2021-03-15T21:41:00.509 INFO:root:teuthology version: 1.1.0-b9656917 2021-03-15T21:41:00.511 DEBUG:teuthology.run:Teuthology command: teuthology --name sage-2021-03-15_21:16:32-rados:cephadm-wip-sage-testing-2021-03-15-1507-distro-basic-smithi --description rados:cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} --archive /home/teuthworker/archive/sage-2021-03-15_21:16:32-rados:cephadm-wip-sage-testing-2021-03-15-1507-distro-basic-smithi/5967954 --verbose --owner scheduled_sage@teuthology -- /home/teuthworker/archive/sage-2021-03-15_21:16:32-rados:cephadm-wip-sage-testing-2021-03-15-1507-distro-basic-smithi/5967954/orig.config.yaml 2021-03-15T21:41:00.550 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-03-15T21:41:00.603 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2021-03-15_21:16:32-rados:cephadm-wip-sage-testing-2021-03-15-1507-distro-basic-smithi/5967954 branch: wip-sage-testing-2021-03-15-1507 description: rados:cephadm/thrash/{0-distro/centos_8.2_kubic_stable 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async root} email: null first_in_suite: false job_id: '5967954' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2021-03-15_21:16:32-rados:cephadm-wip-sage-testing-2021-03-15-1507-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 4 size: 10 os_type: centos os_version: '8.2' overrides: admin_socket: branch: wip-sage-testing-2021-03-15-1507 ceph: conf: global: ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: b658018acf46e32d2ec283d302076352e17476f5 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 cephadm: cephadm_mode: root install: ceph: sha1: b658018acf46e32d2ec283d302076352e17476f5 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: wip-sage-testing-2021-03-15-1507 sha1: b658018acf46e32d2ec283d302076352e17476f5 owner: scheduled_sage@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a sha1: b658018acf46e32d2ec283d302076352e17476f5 sleep_before_teardown: 0 suite: rados:cephadm suite_branch: wip-sage-testing-2021-03-15-1507 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_b658018acf46e32d2ec283d302076352e17476f5/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: b658018acf46e32d2ec283d302076352e17476f5 targets: smithi076.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDOp51JFizPPcnky/2nxgVv0dHwxT4LdlbrEsIfO3v2xBFxZAUEYnx5AMCmlljlv6hgB1eRTycyJpoqf79asGUH2j401sp7akMkzB/Rygmm879okvo5UhjVYdiEqZk3bpec+UcdOJErRY4J/6km93uMNLBllSB3vihhWTlQm4HATeW2rig9pVhfNOzGyqujI9LjtECCtapEg/xmhTFEemWDja2AVZRmdFYyZJ0VfrAAduIsbTN2Y05vatApZbDzszmag9TyZWS9/E1OKmZs787Tbg3l8lJMyJPcIVS6LrBHvDc1IlLCm7ds4IqzuzX5I2na9cYBiLhpHJszBTSCGDFoKO0HnbIjNN+tNOQ5YUV/OR4/1yDcPUrTjoR+JNSJTcu97qfKDKVIul0LHKq0rYTzQWqbn8meICOeLv+oiN/k3wxx2GH2gC5BFIEBmGm7gNI95pie009lh4WLxV3KN8A13CORrQHo+ocU/8tmw5qJ8H2x4+fqvActyP3q6Ap2V+U= smithi125.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZ0SuYpZ9KzamaAdocOx4s0vtG+qbTmhyxoe1dJNwD32ePrey78+bOL3+34r9cbCGZZgWi+dFYjyM6jwtmvN/tNH/cZ5HylH6rSpVolqBvoPwjMDr+WohmP1/iGzhuf6yPJDRXxaL+bu9OlziHxqox3SOwjWD0dLa1avPF6BKVl3xw3iidOqkEqv19i0f8EckohSv/XbxgpHk2zID1XMdfr8hc/A2KgsTgbnSVgmA+drlwk8RPdDKebGxvHRYroiG9eqU4HjzQNu7sZgHFayqZSQwpgXm7M9Gfxhgf0kCYqJr1TB9V7ILu1ILlpZVV2ioGmVskiAA4OFqw3O3748SxXH2Nr1UniFTAnou6et8ncBREJarckbeEpJnUXnk6b6PDdxMIv4GQu5g1rREAXTBfno0DMIPBqYJ3CL0Y06ywAoqBL+UStUW8JUSTgAQwkpfEtBHtuItk7i+34HAxPQJ4dopxGmo9uFoZFMB9jctMNP4AdsqlUYteqB30NzPktC0= tasks: - exec: all: - sudo dnf -y module disable container-tools - sudo dnf -y install 'dnf-command(copr)' - sudo dnf -y copr enable rhcontainerbot/container-selinux - sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo - sudo dnf remove -y podman - sudo dnf -y install podman - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology_branch: master teuthology_sha1: b96569170f15eae4604f361990ea65737b28dff1 timestamp: 2021-03-15_21:16:32 tube: smithi user: sage verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2137 2021-03-15T21:41:00.604 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_b658018acf46e32d2ec283d302076352e17476f5/qa; will attempt to use it 2021-03-15T21:41:00.605 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_b658018acf46e32d2ec283d302076352e17476f5/qa/tasks 2021-03-15T21:41:00.605 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-03-15T21:41:00.607 INFO:teuthology.task.internal:Checking packages... 2021-03-15T21:41:00.637 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash 'b658018acf46e32d2ec283d302076352e17476f5' 2021-03-15T21:41:00.638 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-15T21:41:00.639 INFO:teuthology.packaging:ref: None 2021-03-15T21:41:00.639 INFO:teuthology.packaging:tag: None 2021-03-15T21:41:00.640 INFO:teuthology.packaging:branch: wip-sage-testing-2021-03-15-1507 2021-03-15T21:41:00.640 INFO:teuthology.packaging:sha1: b658018acf46e32d2ec283d302076352e17476f5 2021-03-15T21:41:00.641 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-sage-testing-2021-03-15-1507 2021-03-15T21:41:00.785 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-2063.gb658018a 2021-03-15T21:41:00.785 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-03-15T21:41:00.796 INFO:teuthology.task.internal:no buildpackages task found 2021-03-15T21:41:00.797 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-03-15T21:41:00.866 INFO:teuthology.task.internal:Saving configuration 2021-03-15T21:41:00.880 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-03-15T21:41:00.890 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-15T21:41:00.909 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-15 21:35:34.093843', 'description': '/home/teuthworker/archive/sage-2021-03-15_21:16:32-rados:cephadm-wip-sage-testing-2021-03-15-1507-distro-basic-smithi/5967954', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEk9QZepuaoahO6v6YSw4TbW0R+co8gblbBysObdYFbA3LIITYiYBckhHN2eV40QvN5oiYMMNgk8+VTYmIiiXm8=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi076.front.sepia.ceph.com'} 2021-03-15T21:41:00.927 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-15 21:35:34.093823', 'description': '/home/teuthworker/archive/sage-2021-03-15_21:16:32-rados:cephadm-wip-sage-testing-2021-03-15-1507-distro-basic-smithi/5967954', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFdUFBjjsguwz8rqxWStBWguKiybENiIqursONWCeDt4mggRM99AN13fi3DInwnuCn9Dj2B+0Bz5TzazolpF6cg=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi125.front.sepia.ceph.com'} 2021-03-15T21:41:00.928 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-03-15T21:41:00.938 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2021-03-15T21:41:00.938 INFO:teuthology.task.internal:roles: ubuntu@smithi125.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2021-03-15T21:41:00.938 INFO:teuthology.run_tasks:Running task console_log... 2021-03-15T21:41:01.007 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f19fab0b8c8>, signals=[15]) 2021-03-15T21:41:01.008 INFO:teuthology.run_tasks:Running task internal.connect... 2021-03-15T21:41:01.019 INFO:teuthology.task.internal:Opening connections... 2021-03-15T21:41:01.019 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:41:01.021 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T21:41:01.092 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T21:41:01.093 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T21:41:01.162 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-03-15T21:41:01.173 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2021-03-15T21:41:01.207 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2021-03-15T21:41:01.208 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2021-03-15T21:41:01.261 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Linux" 2021-03-15T21:41:01.262 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="8 (Core)" 2021-03-15T21:41:01.262 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2021-03-15T21:41:01.262 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2021-03-15T21:41:01.262 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="8" 2021-03-15T21:41:01.263 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el8" 2021-03-15T21:41:01.263 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-03-15T21:41:01.263 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2021-03-15T21:41:01.263 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-15T21:41:01.263 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://www.centos.org/" 2021-03-15T21:41:01.264 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-15T21:41:01.264 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:41:01.264 INFO:teuthology.orchestra.run.smithi076.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-15T21:41:01.264 INFO:teuthology.orchestra.run.smithi076.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-15T21:41:01.265 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-15T21:41:01.265 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-03-15T21:41:01.266 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:41:01.267 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2021-03-15T21:41:01.285 DEBUG:teuthology.orchestra.run.smithi125:> uname -m 2021-03-15T21:41:01.301 INFO:teuthology.orchestra.run.smithi125.stdout:x86_64 2021-03-15T21:41:01.302 DEBUG:teuthology.orchestra.run.smithi125:> cat /etc/os-release 2021-03-15T21:41:01.355 INFO:teuthology.orchestra.run.smithi125.stdout:NAME="CentOS Linux" 2021-03-15T21:41:01.356 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION="8 (Core)" 2021-03-15T21:41:01.356 INFO:teuthology.orchestra.run.smithi125.stdout:ID="centos" 2021-03-15T21:41:01.356 INFO:teuthology.orchestra.run.smithi125.stdout:ID_LIKE="rhel fedora" 2021-03-15T21:41:01.357 INFO:teuthology.orchestra.run.smithi125.stdout:VERSION_ID="8" 2021-03-15T21:41:01.357 INFO:teuthology.orchestra.run.smithi125.stdout:PLATFORM_ID="platform:el8" 2021-03-15T21:41:01.357 INFO:teuthology.orchestra.run.smithi125.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-03-15T21:41:01.357 INFO:teuthology.orchestra.run.smithi125.stdout:ANSI_COLOR="0;31" 2021-03-15T21:41:01.357 INFO:teuthology.orchestra.run.smithi125.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-03-15T21:41:01.358 INFO:teuthology.orchestra.run.smithi125.stdout:HOME_URL="https://www.centos.org/" 2021-03-15T21:41:01.358 INFO:teuthology.orchestra.run.smithi125.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-03-15T21:41:01.358 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:41:01.358 INFO:teuthology.orchestra.run.smithi125.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-03-15T21:41:01.359 INFO:teuthology.orchestra.run.smithi125.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-03-15T21:41:01.359 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-03-15T21:41:01.359 INFO:teuthology.orchestra.run.smithi125.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-03-15T21:41:01.359 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:41:01.360 INFO:teuthology.lock.ops:Updating smithi125.front.sepia.ceph.com on lock server 2021-03-15T21:41:01.416 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-03-15T21:41:01.431 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-03-15T21:41:01.465 INFO:teuthology.task.internal:Checking for old test directory... 2021-03-15T21:41:01.465 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2021-03-15T21:41:01.468 DEBUG:teuthology.orchestra.run.smithi125:> test '!' -e /home/ubuntu/cephtest 2021-03-15T21:41:01.482 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-03-15T21:41:01.492 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-03-15T21:41:01.493 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2021-03-15T21:41:01.523 DEBUG:teuthology.orchestra.run.smithi125:> test -z $(ls -A /var/lib/ceph) 2021-03-15T21:41:01.552 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-03-15T21:41:01.602 INFO:teuthology.run_tasks:Running task kernel... 2021-03-15T21:41:01.623 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.y': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}, 'osd.3': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'ceph.rgw.foo.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.a': {'kdb': True, 'sha1': 'distro'}, 'alertmanager.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.4': {'kdb': True, 'sha1': 'distro'}, 'osd.5': {'kdb': True, 'sha1': 'distro'}, 'osd.6': {'kdb': True, 'sha1': 'distro'}, 'osd.7': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'prometheus.a': {'kdb': True, 'sha1': 'distro'}, 'grafana.a': {'kdb': True, 'sha1': 'distro'}, 'node-exporter.b': {'kdb': True, 'sha1': 'distro'}, 'ceph.iscsi.iscsi.a': {'kdb': True, 'sha1': 'distro'}} 2021-03-15T21:41:01.624 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-03-15T21:41:01.625 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2021-03-15T21:41:01.639 INFO:teuthology.orchestra.run.smithi076.stdout:4.18.0-193.28.1.el8_2.x86_64 2021-03-15T21:41:01.640 INFO:teuthology.task.kernel:Running kernel on smithi076: 4.18.0-193.28.1.el8_2.x86_64 2021-03-15T21:41:01.640 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum install -y kernel 2021-03-15T21:41:04.404 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-8 - AppStream 25 MB/s | 6.3 MB 00:00 2021-03-15T21:41:05.483 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-8 - Base 26 MB/s | 2.3 MB 00:00 2021-03-15T21:41:05.925 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-8 - Extras 143 kB/s | 9.2 kB 00:00 2021-03-15T21:41:06.351 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 24 MB/s | 8.9 MB 00:00 2021-03-15T21:41:08.202 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2021-03-15T21:41:09.217 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-4.18.0-193.el8.x86_64 is already installed. 2021-03-15T21:41:09.217 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-4.18.0-193.28.1.el8_2.x86_64 is already installed. 2021-03-15T21:41:09.405 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:41:09.413 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T21:41:09.413 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:41:09.413 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T21:41:09.413 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:41:09.414 INFO:teuthology.orchestra.run.smithi076.stdout: kernel x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 4.3 M 2021-03-15T21:41:09.414 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrading: 2021-03-15T21:41:09.414 INFO:teuthology.orchestra.run.smithi076.stdout: linux-firmware noarch 20200619-101.git3890db36.el8_3 CentOS-Base 101 M 2021-03-15T21:41:09.414 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:41:09.415 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 30 M 2021-03-15T21:41:09.415 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 26 M 2021-03-15T21:41:09.415 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:41:09.415 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:41:09.416 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T21:41:09.416 INFO:teuthology.orchestra.run.smithi076.stdout:Install 3 Packages 2021-03-15T21:41:09.416 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 1 Package 2021-03-15T21:41:09.416 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:41:09.417 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 162 M 2021-03-15T21:41:09.417 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:41:09.720 INFO:teuthology.orchestra.run.smithi076.stdout:(1/4): kernel-4.18.0-240.15.1.el8_3.x86_64.rpm 14 MB/s | 4.3 MB 00:00 2021-03-15T21:41:10.152 INFO:teuthology.orchestra.run.smithi076.stdout:(2/4): kernel-core-4.18.0-240.15.1.el8_3.x86_64 41 MB/s | 30 MB 00:00 2021-03-15T21:41:10.447 INFO:teuthology.orchestra.run.smithi076.stdout:(3/4): kernel-modules-4.18.0-240.15.1.el8_3.x86 25 MB/s | 26 MB 00:01 2021-03-15T21:41:12.061 INFO:teuthology.orchestra.run.smithi076.stdout:(4/4): linux-firmware-20200619-101.git3890db36. 43 MB/s | 101 MB 00:02 2021-03-15T21:41:12.064 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:41:12.064 INFO:teuthology.orchestra.run.smithi076.stdout:Total 61 MB/s | 162 MB 00:02 2021-03-15T21:41:12.155 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:41:12.441 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:41:12.442 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:41:14.134 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:41:14.135 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:41:24.185 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:41:25.962 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : linux-firmware-20200619-101.git3890db36.el8_3.noarch 1/5 2021-03-15T21:41:26.514 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-core-4.18.0-240.15.1.el8_3.x86_64 2/5 2021-03-15T21:41:29.202 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-4.18.0-240.15.1.el8_3.x86_64 2/5 2021-03-15T21:41:29.539 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-4.18.0-240.15.1.el8_3.x86_64 3/5 2021-03-15T21:41:32.416 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-4.18.0-240.15.1.el8_3.x86_64 3/5 2021-03-15T21:41:32.597 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-4.18.0-240.15.1.el8_3.x86_64 4/5 2021-03-15T21:41:32.833 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-03-15T21:41:52.365 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-4.18.0-240.15.1.el8_3.x86_64 5/5 2021-03-15T21:41:52.662 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-03-15T21:41:52.663 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-4.18.0-240.15.1.el8_3.x86_64 1/5 2021-03-15T21:41:52.663 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-core-4.18.0-240.15.1.el8_3.x86_64 2/5 2021-03-15T21:41:52.663 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-4.18.0-240.15.1.el8_3.x86_64 3/5 2021-03-15T21:41:52.663 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : linux-firmware-20200619-101.git3890db36.el8_3.noarch 4/5 2021-03-15T21:41:53.090 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-03-15T21:41:53.090 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:41:53.090 INFO:teuthology.orchestra.run.smithi076.stdout:Upgraded: 2021-03-15T21:41:53.094 INFO:teuthology.orchestra.run.smithi076.stdout: linux-firmware-20200619-101.git3890db36.el8_3.noarch 2021-03-15T21:41:53.094 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:41:53.095 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:41:53.095 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:41:53.095 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core-4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:41:53.095 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:41:53.096 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:41:53.096 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:41:53.297 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel --last | head -n 1 2021-03-15T21:41:53.402 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-4.18.0-240.15.1.el8_3.x86_64 Mon 15 Mar 2021 09:41:32 PM UTC 2021-03-15T21:41:53.403 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:41:53.404 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-193.28.1.el8_2.x86_64 Expected: 4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:41:53.404 DEBUG:teuthology.orchestra.run.smithi125:> uname -r 2021-03-15T21:41:53.421 INFO:teuthology.orchestra.run.smithi125.stdout:4.18.0-193.28.1.el8_2.x86_64 2021-03-15T21:41:53.422 INFO:teuthology.task.kernel:Running kernel on smithi125: 4.18.0-193.28.1.el8_2.x86_64 2021-03-15T21:41:53.422 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum install -y kernel 2021-03-15T21:41:56.089 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS-8 - AppStream 35 MB/s | 6.3 MB 00:00 2021-03-15T21:41:57.171 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS-8 - Base 26 MB/s | 2.3 MB 00:00 2021-03-15T21:41:57.620 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS-8 - Extras 142 kB/s | 9.2 kB 00:00 2021-03-15T21:41:58.023 INFO:teuthology.orchestra.run.smithi125.stdout:Extra Packages for Enterprise Linux 26 MB/s | 8.9 MB 00:00 2021-03-15T21:41:59.869 INFO:teuthology.orchestra.run.smithi125.stdout:lab-extras 4.3 MB/s | 24 kB 00:00 2021-03-15T21:42:00.907 INFO:teuthology.orchestra.run.smithi125.stdout:Package kernel-4.18.0-193.el8.x86_64 is already installed. 2021-03-15T21:42:00.908 INFO:teuthology.orchestra.run.smithi125.stdout:Package kernel-4.18.0-193.28.1.el8_2.x86_64 is already installed. 2021-03-15T21:42:01.092 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T21:42:01.101 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T21:42:01.102 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T21:42:01.102 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T21:42:01.102 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T21:42:01.102 INFO:teuthology.orchestra.run.smithi125.stdout: kernel x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 4.3 M 2021-03-15T21:42:01.102 INFO:teuthology.orchestra.run.smithi125.stdout:Upgrading: 2021-03-15T21:42:01.103 INFO:teuthology.orchestra.run.smithi125.stdout: linux-firmware noarch 20200619-101.git3890db36.el8_3 CentOS-Base 101 M 2021-03-15T21:42:01.103 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T21:42:01.103 INFO:teuthology.orchestra.run.smithi125.stdout: kernel-core x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 30 M 2021-03-15T21:42:01.103 INFO:teuthology.orchestra.run.smithi125.stdout: kernel-modules x86_64 4.18.0-240.15.1.el8_3 CentOS-Base 26 M 2021-03-15T21:42:01.104 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:42:01.104 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T21:42:01.104 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T21:42:01.104 INFO:teuthology.orchestra.run.smithi125.stdout:Install 3 Packages 2021-03-15T21:42:01.104 INFO:teuthology.orchestra.run.smithi125.stdout:Upgrade 1 Package 2021-03-15T21:42:01.105 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:42:01.105 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 162 M 2021-03-15T21:42:01.105 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T21:42:01.206 INFO:teuthology.orchestra.run.smithi125.stdout:(1/4): kernel-4.18.0-240.15.1.el8_3.x86_64.rpm 42 MB/s | 4.3 MB 00:00 2021-03-15T21:42:02.459 INFO:teuthology.orchestra.run.smithi125.stdout:(2/4): kernel-modules-4.18.0-240.15.1.el8_3.x86 19 MB/s | 26 MB 00:01 2021-03-15T21:42:02.710 INFO:teuthology.orchestra.run.smithi125.stdout:(3/4): kernel-core-4.18.0-240.15.1.el8_3.x86_64 19 MB/s | 30 MB 00:01 2021-03-15T21:42:03.645 INFO:teuthology.orchestra.run.smithi125.stdout:(4/4): linux-firmware-20200619-101.git3890db36. 42 MB/s | 101 MB 00:02 2021-03-15T21:42:03.649 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:42:03.649 INFO:teuthology.orchestra.run.smithi125.stdout:Total 63 MB/s | 162 MB 00:02 2021-03-15T21:42:03.740 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T21:42:04.017 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T21:42:04.017 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T21:42:05.808 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T21:42:05.808 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T21:42:15.803 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T21:42:17.606 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : linux-firmware-20200619-101.git3890db36.el8_3.noarch 1/5 2021-03-15T21:42:18.151 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : kernel-core-4.18.0-240.15.1.el8_3.x86_64 2/5 2021-03-15T21:42:20.796 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: kernel-core-4.18.0-240.15.1.el8_3.x86_64 2/5 2021-03-15T21:42:21.109 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : kernel-modules-4.18.0-240.15.1.el8_3.x86_64 3/5 2021-03-15T21:42:23.989 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: kernel-modules-4.18.0-240.15.1.el8_3.x86_64 3/5 2021-03-15T21:42:24.198 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : kernel-4.18.0-240.15.1.el8_3.x86_64 4/5 2021-03-15T21:42:24.418 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-03-15T21:42:44.010 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: kernel-core-4.18.0-240.15.1.el8_3.x86_64 5/5 2021-03-15T21:42:44.326 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-03-15T21:42:44.327 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : kernel-4.18.0-240.15.1.el8_3.x86_64 1/5 2021-03-15T21:42:44.327 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : kernel-core-4.18.0-240.15.1.el8_3.x86_64 2/5 2021-03-15T21:42:44.327 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : kernel-modules-4.18.0-240.15.1.el8_3.x86_64 3/5 2021-03-15T21:42:44.327 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : linux-firmware-20200619-101.git3890db36.el8_3.noarch 4/5 2021-03-15T21:42:44.733 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-03-15T21:42:44.733 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:42:44.734 INFO:teuthology.orchestra.run.smithi125.stdout:Upgraded: 2021-03-15T21:42:44.734 INFO:teuthology.orchestra.run.smithi125.stdout: linux-firmware-20200619-101.git3890db36.el8_3.noarch 2021-03-15T21:42:44.734 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:42:44.734 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T21:42:44.735 INFO:teuthology.orchestra.run.smithi125.stdout: kernel-4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:42:44.735 INFO:teuthology.orchestra.run.smithi125.stdout: kernel-core-4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:42:44.735 INFO:teuthology.orchestra.run.smithi125.stdout: kernel-modules-4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:42:44.735 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:42:44.736 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T21:42:44.908 DEBUG:teuthology.orchestra.run.smithi125:> rpm -q kernel --last | head -n 1 2021-03-15T21:42:45.020 INFO:teuthology.orchestra.run.smithi125.stdout:kernel-4.18.0-240.15.1.el8_3.x86_64 Mon 15 Mar 2021 09:42:23 PM UTC 2021-03-15T21:42:45.022 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:42:45.022 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-193.28.1.el8_2.x86_64 Expected: 4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:42:45.023 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-03-15T21:42:45.023 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-03-15T21:42:45.023 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-03-15T21:42:45.024 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-03-15T21:42:45.024 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2021-03-15T21:42:45.025 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel --last | head -n 1 2021-03-15T21:42:45.120 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-4.18.0-240.15.1.el8_3.x86_64 Mon 15 Mar 2021 09:41:32 PM UTC 2021-03-15T21:42:45.122 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:42:45.122 DEBUG:teuthology.orchestra.run.smithi076:> sudo rpm -qi grub2-tools 2021-03-15T21:42:45.154 INFO:teuthology.orchestra.run.smithi076.stdout:Name : grub2-tools 2021-03-15T21:42:45.155 INFO:teuthology.orchestra.run.smithi076.stdout:Epoch : 1 2021-03-15T21:42:45.155 INFO:teuthology.orchestra.run.smithi076.stdout:Version : 2.02 2021-03-15T21:42:45.155 INFO:teuthology.orchestra.run.smithi076.stdout:Release : 81.el8 2021-03-15T21:42:45.156 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: x86_64 2021-03-15T21:42:45.156 INFO:teuthology.orchestra.run.smithi076.stdout:Install Date: Fri 13 Nov 2020 03:02:55 PM UTC 2021-03-15T21:42:45.156 INFO:teuthology.orchestra.run.smithi076.stdout:Group : System Environment/Base 2021-03-15T21:42:45.156 INFO:teuthology.orchestra.run.smithi076.stdout:Size : 9192705 2021-03-15T21:42:45.156 INFO:teuthology.orchestra.run.smithi076.stdout:License : GPLv3+ 2021-03-15T21:42:45.157 INFO:teuthology.orchestra.run.smithi076.stdout:Signature : RSA/SHA256, Tue 14 Apr 2020 03:18:37 PM UTC, Key ID 05b555b38483c65d 2021-03-15T21:42:45.157 INFO:teuthology.orchestra.run.smithi076.stdout:Source RPM : grub2-2.02-81.el8.src.rpm 2021-03-15T21:42:45.157 INFO:teuthology.orchestra.run.smithi076.stdout:Build Date : Tue 14 Apr 2020 02:54:04 PM UTC 2021-03-15T21:42:45.157 INFO:teuthology.orchestra.run.smithi076.stdout:Build Host : kbuilder.bsys.centos.org 2021-03-15T21:42:45.158 INFO:teuthology.orchestra.run.smithi076.stdout:Relocations : (not relocatable) 2021-03-15T21:42:45.158 INFO:teuthology.orchestra.run.smithi076.stdout:Packager : CentOS BuildSystem 2021-03-15T21:42:45.158 INFO:teuthology.orchestra.run.smithi076.stdout:Vendor : CentOS 2021-03-15T21:42:45.158 INFO:teuthology.orchestra.run.smithi076.stdout:URL : http://www.gnu.org/software/grub/ 2021-03-15T21:42:45.158 INFO:teuthology.orchestra.run.smithi076.stdout:Summary : Support tools for GRUB. 2021-03-15T21:42:45.159 INFO:teuthology.orchestra.run.smithi076.stdout:Description : 2021-03-15T21:42:45.159 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:42:45.159 INFO:teuthology.orchestra.run.smithi076.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-03-15T21:42:45.159 INFO:teuthology.orchestra.run.smithi076.stdout:customizable bootloader with modular architecture. It supports a rich 2021-03-15T21:42:45.159 INFO:teuthology.orchestra.run.smithi076.stdout:variety of kernel formats, file systems, computer architectures and 2021-03-15T21:42:45.160 INFO:teuthology.orchestra.run.smithi076.stdout:hardware devices. 2021-03-15T21:42:45.160 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:42:45.160 INFO:teuthology.orchestra.run.smithi076.stdout:This subpackage provides tools for support of all platforms. 2021-03-15T21:42:45.161 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-03-15T21:42:45.162 INFO:teuthology.task.kernel:Updating grub on smithi076 to boot 4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:42:45.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-03-15T21:42:45.859 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2021-03-15T21:42:47.831 INFO:teuthology.orchestra.run.smithi076.stderr:done 2021-03-15T21:42:47.833 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2021-03-15T21:42:47.848 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.Q7nAZUKS36 2021-03-15T21:42:47.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.Q7nAZUKS36 2021-03-15T21:42:47.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.Q7nAZUKS36 2021-03-15T21:42:48.069 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.Q7nAZUKS36 is 4KB 2021-03-15T21:42:48.117 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.Q7nAZUKS36 2021-03-15T21:42:48.131 DEBUG:teuthology.orchestra.run.smithi076:> sudo /bin/ls /boot/loader/entries || true 2021-03-15T21:42:48.193 INFO:teuthology.orchestra.run.smithi076.stdout:2951a06464bf41f6a3bf91594147ecff-0-rescue.conf 2021-03-15T21:42:48.194 INFO:teuthology.orchestra.run.smithi076.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-193.28.1.el8_2.x86_64.conf 2021-03-15T21:42:48.194 INFO:teuthology.orchestra.run.smithi076.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-193.el8.x86_64.conf 2021-03-15T21:42:48.194 INFO:teuthology.orchestra.run.smithi076.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-240.15.1.el8_3.x86_64.conf 2021-03-15T21:42:48.195 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-set-default 2951a06464bf41f6a3bf91594147ecff-4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:42:48.358 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2021-03-15T21:42:48.361 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2021-03-15T21:42:48.361 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi125.front.sepia.ceph.com, path=None, version=distro) 2021-03-15T21:42:48.361 DEBUG:teuthology.orchestra.run.smithi125:> rpm -q kernel --last | head -n 1 2021-03-15T21:42:48.456 INFO:teuthology.orchestra.run.smithi125.stdout:kernel-4.18.0-240.15.1.el8_3.x86_64 Mon 15 Mar 2021 09:42:23 PM UTC 2021-03-15T21:42:48.457 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:42:48.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo rpm -qi grub2-tools 2021-03-15T21:42:48.488 INFO:teuthology.orchestra.run.smithi125.stdout:Name : grub2-tools 2021-03-15T21:42:48.489 INFO:teuthology.orchestra.run.smithi125.stdout:Epoch : 1 2021-03-15T21:42:48.489 INFO:teuthology.orchestra.run.smithi125.stdout:Version : 2.02 2021-03-15T21:42:48.489 INFO:teuthology.orchestra.run.smithi125.stdout:Release : 81.el8 2021-03-15T21:42:48.489 INFO:teuthology.orchestra.run.smithi125.stdout:Architecture: x86_64 2021-03-15T21:42:48.490 INFO:teuthology.orchestra.run.smithi125.stdout:Install Date: Fri 13 Nov 2020 03:02:55 PM UTC 2021-03-15T21:42:48.490 INFO:teuthology.orchestra.run.smithi125.stdout:Group : System Environment/Base 2021-03-15T21:42:48.490 INFO:teuthology.orchestra.run.smithi125.stdout:Size : 9192705 2021-03-15T21:42:48.490 INFO:teuthology.orchestra.run.smithi125.stdout:License : GPLv3+ 2021-03-15T21:42:48.491 INFO:teuthology.orchestra.run.smithi125.stdout:Signature : RSA/SHA256, Tue 14 Apr 2020 03:18:37 PM UTC, Key ID 05b555b38483c65d 2021-03-15T21:42:48.491 INFO:teuthology.orchestra.run.smithi125.stdout:Source RPM : grub2-2.02-81.el8.src.rpm 2021-03-15T21:42:48.491 INFO:teuthology.orchestra.run.smithi125.stdout:Build Date : Tue 14 Apr 2020 02:54:04 PM UTC 2021-03-15T21:42:48.491 INFO:teuthology.orchestra.run.smithi125.stdout:Build Host : kbuilder.bsys.centos.org 2021-03-15T21:42:48.491 INFO:teuthology.orchestra.run.smithi125.stdout:Relocations : (not relocatable) 2021-03-15T21:42:48.492 INFO:teuthology.orchestra.run.smithi125.stdout:Packager : CentOS BuildSystem 2021-03-15T21:42:48.492 INFO:teuthology.orchestra.run.smithi125.stdout:Vendor : CentOS 2021-03-15T21:42:48.492 INFO:teuthology.orchestra.run.smithi125.stdout:URL : http://www.gnu.org/software/grub/ 2021-03-15T21:42:48.492 INFO:teuthology.orchestra.run.smithi125.stdout:Summary : Support tools for GRUB. 2021-03-15T21:42:48.493 INFO:teuthology.orchestra.run.smithi125.stdout:Description : 2021-03-15T21:42:48.493 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:42:48.493 INFO:teuthology.orchestra.run.smithi125.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-03-15T21:42:48.493 INFO:teuthology.orchestra.run.smithi125.stdout:customizable bootloader with modular architecture. It supports a rich 2021-03-15T21:42:48.493 INFO:teuthology.orchestra.run.smithi125.stdout:variety of kernel formats, file systems, computer architectures and 2021-03-15T21:42:48.494 INFO:teuthology.orchestra.run.smithi125.stdout:hardware devices. 2021-03-15T21:42:48.494 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:42:48.494 INFO:teuthology.orchestra.run.smithi125.stdout:This subpackage provides tools for support of all platforms. 2021-03-15T21:42:48.495 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-03-15T21:42:48.496 INFO:teuthology.task.kernel:Updating grub on smithi125 to boot 4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:42:48.496 DEBUG:teuthology.orchestra.run.smithi125:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-03-15T21:42:49.951 INFO:teuthology.orchestra.run.smithi125.stderr:Generating grub configuration file ... 2021-03-15T21:42:52.177 INFO:teuthology.orchestra.run.smithi125.stderr:done 2021-03-15T21:42:52.179 DEBUG:teuthology.orchestra.run.smithi125:> mktemp 2021-03-15T21:42:52.203 INFO:teuthology.orchestra.run.smithi125.stdout:/tmp/tmp.417nxrNxlV 2021-03-15T21:42:52.204 DEBUG:teuthology.orchestra.run.smithi125:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.417nxrNxlV 2021-03-15T21:42:52.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0666 /tmp/tmp.417nxrNxlV 2021-03-15T21:42:52.423 DEBUG:teuthology.orchestra.remote:smithi125:/tmp/tmp.417nxrNxlV is 4KB 2021-03-15T21:42:52.470 DEBUG:teuthology.orchestra.run.smithi125:> rm -fr /tmp/tmp.417nxrNxlV 2021-03-15T21:42:52.484 DEBUG:teuthology.orchestra.run.smithi125:> sudo /bin/ls /boot/loader/entries || true 2021-03-15T21:42:52.545 INFO:teuthology.orchestra.run.smithi125.stdout:2951a06464bf41f6a3bf91594147ecff-0-rescue.conf 2021-03-15T21:42:52.546 INFO:teuthology.orchestra.run.smithi125.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-193.28.1.el8_2.x86_64.conf 2021-03-15T21:42:52.546 INFO:teuthology.orchestra.run.smithi125.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-193.el8.x86_64.conf 2021-03-15T21:42:52.546 INFO:teuthology.orchestra.run.smithi125.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-240.15.1.el8_3.x86_64.conf 2021-03-15T21:42:52.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo grub2-set-default 2951a06464bf41f6a3bf91594147ecff-4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:42:52.690 DEBUG:teuthology.orchestra.run.smithi125:> sudo shutdown -r now 2021-03-15T21:43:22.722 INFO:teuthology.misc:Re-opening connections... 2021-03-15T21:43:22.723 INFO:teuthology.misc:trying to connect to ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:43:22.723 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T21:43:22.724 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T21:43:57.378 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2021-03-15T21:43:57.380 INFO:teuthology.misc:trying to connect to ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T21:43:57.380 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T21:43:57.381 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T21:44:57.433 DEBUG:teuthology.orchestra.remote:timed out 2021-03-15T21:44:57.435 DEBUG:teuthology.misc:waited 94.71205306053162 2021-03-15T21:44:58.437 INFO:teuthology.misc:trying to connect to ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:44:58.437 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T21:44:58.438 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T21:44:58.785 DEBUG:teuthology.orchestra.run.smithi076:> true 2021-03-15T21:44:59.104 DEBUG:teuthology.misc:waited 96.38125467300415 2021-03-15T21:45:00.106 INFO:teuthology.misc:trying to connect to ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T21:45:00.106 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T21:45:00.107 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T21:45:00.435 DEBUG:teuthology.orchestra.run.smithi125:> true 2021-03-15T21:45:00.744 DEBUG:teuthology.misc:waited 98.02058863639832 2021-03-15T21:45:01.744 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-03-15T21:45:01.745 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-240.15.1.el8_3.x86_64"... 2021-03-15T21:45:01.745 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2021-03-15T21:45:01.762 INFO:teuthology.orchestra.run.smithi076.stdout:4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:45:01.763 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-240.15.1.el8_3.x86_64 vs 4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:45:01.763 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-03-15T21:45:01.764 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-03-15T21:45:01.764 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2021-03-15T21:45:01.764 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "4.18.0-240.15.1.el8_3.x86_64"... 2021-03-15T21:45:01.765 DEBUG:teuthology.orchestra.run.smithi125:> uname -r 2021-03-15T21:45:01.782 INFO:teuthology.orchestra.run.smithi125.stdout:4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:45:01.782 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-240.15.1.el8_3.x86_64 vs 4.18.0-240.15.1.el8_3.x86_64 2021-03-15T21:45:01.782 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-03-15T21:45:01.783 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-03-15T21:45:02.784 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-03-15T21:45:02.785 DEBUG:teuthology.orchestra.run.smithi076:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-15T21:45:02.879 INFO:teuthology.orchestra.run.smithi076.stdout:ttyS1 2021-03-15T21:45:02.892 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2021-03-15T21:45:02.893 DEBUG:teuthology.orchestra.run.smithi125:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-15T21:45:02.972 INFO:teuthology.orchestra.run.smithi125.stdout:ttyS1 2021-03-15T21:45:02.988 INFO:teuthology.run_tasks:Running task internal.base... 2021-03-15T21:45:02.998 INFO:teuthology.task.internal:Creating test directory... 2021-03-15T21:45:02.999 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-15T21:45:03.002 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-15T21:45:03.018 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-03-15T21:45:03.029 INFO:teuthology.run_tasks:Running task internal.archive... 2021-03-15T21:45:03.041 INFO:teuthology.task.internal:Creating archive directory... 2021-03-15T21:45:03.042 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-15T21:45:03.059 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-15T21:45:03.088 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-03-15T21:45:03.099 INFO:teuthology.task.internal:Enabling coredump saving... 2021-03-15T21:45:03.099 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-15T21:45:03.116 DEBUG:teuthology.orchestra.run.smithi125:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-15T21:45:03.143 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-15T21:45:03.150 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-15T21:45:03.154 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-15T21:45:03.162 INFO:teuthology.orchestra.run.smithi125.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-15T21:45:03.163 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-03-15T21:45:03.174 INFO:teuthology.task.internal:Configuring sudo... 2021-03-15T21:45:03.175 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-15T21:45:03.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-15T21:45:03.230 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-03-15T21:45:03.242 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-03-15T21:45:03.243 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-15T21:45:03.267 DEBUG:teuthology.orchestra.run.smithi125:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-15T21:45:03.284 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-15T21:45:03.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-15T21:45:03.436 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-15T21:45:03.470 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-15T21:45:03.519 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T21:45:03.519 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-15T21:45:03.583 DEBUG:teuthology.orchestra.run.smithi125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-15T21:45:03.619 DEBUG:teuthology.orchestra.run.smithi125:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-15T21:45:03.681 DEBUG:teuthology.orchestra.run.smithi125:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-15T21:45:03.756 DEBUG:teuthology.orchestra.run.smithi125:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-15T21:45:03.806 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-15T21:45:03.806 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-15T21:45:03.872 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2021-03-15T21:45:03.874 DEBUG:teuthology.orchestra.run.smithi125:> sudo service rsyslog restart 2021-03-15T21:45:03.919 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-03-15T21:45:03.970 INFO:teuthology.orchestra.run.smithi125.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-03-15T21:45:04.297 INFO:teuthology.run_tasks:Running task internal.timer... 2021-03-15T21:45:04.308 INFO:teuthology.task.internal:Starting timer... 2021-03-15T21:45:04.308 INFO:teuthology.run_tasks:Running task pcp... 2021-03-15T21:45:04.346 INFO:teuthology.run_tasks:Running task selinux... 2021-03-15T21:45:04.455 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-03-15T21:45:04.455 DEBUG:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2021-03-15T21:45:04.533 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: [ OK ] 2021-03-15T21:45:04.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo service auditd rotate 2021-03-15T21:45:04.610 INFO:teuthology.orchestra.run.smithi125.stdout:Rotating logs: [ OK ] 2021-03-15T21:45:04.615 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-03-15T21:45:04.616 DEBUG:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2021-03-15T21:45:04.639 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2021-03-15T21:45:04.639 DEBUG:teuthology.orchestra.run.smithi125:> /usr/sbin/getenforce 2021-03-15T21:45:04.689 INFO:teuthology.orchestra.run.smithi125.stdout:Permissive 2021-03-15T21:45:04.689 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi076.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi125.front.sepia.ceph.com': 'permissive'} 2021-03-15T21:45:04.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-03-15T21:45:04.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T21:45:04.715 DEBUG:teuthology.orchestra.run.smithi125:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|comm="sssd"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-03-15T21:45:04.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T21:45:04.742 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-03-15T21:45:04.743 DEBUG:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2021-03-15T21:45:04.780 DEBUG:teuthology.orchestra.run.smithi125:> sudo /usr/sbin/setenforce permissive 2021-03-15T21:45:04.816 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-03-15T21:45:04.849 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2021-03-15T21:45:04.854 INFO:teuthology.repo_utils:Fetching master from origin 2021-03-15T21:45:04.904 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-03-15T21:45:04.927 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-03-15T21:45:04.928 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi076.front.sepia.ceph.com,smithi125.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-03-15T21:45:05.373 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-03-15T21:45:06.209 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-03-15T21:45:08.304 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-03-15T21:45:08.327 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:08 +0000 (0:00:00.040) 0:00:00.040 ********** 2021-03-15T21:45:08.339 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:08 +0000 (0:00:00.011) 0:00:00.051 ********** 2021-03-15T21:45:09.926 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi125.front.sepia.ceph.com] 2021-03-15T21:45:09.978 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:09 +0000 (0:00:01.639) 0:00:01.690 ********** 2021-03-15T21:45:09.988 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] 2021-03-15T21:45:10.037 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:10 +0000 (0:00:00.058) 0:00:01.749 ********** 2021-03-15T21:45:10.304 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-03-15T21:45:10.306 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-03-15T21:45:10.327 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:10 +0000 (0:00:00.290) 0:00:02.039 ********** 2021-03-15T21:45:10.347 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:10 +0000 (0:00:00.017) 0:00:02.057 ********** 2021-03-15T21:45:10.905 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-03-15T21:45:10.906 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} ok: [smithi076.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-03-15T21:45:10.928 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:10 +0000 (0:00:00.583) 0:00:02.640 ********** 2021-03-15T21:45:10.945 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:10 +0000 (0:00:00.017) 0:00:02.658 ********** 2021-03-15T21:45:11.014 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-03-15T21:45:11.016 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:11.036 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:11 +0000 (0:00:00.090) 0:00:02.748 ********** 2021-03-15T21:45:11.054 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:11 +0000 (0:00:00.018) 0:00:02.766 ********** 2021-03-15T21:45:11.352 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** ok: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-15T21:45:11.354 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-15T21:45:11.375 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:11 +0000 (0:00:00.321) 0:00:03.088 ********** 2021-03-15T21:45:11.394 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:11 +0000 (0:00:00.018) 0:00:03.106 ********** 2021-03-15T21:45:12.233 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-03-15T21:45:12.235 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615844711.4366019-15584-94222377104127/source", "state": "file", "uid": 0} 2021-03-15T21:45:12.255 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:12 +0000 (0:00:00.861) 0:00:03.968 ********** 2021-03-15T21:45:12.262 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615844711.417423-15581-181798339222554/source", "state": "file", "uid": 0} 2021-03-15T21:45:12.285 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:12 +0000 (0:00:00.029) 0:00:03.997 ********** 2021-03-15T21:45:12.810 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-03-15T21:45:12.812 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-15T21:45:12.813 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail 2021-03-15T21:45:12.815 INFO:teuthology.task.ansible.out:.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@ 2021-03-15T21:45:12.816 INFO:teuthology.task.ansible.out:aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-15T21:45:12.894 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-03-15T21:45:12.897 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-03-15T21:45:13.043 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:13 +0000 (0:00:00.758) 0:00:04.756 ********** 2021-03-15T21:45:13.061 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:13 +0000 (0:00:00.017) 0:00:04.773 ********** 2021-03-15T21:45:13.677 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi076.front.sepia.ceph.com] 2021-03-15T21:45:13.708 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] 2021-03-15T21:45:13.739 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Monday 15 March 2021 21:45:13 +0000 (0:00:00.678) 0:00:05.452 ********** 2021-03-15T21:45:13.783 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:13 +0000 (0:00:00.043) 0:00:05.495 ********** 2021-03-15T21:45:14.388 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-03-15T21:45:14.390 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.141016", "end": "2021-03-15 21:45:14.360687", "rc": 0, "start": "2021-03-15 21:45:14.219671", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-15T21:45:14.410 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:14 +0000 (0:00:00.626) 0:00:06.122 ********** 2021-03-15T21:45:14.416 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.152102", "end": "2021-03-15 21:45:14.373083", "rc": 0, "start": "2021-03-15 21:45:14.220981", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-15T21:45:14.440 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:14 +0000 (0:00:00.030) 0:00:06.153 ********** 2021-03-15T21:45:14.613 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-15T21:45:14.615 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.018430", "end": "2021-03-15 21:45:14.597213", "failed_when_result": false, "rc": 0, "start": "2021-03-15 21:45:14.578783", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-03-15T21:45:14.637 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:14 +0000 (0:00:00.196) 0:00:06.349 ********** 2021-03-15T21:45:14.654 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.017856", "end": "2021-03-15 21:45:14.630382", "failed_when_result": false, "rc": 0, "start": "2021-03-15 21:45:14.612526", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-03-15T21:45:14.678 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:14 +0000 (0:00:00.041) 0:00:06.391 ********** 2021-03-15T21:45:14.716 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-15T21:45:14.718 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:14.739 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:14 +0000 (0:00:00.060) 0:00:06.451 ********** 2021-03-15T21:45:14.755 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:14.778 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:14 +0000 (0:00:00.039) 0:00:06.491 ********** 2021-03-15T21:45:14.816 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:14.845 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:14 +0000 (0:00:00.066) 0:00:06.558 ********** 2021-03-15T21:45:14.855 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:14.883 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:14 +0000 (0:00:00.037) 0:00:06.596 ********** 2021-03-15T21:45:14.921 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-03-15T21:45:14.923 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:14.945 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:14 +0000 (0:00:00.061) 0:00:06.657 ********** 2021-03-15T21:45:14.951 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:14.953 INFO:teuthology.task.ansible.out: 2021-03-15T21:45:14.975 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:14 +0000 (0:00:00.030) 0:00:06.688 ********** 2021-03-15T21:45:15.035 INFO:teuthology.task.ansible.out: 2021-03-15T21:45:15.036 INFO:teuthology.task.ansible.out:TASK [Filter the managed_users list] ******************************************* skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:15.060 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:15 +0000 (0:00:00.084) 0:00:06.773 ********** 2021-03-15T21:45:15.066 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:15.090 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:15 +0000 (0:00:00.029) 0:00:06.803 ********** 2021-03-15T21:45:15.117 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-15T21:45:15.119 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:15.141 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:15 +0000 (0:00:00.050) 0:00:06.853 ********** 2021-03-15T21:45:15.160 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:15.185 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:15 +0000 (0:00:00.044) 0:00:06.898 ********** 2021-03-15T21:45:15.380 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-03-15T21:45:15.382 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-15T21:45:15.402 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:15 +0000 (0:00:00.217) 0:00:07.115 ********** 2021-03-15T21:45:15.430 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-15T21:45:15.452 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:15 +0000 (0:00:00.049) 0:00:07.165 ********** 2021-03-15T21:45:15.680 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-15T21:45:15.682 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-15T21:45:15.744 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-15T21:45:15.892 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-15T21:45:15.945 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-15T21:45:16.091 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-15T21:45:16.155 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-15T21:45:16.291 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-15T21:45:16.365 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-15T21:45:16.490 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-15T21:45:16.565 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-15T21:45:16.701 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-15T21:45:16.766 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-15T21:45:16.910 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-15T21:45:16.974 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-15T21:45:17.112 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-15T21:45:17.176 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-15T21:45:17.312 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-15T21:45:17.386 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-15T21:45:17.522 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-15T21:45:17.585 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-15T21:45:17.721 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-15T21:45:17.798 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-15T21:45:17.914 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-15T21:45:18.001 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-15T21:45:18.115 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-15T21:45:18.199 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-15T21:45:18.314 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-15T21:45:18.409 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-15T21:45:18.515 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-15T21:45:18.608 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-15T21:45:18.729 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-15T21:45:18.814 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-15T21:45:18.921 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-15T21:45:19.016 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-15T21:45:19.121 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-15T21:45:19.215 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-15T21:45:19.330 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-15T21:45:19.414 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-15T21:45:19.529 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-15T21:45:19.624 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-15T21:45:19.728 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-15T21:45:19.823 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-15T21:45:19.928 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-15T21:45:20.022 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-15T21:45:20.136 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-15T21:45:20.231 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-15T21:45:20.335 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-15T21:45:20.430 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-15T21:45:20.534 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-15T21:45:20.628 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-15T21:45:20.732 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-15T21:45:20.847 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-15T21:45:20.941 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-15T21:45:21.046 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-15T21:45:21.140 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-15T21:45:21.244 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-15T21:45:21.339 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-15T21:45:21.443 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-15T21:45:21.540 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-15T21:45:21.657 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-15T21:45:21.751 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-15T21:45:21.855 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-15T21:45:21.949 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-15T21:45:22.054 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-15T21:45:22.149 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-15T21:45:22.263 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-15T21:45:22.347 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-15T21:45:22.462 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-15T21:45:22.557 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-15T21:45:22.672 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-15T21:45:22.755 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-15T21:45:22.870 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-15T21:45:22.964 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-15T21:45:23.070 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-15T21:45:23.165 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-15T21:45:23.279 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-15T21:45:23.363 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-15T21:45:23.477 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-15T21:45:23.561 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-15T21:45:23.686 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-15T21:45:23.770 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-15T21:45:23.884 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-15T21:45:23.968 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-15T21:45:24.083 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-15T21:45:24.166 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-15T21:45:24.292 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-15T21:45:24.369 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-15T21:45:24.485 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-15T21:45:24.569 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-15T21:45:24.694 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-15T21:45:24.767 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-15T21:45:24.893 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-15T21:45:24.977 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-15T21:45:25.569 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": fa 2021-03-15T21:45:25.571 INFO:teuthology.task.ansible.out:lse, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-15T21:45:25.573 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-15T21:45:25.657 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-15T21:45:25.783 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-15T21:45:25.867 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-15T21:45:25.983 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-15T21:45:26.067 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-15T21:45:26.183 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-15T21:45:26.267 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-15T21:45:26.391 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-15T21:45:26.465 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-15T21:45:26.591 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-15T21:45:26.674 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-15T21:45:26.792 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-15T21:45:26.876 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-15T21:45:27.001 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-15T21:45:27.075 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-15T21:45:27.200 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-15T21:45:27.273 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-15T21:45:27.399 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-15T21:45:27.482 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-15T21:45:27.608 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-15T21:45:27.681 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-15T21:45:27.807 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-15T21:45:27.880 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-15T21:45:28.005 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-15T21:45:28.079 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-15T21:45:28.204 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-15T21:45:28.277 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-15T21:45:28.413 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-15T21:45:28.476 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-15T21:45:28.613 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-15T21:45:28.676 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-15T21:45:28.823 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-15T21:45:28.886 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-15T21:45:29.023 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-15T21:45:29.086 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-15T21:45:29.232 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-15T21:45:29.234 INFO:teuthology.task.ansible.out: 2021-03-15T21:45:29.285 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-15T21:45:29.431 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-15T21:45:29.494 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-15T21:45:29.630 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-15T21:45:29.693 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-15T21:45:29.829 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-15T21:45:29.903 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-15T21:45:30.029 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-15T21:45:30.115 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-15T21:45:30.232 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-15T21:45:30.337 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-15T21:45:30.452 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-15T21:45:30.547 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-15T21:45:30.674 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-15T21:45:30.749 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-15T21:45:30.917 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-15T21:45:30.969 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-15T21:45:31.149 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-15T21:45:31.182 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-15T21:45:31.350 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-15T21:45:31.382 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-15T21:45:31.562 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-15T21:45:31.604 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-15T21:45:31.774 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-15T21:45:31.806 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-15T21:45:31.994 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-15T21:45:32.017 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-15T21:45:32.195 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-15T21:45:32.216 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-15T21:45:32.393 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-15T21:45:32.415 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-15T21:45:32.603 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-15T21:45:32.615 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-15T21:45:32.802 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-15T21:45:32.823 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-15T21:45:33.010 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-15T21:45:33.022 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-15T21:45:33.208 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-15T21:45:33.220 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-15T21:45:33.418 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-15T21:45:33.419 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-15T21:45:33.616 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-15T21:45:33.617 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-15T21:45:33.825 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-15T21:45:33.826 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-15T21:45:34.024 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-15T21:45:34.025 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-15T21:45:34.222 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-15T21:45:34.233 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-15T21:45:34.442 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-15T21:45:34.443 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-15T21:45:34.653 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-15T21:45:34.654 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-15T21:45:34.858 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-15T21:45:34.858 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-15T21:45:35.057 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-15T21:45:35.069 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-15T21:45:35.267 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-15T21:45:35.268 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-15T21:45:35.465 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-15T21:45:35.467 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-15T21:45:35.666 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-15T21:45:35.667 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-15T21:45:35.875 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-15T21:45:35.877 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-15T21:45:36.072 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-15T21:45:36.074 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-15T21:45:36.270 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-15T21:45:36.282 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-15T21:45:36.480 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-15T21:45:36.481 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-15T21:45:36.690 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-15T21:45:36.691 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-15T21:45:36.883 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-15T21:45:36.895 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-15T21:45:37.083 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-15T21:45:37.094 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-15T21:45:37.291 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-15T21:45:37.292 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-15T21:45:37.489 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-15T21:45:37.490 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-15T21:45:37.687 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-15T21:45:37.688 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-15T21:45:37.888 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-15T21:45:37.889 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-15T21:45:38.087 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-15T21:45:38.088 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-15T21:45:38.295 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-15T21:45:38.297 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-15T21:45:38.504 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-15T21:45:38.505 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-15T21:45:38.702 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-15T21:45:38.703 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-15T21:45:38.911 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-15T21:45:38.913 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-15T21:45:39.109 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-15T21:45:39.109 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-15T21:45:39.307 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-15T21:45:39.308 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-15T21:45:39.506 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-15T21:45:39.517 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-15T21:45:39.714 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-15T21:45:39.715 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-15T21:45:39.913 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-15T21:45:39.914 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-15T21:45:40.111 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-15T21:45:40.123 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-15T21:45:40.320 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-15T21:45:40.321 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-15T21:45:40.519 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-15T21:45:40.520 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-15T21:45:40.716 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-15T21:45:40.728 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-15T21:45:40.914 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-15T21:45:40.925 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-15T21:45:41.123 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-15T21:45:41.124 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-15T21:45:41.331 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-15T21:45:41.332 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-15T21:45:41.532 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-15T21:45:41.533 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-15T21:45:41.730 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-15T21:45:41.731 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-15T21:45:41.928 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-15T21:45:41.940 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-15T21:45:42.139 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-15T21:45:42.140 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-15T21:45:42.338 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-15T21:45:42.340 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-15T21:45:42.538 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-15T21:45:42.540 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-15T21:45:42.739 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-15T21:45:42.740 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-15T21:45:42.940 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-15T21:45:42.942 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-15T21:45:43.153 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-15T21:45:44.635 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-03-15T21:45:44.637 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-03-15T21:45:45.873 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-03-15T21:45:45.967 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-03-15T21:45:47.110 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-03-15T21:45:47.206 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-03-15T21:45:48.370 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-03-15T21:45:48.483 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:48 +0000 (0:00:33.030) 0:00:40.196 ********** 2021-03-15T21:45:48.554 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-15T21:45:48.575 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:48 +0000 (0:00:00.091) 0:00:40.288 ********** 2021-03-15T21:45:48.602 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** 2021-03-15T21:45:48.604 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-03-15T21:45:48.831 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-15T21:45:48.833 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-15T21:45:48.835 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-15T21:45:48.836 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-15T21:45:48.838 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-15T21:45:48.839 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRah 2021-03-15T21:45:48.840 INFO:teuthology.task.ansible.out:w fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-15T21:45:48.841 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-15T21:45:48.843 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-15T21:45:48.844 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-15T21:45:48.846 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-15T21:45:48.848 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-15T21:45:48.850 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-15T21:45:48.851 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-noteboo 2021-03-15T21:45:48.852 INFO:teuthology.task.ansible.out:k LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-15T21:45:48.853 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-15T21:45:48.875 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:48 +0000 (0:00:00.299) 0:00:40.588 ********** 2021-03-15T21:45:48.898 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:48 +0000 (0:00:00.022) 0:00:40.610 ********** 2021-03-15T21:45:48.941 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-15T21:45:48.968 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:48 +0000 (0:00:00.069) 0:00:40.680 ********** 2021-03-15T21:45:49.069 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-15T21:45:49.071 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-15T21:45:49.073 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-15T21:45:49.075 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-15T21:45:49.076 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-15T21:45:49.078 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-15T21:45:49.080 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-15T21:45:49.082 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a 2021-03-15T21:45:49.082 INFO:teuthology.task.ansible.out:4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-15T21:45:49.084 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-15T21:45:49.086 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-15T21:45:49.087 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-15T21:45:49.089 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-15T21:45:49.090 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-15T21:45:49.092 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174f 2021-03-15T21:45:49.092 INFO:teuthology.task.ansible.out:e16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-15T21:45:49.121 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:49 +0000 (0:00:00.153) 0:00:40.834 ********** 2021-03-15T21:45:49.652 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-15T21:45:49.654 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-15T21:45:49.677 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:49 +0000 (0:00:00.555) 0:00:41.389 ********** 2021-03-15T21:45:49.777 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-15T21:45:49.801 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:45:49 +0000 (0:00:00.124) 0:00:41.513 ********** 2021-03-15T21:45:50.006 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-15T21:45:50.007 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-15T21:45:50.122 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-15T21:45:50.218 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-15T21:45:50.344 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-15T21:45:50.440 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-15T21:45:50.568 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-15T21:45:50.662 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-15T21:45:50.664 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-15T21:45:50.788 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-15T21:45:50.789 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-15T21:45:50.884 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-15T21:45:51.001 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-15T21:45:51.583 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-15T21:45:51.585 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7 2021-03-15T21:45:51.587 INFO:teuthology.task.ansible.out:LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIae 2021-03-15T21:45:51.588 INFO:teuthology.task.ansible.out:zzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-15T21:45:51.590 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-15T21:45:51.723 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-15T21:45:51.797 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-15T21:45:51.808 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:51.933 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-15T21:45:51.944 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:52.018 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-15T21:45:52.020 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-15T21:45:52.164 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-15T21:45:52.166 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-15T21:45:52.237 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-15T21:45:52.239 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-15T21:45:52.248 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:52.250 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:52.373 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-15T21:45:52.375 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-15T21:45:52.384 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:52.395 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:45:52.468 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-15T21:45:52.614 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-15T21:45:52.687 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-15T21:45:52.689 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-15T21:45:52.823 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-15T21:45:52.825 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-15T21:45:52.897 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-15T21:45:53.043 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-15T21:45:53.117 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-15T21:45:53.118 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-15T21:45:53.252 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-15T21:45:53.254 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-15T21:45:53.325 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-15T21:45:53.461 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-15T21:45:53.545 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-15T21:45:53.690 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-15T21:45:53.763 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-15T21:45:53.909 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-15T21:45:53.982 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-15T21:45:53.984 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-15T21:45:54.117 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-15T21:45:54.119 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-15T21:45:54.190 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-15T21:45:54.192 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-15T21:45:54.336 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-15T21:45:54.338 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-15T21:45:54.399 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-15T21:45:54.547 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-15T21:45:54.611 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-15T21:45:54.767 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-15T21:45:54.830 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-15T21:45:54.832 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-15T21:45:54.976 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-15T21:45:54.977 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-15T21:45:55.051 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-15T21:45:55.196 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-15T21:45:55.271 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-15T21:45:55.407 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-15T21:45:55.481 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-15T21:45:55.616 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-15T21:45:55.690 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-15T21:45:55.835 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-15T21:45:55.908 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-15T21:45:56.043 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-15T21:45:56.117 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-15T21:45:56.118 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-15T21:45:56.263 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-15T21:45:56.265 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-15T21:45:56.337 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-15T21:45:56.567 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-15T21:45:56.620 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-15T21:45:56.818 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-15T21:45:56.881 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-15T21:45:57.067 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-15T21:45:57.130 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-15T21:45:57.296 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-15T21:45:57.348 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-15T21:45:57.514 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-15T21:45:57.567 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-15T21:45:57.722 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-15T21:45:57.774 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-15T21:45:57.940 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-15T21:45:57.992 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-15T21:45:58.148 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-15T21:45:58.201 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-15T21:45:58.366 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-15T21:45:58.419 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-15T21:45:58.421 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-15T21:45:58.575 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-15T21:45:58.576 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-15T21:45:58.627 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-15T21:45:58.629 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-15T21:45:58.794 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-15T21:45:58.796 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-15T21:45:58.837 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-15T21:45:59.003 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-15T21:45:59.049 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-15T21:45:59.216 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-15T21:45:59.279 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-15T21:45:59.434 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-15T21:45:59.487 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-15T21:45:59.488 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-15T21:45:59.642 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-15T21:45:59.644 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-15T21:45:59.705 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-15T21:45:59.707 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-15T21:45:59.708 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-15T21:45:59.861 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-15T21:45:59.863 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-15T21:45:59.865 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-15T21:45:59.914 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-15T21:46:00.069 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-15T21:46:00.132 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-15T21:46:00.288 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-15T21:46:00.351 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-15T21:46:00.507 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-15T21:46:00.560 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-15T21:46:00.726 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-15T21:46:00.768 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-15T21:46:00.934 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-15T21:46:00.987 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-15T21:46:00.989 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-15T21:46:01.147 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-15T21:46:01.148 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-15T21:46:01.199 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-15T21:46:01.366 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-15T21:46:01.419 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-15T21:46:01.576 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-15T21:46:01.628 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-15T21:46:01.630 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-15T21:46:01.794 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-15T21:46:01.796 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-15T21:46:01.857 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-15T21:46:02.002 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-15T21:46:02.076 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-15T21:46:02.231 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-15T21:46:02.295 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-15T21:46:02.297 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-15T21:46:02.441 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-15T21:46:02.443 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-15T21:46:02.504 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-15T21:46:02.649 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-15T21:46:02.713 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-15T21:46:02.869 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-15T21:46:02.922 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-15T21:46:03.081 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-15T21:46:03.137 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-15T21:46:03.294 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-15T21:46:03.346 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-15T21:46:03.505 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-15T21:46:03.569 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-15T21:46:03.727 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-15T21:46:03.780 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-15T21:46:03.950 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-15T21:46:03.993 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-15T21:46:04.161 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-15T21:46:04.203 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-15T21:46:04.372 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-15T21:46:04.414 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-15T21:46:04.582 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-15T21:46:04.624 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-15T21:46:04.802 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-15T21:46:04.844 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-15T21:46:05.010 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-15T21:46:05.012 INFO:teuthology.task.ansible.out: 2021-03-15T21:46:05.053 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-15T21:46:05.220 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-15T21:46:05.273 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-15T21:46:05.438 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-15T21:46:05.482 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-15T21:46:05.648 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-15T21:46:05.650 INFO:teuthology.task.ansible.out: 2021-03-15T21:46:05.701 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-15T21:46:05.868 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-15T21:46:05.921 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-15T21:46:06.087 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-15T21:46:06.129 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-15T21:46:06.295 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-15T21:46:06.348 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-15T21:46:06.515 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-15T21:46:06.567 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-15T21:46:06.754 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-15T21:46:06.776 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-15T21:46:06.778 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-15T21:46:06.963 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-15T21:46:06.965 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-15T21:46:06.985 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-15T21:46:07.184 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-15T21:46:07.206 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-15T21:46:07.414 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-15T21:46:07.425 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-15T21:46:07.622 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-15T21:46:07.644 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-15T21:46:07.852 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-15T21:46:07.853 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-15T21:46:07.855 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-15T21:46:08.071 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-15T21:46:08.073 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH 2021-03-15T21:46:08.075 INFO:teuthology.task.ansible.out:1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-15T21:46:08.279 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-15T21:46:08.280 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2 2021-03-15T21:46:08.282 INFO:teuthology.task.ansible.out:CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-15T21:46:08.507 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-15T21:46:08.508 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-15T21:46:08.715 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-15T21:46:08.718 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-15T21:46:08.933 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-15T21:46:08.954 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-15T21:46:09.153 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-15T21:46:09.165 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-15T21:46:09.373 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-15T21:46:09.384 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-15T21:46:09.581 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-15T21:46:09.614 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-15T21:46:09.801 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-15T21:46:09.823 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-15T21:46:09.825 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-15T21:46:10.020 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-15T21:46:10.022 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-15T21:46:10.032 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-15T21:46:10.229 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-15T21:46:10.250 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-15T21:46:10.448 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-15T21:46:10.459 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-15T21:46:10.666 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-15T21:46:10.668 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-15T21:46:10.885 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-15T21:46:10.896 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-15T21:46:11.107 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-15T21:46:11.118 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-15T21:46:11.322 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-15T21:46:11.334 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-15T21:46:11.336 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-15T21:46:11.542 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-15T21:46:11.544 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-15T21:46:11.546 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-15T21:46:11.751 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2 2021-03-15T21:46:11.753 INFO:teuthology.task.ansible.out:JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-15T21:46:11.959 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-15T21:46:11.970 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-15T21:46:12.178 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-15T21:46:12.180 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-15T21:46:12.397 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-15T21:46:12.399 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-15T21:46:12.616 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-15T21:46:12.617 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-15T21:46:12.825 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-15T21:46:12.827 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-15T21:46:13.045 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-15T21:46:13.047 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-15T21:46:13.253 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-15T21:46:13.254 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-15T21:46:13.470 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-15T21:46:13.471 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-15T21:46:13.678 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-15T21:46:13.679 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-15T21:46:13.897 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-15T21:46:13.899 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-15T21:46:14.107 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-15T21:46:14.109 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-15T21:46:14.316 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-15T21:46:14.327 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-15T21:46:14.534 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-15T21:46:14.535 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-15T21:46:14.743 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-15T21:46:14.765 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-15T21:46:14.953 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-15T21:46:14.975 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-15T21:46:15.176 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-15T21:46:15.188 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-15T21:46:15.384 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-15T21:46:15.386 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-15T21:46:15.395 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-15T21:46:15.602 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-15T21:46:15.614 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-15T21:46:15.616 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-15T21:46:15.811 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-15T21:46:15.833 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-15T21:46:16.020 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-15T21:46:16.041 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-15T21:46:16.239 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-15T21:46:16.260 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-15T21:46:16.447 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-15T21:46:16.479 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-15T21:46:16.661 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-15T21:46:16.817 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-15T21:46:16.913 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-15T21:46:16.955 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-15T21:46:17.131 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-15T21:46:17.174 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-15T21:46:17.351 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-15T21:46:17.383 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-15T21:46:17.559 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-15T21:46:17.591 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-15T21:46:17.779 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-15T21:46:17.801 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-15T21:46:17.988 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-15T21:46:18.020 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-15T21:46:18.198 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-15T21:46:18.231 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-15T21:46:18.418 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-15T21:46:18.450 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-15T21:46:18.627 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-15T21:46:18.670 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-15T21:46:18.847 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-15T21:46:18.932 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-15T21:46:19.058 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-03-15T21:46:19.060 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-15T21:46:19.142 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-15T21:46:19.267 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-15T21:46:19.360 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:19 +0000 (0:00:29.558) 0:01:11.072 ********** 2021-03-15T21:46:19.368 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-03-15T21:46:19.371 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-15T21:46:19.445 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-15T21:46:19.447 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.456 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.468 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.470 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.480 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.480 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.491 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.493 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.586 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-15T21:46:19.588 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-15T21:46:19.675 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:19 +0000 (0:00:00.315) 0:01:11.388 ********** 2021-03-15T21:46:19.714 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-15T21:46:19.716 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e 2021-03-15T21:46:19.717 INFO:teuthology.task.ansible.out:1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.726 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.758 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.769 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.780 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.782 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.791 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.793 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.803 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.814 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.815 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:19.929 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-15T21:46:20.034 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-15T21:46:20.045 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.047 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.141 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-15T21:46:20.152 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.163 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.164 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.174 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.176 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.185 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.196 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.198 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.208 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.208 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.219 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.230 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.232 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.241 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.252 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.254 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.264 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-15T21:46:20.265 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.275 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.277 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.287 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.289 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.299 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.310 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.311 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.321 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.332 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.333 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.343 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.345 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.354 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.365 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.367 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.377 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.379 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.390 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.402 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.404 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.414 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.416 INFO:teuthology.task.ansible.out: 2021-03-15T21:46:20.426 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.437 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.439 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.450 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.452 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.462 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.464 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.474 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-15T21:46:20.476 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.487 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.489 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.491 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.501 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.502 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.514 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.525 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.537 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.538 INFO:teuthology.task.ansible.out: skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.539 INFO:teuthology.task.ansible.out: skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.550 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.551 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.552 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.552 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.563 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.565 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.575 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.577 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.578 INFO:teuthology.task.ansible.out: skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.589 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.592 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.593 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.594 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.606 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.608 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.611 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.612 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.625 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.628 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.629 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.631 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.641 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.643 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.655 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.657 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.667 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.669 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.678 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.680 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.691 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.692 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.694 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.704 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.705 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.717 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.718 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.729 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.731 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.731 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.742 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.744 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.744 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.745 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.756 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.757 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.767 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.769 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.779 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.780 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.780 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.781 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.791 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.793 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.804 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.805 INFO:teuthology.task.ansible.out: skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.816 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.818 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.829 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.830 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.841 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.842 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.853 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.854 INFO:teuthology.task.ansible.out: skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.856 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.867 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.871 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.879 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.882 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.891 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.893 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.903 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.905 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.915 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.917 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.928 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.931 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.941 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.943 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.953 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.955 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.964 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.966 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.979 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.980 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.982 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.982 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.993 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:20.995 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.005 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.007 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.018 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.019 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.021 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.031 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.033 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.044 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.045 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.056 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.059 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.069 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.073 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.081 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.083 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.094 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.095 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.112 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.114 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.135 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:21 +0000 (0:00:01.460) 0:01:12.848 ********** 2021-03-15T21:46:21.142 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.144 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.144 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.157 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.158 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.169 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.171 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.181 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.183 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.192 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.204 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.205 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.244 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:21 +0000 (0:00:00.108) 0:01:12.957 ********** 2021-03-15T21:46:21.255 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.256 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.258 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.260 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.260 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.262 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.263 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.275 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.276 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.288 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.290 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.292 INFO:teuthology.task.ansible.out: skipping: [smithi076.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.351 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:21 +0000 (0:00:00.106) 0:01:13.064 ********** 2021-03-15T21:46:21.422 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-15T21:46:21.424 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:21.444 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:21 +0000 (0:00:00.092) 0:01:13.156 ********** 2021-03-15T21:46:21.548 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-15T21:46:21.550 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-15T21:46:21.748 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-15T21:46:21.956 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-15T21:46:21.958 INFO:teuthology.task.ansible.out: 2021-03-15T21:46:21.968 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-15T21:46:21.970 INFO:teuthology.task.ansible.out: 2021-03-15T21:46:22.167 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-15T21:46:22.179 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-15T21:46:22.377 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-15T21:46:22.378 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-15T21:46:22.579 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-15T21:46:22.581 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-15T21:46:22.777 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-15T21:46:22.777 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-15T21:46:22.977 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-15T21:46:22.979 INFO:teuthology.task.ansible.out: ok: [smithi125.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-15T21:46:23.167 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-15T21:46:23.178 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-15T21:46:23.369 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-15T21:46:23.380 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-15T21:46:23.567 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-15T21:46:23.578 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-15T21:46:23.765 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-15T21:46:23.776 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-15T21:46:23.953 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-15T21:46:23.975 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-15T21:46:24.151 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-15T21:46:24.173 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-15T21:46:24.349 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-15T21:46:24.371 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-15T21:46:24.547 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-15T21:46:24.558 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-15T21:46:24.755 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-15T21:46:24.756 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-15T21:46:24.953 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-15T21:46:24.965 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-15T21:46:25.151 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-15T21:46:25.152 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-15T21:46:25.349 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-15T21:46:25.360 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-15T21:46:25.547 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-15T21:46:25.558 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-15T21:46:25.747 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-15T21:46:25.749 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-15T21:46:25.944 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-15T21:46:25.946 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-15T21:46:26.142 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-15T21:46:26.144 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-15T21:46:26.340 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-15T21:46:26.341 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-15T21:46:26.533 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-15T21:46:26.545 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-15T21:46:26.732 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-15T21:46:26.743 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-15T21:46:26.930 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-15T21:46:26.941 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-15T21:46:27.138 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-15T21:46:27.139 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-15T21:46:27.329 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-15T21:46:27.340 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-15T21:46:27.527 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-15T21:46:27.528 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-15T21:46:27.727 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-15T21:46:27.727 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-15T21:46:27.925 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-15T21:46:27.926 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-15T21:46:28.123 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-15T21:46:28.134 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-15T21:46:28.321 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-15T21:46:28.332 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-15T21:46:28.519 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-15T21:46:28.530 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-15T21:46:28.717 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-15T21:46:28.729 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-15T21:46:28.906 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-15T21:46:28.929 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-15T21:46:30.149 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-03-15T21:46:30.151 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-03-15T21:46:31.379 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-03-15T21:46:31.390 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-03-15T21:46:31.592 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-15T21:46:31.603 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-15T21:46:31.789 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-15T21:46:31.801 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-15T21:46:31.987 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-15T21:46:31.998 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-15T21:46:32.188 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-15T21:46:32.199 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-15T21:46:32.386 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-15T21:46:32.397 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-15T21:46:32.584 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-15T21:46:32.595 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-15T21:46:32.782 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-15T21:46:32.793 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-15T21:46:32.980 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-15T21:46:32.982 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-15T21:46:33.178 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-15T21:46:33.180 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-15T21:46:33.376 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-15T21:46:33.376 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-15T21:46:33.451 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:33 +0000 (0:00:12.006) 0:01:25.163 ********** 2021-03-15T21:46:33.471 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:33 +0000 (0:00:00.020) 0:01:25.184 ********** 2021-03-15T21:46:34.006 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-15T21:46:34.008 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615844793.5458317-19728-193842961682219/source", "state": "file", "uid": 0} 2021-03-15T21:46:34.028 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:34 +0000 (0:00:00.557) 0:01:25.741 ********** 2021-03-15T21:46:34.036 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615844793.5659716-19729-195122804144429/source", "state": "file", "uid": 0} 2021-03-15T21:46:34.059 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:34 +0000 (0:00:00.030) 0:01:25.772 ********** 2021-03-15T21:46:34.096 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-15T21:46:34.098 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => { "msg": "Host smithi076.front.sepia.ceph.com is running CentOS 8.2 (Core)" } 2021-03-15T21:46:34.120 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:34 +0000 (0:00:00.060) 0:01:25.832 ********** 2021-03-15T21:46:34.137 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => { "msg": "Host smithi125.front.sepia.ceph.com is running CentOS 8.2 (Core)" } 2021-03-15T21:46:34.159 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:34 +0000 (0:00:00.039) 0:01:25.872 ********** 2021-03-15T21:46:34.198 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-15T21:46:34.200 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-03-15T21:46:34.221 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:34 +0000 (0:00:00.061) 0:01:25.934 ********** 2021-03-15T21:46:34.239 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-03-15T21:46:34.262 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:34 +0000 (0:00:00.040) 0:01:25.974 ********** 2021-03-15T21:46:34.288 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-03-15T21:46:34.290 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:34.313 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:34 +0000 (0:00:00.050) 0:01:26.025 ********** 2021-03-15T21:46:34.341 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:34.364 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:34 +0000 (0:00:00.050) 0:01:26.076 ********** 2021-03-15T21:46:34.920 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-15T21:46:34.922 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.322031", "end": "2021-03-15 21:46:34.898473", "rc": 0, "start": "2021-03-15 21:46:34.576442", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-15T21:46:34.943 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:34 +0000 (0:00:00.579) 0:01:26.655 ********** 2021-03-15T21:46:34.950 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.379830", "end": "2021-03-15 21:46:34.920645", "rc": 0, "start": "2021-03-15 21:46:34.540815", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-15T21:46:34.974 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:34 +0000 (0:00:00.031) 0:01:26.687 ********** 2021-03-15T21:46:35.012 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-15T21:46:35.014 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.037 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.062) 0:01:26.749 ********** 2021-03-15T21:46:35.054 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.078 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.040) 0:01:26.790 ********** 2021-03-15T21:46:35.105 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** 2021-03-15T21:46:35.106 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.128 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.050) 0:01:26.840 ********** 2021-03-15T21:46:35.145 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.168 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.039) 0:01:26.880 ********** 2021-03-15T21:46:35.205 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-15T21:46:35.207 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.229 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.061) 0:01:26.942 ********** 2021-03-15T21:46:35.246 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.269 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.040) 0:01:26.982 ********** 2021-03-15T21:46:35.307 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-15T21:46:35.309 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.330 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.060) 0:01:27.042 ********** 2021-03-15T21:46:35.350 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.373 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.043) 0:01:27.086 ********** 2021-03-15T21:46:35.411 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-15T21:46:35.413 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.434 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.061) 0:01:27.147 ********** 2021-03-15T21:46:35.451 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.474 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.039) 0:01:27.187 ********** 2021-03-15T21:46:35.503 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-15T21:46:35.505 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.526 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.051) 0:01:27.238 ********** 2021-03-15T21:46:35.553 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.576 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.050) 0:01:27.288 ********** 2021-03-15T21:46:35.603 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-15T21:46:35.605 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-15T21:46:35.627 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.050) 0:01:27.339 ********** 2021-03-15T21:46:35.654 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-15T21:46:35.656 INFO:teuthology.task.ansible.out: skipping: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-15T21:46:35.678 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.051) 0:01:27.391 ********** 2021-03-15T21:46:35.696 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-03-15T21:46:35.698 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.720 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.041) 0:01:27.433 ********** 2021-03-15T21:46:35.758 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.784 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.063) 0:01:27.496 ********** 2021-03-15T21:46:35.801 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-03-15T21:46:35.803 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.824 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.040) 0:01:27.536 ********** 2021-03-15T21:46:35.852 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:35.880 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.055) 0:01:27.592 ********** 2021-03-15T21:46:35.898 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-15T21:46:35.900 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-15T21:46:35.926 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.045) 0:01:27.638 ********** 2021-03-15T21:46:35.955 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-15T21:46:35.979 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:35 +0000 (0:00:00.053) 0:01:27.692 ********** 2021-03-15T21:46:35.998 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-15T21:46:36.000 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.022 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.042) 0:01:27.734 ********** 2021-03-15T21:46:36.049 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.073 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.051) 0:01:27.786 ********** 2021-03-15T21:46:36.090 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-15T21:46:36.092 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-15T21:46:36.115 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.041) 0:01:27.827 ********** 2021-03-15T21:46:36.143 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-15T21:46:36.166 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.051) 0:01:27.879 ********** 2021-03-15T21:46:36.184 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-15T21:46:36.186 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.207 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.041) 0:01:27.920 ********** 2021-03-15T21:46:36.235 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.259 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.051) 0:01:27.972 ********** 2021-03-15T21:46:36.277 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-03-15T21:46:36.278 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.300 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.040) 0:01:28.012 ********** 2021-03-15T21:46:36.327 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.350 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.050) 0:01:28.062 ********** 2021-03-15T21:46:36.377 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-15T21:46:36.379 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.400 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.050) 0:01:28.113 ********** 2021-03-15T21:46:36.418 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.442 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.041) 0:01:28.154 ********** 2021-03-15T21:46:36.470 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-15T21:46:36.472 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.492 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.050) 0:01:28.205 ********** 2021-03-15T21:46:36.520 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.546 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.053) 0:01:28.258 ********** 2021-03-15T21:46:36.563 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-15T21:46:36.565 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-15T21:46:36.586 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.040) 0:01:28.299 ********** 2021-03-15T21:46:36.624 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-15T21:46:36.648 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.061) 0:01:28.360 ********** 2021-03-15T21:46:36.666 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-15T21:46:36.668 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.689 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.041) 0:01:28.402 ********** 2021-03-15T21:46:36.727 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.750 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.061) 0:01:28.463 ********** 2021-03-15T21:46:36.769 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-15T21:46:36.770 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.792 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.041) 0:01:28.504 ********** 2021-03-15T21:46:36.830 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.854 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.061) 0:01:28.566 ********** 2021-03-15T21:46:36.871 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-15T21:46:36.873 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.894 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.040) 0:01:28.607 ********** 2021-03-15T21:46:36.932 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.958 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.063) 0:01:28.670 ********** 2021-03-15T21:46:36.975 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-15T21:46:36.977 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:36.998 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:36 +0000 (0:00:00.040) 0:01:28.711 ********** 2021-03-15T21:46:37.026 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:37.050 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.051) 0:01:28.762 ********** 2021-03-15T21:46:37.067 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-03-15T21:46:37.068 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:37.089 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.039) 0:01:28.802 ********** 2021-03-15T21:46:37.117 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:37.140 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.050) 0:01:28.853 ********** 2021-03-15T21:46:37.158 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-15T21:46:37.160 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:37.183 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.042) 0:01:28.895 ********** 2021-03-15T21:46:37.210 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:37.234 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.051) 0:01:28.947 ********** 2021-03-15T21:46:37.252 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-15T21:46:37.254 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-15T21:46:37.286 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.051) 0:01:28.998 ********** 2021-03-15T21:46:37.306 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-15T21:46:37.343 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.057) 0:01:29.056 ********** 2021-03-15T21:46:37.367 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-15T21:46:37.369 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:37.390 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.047) 0:01:29.103 ********** 2021-03-15T21:46:37.428 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:37.451 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.060) 0:01:29.163 ********** 2021-03-15T21:46:37.468 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-15T21:46:37.470 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:37.492 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.040) 0:01:29.204 ********** 2021-03-15T21:46:37.519 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:37.543 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.051) 0:01:29.255 ********** 2021-03-15T21:46:37.768 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-15T21:46:37.770 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-15T21:46:37.791 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.247) 0:01:29.503 ********** 2021-03-15T21:46:37.818 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-15T21:46:37.842 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:37 +0000 (0:00:00.050) 0:01:29.554 ********** 2021-03-15T21:46:38.565 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-15T21:46:38.567 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-03-15T21:46:38.577 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-03-15T21:46:39.003 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-03-15T21:46:39.017 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-03-15T21:46:39.040 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:39 +0000 (0:00:01.197) 0:01:30.752 ********** 2021-03-15T21:46:39.059 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:39 +0000 (0:00:00.019) 0:01:30.771 ********** 2021-03-15T21:46:39.555 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-15T21:46:39.557 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-03-15T21:46:39.567 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-03-15T21:46:39.992 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-03-15T21:46:40.005 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-03-15T21:46:40.027 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.968) 0:01:31.740 ********** 2021-03-15T21:46:40.046 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.018) 0:01:31.759 ********** 2021-03-15T21:46:40.116 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-15T21:46:40.118 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.139 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.092) 0:01:31.851 ********** 2021-03-15T21:46:40.157 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.018) 0:01:31.870 ********** 2021-03-15T21:46:40.205 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-15T21:46:40.298 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 15 March 2021 21:46:40 +0000 (0:00:00.069) 0:01:31.939 ********** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 15 March 2021 21:46:40 +0000 (0:00:00.029) 0:01:31.969 ********** TASK [common : Set the timezone in /etc/timezone] ****************************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.316 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.059) 0:01:32.029 ********** 2021-03-15T21:46:40.333 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.356 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.039) 0:01:32.068 ********** 2021-03-15T21:46:40.383 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-15T21:46:40.385 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.406 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.049) 0:01:32.118 ********** 2021-03-15T21:46:40.423 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.447 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.039) 0:01:32.158 ********** 2021-03-15T21:46:40.483 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-03-15T21:46:40.485 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.487 INFO:teuthology.task.ansible.out: skipping: [smithi076.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.498 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.499 INFO:teuthology.task.ansible.out: 2021-03-15T21:46:40.524 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.078) 0:01:32.237 ********** 2021-03-15T21:46:40.525 INFO:teuthology.task.ansible.out: 2021-03-15T21:46:40.532 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.534 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.558 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.034) 0:01:32.271 ********** 2021-03-15T21:46:40.591 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-15T21:46:40.593 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.625 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.066) 0:01:32.338 ********** 2021-03-15T21:46:40.644 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.670 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.044) 0:01:32.382 ********** 2021-03-15T21:46:40.697 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-15T21:46:40.698 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.725 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.055) 0:01:32.438 ********** 2021-03-15T21:46:40.744 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.766 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.041) 0:01:32.479 ********** 2021-03-15T21:46:40.794 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-15T21:46:40.795 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.818 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.051) 0:01:32.530 ********** 2021-03-15T21:46:40.836 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.859 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.040) 0:01:32.571 ********** 2021-03-15T21:46:40.886 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-15T21:46:40.887 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.910 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.051) 0:01:32.623 ********** 2021-03-15T21:46:40.927 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:40.951 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:40 +0000 (0:00:00.041) 0:01:32.664 ********** 2021-03-15T21:46:40.979 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-15T21:46:40.980 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:41.004 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:41 +0000 (0:00:00.052) 0:01:32.717 ********** 2021-03-15T21:46:41.021 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:41.045 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:41 +0000 (0:00:00.041) 0:01:32.758 ********** 2021-03-15T21:46:41.520 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-15T21:46:41.521 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.321994", "end": "2021-03-15 21:46:41.500362", "rc": 0, "start": "2021-03-15 21:46:41.178368", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.276988", "end": "2021-03-15 21:46:41.500361", "rc": 0, "start": "2021-03-15 21:46:41.223373", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:46:41.547 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:41 +0000 (0:00:00.501) 0:01:33.260 ********** 2021-03-15T21:46:41.567 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:41 +0000 (0:00:00.019) 0:01:33.280 ********** 2021-03-15T21:46:45.232 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-15T21:46:45.234 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-15T21:46:45.257 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:45 +0000 (0:00:03.689) 0:01:36.970 ********** 2021-03-15T21:46:45.327 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-15T21:46:45.329 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:45.349 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:45 +0000 (0:00:00.092) 0:01:37.062 ********** 2021-03-15T21:46:45.428 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-15T21:46:45.430 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:45.450 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:45 +0000 (0:00:00.100) 0:01:37.163 ********** 2021-03-15T21:46:45.456 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-15T21:46:45.458 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-15T21:46:45.480 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:45 +0000 (0:00:00.030) 0:01:37.193 ********** 2021-03-15T21:46:45.518 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-15T21:46:45.519 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:45.540 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:45 +0000 (0:00:00.060) 0:01:37.253 ********** 2021-03-15T21:46:45.557 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-15T21:46:45.559 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:45.581 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:45 +0000 (0:00:00.040) 0:01:37.293 ********** 2021-03-15T21:46:45.608 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-15T21:46:45.610 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:45.631 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:45 +0000 (0:00:00.050) 0:01:37.343 ********** 2021-03-15T21:46:45.658 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-15T21:46:45.660 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:45.685 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:45 +0000 (0:00:00.053) 0:01:37.397 ********** 2021-03-15T21:46:45.766 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-15T21:46:45.768 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:45.789 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:45 +0000 (0:00:00.104) 0:01:37.501 ********** 2021-03-15T21:46:45.858 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-15T21:46:45.860 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:45.880 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:45 +0000 (0:00:00.090) 0:01:37.592 ********** 2021-03-15T21:46:46.126 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-15T21:46:46.128 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-03-15T21:46:46.152 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:46 +0000 (0:00:00.272) 0:01:37.865 ********** 2021-03-15T21:46:46.380 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-03-15T21:46:46.405 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:46 +0000 (0:00:00.252) 0:01:38.118 ********** 2021-03-15T21:46:49.451 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-15T21:46:49.453 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: smartmontools-1:7.1-1.el8.x86_64", "Removed: smartmontools-1:6.6-3.el8.x86_64"]} 2021-03-15T21:46:49.476 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:49 +0000 (0:00:03.070) 0:01:41.188 ********** 2021-03-15T21:46:49.870 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: smartmontools-1:7.1-1.el8.x86_64", "Removed: smartmontools-1:6.6-3.el8.x86_64"]} 2021-03-15T21:46:49.895 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:49 +0000 (0:00:00.418) 0:01:41.607 ********** 2021-03-15T21:46:50.058 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-15T21:46:50.060 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-03-15T21:46:50.560 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-03-15T21:46:50.593 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-03-15T21:46:51.011 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-03-15T21:46:51.038 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:51 +0000 (0:00:01.143) 0:01:42.750 ********** 2021-03-15T21:46:51.088 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-03-15T21:46:51.237 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-15T21:46:51.239 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-15T21:46:51.268 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:51 +0000 (0:00:00.230) 0:01:42.981 ********** 2021-03-15T21:46:51.571 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-15T21:46:51.572 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-03-15T21:46:51.597 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:51 +0000 (0:00:00.328) 0:01:43.309 ********** 2021-03-15T21:46:51.792 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-15T21:46:51.795 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-15T21:46:51.818 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:51 +0000 (0:00:00.221) 0:01:43.530 ********** 2021-03-15T21:46:51.868 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-15T21:46:51.870 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615844811.3771055-21023-173357471860434/source", "state": "file", "uid": 0} 2021-03-15T21:46:52.310 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-03-15T21:46:52.405 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615844811.9216828-21076-272056518403873/source", "state": "file", "uid": 0} 2021-03-15T21:46:52.740 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-03-15T21:46:52.765 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:52 +0000 (0:00:00.946) 0:01:44.477 ********** 2021-03-15T21:46:52.844 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-15T21:46:52.851 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** ok: [smithi076.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-03-15T21:46:52.874 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:52 +0000 (0:00:00.109) 0:01:44.587 ********** 2021-03-15T21:46:52.955 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-15T21:46:52.957 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-03-15T21:46:52.978 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:52 +0000 (0:00:00.103) 0:01:44.691 ********** 2021-03-15T21:46:53.223 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-15T21:46:53.225 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-03-15T21:46:53.247 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:53 +0000 (0:00:00.268) 0:01:44.959 ********** 2021-03-15T21:46:53.287 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-15T21:46:53.288 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-03-15T21:46:53.314 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:53 +0000 (0:00:00.067) 0:01:45.027 ********** 2021-03-15T21:46:53.332 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-15T21:46:53.334 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:53.344 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:53.355 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:53.367 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:53.399 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:53 +0000 (0:00:00.085) 0:01:45.112 ********** 2021-03-15T21:46:53.407 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-15T21:46:53.409 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:53.432 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:53 +0000 (0:00:00.032) 0:01:45.144 ********** 2021-03-15T21:46:53.516 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-15T21:46:53.518 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-03-15T21:46:53.539 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:53 +0000 (0:00:00.106) 0:01:45.251 ********** 2021-03-15T21:46:53.786 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-15T21:46:53.788 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-03-15T21:46:53.808 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:53 +0000 (0:00:00.269) 0:01:45.520 ********** 2021-03-15T21:46:53.887 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-15T21:46:53.889 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:53.898 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:53.910 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:53.912 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:53.932 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:53 +0000 (0:00:00.123) 0:01:45.644 ********** 2021-03-15T21:46:54.464 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-15T21:46:54.465 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-15T21:46:54.486 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:54 +0000 (0:00:00.554) 0:01:46.199 ********** 2021-03-15T21:46:54.576 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-15T21:46:54.578 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:54.598 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:54 +0000 (0:00:00.111) 0:01:46.311 ********** 2021-03-15T21:46:54.991 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-15T21:46:54.994 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-15T21:46:55.017 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:55 +0000 (0:00:00.419) 0:01:46.730 ********** 2021-03-15T21:46:55.097 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-15T21:46:55.099 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:55.119 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:55 +0000 (0:00:00.102) 0:01:46.832 ********** 2021-03-15T21:46:55.147 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-15T21:46:55.149 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-03-15T21:46:55.173 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:55 +0000 (0:00:00.053) 0:01:46.886 ********** 2021-03-15T21:46:55.634 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-03-15T21:46:55.657 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:55 +0000 (0:00:00.483) 0:01:47.369 ********** 2021-03-15T21:46:55.694 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-15T21:46:55.696 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-03-15T21:46:55.718 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:55 +0000 (0:00:00.061) 0:01:47.431 ********** 2021-03-15T21:46:55.807 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-15T21:46:55.809 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:55.819 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:55.841 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:55 +0000 (0:00:00.123) 0:01:47.554 ********** 2021-03-15T21:46:56.107 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-15T21:46:56.109 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-15T21:46:56.129 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:56 +0000 (0:00:00.288) 0:01:47.842 ********** 2021-03-15T21:46:56.178 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-15T21:46:56.180 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-03-15T21:46:56.202 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:56 +0000 (0:00:00.072) 0:01:47.915 ********** 2021-03-15T21:46:56.282 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-15T21:46:56.284 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-03-15T21:46:56.306 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:56 +0000 (0:00:00.104) 0:01:48.019 ********** 2021-03-15T21:46:56.407 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-15T21:46:56.408 INFO:teuthology.task.ansible.out:fatal: [smithi125.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-03-15T21:46:56.430 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:56 +0000 (0:00:00.123) 0:01:48.143 ********** 2021-03-15T21:46:56.509 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-15T21:46:56.511 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:56.532 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:56 +0000 (0:00:00.101) 0:01:48.244 ********** 2021-03-15T21:46:56.559 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-15T21:46:56.561 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-15T21:46:56.586 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:56 +0000 (0:00:00.054) 0:01:48.299 ********** 2021-03-15T21:46:56.868 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-15T21:46:56.870 INFO:teuthology.task.ansible.out:fatal: [smithi076.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-03-15T21:46:56.891 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:56 +0000 (0:00:00.304) 0:01:48.603 ********** 2021-03-15T21:46:56.969 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-15T21:46:56.971 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:46:56.992 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:56 +0000 (0:00:00.100) 0:01:48.704 ********** 2021-03-15T21:46:57.051 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-15T21:46:57.093 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} Monday 15 March 2021 21:46:57 +0000 (0:00:00.085) 0:01:48.790 ********** 2021-03-15T21:46:57.570 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-03-15T21:46:57.573 INFO:teuthology.task.ansible.out: 2021-03-15T21:46:57.602 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:57 +0000 (0:00:00.524) 0:01:49.315 ********** 2021-03-15T21:46:57.980 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-15T21:46:57.981 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-03-15 21:44:31 UTC", "ActiveEnterTimestampMonotonic": "12729152", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target -.mount local-fs.target var-run.mount system.slice tmp.mount systemd-journald.socket systemd-tmpfiles-setup.service sysinit.target basic.target time-sync.target nss-lookup.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-03-15 21:44:31 UTC", "AssertTimestampMonotonic": "12727243", "Before": "xdm.service plymouth-quit.service getty@tty1.service multi-user.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-03-15 21:44:31 UTC", "ConditionTimestampMonotonic": "12727243", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "Co 2021-03-15T21:46:57.983 INFO:teuthology.task.ansible.out:ntrolGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "934", "ExecMainStartTimestamp": "Mon 2021-03-15 21:44:31 UTC", "ExecMainStartTimestampMonotonic": "12729113", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-03-15 21:44:31 UTC", "InactiveExitTimestampMonotonic": "12729152", "InvocationID": "7928ffe6262a42b1837e3e6df4c7df28", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", " 2021-03-15T21:46:57.984 INFO:teuthology.task.ansible.out:LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127270", "LimitNPROCSoft": "127270", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127270", "LimitSIGPENDINGSoft": "127270", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "934", "MemoryAccounting": "yes", "MemoryCurrent": "1708032", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/run/nrpe /var/tmp", 2021-03-15T21:46:57.986 INFO:teuthology.task.ansible.out:"Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-03-15 21:44:31 UTC", "StateChangeTimestampMonotonic": "12729152", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203632", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-03-15 21:44:31 UTC", "WatchdogTimestampMonotonic": "12729152", "WatchdogUSec": "0"}} ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-03-15 21:44:26 UTC", "ActiveEnterTimestampMonotonic": "13273773", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "var-run.mount systemd-journald.socket nss-lookup.target system.slice -.mount tmp.m 2021-03-15T21:46:57.987 INFO:teuthology.task.ansible.out:ount basic.target network.target time-sync.target local-fs.target sysinit.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-03-15 21:44:26 UTC", "AssertTimestampMonotonic": "13272977", "Before": "plymouth-quit.service multi-user.target shutdown.target xdm.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-03-15 21:44:26 UTC", "ConditionTimestampMonotonic": "13272977", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "ControlGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMain 2021-03-15T21:46:57.989 INFO:teuthology.task.ansible.out:PID": "946", "ExecMainStartTimestamp": "Mon 2021-03-15 21:44:26 UTC", "ExecMainStartTimestampMonotonic": "13273741", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-03-15 21:44:26 UTC", "InactiveExitTimestampMonotonic": "13273773", "InvocationID": "74fbe0d45c284652af2989c199183022", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", " 2021-03-15T21:46:57.991 INFO:teuthology.task.ansible.out:LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127269", "LimitNPROCSoft": "127269", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127269", "LimitSIGPENDINGSoft": "127269", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "946", "MemoryAccounting": "yes", "MemoryCurrent": "1544192", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount sysinit.target system.slice", "RequiresMountsFor": "/var/tmp /run/nrpe", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIG 2021-03-15T21:46:57.992 INFO:teuthology.task.ansible.out:KILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-03-15 21:44:26 UTC", "StateChangeTimestampMonotonic": "13273773", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203631", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-03-15 21:44:26 UTC", "WatchdogTimestampMonotonic": "13273772", "WatchdogUSec": "0"}} 2021-03-15T21:46:58.007 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:58 +0000 (0:00:00.404) 0:01:49.719 ********** 2021-03-15T21:46:58.026 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:58 +0000 (0:00:00.018) 0:01:49.738 ********** 2021-03-15T21:46:58.239 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-03-15T21:46:58.241 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001805", "end": "2021-03-15 21:46:58.215066", "rc": 0, "start": "2021-03-15 21:46:58.213261", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-03-15T21:46:58.262 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:58 +0000 (0:00:00.236) 0:01:49.975 ********** 2021-03-15T21:46:58.269 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001704", "end": "2021-03-15 21:46:58.239935", "rc": 0, "start": "2021-03-15 21:46:58.238231", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-03-15T21:46:58.292 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:58 +0000 (0:00:00.029) 0:01:50.004 ********** 2021-03-15T21:46:59.316 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-15T21:46:59.318 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-15T21:46:59.339 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:59 +0000 (0:00:01.047) 0:01:51.051 ********** 2021-03-15T21:46:59.369 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-15T21:46:59.403 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:46:59 +0000 (0:00:00.063) 0:01:51.115 ********** 2021-03-15T21:47:37.318 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-15T21:47:37.321 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-54.el8_3.2.noarch", "Installed: selinux-policy-3.14.3-54.el8_3.2.noarch", "Removed: selinux-policy-3.14.3-41.el8_2.8.noarch", "Removed: selinux-policy-targeted-3.14.3-41.el8_2.8.noarch"]} 2021-03-15T21:47:37.342 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:37 +0000 (0:00:37.939) 0:02:29.055 ********** 2021-03-15T21:47:37.645 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted-3.14.3-54.el8_3.2.noarch", "Installed: selinux-policy-3.14.3-54.el8_3.2.noarch", "Removed: selinux-policy-3.14.3-41.el8_2.8.noarch", "Removed: selinux-policy-targeted-3.14.3-41.el8_2.8.noarch"]} 2021-03-15T21:47:37.667 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:37 +0000 (0:00:00.325) 0:02:29.380 ********** 2021-03-15T21:47:38.106 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-03-15T21:47:38.108 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-03-15T21:47:38.132 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:38 +0000 (0:00:00.462) 0:02:29.843 ********** 2021-03-15T21:47:38.169 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-03-15T21:47:38.192 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:38 +0000 (0:00:00.062) 0:02:29.905 ********** 2021-03-15T21:47:54.823 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-15T21:47:54.825 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.451595", "end": "2021-03-15 21:47:54.800260", "failed_when_result": false, "rc": 0, "start": "2021-03-15 21:47:38.348665", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.394486", "end": "2021-03-15 21:47:54.803942", "failed_when_result": false, "rc": 0, "start": "2021-03-15 21:47:38.409456", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-03-15T21:47:54.846 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:54 +0000 (0:00:16.653) 0:02:46.559 ********** 2021-03-15T21:47:54.866 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:54 +0000 (0:00:00.018) 0:02:46.578 ********** 2021-03-15T21:47:55.350 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-15T21:47:55.352 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-03-15T21:47:55.372 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:55 +0000 (0:00:00.506) 0:02:47.085 ********** 2021-03-15T21:47:55.379 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-03-15T21:47:55.404 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:55 +0000 (0:00:00.031) 0:02:47.116 ********** 2021-03-15T21:47:55.618 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-15T21:47:55.620 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.016748", "end": "2021-03-15 21:47:55.597732", "rc": 0, "start": "2021-03-15 21:47:55.580984", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:47:55.642 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:55 +0000 (0:00:00.237) 0:02:47.354 ********** 2021-03-15T21:47:55.659 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.016999", "end": "2021-03-15 21:47:55.633139", "rc": 0, "start": "2021-03-15 21:47:55.616140", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:47:55.683 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:55 +0000 (0:00:00.041) 0:02:47.396 ********** 2021-03-15T21:47:55.876 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-15T21:47:55.877 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.004717", "end": "2021-03-15 21:47:55.853980", "rc": 0, "start": "2021-03-15 21:47:55.849263", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:47:55.898 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:55 +0000 (0:00:00.215) 0:02:47.611 ********** 2021-03-15T21:47:55.937 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.023521", "end": "2021-03-15 21:47:55.919358", "rc": 0, "start": "2021-03-15 21:47:55.895837", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:47:55.963 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:47:55 +0000 (0:00:00.064) 0:02:47.675 ********** 2021-03-15T21:48:12.298 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-15T21:48:12.300 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.153958", "end": "2021-03-15 21:48:12.268501", "rc": 0, "start": "2021-03-15 21:47:56.114543", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:48:12.321 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:12 +0000 (0:00:16.358) 0:03:04.034 ********** 2021-03-15T21:48:12.569 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-15T21:48:12.571 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} TASK [common : nrpe - Load SELinux policy package] ***************************** changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.370315", "end": "2021-03-15 21:48:12.550827", "rc": 0, "start": "2021-03-15 21:47:56.180512", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:48:12.593 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:12 +0000 (0:00:00.271) 0:03:04.305 ********** 2021-03-15T21:48:12.612 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:12 +0000 (0:00:00.019) 0:03:04.325 ********** 2021-03-15T21:48:12.661 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-15T21:48:12.663 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-03-15T21:48:12.688 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:12 +0000 (0:00:00.075) 0:03:04.401 ********** 2021-03-15T21:48:12.787 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-15T21:48:12.789 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-15T21:48:12.809 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:12 +0000 (0:00:00.121) 0:03:04.522 ********** 2021-03-15T21:48:12.857 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-15T21:48:12.859 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-03-15T21:48:12.881 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:12 +0000 (0:00:00.071) 0:03:04.593 ********** 2021-03-15T21:48:12.898 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-15T21:48:12.900 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-03-15T21:48:12.924 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:12 +0000 (0:00:00.043) 0:03:04.637 ********** 2021-03-15T21:48:12.953 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-15T21:48:12.955 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-03-15T21:48:12.977 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:12 +0000 (0:00:00.052) 0:03:04.690 ********** 2021-03-15T21:48:13.004 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-15T21:48:13.007 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-15T21:48:13.029 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.052) 0:03:04.742 ********** 2021-03-15T21:48:13.056 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-15T21:48:13.058 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-15T21:48:13.080 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.050) 0:03:04.793 ********** 2021-03-15T21:48:13.161 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-15T21:48:13.163 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "packages_to_upgrade": ["libgcrypt"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-03-15T21:48:13.187 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.106) 0:03:04.899 ********** 2021-03-15T21:48:13.226 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-15T21:48:13.228 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-03-15T21:48:13.249 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.062) 0:03:04.962 ********** 2021-03-15T21:48:13.277 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-15T21:48:13.279 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-15T21:48:13.302 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.052) 0:03:05.014 ********** 2021-03-15T21:48:13.485 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-15T21:48:13.487 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-03-15T21:48:13.511 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.209) 0:03:05.223 ********** 2021-03-15T21:48:13.518 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-15T21:48:13.520 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-03-15T21:48:13.543 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.031) 0:03:05.255 ********** 2021-03-15T21:48:13.583 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-03-15T21:48:13.584 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:13.607 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.064) 0:03:05.320 ********** 2021-03-15T21:48:13.677 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:13.702 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.095) 0:03:05.415 ********** 2021-03-15T21:48:13.771 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-15T21:48:13.773 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-03-15T21:48:13.799 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.096) 0:03:05.512 ********** 2021-03-15T21:48:13.869 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:13.892 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.092) 0:03:05.604 ********** 2021-03-15T21:48:13.961 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-15T21:48:13.962 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:13.984 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:13 +0000 (0:00:00.092) 0:03:05.696 ********** 2021-03-15T21:48:13.992 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-15T21:48:13.994 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-03-15T21:48:14.017 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:14 +0000 (0:00:00.032) 0:03:05.729 ********** 2021-03-15T21:48:14.246 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-03-15T21:48:14.270 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:14 +0000 (0:00:00.253) 0:03:05.983 ********** 2021-03-15T21:48:14.636 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-15T21:48:14.638 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-03-15T21:48:14.670 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:14 +0000 (0:00:00.399) 0:03:06.382 ********** 2021-03-15T21:48:14.743 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-15T21:48:14.745 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi076"}, "changed": false} 2021-03-15T21:48:14.769 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:14 +0000 (0:00:00.099) 0:03:06.482 ********** 2021-03-15T21:48:14.849 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-15T21:48:14.851 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} 2021-03-15T21:48:14.874 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:14 +0000 (0:00:00.104) 0:03:06.586 ********** 2021-03-15T21:48:14.955 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-15T21:48:14.957 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi125"}, "changed": false} 2021-03-15T21:48:14.978 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:14 +0000 (0:00:00.104) 0:03:06.691 ********** 2021-03-15T21:48:15.893 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-15T21:48:15.894 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi076.front.sepia.ceph.com", "ansible_hostname": "smithi076", "ansible_nodename": "smithi076"}, "changed": true, "name": "smithi076"} changed: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi125.front.sepia.ceph.com", "ansible_hostname": "smithi125", "ansible_nodename": "smithi125"}, "changed": true, "name": "smithi125"} 2021-03-15T21:48:15.916 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:15 +0000 (0:00:00.937) 0:03:07.629 ********** 2021-03-15T21:48:15.940 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:15 +0000 (0:00:00.024) 0:03:07.653 ********** 2021-03-15T21:48:16.429 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-15T21:48:16.430 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-03-15T21:48:16.453 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:16 +0000 (0:00:00.512) 0:03:08.165 ********** 2021-03-15T21:48:16.461 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-03-15T21:48:16.491 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:16 +0000 (0:00:00.037) 0:03:08.203 ********** 2021-03-15T21:48:16.952 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-15T21:48:16.954 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-03-15T21:48:16.976 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:16 +0000 (0:00:00.485) 0:03:08.689 ********** 2021-03-15T21:48:17.015 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-03-15T21:48:17.038 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:17 +0000 (0:00:00.061) 0:03:08.751 ********** 2021-03-15T21:48:17.557 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-15T21:48:17.559 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-15T21:48:17.580 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:17 +0000 (0:00:00.541) 0:03:09.292 ********** 2021-03-15T21:48:17.607 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-15T21:48:17.630 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:17 +0000 (0:00:00.050) 0:03:09.343 ********** 2021-03-15T21:48:18.109 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-15T21:48:18.111 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-03-15T21:48:18.134 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:18 +0000 (0:00:00.503) 0:03:09.846 ********** 2021-03-15T21:48:18.157 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-03-15T21:48:18.181 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:18 +0000 (0:00:00.047) 0:03:09.894 ********** 2021-03-15T21:48:18.388 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-15T21:48:18.389 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-03-15T21:48:18.411 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:18 +0000 (0:00:00.229) 0:03:10.124 ********** 2021-03-15T21:48:18.429 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-03-15T21:48:18.452 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:18 +0000 (0:00:00.040) 0:03:10.164 ********** 2021-03-15T21:48:18.650 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-15T21:48:18.652 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-03-15T21:48:18.675 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:18 +0000 (0:00:00.223) 0:03:10.388 ********** 2021-03-15T21:48:18.704 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-03-15T21:48:18.729 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:18 +0000 (0:00:00.053) 0:03:10.441 ********** 2021-03-15T21:48:18.942 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-15T21:48:18.944 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-15T21:48:18.966 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:18 +0000 (0:00:00.236) 0:03:10.678 ********** 2021-03-15T21:48:19.004 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-15T21:48:19.027 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:19 +0000 (0:00:00.061) 0:03:10.739 ********** 2021-03-15T21:48:19.511 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-15T21:48:19.513 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-03-15T21:48:19.535 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:19 +0000 (0:00:00.508) 0:03:11.248 ********** 2021-03-15T21:48:19.546 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-03-15T21:48:19.570 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:19 +0000 (0:00:00.034) 0:03:11.282 ********** 2021-03-15T21:48:20.050 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-15T21:48:20.052 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-03-15T21:48:20.078 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:20 +0000 (0:00:00.507) 0:03:11.790 ********** 2021-03-15T21:48:20.097 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-03-15T21:48:20.127 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:20 +0000 (0:00:00.049) 0:03:11.839 ********** 2021-03-15T21:48:22.297 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-15T21:48:22.299 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.967200", "end": "2021-03-15 21:48:22.268178", "rc": 0, "start": "2021-03-15 21:48:20.300978", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:48:22.323 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:22 +0000 (0:00:02.196) 0:03:14.035 ********** 2021-03-15T21:48:22.363 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:01.989458", "end": "2021-03-15 21:48:22.336890", "rc": 0, "start": "2021-03-15 21:48:20.347432", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:48:22.390 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:22 +0000 (0:00:00.066) 0:03:14.102 ********** 2021-03-15T21:48:22.881 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-15T21:48:22.882 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-03-15T21:48:22.935 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-03-15T21:48:23.323 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-03-15T21:48:23.365 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-03-15T21:48:23.779 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-03-15T21:48:23.780 INFO:teuthology.task.ansible.out: 2021-03-15T21:48:23.806 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:23 +0000 (0:00:01.416) 0:03:15.519 ********** 2021-03-15T21:48:23.814 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-03-15T21:48:23.840 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:23 +0000 (0:00:00.033) 0:03:15.552 ********** 2021-03-15T21:48:24.322 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-15T21:48:24.324 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-03-15T21:48:24.347 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:24 +0000 (0:00:00.506) 0:03:16.059 ********** 2021-03-15T21:48:24.365 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-03-15T21:48:24.391 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:24 +0000 (0:00:00.044) 0:03:16.103 ********** 2021-03-15T21:48:24.859 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-15T21:48:24.861 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-03-15T21:48:24.893 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-03-15T21:48:25.307 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-03-15T21:48:25.340 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-03-15T21:48:25.765 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-03-15T21:48:25.791 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:25 +0000 (0:00:01.399) 0:03:17.503 ********** 2021-03-15T21:48:25.798 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-03-15T21:48:25.826 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:25 +0000 (0:00:00.035) 0:03:17.539 ********** 2021-03-15T21:48:25.855 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-15T21:48:25.878 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:25 +0000 (0:00:00.052) 0:03:17.591 ********** 2021-03-15T21:48:25.919 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:25 +0000 (0:00:00.040) 0:03:17.631 ********** 2021-03-15T21:48:25.956 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-15T21:48:25.958 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:25.980 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:25 +0000 (0:00:00.060) 0:03:17.692 ********** 2021-03-15T21:48:25.988 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.011 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.031) 0:03:17.724 ********** 2021-03-15T21:48:26.059 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-15T21:48:26.060 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.083 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.071) 0:03:17.796 ********** 2021-03-15T21:48:26.100 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.102 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.125 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.041) 0:03:17.837 ********** 2021-03-15T21:48:26.151 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-15T21:48:26.153 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.178 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.053) 0:03:17.890 ********** 2021-03-15T21:48:26.197 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.224 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.046) 0:03:17.937 ********** 2021-03-15T21:48:26.252 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:48:26.253 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-03-15T21:48:26.275 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.051) 0:03:17.988 ********** 2021-03-15T21:48:26.304 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-03-15T21:48:26.327 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.051) 0:03:18.039 ********** 2021-03-15T21:48:26.343 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-15T21:48:26.345 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.368 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.040) 0:03:18.080 ********** 2021-03-15T21:48:26.395 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.421 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.052) 0:03:18.133 ********** 2021-03-15T21:48:26.437 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-15T21:48:26.439 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.461 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.040) 0:03:18.174 ********** 2021-03-15T21:48:26.490 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.513 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.051) 0:03:18.225 ********** 2021-03-15T21:48:26.700 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-15T21:48:26.702 INFO:teuthology.task.ansible.out:fatal: [smithi076.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.006725", "end": "2021-03-15 21:48:26.682458", "msg": "non-zero return code", "rc": 1, "start": "2021-03-15 21:48:26.675733", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-03-15T21:48:26.724 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.211) 0:03:18.437 ********** 2021-03-15T21:48:26.754 INFO:teuthology.task.ansible.out:fatal: [smithi125.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007328", "end": "2021-03-15 21:48:26.733489", "msg": "non-zero return code", "rc": 1, "start": "2021-03-15 21:48:26.726161", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-03-15T21:48:26.777 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.053) 0:03:18.490 ********** 2021-03-15T21:48:26.794 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-15T21:48:26.796 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.817 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.039) 0:03:18.529 ********** 2021-03-15T21:48:26.846 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:26.874 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:26 +0000 (0:00:00.057) 0:03:18.587 ********** 2021-03-15T21:48:29.234 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-15T21:48:29.236 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64"]} 2021-03-15T21:48:29.257 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:29 +0000 (0:00:02.382) 0:03:20.969 ********** 2021-03-15T21:48:29.370 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64"]} 2021-03-15T21:48:29.393 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:29 +0000 (0:00:00.136) 0:03:21.106 ********** 2021-03-15T21:48:31.312 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-15T21:48:31.314 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64"]} 2021-03-15T21:48:31.336 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:31 +0000 (0:00:01.941) 0:03:23.047 ********** 2021-03-15T21:48:31.519 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64"]} 2021-03-15T21:48:31.543 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:31 +0000 (0:00:00.207) 0:03:23.255 ********** 2021-03-15T21:48:45.948 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-15T21:48:45.950 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: python3-libdnf-0.48.0-5.el8.x86_64", "Installed: libdnf-0.48.0-5.el8.x86_64", "Installed: python3-librepo-1.12.0-2.el8.x86_64", "Installed: boost-chrono-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: yum-4.2.23-4.el8.noarch", "Installed: yum-utils-4.0.17-5.el8.noarch", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: boost-system-1.66.0-10.el8.x86_64", "Installed: boost-thread-1.66.0-10.el8.x86_64", "Installed: librepo-1.12.0-2.el8.x86_64", "Installed: dnf-4.2.23-4.el8.noarch", "Installed: dnf-data-4.2.23-4.el8.noarch", "Installed: dnf-plugins-core-4.0.17-5.el8.noarch", "Installed: qemu-img-15:4.2.0-34.module_el8.3.0+713+8ddc4ca0.4.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+713+8ddc4ca0.4.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+713+8ddc4ca0.4.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+713+8ddc4ca0.4.x86_64", "Installed: python3-hawkey-0.48.0-5.el8.x86_64", "Installed: libmodulemd-2.9.4-2.el8.x86_64", "Installed: python3-dnf-4.2.23-4.el8.noarch", "Installed: python3-dnf-plugins-core-4.0.17-5.el8.noarch", "Installed: seavgabios-bin-1.13.0-2.module_el8.3.0+555+a55c8938.noarch", "Removed: qemu-img-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: qemu-kvm-common-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: boost-chrono-1.66.0-6.el8.x86_64", "Removed: qemu-kvm-core-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: seavgabios-bin-1.11.1-4.module_el8.1.0+248+298dec18.noarch", "Removed: libdnf-0.35.1-9.el8_1.x86_64", "Removed: boost-system-1.66.0-6.el8.x86_64", "Removed: boost-thread-1.66.0-6.el8.x86_64", "Removed: dnf-4.2.7-7.el8_1.noarch", "Removed: dnf-data-4.2.7-7.el8_1.noarch", "Removed: librepo-1.11.0-2.el8.x86_64 2021-03-15T21:48:45.952 INFO:teuthology.task.ansible.out:", "Removed: dnf-plugins-core-4.0.8-3.el8.noarch", "Removed: python3-dnf-4.2.7-7.el8_1.noarch", "Removed: python3-dnf-plugins-core-4.0.8-3.el8.noarch", "Removed: python3-hawkey-0.35.1-9.el8_1.x86_64", "Removed: yum-4.2.7-7.el8_1.noarch", "Removed: yum-utils-4.0.8-3.el8.noarch", "Removed: python3-libdnf-0.35.1-9.el8_1.x86_64", "Removed: python3-librepo-1.11.0-2.el8.x86_64"]} 2021-03-15T21:48:45.971 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:45 +0000 (0:00:14.428) 0:03:37.683 ********** 2021-03-15T21:48:45.973 INFO:teuthology.task.ansible.out: 2021-03-15T21:48:46.041 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-15T21:48:46.042 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:46.063 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:46 +0000 (0:00:00.092) 0:03:37.776 ********** 2021-03-15T21:48:46.547 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-15T21:48:46.548 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: python3-libdnf-0.48.0-5.el8.x86_64", "Installed: libdnf-0.48.0-5.el8.x86_64", "Installed: python3-librepo-1.12.0-2.el8.x86_64", "Installed: boost-chrono-1.66.0-10.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: yum-4.2.23-4.el8.noarch", "Installed: yum-utils-4.0.17-5.el8.noarch", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: boost-system-1.66.0-10.el8.x86_64", "Installed: boost-thread-1.66.0-10.el8.x86_64", "Installed: librepo-1.12.0-2.el8.x86_64", "Installed: dnf-4.2.23-4.el8.noarch", "Installed: dnf-data-4.2.23-4.el8.noarch", "Installed: dnf-plugins-core-4.0.17-5.el8.noarch", "Installed: qemu-img-15:4.2.0-34.module_el8.3.0+713+8ddc4ca0.4.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+713+8ddc4ca0.4.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+713+8ddc4ca0.4.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+713+8ddc4ca0.4.x86_64", "Installed: python3-hawkey-0.48.0-5.el8.x86_64", "Installed: libmodulemd-2.9.4-2.el8.x86_64", "Installed: python3-dnf-4.2.23-4.el8.noarch", "Installed: python3-dnf-plugins-core-4.0.17-5.el8.noarch", "Installed: seavgabios-bin-1.13.0-2.module_el8.3.0+555+a55c8938.noarch", "Removed: qemu-img-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: qemu-kvm-common-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: boost-chrono-1.66.0-6.el8.x86_64", "Removed: qemu-kvm-core-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: seavgabios-bin-1.11.1-4.module_el8.1.0+248+298dec18.noarch", "Removed: libdnf-0.35.1-9.el8_1.x86_64", "Removed: boost-system-1.66.0-6.el8.x86_64", "Removed: boost-thread-1.66.0-6.el8.x86_64", "Removed: dnf-4.2.7-7.el8_1.noarch", "Removed: dnf-data-4.2.7-7.el8_1.noarch", "Removed: librepo-1.11.0-2.el8.x86_64 2021-03-15T21:48:46.550 INFO:teuthology.task.ansible.out:", "Removed: dnf-plugins-core-4.0.8-3.el8.noarch", "Removed: python3-dnf-4.2.7-7.el8_1.noarch", "Removed: python3-dnf-plugins-core-4.0.8-3.el8.noarch", "Removed: python3-hawkey-0.35.1-9.el8_1.x86_64", "Removed: yum-4.2.7-7.el8_1.noarch", "Removed: yum-utils-4.0.8-3.el8.noarch", "Removed: python3-libdnf-0.35.1-9.el8_1.x86_64", "Removed: python3-librepo-1.11.0-2.el8.x86_64"]} 2021-03-15T21:48:46.571 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:46 +0000 (0:00:00.507) 0:03:38.284 ********** 2021-03-15T21:48:46.646 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-15T21:48:46.648 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:46.668 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:46 +0000 (0:00:00.097) 0:03:38.381 ********** 2021-03-15T21:48:47.209 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-15T21:48:47.211 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-15T21:48:47.232 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:47 +0000 (0:00:00.563) 0:03:38.944 ********** 2021-03-15T21:48:47.832 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-15T21:48:47.855 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:47 +0000 (0:00:00.623) 0:03:39.567 ********** 2021-03-15T21:48:49.593 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-15T21:48:49.595 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: libgcrypt-1.8.5-4.el8.x86_64", "Removed: libgcrypt-1.8.3-4.el8.x86_64"]} 2021-03-15T21:48:49.616 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:49 +0000 (0:00:01.761) 0:03:41.329 ********** 2021-03-15T21:48:49.967 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-15T21:48:49.968 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "dbus.socket system.slice sysinit.target polkit.service dbus.service basic.target", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target network-pre.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service ipset.service ip6tables.service shutdown.target iptables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNode 2021-03-15T21:48:49.970 INFO:teuthology.task.ansible.out:s": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127270", "LimitNPROCSoft": "127270", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "L 2021-03-15T21:48:49.972 INFO:teuthology.task.ansible.out:imitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127270", "LimitSIGPENDINGSoft": "127270", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWe 2021-03-15T21:48:49.973 INFO:teuthology.task.ansible.out:ight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203632", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-15T21:48:49.991 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:49 +0000 (0:00:00.374) 0:03:41.704 ********** 2021-03-15T21:48:50.207 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-15T21:48:50.209 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: libgcrypt-1.8.5-4.el8.x86_64", "Removed: libgcrypt-1.8.3-4.el8.x86_64"]} 2021-03-15T21:48:50.230 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:50 +0000 (0:00:00.238) 0:03:41.943 ********** 2021-03-15T21:48:50.341 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ fatal: [smithi076.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-03-15T21:48:50.364 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:50 +0000 (0:00:00.133) 0:03:42.077 ********** 2021-03-15T21:48:50.591 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-15T21:48:50.592 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "basic.target polkit.service sysinit.target dbus.service dbus.socket system.slice", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "ebtables.service ip6tables.service ipset.service iptables.service shutdown.target", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNode 2021-03-15T21:48:50.594 INFO:teuthology.task.ansible.out:s": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127269", "LimitNPROCSoft": "127269", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "L 2021-03-15T21:48:50.596 INFO:teuthology.task.ansible.out:imitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127269", "LimitSIGPENDINGSoft": "127269", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target dbus.socket system.slice", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWe 2021-03-15T21:48:50.597 INFO:teuthology.task.ansible.out:ight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203631", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-15T21:48:50.617 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:50 +0000 (0:00:00.252) 0:03:42.329 ********** 2021-03-15T21:48:50.960 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-15T21:48:50.962 INFO:teuthology.task.ansible.out:fatal: [smithi125.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-03-15T21:48:50.982 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:50 +0000 (0:00:00.365) 0:03:42.694 ********** 2021-03-15T21:48:51.157 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-15T21:48:51.159 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-03-15T21:48:51.180 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.198) 0:03:42.893 ********** 2021-03-15T21:48:51.250 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-15T21:48:51.252 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.272 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.091) 0:03:42.984 ********** 2021-03-15T21:48:51.309 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-15T21:48:51.311 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-03-15T21:48:51.333 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.061) 0:03:43.046 ********** 2021-03-15T21:48:51.339 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-15T21:48:51.341 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.364 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.030) 0:03:43.076 ********** 2021-03-15T21:48:51.401 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-15T21:48:51.403 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.424 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.061) 0:03:43.137 ********** 2021-03-15T21:48:51.431 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-15T21:48:51.433 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.455 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.030) 0:03:43.168 ********** 2021-03-15T21:48:51.493 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-15T21:48:51.494 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.517 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.061) 0:03:43.230 ********** 2021-03-15T21:48:51.524 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.548 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.030) 0:03:43.261 ********** 2021-03-15T21:48:51.587 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-15T21:48:51.589 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.610 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.062) 0:03:43.323 ********** 2021-03-15T21:48:51.627 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-15T21:48:51.629 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.651 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.040) 0:03:43.363 ********** 2021-03-15T21:48:51.678 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-15T21:48:51.680 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.702 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.051) 0:03:43.415 ********** 2021-03-15T21:48:51.719 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-15T21:48:51.721 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.742 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.040) 0:03:43.455 ********** 2021-03-15T21:48:51.770 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-15T21:48:51.772 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.794 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.051) 0:03:43.506 ********** 2021-03-15T21:48:51.811 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-15T21:48:51.834 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.040) 0:03:43.547 ********** 2021-03-15T21:48:51.862 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-15T21:48:51.864 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.885 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.050) 0:03:43.598 ********** 2021-03-15T21:48:51.903 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-15T21:48:51.905 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:51.926 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.040) 0:03:43.639 ********** 2021-03-15T21:48:51.964 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-15T21:48:51.987 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:51 +0000 (0:00:00.061) 0:03:43.700 ********** 2021-03-15T21:48:52.004 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-15T21:48:52.006 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.028 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.040) 0:03:43.740 ********** 2021-03-15T21:48:52.056 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-15T21:48:52.058 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.081 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.052) 0:03:43.793 ********** 2021-03-15T21:48:52.108 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-15T21:48:52.110 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.132 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.051) 0:03:43.845 ********** 2021-03-15T21:48:52.150 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-15T21:48:52.152 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.178 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.045) 0:03:43.890 ********** 2021-03-15T21:48:52.205 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-15T21:48:52.207 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.229 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.051) 0:03:43.942 ********** 2021-03-15T21:48:52.247 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-15T21:48:52.248 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi125.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.274 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.045) 0:03:43.987 ********** 2021-03-15T21:48:52.293 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-15T21:48:52.317 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.042) 0:03:44.030 ********** 2021-03-15T21:48:52.345 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-15T21:48:52.347 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.369 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.051) 0:03:44.082 ********** 2021-03-15T21:48:52.387 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-15T21:48:52.389 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.411 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.041) 0:03:44.123 ********** 2021-03-15T21:48:52.439 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-15T21:48:52.463 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.052) 0:03:44.176 ********** 2021-03-15T21:48:52.480 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-15T21:48:52.481 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.504 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.040) 0:03:44.217 ********** 2021-03-15T21:48:52.533 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-15T21:48:52.535 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.558 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.053) 0:03:44.270 ********** 2021-03-15T21:48:52.576 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-15T21:48:52.577 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.600 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.042) 0:03:44.312 ********** 2021-03-15T21:48:52.629 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-15T21:48:52.631 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.653 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.053) 0:03:44.366 ********** 2021-03-15T21:48:52.670 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-15T21:48:52.672 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.697 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.043) 0:03:44.410 ********** 2021-03-15T21:48:52.717 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-15T21:48:52.718 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.742 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.045) 0:03:44.455 ********** 2021-03-15T21:48:52.760 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-15T21:48:52.761 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.787 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.044) 0:03:44.499 ********** 2021-03-15T21:48:52.816 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-15T21:48:52.818 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.844 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.056) 0:03:44.556 ********** 2021-03-15T21:48:52.861 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-15T21:48:52.863 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.886 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.042) 0:03:44.598 ********** 2021-03-15T21:48:52.913 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.937 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.051) 0:03:44.650 ********** 2021-03-15T21:48:52.955 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-15T21:48:52.957 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:52.980 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:52 +0000 (0:00:00.042) 0:03:44.693 ********** 2021-03-15T21:48:53.007 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-15T21:48:53.009 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.031 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.051) 0:03:44.744 ********** 2021-03-15T21:48:53.059 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-15T21:48:53.061 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.083 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.051) 0:03:44.796 ********** 2021-03-15T21:48:53.110 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-15T21:48:53.112 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.134 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.051) 0:03:44.847 ********** 2021-03-15T21:48:53.151 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-15T21:48:53.153 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.174 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.039) 0:03:44.887 ********** 2021-03-15T21:48:53.202 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-15T21:48:53.204 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.225 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.051) 0:03:44.938 ********** 2021-03-15T21:48:53.243 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-15T21:48:53.245 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.266 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.040) 0:03:44.979 ********** 2021-03-15T21:48:53.294 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-15T21:48:53.296 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.321 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.054) 0:03:45.034 ********** 2021-03-15T21:48:53.338 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-15T21:48:53.340 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.362 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.041) 0:03:45.075 ********** 2021-03-15T21:48:53.390 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-15T21:48:53.392 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.415 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.052) 0:03:45.127 ********** 2021-03-15T21:48:53.432 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-15T21:48:53.434 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.456 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.041) 0:03:45.169 ********** 2021-03-15T21:48:53.484 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-15T21:48:53.486 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.510 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.053) 0:03:45.222 ********** 2021-03-15T21:48:53.528 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.552 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.042) 0:03:45.265 ********** 2021-03-15T21:48:53.601 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-15T21:48:53.602 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.624 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.071) 0:03:45.337 ********** 2021-03-15T21:48:53.633 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-15T21:48:53.635 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.671 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.046) 0:03:45.383 ********** 2021-03-15T21:48:53.689 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-15T21:48:53.691 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.713 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.042) 0:03:45.425 ********** 2021-03-15T21:48:53.740 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-15T21:48:53.742 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.764 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.051) 0:03:45.477 ********** 2021-03-15T21:48:53.781 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-15T21:48:53.783 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.805 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.041) 0:03:45.518 ********** 2021-03-15T21:48:53.834 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-15T21:48:53.836 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.858 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.052) 0:03:45.570 ********** 2021-03-15T21:48:53.874 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-15T21:48:53.876 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.899 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.041) 0:03:45.612 ********** 2021-03-15T21:48:53.901 INFO:teuthology.task.ansible.out: 2021-03-15T21:48:53.927 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-15T21:48:53.929 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.951 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.051) 0:03:45.663 ********** 2021-03-15T21:48:53.969 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-15T21:48:53.971 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:53.996 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:53 +0000 (0:00:00.045) 0:03:45.708 ********** 2021-03-15T21:48:54.023 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-15T21:48:54.025 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.058 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.062) 0:03:45.770 ********** 2021-03-15T21:48:54.078 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-15T21:48:54.081 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.107 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.049) 0:03:45.820 ********** 2021-03-15T21:48:54.136 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-15T21:48:54.138 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.160 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.052) 0:03:45.872 ********** 2021-03-15T21:48:54.188 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-15T21:48:54.190 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.222 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.062) 0:03:45.935 ********** 2021-03-15T21:48:54.243 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-15T21:48:54.245 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.273 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.050) 0:03:45.985 ********** 2021-03-15T21:48:54.302 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-15T21:48:54.304 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.337 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.063) 0:03:46.049 ********** 2021-03-15T21:48:54.367 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-15T21:48:54.369 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.400 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.063) 0:03:46.113 ********** 2021-03-15T21:48:54.418 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-15T21:48:54.420 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.448 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.048) 0:03:46.161 ********** 2021-03-15T21:48:54.480 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-15T21:48:54.482 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.511 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.062) 0:03:46.223 ********** 2021-03-15T21:48:54.533 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-15T21:48:54.534 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.562 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.051) 0:03:46.275 ********** 2021-03-15T21:48:54.590 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-15T21:48:54.592 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.615 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.052) 0:03:46.328 ********** 2021-03-15T21:48:54.635 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-15T21:48:54.636 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.660 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.044) 0:03:46.373 ********** 2021-03-15T21:48:54.700 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:48:54.702 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.725 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.064) 0:03:46.437 ********** 2021-03-15T21:48:54.732 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-15T21:48:54.734 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.765 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.039) 0:03:46.477 ********** 2021-03-15T21:48:54.796 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-15T21:48:54.799 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.820 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.055) 0:03:46.533 ********** 2021-03-15T21:48:54.837 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:48:54.839 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.861 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.041) 0:03:46.574 ********** 2021-03-15T21:48:54.900 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-15T21:48:54.902 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.928 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.066) 0:03:46.640 ********** 2021-03-15T21:48:54.935 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-15T21:48:54.936 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:54.959 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:54 +0000 (0:00:00.031) 0:03:46.672 ********** 2021-03-15T21:48:54.998 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-15T21:48:55.000 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:55.023 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:55 +0000 (0:00:00.063) 0:03:46.736 ********** 2021-03-15T21:48:55.033 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-15T21:48:55.034 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:55.064 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:55 +0000 (0:00:00.041) 0:03:46.777 ********** 2021-03-15T21:48:55.116 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-15T21:48:55.118 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:55.143 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:55 +0000 (0:00:00.078) 0:03:46.855 ********** 2021-03-15T21:48:55.152 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-15T21:48:55.154 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:55.180 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:55 +0000 (0:00:00.036) 0:03:46.892 ********** 2021-03-15T21:48:55.408 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-15T21:48:55.410 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:55.430 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:55 +0000 (0:00:00.250) 0:03:47.143 ********** 2021-03-15T21:48:55.542 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-15T21:48:55.544 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-03-15T21:48:55.566 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:55 +0000 (0:00:00.135) 0:03:47.279 ********** 2021-03-15T21:48:55.703 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-03-15T21:48:55.731 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:55 +0000 (0:00:00.164) 0:03:47.443 ********** 2021-03-15T21:48:55.809 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-15T21:48:55.811 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-15T21:48:55.832 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:55 +0000 (0:00:00.101) 0:03:47.545 ********** 2021-03-15T21:48:55.996 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-15T21:48:56.023 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.190) 0:03:47.736 ********** 2021-03-15T21:48:56.072 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-15T21:48:56.074 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-03-15T21:48:56.098 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.075) 0:03:47.811 ********** 2021-03-15T21:48:56.160 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-15T21:48:56.162 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:56.186 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.087) 0:03:47.898 ********** 2021-03-15T21:48:56.245 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:56.276 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.090) 0:03:47.989 ********** 2021-03-15T21:48:56.285 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-15T21:48:56.287 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-03-15T21:48:56.321 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.044) 0:03:48.034 ********** 2021-03-15T21:48:56.359 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-15T21:48:56.361 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:56.384 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.062) 0:03:48.097 ********** 2021-03-15T21:48:56.391 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-15T21:48:56.392 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:56.416 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.031) 0:03:48.128 ********** 2021-03-15T21:48:56.443 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-15T21:48:56.444 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:56.474 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.058) 0:03:48.186 ********** 2021-03-15T21:48:56.484 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-15T21:48:56.485 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:56.520 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.045) 0:03:48.232 ********** 2021-03-15T21:48:56.538 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-15T21:48:56.540 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:56.563 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.043) 0:03:48.276 ********** 2021-03-15T21:48:56.612 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-15T21:48:56.614 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:56.636 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.072) 0:03:48.348 ********** 2021-03-15T21:48:56.760 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-15T21:48:56.762 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:56.783 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.147) 0:03:48.495 ********** 2021-03-15T21:48:56.844 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-15T21:48:56.846 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:56.867 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.084) 0:03:48.579 ********** 2021-03-15T21:48:56.980 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-15T21:48:56.982 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.004 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:56 +0000 (0:00:00.136) 0:03:48.716 ********** 2021-03-15T21:48:57.083 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-15T21:48:57.085 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.108 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.104) 0:03:48.820 ********** 2021-03-15T21:48:57.188 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-15T21:48:57.190 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.224 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.116) 0:03:48.937 ********** 2021-03-15T21:48:57.309 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.427 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.203) 0:03:49.140 ********** 2021-03-15T21:48:57.433 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-15T21:48:57.435 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.457 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.030) 0:03:49.170 ********** 2021-03-15T21:48:57.495 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-15T21:48:57.565 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Monday 15 March 2021 21:48:57 +0000 (0:00:00.061) 0:03:49.232 ********** TASK [testnode : Set grub config.] ********************************************* skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.571 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.050) 0:03:49.283 ********** 2021-03-15T21:48:57.598 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-15T21:48:57.601 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.633 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.063) 0:03:49.346 ********** 2021-03-15T21:48:57.653 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-15T21:48:57.656 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.677 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.043) 0:03:49.390 ********** 2021-03-15T21:48:57.707 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-15T21:48:57.708 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.730 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.053) 0:03:49.443 ********** 2021-03-15T21:48:57.747 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-15T21:48:57.749 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.771 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.040) 0:03:49.483 ********** 2021-03-15T21:48:57.798 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-15T21:48:57.800 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.822 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.050) 0:03:49.534 ********** 2021-03-15T21:48:57.849 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-15T21:48:57.851 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.873 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.050) 0:03:49.585 ********** 2021-03-15T21:48:57.910 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-15T21:48:57.912 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.934 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.061) 0:03:49.647 ********** 2021-03-15T21:48:57.951 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-15T21:48:57.953 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:57.975 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:57 +0000 (0:00:00.040) 0:03:49.688 ********** 2021-03-15T21:48:58.002 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-15T21:48:58.004 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.027 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.051) 0:03:49.739 ********** 2021-03-15T21:48:58.055 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-15T21:48:58.057 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.081 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.054) 0:03:49.793 ********** 2021-03-15T21:48:58.100 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-15T21:48:58.102 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.127 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.046) 0:03:49.840 ********** 2021-03-15T21:48:58.158 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-15T21:48:58.160 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.182 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.054) 0:03:49.895 ********** 2021-03-15T21:48:58.201 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-15T21:48:58.204 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.237 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.055) 0:03:49.950 ********** 2021-03-15T21:48:58.258 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-15T21:48:58.260 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.281 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.043) 0:03:49.994 ********** 2021-03-15T21:48:58.309 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-15T21:48:58.311 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.333 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.052) 0:03:50.046 ********** 2021-03-15T21:48:58.361 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-15T21:48:58.362 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.384 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.050) 0:03:50.096 ********** 2021-03-15T21:48:58.412 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-15T21:48:58.413 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.436 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.052) 0:03:50.149 ********** 2021-03-15T21:48:58.465 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-15T21:48:58.470 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.488 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.051) 0:03:50.200 ********** 2021-03-15T21:48:58.516 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-15T21:48:58.517 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.540 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.052) 0:03:50.252 ********** 2021-03-15T21:48:58.557 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-15T21:48:58.559 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.579 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.039) 0:03:50.292 ********** 2021-03-15T21:48:58.607 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-15T21:48:58.609 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.634 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.054) 0:03:50.347 ********** 2021-03-15T21:48:58.663 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-15T21:48:58.664 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.686 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.051) 0:03:50.398 ********** 2021-03-15T21:48:58.703 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-15T21:48:58.705 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.729 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.043) 0:03:50.441 ********** 2021-03-15T21:48:58.759 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-15T21:48:58.760 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.794 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.064) 0:03:50.506 ********** 2021-03-15T21:48:58.802 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-15T21:48:58.804 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.838 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.044) 0:03:50.551 ********** 2021-03-15T21:48:58.866 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:48:58.868 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-15T21:48:58.889 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.050) 0:03:50.602 ********** 2021-03-15T21:48:58.906 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-15T21:48:58.908 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.930 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:58 +0000 (0:00:00.040) 0:03:50.643 ********** 2021-03-15T21:48:58.968 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-15T21:48:58.969 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.979 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-15T21:48:58.991 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:58.992 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.003 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-15T21:48:59.004 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Check for NVMe drive] ***************************************** skipping: [smithi076.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.030 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.099) 0:03:50.742 ********** 2021-03-15T21:48:59.049 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.019) 0:03:50.762 ********** 2021-03-15T21:48:59.149 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:48:59.151 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-15T21:48:59.172 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.122) 0:03:50.884 ********** 2021-03-15T21:48:59.252 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-15T21:48:59.254 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.263 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-15T21:48:59.275 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.276 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.286 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.287 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.297 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.323 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.151) 0:03:51.036 ********** 2021-03-15T21:48:59.340 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-15T21:48:59.341 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.006714", "end": "2021-03-15 21:48:59.311364", "rc": 0, "start": "2021-03-15 21:48:59.304650", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/bin/teuthology-lock"]} 2021-03-15T21:48:59.366 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.042) 0:03:51.079 ********** 2021-03-15T21:48:59.435 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-15T21:48:59.437 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.462 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.096) 0:03:51.175 ********** 2021-03-15T21:48:59.532 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-15T21:48:59.534 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.558 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.096) 0:03:51.271 ********** 2021-03-15T21:48:59.576 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-15T21:48:59.577 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003037", "end": "2021-03-15 21:48:59.541983", "rc": 0, "start": "2021-03-15 21:48:59.538946", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_b96569170f15eae4604f361990ea65737b28dff1/virtualenv/bin/teuthology-lock"]} 2021-03-15T21:48:59.604 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.045) 0:03:51.317 ********** 2021-03-15T21:48:59.642 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-15T21:48:59.643 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.665 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.061) 0:03:51.378 ********** 2021-03-15T21:48:59.692 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-15T21:48:59.694 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.723 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.057) 0:03:51.435 ********** 2021-03-15T21:48:59.740 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-15T21:48:59.742 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.768 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.045) 0:03:51.481 ********** 2021-03-15T21:48:59.797 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-15T21:48:59.798 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.822 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.053) 0:03:51.535 ********** 2021-03-15T21:48:59.894 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-15T21:48:59.896 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:48:59.919 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:48:59 +0000 (0:00:00.097) 0:03:51.632 ********** 2021-03-15T21:48:59.990 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-15T21:48:59.992 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:00.015 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:00 +0000 (0:00:00.096) 0:03:51.728 ********** 2021-03-15T21:49:00.884 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-15T21:49:00.886 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-15T21:49:00.908 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:00 +0000 (0:00:00.891) 0:03:52.620 ********** 2021-03-15T21:49:00.999 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-15T21:49:01.001 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356027392, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137702, 'block_used': 12957406, 'inode_total': 61054976, 'inode_available': 60986356, 'inode_used': 68620, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137702, "block_size": 4096, "block_total": 240095108, "block_used": 12957406, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986356, "inode_total": 61054976, "inode_used": 68620, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356027392, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}} 2021-03-15T21:49:01.022 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:01 +0000 (0:00:00.114) 0:03:52.735 ********** 2021-03-15T21:49:01.113 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-15T21:49:01.115 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-03-15T21:49:01.137 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:01 +0000 (0:00:00.114) 0:03:52.849 ********** 2021-03-15T21:49:01.227 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-15T21:49:01.229 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356031488, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137703, 'block_used': 12957405, 'inode_total': 61054976, 'inode_available': 60986356, 'inode_used': 68620, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137703, "block_size": 4096, "block_total": 240095108, "block_used": 12957405, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986356, "inode_total": 61054976, "inode_used": 68620, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356031488, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}} 2021-03-15T21:49:01.250 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:01 +0000 (0:00:00.113) 0:03:52.963 ********** 2021-03-15T21:49:01.277 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-15T21:49:01.278 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005155", "end": "2021-03-15 21:49:01.258046", "rc": 0, "start": "2021-03-15 21:49:01.252891", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:01.301 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:01 +0000 (0:00:00.051) 0:03:53.014 ********** 2021-03-15T21:49:01.383 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-15T21:49:01.385 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356027392, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137702, 'block_used': 12957406, 'inode_total': 61054976, 'inode_available': 60986356, 'inode_used': 68620, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137702, "block_size": 4096, "block_total": 240095108, "block_used": 12957406, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986356, "inode_total": 61054976, "inode_used": 68620, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356027392, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:01.411 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:01 +0000 (0:00:00.109) 0:03:53.124 ********** 2021-03-15T21:49:01.503 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-15T21:49:01.504 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005149", "end": "2021-03-15 21:49:01.478325", "rc": 0, "start": "2021-03-15 21:49:01.473176", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-15T21:49:01.546 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:01 +0000 (0:00:00.134) 0:03:53.258 ********** 2021-03-15T21:49:01.566 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:01 +0000 (0:00:00.020) 0:03:53.279 ********** 2021-03-15T21:49:01.681 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-15T21:49:01.682 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356031488, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137703, 'block_used': 12957405, 'inode_total': 61054976, 'inode_available': 60986356, 'inode_used': 68620, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137703, "block_size": 4096, "block_total": 240095108, "block_used": 12957405, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986356, "inode_total": 61054976, "inode_used": 68620, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356031488, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:01.705 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:01 +0000 (0:00:00.139) 0:03:53.418 ********** 2021-03-15T21:49:01.796 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-15T21:49:01.822 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:01 +0000 (0:00:00.116) 0:03:53.534 ********** 2021-03-15T21:49:03.380 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-15T21:49:03.382 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7YbURu0WXd3IgEjGFqUr4E0zITqbWshV0'], 'uuids': ['d148d2e9-595f-4ff6-a3e4-fa1542318f51'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.564913", "end": "2021-03-15 21:49:03.348060", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7YbURu0WXd3IgEjGFqUr4E0zITqbWshV0"], "labels": [], "masters": [], "uuids": ["d148d2e9-595f-4ff6-a3e4-fa1542318f51"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:01.783147", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:03.568 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MSHkDPY8dIOnISws6Y13nJcbDdEDm7ysv'], 'uuids': ['8232d5a6-ec38-4cbb-9612-947c851c10c9'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.467225", "end": "2021-03-15 21:49:03.541971", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MSHkDPY8dIOnISws6Y13nJcbDdEDm7ysv"], "labels": [], "masters": [], "uuids": ["8232d5a6-ec38-4cbb-9612-947c851c10c9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:02.074746", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:04.660 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-GBi2UW-a6Sw-1URz-1mGP-d5MK-zfnG-uN6uyw', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400U4400BGN', 'nvme-nvme.8086-50484654363230343030553434303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.094060", "end": "2021-03-15 21:49:04.634598", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-GBi2UW-a6Sw-1URz-1mGP-d5MK-zfnG-uN6uyw", "nvme-INTEL_SSDPEDMD400G4_PHFT620400U4400BGN", "nvme-nvme.8086-50484654363230343030553434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:03.540538", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-03-15T21:49:04.662 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:04.850 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-VO8POz-ZH26-BmEJ-VEk7-tSCg-T63c-d2P6FG', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040123400BGN', 'nvme-nvme.8086-50484654363230343031323334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.086554", "end": "2021-03-15 21:49:04.822911", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-VO8POz-ZH26-BmEJ-VEk7-tSCg-T63c-d2P6FG", "nvme-INTEL_SSDPEDMD400G4_PHFT62040123400BGN", "nvme-nvme.8086-50484654363230343031323334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:03.736357", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-03-15T21:49:04.852 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:05.909 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7cw11izvjiC8JLc4UdkPRe58JtWIhrNvF'], 'uuids': ['c4287383-70b5-4076-9480-12f5fd6be4e8'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.054976", "end": "2021-03-15 21:49:05.878045", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7cw11izvjiC8JLc4UdkPRe58JtWIhrNvF"], "labels": [], "masters": [], "uuids": ["c4287383-70b5-4076-9480-12f5fd6be4e8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:04.823069", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:06.119 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6M3ceZNXxeH71Vb65Ox1oBymQrrbAX1ZM5'], 'uuids': ['596daeed-f595-423f-aae9-cbb559271e1b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.080252", "end": "2021-03-15 21:49:06.093173", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6M3ceZNXxeH71Vb65Ox1oBymQrrbAX1ZM5"], "labels": [], "masters": [], "uuids": ["596daeed-f595-423f-aae9-cbb559271e1b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:05.012921", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:07.172 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7rdx3I57nchcOhVLPbnxdS9vzGEzr4NaN'], 'uuids': ['36db6da0-9931-4875-b0fa-dcaff7fbb09a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.071545", "end": "2021-03-15 21:49:07.138685", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7rdx3I57nchcOhVLPbnxdS9vzGEzr4NaN"], "labels": [], "masters": [], "uuids": ["36db6da0-9931-4875-b0fa-dcaff7fbb09a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:06.067140", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:07.360 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MxKqPzMAd1nD7AV6hBxnHUqrqx6yWNrHc'], 'uuids': ['7278bde9-90f6-409a-a496-944626d3fa72'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.053643", "end": "2021-03-15 21:49:07.334243", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MxKqPzMAd1nD7AV6hBxnHUqrqx6yWNrHc"], "labels": [], "masters": [], "uuids": ["7278bde9-90f6-409a-a496-944626d3fa72"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:06.280600", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:08.430 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA79pOX81rx2fOSXSYQkHvDusCUAX3tCLCG'], 'uuids': ['b61cf69c-5502-4750-8ee8-647b14b512c5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.054747", "end": "2021-03-15 21:49:08.394984", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA79pOX81rx2fOSXSYQkHvDusCUAX3tCLCG"], "labels": [], "masters": [], "uuids": ["b61cf69c-5502-4750-8ee8-647b14b512c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:07.340237", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:08.443 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QC9K', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QC9K', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QC9K', 'scsi-35000c50091e06051', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QC9K', 'wwn-0x5000c50091e06051'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QC9K', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e06051', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QC9K-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QC9K-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QC9K-part1', 'scsi-35000c50091e06051-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QC9K-part1', 'wwn-0x5000c50091e06051-part1'], 'uuids': ['a313f13c-d1e6-43a8-a40a-989733f55ee1'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QC9K", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QC9K", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QC9K", "scsi-35000c50091e06051", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QC9K", "wwn-0x5000c50091e06051"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QC9K-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QC9K-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-03-15T21:49:08.445 INFO:teuthology.task.ansible.out:Z1W5QC9K-part1", "scsi-35000c50091e06051-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QC9K-part1", "wwn-0x5000c50091e06051-part1"], "labels": [], "masters": [], "uuids": ["a313f13c-d1e6-43a8-a40a-989733f55ee1"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QC9K", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e06051"}}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:08.638 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MrlYXA3kIYXlCR2B6fddB1SLtu46RjWU9'], 'uuids': ['e25cfacf-85ab-419d-b975-456225105652'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.084709", "end": "2021-03-15 21:49:08.610222", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MrlYXA3kIYXlCR2B6fddB1SLtu46RjWU9"], "labels": [], "masters": [], "uuids": ["e25cfacf-85ab-419d-b975-456225105652"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:07.525513", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:08.651 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHY2', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QHY2', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QHY2', 'scsi-35000c50091e1abc3', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QHY2', 'wwn-0x5000c50091e1abc3'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QHY2', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e1abc3', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHY2-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QHY2-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QHY2-part1', 'scsi-35000c50091e1abc3-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QHY2-part1', 'wwn-0x5000c50091e1abc3-part1'], 'uuids': ['a313f13c-d1e6-43a8-a40a-989733f55ee1'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHY2", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QHY2", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QHY2", "scsi-35000c50091e1abc3", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QHY2", "wwn-0x5000c50091e1abc3"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHY2-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QHY2-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-03-15T21:49:08.653 INFO:teuthology.task.ansible.out:Z1W5QHY2-part1", "scsi-35000c50091e1abc3-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QHY2-part1", "wwn-0x5000c50091e1abc3-part1"], "labels": [], "masters": [], "uuids": ["a313f13c-d1e6-43a8-a40a-989733f55ee1"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QHY2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e1abc3"}}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:09.689 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7cT3xoFkHWjyTwchnms8ekLXxSZdOBSx6'], 'uuids': ['0107d9d3-42c3-412a-acdd-f669e077ad51'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.057789", "end": "2021-03-15 21:49:09.661948", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7cT3xoFkHWjyTwchnms8ekLXxSZdOBSx6"], "labels": [], "masters": [], "uuids": ["0107d9d3-42c3-412a-acdd-f669e077ad51"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:08.604159", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:09.718 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:09 +0000 (0:00:07.895) 0:04:01.430 ********** 2021-03-15T21:49:09.891 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MNjWwEFTcEGepO6nOwzyXYVAybde5M8Y0'], 'uuids': ['9808402f-28a9-4a0b-a786-489b7de4265f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.047078", "end": "2021-03-15 21:49:09.860054", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MNjWwEFTcEGepO6nOwzyXYVAybde5M8Y0"], "labels": [], "masters": [], "uuids": ["9808402f-28a9-4a0b-a786-489b7de4265f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:08.812976", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-15T21:49:09.921 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:09 +0000 (0:00:00.203) 0:04:01.634 ********** 2021-03-15T21:49:10.337 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-15T21:49:10.339 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MSHkDPY8dIOnISws6Y13nJcbDdEDm7ysv'], 'uuids': ['8232d5a6-ec38-4cbb-9612-947c851c10c9'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.143408", "end": "2021-03-15 21:49:10.306159", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MSHkDPY8dIOnISws6Y13nJcbDdEDm7ysv"], "labels": [], "masters": [], "uuids": ["8232d5a6-ec38-4cbb-9612-947c851c10c9"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:10.162751", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140736 s, 820 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140736 s, 820 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:10.549 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-VO8POz-ZH26-BmEJ-VEk7-tSCg-T63c-d2P6FG', 'nvme-INTEL_SSDPEDMD400G4_PHFT62040123400BGN', 'nvme-nvme.8086-50484654363230343031323334303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037301", "end": "2021-03-15 21:49:10.525834", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-VO8POz-ZH26-BmEJ-VEk7-tSCg-T63c-d2P6FG", "nvme-INTEL_SSDPEDMD400G4_PHFT62040123400BGN", "nvme-nvme.8086-50484654363230343031323334303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:10.488533", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0347361 s, 3.3 GB/s", "std 2021-03-15T21:49:10.551 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0347361 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:10.895 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6M3ceZNXxeH71Vb65Ox1oBymQrrbAX1ZM5'], 'uuids': ['596daeed-f595-423f-aae9-cbb559271e1b'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.146810", "end": "2021-03-15 21:49:10.864150", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6M3ceZNXxeH71Vb65Ox1oBymQrrbAX1ZM5"], "labels": [], "masters": [], "uuids": ["596daeed-f595-423f-aae9-cbb559271e1b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:10.717340", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.144267 s, 800 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.144267 s, 800 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:11.134 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7YbURu0WXd3IgEjGFqUr4E0zITqbWshV0'], 'uuids': ['d148d2e9-595f-4ff6-a3e4-fa1542318f51'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:01.142994", "end": "2021-03-15 21:49:11.108606", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7YbURu0WXd3IgEjGFqUr4E0zITqbWshV0"], "labels": [], "masters": [], "uuids": ["d148d2e9-595f-4ff6-a3e4-fa1542318f51"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:09.965612", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139153 s, 829 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139153 s, 829 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:11.219 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MxKqPzMAd1nD7AV6hBxnHUqrqx6yWNrHc'], 'uuids': ['7278bde9-90f6-409a-a496-944626d3fa72'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.145523", "end": "2021-03-15 21:49:11.189013", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MxKqPzMAd1nD7AV6hBxnHUqrqx6yWNrHc"], "labels": [], "masters": [], "uuids": ["7278bde9-90f6-409a-a496-944626d3fa72"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:11.043490", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.142928 s, 807 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.142928 s, 807 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:11.366 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-GBi2UW-a6Sw-1URz-1mGP-d5MK-zfnG-uN6uyw', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400U4400BGN', 'nvme-nvme.8086-50484654363230343030553434303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037732", "end": "2021-03-15 21:49:11.336693", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-GBi2UW-a6Sw-1URz-1mGP-d5MK-zfnG-uN6uyw", "nvme-INTEL_SSDPEDMD400G4_PHFT620400U4400BGN", "nvme-nvme.8086-50484654363230343030553434303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:11.298961", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0350778 s, 3.3 GB/s", "std 2021-03-15T21:49:11.368 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0350778 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:11.544 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MrlYXA3kIYXlCR2B6fddB1SLtu46RjWU9'], 'uuids': ['e25cfacf-85ab-419d-b975-456225105652'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.146256", "end": "2021-03-15 21:49:11.518009", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MrlYXA3kIYXlCR2B6fddB1SLtu46RjWU9"], "labels": [], "masters": [], "uuids": ["e25cfacf-85ab-419d-b975-456225105652"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:11.371753", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143752 s, 802 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143752 s, 802 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:11.556 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHY2', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QHY2', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QHY2', 'scsi-35000c50091e1abc3', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QHY2', 'wwn-0x5000c50091e1abc3'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QHY2', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e1abc3', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QHY2-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QHY2-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QHY2-part1', 'scsi-35000c50091e1abc3-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QHY2-part1', 'wwn-0x5000c50091e1abc3-part1'], 'uuids': ['a313f13c-d1e6-43a8-a40a-989733f55ee1'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHY2", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QHY2", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QHY2", "scsi-35000c50091e1abc3", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QHY2", "wwn-0x5000c50091e1abc3"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QHY2-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QHY2-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-03-15T21:49:11.558 INFO:teuthology.task.ansible.out:Z1W5QHY2-part1", "scsi-35000c50091e1abc3-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QHY2-part1", "wwn-0x5000c50091e1abc3-part1"], "labels": [], "masters": [], "uuids": ["a313f13c-d1e6-43a8-a40a-989733f55ee1"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QHY2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e1abc3"}}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:11.694 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7cw11izvjiC8JLc4UdkPRe58JtWIhrNvF'], 'uuids': ['c4287383-70b5-4076-9480-12f5fd6be4e8'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.144515", "end": "2021-03-15 21:49:11.668084", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7cw11izvjiC8JLc4UdkPRe58JtWIhrNvF"], "labels": [], "masters": [], "uuids": ["c4287383-70b5-4076-9480-12f5fd6be4e8"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:11.523569", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14193 s, 813 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14193 s, 813 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:11.873 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MNjWwEFTcEGepO6nOwzyXYVAybde5M8Y0'], 'uuids': ['9808402f-28a9-4a0b-a786-489b7de4265f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.145848", "end": "2021-03-15 21:49:11.847635", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-1cf4RGGA1C7TZ40JxeNo3MsRxdWcoX6MNjWwEFTcEGepO6nOwzyXYVAybde5M8Y0"], "labels": [], "masters": [], "uuids": ["9808402f-28a9-4a0b-a786-489b7de4265f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:11.701787", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.143209 s, 805 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.143209 s, 805 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:11.905 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:11 +0000 (0:00:01.983) 0:04:03.617 ********** 2021-03-15T21:49:12.026 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7rdx3I57nchcOhVLPbnxdS9vzGEzr4NaN'], 'uuids': ['36db6da0-9931-4875-b0fa-dcaff7fbb09a'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.142759", "end": "2021-03-15 21:49:11.991314", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7rdx3I57nchcOhVLPbnxdS9vzGEzr4NaN"], "labels": [], "masters": [], "uuids": ["36db6da0-9931-4875-b0fa-dcaff7fbb09a"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:11.848555", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14015 s, 823 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14015 s, 823 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:12.226 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-15T21:49:12.228 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.061598", "end": "2021-03-15 21:49:12.148833", "rc": 0, "start": "2021-03-15 21:49:12.087235", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:12.249 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:12 +0000 (0:00:00.344) 0:04:03.962 ********** 2021-03-15T21:49:12.340 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-15T21:49:12.342 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA79pOX81rx2fOSXSYQkHvDusCUAX3tCLCG'], 'uuids': ['b61cf69c-5502-4750-8ee8-647b14b512c5'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.142460", "end": "2021-03-15 21:49:12.317277", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA79pOX81rx2fOSXSYQkHvDusCUAX3tCLCG"], "labels": [], "masters": [], "uuids": ["b61cf69c-5502-4750-8ee8-647b14b512c5"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:12.174817", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.14002 s, 824 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.14002 s, 824 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:12.355 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QC9K', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QC9K', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QC9K', 'scsi-35000c50091e06051', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QC9K', 'wwn-0x5000c50091e06051'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5QC9K', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e06051', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5QC9K-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5QC9K-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QC9K-part1', 'scsi-35000c50091e06051-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5QC9K-part1', 'wwn-0x5000c50091e06051-part1'], 'uuids': ['a313f13c-d1e6-43a8-a40a-989733f55ee1'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QC9K", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QC9K", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5QC9K", "scsi-35000c50091e06051", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QC9K", "wwn-0x5000c50091e06051"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5QC9K-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5QC9K-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-03-15T21:49:12.357 INFO:teuthology.task.ansible.out:Z1W5QC9K-part1", "scsi-35000c50091e06051-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5QC9K-part1", "wwn-0x5000c50091e06051-part1"], "labels": [], "masters": [], "uuids": ["a313f13c-d1e6-43a8-a40a-989733f55ee1"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5QC9K", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e06051"}}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:12.482 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-15T21:49:12.484 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.046543", "end": "2021-03-15 21:49:12.462592", "rc": 0, "start": "2021-03-15 21:49:12.416049", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:12.505 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:12 +0000 (0:00:00.255) 0:04:04.217 ********** 2021-03-15T21:49:12.575 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-15T21:49:12.636 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:12 +0000 (0:00:00.130) 0:04:04.348 ********** 2021-03-15T21:49:12.664 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-15T21:49:12.668 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:12.692 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:12 +0000 (0:00:00.056) 0:04:04.404 ********** 2021-03-15T21:49:12.698 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-15T21:49:12.699 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7cT3xoFkHWjyTwchnms8ekLXxSZdOBSx6'], 'uuids': ['0107d9d3-42c3-412a-acdd-f669e077ad51'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.141458", "end": "2021-03-15 21:49:12.653244", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TyPgR0eBJGJEVGrPwlWf3qun0MMQscA7cT3xoFkHWjyTwchnms8ekLXxSZdOBSx6"], "labels": [], "masters": [], "uuids": ["0107d9d3-42c3-412a-acdd-f669e077ad51"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-15 21:49:12.511786", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138827 s, 831 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138827 s, 831 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:12.731 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:12 +0000 (0:00:00.039) 0:04:04.444 ********** 2021-03-15T21:49:12.738 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-15T21:49:12.740 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:12.762 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:12 +0000 (0:00:00.031) 0:04:04.475 ********** 2021-03-15T21:49:12.789 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-15T21:49:12.791 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:12.816 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:12 +0000 (0:00:00.053) 0:04:04.528 ********** 2021-03-15T21:49:12.843 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-15T21:49:12.845 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:12.866 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:12 +0000 (0:00:00.050) 0:04:04.579 ********** 2021-03-15T21:49:12.990 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-15T21:49:12.991 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.064546", "end": "2021-03-15 21:49:12.960379", "rc": 0, "start": "2021-03-15 21:49:12.895833", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:13.016 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.149) 0:04:04.728 ********** 2021-03-15T21:49:13.067 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-15T21:49:13.069 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-15T21:49:13.097 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.081) 0:04:04.809 ********** 2021-03-15T21:49:13.136 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-15T21:49:13.139 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356031488, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137703, 'block_used': 12957405, 'inode_total': 61054976, 'inode_available': 60986356, 'inode_used': 68620, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137703, "block_size": 4096, "block_total": 240095108, "block_used": 12957405, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986356, "inode_total": 61054976, "inode_used": 68620, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356031488, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.161 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.064) 0:04:04.874 ********** 2021-03-15T21:49:13.188 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-15T21:49:13.190 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.210 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.048) 0:04:04.922 ********** 2021-03-15T21:49:13.250 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:49:13.251 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.253 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.047016", "end": "2021-03-15 21:49:13.229156", "rc": 0, "start": "2021-03-15 21:49:13.182140", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:13.277 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.066) 0:04:04.988 ********** 2021-03-15T21:49:13.297 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.021) 0:04:05.010 ********** 2021-03-15T21:49:13.304 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-15T21:49:13.332 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.034) 0:04:05.045 ********** 2021-03-15T21:49:13.338 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:49:13.340 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.363 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.031) 0:04:05.076 ********** 2021-03-15T21:49:13.370 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-15T21:49:13.372 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.397 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.033) 0:04:05.109 ********** 2021-03-15T21:49:13.404 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-15T21:49:13.407 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.429 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.032) 0:04:05.142 ********** 2021-03-15T21:49:13.435 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-15T21:49:13.436 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.459 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.029) 0:04:05.171 ********** 2021-03-15T21:49:13.465 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-15T21:49:13.467 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.489 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.030) 0:04:05.202 ********** 2021-03-15T21:49:13.496 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-15T21:49:13.498 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.523 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.033) 0:04:05.236 ********** 2021-03-15T21:49:13.529 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-15T21:49:13.531 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.558 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.034) 0:04:05.270 ********** 2021-03-15T21:49:13.564 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-15T21:49:13.566 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.589 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.031) 0:04:05.302 ********** 2021-03-15T21:49:13.788 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-15T21:49:13.790 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-15T21:49:13.811 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.221) 0:04:05.523 ********** 2021-03-15T21:49:13.848 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-15T21:49:13.850 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356027392, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137702, 'block_used': 12957406, 'inode_total': 61054976, 'inode_available': 60986356, 'inode_used': 68620, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137702, "block_size": 4096, "block_total": 240095108, "block_used": 12957406, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986356, "inode_total": 61054976, "inode_used": 68620, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356027392, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.873 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.061) 0:04:05.585 ********** 2021-03-15T21:49:13.901 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-15T21:49:13.902 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.925 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.052) 0:04:05.637 ********** 2021-03-15T21:49:13.963 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:49:13.963 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:13.989 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:13 +0000 (0:00:00.064) 0:04:05.702 ********** 2021-03-15T21:49:14.018 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:49:14.019 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:14.040 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:14 +0000 (0:00:00.050) 0:04:05.752 ********** 2021-03-15T21:49:14.067 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-15T21:49:14.069 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:14.090 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:14 +0000 (0:00:00.050) 0:04:05.803 ********** 2021-03-15T21:49:14.118 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-15T21:49:14.120 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:14.141 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:14 +0000 (0:00:00.050) 0:04:05.853 ********** 2021-03-15T21:49:14.168 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-15T21:49:14.170 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:14.190 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:14 +0000 (0:00:00.049) 0:04:05.903 ********** 2021-03-15T21:49:14.280 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-15T21:49:14.282 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-15T21:49:14.305 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:14 +0000 (0:00:00.114) 0:04:06.017 ********** 2021-03-15T21:49:14.649 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-15T21:49:14.671 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:14 +0000 (0:00:00.366) 0:04:06.384 ********** 2021-03-15T21:49:14.949 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-15T21:49:14.951 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-15T21:49:15.074 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-15T21:49:15.076 INFO:teuthology.task.ansible.out: 2021-03-15T21:49:15.277 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-15T21:49:15.404 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-15T21:49:15.613 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-15T21:49:15.749 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-15T21:49:15.938 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-15T21:49:16.065 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-15T21:49:16.273 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-15T21:49:16.298 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:16 +0000 (0:00:01.626) 0:04:08.011 ********** 2021-03-15T21:49:16.300 INFO:teuthology.task.ansible.out: 2021-03-15T21:49:16.413 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-15T21:49:16.474 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:16 +0000 (0:00:00.141) 0:04:08.152 ********** 2021-03-15T21:49:16.501 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-15T21:49:16.525 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:16 +0000 (0:00:00.085) 0:04:08.238 ********** 2021-03-15T21:49:16.652 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-15T21:49:16.676 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:16 +0000 (0:00:00.150) 0:04:08.389 ********** 2021-03-15T21:49:16.758 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-15T21:49:16.760 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-15T21:49:16.897 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-15T21:49:16.952 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-15T21:49:17.090 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-15T21:49:17.132 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-15T21:49:17.270 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-15T21:49:17.313 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-15T21:49:17.324 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:17.349 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:17 +0000 (0:00:00.673) 0:04:09.062 ********** 2021-03-15T21:49:17.376 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-15T21:49:17.378 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:17.398 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:17 +0000 (0:00:00.048) 0:04:09.111 ********** 2021-03-15T21:49:17.425 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-15T21:49:17.427 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:17.447 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:17 +0000 (0:00:00.048) 0:04:09.159 ********** 2021-03-15T21:49:17.463 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-15T21:49:17.465 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} skipping: [smithi076.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:17.491 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:17 +0000 (0:00:00.044) 0:04:09.204 ********** 2021-03-15T21:49:17.518 INFO:teuthology.task.ansible.out: 2021-03-15T21:49:17.520 INFO:teuthology.task.ansible.out:TASK [testnode : Create filesystems] ******************************************* skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:17.544 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:17 +0000 (0:00:00.052) 0:04:09.256 ********** 2021-03-15T21:49:17.582 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-15T21:49:17.584 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:17.608 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:17 +0000 (0:00:00.063) 0:04:09.320 ********** 2021-03-15T21:49:17.650 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-15T21:49:17.652 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-15T21:49:17.676 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:17 +0000 (0:00:00.068) 0:04:09.389 ********** 2021-03-15T21:49:17.715 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-15T21:49:17.717 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-15T21:49:17.740 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:17 +0000 (0:00:00.063) 0:04:09.453 ********** 2021-03-15T21:49:17.821 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-15T21:49:17.823 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-15T21:49:17.846 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:17 +0000 (0:00:00.105) 0:04:09.558 ********** 2021-03-15T21:49:17.883 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-15T21:49:17.885 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-15T21:49:17.905 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:17 +0000 (0:00:00.059) 0:04:09.618 ********** 2021-03-15T21:49:18.320 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-03-15T21:49:18.322 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true} changed: [smithi076.front.sepia.ceph.com] => {"changed": true} 2021-03-15T21:49:18.344 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:18 +0000 (0:00:00.438) 0:04:10.057 ********** 2021-03-15T21:49:18.363 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:18 +0000 (0:00:00.018) 0:04:10.075 ********** 2021-03-15T21:49:18.925 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-15T21:49:18.926 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-15T21:49:18.947 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:18 +0000 (0:00:00.584) 0:04:10.660 ********** 2021-03-15T21:49:18.967 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:18 +0000 (0:00:00.019) 0:04:10.679 ********** 2021-03-15T21:49:18.973 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-15T21:49:18.975 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:18.997 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:18 +0000 (0:00:00.031) 0:04:10.710 ********** 2021-03-15T21:49:19.005 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.028 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.030) 0:04:10.740 ********** 2021-03-15T21:49:19.036 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-03-15T21:49:19.038 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.060 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.031) 0:04:10.772 ********** 2021-03-15T21:49:19.067 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.090 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.030) 0:04:10.803 ********** 2021-03-15T21:49:19.097 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-15T21:49:19.100 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.126 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.036) 0:04:10.839 ********** 2021-03-15T21:49:19.134 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.157 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.030) 0:04:10.869 ********** 2021-03-15T21:49:19.163 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-15T21:49:19.164 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.192 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.035) 0:04:10.905 ********** 2021-03-15T21:49:19.200 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.224 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.032) 0:04:10.937 ********** 2021-03-15T21:49:19.231 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-03-15T21:49:19.233 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.257 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.032) 0:04:10.969 ********** 2021-03-15T21:49:19.264 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.288 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.031) 0:04:11.000 ********** 2021-03-15T21:49:19.295 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-03-15T21:49:19.297 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.319 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.030) 0:04:11.031 ********** 2021-03-15T21:49:19.325 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.348 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.029) 0:04:11.061 ********** 2021-03-15T21:49:19.354 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.378 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.029) 0:04:11.091 ********** 2021-03-15T21:49:19.385 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.408 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.029) 0:04:11.120 ********** 2021-03-15T21:49:19.414 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-15T21:49:19.415 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.438 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.030) 0:04:11.151 ********** 2021-03-15T21:49:19.446 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.469 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.031) 0:04:11.182 ********** 2021-03-15T21:49:19.476 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.499 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.029) 0:04:11.211 ********** 2021-03-15T21:49:19.505 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.527 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.028) 0:04:11.240 ********** 2021-03-15T21:49:19.534 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-15T21:49:19.536 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.559 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.031) 0:04:11.271 ********** 2021-03-15T21:49:19.568 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.592 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.033) 0:04:11.305 ********** 2021-03-15T21:49:19.599 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-03-15T21:49:19.601 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.623 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.030) 0:04:11.336 ********** 2021-03-15T21:49:19.630 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.654 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.030) 0:04:11.367 ********** 2021-03-15T21:49:19.660 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-03-15T21:49:19.662 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.685 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.030) 0:04:11.398 ********** 2021-03-15T21:49:19.692 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.714 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.028) 0:04:11.426 ********** 2021-03-15T21:49:19.720 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-15T21:49:19.722 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.744 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.030) 0:04:11.457 ********** 2021-03-15T21:49:19.752 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.774 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.029) 0:04:11.487 ********** 2021-03-15T21:49:19.781 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-15T21:49:19.783 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.806 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.031) 0:04:11.518 ********** 2021-03-15T21:49:19.822 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.843 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.037) 0:04:11.556 ********** 2021-03-15T21:49:19.850 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-15T21:49:19.852 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.873 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.029) 0:04:11.586 ********** 2021-03-15T21:49:19.881 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:19.904 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:19 +0000 (0:00:00.030) 0:04:11.617 ********** 2021-03-15T21:49:20.354 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-15T21:49:20.356 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-03-15T21:49:20.378 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:20 +0000 (0:00:00.474) 0:04:12.091 ********** 2021-03-15T21:49:20.380 INFO:teuthology.task.ansible.out: 2021-03-15T21:49:20.396 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-03-15T21:49:20.420 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:20 +0000 (0:00:00.041) 0:04:12.132 ********** 2021-03-15T21:49:20.692 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-15T21:49:20.693 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-03-15 21:44:30 UTC", "ActiveEnterTimestampMonotonic": "12421151", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpd.service systemd-tmpfiles-setup.service ntpdate.service tmp.mount basic.target -.mount systemd-journald.socket sysinit.target system.slice sntp.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-03-15 21:44:29 UTC", "AssertTimestampMonotonic": "10786157", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-03-15 21:44:29 UTC", "ConditionTimestampMonotonic": "10786124", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service ntpd.service shutdown.target", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0 2021-03-15T21:49:20.695 INFO:teuthology.task.ansible.out:", "DefaultDependencies": "yes", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "800", "ExecMainStartTimestamp": "Mon 2021-03-15 21:44:30 UTC", "ExecMainStartTimestampMonotonic": "12349270", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-03-15 21:44:29 UTC", "InactiveExitTimestampMonotonic": "10798672", "InvocationID": "48fa83dc607b4bae94d648a825261d91", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity 2021-03-15T21:49:20.696 INFO:teuthology.task.ansible.out:", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127270", "LimitNPROCSoft": "127270", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127270", "LimitSIGPENDINGSoft": "127270", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "800", "MemoryAccounting": "yes", "MemoryCurrent": "3825664", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOn 2021-03-15T21:49:20.698 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-03-15 21:44:30 UTC", "StateChangeTimestampMonotonic": "12421151", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203632", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-03-15 21:44:30 UTC", "WatchdogTimestampMonotonic": "12349289", "WatchdogUSec": "0"}} 2021-03-15T21:49:20.718 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:20 +0000 (0:00:00.297) 0:04:12.430 ********** 2021-03-15T21:49:20.738 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Mon 2021-03-15 21:44:26 UTC", "ActiveEnterTimestampMonotonic": "12852326", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount system.slice tmp.mount sntp.service ntpd.service basic.target ntpdate.service systemd-journald.socket sysinit.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Mon 2021-03-15 21:44:25 UTC", "AssertTimestampMonotonic": "11803191", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Mon 2021-03-15 21:44:25 UTC", "ConditionTimestampMonotonic": "11803165", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0 2021-03-15T21:49:20.740 INFO:teuthology.task.ansible.out:", "DefaultDependencies": "yes", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "880", "ExecMainStartTimestamp": "Mon 2021-03-15 21:44:26 UTC", "ExecMainStartTimestampMonotonic": "12790307", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Mon 2021-03-15 21:44:25 UTC", "InactiveExitTimestampMonotonic": "11805762", "InvocationID": "f13197dbbde6499ca41249c8db00b375", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity 2021-03-15T21:49:20.742 INFO:teuthology.task.ansible.out:", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127269", "LimitNPROCSoft": "127269", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127269", "LimitSIGPENDINGSoft": "127269", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "880", "MemoryAccounting": "yes", "MemoryCurrent": "3432448", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target -.mount system.slice", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOn 2021-03-15T21:49:20.743 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Mon 2021-03-15 21:44:26 UTC", "StateChangeTimestampMonotonic": "12852326", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203631", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Mon 2021-03-15 21:44:26 UTC", "WatchdogTimestampMonotonic": "12790318", "WatchdogUSec": "0"}} 2021-03-15T21:49:20.762 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:20 +0000 (0:00:00.044) 0:04:12.474 ********** 2021-03-15T21:49:20.770 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:49:20.772 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-15T21:49:20.796 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:20 +0000 (0:00:00.033) 0:04:12.508 ********** 2021-03-15T21:49:20.803 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-15T21:49:20.828 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:20 +0000 (0:00:00.032) 0:04:12.540 ********** 2021-03-15T21:49:20.835 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:49:20.837 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-03-15T21:49:20.859 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:20 +0000 (0:00:00.031) 0:04:12.572 ********** 2021-03-15T21:49:20.876 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-03-15T21:49:20.898 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:20 +0000 (0:00:00.038) 0:04:12.611 ********** 2021-03-15T21:49:20.905 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-15T21:49:20.907 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:20.929 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:20 +0000 (0:00:00.030) 0:04:12.641 ********** 2021-03-15T21:49:20.938 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:20.964 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:20 +0000 (0:00:00.034) 0:04:12.676 ********** 2021-03-15T21:49:20.973 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-15T21:49:20.974 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:20.999 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:20 +0000 (0:00:00.035) 0:04:12.711 ********** 2021-03-15T21:49:21.008 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:21.035 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:21 +0000 (0:00:00.036) 0:04:12.748 ********** 2021-03-15T21:49:21.043 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-15T21:49:21.044 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:21.071 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:21 +0000 (0:00:00.035) 0:04:12.783 ********** 2021-03-15T21:49:21.078 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:21.104 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:21 +0000 (0:00:00.033) 0:04:12.817 ********** 2021-03-15T21:49:21.297 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-15T21:49:21.298 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-03-15T21:49:21.329 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:21 +0000 (0:00:00.225) 0:04:13.042 ********** 2021-03-15T21:49:21.341 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-03-15T21:49:21.366 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:21 +0000 (0:00:00.036) 0:04:13.079 ********** 2021-03-15T21:49:21.850 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-15T21:49:21.852 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-03-15T21:49:21.876 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:21 +0000 (0:00:00.510) 0:04:13.589 ********** 2021-03-15T21:49:21.894 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-03-15T21:49:21.918 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:21 +0000 (0:00:00.041) 0:04:13.630 ********** 2021-03-15T21:49:23.230 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-15T21:49:23.232 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-15T21:49:23.252 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:01.334) 0:04:14.965 ********** 2021-03-15T21:49:23.260 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-15T21:49:23.284 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.032) 0:04:14.997 ********** 2021-03-15T21:49:23.290 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-15T21:49:23.292 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:23.317 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.032) 0:04:15.030 ********** 2021-03-15T21:49:23.325 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:23.351 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.033) 0:04:15.063 ********** 2021-03-15T21:49:23.357 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-15T21:49:23.359 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-15T21:49:23.381 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.030) 0:04:15.094 ********** 2021-03-15T21:49:23.398 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-15T21:49:23.421 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.040) 0:04:15.134 ********** 2021-03-15T21:49:23.576 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-15T21:49:23.578 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-15T21:49:23.601 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.179) 0:04:15.314 ********** 2021-03-15T21:49:23.619 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-15T21:49:23.641 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.040) 0:04:15.354 ********** 2021-03-15T21:49:23.648 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-15T21:49:23.650 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:23.672 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.031) 0:04:15.385 ********** 2021-03-15T21:49:23.681 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:23.717 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.044) 0:04:15.429 ********** 2021-03-15T21:49:23.723 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-15T21:49:23.725 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-15T21:49:23.748 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.031) 0:04:15.461 ********** 2021-03-15T21:49:23.755 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-15T21:49:23.793 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.044) 0:04:15.506 ********** 2021-03-15T21:49:23.804 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-15T21:49:23.805 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:23.828 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.034) 0:04:15.541 ********** 2021-03-15T21:49:23.834 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:23.871 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:23 +0000 (0:00:00.042) 0:04:15.584 ********** 2021-03-15T21:49:24.039 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-15T21:49:24.041 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-15T21:49:24.065 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.193) 0:04:15.778 ********** 2021-03-15T21:49:24.097 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-15T21:49:24.125 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.060) 0:04:15.838 ********** 2021-03-15T21:49:24.269 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-15T21:49:24.271 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-15T21:49:24.294 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.168) 0:04:16.006 ********** 2021-03-15T21:49:24.325 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-15T21:49:24.327 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.351 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.056) 0:04:16.063 ********** 2021-03-15T21:49:24.358 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-15T21:49:24.360 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-15T21:49:24.394 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.043) 0:04:16.107 ********** 2021-03-15T21:49:24.408 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-15T21:49:24.410 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.410 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.411 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.412 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.423 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.448 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.053) 0:04:16.160 ********** 2021-03-15T21:49:24.456 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-15T21:49:24.458 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.481 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.033) 0:04:16.194 ********** 2021-03-15T21:49:24.490 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-15T21:49:24.493 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.494 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.507 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.509 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.510 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.539 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.057) 0:04:16.251 ********** 2021-03-15T21:49:24.545 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-15T21:49:24.547 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.547 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.548 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.575 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.035) 0:04:16.287 ********** 2021-03-15T21:49:24.583 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:49:24.585 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.627 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.051) 0:04:16.339 ********** 2021-03-15T21:49:24.636 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-15T21:49:24.638 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.640 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi076.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.640 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.640 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.664 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.037) 0:04:16.376 ********** 2021-03-15T21:49:24.671 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-15T21:49:24.674 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.697 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.033) 0:04:16.410 ********** 2021-03-15T21:49:24.704 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-15T21:49:24.706 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.739 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.041) 0:04:16.452 ********** 2021-03-15T21:49:24.749 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-15T21:49:24.751 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.785 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.045) 0:04:16.497 ********** 2021-03-15T21:49:24.794 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-15T21:49:24.796 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.820 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.035) 0:04:16.533 ********** 2021-03-15T21:49:24.828 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-15T21:49:24.830 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.854 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.033) 0:04:16.566 ********** 2021-03-15T21:49:24.870 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-15T21:49:24.872 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.896 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.042) 0:04:16.608 ********** 2021-03-15T21:49:24.903 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-15T21:49:24.905 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.936 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.040) 0:04:16.648 ********** 2021-03-15T21:49:24.943 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-15T21:49:24.945 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:24.978 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:24 +0000 (0:00:00.042) 0:04:16.691 ********** 2021-03-15T21:49:24.986 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-15T21:49:24.988 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:25.015 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:25 +0000 (0:00:00.036) 0:04:16.727 ********** 2021-03-15T21:49:25.022 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-15T21:49:25.024 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:25.049 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:25 +0000 (0:00:00.034) 0:04:16.761 ********** 2021-03-15T21:49:25.056 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-15T21:49:25.057 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:25.083 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:25 +0000 (0:00:00.034) 0:04:16.795 ********** 2021-03-15T21:49:25.090 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-15T21:49:25.091 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:25.115 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:25 +0000 (0:00:00.032) 0:04:16.828 ********** 2021-03-15T21:49:25.122 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-15T21:49:25.124 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:25.152 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:25 +0000 (0:00:00.036) 0:04:16.864 ********** 2021-03-15T21:49:25.160 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-15T21:49:25.161 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:25.187 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:25 +0000 (0:00:00.035) 0:04:16.900 ********** 2021-03-15T21:49:25.215 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-15T21:49:25.217 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:25.238 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:25 +0000 (0:00:00.051) 0:04:16.951 ********** 2021-03-15T21:49:25.394 INFO:teuthology.task.ansible.out: 2021-03-15T21:49:25.396 INFO:teuthology.task.ansible.out:TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** ok: [smithi125.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004979", "end": "2021-03-15 21:49:25.352356", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-15 21:49:25.347377", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:25.461 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.004969", "end": "2021-03-15 21:49:25.428444", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-15 21:49:25.423475", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:25.567 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003016", "end": "2021-03-15 21:49:25.535890", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-15 21:49:25.532874", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:25.630 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.002954", "end": "2021-03-15 21:49:25.606054", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-15 21:49:25.603100", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:25.736 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002778", "end": "2021-03-15 21:49:25.712494", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-15 21:49:25.709716", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:25.811 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002865", "end": "2021-03-15 21:49:25.784225", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-15 21:49:25.781360", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:25.928 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003210", "end": "2021-03-15 21:49:25.887622", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-15 21:49:25.884412", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:25.956 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:25 +0000 (0:00:00.717) 0:04:17.669 ********** 2021-03-15T21:49:25.984 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-15T21:49:26.006 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:25 +0000 (0:00:00.049) 0:04:17.718 ********** 2021-03-15T21:49:26.012 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-15T21:49:26.016 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003231", "end": "2021-03-15 21:49:25.964907", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-15 21:49:25.961676", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:26.041 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:26 +0000 (0:00:00.034) 0:04:17.753 ********** 2021-03-15T21:49:26.068 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-15T21:49:26.090 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:26 +0000 (0:00:00.049) 0:04:17.803 ********** 2021-03-15T21:49:26.204 INFO:teuthology.task.ansible.out: 2021-03-15T21:49:26.206 INFO:teuthology.task.ansible.out:TASK [testnode : Touch /ceph-qa-ready] ***************************************** changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-03-15T21:49:26.294 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-03-15T21:49:26.362 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-03-15T21:49:26.382 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:26 +0000 (0:00:00.292) 0:04:18.095 ********** 2021-03-15T21:49:26.404 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:26 +0000 (0:00:00.021) 0:04:18.116 ********** 2021-03-15T21:49:27.017 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi076.front.sepia.ceph.com] 2021-03-15T21:49:27.051 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] 2021-03-15T21:49:27.076 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:27 +0000 (0:00:00.672) 0:04:18.789 ********** 2021-03-15T21:49:27.118 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:27 +0000 (0:00:00.041) 0:04:18.831 ********** 2021-03-15T21:49:27.125 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-15T21:49:27.130 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:27.150 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:27 +0000 (0:00:00.031) 0:04:18.862 ********** 2021-03-15T21:49:27.157 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:27.183 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:27 +0000 (0:00:00.033) 0:04:18.895 ********** 2021-03-15T21:49:27.190 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-15T21:49:27.192 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-03-15T21:49:27.218 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:27 +0000 (0:00:00.034) 0:04:18.930 ********** 2021-03-15T21:49:27.226 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-03-15T21:49:27.250 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:27 +0000 (0:00:00.032) 0:04:18.963 ********** 2021-03-15T21:49:27.258 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-03-15T21:49:27.260 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-03-15T21:49:27.286 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:27 +0000 (0:00:00.035) 0:04:18.998 ********** 2021-03-15T21:49:27.294 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-03-15T21:49:27.322 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:27 +0000 (0:00:00.035) 0:04:19.034 ********** 2021-03-15T21:49:33.725 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-15T21:49:33.727 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-docker-2.2.1-7.module_el8.3.0+699+d61d9c41.noarch", "Installed: runc-1.0.0-70.rc92.module_el8.3.0+699+d61d9c41.x86_64", "Installed: podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64", "Installed: podman-catatonit-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64", "Removed: podman-manpages-1.6.4-4.module_el8.1.0+298+41f9343a.noarch", "Removed: runc-1.0.0-64.rc9.module_el8.1.0+298+41f9343a.x86_64", "Removed: podman-1.6.4-4.module_el8.1.0+298+41f9343a.x86_64", "Removed: podman-docker-1.6.4-4.module_el8.1.0+298+41f9343a.noarch"]} 2021-03-15T21:49:33.749 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:33 +0000 (0:00:06.427) 0:04:25.462 ********** 2021-03-15T21:49:33.788 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-15T21:49:33.790 INFO:teuthology.task.ansible.out:ok: [smithi076.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-03-15T21:49:33.811 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:33 +0000 (0:00:00.061) 0:04:25.524 ********** 2021-03-15T21:49:33.838 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-15T21:49:33.840 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:33.861 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:33 +0000 (0:00:00.049) 0:04:25.573 ********** 2021-03-15T21:49:33.879 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-15T21:49:33.881 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: podman-docker-2.2.1-7.module_el8.3.0+699+d61d9c41.noarch", "Installed: runc-1.0.0-70.rc92.module_el8.3.0+699+d61d9c41.x86_64", "Installed: podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64", "Installed: podman-catatonit-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64", "Removed: podman-manpages-1.6.4-4.module_el8.1.0+298+41f9343a.noarch", "Removed: runc-1.0.0-64.rc9.module_el8.1.0+298+41f9343a.x86_64", "Removed: podman-1.6.4-4.module_el8.1.0+298+41f9343a.x86_64", "Removed: podman-docker-1.6.4-4.module_el8.1.0+298+41f9343a.noarch"]} 2021-03-15T21:49:33.904 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:33 +0000 (0:00:00.043) 0:04:25.617 ********** 2021-03-15T21:49:33.932 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-15T21:49:33.934 INFO:teuthology.task.ansible.out:ok: [smithi125.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-03-15T21:49:33.955 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:33 +0000 (0:00:00.050) 0:04:25.667 ********** 2021-03-15T21:49:33.982 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-15T21:49:33.984 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:34.005 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:33 +0000 (0:00:00.050) 0:04:25.718 ********** 2021-03-15T21:49:34.085 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-15T21:49:34.087 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-15T21:49:34.235 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-15T21:49:34.299 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-15T21:49:34.328 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:34 +0000 (0:00:00.322) 0:04:26.040 ********** 2021-03-15T21:49:34.430 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-03-15T21:49:34.456 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:34 +0000 (0:00:00.127) 0:04:26.168 ********** 2021-03-15T21:49:34.881 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-15T21:49:34.883 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615844974.406025-3100-273623177432694/source", "state": "file", "uid": 0} 2021-03-15T21:49:34.976 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615844974.523379-3128-97229165490020/source", "state": "file", "uid": 0} 2021-03-15T21:49:35.396 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615844974.911985-3100-44326596361815/source", "state": "file", "uid": 0} 2021-03-15T21:49:35.420 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:35 +0000 (0:00:00.964) 0:04:27.133 ********** 2021-03-15T21:49:35.509 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1615844975.015693-3128-94484214088651/source", "state": "file", "uid": 0} 2021-03-15T21:49:35.534 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:35 +0000 (0:00:00.113) 0:04:27.247 ********** 2021-03-15T21:49:38.744 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-15T21:49:38.746 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-a70r21b6-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-a70r21b6-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-03-15T21:49:38.747 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-03-15T21:49:38.768 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:38 +0000 (0:00:03.233) 0:04:30.481 ********** 2021-03-15T21:49:38.806 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-15T21:49:38.808 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:38.832 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:38 +0000 (0:00:00.064) 0:04:30.545 ********** 2021-03-15T21:49:38.861 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-15T21:49:38.863 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-v21tnucn-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-v21tnucn-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-03-15T21:49:38.864 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} TASK [container-host : Create /etc/containers/registries.conf if necessary] **** skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:38.885 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:38 +0000 (0:00:00.052) 0:04:30.598 ********** 2021-03-15T21:49:38.906 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:38 +0000 (0:00:00.020) 0:04:30.619 ********** 2021-03-15T21:49:38.935 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-15T21:49:38.936 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:38.958 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:38 +0000 (0:00:00.051) 0:04:30.670 ********** 2021-03-15T21:49:38.985 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-03-15T21:49:38.987 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:39.008 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:39 +0000 (0:00:00.050) 0:04:30.721 ********** 2021-03-15T21:49:39.244 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-15T21:49:39.246 INFO:teuthology.task.ansible.out:changed: [smithi076.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.164060", "end": "2021-03-15 21:49:39.221963", "rc": 0, "start": "2021-03-15 21:49:39.057903", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:39.270 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:39 +0000 (0:00:00.260) 0:04:30.981 ********** 2021-03-15T21:49:39.296 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-03-15T21:49:39.298 INFO:teuthology.task.ansible.out:skipping: [smithi076.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:39.385 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-15T21:49:39.387 INFO:teuthology.task.ansible.out:changed: [smithi125.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.184467", "end": "2021-03-15 21:49:39.366336", "rc": 0, "start": "2021-03-15 21:49:39.181869", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-15T21:49:39.406 INFO:teuthology.task.ansible.out:Monday 15 March 2021 21:49:39 +0000 (0:00:00.138) 0:04:31.119 ********** 2021-03-15T21:49:39.434 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-03-15T21:49:39.436 INFO:teuthology.task.ansible.out:skipping: [smithi125.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-15T21:49:39.500 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-03-15T21:49:39.502 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-03-15T21:49:39.503 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi076.front.sepia.ceph.com : ok=127 changed=45 unreachable=0 failed=0 skipped=184 rescued=0 ignored=3 smithi125.front.sepia.ceph.com : ok=127 changed=45 unreachable=0 failed=0 skipped=184 rescued=0 ignored=3 Monday 15 March 2021 21:49:39 +0000 (0:00:00.096) 0:04:31.216 ********** =============================================================================== 2021-03-15T21:49:39.504 INFO:teuthology.task.ansible.out:common : nrpe - Ensure SELinux policy is up to date -------------------- 37.94s Create all admin users with sudo access. ------------------------------- 33.03s users : Update authorized_keys using the keys repo --------------------- 29.56s common : nrpe - Remove SELinux policy package -------------------------- 16.65s common : nrpe - Load SELinux policy package ---------------------------- 16.36s testnode : Install packages -------------------------------------------- 14.43s Remove revoked users --------------------------------------------------- 12.01s testnode : Zap all non-root disks --------------------------------------- 7.90s 2021-03-15T21:49:39.504 INFO:teuthology.task.ansible.out:container-host : Install container packages ----------------------------- 6.43s common : Install Kerberos Packages (RedHat) ----------------------------- 3.69s container-host : Install registries-conf-ctl ---------------------------- 3.23s common : Make sure smartmontools is installed --------------------------- 3.07s testnode : Ensure ceph packages are not present. ------------------------ 2.38s testnode : Rebuild rpmdb ------------------------------------------------ 2.20s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.98s testnode : Ensure ceph dependency packages are not present. ------------- 1.94s testnode : Upgrade packages --------------------------------------------- 1.76s Gathering Facts 2021-03-15T21:49:39.505 INFO:teuthology.task.ansible.out: --------------------------------------------------------- 1.64s testnode : Create logical volume(s) ------------------------------------- 1.63s users : Update authorized_keys for each user with literal keys ---------- 1.46s 2021-03-15T21:49:39.867 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi076.front.sepia.ceph.com'), Remote(name='ubuntu@smithi125.front.sepia.ceph.com')] 2021-03-15T21:49:39.868 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T21:49:39.870 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T21:49:39.939 DEBUG:teuthology.orchestra.run.smithi076:> true 2021-03-15T21:49:40.029 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-15T21:49:40.030 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi125.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-15T21:49:40.099 DEBUG:teuthology.orchestra.run.smithi125:> true 2021-03-15T21:49:40.190 INFO:teuthology.run_tasks:Running task clock... 2021-03-15T21:49:40.203 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-03-15T21:49:40.204 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-15T21:49:40.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-15T21:49:40.238 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-03-15T21:49:40.253 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-03-15T21:49:40.290 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: ntpd: command not found 2021-03-15T21:49:40.301 INFO:teuthology.orchestra.run.smithi076.stdout:506 Cannot talk to daemon 2021-03-15T21:49:40.315 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-03-15T21:49:40.330 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-03-15T21:49:40.387 INFO:teuthology.orchestra.run.smithi076.stderr:bash: ntpq: command not found 2021-03-15T21:49:40.389 INFO:teuthology.orchestra.run.smithi076.stdout:210 Number of sources = 4 2021-03-15T21:49:40.390 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-03-15T21:49:40.390 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2021-03-15T21:49:40.390 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-15T21:49:40.390 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-15T21:49:40.391 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-15T21:49:40.391 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-15T21:49:40.391 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-15T21:49:40.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-15T21:49:40.426 INFO:teuthology.orchestra.run.smithi125.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-03-15T21:49:40.441 INFO:teuthology.orchestra.run.smithi125.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-03-15T21:49:40.474 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: ntpd: command not found 2021-03-15T21:49:40.485 INFO:teuthology.orchestra.run.smithi125.stdout:506 Cannot talk to daemon 2021-03-15T21:49:40.499 INFO:teuthology.orchestra.run.smithi125.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-03-15T21:49:40.514 INFO:teuthology.orchestra.run.smithi125.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-03-15T21:49:40.578 INFO:teuthology.orchestra.run.smithi125.stderr:bash: ntpq: command not found 2021-03-15T21:49:40.581 INFO:teuthology.orchestra.run.smithi125.stdout:210 Number of sources = 4 2021-03-15T21:49:40.581 INFO:teuthology.orchestra.run.smithi125.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-03-15T21:49:40.582 INFO:teuthology.orchestra.run.smithi125.stdout:=============================================================================== 2021-03-15T21:49:40.582 INFO:teuthology.orchestra.run.smithi125.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-15T21:49:40.582 INFO:teuthology.orchestra.run.smithi125.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-15T21:49:40.582 INFO:teuthology.orchestra.run.smithi125.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-15T21:49:40.582 INFO:teuthology.orchestra.run.smithi125.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-03-15T21:49:40.583 INFO:teuthology.run_tasks:Running task exec... 2021-03-15T21:49:40.596 INFO:teuthology.task.exec:Executing custom commands... 2021-03-15T21:49:40.596 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:49:40.597 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:49:41.067 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:08:33 ago on Mon 15 Mar 2021 09:41:08 PM UTC. 2021-03-15T21:49:41.535 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:49:41.538 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T21:49:41.538 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2021-03-15T21:49:41.538 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T21:49:41.538 INFO:teuthology.orchestra.run.smithi076.stdout:Disabling modules: 2021-03-15T21:49:41.539 INFO:teuthology.orchestra.run.smithi076.stdout: container-tools 2021-03-15T21:49:41.539 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:49:41.539 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:49:41.539 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T21:49:41.539 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:49:41.729 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:49:41.802 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:49:42.258 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:08:34 ago on Mon 15 Mar 2021 09:41:08 PM UTC. 2021-03-15T21:49:42.583 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:49:42.694 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:49:42.695 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:49:42.695 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:49:42.792 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:49:43.084 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2021-03-15T21:49:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:49:43.091 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2021-03-15T21:49:43.091 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:49:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:49:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:49:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:, 2021-03-15T21:49:43.092 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2021-03-15T21:49:43.092 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:49:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:49:43.093 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:49:43.120 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:49:43.175 INFO:teuthology.orchestra.run.smithi076.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:49:43.176 INFO:teuthology.orchestra.run.smithi076.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:49:43.509 INFO:teuthology.orchestra.run.smithi076.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1074 0 --:--:-- --:--:-- --:--:-- 1074 2021-03-15T21:49:43.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:49:43.946 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:49:43.949 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================================== 2021-03-15T21:49:43.949 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:49:43.950 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================================== 2021-03-15T21:49:43.950 INFO:teuthology.orchestra.run.smithi076.stdout:Removing: 2021-03-15T21:49:43.950 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 2.2.1-7.module_el8.3.0+699+d61d9c41 @CentOS-AppStream 52 M 2021-03-15T21:49:43.950 INFO:teuthology.orchestra.run.smithi076.stdout:Removing dependent packages: 2021-03-15T21:49:43.950 INFO:teuthology.orchestra.run.smithi076.stdout: podman-docker noarch 2.2.1-7.module_el8.3.0+699+d61d9c41 @CentOS-AppStream 5.5 k 2021-03-15T21:49:43.951 INFO:teuthology.orchestra.run.smithi076.stdout:Removing unused dependencies: 2021-03-15T21:49:43.951 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.6-1.module_el8.1.0+298+41f9343a @CentOS-AppStream 85 k 2021-03-15T21:49:43.951 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.124.0-1.module_el8.1.0+298+41f9343a @CentOS-AppStream 44 k 2021-03-15T21:49:43.951 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.8.3-4.module_el8.1.0+298+41f9343a @CentOS-AppStream 71 M 2021-03-15T21:49:43.952 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common x86_64 1:0.1.40-8.module_el8.1.0+298+41f9343a @CentOS-AppStream 42 k 2021-03-15T21:49:43.952 INFO:teuthology.orchestra.run.smithi076.stdout: criu x86_64 3.12-9.el8 @CentOS-AppStream 1.3 M 2021-03-15T21:49:43.952 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 0.7.2-5.module_el8.1.0+298+41f9343a @CentOS-AppStream 122 k 2021-03-15T21:49:43.952 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:49:43.953 INFO:teuthology.orchestra.run.smithi076.stdout: libnet x86_64 1.1.6-15.el8 @CentOS-AppStream 170 k 2021-03-15T21:49:43.953 INFO:teuthology.orchestra.run.smithi076.stdout: libvarlink x86_64 18-3.el8 @CentOS-Base 129 k 2021-03-15T21:49:43.953 INFO:teuthology.orchestra.run.smithi076.stdout: podman-catatonit x86_64 2.2.1-7.module_el8.3.0+699+d61d9c41 @CentOS-AppStream 753 k 2021-03-15T21:49:43.954 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-c x86_64 1.3.0-4.el8 @CentOS-AppStream 56 k 2021-03-15T21:49:43.954 INFO:teuthology.orchestra.run.smithi076.stdout: runc x86_64 1.0.0-70.rc92.module_el8.3.0+699+d61d9c41 @CentOS-AppStream 13 M 2021-03-15T21:49:43.954 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 0.4.2-3.git21fdece.module_el8.1.0+298+41f9343a @CentOS-AppStream 175 k 2021-03-15T21:49:43.954 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:49:43.954 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:49:43.955 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================================== 2021-03-15T21:49:43.955 INFO:teuthology.orchestra.run.smithi076.stdout:Remove 15 Packages 2021-03-15T21:49:43.955 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:49:43.955 INFO:teuthology.orchestra.run.smithi076.stdout:Freed space: 139 M 2021-03-15T21:49:43.956 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:49:43.958 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:49:43.958 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:49:44.035 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:49:44.036 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:49:44.225 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:49:44.241 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-docker-2.2.1-7.module_el8.3.0+699+d61d9c41.no 1/1 2021-03-15T21:49:44.351 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-docker-2.2.1-7.module_el8.3.0+699+d61d9c41. 1/15 2021-03-15T21:49:44.351 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64 2/15 2021-03-15T21:49:44.626 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64 2/15 2021-03-15T21:49:44.627 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containers-common-1:0.1.40-8.module_el8.1.0+298+41 3/15 2021-03-15T21:49:44.627 INFO:teuthology.orchestra.run.smithi076.stdout:warning: /etc/containers/registries.conf saved as /etc/containers/registries.conf.rpmsave 2021-03-15T21:49:44.627 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:49:44.815 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : container-selinux-2:2.124.0-1.module_el8.1.0+298+4 4/15 2021-03-15T21:50:01.073 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.124.0-1.module_el8.1.0+298+4 4/15 2021-03-15T21:50:01.197 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-catatonit-2.2.1-7.module_el8.3.0+699+d61d9c 5/15 2021-03-15T21:50:01.346 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-overlayfs-0.7.2-5.module_el8.1.0+298+41f9343a 6/15 2021-03-15T21:50:01.479 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : runc-1.0.0-70.rc92.module_el8.3.0+699+d61d9c41.x86 7/15 2021-03-15T21:50:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : criu-3.12-9.el8.x86_64 8/15 2021-03-15T21:50:01.634 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libnet-1.1.6-15.el8.x86_64 9/15 2021-03-15T21:50:01.878 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libnet-1.1.6-15.el8.x86_64 9/15 2021-03-15T21:50:02.004 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : protobuf-c-1.3.0-4.el8.x86_64 10/15 2021-03-15T21:50:02.005 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 11/15 2021-03-15T21:50:02.243 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 11/15 2021-03-15T21:50:02.381 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : slirp4netns-0.4.2-3.git21fdece.module_el8.1.0+298+ 12/15 2021-03-15T21:50:02.538 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : conmon-2:2.0.6-1.module_el8.1.0+298+41f9343a.x86_6 13/15 2021-03-15T21:50:02.716 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containernetworking-plugins-0.8.3-4.module_el8.1.0 14/15 2021-03-15T21:50:02.717 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libvarlink-18-3.el8.x86_64 15/15 2021-03-15T21:50:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libvarlink-18-3.el8.x86_64 15/15 2021-03-15T21:50:03.158 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.6-1.module_el8.1.0+298+41f9343a.x86_6 1/15 2021-03-15T21:50:03.158 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.124.0-1.module_el8.1.0+298+4 2/15 2021-03-15T21:50:03.158 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.8.3-4.module_el8.1.0 3/15 2021-03-15T21:50:03.158 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-1:0.1.40-8.module_el8.1.0+298+41 4/15 2021-03-15T21:50:03.159 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : criu-3.12-9.el8.x86_64 5/15 2021-03-15T21:50:03.159 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-0.7.2-5.module_el8.1.0+298+41f9343a 6/15 2021-03-15T21:50:03.159 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 7/15 2021-03-15T21:50:03.159 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libnet-1.1.6-15.el8.x86_64 8/15 2021-03-15T21:50:03.160 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libvarlink-18-3.el8.x86_64 9/15 2021-03-15T21:50:03.160 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64 10/15 2021-03-15T21:50:03.160 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-catatonit-2.2.1-7.module_el8.3.0+699+d61d9c 11/15 2021-03-15T21:50:03.160 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-docker-2.2.1-7.module_el8.3.0+699+d61d9c41. 12/15 2021-03-15T21:50:03.161 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : protobuf-c-1.3.0-4.el8.x86_64 13/15 2021-03-15T21:50:03.161 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.3.0+699+d61d9c41.x86 14/15 2021-03-15T21:50:03.619 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-0.4.2-3.git21fdece.module_el8.1.0+298+ 15/15 2021-03-15T21:50:03.619 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:03.620 INFO:teuthology.orchestra.run.smithi076.stdout:Removed: 2021-03-15T21:50:03.620 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.6-1.module_el8.1.0+298+41f9343a.x86_64 2021-03-15T21:50:03.621 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.124.0-1.module_el8.1.0+298+41f9343a.noarch 2021-03-15T21:50:03.621 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.8.3-4.module_el8.1.0+298+41f9343a.x86_64 2021-03-15T21:50:03.622 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-1:0.1.40-8.module_el8.1.0+298+41f9343a.x86_64 2021-03-15T21:50:03.622 INFO:teuthology.orchestra.run.smithi076.stdout: criu-3.12-9.el8.x86_64 2021-03-15T21:50:03.623 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-0.7.2-5.module_el8.1.0+298+41f9343a.x86_64 2021-03-15T21:50:03.623 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:50:03.623 INFO:teuthology.orchestra.run.smithi076.stdout: libnet-1.1.6-15.el8.x86_64 2021-03-15T21:50:03.624 INFO:teuthology.orchestra.run.smithi076.stdout: libvarlink-18-3.el8.x86_64 2021-03-15T21:50:03.624 INFO:teuthology.orchestra.run.smithi076.stdout: podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64 2021-03-15T21:50:03.625 INFO:teuthology.orchestra.run.smithi076.stdout: podman-catatonit-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64 2021-03-15T21:50:03.625 INFO:teuthology.orchestra.run.smithi076.stdout: podman-docker-2.2.1-7.module_el8.3.0+699+d61d9c41.noarch 2021-03-15T21:50:03.626 INFO:teuthology.orchestra.run.smithi076.stdout: protobuf-c-1.3.0-4.el8.x86_64 2021-03-15T21:50:03.627 INFO:teuthology.orchestra.run.smithi076.stdout: runc-1.0.0-70.rc92.module_el8.3.0+699+d61d9c41.x86_64 2021-03-15T21:50:03.627 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-0.4.2-3.git21fdece.module_el8.1.0+298+41f9343a.x86_64 2021-03-15T21:50:03.628 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:03.628 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:50:03.951 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T21:50:04.511 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 9.3 kB/s | 1.4 kB 00:00 2021-03-15T21:50:05.711 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 58 kB/s | 68 kB 00:01 2021-03-15T21:50:06.346 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:50:06.348 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:50:06.348 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:50:06.348 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:50:06.349 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:50:06.349 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T21:50:06.349 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:50:06.349 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T21:50:06.350 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T21:50:06.350 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T21:50:06.350 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T21:50:06.351 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T21:50:06.351 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T21:50:06.351 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T21:50:06.351 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T21:50:06.351 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T21:50:06.352 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T21:50:06.352 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T21:50:06.352 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T21:50:06.352 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T21:50:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T21:50:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T21:50:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T21:50:06.353 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:06.354 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:50:06.354 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:50:06.354 INFO:teuthology.orchestra.run.smithi076.stdout:Install 16 Packages 2021-03-15T21:50:06.354 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:06.355 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 68 M 2021-03-15T21:50:06.355 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2021-03-15T21:50:06.355 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:50:06.388 INFO:teuthology.orchestra.run.smithi076.stdout:(1/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 8.7 MB/s | 318 kB 00:00 2021-03-15T21:50:06.388 INFO:teuthology.orchestra.run.smithi076.stdout:(2/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T21:50:06.389 INFO:teuthology.orchestra.run.smithi076.stdout:(3/16): fuse3-3.2.1-12.el8.x86_64.rpm 34 MB/s | 50 kB 00:00 2021-03-15T21:50:06.391 INFO:teuthology.orchestra.run.smithi076.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 37 MB/s | 94 kB 00:00 2021-03-15T21:50:06.391 INFO:teuthology.orchestra.run.smithi076.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 535 kB/s | 21 kB 00:00 2021-03-15T21:50:06.429 INFO:teuthology.orchestra.run.smithi076.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.5 MB/s | 59 kB 00:00 2021-03-15T21:50:07.222 INFO:teuthology.orchestra.run.smithi076.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 60 kB/s | 50 kB 00:00 2021-03-15T21:50:07.668 INFO:teuthology.orchestra.run.smithi076.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 227 kB/s | 289 kB 00:01 2021-03-15T21:50:07.975 INFO:teuthology.orchestra.run.smithi076.stdout:(9/16): crun-0.18-3.el8.5.2.x86_64.rpm 608 kB/s | 186 kB 00:00 2021-03-15T21:50:08.061 INFO:teuthology.orchestra.run.smithi076.stdout:(10/16): containers-common-1-8.el8.13.2.noarch. 70 kB/s | 59 kB 00:00 2021-03-15T21:50:08.282 INFO:teuthology.orchestra.run.smithi076.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 234 kB/s | 72 kB 00:00 2021-03-15T21:50:08.568 INFO:teuthology.orchestra.run.smithi076.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 476 kB/s | 241 kB 00:00 2021-03-15T21:50:09.295 INFO:teuthology.orchestra.run.smithi076.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 3.4 MB/s | 2.5 MB 00:00 2021-03-15T21:50:09.626 INFO:teuthology.orchestra.run.smithi076.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T21:50:09.730 INFO:teuthology.orchestra.run.smithi076.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 126 kB/s | 55 kB 00:00 2021-03-15T21:50:10.132 INFO:teuthology.orchestra.run.smithi076.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 13 MB/s | 23 MB 00:01 2021-03-15T21:50:10.139 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:50:10.139 INFO:teuthology.orchestra.run.smithi076.stdout:Total 18 MB/s | 68 MB 00:03 2021-03-15T21:50:10.144 INFO:teuthology.orchestra.run.smithi076.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux-57cf30cbbab27eef/packages/container-selinux-2.145.0-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 62035f7b: NOKEY 2021-03-15T21:50:10.179 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 30 kB/s | 1.0 kB 00:00 2021-03-15T21:50:10.452 INFO:teuthology.orchestra.run.smithi076.stderr:Importing GPG key 0x62035F7B: 2021-03-15T21:50:10.453 INFO:teuthology.orchestra.run.smithi076.stderr: Userid : "rhcontainerbot_container-selinux (None) " 2021-03-15T21:50:10.453 INFO:teuthology.orchestra.run.smithi076.stderr: Fingerprint: F5D6 6271 4052 49DE A189 569F C907 EBD4 6203 5F7B 2021-03-15T21:50:10.453 INFO:teuthology.orchestra.run.smithi076.stderr: From : https://download.copr.fedorainfracloud.org/results/rhcontainerbot/container-selinux/pubkey.gpg 2021-03-15T21:50:10.776 INFO:teuthology.orchestra.run.smithi076.stdout:Key imported successfully 2021-03-15T21:50:10.778 INFO:teuthology.orchestra.run.smithi076.stderr:warning: /var/cache/dnf/devel_kubic_libcontainers_stable-37b272243bc11f7c/packages/catatonit-0.1.5-6.el8.3.2.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 75060aa4: NOKEY 2021-03-15T21:50:11.098 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 3.3 kB/s | 1.1 kB 00:00 2021-03-15T21:50:11.135 INFO:teuthology.orchestra.run.smithi076.stderr:Importing GPG key 0x75060AA4: 2021-03-15T21:50:11.135 INFO:teuthology.orchestra.run.smithi076.stderr: Userid : "devel:kubic OBS Project " 2021-03-15T21:50:11.136 INFO:teuthology.orchestra.run.smithi076.stderr: Fingerprint: 2472 D6D0 D2F6 6AF8 7ABA 8DA3 4D64 3903 7506 0AA4 2021-03-15T21:50:11.136 INFO:teuthology.orchestra.run.smithi076.stderr: From : https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/repodata/repomd.xml.key 2021-03-15T21:50:11.201 INFO:teuthology.orchestra.run.smithi076.stdout:Key imported successfully 2021-03-15T21:50:11.211 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:50:11.255 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:50:11.256 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:50:11.758 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:50:11.760 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:50:12.741 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:50:12.881 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T21:50:15.327 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T21:50:15.499 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T21:50:15.633 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T21:50:15.750 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T21:50:15.766 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:50:15.891 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:50:33.249 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:50:33.415 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:50:33.482 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:50:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:50:33.483 INFO:teuthology.orchestra.run.smithi076.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2021-03-15T21:50:33.483 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:33.746 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T21:50:33.864 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:50:33.921 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:50:34.072 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T21:50:34.232 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:50:34.382 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T21:50:34.470 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T21:50:34.471 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.PGQuky: line 5: 23558 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T21:50:34.774 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:36.602 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:50:36.748 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:50:37.313 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T21:50:37.585 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:50:37.586 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.MQBA7V: line 6: 24508 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T21:50:37.586 INFO:teuthology.orchestra.run.smithi076.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T21:50:37.586 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:37.587 INFO:teuthology.orchestra.run.smithi076.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T21:50:37.779 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T21:50:37.779 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T21:50:37.780 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T21:50:37.780 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T21:50:37.780 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:50:37.780 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:50:37.780 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T21:50:37.781 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T21:50:37.781 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T21:50:37.781 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T21:50:37.781 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T21:50:37.781 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T21:50:37.782 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:50:37.782 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:50:37.782 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:50:38.247 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T21:50:38.248 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:38.248 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:50:38.248 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:50:38.248 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:50:38.249 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:50:38.249 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:50:38.249 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:50:38.249 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:50:38.249 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:50:38.250 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:50:38.250 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:50:38.250 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:50:38.250 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:50:38.251 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:50:38.251 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:50:38.251 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:50:38.252 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:50:38.252 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:50:38.252 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:38.252 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:50:38.458 INFO:teuthology.task.exec:Running commands on role mon.c host ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:50:38.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:50:38.935 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 09:50:05 PM UTC. 2021-03-15T21:50:39.445 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:50:39.446 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:50:39.446 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:50:39.526 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:50:39.988 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:34 ago on Mon 15 Mar 2021 09:50:05 PM UTC. 2021-03-15T21:50:40.326 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:50:40.441 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:50:40.442 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:50:40.442 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:50:40.543 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:50:40.827 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2021-03-15T21:50:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:50:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2021-03-15T21:50:40.835 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:50:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:50:40.835 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:50:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:, 2021-03-15T21:50:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2021-03-15T21:50:40.836 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:50:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:50:40.836 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:50:40.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:50:40.904 INFO:teuthology.orchestra.run.smithi076.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:50:40.904 INFO:teuthology.orchestra.run.smithi076.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:50:41.234 INFO:teuthology.orchestra.run.smithi076.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1094 0 --:--:-- --:--:-- --:--:-- 1094 2021-03-15T21:50:41.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:50:41.719 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:50:41.722 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:50:41.723 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:50:41.724 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:50:41.724 INFO:teuthology.orchestra.run.smithi076.stdout:Removing: 2021-03-15T21:50:41.724 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T21:50:41.726 INFO:teuthology.orchestra.run.smithi076.stdout:Removing unused dependencies: 2021-03-15T21:50:41.727 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T21:50:41.727 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T21:50:41.727 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T21:50:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T21:50:41.728 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T21:50:41.729 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T21:50:41.729 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T21:50:41.729 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T21:50:41.730 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T21:50:41.730 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T21:50:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:50:41.731 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T21:50:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T21:50:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T21:50:41.732 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T21:50:41.733 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:41.733 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:50:41.734 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:50:41.734 INFO:teuthology.orchestra.run.smithi076.stdout:Remove 16 Packages 2021-03-15T21:50:41.735 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:41.736 INFO:teuthology.orchestra.run.smithi076.stdout:Freed space: 173 M 2021-03-15T21:50:41.736 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:50:41.737 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:50:41.738 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:50:41.784 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:50:41.785 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:50:42.035 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:50:42.065 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T21:50:42.066 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:50:42.352 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:50:42.476 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T21:50:42.605 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T21:50:42.748 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T21:50:42.864 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:50:42.981 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:50:43.130 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T21:50:43.131 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:50:43.131 INFO:teuthology.orchestra.run.smithi076.stdout:warning: /etc/fuse.conf saved as /etc/fuse.conf.rpmsave 2021-03-15T21:50:43.131 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:50:43.231 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:50:59.507 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:50:59.620 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T21:50:59.638 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:50:59.638 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:50:59.779 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:50:59.895 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:51:00.011 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:51:00.012 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:51:00.254 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:51:00.364 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T21:51:00.514 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:51:00.743 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:51:00.743 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T21:51:00.743 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T21:51:00.744 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T21:51:00.744 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T21:51:00.744 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T21:51:00.744 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:51:00.744 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T21:51:00.745 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:51:00.745 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T21:51:00.745 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T21:51:00.745 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T21:51:00.746 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T21:51:00.746 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T21:51:00.746 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:51:00.746 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T21:51:01.173 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T21:51:01.173 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:01.174 INFO:teuthology.orchestra.run.smithi076.stdout:Removed: 2021-03-15T21:51:01.174 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:51:01.175 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:51:01.175 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:51:01.176 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:51:01.176 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:51:01.176 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:51:01.177 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:51:01.177 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:51:01.178 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:51:01.178 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:51:01.179 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:51:01.179 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:51:01.180 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:51:01.180 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:51:01.181 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:51:01.181 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:51:01.182 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:01.182 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:51:01.478 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T21:51:01.932 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 89 kB/s | 3.3 kB 00:00 2021-03-15T21:51:02.259 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 5.1 kB/s | 1.7 kB 00:00 2021-03-15T21:51:02.806 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:51:02.808 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:51:02.808 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:51:02.809 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:51:02.809 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:51:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T21:51:02.810 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:51:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T21:51:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T21:51:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T21:51:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T21:51:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T21:51:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T21:51:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T21:51:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T21:51:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T21:51:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T21:51:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T21:51:02.813 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T21:51:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T21:51:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T21:51:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T21:51:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T21:51:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:02.815 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:51:02.815 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:51:02.815 INFO:teuthology.orchestra.run.smithi076.stdout:Install 16 Packages 2021-03-15T21:51:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:02.816 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 68 M 2021-03-15T21:51:02.816 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2021-03-15T21:51:02.817 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:51:02.847 INFO:teuthology.orchestra.run.smithi076.stdout:(1/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T21:51:02.848 INFO:teuthology.orchestra.run.smithi076.stdout:(2/16): fuse3-3.2.1-12.el8.x86_64.rpm 26 MB/s | 50 kB 00:00 2021-03-15T21:51:02.851 INFO:teuthology.orchestra.run.smithi076.stdout:(3/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 7.9 MB/s | 318 kB 00:00 2021-03-15T21:51:02.853 INFO:teuthology.orchestra.run.smithi076.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 28 MB/s | 94 kB 00:00 2021-03-15T21:51:02.854 INFO:teuthology.orchestra.run.smithi076.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 511 kB/s | 21 kB 00:00 2021-03-15T21:51:02.893 INFO:teuthology.orchestra.run.smithi076.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.4 MB/s | 59 kB 00:00 2021-03-15T21:51:03.675 INFO:teuthology.orchestra.run.smithi076.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 60 kB/s | 50 kB 00:00 2021-03-15T21:51:04.121 INFO:teuthology.orchestra.run.smithi076.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 228 kB/s | 289 kB 00:01 2021-03-15T21:51:04.424 INFO:teuthology.orchestra.run.smithi076.stdout:(9/16): crun-0.18-3.el8.5.2.x86_64.rpm 615 kB/s | 186 kB 00:00 2021-03-15T21:51:04.499 INFO:teuthology.orchestra.run.smithi076.stdout:(10/16): containers-common-1-8.el8.13.2.noarch. 71 kB/s | 59 kB 00:00 2021-03-15T21:51:04.726 INFO:teuthology.orchestra.run.smithi076.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 239 kB/s | 72 kB 00:00 2021-03-15T21:51:05.007 INFO:teuthology.orchestra.run.smithi076.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 475 kB/s | 241 kB 00:00 2021-03-15T21:51:05.738 INFO:teuthology.orchestra.run.smithi076.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 3.4 MB/s | 2.5 MB 00:00 2021-03-15T21:51:06.091 INFO:teuthology.orchestra.run.smithi076.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T21:51:06.195 INFO:teuthology.orchestra.run.smithi076.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 120 kB/s | 55 kB 00:00 2021-03-15T21:51:06.372 INFO:teuthology.orchestra.run.smithi076.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 14 MB/s | 23 MB 00:01 2021-03-15T21:51:06.378 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:51:06.378 INFO:teuthology.orchestra.run.smithi076.stdout:Total 19 MB/s | 68 MB 00:03 2021-03-15T21:51:06.394 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:51:06.437 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:51:06.437 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:51:06.915 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:51:06.917 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:51:07.571 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:51:07.757 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T21:51:10.180 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T21:51:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T21:51:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T21:51:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T21:51:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:51:10.768 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:51:28.246 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:51:28.382 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:51:28.447 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:51:28.563 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:51:28.714 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T21:51:28.840 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:51:29.000 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T21:51:29.156 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:51:29.274 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T21:51:29.350 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T21:51:29.350 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.Ovsg1W: line 5: 26153 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T21:51:29.619 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:31.429 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:51:31.549 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:51:32.113 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T21:51:32.369 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:51:32.370 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.dg7BcB: line 6: 27103 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T21:51:32.370 INFO:teuthology.orchestra.run.smithi076.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T21:51:32.370 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:32.371 INFO:teuthology.orchestra.run.smithi076.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T21:51:32.562 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T21:51:32.563 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T21:51:32.563 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T21:51:32.563 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T21:51:32.563 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:51:32.563 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:51:32.564 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T21:51:32.564 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T21:51:32.564 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T21:51:32.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T21:51:32.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T21:51:32.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T21:51:32.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:51:32.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:51:32.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:51:33.037 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T21:51:33.037 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:33.037 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:51:33.038 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:51:33.038 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:51:33.038 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:51:33.038 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:51:33.039 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:51:33.039 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:51:33.039 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:51:33.039 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:51:33.039 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:51:33.040 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:51:33.040 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:51:33.040 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:51:33.040 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:51:33.040 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:51:33.041 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:51:33.041 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:51:33.041 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:33.041 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:51:33.278 INFO:teuthology.task.exec:Running commands on role mgr.y host ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:51:33.279 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:51:33.759 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:31 ago on Mon 15 Mar 2021 09:51:02 PM UTC. 2021-03-15T21:51:34.252 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:51:34.253 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:51:34.253 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:51:34.332 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:51:34.795 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 09:51:02 PM UTC. 2021-03-15T21:51:35.119 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:51:35.232 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:51:35.233 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:51:35.233 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:51:35.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:51:35.603 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2021-03-15T21:51:35.610 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:51:35.610 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2021-03-15T21:51:35.610 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:51:35.611 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:51:35.611 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:51:35.611 INFO:teuthology.orchestra.run.smithi076.stderr:, 2021-03-15T21:51:35.611 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2021-03-15T21:51:35.612 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:51:35.612 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:51:35.612 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:51:35.640 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:51:35.676 INFO:teuthology.orchestra.run.smithi076.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:51:35.676 INFO:teuthology.orchestra.run.smithi076.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:51:36.017 INFO:teuthology.orchestra.run.smithi076.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1052 0 --:--:-- --:--:-- --:--:-- 1052 2021-03-15T21:51:36.020 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:51:36.493 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:51:36.496 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:51:36.496 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:51:36.497 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:51:36.497 INFO:teuthology.orchestra.run.smithi076.stdout:Removing: 2021-03-15T21:51:36.497 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T21:51:36.497 INFO:teuthology.orchestra.run.smithi076.stdout:Removing unused dependencies: 2021-03-15T21:51:36.497 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T21:51:36.498 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T21:51:36.498 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T21:51:36.498 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T21:51:36.498 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T21:51:36.499 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T21:51:36.499 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T21:51:36.499 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T21:51:36.499 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T21:51:36.499 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T21:51:36.500 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:51:36.500 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T21:51:36.500 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T21:51:36.500 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T21:51:36.500 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T21:51:36.501 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:36.501 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:51:36.501 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:51:36.501 INFO:teuthology.orchestra.run.smithi076.stdout:Remove 16 Packages 2021-03-15T21:51:36.502 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:36.502 INFO:teuthology.orchestra.run.smithi076.stdout:Freed space: 173 M 2021-03-15T21:51:36.502 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:51:36.506 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:51:36.507 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:51:36.558 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:51:36.559 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:51:36.775 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:51:36.804 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T21:51:36.805 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:51:37.104 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:51:37.235 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T21:51:37.365 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T21:51:37.490 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T21:51:37.606 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:51:37.749 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:51:37.890 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T21:51:37.990 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:51:37.991 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:51:54.249 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:51:54.388 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T21:51:54.405 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:51:54.406 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:51:54.547 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:51:54.679 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:51:54.804 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:51:54.805 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:51:55.013 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:51:55.123 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T21:51:55.548 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:51:55.757 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:51:55.758 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T21:51:55.758 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T21:51:55.758 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T21:51:55.759 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T21:51:55.759 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T21:51:55.759 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:51:55.759 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T21:51:55.759 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:51:55.760 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T21:51:55.760 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T21:51:55.760 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T21:51:55.760 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T21:51:55.761 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T21:51:55.761 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:51:55.761 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T21:51:56.161 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T21:51:56.161 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:56.161 INFO:teuthology.orchestra.run.smithi076.stdout:Removed: 2021-03-15T21:51:56.162 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:51:56.162 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:51:56.162 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:51:56.162 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:51:56.162 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:51:56.163 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:51:56.163 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:51:56.163 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:51:56.163 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:51:56.164 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:51:56.164 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:51:56.164 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:51:56.164 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:51:56.164 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:51:56.165 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:51:56.165 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:51:56.165 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:56.165 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:51:56.462 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T21:51:56.956 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 43 kB/s | 3.3 kB 00:00 2021-03-15T21:51:57.280 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 5.2 kB/s | 1.7 kB 00:00 2021-03-15T21:51:57.796 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:51:57.798 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:51:57.798 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:51:57.798 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:51:57.799 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:51:57.799 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T21:51:57.799 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:51:57.799 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T21:51:57.799 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T21:51:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T21:51:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T21:51:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T21:51:57.800 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T21:51:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T21:51:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T21:51:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T21:51:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T21:51:57.801 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T21:51:57.802 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T21:51:57.802 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T21:51:57.802 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T21:51:57.802 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T21:51:57.803 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T21:51:57.803 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:57.804 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:51:57.804 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:51:57.805 INFO:teuthology.orchestra.run.smithi076.stdout:Install 16 Packages 2021-03-15T21:51:57.805 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:51:57.806 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 68 M 2021-03-15T21:51:57.806 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2021-03-15T21:51:57.806 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:51:57.839 INFO:teuthology.orchestra.run.smithi076.stdout:(1/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T21:51:57.840 INFO:teuthology.orchestra.run.smithi076.stdout:(2/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 8.0 MB/s | 318 kB 00:00 2021-03-15T21:51:57.842 INFO:teuthology.orchestra.run.smithi076.stdout:(3/16): fuse3-3.2.1-12.el8.x86_64.rpm 18 MB/s | 50 kB 00:00 2021-03-15T21:51:57.844 INFO:teuthology.orchestra.run.smithi076.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 32 MB/s | 94 kB 00:00 2021-03-15T21:51:57.860 INFO:teuthology.orchestra.run.smithi076.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 364 kB/s | 21 kB 00:00 2021-03-15T21:51:57.884 INFO:teuthology.orchestra.run.smithi076.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.4 MB/s | 59 kB 00:00 2021-03-15T21:51:58.708 INFO:teuthology.orchestra.run.smithi076.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 59 kB/s | 50 kB 00:00 2021-03-15T21:51:59.129 INFO:teuthology.orchestra.run.smithi076.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 225 kB/s | 289 kB 00:01 2021-03-15T21:51:59.542 INFO:teuthology.orchestra.run.smithi076.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 70 kB/s | 59 kB 00:00 2021-03-15T21:51:59.553 INFO:teuthology.orchestra.run.smithi076.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 439 kB/s | 186 kB 00:00 2021-03-15T21:51:59.983 INFO:teuthology.orchestra.run.smithi076.stdout:(11/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 562 kB/s | 241 kB 00:00 2021-03-15T21:52:00.049 INFO:teuthology.orchestra.run.smithi076.stdout:(12/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 142 kB/s | 72 kB 00:00 2021-03-15T21:52:00.787 INFO:teuthology.orchestra.run.smithi076.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 3.3 MB/s | 2.5 MB 00:00 2021-03-15T21:52:01.251 INFO:teuthology.orchestra.run.smithi076.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 12 MB/s | 41 MB 00:03 2021-03-15T21:52:01.356 INFO:teuthology.orchestra.run.smithi076.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 96 kB/s | 55 kB 00:00 2021-03-15T21:52:01.892 INFO:teuthology.orchestra.run.smithi076.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 12 MB/s | 23 MB 00:01 2021-03-15T21:52:01.898 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:52:01.898 INFO:teuthology.orchestra.run.smithi076.stdout:Total 17 MB/s | 68 MB 00:04 2021-03-15T21:52:01.914 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:52:01.956 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:52:01.957 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:52:02.437 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:52:02.438 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:52:03.092 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:52:03.216 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T21:52:05.631 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T21:52:06.594 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T21:52:06.594 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T21:52:06.594 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T21:52:06.594 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:52:06.595 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:52:23.638 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:52:23.792 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:52:23.851 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:52:23.966 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:52:24.099 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T21:52:24.233 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:52:24.257 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:52:24.427 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T21:52:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:52:24.700 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T21:52:24.783 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T21:52:24.784 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.r36ek7: line 5: 28746 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T21:52:25.041 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:52:27.129 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:52:27.266 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:52:27.829 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T21:52:28.108 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:52:28.109 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.U1yYDl: line 6: 29696 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T21:52:28.109 INFO:teuthology.orchestra.run.smithi076.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T21:52:28.109 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:52:28.110 INFO:teuthology.orchestra.run.smithi076.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T21:52:28.317 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T21:52:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T21:52:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T21:52:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T21:52:28.318 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:52:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:52:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T21:52:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T21:52:28.319 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T21:52:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T21:52:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T21:52:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T21:52:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:52:28.320 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:52:28.321 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:52:28.832 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T21:52:28.832 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:52:28.833 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:52:28.833 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:52:28.833 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:52:28.833 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:52:28.834 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:52:28.834 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:52:28.834 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:52:28.834 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:52:28.834 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:52:28.835 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:52:28.835 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:52:28.835 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:52:28.835 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:52:28.836 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:52:28.836 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:52:28.836 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:52:28.836 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:52:28.836 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:52:28.837 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:52:29.044 INFO:teuthology.task.exec:Running commands on role osd.0 host ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:52:29.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:52:29.516 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 09:51:57 PM UTC. 2021-03-15T21:52:30.013 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:52:30.014 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:52:30.014 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:52:30.095 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:52:30.559 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 09:51:57 PM UTC. 2021-03-15T21:52:30.903 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:52:31.016 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:52:31.017 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:52:31.017 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:52:31.120 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:52:31.418 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2021-03-15T21:52:31.424 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:52:31.424 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2021-03-15T21:52:31.425 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:52:31.425 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:52:31.425 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:52:31.426 INFO:teuthology.orchestra.run.smithi076.stderr:, 2021-03-15T21:52:31.426 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2021-03-15T21:52:31.426 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:52:31.426 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:52:31.427 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:52:31.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:52:31.494 INFO:teuthology.orchestra.run.smithi076.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:52:31.495 INFO:teuthology.orchestra.run.smithi076.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:52:31.811 INFO:teuthology.orchestra.run.smithi076.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1132 0 --:--:-- --:--:-- --:--:-- 1128 2021-03-15T21:52:31.814 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:52:32.293 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:52:32.296 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:52:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:52:32.297 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:52:32.297 INFO:teuthology.orchestra.run.smithi076.stdout:Removing: 2021-03-15T21:52:32.297 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T21:52:32.298 INFO:teuthology.orchestra.run.smithi076.stdout:Removing unused dependencies: 2021-03-15T21:52:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T21:52:32.298 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T21:52:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T21:52:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T21:52:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T21:52:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T21:52:32.299 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T21:52:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T21:52:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T21:52:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T21:52:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:52:32.300 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T21:52:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T21:52:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T21:52:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T21:52:32.301 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:52:32.302 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:52:32.302 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:52:32.302 INFO:teuthology.orchestra.run.smithi076.stdout:Remove 16 Packages 2021-03-15T21:52:32.303 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:52:32.303 INFO:teuthology.orchestra.run.smithi076.stdout:Freed space: 173 M 2021-03-15T21:52:32.303 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:52:32.307 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:52:32.307 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:52:32.359 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:52:32.360 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:52:32.571 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:52:32.601 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T21:52:32.602 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:52:32.912 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:52:33.044 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T21:52:33.190 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T21:52:33.316 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T21:52:33.449 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:52:33.592 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:52:33.732 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T21:52:33.833 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:52:33.834 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:52:50.275 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:52:50.388 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T21:52:50.406 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:52:50.406 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:52:50.539 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:52:50.664 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:52:50.780 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:52:50.780 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:52:50.989 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:52:51.132 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T21:52:51.299 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:52:51.509 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:52:51.509 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T21:52:51.510 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T21:52:51.510 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T21:52:51.510 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T21:52:51.510 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T21:52:51.511 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:52:51.511 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T21:52:51.511 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:52:51.511 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T21:52:51.512 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T21:52:51.512 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T21:52:51.512 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T21:52:51.512 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T21:52:51.513 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:52:51.513 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T21:52:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T21:52:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:52:51.971 INFO:teuthology.orchestra.run.smithi076.stdout:Removed: 2021-03-15T21:52:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:52:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:52:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:52:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:52:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:52:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:52:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:52:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:52:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:52:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:52:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:52:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:52:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:52:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:52:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:52:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:52:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:52:51.975 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:52:52.268 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T21:52:52.725 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 81 kB/s | 3.3 kB 00:00 2021-03-15T21:52:53.056 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 5.1 kB/s | 1.7 kB 00:00 2021-03-15T21:52:53.572 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:52:53.574 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:52:53.574 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:52:53.575 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:52:53.575 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:52:53.575 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T21:52:53.575 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:52:53.576 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T21:52:53.576 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T21:52:53.576 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T21:52:53.576 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T21:52:53.576 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T21:52:53.577 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T21:52:53.577 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T21:52:53.577 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T21:52:53.577 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T21:52:53.577 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T21:52:53.578 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T21:52:53.578 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T21:52:53.578 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T21:52:53.578 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T21:52:53.579 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T21:52:53.579 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T21:52:53.579 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:52:53.579 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:52:53.579 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:52:53.580 INFO:teuthology.orchestra.run.smithi076.stdout:Install 16 Packages 2021-03-15T21:52:53.580 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:52:53.581 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 68 M 2021-03-15T21:52:53.581 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2021-03-15T21:52:53.581 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:52:53.613 INFO:teuthology.orchestra.run.smithi076.stdout:(1/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T21:52:53.615 INFO:teuthology.orchestra.run.smithi076.stdout:(2/16): fuse3-3.2.1-12.el8.x86_64.rpm 35 MB/s | 50 kB 00:00 2021-03-15T21:52:53.617 INFO:teuthology.orchestra.run.smithi076.stdout:(3/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 51 MB/s | 94 kB 00:00 2021-03-15T21:52:53.624 INFO:teuthology.orchestra.run.smithi076.stdout:(4/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 6.7 MB/s | 318 kB 00:00 2021-03-15T21:52:53.633 INFO:teuthology.orchestra.run.smithi076.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 385 kB/s | 21 kB 00:00 2021-03-15T21:52:53.658 INFO:teuthology.orchestra.run.smithi076.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.4 MB/s | 59 kB 00:00 2021-03-15T21:52:54.466 INFO:teuthology.orchestra.run.smithi076.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 60 kB/s | 50 kB 00:00 2021-03-15T21:52:54.933 INFO:teuthology.orchestra.run.smithi076.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 221 kB/s | 289 kB 00:01 2021-03-15T21:52:55.242 INFO:teuthology.orchestra.run.smithi076.stdout:(9/16): crun-0.18-3.el8.5.2.x86_64.rpm 604 kB/s | 186 kB 00:00 2021-03-15T21:52:55.294 INFO:teuthology.orchestra.run.smithi076.stdout:(10/16): containers-common-1-8.el8.13.2.noarch. 71 kB/s | 59 kB 00:00 2021-03-15T21:52:55.547 INFO:teuthology.orchestra.run.smithi076.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 236 kB/s | 72 kB 00:00 2021-03-15T21:52:57.617 INFO:teuthology.orchestra.run.smithi076.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 104 kB/s | 241 kB 00:02 2021-03-15T21:52:58.041 INFO:teuthology.orchestra.run.smithi076.stdout:(13/16): containernetworking-plugins-0.9.1-4.el 9.4 MB/s | 41 MB 00:04 2021-03-15T21:52:58.165 INFO:teuthology.orchestra.run.smithi076.stdout:(14/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 4.5 MB/s | 2.5 MB 00:00 2021-03-15T21:52:58.347 INFO:teuthology.orchestra.run.smithi076.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 179 kB/s | 55 kB 00:00 2021-03-15T21:52:59.008 INFO:teuthology.orchestra.run.smithi076.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 6.7 MB/s | 23 MB 00:03 2021-03-15T21:52:59.015 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:52:59.015 INFO:teuthology.orchestra.run.smithi076.stdout:Total 13 MB/s | 68 MB 00:05 2021-03-15T21:52:59.030 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:52:59.074 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:52:59.074 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:52:59.558 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:52:59.560 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:53:00.216 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:53:00.342 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T21:53:02.754 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T21:53:02.926 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T21:53:03.043 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T21:53:03.170 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T21:53:03.185 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:53:03.311 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:53:21.279 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:53:21.425 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:53:21.491 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:53:21.606 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:53:21.740 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T21:53:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:53:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:53:22.051 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T21:53:22.265 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:53:22.392 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T21:53:22.481 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T21:53:22.481 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.PuybyM: line 5: 31340 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T21:53:22.770 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:53:24.605 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:53:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:53:25.306 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T21:53:25.633 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:53:25.634 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.jaDOnh: line 6: 32290 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T21:53:25.634 INFO:teuthology.orchestra.run.smithi076.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T21:53:25.635 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:53:25.635 INFO:teuthology.orchestra.run.smithi076.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T21:53:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T21:53:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T21:53:25.834 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T21:53:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T21:53:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:53:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:53:25.835 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T21:53:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T21:53:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T21:53:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T21:53:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T21:53:25.836 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T21:53:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:53:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:53:25.837 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:53:26.417 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T21:53:26.417 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:53:26.417 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:53:26.418 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:53:26.418 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:53:26.418 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:53:26.418 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:53:26.418 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:53:26.419 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:53:26.419 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:53:26.419 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:53:26.419 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:53:26.420 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:53:26.420 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:53:26.420 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:53:26.420 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:53:26.420 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:53:26.421 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:53:26.421 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:53:26.421 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:53:26.421 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:53:26.615 INFO:teuthology.task.exec:Running commands on role osd.1 host ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:53:26.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:53:27.084 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:34 ago on Mon 15 Mar 2021 09:52:53 PM UTC. 2021-03-15T21:53:27.564 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:53:27.566 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:53:27.566 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:53:27.646 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:53:28.113 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:35 ago on Mon 15 Mar 2021 09:52:53 PM UTC. 2021-03-15T21:53:28.454 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:53:28.572 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:53:28.573 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:53:28.573 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:53:28.662 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:53:28.962 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2021-03-15T21:53:28.968 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:53:28.969 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2021-03-15T21:53:28.969 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:53:28.969 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:53:28.969 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:53:28.970 INFO:teuthology.orchestra.run.smithi076.stderr:, 2021-03-15T21:53:28.970 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2021-03-15T21:53:28.970 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:53:28.970 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:53:28.971 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:53:28.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:53:29.035 INFO:teuthology.orchestra.run.smithi076.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:53:29.035 INFO:teuthology.orchestra.run.smithi076.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:53:29.357 INFO:teuthology.orchestra.run.smithi076.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1111 0 --:--:-- --:--:-- --:--:-- 1111 2021-03-15T21:53:29.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:53:29.837 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:53:29.840 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:53:29.841 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:53:29.841 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:53:29.841 INFO:teuthology.orchestra.run.smithi076.stdout:Removing: 2021-03-15T21:53:29.841 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T21:53:29.842 INFO:teuthology.orchestra.run.smithi076.stdout:Removing unused dependencies: 2021-03-15T21:53:29.842 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T21:53:29.842 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T21:53:29.842 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T21:53:29.843 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T21:53:29.843 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T21:53:29.843 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T21:53:29.843 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T21:53:29.843 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T21:53:29.844 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T21:53:29.844 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T21:53:29.844 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:53:29.844 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T21:53:29.845 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T21:53:29.845 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T21:53:29.845 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T21:53:29.845 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:53:29.845 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:53:29.846 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:53:29.846 INFO:teuthology.orchestra.run.smithi076.stdout:Remove 16 Packages 2021-03-15T21:53:29.846 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:53:29.846 INFO:teuthology.orchestra.run.smithi076.stdout:Freed space: 173 M 2021-03-15T21:53:29.847 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:53:29.851 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:53:29.851 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:53:29.903 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:53:29.904 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:53:30.111 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:53:30.140 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T21:53:30.141 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:53:30.446 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:53:30.570 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T21:53:30.708 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T21:53:30.833 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T21:53:30.966 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:53:31.109 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:53:31.258 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T21:53:31.358 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:53:31.359 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:53:47.834 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:53:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T21:53:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:53:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:53:48.115 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:53:48.231 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:53:48.347 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:53:48.348 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:53:48.589 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:53:48.699 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T21:53:48.859 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:53:49.081 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:53:49.081 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T21:53:49.081 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T21:53:49.081 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T21:53:49.082 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T21:53:49.082 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T21:53:49.082 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:53:49.082 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T21:53:49.083 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:53:49.083 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T21:53:49.083 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T21:53:49.083 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T21:53:49.083 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T21:53:49.084 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T21:53:49.084 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:53:49.084 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T21:53:49.522 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T21:53:49.523 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:53:49.523 INFO:teuthology.orchestra.run.smithi076.stdout:Removed: 2021-03-15T21:53:49.523 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:53:49.523 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:53:49.523 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:53:49.524 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:53:49.524 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:53:49.524 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:53:49.524 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:53:49.525 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:53:49.525 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:53:49.525 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:53:49.525 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:53:49.526 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:53:49.526 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:53:49.526 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:53:49.526 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:53:49.526 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:53:49.527 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:53:49.527 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:53:49.812 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T21:53:50.266 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 85 kB/s | 3.3 kB 00:00 2021-03-15T21:53:50.584 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 5.3 kB/s | 1.7 kB 00:00 2021-03-15T21:53:51.117 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:53:51.119 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:53:51.119 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:53:51.119 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:53:51.120 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:53:51.120 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T21:53:51.120 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:53:51.120 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T21:53:51.121 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T21:53:51.121 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T21:53:51.121 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T21:53:51.121 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T21:53:51.122 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T21:53:51.122 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T21:53:51.122 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T21:53:51.122 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T21:53:51.123 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T21:53:51.123 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T21:53:51.123 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T21:53:51.123 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T21:53:51.124 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T21:53:51.124 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T21:53:51.124 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T21:53:51.124 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:53:51.125 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:53:51.125 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:53:51.125 INFO:teuthology.orchestra.run.smithi076.stdout:Install 16 Packages 2021-03-15T21:53:51.125 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:53:51.127 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 68 M 2021-03-15T21:53:51.127 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2021-03-15T21:53:51.127 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:53:51.153 INFO:teuthology.orchestra.run.smithi076.stdout:(1/16): yajl-2.1.0-10.el8.x86_64.rpm 1.3 MB/s | 41 kB 00:00 2021-03-15T21:53:51.154 INFO:teuthology.orchestra.run.smithi076.stdout:(2/16): fuse3-3.2.1-12.el8.x86_64.rpm 28 MB/s | 50 kB 00:00 2021-03-15T21:53:51.157 INFO:teuthology.orchestra.run.smithi076.stdout:(3/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 47 MB/s | 94 kB 00:00 2021-03-15T21:53:51.157 INFO:teuthology.orchestra.run.smithi076.stdout:(4/16): fuse-common-3.2.1-12.el8.x86_64.rpm 613 kB/s | 21 kB 00:00 2021-03-15T21:53:51.166 INFO:teuthology.orchestra.run.smithi076.stdout:(5/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 7.2 MB/s | 318 kB 00:00 2021-03-15T21:53:51.196 INFO:teuthology.orchestra.run.smithi076.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.5 MB/s | 59 kB 00:00 2021-03-15T21:53:51.999 INFO:teuthology.orchestra.run.smithi076.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 60 kB/s | 50 kB 00:00 2021-03-15T21:53:52.576 INFO:teuthology.orchestra.run.smithi076.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 203 kB/s | 289 kB 00:01 2021-03-15T21:53:52.826 INFO:teuthology.orchestra.run.smithi076.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 71 kB/s | 59 kB 00:00 2021-03-15T21:53:52.986 INFO:teuthology.orchestra.run.smithi076.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 454 kB/s | 186 kB 00:00 2021-03-15T21:53:53.144 INFO:teuthology.orchestra.run.smithi076.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 227 kB/s | 72 kB 00:00 2021-03-15T21:53:53.292 INFO:teuthology.orchestra.run.smithi076.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 794 kB/s | 241 kB 00:00 2021-03-15T21:53:54.489 INFO:teuthology.orchestra.run.smithi076.stdout:(13/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T21:53:54.506 INFO:teuthology.orchestra.run.smithi076.stdout:(14/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 2.0 MB/s | 2.5 MB 00:01 2021-03-15T21:53:54.825 INFO:teuthology.orchestra.run.smithi076.stdout:(15/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 14 MB/s | 23 MB 00:01 2021-03-15T21:53:54.827 INFO:teuthology.orchestra.run.smithi076.stdout:(16/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 162 kB/s | 55 kB 00:00 2021-03-15T21:53:54.833 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:53:54.834 INFO:teuthology.orchestra.run.smithi076.stdout:Total 18 MB/s | 68 MB 00:03 2021-03-15T21:53:54.849 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:53:54.893 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:53:54.893 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:53:55.373 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:53:55.375 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:53:56.095 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:53:56.259 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T21:53:58.686 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T21:53:58.860 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T21:53:58.986 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T21:53:59.104 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T21:53:59.117 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:53:59.237 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:54:16.682 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:54:16.843 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:54:16.928 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:54:17.041 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:54:17.166 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T21:54:17.293 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:54:17.316 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:54:17.436 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T21:54:17.575 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:54:17.693 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T21:54:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T21:54:17.780 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.AdNgz9: line 5: 33932 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T21:54:18.050 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:54:19.873 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:54:20.009 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:54:20.573 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T21:54:20.851 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:54:20.851 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.9175k5: line 6: 34882 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T21:54:20.852 INFO:teuthology.orchestra.run.smithi076.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T21:54:20.852 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:54:20.852 INFO:teuthology.orchestra.run.smithi076.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T21:54:21.061 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T21:54:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T21:54:21.062 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T21:54:21.069 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T21:54:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:54:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:54:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T21:54:21.070 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T21:54:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T21:54:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T21:54:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T21:54:21.071 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T21:54:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:54:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:54:21.072 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:54:21.572 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T21:54:21.573 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:54:21.573 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:54:21.573 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:54:21.574 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:54:21.574 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:54:21.575 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:54:21.576 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:54:21.576 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:54:21.576 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:54:21.577 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:54:21.577 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:54:21.578 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:54:21.578 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:54:21.579 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:54:21.580 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:54:21.580 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:54:21.581 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:54:21.581 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:54:21.582 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:54:21.582 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:54:21.815 INFO:teuthology.task.exec:Running commands on role osd.2 host ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:54:21.815 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:54:22.288 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 09:53:50 PM UTC. 2021-03-15T21:54:22.774 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:54:22.775 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:54:22.775 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:54:22.844 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:54:23.310 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 09:53:50 PM UTC. 2021-03-15T21:54:23.655 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:54:23.764 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:54:23.765 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:54:23.766 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:54:23.835 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:54:24.120 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2021-03-15T21:54:24.127 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:54:24.127 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2021-03-15T21:54:24.127 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:54:24.127 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:54:24.128 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:54:24.128 INFO:teuthology.orchestra.run.smithi076.stderr:, 2021-03-15T21:54:24.128 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2021-03-15T21:54:24.128 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:54:24.129 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:54:24.129 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:54:24.157 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:54:24.194 INFO:teuthology.orchestra.run.smithi076.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:54:24.194 INFO:teuthology.orchestra.run.smithi076.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:54:24.509 INFO:teuthology.orchestra.run.smithi076.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1139 0 --:--:-- --:--:-- --:--:-- 1139 2021-03-15T21:54:24.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:54:24.984 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:54:24.987 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:54:24.987 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:54:24.987 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:54:24.988 INFO:teuthology.orchestra.run.smithi076.stdout:Removing: 2021-03-15T21:54:24.988 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T21:54:24.988 INFO:teuthology.orchestra.run.smithi076.stdout:Removing unused dependencies: 2021-03-15T21:54:24.988 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T21:54:24.989 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T21:54:24.989 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T21:54:24.989 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T21:54:24.989 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T21:54:24.989 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T21:54:24.990 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T21:54:24.990 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T21:54:24.990 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T21:54:24.990 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T21:54:24.991 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:54:24.991 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T21:54:24.991 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T21:54:24.991 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T21:54:24.991 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T21:54:24.992 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:54:24.992 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:54:24.992 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:54:24.992 INFO:teuthology.orchestra.run.smithi076.stdout:Remove 16 Packages 2021-03-15T21:54:24.993 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:54:24.993 INFO:teuthology.orchestra.run.smithi076.stdout:Freed space: 173 M 2021-03-15T21:54:24.994 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:54:24.997 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:54:24.998 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:54:25.049 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:54:25.049 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:54:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:54:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T21:54:25.325 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:54:25.647 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:54:25.770 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T21:54:25.892 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T21:54:26.017 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T21:54:26.167 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:54:26.276 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:54:26.425 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T21:54:26.526 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:54:26.526 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:54:42.952 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:54:43.065 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T21:54:43.082 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:54:43.083 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:54:43.508 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:54:43.624 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:54:43.773 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:54:43.774 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:54:43.974 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:54:44.084 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T21:54:44.251 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:54:44.484 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:54:44.484 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T21:54:44.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T21:54:44.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T21:54:44.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T21:54:44.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T21:54:44.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:54:44.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T21:54:44.486 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:54:44.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T21:54:44.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T21:54:44.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T21:54:44.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T21:54:44.487 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T21:54:44.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:54:44.488 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T21:54:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T21:54:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:54:44.954 INFO:teuthology.orchestra.run.smithi076.stdout:Removed: 2021-03-15T21:54:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:54:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:54:44.955 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:54:44.955 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:54:44.955 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:54:44.955 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:54:44.956 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:54:44.956 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:54:44.956 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:54:44.956 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:54:44.957 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:54:44.957 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:54:44.957 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:54:44.957 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:54:44.957 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:54:44.958 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:54:44.958 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:54:44.958 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:54:45.229 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T21:54:45.685 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 94 kB/s | 3.3 kB 00:00 2021-03-15T21:54:46.002 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 5.3 kB/s | 1.7 kB 00:00 2021-03-15T21:54:46.524 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:54:46.526 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:54:46.527 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:54:46.527 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:54:46.527 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:54:46.527 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T21:54:46.528 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:54:46.528 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T21:54:46.528 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T21:54:46.528 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T21:54:46.529 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T21:54:46.529 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T21:54:46.529 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T21:54:46.529 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T21:54:46.529 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T21:54:46.530 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T21:54:46.530 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T21:54:46.530 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T21:54:46.530 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T21:54:46.531 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T21:54:46.531 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T21:54:46.531 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T21:54:46.531 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T21:54:46.532 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:54:46.532 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:54:46.532 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:54:46.533 INFO:teuthology.orchestra.run.smithi076.stdout:Install 16 Packages 2021-03-15T21:54:46.533 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:54:46.534 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 68 M 2021-03-15T21:54:46.534 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2021-03-15T21:54:46.534 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:54:46.564 INFO:teuthology.orchestra.run.smithi076.stdout:(1/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 9.2 MB/s | 318 kB 00:00 2021-03-15T21:54:46.566 INFO:teuthology.orchestra.run.smithi076.stdout:(2/16): fuse3-3.2.1-12.el8.x86_64.rpm 24 MB/s | 50 kB 00:00 2021-03-15T21:54:46.568 INFO:teuthology.orchestra.run.smithi076.stdout:(3/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 45 MB/s | 94 kB 00:00 2021-03-15T21:54:46.569 INFO:teuthology.orchestra.run.smithi076.stdout:(4/16): fuse-common-3.2.1-12.el8.x86_64.rpm 542 kB/s | 21 kB 00:00 2021-03-15T21:54:46.575 INFO:teuthology.orchestra.run.smithi076.stdout:(5/16): yajl-2.1.0-10.el8.x86_64.rpm 906 kB/s | 41 kB 00:00 2021-03-15T21:54:46.610 INFO:teuthology.orchestra.run.smithi076.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.4 MB/s | 59 kB 00:00 2021-03-15T21:54:47.647 INFO:teuthology.orchestra.run.smithi076.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 46 kB/s | 50 kB 00:01 2021-03-15T21:54:47.842 INFO:teuthology.orchestra.run.smithi076.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 227 kB/s | 289 kB 00:01 2021-03-15T21:54:47.959 INFO:teuthology.orchestra.run.smithi076.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 188 kB/s | 59 kB 00:00 2021-03-15T21:54:48.153 INFO:teuthology.orchestra.run.smithi076.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 597 kB/s | 186 kB 00:00 2021-03-15T21:54:48.288 INFO:teuthology.orchestra.run.smithi076.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 220 kB/s | 72 kB 00:00 2021-03-15T21:54:48.460 INFO:teuthology.orchestra.run.smithi076.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 788 kB/s | 241 kB 00:00 2021-03-15T21:54:49.492 INFO:teuthology.orchestra.run.smithi076.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 2.4 MB/s | 2.5 MB 00:01 2021-03-15T21:54:49.920 INFO:teuthology.orchestra.run.smithi076.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 12 MB/s | 41 MB 00:03 2021-03-15T21:54:50.024 INFO:teuthology.orchestra.run.smithi076.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 103 kB/s | 55 kB 00:00 2021-03-15T21:54:50.383 INFO:teuthology.orchestra.run.smithi076.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 11 MB/s | 23 MB 00:02 2021-03-15T21:54:50.390 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:54:50.390 INFO:teuthology.orchestra.run.smithi076.stdout:Total 18 MB/s | 68 MB 00:03 2021-03-15T21:54:50.405 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:54:50.448 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:54:50.449 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:54:50.928 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:54:50.929 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:54:51.582 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:54:51.702 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T21:54:54.130 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T21:54:54.303 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T21:54:54.445 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T21:54:54.588 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T21:54:54.603 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:54:54.729 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:55:12.226 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:55:12.377 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:55:12.441 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:55:12.558 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:55:12.684 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T21:55:12.844 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:55:12.867 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:55:13.003 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T21:55:13.151 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:55:13.269 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T21:55:13.334 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T21:55:13.334 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.gmeXdZ: line 5: 36524 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T21:55:13.587 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:55:15.382 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:55:15.535 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:55:16.104 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T21:55:16.431 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:55:16.432 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.YZbJ4r: line 6: 37475 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T21:55:16.432 INFO:teuthology.orchestra.run.smithi076.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T21:55:16.432 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:55:16.433 INFO:teuthology.orchestra.run.smithi076.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T21:55:16.616 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T21:55:16.617 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T21:55:16.617 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T21:55:16.617 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T21:55:16.618 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:55:16.618 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:55:16.619 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T21:55:16.619 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T21:55:16.620 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T21:55:16.620 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T21:55:16.620 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T21:55:16.621 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T21:55:16.621 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:55:16.622 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:55:16.622 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:55:17.114 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T21:55:17.114 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:55:17.115 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:55:17.115 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:55:17.115 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:55:17.115 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:55:17.116 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:55:17.116 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:55:17.116 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:55:17.116 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:55:17.116 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:55:17.117 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:55:17.117 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:55:17.117 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:55:17.117 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:55:17.118 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:55:17.118 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:55:17.118 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:55:17.118 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:55:17.118 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:55:17.119 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:55:17.335 INFO:teuthology.task.exec:Running commands on role osd.3 host ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:55:17.336 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:55:17.811 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:31 ago on Mon 15 Mar 2021 09:54:46 PM UTC. 2021-03-15T21:55:18.306 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:55:18.308 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:55:18.308 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:55:18.379 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:55:18.845 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 09:54:46 PM UTC. 2021-03-15T21:55:19.172 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:55:19.286 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:55:19.288 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:55:19.288 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:55:19.364 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:55:19.651 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2021-03-15T21:55:19.658 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:55:19.658 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2021-03-15T21:55:19.658 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:55:19.658 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:55:19.659 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:55:19.659 INFO:teuthology.orchestra.run.smithi076.stderr:, 2021-03-15T21:55:19.659 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2021-03-15T21:55:19.659 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:55:19.660 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:55:19.660 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:55:19.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:55:19.725 INFO:teuthology.orchestra.run.smithi076.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:55:19.725 INFO:teuthology.orchestra.run.smithi076.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:55:20.040 INFO:teuthology.orchestra.run.smithi076.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1136 0 --:--:-- --:--:-- --:--:-- 1136 2021-03-15T21:55:20.043 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:55:20.518 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:55:20.522 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:55:20.522 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:55:20.522 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:55:20.523 INFO:teuthology.orchestra.run.smithi076.stdout:Removing: 2021-03-15T21:55:20.523 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T21:55:20.523 INFO:teuthology.orchestra.run.smithi076.stdout:Removing unused dependencies: 2021-03-15T21:55:20.523 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T21:55:20.524 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T21:55:20.524 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T21:55:20.524 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T21:55:20.524 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T21:55:20.524 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T21:55:20.525 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T21:55:20.525 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T21:55:20.525 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T21:55:20.525 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T21:55:20.526 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:55:20.526 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T21:55:20.526 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T21:55:20.526 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T21:55:20.526 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T21:55:20.527 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:55:20.527 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:55:20.527 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:55:20.527 INFO:teuthology.orchestra.run.smithi076.stdout:Remove 16 Packages 2021-03-15T21:55:20.528 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:55:20.528 INFO:teuthology.orchestra.run.smithi076.stdout:Freed space: 173 M 2021-03-15T21:55:20.529 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:55:20.532 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:55:20.533 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:55:20.584 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:55:20.585 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:55:20.795 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:55:20.824 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T21:55:20.825 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:55:21.190 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:55:21.322 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T21:55:21.452 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T21:55:21.568 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T21:55:21.693 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:55:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:55:21.976 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T21:55:22.077 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:55:22.077 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:55:38.571 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:55:38.691 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T21:55:38.708 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:55:38.708 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:55:38.841 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:55:38.966 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:55:39.083 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:55:39.083 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:55:39.291 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:55:39.443 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T21:55:39.602 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:55:39.818 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:55:39.818 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T21:55:39.819 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T21:55:39.819 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T21:55:39.819 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T21:55:39.819 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T21:55:39.820 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:55:39.820 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T21:55:39.820 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:55:39.820 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T21:55:39.821 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T21:55:39.821 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T21:55:39.821 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T21:55:39.822 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T21:55:39.822 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:55:39.822 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T21:55:40.281 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T21:55:40.282 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:55:40.282 INFO:teuthology.orchestra.run.smithi076.stdout:Removed: 2021-03-15T21:55:40.282 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:55:40.283 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:55:40.283 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:55:40.283 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:55:40.283 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:55:40.283 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:55:40.284 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:55:40.284 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:55:40.284 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:55:40.284 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:55:40.285 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:55:40.285 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:55:40.285 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:55:40.285 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:55:40.286 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:55:40.286 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:55:40.286 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:55:40.286 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:55:40.617 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T21:55:41.076 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 84 kB/s | 3.3 kB 00:00 2021-03-15T21:55:41.409 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 5.0 kB/s | 1.7 kB 00:00 2021-03-15T21:55:41.927 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:55:41.929 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:55:41.930 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:55:41.930 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:55:41.930 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:55:41.930 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T21:55:41.931 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:55:41.931 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T21:55:41.931 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T21:55:41.931 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T21:55:41.932 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T21:55:41.932 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T21:55:41.932 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T21:55:41.932 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T21:55:41.932 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T21:55:41.933 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T21:55:41.933 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T21:55:41.933 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T21:55:41.933 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T21:55:41.934 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T21:55:41.934 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T21:55:41.934 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T21:55:41.934 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T21:55:41.935 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:55:41.935 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:55:41.935 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:55:41.935 INFO:teuthology.orchestra.run.smithi076.stdout:Install 16 Packages 2021-03-15T21:55:41.935 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:55:41.936 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 68 M 2021-03-15T21:55:41.937 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2021-03-15T21:55:41.937 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:55:41.969 INFO:teuthology.orchestra.run.smithi076.stdout:(1/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T21:55:41.971 INFO:teuthology.orchestra.run.smithi076.stdout:(2/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 8.1 MB/s | 318 kB 00:00 2021-03-15T21:55:41.972 INFO:teuthology.orchestra.run.smithi076.stdout:(3/16): fuse3-3.2.1-12.el8.x86_64.rpm 17 MB/s | 50 kB 00:00 2021-03-15T21:55:41.973 INFO:teuthology.orchestra.run.smithi076.stdout:(4/16): fuse-common-3.2.1-12.el8.x86_64.rpm 527 kB/s | 21 kB 00:00 2021-03-15T21:55:41.974 INFO:teuthology.orchestra.run.smithi076.stdout:(5/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 40 MB/s | 94 kB 00:00 2021-03-15T21:55:42.012 INFO:teuthology.orchestra.run.smithi076.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.5 MB/s | 59 kB 00:00 2021-03-15T21:55:43.045 INFO:teuthology.orchestra.run.smithi076.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 46 kB/s | 50 kB 00:01 2021-03-15T21:55:43.241 INFO:teuthology.orchestra.run.smithi076.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 228 kB/s | 289 kB 00:01 2021-03-15T21:55:43.353 INFO:teuthology.orchestra.run.smithi076.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 190 kB/s | 59 kB 00:00 2021-03-15T21:55:43.547 INFO:teuthology.orchestra.run.smithi076.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 610 kB/s | 186 kB 00:00 2021-03-15T21:55:43.657 INFO:teuthology.orchestra.run.smithi076.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 237 kB/s | 72 kB 00:00 2021-03-15T21:55:43.855 INFO:teuthology.orchestra.run.smithi076.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 784 kB/s | 241 kB 00:00 2021-03-15T21:55:44.879 INFO:teuthology.orchestra.run.smithi076.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 2.4 MB/s | 2.5 MB 00:01 2021-03-15T21:55:45.228 INFO:teuthology.orchestra.run.smithi076.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T21:55:45.364 INFO:teuthology.orchestra.run.smithi076.stdout:(15/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 14 MB/s | 23 MB 00:01 2021-03-15T21:55:45.366 INFO:teuthology.orchestra.run.smithi076.stdout:(16/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 112 kB/s | 55 kB 00:00 2021-03-15T21:55:45.372 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:55:45.373 INFO:teuthology.orchestra.run.smithi076.stdout:Total 20 MB/s | 68 MB 00:03 2021-03-15T21:55:45.388 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:55:45.432 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:55:45.432 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:55:45.912 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:55:45.914 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:55:46.568 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:55:46.695 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T21:55:49.150 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T21:55:49.329 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T21:55:49.496 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T21:55:49.606 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T21:55:49.620 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:55:49.739 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:56:07.312 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:56:07.469 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:56:07.535 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:56:07.651 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:56:07.801 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T21:56:07.953 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:56:07.976 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:56:08.112 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T21:56:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:56:08.378 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T21:56:08.437 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T21:56:08.437 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.GXjLQy: line 5: 39117 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T21:56:08.724 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:56:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:56:10.686 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:56:11.251 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T21:56:11.535 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:56:11.535 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.F0c2Hp: line 6: 40067 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T21:56:11.536 INFO:teuthology.orchestra.run.smithi076.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T21:56:11.536 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:56:11.536 INFO:teuthology.orchestra.run.smithi076.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T21:56:11.747 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T21:56:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T21:56:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T21:56:11.748 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T21:56:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:56:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:56:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T21:56:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T21:56:11.749 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T21:56:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T21:56:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T21:56:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T21:56:11.750 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:56:11.751 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:56:11.751 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:56:12.394 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T21:56:12.395 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:56:12.395 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:56:12.395 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:56:12.396 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:56:12.396 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:56:12.396 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:56:12.396 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:56:12.396 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:56:12.397 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:56:12.397 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:56:12.397 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:56:12.397 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:56:12.398 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:56:12.398 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:56:12.398 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:56:12.398 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:56:12.398 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:56:12.399 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:56:12.399 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:56:12.399 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:56:12.593 INFO:teuthology.task.exec:Running commands on role client.0 host ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:56:12.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:56:13.068 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 09:55:41 PM UTC. 2021-03-15T21:56:13.550 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:56:13.551 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:56:13.551 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:56:13.640 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:56:14.261 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 09:55:41 PM UTC. 2021-03-15T21:56:14.590 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:56:14.700 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:56:14.701 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:56:14.702 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:56:14.773 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:56:15.060 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2021-03-15T21:56:15.066 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:56:15.067 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2021-03-15T21:56:15.067 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:56:15.067 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:56:15.067 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:56:15.067 INFO:teuthology.orchestra.run.smithi076.stderr:, 2021-03-15T21:56:15.068 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2021-03-15T21:56:15.068 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:56:15.068 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:56:15.068 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:56:15.096 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:56:15.134 INFO:teuthology.orchestra.run.smithi076.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:56:15.135 INFO:teuthology.orchestra.run.smithi076.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:56:15.459 INFO:teuthology.orchestra.run.smithi076.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1101 0 --:--:-- --:--:-- --:--:-- 1101 2021-03-15T21:56:15.462 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:56:15.935 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:56:15.938 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:56:15.939 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:56:15.939 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:56:15.939 INFO:teuthology.orchestra.run.smithi076.stdout:Removing: 2021-03-15T21:56:15.939 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T21:56:15.940 INFO:teuthology.orchestra.run.smithi076.stdout:Removing unused dependencies: 2021-03-15T21:56:15.940 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T21:56:15.940 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T21:56:15.940 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T21:56:15.940 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T21:56:15.941 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T21:56:15.941 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T21:56:15.941 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T21:56:15.941 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T21:56:15.942 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T21:56:15.942 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T21:56:15.942 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:56:15.942 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T21:56:15.942 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T21:56:15.943 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T21:56:15.943 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T21:56:15.943 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:56:15.943 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:56:15.943 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:56:15.944 INFO:teuthology.orchestra.run.smithi076.stdout:Remove 16 Packages 2021-03-15T21:56:15.944 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:56:15.944 INFO:teuthology.orchestra.run.smithi076.stdout:Freed space: 173 M 2021-03-15T21:56:15.945 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:56:15.949 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:56:15.949 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:56:16.001 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:56:16.002 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:56:16.243 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:56:16.274 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T21:56:16.274 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:56:16.574 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:56:16.706 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T21:56:16.852 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T21:56:17.069 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T21:56:17.194 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:56:17.304 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:56:17.453 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T21:56:17.553 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:56:17.554 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:56:34.262 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:56:34.392 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T21:56:34.409 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:56:34.409 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:56:34.551 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:56:34.667 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:56:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:56:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:56:34.992 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:56:35.102 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T21:56:35.278 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:56:35.503 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:56:35.504 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T21:56:35.504 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T21:56:35.504 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T21:56:35.504 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T21:56:35.505 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T21:56:35.505 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:56:35.505 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T21:56:35.505 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:56:35.506 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T21:56:35.506 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T21:56:35.506 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T21:56:35.506 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T21:56:35.507 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T21:56:35.507 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:56:35.507 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T21:56:35.911 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T21:56:35.911 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:56:35.911 INFO:teuthology.orchestra.run.smithi076.stdout:Removed: 2021-03-15T21:56:35.912 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:56:35.912 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:56:35.912 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:56:35.912 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:56:35.913 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:56:35.913 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:56:35.913 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:56:35.913 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:56:35.914 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:56:35.914 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:56:35.914 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:56:35.914 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:56:35.915 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:56:35.915 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:56:35.915 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:56:35.915 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:56:35.916 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:56:35.916 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:56:36.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T21:56:36.711 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 87 kB/s | 3.3 kB 00:00 2021-03-15T21:56:37.031 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 5.2 kB/s | 1.7 kB 00:00 2021-03-15T21:56:37.552 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:56:37.554 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:56:37.555 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:56:37.555 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:56:37.555 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:56:37.556 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T21:56:37.556 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:56:37.556 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T21:56:37.556 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T21:56:37.557 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T21:56:37.557 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T21:56:37.557 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T21:56:37.557 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T21:56:37.558 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T21:56:37.558 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T21:56:37.558 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T21:56:37.558 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T21:56:37.559 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T21:56:37.559 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T21:56:37.559 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T21:56:37.559 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T21:56:37.559 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T21:56:37.560 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T21:56:37.560 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:56:37.560 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:56:37.560 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:56:37.561 INFO:teuthology.orchestra.run.smithi076.stdout:Install 16 Packages 2021-03-15T21:56:37.561 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:56:37.562 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 68 M 2021-03-15T21:56:37.562 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2021-03-15T21:56:37.562 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:56:37.591 INFO:teuthology.orchestra.run.smithi076.stdout:(1/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 9.5 MB/s | 318 kB 00:00 2021-03-15T21:56:37.593 INFO:teuthology.orchestra.run.smithi076.stdout:(2/16): fuse3-3.2.1-12.el8.x86_64.rpm 33 MB/s | 50 kB 00:00 2021-03-15T21:56:37.595 INFO:teuthology.orchestra.run.smithi076.stdout:(3/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T21:56:37.596 INFO:teuthology.orchestra.run.smithi076.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 33 MB/s | 94 kB 00:00 2021-03-15T21:56:37.598 INFO:teuthology.orchestra.run.smithi076.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 531 kB/s | 21 kB 00:00 2021-03-15T21:56:37.635 INFO:teuthology.orchestra.run.smithi076.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.5 MB/s | 59 kB 00:00 2021-03-15T21:56:38.635 INFO:teuthology.orchestra.run.smithi076.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 48 kB/s | 50 kB 00:01 2021-03-15T21:56:38.893 INFO:teuthology.orchestra.run.smithi076.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 223 kB/s | 289 kB 00:01 2021-03-15T21:56:39.040 INFO:teuthology.orchestra.run.smithi076.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 144 kB/s | 59 kB 00:00 2021-03-15T21:56:39.300 INFO:teuthology.orchestra.run.smithi076.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 458 kB/s | 186 kB 00:00 2021-03-15T21:56:39.345 INFO:teuthology.orchestra.run.smithi076.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 237 kB/s | 72 kB 00:00 2021-03-15T21:56:39.607 INFO:teuthology.orchestra.run.smithi076.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 788 kB/s | 241 kB 00:00 2021-03-15T21:56:40.228 INFO:teuthology.orchestra.run.smithi076.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 4.0 MB/s | 2.5 MB 00:00 2021-03-15T21:56:40.536 INFO:teuthology.orchestra.run.smithi076.stdout:(14/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 178 kB/s | 55 kB 00:00 2021-03-15T21:56:40.833 INFO:teuthology.orchestra.run.smithi076.stdout:(15/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T21:56:41.093 INFO:teuthology.orchestra.run.smithi076.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 13 MB/s | 23 MB 00:01 2021-03-15T21:56:41.101 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:56:41.102 INFO:teuthology.orchestra.run.smithi076.stdout:Total 19 MB/s | 68 MB 00:03 2021-03-15T21:56:41.116 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:56:41.159 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:56:41.160 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:56:41.642 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:56:41.644 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:56:42.303 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:56:42.446 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T21:56:44.892 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T21:56:45.072 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T21:56:45.222 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T21:56:45.332 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T21:56:45.347 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:56:45.473 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:57:03.056 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:57:03.204 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:57:03.270 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:57:03.377 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:57:03.527 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T21:57:03.679 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:57:03.703 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:57:03.839 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T21:57:03.986 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:57:04.104 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T21:57:04.177 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T21:57:04.178 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.zp8HQH: line 5: 41708 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T21:57:04.462 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:57:06.267 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:57:06.420 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:57:06.983 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T21:57:07.302 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:57:07.303 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.pIsxma: line 6: 42658 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T21:57:07.303 INFO:teuthology.orchestra.run.smithi076.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T21:57:07.303 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:57:07.304 INFO:teuthology.orchestra.run.smithi076.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T21:57:07.499 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:57:07.500 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T21:57:07.501 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:57:07.502 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:57:08.062 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T21:57:08.062 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:57:08.063 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:57:08.063 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:57:08.063 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:57:08.063 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:57:08.064 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:57:08.064 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:57:08.064 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:57:08.064 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:57:08.064 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:57:08.065 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:57:08.065 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:57:08.065 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:57:08.065 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:57:08.066 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:57:08.066 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:57:08.066 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:57:08.066 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:57:08.066 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:57:08.067 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:57:08.278 INFO:teuthology.task.exec:Running commands on role ceph.rgw.foo.a host ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:57:08.279 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:57:08.751 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:31 ago on Mon 15 Mar 2021 09:56:37 PM UTC. 2021-03-15T21:57:09.234 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:57:09.235 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:57:09.235 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:57:09.308 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:57:09.772 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 09:56:37 PM UTC. 2021-03-15T21:57:10.116 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:57:10.228 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:57:10.229 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:57:10.229 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:57:10.304 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:57:10.597 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2021-03-15T21:57:10.603 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:57:10.604 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2021-03-15T21:57:10.604 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:57:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:57:10.605 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:57:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:, 2021-03-15T21:57:10.606 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2021-03-15T21:57:10.606 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:57:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:57:10.607 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:57:10.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:57:10.671 INFO:teuthology.orchestra.run.smithi076.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:57:10.671 INFO:teuthology.orchestra.run.smithi076.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:57:10.989 INFO:teuthology.orchestra.run.smithi076.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1128 0 --:--:-- --:--:-- --:--:-- 1128 2021-03-15T21:57:11.164 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:57:11.640 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:57:11.644 INFO:teuthology.orchestra.run.smithi076.stdout:Removing: 2021-03-15T21:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T21:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout:Removing unused dependencies: 2021-03-15T21:57:11.645 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T21:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T21:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T21:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T21:57:11.646 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T21:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T21:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T21:57:11.647 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T21:57:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T21:57:11.648 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T21:57:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:57:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T21:57:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T21:57:11.649 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T21:57:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T21:57:11.650 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:57:11.650 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:57:11.651 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:57:11.651 INFO:teuthology.orchestra.run.smithi076.stdout:Remove 16 Packages 2021-03-15T21:57:11.651 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:57:11.651 INFO:teuthology.orchestra.run.smithi076.stdout:Freed space: 173 M 2021-03-15T21:57:11.652 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:57:11.654 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:57:11.654 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:57:11.706 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:57:11.707 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:57:11.949 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:57:11.978 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T21:57:11.978 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:57:12.291 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:57:12.424 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T21:57:12.570 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T21:57:12.745 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T21:57:12.861 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:57:12.971 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:57:13.128 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T21:57:13.220 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:57:13.221 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:57:29.913 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:57:30.026 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T21:57:30.044 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:57:30.044 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:57:30.493 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:57:30.618 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:57:30.734 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:57:30.735 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:57:30.985 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:57:31.086 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T21:57:31.237 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:57:31.463 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:57:31.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T21:57:31.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T21:57:31.463 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T21:57:31.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T21:57:31.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T21:57:31.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:57:31.464 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T21:57:31.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:57:31.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T21:57:31.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T21:57:31.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T21:57:31.465 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T21:57:31.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T21:57:31.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:57:31.466 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T21:57:31.950 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T21:57:31.950 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:57:31.950 INFO:teuthology.orchestra.run.smithi076.stdout:Removed: 2021-03-15T21:57:31.951 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:57:31.951 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:57:31.951 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:57:31.951 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:57:31.952 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:57:31.952 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:57:31.953 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:57:31.953 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:57:31.954 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:57:31.954 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:57:31.954 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:57:31.955 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:57:31.955 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:57:31.955 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:57:31.956 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:57:31.956 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:57:31.956 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:57:31.956 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:57:32.223 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T21:57:32.679 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 83 kB/s | 3.3 kB 00:00 2021-03-15T21:57:32.992 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 5.4 kB/s | 1.7 kB 00:00 2021-03-15T21:57:33.493 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:57:33.496 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:57:33.496 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:57:33.496 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:57:33.497 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:57:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T21:57:33.497 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:57:33.497 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T21:57:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T21:57:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T21:57:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T21:57:33.498 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T21:57:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T21:57:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T21:57:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T21:57:33.499 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T21:57:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T21:57:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T21:57:33.500 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T21:57:33.500 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T21:57:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T21:57:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T21:57:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T21:57:33.501 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:57:33.501 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:57:33.502 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:57:33.502 INFO:teuthology.orchestra.run.smithi076.stdout:Install 16 Packages 2021-03-15T21:57:33.502 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:57:33.503 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 68 M 2021-03-15T21:57:33.503 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2021-03-15T21:57:33.503 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:57:33.534 INFO:teuthology.orchestra.run.smithi076.stdout:(1/16): yajl-2.1.0-10.el8.x86_64.rpm 1.2 MB/s | 41 kB 00:00 2021-03-15T21:57:33.535 INFO:teuthology.orchestra.run.smithi076.stdout:(2/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 8.6 MB/s | 318 kB 00:00 2021-03-15T21:57:33.536 INFO:teuthology.orchestra.run.smithi076.stdout:(3/16): fuse3-3.2.1-12.el8.x86_64.rpm 21 MB/s | 50 kB 00:00 2021-03-15T21:57:33.537 INFO:teuthology.orchestra.run.smithi076.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 47 MB/s | 94 kB 00:00 2021-03-15T21:57:33.544 INFO:teuthology.orchestra.run.smithi076.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 475 kB/s | 21 kB 00:00 2021-03-15T21:57:33.577 INFO:teuthology.orchestra.run.smithi076.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.4 MB/s | 59 kB 00:00 2021-03-15T21:57:34.606 INFO:teuthology.orchestra.run.smithi076.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 47 kB/s | 50 kB 00:01 2021-03-15T21:57:34.826 INFO:teuthology.orchestra.run.smithi076.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 224 kB/s | 289 kB 00:01 2021-03-15T21:57:35.017 INFO:teuthology.orchestra.run.smithi076.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 140 kB/s | 59 kB 00:00 2021-03-15T21:57:35.235 INFO:teuthology.orchestra.run.smithi076.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 455 kB/s | 186 kB 00:00 2021-03-15T21:57:35.422 INFO:teuthology.orchestra.run.smithi076.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 178 kB/s | 72 kB 00:00 2021-03-15T21:57:35.657 INFO:teuthology.orchestra.run.smithi076.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 572 kB/s | 241 kB 00:00 2021-03-15T21:57:36.355 INFO:teuthology.orchestra.run.smithi076.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 3.5 MB/s | 2.5 MB 00:00 2021-03-15T21:57:36.789 INFO:teuthology.orchestra.run.smithi076.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T21:57:36.893 INFO:teuthology.orchestra.run.smithi076.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 102 kB/s | 55 kB 00:00 2021-03-15T21:57:37.333 INFO:teuthology.orchestra.run.smithi076.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 12 MB/s | 23 MB 00:01 2021-03-15T21:57:37.339 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:57:37.340 INFO:teuthology.orchestra.run.smithi076.stdout:Total 18 MB/s | 68 MB 00:03 2021-03-15T21:57:37.355 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:57:37.398 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:57:37.398 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:57:37.878 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:57:37.880 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:57:38.550 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:57:38.688 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T21:57:41.100 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T21:57:41.264 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T21:57:41.389 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T21:57:41.508 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T21:57:41.520 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:57:41.648 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:57:59.551 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:57:59.696 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:57:59.745 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:57:59.861 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:57:59.969 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T21:58:00.113 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:58:00.135 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:58:00.306 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T21:58:00.453 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:58:00.580 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T21:58:00.652 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T21:58:00.652 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.gWHQim: line 5: 44299 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T21:58:00.901 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:02.684 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:58:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:58:03.384 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T21:58:03.639 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:58:03.639 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.LsIcEF: line 6: 45249 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T21:58:03.639 INFO:teuthology.orchestra.run.smithi076.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T21:58:03.640 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:03.640 INFO:teuthology.orchestra.run.smithi076.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T21:58:03.838 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T21:58:03.838 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T21:58:03.838 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T21:58:03.839 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T21:58:03.839 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:58:03.839 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:58:03.839 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T21:58:03.840 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T21:58:03.840 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T21:58:03.840 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T21:58:03.840 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T21:58:03.840 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T21:58:03.841 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:58:03.841 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:58:03.841 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:58:04.288 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T21:58:04.288 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:04.288 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:58:04.289 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:58:04.289 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:58:04.289 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:58:04.289 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:58:04.290 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:58:04.290 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:58:04.290 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:58:04.290 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:58:04.290 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:58:04.291 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:58:04.291 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:58:04.291 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:58:04.291 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:58:04.292 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:58:04.292 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:58:04.292 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:58:04.292 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:04.292 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:58:04.530 INFO:teuthology.task.exec:Running commands on role node-exporter.a host ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:58:04.530 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:58:05.004 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 09:57:32 PM UTC. 2021-03-15T21:58:05.503 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:58:05.504 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:58:05.504 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:58:05.575 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:58:06.044 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 09:57:32 PM UTC. 2021-03-15T21:58:06.383 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:58:06.497 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:58:06.498 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:58:06.498 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:58:06.583 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:58:06.871 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2021-03-15T21:58:06.877 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:58:06.878 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2021-03-15T21:58:06.878 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:58:06.878 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:58:06.879 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:58:06.879 INFO:teuthology.orchestra.run.smithi076.stderr:, 2021-03-15T21:58:06.879 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2021-03-15T21:58:06.879 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:58:06.879 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:58:06.880 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:58:06.907 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:58:06.944 INFO:teuthology.orchestra.run.smithi076.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:58:06.944 INFO:teuthology.orchestra.run.smithi076.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:58:07.277 INFO:teuthology.orchestra.run.smithi076.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1078 0 --:--:-- --:--:-- --:--:-- 1078 2021-03-15T21:58:07.280 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:58:07.758 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:58:07.762 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:58:07.762 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:58:07.762 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:58:07.763 INFO:teuthology.orchestra.run.smithi076.stdout:Removing: 2021-03-15T21:58:07.763 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T21:58:07.763 INFO:teuthology.orchestra.run.smithi076.stdout:Removing unused dependencies: 2021-03-15T21:58:07.763 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T21:58:07.764 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T21:58:07.764 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T21:58:07.764 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T21:58:07.764 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T21:58:07.764 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T21:58:07.765 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T21:58:07.765 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T21:58:07.765 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T21:58:07.765 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T21:58:07.766 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:58:07.766 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T21:58:07.766 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T21:58:07.766 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T21:58:07.767 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T21:58:07.767 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:07.767 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:58:07.767 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:58:07.767 INFO:teuthology.orchestra.run.smithi076.stdout:Remove 16 Packages 2021-03-15T21:58:07.768 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:07.768 INFO:teuthology.orchestra.run.smithi076.stdout:Freed space: 173 M 2021-03-15T21:58:07.769 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:58:07.773 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:58:07.773 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:58:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:58:07.825 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:58:08.037 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:58:08.066 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T21:58:08.067 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:58:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:58:08.525 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T21:58:08.671 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T21:58:08.796 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T21:58:08.904 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:58:09.021 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:58:09.162 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T21:58:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:58:09.263 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:58:25.739 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:58:25.852 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T21:58:25.868 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:58:25.869 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:58:26.011 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:58:26.152 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:58:26.269 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:58:26.269 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:58:26.494 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:58:26.604 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T21:58:26.754 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:58:26.975 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:58:26.976 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T21:58:26.976 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T21:58:26.976 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T21:58:26.976 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T21:58:26.977 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T21:58:26.977 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:58:26.977 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T21:58:26.977 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:58:26.977 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T21:58:26.978 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T21:58:26.978 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T21:58:26.978 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T21:58:26.978 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T21:58:26.978 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:58:26.979 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T21:58:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T21:58:27.438 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:27.439 INFO:teuthology.orchestra.run.smithi076.stdout:Removed: 2021-03-15T21:58:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:58:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:58:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:58:27.439 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:58:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:58:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:58:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:58:27.440 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:58:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:58:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:58:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:58:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:58:27.441 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:58:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:58:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:58:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:58:27.442 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:27.443 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:58:27.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T21:58:28.201 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 86 kB/s | 3.3 kB 00:00 2021-03-15T21:58:28.515 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 5.3 kB/s | 1.7 kB 00:00 2021-03-15T21:58:29.039 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:58:29.041 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:58:29.041 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:58:29.042 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:58:29.042 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:58:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T21:58:29.043 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:58:29.043 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T21:58:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T21:58:29.044 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T21:58:29.045 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T21:58:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T21:58:29.046 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T21:58:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T21:58:29.047 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T21:58:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T21:58:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T21:58:29.048 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T21:58:29.048 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T21:58:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T21:58:29.049 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T21:58:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T21:58:29.050 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T21:58:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:29.051 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:58:29.051 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:58:29.051 INFO:teuthology.orchestra.run.smithi076.stdout:Install 16 Packages 2021-03-15T21:58:29.051 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:29.052 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 68 M 2021-03-15T21:58:29.053 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2021-03-15T21:58:29.053 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:58:29.079 INFO:teuthology.orchestra.run.smithi076.stdout:(1/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 9.1 MB/s | 318 kB 00:00 2021-03-15T21:58:29.079 INFO:teuthology.orchestra.run.smithi076.stdout:(2/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T21:58:29.080 INFO:teuthology.orchestra.run.smithi076.stdout:(3/16): fuse3-3.2.1-12.el8.x86_64.rpm 35 MB/s | 50 kB 00:00 2021-03-15T21:58:29.081 INFO:teuthology.orchestra.run.smithi076.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 48 MB/s | 94 kB 00:00 2021-03-15T21:58:29.086 INFO:teuthology.orchestra.run.smithi076.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 505 kB/s | 21 kB 00:00 2021-03-15T21:58:29.126 INFO:teuthology.orchestra.run.smithi076.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.3 MB/s | 59 kB 00:00 2021-03-15T21:58:30.136 INFO:teuthology.orchestra.run.smithi076.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 47 kB/s | 50 kB 00:01 2021-03-15T21:58:30.333 INFO:teuthology.orchestra.run.smithi076.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 231 kB/s | 289 kB 00:01 2021-03-15T21:58:30.446 INFO:teuthology.orchestra.run.smithi076.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 189 kB/s | 59 kB 00:00 2021-03-15T21:58:30.644 INFO:teuthology.orchestra.run.smithi076.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 598 kB/s | 186 kB 00:00 2021-03-15T21:58:30.747 INFO:teuthology.orchestra.run.smithi076.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 239 kB/s | 72 kB 00:00 2021-03-15T21:58:30.964 INFO:teuthology.orchestra.run.smithi076.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 756 kB/s | 241 kB 00:00 2021-03-15T21:58:31.815 INFO:teuthology.orchestra.run.smithi076.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 2.9 MB/s | 2.5 MB 00:00 2021-03-15T21:58:32.397 INFO:teuthology.orchestra.run.smithi076.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T21:58:32.398 INFO:teuthology.orchestra.run.smithi076.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 94 kB/s | 55 kB 00:00 2021-03-15T21:58:33.008 INFO:teuthology.orchestra.run.smithi076.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 10 MB/s | 23 MB 00:02 2021-03-15T21:58:33.016 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:58:33.017 INFO:teuthology.orchestra.run.smithi076.stdout:Total 17 MB/s | 68 MB 00:03 2021-03-15T21:58:33.032 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:58:33.077 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:58:33.078 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:58:33.559 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:58:33.561 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:58:34.218 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:58:34.339 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T21:58:36.795 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T21:58:36.989 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T21:58:37.116 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T21:58:37.225 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T21:58:37.239 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:58:37.358 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:58:55.144 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:58:55.322 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:58:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:58:55.503 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:58:55.637 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T21:58:55.764 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:58:55.786 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:58:55.923 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T21:58:56.096 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:58:56.214 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T21:58:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T21:58:56.274 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.29Xaob: line 5: 46890 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T21:58:56.543 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:58.335 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:58:58.464 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:58:59.030 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T21:58:59.362 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:58:59.363 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.3XYwr4: line 6: 47840 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T21:58:59.363 INFO:teuthology.orchestra.run.smithi076.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T21:58:59.363 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:58:59.364 INFO:teuthology.orchestra.run.smithi076.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T21:58:59.562 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T21:58:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T21:58:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T21:58:59.563 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T21:58:59.564 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:58:59.564 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:58:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T21:58:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T21:58:59.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T21:58:59.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T21:58:59.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T21:58:59.566 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T21:58:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:58:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:58:59.567 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:59:00.089 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T21:59:00.089 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:00.089 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:59:00.090 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:59:00.090 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:59:00.090 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:59:00.090 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:59:00.091 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:59:00.091 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:59:00.091 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:59:00.091 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:59:00.091 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:59:00.092 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:59:00.092 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:59:00.092 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:59:00.092 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:59:00.093 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:59:00.093 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:59:00.093 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:59:00.093 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:00.093 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:59:00.351 INFO:teuthology.task.exec:Running commands on role alertmanager.a host ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T21:59:00.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:59:00.825 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 09:58:28 PM UTC. 2021-03-15T21:59:01.328 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:59:01.329 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:59:01.329 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:59:01.406 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:59:01.871 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 09:58:28 PM UTC. 2021-03-15T21:59:02.209 INFO:teuthology.orchestra.run.smithi076.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:59:02.322 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:59:02.323 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T21:59:02.323 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:59:02.392 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:59:02.679 INFO:teuthology.orchestra.run.smithi076.stdout:Repository successfully enabled. 2021-03-15T21:59:02.686 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:59:02.687 INFO:teuthology.orchestra.run.smithi076.stderr:of the main distribution, and quality may vary. 2021-03-15T21:59:02.687 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:59:02.687 INFO:teuthology.orchestra.run.smithi076.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:59:02.687 INFO:teuthology.orchestra.run.smithi076.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:59:02.687 INFO:teuthology.orchestra.run.smithi076.stderr:, 2021-03-15T21:59:02.688 INFO:teuthology.orchestra.run.smithi076.stderr:and packages are not held to any quality or security level. 2021-03-15T21:59:02.688 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T21:59:02.688 INFO:teuthology.orchestra.run.smithi076.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:59:02.688 INFO:teuthology.orchestra.run.smithi076.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:59:02.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:59:02.753 INFO:teuthology.orchestra.run.smithi076.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:59:02.753 INFO:teuthology.orchestra.run.smithi076.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:59:03.086 INFO:teuthology.orchestra.run.smithi076.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1078 0 --:--:-- --:--:-- --:--:-- 1078 2021-03-15T21:59:03.089 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:59:03.567 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:59:03.570 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:59:03.570 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:59:03.571 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:59:03.571 INFO:teuthology.orchestra.run.smithi076.stdout:Removing: 2021-03-15T21:59:03.571 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T21:59:03.571 INFO:teuthology.orchestra.run.smithi076.stdout:Removing unused dependencies: 2021-03-15T21:59:03.571 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T21:59:03.572 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T21:59:03.572 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T21:59:03.572 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T21:59:03.572 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T21:59:03.573 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T21:59:03.573 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T21:59:03.573 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T21:59:03.573 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T21:59:03.573 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T21:59:03.574 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:59:03.574 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T21:59:03.574 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T21:59:03.574 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T21:59:03.575 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T21:59:03.575 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:03.575 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:59:03.575 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================= 2021-03-15T21:59:03.575 INFO:teuthology.orchestra.run.smithi076.stdout:Remove 16 Packages 2021-03-15T21:59:03.576 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:03.576 INFO:teuthology.orchestra.run.smithi076.stdout:Freed space: 173 M 2021-03-15T21:59:03.577 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:59:03.581 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:59:03.581 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:59:03.631 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:59:03.632 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:59:03.925 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:59:03.954 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T21:59:03.954 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:59:04.268 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T21:59:04.433 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T21:59:04.563 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T21:59:04.689 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T21:59:04.821 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:59:04.931 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:59:05.071 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T21:59:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:59:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:59:21.673 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T21:59:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T21:59:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:59:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:59:21.979 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T21:59:22.111 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:59:22.219 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:59:22.220 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:59:22.420 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T21:59:22.530 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T21:59:22.705 INFO:teuthology.orchestra.run.smithi076.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:59:22.926 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T21:59:22.926 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T21:59:22.926 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T21:59:22.926 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T21:59:22.927 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T21:59:22.927 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T21:59:22.927 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T21:59:22.927 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T21:59:22.928 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:59:22.928 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T21:59:22.928 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T21:59:22.928 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T21:59:22.928 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T21:59:22.929 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T21:59:22.929 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:59:22.929 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T21:59:23.357 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T21:59:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:23.358 INFO:teuthology.orchestra.run.smithi076.stdout:Removed: 2021-03-15T21:59:23.358 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:59:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:59:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:59:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:59:23.359 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:59:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:59:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:59:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:59:23.360 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:59:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:59:23.361 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:59:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:59:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:59:23.362 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:59:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:59:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:59:23.363 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:23.363 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:59:23.670 DEBUG:teuthology.orchestra.run.smithi076:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T21:59:24.126 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 90 kB/s | 3.3 kB 00:00 2021-03-15T21:59:24.449 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 5.2 kB/s | 1.7 kB 00:00 2021-03-15T21:59:24.959 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T21:59:24.961 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:59:24.961 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T21:59:24.962 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:59:24.962 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T21:59:24.962 INFO:teuthology.orchestra.run.smithi076.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T21:59:24.962 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T21:59:24.963 INFO:teuthology.orchestra.run.smithi076.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T21:59:24.963 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T21:59:24.963 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T21:59:24.963 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T21:59:24.963 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T21:59:24.964 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T21:59:24.964 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T21:59:24.964 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T21:59:24.964 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T21:59:24.965 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T21:59:24.965 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T21:59:24.965 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T21:59:24.965 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T21:59:24.965 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T21:59:24.966 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T21:59:24.966 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T21:59:24.966 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:24.966 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T21:59:24.967 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================================================================ 2021-03-15T21:59:24.967 INFO:teuthology.orchestra.run.smithi076.stdout:Install 16 Packages 2021-03-15T21:59:24.967 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:24.968 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 68 M 2021-03-15T21:59:24.968 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 173 M 2021-03-15T21:59:24.968 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T21:59:24.998 INFO:teuthology.orchestra.run.smithi076.stdout:(1/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 9.4 MB/s | 318 kB 00:00 2021-03-15T21:59:25.000 INFO:teuthology.orchestra.run.smithi076.stdout:(2/16): fuse3-3.2.1-12.el8.x86_64.rpm 35 MB/s | 50 kB 00:00 2021-03-15T21:59:25.002 INFO:teuthology.orchestra.run.smithi076.stdout:(3/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T21:59:25.002 INFO:teuthology.orchestra.run.smithi076.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 50 MB/s | 94 kB 00:00 2021-03-15T21:59:25.010 INFO:teuthology.orchestra.run.smithi076.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 473 kB/s | 21 kB 00:00 2021-03-15T21:59:25.045 INFO:teuthology.orchestra.run.smithi076.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.3 MB/s | 59 kB 00:00 2021-03-15T21:59:26.088 INFO:teuthology.orchestra.run.smithi076.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 46 kB/s | 50 kB 00:01 2021-03-15T21:59:26.300 INFO:teuthology.orchestra.run.smithi076.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 223 kB/s | 289 kB 00:01 2021-03-15T21:59:26.426 INFO:teuthology.orchestra.run.smithi076.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 174 kB/s | 59 kB 00:00 2021-03-15T21:59:26.604 INFO:teuthology.orchestra.run.smithi076.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 612 kB/s | 186 kB 00:00 2021-03-15T21:59:26.736 INFO:teuthology.orchestra.run.smithi076.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 233 kB/s | 72 kB 00:00 2021-03-15T21:59:26.912 INFO:teuthology.orchestra.run.smithi076.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 786 kB/s | 241 kB 00:00 2021-03-15T21:59:27.935 INFO:teuthology.orchestra.run.smithi076.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 2.4 MB/s | 2.5 MB 00:01 2021-03-15T21:59:28.260 INFO:teuthology.orchestra.run.smithi076.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T21:59:28.418 INFO:teuthology.orchestra.run.smithi076.stdout:(15/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 14 MB/s | 23 MB 00:01 2021-03-15T21:59:28.420 INFO:teuthology.orchestra.run.smithi076.stdout:(16/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 113 kB/s | 55 kB 00:00 2021-03-15T21:59:28.426 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T21:59:28.426 INFO:teuthology.orchestra.run.smithi076.stdout:Total 20 MB/s | 68 MB 00:03 2021-03-15T21:59:28.442 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T21:59:28.484 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T21:59:28.484 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T21:59:28.964 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T21:59:28.966 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T21:59:29.618 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T21:59:29.731 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T21:59:32.194 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T21:59:32.374 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T21:59:32.500 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T21:59:32.618 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T21:59:32.632 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:59:32.801 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:59:50.568 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:59:50.698 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:59:50.766 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T21:59:50.879 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T21:59:51.038 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T21:59:51.173 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:59:51.196 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T21:59:51.341 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T21:59:51.489 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T21:59:51.615 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T21:59:51.699 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T21:59:51.699 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.VIE4Ky: line 5: 49484 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T21:59:52.005 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:53.820 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:59:53.982 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:59:54.548 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T21:59:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T21:59:54.941 INFO:teuthology.orchestra.run.smithi076.stdout:/var/tmp/rpm-tmp.UgAP3X: line 6: 50434 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T21:59:54.942 INFO:teuthology.orchestra.run.smithi076.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T21:59:54.942 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:54.942 INFO:teuthology.orchestra.run.smithi076.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T21:59:55.050 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T21:59:55.050 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T21:59:55.050 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T21:59:55.051 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T21:59:55.051 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T21:59:55.051 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T21:59:55.051 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T21:59:55.052 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T21:59:55.052 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T21:59:55.052 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T21:59:55.052 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T21:59:55.052 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T21:59:55.053 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T21:59:55.053 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T21:59:55.053 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T21:59:55.631 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T21:59:55.632 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:55.632 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T21:59:55.632 INFO:teuthology.orchestra.run.smithi076.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T21:59:55.632 INFO:teuthology.orchestra.run.smithi076.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T21:59:55.632 INFO:teuthology.orchestra.run.smithi076.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T21:59:55.633 INFO:teuthology.orchestra.run.smithi076.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T21:59:55.633 INFO:teuthology.orchestra.run.smithi076.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T21:59:55.633 INFO:teuthology.orchestra.run.smithi076.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T21:59:55.633 INFO:teuthology.orchestra.run.smithi076.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T21:59:55.634 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T21:59:55.634 INFO:teuthology.orchestra.run.smithi076.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T21:59:55.634 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T21:59:55.634 INFO:teuthology.orchestra.run.smithi076.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T21:59:55.635 INFO:teuthology.orchestra.run.smithi076.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T21:59:55.635 INFO:teuthology.orchestra.run.smithi076.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:59:55.635 INFO:teuthology.orchestra.run.smithi076.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T21:59:55.635 INFO:teuthology.orchestra.run.smithi076.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T21:59:55.635 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T21:59:55.636 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T21:59:55.636 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T21:59:55.839 INFO:teuthology.task.exec:Running commands on role mon.b host ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T21:59:55.840 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T21:59:56.334 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:17:57 ago on Mon 15 Mar 2021 09:41:59 PM UTC. 2021-03-15T21:59:56.797 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T21:59:56.809 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T21:59:56.809 INFO:teuthology.orchestra.run.smithi125.stdout: Package Architecture Version Repository Size 2021-03-15T21:59:56.809 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T21:59:56.809 INFO:teuthology.orchestra.run.smithi125.stdout:Disabling modules: 2021-03-15T21:59:56.810 INFO:teuthology.orchestra.run.smithi125.stdout: container-tools 2021-03-15T21:59:56.810 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:59:56.810 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T21:59:56.810 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T21:59:56.811 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:59:57.024 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T21:59:57.080 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T21:59:57.531 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:17:58 ago on Mon 15 Mar 2021 09:41:59 PM UTC. 2021-03-15T21:59:57.840 INFO:teuthology.orchestra.run.smithi125.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T21:59:57.951 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T21:59:57.952 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T21:59:57.952 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T21:59:58.025 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T21:59:58.310 INFO:teuthology.orchestra.run.smithi125.stdout:Repository successfully enabled. 2021-03-15T21:59:58.316 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T21:59:58.316 INFO:teuthology.orchestra.run.smithi125.stderr:of the main distribution, and quality may vary. 2021-03-15T21:59:58.316 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T21:59:58.317 INFO:teuthology.orchestra.run.smithi125.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T21:59:58.317 INFO:teuthology.orchestra.run.smithi125.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T21:59:58.317 INFO:teuthology.orchestra.run.smithi125.stderr:, 2021-03-15T21:59:58.317 INFO:teuthology.orchestra.run.smithi125.stderr:and packages are not held to any quality or security level. 2021-03-15T21:59:58.318 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T21:59:58.318 INFO:teuthology.orchestra.run.smithi125.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T21:59:58.318 INFO:teuthology.orchestra.run.smithi125.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T21:59:58.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T21:59:58.397 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T21:59:58.397 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-15T21:59:58.710 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1139 0 --:--:-- --:--:-- --:--:-- 1136 2021-03-15T21:59:58.714 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T21:59:59.143 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T21:59:59.146 INFO:teuthology.orchestra.run.smithi125.stdout:============================================================================================================== 2021-03-15T21:59:59.146 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T21:59:59.147 INFO:teuthology.orchestra.run.smithi125.stdout:============================================================================================================== 2021-03-15T21:59:59.147 INFO:teuthology.orchestra.run.smithi125.stdout:Removing: 2021-03-15T21:59:59.147 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 2.2.1-7.module_el8.3.0+699+d61d9c41 @CentOS-AppStream 52 M 2021-03-15T21:59:59.148 INFO:teuthology.orchestra.run.smithi125.stdout:Removing dependent packages: 2021-03-15T21:59:59.148 INFO:teuthology.orchestra.run.smithi125.stdout: podman-docker noarch 2.2.1-7.module_el8.3.0+699+d61d9c41 @CentOS-AppStream 5.5 k 2021-03-15T21:59:59.148 INFO:teuthology.orchestra.run.smithi125.stdout:Removing unused dependencies: 2021-03-15T21:59:59.148 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.6-1.module_el8.1.0+298+41f9343a @CentOS-AppStream 85 k 2021-03-15T21:59:59.148 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.124.0-1.module_el8.1.0+298+41f9343a @CentOS-AppStream 44 k 2021-03-15T21:59:59.149 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.8.3-4.module_el8.1.0+298+41f9343a @CentOS-AppStream 71 M 2021-03-15T21:59:59.149 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common x86_64 1:0.1.40-8.module_el8.1.0+298+41f9343a @CentOS-AppStream 42 k 2021-03-15T21:59:59.149 INFO:teuthology.orchestra.run.smithi125.stdout: criu x86_64 3.12-9.el8 @CentOS-AppStream 1.3 M 2021-03-15T21:59:59.149 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 0.7.2-5.module_el8.1.0+298+41f9343a @CentOS-AppStream 122 k 2021-03-15T21:59:59.150 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T21:59:59.150 INFO:teuthology.orchestra.run.smithi125.stdout: libnet x86_64 1.1.6-15.el8 @CentOS-AppStream 170 k 2021-03-15T21:59:59.150 INFO:teuthology.orchestra.run.smithi125.stdout: libvarlink x86_64 18-3.el8 @CentOS-Base 129 k 2021-03-15T21:59:59.150 INFO:teuthology.orchestra.run.smithi125.stdout: podman-catatonit x86_64 2.2.1-7.module_el8.3.0+699+d61d9c41 @CentOS-AppStream 753 k 2021-03-15T21:59:59.151 INFO:teuthology.orchestra.run.smithi125.stdout: protobuf-c x86_64 1.3.0-4.el8 @CentOS-AppStream 56 k 2021-03-15T21:59:59.151 INFO:teuthology.orchestra.run.smithi125.stdout: runc x86_64 1.0.0-70.rc92.module_el8.3.0+699+d61d9c41 @CentOS-AppStream 13 M 2021-03-15T21:59:59.151 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 0.4.2-3.git21fdece.module_el8.1.0+298+41f9343a @CentOS-AppStream 175 k 2021-03-15T21:59:59.151 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:59:59.151 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T21:59:59.152 INFO:teuthology.orchestra.run.smithi125.stdout:============================================================================================================== 2021-03-15T21:59:59.152 INFO:teuthology.orchestra.run.smithi125.stdout:Remove 15 Packages 2021-03-15T21:59:59.152 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T21:59:59.152 INFO:teuthology.orchestra.run.smithi125.stdout:Freed space: 139 M 2021-03-15T21:59:59.153 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T21:59:59.155 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T21:59:59.155 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T21:59:59.243 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T21:59:59.243 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T21:59:59.441 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T21:59:59.457 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-docker-2.2.1-7.module_el8.3.0+699+d61d9c41.no 1/1 2021-03-15T21:59:59.576 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-docker-2.2.1-7.module_el8.3.0+699+d61d9c41. 1/15 2021-03-15T21:59:59.576 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64 2/15 2021-03-15T21:59:59.845 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64 2/15 2021-03-15T21:59:59.845 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containers-common-1:0.1.40-8.module_el8.1.0+298+41 3/15 2021-03-15T21:59:59.845 INFO:teuthology.orchestra.run.smithi125.stdout:warning: /etc/containers/registries.conf saved as /etc/containers/registries.conf.rpmsave 2021-03-15T21:59:59.846 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:00.035 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : container-selinux-2:2.124.0-1.module_el8.1.0+298+4 4/15 2021-03-15T22:00:16.255 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.124.0-1.module_el8.1.0+298+4 4/15 2021-03-15T22:00:16.384 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-catatonit-2.2.1-7.module_el8.3.0+699+d61d9c 5/15 2021-03-15T22:00:16.562 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-overlayfs-0.7.2-5.module_el8.1.0+298+41f9343a 6/15 2021-03-15T22:00:16.690 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : runc-1.0.0-70.rc92.module_el8.3.0+699+d61d9c41.x86 7/15 2021-03-15T22:00:16.870 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : criu-3.12-9.el8.x86_64 8/15 2021-03-15T22:00:16.870 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libnet-1.1.6-15.el8.x86_64 9/15 2021-03-15T22:00:17.086 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libnet-1.1.6-15.el8.x86_64 9/15 2021-03-15T22:00:17.202 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : protobuf-c-1.3.0-4.el8.x86_64 10/15 2021-03-15T22:00:17.202 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 11/15 2021-03-15T22:00:17.476 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 11/15 2021-03-15T22:00:17.628 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : slirp4netns-0.4.2-3.git21fdece.module_el8.1.0+298+ 12/15 2021-03-15T22:00:17.803 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : conmon-2:2.0.6-1.module_el8.1.0+298+41f9343a.x86_6 13/15 2021-03-15T22:00:17.966 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containernetworking-plugins-0.8.3-4.module_el8.1.0 14/15 2021-03-15T22:00:17.966 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libvarlink-18-3.el8.x86_64 15/15 2021-03-15T22:00:18.422 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libvarlink-18-3.el8.x86_64 15/15 2021-03-15T22:00:18.423 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.6-1.module_el8.1.0+298+41f9343a.x86_6 1/15 2021-03-15T22:00:18.423 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.124.0-1.module_el8.1.0+298+4 2/15 2021-03-15T22:00:18.423 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.8.3-4.module_el8.1.0 3/15 2021-03-15T22:00:18.424 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-1:0.1.40-8.module_el8.1.0+298+41 4/15 2021-03-15T22:00:18.424 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : criu-3.12-9.el8.x86_64 5/15 2021-03-15T22:00:18.424 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-0.7.2-5.module_el8.1.0+298+41f9343a 6/15 2021-03-15T22:00:18.424 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 7/15 2021-03-15T22:00:18.425 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libnet-1.1.6-15.el8.x86_64 8/15 2021-03-15T22:00:18.425 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libvarlink-18-3.el8.x86_64 9/15 2021-03-15T22:00:18.425 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64 10/15 2021-03-15T22:00:18.425 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-catatonit-2.2.1-7.module_el8.3.0+699+d61d9c 11/15 2021-03-15T22:00:18.425 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-docker-2.2.1-7.module_el8.3.0+699+d61d9c41. 12/15 2021-03-15T22:00:18.426 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : protobuf-c-1.3.0-4.el8.x86_64 13/15 2021-03-15T22:00:18.426 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : runc-1.0.0-70.rc92.module_el8.3.0+699+d61d9c41.x86 14/15 2021-03-15T22:00:18.888 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-0.4.2-3.git21fdece.module_el8.1.0+298+ 15/15 2021-03-15T22:00:18.889 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:18.889 INFO:teuthology.orchestra.run.smithi125.stdout:Removed: 2021-03-15T22:00:18.890 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.6-1.module_el8.1.0+298+41f9343a.x86_64 2021-03-15T22:00:18.890 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.124.0-1.module_el8.1.0+298+41f9343a.noarch 2021-03-15T22:00:18.890 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.8.3-4.module_el8.1.0+298+41f9343a.x86_64 2021-03-15T22:00:18.890 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-1:0.1.40-8.module_el8.1.0+298+41f9343a.x86_64 2021-03-15T22:00:18.891 INFO:teuthology.orchestra.run.smithi125.stdout: criu-3.12-9.el8.x86_64 2021-03-15T22:00:18.891 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-0.7.2-5.module_el8.1.0+298+41f9343a.x86_64 2021-03-15T22:00:18.891 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:00:18.892 INFO:teuthology.orchestra.run.smithi125.stdout: libnet-1.1.6-15.el8.x86_64 2021-03-15T22:00:18.892 INFO:teuthology.orchestra.run.smithi125.stdout: libvarlink-18-3.el8.x86_64 2021-03-15T22:00:18.892 INFO:teuthology.orchestra.run.smithi125.stdout: podman-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64 2021-03-15T22:00:18.893 INFO:teuthology.orchestra.run.smithi125.stdout: podman-catatonit-2.2.1-7.module_el8.3.0+699+d61d9c41.x86_64 2021-03-15T22:00:18.893 INFO:teuthology.orchestra.run.smithi125.stdout: podman-docker-2.2.1-7.module_el8.3.0+699+d61d9c41.noarch 2021-03-15T22:00:18.893 INFO:teuthology.orchestra.run.smithi125.stdout: protobuf-c-1.3.0-4.el8.x86_64 2021-03-15T22:00:18.894 INFO:teuthology.orchestra.run.smithi125.stdout: runc-1.0.0-70.rc92.module_el8.3.0+699+d61d9c41.x86_64 2021-03-15T22:00:18.894 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-0.4.2-3.git21fdece.module_el8.1.0+298+41f9343a.x86_64 2021-03-15T22:00:18.894 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:18.895 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:00:19.160 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T22:00:19.664 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 15 kB/s | 1.4 kB 00:00 2021-03-15T22:00:21.065 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 50 kB/s | 68 kB 00:01 2021-03-15T22:00:21.694 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:00:21.697 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:00:21.697 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:00:21.697 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:00:21.698 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:00:21.698 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T22:00:21.698 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:00:21.698 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T22:00:21.699 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T22:00:21.699 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T22:00:21.699 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T22:00:21.699 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T22:00:21.700 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T22:00:21.700 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T22:00:21.700 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T22:00:21.700 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T22:00:21.700 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T22:00:21.701 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T22:00:21.701 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:00:21.701 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T22:00:21.701 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T22:00:21.702 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T22:00:21.702 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T22:00:21.702 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:21.702 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:00:21.703 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:00:21.703 INFO:teuthology.orchestra.run.smithi125.stdout:Install 16 Packages 2021-03-15T22:00:21.703 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:21.704 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 68 M 2021-03-15T22:00:21.704 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 173 M 2021-03-15T22:00:21.704 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:00:21.738 INFO:teuthology.orchestra.run.smithi125.stdout:(1/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T22:00:21.739 INFO:teuthology.orchestra.run.smithi125.stdout:(2/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 7.9 MB/s | 318 kB 00:00 2021-03-15T22:00:21.740 INFO:teuthology.orchestra.run.smithi125.stdout:(3/16): fuse-common-3.2.1-12.el8.x86_64.rpm 532 kB/s | 21 kB 00:00 2021-03-15T22:00:21.741 INFO:teuthology.orchestra.run.smithi125.stdout:(4/16): fuse3-3.2.1-12.el8.x86_64.rpm 21 MB/s | 50 kB 00:00 2021-03-15T22:00:21.742 INFO:teuthology.orchestra.run.smithi125.stdout:(5/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 32 MB/s | 94 kB 00:00 2021-03-15T22:00:21.782 INFO:teuthology.orchestra.run.smithi125.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.4 MB/s | 59 kB 00:00 2021-03-15T22:00:22.805 INFO:teuthology.orchestra.run.smithi125.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 47 kB/s | 50 kB 00:01 2021-03-15T22:00:23.027 INFO:teuthology.orchestra.run.smithi125.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 224 kB/s | 289 kB 00:01 2021-03-15T22:00:23.115 INFO:teuthology.orchestra.run.smithi125.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 189 kB/s | 59 kB 00:00 2021-03-15T22:00:23.417 INFO:teuthology.orchestra.run.smithi125.stdout:(10/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 239 kB/s | 72 kB 00:00 2021-03-15T22:00:23.443 INFO:teuthology.orchestra.run.smithi125.stdout:(11/16): crun-0.18-3.el8.5.2.x86_64.rpm 448 kB/s | 186 kB 00:00 2021-03-15T22:00:23.730 INFO:teuthology.orchestra.run.smithi125.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 773 kB/s | 241 kB 00:00 2021-03-15T22:00:24.359 INFO:teuthology.orchestra.run.smithi125.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 3.9 MB/s | 2.5 MB 00:00 2021-03-15T22:00:24.671 INFO:teuthology.orchestra.run.smithi125.stdout:(14/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 175 kB/s | 55 kB 00:00 2021-03-15T22:00:25.068 INFO:teuthology.orchestra.run.smithi125.stdout:(15/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T22:00:25.304 INFO:teuthology.orchestra.run.smithi125.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 12 MB/s | 23 MB 00:01 2021-03-15T22:00:25.312 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:00:25.312 INFO:teuthology.orchestra.run.smithi125.stdout:Total 19 MB/s | 68 MB 00:03 2021-03-15T22:00:25.317 INFO:teuthology.orchestra.run.smithi125.stderr:warning: /var/cache/dnf/copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux-57cf30cbbab27eef/packages/container-selinux-2.145.0-1.el8.noarch.rpm: Header V3 RSA/SHA1 Signature, key ID 62035f7b: NOKEY 2021-03-15T22:00:25.353 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 29 kB/s | 1.0 kB 00:00 2021-03-15T22:00:25.512 INFO:teuthology.orchestra.run.smithi125.stderr:Importing GPG key 0x62035F7B: 2021-03-15T22:00:25.512 INFO:teuthology.orchestra.run.smithi125.stderr: Userid : "rhcontainerbot_container-selinux (None) " 2021-03-15T22:00:25.513 INFO:teuthology.orchestra.run.smithi125.stderr: Fingerprint: F5D6 6271 4052 49DE A189 569F C907 EBD4 6203 5F7B 2021-03-15T22:00:25.513 INFO:teuthology.orchestra.run.smithi125.stderr: From : https://download.copr.fedorainfracloud.org/results/rhcontainerbot/container-selinux/pubkey.gpg 2021-03-15T22:00:25.583 INFO:teuthology.orchestra.run.smithi125.stdout:Key imported successfully 2021-03-15T22:00:25.584 INFO:teuthology.orchestra.run.smithi125.stderr:warning: /var/cache/dnf/devel_kubic_libcontainers_stable-37b272243bc11f7c/packages/catatonit-0.1.5-6.el8.3.2.x86_64.rpm: Header V3 RSA/SHA256 Signature, key ID 75060aa4: NOKEY 2021-03-15T22:00:25.904 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 3.4 kB/s | 1.1 kB 00:00 2021-03-15T22:00:25.940 INFO:teuthology.orchestra.run.smithi125.stderr:Importing GPG key 0x75060AA4: 2021-03-15T22:00:25.941 INFO:teuthology.orchestra.run.smithi125.stderr: Userid : "devel:kubic OBS Project " 2021-03-15T22:00:25.941 INFO:teuthology.orchestra.run.smithi125.stderr: Fingerprint: 2472 D6D0 D2F6 6AF8 7ABA 8DA3 4D64 3903 7506 0AA4 2021-03-15T22:00:25.942 INFO:teuthology.orchestra.run.smithi125.stderr: From : https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/repodata/repomd.xml.key 2021-03-15T22:00:26.000 INFO:teuthology.orchestra.run.smithi125.stdout:Key imported successfully 2021-03-15T22:00:26.009 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:00:26.052 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:00:26.053 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:00:26.534 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:00:26.536 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:00:27.493 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:00:27.613 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T22:00:30.100 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T22:00:30.256 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T22:00:30.440 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T22:00:30.558 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T22:00:30.573 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:00:30.690 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:00:48.064 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:00:48.246 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:00:48.307 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:00:48.307 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:00:48.308 INFO:teuthology.orchestra.run.smithi125.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2021-03-15T22:00:48.308 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:48.552 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T22:00:48.687 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:00:48.744 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:00:48.949 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T22:00:49.095 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:00:49.230 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T22:00:49.316 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T22:00:49.317 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.AkAQnI: line 5: 23795 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T22:00:49.679 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:51.446 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:00:51.587 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:00:52.150 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T22:00:52.431 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:00:52.431 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.c6Whex: line 6: 24745 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T22:00:52.431 INFO:teuthology.orchestra.run.smithi125.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T22:00:52.432 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:52.432 INFO:teuthology.orchestra.run.smithi125.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T22:00:52.636 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T22:00:52.636 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T22:00:52.637 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T22:00:52.637 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T22:00:52.637 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:00:52.637 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:00:52.637 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T22:00:52.638 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T22:00:52.638 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T22:00:52.638 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T22:00:52.638 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T22:00:52.639 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T22:00:52.639 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:00:52.639 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:00:52.639 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:00:53.141 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T22:00:53.142 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:53.142 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:00:53.143 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:00:53.143 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:00:53.144 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:00:53.144 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:00:53.144 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:00:53.145 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:00:53.145 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:00:53.145 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:00:53.145 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:00:53.146 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:00:53.146 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:00:53.146 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:00:53.146 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:00:53.147 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:00:53.147 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:00:53.147 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:00:53.147 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:53.148 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:00:53.331 INFO:teuthology.task.exec:Running commands on role mgr.x host ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:00:53.331 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T22:00:53.805 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 10:00:21 PM UTC. 2021-03-15T22:00:54.281 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:00:54.282 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:00:54.282 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:00:54.346 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T22:00:54.804 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 10:00:21 PM UTC. 2021-03-15T22:00:55.139 INFO:teuthology.orchestra.run.smithi125.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T22:00:55.251 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:00:55.252 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:00:55.253 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:00:55.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T22:00:55.603 INFO:teuthology.orchestra.run.smithi125.stdout:Repository successfully enabled. 2021-03-15T22:00:55.610 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T22:00:55.610 INFO:teuthology.orchestra.run.smithi125.stderr:of the main distribution, and quality may vary. 2021-03-15T22:00:55.611 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:00:55.611 INFO:teuthology.orchestra.run.smithi125.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T22:00:55.612 INFO:teuthology.orchestra.run.smithi125.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T22:00:55.612 INFO:teuthology.orchestra.run.smithi125.stderr:, 2021-03-15T22:00:55.613 INFO:teuthology.orchestra.run.smithi125.stderr:and packages are not held to any quality or security level. 2021-03-15T22:00:55.613 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:00:55.614 INFO:teuthology.orchestra.run.smithi125.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T22:00:55.614 INFO:teuthology.orchestra.run.smithi125.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T22:00:55.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T22:00:55.678 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T22:00:55.678 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-15T22:00:56.021 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1043 0 --:--:-- --:--:-- --:--:-- 1043 2021-03-15T22:00:56.025 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T22:00:56.495 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:00:56.498 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:00:56.498 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:00:56.498 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:00:56.499 INFO:teuthology.orchestra.run.smithi125.stdout:Removing: 2021-03-15T22:00:56.499 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T22:00:56.499 INFO:teuthology.orchestra.run.smithi125.stdout:Removing unused dependencies: 2021-03-15T22:00:56.499 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T22:00:56.500 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T22:00:56.500 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T22:00:56.500 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T22:00:56.500 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T22:00:56.500 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T22:00:56.501 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T22:00:56.501 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T22:00:56.501 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T22:00:56.501 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T22:00:56.501 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T22:00:56.502 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T22:00:56.502 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T22:00:56.502 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T22:00:56.502 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T22:00:56.503 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:56.503 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:00:56.503 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:00:56.503 INFO:teuthology.orchestra.run.smithi125.stdout:Remove 16 Packages 2021-03-15T22:00:56.503 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:56.504 INFO:teuthology.orchestra.run.smithi125.stdout:Freed space: 173 M 2021-03-15T22:00:56.504 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:00:56.508 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:00:56.509 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:00:56.560 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:00:56.612 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:00:56.773 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:00:56.801 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T22:00:56.802 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:00:57.078 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:00:57.252 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T22:00:57.387 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T22:00:57.512 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T22:00:57.628 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:00:57.762 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:00:57.903 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T22:00:57.904 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:00:57.904 INFO:teuthology.orchestra.run.smithi125.stdout:warning: /etc/fuse.conf saved as /etc/fuse.conf.rpmsave 2021-03-15T22:00:57.904 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:00:58.037 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:01:14.245 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:01:14.359 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T22:01:14.376 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:01:14.377 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:01:14.534 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:01:14.650 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:01:14.750 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:01:14.750 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:01:14.992 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:01:15.127 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T22:01:15.278 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:01:15.495 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:01:15.495 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T22:01:15.495 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T22:01:15.496 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T22:01:15.496 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T22:01:15.496 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T22:01:15.497 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:01:15.498 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T22:01:15.498 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:01:15.499 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T22:01:15.499 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T22:01:15.500 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T22:01:15.500 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T22:01:15.500 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T22:01:15.501 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:01:15.501 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T22:01:15.990 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T22:01:15.991 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:01:15.991 INFO:teuthology.orchestra.run.smithi125.stdout:Removed: 2021-03-15T22:01:15.991 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:01:15.991 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:01:15.992 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:01:15.992 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:01:15.992 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:01:15.992 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:01:15.993 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:01:15.993 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:01:15.993 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:01:15.993 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:01:15.993 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:01:15.994 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:01:15.994 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:01:15.994 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:01:15.994 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:01:15.995 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:01:15.995 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:01:15.995 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:01:16.269 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T22:01:16.725 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 82 kB/s | 3.3 kB 00:00 2021-03-15T22:01:17.052 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 5.1 kB/s | 1.7 kB 00:00 2021-03-15T22:01:17.565 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:01:17.567 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:01:17.567 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:01:17.568 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:01:17.568 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:01:17.568 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T22:01:17.568 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:01:17.569 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T22:01:17.569 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T22:01:17.569 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T22:01:17.569 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T22:01:17.569 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T22:01:17.570 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T22:01:17.570 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T22:01:17.570 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T22:01:17.570 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T22:01:17.571 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T22:01:17.571 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T22:01:17.571 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:01:17.572 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T22:01:17.572 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T22:01:17.572 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T22:01:17.572 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T22:01:17.572 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:01:17.573 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:01:17.573 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:01:17.573 INFO:teuthology.orchestra.run.smithi125.stdout:Install 16 Packages 2021-03-15T22:01:17.573 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:01:17.574 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 68 M 2021-03-15T22:01:17.574 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 173 M 2021-03-15T22:01:17.575 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:01:17.614 INFO:teuthology.orchestra.run.smithi125.stdout:(1/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 7.3 MB/s | 318 kB 00:00 2021-03-15T22:01:17.615 INFO:teuthology.orchestra.run.smithi125.stdout:(2/16): fuse3-3.2.1-12.el8.x86_64.rpm 30 MB/s | 50 kB 00:00 2021-03-15T22:01:17.616 INFO:teuthology.orchestra.run.smithi125.stdout:(3/16): yajl-2.1.0-10.el8.x86_64.rpm 909 kB/s | 41 kB 00:00 2021-03-15T22:01:17.616 INFO:teuthology.orchestra.run.smithi125.stdout:(4/16): fuse-common-3.2.1-12.el8.x86_64.rpm 465 kB/s | 21 kB 00:00 2021-03-15T22:01:17.617 INFO:teuthology.orchestra.run.smithi125.stdout:(5/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 47 MB/s | 94 kB 00:00 2021-03-15T22:01:17.668 INFO:teuthology.orchestra.run.smithi125.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.1 MB/s | 59 kB 00:00 2021-03-15T22:01:18.678 INFO:teuthology.orchestra.run.smithi125.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 47 kB/s | 50 kB 00:01 2021-03-15T22:01:18.870 INFO:teuthology.orchestra.run.smithi125.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 230 kB/s | 289 kB 00:01 2021-03-15T22:01:18.983 INFO:teuthology.orchestra.run.smithi125.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 192 kB/s | 59 kB 00:00 2021-03-15T22:01:19.183 INFO:teuthology.orchestra.run.smithi125.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 595 kB/s | 186 kB 00:00 2021-03-15T22:01:19.287 INFO:teuthology.orchestra.run.smithi125.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 237 kB/s | 72 kB 00:00 2021-03-15T22:01:19.498 INFO:teuthology.orchestra.run.smithi125.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 767 kB/s | 241 kB 00:00 2021-03-15T22:01:20.029 INFO:teuthology.orchestra.run.smithi125.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 4.7 MB/s | 2.5 MB 00:00 2021-03-15T22:01:20.342 INFO:teuthology.orchestra.run.smithi125.stdout:(14/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 175 kB/s | 55 kB 00:00 2021-03-15T22:01:21.005 INFO:teuthology.orchestra.run.smithi125.stdout:(15/16): containernetworking-plugins-0.9.1-4.el 12 MB/s | 41 MB 00:03 2021-03-15T22:01:21.442 INFO:teuthology.orchestra.run.smithi125.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 11 MB/s | 23 MB 00:02 2021-03-15T22:01:21.450 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:01:21.450 INFO:teuthology.orchestra.run.smithi125.stdout:Total 18 MB/s | 68 MB 00:03 2021-03-15T22:01:21.466 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:01:21.510 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:01:21.510 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:01:21.989 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:01:21.991 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:01:22.648 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:01:22.762 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T22:01:25.205 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T22:01:25.363 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T22:01:25.522 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T22:01:25.715 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T22:01:25.730 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:01:25.847 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:01:43.301 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:01:43.470 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:01:43.523 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:01:43.634 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:01:43.768 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T22:01:43.895 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:01:43.919 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:01:44.057 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T22:01:44.277 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:01:44.412 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T22:01:44.505 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T22:01:44.505 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.KOX1AX: line 5: 26394 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T22:01:44.802 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:01:46.610 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:01:46.761 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:01:47.323 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T22:01:47.630 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:01:47.630 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.wB5i2a: line 6: 27352 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T22:01:47.631 INFO:teuthology.orchestra.run.smithi125.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T22:01:47.631 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:01:47.631 INFO:teuthology.orchestra.run.smithi125.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T22:01:47.838 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T22:01:47.839 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T22:01:47.839 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T22:01:47.839 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T22:01:47.839 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:01:47.839 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:01:47.840 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T22:01:47.840 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T22:01:47.840 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T22:01:47.840 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T22:01:47.841 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T22:01:47.841 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T22:01:47.841 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:01:47.841 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:01:47.842 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:01:48.432 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T22:01:48.432 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:01:48.432 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:01:48.433 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:01:48.433 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:01:48.433 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:01:48.433 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:01:48.433 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:01:48.434 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:01:48.434 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:01:48.434 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:01:48.434 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:01:48.435 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:01:48.435 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:01:48.435 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:01:48.435 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:01:48.436 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:01:48.436 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:01:48.436 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:01:48.436 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:01:48.436 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:01:48.616 INFO:teuthology.task.exec:Running commands on role osd.4 host ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:01:48.617 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T22:01:49.084 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 10:01:17 PM UTC. 2021-03-15T22:01:49.565 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:01:49.566 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:01:49.566 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:01:49.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T22:01:50.101 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 10:01:17 PM UTC. 2021-03-15T22:01:50.446 INFO:teuthology.orchestra.run.smithi125.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T22:01:50.560 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:01:50.561 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:01:50.561 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:01:50.639 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T22:01:50.924 INFO:teuthology.orchestra.run.smithi125.stdout:Repository successfully enabled. 2021-03-15T22:01:50.930 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T22:01:50.930 INFO:teuthology.orchestra.run.smithi125.stderr:of the main distribution, and quality may vary. 2021-03-15T22:01:50.931 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:01:50.931 INFO:teuthology.orchestra.run.smithi125.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T22:01:50.931 INFO:teuthology.orchestra.run.smithi125.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T22:01:50.932 INFO:teuthology.orchestra.run.smithi125.stderr:, 2021-03-15T22:01:50.932 INFO:teuthology.orchestra.run.smithi125.stderr:and packages are not held to any quality or security level. 2021-03-15T22:01:50.932 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:01:50.932 INFO:teuthology.orchestra.run.smithi125.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T22:01:50.933 INFO:teuthology.orchestra.run.smithi125.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T22:01:50.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T22:01:50.998 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T22:01:50.998 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-15T22:01:51.314 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1132 0 --:--:-- --:--:-- --:--:-- 1132 2021-03-15T22:01:51.316 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T22:01:51.789 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:01:51.793 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:01:51.793 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:01:51.794 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:01:51.794 INFO:teuthology.orchestra.run.smithi125.stdout:Removing: 2021-03-15T22:01:51.794 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T22:01:51.795 INFO:teuthology.orchestra.run.smithi125.stdout:Removing unused dependencies: 2021-03-15T22:01:51.795 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T22:01:51.795 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T22:01:51.795 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T22:01:51.795 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T22:01:51.796 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T22:01:51.796 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T22:01:51.796 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T22:01:51.796 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T22:01:51.797 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T22:01:51.797 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T22:01:51.797 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T22:01:51.797 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T22:01:51.797 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T22:01:51.798 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T22:01:51.798 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T22:01:51.798 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:01:51.798 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:01:51.799 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:01:51.799 INFO:teuthology.orchestra.run.smithi125.stdout:Remove 16 Packages 2021-03-15T22:01:51.799 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:01:51.799 INFO:teuthology.orchestra.run.smithi125.stdout:Freed space: 173 M 2021-03-15T22:01:51.800 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:01:51.803 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:01:51.803 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:01:51.855 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:01:51.855 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:01:52.065 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:01:52.093 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T22:01:52.094 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:01:52.386 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:01:52.553 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T22:01:52.669 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T22:01:52.795 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T22:01:52.911 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:01:53.036 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:01:53.177 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T22:01:53.308 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:01:53.308 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:02:09.586 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:02:09.708 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T22:02:09.727 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:02:09.728 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:02:09.892 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:02:10.016 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:02:10.116 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:02:10.117 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:02:10.333 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:02:10.485 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T22:02:10.627 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:02:10.842 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:02:10.842 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T22:02:10.843 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T22:02:10.843 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T22:02:10.843 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T22:02:10.843 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T22:02:10.844 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:02:10.844 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T22:02:10.844 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:02:10.844 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T22:02:10.845 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T22:02:10.845 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T22:02:10.845 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T22:02:10.845 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T22:02:10.846 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:02:10.846 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T22:02:11.321 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T22:02:11.321 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:02:11.322 INFO:teuthology.orchestra.run.smithi125.stdout:Removed: 2021-03-15T22:02:11.322 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:02:11.322 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:02:11.323 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:02:11.323 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:02:11.323 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:02:11.323 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:02:11.323 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:02:11.324 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:02:11.324 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:02:11.324 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:02:11.324 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:02:11.325 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:02:11.325 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:02:11.325 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:02:11.325 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:02:11.325 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:02:11.326 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:02:11.326 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:02:11.589 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T22:02:12.041 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 88 kB/s | 3.3 kB 00:00 2021-03-15T22:02:12.358 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 5.3 kB/s | 1.7 kB 00:00 2021-03-15T22:02:12.868 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:02:12.870 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:02:12.871 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:02:12.871 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:02:12.871 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:02:12.872 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T22:02:12.872 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:02:12.873 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T22:02:12.873 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T22:02:12.874 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T22:02:12.874 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T22:02:12.874 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T22:02:12.875 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T22:02:12.875 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T22:02:12.876 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T22:02:12.876 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T22:02:12.877 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T22:02:12.877 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T22:02:12.877 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:02:12.878 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T22:02:12.878 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T22:02:12.879 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T22:02:12.879 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T22:02:12.880 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:02:12.880 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:02:12.880 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:02:12.881 INFO:teuthology.orchestra.run.smithi125.stdout:Install 16 Packages 2021-03-15T22:02:12.881 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:02:12.882 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 68 M 2021-03-15T22:02:12.883 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 173 M 2021-03-15T22:02:12.883 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:02:12.909 INFO:teuthology.orchestra.run.smithi125.stdout:(1/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 8.9 MB/s | 318 kB 00:00 2021-03-15T22:02:12.910 INFO:teuthology.orchestra.run.smithi125.stdout:(2/16): fuse3-3.2.1-12.el8.x86_64.rpm 33 MB/s | 50 kB 00:00 2021-03-15T22:02:12.911 INFO:teuthology.orchestra.run.smithi125.stdout:(3/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T22:02:12.912 INFO:teuthology.orchestra.run.smithi125.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 52 MB/s | 94 kB 00:00 2021-03-15T22:02:12.919 INFO:teuthology.orchestra.run.smithi125.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 470 kB/s | 21 kB 00:00 2021-03-15T22:02:12.951 INFO:teuthology.orchestra.run.smithi125.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.5 MB/s | 59 kB 00:00 2021-03-15T22:02:13.983 INFO:teuthology.orchestra.run.smithi125.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 47 kB/s | 50 kB 00:01 2021-03-15T22:02:14.182 INFO:teuthology.orchestra.run.smithi125.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 227 kB/s | 289 kB 00:01 2021-03-15T22:02:14.288 INFO:teuthology.orchestra.run.smithi125.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 192 kB/s | 59 kB 00:00 2021-03-15T22:02:14.490 INFO:teuthology.orchestra.run.smithi125.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 606 kB/s | 186 kB 00:00 2021-03-15T22:02:14.595 INFO:teuthology.orchestra.run.smithi125.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 235 kB/s | 72 kB 00:00 2021-03-15T22:02:14.793 INFO:teuthology.orchestra.run.smithi125.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 796 kB/s | 241 kB 00:00 2021-03-15T22:02:15.836 INFO:teuthology.orchestra.run.smithi125.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 2.4 MB/s | 2.5 MB 00:01 2021-03-15T22:02:16.043 INFO:teuthology.orchestra.run.smithi125.stdout:(14/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 16 MB/s | 23 MB 00:01 2021-03-15T22:02:16.149 INFO:teuthology.orchestra.run.smithi125.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 175 kB/s | 55 kB 00:00 2021-03-15T22:02:16.510 INFO:teuthology.orchestra.run.smithi125.stdout:(16/16): containernetworking-plugins-0.9.1-4.el 12 MB/s | 41 MB 00:03 2021-03-15T22:02:16.517 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:02:16.517 INFO:teuthology.orchestra.run.smithi125.stdout:Total 19 MB/s | 68 MB 00:03 2021-03-15T22:02:16.533 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:02:16.576 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:02:16.577 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:02:17.055 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:02:17.057 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:02:17.732 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:02:17.908 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T22:02:20.370 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T22:02:20.537 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T22:02:20.679 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T22:02:20.805 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T22:02:20.820 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:02:20.947 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:02:38.491 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:02:38.677 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:02:38.745 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:02:38.875 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:02:39.000 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T22:02:39.136 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:02:39.160 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:02:39.298 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T22:02:39.454 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:02:39.594 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T22:02:39.663 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T22:02:39.663 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.PHFvss: line 5: 28987 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T22:02:39.942 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:02:41.743 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:02:41.885 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:02:42.447 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T22:02:42.743 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:02:42.744 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.WP0PP3: line 6: 29937 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T22:02:42.744 INFO:teuthology.orchestra.run.smithi125.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T22:02:42.744 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:02:42.745 INFO:teuthology.orchestra.run.smithi125.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T22:02:42.943 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T22:02:42.943 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T22:02:42.943 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T22:02:42.943 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T22:02:42.944 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:02:42.944 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:02:42.944 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T22:02:42.944 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T22:02:42.945 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T22:02:42.945 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T22:02:42.945 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T22:02:42.945 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T22:02:42.946 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:02:42.946 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:02:42.946 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:02:43.396 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T22:02:43.396 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:02:43.396 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:02:43.396 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:02:43.397 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:02:43.397 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:02:43.397 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:02:43.397 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:02:43.398 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:02:43.398 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:02:43.398 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:02:43.398 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:02:43.399 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:02:43.399 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:02:43.399 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:02:43.399 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:02:43.400 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:02:43.400 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:02:43.400 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:02:43.400 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:02:43.401 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:02:43.603 INFO:teuthology.task.exec:Running commands on role osd.5 host ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:02:43.604 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T22:02:44.080 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 10:02:12 PM UTC. 2021-03-15T22:02:44.550 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:02:44.552 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:02:44.552 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:02:44.613 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T22:02:45.077 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 10:02:12 PM UTC. 2021-03-15T22:02:45.412 INFO:teuthology.orchestra.run.smithi125.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T22:02:45.523 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:02:45.524 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:02:45.525 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:02:45.585 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T22:02:45.872 INFO:teuthology.orchestra.run.smithi125.stdout:Repository successfully enabled. 2021-03-15T22:02:45.878 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T22:02:45.878 INFO:teuthology.orchestra.run.smithi125.stderr:of the main distribution, and quality may vary. 2021-03-15T22:02:45.878 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:02:45.879 INFO:teuthology.orchestra.run.smithi125.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T22:02:45.879 INFO:teuthology.orchestra.run.smithi125.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T22:02:45.879 INFO:teuthology.orchestra.run.smithi125.stderr:, 2021-03-15T22:02:45.879 INFO:teuthology.orchestra.run.smithi125.stderr:and packages are not held to any quality or security level. 2021-03-15T22:02:45.880 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:02:45.880 INFO:teuthology.orchestra.run.smithi125.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T22:02:45.880 INFO:teuthology.orchestra.run.smithi125.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T22:02:45.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T22:02:45.944 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T22:02:45.945 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-15T22:02:46.269 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1104 0 --:--:-- --:--:-- --:--:-- 1104 2021-03-15T22:02:46.272 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T22:02:46.746 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:02:46.749 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:02:46.750 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:02:46.750 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:02:46.750 INFO:teuthology.orchestra.run.smithi125.stdout:Removing: 2021-03-15T22:02:46.750 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T22:02:46.751 INFO:teuthology.orchestra.run.smithi125.stdout:Removing unused dependencies: 2021-03-15T22:02:46.751 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T22:02:46.751 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T22:02:46.751 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T22:02:46.752 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T22:02:46.752 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T22:02:46.752 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T22:02:46.752 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T22:02:46.753 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T22:02:46.753 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T22:02:46.753 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T22:02:46.753 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T22:02:46.753 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T22:02:46.754 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T22:02:46.754 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T22:02:46.754 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T22:02:46.754 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:02:46.755 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:02:46.755 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:02:46.755 INFO:teuthology.orchestra.run.smithi125.stdout:Remove 16 Packages 2021-03-15T22:02:46.755 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:02:46.756 INFO:teuthology.orchestra.run.smithi125.stdout:Freed space: 173 M 2021-03-15T22:02:46.756 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:02:46.760 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:02:46.761 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:02:46.811 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:02:46.812 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:02:47.019 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:02:47.048 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T22:02:47.049 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:02:47.343 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:02:47.483 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T22:02:47.609 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T22:02:47.726 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T22:02:47.859 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:02:48.027 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:02:48.168 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T22:02:48.276 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:02:48.277 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:03:04.568 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:03:04.682 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T22:03:04.699 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:03:04.700 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:03:04.890 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:03:05.282 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:03:05.390 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:03:05.390 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:03:05.616 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:03:05.750 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T22:03:05.893 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:03:06.133 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:03:06.133 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T22:03:06.134 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T22:03:06.134 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T22:03:06.135 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T22:03:06.135 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T22:03:06.135 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:03:06.135 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T22:03:06.135 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:03:06.136 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T22:03:06.136 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T22:03:06.136 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T22:03:06.137 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T22:03:06.137 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T22:03:06.137 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:03:06.138 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T22:03:06.613 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T22:03:06.614 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:03:06.614 INFO:teuthology.orchestra.run.smithi125.stdout:Removed: 2021-03-15T22:03:06.614 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:03:06.615 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:03:06.615 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:03:06.615 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:03:06.615 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:03:06.616 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:03:06.616 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:03:06.616 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:03:06.617 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:03:06.617 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:03:06.617 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:03:06.618 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:03:06.618 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:03:06.618 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:03:06.619 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:03:06.619 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:03:06.619 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:03:06.619 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:03:06.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T22:03:07.341 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 86 kB/s | 3.3 kB 00:00 2021-03-15T22:03:07.669 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 5.1 kB/s | 1.7 kB 00:00 2021-03-15T22:03:08.176 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:03:08.178 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:03:08.178 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:03:08.178 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:03:08.179 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:03:08.179 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T22:03:08.179 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:03:08.179 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T22:03:08.180 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T22:03:08.180 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T22:03:08.180 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T22:03:08.180 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T22:03:08.181 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T22:03:08.181 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T22:03:08.181 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T22:03:08.181 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T22:03:08.182 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T22:03:08.182 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T22:03:08.182 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:03:08.183 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T22:03:08.183 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T22:03:08.183 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T22:03:08.183 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T22:03:08.183 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:03:08.184 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:03:08.184 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:03:08.184 INFO:teuthology.orchestra.run.smithi125.stdout:Install 16 Packages 2021-03-15T22:03:08.185 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:03:08.186 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 68 M 2021-03-15T22:03:08.186 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 173 M 2021-03-15T22:03:08.186 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:03:08.217 INFO:teuthology.orchestra.run.smithi125.stdout:(1/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 9.0 MB/s | 318 kB 00:00 2021-03-15T22:03:08.218 INFO:teuthology.orchestra.run.smithi125.stdout:(2/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T22:03:08.218 INFO:teuthology.orchestra.run.smithi125.stdout:(3/16): fuse3-3.2.1-12.el8.x86_64.rpm 27 MB/s | 50 kB 00:00 2021-03-15T22:03:08.219 INFO:teuthology.orchestra.run.smithi125.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 45 MB/s | 94 kB 00:00 2021-03-15T22:03:08.221 INFO:teuthology.orchestra.run.smithi125.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 545 kB/s | 21 kB 00:00 2021-03-15T22:03:08.258 INFO:teuthology.orchestra.run.smithi125.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.5 MB/s | 59 kB 00:00 2021-03-15T22:03:09.284 INFO:teuthology.orchestra.run.smithi125.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 47 kB/s | 50 kB 00:01 2021-03-15T22:03:09.507 INFO:teuthology.orchestra.run.smithi125.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 224 kB/s | 289 kB 00:01 2021-03-15T22:03:09.786 INFO:teuthology.orchestra.run.smithi125.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 117 kB/s | 59 kB 00:00 2021-03-15T22:03:09.918 INFO:teuthology.orchestra.run.smithi125.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 453 kB/s | 186 kB 00:00 2021-03-15T22:03:10.191 INFO:teuthology.orchestra.run.smithi125.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 178 kB/s | 72 kB 00:00 2021-03-15T22:03:10.334 INFO:teuthology.orchestra.run.smithi125.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 580 kB/s | 241 kB 00:00 2021-03-15T22:03:11.062 INFO:teuthology.orchestra.run.smithi125.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 3.4 MB/s | 2.5 MB 00:00 2021-03-15T22:03:11.453 INFO:teuthology.orchestra.run.smithi125.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T22:03:11.557 INFO:teuthology.orchestra.run.smithi125.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 111 kB/s | 55 kB 00:00 2021-03-15T22:03:12.007 INFO:teuthology.orchestra.run.smithi125.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 13 MB/s | 23 MB 00:01 2021-03-15T22:03:12.014 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:03:12.014 INFO:teuthology.orchestra.run.smithi125.stdout:Total 18 MB/s | 68 MB 00:03 2021-03-15T22:03:12.030 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:03:12.074 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:03:12.074 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:03:12.553 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:03:12.555 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:03:13.212 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:03:13.360 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T22:03:15.824 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T22:03:15.970 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T22:03:16.120 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T22:03:16.246 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T22:03:16.265 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:03:16.421 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:03:34.413 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:03:34.584 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:03:34.667 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:03:34.807 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:03:34.941 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T22:03:35.076 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:03:35.101 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:03:35.239 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T22:03:35.375 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:03:35.526 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T22:03:35.595 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T22:03:35.595 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.GCrNaa: line 5: 31578 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T22:03:35.874 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:03:37.633 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:03:37.834 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:03:38.397 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T22:03:38.665 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:03:38.665 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.PgGFKr: line 6: 32528 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T22:03:38.666 INFO:teuthology.orchestra.run.smithi125.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T22:03:38.666 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:03:38.666 INFO:teuthology.orchestra.run.smithi125.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T22:03:38.870 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T22:03:38.871 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T22:03:38.871 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T22:03:38.872 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T22:03:38.872 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:03:38.873 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:03:38.873 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T22:03:38.874 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T22:03:38.875 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T22:03:38.875 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T22:03:38.875 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T22:03:38.876 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T22:03:38.876 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:03:38.876 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:03:38.876 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:03:39.301 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T22:03:39.302 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:03:39.302 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:03:39.302 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:03:39.303 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:03:39.303 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:03:39.303 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:03:39.304 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:03:39.304 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:03:39.304 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:03:39.304 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:03:39.305 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:03:39.305 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:03:39.305 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:03:39.305 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:03:39.306 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:03:39.306 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:03:39.306 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:03:39.306 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:03:39.307 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:03:39.307 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:03:39.534 INFO:teuthology.task.exec:Running commands on role osd.6 host ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:03:39.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T22:03:40.005 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 10:03:07 PM UTC. 2021-03-15T22:03:40.486 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:03:40.487 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:03:40.487 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:03:40.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T22:03:41.013 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 10:03:07 PM UTC. 2021-03-15T22:03:41.352 INFO:teuthology.orchestra.run.smithi125.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T22:03:41.465 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:03:41.466 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:03:41.467 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:03:41.528 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T22:03:41.826 INFO:teuthology.orchestra.run.smithi125.stdout:Repository successfully enabled. 2021-03-15T22:03:41.833 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T22:03:41.834 INFO:teuthology.orchestra.run.smithi125.stderr:of the main distribution, and quality may vary. 2021-03-15T22:03:41.834 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:03:41.835 INFO:teuthology.orchestra.run.smithi125.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T22:03:41.835 INFO:teuthology.orchestra.run.smithi125.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T22:03:41.836 INFO:teuthology.orchestra.run.smithi125.stderr:, 2021-03-15T22:03:41.836 INFO:teuthology.orchestra.run.smithi125.stderr:and packages are not held to any quality or security level. 2021-03-15T22:03:41.837 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:03:41.837 INFO:teuthology.orchestra.run.smithi125.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T22:03:41.837 INFO:teuthology.orchestra.run.smithi125.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T22:03:41.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T22:03:41.904 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T22:03:41.904 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-15T22:03:42.234 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1084 0 --:--:-- --:--:-- --:--:-- 1084 2021-03-15T22:03:42.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T22:03:42.711 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:03:42.715 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:03:42.715 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:03:42.715 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:03:42.715 INFO:teuthology.orchestra.run.smithi125.stdout:Removing: 2021-03-15T22:03:42.716 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T22:03:42.716 INFO:teuthology.orchestra.run.smithi125.stdout:Removing unused dependencies: 2021-03-15T22:03:42.716 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T22:03:42.716 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T22:03:42.717 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T22:03:42.717 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T22:03:42.717 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T22:03:42.717 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T22:03:42.717 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T22:03:42.718 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T22:03:42.718 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T22:03:42.718 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T22:03:42.718 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T22:03:42.719 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T22:03:42.719 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T22:03:42.719 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T22:03:42.719 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T22:03:42.719 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:03:42.720 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:03:42.720 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:03:42.720 INFO:teuthology.orchestra.run.smithi125.stdout:Remove 16 Packages 2021-03-15T22:03:42.720 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:03:42.721 INFO:teuthology.orchestra.run.smithi125.stdout:Freed space: 173 M 2021-03-15T22:03:42.721 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:03:42.725 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:03:42.725 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:03:42.776 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:03:42.777 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:03:42.980 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:03:43.008 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T22:03:43.009 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:03:43.316 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:03:43.456 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T22:03:43.583 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T22:03:43.717 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T22:03:43.841 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:03:44.009 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:03:44.166 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T22:03:44.284 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:03:44.284 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:04:00.726 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:04:00.839 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T22:04:00.908 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:04:00.908 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:04:01.039 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:04:01.164 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:04:01.280 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:04:01.281 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:04:01.497 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:04:01.641 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T22:04:01.808 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:04:02.033 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:04:02.033 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T22:04:02.034 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T22:04:02.034 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T22:04:02.034 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T22:04:02.035 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T22:04:02.035 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:04:02.035 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T22:04:02.036 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:04:02.036 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T22:04:02.036 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T22:04:02.036 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T22:04:02.037 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T22:04:02.037 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T22:04:02.037 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:04:02.037 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T22:04:02.503 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T22:04:02.504 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:02.504 INFO:teuthology.orchestra.run.smithi125.stdout:Removed: 2021-03-15T22:04:02.504 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:04:02.504 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:04:02.504 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:04:02.505 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:04:02.505 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:04:02.505 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:04:02.505 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:04:02.506 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:04:02.506 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:04:02.506 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:04:02.506 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:04:02.506 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:04:02.507 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:04:02.507 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:04:02.507 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:04:02.507 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:04:02.508 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:02.508 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:04:02.776 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T22:04:03.225 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 88 kB/s | 3.3 kB 00:00 2021-03-15T22:04:03.554 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 5.1 kB/s | 1.7 kB 00:00 2021-03-15T22:04:04.072 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:04:04.074 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:04:04.074 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:04:04.075 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:04:04.075 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:04:04.075 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T22:04:04.075 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:04:04.075 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T22:04:04.076 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T22:04:04.076 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T22:04:04.076 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T22:04:04.076 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T22:04:04.077 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T22:04:04.077 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T22:04:04.077 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T22:04:04.077 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T22:04:04.077 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T22:04:04.078 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T22:04:04.078 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:04:04.078 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T22:04:04.078 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T22:04:04.079 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T22:04:04.079 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T22:04:04.079 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:04.079 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:04:04.079 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:04:04.080 INFO:teuthology.orchestra.run.smithi125.stdout:Install 16 Packages 2021-03-15T22:04:04.080 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:04.081 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 68 M 2021-03-15T22:04:04.081 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 173 M 2021-03-15T22:04:04.081 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:04:04.154 INFO:teuthology.orchestra.run.smithi125.stdout:(1/16): yajl-2.1.0-10.el8.x86_64.rpm 535 kB/s | 41 kB 00:00 2021-03-15T22:04:04.165 INFO:teuthology.orchestra.run.smithi125.stdout:(2/16): fuse-common-3.2.1-12.el8.x86_64.rpm 244 kB/s | 21 kB 00:00 2021-03-15T22:04:04.175 INFO:teuthology.orchestra.run.smithi125.stdout:(3/16): fuse3-3.2.1-12.el8.x86_64.rpm 2.4 MB/s | 50 kB 00:00 2021-03-15T22:04:04.178 INFO:teuthology.orchestra.run.smithi125.stdout:(4/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 3.1 MB/s | 318 kB 00:00 2021-03-15T22:04:04.179 INFO:teuthology.orchestra.run.smithi125.stdout:(5/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 6.4 MB/s | 94 kB 00:00 2021-03-15T22:04:04.217 INFO:teuthology.orchestra.run.smithi125.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.4 MB/s | 59 kB 00:00 2021-03-15T22:04:05.266 INFO:teuthology.orchestra.run.smithi125.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 46 kB/s | 50 kB 00:01 2021-03-15T22:04:05.465 INFO:teuthology.orchestra.run.smithi125.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 224 kB/s | 289 kB 00:01 2021-03-15T22:04:05.685 INFO:teuthology.orchestra.run.smithi125.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 140 kB/s | 59 kB 00:00 2021-03-15T22:04:05.768 INFO:teuthology.orchestra.run.smithi125.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 616 kB/s | 186 kB 00:00 2021-03-15T22:04:06.093 INFO:teuthology.orchestra.run.smithi125.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 177 kB/s | 72 kB 00:00 2021-03-15T22:04:06.181 INFO:teuthology.orchestra.run.smithi125.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 584 kB/s | 241 kB 00:00 2021-03-15T22:04:06.704 INFO:teuthology.orchestra.run.smithi125.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 4.7 MB/s | 2.5 MB 00:00 2021-03-15T22:04:07.022 INFO:teuthology.orchestra.run.smithi125.stdout:(14/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 172 kB/s | 55 kB 00:00 2021-03-15T22:04:07.569 INFO:teuthology.orchestra.run.smithi125.stdout:(15/16): containernetworking-plugins-0.9.1-4.el 12 MB/s | 41 MB 00:03 2021-03-15T22:04:09.178 INFO:teuthology.orchestra.run.smithi125.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 7.5 MB/s | 23 MB 00:03 2021-03-15T22:04:09.186 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:04:09.186 INFO:teuthology.orchestra.run.smithi125.stdout:Total 13 MB/s | 68 MB 00:05 2021-03-15T22:04:09.201 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:04:09.245 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:04:09.246 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:04:09.726 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:04:09.728 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:04:10.435 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:04:10.558 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T22:04:13.001 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T22:04:13.159 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T22:04:13.335 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T22:04:13.461 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T22:04:13.475 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:04:13.594 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:04:31.071 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:04:31.233 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:04:31.302 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:04:31.430 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:04:31.572 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T22:04:31.733 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:04:31.756 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:04:31.903 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T22:04:32.048 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:04:32.175 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T22:04:32.251 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T22:04:32.251 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.2BrTDN: line 5: 34169 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T22:04:32.576 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:34.349 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:04:34.491 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:04:35.051 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T22:04:35.331 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:04:35.331 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.j5an71: line 6: 35119 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T22:04:35.331 INFO:teuthology.orchestra.run.smithi125.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T22:04:35.331 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:35.332 INFO:teuthology.orchestra.run.smithi125.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T22:04:35.530 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T22:04:35.530 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T22:04:35.530 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T22:04:35.530 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T22:04:35.531 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:04:35.531 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:04:35.531 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T22:04:35.531 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T22:04:35.532 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T22:04:35.532 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T22:04:35.532 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T22:04:35.532 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T22:04:35.532 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:04:35.533 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:04:35.533 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:04:35.995 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T22:04:35.995 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:35.995 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:04:35.996 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:04:35.996 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:04:35.996 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:04:35.996 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:04:35.996 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:04:35.997 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:04:35.997 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:04:35.997 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:04:35.997 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:04:35.998 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:04:35.998 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:04:35.998 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:04:35.998 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:04:35.998 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:04:35.999 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:04:35.999 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:04:35.999 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:35.999 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:04:36.227 INFO:teuthology.task.exec:Running commands on role osd.7 host ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:04:36.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T22:04:36.706 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 10:04:03 PM UTC. 2021-03-15T22:04:37.213 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:04:37.214 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:04:37.215 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:04:37.283 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T22:04:37.746 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:34 ago on Mon 15 Mar 2021 10:04:03 PM UTC. 2021-03-15T22:04:38.076 INFO:teuthology.orchestra.run.smithi125.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T22:04:38.186 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:04:38.187 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:04:38.187 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:04:38.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T22:04:38.556 INFO:teuthology.orchestra.run.smithi125.stdout:Repository successfully enabled. 2021-03-15T22:04:38.562 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T22:04:38.563 INFO:teuthology.orchestra.run.smithi125.stderr:of the main distribution, and quality may vary. 2021-03-15T22:04:38.563 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:04:38.563 INFO:teuthology.orchestra.run.smithi125.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T22:04:38.563 INFO:teuthology.orchestra.run.smithi125.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T22:04:38.564 INFO:teuthology.orchestra.run.smithi125.stderr:, 2021-03-15T22:04:38.564 INFO:teuthology.orchestra.run.smithi125.stderr:and packages are not held to any quality or security level. 2021-03-15T22:04:38.564 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:04:38.564 INFO:teuthology.orchestra.run.smithi125.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T22:04:38.565 INFO:teuthology.orchestra.run.smithi125.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T22:04:38.595 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T22:04:38.631 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T22:04:38.632 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-15T22:04:38.953 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1118 0 --:--:-- --:--:-- --:--:-- 1118 2021-03-15T22:04:38.956 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T22:04:39.430 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:04:39.433 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:04:39.434 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:04:39.434 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:04:39.434 INFO:teuthology.orchestra.run.smithi125.stdout:Removing: 2021-03-15T22:04:39.434 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T22:04:39.435 INFO:teuthology.orchestra.run.smithi125.stdout:Removing unused dependencies: 2021-03-15T22:04:39.435 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T22:04:39.435 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T22:04:39.435 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T22:04:39.436 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T22:04:39.436 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T22:04:39.436 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T22:04:39.436 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T22:04:39.436 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T22:04:39.437 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T22:04:39.437 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T22:04:39.437 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T22:04:39.437 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T22:04:39.438 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T22:04:39.438 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T22:04:39.438 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T22:04:39.438 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:39.438 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:04:39.439 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:04:39.439 INFO:teuthology.orchestra.run.smithi125.stdout:Remove 16 Packages 2021-03-15T22:04:39.439 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:39.440 INFO:teuthology.orchestra.run.smithi125.stdout:Freed space: 173 M 2021-03-15T22:04:39.440 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:04:39.444 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:04:39.444 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:04:39.496 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:04:39.496 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:04:39.703 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:04:39.731 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T22:04:39.732 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:04:40.023 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:04:40.188 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T22:04:40.315 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T22:04:40.440 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T22:04:40.590 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:04:40.716 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:04:40.890 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T22:04:41.007 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:04:41.008 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:04:57.549 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:04:57.662 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T22:04:57.680 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:04:57.681 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:04:57.871 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:04:57.987 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:04:58.104 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:04:58.104 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:04:58.313 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:04:58.456 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T22:04:58.640 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:04:58.861 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:04:58.861 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T22:04:58.862 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T22:04:58.862 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T22:04:58.862 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T22:04:58.862 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T22:04:58.863 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:04:58.863 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T22:04:58.863 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:04:58.863 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T22:04:58.864 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T22:04:58.864 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T22:04:58.864 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T22:04:58.864 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T22:04:58.865 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:04:58.865 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T22:04:59.331 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T22:04:59.331 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:59.332 INFO:teuthology.orchestra.run.smithi125.stdout:Removed: 2021-03-15T22:04:59.332 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:04:59.332 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:04:59.332 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:04:59.333 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:04:59.333 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:04:59.333 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:04:59.333 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:04:59.333 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:04:59.334 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:04:59.334 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:04:59.334 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:04:59.334 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:04:59.334 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:04:59.335 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:04:59.335 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:04:59.335 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:04:59.335 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:04:59.336 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:04:59.588 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T22:05:00.046 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 89 kB/s | 3.3 kB 00:00 2021-03-15T22:05:00.376 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 5.1 kB/s | 1.7 kB 00:00 2021-03-15T22:05:00.887 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:05:00.889 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:05:00.889 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:05:00.890 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:05:00.890 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:05:00.890 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T22:05:00.890 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:05:00.891 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T22:05:00.891 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T22:05:00.891 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T22:05:00.891 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T22:05:00.891 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T22:05:00.892 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T22:05:00.892 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T22:05:00.892 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T22:05:00.892 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T22:05:00.892 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T22:05:00.893 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T22:05:00.893 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:05:00.893 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T22:05:00.893 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T22:05:00.894 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T22:05:00.894 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T22:05:00.894 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:00.894 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:05:00.894 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:05:00.895 INFO:teuthology.orchestra.run.smithi125.stdout:Install 16 Packages 2021-03-15T22:05:00.895 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:00.896 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 68 M 2021-03-15T22:05:00.896 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 173 M 2021-03-15T22:05:00.896 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:05:00.958 INFO:teuthology.orchestra.run.smithi125.stdout:(1/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 4.7 MB/s | 318 kB 00:00 2021-03-15T22:05:00.959 INFO:teuthology.orchestra.run.smithi125.stdout:(2/16): yajl-2.1.0-10.el8.x86_64.rpm 619 kB/s | 41 kB 00:00 2021-03-15T22:05:00.960 INFO:teuthology.orchestra.run.smithi125.stdout:(3/16): fuse3-3.2.1-12.el8.x86_64.rpm 27 MB/s | 50 kB 00:00 2021-03-15T22:05:00.961 INFO:teuthology.orchestra.run.smithi125.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 39 MB/s | 94 kB 00:00 2021-03-15T22:05:00.963 INFO:teuthology.orchestra.run.smithi125.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 299 kB/s | 21 kB 00:00 2021-03-15T22:05:01.001 INFO:teuthology.orchestra.run.smithi125.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.4 MB/s | 59 kB 00:00 2021-03-15T22:05:02.003 INFO:teuthology.orchestra.run.smithi125.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 48 kB/s | 50 kB 00:01 2021-03-15T22:05:02.267 INFO:teuthology.orchestra.run.smithi125.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 221 kB/s | 289 kB 00:01 2021-03-15T22:05:02.413 INFO:teuthology.orchestra.run.smithi125.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 143 kB/s | 59 kB 00:00 2021-03-15T22:05:02.828 INFO:teuthology.orchestra.run.smithi125.stdout:(10/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 174 kB/s | 72 kB 00:00 2021-03-15T22:05:02.928 INFO:teuthology.orchestra.run.smithi125.stdout:(11/16): crun-0.18-3.el8.5.2.x86_64.rpm 282 kB/s | 186 kB 00:00 2021-03-15T22:05:03.248 INFO:teuthology.orchestra.run.smithi125.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 574 kB/s | 241 kB 00:00 2021-03-15T22:05:03.779 INFO:teuthology.orchestra.run.smithi125.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 4.7 MB/s | 2.5 MB 00:00 2021-03-15T22:05:04.379 INFO:teuthology.orchestra.run.smithi125.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 12 MB/s | 41 MB 00:03 2021-03-15T22:05:04.380 INFO:teuthology.orchestra.run.smithi125.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 91 kB/s | 55 kB 00:00 2021-03-15T22:05:04.883 INFO:teuthology.orchestra.run.smithi125.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 12 MB/s | 23 MB 00:01 2021-03-15T22:05:04.890 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:05:04.890 INFO:teuthology.orchestra.run.smithi125.stdout:Total 17 MB/s | 68 MB 00:03 2021-03-15T22:05:04.905 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:05:04.947 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:05:04.948 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:05:05.428 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:05:05.429 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:05:06.087 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:05:06.232 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T22:05:08.673 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T22:05:08.841 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T22:05:08.984 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T22:05:09.110 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T22:05:09.124 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:05:09.285 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:05:26.760 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:05:26.948 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:05:27.002 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:05:27.121 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:05:27.246 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T22:05:27.373 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:05:27.396 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:05:27.544 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T22:05:27.713 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:05:27.849 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T22:05:27.921 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T22:05:27.922 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.Kt13Wk: line 5: 36759 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T22:05:28.206 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:29.981 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:05:30.148 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:05:30.712 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T22:05:30.999 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:05:31.000 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.Jfx7z9: line 6: 37709 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T22:05:31.000 INFO:teuthology.orchestra.run.smithi125.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T22:05:31.001 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:31.001 INFO:teuthology.orchestra.run.smithi125.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T22:05:31.193 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T22:05:31.194 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T22:05:31.194 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T22:05:31.194 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T22:05:31.194 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:05:31.195 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:05:31.195 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T22:05:31.195 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T22:05:31.195 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T22:05:31.195 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T22:05:31.196 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T22:05:31.196 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T22:05:31.196 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:05:31.197 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:05:31.197 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:05:31.686 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T22:05:31.686 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:31.686 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:05:31.687 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:05:31.687 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:05:31.687 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:05:31.687 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:05:31.688 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:05:31.688 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:05:31.688 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:05:31.688 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:05:31.688 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:05:31.689 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:05:31.689 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:05:31.689 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:05:31.689 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:05:31.689 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:05:31.690 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:05:31.690 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:05:31.690 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:31.690 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:05:31.867 INFO:teuthology.task.exec:Running commands on role client.1 host ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:05:31.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T22:05:32.339 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 10:05:00 PM UTC. 2021-03-15T22:05:32.819 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:05:32.820 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:05:32.820 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:05:32.879 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T22:05:33.341 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 10:05:00 PM UTC. 2021-03-15T22:05:33.675 INFO:teuthology.orchestra.run.smithi125.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T22:05:33.785 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:05:33.786 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:05:33.787 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:05:33.854 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T22:05:34.163 INFO:teuthology.orchestra.run.smithi125.stdout:Repository successfully enabled. 2021-03-15T22:05:34.169 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T22:05:34.169 INFO:teuthology.orchestra.run.smithi125.stderr:of the main distribution, and quality may vary. 2021-03-15T22:05:34.170 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:05:34.170 INFO:teuthology.orchestra.run.smithi125.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T22:05:34.170 INFO:teuthology.orchestra.run.smithi125.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T22:05:34.170 INFO:teuthology.orchestra.run.smithi125.stderr:, 2021-03-15T22:05:34.171 INFO:teuthology.orchestra.run.smithi125.stderr:and packages are not held to any quality or security level. 2021-03-15T22:05:34.171 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:05:34.171 INFO:teuthology.orchestra.run.smithi125.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T22:05:34.171 INFO:teuthology.orchestra.run.smithi125.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T22:05:34.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T22:05:34.236 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T22:05:34.236 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-15T22:05:34.584 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1028 0 --:--:-- --:--:-- --:--:-- 1025 2021-03-15T22:05:34.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T22:05:35.059 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:05:35.063 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:05:35.063 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:05:35.063 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:05:35.063 INFO:teuthology.orchestra.run.smithi125.stdout:Removing: 2021-03-15T22:05:35.064 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T22:05:35.064 INFO:teuthology.orchestra.run.smithi125.stdout:Removing unused dependencies: 2021-03-15T22:05:35.064 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T22:05:35.064 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T22:05:35.064 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T22:05:35.065 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T22:05:35.065 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T22:05:35.065 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T22:05:35.065 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T22:05:35.065 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T22:05:35.066 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T22:05:35.066 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T22:05:35.066 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T22:05:35.066 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T22:05:35.067 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T22:05:35.067 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T22:05:35.067 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T22:05:35.067 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:35.067 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:05:35.068 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:05:35.068 INFO:teuthology.orchestra.run.smithi125.stdout:Remove 16 Packages 2021-03-15T22:05:35.068 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:35.069 INFO:teuthology.orchestra.run.smithi125.stdout:Freed space: 173 M 2021-03-15T22:05:35.069 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:05:35.073 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:05:35.073 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:05:35.125 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:05:35.125 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:05:35.331 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:05:35.360 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T22:05:35.360 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:05:35.647 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:05:35.795 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T22:05:35.914 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T22:05:36.031 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T22:05:36.164 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:05:36.290 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:05:36.431 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T22:05:36.531 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:05:36.531 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:05:53.281 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:05:53.394 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T22:05:53.411 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:05:53.412 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:05:53.570 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:05:53.711 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:05:53.811 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:05:53.812 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:05:54.062 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:05:54.206 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T22:05:54.364 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:05:54.587 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:05:54.587 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T22:05:54.587 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T22:05:54.588 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T22:05:54.588 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T22:05:54.588 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T22:05:54.588 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:05:54.589 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T22:05:54.589 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:05:54.589 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T22:05:54.589 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T22:05:54.590 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T22:05:54.590 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T22:05:54.590 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T22:05:54.590 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:05:54.591 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T22:05:55.051 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T22:05:55.051 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:55.052 INFO:teuthology.orchestra.run.smithi125.stdout:Removed: 2021-03-15T22:05:55.052 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:05:55.052 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:05:55.052 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:05:55.052 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:05:55.053 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:05:55.053 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:05:55.053 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:05:55.053 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:05:55.053 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:05:55.054 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:05:55.054 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:05:55.054 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:05:55.054 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:05:55.055 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:05:55.055 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:05:55.055 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:05:55.055 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:55.055 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:05:55.340 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T22:05:55.791 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 91 kB/s | 3.3 kB 00:00 2021-03-15T22:05:56.109 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 5.3 kB/s | 1.7 kB 00:00 2021-03-15T22:05:56.629 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:05:56.631 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:05:56.631 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:05:56.632 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:05:56.632 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:05:56.632 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T22:05:56.632 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:05:56.633 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T22:05:56.633 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T22:05:56.633 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T22:05:56.633 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T22:05:56.633 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T22:05:56.634 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T22:05:56.634 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T22:05:56.634 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T22:05:56.635 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T22:05:56.635 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T22:05:56.635 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T22:05:56.635 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:05:56.636 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T22:05:56.636 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T22:05:56.636 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T22:05:56.636 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T22:05:56.636 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:56.637 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:05:56.637 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:05:56.637 INFO:teuthology.orchestra.run.smithi125.stdout:Install 16 Packages 2021-03-15T22:05:56.637 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:05:56.638 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 68 M 2021-03-15T22:05:56.638 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 173 M 2021-03-15T22:05:56.639 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:05:56.755 INFO:teuthology.orchestra.run.smithi125.stdout:(1/16): yajl-2.1.0-10.el8.x86_64.rpm 339 kB/s | 41 kB 00:00 2021-03-15T22:05:56.757 INFO:teuthology.orchestra.run.smithi125.stdout:(2/16): fuse-common-3.2.1-12.el8.x86_64.rpm 172 kB/s | 21 kB 00:00 2021-03-15T22:05:56.784 INFO:teuthology.orchestra.run.smithi125.stdout:(3/16): fuse3-3.2.1-12.el8.x86_64.rpm 1.7 MB/s | 50 kB 00:00 2021-03-15T22:05:56.787 INFO:teuthology.orchestra.run.smithi125.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 3.1 MB/s | 94 kB 00:00 2021-03-15T22:05:56.792 INFO:teuthology.orchestra.run.smithi125.stdout:(5/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 2.0 MB/s | 318 kB 00:00 2021-03-15T22:05:56.826 INFO:teuthology.orchestra.run.smithi125.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.4 MB/s | 59 kB 00:00 2021-03-15T22:05:57.865 INFO:teuthology.orchestra.run.smithi125.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 46 kB/s | 50 kB 00:01 2021-03-15T22:05:58.127 INFO:teuthology.orchestra.run.smithi125.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 215 kB/s | 289 kB 00:01 2021-03-15T22:05:58.279 INFO:teuthology.orchestra.run.smithi125.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 142 kB/s | 59 kB 00:00 2021-03-15T22:05:58.434 INFO:teuthology.orchestra.run.smithi125.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 608 kB/s | 186 kB 00:00 2021-03-15T22:05:58.588 INFO:teuthology.orchestra.run.smithi125.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 234 kB/s | 72 kB 00:00 2021-03-15T22:05:58.749 INFO:teuthology.orchestra.run.smithi125.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 768 kB/s | 241 kB 00:00 2021-03-15T22:05:59.777 INFO:teuthology.orchestra.run.smithi125.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 2.4 MB/s | 2.5 MB 00:01 2021-03-15T22:06:00.178 INFO:teuthology.orchestra.run.smithi125.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 12 MB/s | 41 MB 00:03 2021-03-15T22:06:00.284 INFO:teuthology.orchestra.run.smithi125.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 108 kB/s | 55 kB 00:00 2021-03-15T22:06:00.465 INFO:teuthology.orchestra.run.smithi125.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 12 MB/s | 23 MB 00:01 2021-03-15T22:06:00.471 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:06:00.472 INFO:teuthology.orchestra.run.smithi125.stdout:Total 18 MB/s | 68 MB 00:03 2021-03-15T22:06:00.487 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:06:00.530 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:06:00.531 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:06:01.011 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:06:01.013 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:06:01.675 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:06:01.798 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T22:06:04.246 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T22:06:04.398 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T22:06:04.549 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T22:06:04.701 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T22:06:04.715 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:06:04.842 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:06:22.329 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:06:22.497 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:06:22.568 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:06:22.678 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:06:22.812 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T22:06:22.939 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:06:22.962 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:06:23.160 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T22:06:23.304 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:06:23.431 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T22:06:23.486 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T22:06:23.486 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.3z1uBE: line 5: 39351 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T22:06:23.789 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:06:25.597 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:06:25.764 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:06:26.325 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T22:06:26.632 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:06:26.633 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.Mh7aC1: line 6: 40301 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T22:06:26.633 INFO:teuthology.orchestra.run.smithi125.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T22:06:26.633 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:06:26.634 INFO:teuthology.orchestra.run.smithi125.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T22:06:26.832 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T22:06:26.832 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T22:06:26.832 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T22:06:26.833 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T22:06:26.833 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:06:26.833 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:06:26.833 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T22:06:26.833 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T22:06:26.834 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T22:06:26.834 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T22:06:26.834 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T22:06:26.834 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T22:06:26.835 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:06:26.835 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:06:26.835 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:06:27.384 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T22:06:27.385 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:06:27.385 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:06:27.385 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:06:27.385 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:06:27.385 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:06:27.386 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:06:27.386 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:06:27.386 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:06:27.386 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:06:27.387 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:06:27.387 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:06:27.387 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:06:27.387 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:06:27.388 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:06:27.388 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:06:27.388 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:06:27.389 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:06:27.389 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:06:27.389 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:06:27.389 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:06:27.569 INFO:teuthology.task.exec:Running commands on role prometheus.a host ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:06:27.569 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T22:06:28.040 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:31 ago on Mon 15 Mar 2021 10:05:56 PM UTC. 2021-03-15T22:06:28.525 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:06:28.526 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:06:28.526 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:06:28.586 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T22:06:29.050 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 10:05:56 PM UTC. 2021-03-15T22:06:29.390 INFO:teuthology.orchestra.run.smithi125.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T22:06:29.501 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:06:29.502 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:06:29.502 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:06:29.571 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T22:06:29.861 INFO:teuthology.orchestra.run.smithi125.stdout:Repository successfully enabled. 2021-03-15T22:06:29.868 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T22:06:29.868 INFO:teuthology.orchestra.run.smithi125.stderr:of the main distribution, and quality may vary. 2021-03-15T22:06:29.868 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:06:29.869 INFO:teuthology.orchestra.run.smithi125.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T22:06:29.869 INFO:teuthology.orchestra.run.smithi125.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T22:06:29.869 INFO:teuthology.orchestra.run.smithi125.stderr:, 2021-03-15T22:06:29.869 INFO:teuthology.orchestra.run.smithi125.stderr:and packages are not held to any quality or security level. 2021-03-15T22:06:29.870 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:06:29.870 INFO:teuthology.orchestra.run.smithi125.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T22:06:29.870 INFO:teuthology.orchestra.run.smithi125.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T22:06:29.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T22:06:29.938 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T22:06:29.939 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-15T22:06:30.274 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1068 0 --:--:-- --:--:-- --:--:-- 1068 2021-03-15T22:06:30.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T22:06:30.749 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:06:30.752 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:06:30.752 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:06:30.752 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:06:30.753 INFO:teuthology.orchestra.run.smithi125.stdout:Removing: 2021-03-15T22:06:30.753 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T22:06:30.753 INFO:teuthology.orchestra.run.smithi125.stdout:Removing unused dependencies: 2021-03-15T22:06:30.754 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T22:06:30.754 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T22:06:30.755 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T22:06:30.755 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T22:06:30.755 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T22:06:30.755 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T22:06:30.755 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T22:06:30.756 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T22:06:30.756 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T22:06:30.756 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T22:06:30.756 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T22:06:30.757 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T22:06:30.757 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T22:06:30.757 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T22:06:30.757 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T22:06:30.757 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:06:30.758 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:06:30.758 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:06:30.758 INFO:teuthology.orchestra.run.smithi125.stdout:Remove 16 Packages 2021-03-15T22:06:30.758 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:06:30.759 INFO:teuthology.orchestra.run.smithi125.stdout:Freed space: 173 M 2021-03-15T22:06:30.759 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:06:30.763 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:06:30.763 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:06:30.814 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:06:30.815 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:06:31.026 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:06:31.054 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T22:06:31.055 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:06:31.371 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:06:31.544 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T22:06:31.663 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T22:06:31.788 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T22:06:31.929 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:06:32.047 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:06:32.213 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T22:06:32.355 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:06:32.355 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:06:49.155 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:06:49.310 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T22:06:49.327 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:06:49.328 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:06:49.494 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:06:49.610 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:06:49.718 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:06:49.719 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:06:49.944 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:06:50.129 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T22:06:50.279 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:06:50.483 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:06:50.484 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T22:06:50.484 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T22:06:50.484 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T22:06:50.484 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T22:06:50.485 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T22:06:50.485 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:06:50.485 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T22:06:50.485 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:06:50.485 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T22:06:50.486 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T22:06:50.486 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T22:06:50.486 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T22:06:50.486 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T22:06:50.487 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:06:50.487 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T22:06:50.925 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T22:06:50.925 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:06:50.925 INFO:teuthology.orchestra.run.smithi125.stdout:Removed: 2021-03-15T22:06:50.926 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:06:50.926 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:06:50.926 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:06:50.926 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:06:50.927 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:06:50.927 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:06:50.927 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:06:50.927 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:06:50.928 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:06:50.928 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:06:50.928 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:06:50.928 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:06:50.928 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:06:50.929 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:06:50.929 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:06:50.929 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:06:50.929 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:06:50.929 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:06:51.172 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T22:06:51.628 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 80 kB/s | 3.3 kB 00:00 2021-03-15T22:06:51.944 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 5.3 kB/s | 1.7 kB 00:00 2021-03-15T22:06:52.490 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:06:52.492 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:06:52.492 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:06:52.493 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:06:52.493 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:06:52.493 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T22:06:52.493 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:06:52.494 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T22:06:52.494 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T22:06:52.494 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T22:06:52.494 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T22:06:52.495 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T22:06:52.495 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T22:06:52.495 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T22:06:52.495 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T22:06:52.495 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T22:06:52.496 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T22:06:52.496 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T22:06:52.496 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:06:52.496 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T22:06:52.497 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T22:06:52.497 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T22:06:52.497 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T22:06:52.497 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:06:52.498 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:06:52.498 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:06:52.498 INFO:teuthology.orchestra.run.smithi125.stdout:Install 16 Packages 2021-03-15T22:06:52.498 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:06:52.499 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 68 M 2021-03-15T22:06:52.499 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 173 M 2021-03-15T22:06:52.500 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:06:52.534 INFO:teuthology.orchestra.run.smithi125.stdout:(1/16): yajl-2.1.0-10.el8.x86_64.rpm 1.0 MB/s | 41 kB 00:00 2021-03-15T22:06:52.536 INFO:teuthology.orchestra.run.smithi125.stdout:(2/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 7.7 MB/s | 318 kB 00:00 2021-03-15T22:06:52.537 INFO:teuthology.orchestra.run.smithi125.stdout:(3/16): fuse-common-3.2.1-12.el8.x86_64.rpm 517 kB/s | 21 kB 00:00 2021-03-15T22:06:52.537 INFO:teuthology.orchestra.run.smithi125.stdout:(4/16): fuse3-3.2.1-12.el8.x86_64.rpm 18 MB/s | 50 kB 00:00 2021-03-15T22:06:52.539 INFO:teuthology.orchestra.run.smithi125.stdout:(5/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 37 MB/s | 94 kB 00:00 2021-03-15T22:06:52.652 INFO:teuthology.orchestra.run.smithi125.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 514 kB/s | 59 kB 00:00 2021-03-15T22:06:53.618 INFO:teuthology.orchestra.run.smithi125.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 46 kB/s | 50 kB 00:01 2021-03-15T22:06:53.826 INFO:teuthology.orchestra.run.smithi125.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 224 kB/s | 289 kB 00:01 2021-03-15T22:06:53.923 INFO:teuthology.orchestra.run.smithi125.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 192 kB/s | 59 kB 00:00 2021-03-15T22:06:54.231 INFO:teuthology.orchestra.run.smithi125.stdout:(10/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 235 kB/s | 72 kB 00:00 2021-03-15T22:06:54.235 INFO:teuthology.orchestra.run.smithi125.stdout:(11/16): crun-0.18-3.el8.5.2.x86_64.rpm 456 kB/s | 186 kB 00:00 2021-03-15T22:06:54.544 INFO:teuthology.orchestra.run.smithi125.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 771 kB/s | 241 kB 00:00 2021-03-15T22:06:55.077 INFO:teuthology.orchestra.run.smithi125.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 4.6 MB/s | 2.5 MB 00:00 2021-03-15T22:06:55.908 INFO:teuthology.orchestra.run.smithi125.stdout:(14/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T22:06:56.182 INFO:teuthology.orchestra.run.smithi125.stdout:(15/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 12 MB/s | 23 MB 00:01 2021-03-15T22:07:01.468 INFO:teuthology.orchestra.run.smithi125.stdout:(16/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 8.6 kB/s | 55 kB 00:06 2021-03-15T22:07:01.476 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:07:01.477 INFO:teuthology.orchestra.run.smithi125.stdout:Total 7.6 MB/s | 68 MB 00:08 2021-03-15T22:07:01.492 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:07:01.536 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:07:01.536 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:07:02.017 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:07:02.019 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:07:02.668 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:07:02.796 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T22:07:05.240 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T22:07:05.446 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T22:07:05.597 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T22:07:05.723 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T22:07:05.736 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:07:05.873 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:07:23.537 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:07:23.712 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:07:23.795 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:07:23.926 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:07:24.060 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T22:07:24.186 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:07:24.209 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:07:24.358 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T22:07:24.502 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:07:24.662 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T22:07:24.757 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T22:07:24.758 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.hi7jLe: line 5: 41944 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T22:07:25.040 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:07:26.844 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:07:27.028 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:07:27.588 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T22:07:27.906 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:07:27.907 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.q8HWdg: line 6: 42894 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T22:07:27.907 INFO:teuthology.orchestra.run.smithi125.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T22:07:27.907 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:07:27.908 INFO:teuthology.orchestra.run.smithi125.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T22:07:28.117 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T22:07:28.118 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T22:07:28.118 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T22:07:28.118 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T22:07:28.118 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:07:28.119 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:07:28.119 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T22:07:28.119 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T22:07:28.119 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T22:07:28.120 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T22:07:28.120 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T22:07:28.120 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T22:07:28.121 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:07:28.121 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:07:28.121 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:07:28.649 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T22:07:28.649 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:07:28.650 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:07:28.650 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:07:28.651 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:07:28.651 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:07:28.652 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:07:28.652 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:07:28.653 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:07:28.653 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:07:28.654 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:07:28.654 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:07:28.655 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:07:28.655 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:07:28.656 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:07:28.657 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:07:28.657 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:07:28.657 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:07:28.658 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:07:28.658 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:07:28.659 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:07:28.877 INFO:teuthology.task.exec:Running commands on role grafana.a host ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:07:28.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T22:07:29.348 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:38 ago on Mon 15 Mar 2021 10:06:51 PM UTC. 2021-03-15T22:07:29.830 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:07:29.831 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:07:29.831 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:07:29.884 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T22:07:30.343 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:39 ago on Mon 15 Mar 2021 10:06:51 PM UTC. 2021-03-15T22:07:30.673 INFO:teuthology.orchestra.run.smithi125.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T22:07:30.783 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:07:30.784 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:07:30.785 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:07:30.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T22:07:31.129 INFO:teuthology.orchestra.run.smithi125.stdout:Repository successfully enabled. 2021-03-15T22:07:31.136 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T22:07:31.136 INFO:teuthology.orchestra.run.smithi125.stderr:of the main distribution, and quality may vary. 2021-03-15T22:07:31.136 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:07:31.137 INFO:teuthology.orchestra.run.smithi125.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T22:07:31.137 INFO:teuthology.orchestra.run.smithi125.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T22:07:31.137 INFO:teuthology.orchestra.run.smithi125.stderr:, 2021-03-15T22:07:31.137 INFO:teuthology.orchestra.run.smithi125.stderr:and packages are not held to any quality or security level. 2021-03-15T22:07:31.137 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:07:31.138 INFO:teuthology.orchestra.run.smithi125.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T22:07:31.138 INFO:teuthology.orchestra.run.smithi125.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T22:07:31.166 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T22:07:31.202 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T22:07:31.202 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-15T22:07:31.524 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1111 0 --:--:-- --:--:-- --:--:-- 1108 2021-03-15T22:07:31.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T22:07:31.998 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:07:32.002 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:07:32.002 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:07:32.002 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:07:32.003 INFO:teuthology.orchestra.run.smithi125.stdout:Removing: 2021-03-15T22:07:32.003 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T22:07:32.003 INFO:teuthology.orchestra.run.smithi125.stdout:Removing unused dependencies: 2021-03-15T22:07:32.003 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T22:07:32.003 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T22:07:32.004 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T22:07:32.004 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T22:07:32.004 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T22:07:32.004 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T22:07:32.005 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T22:07:32.005 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T22:07:32.005 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T22:07:32.005 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T22:07:32.006 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T22:07:32.006 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T22:07:32.006 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T22:07:32.006 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T22:07:32.007 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T22:07:32.007 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:07:32.007 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:07:32.007 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:07:32.008 INFO:teuthology.orchestra.run.smithi125.stdout:Remove 16 Packages 2021-03-15T22:07:32.008 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:07:32.008 INFO:teuthology.orchestra.run.smithi125.stdout:Freed space: 173 M 2021-03-15T22:07:32.009 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:07:32.012 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:07:32.013 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:07:32.064 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:07:32.065 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:07:32.268 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:07:32.298 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T22:07:32.298 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:07:32.602 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:07:32.743 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T22:07:32.869 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T22:07:33.014 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T22:07:33.160 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:07:33.287 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:07:33.444 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T22:07:33.553 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:07:33.553 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:07:50.094 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:07:50.216 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T22:07:50.233 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:07:50.234 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:07:50.392 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:07:50.508 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:07:50.624 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:07:50.625 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:07:50.883 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:07:51.035 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T22:07:51.185 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:07:51.411 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:07:51.411 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T22:07:51.411 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T22:07:51.412 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T22:07:51.412 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T22:07:51.412 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T22:07:51.412 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:07:51.412 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T22:07:51.413 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:07:51.413 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T22:07:51.413 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T22:07:51.413 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T22:07:51.413 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T22:07:51.414 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T22:07:51.414 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:07:51.414 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T22:07:51.838 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T22:07:51.838 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:07:51.839 INFO:teuthology.orchestra.run.smithi125.stdout:Removed: 2021-03-15T22:07:51.839 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:07:51.839 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:07:51.839 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:07:51.840 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:07:51.840 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:07:51.840 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:07:51.840 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:07:51.840 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:07:51.841 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:07:51.841 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:07:51.841 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:07:51.841 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:07:51.842 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:07:51.842 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:07:51.842 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:07:51.842 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:07:51.842 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:07:51.843 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:07:52.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T22:07:52.583 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 67 kB/s | 3.3 kB 00:00 2021-03-15T22:07:52.918 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 5.0 kB/s | 1.7 kB 00:00 2021-03-15T22:07:53.435 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:07:53.437 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:07:53.437 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:07:53.437 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:07:53.438 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:07:53.438 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T22:07:53.438 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:07:53.438 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T22:07:53.438 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T22:07:53.439 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T22:07:53.439 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T22:07:53.439 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T22:07:53.439 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T22:07:53.440 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T22:07:53.440 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T22:07:53.440 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T22:07:53.440 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T22:07:53.441 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T22:07:53.441 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:07:53.441 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T22:07:53.441 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T22:07:53.442 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T22:07:53.442 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T22:07:53.442 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:07:53.442 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:07:53.443 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:07:53.443 INFO:teuthology.orchestra.run.smithi125.stdout:Install 16 Packages 2021-03-15T22:07:53.443 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:07:53.444 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 68 M 2021-03-15T22:07:53.444 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 173 M 2021-03-15T22:07:53.444 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:07:53.489 INFO:teuthology.orchestra.run.smithi125.stdout:(1/16): yajl-2.1.0-10.el8.x86_64.rpm 853 kB/s | 41 kB 00:00 2021-03-15T22:07:53.489 INFO:teuthology.orchestra.run.smithi125.stdout:(2/16): fuse-common-3.2.1-12.el8.x86_64.rpm 438 kB/s | 21 kB 00:00 2021-03-15T22:07:53.490 INFO:teuthology.orchestra.run.smithi125.stdout:(3/16): fuse3-3.2.1-12.el8.x86_64.rpm 30 MB/s | 50 kB 00:00 2021-03-15T22:07:53.492 INFO:teuthology.orchestra.run.smithi125.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 31 MB/s | 94 kB 00:00 2021-03-15T22:07:53.493 INFO:teuthology.orchestra.run.smithi125.stdout:(5/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 5.9 MB/s | 318 kB 00:00 2021-03-15T22:07:53.534 INFO:teuthology.orchestra.run.smithi125.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.3 MB/s | 59 kB 00:00 2021-03-15T22:07:54.582 INFO:teuthology.orchestra.run.smithi125.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 46 kB/s | 50 kB 00:01 2021-03-15T22:07:54.777 INFO:teuthology.orchestra.run.smithi125.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 225 kB/s | 289 kB 00:01 2021-03-15T22:07:54.884 INFO:teuthology.orchestra.run.smithi125.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 194 kB/s | 59 kB 00:00 2021-03-15T22:07:55.087 INFO:teuthology.orchestra.run.smithi125.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 601 kB/s | 186 kB 00:00 2021-03-15T22:07:55.192 INFO:teuthology.orchestra.run.smithi125.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 234 kB/s | 72 kB 00:00 2021-03-15T22:07:55.405 INFO:teuthology.orchestra.run.smithi125.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 759 kB/s | 241 kB 00:00 2021-03-15T22:07:55.945 INFO:teuthology.orchestra.run.smithi125.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 4.6 MB/s | 2.5 MB 00:00 2021-03-15T22:07:56.247 INFO:teuthology.orchestra.run.smithi125.stdout:(14/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 181 kB/s | 55 kB 00:00 2021-03-15T22:07:56.952 INFO:teuthology.orchestra.run.smithi125.stdout:(15/16): containernetworking-plugins-0.9.1-4.el 12 MB/s | 41 MB 00:03 2021-03-15T22:07:57.428 INFO:teuthology.orchestra.run.smithi125.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 10 MB/s | 23 MB 00:02 2021-03-15T22:07:57.436 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:07:57.436 INFO:teuthology.orchestra.run.smithi125.stdout:Total 17 MB/s | 68 MB 00:03 2021-03-15T22:07:57.452 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:07:57.496 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:07:57.496 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:07:57.975 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:07:57.977 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:07:58.627 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:07:58.770 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T22:08:01.197 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T22:08:01.362 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T22:08:01.546 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T22:08:01.664 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T22:08:01.678 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:08:01.813 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:08:19.388 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:08:19.552 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:08:19.616 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:08:19.733 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:08:19.883 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T22:08:20.027 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:08:20.050 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:08:20.214 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T22:08:20.359 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:08:20.494 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T22:08:20.559 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T22:08:20.560 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.l0q3T1: line 5: 44537 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T22:08:20.880 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:08:22.634 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:08:22.810 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:08:23.373 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T22:08:23.832 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:08:23.833 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.RvRpMH: line 6: 45487 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T22:08:23.833 INFO:teuthology.orchestra.run.smithi125.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T22:08:23.833 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:08:23.834 INFO:teuthology.orchestra.run.smithi125.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T22:08:24.020 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T22:08:24.020 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T22:08:24.020 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T22:08:24.020 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T22:08:24.021 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:08:24.021 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:08:24.021 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T22:08:24.021 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T22:08:24.022 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T22:08:24.022 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T22:08:24.022 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T22:08:24.022 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T22:08:24.023 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:08:24.023 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:08:24.023 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:08:24.565 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T22:08:24.565 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:08:24.565 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:08:24.565 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:08:24.566 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:08:24.566 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:08:24.566 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:08:24.566 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:08:24.567 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:08:24.567 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:08:24.567 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:08:24.567 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:08:24.567 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:08:24.568 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:08:24.568 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:08:24.568 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:08:24.568 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:08:24.569 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:08:24.569 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:08:24.569 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:08:24.569 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:08:24.732 INFO:teuthology.task.exec:Running commands on role node-exporter.b host ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:08:24.732 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T22:08:25.203 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 10:07:52 PM UTC. 2021-03-15T22:08:25.679 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:08:25.680 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:08:25.681 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:08:25.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T22:08:26.204 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:34 ago on Mon 15 Mar 2021 10:07:52 PM UTC. 2021-03-15T22:08:26.537 INFO:teuthology.orchestra.run.smithi125.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T22:08:26.649 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:08:26.652 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:08:26.652 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:08:26.716 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T22:08:27.015 INFO:teuthology.orchestra.run.smithi125.stdout:Repository successfully enabled. 2021-03-15T22:08:27.022 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T22:08:27.022 INFO:teuthology.orchestra.run.smithi125.stderr:of the main distribution, and quality may vary. 2021-03-15T22:08:27.022 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:08:27.023 INFO:teuthology.orchestra.run.smithi125.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T22:08:27.023 INFO:teuthology.orchestra.run.smithi125.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T22:08:27.023 INFO:teuthology.orchestra.run.smithi125.stderr:, 2021-03-15T22:08:27.023 INFO:teuthology.orchestra.run.smithi125.stderr:and packages are not held to any quality or security level. 2021-03-15T22:08:27.023 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:08:27.024 INFO:teuthology.orchestra.run.smithi125.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T22:08:27.024 INFO:teuthology.orchestra.run.smithi125.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T22:08:27.052 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T22:08:27.088 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T22:08:27.089 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-15T22:08:27.402 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1143 0 --:--:-- --:--:-- --:--:-- 1143 2021-03-15T22:08:27.405 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T22:08:27.875 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:08:27.878 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:08:27.879 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:08:27.879 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:08:27.880 INFO:teuthology.orchestra.run.smithi125.stdout:Removing: 2021-03-15T22:08:27.880 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T22:08:27.880 INFO:teuthology.orchestra.run.smithi125.stdout:Removing unused dependencies: 2021-03-15T22:08:27.881 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T22:08:27.881 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T22:08:27.881 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T22:08:27.882 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T22:08:27.882 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T22:08:27.882 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T22:08:27.882 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T22:08:27.883 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T22:08:27.883 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T22:08:27.883 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T22:08:27.884 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T22:08:27.884 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T22:08:27.884 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T22:08:27.885 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T22:08:27.885 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T22:08:27.885 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:08:27.885 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:08:27.886 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:08:27.886 INFO:teuthology.orchestra.run.smithi125.stdout:Remove 16 Packages 2021-03-15T22:08:27.886 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:08:27.886 INFO:teuthology.orchestra.run.smithi125.stdout:Freed space: 173 M 2021-03-15T22:08:27.887 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:08:27.888 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:08:27.889 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:08:27.940 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:08:27.941 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:08:28.146 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:08:28.176 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T22:08:28.177 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:08:28.509 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:08:28.658 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T22:08:28.801 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T22:08:28.918 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T22:08:29.043 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:08:29.369 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:08:29.543 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T22:08:29.652 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:08:29.652 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:08:46.360 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:08:46.473 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T22:08:46.489 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:08:46.490 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:08:46.649 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:08:46.765 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:08:46.881 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:08:46.882 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:08:47.132 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:08:47.284 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T22:08:47.442 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:08:47.662 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:08:47.662 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T22:08:47.663 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T22:08:47.663 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T22:08:47.663 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T22:08:47.663 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T22:08:47.664 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:08:47.664 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T22:08:47.664 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:08:47.664 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T22:08:47.664 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T22:08:47.665 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T22:08:47.665 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T22:08:47.665 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T22:08:47.665 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:08:47.666 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T22:08:48.238 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T22:08:48.239 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:08:48.239 INFO:teuthology.orchestra.run.smithi125.stdout:Removed: 2021-03-15T22:08:48.239 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:08:48.239 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:08:48.240 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:08:48.240 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:08:48.240 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:08:48.240 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:08:48.241 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:08:48.241 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:08:48.241 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:08:48.241 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:08:48.241 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:08:48.242 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:08:48.242 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:08:48.242 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:08:48.243 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:08:48.243 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:08:48.243 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:08:48.243 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:08:48.556 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T22:08:49.006 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 96 kB/s | 3.3 kB 00:00 2021-03-15T22:08:49.331 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 5.1 kB/s | 1.7 kB 00:00 2021-03-15T22:08:49.842 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:08:49.844 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:08:49.844 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:08:49.845 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:08:49.845 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:08:49.845 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T22:08:49.845 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:08:49.846 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T22:08:49.846 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T22:08:49.846 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T22:08:49.846 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T22:08:49.847 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T22:08:49.847 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T22:08:49.847 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T22:08:49.847 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T22:08:49.847 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T22:08:49.848 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T22:08:49.848 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T22:08:49.848 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:08:49.848 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T22:08:49.849 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T22:08:49.849 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T22:08:49.849 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T22:08:49.849 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:08:49.849 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:08:49.850 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:08:49.850 INFO:teuthology.orchestra.run.smithi125.stdout:Install 16 Packages 2021-03-15T22:08:49.850 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:08:49.851 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 68 M 2021-03-15T22:08:49.851 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 173 M 2021-03-15T22:08:49.852 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:08:49.882 INFO:teuthology.orchestra.run.smithi125.stdout:(1/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 9.0 MB/s | 318 kB 00:00 2021-03-15T22:08:49.883 INFO:teuthology.orchestra.run.smithi125.stdout:(2/16): yajl-2.1.0-10.el8.x86_64.rpm 1.1 MB/s | 41 kB 00:00 2021-03-15T22:08:49.885 INFO:teuthology.orchestra.run.smithi125.stdout:(3/16): fuse3-3.2.1-12.el8.x86_64.rpm 22 MB/s | 50 kB 00:00 2021-03-15T22:08:49.886 INFO:teuthology.orchestra.run.smithi125.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 29 MB/s | 94 kB 00:00 2021-03-15T22:08:49.888 INFO:teuthology.orchestra.run.smithi125.stdout:(5/16): fuse-common-3.2.1-12.el8.x86_64.rpm 527 kB/s | 21 kB 00:00 2021-03-15T22:08:49.927 INFO:teuthology.orchestra.run.smithi125.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.4 MB/s | 59 kB 00:00 2021-03-15T22:08:50.935 INFO:teuthology.orchestra.run.smithi125.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 47 kB/s | 50 kB 00:01 2021-03-15T22:08:51.174 INFO:teuthology.orchestra.run.smithi125.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 224 kB/s | 289 kB 00:01 2021-03-15T22:08:51.238 INFO:teuthology.orchestra.run.smithi125.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 194 kB/s | 59 kB 00:00 2021-03-15T22:08:51.548 INFO:teuthology.orchestra.run.smithi125.stdout:(10/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 233 kB/s | 72 kB 00:00 2021-03-15T22:08:51.587 INFO:teuthology.orchestra.run.smithi125.stdout:(11/16): crun-0.18-3.el8.5.2.x86_64.rpm 451 kB/s | 186 kB 00:00 2021-03-15T22:08:51.858 INFO:teuthology.orchestra.run.smithi125.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 779 kB/s | 241 kB 00:00 2021-03-15T22:08:52.390 INFO:teuthology.orchestra.run.smithi125.stdout:(13/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 4.6 MB/s | 2.5 MB 00:00 2021-03-15T22:08:52.705 INFO:teuthology.orchestra.run.smithi125.stdout:(14/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 174 kB/s | 55 kB 00:00 2021-03-15T22:08:53.159 INFO:teuthology.orchestra.run.smithi125.stdout:(15/16): containernetworking-plugins-0.9.1-4.el 13 MB/s | 41 MB 00:03 2021-03-15T22:08:53.512 INFO:teuthology.orchestra.run.smithi125.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 12 MB/s | 23 MB 00:01 2021-03-15T22:08:53.520 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:08:53.520 INFO:teuthology.orchestra.run.smithi125.stdout:Total 19 MB/s | 68 MB 00:03 2021-03-15T22:08:53.536 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:08:53.581 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:08:53.581 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:08:54.062 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:08:54.064 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:08:54.722 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:08:54.860 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T22:08:57.304 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T22:08:57.460 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T22:08:57.620 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T22:08:57.771 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T22:08:57.786 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:08:57.912 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:09:15.745 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:09:15.917 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:09:15.983 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:09:16.090 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:09:16.257 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T22:09:16.392 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:09:16.416 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:09:16.572 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T22:09:16.716 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:09:16.851 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T22:09:16.918 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T22:09:16.919 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.yPwmqg: line 5: 47127 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T22:09:17.290 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:09:19.058 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:09:19.209 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:09:19.774 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T22:09:20.116 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:09:20.116 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.JgqewL: line 6: 48077 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T22:09:20.116 INFO:teuthology.orchestra.run.smithi125.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T22:09:20.117 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:09:20.117 INFO:teuthology.orchestra.run.smithi125.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T22:09:20.324 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T22:09:20.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T22:09:20.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T22:09:20.325 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T22:09:20.326 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:09:20.326 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:09:20.326 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T22:09:20.326 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T22:09:20.327 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T22:09:20.327 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T22:09:20.327 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T22:09:20.327 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T22:09:20.327 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:09:20.328 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:09:20.328 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:09:20.863 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T22:09:20.863 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:09:20.864 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:09:20.864 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:09:20.864 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:09:20.864 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:09:20.865 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:09:20.865 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:09:20.865 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:09:20.865 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:09:20.865 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:09:20.866 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:09:20.866 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:09:20.866 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:09:20.866 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:09:20.867 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:09:20.867 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:09:20.867 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:09:20.867 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:09:20.867 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:09:20.868 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:09:21.079 INFO:teuthology.task.exec:Running commands on role ceph.iscsi.iscsi.a host ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:09:21.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y module disable container-tools' 2021-03-15T22:09:21.548 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:32 ago on Mon 15 Mar 2021 10:08:49 PM UTC. 2021-03-15T22:09:22.039 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:09:22.040 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:09:22.040 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:09:22.107 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install '"'"'dnf-command(copr)'"'"'' 2021-03-15T22:09:22.570 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:33 ago on Mon 15 Mar 2021 10:08:49 PM UTC. 2021-03-15T22:09:22.902 INFO:teuthology.orchestra.run.smithi125.stdout:Package dnf-plugins-core-4.0.17-5.el8.noarch is already installed. 2021-03-15T22:09:23.015 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:09:23.016 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:09:23.016 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:09:23.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y copr enable rhcontainerbot/container-selinux' 2021-03-15T22:09:23.370 INFO:teuthology.orchestra.run.smithi125.stdout:Repository successfully enabled. 2021-03-15T22:09:23.377 INFO:teuthology.orchestra.run.smithi125.stderr:Enabling a Copr repository. Please note that this repository is not part 2021-03-15T22:09:23.377 INFO:teuthology.orchestra.run.smithi125.stderr:of the main distribution, and quality may vary. 2021-03-15T22:09:23.377 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:09:23.378 INFO:teuthology.orchestra.run.smithi125.stderr:The Fedora Project does not exercise any power over the contents of 2021-03-15T22:09:23.378 INFO:teuthology.orchestra.run.smithi125.stderr:this repository beyond the rules outlined in the Copr FAQ at 2021-03-15T22:09:23.378 INFO:teuthology.orchestra.run.smithi125.stderr:, 2021-03-15T22:09:23.379 INFO:teuthology.orchestra.run.smithi125.stderr:and packages are not held to any quality or security level. 2021-03-15T22:09:23.379 INFO:teuthology.orchestra.run.smithi125.stderr: 2021-03-15T22:09:23.379 INFO:teuthology.orchestra.run.smithi125.stderr:Please do not file bug reports about these packages in Fedora 2021-03-15T22:09:23.380 INFO:teuthology.orchestra.run.smithi125.stderr:Bugzilla. In case of problems, contact the owner of this repository. 2021-03-15T22:09:23.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/CentOS_8/devel:kubic:libcontainers:stable.repo' 2021-03-15T22:09:23.443 INFO:teuthology.orchestra.run.smithi125.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-15T22:09:23.444 INFO:teuthology.orchestra.run.smithi125.stderr: Dload Upload Total Spent Left Speed 2021-03-15T22:09:23.758 INFO:teuthology.orchestra.run.smithi125.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 359 100 359 0 0 1139 0 --:--:-- --:--:-- --:--:-- 1136 2021-03-15T22:09:23.760 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf remove -y podman' 2021-03-15T22:09:24.235 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:09:24.236 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:09:24.236 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:09:24.236 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:09:24.237 INFO:teuthology.orchestra.run.smithi125.stdout:Removing: 2021-03-15T22:09:24.237 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 66 M 2021-03-15T22:09:24.237 INFO:teuthology.orchestra.run.smithi125.stdout:Removing unused dependencies: 2021-03-15T22:09:24.237 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 @devel_kubic_libcontainers_stable 744 k 2021-03-15T22:09:24.238 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 @devel_kubic_libcontainers_stable 272 k 2021-03-15T22:09:24.238 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 @copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 46 k 2021-03-15T22:09:24.238 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 @devel_kubic_libcontainers_stable 98 M 2021-03-15T22:09:24.238 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 @devel_kubic_libcontainers_stable 76 k 2021-03-15T22:09:24.238 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 @devel_kubic_libcontainers_stable 466 k 2021-03-15T22:09:24.239 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 @CentOS-AppStream 647 k 2021-03-15T22:09:24.239 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 @CentOS-Base 4.7 k 2021-03-15T22:09:24.239 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 @devel_kubic_libcontainers_stable 125 k 2021-03-15T22:09:24.239 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 @CentOS-Base 90 k 2021-03-15T22:09:24.240 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 @CentOS-Base 279 k 2021-03-15T22:09:24.240 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 @devel_kubic_libcontainers_stable 870 k 2021-03-15T22:09:24.240 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 @devel_kubic_libcontainers_stable 5.6 M 2021-03-15T22:09:24.240 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 @devel_kubic_libcontainers_stable 93 k 2021-03-15T22:09:24.240 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 @CentOS-AppStream 85 k 2021-03-15T22:09:24.241 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:09:24.241 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:09:24.241 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================= 2021-03-15T22:09:24.241 INFO:teuthology.orchestra.run.smithi125.stdout:Remove 16 Packages 2021-03-15T22:09:24.241 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:09:24.242 INFO:teuthology.orchestra.run.smithi125.stdout:Freed space: 173 M 2021-03-15T22:09:24.242 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:09:24.246 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:09:24.246 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:09:24.297 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:09:24.298 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:09:24.545 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:09:24.573 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/1 2021-03-15T22:09:24.574 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:09:24.866 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-3.0.1-2.el8.3.2.x86_64 1/16 2021-03-15T22:09:25.023 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containers-common-4:1-8.el8.13.2.noarch 2/16 2021-03-15T22:09:25.141 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 3/16 2021-03-15T22:09:25.267 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : slirp4netns-1.1.8-4.el8.7.2.x86_64 4/16 2021-03-15T22:09:25.416 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:09:25.551 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:09:25.691 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : podman-plugins-3.0.1-2.el8.3.2.x86_64 7/16 2021-03-15T22:09:25.792 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:09:25.792 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:09:42.401 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 9/16 2021-03-15T22:09:42.514 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : catatonit-0.1.5-6.el8.3.2.x86_64 10/16 2021-03-15T22:09:42.530 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:09:42.531 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:09:42.689 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 11/16 2021-03-15T22:09:42.805 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:09:42.922 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:09:42.922 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:09:43.164 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 14/16 2021-03-15T22:09:43.316 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : conmon-2:2.0.27-1.el8.1.1.x86_64 15/16 2021-03-15T22:09:43.466 INFO:teuthology.orchestra.run.smithi125.stdout: Erasing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:09:43.681 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 16/16 2021-03-15T22:09:43.682 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 1/16 2021-03-15T22:09:43.682 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 2/16 2021-03-15T22:09:43.682 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 3/16 2021-03-15T22:09:43.682 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 4/16 2021-03-15T22:09:43.683 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 5/16 2021-03-15T22:09:43.683 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 6/16 2021-03-15T22:09:43.683 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 7/16 2021-03-15T22:09:43.683 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:09:43.683 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 9/16 2021-03-15T22:09:43.684 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 10/16 2021-03-15T22:09:43.684 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 11/16 2021-03-15T22:09:43.684 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 12/16 2021-03-15T22:09:43.685 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 13/16 2021-03-15T22:09:43.685 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:09:43.685 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 15/16 2021-03-15T22:09:44.137 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 16/16 2021-03-15T22:09:44.138 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:09:44.138 INFO:teuthology.orchestra.run.smithi125.stdout:Removed: 2021-03-15T22:09:44.139 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:09:44.139 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:09:44.139 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:09:44.140 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:09:44.141 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:09:44.141 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:09:44.142 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:09:44.142 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:09:44.142 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:09:44.142 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:09:44.143 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:09:44.143 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:09:44.143 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:09:44.143 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:09:44.144 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:09:44.144 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:09:44.144 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:09:44.144 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:09:44.396 DEBUG:teuthology.orchestra.run.smithi125:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo dnf -y install podman' 2021-03-15T22:09:44.859 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 68 kB/s | 3.3 kB 00:00 2021-03-15T22:09:45.185 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 5.2 kB/s | 1.7 kB 00:00 2021-03-15T22:09:45.684 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:09:45.687 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:09:45.687 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:09:45.687 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:09:45.687 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:09:45.688 INFO:teuthology.orchestra.run.smithi125.stdout: podman x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 23 M 2021-03-15T22:09:45.688 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:09:45.688 INFO:teuthology.orchestra.run.smithi125.stdout: conmon x86_64 2:2.0.27-1.el8.1.1 devel_kubic_libcontainers_stable 50 k 2021-03-15T22:09:45.688 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux noarch 2:2.145.0-1.el8 copr:copr.fedorainfracloud.org:rhcontainerbot:container-selinux 59 k 2021-03-15T22:09:45.689 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins x86_64 0.9.1-4.el8.5.2 devel_kubic_libcontainers_stable 41 M 2021-03-15T22:09:45.689 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common noarch 4:1-8.el8.13.2 devel_kubic_libcontainers_stable 59 k 2021-03-15T22:09:45.689 INFO:teuthology.orchestra.run.smithi125.stdout: crun x86_64 0.18-3.el8.5.2 devel_kubic_libcontainers_stable 186 k 2021-03-15T22:09:45.689 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq x86_64 2.79-13.el8_3.1 CentOS-AppStream 318 k 2021-03-15T22:09:45.689 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-03-15T22:09:45.690 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3 x86_64 3.2.1-12.el8 CentOS-Base 50 k 2021-03-15T22:09:45.690 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs x86_64 3.2.1-12.el8 CentOS-Base 94 k 2021-03-15T22:09:45.690 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp x86_64 4.3.1-4.el8.4.2 devel_kubic_libcontainers_stable 241 k 2021-03-15T22:09:45.690 INFO:teuthology.orchestra.run.smithi125.stdout: yajl x86_64 2.1.0-10.el8 CentOS-AppStream 41 k 2021-03-15T22:09:45.691 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:09:45.691 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit x86_64 0.1.5-6.el8.3.2 devel_kubic_libcontainers_stable 289 k 2021-03-15T22:09:45.691 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs x86_64 1.4.0-3.el8.4.2 devel_kubic_libcontainers_stable 72 k 2021-03-15T22:09:45.691 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins x86_64 3.0.1-2.el8.3.2 devel_kubic_libcontainers_stable 2.5 M 2021-03-15T22:09:45.691 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns x86_64 1.1.8-4.el8.7.2 devel_kubic_libcontainers_stable 55 k 2021-03-15T22:09:45.692 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:09:45.692 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:09:45.692 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================================================================ 2021-03-15T22:09:45.694 INFO:teuthology.orchestra.run.smithi125.stdout:Install 16 Packages 2021-03-15T22:09:45.695 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:09:45.696 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 68 M 2021-03-15T22:09:45.696 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 173 M 2021-03-15T22:09:45.696 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:09:45.742 INFO:teuthology.orchestra.run.smithi125.stdout:(1/16): fuse-common-3.2.1-12.el8.x86_64.rpm 415 kB/s | 21 kB 00:00 2021-03-15T22:09:45.744 INFO:teuthology.orchestra.run.smithi125.stdout:(2/16): fuse3-3.2.1-12.el8.x86_64.rpm 24 MB/s | 50 kB 00:00 2021-03-15T22:09:45.747 INFO:teuthology.orchestra.run.smithi125.stdout:(3/16): dnsmasq-2.79-13.el8_3.1.x86_64.rpm 5.4 MB/s | 318 kB 00:00 2021-03-15T22:09:45.748 INFO:teuthology.orchestra.run.smithi125.stdout:(4/16): fuse3-libs-3.2.1-12.el8.x86_64.rpm 22 MB/s | 94 kB 00:00 2021-03-15T22:09:45.749 INFO:teuthology.orchestra.run.smithi125.stdout:(5/16): yajl-2.1.0-10.el8.x86_64.rpm 690 kB/s | 41 kB 00:00 2021-03-15T22:09:45.788 INFO:teuthology.orchestra.run.smithi125.stdout:(6/16): container-selinux-2.145.0-1.el8.noarch. 1.4 MB/s | 59 kB 00:00 2021-03-15T22:09:47.050 INFO:teuthology.orchestra.run.smithi125.stdout:(7/16): conmon-2.0.27-1.el8.1.1.x86_64.rpm 38 kB/s | 50 kB 00:01 2021-03-15T22:09:47.161 INFO:teuthology.orchestra.run.smithi125.stdout:(8/16): catatonit-0.1.5-6.el8.3.2.x86_64.rpm 204 kB/s | 289 kB 00:01 2021-03-15T22:09:47.352 INFO:teuthology.orchestra.run.smithi125.stdout:(9/16): containers-common-1-8.el8.13.2.noarch.r 194 kB/s | 59 kB 00:00 2021-03-15T22:09:47.470 INFO:teuthology.orchestra.run.smithi125.stdout:(10/16): crun-0.18-3.el8.5.2.x86_64.rpm 604 kB/s | 186 kB 00:00 2021-03-15T22:09:47.654 INFO:teuthology.orchestra.run.smithi125.stdout:(11/16): fuse-overlayfs-1.4.0-3.el8.4.2.x86_64. 239 kB/s | 72 kB 00:00 2021-03-15T22:09:47.784 INFO:teuthology.orchestra.run.smithi125.stdout:(12/16): libslirp-4.3.1-4.el8.4.2.x86_64.rpm 768 kB/s | 241 kB 00:00 2021-03-15T22:09:49.260 INFO:teuthology.orchestra.run.smithi125.stdout:(13/16): containernetworking-plugins-0.9.1-4.el 12 MB/s | 41 MB 00:03 2021-03-15T22:09:49.490 INFO:teuthology.orchestra.run.smithi125.stdout:(14/16): podman-plugins-3.0.1-2.el8.3.2.x86_64. 1.4 MB/s | 2.5 MB 00:01 2021-03-15T22:09:49.563 INFO:teuthology.orchestra.run.smithi125.stdout:(15/16): slirp4netns-1.1.8-4.el8.7.2.x86_64.rpm 181 kB/s | 55 kB 00:00 2021-03-15T22:09:49.730 INFO:teuthology.orchestra.run.smithi125.stdout:(16/16): podman-3.0.1-2.el8.3.2.x86_64.rpm 11 MB/s | 23 MB 00:02 2021-03-15T22:09:49.738 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:09:49.738 INFO:teuthology.orchestra.run.smithi125.stdout:Total 17 MB/s | 68 MB 00:04 2021-03-15T22:09:49.755 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:09:49.801 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:09:49.940 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:09:50.281 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:09:50.282 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:09:50.935 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:09:51.075 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libslirp-4.3.1-4.el8.4.2.x86_64 1/16 2021-03-15T22:09:53.513 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : slirp4netns-1.1.8-4.el8.7.2.x86_64 2/16 2021-03-15T22:09:53.676 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 3/16 2021-03-15T22:09:53.835 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : conmon-2:2.0.27-1.el8.1.1.x86_64 4/16 2021-03-15T22:09:53.986 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : catatonit-0.1.5-6.el8.3.2.x86_64 5/16 2021-03-15T22:09:54.000 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:09:54.128 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:10:11.849 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:10:12.016 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:10:12.084 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse3-libs-3.2.1-12.el8.x86_64 7/16 2021-03-15T22:10:12.214 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 8/16 2021-03-15T22:10:12.382 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse3-3.2.1-12.el8.x86_64 9/16 2021-03-15T22:10:12.508 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:10:12.531 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 10/16 2021-03-15T22:10:12.679 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : containers-common-4:1-8.el8.13.2.noarch 11/16 2021-03-15T22:10:12.823 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : yajl-2.1.0-10.el8.x86_64 12/16 2021-03-15T22:10:12.950 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : crun-0.18-3.el8.5.2.x86_64 13/16 2021-03-15T22:10:13.020 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: dnsmasq-2.79-13.el8_3.1.x86_64 14/16 2021-03-15T22:10:13.020 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.BpzdvG: line 5: 49723 Segmentation fault (core dumped) systemd-sysusers - &> /dev/null < /dev/null 2021-03-15T22:10:13.333 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:10:15.107 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:10:15.266 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:10:15.829 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: container-selinux-2:2.145.0-1.el8.noarch 16/16 2021-03-15T22:10:16.140 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: podman-plugins-3.0.1-2.el8.3.2.x86_64 16/16 2021-03-15T22:10:16.140 INFO:teuthology.orchestra.run.smithi125.stdout:/var/tmp/rpm-tmp.Mg8gtU: line 6: 50673 Segmentation fault (core dumped) /usr/bin/systemd-sysusers 2021-03-15T22:10:16.140 INFO:teuthology.orchestra.run.smithi125.stdout:warning: %triggerin(systemd-239-29.el8.x86_64) scriptlet failed, exit status 139 2021-03-15T22:10:16.140 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:10:16.141 INFO:teuthology.orchestra.run.smithi125.stderr:Error in scriptlet in rpm package podman-plugins 2021-03-15T22:10:16.336 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : dnsmasq-2.79-13.el8_3.1.x86_64 1/16 2021-03-15T22:10:16.336 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : yajl-2.1.0-10.el8.x86_64 2/16 2021-03-15T22:10:16.337 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/16 2021-03-15T22:10:16.337 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-3.2.1-12.el8.x86_64 4/16 2021-03-15T22:10:16.337 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse3-libs-3.2.1-12.el8.x86_64 5/16 2021-03-15T22:10:16.337 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : container-selinux-2:2.145.0-1.el8.noarch 6/16 2021-03-15T22:10:16.338 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : catatonit-0.1.5-6.el8.3.2.x86_64 7/16 2021-03-15T22:10:16.338 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : conmon-2:2.0.27-1.el8.1.1.x86_64 8/16 2021-03-15T22:10:16.338 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 9/16 2021-03-15T22:10:16.338 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : containers-common-4:1-8.el8.13.2.noarch 10/16 2021-03-15T22:10:16.339 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : crun-0.18-3.el8.5.2.x86_64 11/16 2021-03-15T22:10:16.339 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 12/16 2021-03-15T22:10:16.339 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libslirp-4.3.1-4.el8.4.2.x86_64 13/16 2021-03-15T22:10:16.339 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-3.0.1-2.el8.3.2.x86_64 14/16 2021-03-15T22:10:16.339 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : podman-plugins-3.0.1-2.el8.3.2.x86_64 15/16 2021-03-15T22:10:16.795 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : slirp4netns-1.1.8-4.el8.7.2.x86_64 16/16 2021-03-15T22:10:16.795 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:10:16.795 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:10:16.796 INFO:teuthology.orchestra.run.smithi125.stdout: catatonit-0.1.5-6.el8.3.2.x86_64 2021-03-15T22:10:16.796 INFO:teuthology.orchestra.run.smithi125.stdout: conmon-2:2.0.27-1.el8.1.1.x86_64 2021-03-15T22:10:16.796 INFO:teuthology.orchestra.run.smithi125.stdout: container-selinux-2:2.145.0-1.el8.noarch 2021-03-15T22:10:16.796 INFO:teuthology.orchestra.run.smithi125.stdout: containernetworking-plugins-0.9.1-4.el8.5.2.x86_64 2021-03-15T22:10:16.796 INFO:teuthology.orchestra.run.smithi125.stdout: containers-common-4:1-8.el8.13.2.noarch 2021-03-15T22:10:16.797 INFO:teuthology.orchestra.run.smithi125.stdout: crun-0.18-3.el8.5.2.x86_64 2021-03-15T22:10:16.797 INFO:teuthology.orchestra.run.smithi125.stdout: dnsmasq-2.79-13.el8_3.1.x86_64 2021-03-15T22:10:16.797 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-03-15T22:10:16.797 INFO:teuthology.orchestra.run.smithi125.stdout: fuse-overlayfs-1.4.0-3.el8.4.2.x86_64 2021-03-15T22:10:16.798 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-3.2.1-12.el8.x86_64 2021-03-15T22:10:16.798 INFO:teuthology.orchestra.run.smithi125.stdout: fuse3-libs-3.2.1-12.el8.x86_64 2021-03-15T22:10:16.798 INFO:teuthology.orchestra.run.smithi125.stdout: libslirp-4.3.1-4.el8.4.2.x86_64 2021-03-15T22:10:16.798 INFO:teuthology.orchestra.run.smithi125.stdout: podman-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:10:16.798 INFO:teuthology.orchestra.run.smithi125.stdout: podman-plugins-3.0.1-2.el8.3.2.x86_64 2021-03-15T22:10:16.799 INFO:teuthology.orchestra.run.smithi125.stdout: slirp4netns-1.1.8-4.el8.7.2.x86_64 2021-03-15T22:10:16.799 INFO:teuthology.orchestra.run.smithi125.stdout: yajl-2.1.0-10.el8.x86_64 2021-03-15T22:10:16.799 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:10:16.799 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:10:17.018 INFO:teuthology.run_tasks:Running task install... 2021-03-15T22:10:17.026 DEBUG:teuthology.task.install:project ceph 2021-03-15T22:10:17.027 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': 'b658018acf46e32d2ec283d302076352e17476f5'}} 2021-03-15T22:10:17.028 DEBUG:teuthology.task.install:config {'sha1': 'b658018acf46e32d2ec283d302076352e17476f5'} 2021-03-15T22:10:17.028 INFO:teuthology.task.install:Using flavor: basic 2021-03-15T22:10:17.036 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-03-15T22:10:17.036 INFO:teuthology.task.install:extra packages: [] 2021-03-15T22:10:17.037 DEBUG:teuthology.orchestra.run.smithi076:> sudo lsb_release -is 2021-03-15T22:10:17.094 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS 2021-03-15T22:10:17.095 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-15T22:10:17.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo lsb_release -is 2021-03-15T22:10:17.097 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b658018acf46e32d2ec283d302076352e17476f5', 'tag': None, 'wait_for_package': False} 2021-03-15T22:10:17.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo lsb_release -is 2021-03-15T22:10:17.138 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS 2021-03-15T22:10:17.139 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-15T22:10:17.140 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b658018acf46e32d2ec283d302076352e17476f5 2021-03-15T22:10:17.148 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS 2021-03-15T22:10:17.149 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-15T22:10:17.149 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'b658018acf46e32d2ec283d302076352e17476f5', 'tag': None, 'wait_for_package': False} 2021-03-15T22:10:17.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo lsb_release -is 2021-03-15T22:10:17.221 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS 2021-03-15T22:10:17.222 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-15T22:10:17.222 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b658018acf46e32d2ec283d302076352e17476f5 2021-03-15T22:10:17.291 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2021-03-15-1507/b658018acf46e32d2ec283d302076352e17476f5/centos/8/flavors/default/ 2021-03-15T22:10:17.292 INFO:teuthology.task.install.rpm:Package version is 17.0.0-2063.gb658018a 2021-03-15T22:10:17.353 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2021-03-15-1507/b658018acf46e32d2ec283d302076352e17476f5/centos/8/flavors/default/ 2021-03-15T22:10:17.353 INFO:teuthology.task.install.rpm:Package version is 17.0.0-2063.gb658018a 2021-03-15T22:10:17.458 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2021-03-15-1507/b658018acf46e32d2ec283d302076352e17476f5/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2021-03-15-1507/b658018acf46e32d2ec283d302076352e17476f5/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2021-03-15-1507/b658018acf46e32d2ec283d302076352e17476f5/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-03-15T22:10:17.458 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:10:17.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-03-15T22:10:17.487 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-03-15T22:10:17.488 DEBUG:teuthology.orchestra.run.smithi076:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b658018acf46e32d2ec283d302076352e17476f5/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-03-15T22:10:17.506 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2021-03-15-1507/b658018acf46e32d2ec283d302076352e17476f5/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2021-03-15-1507/b658018acf46e32d2ec283d302076352e17476f5/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-sage-testing-2021-03-15-1507/b658018acf46e32d2ec283d302076352e17476f5/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-03-15T22:10:17.507 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-15T22:10:17.507 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-03-15T22:10:17.532 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-03-15T22:10:17.533 DEBUG:teuthology.orchestra.run.smithi125:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/b658018acf46e32d2ec283d302076352e17476f5/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-03-15T22:10:17.557 DEBUG:teuthology.orchestra.run.smithi076:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-03-15T22:10:17.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-03-15T22:10:17.638 DEBUG:teuthology.orchestra.run.smithi076:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-03-15T22:10:17.673 DEBUG:teuthology.orchestra.run.smithi125:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-03-15T22:10:17.703 INFO:teuthology.orchestra.run.smithi076.stdout:check_obsoletes = 1 2021-03-15T22:10:17.704 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum clean all 2021-03-15T22:10:17.737 INFO:teuthology.orchestra.run.smithi125.stdout:check_obsoletes = 1 2021-03-15T22:10:17.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum clean all 2021-03-15T22:10:18.004 INFO:teuthology.orchestra.run.smithi076.stdout:80 files removed 2021-03-15T22:10:18.033 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw 2021-03-15T22:10:18.039 INFO:teuthology.orchestra.run.smithi125.stdout:80 files removed 2021-03-15T22:10:18.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph-radosgw 2021-03-15T22:10:18.630 INFO:teuthology.orchestra.run.smithi076.stdout:ceph packages for x86_64 339 kB/s | 74 kB 00:00 2021-03-15T22:10:18.662 INFO:teuthology.orchestra.run.smithi125.stdout:ceph packages for x86_64 338 kB/s | 74 kB 00:00 2021-03-15T22:10:18.847 INFO:teuthology.orchestra.run.smithi076.stdout:ceph noarch packages 83 kB/s | 15 kB 00:00 2021-03-15T22:10:18.879 INFO:teuthology.orchestra.run.smithi125.stdout:ceph noarch packages 83 kB/s | 15 kB 00:00 2021-03-15T22:10:19.056 INFO:teuthology.orchestra.run.smithi076.stdout:ceph source packages 7.8 kB/s | 1.5 kB 00:00 2021-03-15T22:10:19.086 INFO:teuthology.orchestra.run.smithi125.stdout:ceph source packages 8.0 kB/s | 1.5 kB 00:00 2021-03-15T22:10:19.283 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-8 - AppStream 31 MB/s | 6.3 MB 00:00 2021-03-15T22:10:19.324 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS-8 - AppStream 29 MB/s | 6.3 MB 00:00 2021-03-15T22:10:20.730 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS-8 - Base 6.7 MB/s | 2.3 MB 00:00 2021-03-15T22:10:20.754 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-8 - Base 5.1 MB/s | 2.3 MB 00:00 2021-03-15T22:10:21.236 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS-8 - Extras 85 kB/s | 9.2 kB 00:00 2021-03-15T22:10:21.247 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS-8 - Extras 75 kB/s | 9.2 kB 00:00 2021-03-15T22:10:21.364 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for container-selinux owned by rhcont 18 kB/s | 1.4 kB 00:00 2021-03-15T22:10:21.392 INFO:teuthology.orchestra.run.smithi125.stdout:Copr repo for container-selinux owned by rhcont 15 kB/s | 1.4 kB 00:00 2021-03-15T22:10:22.808 INFO:teuthology.orchestra.run.smithi076.stdout:Stable Releases of Upstream github.com/containe 48 kB/s | 68 kB 00:01 2021-03-15T22:10:22.846 INFO:teuthology.orchestra.run.smithi125.stdout:Stable Releases of Upstream github.com/containe 48 kB/s | 68 kB 00:01 2021-03-15T22:10:23.201 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 27 MB/s | 8.9 MB 00:00 2021-03-15T22:10:23.303 INFO:teuthology.orchestra.run.smithi125.stdout:Extra Packages for Enterprise Linux 23 MB/s | 8.9 MB 00:00 2021-03-15T22:10:25.051 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 4.1 MB/s | 24 kB 00:00 2021-03-15T22:10:25.158 INFO:teuthology.orchestra.run.smithi125.stdout:lab-extras 4.6 MB/s | 24 kB 00:00 2021-03-15T22:10:26.256 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:10:26.259 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:10:26.259 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T22:10:26.260 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:10:26.260 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:10:26.260 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw x86_64 2:17.0.0-2063.gb658018a.el8 ceph 9.1 M 2021-03-15T22:10:26.260 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrading: 2021-03-15T22:10:26.261 INFO:teuthology.orchestra.run.smithi076.stdout: libpmem x86_64 1.6.1-1.el8 CentOS-AppStream 79 k 2021-03-15T22:10:26.261 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 x86_64 2:17.0.0-2063.gb658018a.el8 ceph 3.2 M 2021-03-15T22:10:26.261 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1 x86_64 2:17.0.0-2063.gb658018a.el8 ceph 3.3 M 2021-03-15T22:10:26.261 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T22:10:26.262 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base x86_64 2:17.0.0-2063.gb658018a.el8 ceph 5.1 M 2021-03-15T22:10:26.262 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common x86_64 2:17.0.0-2063.gb658018a.el8 ceph 19 M 2021-03-15T22:10:26.262 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux x86_64 2:17.0.0-2063.gb658018a.el8 ceph 23 k 2021-03-15T22:10:26.262 INFO:teuthology.orchestra.run.smithi076.stdout: daxctl-libs x86_64 67-2.el8 CentOS-Base 40 k 2021-03-15T22:10:26.263 INFO:teuthology.orchestra.run.smithi076.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-03-15T22:10:26.263 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs x86_64 2.8-1.el8 epel 306 k 2021-03-15T22:10:26.263 INFO:teuthology.orchestra.run.smithi076.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-03-15T22:10:26.263 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2 x86_64 2:17.0.0-2063.gb658018a.el8 ceph 749 k 2021-03-15T22:10:26.263 INFO:teuthology.orchestra.run.smithi076.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-03-15T22:10:26.264 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-03-15T22:10:26.264 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq x86_64 0.9.0-2.el8 CentOS-Base 47 k 2021-03-15T22:10:26.264 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1 x86_64 2:17.0.0-2063.gb658018a.el8 ceph 457 k 2021-03-15T22:10:26.264 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-03-15T22:10:26.265 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2 x86_64 2:17.0.0-2063.gb658018a.el8 ceph 3.7 M 2021-03-15T22:10:26.265 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-03-15T22:10:26.265 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-03-15T22:10:26.265 INFO:teuthology.orchestra.run.smithi076.stdout: ndctl-libs x86_64 67-2.el8 CentOS-Base 74 k 2021-03-15T22:10:26.265 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse 2021-03-15T22:10:26.266 INFO:teuthology.orchestra.run.smithi076.stdout: x86_64 2:17.0.0-2063.gb658018a.el8 ceph 42 k 2021-03-15T22:10:26.266 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common x86_64 2:17.0.0-2063.gb658018a.el8 ceph 74 k 2021-03-15T22:10:26.266 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs x86_64 2:17.0.0-2063.gb658018a.el8 ceph 176 k 2021-03-15T22:10:26.266 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-03-15T22:10:26.267 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados x86_64 2:17.0.0-2063.gb658018a.el8 ceph 365 k 2021-03-15T22:10:26.267 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd x86_64 2:17.0.0-2063.gb658018a.el8 ceph 346 k 2021-03-15T22:10:26.267 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw x86_64 2:17.0.0-2063.gb658018a.el8 ceph 110 k 2021-03-15T22:10:26.267 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:10:26.267 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:10:26.268 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:10:26.268 INFO:teuthology.orchestra.run.smithi076.stdout:Install 25 Packages 2021-03-15T22:10:26.268 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 3 Packages 2021-03-15T22:10:26.268 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:10:26.269 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 48 M 2021-03-15T22:10:26.269 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:10:26.380 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:10:26.383 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:10:26.383 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:10:26.383 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:10:26.384 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:10:26.384 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-radosgw x86_64 2:17.0.0-2063.gb658018a.el8 ceph 9.1 M 2021-03-15T22:10:26.384 INFO:teuthology.orchestra.run.smithi125.stdout:Upgrading: 2021-03-15T22:10:26.384 INFO:teuthology.orchestra.run.smithi125.stdout: libpmem x86_64 1.6.1-1.el8 CentOS-AppStream 79 k 2021-03-15T22:10:26.385 INFO:teuthology.orchestra.run.smithi125.stdout: librados2 x86_64 2:17.0.0-2063.gb658018a.el8 ceph 3.2 M 2021-03-15T22:10:26.385 INFO:teuthology.orchestra.run.smithi125.stdout: librbd1 x86_64 2:17.0.0-2063.gb658018a.el8 ceph 3.3 M 2021-03-15T22:10:26.385 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:10:26.385 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-base x86_64 2:17.0.0-2063.gb658018a.el8 ceph 5.1 M 2021-03-15T22:10:26.385 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-common x86_64 2:17.0.0-2063.gb658018a.el8 ceph 19 M 2021-03-15T22:10:26.386 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-selinux x86_64 2:17.0.0-2063.gb658018a.el8 ceph 23 k 2021-03-15T22:10:26.386 INFO:teuthology.orchestra.run.smithi125.stdout: daxctl-libs x86_64 67-2.el8 CentOS-Base 40 k 2021-03-15T22:10:26.386 INFO:teuthology.orchestra.run.smithi125.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-03-15T22:10:26.386 INFO:teuthology.orchestra.run.smithi125.stdout: gperftools-libs x86_64 2.8-1.el8 epel 306 k 2021-03-15T22:10:26.387 INFO:teuthology.orchestra.run.smithi125.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-03-15T22:10:26.387 INFO:teuthology.orchestra.run.smithi125.stdout: libcephfs2 x86_64 2:17.0.0-2063.gb658018a.el8 ceph 749 k 2021-03-15T22:10:26.387 INFO:teuthology.orchestra.run.smithi125.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-03-15T22:10:26.387 INFO:teuthology.orchestra.run.smithi125.stdout: libpmemobj x86_64 1.6.1-1.el8 CentOS-AppStream 145 k 2021-03-15T22:10:26.387 INFO:teuthology.orchestra.run.smithi125.stdout: librabbitmq x86_64 0.9.0-2.el8 CentOS-Base 47 k 2021-03-15T22:10:26.388 INFO:teuthology.orchestra.run.smithi125.stdout: libradosstriper1 x86_64 2:17.0.0-2063.gb658018a.el8 ceph 457 k 2021-03-15T22:10:26.388 INFO:teuthology.orchestra.run.smithi125.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-03-15T22:10:26.388 INFO:teuthology.orchestra.run.smithi125.stdout: librgw2 x86_64 2:17.0.0-2063.gb658018a.el8 ceph 3.7 M 2021-03-15T22:10:26.388 INFO:teuthology.orchestra.run.smithi125.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-03-15T22:10:26.389 INFO:teuthology.orchestra.run.smithi125.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-03-15T22:10:26.389 INFO:teuthology.orchestra.run.smithi125.stdout: ndctl-libs x86_64 67-2.el8 CentOS-Base 74 k 2021-03-15T22:10:26.389 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-argparse 2021-03-15T22:10:26.389 INFO:teuthology.orchestra.run.smithi125.stdout: x86_64 2:17.0.0-2063.gb658018a.el8 ceph 42 k 2021-03-15T22:10:26.389 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-common x86_64 2:17.0.0-2063.gb658018a.el8 ceph 74 k 2021-03-15T22:10:26.390 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cephfs x86_64 2:17.0.0-2063.gb658018a.el8 ceph 176 k 2021-03-15T22:10:26.390 INFO:teuthology.orchestra.run.smithi125.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-03-15T22:10:26.390 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rados x86_64 2:17.0.0-2063.gb658018a.el8 ceph 365 k 2021-03-15T22:10:26.390 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rbd x86_64 2:17.0.0-2063.gb658018a.el8 ceph 346 k 2021-03-15T22:10:26.391 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rgw x86_64 2:17.0.0-2063.gb658018a.el8 ceph 110 k 2021-03-15T22:10:26.391 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:10:26.391 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:10:26.391 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:10:26.391 INFO:teuthology.orchestra.run.smithi125.stdout:Install 25 Packages 2021-03-15T22:10:26.392 INFO:teuthology.orchestra.run.smithi125.stdout:Upgrade 3 Packages 2021-03-15T22:10:26.392 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:10:26.393 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 48 M 2021-03-15T22:10:26.393 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:10:26.963 INFO:teuthology.orchestra.run.smithi076.stdout:(1/28): ceph-base-17.0.0-2063.gb658018a.el8.x86 7.3 MB/s | 5.1 MB 00:00 2021-03-15T22:10:27.011 INFO:teuthology.orchestra.run.smithi076.stdout:(2/28): ceph-selinux-17.0.0-2063.gb658018a.el8. 497 kB/s | 23 kB 00:00 2021-03-15T22:10:27.138 INFO:teuthology.orchestra.run.smithi076.stdout:(3/28): libcephfs2-17.0.0-2063.gb658018a.el8.x8 5.8 MB/s | 749 kB 00:00 2021-03-15T22:10:27.208 INFO:teuthology.orchestra.run.smithi076.stdout:(4/28): libradosstriper1-17.0.0-2063.gb658018a. 6.3 MB/s | 457 kB 00:00 2021-03-15T22:10:27.517 INFO:teuthology.orchestra.run.smithi076.stdout:(5/28): ceph-radosgw-17.0.0-2063.gb658018a.el8. 7.3 MB/s | 9.1 MB 00:01 2021-03-15T22:10:27.565 INFO:teuthology.orchestra.run.smithi076.stdout:(6/28): python3-ceph-argparse-17.0.0-2063.gb658 890 kB/s | 42 kB 00:00 2021-03-15T22:10:27.618 INFO:teuthology.orchestra.run.smithi076.stdout:(7/28): python3-ceph-common-17.0.0-2063.gb65801 1.4 MB/s | 74 kB 00:00 2021-03-15T22:10:27.664 INFO:teuthology.orchestra.run.smithi076.stdout:(8/28): librgw2-17.0.0-2063.gb658018a.el8.x86_6 8.1 MB/s | 3.7 MB 00:00 2021-03-15T22:10:27.671 INFO:teuthology.orchestra.run.smithi076.stdout:(9/28): python3-cephfs-17.0.0-2063.gb658018a.el 3.2 MB/s | 176 kB 00:00 2021-03-15T22:10:27.715 INFO:teuthology.orchestra.run.smithi076.stdout:(10/28): python3-rados-17.0.0-2063.gb658018a.el 7.0 MB/s | 365 kB 00:00 2021-03-15T22:10:27.729 INFO:teuthology.orchestra.run.smithi076.stdout:(11/28): python3-rbd-17.0.0-2063.gb658018a.el8. 5.8 MB/s | 346 kB 00:00 2021-03-15T22:10:27.764 INFO:teuthology.orchestra.run.smithi076.stdout:(12/28): python3-rgw-17.0.0-2063.gb658018a.el8. 2.2 MB/s | 110 kB 00:00 2021-03-15T22:10:28.043 INFO:teuthology.orchestra.run.smithi076.stdout:(13/28): libpmemobj-1.6.1-1.el8.x86_64.rpm 462 kB/s | 145 kB 00:00 2021-03-15T22:10:28.079 INFO:teuthology.orchestra.run.smithi076.stdout:(14/28): librdkafka-0.11.4-1.el8.x86_64.rpm 1.1 MB/s | 353 kB 00:00 2021-03-15T22:10:28.118 INFO:teuthology.orchestra.run.smithi076.stdout:(15/28): python3-prettytable-0.7.2-14.el8.noarc 1.1 MB/s | 44 kB 00:00 2021-03-15T22:10:28.128 INFO:teuthology.orchestra.run.smithi076.stdout:(16/28): lttng-ust-2.8.1-11.el8.x86_64.rpm 3.0 MB/s | 259 kB 00:00 2021-03-15T22:10:28.151 INFO:teuthology.orchestra.run.smithi076.stdout:(17/28): daxctl-libs-67-2.el8.x86_64.rpm 1.2 MB/s | 40 kB 00:00 2021-03-15T22:10:28.268 INFO:teuthology.orchestra.run.smithi076.stdout:(18/28): ceph-common-17.0.0-2063.gb658018a.el8. 9.7 MB/s | 19 MB 00:02 2021-03-15T22:10:28.320 INFO:teuthology.orchestra.run.smithi076.stdout:(19/28): librabbitmq-0.9.0-2.el8.x86_64.rpm 244 kB/s | 47 kB 00:00 2021-03-15T22:10:28.320 INFO:teuthology.orchestra.run.smithi076.stdout:(20/28): ndctl-libs-67-2.el8.x86_64.rpm 436 kB/s | 74 kB 00:00 2021-03-15T22:10:28.330 INFO:teuthology.orchestra.run.smithi125.stdout:(1/28): ceph-base-17.0.0-2063.gb658018a.el8.x86 2.6 MB/s | 5.1 MB 00:01 2021-03-15T22:10:28.356 INFO:teuthology.orchestra.run.smithi125.stdout:(2/28): ceph-selinux-17.0.0-2063.gb658018a.el8. 905 kB/s | 23 kB 00:00 2021-03-15T22:10:28.411 INFO:teuthology.orchestra.run.smithi076.stdout:(21/28): leveldb-1.22-1.el8.x86_64.rpm 1.9 MB/s | 181 kB 00:00 2021-03-15T22:10:28.433 INFO:teuthology.orchestra.run.smithi076.stdout:(22/28): fmt-6.2.1-1.el8.x86_64.rpm 707 kB/s | 116 kB 00:00 2021-03-15T22:10:28.443 INFO:teuthology.orchestra.run.smithi076.stdout:(23/28): liboath-2.6.2-3.el8.x86_64.rpm 1.8 MB/s | 59 kB 00:00 2021-03-15T22:10:28.453 INFO:teuthology.orchestra.run.smithi076.stdout:(24/28): gperftools-libs-2.8-1.el8.x86_64.rpm 2.3 MB/s | 306 kB 00:00 2021-03-15T22:10:28.463 INFO:teuthology.orchestra.run.smithi125.stdout:(3/28): libcephfs2-17.0.0-2063.gb658018a.el8.x8 6.9 MB/s | 749 kB 00:00 2021-03-15T22:10:28.469 INFO:teuthology.orchestra.run.smithi076.stdout:(25/28): libunwind-1.3.1-3.el8.x86_64.rpm 2.0 MB/s | 75 kB 00:00 2021-03-15T22:10:28.533 INFO:teuthology.orchestra.run.smithi076.stdout:(26/28): libpmem-1.6.1-1.el8.x86_64.rpm 1.2 MB/s | 79 kB 00:00 2021-03-15T22:10:28.581 INFO:teuthology.orchestra.run.smithi125.stdout:(4/28): libradosstriper1-17.0.0-2063.gb658018a. 3.8 MB/s | 457 kB 00:00 2021-03-15T22:10:28.912 INFO:teuthology.orchestra.run.smithi125.stdout:(5/28): ceph-radosgw-17.0.0-2063.gb658018a.el8. 3.6 MB/s | 9.1 MB 00:02 2021-03-15T22:10:28.932 INFO:teuthology.orchestra.run.smithi076.stdout:(27/28): librbd1-17.0.0-2063.gb658018a.el8.x86_ 6.9 MB/s | 3.3 MB 00:00 2021-03-15T22:10:28.961 INFO:teuthology.orchestra.run.smithi125.stdout:(6/28): python3-ceph-argparse-17.0.0-2063.gb658 889 kB/s | 42 kB 00:00 2021-03-15T22:10:28.999 INFO:teuthology.orchestra.run.smithi125.stdout:(7/28): python3-ceph-common-17.0.0-2063.gb65801 1.9 MB/s | 74 kB 00:00 2021-03-15T22:10:29.034 INFO:teuthology.orchestra.run.smithi076.stdout:(28/28): librados2-17.0.0-2063.gb658018a.el8.x8 5.5 MB/s | 3.2 MB 00:00 2021-03-15T22:10:29.038 INFO:teuthology.orchestra.run.smithi125.stdout:(8/28): python3-cephfs-17.0.0-2063.gb658018a.el 4.4 MB/s | 176 kB 00:00 2021-03-15T22:10:29.038 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:10:29.039 INFO:teuthology.orchestra.run.smithi076.stdout:Total 17 MB/s | 48 MB 00:02 2021-03-15T22:10:29.046 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:10:29.098 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:10:29.098 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:10:29.103 INFO:teuthology.orchestra.run.smithi125.stdout:(9/28): python3-rados-17.0.0-2063.gb658018a.el8 5.5 MB/s | 365 kB 00:00 2021-03-15T22:10:29.169 INFO:teuthology.orchestra.run.smithi125.stdout:(10/28): python3-rbd-17.0.0-2063.gb658018a.el8. 5.1 MB/s | 346 kB 00:00 2021-03-15T22:10:29.201 INFO:teuthology.orchestra.run.smithi125.stdout:(11/28): python3-rgw-17.0.0-2063.gb658018a.el8. 3.4 MB/s | 110 kB 00:00 2021-03-15T22:10:29.237 INFO:teuthology.orchestra.run.smithi125.stdout:(12/28): librgw2-17.0.0-2063.gb658018a.el8.x86_ 5.6 MB/s | 3.7 MB 00:00 2021-03-15T22:10:29.371 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:10:29.373 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:10:29.436 INFO:teuthology.orchestra.run.smithi125.stdout:(13/28): libpmemobj-1.6.1-1.el8.x86_64.rpm 620 kB/s | 145 kB 00:00 2021-03-15T22:10:29.538 INFO:teuthology.orchestra.run.smithi125.stdout:(14/28): librdkafka-0.11.4-1.el8.x86_64.rpm 1.1 MB/s | 353 kB 00:00 2021-03-15T22:10:29.539 INFO:teuthology.orchestra.run.smithi125.stdout:(15/28): lttng-ust-2.8.1-11.el8.x86_64.rpm 2.5 MB/s | 259 kB 00:00 2021-03-15T22:10:29.553 INFO:teuthology.orchestra.run.smithi125.stdout:(16/28): python3-prettytable-0.7.2-14.el8.noarc 2.8 MB/s | 44 kB 00:00 2021-03-15T22:10:29.555 INFO:teuthology.orchestra.run.smithi125.stdout:(17/28): daxctl-libs-67-2.el8.x86_64.rpm 2.5 MB/s | 40 kB 00:00 2021-03-15T22:10:29.589 INFO:teuthology.orchestra.run.smithi125.stdout:(18/28): ndctl-libs-67-2.el8.x86_64.rpm 2.1 MB/s | 74 kB 00:00 2021-03-15T22:10:29.590 INFO:teuthology.orchestra.run.smithi125.stdout:(19/28): librabbitmq-0.9.0-2.el8.x86_64.rpm 1.2 MB/s | 47 kB 00:00 2021-03-15T22:10:29.755 INFO:teuthology.orchestra.run.smithi125.stdout:(20/28): ceph-common-17.0.0-2063.gb658018a.el8. 5.8 MB/s | 19 MB 00:03 2021-03-15T22:10:29.756 INFO:teuthology.orchestra.run.smithi125.stdout:(21/28): fmt-6.2.1-1.el8.x86_64.rpm 700 kB/s | 116 kB 00:00 2021-03-15T22:10:29.773 INFO:teuthology.orchestra.run.smithi125.stdout:(22/28): liboath-2.6.2-3.el8.x86_64.rpm 3.4 MB/s | 59 kB 00:00 2021-03-15T22:10:29.776 INFO:teuthology.orchestra.run.smithi125.stdout:(23/28): gperftools-libs-2.8-1.el8.x86_64.rpm 1.6 MB/s | 306 kB 00:00 2021-03-15T22:10:29.790 INFO:teuthology.orchestra.run.smithi125.stdout:(24/28): libunwind-1.3.1-3.el8.x86_64.rpm 4.3 MB/s | 75 kB 00:00 2021-03-15T22:10:29.840 INFO:teuthology.orchestra.run.smithi125.stdout:(25/28): leveldb-1.22-1.el8.x86_64.rpm 2.1 MB/s | 181 kB 00:00 2021-03-15T22:10:29.887 INFO:teuthology.orchestra.run.smithi125.stdout:(26/28): libpmem-1.6.1-1.el8.x86_64.rpm 1.6 MB/s | 79 kB 00:00 2021-03-15T22:10:30.007 INFO:teuthology.orchestra.run.smithi125.stdout:(27/28): librbd1-17.0.0-2063.gb658018a.el8.x86_ 15 MB/s | 3.3 MB 00:00 2021-03-15T22:10:30.079 INFO:teuthology.orchestra.run.smithi125.stdout:(28/28): librados2-17.0.0-2063.gb658018a.el8.x8 11 MB/s | 3.2 MB 00:00 2021-03-15T22:10:30.082 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:10:30.083 INFO:teuthology.orchestra.run.smithi125.stdout:Total 13 MB/s | 48 MB 00:03 2021-03-15T22:10:30.090 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:10:30.138 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:10:30.139 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:10:30.417 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:10:30.419 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:10:30.459 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:10:30.636 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/31 2021-03-15T22:10:30.812 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/31 2021-03-15T22:10:31.012 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/31 2021-03-15T22:10:31.398 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/31 2021-03-15T22:10:31.530 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:10:31.554 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librados2-2:17.0.0-2063.gb658018a.el8.x86_64 4/31 2021-03-15T22:10:31.636 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:17.0.0-2063.gb658018a.el8.x86_64 4/31 2021-03-15T22:10:31.729 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/31 2021-03-15T22:10:31.844 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rados-2:17.0.0-2063.gb658018a.el8.x86_64 5/31 2021-03-15T22:10:31.885 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/31 2021-03-15T22:10:31.996 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs2-2:17.0.0-2063.gb658018a.el8.x86_64 6/31 2021-03-15T22:10:32.062 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs2-2:17.0.0-2063.gb658018a.el8.x86_64 6/31 2021-03-15T22:10:32.062 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/31 2021-03-15T22:10:32.212 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : libpmem-1.6.1-1.el8.x86_64 7/31 2021-03-15T22:10:32.275 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libpmem-1.6.1-1.el8.x86_64 7/31 2021-03-15T22:10:32.420 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 8/31 2021-03-15T22:10:32.440 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/31 2021-03-15T22:10:32.604 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/31 2021-03-15T22:10:32.612 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : librados2-2:17.0.0-2063.gb658018a.el8.x86_64 4/31 2021-03-15T22:10:32.693 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librados2-2:17.0.0-2063.gb658018a.el8.x86_64 4/31 2021-03-15T22:10:32.903 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-rados-2:17.0.0-2063.gb658018a.el8.x86_64 5/31 2021-03-15T22:10:33.010 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/31 2021-03-15T22:10:33.054 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libcephfs2-2:17.0.0-2063.gb658018a.el8.x86_64 6/31 2021-03-15T22:10:33.144 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libcephfs2-2:17.0.0-2063.gb658018a.el8.x86_64 6/31 2021-03-15T22:10:33.162 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librgw2-2:17.0.0-2063.gb658018a.el8.x86_64 10/31 2021-03-15T22:10:33.248 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librgw2-2:17.0.0-2063.gb658018a.el8.x86_64 10/31 2021-03-15T22:10:33.312 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : libpmem-1.6.1-1.el8.x86_64 7/31 2021-03-15T22:10:33.374 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libpmem-1.6.1-1.el8.x86_64 7/31 2021-03-15T22:10:33.451 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/31 2021-03-15T22:10:33.540 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 8/31 2021-03-15T22:10:33.570 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : daxctl-libs-67-2.el8.x86_64 12/31 2021-03-15T22:10:33.639 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: daxctl-libs-67-2.el8.x86_64 12/31 2021-03-15T22:10:33.687 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/31 2021-03-15T22:10:33.755 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-argparse-2:17.0.0-2063.gb658018a.el8. 13/31 2021-03-15T22:10:33.878 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cephfs-2:17.0.0-2063.gb658018a.el8.x86_64 14/31 2021-03-15T22:10:34.020 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ndctl-libs-67-2.el8.x86_64 15/31 2021-03-15T22:10:34.098 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/31 2021-03-15T22:10:34.106 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ndctl-libs-67-2.el8.x86_64 15/31 2021-03-15T22:10:34.245 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : librgw2-2:17.0.0-2063.gb658018a.el8.x86_64 10/31 2021-03-15T22:10:34.253 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 16/31 2021-03-15T22:10:34.330 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librgw2-2:17.0.0-2063.gb658018a.el8.x86_64 10/31 2021-03-15T22:10:34.509 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/31 2021-03-15T22:10:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 16/31 2021-03-15T22:10:34.670 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : daxctl-libs-67-2.el8.x86_64 12/31 2021-03-15T22:10:34.728 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: daxctl-libs-67-2.el8.x86_64 12/31 2021-03-15T22:10:34.795 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librbd1-2:17.0.0-2063.gb658018a.el8.x86_64 17/31 2021-03-15T22:10:34.855 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-ceph-argparse-2:17.0.0-2063.gb658018a.el8. 13/31 2021-03-15T22:10:34.896 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:17.0.0-2063.gb658018a.el8.x86_64 17/31 2021-03-15T22:10:35.012 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-cephfs-2:17.0.0-2063.gb658018a.el8.x86_64 14/31 2021-03-15T22:10:35.082 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rbd-2:17.0.0-2063.gb658018a.el8.x86_64 18/31 2021-03-15T22:10:35.171 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ndctl-libs-67-2.el8.x86_64 15/31 2021-03-15T22:10:35.228 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ndctl-libs-67-2.el8.x86_64 15/31 2021-03-15T22:10:35.238 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rgw-2:17.0.0-2063.gb658018a.el8.x86_64 19/31 2021-03-15T22:10:35.354 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 16/31 2021-03-15T22:10:35.378 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libradosstriper1-2:17.0.0-2063.gb658018a.el8.x86_6 20/31 2021-03-15T22:10:35.431 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libradosstriper1-2:17.0.0-2063.gb658018a.el8.x86_6 20/31 2021-03-15T22:10:35.579 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 21/31 2021-03-15T22:10:35.753 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : gperftools-libs-2.8-1.el8.x86_64 22/31 2021-03-15T22:10:35.763 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 16/31 2021-03-15T22:10:35.912 INFO:teuthology.orchestra.run.smithi125.stdout: Upgrading : librbd1-2:17.0.0-2063.gb658018a.el8.x86_64 17/31 2021-03-15T22:10:35.937 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 23/31 2021-03-15T22:10:35.992 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librbd1-2:17.0.0-2063.gb658018a.el8.x86_64 17/31 2021-03-15T22:10:36.062 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-common-2:17.0.0-2063.gb658018a.el8.x8 24/31 2021-03-15T22:10:36.198 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-rbd-2:17.0.0-2063.gb658018a.el8.x86_64 18/31 2021-03-15T22:10:36.397 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-rgw-2:17.0.0-2063.gb658018a.el8.x86_64 19/31 2021-03-15T22:10:36.520 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libradosstriper1-2:17.0.0-2063.gb658018a.el8.x86_6 20/31 2021-03-15T22:10:36.594 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libradosstriper1-2:17.0.0-2063.gb658018a.el8.x86_6 20/31 2021-03-15T22:10:36.762 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 21/31 2021-03-15T22:10:36.995 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : gperftools-libs-2.8-1.el8.x86_64 22/31 2021-03-15T22:10:37.145 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 23/31 2021-03-15T22:10:37.288 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-ceph-common-2:17.0.0-2063.gb658018a.el8.x8 24/31 2021-03-15T22:10:39.383 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 25/31 2021-03-15T22:10:39.553 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 25/31 2021-03-15T22:10:40.230 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 25/31 2021-03-15T22:10:40.606 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-base-2:17.0.0-2063.gb658018a.el8.x86_64 26/31 2021-03-15T22:10:40.643 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 25/31 2021-03-15T22:10:40.676 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-base-2:17.0.0-2063.gb658018a.el8.x86_64 26/31 2021-03-15T22:10:40.786 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-selinux-2:17.0.0-2063.gb658018a.el8.x86_64 27/31 2021-03-15T22:10:40.870 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 25/31 2021-03-15T22:10:41.555 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 25/31 2021-03-15T22:10:41.801 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-base-2:17.0.0-2063.gb658018a.el8.x86_64 26/31 2021-03-15T22:10:41.886 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-base-2:17.0.0-2063.gb658018a.el8.x86_64 26/31 2021-03-15T22:10:41.986 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-selinux-2:17.0.0-2063.gb658018a.el8.x86_64 27/31 2021-03-15T22:10:57.656 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-selinux-2:17.0.0-2063.gb658018a.el8.x86_64 27/31 2021-03-15T22:10:57.656 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /sys 2021-03-15T22:10:57.657 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /proc 2021-03-15T22:10:57.657 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /dev 2021-03-15T22:10:57.657 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /run 2021-03-15T22:10:57.657 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /mnt 2021-03-15T22:10:57.658 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /var/tmp 2021-03-15T22:10:57.658 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /home 2021-03-15T22:10:57.658 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /tmp 2021-03-15T22:10:57.658 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /dev 2021-03-15T22:10:57.659 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:10:58.812 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-radosgw-2:17.0.0-2063.gb658018a.el8.x86_64 28/31 2021-03-15T22:10:58.878 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-2063.gb658018a.el8.x86_64 28/31 2021-03-15T22:10:58.879 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 29/31 2021-03-15T22:10:59.025 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-selinux-2:17.0.0-2063.gb658018a.el8.x86_64 27/31 2021-03-15T22:10:59.026 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /sys 2021-03-15T22:10:59.026 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /proc 2021-03-15T22:10:59.026 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /dev 2021-03-15T22:10:59.026 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /run 2021-03-15T22:10:59.027 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /mnt 2021-03-15T22:10:59.027 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /var/tmp 2021-03-15T22:10:59.027 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /home 2021-03-15T22:10:59.028 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /tmp 2021-03-15T22:10:59.028 INFO:teuthology.orchestra.run.smithi125.stdout:skipping the directory /dev 2021-03-15T22:10:59.028 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:10:59.078 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 29/31 2021-03-15T22:10:59.078 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 30/31 2021-03-15T22:10:59.312 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 30/31 2021-03-15T22:10:59.313 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : libpmem-1.6-2.el8.x86_64 31/31 2021-03-15T22:11:00.001 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libpmem-1.6-2.el8.x86_64 31/31 2021-03-15T22:11:00.002 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-base-2:17.0.0-2063.gb658018a.el8.x86_64 1/31 2021-03-15T22:11:00.002 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 2/31 2021-03-15T22:11:00.003 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-radosgw-2:17.0.0-2063.gb658018a.el8.x86_64 3/31 2021-03-15T22:11:00.003 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-selinux-2:17.0.0-2063.gb658018a.el8.x86_64 4/31 2021-03-15T22:11:00.004 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs2-2:17.0.0-2063.gb658018a.el8.x86_64 5/31 2021-03-15T22:11:00.004 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libradosstriper1-2:17.0.0-2063.gb658018a.el8.x86_6 6/31 2021-03-15T22:11:00.005 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librgw2-2:17.0.0-2063.gb658018a.el8.x86_64 7/31 2021-03-15T22:11:00.006 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-argparse-2:17.0.0-2063.gb658018a.el8. 8/31 2021-03-15T22:11:00.006 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-common-2:17.0.0-2063.gb658018a.el8.x8 9/31 2021-03-15T22:11:00.007 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cephfs-2:17.0.0-2063.gb658018a.el8.x86_64 10/31 2021-03-15T22:11:00.008 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rados-2:17.0.0-2063.gb658018a.el8.x86_64 11/31 2021-03-15T22:11:00.008 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rbd-2:17.0.0-2063.gb658018a.el8.x86_64 12/31 2021-03-15T22:11:00.008 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rgw-2:17.0.0-2063.gb658018a.el8.x86_64 13/31 2021-03-15T22:11:00.009 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/31 2021-03-15T22:11:00.009 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 15/31 2021-03-15T22:11:00.009 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/31 2021-03-15T22:11:00.009 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/31 2021-03-15T22:11:00.009 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : daxctl-libs-67-2.el8.x86_64 18/31 2021-03-15T22:11:00.010 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 19/31 2021-03-15T22:11:00.010 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ndctl-libs-67-2.el8.x86_64 20/31 2021-03-15T22:11:00.010 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/31 2021-03-15T22:11:00.010 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : gperftools-libs-2.8-1.el8.x86_64 22/31 2021-03-15T22:11:00.011 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/31 2021-03-15T22:11:00.011 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/31 2021-03-15T22:11:00.011 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/31 2021-03-15T22:11:00.012 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:17.0.0-2063.gb658018a.el8.x86_64 26/31 2021-03-15T22:11:00.012 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/31 2021-03-15T22:11:00.012 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:17.0.0-2063.gb658018a.el8.x86_64 28/31 2021-03-15T22:11:00.012 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/31 2021-03-15T22:11:00.013 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libpmem-1.6.1-1.el8.x86_64 30/31 2021-03-15T22:11:00.165 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-radosgw-2:17.0.0-2063.gb658018a.el8.x86_64 28/31 2021-03-15T22:11:00.235 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-2063.gb658018a.el8.x86_64 28/31 2021-03-15T22:11:00.235 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 29/31 2021-03-15T22:11:00.435 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 29/31 2021-03-15T22:11:00.435 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 30/31 2021-03-15T22:11:00.657 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 30/31 2021-03-15T22:11:00.658 INFO:teuthology.orchestra.run.smithi125.stdout: Cleanup : libpmem-1.6-2.el8.x86_64 31/31 2021-03-15T22:11:00.904 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libpmem-1.6-2.el8.x86_64 31/31 2021-03-15T22:11:00.904 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:00.904 INFO:teuthology.orchestra.run.smithi076.stdout:Upgraded: 2021-03-15T22:11:00.905 INFO:teuthology.orchestra.run.smithi076.stdout: libpmem-1.6.1-1.el8.x86_64 2021-03-15T22:11:00.905 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.905 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.905 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:00.906 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:11:00.906 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.906 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.906 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.907 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.907 INFO:teuthology.orchestra.run.smithi076.stdout: daxctl-libs-67-2.el8.x86_64 2021-03-15T22:11:00.907 INFO:teuthology.orchestra.run.smithi076.stdout: fmt-6.2.1-1.el8.x86_64 2021-03-15T22:11:00.907 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs-2.8-1.el8.x86_64 2021-03-15T22:11:00.907 INFO:teuthology.orchestra.run.smithi076.stdout: leveldb-1.22-1.el8.x86_64 2021-03-15T22:11:00.908 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.908 INFO:teuthology.orchestra.run.smithi076.stdout: liboath-2.6.2-3.el8.x86_64 2021-03-15T22:11:00.908 INFO:teuthology.orchestra.run.smithi076.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-03-15T22:11:00.908 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-03-15T22:11:00.909 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.909 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-03-15T22:11:00.909 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.909 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind-1.3.1-3.el8.x86_64 2021-03-15T22:11:00.909 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-03-15T22:11:00.910 INFO:teuthology.orchestra.run.smithi076.stdout: ndctl-libs-67-2.el8.x86_64 2021-03-15T22:11:00.910 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.911 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.911 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.911 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-03-15T22:11:00.912 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.912 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.913 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:00.913 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:00.914 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:11:01.385 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libpmem-1.6-2.el8.x86_64 31/31 2021-03-15T22:11:01.385 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-base-2:17.0.0-2063.gb658018a.el8.x86_64 1/31 2021-03-15T22:11:01.386 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 2/31 2021-03-15T22:11:01.386 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-radosgw-2:17.0.0-2063.gb658018a.el8.x86_64 3/31 2021-03-15T22:11:01.386 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-selinux-2:17.0.0-2063.gb658018a.el8.x86_64 4/31 2021-03-15T22:11:01.386 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libcephfs2-2:17.0.0-2063.gb658018a.el8.x86_64 5/31 2021-03-15T22:11:01.387 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libradosstriper1-2:17.0.0-2063.gb658018a.el8.x86_6 6/31 2021-03-15T22:11:01.387 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librgw2-2:17.0.0-2063.gb658018a.el8.x86_64 7/31 2021-03-15T22:11:01.387 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-ceph-argparse-2:17.0.0-2063.gb658018a.el8. 8/31 2021-03-15T22:11:01.387 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-ceph-common-2:17.0.0-2063.gb658018a.el8.x8 9/31 2021-03-15T22:11:01.387 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-cephfs-2:17.0.0-2063.gb658018a.el8.x86_64 10/31 2021-03-15T22:11:01.388 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-rados-2:17.0.0-2063.gb658018a.el8.x86_64 11/31 2021-03-15T22:11:01.388 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-rbd-2:17.0.0-2063.gb658018a.el8.x86_64 12/31 2021-03-15T22:11:01.388 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-rgw-2:17.0.0-2063.gb658018a.el8.x86_64 13/31 2021-03-15T22:11:01.388 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/31 2021-03-15T22:11:01.389 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 15/31 2021-03-15T22:11:01.389 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/31 2021-03-15T22:11:01.389 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/31 2021-03-15T22:11:01.389 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : daxctl-libs-67-2.el8.x86_64 18/31 2021-03-15T22:11:01.389 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 19/31 2021-03-15T22:11:01.390 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ndctl-libs-67-2.el8.x86_64 20/31 2021-03-15T22:11:01.390 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/31 2021-03-15T22:11:01.390 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : gperftools-libs-2.8-1.el8.x86_64 22/31 2021-03-15T22:11:01.390 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/31 2021-03-15T22:11:01.391 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/31 2021-03-15T22:11:01.391 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/31 2021-03-15T22:11:01.391 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librados2-2:17.0.0-2063.gb658018a.el8.x86_64 26/31 2021-03-15T22:11:01.391 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/31 2021-03-15T22:11:01.391 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librbd1-2:17.0.0-2063.gb658018a.el8.x86_64 28/31 2021-03-15T22:11:01.392 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/31 2021-03-15T22:11:01.392 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libpmem-1.6.1-1.el8.x86_64 30/31 2021-03-15T22:11:01.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-test 2021-03-15T22:11:01.870 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:36 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:02.326 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:11:02.327 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:02.327 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T22:11:02.327 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:02.327 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:11:02.328 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test x86_64 2:17.0.0-2063.gb658018a.el8 ceph 40 M 2021-03-15T22:11:02.328 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T22:11:02.328 INFO:teuthology.orchestra.run.smithi076.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-03-15T22:11:02.328 INFO:teuthology.orchestra.run.smithi076.stdout: libxslt x86_64 1.1.32-5.el8 CentOS-Base 250 k 2021-03-15T22:11:02.329 INFO:teuthology.orchestra.run.smithi076.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-03-15T22:11:02.329 INFO:teuthology.orchestra.run.smithi076.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-03-15T22:11:02.329 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-03-15T22:11:02.329 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:02.329 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:11:02.330 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:02.330 INFO:teuthology.orchestra.run.smithi076.stdout:Install 6 Packages 2021-03-15T22:11:02.330 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:02.331 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 41 M 2021-03-15T22:11:02.331 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 229 M 2021-03-15T22:11:02.331 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:11:02.359 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libpmem-1.6-2.el8.x86_64 31/31 2021-03-15T22:11:02.359 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:02.359 INFO:teuthology.orchestra.run.smithi125.stdout:Upgraded: 2021-03-15T22:11:02.360 INFO:teuthology.orchestra.run.smithi125.stdout: libpmem-1.6.1-1.el8.x86_64 2021-03-15T22:11:02.360 INFO:teuthology.orchestra.run.smithi125.stdout: librados2-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.360 INFO:teuthology.orchestra.run.smithi125.stdout: librbd1-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.360 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:02.361 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:11:02.361 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-base-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.361 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.361 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-radosgw-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.361 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-selinux-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.362 INFO:teuthology.orchestra.run.smithi125.stdout: daxctl-libs-67-2.el8.x86_64 2021-03-15T22:11:02.362 INFO:teuthology.orchestra.run.smithi125.stdout: fmt-6.2.1-1.el8.x86_64 2021-03-15T22:11:02.362 INFO:teuthology.orchestra.run.smithi125.stdout: gperftools-libs-2.8-1.el8.x86_64 2021-03-15T22:11:02.362 INFO:teuthology.orchestra.run.smithi125.stdout: leveldb-1.22-1.el8.x86_64 2021-03-15T22:11:02.363 INFO:teuthology.orchestra.run.smithi125.stdout: libcephfs2-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.363 INFO:teuthology.orchestra.run.smithi125.stdout: liboath-2.6.2-3.el8.x86_64 2021-03-15T22:11:02.363 INFO:teuthology.orchestra.run.smithi125.stdout: libpmemobj-1.6.1-1.el8.x86_64 2021-03-15T22:11:02.363 INFO:teuthology.orchestra.run.smithi125.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-03-15T22:11:02.364 INFO:teuthology.orchestra.run.smithi125.stdout: libradosstriper1-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.364 INFO:teuthology.orchestra.run.smithi125.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-03-15T22:11:02.364 INFO:teuthology.orchestra.run.smithi125.stdout: librgw2-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.364 INFO:teuthology.orchestra.run.smithi125.stdout: libunwind-1.3.1-3.el8.x86_64 2021-03-15T22:11:02.364 INFO:teuthology.orchestra.run.smithi125.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-03-15T22:11:02.365 INFO:teuthology.orchestra.run.smithi125.stdout: ndctl-libs-67-2.el8.x86_64 2021-03-15T22:11:02.365 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-argparse-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.365 INFO:teuthology.orchestra.run.smithi125.stdout: python3-ceph-common-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.365 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cephfs-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.366 INFO:teuthology.orchestra.run.smithi125.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-03-15T22:11:02.366 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rados-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.366 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rbd-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.366 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rgw-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:02.366 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:02.367 INFO:teuthology.orchestra.run.smithi076.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.4 MB/s | 161 kB 00:00 2021-03-15T22:11:02.367 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:11:02.368 INFO:teuthology.orchestra.run.smithi076.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.8 MB/s | 187 kB 00:00 2021-03-15T22:11:02.373 INFO:teuthology.orchestra.run.smithi076.stdout:(3/6): socat-1.7.3.3-2.el8.x86_64.rpm 42 MB/s | 302 kB 00:00 2021-03-15T22:11:02.374 INFO:teuthology.orchestra.run.smithi076.stdout:(4/6): libxslt-1.1.32-5.el8.x86_64.rpm 42 MB/s | 250 kB 00:00 2021-03-15T22:11:02.466 INFO:teuthology.orchestra.run.smithi076.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 746 kB/s | 69 kB 00:00 2021-03-15T22:11:02.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph-test 2021-03-15T22:11:03.034 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:38 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:03.472 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:11:03.474 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:03.474 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:11:03.474 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:03.474 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:11:03.475 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-test x86_64 2:17.0.0-2063.gb658018a.el8 ceph 40 M 2021-03-15T22:11:03.475 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:11:03.475 INFO:teuthology.orchestra.run.smithi125.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-03-15T22:11:03.475 INFO:teuthology.orchestra.run.smithi125.stdout: libxslt x86_64 1.1.32-5.el8 CentOS-Base 250 k 2021-03-15T22:11:03.475 INFO:teuthology.orchestra.run.smithi125.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-03-15T22:11:03.476 INFO:teuthology.orchestra.run.smithi125.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-03-15T22:11:03.476 INFO:teuthology.orchestra.run.smithi125.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-03-15T22:11:03.476 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:03.476 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:11:03.477 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:03.477 INFO:teuthology.orchestra.run.smithi125.stdout:Install 6 Packages 2021-03-15T22:11:03.477 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:03.478 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 41 M 2021-03-15T22:11:03.478 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 229 M 2021-03-15T22:11:03.479 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:11:03.566 INFO:teuthology.orchestra.run.smithi125.stdout:(1/6): oniguruma-6.8.2-2.el8.x86_64.rpm 2.1 MB/s | 187 kB 00:00 2021-03-15T22:11:03.571 INFO:teuthology.orchestra.run.smithi125.stdout:(2/6): socat-1.7.3.3-2.el8.x86_64.rpm 57 MB/s | 302 kB 00:00 2021-03-15T22:11:03.572 INFO:teuthology.orchestra.run.smithi125.stdout:(3/6): jq-1.5-12.el8.x86_64.rpm 1.7 MB/s | 161 kB 00:00 2021-03-15T22:11:03.580 INFO:teuthology.orchestra.run.smithi125.stdout:(4/6): libxslt-1.1.32-5.el8.x86_64.rpm 26 MB/s | 250 kB 00:00 2021-03-15T22:11:03.650 INFO:teuthology.orchestra.run.smithi125.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 882 kB/s | 69 kB 00:00 2021-03-15T22:11:04.282 INFO:teuthology.orchestra.run.smithi076.stdout:(6/6): ceph-test-17.0.0-2063.gb658018a.el8.x86_ 21 MB/s | 40 MB 00:01 2021-03-15T22:11:04.285 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:04.286 INFO:teuthology.orchestra.run.smithi076.stdout:Total 21 MB/s | 41 MB 00:01 2021-03-15T22:11:04.290 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:11:04.314 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:11:04.315 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:11:04.544 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:11:04.546 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:11:04.996 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:11:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/6 2021-03-15T22:11:05.352 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2021-03-15T22:11:05.493 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/6 2021-03-15T22:11:05.624 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-03-15T22:11:05.705 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/6 2021-03-15T22:11:10.123 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2021-03-15T22:11:10.245 INFO:teuthology.orchestra.run.smithi125.stdout:(6/6): ceph-test-17.0.0-2063.gb658018a.el8.x86_ 6.0 MB/s | 40 MB 00:06 2021-03-15T22:11:10.249 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:10.249 INFO:teuthology.orchestra.run.smithi125.stdout:Total 6.1 MB/s | 41 MB 00:06 2021-03-15T22:11:10.253 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:11:10.277 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:11:10.278 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:11:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-test-2:17.0.0-2063.gb658018a.el8.x86_64 6/6 2021-03-15T22:11:10.509 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:11:10.511 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:11:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-test-2:17.0.0-2063.gb658018a.el8.x86_64 6/6 2021-03-15T22:11:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-test-2:17.0.0-2063.gb658018a.el8.x86_64 1/6 2021-03-15T22:11:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2021-03-15T22:11:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-03-15T22:11:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 4/6 2021-03-15T22:11:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 5/6 2021-03-15T22:11:10.929 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-03-15T22:11:10.929 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:10.930 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:11:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-2:17.0.0-2063.gb658018a.el8.x86_64 jq-1.5-12.el8.x86_64 2021-03-15T22:11:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: libxslt-1.1.32-5.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-03-15T22:11:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: socat-1.7.3.3-2.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-03-15T22:11:10.930 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:10.931 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:11:10.952 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:11:11.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph 2021-03-15T22:11:11.166 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/6 2021-03-15T22:11:11.334 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2021-03-15T22:11:11.509 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/6 2021-03-15T22:11:11.585 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:46 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:11.681 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-03-15T22:11:11.782 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/6 2021-03-15T22:11:12.067 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:11:12.071 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2021-03-15T22:11:12.071 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T22:11:12.072 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2021-03-15T22:11:12.072 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:11:12.072 INFO:teuthology.orchestra.run.smithi076.stdout: ceph x86_64 2:17.0.0-2063.gb658018a.el8 ceph 6.4 k 2021-03-15T22:11:12.072 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T22:11:12.073 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds x86_64 2:17.0.0-2063.gb658018a.el8 ceph 2.2 M 2021-03-15T22:11:12.073 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr x86_64 2:17.0.0-2063.gb658018a.el8 ceph 1.4 M 2021-03-15T22:11:12.073 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 201 k 2021-03-15T22:11:12.074 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon x86_64 2:17.0.0-2063.gb658018a.el8 ceph 3.5 M 2021-03-15T22:11:12.074 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd x86_64 2:17.0.0-2063.gb658018a.el8 ceph 16 M 2021-03-15T22:11:12.074 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-03-15T22:11:12.074 INFO:teuthology.orchestra.run.smithi076.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-03-15T22:11:12.075 INFO:teuthology.orchestra.run.smithi076.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-03-15T22:11:12.075 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-03-15T22:11:12.075 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-03-15T22:11:12.075 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-03-15T22:11:12.076 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-03-15T22:11:12.076 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-03-15T22:11:12.076 INFO:teuthology.orchestra.run.smithi076.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2021-03-15T22:11:12.077 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-03-15T22:11:12.077 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-03-15T22:11:12.078 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-03-15T22:11:12.078 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-03-15T22:11:12.078 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-03-15T22:11:12.078 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-03-15T22:11:12.079 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-03-15T22:11:12.079 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-03-15T22:11:12.079 INFO:teuthology.orchestra.run.smithi076.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-03-15T22:11:12.080 INFO:teuthology.orchestra.run.smithi076.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-03-15T22:11:12.080 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-03-15T22:11:12.080 INFO:teuthology.orchestra.run.smithi076.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-03-15T22:11:12.081 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3 noarch 1.24.2-4.el8 CentOS-Base 176 k 2021-03-15T22:11:12.081 INFO:teuthology.orchestra.run.smithi076.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-03-15T22:11:12.081 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-03-15T22:11:12.081 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-03-15T22:11:12.081 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-03-15T22:11:12.082 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-03-15T22:11:12.082 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-03-15T22:11:12.082 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2021-03-15T22:11:12.082 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-03-15T22:11:12.083 INFO:teuthology.orchestra.run.smithi076.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-03-15T22:11:12.083 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:12.083 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:11:12.083 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================= 2021-03-15T22:11:12.083 INFO:teuthology.orchestra.run.smithi076.stdout:Install 36 Packages 2021-03-15T22:11:12.084 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:12.085 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 28 M 2021-03-15T22:11:12.085 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 114 M 2021-03-15T22:11:12.085 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:11:12.335 INFO:teuthology.orchestra.run.smithi076.stdout:(1/36): ceph-17.0.0-2063.gb658018a.el8.x86_64.r 25 kB/s | 6.4 kB 00:00 2021-03-15T22:11:13.439 INFO:teuthology.orchestra.run.smithi076.stdout:(2/36): ceph-mds-17.0.0-2063.gb658018a.el8.x86_ 1.6 MB/s | 2.2 MB 00:01 2021-03-15T22:11:13.502 INFO:teuthology.orchestra.run.smithi076.stdout:(3/36): ceph-mgr-17.0.0-2063.gb658018a.el8.x86_ 1.0 MB/s | 1.4 MB 00:01 2021-03-15T22:11:13.658 INFO:teuthology.orchestra.run.smithi076.stdout:(4/36): ceph-mgr-modules-core-17.0.0-2063.gb658 1.3 MB/s | 201 kB 00:00 2021-03-15T22:11:13.694 INFO:teuthology.orchestra.run.smithi076.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 5.9 MB/s | 214 kB 00:00 2021-03-15T22:11:13.716 INFO:teuthology.orchestra.run.smithi076.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 70 MB/s | 1.5 MB 00:00 2021-03-15T22:11:13.718 INFO:teuthology.orchestra.run.smithi076.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 53 MB/s | 157 kB 00:00 2021-03-15T22:11:13.720 INFO:teuthology.orchestra.run.smithi076.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 27 MB/s | 39 kB 00:00 2021-03-15T22:11:13.722 INFO:teuthology.orchestra.run.smithi076.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 37 MB/s | 54 kB 00:00 2021-03-15T22:11:13.723 INFO:teuthology.orchestra.run.smithi076.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 20 MB/s | 27 kB 00:00 2021-03-15T22:11:13.729 INFO:teuthology.orchestra.run.smithi076.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 74 MB/s | 457 kB 00:00 2021-03-15T22:11:13.733 INFO:teuthology.orchestra.run.smithi076.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 61 MB/s | 195 kB 00:00 2021-03-15T22:11:13.734 INFO:teuthology.orchestra.run.smithi076.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 27 MB/s | 34 kB 00:00 2021-03-15T22:11:13.736 INFO:teuthology.orchestra.run.smithi076.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 56 MB/s | 123 kB 00:00 2021-03-15T22:11:13.739 INFO:teuthology.orchestra.run.smithi076.stdout:(15/36): python3-urllib3-1.24.2-4.el8.noarch.rp 61 MB/s | 176 kB 00:00 2021-03-15T22:11:13.818 INFO:teuthology.orchestra.run.smithi076.stdout:(16/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 557 kB/s | 44 kB 00:00 2021-03-15T22:11:14.724 INFO:teuthology.orchestra.run.smithi076.stdout:(17/36): ceph-mon-17.0.0-2063.gb658018a.el8.x86 1.5 MB/s | 3.5 MB 00:02 2021-03-15T22:11:16.200 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2021-03-15T22:11:16.453 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-test-2:17.0.0-2063.gb658018a.el8.x86_64 6/6 2021-03-15T22:11:16.682 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-test-2:17.0.0-2063.gb658018a.el8.x86_64 6/6 2021-03-15T22:11:16.682 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-test-2:17.0.0-2063.gb658018a.el8.x86_64 1/6 2021-03-15T22:11:16.683 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2021-03-15T22:11:16.683 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-03-15T22:11:16.683 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 4/6 2021-03-15T22:11:16.683 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 5/6 2021-03-15T22:11:17.033 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-03-15T22:11:17.034 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:17.034 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:11:17.034 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-test-2:17.0.0-2063.gb658018a.el8.x86_64 jq-1.5-12.el8.x86_64 2021-03-15T22:11:17.034 INFO:teuthology.orchestra.run.smithi125.stdout: libxslt-1.1.32-5.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-03-15T22:11:17.034 INFO:teuthology.orchestra.run.smithi125.stdout: socat-1.7.3.3-2.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-03-15T22:11:17.035 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:17.035 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:11:17.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph 2021-03-15T22:11:17.543 INFO:teuthology.orchestra.run.smithi076.stdout:(18/36): python3-cheroot-8.5.2-1.el8.noarch.rpm 61 kB/s | 173 kB 00:02 2021-03-15T22:11:17.544 INFO:teuthology.orchestra.run.smithi076.stdout:(19/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 50 kB/s | 185 kB 00:03 2021-03-15T22:11:17.577 INFO:teuthology.orchestra.run.smithi076.stdout:(20/36): python3-cssselect-0.9.2-10.el8.noarch. 1.2 MB/s | 40 kB 00:00 2021-03-15T22:11:17.606 INFO:teuthology.orchestra.run.smithi076.stdout:(21/36): python3-jaraco-6.2-6.el8.noarch.rpm 388 kB/s | 11 kB 00:00 2021-03-15T22:11:17.627 INFO:teuthology.orchestra.run.smithi076.stdout:(22/36): python3-cherrypy-18.4.0-1.el8.noarch.r 4.5 MB/s | 384 kB 00:00 2021-03-15T22:11:17.631 INFO:teuthology.orchestra.run.smithi076.stdout:(23/36): python3-jaraco-functools-2.0-4.el8.noa 709 kB/s | 18 kB 00:00 2021-03-15T22:11:17.660 INFO:teuthology.orchestra.run.smithi076.stdout:(24/36): python3-logutils-0.3.5-11.el8.noarch.r 1.4 MB/s | 49 kB 00:00 2021-03-15T22:11:17.665 INFO:teuthology.orchestra.run.smithi076.stdout:(25/36): python3-more-itertools-7.2.0-3.el8.noa 1.7 MB/s | 59 kB 00:00 2021-03-15T22:11:17.686 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:00:52 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:17.720 INFO:teuthology.orchestra.run.smithi076.stdout:(26/36): python3-portend-2.6-1.el8.noarch.rpm 286 kB/s | 16 kB 00:00 2021-03-15T22:11:17.785 INFO:teuthology.orchestra.run.smithi076.stdout:(27/36): python3-pecan-1.3.2-9.el8.noarch.rpm 2.2 MB/s | 283 kB 00:00 2021-03-15T22:11:17.810 INFO:teuthology.orchestra.run.smithi076.stdout:(28/36): python3-simplegeneric-0.8.1-17.el8.noa 206 kB/s | 19 kB 00:00 2021-03-15T22:11:17.845 INFO:teuthology.orchestra.run.smithi076.stdout:(29/36): python3-singledispatch-3.4.0.3-18.el8. 398 kB/s | 24 kB 00:00 2021-03-15T22:11:17.850 INFO:teuthology.orchestra.run.smithi076.stdout:(30/36): python3-tempora-1.14.1-5.el8.noarch.rp 744 kB/s | 29 kB 00:00 2021-03-15T22:11:17.872 INFO:teuthology.orchestra.run.smithi076.stdout:(31/36): python3-trustme-0.6.0-4.el8.noarch.rpm 1.0 MB/s | 27 kB 00:00 2021-03-15T22:11:17.906 INFO:teuthology.orchestra.run.smithi076.stdout:(32/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 7.2 MB/s | 251 kB 00:00 2021-03-15T22:11:17.908 INFO:teuthology.orchestra.run.smithi076.stdout:(33/36): python3-waitress-1.2.1-2.el8.1.noarch. 4.1 MB/s | 241 kB 00:00 2021-03-15T22:11:17.938 INFO:teuthology.orchestra.run.smithi076.stdout:(34/36): python3-webtest-2.0.33-1.el8.noarch.rp 2.6 MB/s | 85 kB 00:00 2021-03-15T22:11:17.946 INFO:teuthology.orchestra.run.smithi076.stdout:(35/36): python3-zc-lockfile-2.0-2.el8.noarch.r 607 kB/s | 23 kB 00:00 2021-03-15T22:11:18.140 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:11:18.144 INFO:teuthology.orchestra.run.smithi125.stdout:======================================================================================= 2021-03-15T22:11:18.144 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:11:18.145 INFO:teuthology.orchestra.run.smithi125.stdout:======================================================================================= 2021-03-15T22:11:18.145 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:11:18.145 INFO:teuthology.orchestra.run.smithi125.stdout: ceph x86_64 2:17.0.0-2063.gb658018a.el8 ceph 6.4 k 2021-03-15T22:11:18.145 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:11:18.145 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mds x86_64 2:17.0.0-2063.gb658018a.el8 ceph 2.2 M 2021-03-15T22:11:18.146 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr x86_64 2:17.0.0-2063.gb658018a.el8 ceph 1.4 M 2021-03-15T22:11:18.146 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-modules-core noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 201 k 2021-03-15T22:11:18.146 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mon x86_64 2:17.0.0-2063.gb658018a.el8 ceph 3.5 M 2021-03-15T22:11:18.146 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-osd x86_64 2:17.0.0-2063.gb658018a.el8 ceph 16 M 2021-03-15T22:11:18.146 INFO:teuthology.orchestra.run.smithi125.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-03-15T22:11:18.147 INFO:teuthology.orchestra.run.smithi125.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-03-15T22:11:18.147 INFO:teuthology.orchestra.run.smithi125.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-03-15T22:11:18.147 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-03-15T22:11:18.147 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-03-15T22:11:18.148 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-03-15T22:11:18.148 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-03-15T22:11:18.148 INFO:teuthology.orchestra.run.smithi125.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-03-15T22:11:18.148 INFO:teuthology.orchestra.run.smithi125.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2021-03-15T22:11:18.148 INFO:teuthology.orchestra.run.smithi125.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-03-15T22:11:18.149 INFO:teuthology.orchestra.run.smithi125.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-03-15T22:11:18.149 INFO:teuthology.orchestra.run.smithi125.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-03-15T22:11:18.149 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-03-15T22:11:18.149 INFO:teuthology.orchestra.run.smithi125.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-03-15T22:11:18.150 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-03-15T22:11:18.150 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-03-15T22:11:18.150 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-03-15T22:11:18.150 INFO:teuthology.orchestra.run.smithi125.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-03-15T22:11:18.151 INFO:teuthology.orchestra.run.smithi125.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-03-15T22:11:18.151 INFO:teuthology.orchestra.run.smithi125.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-03-15T22:11:18.151 INFO:teuthology.orchestra.run.smithi125.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-03-15T22:11:18.151 INFO:teuthology.orchestra.run.smithi125.stdout: python3-urllib3 noarch 1.24.2-4.el8 CentOS-Base 176 k 2021-03-15T22:11:18.151 INFO:teuthology.orchestra.run.smithi125.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-03-15T22:11:18.152 INFO:teuthology.orchestra.run.smithi125.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-03-15T22:11:18.152 INFO:teuthology.orchestra.run.smithi125.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-03-15T22:11:18.152 INFO:teuthology.orchestra.run.smithi125.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-03-15T22:11:18.152 INFO:teuthology.orchestra.run.smithi125.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-03-15T22:11:18.152 INFO:teuthology.orchestra.run.smithi125.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-03-15T22:11:18.153 INFO:teuthology.orchestra.run.smithi125.stdout:Installing weak dependencies: 2021-03-15T22:11:18.153 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-03-15T22:11:18.153 INFO:teuthology.orchestra.run.smithi125.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-03-15T22:11:18.153 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:18.153 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:11:18.154 INFO:teuthology.orchestra.run.smithi125.stdout:======================================================================================= 2021-03-15T22:11:18.154 INFO:teuthology.orchestra.run.smithi125.stdout:Install 36 Packages 2021-03-15T22:11:18.154 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:18.155 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 28 M 2021-03-15T22:11:18.155 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 114 M 2021-03-15T22:11:18.155 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:11:18.473 INFO:teuthology.orchestra.run.smithi125.stdout:(1/36): ceph-17.0.0-2063.gb658018a.el8.x86_64.r 20 kB/s | 6.4 kB 00:00 2021-03-15T22:11:18.711 INFO:teuthology.orchestra.run.smithi076.stdout:(36/36): ceph-osd-17.0.0-2063.gb658018a.el8.x86 3.0 MB/s | 16 MB 00:05 2021-03-15T22:11:18.714 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:18.714 INFO:teuthology.orchestra.run.smithi076.stdout:Total 4.2 MB/s | 28 MB 00:06 2021-03-15T22:11:18.726 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:11:18.769 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:11:18.769 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:11:19.059 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:11:19.061 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:11:19.540 INFO:teuthology.orchestra.run.smithi125.stdout:(2/36): ceph-mds-17.0.0-2063.gb658018a.el8.x86_ 1.6 MB/s | 2.2 MB 00:01 2021-03-15T22:11:20.294 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:11:20.480 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/36 2021-03-15T22:11:20.656 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/36 2021-03-15T22:11:20.675 INFO:teuthology.orchestra.run.smithi125.stdout:(3/36): ceph-mon-17.0.0-2063.gb658018a.el8.x86_ 1.6 MB/s | 3.5 MB 00:02 2021-03-15T22:11:20.745 INFO:teuthology.orchestra.run.smithi125.stdout:(4/36): ceph-mgr-modules-core-17.0.0-2063.gb658 2.8 MB/s | 201 kB 00:00 2021-03-15T22:11:20.818 INFO:teuthology.orchestra.run.smithi125.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 2.9 MB/s | 214 kB 00:00 2021-03-15T22:11:20.857 INFO:teuthology.orchestra.run.smithi125.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 39 MB/s | 1.5 MB 00:00 2021-03-15T22:11:20.859 INFO:teuthology.orchestra.run.smithi125.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 56 MB/s | 157 kB 00:00 2021-03-15T22:11:20.861 INFO:teuthology.orchestra.run.smithi125.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 26 MB/s | 39 kB 00:00 2021-03-15T22:11:20.862 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/36 2021-03-15T22:11:20.863 INFO:teuthology.orchestra.run.smithi125.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 35 MB/s | 54 kB 00:00 2021-03-15T22:11:20.864 INFO:teuthology.orchestra.run.smithi125.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 19 MB/s | 27 kB 00:00 2021-03-15T22:11:20.872 INFO:teuthology.orchestra.run.smithi125.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 61 MB/s | 457 kB 00:00 2021-03-15T22:11:20.875 INFO:teuthology.orchestra.run.smithi125.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 61 MB/s | 195 kB 00:00 2021-03-15T22:11:20.877 INFO:teuthology.orchestra.run.smithi125.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 25 MB/s | 34 kB 00:00 2021-03-15T22:11:20.879 INFO:teuthology.orchestra.run.smithi125.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 53 MB/s | 123 kB 00:00 2021-03-15T22:11:20.882 INFO:teuthology.orchestra.run.smithi125.stdout:(15/36): python3-urllib3-1.24.2-4.el8.noarch.rp 59 MB/s | 176 kB 00:00 2021-03-15T22:11:20.966 INFO:teuthology.orchestra.run.smithi125.stdout:(16/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 525 kB/s | 44 kB 00:00 2021-03-15T22:11:20.997 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/36 2021-03-15T22:11:21.024 INFO:teuthology.orchestra.run.smithi125.stdout:(17/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 3.1 MB/s | 185 kB 00:00 2021-03-15T22:11:21.063 INFO:teuthology.orchestra.run.smithi125.stdout:(18/36): python3-cheroot-8.5.2-1.el8.noarch.rpm 4.3 MB/s | 173 kB 00:00 2021-03-15T22:11:21.071 INFO:teuthology.orchestra.run.smithi125.stdout:(19/36): ceph-mgr-17.0.0-2063.gb658018a.el8.x86 508 kB/s | 1.4 MB 00:02 2021-03-15T22:11:21.103 INFO:teuthology.orchestra.run.smithi125.stdout:(20/36): python3-cherrypy-18.4.0-1.el8.noarch.r 9.5 MB/s | 384 kB 00:00 2021-03-15T22:11:21.120 INFO:teuthology.orchestra.run.smithi125.stdout:(21/36): python3-cssselect-0.9.2-10.el8.noarch. 820 kB/s | 40 kB 00:00 2021-03-15T22:11:21.123 INFO:teuthology.orchestra.run.smithi125.stdout:(22/36): python3-jaraco-6.2-6.el8.noarch.rpm 567 kB/s | 11 kB 00:00 2021-03-15T22:11:21.129 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 5/36 2021-03-15T22:11:21.136 INFO:teuthology.orchestra.run.smithi125.stdout:(23/36): python3-jaraco-functools-2.0-4.el8.noa 1.1 MB/s | 18 kB 00:00 2021-03-15T22:11:21.143 INFO:teuthology.orchestra.run.smithi125.stdout:(24/36): python3-logutils-0.3.5-11.el8.noarch.r 2.3 MB/s | 49 kB 00:00 2021-03-15T22:11:21.166 INFO:teuthology.orchestra.run.smithi125.stdout:(25/36): python3-pecan-1.3.2-9.el8.noarch.rpm 12 MB/s | 283 kB 00:00 2021-03-15T22:11:21.168 INFO:teuthology.orchestra.run.smithi125.stdout:(26/36): python3-more-itertools-7.2.0-3.el8.noa 1.8 MB/s | 59 kB 00:00 2021-03-15T22:11:21.186 INFO:teuthology.orchestra.run.smithi125.stdout:(27/36): python3-simplegeneric-0.8.1-17.el8.noa 1.1 MB/s | 19 kB 00:00 2021-03-15T22:11:21.186 INFO:teuthology.orchestra.run.smithi125.stdout:(28/36): python3-portend-2.6-1.el8.noarch.rpm 780 kB/s | 16 kB 00:00 2021-03-15T22:11:21.202 INFO:teuthology.orchestra.run.smithi125.stdout:(29/36): python3-tempora-1.14.1-5.el8.noarch.rp 1.8 MB/s | 29 kB 00:00 2021-03-15T22:11:21.205 INFO:teuthology.orchestra.run.smithi125.stdout:(30/36): python3-singledispatch-3.4.0.3-18.el8. 1.2 MB/s | 24 kB 00:00 2021-03-15T22:11:21.219 INFO:teuthology.orchestra.run.smithi125.stdout:(31/36): python3-trustme-0.6.0-4.el8.noarch.rpm 1.7 MB/s | 27 kB 00:00 2021-03-15T22:11:21.227 INFO:teuthology.orchestra.run.smithi125.stdout:(32/36): python3-waitress-1.2.1-2.el8.1.noarch. 11 MB/s | 241 kB 00:00 2021-03-15T22:11:21.247 INFO:teuthology.orchestra.run.smithi125.stdout:(33/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.2 MB/s | 85 kB 00:00 2021-03-15T22:11:21.252 INFO:teuthology.orchestra.run.smithi125.stdout:(34/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 7.4 MB/s | 251 kB 00:00 2021-03-15T22:11:21.267 INFO:teuthology.orchestra.run.smithi125.stdout:(35/36): python3-zc-lockfile-2.0-2.el8.noarch.r 1.1 MB/s | 23 kB 00:00 2021-03-15T22:11:21.302 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/36 2021-03-15T22:11:21.441 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/36 2021-03-15T22:11:21.642 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/36 2021-03-15T22:11:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/36 2021-03-15T22:11:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/36 2021-03-15T22:11:22.143 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 11/36 2021-03-15T22:11:22.290 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 12/36 2021-03-15T22:11:22.463 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 13/36 2021-03-15T22:11:22.647 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2021-03-15T22:11:22.660 INFO:teuthology.orchestra.run.smithi125.stdout:(36/36): ceph-osd-17.0.0-2063.gb658018a.el8.x86 5.0 MB/s | 16 MB 00:03 2021-03-15T22:11:22.664 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:22.664 INFO:teuthology.orchestra.run.smithi125.stdout:Total 6.1 MB/s | 28 MB 00:04 2021-03-15T22:11:22.676 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:11:22.719 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:11:22.720 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:11:22.887 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-urllib3-1.24.2-4.el8.noarch 15/36 2021-03-15T22:11:23.004 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:11:23.017 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:11:23.060 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2021-03-15T22:11:23.282 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2021-03-15T22:11:23.483 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2021-03-15T22:11:23.682 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2021-03-15T22:11:23.885 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2021-03-15T22:11:24.208 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2021-03-15T22:11:24.241 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:11:24.427 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/36 2021-03-15T22:11:24.450 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2021-03-15T22:11:24.595 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/36 2021-03-15T22:11:24.628 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2021-03-15T22:11:24.783 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2021-03-15T22:11:24.826 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/36 2021-03-15T22:11:24.969 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2021-03-15T22:11:24.987 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/36 2021-03-15T22:11:25.127 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 5/36 2021-03-15T22:11:25.189 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2021-03-15T22:11:25.308 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/36 2021-03-15T22:11:25.368 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2021-03-15T22:11:25.505 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/36 2021-03-15T22:11:25.584 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2021-03-15T22:11:25.658 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/36 2021-03-15T22:11:25.801 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/36 2021-03-15T22:11:25.936 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2021-03-15T22:11:25.980 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/36 2021-03-15T22:11:26.223 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2021-03-15T22:11:26.242 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 11/36 2021-03-15T22:11:26.405 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 12/36 2021-03-15T22:11:26.563 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 13/36 2021-03-15T22:11:26.569 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-2063.gb658018a.el8. 31/36 2021-03-15T22:11:26.729 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2021-03-15T22:11:26.758 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-2:17.0.0-2063.gb658018a.el8.x86_64 32/36 2021-03-15T22:11:26.992 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-urllib3-1.24.2-4.el8.noarch 15/36 2021-03-15T22:11:27.174 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2021-03-15T22:11:27.429 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2021-03-15T22:11:27.639 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2021-03-15T22:11:27.898 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2021-03-15T22:11:28.107 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2021-03-15T22:11:28.185 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-2:17.0.0-2063.gb658018a.el8.x86_64 32/36 2021-03-15T22:11:28.425 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-osd-2:17.0.0-2063.gb658018a.el8.x86_64 33/36 2021-03-15T22:11:28.463 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2021-03-15T22:11:28.693 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2021-03-15T22:11:28.800 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-osd-2:17.0.0-2063.gb658018a.el8.x86_64 33/36 2021-03-15T22:11:28.859 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2021-03-15T22:11:29.000 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mon-2:17.0.0-2063.gb658018a.el8.x86_64 34/36 2021-03-15T22:11:29.023 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2021-03-15T22:11:29.191 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2021-03-15T22:11:29.218 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mon-2:17.0.0-2063.gb658018a.el8.x86_64 34/36 2021-03-15T22:11:29.444 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2021-03-15T22:11:29.648 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2021-03-15T22:11:29.864 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2021-03-15T22:11:29.900 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mds-2:17.0.0-2063.gb658018a.el8.x86_64 35/36 2021-03-15T22:11:29.923 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mds-2:17.0.0-2063.gb658018a.el8.x86_64 35/36 2021-03-15T22:11:30.071 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-2:17.0.0-2063.gb658018a.el8.x86_64 36/36 2021-03-15T22:11:30.220 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2021-03-15T22:11:30.433 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-2:17.0.0-2063.gb658018a.el8.x86_64 36/36 2021-03-15T22:11:30.434 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-2:17.0.0-2063.gb658018a.el8.x86_64 1/36 2021-03-15T22:11:30.434 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mds-2:17.0.0-2063.gb658018a.el8.x86_64 2/36 2021-03-15T22:11:30.434 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-2:17.0.0-2063.gb658018a.el8.x86_64 3/36 2021-03-15T22:11:30.434 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mon-2:17.0.0-2063.gb658018a.el8.x86_64 4/36 2021-03-15T22:11:30.435 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-osd-2:17.0.0-2063.gb658018a.el8.x86_64 5/36 2021-03-15T22:11:30.435 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-2063.gb658018a.el8. 6/36 2021-03-15T22:11:30.435 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2021-03-15T22:11:30.435 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2021-03-15T22:11:30.436 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2021-03-15T22:11:30.436 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2021-03-15T22:11:30.436 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2021-03-15T22:11:30.437 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2021-03-15T22:11:30.437 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2021-03-15T22:11:30.437 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2021-03-15T22:11:30.438 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2021-03-15T22:11:30.438 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2021-03-15T22:11:30.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-urllib3-1.24.2-4.el8.noarch 17/36 2021-03-15T22:11:30.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/36 2021-03-15T22:11:30.439 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/36 2021-03-15T22:11:30.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 20/36 2021-03-15T22:11:30.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 21/36 2021-03-15T22:11:30.440 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 22/36 2021-03-15T22:11:30.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 23/36 2021-03-15T22:11:30.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 24/36 2021-03-15T22:11:30.441 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 25/36 2021-03-15T22:11:30.443 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 26/36 2021-03-15T22:11:30.444 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 27/36 2021-03-15T22:11:30.444 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-portend-2.6-1.el8.noarch 28/36 2021-03-15T22:11:30.445 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 29/36 2021-03-15T22:11:30.445 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 30/36 2021-03-15T22:11:30.445 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 31/36 2021-03-15T22:11:30.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 32/36 2021-03-15T22:11:30.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 33/36 2021-03-15T22:11:30.446 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 34/36 2021-03-15T22:11:30.447 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 35/36 2021-03-15T22:11:30.503 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2021-03-15T22:11:30.878 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-2063.gb658018a.el8. 31/36 2021-03-15T22:11:31.236 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 36/36 2021-03-15T22:11:31.237 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:31.237 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:11:31.237 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:31.237 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:31.238 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:31.238 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:31.238 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:31.238 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:31.239 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-03-15T22:11:31.239 INFO:teuthology.orchestra.run.smithi076.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-03-15T22:11:31.239 INFO:teuthology.orchestra.run.smithi076.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-03-15T22:11:31.239 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-03-15T22:11:31.240 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-03-15T22:11:31.240 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-03-15T22:11:31.240 INFO:teuthology.orchestra.run.smithi076.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-03-15T22:11:31.240 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-6.2-6.el8.noarch 2021-03-15T22:11:31.240 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-03-15T22:11:31.241 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-03-15T22:11:31.241 INFO:teuthology.orchestra.run.smithi076.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-03-15T22:11:31.241 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako-1.0.6-13.el8.noarch 2021-03-15T22:11:31.241 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-03-15T22:11:31.241 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-03-15T22:11:31.242 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-03-15T22:11:31.242 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend-2.6-1.el8.noarch 2021-03-15T22:11:31.243 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-03-15T22:11:31.243 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz-2017.2-9.el8.noarch 2021-03-15T22:11:31.243 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-03-15T22:11:31.244 INFO:teuthology.orchestra.run.smithi076.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-03-15T22:11:31.244 INFO:teuthology.orchestra.run.smithi076.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-03-15T22:11:31.247 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-03-15T22:11:31.249 INFO:teuthology.orchestra.run.smithi076.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-03-15T22:11:31.249 INFO:teuthology.orchestra.run.smithi076.stdout: python3-urllib3-1.24.2-4.el8.noarch 2021-03-15T22:11:31.249 INFO:teuthology.orchestra.run.smithi076.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-03-15T22:11:31.249 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-03-15T22:11:31.250 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-03-15T22:11:31.250 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-03-15T22:11:31.250 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-03-15T22:11:31.250 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-03-15T22:11:31.251 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:31.251 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:11:31.359 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-2:17.0.0-2063.gb658018a.el8.x86_64 32/36 2021-03-15T22:11:31.469 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-base 2021-03-15T22:11:31.940 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:06 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:32.276 INFO:teuthology.orchestra.run.smithi076.stdout:Package ceph-base-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:11:32.389 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:11:32.390 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T22:11:32.391 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:11:32.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install cephadm 2021-03-15T22:11:32.795 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mgr-2:17.0.0-2063.gb658018a.el8.x86_64 32/36 2021-03-15T22:11:32.969 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:07 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:32.984 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-osd-2:17.0.0-2063.gb658018a.el8.x86_64 33/36 2021-03-15T22:11:33.357 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-osd-2:17.0.0-2063.gb658018a.el8.x86_64 33/36 2021-03-15T22:11:33.458 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:11:33.459 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T22:11:33.460 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:33.460 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:11:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 65 k 2021-03-15T22:11:33.460 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:33.461 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:11:33.461 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:33.461 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2021-03-15T22:11:33.461 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:33.462 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 65 k 2021-03-15T22:11:33.462 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 284 k 2021-03-15T22:11:33.462 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:11:33.559 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mon-2:17.0.0-2063.gb658018a.el8.x86_64 34/36 2021-03-15T22:11:33.777 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mon-2:17.0.0-2063.gb658018a.el8.x86_64 34/36 2021-03-15T22:11:33.848 INFO:teuthology.orchestra.run.smithi076.stdout:cephadm-17.0.0-2063.gb658018a.el8.noarch.rpm 167 kB/s | 65 kB 00:00 2021-03-15T22:11:33.849 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:33.849 INFO:teuthology.orchestra.run.smithi076.stdout:Total 166 kB/s | 65 kB 00:00 2021-03-15T22:11:33.850 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:11:33.858 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:11:33.858 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:11:33.859 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:11:33.862 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:11:33.906 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:11:34.018 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mds-2:17.0.0-2063.gb658018a.el8.x86_64 35/36 2021-03-15T22:11:34.037 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mds-2:17.0.0-2063.gb658018a.el8.x86_64 35/36 2021-03-15T22:11:34.168 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-2:17.0.0-2063.gb658018a.el8.x86_64 36/36 2021-03-15T22:11:34.512 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-2:17.0.0-2063.gb658018a.el8.x86_64 36/36 2021-03-15T22:11:34.513 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-2:17.0.0-2063.gb658018a.el8.x86_64 1/36 2021-03-15T22:11:34.513 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mds-2:17.0.0-2063.gb658018a.el8.x86_64 2/36 2021-03-15T22:11:34.514 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-2:17.0.0-2063.gb658018a.el8.x86_64 3/36 2021-03-15T22:11:34.514 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mon-2:17.0.0-2063.gb658018a.el8.x86_64 4/36 2021-03-15T22:11:34.514 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-osd-2:17.0.0-2063.gb658018a.el8.x86_64 5/36 2021-03-15T22:11:34.514 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-2063.gb658018a.el8. 6/36 2021-03-15T22:11:34.514 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2021-03-15T22:11:34.515 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2021-03-15T22:11:34.515 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2021-03-15T22:11:34.515 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2021-03-15T22:11:34.515 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2021-03-15T22:11:34.516 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2021-03-15T22:11:34.516 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2021-03-15T22:11:34.516 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2021-03-15T22:11:34.516 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2021-03-15T22:11:34.517 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2021-03-15T22:11:34.517 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-urllib3-1.24.2-4.el8.noarch 17/36 2021-03-15T22:11:34.517 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/36 2021-03-15T22:11:34.517 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/36 2021-03-15T22:11:34.517 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 20/36 2021-03-15T22:11:34.518 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 21/36 2021-03-15T22:11:34.518 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 22/36 2021-03-15T22:11:34.518 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 23/36 2021-03-15T22:11:34.519 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 24/36 2021-03-15T22:11:34.519 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 25/36 2021-03-15T22:11:34.519 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 26/36 2021-03-15T22:11:34.520 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 27/36 2021-03-15T22:11:34.521 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-portend-2.6-1.el8.noarch 28/36 2021-03-15T22:11:34.522 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 29/36 2021-03-15T22:11:34.522 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 30/36 2021-03-15T22:11:34.522 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 31/36 2021-03-15T22:11:34.522 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 32/36 2021-03-15T22:11:34.523 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 33/36 2021-03-15T22:11:34.523 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 34/36 2021-03-15T22:11:34.523 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 35/36 2021-03-15T22:11:35.361 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 36/36 2021-03-15T22:11:35.361 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:35.362 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:11:35.362 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:35.362 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mds-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:35.362 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:35.363 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-modules-core-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:35.363 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mon-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:35.363 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-osd-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:35.363 INFO:teuthology.orchestra.run.smithi125.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-03-15T22:11:35.364 INFO:teuthology.orchestra.run.smithi125.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-03-15T22:11:35.364 INFO:teuthology.orchestra.run.smithi125.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-03-15T22:11:35.364 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-03-15T22:11:35.364 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-03-15T22:11:35.364 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-03-15T22:11:35.365 INFO:teuthology.orchestra.run.smithi125.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-03-15T22:11:35.365 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-6.2-6.el8.noarch 2021-03-15T22:11:35.365 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-03-15T22:11:35.365 INFO:teuthology.orchestra.run.smithi125.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-03-15T22:11:35.366 INFO:teuthology.orchestra.run.smithi125.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-03-15T22:11:35.366 INFO:teuthology.orchestra.run.smithi125.stdout: python3-mako-1.0.6-13.el8.noarch 2021-03-15T22:11:35.366 INFO:teuthology.orchestra.run.smithi125.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-03-15T22:11:35.366 INFO:teuthology.orchestra.run.smithi125.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-03-15T22:11:35.367 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-03-15T22:11:35.367 INFO:teuthology.orchestra.run.smithi125.stdout: python3-portend-2.6-1.el8.noarch 2021-03-15T22:11:35.367 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-03-15T22:11:35.367 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pytz-2017.2-9.el8.noarch 2021-03-15T22:11:35.367 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-03-15T22:11:35.368 INFO:teuthology.orchestra.run.smithi125.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-03-15T22:11:35.368 INFO:teuthology.orchestra.run.smithi125.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-03-15T22:11:35.368 INFO:teuthology.orchestra.run.smithi125.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-03-15T22:11:35.368 INFO:teuthology.orchestra.run.smithi125.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-03-15T22:11:35.369 INFO:teuthology.orchestra.run.smithi125.stdout: python3-urllib3-1.24.2-4.el8.noarch 2021-03-15T22:11:35.369 INFO:teuthology.orchestra.run.smithi125.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-03-15T22:11:35.369 INFO:teuthology.orchestra.run.smithi125.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-03-15T22:11:35.369 INFO:teuthology.orchestra.run.smithi125.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-03-15T22:11:35.370 INFO:teuthology.orchestra.run.smithi125.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-03-15T22:11:35.370 INFO:teuthology.orchestra.run.smithi125.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-03-15T22:11:35.370 INFO:teuthology.orchestra.run.smithi125.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-03-15T22:11:35.370 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:35.370 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:11:35.420 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:17.0.0-2063.gb658018a.el8.noarch 1/1 2021-03-15T22:11:35.576 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph-base 2021-03-15T22:11:35.586 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephadm-2:17.0.0-2063.gb658018a.el8.noarch 1/1 2021-03-15T22:11:35.754 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:17.0.0-2063.gb658018a.el8.noarch 1/1 2021-03-15T22:11:36.031 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephadm-2:17.0.0-2063.gb658018a.el8.noarch 1/1 2021-03-15T22:11:36.032 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:36.032 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:11:36.032 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:36.032 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:36.033 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:11:36.044 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:11 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:36.190 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-immutable-object-cache 2021-03-15T22:11:36.364 INFO:teuthology.orchestra.run.smithi125.stdout:Package ceph-base-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:11:36.480 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:11:36.481 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:11:36.482 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:11:36.568 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install cephadm 2021-03-15T22:11:36.669 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:11 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:37.027 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:12 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:37.121 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:11:37.122 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:37.122 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2021-03-15T22:11:37.123 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:37.123 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:11:37.123 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-2063.gb658018a.el8 ceph 155 k 2021-03-15T22:11:37.124 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:37.124 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:11:37.124 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:37.124 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2021-03-15T22:11:37.125 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:37.125 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 155 k 2021-03-15T22:11:37.125 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 587 k 2021-03-15T22:11:37.125 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:11:37.488 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:11:37.489 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:37.490 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:11:37.490 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:37.490 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:11:37.491 INFO:teuthology.orchestra.run.smithi125.stdout: cephadm noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 65 k 2021-03-15T22:11:37.491 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:37.491 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:11:37.491 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:37.492 INFO:teuthology.orchestra.run.smithi125.stdout:Install 1 Package 2021-03-15T22:11:37.492 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:37.492 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 65 k 2021-03-15T22:11:37.493 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 284 k 2021-03-15T22:11:37.493 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:11:37.634 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-immutable-object-cache-17.0.0-2063.gb65801 304 kB/s | 155 kB 00:00 2021-03-15T22:11:37.635 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:37.636 INFO:teuthology.orchestra.run.smithi076.stdout:Total 303 kB/s | 155 kB 00:00 2021-03-15T22:11:37.636 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:11:37.648 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:11:37.648 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:11:37.695 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:11:37.697 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:11:37.872 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:11:37.933 INFO:teuthology.orchestra.run.smithi125.stdout:cephadm-17.0.0-2063.gb658018a.el8.noarch.rpm 146 kB/s | 65 kB 00:00 2021-03-15T22:11:37.934 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:37.935 INFO:teuthology.orchestra.run.smithi125.stdout:Total 145 kB/s | 65 kB 00:00 2021-03-15T22:11:37.935 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:11:37.943 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:11:37.943 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:11:37.944 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:11:37.947 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:11:37.989 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:11:38.070 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-2063.gb658018a. 1/1 2021-03-15T22:11:38.423 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-2063.gb658018a. 1/1 2021-03-15T22:11:38.715 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-2063.gb658018a. 1/1 2021-03-15T22:11:38.716 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:38.716 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:11:38.716 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:38.716 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:38.717 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:11:38.851 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr 2021-03-15T22:11:39.322 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:14 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:39.483 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: cephadm-2:17.0.0-2063.gb658018a.el8.noarch 1/1 2021-03-15T22:11:39.646 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : cephadm-2:17.0.0-2063.gb658018a.el8.noarch 1/1 2021-03-15T22:11:39.667 INFO:teuthology.orchestra.run.smithi076.stdout:Package ceph-mgr-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:11:39.781 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:11:39.782 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T22:11:39.783 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:11:39.814 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: cephadm-2:17.0.0-2063.gb658018a.el8.noarch 1/1 2021-03-15T22:11:39.821 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-dashboard 2021-03-15T22:11:40.070 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : cephadm-2:17.0.0-2063.gb658018a.el8.noarch 1/1 2021-03-15T22:11:40.070 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:40.071 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:11:40.071 INFO:teuthology.orchestra.run.smithi125.stdout: cephadm-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:40.071 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:40.071 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:11:40.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph-immutable-object-cache 2021-03-15T22:11:40.287 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:15 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:40.706 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:15 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:41.166 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:11:41.167 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:41.168 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repo Size 2021-03-15T22:11:41.168 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:41.168 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:11:41.168 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-2063.gb658018a.el8 ceph 155 k 2021-03-15T22:11:41.169 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:41.169 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:11:41.169 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:41.169 INFO:teuthology.orchestra.run.smithi125.stdout:Install 1 Package 2021-03-15T22:11:41.170 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:41.170 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 155 k 2021-03-15T22:11:41.170 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 587 k 2021-03-15T22:11:41.170 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:11:41.322 INFO:teuthology.orchestra.run.smithi125.stdout:ceph-immutable-object-cache-17.0.0-2063.gb65801 1.0 MB/s | 155 kB 00:00 2021-03-15T22:11:41.323 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:41.323 INFO:teuthology.orchestra.run.smithi125.stdout:Total 1.0 MB/s | 155 kB 00:00 2021-03-15T22:11:41.324 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:11:41.336 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:11:41.336 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:11:41.383 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:11:41.384 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:11:41.533 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:11:41.570 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:11:41.571 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:41.571 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T22:11:41.572 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:41.572 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:11:41.572 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 2.3 M 2021-03-15T22:11:41.572 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T22:11:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 23 k 2021-03-15T22:11:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 8.9 k 2021-03-15T22:11:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-03-15T22:11:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-03-15T22:11:41.573 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-03-15T22:11:41.574 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:41.574 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:11:41.574 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:11:41.574 INFO:teuthology.orchestra.run.smithi076.stdout:Install 6 Packages 2021-03-15T22:11:41.575 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:41.575 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 2.6 M 2021-03-15T22:11:41.576 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 15 M 2021-03-15T22:11:41.576 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:11:41.730 INFO:teuthology.orchestra.run.smithi076.stdout:(1/6): ceph-prometheus-alerts-17.0.0-2063.gb658 57 kB/s | 8.9 kB 00:00 2021-03-15T22:11:41.730 INFO:teuthology.orchestra.run.smithi076.stdout:(2/6): ceph-grafana-dashboards-17.0.0-2063.gb65 146 kB/s | 23 kB 00:00 2021-03-15T22:11:41.788 INFO:teuthology.orchestra.run.smithi076.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 733 kB/s | 43 kB 00:00 2021-03-15T22:11:41.818 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-2063.gb658018a. 1/1 2021-03-15T22:11:41.838 INFO:teuthology.orchestra.run.smithi076.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 314 kB/s | 34 kB 00:00 2021-03-15T22:11:41.944 INFO:teuthology.orchestra.run.smithi076.stdout:(5/6): ceph-mgr-dashboard-17.0.0-2063.gb658018a 6.1 MB/s | 2.3 MB 00:00 2021-03-15T22:11:41.951 INFO:teuthology.orchestra.run.smithi076.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2021-03-15T22:11:41.953 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:41.954 INFO:teuthology.orchestra.run.smithi076.stdout:Total 6.7 MB/s | 2.6 MB 00:00 2021-03-15T22:11:41.956 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:11:41.966 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:11:41.967 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:11:42.092 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:11:42.094 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:11:42.166 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-2063.gb658018a. 1/1 2021-03-15T22:11:42.381 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:11:42.435 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-2063.gb658018a. 1/1 2021-03-15T22:11:42.495 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:42.496 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:11:42.496 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-immutable-object-cache-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:11:42.496 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:42.497 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:11:42.574 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph-mgr 2021-03-15T22:11:42.594 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-03-15T22:11:42.838 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-03-15T22:11:43.013 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-03-15T22:11:43.044 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:18 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:43.143 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-2063.gb658018a.el8.n 4/6 2021-03-15T22:11:43.379 INFO:teuthology.orchestra.run.smithi125.stdout:Package ceph-mgr-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:11:43.491 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:11:43.492 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:11:43.493 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:11:43.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph-mgr-dashboard 2021-03-15T22:11:43.992 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:18 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:44.128 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-2063.gb658018a.el8. 5/6 2021-03-15T22:11:44.389 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-2063.gb658018a.el8.noarc 6/6 2021-03-15T22:11:44.423 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:11:44.424 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:44.425 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:11:44.425 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:44.425 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:11:44.425 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-dashboard noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 2.3 M 2021-03-15T22:11:44.426 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:11:44.426 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-grafana-dashboards noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 23 k 2021-03-15T22:11:44.426 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-prometheus-alerts noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 8.9 k 2021-03-15T22:11:44.426 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-03-15T22:11:44.426 INFO:teuthology.orchestra.run.smithi125.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-03-15T22:11:44.427 INFO:teuthology.orchestra.run.smithi125.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-03-15T22:11:44.427 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:44.427 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:11:44.427 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:11:44.428 INFO:teuthology.orchestra.run.smithi125.stdout:Install 6 Packages 2021-03-15T22:11:44.428 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:44.429 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 2.6 M 2021-03-15T22:11:44.429 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 15 M 2021-03-15T22:11:44.429 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:11:44.520 INFO:teuthology.orchestra.run.smithi125.stdout:(1/6): ceph-prometheus-alerts-17.0.0-2063.gb658 95 kB/s | 8.9 kB 00:00 2021-03-15T22:11:44.545 INFO:teuthology.orchestra.run.smithi125.stdout:(2/6): ceph-grafana-dashboards-17.0.0-2063.gb65 193 kB/s | 23 kB 00:00 2021-03-15T22:11:44.551 INFO:teuthology.orchestra.run.smithi125.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 1.4 MB/s | 43 kB 00:00 2021-03-15T22:11:44.605 INFO:teuthology.orchestra.run.smithi125.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 565 kB/s | 34 kB 00:00 2021-03-15T22:11:44.618 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-2063.gb658018a.el8.noarc 6/6 2021-03-15T22:11:44.619 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-2063.gb658018a.el8. 1/6 2021-03-15T22:11:44.619 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-2063.gb658018a.el8.noarc 2/6 2021-03-15T22:11:44.619 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-2063.gb658018a.el8.n 3/6 2021-03-15T22:11:44.619 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-03-15T22:11:44.620 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-03-15T22:11:44.648 INFO:teuthology.orchestra.run.smithi125.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 2.0 MB/s | 196 kB 00:00 2021-03-15T22:11:44.782 INFO:teuthology.orchestra.run.smithi125.stdout:(6/6): ceph-mgr-dashboard-17.0.0-2063.gb658018a 6.3 MB/s | 2.3 MB 00:00 2021-03-15T22:11:44.784 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:44.785 INFO:teuthology.orchestra.run.smithi125.stdout:Total 7.1 MB/s | 2.6 MB 00:00 2021-03-15T22:11:44.786 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:11:44.796 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:11:44.796 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:11:44.889 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-03-15T22:11:44.890 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:44.890 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:11:44.890 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:44.891 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:44.891 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:44.891 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-03-15T22:11:44.891 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-03-15T22:11:44.892 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes-2.4.1-12.el8.noarch 2021-03-15T22:11:44.892 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:44.892 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:11:44.917 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:11:44.919 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:11:44.995 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-diskprediction-local 2021-03-15T22:11:45.175 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:11:45.433 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-03-15T22:11:45.461 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:20 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:45.669 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-03-15T22:11:45.877 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-03-15T22:11:45.905 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:11:45.906 INFO:teuthology.orchestra.run.smithi076.stdout:===================================================================================================== 2021-03-15T22:11:45.906 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T22:11:45.907 INFO:teuthology.orchestra.run.smithi076.stdout:===================================================================================================== 2021-03-15T22:11:45.907 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:11:45.907 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 7.4 M 2021-03-15T22:11:45.907 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T22:11:45.908 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran x86_64 8.3.1-5.1.el8 CentOS-Base 639 k 2021-03-15T22:11:45.908 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath x86_64 8.3.1-5.1.el8 CentOS-Base 169 k 2021-03-15T22:11:45.908 INFO:teuthology.orchestra.run.smithi076.stdout: openblas x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-03-15T22:11:45.908 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-threads x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-03-15T22:11:45.909 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-03-15T22:11:45.909 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-03-15T22:11:45.909 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy x86_64 1.0.0-20.module_el8.3.0+562+e162826a CentOS-AppStream 15 M 2021-03-15T22:11:45.909 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:45.909 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:11:45.910 INFO:teuthology.orchestra.run.smithi076.stdout:===================================================================================================== 2021-03-15T22:11:45.910 INFO:teuthology.orchestra.run.smithi076.stdout:Install 8 Packages 2021-03-15T22:11:45.910 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:45.911 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 35 M 2021-03-15T22:11:45.911 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 194 M 2021-03-15T22:11:45.911 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:11:46.007 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-2063.gb658018a.el8.n 4/6 2021-03-15T22:11:46.106 INFO:teuthology.orchestra.run.smithi076.stdout:(1/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 22 MB/s | 4.3 MB 00:00 2021-03-15T22:11:46.144 INFO:teuthology.orchestra.run.smithi076.stdout:(2/8): openblas-0.3.3-5.el8.x86_64.rpm 18 MB/s | 4.3 MB 00:00 2021-03-15T22:11:46.153 INFO:teuthology.orchestra.run.smithi076.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 25 MB/s | 225 kB 00:00 2021-03-15T22:11:46.182 INFO:teuthology.orchestra.run.smithi076.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 49 MB/s | 3.7 MB 00:00 2021-03-15T22:11:46.208 INFO:teuthology.orchestra.run.smithi076.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 25 MB/s | 639 kB 00:00 2021-03-15T22:11:46.215 INFO:teuthology.orchestra.run.smithi076.stdout:(6/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 25 MB/s | 169 kB 00:00 2021-03-15T22:11:46.435 INFO:teuthology.orchestra.run.smithi076.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.3.0+56 52 MB/s | 15 MB 00:00 2021-03-15T22:11:46.475 INFO:teuthology.orchestra.run.smithi076.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-206 13 MB/s | 7.4 MB 00:00 2021-03-15T22:11:46.478 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:46.478 INFO:teuthology.orchestra.run.smithi076.stdout:Total 62 MB/s | 35 MB 00:00 2021-03-15T22:11:46.488 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:11:46.507 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:11:46.507 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:11:46.793 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:11:46.795 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:11:47.052 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-2063.gb658018a.el8. 5/6 2021-03-15T22:11:47.354 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:11:47.752 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-03-15T22:11:47.875 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-03-15T22:11:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-03-15T22:11:48.527 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-03-15T22:11:48.702 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-0.3.3-5.el8.x86_64 3/8 2021-03-15T22:11:49.031 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-2063.gb658018a.el8.noarc 6/6 2021-03-15T22:11:49.203 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 3/8 2021-03-15T22:11:49.271 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-2063.gb658018a.el8.noarc 6/6 2021-03-15T22:11:49.272 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-2063.gb658018a.el8. 1/6 2021-03-15T22:11:49.272 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-2063.gb658018a.el8.noarc 2/6 2021-03-15T22:11:49.272 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-2063.gb658018a.el8.n 3/6 2021-03-15T22:11:49.272 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-03-15T22:11:49.273 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-03-15T22:11:49.377 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-03-15T22:11:49.543 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-03-15T22:11:49.543 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:49.543 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:11:49.544 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-grafana-dashboards-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:49.544 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-dashboard-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:49.544 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-prometheus-alerts-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:49.544 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-03-15T22:11:49.545 INFO:teuthology.orchestra.run.smithi125.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-03-15T22:11:49.545 INFO:teuthology.orchestra.run.smithi125.stdout: python3-routes-2.4.1-12.el8.noarch 2021-03-15T22:11:49.545 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:49.545 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:11:49.631 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph-mgr-diskprediction-local 2021-03-15T22:11:50.094 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:25 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:50.109 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-03-15T22:11:50.476 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-03-15T22:11:50.539 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:11:50.540 INFO:teuthology.orchestra.run.smithi125.stdout:===================================================================================================== 2021-03-15T22:11:50.541 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:11:50.541 INFO:teuthology.orchestra.run.smithi125.stdout:===================================================================================================== 2021-03-15T22:11:50.541 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:11:50.541 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 7.4 M 2021-03-15T22:11:50.541 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:11:50.542 INFO:teuthology.orchestra.run.smithi125.stdout: libgfortran x86_64 8.3.1-5.1.el8 CentOS-Base 639 k 2021-03-15T22:11:50.542 INFO:teuthology.orchestra.run.smithi125.stdout: libquadmath x86_64 8.3.1-5.1.el8 CentOS-Base 169 k 2021-03-15T22:11:50.542 INFO:teuthology.orchestra.run.smithi125.stdout: openblas x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-03-15T22:11:50.542 INFO:teuthology.orchestra.run.smithi125.stdout: openblas-threads x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-03-15T22:11:50.543 INFO:teuthology.orchestra.run.smithi125.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-03-15T22:11:50.543 INFO:teuthology.orchestra.run.smithi125.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-03-15T22:11:50.543 INFO:teuthology.orchestra.run.smithi125.stdout: python3-scipy x86_64 1.0.0-20.module_el8.3.0+562+e162826a CentOS-AppStream 15 M 2021-03-15T22:11:50.543 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:50.543 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:11:50.544 INFO:teuthology.orchestra.run.smithi125.stdout:===================================================================================================== 2021-03-15T22:11:50.544 INFO:teuthology.orchestra.run.smithi125.stdout:Install 8 Packages 2021-03-15T22:11:50.544 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:50.545 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 35 M 2021-03-15T22:11:50.545 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 194 M 2021-03-15T22:11:50.545 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:11:50.651 INFO:teuthology.orchestra.run.smithi125.stdout:(1/8): openblas-0.3.3-5.el8.x86_64.rpm 40 MB/s | 4.3 MB 00:00 2021-03-15T22:11:50.677 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-03-15T22:11:50.719 INFO:teuthology.orchestra.run.smithi125.stdout:(2/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 55 MB/s | 3.7 MB 00:00 2021-03-15T22:11:50.726 INFO:teuthology.orchestra.run.smithi125.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 33 MB/s | 225 kB 00:00 2021-03-15T22:11:50.862 INFO:teuthology.orchestra.run.smithi125.stdout:(4/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 14 MB/s | 4.3 MB 00:00 2021-03-15T22:11:50.898 INFO:teuthology.orchestra.run.smithi125.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 17 MB/s | 639 kB 00:00 2021-03-15T22:11:50.911 INFO:teuthology.orchestra.run.smithi125.stdout:(6/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 13 MB/s | 169 kB 00:00 2021-03-15T22:11:51.050 INFO:teuthology.orchestra.run.smithi125.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.3.0+56 45 MB/s | 15 MB 00:00 2021-03-15T22:11:51.109 INFO:teuthology.orchestra.run.smithi125.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-206 13 MB/s | 7.4 MB 00:00 2021-03-15T22:11:51.112 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:51.113 INFO:teuthology.orchestra.run.smithi125.stdout:Total 62 MB/s | 35 MB 00:00 2021-03-15T22:11:51.121 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:11:51.141 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:11:51.141 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:11:51.424 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:11:51.426 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:11:51.997 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:11:52.132 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-03-15T22:11:52.255 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-03-15T22:11:52.424 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-03-15T22:11:52.667 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-03-15T22:11:52.932 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-03-15T22:11:53.149 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : openblas-0.3.3-5.el8.x86_64 3/8 2021-03-15T22:11:53.643 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 3/8 2021-03-15T22:11:53.857 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-03-15T22:11:54.220 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 7/8 2021-03-15T22:11:54.409 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-2063.gb658018 8/8 2021-03-15T22:11:54.550 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-03-15T22:11:54.684 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-2063.gb658018 8/8 2021-03-15T22:11:54.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-2063.gb658018 1/8 2021-03-15T22:11:54.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 2/8 2021-03-15T22:11:54.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-03-15T22:11:54.685 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-03-15T22:11:54.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-03-15T22:11:54.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 6/8 2021-03-15T22:11:54.686 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-03-15T22:11:54.888 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-03-15T22:11:54.972 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-03-15T22:11:54.972 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:54.972 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:11:54.973 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:54.973 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-03-15T22:11:54.973 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-03-15T22:11:54.973 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-0.3.3-5.el8.x86_64 2021-03-15T22:11:54.973 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-03-15T22:11:54.974 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-03-15T22:11:54.974 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-03-15T22:11:54.974 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x86_64 2021-03-15T22:11:54.974 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:54.975 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:11:55.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-rook 2021-03-15T22:11:55.107 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-03-15T22:11:55.520 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:30 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:11:55.964 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:11:55.966 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2021-03-15T22:11:55.966 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T22:11:55.966 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2021-03-15T22:11:55.967 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:11:55.967 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 26 k 2021-03-15T22:11:55.967 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T22:11:55.967 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-03-15T22:11:55.967 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-03-15T22:11:55.968 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-03-15T22:11:55.968 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-03-15T22:11:55.968 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-03-15T22:11:55.968 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-03-15T22:11:55.969 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-03-15T22:11:55.969 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-03-15T22:11:55.969 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-03-15T22:11:55.969 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-03-15T22:11:55.969 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-03-15T22:11:55.970 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-03-15T22:11:55.970 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:55.970 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:11:55.970 INFO:teuthology.orchestra.run.smithi076.stdout:======================================================================================== 2021-03-15T22:11:55.971 INFO:teuthology.orchestra.run.smithi076.stdout:Install 13 Packages 2021-03-15T22:11:55.971 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:11:55.972 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 1.9 M 2021-03-15T22:11:55.972 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 24 M 2021-03-15T22:11:55.972 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:11:56.000 INFO:teuthology.orchestra.run.smithi076.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 855 kB/s | 27 kB 00:00 2021-03-15T22:11:56.005 INFO:teuthology.orchestra.run.smithi076.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 26 MB/s | 126 kB 00:00 2021-03-15T22:11:56.006 INFO:teuthology.orchestra.run.smithi076.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 547 kB/s | 20 kB 00:00 2021-03-15T22:11:56.009 INFO:teuthology.orchestra.run.smithi076.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 33 MB/s | 110 kB 00:00 2021-03-15T22:11:56.009 INFO:teuthology.orchestra.run.smithi076.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 42 MB/s | 155 kB 00:00 2021-03-15T22:11:56.010 INFO:teuthology.orchestra.run.smithi076.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 27 MB/s | 43 kB 00:00 2021-03-15T22:11:56.068 INFO:teuthology.orchestra.run.smithi076.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 568 kB/s | 33 kB 00:00 2021-03-15T22:11:56.070 INFO:teuthology.orchestra.run.smithi076.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 276 kB/s | 16 kB 00:00 2021-03-15T22:11:56.078 INFO:teuthology.orchestra.run.smithi076.stdout:(9/13): ceph-mgr-rook-17.0.0-2063.gb658018a.el8 238 kB/s | 26 kB 00:00 2021-03-15T22:11:56.109 INFO:teuthology.orchestra.run.smithi076.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 2.0 MB/s | 81 kB 00:00 2021-03-15T22:11:56.137 INFO:teuthology.orchestra.run.smithi076.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 2.1 MB/s | 61 kB 00:00 2021-03-15T22:11:56.165 INFO:teuthology.orchestra.run.smithi076.stdout:(12/13): python3-rsa-3.4.2-16.el8.noarch.rpm 847 kB/s | 73 kB 00:00 2021-03-15T22:11:56.184 INFO:teuthology.orchestra.run.smithi076.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 10 MB/s | 1.2 MB 00:00 2021-03-15T22:11:56.186 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:11:56.187 INFO:teuthology.orchestra.run.smithi076.stdout:Total 8.8 MB/s | 1.9 MB 00:00 2021-03-15T22:11:56.192 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:11:56.207 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:11:56.207 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:11:56.278 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:11:56.280 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:11:56.746 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:11:57.013 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-03-15T22:11:57.067 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-03-15T22:11:57.266 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-03-15T22:11:57.496 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-03-15T22:11:57.750 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-03-15T22:11:57.975 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-03-15T22:11:58.184 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-03-15T22:11:58.307 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 7/8 2021-03-15T22:11:58.428 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-03-15T22:11:58.489 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-2063.gb658018 8/8 2021-03-15T22:11:58.765 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-2063.gb658018 8/8 2021-03-15T22:11:58.765 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-2063.gb658018 1/8 2021-03-15T22:11:58.766 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 2/8 2021-03-15T22:11:58.766 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-03-15T22:11:58.766 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-03-15T22:11:58.766 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-03-15T22:11:58.767 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 6/8 2021-03-15T22:11:58.767 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-03-15T22:11:59.058 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-03-15T22:11:59.059 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:59.059 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:11:59.059 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-diskprediction-local-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:11:59.059 INFO:teuthology.orchestra.run.smithi125.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-03-15T22:11:59.060 INFO:teuthology.orchestra.run.smithi125.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-03-15T22:11:59.060 INFO:teuthology.orchestra.run.smithi125.stdout: openblas-0.3.3-5.el8.x86_64 2021-03-15T22:11:59.060 INFO:teuthology.orchestra.run.smithi125.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-03-15T22:11:59.060 INFO:teuthology.orchestra.run.smithi125.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-03-15T22:11:59.060 INFO:teuthology.orchestra.run.smithi125.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-03-15T22:11:59.061 INFO:teuthology.orchestra.run.smithi125.stdout: python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x86_64 2021-03-15T22:11:59.061 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:11:59.061 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:11:59.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph-mgr-rook 2021-03-15T22:11:59.465 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-03-15T22:11:59.601 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:34 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:00.040 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:00.042 INFO:teuthology.orchestra.run.smithi125.stdout:======================================================================================== 2021-03-15T22:12:00.042 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:12:00.042 INFO:teuthology.orchestra.run.smithi125.stdout:======================================================================================== 2021-03-15T22:12:00.043 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:12:00.043 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-rook noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 26 k 2021-03-15T22:12:00.043 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:12:00.043 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-03-15T22:12:00.043 INFO:teuthology.orchestra.run.smithi125.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-03-15T22:12:00.044 INFO:teuthology.orchestra.run.smithi125.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-03-15T22:12:00.044 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-03-15T22:12:00.044 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-03-15T22:12:00.044 INFO:teuthology.orchestra.run.smithi125.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-03-15T22:12:00.045 INFO:teuthology.orchestra.run.smithi125.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-03-15T22:12:00.045 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-03-15T22:12:00.045 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-03-15T22:12:00.045 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-03-15T22:12:00.045 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-03-15T22:12:00.046 INFO:teuthology.orchestra.run.smithi125.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-03-15T22:12:00.046 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:00.047 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:12:00.047 INFO:teuthology.orchestra.run.smithi125.stdout:======================================================================================== 2021-03-15T22:12:00.047 INFO:teuthology.orchestra.run.smithi125.stdout:Install 13 Packages 2021-03-15T22:12:00.047 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:00.048 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 1.9 M 2021-03-15T22:12:00.048 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 24 M 2021-03-15T22:12:00.049 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:12:00.110 INFO:teuthology.orchestra.run.smithi125.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 413 kB/s | 27 kB 00:00 2021-03-15T22:12:00.125 INFO:teuthology.orchestra.run.smithi125.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 251 kB/s | 20 kB 00:00 2021-03-15T22:12:00.154 INFO:teuthology.orchestra.run.smithi125.stdout:(3/13): ceph-mgr-rook-17.0.0-2063.gb658018a.el8 239 kB/s | 26 kB 00:00 2021-03-15T22:12:00.159 INFO:teuthology.orchestra.run.smithi125.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 2.5 MB/s | 126 kB 00:00 2021-03-15T22:12:00.174 INFO:teuthology.orchestra.run.smithi125.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 2.2 MB/s | 110 kB 00:00 2021-03-15T22:12:00.175 INFO:teuthology.orchestra.run.smithi125.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 2.8 MB/s | 43 kB 00:00 2021-03-15T22:12:00.198 INFO:teuthology.orchestra.run.smithi125.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 3.5 MB/s | 155 kB 00:00 2021-03-15T22:12:00.233 INFO:teuthology.orchestra.run.smithi125.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 278 kB/s | 16 kB 00:00 2021-03-15T22:12:00.234 INFO:teuthology.orchestra.run.smithi125.stdout:(9/13): python3-cachetools-3.1.1-4.el8.noarch.r 555 kB/s | 33 kB 00:00 2021-03-15T22:12:00.263 INFO:teuthology.orchestra.run.smithi125.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 1.2 MB/s | 81 kB 00:00 2021-03-15T22:12:00.272 INFO:teuthology.orchestra.run.smithi125.stdout:(11/13): python3-rsa-3.4.2-16.el8.noarch.rpm 1.8 MB/s | 73 kB 00:00 2021-03-15T22:12:00.279 INFO:teuthology.orchestra.run.smithi125.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 3.6 MB/s | 61 kB 00:00 2021-03-15T22:12:00.352 INFO:teuthology.orchestra.run.smithi125.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 9.9 MB/s | 1.2 MB 00:00 2021-03-15T22:12:00.354 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:00.354 INFO:teuthology.orchestra.run.smithi125.stdout:Total 6.2 MB/s | 1.9 MB 00:00 2021-03-15T22:12:00.359 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:12:00.374 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:12:00.374 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:12:00.444 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:12:00.446 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:12:00.790 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-03-15T22:12:00.951 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:12:01.007 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-03-15T22:12:01.177 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-03-15T22:12:01.216 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-03-15T22:12:01.367 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-03-15T22:12:01.430 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-03-15T22:12:01.484 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-rook-2:17.0.0-2063.gb658018a.el8.noarch 13/13 2021-03-15T22:12:01.668 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-03-15T22:12:01.727 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-2063.gb658018a.el8.noarch 13/13 2021-03-15T22:12:01.727 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-rook-2:17.0.0-2063.gb658018a.el8.noarch 1/13 2021-03-15T22:12:01.727 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-03-15T22:12:01.727 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-03-15T22:12:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-03-15T22:12:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-03-15T22:12:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-03-15T22:12:01.728 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-03-15T22:12:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-03-15T22:12:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-03-15T22:12:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-03-15T22:12:01.729 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-03-15T22:12:01.730 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-03-15T22:12:01.874 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-03-15T22:12:02.071 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-03-15T22:12:02.088 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-03-15T22:12:02.088 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:02.089 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:12:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:12:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-03-15T22:12:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-03-15T22:12:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-03-15T22:12:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-03-15T22:12:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-03-15T22:12:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-03-15T22:12:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-03-15T22:12:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-03-15T22:12:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-03-15T22:12:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-03-15T22:12:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-03-15T22:12:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-03-15T22:12:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:02.092 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:02.175 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-cephadm 2021-03-15T22:12:02.320 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-03-15T22:12:02.579 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-03-15T22:12:02.641 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:37 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:02.803 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-03-15T22:12:03.096 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:03.145 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:03.145 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T22:12:03.145 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:03.146 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:12:03.146 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 79 k 2021-03-15T22:12:03.146 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T22:12:03.146 INFO:teuthology.orchestra.run.smithi076.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-03-15T22:12:03.147 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-03-15T22:12:03.147 INFO:teuthology.orchestra.run.smithi076.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-03-15T22:12:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-03-15T22:12:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-03-15T22:12:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:03.148 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:12:03.148 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:03.149 INFO:teuthology.orchestra.run.smithi076.stdout:Install 6 Packages 2021-03-15T22:12:03.149 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:03.149 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 5.8 M 2021-03-15T22:12:03.149 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 24 M 2021-03-15T22:12:03.150 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:12:03.150 INFO:teuthology.orchestra.run.smithi076.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 11 MB/s | 538 kB 00:00 2021-03-15T22:12:03.201 INFO:teuthology.orchestra.run.smithi076.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 48 MB/s | 4.8 MB 00:00 2021-03-15T22:12:03.215 INFO:teuthology.orchestra.run.smithi076.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 327 kB/s | 21 kB 00:00 2021-03-15T22:12:03.232 INFO:teuthology.orchestra.run.smithi076.stdout:(4/6): ceph-mgr-cephadm-17.0.0-2063.gb658018a.e 596 kB/s | 79 kB 00:00 2021-03-15T22:12:03.256 INFO:teuthology.orchestra.run.smithi076.stdout:(5/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.2 MB/s | 48 kB 00:00 2021-03-15T22:12:03.336 INFO:teuthology.orchestra.run.smithi076.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 2.6 MB/s | 355 kB 00:00 2021-03-15T22:12:03.337 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:03.338 INFO:teuthology.orchestra.run.smithi076.stdout:Total 24 MB/s | 5.8 MB 00:00 2021-03-15T22:12:03.341 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:12:03.351 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:12:03.351 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:12:03.428 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:12:03.429 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:12:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:12:03.853 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-03-15T22:12:03.953 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-03-15T22:12:04.079 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-03-15T22:12:04.171 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-03-15T22:12:04.271 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-03-15T22:12:04.430 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-03-15T22:12:04.786 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-rook-2:17.0.0-2063.gb658018a.el8.noarch 13/13 2021-03-15T22:12:05.061 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-03-15T22:12:05.536 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-03-15T22:12:05.601 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-2063.gb658018a.el8.noarch 13/13 2021-03-15T22:12:05.601 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-rook-2:17.0.0-2063.gb658018a.el8.noarch 1/13 2021-03-15T22:12:05.602 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-03-15T22:12:05.602 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-03-15T22:12:05.602 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-03-15T22:12:05.602 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-03-15T22:12:05.603 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-03-15T22:12:05.603 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-03-15T22:12:05.603 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-03-15T22:12:05.603 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-03-15T22:12:05.603 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-03-15T22:12:05.604 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-03-15T22:12:05.604 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-03-15T22:12:05.731 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-03-15T22:12:05.891 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-2063.gb658018a.el8.noarch 6/6 2021-03-15T22:12:05.924 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-03-15T22:12:05.925 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:05.925 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:12:05.925 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-rook-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:12:05.925 INFO:teuthology.orchestra.run.smithi125.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-03-15T22:12:05.926 INFO:teuthology.orchestra.run.smithi125.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-03-15T22:12:05.926 INFO:teuthology.orchestra.run.smithi125.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-03-15T22:12:05.926 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-03-15T22:12:05.926 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-03-15T22:12:05.927 INFO:teuthology.orchestra.run.smithi125.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-03-15T22:12:05.927 INFO:teuthology.orchestra.run.smithi125.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-03-15T22:12:05.927 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-03-15T22:12:05.927 INFO:teuthology.orchestra.run.smithi125.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-03-15T22:12:05.927 INFO:teuthology.orchestra.run.smithi125.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-03-15T22:12:05.928 INFO:teuthology.orchestra.run.smithi125.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-03-15T22:12:05.928 INFO:teuthology.orchestra.run.smithi125.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-03-15T22:12:05.928 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:05.928 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:06.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph-mgr-cephadm 2021-03-15T22:12:06.122 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-2063.gb658018a.el8.noarch 6/6 2021-03-15T22:12:06.122 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-2063.gb658018a.el8.noarch 1/6 2021-03-15T22:12:06.122 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-03-15T22:12:06.123 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-03-15T22:12:06.123 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-03-15T22:12:06.123 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-03-15T22:12:06.395 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-03-15T22:12:06.396 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:06.396 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:12:06.396 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:12:06.396 INFO:teuthology.orchestra.run.smithi076.stdout: python3-apipkg-1.5-6.el8.noarch 2021-03-15T22:12:06.396 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel-2.5.1-5.el8.noarch 2021-03-15T22:12:06.397 INFO:teuthology.orchestra.run.smithi076.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-03-15T22:12:06.397 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-03-15T22:12:06.397 INFO:teuthology.orchestra.run.smithi076.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-03-15T22:12:06.397 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:06.398 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:06.476 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:41 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:06.497 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-fuse 2021-03-15T22:12:06.921 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:06.922 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:06.922 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:12:06.922 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:06.922 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:12:06.923 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-cephadm noarch 2:17.0.0-2063.gb658018a.el8 ceph-noarch 79 k 2021-03-15T22:12:06.923 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:12:06.923 INFO:teuthology.orchestra.run.smithi125.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-03-15T22:12:06.923 INFO:teuthology.orchestra.run.smithi125.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-03-15T22:12:06.924 INFO:teuthology.orchestra.run.smithi125.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-03-15T22:12:06.924 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-03-15T22:12:06.924 INFO:teuthology.orchestra.run.smithi125.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-03-15T22:12:06.924 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:06.925 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:12:06.925 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:06.925 INFO:teuthology.orchestra.run.smithi125.stdout:Install 6 Packages 2021-03-15T22:12:06.925 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:06.926 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 5.8 M 2021-03-15T22:12:06.926 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 24 M 2021-03-15T22:12:06.926 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:12:06.962 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:41 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:07.039 INFO:teuthology.orchestra.run.smithi125.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 4.6 MB/s | 538 kB 00:00 2021-03-15T22:12:07.054 INFO:teuthology.orchestra.run.smithi125.stdout:(2/6): ceph-mgr-cephadm-17.0.0-2063.gb658018a.e 607 kB/s | 79 kB 00:00 2021-03-15T22:12:07.089 INFO:teuthology.orchestra.run.smithi125.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 428 kB/s | 21 kB 00:00 2021-03-15T22:12:07.121 INFO:teuthology.orchestra.run.smithi125.stdout:(4/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.4 MB/s | 48 kB 00:00 2021-03-15T22:12:07.154 INFO:teuthology.orchestra.run.smithi125.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 3.5 MB/s | 355 kB 00:00 2021-03-15T22:12:07.184 INFO:teuthology.orchestra.run.smithi125.stdout:(6/6): python3-babel-2.5.1-5.el8.noarch.rpm 18 MB/s | 4.8 MB 00:00 2021-03-15T22:12:07.185 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:07.186 INFO:teuthology.orchestra.run.smithi125.stdout:Total 22 MB/s | 5.8 MB 00:00 2021-03-15T22:12:07.189 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:12:07.198 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:12:07.199 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:12:07.275 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:12:07.277 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:12:07.411 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:07.411 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:07.412 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2021-03-15T22:12:07.412 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:07.412 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:12:07.412 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse x86_64 2:17.0.0-2063.gb658018a.el8 ceph 827 k 2021-03-15T22:12:07.412 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2021-03-15T22:12:07.413 INFO:teuthology.orchestra.run.smithi076.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-03-15T22:12:07.413 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:07.413 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:12:07.413 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:07.414 INFO:teuthology.orchestra.run.smithi076.stdout:Install 2 Packages 2021-03-15T22:12:07.414 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:07.415 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 909 k 2021-03-15T22:12:07.415 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 3.2 M 2021-03-15T22:12:07.415 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:12:07.446 INFO:teuthology.orchestra.run.smithi076.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.4 MB/s | 82 kB 00:00 2021-03-15T22:12:07.529 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:12:07.625 INFO:teuthology.orchestra.run.smithi076.stdout:(2/2): ceph-fuse-17.0.0-2063.gb658018a.el8.x86_ 3.8 MB/s | 827 kB 00:00 2021-03-15T22:12:07.625 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:07.626 INFO:teuthology.orchestra.run.smithi076.stdout:Total 4.2 MB/s | 909 kB 00:00 2021-03-15T22:12:07.627 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:12:07.650 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:12:07.650 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:12:07.723 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:12:07.725 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:12:07.782 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-03-15T22:12:07.884 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:12:08.000 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-03-15T22:12:08.141 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-03-15T22:12:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 2/2 2021-03-15T22:12:08.631 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 2/2 2021-03-15T22:12:08.631 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 1/2 2021-03-15T22:12:08.900 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-03-15T22:12:08.944 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-03-15T22:12:08.945 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:08.945 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:12:08.945 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-03-15T22:12:08.946 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:08.946 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:09.117 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados-devel 2021-03-15T22:12:09.246 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-03-15T22:12:09.501 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-03-15T22:12:09.588 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:44 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:09.688 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-2063.gb658018a.el8.noarch 6/6 2021-03-15T22:12:09.910 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-2063.gb658018a.el8.noarch 6/6 2021-03-15T22:12:09.911 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-2063.gb658018a.el8.noarch 1/6 2021-03-15T22:12:09.911 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-03-15T22:12:09.911 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-03-15T22:12:09.911 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-03-15T22:12:09.912 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-03-15T22:12:10.040 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:10.042 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:10.042 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2021-03-15T22:12:10.042 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:10.042 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:12:10.043 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel x86_64 2:17.0.0-2063.gb658018a.el8 ceph 125 k 2021-03-15T22:12:10.043 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:10.043 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:12:10.043 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:10.044 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2021-03-15T22:12:10.044 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:10.044 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 125 k 2021-03-15T22:12:10.044 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 439 k 2021-03-15T22:12:10.044 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:12:10.193 INFO:teuthology.orchestra.run.smithi076.stdout:librados-devel-17.0.0-2063.gb658018a.el8.x86_64 828 kB/s | 125 kB 00:00 2021-03-15T22:12:10.194 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:10.194 INFO:teuthology.orchestra.run.smithi076.stdout:Total 821 kB/s | 125 kB 00:00 2021-03-15T22:12:10.195 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:12:10.198 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-03-15T22:12:10.198 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:10.198 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:12:10.198 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-mgr-cephadm-2:17.0.0-2063.gb658018a.el8.noarch 2021-03-15T22:12:10.199 INFO:teuthology.orchestra.run.smithi125.stdout: python3-apipkg-1.5-6.el8.noarch 2021-03-15T22:12:10.199 INFO:teuthology.orchestra.run.smithi125.stdout: python3-babel-2.5.1-5.el8.noarch 2021-03-15T22:12:10.199 INFO:teuthology.orchestra.run.smithi125.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-03-15T22:12:10.199 INFO:teuthology.orchestra.run.smithi125.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-03-15T22:12:10.200 INFO:teuthology.orchestra.run.smithi125.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-03-15T22:12:10.200 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:10.200 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:10.202 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:12:10.202 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:12:10.251 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:12:10.254 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:12:10.315 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install ceph-fuse 2021-03-15T22:12:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:12:10.714 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:10.777 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:45 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:10.949 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:11.211 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:11.211 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:11.211 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:12:11.212 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:12:11.212 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:11.212 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:11.299 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:11.300 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:11.300 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repository Size 2021-03-15T22:12:11.300 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:11.300 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:12:11.300 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-fuse x86_64 2:17.0.0-2063.gb658018a.el8 ceph 827 k 2021-03-15T22:12:11.301 INFO:teuthology.orchestra.run.smithi125.stdout:Installing dependencies: 2021-03-15T22:12:11.301 INFO:teuthology.orchestra.run.smithi125.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-03-15T22:12:11.302 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:11.302 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:12:11.302 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:11.302 INFO:teuthology.orchestra.run.smithi125.stdout:Install 2 Packages 2021-03-15T22:12:11.303 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:11.304 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 909 k 2021-03-15T22:12:11.304 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 3.2 M 2021-03-15T22:12:11.304 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:12:11.333 INFO:teuthology.orchestra.run.smithi125.stdout:(1/2): fuse-2.9.7-12.el8.x86_64.rpm 2.6 MB/s | 82 kB 00:00 2021-03-15T22:12:11.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install libcephfs2 2021-03-15T22:12:11.512 INFO:teuthology.orchestra.run.smithi125.stdout:(2/2): ceph-fuse-17.0.0-2063.gb658018a.el8.x86_ 3.8 MB/s | 827 kB 00:00 2021-03-15T22:12:11.513 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:11.514 INFO:teuthology.orchestra.run.smithi125.stdout:Total 4.2 MB/s | 909 kB 00:00 2021-03-15T22:12:11.515 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:12:11.538 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:12:11.538 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:12:11.609 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:12:11.611 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:12:11.787 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:12:11.883 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:46 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:12.037 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : fuse-2.9.7-12.el8.x86_64 1/2 2021-03-15T22:12:12.186 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : ceph-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 2/2 2021-03-15T22:12:12.229 INFO:teuthology.orchestra.run.smithi076.stdout:Package libcephfs2-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:12.343 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:12.344 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T22:12:12.344 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:12.418 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install libcephfs-devel 2021-03-15T22:12:12.495 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: ceph-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 2/2 2021-03-15T22:12:12.495 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : ceph-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 1/2 2021-03-15T22:12:12.769 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/2 2021-03-15T22:12:12.770 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:12.770 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:12:12.770 INFO:teuthology.orchestra.run.smithi125.stdout: ceph-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-03-15T22:12:12.770 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:12.771 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:12.886 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:47 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:12.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install librados-devel 2021-03-15T22:12:13.335 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:13.336 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2021-03-15T22:12:13.337 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:13.337 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:12:13.337 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel x86_64 2:17.0.0-2063.gb658018a.el8 ceph 23 k 2021-03-15T22:12:13.338 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:13.338 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:12:13.338 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:13.338 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2021-03-15T22:12:13.339 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:13.339 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 23 k 2021-03-15T22:12:13.339 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 93 k 2021-03-15T22:12:13.339 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:12:13.411 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:48 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:13.447 INFO:teuthology.orchestra.run.smithi076.stdout:libcephfs-devel-17.0.0-2063.gb658018a.el8.x86_6 212 kB/s | 23 kB 00:00 2021-03-15T22:12:13.448 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:13.448 INFO:teuthology.orchestra.run.smithi076.stdout:Total 210 kB/s | 23 kB 00:00 2021-03-15T22:12:13.449 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:12:13.453 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:12:13.453 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:12:13.455 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:12:13.458 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:12:13.541 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:12:13.681 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:13.863 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:13.864 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:13.864 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repo Size 2021-03-15T22:12:13.864 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:13.865 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:12:13.865 INFO:teuthology.orchestra.run.smithi125.stdout: librados-devel x86_64 2:17.0.0-2063.gb658018a.el8 ceph 125 k 2021-03-15T22:12:13.865 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:13.865 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:12:13.866 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:13.866 INFO:teuthology.orchestra.run.smithi125.stdout:Install 1 Package 2021-03-15T22:12:13.866 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:13.866 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 125 k 2021-03-15T22:12:13.867 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 439 k 2021-03-15T22:12:13.867 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:12:13.881 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:14.014 INFO:teuthology.orchestra.run.smithi125.stdout:librados-devel-17.0.0-2063.gb658018a.el8.x86_64 833 kB/s | 125 kB 00:00 2021-03-15T22:12:14.015 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:14.016 INFO:teuthology.orchestra.run.smithi125.stdout:Total 826 kB/s | 125 kB 00:00 2021-03-15T22:12:14.016 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:12:14.023 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:12:14.023 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:12:14.073 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:12:14.074 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:12:14.110 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:14.111 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:14.111 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:12:14.111 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:12:14.112 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:14.112 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:14.185 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:12:14.194 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados2 2021-03-15T22:12:14.361 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : librados-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:14.595 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: librados-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:14.661 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:49 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:14.882 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : librados-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:14.882 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:14.883 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:12:14.883 INFO:teuthology.orchestra.run.smithi125.stdout: librados-devel-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:12:14.883 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:14.884 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:15.004 INFO:teuthology.orchestra.run.smithi076.stdout:Package librados2-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:15.034 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install libcephfs2 2021-03-15T22:12:15.119 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:15.121 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T22:12:15.121 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:15.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librbd1 2021-03-15T22:12:15.505 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:50 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:15.659 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:50 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:15.840 INFO:teuthology.orchestra.run.smithi125.stdout:Package libcephfs2-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:15.954 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:15.955 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:12:15.955 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:16.027 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install libcephfs-devel 2021-03-15T22:12:16.104 INFO:teuthology.orchestra.run.smithi076.stdout:Package librbd1-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:16.219 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:16.220 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T22:12:16.220 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:16.294 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rados 2021-03-15T22:12:16.493 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:51 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:16.762 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:51 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:16.944 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:16.945 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:16.945 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repo Size 2021-03-15T22:12:16.945 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:16.946 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:12:16.946 INFO:teuthology.orchestra.run.smithi125.stdout: libcephfs-devel x86_64 2:17.0.0-2063.gb658018a.el8 ceph 23 k 2021-03-15T22:12:16.946 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:16.946 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:12:16.946 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:16.947 INFO:teuthology.orchestra.run.smithi125.stdout:Install 1 Package 2021-03-15T22:12:16.947 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:16.947 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 23 k 2021-03-15T22:12:16.947 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 93 k 2021-03-15T22:12:16.947 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:12:17.054 INFO:teuthology.orchestra.run.smithi125.stdout:libcephfs-devel-17.0.0-2063.gb658018a.el8.x86_6 214 kB/s | 23 kB 00:00 2021-03-15T22:12:17.055 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:17.056 INFO:teuthology.orchestra.run.smithi125.stdout:Total 212 kB/s | 23 kB 00:00 2021-03-15T22:12:17.056 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:12:17.060 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:12:17.060 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:12:17.062 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:12:17.064 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:12:17.095 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rados-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:17.119 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:12:17.206 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:17.207 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T22:12:17.208 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:17.277 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : libcephfs-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:17.293 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rgw 2021-03-15T22:12:17.497 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:17.721 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : libcephfs-devel-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:17.721 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:17.722 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:12:17.722 INFO:teuthology.orchestra.run.smithi125.stdout: libcephfs-devel-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:12:17.722 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:17.722 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:17.762 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:52 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:17.831 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install librados2 2021-03-15T22:12:18.112 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rgw-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:18.227 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:18.229 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T22:12:18.229 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:18.295 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:53 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:18.307 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-cephfs 2021-03-15T22:12:18.631 INFO:teuthology.orchestra.run.smithi125.stdout:Package librados2-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:18.745 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:18.746 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:12:18.746 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:18.775 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:53 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:18.829 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install librbd1 2021-03-15T22:12:19.112 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-cephfs-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:19.223 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:19.224 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T22:12:19.224 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:19.293 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rbd 2021-03-15T22:12:19.296 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:54 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:19.631 INFO:teuthology.orchestra.run.smithi125.stdout:Package librbd1-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:19.743 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:19.744 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:12:19.744 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:19.759 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:54 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:19.817 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install python3-rados 2021-03-15T22:12:20.111 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rbd-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:20.224 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:20.225 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2021-03-15T22:12:20.225 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:20.282 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:55 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:20.302 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-fuse 2021-03-15T22:12:20.614 INFO:teuthology.orchestra.run.smithi125.stdout:Package python3-rados-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:20.729 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:20.730 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:12:20.731 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:20.769 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:55 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:20.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install python3-rgw 2021-03-15T22:12:21.275 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:56 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:21.810 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:21.811 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2021-03-15T22:12:21.812 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:21.812 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:12:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse x86_64 2:17.0.0-2063.gb658018a.el8 ceph 86 k 2021-03-15T22:12:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:21.813 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:12:21.813 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:21.814 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2021-03-15T22:12:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:21.814 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 86 k 2021-03-15T22:12:21.814 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 298 k 2021-03-15T22:12:21.815 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:12:21.859 INFO:teuthology.orchestra.run.smithi125.stdout:Package python3-rgw-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:21.956 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-fuse-17.0.0-2063.gb658018a.el8.x86_64.rpm 595 kB/s | 86 kB 00:00 2021-03-15T22:12:21.957 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:21.958 INFO:teuthology.orchestra.run.smithi076.stdout:Total 588 kB/s | 86 kB 00:00 2021-03-15T22:12:21.958 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:12:21.970 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:21.970 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:12:21.971 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:12:21.971 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:12:21.972 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:22.026 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:12:22.027 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:12:22.042 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install python3-cephfs 2021-03-15T22:12:22.508 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:57 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:22.708 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:12:22.847 INFO:teuthology.orchestra.run.smithi125.stdout:Package python3-cephfs-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:22.871 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:22.958 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:22.959 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:12:22.960 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:23.033 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install python3-rbd 2021-03-15T22:12:23.111 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:23.397 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:23.398 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:12:23.398 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:12:23.399 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:23.399 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:23.498 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:58 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:23.638 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-mirror 2021-03-15T22:12:23.831 INFO:teuthology.orchestra.run.smithi125.stdout:Package python3-rbd-2:17.0.0-2063.gb658018a.el8.x86_64 is already installed. 2021-03-15T22:12:23.941 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:23.942 INFO:teuthology.orchestra.run.smithi125.stdout:Nothing to do. 2021-03-15T22:12:23.943 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:24.016 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install rbd-fuse 2021-03-15T22:12:24.111 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:59 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:24.480 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:01:59 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:24.564 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:24.565 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2021-03-15T22:12:24.566 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:24.566 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:12:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror x86_64 2:17.0.0-2063.gb658018a.el8 ceph 3.4 M 2021-03-15T22:12:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:24.567 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:12:24.567 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:24.567 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2021-03-15T22:12:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:24.567 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 3.4 M 2021-03-15T22:12:24.568 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 19 M 2021-03-15T22:12:24.568 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:12:24.906 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-mirror-17.0.0-2063.gb658018a.el8.x86_64.rpm 9.9 MB/s | 3.4 MB 00:00 2021-03-15T22:12:24.907 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:24.908 INFO:teuthology.orchestra.run.smithi076.stdout:Total 9.9 MB/s | 3.4 MB 00:00 2021-03-15T22:12:24.908 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:12:24.922 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:12:24.923 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:12:24.938 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:24.939 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:24.939 INFO:teuthology.orchestra.run.smithi125.stdout: Package Architecture Version Repository Size 2021-03-15T22:12:24.939 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:24.939 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:12:24.940 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-fuse x86_64 2:17.0.0-2063.gb658018a.el8 ceph 86 k 2021-03-15T22:12:24.940 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:24.940 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:12:24.940 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:24.941 INFO:teuthology.orchestra.run.smithi125.stdout:Install 1 Package 2021-03-15T22:12:24.941 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:24.941 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 86 k 2021-03-15T22:12:24.941 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 298 k 2021-03-15T22:12:24.941 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:12:24.984 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:12:24.986 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:12:25.069 INFO:teuthology.orchestra.run.smithi125.stdout:rbd-fuse-17.0.0-2063.gb658018a.el8.x86_64.rpm 661 kB/s | 86 kB 00:00 2021-03-15T22:12:25.070 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:25.070 INFO:teuthology.orchestra.run.smithi125.stdout:Total 654 kB/s | 86 kB 00:00 2021-03-15T22:12:25.070 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:12:25.082 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:12:25.082 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:12:25.131 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:12:25.133 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:12:25.246 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:12:25.451 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : rbd-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:25.452 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:12:25.606 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-mirror-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:25.686 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: rbd-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:25.951 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-mirror-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:25.958 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : rbd-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:25.959 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:25.959 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:12:25.959 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-fuse-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:12:25.959 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:25.960 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:26.164 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install rbd-mirror 2021-03-15T22:12:26.263 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-mirror-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:26.263 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:26.264 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:12:26.264 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:12:26.264 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:26.264 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:26.430 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-nbd 2021-03-15T22:12:26.642 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:02:01 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:26.906 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:01 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:27.099 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:27.100 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:27.101 INFO:teuthology.orchestra.run.smithi125.stdout: Package Arch Version Repo Size 2021-03-15T22:12:27.101 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:27.101 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:12:27.101 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-mirror x86_64 2:17.0.0-2063.gb658018a.el8 ceph 3.4 M 2021-03-15T22:12:27.102 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:27.102 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:12:27.102 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:27.102 INFO:teuthology.orchestra.run.smithi125.stdout:Install 1 Package 2021-03-15T22:12:27.102 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:27.103 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 3.4 M 2021-03-15T22:12:27.103 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 19 M 2021-03-15T22:12:27.103 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:12:27.365 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2021-03-15T22:12:27.366 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2021-03-15T22:12:27.367 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:27.367 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2021-03-15T22:12:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd x86_64 2:17.0.0-2063.gb658018a.el8 ceph 165 k 2021-03-15T22:12:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:27.368 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2021-03-15T22:12:27.368 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2021-03-15T22:12:27.368 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2021-03-15T22:12:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:27.369 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 165 k 2021-03-15T22:12:27.369 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 649 k 2021-03-15T22:12:27.369 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2021-03-15T22:12:27.465 INFO:teuthology.orchestra.run.smithi125.stdout:rbd-mirror-17.0.0-2063.gb658018a.el8.x86_64.rpm 9.3 MB/s | 3.4 MB 00:00 2021-03-15T22:12:27.466 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:27.466 INFO:teuthology.orchestra.run.smithi125.stdout:Total 9.2 MB/s | 3.4 MB 00:00 2021-03-15T22:12:27.467 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:12:27.486 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:12:27.487 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:12:27.523 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-nbd-17.0.0-2063.gb658018a.el8.x86_64.rpm 1.0 MB/s | 165 kB 00:00 2021-03-15T22:12:27.524 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:27.525 INFO:teuthology.orchestra.run.smithi076.stdout:Total 1.0 MB/s | 165 kB 00:00 2021-03-15T22:12:27.525 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2021-03-15T22:12:27.538 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2021-03-15T22:12:27.538 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2021-03-15T22:12:27.548 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:12:27.550 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:12:27.589 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2021-03-15T22:12:27.590 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2021-03-15T22:12:27.864 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2021-03-15T22:12:28.390 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:12:28.521 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-nbd-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:28.753 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-nbd-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:29.004 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-nbd-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:29.004 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:29.005 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2021-03-15T22:12:29.005 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:12:29.005 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:12:29.005 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2021-03-15T22:12:29.152 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : rbd-mirror-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:29.200 DEBUG:teuthology.parallel:result is None 2021-03-15T22:12:29.510 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: rbd-mirror-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:29.779 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : rbd-mirror-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:29.779 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:29.780 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:12:29.780 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-mirror-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:12:29.780 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:29.780 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:29.941 DEBUG:teuthology.orchestra.run.smithi125:> sudo yum -y install rbd-nbd 2021-03-15T22:12:30.412 INFO:teuthology.orchestra.run.smithi125.stdout:Last metadata expiration check: 0:02:05 ago on Mon 15 Mar 2021 10:10:25 PM UTC. 2021-03-15T22:12:30.863 INFO:teuthology.orchestra.run.smithi125.stdout:Dependencies resolved. 2021-03-15T22:12:30.863 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:30.864 INFO:teuthology.orchestra.run.smithi125.stdout: Package Architecture Version Repository Size 2021-03-15T22:12:30.864 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:30.864 INFO:teuthology.orchestra.run.smithi125.stdout:Installing: 2021-03-15T22:12:30.864 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-nbd x86_64 2:17.0.0-2063.gb658018a.el8 ceph 165 k 2021-03-15T22:12:30.865 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:30.865 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction Summary 2021-03-15T22:12:30.865 INFO:teuthology.orchestra.run.smithi125.stdout:================================================================================ 2021-03-15T22:12:30.865 INFO:teuthology.orchestra.run.smithi125.stdout:Install 1 Package 2021-03-15T22:12:30.865 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:30.866 INFO:teuthology.orchestra.run.smithi125.stdout:Total download size: 165 k 2021-03-15T22:12:30.867 INFO:teuthology.orchestra.run.smithi125.stdout:Installed size: 649 k 2021-03-15T22:12:30.867 INFO:teuthology.orchestra.run.smithi125.stdout:Downloading Packages: 2021-03-15T22:12:31.014 INFO:teuthology.orchestra.run.smithi125.stdout:rbd-nbd-17.0.0-2063.gb658018a.el8.x86_64.rpm 1.1 MB/s | 165 kB 00:00 2021-03-15T22:12:31.015 INFO:teuthology.orchestra.run.smithi125.stdout:-------------------------------------------------------------------------------- 2021-03-15T22:12:31.015 INFO:teuthology.orchestra.run.smithi125.stdout:Total 1.0 MB/s | 165 kB 00:00 2021-03-15T22:12:31.015 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction check 2021-03-15T22:12:31.029 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction check succeeded. 2021-03-15T22:12:31.029 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction test 2021-03-15T22:12:31.079 INFO:teuthology.orchestra.run.smithi125.stdout:Transaction test succeeded. 2021-03-15T22:12:31.081 INFO:teuthology.orchestra.run.smithi125.stdout:Running transaction 2021-03-15T22:12:31.196 INFO:teuthology.orchestra.run.smithi125.stdout: Preparing : 1/1 2021-03-15T22:12:31.383 INFO:teuthology.orchestra.run.smithi125.stdout: Installing : rbd-nbd-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:31.624 INFO:teuthology.orchestra.run.smithi125.stdout: Running scriptlet: rbd-nbd-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:31.937 INFO:teuthology.orchestra.run.smithi125.stdout: Verifying : rbd-nbd-2:17.0.0-2063.gb658018a.el8.x86_64 1/1 2021-03-15T22:12:31.937 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:31.938 INFO:teuthology.orchestra.run.smithi125.stdout:Installed: 2021-03-15T22:12:31.938 INFO:teuthology.orchestra.run.smithi125.stdout: rbd-nbd-2:17.0.0-2063.gb658018a.el8.x86_64 2021-03-15T22:12:31.938 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:12:31.939 INFO:teuthology.orchestra.run.smithi125.stdout:Complete! 2021-03-15T22:12:32.111 DEBUG:teuthology.parallel:result is None 2021-03-15T22:12:32.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo lsb_release -is 2021-03-15T22:12:32.152 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS 2021-03-15T22:12:32.153 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-15T22:12:32.154 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b658018acf46e32d2ec283d302076352e17476f5 2021-03-15T22:12:32.292 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-03-15T22:12:32.313 INFO:teuthology.orchestra.run.smithi076.stdout:17.0.0-2063.gb658018a.el8 2021-03-15T22:12:32.314 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-2063.gb658018a.el8 2021-03-15T22:12:32.315 INFO:teuthology.task.install:The correct ceph version 17.0.0-2063.gb658018a is installed. 2021-03-15T22:12:32.315 DEBUG:teuthology.orchestra.run.smithi125:> sudo lsb_release -is 2021-03-15T22:12:32.350 INFO:teuthology.orchestra.run.smithi125.stdout:CentOS 2021-03-15T22:12:32.351 DEBUG:teuthology.misc:System to be installed: CentOS 2021-03-15T22:12:32.351 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=b658018acf46e32d2ec283d302076352e17476f5 2021-03-15T22:12:32.485 DEBUG:teuthology.orchestra.run.smithi125:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-03-15T22:12:32.504 INFO:teuthology.orchestra.run.smithi125.stdout:17.0.0-2063.gb658018a.el8 2021-03-15T22:12:32.505 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-2063.gb658018a.el8 2021-03-15T22:12:32.505 INFO:teuthology.task.install:The correct ceph version 17.0.0-2063.gb658018a is installed. 2021-03-15T22:12:32.506 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-03-15T22:12:32.506 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:12:32.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-15T22:12:32.535 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-15T22:12:32.536 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-15T22:12:32.568 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-03-15T22:12:32.569 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:12:32.569 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2021-03-15T22:12:32.602 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-15T22:12:32.669 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-15T22:12:32.670 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/daemon-helper 2021-03-15T22:12:32.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-15T22:12:32.759 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-03-15T22:12:32.759 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:12:32.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-15T22:12:32.786 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-15T22:12:32.851 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-15T22:12:32.852 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-15T22:12:32.877 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-15T22:12:32.941 INFO:teuthology.run_tasks:Running task cephadm... 2021-03-15T22:12:33.021 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'global': {'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b658018acf46e32d2ec283d302076352e17476f5', 'cephadm_mode': 'root'} 2021-03-15T22:12:33.022 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 2021-03-15T22:12:33.023 INFO:tasks.cephadm:Cluster fsid is 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:12:33.023 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2021-03-15T22:12:33.024 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.76', 'mon.c': '[v2:172.21.15.76:3301,v1:172.21.15.76:6790]', 'mon.b': '172.21.15.125'} 2021-03-15T22:12:33.024 INFO:tasks.cephadm:First mon is mon.a on smithi076 2021-03-15T22:12:33.025 INFO:tasks.cephadm:First mgr is y 2021-03-15T22:12:33.025 INFO:tasks.cephadm:Normalizing hostnames... 2021-03-15T22:12:33.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname $(hostname -s) 2021-03-15T22:12:33.055 DEBUG:teuthology.orchestra.run.smithi125:> sudo hostname $(hostname -s) 2021-03-15T22:12:33.084 INFO:tasks.cephadm:Downloading cephadm (repo git://git.ceph.com/ceph-ci.git ref b658018acf46e32d2ec283d302076352e17476f5)... 2021-03-15T22:12:33.084 DEBUG:teuthology.orchestra.run.smithi076:> git archive --remote=git://git.ceph.com/ceph-ci.git b658018acf46e32d2ec283d302076352e17476f5 src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2021-03-15T22:12:33.237 DEBUG:teuthology.orchestra.run.smithi125:> git archive --remote=git://git.ceph.com/ceph-ci.git b658018acf46e32d2ec283d302076352e17476f5 src/cephadm/cephadm | tar -xO src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm 2021-03-15T22:12:33.375 DEBUG:teuthology.orchestra.run.smithi076:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-03-15T22:12:33.392 DEBUG:teuthology.orchestra.run.smithi125:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2021-03-15T22:12:33.448 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph 2021-03-15T22:12:33.475 DEBUG:teuthology.orchestra.run.smithi125:> sudo mkdir -p /etc/ceph 2021-03-15T22:12:33.500 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /etc/ceph 2021-03-15T22:12:33.541 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 777 /etc/ceph 2021-03-15T22:12:33.566 INFO:tasks.cephadm:Writing seed config... 2021-03-15T22:12:33.567 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2021-03-15T22:12:33.568 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2021-03-15T22:12:33.568 INFO:tasks.cephadm: override: [global] ms type = async 2021-03-15T22:12:33.568 INFO:tasks.cephadm: override: [mon] debug mon = 20 2021-03-15T22:12:33.569 INFO:tasks.cephadm: override: [mon] debug ms = 1 2021-03-15T22:12:33.569 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2021-03-15T22:12:33.569 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2021-03-15T22:12:33.570 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2021-03-15T22:12:33.570 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2021-03-15T22:12:33.570 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2021-03-15T22:12:33.571 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2021-03-15T22:12:33.571 INFO:tasks.cephadm: override: [osd] debug ms = 1 2021-03-15T22:12:33.571 INFO:tasks.cephadm: override: [osd] debug osd = 20 2021-03-15T22:12:33.572 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2021-03-15T22:12:33.572 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2021-03-15T22:12:33.572 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2021-03-15T22:12:33.573 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2021-03-15T22:12:33.573 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2021-03-15T22:12:33.573 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2021-03-15T22:12:33.573 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2021-03-15T22:12:33.574 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:12:33.575 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2021-03-15T22:12:33.597 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8a572802-85db-11eb-9091-001a4aab830c ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2021-03-15T22:12:33.598 DEBUG:teuthology.orchestra.run.smithi076:mon.a> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@mon.a.service 2021-03-15T22:12:33.640 DEBUG:teuthology.orchestra.run.smithi076:mgr.y> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@mgr.y.service 2021-03-15T22:12:33.665 INFO:journalctl@ceph.mon.a.smithi076.stdout:-- Logs begin at Mon 2021-03-15 21:44:23 UTC. -- 2021-03-15T22:12:33.666 INFO:tasks.cephadm:Bootstrapping... 2021-03-15T22:12:33.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -v bootstrap --fsid 8a572802-85db-11eb-9091-001a4aab830c --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.76 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2021-03-15T22:12:33.690 INFO:journalctl@ceph.mgr.y.smithi076.stdout:-- Logs begin at Mon 2021-03-15 21:44:23 UTC. -- 2021-03-15T22:12:34.131 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman --version 2021-03-15T22:12:34.170 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: podman version 3.0.1 2021-03-15T22:12:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:Verifying podman|docker is present... 2021-03-15T22:12:34.172 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman --version 2021-03-15T22:12:34.205 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: podman version 3.0.1 2021-03-15T22:12:34.207 INFO:teuthology.orchestra.run.smithi076.stderr:Verifying lvm2 is present... 2021-03-15T22:12:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:Verifying time synchronization is in place... 2021-03-15T22:12:34.208 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-enabled chrony.service 2021-03-15T22:12:34.215 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-03-15T22:12:34.215 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-active chrony.service 2021-03-15T22:12:34.220 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: inactive 2021-03-15T22:12:34.221 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-enabled chronyd.service 2021-03-15T22:12:34.227 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: enabled 2021-03-15T22:12:34.228 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-active chronyd.service 2021-03-15T22:12:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: active 2021-03-15T22:12:34.233 INFO:teuthology.orchestra.run.smithi076.stderr:Unit chronyd.service is enabled and running 2021-03-15T22:12:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:Repeating the final host check... 2021-03-15T22:12:34.234 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman --version 2021-03-15T22:12:34.266 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: podman version 3.0.1 2021-03-15T22:12:34.268 INFO:teuthology.orchestra.run.smithi076.stderr:podman|docker (/bin/podman) is present 2021-03-15T22:12:34.268 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl is present 2021-03-15T22:12:34.268 INFO:teuthology.orchestra.run.smithi076.stderr:lvcreate is present 2021-03-15T22:12:34.269 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-enabled chrony.service 2021-03-15T22:12:34.274 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2021-03-15T22:12:34.275 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-active chrony.service 2021-03-15T22:12:34.280 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: inactive 2021-03-15T22:12:34.280 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-enabled chronyd.service 2021-03-15T22:12:34.286 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: enabled 2021-03-15T22:12:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-active chronyd.service 2021-03-15T22:12:34.292 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: active 2021-03-15T22:12:34.293 INFO:teuthology.orchestra.run.smithi076.stderr:Unit chronyd.service is enabled and running 2021-03-15T22:12:34.293 INFO:teuthology.orchestra.run.smithi076.stderr:Host looks OK 2021-03-15T22:12:34.293 INFO:teuthology.orchestra.run.smithi076.stderr:Cluster fsid: 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:12:34.294 INFO:teuthology.orchestra.run.smithi076.stderr:Acquiring lock 140588105407848 on /run/cephadm/8a572802-85db-11eb-9091-001a4aab830c.lock 2021-03-15T22:12:34.294 INFO:teuthology.orchestra.run.smithi076.stderr:Lock 140588105407848 acquired on /run/cephadm/8a572802-85db-11eb-9091-001a4aab830c.lock 2021-03-15T22:12:34.295 INFO:teuthology.orchestra.run.smithi076.stderr:Verifying IP 172.21.15.76 port 3300 ... 2021-03-15T22:12:34.295 INFO:teuthology.orchestra.run.smithi076.stderr:Verifying IP 172.21.15.76 port 6789 ... 2021-03-15T22:12:34.295 INFO:teuthology.orchestra.run.smithi076.stderr:Base mon IP is 172.21.15.76, final addrv is [v2:172.21.15.76:3300,v1:172.21.15.76:6789] 2021-03-15T22:12:34.296 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /sbin/ip route ls 2021-03-15T22:12:34.296 INFO:teuthology.orchestra.run.smithi076.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp metric 100 2021-03-15T22:12:34.297 INFO:teuthology.orchestra.run.smithi076.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.76 metric 100 2021-03-15T22:12:34.297 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /sbin/ip -6 route ls 2021-03-15T22:12:34.298 INFO:teuthology.orchestra.run.smithi076.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2021-03-15T22:12:34.299 INFO:teuthology.orchestra.run.smithi076.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2021-03-15T22:12:34.299 INFO:teuthology.orchestra.run.smithi076.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2021-03-15T22:12:34.300 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /sbin/ip -6 addr ls 2021-03-15T22:12:34.301 INFO:teuthology.orchestra.run.smithi076.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2021-03-15T22:12:34.302 INFO:teuthology.orchestra.run.smithi076.stderr:/sbin/ip: inet6 ::1/128 scope host 2021-03-15T22:12:34.302 INFO:teuthology.orchestra.run.smithi076.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-03-15T22:12:34.302 INFO:teuthology.orchestra.run.smithi076.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2021-03-15T22:12:34.302 INFO:teuthology.orchestra.run.smithi076.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:47e1/64 scope link noprefixroute 2021-03-15T22:12:34.302 INFO:teuthology.orchestra.run.smithi076.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2021-03-15T22:12:34.303 INFO:teuthology.orchestra.run.smithi076.stderr:Mon IP 172.21.15.76 is in CIDR network 172.21.0.0/20 2021-03-15T22:12:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2021-03-15T22:12:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5... 2021-03-15T22:12:34.304 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman pull quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 2021-03-15T22:12:35.448 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Trying to pull quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5... 2021-03-15T22:12:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Getting image source signatures 2021-03-15T22:12:35.620 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:35.678 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:889524391d554cc9bd269d2e47820d562c512d9d60fd7d4da881803a833083d7 2021-03-15T22:12:35.695 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:35.720 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:35.762 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:35.970 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:36.037 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:36.038 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:36.038 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:36.038 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:36.039 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:36.039 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:36.039 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:36.039 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:36.039 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 2021-03-15T22:12:36.137 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Copying blob sha256:2cc651cf80bf41c5b7e064a18b3ea49b2f6eec0f1e2a9d66b35ac5d762f8bbe1 2021-03-15T22:12:53.993 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Writing manifest to image destination 2021-03-15T22:12:53.994 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: Storing signatures 2021-03-15T22:13:03.522 INFO:teuthology.orchestra.run.smithi076.stderr:/bin/podman: 0568d198570d73d7f62cf803fde56ce5825204163bed23bf36718971e3f940d9 2021-03-15T22:13:03.536 INFO:teuthology.orchestra.run.smithi076.stderr:Extracting ceph user uid/gid from container image... 2021-03-15T22:13:03.536 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -c %u %g /var/lib/ceph 2021-03-15T22:13:04.117 INFO:teuthology.orchestra.run.smithi076.stderr:stat: 167 167 2021-03-15T22:13:14.708 INFO:teuthology.orchestra.run.smithi076.stderr:Creating initial keys... 2021-03-15T22:13:14.709 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 --gen-print-key 2021-03-15T22:13:15.157 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-authtool: AQD7209gUA1cBxAABGOgN0A54So66Zz2FIvBBA== 2021-03-15T22:13:15.609 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 --gen-print-key 2021-03-15T22:13:16.040 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-authtool: AQD8209ggfJLARAATHBrheyY+ieU+xFqF8mZ2A== 2021-03-15T22:13:16.501 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 --gen-print-key 2021-03-15T22:13:16.890 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-authtool: AQD8209gt9SGNBAA8nxH6LE4vaAgf+eBDB764A== 2021-03-15T22:13:17.380 INFO:teuthology.orchestra.run.smithi076.stderr:Creating initial monmap... 2021-03-15T22:13:17.380 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmpu_87xhw1:/tmp/monmap:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 --create --clobber --fsid 8a572802-85db-11eb-9091-001a4aab830c --addv a [v2:172.21.15.76:3300,v1:172.21.15.76:6789] /tmp/monmap 2021-03-15T22:13:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2021-03-15T22:13:17.840 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/monmaptool: setting min_mon_release = octopus 2021-03-15T22:13:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:17.841 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-03-15T22:13:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:monmaptool for a [v2:172.21.15.76:3300,v1:172.21.15.76:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2021-03-15T22:13:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:setting min_mon_release = octopus 2021-03-15T22:13:18.363 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/monmaptool: set fsid to 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2021-03-15T22:13:18.364 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T22:13:18.364 INFO:teuthology.orchestra.run.smithi076.stderr:Creating mon... 2021-03-15T22:13:18.365 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmpsvx7blos:/tmp/keyring:z -v /tmp/ceph-tmpu_87xhw1:/tmp/monmap:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 --mkfs -i a --fsid 8a572802-85db-11eb-9091-001a4aab830c -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2021-03-15T22:13:18.844 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.803+0000 7f71ca92a700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-15T22:13:18.844 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.803+0000 7f71ca92a700 1 imported monmap: 2021-03-15T22:13:18.845 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: epoch 0 2021-03-15T22:13:18.845 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: fsid 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:18.845 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: last_changed 2021-03-15T22:13:17.824467+0000 2021-03-15T22:13:18.845 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: created 2021-03-15T22:13:17.824467+0000 2021-03-15T22:13:18.846 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: min_mon_release 15 (octopus) 2021-03-15T22:13:18.846 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: election_strategy: 1 2021-03-15T22:13:18.846 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon.a 2021-03-15T22:13:18.846 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.847 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.803+0000 7f71ca92a700 0 /usr/bin/ceph-mon: set fsid to 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:18.847 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: RocksDB version: 6.8.1 2021-03-15T22:13:18.847 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.847 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-15T22:13:18.848 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Compile date Mar 15 2021 2021-03-15T22:13:18.848 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: DB SUMMARY 2021-03-15T22:13:18.848 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.848 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 0, files: 2021-03-15T22:13:18.849 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.849 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 2021-03-15T22:13:18.849 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.849 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.error_if_exists: 0 2021-03-15T22:13:18.849 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.create_if_missing: 1 2021-03-15T22:13:18.850 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.paranoid_checks: 1 2021-03-15T22:13:18.850 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.env: 0x558e9cd221c0 2021-03-15T22:13:18.850 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.fs: Posix File System 2021-03-15T22:13:18.850 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.info_log: 0x558e9d992ee0 2021-03-15T22:13:18.851 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-15T22:13:18.851 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.statistics: (nil) 2021-03-15T22:13:18.851 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.use_fsync: 0 2021-03-15T22:13:18.851 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.max_log_file_size: 0 2021-03-15T22:13:18.852 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-15T22:13:18.852 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-15T22:13:18.852 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-15T22:13:18.852 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-15T22:13:18.853 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.allow_fallocate: 1 2021-03-15T22:13:18.853 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-15T22:13:18.853 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-15T22:13:18.853 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.use_direct_reads: 0 2021-03-15T22:13:18.853 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-15T22:13:18.854 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-15T22:13:18.854 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.db_log_dir: 2021-03-15T22:13:18.854 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-03-15T22:13:18.854 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-15T22:13:18.855 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.max_subcompactions: 1 2021-03-15T22:13:18.855 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.max_background_flushes: -1 2021-03-15T22:13:18.855 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-15T22:13:18.855 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-15T22:13:18.855 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-15T22:13:18.856 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-15T22:13:18.856 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-15T22:13:18.856 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-15T22:13:18.856 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-15T22:13:18.857 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.write_buffer_manager: 0x558e9d99d290 2021-03-15T22:13:18.857 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-15T22:13:18.857 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-15T22:13:18.857 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-15T22:13:18.858 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-15T22:13:18.858 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-15T22:13:18.858 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-15T22:13:18.858 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-15T22:13:18.859 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-15T22:13:18.859 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-15T22:13:18.859 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.unordered_write: 0 2021-03-15T22:13:18.859 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-15T22:13:18.859 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-15T22:13:18.860 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-15T22:13:18.860 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-15T22:13:18.860 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.row_cache: None 2021-03-15T22:13:18.860 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.wal_filter: None 2021-03-15T22:13:18.861 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-15T22:13:18.861 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-15T22:13:18.861 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.preserve_deletes: 0 2021-03-15T22:13:18.861 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.two_write_queues: 0 2021-03-15T22:13:18.862 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-15T22:13:18.862 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.atomic_flush: 0 2021-03-15T22:13:18.862 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-15T22:13:18.862 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-15T22:13:18.862 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-15T22:13:18.863 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.log_readahead_size: 0 2021-03-15T22:13:18.863 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-15T22:13:18.863 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.max_background_jobs: 2 2021-03-15T22:13:18.863 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.max_background_compactions: -1 2021-03-15T22:13:18.864 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-15T22:13:18.864 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-15T22:13:18.864 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-15T22:13:18.864 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-15T22:13:18.864 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-15T22:13:18.865 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-15T22:13:18.865 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-15T22:13:18.865 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-15T22:13:18.865 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.max_open_files: -1 2021-03-15T22:13:18.865 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-15T22:13:18.866 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-15T22:13:18.866 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-15T22:13:18.866 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-15T22:13:18.866 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Compression algorithms supported: 2021-03-15T22:13:18.866 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-15T22:13:18.867 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: kZSTD supported: 0 2021-03-15T22:13:18.867 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: kXpressCompression supported: 0 2021-03-15T22:13:18.867 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-15T22:13:18.867 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: kLZ4Compression supported: 1 2021-03-15T22:13:18.867 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: kBZip2Compression supported: 0 2021-03-15T22:13:18.868 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: kZlibCompression supported: 1 2021-03-15T22:13:18.868 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: kSnappyCompression supported: 1 2021-03-15T22:13:18.868 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-15T22:13:18.868 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2021-03-15T22:13:18.868 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.869 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 2021-03-15T22:13:18.869 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.869 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-15T22:13:18.869 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.870 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-15T22:13:18.870 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.merge_operator: 2021-03-15T22:13:18.870 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.compaction_filter: None 2021-03-15T22:13:18.870 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-15T22:13:18.870 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-15T22:13:18.871 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.804+0000 7f71ca92a700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-15T22:13:18.871 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558e9d8a4230) 2021-03-15T22:13:18.871 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2021-03-15T22:13:18.871 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-15T22:13:18.872 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-15T22:13:18.872 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2021-03-15T22:13:18.872 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: index_type: 0 2021-03-15T22:13:18.872 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2021-03-15T22:13:18.873 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: index_shortening: 1 2021-03-15T22:13:18.873 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2021-03-15T22:13:18.873 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2021-03-15T22:13:18.873 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: checksum: 1 2021-03-15T22:13:18.873 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2021-03-15T22:13:18.874 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: block_cache: 0x558e9d8ded10 2021-03-15T22:13:18.874 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2021-03-15T22:13:18.874 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: block_cache_options: 2021-03-15T22:13:18.874 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: capacity : 536870912 2021-03-15T22:13:18.874 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2021-03-15T22:13:18.875 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2021-03-15T22:13:18.875 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2021-03-15T22:13:18.875 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2021-03-15T22:13:18.875 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2021-03-15T22:13:18.876 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: block_size: 4096 2021-03-15T22:13:18.876 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2021-03-15T22:13:18.876 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2021-03-15T22:13:18.876 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2021-03-15T22:13:18.877 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2021-03-15T22:13:18.877 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: partition_filters: 0 2021-03-15T22:13:18.877 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2021-03-15T22:13:18.877 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-15T22:13:18.877 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2021-03-15T22:13:18.878 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: verify_compression: 0 2021-03-15T22:13:18.878 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2021-03-15T22:13:18.878 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: format_version: 2 2021-03-15T22:13:18.878 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2021-03-15T22:13:18.879 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: block_align: 0 2021-03-15T22:13:18.879 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.879 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-15T22:13:18.879 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-15T22:13:18.879 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compression: NoCompression 2021-03-15T22:13:18.880 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-15T22:13:18.880 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-15T22:13:18.880 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-15T22:13:18.880 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.num_levels: 7 2021-03-15T22:13:18.880 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-15T22:13:18.881 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-15T22:13:18.881 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-15T22:13:18.881 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-15T22:13:18.881 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-15T22:13:18.882 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-15T22:13:18.882 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-15T22:13:18.882 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-15T22:13:18.882 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-15T22:13:18.883 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-15T22:13:18.883 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-15T22:13:18.883 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-15T22:13:18.883 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-15T22:13:18.883 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-15T22:13:18.884 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-15T22:13:18.884 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-15T22:13:18.884 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-15T22:13:18.884 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-15T22:13:18.884 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-15T22:13:18.885 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-15T22:13:18.885 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-15T22:13:18.885 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-15T22:13:18.885 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-15T22:13:18.886 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-15T22:13:18.886 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-15T22:13:18.886 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-15T22:13:18.886 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-15T22:13:18.887 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-15T22:13:18.887 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-15T22:13:18.887 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-15T22:13:18.887 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-15T22:13:18.888 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-15T22:13:18.888 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-15T22:13:18.888 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-15T22:13:18.888 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-15T22:13:18.888 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-15T22:13:18.889 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-15T22:13:18.889 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-15T22:13:18.889 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-15T22:13:18.889 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-15T22:13:18.889 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-15T22:13:18.890 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-15T22:13:18.890 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-15T22:13:18.890 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-15T22:13:18.890 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-15T22:13:18.891 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-15T22:13:18.891 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-15T22:13:18.891 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.table_properties_collectors: 2021-03-15T22:13:18.891 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.inplace_update_support: 0 2021-03-15T22:13:18.892 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-15T22:13:18.892 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-15T22:13:18.892 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-15T22:13:18.892 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-15T22:13:18.892 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.bloom_locality: 0 2021-03-15T22:13:18.893 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.max_successive_merges: 0 2021-03-15T22:13:18.893 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-15T22:13:18.893 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-15T22:13:18.893 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-15T22:13:18.894 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-15T22:13:18.894 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.ttl: 2592000 2021-03-15T22:13:18.894 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-15T22:13:18.894 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-15T22:13:18.894 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.895 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.805+0000 7f71ca92a700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2021-03-15T22:13:18.895 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.895 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.806+0000 7f71ca92a700 4 rocksdb: DB pointer 0x558e9d9a7800 2021-03-15T22:13:18.895 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.806+0000 7f71bb9d2700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-15T22:13:18.896 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.806+0000 7f71bb9d2700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-15T22:13:18.896 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2021-03-15T22:13:18.896 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-03-15T22:13:18.896 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-15T22:13:18.897 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-15T22:13:18.897 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-15T22:13:18.897 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-15T22:13:18.897 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-15T22:13:18.897 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-15T22:13:18.898 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.898 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-15T22:13:18.898 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-15T22:13:18.898 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-15T22:13:18.898 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-15T22:13:18.899 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-15T22:13:18.899 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.899 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-15T22:13:18.899 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-15T22:13:18.900 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-15T22:13:18.900 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-03-15T22:13:18.900 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-03-15T22:13:18.900 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-15T22:13:18.900 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-03-15T22:13:18.901 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-03-15T22:13:18.901 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-03-15T22:13:18.901 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-15T22:13:18.901 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-15T22:13:18.901 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-15T22:13:18.902 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.902 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-03-15T22:13:18.902 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.902 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-15T22:13:18.903 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-15T22:13:18.903 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-15T22:13:18.903 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-15T22:13:18.903 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-15T22:13:18.903 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.904 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2021-03-15T22:13:18.904 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-15T22:13:18.904 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-15T22:13:18.904 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2021-03-15T22:13:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2021-03-15T22:13:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-15T22:13:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2021-03-15T22:13:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2021-03-15T22:13:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2021-03-15T22:13:18.906 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-15T22:13:18.906 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-15T22:13:18.906 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-15T22:13:18.906 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2021-03-15T22:13:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: 2021-03-15T22:13:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.807+0000 7f71ca92a700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2021-03-15T22:13:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.807+0000 7f71ca92a700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2021-03-15T22:13:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph-mon: debug 2021-03-15T22:13:18.807+0000 7f71ca92a700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-a for mon.a 2021-03-15T22:13:19.425 INFO:teuthology.orchestra.run.smithi076.stderr:create mon.a on 2021-03-15T22:13:19.426 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:19.432 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl disable ceph.target 2021-03-15T22:13:19.440 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2021-03-15T22:13:19.517 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl enable ceph.target 2021-03-15T22:13:19.525 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2021-03-15T22:13:19.601 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl start ceph.target 2021-03-15T22:13:19.608 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl enable ceph-8a572802-85db-11eb-9091-001a4aab830c.target 2021-03-15T22:13:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8a572802-85db-11eb-9091-001a4aab830c.target → /etc/systemd/system/ceph-8a572802-85db-11eb-9091-001a4aab830c.target. 2021-03-15T22:13:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-8a572802-85db-11eb-9091-001a4aab830c.target → /etc/systemd/system/ceph-8a572802-85db-11eb-9091-001a4aab830c.target. 2021-03-15T22:13:19.689 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl start ceph-8a572802-85db-11eb-9091-001a4aab830c.target 2021-03-15T22:13:19.696 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl daemon-reload 2021-03-15T22:13:19.776 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl stop ceph-8a572802-85db-11eb-9091-001a4aab830c@mon.a 2021-03-15T22:13:19.802 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl reset-failed ceph-8a572802-85db-11eb-9091-001a4aab830c@mon.a 2021-03-15T22:13:19.808 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: Failed to reset failed state of unit ceph-8a572802-85db-11eb-9091-001a4aab830c@mon.a.service: Unit ceph-8a572802-85db-11eb-9091-001a4aab830c@mon.a.service not loaded. 2021-03-15T22:13:19.809 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl enable ceph-8a572802-85db-11eb-9091-001a4aab830c@mon.a 2021-03-15T22:13:19.815 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: Created symlink /etc/systemd/system/ceph-8a572802-85db-11eb-9091-001a4aab830c.target.wants/ceph-8a572802-85db-11eb-9091-001a4aab830c@mon.a.service → /etc/systemd/system/ceph-8a572802-85db-11eb-9091-001a4aab830c@.service. 2021-03-15T22:13:19.889 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl start ceph-8a572802-85db-11eb-9091-001a4aab830c@mon.a 2021-03-15T22:13:20.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:19 smithi076 systemd[1]: Starting Ceph mon.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:13:20.348 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-enabled firewalld.service 2021-03-15T22:13:20.357 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: disabled 2021-03-15T22:13:20.358 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-active firewalld.service 2021-03-15T22:13:20.364 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: inactive 2021-03-15T22:13:20.365 INFO:teuthology.orchestra.run.smithi076.stderr:firewalld.service is not enabled 2021-03-15T22:13:20.366 INFO:teuthology.orchestra.run.smithi076.stderr:Not possible to enable service . firewalld.service is not available 2021-03-15T22:13:20.366 INFO:teuthology.orchestra.run.smithi076.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-03-15T22:13:20.367 INFO:teuthology.orchestra.run.smithi076.stderr:Waiting for mon to start... 2021-03-15T22:13:20.367 INFO:teuthology.orchestra.run.smithi076.stderr:Waiting for mon... 2021-03-15T22:13:20.367 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 status 2021-03-15T22:13:20.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:20 smithi076 podman[55873]: 2021-03-15 22:13:20.188623293 +0000 UTC m=+0.164528458 container create 23ca579867fc61ca335ea77f5b08b46ad294a0c50bfceaabd1ce6f1619de208b (image=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mon.a, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20201204, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) 2021-03-15T22:13:20.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:20 smithi076 podman[55873]: 2021-03-15 22:13:20.296942759 +0000 UTC m=+0.272847792 container init 23ca579867fc61ca335ea77f5b08b46ad294a0c50bfceaabd1ce6f1619de208b (image=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mon.a, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Dimitri Savineau , io.buildah.version=1.18.0, GIT_BRANCH=HEAD) 2021-03-15T22:13:20.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:20 smithi076 podman[55873]: 2021-03-15 22:13:20.338486603 +0000 UTC m=+0.314391643 container start 23ca579867fc61ca335ea77f5b08b46ad294a0c50bfceaabd1ce6f1619de208b (image=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mon.a, org.label-schema.build-date=20201204, GIT_CLEAN=True, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD) 2021-03-15T22:13:20.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:20 smithi076 bash[55799]: 23ca579867fc61ca335ea77f5b08b46ad294a0c50bfceaabd1ce6f1619de208b 2021-03-15T22:13:20.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:20 smithi076 systemd[1]: Started Ceph mon.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:13:20.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:20 smithi076 conmon[55914]: cluster 2021-03-15T22:13:20.370967+0000 mon.a (mon.0) 0 : [INF] 2021-03-15T22:13:20.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:20 smithi076 conmon[55914]: mkfs 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:20.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:20 smithi076 conmon[55914]: cluster 2021-03-15T22:13:20.353310+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-03-15T22:13:20.976 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: cluster: 2021-03-15T22:13:20.976 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: id: 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:20.976 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: health: HEALTH_OK 2021-03-15T22:13:20.977 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 2021-03-15T22:13:20.977 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: services: 2021-03-15T22:13:20.977 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: mon: 1 daemons, quorum a (age 0.60768s) 2021-03-15T22:13:20.977 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: mgr: no daemons active 2021-03-15T22:13:20.978 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2021-03-15T22:13:20.978 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 2021-03-15T22:13:20.978 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: data: 2021-03-15T22:13:20.978 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2021-03-15T22:13:20.979 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2021-03-15T22:13:20.979 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2021-03-15T22:13:20.979 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: pgs: 2021-03-15T22:13:20.979 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 2021-03-15T22:13:21.627 INFO:teuthology.orchestra.run.smithi076.stderr:mon is available 2021-03-15T22:13:21.628 INFO:teuthology.orchestra.run.smithi076.stderr:Assimilating anything we can from ceph.conf... 2021-03-15T22:13:21.628 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z -v /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 config assimilate-conf -i /var/lib/ceph/mon/ceph-a/config 2021-03-15T22:13:21.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:21 smithi076 conmon[55914]: cluster 2021-03-15T22:13:20.365840+0000 mon.a (mon.0) 2 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-03-15T22:13:21.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:21 smithi076 conmon[55914]: cluster 2021-03-15T22: 2021-03-15T22:13:21.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:21 smithi076 conmon[55914]: 13:20.368155+0000 mon.a (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0]} 2021-03-15T22:13:21.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:21 smithi076 conmon[55914]: cluster 2021-03-15T22:13:20.376419+0000 mon.a (mon.0) 4 : cluster [DBG] fsmap 2021-03-15T22:13:21.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:21 smithi076 conmon[55914]: cluster 2021-03-15T22:13:20.379170+0000 mon.a (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-03-15T22:13:21.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:21 smithi076 conmon[55914]: cluster 2021-03-15T22:13:20.379599+0000 mon.a (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2021-03-15T22:13:21.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:21 smithi076 conmon[55914]: audit 2021-03-15T22:13:20.973484+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.76:0/4219000721' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2021-03-15T22:13:22.237 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 2021-03-15T22:13:22.238 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: [global] 2021-03-15T22:13:22.238 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: fsid = 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:22.238 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.76:3300,v1:172.21.15.76:6789] 2021-03-15T22:13:22.239 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2021-03-15T22:13:22.239 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2021-03-15T22:13:22.239 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2021-03-15T22:13:22.239 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2021-03-15T22:13:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 2021-03-15T22:13:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: [mgr] 2021-03-15T22:13:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2021-03-15T22:13:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 2021-03-15T22:13:22.241 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: [osd] 2021-03-15T22:13:22.241 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2021-03-15T22:13:22.241 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2021-03-15T22:13:22.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:22 smithi076 conmon[55914]: audit 2021-03-15T22:13:22.231983+0000 mon.a (mon. 2021-03-15T22:13:22.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:22 smithi076 conmon[55914]: 0) 8 : audit [INF] from='client.? 172.21.15.76:0/1762421956' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2021-03-15T22:13:22.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:22 smithi076 conmon[55914]: audit 2021-03-15T22:13:22.234484+0000 mon.a (mon.0) 9 : audit [INF] from='client.? 172.21.15.76:0/1762421956' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2021-03-15T22:13:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:Generating new minimal ceph.conf... 2021-03-15T22:13:22.791 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z -v /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a:/var/lib/ceph/mon/ceph-a:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-a/config 2021-03-15T22:13:23.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:23 smithi076 conmon[55914]: audit 2021-03-15T22:13:23.367971+0000 mon.a (mon.0 2021-03-15T22:13:23.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:23 smithi076 conmon[55914]: ) 10 : audit [DBG] from='client.? 172.21.15.76:0/2228179409' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:13:23.875 INFO:teuthology.orchestra.run.smithi076.stderr:Restarting the monitor... 2021-03-15T22:13:23.875 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl restart ceph-8a572802-85db-11eb-9091-001a4aab830c@mon.a 2021-03-15T22:13:24.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:23 smithi076 systemd[1]: Stopping Ceph mon.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:13:24.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[55914]: debug 2021-03-15T22:13:23.999+0000 7f228453e700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2021-03-15T22:13:24.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[55914]: debug 2021-03-15T22:13:23.999+0000 7f228453e700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2021-03-15T22:13:24.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 podman[56347]: 2021-03-15 22:13:24.43802479 +0000 UTC m=+0.385602323 container remove 23ca579867fc61ca335ea77f5b08b46ad294a0c50bfceaabd1ce6f1619de208b (image=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mon.a, ceph=True, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_CLEAN=True, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.license=GPLv2, io.buildah.version=1.18.0, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0) 2021-03-15T22:13:24.756 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 podman[56306]: 23ca579867fc61ca335ea77f5b08b46ad294a0c50bfceaabd1ce6f1619de208b 2021-03-15T22:13:24.756 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 systemd[1]: Stopped Ceph mon.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:13:24.756 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 systemd[1]: Starting Ceph mon.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:13:24.888 INFO:teuthology.orchestra.run.smithi076.stderr:Setting mon public_network to 172.21.0.0/20 2021-03-15T22:13:24.889 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 config set mon public_network 172.21.0.0/20 2021-03-15T22:13:25.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 podman[56472]: 2021-03-15 22:13:24.754603646 +0000 UTC m=+0.149527427 container create 1884de2119e1f8c3f49d6df74d9192fdb0b12c28c6df2a81b093c2f0056738a0 (image=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mon.a, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.18.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) 2021-03-15T22:13:25.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 podman[56472]: 2021-03-15 22:13:24.846340592 +0000 UTC m=+0.241264471 container init 1884de2119e1f8c3f49d6df74d9192fdb0b12c28c6df2a81b093c2f0056738a0 (image=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mon.a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.build-date=20201204, GIT_CLEAN=True, ceph=True, io.buildah.version=1.18.0, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2021-03-15T22:13:25.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.878+0000 7f054645c700 0 set uid:gid to 167:167 (ceph:ceph) 2021-03-15T22:13:25.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.878+0000 7f054645c700 0 ceph version 17.0.0-2063-gb658018a (b658018acf46e32d2ec283d302076352e17476f5) quincy (dev), process ceph-mon, pid 2 2021-03-15T22:13:25.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.878+0000 7f054645c700 0 pidfile_write: ignore empty --pid-file 2021-03-15T22:13:25.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 podman[56472]: 2021-03-15 22:13:24.879559209 +0000 UTC m=+0.274482983 container start 1884de2119e1f8c3f49d6df74d9192fdb0b12c28c6df2a81b093c2f0056738a0 (image=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mon.a, org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True) 2021-03-15T22:13:25.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 bash[56397]: 1884de2119e1f8c3f49d6df74d9192fdb0b12c28c6df2a81b093c2f0056738a0 2021-03-15T22:13:25.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.882+0000 7f054645c700 0 load: jerasure load: lrc load: isa 2021-03-15T22:13:25.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: RocksDB version: 6.8.1 2021-03-15T22:13:25.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2021-03-15T22:13:25.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Compile date Mar 15 2021 2021-03-15T22:13:25.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: DB SUMMARY 2021-03-15T22:13:25.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: CURRENT file: CURRENT 2021-03-15T22:13:25.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: IDENTITY file: IDENTITY 2021-03-15T22:13:25.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2021-03-15T22:13:25.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000004.sst 2021-03-15T22:13:25.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000006.log size: 87451 ; 2021-03-15T22:13:25.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.error_if_exists: 0 2021-03-15T22:13:25.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.create_if_missing: 0 2021-03-15T22:13:25.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.paranoid_checks: 1 2021-03-15T22:13:25.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.env: 0x562d9d4ca1c0 2021-03-15T22:13:25.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.fs: Posix File System 2021-03-15T22:13:25.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.info_log: 0x562d9fa6d1c0 2021-03-15T22:13:25.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_file_opening_threads: 16 2021-03-15T22:13:25.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.statistics: (nil) 2021-03-15T22:13:25.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.use_fsync: 0 2021-03-15T22:13:25.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_log_file_size: 0 2021-03-15T22:13:25.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2021-03-15T22:13:25.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.log_file_time_to_roll: 0 2021-03-15T22:13:25.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.keep_log_file_num: 1000 2021-03-15T22:13:25.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.recycle_log_file_num: 0 2021-03-15T22:13:25.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.allow_fallocate: 1 2021-03-15T22:13:25.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.allow_mmap_reads: 0 2021-03-15T22:13:25.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.allow_mmap_writes: 0 2021-03-15T22:13:25.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.use_direct_reads: 0 2021-03-15T22:13:25.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2021-03-15T22:13:25.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.create_missing_column_families: 0 2021-03-15T22:13:25.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.db_log_dir: 2021-03-15T22:13:25.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-a/store.db 2021-03-15T22:13:25.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.table_cache_numshardbits: 6 2021-03-15T22:13:25.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_subcompactions: 1 2021-03-15T22:13:25.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_background_flushes: -1 2021-03-15T22:13:25.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.WAL_ttl_seconds: 0 2021-03-15T22:13:25.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.WAL_size_limit_MB: 0 2021-03-15T22:13:25.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2021-03-15T22:13:25.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2021-03-15T22:13:25.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.is_fd_close_on_exec: 1 2021-03-15T22:13:25.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.advise_random_on_open: 1 2021-03-15T22:13:25.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.db_write_buffer_size: 0 2021-03-15T22:13:25.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.write_buffer_manager: 0x562d9fb07d10 2021-03-15T22:13:25.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2021-03-15T22:13:25.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2021-03-15T22:13:25.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2021-03-15T22:13:25.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.use_adaptive_mutex: 0 2021-03-15T22:13:25.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.rate_limiter: (nil) 2021-03-15T22:13:25.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2021-03-15T22:13:25.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.wal_recovery_mode: 2 2021-03-15T22:13:25.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.enable_thread_tracking: 0 2021-03-15T22:13:25.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.enable_pipelined_write: 0 2021-03-15T22:13:25.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.unordered_write: 0 2021-03-15T22:13:25.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2021-03-15T22:13:25.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2021-03-15T22:13:25.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2021-03-15T22:13:25.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2021-03-15T22:13:25.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.row_cache: None 2021-03-15T22:13:25.038 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.wal_filter: None 2021-03-15T22:13:25.038 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2021-03-15T22:13:25.038 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.allow_ingest_behind: 0 2021-03-15T22:13:25.038 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.preserve_deletes: 0 2021-03-15T22:13:25.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.two_write_queues: 0 2021-03-15T22:13:25.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.manual_wal_flush: 0 2021-03-15T22:13:25.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.atomic_flush: 0 2021-03-15T22:13:25.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2021-03-15T22:13:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.persist_stats_to_disk: 0 2021-03-15T22:13:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.write_dbid_to_manifest: 0 2021-03-15T22:13:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.log_readahead_size: 0 2021-03-15T22:13:25.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2021-03-15T22:13:25.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_background_jobs: 2 2021-03-15T22:13:25.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_background_compactions: -1 2021-03-15T22:13:25.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2021-03-15T22:13:25.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2021-03-15T22:13:25.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.delayed_write_rate : 16777216 2021-03-15T22:13:25.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_total_wal_size: 0 2021-03-15T22:13:25.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2021-03-15T22:13:25.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.stats_dump_period_sec: 600 2021-03-15T22:13:25.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.stats_persist_period_sec: 600 2021-03-15T22:13:25.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2021-03-15T22:13:25.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_open_files: -1 2021-03-15T22:13:25.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.bytes_per_sync: 0 2021-03-15T22:13:25.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.wal_bytes_per_sync: 0 2021-03-15T22:13:25.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.strict_bytes_per_sync: 0 2021-03-15T22:13:25.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_readahead_size: 0 2021-03-15T22:13:25.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Compression algorithms supported: 2021-03-15T22:13:25.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2021-03-15T22:13:25.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: kZSTD supported: 0 2021-03-15T22:13:25.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: kXpressCompression supported: 0 2021-03-15T22:13:25.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: kLZ4HCCompression supported: 1 2021-03-15T22:13:25.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: kLZ4Compression supported: 1 2021-03-15T22:13:25.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: kBZip2Compression supported: 0 2021-03-15T22:13:25.046 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: kZlibCompression supported: 1 2021-03-15T22:13:25.046 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: kSnappyCompression supported: 1 2021-03-15T22:13:25.046 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Fast CRC32 supported: Supported on x86 2021-03-15T22:13:25.047 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 2021-03-15T22:13:25.047 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.047 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2021-03-15T22:13:25.047 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.048 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2021-03-15T22:13:25.048 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.merge_operator: 2021-03-15T22:13:25.048 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_filter: None 2021-03-15T22:13:25.048 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_filter_factory: None 2021-03-15T22:13:25.048 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.memtable_factory: SkipListFactory 2021-03-15T22:13:25.049 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.table_factory: BlockBasedTable 2021-03-15T22:13:25.049 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x562d9fa0e240) 2021-03-15T22:13:25.049 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: cache_index_and_filter_blocks: 1 2021-03-15T22:13:25.049 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: cache_index_and_filter_blocks_with_high_priority: 0 2021-03-15T22:13:25.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: pin_l0_filter_and_index_blocks_in_cache: 0 2021-03-15T22:13:25.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: pin_top_level_index_and_filter: 1 2021-03-15T22:13:25.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: index_type: 0 2021-03-15T22:13:25.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: data_block_index_type: 0 2021-03-15T22:13:25.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: index_shortening: 1 2021-03-15T22:13:25.051 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: data_block_hash_table_util_ratio: 0.750000 2021-03-15T22:13:25.051 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: hash_index_allow_collision: 1 2021-03-15T22:13:25.051 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: checksum: 1 2021-03-15T22:13:25.051 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: no_block_cache: 0 2021-03-15T22:13:25.052 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: block_cache: 0x562d9fa48f10 2021-03-15T22:13:25.052 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: block_cache_name: BinnedLRUCache 2021-03-15T22:13:25.052 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: block_cache_options: 2021-03-15T22:13:25.053 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: capacity : 536870912 2021-03-15T22:13:25.053 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: num_shard_bits : 4 2021-03-15T22:13:25.053 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: strict_capacity_limit : 0 2021-03-15T22:13:25.053 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: high_pri_pool_ratio: 0.000 2021-03-15T22:13:25.053 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: block_cache_compressed: (nil) 2021-03-15T22:13:25.055 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: persistent_cache: (nil) 2021-03-15T22:13:25.055 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: block_size: 4096 2021-03-15T22:13:25.056 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: block_size_deviation: 10 2021-03-15T22:13:25.056 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: block_restart_interval: 16 2021-03-15T22:13:25.056 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: index_block_restart_interval: 1 2021-03-15T22:13:25.056 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: metadata_block_size: 4096 2021-03-15T22:13:25.056 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: partition_filters: 0 2021-03-15T22:13:25.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: use_delta_encoding: 1 2021-03-15T22:13:25.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: filter_policy: rocksdb.BuiltinBloomFilter 2021-03-15T22:13:25.057 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: whole_key_filtering: 1 2021-03-15T22:13:25.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: verify_compression: 0 2021-03-15T22:13:25.058 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: read_amp_bytes_per_bit: 0 2021-03-15T22:13:25.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: format_version: 2 2021-03-15T22:13:25.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: enable_index_compression: 1 2021-03-15T22:13:25.059 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: block_align: 0 2021-03-15T22:13:25.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.write_buffer_size: 33554432 2021-03-15T22:13:25.060 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_write_buffer_number: 2 2021-03-15T22:13:25.061 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compression: NoCompression 2021-03-15T22:13:25.061 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.bottommost_compression: Disabled 2021-03-15T22:13:25.061 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.prefix_extractor: nullptr 2021-03-15T22:13:25.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2021-03-15T22:13:25.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.num_levels: 7 2021-03-15T22:13:25.063 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2021-03-15T22:13:25.063 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2021-03-15T22:13:25.063 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2021-03-15T22:13:25.063 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2021-03-15T22:13:25.063 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2021-03-15T22:13:25.064 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2021-03-15T22:13:25.064 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2021-03-15T22:13:25.064 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2021-03-15T22:13:25.064 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2021-03-15T22:13:25.065 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compression_opts.window_bits: -14 2021-03-15T22:13:25.065 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compression_opts.level: 32767 2021-03-15T22:13:25.065 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compression_opts.strategy: 0 2021-03-15T22:13:25.065 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2021-03-15T22:13:25.065 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2021-03-15T22:13:25.066 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compression_opts.enabled: false 2021-03-15T22:13:25.066 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2021-03-15T22:13:25.066 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2021-03-15T22:13:25.066 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2021-03-15T22:13:25.067 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.target_file_size_base: 67108864 2021-03-15T22:13:25.067 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.target_file_size_multiplier: 1 2021-03-15T22:13:25.067 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2021-03-15T22:13:25.067 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2021-03-15T22:13:25.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2021-03-15T22:13:25.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2021-03-15T22:13:25.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2021-03-15T22:13:25.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2021-03-15T22:13:25.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2021-03-15T22:13:25.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2021-03-15T22:13:25.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2021-03-15T22:13:25.070 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2021-03-15T22:13:25.070 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2021-03-15T22:13:25.070 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2021-03-15T22:13:25.070 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.arena_block_size: 4194304 2021-03-15T22:13:25.071 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2021-03-15T22:13:25.071 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2021-03-15T22:13:25.071 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2021-03-15T22:13:25.071 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.disable_auto_compactions: 0 2021-03-15T22:13:25.071 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2021-03-15T22:13:25.072 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2021-03-15T22:13:25.072 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2021-03-15T22:13:25.072 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2021-03-15T22:13:25.072 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2021-03-15T22:13:25.073 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2021-03-15T22:13:25.073 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2021-03-15T22:13:25.073 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2021-03-15T22:13:25.074 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2021-03-15T22:13:25.074 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2021-03-15T22:13:25.074 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.table_properties_collectors: 2021-03-15T22:13:25.074 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.inplace_update_support: 0 2021-03-15T22:13:25.074 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.inplace_update_num_locks: 10000 2021-03-15T22:13:25.075 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2021-03-15T22:13:25.075 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2021-03-15T22:13:25.075 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.memtable_huge_page_size: 0 2021-03-15T22:13:25.075 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.bloom_locality: 0 2021-03-15T22:13:25.076 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.max_successive_merges: 0 2021-03-15T22:13:25.076 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.optimize_filters_for_hits: 0 2021-03-15T22:13:25.076 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.paranoid_file_checks: 0 2021-03-15T22:13:25.076 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.force_consistency_checks: 0 2021-03-15T22:13:25.076 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.report_bg_io_stats: 0 2021-03-15T22:13:25.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.ttl: 2592000 2021-03-15T22:13:25.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.883+0000 7f054645c700 4 rocksdb: Options.periodic_compaction_seconds: 0 2021-03-15T22:13:25.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.884+0000 7f054645c700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2021-03-15T22:13:25.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.078 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.884+0000 7f054645c700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2021-03-15T22:13:25.078 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.078 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.884+0000 7f054645c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615846404884954, "job": 1, "event": "recovery_started", "log_files": [6]} 2021-03-15T22:13:25.079 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.884+0000 7f054645c700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2021-03-15T22:13:25.079 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.884+0000 7f054645c700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2021-03-15T22:13:25.079 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.885+0000 7f054645c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615846404885771, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 83947, "table_properties": {"data_size": 82216, "index_size": 336, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 581, "raw_key_size": 9842, "raw_average_key_size": 47, "raw_value_size": 76642, "raw_average_value_size": 368, "num_data_blocks": 11, "num_entries": 208, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1615846404, "oldest_key_time": 3, "file_creation_time": 0}} 2021-03-15T22:13:25.080 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.885+0000 7f054645c700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2021-03-15T22:13:25.080 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.080 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.885+0000 7f054645c700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1615846404886272, "job": 1, "event": "recovery_finished"} 2021-03-15T22:13:25.080 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 systemd[1]: Started Ceph mon.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:13:25.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.886+0000 7f054645c700 4 rocksdb: DB pointer 0x562d9fb11800 2021-03-15T22:13:25.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.886+0000 7f0535483700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2021-03-15T22:13:25.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.886+0000 7f0535483700 4 rocksdb: [db_impl/db_impl.cc:851] 2021-03-15T22:13:25.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: ** DB Stats ** 2021-03-15T22:13:25.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-15T22:13:25.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2021-03-15T22:13:25.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2021-03-15T22:13:25.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-15T22:13:25.082 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2021-03-15T22:13:25.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2021-03-15T22:13:25.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2021-03-15T22:13:25.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.083 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: ** Compaction Stats [default] ** 2021-03-15T22:13:25.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-15T22:13:25.084 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-15T22:13:25.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: L0 2/0 83.63 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 140.9 0.00 0.00 1 0.001 0 0 2021-03-15T22:13:25.085 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Sum 2/0 83.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 140.9 0.00 0.00 1 0.001 0 0 2021-03-15T22:13:25.086 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 140.9 0.00 0.00 1 0.001 0 0 2021-03-15T22:13:25.086 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.087 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: ** Compaction Stats [default] ** 2021-03-15T22:13:25.087 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-15T22:13:25.087 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-15T22:13:25.088 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 140.9 0.00 0.00 1 0.001 0 0 2021-03-15T22:13:25.088 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-15T22:13:25.088 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-15T22:13:25.088 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-15T22:13:25.089 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: AddFile(Total Files): cumulative 0, interval 0 2021-03-15T22:13:25.089 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-15T22:13:25.090 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: AddFile(Keys): cumulative 0, interval 0 2021-03-15T22:13:25.090 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Cumulative compaction: 0.00 GB write, 27.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-15T22:13:25.090 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Interval compaction: 0.00 GB write, 27.27 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-15T22:13:25.090 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-15T22:13:25.091 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.091 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: ** File Read Latency Histogram By Level [default] ** 2021-03-15T22:13:25.091 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.091 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: ** Compaction Stats [default] ** 2021-03-15T22:13:25.091 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-15T22:13:25.092 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-15T22:13:25.092 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: L0 2/0 83.63 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 140.9 0.00 0.00 1 0.001 0 0 2021-03-15T22:13:25.092 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Sum 2/0 83.63 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 140.9 0.00 0.00 1 0.001 0 0 2021-03-15T22:13:25.092 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2021-03-15T22:13:25.093 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.093 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: ** Compaction Stats [default] ** 2021-03-15T22:13:25.093 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2021-03-15T22:13:25.093 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2021-03-15T22:13:25.093 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 140.9 0.00 0.00 1 0.001 0 0 2021-03-15T22:13:25.094 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Uptime(secs): 0.0 total, 0.0 interval 2021-03-15T22:13:25.094 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Flush(GB): cumulative 0.000, interval 0.000 2021-03-15T22:13:25.094 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: AddFile(GB): cumulative 0.000, interval 0.000 2021-03-15T22:13:25.095 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: AddFile(Total Files): cumulative 0, interval 0 2021-03-15T22:13:25.095 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: AddFile(L0 Files): cumulative 0, interval 0 2021-03-15T22:13:25.095 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: AddFile(Keys): cumulative 0, interval 0 2021-03-15T22:13:25.096 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Cumulative compaction: 0.00 GB write, 26.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-15T22:13:25.096 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2021-03-15T22:13:25.096 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2021-03-15T22:13:25.096 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.097 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: ** File Read Latency Histogram By Level [default] ** 2021-03-15T22:13:25.097 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.097 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.887+0000 7f054645c700 0 starting mon.a rank 0 at public addrs [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] at bind addrs [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:25.097 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.888+0000 7f054645c700 1 mon.a@-1(???) e1 preinit fsid 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:25.097 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.888+0000 7f054645c700 0 mon.a@-1(???).mds e1 new map 2021-03-15T22:13:25.098 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:25.098 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:24.888+0000 7f054645c700 0 mon.a@-1(???).mds e1 print_map 2021-03-15T22:13:25.098 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: e1 2021-03-15T22:13:25.099 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: enable_multiple, ever_enabled_multiple: 1,1 2021-03-15T22:13:25.099 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2021-03-15T22:13:25.099 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: legacy client fscid: -1 2021-03-15T22:13:25.099 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.100 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: No filesystems configured 2021-03-15T22:13:25.100 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.100 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.888+0000 7f054645c700 0 mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2021-03-15T22:13:25.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.888+0000 7f054645c700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-03-15T22:13:25.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.888+0000 7f054645c700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-03-15T22:13:25.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.888+0000 7f054645c700 0 mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2021-03-15T22:13:25.101 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: audit 2021-03-15T22 2021-03-15T22:13:25.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: :13:23.367971+0000 mon.a (mon.0) 10 : audit 2021-03-15T22:13:25.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: [DBG] from='client.? 172.21.15.76:0/2228179409' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:13:25.102 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.888+0000 7f054645c700 1 mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2021-03-15T22:13:25.103 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta expand map: {default=false} 2021-03-15T22:13:25.103 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta from 'false' to 'false' 2021-03-15T22:13:25.103 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta expanded map: {default=false} 2021-03-15T22:13:25.103 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta expand map: {default=info} 2021-03-15T22:13:25.103 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta from 'info' to 'info' 2021-03-15T22:13:25.104 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta expanded map: {default=info} 2021-03-15T22:13:25.104 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta expand map: {default=daemon} 2021-03-15T22:13:25.104 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta from 'daemon' to 'daemon' 2021-03-15T22:13:25.104 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta expanded map: {default=daemon} 2021-03-15T22:13:25.105 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta expand map: {default=debug} 2021-03-15T22:13:25.105 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta from 'debug' to 'debug' 2021-03-15T22:13:25.105 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: debug 2021-03-15T22:13:24.889+0000 7f054645c700 20 expand_channel_meta expanded map: {default=debug} 2021-03-15T22:13:25.105 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 2021-03-15T22:13:25.106 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: cluster 2021-03-15T22:13:24.904607+0000 mon.a (mon.0) 1 : cluster [INF] mon.a is new leader, mons a in quorum (ranks 0) 2021-03-15T22:13:25.106 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: cluster 2021-03-15T22:13:24.904757+0000 mon.a (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0]} 2021-03-15T22:13:25.106 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: cluster 2021-03-15T22:13:24.904824+0000 mon.a (mon.0) 3 : cluster [DBG] fsmap 2021-03-15T22:13:25.106 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: cluster 2021-03-15T22:13:24.904872+0000 mon.a (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2021-03-15T22:13:25.107 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:13:25.107 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:24 smithi076 conmon[56514]: 15T22:13:24.905437+0000 mon.a (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2021-03-15T22:13:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:Wrote config to /etc/ceph/ceph.conf 2021-03-15T22:13:26.163 INFO:teuthology.orchestra.run.smithi076.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2021-03-15T22:13:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:Creating mgr... 2021-03-15T22:13:26.164 INFO:teuthology.orchestra.run.smithi076.stderr:Verifying port 9283 ... 2021-03-15T22:13:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl daemon-reload 2021-03-15T22:13:26.250 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl stop ceph-8a572802-85db-11eb-9091-001a4aab830c@mgr.y 2021-03-15T22:13:26.276 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl reset-failed ceph-8a572802-85db-11eb-9091-001a4aab830c@mgr.y 2021-03-15T22:13:26.282 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: Failed to reset failed state of unit ceph-8a572802-85db-11eb-9091-001a4aab830c@mgr.y.service: Unit ceph-8a572802-85db-11eb-9091-001a4aab830c@mgr.y.service not loaded. 2021-03-15T22:13:26.282 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl enable ceph-8a572802-85db-11eb-9091-001a4aab830c@mgr.y 2021-03-15T22:13:26.289 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: Created symlink /etc/systemd/system/ceph-8a572802-85db-11eb-9091-001a4aab830c.target.wants/ceph-8a572802-85db-11eb-9091-001a4aab830c@mgr.y.service → /etc/systemd/system/ceph-8a572802-85db-11eb-9091-001a4aab830c@.service. 2021-03-15T22:13:26.362 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl start ceph-8a572802-85db-11eb-9091-001a4aab830c@mgr.y 2021-03-15T22:13:26.487 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:26 smithi076 systemd[1]: Starting Ceph mgr.y for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:13:26.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:26 smithi076 conmon[56514]: audit 2021-03-15T22:13:25.486579+0000 mon.a (mon.0) 6 : audit [INF] from='client.? 172.21.15.76:0/3980769354' entity='client.admin' 2021-03-15T22:13:26.767 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:26 smithi076 podman[56803]: 2021-03-15 22:13:26.63769533 +0000 UTC m=+0.140943283 container create 200982e31074e786605f9f74f50e01d4a2934c38423a9e418b77317a280982dd (image=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mgr.y, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image) 2021-03-15T22:13:26.913 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-enabled firewalld.service 2021-03-15T22:13:26.922 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: disabled 2021-03-15T22:13:26.923 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-active firewalld.service 2021-03-15T22:13:26.930 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: inactive 2021-03-15T22:13:26.931 INFO:teuthology.orchestra.run.smithi076.stderr:firewalld.service is not enabled 2021-03-15T22:13:26.932 INFO:teuthology.orchestra.run.smithi076.stderr:Not possible to enable service . firewalld.service is not available 2021-03-15T22:13:26.933 INFO:teuthology.orchestra.run.smithi076.stderr:Not possible to open ports <[]>. firewalld.service is not available 2021-03-15T22:13:26.934 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-enabled firewalld.service 2021-03-15T22:13:26.938 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: disabled 2021-03-15T22:13:26.939 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-active firewalld.service 2021-03-15T22:13:26.946 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: inactive 2021-03-15T22:13:26.947 INFO:teuthology.orchestra.run.smithi076.stderr:firewalld.service is not enabled 2021-03-15T22:13:26.947 INFO:teuthology.orchestra.run.smithi076.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2021-03-15T22:13:26.947 INFO:teuthology.orchestra.run.smithi076.stderr:Waiting for mgr to start... 2021-03-15T22:13:26.948 INFO:teuthology.orchestra.run.smithi076.stderr:Waiting for mgr... 2021-03-15T22:13:26.948 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 status -f json-pretty 2021-03-15T22:13:27.148 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:26 smithi076 podman[56803]: 2021-03-15 22:13:26.871079472 +0000 UTC m=+0.374327524 container init 200982e31074e786605f9f74f50e01d4a2934c38423a9e418b77317a280982dd (image=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mgr.y, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, org.label-schema.name=CentOS Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, maintainer=Dimitri Savineau ) 2021-03-15T22:13:27.149 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:26 smithi076 podman[56803]: 2021-03-15 22:13:26.904264892 +0000 UTC m=+0.407512836 container start 200982e31074e786605f9f74f50e01d4a2934c38423a9e418b77317a280982dd (image=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mgr.y, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, org.label-schema.build-date=20201204, io.buildah.version=1.18.0, RELEASE=wip-sage-testing-2021-03-15-1507-b658018) 2021-03-15T22:13:27.149 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:26 smithi076 bash[56726]: 200982e31074e786605f9f74f50e01d4a2934c38423a9e418b77317a280982dd 2021-03-15T22:13:27.149 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:26 smithi076 systemd[1]: Started Ceph mgr.y for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:13:27.539 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 2021-03-15T22:13:27.540 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: { 2021-03-15T22:13:27.540 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:13:27.540 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "health": { 2021-03-15T22:13:27.541 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-03-15T22:13:27.541 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "checks": {}, 2021-03-15T22:13:27.541 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "mutes": [] 2021-03-15T22:13:27.541 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:27.542 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-03-15T22:13:27.542 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "quorum": [ 2021-03-15T22:13:27.542 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 0 2021-03-15T22:13:27.542 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: ], 2021-03-15T22:13:27.543 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "quorum_names": [ 2021-03-15T22:13:27.543 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "a" 2021-03-15T22:13:27.543 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: ], 2021-03-15T22:13:27.543 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "quorum_age": 2, 2021-03-15T22:13:27.543 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "monmap": { 2021-03-15T22:13:27.544 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:27.544 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-03-15T22:13:27.544 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_mons": 1 2021-03-15T22:13:27.544 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:27.544 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "osdmap": { 2021-03-15T22:13:27.545 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:27.545 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_osds": 0, 2021-03-15T22:13:27.545 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-03-15T22:13:27.545 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-03-15T22:13:27.546 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-03-15T22:13:27.546 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-03-15T22:13:27.546 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-03-15T22:13:27.547 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:27.547 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "pgmap": { 2021-03-15T22:13:27.547 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-03-15T22:13:27.548 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-03-15T22:13:27.548 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_pools": 0, 2021-03-15T22:13:27.548 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_objects": 0, 2021-03-15T22:13:27.548 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-03-15T22:13:27.549 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-03-15T22:13:27.549 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-03-15T22:13:27.549 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "bytes_total": 0 2021-03-15T22:13:27.550 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:27.550 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "fsmap": { 2021-03-15T22:13:27.550 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:27.550 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "by_rank": [], 2021-03-15T22:13:27.551 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "up:standby": 0 2021-03-15T22:13:27.551 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:27.551 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "mgrmap": { 2021-03-15T22:13:27.552 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "available": false, 2021-03-15T22:13:27.552 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-03-15T22:13:27.552 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "modules": [ 2021-03-15T22:13:27.552 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "iostat", 2021-03-15T22:13:27.553 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "restful" 2021-03-15T22:13:27.553 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: ], 2021-03-15T22:13:27.553 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "services": {} 2021-03-15T22:13:27.553 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:27.554 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "servicemap": { 2021-03-15T22:13:27.554 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:27.554 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "modified": "2021-03-15T22:13:20.371389+0000", 2021-03-15T22:13:27.554 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "services": {} 2021-03-15T22:13:27.554 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:27.555 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "progress_events": {} 2021-03-15T22:13:27.555 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: } 2021-03-15T22:13:28.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:27 smithi076 conmon[56514]: audit 2021-03-15T22:13:27.530958+0000 mon.a (mon.0) 7 : audit [DBG] from='client.? 172.21.15.76:0/3539361220' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-03-15T22:13:28.133 INFO:teuthology.orchestra.run.smithi076.stderr:mgr not available, waiting (1/15)... 2021-03-15T22:13:30.135 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 status -f json-pretty 2021-03-15T22:13:30.738 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 2021-03-15T22:13:30.738 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: { 2021-03-15T22:13:30.739 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:13:30.739 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "health": { 2021-03-15T22:13:30.739 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-03-15T22:13:30.739 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "checks": {}, 2021-03-15T22:13:30.739 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "mutes": [] 2021-03-15T22:13:30.740 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:30.740 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-03-15T22:13:30.741 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "quorum": [ 2021-03-15T22:13:30.741 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 0 2021-03-15T22:13:30.741 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: ], 2021-03-15T22:13:30.742 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "quorum_names": [ 2021-03-15T22:13:30.742 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "a" 2021-03-15T22:13:30.743 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: ], 2021-03-15T22:13:30.743 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "quorum_age": 5, 2021-03-15T22:13:30.743 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "monmap": { 2021-03-15T22:13:30.743 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:30.744 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-03-15T22:13:30.744 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_mons": 1 2021-03-15T22:13:30.744 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:30.745 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "osdmap": { 2021-03-15T22:13:30.745 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:30.745 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_osds": 0, 2021-03-15T22:13:30.745 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-03-15T22:13:30.746 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-03-15T22:13:30.746 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-03-15T22:13:30.746 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-03-15T22:13:30.746 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-03-15T22:13:30.748 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:30.749 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "pgmap": { 2021-03-15T22:13:30.749 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-03-15T22:13:30.749 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-03-15T22:13:30.750 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_pools": 0, 2021-03-15T22:13:30.750 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_objects": 0, 2021-03-15T22:13:30.750 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-03-15T22:13:30.750 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-03-15T22:13:30.750 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-03-15T22:13:30.751 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "bytes_total": 0 2021-03-15T22:13:30.751 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:30.751 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "fsmap": { 2021-03-15T22:13:30.751 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:30.752 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "by_rank": [], 2021-03-15T22:13:30.752 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "up:standby": 0 2021-03-15T22:13:30.752 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:30.752 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "mgrmap": { 2021-03-15T22:13:30.752 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "available": false, 2021-03-15T22:13:30.753 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-03-15T22:13:30.753 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "modules": [ 2021-03-15T22:13:30.753 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "iostat", 2021-03-15T22:13:30.753 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "restful" 2021-03-15T22:13:30.753 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: ], 2021-03-15T22:13:30.754 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "services": {} 2021-03-15T22:13:30.754 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:30.754 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "servicemap": { 2021-03-15T22:13:30.754 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:30.755 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "modified": "2021-03-15T22:13:20.371389+0000", 2021-03-15T22:13:30.755 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "services": {} 2021-03-15T22:13:30.755 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:30.755 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "progress_events": {} 2021-03-15T22:13:30.755 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: } 2021-03-15T22:13:31.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:30 smithi076 conmon[56514]: audit 2021-03-15T22:13:30.727199+0000 mon.a (mon.0) 8 2021-03-15T22:13:31.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:30 smithi076 conmon[56514]: : audit [DBG] from='client.? 172.21.15.76:0/435718193' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-03-15T22:13:31.332 INFO:teuthology.orchestra.run.smithi076.stderr:mgr not available, waiting (2/15)... 2021-03-15T22:13:32.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: cluster 2021-03-15T22:13:31.775521+0000 mon.a (mon.0) 9 : cluster [INF] Activating manager daemon y 2021-03-15T22:13:32.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: cluster 2021-03-15T22:13:31.778207+0000 2021-03-15T22:13:32.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: mon.a (mon.0) 10 : cluster [DBG] mgrmap e2: y(active, starting, since 0.00276434s) 2021-03-15T22:13:32.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: audit 2021-03-15T22:13:31.778879+0000 mon.a (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.76:0/425698741' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-03-15T22:13:32.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:13:32.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: -15T22:13:31.779084+0000 mon.a (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.76:0/425698741' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-03-15T22:13:32.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:13:32.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: -15T22:13:31.779259+0000 mon.a 2021-03-15T22:13:32.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.76:0/425698741' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-03-15T22:13:32.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: 2021-03-15T22:13:32.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: audit 2021-03-15T22:13:32.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: 2021-03-15T22:13:31.779418+0000 mon.a ( 2021-03-15T22:13:32.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.76:0/425698741' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-15T22:13:32.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:13:32.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: -15T22:13:31.779594+0000 2021-03-15T22:13:32.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: mon.a (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.76:0/425698741' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-03-15T22:13:32.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: 2021-03-15T22:13:32.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:13:32.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: 15T22:13:31.781011+0000 mon.a (mon.0) 2021-03-15T22:13:32.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: 16 : cluster [INF] Manager daemon y is now available 2021-03-15T22:13:32.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: audit 2021-03-15 2021-03-15T22:13:32.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: T22:13:31.788420+0000 mon.a (mon 2021-03-15T22:13:32.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: .0) 17 : audit [INF] from='mgr.14100 172.21.15.76:0/425698741' entity='mgr.y' 2021-03-15T22:13:32.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:13:32.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: 03-15T22:13:31.792493+0000 2021-03-15T22:13:32.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: mon.a (mon.0) 18 : audit [INF] 2021-03-15T22:13:32.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: from='mgr.14100 172.21.15.76:0/425698741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:13:32.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: audit 2021-03-15T22:13:31 2021-03-15T22:13:32.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: .793544+0000 mon.a (mon.0) 19 : 2021-03-15T22:13:32.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: audit [INF] from='mgr.14100 172.21.15.76:0/425698741' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:13:32.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: audit 2021-03-15T22 2021-03-15T22:13:32.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: :13:31.798090+0000 mon.a (mon. 2021-03-15T22:13:32.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: 0) 20 : audit [INF] from='mgr.14100 172.21.15.76:0/425698741' entity='mgr.y' 2021-03-15T22:13:32.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:13:32.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: -15T22:13:31.799863+0000 mon.a (mon.0) 21 : audit 2021-03-15T22:13:32.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:31 smithi076 conmon[56514]: [INF] from='mgr.14100 172.21.15.76:0/425698741' entity='mgr.y' 2021-03-15T22:13:33.335 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 status -f json-pretty 2021-03-15T22:13:34.068 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 2021-03-15T22:13:34.068 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: { 2021-03-15T22:13:34.069 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:13:34.069 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "health": { 2021-03-15T22:13:34.069 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2021-03-15T22:13:34.069 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "checks": {}, 2021-03-15T22:13:34.070 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "mutes": [] 2021-03-15T22:13:34.070 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:34.070 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "election_epoch": 5, 2021-03-15T22:13:34.070 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "quorum": [ 2021-03-15T22:13:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 0 2021-03-15T22:13:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: ], 2021-03-15T22:13:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "quorum_names": [ 2021-03-15T22:13:34.071 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "a" 2021-03-15T22:13:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: ], 2021-03-15T22:13:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "quorum_age": 9, 2021-03-15T22:13:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "monmap": { 2021-03-15T22:13:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "min_mon_release_name": "quincy", 2021-03-15T22:13:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_mons": 1 2021-03-15T22:13:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "osdmap": { 2021-03-15T22:13:34.073 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_osds": 0, 2021-03-15T22:13:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_up_osds": 0, 2021-03-15T22:13:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "osd_up_since": 0, 2021-03-15T22:13:34.074 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_in_osds": 0, 2021-03-15T22:13:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "osd_in_since": 0, 2021-03-15T22:13:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2021-03-15T22:13:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "pgmap": { 2021-03-15T22:13:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "pgs_by_state": [], 2021-03-15T22:13:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_pgs": 0, 2021-03-15T22:13:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_pools": 0, 2021-03-15T22:13:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_objects": 0, 2021-03-15T22:13:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "data_bytes": 0, 2021-03-15T22:13:34.076 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "bytes_used": 0, 2021-03-15T22:13:34.077 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "bytes_avail": 0, 2021-03-15T22:13:34.077 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "bytes_total": 0 2021-03-15T22:13:34.077 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:34.077 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "fsmap": { 2021-03-15T22:13:34.078 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:34.078 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "by_rank": [], 2021-03-15T22:13:34.078 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "up:standby": 0 2021-03-15T22:13:34.078 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:34.078 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "mgrmap": { 2021-03-15T22:13:34.079 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "available": true, 2021-03-15T22:13:34.079 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_standbys": 0, 2021-03-15T22:13:34.079 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "modules": [ 2021-03-15T22:13:34.079 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "iostat", 2021-03-15T22:13:34.079 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "restful" 2021-03-15T22:13:34.080 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: ], 2021-03-15T22:13:34.080 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "services": {} 2021-03-15T22:13:34.080 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:34.080 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "servicemap": { 2021-03-15T22:13:34.081 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 1, 2021-03-15T22:13:34.081 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "modified": "2021-03-15T22:13:20.371389+0000", 2021-03-15T22:13:34.081 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "services": {} 2021-03-15T22:13:34.081 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: }, 2021-03-15T22:13:34.081 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "progress_events": {} 2021-03-15T22:13:34.082 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: } 2021-03-15T22:13:34.124 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:33 smithi076 conmon[56514]: cluster 2021-03-15T22:13:32.781091+0000 mon.a 2021-03-15T22:13:34.125 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:33 smithi076 conmon[56514]: (mon.0) 22 : cluster [DBG] mgrmap e3: y(active, since 1.00565s) 2021-03-15T22:13:34.582 INFO:teuthology.orchestra.run.smithi076.stderr:mgr is available 2021-03-15T22:13:34.583 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 config-key set mgr/cephadm/ssh_user root 2021-03-15T22:13:34.939 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:34 smithi076 conmon[56514]: audit 2021-03-15T22:13:34.060736+0000 mon.a ( 2021-03-15T22:13:34.940 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:34 smithi076 conmon[56514]: mon.0) 23 : audit [DBG] from='client.? 172.21.15.76:0/2905876961' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2021-03-15T22:13:35.303 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: set mgr/cephadm/ssh_user 2021-03-15T22:13:35.798 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling cephadm module... 2021-03-15T22:13:35.798 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 mgr module enable cephadm 2021-03-15T22:13:36.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:35 smithi076 conmon[56514]: cluster 2021-03-15T22:13:34.780461+0000 mon.a (mon.0) 24 2021-03-15T22:13:36.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:35 smithi076 conmon[56514]: : cluster [DBG] mgrmap e4: y(active, since 3s) 2021-03-15T22:13:36.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:35 smithi076 conmon[56514]: audit 2021-03-15T22:13:35.299385+0000 mon.a (mon.0) 25 : audit [INF] from='client.? 172.21.15.76:0/1716013327' entity='client.admin' 2021-03-15T22:13:37.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:36 smithi076 conmon[56514]: audit 2021-03-15T22:13:36.494958+0000 mon.a (mon.0) 26 2021-03-15T22:13:37.237 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:36 smithi076 conmon[56514]: : audit [INF] from='client.? 172.21.15.76:0/4125498545' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2021-03-15T22:13:37.238 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:36 smithi076 conmon[56844]: ignoring --setuser ceph since I am not root 2021-03-15T22:13:37.238 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:36 smithi076 conmon[56844]: ignoring --setgroup ceph since I am not root 2021-03-15T22:13:37.239 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 mgr stat 2021-03-15T22:13:37.924 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: { 2021-03-15T22:13:37.924 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 5, 2021-03-15T22:13:37.924 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "available": true, 2021-03-15T22:13:37.924 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "active_name": "y", 2021-03-15T22:13:37.925 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_standby": 0 2021-03-15T22:13:37.925 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: } 2021-03-15T22:13:37.969 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:37 smithi076 conmon[56514]: audit 2021-03-15T22:13:36.785775 2021-03-15T22:13:37.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:37 smithi076 conmon[56514]: +0000 mon.a (mon.0) 27 : audit [INF] from='client.? 172.21.15.76:0/4125498545' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2021-03-15T22:13:37.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:37 smithi076 conmon[56514]: cluster 2021-03-15T22:13:36.785827+0000 mon.a (mon.0) 28 : cluster [DBG] mgrmap e5: y(active, since 5s) 2021-03-15T22:13:38.384 INFO:teuthology.orchestra.run.smithi076.stderr:Waiting for the mgr to restart... 2021-03-15T22:13:38.384 INFO:teuthology.orchestra.run.smithi076.stderr:Waiting for mgr epoch 5... 2021-03-15T22:13:38.385 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 tell mgr mgr_status 2021-03-15T22:13:39.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:38 smithi076 conmon[56514]: audit 2021-03-15T22:13:37.921555+0000 mon.a (mon.0) 29 : audit [DBG] from='client.? 172.21.15.76:0/3056325517' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-03-15T22:13:42.672 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: { 2021-03-15T22:13:42.672 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2021-03-15T22:13:42.672 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "initialized": true 2021-03-15T22:13:42.673 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: } 2021-03-15T22:13:43.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: cluster 2021-03-15T22:13:41.603627+0000 mon.a (mon.0) 30 : cluster [INF] Active manager daemon y restarted 2021-03-15T22:13:43.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: cluster 2021-03-15T22:13:41.604109 2021-03-15T22:13:43.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: +0000 mon.a (mon.0) 31 : cluster [INF] Activating manager daemon y 2021-03-15T22:13:43.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: cluster 2021-03-15T22:13:41.607923+0000 mon.a (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2021-03-15T22:13:43.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: cluster 2021-03-15T22:13:41.658695+0000 mon.a (mon.0) 33 2021-03-15T22:13:43.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: : cluster [DBG] mgrmap e6: y(active, starting, since 0.054688s) 2021-03-15T22:13:43.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021-03-15T 2021-03-15T22:13:43.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 22:13:41.659537+0000 mon.a (mon.0) 34 : audit 2021-03-15T22:13:43.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: [DBG] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-15T22:13:43.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021-03-15T22: 2021-03-15T22:13:43.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 13:41.660547+0000 mon.a (mon.0) 2021-03-15T22:13:43.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 35 : audit [DBG] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-03-15T22:13:43.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:13:43.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 15T22:13:41.661179+0000 mon.a ( 2021-03-15T22:13:43.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-03-15T22:13:43.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:13:43.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 03-15T22:13:41.661349+0000 2021-03-15T22:13:43.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: mon.a (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-03-15T22:13:43.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 2021-03-15T22:13:43.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021-03-15T22:13: 2021-03-15T22:13:43.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 41.661494+0000 mon.a (mon.0) 38 : 2021-03-15T22:13:43.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit [DBG] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-03-15T22:13:43.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:13:43.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: T22:13:41.663234+0000 mon.a (mon 2021-03-15T22:13:43.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: .0) 39 : cluster [INF] Manager daemon y is now available 2021-03-15T22:13:43.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021 2021-03-15T22:13:43.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: -03-15T22:13:41.676694+0000 2021-03-15T22:13:43.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: mon.a (mon.0) 40 : audit [INF] 2021-03-15T22:13:43.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:43.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021-03-15T22: 2021-03-15T22:13:43.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 13:41.682020+0000 mon.a (mon.0) 2021-03-15T22:13:43.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 41 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:43.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:13:43.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 15T22:13:41.687546+0000 mon.a ( 2021-03-15T22:13:43.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:43.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021-03-15T22:13:43.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 2021-03-15T22:13:41.778277+0000 mon.a 2021-03-15T22:13:43.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:43.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021 2021-03-15T22:13:43.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: -03-15T22:13:41.778895 2021-03-15T22:13:43.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: +0000 mon.a (mon.0) 44 : audit 2021-03-15T22:13:43.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: [DBG] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:13:43.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021-03-15T 2021-03-15T22:13:43.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 22:13:41.779875+0000 mon.a (mon 2021-03-15T22:13:43.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: .0) 45 : audit [DBG] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:13:43.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:13:43.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 03-15T22:13:41.781020+0000 2021-03-15T22:13:43.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: mon.a (mon.0) 46 : audit [DBG] 2021-03-15T22:13:43.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:13:43.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021-03-15T22 2021-03-15T22:13:43.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: :13:41.791190+0000 mon.a (mon. 2021-03-15T22:13:43.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 0) 47 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:13:43.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:13:43.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: 03-15T22:13:41.792396+0000 2021-03-15T22:13:43.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: mon.a (mon.0) 48 : audit [INF] 2021-03-15T22:13:43.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:42 smithi076 conmon[56514]: from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:13:43.181 INFO:teuthology.orchestra.run.smithi076.stderr:mgr epoch 5 is available 2021-03-15T22:13:43.182 INFO:teuthology.orchestra.run.smithi076.stderr:Setting orchestrator backend to cephadm... 2021-03-15T22:13:43.182 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 orch set backend cephadm 2021-03-15T22:13:43.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:43 smithi076 conmon[56514]: audit 2021-03-15T22:13:42.663585+0000 mgr.y (mgr.14116) 1 2021-03-15T22:13:43.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:43 smithi076 conmon[56514]: : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-03-15T22:13:43.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:43 smithi076 conmon[56514]: cluster 2021-03-15T22:13:42.663698+0000 mon.a (mon.0) 49 : cluster [DBG] mgrmap e7: y(active, since 1.05968s) 2021-03-15T22:13:43.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:43 smithi076 conmon[56514]: audit 2021-03-15T22:13:42.670204+0000 mgr.y (mgr.14116) 2 2021-03-15T22:13:43.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:43 smithi076 conmon[56514]: : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-03-15T22:13:44.473 INFO:teuthology.orchestra.run.smithi076.stderr:Generating ssh key... 2021-03-15T22:13:44.473 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 cephadm generate-key 2021-03-15T22:13:44.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:44 smithi076 conmon[56514]: cluster 2021-03-15T22:13:43.663795+0000 2021-03-15T22:13:44.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:44 smithi076 conmon[56514]: mon.a (mon.0) 50 : cluster [DBG] mgrmap e8: y(active, since 2s) 2021-03-15T22:13:44.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:44 smithi076 conmon[56514]: audit 2021-03-15T22:13:43.902906+0000 mon.a (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:44.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:44 smithi076 conmon[56514]: audit 2021-03-15T22:13:43.906318+0000 mon.a (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:13:45.669 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: Generating public/private rsa key pair. 2021-03-15T22:13:45.669 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: Your identification has been saved in /tmp/tmppj7m1l7r/key. 2021-03-15T22:13:45.669 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: Your public key has been saved in /tmp/tmppj7m1l7r/key.pub. 2021-03-15T22:13:45.669 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: The key fingerprint is: 2021-03-15T22:13:45.670 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: SHA256:yu0RAO7ZzFSUJRLXCD4WJTUwT7JITgYsCcrInJ3jSrE ceph-8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:45.670 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: The key's randomart image is: 2021-03-15T22:13:45.670 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: +---[RSA 3072]----+ 2021-03-15T22:13:45.670 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: |oo..= XBX=. | 2021-03-15T22:13:45.671 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: |O +*.+ %oo. | 2021-03-15T22:13:45.671 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: |oB ++ B . | 2021-03-15T22:13:45.671 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: | +..B o | 2021-03-15T22:13:45.671 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: | E .o + S | 2021-03-15T22:13:45.672 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: |. . . o . | 2021-03-15T22:13:45.672 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: | . o o | 2021-03-15T22:13:45.672 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: | . . | 2021-03-15T22:13:45.672 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: | . | 2021-03-15T22:13:45.672 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56844]: +----[SHA256]-----+ 2021-03-15T22:13:45.814 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 cephadm get-pub-key 2021-03-15T22:13:46.011 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56514]: audit 2021-03-15T22:13:43.896297+0000 mgr.y (mgr.14116) 3 : 2021-03-15T22:13:46.011 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56514]: audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:13:46.011 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56514]: audit 2021-03-15T22:13:45.289598+0000 mon.a (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:46.011 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:45 smithi076 conmon[56514]: audit 2021-03-15T22:13:45.291301+0000 mon.a (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:46.502 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvhWU99/nHSuIiP/iCJP3EPyx/w3i0CoSoWCzlnYoW7W3iGBYDOgnKMkggrFpOvZGKlzL44n4ZG5fLVpoOWj9GOGjRIZuTJzV7llJhmCfnDHCcedsMiGTcz76gnXxeXfUG1NHxQOztcVOCPM7HaFBT2J9hue9zjThl6YbtrrkXHbeeVDfdzZkQ7B2Aft4sLPK727R2kNA8PlK4kCJEVHPgifHIraJe9Dp1L5bCKSj2pNt8aqQPt8jz1eyaliFmsg4jyrtG4zW5nOyggmHjjM1LZzUNb2wLcC2DEwwizZV3EBOTa1OTgb/GtR/CeinXobKlgdYiPqfv9ZBOLG7eoGxbfsyIWsXaLnjsFjiZTyj4nfGUEtjqNOkFYF2GAvnqFCpL1ZO5oNPrpjfk4XCpeyvOyczA130fKXVBQdTw5ABIJyz5ynyk1OzHuMr4Wq6g5njzmODhE5WOrxxGlC6WfelaunGI5yMti15hUyrI0q4/LCLKXGN+fOi0THfwec4K6Mc= ceph-8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:13:47.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:13:45.128779+0000 mgr.y (mgr.14116) 5 : 2021-03-15T22:13:47.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:46 smithi076 conmon[56514]: cephadm [INF] Generating ssh key... 2021-03-15T22:13:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:Wrote public SSH key to to /home/ubuntu/cephtest/ceph.pub 2021-03-15T22:13:47.033 INFO:teuthology.orchestra.run.smithi076.stderr:Adding key to root@localhost authorized_keys... 2021-03-15T22:13:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:Adding host smithi076... 2021-03-15T22:13:47.035 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 orch host add smithi076 2021-03-15T22:13:48.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:47 smithi076 conmon[56514]: audit 2021-03-15T22:13:46.500793+0000 mgr.y (mgr.14116) 6 2021-03-15T22:13:48.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:47 smithi076 conmon[56514]: : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:13:48.017 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:47 smithi076 conmon[56844]: [76B blob data] 2021-03-15T22:13:48.346 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: Added host 'smithi076' 2021-03-15T22:13:49.002 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:48 smithi076 conmon[56514]: audit 2021-03-15T22:13:47.690982+0000 mgr.y (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi076", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:13:49.002 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:48 smithi076 conmon[56514]: audit 2021-03-15T22:13:48.343425+0000 mon.a (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:49.003 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:48 smithi076 conmon[56514]: audit 2021-03-15T22:13:48.344001+0000 mon.a (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:13:49.005 INFO:teuthology.orchestra.run.smithi076.stderr:Deploying unmanaged mon service... 2021-03-15T22:13:49.005 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 orch apply mon --unmanaged 2021-03-15T22:13:49.817 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: Scheduled mon update... 2021-03-15T22:13:49.879 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:13:48.343794+0000 mgr.y (mgr.14116) 8 : cephadm [INF] Added host smithi076 2021-03-15T22:13:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:Deploying unmanaged mgr service... 2021-03-15T22:13:50.447 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 orch apply mgr --unmanaged 2021-03-15T22:13:51.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:50 smithi076 conmon[56514]: audit 2021-03-15T22:13:49.812057+0000 mgr.y (mgr 2021-03-15T22:13:51.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:50 smithi076 conmon[56514]: .14116) 9 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:13:51.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:50 smithi076 conmon[56514]: cephadm 2021-03-15T22:13:49.812806+0000 mgr.y (mgr.14116) 10 : cephadm [INF] Saving service mon spec with placement count:5 2021-03-15T22:13:51.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:50 smithi076 conmon[56514]: audit 2021-03-15T22:13:49.815839+0000 mon.a (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:51.217 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: Scheduled mgr update... 2021-03-15T22:13:51.729 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 config set mgr mgr/cephadm/container_init True --force 2021-03-15T22:13:52.168 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:52 smithi076 conmon[56514]: audit 2021-03-15T22:13:51.142098+0000 mgr.y (mgr. 2021-03-15T22:13:52.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:52 smithi076 conmon[56514]: 14116) 11 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:13:52.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:52 smithi076 conmon[56514]: cephadm 2021-03-15T22:13:51.142628+0000 mgr.y (mgr.14116) 12 : cephadm [INF] Saving service mgr spec with placement count:2 2021-03-15T22:13:52.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:52 smithi076 conmon[56514]: audit 2021-03-15T22:13:51.145782+0000 mon.a (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:52.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:52 smithi076 conmon[56514]: audit 2021-03-15T22:13:51.615629+0000 mon.a (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:52.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:52 smithi076 conmon[56514]: audit 2021-03-15T22:13:51.874310+0000 mon.a (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:53.096 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 config set mgr mgr/dashboard/ssl_server_port 8443 2021-03-15T22:13:53.702 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:53 smithi076 conmon[56514]: audit 2021-03-15T22:13:52.443454+0000 mon.a (mon.0) 61 : audit [INF] from='client.? 172.21.15.76:0/2890833256' entity='client.admin' 2021-03-15T22:13:53.703 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:53 smithi076 conmon[56514]: audit 2021-03-15T22:13:53.089130+0000 mon.a (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.76:0/3638782581' entity='mgr.y' 2021-03-15T22:13:54.596 INFO:teuthology.orchestra.run.smithi076.stderr:Enabling the dashboard module... 2021-03-15T22:13:54.597 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 mgr module enable dashboard 2021-03-15T22:13:55.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:54 smithi076 conmon[56514]: audit 2021-03-15T22:13:53. 2021-03-15T22:13:55.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:54 smithi076 conmon[56514]: 817528+0000 mon.a (mon.0) 63 : audit [INF] from='client.? 172.21.15.76:0/2704347321' entity='client.admin' 2021-03-15T22:13:56.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:55 smithi076 conmon[56514]: audit 2021-03-15T22:13:55.289755+0000 mon.a 2021-03-15T22:13:56.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:55 smithi076 conmon[56514]: (mon.0) 64 : audit [INF] from='client.? 172.21.15.76:0/3015520022' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2021-03-15T22:13:56.267 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:55 smithi076 conmon[56844]: ignoring --setuser ceph since I am not root 2021-03-15T22:13:56.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:13:55 smithi076 conmon[56844]: ignoring --setgroup ceph since I am not root 2021-03-15T22:13:56.420 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 mgr stat 2021-03-15T22:13:57.180 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: { 2021-03-15T22:13:57.180 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "epoch": 9, 2021-03-15T22:13:57.180 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "available": true, 2021-03-15T22:13:57.181 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "active_name": "y", 2021-03-15T22:13:57.181 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "num_standby": 0 2021-03-15T22:13:57.181 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: } 2021-03-15T22:13:57.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:56 smithi076 conmon[56514]: audit 2021-03-15T22:13:55.901321+0000 2021-03-15T22:13:57.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:56 smithi076 conmon[56514]: mon.a (mon.0) 65 : audit [INF] from='client.? 172.21.15.76:0/3015520022' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2021-03-15T22:13:57.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:56 smithi076 conmon[56514]: cluster 2021-03-15T22:13:55.901381+0000 mon.a (mon.0) 66 : cluster [DBG] mgrmap e9: y(active, since 14s) 2021-03-15T22:13:57.704 INFO:teuthology.orchestra.run.smithi076.stderr:Waiting for the mgr to restart... 2021-03-15T22:13:57.704 INFO:teuthology.orchestra.run.smithi076.stderr:Waiting for mgr epoch 9... 2021-03-15T22:13:57.704 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 tell mgr mgr_status 2021-03-15T22:13:58.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:57 smithi076 conmon[56514]: audit 2021-03-15T22:13:57.177024+0000 mon.a ( 2021-03-15T22:13:58.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:13:57 smithi076 conmon[56514]: mon.0) 67 : audit [DBG] from='client.? 172.21.15.76:0/1120663278' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2021-03-15T22:14:01.149 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:00 smithi076 conmon[56514]: cluster 2021-03-15T22:14:00.771676+0000 mon.a (mon.0) 68 2021-03-15T22:14:01.150 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:00 smithi076 conmon[56514]: : cluster [INF] Active manager daemon y restarted 2021-03-15T22:14:01.150 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:00 smithi076 conmon[56514]: cluster 2021-03-15T22:14:00.772066+0000 mon.a (mon.0) 69 : cluster [INF] Activating manager daemon y 2021-03-15T22:14:01.150 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:00 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:14:01.151 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:00 smithi076 conmon[56514]: T22:14:00.776156+0000 mon.a 2021-03-15T22:14:01.151 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:00 smithi076 conmon[56514]: (mon.0) 70 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2021-03-15T22:14:01.151 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56844]: [76B blob data] 2021-03-15T22:14:01.911 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: { 2021-03-15T22:14:01.911 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2021-03-15T22:14:01.912 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: "initialized": true 2021-03-15T22:14:01.912 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: } 2021-03-15T22:14:02.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: cluster 2021-03-15T22:14:00.901154+0000 mon.a (mon.0) 71 : cluster [DBG] mgrmap e10: y(active, starting, since 0.129173s) 2021-03-15T22:14:02.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: audit 2021-03-15T22:14:00.902216+0000 mon.a (mon.0) 72 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-15T22:14:02.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: audit 2021-03-15T22:14:00.902772+0000 mon.a (mon.0) 73 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2021-03-15T22:14:02.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: audit 2021-03-15T22:14:00.903391+0000 mon.a (mon.0) 74 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2021-03-15T22:14:02.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: audit 2021-03-15T22:14:00.903554+0000 mon.a (mon.0) 75 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2021-03-15T22:14:02.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: audit 2021-03-15T22:14:00.903702+0000 mon.a (mon.0) 76 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2021-03-15T22:14:02.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: cluster 2021-03-15T22:14:00.905473+0000 mon.a (mon.0) 77 : cluster [INF] Manager daemon y is now available 2021-03-15T22:14:02.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: audit 2021-03-15T22:14:01.007674+0000 mon.a (mon. 2021-03-15T22:14:02.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: 0) 78 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:14:02.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: audit 2021-03-15T22:14:01.009882+0000 mon.a (mon.0) 79 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:14:02.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: audit 2021-03-15T22:14:01.028112+0000 mon.a (mon.0) 80 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:14:02.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:01 smithi076 conmon[56514]: audit 2021-03-15T22:14:01.029509+0000 mon.a (mon.0) 81 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:14:02.495 INFO:teuthology.orchestra.run.smithi076.stderr:mgr epoch 9 is available 2021-03-15T22:14:02.496 INFO:teuthology.orchestra.run.smithi076.stderr:Generating a dashboard self-signed certificate... 2021-03-15T22:14:02.496 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 dashboard create-self-signed-cert 2021-03-15T22:14:03.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:02 smithi076 conmon[56514]: audit 2021-03-15T22:14:01.902663+0000 mgr.y (mgr.14144) 1 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2021-03-15T22:14:03.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:02 smithi076 conmon[56514]: cluster 2021-03-15T22:14:01.903203+0000 mon.a (mon.0) 82 : cluster [DBG] mgrmap e11: y(active, since 1.13122s) 2021-03-15T22:14:03.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:02 smithi076 conmon[56514]: audit 2021-03-15T22:14:01.909612+0000 mgr.y (mgr.14144) 2 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2021-03-15T22:14:03.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:02 smithi076 conmon[56514]: audit 2021-03-15T22:14:02.484893+0000 mon.a (mon.0) 83 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:03.316 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: Self-signed certificate created 2021-03-15T22:14:04.090 INFO:teuthology.orchestra.run.smithi076.stderr:Creating initial admin user... 2021-03-15T22:14:04.091 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmpwt_99u7v:/tmp/dashboard.pw:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2021-03-15T22:14:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:04 smithi076 conmon[56514]: audit 2021-03-15T22:14:03.217354+0000 mgr.y (mgr.14144) 3 2021-03-15T22:14:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:04 smithi076 conmon[56514]: : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:14:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:04 smithi076 conmon[56514]: audit 2021-03-15T22:14:03.308969+0000 mon.a (mon.0) 84 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:04 smithi076 conmon[56514]: audit 2021-03-15T22:14:03.313352+0000 mon.a (mon.0) 85 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:04 smithi076 conmon[56514]: cluster 2021-03-15T22:14:03.489012+0000 mon.a (mon.0) 86 : cluster [DBG] mgrmap e12: y(active, since 2s) 2021-03-15T22:14:05.021 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$nnneBcHK2QOwllNBTGc6T.fh1jx1yRWBSm2VTGEjZsQWtfnSqYuVW", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1615846445, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2021-03-15T22:14:05.569 INFO:teuthology.orchestra.run.smithi076.stderr:Fetching dashboard port number... 2021-03-15T22:14:05.570 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /bin/podman run --rm --ipc=host --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 -e NODE_NAME=smithi076 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c:/var/log/ceph:z -v /tmp/ceph-tmp71midtim:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpb6n5a19i:/etc/ceph/ceph.conf:z quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 config get mgr mgr/dashboard/ssl_server_port 2021-03-15T22:14:06.249 INFO:teuthology.orchestra.run.smithi076.stderr:/usr/bin/ceph: 8443 2021-03-15T22:14:06.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:05 smithi076 conmon[56514]: audit 2021-03-15T22:14:04.803816+0000 mgr.y (mgr 2021-03-15T22:14:06.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:05 smithi076 conmon[56514]: .14144) 4 : audit [DBG] from='client.14156 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:14:06.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:05 smithi076 conmon[56514]: audit 2021-03-15T22:14:05.019971+0000 mon.a (mon.0) 87 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:06.819 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-enabled firewalld.service 2021-03-15T22:14:06.828 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: disabled 2021-03-15T22:14:06.829 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: systemctl is-active firewalld.service 2021-03-15T22:14:06.838 INFO:teuthology.orchestra.run.smithi076.stderr:systemctl: inactive 2021-03-15T22:14:06.839 INFO:teuthology.orchestra.run.smithi076.stderr:firewalld.service is not enabled 2021-03-15T22:14:06.839 INFO:teuthology.orchestra.run.smithi076.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2021-03-15T22:14:06.839 INFO:teuthology.orchestra.run.smithi076.stderr:Ceph Dashboard is now available at: 2021-03-15T22:14:06.840 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T22:14:06.840 INFO:teuthology.orchestra.run.smithi076.stderr: URL: https://smithi076.front.sepia.ceph.com:8443/ 2021-03-15T22:14:06.840 INFO:teuthology.orchestra.run.smithi076.stderr: User: admin 2021-03-15T22:14:06.840 INFO:teuthology.orchestra.run.smithi076.stderr: Password: owznxfe512 2021-03-15T22:14:06.841 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T22:14:06.842 INFO:teuthology.orchestra.run.smithi076.stderr:You can access the Ceph CLI with: 2021-03-15T22:14:06.842 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T22:14:06.842 INFO:teuthology.orchestra.run.smithi076.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2021-03-15T22:14:06.842 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T22:14:06.843 INFO:teuthology.orchestra.run.smithi076.stderr:Please consider enabling telemetry to help improve Ceph: 2021-03-15T22:14:06.843 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T22:14:06.843 INFO:teuthology.orchestra.run.smithi076.stderr: ceph telemetry on 2021-03-15T22:14:06.843 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T22:14:06.843 INFO:teuthology.orchestra.run.smithi076.stderr:For more information see: 2021-03-15T22:14:06.844 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T22:14:06.844 INFO:teuthology.orchestra.run.smithi076.stderr: https://docs.ceph.com/docs/master/mgr/telemetry/ 2021-03-15T22:14:06.844 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T22:14:06.844 INFO:teuthology.orchestra.run.smithi076.stderr:Bootstrap complete. 2021-03-15T22:14:06.845 INFO:teuthology.orchestra.run.smithi076.stderr:Releasing lock 140588105407848 on /run/cephadm/8a572802-85db-11eb-9091-001a4aab830c.lock 2021-03-15T22:14:06.845 INFO:teuthology.orchestra.run.smithi076.stderr:Lock 140588105407848 released on /run/cephadm/8a572802-85db-11eb-9091-001a4aab830c.lock 2021-03-15T22:14:06.878 INFO:tasks.cephadm:Fetching config... 2021-03-15T22:14:06.879 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:14:06.879 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2021-03-15T22:14:06.894 INFO:tasks.cephadm:Fetching client.admin keyring... 2021-03-15T22:14:06.895 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:14:06.895 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2021-03-15T22:14:06.949 INFO:tasks.cephadm:Fetching mon keyring... 2021-03-15T22:14:06.950 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:14:06.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd if=/var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/keyring of=/dev/stdout 2021-03-15T22:14:07.015 INFO:tasks.cephadm:Fetching pub ssh key... 2021-03-15T22:14:07.016 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:14:07.016 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2021-03-15T22:14:07.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:06 smithi076 conmon[56514]: audit 2021-03-15T22:14:06.247947+0000 mon.a (mon.0) 88 : audit [DBG] from='client.? 172.21.15.76:0/974196692' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2021-03-15T22:14:07.070 INFO:tasks.cephadm:Installing pub ssh key for root users... 2021-03-15T22:14:07.071 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvhWU99/nHSuIiP/iCJP3EPyx/w3i0CoSoWCzlnYoW7W3iGBYDOgnKMkggrFpOvZGKlzL44n4ZG5fLVpoOWj9GOGjRIZuTJzV7llJhmCfnDHCcedsMiGTcz76gnXxeXfUG1NHxQOztcVOCPM7HaFBT2J9hue9zjThl6YbtrrkXHbeeVDfdzZkQ7B2Aft4sLPK727R2kNA8PlK4kCJEVHPgifHIraJe9Dp1L5bCKSj2pNt8aqQPt8jz1eyaliFmsg4jyrtG4zW5nOyggmHjjM1LZzUNb2wLcC2DEwwizZV3EBOTa1OTgb/GtR/CeinXobKlgdYiPqfv9ZBOLG7eoGxbfsyIWsXaLnjsFjiZTyj4nfGUEtjqNOkFYF2GAvnqFCpL1ZO5oNPrpjfk4XCpeyvOyczA130fKXVBQdTw5ABIJyz5ynyk1OzHuMr4Wq6g5njzmODhE5WOrxxGlC6WfelaunGI5yMti15hUyrI0q4/LCLKXGN+fOi0THfwec4K6Mc= ceph-8a572802-85db-11eb-9091-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-03-15T22:14:07.148 INFO:teuthology.orchestra.run.smithi076.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvhWU99/nHSuIiP/iCJP3EPyx/w3i0CoSoWCzlnYoW7W3iGBYDOgnKMkggrFpOvZGKlzL44n4ZG5fLVpoOWj9GOGjRIZuTJzV7llJhmCfnDHCcedsMiGTcz76gnXxeXfUG1NHxQOztcVOCPM7HaFBT2J9hue9zjThl6YbtrrkXHbeeVDfdzZkQ7B2Aft4sLPK727R2kNA8PlK4kCJEVHPgifHIraJe9Dp1L5bCKSj2pNt8aqQPt8jz1eyaliFmsg4jyrtG4zW5nOyggmHjjM1LZzUNb2wLcC2DEwwizZV3EBOTa1OTgb/GtR/CeinXobKlgdYiPqfv9ZBOLG7eoGxbfsyIWsXaLnjsFjiZTyj4nfGUEtjqNOkFYF2GAvnqFCpL1ZO5oNPrpjfk4XCpeyvOyczA130fKXVBQdTw5ABIJyz5ynyk1OzHuMr4Wq6g5njzmODhE5WOrxxGlC6WfelaunGI5yMti15hUyrI0q4/LCLKXGN+fOi0THfwec4K6Mc= ceph-8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:14:07.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvhWU99/nHSuIiP/iCJP3EPyx/w3i0CoSoWCzlnYoW7W3iGBYDOgnKMkggrFpOvZGKlzL44n4ZG5fLVpoOWj9GOGjRIZuTJzV7llJhmCfnDHCcedsMiGTcz76gnXxeXfUG1NHxQOztcVOCPM7HaFBT2J9hue9zjThl6YbtrrkXHbeeVDfdzZkQ7B2Aft4sLPK727R2kNA8PlK4kCJEVHPgifHIraJe9Dp1L5bCKSj2pNt8aqQPt8jz1eyaliFmsg4jyrtG4zW5nOyggmHjjM1LZzUNb2wLcC2DEwwizZV3EBOTa1OTgb/GtR/CeinXobKlgdYiPqfv9ZBOLG7eoGxbfsyIWsXaLnjsFjiZTyj4nfGUEtjqNOkFYF2GAvnqFCpL1ZO5oNPrpjfk4XCpeyvOyczA130fKXVBQdTw5ABIJyz5ynyk1OzHuMr4Wq6g5njzmODhE5WOrxxGlC6WfelaunGI5yMti15hUyrI0q4/LCLKXGN+fOi0THfwec4K6Mc= ceph-8a572802-85db-11eb-9091-001a4aab830c' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2021-03-15T22:14:07.204 INFO:teuthology.orchestra.run.smithi125.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCvhWU99/nHSuIiP/iCJP3EPyx/w3i0CoSoWCzlnYoW7W3iGBYDOgnKMkggrFpOvZGKlzL44n4ZG5fLVpoOWj9GOGjRIZuTJzV7llJhmCfnDHCcedsMiGTcz76gnXxeXfUG1NHxQOztcVOCPM7HaFBT2J9hue9zjThl6YbtrrkXHbeeVDfdzZkQ7B2Aft4sLPK727R2kNA8PlK4kCJEVHPgifHIraJe9Dp1L5bCKSj2pNt8aqQPt8jz1eyaliFmsg4jyrtG4zW5nOyggmHjjM1LZzUNb2wLcC2DEwwizZV3EBOTa1OTgb/GtR/CeinXobKlgdYiPqfv9ZBOLG7eoGxbfsyIWsXaLnjsFjiZTyj4nfGUEtjqNOkFYF2GAvnqFCpL1ZO5oNPrpjfk4XCpeyvOyczA130fKXVBQdTw5ABIJyz5ynyk1OzHuMr4Wq6g5njzmODhE5WOrxxGlC6WfelaunGI5yMti15hUyrI0q4/LCLKXGN+fOi0THfwec4K6Mc= ceph-8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:14:07.216 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph config set mgr mgr/cephadm/allow_ptrace true 2021-03-15T22:14:07.950 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:07 smithi076 conmon[56514]: cluster 2021-03-15T22:14:06.901735+0000 2021-03-15T22:14:07.951 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:07 smithi076 conmon[56514]: mon.a (mon.0) 89 : cluster [DBG] mgrmap e13: y(active, since 6s) 2021-03-15T22:14:09.866 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi125 2021-03-15T22:14:09.867 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-15T22:14:09.867 DEBUG:teuthology.orchestra.run.smithi125:> dd of=/etc/ceph/ceph.conf 2021-03-15T22:14:09.884 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-15T22:14:09.885 DEBUG:teuthology.orchestra.run.smithi125:> dd of=/etc/ceph/ceph.client.admin.keyring 2021-03-15T22:14:09.940 INFO:tasks.cephadm:Adding host smithi125 to orchestrator... 2021-03-15T22:14:09.941 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch host add smithi125 2021-03-15T22:14:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:10 smithi076 conmon[56514]: audit 2021-03-15T22:14:09.102322+0000 mon.a (mon.0) 90 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:10 smithi076 conmon[56514]: cephadm 2021-03-15T22:14:09.396710+0000 mgr.y (mgr.14144) 5 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:14:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:10 smithi076 conmon[56514]: audit 2021-03-15T22:14:09.399880+0000 mon.a (mon.0) 91 : audit [INF] from='client.? 172.21.15.76:0/618607428' entity='client.admin' 2021-03-15T22:14:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:10 smithi076 conmon[56514]: audit 2021-03-15T22:14:09.404519+0000 mon.a (mon.0) 92 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:10 smithi076 conmon[56514]: audit 2021-03-15T22:14:09.406210+0000 mon.a (mon.0) 93 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:14:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:10 smithi076 conmon[56514]: cephadm 2021-03-15T22:14:09.407801+0000 mgr.y (mgr.14144) 6 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:14:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:10 smithi076 conmon[56514]: audit 2021-03-15T22:14:09.413360+0000 mon.a (mon.0) 94 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:22.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:21 smithi076 conmon[56514]: cluster 2021-03-15T22:14:20.904628+0000 mgr.y (mgr.14144) 7 : cluster 2021-03-15T22:14:22.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:21 smithi076 conmon[56514]: [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:24.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:23 smithi076 conmon[56514]: cluster 2021-03-15T22:14:22.904947+0000 mgr.y (mgr.14144) 8 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:26.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:25 smithi076 conmon[56514]: cluster 2021-03-15T22:14:24.905198+0000 mgr.y (mgr.14144) 9 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:28.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:27 smithi076 conmon[56514]: cluster 2021-03-15T22:14:26.905380+0000 mgr.y (mgr.14144) 10 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:30.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:29 smithi076 conmon[56514]: cluster 2021-03-15T22:14:28.905686+0000 mgr.y (mgr.14144) 11 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:32.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:31 smithi076 conmon[56514]: cluster 2021-03-15T22:14:30.905900+0000 mgr.y (mgr.14144) 12 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:34.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:33 smithi076 conmon[56514]: cluster 2021-03-15T22:14:32.906217+0000 mgr.y (mgr.14144 2021-03-15T22:14:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:33 smithi076 conmon[56514]: ) 13 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:36.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:35 smithi076 conmon[56514]: cluster 2021-03-15T22:14:34. 2021-03-15T22:14:36.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:35 smithi076 conmon[56514]: 906433+0000 mgr.y (mgr.14144) 14 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:37.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:36 smithi076 conmon[56514]: cluster 2021-03-15T22:14:36.906802+0000 mgr.y ( 2021-03-15T22:14:37.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:36 smithi076 conmon[56514]: mgr.14144) 15 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:39.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:38 smithi076 conmon[56514]: cluster 2021-03-15T22:14:38.907008+0000 mgr.y (mgr.14144) 16 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:41.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:40 smithi076 conmon[56514]: cluster 2021-03-15T22:14:40.907209+0000 mgr.y (mgr.14144 2021-03-15T22:14:41.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:40 smithi076 conmon[56514]: ) 17 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:43.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:42 smithi076 conmon[56514]: cluster 2021-03-15T22:14:42.907530+0000 mgr.y (mgr.14144) 18 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:46.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:45 smithi076 conmon[56514]: cluster 2021-03-15T22:14:44.907819+0000 mgr.y (mgr.14144) 19 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:47.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:46 smithi076 conmon[56514]: cluster 2021-03-15T22:14:46.908149+0000 mgr.y (mgr.14144) 20 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:49.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:48 smithi076 conmon[56514]: cluster 2021-03-15T22:14:48.908434+0000 mgr.y (mgr.14144) 21 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:50.766 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:14:50 smithi076 conmon[56844]: [90B blob data] 2021-03-15T22:14:51.135 INFO:teuthology.orchestra.run.smithi125.stdout:Added host 'smithi125' 2021-03-15T22:14:51.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:50 smithi076 conmon[56514]: audit 2021-03-15T22:14:50.476678+0000 mgr.y (mgr.14144) 22 : audit [DBG] from='client.14162 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi125", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:14:51.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:50 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:14:51.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:50 smithi076 conmon[56514]: 14:50.908726+0000 mgr.y (mgr.14144) 23 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:51.801 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch host ls --format=json 2021-03-15T22:14:52.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:52 smithi076 conmon[56514]: audit 2021-03-15T22:14:51.132132+0000 mon.a (mon.0) 95 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:52.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:52 smithi076 conmon[56514]: cephadm 2021-03-15T22:14:51.132513+0000 mgr.y (mgr.14144) 24 : cephadm [INF] Added host smithi125 2021-03-15T22:14:52.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:52 smithi076 conmon[56514]: audit 2021-03-15T22:14:51.132870+0000 mon.a (mon.0) 96 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:14:52.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:52 smithi076 conmon[56514]: audit 2021-03-15T22:14:51.386299+0000 mon.a (mon.0) 2021-03-15T22:14:52.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:52 smithi076 conmon[56514]: 97 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:52.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:52 smithi076 conmon[56514]: audit 2021-03-15T22: 2021-03-15T22:14:52.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:52 smithi076 conmon[56514]: 14:51.836928+0000 mon.a (mon.0) 98 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:53.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:53 smithi076 conmon[56514]: cluster 2021-03-15T22:14:52.909084+0000 mgr.y (mgr.14144) 2021-03-15T22:14:53.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:53 smithi076 conmon[56514]: 25 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:54.178 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:14:54.178 INFO:teuthology.orchestra.run.smithi125.stdout:[{"addr": "smithi076", "hostname": "smithi076", "labels": [], "status": ""}, {"addr": "smithi125", "hostname": "smithi125", "labels": [], "status": ""}] 2021-03-15T22:14:54.827 INFO:tasks.cephadm:Setting crush tunables to default 2021-03-15T22:14:54.828 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd crush tunables default 2021-03-15T22:14:56.195 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:55 smithi076 conmon[56514]: audit 2021-03-15T22:14:54.176450+0000 mgr.y (mgr.14144) 26 : audit [DBG] from='client.14164 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:14:56.195 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:55 smithi076 conmon[56514]: cluster 2021-03-15T22:14:54.909367+0000 mgr.y (mgr.14144) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:56.196 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:55 smithi076 conmon[56514]: audit 2021-03-15T22:14:55.719586+0000 mon.a (mon.0) 99 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:57.009 INFO:teuthology.orchestra.run.smithi076.stderr:adjusted tunables profile to default 2021-03-15T22:14:57.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:56 smithi076 conmon[56514]: audit 2021-03-15T22:14:56.000292+0000 mon.a (mon.0) 100 : audit 2021-03-15T22:14:57.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:56 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:57.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:14:56.000851+0000 mgr.y (mgr.14144) 28 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:14:57.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:56 smithi076 conmon[56514]: audit 2021-03-15T22:14:56.003142+0000 mon.a (mon.0) 101 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:14:57.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:56 smithi076 conmon[56514]: audit 2021-03-15T22:14:56.523909+0000 mon.a (mon.0) 102 : audit [INF] from='client.? 172.21.15.76:0/33838333' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2021-03-15T22:14:57.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:56 smithi076 conmon[56514]: cluster 2021-03-15T22:14:56.909710+0000 mgr.y (mgr.14144) 29 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:14:57.645 INFO:tasks.cephadm:Adding mon.c on smithi076 2021-03-15T22:14:57.646 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch daemon add mon 'smithi076:[v2:172.21.15.76:3301,v1:172.21.15.76:6790]=c' 2021-03-15T22:14:58.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:58 smithi076 conmon[56514]: audit 2021-03-15T22:14:57.004764+0000 mon.a (mon.0) 103 : audit 2021-03-15T22:14:58.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:58 smithi076 conmon[56514]: [INF] from='client.? 172.21.15.76:0/33838333' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2021-03-15T22:14:58.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:58 smithi076 conmon[56514]: cluster 2021-03-15T22:14:57.004818+0000 mon.a (mon.0) 104 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-15T22:14:59.046 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:14:59 smithi076 conmon[56514]: cluster 2021-03-15T22:14:58.909993+0000 mgr.y (mgr.14144) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:00.113 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:00 smithi076 conmon[56514]: audit 2021-03-15T22:14:59.396804+0000 mgr.y (mgr.14144) 31 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi076:[v2:172.21.15.76:3301,v1:172.21.15.76:6790]=c", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:00.114 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:00 smithi076 conmon[56514]: audit 2021-03-15T22:14:59.398334+0000 mon.a (mon.0) 105 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-15T22:15:00.114 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:00 smithi076 conmon[56514]: audit 2021-03-15T22:14:59.398905+0000 mon.a (mon.0) 106 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:00.114 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:00 smithi076 conmon[56514]: cephadm 2021-03-15T22:14:59.399359+0000 mgr.y (mgr.14144) 32 : cephadm [INF] Deploying daemon mon.c on smithi076 2021-03-15T22:15:01.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:01 smithi076 conmon[56514]: cluster 2021-03-15T22:15:00.910361+0000 mgr.y (mgr 2021-03-15T22:15:01.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:01 smithi076 conmon[56514]: .14144) 33 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:02.265 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:02 smithi076 conmon[56514]: audit 2021-03-15T22:15: 2021-03-15T22:15:02.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:02 smithi076 conmon[56514]: 01.029222+0000 mon.a (mon.0) 107 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:15:02.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:02 smithi076 conmon[56514]: audit 2021-03-15T22:15:01.030349+0000 mon.a (mon.0) 108 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:15:07.330 INFO:teuthology.orchestra.run.smithi076.stdout:Deployed mon.c on host 'smithi076' 2021-03-15T22:15:07.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: audit 2021-03-15T22:15:02.291922+0000 mon.a ( 2021-03-15T22:15:07.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: mon.0) 110 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-15T22:15:07.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: audit 2021-03-15T22:15:02.292165+0000 mon.a (mon.0) 111 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-15T22:15:07.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster 2021-03-15T22:15:02.292304+0000 mon.a (mon.0) 112 : cluster [INF] mon.a calling monitor election 2021-03-15T22:15:07.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: audit 2021-03-15T22:15:02.316979+0000 mon.a (mon.0) 113 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:07.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster 2021-03-15T22:15:02.910611+0000 mgr.y (mgr.14144) 34 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:07.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: audit 2021-03-15T22:15:03.288851+0000 mon.a (mon.0) 114 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-15T22:15:07.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: audit 2021-03-15T22:15:04.288812+0000 mon.a (mon.0) 115 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-15T22:15:07.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster 2021-03-15T22:15:04.290509+0000 mon.c (mon.1) 1 : cluster [INF] mon.c calling monitor election 2021-03-15T22:15:07.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster 2021-03-15T22:15:04.910916+0000 mgr.y (mgr.14144) 35 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:07.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: audit 2021-03-15T22:15:05.288903+0000 mon.a (mon.0) 116 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-15T22:15:07.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: audit 2021-03-15T22:15:07.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 2021-03-15T22:15:06.289205+0000 mon.a (mon.0) 117 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-15T22:15:07.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster 2021-03-15T22:15:07.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 2021-03-15T22 2021-03-15T22:15:07.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: :15:06.911224+0000 mgr.y ( 2021-03-15T22:15:07.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: mgr.14144) 36 : cluster [DBG] 2021-03-15T22:15:07.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:07.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: audit 2021-03-15T22:15:07.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 2021-03- 2021-03-15T22:15:07.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 15T22:15:07.289237 2021-03-15T22:15:07.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: +0000 mon.a (mon.0) 118 2021-03-15T22:15:07.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-15T22:15:07.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:15:07.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 15T22:15:07.309589+0000 2021-03-15T22:15:07.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: mon.a (mon.0) 119 : 2021-03-15T22:15:07.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster [INF] mon.a is new leader, mons a,c in quorum (ranks 0,1) 2021-03-15T22:15:07.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:15:07.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 03-15T22:15:07.319702 2021-03-15T22:15:07.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: +0000 mon.a (mon.0) 120 2021-03-15T22:15:07.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: : cluster [DBG] monmap e2: 2 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],c=[v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0]} 2021-03-15T22:15:07.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster 2021-03-15T22:15:07.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:15:07.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 15:07.319752+0000 mon.a ( 2021-03-15T22:15:07.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: mon.0) 121 : cluster [DBG] fsmap 2021-03-15T22:15:07.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:15:07.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: -03-15T22:15:07. 2021-03-15T22:15:07.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 319781+0000 mon.a (mon.0) 122 : cluster 2021-03-15T22:15:07.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-15T22:15:07.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:15:07.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 03-15T22:15:07.319983+0000 mon.a (mon 2021-03-15T22:15:07.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: .0) 123 : cluster [DBG] mgrmap e13: y(active, since 66s) 2021-03-15T22:15:07.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:15:07.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 03-15T22:15:07.323429+0000 mon.a (mon. 2021-03-15T22:15:07.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 0) 124 : cluster [INF] overall HEALTH_OK 2021-03-15T22:15:07.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: audit 2021-03-15T22:15:07.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 2021-03-15T22:15: 2021-03-15T22:15:07.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: 07.325963+0000 mon.a (mon.0) 125 : audit 2021-03-15T22:15:07.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:07 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:07.959 DEBUG:teuthology.orchestra.run.smithi076:mon.c> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@mon.c.service 2021-03-15T22:15:07.962 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2021-03-15T22:15:07.962 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph mon dump -f json 2021-03-15T22:15:07.992 INFO:journalctl@ceph.mon.c.smithi076.stdout:-- Logs begin at Mon 2021-03-15 21:44:23 UTC. -- 2021-03-15T22:15:08.508 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:08 smithi076 conmon[56514]: audit 2021-03-15T22:15:08.289382+0000 mon.a (mon.0) 126 : audit 2021-03-15T22:15:08.508 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:08 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-15T22:15:08.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:08 smithi076 conmon[62156]: audit 2021-03-15T22:15:08.289382+0000 mon.a (mon. 2021-03-15T22:15:08.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:08 smithi076 conmon[62156]: 0) 126 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-15T22:15:09.873 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:15:09.874 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":2,"fsid":"8a572802-85db-11eb-9091-001a4aab830c","modified":"2021-03-15T22:15:02.289727Z","created":"2021-03-15T22:13:17.824467Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3301","nonce":0},{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2021-03-15T22:15:09.875 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 2 2021-03-15T22:15:09.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:09 smithi076 conmon[56514]: audit 2021-03-15T22:15:08.662457+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:09.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:09 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:08.662964+0000 mgr.y (mgr.14144) 37 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:09.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:09 smithi076 conmon[56514]: audit 2021-03-15T22:15:08.666210+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:09.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:09 smithi076 conmon[56514]: cluster 2021-03-15T22:15:08.911494+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:09.962 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:09 smithi076 conmon[62156]: audit 2021-03-15T22:15:08 2021-03-15T22:15:09.963 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:09 smithi076 conmon[62156]: .662457+0000 mon.a (mon.0) 127 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:09.963 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:09 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:08.662964+0000 mgr.y (mgr.14144) 37 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:09.963 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:09 smithi076 conmon[62156]: audit 2021-03-15T22:15:08.666210+0000 mon.a (mon.0) 128 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:09.963 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:09 smithi076 conmon[62156]: cluster 2021-03-15T22:15:08.911494+0000 mgr.y (mgr.14144) 38 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:10.457 INFO:tasks.cephadm:Adding mon.b on smithi125 2021-03-15T22:15:10.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch daemon add mon smithi125:172.21.15.125=b 2021-03-15T22:15:10.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:10 smithi076 conmon[56514]: audit 2021-03-15T22:15:09. 2021-03-15T22:15:10.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:10 smithi076 conmon[56514]: 872137+0000 mon.a (mon.0) 129 : audit [DBG] from='client.? 172.21.15.76:0/4242729353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-15T22:15:10.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:10 smithi076 conmon[62156]: audit 2021-03-15T22:15:09.872137+0000 mon.a (mon 2021-03-15T22:15:10.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:10 smithi076 conmon[62156]: .0) 129 : audit [DBG] from='client.? 172.21.15.76:0/4242729353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-15T22:15:12.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:11 smithi076 conmon[56514]: cluster 2021-03-15T22:15:10.911859+0000 mgr.y (mgr.14144) 39 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:12.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:11 smithi076 conmon[62156]: cluster 2021-03-15T22:15:10.911859+0000 mgr.y (mgr.14144) 39 : 2021-03-15T22:15:12.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:11 smithi076 conmon[62156]: cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:13.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:12 smithi076 conmon[56514]: audit 2021-03-15T22:15:12.213079+0000 mon.a (mon.0) 130 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-15T22:15:13.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:12 smithi076 conmon[56514]: audit 2021-03-15T22:15:12.214041+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:13.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:12 smithi076 conmon[62156]: audit 2021-03-15T22:15:12.213079+0000 mon.a (mon.0) 2021-03-15T22:15:13.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:12 smithi076 conmon[62156]: 130 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2021-03-15T22:15:13.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:12 smithi076 conmon[62156]: audit 2021-03-15T22:15:12.214041+0000 mon.a (mon.0) 131 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:14.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:13 smithi076 conmon[56514]: audit 2021-03-15T22:15:12.211776+0000 mgr.y (mgr.14144) 40 : audit 2021-03-15T22:15:14.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:13 smithi076 conmon[56514]: [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi125:172.21.15.125=b", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:14.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:13 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:12.214754+0000 mgr.y (mgr.14144) 41 : cephadm [INF] Deploying daemon mon.b on smithi125 2021-03-15T22:15:14.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:13 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:15:14.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:13 smithi076 conmon[56514]: :15:12.912185+0000 mgr.y (mgr.14144) 42 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:14.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:13 smithi076 conmon[62156]: audit 2021-03-15T22:15:12.211776+0000 mgr.y (mgr.14144) 2021-03-15T22:15:14.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:13 smithi076 conmon[62156]: 40 : audit [DBG] from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch daemon add", "daemon_type": "mon", "placement": "smithi125:172.21.15.125=b", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:14.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:13 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:12.214754+0000 mgr.y (mgr.14144) 41 : cephadm [INF] Deploying daemon mon.b on smithi125 2021-03-15T22:15:14.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:13 smithi076 conmon[62156]: cluster 2021-03-15T22:15:12.912185+0000 mgr.y (mgr.14144) 42 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:20.225 INFO:teuthology.orchestra.run.smithi125.stdout:Deployed mon.b on host 'smithi125' 2021-03-15T22:15:20.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: audit 2021-03-15T22:15:15.194678+0000 mon.a ( 2021-03-15T22:15:20.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: mon.0) 133 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-15T22:15:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster 2021-03-15T22:15:15.194985+0000 mon.a (mon.0) 134 : cluster [INF] mon.a calling monitor election 2021-03-15T22:15:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: audit 2021-03-15T22:15:15.196762+0000 mon.a (mon.0 2021-03-15T22:15:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: ) 135 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster 2021-03-15T22:15:15.197055+0000 mon.c (mon 2021-03-15T22:15:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: .1) 2 : cluster [INF] mon.c calling monitor election 2021-03-15T22:15:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: audit 2021-03-15T22:15:15.197073+0000 mon.a 2021-03-15T22:15:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: (mon.0) 136 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-15T22:15:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: audit 2021 2021-03-15T22:15:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: -03-15T22:15:15.204996+0000 mon.a ( 2021-03-15T22:15:20.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: mon.0) 137 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:20.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: audit 2021-03-15T22:15:16. 2021-03-15T22:15:20.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 191177+0000 mon.a (mon.0) 138 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:15:20.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 03-15T22:15:16.912869+0000 mgr.y (mgr 2021-03-15T22:15:20.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: .14144) 44 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:20.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:15:20.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 03-15T22:15:17.191255+0000 mon.a (mon 2021-03-15T22:15:20.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: .0) 139 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:15:20.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: T22:15:17.192446+0000 mon.b (mon.2) 2021-03-15T22:15:20.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 1 : cluster [INF] mon.b calling monitor election 2021-03-15T22:15:20.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: audit 2021-03-15T22: 2021-03-15T22:15:20.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 15:18.191610+0000 mon.a (mon.0) 140 : audit 2021-03-15T22:15:20.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster 2021-03-15T22:15:18.913188 2021-03-15T22:15:20.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 45 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:20.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: audit 2021 2021-03-15T22:15:20.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: -03-15T22:15:19.191598+0000 mon.a ( 2021-03-15T22:15:20.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: mon.0) 141 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: audit 2021 2021-03-15T22:15:20.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: -03-15T22:15:20.191669+0000 mon.a (mon 2021-03-15T22:15:20.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: .0) 142 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:15:20.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: -03-15T22:15:20.210366+0000 mon.a (mon.0) 143 : cluster [INF] 2021-03-15T22:15:20.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-03-15T22:15:20.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster 2021-03-15T22:15:20.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 2021-03-15T22:15:20.213729+0000 mon.a 2021-03-15T22:15:20.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: audit 2021-03-15T22:15:15.194678 2021-03-15T22:15:20.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: +0000 mon.a (mon.0) 133 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2021-03-15T22:15:20.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster 2021-03-15T22:15:15.194985 2021-03-15T22:15:20.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: +0000 mon.a (mon.0) 134 : 2021-03-15T22:15:20.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster [INF] mon.a calling monitor election 2021-03-15T22:15:20.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: audit 2021-03-15T22:15:20.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:15:20.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 22:15:15.196762+0000 mon.a (mon. 2021-03-15T22:15:20.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 0) 135 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:15:20.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: -03-15T22:15:15.197055+0000 2021-03-15T22:15:20.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: mon.c (mon.1) 2 : cluster [INF] mon.c calling monitor election 2021-03-15T22:15:20.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 2021-03-15T22:15:20.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: audit 2021-03-15T22:15:20.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 2021-03-15T22:15 2021-03-15T22:15:20.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: :15.197073+0000 mon.a (mon.0) 136 : audit 2021-03-15T22:15:20.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2021-03-15T22:15:20.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: audit 2021-03-15T22:15:20.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 2021-03-15T22:15 2021-03-15T22:15:20.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: :15.204996+0000 mon.a (mon.0) 137 : audit 2021-03-15T22:15:20.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:20.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: audit 2021 2021-03-15T22:15:20.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: -03-15T22:15:16. 2021-03-15T22:15:20.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 191177+0000 mon.a (mon.0) 138 : 2021-03-15T22:15:20.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:15:20.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: -03-15T22:15:16.912869 2021-03-15T22:15:20.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 44 : cluster [DBG] 2021-03-15T22:15:20.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:20.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: audit 2021-03-15T22:15:20.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 2021- 2021-03-15T22:15:20.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 03-15T22:15:17 2021-03-15T22:15:20.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: .191255+0000 mon.a (mon.0) 139 2021-03-15T22:15:20.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:15:20.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: :15:17.192446+0000 mon.b (mon.2) 1 : cluster [INF] mon.b calling monitor election 2021-03-15T22:15:20.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: audit 2021-03-15T22:15:18.191610+0000 mon.a 2021-03-15T22:15:20.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: (mon.0) 140 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster 2021-03-15T22:15:18.913188+0000 mgr.y 2021-03-15T22:15:20.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: (mgr.14144) 45 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:20.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:15:20.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 22:15:19.191598+0000 2021-03-15T22:15:20.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: mon.a (mon.0) 141 : audit 2021-03-15T22:15:20.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: audit 2021-03-15T22:15:20.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 2021-03-15T22:15:20.191669+0000 2021-03-15T22:15:20.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: mon.a (mon.0) 142 : audit [DBG] 2021-03-15T22:15:20.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: (mon.0) 144 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0],c=[v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0]} 2021-03-15T22:15:20.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster 2021-03-15T22:15:20.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 2021-03-15T22:15:20.213782 2021-03-15T22:15:20.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: +0000 mon.a (mon.0) 145 : cluster [DBG] fsmap 2021-03-15T22:15:20.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 2021-03-15T22:15:20.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster 2021-03-15T22:15:20 2021-03-15T22:15:20.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: .213841+0000 mon.a (mon.0) 146 : 2021-03-15T22:15:20.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-15T22:15:20.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster 2021-03-15T22:15:20.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 2021-03-15T22:15: 2021-03-15T22:15:20.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 20.214078+0000 mon.a (mon.0) 147 : cluster 2021-03-15T22:15:20.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: [DBG] mgrmap e13: y(active, since 79s) 2021-03-15T22:15:20.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:15:20.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: -15T22:15:20.218345+0000 mon.a ( 2021-03-15T22:15:20.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: mon.0) 148 : cluster [INF] overall HEALTH_OK 2021-03-15T22:15:20.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: audit 2021-03-15T22:15:20.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 2021-03-15T22:15:20.221536 2021-03-15T22:15:20.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: +0000 mon.a (mon.0) 149 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:20.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[56514]: 2021-03-15T22:15:20.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:20.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster 2021-03-15T22:15:20.210366+0000 mon.a 2021-03-15T22:15:20.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: (mon.0) 143 : cluster [INF] mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2021-03-15T22:15:20.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:15:20.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 15T22:15:20.213729+0000 mon.a ( 2021-03-15T22:15:20.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: mon.0) 144 : cluster [DBG] monmap e3: 3 mons at {a=[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0],b=[v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0],c=[v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0]} 2021-03-15T22:15:20.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster 2021-03-15T22:15:20.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 2021-03-15T22:15 2021-03-15T22:15:20.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: :20.213782+0000 mon.a (mon 2021-03-15T22:15:20.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: .0) 145 : cluster [DBG] 2021-03-15T22:15:20.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: fsmap 2021-03-15T22:15:20.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:15:20.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 15T22:15:20.213841+0000 mon.a (mon. 2021-03-15T22:15:20.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 0) 146 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2021-03-15T22:15:20.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:15:20.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: 03-15T22:15:20.214078+0000 mon.a (mon.0 2021-03-15T22:15:20.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: ) 147 : cluster [DBG] mgrmap e13: y(active, since 79s) 2021-03-15T22:15:20.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:15:20.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: :15:20.218345+0000 mon.a (mon.0) 148 : cluster [INF] 2021-03-15T22:15:20.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: overall HEALTH_OK 2021-03-15T22:15:20.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:15:20.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: T22:15:20.221536+0000 mon.a (mon.0) 149 : audit 2021-03-15T22:15:20.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:20 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:20.852 DEBUG:teuthology.orchestra.run.smithi125:mon.b> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@mon.b.service 2021-03-15T22:15:20.855 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2021-03-15T22:15:20.855 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph mon dump -f json 2021-03-15T22:15:20.883 INFO:journalctl@ceph.mon.b.smithi125.stdout:-- Logs begin at Mon 2021-03-15 21:44:18 UTC. -- 2021-03-15T22:15:21.433 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:21 smithi125 conmon[56739]: cluster 2021-03-15T22:15:20.913436+0000 mgr.y (mgr.14144 2021-03-15T22:15:21.433 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:21 smithi125 conmon[56739]: ) 46 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:21.433 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:21 smithi125 conmon[56739]: audit 2021-03-15T22:15:21.191509+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:21.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:21 smithi076 conmon[56514]: cluster 2021-03-15T22:15:20.913436+0000 mgr.y (mgr.14144) 46 2021-03-15T22:15:21.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:21 smithi076 conmon[56514]: : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:21.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:21 smithi076 conmon[56514]: audit 2021-03-15T22:15:21.191509+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:21.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:21 smithi076 conmon[62156]: cluster 2021-03-15T22:15:20.913436+0000 mgr.y (mgr.14144) 46 : cluster [DBG] 2021-03-15T22:15:21.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:21 smithi076 conmon[62156]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:21.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:21 smithi076 conmon[62156]: audit 2021-03-15T22:15:21.191509+0000 mon.a (mon.0) 150 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2021-03-15T22:15:22.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:22 smithi125 conmon[56739]: audit 2021-03-15T22:15:21.414042+0000 mon.a (mon.0) 151 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:22.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:22 smithi125 conmon[56739]: cephadm 2021-03-15T22:15:21.414523+0000 mgr.y (mgr.14144) 47 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:22.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:22 smithi125 conmon[56739]: audit 2021-03-15T22:15:21.419792+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:22.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:22 smithi076 conmon[56514]: audit 2021-03-15T22:15:21.414042+0000 mon.a (mon.0) 151 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:22.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:22 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:21.414523+0000 mgr.y (mgr.14144) 47 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:22.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:22 smithi076 conmon[56514]: audit 2021-03-15T22:15:21.419792+0000 mon.a (mon.0) 152 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:22.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:22 smithi076 conmon[62156]: audit 2021-03-15T22:15:21.414042+0000 mon.a (mon 2021-03-15T22:15:22.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:22 smithi076 conmon[62156]: .0) 151 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:22.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:22 smithi076 conmon[62156]: cephadm 2021-03-15T22:15 2021-03-15T22:15:22.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:22 smithi076 conmon[62156]: :21.414523+0000 mgr.y (mgr.14144) 47 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:22.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:22 smithi076 conmon[62156]: audit 2021-03-15T22:15:21.419792+0000 mon.a (mon.0) 152 : 2021-03-15T22:15:22.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:22 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:22.830 INFO:teuthology.orchestra.run.smithi125.stdout: 2021-03-15T22:15:22.831 INFO:teuthology.orchestra.run.smithi125.stdout:{"epoch":3,"fsid":"8a572802-85db-11eb-9091-001a4aab830c","modified":"2021-03-15T22:15:15.191438Z","created":"2021-03-15T22:13:17.824467Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3300","nonce":0},{"type":"v1","addr":"172.21.15.76:6789","nonce":0}]},"addr":"172.21.15.76:6789/0","public_addr":"172.21.15.76:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:3301","nonce":0},{"type":"v1","addr":"172.21.15.76:6790","nonce":0}]},"addr":"172.21.15.76:6790/0","public_addr":"172.21.15.76:6790/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:3300","nonce":0},{"type":"v1","addr":"172.21.15.125:6789","nonce":0}]},"addr":"172.21.15.125:6789/0","public_addr":"172.21.15.125:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2021-03-15T22:15:22.832 INFO:teuthology.orchestra.run.smithi125.stderr:dumped monmap epoch 3 2021-03-15T22:15:23.363 INFO:tasks.cephadm:Generating final ceph.conf file... 2021-03-15T22:15:23.364 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph config generate-minimal-conf 2021-03-15T22:15:23.756 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:23 smithi076 conmon[56514]: audit 2021-03-15T22:15:22.828933+0000 mon.c (mon.1) 3 : 2021-03-15T22:15:23.757 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:23 smithi076 conmon[56514]: audit [DBG] from='client.? 172.21.15.125:0/529976481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-15T22:15:23.757 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:23 smithi076 conmon[56514]: cluster 2021-03-15T22:15:22.913791+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:23.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:23 smithi076 conmon[62156]: audit 2021-03-15T22:15:22.828933+0000 mon.c 2021-03-15T22:15:23.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:23 smithi076 conmon[62156]: (mon.1) 3 : audit [DBG] from='client.? 172.21.15.125:0/529976481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-15T22:15:23.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:23 smithi076 conmon[62156]: cluster 2021-03-15T22:15:23.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:23 smithi076 conmon[62156]: 2021-03-15T22:15:22.913791+0000 mgr.y (mgr.14144) 48 : cluster [DBG] 2021-03-15T22:15:23.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:23 smithi076 conmon[62156]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:23.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:23 smithi125 conmon[56739]: audit 2021-03-15T22:15:22.828933+0000 mon.c (mon.1) 3 : audit [DBG] from='client.? 172.21.15.125:0/529976481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-15T22:15:23.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:23 smithi125 conmon[56739]: cluster 2021-03-15T22:15 2021-03-15T22:15:23.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:23 smithi125 conmon[56739]: :22.913791+0000 mgr.y (mgr.14144) 48 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:25.051 INFO:teuthology.orchestra.run.smithi076.stdout:# minimal ceph.conf for 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:15:25.052 INFO:teuthology.orchestra.run.smithi076.stdout:[global] 2021-03-15T22:15:25.052 INFO:teuthology.orchestra.run.smithi076.stdout: fsid = 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:15:25.052 INFO:teuthology.orchestra.run.smithi076.stdout: mon_host = [v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0] [v2:172.21.15.125:3300/0,v1:172.21.15.125:6789/0] [v2:172.21.15.76:3301/0,v1:172.21.15.76:6790/0] 2021-03-15T22:15:25.629 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2021-03-15T22:15:25.630 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:15:25.630 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.conf 2021-03-15T22:15:25.660 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:15:25.660 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-03-15T22:15:25.725 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-15T22:15:25.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/ceph/ceph.conf 2021-03-15T22:15:25.761 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-15T22:15:25.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2021-03-15T22:15:25.829 INFO:tasks.cephadm:Adding mgr.y on smithi076 2021-03-15T22:15:25.830 INFO:tasks.cephadm:Adding mgr.x on smithi125 2021-03-15T22:15:25.830 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch apply mgr '2;smithi076=y;smithi125=x' 2021-03-15T22:15:26.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:25 smithi125 conmon[56739]: cluster 2021-03-15T22:15:24.914043+0000 mgr.y (mgr.14144) 49 : 2021-03-15T22:15:26.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:25 smithi125 conmon[56739]: cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:26.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:25 smithi125 conmon[56739]: audit 2021-03-15T22:15:25.050458+0000 mon.a (mon.0) 153 : audit [DBG] from='client.? 172.21.15.76:0/2488190683' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:26.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:25 smithi076 conmon[56514]: cluster 2021-03-15T22:15:24.914043+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:26.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:25 smithi076 conmon[56514]: audit 2021-03-15T22:15:25.050458+0000 mon.a (mon.0) 153 : audit [DBG] from='client.? 172.21.15.76:0/2488190683' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:26.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:25 smithi076 conmon[62156]: cluster 2021-03-15T22:15:24.914043+0000 mgr.y (mgr.14144) 49 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:26.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:25 smithi076 conmon[62156]: audit 2021-03-15T22:15:25.050458+0000 mon.a (mon.0) 153 : audit [DBG] from='client.? 172.21.15.76:0/2488190683' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:27.671 INFO:teuthology.orchestra.run.smithi125.stdout:Scheduled mgr update... 2021-03-15T22:15:28.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:27 smithi125 conmon[56739]: cluster 2021-03-15T22:15:26.914370+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:27 smithi125 conmon[56739]: audit 2021-03-15T22:15:27.669294+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:27 smithi125 conmon[56739]: audit 2021-03-15T22:15:27.669983+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:27 smithi125 conmon[56739]: audit 2021-03-15T22:15:27.674942+0000 mon.a (mon.0) 156 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:28.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:27 smithi125 conmon[56739]: audit 2021-03-15T22:15:27.675861+0000 mon.a (mon.0) 157 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-15T22:15:28.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:27 smithi125 conmon[56739]: audit 2021-03-15T22:15:27.678863+0000 mon.a (mon.0) 158 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-03-15T22:15:28.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:27 smithi125 conmon[56739]: audit 2021-03-15T22:15:27.679722+0000 mon.a (mon.0) 159 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-15T22:15:28.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:27 smithi125 conmon[56739]: audit 2021-03-15T22:15:27.680654+0000 mon.a (mon.0) 160 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:28.250 DEBUG:teuthology.orchestra.run.smithi125:mgr.x> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@mgr.x.service 2021-03-15T22:15:28.253 INFO:tasks.cephadm:Deploying OSDs... 2021-03-15T22:15:28.254 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:15:28.254 DEBUG:teuthology.orchestra.run.smithi076:> dd if=/scratch_devs of=/dev/stdout 2021-03-15T22:15:28.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: cluster 2021-03-15T22:15:26.914370+0000 mgr.y (mgr. 2021-03-15T22:15:28.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: 14144) 50 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:28.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: audit 2021-03-15T22:15:27.669294+0000 mon.a (mon.0) 2021-03-15T22:15:28.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: 154 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: audit 2021-03-15T22:15:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: 2021-03-15T22:15:27.669983 2021-03-15T22:15:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: +0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:15:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: 15:27.674942+0000 mon.a (mon.0) 156 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:28.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:15:28.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: :15:27.675861+0000 mon.a (mon.0) 157 : audit 2021-03-15T22:15:28.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-15T22:15:28.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: audit 2021 2021-03-15T22:15:28.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: -03-15T22:15:27.678863+0000 mon.a (mon.0) 158 : audit [INF] 2021-03-15T22:15:28.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-03-15T22:15:28.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: audit 2021 2021-03-15T22:15:28.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: -03-15T22:15:27.679722+0000 mon.a (mon.0) 159 : audit 2021-03-15T22:15:28.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-15T22:15:28.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: audit 2021-03-15T22:15:28.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: 2021-03-15T22:15:27. 2021-03-15T22:15:28.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: 680654+0000 mon.a (mon.0) 160 : audit 2021-03-15T22:15:28.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:28.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: cluster 2021-03-15T22:15:26.914370+0000 mgr.y (mgr.14144) 50 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:28.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: audit 2021-03-15T22:15:27.669294+0000 mon.a (mon.0) 154 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:28.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: audit 2021-03-15T22:15:27.669983+0000 mon.a (mon.0) 155 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:28.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: audit 2021-03-15T22:15:27.674942+0000 mon.a (mon.0) 156 : audit [INF] 2021-03-15T22:15:28.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:28.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: audit 2021-03-15T22:15:28.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: 2021-03-15T 2021-03-15T22:15:28.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: 22:15:27.675861+0000 mon.a (mon.0 2021-03-15T22:15:28.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: ) 157 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-15T22:15:28.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: audit 2021 2021-03-15T22:15:28.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: -03-15T22:15:27 2021-03-15T22:15:28.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: .678863+0000 mon.a (mon.0) 158 : 2021-03-15T22:15:28.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2021-03-15T22:15:28.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: audit 2021 2021-03-15T22:15:28.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: -03-15T22:15:27 2021-03-15T22:15:28.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: .679722+0000 mon.a (mon.0 2021-03-15T22:15:28.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: ) 159 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-15T22:15:28.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: audit 2021-03-15T22:15:28.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: 2021-03-15T22:15: 2021-03-15T22:15:28.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: 27.680654+0000 mon.a (mon.0) 160 2021-03-15T22:15:28.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:27 smithi076 conmon[56514]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:28.278 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-15T22:15:28.278 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-15T22:15:28.282 INFO:journalctl@ceph.mgr.x.smithi125.stdout:-- Logs begin at Mon 2021-03-15 21:44:18 UTC. -- 2021-03-15T22:15:28.335 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-15T22:15:28.336 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-15T22:15:28.336 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 103070 Links: 1 2021-03-15T22:15:28.336 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-15T22:15:28.337 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2021-03-15T22:15:28.337 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2021-03-15 22:15:01.810132819 +0000 2021-03-15T22:15:28.337 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2021-03-15 21:49:15.031244048 +0000 2021-03-15T22:15:28.338 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2021-03-15 21:49:15.031244048 +0000 2021-03-15T22:15:28.338 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2021-03-15T22:15:28.347 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2021-03-15T22:15:28.347 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2021-03-15T22:15:28.347 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000232601 s, 2.2 MB/s 2021-03-15T22:15:28.350 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-15T22:15:28.406 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-15T22:15:28.407 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-15T22:15:28.407 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 98242 Links: 1 2021-03-15T22:15:28.407 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-15T22:15:28.407 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2021-03-15T22:15:28.408 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2021-03-15 22:15:01.811132786 +0000 2021-03-15T22:15:28.408 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2021-03-15 21:49:15.378232698 +0000 2021-03-15T22:15:28.408 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2021-03-15 21:49:15.378232698 +0000 2021-03-15T22:15:28.408 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2021-03-15T22:15:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2021-03-15T22:15:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2021-03-15T22:15:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000224178 s, 2.3 MB/s 2021-03-15T22:15:28.421 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-15T22:15:28.477 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-15T22:15:28.478 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-15T22:15:28.478 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 103126 Links: 1 2021-03-15T22:15:28.478 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-15T22:15:28.479 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2021-03-15T22:15:28.479 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2021-03-15 22:15:01.811132786 +0000 2021-03-15T22:15:28.479 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2021-03-15 21:49:15.720221511 +0000 2021-03-15T22:15:28.479 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2021-03-15 21:49:15.720221511 +0000 2021-03-15T22:15:28.480 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2021-03-15T22:15:28.488 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2021-03-15T22:15:28.488 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2021-03-15T22:15:28.489 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.00021977 s, 2.3 MB/s 2021-03-15T22:15:28.491 DEBUG:teuthology.orchestra.run.smithi076:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-15T22:15:28.546 INFO:teuthology.orchestra.run.smithi076.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-15T22:15:28.547 INFO:teuthology.orchestra.run.smithi076.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-15T22:15:28.547 INFO:teuthology.orchestra.run.smithi076.stdout:Device: 6h/6d Inode: 98304 Links: 1 2021-03-15T22:15:28.547 INFO:teuthology.orchestra.run.smithi076.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-15T22:15:28.547 INFO:teuthology.orchestra.run.smithi076.stdout:Context: system_u:object_r:device_t:s0 2021-03-15T22:15:28.548 INFO:teuthology.orchestra.run.smithi076.stdout:Access: 2021-03-15 22:15:01.811132786 +0000 2021-03-15T22:15:28.548 INFO:teuthology.orchestra.run.smithi076.stdout:Modify: 2021-03-15 21:49:16.044210914 +0000 2021-03-15T22:15:28.548 INFO:teuthology.orchestra.run.smithi076.stdout:Change: 2021-03-15 21:49:16.044210914 +0000 2021-03-15T22:15:28.548 INFO:teuthology.orchestra.run.smithi076.stdout: Birth: - 2021-03-15T22:15:28.557 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records in 2021-03-15T22:15:28.558 INFO:teuthology.orchestra.run.smithi076.stderr:1+0 records out 2021-03-15T22:15:28.558 INFO:teuthology.orchestra.run.smithi076.stderr:512 bytes copied, 0.000204741 s, 2.5 MB/s 2021-03-15T22:15:28.560 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-15T22:15:28.561 DEBUG:teuthology.orchestra.run.smithi125:> dd if=/scratch_devs of=/dev/stdout 2021-03-15T22:15:28.576 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-15T22:15:28.576 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-15T22:15:28.631 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-15T22:15:28.631 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-15T22:15:28.631 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 6h/6d Inode: 97151 Links: 1 2021-03-15T22:15:28.632 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-15T22:15:28.632 INFO:teuthology.orchestra.run.smithi125.stdout:Context: system_u:object_r:device_t:s0 2021-03-15T22:15:28.632 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2021-03-15 22:15:14.681875286 +0000 2021-03-15T22:15:28.632 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2021-03-15 21:49:14.921049577 +0000 2021-03-15T22:15:28.633 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2021-03-15 21:49:14.921049577 +0000 2021-03-15T22:15:28.633 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2021-03-15T22:15:28.641 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2021-03-15T22:15:28.641 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2021-03-15T22:15:28.642 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000209853 s, 2.4 MB/s 2021-03-15T22:15:28.644 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-15T22:15:28.699 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-15T22:15:28.700 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-15T22:15:28.700 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 6h/6d Inode: 105029 Links: 1 2021-03-15T22:15:28.700 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-15T22:15:28.700 INFO:teuthology.orchestra.run.smithi125.stdout:Context: system_u:object_r:device_t:s0 2021-03-15T22:15:28.700 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2021-03-15 22:15:14.681875286 +0000 2021-03-15T22:15:28.701 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2021-03-15 21:49:15.246044258 +0000 2021-03-15T22:15:28.701 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2021-03-15 21:49:15.246044258 +0000 2021-03-15T22:15:28.701 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2021-03-15T22:15:28.709 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2021-03-15T22:15:28.710 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2021-03-15T22:15:28.710 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000188211 s, 2.7 MB/s 2021-03-15T22:15:28.712 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-15T22:15:28.768 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-15T22:15:28.768 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-15T22:15:28.768 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 6h/6d Inode: 100893 Links: 1 2021-03-15T22:15:28.769 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-15T22:15:28.769 INFO:teuthology.orchestra.run.smithi125.stdout:Context: system_u:object_r:device_t:s0 2021-03-15T22:15:28.769 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2021-03-15 22:15:14.681875286 +0000 2021-03-15T22:15:28.770 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2021-03-15 21:49:15.582038760 +0000 2021-03-15T22:15:28.770 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2021-03-15 21:49:15.582038760 +0000 2021-03-15T22:15:28.770 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2021-03-15T22:15:28.778 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2021-03-15T22:15:28.778 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2021-03-15T22:15:28.779 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000217987 s, 2.3 MB/s 2021-03-15T22:15:28.781 DEBUG:teuthology.orchestra.run.smithi125:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-15T22:15:28.837 INFO:teuthology.orchestra.run.smithi125.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-15T22:15:28.837 INFO:teuthology.orchestra.run.smithi125.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-15T22:15:28.838 INFO:teuthology.orchestra.run.smithi125.stdout:Device: 6h/6d Inode: 97270 Links: 1 2021-03-15T22:15:28.838 INFO:teuthology.orchestra.run.smithi125.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-15T22:15:28.838 INFO:teuthology.orchestra.run.smithi125.stdout:Context: system_u:object_r:device_t:s0 2021-03-15T22:15:28.838 INFO:teuthology.orchestra.run.smithi125.stdout:Access: 2021-03-15 22:15:14.681875286 +0000 2021-03-15T22:15:28.839 INFO:teuthology.orchestra.run.smithi125.stdout:Modify: 2021-03-15 21:49:15.909033411 +0000 2021-03-15T22:15:28.839 INFO:teuthology.orchestra.run.smithi125.stdout:Change: 2021-03-15 21:49:15.909033411 +0000 2021-03-15T22:15:28.839 INFO:teuthology.orchestra.run.smithi125.stdout: Birth: - 2021-03-15T22:15:28.850 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records in 2021-03-15T22:15:28.851 INFO:teuthology.orchestra.run.smithi125.stderr:1+0 records out 2021-03-15T22:15:28.851 INFO:teuthology.orchestra.run.smithi125.stderr:512 bytes copied, 0.000220532 s, 2.3 MB/s 2021-03-15T22:15:28.853 INFO:tasks.cephadm:Deploying osd.0 on smithi076 with /dev/vg_nvme/lv_4... 2021-03-15T22:15:28.853 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-03-15T22:15:29.207 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:28 smithi125 conmon[56739]: audit 2021-03-15T22:15:27.665386+0000 mgr.y (mgr.14144 2021-03-15T22:15:29.207 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:28 smithi125 conmon[56739]: ) 51 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi076=y;smithi125=x", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:29.207 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:28 smithi125 conmon[56739]: cephadm 2021-03-15T22:15:27.666191+0000 mgr.y (mgr.14144) 52 : cephadm [INF] Saving service mgr spec with placement smithi076=y;smithi125=x;count:2 2021-03-15T22:15:29.207 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:28 smithi125 conmon[56739]: cephadm 2021-03-15T22:15:27.671792+0000 mgr.y (mgr.14144) 53 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:29.208 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:28 smithi125 conmon[56739]: cephadm 2021-03-15T22:15:27.681440+0000 mgr.y (mgr.14144) 54 : cephadm [INF] Deploying daemon mgr.x on smithi125 2021-03-15T22:15:29.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[56514]: audit 2021-03-15T22:15:27.665386+0000 mgr.y (mgr.14144) 51 : 2021-03-15T22:15:29.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[56514]: audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi076=y;smithi125=x", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:29.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:27.666191+0000 mgr.y (mgr.14144) 52 : cephadm [INF] Saving service mgr spec with placement smithi076=y;smithi125=x;count:2 2021-03-15T22:15:29.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:29.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[56514]: 2021-03-15T22:15:27.671792+0000 mgr.y (mgr.14144) 53 : cephadm 2021-03-15T22:15:29.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[56514]: [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:29.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:15:29.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[56514]: -15T22:15:27.681440 2021-03-15T22:15:29.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 54 : cephadm 2021-03-15T22:15:29.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[56514]: [INF] Deploying daemon mgr.x on smithi125 2021-03-15T22:15:29.258 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[62156]: audit 2021-03-15T22:15:27.665386+0000 2021-03-15T22:15:29.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[62156]: mgr.y (mgr.14144) 51 : audit [DBG] from='client.34103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi076=y;smithi125=x", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:29.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:29.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[62156]: 2021-03-15T22:15:27.666191+0000 mgr.y (mgr.14144) 52 : cephadm 2021-03-15T22:15:29.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[62156]: [INF] Saving service mgr spec with placement smithi076=y;smithi125=x;count:2 2021-03-15T22:15:29.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[62156]: cephadm 2021-03-15T 2021-03-15T22:15:29.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[62156]: 22:15:27.671792+0000 mgr.y (mgr.14144 2021-03-15T22:15:29.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[62156]: ) 53 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:29.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:15:29.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[62156]: 03-15T22:15:27.681440+0000 mgr.y (mgr 2021-03-15T22:15:29.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:28 smithi076 conmon[62156]: .14144) 54 : cephadm [INF] Deploying daemon mgr.x on smithi125 2021-03-15T22:15:29.536 INFO:journalctl@ceph.mgr.x.smithi125.stdout:Mar 15 22:15:29 smithi125 systemd[1]: Starting Ceph mgr.x for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:15:29.973 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:29 smithi125 conmon[56739]: cluster 2021-03-15T22:15:28.914670+0000 2021-03-15T22:15:29.973 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:29 smithi125 conmon[56739]: mgr.y (mgr.14144) 55 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:29.974 INFO:journalctl@ceph.mgr.x.smithi125.stdout:Mar 15 22:15:29 smithi125 podman[57947]: 2021-03-15 22:15:29.713531133 +0000 UTC m=+0.173068601 container create ce4650b1392c44957fb849d437b65de9518538fa58299a1a928fd91231dcdcd4 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mgr.x, org.label-schema.schema-version=1.0, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, ceph=True, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2021-03-15T22:15:29.974 INFO:journalctl@ceph.mgr.x.smithi125.stdout:Mar 15 22:15:29 smithi125 podman[57947]: 2021-03-15 22:15:29.971898918 +0000 UTC m=+0.431436501 container init ce4650b1392c44957fb849d437b65de9518538fa58299a1a928fd91231dcdcd4 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mgr.x, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, GIT_CLEAN=True, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.license=GPLv2, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2021-03-15T22:15:30.230 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:29 smithi076 conmon[56514]: cluster 2021-03-15T22:15:28.914670+0000 mgr.y (mgr.14144) 55 : 2021-03-15T22:15:30.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:29 smithi076 conmon[56514]: cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:30.231 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:29 smithi076 conmon[62156]: cluster 2021-03-15T22:15:28.914670+0000 mgr.y ( 2021-03-15T22:15:30.231 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:29 smithi076 conmon[62156]: mgr.14144) 55 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:30.232 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:30 smithi076 conmon[56844]: debug 2021-03-15T22:15:30.084+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:30.232 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:30 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:15:30.232 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:30 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:30.233 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:30 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:15:30.233 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:30 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:30.233 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:30 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:15:30.233 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:30 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:30.233 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:30 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:30.234 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:30 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:30.286 INFO:journalctl@ceph.mgr.x.smithi125.stdout:Mar 15 22:15:30 smithi125 conmon[57988]: debug 2021-03-15T22:15:30.003+0000 7f2c03185700 1 -- 172.21.15.125:0/857554858 <== mon.2 v2:172.21.15.125:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557dbe4044e0 con 0x557dbe3dd000 2021-03-15T22:15:30.286 INFO:journalctl@ceph.mgr.x.smithi125.stdout:Mar 15 22:15:30 smithi125 podman[57947]: 2021-03-15 22:15:30.013442244 +0000 UTC m=+0.472979718 container start ce4650b1392c44957fb849d437b65de9518538fa58299a1a928fd91231dcdcd4 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-mgr.x, org.label-schema.name=CentOS Base Image, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_BRANCH=HEAD, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, ceph=True, io.buildah.version=1.18.0, org.label-schema.build-date=20201204, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Dimitri Savineau ) 2021-03-15T22:15:30.287 INFO:journalctl@ceph.mgr.x.smithi125.stdout:Mar 15 22:15:30 smithi125 bash[57874]: ce4650b1392c44957fb849d437b65de9518538fa58299a1a928fd91231dcdcd4 2021-03-15T22:15:30.287 INFO:journalctl@ceph.mgr.x.smithi125.stdout:Mar 15 22:15:30 smithi125 systemd[1]: Started Ceph mgr.x for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:15:30.728 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-03-15T22:15:30.728 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-03-15T22:15:30.728 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2021-03-15T22:15:30.728 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2021-03-15T22:15:30.729 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0107803 s, 973 MB/s 2021-03-15T22:15:30.729 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2021-03-15T22:15:31.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: audit 2021-03-15T22:15:30.083758+0000 2021-03-15T22:15:31.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: mon.a (mon.0) 161 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:31.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:30.085120+0000 mgr.y (mgr.14144) 56 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:31.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:15:31.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:31.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:15:31.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:31.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:15:31.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:31.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:31.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:31.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:30.085285+0000 mgr.y (mgr.14144) 57 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2021-03-15T22:15:31.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: audit 2021-03-15T22:15:30.085540+0000 mon.a (mon.0) 162 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-15T22:15:31.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: audit 2021-03-15T22:15:30.086361+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-15T22:15:31.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: audit 2021-03-15T22:15:30.087091+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:31.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:30.087631+0000 mgr.y (mgr.14144) 58 : cephadm [INF] Reconfiguring daemon mgr.y on smithi076 2021-03-15T22:15:31.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[56514]: cluster 2021-03-15T22:15:30.914920+0000 mgr.y (mgr.14144) 59 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:31.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: audit 2021-03-15T22:15:30.083758+0000 mon.a ( 2021-03-15T22:15:31.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: mon.0) 161 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:31.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:30.085120+0000 mgr.y (mgr.14144) 56 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:31.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:15:31.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:31.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:15:31.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:31.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:15:31.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:31.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:31.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:31.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:30.085285+0000 mgr.y (mgr.14144) 57 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2021-03-15T22:15:31.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: audit 2021-03-15T22:15:30.085540+0000 mon.a (mon.0) 162 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-15T22:15:31.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: audit 2021-03-15T22:15:30.086361+0000 mon.a (mon.0) 163 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-15T22:15:31.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: audit 2021-03-15T22:15:30.087091+0000 mon.a (mon.0) 164 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:31.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:30.087631+0000 mgr.y (mgr.14144) 58 : cephadm [INF] Reconfiguring daemon mgr.y on smithi076 2021-03-15T22:15:31.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:31 smithi076 conmon[62156]: cluster 2021-03-15T22:15:30.914920+0000 mgr.y (mgr.14144) 59 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:31.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: audit 2021-03-15T22:15:30.083758+0000 mon.a (mon.0) 161 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: cephadm 2021-03-15T22:15:30.085120+0000 mgr.y (mgr.14144) 56 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:15:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:15:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:15:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: cephadm 2021-03-15T22:15:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: 2021-03-15T22:15:30.085285+0000 mgr.y (mgr.14144) 57 : cephadm [INF] Reconfiguring mgr.y (unknown last config time)... 2021-03-15T22:15:31.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:15:31.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: 15:30.085540+0000 mon.a (mon.0) 162 : audit 2021-03-15T22:15:31.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2021-03-15T22:15:31.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:15:31.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: 15T22:15:30.086361+0000 2021-03-15T22:15:31.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: mon.a (mon.0) 163 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2021-03-15T22:15:31.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: 2021-03-15T22:15:31.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: audit 2021-03-15T22:15:31.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: 2021-03-15T22:15:30. 2021-03-15T22:15:31.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: 087091+0000 mon.a (mon.0) 164 : audit 2021-03-15T22:15:31.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:31.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: cephadm 2021-03- 2021-03-15T22:15:31.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: 15T22:15:30.087631 2021-03-15T22:15:31.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 58 : cephadm 2021-03-15T22:15:31.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: [INF] Reconfiguring daemon mgr.y on smithi076 2021-03-15T22:15:31.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:15:31.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: 15T22:15:30.914920+0000 2021-03-15T22:15:31.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: mgr.y (mgr.14144) 59 : cluster [DBG] 2021-03-15T22:15:31.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:31 smithi125 conmon[56739]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:31.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch daemon add osd smithi076:vg_nvme/lv_4 2021-03-15T22:15:32.897 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56844]: debug 2021-03-15T22:15:32.894+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:32.897 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:15:33.171 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:33.172 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:15:33.172 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:33.172 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:15:33.173 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:33.173 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:33.173 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:33.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: audit 2021-03-15T22:15:31.984676+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:33.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: audit 2021-03-15T22:15:31.985386+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:33.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: audit 2021-03-15T22:15:33.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: 2021-03-15T22:15:32.889287+0000 mon.a ( 2021-03-15T22:15:33.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: mon.0) 167 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:33.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:15:33.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: -15T22:15:32.889740+0000 mgr.y 2021-03-15T22:15:33.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: (mgr.14144) 60 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:33.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: 2021-03-15T22:15:33.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: audit 2021-03-15T22:15:33.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: 2021-03-15T22:15:32.894369+0000 mon.a ( 2021-03-15T22:15:33.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: mon.0) 168 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:33.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: cephadm 2021-03- 2021-03-15T22:15:33.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: 15T22:15:32.895087+0000 2021-03-15T22:15:33.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: mgr.y (mgr.14144) 61 : cephadm 2021-03-15T22:15:33.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:33.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:15:33.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:33.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:15:33.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:33.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:15:33.179 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:33.179 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:33.179 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:33.179 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[56514]: cluster 2021-03-15T22:15:32.915151+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:33.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: audit 2021-03-15T22:15:31.984676+0000 mon.a ( 2021-03-15T22:15:33.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: mon.0) 165 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:33.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: audit 2021-03-15T22:15:31.985386+0000 mon.a (mon.0) 2021-03-15T22:15:33.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: 166 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:33.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:15:33.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: 03-15T22:15:32.889287+0000 mon.a (mon.0) 167 2021-03-15T22:15:33.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:33.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: cephadm 2021-03- 2021-03-15T22:15:33.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: 15T22:15:32.889740+0000 mgr.y 2021-03-15T22:15:33.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: (mgr.14144) 60 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:33.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: 2021-03-15T22:15:33.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: audit 2021-03-15T22:15:33.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: 2021-03-15T22:15:32.894369+0000 2021-03-15T22:15:33.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: mon.a (mon.0) 168 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:33.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: 2021-03-15T22:15:33.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: cephadm 2021-03-15T 2021-03-15T22:15:33.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: 22:15:32.895087+0000 mgr.y (mgr.14144) 61 : cephadm 2021-03-15T22:15:33.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:33.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:15:33.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:33.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:15:33.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:33.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:15:33.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:33.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:33.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:33.186 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:32 smithi076 conmon[62156]: cluster 2021-03-15T22:15:32.915151+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:33.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: audit 2021-03-15T22:15:31.984676+0000 mon.a (mon.0) 165 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:33.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: audit 2021-03-15T22:15:31.985386+0000 mon.a (mon.0) 166 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:33.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: audit 2021-03-15T22:15:32.889287+0000 mon.a (mon.0) 167 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:33.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: cephadm 2021-03-15T22:15:32.889740+0000 mgr.y (mgr.14144) 60 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:33.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: audit 2021-03-15T22:15:32.894369+0000 mon.a (mon.0) 168 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:33.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: cephadm 2021-03-15T22:15:32.895087+0000 mgr.y (mgr.14144) 61 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:33.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:15:33.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:33.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:15:33.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:33.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:15:33.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:33.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:33.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:33.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:32 smithi125 conmon[56739]: cluster 2021-03-15T22:15:32.915151+0000 mgr.y (mgr.14144) 62 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[56514]: audit 2021-03-15T22:15:33.769472+0000 mgr.y (mgr.14144) 2021-03-15T22:15:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[56514]: 63 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[56514]: audit 2021-03-15T22:15:33.770229+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:15:34.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:33.771343+0000 mgr.y (mgr.14144) 64 : cephadm [INF] Applying service osd.None on host smithi076... 2021-03-15T22:15:34.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[56514]: audit 2021-03-15T22:15:33.771558+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:15:34.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[56514]: audit 2021-03-15T22:15:33.772024+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:34.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[62156]: audit 2021-03-15T22:15:33.769472 2021-03-15T22:15:34.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 63 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:34.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[62156]: audit 2021-03-15T22:15:33.770229+0000 mon.a (mon.0) 169 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:15:34.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:33.771343+0000 mgr.y (mgr.14144) 64 : cephadm [INF] Applying service osd.None on host smithi076... 2021-03-15T22:15:34.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[62156]: audit 2021-03-15T22:15:33.771558+0000 mon.a (mon.0) 170 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:15:34.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:33 smithi076 conmon[62156]: audit 2021-03-15T22:15:33.772024+0000 mon.a (mon.0) 171 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:34.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: audit 2021-03-15T22:15:33.769472+0000 mgr.y 2021-03-15T22:15:34.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: (mgr.14144) 63 : audit [DBG] from='client.14205 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: audit 2021-03-15T22:15:33.770229+0000 2021-03-15T22:15:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: mon.a (mon.0) 169 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:15:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: cephadm 2021-03-15T22:15:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: 2021-03-15T 2021-03-15T22:15:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: 22:15:33.771343+0000 2021-03-15T22:15:34.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: mgr.y (mgr.14144) 64 : 2021-03-15T22:15:34.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: cephadm [INF] Applying service osd.None on host smithi076... 2021-03-15T22:15:34.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:15:34.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: 03-15T22:15:33.771558+0000 2021-03-15T22:15:34.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: mon.a (mon.0) 170 2021-03-15T22:15:34.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:15:34.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:15:34.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: -15T22:15:33.772024+0000 2021-03-15T22:15:34.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: mon.a (mon.0) 171 : 2021-03-15T22:15:34.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:33 smithi125 conmon[56739]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:35.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[56514]: cluster 2021-03-15T22:15:34.812665+0000 mon.a (mon 2021-03-15T22:15:35.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[56514]: .0) 172 : cluster [DBG] Standby manager daemon x started 2021-03-15T22:15:35.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[56514]: audit 2021-03-15T22:15:34.813483+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-03-15T22:15:35.221 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[56514]: audit 2021-03-15T22:15:34.813753+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-03-15T22:15:35.221 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[56514]: audit 2021-03-15T22:15:34.814710+0000 mon.b (mon.2) 4 : audit [DBG] from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-03-15T22:15:35.221 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[56514]: audit 2021-03-15T22:15:34.814932+0000 mon.b (mon.2) 5 : audit [DBG] from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-03-15T22:15:35.221 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[56514]: cluster 2021-03-15T22:15:34.915359+0000 mgr.y (mgr.14144) 65 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:35.222 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: cluster 2021-03-15T22:15:34.812665 2021-03-15T22:15:35.222 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: +0000 mon.a (mon.0) 172 : cluster [DBG] Standby manager daemon x started 2021-03-15T22:15:35.222 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:15:35.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: 15T22:15:34.813483+0000 mon.b (mon.2) 2 : audit [DBG] 2021-03-15T22:15:35.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-03-15T22:15:35.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: audit 2021-03-15T22:15 2021-03-15T22:15:35.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: :34.813753+0000 mon.b (mon.2) 3 : 2021-03-15T22:15:35.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: audit [DBG] from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-03-15T22:15:35.224 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:15:35.224 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: T22:15:34.814710+0000 mon.b (mon 2021-03-15T22:15:35.224 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: .2) 4 : audit [DBG] from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-03-15T22:15:35.224 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:15:35.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: 03-15T22:15:34.814932+0000 mon.b (mon 2021-03-15T22:15:35.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: .2) 5 : audit [DBG] from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-03-15T22:15:35.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:15:35.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: 22:15:34.915359+0000 mgr.y (mgr.14144) 65 : cluster 2021-03-15T22:15:35.226 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:34 smithi076 conmon[62156]: [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:35.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: cluster 2021-03-15T22:15:34.812665+0000 mon.a (mon.0) 172 : cluster [DBG] Standby manager daemon x started 2021-03-15T22:15:35.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: audit 2021-03-15T22:15:34.813483+0000 mon.b (mon.2) 2 : audit [DBG] from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2021-03-15T22:15:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: audit 2021-03-15T22:15:34. 2021-03-15T22:15:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: 813753+0000 mon.b (mon.2) 3 : audit [DBG] from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2021-03-15T22:15:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:15:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: 22:15:34.814710+0000 mon.b (mon 2021-03-15T22:15:35.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: .2) 4 : audit [DBG] from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2021-03-15T22:15:35.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: audit 2021 2021-03-15T22:15:35.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: -03-15T22:15:34.814932 2021-03-15T22:15:35.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: +0000 mon.b (mon.2) 5 : audit 2021-03-15T22:15:35.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: [DBG] from='mgr.? 172.21.15.125:0/2053288108' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2021-03-15T22:15:35.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:15:35.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: 15:34.915359+0000 mgr.y (mgr.14144 2021-03-15T22:15:35.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:34 smithi125 conmon[56739]: ) 65 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:36.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[56514]: cluster 2021-03-15T22:15:35.001585+0000 mon.a (mon.0) 173 : cluster [DBG] mgrmap e14: y(active, since 94s), standbys: x 2021-03-15T22:15:36.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[56514]: audit 2021-03-15T22:15:35.001721+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-03-15T22:15:36.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[56514]: audit 2021-03-15T22:15:35.826180+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.76:0/1571792403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd"}]: dispatch 2021-03-15T22:15:36.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[56514]: audit 2021-03-15T22:15:35.826561+0000 mon.a (mon.0) 175 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd"}]: dispatch 2021-03-15T22:15:36.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[56514]: audit 2021-03-15T22:15:35.831425+0000 mon.a (mon.0) 176 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd"}]': finished 2021-03-15T22:15:36.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[56514]: cluster 2021-03-15T22:15:35.831499+0000 mon.a (mon.0) 177 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2021-03-15T22:15:36.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[56514]: audit 2021-03-15T22:15:35.831586+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:36.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[62156]: cluster 2021-03-15T22:15:35.001585+0000 mon.a (mon.0) 173 : cluster [DBG] mgrmap e14: y(active, since 94s), standbys: x 2021-03-15T22:15:36.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[62156]: audit 2021-03-15T22:15:35.001721+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-03-15T22:15:36.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[62156]: audit 2021-03-15T22:15:35.826180+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.76:0/1571792403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd"}]: dispatch 2021-03-15T22:15:36.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[62156]: audit 2021-03-15T22:15:35.826561+0000 mon.a (mon.0) 175 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd"}]: dispatch 2021-03-15T22:15:36.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[62156]: audit 2021-03-15T22:15:35.831425+0000 mon.a (mon.0) 176 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd"}]': finished 2021-03-15T22:15:36.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[62156]: cluster 2021-03-15T22:15:35.831499+0000 mon.a (mon.0) 177 : cluster [DBG] osdmap e5: 1 total, 0 up, 1 in 2021-03-15T22:15:36.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:36 smithi076 conmon[62156]: audit 2021-03-15T22:15:35.831586+0000 mon.a (mon.0) 178 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:36.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: cluster 2021-03-15T22:15:35.001585+0000 mon.a (mon.0) 173 : cluster [DBG] mgrmap e14: y(active, since 94s), standbys: x 2021-03-15T22:15:36.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: audit 2021-03-15T22:15:35.001721+0000 mon.a (mon.0) 174 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2021-03-15T22:15:36.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: audit 2021-03-15T22:15:35.826180+0000 mon.c (mon.1) 4 : audit [INF] from='client.? 172.21.15.76:0/1571792403' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd"}]: dispatch 2021-03-15T22:15:36.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: 2021-03-15T22:15:36.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: audit 2021-03-15T22:15 2021-03-15T22:15:36.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: :35.826561+0000 mon.a (mon.0) 175 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd"}]: dispatch 2021-03-15T22:15:36.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: 2021-03-15T22:15:36.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: audit 2021-03-15T22:15:36.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: 2021-03-15T22:15:35.831425+0000 mon.a 2021-03-15T22:15:36.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: (mon.0) 176 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd"}]': finished 2021-03-15T22:15:36.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: cluster 2021-03-15T22:15:36.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: 2021-03-15T22:15:35.831499 2021-03-15T22:15:36.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: +0000 mon.a (mon.0) 177 : cluster [DBG] 2021-03-15T22:15:36.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: osdmap e5: 1 total, 0 up, 1 in 2021-03-15T22:15:36.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: audit 2021-03-15T22:15 2021-03-15T22:15:36.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: :35.831586+0000 mon.a (mon.0) 178 2021-03-15T22:15:36.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:36 smithi125 conmon[56739]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:37.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:37 smithi076 conmon[56514]: audit 2021-03-15T22:15:36.304290+0000 mon.a (mon.0) 179 2021-03-15T22:15:37.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:37 smithi076 conmon[56514]: : audit [DBG] from='client.? 172.21.15.76:0/2964528539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:15:37.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:37 smithi076 conmon[56514]: cluster 2021-03-15T22:15:36.915745+0000 mgr.y (mgr.14144) 66 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:37.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:37 smithi076 conmon[62156]: audit 2021-03-15T22:15:36.304290+0000 mon.a ( 2021-03-15T22:15:37.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:37 smithi076 conmon[62156]: mon.0) 179 : audit [DBG] from='client.? 172.21.15.76:0/2964528539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:15:37.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:37 smithi076 conmon[62156]: cluster 2021-03-15T22:15 2021-03-15T22:15:37.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:37 smithi076 conmon[62156]: :36.915745+0000 mgr.y (mgr.14144) 66 : cluster [DBG] 2021-03-15T22:15:37.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:37 smithi076 conmon[62156]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:37.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:37 smithi125 conmon[56739]: audit 2021-03-15T22:15:36.304290+0000 mon.a (mon.0) 179 2021-03-15T22:15:37.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:37 smithi125 conmon[56739]: : audit [DBG] from='client.? 172.21.15.76:0/2964528539' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:15:37.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:37 smithi125 conmon[56739]: cluster 2021-03-15T22:15:36.915745+0000 mgr.y (mgr.14144) 2021-03-15T22:15:37.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:37 smithi125 conmon[56739]: 66 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:39.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:39 smithi076 conmon[56514]: cluster 2021-03-15T22:15:38.916134+0000 mgr.y (mgr.14144) 67 : cluster [DBG] 2021-03-15T22:15:39.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:39 smithi076 conmon[56514]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:39.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:39 smithi076 conmon[62156]: cluster 2021-03-15T22:15:38.916134 2021-03-15T22:15:39.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:39 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:39.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:39 smithi125 conmon[56739]: cluster 2021-03-15T22:15:38.916134+0000 mgr.y (mgr.14144) 67 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:42.129 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:41 smithi076 conmon[56514]: cluster 2021-03-15T22:15:40.916448+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:42.130 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:41 smithi076 conmon[56514]: audit 2021-03-15T22:15:41.359393+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-03-15T22:15:42.130 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:41 smithi076 conmon[56514]: audit 2021-03-15T22:15:41.359958+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:42.130 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:41 smithi076 conmon[62156]: cluster 2021-03-15T22:15:40 2021-03-15T22:15:42.130 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:41 smithi076 conmon[62156]: .916448+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:42.131 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:41 smithi076 conmon[62156]: audit 2021-03-15T22:15:41.359393+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-03-15T22:15:42.131 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:41 smithi076 conmon[62156]: audit 2021-03-15T22:15:41.359958+0000 mon.a (mon.0) 181 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:42.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:41 smithi125 conmon[56739]: cluster 2021-03-15T22:15:40.916448+0000 mgr.y (mgr.14144) 68 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:42.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:41 smithi125 conmon[56739]: audit 2021-03-15T22:15:41.359393+0000 mon.a (mon.0) 180 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2021-03-15T22:15:42.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:41 smithi125 conmon[56739]: audit 2021-03-15T22:15:42.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:41 smithi125 conmon[56739]: 2021-03-15T22:15:41.359958+0000 mon.a ( 2021-03-15T22:15:42.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:41 smithi125 conmon[56739]: mon.0) 181 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:43.205 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:42 smithi076 conmon[56514]: cephadm 2021-03-15T22 2021-03-15T22:15:43.205 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:42 smithi076 conmon[56514]: :15:41.360450+0000 mgr.y (mgr.14144) 69 : cephadm [INF] Deploying daemon osd.0 on smithi076 2021-03-15T22:15:43.206 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:42 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:41.360450+0000 mgr.y ( 2021-03-15T22:15:43.206 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:42 smithi076 conmon[62156]: mgr.14144) 69 : cephadm [INF] Deploying daemon osd.0 on smithi076 2021-03-15T22:15:43.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:42 smithi125 conmon[56739]: cephadm 2021-03-15T22:15:41.360450+0000 mgr.y (mgr 2021-03-15T22:15:43.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:42 smithi125 conmon[56739]: .14144) 69 : cephadm [INF] Deploying daemon osd.0 on smithi076 2021-03-15T22:15:44.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:43 smithi076 conmon[56514]: cluster 2021-03-15T22:15:42.916715+0000 2021-03-15T22:15:44.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:43 smithi076 conmon[56514]: mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:44.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:43 smithi076 conmon[62156]: cluster 2021-03-15T22:15:42. 2021-03-15T22:15:44.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:43 smithi076 conmon[62156]: 916715+0000 mgr.y (mgr.14144) 70 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:44.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:43 smithi125 conmon[56739]: cluster 2021-03-15T22:15:42.916715+0000 mgr.y (mgr.14144) 70 : cluster 2021-03-15T22:15:44.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:43 smithi125 conmon[56739]: [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:44.636 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 0 on host 'smithi076' 2021-03-15T22:15:45.187 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:44 smithi076 conmon[56514]: audit 2021-03-15T22:15:44.625587+0000 mon.a (mon.0 2021-03-15T22:15:45.187 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:44 smithi076 conmon[56514]: ) 182 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:45.187 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:44 smithi076 conmon[56514]: audit 2021-03-15T22:15:44.629260+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:45.188 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:44 smithi076 conmon[56514]: audit 2021-03-15T22:15:44.633064+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:45.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:44 smithi076 conmon[62156]: audit 2021-03-15T22:15:44.625587+0000 mon.a 2021-03-15T22:15:45.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:44 smithi076 conmon[62156]: (mon.0) 182 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:45.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:44 smithi076 conmon[62156]: audit 2021-03-15T22:15:44.629260+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:45.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:44 smithi076 conmon[62156]: audit 2021-03-15T22:15:44.633064+0000 mon.a (mon.0) 184 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:45.210 DEBUG:teuthology.orchestra.run.smithi076:osd.0> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.0.service 2021-03-15T22:15:45.212 INFO:tasks.cephadm:Deploying osd.1 on smithi076 with /dev/vg_nvme/lv_3... 2021-03-15T22:15:45.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-03-15T22:15:45.240 INFO:journalctl@ceph.osd.0.smithi076.stdout:-- Logs begin at Mon 2021-03-15 21:44:23 UTC. -- 2021-03-15T22:15:45.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:44 smithi125 conmon[56739]: audit 2021-03-15T22:15:44.625587+0000 mon.a (mon.0) 182 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:45.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:44 smithi125 conmon[56739]: 2021-03-15T22:15:45.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:44 smithi125 conmon[56739]: audit 2021-03-15T22:15:44.629260+0000 mon.a (mon.0) 183 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:45.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:44 smithi125 conmon[56739]: audit 2021-03-15T22:15:44.633064+0000 2021-03-15T22:15:45.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:44 smithi125 conmon[56739]: mon.a (mon.0) 184 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:45.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:44 smithi125 conmon[56739]: 2021-03-15T22:15:46.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:45 smithi076 conmon[56514]: cluster 2021-03-15T22:15:44.916986+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:46.045 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:45 smithi076 conmon[62156]: cluster 2021-03-15T22:15:44.916986+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:46.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:45 smithi125 conmon[56739]: cluster 2021-03-15T22:15:44.916986+0000 mgr.y (mgr.14144) 71 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:47.145 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:46 smithi076 conmon[56514]: audit 2021-03-15T22:15:45 2021-03-15T22:15:47.145 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:46 smithi076 conmon[56514]: .992175+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:47.146 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:46 smithi076 conmon[56514]: cluster 2021-03-15T22:15:46.917205+0000 mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:47.146 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:46 smithi076 conmon[62156]: audit 2021-03-15T22:15:45.992175+0000 2021-03-15T22:15:47.146 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:46 smithi076 conmon[62156]: mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:47.146 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:46 smithi076 conmon[62156]: cluster 2021-03-15T22:15:46.917205+0000 2021-03-15T22:15:47.147 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:46 smithi076 conmon[62156]: mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:47.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:46 smithi125 conmon[56739]: audit 2021-03-15T22:15:45.992175+0000 mon.a (mon.0) 185 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:47.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:46 smithi125 conmon[56739]: cluster 2021-03-15T22:15:46.917205+0000 mgr.y (mgr.14144) 72 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:47.621 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-03-15T22:15:47.622 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-03-15T22:15:47.622 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2021-03-15T22:15:47.622 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2021-03-15T22:15:47.622 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0122573 s, 855 MB/s 2021-03-15T22:15:47.623 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2021-03-15T22:15:47.996 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:15:47 smithi076 conmon[65511]: debug 2021-03-15T22:15:47.813+0000 7f1a4b37b080 -1 osd.0 0 log_to_monitors {default=true} 2021-03-15T22:15:48.153 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch daemon add osd smithi076:vg_nvme/lv_3 2021-03-15T22:15:48.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:47 smithi076 conmon[56514]: audit 2021-03-15T22:15: 2021-03-15T22:15:48.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:47 smithi076 conmon[56514]: 47.816680+0000 mon.a (mon.0) 186 : audit [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-15T22:15:48.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:47 smithi076 conmon[62156]: audit 2021-03-15T22:15:47.816680+0000 2021-03-15T22:15:48.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:47 smithi076 conmon[62156]: mon.a (mon.0) 186 : audit [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-15T22:15:48.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:47 smithi125 conmon[56739]: audit 2021-03-15T22:15:47.816680+0000 mon.a (mon.0) 186 : audit [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2021-03-15T22:15:48.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:47 smithi125 conmon[56739]: 2021-03-15T22:15:49.267 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[65511]: debug 2021-03-15T22:15:49.004+0000 7f1a418ec700 -1 osd.0 0 waiting for initial osdmap 2021-03-15T22:15:49.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[56514]: audit 2021-03-15T22:15:48.000719+0000 mon.a (mon.0) 187 : audit [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-03-15T22:15:49.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[56514]: cluster 2021-03-15T22:15:48.000771+0000 mon.a (mon.0) 188 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-03-15T22:15:49.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[56514]: audit 2021-03-15T22:15:48.000860+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:49.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[56514]: audit 2021-03-15T22:15:48.001198+0000 mon.a (mon.0) 190 : audit [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:15:49.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[56514]: cluster 2021-03-15T22:15:48.917434+0000 mgr.y (mgr.14144) 73 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:49.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: audit 2021-03-15T22:15:48.000719 2021-03-15T22:15:49.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: +0000 mon.a (mon.0) 187 : audit [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-03-15T22:15:49.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: cluster 2021-03-15T22:15:49.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: 2021-03-15T22:15:48.000771+0000 mon.a (mon.0) 188 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-03-15T22:15:49.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:15:49.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: -15T22:15:48.000860+0000 mon.a (mon.0) 2021-03-15T22:15:49.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: 189 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:49.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:15:49.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: 03-15T22:15:48 2021-03-15T22:15:49.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: .001198+0000 mon.a (mon.0) 190 : 2021-03-15T22:15:49.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: audit [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:15:49.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:15:49.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: -15T22:15:48. 2021-03-15T22:15:49.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: 917434+0000 mgr.y (mgr.14144) 73 : cluster 2021-03-15T22:15:49.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:49 smithi076 conmon[62156]: [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:49.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: audit 2021-03-15T22:15:48.000719+0000 mon.a (mon.0) 187 : audit [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2021-03-15T22:15:49.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:15:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: 03-15T22:15:48.000771+0000 mon.a (mon.0) 188 : cluster [DBG] osdmap e6: 1 total, 0 up, 1 in 2021-03-15T22:15:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: audit 2021-03-15T22:15:48. 2021-03-15T22:15:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: 000860+0000 mon.a (mon.0) 189 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:15:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: 03-15T22:15:48. 2021-03-15T22:15:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: 001198+0000 mon.a (mon.0) 190 : audit 2021-03-15T22:15:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:15:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:15:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: -15T22:15:48.917434 2021-03-15T22:15:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 73 : cluster [DBG] 2021-03-15T22:15:49.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:49 smithi125 conmon[56739]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:50.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56514]: audit 2021-03-15T22:15:49.001966+0000 mon.a ( 2021-03-15T22:15:50.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56514]: mon.0) 191 : audit [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:15:50.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56514]: cluster 2021-03-15T22:15:49.002044+0000 mon.a (mon.0) 192 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-03-15T22:15:50.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56514]: audit 2021-03-15T22:15:49.002393+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:50.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56514]: audit 2021-03-15T22:15:49.005952+0000 mon.a (mon.0) 194 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:50.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[62156]: audit 2021-03-15T22:15:49. 2021-03-15T22:15:50.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[62156]: 001966+0000 mon.a (mon.0) 191 : audit [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:15:50.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[62156]: cluster 2021-03-15T22:15:49.002044+0000 mon.a (mon.0) 192 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-03-15T22:15:50.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[62156]: audit 2021-03-15T22:15:49.002393+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:50.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[62156]: audit 2021-03-15T22:15:49.005952+0000 mon.a (mon.0) 194 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:50.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:50 smithi125 conmon[56739]: audit 2021-03-15T22:15:49.001966+0000 2021-03-15T22:15:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:50 smithi125 conmon[56739]: mon.a (mon.0) 191 : audit [INF] from='osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:15:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:50 smithi125 conmon[56739]: cluster 2021-03-15T22:15: 2021-03-15T22:15:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:50 smithi125 conmon[56739]: 49.002044+0000 mon.a (mon.0) 192 : cluster [DBG] osdmap e7: 1 total, 0 up, 1 in 2021-03-15T22:15:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:50 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:15:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:50 smithi125 conmon[56739]: 15:49.002393+0000 mon.a (mon.0) 193 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:50.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:50 smithi125 conmon[56739]: audit 2021-03-15T22:15:49.005952+0000 2021-03-15T22:15:50.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:50 smithi125 conmon[56739]: mon.a (mon.0) 194 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:50.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: debug 2021-03-15T22:15:50.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: 2021-03-15T22:15:50.189+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:15:50.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:15:50.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:50.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:15:50.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:50.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:15:50.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:50.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:50.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:15:50.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: debug 2021-03-15T22:15:50.197+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:50.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:15:50.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:50.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:15:50.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:50.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:15:50.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:50.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:50.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:15:50 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:51.155 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: cluster 2021-03-15T22:15:48.855526+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:15:51.155 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: cluster 2021-03-15T22:15:48.855719+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:15:51.156 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: cluster 2021-03-15T22:15:50.007028+0000 mon.a (mon.0) 195 : cluster [INF] osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176] boot 2021-03-15T22:15:51.156 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: cluster 2021-03-15T22:15:50.007062+0000 mon.a (mon.0) 196 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-03-15T22:15:51.156 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: audit 2021-03-15T22:15:50.007557+0000 mon.a (mon.0) 197 : audit 2021-03-15T22:15:51.156 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:51.157 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:50.180011+0000 mgr.y (mgr.14144) 74 : cephadm [INF] Detected new or changed devices on smithi076 2021-03-15T22:15:51.157 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: audit 2021-03-15T22:15: 2021-03-15T22:15:51.157 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 50.184013+0000 mon.a (mon. 2021-03-15T22:15:51.157 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 0) 198 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:51.158 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 2021-03-15T22:15:51.158 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:15:51.158 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: -15T22:15:50.184532 2021-03-15T22:15:51.158 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 75 2021-03-15T22:15:51.159 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:51.159 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: audit 2021-03-15T22:15:51.159 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 2021-03-15T22 2021-03-15T22:15:51.159 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: :15:50.189532+0000 2021-03-15T22:15:51.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: mon.a (mon.0) 199 : audit 2021-03-15T22:15:51.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:51.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:51.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:15:51.160 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 15:50.190400+0000 mgr.y ( 2021-03-15T22:15:51.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: mgr.14144) 76 : cephadm [ERR] 2021-03-15T22:15:51.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:15:51.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:15:51.161 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:51.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:15:51.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:51.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:15:51.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:51.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:51.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:15:51.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: audit 2021-03-15T22:15:50.190930+0000 mon.a (mon.0) 200 : audit [DBG] 2021-03-15T22:15:51.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:51.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:15:51.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: -15T22:15:50. 2021-03-15T22:15:51.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 192268+0000 mgr.y (mgr.14144 2021-03-15T22:15:51.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: ) 77 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:51.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: cluster 2021-03-15T22:15:48.855526+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:15:51.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: cluster 2021-03-15T22:15:48.855719+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:15:51.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: cluster 2021-03-15T22:15:50.007028+0000 mon.a (mon.0) 195 : cluster [INF] osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176] boot 2021-03-15T22:15:51.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: cluster 2021-03-15T22:15:50.007062+0000 mon.a (mon.0) 196 : cluster [DBG] osdmap e8: 1 total, 1 up, 1 in 2021-03-15T22:15:51.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: audit 2021-03-15T22:15:50.007557+0000 mon.a (mon.0) 197 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:51.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: cephadm 2021-03 2021-03-15T22:15:51.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: -15T22:15:50.180011+0000 mgr.y (mgr. 2021-03-15T22:15:51.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 14144) 74 : cephadm [INF] Detected new or changed devices on smithi076 2021-03-15T22:15:51.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: audit 2021 2021-03-15T22:15:51.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: -03-15T22:15:50.184013+0000 2021-03-15T22:15:51.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: mon.a (mon.0) 198 : audit [INF] 2021-03-15T22:15:51.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:51.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:51.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 2021- 2021-03-15T22:15:51.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 03-15T22:15:50.184532+0000 2021-03-15T22:15:51.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: mgr.y (mgr.14144) 75 : cephadm [INF] 2021-03-15T22:15:51.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:51.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: audit 2021-03-15T22:15:51.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:15:51.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 15T22:15:50.189532+0000 mon.a ( 2021-03-15T22:15:51.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: mon.0) 199 : audit [INF] 2021-03-15T22:15:51.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:51.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:51.170 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:15:51.170 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 15T22:15:50.190400+0000 mgr.y 2021-03-15T22:15:51.170 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: (mgr.14144) 76 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:15:51.170 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:15:51.171 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:51.171 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:15:51.171 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:51.171 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:15:51.172 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:51.172 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:51.172 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:15:51.172 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: audit 2021-03-15T22:15:50.190930+0000 mon.a (mon.0) 200 : audit [DBG] 2021-03-15T22:15:51.173 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:51.173 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:51.173 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:15:51.173 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 15T22:15:50.192268+0000 mgr.y ( 2021-03-15T22:15:51.174 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: mgr.14144) 77 : cephadm [INF] 2021-03-15T22:15:51.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: audit 2021-03-15T22:15:51.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 2021-03-15T22:15:50.197634+0000 2021-03-15T22:15:51.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: mon.a (mon.0) 201 : audit 2021-03-15T22:15:51.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:51.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:51.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:15:51.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 15:50.198255+0000 mgr.y 2021-03-15T22:15:51.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: (mgr.14144) 78 : cephadm 2021-03-15T22:15:51.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:51.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:15:51.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:51.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:15:51.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:51.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:15:51.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:51.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:51.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:51.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: audit 2021-03-15T22:15:50.493549+0000 mgr.y (mgr.14144) 79 : audit [DBG] from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:51.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: audit 2021-03-15T22:15:51.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:15:51.179 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 15:50.494323+0000 mon.a 2021-03-15T22:15:51.179 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: (mon.0) 202 : audit 2021-03-15T22:15:51.179 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:15:51.179 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: cephadm 2021 2021-03-15T22:15:51.179 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: -03-15T22:15: 2021-03-15T22:15:51.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 50.495387+0000 mgr.y (mgr 2021-03-15T22:15:51.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: .14144) 80 : cephadm [INF] 2021-03-15T22:15:51.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: Applying service osd.None on host smithi076... 2021-03-15T22:15:51.180 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: audit 2021-03-15T22:15:51.181 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 2021-03-15T22:15: 2021-03-15T22:15:51.181 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 50.495610+0000 mon.a (mon. 2021-03-15T22:15:51.181 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 0) 203 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:15:51.181 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 2021-03-15T22:15:51.181 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: audit 2021-03-15T22:15:51.182 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 2021-03-15T 2021-03-15T22:15:51.182 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 22:15:50.496136+0000 2021-03-15T22:15:51.182 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: mon.a (mon.0) 204 : audit 2021-03-15T22:15:51.182 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:51.183 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: cluster 2021-03-15T22:15:51.183 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:15:51.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: 15:50.917667+0000 mgr.y 2021-03-15T22:15:51.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: (mgr.14144) 81 : cluster [DBG] 2021-03-15T22:15:51.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[56514]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:51.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:51.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: audit 2021-03-15T22:15:51.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:15:51.186 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 15:50.197634+0000 mon.a (mon.0 2021-03-15T22:15:51.186 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: ) 201 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:51.186 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:15:51.186 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: -03-15T22:15:50.198255 2021-03-15T22:15:51.186 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 78 : cephadm [ERR] 2021-03-15T22:15:51.187 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:51.187 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:15:51.187 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:51.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:15:51.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:51.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:15:51.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:51.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:51.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:51.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: audit 2021-03-15T22:15:50.493549+0000 mgr.y (mgr.14144) 79 : audit [DBG] from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:51.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: audit 2021-03-15T22:15:51.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 2021-03 2021-03-15T22:15:51.190 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: -15T22:15:50.494323+0000 2021-03-15T22:15:51.190 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: mon.a (mon.0) 202 : audit [DBG] 2021-03-15T22:15:51.190 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:15:51.190 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: cephadm 2021-03-15T22:15:51.190 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 2021-03 2021-03-15T22:15:51.191 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: -15T22:15:50.495387+0000 mgr.y (mgr 2021-03-15T22:15:51.191 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: .14144) 80 : cephadm [INF] Applying service osd.None on host smithi076... 2021-03-15T22:15:51.191 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: audit 2021 2021-03-15T22:15:51.191 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: -03-15T22:15:50.495610 2021-03-15T22:15:51.192 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: +0000 mon.a (mon.0) 203 : audit [INF] 2021-03-15T22:15:51.192 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:15:51.192 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: audit 2021-03-15T22:15:51.192 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:15:51.193 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: :15:50.496136+0000 mon.a ( 2021-03-15T22:15:51.193 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: mon.0) 204 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:51.193 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: cluster 2021-03-15T22:15:51.193 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: 2021-03-15T22:15:50.917667 2021-03-15T22:15:51.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 81 : cluster [DBG] 2021-03-15T22:15:51.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:51 smithi076 conmon[62156]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:51.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cluster 2021-03-15T22:15:48.855526+0000 osd.0 (osd.0) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:15:51.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cluster 2021-03-15T22:15:48.855719+0000 osd.0 (osd.0) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:15:51.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cluster 2021-03-15T22:15:51.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 2021-03-15T22:15:50.007028+0000 mon.a 2021-03-15T22:15:51.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: (mon.0) 195 : cluster [INF] osd.0 [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176] boot 2021-03-15T22:15:51.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:15:51.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 15T22:15:50.007062+0000 2021-03-15T22:15:51.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: mon.a (mon.0) 196 : cluster [DBG] 2021-03-15T22:15:51.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: osdmap e8: 1 total, 1 up, 1 in 2021-03-15T22:15:51.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:15:51.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: -15T22:15:50.007557 2021-03-15T22:15:51.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: +0000 mon.a (mon.0) 197 : audit 2021-03-15T22:15:51.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2021-03-15T22:15:51.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cephadm 2021-03 2021-03-15T22:15:51.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: -15T22:15:50. 2021-03-15T22:15:51.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 180011+0000 mgr.y (mgr.14144) 74 : cephadm 2021-03-15T22:15:51.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: [INF] Detected new or changed devices on smithi076 2021-03-15T22:15:51.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:15:51.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: -15T22:15:50. 2021-03-15T22:15:51.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 184013+0000 mon.a (mon.0) 198 : audit 2021-03-15T22:15:51.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:51.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cephadm 2021 2021-03-15T22:15:51.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: -03-15T22:15: 2021-03-15T22:15:51.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 50.184532+0000 mgr.y (mgr.14144) 2021-03-15T22:15:51.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 75 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:51.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: audit 2021-03-15T22:15:51.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:15:51.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 15:50.189532+0000 mon.a 2021-03-15T22:15:51.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: (mon.0) 199 : audit [INF] 2021-03-15T22:15:51.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:51.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cephadm 2021- 2021-03-15T22:15:51.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 03-15T22:15:50.190400 2021-03-15T22:15:51.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 76 : cephadm 2021-03-15T22:15:51.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:15:51.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:15:51.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:51.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:15:51.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:51.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:15:51.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:51.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:51.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:15:51.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: audit 2021 2021-03-15T22:15:51.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: -03-15T22:15:50.190930+0000 mon.a (mon.0) 200 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:15:51.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cephadm 2021-03-15T22:15 2021-03-15T22:15:51.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: :50.192268+0000 mgr.y (mgr.14144) 77 : cephadm [INF] 2021-03-15T22:15:51.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:15:51.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:15:51.300 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 15T22:15:50.197634+0000 2021-03-15T22:15:51.300 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: mon.a (mon.0) 201 : audit [INF] 2021-03-15T22:15:51.300 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:15:51.300 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cephadm 2021-03 2021-03-15T22:15:51.301 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: -15T22:15:50. 2021-03-15T22:15:51.301 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 198255+0000 mgr.y (mgr.14144) 78 : 2021-03-15T22:15:51.301 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:15:51.301 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:15:51.301 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:15:51.302 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:15:51.302 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:15:51.302 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:15:51.303 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:15:51.303 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:15:51.303 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:15:51.303 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: audit 2021-03-15T22:15:50.493549+0000 mgr.y (mgr.14144) 79 : audit [DBG] from='client.24148 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:15:51.303 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: audit 2021-03-15T22:15:51.304 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 2021-03-15T22:15 2021-03-15T22:15:51.304 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: :50.494323+0000 mon.a (mon.0) 2021-03-15T22:15:51.304 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 202 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:15:51.305 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cephadm 2021-03- 2021-03-15T22:15:51.305 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 15T22:15:50.495387+0000 mgr.y (mgr.14144) 80 : cephadm [INF] 2021-03-15T22:15:51.306 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: Applying service osd.None on host smithi076... 2021-03-15T22:15:51.306 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:15:51.306 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 15T22:15:50.495610+0000 2021-03-15T22:15:51.307 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: mon.a (mon.0) 203 : audit [INF] 2021-03-15T22:15:51.308 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:15:51.308 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: audit 2021 2021-03-15T22:15:51.309 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: -03-15T22:15: 2021-03-15T22:15:51.309 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 50.496136+0000 mon.a (mon 2021-03-15T22:15:51.309 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: .0) 204 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:51.310 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 2021-03-15T22:15:51.310 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:15:51.310 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: 03-15T22:15:50 2021-03-15T22:15:51.311 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: .917667+0000 mgr.y (mgr.14144) 81 : 2021-03-15T22:15:51.311 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:51 smithi125 conmon[56739]: cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2021-03-15T22:15:52.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:52 smithi076 conmon[56514]: cluster 2021-03-15T22:15:51.212010+0000 2021-03-15T22:15:52.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:52 smithi076 conmon[56514]: mon.a (mon.0) 205 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-03-15T22:15:52.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:52 smithi076 conmon[62156]: cluster 2021-03-15T22:15:51.212010+0000 mon.a ( 2021-03-15T22:15:52.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:52 smithi076 conmon[62156]: mon.0) 205 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-03-15T22:15:52.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:52 smithi125 conmon[56739]: cluster 2021-03-15T22:15:51.212010+0000 mon.a (mon.0) 205 : cluster [DBG] osdmap e9: 1 total, 1 up, 1 in 2021-03-15T22:15:53.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[56514]: audit 2021-03-15T22:15:52.578770+0000 mon.a (mon.0) 206 : audit 2021-03-15T22:15:53.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[56514]: [INF] from='client.? 172.21.15.76:0/2933940967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526"}]: dispatch 2021-03-15T22:15:53.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[56514]: audit 2021-03-15T22:15:52.584487+0000 mon.a (mon.0) 207 : audit [INF] from='client.? 172.21.15.76:0/2933940967' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526"}]': finished 2021-03-15T22:15:53.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[56514]: cluster 2021-03-15T22:15:52.584544+0000 mon.a (mon.0) 208 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-03-15T22:15:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[56514]: audit 2021-03-15T22:15:52.584651+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:15:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[56514]: cluster 2021-03-15T22:15:52.917959+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:15:53.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[56514]: audit 2021-03-15T22:15:53.061083+0000 mon.c (mon.1) 5 : audit [DBG] from='client.? 172.21.15.76:0/2460898044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:15:53.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: audit 2021-03-15T22:15: 2021-03-15T22:15:53.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: 52.578770+0000 mon.a (mon.0) 206 : audit [INF] from='client.? 172.21.15.76:0/2933940967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526"}]: dispatch 2021-03-15T22:15:53.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:15:53.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: 15:52.584487+0000 mon.a (mon.0) 207 : audit 2021-03-15T22:15:53.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/2933940967' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526"}]': finished 2021-03-15T22:15:53.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: cluster 2021-03-15T22:15:52.584544+0000 mon.a (mon.0) 2021-03-15T22:15:53.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: 208 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-03-15T22:15:53.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: audit 2021-03-15T22:15: 2021-03-15T22:15:53.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: 52.584651+0000 mon.a (mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:15:53.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: 2021-03-15T22:15:53.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:15:53.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: T22:15:52.917959+0000 mgr.y (mgr.14144) 82 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:15:53.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: audit 2021-03-15T22:15: 2021-03-15T22:15:53.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: 53.061083+0000 mon.c (mon.1) 5 : audit [DBG] 2021-03-15T22:15:53.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:53 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2460898044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:15:53.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: audit 2021-03-15T22:15:52.578770+0000 mon.a (mon.0) 206 : audit [INF] from='client.? 172.21.15.76:0/2933940967' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526"}]: dispatch 2021-03-15T22:15:53.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:15:53.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: 22:15:52.584487+0000 mon.a (mon.0) 207 : audit [INF] from='client.? 172.21.15.76:0/2933940967' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526"}]': finished 2021-03-15T22:15:53.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: cluster 2021-03-15T22:15:52.584544+0000 mon.a (mon.0) 208 : cluster [DBG] osdmap e10: 2 total, 1 up, 2 in 2021-03-15T22:15:53.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: audit 2021-03-15T22:15:52.584651+0000 mon.a ( 2021-03-15T22:15:53.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: mon.0) 209 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:15:53.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:15:53.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: -15T22:15:52.917959+0000 mgr.y 2021-03-15T22:15:53.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: (mgr.14144) 82 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:15:53.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: audit 2021 2021-03-15T22:15:53.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: -03-15T22:15:53.061083 2021-03-15T22:15:53.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: +0000 mon.c (mon.1) 5 : audit [DBG] 2021-03-15T22:15:53.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:53 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2460898044' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:15:56.202 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:55 smithi076 conmon[56514]: cluster 2021-03-15T22:15:54.918281+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:15:56.202 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:55 smithi076 conmon[62156]: cluster 2021-03-15T22:15:54.918281+0000 mgr.y (mgr.14144) 83 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:15:56.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:55 smithi125 conmon[56739]: cluster 2021-03-15T22:15:54.918281+0000 mgr.y (mgr.14144) 83 : cluster 2021-03-15T22:15:56.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:55 smithi125 conmon[56739]: [DBG] pgmap v57: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:15:58.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:57 smithi076 conmon[56514]: cluster 2021-03-15T22:15:56. 2021-03-15T22:15:58.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:57 smithi076 conmon[56514]: 918552+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:15:58.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:57 smithi076 conmon[62156]: cluster 2021-03-15T22:15: 2021-03-15T22:15:58.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:57 smithi076 conmon[62156]: 56.918552+0000 mgr.y (mgr.14144) 84 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:15:58.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:57 smithi125 conmon[56739]: cluster 2021-03-15T22:15:56.918552+0000 mgr.y (mgr.14144) 2021-03-15T22:15:58.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:57 smithi125 conmon[56739]: 84 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:15:59.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:58 smithi076 conmon[56514]: audit 2021-03-15T22:15:58.115954+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-03-15T22:15:59.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:58 smithi076 conmon[56514]: audit 2021-03-15T22:15:58.116489+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:59.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:58 smithi076 conmon[62156]: audit 2021-03-15T22:15:58.115954+0000 mon.a (mon.0) 210 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-03-15T22:15:59.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:58 smithi076 conmon[62156]: audit 2021-03-15T22:15:58.116489+0000 mon.a (mon.0) 211 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:15:59.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:58 smithi125 conmon[56739]: audit 2021-03-15T22:15:58.115954+0000 2021-03-15T22:15:59.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:58 smithi125 conmon[56739]: mon.a (mon.0) 210 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2021-03-15T22:15:59.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:58 smithi125 conmon[56739]: audit 2021-03-15T22:15:58.116489+0000 mon.a ( 2021-03-15T22:15:59.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:58 smithi125 conmon[56739]: mon.0) 211 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:00.202 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:59 smithi076 conmon[56514]: cephadm 2021-03-15T22:15:58.116928+0000 2021-03-15T22:16:00.202 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:59 smithi076 conmon[56514]: mgr.y (mgr.14144) 85 : cephadm [INF] Deploying daemon osd.1 on smithi076 2021-03-15T22:16:00.202 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:59 smithi076 conmon[56514]: cluster 2021-03-15T22:15:58 2021-03-15T22:16:00.203 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:15:59 smithi076 conmon[56514]: .918914+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:00.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:59 smithi076 conmon[62156]: cephadm 2021-03-15T22:15 2021-03-15T22:16:00.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:59 smithi076 conmon[62156]: :58.116928+0000 mgr.y (mgr.14144) 85 : cephadm [INF] Deploying daemon osd.1 on smithi076 2021-03-15T22:16:00.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:59 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:16:00.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:59 smithi076 conmon[62156]: -03-15T22:15: 2021-03-15T22:16:00.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:59 smithi076 conmon[62156]: 58.918914+0000 mgr.y (mgr 2021-03-15T22:16:00.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:59 smithi076 conmon[62156]: .14144) 86 : cluster [DBG] 2021-03-15T22:16:00.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:15:59 smithi076 conmon[62156]: pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:00.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:59 smithi125 conmon[56739]: cephadm 2021-03-15T22:15:58.116928+0000 mgr.y (mgr.14144) 85 : cephadm [INF] Deploying daemon osd.1 on smithi076 2021-03-15T22:16:00.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:15:59 smithi125 conmon[56739]: cluster 2021-03-15T22:15:58.918914+0000 mgr.y (mgr.14144) 86 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:01.251 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 1 on host 'smithi076' 2021-03-15T22:16:01.860 DEBUG:teuthology.orchestra.run.smithi076:osd.1> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.1.service 2021-03-15T22:16:01.863 INFO:tasks.cephadm:Deploying osd.2 on smithi076 with /dev/vg_nvme/lv_2... 2021-03-15T22:16:01.863 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-03-15T22:16:01.894 INFO:journalctl@ceph.osd.1.smithi076.stdout:-- Logs begin at Mon 2021-03-15 21:44:23 UTC. -- 2021-03-15T22:16:02.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[56514]: cluster 2021-03-15T22:16:00.919397 2021-03-15T22:16:02.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 87 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:02.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[56514]: audit 2021-03-15T22:16:01.030104+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:16:02.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[56514]: audit 2021-03-15T22:16:01.030987+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:16:02.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[56514]: audit 2021-03-15T22:16:01.238912+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:02.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[56514]: audit 2021-03-15T22:16:01.242407+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:02.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[56514]: audit 2021-03-15T22:16:01.247456+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:02.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: cluster 2021-03-15T22:16:00.919397 2021-03-15T22:16:02.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 87 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:02.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:16:02.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: T22:16:01.030104+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:16:02.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: audit 2021-03-15T22:16:02.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: 2021-03-15T22:16:01.030987 2021-03-15T22:16:02.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: +0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:16:02.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: 2021-03-15T22:16:02.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: audit 2021-03-15T22:16:02.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: 2021-03-15T22:16:01.238912+0000 mon.a (mon 2021-03-15T22:16:02.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: .0) 214 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:02.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:16:02.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: 03-15T22:16:01.242407+0000 2021-03-15T22:16:02.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: mon.a (mon.0) 215 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:02.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: audit 2021-03-15T22:16:02.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: 2021-03-15T22:16:01.247456 2021-03-15T22:16:02.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: +0000 mon.a (mon.0) 216 : audit [INF] 2021-03-15T22:16:02.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:01 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:02.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:01 smithi125 conmon[56739]: cluster 2021-03-15T22:16:00.919397+0000 mgr.y (mgr.14144) 87 : 2021-03-15T22:16:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:01 smithi125 conmon[56739]: cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:01 smithi125 conmon[56739]: audit 2021-03-15T22:16:01.030104+0000 mon.a (mon.0) 212 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:16:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:01 smithi125 conmon[56739]: audit 2021-03-15T22:16:01.030987+0000 mon.a (mon.0) 213 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:16:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:01 smithi125 conmon[56739]: audit 2021-03-15T22:16:01.238912+0000 mon.a (mon.0) 214 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:02.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:01 smithi125 conmon[56739]: audit 2021-03-15T22:16:01.242407+0000 mon.a (mon.0) 215 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:02.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:01 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:02.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:01 smithi125 conmon[56739]: -03-15T22:16:01.247456+0000 mon.a (mon.0) 216 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:03.841 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-03-15T22:16:03.841 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-03-15T22:16:03.842 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2021-03-15T22:16:03.842 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2021-03-15T22:16:03.842 INFO:teuthology.orchestra.run.smithi076.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.012453 s, 842 MB/s 2021-03-15T22:16:03.842 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 2021-03-15T22:16:03.843 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2021-03-15T22:16:04.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:03 smithi076 conmon[56514]: cluster 2021-03-15T22:16:02.919657+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:04.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:03 smithi076 conmon[56514]: audit 2021-03-15T22:16:03.220543+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:04.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:03 smithi076 conmon[62156]: cluster 2021-03-15T22:16:02.919657+0000 mgr.y (mgr.14144) 88 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:04.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:03 smithi076 conmon[62156]: audit 2021-03-15T22:16:03.220543+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:04.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:03 smithi125 conmon[56739]: cluster 2021-03-15T22:16:02.919657+0000 mgr.y (mgr.14144) 88 2021-03-15T22:16:04.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:03 smithi125 conmon[56739]: : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:04.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:03 smithi125 conmon[56739]: audit 2021-03-15T22:16:03.220543+0000 mon.a (mon.0) 217 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:04.759 INFO:journalctl@ceph.osd.1.smithi076.stdout:Mar 15 22:16:04 smithi076 conmon[68886]: debug 2021-03-15T22:16:04.478+0000 7fae5a8a5080 -1 osd.1 0 log_to_monitors {default=true} 2021-03-15T22:16:04.783 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch daemon add osd smithi076:vg_nvme/lv_2 2021-03-15T22:16:05.014 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:04 smithi076 conmon[56514]: audit 2021-03-15T22:16:04.481378+0000 mon.c (mon.1) 6 : audit [INF] from='osd.1 [v2:172.21.15.76:6810/3924955423,v1:172.21.15.76:6811/3924955423]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-15T22:16:05.015 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:04 smithi076 conmon[56514]: audit 2021-03-15T22:16:04.481740+0000 mon.a (mon.0) 218 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-15T22:16:05.015 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:04 smithi076 conmon[62156]: audit 2021-03-15T22:16:04.481378+0000 mon.c (mon.1) 6 : audit [INF] from='osd.1 [v2:172.21.15.76:6810/3924955423,v1:172.21.15.76:6811/3924955423]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-15T22:16:05.015 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:04 smithi076 conmon[62156]: audit 2021-03-15T22:16:04.481740+0000 mon.a (mon.0) 218 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-15T22:16:05.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:04 smithi125 conmon[56739]: audit 2021-03-15T22:16:04.481378+0000 mon.c (mon. 2021-03-15T22:16:05.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:04 smithi125 conmon[56739]: 1) 6 : audit [INF] from='osd.1 [v2:172.21.15.76:6810/3924955423,v1:172.21.15.76:6811/3924955423]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-15T22:16:05.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:04 smithi125 conmon[56739]: audit 2021-03-15T22:16:04.481740+0000 mon.a (mon.0) 218 2021-03-15T22:16:05.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:04 smithi125 conmon[56739]: : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2021-03-15T22:16:06.267 INFO:journalctl@ceph.osd.1.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[68886]: debug 2021-03-15T22:16:05.961+0000 7fae50e16700 -1 osd.1 0 waiting for initial osdmap 2021-03-15T22:16:06.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[62156]: cluster 2021-03-15T22:16:04.919926+0000 2021-03-15T22:16:06.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[62156]: mgr.y (mgr.14144) 89 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:06.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[62156]: audit 2021-03-15T22:16:04.950770+0000 mon.a (mon.0) 219 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-03-15T22:16:06.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[62156]: cluster 2021-03-15T22:16:04.950821+0000 mon.a (mon.0) 220 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-03-15T22:16:06.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[62156]: audit 2021-03-15T22:16:04.950951+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:06.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[62156]: audit 2021-03-15T22:16:04.951404+0000 mon.c (mon.1) 7 : audit [INF] from='osd.1 [v2:172.21.15.76:6810/3924955423,v1:172.21.15.76:6811/3924955423]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:06.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[62156]: audit 2021-03-15T22:16:04.951686+0000 mon.a (mon.0) 222 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:06.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[56514]: cluster 2021-03-15T22:16:04.919926+0000 mgr.y (mgr.14144) 89 : cluster [DBG] 2021-03-15T22:16:06.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[56514]: pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:06.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[56514]: audit 2021-03-15T22:16:04.950770+0000 mon.a (mon.0) 219 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-03-15T22:16:06.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[56514]: cluster 2021-03-15T22:16:04.950821+0000 mon.a (mon.0) 220 : cluster [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-03-15T22:16:06.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[56514]: audit 2021-03-15T22:16:04.950951+0000 mon.a (mon.0) 221 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:06.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[56514]: audit 2021-03-15T22:16:04.951404+0000 mon.c (mon.1) 7 : audit [INF] from='osd.1 [v2:172.21.15.76:6810/3924955423,v1:172.21.15.76:6811/3924955423]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:06.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:05 smithi076 conmon[56514]: audit 2021-03-15T22:16:04.951686+0000 mon.a (mon.0) 222 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:06.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: cluster 2021-03-15T22:16:04.919926+0000 mgr.y (mgr.14144) 2021-03-15T22:16:06.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: 89 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:06.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: audit 2021-03-15T22:16:04.950770+0000 mon.a (mon.0) 219 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2021-03-15T22:16:06.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: 2021-03-15T22:16:06.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:16:06.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: T22:16:04.950821+0000 2021-03-15T22:16:06.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: mon.a (mon.0) 220 : cluster 2021-03-15T22:16:06.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: [DBG] osdmap e11: 2 total, 1 up, 2 in 2021-03-15T22:16:06.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:06.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: 03-15T22:16:04.950951+0000 2021-03-15T22:16:06.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: mon.a (mon.0) 221 : 2021-03-15T22:16:06.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:06.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:06.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: 03-15T22:16:04.951404 2021-03-15T22:16:06.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: +0000 mon.c (mon.1) 7 2021-03-15T22:16:06.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: : audit [INF] from='osd.1 [v2:172.21.15.76:6810/3924955423,v1:172.21.15.76:6811/3924955423]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:06.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:06.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: -03-15T22:16:04. 2021-03-15T22:16:06.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: 951686+0000 mon.a (mon.0 2021-03-15T22:16:06.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:05 smithi125 conmon[56739]: ) 222 : audit [INF] from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:07.267 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: debug 2021-03-15T22:16:06.817+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:07.267 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:16:07.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:07.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:16:07.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:07.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:16:07.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:07.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:07.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:07.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: debug 2021-03-15T22:16:06.826+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:07.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:16:07.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:07.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:16:07.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:07.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:16:07.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:07.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:07.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:07.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: audit 2021-03-15T22:16:05.954503+0000 mon.a ( 2021-03-15T22:16:07.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: mon.0) 223 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:16:07.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: cluster 2021-03-15T22:16:05.954582+0000 mon.a (mon.0) 224 : cluster [DBG] osdmap e12: 2 total, 1 up, 2 in 2021-03-15T22:16:07.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: audit 2021-03-15T22:16:05. 2021-03-15T22:16:07.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: 955004+0000 mon.a (mon.0) 225 : audit [DBG] 2021-03-15T22:16:07.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:07.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: audit 2021-03-15T22:16:07.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: 2021-03-15T22:16:05.957375 2021-03-15T22:16:07.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: +0000 mon.a (mon.0) 226 : audit 2021-03-15T22:16:07.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:07.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: cephadm 2021-03-15T 2021-03-15T22:16:07.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: 22:16:06.807975+0000 mgr.y (mgr. 2021-03-15T22:16:07.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: 14144) 90 : cephadm [INF] Detected new or changed devices on smithi076 2021-03-15T22:16:07.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: audit 2021-03-15T22:16:07.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: 2021-03-15T22:16:06.812037 2021-03-15T22:16:07.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: +0000 mon.a (mon.0) 227 : audit [INF] 2021-03-15T22:16:07.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:07.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: cephadm 2021-03-15T22 2021-03-15T22:16:07.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: :16:06.812718+0000 mgr.y (mgr.14144 2021-03-15T22:16:07.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: ) 91 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:07.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: audit 2021-03-15T22:16:07.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: 2021-03-15T22:16 2021-03-15T22:16:07.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: :06.817893+0000 mon.a (mon.0) 228 2021-03-15T22:16:07.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:07.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:07.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: 2021-03-15T22:16: 2021-03-15T22:16:07.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: 06.818708+0000 mgr.y (mgr.14144) 92 : 2021-03-15T22:16:07.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:07.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:16:07.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:07.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:16:07.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:07.279 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:16:07.279 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:07.279 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:07.279 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:07.280 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: audit 2021-03-15T22:16:06.819120+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:07.280 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:07.280 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: 2021-03-15T22:16:06.820553+0000 mgr.y (mgr.14144 2021-03-15T22:16:07.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: audit 2021-03-15T22:16:05.954503+0000 2021-03-15T22:16:07.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: mon.a (mon.0) 223 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:16:07.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:16:07.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 03-15T22:16:05.954582+0000 mon.a (mon.0) 224 : cluster [DBG] 2021-03-15T22:16:07.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: osdmap e12: 2 total, 1 up, 2 in 2021-03-15T22:16:07.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: audit 2021-03-15T22:16 2021-03-15T22:16:07.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: :05.955004+0000 mon.a (mon.0) 2021-03-15T22:16:07.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 225 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:07.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: audit 2021-03-15T22:16:07.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 2021-03-15T22:16: 2021-03-15T22:16:07.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 05.957375+0000 mon.a (mon.0) 226 2021-03-15T22:16:07.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:07.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:16:07.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 03-15T22:16:06.807975+0000 2021-03-15T22:16:07.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: mgr.y (mgr.14144) 90 : cephadm [INF] Detected new or changed devices on smithi076 2021-03-15T22:16:07.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 2021-03-15T22:16:07.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:16:07.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 15T22:16:06.812037+0000 mon.a ( 2021-03-15T22:16:07.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: mon.0) 227 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:07.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:16:07.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: -03-15T22:16:06.812718+0000 2021-03-15T22:16:07.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: mgr.y (mgr.14144) 91 : cephadm [INF] 2021-03-15T22:16:07.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:07.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: audit 2021-03-15T22:16 2021-03-15T22:16:07.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: :06.817893+0000 mon.a (mon.0) 2021-03-15T22:16:07.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 228 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:07.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:07.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 2021-03-15T22:16: 2021-03-15T22:16:07.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 06.818708+0000 mgr.y (mgr.14144) 2021-03-15T22:16:07.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 92 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:07.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:16:07.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:07.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:16:07.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:07.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:16:07.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:07.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:07.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:07.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: audit 2021-03-15T22:16:06.819120+0000 mon.a (mon.0) 229 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:07.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:06 2021-03-15T22:16:07.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: .820553+0000 mgr.y (mgr.14144) 93 : 2021-03-15T22:16:07.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: ) 93 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:07.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: audit 2021-03-15T22:16:07.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: 2021-03-15T22:16:06 2021-03-15T22:16:07.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: .827455+0000 mon.a (mon.0) 230 : audit 2021-03-15T22:16:07.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:07.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: cephadm 2021 2021-03-15T22:16:07.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: -03-15T22:16:06.828406+0000 2021-03-15T22:16:07.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: mgr.y (mgr.14144) 94 : cephadm [ERR] 2021-03-15T22:16:07.294 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:07.294 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:16:07.294 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:07.294 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:16:07.294 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:07.295 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:16:07.295 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:07.295 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:07.295 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:07.296 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: cluster 2021-03-15T22:16:06.920176+0000 mgr.y (mgr.14144) 95 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:07.296 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:16:07.296 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: 03-15T22:16:06.957069+0000 mon.a (mon.0) 231 2021-03-15T22:16:07.296 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[56514]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:07.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:07.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:16:07.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 15T22:16:06.827455+0000 mon.a 2021-03-15T22:16:07.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: (mon.0) 230 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:07.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 2021-03-15T22:16:07.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:07.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 2021-03-15T22:16:06.828406+0000 2021-03-15T22:16:07.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: mgr.y (mgr.14144) 94 : cephadm [ERR] 2021-03-15T22:16:07.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:07.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:16:07.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:07.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:16:07.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:07.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:16:07.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:07.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:07.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:07.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: cluster 2021-03-15T22:16:06.920176+0000 mgr.y (mgr.14144) 95 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:07.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:16:07.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: 16:06.957069+0000 mon.a (mon.0 2021-03-15T22:16:07.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:06 smithi076 conmon[62156]: ) 231 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:07.302 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: audit 2021-03-15T22:16:05.954503+0000 mon.a (mon.0) 2021-03-15T22:16:07.302 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 223 : audit [INF] from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:16:07.302 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: cluster 2021-03-15T22:16:05.954582+0000 mon.a (mon.0) 224 : cluster [DBG] 2021-03-15T22:16:07.303 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: osdmap e12: 2 total, 1 up, 2 in 2021-03-15T22:16:07.303 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:16:07.303 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 15T22:16:05.955004+0000 2021-03-15T22:16:07.303 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: mon.a (mon.0) 225 : 2021-03-15T22:16:07.303 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:07.304 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:07.304 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 03-15T22:16:05.957375+0000 2021-03-15T22:16:07.304 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: mon.a (mon.0) 226 : audit 2021-03-15T22:16:07.304 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:07.305 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: cephadm 2021- 2021-03-15T22:16:07.305 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 03-15T22:16:06.807975 2021-03-15T22:16:07.305 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 90 2021-03-15T22:16:07.305 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: : cephadm [INF] Detected new or changed devices on smithi076 2021-03-15T22:16:07.305 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:07.306 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: -03-15T22:16:06 2021-03-15T22:16:07.306 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: .812037+0000 mon.a (mon. 2021-03-15T22:16:07.306 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 0) 227 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:07.307 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 2021-03-15T22:16:07.307 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: cephadm 2021 2021-03-15T22:16:07.307 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: -03-15T22:16:06 2021-03-15T22:16:07.307 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: .812718+0000 mgr.y (mgr.14144 2021-03-15T22:16:07.307 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: ) 91 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:07.308 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 2021-03-15T22:16:07.308 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:07.308 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: -03-15T22:16:06 2021-03-15T22:16:07.308 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: .817893+0000 mon.a (mon. 2021-03-15T22:16:07.309 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 0) 228 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:07.309 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 2021-03-15T22:16:07.309 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: cephadm 2021 2021-03-15T22:16:07.309 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: -03-15T22:16:06 2021-03-15T22:16:07.309 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: .818708+0000 mgr.y (mgr. 2021-03-15T22:16:07.310 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 14144) 92 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:07.310 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:16:07.310 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:07.310 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:16:07.310 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:07.311 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:16:07.311 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:07.311 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:07.311 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:07.312 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: audit 2021-03-15T22:16:06.819120+0000 mon.a (mon.0) 229 : audit 2021-03-15T22:16:07.312 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:07.312 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:07.312 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 2021-03-15T22:16: 2021-03-15T22:16:07.313 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 06.820553+0000 mgr.y (mgr 2021-03-15T22:16:07.313 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: .14144) 93 : cephadm [INF] 2021-03-15T22:16:07.313 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:07.313 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:07.313 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: -03-15T22:16:06 2021-03-15T22:16:07.314 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: .827455+0000 mon.a (mon. 2021-03-15T22:16:07.314 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: 0) 230 : audit [INF] 2021-03-15T22:16:07.314 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:07.314 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: cephadm 2021 2021-03-15T22:16:07.315 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: -03-15T22:16:06.828406 2021-03-15T22:16:07.315 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 94 2021-03-15T22:16:07.315 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:07.315 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:16:07.315 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:07.316 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:16:07.316 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:07.316 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:16:07.316 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:07.316 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:07.317 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:07.317 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: cluster 2021-03-15T22:16:06.920176 2021-03-15T22:16:07.317 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 95 2021-03-15T22:16:07.317 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 4.7 MiB used, 89 GiB / 89 GiB avail 2021-03-15T22:16:07.318 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:07.318 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: -03-15T22:16:06 2021-03-15T22:16:07.318 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: .957069+0000 mon.a (mon 2021-03-15T22:16:07.318 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: .0) 231 : audit [DBG] 2021-03-15T22:16:07.318 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:06 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:07.981 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[62156]: cluster 2021-03-15T22:16:05.484684 2021-03-15T22:16:07.981 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:07.982 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[62156]: cluster 2021-03-15T22:16:05.484779+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:07.982 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[62156]: cluster 2021-03-15T22:16:06.965554+0000 mon.a (mon.0) 232 : cluster [INF] osd.1 [v2:172.21.15.76:6810/3924955423,v1:172.21.15.76:6811/3924955423] boot 2021-03-15T22:16:07.982 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[62156]: cluster 2021-03-15T22:16:06.965608+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-03-15T22:16:07.982 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[62156]: audit 2021-03-15T22:16:06.966246+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:07.983 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[62156]: audit 2021-03-15T22:16:07.000614+0000 mgr.y (mgr.14144) 96 : audit [DBG] from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:07.983 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[62156]: audit 2021-03-15T22:16:07.001267+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:07.983 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:07.002241+0000 mgr.y (mgr.14144) 97 : cephadm [INF] Applying service osd.None on host smithi076... 2021-03-15T22:16:07.983 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[62156]: audit 2021-03-15T22:16:07.002420+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:07.984 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[62156]: audit 2021-03-15T22:16:07.002958+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[56514]: cluster 2021-03-15T22:16:05.484684+0000 osd.1 ( 2021-03-15T22:16:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[56514]: osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[56514]: cluster 2021-03-15T22:16:05.484779+0000 osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[56514]: cluster 2021-03-15T22:16:06.965554+0000 mon.a (mon.0) 232 : cluster [INF] osd.1 [v2:172.21.15.76:6810/3924955423,v1:172.21.15.76:6811/3924955423] boot 2021-03-15T22:16:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[56514]: cluster 2021-03-15T22:16:06.965608+0000 mon.a (mon.0) 233 : cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-03-15T22:16:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[56514]: audit 2021-03-15T22:16:06.966246+0000 mon.a (mon.0) 234 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[56514]: audit 2021-03-15T22:16:07.000614+0000 mgr.y (mgr.14144) 96 : audit [DBG] from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:08.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[56514]: audit 2021-03-15T22:16:07.001267+0000 mon.a (mon.0) 235 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:08.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:07.002241+0000 mgr.y (mgr.14144) 97 : cephadm [INF] Applying service osd.None on host smithi076... 2021-03-15T22:16:08.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[56514]: audit 2021-03-15T22:16:07.002420+0000 mon.a (mon.0) 236 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:08.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:07 smithi076 conmon[56514]: audit 2021-03-15T22:16:07.002958+0000 mon.a (mon.0) 237 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:08.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: cluster 2021-03-15T22:16:05.484684+0000 osd.1 (osd.1) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:08.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: cluster 2021-03-15T22:16:05.484779+0000 2021-03-15T22:16:08.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: osd.1 (osd.1) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:08.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: cluster 2021-03-15T22:16:06.965554+0000 mon.a (mon.0) 232 : cluster [INF] osd.1 [v2:172.21.15.76:6810/3924955423,v1:172.21.15.76:6811/3924955423] boot 2021-03-15T22:16:08.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:16:08.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: -15T22:16:06.965608 2021-03-15T22:16:08.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: +0000 mon.a (mon.0) 233 : 2021-03-15T22:16:08.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: cluster [DBG] osdmap e13: 2 total, 2 up, 2 in 2021-03-15T22:16:08.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:08.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: 03-15T22:16:06.966246+0000 2021-03-15T22:16:08.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: mon.a (mon.0) 234 : audit 2021-03-15T22:16:08.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2021-03-15T22:16:08.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:08.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: 03-15T22:16:07. 2021-03-15T22:16:08.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: 000614+0000 mgr.y (mgr.14144) 2021-03-15T22:16:08.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: 96 : audit [DBG] from='client.24175 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:08.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:08.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: 03-15T22:16:07. 2021-03-15T22:16:08.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: 001267+0000 mon.a (mon.0) 2021-03-15T22:16:08.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: 235 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:08.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: cephadm 2021 2021-03-15T22:16:08.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: -03-15T22:16:07 2021-03-15T22:16:08.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: .002241+0000 mgr.y (mgr.14144 2021-03-15T22:16:08.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: ) 97 : cephadm [INF] Applying service osd.None on host smithi076... 2021-03-15T22:16:08.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:08.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: -03-15T22:16: 2021-03-15T22:16:08.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: 07.002420+0000 mon.a (mon 2021-03-15T22:16:08.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: .0) 236 : audit [INF] 2021-03-15T22:16:08.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:08.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:08.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: -03-15T22:16:07 2021-03-15T22:16:08.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: .002958+0000 mon.a (mon. 2021-03-15T22:16:08.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: 0) 237 : audit [DBG] 2021-03-15T22:16:08.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:07 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:09.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:08 smithi076 conmon[56514]: cluster 2021-03-15T22:16:07.974476+0000 mon.a (mon.0) 238 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-03-15T22:16:09.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:08 smithi076 conmon[56514]: cluster 2021-03-15T22:16:08.920445+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:09.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:08 smithi076 conmon[62156]: cluster 2021-03-15T22:16:07.974476+0000 mon.a (mon.0) 238 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-03-15T22:16:09.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:08 smithi076 conmon[62156]: cluster 2021-03-15T22:16:08.920445+0000 mgr.y (mgr.14144) 98 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:09.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:08 smithi125 conmon[56739]: cluster 2021-03-15T22:16:07.974476+0000 mon.a (mon.0 2021-03-15T22:16:09.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:08 smithi125 conmon[56739]: ) 238 : cluster [DBG] osdmap e14: 2 total, 2 up, 2 in 2021-03-15T22:16:09.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:08 smithi125 conmon[56739]: cluster 2021-03-15T22:16:08.920445+0000 mgr.y (mgr.14144) 98 : 2021-03-15T22:16:09.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:08 smithi125 conmon[56739]: cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:10.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[56514]: audit 2021-03-15T22:16:08.983838+0000 mon.a (mon 2021-03-15T22:16:10.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[56514]: .0) 239 : audit [INF] from='client.? 172.21.15.76:0/2028981606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05"}]: dispatch 2021-03-15T22:16:10.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[56514]: audit 2021-03-15T22:16:08.986921+0000 mon.a (mon.0) 240 : audit [INF] from='client.? 172.21.15.76:0/2028981606' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05"}]': finished 2021-03-15T22:16:10.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[56514]: cluster 2021-03-15T22:16:08.986978+0000 mon.a (mon.0) 241 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-03-15T22:16:10.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[56514]: audit 2021-03-15T22:16:08.987082+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:10.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[56514]: audit 2021-03-15T22:16:09.462464+0000 mon.c (mon.1) 8 : audit [DBG] from='client.? 172.21.15.76:0/3108781005' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:16:10.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: audit 2021-03-15T22:16:08.983838+0000 2021-03-15T22:16:10.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: mon.a (mon.0) 239 : audit [INF] from='client.? 172.21.15.76:0/2028981606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05"}]: dispatch 2021-03-15T22:16:10.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:16:10.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: :16:08.986921+0000 mon.a (mon. 2021-03-15T22:16:10.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: 0) 240 : audit [INF] from='client.? 172.21.15.76:0/2028981606' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05"}]': finished 2021-03-15T22:16:10.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:16:10.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: -03-15T22:16:08.986978+0000 mon.a (mon. 2021-03-15T22:16:10.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: 0) 241 : cluster [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-03-15T22:16:10.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:16:10.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: 15T22:16:08.987082+0000 mon.a (mon.0) 242 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:10.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: 2021-03-15T22:16:10.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:16:10.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: 03-15T22:16:09.462464+0000 mon.c (mon.1) 8 2021-03-15T22:16:10.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:09 smithi076 conmon[62156]: : audit [DBG] from='client.? 172.21.15.76:0/3108781005' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:16:10.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: audit 2021-03-15T22:16:08.983838+0000 mon.a (mon.0) 239 : audit [INF] from='client.? 172.21.15.76:0/2028981606' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05"}]: dispatch 2021-03-15T22:16:10.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: audit 2021-03-15T22:16:08.986921+0000 mon.a (mon.0) 240 : audit [INF] from='client.? 172.21.15.76:0/2028981606' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05"}]': finished 2021-03-15T22:16:10.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: cluster 2021-03-15T22:16:10.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: 2021-03-15T22:16:08.986978+0000 2021-03-15T22:16:10.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: mon.a (mon.0) 241 : cluster 2021-03-15T22:16:10.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: [DBG] osdmap e15: 3 total, 2 up, 3 in 2021-03-15T22:16:10.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:10.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: 03-15T22:16:08.987082+0000 mon.a 2021-03-15T22:16:10.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: (mon.0) 242 : audit [DBG] 2021-03-15T22:16:10.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:10.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:10.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: -03-15T22:16:09 2021-03-15T22:16:10.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: .462464+0000 mon.c (mon. 2021-03-15T22:16:10.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: 1) 8 : audit [DBG] from='client.? 172.21.15.76:0/3108781005' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:16:10.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:09 smithi125 conmon[56739]: 2021-03-15T22:16:11.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:10 smithi076 conmon[56514]: cluster 2021-03-15T22:16:10.920846+0000 mgr.y (mgr.14144 2021-03-15T22:16:11.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:10 smithi076 conmon[56514]: ) 99 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:11.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:10 smithi076 conmon[62156]: cluster 2021-03-15T22:16:10.920846+0000 mgr.y (mgr 2021-03-15T22:16:11.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:10 smithi076 conmon[62156]: .14144) 99 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:11.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:10 smithi125 conmon[56739]: cluster 2021-03-15T22:16:10.920846+0000 mgr.y (mgr.14144) 99 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 9.5 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:13.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:13 smithi076 conmon[56514]: cluster 2021-03-15T22:16:12. 2021-03-15T22:16:13.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:13 smithi076 conmon[56514]: 921103+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:13.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:13 smithi076 conmon[62156]: cluster 2021-03-15T22:16:12.921103+0000 2021-03-15T22:16:13.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:13 smithi076 conmon[62156]: mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:13.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:13 smithi125 conmon[56739]: cluster 2021-03-15T22:16:12.921103+0000 mgr.y (mgr.14144) 100 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:15.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:14 smithi076 conmon[62156]: audit 2021-03-15T22:16:14.664335+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-03-15T22:16:15.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:14 smithi076 conmon[62156]: audit 2021-03-15T22:16:14.664957+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:15.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:14 smithi076 conmon[56514]: audit 2021-03-15T22:16:14.664335+0000 mon.a (mon.0) 243 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-03-15T22:16:15.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:14 smithi076 conmon[56514]: audit 2021-03-15T22:16:14.664957+0000 mon.a (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:15.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:14 smithi125 conmon[56739]: audit 2021-03-15T22:16:14.664335+0000 mon.a ( 2021-03-15T22:16:15.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:14 smithi125 conmon[56739]: mon.0) 243 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2021-03-15T22:16:15.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:14 smithi125 conmon[56739]: audit 2021-03-15T22:16:14.664957+0000 mon.a 2021-03-15T22:16:15.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:14 smithi125 conmon[56739]: (mon.0) 244 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:16.201 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:15 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:14.665445+0000 mgr.y (mgr.14144) 101 2021-03-15T22:16:16.202 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:15 smithi076 conmon[56514]: : cephadm [INF] Deploying daemon osd.2 on smithi076 2021-03-15T22:16:16.202 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:15 smithi076 conmon[56514]: cluster 2021-03-15T22:16:14.921341+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:16.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:15 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:14.665445+0000 2021-03-15T22:16:16.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:15 smithi076 conmon[62156]: mgr.y (mgr.14144) 101 : cephadm [INF] Deploying daemon osd.2 on smithi076 2021-03-15T22:16:16.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:15 smithi076 conmon[62156]: cluster 2021-03-15T22:16:14.921341+0000 mgr.y ( 2021-03-15T22:16:16.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:15 smithi076 conmon[62156]: mgr.14144) 102 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:16.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:15 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:14.665445+0000 mgr.y (mgr.14144) 101 : cephadm [INF] Deploying daemon osd.2 on smithi076 2021-03-15T22:16:16.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:15 smithi125 conmon[56739]: cluster 2021-03-15T22:16:14.921341+0000 mgr.y (mgr.14144) 102 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:17.903 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 2 on host 'smithi076' 2021-03-15T22:16:17.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[56514]: cluster 2021-03-15T22:16:16.921766+0000 mgr.y (mgr 2021-03-15T22:16:17.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[56514]: .14144) 103 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:17.965 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[56514]: audit 2021-03-15T22:16:17.888063+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:17.966 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[56514]: audit 2021-03-15T22:16:17.892291+0000 mon.a (mon.0) 246 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:17.966 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[56514]: audit 2021-03-15T22:16:17.898352+0000 mon.a (mon.0) 247 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:17.966 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: cluster 2021-03-15T22:16:16 2021-03-15T22:16:17.967 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: .921766+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:17.967 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: audit 2021-03-15T22:16:17 2021-03-15T22:16:17.967 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: .888063+0000 mon.a (mon.0) 245 : audit [DBG] 2021-03-15T22:16:17.967 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:17.968 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: audit 2021-03-15T22:16:17.968 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:16:17.968 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: 16:17.892291+0000 mon.a (mon.0) 2021-03-15T22:16:17.968 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: 246 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:17.968 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:16:17.969 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: 03-15T22:16:17.898352+0000 mon.a 2021-03-15T22:16:17.969 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:17 smithi076 conmon[62156]: (mon.0) 247 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:18.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:17 smithi125 conmon[56739]: cluster 2021-03-15T22:16:16.921766+0000 mgr.y (mgr.14144) 103 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:18.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:17 smithi125 conmon[56739]: audit 2021-03-15T22:16:17.888063+0000 mon.a (mon.0) 245 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:18.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:17 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:18.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:17 smithi125 conmon[56739]: -03-15T22:16:17.892291+0000 mon.a (mon 2021-03-15T22:16:18.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:17 smithi125 conmon[56739]: .0) 246 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:18.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:17 smithi125 conmon[56739]: 2021-03-15T22:16:18.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:17 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:16:18.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:17 smithi125 conmon[56739]: 15T22:16:17.898352+0000 mon.a ( 2021-03-15T22:16:18.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:17 smithi125 conmon[56739]: mon.0) 247 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:18.462 DEBUG:teuthology.orchestra.run.smithi076:osd.2> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.2.service 2021-03-15T22:16:18.464 INFO:tasks.cephadm:Deploying osd.3 on smithi076 with /dev/vg_nvme/lv_1... 2021-03-15T22:16:18.465 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2021-03-15T22:16:18.494 INFO:journalctl@ceph.osd.2.smithi076.stdout:-- Logs begin at Mon 2021-03-15 21:44:23 UTC. -- 2021-03-15T22:16:20.089 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:19 smithi076 conmon[56514]: cluster 2021-03-15T22:16:18.922060+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:20.089 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:19 smithi076 conmon[56514]: audit 2021-03-15T22:16:19.815778+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:20.090 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:19 smithi076 conmon[62156]: cluster 2021-03-15T22:16:18.922060 2021-03-15T22:16:20.090 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:19 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:20.090 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:19 smithi076 conmon[62156]: audit 2021-03-15T22:16:19 2021-03-15T22:16:20.090 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:19 smithi076 conmon[62156]: .815778+0000 mon.a (mon.0) 248 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:20.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:19 smithi125 conmon[56739]: cluster 2021-03-15T22:16:18.922060+0000 mgr.y (mgr.14144) 104 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:20.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:19 smithi125 conmon[56739]: audit 2021-03-15T22:16:19.815778+0000 mon.a (mon 2021-03-15T22:16:20.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:19 smithi125 conmon[56739]: .0) 248 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:20.510 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping: /dev/vg_nvme/lv_1 2021-03-15T22:16:20.511 INFO:teuthology.orchestra.run.smithi076.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2021-03-15T22:16:20.511 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10+0 records in 2021-03-15T22:16:20.511 INFO:teuthology.orchestra.run.smithi076.stderr:10+0 records out 2021-03-15T22:16:20.511 INFO:teuthology.orchestra.run.smithi076.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140429 s, 747 MB/s 2021-03-15T22:16:20.511 INFO:teuthology.orchestra.run.smithi076.stderr:--> Zapping successful for: 2021-03-15T22:16:21.415 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:16:21 smithi076 conmon[72291]: debug 2021-03-15T22:16:21.070+0000 7fb95c47a080 -1 osd.2 0 log_to_monitors {default=true} 2021-03-15T22:16:21.503 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch daemon add osd smithi076:vg_nvme/lv_1 2021-03-15T22:16:22.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:21 smithi076 conmon[56514]: cluster 2021-03-15T22:16:20.922315+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:22.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:21 smithi076 conmon[56514]: audit 2021-03-15T22:16:21.074101+0000 mon.a (mon.0) 249 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-15T22:16:22.216 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:21 smithi076 conmon[62156]: cluster 2021-03-15T22:16:20.922315+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:22.217 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:21 smithi076 conmon[62156]: audit 2021-03-15T22:16:21.074101+0000 mon.a (mon.0) 249 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-15T22:16:22.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:21 smithi125 conmon[56739]: cluster 2021-03-15T22:16:20.922315+0000 mgr.y (mgr.14144) 105 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:22.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:21 smithi125 conmon[56739]: audit 2021-03-15T22:16:21.074101+0000 mon.a (mon.0) 249 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-15T22:16:23.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[56514]: audit 2021-03-15T22:16:21.951113+0000 mon.a (mon.0) 250 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-15T22:16:23.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[56514]: cluster 2021-03-15T22:16:21.951178+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-03-15T22:16:23.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[56514]: audit 2021-03-15T22:16:21.951713+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:23.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[56514]: audit 2021-03-15T22:16:21.952079+0000 mon.a (mon.0) 253 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:23.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[56514]: cluster 2021-03-15T22:16:22.922620+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:23.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[56514]: audit 2021-03-15T22:16:22.949869+0000 mon.a (mon.0) 254 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:16:23.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[56514]: cluster 2021-03-15T22:16:22.949947+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-15T22:16:23.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[56514]: audit 2021-03-15T22:16:22.950064+0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:23.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[56514]: audit 2021-03-15T22:16:22.953067+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:23.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[62156]: audit 2021-03-15T22:16:21.951113+0000 mon.a ( 2021-03-15T22:16:23.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[62156]: mon.0) 250 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-15T22:16:23.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[62156]: cluster 2021-03-15T22:16:21.951178+0000 mon.a (mon.0) 251 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-03-15T22:16:23.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[62156]: audit 2021-03-15T22:16:21.951713+0000 mon.a (mon.0) 252 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:23.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[62156]: audit 2021-03-15T22:16:21.952079+0000 mon.a (mon.0) 253 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:23.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[62156]: cluster 2021-03-15T22:16:22.922620+0000 mgr.y (mgr.14144) 106 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:23.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[62156]: audit 2021-03-15T22:16:22.949869+0000 mon.a (mon.0) 254 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:16:23.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[62156]: cluster 2021-03-15T22:16:22.949947+0000 mon.a (mon.0) 255 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-15T22:16:23.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[62156]: audit 2021-03-15T22:16:22.950064 2021-03-15T22:16:23.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[62156]: +0000 mon.a (mon.0) 256 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:23.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[62156]: audit 2021-03-15T22:16:22.953067+0000 mon.a (mon.0) 257 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:23.022 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:16:22 smithi076 conmon[72291]: debug 2021-03-15T22:16:22.952+0000 7fb9529eb700 -1 osd.2 0 waiting for initial osdmap 2021-03-15T22:16:23.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: audit 2021-03-15T22:16:21.951113+0000 mon.a ( 2021-03-15T22:16:23.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: mon.0) 250 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-15T22:16:23.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: cluster 2021-03-15T22:16:21.951178+0000 mon.a (mon. 2021-03-15T22:16:23.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 0) 251 : cluster [DBG] osdmap e16: 3 total, 2 up, 3 in 2021-03-15T22:16:23.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:16:23.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: :16:21.951713+0000 mon.a (mon. 2021-03-15T22:16:23.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 0) 252 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:23.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 2021-03-15T22:16:23.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:23.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 03-15T22:16:21.952079 2021-03-15T22:16:23.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2021-03-15T22:16:23.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 253 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:23.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:16:23.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: -03-15T22:16:22. 2021-03-15T22:16:23.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 922620+0000 mgr.y (mgr.14144) 2021-03-15T22:16:23.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 106 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:23.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:23.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: -03-15T22:16:22. 2021-03-15T22:16:23.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 949869+0000 mon.a (mon.0) 2021-03-15T22:16:23.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 254 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:16:23.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:16:23.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 03-15T22:16:22. 2021-03-15T22:16:23.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 949947+0000 mon.a (mon.0 2021-03-15T22:16:23.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: ) 255 : cluster [DBG] osdmap e17: 3 total, 2 up, 3 in 2021-03-15T22:16:23.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 2021-03-15T22:16:23.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:23.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: -03-15T22:16:22 2021-03-15T22:16:23.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: .950064+0000 mon.a (mon. 2021-03-15T22:16:23.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 0) 256 : audit [DBG] 2021-03-15T22:16:23.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:23.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:23.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: 03-15T22:16:22.953067+0000 mon.a ( 2021-03-15T22:16:23.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:22 smithi125 conmon[56739]: mon.0) 257 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:24.017 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: debug 2021-03-15T22:16:23.734+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:24.310 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:16:24.311 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:24.311 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:16:24.311 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:24.311 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:16:24.312 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:24.312 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:24.312 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:24.312 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: debug 2021-03-15T22:16:23.742+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:24.313 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:16:24.313 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:24.313 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:16:24.313 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:24.313 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:16:24.314 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:24.314 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:24.314 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:23 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:24.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: cluster 2021-03-15T22:16:22.060716 2021-03-15T22:16:24.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: +0000 osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:24.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: cluster 2021-03-15T22:16:22.060800+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:24.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:23.453507+0000 mgr.y (mgr.14144) 107 : cephadm [INF] Detected new or changed devices on smithi076 2021-03-15T22:16:24.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit 2021-03-15T22:16:23.457637+0000 mon.a (mon.0) 258 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:24.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:23.728602+0000 mgr.y (mgr.14144) 108 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:24.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit 2021-03-15T22:16:23.734133+0000 mon.a (mon.0) 259 : audit [INF] 2021-03-15T22:16:24.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:24.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:23.735019+0000 mgr.y (mgr.14144) 109 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:24.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:16:24.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:24.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:16:24.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:24.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:16:24.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:24.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:24.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:24.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit 2021-03-15T22: 2021-03-15T22:16:24.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: 16:23.735436+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:24.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:23.736917+0000 mgr.y (mgr.14144) 110 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:24.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit 2021-03-15T22:16:23.742248 2021-03-15T22:16:24.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: +0000 mon.a (mon.0) 261 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:24.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: 2021-03-15T22:16:24.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:24.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: 2021-03-15T22:16:23.742997+0000 mgr.y (mgr.14144) 111 : cephadm 2021-03-15T22:16:24.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:24.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:16:24.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:24.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:16:24.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:24.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:16:24.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:24.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:24.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:24.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit 2021-03-15T22:16:23.773125+0000 mgr.y (mgr.14144) 112 : audit [DBG] from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:24.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit 2021-03-15T22:16:23.773800+0000 mon.a (mon.0) 262 : 2021-03-15T22:16:24.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:24.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: cephadm 2021-03-15T22:16: 2021-03-15T22:16:24.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: 23.774802+0000 mgr.y (mgr.14144) 113 : cephadm [INF] Applying service osd.None on host smithi076... 2021-03-15T22:16:24.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit 2021-03-15T22:16: 2021-03-15T22:16:24.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: 23.775015+0000 mon.a (mon.0) 263 : audit [INF] 2021-03-15T22:16:24.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:24.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit 2021-03-15T22:16:24.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: 2021-03-15T22:16:23.775557+0000 mon.a (mon.0) 264 : audit 2021-03-15T22:16:24.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:24.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit 2021-03-15T22:16:24.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: 2021-03-15T22:16:23.952823+0000 mon.a (mon.0) 265 : 2021-03-15T22:16:24.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:24.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: cluster 2021-03-15T22:16:23.964753+0000 2021-03-15T22:16:24.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: mon.a (mon.0) 266 : cluster [INF] osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050] boot 2021-03-15T22:16:24.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: cluster 2021-03-15T22:16:23 2021-03-15T22:16:24.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: .964788+0000 mon.a (mon.0) 267 : cluster [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-03-15T22:16:24.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: 2021-03-15T22:16:24.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:16:24.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: 03-15T22:16:23.965049+0000 mon.a (mon.0) 268 : audit [DBG] 2021-03-15T22:16:24.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:24.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: cluster 2021-03-15T22:16:22.060716+0000 2021-03-15T22:16:24.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:24.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:16:24.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 22:16:22.060800+0000 osd.2 (osd.2) 2 : cluster [DBG] 2021-03-15T22:16:24.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: purged_snaps scrub ok 2021-03-15T22:16:24.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:16:24.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: -03-15T22:16:23 2021-03-15T22:16:24.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: .453507+0000 mgr.y (mgr.14144) 107 : 2021-03-15T22:16:24.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: cephadm [INF] Detected new or changed devices on smithi076 2021-03-15T22:16:24.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: audit 2021 2021-03-15T22:16:24.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: -03-15T22:16: 2021-03-15T22:16:24.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 23.457637+0000 mon.a (mon.0) 258 2021-03-15T22:16:24.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:24.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:16:24.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: -03-15T22:16: 2021-03-15T22:16:24.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 23.728602+0000 mgr.y (mgr.14144) 108 2021-03-15T22:16:24.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:24.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:16:24.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: -15T22:16:23.734133 2021-03-15T22:16:24.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: +0000 mon.a (mon.0) 259 : audit 2021-03-15T22:16:24.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:24.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: cephadm 2021-03 2021-03-15T22:16:24.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: -15T22:16:23.735019 2021-03-15T22:16:24.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 109 : cephadm 2021-03-15T22:16:24.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:24.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:16:24.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:24.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:16:24.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:24.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:16:24.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:24.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:24.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:24.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: audit 2021-03-15T22:16:23.735436+0000 mon.a (mon.0) 260 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:24.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:23 2021-03-15T22:16:24.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: .736917+0000 mgr.y (mgr.14144) 110 : cephadm 2021-03-15T22:16:24.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:24.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:16:24.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: -15T22:16:23.742248 2021-03-15T22:16:24.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: +0000 mon.a (mon.0) 261 : audit 2021-03-15T22:16:24.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:24.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:16:24.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 03-15T22:16:23 2021-03-15T22:16:24.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: .742997+0000 mgr.y (mgr.14144) 2021-03-15T22:16:24.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 111 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:24.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:16:24.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:24.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:16:24.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:24.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:16:24.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:24.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:24.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:24.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: audit 2021-03-15T22:16:23.773125+0000 mgr.y (mgr.14144) 112 : audit [DBG] from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:24.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: audit 2021-03-15T22:16:24.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 2021-03-15T22:16 2021-03-15T22:16:24.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: :23.773800+0000 mon.a (mon.0) 2021-03-15T22:16:24.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 262 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:24.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: cephadm 2021-03 2021-03-15T22:16:24.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: -15T22:16:23.774802 2021-03-15T22:16:24.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 113 : cephadm 2021-03-15T22:16:24.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: [INF] Applying service osd.None on host smithi076... 2021-03-15T22:16:24.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:16:24.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 15T22:16:23.775015+0000 2021-03-15T22:16:24.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: mon.a (mon.0) 263 : audit [INF] 2021-03-15T22:16:24.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:24.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:16:24.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 16:23.775557+0000 mon.a (mon.0 2021-03-15T22:16:24.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: ) 264 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:24.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:16:24.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 03-15T22:16:23.952823 2021-03-15T22:16:24.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: +0000 mon.a (mon.0) 265 : audit 2021-03-15T22:16:24.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:24.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:16:24.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: -15T22:16:23.964753 2021-03-15T22:16:24.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: +0000 mon.a (mon.0) 266 : cluster 2021-03-15T22:16:24.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: [INF] osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050] boot 2021-03-15T22:16:24.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:16:24.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: -15T22:16:23.964788 2021-03-15T22:16:24.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: +0000 mon.a (mon.0) 267 : cluster 2021-03-15T22:16:24.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: [DBG] osdmap e18: 3 total, 3 up, 3 in 2021-03-15T22:16:24.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:16:24.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 15T22:16:23.965049+0000 2021-03-15T22:16:24.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: mon.a (mon.0) 268 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:24.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:24 smithi076 conmon[62156]: 2021-03-15T22:16:24.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: cluster 2021-03-15T22:16:22.060716+0000 osd.2 (osd. 2021-03-15T22:16:24.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 2) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:24.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: cluster 2021-03-15T22:16:22.060800+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:24.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 2021-03-15T22:16:24.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: cephadm 2021-03-15T 2021-03-15T22:16:24.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 22:16:23.453507+0000 2021-03-15T22:16:24.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: mgr.y (mgr.14144) 107 : cephadm 2021-03-15T22:16:24.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: [INF] Detected new or changed devices on smithi076 2021-03-15T22:16:24.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:24.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 03-15T22:16:23.457637+0000 2021-03-15T22:16:24.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: mon.a (mon.0) 258 : audit 2021-03-15T22:16:24.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:24.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: cephadm 2021- 2021-03-15T22:16:24.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 03-15T22:16:23. 2021-03-15T22:16:24.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 728602+0000 mgr.y (mgr.14144 2021-03-15T22:16:24.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: ) 108 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:24.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:24.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: -03-15T22:16:23. 2021-03-15T22:16:24.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 734133+0000 mon.a (mon.0) 2021-03-15T22:16:24.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 259 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:24.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: cephadm 2021 2021-03-15T22:16:24.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: -03-15T22:16:23. 2021-03-15T22:16:24.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 735019+0000 mgr.y (mgr.14144 2021-03-15T22:16:24.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: ) 109 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:24.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:16:24.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:24.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:16:24.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:24.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:16:24.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:24.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:24.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:24.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 2021-03-15T22:16:24.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: audit 2021-03-15T22:16:23.735436+0000 mon.a (mon.0 2021-03-15T22:16:24.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: ) 260 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:24.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: cephadm 2021-03-15T22:16: 2021-03-15T22:16:24.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 23.736917+0000 mgr.y (mgr 2021-03-15T22:16:24.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: .14144) 110 : cephadm [INF] 2021-03-15T22:16:24.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:24.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:24.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 03-15T22:16:23.742248+0000 2021-03-15T22:16:24.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: mon.a (mon.0) 261 : 2021-03-15T22:16:24.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:24.812 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: cephadm 2021 2021-03-15T22:16:24.812 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: -03-15T22:16:23.742997 2021-03-15T22:16:24.812 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 2021-03-15T22:16:24.813 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 111 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:24.813 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:16:24.813 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:24.813 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:16:24.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:24.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:16:24.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:24.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:24.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:24.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: audit 2021-03-15T22:16 2021-03-15T22:16:24.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: :23.773125+0000 mgr.y ( 2021-03-15T22:16:24.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: mgr.14144) 112 : audit [DBG] 2021-03-15T22:16:24.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi076:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:24.816 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:24.816 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: -03-15T22:16:23. 2021-03-15T22:16:24.816 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 773800+0000 mon.a (mon.0 2021-03-15T22:16:24.816 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: ) 262 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:24.816 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 2021-03-15T22:16:24.817 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: cephadm 2021- 2021-03-15T22:16:24.817 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 03-15T22:16:23.774802 2021-03-15T22:16:24.817 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 113 : cephadm [INF] Applying service osd.None on host smithi076... 2021-03-15T22:16:24.817 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:24.817 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: -03-15T22:16:23 2021-03-15T22:16:24.818 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: .775015+0000 mon.a (mon. 2021-03-15T22:16:24.818 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 0) 263 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:24.818 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 2021-03-15T22:16:24.818 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:24.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: -03-15T22:16:23 2021-03-15T22:16:24.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: .775557+0000 mon.a (mon. 2021-03-15T22:16:24.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 0) 264 : audit [DBG] 2021-03-15T22:16:24.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:24.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:24.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 03-15T22:16:23.952823 2021-03-15T22:16:24.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: +0000 mon.a (mon.0) 265 2021-03-15T22:16:24.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:24.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:16:24.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: -03-15T22:16:23 2021-03-15T22:16:24.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: .964753+0000 mon.a (mon. 2021-03-15T22:16:24.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 0) 266 : cluster [INF] osd.2 [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050] boot 2021-03-15T22:16:24.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 2021-03-15T22:16:24.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:16:24.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: -03-15T22:16:23 2021-03-15T22:16:24.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: .964788+0000 mon.a (mon. 2021-03-15T22:16:24.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 0) 267 : cluster [DBG] 2021-03-15T22:16:24.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: osdmap e18: 3 total, 3 up, 3 in 2021-03-15T22:16:24.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:24.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: 03-15T22:16:23.965049 2021-03-15T22:16:24.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: +0000 mon.a (mon.0) 268 2021-03-15T22:16:24.824 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:24 smithi125 conmon[56739]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:16:26.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[56514]: cluster 2021-03-15T22:16:24.922825+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:26.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[56514]: cluster 2021-03-15T22:16:24.967925+0000 mon.a (mon.0) 269 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-03-15T22:16:26.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[56514]: audit 2021-03-15T22:16:25.783173+0000 mon.c (mon.1) 9 : audit [INF] from='client.? 172.21.15.76:0/776983110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56"}]: dispatch 2021-03-15T22:16:26.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[56514]: audit 2021-03-15T22:16:25.783513+0000 mon.a (mon.0) 270 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56"}]: dispatch 2021-03-15T22:16:26.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[56514]: audit 2021-03-15T22:16:25.789433+0000 mon.a (mon.0) 271 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56"}]': finished 2021-03-15T22:16:26.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[56514]: cluster 2021-03-15T22:16:25.789486+0000 mon.a (mon.0) 272 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-03-15T22:16:26.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[56514]: audit 2021-03-15T22:16:25.789585+0000 mon.a (mon.0) 273 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:26.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: cluster 2021-03-15T22:16:24.922825+0000 mgr.y (mgr.14144) 114 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:26.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:16:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: 22:16:24.967925+0000 mon.a (mon.0) 269 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-03-15T22:16:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: audit 2021-03-15T22:16:25.783173+0000 mon.c 2021-03-15T22:16:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: (mon.1) 9 : audit [INF] from='client.? 172.21.15.76:0/776983110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56"}]: dispatch 2021-03-15T22:16:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: audit 2021 2021-03-15T22:16:26.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: -03-15T22:16:25. 2021-03-15T22:16:26.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: 783513+0000 mon.a (mon.0) 2021-03-15T22:16:26.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: 270 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56"}]: dispatch 2021-03-15T22:16:26.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: audit 2021 2021-03-15T22:16:26.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: -03-15T22:16 2021-03-15T22:16:26.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: :25.789433+0000 mon.a ( 2021-03-15T22:16:26.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: mon.0) 271 : audit [INF] 2021-03-15T22:16:26.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56"}]': finished 2021-03-15T22:16:26.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:16:26.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: -03-15T22:16:25 2021-03-15T22:16:26.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: .789486+0000 mon.a (mon. 2021-03-15T22:16:26.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: 0) 272 : cluster [DBG] 2021-03-15T22:16:26.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: osdmap e20: 4 total, 3 up, 4 in 2021-03-15T22:16:26.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: audit 2021-03-15T22:16:26.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: 2021-03-15T22:16: 2021-03-15T22:16:26.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: 25.789585+0000 mon.a (mon 2021-03-15T22:16:26.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: .0) 273 : audit [DBG] 2021-03-15T22:16:26.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:25 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: cluster 2021-03-15T22:16:24.922825+0000 mgr.y (mgr.14144) 114 : 2021-03-15T22:16:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 9.6 MiB used, 179 GiB / 179 GiB avail 2021-03-15T22:16:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: cluster 2021-03-15T22:16:24.967925+0000 mon.a (mon.0) 269 : cluster [DBG] osdmap e19: 3 total, 3 up, 3 in 2021-03-15T22:16:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:16:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: 15T22:16:25.783173+0000 2021-03-15T22:16:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: mon.c (mon.1) 9 : 2021-03-15T22:16:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: audit [INF] from='client.? 172.21.15.76:0/776983110' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56"}]: dispatch 2021-03-15T22:16:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:26.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: 03-15T22:16:25.783513+0000 2021-03-15T22:16:26.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: mon.a (mon.0) 270 : 2021-03-15T22:16:26.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56"}]: dispatch 2021-03-15T22:16:26.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:16:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: 03-15T22:16:25.789433 2021-03-15T22:16:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2021-03-15T22:16:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: 271 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56"}]': finished 2021-03-15T22:16:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:16:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: -03-15T22:16:25. 2021-03-15T22:16:26.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: 789486+0000 mon.a (mon.0 2021-03-15T22:16:26.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: ) 272 : cluster [DBG] osdmap e20: 4 total, 3 up, 4 in 2021-03-15T22:16:26.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: 2021-03-15T22:16:26.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: audit 2021 2021-03-15T22:16:26.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: -03-15T22:16:25 2021-03-15T22:16:26.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: .789585+0000 mon.a (mon. 2021-03-15T22:16:26.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: 0) 273 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:26.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:25 smithi125 conmon[56739]: 2021-03-15T22:16:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:26 smithi076 conmon[56514]: audit 2021-03-15T22:16:26.273335+0000 mon.c (mon.1) 10 : audit 2021-03-15T22:16:27.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:26 smithi076 conmon[56514]: [DBG] from='client.? 172.21.15.76:0/3648338043' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:16:27.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:26 smithi076 conmon[62156]: audit 2021-03-15T22:16:26.273335+0000 mon.c (mon.1) 10 : audit [DBG] from='client.? 172.21.15.76:0/3648338043' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:16:27.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:26 smithi076 conmon[62156]: 2021-03-15T22:16:27.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:26 smithi125 conmon[56739]: audit 2021-03-15T22:16:26.273335+0000 mon.c (mon. 2021-03-15T22:16:27.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:26 smithi125 conmon[56739]: 1) 10 : audit [DBG] from='client.? 172.21.15.76:0/3648338043' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:16:28.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:27 smithi076 conmon[56514]: cluster 2021-03-15T22:16:26.923130+0000 mgr.y ( 2021-03-15T22:16:28.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:27 smithi076 conmon[56514]: mgr.14144) 115 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:27 smithi076 conmon[62156]: cluster 2021-03-15T22:16:26.923130+0000 mgr.y (mgr.14144) 115 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:28.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:27 smithi125 conmon[56739]: cluster 2021-03-15T22:16:26.923130+0000 mgr.y ( 2021-03-15T22:16:28.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:27 smithi125 conmon[56739]: mgr.14144) 115 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:30.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:29 smithi076 conmon[56514]: cluster 2021-03-15T22:16:28.923426+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:30.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:29 smithi076 conmon[62156]: cluster 2021-03-15T22:16:28.923426+0000 mgr.y (mgr.14144) 116 : cluster [DBG] 2021-03-15T22:16:30.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:29 smithi076 conmon[62156]: pgmap v84: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:30.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:29 smithi125 conmon[56739]: cluster 2021-03-15T22:16:28.923426+0000 mgr.y (mgr.14144) 116 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:32.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:31 smithi076 conmon[56514]: cluster 2021-03-15T22:16:30 2021-03-15T22:16:32.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:31 smithi076 conmon[56514]: .923654+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:32.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:31 smithi076 conmon[56514]: audit 2021-03-15T22:16:31.420713+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-03-15T22:16:32.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:31 smithi076 conmon[56514]: audit 2021-03-15T22:16:31.421322+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:32.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:31 smithi076 conmon[62156]: cluster 2021-03-15T22:16:30 2021-03-15T22:16:32.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:31 smithi076 conmon[62156]: .923654+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:32.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:31 smithi076 conmon[62156]: audit 2021-03-15T22:16:31.420713+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-03-15T22:16:32.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:31 smithi076 conmon[62156]: audit 2021-03-15T22:16:31.421322+0000 mon.a (mon.0) 275 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:32.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:31 smithi125 conmon[56739]: cluster 2021-03-15T22:16:30.923654+0000 mgr.y (mgr.14144) 117 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 14 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:32.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:31 smithi125 conmon[56739]: audit 2021-03-15T22:16:31.420713+0000 mon.a (mon.0) 274 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2021-03-15T22:16:32.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:31 smithi125 conmon[56739]: audit 2021-03-15T22:16:31.421322+0000 2021-03-15T22:16:32.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:31 smithi125 conmon[56739]: mon.a (mon.0) 275 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:33.219 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:32 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:31.421808+0000 mgr.y (mgr.14144) 118 : cephadm [INF] Deploying daemon osd.3 on smithi076 2021-03-15T22:16:33.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:32 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:31.421808+0000 mgr.y (mgr.14144 2021-03-15T22:16:33.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:32 smithi076 conmon[56514]: ) 118 : cephadm [INF] Deploying daemon osd.3 on smithi076 2021-03-15T22:16:33.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:32 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:31.421808+0000 mgr.y (mgr 2021-03-15T22:16:33.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:32 smithi125 conmon[56739]: .14144) 118 : cephadm [INF] Deploying daemon osd.3 on smithi076 2021-03-15T22:16:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:33 smithi076 conmon[56514]: cluster 2021-03-15T22:16:32.923911+0000 mgr.y (mgr.14144) 119 : 2021-03-15T22:16:34.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:33 smithi076 conmon[56514]: cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:34.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:33 smithi076 conmon[62156]: cluster 2021-03-15T22:16:32.923911+0000 2021-03-15T22:16:34.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:33 smithi076 conmon[62156]: mgr.y (mgr.14144) 119 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:34.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:33 smithi125 conmon[56739]: cluster 2021-03-15T22:16:32.923911+0000 mgr.y (mgr.14144) 119 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:34.542 INFO:teuthology.orchestra.run.smithi076.stdout:Created osd(s) 3 on host 'smithi076' 2021-03-15T22:16:35.070 DEBUG:teuthology.orchestra.run.smithi076:osd.3> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.3.service 2021-03-15T22:16:35.073 INFO:tasks.cephadm:Deploying osd.4 on smithi125 with /dev/vg_nvme/lv_4... 2021-03-15T22:16:35.073 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2021-03-15T22:16:35.102 INFO:journalctl@ceph.osd.3.smithi076.stdout:-- Logs begin at Mon 2021-03-15 21:44:23 UTC. -- 2021-03-15T22:16:35.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:34 smithi076 conmon[56514]: audit 2021-03-15T22:16:34.528991+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:35.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:34 smithi076 conmon[56514]: audit 2021-03-15T22:16:34.532832+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:35.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:34 smithi076 conmon[56514]: audit 2021-03-15T22:16:34.538020+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:35.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:34 smithi076 conmon[56514]: audit 2021-03-15T22:16:34.539585+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-03-15T22:16:35.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:34 smithi076 conmon[62156]: audit 2021-03-15T22:16:34.528991+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:35.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:34 smithi076 conmon[62156]: audit 2021-03-15T22:16:34.532832+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:35.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:34 smithi076 conmon[62156]: audit 2021-03-15T22:16:34.538020+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:35.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:34 smithi076 conmon[62156]: audit 2021-03-15T22:16:34.539585+0000 mon.a (mon.0) 279 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-03-15T22:16:35.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:34 smithi125 conmon[56739]: audit 2021-03-15T22:16:34.528991+0000 mon.a (mon.0) 276 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:35.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:34 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:16:35.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:34 smithi125 conmon[56739]: T22:16:34.532832+0000 mon.a (mon.0) 277 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:34 smithi125 conmon[56739]: audit 2021-03-15T22:16:34.538020+0000 mon.a (mon.0) 278 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:34 smithi125 conmon[56739]: 2021-03-15T22:16:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:34 smithi125 conmon[56739]: audit 2021-03-15T22:16:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:34 smithi125 conmon[56739]: 2021-03-15T22:16:34.539585+0000 mon.a (mon 2021-03-15T22:16:35.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:34 smithi125 conmon[56739]: .0) 279 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]: dispatch 2021-03-15T22:16:36.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:35 smithi125 conmon[56739]: cluster 2021-03-15T22:16:34.924492+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:36.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:35 smithi125 conmon[56739]: audit 2021-03-15T22:16:35.543213+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-03-15T22:16:36.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:35 smithi125 conmon[56739]: cluster 2021-03-15T22:16:35.543265+0000 mon.a (mon.0) 281 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-03-15T22:16:36.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:35 smithi125 conmon[56739]: audit 2021-03-15T22:16:35.543397+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:36.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:35 smithi125 conmon[56739]: audit 2021-03-15T22:16:35.544079+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-03-15T22:16:36.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:35 smithi076 conmon[56514]: cluster 2021-03-15T22:16:34.924492+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:36.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:35 smithi076 conmon[56514]: audit 2021-03-15T22:16:35.543213+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-03-15T22:16:36.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:35 smithi076 conmon[56514]: cluster 2021-03-15T22:16:35.543265+0000 mon.a (mon.0) 281 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-03-15T22:16:36.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:35 smithi076 conmon[56514]: audit 2021-03-15T22:16:35.543397+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:36.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:35 smithi076 conmon[56514]: audit 2021-03-15T22:16:35.544079+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-03-15T22:16:36.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:35 smithi076 conmon[62156]: cluster 2021-03-15T22:16:34.924492+0000 mgr.y (mgr.14144) 120 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:36.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:35 smithi076 conmon[62156]: audit 2021-03-15T22:16:35.543213+0000 mon.a (mon.0) 280 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": "device_health_metrics", "pg_num": 1, "pg_num_min": 1}]': finished 2021-03-15T22:16:36.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:35 smithi076 conmon[62156]: cluster 2021-03-15T22:16:35.543265+0000 mon.a (mon.0) 281 : cluster [DBG] osdmap e21: 4 total, 3 up, 4 in 2021-03-15T22:16:36.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:35 smithi076 conmon[62156]: audit 2021-03-15T22:16:35.543397+0000 mon.a (mon.0) 282 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:36.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:35 smithi076 conmon[62156]: audit 2021-03-15T22:16:35.544079+0000 mon.a (mon.0) 283 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]: dispatch 2021-03-15T22:16:36.993 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping: /dev/vg_nvme/lv_4 2021-03-15T22:16:36.994 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2021-03-15T22:16:36.994 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10+0 records in 2021-03-15T22:16:36.994 INFO:teuthology.orchestra.run.smithi125.stderr:10+0 records out 2021-03-15T22:16:36.995 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0109663 s, 956 MB/s 2021-03-15T22:16:36.995 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping successful for: 2021-03-15T22:16:37.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[56514]: audit 2021-03-15T22:16:36.052185+0000 mon.a (mon. 2021-03-15T22:16:37.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[56514]: 0) 284 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:37.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[56514]: audit 2021-03-15T22:16:36.546052+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-03-15T22:16:37.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[56514]: cluster 2021-03-15T22:16:36.546112+0000 mon.a (mon.0) 286 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-03-15T22:16:37.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[56514]: audit 2021-03-15T22:16:36.546238+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:37.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[56514]: cluster 2021-03-15T22:16:36.924978+0000 mgr.y (mgr.14144) 121 : cluster [DBG] pgmap v90: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:37.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[62156]: audit 2021-03-15T22:16:36.052185+0000 mon.a 2021-03-15T22:16:37.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[62156]: (mon.0) 284 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:37.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[62156]: audit 2021-03-15T22:16:36.546052+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-03-15T22:16:37.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[62156]: cluster 2021-03-15T22:16:36.546112+0000 mon.a (mon.0) 286 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-03-15T22:16:37.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[62156]: audit 2021-03-15T22:16:36.546238+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:37.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[62156]: cluster 2021-03-15T22:16:36.924978+0000 mgr.y (mgr.14144) 121 : cluster [DBG] pgmap v90: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:37.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:37 smithi125 conmon[56739]: audit 2021-03-15T22:16:36.052185+0000 mon.a (mon.0) 284 2021-03-15T22:16:37.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:37 smithi125 conmon[56739]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:37 smithi125 conmon[56739]: audit 2021-03-15T22:16:36.546052+0000 mon.a (mon.0) 285 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": "device_health_metrics", "app": "mgr_devicehealth"}]': finished 2021-03-15T22:16:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:37 smithi125 conmon[56739]: cluster 2021-03-15T22:16:36.546112+0000 mon.a (mon.0) 286 : cluster [DBG] osdmap e22: 4 total, 3 up, 4 in 2021-03-15T22:16:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:37 smithi125 conmon[56739]: audit 2021-03-15T22:16:36.546238+0000 mon.a (mon.0) 287 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:37 smithi125 conmon[56739]: cluster 2021-03-15T22:16:36.924978+0000 mgr.y (mgr.14144) 121 : cluster [DBG] pgmap v90: 1 pgs: 1 unknown; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:37.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch daemon add osd smithi125:vg_nvme/lv_4 2021-03-15T22:16:37.767 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:16:37 smithi076 conmon[75764]: debug 2021-03-15T22:16:37.694+0000 7f9fe3aad080 -1 osd.3 0 log_to_monitors {default=true} 2021-03-15T22:16:38.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:38 smithi076 conmon[56514]: cluster 2021-03-15T22:16:37.546606+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-03-15T22:16:38.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:38 smithi076 conmon[56514]: audit 2021-03-15T22:16:37.546733+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:38.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:38 smithi076 conmon[56514]: audit 2021-03-15T22:16:37.698733+0000 mon.c (mon.1) 11 : audit [INF] from='osd.3 [v2:172.21.15.76:6826/3960968636,v1:172.21.15.76:6827/3960968636]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-03-15T22:16:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:38 smithi076 conmon[56514]: audit 2021-03-15T22:16:37.699024+0000 mon.a (mon.0) 290 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-03-15T22:16:38.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:38 smithi076 conmon[62156]: cluster 2021-03-15T22:16:37.546606+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-03-15T22:16:38.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:38 smithi076 conmon[62156]: audit 2021-03-15T22:16:37.546733+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:38.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:38 smithi076 conmon[62156]: audit 2021-03-15T22:16:37.698733+0000 mon.c (mon.1) 11 : audit [INF] from='osd.3 [v2:172.21.15.76:6826/3960968636,v1:172.21.15.76:6827/3960968636]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-03-15T22:16:38.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:38 smithi076 conmon[62156]: audit 2021-03-15T22:16:37.699024+0000 mon.a (mon.0) 290 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-03-15T22:16:38.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:38 smithi125 conmon[56739]: cluster 2021-03-15T22:16:37.546606+0000 mon.a (mon.0) 288 : cluster [DBG] osdmap e23: 4 total, 3 up, 4 in 2021-03-15T22:16:38.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:38 smithi125 conmon[56739]: audit 2021-03-15T22:16:37.546733+0000 mon.a (mon.0) 289 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:38 smithi125 conmon[56739]: audit 2021-03-15T22:16:37.698733+0000 mon.c (mon.1) 11 : audit [INF] from='osd.3 [v2:172.21.15.76:6826/3960968636,v1:172.21.15.76:6827/3960968636]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-03-15T22:16:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:38 smithi125 conmon[56739]: audit 2021-03-15T22:16:37.699024+0000 mon.a (mon.0) 290 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2021-03-15T22:16:39.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: debug 2021-03-15T22:16:39.119+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:39.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:16:39.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:39.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:16:39.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:39.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:16:39.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:39.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:39.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:39.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: debug 2021-03-15T22:16:39.125+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:39.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:16:39.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:39.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:16:39.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:39.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:16:39.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:39.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:39.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:39.958 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: audit 2021-03-15T22:16:38.548013+0000 mon.a (mon.0) 291 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-03-15T22:16:39.958 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: cluster 2021-03-15T22:16:38. 2021-03-15T22:16:39.959 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: 548088+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-03-15T22:16:39.959 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: audit 2021-03-15T22:16:38.548416+0000 mon.a (mon.0) 293 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:39.959 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: audit 2021-03-15T22:16:38.549416+0000 mon.c (mon.1) 12 : audit [INF] from='osd.3 [v2:172.21.15.76:6826/3960968636,v1:172.21.15.76:6827/3960968636]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:39.959 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: audit 2021-03-15T22:16:38.552367+0000 mon.a (mon.0) 294 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:39.960 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: cluster 2021-03-15T22:16:38.925265+0000 mgr.y (mgr.14144) 122 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:39.960 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: audit 2021-03-15T22:16:39.117794+0000 mon.a (mon.0) 295 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:39.960 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: audit 2021-03-15T22:16:39.120815+0000 mon.a (mon.0) 296 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:39.960 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: audit 2021-03-15T22:16:39.121972+0000 mon.a (mon.0) 297 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:39.960 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: audit 2021-03-15T22:16:39.126474+0000 mon.a (mon.0) 298 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:39.961 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: audit 2021-03-15T22:16:39.396566+0000 mon.a (mon.0) 299 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:39.961 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: audit 2021-03-15T22:16:39.397956+0000 mon.a (mon.0) 300 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:39.961 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:39 smithi125 conmon[56739]: audit 2021-03-15T22:16:39.398522+0000 mon.a (mon.0) 301 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:40.017 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[75764]: debug 2021-03-15T22:16:39.558+0000 7f9fda01e700 -1 osd.3 0 waiting for initial osdmap 2021-03-15T22:16:40.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit 2021-03-15T22:16:38.548013+0000 mon.a (mon.0) 291 2021-03-15T22:16:40.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-03-15T22:16:40.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: cluster 2021-03-15T22:16:38.548088+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-03-15T22:16:40.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit 2021-03-15T22:16:38.548416+0000 mon.a (mon.0) 293 : audit [DBG] 2021-03-15T22:16:40.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:40.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit 2021-03-15T22:16:40.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: 2021-03-15T22:16:38.549416+0000 mon.c (mon. 2021-03-15T22:16:40.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: 1) 12 : audit [INF] from='osd.3 [v2:172.21.15.76:6826/3960968636,v1:172.21.15.76:6827/3960968636]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:40.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:16:40.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: 03-15T22:16:38.552367+0000 mon.a (mon. 2021-03-15T22:16:40.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: 0) 294 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:40.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:16:40.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: -15T22:16:38.925265+0000 mgr.y (mgr. 2021-03-15T22:16:40.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: 14144) 122 : cluster [DBG] pgmap v93: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:40.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:16:40.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: -15T22:16:39.117794+0000 mon.a ( 2021-03-15T22:16:40.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: mon.0) 295 : audit [INF] 2021-03-15T22:16:40.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:40.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit 2021-03-15T22:16:40.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: 2021-03-15T22:16:39.120815 2021-03-15T22:16:40.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: +0000 mon.a (mon.0) 296 : 2021-03-15T22:16:40.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:40.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:16:40.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: -15T22:16:39.121972+0000 mon.a ( 2021-03-15T22:16:40.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: mon.0) 297 : audit [DBG] 2021-03-15T22:16:40.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:40.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit 2021-03-15T22:16:40.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: 2021-03-15T22:16:39.126474+0000 2021-03-15T22:16:40.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: mon.a (mon.0) 298 : audit 2021-03-15T22:16:40.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:40.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:16:40.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: 03-15T22:16:39.396566+0000 mon.a ( 2021-03-15T22:16:40.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: mon.0) 299 : audit [DBG] 2021-03-15T22:16:40.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:40.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit 2021-03-15T22:16:40.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: 2021-03-15T22 2021-03-15T22:16:40.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: :16:39.397956+0000 mon.a (mon 2021-03-15T22:16:40.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: .0) 300 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:40.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: 2021-03-15T22:16:40.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: audit 2021 2021-03-15T22:16:40.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: -03-15T22:16:39.398522+0000 2021-03-15T22:16:40.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: mon.a (mon.0) 301 : audit 2021-03-15T22:16:40.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:40.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: audit 2021-03-15T22:16:38.548013 2021-03-15T22:16:40.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: +0000 mon.a (mon.0) 291 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2021-03-15T22:16:40.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:16:40.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: 03-15T22:16:38.548088+0000 mon.a (mon.0) 292 : cluster [DBG] osdmap e24: 4 total, 3 up, 4 in 2021-03-15T22:16:40.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:16:40.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: -15T22:16:38.548416+0000 mon.a (mon.0) 2021-03-15T22:16:40.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: 293 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:40.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:16:40.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: 03-15T22:16:38.549416+0000 2021-03-15T22:16:40.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: mon.c (mon.1) 12 : audit [INF] from='osd.3 [v2:172.21.15.76:6826/3960968636,v1:172.21.15.76:6827/3960968636]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:40.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: 2021-03-15T22:16:40.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: audit 2021-03-15T22:16:38.552367+0000 mon.a (mon. 2021-03-15T22:16:40.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: 0) 294 : audit [INF] from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:16:40.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: cluster 2021-03-15T22:16:40.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: 2021-03-15T22:16:38.925265+0000 2021-03-15T22:16:40.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: mgr.y (mgr.14144) 122 : cluster [DBG] 2021-03-15T22:16:40.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: pgmap v93: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:40.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:16:40.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: :16:39.117794+0000 mon.a (mon.0 2021-03-15T22:16:40.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: ) 295 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:40.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: audit 2021-03-15T22:16:40.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: 2021-03-15T22:16:39.120815+0000 2021-03-15T22:16:40.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: mon.a (mon.0) 296 : audit [INF] 2021-03-15T22:16:40.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:40.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: audit 2021-03-15T22:16 2021-03-15T22:16:40.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: :39.121972+0000 mon.a (mon.0) 297 2021-03-15T22:16:40.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:40.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:16:40.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: 15T22:16:39.126474+0000 mon.a ( 2021-03-15T22:16:40.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: mon.0) 298 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:40.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: audit 2021-03-15T22:16:40.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: 2021-03-15T22:16:39.396566+0000 2021-03-15T22:16:40.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: mon.a (mon.0) 299 : audit [DBG] 2021-03-15T22:16:40.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:40.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: audit 2021-03-15T22:16:40.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: 2021-03-15T22:16:39.397956+0000 mon.a (mon 2021-03-15T22:16:40.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: .0) 300 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:40.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:16:40.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: T22:16:39.398522+0000 mon.a (mon 2021-03-15T22:16:40.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:39 smithi076 conmon[62156]: .0) 301 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:40.826 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:39.113646 2021-03-15T22:16:40.827 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 123 : cephadm [INF] Detected new or changed devices on smithi076 2021-03-15T22:16:40.827 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:39.118372+0000 mgr.y (mgr.14144) 124 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:40.827 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:39.121497+0000 mgr.y (mgr.14144) 125 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:40.827 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:16:40.828 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:40.828 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:16:40.828 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:40.828 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:16:40.829 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:40.829 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:40.829 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:40.829 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:39.123167+0000 mgr.y (mgr.14144) 126 : cephadm 2021-03-15T22:16:40.830 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:40.830 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:39.127042+0000 mgr.y (mgr.14144) 127 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:40.830 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:16:40.830 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:40.831 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:16:40.831 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:40.831 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:16:40.831 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:40.831 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:40.832 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:40.832 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:16:40.832 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: T22:16:39.395930+0000 mgr.y (mgr.14144) 128 : audit [DBG] from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:40.832 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:39.397744+0000 mgr.y (mgr. 2021-03-15T22:16:40.833 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: 14144) 129 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-15T22:16:40.833 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:16:40.833 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: 22:16:39.555442+0000 mon.a (mon 2021-03-15T22:16:40.833 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: .0) 302 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:16:40.834 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: cluster 2021-03-15T22:16:40.834 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: 2021-03-15T22:16:39.555521 2021-03-15T22:16:40.834 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: +0000 mon.a (mon.0) 303 : cluster 2021-03-15T22:16:40.834 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-03-15T22:16:40.834 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: audit 2021-03-15T22:16:39 2021-03-15T22:16:40.835 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: .555949+0000 mon.a (mon.0) 304 : 2021-03-15T22:16:40.835 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:40.835 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:16:40.835 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: 22:16:39.560751+0000 mon.a (mon 2021-03-15T22:16:40.836 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:40 smithi125 conmon[56739]: .0) 305 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:41.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:39.113646+0000 mgr.y (mgr.14144) 123 : cephadm 2021-03-15T22:16:41.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: [INF] Detected new or changed devices on smithi076 2021-03-15T22:16:41.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:39.118372+0000 mgr.y (mgr.14144) 124 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:41.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:39.121497+0000 mgr.y (mgr.14144) 125 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:41.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:16:41.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:41.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:16:41.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:41.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:16:41.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:41.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:41.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:41.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:39.123167+0000 mgr.y (mgr.14144) 126 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:41.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:39.127042+0000 mgr.y (mgr 2021-03-15T22:16:41.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: .14144) 127 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:41.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:16:41.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:41.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:16:41.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:41.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:16:41.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:41.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:41.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:41.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: audit 2021-03-15T22:16:39.395930+0000 mgr.y (mgr.14144) 128 : audit [DBG] from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:41.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:39.397744+0000 2021-03-15T22:16:41.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: mgr.y (mgr.14144) 129 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-15T22:16:41.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: audit 2021-03-15T22:16:39.555442+0000 mon.a (mon.0) 302 : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:16:41.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:16:41.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: T22:16:39.555521+0000 mon.a (mon.0) 2021-03-15T22:16:41.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: 303 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-03-15T22:16:41.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:16:41.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: 03-15T22:16:39.555949+0000 mon.a (mon. 2021-03-15T22:16:41.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: 0) 304 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:41.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:16:41.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: 03-15T22:16:39.560751+0000 mon.a (mon.0) 305 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:41.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[56514]: 2021-03-15T22:16:41.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:41.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: 2021-03-15T22:16:39 2021-03-15T22:16:41.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: .113646+0000 mgr.y (mgr.14144) 123 : cephadm [INF] Detected new or changed devices on smithi076 2021-03-15T22:16:41.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:39.118372+0000 mgr.y (mgr.14144) 124 2021-03-15T22:16:41.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:41.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:41.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: 2021-03-15T22:16: 2021-03-15T22:16:41.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: 39.121497+0000 mgr.y (mgr.14144) 125 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:41.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:16:41.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:41.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:16:41.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:41.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:16:41.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:41.037 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:41.037 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:41.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: 2021-03-15T22:16:41.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:39.123167+0000 mgr.y (mgr.14144) 126 : 2021-03-15T22:16:41.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:41.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:41.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: 2021-03-15T22:16:39.127042+0000 mgr.y (mgr.14144) 2021-03-15T22:16:41.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: 127 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:41.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:16:41.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:41.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:16:41.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:41.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:16:41.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:41.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:41.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:41.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: audit 2021-03-15T22:16:39.395930+0000 mgr.y (mgr.14144) 128 : audit [DBG] from='client.34193 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:41.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:41.045 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: 2021-03-15T22:16:39.397744+0000 mgr.y (mgr.14144) 129 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-15T22:16:41.045 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:16:41.046 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: 15T22:16:39.555442+0000 mon.a (mon.0) 302 2021-03-15T22:16:41.046 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: : audit [INF] from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]': finished 2021-03-15T22:16:41.046 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: cluster 2021-03-15T22:16:39.555521 2021-03-15T22:16:41.047 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: +0000 mon.a (mon.0) 303 : cluster [DBG] osdmap e25: 4 total, 3 up, 4 in 2021-03-15T22:16:41.047 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: audit 2021-03-15T22:16:39. 2021-03-15T22:16:41.048 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: 555949+0000 mon.a (mon.0) 304 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:41.048 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: audit 2021-03-15T22:16:41.049 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: 2021-03-15T22:16 2021-03-15T22:16:41.049 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: :39.560751+0000 mon.a (mon.0) 305 : audit 2021-03-15T22:16:41.050 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:40 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:42.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: cluster 2021-03-15T22:16:38.697035+0000 osd.3 (osd.3 2021-03-15T22:16:42.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: ) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:42.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: cluster 2021-03-15T22:16:38.697203+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:42.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: cluster 2021-03-15T22:16:40.561873+0000 mon.a (mon.0) 306 2021-03-15T22:16:42.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: : cluster [INF] osd.3 [v2:172.21.15.76:6826/3960968636,v1:172.21.15.76:6827/3960968636] boot 2021-03-15T22:16:42.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:16:42.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: -15T22:16:40.561929+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-03-15T22:16:42.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: audit 2021-03-15T22:16: 2021-03-15T22:16:42.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: 40.562395+0000 mon.a (mon.0) 308 : audit 2021-03-15T22:16:42.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:42.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:16:42.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: -15T22:16:40. 2021-03-15T22:16:42.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: 925616+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:42.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:16:42.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: :16:41.433421+0000 mon.b ( 2021-03-15T22:16:42.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: mon.2) 6 : audit [INF] from='client.? 172.21.15.125:0/3678594868' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237"}]: dispatch 2021-03-15T22:16:42.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:16:42.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: 16:41.433997+0000 mon.a (mon.0) 309 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237"}]: dispatch 2021-03-15T22:16:42.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: audit 2021-03-15T22:16 2021-03-15T22:16:42.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: :41.438415+0000 mon.a (mon.0) 310 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237"}]': finished 2021-03-15T22:16:42.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:16:42.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: :16:41.438499+0000 mon.a (mon.0) 311 : cluster [DBG] 2021-03-15T22:16:42.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: osdmap e27: 5 total, 4 up, 5 in 2021-03-15T22:16:42.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: audit 2021-03-15T22:16:42.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: 2021-03-15T22:16:41.438648+0000 mon.a (mon.0) 2021-03-15T22:16:42.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[62156]: 312 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:42.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: cluster 2021-03-15T22:16:38.697035+0000 osd.3 (osd.3) 1 : cluster 2021-03-15T22:16:42.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: [DBG] purged_snaps scrub starts 2021-03-15T22:16:42.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: cluster 2021-03-15T22:16:38.697203+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:42.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: cluster 2021-03-15T22:16:42.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: 2021-03-15T22:16:40.561873+0000 mon.a (mon.0) 306 : cluster [INF] osd.3 [v2:172.21.15.76:6826/3960968636,v1:172.21.15.76:6827/3960968636] boot 2021-03-15T22:16:42.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: cluster 2021-03-15T22:16:42.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: 2021-03-15T22:16:40.561929+0000 mon.a 2021-03-15T22:16:42.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: (mon.0) 307 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-03-15T22:16:42.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:16:42.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: -15T22:16:40.562395+0000 mon.a (mon.0) 308 : audit [DBG] 2021-03-15T22:16:42.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:42.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:16:42.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: 16:40.925616+0000 mgr.y (mgr.14144) 130 2021-03-15T22:16:42.026 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:42.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: audit 2021-03-15T22:16:42.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: 2021-03-15T22:16:41.433421+0000 mon.b 2021-03-15T22:16:42.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: (mon.2) 6 : audit [INF] from='client.? 172.21.15.125:0/3678594868' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237"}]: dispatch 2021-03-15T22:16:42.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: 2021-03-15T22:16:42.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: audit 2021-03-15T22:16 2021-03-15T22:16:42.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: :41.433997+0000 mon.a (mon.0) 309 2021-03-15T22:16:42.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237"}]: dispatch 2021-03-15T22:16:42.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: audit 2021-03-15T22:16:42.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: 2021-03-15T22:16:41.438415+0000 2021-03-15T22:16:42.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: mon.a (mon.0) 310 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237"}]': finished 2021-03-15T22:16:42.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: 2021-03-15T22:16:42.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: cluster 2021-03-15T22:16 2021-03-15T22:16:42.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: :41.438499+0000 mon.a (mon.0) 311 : 2021-03-15T22:16:42.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2021-03-15T22:16:42.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: audit 2021 2021-03-15T22:16:42.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: -03-15T22:16:41.438648+0000 mon.a 2021-03-15T22:16:42.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:41 smithi076 conmon[56514]: (mon.0) 312 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:42.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: cluster 2021-03-15T22:16:38.697035+0000 osd.3 (osd.3) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:42.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: cluster 2021-03-15T22:16:38.697203+0000 osd.3 (osd.3) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:42.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: cluster 2021-03-15T22:16:40.561873+0000 mon.a (mon.0) 306 : cluster [INF] osd.3 [v2:172.21.15.76:6826/3960968636,v1:172.21.15.76:6827/3960968636] boot 2021-03-15T22:16:42.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: cluster 2021-03-15T22:16:40.561929+0000 mon.a (mon.0) 307 : cluster [DBG] osdmap e26: 4 total, 4 up, 4 in 2021-03-15T22:16:42.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: audit 2021-03-15T22:16:40.562395+0000 mon.a (mon.0) 308 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2021-03-15T22:16:42.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: cluster 2021-03-15T22:16:40.925616+0000 mgr.y (mgr.14144) 130 : cluster [DBG] pgmap v96: 1 pgs: 1 active+clean; 0 B data, 15 MiB used, 268 GiB / 268 GiB avail 2021-03-15T22:16:42.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: audit 2021-03-15T22:16:41.433421+0000 mon.b (mon.2) 6 : audit [INF] from='client.? 172.21.15.125:0/3678594868' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237"}]: dispatch 2021-03-15T22:16:42.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: audit 2021-03-15T22:16:41.433997+0000 mon.a (mon.0) 309 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237"}]: dispatch 2021-03-15T22:16:42.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: audit 2021-03-15T22:16:41.438415+0000 mon.a (mon.0) 310 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237"}]': finished 2021-03-15T22:16:42.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: cluster 2021-03-15T22:16:42.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: 2021-03-15T22:16:41.438499+0000 mon.a (mon.0) 311 : cluster [DBG] osdmap e27: 5 total, 4 up, 5 in 2021-03-15T22:16:42.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:41 smithi125 conmon[56739]: audit 2021-03-15T22:16:41.438648+0000 mon.a (mon.0) 312 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:43.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:42 smithi076 conmon[56514]: audit 2021-03-15T22:16:41.901019+0000 mon.b (mon.2) 7 : 2021-03-15T22:16:43.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:42 smithi076 conmon[56514]: audit [DBG] from='client.? 172.21.15.125:0/121130922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:16:43.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:42 smithi076 conmon[62156]: audit 2021-03-15T22:16:41.901019+0000 mon.b (mon 2021-03-15T22:16:43.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:42 smithi076 conmon[62156]: .2) 7 : audit [DBG] from='client.? 172.21.15.125:0/121130922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:16:43.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:42 smithi125 conmon[56739]: audit 2021-03-15T22:16:41.901019+0000 mon.b (mon.2) 7 : audit 2021-03-15T22:16:43.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:42 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.125:0/121130922' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:16:44.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:43 smithi076 conmon[56514]: cluster 2021-03-15T22:16:42.926209 2021-03-15T22:16:44.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:43 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:44.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:43 smithi076 conmon[62156]: cluster 2021-03-15T22:16:42.926209 2021-03-15T22:16:44.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:43 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:44.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:43 smithi125 conmon[56739]: cluster 2021-03-15T22:16:42.926209+0000 mgr.y (mgr.14144) 131 : cluster [DBG] pgmap v98: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:46.196 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:45 smithi125 conmon[56739]: cluster 2021-03-15T22:16:44.926759+0000 mgr.y (mgr.14144) 132 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:46.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:45 smithi076 conmon[56514]: cluster 2021-03-15T22:16:44.926759+0000 mgr.y (mgr. 2021-03-15T22:16:46.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:45 smithi076 conmon[56514]: 14144) 132 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:46.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:45 smithi076 conmon[62156]: cluster 2021-03-15T22:16:44.926759+0000 mgr.y ( 2021-03-15T22:16:46.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:45 smithi076 conmon[62156]: mgr.14144) 132 : cluster [DBG] pgmap v99: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:48.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[56514]: cluster 2021-03-15T22:16:46.927294+0000 mgr.y (mgr.14144) 133 : cluster [DBG] 2021-03-15T22:16:48.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[56514]: pgmap v100: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:48.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[56514]: audit 2021-03-15T22:16:47.090519+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-03-15T22:16:48.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[56514]: audit 2021-03-15T22:16:47.091325+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:48.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[62156]: cluster 2021-03-15T22:16:46.927294+0000 mgr.y (mgr.14144) 2021-03-15T22:16:48.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[62156]: 133 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:48.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[62156]: audit 2021-03-15T22:16:47.090519+0000 mon.a (mon.0) 313 : audit 2021-03-15T22:16:48.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-03-15T22:16:48.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:16:48.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[62156]: T22:16:47.091325+0000 2021-03-15T22:16:48.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[62156]: mon.a (mon.0) 314 : audit 2021-03-15T22:16:48.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:47 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:48.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:47 smithi125 conmon[56739]: cluster 2021-03-15T22:16:46.927294+0000 mgr.y 2021-03-15T22:16:48.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:47 smithi125 conmon[56739]: (mgr.14144) 133 : cluster [DBG] pgmap v100: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:48.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:47 smithi125 conmon[56739]: audit 2021-03-15T22:16:47.090519+0000 mon.a (mon.0) 313 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2021-03-15T22:16:48.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:47 smithi125 conmon[56739]: audit 2021-03-15T22:16:47.091325+0000 mon.a (mon.0) 314 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:49.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:47.092034+0000 mgr.y (mgr.14144) 134 : cephadm [INF] Deploying daemon osd.4 on smithi125 2021-03-15T22:16:49.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:48 smithi076 conmon[62156]: 2021-03-15T22:16:49.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:47.092034+0000 mgr.y (mgr.14144) 2021-03-15T22:16:49.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:48 smithi076 conmon[56514]: 134 : cephadm [INF] Deploying daemon osd.4 on smithi125 2021-03-15T22:16:49.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:47.092034+0000 mgr.y (mgr.14144) 134 : cephadm [INF] Deploying daemon osd.4 on smithi125 2021-03-15T22:16:50.168 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:49 smithi125 conmon[56739]: cluster 2021-03-15T22:16:48.924959+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:50.168 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:49 smithi125 conmon[56739]: cluster 2021-03-15T22:16:48.925148+0000 osd.0 (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:50.169 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:49 smithi125 conmon[56739]: cluster 2021-03-15T22:16:48.927644+0000 mgr.y (mgr.14144) 135 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:50.169 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:49 smithi125 conmon[56739]: cluster 2021-03-15T22:16:48.942374+0000 mon.a (mon.0) 315 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-03-15T22:16:50.169 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:49 smithi125 conmon[56739]: audit 2021-03-15T22:16:48.942631+0000 mon.a (mon 2021-03-15T22:16:50.169 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:49 smithi125 conmon[56739]: .0) 316 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:50.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: cluster 2021-03-15T22:16:48.924959+0000 osd.0 (osd. 2021-03-15T22:16:50.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: 0) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:50.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: cluster 2021-03-15T22:16:48.925148+0000 osd.0 2021-03-15T22:16:50.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: (osd.0) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:50.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: cluster 2021-03-15T22:16:50.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: 2021-03-15T22:16:48 2021-03-15T22:16:50.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: .927644+0000 mgr.y (mgr.14144) 2021-03-15T22:16:50.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: 135 : cluster [DBG] pgmap v101: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:50.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:16:50.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: 22:16:48.942374+0000 mon.a (mon 2021-03-15T22:16:50.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: .0) 315 : cluster [DBG] 2021-03-15T22:16:50.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: osdmap e28: 5 total, 4 up, 5 in 2021-03-15T22:16:50.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:16:50.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: 03-15T22:16:48.942631 2021-03-15T22:16:50.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: +0000 mon.a (mon.0) 316 2021-03-15T22:16:50.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[56514]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:50.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: cluster 2021-03-15T22:16:48. 2021-03-15T22:16:50.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: 924959+0000 osd.0 (osd.0) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:50.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: cluster 2021-03-15T22:16:50.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:16:50.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: T22:16:48.925148+0000 2021-03-15T22:16:50.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: osd.0 (osd.0) 4 : 2021-03-15T22:16:50.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:50.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:16:50.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: -03-15T22:16:48 2021-03-15T22:16:50.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: .927644+0000 mgr.y (mgr. 2021-03-15T22:16:50.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: 14144) 135 : cluster [DBG] 2021-03-15T22:16:50.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: pgmap v101: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:50.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:16:50.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: -03-15T22:16:48. 2021-03-15T22:16:50.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: 942374+0000 mon.a (mon.0 2021-03-15T22:16:50.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: ) 315 : cluster [DBG] osdmap e28: 5 total, 4 up, 5 in 2021-03-15T22:16:50.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: 2021-03-15T22:16:50.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: audit 2021-03-15T22:16:50.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:16:50.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: T22:16:48.942631+0000 2021-03-15T22:16:50.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: mon.a (mon.0) 316 : 2021-03-15T22:16:50.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:49 smithi076 conmon[62156]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:50.413 INFO:teuthology.orchestra.run.smithi125.stdout:Created osd(s) 4 on host 'smithi125' 2021-03-15T22:16:50.981 DEBUG:teuthology.orchestra.run.smithi125:osd.4> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.4.service 2021-03-15T22:16:50.985 INFO:tasks.cephadm:Deploying osd.5 on smithi125 with /dev/vg_nvme/lv_3... 2021-03-15T22:16:50.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2021-03-15T22:16:51.019 INFO:journalctl@ceph.osd.4.smithi125.stdout:-- Logs begin at Mon 2021-03-15 21:44:18 UTC. -- 2021-03-15T22:16:51.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[56514]: audit 2021-03-15T22:16:50.397797+0000 mon.a (mon.0) 317 : 2021-03-15T22:16:51.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[56514]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:51.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[56514]: audit 2021-03-15T22:16:50.402030+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:51.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:16:51.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[56514]: 03-15T22:16:50.409841 2021-03-15T22:16:51.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[56514]: +0000 mon.a (mon.0) 319 2021-03-15T22:16:51.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[56514]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:51.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[62156]: audit 2021-03-15T22:16:50.397797+0000 mon.a ( 2021-03-15T22:16:51.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[62156]: mon.0) 317 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:51.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[62156]: audit 2021-03-15T22:16:50 2021-03-15T22:16:51.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[62156]: .402030+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:51.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:16:51.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:50 smithi076 conmon[62156]: T22:16:50.409841+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:51.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:50 smithi125 conmon[56739]: audit 2021-03-15T22:16:50.397797+0000 mon.a (mon.0) 317 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:51.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:50 smithi125 conmon[56739]: audit 2021-03-15T22:16: 2021-03-15T22:16:51.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:50 smithi125 conmon[56739]: 50.402030+0000 mon.a (mon.0) 318 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:51.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:50 smithi125 conmon[56739]: audit 2021-03-15T22:16:50.409841+0000 mon.a (mon.0) 319 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:52.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:51 smithi076 conmon[56514]: cluster 2021-03-15T22:16:50.928084+0000 mgr.y (mgr.14144) 136 : cluster [DBG] 2021-03-15T22:16:52.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:51 smithi076 conmon[56514]: pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:52.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:51 smithi076 conmon[56514]: audit 2021-03-15T22:16:51.745545+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:52.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:51 smithi076 conmon[62156]: cluster 2021-03-15T22:16:50.928084+0000 mgr.y 2021-03-15T22:16:52.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:51 smithi076 conmon[62156]: (mgr.14144) 136 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:52.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:51 smithi076 conmon[62156]: audit 2021-03-15T22:16: 2021-03-15T22:16:52.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:51 smithi076 conmon[62156]: 51.745545+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:52.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:51 smithi125 conmon[56739]: cluster 2021-03-15T22:16:50.928084+0000 mgr.y (mgr.14144) 136 : cluster [DBG] pgmap v103: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:52.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:51 smithi125 conmon[56739]: audit 2021-03-15T22:16:51.745545+0000 mon.a (mon.0) 320 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:53.420 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping: /dev/vg_nvme/lv_3 2021-03-15T22:16:53.421 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2021-03-15T22:16:53.421 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10+0 records in 2021-03-15T22:16:53.421 INFO:teuthology.orchestra.run.smithi125.stderr:10+0 records out 2021-03-15T22:16:53.421 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125834 s, 833 MB/s 2021-03-15T22:16:53.422 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping successful for: 2021-03-15T22:16:53.962 INFO:journalctl@ceph.osd.4.smithi125.stdout:Mar 15 22:16:53 smithi125 conmon[60581]: debug 2021-03-15T22:16:53.553+0000 7fe6d1e9c080 -1 osd.4 0 log_to_monitors {default=true} 2021-03-15T22:16:54.107 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch daemon add osd smithi125:vg_nvme/lv_3 2021-03-15T22:16:54.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:53 smithi125 conmon[56739]: cluster 2021-03-15T22:16:52.928571+0000 mgr.y (mgr.14144) 137 : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:54.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:53 smithi125 conmon[56739]: audit 2021-03-15T22:16:53.556826+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 [v2:172.21.15.125:6800/37386122,v1:172.21.15.125:6801/37386122]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-15T22:16:54.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:53 smithi125 conmon[56739]: audit 2021-03-15T22:16:53.557353+0000 mon.a (mon.0) 321 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-15T22:16:54.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:53 smithi076 conmon[56514]: cluster 2021-03-15T22:16:52.928571+0000 mgr.y (mgr.14144) 137 2021-03-15T22:16:54.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:53 smithi076 conmon[56514]: : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:54.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:53 smithi076 conmon[56514]: audit 2021-03-15T22:16:53.556826+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 [v2:172.21.15.125:6800/37386122,v1:172.21.15.125:6801/37386122]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-15T22:16:54.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:53 smithi076 conmon[56514]: audit 2021-03-15T22:16:53.557353+0000 mon.a (mon.0) 321 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-15T22:16:54.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:53 smithi076 conmon[62156]: cluster 2021-03-15T22:16:52.928571+0000 mgr.y (mgr.14144) 137 2021-03-15T22:16:54.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:53 smithi076 conmon[62156]: : cluster [DBG] pgmap v104: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:54.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:53 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:16:54.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:53 smithi076 conmon[62156]: :16:53.556826+0000 mon.b (mon.2) 8 : audit [INF] from='osd.4 [v2:172.21.15.125:6800/37386122,v1:172.21.15.125:6801/37386122]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-15T22:16:54.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:53 smithi076 conmon[62156]: audit 2021 2021-03-15T22:16:54.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:53 smithi076 conmon[62156]: -03-15T22:16:53.557353+0000 mon.a (mon 2021-03-15T22:16:54.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:53 smithi076 conmon[62156]: .0) 321 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2021-03-15T22:16:55.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[56514]: audit 2021-03-15T22:16:53.952686+0000 mon.a (mon.0) 322 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-03-15T22:16:55.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[56514]: cluster 2021-03-15T22:16:53.952764+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-03-15T22:16:55.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[56514]: audit 2021-03-15T22:16:53.953304+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:55.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[56514]: audit 2021-03-15T22:16:53.953676+0000 mon.b (mon.2) 9 2021-03-15T22:16:55.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[56514]: : audit [INF] from='osd.4 [v2:172.21.15.125:6800/37386122,v1:172.21.15.125:6801/37386122]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:16:55.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[56514]: audit 2021-03-15T22:16:55.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[56514]: 2021-03-15T22:16:53.964149+0000 mon.a (mon.0) 325 : audit [INF] 2021-03-15T22:16:55.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[56514]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:16:55.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:16:55.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[56514]: -03-15T22:16:54.928868+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:55.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: audit 2021-03-15T22:16:55.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: 2021-03-15T22:16:55.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: 2021 2021-03-15T22:16:55.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: - 2021-03-15T22:16:55.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: 03 2021-03-15T22:16:55.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: - 2021-03-15T22:16:55.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: 15T22:16:53.952686+0000 mon.a ( 2021-03-15T22:16:55.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: mon.0) 322 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-03-15T22:16:55.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: cluster 2021-03-15T22:16:55.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: 2021-03-15T22:16:53.952764+0000 mon.a (mon.0) 323 : cluster [DBG] 2021-03-15T22:16:55.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: osdmap e29: 5 total, 4 up, 5 in 2021-03-15T22:16:55.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: audit 2021-03-15T22:16:53.953304+0000 2021-03-15T22:16:55.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: mon.a (mon.0) 324 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:55.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:16:55.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: -15T22:16:53.953676+0000 mon.b (mon. 2021-03-15T22:16:55.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: 2) 9 : audit [INF] from='osd.4 [v2:172.21.15.125:6800/37386122,v1:172.21.15.125:6801/37386122]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:16:55.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:16:55.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: :16:53.964149+0000 mon.a (mon.0) 325 : audit 2021-03-15T22:16:55.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:16:55.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: cluster 2021-03-15T22:16:54 2021-03-15T22:16:55.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: .928868+0000 mgr.y (mgr.14144) 138 : cluster [DBG] 2021-03-15T22:16:55.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:54 smithi076 conmon[62156]: pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:55.286 INFO:journalctl@ceph.osd.4.smithi125.stdout:Mar 15 22:16:54 smithi125 conmon[60581]: debug 2021-03-15T22:16:54.970+0000 7fe6c840d700 -1 osd.4 0 waiting for initial osdmap 2021-03-15T22:16:55.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:54 smithi125 conmon[56739]: audit 2021-03-15T22:16:53.952686+0000 mon.a (mon.0) 322 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2021-03-15T22:16:55.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:54 smithi125 conmon[56739]: cluster 2021-03-15T22:16:53.952764+0000 mon.a (mon.0) 323 : cluster [DBG] osdmap e29: 5 total, 4 up, 5 in 2021-03-15T22:16:55.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:54 smithi125 conmon[56739]: audit 2021-03-15T22:16:53.953304+0000 mon.a (mon.0) 324 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:55.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:54 smithi125 conmon[56739]: audit 2021-03-15T22:16:53.953676+0000 mon.b (mon.2) 9 : audit [INF] from='osd.4 [v2:172.21.15.125:6800/37386122,v1:172.21.15.125:6801/37386122]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:16:55.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:54 smithi125 conmon[56739]: audit 2021-03-15T22:16:53.964149+0000 mon.a (mon.0) 325 : audit [INF] from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:16:55.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:54 smithi125 conmon[56739]: cluster 2021-03-15T22:16:54.928868+0000 mgr.y (mgr.14144) 138 : cluster [DBG] pgmap v106: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:56.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:55 smithi125 conmon[56739]: audit 2021-03-15T22:16:54.966855+0000 mon.a (mon.0) 326 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:16:56.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:55 smithi125 conmon[56739]: cluster 2021-03-15T22:16:54.966944+0000 mon.a (mon.0) 327 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-03-15T22:16:56.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:55 smithi125 conmon[56739]: audit 2021-03-15T22:16:54.967349+0000 mon.a (mon.0) 328 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:56.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:55 smithi125 conmon[56739]: audit 2021-03-15T22:16:54.971728+0000 mon.a (mon.0) 329 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:56.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[56514]: audit 2021-03-15T22:16:54.966855+0000 mon.a (mon.0 2021-03-15T22:16:56.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[56514]: ) 326 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:16:56.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[56514]: cluster 2021-03-15T22:16:54.966944+0000 mon.a (mon.0) 327 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-03-15T22:16:56.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[56514]: audit 2021-03-15T22:16:54.967349+0000 mon.a 2021-03-15T22:16:56.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[56514]: (mon.0) 328 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:56.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[56514]: audit 2021 2021-03-15T22:16:56.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[56514]: -03-15T22:16:54.971728+0000 mon.a (mon.0 2021-03-15T22:16:56.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[56514]: ) 329 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:56.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[62156]: audit 2021-03-15T22:16:54.966855+0000 mon.a 2021-03-15T22:16:56.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[62156]: (mon.0) 326 : audit [INF] from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:16:56.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:16:56.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[62156]: 16:54.966944+0000 mon.a (mon.0) 327 : cluster [DBG] osdmap e30: 5 total, 4 up, 5 in 2021-03-15T22:16:56.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[62156]: audit 2021 2021-03-15T22:16:56.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[62156]: -03-15T22:16:54.967349+0000 mon.a (mon. 2021-03-15T22:16:56.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[62156]: 0) 328 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:56.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[62156]: audit 2021-03-15T22:16:56.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[62156]: 2021-03-15T22:16 2021-03-15T22:16:56.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[62156]: :54.971728+0000 mon.a (mon. 2021-03-15T22:16:56.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:55 smithi076 conmon[62156]: 0) 329 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:56.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: debug 2021-03-15T22:16:56.211+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:56.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:16:56.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:56.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:16:56.273 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:56.273 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:16:56.273 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:56.273 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:56.273 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:56.274 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: debug 2021-03-15T22:16:56.218+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:56.274 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:16:56.274 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:56.274 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:16:56.275 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:56.275 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:16:56.275 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:56.275 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:56.275 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:57.109 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: cluster 2021-03-15T22:16:54.541887+0000 osd.4 ( 2021-03-15T22:16:57.109 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:57.110 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: cluster 2021-03-15T22:16:54.541980+0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:57.110 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: cluster 2021-03-15T22:16:55.974812+0000 mon.a (mon.0) 330 : cluster [INF] osd.4 [v2:172.21.15.125:6800/37386122,v1:172.21.15.125:6801/37386122] boot 2021-03-15T22:16:57.110 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: cluster 2021-03-15T22:16:55.974876+0000 mon.a (mon.0) 331 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-03-15T22:16:57.111 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: audit 2021-03-15T22:16:55.977638+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:57.111 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:56.203234+0000 mgr.y (mgr.14144) 139 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-15T22:16:57.111 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: audit 2021-03-15T22:16:56.207571+0000 mon.a (mon.0) 333 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:57.112 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:56.208229+0000 mgr.y (mgr.14144) 140 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:57.112 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: audit 2021-03-15T22:16:56.211273+0000 mon.a (mon.0) 334 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:57.112 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:56.212222+0000 mgr.y (mgr.14144) 141 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:57.113 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:16:57.113 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:57.113 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:16:57.113 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:57.114 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:16:57.114 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:57.114 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:57.115 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:57.115 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: audit 2021-03-15T22:16:56.212884+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:57.115 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:56.214462+0000 mgr.y (mgr.14144) 142 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:57.115 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: audit 2021-03-15T22:16:56.218436+0000 mon.a (mon.0) 336 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:57.116 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:56.219275+0000 mgr.y (mgr.14144) 143 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:57.116 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:16:57.116 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:57.116 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:16:57.117 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:57.117 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:16:57.117 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:57.118 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:57.118 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:57.118 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: audit 2021-03-15T22:16:56.479851+0000 mgr.y (mgr.14144) 144 : audit [DBG] from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:57.118 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: audit 2021-03-15T22:16:56.480751+0000 mon.a (mon.0) 337 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:57.119 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:16:56.482194+0000 mgr.y (mgr.14144) 145 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-15T22:16:57.119 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: audit 2021-03-15T22:16:56.482497+0000 mon.a (mon.0) 338 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:57.119 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: audit 2021-03-15T22:16:56.483318+0000 mon.a (mon.0) 339 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:57.120 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:56 smithi125 conmon[56739]: cluster 2021-03-15T22:16:56.929179+0000 mgr.y (mgr.14144) 146 : cluster [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:57.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:16:57.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: -15T22:16:54.541887+0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:57.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cluster 2021-03-15T22:16:54.541980 2021-03-15T22:16:57.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: +0000 osd.4 (osd.4) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:57.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cluster 2021-03-15T22:16:57.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 2021-03-15T22:16: 2021-03-15T22:16:57.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 55.974812+0000 mon.a (mon.0) 330 2021-03-15T22:16:57.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: : cluster [INF] osd.4 [v2:172.21.15.125:6800/37386122,v1:172.21.15.125:6801/37386122] boot 2021-03-15T22:16:57.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:16:57.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: -03-15T22:16: 2021-03-15T22:16:57.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 55.974876+0000 mon.a (mon.0) 331 : 2021-03-15T22:16:57.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-03-15T22:16:57.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:16:57.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 15T22:16:55.977638+0000 2021-03-15T22:16:57.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: mon.a (mon.0) 332 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:57.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 2021-03-15T22:16:57.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:57.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 2021-03-15T22:16:56 2021-03-15T22:16:57.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: .203234+0000 mgr.y (mgr.14144) 139 : 2021-03-15T22:16:57.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cephadm [INF] Detected new or changed devices on smithi125 2021-03-15T22:16:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:16:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 03-15T22:16:56.207571 2021-03-15T22:16:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: +0000 mon.a (mon.0) 333 : audit [INF] 2021-03-15T22:16:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cephadm 2021-03-15T 2021-03-15T22:16:57.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 22:16:56.208229+0000 mgr.y ( 2021-03-15T22:16:57.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: mgr.14144) 140 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:57.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: audit 2021 2021-03-15T22:16:57.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: -03-15T22:16:56 2021-03-15T22:16:57.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: .211273+0000 mon.a (mon.0) 334 : audit 2021-03-15T22:16:57.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:57.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:16:57.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: -15T22:16:56. 2021-03-15T22:16:57.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 212222+0000 mgr.y (mgr.14144) 2021-03-15T22:16:57.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 141 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:57.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:16:57.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:57.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:16:57.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:57.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:16:57.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:57.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:57.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:57.279 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: audit 2021-03-15T22:16:56.212884+0000 mon.a (mon.0) 335 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:57.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: cluster 2021-03-15T22:16:54.541887 2021-03-15T22:16:57.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:16:57.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: cluster 2021-03-15T22:16:57.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 2021-03-15T22:16:54.541980+0000 osd.4 (osd.4 2021-03-15T22:16:57.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: ) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:16:57.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:16:57.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: -15T22:16:55.974812+0000 mon.a (mon.0) 2021-03-15T22:16:57.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 330 : cluster [INF] osd.4 [v2:172.21.15.125:6800/37386122,v1:172.21.15.125:6801/37386122] boot 2021-03-15T22:16:57.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: cluster 2021-03-15T22:16:57.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 2021-03-15T22:16:55.974876+0000 2021-03-15T22:16:57.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: mon.a (mon.0) 331 : cluster [DBG] osdmap e31: 5 total, 5 up, 5 in 2021-03-15T22:16:57.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 2021-03-15T22:16:57.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: audit 2021-03-15T22:16: 2021-03-15T22:16:57.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 55.977638+0000 mon.a (mon.0) 332 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2021-03-15T22:16:57.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:57.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:16:57.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 16:56.203234+0000 mgr.y (mgr.14144 2021-03-15T22:16:57.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: ) 139 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-15T22:16:57.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: audit 2021-03-15T22:16:57.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 2021-03-15T22:16:56.207571+0000 mon.a ( 2021-03-15T22:16:57.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: mon.0) 333 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:57.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 2021-03-15T22:16:57.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:16 2021-03-15T22:16:57.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: :56.208229+0000 mgr.y (mgr.14144) 140 : cephadm [INF] 2021-03-15T22:16:57.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:57.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:16:57.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: :16:56.211273+0000 2021-03-15T22:16:57.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: mon.a (mon.0) 334 : 2021-03-15T22:16:57.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:57.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:57.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 2021-03-15T22:16: 2021-03-15T22:16:57.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 56.212222+0000 mgr.y (mgr.14144) 141 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:16:57.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:16:57.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:57.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:16:57.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:57.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:16:57.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:57.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:57.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:16:57.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 2021-03-15T22:16:57.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: audit 2021-03-15T22:16:56.212884 2021-03-15T22:16:57.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: +0000 mon.a (mon.0) 335 : audit [DBG] 2021-03-15T22:16:57.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:16:57.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:16:57.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 03-15T22:16:56.214462+0000 mgr.y (mgr 2021-03-15T22:16:57.292 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: .14144) 142 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:57.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 2021-03-15T22:16:57.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: audit 2021-03-15T22:16:57.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 2021-03-15T22:16: 2021-03-15T22:16:57.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 56.218436+0000 mon.a (mon.0) 336 : audit [INF] 2021-03-15T22:16:57.293 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:57.294 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:16:57.294 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 2021-03-15T22:16:56.219275+0000 2021-03-15T22:16:57.294 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: mgr.y (mgr.14144) 143 : cephadm 2021-03-15T22:16:57.294 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:57.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:16:57.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:57.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:16:57.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:57.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:16:57.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:57.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:57.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:57.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: audit 2021-03-15T22:16:56.479851+0000 mgr.y (mgr.14144) 144 : audit [DBG] 2021-03-15T22:16:57.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:57.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:16:57.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 03-15T22:16:56.480751+0000 mon.a ( 2021-03-15T22:16:57.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: mon.0) 337 : audit [DBG] 2021-03-15T22:16:57.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:57.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:16:57.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 03-15T22:16:56.482194+0000 mgr.y 2021-03-15T22:16:57.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: (mgr.14144) 145 : cephadm 2021-03-15T22:16:57.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: [INF] Applying service osd.None on host smithi125... 2021-03-15T22:16:57.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:16:57.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 03-15T22:16:56.482497+0000 mon.a ( 2021-03-15T22:16:57.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: mon.0) 338 : audit [INF] 2021-03-15T22:16:57.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:57.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:16:57.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 15T22:16:56.483318+0000 mon.a (mon.0 2021-03-15T22:16:57.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: ) 339 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:57.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:16:57.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: 03-15T22:16:56.929179+0000 mgr.y 2021-03-15T22:16:57.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: (mgr.14144) 146 : cluster 2021-03-15T22:16:57.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[62156]: [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:57.303 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cephadm 2021- 2021-03-15T22:16:57.303 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 03-15T22:16:56.214462 2021-03-15T22:16:57.303 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 142 : cephadm [INF] 2021-03-15T22:16:57.303 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:16:57.304 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: audit 2021-03-15T 2021-03-15T22:16:57.304 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 22:16:56.218436+0000 mon.a ( 2021-03-15T22:16:57.304 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: mon.0) 336 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:16:57.304 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:16:57.305 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:16:57.305 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 16:56.219275+0000 mgr.y (mgr.14144 2021-03-15T22:16:57.305 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: ) 143 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:16:57.306 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:16:57.306 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:16:57.306 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:16:57.306 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:16:57.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:16:57.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:16:57.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:16:57.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:16:57.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: audit 2021-03-15T22:16:56.479851+0000 mgr.y (mgr.14144) 144 : audit [DBG] from='client.34220 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:16:57.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: audit 2021 2021-03-15T22:16:57.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: -03-15T22:16: 2021-03-15T22:16:57.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 56.480751+0000 mon.a (mon.0) 337 : audit 2021-03-15T22:16:57.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:16:57.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:16:57.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: -15T22:16:56.482194+0000 mgr.y (mgr 2021-03-15T22:16:57.310 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: .14144) 145 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-15T22:16:57.310 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: audit 2021 2021-03-15T22:16:57.310 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: -03-15T22:16: 2021-03-15T22:16:57.310 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: 56.482497+0000 mon.a (mon.0) 338 2021-03-15T22:16:57.311 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:16:57.311 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:16:57.311 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: -15T22:16:56.483318 2021-03-15T22:16:57.311 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: +0000 mon.a (mon.0) 339 : audit 2021-03-15T22:16:57.312 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:16:57.312 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:16:57.312 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: T22:16:56.929179 2021-03-15T22:16:57.313 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 146 : cluster 2021-03-15T22:16:57.313 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:56 smithi076 conmon[56514]: [DBG] pgmap v109: 1 pgs: 1 active+clean; 0 B data, 20 MiB used, 358 GiB / 358 GiB avail 2021-03-15T22:16:58.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:57 smithi076 conmon[56514]: cluster 2021-03-15T22:16:56.984545+0000 mon.a ( 2021-03-15T22:16:58.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:57 smithi076 conmon[56514]: mon.0) 340 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-03-15T22:16:58.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:57 smithi076 conmon[62156]: cluster 2021-03-15T22:16:56.984545+0000 mon.a (mon.0) 340 : cluster 2021-03-15T22:16:58.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:57 smithi076 conmon[62156]: [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-03-15T22:16:58.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:57 smithi125 conmon[56739]: cluster 2021-03-15T22:16:56.984545+0000 mon.a (mon.0) 340 : cluster [DBG] osdmap e32: 5 total, 5 up, 5 in 2021-03-15T22:16:59.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: cluster 2021-03-15T22:16:57.993431+0000 mon.a ( 2021-03-15T22:16:59.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: mon.0) 341 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2021-03-15T22:16:59.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: audit 2021-03-15T22:16:58.603014+0000 mon.b ( 2021-03-15T22:16:59.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: mon.2) 10 : audit [INF] from='client.? 172.21.15.125:0/400399058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2"}]: dispatch 2021-03-15T22:16:59.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: audit 2021-03-15T22:16: 2021-03-15T22:16:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: 58.603655+0000 mon.a (mon.0) 342 : audit [INF] 2021-03-15T22:16:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2"}]: dispatch 2021-03-15T22:16:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:16:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: 03-15T22:16:58.613312+0000 2021-03-15T22:16:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: mon.a (mon.0) 343 : audit 2021-03-15T22:16:59.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2"}]': finished 2021-03-15T22:16:59.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:16:59.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: -03-15T22:16:58.613411+0000 2021-03-15T22:16:59.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: mon.a (mon.0) 344 : cluster [DBG] 2021-03-15T22:16:59.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: osdmap e34: 6 total, 5 up, 6 in 2021-03-15T22:16:59.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: audit 2021-03-15T22: 2021-03-15T22:16:59.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: 16:58.613561+0000 mon.a (mon.0) 2021-03-15T22:16:59.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: 345 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:16:59.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:16:59.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: T22:16:58.929486+0000 mgr.y (mgr 2021-03-15T22:16:59.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[56514]: .14144) 147 : cluster [DBG] pgmap v113: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:16:59.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: cluster 2021-03-15T22:16:57.993431 2021-03-15T22:16:59.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: +0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2021-03-15T22:16:59.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: audit 2021-03-15T22:16:59.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: 2021-03-15T22:16: 2021-03-15T22:16:59.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: 58.603014+0000 mon.b (mon.2 2021-03-15T22:16:59.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: ) 10 : audit [INF] from='client.? 172.21.15.125:0/400399058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2"}]: dispatch 2021-03-15T22:16:59.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: 2021-03-15T22:16:59.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: audit 2021 2021-03-15T22:16:59.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: -03-15T22:16:58.603655 2021-03-15T22:16:59.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2021-03-15T22:16:59.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: 342 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2"}]: dispatch 2021-03-15T22:16:59.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:16:59.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: 03-15T22:16:58.613312 2021-03-15T22:16:59.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: +0000 mon.a (mon.0) 343 2021-03-15T22:16:59.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2"}]': finished 2021-03-15T22:16:59.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:16:59.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: 03-15T22:16:58.613411+0000 mon.a (mon.0 2021-03-15T22:16:59.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: ) 344 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-03-15T22:16:59.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: audit 2021 2021-03-15T22:16:59.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: -03-15T22:16:58. 2021-03-15T22:16:59.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: 613561+0000 mon.a (mon.0 2021-03-15T22:16:59.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: ) 345 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:16:59.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: 2021-03-15T22:16:59.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:16:59.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: 03-15T22:16:58.929486 2021-03-15T22:16:59.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 147 2021-03-15T22:16:59.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:16:59 smithi076 conmon[62156]: : cluster [DBG] pgmap v113: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:16:59.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:59 smithi125 conmon[56739]: cluster 2021-03-15T22:16:57.993431+0000 mon.a (mon.0) 341 : cluster [DBG] osdmap e33: 5 total, 5 up, 5 in 2021-03-15T22:16:59.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:59 smithi125 conmon[56739]: audit 2021-03-15T22:16:58.603014+0000 mon.b (mon.2) 10 : audit [INF] from='client.? 172.21.15.125:0/400399058' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2"}]: dispatch 2021-03-15T22:16:59.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:59 smithi125 conmon[56739]: audit 2021-03-15T22:16:58.603655+0000 mon.a (mon.0) 342 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2"}]: dispatch 2021-03-15T22:16:59.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:59 smithi125 conmon[56739]: audit 2021-03-15T22:16:58.613312+0000 mon.a (mon.0) 343 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2"}]': finished 2021-03-15T22:16:59.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:59 smithi125 conmon[56739]: cluster 2021-03-15T22:16:58.613411+0000 mon.a (mon.0) 344 : cluster [DBG] osdmap e34: 6 total, 5 up, 6 in 2021-03-15T22:16:59.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:59 smithi125 conmon[56739]: audit 2021-03-15T22:16:58.613561+0000 mon.a (mon.0) 345 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:16:59.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:16:59 smithi125 conmon[56739]: cluster 2021-03-15T22:16:58.929486+0000 mgr.y (mgr.14144) 147 : cluster [DBG] pgmap v113: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:00.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:00 smithi076 conmon[56514]: audit 2021-03-15T22:16:59.131749+0000 mon.b (mon.2) 11 : audit 2021-03-15T22:17:00.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:00 smithi076 conmon[56514]: [DBG] from='client.? 172.21.15.125:0/1037162127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:17:00.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:00 smithi076 conmon[62156]: audit 2021-03-15T22:16:59.131749+0000 mon.b (mon.2) 2021-03-15T22:17:00.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:00 smithi076 conmon[62156]: 11 : audit [DBG] from='client.? 172.21.15.125:0/1037162127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:17:00.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:00 smithi125 conmon[56739]: audit 2021-03-15T22:16:59.131749+0000 mon.b (mon.2) 11 : audit [DBG] from='client.? 172.21.15.125:0/1037162127' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:17:01.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:01 smithi076 conmon[62156]: cluster 2021-03-15T22:17:00.929940+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v114: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:01.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:01 smithi076 conmon[56514]: cluster 2021-03-15T22:17:00.929940+0000 mgr.y (mgr.14144) 2021-03-15T22:17:01.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:01 smithi076 conmon[56514]: 148 : cluster [DBG] pgmap v114: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:01.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:01 smithi125 conmon[56739]: cluster 2021-03-15T22:17:00.929940+0000 mgr.y (mgr.14144) 148 : cluster [DBG] pgmap v114: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:02.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:02 smithi076 conmon[56514]: audit 2021-03-15T22:17:01.031013+0000 mon.a (mon 2021-03-15T22:17:02.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:02 smithi076 conmon[56514]: .0) 346 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:17:02.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:02 smithi076 conmon[56514]: audit 2021-03-15T22:17:01.032050+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:17:02.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:02 smithi076 conmon[56514]: 2021-03-15T22:17:02.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:02 smithi076 conmon[62156]: audit 2021-03-15T22:17:01.031013+0000 mon.a (mon.0) 346 : 2021-03-15T22:17:02.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:02 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:17:02.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:02 smithi076 conmon[62156]: audit 2021-03-15T22:17:01.032050+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:17:02.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:02 smithi125 conmon[56739]: audit 2021-03-15T22:17:01.031013+0000 mon.a (mon.0) 346 : audit 2021-03-15T22:17:02.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:02 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:17:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:02 smithi125 conmon[56739]: audit 2021-03-15T22:17:01.032050+0000 mon.a (mon.0) 347 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:17:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:03 smithi076 conmon[56514]: cluster 2021-03-15T22:17:02.930277+0000 mgr.y (mgr.14144 2021-03-15T22:17:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:03 smithi076 conmon[56514]: ) 149 : cluster [DBG] pgmap v115: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:03.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:03 smithi076 conmon[62156]: cluster 2021-03-15T22:17:02.930277+0000 mgr.y (mgr.14144) 149 2021-03-15T22:17:03.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:03 smithi076 conmon[62156]: : cluster [DBG] pgmap v115: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:03.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:03 smithi125 conmon[56739]: cluster 2021-03-15T22:17:02.930277+0000 mgr.y (mgr.14144) 149 2021-03-15T22:17:03.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:03 smithi125 conmon[56739]: : cluster [DBG] pgmap v115: 1 pgs: 1 peering; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:04.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:04 smithi076 conmon[56514]: audit 2021-03-15T22:17:04.429755+0000 mon.a (mon.0) 2021-03-15T22:17:04.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:04 smithi076 conmon[56514]: 348 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-03-15T22:17:04.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:04 smithi076 conmon[56514]: audit 2021-03-15T22:17:04.430575+0000 mon.a (mon.0) 349 : audit [DBG] 2021-03-15T22:17:04.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:04 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:04.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:04 smithi076 conmon[62156]: audit 2021-03-15T22:17:04.429755+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-03-15T22:17:04.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:04 smithi076 conmon[62156]: 2021-03-15T22:17:04.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:04 smithi076 conmon[62156]: audit 2021-03-15T22:17:04.430575+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:04.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:04 smithi125 conmon[56739]: audit 2021-03-15T22:17:04.429755+0000 mon.a (mon.0) 348 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2021-03-15T22:17:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:04 smithi125 conmon[56739]: audit 2021-03-15T22:17:04.430575+0000 mon.a (mon.0) 349 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:06.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:05 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:04.431220+0000 mgr.y (mgr.14144) 150 : cephadm [INF] Deploying daemon osd.5 on smithi125 2021-03-15T22:17:06.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:05 smithi125 conmon[56739]: cluster 2021-03-15T22:17:04.930802+0000 mgr.y (mgr.14144) 151 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:06.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:05 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:04.431220+0000 mgr.y (mgr.14144) 150 : cephadm [INF] Deploying daemon osd.5 on smithi125 2021-03-15T22:17:06.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:05 smithi076 conmon[56514]: cluster 2021-03-15T22:17:04.930802+0000 mgr.y (mgr.14144) 151 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:06.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:05 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:04.431220+0000 mgr.y (mgr.14144) 150 : 2021-03-15T22:17:06.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:05 smithi076 conmon[62156]: cephadm [INF] Deploying daemon osd.5 on smithi125 2021-03-15T22:17:06.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:05 smithi076 conmon[62156]: cluster 2021-03-15T22:17:04.930802+0000 mgr.y (mgr.14144) 151 : cluster [DBG] pgmap v116: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:06.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:05 smithi076 conmon[62156]: 2021-03-15T22:17:07.248 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:06 smithi125 conmon[56739]: cluster 2021-03-15T22:17:06.232589+0000 osd.1 (osd.1) 3 2021-03-15T22:17:07.248 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:06 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:07.249 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:06 smithi125 conmon[56739]: cluster 2021-03-15T22:17:06.232748+0000 osd.1 (osd.1) 4 : cluster 2021-03-15T22:17:07.249 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:06 smithi125 conmon[56739]: [DBG] purged_snaps scrub ok 2021-03-15T22:17:07.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[56514]: cluster 2021-03-15T22:17:07.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[56514]: 2021-03-15T22:17:06. 2021-03-15T22:17:07.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[56514]: 232589+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:07.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[56514]: cluster 2021-03-15T22:17:07.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[56514]: 2021-03-15T22:17:06.232748+0000 osd.1 (osd.1) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:07.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[62156]: cluster 2021-03-15T22:17:07.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[62156]: 2021-03-15T22:17:07.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[62156]: 2021 2021-03-15T22:17:07.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[62156]: -03-15T22:17 2021-03-15T22:17:07.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[62156]: :06.232589+0000 osd.1 (osd.1) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:07.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[62156]: cluster 2021-03-15T22:17:07.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[62156]: 2021-03 2021-03-15T22:17:07.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[62156]: -15T22:17:06.232748+0000 osd.1 (osd.1) 4 : cluster [DBG] 2021-03-15T22:17:07.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:06 smithi076 conmon[62156]: purged_snaps scrub ok 2021-03-15T22:17:07.715 INFO:teuthology.orchestra.run.smithi125.stdout:Created osd(s) 5 on host 'smithi125' 2021-03-15T22:17:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[56514]: cluster 2021-03-15T22:17:06.931248+0000 mgr.y (mgr.14144) 152 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[56514]: cluster 2021-03-15T22:17:06.943921+0000 mon.a (mon.0) 350 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-03-15T22:17:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[56514]: audit 2021-03-15T22:17:06.944078+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[56514]: audit 2021-03-15T22:17:07.704056+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[56514]: audit 2021-03-15T22:17:07.707257+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:17:08.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[56514]: -15T22:17:07.711685+0000 mon.a (mon.0) 354 : audit 2021-03-15T22:17:08.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: cluster 2021-03-15T22:17:06.931248+0000 2021-03-15T22:17:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: mgr.y (mgr.14144) 152 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:17:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: 22:17:06.943921+0000 mon.a (mon.0) 350 : 2021-03-15T22:17:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-03-15T22:17:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:17:08.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: 03-15T22:17:06.944078+0000 mon.a (mon. 2021-03-15T22:17:08.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: 0) 351 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:08.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: 2021-03-15T22:17:08.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:17:08.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: 03-15T22:17:07.704056+0000 mon.a ( 2021-03-15T22:17:08.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: mon.0) 352 : audit [DBG] 2021-03-15T22:17:08.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:08.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:17:08.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: -15T22:17:07.707257+0000 mon.a (mon 2021-03-15T22:17:08.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: .0) 353 : audit [INF] 2021-03-15T22:17:08.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:08.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:17:08.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: -15T22:17:07.711685+0000 mon.a (mon 2021-03-15T22:17:08.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: .0) 354 : audit [INF] 2021-03-15T22:17:08.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:07 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:08.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:07 smithi125 conmon[56739]: cluster 2021-03-15T22:17:06.931248+0000 mgr.y (mgr.14144) 152 : cluster [DBG] pgmap v117: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:08.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:07 smithi125 conmon[56739]: cluster 2021-03-15T22:17:06.943921+0000 mon.a (mon.0) 350 : cluster [DBG] osdmap e35: 6 total, 5 up, 6 in 2021-03-15T22:17:08.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:07 smithi125 conmon[56739]: audit 2021-03-15T22:17:06.944078+0000 mon.a (mon.0) 351 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:08.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:07 smithi125 conmon[56739]: audit 2021-03-15T22:17:07.704056+0000 mon.a (mon.0) 352 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:08.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:07 smithi125 conmon[56739]: audit 2021-03-15T22:17:07.707257+0000 mon.a (mon.0) 353 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:08.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:07 smithi125 conmon[56739]: audit 2021-03-15T22:17:07.711685+0000 mon.a (mon.0) 354 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:08.380 DEBUG:teuthology.orchestra.run.smithi125:osd.5> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.5.service 2021-03-15T22:17:08.383 INFO:tasks.cephadm:Deploying osd.6 on smithi125 with /dev/vg_nvme/lv_2... 2021-03-15T22:17:08.383 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2021-03-15T22:17:08.411 INFO:journalctl@ceph.osd.5.smithi125.stdout:-- Logs begin at Mon 2021-03-15 21:44:18 UTC. -- 2021-03-15T22:17:10.201 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:09 smithi125 conmon[56739]: cluster 2021-03-15T22:17:08.931755+0000 mgr.y (mgr.14144) 153 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:10.201 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:09 smithi125 conmon[56739]: audit 2021-03-15T22:17:09.220477+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:10.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:09 smithi076 conmon[56514]: cluster 2021-03-15T22:17:08.931755+0000 mgr.y (mgr.14144) 153 : cluster [DBG] 2021-03-15T22:17:10.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:09 smithi076 conmon[56514]: pgmap v119: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:10.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:09 smithi076 conmon[56514]: audit 2021-03-15T22:17:09.220477+0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:10.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:09 smithi076 conmon[62156]: cluster 2021-03-15T22:17:08.931755+0000 mgr.y (mgr.14144) 2021-03-15T22:17:10.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:09 smithi076 conmon[62156]: 153 : cluster [DBG] pgmap v119: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:10.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:09 smithi076 conmon[62156]: audit 2021-03-15T22:17:09.220477 2021-03-15T22:17:10.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:09 smithi076 conmon[62156]: +0000 mon.a (mon.0) 355 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:10.791 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping: /dev/vg_nvme/lv_2 2021-03-15T22:17:10.791 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2021-03-15T22:17:10.791 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10+0 records in 2021-03-15T22:17:10.791 INFO:teuthology.orchestra.run.smithi125.stderr:10+0 records out 2021-03-15T22:17:10.792 INFO:teuthology.orchestra.run.smithi125.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126449 s, 829 MB/s 2021-03-15T22:17:10.792 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping successful for: 2021-03-15T22:17:11.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:10 smithi076 conmon[56514]: audit 2021-03-15T22:17:10.882392+0000 mon.b (mon.2) 12 : audit 2021-03-15T22:17:11.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:10 smithi076 conmon[56514]: [INF] from='osd.5 [v2:172.21.15.125:6808/3610838738,v1:172.21.15.125:6809/3610838738]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:17:11.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:10 smithi076 conmon[56514]: audit 2021-03-15T22:17:10.882791+0000 mon.a (mon.0) 356 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:17:11.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:10 smithi076 conmon[62156]: audit 2021-03-15T22:17:10.882392+0000 mon.b (mon.2) 12 2021-03-15T22:17:11.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:10 smithi076 conmon[62156]: : audit [INF] from='osd.5 [v2:172.21.15.125:6808/3610838738,v1:172.21.15.125:6809/3610838738]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:17:11.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:10 smithi076 conmon[62156]: audit 2021-03-15T22:17:10.882791+0000 2021-03-15T22:17:11.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:10 smithi076 conmon[62156]: mon.a (mon.0) 356 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:17:11.286 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:17:10 smithi125 conmon[63905]: debug 2021-03-15T22:17:10.878+0000 7fce2872d080 -1 osd.5 0 log_to_monitors {default=true} 2021-03-15T22:17:11.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:10 smithi125 conmon[56739]: audit 2021-03-15T22:17:10.882392+0000 mon.b 2021-03-15T22:17:11.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:10 smithi125 conmon[56739]: (mon.2) 12 : audit [INF] from='osd.5 [v2:172.21.15.125:6808/3610838738,v1:172.21.15.125:6809/3610838738]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:17:11.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:10 smithi125 conmon[56739]: audit 2021-03-15T22:17:10.882791+0000 mon.a (mon.0) 2021-03-15T22:17:11.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:10 smithi125 conmon[56739]: 356 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:17:11.595 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch daemon add osd smithi125:vg_nvme/lv_2 2021-03-15T22:17:12.036 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[63905]: debug 2021-03-15T22:17:11.959+0000 7fce1ec9e700 -1 osd.5 0 waiting for initial osdmap 2021-03-15T22:17:12.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[56739]: cluster 2021-03-15T22:17:10.932256+0000 mgr.y (mgr.14144) 154 2021-03-15T22:17:12.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[56739]: : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:12.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[56739]: audit 2021-03-15T22:17:10.959418+0000 mon.a (mon.0) 357 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-15T22:17:12.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[56739]: cluster 2021-03-15T22:17:10.959526+0000 mon.a (mon.0) 358 : 2021-03-15T22:17:12.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[56739]: cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2021-03-15T22:17:12.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:17:12.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[56739]: -15T22:17:10.959658+0000 mon.a (mon.0) 359 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:12.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[56739]: audit 2021-03-15T22:17:10.959916 2021-03-15T22:17:12.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[56739]: +0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.125:6808/3610838738,v1:172.21.15.125:6809/3610838738]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:12.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[56739]: audit 2021-03-15T22:17:10.960250+0000 mon.a 2021-03-15T22:17:12.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:11 smithi125 conmon[56739]: (mon.0) 360 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:12.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[56514]: cluster 2021-03-15T22:17:10.932256+0000 mgr.y (mgr.14144) 154 : cluster [DBG] 2021-03-15T22:17:12.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[56514]: pgmap v120: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:12.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[56514]: audit 2021-03-15T22:17:10.959418+0000 mon.a (mon.0) 357 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-15T22:17:12.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:17:12.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[56514]: 17:10.959526+0000 mon.a (mon.0) 358 : cluster [DBG] osdmap e36: 6 total, 5 up, 6 in 2021-03-15T22:17:12.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[56514]: audit 2021-03-15T22:17:10.959658+0000 mon.a (mon.0) 2021-03-15T22:17:12.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[56514]: 359 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:12.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[56514]: audit 2021-03-15T22:17:10.959916 2021-03-15T22:17:12.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[56514]: +0000 mon.b (mon.2) 13 : audit [INF] from='osd.5 [v2:172.21.15.125:6808/3610838738,v1:172.21.15.125:6809/3610838738]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:12.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[56514]: audit 2021-03-15T22:17 2021-03-15T22:17:12.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[56514]: :10.960250+0000 mon.a (mon.0) 360 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:12.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: cluster 2021-03-15T22:17:10.932256+0000 mgr.y ( 2021-03-15T22:17:12.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: mgr.14144) 154 : cluster [DBG] pgmap v120: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:12.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:17:12.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: 15T22:17:10.959418+0000 mon.a (mon.0) 357 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-15T22:17:12.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:17:12.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: T22:17:10.959526+0000 mon.a (mon.0) 358 : cluster [DBG] 2021-03-15T22:17:12.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: osdmap e36: 6 total, 5 up, 6 in 2021-03-15T22:17:12.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: audit 2021-03-15T22:17:12.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: 2021-03-15T22:17:10 2021-03-15T22:17:12.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: .959658+0000 mon.a (mon.0) 359 : 2021-03-15T22:17:12.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:12.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: audit 2021-03-15T22:17:12.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:17:12.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: :17:10.959916+0000 mon.b (mon.2 2021-03-15T22:17:12.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: ) 13 : audit [INF] from='osd.5 [v2:172.21.15.125:6808/3610838738,v1:172.21.15.125:6809/3610838738]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:12.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: audit 2021-03-15T22:17:12.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: 2021-03-15T22:17:10.960250+0000 mon.a (mon 2021-03-15T22:17:12.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:11 smithi076 conmon[62156]: .0) 360 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:13.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:12 smithi125 conmon[56739]: audit 2021-03-15T22:17:11.957672+0000 mon.a (mon.0) 361 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:17:13.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:12 smithi125 conmon[56739]: cluster 2021-03-15T22:17:11.957784+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2021-03-15T22:17:13.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:12 smithi125 conmon[56739]: audit 2021-03-15T22:17:11.958308+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:13.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:12 smithi125 conmon[56739]: audit 2021-03-15T22:17:11.961002+0000 mon.a (mon.0) 364 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:13.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:12 smithi125 conmon[56739]: cluster 2021-03-15T22:17:12.932620+0000 mgr.y (mgr.14144) 155 : cluster [DBG] pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:13.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:12 smithi125 conmon[56739]: cluster 2021-03-15T22:17:12.959783+0000 mon.a (mon.0) 365 : cluster [INF] osd.5 [v2:172.21.15.125:6808/3610838738,v1:172.21.15.125:6809/3610838738] boot 2021-03-15T22:17:13.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:12 smithi125 conmon[56739]: cluster 2021-03-15T22:17:12.959847+0000 mon.a (mon.0) 366 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-03-15T22:17:13.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:12 smithi125 conmon[56739]: audit 2021-03-15T22:17:12.960066+0000 mon.a (mon.0) 367 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:13.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: audit 2021-03-15T22:17:11.957672+0000 mon.a (mon.0) 361 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:17:13.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: 2021-03-15T22:17:13.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: cluster 2021-03-15T22:17:11.957784+0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2021-03-15T22:17:13.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: audit 2021-03-15T22:17 2021-03-15T22:17:13.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: :11.958308+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:13.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: audit 2021-03-15T22:17:11.961002+0000 mon.a (mon 2021-03-15T22:17:13.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: .0) 364 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:13.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: 2021-03-15T22:17:13.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: cluster 2021-03-15T22:17:13.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: 2021-03-15T 2021-03-15T22:17:13.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: 22:17:12.932620+0000 mgr.y (mgr 2021-03-15T22:17:13.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: .14144) 155 : cluster [DBG] 2021-03-15T22:17:13.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:13.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:17:13.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: -03-15T22:17:12 2021-03-15T22:17:13.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: .959783+0000 mon.a (mon.0 2021-03-15T22:17:13.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: ) 365 : cluster [INF] osd.5 [v2:172.21.15.125:6808/3610838738,v1:172.21.15.125:6809/3610838738] boot 2021-03-15T22:17:13.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:17:13.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: -03-15T22:17:12. 2021-03-15T22:17:13.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: 959847+0000 mon.a (mon.0) 2021-03-15T22:17:13.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: 366 : cluster [DBG] osdmap e38: 6 total, 6 up, 6 in 2021-03-15T22:17:13.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: audit 2021-03-15T22:17:13.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: 2021-03-15T22:17:12. 2021-03-15T22:17:13.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: 960066+0000 mon.a (mon.0) 2021-03-15T22:17:13.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[56514]: 367 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:13.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: audit 2021-03-15T22:17:11.957672+0000 mon.a 2021-03-15T22:17:13.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: (mon.0) 361 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:17:13.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: cluster 2021-03-15T22:17:11.957784 2021-03-15T22:17:13.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: +0000 mon.a (mon.0) 362 : cluster [DBG] osdmap e37: 6 total, 5 up, 6 in 2021-03-15T22:17:13.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:17:13.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: 22:17:11.958308+0000 mon.a (mon.0) 363 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:13.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: 2021-03-15T22:17:13.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: audit 2021-03-15T22:17:13.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: 2021-03-15T22:17: 2021-03-15T22:17:13.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: 11.961002+0000 mon.a (mon. 2021-03-15T22:17:13.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: 0) 364 : audit [DBG] 2021-03-15T22:17:13.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:13.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:17:13.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: -15T22:17:12.932620+0000 mgr.y ( 2021-03-15T22:17:13.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: mgr.14144) 155 : cluster [DBG] 2021-03-15T22:17:13.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: pgmap v123: 1 pgs: 1 active+clean; 0 B data, 25 MiB used, 447 GiB / 447 GiB avail 2021-03-15T22:17:13.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:17:13.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: -15T22:17:12.959783+0000 mon.a 2021-03-15T22:17:13.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: (mon.0) 365 : cluster 2021-03-15T22:17:13.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: [INF] osd.5 [v2:172.21.15.125:6808/3610838738,v1:172.21.15.125:6809/3610838738] boot 2021-03-15T22:17:13.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:17:13.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: 03-15T22:17:12.959847+0000 mon.a ( 2021-03-15T22:17:13.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: mon.0) 366 : cluster [DBG] 2021-03-15T22:17:13.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: osdmap e38: 6 total, 6 up, 6 in 2021-03-15T22:17:13.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:13.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: -03-15T22:17:12 2021-03-15T22:17:13.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: .960066+0000 mon.a (mon.0 2021-03-15T22:17:13.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:12 smithi076 conmon[62156]: ) 367 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:17:14.267 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: debug 2021-03-15T22:17:13.902+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:14.267 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:17:14.267 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:14.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:17:14.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:14.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:17:14.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:14.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:14.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:14.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: debug 2021-03-15T22:17:13.911+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:14.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:17:14.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:14.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:17:14.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:14.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:17:14.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:14.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:14.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:13 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:15.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: cluster 2021-03-15T22:17:11.836071+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:15.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: 2021-03-15T22:17:15.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: cluster 2021-03-15T22:17:11.836158+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:15.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: cephadm 2021 2021-03-15T22:17:15.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: -03-15T22:17:13.626961+0000 mgr.y (mgr.14144) 156 : cephadm 2021-03-15T22:17:15.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: [INF] Detected new or changed devices on smithi125 2021-03-15T22:17:15.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: audit 2021-03-15T22:17:15.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: 2021-03-15T22:17:13.631484+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:15.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: cephadm 2021-03- 2021-03-15T22:17:15.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: 15T22:17:13.896458+0000 mgr.y (mgr.14144) 157 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:15.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: audit 2021-03-15T22:17 2021-03-15T22:17:15.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: :13.902782+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:15.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: cephadm 2021-03-15T22:17 2021-03-15T22:17:15.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: :13.903823+0000 mgr.y (mgr.14144) 158 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:15.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:17:15.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:15.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:17:15.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:15.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:17:15.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:15.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:15.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:15.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: audit 2021-03-15T22:17:13.904493+0000 mon.a (mon.0) 370 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:15.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:13.906123+0000 mgr.y (mgr.14144) 159 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:15.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: audit 2021-03-15T22:17:13.912134+0000 mon.a (mon.0) 371 : audit 2021-03-15T22:17:15.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:15.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:15.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: 2021-03-15T22:17:13.913003+0000 mgr.y (mgr.14144) 160 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:15.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:17:15.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:15.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:17:15.025 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:15.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: cluster 2021-03-15T22:17:11.836071+0000 osd.5 ( 2021-03-15T22:17:15.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:15.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: cluster 2021-03-15T22:17:15.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 2021-03-15T22:17: 2021-03-15T22:17:15.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 11.836158+0000 osd.5 (osd.5) 2021-03-15T22:17:15.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:15.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:13.626961+0000 mgr.y 2021-03-15T22:17:15.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: (mgr.14144) 156 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-15T22:17:15.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:15.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: -03-15T22:17:13.631484+0000 mon.a 2021-03-15T22:17:15.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: (mon.0) 368 : audit 2021-03-15T22:17:15.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:15.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:17:15.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 03-15T22:17:13.896458+0000 2021-03-15T22:17:15.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: mgr.y (mgr.14144) 157 : cephadm 2021-03-15T22:17:15.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:15.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:15.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: -03-15T22:17:13. 2021-03-15T22:17:15.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 902782+0000 mon.a (mon.0) 2021-03-15T22:17:15.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 369 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:15.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:17:15.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: -03-15T22:17:13 2021-03-15T22:17:15.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: .903823+0000 mgr.y (mgr. 2021-03-15T22:17:15.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 14144) 158 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:15.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:17:15.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:15.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:17:15.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:15.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:17:15.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:15.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:15.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:15.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 2021-03-15T22:17:15.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: audit 2021-03-15T22:17: 2021-03-15T22:17:15.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 13.904493+0000 mon.a (mon.0) 370 2021-03-15T22:17:15.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:15.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:17:15.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: -03-15T22:17:13. 2021-03-15T22:17:15.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 906123+0000 mgr.y (mgr.14144 2021-03-15T22:17:15.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: ) 159 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:15.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: audit 2021-03-15T22:17:15.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:17:15.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 17:13.912134+0000 mon.a 2021-03-15T22:17:15.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: (mon.0) 371 : audit 2021-03-15T22:17:15.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:15.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:17:15.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:17:15.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:15.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:15.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:15.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: cluster 2021-03-15T22:17:13.968188+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-03-15T22:17:15.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: audit 2021-03-15T22:17:13.975931+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:17:15.038 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:17:15.038 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: -15T22:17:13.977575+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:17:15.038 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: audit 2021-03-15T22:17:15.038 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: 2021-03-15T22:17:13.978376+0000 mon.a (mon 2021-03-15T22:17:15.039 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[56514]: .0) 375 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:15.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 03-15T22:17:13.913003 2021-03-15T22:17:15.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 160 2021-03-15T22:17:15.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:15.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:17:15.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:15.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:17:15.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:15.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:17:15.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:15.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:15.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:15.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: cluster 2021-03-15T22:17:13 2021-03-15T22:17:15.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: .968188+0000 mon.a (mon.0 2021-03-15T22:17:15.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: ) 372 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-03-15T22:17:15.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:15.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: -03-15T22:17:13.975931 2021-03-15T22:17:15.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: +0000 mon.a (mon.0) 373 2021-03-15T22:17:15.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:17:15.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:15.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: -03-15T22:17:13. 2021-03-15T22:17:15.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 977575+0000 mon.a (mon.0) 2021-03-15T22:17:15.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 374 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:17:15.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:15.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: -03-15T22:17:13. 2021-03-15T22:17:15.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: 978376+0000 mon.a (mon.0 2021-03-15T22:17:15.045 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:14 smithi076 conmon[62156]: ) 375 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:15.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: cluster 2021-03-15T22:17:11.836071+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:15.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: cluster 2021-03-15T22:17:11.836158+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:15.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:13.626961+0000 mgr.y (mgr.14144) 156 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-15T22:17:15.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: audit 2021-03-15T22:17:13.631484+0000 mon.a (mon.0) 368 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:15.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:13.896458+0000 mgr.y (mgr.14144) 157 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:15.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: audit 2021-03-15T22:17:13.902782+0000 mon.a (mon.0) 369 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:15.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:13.903823+0000 mgr.y (mgr.14144) 158 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:15.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:17:15.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:15.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:17:15.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:15.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:17:15.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:15.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:15.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:15.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: audit 2021-03-15T22:17:13.904493+0000 mon.a (mon.0 2021-03-15T22:17:15.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: ) 370 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:15.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:13.906123+0000 mgr.y (mgr.14144) 159 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:15.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: audit 2021-03-15T22:17:13.912134+0000 mon.a (mon.0) 371 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:15.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:13.913003+0000 mgr.y (mgr.14144) 160 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:15.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:17:15.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:15.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:17:15.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:15.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:17:15.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:15.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:15.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:15.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: cluster 2021-03-15T22:17:13.968188+0000 mon.a (mon.0) 372 : cluster [DBG] osdmap e39: 6 total, 6 up, 6 in 2021-03-15T22:17:15.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: audit 2021-03-15T22:17:13.975931+0000 mon.a (mon.0) 373 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:17:15.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: audit 2021-03-15T22:17:13.977575+0000 mon.a (mon.0) 374 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:17:15.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:14 smithi125 conmon[56739]: audit 2021-03-15T22:17:13.978376+0000 mon.a (mon.0) 375 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:16.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[56514]: audit 2021-03-15T22:17:13.975099+0000 mgr.y (mgr.14144) 161 : audit [DBG] from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:16.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[56514]: 2021-03-15T22:17:16.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:13.977276+0000 mgr.y (mgr.14144) 162 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-15T22:17:16.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[56514]: cluster 2021-03-15T22:17:16.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[56514]: 2021-03-15T22:17:14.933171+0000 2021-03-15T22:17:16.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[56514]: mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:16.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[56514]: 2021-03-15T22:17:16.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:17:16.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[56514]: T22:17:14.964291+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-03-15T22:17:16.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[56514]: 2021-03-15T22:17:16.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[62156]: audit 2021-03-15T22:17:13.975099+0000 mgr.y ( 2021-03-15T22:17:16.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[62156]: mgr.14144) 161 : audit [DBG] from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:16.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:13.977276+0000 2021-03-15T22:17:16.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[62156]: mgr.y (mgr.14144) 162 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-15T22:17:16.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[62156]: cluster 2021-03-15T22:17: 2021-03-15T22:17:16.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[62156]: 14.933171+0000 mgr.y (mgr.14144) 163 : cluster 2021-03-15T22:17:16.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[62156]: [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:16.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:17:16.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[62156]: 22:17:14.964291+0000 mon.a (mon. 2021-03-15T22:17:16.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:15 smithi076 conmon[62156]: 0) 376 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-03-15T22:17:16.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:15 smithi125 conmon[56739]: audit 2021-03-15T22:17:13.975099+0000 mgr.y (mgr.14144) 161 2021-03-15T22:17:16.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:15 smithi125 conmon[56739]: : audit [DBG] from='client.34247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:16.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:15 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:13.977276+0000 mgr.y (mgr.14144) 162 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-15T22:17:16.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:15 smithi125 conmon[56739]: cluster 2021-03-15T22:17:14.933171+0000 mgr.y (mgr.14144) 163 : cluster [DBG] pgmap v126: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:16.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:15 smithi125 conmon[56739]: cluster 2021-03-15T22:17:14.964291+0000 mon.a (mon.0) 376 : cluster [DBG] osdmap e40: 6 total, 6 up, 6 in 2021-03-15T22:17:17.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: audit 2021-03-15T22:17:16.268278+0000 mon.b (mon.2) 14 : audit 2021-03-15T22:17:17.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.125:0/224106860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe"}]: dispatch 2021-03-15T22:17:17.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: audit 2021-03-15T22:17:16.268662+0000 mon.a (mon.0) 377 : audit 2021-03-15T22:17:17.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe"}]: dispatch 2021-03-15T22:17:17.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:17:17.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: 03-15T22:17:16.273205+0000 mon.a (mon.0) 378 2021-03-15T22:17:17.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe"}]': finished 2021-03-15T22:17:17.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: cluster 2021-03-15T22:17:16 2021-03-15T22:17:17.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: .273290+0000 mon.a (mon.0) 379 : cluster 2021-03-15T22:17:17.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: [DBG] osdmap e41: 7 total, 6 up, 7 in 2021-03-15T22:17:17.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:17:17.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: 15T22:17:16.273456+0000 mon.a ( 2021-03-15T22:17:17.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: mon.0) 380 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:17.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:17:17.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: 03-15T22:17:16.729844+0000 2021-03-15T22:17:17.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: mon.b (mon.2) 15 : audit [DBG] from='client.? 172.21.15.125:0/3073182613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:17:17.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[62156]: 2021-03-15T22:17:17.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: audit 2021-03-15T22:17:16.268278+0000 mon.b (mon 2021-03-15T22:17:17.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: .2) 14 : audit [INF] from='client.? 172.21.15.125:0/224106860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe"}]: dispatch 2021-03-15T22:17:17.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: audit 2021-03-15T22:17:16.268662+0000 mon.a (mon.0) 377 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe"}]: dispatch 2021-03-15T22:17:17.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: audit 2021-03-15T22:17:16 2021-03-15T22:17:17.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: .273205+0000 mon.a (mon.0) 378 : audit 2021-03-15T22:17:17.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe"}]': finished 2021-03-15T22:17:17.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:17:17.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: -15T22:17:16.273290 2021-03-15T22:17:17.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: +0000 mon.a (mon.0) 379 : cluster [DBG] 2021-03-15T22:17:17.279 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: osdmap e41: 7 total, 6 up, 7 in 2021-03-15T22:17:17.279 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:17:17.280 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: 15T22:17:16.273456+0000 2021-03-15T22:17:17.280 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: mon.a (mon.0) 380 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:17.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: 2021-03-15T22:17:17.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: audit 2021-03-15T22:17:17.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: 2021-03-15T22:17:16. 2021-03-15T22:17:17.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: 729844+0000 mon.b (mon.2) 15 : audit 2021-03-15T22:17:17.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:16 smithi076 conmon[56514]: [DBG] from='client.? 172.21.15.125:0/3073182613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:17:17.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:16 smithi125 conmon[56739]: audit 2021-03-15T22:17:16.268278+0000 mon.b (mon.2) 14 : audit 2021-03-15T22:17:17.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:16 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.125:0/224106860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe"}]: dispatch 2021-03-15T22:17:17.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:16 smithi125 conmon[56739]: audit 2021-03-15T22:17:16.268662+0000 mon.a (mon.0) 377 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe"}]: dispatch 2021-03-15T22:17:17.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:16 smithi125 conmon[56739]: audit 2021-03-15T22:17:16.273205+0000 mon.a (mon.0) 378 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe"}]': finished 2021-03-15T22:17:17.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:16 smithi125 conmon[56739]: cluster 2021-03-15T22:17:16.273290+0000 mon.a (mon.0) 379 : cluster [DBG] osdmap e41: 7 total, 6 up, 7 in 2021-03-15T22:17:17.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:16 smithi125 conmon[56739]: audit 2021-03-15T22:17:16.273456+0000 mon.a (mon.0) 380 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:17.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:16 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:17:17.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:16 smithi125 conmon[56739]: 15T22:17:16.729844+0000 mon.b (mon.2) 15 : audit [DBG] from='client.? 172.21.15.125:0/3073182613' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:17:18.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:17 smithi076 conmon[56514]: cluster 2021-03-15T22:17:16.933728+0000 mgr.y (mgr.14144) 164 : 2021-03-15T22:17:18.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:17 smithi076 conmon[56514]: cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:18.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:17 smithi076 conmon[62156]: cluster 2021-03-15T22:17:16.933728+0000 mgr.y 2021-03-15T22:17:18.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:17 smithi076 conmon[62156]: (mgr.14144) 164 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:18.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:17 smithi125 conmon[56739]: cluster 2021-03-15T22:17:16.933728+0000 mgr.y (mgr.14144) 164 : cluster [DBG] pgmap v129: 1 pgs: 1 active+clean; 0 B data, 30 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:20.238 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:19 smithi125 conmon[56739]: cluster 2021-03-15T22:17:18.934182+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:20.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:19 smithi076 conmon[56514]: cluster 2021-03-15T22:17:18.934182+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:20.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:19 smithi076 conmon[56514]: 2021-03-15T22:17:20.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:19 smithi076 conmon[62156]: cluster 2021-03-15T22:17:20.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:19 smithi076 conmon[62156]: 2021-03 2021-03-15T22:17:20.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:19 smithi076 conmon[62156]: -15T22:17:18.934182+0000 mgr.y (mgr.14144) 165 : cluster [DBG] pgmap v130: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:22.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:21 smithi076 conmon[56514]: cluster 2021-03-15T22:17:20.934583+0000 mgr.y (mgr. 2021-03-15T22:17:22.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:21 smithi076 conmon[56514]: 14144) 166 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:22.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:21 smithi076 conmon[62156]: cluster 2021-03-15T22:17:20.934583+0000 mgr.y (mgr.14144) 2021-03-15T22:17:22.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:21 smithi076 conmon[62156]: 166 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:22.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:21 smithi125 conmon[56739]: cluster 2021-03-15T22:17:20.934583+0000 mgr.y (mgr.14144) 2021-03-15T22:17:22.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:21 smithi125 conmon[56739]: 166 : cluster [DBG] pgmap v131: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:23.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:22 smithi076 conmon[56514]: audit 2021-03-15T22:17:22.076987+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-03-15T22:17:23.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:22 smithi076 conmon[56514]: audit 2021-03-15T22:17:22.077899+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:23.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:22 smithi076 conmon[62156]: audit 2021-03-15T22:17:22.076987+0000 mon.a (mon.0 2021-03-15T22:17:23.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:22 smithi076 conmon[62156]: ) 381 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-03-15T22:17:23.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:22 smithi076 conmon[62156]: audit 2021-03-15T22:17:22.077899+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:23.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:22 smithi076 conmon[62156]: 2021-03-15T22:17:23.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:22 smithi125 conmon[56739]: audit 2021-03-15T22:17:22.076987+0000 mon.a (mon.0) 381 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2021-03-15T22:17:23.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:22 smithi125 conmon[56739]: audit 2021-03-15T22:17:22.077899+0000 mon.a (mon.0) 382 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:24.253 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:22.078570+0000 mgr.y (mgr.14144) 167 : cephadm [INF] 2021-03-15T22:17:24.253 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: Deploying daemon osd.6 on smithi125 2021-03-15T22:17:24.254 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: cluster 2021-03-15T22:17:22.875868+0000 osd.2 (osd.2) 3 : cluster 2021-03-15T22:17:24.254 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: [DBG] purged_snaps scrub starts 2021-03-15T22:17:24.254 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:17:24.255 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: -15T22:17:22.876063+0000 osd.2 (osd.2 2021-03-15T22:17:24.255 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: ) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:24.256 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:17:24.256 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: -15T22:17:22.934945+0000 2021-03-15T22:17:24.257 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: mgr.y (mgr.14144) 168 : cluster [DBG] 2021-03-15T22:17:24.257 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: pgmap v132: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:24.257 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: cluster 2021-03-15T22:17: 2021-03-15T22:17:24.258 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: 22.962774+0000 mon.a (mon.0) 383 2021-03-15T22:17:24.258 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2021-03-15T22:17:24.259 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:17:24.259 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: T22:17:22.963128+0000 mon.a ( 2021-03-15T22:17:24.260 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:23 smithi125 conmon[56739]: mon.0) 384 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:24.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:22.078570+0000 mgr.y (mgr.14144) 167 : cephadm [INF] Deploying daemon osd.6 on smithi125 2021-03-15T22:17:24.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: cluster 2021-03-15T22:17:22.875868+0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:24.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: cluster 2021-03-15T22:17:22.876063+0000 2021-03-15T22:17:24.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: osd.2 (osd.2) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:24.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: cluster 2021-03-15T22:17:22. 2021-03-15T22:17:24.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: 934945+0000 mgr.y (mgr.14144) 168 : cluster 2021-03-15T22:17:24.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:24.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:17:24.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: 17:22.962774+0000 mon.a (mon.0) 2021-03-15T22:17:24.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: 383 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2021-03-15T22:17:24.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: audit 2021-03-15T22:17:22.963128 2021-03-15T22:17:24.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: +0000 mon.a (mon.0) 384 : audit [DBG] 2021-03-15T22:17:24.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:24.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:22.078570+0000 mgr.y (mgr.14144 2021-03-15T22:17:24.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: ) 167 : cephadm [INF] Deploying daemon osd.6 on smithi125 2021-03-15T22:17:24.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: cluster 2021-03-15T22:17:22.875868 2021-03-15T22:17:24.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: +0000 osd.2 (osd.2) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:24.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:17:24.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: 03-15T22:17:22.876063+0000 osd.2 (osd.2) 2021-03-15T22:17:24.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:24.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: cluster 2021-03-15T22:17:24.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: 2021-03-15T22:17:22.934945+0000 mgr.y (mgr.14144) 168 : cluster [DBG] pgmap v132: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:24.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: 2021-03-15T22:17:24.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: cluster 2021-03-15T22:17:24.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: 2021-03-15T22:17:22.962774+0000 2021-03-15T22:17:24.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: mon.a (mon.0) 383 : cluster [DBG] osdmap e42: 7 total, 6 up, 7 in 2021-03-15T22:17:24.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: audit 2021-03-15T22:17:22.963128+0000 mon.a (mon.0) 384 : audit [DBG] 2021-03-15T22:17:24.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:23 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:25.375 INFO:teuthology.orchestra.run.smithi125.stdout:Created osd(s) 6 on host 'smithi125' 2021-03-15T22:17:26.027 DEBUG:teuthology.orchestra.run.smithi125:osd.6> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.6.service 2021-03-15T22:17:26.030 INFO:tasks.cephadm:Deploying osd.7 on smithi125 with /dev/vg_nvme/lv_1... 2021-03-15T22:17:26.030 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2021-03-15T22:17:26.058 INFO:journalctl@ceph.osd.6.smithi125.stdout:-- Logs begin at Mon 2021-03-15 21:44:18 UTC. -- 2021-03-15T22:17:26.081 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:25 smithi125 conmon[56739]: cluster 2021-03-15T22:17:24.935333+0000 mgr.y (mgr.14144) 169 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:26.082 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:25 smithi125 conmon[56739]: audit 2021-03-15T22:17:25.359069+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:26.082 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:25 smithi125 conmon[56739]: audit 2021-03-15T22:17:25.364582+0000 mon.a ( 2021-03-15T22:17:26.082 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:25 smithi125 conmon[56739]: mon.0) 386 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:26.083 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:25 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:17:26.083 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:25 smithi125 conmon[56739]: :17:25.371879+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:26.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[56514]: cluster 2021-03-15T22:17:24.935333+0000 mgr.y (mgr.14144) 169 2021-03-15T22:17:26.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[56514]: : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:26.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[56514]: audit 2021-03-15T22:17:25.359069+0000 mon.a (mon.0) 385 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:26.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[56514]: audit 2021-03-15T22:17:25.364582+0000 mon.a (mon.0) 386 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:26.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[56514]: audit 2021-03-15T22:17:25.371879+0000 mon.a (mon.0) 387 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:26.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[62156]: cluster 2021-03-15T22:17:24.935333+0000 mgr.y (mgr 2021-03-15T22:17:26.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[62156]: .14144) 169 : cluster [DBG] pgmap v134: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:26.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[62156]: audit 2021-03-15T22:17:25.359069+0000 mon.a 2021-03-15T22:17:26.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[62156]: (mon.0) 385 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:26.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[62156]: audit 2021-03-15T22:17 2021-03-15T22:17:26.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[62156]: :25.364582+0000 mon.a (mon.0) 386 2021-03-15T22:17:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[62156]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:17:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[62156]: 15T22:17:25.371879+0000 mon.a ( 2021-03-15T22:17:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:25 smithi076 conmon[62156]: mon.0) 387 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:28.070 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:27 smithi125 conmon[56739]: cluster 2021-03-15T22:17:26.935698+0000 mgr.y (mgr.14144) 170 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:28.071 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:27 smithi125 conmon[56739]: audit 2021-03-15T22:17:27.521739+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:28.209 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping: /dev/vg_nvme/lv_1 2021-03-15T22:17:28.209 INFO:teuthology.orchestra.run.smithi125.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2021-03-15T22:17:28.210 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10+0 records in 2021-03-15T22:17:28.210 INFO:teuthology.orchestra.run.smithi125.stderr:10+0 records out 2021-03-15T22:17:28.210 INFO:teuthology.orchestra.run.smithi125.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0120372 s, 871 MB/s 2021-03-15T22:17:28.210 INFO:teuthology.orchestra.run.smithi125.stderr:--> Zapping successful for: 2021-03-15T22:17:28.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:27 smithi076 conmon[56514]: cluster 2021-03-15T22:17:26.935698+0000 mgr.y (mgr.14144) 170 : cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:28.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:27 smithi076 conmon[56514]: audit 2021-03-15T22:17:27.521739+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:28.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:27 smithi076 conmon[62156]: cluster 2021-03-15T22:17:26.935698+0000 mgr.y (mgr.14144) 170 : 2021-03-15T22:17:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:27 smithi076 conmon[62156]: cluster [DBG] pgmap v135: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:27 smithi076 conmon[62156]: audit 2021-03-15T22:17:27.521739+0000 mon.a (mon.0) 388 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:27 smithi076 conmon[62156]: 2021-03-15T22:17:28.786 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:17:28 smithi125 conmon[67289]: debug 2021-03-15T22:17:28.551+0000 7fd50032d080 -1 osd.6 0 log_to_monitors {default=true} 2021-03-15T22:17:29.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:28 smithi076 conmon[56514]: audit 2021-03-15T22:17:28.555260+0000 mon.b (mon.2) 16 : 2021-03-15T22:17:29.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:28 smithi076 conmon[56514]: audit [INF] from='osd.6 [v2:172.21.15.125:6816/171378275,v1:172.21.15.125:6817/171378275]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-15T22:17:29.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:28 smithi076 conmon[56514]: audit 2021-03-15T22:17:28.555695+0000 mon.a (mon.0) 389 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-15T22:17:29.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:28 smithi076 conmon[62156]: audit 2021-03-15T22:17:28.555260+0000 mon.b (mon.2) 2021-03-15T22:17:29.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:28 smithi076 conmon[62156]: 16 : audit [INF] from='osd.6 [v2:172.21.15.125:6816/171378275,v1:172.21.15.125:6817/171378275]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-15T22:17:29.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:28 smithi076 conmon[62156]: audit 2021-03-15T22:17:28.555695+0000 mon.a (mon.0) 389 : audit [INF] 2021-03-15T22:17:29.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:28 smithi076 conmon[62156]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-15T22:17:29.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:28 smithi125 conmon[56739]: audit 2021-03-15T22:17:28.555260+0000 mon.b (mon.2) 16 : audit 2021-03-15T22:17:29.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:28 smithi125 conmon[56739]: [INF] from='osd.6 [v2:172.21.15.125:6816/171378275,v1:172.21.15.125:6817/171378275]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-15T22:17:29.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:28 smithi125 conmon[56739]: audit 2021-03-15T22:17:28.555695+0000 mon.a (mon.0) 389 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2021-03-15T22:17:29.466 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch daemon add osd smithi125:vg_nvme/lv_1 2021-03-15T22:17:30.213 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:29 smithi125 conmon[56739]: cluster 2021-03-15T22:17:28.936017+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:30.214 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:29 smithi125 conmon[56739]: audit 2021-03-15T22:17:28.957508+0000 mon.a (mon.0) 390 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-03-15T22:17:30.214 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:29 smithi125 conmon[56739]: cluster 2021-03-15T22:17:28.957622+0000 mon.a (mon.0) 391 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2021-03-15T22:17:30.214 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:29 smithi125 conmon[56739]: audit 2021-03-15T22:17:28.957868+0000 mon.a (mon.0) 392 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:30.214 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:29 smithi125 conmon[56739]: audit 2021-03-15T22:17:28.958111+0000 mon.b (mon.2) 17 : audit [INF] from='osd.6 [v2:172.21.15.125:6816/171378275,v1:172.21.15.125:6817/171378275]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:30.215 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:29 smithi125 conmon[56739]: audit 2021-03-15T22:17:28.958485+0000 mon.a (mon.0) 393 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:30.215 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:17:29 smithi125 conmon[67289]: debug 2021-03-15T22:17:29.986+0000 7fd4f689e700 -1 osd.6 0 waiting for initial osdmap 2021-03-15T22:17:30.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: cluster 2021-03-15T22:17:28.936017+0000 mgr.y ( 2021-03-15T22:17:30.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: mgr.14144) 171 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:30.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: audit 2021-03-15T22:17:28.957508+0000 mon.a (mon.0) 390 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-03-15T22:17:30.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: cluster 2021-03-15T22:17:28.957622+0000 mon.a (mon.0) 2021-03-15T22:17:30.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: 391 : cluster [DBG] osdmap e43: 7 total, 6 up, 7 in 2021-03-15T22:17:30.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:17:30.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: 03-15T22:17:28. 2021-03-15T22:17:30.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: 957868+0000 mon.a (mon.0) 2021-03-15T22:17:30.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: 392 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:30.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: audit 2021-03-15T22:17:30.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: 2021-03-15T22:17 2021-03-15T22:17:30.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: :28.958111+0000 mon.b (mon 2021-03-15T22:17:30.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: .2) 17 : audit [INF] 2021-03-15T22:17:30.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: from='osd.6 [v2:172.21.15.125:6816/171378275,v1:172.21.15.125:6817/171378275]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:30.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:17:30.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: 15T22:17:28.958485 2021-03-15T22:17:30.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: +0000 mon.a (mon.0) 393 2021-03-15T22:17:30.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[56514]: : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:30.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: cluster 2021-03-15T22:17:30.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 2021-03-15T22:17:30.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 2021 2021-03-15T22:17:30.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: -03-15T22 2021-03-15T22:17:30.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: :17:28.936017+0000 mgr.y (mgr.14144) 171 : cluster [DBG] pgmap v136: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:30.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: audit 2021-03-15T22:17:30.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 2021- 2021-03-15T22:17:30.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 03-15T22:17:28.957508+0000 mon.a (mon. 2021-03-15T22:17:30.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 0) 390 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2021-03-15T22:17:30.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: cluster 2021-03-15T22:17:30.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 2021-03-15T22:17:30.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 2021 2021-03-15T22:17:30.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: - 2021-03-15T22:17:30.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 03 2021-03-15T22:17:30.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: - 2021-03-15T22:17:30.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 15 2021-03-15T22:17:30.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: T 2021-03-15T22:17:30.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 22 2021-03-15T22:17:30.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: : 2021-03-15T22:17:30.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 17 2021-03-15T22:17:30.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: : 2021-03-15T22:17:30.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 28 2021-03-15T22:17:30.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: . 2021-03-15T22:17:30.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 957622+0000 mon.a (mon.0) 391 : cluster 2021-03-15T22:17:30.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: [DBG] osdmap e43: 7 total, 6 up, 7 in 2021-03-15T22:17:30.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:17:30.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: -15T22:17:28. 2021-03-15T22:17:30.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 957868+0000 mon.a (mon.0 2021-03-15T22:17:30.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: ) 392 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:30.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 2021-03-15T22:17:30.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: audit 2021-03-15T22:17:30.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 2021-03-15T22:17:30.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 2021 2021-03-15T22:17:30.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: - 2021-03-15T22:17:30.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 03 2021-03-15T22:17:30.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: - 2021-03-15T22:17:30.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 15 2021-03-15T22:17:30.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: T 2021-03-15T22:17:30.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 22 2021-03-15T22:17:30.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: : 2021-03-15T22:17:30.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 17 2021-03-15T22:17:30.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: : 2021-03-15T22:17:30.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 28 2021-03-15T22:17:30.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: . 2021-03-15T22:17:30.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 958111 2021-03-15T22:17:30.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: +0000 2021-03-15T22:17:30.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 2021-03-15T22:17:30.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: mon.b 2021-03-15T22:17:30.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: ( 2021-03-15T22:17:30.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: mon 2021-03-15T22:17:30.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: .2) 17 : audit [INF] 2021-03-15T22:17:30.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: from='osd.6 [v2:172.21.15.125:6816/171378275,v1:172.21.15.125:6817/171378275]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:30.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: audit 2021-03-15T22:17:30.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 2021-03-15T22:17:28.958485+0000 mon.a (mon. 2021-03-15T22:17:30.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:29 smithi076 conmon[62156]: 0) 393 : audit [INF] from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:31.241 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:30 smithi125 conmon[56739]: audit 2021-03-15T22:17:29.956323+0000 mon.a (mon 2021-03-15T22:17:31.242 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:30 smithi125 conmon[56739]: .0) 394 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:17:31.242 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:30 smithi125 conmon[56739]: cluster 2021-03-15T22:17:29.956439+0000 mon.a (mon.0) 395 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2021-03-15T22:17:31.242 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:30 smithi125 conmon[56739]: audit 2021-03-15T22:17:29.957052+0000 mon.a (mon.0) 396 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:31.243 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:30 smithi125 conmon[56739]: audit 2021-03-15T22:17:29.968323+0000 mon.a (mon.0) 397 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:31.243 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:30 smithi125 conmon[56739]: cluster 2021-03-15T22:17:30.936368+0000 mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:31.243 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:30 smithi125 conmon[56739]: cluster 2021-03-15T22:17:30.957316+0000 mon.a (mon.0) 398 : cluster [INF] osd.6 [v2:172.21.15.125:6816/171378275,v1:172.21.15.125:6817/171378275] boot 2021-03-15T22:17:31.243 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:30 smithi125 conmon[56739]: cluster 2021-03-15T22:17:30.957387+0000 mon.a (mon.0) 399 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2021-03-15T22:17:31.243 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:30 smithi125 conmon[56739]: audit 2021-03-15T22:17:30.957555+0000 mon.a (mon.0) 400 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:31.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: audit 2021-03-15T22:17:29.956323+0000 mon.a (mon.0 2021-03-15T22:17:31.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: ) 394 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:17:31.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: cluster 2021-03-15T22:17:29.956439+0000 mon.a (mon.0) 395 : cluster [DBG] osdmap e44: 7 total, 6 up, 7 in 2021-03-15T22:17:31.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: audit 2021-03-15T22: 2021-03-15T22:17:31.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: 17:29.957052+0000 mon.a ( 2021-03-15T22:17:31.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: mon.0) 396 : audit 2021-03-15T22:17:31.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:31.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:17:31.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: 03-15T22:17:29.968323+0000 2021-03-15T22:17:31.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: mon.a (mon.0) 397 : audit 2021-03-15T22:17:31.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:31.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:17:31.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: 03-15T22:17:30.936368+0000 2021-03-15T22:17:31.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: mgr.y (mgr.14144) 172 : cluster [DBG] 2021-03-15T22:17:31.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: pgmap v139: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:31.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: cluster 2021-03-15T22:17:30.957316 2021-03-15T22:17:31.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: +0000 mon.a (mon.0) 398 : cluster [INF] osd.6 [v2:172.21.15.125:6816/171378275,v1:172.21.15.125:6817/171378275] boot 2021-03-15T22:17:31.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: 2021-03-15T22:17:31.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:17:31.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: 03-15T22:17:30 2021-03-15T22:17:31.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: .957387+0000 mon.a ( 2021-03-15T22:17:31.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: mon.0) 399 : cluster 2021-03-15T22:17:31.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: [DBG] osdmap e45: 7 total, 7 up, 7 in 2021-03-15T22:17:31.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: audit 2021 2021-03-15T22:17:31.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: -03-15T22:17:30.957555+0000 2021-03-15T22:17:31.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[56514]: mon.a (mon.0) 400 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:31.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: audit 2021-03-15T22:17:29.956323 2021-03-15T22:17:31.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: +0000 mon.a (mon.0) 394 : audit [INF] from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:17:31.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:17:31.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: 22:17:29.956439+0000 mon.a (mon.0) 395 : cluster [DBG] 2021-03-15T22:17:31.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: osdmap e44: 7 total, 6 up, 7 in 2021-03-15T22:17:31.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: audit 2021-03-15T22:17:31.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:17:31.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: 15T22:17:29.957052 2021-03-15T22:17:31.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: +0000 mon.a (mon.0) 396 : audit [DBG] 2021-03-15T22:17:31.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:31.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:17:31.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: 15T22:17:29.968323+0000 mon.a ( 2021-03-15T22:17:31.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: mon.0) 397 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:31.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: cluster 2021-03-15T22:17:31.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: 2021-03-15T22:17:30.936368+0000 2021-03-15T22:17:31.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: mgr.y (mgr.14144) 172 : cluster [DBG] pgmap v139: 1 pgs: 1 active+clean; 0 B data, 31 MiB used, 536 GiB / 536 GiB avail 2021-03-15T22:17:31.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: 2021-03-15T22:17:31.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:17:31.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: 17:30.957316+0000 mon.a (mon.0) 398 : 2021-03-15T22:17:31.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: cluster [INF] osd.6 [v2:172.21.15.125:6816/171378275,v1:172.21.15.125:6817/171378275] boot 2021-03-15T22:17:31.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:17:31.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: -03-15T22:17:30.957387+0000 2021-03-15T22:17:31.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: mon.a (mon.0) 399 : cluster [DBG] osdmap e45: 7 total, 7 up, 7 in 2021-03-15T22:17:31.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: audit 2021-03-15T22:17:31.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: 2021-03-15T22:17:30.957555+0000 2021-03-15T22:17:31.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: mon.a (mon.0) 400 : audit 2021-03-15T22:17:31.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:30 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2021-03-15T22:17:31.767 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: debug 2021-03-15T22:17:31.510+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:31.767 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:17:31.768 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:31.768 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:17:31.768 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:31.768 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:17:31.769 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:31.769 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:31.769 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:31.769 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: debug 2021-03-15T22:17:31.520+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:31.769 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:17:31.770 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:31.770 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:17:31.770 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:31.770 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:17:31.771 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:31.771 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:31.771 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:31 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:32.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: cluster 2021-03-15T22:17:29.507795+0000 osd.6 (osd.6) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:32.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: 2021-03-15T22:17:32.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: cluster 2021-03-15T22:17:29.507884+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:32.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:31.498414+0000 mgr.y ( 2021-03-15T22:17:32.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: mgr.14144) 173 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-15T22:17:32.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: audit 2021-03-15T22:17:31.504141 2021-03-15T22:17:32.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: +0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:32.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:31.504863 2021-03-15T22:17:32.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 174 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:32.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: audit 2021-03-15T22 2021-03-15T22:17:32.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: :17:31.510393+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:32.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: 2021-03-15T22:17:32.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:32.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: 2021-03-15T22:17:31.511561+0000 mgr.y (mgr.14144) 2021-03-15T22:17:32.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: 175 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:32.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:17:32.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:32.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:17:32.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:32.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:17:32.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:32.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:32.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:32.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: audit 2021-03-15T22:17:31.512318+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:32.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:31.514016+0000 mgr.y (mgr 2021-03-15T22:17:32.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: .14144) 176 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:32.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:17:32.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: 03-15T22:17:31.519789+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:32.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:17:32.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: -15T22:17:31.520884+0000 mgr.y (mgr.14144) 2021-03-15T22:17:32.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: 177 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:32.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:17:32.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:32.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: cluster 2021-03-15T22:17:29.507795+0000 osd.6 (osd.6 2021-03-15T22:17:32.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: ) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:32.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: cluster 2021-03-15T22:17:29.507884+0000 osd.6 (osd. 2021-03-15T22:17:32.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 6) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:32.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:31.498414+0000 2021-03-15T22:17:32.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: mgr.y (mgr.14144) 173 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-15T22:17:32.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 2021-03-15T22:17:32.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: audit 2021-03-15T22:17 2021-03-15T22:17:32.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: :31.504141+0000 mon.a (mon.0) 401 : 2021-03-15T22:17:32.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:32.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:17:32.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: -03-15T22:17:31.504863+0000 mgr.y 2021-03-15T22:17:32.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: (mgr.14144) 174 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:32.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:32.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: -03-15T22:17:31.510393+0000 mon.a (mon.0) 402 2021-03-15T22:17:32.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:32.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:17:32.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: -03-15T22:17:31.511561+0000 mgr.y 2021-03-15T22:17:32.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: (mgr.14144) 175 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:32.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:17:32.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:32.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:17:32.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:32.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:17:32.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:32.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:32.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:32.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: audit 2021-03-15T22:17:31.512318+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:32.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:32.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:17:32.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: :17:31.514016+0000 mgr.y (mgr.14144) 2021-03-15T22:17:32.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 176 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:32.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: audit 2021-03-15T22:17:32.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 2021-03-15T22:17:31.519789+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:32.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:32.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 2021-03-15T22:17:31.520884+0000 mgr.y ( 2021-03-15T22:17:32.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: mgr.14144) 177 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:32.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:17:32.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:32.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:17:32.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:32.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:32.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:32.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: audit 2021-03-15T22:17:31.873827+0000 mgr.y (mgr.14144) 178 : audit [DBG] from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:32.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: audit 2021-03-15T22:17:31.874574+0000 mon.a (mon.0) 2021-03-15T22:17:32.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: 405 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:17:32.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:17:32.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: -15T22:17:31.875925+0000 mgr.y (mgr.14144) 179 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-15T22:17:32.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: audit 2021-03-15T 2021-03-15T22:17:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: 22:17:31.876234+0000 mon.a (mon.0) 406 : audit 2021-03-15T22:17:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:17:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: audit 2021-03-15T22:17:32.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: 2021-03-15T22:17:31.876972+0000 mon.a (mon.0) 407 2021-03-15T22:17:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: cluster 2021-03-15T22:17:31 2021-03-15T22:17:32.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[56514]: .959914+0000 mon.a (mon.0) 408 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2021-03-15T22:17:32.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:17:32.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:32.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:17:32.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:32.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:17:32.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:32.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:32.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:32.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: audit 2021-03-15T22:17:31.873827+0000 mgr.y (mgr.14144) 178 : audit [DBG] from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:32.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:32.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: -03-15T22:17:31.874574+0000 mon.a (mon. 2021-03-15T22:17:32.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 0) 405 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:17:32.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: cephadm 2021-03 2021-03-15T22:17:32.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: -15T22:17:31.875925+0000 mgr.y (mgr.14144) 179 : cephadm 2021-03-15T22:17:32.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: [INF] Applying service osd.None on host smithi125... 2021-03-15T22:17:32.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:17:32.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 15T22:17:31.876234+0000 mon.a (mon. 2021-03-15T22:17:32.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 0) 406 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:17:32.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:17:32.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: -15T22:17:31.876972+0000 mon.a (mon. 2021-03-15T22:17:32.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 0) 407 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:32.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: cluster 2021-03-15T22:17:32.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 2021-03-15T22:17:31.959914+0000 2021-03-15T22:17:32.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: mon.a (mon.0) 408 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2021-03-15T22:17:32.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:32 smithi076 conmon[62156]: 2021-03-15T22:17:32.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: cluster 2021-03-15T22:17:29.507795+0000 osd.6 (osd. 2021-03-15T22:17:32.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: 6) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:32.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: cluster 2021-03-15T22:17:29.507884+0000 osd.6 (osd.6) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:32.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:31.498414+0000 mgr.y (mgr.14144) 173 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-15T22:17:32.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: audit 2021-03-15T22:17:31.504141+0000 mon.a (mon.0) 401 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:32.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:31.504863+0000 mgr.y (mgr.14144) 174 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:32.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: audit 2021-03-15T22:17:31.510393+0000 mon.a (mon.0) 402 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:32.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:31.511561+0000 mgr.y (mgr.14144) 175 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:32.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:17:32.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:32.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:17:32.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:32.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:17:32.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:32.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:32.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:32.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: audit 2021-03-15T22:17:31.512318+0000 mon.a (mon.0) 403 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:32.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:31.514016+0000 mgr.y (mgr.14144) 176 : cephadm [INF] 2021-03-15T22:17:32.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:32.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: audit 2021-03-15T22:17:31.519789+0000 mon.a (mon.0) 404 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:32.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:31.520884+0000 mgr.y (mgr.14144) 177 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:32.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:17:32.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:32.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:17:32.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:32.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:17:32.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:32.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:32.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:32.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: audit 2021-03-15T22:17:31.873827+0000 mgr.y 2021-03-15T22:17:32.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: (mgr.14144) 178 : audit [DBG] from='client.34274 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi125:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:32.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: audit 2021-03-15T22:17:31.874574+0000 mon.a (mon.0) 405 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2021-03-15T22:17:32.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:31.875925+0000 mgr.y (mgr.14144 2021-03-15T22:17:32.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: ) 179 : cephadm [INF] Applying service osd.None on host smithi125... 2021-03-15T22:17:32.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:17:32.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: T22:17:31.876234+0000 mon.a (mon.0) 406 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2021-03-15T22:17:32.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: 2021-03-15T22:17:32.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: audit 2021-03-15T22:17 2021-03-15T22:17:32.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: :31.876972+0000 mon.a (mon.0) 407 : audit [DBG] 2021-03-15T22:17:32.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:32.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:17:32.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:32 smithi125 conmon[56739]: 22:17:31.959914+0000 mon.a (mon.0) 408 : cluster [DBG] osdmap e46: 7 total, 7 up, 7 in 2021-03-15T22:17:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:33 smithi076 conmon[56514]: cluster 2021-03-15T22:17:32.936730+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v142: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:33 smithi076 conmon[56514]: 2021-03-15T22:17:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:33 smithi076 conmon[56514]: cluster 2021-03-15T22:17:32.960267+0000 mon.a (mon.0) 2021-03-15T22:17:34.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:33 smithi076 conmon[56514]: 409 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2021-03-15T22:17:34.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:33 smithi076 conmon[62156]: cluster 2021-03-15T22:17:32.936730+0000 mgr.y (mgr.14144) 180 2021-03-15T22:17:34.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:33 smithi076 conmon[62156]: : cluster [DBG] pgmap v142: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:34.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:33 smithi076 conmon[62156]: cluster 2021-03-15T22:17:32.960267+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2021-03-15T22:17:34.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:33 smithi076 conmon[62156]: 2021-03-15T22:17:34.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:33 smithi125 conmon[56739]: cluster 2021-03-15T22:17:32.936730+0000 mgr.y (mgr.14144) 180 : cluster [DBG] pgmap v142: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:33 smithi125 conmon[56739]: cluster 2021-03-15T22:17:32.960267+0000 mon.a (mon.0) 409 : cluster [DBG] osdmap e47: 7 total, 7 up, 7 in 2021-03-15T22:17:35.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: audit 2021-03-15T22:17:34.079318+0000 mon.b (mon.2) 18 : audit [INF] from='client.? 172.21.15.125:0/4147797443' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614"}]: dispatch 2021-03-15T22:17:35.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: audit 2021-03-15T22:17:34.079877+0000 mon.a (mon.0) 410 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614"}]: dispatch 2021-03-15T22:17:35.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: audit 2021-03-15T22:17:34.084180+0000 mon.a (mon.0) 2021-03-15T22:17:35.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: 411 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614"}]': finished 2021-03-15T22:17:35.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: cluster 2021-03-15T22:17:34.084259+0000 2021-03-15T22:17:35.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: mon.a (mon.0) 412 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2021-03-15T22:17:35.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: audit 2021-03-15T22:17:35.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: 2021-03-15T22:17:34.084375+0000 mon.a (mon.0) 413 : audit 2021-03-15T22:17:35.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:35.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: audit 2021-03-15T22:17:35.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: 2021-03-15T22:17:34.599931+0000 mon.b (mon.2) 19 : audit [DBG] 2021-03-15T22:17:35.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[62156]: from='client.? 172.21.15.125:0/2715234691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:17:35.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: audit 2021-03-15T22:17:34.079318+0000 mon.b (mon.2 2021-03-15T22:17:35.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: ) 18 : audit [INF] from='client.? 172.21.15.125:0/4147797443' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614"}]: dispatch 2021-03-15T22:17:35.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: audit 2021-03-15T22:17:34.079877+0000 mon.a (mon.0) 410 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614"}]: dispatch 2021-03-15T22:17:35.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: audit 2021-03-15T22:17:35.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: 2021-03-15T22:17: 2021-03-15T22:17:35.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: 34.084180+0000 mon.a (mon.0) 411 2021-03-15T22:17:35.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614"}]': finished 2021-03-15T22:17:35.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:17:35.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: -03-15T22:17:34.084259+0000 2021-03-15T22:17:35.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: mon.a (mon.0) 412 : cluster [DBG] osdmap e48: 8 total, 7 up, 8 in 2021-03-15T22:17:35.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: 2021-03-15T22:17:35.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: audit 2021-03-15T22:17 2021-03-15T22:17:35.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: :34.084375+0000 mon.a (mon.0) 413 2021-03-15T22:17:35.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:35.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: audit 2021-03-15T 2021-03-15T22:17:35.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: 22:17:34.599931+0000 mon.b (mon. 2021-03-15T22:17:35.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:34 smithi076 conmon[56514]: 2) 19 : audit [DBG] from='client.? 172.21.15.125:0/2715234691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:17:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: audit 2021-03-15T22:17:34.079318+0000 mon.b (mon.2) 18 : audit [INF] from='client.? 172.21.15.125:0/4147797443' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614"}]: dispatch 2021-03-15T22:17:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: audit 2021-03-15T22:17:34.079877+0000 2021-03-15T22:17:35.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: mon.a (mon.0) 410 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614"}]: dispatch 2021-03-15T22:17:35.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: audit 2021-03-15T22:17:34.084180+0000 mon.a (mon.0) 411 : audit [INF] from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614"}]': finished 2021-03-15T22:17:35.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:17:35.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: T22:17:34.084259+0000 mon.a (mon.0) 412 : cluster [DBG] 2021-03-15T22:17:35.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: osdmap e48: 8 total, 7 up, 8 in 2021-03-15T22:17:35.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: audit 2021 2021-03-15T22:17:35.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: -03-15T22:17:34 2021-03-15T22:17:35.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: .084375+0000 mon.a (mon.0 2021-03-15T22:17:35.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: ) 413 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:35.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:17:35.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: 03-15T22:17:34.599931+0000 2021-03-15T22:17:35.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: mon.b (mon.2) 19 : audit 2021-03-15T22:17:35.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:34 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.125:0/2715234691' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2021-03-15T22:17:36.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:35 smithi076 conmon[56514]: cluster 2021-03-15T22:17:34.937232+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v145: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:36.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:35 smithi076 conmon[56514]: 2021-03-15T22:17:36.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:35 smithi076 conmon[62156]: cluster 2021-03-15T22:17:34.937232+0000 mgr.y (mgr.14144) 181 2021-03-15T22:17:36.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:35 smithi076 conmon[62156]: : cluster [DBG] pgmap v145: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:36.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:35 smithi125 conmon[56739]: cluster 2021-03-15T22:17:34.937232+0000 mgr.y (mgr.14144) 181 : cluster [DBG] pgmap v145: 1 pgs: 1 remapped+peering; 0 B data, 36 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:36.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:35 smithi125 conmon[56739]: 2021-03-15T22:17:38.093 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:37 smithi125 conmon[56739]: cluster 2021-03-15T22:17:36.937665+0000 mgr.y (mgr.14144) 182 : cluster [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:38.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:37 smithi076 conmon[56514]: cluster 2021-03-15T22:17:36.937665+0000 mgr.y (mgr.14144) 182 : cluster [DBG] 2021-03-15T22:17:38.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:37 smithi076 conmon[56514]: pgmap v146: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:38.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:37 smithi076 conmon[62156]: cluster 2021-03-15T22:17:36.937665+0000 mgr.y (mgr.14144) 182 : cluster 2021-03-15T22:17:38.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:37 smithi076 conmon[62156]: [DBG] pgmap v146: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:39.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:38 smithi076 conmon[56514]: cluster 2021-03-15T22:17:38.937995+0000 mgr.y (mgr.14144) 183 : cluster [DBG] 2021-03-15T22:17:39.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:38 smithi076 conmon[56514]: pgmap v147: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:39.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:38 smithi076 conmon[62156]: cluster 2021-03-15T22:17:38.937995+0000 mgr.y ( 2021-03-15T22:17:39.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:38 smithi076 conmon[62156]: mgr.14144) 183 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:39.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:38 smithi125 conmon[56739]: cluster 2021-03-15T22:17:38.937995+0000 mgr.y (mgr.14144) 183 : cluster [DBG] pgmap v147: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:40.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[62156]: cluster 2021-03-15T22:17:38.835670+0000 osd.3 (osd. 2021-03-15T22:17:40.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[62156]: 3) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:40.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[62156]: cluster 2021-03-15T22:17:38.835875+0000 osd.3 (osd 2021-03-15T22:17:40.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[62156]: .3) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:40.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:17:40.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[62156]: T22:17:38.977065+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2021-03-15T22:17:40.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[62156]: audit 2021-03-15T22:17 2021-03-15T22:17:40.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[62156]: :38.977358+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:40.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[56514]: cluster 2021-03-15T22:17:38.835670+0000 osd.3 (osd.3) 3 : cluster 2021-03-15T22:17:40.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[56514]: [DBG] purged_snaps scrub starts 2021-03-15T22:17:40.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[56514]: cluster 2021-03-15T22:17:38.835875+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:40.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:17:40.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[56514]: 15T22:17:38.977065+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2021-03-15T22:17:40.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:17:40.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[56514]: -15T22:17:38.977358+0000 mon.a 2021-03-15T22:17:40.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[56514]: (mon.0) 415 : audit [DBG] 2021-03-15T22:17:40.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:39 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:40.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:39 smithi125 conmon[56739]: cluster 2021-03-15T22:17:38.835670+0000 osd.3 (osd.3) 3 : cluster 2021-03-15T22:17:40.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:39 smithi125 conmon[56739]: [DBG] purged_snaps scrub starts 2021-03-15T22:17:40.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:39 smithi125 conmon[56739]: cluster 2021-03-15T22:17:38.835875+0000 osd.3 (osd.3) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:40.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:39 smithi125 conmon[56739]: cluster 2021-03-15T22:17:38.977065+0000 mon.a (mon.0) 414 : cluster [DBG] osdmap e49: 8 total, 7 up, 8 in 2021-03-15T22:17:40.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:39 smithi125 conmon[56739]: audit 2021-03-15T22:17:38.977358+0000 mon.a (mon.0) 415 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:41.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[56514]: audit 2021-03-15T22:17:40.025316+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-03-15T22:17:41.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[56514]: audit 2021-03-15T22:17 2021-03-15T22:17:41.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[56514]: :40.026016+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:41.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:40.026599+0000 mgr.y (mgr.14144) 184 : cephadm [INF] Deploying daemon osd.7 on smithi125 2021-03-15T22:17:41.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:17:41.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[56514]: 15T22:17:40.938476+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:41.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[62156]: audit 2021-03-15T22:17:40.025316+0000 mon.a (mon.0) 416 : audit 2021-03-15T22:17:41.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-03-15T22:17:41.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[62156]: audit 2021-03-15T22:17:40.026016+0000 mon.a (mon.0) 417 : audit [DBG] 2021-03-15T22:17:41.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:41.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[62156]: cephadm 2021-03-15 2021-03-15T22:17:41.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[62156]: T22:17:40.026599+0000 mgr.y (mgr.14144) 184 : 2021-03-15T22:17:41.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[62156]: cephadm [INF] Deploying daemon osd.7 on smithi125 2021-03-15T22:17:41.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:17:41.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[62156]: -15T22:17:40.938476+0000 mgr.y 2021-03-15T22:17:41.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:40 smithi076 conmon[62156]: (mgr.14144) 185 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:41.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:40 smithi125 conmon[56739]: audit 2021-03-15T22:17:40.025316+0000 mon.a (mon.0) 416 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2021-03-15T22:17:41.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:40 smithi125 conmon[56739]: audit 2021-03-15T22:17:40.026016+0000 mon.a (mon.0) 417 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:41.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:40 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:40.026599+0000 mgr.y (mgr.14144) 184 : cephadm [INF] Deploying daemon osd.7 on smithi125 2021-03-15T22:17:41.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:40 smithi125 conmon[56739]: cluster 2021-03-15T22:17:40.938476+0000 mgr.y (mgr.14144) 185 : cluster [DBG] pgmap v149: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:43.225 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:43 smithi125 conmon[56739]: cluster 2021-03-15T22:17:42.938882+0000 mgr.y (mgr.14144) 186 : cluster 2021-03-15T22:17:43.226 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:43 smithi125 conmon[56739]: [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:43.452 INFO:teuthology.orchestra.run.smithi125.stdout:Created osd(s) 7 on host 'smithi125' 2021-03-15T22:17:43.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:43 smithi076 conmon[56514]: cluster 2021-03-15T22:17:42.938882+0000 mgr.y (mgr.14144) 186 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:43.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:43 smithi076 conmon[62156]: cluster 2021-03-15T22:17:42.938882+0000 mgr.y (mgr. 2021-03-15T22:17:43.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:43 smithi076 conmon[62156]: 14144) 186 : cluster [DBG] pgmap v150: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:44.081 DEBUG:teuthology.orchestra.run.smithi125:osd.7> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.7.service 2021-03-15T22:17:44.084 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi076 2021-03-15T22:17:44.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch apply rgw foo.a --placement '1;smithi076=foo.a' 2021-03-15T22:17:44.113 INFO:journalctl@ceph.osd.7.smithi125.stdout:-- Logs begin at Mon 2021-03-15 21:44:18 UTC. -- 2021-03-15T22:17:44.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:44 smithi125 conmon[56739]: cluster 2021-03-15T22:17:43.395888+0000 osd.6 ( 2021-03-15T22:17:44.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:44 smithi125 conmon[56739]: osd.6) 3 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:17:44.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:44 smithi125 conmon[56739]: audit 2021-03-15T22:17:43.439045+0000 mon.a (mon 2021-03-15T22:17:44.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:44 smithi125 conmon[56739]: .0) 418 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:44.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:44 smithi125 conmon[56739]: audit 2021-03-15T22:17:43.443745 2021-03-15T22:17:44.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:44 smithi125 conmon[56739]: +0000 mon.a (mon.0) 419 : audit 2021-03-15T22:17:44.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:44 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:44.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:44 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:17:44.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:44 smithi125 conmon[56739]: 15T22:17:43.448711+0000 mon.a 2021-03-15T22:17:44.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:44 smithi125 conmon[56739]: (mon.0) 420 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:44.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[56514]: cluster 2021-03-15T22:17:43.395888+0000 osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:17:44.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[56514]: 2021-03-15T22:17:44.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[56514]: audit 2021-03-15T22:17:43.439045+0000 mon.a (mon.0) 418 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:44.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[56514]: audit 2021-03-15T22:17:44.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[56514]: 2021-03-15T22:17:43.443745+0000 mon.a (mon.0) 419 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:44.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[56514]: audit 2021-03-15T22:17:43.448711+0000 mon.a (mon.0) 420 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:44.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: cluster 2021-03-15T22:17:43.395888+0000 2021-03-15T22:17:44.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: osd.6 (osd.6) 3 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:17:44.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:17:44.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: 15T22:17:43.439045+0000 mon.a (mon 2021-03-15T22:17:44.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: .0) 418 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:44.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: 2021-03-15T22:17:44.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: audit 2021-03-15T22:17:44.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: 2021-03-15T22:17 2021-03-15T22:17:44.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: :43.443745+0000 mon.a (mon 2021-03-15T22:17:44.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: .0) 419 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:44.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:17:44.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: 03-15T22:17:43.448711+0000 2021-03-15T22:17:44.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: mon.a (mon.0) 420 : audit 2021-03-15T22:17:44.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:44 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:46.125 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:45 smithi125 conmon[56739]: cluster 2021-03-15T22:17:44.939316+0000 mgr.y (mgr.14144) 187 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:46.125 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:45 smithi125 conmon[56739]: audit 2021-03-15T22:17:45.679054+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:46.160 INFO:teuthology.orchestra.run.smithi125.stdout:Scheduled rgw.foo.a update... 2021-03-15T22:17:46.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:45 smithi076 conmon[56514]: cluster 2021-03-15T22:17:44.939316+0000 mgr.y (mgr 2021-03-15T22:17:46.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:45 smithi076 conmon[56514]: .14144) 187 : cluster [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:46.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:45 smithi076 conmon[56514]: audit 2021-03-15T22:17:45.679054+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:46.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:45 smithi076 conmon[62156]: cluster 2021-03-15T22:17:44.939316+0000 mgr.y (mgr.14144) 187 : cluster 2021-03-15T22:17:46.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:45 smithi076 conmon[62156]: [DBG] pgmap v151: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:46.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:45 smithi076 conmon[62156]: audit 2021-03-15T22:17:45.679054+0000 mon.a (mon.0) 421 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:46.861 INFO:journalctl@ceph.osd.7.smithi125.stdout:Mar 15 22:17:46 smithi125 conmon[70700]: debug 2021-03-15T22:17:46.594+0000 7f382f833080 -1 osd.7 0 log_to_monitors {default=true} 2021-03-15T22:17:47.238 DEBUG:teuthology.orchestra.run.smithi076:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@rgw.foo.a.service 2021-03-15T22:17:47.242 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi125 2021-03-15T22:17:47.242 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool create iscsi 3 3 replicated 2021-03-15T22:17:47.284 INFO:journalctl@ceph.rgw.foo.a.smithi076.stdout:-- Logs begin at Mon 2021-03-15 21:44:23 UTC. -- 2021-03-15T22:17:47.424 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:47 smithi125 conmon[56739]: audit 2021-03-15T22:17:46.154266+0000 mgr.y (mgr.14144) 188 : audit [DBG] from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi076=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:47.425 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:46.154907+0000 mgr.y (mgr.14144) 189 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi076=foo.a;count:1 2021-03-15T22:17:47.425 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:47 smithi125 conmon[56739]: audit 2021-03-15T22:17:46.158558+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:47.425 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:47 smithi125 conmon[56739]: audit 2021-03-15T22:17:46.598179+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 [v2:172.21.15.125:6824/652003948,v1:172.21.15.125:6825/652003948]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-15T22:17:47.426 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:47 smithi125 conmon[56739]: audit 2021-03-15T22:17:46.598654+0000 mon.a (mon.0) 423 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-15T22:17:47.426 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:47 smithi125 conmon[56739]: cluster 2021-03-15T22:17:46.939841+0000 mgr.y (mgr.14144) 190 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:47.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[56514]: audit 2021-03-15T22:17:46.154266+0000 mgr.y (mgr.14144) 188 : audit [DBG] from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi076=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:47.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:46.154907+0000 mgr.y (mgr.14144) 189 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi076=foo.a;count:1 2021-03-15T22:17:47.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[56514]: audit 2021-03-15T22:17:46.158558+0000 mon.a (mon.0) 422 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:47.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[56514]: audit 2021-03-15T22:17:46.598179+0000 mon.b (mon.2) 20 : audit [INF] from='osd.7 [v2:172.21.15.125:6824/652003948,v1:172.21.15.125:6825/652003948]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-15T22:17:47.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[56514]: audit 2021-03-15 2021-03-15T22:17:47.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[56514]: T22:17:46.598654+0000 mon.a (mon.0) 2021-03-15T22:17:47.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[56514]: 423 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-15T22:17:47.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[56514]: cluster 2021-03-15T22:17:46.939841 2021-03-15T22:17:47.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 190 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:47.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: audit 2021-03-15T22:17:46. 2021-03-15T22:17:47.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: 154266+0000 mgr.y (mgr.14144) 188 : audit [DBG] from='client.34298 -' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi076=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:47.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: cephadm 2021-03-15T 2021-03-15T22:17:47.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: 22:17:46.154907+0000 mgr.y (mgr.14144) 189 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi076=foo.a;count:1 2021-03-15T22:17:47.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:17:47.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: 03-15T22:17:46.158558+0000 mon.a 2021-03-15T22:17:47.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: (mon.0) 422 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:47.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: audit 2021-03-15T22:17:47.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: 2021-03-15T22:17:46.598179+0000 2021-03-15T22:17:47.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: mon.b (mon.2) 20 : audit [INF] from='osd.7 [v2:172.21.15.125:6824/652003948,v1:172.21.15.125:6825/652003948]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-15T22:17:47.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: audit 2021-03-15T22:17:47.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: 2021-03-15T22:17:46.598654 2021-03-15T22:17:47.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: +0000 mon.a (mon.0) 423 : audit [INF] 2021-03-15T22:17:47.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2021-03-15T22:17:47.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:17:47.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: T22:17:46.939841+0000 mgr.y (mgr.14144 2021-03-15T22:17:47.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:47 smithi076 conmon[62156]: ) 190 : cluster [DBG] pgmap v152: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:48.286 INFO:journalctl@ceph.osd.7.smithi125.stdout:Mar 15 22:17:48 smithi125 conmon[70700]: debug 2021-03-15T22:17:48.168+0000 7f3825da4700 -1 osd.7 0 waiting for initial osdmap 2021-03-15T22:17:48.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:48 smithi125 conmon[56739]: audit 2021-03-15T22:17:47.164924+0000 2021-03-15T22:17:48.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:48 smithi125 conmon[56739]: mon.a (mon.0) 424 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-03-15T22:17:48.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:48 smithi125 conmon[56739]: cluster 2021-03-15T22:17:47.165014+0000 mon.a (mon.0) 425 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2021-03-15T22:17:48.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:48 smithi125 conmon[56739]: audit 2021-03-15T22:17:47.165347+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:48.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:48 smithi125 conmon[56739]: audit 2021-03-15T22:17:47.165526+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 [v2:172.21.15.125:6824/652003948,v1:172.21.15.125:6825/652003948]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:48.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:48 smithi125 conmon[56739]: audit 2021-03-15T22:17:47.165920+0000 mon.a (mon.0) 427 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[56514]: audit 2021-03-15T22:17:47.164924+0000 mon.a (mon.0) 424 2021-03-15T22:17:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[56514]: : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-03-15T22:17:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[56514]: cluster 2021-03-15T22:17:47.165014+0000 mon.a (mon.0) 425 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2021-03-15T22:17:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[56514]: audit 2021-03-15T22:17:47.165347+0000 mon.a (mon.0) 426 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[56514]: audit 2021-03-15T22:17:47.165526+0000 mon.b (mon.2) 21 : audit [INF] from='osd.7 [v2:172.21.15.125:6824/652003948,v1:172.21.15.125:6825/652003948]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[56514]: audit 2021-03-15T22:17:47.165920+0000 mon.a (mon.0) 427 : audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:48.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: audit 2021-03-15T22:17:47.164924+0000 mon.a ( 2021-03-15T22:17:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: mon.0) 424 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2021-03-15T22:17:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: cluster 2021-03-15T22:17:47.165014 2021-03-15T22:17:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: +0000 mon.a (mon.0) 425 : cluster [DBG] osdmap e50: 8 total, 7 up, 8 in 2021-03-15T22:17:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:17:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: :17:47.165347+0000 mon.a ( 2021-03-15T22:17:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: mon.0) 426 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:17:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: 03-15T22:17:47.165526+0000 mon.b (mon.2) 21 2021-03-15T22:17:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: : audit [INF] from='osd.7 [v2:172.21.15.125:6824/652003948,v1:172.21.15.125:6825/652003948]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: -03-15T22:17:47.165920+0000 2021-03-15T22:17:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: mon.a (mon.0) 427 : 2021-03-15T22:17:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:48 smithi076 conmon[62156]: audit [INF] from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:17:49.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:49 smithi125 conmon[56739]: audit 2021-03-15T22:17:48.166584+0000 mon.a (mon.0) 428 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:17:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:49 smithi125 conmon[56739]: cluster 2021-03-15T22:17:48.166720+0000 mon.a (mon.0) 429 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2021-03-15T22:17:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:49 smithi125 conmon[56739]: audit 2021-03-15T22:17:48.167336+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:49 smithi125 conmon[56739]: audit 2021-03-15T22:17:48.171233+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:49 smithi125 conmon[56739]: cluster 2021-03-15T22:17:48.940340+0000 mgr.y (mgr.14144) 191 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:49 smithi125 conmon[56739]: cluster 2021-03-15T22:17:49.167486+0000 mon.a (mon.0) 432 : cluster [INF] osd.7 [v2:172.21.15.125:6824/652003948,v1:172.21.15.125:6825/652003948] boot 2021-03-15T22:17:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:49 smithi125 conmon[56739]: cluster 2021-03-15T22:17:49.167543+0000 mon.a (mon.0) 433 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2021-03-15T22:17:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:49 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:17:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:49 smithi125 conmon[56739]: 17:49.167740+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:49.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: audit 2021-03-15T22:17:48.166584+0000 mon.a (mon.0) 428 : 2021-03-15T22:17:49.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:17:49.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: cluster 2021-03-15T22:17:48.166720+0000 mon.a (mon.0) 429 : cluster [DBG] osdmap e51: 8 total, 7 up, 8 in 2021-03-15T22:17:49.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: audit 2021-03-15T22:17 2021-03-15T22:17:49.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: :48.167336+0000 mon.a (mon.0) 430 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:49.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: audit 2021-03-15T22:17:49.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: 2021-03-15T22:17:48.171233+0000 mon.a (mon.0) 431 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:49.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:17:49.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: -03-15T22:17:48.940340+0000 mgr.y (mgr 2021-03-15T22:17:49.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: .14144) 191 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:49.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: cluster 2021-03-15T22:17:49.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: 2021-03-15T22:17:49.167486+0000 mon.a (mon 2021-03-15T22:17:49.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: .0) 432 : cluster [INF] osd.7 [v2:172.21.15.125:6824/652003948,v1:172.21.15.125:6825/652003948] boot 2021-03-15T22:17:49.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:17:49.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: -03-15T22:17:49.167543+0000 mon.a (mon.0 2021-03-15T22:17:49.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: ) 433 : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2021-03-15T22:17:49.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: audit 2021-03-15T22:17:49.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: 2021-03-15T22:17:49.167740+0000 mon.a ( 2021-03-15T22:17:49.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56514]: mon.0) 434 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:49.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: audit 2021-03-15T22:17:48.166584+0000 mon.a ( 2021-03-15T22:17:49.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: mon.0) 428 : audit [INF] from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]': finished 2021-03-15T22:17:49.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: cluster 2021-03-15T22:17:48.166720+0000 mon.a (mon.0) 429 : cluster 2021-03-15T22:17:49.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: [DBG] osdmap e51: 8 total, 7 up, 8 in 2021-03-15T22:17:49.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: audit 2021-03-15T22:17:48.167336+0000 2021-03-15T22:17:49.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: mon.a (mon.0) 430 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:49.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: audit 2021-03-15T22:17:48.171233+0000 mon.a 2021-03-15T22:17:49.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: (mon.0) 431 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:49.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:17:49.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: 03-15T22:17:48. 2021-03-15T22:17:49.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: 940340+0000 mgr.y (mgr.14144 2021-03-15T22:17:49.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: ) 191 : cluster [DBG] pgmap v155: 1 pgs: 1 active+clean; 0 B data, 37 MiB used, 626 GiB / 626 GiB avail 2021-03-15T22:17:49.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: 2021-03-15T22:17:49.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:17:49.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: 15T22:17:49.167486 2021-03-15T22:17:49.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: +0000 mon.a (mon.0) 432 : 2021-03-15T22:17:49.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: cluster [INF] osd.7 [v2:172.21.15.125:6824/652003948,v1:172.21.15.125:6825/652003948] boot 2021-03-15T22:17:49.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:17:49.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: 03-15T22:17:49.167543 2021-03-15T22:17:49.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: +0000 mon.a (mon.0) 433 2021-03-15T22:17:49.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: : cluster [DBG] osdmap e52: 8 total, 8 up, 8 in 2021-03-15T22:17:49.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:17:49.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: T22:17:49.167740+0000 mon.a (mon.0) 434 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2021-03-15T22:17:49.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[62156]: 2021-03-15T22:17:49.950 INFO:teuthology.orchestra.run.smithi125.stderr:pool 'iscsi' created 2021-03-15T22:17:50.178 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56844]: debug 2021-03-15T22:17:49.915+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:50.179 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:17:50.179 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:50.179 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:17:50.180 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:50.180 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:17:50.180 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:50.180 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:50.180 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:49 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:50.182 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: cluster 2021-03-15T22:17:47.645079+0000 osd.7 (osd. 2021-03-15T22:17:50.182 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 7) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:50.182 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: cluster 2021-03-15T22:17:47.645169+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:50.183 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: cluster 2021-03-15T22:17:49.279728+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:50.183 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: cluster 2021-03-15T22:17:49.279863+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:50.183 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: audit 2021-03-15T22:17:49.664930+0000 mon.b (mon.2) 22 : audit [INF] from='client.? 172.21.15.125:0/3768394254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-15T22:17:50.183 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:17:50.183 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 03-15T22:17:49.665622+0000 mon.a (mon.0) 435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-15T22:17:50.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:49.901159+0000 mgr.y (mgr.14144) 192 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-15T22:17:50.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 2021-03-15T22:17:50.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:17:50.184 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 15T22:17:49.911461+0000 mon.a (mon.0) 436 : audit [INF] 2021-03-15T22:17:50.185 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.185 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: cephadm 2021-03-15 2021-03-15T22:17:50.185 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: T22:17:49.912017+0000 mgr.y (mgr 2021-03-15T22:17:50.185 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: .14144) 193 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:50.185 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: audit 2021-03-15T22:17:50.186 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 2021-03-15T22 2021-03-15T22:17:50.186 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: :17:49.915271+0000 mon.a (mon.0 2021-03-15T22:17:50.187 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: ) 437 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.187 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:49.916021+0000 mgr.y (mgr.14144) 194 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:50.187 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:17:50.187 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:50.187 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:17:50.188 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:50.188 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:17:50.188 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:50.188 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:50.189 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:50.189 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: audit 2021-03-15T22:17:49.919956+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.189 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:49.928952+0000 mgr.y (mgr.14144) 195 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi076=foo.a;count:1 2021-03-15T22:17:50.189 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: audit 2021-03-15T22:17:50.189 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 2021-03-15T22:17:49.932036+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.190 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: audit 2021-03-15T22:17:49.932512+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-03-15T22:17:50.190 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: audit 2021-03-15T22:17:50.190 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 2021-03-15T22:17: 2021-03-15T22:17:50.190 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 49.935853+0000 mon.a (mon.0) 441 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-03-15T22:17:50.191 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:17:50.191 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 15T22:17:49.936462+0000 2021-03-15T22:17:50.191 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: mon.a (mon.0) 442 : 2021-03-15T22:17:50.191 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:50.191 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: cephadm 2021-03-15 2021-03-15T22:17:50.192 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: T22:17:49.936969+0000 mgr.y (mgr.14144) 196 : cephadm [INF] Deploying daemon rgw.foo.a on smithi076 2021-03-15T22:17:50.192 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 2021-03-15T22:17:50.192 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: audit 2021-03-15T22:17:50.192 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 2021-03-15T22:17:49 2021-03-15T22:17:50.193 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: cluster 2021-03-15T22:17:47 2021-03-15T22:17:50.193 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: .645079+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:50.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: cluster 2021-03-15T22:17:50.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 2021-03-15T22:17 2021-03-15T22:17:50.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: :47.645169+0000 osd.7 (osd.7) 2021-03-15T22:17:50.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:50.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: cluster 2021-03-15T22:17:50.195 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:17:50.195 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: T22:17:49.279728+0000 osd.0 ( 2021-03-15T22:17:50.195 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:50.195 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: cluster 2021-03-15T22:17:50.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 2021-03-15T22:17: 2021-03-15T22:17:50.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 49.279863+0000 osd.0 (osd.0) 6 : cluster 2021-03-15T22:17:50.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: [DBG] purged_snaps scrub ok 2021-03-15T22:17:50.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: audit 2021-03-15T22:17:50.197 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 2021-03-15T22:17:49. 2021-03-15T22:17:50.197 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 664930+0000 mon.b (mon.2) 22 : audit 2021-03-15T22:17:50.197 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.125:0/3768394254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-15T22:17:50.198 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:17:50.198 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 03-15T22:17:49.665622+0000 2021-03-15T22:17:50.198 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: mon.a (mon.0) 435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-15T22:17:50.198 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 2021-03-15T22:17:50.199 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: cephadm 2021-03 2021-03-15T22:17:50.199 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: -15T22:17:49.901159+0000 2021-03-15T22:17:50.199 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: mgr.y (mgr.14144) 192 : cephadm [INF] 2021-03-15T22:17:50.199 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: Detected new or changed devices on smithi125 2021-03-15T22:17:50.199 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:17:50.200 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 17:49.911461+0000 mon.a (mon.0) 2021-03-15T22:17:50.200 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 436 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.200 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:50.200 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:17:50.200 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 17:49.912017+0000 mgr.y (mgr.14144) 2021-03-15T22:17:50.201 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 193 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:50.201 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: audit 2021-03-15T22:17:49. 2021-03-15T22:17:50.201 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 915271+0000 mon.a (mon.0) 437 : audit [INF] 2021-03-15T22:17:50.201 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.201 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: cephadm 2021-03-15T 2021-03-15T22:17:50.202 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 22:17:49.916021+0000 mgr.y (mgr. 2021-03-15T22:17:50.202 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 14144) 194 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:50.202 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:17:50.202 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:50.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:17:50.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:50.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:17:50.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:50.203 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:50.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:50.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: audit 2021-03-15T22:17:49.919956+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: cephadm 2021-03-15 2021-03-15T22:17:50.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: T22:17:49.928952+0000 mgr.y (mgr 2021-03-15T22:17:50.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: .14144) 195 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi076=foo.a;count:1 2021-03-15T22:17:50.205 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: audit 2021-03-15T22:17:50.205 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:17:50.205 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: :17:49.932036+0000 mon.a (mon 2021-03-15T22:17:50.205 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: .0) 439 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.206 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:50.206 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: -03-15T22:17:49.932512+0000 2021-03-15T22:17:50.206 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: mon.a (mon.0) 440 : audit [INF] 2021-03-15T22:17:50.206 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-03-15T22:17:50.206 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: audit 2021-03-15T22:17:49 2021-03-15T22:17:50.207 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: .935853+0000 mon.a (mon.0) 441 : audit 2021-03-15T22:17:50.208 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-03-15T22:17:50.209 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:17:50.209 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 22:17:49.936462+0000 mon.a (mon. 2021-03-15T22:17:50.209 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 0) 442 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:50.210 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: cephadm 2021-03-15T22: 2021-03-15T22:17:50.210 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 17:49.936969+0000 mgr.y (mgr.14144) 196 2021-03-15T22:17:50.210 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: : cephadm [INF] Deploying daemon rgw.foo.a on smithi076 2021-03-15T22:17:50.211 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:17:50.211 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 22:17:49.945797+0000 mon.a (mon. 2021-03-15T22:17:50.212 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 0) 443 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-03-15T22:17:50.212 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:17:50.212 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: -15T22:17:49.945881+0000 2021-03-15T22:17:50.213 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: mon.a (mon.0) 444 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2021-03-15T22:17:50.213 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[62156]: 2021-03-15T22:17:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: cluster 2021-03-15T22:17:47.645079+0000 osd.7 (osd.7) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: cluster 2021-03-15T22:17:47.645169+0000 osd.7 (osd.7) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: cluster 2021-03-15T22:17:49.279728+0000 osd.0 (osd.0) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:50.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: cluster 2021-03-15T22:17:49.279863+0000 osd.0 (osd.0) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:50.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: audit 2021-03-15T22:17:49.664930+0000 mon.b (mon.2) 22 : audit [INF] from='client.? 172.21.15.125:0/3768394254' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-15T22:17:50.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: audit 2021-03-15T22:17:49.665622+0000 mon.a (mon.0) 435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2021-03-15T22:17:50.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:49.901159+0000 mgr.y (mgr.14144) 192 : cephadm [INF] Detected new or changed devices on smithi125 2021-03-15T22:17:50.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: audit 2021-03-15T22:17:49.911461+0000 mon.a (mon.0) 436 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:49.912017+0000 mgr.y (mgr.14144) 193 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:50.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: audit 2021-03-15T22:17:49.915271+0000 mon.a (mon.0) 437 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:49.916021+0000 mgr.y (mgr.14144) 194 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:50.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:17:50.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:50.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:17:50.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:50.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:17:50.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:50.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:50.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:50.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: audit 2021-03-15T22:17:49.919956+0000 mon.a (mon.0) 438 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: cephadm 2021-03-15T22:17: 2021-03-15T22:17:50.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: 49.928952+0000 mgr.y (mgr.14144) 195 : cephadm [INF] Saving service rgw.foo.a spec with placement smithi076=foo.a;count:1 2021-03-15T22:17:50.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: audit 2021-03-15T22:17:49.932036+0000 mon.a (mon.0) 439 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:50.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: audit 2021-03-15T22:17:49.932512+0000 mon.a (mon.0) 440 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2021-03-15T22:17:50.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: audit 2021-03-15T22:17:50.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: 2021-03-15T22:17:49.935853+0000 mon.a (mon.0) 441 : audit [INF] 2021-03-15T22:17:50.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2021-03-15T22:17:50.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: audit 2021-03-15T22:17:50.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: 2021-03-15T22:17:49.936462+0000 2021-03-15T22:17:50.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: mon.a (mon.0) 442 : audit [DBG] 2021-03-15T22:17:50.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:50.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:50.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: 2021-03-15T22:17:49.936969+0000 2021-03-15T22:17:50.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: mgr.y (mgr.14144) 196 : cephadm [INF] 2021-03-15T22:17:50.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: Deploying daemon rgw.foo.a on smithi076 2021-03-15T22:17:50.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:17:50.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: T22:17:49.945797+0000 mon.a (mon.0) 443 : audit [INF] 2021-03-15T22:17:50.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-03-15T22:17:50.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: cluster 2021-03-15T22:17 2021-03-15T22:17:50.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: :49.945881+0000 mon.a (mon.0) 2021-03-15T22:17:50.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:50 smithi125 conmon[56739]: 444 : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2021-03-15T22:17:50.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: .945797+0000 mon.a (mon.0) 443 : audit [INF] 2021-03-15T22:17:50.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "iscsi", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2021-03-15T22:17:50.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:17:50.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: 22:17:49.945881+0000 mon.a (mon.0) 444 2021-03-15T22:17:50.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:50 smithi076 conmon[56514]: : cluster [DBG] osdmap e53: 8 total, 8 up, 8 in 2021-03-15T22:17:50.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool application enable iscsi rbd 2021-03-15T22:17:51.517 INFO:journalctl@ceph.rgw.foo.a.smithi076.stdout:Mar 15 22:17:51 smithi076 systemd[1]: Starting Ceph rgw.foo.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:17:51.994 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:51 smithi076 conmon[56514]: cluster 2021-03-15T22:17:50.940747+0000 2021-03-15T22:17:51.995 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:51 smithi076 conmon[56514]: mgr.y (mgr.14144) 197 : cluster [DBG] pgmap v158: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:51.995 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:51 smithi076 conmon[56514]: cluster 2021-03-15T22:17:50.948729+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2021-03-15T22:17:51.996 INFO:journalctl@ceph.rgw.foo.a.smithi076.stdout:Mar 15 22:17:51 smithi076 podman[77382]: 2021-03-15 22:17:51.710357471 +0000 UTC m=+0.149366099 container create faa4098a6d54d2961b12e1f5d661f3ae4579a5b488ceaf276f0e6ca83edf078b (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-rgw.foo.a, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, io.buildah.version=1.18.0, maintainer=Dimitri Savineau ) 2021-03-15T22:17:51.996 INFO:journalctl@ceph.rgw.foo.a.smithi076.stdout:Mar 15 22:17:51 smithi076 podman[77382]: 2021-03-15 22:17:51.960337319 +0000 UTC m=+0.399345931 container init faa4098a6d54d2961b12e1f5d661f3ae4579a5b488ceaf276f0e6ca83edf078b (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-rgw.foo.a, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_CLEAN=True, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.name=CentOS Base Image) 2021-03-15T22:17:51.996 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:51 smithi076 conmon[62156]: cluster 2021-03-15T22:17:50.940747+0000 mgr.y (mgr.14144) 197 : cluster [DBG] pgmap v158: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:51.997 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:51 smithi076 conmon[62156]: cluster 2021-03-15T22:17:50.948729+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2021-03-15T22:17:52.068 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:51 smithi125 conmon[56739]: cluster 2021-03-15T22:17:50.940747+0000 mgr.y (mgr.14144) 197 : cluster [DBG] pgmap v158: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:52.068 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:51 smithi125 conmon[56739]: cluster 2021-03-15T22:17:50.948729+0000 mon.a (mon.0) 445 : cluster [DBG] osdmap e54: 8 total, 8 up, 8 in 2021-03-15T22:17:52.266 INFO:journalctl@ceph.rgw.foo.a.smithi076.stdout:Mar 15 22:17:51 smithi076 podman[77382]: 2021-03-15 22:17:51.993633676 +0000 UTC m=+0.432642309 container start faa4098a6d54d2961b12e1f5d661f3ae4579a5b488ceaf276f0e6ca83edf078b (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-rgw.foo.a, org.label-schema.schema-version=1.0, maintainer=Dimitri Savineau , RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2021-03-15T22:17:52.267 INFO:journalctl@ceph.rgw.foo.a.smithi076.stdout:Mar 15 22:17:51 smithi076 bash[77307]: faa4098a6d54d2961b12e1f5d661f3ae4579a5b488ceaf276f0e6ca83edf078b 2021-03-15T22:17:52.267 INFO:journalctl@ceph.rgw.foo.a.smithi076.stdout:Mar 15 22:17:51 smithi076 systemd[1]: Started Ceph rgw.foo.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:17:52.969 INFO:teuthology.orchestra.run.smithi125.stderr:enabled application 'rbd' on pool 'iscsi' 2021-03-15T22:17:53.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[56514]: cluster 2021-03-15T22:17:51 2021-03-15T22:17:53.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[56514]: .949289+0000 mon.a (mon.0) 446 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2021-03-15T22:17:53.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[56514]: audit 2021-03-15T22:17:52.263959+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:53.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[56514]: audit 2021-03-15T22:17:52.264568+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:53.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[56514]: audit 2021-03-15T22:17:52.390455+0000 mon.b (mon.2) 23 : audit [INF] from='client.? 172.21.15.125:0/2476573332' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-15T22:17:53.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[56514]: audit 2021-03-15T22:17:52.390931+0000 mon.a (mon.0) 449 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-15T22:17:53.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: cluster 2021-03-15T22:17:51.949289+0000 2021-03-15T22:17:53.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: mon.a (mon.0) 446 : cluster [DBG] osdmap e55: 8 total, 8 up, 8 in 2021-03-15T22:17:53.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:17:53.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: T22:17:52.263959+0000 mon.a ( 2021-03-15T22:17:53.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: mon.0) 447 : audit 2021-03-15T22:17:53.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:53.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: audit 2021-03-15T22:17:53.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:17:53.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: 17:52.264568+0000 mon.a 2021-03-15T22:17:53.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: (mon.0) 448 : audit 2021-03-15T22:17:53.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:53.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:53.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: -03-15T22:17:52.390455+0000 mon.b (mon.2) 23 : audit [INF] from='client.? 172.21.15.125:0/2476573332' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-15T22:17:53.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:52 smithi076 conmon[62156]: audit 2021-03-15T22:17:52.390931+0000 mon.a (mon.0) 449 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-15T22:17:53.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:52 smithi125 conmon[56739]: cluster 2021-03-15T22:17:51.949289+0000 mon.a (mon.0) 446 : cluster 2021-03-15T22:17:53.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:52 smithi125 conmon[56739]: [DBG] osdmap e55: 8 total, 8 up, 8 in 2021-03-15T22:17:53.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:52 smithi125 conmon[56739]: audit 2021-03-15T22:17:52.263959+0000 mon.a (mon.0) 447 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:53.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:52 smithi125 conmon[56739]: audit 2021-03-15T22:17:52.264568+0000 mon.a (mon.0) 448 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:53.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:52 smithi125 conmon[56739]: audit 2021-03-15T22:17:52.390455+0000 mon.b (mon.2) 23 : audit [INF] from='client.? 172.21.15.125:0/2476573332' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-15T22:17:53.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:52 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:17:53.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:52 smithi125 conmon[56739]: 03-15T22:17:52.390931+0000 mon.a (mon.0) 449 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]: dispatch 2021-03-15T22:17:53.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch apply iscsi iscsi user password --placement '1;smithi125=iscsi.a' 2021-03-15T22:17:53.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: cluster 2021-03-15T22:17:53.971 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: 2021-03-15T22:17:53.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: 2021 2021-03-15T22:17:53.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: - 2021-03-15T22:17:53.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: 03 2021-03-15T22:17:53.972 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: - 2021-03-15T22:17:53.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: 15 2021-03-15T22:17:53.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: T 2021-03-15T22:17:53.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: 22 2021-03-15T22:17:53.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: :17:52.941064+0000 2021-03-15T22:17:53.973 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: mgr.y (mgr.14144) 198 : cluster [DBG] pgmap v161: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:53.974 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: audit 2021-03-15T22:17:53.974 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: debug 2021-03-15T22:17:53.523+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:53.974 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:17:53.975 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:53.975 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:17:53.975 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:53.975 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:17:53.975 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:53.976 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:53.976 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:53.976 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: debug 2021-03-15T22:17:53.524+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:53.976 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:17:53.977 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:53.977 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:17:53.977 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:53.977 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:17:53.977 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:53.978 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:53.978 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:53.978 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: debug 2021-03-15T22:17:53.529+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:53.978 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:17:53.979 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:53.979 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:17:53.979 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:53.979 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:17:53.979 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:53.980 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:53.980 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:53.980 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: debug 2021-03-15T22:17:53.529+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:53.980 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:17:53.980 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:53.981 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:17:53.981 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:53.981 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:17:53.981 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:53.981 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:53.982 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:54.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: 2021-03-15T22:17:52.961237+0000 2021-03-15T22:17:54.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: mon.a (mon.0) 450 : audit [INF] 2021-03-15T22:17:54.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2021-03-15T22:17:54.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: cluster 2021-03-15T22:17:54.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: 2021-03-15T22:17:52.961309+0000 2021-03-15T22:17:54.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: mon.a (mon.0) 451 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2021-03-15T22:17:54.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: 2021-03-15T22:17:54.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: audit 2021-03-15T22:17:52. 2021-03-15T22:17:54.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: 963348+0000 mon.a (mon.0) 452 : audit 2021-03-15T22:17:54.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-03-15T22:17:54.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: audit 2021-03-15T22:17:53 2021-03-15T22:17:54.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: .520700+0000 mon.a (mon.0) 453 : audit 2021-03-15T22:17:54.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:54.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: audit 2021-03-15T22:17 2021-03-15T22:17:54.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: :53.523698+0000 mon.a (mon.0) 454 : 2021-03-15T22:17:54.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:54.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: audit 2021-03-15T22: 2021-03-15T22:17:54.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: 17:53.525390+0000 mon.a (mon.0) 455 2021-03-15T22:17:54.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:54.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: audit 2021-03-15T22 2021-03-15T22:17:54.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: :17:53.529330+0000 mon.a (mon.0 2021-03-15T22:17:54.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[56514]: ) 456 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:54.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: cluster 2021-03-15T22:17:54.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: 2021-03-15T22:17:54.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: 2021 2021-03-15T22:17:54.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: - 2021-03-15T22:17:54.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: 03 2021-03-15T22:17:54.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: - 2021-03-15T22:17:54.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: 15 2021-03-15T22:17:54.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: T22:17:52.941064+0000 mgr.y (mgr.14144 2021-03-15T22:17:54.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: ) 198 : cluster [DBG] pgmap v161: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:54.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:54.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: -03-15T22:17:52.961237+0000 mon.a (mon.0) 450 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2021-03-15T22:17:54.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:17:54.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: -15T22:17:52.961309+0000 mon.a (mon. 2021-03-15T22:17:54.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: 0) 451 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2021-03-15T22:17:54.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:17:54.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: 15T22:17:52.963348+0000 mon.a (mon. 2021-03-15T22:17:54.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: 0) 452 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-03-15T22:17:54.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:17:54.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: 15T22:17:53.520700+0000 mon.a (mon. 2021-03-15T22:17:54.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: 0) 453 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:54.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:17:54.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: -15T22:17:53.523698+0000 mon.a ( 2021-03-15T22:17:54.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: mon.0) 454 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:54.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: audit 2021 2021-03-15T22:17:54.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: -03-15T22:17:53.525390+0000 mon.a 2021-03-15T22:17:54.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: (mon.0) 455 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:54.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:17:54.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: 22:17:53.529330+0000 mon.a (mon.0) 2021-03-15T22:17:54.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:53 smithi076 conmon[62156]: 456 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:54.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: cluster 2021-03-15T22:17:52.941064+0000 2021-03-15T22:17:54.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: mgr.y (mgr.14144) 198 : cluster [DBG] pgmap v161: 4 pgs: 1 creating+peering, 1 active+clean, 2 unknown; 0 B data, 42 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:54.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: audit 2021-03-15T22:17:52.961237+0000 mon.a (mon.0) 450 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "iscsi", "app": "rbd"}]': finished 2021-03-15T22:17:54.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: cluster 2021-03-15T22:17:52.961309+0000 mon.a (mon.0) 451 : cluster [DBG] osdmap e56: 8 total, 8 up, 8 in 2021-03-15T22:17:54.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: audit 2021-03-15T22:17:52.963348+0000 mon.a (mon.0) 452 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2021-03-15T22:17:54.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: audit 2021-03-15T22:17:53.520700+0000 mon.a (mon.0) 453 : audit [INF] 2021-03-15T22:17:54.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:54.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: audit 2021-03-15T22:17:53.523698+0000 mon.a (mon.0) 454 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:54.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: 2021-03-15T22:17:54.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:17:54.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: 15T22:17:53.525390+0000 mon.a (mon.0) 455 2021-03-15T22:17:54.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:54.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:17:54.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: 17:53.529330+0000 mon.a (mon.0) 456 2021-03-15T22:17:54.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:53 smithi125 conmon[56739]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:55.101 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:53.521248+0000 mgr.y (mgr.14144) 199 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:55.101 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:53.524456+0000 mgr.y (mgr.14144) 200 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:55.101 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:17:55.102 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.102 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:17:55.102 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.102 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:17:55.103 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.103 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.103 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:55.103 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:53.524860+0000 mgr.y (mgr.14144) 201 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:55.104 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:17:55.104 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.104 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:17:55.104 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.105 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:17:55.106 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.106 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.107 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:55.107 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:53.526521+0000 mgr.y (mgr.14144) 202 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:55.108 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:53.530038+0000 mgr.y (mgr.14144) 203 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:55.108 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:17:55.108 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.109 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:17:55.109 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.109 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:17:55.110 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.110 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.110 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:55.111 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:53.530433+0000 mgr.y (mgr.14144) 204 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:55.111 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:17:55.111 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.111 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:17:55.112 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.112 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:17:55.112 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.113 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.113 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:55.113 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: audit 2021-03-15T22:17:53.962147+0000 mon.a (mon.0) 457 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-03-15T22:17:55.113 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: cluster 2021-03-15T22:17:53.962300+0000 mon.a (mon.0) 458 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2021-03-15T22:17:55.114 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: cluster 2021-03-15T22:17 2021-03-15T22:17:55.114 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:54 smithi125 conmon[56739]: :54.941442+0000 mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v164: 36 pgs: 1 creating+activating, 9 creating+peering, 8 active+clean, 18 unknown; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:55.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:53.521248+0000 mgr.y ( 2021-03-15T22:17:55.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: mgr.14144) 199 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:55.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:53.524456+0000 mgr.y (mgr.14144) 200 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:55.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:17:55.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:17:55.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:17:55.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:55.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:53.524860+0000 mgr.y (mgr.14144) 201 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:55.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:17:55.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:17:55.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:17:55.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:55.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: cephadm 2021 2021-03-15T22:17:55.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: -03-15T22:17:53.526521+0000 mgr.y (mgr.14144) 202 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:55.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:53. 2021-03-15T22:17:55.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: 530038+0000 mgr.y (mgr.14144) 203 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:55.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:17:55.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:17:55.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:17:55.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:53.521248+0000 2021-03-15T22:17:55.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: mgr.y (mgr.14144) 199 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:55.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:55.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: 2021-03-15T22:17:53.524456+0000 mgr.y (mgr.14144) 200 2021-03-15T22:17:55.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:55.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:17:55.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:17:55.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:17:55.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:55.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:53.524860+0000 mgr.y (mgr.14144) 201 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:55.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:17:55.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:17:55.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:17:55.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:55.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:53.526521+0000 mgr.y (mgr.14144) 202 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:55.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:17:55.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: -03-15T22:17:53.530038+0000 2021-03-15T22:17:55.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: mgr.y (mgr.14144) 203 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:17:55.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:17:55.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:17:55.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.289 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:53.530433+0000 mgr.y (mgr.14144) 204 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:17:55.290 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:17:55.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:17:55.291 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:55.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: audit 2021-03-15T22:17:53.962147+0000 mon.a (mon.0) 457 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-03-15T22:17:55.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: cluster 2021-03-15T22:17:53.962300 2021-03-15T22:17:55.292 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: +0000 mon.a (mon.0) 458 : cluster [DBG] 2021-03-15T22:17:55.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: osdmap e57: 8 total, 8 up, 8 in 2021-03-15T22:17:55.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: cluster 2021-03-15T22:17 2021-03-15T22:17:55.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: :54.941442+0000 mgr.y (mgr.14144) 2021-03-15T22:17:55.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[56514]: 205 : cluster [DBG] pgmap v164: 36 pgs: 1 creating+activating, 9 creating+peering, 8 active+clean, 18 unknown; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:55.294 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:17:55.294 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.294 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.294 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:17:55.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: 2021-03-15T22:17:55.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:53.530433+0000 mgr.y (mgr.14144) 204 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:55.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:17:55.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:17:55.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:17:55.296 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:55.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: 2021-03-15T22:17:55.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: audit 2021-03-15T22:17:53.962147+0000 mon.a (mon.0) 457 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2021-03-15T22:17:55.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:17:55.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: 22:17:53.962300+0000 mon.a (mon.0 2021-03-15T22:17:55.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: ) 458 : cluster [DBG] osdmap e57: 8 total, 8 up, 8 in 2021-03-15T22:17:55.298 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:17:55.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: -03-15T22:17:54.941442+0000 2021-03-15T22:17:55.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:54 smithi076 conmon[62156]: mgr.y (mgr.14144) 205 : cluster [DBG] pgmap v164: 36 pgs: 1 creating+activating, 9 creating+peering, 8 active+clean, 18 unknown; 0 B data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:55.460 INFO:teuthology.orchestra.run.smithi125.stdout:Scheduled iscsi.iscsi update... 2021-03-15T22:17:55.767 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: debug 2021-03-15T22:17:55.463+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:55.767 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:17:55.767 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.768 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:17:55.768 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.768 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:17:55.768 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.769 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.769 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:55.769 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: debug 2021-03-15T22:17:55.464+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:55.769 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:17:55.770 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:55.770 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:17:55.770 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:55.770 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:17:55.770 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:55.771 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:55.771 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:56.116 DEBUG:teuthology.orchestra.run.smithi125:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@iscsi.iscsi.a.service 2021-03-15T22:17:56.119 INFO:tasks.cephadm:Adding prometheus.a on smithi125 2021-03-15T22:17:56.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch apply prometheus '1;smithi125=a' 2021-03-15T22:17:56.147 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:-- Logs begin at Mon 2021-03-15 21:44:18 UTC. -- 2021-03-15T22:17:56.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: cluster 2021-03-15T22:17:56.448 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: 2021-03-15T22:17:54 2021-03-15T22:17:56.449 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: .977079+0000 mon.a (mon.0) 459 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2021-03-15T22:17:56.449 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: audit 2021-03-15T22:17:54.982118+0000 mon.a (mon.0) 2021-03-15T22:17:56.450 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: 460 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-03-15T22:17:56.450 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: audit 2021-03-15T22:17: 2021-03-15T22:17:56.451 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: 55.453513+0000 mgr.y (mgr.14144) 206 : audit [DBG] from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi125=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:56.451 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:17:56.452 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: -03-15T22:17:55.454229+0000 mgr.y (mgr 2021-03-15T22:17:56.452 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: .14144) 207 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi125=iscsi.a;count:1 2021-03-15T22:17:56.452 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:17:56.453 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: 22:17:55.458047+0000 mon.a (mon.0) 461 : 2021-03-15T22:17:56.453 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:56.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:17:56.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: 22:17:55.458767+0000 mon.a (mon.0) 2021-03-15T22:17:56.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: 462 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:56.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: cephadm 2021-03-15 2021-03-15T22:17:56.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: T22:17:55.460265+0000 mgr.y (mgr.14144 2021-03-15T22:17:56.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: ) 208 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:56.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:17:56.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: -15T22:17:55.463431+0000 mon.a ( 2021-03-15T22:17:56.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: mon.0) 463 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:56.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:56.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: 2021-03-15T22:17:55.464558+0000 2021-03-15T22:17:56.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: mgr.y (mgr.14144) 209 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:56.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:17:56.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:56.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:17:56.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:56.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:17:56.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:56.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:56.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:56.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: 2021-03-15T22:17:56.461 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: cluster 2021-03-15T22:17:56.461 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 2021-03-15T22:17:56.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 2021 2021-03-15T22:17:56.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: - 2021-03-15T22:17:56.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 03 2021-03-15T22:17:56.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: -15T22:17:54.977079+0000 mon.a (mon.0 2021-03-15T22:17:56.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: ) 459 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2021-03-15T22:17:56.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: audit 2021 2021-03-15T22:17:56.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: -03-15T22:17:54.982118+0000 mon.a (mon.0) 460 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-03-15T22:17:56.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:17:56.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 15T22:17:55.453513+0000 mgr.y (mgr.14144) 206 : 2021-03-15T22:17:56.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: audit [DBG] from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi125=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:56.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:17: 2021-03-15T22:17:56.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 55.454229+0000 mgr.y (mgr.14144) 207 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi125=iscsi.a;count:1 2021-03-15T22:17:56.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: audit 2021-03-15T22:17:56.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 2021-03-15T22:17:56.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 2021 2021-03-15T22:17:56.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: - 2021-03-15T22:17:56.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 03 2021-03-15T22:17:56.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: - 2021-03-15T22:17:56.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 15 2021-03-15T22:17:56.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: T 2021-03-15T22:17:56.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 22 2021-03-15T22:17:56.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: : 2021-03-15T22:17:56.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 17 2021-03-15T22:17:56.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: : 2021-03-15T22:17:56.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 55 2021-03-15T22:17:56.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: . 2021-03-15T22:17:56.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 458047 2021-03-15T22:17:56.467 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: +0000 2021-03-15T22:17:56.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 2021-03-15T22:17:56.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: mon.a 2021-03-15T22:17:56.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: ( 2021-03-15T22:17:56.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: mon 2021-03-15T22:17:56.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: . 2021-03-15T22:17:56.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 0 2021-03-15T22:17:56.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: ) 2021-03-15T22:17:56.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 461 2021-03-15T22:17:56.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: : 2021-03-15T22:17:56.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: audit 2021-03-15T22:17:56.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 2021-03-15T22:17:56.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: [INF] 2021-03-15T22:17:56.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 2021-03-15T22:17:56.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:56.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:17:56.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: -15T22:17:55.458767+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:56.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:56.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 2021-03-15T22:17:56.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 2021 2021-03-15T22:17:56.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: - 2021-03-15T22:17:56.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 03 2021-03-15T22:17:56.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: - 2021-03-15T22:17:56.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 15T22:17:55. 2021-03-15T22:17:56.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 460265+0000 mgr.y (mgr.14144) 208 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:56.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: audit 2021-03-15T22:17:56.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 2021-03-15T22:17:56.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 2021 2021-03-15T22:17:56.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: - 2021-03-15T22:17:56.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 03 2021-03-15T22:17:56.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: -15T22:17:55.463431+0000 mon.a 2021-03-15T22:17:56.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: (mon.0) 463 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:56.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:17:56.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:55.465376+0000 mgr.y (mgr.14144) 210 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:56.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:17:56.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:56.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:17:56.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:56.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:17:56.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:56.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:56.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:56.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:17:55.465938+0000 mgr.y (mgr.14144) 211 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2021-03-15T22:17:56.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: audit 2021-03-15T22:17:55.471962+0000 mon.a (mon 2021-03-15T22:17:56.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: .0) 464 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-03-15T22:17:56.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: audit 2021-03-15T22:17:56.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: 2021-03-15T22:17:55.475637+0000 mon.a (mon.0) 465 : audit [INF] 2021-03-15T22:17:56.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-03-15T22:17:56.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: cluster 2021-03-15T22:17:55.480376+0000 2021-03-15T22:17:56.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:56.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:17:56.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: -03-15T22:17:55.480479+0000 osd.4 2021-03-15T22:17:56.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:56.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:17:56.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: 03-15T22:17:55.487171+0000 mon.a ( 2021-03-15T22:17:56.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: mon.0) 466 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:56.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: cephadm 2021-03 2021-03-15T22:17:56.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: -15T22:17:55.487950+0000 mgr.y ( 2021-03-15T22:17:56.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[62156]: mgr.14144) 212 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi125 2021-03-15T22:17:56.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: -15T22:17:55.464558+0000 mgr.y (mgr.14144) 209 : cephadm [ERR] 2021-03-15T22:17:56.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:56.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:17:56.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:56.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:17:56.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:56.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:17:56.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:56.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:56.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:56.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:55.465376+0000 mgr.y (mgr.14144) 210 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:56.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:17:56.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:56.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:17:56.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:56.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:17:56.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:56.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:56.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:56.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:55. 2021-03-15T22:17:56.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: 465938+0000 mgr.y (mgr.14144) 211 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2021-03-15T22:17:56.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: audit 2021-03-15T22:17:55.471962+0000 mon.a (mon.0) 464 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-03-15T22:17:56.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: audit 2021-03-15T22:17 2021-03-15T22:17:56.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: :55.475637+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-03-15T22:17:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: cluster 2021-03-15T22:17:55.480376+0000 osd.4 (osd.4 2021-03-15T22:17:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: ) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: cluster 2021-03-15T22:17:55.480479+0000 2021-03-15T22:17:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:56.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: audit 2021-03-15T22:17:55.487171 2021-03-15T22:17:56.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: +0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:56.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:55.487950+0000 mgr.y (mgr.14144 2021-03-15T22:17:56.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:55 smithi076 conmon[56514]: ) 212 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi125 2021-03-15T22:17:56.489 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: cluster 2021-03-15T22:17:54.977079+0000 mon.a 2021-03-15T22:17:56.489 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: (mon.0) 459 : cluster [DBG] osdmap e58: 8 total, 8 up, 8 in 2021-03-15T22:17:56.489 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: audit 2021-03-15T22:17:54.982118+0000 mon.a (mon.0) 460 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2021-03-15T22:17:56.489 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: audit 2021-03-15T22:17:55.453513+0000 mgr.y (mgr.14144) 206 : audit [DBG] from='client.34328 -' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "iscsi", "api_user": "user", "api_password": "password", "placement": "1;smithi125=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:56.490 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:55.454229+0000 mgr.y (mgr.14144) 207 : cephadm [INF] Saving service iscsi.iscsi spec with placement smithi125=iscsi.a;count:1 2021-03-15T22:17:56.490 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: audit 2021-03-15T22:17:55.458047+0000 mon.a (mon.0) 461 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:56.490 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: audit 2021-03-15T22:17:55.458767+0000 mon.a (mon.0) 462 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:56.490 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:55.460265+0000 mgr.y (mgr.14144) 208 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:17:56.491 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: audit 2021-03-15T22:17:55.463431+0000 mon.a (mon.0) 463 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:56.491 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:55.464558+0000 mgr.y (mgr.14144) 209 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:17:56.491 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:17:56.491 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:56.492 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:17:56.492 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:56.492 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:17:56.492 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:56.492 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:56.493 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:17:56.493 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:55.465376+0000 mgr.y (mgr.14144) 210 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:17:56.493 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:17:56.493 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:17:56.493 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:17:56.494 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:17:56.494 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:17:56.494 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:17:56.494 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:17:56.495 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:17:56.495 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:55.465938+0000 mgr.y (mgr.14144) 211 : cephadm [INF] Checking pool "iscsi" exists for service iscsi.iscsi 2021-03-15T22:17:56.495 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: audit 2021-03-15T22:17:55.471962+0000 mon.a (mon.0) 464 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2021-03-15T22:17:56.495 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: audit 2021-03-15T22:17:55.475637+0000 mon.a (mon.0) 465 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2021-03-15T22:17:56.496 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: cluster 2021-03-15T22:17:55.480376+0000 osd.4 (osd.4) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:17:56.496 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: cluster 2021-03-15T22:17:55.480479+0000 osd.4 (osd.4) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:17:56.496 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: audit 2021-03-15T22:17:55.487171+0000 mon.a (mon.0) 466 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2021-03-15T22:17:56.496 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:55.487950+0000 mgr.y (mgr.14144) 212 : cephadm [INF] Deploying daemon iscsi.iscsi.a on smithi125 2021-03-15T22:17:57.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[62156]: audit 2021-03-15T22:17:57.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[62156]: 2021-03-15T22:17:55.981746+0000 2021-03-15T22:17:57.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[62156]: mon.a (mon.0) 467 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2021-03-15T22:17:57.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[62156]: cluster 2021-03-15T22:17:57.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[62156]: 2021-03-15T22:17:55.981920+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2021-03-15T22:17:57.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:17:57.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[62156]: 15T22:17:56.480503+0000 mon.a (mon.0) 469 : cluster [DBG] 2021-03-15T22:17:57.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[62156]: mgrmap e15: y(active, since 3m), standbys: x 2021-03-15T22:17:57.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[62156]: cluster 2021-03-15T22:17:56.941920+0000 2021-03-15T22:17:57.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[62156]: mgr.y (mgr.14144) 213 : cluster [DBG] pgmap v167: 68 pgs: 4 creating+activating, 11 creating+peering, 32 active+clean, 21 unknown; 1.2 KiB data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:57.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: audit 2021-03-15T22:17:57.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: 2021-03-15T22:17:57.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: 2021 2021-03-15T22:17:57.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: - 2021-03-15T22:17:57.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: 03 2021-03-15T22:17:57.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: - 2021-03-15T22:17:57.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: 15 2021-03-15T22:17:57.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: T 2021-03-15T22:17:57.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: 22 2021-03-15T22:17:57.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: :17:55.981746+0000 mon.a (mon. 2021-03-15T22:17:57.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: 0) 467 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2021-03-15T22:17:57.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: cluster 2021-03-15T22:17:57.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: 2021-03-15T22:17:55.981920+0000 2021-03-15T22:17:57.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: mon.a (mon.0) 468 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2021-03-15T22:17:57.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: 2021-03-15T22:17:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:17:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: :17:56.480503+0000 mon.a (mon.0) 2021-03-15T22:17:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: 469 : cluster [DBG] mgrmap e15: y(active, since 3m), standbys: x 2021-03-15T22:17:57.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:17:57.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: T22:17:56.941920+0000 mgr.y (mgr. 2021-03-15T22:17:57.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:56 smithi076 conmon[56514]: 14144) 213 : cluster [DBG] pgmap v167: 68 pgs: 4 creating+activating, 11 creating+peering, 32 active+clean, 21 unknown; 1.2 KiB data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:57.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:56 smithi125 conmon[56739]: audit 2021-03-15T22:17:55.981746+0000 mon.a (mon.0) 467 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2021-03-15T22:17:57.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:56 smithi125 conmon[56739]: cluster 2021-03-15T22:17:55.981920+0000 mon.a (mon.0) 468 : cluster [DBG] osdmap e59: 8 total, 8 up, 8 in 2021-03-15T22:17:57.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:56 smithi125 conmon[56739]: cluster 2021-03-15T22:17:56.480503+0000 mon.a (mon.0) 469 : cluster [DBG] mgrmap e15: y(active, since 3m), standbys: x 2021-03-15T22:17:57.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:56 smithi125 conmon[56739]: cluster 2021-03-15T22:17:56.941920+0000 mgr.y (mgr.14144) 213 : cluster [DBG] pgmap v167: 68 pgs: 4 creating+activating, 11 creating+peering, 32 active+clean, 21 unknown; 1.2 KiB data, 43 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:58.005 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:57 smithi125 systemd[1]: Starting Ceph iscsi.iscsi.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:17:58.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: cluster 2021-03-15T22:17:58.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: 2021-03-15T22:17:58.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: 2021 2021-03-15T22:17:58.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: - 2021-03-15T22:17:58.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: 03 2021-03-15T22:17:58.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: - 2021-03-15T22:17:58.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: 15 2021-03-15T22:17:58.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: T 2021-03-15T22:17:58.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: 22 2021-03-15T22:17:58.010 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: :17:56.991997+0000 mon.a (mon.0) 470 : 2021-03-15T22:17:58.010 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2021-03-15T22:17:58.010 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: audit 2021-03-15 2021-03-15T22:17:58.010 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: T22:17:56.999476+0000 mon.a (mon.0) 471 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-03-15T22:17:58.011 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:17:58.011 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: 03-15T22:17:57.995254+0000 mon.a ( 2021-03-15T22:17:58.011 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: mon.0) 472 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-03-15T22:17:58.011 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:17:58.012 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: -03-15T22:17:57.995402+0000 mon.a 2021-03-15T22:17:58.012 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:57 smithi076 conmon[56514]: (mon.0) 473 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2021-03-15T22:17:58.012 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: cluster 2021-03-15T22:17:58.013 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.013 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021 2021-03-15T22:17:58.013 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: - 2021-03-15T22:17:58.013 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 03 2021-03-15T22:17:58.013 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: - 2021-03-15T22:17:58.014 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 15 2021-03-15T22:17:58.014 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: T 2021-03-15T22:17:58.014 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 22 2021-03-15T22:17:58.014 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.015 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 17 2021-03-15T22:17:58.015 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.015 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 56 2021-03-15T22:17:58.015 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: . 2021-03-15T22:17:58.015 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 991997 2021-03-15T22:17:58.016 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: +0000 2021-03-15T22:17:58.016 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.016 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: mon.a 2021-03-15T22:17:58.016 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: ( 2021-03-15T22:17:58.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: mon 2021-03-15T22:17:58.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: . 2021-03-15T22:17:58.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 0 2021-03-15T22:17:58.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: ) 2021-03-15T22:17:58.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 470 2021-03-15T22:17:58.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: cluster 2021-03-15T22:17:58.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: [DBG] 2021-03-15T22:17:58.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: osdmap e60: 8 total, 8 up, 8 in 2021-03-15T22:17:58.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: audit 2021-03-15T22:17:58.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021 2021-03-15T22:17:58.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: - 2021-03-15T22:17:58.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 03 2021-03-15T22:17:58.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: - 2021-03-15T22:17:58.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 15 2021-03-15T22:17:58.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: T 2021-03-15T22:17:58.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 22 2021-03-15T22:17:58.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 17 2021-03-15T22:17:58.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 56 2021-03-15T22:17:58.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: . 2021-03-15T22:17:58.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 999476 2021-03-15T22:17:58.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: +0000 2021-03-15T22:17:58.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: mon.a 2021-03-15T22:17:58.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: ( 2021-03-15T22:17:58.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: mon 2021-03-15T22:17:58.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: . 2021-03-15T22:17:58.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 0 2021-03-15T22:17:58.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: ) 2021-03-15T22:17:58.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 471 2021-03-15T22:17:58.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: audit 2021-03-15T22:17:58.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: [INF] 2021-03-15T22:17:58.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-03-15T22:17:58.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: audit 2021-03-15T22:17:58.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021 2021-03-15T22:17:58.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: - 2021-03-15T22:17:58.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 03 2021-03-15T22:17:58.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: - 2021-03-15T22:17:58.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 15 2021-03-15T22:17:58.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: T 2021-03-15T22:17:58.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 22 2021-03-15T22:17:58.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 17 2021-03-15T22:17:58.264 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:58 smithi125 conmon[56739]: cluster 2021-03-15T22:17:56.991997+0000 mon.a (mon.0) 470 : cluster [DBG] osdmap e60: 8 total, 8 up, 8 in 2021-03-15T22:17:58.265 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:58 smithi125 conmon[56739]: audit 2021-03-15T22:17:56.999476+0000 mon.a (mon.0) 471 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2021-03-15T22:17:58.265 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:58 smithi125 conmon[56739]: audit 2021-03-15T22:17:57.995254+0000 mon.a (mon.0) 472 : audit [INF] from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-03-15T22:17:58.265 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:58 smithi125 conmon[56739]: cluster 2021-03-15T22:17:57.995402+0000 mon.a (mon.0) 473 : cluster [DBG] osdmap e61: 8 total, 8 up, 8 in 2021-03-15T22:17:58.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 57 2021-03-15T22:17:58.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: . 2021-03-15T22:17:58.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 995254 2021-03-15T22:17:58.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: +0000 2021-03-15T22:17:58.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: mon.a 2021-03-15T22:17:58.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: ( 2021-03-15T22:17:58.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: mon 2021-03-15T22:17:58.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: . 2021-03-15T22:17:58.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 0 2021-03-15T22:17:58.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: ) 2021-03-15T22:17:58.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 472 2021-03-15T22:17:58.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: audit 2021-03-15T22:17:58.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: [INF] 2021-03-15T22:17:58.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/926017807' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2021-03-15T22:17:58.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: cluster 2021-03-15T22:17:58.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021 2021-03-15T22:17:58.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: - 2021-03-15T22:17:58.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 03 2021-03-15T22:17:58.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: - 2021-03-15T22:17:58.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 15 2021-03-15T22:17:58.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: T 2021-03-15T22:17:58.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 22 2021-03-15T22:17:58.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 17 2021-03-15T22:17:58.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 57 2021-03-15T22:17:58.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: . 2021-03-15T22:17:58.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 995402 2021-03-15T22:17:58.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: +0000 2021-03-15T22:17:58.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: mon.a 2021-03-15T22:17:58.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: ( 2021-03-15T22:17:58.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: mon 2021-03-15T22:17:58.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: . 2021-03-15T22:17:58.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 0 2021-03-15T22:17:58.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: ) 2021-03-15T22:17:58.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 473 2021-03-15T22:17:58.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: : 2021-03-15T22:17:58.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: cluster 2021-03-15T22:17:58.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: [DBG] 2021-03-15T22:17:58.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: osdmap e61: 8 total, 8 up, 8 in 2021-03-15T22:17:58.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:58 smithi076 conmon[62156]: 2021-03-15T22:17:58.737 INFO:teuthology.orchestra.run.smithi125.stdout:Scheduled prometheus update... 2021-03-15T22:17:58.781 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:58 smithi125 podman[73347]: 2021-03-15 22:17:58.528298332 +0000 UTC m=+0.281095739 container create e0d98c49cc76b86aab80b758a5db2f907777e24929bef1878cd813832384186f (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-iscsi.iscsi.a-tcmu, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, ceph=True, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, io.buildah.version=1.18.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2021-03-15T22:17:58.781 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:58 smithi125 podman[73479]: 2021-03-15 22:17:58.671376027 +0000 UTC m=+0.254034402 container create 066e286aec797d6038d767e373bab6c94a0e7f6748b110b5d85c033df90b086a (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-iscsi.iscsi.a, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, io.buildah.version=1.18.0) 2021-03-15T22:17:58.782 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:58 smithi125 podman[73347]: 2021-03-15 22:17:58.780084717 +0000 UTC m=+0.532882127 container init e0d98c49cc76b86aab80b758a5db2f907777e24929bef1878cd813832384186f (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-iscsi.iscsi.a-tcmu, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0, org.label-schema.build-date=20201204, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau ) 2021-03-15T22:17:59.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:58 smithi125 podman[73347]: 2021-03-15 22:17:58.876583515 +0000 UTC m=+0.629380960 container start e0d98c49cc76b86aab80b758a5db2f907777e24929bef1878cd813832384186f (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-iscsi.iscsi.a-tcmu, GIT_CLEAN=True, maintainer=Dimitri Savineau , io.buildah.version=1.18.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2021-03-15T22:17:59.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:58 smithi125 podman[73347]: 2021-03-15 22:17:58.876681244 +0000 UTC m=+0.629478687 container attach e0d98c49cc76b86aab80b758a5db2f907777e24929bef1878cd813832384186f (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-iscsi.iscsi.a-tcmu, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20201204, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2021-03-15T22:17:59.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:58 smithi125 bash[73240]: log file path now is '/var/log/tcmu-runner.log' 2021-03-15T22:17:59.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:59 smithi125 podman[73479]: 2021-03-15 22:17:59.00465648 +0000 UTC m=+0.587314763 container init 066e286aec797d6038d767e373bab6c94a0e7f6748b110b5d85c033df90b086a (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-iscsi.iscsi.a, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, GIT_CLEAN=True, io.buildah.version=1.18.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2021-03-15T22:17:59.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:59 smithi125 podman[73479]: 2021-03-15 22:17:59.037971949 +0000 UTC m=+0.620630205 container start 066e286aec797d6038d767e373bab6c94a0e7f6748b110b5d85c033df90b086a (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-iscsi.iscsi.a, GIT_CLEAN=True, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0) 2021-03-15T22:17:59.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:59 smithi125 bash[73240]: 066e286aec797d6038d767e373bab6c94a0e7f6748b110b5d85c033df90b086a 2021-03-15T22:17:59.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:59 smithi125 systemd[1]: Started Ceph iscsi.iscsi.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:17:59.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[73528]: debug Started the configuration object watcher 2021-03-15T22:17:59.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[73528]: debug Checking for config object changes every 1s 2021-03-15T22:17:59.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[73528]: debug Processing osd blocklist entries for this node 2021-03-15T22:17:59.790 DEBUG:teuthology.orchestra.run.smithi125:prometheus.a> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@prometheus.a.service 2021-03-15T22:17:59.793 INFO:tasks.cephadm:Adding node-exporter.a on smithi076 2021-03-15T22:17:59.793 INFO:tasks.cephadm:Adding node-exporter.b on smithi125 2021-03-15T22:17:59.794 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch apply node-exporter '2;smithi076=a;smithi125=b' 2021-03-15T22:17:59.816 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: audit 2021-03-15T22:17:58.591661+0000 mgr.y (mgr.14144 2021-03-15T22:17:59.817 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: ) 214 : audit [DBG] from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi125=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:17:59.817 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: cephadm 2021-03-15T22:17:58.592280+0000 mgr.y (mgr.14144) 215 : cephadm [INF] Saving service prometheus spec with placement smithi125=a;count:1 2021-03-15T22:17:59.818 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:17:59.818 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: T22:17:58.734627+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:59.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: cluster 2021-03-15T22:17 2021-03-15T22:17:59.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: :58.942453+0000 mgr.y (mgr.14144) 2021-03-15T22:17:59.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: 216 : cluster [DBG] pgmap v170: 100 pgs: 4 creating+activating, 12 creating+peering, 60 active+clean, 24 unknown; 1.2 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:17:59.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:17:59.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: -15T22:17:59.003911+0000 2021-03-15T22:17:59.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: mon.a (mon.0) 475 : cluster [DBG] 2021-03-15T22:17:59.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: osdmap e62: 8 total, 8 up, 8 in 2021-03-15T22:17:59.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: audit 2021-03-15T22:17: 2021-03-15T22:17:59.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: 59.004601+0000 mon.a (mon.0) 476 2021-03-15T22:17:59.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-03-15T22:17:59.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: audit 2021 2021-03-15T22:17:59.824 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: -03-15T22:17:59.084958+0000 2021-03-15T22:17:59.824 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: mon.a (mon.0) 477 : audit [INF] 2021-03-15T22:17:59.825 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:17:59.825 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: audit 2021 2021-03-15T22:17:59.826 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: -03-15T22:17:59.085572 2021-03-15T22:17:59.826 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: +0000 mon.a (mon.0) 478 : audit 2021-03-15T22:17:59.826 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:17:59.827 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:17:59.827 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: 22:17:59.550627+0000 mon.b (mon 2021-03-15T22:17:59.828 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[56739]: .2) 24 : audit [DBG] from='client.? 172.21.15.125:0/292658556' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-03-15T22:17:59.829 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[73528]: debug Reading the configuration object to update local LIO configuration 2021-03-15T22:17:59.829 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[73528]: debug Configuration does not have an entry for this host(ceph-8a572802-85db-11eb-9091-001a4aab830c-iscsi.iscsi.a) - nothing to define to LIO 2021-03-15T22:17:59.830 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:17:59 smithi125 conmon[73528]: debug * Running on http://[::]:5000/ (Press CTRL+C to quit) 2021-03-15T22:17:59.831 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:-- Logs begin at Mon 2021-03-15 21:44:18 UTC. -- 2021-03-15T22:18:00.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: audit 2021-03-15T22:17:58.591661+0000 mgr.y (mgr.14144) 214 : audit [DBG] 2021-03-15T22:18:00.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi125=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:18:00.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: cephadm 2021-03-15T22:17:58.592280+0000 mgr.y (mgr.14144) 215 : cephadm [INF] Saving service prometheus spec with placement smithi125=a;count:1 2021-03-15T22:18:00.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: audit 2021-03-15T22:17:58.734627+0000 mon.a (mon.0) 474 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:00.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: cluster 2021-03-15T22:17:58.942453+0000 2021-03-15T22:18:00.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: mgr.y (mgr.14144) 216 : cluster [DBG] pgmap v170: 100 pgs: 4 creating+activating, 12 creating+peering, 60 active+clean, 24 unknown; 1.2 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:18:00.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: cluster 2021-03-15T22:17:59.003911+0000 mon.a (mon.0 2021-03-15T22:18:00.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: ) 475 : cluster [DBG] osdmap e62: 8 total, 8 up, 8 in 2021-03-15T22:18:00.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: audit 2021-03-15T22:18:00.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: 2021-03-15T22:17:59.004601+0000 mon.a (mon.0) 476 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-03-15T22:18:00.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: audit 2021-03-15T22 2021-03-15T22:18:00.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: :17:59.084958+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:00.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: audit 2021-03-15T22:17:59. 2021-03-15T22:18:00.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: 085572+0000 mon.a (mon.0) 478 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:18:00.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: audit 2021-03-15T22:17:59.550627+0000 mon.b (mon 2021-03-15T22:18:00.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[56514]: .2) 24 : audit [DBG] from='client.? 172.21.15.125:0/292658556' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-03-15T22:18:00.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: audit 2021-03-15T22:17:58.591661+0000 mgr.y ( 2021-03-15T22:18:00.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: mgr.14144) 214 : audit [DBG] from='client.34340 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi125=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:18:00.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: cephadm 2021-03-15T 2021-03-15T22:18:00.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: 22:17:58.592280+0000 mgr.y (mgr.14144) 215 : cephadm [INF] 2021-03-15T22:18:00.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: Saving service prometheus spec with placement smithi125=a;count:1 2021-03-15T22:18:00.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:18:00.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: -15T22:17:58.734627+0000 2021-03-15T22:18:00.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: mon.a (mon.0) 474 : 2021-03-15T22:18:00.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:00.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:18:00.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: 15T22:17:58.942453+0000 mgr.y (mgr.14144) 216 : cluster [DBG] 2021-03-15T22:18:00.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: pgmap v170: 100 pgs: 4 creating+activating, 12 creating+peering, 60 active+clean, 24 unknown; 1.2 KiB data, 44 MiB used, 715 GiB / 715 GiB avail 2021-03-15T22:18:00.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: cluster 2021-03-15T22:18:00.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: 2021-03-15T22:17:59.003911+0000 mon.a (mon.0) 475 : cluster 2021-03-15T22:18:00.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: [DBG] osdmap e62: 8 total, 8 up, 8 in 2021-03-15T22:18:00.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: audit 2021-03-15T22:17:59.004601+0000 mon.a (mon.0) 476 2021-03-15T22:18:00.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2021-03-15T22:18:00.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:18:00.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: 03-15T22:17:59.084958+0000 mon.a (mon.0) 477 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:00.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: audit 2021-03-15T22:17:59.085572+0000 mon.a (mon.0) 478 : audit [DBG] 2021-03-15T22:18:00.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:18:00.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: audit 2021-03-15T22:17 2021-03-15T22:18:00.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:17:59 smithi076 conmon[62156]: :59.550627+0000 mon.b (mon.2) 24 : audit [DBG] from='client.? 172.21.15.125:0/292658556' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2021-03-15T22:18:01.010 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:01 smithi125 conmon[56739]: audit 2021-03-15T22:18:00.005567+0000 2021-03-15T22:18:01.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[56514]: audit 2021-03-15T22:18:00.005567+0000 mon.a (mon.0) 479 : audit [INF] 2021-03-15T22:18:01.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-03-15T22:18:01.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[56514]: cluster 2021-03-15T22:18:00.005788+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-03-15T22:18:01.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[56514]: audit 2021-03-15T22:18: 2021-03-15T22:18:01.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[56514]: 00.007836+0000 mon.a (mon.0) 481 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-03-15T22:18:01.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:18:01.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[56514]: T22:18:00.942907+0000 mgr.y (mgr.14144) 217 2021-03-15T22:18:01.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[56514]: : cluster [DBG] pgmap v173: 132 pgs: 3 creating+activating, 20 creating+peering, 71 active+clean, 38 unknown; 1.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 0 op/s 2021-03-15T22:18:01.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: audit 2021-03-15T22:18:00.005567+0000 mon.a ( 2021-03-15T22:18:01.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: mon.0) 479 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-03-15T22:18:01.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: cluster 2021-03-15T22:18:01.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: 2021-03-15T22:18:00.005788+0000 mon.a (mon.0) 480 : 2021-03-15T22:18:01.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-03-15T22:18:01.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:18:01.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: 22:18:00.007836+0000 mon.a (mon.0) 481 : audit 2021-03-15T22:18:01.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-03-15T22:18:01.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:18:01.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: -03-15T22:18:00.942907+0000 2021-03-15T22:18:01.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: mgr.y (mgr.14144) 217 : cluster [DBG] 2021-03-15T22:18:01.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:01 smithi076 conmon[62156]: pgmap v173: 132 pgs: 3 creating+activating, 20 creating+peering, 71 active+clean, 38 unknown; 1.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 0 op/s 2021-03-15T22:18:01.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:01 smithi125 conmon[56739]: mon.a (mon.0) 479 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2021-03-15T22:18:01.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:01 smithi125 conmon[56739]: cluster 2021-03-15T22:18:00.005788+0000 mon.a (mon.0) 480 : cluster [DBG] osdmap e63: 8 total, 8 up, 8 in 2021-03-15T22:18:01.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:01 smithi125 conmon[56739]: audit 2021-03-15T22:18:00.007836+0000 mon.a (mon.0) 481 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2021-03-15T22:18:01.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:01 smithi125 conmon[56739]: cluster 2021-03-15T22:18:00.942907+0000 mgr.y (mgr.14144) 217 : cluster [DBG] pgmap v173: 132 pgs: 3 creating+activating, 20 creating+peering, 71 active+clean, 38 unknown; 1.3 KiB data, 45 MiB used, 715 GiB / 715 GiB avail; 255 B/s rd, 511 B/s wr, 0 op/s 2021-03-15T22:18:02.008 INFO:teuthology.orchestra.run.smithi125.stdout:Scheduled node-exporter update... 2021-03-15T22:18:02.267 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: debug 2021-03-15T22:18:02.000+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:18:02.267 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:02.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:02.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:02.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:02.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:02.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:02.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:02.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:18:02.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: debug 2021-03-15T22:18:02.001+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:02.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:02.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:02.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:02.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:02.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:02.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:02.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:02.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:02.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: debug 2021-03-15T22:18:02.002+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:02.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:02.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:02.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:02.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:02.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:02.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:02.273 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:02.273 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:02.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: audit 2021-03-15T22:18:01.011149+0000 mon.a (mon.0) 482 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-03-15T22:18:02.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: cluster 2021-03-15T22:18:01.011258+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-03-15T22:18:02.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:18:02.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: 15T22:18:01.013279+0000 mon.a (mon.0) 484 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-03-15T22:18:02.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:18:02.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: 15T22:18:01.032246+0000 mon.a (mon.0) 485 : 2021-03-15T22:18:02.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:18:02.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: audit 2021-03-15T22:18:02.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: 2021-03-15T22:18:01.033419+0000 2021-03-15T22:18:02.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:18:02.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: 2021-03-15T22:18:02.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: audit 2021-03-15T22:18:02.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: 2021-03-15T22:18: 2021-03-15T22:18:02.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: 01.983153+0000 mon.a (mon.0) 487 : audit 2021-03-15T22:18:02.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:02.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: audit 2021-03-15T22:18:02.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: 2021-03-15T22 2021-03-15T22:18:02.277 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: :18:01.990284+0000 mon.a (mon.0) 2021-03-15T22:18:02.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: 488 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:02.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:18:02.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: -15T22:18:02.005773+0000 mon.a (mon.0) 2021-03-15T22:18:02.278 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[56514]: 489 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:02.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: audit 2021-03-15T22:18:01.011149+0000 mon.a (mon. 2021-03-15T22:18:02.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: 0) 482 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-03-15T22:18:02.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:18:02.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: -03-15T22:18:01.011258+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-03-15T22:18:02.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: audit 2021-03-15T22:18:02.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: 2021-03-15T22:18 2021-03-15T22:18:02.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: :01.013279+0000 mon.a (mon.0) 484 : audit 2021-03-15T22:18:02.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-03-15T22:18:02.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: audit 2021-03-15T22:18:02.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:18:02.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: :18:01.032246+0000 mon.a (mon.0) 485 : 2021-03-15T22:18:02.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:18:02.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: audit 2021-03-15T22:18:02.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:18:02.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: 18:01.033419+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:18:02.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: 2021-03-15T22:18:02.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:18:02.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: 18:01.983153+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:02.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: audit 2021-03-15T22:18:02.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: 2021-03-15T22:18:01 2021-03-15T22:18:02.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: .990284+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:02.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: audit 2021-03-15T22:18:02.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: 2021-03-15T22:18:02.005773+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:02.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:02 smithi076 conmon[62156]: 2021-03-15T22:18:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:02 smithi125 conmon[56739]: audit 2021-03-15T22:18:01.011149+0000 mon.a (mon.0) 482 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2021-03-15T22:18:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:02 smithi125 conmon[56739]: cluster 2021-03-15T22:18:01.011258+0000 mon.a (mon.0) 483 : cluster [DBG] osdmap e64: 8 total, 8 up, 8 in 2021-03-15T22:18:02.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:02 smithi125 conmon[56739]: audit 2021-03-15T22:18:01.013279+0000 mon.a (mon.0) 484 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]: dispatch 2021-03-15T22:18:02.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:02 smithi125 conmon[56739]: audit 2021-03-15T22:18:01.032246+0000 mon.a (mon.0) 485 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:18:02.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:02 smithi125 conmon[56739]: audit 2021-03-15T22:18:01.033419+0000 mon.a (mon.0) 486 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:18:02.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:02 smithi125 conmon[56739]: audit 2021-03-15T22:18:01.983153+0000 mon.a (mon.0) 487 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:02.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:02 smithi125 conmon[56739]: audit 2021-03-15T22:18:01.990284+0000 mon.a (mon.0) 488 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:02.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:02 smithi125 conmon[56739]: audit 2021-03-15T22:18:02.005773+0000 mon.a (mon.0) 489 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:02.746 DEBUG:teuthology.orchestra.run.smithi076:node-exporter.a> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@node-exporter.a.service 2021-03-15T22:18:02.749 DEBUG:teuthology.orchestra.run.smithi125:node-exporter.b> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@node-exporter.b.service 2021-03-15T22:18:02.752 INFO:tasks.cephadm:Adding alertmanager.a on smithi076 2021-03-15T22:18:02.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch apply alertmanager '1;smithi076=a' 2021-03-15T22:18:02.778 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:-- Logs begin at Mon 2021-03-15 21:44:23 UTC. -- 2021-03-15T22:18:02.781 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:-- Logs begin at Mon 2021-03-15 21:44:18 UTC. -- 2021-03-15T22:18:03.027 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:01. 2021-03-15T22:18:03.027 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: 983933+0000 mgr.y (mgr.14144) 218 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:18:03.027 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: audit 2021-03-15T22:18:01.998965+0000 mgr.y (mgr.14144) 219 : audit [DBG] from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi076=a;smithi125=b", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:18:03.028 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:01.999697+0000 mgr.y (mgr.14144) 220 : cephadm [INF] Saving service node-exporter spec with placement smithi076=a;smithi125=b;count:2 2021-03-15T22:18:03.028 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:02.001679+0000 mgr.y (mgr.14144) 221 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:18:03.028 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:03.028 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:03.028 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:03.029 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:03.029 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:03.029 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:03.029 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:03.030 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:18:03.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:03.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: 2021-03 2021-03-15T22:18:03.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: -15T22:18:01.983933+0000 mgr.y (mgr.14144) 218 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:18:03.031 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: audit 2021 2021-03-15T22:18:03.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: -03-15T22:18:01.998965+0000 mgr.y (mgr.14144 2021-03-15T22:18:03.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: ) 219 : audit [DBG] from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi076=a;smithi125=b", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:18:03.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:18:03.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: -03-15T22:18:01.999697+0000 mgr.y 2021-03-15T22:18:03.032 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: (mgr.14144) 220 : cephadm [INF] Saving service node-exporter spec with placement smithi076=a;smithi125=b;count:2 2021-03-15T22:18:03.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: 2021-03-15T22:18:03.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: cephadm 2021-03-15T22 2021-03-15T22:18:03.033 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: :18:02.001679+0000 mgr.y (mgr.14144) 2021-03-15T22:18:03.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: 221 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:18:03.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:03.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:03.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:03.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:03.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:03.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:03.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:03.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:18:03.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:02.002355+0000 mgr.y (mgr.14144) 222 : cephadm [ERR] 2021-03-15T22:18:03.037 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:03.037 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:03.037 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:03.037 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:03.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:03.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:03.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:03.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:03.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:03.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:02.002928+0000 mgr.y (mgr.14144) 223 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:03.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:03.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:03.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:03.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:03.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:03.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:03.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: 2021-03-15T22:18:02.012341+0000 mgr.y (mgr.14144) 224 : cephadm [INF] Deploying daemon prometheus.a on smithi125 2021-03-15T22:18:03.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: audit 2021-03-15T22:18:02.015796+0000 mon.a ( 2021-03-15T22:18:03.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: mon.0) 490 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-03-15T22:18:03.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:18:03.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: -15T22:18:02.015887+0000 mon.a 2021-03-15T22:18:03.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: (mon.0) 491 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-03-15T22:18:03.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: 2021-03-15T22:18:03.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: cluster 2021-03-15T22:18: 2021-03-15T22:18:03.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: 02.943334+0000 mgr.y (mgr.14144) 225 : 2021-03-15T22:18:03.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[62156]: cluster [DBG] pgmap v176: 132 pgs: 17 creating+peering, 108 active+clean, 7 unknown; 17 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 26 KiB/s wr, 101 op/s 2021-03-15T22:18:03.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:02.002355+0000 mgr.y (mgr.14144) 222 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:03.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:03.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:03.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:03.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:03.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:03.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:03.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:03.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:03.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:02.002928+0000 mgr.y (mgr.14144) 223 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:03.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:03.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:03.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:03.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:03.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:03.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:03.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:03.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:03.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:02.012341+0000 mgr.y (mgr.14144) 224 : cephadm [INF] Deploying daemon prometheus.a on smithi125 2021-03-15T22:18:03.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: audit 2021-03-15T22:18:02.015796+0000 mon.a (mon.0) 490 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-03-15T22:18:03.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: cluster 2021-03-15T22:18:02.015887 2021-03-15T22:18:03.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: +0000 mon.a (mon.0) 491 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-03-15T22:18:03.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:18:03.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: :18:02.943334+0000 mgr.y (mgr.14144) 225 2021-03-15T22:18:03.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:03 smithi125 conmon[56739]: : cluster [DBG] pgmap v176: 132 pgs: 17 creating+peering, 108 active+clean, 7 unknown; 17 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 26 KiB/s wr, 101 op/s 2021-03-15T22:18:03.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:01.983933+0000 mgr.y (mgr.14144) 218 : cephadm [INF] 2021-03-15T22:18:03.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:18:03.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: audit 2021-03-15T22:18:01.998965+0000 mgr.y (mgr.14144) 219 : audit [DBG] from='client.34376 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi076=a;smithi125=b", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:18:03.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:01.999697+0000 mgr.y (mgr.14144) 220 : cephadm [INF] Saving service node-exporter spec with placement smithi076=a;smithi125=b;count:2 2021-03-15T22:18:03.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:02.001679+0000 mgr.y (mgr.14144) 221 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:18:03.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:03.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:03.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:03.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:03.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:03.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:03.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:03.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:18:03.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:02.002355+0000 mgr.y (mgr.14144) 222 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:03.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:03.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:03.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:03.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:03.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:03.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:03.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:03.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:03.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:02.002928+0000 mgr.y ( 2021-03-15T22:18:03.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: mgr.14144) 223 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:03.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:03.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:03.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:03.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:03.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:03.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:03.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:03.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:03.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:02.012341+0000 mgr.y (mgr.14144) 224 : cephadm [INF] Deploying daemon prometheus.a on smithi125 2021-03-15T22:18:03.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: audit 2021-03-15T22:18:02.015796+0000 mon.a (mon.0) 490 : audit [INF] from='client.? 172.21.15.76:0/2912955800' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_min", "val": "8"}]': finished 2021-03-15T22:18:03.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: cluster 2021-03-15T22:18:02. 2021-03-15T22:18:03.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: 015887+0000 mon.a (mon.0) 491 : cluster [DBG] osdmap e65: 8 total, 8 up, 8 in 2021-03-15T22:18:03.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: cluster 2021-03-15T22:18:02.943334+0000 mgr.y (mgr.14144) 225 : cluster 2021-03-15T22:18:03.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:03 smithi076 conmon[56514]: [DBG] pgmap v176: 132 pgs: 17 creating+peering, 108 active+clean, 7 unknown; 17 KiB data, 47 MiB used, 715 GiB / 715 GiB avail; 25 KiB/s rd, 26 KiB/s wr, 101 op/s 2021-03-15T22:18:05.010 INFO:teuthology.orchestra.run.smithi125.stdout:Scheduled alertmanager update... 2021-03-15T22:18:06.229 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:05 smithi125 conmon[56739]: cluster 2021-03-15T22:18:04.944003+0000 mgr.y (mgr.14144) 226 : cluster [DBG] pgmap v177: 132 pgs: 2 creating+peering, 130 active+clean; 22 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 24 KiB/s wr, 135 op/s 2021-03-15T22:18:06.230 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:05 smithi125 conmon[56739]: audit 2021-03-15T22:18:05.008773+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:06.258 DEBUG:teuthology.orchestra.run.smithi076:alertmanager.a> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@alertmanager.a.service 2021-03-15T22:18:06.261 INFO:tasks.cephadm:Adding grafana.a on smithi125 2021-03-15T22:18:06.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph orch apply grafana '1;smithi125=a' 2021-03-15T22:18:06.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:05 smithi076 conmon[56514]: cluster 2021-03-15T22:18:04.944003+0000 mgr.y (mgr.14144) 226 : cluster 2021-03-15T22:18:06.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:05 smithi076 conmon[56514]: [DBG] pgmap v177: 132 pgs: 2 creating+peering, 130 active+clean; 22 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 24 KiB/s wr, 135 op/s 2021-03-15T22:18:06.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:05 smithi076 conmon[56514]: audit 2021-03-15T22:18:05.008773+0000 mon.a (mon.0) 492 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:06.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:05 smithi076 conmon[62156]: cluster 2021-03-15T22:18:04.944003+0000 mgr.y (mgr.14144) 226 : cluster [DBG] 2021-03-15T22:18:06.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:05 smithi076 conmon[62156]: pgmap v177: 132 pgs: 2 creating+peering, 130 active+clean; 22 KiB data, 49 MiB used, 715 GiB / 715 GiB avail; 41 KiB/s rd, 24 KiB/s wr, 135 op/s 2021-03-15T22:18:06.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:05 smithi076 conmon[62156]: audit 2021-03-15T22:18:05.008773+0000 mon.a (mon.0) 492 2021-03-15T22:18:06.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:05 smithi076 conmon[62156]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:06.290 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:-- Logs begin at Mon 2021-03-15 21:44:23 UTC. -- 2021-03-15T22:18:07.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:06 smithi076 conmon[56514]: audit 2021-03-15T22:18:05.002372+0000 mgr.y (mgr.14144) 227 : audit [DBG] from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi076=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:18:07.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:06 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:05.003072+0000 mgr.y (mgr.14144) 228 : cephadm [INF] Saving service alertmanager spec with placement smithi076=a;count:1 2021-03-15T22:18:07.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:06 smithi076 conmon[62156]: audit 2021-03-15T22:18:05.002372+0000 2021-03-15T22:18:07.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:06 smithi076 conmon[62156]: mgr.y (mgr.14144) 227 : audit [DBG] from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi076=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:18:07.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:06 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:18:07.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:06 smithi076 conmon[62156]: 03-15T22:18:05. 2021-03-15T22:18:07.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:06 smithi076 conmon[62156]: 003072+0000 mgr.y (mgr.14144) 228 : cephadm 2021-03-15T22:18:07.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:06 smithi076 conmon[62156]: [INF] Saving service alertmanager spec with placement smithi076=a;count:1 2021-03-15T22:18:07.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:06 smithi125 conmon[56739]: audit 2021-03-15T22:18:05.002372+0000 mgr.y (mgr.14144) 227 : audit [DBG] from='client.34382 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi076=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:18:07.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:06 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:05.003072+0000 mgr.y (mgr.14144) 228 : cephadm [INF] Saving service alertmanager spec with placement smithi076=a;count:1 2021-03-15T22:18:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[56514]: cluster 2021-03-15T22:18:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[56514]: 2021-03-15T22:18:06.944765+0000 2021-03-15T22:18:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[56514]: mgr.y (mgr.14144) 229 : cluster [DBG] pgmap v178: 132 pgs: 132 active+clean; 24 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 22 KiB/s wr, 193 op/s 2021-03-15T22:18:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:18:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[56514]: :18:07.166373+0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[56514]: cluster 2021-03-15T22:18: 2021-03-15T22:18:08.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[56514]: 07.166564+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[62156]: cluster 2021-03-15T22:18:06.944765+0000 mgr.y (mgr 2021-03-15T22:18:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[62156]: .14144) 229 : cluster [DBG] pgmap v178: 132 pgs: 132 active+clean; 24 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 22 KiB/s wr, 193 op/s 2021-03-15T22:18:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[62156]: cluster 2021-03-15T22:18:07.166373 2021-03-15T22:18:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[62156]: cluster 2021-03-15T22:18 2021-03-15T22:18:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[62156]: :07.166564+0000 osd.1 (osd.1) 6 : 2021-03-15T22:18:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:07 smithi076 conmon[62156]: cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:08.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:07 smithi125 conmon[56739]: cluster 2021-03-15T22:18:06.944765+0000 mgr.y (mgr.14144) 229 : cluster [DBG] pgmap v178: 132 pgs: 132 active+clean; 24 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 67 KiB/s rd, 22 KiB/s wr, 193 op/s 2021-03-15T22:18:08.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:07 smithi125 conmon[56739]: cluster 2021-03-15T22:18:07.166373 2021-03-15T22:18:08.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:07 smithi125 conmon[56739]: +0000 osd.1 (osd.1) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:08.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:07 smithi125 conmon[56739]: cluster 2021-03-15T22:18:07.166564+0000 osd.1 (osd.1) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:09.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:08 smithi076 conmon[56514]: cluster 2021-03-15T22:18:07.963492+0000 mon.a (mon.0) 493 : cluster 2021-03-15T22:18:09.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:08 smithi076 conmon[56514]: [DBG] osdmap e66: 8 total, 8 up, 8 in 2021-03-15T22:18:09.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:08 smithi076 conmon[62156]: cluster 2021-03-15T22:18:07.963492+0000 2021-03-15T22:18:09.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:08 smithi076 conmon[62156]: mon.a (mon.0) 493 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2021-03-15T22:18:09.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:08 smithi125 conmon[56739]: cluster 2021-03-15T22:18:07.963492+0000 mon.a (mon.0 2021-03-15T22:18:09.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:08 smithi125 conmon[56739]: ) 493 : cluster [DBG] osdmap e66: 8 total, 8 up, 8 in 2021-03-15T22:18:09.579 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:18:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:18:10.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[56514]: cluster 2021-03-15T22:18:08.945490+0000 mgr.y (mgr.14144) 230 : cluster [DBG] 2021-03-15T22:18:10.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[56514]: pgmap v180: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 19 KiB/s wr, 198 op/s 2021-03-15T22:18:10.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[56514]: audit 2021-03-15T22:18:09.300994+0000 mgr.y (mgr.14144) 231 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:10.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[56514]: cluster 2021-03-15T22:18:09.951017+0000 mon.a (mon.0) 494 2021-03-15T22:18:10.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[56514]: : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-03-15T22:18:10.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[62156]: cluster 2021-03-15T22:18:08.945490+0000 mgr.y (mgr 2021-03-15T22:18:10.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[62156]: .14144) 230 : cluster [DBG] pgmap v180: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 19 KiB/s wr, 198 op/s 2021-03-15T22:18:10.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[62156]: audit 2021-03-15T22:18:09.300994+0000 mgr.y (mgr. 2021-03-15T22:18:10.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[62156]: 14144) 231 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:10.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:18:10.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[62156]: -03-15T22:18:09.951017+0000 mon.a (mon.0 2021-03-15T22:18:10.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:09 smithi076 conmon[62156]: ) 494 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-03-15T22:18:10.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:09 smithi125 conmon[56739]: cluster 2021-03-15T22:18:08.945490+0000 2021-03-15T22:18:10.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:09 smithi125 conmon[56739]: mgr.y (mgr.14144) 230 : cluster [DBG] pgmap v180: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 71 KiB/s rd, 19 KiB/s wr, 198 op/s 2021-03-15T22:18:10.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:09 smithi125 conmon[56739]: audit 2021-03-15T22:18:09.300994+0000 mgr.y (mgr.14144) 231 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:10.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:09 smithi125 conmon[56739]: cluster 2021-03-15T22:18:09.951017+0000 mon.a (mon.0) 494 : cluster [DBG] osdmap e67: 8 total, 8 up, 8 in 2021-03-15T22:18:11.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:10 smithi076 conmon[56514]: cluster 2021-03-15T22:18:10.946200+0000 mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v182: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 7.5 KiB/s wr, 153 op/s 2021-03-15T22:18:11.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:10 smithi076 conmon[56514]: 2021-03-15T22:18:11.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:10 smithi076 conmon[62156]: cluster 2021-03-15T22:18:10.946200+0000 2021-03-15T22:18:11.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:10 smithi076 conmon[62156]: mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v182: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 7.5 KiB/s wr, 153 op/s 2021-03-15T22:18:11.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:10 smithi125 conmon[56739]: cluster 2021-03-15T22:18:10.946200+0000 mgr.y (mgr.14144) 232 : cluster [DBG] pgmap v182: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 60 KiB/s rd, 7.5 KiB/s wr, 153 op/s 2021-03-15T22:18:12.475 INFO:teuthology.orchestra.run.smithi125.stdout:Scheduled grafana update... 2021-03-15T22:18:13.175 DEBUG:teuthology.orchestra.run.smithi125:grafana.a> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@grafana.a.service 2021-03-15T22:18:13.179 INFO:tasks.cephadm:Setting up client nodes... 2021-03-15T22:18:13.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-03-15T22:18:13.209 INFO:journalctl@ceph.grafana.a.smithi125.stdout:-- Logs begin at Mon 2021-03-15 21:44:18 UTC. -- 2021-03-15T22:18:13.759 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: audit 2021-03-15T22:18:12.468893+0000 mgr.y ( 2021-03-15T22:18:13.759 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: mgr.14144) 233 : audit [DBG] from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi125=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:18:13.759 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:12.469615+0000 mgr.y (mgr.14144) 234 : cephadm [INF] Saving service grafana spec with placement smithi125=a;count:1 2021-03-15T22:18:13.760 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: audit 2021-03-15T22:18:12.473336+0000 mon.a (mon.0 2021-03-15T22:18:13.760 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: ) 495 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:13.760 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:18:13.760 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: 18:12.704285+0000 osd.5 (osd.5) 2021-03-15T22:18:13.761 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:13.761 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: cluster 2021-03-15T22:18:13.761 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: 2021-03-15T22:18 2021-03-15T22:18:13.761 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: :12.704361+0000 osd.5 (osd.5) 2021-03-15T22:18:13.761 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:13.762 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: cluster 2021-03-15T22:18:13.762 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[56514]: 2021-03-15T22:18:12.946812+0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v183: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.7 KiB/s wr, 103 op/s 2021-03-15T22:18:13.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[62156]: audit 2021-03-15T22:18:12.468893+0000 mgr.y (mgr.14144) 233 : audit [DBG] from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi125=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:18:13.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:12.469615+0000 mgr.y (mgr.14144) 234 : cephadm [INF] Saving service grafana spec with placement smithi125=a;count:1 2021-03-15T22:18:13.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[62156]: audit 2021-03-15T22:18:12.473336+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:13.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[62156]: cluster 2021-03-15T22:18:12.704285+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:13.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[62156]: cluster 2021-03-15T22:18:12.704361+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:13.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:13 smithi076 conmon[62156]: cluster 2021-03-15T22:18:12.946812+0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v183: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.7 KiB/s wr, 103 op/s 2021-03-15T22:18:13.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:13 smithi125 conmon[56739]: audit 2021-03-15T22:18:12.468893+0000 mgr.y (mgr.14144) 233 : audit [DBG] from='client.34388 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi125=a", "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:18:13.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:13 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:12.469615+0000 mgr.y (mgr.14144) 234 : cephadm [INF] Saving service grafana spec with placement smithi125=a;count:1 2021-03-15T22:18:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:13 smithi125 conmon[56739]: audit 2021-03-15T22:18:12.473336+0000 mon.a (mon.0) 495 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:13 smithi125 conmon[56739]: cluster 2021-03-15T22:18:12.704285+0000 osd.5 (osd.5) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:13 smithi125 conmon[56739]: cluster 2021-03-15T22:18:12.704361+0000 osd.5 (osd.5) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:13 smithi125 conmon[56739]: cluster 2021-03-15T22:18:12.946812+0000 mgr.y (mgr.14144) 235 : cluster [DBG] pgmap v183: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 2.7 KiB/s wr, 103 op/s 2021-03-15T22:18:14.187 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:13 smithi125 systemd[1]: Starting Ceph prometheus.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:18:14.471 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 podman[75426]: 2021-03-15 22:18:14.185388506 +0000 UTC m=+0.170952184 container create be16bcbf856b4e82c3c3dafcd87297349b3272d00044064e17eb63b25e1b4a1a (image=docker.io/prom/prometheus:v2.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-15T22:18:14.472 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 podman[75426]: 2021-03-15 22:18:14.435433136 +0000 UTC m=+0.420996922 container init be16bcbf856b4e82c3c3dafcd87297349b3272d00044064e17eb63b25e1b4a1a (image=docker.io/prom/prometheus:v2.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-15T22:18:14.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:14 smithi076 conmon[62156]: cluster 2021-03-15T22:18:13.484227+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-03-15T22:18:14.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:14 smithi076 conmon[56514]: cluster 2021-03-15T22:18:13.484227+0000 mon.a (mon.0) 496 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-03-15T22:18:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[56739]: cluster 2021-03-15T22:18:13.484227+0000 mon.a ( 2021-03-15T22:18:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[56739]: mon.0) 496 : cluster [DBG] osdmap e68: 8 total, 8 up, 8 in 2021-03-15T22:18:14.787 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 podman[75426]: 2021-03-15 22:18:14.468677841 +0000 UTC m=+0.454241504 container start be16bcbf856b4e82c3c3dafcd87297349b3272d00044064e17eb63b25e1b4a1a (image=docker.io/prom/prometheus:v2.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-15T22:18:14.787 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 bash[75353]: be16bcbf856b4e82c3c3dafcd87297349b3272d00044064e17eb63b25e1b4a1a 2021-03-15T22:18:14.787 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 systemd[1]: Started Ceph prometheus.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:18:14.788 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.480Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2021-03-15T22:18:14.788 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.480Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2021-03-15T22:18:14.788 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.480Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2021-03-15T22:18:14.788 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.480Z caller=main.go:339 host_details="(Linux 4.18.0-240.15.1.el8_3.x86_64 #1 SMP Mon Mar 1 17:16:16 UTC 2021 x86_64 smithi125 (none))" 2021-03-15T22:18:14.788 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.480Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2021-03-15T22:18:14.789 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.480Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2021-03-15T22:18:14.789 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.481Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2021-03-15T22:18:14.789 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.481Z caller=main.go:678 msg="Starting TSDB ..." 2021-03-15T22:18:14.789 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.483Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2021-03-15T22:18:14.790 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.483Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2021-03-15T22:18:14.790 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.483Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=120.074µs 2021-03-15T22:18:14.790 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.484Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2021-03-15T22:18:14.790 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.484Z caller=main.go:695 msg="TSDB started" 2021-03-15T22:18:14.790 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.484Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2021-03-15T22:18:14.791 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.485Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2021-03-15T22:18:14.791 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:18:14 smithi125 conmon[75467]: level=info ts=2021-03-15T22:18:14.485Z caller=main.go:646 msg="Server is ready to receive web requests." 2021-03-15T22:18:15.194 INFO:teuthology.orchestra.run.smithi076.stdout:[client.0] 2021-03-15T22:18:15.195 INFO:teuthology.orchestra.run.smithi076.stdout: key = AQAn3U9g2NUdCxAAyxDPUlcs+1XM1Qg/FAGzEg== 2021-03-15T22:18:15.449 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:15 smithi076 systemd[1]: Starting Ceph node-exporter.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:18:15.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[56514]: audit 2021-03-15T22:18:14.532721 2021-03-15T22:18:15.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[56514]: +0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:15.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:14.533700+0000 mgr.y (mgr.14144) 236 : cephadm [INF] Deploying daemon node-exporter.a on smithi076 2021-03-15T22:18:15.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[56514]: cluster 2021-03-15T22:18:14.947198+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-03-15T22:18:15.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[56514]: cluster 2021-03-15T22:18:14.953223+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2021-03-15T22:18:15.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[56514]: audit 2021-03-15T22:18:15.186345+0000 mon.a (mon.0) 499 : audit [INF] from='client.? 172.21.15.76:0/3040984191' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-15T22:18:15.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[56514]: audit 2021-03-15T22:18:15.192317+0000 mon.a (mon.0) 500 : audit [INF] from='client.? 172.21.15.76:0/3040984191' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-15T22:18:15.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[62156]: audit 2021-03-15T22:18:14.532721+0000 mon.a 2021-03-15T22:18:15.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[62156]: (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:15.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:14.533700+0000 mgr.y (mgr.14144) 236 : cephadm [INF] Deploying daemon node-exporter.a on smithi076 2021-03-15T22:18:15.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[62156]: cluster 2021-03-15T22:18:14.947198+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-03-15T22:18:15.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[62156]: cluster 2021-03-15T22:18:14.953223+0000 mon.a (mon.0) 498 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2021-03-15T22:18:15.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[62156]: audit 2021-03-15T22:18:15.186345+0000 mon.a (mon.0) 499 : audit [INF] from='client.? 172.21.15.76:0/3040984191' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-15T22:18:15.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:15 smithi076 conmon[62156]: audit 2021-03-15T22:18:15.192317+0000 mon.a (mon.0) 500 : audit [INF] from='client.? 172.21.15.76:0/3040984191' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-15T22:18:15.771 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:15 smithi076 bash[78891]: Trying to pull docker.io/prom/node-exporter:v0.18.1... 2021-03-15T22:18:15.798 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2021-03-15T22:18:15.798 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2021-03-15T22:18:15.799 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2021-03-15T22:18:15.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2021-03-15T22:18:15.870 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: audit 2021-03-15T22:18:14.532721+0000 mon.a (mon.0) 497 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:15.870 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:14.533700+0000 mgr.y (mgr.14144) 236 : cephadm [INF] Deploying daemon node-exporter.a on smithi076 2021-03-15T22:18:15.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: cluster 2021-03-15T22:18:14.947198+0000 mgr.y (mgr.14144) 237 : cluster [DBG] pgmap v185: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s 2021-03-15T22:18:15.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: cluster 2021-03-15T22:18:14.953223+0000 mon.a (mon.0 2021-03-15T22:18:15.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: ) 498 : cluster [DBG] osdmap e69: 8 total, 8 up, 8 in 2021-03-15T22:18:15.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:18:15.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: 15T22:18:15.186345+0000 mon.a (mon 2021-03-15T22:18:15.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: .0) 499 : audit [INF] from='client.? 172.21.15.76:0/3040984191' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-15T22:18:15.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:18:15.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: 03-15T22:18:15.192317+0000 2021-03-15T22:18:15.873 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: mon.a (mon.0) 500 : audit [INF] 2021-03-15T22:18:15.873 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:15 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/3040984191' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-15T22:18:16.066 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:15 smithi076 bash[78891]: Getting image source signatures 2021-03-15T22:18:16.067 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:15 smithi076 bash[78891]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-03-15T22:18:16.341 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:16 smithi076 bash[78891]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-03-15T22:18:16.342 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:16 smithi076 bash[78891]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-03-15T22:18:16.767 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:16 smithi076 bash[78891]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2021-03-15T22:18:16.767 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:16 smithi076 bash[78891]: Writing manifest to image destination 2021-03-15T22:18:16.767 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:16 smithi076 bash[78891]: Storing signatures 2021-03-15T22:18:17.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:17 smithi125 conmon[56739]: cluster 2021-03-15T22:18:16.947658+0000 mgr.y (mgr.14144) 238 : cluster [DBG] pgmap v187: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2021-03-15T22:18:17.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[56514]: cluster 2021-03-15T22:18:16.947658+0000 2021-03-15T22:18:17.421 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[56514]: mgr.y (mgr.14144) 238 : cluster [DBG] pgmap v187: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2021-03-15T22:18:17.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[62156]: cluster 2021-03-15T22:18:16.947658+0000 mgr.y 2021-03-15T22:18:17.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[62156]: (mgr.14144) 238 : cluster [DBG] pgmap v187: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2021-03-15T22:18:17.747 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 podman[78968]: 2021-03-15 22:18:17.419393848 +0000 UTC m=+2.163242231 image pull 2021-03-15T22:18:17.748 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 podman[78968]: 2021-03-15 22:18:17.531928974 +0000 UTC m=+2.275777472 container create 235b466aaad3e2382437593404a8626aa93ebc6f6391ca41588729aff2300c10 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-node-exporter.a, maintainer=The Prometheus Authors ) 2021-03-15T22:18:17.935 INFO:teuthology.orchestra.run.smithi125.stdout:[client.1] 2021-03-15T22:18:17.936 INFO:teuthology.orchestra.run.smithi125.stdout: key = AQAp3U9g/yZ2NxAAqWB8RmspFuPb0cx6IqN3GQ== 2021-03-15T22:18:18.267 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 podman[78968]: 2021-03-15 22:18:17.773639235 +0000 UTC m=+2.517487719 container init 235b466aaad3e2382437593404a8626aa93ebc6f6391ca41588729aff2300c10 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-node-exporter.a, maintainer=The Prometheus Authors ) 2021-03-15T22:18:18.268 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2021-03-15T22:18:18.268 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2021-03-15T22:18:18.268 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2021-03-15T22:18:18.268 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - arp" source="node_exporter.go:104" 2021-03-15T22:18:18.269 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - bcache" source="node_exporter.go:104" 2021-03-15T22:18:18.269 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - bonding" source="node_exporter.go:104" 2021-03-15T22:18:18.269 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2021-03-15T22:18:18.269 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - cpu" source="node_exporter.go:104" 2021-03-15T22:18:18.269 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2021-03-15T22:18:18.270 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2021-03-15T22:18:18.270 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - edac" source="node_exporter.go:104" 2021-03-15T22:18:18.270 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - entropy" source="node_exporter.go:104" 2021-03-15T22:18:18.270 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - filefd" source="node_exporter.go:104" 2021-03-15T22:18:18.271 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2021-03-15T22:18:18.271 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2021-03-15T22:18:18.271 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2021-03-15T22:18:18.271 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2021-03-15T22:18:18.271 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2021-03-15T22:18:18.272 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2021-03-15T22:18:18.272 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2021-03-15T22:18:18.272 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - netclass" source="node_exporter.go:104" 2021-03-15T22:18:18.272 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - netdev" source="node_exporter.go:104" 2021-03-15T22:18:18.272 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - netstat" source="node_exporter.go:104" 2021-03-15T22:18:18.273 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - nfs" source="node_exporter.go:104" 2021-03-15T22:18:18.273 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2021-03-15T22:18:18.273 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - pressure" source="node_exporter.go:104" 2021-03-15T22:18:18.273 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2021-03-15T22:18:18.274 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - stat" source="node_exporter.go:104" 2021-03-15T22:18:18.274 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - textfile" source="node_exporter.go:104" 2021-03-15T22:18:18.274 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - time" source="node_exporter.go:104" 2021-03-15T22:18:18.275 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - uname" source="node_exporter.go:104" 2021-03-15T22:18:18.275 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2021-03-15T22:18:18.275 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - xfs" source="node_exporter.go:104" 2021-03-15T22:18:18.275 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg=" - zfs" source="node_exporter.go:104" 2021-03-15T22:18:18.276 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 conmon[79108]: time="2021-03-15T22:18:17Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2021-03-15T22:18:18.276 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 podman[78968]: 2021-03-15 22:18:17.806849762 +0000 UTC m=+2.550698141 container start 235b466aaad3e2382437593404a8626aa93ebc6f6391ca41588729aff2300c10 (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-node-exporter.a, maintainer=The Prometheus Authors ) 2021-03-15T22:18:18.276 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 bash[78891]: 235b466aaad3e2382437593404a8626aa93ebc6f6391ca41588729aff2300c10 2021-03-15T22:18:18.276 INFO:journalctl@ceph.node-exporter.a.smithi076.stdout:Mar 15 22:18:17 smithi076 systemd[1]: Started Ceph node-exporter.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:18:18.609 DEBUG:teuthology.orchestra.run.smithi125:> set -ex 2021-03-15T22:18:18.610 DEBUG:teuthology.orchestra.run.smithi125:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2021-03-15T22:18:18.610 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2021-03-15T22:18:18.777 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump --format=json 2021-03-15T22:18:19.072 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:19.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[56514]: audit 2021-03-15T22:18:17.864983+0000 mon.a (mon.0) 2021-03-15T22:18:19.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[56514]: 501 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:19.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:17.865630+0000 mgr.y (mgr.14144) 239 : cephadm [INF] Deploying daemon node-exporter.b on smithi125 2021-03-15T22:18:19.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[56514]: audit 2021-03-15T22:18:17.929749+0000 mon.b (mon.2) 25 : audit [INF] from='client.? 172.21.15.125:0/1448054198' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-15T22:18:19.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[56514]: audit 2021-03-15T22:18:17.930355+0000 mon.a (mon.0) 502 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-15T22:18:19.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:18:19.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[56514]: 15T22:18:17.933266+0000 mon.a (mon.0) 503 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-15T22:18:19.243 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: audit 2021-03-15T22:18:17.864983+0000 mon.a 2021-03-15T22:18:19.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: (mon.0) 501 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:19.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:19.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: 2021-03-15T22:18:17.865630+0000 mgr.y (mgr.14144) 239 : cephadm [INF] 2021-03-15T22:18:19.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: Deploying daemon node-exporter.b on smithi125 2021-03-15T22:18:19.245 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:18:19.245 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: T22:18:17.929749+0000 mon.b (mon.2) 2021-03-15T22:18:19.245 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: 25 : audit [INF] from='client.? 172.21.15.125:0/1448054198' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-15T22:18:19.245 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: audit 2021-03-15T22:18:19.246 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: 2021-03-15T22:18:17.930355 2021-03-15T22:18:19.246 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: +0000 mon.a (mon.0) 502 : audit [INF] 2021-03-15T22:18:19.246 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-15T22:18:19.246 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:18:19.246 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: T22:18:17.933266+0000 mon.a (mon.0 2021-03-15T22:18:19.247 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:18 smithi076 conmon[62156]: ) 503 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-15T22:18:19.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:18 smithi125 conmon[56739]: audit 2021-03-15T22:18:17.864983+0000 mon.a (mon.0) 501 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:19.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:18 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:17.865630+0000 mgr.y (mgr.14144) 239 : cephadm [INF] Deploying daemon node-exporter.b on smithi125 2021-03-15T22:18:19.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:18 smithi125 conmon[56739]: audit 2021-03-15T22:18:17.929749+0000 mon.b (mon.2) 25 : audit [INF] from='client.? 172.21.15.125:0/1448054198' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-15T22:18:19.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:18 smithi125 conmon[56739]: audit 2021-03-15T22:18:17.930355+0000 mon.a (mon.0) 502 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2021-03-15T22:18:19.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:18 smithi125 conmon[56739]: audit 2021-03-15T22:18:17.933266+0000 mon.a (mon.0) 503 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2021-03-15T22:18:19.290 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:18 smithi125 systemd[1]: Starting Ceph node-exporter.b for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:18:19.291 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:19 smithi125 bash[75862]: Trying to pull docker.io/prom/node-exporter:v0.18.1... 2021-03-15T22:18:19.577 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:18:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:18:19.578 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:19 smithi125 bash[75862]: Getting image source signatures 2021-03-15T22:18:19.578 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:19 smithi125 bash[75862]: Copying blob sha256:190160031744b0810999dd6f293478be0b41524654c70782cf09437095430f2f 2021-03-15T22:18:19.578 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:19 smithi125 bash[75862]: Copying blob sha256:3589a6efd9ce96b2f2218a80bacb84f60d5e584d024023617b0f56b87fe798c3 2021-03-15T22:18:19.870 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:19 smithi125 bash[75862]: Copying blob sha256:49a2d53aa1af115e87f7e23a7fe4b7a3e12304d37982fb6528b035d0f68a3b56 2021-03-15T22:18:19.871 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:19 smithi076 conmon[62156]: cluster 2021-03-15T22:18:18.948249+0000 mgr.y (mgr 2021-03-15T22:18:19.871 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:19 smithi076 conmon[62156]: .14144) 240 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:20.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:19 smithi076 conmon[56514]: cluster 2021-03-15T22:18:18.948249+0000 mgr.y (mgr. 2021-03-15T22:18:20.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:19 smithi076 conmon[56514]: 14144) 240 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:20.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:19 smithi125 conmon[56739]: cluster 2021-03-15T22:18:18.948249+0000 mgr.y (mgr.14144) 240 : cluster [DBG] pgmap v188: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:20.287 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:19 smithi125 bash[75862]: Copying config sha256:e5a616e4b9cf68dfcad7782b78e118be4310022e874d52da85c55923fb615f87 2021-03-15T22:18:20.287 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:20 smithi125 bash[75862]: Writing manifest to image destination 2021-03-15T22:18:20.287 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:20 smithi125 bash[75862]: Storing signatures 2021-03-15T22:18:20.937 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:18:20.938 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":70,"fsid":"8a572802-85db-11eb-9091-001a4aab830c","created":"2021-03-15T22:13:20.370795+0000","modified":"2021-03-15T22:18:19.949065+0000","last_up_change":"2021-03-15T22:17:49.162889+0000","last_in_change":"2021-03-15T22:17:34.080423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-15T22:16:34.817191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"iscsi","create_time":"2021-03-15T22:17:49.906400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2021-03-15T22:17:52.257970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2021-03-15T22:17:54.562498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2021-03-15T22:17:56.243806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2021-03-15T22:17:58.730534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6803","nonce":188228176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6805","nonce":188228176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6809","nonce":188228176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6807","nonce":188228176}]},"public_addr":"172.21.15.76:6803/188228176","cluster_addr":"172.21.15.76:6805/188228176","heartbeat_back_addr":"172.21.15.76:6809/188228176","heartbeat_front_addr":"172.21.15.76:6807/188228176","state":["exists","up"]},{"osd":1,"uuid":"97bd3ffc-15c8-4a6f-b37a-217813def526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6811","nonce":3924955423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6813","nonce":3924955423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6817","nonce":3924955423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6815","nonce":3924955423}]},"public_addr":"172.21.15.76:6811/3924955423","cluster_addr":"172.21.15.76:6813/3924955423","heartbeat_back_addr":"172.21.15.76:6817/3924955423","heartbeat_front_addr":"172.21.15.76:6815/3924955423","state":["exists","up"]},{"osd":2,"uuid":"a7f5f115-e69e-47d9-b6c3-ff386ddfaa05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6819","nonce":1401263050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6821","nonce":1401263050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6825","nonce":1401263050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6823","nonce":1401263050}]},"public_addr":"172.21.15.76:6819/1401263050","cluster_addr":"172.21.15.76:6821/1401263050","heartbeat_back_addr":"172.21.15.76:6825/1401263050","heartbeat_front_addr":"172.21.15.76:6823/1401263050","state":["exists","up"]},{"osd":3,"uuid":"a0bb7f88-0114-47cb-a82c-d8fb002aac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6827","nonce":3960968636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6829","nonce":3960968636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6833","nonce":3960968636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6831","nonce":3960968636}]},"public_addr":"172.21.15.76:6827/3960968636","cluster_addr":"172.21.15.76:6829/3960968636","heartbeat_back_addr":"172.21.15.76:6833/3960968636","heartbeat_front_addr":"172.21.15.76:6831/3960968636","state":["exists","up"]},{"osd":4,"uuid":"b650611c-6101-47e8-9ca9-94e02cf8b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6801","nonce":37386122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6803","nonce":37386122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6807","nonce":37386122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6805","nonce":37386122}]},"public_addr":"172.21.15.125:6801/37386122","cluster_addr":"172.21.15.125:6803/37386122","heartbeat_back_addr":"172.21.15.125:6807/37386122","heartbeat_front_addr":"172.21.15.125:6805/37386122","state":["exists","up"]},{"osd":5,"uuid":"7d6ceff2-7093-4ab9-8b96-ade448093bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6809","nonce":3610838738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6811","nonce":3610838738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6815","nonce":3610838738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6813","nonce":3610838738}]},"public_addr":"172.21.15.125:6809/3610838738","cluster_addr":"172.21.15.125:6811/3610838738","heartbeat_back_addr":"172.21.15.125:6815/3610838738","heartbeat_front_addr":"172.21.15.125:6813/3610838738","state":["exists","up"]},{"osd":6,"uuid":"98b72022-a9a7-4ee2-89a1-cac037de7afe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6817","nonce":171378275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6819","nonce":171378275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6823","nonce":171378275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6821","nonce":171378275}]},"public_addr":"172.21.15.125:6817/171378275","cluster_addr":"172.21.15.125:6819/171378275","heartbeat_back_addr":"172.21.15.125:6823/171378275","heartbeat_front_addr":"172.21.15.125:6821/171378275","state":["exists","up"]},{"osd":7,"uuid":"f99ea650-b2dc-4d87-a44c-56a897edd614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6825","nonce":652003948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6827","nonce":652003948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6831","nonce":652003948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6829","nonce":652003948}]},"public_addr":"172.21.15.125:6825/652003948","cluster_addr":"172.21.15.125:6827/652003948","heartbeat_back_addr":"172.21.15.125:6831/652003948","heartbeat_front_addr":"172.21.15.125:6829/652003948","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:17:49.279866+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:07.166571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:17:22.876068+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:17:38.835880+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:17:55.480481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:12.704364+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:17:29.507887+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:17:47.645171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6800/2604045245":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/4087324200":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/4141143304":"2021-03-16T22:14:00.771702+0000","172.21.15.76:6801/2604045245":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/2779879991":"2021-03-16T22:13:41.603652+0000","172.21.15.76:6801/2764179881":"2021-03-16T22:13:41.603652+0000","172.21.15.76:6800/2764179881":"2021-03-16T22:13:41.603652+0000","172.21.15.76:0/2483102204":"2021-03-16T22:13:41.603652+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-15T22:18:21.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:20 smithi076 conmon[56514]: audit 2021-03-15T22:18:19.303500+0000 mgr.y (mgr.14144) 241 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:21.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:20 smithi076 conmon[56514]: cluster 2021-03-15T22:18:19.953059+0000 mon.a (mon.0) 504 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-03-15T22:18:21.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:20 smithi076 conmon[56514]: audit 2021-03-15T22:18:20.935203+0000 mon.a (mon.0) 505 : audit [DBG] from='client.? 172.21.15.76:0/2212854333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:18:21.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:20 smithi076 conmon[62156]: audit 2021-03-15T22:18:19.303500+0000 mgr.y (mgr.14144) 241 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:21.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:20 smithi076 conmon[62156]: cluster 2021-03-15T22:18:19.953059+0000 mon.a (mon.0) 504 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-03-15T22:18:21.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:20 smithi076 conmon[62156]: audit 2021-03-15T22:18:20.935203+0000 mon.a (mon.0) 505 : audit [DBG] from='client.? 172.21.15.76:0/2212854333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:18:21.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:20 smithi125 conmon[56739]: audit 2021-03-15T22:18:19.303500+0000 mgr.y (mgr.14144) 241 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:21.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:20 smithi125 conmon[56739]: cluster 2021-03-15T22:18:19.953059+0000 mon.a (mon.0) 504 : cluster [DBG] osdmap e70: 8 total, 8 up, 8 in 2021-03-15T22:18:21.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:20 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:18:21.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:20 smithi125 conmon[56739]: 15T22:18:20.935203+0000 mon.a (mon.0) 505 : audit [DBG] 2021-03-15T22:18:21.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:20 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2212854333' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:18:21.287 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:20 smithi125 podman[75936]: 2021-03-15 22:18:20.989424037 +0000 UTC m=+1.987385927 image pull 2021-03-15T22:18:21.288 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 podman[75936]: 2021-03-15 22:18:21.125972857 +0000 UTC m=+2.123934740 container create f0e52f22ab74421fadbab502f0b9cca1562f10b4983bd6c1a22c96f52a005a0e (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-node-exporter.b, maintainer=The Prometheus Authors ) 2021-03-15T22:18:21.516 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-15T22:16:34.817191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-15T22:17:49.906400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-15T22:17:52.257970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-15T22:17:54.562498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-15T22:17:56.243806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '62', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-15T22:17:58.730534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '65', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}] 2021-03-15T22:18:21.517 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool get device_health_metrics pg_num 2021-03-15T22:18:21.786 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 podman[75936]: 2021-03-15 22:18:21.37593681 +0000 UTC m=+2.373898798 container init f0e52f22ab74421fadbab502f0b9cca1562f10b4983bd6c1a22c96f52a005a0e (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-node-exporter.b, maintainer=The Prometheus Authors ) 2021-03-15T22:18:21.787 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg="Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)" source="node_exporter.go:156" 2021-03-15T22:18:21.787 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg="Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)" source="node_exporter.go:157" 2021-03-15T22:18:21.787 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg="Enabled collectors:" source="node_exporter.go:97" 2021-03-15T22:18:21.787 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - arp" source="node_exporter.go:104" 2021-03-15T22:18:21.788 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - bcache" source="node_exporter.go:104" 2021-03-15T22:18:21.788 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - bonding" source="node_exporter.go:104" 2021-03-15T22:18:21.788 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - conntrack" source="node_exporter.go:104" 2021-03-15T22:18:21.788 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - cpu" source="node_exporter.go:104" 2021-03-15T22:18:21.789 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - cpufreq" source="node_exporter.go:104" 2021-03-15T22:18:21.789 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - diskstats" source="node_exporter.go:104" 2021-03-15T22:18:21.789 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - edac" source="node_exporter.go:104" 2021-03-15T22:18:21.789 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - entropy" source="node_exporter.go:104" 2021-03-15T22:18:21.789 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - filefd" source="node_exporter.go:104" 2021-03-15T22:18:21.790 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - filesystem" source="node_exporter.go:104" 2021-03-15T22:18:21.790 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - hwmon" source="node_exporter.go:104" 2021-03-15T22:18:21.790 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - infiniband" source="node_exporter.go:104" 2021-03-15T22:18:21.790 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - ipvs" source="node_exporter.go:104" 2021-03-15T22:18:21.791 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - loadavg" source="node_exporter.go:104" 2021-03-15T22:18:21.791 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - mdadm" source="node_exporter.go:104" 2021-03-15T22:18:21.791 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - meminfo" source="node_exporter.go:104" 2021-03-15T22:18:21.791 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - netclass" source="node_exporter.go:104" 2021-03-15T22:18:21.791 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - netdev" source="node_exporter.go:104" 2021-03-15T22:18:21.792 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - netstat" source="node_exporter.go:104" 2021-03-15T22:18:21.792 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - nfs" source="node_exporter.go:104" 2021-03-15T22:18:21.792 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - nfsd" source="node_exporter.go:104" 2021-03-15T22:18:21.792 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - pressure" source="node_exporter.go:104" 2021-03-15T22:18:21.793 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - sockstat" source="node_exporter.go:104" 2021-03-15T22:18:21.793 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - stat" source="node_exporter.go:104" 2021-03-15T22:18:21.793 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - textfile" source="node_exporter.go:104" 2021-03-15T22:18:21.793 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - time" source="node_exporter.go:104" 2021-03-15T22:18:21.793 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - uname" source="node_exporter.go:104" 2021-03-15T22:18:21.794 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - vmstat" source="node_exporter.go:104" 2021-03-15T22:18:21.794 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - xfs" source="node_exporter.go:104" 2021-03-15T22:18:21.794 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg=" - zfs" source="node_exporter.go:104" 2021-03-15T22:18:21.794 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[76011]: time="2021-03-15T22:18:21Z" level=info msg="Listening on :9100" source="node_exporter.go:170" 2021-03-15T22:18:21.795 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 podman[75936]: 2021-03-15 22:18:21.417552787 +0000 UTC m=+2.415514759 container start f0e52f22ab74421fadbab502f0b9cca1562f10b4983bd6c1a22c96f52a005a0e (image=docker.io/prom/node-exporter:v0.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-node-exporter.b, maintainer=The Prometheus Authors ) 2021-03-15T22:18:21.795 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 bash[75862]: f0e52f22ab74421fadbab502f0b9cca1562f10b4983bd6c1a22c96f52a005a0e 2021-03-15T22:18:21.795 INFO:journalctl@ceph.node-exporter.b.smithi125.stdout:Mar 15 22:18:21 smithi125 systemd[1]: Started Ceph node-exporter.b for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:18:21.813 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:22.232 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:21 smithi076 conmon[56514]: cluster 2021-03-15T22:18:20.948813+0000 2021-03-15T22:18:22.232 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:21 smithi076 conmon[56514]: mgr.y (mgr.14144) 242 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-03-15T22:18:22.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:21 smithi076 conmon[56514]: audit 2021-03-15T22:18:21.474681+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:22.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:21 smithi076 conmon[56514]: audit 2021-03-15T22:18:21.475297+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:18:22.233 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:21 smithi076 conmon[62156]: cluster 2021-03-15T22:18:20.948813 2021-03-15T22:18:22.234 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:21 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 242 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-03-15T22:18:22.234 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:21 smithi076 conmon[62156]: audit 2021-03-15T22:18:21.474681+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:22.234 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:21 smithi076 conmon[62156]: audit 2021-03-15T22:18:21.475297+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:18:22.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[56739]: cluster 2021-03-15T22:18:20.948813+0000 mgr.y (mgr.14144) 2021-03-15T22:18:22.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[56739]: 242 : cluster [DBG] pgmap v190: 132 pgs: 132 active+clean; 25 KiB data, 50 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 1 op/s 2021-03-15T22:18:22.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[56739]: audit 2021-03-15T22:18:21.474681+0000 mon.a (mon.0) 506 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:22.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:21 smithi125 conmon[56739]: audit 2021-03-15T22:18:21.475297+0000 mon.a (mon.0) 507 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:18:23.541 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 1 2021-03-15T22:18:23.968 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:23 smithi076 conmon[62156]: cluster 2021-03-15T22:18:22.949311+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:23.969 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:23 smithi076 conmon[62156]: cluster 2021-03-15T22:18:23.085398+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:23.969 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:23 smithi076 conmon[62156]: cluster 2021-03-15T22:18:23.085492+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:23.969 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:23 smithi076 conmon[62156]: audit 2021-03-15T22:18:23.539706+0000 mon.a (mon.0) 508 : audit [DBG] from='client.? 172.21.15.76:0/4126595915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-03-15T22:18:24.128 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool get iscsi pg_num 2021-03-15T22:18:24.239 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:23 smithi076 conmon[56514]: cluster 2021-03-15T22:18:22.949311+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:24.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:23 smithi076 conmon[56514]: cluster 2021-03-15T22:18:23.085398+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:24.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:23 smithi076 conmon[56514]: cluster 2021-03-15T22:18:23.085492+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:24.240 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:23 smithi076 conmon[56514]: audit 2021-03-15T22:18:23.539706+0000 mon.a (mon.0) 508 : audit [DBG] from='client.? 172.21.15.76:0/4126595915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-03-15T22:18:24.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:23 smithi125 conmon[56739]: cluster 2021-03-15T22:18:22.949311+0000 mgr.y (mgr.14144) 243 : cluster [DBG] pgmap v191: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:24.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:23 smithi125 conmon[56739]: cluster 2021-03-15T22:18:23.085398+0000 osd.2 (osd.2) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:24.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:23 smithi125 conmon[56739]: cluster 2021-03-15T22:18:23.085492+0000 osd.2 (osd.2) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:24.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:23 smithi125 conmon[56739]: audit 2021-03-15T22:18:23.539706+0000 mon.a (mon.0) 508 : audit [DBG] from='client.? 172.21.15.76:0/4126595915' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]: dispatch 2021-03-15T22:18:24.472 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:24.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: debug 2021-03-15T22:18:24.443+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:18:24.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:24.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:24.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:24.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:24.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:24.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:24.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:24.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:18:24.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: debug 2021-03-15T22:18:24.443+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:24.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:24.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:24.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:24.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:24.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:24.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:24.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:24.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:24.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: debug 2021-03-15T22:18:24.444+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:24.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:24.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:24.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:24.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:24.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:24.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:24.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:24.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:24.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: debug 2021-03-15T22:18:24.444+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:18:24.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:24.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:24.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:24.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:24.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:24.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:24.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:24.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:18:24.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: debug 2021-03-15T22:18:24.444+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:18:24.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:24.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:24.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:24.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:24.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:24.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:24.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:24.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:18:25.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56514]: cluster 2021-03-15T22:18:23.963006+0000 mon.a (mon.0) 509 2021-03-15T22:18:25.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56514]: : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-03-15T22:18:25.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56514]: audit 2021-03-15T22:18:24.063088+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:25.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56514]: audit 2021-03-15T22:18:24.437929+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:25.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56514]: audit 2021-03-15T22:18:24.443030+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:25.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[56514]: cluster 2021-03-15T22:18:24.956231+0000 mon.a (mon.0) 513 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2021-03-15T22:18:25.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[62156]: cluster 2021-03-15T22:18:23.963006+0000 mon.a (mon. 2021-03-15T22:18:25.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[62156]: 0) 509 : cluster [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-03-15T22:18:25.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[62156]: audit 2021-03-15T22:18:24.063088+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:25.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[62156]: audit 2021-03-15T22:18:24.437929+0000 mon.a (mon.0) 511 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:25.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[62156]: audit 2021-03-15T22:18:24. 2021-03-15T22:18:25.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[62156]: 443030+0000 mon.a (mon.0) 512 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:25.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[62156]: cluster 2021-03-15T22:18:24. 2021-03-15T22:18:25.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:24 smithi076 conmon[62156]: 956231+0000 mon.a (mon.0) 513 : cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2021-03-15T22:18:25.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: cluster 2021-03-15T22:18:23.963006+0000 mon.a (mon.0) 509 : cluster 2021-03-15T22:18:25.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: [DBG] osdmap e71: 8 total, 8 up, 8 in 2021-03-15T22:18:25.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: audit 2021-03-15T22:18:24.063088+0000 mon.a (mon.0) 510 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:25.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: audit 2021-03-15T22:18: 2021-03-15T22:18:25.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: 24.437929+0000 mon.a (mon.0) 511 2021-03-15T22:18:25.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:25.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: audit 2021-03-15T22:18: 2021-03-15T22:18:25.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: 24.443030+0000 mon.a (mon.0) 512 2021-03-15T22:18:25.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:25.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:18:25.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: -03-15T22:18:24 2021-03-15T22:18:25.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: .956231+0000 mon.a (mon.0) 513 : 2021-03-15T22:18:25.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:24 smithi125 conmon[56739]: cluster [DBG] osdmap e72: 8 total, 8 up, 8 in 2021-03-15T22:18:26.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:26.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: 2021-03-15T22:18:26.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: 2021 2021-03-15T22:18:26.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: -03-15T22: 2021-03-15T22:18:26.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: 18:24.438556+0000 mgr.y (mgr.14144) 244 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:18:26.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:18:26.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: 03-15T22:18:24.443916+0000 mgr.y (mgr.14144 2021-03-15T22:18:26.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: ) 245 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:18:26.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:26.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:26.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:26.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:18:26.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:24.444351+0000 mgr.y (mgr.14144) 246 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:26.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:26.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:26.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:26.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:26.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: cephadm 2021-03-15T22:18 2021-03-15T22:18:26.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: :24.444735+0000 mgr.y (mgr.14144) 247 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:26.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:26.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:26.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:24.438556+0000 mgr.y (mgr.14144) 244 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:18:26.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:24.443916+0000 mgr.y (mgr.14144) 245 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:18:26.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:26.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.028 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:26.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:26.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.029 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:18:26.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:24.444351+0000 mgr.y (mgr.14144) 246 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:26.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:26.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.030 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:26.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:26.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:26.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:24.444735+0000 mgr.y (mgr.14144) 247 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:26.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:26.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:26.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:26.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:26.034 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:26.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:24.445099+0000 mgr.y (mgr.14144) 248 : cephadm 2021-03-15T22:18:26.035 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:18:26.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:26.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:26.036 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.037 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:26.037 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.037 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.037 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:18:26.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:24.445480+0000 mgr.y (mgr.14144) 249 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:18:26.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:26.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:26.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:26.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:18:26.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:24.449639+0000 mgr.y (mgr.14144) 250 : cephadm [INF] Deploying daemon alertmanager.a on smithi076 2021-03-15T22:18:26.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[62156]: cluster 2021-03-15T22:18:24.949732+0000 mgr.y (mgr.14144) 251 : cluster [DBG] pgmap v193: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:26.040 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:26.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:24.445099+0000 mgr.y (mgr.14144) 248 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:18:26.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:26.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.041 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:26.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:26.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.042 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:18:26.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:24. 2021-03-15T22:18:26.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: 445480+0000 mgr.y (mgr.14144) 249 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:18:26.043 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:26.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:26.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.044 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:26.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:18:26.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:24.449639+0000 mgr.y (mgr.14144) 250 : cephadm [INF] Deploying daemon alertmanager.a on smithi076 2021-03-15T22:18:26.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:25 smithi076 conmon[56514]: cluster 2021-03-15T22:18:24.949732+0000 mgr.y (mgr.14144) 251 : cluster [DBG] pgmap v193: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:26.095 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 3 2021-03-15T22:18:26.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:24.438556+0000 mgr.y (mgr.14144) 244 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:18:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: 2021-03-15T22:18:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:24.443916+0000 mgr.y (mgr.14144) 245 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:18:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:26.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:18:26.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:24.444351+0000 2021-03-15T22:18:26.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: mgr.y (mgr.14144) 246 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:26.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:26.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:24.444735+0000 mgr.y (mgr.14144) 247 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:26.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:26.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:26.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:26.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:26.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:24.445099+0000 mgr.y (mgr.14144) 248 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:18:26.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:26.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:26.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:26.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:18:26.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:24.445480+0000 mgr.y (mgr. 2021-03-15T22:18:26.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: 14144) 249 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:18:26.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:26.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:26.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:26.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:26.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:26.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:26.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:26.300 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:18:26.300 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:24.449639+0000 mgr.y (mgr.14144 2021-03-15T22:18:26.300 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: ) 250 : cephadm [INF] Deploying daemon alertmanager.a on smithi076 2021-03-15T22:18:26.301 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: cluster 2021-03-15T22:18:24.949732+0000 mgr.y (mgr. 2021-03-15T22:18:26.301 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:25 smithi125 conmon[56739]: 14144) 251 : cluster [DBG] pgmap v193: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:26 smithi076 conmon[56514]: audit 2021-03-15 2021-03-15T22:18:27.303 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:26 smithi076 conmon[56514]: T22:18:26.094470+0000 mon.a (mon.0) 514 : audit [DBG] from='client.? 172.21.15.76:0/3424414392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2021-03-15T22:18:27.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:26 smithi076 conmon[62156]: audit 2021-03-15T22:18:26.094470+0000 mon.a ( 2021-03-15T22:18:27.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:26 smithi076 conmon[62156]: mon.0) 514 : audit [DBG] from='client.? 172.21.15.76:0/3424414392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2021-03-15T22:18:27.305 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:26 smithi125 conmon[56739]: audit 2021-03-15T22:18:26.094470+0000 mon.a (mon.0) 514 : audit [DBG] from='client.? 172.21.15.76:0/3424414392' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "iscsi", "var": "pg_num"}]: dispatch 2021-03-15T22:18:27.305 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:26 smithi125 conmon[56739]: 2021-03-15T22:18:27.329 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool get .rgw.root pg_num 2021-03-15T22:18:27.958 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:28.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:27 smithi076 conmon[56514]: cluster 2021-03-15T22:18:26.950339+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v195: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2021-03-15T22:18:28.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:27 smithi076 conmon[62156]: cluster 2021-03-15T22:18:26 2021-03-15T22:18:28.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:27 smithi076 conmon[62156]: .950339+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v195: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2021-03-15T22:18:28.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:27 smithi125 conmon[56739]: cluster 2021-03-15T22:18:26.950339+0000 mgr.y (mgr.14144) 252 : cluster [DBG] pgmap v195: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s 2021-03-15T22:18:28.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:27 smithi125 conmon[56739]: 2021-03-15T22:18:29.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:28 smithi076 conmon[56514]: cluster 2021-03-15T22:18:28.951104+0000 mgr.y ( 2021-03-15T22:18:29.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:28 smithi076 conmon[56514]: mgr.14144) 253 : cluster [DBG] pgmap v196: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:29.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:28 smithi076 conmon[62156]: cluster 2021-03-15T22:18:28.951104+0000 2021-03-15T22:18:29.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:28 smithi076 conmon[62156]: mgr.y (mgr.14144) 253 : cluster [DBG] pgmap v196: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:29.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:28 smithi125 conmon[56739]: cluster 2021-03-15T22:18:28.951104+0000 mgr.y (mgr.14144) 253 : cluster 2021-03-15T22:18:29.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:28 smithi125 conmon[56739]: [DBG] pgmap v196: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:29.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:18:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:18:31.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:30 smithi076 conmon[56514]: audit 2021-03-15T22:18:29.312132+0000 mgr.y (mgr.14144) 254 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:31.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:30 smithi076 conmon[56514]: cluster 2021-03-15T22:18:30.283900+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:31.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:30 smithi076 conmon[56514]: cluster 2021-03-15T22:18:30.284117+0000 osd.6 (osd.6) 5 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:31.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:30 smithi076 conmon[62156]: audit 2021-03-15T22:18:29.312132 2021-03-15T22:18:31.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:30 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 254 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:31.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:30 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:18:31.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:30 smithi076 conmon[62156]: -03-15T22:18:30.283900+0000 osd.6 (osd.6) 4 : cluster [DBG] 2021-03-15T22:18:31.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:30 smithi076 conmon[62156]: purged_snaps scrub starts 2021-03-15T22:18:31.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:30 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:18:31.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:30 smithi076 conmon[62156]: 22:18:30.284117+0000 osd.6 (osd.6 2021-03-15T22:18:31.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:30 smithi076 conmon[62156]: ) 5 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:31.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:30 smithi125 conmon[56739]: audit 2021-03-15T22:18:29.312132+0000 mgr.y (mgr.14144 2021-03-15T22:18:31.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:30 smithi125 conmon[56739]: ) 254 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:31.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:30 smithi125 conmon[56739]: cluster 2021-03-15T22:18:30.283900+0000 osd.6 (osd.6) 4 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:31.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:30 smithi125 conmon[56739]: cluster 2021-03-15T22:18:30.284117+0000 osd.6 (osd.6) 5 : 2021-03-15T22:18:31.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:30 smithi125 conmon[56739]: cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:31.917 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2021-03-15T22:18:32.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[56514]: cluster 2021-03-15T22:18:30.951531+0000 2021-03-15T22:18:32.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[56514]: mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v197: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:32.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[56514]: cluster 2021-03-15T22:18:30.963668+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-03-15T22:18:32.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[56514]: audit 2021-03-15T22:18:31.915547+0000 mon.a (mon.0) 516 : audit [DBG] from='client.? 172.21.15.76:0/3068333556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-03-15T22:18:32.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[62156]: cluster 2021-03-15T22:18: 2021-03-15T22:18:32.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[62156]: 30.951531+0000 mgr.y (mgr.14144) 255 : cluster [DBG] pgmap v197: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:32.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[62156]: cluster 2021-03-15T22:18:32.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[62156]: 2021-03-15T22:18:30.963668+0000 2021-03-15T22:18:32.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[62156]: mon.a (mon.0) 515 : 2021-03-15T22:18:32.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[62156]: cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-03-15T22:18:32.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:18:32.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[62156]: -15T22:18:31.915547+0000 mon.a (mon.0) 2021-03-15T22:18:32.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:31 smithi076 conmon[62156]: 516 : audit [DBG] from='client.? 172.21.15.76:0/3068333556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-03-15T22:18:32.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:31 smithi125 conmon[56739]: cluster 2021-03-15T22:18:30.951531+0000 mgr.y (mgr.14144 2021-03-15T22:18:32.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:31 smithi125 conmon[56739]: ) 255 : cluster [DBG] pgmap v197: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:32.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:31 smithi125 conmon[56739]: cluster 2021-03-15T22:18:30.963668+0000 mon.a (mon.0) 515 : cluster [DBG] osdmap e73: 8 total, 8 up, 8 in 2021-03-15T22:18:32.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:31 smithi125 conmon[56739]: audit 2021-03-15T22:18:31.915547+0000 mon.a ( 2021-03-15T22:18:32.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:31 smithi125 conmon[56739]: mon.0) 516 : audit [DBG] from='client.? 172.21.15.76:0/3068333556' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".rgw.root", "var": "pg_num"}]: dispatch 2021-03-15T22:18:33.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool get default.rgw.log pg_num 2021-03-15T22:18:33.431 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:33.699 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:33 smithi076 systemd[1]: Starting Ceph alertmanager.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:18:33.964 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:33 smithi076 podman[81400]: 2021-03-15 22:18:33.896450282 +0000 UTC m=+0.148422652 volume create 449ecb7136938e2a61d5eb3d296efd97fcff415fdc2720c9e35f0ef5d659d3ee 2021-03-15T22:18:33.964 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:33 smithi076 podman[81400]: 2021-03-15 22:18:33.921373372 +0000 UTC m=+0.173345725 container create 979ec48afd0e4720782d746f9e0da009d1b6e60e482841b43fa462157861ad8e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-15T22:18:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:33 smithi076 conmon[56514]: cluster 2021-03-15T22:18:32.952068+0000 mgr.y (mgr. 2021-03-15T22:18:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:33 smithi076 conmon[56514]: 14144) 256 : cluster [DBG] pgmap v199: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:34.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:33 smithi076 conmon[62156]: cluster 2021-03-15T22:18:32.952068+0000 mgr.y (mgr.14144) 256 : cluster [DBG] pgmap v199: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:34.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:33 smithi125 conmon[56739]: cluster 2021-03-15T22:18:32.952068+0000 mgr.y (mgr.14144) 256 : cluster 2021-03-15T22:18:34.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:33 smithi125 conmon[56739]: [DBG] pgmap v199: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:34.889 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:34 smithi076 podman[81400]: 2021-03-15 22:18:34.626264875 +0000 UTC m=+0.878237334 container init 979ec48afd0e4720782d746f9e0da009d1b6e60e482841b43fa462157861ad8e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-15T22:18:34.890 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:34 smithi076 conmon[81441]: level=info ts=2021-03-15T22:18:34.680Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2021-03-15T22:18:34.890 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:34 smithi076 conmon[81441]: level=info ts=2021-03-15T22:18:34.680Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2021-03-15T22:18:34.890 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:34 smithi076 conmon[81441]: level=info ts=2021-03-15T22:18:34.681Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.76 port=9094 2021-03-15T22:18:34.890 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:34 smithi076 conmon[81441]: level=info ts=2021-03-15T22:18:34.682Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2021-03-15T22:18:34.891 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:34 smithi076 conmon[81441]: level=info ts=2021-03-15T22:18:34.700Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2021-03-15T22:18:34.891 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:34 smithi076 conmon[81441]: level=info ts=2021-03-15T22:18:34.701Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2021-03-15T22:18:34.891 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:34 smithi076 conmon[81441]: level=info ts=2021-03-15T22:18:34.703Z caller=main.go:497 msg=Listening address=:9093 2021-03-15T22:18:34.891 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:34 smithi076 podman[81400]: 2021-03-15 22:18:34.887868299 +0000 UTC m=+1.139840649 container start 979ec48afd0e4720782d746f9e0da009d1b6e60e482841b43fa462157861ad8e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-15T22:18:35.166 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:34 smithi076 bash[81294]: 979ec48afd0e4720782d746f9e0da009d1b6e60e482841b43fa462157861ad8e 2021-03-15T22:18:35.167 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:34 smithi076 systemd[1]: Started Ceph alertmanager.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:18:35.501 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2021-03-15T22:18:36.227 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[56514]: cluster 2021-03-15T22:18:34.952544+0000 mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v200: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2021-03-15T22:18:36.227 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[56514]: audit 2021-03-15T22:18:34.954433+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.227 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[56514]: cluster 2021-03-15T22:18:34.959577+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-03-15T22:18:36.227 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[56514]: audit 2021-03-15T22:18:35.041150+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[56514]: audit 2021-03-15T22:18:35.044125+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[56514]: audit 2021-03-15T22:18:35.044552+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-15T22:18:36.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[56514]: audit 2021-03-15T22:18:35.048351+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[56514]: audit 2021-03-15T22:18:35.499863+0000 mon.a (mon.0) 523 : audit [DBG] from='client.? 172.21.15.76:0/2191193069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2021-03-15T22:18:36.229 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[62156]: cluster 2021-03-15T22:18:34.952544+0000 2021-03-15T22:18:36.229 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[62156]: mgr.y (mgr.14144) 257 : cluster [DBG] pgmap v200: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2021-03-15T22:18:36.229 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[62156]: audit 2021-03-15T22:18:34.954433+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.230 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[62156]: cluster 2021-03-15T22:18:34.959577+0000 mon.a (mon.0) 518 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-03-15T22:18:36.230 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[62156]: audit 2021-03-15T22:18:35.041150+0000 mon.a (mon.0) 519 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.230 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[62156]: audit 2021-03-15T22:18:35.044125+0000 mon.a (mon.0) 520 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.230 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[62156]: audit 2021-03-15T22:18:35.044552+0000 mon.a (mon.0) 521 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-15T22:18:36.231 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[62156]: audit 2021-03-15T22:18:36.232 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[62156]: 2021-03-15T22:18:35.048351+0000 mon.a (mon.0) 522 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.232 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:18:36.232 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:35 smithi076 conmon[62156]: 18:35.499863+0000 mon.a (mon.0) 523 : audit [DBG] from='client.? 172.21.15.76:0/2191193069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2021-03-15T22:18:36.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool get default.rgw.control pg_num 2021-03-15T22:18:36.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: cluster 2021-03-15T22:18:34.952544+0000 mgr.y (mgr.14144 2021-03-15T22:18:36.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: ) 257 : cluster [DBG] pgmap v200: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2021-03-15T22:18:36.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: audit 2021-03-15T22:18:34.954433+0000 mon.a (mon.0) 517 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: cluster 2021-03-15T22:18:34.959577+0000 mon.a 2021-03-15T22:18:36.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: (mon.0) 518 : cluster [DBG] osdmap e74: 8 total, 8 up, 8 in 2021-03-15T22:18:36.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: audit 2021-03-15T22:18:36.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: 2021-03-15T22:18:35. 2021-03-15T22:18:36.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: 041150+0000 mon.a (mon.0) 519 : audit 2021-03-15T22:18:36.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:18:36.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: :18:35.044125+0000 mon.a (mon. 2021-03-15T22:18:36.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: 0) 520 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:18:36.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: 03-15T22:18:35.044552+0000 2021-03-15T22:18:36.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: mon.a (mon.0) 521 : audit [INF] 2021-03-15T22:18:36.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-15T22:18:36.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: audit 2021-03-15T22:18: 2021-03-15T22:18:36.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: 35.048351+0000 mon.a (mon.0) 522 2021-03-15T22:18:36.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:36.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: audit 2021-03-15T22:18:36.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:18:36.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: 18:35.499863+0000 mon.a (mon.0 2021-03-15T22:18:36.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:35 smithi125 conmon[56739]: ) 523 : audit [DBG] from='client.? 172.21.15.76:0/2191193069' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.log", "var": "pg_num"}]: dispatch 2021-03-15T22:18:36.512 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:36.955 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:36 smithi076 conmon[81441]: level=info ts=2021-03-15T22:18:36.682Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000080603s 2021-03-15T22:18:37.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:36 smithi076 conmon[56514]: audit 2021-03-15T22:18:35.045024+0000 mgr.y (mgr.14144) 258 : audit [DBG] 2021-03-15T22:18:37.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:36 smithi076 conmon[56514]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-15T22:18:37.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:36 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:35.051854+0000 mgr.y (mgr.14144) 259 : cephadm [INF] Deploying daemon grafana.a on smithi125 2021-03-15T22:18:37.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:36 smithi076 conmon[62156]: audit 2021-03-15T22:18:35.045024+0000 mgr.y 2021-03-15T22:18:37.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:36 smithi076 conmon[62156]: (mgr.14144) 258 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-15T22:18:37.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:36 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:35.051854 2021-03-15T22:18:37.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:36 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 259 : cephadm [INF] Deploying daemon grafana.a on smithi125 2021-03-15T22:18:37.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:36 smithi125 conmon[56739]: audit 2021-03-15T22:18:35.045024+0000 mgr.y (mgr.14144) 258 : audit [DBG] 2021-03-15T22:18:37.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:36 smithi125 conmon[56739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2021-03-15T22:18:37.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:36 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:35.051854+0000 mgr.y (mgr.14144) 259 : cephadm [INF] Deploying daemon grafana.a on smithi125 2021-03-15T22:18:38.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:37 smithi076 conmon[56514]: cluster 2021-03-15T22:18:36.953262+0000 mgr.y ( 2021-03-15T22:18:38.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:37 smithi076 conmon[56514]: mgr.14144) 260 : cluster [DBG] pgmap v202: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:38.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:37 smithi076 conmon[62156]: cluster 2021-03-15T22:18:36.953262+0000 mgr.y (mgr. 2021-03-15T22:18:38.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:37 smithi076 conmon[62156]: 14144) 260 : cluster [DBG] pgmap v202: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:38.273 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2021-03-15T22:18:38.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:37 smithi125 conmon[56739]: cluster 2021-03-15T22:18:36.953262+0000 mgr.y (mgr.14144) 260 : 2021-03-15T22:18:38.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:37 smithi125 conmon[56739]: cluster [DBG] pgmap v202: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:38.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool get default.rgw.meta pg_num 2021-03-15T22:18:39.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:38 smithi076 conmon[56514]: audit 2021-03-15T22:18:38.272612+0000 mon.a (mon.0) 524 : audit [DBG] from='client.? 172.21.15.76:0/1417241797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2021-03-15T22:18:39.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:38 smithi076 conmon[62156]: audit 2021-03-15T22:18:38. 2021-03-15T22:18:39.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:38 smithi076 conmon[62156]: 272612+0000 mon.a (mon.0) 524 : audit [DBG] from='client.? 172.21.15.76:0/1417241797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2021-03-15T22:18:39.275 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:39.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:38 smithi125 conmon[56739]: audit 2021-03-15T22:18:38.272612+0000 mon.a (mon.0) 2021-03-15T22:18:39.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:38 smithi125 conmon[56739]: 524 : audit [DBG] from='client.? 172.21.15.76:0/1417241797' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.control", "var": "pg_num"}]: dispatch 2021-03-15T22:18:39.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:18:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:18:39.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:39 smithi076 conmon[56514]: cluster 2021-03-15T22:18:38.953788+0000 mgr.y (mgr.14144) 261 : cluster [DBG] pgmap v203: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:39.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:39 smithi076 conmon[56514]: cluster 2021-03-15T22:18:39.017731+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:39.975 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:39 smithi076 conmon[56514]: cluster 2021-03-15T22:18:39.017830+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:40.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:39 smithi076 conmon[62156]: cluster 2021-03-15T22:18:38.953788+0000 mgr.y (mgr.14144) 261 : cluster [DBG] pgmap v203: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:40.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:39 smithi076 conmon[62156]: cluster 2021-03-15T22:18:39.017731+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:40.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:39 smithi076 conmon[62156]: cluster 2021-03-15T22:18:39.017830+0000 osd.3 (osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:40.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:39 smithi125 conmon[56739]: cluster 2021-03-15T22:18:38.953788+0000 mgr.y (mgr. 2021-03-15T22:18:40.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:39 smithi125 conmon[56739]: 14144) 261 : cluster [DBG] pgmap v203: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:40.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:39 smithi125 conmon[56739]: cluster 2021-03-15T22:18:39.017731+0000 osd.3 (osd.3) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:40.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:39 smithi125 conmon[56739]: 2021-03-15T22:18:40.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:39 smithi125 conmon[56739]: cluster 2021-03-15T22:18:40.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:39 smithi125 conmon[56739]: 2021-03-15T22:18:39.017830+0000 osd.3 ( 2021-03-15T22:18:40.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:39 smithi125 conmon[56739]: osd.3) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:40.945 INFO:teuthology.orchestra.run.smithi076.stdout:pg_num: 32 2021-03-15T22:18:41.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[56514]: audit 2021-03-15T22:18:39.322368+0000 mgr.y (mgr. 2021-03-15T22:18:41.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[56514]: 14144) 262 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:41.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[56514]: cluster 2021-03-15T22:18:40.015889+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2021-03-15T22:18:41.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[56514]: audit 2021-03-15T22:18:40.942940+0000 mon.a (mon.0) 526 : audit [DBG] from='client.? 172.21.15.76:0/4164087033' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2021-03-15T22:18:41.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[62156]: audit 2021-03-15T22:18:39. 2021-03-15T22:18:41.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[62156]: 322368+0000 mgr.y (mgr.14144) 262 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:41.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[62156]: cluster 2021-03-15T22:18:41.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[62156]: 2021-03-15T22:18:40.015889+0000 mon.a (mon.0) 525 : 2021-03-15T22:18:41.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[62156]: cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2021-03-15T22:18:41.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:18:41.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[62156]: 18:40.942940+0000 mon.a (mon.0) 526 : 2021-03-15T22:18:41.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:40 smithi076 conmon[62156]: audit [DBG] from='client.? 172.21.15.76:0/4164087033' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2021-03-15T22:18:41.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:40 smithi125 conmon[56739]: audit 2021-03-15T22:18:39.322368+0000 mgr.y (mgr.14144) 262 : audit [DBG] 2021-03-15T22:18:41.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:40 smithi125 conmon[56739]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:41.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:40 smithi125 conmon[56739]: cluster 2021-03-15T22:18:40.015889+0000 mon.a (mon.0) 525 : cluster [DBG] osdmap e75: 8 total, 8 up, 8 in 2021-03-15T22:18:41.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:40 smithi125 conmon[56739]: audit 2021-03-15T22:18:40.942940+0000 mon.a 2021-03-15T22:18:41.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:40 smithi125 conmon[56739]: (mon.0) 526 : audit [DBG] from='client.? 172.21.15.76:0/4164087033' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "default.rgw.meta", "var": "pg_num"}]: dispatch 2021-03-15T22:18:41.662 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-03-15T22:18:41.664 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2021-03-15T22:18:41.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph mgr dump --format=json 2021-03-15T22:18:41.942 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:42.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:41 smithi076 conmon[56514]: cluster 2021-03-15T22:18:40.954113+0000 mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:42.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:41 smithi076 conmon[62156]: cluster 2021-03-15T22:18:40.954113+0000 2021-03-15T22:18:42.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:41 smithi076 conmon[62156]: mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:42.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:41 smithi125 conmon[56739]: cluster 2021-03-15T22:18:40.954113 2021-03-15T22:18:42.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:41 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 263 : cluster [DBG] pgmap v205: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:43.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:43 smithi076 conmon[56514]: cluster 2021-03-15T22:18:42.954683+0000 mgr.y (mgr.14144) 2021-03-15T22:18:43.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:43 smithi076 conmon[56514]: 264 : cluster [DBG] pgmap v206: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:43.256 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:43 smithi076 conmon[62156]: cluster 2021-03-15T22:18:42.954683+0000 mgr.y (mgr.14144) 264 : cluster [DBG] pgmap v206: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:43.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:43 smithi125 conmon[56739]: cluster 2021-03-15T22:18:42.954683+0000 mgr.y (mgr 2021-03-15T22:18:43.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:43 smithi125 conmon[56739]: .14144) 264 : cluster [DBG] pgmap v206: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:18:43.591 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:18:44.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:44 smithi125 conmon[56739]: audit 2021-03-15T22:18:43.585481 2021-03-15T22:18:44.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:44 smithi125 conmon[56739]: +0000 mon.a (mon.0) 527 : audit [DBG] from='client.? 172.21.15.76:0/3020678024' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-03-15T22:18:44.304 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:44 smithi076 conmon[56514]: audit 2021-03-15T22:18:43.585481+0000 2021-03-15T22:18:44.305 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:44 smithi076 conmon[56514]: mon.a (mon.0) 527 : audit [DBG] from='client.? 172.21.15.76:0/3020678024' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-03-15T22:18:44.305 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:44 smithi076 conmon[62156]: audit 2021-03-15T22:18:43.585481+0000 mon.a ( 2021-03-15T22:18:44.305 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:44 smithi076 conmon[62156]: mon.0) 527 : audit [DBG] from='client.? 172.21.15.76:0/3020678024' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2021-03-15T22:18:44.328 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":15,"active_gid":14144,"active_name":"y","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6800","nonce":1703655353},{"type":"v1","addr":"172.21.15.76:6801","nonce":1703655353}]},"active_addr":"172.21.15.76:6801/1703655353","active_change":"2021-03-15T22:14:00.771971+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[{"gid":34109,"name":"x","mgr_features":4540138303579357183,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"str","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://ceph-8a572802-85db-11eb-9091-001a4aab830c-mgr.y:8443/"},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":3,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":3776285537}]},{"addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":3089063243}]},{"addrvec":[{"type":"v2","addr":"172.21.15.76:0","nonce":3587538366}]}]}} 2021-03-15T22:18:44.333 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2021-03-15T22:18:44.333 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2021-03-15T22:18:44.333 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump --format=json 2021-03-15T22:18:44.611 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:45.011 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:44 smithi076 conmon[81441]: level=info ts=2021-03-15T22:18:44.683Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000882432s 2021-03-15T22:18:46.007 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:45 smithi076 conmon[56514]: cluster 2021-03-15T22:18:44.955154+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2021-03-15T22:18:46.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:45 smithi076 conmon[56514]: cluster 2021-03-15T22:18:44.962039+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-03-15T22:18:46.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:45 smithi076 conmon[62156]: cluster 2021-03-15T22:18:44.955154+0000 2021-03-15T22:18:46.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:45 smithi076 conmon[62156]: mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2021-03-15T22:18:46.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:45 smithi076 conmon[62156]: cluster 2021-03-15T22:18:44.962039+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-03-15T22:18:46.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:45 smithi125 conmon[56739]: cluster 2021-03-15T22:18:44.955154+0000 mgr.y (mgr.14144) 265 : cluster [DBG] pgmap v207: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2021-03-15T22:18:46.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:45 smithi125 conmon[56739]: cluster 2021-03-15T22:18:44.962039+0000 mon.a (mon.0) 528 : cluster [DBG] osdmap e76: 8 total, 8 up, 8 in 2021-03-15T22:18:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:18:46.359 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":76,"fsid":"8a572802-85db-11eb-9091-001a4aab830c","created":"2021-03-15T22:13:20.370795+0000","modified":"2021-03-15T22:18:44.958032+0000","last_up_change":"2021-03-15T22:17:49.162889+0000","last_in_change":"2021-03-15T22:17:34.080423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-15T22:16:34.817191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"iscsi","create_time":"2021-03-15T22:17:49.906400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2021-03-15T22:17:52.257970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2021-03-15T22:17:54.562498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2021-03-15T22:17:56.243806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2021-03-15T22:17:58.730534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6803","nonce":188228176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6805","nonce":188228176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6809","nonce":188228176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6807","nonce":188228176}]},"public_addr":"172.21.15.76:6803/188228176","cluster_addr":"172.21.15.76:6805/188228176","heartbeat_back_addr":"172.21.15.76:6809/188228176","heartbeat_front_addr":"172.21.15.76:6807/188228176","state":["exists","up"]},{"osd":1,"uuid":"97bd3ffc-15c8-4a6f-b37a-217813def526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6811","nonce":3924955423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6813","nonce":3924955423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6817","nonce":3924955423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6815","nonce":3924955423}]},"public_addr":"172.21.15.76:6811/3924955423","cluster_addr":"172.21.15.76:6813/3924955423","heartbeat_back_addr":"172.21.15.76:6817/3924955423","heartbeat_front_addr":"172.21.15.76:6815/3924955423","state":["exists","up"]},{"osd":2,"uuid":"a7f5f115-e69e-47d9-b6c3-ff386ddfaa05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6819","nonce":1401263050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6821","nonce":1401263050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6825","nonce":1401263050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6823","nonce":1401263050}]},"public_addr":"172.21.15.76:6819/1401263050","cluster_addr":"172.21.15.76:6821/1401263050","heartbeat_back_addr":"172.21.15.76:6825/1401263050","heartbeat_front_addr":"172.21.15.76:6823/1401263050","state":["exists","up"]},{"osd":3,"uuid":"a0bb7f88-0114-47cb-a82c-d8fb002aac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6827","nonce":3960968636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6829","nonce":3960968636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6833","nonce":3960968636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6831","nonce":3960968636}]},"public_addr":"172.21.15.76:6827/3960968636","cluster_addr":"172.21.15.76:6829/3960968636","heartbeat_back_addr":"172.21.15.76:6833/3960968636","heartbeat_front_addr":"172.21.15.76:6831/3960968636","state":["exists","up"]},{"osd":4,"uuid":"b650611c-6101-47e8-9ca9-94e02cf8b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6801","nonce":37386122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6803","nonce":37386122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6807","nonce":37386122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6805","nonce":37386122}]},"public_addr":"172.21.15.125:6801/37386122","cluster_addr":"172.21.15.125:6803/37386122","heartbeat_back_addr":"172.21.15.125:6807/37386122","heartbeat_front_addr":"172.21.15.125:6805/37386122","state":["exists","up"]},{"osd":5,"uuid":"7d6ceff2-7093-4ab9-8b96-ade448093bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6809","nonce":3610838738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6811","nonce":3610838738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6815","nonce":3610838738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6813","nonce":3610838738}]},"public_addr":"172.21.15.125:6809/3610838738","cluster_addr":"172.21.15.125:6811/3610838738","heartbeat_back_addr":"172.21.15.125:6815/3610838738","heartbeat_front_addr":"172.21.15.125:6813/3610838738","state":["exists","up"]},{"osd":6,"uuid":"98b72022-a9a7-4ee2-89a1-cac037de7afe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6817","nonce":171378275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6819","nonce":171378275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6823","nonce":171378275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6821","nonce":171378275}]},"public_addr":"172.21.15.125:6817/171378275","cluster_addr":"172.21.15.125:6819/171378275","heartbeat_back_addr":"172.21.15.125:6823/171378275","heartbeat_front_addr":"172.21.15.125:6821/171378275","state":["exists","up"]},{"osd":7,"uuid":"f99ea650-b2dc-4d87-a44c-56a897edd614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6825","nonce":652003948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6827","nonce":652003948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6831","nonce":652003948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6829","nonce":652003948}]},"public_addr":"172.21.15.125:6825/652003948","cluster_addr":"172.21.15.125:6827/652003948","heartbeat_back_addr":"172.21.15.125:6831/652003948","heartbeat_front_addr":"172.21.15.125:6829/652003948","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:17:49.279866+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:07.166571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:23.085495+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:39.017833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:17:55.480481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:12.704364+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:30.284120+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:17:47.645171+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6800/2604045245":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/4087324200":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/4141143304":"2021-03-16T22:14:00.771702+0000","172.21.15.76:6801/2604045245":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/2779879991":"2021-03-16T22:13:41.603652+0000","172.21.15.76:6801/2764179881":"2021-03-16T22:13:41.603652+0000","172.21.15.76:6800/2764179881":"2021-03-16T22:13:41.603652+0000","172.21.15.76:0/2483102204":"2021-03-16T22:13:41.603652+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-15T22:18:47.011 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2021-03-15T22:18:47.012 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump --format=json 2021-03-15T22:18:47.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:46 smithi076 conmon[56514]: audit 2021-03-15T22:18:46.355661+0000 2021-03-15T22:18:47.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:46 smithi076 conmon[56514]: mon.a (mon.0) 529 : audit [DBG] from='client.? 172.21.15.76:0/538312185' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:18:47.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:46 smithi076 conmon[62156]: audit 2021-03-15T22:18:46.355661+0000 mon.a 2021-03-15T22:18:47.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:46 smithi076 conmon[62156]: (mon.0) 529 : audit [DBG] from='client.? 172.21.15.76:0/538312185' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:18:47.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:46 smithi125 conmon[56739]: audit 2021-03-15T22:18:46.355661+0000 mon.a (mon.0) 529 : audit [DBG] from='client.? 172.21.15.76:0/538312185' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:18:47.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:46 smithi125 conmon[56739]: 2021-03-15T22:18:47.288 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:48.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:47 smithi076 conmon[62156]: cluster 2021-03-15T22:18:46.955827+0000 mgr.y (mgr.14144) 266 : cluster [DBG] pgmap v209: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:48.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:47 smithi076 conmon[56514]: cluster 2021-03-15T22:18:46.955827+0000 mgr.y (mgr.14144) 266 : cluster [DBG] pgmap v209: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:48.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:47 smithi125 conmon[56739]: cluster 2021-03-15T22:18:46.955827+0000 mgr.y (mgr.14144) 266 : cluster [DBG] pgmap v209: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:49.007 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:18:49.008 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":77,"fsid":"8a572802-85db-11eb-9091-001a4aab830c","created":"2021-03-15T22:13:20.370795+0000","modified":"2021-03-15T22:18:48.967291+0000","last_up_change":"2021-03-15T22:17:49.162889+0000","last_in_change":"2021-03-15T22:17:34.080423+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-15T22:16:34.817191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"iscsi","create_time":"2021-03-15T22:17:49.906400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2021-03-15T22:17:52.257970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2021-03-15T22:17:54.562498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2021-03-15T22:17:56.243806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"62","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2021-03-15T22:17:58.730534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"65","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6803","nonce":188228176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6805","nonce":188228176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6809","nonce":188228176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6807","nonce":188228176}]},"public_addr":"172.21.15.76:6803/188228176","cluster_addr":"172.21.15.76:6805/188228176","heartbeat_back_addr":"172.21.15.76:6809/188228176","heartbeat_front_addr":"172.21.15.76:6807/188228176","state":["exists","up"]},{"osd":1,"uuid":"97bd3ffc-15c8-4a6f-b37a-217813def526","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6811","nonce":3924955423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6813","nonce":3924955423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6817","nonce":3924955423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6815","nonce":3924955423}]},"public_addr":"172.21.15.76:6811/3924955423","cluster_addr":"172.21.15.76:6813/3924955423","heartbeat_back_addr":"172.21.15.76:6817/3924955423","heartbeat_front_addr":"172.21.15.76:6815/3924955423","state":["exists","up"]},{"osd":2,"uuid":"a7f5f115-e69e-47d9-b6c3-ff386ddfaa05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6819","nonce":1401263050}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6821","nonce":1401263050}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6825","nonce":1401263050}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1401263050},{"type":"v1","addr":"172.21.15.76:6823","nonce":1401263050}]},"public_addr":"172.21.15.76:6819/1401263050","cluster_addr":"172.21.15.76:6821/1401263050","heartbeat_back_addr":"172.21.15.76:6825/1401263050","heartbeat_front_addr":"172.21.15.76:6823/1401263050","state":["exists","up"]},{"osd":3,"uuid":"a0bb7f88-0114-47cb-a82c-d8fb002aac56","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6827","nonce":3960968636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6829","nonce":3960968636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6833","nonce":3960968636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6831","nonce":3960968636}]},"public_addr":"172.21.15.76:6827/3960968636","cluster_addr":"172.21.15.76:6829/3960968636","heartbeat_back_addr":"172.21.15.76:6833/3960968636","heartbeat_front_addr":"172.21.15.76:6831/3960968636","state":["exists","up"]},{"osd":4,"uuid":"b650611c-6101-47e8-9ca9-94e02cf8b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6801","nonce":37386122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6803","nonce":37386122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6807","nonce":37386122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6805","nonce":37386122}]},"public_addr":"172.21.15.125:6801/37386122","cluster_addr":"172.21.15.125:6803/37386122","heartbeat_back_addr":"172.21.15.125:6807/37386122","heartbeat_front_addr":"172.21.15.125:6805/37386122","state":["exists","up"]},{"osd":5,"uuid":"7d6ceff2-7093-4ab9-8b96-ade448093bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6809","nonce":3610838738}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6811","nonce":3610838738}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6815","nonce":3610838738}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":3610838738},{"type":"v1","addr":"172.21.15.125:6813","nonce":3610838738}]},"public_addr":"172.21.15.125:6809/3610838738","cluster_addr":"172.21.15.125:6811/3610838738","heartbeat_back_addr":"172.21.15.125:6815/3610838738","heartbeat_front_addr":"172.21.15.125:6813/3610838738","state":["exists","up"]},{"osd":6,"uuid":"98b72022-a9a7-4ee2-89a1-cac037de7afe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":60,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6817","nonce":171378275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6819","nonce":171378275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6823","nonce":171378275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6821","nonce":171378275}]},"public_addr":"172.21.15.125:6817/171378275","cluster_addr":"172.21.15.125:6819/171378275","heartbeat_back_addr":"172.21.15.125:6823/171378275","heartbeat_front_addr":"172.21.15.125:6821/171378275","state":["exists","up"]},{"osd":7,"uuid":"f99ea650-b2dc-4d87-a44c-56a897edd614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":62,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6825","nonce":652003948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6827","nonce":652003948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6831","nonce":652003948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6829","nonce":652003948}]},"public_addr":"172.21.15.125:6825/652003948","cluster_addr":"172.21.15.125:6827/652003948","heartbeat_back_addr":"172.21.15.125:6831/652003948","heartbeat_front_addr":"172.21.15.125:6829/652003948","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:17:49.279866+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:07.166571+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:23.085495+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:39.017833+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:17:55.480481+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:12.704364+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:30.284120+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:18:48.567392+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6800/2604045245":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/4087324200":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/4141143304":"2021-03-16T22:14:00.771702+0000","172.21.15.76:6801/2604045245":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/2779879991":"2021-03-16T22:13:41.603652+0000","172.21.15.76:6801/2764179881":"2021-03-16T22:13:41.603652+0000","172.21.15.76:6800/2764179881":"2021-03-16T22:13:41.603652+0000","172.21.15.76:0/2483102204":"2021-03-16T22:13:41.603652+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-15T22:18:49.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:48 smithi076 conmon[56514]: cluster 2021-03-15T22:18:48. 2021-03-15T22:18:49.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:48 smithi076 conmon[56514]: 567280+0000 osd.7 (osd.7) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:49.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:48 smithi076 conmon[56514]: cluster 2021-03-15T22:18:48.567390+0000 osd.7 (osd.7) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:48 smithi076 conmon[62156]: cluster 2021-03-15T22:18: 2021-03-15T22:18:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:48 smithi076 conmon[62156]: 48.567280+0000 osd.7 (osd.7) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:48 smithi076 conmon[62156]: cluster 2021-03-15T22:18:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:48 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:18:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:48 smithi076 conmon[62156]: :18:48.567390+0000 osd.7 2021-03-15T22:18:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:48 smithi076 conmon[62156]: (osd.7) 4 : cluster 2021-03-15T22:18:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:48 smithi076 conmon[62156]: [DBG] purged_snaps scrub ok 2021-03-15T22:18:49.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:48 smithi125 conmon[56739]: cluster 2021-03-15T22:18:48.567280+0000 osd.7 (osd. 2021-03-15T22:18:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:48 smithi125 conmon[56739]: 7) 3 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:48 smithi125 conmon[56739]: cluster 2021-03-15T22:18:48.567390+0000 osd.7 (osd.7) 4 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:49.643 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph tell osd.0 flush_pg_stats 2021-03-15T22:18:49.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:18:49.921 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:49.980 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[56514]: cluster 2021-03-15T22:18:48.956331+0000 mgr.y (mgr.14144) 267 : cluster 2021-03-15T22:18:49.981 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[56514]: [DBG] pgmap v210: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2021-03-15T22:18:49.981 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[56514]: cluster 2021-03-15T22:18:48.974564+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-03-15T22:18:49.981 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[56514]: audit 2021-03-15T22:18:49.005513+0000 mon.a (mon.0) 531 : audit [DBG] from='client.? 172.21.15.76:0/3042441404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:18:49.982 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[56514]: cluster 2021-03-15T22:18:49.323230+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:49.982 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[56514]: cluster 2021-03-15T22:18:49.323305+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:49.982 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[56514]: cluster 2021-03-15T22:18:49.966544+0000 mon.a (mon.0) 532 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2021-03-15T22:18:50.156 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: cluster 2021-03-15T22:18:48.956331+0000 mgr.y ( 2021-03-15T22:18:50.156 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: mgr.14144) 267 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2021-03-15T22:18:50.157 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: cluster 2021-03-15T22:18:48.974564+0000 mon.a (mon.0) 530 : cluster 2021-03-15T22:18:50.157 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-03-15T22:18:50.157 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: audit 2021-03-15T22:18:50.157 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: 2021 2021-03-15T22:18:50.157 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: -03-15T22:18:49.005513+0000 mon.a (mon.0 2021-03-15T22:18:50.158 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: ) 531 : audit [DBG] from='client.? 172.21.15.76:0/3042441404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:18:50.158 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:18:50.158 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: -03-15T22:18:49.323230+0000 2021-03-15T22:18:50.158 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:50.159 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:18:50.159 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: -03-15T22:18:49.323305 2021-03-15T22:18:50.159 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: +0000 osd.0 (osd.0) 8 : cluster 2021-03-15T22:18:50.159 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: [DBG] purged_snaps scrub ok 2021-03-15T22:18:50.160 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:18:50.160 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: -15T22:18:49.966544+0000 2021-03-15T22:18:50.160 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: mon.a (mon.0) 532 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2021-03-15T22:18:50.160 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:49 smithi125 conmon[56739]: 2021-03-15T22:18:50.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[62156]: cluster 2021-03-15T22:18:48.956331+0000 mgr.y (mgr.14144) 267 : cluster [DBG] pgmap v210: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 572 B/s rd, 0 op/s 2021-03-15T22:18:50.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[62156]: cluster 2021-03-15T22:18:48.974564+0000 mon.a (mon.0) 530 : cluster [DBG] osdmap e77: 8 total, 8 up, 8 in 2021-03-15T22:18:50.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[62156]: audit 2021-03-15T22:18:49.005513+0000 mon.a (mon.0) 531 : audit [DBG] from='client.? 172.21.15.76:0/3042441404' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:18:50.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[62156]: cluster 2021-03-15T22:18:49.323230+0000 osd.0 (osd.0) 7 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:50.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[62156]: cluster 2021-03-15T22:18:49.323305+0000 osd.0 (osd.0) 8 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:50.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:49 smithi076 conmon[62156]: cluster 2021-03-15T22:18:49.966544+0000 mon.a (mon.0) 532 : cluster [DBG] osdmap e78: 8 total, 8 up, 8 in 2021-03-15T22:18:51.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:50 smithi076 conmon[56514]: audit 2021-03-15T22:18:49.333164+0000 mgr.y (mgr.14144 2021-03-15T22:18:51.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:50 smithi076 conmon[56514]: ) 268 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:51.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:50 smithi076 conmon[56514]: cluster 2021-03-15T22:18:50.956807+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v213: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:18:51.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:50 smithi076 conmon[62156]: audit 2021-03-15T22:18:49.333164+0000 mgr.y (mgr.14144) 268 : 2021-03-15T22:18:51.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:50 smithi076 conmon[62156]: audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:51.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:50 smithi076 conmon[62156]: cluster 2021-03-15T22:18:50.956807+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v213: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:18:51.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:50 smithi125 conmon[56739]: audit 2021-03-15T22:18:49.333164+0000 mgr.y (mgr. 2021-03-15T22:18:51.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:50 smithi125 conmon[56739]: 14144) 268 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:18:51.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:50 smithi125 conmon[56739]: cluster 2021-03-15T22:18:50.956807+0000 mgr.y (mgr.14144) 269 : cluster [DBG] pgmap v213: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:18:51.287 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 systemd[1]: Starting Ceph grafana.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:18:51.650 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 podman[77222]: 2021-03-15 22:18:51.379099535 +0000 UTC m=+0.166718214 container create 8c9c27280f1f3b9b7cadf01184a8696b0612149b0df3bfef3d4cb1d1c84847b0 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-grafana.a, description=Ceph Grafana Container, org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner , org.label-schema.schema-version=1.0) 2021-03-15T22:18:51.651 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 podman[77222]: 2021-03-15 22:18:51.620712863 +0000 UTC m=+0.408331441 container init 8c9c27280f1f3b9b7cadf01184a8696b0612149b0df3bfef3d4cb1d1c84847b0 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-grafana.a, org.label-schema.schema-version=1.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, description=Ceph Grafana Container, org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0, org.label-schema.license=GPLv2, maintainer=Paul Cuzner , org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS) 2021-03-15T22:18:51.956 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Starting Grafana" logger=server version=6.7.4 commit=8e44bbc5f5 branch=HEAD compiled=2020-05-26T17:35:38+0000 2021-03-15T22:18:51.957 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Config loaded from" logger=settings file=/usr/share/grafana/conf/defaults.ini 2021-03-15T22:18:51.957 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Config loaded from" logger=settings file=/etc/grafana/grafana.ini 2021-03-15T22:18:51.957 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_DATA=/var/lib/grafana" 2021-03-15T22:18:51.957 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_LOGS=/var/log/grafana" 2021-03-15T22:18:51.958 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2021-03-15T22:18:51.958 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Config overridden from Environment variable" logger=settings var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2021-03-15T22:18:51.958 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Path Home" logger=settings path=/usr/share/grafana 2021-03-15T22:18:51.958 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Path Data" logger=settings path=/var/lib/grafana 2021-03-15T22:18:51.958 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Path Logs" logger=settings path=/var/log/grafana 2021-03-15T22:18:51.959 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Path Plugins" logger=settings path=/var/lib/grafana/plugins 2021-03-15T22:18:51.959 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Path Provisioning" logger=settings path=/etc/grafana/provisioning 2021-03-15T22:18:51.959 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="App mode production" logger=settings 2021-03-15T22:18:51.959 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Initializing SqlStore" logger=server 2021-03-15T22:18:51.960 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Connecting to DB" logger=sqlstore dbtype=sqlite3 2021-03-15T22:18:51.960 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Starting DB migration" logger=migrator 2021-03-15T22:18:51.960 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Executing migration" logger=migrator id="create migration_log table" 2021-03-15T22:18:51.960 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 podman[77222]: 2021-03-15 22:18:51.653981877 +0000 UTC m=+0.441600552 container start 8c9c27280f1f3b9b7cadf01184a8696b0612149b0df3bfef3d4cb1d1c84847b0 (image=docker.io/ceph/ceph-grafana:6.7.4, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-grafana.a, org.label-schema.vendor=CentOS, description=Ceph Grafana Container, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0, summary=Grafana Container configured for Ceph mgr/dashboard integration, maintainer=Paul Cuzner ) 2021-03-15T22:18:51.960 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 bash[77148]: 8c9c27280f1f3b9b7cadf01184a8696b0612149b0df3bfef3d4cb1d1c84847b0 2021-03-15T22:18:51.961 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 systemd[1]: Started Ceph grafana.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:18:51.961 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Executing migration" logger=migrator id="create user table" 2021-03-15T22:18:51.961 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.login" 2021-03-15T22:18:51.961 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user.email" 2021-03-15T22:18:51.962 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:51 smithi125 conmon[77264]: t=2021-03-15T22:18:51+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_login - v1" 2021-03-15T22:18:52.286 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_user_email - v1" 2021-03-15T22:18:52.286 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table user to user_v1 - v1" 2021-03-15T22:18:52.287 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="create user table v2" 2021-03-15T22:18:52.360 INFO:teuthology.orchestra.run.smithi076.stdout:34359738407 2021-03-15T22:18:52.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph tell osd.1 flush_pg_stats 2021-03-15T22:18:52.557 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_login - v2" 2021-03-15T22:18:52.753 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_user_email - v2" 2021-03-15T22:18:52.754 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2021-03-15T22:18:52.765 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:52.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[56739]: audit 2021-03-15T22:18:51.712327+0000 mon.a (mon.0) 533 2021-03-15T22:18:52.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[56739]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:52.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[56739]: audit 2021-03-15T22:18:51.713102+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:18:52.820 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table user_v1" 2021-03-15T22:18:52.820 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add column help_flags1 to user table" 2021-03-15T22:18:52.821 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="Update user table charset" 2021-03-15T22:18:52.821 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add last_seen_at column to user" 2021-03-15T22:18:53.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:52 smithi076 conmon[56514]: audit 2021-03-15T22:18:51.712327+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:53.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:52 smithi076 conmon[56514]: audit 2021-03-15T22:18:51.713102+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:18:53.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:52 smithi076 conmon[62156]: audit 2021-03-15T22:18:51.712327+0000 mon.a (mon.0) 533 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:53.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:52 smithi076 conmon[62156]: audit 2021-03-15T22:18:51.713102+0000 mon.a (mon.0) 534 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:18:53.149 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add missing user data" 2021-03-15T22:18:53.149 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="Add is_disabled column to user" 2021-03-15T22:18:53.149 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:52 smithi125 conmon[77264]: t=2021-03-15T22:18:52+0000 lvl=info msg="Executing migration" logger=migrator id="create temp user table v1-7" 2021-03-15T22:18:53.149 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:53 smithi125 conmon[77264]: t=2021-03-15T22:18:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_email - v1-7" 2021-03-15T22:18:53.448 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:53 smithi125 conmon[77264]: t=2021-03-15T22:18:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_org_id - v1-7" 2021-03-15T22:18:53.448 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:53 smithi125 conmon[77264]: t=2021-03-15T22:18:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_code - v1-7" 2021-03-15T22:18:53.707 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:53 smithi125 conmon[77264]: t=2021-03-15T22:18:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_temp_user_status - v1-7" 2021-03-15T22:18:53.708 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:53 smithi125 conmon[77264]: t=2021-03-15T22:18:53+0000 lvl=info msg="Executing migration" logger=migrator id="Update temp_user table charset" 2021-03-15T22:18:53.708 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:53 smithi125 conmon[77264]: t=2021-03-15T22:18:53+0000 lvl=info msg="Executing migration" logger=migrator id="create star table" 2021-03-15T22:18:53.959 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:53 smithi125 conmon[56739]: cluster 2021-03-15T22:18:52.957385+0000 mgr.y (mgr.14144 2021-03-15T22:18:53.959 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:53 smithi125 conmon[56739]: ) 270 : cluster [DBG] pgmap v214: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2021-03-15T22:18:53.959 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:53 smithi125 conmon[77264]: t=2021-03-15T22:18:53+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index star.user_id_dashboard_id" 2021-03-15T22:18:53.960 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:53 smithi125 conmon[77264]: t=2021-03-15T22:18:53+0000 lvl=info msg="Executing migration" logger=migrator id="create org table v1" 2021-03-15T22:18:53.960 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:53 smithi125 conmon[77264]: t=2021-03-15T22:18:53+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_name - v1" 2021-03-15T22:18:54.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:53 smithi076 conmon[56514]: cluster 2021-03-15T22:18:52.957385+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v214: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2021-03-15T22:18:54.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:53 smithi076 conmon[62156]: cluster 2021-03-15T22:18:52. 2021-03-15T22:18:54.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:53 smithi076 conmon[62156]: 957385+0000 mgr.y (mgr.14144) 270 : cluster [DBG] pgmap v214: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 640 B/s rd, 0 op/s 2021-03-15T22:18:54.277 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:54 smithi125 conmon[77264]: t=2021-03-15T22:18:54+0000 lvl=info msg="Executing migration" logger=migrator id="create org_user table v1" 2021-03-15T22:18:54.277 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:54 smithi125 conmon[77264]: t=2021-03-15T22:18:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_org_user_org_id - v1" 2021-03-15T22:18:54.277 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:54 smithi125 conmon[77264]: t=2021-03-15T22:18:54+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_org_user_org_id_user_id - v1" 2021-03-15T22:18:54.536 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:54 smithi125 conmon[77264]: t=2021-03-15T22:18:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update org table charset" 2021-03-15T22:18:54.537 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:54 smithi125 conmon[77264]: t=2021-03-15T22:18:54+0000 lvl=info msg="Executing migration" logger=migrator id="Update org_user table charset" 2021-03-15T22:18:54.913 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:54 smithi125 conmon[56739]: cluster 2021-03-15T22:18:53.334472+0000 osd.5 ( 2021-03-15T22:18:54.914 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:54 smithi125 conmon[56739]: osd.5) 5 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:18:54.914 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:54 smithi125 conmon[77264]: t=2021-03-15T22:18:54+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate all Read Only Viewers to Viewers" 2021-03-15T22:18:54.914 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:54 smithi125 conmon[77264]: t=2021-03-15T22:18:54+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard table" 2021-03-15T22:18:54.914 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:54 smithi125 conmon[77264]: t=2021-03-15T22:18:54+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard.account_id" 2021-03-15T22:18:55.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:54 smithi076 conmon[56514]: cluster 2021-03-15T22:18:53.334472 2021-03-15T22:18:55.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:54 smithi076 conmon[56514]: +0000 osd.5 (osd.5) 5 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:18:55.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:54 smithi076 conmon[62156]: cluster 2021-03-15T22:18:53.334472+0000 osd.5 ( 2021-03-15T22:18:55.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:54 smithi076 conmon[62156]: osd.5) 5 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:18:55.187 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[77264]: t=2021-03-15T22:18:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_account_id_slug" 2021-03-15T22:18:55.187 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[77264]: t=2021-03-15T22:18:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_tag table" 2021-03-15T22:18:55.353 INFO:teuthology.orchestra.run.smithi076.stdout:55834574885 2021-03-15T22:18:55.354 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph tell osd.2 flush_pg_stats 2021-03-15T22:18:55.536 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[77264]: t=2021-03-15T22:18:55+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_tag.dasboard_id_term" 2021-03-15T22:18:55.537 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[77264]: t=2021-03-15T22:18:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2021-03-15T22:18:55.661 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:55.867 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[77264]: t=2021-03-15T22:18:55+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard to dashboard_v1 - v1" 2021-03-15T22:18:55.868 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[77264]: t=2021-03-15T22:18:55+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard v2" 2021-03-15T22:18:55.868 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[77264]: t=2021-03-15T22:18:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_org_id - v2" 2021-03-15T22:18:55.868 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[77264]: t=2021-03-15T22:18:55+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_org_id_slug - v2" 2021-03-15T22:18:55.868 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[77264]: t=2021-03-15T22:18:55+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard v1 to v2" 2021-03-15T22:18:56.119 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[56739]: cluster 2021-03-15T22:18:54.957887+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v215: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:56.120 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[56739]: audit 2021-03-15T22:18:55.555020+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:56.120 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:18:56.120 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[56739]: 22:18:55.769581+0000 mon.a 2021-03-15T22:18:56.120 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[56739]: (mon.0) 536 : audit 2021-03-15T22:18:56.120 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:56.121 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[77264]: t=2021-03-15T22:18:55+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_v1" 2021-03-15T22:18:56.121 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:55 smithi125 conmon[77264]: t=2021-03-15T22:18:55+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard.data to mediumtext v1" 2021-03-15T22:18:56.121 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add column updated_by in dashboard - v2" 2021-03-15T22:18:56.122 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add column created_by in dashboard - v2" 2021-03-15T22:18:56.122 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add column gnetId in dashboard" 2021-03-15T22:18:56.152 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:55 smithi076 conmon[56514]: cluster 2021-03-15T22:18:54.957887+0000 mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v215: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:56.153 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:55 smithi076 conmon[56514]: audit 2021-03-15T22:18:55.555020+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:56.153 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:55 smithi076 conmon[56514]: audit 2021-03-15T22:18:55.769581+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:56.153 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:55 smithi076 conmon[62156]: cluster 2021-03-15T22:18:54.957887+0000 2021-03-15T22:18:56.154 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:55 smithi076 conmon[62156]: mgr.y (mgr.14144) 271 : cluster [DBG] pgmap v215: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:56.154 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:55 smithi076 conmon[62156]: audit 2021-03-15T22:18:55.555020+0000 mon.a (mon.0) 535 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:56.154 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:55 smithi076 conmon[62156]: audit 2021-03-15T22:18:55.769581+0000 mon.a (mon.0) 536 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:56.155 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: debug 2021-03-15T22:18:56.059+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:18:56.155 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:56.155 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:56.156 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:56.156 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:56.156 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:56.156 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:56.157 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:56.157 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:18:56.157 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: debug 2021-03-15T22:18:56.059+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:56.157 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:56.158 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:56.158 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:56.158 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:56.158 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:56.158 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:56.159 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:56.159 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:56.159 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: debug 2021-03-15T22:18:56.059+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:56.159 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:56.160 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:56.160 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:56.160 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:56.160 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:56.160 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:56.161 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:56.161 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:56.161 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: debug 2021-03-15T22:18:56.060+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:18:56.161 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:56.162 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:56.162 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:56.162 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:56.162 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:56.162 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:56.163 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:56.163 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:18:56.164 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: debug 2021-03-15T22:18:56.060+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:18:56.164 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:56.164 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:56.164 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:56.165 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:56.165 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:56.165 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:56.165 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:56.166 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:18:56.166 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: debug 2021-03-15T22:18:56.061+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:18:56.166 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:56.166 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:56.166 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:56.167 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:56.167 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:56.167 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:56.167 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:56.167 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:18:56.168 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: debug 2021-03-15T22:18:56.061+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:18:56.168 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:18:56.168 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:56.168 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:18:56.169 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:56.169 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:18:56.169 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:56.169 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:56.169 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:18:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:18:56.437 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for gnetId in dashboard" 2021-03-15T22:18:56.437 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_id in dashboard" 2021-03-15T22:18:56.438 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for plugin_id in dashboard" 2021-03-15T22:18:56.438 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for dashboard_id in dashboard_tag" 2021-03-15T22:18:56.705 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard table charset" 2021-03-15T22:18:56.705 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_tag table charset" 2021-03-15T22:18:56.706 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add column folder_id in dashboard" 2021-03-15T22:18:56.706 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add column isFolder in dashboard" 2021-03-15T22:18:56.973 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add column has_acl in dashboard" 2021-03-15T22:18:56.973 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in dashboard" 2021-03-15T22:18:56.974 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in dashboard" 2021-03-15T22:18:56.974 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:56 smithi125 conmon[77264]: t=2021-03-15T22:18:56+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index dashboard_org_id_uid" 2021-03-15T22:18:57.137 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:56.055718 2021-03-15T22:18:57.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 272 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:18:57.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: audit 2021-03-15T22:18:56.059011+0000 mon.a (mon.0) 537 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:57.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:56.059915+0000 mgr.y (mgr.14144) 273 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:18:57.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:57.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:57.139 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:57.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.140 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:18:57.141 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:56.060317+0000 mgr.y (mgr.14144) 274 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:57.141 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:57.141 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.141 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:57.141 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.142 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:57.142 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.142 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.142 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:57.143 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:56.060670+0000 mgr.y (mgr.14144) 275 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:57.143 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:57.143 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.143 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:57.143 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.144 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:57.144 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.144 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.144 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:57.145 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:56.061031+0000 mgr.y (mgr.14144) 276 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:18:57.145 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:57.145 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.145 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:57.145 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.146 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:57.146 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.146 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.146 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:18:57.147 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:56.061384+0000 mgr.y (mgr.14144) 277 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:18:57.147 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:57.147 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.147 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:57.147 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.148 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:57.148 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.148 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.148 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:18:57.148 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:56.061730+0000 mgr.y (mgr.14144) 278 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:18:57.149 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:57.149 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.149 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:57.149 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.150 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:57.150 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.150 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.150 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:18:57.151 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:56.062068+0000 mgr.y (mgr.14144) 279 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:18:57.151 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:18:57.151 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.151 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:18:57.151 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.152 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:18:57.152 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.152 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.152 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:18:57.152 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:56.062404+0000 mgr.y (mgr.14144) 280 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-03-15T22:18:57.153 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:18:56.064573+0000 mgr.y (mgr.14144) 281 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi076 2021-03-15T22:18:57.153 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cluster 2021-03-15T22:18:56.337223+0000 osd.4 (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:57.153 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cluster 2021-03-15T22:18:56.337345+0000 osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:57.153 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cluster 2021-03-15T22:18:56.832633+0000 osd.0 (osd.0) 9 : cluster [DBG] 4.3 deep-scrub ok 2021-03-15T22:18:57.154 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[56514]: cluster 2021-03-15T22:18:56.958598+0000 mgr.y (mgr.14144) 282 : cluster [DBG] pgmap v216: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:57.154 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:57.155 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.155 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021 2021-03-15T22:18:57.155 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: - 2021-03-15T22:18:57.155 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 03 2021-03-15T22:18:57.156 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: - 2021-03-15T22:18:57.156 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 15 2021-03-15T22:18:57.156 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: T 2021-03-15T22:18:57.156 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 22 2021-03-15T22:18:57.157 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: : 2021-03-15T22:18:57.157 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 18 2021-03-15T22:18:57.157 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: : 2021-03-15T22:18:57.157 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 56 2021-03-15T22:18:57.157 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: . 2021-03-15T22:18:57.158 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 055718 2021-03-15T22:18:57.158 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: +0000 2021-03-15T22:18:57.158 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.158 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: mgr.y 2021-03-15T22:18:57.159 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: ( 2021-03-15T22:18:57.159 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: mgr 2021-03-15T22:18:57.159 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: . 2021-03-15T22:18:57.159 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 14144 2021-03-15T22:18:57.159 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: ) 2021-03-15T22:18:57.160 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 272 2021-03-15T22:18:57.160 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: : 2021-03-15T22:18:57.160 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:57.160 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.160 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: [INF] 2021-03-15T22:18:57.161 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.161 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:18:57.161 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.161 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: audit 2021-03-15T22:18:57.162 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.162 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021 2021-03-15T22:18:57.162 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: - 2021-03-15T22:18:57.162 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 03 2021-03-15T22:18:57.163 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: - 2021-03-15T22:18:57.163 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 15 2021-03-15T22:18:57.163 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: T 2021-03-15T22:18:57.163 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 22 2021-03-15T22:18:57.163 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: : 2021-03-15T22:18:57.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 18 2021-03-15T22:18:57.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: : 2021-03-15T22:18:57.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 56 2021-03-15T22:18:57.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: . 2021-03-15T22:18:57.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 059011 2021-03-15T22:18:57.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: +0000 2021-03-15T22:18:57.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: mon.a 2021-03-15T22:18:57.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: ( 2021-03-15T22:18:57.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: mon 2021-03-15T22:18:57.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: . 2021-03-15T22:18:57.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 0 2021-03-15T22:18:57.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: ) 2021-03-15T22:18:57.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 537 2021-03-15T22:18:57.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: : 2021-03-15T22:18:57.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: audit 2021-03-15T22:18:57.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: [INF] 2021-03-15T22:18:57.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:57.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:57.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021 2021-03-15T22:18:57.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: - 2021-03-15T22:18:57.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 03 2021-03-15T22:18:57.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: - 2021-03-15T22:18:57.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 15 2021-03-15T22:18:57.170 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: T 2021-03-15T22:18:57.170 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 22 2021-03-15T22:18:57.170 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: : 2021-03-15T22:18:57.170 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 18 2021-03-15T22:18:57.170 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: :56.059915+0000 mgr.y (mgr 2021-03-15T22:18:57.171 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: .14144) 273 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:18:57.171 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:57.171 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.171 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:57.171 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.172 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:57.172 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.172 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.172 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:18:57.173 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:56.060317+0000 mgr.y (mgr.14144 2021-03-15T22:18:57.173 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: ) 274 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:57.173 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:57.173 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.173 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:57.174 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.174 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:57.174 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.174 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.175 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:57.175 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18:57.175 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:56.060670+0000 mgr.y (mgr.14144 2021-03-15T22:18:57.175 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: ) 275 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:57.175 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:57.176 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.176 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:57.176 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.176 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:57.176 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.177 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.177 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:57.177 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:56.061031+0000 mgr.y (mgr.14144) 276 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:18:57.177 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:57.178 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.178 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:57.178 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.178 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:57.178 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:18:57.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:56.061384+0000 mgr.y (mgr.14144) 277 : 2021-03-15T22:18:57.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:18:57.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:57.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:57.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:57.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:18:57.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:56.061730+0000 mgr.y (mgr.14144 2021-03-15T22:18:57.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: ) 278 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:18:57.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:57.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:57.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:57.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:18:57.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:56.062068+0000 mgr.y (mgr.14144) 279 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:18:57.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:18:57.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:18:57.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:18:57.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:18:57.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm 2021-03 2021-03-15T22:18:57.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: -15T22:18:56.062404+0000 mgr.y (mgr.14144) 280 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-03-15T22:18:57.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:18:56.064573+0000 mgr.y (mgr. 2021-03-15T22:18:57.186 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 14144) 281 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi076 2021-03-15T22:18:57.186 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:18:57.186 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 22:18:56.337223+0000 2021-03-15T22:18:57.186 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: osd.4 (osd.4) 5 2021-03-15T22:18:57.187 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:57.187 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:18:57.187 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: -03-15T22:18:56 2021-03-15T22:18:57.187 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: .337345+0000 osd.4 (osd 2021-03-15T22:18:57.187 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: .4) 6 : cluster [DBG] 2021-03-15T22:18:57.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: purged_snaps scrub ok 2021-03-15T22:18:57.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:18:57.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 03-15T22:18:56 2021-03-15T22:18:57.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: .832633+0000 osd.0 (osd 2021-03-15T22:18:57.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: .0) 9 : cluster [DBG] 2021-03-15T22:18:57.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 4.3 deep-scrub ok 2021-03-15T22:18:57.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: cluster 2021-03-15T22:18:57.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: 2021-03-15T22:18 2021-03-15T22:18:57.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: :56.958598+0000 mgr.y 2021-03-15T22:18:57.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: (mgr.14144) 282 : cluster 2021-03-15T22:18:57.190 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:57 smithi076 conmon[62156]: [DBG] pgmap v216: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:57.257 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_slug" 2021-03-15T22:18:57.258 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard title length" 2021-03-15T22:18:57.258 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index for dashboard_org_id_title_folder_id" 2021-03-15T22:18:57.258 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning" 2021-03-15T22:18:57.259 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:56.055718+0000 mgr.y (mgr.14144 2021-03-15T22:18:57.259 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: ) 272 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:18:57.259 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: audit 2021-03-15T22:18:56.059011+0000 mon.a (mon.0) 537 : 2021-03-15T22:18:57.260 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:18:57.260 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:57.260 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 2021-03-15T22:18:56.059915+0000 mgr.y (mgr.14144) 2021-03-15T22:18:57.260 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 273 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:18:57.261 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:57.261 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.261 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:57.261 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.261 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:57.262 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.262 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.262 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:18:57.262 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:56.060317+0000 mgr.y (mgr.14144) 274 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:18:57.263 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:57.263 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.263 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:57.263 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.264 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:57.264 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.264 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.264 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:18:57.264 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:57.265 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 2021-03-15T22:18:56.060670+0000 mgr.y (mgr.14144) 275 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:18:57.265 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:57.265 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.265 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:57.266 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.266 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:57.266 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.266 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.267 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:18:57.267 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:57.267 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 2021-03-15T22:18:56.061031+0000 mgr.y (mgr.14144) 276 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:18:57.267 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:57.267 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.268 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:57.268 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.268 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:57.268 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.268 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.269 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:18:57.269 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:56.061384 2021-03-15T22:18:57.269 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 277 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:18:57.269 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:57.270 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.270 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:57.270 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.270 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:57.271 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.271 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.271 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:18:57.271 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:56. 2021-03-15T22:18:57.271 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 061730+0000 mgr.y (mgr.14144) 278 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:18:57.272 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:57.272 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.272 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:57.272 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.273 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:57.273 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.273 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.273 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:18:57.274 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:18: 2021-03-15T22:18:57.274 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 56.062068+0000 mgr.y (mgr.14144) 279 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:18:57.274 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:18:57.274 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:18:57.275 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:18:57.275 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:18:57.275 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:18:57.275 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:18:57.275 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:18:57.276 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:18:57.276 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:56.062404+0000 2021-03-15T22:18:57.276 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: mgr.y (mgr.14144) 280 : cephadm [INF] Reconfiguring alertmanager.a (dependencies changed)... 2021-03-15T22:18:57.276 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:18:56.064573+0000 mgr.y (mgr.14144) 281 : cephadm [INF] Reconfiguring daemon alertmanager.a on smithi076 2021-03-15T22:18:57.276 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cluster 2021-03-15T22:18:57.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 2021-03-15T22:18:56.337223+0000 osd.4 2021-03-15T22:18:57.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: (osd.4) 5 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:18:57.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cluster 2021-03-15T22:18:57.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 2021-03-15T22:18:56.337345+0000 2021-03-15T22:18:57.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: osd.4 (osd.4) 6 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:18:57.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 2021-03-15T22:18:57.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cluster 2021-03-15T22:18:57.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:18:57.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 18:56.832633+0000 osd.0 (osd.0) 2021-03-15T22:18:57.279 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 9 : cluster [DBG] 4.3 deep-scrub ok 2021-03-15T22:18:57.279 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: cluster 2021-03-15T22:18:57.279 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:18:57.279 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 18:56.958598+0000 mgr.y (mgr.14144) 2021-03-15T22:18:57.280 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[56739]: 282 : cluster [DBG] pgmap v216: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:18:57.518 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2021-03-15T22:18:57.518 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_provisioning v2" 2021-03-15T22:18:57.518 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2021-03-15T22:18:57.519 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2021-03-15T22:18:57.519 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="copy dashboard_provisioning v1 to v2" 2021-03-15T22:18:57.786 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop dashboard_provisioning_tmp_qwerty" 2021-03-15T22:18:57.786 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="Add check_sum column" 2021-03-15T22:18:57.786 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table" 2021-03-15T22:18:57.786 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="add index data_source.account_id" 2021-03-15T22:18:58.036 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index data_source.account_id_name" 2021-03-15T22:18:58.036 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:57 smithi125 conmon[77264]: t=2021-03-15T22:18:57+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_data_source_account_id - v1" 2021-03-15T22:18:58.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:58 smithi076 conmon[56514]: cluster 2021-03-15T22:18:57.065277+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-03-15T22:18:58.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:58 smithi076 conmon[56514]: 2021-03-15T22:18:58.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:58 smithi076 conmon[56514]: cluster 2021-03-15T22:18:57.649995+0000 osd.3 (osd.3) 7 : cluster [DBG] 3.b scrub ok 2021-03-15T22:18:58.337 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:58 smithi076 conmon[62156]: cluster 2021-03-15T22:18:57. 2021-03-15T22:18:58.337 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:58 smithi076 conmon[62156]: 065277+0000 mon.a (mon.0) 538 : cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-03-15T22:18:58.338 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:58 smithi076 conmon[62156]: cluster 2021-03-15T22:18:57.649995+0000 osd.3 (osd.3) 7 : cluster [DBG] 3.b scrub ok 2021-03-15T22:18:58.363 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[56739]: cluster 2021-03-15T22:18:57.065277+0000 mon.a (mon.0) 538 : 2021-03-15T22:18:58.363 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[56739]: cluster [DBG] osdmap e79: 8 total, 8 up, 8 in 2021-03-15T22:18:58.364 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[56739]: cluster 2021-03-15T22:18:57.649995+0000 osd.3 ( 2021-03-15T22:18:58.364 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[56739]: osd.3) 7 : cluster [DBG] 3.b scrub ok 2021-03-15T22:18:58.364 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_data_source_account_id_name - v1" 2021-03-15T22:18:58.365 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table data_source to data_source_v1 - v1" 2021-03-15T22:18:58.365 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="create data_source table v2" 2021-03-15T22:18:58.365 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_data_source_org_id - v2" 2021-03-15T22:18:58.365 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_data_source_org_id_name - v2" 2021-03-15T22:18:58.614 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="copy data_source v1 to v2" 2021-03-15T22:18:58.614 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table data_source_v1 #2" 2021-03-15T22:18:58.615 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add column with_credentials" 2021-03-15T22:18:58.651 INFO:teuthology.orchestra.run.smithi076.stdout:77309411362 2021-03-15T22:18:58.651 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph tell osd.3 flush_pg_stats 2021-03-15T22:18:58.899 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add secure json data column" 2021-03-15T22:18:58.899 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update data_source table charset" 2021-03-15T22:18:58.899 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="Update initial version to 1" 2021-03-15T22:18:59.166 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[56739]: cluster 2021-03-15T22:18:58.347611+0000 osd.5 (osd.5) 6 : 2021-03-15T22:18:59.167 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[56739]: cluster [DBG] 5.f scrub ok 2021-03-15T22:18:59.167 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[56739]: cluster 2021-03-15T22:18:58.959053+0000 mgr.y (mgr.14144) 2021-03-15T22:18:59.167 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[56739]: 283 : cluster [DBG] pgmap v218: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2021-03-15T22:18:59.168 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="Add read_only data column" 2021-03-15T22:18:59.168 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:58 smithi125 conmon[77264]: t=2021-03-15T22:18:58+0000 lvl=info msg="Executing migration" logger=migrator id="Migrate logging ds to loki ds" 2021-03-15T22:18:59.169 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="Update json_data with nulls" 2021-03-15T22:18:59.169 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table" 2021-03-15T22:18:59.169 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id" 2021-03-15T22:18:59.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:59 smithi076 conmon[56514]: cluster 2021-03-15T22:18:58.347611+0000 osd.5 (osd.5) 6 : cluster [DBG] 5.f scrub ok 2021-03-15T22:18:59.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:18:59 smithi076 conmon[56514]: cluster 2021-03-15T22:18:58.959053+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v218: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2021-03-15T22:18:59.174 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:59 smithi076 conmon[62156]: cluster 2021-03-15T22:18:58.347611+0000 osd.5 (osd.5) 6 : cluster [DBG] 5.f scrub ok 2021-03-15T22:18:59.175 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:18:59 smithi076 conmon[62156]: cluster 2021-03-15T22:18:58.959053+0000 mgr.y (mgr.14144) 283 : cluster [DBG] pgmap v218: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 569 B/s rd, 0 op/s 2021-03-15T22:18:59.176 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:18:59.460 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:18:59.460 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.key" 2021-03-15T22:18:59.461 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="add index api_key.account_id_name" 2021-03-15T22:18:59.461 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_api_key_account_id - v1" 2021-03-15T22:18:59.461 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_key - v1" 2021-03-15T22:18:59.735 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="drop index UQE_api_key_account_id_name - v1" 2021-03-15T22:18:59.736 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table api_key to api_key_v1 - v1" 2021-03-15T22:18:59.736 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="create api_key table v2" 2021-03-15T22:18:59.736 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_api_key_org_id - v2" 2021-03-15T22:19:00.036 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_key - v2" 2021-03-15T22:19:00.037 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_api_key_org_id_name - v2" 2021-03-15T22:19:00.037 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:18:59 smithi125 conmon[77264]: t=2021-03-15T22:18:59+0000 lvl=info msg="Executing migration" logger=migrator id="copy api_key v1 to v2" 2021-03-15T22:19:00.064 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:59 smithi076 systemd[1]: Stopping Ceph alertmanager.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:19:00.065 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:18:59 smithi076 conmon[81441]: level=info ts=2021-03-15T22:18:59.945Z caller=main.go:536 msg="Received SIGTERM, exiting gracefully..." 2021-03-15T22:19:00.322 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table api_key_v1" 2021-03-15T22:19:00.322 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update api_key table charset" 2021-03-15T22:19:00.323 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add expires to api_key table" 2021-03-15T22:19:00.323 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v4" 2021-03-15T22:19:00.324 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[56739]: cluster 2021-03-15T22:18:59.037497+0000 osd.4 ( 2021-03-15T22:19:00.324 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[56739]: osd.4) 7 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:19:00.324 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:19:00.325 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[56739]: 18:59.343433+0000 mgr.y (mgr.14144) 284 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:00.325 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:19:00.325 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[56739]: T22:18:59.667382+0000 osd.3 (osd.3 2021-03-15T22:19:00.325 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[56739]: ) 8 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:19:00.325 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:19:00.326 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[56739]: 03-15T22:18:59.967830+0000 mon.a (mon 2021-03-15T22:19:00.326 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[56739]: .0) 539 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2021-03-15T22:19:00.496 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[56514]: cluster 2021-03-15T22:18:59.037497+0000 osd.4 2021-03-15T22:19:00.496 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[56514]: (osd.4) 7 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:19:00.496 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[56514]: audit 2021-03-15T22:18:59.343433+0000 mgr.y (mgr.14144) 284 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:00.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[56514]: cluster 2021-03-15T22:18:59.667382+0000 osd.3 (osd.3) 8 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:19:00.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:19:00.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[56514]: 15T22:18:59.967830+0000 mon.a (mon.0) 539 : cluster [DBG] osdmap e80: 8 total, 8 up, 8 in 2021-03-15T22:19:00.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: cluster 2021-03-15T22:18:59.037497+0000 2021-03-15T22:19:00.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: osd.4 (osd.4) 7 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:19:00.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: audit 2021-03-15T22:18:59. 2021-03-15T22:19:00.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: 343433+0000 mgr.y (mgr.14144) 284 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:00.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: 2021-03-15T22:19:00.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:19:00.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: -03-15T22:18: 2021-03-15T22:19:00.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: 59.667382+0000 osd.3 (osd.3) 2021-03-15T22:19:00.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: 8 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:19:00.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:19:00.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: -03-15T22:18: 2021-03-15T22:19:00.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: 59.967830+0000 mon.a (mon 2021-03-15T22:19:00.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: .0) 539 : cluster [DBG] 2021-03-15T22:19:00.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:00 smithi076 conmon[62156]: osdmap e80: 8 total, 8 up, 8 in 2021-03-15T22:19:00.502 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:00 smithi076 podman[84951]: 2021-03-15 22:19:00.095792075 +0000 UTC m=+0.246763497 container stop 979ec48afd0e4720782d746f9e0da009d1b6e60e482841b43fa462157861ad8e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-15T22:19:00.502 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:00 smithi076 podman[84951]: 2021-03-15 22:19:00.144779793 +0000 UTC m=+0.295751228 container died 979ec48afd0e4720782d746f9e0da009d1b6e60e482841b43fa462157861ad8e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-alertmanager.a) 2021-03-15T22:19:00.503 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:00 smithi076 podman[84951]: 979ec48afd0e4720782d746f9e0da009d1b6e60e482841b43fa462157861ad8e 2021-03-15T22:19:00.572 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="drop table dashboard_snapshot_v4 #1" 2021-03-15T22:19:00.573 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_snapshot table v5 #2" 2021-03-15T22:19:00.573 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_key - v5" 2021-03-15T22:19:00.849 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_dashboard_snapshot_delete_key - v5" 2021-03-15T22:19:00.849 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_dashboard_snapshot_user_id - v5" 2021-03-15T22:19:00.849 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_snapshot to mediumtext v2" 2021-03-15T22:19:00.850 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="Update dashboard_snapshot table charset" 2021-03-15T22:19:01.134 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external_delete_url to dashboard_snapshots table" 2021-03-15T22:19:01.135 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:00 smithi125 conmon[77264]: t=2021-03-15T22:19:00+0000 lvl=info msg="Executing migration" logger=migrator id="create quota table v1" 2021-03-15T22:19:01.135 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_quota_org_id_user_id_target - v1" 2021-03-15T22:19:01.135 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update quota table charset" 2021-03-15T22:19:01.136 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: cluster 2021-03-15T22:18:59.946557+0000 osd.7 2021-03-15T22:19:01.137 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: (osd.7) 5 : cluster [DBG] 3.e scrub ok 2021-03-15T22:19:01.137 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: cluster 2021-03-15T22:19:00.305798+0000 osd.5 (osd.5) 7 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:19:01.137 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: audit 2021-03-15T22:19:00.914551+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2021-03-15T22:19:01.138 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: 2021-03-15T22:19:01.138 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: audit 2021 2021-03-15T22:19:01.139 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: -03-15T22:19:00.914764+0000 mon.a (mon.0) 541 : 2021-03-15T22:19:01.139 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2021-03-15T22:19:01.140 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:19:01.140 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: 19:00.914924+0000 mon.a (mon.0) 542 : 2021-03-15T22:19:01.140 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2021-03-15T22:19:01.141 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:19:01.141 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: -03-15T22:19:00.959717+0000 2021-03-15T22:19:01.141 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v220: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:01.142 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: 2021-03-15T22:19:01.142 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:19:01.143 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: -15T22:19:01.036737+0000 2021-03-15T22:19:01.144 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: mon.a (mon.0) 543 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:19:01.144 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:19:01.144 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: 19:01.037875+0000 mon.a (mon.0) 544 : audit [INF] 2021-03-15T22:19:01.145 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:19:01.385 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="create plugin_setting table" 2021-03-15T22:19:01.386 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2021-03-15T22:19:01.386 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="Add column plugin_version to plugin_settings" 2021-03-15T22:19:01.386 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update plugin_setting table charset" 2021-03-15T22:19:01.461 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:01 smithi076 podman[84997]: 2021-03-15 22:19:01.09778346 +0000 UTC m=+1.095558217 container remove 979ec48afd0e4720782d746f9e0da009d1b6e60e482841b43fa462157861ad8e (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-15T22:19:01.461 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:01 smithi076 podman[84997]: 2021-03-15 22:19:01.109697651 +0000 UTC m=+1.107472303 volume remove 449ecb7136938e2a61d5eb3d296efd97fcff415fdc2720c9e35f0ef5d659d3ee 2021-03-15T22:19:01.461 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:01 smithi076 systemd[1]: Stopped Ceph alertmanager.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:19:01.462 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:01 smithi076 systemd[1]: Starting Ceph alertmanager.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:19:01.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[56514]: cluster 2021-03-15T22:18:59.946557+0000 osd.7 (osd.7) 5 2021-03-15T22:19:01.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[56514]: : cluster [DBG] 3.e scrub ok 2021-03-15T22:19:01.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[56514]: cluster 2021-03-15T22:19:00.305798+0000 osd.5 (osd.5) 7 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:19:01.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[56514]: audit 2021-03-15T22:19:00.914551+0000 mon.a (mon.0) 540 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2021-03-15T22:19:01.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[56514]: audit 2021-03-15T22:19:00.914764+0000 mon.a (mon.0) 541 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2021-03-15T22:19:01.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[56514]: audit 2021-03-15T22:19:00.914924+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2021-03-15T22:19:01.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[56514]: cluster 2021-03-15T22:19:00.959717+0000 mgr.y (mgr.14144) 285 : cluster [DBG] pgmap v220: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:01.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[56514]: audit 2021-03-15T22:19:01.036737+0000 mon.a (mon.0) 543 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:19:01.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[56514]: audit 2021-03-15T22:19:01.037875+0000 mon.a (mon.0) 544 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:19:01.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: cluster 2021-03-15T22:19:01.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 2021-03-15T22:19:01.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 2021 2021-03-15T22:19:01.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: - 2021-03-15T22:19:01.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 03 2021-03-15T22:19:01.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: - 2021-03-15T22:19:01.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 15 2021-03-15T22:19:01.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: T 2021-03-15T22:19:01.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 22 2021-03-15T22:19:01.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: : 2021-03-15T22:19:01.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 18 2021-03-15T22:19:01.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: : 2021-03-15T22:19:01.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 59 2021-03-15T22:19:01.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: . 2021-03-15T22:19:01.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 946557 2021-03-15T22:19:01.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: +0000 2021-03-15T22:19:01.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 2021-03-15T22:19:01.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: osd.7 2021-03-15T22:19:01.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: ( 2021-03-15T22:19:01.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: osd 2021-03-15T22:19:01.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: . 2021-03-15T22:19:01.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 7 2021-03-15T22:19:01.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: ) 2021-03-15T22:19:01.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 5 2021-03-15T22:19:01.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: : 2021-03-15T22:19:01.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: cluster [DBG] 3.e scrub ok 2021-03-15T22:19:01.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: cluster 2021-03-15T22:19:00. 2021-03-15T22:19:01.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 305798+0000 osd.5 (osd.5) 7 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:19:01.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:19:01.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 03-15T22:19:00.914551+0000 mon.a (mon.0) 540 : audit [INF] 2021-03-15T22:19:01.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]: dispatch 2021-03-15T22:19:01.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:19:01.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: T22:19:00.914764+0000 mon.a (mon.0) 541 : audit [INF] 2021-03-15T22:19:01.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]: dispatch 2021-03-15T22:19:01.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: audit 2021-03-15T22:19 2021-03-15T22:19:01.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: :00.914924+0000 mon.a (mon.0) 542 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2021-03-15T22:19:01.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:19:01.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: T22:19:00.959717 2021-03-15T22:19:01.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 285 2021-03-15T22:19:01.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: : cluster [DBG] pgmap v220: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:01.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:19:01.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 15T22:19:01.036737+0000 2021-03-15T22:19:01.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: mon.a (mon.0) 543 : audit 2021-03-15T22:19:01.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:19:01.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: audit 2021-03-15T22:19:01.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 2021-03-15T22:19:01.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 2021 2021-03-15T22:19:01.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: - 2021-03-15T22:19:01.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 03 2021-03-15T22:19:01.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: - 2021-03-15T22:19:01.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 15 2021-03-15T22:19:01.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: T 2021-03-15T22:19:01.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 22 2021-03-15T22:19:01.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: : 2021-03-15T22:19:01.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 19 2021-03-15T22:19:01.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: : 2021-03-15T22:19:01.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 01 2021-03-15T22:19:01.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: . 2021-03-15T22:19:01.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 037875 2021-03-15T22:19:01.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: +0000 2021-03-15T22:19:01.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 2021-03-15T22:19:01.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: mon.a 2021-03-15T22:19:01.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: ( 2021-03-15T22:19:01.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: mon 2021-03-15T22:19:01.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: . 2021-03-15T22:19:01.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 0 2021-03-15T22:19:01.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: ) 2021-03-15T22:19:01.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 544 2021-03-15T22:19:01.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: : 2021-03-15T22:19:01.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: audit 2021-03-15T22:19:01.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 2021-03-15T22:19:01.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: [INF] 2021-03-15T22:19:01.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 2021-03-15T22:19:01.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:19:01.482 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:01 smithi076 conmon[62156]: 2021-03-15T22:19:01.713 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="create session table" 2021-03-15T22:19:01.713 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist table" 2021-03-15T22:19:01.714 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old table playlist_item table" 2021-03-15T22:19:01.714 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist table v2" 2021-03-15T22:19:01.714 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="create playlist item table v2" 2021-03-15T22:19:01.979 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist table charset" 2021-03-15T22:19:01.980 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="Update playlist_item table charset" 2021-03-15T22:19:01.980 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v2" 2021-03-15T22:19:01.980 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:01 smithi125 conmon[77264]: t=2021-03-15T22:19:01+0000 lvl=info msg="Executing migration" logger=migrator id="drop preferences table v3" 2021-03-15T22:19:02.017 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:01 smithi076 podman[85203]: 2021-03-15 22:19:01.742703474 +0000 UTC m=+0.331310612 volume create 5570fd27d97091c455b766ed176367d89ebae18b459368b20e7bf124b94effd0 2021-03-15T22:19:02.017 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:01 smithi076 podman[85203]: 2021-03-15 22:19:01.784253467 +0000 UTC m=+0.372860598 container create 8c6e56e6b90f1141e709b0d5125f80d3767117866ae86adb243fcdeadf83e388 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-15T22:19:02.264 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[77264]: t=2021-03-15T22:19:02+0000 lvl=info msg="Executing migration" logger=migrator id="create preferences table v3" 2021-03-15T22:19:02.264 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[77264]: t=2021-03-15T22:19:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update preferences table charset" 2021-03-15T22:19:02.264 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[77264]: t=2021-03-15T22:19:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add column team_id in preferences" 2021-03-15T22:19:02.265 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: cluster 2021-03-15T22:19:01.036032+0000 osd.2 (osd.2) 7 : cluster 2021-03-15T22:19:02.265 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: [DBG] 5.a scrub ok 2021-03-15T22:19:02.265 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: audit 2021-03-15T22:19:01.081422+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2021-03-15T22:19:02.266 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: 2021-03-15T22:19:02.266 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: audit 2021-03-15T22:19 2021-03-15T22:19:02.266 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: :01.081507+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2021-03-15T22:19:02.266 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: 2021-03-15T22:19:02.267 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: audit 2021-03-15T22:19:02.267 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: 2021-03-15T22:19:01.081581+0000 mon.a 2021-03-15T22:19:02.267 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: (mon.0) 547 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2021-03-15T22:19:02.267 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:19:02.267 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: -03-15T22:19:01.081642 2021-03-15T22:19:02.268 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: +0000 mon.a (mon.0) 548 : cluster 2021-03-15T22:19:02.268 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: [DBG] osdmap e81: 8 total, 8 up, 8 in 2021-03-15T22:19:02.269 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:19:02.269 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: :19:01.740488+0000 osd.0 (osd. 2021-03-15T22:19:02.269 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[56739]: 0) 10 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:19:02.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[56514]: cluster 2021-03-15T22:19:01.036032 2021-03-15T22:19:02.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[56514]: +0000 osd.2 (osd.2) 7 : cluster [DBG] 5.a scrub ok 2021-03-15T22:19:02.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[56514]: audit 2021-03-15T22:19:01.081422+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2021-03-15T22:19:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[56514]: audit 2021-03-15T22:19:01.081507+0000 mon.a (mon.0) 546 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2021-03-15T22:19:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[56514]: audit 2021-03-15T22:19:01.081581+0000 mon.a (mon.0) 547 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2021-03-15T22:19:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[56514]: cluster 2021-03-15T22:19:01.081642+0000 mon.a (mon.0) 548 : cluster [DBG] osdmap e81: 8 total, 8 up, 8 in 2021-03-15T22:19:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[56514]: cluster 2021-03-15T22:19:01.740488+0000 osd.0 (osd.0) 10 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:19:02.519 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:02 smithi076 podman[85203]: 2021-03-15 22:19:02.084228289 +0000 UTC m=+0.672835427 container init 8c6e56e6b90f1141e709b0d5125f80d3767117866ae86adb243fcdeadf83e388 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-15T22:19:02.519 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:02 smithi076 podman[85203]: 2021-03-15 22:19:02.117525573 +0000 UTC m=+0.706132691 container start 8c6e56e6b90f1141e709b0d5125f80d3767117866ae86adb243fcdeadf83e388 (image=docker.io/prom/alertmanager:v0.20.0, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-alertmanager.a, maintainer=The Prometheus Authors ) 2021-03-15T22:19:02.519 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:02 smithi076 bash[85086]: 8c6e56e6b90f1141e709b0d5125f80d3767117866ae86adb243fcdeadf83e388 2021-03-15T22:19:02.519 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:02 smithi076 systemd[1]: Started Ceph alertmanager.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:19:02.520 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[85285]: level=info ts=2021-03-15T22:19:02.127Z caller=main.go:231 msg="Starting Alertmanager" version="(version=0.20.0, branch=HEAD, revision=f74be0400a6243d10bb53812d6fa408ad71ff32d)" 2021-03-15T22:19:02.520 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[85285]: level=info ts=2021-03-15T22:19:02.127Z caller=main.go:232 build_context="(go=go1.13.5, user=root@00c3106655f8, date=20191211-14:13:14)" 2021-03-15T22:19:02.520 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[85285]: level=info ts=2021-03-15T22:19:02.129Z caller=cluster.go:161 component=cluster msg="setting advertise address explicitly" addr=172.21.15.76 port=9094 2021-03-15T22:19:02.520 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[85285]: level=info ts=2021-03-15T22:19:02.130Z caller=cluster.go:623 component=cluster msg="Waiting for gossip to settle..." interval=2s 2021-03-15T22:19:02.521 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[85285]: level=info ts=2021-03-15T22:19:02.155Z caller=coordinator.go:119 component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2021-03-15T22:19:02.521 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[85285]: level=info ts=2021-03-15T22:19:02.155Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2021-03-15T22:19:02.521 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[85285]: level=info ts=2021-03-15T22:19:02.158Z caller=main.go:497 msg=Listening address=:9093 2021-03-15T22:19:02.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: cluster 2021-03-15T22:19:01.036032 2021-03-15T22:19:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: +0000 osd.2 (osd.2) 7 : cluster [DBG] 5.a scrub ok 2021-03-15T22:19:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: audit 2021-03-15T22:19 2021-03-15T22:19:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: :01.081422+0000 mon.a (mon.0) 545 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.b", "id": [1, 2]}]': finished 2021-03-15T22:19:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: audit 2021-03-15T22:19:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: 2021-03-15T22:19:01.081507+0000 mon.a (mon.0 2021-03-15T22:19:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: ) 546 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.f", "id": [1, 2]}]': finished 2021-03-15T22:19:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:19:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: T22:19:01.081581+0000 mon.a (mon.0) 547 : 2021-03-15T22:19:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2021-03-15T22:19:02.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: cluster 2021-03-15T22:19:02.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: 2021-03-15T22:19:01.081642+0000 mon.a (mon.0) 548 : cluster [DBG] 2021-03-15T22:19:02.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: osdmap e81: 8 total, 8 up, 8 in 2021-03-15T22:19:02.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: cluster 2021-03-15T22:19 2021-03-15T22:19:02.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:02 smithi076 conmon[62156]: :01.740488+0000 osd.0 (osd.0) 10 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:19:02.536 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[77264]: t=2021-03-15T22:19:02+0000 lvl=info msg="Executing migration" logger=migrator id="Update team_id column values in preferences" 2021-03-15T22:19:02.536 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[77264]: t=2021-03-15T22:19:02+0000 lvl=info msg="Executing migration" logger=migrator id="create alert table v1" 2021-03-15T22:19:02.537 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[77264]: t=2021-03-15T22:19:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert org_id & id " 2021-03-15T22:19:02.840 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[77264]: t=2021-03-15T22:19:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert state" 2021-03-15T22:19:02.841 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[77264]: t=2021-03-15T22:19:02+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert dashboard_id" 2021-03-15T22:19:02.841 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[77264]: t=2021-03-15T22:19:02+0000 lvl=info msg="Executing migration" logger=migrator id="Create alert_rule_tag table v1" 2021-03-15T22:19:02.883 INFO:teuthology.orchestra.run.smithi076.stdout:111669149727 2021-03-15T22:19:02.884 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph tell osd.4 flush_pg_stats 2021-03-15T22:19:03.175 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:03.182 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[77264]: t=2021-03-15T22:19:02+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_rule_tag.alert_id_tag_id" 2021-03-15T22:19:03.182 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:02 smithi125 conmon[77264]: t=2021-03-15T22:19:02+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification table v1" 2021-03-15T22:19:03.182 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[77264]: t=2021-03-15T22:19:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column is_default" 2021-03-15T22:19:03.183 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: cluster 2021-03-15T22:19:02.085821+0000 mon.a (mon. 2021-03-15T22:19:03.183 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: 0) 549 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2021-03-15T22:19:03.183 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: audit 2021-03-15T22:19:02.150654+0000 mon.a (mon 2021-03-15T22:19:03.183 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: .0) 550 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:03.184 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:03.184 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:19:03.184 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: 19:02.151568+0000 mgr.y (mgr 2021-03-15T22:19:03.184 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: .14144) 286 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-03-15T22:19:03.185 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: 2021-03-15T22:19:03.185 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:03.185 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: 2021-03-15T22:19:02. 2021-03-15T22:19:03.185 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: 154790+0000 mgr.y (mgr.14144) 287 : 2021-03-15T22:19:03.186 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: cephadm [INF] Reconfiguring daemon prometheus.a on smithi125 2021-03-15T22:19:03.186 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:19:03.186 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: -03-15T22:19:02.960158+0000 mgr.y (mgr 2021-03-15T22:19:03.186 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: .14144) 288 : cluster [DBG] pgmap v223: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2021-03-15T22:19:03.186 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[56739]: 2021-03-15T22:19:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:03 smithi076 conmon[56514]: cluster 2021-03-15T22:19:02.085821+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2021-03-15T22:19:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:03 smithi076 conmon[56514]: audit 2021-03-15T22:19:02.150654+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:03 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:02.151568+0000 mgr.y (mgr.14144) 286 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-03-15T22:19:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:03 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:02.154790+0000 mgr.y (mgr.14144) 287 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi125 2021-03-15T22:19:03.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:03 smithi076 conmon[56514]: cluster 2021-03-15T22:19:02.960158+0000 mgr.y (mgr.14144) 288 : cluster [DBG] pgmap v223: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2021-03-15T22:19:03.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:03 smithi076 conmon[62156]: cluster 2021-03-15T22:19:02.085821+0000 mon.a (mon.0) 549 : cluster [DBG] osdmap e82: 8 total, 8 up, 8 in 2021-03-15T22:19:03.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:03 smithi076 conmon[62156]: audit 2021-03-15T22:19:02.150654+0000 mon.a (mon.0) 550 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:03 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:02.151568+0000 mgr.y (mgr.14144) 286 : cephadm [INF] Reconfiguring prometheus.a (dependencies changed)... 2021-03-15T22:19:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:03 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:02.154790+0000 mgr.y (mgr.14144) 287 : cephadm [INF] Reconfiguring daemon prometheus.a on smithi125 2021-03-15T22:19:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:03 smithi076 conmon[62156]: cluster 2021-03-15T22:19:02.960158+0000 mgr.y (mgr.14144) 288 : cluster [DBG] pgmap v223: 132 pgs: 1 peering, 131 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 868 B/s rd, 0 op/s 2021-03-15T22:19:03.519 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[77264]: t=2021-03-15T22:19:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column frequency" 2021-03-15T22:19:03.520 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[77264]: t=2021-03-15T22:19:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column send_reminder" 2021-03-15T22:19:03.520 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[77264]: t=2021-03-15T22:19:03+0000 lvl=info msg="Executing migration" logger=migrator id="Add column disable_resolve_message" 2021-03-15T22:19:04.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:04 smithi125 conmon[56739]: cluster 2021-03-15T22:19:03.699192+0000 osd.3 (osd.3 2021-03-15T22:19:04.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:04 smithi125 conmon[56739]: ) 9 : cluster [DBG] 5.13 deep-scrub ok 2021-03-15T22:19:04.287 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:03 smithi125 conmon[77264]: t=2021-03-15T22:19:03+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification org_id & name" 2021-03-15T22:19:04.288 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:04 smithi125 conmon[77264]: t=2021-03-15T22:19:04+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert table charset" 2021-03-15T22:19:04.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:04 smithi076 conmon[56514]: cluster 2021-03-15T22:19:03.699192+0000 osd.3 (osd.3 2021-03-15T22:19:04.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:04 smithi076 conmon[56514]: ) 9 : cluster [DBG] 5.13 deep-scrub ok 2021-03-15T22:19:04.419 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:04 smithi076 conmon[85285]: level=info ts=2021-03-15T22:19:04.131Z caller=cluster.go:648 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000089737s 2021-03-15T22:19:04.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:04 smithi076 conmon[62156]: cluster 2021-03-15T22:19:03.699192+0000 osd.3 (osd.3) 2021-03-15T22:19:04.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:04 smithi076 conmon[62156]: 9 : cluster [DBG] 5.13 deep-scrub ok 2021-03-15T22:19:04.604 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:04 smithi125 conmon[77264]: t=2021-03-15T22:19:04+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert_notification table charset" 2021-03-15T22:19:04.604 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:04 smithi125 conmon[77264]: t=2021-03-15T22:19:04+0000 lvl=info msg="Executing migration" logger=migrator id="create notification_journal table v1" 2021-03-15T22:19:04.604 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:04 smithi125 conmon[77264]: t=2021-03-15T22:19:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index notification_journal org_id & alert_id & notifier_id" 2021-03-15T22:19:04.909 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:04 smithi125 conmon[77264]: t=2021-03-15T22:19:04+0000 lvl=info msg="Executing migration" logger=migrator id="drop alert_notification_journal" 2021-03-15T22:19:04.909 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:04 smithi125 conmon[77264]: t=2021-03-15T22:19:04+0000 lvl=info msg="Executing migration" logger=migrator id="create alert_notification_state table v1" 2021-03-15T22:19:05.214 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:05 smithi125 conmon[56739]: cluster 2021-03-15T22:19:03.966798+0000 osd.7 (osd.7) 6 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:19:05.215 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:05 smithi125 conmon[56739]: cluster 2021-03-15T22:19:04.960869+0000 mgr.y (mgr.14144) 289 : cluster [DBG] pgmap v224: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 43 B/s, 0 objects/s recovering 2021-03-15T22:19:05.215 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:04 smithi125 conmon[77264]: t=2021-03-15T22:19:04+0000 lvl=info msg="Executing migration" logger=migrator id="add index alert_notification_state org_id & alert_id & notifier_id" 2021-03-15T22:19:05.216 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:05 smithi125 conmon[77264]: t=2021-03-15T22:19:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add for to alert table" 2021-03-15T22:19:05.216 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:05 smithi125 conmon[77264]: t=2021-03-15T22:19:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add column uid in alert_notification" 2021-03-15T22:19:05.333 INFO:teuthology.orchestra.run.smithi076.stdout:133143986204 2021-03-15T22:19:05.333 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph tell osd.5 flush_pg_stats 2021-03-15T22:19:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:05 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:19:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:05 smithi076 conmon[56514]: :19:03.966798+0000 osd.7 (osd.7) 6 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:19:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:05 smithi076 conmon[56514]: cluster 2021-03-15T22:19:04.960869+0000 mgr.y (mgr.14144) 289 : cluster [DBG] pgmap v224: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 43 B/s, 0 objects/s recovering 2021-03-15T22:19:05.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:05 smithi076 conmon[62156]: cluster 2021-03-15T22:19:03.966798+0000 osd.7 2021-03-15T22:19:05.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:05 smithi076 conmon[62156]: (osd.7) 6 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:19:05.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:05 smithi076 conmon[62156]: cluster 2021-03-15T22:19:04.960869+0000 mgr.y (mgr.14144) 289 : cluster [DBG] pgmap v224: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.7 KiB/s rd, 1 op/s; 43 B/s, 0 objects/s recovering 2021-03-15T22:19:05.536 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:05 smithi125 conmon[77264]: t=2021-03-15T22:19:05+0000 lvl=info msg="Executing migration" logger=migrator id="Update uid column values in alert_notification" 2021-03-15T22:19:05.536 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:05 smithi125 conmon[77264]: t=2021-03-15T22:19:05+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index alert_notification_org_id_uid" 2021-03-15T22:19:05.537 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:05 smithi125 conmon[77264]: t=2021-03-15T22:19:05+0000 lvl=info msg="Executing migration" logger=migrator id="Remove unique index org_id_name" 2021-03-15T22:19:05.610 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:05.929 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:05 smithi125 conmon[77264]: t=2021-03-15T22:19:05+0000 lvl=info msg="Executing migration" logger=migrator id="Drop old annotation table v4" 2021-03-15T22:19:06.203 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[56739]: cluster 2021-03-15T22:19:04.746214+0000 osd.3 ( 2021-03-15T22:19:06.203 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[56739]: osd.3) 10 : cluster [DBG] 4.0 deep-scrub ok 2021-03-15T22:19:06.203 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[56739]: cluster 2021-03-15T22:19 2021-03-15T22:19:06.203 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[56739]: :05.557529+0000 osd.6 (osd.6) 6 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:19:06.204 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:05 smithi125 conmon[77264]: t=2021-03-15T22:19:05+0000 lvl=info msg="Executing migration" logger=migrator id="create annotation table v5" 2021-03-15T22:19:06.204 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[77264]: t=2021-03-15T22:19:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 0 v3" 2021-03-15T22:19:06.204 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[77264]: t=2021-03-15T22:19:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 1 v3" 2021-03-15T22:19:06.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:06 smithi076 conmon[56514]: cluster 2021-03-15T22:19:04.746214+0000 osd.3 (osd.3) 10 : cluster [DBG] 4.0 deep-scrub ok 2021-03-15T22:19:06.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:06 smithi076 conmon[56514]: cluster 2021-03-15T22:19:05.557529+0000 osd.6 (osd.6) 6 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:19:06.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:06 smithi076 conmon[62156]: cluster 2021-03-15T22:19:04.746214+0000 osd.3 (osd.3) 10 : cluster [DBG] 4.0 deep-scrub ok 2021-03-15T22:19:06.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:06 smithi076 conmon[62156]: cluster 2021-03-15T22:19:05.557529+0000 osd.6 (osd.6) 6 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:19:06.536 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[77264]: t=2021-03-15T22:19:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 2 v3" 2021-03-15T22:19:06.536 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[77264]: t=2021-03-15T22:19:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 3 v3" 2021-03-15T22:19:06.537 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 systemd[1]: Stopping Ceph prometheus.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:19:06.537 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=warn ts=2021-03-15T22:19:06.393Z caller=main.go:524 msg="Received SIGTERM, exiting gracefully..." 2021-03-15T22:19:06.537 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=info ts=2021-03-15T22:19:06.393Z caller=main.go:547 msg="Stopping scrape discovery manager..." 2021-03-15T22:19:06.538 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=info ts=2021-03-15T22:19:06.393Z caller=main.go:561 msg="Stopping notify discovery manager..." 2021-03-15T22:19:06.538 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=info ts=2021-03-15T22:19:06.393Z caller=main.go:583 msg="Stopping scrape manager..." 2021-03-15T22:19:06.538 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=info ts=2021-03-15T22:19:06.393Z caller=manager.go:882 component="rule manager" msg="Stopping rule manager..." 2021-03-15T22:19:06.538 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=info ts=2021-03-15T22:19:06.393Z caller=manager.go:892 component="rule manager" msg="Rule manager stopped" 2021-03-15T22:19:06.538 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=info ts=2021-03-15T22:19:06.393Z caller=main.go:557 msg="Notify discovery manager stopped" 2021-03-15T22:19:06.539 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=info ts=2021-03-15T22:19:06.393Z caller=main.go:543 msg="Scrape discovery manager stopped" 2021-03-15T22:19:06.539 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=info ts=2021-03-15T22:19:06.393Z caller=main.go:577 msg="Scrape manager stopped" 2021-03-15T22:19:06.539 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=info ts=2021-03-15T22:19:06.393Z caller=notifier.go:601 component=notifier msg="Stopping notification manager..." 2021-03-15T22:19:06.540 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=info ts=2021-03-15T22:19:06.393Z caller=main.go:749 msg="Notifier manager stopped" 2021-03-15T22:19:06.540 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[75467]: level=info ts=2021-03-15T22:19:06.393Z caller=main.go:761 msg="See you next time!" 2021-03-15T22:19:06.913 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[77264]: t=2021-03-15T22:19:06+0000 lvl=info msg="Executing migration" logger=migrator id="add index annotation 4 v3" 2021-03-15T22:19:07.164 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:06 smithi125 conmon[77264]: t=2021-03-15T22:19:06+0000 lvl=info msg="Executing migration" logger=migrator id="Update annotation table charset" 2021-03-15T22:19:07.164 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[77264]: t=2021-03-15T22:19:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column region_id to annotation table" 2021-03-15T22:19:07.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[56514]: cluster 2021-03-15T22:19:05.537491+0000 osd.1 (osd 2021-03-15T22:19:07.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[56514]: .1) 7 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:19:07.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[56514]: cluster 2021-03-15T22:19:06.274411+0000 osd.5 (osd.5) 8 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:19:07.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[56514]: cluster 2021-03-15T22:19:06.778618+0000 osd.0 (osd.0) 11 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:19:07.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[56514]: cluster 2021-03-15T22:19:06.961315+0000 mgr.y (mgr.14144) 290 : cluster [DBG] pgmap v225: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s; 225 B/s, 3 objects/s recovering 2021-03-15T22:19:07.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: cluster 2021-03-15T22:19:05.537491 2021-03-15T22:19:07.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 7 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:19:07.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:19:07.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: T22:19:06.274411+0000 osd.5 (osd.5) 8 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:19:07.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: 2021-03-15T22:19:07.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:19:07.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: 15T22:19:06.778618+0000 2021-03-15T22:19:07.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: osd.0 (osd.0) 11 : 2021-03-15T22:19:07.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: cluster [DBG] 4.16 scrub ok 2021-03-15T22:19:07.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:19:07.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: -03-15T22:19:06. 2021-03-15T22:19:07.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: 961315+0000 mgr.y (mgr.14144) 2021-03-15T22:19:07.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:07 smithi076 conmon[62156]: 290 : cluster [DBG] pgmap v225: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s; 225 B/s, 3 objects/s recovering 2021-03-15T22:19:07.485 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 podman[78459]: 2021-03-15 22:19:07.184915258 +0000 UTC m=+0.773227229 container remove be16bcbf856b4e82c3c3dafcd87297349b3272d00044064e17eb63b25e1b4a1a (image=docker.io/prom/prometheus:v2.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-15T22:19:07.485 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 podman[78419]: be16bcbf856b4e82c3c3dafcd87297349b3272d00044064e17eb63b25e1b4a1a 2021-03-15T22:19:07.486 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 systemd[1]: Stopped Ceph prometheus.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:19:07.486 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 systemd[1]: Starting Ceph prometheus.a for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:19:07.486 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[56739]: cluster 2021-03-15T22:19:05.537491+0000 osd.1 (osd.1 2021-03-15T22:19:07.487 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[56739]: ) 7 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:19:07.487 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[56739]: cluster 2021-03-15T22:19:06.274411+0000 osd.5 (osd.5) 8 : cluster 2021-03-15T22:19:07.487 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[56739]: [DBG] 3.1c scrub ok 2021-03-15T22:19:07.487 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:19:07.488 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[56739]: 03-15T22:19:06.778618+0000 osd.0 (osd.0) 11 2021-03-15T22:19:07.488 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[56739]: : cluster [DBG] 4.16 scrub ok 2021-03-15T22:19:07.488 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:19:07.488 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[56739]: 03-15T22:19:06.961315+0000 mgr.y ( 2021-03-15T22:19:07.488 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[56739]: mgr.14144) 290 : cluster [DBG] pgmap v225: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 731 B/s rd, 0 op/s; 225 B/s, 3 objects/s recovering 2021-03-15T22:19:07.490 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[77264]: t=2021-03-15T22:19:07+0000 lvl=info msg="Executing migration" logger=migrator id="Drop category_id index" 2021-03-15T22:19:07.490 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[77264]: t=2021-03-15T22:19:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add column tags to annotation table" 2021-03-15T22:19:07.490 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[77264]: t=2021-03-15T22:19:07+0000 lvl=info msg="Executing migration" logger=migrator id="Create annotation_tag table v2" 2021-03-15T22:19:07.491 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[77264]: t=2021-03-15T22:19:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add unique index annotation_tag.annotation_id_tag_id" 2021-03-15T22:19:07.776 INFO:teuthology.orchestra.run.smithi076.stdout:163208757273 2021-03-15T22:19:07.776 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph tell osd.6 flush_pg_stats 2021-03-15T22:19:07.786 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 podman[78582]: 2021-03-15 22:19:07.568229781 +0000 UTC m=+0.248520901 container create 4f0fae298426fcb15e361588c02b660fa9eef3b97325654b4a532826434246e3 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-15T22:19:07.786 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[77264]: t=2021-03-15T22:19:07+0000 lvl=info msg="Executing migration" logger=migrator id="Update alert annotations and set TEXT to empty" 2021-03-15T22:19:07.787 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[77264]: t=2021-03-15T22:19:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add created time to annotation table" 2021-03-15T22:19:07.787 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[77264]: t=2021-03-15T22:19:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add updated time to annotation table" 2021-03-15T22:19:08.055 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[77264]: t=2021-03-15T22:19:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for created in annotation table" 2021-03-15T22:19:08.055 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[77264]: t=2021-03-15T22:19:07+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for updated in annotation table" 2021-03-15T22:19:08.056 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[77264]: t=2021-03-15T22:19:08+0000 lvl=info msg="Executing migration" logger=migrator id="Convert existing annotations from seconds to milliseconds" 2021-03-15T22:19:08.056 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 podman[78582]: 2021-03-15 22:19:07.895063098 +0000 UTC m=+0.575354229 container init 4f0fae298426fcb15e361588c02b660fa9eef3b97325654b4a532826434246e3 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-15T22:19:08.056 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.956Z caller=main.go:302 msg="No time or size retention was set so using the default time retention" duration=15d 2021-03-15T22:19:08.056 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.956Z caller=main.go:337 msg="Starting Prometheus" version="(version=2.18.1, branch=HEAD, revision=ecee9c8abfd118f139014cb1b174b08db3f342cf)" 2021-03-15T22:19:08.057 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.956Z caller=main.go:338 build_context="(go=go1.14.2, user=root@2117a9e64a7e, date=20200507-16:51:47)" 2021-03-15T22:19:08.057 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.956Z caller=main.go:339 host_details="(Linux 4.18.0-240.15.1.el8_3.x86_64 #1 SMP Mon Mar 1 17:16:16 UTC 2021 x86_64 smithi125 (none))" 2021-03-15T22:19:08.057 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.956Z caller=main.go:340 fd_limits="(soft=1048576, hard=1048576)" 2021-03-15T22:19:08.057 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.956Z caller=main.go:341 vm_limits="(soft=unlimited, hard=unlimited)" 2021-03-15T22:19:08.058 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.957Z caller=main.go:678 msg="Starting TSDB ..." 2021-03-15T22:19:08.058 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.957Z caller=web.go:523 component=web msg="Start listening for connections" address=:9095 2021-03-15T22:19:08.058 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.959Z caller=head.go:575 component=tsdb msg="Replaying WAL, this may take awhile" 2021-03-15T22:19:08.058 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.959Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2021-03-15T22:19:08.059 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.959Z caller=head.go:624 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2021-03-15T22:19:08.059 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.960Z caller=head.go:627 component=tsdb msg="WAL replay completed" duration=719.326µs 2021-03-15T22:19:08.059 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.961Z caller=main.go:694 fs_type=XFS_SUPER_MAGIC 2021-03-15T22:19:08.059 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.961Z caller=main.go:695 msg="TSDB started" 2021-03-15T22:19:08.059 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.961Z caller=main.go:799 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2021-03-15T22:19:08.060 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.961Z caller=main.go:827 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml 2021-03-15T22:19:08.060 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:07 smithi125 conmon[78622]: level=info ts=2021-03-15T22:19:07.961Z caller=main.go:646 msg="Server is ready to receive web requests." 2021-03-15T22:19:08.060 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:08 smithi125 podman[78582]: 2021-03-15 22:19:08.003877477 +0000 UTC m=+0.684168623 container start 4f0fae298426fcb15e361588c02b660fa9eef3b97325654b4a532826434246e3 (image=docker.io/prom/prometheus:v2.18.1, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-prometheus.a, maintainer=The Prometheus Authors ) 2021-03-15T22:19:08.060 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:08 smithi125 bash[78507]: 4f0fae298426fcb15e361588c02b660fa9eef3b97325654b4a532826434246e3 2021-03-15T22:19:08.061 INFO:journalctl@ceph.prometheus.a.smithi125.stdout:Mar 15 22:19:08 smithi125 systemd[1]: Started Ceph prometheus.a for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:19:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:08.373 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[77264]: t=2021-03-15T22:19:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add epoch_end column" 2021-03-15T22:19:08.374 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[77264]: t=2021-03-15T22:19:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for epoch_end" 2021-03-15T22:19:08.374 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[77264]: t=2021-03-15T22:19:08+0000 lvl=info msg="Executing migration" logger=migrator id="Make epoch_end the same as epoch" 2021-03-15T22:19:08.375 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[77264]: t=2021-03-15T22:19:08+0000 lvl=info msg="Executing migration" logger=migrator id="Move region to single row" 2021-03-15T22:19:08.376 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: cluster 2021-03-15T22:19:06.942864+0000 osd.7 (osd.7) 2021-03-15T22:19:08.376 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 7 : cluster [DBG] 3.f scrub ok 2021-03-15T22:19:08.376 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: cluster 2021-03-15T22:19:07.795069+0000 osd.0 (osd.0) 12 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:19:08.376 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15T22:19:08.377 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 2021-03-15T22:19:08.030295+0000 mon.a (mon.0) 551 : audit [INF] 2021-03-15T22:19:08.377 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.377 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15T22:19:08.377 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 2021-03-15T22:19:08.032720+0000 mon.a ( 2021-03-15T22:19:08.378 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: mon.0) 552 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-15T22:19:08.378 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:19:08.378 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: T22:19:08.033837+0000 mon.a ( 2021-03-15T22:19:08.378 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: mon.0) 553 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076:9093"}]: dispatch 2021-03-15T22:19:08.379 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:19:08.379 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 15T22:19:08.039387 2021-03-15T22:19:08.379 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: +0000 mon.a (mon.0) 554 : audit 2021-03-15T22:19:08.379 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.380 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15T22:19 2021-03-15T22:19:08.380 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: :08.043754+0000 mon.a (mon.0) 2021-03-15T22:19:08.380 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 555 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-15T22:19:08.380 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15T22:19 2021-03-15T22:19:08.381 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: :08.051011+0000 mon.a (mon.0) 2021-03-15T22:19:08.381 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 556 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-15T22:19:08.381 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:19:08.381 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 22:19:08.056599+0000 mon.a 2021-03-15T22:19:08.381 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: (mon.0) 557 : audit 2021-03-15T22:19:08.382 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.382 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15T22:19:08.382 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:19:08.382 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 19:08.059118+0000 mon.a (mon. 2021-03-15T22:19:08.383 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 0) 558 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi125"}]: dispatch 2021-03-15T22:19:08.383 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021 2021-03-15T22:19:08.383 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: -03-15T22:19 2021-03-15T22:19:08.383 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: :08.066281+0000 mon.a (mon.0) 2021-03-15T22:19:08.384 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 559 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.384 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:19:08.384 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 03-15T22:19:08. 2021-03-15T22:19:08.384 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 069209+0000 mon.a (mon.0) 560 : audit 2021-03-15T22:19:08.384 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-15T22:19:08.385 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:19:08.385 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 15T22:19:08.070397+0000 2021-03-15T22:19:08.385 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: mon.a (mon.0) 561 : audit [INF] 2021-03-15T22:19:08.385 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi125:9095"}]: dispatch 2021-03-15T22:19:08.386 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:19:08.386 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: T22:19:08.075478+0000 2021-03-15T22:19:08.386 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: mon.a (mon.0) 562 : audit [INF] 2021-03-15T22:19:08.386 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.387 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:19:08.387 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: T22:19:08.078419+0000 mon.a 2021-03-15T22:19:08.387 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: (mon.0) 563 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-15T22:19:08.387 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 2021-03-15T22:19:08.387 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:19:08.388 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 22:19:08.082881+0000 mon.a 2021-03-15T22:19:08.388 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi125:3000"}]: dispatch 2021-03-15T22:19:08.388 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 2021-03-15T22:19:08.388 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:19:08.388 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 22:19:08.087588+0000 mon.a ( 2021-03-15T22:19:08.389 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: mon.0) 565 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.389 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: audit 2021-03-15T22:19:08.389 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:19:08.389 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: 19:08.090348+0000 mon.a (mon.0 2021-03-15T22:19:08.390 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[56739]: ) 566 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:19:08.446 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: cluster 2021-03-15T22:19:06.942864 2021-03-15T22:19:08.446 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: +0000 osd.7 (osd.7) 7 : cluster [DBG] 3.f scrub ok 2021-03-15T22:19:08.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: cluster 2021-03-15T22:19:07.795069+0000 osd.0 (osd.0) 12 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:19:08.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.030295+0000 mon.a (mon.0) 551 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.032720+0000 mon.a (mon.0) 552 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-15T22:19:08.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.033837+0000 mon.a (mon.0) 553 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076:9093"}]: dispatch 2021-03-15T22:19:08.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.039387+0000 mon.a (mon.0) 554 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.043754+0000 mon.a (mon.0) 555 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-15T22:19:08.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.051011+0000 mon.a (mon.0) 556 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-15T22:19:08.449 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.056599+0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.449 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.059118+0000 mon.a (mon.0) 558 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi125"}]: dispatch 2021-03-15T22:19:08.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.066281+0000 mon.a (mon.0) 559 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.451 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.069209+0000 mon.a (mon.0) 560 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-15T22:19:08.451 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.070397+0000 mon.a (mon.0) 561 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi125:9095"}]: dispatch 2021-03-15T22:19:08.451 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.075478+0000 mon.a (mon.0) 562 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.451 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.078419+0000 mon.a (mon.0) 563 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-15T22:19:08.452 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.082881+0000 mon.a (mon.0) 564 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi125:3000"}]: dispatch 2021-03-15T22:19:08.452 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.087588+0000 mon.a (mon.0) 565 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.452 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.090348+0000 mon.a (mon.0) 566 : 2021-03-15T22:19:08.452 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[56514]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:19:08.453 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: cluster 2021-03-15T22:19:06.942864 2021-03-15T22:19:08.453 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: +0000 osd.7 (osd.7) 7 : cluster [DBG] 3.f scrub ok 2021-03-15T22:19:08.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: cluster 2021-03-15T22:19:08.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 2021-03-15T22:19:07.795069+0000 osd.0 (osd.0) 12 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:19:08.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:19:08.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 19:08.030295+0000 mon.a (mon.0) 551 : 2021-03-15T22:19:08.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 2021-03-15T22:19:08.032720+0000 mon.a (mon.0) 552 2021-03-15T22:19:08.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-15T22:19:08.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:19:08.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 03-15T22:19:08.033837+0000 mon.a (mon.0) 553 : 2021-03-15T22:19:08.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076:9093"}]: dispatch 2021-03-15T22:19:08.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.039387+0000 2021-03-15T22:19:08.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: mon.a (mon.0) 554 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.043754+0000 2021-03-15T22:19:08.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: mon.a (mon.0) 555 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-15T22:19:08.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.051011+0000 2021-03-15T22:19:08.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: mon.a (mon.0) 556 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-15T22:19:08.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.056599 2021-03-15T22:19:08.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: +0000 mon.a (mon.0) 557 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.059118+0000 2021-03-15T22:19:08.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: mon.a (mon.0) 558 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi125"}]: dispatch 2021-03-15T22:19:08.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.066281+0000 2021-03-15T22:19:08.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: mon.a (mon.0) 559 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.069209+0000 mon.a ( 2021-03-15T22:19:08.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: mon.0) 560 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-15T22:19:08.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 2021-03-15T22:19:08.070397+0000 mon.a (mon. 2021-03-15T22:19:08.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 0) 561 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi125:9095"}]: dispatch 2021-03-15T22:19:08.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:19:08.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 22:19:08.075478+0000 mon.a (mon 2021-03-15T22:19:08.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: .0) 562 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 2021-03-15T22:19:08.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:19:08.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 15T22:19:08.078419+0000 mon.a ( 2021-03-15T22:19:08.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: mon.0) 563 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-15T22:19:08.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021 2021-03-15T22:19:08.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: -03-15T22:19: 2021-03-15T22:19:08.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 08.082881+0000 mon.a (mon.0) 564 : audit [INF] 2021-03-15T22:19:08.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi125:3000"}]: dispatch 2021-03-15T22:19:08.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:19:08.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 15T22:19:08.087588+0000 mon.a (mon. 2021-03-15T22:19:08.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 0) 565 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:08.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:19:08.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: 15T22:19:08.090348+0000 mon.a (mon.0 2021-03-15T22:19:08.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:08 smithi076 conmon[62156]: ) 566 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:19:08.670 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[77264]: t=2021-03-15T22:19:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch from annotation table" 2021-03-15T22:19:08.671 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[77264]: t=2021-03-15T22:19:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2021-03-15T22:19:08.984 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[77264]: t=2021-03-15T22:19:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2021-03-15T22:19:08.985 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[77264]: t=2021-03-15T22:19:08+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for org_id_epoch_end_epoch on annotation table" 2021-03-15T22:19:08.985 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:08 smithi125 conmon[77264]: t=2021-03-15T22:19:08+0000 lvl=info msg="Executing migration" logger=migrator id="Remove index org_id_epoch_epoch_end from annotation table" 2021-03-15T22:19:09.269 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[77264]: t=2021-03-15T22:19:09+0000 lvl=info msg="Executing migration" logger=migrator id="Add index for alert_id on annotation table" 2021-03-15T22:19:09.269 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[77264]: t=2021-03-15T22:19:09+0000 lvl=info msg="Executing migration" logger=migrator id="create test_data table" 2021-03-15T22:19:09.270 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[77264]: t=2021-03-15T22:19:09+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard_version table v1" 2021-03-15T22:19:09.270 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: cluster 2021-03-15T22:19:07.544850+0000 2021-03-15T22:19:09.271 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: osd.1 (osd.1) 8 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:19:09.271 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: cluster 2021-03-15T22:19:07.874456 2021-03-15T22:19:09.271 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: +0000 osd.3 (osd.3) 11 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:19:09.271 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: cluster 2021-03-15T22:19 2021-03-15T22:19:09.271 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: :07.981216+0000 osd.7 (osd.7) 8 : cluster 2021-03-15T22:19:09.272 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: [DBG] 5.4 deep-scrub ok 2021-03-15T22:19:09.272 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:19:09.272 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 19:08.033024+0000 mgr.y (mgr.14144 2021-03-15T22:19:09.272 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: ) 291 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-15T22:19:09.273 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: audit 2021 2021-03-15T22:19:09.273 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: -03-15T22:19:08.034093 2021-03-15T22:19:09.273 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 292 : audit 2021-03-15T22:19:09.273 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076:9093"}]: dispatch 2021-03-15T22:19:09.273 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:19:09.274 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 19:08.044216+0000 mgr.y (mgr.14144) 293 : 2021-03-15T22:19:09.274 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-15T22:19:09.274 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: cephadm 2021-03-15T22: 2021-03-15T22:19:09.274 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 19:08.050524+0000 mgr.y (mgr.14144) 2021-03-15T22:19:09.275 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 294 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.125:5000 to Dashboard 2021-03-15T22:19:09.275 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: audit 2021-03-15T22:19:09.275 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 2021- 2021-03-15T22:19:09.275 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 03-15T22:19:08.051355 2021-03-15T22:19:09.275 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 295 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-15T22:19:09.276 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: audit 2021-03-15T22:19:08.059563+0000 mgr.y (mgr. 2021-03-15T22:19:09.276 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 14144) 296 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi125"}]: dispatch 2021-03-15T22:19:09.276 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: audit 2021 2021-03-15T22:19:09.276 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: -03-15T22:19:08.069560+0000 mgr.y (mgr.14144) 297 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-15T22:19:09.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 2021-03-15T22:19:09.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:19:09.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: T22:19:08.070712+0000 mgr.y (mgr.14144) 298 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi125:9095"}]: dispatch 2021-03-15T22:19:09.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 2021-03-15T22:19:09.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:19:09.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: T22:19:08.078810+0000 mgr.y (mgr.14144) 299 : audit [DBG] 2021-03-15T22:19:09.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-15T22:19:09.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:19:09.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 15T22:19:08.083256+0000 mgr.y (mgr.14144) 300 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi125:3000"}]: dispatch 2021-03-15T22:19:09.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 2021-03-15T22:19:09.279 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:19:09.279 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: :19:08.176462+0000 osd.1 (osd.1) 9 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:09.279 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: cluster 2021-03-15T22:19 2021-03-15T22:19:09.279 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: :08.176553+0000 osd.1 (osd.1) 10 : cluster [DBG] 2021-03-15T22:19:09.280 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: purged_snaps scrub ok 2021-03-15T22:19:09.280 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:19:09.280 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: -03-15T22:19:08.268699+0000 osd.5 ( 2021-03-15T22:19:09.280 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: osd.5) 9 : cluster [DBG] 2.2 deep-scrub ok 2021-03-15T22:19:09.280 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:19:09.281 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 15T22:19:08.788961+0000 osd.0 (osd.0) 13 : 2021-03-15T22:19:09.281 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: cluster [DBG] 3.1f scrub ok 2021-03-15T22:19:09.281 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: cluster 2021-03-15T22:19:09.281 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 2021-03-15T22:19:08.961791+0000 mgr.y (mgr. 2021-03-15T22:19:09.281 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[56739]: 14144) 301 : cluster [DBG] pgmap v226: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 197 B/s, 2 objects/s recovering 2021-03-15T22:19:09.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: cluster 2021-03-15T22:19:07.544850+0000 osd.1 (osd.1) 8 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:19:09.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: cluster 2021-03-15T22:19:07.874456+0000 osd.3 (osd.3) 11 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:19:09.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: cluster 2021-03-15T22:19:07.981216+0000 osd.7 (osd.7) 8 : cluster [DBG] 5.4 deep-scrub ok 2021-03-15T22:19:09.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.033024+0000 mgr.y (mgr.14144) 291 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-15T22:19:09.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.034093+0000 mgr.y (mgr.14144) 292 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076:9093"}]: dispatch 2021-03-15T22:19:09.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.044216+0000 mgr.y (mgr.14144) 293 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-15T22:19:09.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:08.050524+0000 mgr.y (mgr.14144) 294 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.125:5000 to Dashboard 2021-03-15T22:19:09.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.051355+0000 mgr.y (mgr.14144) 295 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-15T22:19:09.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.059563+0000 mgr.y (mgr.14144) 296 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi125"}]: dispatch 2021-03-15T22:19:09.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.069560+0000 mgr.y (mgr.14144) 297 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-15T22:19:09.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.070712+0000 mgr.y (mgr.14144) 298 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi125:9095"}]: dispatch 2021-03-15T22:19:09.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.078810+0000 mgr.y (mgr.14144) 299 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-15T22:19:09.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: audit 2021-03-15T22:19:08.083256+0000 mgr.y (mgr.14144) 300 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi125:3000"}]: dispatch 2021-03-15T22:19:09.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: cluster 2021-03-15T22:19:08.176462+0000 osd.1 (osd.1) 9 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:09.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: cluster 2021-03-15T22:19:08.176553+0000 osd.1 (osd.1) 10 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:09.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: cluster 2021-03-15T22:19:08.268699+0000 osd.5 (osd.5) 9 : cluster [DBG] 2.2 deep-scrub ok 2021-03-15T22:19:09.389 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: cluster 2021-03-15T22:19:08.788961+0000 osd.0 (osd.0) 13 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:19:09.389 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[56514]: cluster 2021-03-15T22:19:08.961791+0000 mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v226: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 197 B/s, 2 objects/s recovering 2021-03-15T22:19:09.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: cluster 2021-03-15T22:19:07.544850+0000 osd.1 (osd.1) 8 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:19:09.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: cluster 2021-03-15T22:19:07.874456+0000 osd.3 (osd.3) 11 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:19:09.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: cluster 2021-03-15T22:19:07.981216+0000 osd.7 (osd.7) 8 : cluster [DBG] 5.4 deep-scrub ok 2021-03-15T22:19:09.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.033024+0000 mgr.y (mgr.14144) 291 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2021-03-15T22:19:09.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.034093+0000 2021-03-15T22:19:09.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: mgr.y (mgr.14144) 292 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi076:9093"}]: dispatch 2021-03-15T22:19:09.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.044216+0000 mgr.y (mgr.14144) 293 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-list"}]: dispatch 2021-03-15T22:19:09.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:08.050524+0000 mgr.y (mgr.14144) 294 : cephadm [INF] Adding iSCSI gateway http://:@172.21.15.125:5000 to Dashboard 2021-03-15T22:19:09.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.051355+0000 mgr.y (mgr.14144) 295 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-iscsi-api-ssl-verification", "value": "true"}]: dispatch 2021-03-15T22:19:09.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.059563+0000 mgr.y (mgr.14144) 296 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard iscsi-gateway-add", "name": "smithi125"}]: dispatch 2021-03-15T22:19:09.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.069560+0000 mgr.y (mgr.14144) 297 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2021-03-15T22:19:09.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.070712+0000 mgr.y (mgr.14144) 298 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi125:9095"}]: dispatch 2021-03-15T22:19:09.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.078810+0000 mgr.y (mgr.14144) 299 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2021-03-15T22:19:09.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: audit 2021-03-15T22:19:08.083256+0000 mgr.y (mgr.14144) 300 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi125:3000"}]: dispatch 2021-03-15T22:19:09.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: cluster 2021-03-15T22:19:08.176462+0000 osd.1 (osd.1) 9 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:09.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: cluster 2021-03-15T22:19:08.176553+0000 osd.1 (osd. 2021-03-15T22:19:09.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: 1) 10 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:09.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: cluster 2021-03-15T22:19:08.268699+0000 osd.5 (osd.5) 9 : cluster [DBG] 2.2 deep-scrub ok 2021-03-15T22:19:09.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: cluster 2021-03-15T22:19:08.788961+0000 2021-03-15T22:19:09.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: osd.0 (osd.0) 13 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:19:09.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: cluster 2021-03-15T22:19:08.961791+0000 2021-03-15T22:19:09.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:09 smithi076 conmon[62156]: mgr.y (mgr.14144) 301 : cluster [DBG] pgmap v226: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s; 197 B/s, 2 objects/s recovering 2021-03-15T22:19:09.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:19:09.537 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[77264]: t=2021-03-15T22:19:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_version.dashboard_id" 2021-03-15T22:19:09.537 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[77264]: t=2021-03-15T22:19:09+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2021-03-15T22:19:09.537 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[77264]: t=2021-03-15T22:19:09+0000 lvl=info msg="Executing migration" logger=migrator id="Set dashboard version to 1 where 0" 2021-03-15T22:19:09.956 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[77264]: t=2021-03-15T22:19:09+0000 lvl=info msg="Executing migration" logger=migrator id="save existing dashboard data in dashboard_version table v1" 2021-03-15T22:19:09.957 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[77264]: t=2021-03-15T22:19:09+0000 lvl=info msg="Executing migration" logger=migrator id="alter dashboard_version.data to mediumtext v1" 2021-03-15T22:19:09.957 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[77264]: t=2021-03-15T22:19:09+0000 lvl=info msg="Executing migration" logger=migrator id="create team table" 2021-03-15T22:19:09.958 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:09 smithi125 conmon[77264]: t=2021-03-15T22:19:09+0000 lvl=info msg="Executing migration" logger=migrator id="add index team.org_id" 2021-03-15T22:19:10.286 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[77264]: t=2021-03-15T22:19:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_org_id_name" 2021-03-15T22:19:10.287 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[77264]: t=2021-03-15T22:19:10+0000 lvl=info msg="Executing migration" logger=migrator id="create team member table" 2021-03-15T22:19:10.287 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[77264]: t=2021-03-15T22:19:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index team_member.org_id" 2021-03-15T22:19:10.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[56739]: cluster 2021-03-15T22:19:09.175236+0000 mon.a ( 2021-03-15T22:19:10.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[56739]: mon.0) 567 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2021-03-15T22:19:10.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[56739]: audit 2021-03-15T22:19:09.353317+0000 2021-03-15T22:19:10.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[56739]: mgr.y (mgr.14144) 302 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:10.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[56739]: cluster 2021-03-15T22:19:09 2021-03-15T22:19:10.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[56739]: .820943+0000 osd.0 (osd.0) 14 : 2021-03-15T22:19:10.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[56739]: cluster [DBG] 5.5 scrub ok 2021-03-15T22:19:10.304 INFO:teuthology.orchestra.run.smithi076.stdout:193273528342 2021-03-15T22:19:10.305 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph tell osd.7 flush_pg_stats 2021-03-15T22:19:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:10 smithi076 conmon[56514]: cluster 2021-03-15T22:19:09.175236+0000 mon.a (mon.0) 567 : cluster [DBG] 2021-03-15T22:19:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:10 smithi076 conmon[56514]: osdmap e83: 8 total, 8 up, 8 in 2021-03-15T22:19:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:10 smithi076 conmon[56514]: audit 2021-03-15T22:19:09.353317+0000 mgr.y (mgr.14144) 302 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:10 smithi076 conmon[56514]: cluster 2021-03-15T22:19:09.820943+0000 osd.0 (osd.0) 14 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:19:10.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:10 smithi076 conmon[62156]: cluster 2021-03-15T22:19:09.175236+0000 mon.a 2021-03-15T22:19:10.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:10 smithi076 conmon[62156]: (mon.0) 567 : cluster [DBG] osdmap e83: 8 total, 8 up, 8 in 2021-03-15T22:19:10.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:10 smithi076 conmon[62156]: audit 2021-03-15T22:19:09.353317+0000 mgr.y 2021-03-15T22:19:10.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:10 smithi076 conmon[62156]: (mgr.14144) 302 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:10.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:10 smithi076 conmon[62156]: cluster 2021-03-15T22:19:09.820943+0000 osd.0 (osd.0) 14 : 2021-03-15T22:19:10.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:10 smithi076 conmon[62156]: cluster [DBG] 5.5 scrub ok 2021-03-15T22:19:10.585 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[77264]: t=2021-03-15T22:19:10+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index team_member_org_id_team_id_user_id" 2021-03-15T22:19:10.585 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[77264]: t=2021-03-15T22:19:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column email to team table" 2021-03-15T22:19:10.586 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[77264]: t=2021-03-15T22:19:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column external to team_member table" 2021-03-15T22:19:10.620 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:10.887 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[77264]: t=2021-03-15T22:19:10+0000 lvl=info msg="Executing migration" logger=migrator id="Add column permission to team_member table" 2021-03-15T22:19:10.888 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[77264]: t=2021-03-15T22:19:10+0000 lvl=info msg="Executing migration" logger=migrator id="create dashboard acl table" 2021-03-15T22:19:11.178 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:10 smithi125 conmon[77264]: t=2021-03-15T22:19:10+0000 lvl=info msg="Executing migration" logger=migrator id="add index dashboard_acl_dashboard_id" 2021-03-15T22:19:11.178 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[77264]: t=2021-03-15T22:19:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_user_id" 2021-03-15T22:19:11.456 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[77264]: t=2021-03-15T22:19:11+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index dashboard_acl_dashboard_id_team_id" 2021-03-15T22:19:11.456 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[77264]: t=2021-03-15T22:19:11+0000 lvl=info msg="Executing migration" logger=migrator id="save default acl rules in dashboard_acl table" 2021-03-15T22:19:11.457 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[77264]: t=2021-03-15T22:19:11+0000 lvl=info msg="Executing migration" logger=migrator id="create tag table" 2021-03-15T22:19:11.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[56739]: cluster 2021-03-15T22:19:09.989224+0000 osd.7 2021-03-15T22:19:11.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[56739]: (osd.7) 9 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:19:11.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[56739]: cluster 2021-03-15T22:19:10.132510+0000 2021-03-15T22:19:11.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[56739]: osd.2 (osd.2) 8 : cluster [DBG] 4.f scrub ok 2021-03-15T22:19:11.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:19:11.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[56739]: 22:19:10.962224+0000 mgr.y (mgr.14144 2021-03-15T22:19:11.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[56739]: ) 303 : cluster [DBG] pgmap v228: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 177 B/s, 2 objects/s recovering 2021-03-15T22:19:11.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:11 smithi076 conmon[56514]: cluster 2021-03-15T22:19:09.989224+0000 osd.7 (osd.7) 9 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:19:11.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:11 smithi076 conmon[56514]: cluster 2021-03-15T22:19:10.132510+0000 osd.2 (osd.2) 8 : cluster [DBG] 4.f scrub ok 2021-03-15T22:19:11.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:11 smithi076 conmon[56514]: cluster 2021-03-15T22:19:10.962224+0000 mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v228: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 177 B/s, 2 objects/s recovering 2021-03-15T22:19:11.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:11 smithi076 conmon[62156]: cluster 2021-03-15T22:19:09.989224+0000 osd.7 2021-03-15T22:19:11.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:11 smithi076 conmon[62156]: (osd.7) 9 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:19:11.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:11 smithi076 conmon[62156]: cluster 2021-03-15T22:19:10.132510+0000 osd.2 (osd.2) 8 : cluster [DBG] 4.f scrub ok 2021-03-15T22:19:11.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:11 smithi076 conmon[62156]: cluster 2021-03-15T22:19:10.962224+0000 mgr.y (mgr.14144) 303 : cluster [DBG] pgmap v228: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s; 177 B/s, 2 objects/s recovering 2021-03-15T22:19:11.786 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[77264]: t=2021-03-15T22:19:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index tag.key_value" 2021-03-15T22:19:11.787 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[77264]: t=2021-03-15T22:19:11+0000 lvl=info msg="Executing migration" logger=migrator id="create login attempt table" 2021-03-15T22:19:11.787 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[77264]: t=2021-03-15T22:19:11+0000 lvl=info msg="Executing migration" logger=migrator id="add index login_attempt.username" 2021-03-15T22:19:12.084 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[77264]: t=2021-03-15T22:19:11+0000 lvl=info msg="Executing migration" logger=migrator id="drop index IDX_login_attempt_username - v1" 2021-03-15T22:19:12.085 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:11 smithi125 conmon[77264]: t=2021-03-15T22:19:11+0000 lvl=info msg="Executing migration" logger=migrator id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2021-03-15T22:19:12.085 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="create login_attempt v2" 2021-03-15T22:19:12.085 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_login_attempt_username - v2" 2021-03-15T22:19:12.178 INFO:journalctl@ceph.alertmanager.a.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[85285]: level=info ts=2021-03-15T22:19:12.131Z caller=cluster.go:640 component=cluster msg="gossip settled; proceeding" elapsed=10.000784149s 2021-03-15T22:19:12.352 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="copy login_attempt v1 to v2" 2021-03-15T22:19:12.353 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="drop login_attempt_tmp_qwerty" 2021-03-15T22:19:12.353 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth table" 2021-03-15T22:19:12.353 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="create index IDX_user_auth_auth_module_auth_id - v1" 2021-03-15T22:19:12.354 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[56739]: cluster 2021-03-15T22:19:10.851252+0000 osd.3 (osd. 2021-03-15T22:19:12.354 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[56739]: 3) 12 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:19:12.354 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[56739]: cluster 2021-03-15T22:19:11.318124+0000 osd.5 (osd 2021-03-15T22:19:12.354 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[56739]: .5) 10 : cluster [DBG] 6.a scrub ok 2021-03-15T22:19:12.355 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[56739]: cluster 2021-03-15T22:19:12.355 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[56739]: 2021-03-15T22:19:11.831960+0000 osd.0 2021-03-15T22:19:12.355 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[56739]: (osd.0) 15 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:19:12.355 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[56739]: audit 2021 2021-03-15T22:19:12.356 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[56739]: -03-15T22:19:11.966349+0000 2021-03-15T22:19:12.356 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[56739]: mon.a (mon.0) 568 : audit [INF] 2021-03-15T22:19:12.356 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:12.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56514]: cluster 2021-03-15T22:19:10 2021-03-15T22:19:12.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56514]: .851252+0000 osd.3 (osd.3) 12 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:19:12.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56514]: cluster 2021-03-15T22:19:11.318124+0000 osd.5 (osd.5) 10 : cluster [DBG] 6.a scrub ok 2021-03-15T22:19:12.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:19:12.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56514]: -03-15T22:19:11.831960+0000 osd.0 (osd.0) 15 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:19:12.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56514]: audit 2021-03-15T22:19:11.966349+0000 mon.a ( 2021-03-15T22:19:12.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56514]: mon.0) 568 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:12.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[62156]: cluster 2021-03-15T22:19:10.851252+0000 osd.3 (osd 2021-03-15T22:19:12.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[62156]: .3) 12 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:19:12.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:19:12.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[62156]: :19:11.318124+0000 osd.5 (osd.5) 10 : cluster [DBG] 6.a scrub ok 2021-03-15T22:19:12.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:19:12.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[62156]: -15T22:19:11.831960+0000 osd.0 2021-03-15T22:19:12.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[62156]: (osd.0) 15 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:19:12.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[62156]: 2021-03-15T22:19:12.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:19:12.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[62156]: :19:11.966349+0000 mon.a (mon.0) 568 : audit 2021-03-15T22:19:12.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:12.628 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="alter user_auth.auth_id to length 190" 2021-03-15T22:19:12.629 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth access token to user_auth" 2021-03-15T22:19:12.629 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth refresh token to user_auth" 2021-03-15T22:19:12.922 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth token type to user_auth" 2021-03-15T22:19:12.922 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add OAuth expiry to user_auth" 2021-03-15T22:19:12.922 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="Add index to user_id column in user_auth" 2021-03-15T22:19:12.923 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="create server_lock table" 2021-03-15T22:19:13.017 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: debug 2021-03-15T22:19:12.624+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:19:13.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:13.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:13.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:13.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:19:13.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: debug 2021-03-15T22:19:12.624+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:13.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:13.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:13.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:13.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:13.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: debug 2021-03-15T22:19:12.625+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:13.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:13.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:13.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:13.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:13.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: debug 2021-03-15T22:19:12.625+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:13.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:13.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:13.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:13.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:13.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: debug 2021-03-15T22:19:12.625+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:13.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:13.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:13.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:13.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:13.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: debug 2021-03-15T22:19:12.626+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:13.028 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:13.028 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.028 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:13.028 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:13.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:13.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: debug 2021-03-15T22:19:12.626+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:13.030 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:13.030 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.030 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:13.030 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:13.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:12 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:13.181 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:12 smithi125 conmon[77264]: t=2021-03-15T22:19:12+0000 lvl=info msg="Executing migration" logger=migrator id="add index server_lock.operation_uid" 2021-03-15T22:19:13.181 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Executing migration" logger=migrator id="create user auth token table" 2021-03-15T22:19:13.182 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.auth_token" 2021-03-15T22:19:13.239 INFO:teuthology.orchestra.run.smithi076.stdout:223338299411 2021-03-15T22:19:13.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd last-stat-seq osd.0 2021-03-15T22:19:13.449 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index user_auth_token.prev_auth_token" 2021-03-15T22:19:13.450 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Executing migration" logger=migrator id="create cache_data table" 2021-03-15T22:19:13.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cluster 2021-03-15T22:19:11.891459+0000 osd.4 (osd.4 2021-03-15T22:19:13.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: ) 8 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:19:13.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cluster 2021-03-15T22:19:12.158853+0000 osd.2 (osd.2) 9 : 2021-03-15T22:19:13.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cluster [DBG] 5.d scrub ok 2021-03-15T22:19:13.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:19:13.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: 15T22:19:12.621335+0000 mon.a (mon.0) 569 2021-03-15T22:19:13.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:13.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cephadm 2021-03-15T22 2021-03-15T22:19:13.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: :19:12.621941+0000 mgr.y (mgr. 2021-03-15T22:19:13.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: 14144) 304 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:19:13.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:19:13.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: -15T22:19:12.625199+0000 mon.a ( 2021-03-15T22:19:13.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: mon.0) 570 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:13.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cephadm 2021 2021-03-15T22:19:13.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: -03-15T22:19:12.625951+0000 mgr.y (mgr 2021-03-15T22:19:13.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: .14144) 305 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:19:13.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:13.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:13.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:13.456 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.456 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.456 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:19:13.456 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:12.626341+0000 mgr.y (mgr.14144) 306 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:13.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:13.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:13.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:13.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:13.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:12.626700+0000 mgr.y (mgr.14144) 307 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:13.459 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:13.459 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.459 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:13.459 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:13.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:13.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cephadm 2021-03-15T22 2021-03-15T22:19:13.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: :19:12.627040+0000 mgr.y (mgr.14144) 308 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:13.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:13.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:13.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:13.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:13.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:12.627388+0000 mgr.y (mgr.14144) 309 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:13.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:13.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:13.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:13.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:13.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cephadm 2021-03-15T22:19 2021-03-15T22:19:13.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: :12.627737+0000 mgr.y (mgr.14144) 310 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:13.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:13.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.466 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:13.466 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.466 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:13.466 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.466 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:13.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:12.628071+0000 mgr.y (mgr.14144) 311 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:13.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:13.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:13.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:13.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.469 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:13.469 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: audit 2021-03-15T22:19:13.469 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: 2021-03-15T22:19:12.628792+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:19:13.469 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[56739]: cluster 2021-03-15T22:19:12.962878+0000 mgr.y (mgr.14144) 312 : cluster [DBG] pgmap v229: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 157 B/s, 2 objects/s recovering 2021-03-15T22:19:13.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: cluster 2021-03-15T22:19:11.891459 2021-03-15T22:19:13.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: +0000 osd.4 (osd.4) 8 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:19:13.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: cluster 2021-03-15T22:19:13.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: 2021-03-15T22:19:12.158853+0000 osd.2 (osd.2 2021-03-15T22:19:13.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: ) 9 : cluster [DBG] 5.d scrub ok 2021-03-15T22:19:13.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:19:13.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: 03-15T22:19:12.621335+0000 2021-03-15T22:19:13.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: mon.a (mon.0) 569 : 2021-03-15T22:19:13.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:13.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:19:13.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: -15T22:19:12.621941+0000 mgr.y (mgr 2021-03-15T22:19:13.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: .14144) 304 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:19:13.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: 2021-03-15T22:19:13.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: audit 2021-03-15T22:19:12.625199 2021-03-15T22:19:13.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: +0000 mon.a (mon.0) 570 : audit 2021-03-15T22:19:13.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:13.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: cephadm 2021-03-15T 2021-03-15T22:19:13.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: 22:19:12.625951+0000 mgr.y (mgr.14144) 2021-03-15T22:19:13.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: 305 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:19:13.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:13.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:13.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:13.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:19:13.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:12.626341+0000 mgr.y (mgr.14144) 306 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:13.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:13.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:13.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:13.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:13.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:12.626700+0000 mgr.y (mgr.14144) 307 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:13.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:13.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:13.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:13.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:13.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:12.627040+0000 mgr.y (mgr.14144) 308 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:13.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:13.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:13.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:13.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:13.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:12.627388+0000 mgr.y (mgr 2021-03-15T22:19:13.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: .14144) 309 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:13.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:13.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:13.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:13.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:13.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:12.627737+0000 mgr.y ( 2021-03-15T22:19:13.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: mgr.14144) 310 : cephadm [ERR] 2021-03-15T22:19:13.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: cluster 2021-03-15T22:19:11.891459+0000 2021-03-15T22:19:13.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: osd.4 (osd.4) 8 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:19:13.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:19:13.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: -03-15T22:19:12.158853+0000 osd.2 (osd.2) 9 : cluster [DBG] 2021-03-15T22:19:13.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: 5.d scrub ok 2021-03-15T22:19:13.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:19:13.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: -15T22:19:12.621335+0000 mon.a ( 2021-03-15T22:19:13.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: mon.0) 569 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:13.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:13.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: 2021-03-15T22:19: 2021-03-15T22:19:13.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: 12.621941+0000 mgr.y (mgr.14144) 304 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:19:13.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:19:13.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: -15T22:19:12.625199+0000 mon.a (mon.0) 570 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:13.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: 2021-03-15T22:19:13.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:19:13.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: -03-15T22:19:12.625951+0000 2021-03-15T22:19:13.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: mgr.y (mgr.14144) 305 : cephadm [ERR] 2021-03-15T22:19:13.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:19:13.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:13.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:13.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:13.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:19:13.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:12.626341+0000 mgr.y (mgr.14144) 306 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:13.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:13.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:13.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:13.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:13.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: cephadm 2021-03-15T22 2021-03-15T22:19:13.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:13.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:13.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:13.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:13.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:13.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:13.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: 2021-03-15T22:19:12.628071+0000 mgr.y (mgr.14144) 311 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:13.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:13.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:13.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:13.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:13.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:19:13.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: 03-15T22:19:12.628792+0000 mon.a (mon.0) 571 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:19:13.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:19:13.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[56514]: -15T22:19:12.962878+0000 mgr.y (mgr.14144) 312 : cluster [DBG] pgmap v229: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 157 B/s, 2 objects/s recovering 2021-03-15T22:19:13.547 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:13.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: :19:12.626700+0000 mgr.y (mgr.14144) 307 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:13.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:13.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:13.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:13.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:13.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:12.627040+0000 mgr.y (mgr.14144) 308 : cephadm [ERR] 2021-03-15T22:19:13.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:13.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:13.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:13.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:13.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:13.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:12.627388+0000 mgr.y (mgr.14144) 309 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:13.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:13.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:13.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:13.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:13.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: cephadm 2021-03 2021-03-15T22:19:13.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: -15T22:19:12.627737+0000 mgr.y (mgr.14144) 310 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:13.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:13.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:13.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:13.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:13.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:12.628071+0000 mgr.y (mgr.14144) 311 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:13.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:13.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:13.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:13.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:13.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:13.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:13.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:13.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:13.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: audit 2021-03-15T22:19:12.628792+0000 mon.a ( 2021-03-15T22:19:13.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: mon.0) 571 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:19:13.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:13 smithi076 conmon[62156]: cluster 2021-03-15T22:19:12.962878+0000 mgr.y (mgr.14144) 312 : cluster [DBG] pgmap v229: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 157 B/s, 2 objects/s recovering 2021-03-15T22:19:13.759 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Executing migration" logger=migrator id="add unique index cache_data.cache_key" 2021-03-15T22:19:13.760 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Created default admin" logger=sqlstore user=admin 2021-03-15T22:19:14.036 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing HTTPServer" logger=server 2021-03-15T22:19:14.037 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing BackendPluginManager" logger=server 2021-03-15T22:19:14.037 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing PluginManager" logger=server 2021-03-15T22:19:14.037 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Starting plugin search" logger=plugins 2021-03-15T22:19:14.037 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Registering plugin" logger=plugins name="Pie Chart" 2021-03-15T22:19:14.038 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Registering plugin" logger=plugins name="Status Panel" 2021-03-15T22:19:14.038 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing HooksService" logger=server 2021-03-15T22:19:14.038 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing OSSLicensingService" logger=server 2021-03-15T22:19:14.039 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing InternalMetricsService" logger=server 2021-03-15T22:19:14.039 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing RemoteCache" logger=server 2021-03-15T22:19:14.039 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing RenderingService" logger=server 2021-03-15T22:19:14.040 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing AlertEngine" logger=server 2021-03-15T22:19:14.040 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing QuotaService" logger=server 2021-03-15T22:19:14.040 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing ServerLockService" logger=server 2021-03-15T22:19:14.040 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing UserAuthTokenService" logger=server 2021-03-15T22:19:14.040 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing DatasourceCacheService" logger=server 2021-03-15T22:19:14.041 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing LoginService" logger=server 2021-03-15T22:19:14.041 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing SearchService" logger=server 2021-03-15T22:19:14.041 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing TracingService" logger=server 2021-03-15T22:19:14.042 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing UsageStatsService" logger=server 2021-03-15T22:19:14.042 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing CleanUpService" logger=server 2021-03-15T22:19:14.042 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing NotificationService" logger=server 2021-03-15T22:19:14.042 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing provisioningServiceImpl" logger=server 2021-03-15T22:19:14.043 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=warn msg="[Deprecated] the datasource provisioning config is outdated. please upgrade" logger=provisioning.datasources filename=/etc/grafana/provisioning/datasources/ceph-dashboard.yml 2021-03-15T22:19:14.043 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="inserting datasource from configuration " logger=provisioning.datasources name=Dashboard1 2021-03-15T22:19:14.043 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Backend rendering via phantomJS" logger=rendering renderer=phantomJS 2021-03-15T22:19:14.043 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=warn msg="phantomJS is deprecated and will be removed in a future release. You should consider migrating from phantomJS to grafana-image-renderer plugin. Read more at https://grafana.com/docs/grafana/latest/administration/image_rendering/" logger=rendering renderer=phantomJS 2021-03-15T22:19:14.044 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="Initializing Stream Manager" 2021-03-15T22:19:14.044 INFO:journalctl@ceph.grafana.a.smithi125.stdout:Mar 15 22:19:13 smithi125 conmon[77264]: t=2021-03-15T22:19:13+0000 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=https subUrl= socket= 2021-03-15T22:19:14.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:14 smithi076 conmon[56514]: cluster 2021-03-15T22:19:13.525878+0000 osd.6 (osd.6) 7 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:19:14.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:14 smithi076 conmon[56514]: cluster 2021-03-15T22:19:13.733795+0000 osd.5 (osd.5) 11 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:14.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:14 smithi076 conmon[56514]: cluster 2021-03-15T22:19:13.733891+0000 osd.5 (osd.5) 12 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:14.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:14 smithi076 conmon[62156]: cluster 2021-03-15T22:19:13.525878+0000 2021-03-15T22:19:14.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:14 smithi076 conmon[62156]: osd.6 (osd.6) 7 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:19:14.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:14 smithi076 conmon[62156]: cluster 2021-03-15T22:19:13.733795+0000 osd.5 (osd.5) 11 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:14.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:14 smithi076 conmon[62156]: cluster 2021-03-15T22:19:13.733891+0000 osd.5 (osd.5) 12 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:14.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:14 smithi125 conmon[56739]: cluster 2021-03-15T22:19:13.525878+0000 osd.6 (osd.6) 2021-03-15T22:19:14.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:14 smithi125 conmon[56739]: 7 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:19:14.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:14 smithi125 conmon[56739]: cluster 2021-03-15T22:19:13.733795+0000 osd.5 (osd.5) 11 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:14.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:14 smithi125 conmon[56739]: cluster 2021-03-15T22:19:13.733891+0000 osd.5 (osd.5) 12 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:15.163 INFO:teuthology.orchestra.run.smithi076.stdout:34359738411 2021-03-15T22:19:15.449 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: cluster 2021-03-15T22:19:13.514434+0000 osd.1 (osd.1) 2021-03-15T22:19:15.449 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: 11 : cluster [DBG] 5.c scrub ok 2021-03-15T22:19:15.450 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: cluster 2021-03-15T22:19:13.870199+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:19:15.450 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: cluster 2021-03-15T22:19:15.450 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: 2021-03-15T22:19:14.629704+0000 mon.a (mon.0) 572 2021-03-15T22:19:15.450 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-03-15T22:19:15.450 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: cluster 2021-03-15T22:19:15.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: 2021-03-15T 2021-03-15T22:19:15.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: 22:19:14.963205+0000 mgr.y (mgr 2021-03-15T22:19:15.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: .14144) 313 : cluster [DBG] pgmap v231: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:15.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: audit 2021-03-15T22:19:15.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: 2021-03-15T22:19: 2021-03-15T22:19:15.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: 15.161537+0000 mon.a (mon. 2021-03-15T22:19:15.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:15 smithi125 conmon[56739]: 0) 573 : audit [DBG] from='client.? 172.21.15.76:0/2683057873' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-03-15T22:19:15.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[56514]: cluster 2021-03-15T22:19:13.514434 2021-03-15T22:19:15.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[56514]: +0000 osd.1 (osd.1) 11 : cluster [DBG] 5.c scrub ok 2021-03-15T22:19:15.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[56514]: cluster 2021-03-15T22:19:13.870199+0000 osd.0 (osd.0) 16 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:19:15.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[56514]: cluster 2021-03-15T22:19:14.629704+0000 mon.a (mon.0) 572 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-03-15T22:19:15.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[56514]: cluster 2021-03-15T22:19:14.963205+0000 mgr.y (mgr.14144) 313 : cluster [DBG] pgmap v231: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:15.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[56514]: audit 2021-03-15T22:19:15.161537+0000 mon.a (mon.0) 573 : audit [DBG] from='client.? 172.21.15.76:0/2683057873' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-03-15T22:19:15.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: cluster 2021-03-15T22:19:13.514434+0000 2021-03-15T22:19:15.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: osd.1 (osd.1) 11 : cluster [DBG] 5.c scrub ok 2021-03-15T22:19:15.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: cluster 2021-03-15T22:19:13.870199+0000 osd.0 (osd.0 2021-03-15T22:19:15.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: ) 16 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:19:15.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: cluster 2021-03-15T22:19:15.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: 2021-03-15T22:19:14.629704+0000 mon.a (mon 2021-03-15T22:19:15.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: .0) 572 : cluster [DBG] osdmap e84: 8 total, 8 up, 8 in 2021-03-15T22:19:15.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: cluster 2021-03-15T22:19:15.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:19:15.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: :19:14.963205+0000 mgr.y (mgr. 2021-03-15T22:19:15.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: 14144) 313 : cluster [DBG] pgmap v231: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:15.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: audit 2021-03-15T22:19:15.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: 2021-03-15T22:19: 2021-03-15T22:19:15.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: 15.161537+0000 mon.a (mon.0) 573 2021-03-15T22:19:15.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:15 smithi076 conmon[62156]: : audit [DBG] from='client.? 172.21.15.76:0/2683057873' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2021-03-15T22:19:15.799 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738407 got 34359738411 for osd.0 2021-03-15T22:19:15.800 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd last-stat-seq osd.1 2021-03-15T22:19:16.078 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:16.509 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: debug 2021-03-15T22:19:16.294+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:19:16.509 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:16.510 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:16.510 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:16.510 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:16.510 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:16.511 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:16.511 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:16.511 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:19:16.511 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: debug 2021-03-15T22:19:16.294+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:16.512 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:16.512 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:16.512 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:16.512 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:16.512 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:16.513 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:16.513 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:16.513 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:16.514 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: debug 2021-03-15T22:19:16.295+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:16.514 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:16.514 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:16.514 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:16.515 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:16.515 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:16.515 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:16.516 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:16.516 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:16.516 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: debug 2021-03-15T22:19:16.295+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:16.516 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:16.516 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:16.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:16.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:16.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:16.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:16.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:16.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:16.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: debug 2021-03-15T22:19:16.295+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:16.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:16.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:16.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:16.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:16.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:16.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:16.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:16.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:16.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: debug 2021-03-15T22:19:16.296+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:16.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:16.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:16.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:16.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:16.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:16.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:16.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:16.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:16.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: debug 2021-03-15T22:19:16.296+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:16.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:16.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:16.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:16.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:16.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:16.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:16.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:16.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:16.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56514]: cluster 2021-03-15T22:19:14.850599+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:19:16.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56514]: cluster 2021-03-15T22:19:14.859439+0000 osd.0 (osd.0) 17 : cluster [DBG] 3.1b deep-scrub ok 2021-03-15T22:19:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56514]: cluster 2021-03-15T22:19:15.249072+0000 osd.2 (osd.2) 10 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:19:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56514]: cluster 2021-03-15T22:19:15.593300+0000 osd.1 (osd.1) 12 : cluster [DBG] 6.1 deep-scrub ok 2021-03-15T22:19:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56514]: audit 2021-03-15T22:19:16.288629+0000 mon.a (mon.0) 574 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[56514]: audit 2021-03-15T22:19:16.294942+0000 mon.a (mon.0) 575 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:16.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: cluster 2021-03-15T22:19:14.850599+0000 osd.3 2021-03-15T22:19:16.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: (osd.3) 13 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:19:16.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: cluster 2021-03-15T22:19:14.859439+0000 osd.0 (osd.0 2021-03-15T22:19:16.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: ) 17 : cluster [DBG] 3.1b deep-scrub ok 2021-03-15T22:19:16.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:19:16.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: T22:19:15.249072+0000 osd.2 ( 2021-03-15T22:19:16.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: osd.2) 10 : cluster [DBG] 2021-03-15T22:19:16.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: 5.6 scrub ok 2021-03-15T22:19:16.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:19:16.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: 15T22:19:15.593300+0000 osd.1 (osd. 2021-03-15T22:19:16.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: 1) 12 : cluster [DBG] 6.1 deep-scrub ok 2021-03-15T22:19:16.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:19:16.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: 03-15T22:19:16.288629+0000 2021-03-15T22:19:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: mon.a (mon.0) 574 : audit 2021-03-15T22:19:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:19:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: 03-15T22:19:16.294942+0000 2021-03-15T22:19:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: mon.a (mon.0) 575 : 2021-03-15T22:19:16.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:16 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:17.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:16 smithi125 conmon[56739]: cluster 2021-03-15T22:19:14.850599+0000 osd.3 (osd.3) 13 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:19:17.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:16 smithi125 conmon[56739]: cluster 2021-03-15T22:19:14.859439+0000 osd.0 (osd.0) 17 : cluster [DBG] 3.1b deep-scrub ok 2021-03-15T22:19:17.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:16 smithi125 conmon[56739]: cluster 2021-03-15T22:19:15.249072+0000 osd.2 (osd.2) 10 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:19:17.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:16 smithi125 conmon[56739]: cluster 2021-03-15T22:19:15.593300+0000 osd.1 (osd.1) 12 : cluster [DBG] 6.1 deep-scrub ok 2021-03-15T22:19:17.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:16 smithi125 conmon[56739]: audit 2021-03-15T22:19: 2021-03-15T22:19:17.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:16 smithi125 conmon[56739]: 16.288629+0000 mon.a (mon.0 2021-03-15T22:19:17.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:16 smithi125 conmon[56739]: ) 574 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:17.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:16 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:19:17.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:16 smithi125 conmon[56739]: -15T22:19:16.294942+0000 mon.a ( 2021-03-15T22:19:17.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:16 smithi125 conmon[56739]: mon.0) 575 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:17.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cluster 2021-03-15T22:19:15.850213+0000 osd.3 ( 2021-03-15T22:19:17.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: osd.3) 14 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:19:17.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cluster 2021-03-15T22:19:16.224868+0000 osd.2 (osd.2) 11 : cluster [DBG] 6.14 deep-scrub ok 2021-03-15T22:19:17.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:16.289362+0000 mgr.y (mgr.14144) 314 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:19:17.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:16.295858+0000 mgr.y (mgr.14144) 315 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:19:17.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:17.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:17.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:17.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:19:17.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:16.296404+0000 mgr.y (mgr.14144) 316 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:17.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:17.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:17.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:17.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:17.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:16.296818+0000 mgr.y (mgr.14144) 317 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:17.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:17.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:17.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cluster 2021-03-15T22:19:15.850213+0000 2021-03-15T22:19:17.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: osd.3 (osd.3) 14 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:19:17.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cluster 2021-03-15T22:19:16.224868+0000 osd.2 (osd.2) 11 : cluster 2021-03-15T22:19:17.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: [DBG] 6.14 deep-scrub ok 2021-03-15T22:19:17.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:17.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:19:17.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: 22:19:16.289362+0000 mgr.y (mgr.14144) 2021-03-15T22:19:17.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: 314 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:19:17.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cephadm 2021-03 2021-03-15T22:19:17.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: -15T22:19:16.295858+0000 mgr.y (mgr. 2021-03-15T22:19:17.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: 14144) 315 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:19:17.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:17.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:17.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:17.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:19:17.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:16.296404+0000 mgr.y (mgr.14144) 316 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:17.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:17.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:17.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:17.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:17.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:19:17.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: -03-15T22:19:16.296818+0000 mgr.y (mgr.14144) 317 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:17.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:17.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:17.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:17.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:16.297199+0000 mgr.y (mgr.14144) 318 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:17.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:17.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:17.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:17.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:17.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:16.297585+0000 mgr.y (mgr.14144) 319 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:17.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:17.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:17.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:17.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:17.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:16.297967+0000 mgr.y (mgr.14144) 320 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:17.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:16.298334+0000 mgr.y (mgr.14144) 321 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:17.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:17.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:17.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:17.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:17.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cluster 2021-03-15T22:19:16.581507+0000 osd.6 (osd.6) 8 : cluster [DBG] 5.2 deep-scrub ok 2021-03-15T22:19:17.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cluster 2021-03-15T22:19:16.638421+0000 osd.1 (osd.1) 13 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:19:17.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[56514]: cluster 2021-03-15T22:19:16.963829+0000 mgr.y (mgr.14144) 322 : cluster [DBG] pgmap v232: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:17.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:17.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:17.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:17.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:16.297199+0000 mgr.y (mgr 2021-03-15T22:19:17.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: .14144) 318 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:17.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:17.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:17.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:17.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:17.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:16.297585+0000 mgr.y (mgr.14144) 319 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:17.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:17.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:17.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:17.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:17.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:16.297967+0000 mgr.y (mgr.14144) 320 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:17.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:17.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:17.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:17.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:17.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:16.298334+0000 mgr.y (mgr.14144) 321 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:17.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:17.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:17.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:17.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:17.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:17.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:17.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:17.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:17.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:19:17.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: 22:19:16.581507+0000 osd.6 (osd.6) 8 : cluster [DBG] 5.2 deep-scrub ok 2021-03-15T22:19:17.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cluster 2021-03-15T22:19:16.638421+0000 osd.1 (osd.1) 13 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:19:17.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: cluster 2021-03-15T22:19:16.963829+0000 mgr.y (mgr.14144) 322 : cluster 2021-03-15T22:19:17.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:17 smithi076 conmon[62156]: [DBG] pgmap v232: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:17.804 INFO:teuthology.orchestra.run.smithi076.stdout:55834574889 2021-03-15T22:19:18.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cluster 2021-03-15T22:19:15.850213+0000 osd.3 (osd.3) 14 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:19:18.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cluster 2021-03-15T22:19:16.224868+0000 2021-03-15T22:19:18.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: osd.2 (osd.2) 11 : cluster [DBG] 6.14 deep-scrub ok 2021-03-15T22:19:18.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:16.289362+0000 mgr.y (mgr.14144) 314 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:19:18.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cephadm 2021-03- 2021-03-15T22:19:18.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: 15T22:19:16.295858+0000 2021-03-15T22:19:18.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: mgr.y (mgr.14144) 315 : cephadm [ERR] 2021-03-15T22:19:18.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:19:18.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:18.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:18.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:18.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:18.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:18.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:18.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:18.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:19:18.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:16.296404+0000 mgr.y (mgr.14144) 316 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:18.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:18.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:18.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:18.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:18.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:18.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:18.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:18.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:18.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:16.296818+0000 mgr.y (mgr.14144) 317 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:18.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:18.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:18.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:18.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:18.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:18.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:18.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:18.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:18.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:18.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: 2021-03-15T22:19:16.297199+0000 mgr.y (mgr.14144) 318 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:18.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:18.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:18.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:18.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:18.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:18.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:18.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:18.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:18.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:18.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: 2021-03-15T22:19:16.297585+0000 mgr.y (mgr.14144) 319 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:18.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:18.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:18.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:18.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:18.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:18.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:18.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:18.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:18.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:16. 2021-03-15T22:19:18.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: 297967+0000 mgr.y (mgr.14144) 320 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:18.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:18.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:18.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:18.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:18.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:18.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:18.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:18.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:18.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cephadm 2021-03-15T22:19 2021-03-15T22:19:18.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: :16.298334+0000 mgr.y (mgr.14144) 321 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:18.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:18.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:18.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:18.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:18.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:18.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:18.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:18.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:18.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cluster 2021-03-15T22:19: 2021-03-15T22:19:18.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: 16.581507+0000 osd.6 (osd.6) 8 : cluster [DBG] 5.2 deep-scrub ok 2021-03-15T22:19:18.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cluster 2021-03-15T22:19:16.638421+0000 osd.1 (osd.1 2021-03-15T22:19:18.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: ) 13 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:19:18.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:19:18.055 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: 15T22:19:16.963829+0000 2021-03-15T22:19:18.055 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: mgr.y (mgr.14144) 322 : cluster [DBG] 2021-03-15T22:19:18.055 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:17 smithi125 conmon[56739]: pgmap v232: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:18.507 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574885 got 55834574889 for osd.1 2021-03-15T22:19:18.507 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd last-stat-seq osd.2 2021-03-15T22:19:18.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[56514]: cluster 2021-03-15T22:19:17.295402+0000 osd.5 (osd. 2021-03-15T22:19:18.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[56514]: 5) 13 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:19:18.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[56514]: cluster 2021-03-15T22:19:17.649958+0000 osd.1 (osd.1) 14 : cluster [DBG] 4.4 deep-scrub ok 2021-03-15T22:19:18.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[56514]: cluster 2021-03-15T22:19:17.752489+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.1e scrub ok 2021-03-15T22:19:18.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[56514]: audit 2021-03-15T22:19:17.785345+0000 mon.a (mon.0) 576 : audit [DBG] from='client.? 172.21.15.76:0/2766939446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-03-15T22:19:18.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[56514]: cluster 2021-03-15T22:19:17.797280+0000 osd.7 (osd.7) 10 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:19:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: cluster 2021-03-15T22:19:17.295402+0000 osd.5 (osd 2021-03-15T22:19:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: .5) 13 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:19:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:19:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: 19:17.649958+0000 osd.1 (osd.1) 14 : cluster [DBG] 4.4 deep-scrub ok 2021-03-15T22:19:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:19:18.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: :19:17.752489+0000 osd.0 (osd.0) 18 : cluster 2021-03-15T22:19:18.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: [DBG] 5.1e scrub ok 2021-03-15T22:19:18.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: audit 2021 2021-03-15T22:19:18.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: -03-15T22:19:17.785345+0000 mon.a (mon.0) 576 : audit 2021-03-15T22:19:18.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/2766939446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-03-15T22:19:18.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: cluster 2021-03-15T22:19:18.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: 2021-03-15T22:19:17.797280+0000 osd.7 (osd.7) 2021-03-15T22:19:18.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:18 smithi076 conmon[62156]: 10 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:19:18.797 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:19.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:18 smithi125 conmon[56739]: cluster 2021-03-15T22:19:17.295402+0000 osd.5 (osd.5) 13 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:19:19.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:18 smithi125 conmon[56739]: cluster 2021-03-15T22:19:17.649958+0000 2021-03-15T22:19:19.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:18 smithi125 conmon[56739]: osd.1 (osd.1) 14 : cluster [DBG] 4.4 deep-scrub ok 2021-03-15T22:19:19.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:18 smithi125 conmon[56739]: cluster 2021-03-15T22:19:17.752489+0000 osd.0 (osd.0) 18 : cluster [DBG] 5.1e scrub ok 2021-03-15T22:19:19.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:18 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:19:19.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:18 smithi125 conmon[56739]: T22:19:17.785345+0000 mon.a (mon.0) 576 : audit [DBG] from='client.? 172.21.15.76:0/2766939446' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2021-03-15T22:19:19.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:18 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:19:19.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:18 smithi125 conmon[56739]: -03-15T22:19:17 2021-03-15T22:19:19.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:18 smithi125 conmon[56739]: .797280+0000 osd.7 (osd.7) 10 : 2021-03-15T22:19:19.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:18 smithi125 conmon[56739]: cluster [DBG] 6.15 scrub ok 2021-03-15T22:19:19.655 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:19:19.924 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[56514]: cluster 2021-03-15T22:19:18.197221+0000 osd.2 (osd.2) 12 : 2021-03-15T22:19:19.925 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[56514]: cluster [DBG] 5.8 scrub ok 2021-03-15T22:19:19.925 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[56514]: cluster 2021-03-15T22:19:18.252453+0000 osd.5 (osd.5) 14 : cluster [DBG] 3.c scrub ok 2021-03-15T22:19:19.925 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[56514]: cluster 2021-03-15T22:19:18.591325+0000 osd.6 (osd.6) 9 : cluster [DBG] 3.a scrub ok 2021-03-15T22:19:19.926 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[56514]: cluster 2021-03-15T22:19:18.783289+0000 osd.7 (osd.7) 11 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:19:19.926 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[56514]: cluster 2021-03-15T22:19:18.964295+0000 mgr.y (mgr.14144) 323 : cluster [DBG] pgmap v233: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:19:19.926 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[62156]: cluster 2021-03-15T22:19:18.197221+0000 2021-03-15T22:19:19.927 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[62156]: osd.2 (osd.2) 12 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:19:19.927 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[62156]: cluster 2021-03-15T22:19:18.252453+0000 osd.5 (osd.5) 14 : cluster [DBG] 3.c scrub ok 2021-03-15T22:19:19.927 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[62156]: cluster 2021-03-15T22:19:18.591325+0000 osd.6 (osd.6) 9 : cluster [DBG] 3.a scrub ok 2021-03-15T22:19:19.927 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[62156]: cluster 2021-03-15T22:19:18.783289+0000 osd.7 (osd.7) 11 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:19:19.927 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:19 smithi076 conmon[62156]: cluster 2021-03-15T22:19:18.964295+0000 mgr.y (mgr.14144) 323 : cluster [DBG] pgmap v233: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:19:20.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[56739]: cluster 2021-03-15T22:19:18.197221+0000 osd.2 (osd.2) 12 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:19:20.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[56739]: cluster 2021-03-15T22:19:18.252453+0000 osd.5 (osd.5) 14 : cluster [DBG] 3.c scrub ok 2021-03-15T22:19:20.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[56739]: cluster 2021-03-15T22:19: 2021-03-15T22:19:20.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[56739]: 18.591325+0000 osd.6 (osd.6) 9 : cluster [DBG] 3.a scrub ok 2021-03-15T22:19:20.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[56739]: cluster 2021-03-15T22:19:20.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[56739]: 2021-03-15T22:19:18. 2021-03-15T22:19:20.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[56739]: 783289+0000 osd.7 (osd.7) 2021-03-15T22:19:20.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[56739]: 11 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:19:20.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[56739]: cluster 2021-03-15T22:19:20.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[56739]: 2021-03-15T22:19:18.964295+0000 mgr.y 2021-03-15T22:19:20.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:19 smithi125 conmon[56739]: (mgr.14144) 323 : cluster [DBG] pgmap v233: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:19:20.581 INFO:teuthology.orchestra.run.smithi076.stdout:77309411366 2021-03-15T22:19:21.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:20 smithi076 conmon[56514]: audit 2021-03-15T22:19:19.364074+0000 mgr.y (mgr.14144) 324 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:21.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:20 smithi076 conmon[56514]: cluster 2021-03-15T22:19:19.751277+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:19:21.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:20 smithi076 conmon[56514]: audit 2021-03-15T22:19:20.580020+0000 mon.a (mon.0) 577 : audit [DBG] from='client.? 172.21.15.76:0/1032888868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-03-15T22:19:21.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:20 smithi076 conmon[62156]: audit 2021-03-15T22:19:19.364074+0000 2021-03-15T22:19:21.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:20 smithi076 conmon[62156]: mgr.y (mgr.14144) 324 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:21.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:20 smithi076 conmon[62156]: cluster 2021-03-15T22:19:19.751277+0000 2021-03-15T22:19:21.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:20 smithi076 conmon[62156]: osd.0 (osd.0) 19 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:19:21.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:20 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:19:21.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:20 smithi076 conmon[62156]: 19:20.580020+0000 mon.a (mon.0) 577 2021-03-15T22:19:21.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:20 smithi076 conmon[62156]: : audit [DBG] from='client.? 172.21.15.76:0/1032888868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-03-15T22:19:21.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:20 smithi125 conmon[56739]: audit 2021-03-15T22:19:19.364074+0000 mgr.y (mgr.14144) 324 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:21.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:20 smithi125 conmon[56739]: cluster 2021-03-15T22:19:19 2021-03-15T22:19:21.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:20 smithi125 conmon[56739]: .751277+0000 osd.0 (osd.0) 19 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:19:21.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:20 smithi125 conmon[56739]: audit 2021-03-15T22:19:20.580020+0000 mon.a (mon.0) 577 : audit [DBG] from='client.? 172.21.15.76:0/1032888868' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2021-03-15T22:19:21.297 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411362 got 77309411366 for osd.2 2021-03-15T22:19:21.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd last-stat-seq osd.3 2021-03-15T22:19:21.577 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:21.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[62156]: cluster 2021-03-15T22:19:19.908211 2021-03-15T22:19:21.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[62156]: +0000 osd.3 (osd.3) 15 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:19:21.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:19:21.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[62156]: 03-15T22:19:20.964752+0000 mgr.y (mgr.14144 2021-03-15T22:19:21.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[62156]: ) 325 : cluster [DBG] pgmap v234: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:21.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[62156]: cluster 2021-03-15T22:19:21.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[62156]: 2021-03-15T22:19: 2021-03-15T22:19:21.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[62156]: 21.043255+0000 osd.4 (osd.4 2021-03-15T22:19:21.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[62156]: ) 9 : cluster [DBG] 5.1c scrub ok 2021-03-15T22:19:21.969 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[56514]: cluster 2021-03-15T22:19:19.908211 2021-03-15T22:19:21.969 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[56514]: +0000 osd.3 (osd.3) 15 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:19:21.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[56514]: cluster 2021-03-15T22:19:20.964752+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v234: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:21.970 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:21 smithi076 conmon[56514]: cluster 2021-03-15T22:19:21.043255+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.1c scrub ok 2021-03-15T22:19:22.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:21 smithi125 conmon[56739]: cluster 2021-03-15T22:19:19.908211+0000 osd.3 (osd.3) 15 : cluster [DBG] 2021-03-15T22:19:22.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:21 smithi125 conmon[56739]: 6.6 scrub ok 2021-03-15T22:19:22.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:21 smithi125 conmon[56739]: cluster 2021-03-15T22:19:20.964752+0000 mgr.y (mgr.14144) 325 : cluster [DBG] pgmap v234: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:22.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:21 smithi125 conmon[56739]: cluster 2021-03-15T22:19:21.043255+0000 osd.4 (osd.4) 9 : cluster [DBG] 5.1c scrub ok 2021-03-15T22:19:22.941 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:22 smithi076 conmon[56514]: cluster 2021-03-15T22:19:20.880632+0000 osd.3 (osd.3) 16 : cluster 2021-03-15T22:19:22.941 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:22 smithi076 conmon[56514]: [DBG] 3.2 scrub ok 2021-03-15T22:19:22.941 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:22 smithi076 conmon[56514]: cluster 2021-03-15T22:19:21.655915+0000 osd.6 (osd.6) 10 : cluster [DBG] 5.1f scrub ok 2021-03-15T22:19:22.942 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:22 smithi076 conmon[62156]: cluster 2021-03-15T22:19:20.880632+0000 osd.3 2021-03-15T22:19:22.942 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:22 smithi076 conmon[62156]: (osd.3) 16 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:19:22.942 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:22 smithi076 conmon[62156]: cluster 2021-03-15T22:19:21. 2021-03-15T22:19:22.942 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:22 smithi076 conmon[62156]: 655915+0000 osd.6 (osd.6) 10 : cluster [DBG] 5.1f scrub ok 2021-03-15T22:19:23.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:22 smithi125 conmon[56739]: cluster 2021-03-15T22:19:20.880632+0000 osd.3 (osd.3) 16 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:19:23.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:22 smithi125 conmon[56739]: cluster 2021-03-15T22:19:21.655915+0000 osd.6 (osd.6) 10 : cluster [DBG] 5.1f scrub ok 2021-03-15T22:19:23.349 INFO:teuthology.orchestra.run.smithi076.stdout:111669149730 2021-03-15T22:19:23.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: cluster 2021-03-15T22:19:21.899965+0000 2021-03-15T22:19:23.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: osd.3 (osd.3) 17 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:19:23.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: cluster 2021-03-15T22:19:22.172600+0000 osd.2 (osd.2) 13 : cluster [DBG] 6.f deep-scrub ok 2021-03-15T22:19:23.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: cluster 2021-03-15T22:19: 2021-03-15T22:19:23.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: 22.700015+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.a scrub ok 2021-03-15T22:19:23.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:19:23.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: 15T22:19:22.965283+0000 mgr.y ( 2021-03-15T22:19:23.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: mgr.14144) 326 : cluster [DBG] pgmap v235: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2021-03-15T22:19:23.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: audit 2021-03-15T22:19:23.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: 2021-03-15T22:19:23.348488+0000 2021-03-15T22:19:23.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: mon.a (mon.0) 578 : audit [DBG] 2021-03-15T22:19:23.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/4170615334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-03-15T22:19:24.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[56514]: cluster 2021-03-15T22:19:21.899965+0000 osd.3 (osd.3) 17 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:19:24.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[56514]: cluster 2021-03-15T22:19:22.172600+0000 osd.2 (osd.2) 13 : cluster [DBG] 6.f deep-scrub ok 2021-03-15T22:19:24.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[56514]: cluster 2021-03-15T22:19:22.700015+0000 osd.6 (osd.6) 11 : cluster [DBG] 4.a scrub ok 2021-03-15T22:19:24.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[56514]: cluster 2021-03-15T22:19:22.965283+0000 mgr.y (mgr.14144) 326 : cluster [DBG] pgmap v235: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2021-03-15T22:19:24.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[56514]: audit 2021-03-15T22:19:23.348488+0000 2021-03-15T22:19:24.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:23 smithi076 conmon[56514]: mon.a (mon.0) 578 : audit [DBG] from='client.? 172.21.15.76:0/4170615334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-03-15T22:19:24.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: cluster 2021-03-15T22:19: 2021-03-15T22:19:24.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: 21.899965+0000 osd.3 (osd.3) 17 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:19:24.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: cluster 2021-03-15T22:19:24.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: 2021-03-15T22:19:22.172600+0000 osd.2 (osd.2) 13 : cluster 2021-03-15T22:19:24.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: [DBG] 6.f deep-scrub ok 2021-03-15T22:19:24.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:19:24.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: :19:22.700015+0000 osd.6 (osd.6) 2021-03-15T22:19:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: 11 : cluster [DBG] 4.a scrub ok 2021-03-15T22:19:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: cluster 2021-03-15T22:19:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: 2021-03-15T22:19:22.965283+0000 2021-03-15T22:19:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: mgr.y (mgr.14144) 326 : cluster [DBG] 2021-03-15T22:19:24.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: pgmap v235: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 614 B/s rd, 0 op/s 2021-03-15T22:19:24.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: audit 2021-03-15T22:19 2021-03-15T22:19:24.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: :23.348488+0000 mon.a (mon.0) 578 2021-03-15T22:19:24.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:23 smithi125 conmon[56739]: : audit [DBG] from='client.? 172.21.15.76:0/4170615334' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2021-03-15T22:19:24.040 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149727 got 111669149730 for osd.3 2021-03-15T22:19:24.041 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd last-stat-seq osd.4 2021-03-15T22:19:24.322 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:24.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[56514]: cluster 2021-03-15T22:19:23.578005+0000 osd.2 (osd.2) 14 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:24.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[56514]: 2021-03-15T22:19:24.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[56514]: cluster 2021-03-15T22:19:23.578092+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:24.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[56514]: cluster 2021-03-15T22:19:23.669649+0000 mon.a (mon.0) 579 : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-03-15T22:19:24.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[56514]: cluster 2021-03-15T22:19:23.732183+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:19:24.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[56514]: cluster 2021-03-15T22:19:23.843887+0000 osd.7 (osd.7) 12 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:19:24.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[62156]: cluster 2021-03-15T22:19:23.578005+0000 osd.2 (osd.2) 14 : 2021-03-15T22:19:24.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[62156]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:24.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[62156]: cluster 2021-03-15T22:19:23.578092+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:24.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[62156]: cluster 2021-03-15T22:19:23.669649+0000 mon.a (mon.0) 579 2021-03-15T22:19:24.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[62156]: : cluster [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-03-15T22:19:24.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[62156]: cluster 2021-03-15T22:19:24.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[62156]: 2021-03-15T22:19:23.732183+0000 osd.6 (osd.6) 12 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:19:24.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[62156]: 2021-03-15T22:19:24.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:19:24.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:24 smithi076 conmon[62156]: 15T22:19:23.843887+0000 osd.7 (osd.7) 12 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:19:25.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:24 smithi125 conmon[56739]: cluster 2021-03-15T22:19:23.578005+0000 osd.2 (osd.2) 14 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:24 smithi125 conmon[56739]: cluster 2021-03-15T22:19:23.578092+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:24 smithi125 conmon[56739]: cluster 2021-03-15T22:19:23.669649+0000 mon.a (mon.0) 579 : cluster 2021-03-15T22:19:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:24 smithi125 conmon[56739]: [DBG] osdmap e85: 8 total, 8 up, 8 in 2021-03-15T22:19:25.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:24 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:19:25.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:24 smithi125 conmon[56739]: T22:19:23.732183+0000 2021-03-15T22:19:25.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:24 smithi125 conmon[56739]: osd.6 (osd.6) 12 : cluster 2021-03-15T22:19:25.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:24 smithi125 conmon[56739]: [DBG] 4.5 scrub ok 2021-03-15T22:19:25.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:24 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:19:25.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:24 smithi125 conmon[56739]: :19:23.843887+0000 osd.7 (osd.7 2021-03-15T22:19:25.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:24 smithi125 conmon[56739]: ) 12 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:19:26.010 INFO:teuthology.orchestra.run.smithi076.stdout:133143986208 2021-03-15T22:19:26.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[56514]: cluster 2021-03-15T22:19:24.142546+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:19:26.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[56514]: cluster 2021-03-15T22:19:24.648196+0000 osd.1 (osd.1) 15 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:19:26.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[56514]: cluster 2021-03-15T22:19:24.902800+0000 osd.3 (osd.3) 18 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:19:26.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[56514]: cluster 2021-03-15T22:19:24.965647+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v237: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:26.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[56514]: cluster 2021-03-15T22:19:25.184256+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:19:26.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[56514]: cluster 2021-03-15T22:19:25.604213+0000 osd.0 (osd.0) 20 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:19:26.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[62156]: cluster 2021-03-15T22:19:24.142546+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:19:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[62156]: cluster 2021-03-15T22:19:24.648196+0000 osd.1 (osd.1) 15 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:19:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[62156]: cluster 2021-03-15T22:19:24.902800+0000 osd.3 (osd.3) 18 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:19:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[62156]: cluster 2021-03-15T22:19:24.965647+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v237: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:26.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[62156]: cluster 2021-03-15T22:19:25.184256+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:19:26.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:25 smithi076 conmon[62156]: cluster 2021-03-15T22:19:25.604213+0000 osd.0 (osd.0) 20 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:19:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:25 smithi125 conmon[56739]: cluster 2021-03-15T22:19:24.142546+0000 osd.2 (osd.2) 16 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:19:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:25 smithi125 conmon[56739]: cluster 2021-03-15T22:19:24.648196+0000 2021-03-15T22:19:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:25 smithi125 conmon[56739]: osd.1 (osd.1) 15 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:19:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:25 smithi125 conmon[56739]: cluster 2021-03-15T22:19:24.902800+0000 osd.3 (osd.3) 18 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:19:26.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:25 smithi125 conmon[56739]: cluster 2021-03-15T22:19:24.965647+0000 mgr.y (mgr.14144) 327 : cluster [DBG] pgmap v237: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:26.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:25 smithi125 conmon[56739]: cluster 2021-03-15T22:19:25.184256+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:19:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:25 smithi125 conmon[56739]: cluster 2021-03-15T22:19:25. 2021-03-15T22:19:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:25 smithi125 conmon[56739]: 604213+0000 osd.0 (osd.0) 20 : cluster 2021-03-15T22:19:26.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:25 smithi125 conmon[56739]: [DBG] 5.3 scrub ok 2021-03-15T22:19:26.796 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986204 got 133143986208 for osd.4 2021-03-15T22:19:26.797 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd last-stat-seq osd.5 2021-03-15T22:19:27.080 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[56514]: cluster 2021-03-15T22:19:25.759019+0000 osd.7 (osd. 2021-03-15T22:19:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[56514]: 7) 13 : cluster [DBG] 6.12 deep-scrub ok 2021-03-15T22:19:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[56514]: cluster 2021-03-15T22:19:25.878003+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.c deep-scrub ok 2021-03-15T22:19:27.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[56514]: audit 2021-03-15T22:19:26.009241+0000 mon.a (mon.0) 580 : audit [DBG] from='client.? 172.21.15.76:0/185177531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-03-15T22:19:27.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[56514]: cluster 2021-03-15T22:19:26.087616+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:19:27.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[56514]: cluster 2021-03-15T22:19:26.615374+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.10 scrub ok 2021-03-15T22:19:27.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[62156]: cluster 2021-03-15T22:19:25.759019+0000 2021-03-15T22:19:27.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[62156]: osd.7 (osd.7) 13 : cluster [DBG] 6.12 deep-scrub ok 2021-03-15T22:19:27.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[62156]: cluster 2021-03-15T22:19:25.878003+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.c deep-scrub ok 2021-03-15T22:19:27.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[62156]: audit 2021-03-15T22:19:26.009241+0000 mon.a (mon.0) 580 : audit [DBG] from='client.? 172.21.15.76:0/185177531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-03-15T22:19:27.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[62156]: cluster 2021-03-15T22:19:26.087616+0000 osd.4 (osd.4) 10 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:19:27.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:26 smithi076 conmon[62156]: cluster 2021-03-15T22:19:26.615374+0000 osd.0 (osd.0) 21 : cluster [DBG] 6.10 scrub ok 2021-03-15T22:19:27.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: cluster 2021-03-15T22:19:25.759019+0000 osd.7 (osd.7) 13 : cluster [DBG] 6.12 deep-scrub ok 2021-03-15T22:19:27.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: cluster 2021-03-15T22:19:25.878003+0000 osd.3 (osd.3) 19 : cluster [DBG] 6.c deep-scrub ok 2021-03-15T22:19:27.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: audit 2021-03-15T22:19:26.009241+0000 2021-03-15T22:19:27.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: mon.a (mon.0) 580 : audit [DBG] from='client.? 172.21.15.76:0/185177531' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2021-03-15T22:19:27.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:19:27.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: 03-15T22:19:26. 2021-03-15T22:19:27.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: 087616+0000 osd.4 (osd.4) 10 : cluster 2021-03-15T22:19:27.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: [DBG] 4.1 scrub ok 2021-03-15T22:19:27.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:19:27.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: 03-15T22:19:26. 2021-03-15T22:19:27.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: 615374+0000 osd.0 (osd.0) 21 : cluster 2021-03-15T22:19:27.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:26 smithi125 conmon[56739]: [DBG] 6.10 scrub ok 2021-03-15T22:19:28.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:27 smithi076 conmon[56514]: cluster 2021-03-15T22:19:26.865891+0000 osd.3 (osd.3) 20 2021-03-15T22:19:28.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:27 smithi076 conmon[56514]: : cluster [DBG] 6.1b scrub ok 2021-03-15T22:19:28.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:27 smithi076 conmon[56514]: cluster 2021-03-15T22:19:26.966150+0000 mgr.y (mgr.14144) 328 : cluster [DBG] pgmap v238: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:27 smithi076 conmon[62156]: cluster 2021-03-15T22:19: 2021-03-15T22:19:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:27 smithi076 conmon[62156]: 26.865891+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:19:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:27 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:19:28.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:27 smithi076 conmon[62156]: -03-15T22:19:26. 2021-03-15T22:19:28.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:27 smithi076 conmon[62156]: 966150+0000 mgr.y (mgr.14144) 2021-03-15T22:19:28.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:27 smithi076 conmon[62156]: 328 : cluster [DBG] pgmap v238: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:28.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:27 smithi125 conmon[56739]: cluster 2021-03-15T22:19:26.865891+0000 osd.3 (osd.3) 20 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:19:28.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:27 smithi125 conmon[56739]: cluster 2021-03-15T22:19:26.966150+0000 mgr.y (mgr.14144) 328 : cluster [DBG] pgmap v238: 132 pgs: 132 active+clean; 25 KiB data, 51 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:28.825 INFO:teuthology.orchestra.run.smithi076.stdout:163208757277 2021-03-15T22:19:29.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[56514]: cluster 2021-03-15T22:19:27.165113+0000 osd.2 (osd.2) 17 : cluster 2021-03-15T22:19:29.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[56514]: [DBG] 4.1c scrub ok 2021-03-15T22:19:29.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[56514]: cluster 2021-03-15T22:19:27.800101+0000 osd.7 (osd.7) 14 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:19:29.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[56514]: cluster 2021-03-15T22:19:28.122072+0000 osd.4 (osd.4) 11 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:19:29.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[56514]: cluster 2021-03-15T22:19:28.581915+0000 osd.0 (osd.0) 22 : cluster [DBG] 4.b scrub ok 2021-03-15T22:19:29.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[56514]: audit 2021-03-15T22:19:28.823932+0000 mon.a (mon.0) 581 : audit [DBG] from='client.? 172.21.15.76:0/3523943796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-03-15T22:19:29.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: cluster 2021-03-15T22:19:27.165113+0000 osd.2 (osd.2 2021-03-15T22:19:29.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: ) 17 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:19:29.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: cluster 2021-03-15T22:19:27.800101+0000 osd.7 (osd. 2021-03-15T22:19:29.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: 7) 14 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:19:29.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:19:29.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: -03-15T22:19:28.122072+0000 osd.4 (osd.4) 11 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:19:29.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:19:29.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: -15T22:19:28.581915+0000 osd.0 (osd.0) 22 : 2021-03-15T22:19:29.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: cluster [DBG] 4.b scrub ok 2021-03-15T22:19:29.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: audit 2021-03-15T22:19:29.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: 2021-03-15T22:19:28.823932+0000 mon.a (mon.0) 2021-03-15T22:19:29.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:28 smithi076 conmon[62156]: 581 : audit [DBG] from='client.? 172.21.15.76:0/3523943796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-03-15T22:19:29.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: cluster 2021-03-15T22:19:27.165113+0000 osd.2 (osd.2) 17 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:19:29.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: cluster 2021-03-15T22:19:27.800101+0000 osd.7 (osd.7) 14 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:19:29.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: cluster 2021-03-15T22:19:28.122072 2021-03-15T22:19:29.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: +0000 osd.4 (osd.4) 11 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:19:29.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:19:29.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: -03-15T22:19: 2021-03-15T22:19:29.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: 28.581915+0000 osd.0 (osd.0) 22 2021-03-15T22:19:29.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: : cluster [DBG] 4.b scrub ok 2021-03-15T22:19:29.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: audit 2021-03-15T22:19:29.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: 2021-03-15T22:19 2021-03-15T22:19:29.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: :28.823932+0000 mon.a (mon.0) 2021-03-15T22:19:29.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:28 smithi125 conmon[56739]: 581 : audit [DBG] from='client.? 172.21.15.76:0/3523943796' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2021-03-15T22:19:29.471 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757273 got 163208757277 for osd.5 2021-03-15T22:19:29.472 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd last-stat-seq osd.6 2021-03-15T22:19:29.751 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:29.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:19:30.182 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[56514]: cluster 2021-03-15T22:19:28.737789+0000 osd.6 (osd.6) 13 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:19:30.182 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[56514]: cluster 2021-03-15T22:19:28.773742+0000 osd.1 (osd.1) 16 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:19:30.182 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[56514]: cluster 2021-03-15T22:19:28.966790+0000 mgr.y (mgr.14144) 329 : cluster [DBG] pgmap v239: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:30.182 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[56514]: cluster 2021-03-15T22:19:29.179796+0000 osd.5 (osd.5) 16 : cluster [DBG] 5.16 deep-scrub ok 2021-03-15T22:19:30.183 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[56514]: audit 2021-03-15T22:19:29.375068+0000 mgr.y (mgr.14144) 330 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:30.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[62156]: cluster 2021-03-15T22:19:28.737789+0000 osd.6 (osd.6) 2021-03-15T22:19:30.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[62156]: 13 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:19:30.183 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[62156]: cluster 2021-03-15T22:19:28.773742+0000 osd.1 (osd.1) 16 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:19:30.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[62156]: 2021-03-15T22:19:30.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[62156]: cluster 2021-03-15T22:19:28.966790+0000 mgr.y 2021-03-15T22:19:30.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[62156]: (mgr.14144) 329 : cluster [DBG] pgmap v239: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:30.184 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[62156]: cluster 2021-03-15T22:19:30.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[62156]: 2021-03-15T22:19:29.179796+0000 osd.5 (osd.5) 16 : cluster 2021-03-15T22:19:30.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[62156]: [DBG] 5.16 deep-scrub ok 2021-03-15T22:19:30.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:19:30.185 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:29 smithi076 conmon[62156]: 03-15T22:19:29.375068+0000 mgr.y (mgr.14144) 330 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:30.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: cluster 2021-03-15T22:19:28.737789+0000 osd.6 (osd.6) 13 : cluster 2021-03-15T22:19:30.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: [DBG] 5.11 scrub ok 2021-03-15T22:19:30.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: cluster 2021-03-15T22:19:28.773742+0000 osd.1 (osd.1) 16 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:19:30.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: cluster 2021-03-15T22:19:28.966790+0000 2021-03-15T22:19:30.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: mgr.y (mgr.14144) 329 : cluster [DBG] pgmap v239: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:30.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:19:30.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: -15T22:19:29.179796+0000 2021-03-15T22:19:30.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: osd.5 (osd.5) 16 : cluster [DBG] 2021-03-15T22:19:30.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: 5.16 deep-scrub ok 2021-03-15T22:19:30.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:19:30.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: 03-15T22:19:29 2021-03-15T22:19:30.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: .375068+0000 mgr.y (mgr.14144) 330 2021-03-15T22:19:30.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:29 smithi125 conmon[56739]: : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:31.116 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[56514]: cluster 2021-03-15T22:19:29.099642+0000 2021-03-15T22:19:31.117 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[56514]: osd.2 (osd.2) 18 : cluster [DBG] 5.b scrub ok 2021-03-15T22:19:31.117 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[56514]: cluster 2021-03-15T22:19:29.812441+0000 osd.1 (osd.1) 17 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:19:31.117 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[56514]: cluster 2021-03-15T22:19:30.205908+0000 osd.5 (osd.5) 17 : cluster [DBG] 5.1b scrub ok 2021-03-15T22:19:31.117 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[56514]: cluster 2021-03-15T22:19:30.589057+0000 osd.0 (osd.0) 23 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:19:31.118 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[56514]: cluster 2021-03-15T22:19:30.967227+0000 mgr.y (mgr.14144) 331 : cluster [DBG] pgmap v240: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:31.118 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: cluster 2021-03-15T22:19:29.099642+0000 osd.2 2021-03-15T22:19:31.118 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: (osd.2) 18 : cluster [DBG] 5.b scrub ok 2021-03-15T22:19:31.119 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: cluster 2021-03-15T22:19:29.812441+0000 osd.1 (osd 2021-03-15T22:19:31.119 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: .1) 17 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:19:31.119 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:19:31.119 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: 19:30.205908+0000 osd.5 (osd.5) 2021-03-15T22:19:31.119 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: 17 : cluster [DBG] 5.1b scrub ok 2021-03-15T22:19:31.120 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:19:31.120 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: 03-15T22:19:30.589057+0000 2021-03-15T22:19:31.120 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: osd.0 (osd.0) 23 : cluster [DBG] 2021-03-15T22:19:31.120 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: 4.6 scrub ok 2021-03-15T22:19:31.121 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:19:31.121 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: 15T22:19:30.967227+0000 mgr.y (mgr. 2021-03-15T22:19:31.121 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:30 smithi076 conmon[62156]: 14144) 331 : cluster [DBG] pgmap v240: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:31.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: cluster 2021-03-15T22:19:29.099642+0000 osd.2 (osd.2) 18 : cluster [DBG] 5.b scrub ok 2021-03-15T22:19:31.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: cluster 2021-03-15T22:19:29.812441+0000 osd.1 (osd.1) 17 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:19:31.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:19:31.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: 03-15T22:19:30.205908+0000 osd.5 (osd.5) 17 : cluster 2021-03-15T22:19:31.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: [DBG] 5.1b scrub ok 2021-03-15T22:19:31.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:19:31.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: 03-15T22:19:30.589057+0000 2021-03-15T22:19:31.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: osd.0 (osd.0) 23 : cluster 2021-03-15T22:19:31.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: [DBG] 4.6 scrub ok 2021-03-15T22:19:31.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:19:31.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: -15T22:19:30.967227+0000 mgr.y (mgr 2021-03-15T22:19:31.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:30 smithi125 conmon[56739]: .14144) 331 : cluster [DBG] pgmap v240: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:31.452 INFO:teuthology.orchestra.run.smithi076.stdout:193273528346 2021-03-15T22:19:32.170 INFO:tasks.cephadm.ceph_manager.ceph:need seq 193273528342 got 193273528346 for osd.6 2021-03-15T22:19:32.171 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd last-stat-seq osd.7 2021-03-15T22:19:32.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[56514]: cluster 2021-03-15T22:19:30.852585+0000 osd.6 (osd.6) 2021-03-15T22:19:32.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[56514]: 14 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:32.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[56514]: cluster 2021-03-15T22:19:30.852731+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:32.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[56514]: cluster 2021-03-15T22:19:30.854892+0000 osd.7 (osd.7) 15 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:19:32.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[56514]: cluster 2021-03-15T22:19:30.994664+0000 mon.a (mon.0) 582 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2021-03-15T22:19:32.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[56514]: audit 2021-03-15T22:19:31.451253+0000 mon.a (mon.0) 583 : audit [DBG] from='client.? 172.21.15.76:0/2579511464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-03-15T22:19:32.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[62156]: cluster 2021-03-15T22:19:30.852585+0000 osd.6 (osd.6 2021-03-15T22:19:32.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[62156]: ) 14 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:32.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[62156]: cluster 2021-03-15T22:19:30.852731+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:32.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[62156]: cluster 2021-03-15T22:19:30.854892+0000 osd.7 (osd. 2021-03-15T22:19:32.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[62156]: 7) 15 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:19:32.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:19:32.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[62156]: 15T22:19:30.994664+0000 mon.a (mon. 2021-03-15T22:19:32.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[62156]: 0) 582 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2021-03-15T22:19:32.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:19:32.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[62156]: 03-15T22:19:31.451253+0000 mon.a ( 2021-03-15T22:19:32.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:31 smithi076 conmon[62156]: mon.0) 583 : audit [DBG] from='client.? 172.21.15.76:0/2579511464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-03-15T22:19:32.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:31 smithi125 conmon[56739]: cluster 2021-03-15T22:19:30.852585+0000 osd.6 (osd.6) 14 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:32.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:31 smithi125 conmon[56739]: 2021-03-15T22:19:32.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:31 smithi125 conmon[56739]: cluster 2021-03-15T22:19:30.852731+0000 osd.6 (osd.6) 15 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:32.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:31 smithi125 conmon[56739]: cluster 2021-03-15T22:19:30.854892+0000 osd.7 (osd.7) 15 2021-03-15T22:19:32.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:31 smithi125 conmon[56739]: : cluster [DBG] 6.1c scrub ok 2021-03-15T22:19:32.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:31 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:19:32.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:31 smithi125 conmon[56739]: :19:30.994664+0000 mon.a (mon.0) 582 : cluster [DBG] osdmap e86: 8 total, 8 up, 8 in 2021-03-15T22:19:32.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:31 smithi125 conmon[56739]: audit 2021-03-15T22:19:32.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:31 smithi125 conmon[56739]: 2021-03-15T22:19:31. 2021-03-15T22:19:32.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:31 smithi125 conmon[56739]: 451253+0000 mon.a (mon.0) 583 : audit [DBG] 2021-03-15T22:19:32.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:31 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2579511464' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2021-03-15T22:19:32.449 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:33.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:33 smithi076 conmon[56514]: cluster 2021-03-15T22:19:31.884958+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:19:33.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:33 smithi076 conmon[56514]: cluster 2021-03-15T22:19:32.967846+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v242: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-15T22:19:33.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:33 smithi076 conmon[62156]: cluster 2021-03-15T22:19:31.884958+0000 osd.7 (osd.7) 16 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:19:33.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:33 smithi076 conmon[62156]: cluster 2021-03-15T22:19:32.967846+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v242: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-15T22:19:33.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:33 smithi125 conmon[56739]: cluster 2021-03-15T22:19:31.884958+0000 osd.7 (osd.7) 16 : cluster 2021-03-15T22:19:33.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:33 smithi125 conmon[56739]: [DBG] 6.5 scrub ok 2021-03-15T22:19:33.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:33 smithi125 conmon[56739]: cluster 2021-03-15T22:19:32.967846+0000 mgr.y (mgr.14144) 332 : cluster [DBG] pgmap v242: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-15T22:19:34.147 INFO:teuthology.orchestra.run.smithi076.stdout:223338299415 2021-03-15T22:19:34.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[56514]: cluster 2021-03-15T22:19:32.759800+0000 osd.1 (osd.1) 18 2021-03-15T22:19:34.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[56514]: : cluster [DBG] 5.12 scrub ok 2021-03-15T22:19:34.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[56514]: cluster 2021-03-15T22:19:32.880356+0000 osd.3 (osd.3) 21 : cluster [DBG] 5.17 scrub ok 2021-03-15T22:19:34.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[56514]: cluster 2021-03-15T22:19:33.261208+0000 osd.5 (osd.5) 18 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:19:34.234 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[62156]: cluster 2021-03-15T22:19:32.759800+0000 2021-03-15T22:19:34.235 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[62156]: osd.1 (osd.1) 18 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:19:34.235 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[62156]: cluster 2021-03-15T22:19:32. 2021-03-15T22:19:34.235 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[62156]: 880356+0000 osd.3 (osd.3) 21 : cluster [DBG] 5.17 scrub ok 2021-03-15T22:19:34.236 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[62156]: cluster 2021-03-15T22:19:34.236 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[62156]: 2021-03-15T22:19:33.261208 2021-03-15T22:19:34.236 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 18 : cluster 2021-03-15T22:19:34.237 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:34 smithi076 conmon[62156]: [DBG] 4.15 scrub ok 2021-03-15T22:19:34.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:34 smithi125 conmon[56739]: cluster 2021-03-15T22:19:32.759800+0000 osd.1 (osd.1) 18 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:19:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:34 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:19:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:34 smithi125 conmon[56739]: -15T22:19:32.880356+0000 osd.3 (osd.3) 21 : cluster [DBG] 5.17 scrub ok 2021-03-15T22:19:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:34 smithi125 conmon[56739]: cluster 2021-03-15T22:19:33.261208+0000 osd.5 (osd.5) 18 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:19:34.821 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299411 got 223338299415 for osd.7 2021-03-15T22:19:34.822 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-03-15T22:19:34.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:19:35.114 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:35.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[56514]: audit 2021-03-15T22:19:34.145545+0000 mon.a (mon.0) 584 2021-03-15T22:19:35.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[56514]: : audit [DBG] from='client.? 172.21.15.76:0/2235781559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-03-15T22:19:35.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[56514]: cluster 2021-03-15T22:19:34.179424+0000 osd.4 (osd.4) 12 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:19:35.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[56514]: cluster 2021-03-15T22:19:34.968294+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v243: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:35.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: audit 2021-03-15T22:19: 2021-03-15T22:19:35.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: 34.145545+0000 mon.a (mon.0) 584 : audit [DBG] from='client.? 172.21.15.76:0/2235781559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-03-15T22:19:35.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: cluster 2021-03-15T22:19:35.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: 2021-03 2021-03-15T22:19:35.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: -15T22:19:34 2021-03-15T22:19:35.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: .179424+0000 osd.4 (osd. 2021-03-15T22:19:35.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: 4) 12 : cluster [DBG] 2021-03-15T22:19:35.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: 6.2 scrub ok 2021-03-15T22:19:35.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:19:35.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: -03-15T22:19:34.968294 2021-03-15T22:19:35.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 333 2021-03-15T22:19:35.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:35 smithi076 conmon[62156]: : cluster [DBG] pgmap v243: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:35.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:35 smithi125 conmon[56739]: audit 2021-03-15T22:19:34.145545+0000 mon.a (mon.0) 584 : audit [DBG] from='client.? 172.21.15.76:0/2235781559' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2021-03-15T22:19:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:35 smithi125 conmon[56739]: cluster 2021-03-15T22:19:34.179424+0000 osd.4 (osd.4) 12 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:19:35.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:35 smithi125 conmon[56739]: cluster 2021-03-15T22:19:34.968294+0000 mgr.y (mgr.14144) 333 : cluster [DBG] pgmap v243: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:36.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:36 smithi076 conmon[56514]: cluster 2021-03-15T22:19:35.185883+0000 osd.4 (osd.4) 13 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:19:36.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:36 smithi076 conmon[62156]: cluster 2021-03-15T22:19:35.185883+0000 2021-03-15T22:19:36.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:36 smithi076 conmon[62156]: osd.4 (osd.4) 13 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:19:36.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:36 smithi125 conmon[56739]: cluster 2021-03-15T22:19:35.185883+0000 osd.4 (osd.4) 13 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:19:36.865 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:19:36.867 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:19:37.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:37 smithi076 conmon[56514]: cluster 2021-03-15T22:19:35.779098+0000 osd.1 (osd. 2021-03-15T22:19:37.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:37 smithi076 conmon[56514]: 1) 19 : cluster [DBG] 5.1d scrub ok 2021-03-15T22:19:37.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:37 smithi076 conmon[56514]: cluster 2021-03-15T22:19:35.934467+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:19:37.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:37 smithi076 conmon[56514]: audit 2021-03-15T22:19:36.848868+0000 mgr.y (mgr.14144) 334 : audit [DBG] from='client.14631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:37.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:37 smithi076 conmon[56514]: cluster 2021-03-15T22:19:36.968910+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v244: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:37.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:37 smithi076 conmon[62156]: cluster 2021-03-15T22:19:35.779098+0000 osd.1 (osd.1) 2021-03-15T22:19:37.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:37 smithi076 conmon[62156]: 19 : cluster [DBG] 5.1d scrub ok 2021-03-15T22:19:37.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:37 smithi076 conmon[62156]: cluster 2021-03-15T22:19:35.934467+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:19:37.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:37 smithi076 conmon[62156]: audit 2021-03-15T22:19:36.848868+0000 mgr.y (mgr.14144) 334 : audit [DBG] from='client.14631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:37.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:37 smithi076 conmon[62156]: cluster 2021-03-15T22:19:36.968910+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v244: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:37.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:37 smithi125 conmon[56739]: cluster 2021-03-15T22:19:35.779098+0000 osd.1 (osd.1) 19 : cluster [DBG] 5.1d scrub ok 2021-03-15T22:19:37.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:37 smithi125 conmon[56739]: 2021-03-15T22:19:37.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:37 smithi125 conmon[56739]: cluster 2021-03-15T22:19:35.934467+0000 osd.3 (osd.3) 22 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:19:37.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:37 smithi125 conmon[56739]: audit 2021-03-15T22:19:36.848868+0000 mgr.y (mgr.14144) 334 : audit [DBG] 2021-03-15T22:19:37.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:37 smithi125 conmon[56739]: from='client.14631 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:37.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:37 smithi125 conmon[56739]: cluster 2021-03-15T22:19: 2021-03-15T22:19:37.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:37 smithi125 conmon[56739]: 36.968910+0000 mgr.y (mgr.14144) 335 : cluster [DBG] pgmap v244: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:37.579 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":243,"stamp":"2021-03-15T22:19:34.967987+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25236,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1173,"num_read_kb":673,"num_write":550,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":571,"ondisk_log_size":571,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":52968,"kb_used_data":7080,"kb_used_omap":0,"kb_used_meta":45824,"kb_avail":749908248,"statfs":{"total":767960285184,"available":767906045952,"internally_reserved":0,"allocated":7249920,"data_stored":2870763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46923776},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002561"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:28.122186+0000","last_change":"2021-03-15T22:19:28.122158+0000","last_active":"2021-03-15T22:19:28.122186+0000","last_peered":"2021-03-15T22:19:28.122186+0000","last_clean":"2021-03-15T22:19:28.122186+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:19:28.122186+0000","last_undegraded":"2021-03-15T22:19:28.122186+0000","last_fullsized":"2021-03-15T22:19:28.122186+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.122079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.122079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"34","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646338+0000","last_change":"2021-03-15T22:17:58.015465+0000","last_active":"2021-03-15T22:19:24.646338+0000","last_peered":"2021-03-15T22:19:24.646338+0000","last_clean":"2021-03-15T22:19:24.646338+0000","last_became_active":"2021-03-15T22:17:58.014719+0000","last_became_peered":"2021-03-15T22:17:58.014719+0000","last_unstale":"2021-03-15T22:19:24.646338+0000","last_undegraded":"2021-03-15T22:19:24.646338+0000","last_fullsized":"2021-03-15T22:19:24.646338+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"45","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842608+0000","last_change":"2021-03-15T22:19:08.789028+0000","last_active":"2021-03-15T22:19:23.842608+0000","last_peered":"2021-03-15T22:19:23.842608+0000","last_clean":"2021-03-15T22:19:23.842608+0000","last_became_active":"2021-03-15T22:17:54.284855+0000","last_became_peered":"2021-03-15T22:17:54.284855+0000","last_unstale":"2021-03-15T22:19:23.842608+0000","last_undegraded":"2021-03-15T22:19:23.842608+0000","last_fullsized":"2021-03-15T22:19:23.842608+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:19:08.788964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:08.788964+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"69","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672254+0000","last_change":"2021-03-15T22:19:11.891528+0000","last_active":"2021-03-15T22:19:23.672254+0000","last_peered":"2021-03-15T22:19:23.672254+0000","last_clean":"2021-03-15T22:19:23.672254+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:19:23.672254+0000","last_undegraded":"2021-03-15T22:19:23.672254+0000","last_fullsized":"2021-03-15T22:19:23.672254+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:11.891465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.891465+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:26.865995+0000","last_change":"2021-03-15T22:19:26.865961+0000","last_active":"2021-03-15T22:19:26.865995+0000","last_peered":"2021-03-15T22:19:26.865995+0000","last_clean":"2021-03-15T22:19:26.865995+0000","last_became_active":"2021-03-15T22:18:00.288598+0000","last_became_peered":"2021-03-15T22:18:00.288598+0000","last_unstale":"2021-03-15T22:19:26.865995+0000","last_undegraded":"2021-03-15T22:19:26.865995+0000","last_fullsized":"2021-03-15T22:19:26.865995+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.865895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.865895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"34","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672252+0000","last_change":"2021-03-15T22:17:58.011237+0000","last_active":"2021-03-15T22:19:23.672252+0000","last_peered":"2021-03-15T22:19:23.672252+0000","last_clean":"2021-03-15T22:19:23.672252+0000","last_became_active":"2021-03-15T22:17:58.010916+0000","last_became_peered":"2021-03-15T22:17:58.010916+0000","last_unstale":"2021-03-15T22:19:23.672252+0000","last_undegraded":"2021-03-15T22:19:23.672252+0000","last_fullsized":"2021-03-15T22:19:23.672252+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669169+0000","last_change":"2021-03-15T22:19:10.851317+0000","last_active":"2021-03-15T22:19:23.669169+0000","last_peered":"2021-03-15T22:19:23.669169+0000","last_clean":"2021-03-15T22:19:23.669169+0000","last_became_active":"2021-03-15T22:17:54.283819+0000","last_became_peered":"2021-03-15T22:17:54.283819+0000","last_unstale":"2021-03-15T22:19:23.669169+0000","last_undegraded":"2021-03-15T22:19:23.669169+0000","last_fullsized":"2021-03-15T22:19:23.669169+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:10.851256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:10.851256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"71","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669284+0000","last_change":"2021-03-15T22:17:55.991923+0000","last_active":"2021-03-15T22:19:23.669284+0000","last_peered":"2021-03-15T22:19:23.669284+0000","last_clean":"2021-03-15T22:19:23.669284+0000","last_became_active":"2021-03-15T22:17:55.991597+0000","last_became_peered":"2021-03-15T22:17:55.991597+0000","last_unstale":"2021-03-15T22:19:23.669284+0000","last_undegraded":"2021-03-15T22:19:23.669284+0000","last_fullsized":"2021-03-15T22:19:23.669284+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"36","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842172+0000","last_change":"2021-03-15T22:19:11.832031+0000","last_active":"2021-03-15T22:19:23.842172+0000","last_peered":"2021-03-15T22:19:23.842172+0000","last_clean":"2021-03-15T22:19:23.842172+0000","last_became_active":"2021-03-15T22:18:00.015776+0000","last_became_peered":"2021-03-15T22:18:00.015776+0000","last_unstale":"2021-03-15T22:19:23.842172+0000","last_undegraded":"2021-03-15T22:19:23.842172+0000","last_fullsized":"2021-03-15T22:19:23.842172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:11.831963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.831963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:30.206036+0000","last_change":"2021-03-15T22:19:30.205991+0000","last_active":"2021-03-15T22:19:30.206036+0000","last_peered":"2021-03-15T22:19:30.206036+0000","last_clean":"2021-03-15T22:19:30.206036+0000","last_became_active":"2021-03-15T22:17:58.008828+0000","last_became_peered":"2021-03-15T22:17:58.008828+0000","last_unstale":"2021-03-15T22:19:30.206036+0000","last_undegraded":"2021-03-15T22:19:30.206036+0000","last_fullsized":"2021-03-15T22:19:30.206036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.205915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.205915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175672+0000","last_change":"2021-03-15T22:18:53.334542+0000","last_active":"2021-03-15T22:19:24.175672+0000","last_peered":"2021-03-15T22:19:24.175672+0000","last_clean":"2021-03-15T22:19:24.175672+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:19:24.175672+0000","last_undegraded":"2021-03-15T22:19:24.175672+0000","last_fullsized":"2021-03-15T22:19:24.175672+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:53.334478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:53.334478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"60","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672688+0000","last_change":"2021-03-15T22:17:55.990767+0000","last_active":"2021-03-15T22:19:23.672688+0000","last_peered":"2021-03-15T22:19:23.672688+0000","last_clean":"2021-03-15T22:19:23.672688+0000","last_became_active":"2021-03-15T22:17:55.990528+0000","last_became_peered":"2021-03-15T22:17:55.990528+0000","last_unstale":"2021-03-15T22:19:23.672688+0000","last_undegraded":"2021-03-15T22:19:23.672688+0000","last_fullsized":"2021-03-15T22:19:23.672688+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"31","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175015+0000","last_change":"2021-03-15T22:18:00.019861+0000","last_active":"2021-03-15T22:19:24.175015+0000","last_peered":"2021-03-15T22:19:24.175015+0000","last_clean":"2021-03-15T22:19:24.175015+0000","last_became_active":"2021-03-15T22:18:00.019585+0000","last_became_peered":"2021-03-15T22:18:00.019585+0000","last_unstale":"2021-03-15T22:19:24.175015+0000","last_undegraded":"2021-03-15T22:19:24.175015+0000","last_fullsized":"2021-03-15T22:19:24.175015+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"34","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733091+0000","last_change":"2021-03-15T22:17:58.013492+0000","last_active":"2021-03-15T22:19:23.733091+0000","last_peered":"2021-03-15T22:19:23.733091+0000","last_clean":"2021-03-15T22:19:23.733091+0000","last_became_active":"2021-03-15T22:17:58.013236+0000","last_became_peered":"2021-03-15T22:17:58.013236+0000","last_unstale":"2021-03-15T22:19:23.733091+0000","last_undegraded":"2021-03-15T22:19:23.733091+0000","last_fullsized":"2021-03-15T22:19:23.733091+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175442+0000","last_change":"2021-03-15T22:19:06.274483+0000","last_active":"2021-03-15T22:19:24.175442+0000","last_peered":"2021-03-15T22:19:24.175442+0000","last_clean":"2021-03-15T22:19:24.175442+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:19:24.175442+0000","last_undegraded":"2021-03-15T22:19:24.175442+0000","last_fullsized":"2021-03-15T22:19:24.175442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:06.274416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.274416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'6","reported_seq":"70","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672984+0000","last_change":"2021-03-15T22:19:02.277922+0000","last_active":"2021-03-15T22:19:23.672984+0000","last_peered":"2021-03-15T22:19:23.672984+0000","last_clean":"2021-03-15T22:19:23.672984+0000","last_became_active":"2021-03-15T22:19:02.274105+0000","last_became_peered":"2021-03-15T22:19:02.274105+0000","last_unstale":"2021-03-15T22:19:23.672984+0000","last_undegraded":"2021-03-15T22:19:23.672984+0000","last_fullsized":"2021-03-15T22:19:23.672984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2021-03-15T22:18:59.037504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.037504+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.673225+0000","last_change":"2021-03-15T22:18:00.287135+0000","last_active":"2021-03-15T22:19:23.673225+0000","last_peered":"2021-03-15T22:19:23.673225+0000","last_clean":"2021-03-15T22:19:23.673225+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:19:23.673225+0000","last_undegraded":"2021-03-15T22:19:23.673225+0000","last_fullsized":"2021-03-15T22:19:23.673225+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"34","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646716+0000","last_change":"2021-03-15T22:17:58.016331+0000","last_active":"2021-03-15T22:19:24.646716+0000","last_peered":"2021-03-15T22:19:24.646716+0000","last_clean":"2021-03-15T22:19:24.646716+0000","last_became_active":"2021-03-15T22:17:58.016067+0000","last_became_peered":"2021-03-15T22:17:58.016067+0000","last_unstale":"2021-03-15T22:19:24.646716+0000","last_undegraded":"2021-03-15T22:19:24.646716+0000","last_fullsized":"2021-03-15T22:19:24.646716+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"45","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842757+0000","last_change":"2021-03-15T22:19:14.859504+0000","last_active":"2021-03-15T22:19:23.842757+0000","last_peered":"2021-03-15T22:19:23.842757+0000","last_clean":"2021-03-15T22:19:23.842757+0000","last_became_active":"2021-03-15T22:17:53.976875+0000","last_became_peered":"2021-03-15T22:17:53.976875+0000","last_unstale":"2021-03-15T22:19:23.842757+0000","last_undegraded":"2021-03-15T22:19:23.842757+0000","last_fullsized":"2021-03-15T22:19:23.842757+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.859443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"66","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:27.165241+0000","last_change":"2021-03-15T22:19:27.165212+0000","last_active":"2021-03-15T22:19:27.165241+0000","last_peered":"2021-03-15T22:19:27.165241+0000","last_clean":"2021-03-15T22:19:27.165241+0000","last_became_active":"2021-03-15T22:17:55.992247+0000","last_became_peered":"2021-03-15T22:17:55.992247+0000","last_unstale":"2021-03-15T22:19:27.165241+0000","last_undegraded":"2021-03-15T22:19:27.165241+0000","last_fullsized":"2021-03-15T22:19:27.165241+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:27.165119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.165119+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668832+0000","last_change":"2021-03-15T22:18:00.287310+0000","last_active":"2021-03-15T22:19:23.668832+0000","last_peered":"2021-03-15T22:19:23.668832+0000","last_clean":"2021-03-15T22:19:23.668832+0000","last_became_active":"2021-03-15T22:18:00.286974+0000","last_became_peered":"2021-03-15T22:18:00.286974+0000","last_unstale":"2021-03-15T22:19:23.668832+0000","last_undegraded":"2021-03-15T22:19:23.668832+0000","last_fullsized":"2021-03-15T22:19:23.668832+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672814+0000","last_change":"2021-03-15T22:19:21.043341+0000","last_active":"2021-03-15T22:19:23.672814+0000","last_peered":"2021-03-15T22:19:23.672814+0000","last_clean":"2021-03-15T22:19:23.672814+0000","last_became_active":"2021-03-15T22:17:58.005477+0000","last_became_peered":"2021-03-15T22:17:58.005477+0000","last_unstale":"2021-03-15T22:19:23.672814+0000","last_undegraded":"2021-03-15T22:19:23.672814+0000","last_fullsized":"2021-03-15T22:19:23.672814+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.043261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.043261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672556+0000","last_change":"2021-03-15T22:17:54.284146+0000","last_active":"2021-03-15T22:19:23.672556+0000","last_peered":"2021-03-15T22:19:23.672556+0000","last_clean":"2021-03-15T22:19:23.672556+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:19:23.672556+0000","last_undegraded":"2021-03-15T22:19:23.672556+0000","last_fullsized":"2021-03-15T22:19:23.672556+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"67","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668551+0000","last_change":"2021-03-15T22:17:55.996079+0000","last_active":"2021-03-15T22:19:23.668551+0000","last_peered":"2021-03-15T22:19:23.668551+0000","last_clean":"2021-03-15T22:19:23.668551+0000","last_became_active":"2021-03-15T22:17:55.995747+0000","last_became_peered":"2021-03-15T22:17:55.995747+0000","last_unstale":"2021-03-15T22:19:23.668551+0000","last_undegraded":"2021-03-15T22:19:23.668551+0000","last_fullsized":"2021-03-15T22:19:23.668551+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:30.855012+0000","last_change":"2021-03-15T22:19:30.854972+0000","last_active":"2021-03-15T22:19:30.855012+0000","last_peered":"2021-03-15T22:19:30.855012+0000","last_clean":"2021-03-15T22:19:30.855012+0000","last_became_active":"2021-03-15T22:18:00.015889+0000","last_became_peered":"2021-03-15T22:18:00.015889+0000","last_unstale":"2021-03-15T22:19:30.855012+0000","last_undegraded":"2021-03-15T22:19:30.855012+0000","last_fullsized":"2021-03-15T22:19:30.855012+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.854899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.854899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"61'2","reported_seq":"57","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.731510+0000","last_change":"2021-03-15T22:19:21.656004+0000","last_active":"2021-03-15T22:19:23.731510+0000","last_peered":"2021-03-15T22:19:23.731510+0000","last_clean":"2021-03-15T22:19:23.731510+0000","last_became_active":"2021-03-15T22:17:58.013571+0000","last_became_peered":"2021-03-15T22:17:58.013571+0000","last_unstale":"2021-03-15T22:19:23.731510+0000","last_undegraded":"2021-03-15T22:19:23.731510+0000","last_fullsized":"2021-03-15T22:19:23.731510+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:21.655918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.655918+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646367+0000","last_change":"2021-03-15T22:19:16.638497+0000","last_active":"2021-03-15T22:19:24.646367+0000","last_peered":"2021-03-15T22:19:24.646367+0000","last_clean":"2021-03-15T22:19:24.646367+0000","last_became_active":"2021-03-15T22:17:53.975549+0000","last_became_peered":"2021-03-15T22:17:53.975549+0000","last_unstale":"2021-03-15T22:19:24.646367+0000","last_undegraded":"2021-03-15T22:19:24.646367+0000","last_fullsized":"2021-03-15T22:19:24.646367+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.638428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.638428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'16","reported_seq":"62","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842457+0000","last_change":"2021-03-15T22:17:56.288831+0000","last_active":"2021-03-15T22:19:23.842457+0000","last_peered":"2021-03-15T22:19:23.842457+0000","last_clean":"2021-03-15T22:19:23.842457+0000","last_became_active":"2021-03-15T22:17:56.288283+0000","last_became_peered":"2021-03-15T22:17:56.288283+0000","last_unstale":"2021-03-15T22:19:23.842457+0000","last_undegraded":"2021-03-15T22:19:23.842457+0000","last_fullsized":"2021-03-15T22:19:23.842457+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646797+0000","last_change":"2021-03-15T22:19:07.544919+0000","last_active":"2021-03-15T22:19:24.646797+0000","last_peered":"2021-03-15T22:19:24.646797+0000","last_clean":"2021-03-15T22:19:24.646797+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:19:24.646797+0000","last_undegraded":"2021-03-15T22:19:24.646797+0000","last_fullsized":"2021-03-15T22:19:24.646797+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.544856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.544856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842614+0000","last_change":"2021-03-15T22:19:17.752562+0000","last_active":"2021-03-15T22:19:23.842614+0000","last_peered":"2021-03-15T22:19:23.842614+0000","last_clean":"2021-03-15T22:19:23.842614+0000","last_became_active":"2021-03-15T22:17:58.009184+0000","last_became_peered":"2021-03-15T22:17:58.009184+0000","last_unstale":"2021-03-15T22:19:23.842614+0000","last_undegraded":"2021-03-15T22:19:23.842614+0000","last_fullsized":"2021-03-15T22:19:23.842614+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.752495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.752495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668285+0000","last_change":"2021-03-15T22:19:21.900042+0000","last_active":"2021-03-15T22:19:23.668285+0000","last_peered":"2021-03-15T22:19:23.668285+0000","last_clean":"2021-03-15T22:19:23.668285+0000","last_became_active":"2021-03-15T22:17:53.967947+0000","last_became_peered":"2021-03-15T22:17:53.967947+0000","last_unstale":"2021-03-15T22:19:23.668285+0000","last_undegraded":"2021-03-15T22:19:23.668285+0000","last_fullsized":"2021-03-15T22:19:23.668285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.899969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.899969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"70","reported_epoch":"84","state":"active+clean","last_fresh":"2021-03-15T22:19:14.628002+0000","last_change":"2021-03-15T22:19:13.525956+0000","last_active":"2021-03-15T22:19:14.628002+0000","last_peered":"2021-03-15T22:19:14.628002+0000","last_clean":"2021-03-15T22:19:14.628002+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:19:14.628002+0000","last_undegraded":"2021-03-15T22:19:14.628002+0000","last_fullsized":"2021-03-15T22:19:14.628002+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:19:13.525883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.525883+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:25.759163+0000","last_change":"2021-03-15T22:19:25.759114+0000","last_active":"2021-03-15T22:19:25.759163+0000","last_peered":"2021-03-15T22:19:25.759163+0000","last_clean":"2021-03-15T22:19:25.759163+0000","last_became_active":"2021-03-15T22:18:00.015858+0000","last_became_peered":"2021-03-15T22:18:00.015858+0000","last_unstale":"2021-03-15T22:19:25.759163+0000","last_undegraded":"2021-03-15T22:19:25.759163+0000","last_fullsized":"2021-03-15T22:19:25.759163+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.759034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:28.737870+0000","last_change":"2021-03-15T22:19:28.737870+0000","last_active":"2021-03-15T22:19:28.737870+0000","last_peered":"2021-03-15T22:19:28.737870+0000","last_clean":"2021-03-15T22:19:28.737870+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:19:28.737870+0000","last_undegraded":"2021-03-15T22:19:28.737870+0000","last_fullsized":"2021-03-15T22:19:28.737870+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.737796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.737796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842625+0000","last_change":"2021-03-15T22:19:01.740592+0000","last_active":"2021-03-15T22:19:23.842625+0000","last_peered":"2021-03-15T22:19:23.842625+0000","last_clean":"2021-03-15T22:19:23.842625+0000","last_became_active":"2021-03-15T22:17:53.973507+0000","last_became_peered":"2021-03-15T22:17:53.973507+0000","last_unstale":"2021-03-15T22:19:23.842625+0000","last_undegraded":"2021-03-15T22:19:23.842625+0000","last_fullsized":"2021-03-15T22:19:23.842625+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:01.740496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:01.740496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"58","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668372+0000","last_change":"2021-03-15T22:17:55.996994+0000","last_active":"2021-03-15T22:19:23.668372+0000","last_peered":"2021-03-15T22:19:23.668372+0000","last_clean":"2021-03-15T22:19:23.668372+0000","last_became_active":"2021-03-15T22:17:55.996707+0000","last_became_peered":"2021-03-15T22:17:55.996707+0000","last_unstale":"2021-03-15T22:19:23.668372+0000","last_undegraded":"2021-03-15T22:19:23.668372+0000","last_fullsized":"2021-03-15T22:19:23.668372+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668484+0000","last_change":"2021-03-15T22:19:14.850664+0000","last_active":"2021-03-15T22:19:23.668484+0000","last_peered":"2021-03-15T22:19:23.668484+0000","last_clean":"2021-03-15T22:19:23.668484+0000","last_became_active":"2021-03-15T22:18:00.287972+0000","last_became_peered":"2021-03-15T22:18:00.287972+0000","last_unstale":"2021-03-15T22:19:23.668484+0000","last_undegraded":"2021-03-15T22:19:23.668484+0000","last_fullsized":"2021-03-15T22:19:23.668484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:14.850605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.850605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:27.800237+0000","last_change":"2021-03-15T22:19:27.800200+0000","last_active":"2021-03-15T22:19:27.800237+0000","last_peered":"2021-03-15T22:19:27.800237+0000","last_clean":"2021-03-15T22:19:27.800237+0000","last_became_active":"2021-03-15T22:17:58.013174+0000","last_became_peered":"2021-03-15T22:17:58.013174+0000","last_unstale":"2021-03-15T22:19:27.800237+0000","last_undegraded":"2021-03-15T22:19:27.800237+0000","last_fullsized":"2021-03-15T22:19:27.800237+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:27.800106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.800106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175740+0000","last_change":"2021-03-15T22:19:17.295504+0000","last_active":"2021-03-15T22:19:24.175740+0000","last_peered":"2021-03-15T22:19:24.175740+0000","last_clean":"2021-03-15T22:19:24.175740+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:19:24.175740+0000","last_undegraded":"2021-03-15T22:19:24.175740+0000","last_fullsized":"2021-03-15T22:19:24.175740+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.295412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.295412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'14","reported_seq":"61","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669240+0000","last_change":"2021-03-15T22:17:56.289858+0000","last_active":"2021-03-15T22:19:23.669240+0000","last_peered":"2021-03-15T22:19:23.669240+0000","last_clean":"2021-03-15T22:19:23.669240+0000","last_became_active":"2021-03-15T22:17:56.289456+0000","last_became_peered":"2021-03-15T22:17:56.289456+0000","last_unstale":"2021-03-15T22:19:23.669240+0000","last_undegraded":"2021-03-15T22:19:23.669240+0000","last_fullsized":"2021-03-15T22:19:23.669240+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"36","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:26.615466+0000","last_change":"2021-03-15T22:19:26.615442+0000","last_active":"2021-03-15T22:19:26.615466+0000","last_peered":"2021-03-15T22:19:26.615466+0000","last_clean":"2021-03-15T22:19:26.615466+0000","last_became_active":"2021-03-15T22:18:00.014112+0000","last_became_peered":"2021-03-15T22:18:00.014112+0000","last_unstale":"2021-03-15T22:19:26.615466+0000","last_undegraded":"2021-03-15T22:19:26.615466+0000","last_fullsized":"2021-03-15T22:19:26.615466+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.615380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.615380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668729+0000","last_change":"2021-03-15T22:19:03.699264+0000","last_active":"2021-03-15T22:19:23.668729+0000","last_peered":"2021-03-15T22:19:23.668729+0000","last_clean":"2021-03-15T22:19:23.668729+0000","last_became_active":"2021-03-15T22:17:58.009365+0000","last_became_peered":"2021-03-15T22:17:58.009365+0000","last_unstale":"2021-03-15T22:19:23.668729+0000","last_undegraded":"2021-03-15T22:19:23.668729+0000","last_fullsized":"2021-03-15T22:19:23.668729+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_clean_scrub_stamp":"2021-03-15T22:19:03.699198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733442+0000","last_change":"2021-03-15T22:17:53.980472+0000","last_active":"2021-03-15T22:19:23.733442+0000","last_peered":"2021-03-15T22:19:23.733442+0000","last_clean":"2021-03-15T22:19:23.733442+0000","last_became_active":"2021-03-15T22:17:53.979947+0000","last_became_peered":"2021-03-15T22:17:53.979947+0000","last_unstale":"2021-03-15T22:19:23.733442+0000","last_undegraded":"2021-03-15T22:19:23.733442+0000","last_fullsized":"2021-03-15T22:19:23.733442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"64","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.645843+0000","last_change":"2021-03-15T22:17:55.993221+0000","last_active":"2021-03-15T22:19:24.645843+0000","last_peered":"2021-03-15T22:19:24.645843+0000","last_clean":"2021-03-15T22:19:24.645843+0000","last_became_active":"2021-03-15T22:17:55.991537+0000","last_became_peered":"2021-03-15T22:17:55.991537+0000","last_unstale":"2021-03-15T22:19:24.645843+0000","last_undegraded":"2021-03-15T22:19:24.645843+0000","last_fullsized":"2021-03-15T22:19:24.645843+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668832+0000","last_change":"2021-03-15T22:18:00.018406+0000","last_active":"2021-03-15T22:19:23.668832+0000","last_peered":"2021-03-15T22:19:23.668832+0000","last_clean":"2021-03-15T22:19:23.668832+0000","last_became_active":"2021-03-15T22:18:00.018117+0000","last_became_peered":"2021-03-15T22:18:00.018117+0000","last_unstale":"2021-03-15T22:19:23.668832+0000","last_undegraded":"2021-03-15T22:19:23.668832+0000","last_fullsized":"2021-03-15T22:19:23.668832+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:32.759869+0000","last_change":"2021-03-15T22:19:32.759869+0000","last_active":"2021-03-15T22:19:32.759869+0000","last_peered":"2021-03-15T22:19:32.759869+0000","last_clean":"2021-03-15T22:19:32.759869+0000","last_became_active":"2021-03-15T22:17:58.014621+0000","last_became_peered":"2021-03-15T22:17:58.014621+0000","last_unstale":"2021-03-15T22:19:32.759869+0000","last_undegraded":"2021-03-15T22:19:32.759869+0000","last_fullsized":"2021-03-15T22:19:32.759869+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:32.759804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:32.759804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672933+0000","last_change":"2021-03-15T22:17:53.975528+0000","last_active":"2021-03-15T22:19:23.672933+0000","last_peered":"2021-03-15T22:19:23.672933+0000","last_clean":"2021-03-15T22:19:23.672933+0000","last_became_active":"2021-03-15T22:17:53.975292+0000","last_became_peered":"2021-03-15T22:17:53.975292+0000","last_unstale":"2021-03-15T22:19:23.672933+0000","last_undegraded":"2021-03-15T22:19:23.672933+0000","last_fullsized":"2021-03-15T22:19:23.672933+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"63","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672761+0000","last_change":"2021-03-15T22:17:55.992828+0000","last_active":"2021-03-15T22:19:23.672761+0000","last_peered":"2021-03-15T22:19:23.672761+0000","last_clean":"2021-03-15T22:19:23.672761+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:19:23.672761+0000","last_undegraded":"2021-03-15T22:19:23.672761+0000","last_fullsized":"2021-03-15T22:19:23.672761+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"31","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842408+0000","last_change":"2021-03-15T22:18:00.015339+0000","last_active":"2021-03-15T22:19:23.842408+0000","last_peered":"2021-03-15T22:19:23.842408+0000","last_clean":"2021-03-15T22:19:23.842408+0000","last_became_active":"2021-03-15T22:18:00.015034+0000","last_became_peered":"2021-03-15T22:18:00.015034+0000","last_unstale":"2021-03-15T22:19:23.842408+0000","last_undegraded":"2021-03-15T22:19:23.842408+0000","last_fullsized":"2021-03-15T22:19:23.842408+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"61'2","reported_seq":"54","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500649+0000","last_change":"2021-03-15T22:17:58.011190+0000","last_active":"2021-03-15T22:19:31.500649+0000","last_peered":"2021-03-15T22:19:31.500649+0000","last_clean":"2021-03-15T22:19:31.500649+0000","last_became_active":"2021-03-15T22:17:58.010903+0000","last_became_peered":"2021-03-15T22:17:58.010903+0000","last_unstale":"2021-03-15T22:19:31.500649+0000","last_undegraded":"2021-03-15T22:19:31.500649+0000","last_fullsized":"2021-03-15T22:19:31.500649+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733105+0000","last_change":"2021-03-15T22:17:54.286237+0000","last_active":"2021-03-15T22:19:23.733105+0000","last_peered":"2021-03-15T22:19:23.733105+0000","last_clean":"2021-03-15T22:19:23.733105+0000","last_became_active":"2021-03-15T22:17:54.285851+0000","last_became_peered":"2021-03-15T22:17:54.285851+0000","last_unstale":"2021-03-15T22:19:23.733105+0000","last_undegraded":"2021-03-15T22:19:23.733105+0000","last_fullsized":"2021-03-15T22:19:23.733105+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"64","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668900+0000","last_change":"2021-03-15T22:19:07.874527+0000","last_active":"2021-03-15T22:19:23.668900+0000","last_peered":"2021-03-15T22:19:23.668900+0000","last_clean":"2021-03-15T22:19:23.668900+0000","last_became_active":"2021-03-15T22:17:56.288102+0000","last_became_peered":"2021-03-15T22:17:56.288102+0000","last_unstale":"2021-03-15T22:19:23.668900+0000","last_undegraded":"2021-03-15T22:19:23.668900+0000","last_fullsized":"2021-03-15T22:19:23.668900+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:19:07.874462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.874462+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.673452+0000","last_change":"2021-03-15T22:18:00.015238+0000","last_active":"2021-03-15T22:19:23.673452+0000","last_peered":"2021-03-15T22:19:23.673452+0000","last_clean":"2021-03-15T22:19:23.673452+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:19:23.673452+0000","last_undegraded":"2021-03-15T22:19:23.673452+0000","last_fullsized":"2021-03-15T22:19:23.673452+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"61'2","reported_seq":"59","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:27.023220+0000","last_change":"2021-03-15T22:19:15.850282+0000","last_active":"2021-03-15T22:19:27.023220+0000","last_peered":"2021-03-15T22:19:27.023220+0000","last_clean":"2021-03-15T22:19:27.023220+0000","last_became_active":"2021-03-15T22:17:58.003748+0000","last_became_peered":"2021-03-15T22:17:58.003748+0000","last_unstale":"2021-03-15T22:19:27.023220+0000","last_undegraded":"2021-03-15T22:19:27.023220+0000","last_fullsized":"2021-03-15T22:19:27.023220+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:15.850217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.850217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842221+0000","last_change":"2021-03-15T22:19:07.795140+0000","last_active":"2021-03-15T22:19:23.842221+0000","last_peered":"2021-03-15T22:19:23.842221+0000","last_clean":"2021-03-15T22:19:23.842221+0000","last_became_active":"2021-03-15T22:17:53.968736+0000","last_became_peered":"2021-03-15T22:17:53.968736+0000","last_unstale":"2021-03-15T22:19:23.842221+0000","last_undegraded":"2021-03-15T22:19:23.842221+0000","last_fullsized":"2021-03-15T22:19:23.842221+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.795072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.795072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"59","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175774+0000","last_change":"2021-03-15T22:17:56.290858+0000","last_active":"2021-03-15T22:19:24.175774+0000","last_peered":"2021-03-15T22:19:24.175774+0000","last_clean":"2021-03-15T22:19:24.175774+0000","last_became_active":"2021-03-15T22:17:56.290563+0000","last_became_peered":"2021-03-15T22:17:56.290563+0000","last_unstale":"2021-03-15T22:19:24.175774+0000","last_undegraded":"2021-03-15T22:19:24.175774+0000","last_fullsized":"2021-03-15T22:19:24.175774+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.670156+0000","last_change":"2021-03-15T22:19:16.224939+0000","last_active":"2021-03-15T22:19:23.670156+0000","last_peered":"2021-03-15T22:19:23.670156+0000","last_clean":"2021-03-15T22:19:23.670156+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:19:23.670156+0000","last_undegraded":"2021-03-15T22:19:23.670156+0000","last_fullsized":"2021-03-15T22:19:23.670156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.224872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"34","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668803+0000","last_change":"2021-03-15T22:17:58.001653+0000","last_active":"2021-03-15T22:19:23.668803+0000","last_peered":"2021-03-15T22:19:23.668803+0000","last_clean":"2021-03-15T22:19:23.668803+0000","last_became_active":"2021-03-15T22:17:58.001340+0000","last_became_peered":"2021-03-15T22:17:58.001340+0000","last_unstale":"2021-03-15T22:19:23.668803+0000","last_undegraded":"2021-03-15T22:19:23.668803+0000","last_fullsized":"2021-03-15T22:19:23.668803+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.732886+0000","last_change":"2021-03-15T22:19:13.870789+0000","last_active":"2021-03-15T22:19:23.732886+0000","last_peered":"2021-03-15T22:19:23.732886+0000","last_clean":"2021-03-15T22:19:23.732886+0000","last_became_active":"2021-03-15T22:17:53.976777+0000","last_became_peered":"2021-03-15T22:17:53.976777+0000","last_unstale":"2021-03-15T22:19:23.732886+0000","last_undegraded":"2021-03-15T22:19:23.732886+0000","last_fullsized":"2021-03-15T22:19:23.732886+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"70","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842532+0000","last_change":"2021-03-15T22:19:06.778692+0000","last_active":"2021-03-15T22:19:23.842532+0000","last_peered":"2021-03-15T22:19:23.842532+0000","last_clean":"2021-03-15T22:19:23.842532+0000","last_became_active":"2021-03-15T22:17:56.288678+0000","last_became_peered":"2021-03-15T22:17:56.288678+0000","last_unstale":"2021-03-15T22:19:23.842532+0000","last_undegraded":"2021-03-15T22:19:23.842532+0000","last_fullsized":"2021-03-15T22:19:23.842532+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:06.778622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.778622+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733386+0000","last_change":"2021-03-15T22:19:17.797392+0000","last_active":"2021-03-15T22:19:23.733386+0000","last_peered":"2021-03-15T22:19:23.733386+0000","last_clean":"2021-03-15T22:19:23.733386+0000","last_became_active":"2021-03-15T22:18:00.288462+0000","last_became_peered":"2021-03-15T22:18:00.288462+0000","last_unstale":"2021-03-15T22:19:23.733386+0000","last_undegraded":"2021-03-15T22:19:23.733386+0000","last_fullsized":"2021-03-15T22:19:23.733386+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.797291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.797291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:29.179916+0000","last_change":"2021-03-15T22:19:29.179877+0000","last_active":"2021-03-15T22:19:29.179916+0000","last_peered":"2021-03-15T22:19:29.179916+0000","last_clean":"2021-03-15T22:19:29.179916+0000","last_became_active":"2021-03-15T22:17:58.015682+0000","last_became_peered":"2021-03-15T22:17:58.015682+0000","last_unstale":"2021-03-15T22:19:29.179916+0000","last_undegraded":"2021-03-15T22:19:29.179916+0000","last_fullsized":"2021-03-15T22:19:29.179916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.179800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"41","reported_epoch":"84","state":"active+clean","last_fresh":"2021-03-15T22:19:14.627382+0000","last_change":"2021-03-15T22:19:05.557588+0000","last_active":"2021-03-15T22:19:14.627382+0000","last_peered":"2021-03-15T22:19:14.627382+0000","last_clean":"2021-03-15T22:19:14.627382+0000","last_became_active":"2021-03-15T22:17:53.977470+0000","last_became_peered":"2021-03-15T22:17:53.977470+0000","last_unstale":"2021-03-15T22:19:14.627382+0000","last_undegraded":"2021-03-15T22:19:14.627382+0000","last_fullsized":"2021-03-15T22:19:14.627382+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:05.557534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:05.557534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'7","reported_seq":"47","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668886+0000","last_change":"2021-03-15T22:17:56.289796+0000","last_active":"2021-03-15T22:19:23.668886+0000","last_peered":"2021-03-15T22:19:23.668886+0000","last_clean":"2021-03-15T22:19:23.668886+0000","last_became_active":"2021-03-15T22:17:56.289454+0000","last_became_peered":"2021-03-15T22:17:56.289454+0000","last_unstale":"2021-03-15T22:19:23.668886+0000","last_undegraded":"2021-03-15T22:19:23.668886+0000","last_fullsized":"2021-03-15T22:19:23.668886+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"36","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175506+0000","last_change":"2021-03-15T22:19:11.318218+0000","last_active":"2021-03-15T22:19:24.175506+0000","last_peered":"2021-03-15T22:19:24.175506+0000","last_clean":"2021-03-15T22:19:24.175506+0000","last_became_active":"2021-03-15T22:18:00.286699+0000","last_became_peered":"2021-03-15T22:18:00.286699+0000","last_unstale":"2021-03-15T22:19:24.175506+0000","last_undegraded":"2021-03-15T22:19:24.175506+0000","last_fullsized":"2021-03-15T22:19:24.175506+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:11.318132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.318132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"46","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733721+0000","last_change":"2021-03-15T22:19:06.942939+0000","last_active":"2021-03-15T22:19:23.733721+0000","last_peered":"2021-03-15T22:19:23.733721+0000","last_clean":"2021-03-15T22:19:23.733721+0000","last_became_active":"2021-03-15T22:17:53.979246+0000","last_became_peered":"2021-03-15T22:17:53.979246+0000","last_unstale":"2021-03-15T22:19:23.733721+0000","last_undegraded":"2021-03-15T22:19:23.733721+0000","last_fullsized":"2021-03-15T22:19:23.733721+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:19:06.942870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.942870+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"66","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175012+0000","last_change":"2021-03-15T22:17:56.289996+0000","last_active":"2021-03-15T22:19:24.175012+0000","last_peered":"2021-03-15T22:19:24.175012+0000","last_clean":"2021-03-15T22:19:24.175012+0000","last_became_active":"2021-03-15T22:17:56.289684+0000","last_became_peered":"2021-03-15T22:17:56.289684+0000","last_unstale":"2021-03-15T22:19:24.175012+0000","last_undegraded":"2021-03-15T22:19:24.175012+0000","last_fullsized":"2021-03-15T22:19:24.175012+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"61'2","reported_seq":"55","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884865+0000","last_change":"2021-03-15T22:17:58.015052+0000","last_active":"2021-03-15T22:19:31.884865+0000","last_peered":"2021-03-15T22:19:31.884865+0000","last_clean":"2021-03-15T22:19:31.884865+0000","last_became_active":"2021-03-15T22:17:58.013784+0000","last_became_peered":"2021-03-15T22:17:58.013784+0000","last_unstale":"2021-03-15T22:19:31.884865+0000","last_undegraded":"2021-03-15T22:19:31.884865+0000","last_fullsized":"2021-03-15T22:19:31.884865+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668564+0000","last_change":"2021-03-15T22:18:00.020025+0000","last_active":"2021-03-15T22:19:23.668564+0000","last_peered":"2021-03-15T22:19:23.668564+0000","last_clean":"2021-03-15T22:19:23.668564+0000","last_became_active":"2021-03-15T22:18:00.019686+0000","last_became_peered":"2021-03-15T22:18:00.019686+0000","last_unstale":"2021-03-15T22:19:23.668564+0000","last_undegraded":"2021-03-15T22:19:23.668564+0000","last_fullsized":"2021-03-15T22:19:23.668564+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733591+0000","last_change":"2021-03-15T22:18:59.946642+0000","last_active":"2021-03-15T22:19:23.733591+0000","last_peered":"2021-03-15T22:19:23.733591+0000","last_clean":"2021-03-15T22:19:23.733591+0000","last_became_active":"2021-03-15T22:17:53.978907+0000","last_became_peered":"2021-03-15T22:17:53.978907+0000","last_unstale":"2021-03-15T22:19:23.733591+0000","last_undegraded":"2021-03-15T22:19:23.733591+0000","last_fullsized":"2021-03-15T22:19:23.733591+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:59.946561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.946561+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"65","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672922+0000","last_change":"2021-03-15T22:17:55.993082+0000","last_active":"2021-03-15T22:19:23.672922+0000","last_peered":"2021-03-15T22:19:23.672922+0000","last_clean":"2021-03-15T22:19:23.672922+0000","last_became_active":"2021-03-15T22:17:55.992386+0000","last_became_peered":"2021-03-15T22:17:55.992386+0000","last_unstale":"2021-03-15T22:19:23.672922+0000","last_undegraded":"2021-03-15T22:19:23.672922+0000","last_fullsized":"2021-03-15T22:19:23.672922+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.667985+0000","last_change":"2021-03-15T22:19:18.197284+0000","last_active":"2021-03-15T22:19:23.667985+0000","last_peered":"2021-03-15T22:19:23.667985+0000","last_clean":"2021-03-15T22:19:23.667985+0000","last_became_active":"2021-03-15T22:17:58.007916+0000","last_became_peered":"2021-03-15T22:17:58.007916+0000","last_unstale":"2021-03-15T22:19:23.667985+0000","last_undegraded":"2021-03-15T22:19:23.667985+0000","last_fullsized":"2021-03-15T22:19:23.667985+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.197224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.197224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.732771+0000","last_change":"2021-03-15T22:19:09.989290+0000","last_active":"2021-03-15T22:19:23.732771+0000","last_peered":"2021-03-15T22:19:23.732771+0000","last_clean":"2021-03-15T22:19:23.732771+0000","last_became_active":"2021-03-15T22:18:00.016271+0000","last_became_peered":"2021-03-15T22:18:00.016271+0000","last_unstale":"2021-03-15T22:19:23.732771+0000","last_undegraded":"2021-03-15T22:19:23.732771+0000","last_fullsized":"2021-03-15T22:19:23.732771+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:09.989228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:09.989228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:29.099771+0000","last_change":"2021-03-15T22:19:29.099735+0000","last_active":"2021-03-15T22:19:29.099771+0000","last_peered":"2021-03-15T22:19:29.099771+0000","last_clean":"2021-03-15T22:19:29.099771+0000","last_became_active":"2021-03-15T22:17:58.010511+0000","last_became_peered":"2021-03-15T22:17:58.010511+0000","last_unstale":"2021-03-15T22:19:29.099771+0000","last_undegraded":"2021-03-15T22:19:29.099771+0000","last_fullsized":"2021-03-15T22:19:29.099771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.099650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.099650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733550+0000","last_change":"2021-03-15T22:17:53.974424+0000","last_active":"2021-03-15T22:19:23.733550+0000","last_peered":"2021-03-15T22:19:23.733550+0000","last_clean":"2021-03-15T22:19:23.733550+0000","last_became_active":"2021-03-15T22:17:53.974108+0000","last_became_peered":"2021-03-15T22:17:53.974108+0000","last_unstale":"2021-03-15T22:19:23.733550+0000","last_undegraded":"2021-03-15T22:19:23.733550+0000","last_fullsized":"2021-03-15T22:19:23.733550+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"78","reported_epoch":"84","state":"active+clean","last_fresh":"2021-03-15T22:19:22.700192+0000","last_change":"2021-03-15T22:19:22.700145+0000","last_active":"2021-03-15T22:19:22.700192+0000","last_peered":"2021-03-15T22:19:22.700192+0000","last_clean":"2021-03-15T22:19:22.700192+0000","last_became_active":"2021-03-15T22:17:56.288870+0000","last_became_peered":"2021-03-15T22:17:56.288870+0000","last_unstale":"2021-03-15T22:19:22.700192+0000","last_undegraded":"2021-03-15T22:19:22.700192+0000","last_fullsized":"2021-03-15T22:19:22.700192+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:19:22.700049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.700049+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"31","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842079+0000","last_change":"2021-03-15T22:18:00.017290+0000","last_active":"2021-03-15T22:19:23.842079+0000","last_peered":"2021-03-15T22:19:23.842079+0000","last_clean":"2021-03-15T22:19:23.842079+0000","last_became_active":"2021-03-15T22:18:00.016982+0000","last_became_peered":"2021-03-15T22:18:00.016982+0000","last_unstale":"2021-03-15T22:19:23.842079+0000","last_undegraded":"2021-03-15T22:19:23.842079+0000","last_fullsized":"2021-03-15T22:19:23.842079+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669630+0000","last_change":"2021-03-15T22:19:01.036104+0000","last_active":"2021-03-15T22:19:23.669630+0000","last_peered":"2021-03-15T22:19:23.669630+0000","last_clean":"2021-03-15T22:19:23.669630+0000","last_became_active":"2021-03-15T22:17:58.001399+0000","last_became_peered":"2021-03-15T22:17:58.001399+0000","last_unstale":"2021-03-15T22:19:23.669630+0000","last_undegraded":"2021-03-15T22:19:23.669630+0000","last_fullsized":"2021-03-15T22:19:23.669630+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:01.036038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:01.036038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175566+0000","last_change":"2021-03-15T22:19:18.252532+0000","last_active":"2021-03-15T22:19:24.175566+0000","last_peered":"2021-03-15T22:19:24.175566+0000","last_clean":"2021-03-15T22:19:24.175566+0000","last_became_active":"2021-03-15T22:17:53.972006+0000","last_became_peered":"2021-03-15T22:17:53.972006+0000","last_unstale":"2021-03-15T22:19:24.175566+0000","last_undegraded":"2021-03-15T22:19:24.175566+0000","last_fullsized":"2021-03-15T22:19:24.175566+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.252456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.252456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"95","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:28.582021+0000","last_change":"2021-03-15T22:19:28.581985+0000","last_active":"2021-03-15T22:19:28.582021+0000","last_peered":"2021-03-15T22:19:28.582021+0000","last_clean":"2021-03-15T22:19:28.582021+0000","last_became_active":"2021-03-15T22:19:02.093463+0000","last_became_peered":"2021-03-15T22:19:02.093463+0000","last_unstale":"2021-03-15T22:19:28.582021+0000","last_undegraded":"2021-03-15T22:19:28.582021+0000","last_fullsized":"2021-03-15T22:19:28.582021+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:19:28.581920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.581920+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669025+0000","last_change":"2021-03-15T22:19:19.908283+0000","last_active":"2021-03-15T22:19:23.669025+0000","last_peered":"2021-03-15T22:19:23.669025+0000","last_clean":"2021-03-15T22:19:23.669025+0000","last_became_active":"2021-03-15T22:18:00.020501+0000","last_became_peered":"2021-03-15T22:18:00.020501+0000","last_unstale":"2021-03-15T22:19:23.669025+0000","last_undegraded":"2021-03-15T22:19:23.669025+0000","last_fullsized":"2021-03-15T22:19:23.669025+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.908218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.908218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"44","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175273+0000","last_change":"2021-03-15T22:19:08.268784+0000","last_active":"2021-03-15T22:19:24.175273+0000","last_peered":"2021-03-15T22:19:24.175273+0000","last_clean":"2021-03-15T22:19:24.175273+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:19:24.175273+0000","last_undegraded":"2021-03-15T22:19:24.175273+0000","last_fullsized":"2021-03-15T22:19:24.175273+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:19:08.268707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672896+0000","last_change":"2021-03-15T22:17:53.978932+0000","last_active":"2021-03-15T22:19:23.672896+0000","last_peered":"2021-03-15T22:19:23.672896+0000","last_clean":"2021-03-15T22:19:23.672896+0000","last_became_active":"2021-03-15T22:17:53.978521+0000","last_became_peered":"2021-03-15T22:17:53.978521+0000","last_unstale":"2021-03-15T22:19:23.672896+0000","last_undegraded":"2021-03-15T22:19:23.672896+0000","last_fullsized":"2021-03-15T22:19:23.672896+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"101","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646127+0000","last_change":"2021-03-15T22:19:17.650024+0000","last_active":"2021-03-15T22:19:24.646127+0000","last_peered":"2021-03-15T22:19:24.646127+0000","last_clean":"2021-03-15T22:19:24.646127+0000","last_became_active":"2021-03-15T22:17:55.995319+0000","last_became_peered":"2021-03-15T22:17:55.995319+0000","last_unstale":"2021-03-15T22:19:24.646127+0000","last_undegraded":"2021-03-15T22:19:24.646127+0000","last_fullsized":"2021-03-15T22:19:24.646127+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.649963+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842338+0000","last_change":"2021-03-15T22:19:09.821008+0000","last_active":"2021-03-15T22:19:23.842338+0000","last_peered":"2021-03-15T22:19:23.842338+0000","last_clean":"2021-03-15T22:19:23.842338+0000","last_became_active":"2021-03-15T22:17:58.017434+0000","last_became_peered":"2021-03-15T22:17:58.017434+0000","last_unstale":"2021-03-15T22:19:23.842338+0000","last_undegraded":"2021-03-15T22:19:23.842338+0000","last_fullsized":"2021-03-15T22:19:23.842338+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:09.820947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:09.820947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885111+0000","last_change":"2021-03-15T22:19:31.885077+0000","last_active":"2021-03-15T22:19:31.885111+0000","last_peered":"2021-03-15T22:19:31.885111+0000","last_clean":"2021-03-15T22:19:31.885111+0000","last_became_active":"2021-03-15T22:18:00.287780+0000","last_became_peered":"2021-03-15T22:18:00.287780+0000","last_unstale":"2021-03-15T22:19:31.885111+0000","last_undegraded":"2021-03-15T22:19:31.885111+0000","last_fullsized":"2021-03-15T22:19:31.885111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:31.884965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:31.884965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646533+0000","last_change":"2021-03-15T22:19:05.537563+0000","last_active":"2021-03-15T22:19:24.646533+0000","last_peered":"2021-03-15T22:19:24.646533+0000","last_clean":"2021-03-15T22:19:24.646533+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:19:24.646533+0000","last_undegraded":"2021-03-15T22:19:24.646533+0000","last_fullsized":"2021-03-15T22:19:24.646533+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:05.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:05.537497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"46","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.142643+0000","last_change":"2021-03-15T22:19:24.142618+0000","last_active":"2021-03-15T22:19:24.142643+0000","last_peered":"2021-03-15T22:19:24.142643+0000","last_clean":"2021-03-15T22:19:24.142643+0000","last_became_active":"2021-03-15T22:17:51.631460+0000","last_became_peered":"2021-03-15T22:17:51.631460+0000","last_unstale":"2021-03-15T22:19:24.142643+0000","last_undegraded":"2021-03-15T22:19:24.142643+0000","last_fullsized":"2021-03-15T22:19:24.142643+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.142558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.142558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'15","reported_seq":"72","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.648283+0000","last_change":"2021-03-15T22:19:24.648256+0000","last_active":"2021-03-15T22:19:24.648283+0000","last_peered":"2021-03-15T22:19:24.648283+0000","last_clean":"2021-03-15T22:19:24.648283+0000","last_became_active":"2021-03-15T22:17:55.992952+0000","last_became_peered":"2021-03-15T22:17:55.992952+0000","last_unstale":"2021-03-15T22:19:24.648283+0000","last_undegraded":"2021-03-15T22:19:24.648283+0000","last_fullsized":"2021-03-15T22:19:24.648283+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:24.648200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.648200+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.670323+0000","last_change":"2021-03-15T22:19:15.249150+0000","last_active":"2021-03-15T22:19:23.670323+0000","last_peered":"2021-03-15T22:19:23.670323+0000","last_clean":"2021-03-15T22:19:23.670323+0000","last_became_active":"2021-03-15T22:17:58.014849+0000","last_became_peered":"2021-03-15T22:17:58.014849+0000","last_unstale":"2021-03-15T22:19:23.670323+0000","last_undegraded":"2021-03-15T22:19:23.670323+0000","last_fullsized":"2021-03-15T22:19:23.670323+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:15.249076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.249076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"36","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:25.184377+0000","last_change":"2021-03-15T22:19:25.184338+0000","last_active":"2021-03-15T22:19:25.184377+0000","last_peered":"2021-03-15T22:19:25.184377+0000","last_clean":"2021-03-15T22:19:25.184377+0000","last_became_active":"2021-03-15T22:18:00.019363+0000","last_became_peered":"2021-03-15T22:18:00.019363+0000","last_unstale":"2021-03-15T22:19:25.184377+0000","last_undegraded":"2021-03-15T22:19:25.184377+0000","last_fullsized":"2021-03-15T22:19:25.184377+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.184260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.184260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"46","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.844005+0000","last_change":"2021-03-15T22:19:23.843965+0000","last_active":"2021-03-15T22:19:23.844005+0000","last_peered":"2021-03-15T22:19:23.844005+0000","last_clean":"2021-03-15T22:19:23.844005+0000","last_became_active":"2021-03-15T22:17:51.044140+0000","last_became_peered":"2021-03-15T22:17:51.044140+0000","last_unstale":"2021-03-15T22:19:23.844005+0000","last_undegraded":"2021-03-15T22:19:23.844005+0000","last_fullsized":"2021-03-15T22:19:23.844005+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:23.843892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.843892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669057+0000","last_change":"2021-03-15T22:19:20.880709+0000","last_active":"2021-03-15T22:19:23.669057+0000","last_peered":"2021-03-15T22:19:23.669057+0000","last_clean":"2021-03-15T22:19:23.669057+0000","last_became_active":"2021-03-15T22:17:53.972080+0000","last_became_peered":"2021-03-15T22:17:53.972080+0000","last_unstale":"2021-03-15T22:19:23.669057+0000","last_undegraded":"2021-03-15T22:19:23.669057+0000","last_fullsized":"2021-03-15T22:19:23.669057+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:20.880638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:20.880638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"63","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.732253+0000","last_change":"2021-03-15T22:19:23.732253+0000","last_active":"2021-03-15T22:19:23.732253+0000","last_peered":"2021-03-15T22:19:23.732253+0000","last_clean":"2021-03-15T22:19:23.732253+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:19:23.732253+0000","last_undegraded":"2021-03-15T22:19:23.732253+0000","last_fullsized":"2021-03-15T22:19:23.732253+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:23.732187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.732187+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733427+0000","last_change":"2021-03-15T22:19:07.981284+0000","last_active":"2021-03-15T22:19:23.733427+0000","last_peered":"2021-03-15T22:19:23.733427+0000","last_clean":"2021-03-15T22:19:23.733427+0000","last_became_active":"2021-03-15T22:17:58.011911+0000","last_became_peered":"2021-03-15T22:17:58.011911+0000","last_unstale":"2021-03-15T22:19:23.733427+0000","last_undegraded":"2021-03-15T22:19:23.733427+0000","last_fullsized":"2021-03-15T22:19:23.733427+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.981223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:28.773845+0000","last_change":"2021-03-15T22:19:28.773808+0000","last_active":"2021-03-15T22:19:28.773845+0000","last_peered":"2021-03-15T22:19:28.773845+0000","last_clean":"2021-03-15T22:19:28.773845+0000","last_became_active":"2021-03-15T22:18:00.014336+0000","last_became_peered":"2021-03-15T22:18:00.014336+0000","last_unstale":"2021-03-15T22:19:28.773845+0000","last_undegraded":"2021-03-15T22:19:28.773845+0000","last_fullsized":"2021-03-15T22:19:28.773845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.773747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.773747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842245+0000","last_change":"2021-03-15T22:17:53.977752+0000","last_active":"2021-03-15T22:19:23.842245+0000","last_peered":"2021-03-15T22:19:23.842245+0000","last_clean":"2021-03-15T22:19:23.842245+0000","last_became_active":"2021-03-15T22:17:53.977311+0000","last_became_peered":"2021-03-15T22:17:53.977311+0000","last_unstale":"2021-03-15T22:19:23.842245+0000","last_undegraded":"2021-03-15T22:19:23.842245+0000","last_fullsized":"2021-03-15T22:19:23.842245+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"148","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:32.419432+0000","last_change":"2021-03-15T22:19:03.966872+0000","last_active":"2021-03-15T22:19:32.419432+0000","last_peered":"2021-03-15T22:19:32.419432+0000","last_clean":"2021-03-15T22:19:32.419432+0000","last_became_active":"2021-03-15T22:17:51.044139+0000","last_became_peered":"2021-03-15T22:17:51.044139+0000","last_unstale":"2021-03-15T22:19:32.419432+0000","last_undegraded":"2021-03-15T22:19:32.419432+0000","last_fullsized":"2021-03-15T22:19:32.419432+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:19:03.966803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:19:03.966803+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":100,"num_read_kb":95,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"66","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:30.589175+0000","last_change":"2021-03-15T22:19:30.589143+0000","last_active":"2021-03-15T22:19:30.589175+0000","last_peered":"2021-03-15T22:19:30.589175+0000","last_clean":"2021-03-15T22:19:30.589175+0000","last_became_active":"2021-03-15T22:17:55.999564+0000","last_became_peered":"2021-03-15T22:17:55.999564+0000","last_unstale":"2021-03-15T22:19:30.589175+0000","last_undegraded":"2021-03-15T22:19:30.589175+0000","last_fullsized":"2021-03-15T22:19:30.589175+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:30.589062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.589062+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.176080+0000","last_change":"2021-03-15T22:19:00.305911+0000","last_active":"2021-03-15T22:19:24.176080+0000","last_peered":"2021-03-15T22:19:24.176080+0000","last_clean":"2021-03-15T22:19:24.176080+0000","last_became_active":"2021-03-15T22:17:58.010798+0000","last_became_peered":"2021-03-15T22:17:58.010798+0000","last_unstale":"2021-03-15T22:19:24.176080+0000","last_undegraded":"2021-03-15T22:19:24.176080+0000","last_fullsized":"2021-03-15T22:19:24.176080+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:00.305805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:00.305805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646252+0000","last_change":"2021-03-15T22:19:15.593372+0000","last_active":"2021-03-15T22:19:24.646252+0000","last_peered":"2021-03-15T22:19:24.646252+0000","last_clean":"2021-03-15T22:19:24.646252+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:19:24.646252+0000","last_undegraded":"2021-03-15T22:19:24.646252+0000","last_fullsized":"2021-03-15T22:19:24.646252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.593307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:29.812553+0000","last_change":"2021-03-15T22:19:29.812510+0000","last_active":"2021-03-15T22:19:29.812553+0000","last_peered":"2021-03-15T22:19:29.812553+0000","last_clean":"2021-03-15T22:19:29.812553+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:19:29.812553+0000","last_undegraded":"2021-03-15T22:19:29.812553+0000","last_fullsized":"2021-03-15T22:19:29.812553+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.812446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.812446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"90","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842817+0000","last_change":"2021-03-15T22:18:56.832718+0000","last_active":"2021-03-15T22:19:23.842817+0000","last_peered":"2021-03-15T22:19:23.842817+0000","last_clean":"2021-03-15T22:19:23.842817+0000","last_became_active":"2021-03-15T22:17:56.288438+0000","last_became_peered":"2021-03-15T22:17:56.288438+0000","last_unstale":"2021-03-15T22:19:23.842817+0000","last_undegraded":"2021-03-15T22:19:23.842817+0000","last_fullsized":"2021-03-15T22:19:23.842817+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:18:56.832638+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"37","reported_epoch":"84","state":"active+clean","last_fresh":"2021-03-15T22:19:16.581620+0000","last_change":"2021-03-15T22:19:16.581584+0000","last_active":"2021-03-15T22:19:16.581620+0000","last_peered":"2021-03-15T22:19:16.581620+0000","last_clean":"2021-03-15T22:19:16.581620+0000","last_became_active":"2021-03-15T22:17:58.008731+0000","last_became_peered":"2021-03-15T22:17:58.008731+0000","last_unstale":"2021-03-15T22:19:16.581620+0000","last_undegraded":"2021-03-15T22:19:16.581620+0000","last_fullsized":"2021-03-15T22:19:16.581620+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.581514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"36","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842015+0000","last_change":"2021-03-15T22:19:19.751342+0000","last_active":"2021-03-15T22:19:23.842015+0000","last_peered":"2021-03-15T22:19:23.842015+0000","last_clean":"2021-03-15T22:19:23.842015+0000","last_became_active":"2021-03-15T22:18:00.015069+0000","last_became_peered":"2021-03-15T22:18:00.015069+0000","last_unstale":"2021-03-15T22:19:23.842015+0000","last_undegraded":"2021-03-15T22:19:23.842015+0000","last_fullsized":"2021-03-15T22:19:23.842015+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.751282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.751282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175699+0000","last_change":"2021-03-15T22:17:54.284532+0000","last_active":"2021-03-15T22:19:24.175699+0000","last_peered":"2021-03-15T22:19:24.175699+0000","last_clean":"2021-03-15T22:19:24.175699+0000","last_became_active":"2021-03-15T22:17:54.284266+0000","last_became_peered":"2021-03-15T22:17:54.284266+0000","last_unstale":"2021-03-15T22:19:24.175699+0000","last_undegraded":"2021-03-15T22:19:24.175699+0000","last_fullsized":"2021-03-15T22:19:24.175699+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"65","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.645992+0000","last_change":"2021-03-15T22:19:02.550860+0000","last_active":"2021-03-15T22:19:24.645992+0000","last_peered":"2021-03-15T22:19:24.645992+0000","last_clean":"2021-03-15T22:19:24.645992+0000","last_became_active":"2021-03-15T22:17:55.992257+0000","last_became_peered":"2021-03-15T22:17:55.992257+0000","last_unstale":"2021-03-15T22:19:24.645992+0000","last_undegraded":"2021-03-15T22:19:24.645992+0000","last_fullsized":"2021-03-15T22:19:24.645992+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"61'2","reported_seq":"58","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:27.023238+0000","last_change":"2021-03-15T22:19:25.604285+0000","last_active":"2021-03-15T22:19:27.023238+0000","last_peered":"2021-03-15T22:19:27.023238+0000","last_clean":"2021-03-15T22:19:27.023238+0000","last_became_active":"2021-03-15T22:17:58.008973+0000","last_became_peered":"2021-03-15T22:17:58.008973+0000","last_unstale":"2021-03-15T22:19:27.023238+0000","last_undegraded":"2021-03-15T22:19:27.023238+0000","last_fullsized":"2021-03-15T22:19:27.023238+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:25.604217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.604217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733294+0000","last_change":"2021-03-15T22:19:18.783378+0000","last_active":"2021-03-15T22:19:23.733294+0000","last_peered":"2021-03-15T22:19:23.733294+0000","last_clean":"2021-03-15T22:19:23.733294+0000","last_became_active":"2021-03-15T22:18:00.288441+0000","last_became_peered":"2021-03-15T22:18:00.288441+0000","last_unstale":"2021-03-15T22:19:23.733294+0000","last_undegraded":"2021-03-15T22:19:23.733294+0000","last_fullsized":"2021-03-15T22:19:23.733294+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.783297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.783297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842459+0000","last_change":"2021-03-15T22:19:13.870267+0000","last_active":"2021-03-15T22:19:23.842459+0000","last_peered":"2021-03-15T22:19:23.842459+0000","last_clean":"2021-03-15T22:19:23.842459+0000","last_became_active":"2021-03-15T22:17:53.976307+0000","last_became_peered":"2021-03-15T22:17:53.976307+0000","last_unstale":"2021-03-15T22:19:23.842459+0000","last_undegraded":"2021-03-15T22:19:23.842459+0000","last_fullsized":"2021-03-15T22:19:23.842459+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:13.870202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.870202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"74","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:26.087739+0000","last_change":"2021-03-15T22:19:26.087704+0000","last_active":"2021-03-15T22:19:26.087739+0000","last_peered":"2021-03-15T22:19:26.087739+0000","last_clean":"2021-03-15T22:19:26.087739+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:19:26.087739+0000","last_undegraded":"2021-03-15T22:19:26.087739+0000","last_fullsized":"2021-03-15T22:19:26.087739+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:19:26.087622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.087622+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"61'2","reported_seq":"54","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:27.023203+0000","last_change":"2021-03-15T22:17:58.011617+0000","last_active":"2021-03-15T22:19:27.023203+0000","last_peered":"2021-03-15T22:19:27.023203+0000","last_clean":"2021-03-15T22:19:27.023203+0000","last_became_active":"2021-03-15T22:17:58.011279+0000","last_became_peered":"2021-03-15T22:17:58.011279+0000","last_unstale":"2021-03-15T22:19:27.023203+0000","last_undegraded":"2021-03-15T22:19:27.023203+0000","last_fullsized":"2021-03-15T22:19:27.023203+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672429+0000","last_change":"2021-03-15T22:18:00.021409+0000","last_active":"2021-03-15T22:19:23.672429+0000","last_peered":"2021-03-15T22:19:23.672429+0000","last_clean":"2021-03-15T22:19:23.672429+0000","last_became_active":"2021-03-15T22:18:00.021088+0000","last_became_peered":"2021-03-15T22:18:00.021088+0000","last_unstale":"2021-03-15T22:19:23.672429+0000","last_undegraded":"2021-03-15T22:19:23.672429+0000","last_fullsized":"2021-03-15T22:19:23.672429+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668597+0000","last_change":"2021-03-15T22:18:59.667448+0000","last_active":"2021-03-15T22:19:23.668597+0000","last_peered":"2021-03-15T22:19:23.668597+0000","last_clean":"2021-03-15T22:19:23.668597+0000","last_became_active":"2021-03-15T22:17:53.970585+0000","last_became_peered":"2021-03-15T22:17:53.970585+0000","last_unstale":"2021-03-15T22:19:23.668597+0000","last_undegraded":"2021-03-15T22:19:23.668597+0000","last_fullsized":"2021-03-15T22:19:23.668597+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:59.667388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.667388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"93","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668592+0000","last_change":"2021-03-15T22:19:04.746329+0000","last_active":"2021-03-15T22:19:23.668592+0000","last_peered":"2021-03-15T22:19:23.668592+0000","last_clean":"2021-03-15T22:19:23.668592+0000","last_became_active":"2021-03-15T22:17:56.288319+0000","last_became_peered":"2021-03-15T22:17:56.288319+0000","last_unstale":"2021-03-15T22:19:23.668592+0000","last_undegraded":"2021-03-15T22:19:23.668592+0000","last_fullsized":"2021-03-15T22:19:23.668592+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:19:04.746224+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"34","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672365+0000","last_change":"2021-03-15T22:17:58.007412+0000","last_active":"2021-03-15T22:19:23.672365+0000","last_peered":"2021-03-15T22:19:23.672365+0000","last_clean":"2021-03-15T22:19:23.672365+0000","last_became_active":"2021-03-15T22:17:58.006706+0000","last_became_peered":"2021-03-15T22:17:58.006706+0000","last_unstale":"2021-03-15T22:19:23.672365+0000","last_undegraded":"2021-03-15T22:19:23.672365+0000","last_fullsized":"2021-03-15T22:19:23.672365+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"31","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175556+0000","last_change":"2021-03-15T22:18:00.019700+0000","last_active":"2021-03-15T22:19:24.175556+0000","last_peered":"2021-03-15T22:19:24.175556+0000","last_clean":"2021-03-15T22:19:24.175556+0000","last_became_active":"2021-03-15T22:18:00.019457+0000","last_became_peered":"2021-03-15T22:18:00.019457+0000","last_unstale":"2021-03-15T22:19:24.175556+0000","last_undegraded":"2021-03-15T22:19:24.175556+0000","last_fullsized":"2021-03-15T22:19:24.175556+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"61'2","reported_seq":"54","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:27.023477+0000","last_change":"2021-03-15T22:17:58.010787+0000","last_active":"2021-03-15T22:19:27.023477+0000","last_peered":"2021-03-15T22:19:27.023477+0000","last_clean":"2021-03-15T22:19:27.023477+0000","last_became_active":"2021-03-15T22:17:58.010485+0000","last_became_peered":"2021-03-15T22:17:58.010485+0000","last_unstale":"2021-03-15T22:19:27.023477+0000","last_undegraded":"2021-03-15T22:19:27.023477+0000","last_fullsized":"2021-03-15T22:19:27.023477+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.902895+0000","last_change":"2021-03-15T22:19:24.902865+0000","last_active":"2021-03-15T22:19:24.902895+0000","last_peered":"2021-03-15T22:19:24.902895+0000","last_clean":"2021-03-15T22:19:24.902895+0000","last_became_active":"2021-03-15T22:17:53.967568+0000","last_became_peered":"2021-03-15T22:17:53.967568+0000","last_unstale":"2021-03-15T22:19:24.902895+0000","last_undegraded":"2021-03-15T22:19:24.902895+0000","last_fullsized":"2021-03-15T22:19:24.902895+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.902804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.902804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.670407+0000","last_change":"2021-03-15T22:19:10.132572+0000","last_active":"2021-03-15T22:19:23.670407+0000","last_peered":"2021-03-15T22:19:23.670407+0000","last_clean":"2021-03-15T22:19:23.670407+0000","last_became_active":"2021-03-15T22:19:02.092703+0000","last_became_peered":"2021-03-15T22:19:02.092703+0000","last_unstale":"2021-03-15T22:19:23.670407+0000","last_undegraded":"2021-03-15T22:19:23.670407+0000","last_fullsized":"2021-03-15T22:19:23.670407+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:19:10.132514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:10.132514+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:25.878099+0000","last_change":"2021-03-15T22:19:25.878071+0000","last_active":"2021-03-15T22:19:25.878099+0000","last_peered":"2021-03-15T22:19:25.878099+0000","last_clean":"2021-03-15T22:19:25.878099+0000","last_became_active":"2021-03-15T22:18:00.287019+0000","last_became_peered":"2021-03-15T22:18:00.287019+0000","last_unstale":"2021-03-15T22:19:25.878099+0000","last_undegraded":"2021-03-15T22:19:25.878099+0000","last_fullsized":"2021-03-15T22:19:25.878099+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.878008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.176094+0000","last_change":"2021-03-15T22:18:58.347698+0000","last_active":"2021-03-15T22:19:24.176094+0000","last_peered":"2021-03-15T22:19:24.176094+0000","last_clean":"2021-03-15T22:19:24.176094+0000","last_became_active":"2021-03-15T22:17:58.008735+0000","last_became_peered":"2021-03-15T22:17:58.008735+0000","last_unstale":"2021-03-15T22:19:24.176094+0000","last_undegraded":"2021-03-15T22:19:24.176094+0000","last_fullsized":"2021-03-15T22:19:24.176094+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:58.347616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:18:58.347616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672557+0000","last_change":"2021-03-15T22:17:54.285942+0000","last_active":"2021-03-15T22:19:23.672557+0000","last_peered":"2021-03-15T22:19:23.672557+0000","last_clean":"2021-03-15T22:19:23.672557+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:19:23.672557+0000","last_undegraded":"2021-03-15T22:19:23.672557+0000","last_fullsized":"2021-03-15T22:19:23.672557+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"63","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.673146+0000","last_change":"2021-03-15T22:17:55.992532+0000","last_active":"2021-03-15T22:19:23.673146+0000","last_peered":"2021-03-15T22:19:23.673146+0000","last_clean":"2021-03-15T22:19:23.673146+0000","last_became_active":"2021-03-15T22:17:55.992213+0000","last_became_peered":"2021-03-15T22:17:55.992213+0000","last_unstale":"2021-03-15T22:19:23.673146+0000","last_undegraded":"2021-03-15T22:19:23.673146+0000","last_fullsized":"2021-03-15T22:19:23.673146+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.670481+0000","last_change":"2021-03-15T22:19:22.172668+0000","last_active":"2021-03-15T22:19:23.670481+0000","last_peered":"2021-03-15T22:19:23.670481+0000","last_clean":"2021-03-15T22:19:23.670481+0000","last_became_active":"2021-03-15T22:18:00.019205+0000","last_became_peered":"2021-03-15T22:18:00.019205+0000","last_unstale":"2021-03-15T22:19:23.670481+0000","last_undegraded":"2021-03-15T22:19:23.670481+0000","last_fullsized":"2021-03-15T22:19:23.670481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.172605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.647040+0000","last_change":"2021-03-15T22:19:13.514504+0000","last_active":"2021-03-15T22:19:24.647040+0000","last_peered":"2021-03-15T22:19:24.647040+0000","last_clean":"2021-03-15T22:19:24.647040+0000","last_became_active":"2021-03-15T22:17:58.013512+0000","last_became_peered":"2021-03-15T22:17:58.013512+0000","last_unstale":"2021-03-15T22:19:24.647040+0000","last_undegraded":"2021-03-15T22:19:24.647040+0000","last_fullsized":"2021-03-15T22:19:24.647040+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:13.514438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.514438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"41","reported_epoch":"84","state":"active+clean","last_fresh":"2021-03-15T22:19:18.591435+0000","last_change":"2021-03-15T22:19:18.591404+0000","last_active":"2021-03-15T22:19:18.591435+0000","last_peered":"2021-03-15T22:19:18.591435+0000","last_clean":"2021-03-15T22:19:18.591435+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:19:18.591435+0000","last_undegraded":"2021-03-15T22:19:18.591435+0000","last_fullsized":"2021-03-15T22:19:18.591435+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.591331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.591331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"65'21","reported_seq":"72","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.673084+0000","last_change":"2021-03-15T22:17:55.992521+0000","last_active":"2021-03-15T22:19:23.673084+0000","last_peered":"2021-03-15T22:19:23.673084+0000","last_clean":"2021-03-15T22:19:23.673084+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:19:23.673084+0000","last_undegraded":"2021-03-15T22:19:23.673084+0000","last_fullsized":"2021-03-15T22:19:23.673084+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.673372+0000","last_change":"2021-03-15T22:18:00.019688+0000","last_active":"2021-03-15T22:19:23.673372+0000","last_peered":"2021-03-15T22:19:23.673372+0000","last_clean":"2021-03-15T22:19:23.673372+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:19:23.673372+0000","last_undegraded":"2021-03-15T22:19:23.673372+0000","last_fullsized":"2021-03-15T22:19:23.673372+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"61'2","reported_seq":"60","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944385+0000","last_change":"2021-03-15T22:19:12.158924+0000","last_active":"2021-03-15T22:19:31.944385+0000","last_peered":"2021-03-15T22:19:31.944385+0000","last_clean":"2021-03-15T22:19:31.944385+0000","last_became_active":"2021-03-15T22:17:58.010539+0000","last_became_peered":"2021-03-15T22:17:58.010539+0000","last_unstale":"2021-03-15T22:19:31.944385+0000","last_undegraded":"2021-03-15T22:19:31.944385+0000","last_fullsized":"2021-03-15T22:19:31.944385+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:12.158857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:12.158857+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668932+0000","last_change":"2021-03-15T22:18:57.650089+0000","last_active":"2021-03-15T22:19:23.668932+0000","last_peered":"2021-03-15T22:19:23.668932+0000","last_clean":"2021-03-15T22:19:23.668932+0000","last_became_active":"2021-03-15T22:17:53.976045+0000","last_became_peered":"2021-03-15T22:17:53.976045+0000","last_unstale":"2021-03-15T22:19:23.668932+0000","last_undegraded":"2021-03-15T22:19:23.668932+0000","last_fullsized":"2021-03-15T22:19:23.668932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:57.650003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:57.650003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"57","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672725+0000","last_change":"2021-03-15T22:17:55.986249+0000","last_active":"2021-03-15T22:19:23.672725+0000","last_peered":"2021-03-15T22:19:23.672725+0000","last_clean":"2021-03-15T22:19:23.672725+0000","last_became_active":"2021-03-15T22:17:55.986062+0000","last_became_peered":"2021-03-15T22:17:55.986062+0000","last_unstale":"2021-03-15T22:19:23.672725+0000","last_undegraded":"2021-03-15T22:19:23.672725+0000","last_fullsized":"2021-03-15T22:19:23.672725+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16,"ondisk_log_size":16,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1069,"num_read_kb":574,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":545,"ondisk_log_size":545,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":100,"num_read_kb":95,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299416,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6440,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738712,"statfs":{"total":95995035648,"available":95988441088,"internally_reserved":0,"allocated":884736,"data_stored":360759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.376},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.58899999999999997}]},{"osd":1,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.51400000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.65100000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.63200000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528347,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6396,"kb_used_data":884,"kb_used_omap":0,"kb_used_meta":5504,"kb_avail":93738756,"statfs":{"total":95995035648,"available":95988486144,"internally_reserved":0,"allocated":905216,"data_stored":353920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5636096},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47699999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.69599999999999995,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.51900000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.495,"15min":0.495},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.624},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.58999999999999997}]},{"osd":3,"last update":"Mon Mar 15 22:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.72399999999999998,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.69499999999999995}]},{"osd":4,"last update":"Mon Mar 15 22:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.42399999999999999,"5min":0.42399999999999999,"15min":0.42399999999999999},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.60099999999999998,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.60599999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.42999999999999999,"15min":0.42999999999999999},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61599999999999999},"last":0.76400000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.67400000000000004},"last":0.77300000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:18:52 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.71999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7088,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":6080,"kb_avail":93738064,"statfs":{"total":95995035648,"available":95987777536,"internally_reserved":0,"allocated":1024000,"data_stored":365735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6225920},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:13 2021","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38400000000000001,"15min":0.35099999999999998},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66400000000000003,"5min":3.387,"15min":3.387},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.377},"min":{"1min":0.23699999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.71899999999999997,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.433,"15min":0.42099999999999999},"min":{"1min":0.28899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.43099999999999999,"15min":0.42499999999999999},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":2.0760000000000001,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.28199999999999997}]},{"osd":3,"last update":"Mon Mar 15 22:18:42 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.505,"15min":0.502},"min":{"1min":0.318,"5min":0.317,"15min":0.317},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.308}]},{"osd":4,"last update":"Mon Mar 15 22:19:00 2021","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.65300000000000002,"15min":0.63500000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":4.3499999999999996,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.35699999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:19:16 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.65200000000000002,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.94199999999999995,"5min":2.153,"15min":2.153},"last":0.372},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.151,"15min":1.2450000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95099999999999996,"5min":15.472,"15min":15.472},"last":0.46100000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":1.75,"5min":1.75,"15min":1.75},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.65900000000000003,"5min":0.65900000000000003,"15min":0.65900000000000003},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":1.3580000000000001,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.47399999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:18:53 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":8.8100000000000005,"5min":8.8100000000000005,"15min":8.8100000000000005},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.44400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738415,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6688,"kb_used_data":856,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738464,"statfs":{"total":95995035648,"available":95988187136,"internally_reserved":0,"allocated":876544,"data_stored":354133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:19:12 2021","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.253,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.38200000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59299999999999997,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.37}]},{"osd":2,"last update":"Mon Mar 15 22:19:29 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.39700000000000002,"15min":0.40999999999999998},"min":{"1min":0.24199999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.42899999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.39200000000000002,"15min":0.40400000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.442,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.28499999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:18:48 2021","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.432,"15min":0.442},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54700000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.442,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.53900000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:19:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73599999999999999,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60499999999999998,"15min":0.61499999999999999},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72099999999999997,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.501}]},{"osd":5,"last update":"Mon Mar 15 22:19:18 2021","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.443},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70199999999999996,"5min":0.871,"15min":0.871},"last":0.47199999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.75700000000000001,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.443,"5min":0.443,"15min":0.443},"max":{"1min":0.872,"5min":0.872,"15min":0.872},"last":0.52900000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:18:52 2021","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.48999999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6508,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738644,"statfs":{"total":95995035648,"available":95988371456,"internally_reserved":0,"allocated":823296,"data_stored":358977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.35399999999999998,"15min":0.35999999999999999},"min":{"1min":0.20799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.377,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.191,"15min":0.191},"max":{"1min":0.53600000000000003,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.38800000000000001},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55200000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.50700000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.36299999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:18:44 2021","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.501,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.309},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.55800000000000005,"5min":0.874,"15min":0.874},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Mar 15 22:18:59 2021","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.69199999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.56100000000000005,"15min":0.56899999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73399999999999999,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.45800000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:19:19 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69799999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.78000000000000003,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.77300000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56000000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149733,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6784,"kb_used_data":952,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738368,"statfs":{"total":95995035648,"available":95988088832,"internally_reserved":0,"allocated":974848,"data_stored":356713,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.36199999999999999,"15min":0.371},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.42499999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.20100000000000001},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.38700000000000001,"15min":0.39700000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.47899999999999998,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.249}]},{"osd":1,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42599999999999999,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52300000000000002,"5min":0.621,"15min":0.621},"last":0.307},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.42199999999999999,"15min":0.436},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.42699999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.27000000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.42699999999999999,"15min":0.44},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.49099999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39700000000000002,"15min":0.40500000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54100000000000004,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.23799999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:19:01 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.51400000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63800000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73899999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.56499999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:18:33 2021","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.59699999999999998},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.59299999999999997,"15min":0.59299999999999997},"min":{"1min":0.435,"5min":0.435,"15min":0.435},"max":{"1min":0.88,"5min":0.88,"15min":0.88},"last":0.57399999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.497},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.53300000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986210,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6692,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738460,"statfs":{"total":95995035648,"available":95988183040,"internally_reserved":0,"allocated":946176,"data_stored":362780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68600000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.497},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.47399999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51100000000000001,"15min":0.52100000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66000000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63400000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.626}]},{"osd":2,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.64300000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.56399999999999995},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.58899999999999997}]},{"osd":3,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.67500000000000004,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.57499999999999996,"15min":0.58699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68100000000000005,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.60999999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:19:19 2021","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.40699999999999997,"15min":0.40200000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.371}]},{"osd":6,"last update":"Mon Mar 15 22:18:31 2021","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.40999999999999998,"15min":0.40999999999999998},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55300000000000005,"5min":0.55300000000000005,"15min":0.55300000000000005},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.63800000000000001,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:18:49 2021","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.437},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.41699999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757279,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6372,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":815104,"data_stored":357746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47799999999999998,"15min":0.47899999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59299999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.755,"15min":0.755},"last":0.52800000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73399999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.502},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.42399999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.439}]},{"osd":3,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69999999999999996,"5min":0.745,"15min":0.745},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.57699999999999996}]},{"osd":4,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.376,"15min":0.36399999999999999},"min":{"1min":0.30099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.314,"5min":0.25,"15min":0.25},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.54700000000000004}]},{"osd":6,"last update":"Mon Mar 15 22:18:33 2021","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.44900000000000001,"15min":0.44900000000000001},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72899999999999998,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.55800000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.56299999999999994}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":12926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":5792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:19:37.590 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:19:37.868 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:38.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:38 smithi076 conmon[56514]: cluster 2021-03-15T22:19:36.926954+0000 osd.7 (osd.7) 2021-03-15T22:19:38.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:38 smithi076 conmon[56514]: 17 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:19:38.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:38 smithi076 conmon[56514]: cluster 2021-03-15T22:19:37.268358+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:19:38.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:38 smithi076 conmon[62156]: cluster 2021-03-15T22:19:36.926954+0000 2021-03-15T22:19:38.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:38 smithi076 conmon[62156]: osd.7 (osd.7) 17 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:19:38.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:38 smithi076 conmon[62156]: cluster 2021-03-15T22:19:37.268358+0000 osd.5 ( 2021-03-15T22:19:38.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:38 smithi076 conmon[62156]: osd.5) 19 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:19:38.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:38 smithi125 conmon[56739]: cluster 2021-03-15T22:19:36.926954+0000 osd.7 (osd.7) 17 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:19:38.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:38 smithi125 conmon[56739]: cluster 2021-03-15T22:19:37.268358+0000 2021-03-15T22:19:38.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:38 smithi125 conmon[56739]: osd.5 (osd.5) 19 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:19:39.250 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:39 smithi076 conmon[62156]: cluster 2021-03-15T22:19:37.782576+0000 osd.1 (osd.1) 20 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:19:39.250 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:39 smithi076 conmon[62156]: cluster 2021-03-15T22:19:37.905263+0000 osd.3 (osd.3) 23 : cluster [DBG] 6.b scrub ok 2021-03-15T22:19:39.250 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:39 smithi076 conmon[62156]: cluster 2021-03-15T22:19:38.156098+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:19:39.251 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:39 smithi076 conmon[62156]: cluster 2021-03-15T22:19:38.721529+0000 osd.0 (osd.0) 24 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:19:39.251 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:39 smithi076 conmon[62156]: cluster 2021-03-15T22:19:38.969408+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v245: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:39.251 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:39 smithi076 conmon[56514]: cluster 2021-03-15T22:19:37.782576+0000 osd.1 (osd.1) 20 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:19:39.251 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:39 smithi076 conmon[56514]: cluster 2021-03-15T22:19:37.905263+0000 osd.3 (osd.3) 23 : cluster [DBG] 6.b scrub ok 2021-03-15T22:19:39.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:39 smithi076 conmon[56514]: cluster 2021-03-15T22:19:38.156098+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:19:39.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:39 smithi076 conmon[56514]: cluster 2021-03-15T22:19:38.721529+0000 osd.0 (osd.0) 24 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:19:39.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:39 smithi076 conmon[56514]: cluster 2021-03-15T22:19:38.969408+0000 mgr.y (mgr.14144) 336 : cluster [DBG] pgmap v245: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:39.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:39 smithi125 conmon[56739]: cluster 2021-03-15T22:19:37.782576+0000 osd.1 (osd.1) 20 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:19:39.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:39 smithi125 conmon[56739]: cluster 2021-03-15T22:19:37 2021-03-15T22:19:39.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:39 smithi125 conmon[56739]: .905263+0000 osd.3 (osd.3) 23 : cluster [DBG] 6.b scrub ok 2021-03-15T22:19:39.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:39 smithi125 conmon[56739]: cluster 2021-03-15T22:19:38.156098+0000 osd.4 (osd.4) 14 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:19:39.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:39 smithi125 conmon[56739]: cluster 2021-03-15T22:19:38.721529+0000 osd.0 (osd. 2021-03-15T22:19:39.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:39 smithi125 conmon[56739]: 0) 24 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:19:39.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:39 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:19:39.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:39 smithi125 conmon[56739]: -03-15T22:19:38.969408+0000 mgr.y (mgr.14144) 336 2021-03-15T22:19:39.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:39 smithi125 conmon[56739]: : cluster [DBG] pgmap v245: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:39.610 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:19:39.613 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:19:39.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:19:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:19:40.296 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":245,"stamp":"2021-03-15T22:19:38.969091+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25236,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1173,"num_read_kb":673,"num_write":550,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":571,"ondisk_log_size":571,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":53132,"kb_used_data":7116,"kb_used_omap":0,"kb_used_meta":45952,"kb_avail":749908084,"statfs":{"total":767960285184,"available":767905878016,"internally_reserved":0,"allocated":7286784,"data_stored":2893020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47054848},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002753"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:28.122186+0000","last_change":"2021-03-15T22:19:28.122158+0000","last_active":"2021-03-15T22:19:28.122186+0000","last_peered":"2021-03-15T22:19:28.122186+0000","last_clean":"2021-03-15T22:19:28.122186+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:19:28.122186+0000","last_undegraded":"2021-03-15T22:19:28.122186+0000","last_fullsized":"2021-03-15T22:19:28.122186+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.122079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.122079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:37.782674+0000","last_change":"2021-03-15T22:19:37.782674+0000","last_active":"2021-03-15T22:19:37.782674+0000","last_peered":"2021-03-15T22:19:37.782674+0000","last_clean":"2021-03-15T22:19:37.782674+0000","last_became_active":"2021-03-15T22:17:58.014719+0000","last_became_peered":"2021-03-15T22:17:58.014719+0000","last_unstale":"2021-03-15T22:19:37.782674+0000","last_undegraded":"2021-03-15T22:19:37.782674+0000","last_fullsized":"2021-03-15T22:19:37.782674+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.782584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.782584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"45","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842608+0000","last_change":"2021-03-15T22:19:08.789028+0000","last_active":"2021-03-15T22:19:23.842608+0000","last_peered":"2021-03-15T22:19:23.842608+0000","last_clean":"2021-03-15T22:19:23.842608+0000","last_became_active":"2021-03-15T22:17:54.284855+0000","last_became_peered":"2021-03-15T22:17:54.284855+0000","last_unstale":"2021-03-15T22:19:23.842608+0000","last_undegraded":"2021-03-15T22:19:23.842608+0000","last_fullsized":"2021-03-15T22:19:23.842608+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:19:08.788964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:08.788964+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"69","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672254+0000","last_change":"2021-03-15T22:19:11.891528+0000","last_active":"2021-03-15T22:19:23.672254+0000","last_peered":"2021-03-15T22:19:23.672254+0000","last_clean":"2021-03-15T22:19:23.672254+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:19:23.672254+0000","last_undegraded":"2021-03-15T22:19:23.672254+0000","last_fullsized":"2021-03-15T22:19:23.672254+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:11.891465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.891465+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:26.865995+0000","last_change":"2021-03-15T22:19:26.865961+0000","last_active":"2021-03-15T22:19:26.865995+0000","last_peered":"2021-03-15T22:19:26.865995+0000","last_clean":"2021-03-15T22:19:26.865995+0000","last_became_active":"2021-03-15T22:18:00.288598+0000","last_became_peered":"2021-03-15T22:18:00.288598+0000","last_unstale":"2021-03-15T22:19:26.865995+0000","last_undegraded":"2021-03-15T22:19:26.865995+0000","last_fullsized":"2021-03-15T22:19:26.865995+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.865895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.865895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"34","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672252+0000","last_change":"2021-03-15T22:17:58.011237+0000","last_active":"2021-03-15T22:19:23.672252+0000","last_peered":"2021-03-15T22:19:23.672252+0000","last_clean":"2021-03-15T22:19:23.672252+0000","last_became_active":"2021-03-15T22:17:58.010916+0000","last_became_peered":"2021-03-15T22:17:58.010916+0000","last_unstale":"2021-03-15T22:19:23.672252+0000","last_undegraded":"2021-03-15T22:19:23.672252+0000","last_fullsized":"2021-03-15T22:19:23.672252+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669169+0000","last_change":"2021-03-15T22:19:10.851317+0000","last_active":"2021-03-15T22:19:23.669169+0000","last_peered":"2021-03-15T22:19:23.669169+0000","last_clean":"2021-03-15T22:19:23.669169+0000","last_became_active":"2021-03-15T22:17:54.283819+0000","last_became_peered":"2021-03-15T22:17:54.283819+0000","last_unstale":"2021-03-15T22:19:23.669169+0000","last_undegraded":"2021-03-15T22:19:23.669169+0000","last_fullsized":"2021-03-15T22:19:23.669169+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:10.851256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:10.851256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"71","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669284+0000","last_change":"2021-03-15T22:17:55.991923+0000","last_active":"2021-03-15T22:19:23.669284+0000","last_peered":"2021-03-15T22:19:23.669284+0000","last_clean":"2021-03-15T22:19:23.669284+0000","last_became_active":"2021-03-15T22:17:55.991597+0000","last_became_peered":"2021-03-15T22:17:55.991597+0000","last_unstale":"2021-03-15T22:19:23.669284+0000","last_undegraded":"2021-03-15T22:19:23.669284+0000","last_fullsized":"2021-03-15T22:19:23.669284+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"36","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842172+0000","last_change":"2021-03-15T22:19:11.832031+0000","last_active":"2021-03-15T22:19:23.842172+0000","last_peered":"2021-03-15T22:19:23.842172+0000","last_clean":"2021-03-15T22:19:23.842172+0000","last_became_active":"2021-03-15T22:18:00.015776+0000","last_became_peered":"2021-03-15T22:18:00.015776+0000","last_unstale":"2021-03-15T22:19:23.842172+0000","last_undegraded":"2021-03-15T22:19:23.842172+0000","last_fullsized":"2021-03-15T22:19:23.842172+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:11.831963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.831963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:30.206036+0000","last_change":"2021-03-15T22:19:30.205991+0000","last_active":"2021-03-15T22:19:30.206036+0000","last_peered":"2021-03-15T22:19:30.206036+0000","last_clean":"2021-03-15T22:19:30.206036+0000","last_became_active":"2021-03-15T22:17:58.008828+0000","last_became_peered":"2021-03-15T22:17:58.008828+0000","last_unstale":"2021-03-15T22:19:30.206036+0000","last_undegraded":"2021-03-15T22:19:30.206036+0000","last_fullsized":"2021-03-15T22:19:30.206036+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.205915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.205915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175672+0000","last_change":"2021-03-15T22:18:53.334542+0000","last_active":"2021-03-15T22:19:24.175672+0000","last_peered":"2021-03-15T22:19:24.175672+0000","last_clean":"2021-03-15T22:19:24.175672+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:19:24.175672+0000","last_undegraded":"2021-03-15T22:19:24.175672+0000","last_fullsized":"2021-03-15T22:19:24.175672+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:53.334478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:53.334478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"60","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672688+0000","last_change":"2021-03-15T22:17:55.990767+0000","last_active":"2021-03-15T22:19:23.672688+0000","last_peered":"2021-03-15T22:19:23.672688+0000","last_clean":"2021-03-15T22:19:23.672688+0000","last_became_active":"2021-03-15T22:17:55.990528+0000","last_became_peered":"2021-03-15T22:17:55.990528+0000","last_unstale":"2021-03-15T22:19:23.672688+0000","last_undegraded":"2021-03-15T22:19:23.672688+0000","last_fullsized":"2021-03-15T22:19:23.672688+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"36","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:37.268478+0000","last_change":"2021-03-15T22:19:37.268478+0000","last_active":"2021-03-15T22:19:37.268478+0000","last_peered":"2021-03-15T22:19:37.268478+0000","last_clean":"2021-03-15T22:19:37.268478+0000","last_became_active":"2021-03-15T22:18:00.019585+0000","last_became_peered":"2021-03-15T22:18:00.019585+0000","last_unstale":"2021-03-15T22:19:37.268478+0000","last_undegraded":"2021-03-15T22:19:37.268478+0000","last_fullsized":"2021-03-15T22:19:37.268478+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.268362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.268362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"34","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733091+0000","last_change":"2021-03-15T22:17:58.013492+0000","last_active":"2021-03-15T22:19:23.733091+0000","last_peered":"2021-03-15T22:19:23.733091+0000","last_clean":"2021-03-15T22:19:23.733091+0000","last_became_active":"2021-03-15T22:17:58.013236+0000","last_became_peered":"2021-03-15T22:17:58.013236+0000","last_unstale":"2021-03-15T22:19:23.733091+0000","last_undegraded":"2021-03-15T22:19:23.733091+0000","last_fullsized":"2021-03-15T22:19:23.733091+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175442+0000","last_change":"2021-03-15T22:19:06.274483+0000","last_active":"2021-03-15T22:19:24.175442+0000","last_peered":"2021-03-15T22:19:24.175442+0000","last_clean":"2021-03-15T22:19:24.175442+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:19:24.175442+0000","last_undegraded":"2021-03-15T22:19:24.175442+0000","last_fullsized":"2021-03-15T22:19:24.175442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:06.274416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.274416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'6","reported_seq":"70","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672984+0000","last_change":"2021-03-15T22:19:02.277922+0000","last_active":"2021-03-15T22:19:23.672984+0000","last_peered":"2021-03-15T22:19:23.672984+0000","last_clean":"2021-03-15T22:19:23.672984+0000","last_became_active":"2021-03-15T22:19:02.274105+0000","last_became_peered":"2021-03-15T22:19:02.274105+0000","last_unstale":"2021-03-15T22:19:23.672984+0000","last_undegraded":"2021-03-15T22:19:23.672984+0000","last_fullsized":"2021-03-15T22:19:23.672984+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2021-03-15T22:18:59.037504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.037504+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.673225+0000","last_change":"2021-03-15T22:18:00.287135+0000","last_active":"2021-03-15T22:19:23.673225+0000","last_peered":"2021-03-15T22:19:23.673225+0000","last_clean":"2021-03-15T22:19:23.673225+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:19:23.673225+0000","last_undegraded":"2021-03-15T22:19:23.673225+0000","last_fullsized":"2021-03-15T22:19:23.673225+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:35.779203+0000","last_change":"2021-03-15T22:19:35.779203+0000","last_active":"2021-03-15T22:19:35.779203+0000","last_peered":"2021-03-15T22:19:35.779203+0000","last_clean":"2021-03-15T22:19:35.779203+0000","last_became_active":"2021-03-15T22:17:58.016067+0000","last_became_peered":"2021-03-15T22:17:58.016067+0000","last_unstale":"2021-03-15T22:19:35.779203+0000","last_undegraded":"2021-03-15T22:19:35.779203+0000","last_fullsized":"2021-03-15T22:19:35.779203+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:35.779105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.779105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"45","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842757+0000","last_change":"2021-03-15T22:19:14.859504+0000","last_active":"2021-03-15T22:19:23.842757+0000","last_peered":"2021-03-15T22:19:23.842757+0000","last_clean":"2021-03-15T22:19:23.842757+0000","last_became_active":"2021-03-15T22:17:53.976875+0000","last_became_peered":"2021-03-15T22:17:53.976875+0000","last_unstale":"2021-03-15T22:19:23.842757+0000","last_undegraded":"2021-03-15T22:19:23.842757+0000","last_fullsized":"2021-03-15T22:19:23.842757+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.859443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"66","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:27.165241+0000","last_change":"2021-03-15T22:19:27.165212+0000","last_active":"2021-03-15T22:19:27.165241+0000","last_peered":"2021-03-15T22:19:27.165241+0000","last_clean":"2021-03-15T22:19:27.165241+0000","last_became_active":"2021-03-15T22:17:55.992247+0000","last_became_peered":"2021-03-15T22:17:55.992247+0000","last_unstale":"2021-03-15T22:19:27.165241+0000","last_undegraded":"2021-03-15T22:19:27.165241+0000","last_fullsized":"2021-03-15T22:19:27.165241+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:27.165119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.165119+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668832+0000","last_change":"2021-03-15T22:18:00.287310+0000","last_active":"2021-03-15T22:19:23.668832+0000","last_peered":"2021-03-15T22:19:23.668832+0000","last_clean":"2021-03-15T22:19:23.668832+0000","last_became_active":"2021-03-15T22:18:00.286974+0000","last_became_peered":"2021-03-15T22:18:00.286974+0000","last_unstale":"2021-03-15T22:19:23.668832+0000","last_undegraded":"2021-03-15T22:19:23.668832+0000","last_fullsized":"2021-03-15T22:19:23.668832+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672814+0000","last_change":"2021-03-15T22:19:21.043341+0000","last_active":"2021-03-15T22:19:23.672814+0000","last_peered":"2021-03-15T22:19:23.672814+0000","last_clean":"2021-03-15T22:19:23.672814+0000","last_became_active":"2021-03-15T22:17:58.005477+0000","last_became_peered":"2021-03-15T22:17:58.005477+0000","last_unstale":"2021-03-15T22:19:23.672814+0000","last_undegraded":"2021-03-15T22:19:23.672814+0000","last_fullsized":"2021-03-15T22:19:23.672814+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.043261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.043261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:35.186018+0000","last_change":"2021-03-15T22:19:35.186018+0000","last_active":"2021-03-15T22:19:35.186018+0000","last_peered":"2021-03-15T22:19:35.186018+0000","last_clean":"2021-03-15T22:19:35.186018+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:19:35.186018+0000","last_undegraded":"2021-03-15T22:19:35.186018+0000","last_fullsized":"2021-03-15T22:19:35.186018+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:35.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.185891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"72","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:35.934561+0000","last_change":"2021-03-15T22:19:35.934561+0000","last_active":"2021-03-15T22:19:35.934561+0000","last_peered":"2021-03-15T22:19:35.934561+0000","last_clean":"2021-03-15T22:19:35.934561+0000","last_became_active":"2021-03-15T22:17:55.995747+0000","last_became_peered":"2021-03-15T22:17:55.995747+0000","last_unstale":"2021-03-15T22:19:35.934561+0000","last_undegraded":"2021-03-15T22:19:35.934561+0000","last_fullsized":"2021-03-15T22:19:35.934561+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:35.934475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.934475+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:30.855012+0000","last_change":"2021-03-15T22:19:30.854972+0000","last_active":"2021-03-15T22:19:30.855012+0000","last_peered":"2021-03-15T22:19:30.855012+0000","last_clean":"2021-03-15T22:19:30.855012+0000","last_became_active":"2021-03-15T22:18:00.015889+0000","last_became_peered":"2021-03-15T22:18:00.015889+0000","last_unstale":"2021-03-15T22:19:30.855012+0000","last_undegraded":"2021-03-15T22:19:30.855012+0000","last_fullsized":"2021-03-15T22:19:30.855012+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.854899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.854899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"61'2","reported_seq":"59","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.886591+0000","last_change":"2021-03-15T22:19:21.656004+0000","last_active":"2021-03-15T22:19:31.886591+0000","last_peered":"2021-03-15T22:19:31.886591+0000","last_clean":"2021-03-15T22:19:31.886591+0000","last_became_active":"2021-03-15T22:17:58.013571+0000","last_became_peered":"2021-03-15T22:17:58.013571+0000","last_unstale":"2021-03-15T22:19:31.886591+0000","last_undegraded":"2021-03-15T22:19:31.886591+0000","last_fullsized":"2021-03-15T22:19:31.886591+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:21.655918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.655918+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646367+0000","last_change":"2021-03-15T22:19:16.638497+0000","last_active":"2021-03-15T22:19:24.646367+0000","last_peered":"2021-03-15T22:19:24.646367+0000","last_clean":"2021-03-15T22:19:24.646367+0000","last_became_active":"2021-03-15T22:17:53.975549+0000","last_became_peered":"2021-03-15T22:17:53.975549+0000","last_unstale":"2021-03-15T22:19:24.646367+0000","last_undegraded":"2021-03-15T22:19:24.646367+0000","last_fullsized":"2021-03-15T22:19:24.646367+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.638428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.638428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"65'16","reported_seq":"62","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842457+0000","last_change":"2021-03-15T22:17:56.288831+0000","last_active":"2021-03-15T22:19:23.842457+0000","last_peered":"2021-03-15T22:19:23.842457+0000","last_clean":"2021-03-15T22:19:23.842457+0000","last_became_active":"2021-03-15T22:17:56.288283+0000","last_became_peered":"2021-03-15T22:17:56.288283+0000","last_unstale":"2021-03-15T22:19:23.842457+0000","last_undegraded":"2021-03-15T22:19:23.842457+0000","last_fullsized":"2021-03-15T22:19:23.842457+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646797+0000","last_change":"2021-03-15T22:19:07.544919+0000","last_active":"2021-03-15T22:19:24.646797+0000","last_peered":"2021-03-15T22:19:24.646797+0000","last_clean":"2021-03-15T22:19:24.646797+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:19:24.646797+0000","last_undegraded":"2021-03-15T22:19:24.646797+0000","last_fullsized":"2021-03-15T22:19:24.646797+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.544856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.544856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842614+0000","last_change":"2021-03-15T22:19:17.752562+0000","last_active":"2021-03-15T22:19:23.842614+0000","last_peered":"2021-03-15T22:19:23.842614+0000","last_clean":"2021-03-15T22:19:23.842614+0000","last_became_active":"2021-03-15T22:17:58.009184+0000","last_became_peered":"2021-03-15T22:17:58.009184+0000","last_unstale":"2021-03-15T22:19:23.842614+0000","last_undegraded":"2021-03-15T22:19:23.842614+0000","last_fullsized":"2021-03-15T22:19:23.842614+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.752495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.752495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668285+0000","last_change":"2021-03-15T22:19:21.900042+0000","last_active":"2021-03-15T22:19:23.668285+0000","last_peered":"2021-03-15T22:19:23.668285+0000","last_clean":"2021-03-15T22:19:23.668285+0000","last_became_active":"2021-03-15T22:17:53.967947+0000","last_became_peered":"2021-03-15T22:17:53.967947+0000","last_unstale":"2021-03-15T22:19:23.668285+0000","last_undegraded":"2021-03-15T22:19:23.668285+0000","last_fullsized":"2021-03-15T22:19:23.668285+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.899969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.899969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"71","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.731411+0000","last_change":"2021-03-15T22:19:13.525956+0000","last_active":"2021-03-15T22:19:23.731411+0000","last_peered":"2021-03-15T22:19:23.731411+0000","last_clean":"2021-03-15T22:19:23.731411+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:19:23.731411+0000","last_undegraded":"2021-03-15T22:19:23.731411+0000","last_fullsized":"2021-03-15T22:19:23.731411+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:19:13.525883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.525883+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:25.759163+0000","last_change":"2021-03-15T22:19:25.759114+0000","last_active":"2021-03-15T22:19:25.759163+0000","last_peered":"2021-03-15T22:19:25.759163+0000","last_clean":"2021-03-15T22:19:25.759163+0000","last_became_active":"2021-03-15T22:18:00.015858+0000","last_became_peered":"2021-03-15T22:18:00.015858+0000","last_unstale":"2021-03-15T22:19:25.759163+0000","last_undegraded":"2021-03-15T22:19:25.759163+0000","last_fullsized":"2021-03-15T22:19:25.759163+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.759034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:28.737904+0000","last_change":"2021-03-15T22:19:28.737870+0000","last_active":"2021-03-15T22:19:28.737904+0000","last_peered":"2021-03-15T22:19:28.737904+0000","last_clean":"2021-03-15T22:19:28.737904+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:19:28.737904+0000","last_undegraded":"2021-03-15T22:19:28.737904+0000","last_fullsized":"2021-03-15T22:19:28.737904+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.737796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.737796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842625+0000","last_change":"2021-03-15T22:19:01.740592+0000","last_active":"2021-03-15T22:19:23.842625+0000","last_peered":"2021-03-15T22:19:23.842625+0000","last_clean":"2021-03-15T22:19:23.842625+0000","last_became_active":"2021-03-15T22:17:53.973507+0000","last_became_peered":"2021-03-15T22:17:53.973507+0000","last_unstale":"2021-03-15T22:19:23.842625+0000","last_undegraded":"2021-03-15T22:19:23.842625+0000","last_fullsized":"2021-03-15T22:19:23.842625+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:01.740496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:01.740496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"58","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668372+0000","last_change":"2021-03-15T22:17:55.996994+0000","last_active":"2021-03-15T22:19:23.668372+0000","last_peered":"2021-03-15T22:19:23.668372+0000","last_clean":"2021-03-15T22:19:23.668372+0000","last_became_active":"2021-03-15T22:17:55.996707+0000","last_became_peered":"2021-03-15T22:17:55.996707+0000","last_unstale":"2021-03-15T22:19:23.668372+0000","last_undegraded":"2021-03-15T22:19:23.668372+0000","last_fullsized":"2021-03-15T22:19:23.668372+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668484+0000","last_change":"2021-03-15T22:19:14.850664+0000","last_active":"2021-03-15T22:19:23.668484+0000","last_peered":"2021-03-15T22:19:23.668484+0000","last_clean":"2021-03-15T22:19:23.668484+0000","last_became_active":"2021-03-15T22:18:00.287972+0000","last_became_peered":"2021-03-15T22:18:00.287972+0000","last_unstale":"2021-03-15T22:19:23.668484+0000","last_undegraded":"2021-03-15T22:19:23.668484+0000","last_fullsized":"2021-03-15T22:19:23.668484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:14.850605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.850605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:27.800237+0000","last_change":"2021-03-15T22:19:27.800200+0000","last_active":"2021-03-15T22:19:27.800237+0000","last_peered":"2021-03-15T22:19:27.800237+0000","last_clean":"2021-03-15T22:19:27.800237+0000","last_became_active":"2021-03-15T22:17:58.013174+0000","last_became_peered":"2021-03-15T22:17:58.013174+0000","last_unstale":"2021-03-15T22:19:27.800237+0000","last_undegraded":"2021-03-15T22:19:27.800237+0000","last_fullsized":"2021-03-15T22:19:27.800237+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:27.800106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.800106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175740+0000","last_change":"2021-03-15T22:19:17.295504+0000","last_active":"2021-03-15T22:19:24.175740+0000","last_peered":"2021-03-15T22:19:24.175740+0000","last_clean":"2021-03-15T22:19:24.175740+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:19:24.175740+0000","last_undegraded":"2021-03-15T22:19:24.175740+0000","last_fullsized":"2021-03-15T22:19:24.175740+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.295412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.295412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'14","reported_seq":"61","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669240+0000","last_change":"2021-03-15T22:17:56.289858+0000","last_active":"2021-03-15T22:19:23.669240+0000","last_peered":"2021-03-15T22:19:23.669240+0000","last_clean":"2021-03-15T22:19:23.669240+0000","last_became_active":"2021-03-15T22:17:56.289456+0000","last_became_peered":"2021-03-15T22:17:56.289456+0000","last_unstale":"2021-03-15T22:19:23.669240+0000","last_undegraded":"2021-03-15T22:19:23.669240+0000","last_fullsized":"2021-03-15T22:19:23.669240+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"36","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:26.615466+0000","last_change":"2021-03-15T22:19:26.615442+0000","last_active":"2021-03-15T22:19:26.615466+0000","last_peered":"2021-03-15T22:19:26.615466+0000","last_clean":"2021-03-15T22:19:26.615466+0000","last_became_active":"2021-03-15T22:18:00.014112+0000","last_became_peered":"2021-03-15T22:18:00.014112+0000","last_unstale":"2021-03-15T22:19:26.615466+0000","last_undegraded":"2021-03-15T22:19:26.615466+0000","last_fullsized":"2021-03-15T22:19:26.615466+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.615380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.615380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668729+0000","last_change":"2021-03-15T22:19:03.699264+0000","last_active":"2021-03-15T22:19:23.668729+0000","last_peered":"2021-03-15T22:19:23.668729+0000","last_clean":"2021-03-15T22:19:23.668729+0000","last_became_active":"2021-03-15T22:17:58.009365+0000","last_became_peered":"2021-03-15T22:17:58.009365+0000","last_unstale":"2021-03-15T22:19:23.668729+0000","last_undegraded":"2021-03-15T22:19:23.668729+0000","last_fullsized":"2021-03-15T22:19:23.668729+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_clean_scrub_stamp":"2021-03-15T22:19:03.699198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733442+0000","last_change":"2021-03-15T22:17:53.980472+0000","last_active":"2021-03-15T22:19:23.733442+0000","last_peered":"2021-03-15T22:19:23.733442+0000","last_clean":"2021-03-15T22:19:23.733442+0000","last_became_active":"2021-03-15T22:17:53.979947+0000","last_became_peered":"2021-03-15T22:17:53.979947+0000","last_unstale":"2021-03-15T22:19:23.733442+0000","last_undegraded":"2021-03-15T22:19:23.733442+0000","last_fullsized":"2021-03-15T22:19:23.733442+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"64","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.645843+0000","last_change":"2021-03-15T22:17:55.993221+0000","last_active":"2021-03-15T22:19:24.645843+0000","last_peered":"2021-03-15T22:19:24.645843+0000","last_clean":"2021-03-15T22:19:24.645843+0000","last_became_active":"2021-03-15T22:17:55.991537+0000","last_became_peered":"2021-03-15T22:17:55.991537+0000","last_unstale":"2021-03-15T22:19:24.645843+0000","last_undegraded":"2021-03-15T22:19:24.645843+0000","last_fullsized":"2021-03-15T22:19:24.645843+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668832+0000","last_change":"2021-03-15T22:18:00.018406+0000","last_active":"2021-03-15T22:19:23.668832+0000","last_peered":"2021-03-15T22:19:23.668832+0000","last_clean":"2021-03-15T22:19:23.668832+0000","last_became_active":"2021-03-15T22:18:00.018117+0000","last_became_peered":"2021-03-15T22:18:00.018117+0000","last_unstale":"2021-03-15T22:19:23.668832+0000","last_undegraded":"2021-03-15T22:19:23.668832+0000","last_fullsized":"2021-03-15T22:19:23.668832+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:32.759869+0000","last_change":"2021-03-15T22:19:32.759869+0000","last_active":"2021-03-15T22:19:32.759869+0000","last_peered":"2021-03-15T22:19:32.759869+0000","last_clean":"2021-03-15T22:19:32.759869+0000","last_became_active":"2021-03-15T22:17:58.014621+0000","last_became_peered":"2021-03-15T22:17:58.014621+0000","last_unstale":"2021-03-15T22:19:32.759869+0000","last_undegraded":"2021-03-15T22:19:32.759869+0000","last_fullsized":"2021-03-15T22:19:32.759869+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:32.759804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:32.759804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672933+0000","last_change":"2021-03-15T22:17:53.975528+0000","last_active":"2021-03-15T22:19:23.672933+0000","last_peered":"2021-03-15T22:19:23.672933+0000","last_clean":"2021-03-15T22:19:23.672933+0000","last_became_active":"2021-03-15T22:17:53.975292+0000","last_became_peered":"2021-03-15T22:17:53.975292+0000","last_unstale":"2021-03-15T22:19:23.672933+0000","last_undegraded":"2021-03-15T22:19:23.672933+0000","last_fullsized":"2021-03-15T22:19:23.672933+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"63","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672761+0000","last_change":"2021-03-15T22:17:55.992828+0000","last_active":"2021-03-15T22:19:23.672761+0000","last_peered":"2021-03-15T22:19:23.672761+0000","last_clean":"2021-03-15T22:19:23.672761+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:19:23.672761+0000","last_undegraded":"2021-03-15T22:19:23.672761+0000","last_fullsized":"2021-03-15T22:19:23.672761+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"31","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842408+0000","last_change":"2021-03-15T22:18:00.015339+0000","last_active":"2021-03-15T22:19:23.842408+0000","last_peered":"2021-03-15T22:19:23.842408+0000","last_clean":"2021-03-15T22:19:23.842408+0000","last_became_active":"2021-03-15T22:18:00.015034+0000","last_became_peered":"2021-03-15T22:18:00.015034+0000","last_unstale":"2021-03-15T22:19:23.842408+0000","last_undegraded":"2021-03-15T22:19:23.842408+0000","last_fullsized":"2021-03-15T22:19:23.842408+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"61'2","reported_seq":"55","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:32.024438+0000","last_change":"2021-03-15T22:17:58.011190+0000","last_active":"2021-03-15T22:19:32.024438+0000","last_peered":"2021-03-15T22:19:32.024438+0000","last_clean":"2021-03-15T22:19:32.024438+0000","last_became_active":"2021-03-15T22:17:58.010903+0000","last_became_peered":"2021-03-15T22:17:58.010903+0000","last_unstale":"2021-03-15T22:19:32.024438+0000","last_undegraded":"2021-03-15T22:19:32.024438+0000","last_fullsized":"2021-03-15T22:19:32.024438+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733105+0000","last_change":"2021-03-15T22:17:54.286237+0000","last_active":"2021-03-15T22:19:23.733105+0000","last_peered":"2021-03-15T22:19:23.733105+0000","last_clean":"2021-03-15T22:19:23.733105+0000","last_became_active":"2021-03-15T22:17:54.285851+0000","last_became_peered":"2021-03-15T22:17:54.285851+0000","last_unstale":"2021-03-15T22:19:23.733105+0000","last_undegraded":"2021-03-15T22:19:23.733105+0000","last_fullsized":"2021-03-15T22:19:23.733105+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"64","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668900+0000","last_change":"2021-03-15T22:19:07.874527+0000","last_active":"2021-03-15T22:19:23.668900+0000","last_peered":"2021-03-15T22:19:23.668900+0000","last_clean":"2021-03-15T22:19:23.668900+0000","last_became_active":"2021-03-15T22:17:56.288102+0000","last_became_peered":"2021-03-15T22:17:56.288102+0000","last_unstale":"2021-03-15T22:19:23.668900+0000","last_undegraded":"2021-03-15T22:19:23.668900+0000","last_fullsized":"2021-03-15T22:19:23.668900+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:19:07.874462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.874462+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.673452+0000","last_change":"2021-03-15T22:18:00.015238+0000","last_active":"2021-03-15T22:19:23.673452+0000","last_peered":"2021-03-15T22:19:23.673452+0000","last_clean":"2021-03-15T22:19:23.673452+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:19:23.673452+0000","last_undegraded":"2021-03-15T22:19:23.673452+0000","last_fullsized":"2021-03-15T22:19:23.673452+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"61'2","reported_seq":"60","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.997315+0000","last_change":"2021-03-15T22:19:15.850282+0000","last_active":"2021-03-15T22:19:30.997315+0000","last_peered":"2021-03-15T22:19:30.997315+0000","last_clean":"2021-03-15T22:19:30.997315+0000","last_became_active":"2021-03-15T22:17:58.003748+0000","last_became_peered":"2021-03-15T22:17:58.003748+0000","last_unstale":"2021-03-15T22:19:30.997315+0000","last_undegraded":"2021-03-15T22:19:30.997315+0000","last_fullsized":"2021-03-15T22:19:30.997315+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:15.850217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.850217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842221+0000","last_change":"2021-03-15T22:19:07.795140+0000","last_active":"2021-03-15T22:19:23.842221+0000","last_peered":"2021-03-15T22:19:23.842221+0000","last_clean":"2021-03-15T22:19:23.842221+0000","last_became_active":"2021-03-15T22:17:53.968736+0000","last_became_peered":"2021-03-15T22:17:53.968736+0000","last_unstale":"2021-03-15T22:19:23.842221+0000","last_undegraded":"2021-03-15T22:19:23.842221+0000","last_fullsized":"2021-03-15T22:19:23.842221+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.795072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.795072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"64","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:33.261320+0000","last_change":"2021-03-15T22:19:33.261320+0000","last_active":"2021-03-15T22:19:33.261320+0000","last_peered":"2021-03-15T22:19:33.261320+0000","last_clean":"2021-03-15T22:19:33.261320+0000","last_became_active":"2021-03-15T22:17:56.290563+0000","last_became_peered":"2021-03-15T22:17:56.290563+0000","last_unstale":"2021-03-15T22:19:33.261320+0000","last_undegraded":"2021-03-15T22:19:33.261320+0000","last_fullsized":"2021-03-15T22:19:33.261320+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:19:33.261213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:33.261213+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.670156+0000","last_change":"2021-03-15T22:19:16.224939+0000","last_active":"2021-03-15T22:19:23.670156+0000","last_peered":"2021-03-15T22:19:23.670156+0000","last_clean":"2021-03-15T22:19:23.670156+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:19:23.670156+0000","last_undegraded":"2021-03-15T22:19:23.670156+0000","last_fullsized":"2021-03-15T22:19:23.670156+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.224872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:32.880446+0000","last_change":"2021-03-15T22:19:32.880446+0000","last_active":"2021-03-15T22:19:32.880446+0000","last_peered":"2021-03-15T22:19:32.880446+0000","last_clean":"2021-03-15T22:19:32.880446+0000","last_became_active":"2021-03-15T22:17:58.001340+0000","last_became_peered":"2021-03-15T22:17:58.001340+0000","last_unstale":"2021-03-15T22:19:32.880446+0000","last_undegraded":"2021-03-15T22:19:32.880446+0000","last_fullsized":"2021-03-15T22:19:32.880446+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:32.880362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:32.880362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"40","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.732886+0000","last_change":"2021-03-15T22:19:13.870789+0000","last_active":"2021-03-15T22:19:23.732886+0000","last_peered":"2021-03-15T22:19:23.732886+0000","last_clean":"2021-03-15T22:19:23.732886+0000","last_became_active":"2021-03-15T22:17:53.976777+0000","last_became_peered":"2021-03-15T22:17:53.976777+0000","last_unstale":"2021-03-15T22:19:23.732886+0000","last_undegraded":"2021-03-15T22:19:23.732886+0000","last_fullsized":"2021-03-15T22:19:23.732886+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"70","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842532+0000","last_change":"2021-03-15T22:19:06.778692+0000","last_active":"2021-03-15T22:19:23.842532+0000","last_peered":"2021-03-15T22:19:23.842532+0000","last_clean":"2021-03-15T22:19:23.842532+0000","last_became_active":"2021-03-15T22:17:56.288678+0000","last_became_peered":"2021-03-15T22:17:56.288678+0000","last_unstale":"2021-03-15T22:19:23.842532+0000","last_undegraded":"2021-03-15T22:19:23.842532+0000","last_fullsized":"2021-03-15T22:19:23.842532+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:06.778622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.778622+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733386+0000","last_change":"2021-03-15T22:19:17.797392+0000","last_active":"2021-03-15T22:19:23.733386+0000","last_peered":"2021-03-15T22:19:23.733386+0000","last_clean":"2021-03-15T22:19:23.733386+0000","last_became_active":"2021-03-15T22:18:00.288462+0000","last_became_peered":"2021-03-15T22:18:00.288462+0000","last_unstale":"2021-03-15T22:19:23.733386+0000","last_undegraded":"2021-03-15T22:19:23.733386+0000","last_fullsized":"2021-03-15T22:19:23.733386+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.797291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.797291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:29.179916+0000","last_change":"2021-03-15T22:19:29.179877+0000","last_active":"2021-03-15T22:19:29.179916+0000","last_peered":"2021-03-15T22:19:29.179916+0000","last_clean":"2021-03-15T22:19:29.179916+0000","last_became_active":"2021-03-15T22:17:58.015682+0000","last_became_peered":"2021-03-15T22:17:58.015682+0000","last_unstale":"2021-03-15T22:19:29.179916+0000","last_undegraded":"2021-03-15T22:19:29.179916+0000","last_fullsized":"2021-03-15T22:19:29.179916+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.179800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.730680+0000","last_change":"2021-03-15T22:19:05.557588+0000","last_active":"2021-03-15T22:19:23.730680+0000","last_peered":"2021-03-15T22:19:23.730680+0000","last_clean":"2021-03-15T22:19:23.730680+0000","last_became_active":"2021-03-15T22:17:53.977470+0000","last_became_peered":"2021-03-15T22:17:53.977470+0000","last_unstale":"2021-03-15T22:19:23.730680+0000","last_undegraded":"2021-03-15T22:19:23.730680+0000","last_fullsized":"2021-03-15T22:19:23.730680+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:05.557534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:05.557534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'7","reported_seq":"47","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668886+0000","last_change":"2021-03-15T22:17:56.289796+0000","last_active":"2021-03-15T22:19:23.668886+0000","last_peered":"2021-03-15T22:19:23.668886+0000","last_clean":"2021-03-15T22:19:23.668886+0000","last_became_active":"2021-03-15T22:17:56.289454+0000","last_became_peered":"2021-03-15T22:17:56.289454+0000","last_unstale":"2021-03-15T22:19:23.668886+0000","last_undegraded":"2021-03-15T22:19:23.668886+0000","last_fullsized":"2021-03-15T22:19:23.668886+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"36","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175506+0000","last_change":"2021-03-15T22:19:11.318218+0000","last_active":"2021-03-15T22:19:24.175506+0000","last_peered":"2021-03-15T22:19:24.175506+0000","last_clean":"2021-03-15T22:19:24.175506+0000","last_became_active":"2021-03-15T22:18:00.286699+0000","last_became_peered":"2021-03-15T22:18:00.286699+0000","last_unstale":"2021-03-15T22:19:24.175506+0000","last_undegraded":"2021-03-15T22:19:24.175506+0000","last_fullsized":"2021-03-15T22:19:24.175506+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:11.318132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.318132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"46","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733721+0000","last_change":"2021-03-15T22:19:06.942939+0000","last_active":"2021-03-15T22:19:23.733721+0000","last_peered":"2021-03-15T22:19:23.733721+0000","last_clean":"2021-03-15T22:19:23.733721+0000","last_became_active":"2021-03-15T22:17:53.979246+0000","last_became_peered":"2021-03-15T22:17:53.979246+0000","last_unstale":"2021-03-15T22:19:23.733721+0000","last_undegraded":"2021-03-15T22:19:23.733721+0000","last_fullsized":"2021-03-15T22:19:23.733721+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:19:06.942870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.942870+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"66","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175012+0000","last_change":"2021-03-15T22:17:56.289996+0000","last_active":"2021-03-15T22:19:24.175012+0000","last_peered":"2021-03-15T22:19:24.175012+0000","last_clean":"2021-03-15T22:19:24.175012+0000","last_became_active":"2021-03-15T22:17:56.289684+0000","last_became_peered":"2021-03-15T22:17:56.289684+0000","last_unstale":"2021-03-15T22:19:24.175012+0000","last_undegraded":"2021-03-15T22:19:24.175012+0000","last_fullsized":"2021-03-15T22:19:24.175012+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"61'2","reported_seq":"55","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884865+0000","last_change":"2021-03-15T22:17:58.015052+0000","last_active":"2021-03-15T22:19:31.884865+0000","last_peered":"2021-03-15T22:19:31.884865+0000","last_clean":"2021-03-15T22:19:31.884865+0000","last_became_active":"2021-03-15T22:17:58.013784+0000","last_became_peered":"2021-03-15T22:17:58.013784+0000","last_unstale":"2021-03-15T22:19:31.884865+0000","last_undegraded":"2021-03-15T22:19:31.884865+0000","last_fullsized":"2021-03-15T22:19:31.884865+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668564+0000","last_change":"2021-03-15T22:18:00.020025+0000","last_active":"2021-03-15T22:19:23.668564+0000","last_peered":"2021-03-15T22:19:23.668564+0000","last_clean":"2021-03-15T22:19:23.668564+0000","last_became_active":"2021-03-15T22:18:00.019686+0000","last_became_peered":"2021-03-15T22:18:00.019686+0000","last_unstale":"2021-03-15T22:19:23.668564+0000","last_undegraded":"2021-03-15T22:19:23.668564+0000","last_fullsized":"2021-03-15T22:19:23.668564+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733591+0000","last_change":"2021-03-15T22:18:59.946642+0000","last_active":"2021-03-15T22:19:23.733591+0000","last_peered":"2021-03-15T22:19:23.733591+0000","last_clean":"2021-03-15T22:19:23.733591+0000","last_became_active":"2021-03-15T22:17:53.978907+0000","last_became_peered":"2021-03-15T22:17:53.978907+0000","last_unstale":"2021-03-15T22:19:23.733591+0000","last_undegraded":"2021-03-15T22:19:23.733591+0000","last_fullsized":"2021-03-15T22:19:23.733591+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:59.946561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.946561+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"65","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672922+0000","last_change":"2021-03-15T22:17:55.993082+0000","last_active":"2021-03-15T22:19:23.672922+0000","last_peered":"2021-03-15T22:19:23.672922+0000","last_clean":"2021-03-15T22:19:23.672922+0000","last_became_active":"2021-03-15T22:17:55.992386+0000","last_became_peered":"2021-03-15T22:17:55.992386+0000","last_unstale":"2021-03-15T22:19:23.672922+0000","last_undegraded":"2021-03-15T22:19:23.672922+0000","last_fullsized":"2021-03-15T22:19:23.672922+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.667985+0000","last_change":"2021-03-15T22:19:18.197284+0000","last_active":"2021-03-15T22:19:23.667985+0000","last_peered":"2021-03-15T22:19:23.667985+0000","last_clean":"2021-03-15T22:19:23.667985+0000","last_became_active":"2021-03-15T22:17:58.007916+0000","last_became_peered":"2021-03-15T22:17:58.007916+0000","last_unstale":"2021-03-15T22:19:23.667985+0000","last_undegraded":"2021-03-15T22:19:23.667985+0000","last_fullsized":"2021-03-15T22:19:23.667985+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.197224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.197224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.732771+0000","last_change":"2021-03-15T22:19:09.989290+0000","last_active":"2021-03-15T22:19:23.732771+0000","last_peered":"2021-03-15T22:19:23.732771+0000","last_clean":"2021-03-15T22:19:23.732771+0000","last_became_active":"2021-03-15T22:18:00.016271+0000","last_became_peered":"2021-03-15T22:18:00.016271+0000","last_unstale":"2021-03-15T22:19:23.732771+0000","last_undegraded":"2021-03-15T22:19:23.732771+0000","last_fullsized":"2021-03-15T22:19:23.732771+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:09.989228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:09.989228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:29.099771+0000","last_change":"2021-03-15T22:19:29.099735+0000","last_active":"2021-03-15T22:19:29.099771+0000","last_peered":"2021-03-15T22:19:29.099771+0000","last_clean":"2021-03-15T22:19:29.099771+0000","last_became_active":"2021-03-15T22:17:58.010511+0000","last_became_peered":"2021-03-15T22:17:58.010511+0000","last_unstale":"2021-03-15T22:19:29.099771+0000","last_undegraded":"2021-03-15T22:19:29.099771+0000","last_fullsized":"2021-03-15T22:19:29.099771+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.099650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.099650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733550+0000","last_change":"2021-03-15T22:17:53.974424+0000","last_active":"2021-03-15T22:19:23.733550+0000","last_peered":"2021-03-15T22:19:23.733550+0000","last_clean":"2021-03-15T22:19:23.733550+0000","last_became_active":"2021-03-15T22:17:53.974108+0000","last_became_peered":"2021-03-15T22:17:53.974108+0000","last_unstale":"2021-03-15T22:19:23.733550+0000","last_undegraded":"2021-03-15T22:19:23.733550+0000","last_fullsized":"2021-03-15T22:19:23.733550+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"79","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.731630+0000","last_change":"2021-03-15T22:19:22.700145+0000","last_active":"2021-03-15T22:19:23.731630+0000","last_peered":"2021-03-15T22:19:23.731630+0000","last_clean":"2021-03-15T22:19:23.731630+0000","last_became_active":"2021-03-15T22:17:56.288870+0000","last_became_peered":"2021-03-15T22:17:56.288870+0000","last_unstale":"2021-03-15T22:19:23.731630+0000","last_undegraded":"2021-03-15T22:19:23.731630+0000","last_fullsized":"2021-03-15T22:19:23.731630+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:19:22.700049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.700049+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"31","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842079+0000","last_change":"2021-03-15T22:18:00.017290+0000","last_active":"2021-03-15T22:19:23.842079+0000","last_peered":"2021-03-15T22:19:23.842079+0000","last_clean":"2021-03-15T22:19:23.842079+0000","last_became_active":"2021-03-15T22:18:00.016982+0000","last_became_peered":"2021-03-15T22:18:00.016982+0000","last_unstale":"2021-03-15T22:19:23.842079+0000","last_undegraded":"2021-03-15T22:19:23.842079+0000","last_fullsized":"2021-03-15T22:19:23.842079+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669630+0000","last_change":"2021-03-15T22:19:01.036104+0000","last_active":"2021-03-15T22:19:23.669630+0000","last_peered":"2021-03-15T22:19:23.669630+0000","last_clean":"2021-03-15T22:19:23.669630+0000","last_became_active":"2021-03-15T22:17:58.001399+0000","last_became_peered":"2021-03-15T22:17:58.001399+0000","last_unstale":"2021-03-15T22:19:23.669630+0000","last_undegraded":"2021-03-15T22:19:23.669630+0000","last_fullsized":"2021-03-15T22:19:23.669630+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:01.036038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:01.036038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175566+0000","last_change":"2021-03-15T22:19:18.252532+0000","last_active":"2021-03-15T22:19:24.175566+0000","last_peered":"2021-03-15T22:19:24.175566+0000","last_clean":"2021-03-15T22:19:24.175566+0000","last_became_active":"2021-03-15T22:17:53.972006+0000","last_became_peered":"2021-03-15T22:17:53.972006+0000","last_unstale":"2021-03-15T22:19:24.175566+0000","last_undegraded":"2021-03-15T22:19:24.175566+0000","last_fullsized":"2021-03-15T22:19:24.175566+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.252456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.252456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"95","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:28.582021+0000","last_change":"2021-03-15T22:19:28.581985+0000","last_active":"2021-03-15T22:19:28.582021+0000","last_peered":"2021-03-15T22:19:28.582021+0000","last_clean":"2021-03-15T22:19:28.582021+0000","last_became_active":"2021-03-15T22:19:02.093463+0000","last_became_peered":"2021-03-15T22:19:02.093463+0000","last_unstale":"2021-03-15T22:19:28.582021+0000","last_undegraded":"2021-03-15T22:19:28.582021+0000","last_fullsized":"2021-03-15T22:19:28.582021+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:19:28.581920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.581920+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669025+0000","last_change":"2021-03-15T22:19:19.908283+0000","last_active":"2021-03-15T22:19:23.669025+0000","last_peered":"2021-03-15T22:19:23.669025+0000","last_clean":"2021-03-15T22:19:23.669025+0000","last_became_active":"2021-03-15T22:18:00.020501+0000","last_became_peered":"2021-03-15T22:18:00.020501+0000","last_unstale":"2021-03-15T22:19:23.669025+0000","last_undegraded":"2021-03-15T22:19:23.669025+0000","last_fullsized":"2021-03-15T22:19:23.669025+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.908218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.908218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"44","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175273+0000","last_change":"2021-03-15T22:19:08.268784+0000","last_active":"2021-03-15T22:19:24.175273+0000","last_peered":"2021-03-15T22:19:24.175273+0000","last_clean":"2021-03-15T22:19:24.175273+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:19:24.175273+0000","last_undegraded":"2021-03-15T22:19:24.175273+0000","last_fullsized":"2021-03-15T22:19:24.175273+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:19:08.268707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672896+0000","last_change":"2021-03-15T22:17:53.978932+0000","last_active":"2021-03-15T22:19:23.672896+0000","last_peered":"2021-03-15T22:19:23.672896+0000","last_clean":"2021-03-15T22:19:23.672896+0000","last_became_active":"2021-03-15T22:17:53.978521+0000","last_became_peered":"2021-03-15T22:17:53.978521+0000","last_unstale":"2021-03-15T22:19:23.672896+0000","last_undegraded":"2021-03-15T22:19:23.672896+0000","last_fullsized":"2021-03-15T22:19:23.672896+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"101","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646127+0000","last_change":"2021-03-15T22:19:17.650024+0000","last_active":"2021-03-15T22:19:24.646127+0000","last_peered":"2021-03-15T22:19:24.646127+0000","last_clean":"2021-03-15T22:19:24.646127+0000","last_became_active":"2021-03-15T22:17:55.995319+0000","last_became_peered":"2021-03-15T22:17:55.995319+0000","last_unstale":"2021-03-15T22:19:24.646127+0000","last_undegraded":"2021-03-15T22:19:24.646127+0000","last_fullsized":"2021-03-15T22:19:24.646127+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.649963+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842338+0000","last_change":"2021-03-15T22:19:09.821008+0000","last_active":"2021-03-15T22:19:23.842338+0000","last_peered":"2021-03-15T22:19:23.842338+0000","last_clean":"2021-03-15T22:19:23.842338+0000","last_became_active":"2021-03-15T22:17:58.017434+0000","last_became_peered":"2021-03-15T22:17:58.017434+0000","last_unstale":"2021-03-15T22:19:23.842338+0000","last_undegraded":"2021-03-15T22:19:23.842338+0000","last_fullsized":"2021-03-15T22:19:23.842338+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:09.820947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:09.820947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885111+0000","last_change":"2021-03-15T22:19:31.885077+0000","last_active":"2021-03-15T22:19:31.885111+0000","last_peered":"2021-03-15T22:19:31.885111+0000","last_clean":"2021-03-15T22:19:31.885111+0000","last_became_active":"2021-03-15T22:18:00.287780+0000","last_became_peered":"2021-03-15T22:18:00.287780+0000","last_unstale":"2021-03-15T22:19:31.885111+0000","last_undegraded":"2021-03-15T22:19:31.885111+0000","last_fullsized":"2021-03-15T22:19:31.885111+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:31.884965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:31.884965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646533+0000","last_change":"2021-03-15T22:19:05.537563+0000","last_active":"2021-03-15T22:19:24.646533+0000","last_peered":"2021-03-15T22:19:24.646533+0000","last_clean":"2021-03-15T22:19:24.646533+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:19:24.646533+0000","last_undegraded":"2021-03-15T22:19:24.646533+0000","last_fullsized":"2021-03-15T22:19:24.646533+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:05.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:05.537497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"46","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.142643+0000","last_change":"2021-03-15T22:19:24.142618+0000","last_active":"2021-03-15T22:19:24.142643+0000","last_peered":"2021-03-15T22:19:24.142643+0000","last_clean":"2021-03-15T22:19:24.142643+0000","last_became_active":"2021-03-15T22:17:51.631460+0000","last_became_peered":"2021-03-15T22:17:51.631460+0000","last_unstale":"2021-03-15T22:19:24.142643+0000","last_undegraded":"2021-03-15T22:19:24.142643+0000","last_fullsized":"2021-03-15T22:19:24.142643+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.142558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.142558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'15","reported_seq":"72","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.648283+0000","last_change":"2021-03-15T22:19:24.648256+0000","last_active":"2021-03-15T22:19:24.648283+0000","last_peered":"2021-03-15T22:19:24.648283+0000","last_clean":"2021-03-15T22:19:24.648283+0000","last_became_active":"2021-03-15T22:17:55.992952+0000","last_became_peered":"2021-03-15T22:17:55.992952+0000","last_unstale":"2021-03-15T22:19:24.648283+0000","last_undegraded":"2021-03-15T22:19:24.648283+0000","last_fullsized":"2021-03-15T22:19:24.648283+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:24.648200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.648200+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.670323+0000","last_change":"2021-03-15T22:19:15.249150+0000","last_active":"2021-03-15T22:19:23.670323+0000","last_peered":"2021-03-15T22:19:23.670323+0000","last_clean":"2021-03-15T22:19:23.670323+0000","last_became_active":"2021-03-15T22:17:58.014849+0000","last_became_peered":"2021-03-15T22:17:58.014849+0000","last_unstale":"2021-03-15T22:19:23.670323+0000","last_undegraded":"2021-03-15T22:19:23.670323+0000","last_fullsized":"2021-03-15T22:19:23.670323+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:15.249076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.249076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"36","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:25.184377+0000","last_change":"2021-03-15T22:19:25.184338+0000","last_active":"2021-03-15T22:19:25.184377+0000","last_peered":"2021-03-15T22:19:25.184377+0000","last_clean":"2021-03-15T22:19:25.184377+0000","last_became_active":"2021-03-15T22:18:00.019363+0000","last_became_peered":"2021-03-15T22:18:00.019363+0000","last_unstale":"2021-03-15T22:19:25.184377+0000","last_undegraded":"2021-03-15T22:19:25.184377+0000","last_fullsized":"2021-03-15T22:19:25.184377+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.184260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.184260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"46","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.844005+0000","last_change":"2021-03-15T22:19:23.843965+0000","last_active":"2021-03-15T22:19:23.844005+0000","last_peered":"2021-03-15T22:19:23.844005+0000","last_clean":"2021-03-15T22:19:23.844005+0000","last_became_active":"2021-03-15T22:17:51.044140+0000","last_became_peered":"2021-03-15T22:17:51.044140+0000","last_unstale":"2021-03-15T22:19:23.844005+0000","last_undegraded":"2021-03-15T22:19:23.844005+0000","last_fullsized":"2021-03-15T22:19:23.844005+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:23.843892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.843892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.669057+0000","last_change":"2021-03-15T22:19:20.880709+0000","last_active":"2021-03-15T22:19:23.669057+0000","last_peered":"2021-03-15T22:19:23.669057+0000","last_clean":"2021-03-15T22:19:23.669057+0000","last_became_active":"2021-03-15T22:17:53.972080+0000","last_became_peered":"2021-03-15T22:17:53.972080+0000","last_unstale":"2021-03-15T22:19:23.669057+0000","last_undegraded":"2021-03-15T22:19:23.669057+0000","last_fullsized":"2021-03-15T22:19:23.669057+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:20.880638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:20.880638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"64","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.732287+0000","last_change":"2021-03-15T22:19:23.732253+0000","last_active":"2021-03-15T22:19:23.732287+0000","last_peered":"2021-03-15T22:19:23.732287+0000","last_clean":"2021-03-15T22:19:23.732287+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:19:23.732287+0000","last_undegraded":"2021-03-15T22:19:23.732287+0000","last_fullsized":"2021-03-15T22:19:23.732287+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:23.732187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.732187+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733427+0000","last_change":"2021-03-15T22:19:07.981284+0000","last_active":"2021-03-15T22:19:23.733427+0000","last_peered":"2021-03-15T22:19:23.733427+0000","last_clean":"2021-03-15T22:19:23.733427+0000","last_became_active":"2021-03-15T22:17:58.011911+0000","last_became_peered":"2021-03-15T22:17:58.011911+0000","last_unstale":"2021-03-15T22:19:23.733427+0000","last_undegraded":"2021-03-15T22:19:23.733427+0000","last_fullsized":"2021-03-15T22:19:23.733427+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.981223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:28.773845+0000","last_change":"2021-03-15T22:19:28.773808+0000","last_active":"2021-03-15T22:19:28.773845+0000","last_peered":"2021-03-15T22:19:28.773845+0000","last_clean":"2021-03-15T22:19:28.773845+0000","last_became_active":"2021-03-15T22:18:00.014336+0000","last_became_peered":"2021-03-15T22:18:00.014336+0000","last_unstale":"2021-03-15T22:19:28.773845+0000","last_undegraded":"2021-03-15T22:19:28.773845+0000","last_fullsized":"2021-03-15T22:19:28.773845+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.773747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.773747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842245+0000","last_change":"2021-03-15T22:17:53.977752+0000","last_active":"2021-03-15T22:19:23.842245+0000","last_peered":"2021-03-15T22:19:23.842245+0000","last_clean":"2021-03-15T22:19:23.842245+0000","last_became_active":"2021-03-15T22:17:53.977311+0000","last_became_peered":"2021-03-15T22:17:53.977311+0000","last_unstale":"2021-03-15T22:19:23.842245+0000","last_undegraded":"2021-03-15T22:19:23.842245+0000","last_fullsized":"2021-03-15T22:19:23.842245+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"148","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:32.419432+0000","last_change":"2021-03-15T22:19:03.966872+0000","last_active":"2021-03-15T22:19:32.419432+0000","last_peered":"2021-03-15T22:19:32.419432+0000","last_clean":"2021-03-15T22:19:32.419432+0000","last_became_active":"2021-03-15T22:17:51.044139+0000","last_became_peered":"2021-03-15T22:17:51.044139+0000","last_unstale":"2021-03-15T22:19:32.419432+0000","last_undegraded":"2021-03-15T22:19:32.419432+0000","last_fullsized":"2021-03-15T22:19:32.419432+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:19:03.966803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:19:03.966803+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":100,"num_read_kb":95,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"66","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:30.589175+0000","last_change":"2021-03-15T22:19:30.589143+0000","last_active":"2021-03-15T22:19:30.589175+0000","last_peered":"2021-03-15T22:19:30.589175+0000","last_clean":"2021-03-15T22:19:30.589175+0000","last_became_active":"2021-03-15T22:17:55.999564+0000","last_became_peered":"2021-03-15T22:17:55.999564+0000","last_unstale":"2021-03-15T22:19:30.589175+0000","last_undegraded":"2021-03-15T22:19:30.589175+0000","last_fullsized":"2021-03-15T22:19:30.589175+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:30.589062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.589062+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.176080+0000","last_change":"2021-03-15T22:19:00.305911+0000","last_active":"2021-03-15T22:19:24.176080+0000","last_peered":"2021-03-15T22:19:24.176080+0000","last_clean":"2021-03-15T22:19:24.176080+0000","last_became_active":"2021-03-15T22:17:58.010798+0000","last_became_peered":"2021-03-15T22:17:58.010798+0000","last_unstale":"2021-03-15T22:19:24.176080+0000","last_undegraded":"2021-03-15T22:19:24.176080+0000","last_fullsized":"2021-03-15T22:19:24.176080+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:00.305805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:00.305805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.646252+0000","last_change":"2021-03-15T22:19:15.593372+0000","last_active":"2021-03-15T22:19:24.646252+0000","last_peered":"2021-03-15T22:19:24.646252+0000","last_clean":"2021-03-15T22:19:24.646252+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:19:24.646252+0000","last_undegraded":"2021-03-15T22:19:24.646252+0000","last_fullsized":"2021-03-15T22:19:24.646252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.593307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:29.812553+0000","last_change":"2021-03-15T22:19:29.812510+0000","last_active":"2021-03-15T22:19:29.812553+0000","last_peered":"2021-03-15T22:19:29.812553+0000","last_clean":"2021-03-15T22:19:29.812553+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:19:29.812553+0000","last_undegraded":"2021-03-15T22:19:29.812553+0000","last_fullsized":"2021-03-15T22:19:29.812553+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.812446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.812446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"90","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842817+0000","last_change":"2021-03-15T22:18:56.832718+0000","last_active":"2021-03-15T22:19:23.842817+0000","last_peered":"2021-03-15T22:19:23.842817+0000","last_clean":"2021-03-15T22:19:23.842817+0000","last_became_active":"2021-03-15T22:17:56.288438+0000","last_became_peered":"2021-03-15T22:17:56.288438+0000","last_unstale":"2021-03-15T22:19:23.842817+0000","last_undegraded":"2021-03-15T22:19:23.842817+0000","last_fullsized":"2021-03-15T22:19:23.842817+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:18:56.832638+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.731678+0000","last_change":"2021-03-15T22:19:16.581584+0000","last_active":"2021-03-15T22:19:23.731678+0000","last_peered":"2021-03-15T22:19:23.731678+0000","last_clean":"2021-03-15T22:19:23.731678+0000","last_became_active":"2021-03-15T22:17:58.008731+0000","last_became_peered":"2021-03-15T22:17:58.008731+0000","last_unstale":"2021-03-15T22:19:23.731678+0000","last_undegraded":"2021-03-15T22:19:23.731678+0000","last_fullsized":"2021-03-15T22:19:23.731678+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.581514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"36","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842015+0000","last_change":"2021-03-15T22:19:19.751342+0000","last_active":"2021-03-15T22:19:23.842015+0000","last_peered":"2021-03-15T22:19:23.842015+0000","last_clean":"2021-03-15T22:19:23.842015+0000","last_became_active":"2021-03-15T22:18:00.015069+0000","last_became_peered":"2021-03-15T22:18:00.015069+0000","last_unstale":"2021-03-15T22:19:23.842015+0000","last_undegraded":"2021-03-15T22:19:23.842015+0000","last_fullsized":"2021-03-15T22:19:23.842015+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.751282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.751282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175699+0000","last_change":"2021-03-15T22:17:54.284532+0000","last_active":"2021-03-15T22:19:24.175699+0000","last_peered":"2021-03-15T22:19:24.175699+0000","last_clean":"2021-03-15T22:19:24.175699+0000","last_became_active":"2021-03-15T22:17:54.284266+0000","last_became_peered":"2021-03-15T22:17:54.284266+0000","last_unstale":"2021-03-15T22:19:24.175699+0000","last_undegraded":"2021-03-15T22:19:24.175699+0000","last_fullsized":"2021-03-15T22:19:24.175699+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"65","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.645992+0000","last_change":"2021-03-15T22:19:02.550860+0000","last_active":"2021-03-15T22:19:24.645992+0000","last_peered":"2021-03-15T22:19:24.645992+0000","last_clean":"2021-03-15T22:19:24.645992+0000","last_became_active":"2021-03-15T22:17:55.992257+0000","last_became_peered":"2021-03-15T22:17:55.992257+0000","last_unstale":"2021-03-15T22:19:24.645992+0000","last_undegraded":"2021-03-15T22:19:24.645992+0000","last_fullsized":"2021-03-15T22:19:24.645992+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"61'2","reported_seq":"59","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995893+0000","last_change":"2021-03-15T22:19:25.604285+0000","last_active":"2021-03-15T22:19:30.995893+0000","last_peered":"2021-03-15T22:19:30.995893+0000","last_clean":"2021-03-15T22:19:30.995893+0000","last_became_active":"2021-03-15T22:17:58.008973+0000","last_became_peered":"2021-03-15T22:17:58.008973+0000","last_unstale":"2021-03-15T22:19:30.995893+0000","last_undegraded":"2021-03-15T22:19:30.995893+0000","last_fullsized":"2021-03-15T22:19:30.995893+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:25.604217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.604217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.733294+0000","last_change":"2021-03-15T22:19:18.783378+0000","last_active":"2021-03-15T22:19:23.733294+0000","last_peered":"2021-03-15T22:19:23.733294+0000","last_clean":"2021-03-15T22:19:23.733294+0000","last_became_active":"2021-03-15T22:18:00.288441+0000","last_became_peered":"2021-03-15T22:18:00.288441+0000","last_unstale":"2021-03-15T22:19:23.733294+0000","last_undegraded":"2021-03-15T22:19:23.733294+0000","last_fullsized":"2021-03-15T22:19:23.733294+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.783297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.783297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.842459+0000","last_change":"2021-03-15T22:19:13.870267+0000","last_active":"2021-03-15T22:19:23.842459+0000","last_peered":"2021-03-15T22:19:23.842459+0000","last_clean":"2021-03-15T22:19:23.842459+0000","last_became_active":"2021-03-15T22:17:53.976307+0000","last_became_peered":"2021-03-15T22:17:53.976307+0000","last_unstale":"2021-03-15T22:19:23.842459+0000","last_undegraded":"2021-03-15T22:19:23.842459+0000","last_fullsized":"2021-03-15T22:19:23.842459+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:13.870202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.870202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"74","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:26.087739+0000","last_change":"2021-03-15T22:19:26.087704+0000","last_active":"2021-03-15T22:19:26.087739+0000","last_peered":"2021-03-15T22:19:26.087739+0000","last_clean":"2021-03-15T22:19:26.087739+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:19:26.087739+0000","last_undegraded":"2021-03-15T22:19:26.087739+0000","last_fullsized":"2021-03-15T22:19:26.087739+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:19:26.087622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.087622+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"61'2","reported_seq":"55","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996668+0000","last_change":"2021-03-15T22:17:58.011617+0000","last_active":"2021-03-15T22:19:30.996668+0000","last_peered":"2021-03-15T22:19:30.996668+0000","last_clean":"2021-03-15T22:19:30.996668+0000","last_became_active":"2021-03-15T22:17:58.011279+0000","last_became_peered":"2021-03-15T22:17:58.011279+0000","last_unstale":"2021-03-15T22:19:30.996668+0000","last_undegraded":"2021-03-15T22:19:30.996668+0000","last_fullsized":"2021-03-15T22:19:30.996668+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:34.179513+0000","last_change":"2021-03-15T22:19:34.179513+0000","last_active":"2021-03-15T22:19:34.179513+0000","last_peered":"2021-03-15T22:19:34.179513+0000","last_clean":"2021-03-15T22:19:34.179513+0000","last_became_active":"2021-03-15T22:18:00.021088+0000","last_became_peered":"2021-03-15T22:18:00.021088+0000","last_unstale":"2021-03-15T22:19:34.179513+0000","last_undegraded":"2021-03-15T22:19:34.179513+0000","last_fullsized":"2021-03-15T22:19:34.179513+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:34.179433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:34.179433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668597+0000","last_change":"2021-03-15T22:18:59.667448+0000","last_active":"2021-03-15T22:19:23.668597+0000","last_peered":"2021-03-15T22:19:23.668597+0000","last_clean":"2021-03-15T22:19:23.668597+0000","last_became_active":"2021-03-15T22:17:53.970585+0000","last_became_peered":"2021-03-15T22:17:53.970585+0000","last_unstale":"2021-03-15T22:19:23.668597+0000","last_undegraded":"2021-03-15T22:19:23.668597+0000","last_fullsized":"2021-03-15T22:19:23.668597+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:59.667388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.667388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"93","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668592+0000","last_change":"2021-03-15T22:19:04.746329+0000","last_active":"2021-03-15T22:19:23.668592+0000","last_peered":"2021-03-15T22:19:23.668592+0000","last_clean":"2021-03-15T22:19:23.668592+0000","last_became_active":"2021-03-15T22:17:56.288319+0000","last_became_peered":"2021-03-15T22:17:56.288319+0000","last_unstale":"2021-03-15T22:19:23.668592+0000","last_undegraded":"2021-03-15T22:19:23.668592+0000","last_fullsized":"2021-03-15T22:19:23.668592+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:19:04.746224+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"34","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672365+0000","last_change":"2021-03-15T22:17:58.007412+0000","last_active":"2021-03-15T22:19:23.672365+0000","last_peered":"2021-03-15T22:19:23.672365+0000","last_clean":"2021-03-15T22:19:23.672365+0000","last_became_active":"2021-03-15T22:17:58.006706+0000","last_became_peered":"2021-03-15T22:17:58.006706+0000","last_unstale":"2021-03-15T22:19:23.672365+0000","last_undegraded":"2021-03-15T22:19:23.672365+0000","last_fullsized":"2021-03-15T22:19:23.672365+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"31","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.175556+0000","last_change":"2021-03-15T22:18:00.019700+0000","last_active":"2021-03-15T22:19:24.175556+0000","last_peered":"2021-03-15T22:19:24.175556+0000","last_clean":"2021-03-15T22:19:24.175556+0000","last_became_active":"2021-03-15T22:18:00.019457+0000","last_became_peered":"2021-03-15T22:18:00.019457+0000","last_unstale":"2021-03-15T22:19:24.175556+0000","last_undegraded":"2021-03-15T22:19:24.175556+0000","last_fullsized":"2021-03-15T22:19:24.175556+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"61'2","reported_seq":"55","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995388+0000","last_change":"2021-03-15T22:17:58.010787+0000","last_active":"2021-03-15T22:19:30.995388+0000","last_peered":"2021-03-15T22:19:30.995388+0000","last_clean":"2021-03-15T22:19:30.995388+0000","last_became_active":"2021-03-15T22:17:58.010485+0000","last_became_peered":"2021-03-15T22:17:58.010485+0000","last_unstale":"2021-03-15T22:19:30.995388+0000","last_undegraded":"2021-03-15T22:19:30.995388+0000","last_fullsized":"2021-03-15T22:19:30.995388+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.902895+0000","last_change":"2021-03-15T22:19:24.902865+0000","last_active":"2021-03-15T22:19:24.902895+0000","last_peered":"2021-03-15T22:19:24.902895+0000","last_clean":"2021-03-15T22:19:24.902895+0000","last_became_active":"2021-03-15T22:17:53.967568+0000","last_became_peered":"2021-03-15T22:17:53.967568+0000","last_unstale":"2021-03-15T22:19:24.902895+0000","last_undegraded":"2021-03-15T22:19:24.902895+0000","last_fullsized":"2021-03-15T22:19:24.902895+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.902804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.902804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.670407+0000","last_change":"2021-03-15T22:19:10.132572+0000","last_active":"2021-03-15T22:19:23.670407+0000","last_peered":"2021-03-15T22:19:23.670407+0000","last_clean":"2021-03-15T22:19:23.670407+0000","last_became_active":"2021-03-15T22:19:02.092703+0000","last_became_peered":"2021-03-15T22:19:02.092703+0000","last_unstale":"2021-03-15T22:19:23.670407+0000","last_undegraded":"2021-03-15T22:19:23.670407+0000","last_fullsized":"2021-03-15T22:19:23.670407+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:19:10.132514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:10.132514+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:25.878099+0000","last_change":"2021-03-15T22:19:25.878071+0000","last_active":"2021-03-15T22:19:25.878099+0000","last_peered":"2021-03-15T22:19:25.878099+0000","last_clean":"2021-03-15T22:19:25.878099+0000","last_became_active":"2021-03-15T22:18:00.287019+0000","last_became_peered":"2021-03-15T22:18:00.287019+0000","last_unstale":"2021-03-15T22:19:25.878099+0000","last_undegraded":"2021-03-15T22:19:25.878099+0000","last_fullsized":"2021-03-15T22:19:25.878099+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.878008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.176094+0000","last_change":"2021-03-15T22:18:58.347698+0000","last_active":"2021-03-15T22:19:24.176094+0000","last_peered":"2021-03-15T22:19:24.176094+0000","last_clean":"2021-03-15T22:19:24.176094+0000","last_became_active":"2021-03-15T22:17:58.008735+0000","last_became_peered":"2021-03-15T22:17:58.008735+0000","last_unstale":"2021-03-15T22:19:24.176094+0000","last_undegraded":"2021-03-15T22:19:24.176094+0000","last_fullsized":"2021-03-15T22:19:24.176094+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:58.347616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:18:58.347616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"38","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672557+0000","last_change":"2021-03-15T22:17:54.285942+0000","last_active":"2021-03-15T22:19:23.672557+0000","last_peered":"2021-03-15T22:19:23.672557+0000","last_clean":"2021-03-15T22:19:23.672557+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:19:23.672557+0000","last_undegraded":"2021-03-15T22:19:23.672557+0000","last_fullsized":"2021-03-15T22:19:23.672557+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"63","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.673146+0000","last_change":"2021-03-15T22:17:55.992532+0000","last_active":"2021-03-15T22:19:23.673146+0000","last_peered":"2021-03-15T22:19:23.673146+0000","last_clean":"2021-03-15T22:19:23.673146+0000","last_became_active":"2021-03-15T22:17:55.992213+0000","last_became_peered":"2021-03-15T22:17:55.992213+0000","last_unstale":"2021-03-15T22:19:23.673146+0000","last_undegraded":"2021-03-15T22:19:23.673146+0000","last_fullsized":"2021-03-15T22:19:23.673146+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"37","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.670481+0000","last_change":"2021-03-15T22:19:22.172668+0000","last_active":"2021-03-15T22:19:23.670481+0000","last_peered":"2021-03-15T22:19:23.670481+0000","last_clean":"2021-03-15T22:19:23.670481+0000","last_became_active":"2021-03-15T22:18:00.019205+0000","last_became_peered":"2021-03-15T22:18:00.019205+0000","last_unstale":"2021-03-15T22:19:23.670481+0000","last_undegraded":"2021-03-15T22:19:23.670481+0000","last_fullsized":"2021-03-15T22:19:23.670481+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.172605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"39","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:24.647040+0000","last_change":"2021-03-15T22:19:13.514504+0000","last_active":"2021-03-15T22:19:24.647040+0000","last_peered":"2021-03-15T22:19:24.647040+0000","last_clean":"2021-03-15T22:19:24.647040+0000","last_became_active":"2021-03-15T22:17:58.013512+0000","last_became_peered":"2021-03-15T22:17:58.013512+0000","last_unstale":"2021-03-15T22:19:24.647040+0000","last_undegraded":"2021-03-15T22:19:24.647040+0000","last_fullsized":"2021-03-15T22:19:24.647040+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:13.514438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.514438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"42","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.730813+0000","last_change":"2021-03-15T22:19:18.591404+0000","last_active":"2021-03-15T22:19:23.730813+0000","last_peered":"2021-03-15T22:19:23.730813+0000","last_clean":"2021-03-15T22:19:23.730813+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:19:23.730813+0000","last_undegraded":"2021-03-15T22:19:23.730813+0000","last_fullsized":"2021-03-15T22:19:23.730813+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.591331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.591331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"65'21","reported_seq":"72","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.673084+0000","last_change":"2021-03-15T22:17:55.992521+0000","last_active":"2021-03-15T22:19:23.673084+0000","last_peered":"2021-03-15T22:19:23.673084+0000","last_clean":"2021-03-15T22:19:23.673084+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:19:23.673084+0000","last_undegraded":"2021-03-15T22:19:23.673084+0000","last_fullsized":"2021-03-15T22:19:23.673084+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"32","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.673372+0000","last_change":"2021-03-15T22:18:00.019688+0000","last_active":"2021-03-15T22:19:23.673372+0000","last_peered":"2021-03-15T22:19:23.673372+0000","last_clean":"2021-03-15T22:19:23.673372+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:19:23.673372+0000","last_undegraded":"2021-03-15T22:19:23.673372+0000","last_fullsized":"2021-03-15T22:19:23.673372+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"61'2","reported_seq":"60","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944385+0000","last_change":"2021-03-15T22:19:12.158924+0000","last_active":"2021-03-15T22:19:31.944385+0000","last_peered":"2021-03-15T22:19:31.944385+0000","last_clean":"2021-03-15T22:19:31.944385+0000","last_became_active":"2021-03-15T22:17:58.010539+0000","last_became_peered":"2021-03-15T22:17:58.010539+0000","last_unstale":"2021-03-15T22:19:31.944385+0000","last_undegraded":"2021-03-15T22:19:31.944385+0000","last_fullsized":"2021-03-15T22:19:31.944385+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:12.158857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:12.158857+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"43","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.668932+0000","last_change":"2021-03-15T22:18:57.650089+0000","last_active":"2021-03-15T22:19:23.668932+0000","last_peered":"2021-03-15T22:19:23.668932+0000","last_clean":"2021-03-15T22:19:23.668932+0000","last_became_active":"2021-03-15T22:17:53.976045+0000","last_became_peered":"2021-03-15T22:17:53.976045+0000","last_unstale":"2021-03-15T22:19:23.668932+0000","last_undegraded":"2021-03-15T22:19:23.668932+0000","last_fullsized":"2021-03-15T22:19:23.668932+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:57.650003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:57.650003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"57","reported_epoch":"85","state":"active+clean","last_fresh":"2021-03-15T22:19:23.672725+0000","last_change":"2021-03-15T22:17:55.986249+0000","last_active":"2021-03-15T22:19:23.672725+0000","last_peered":"2021-03-15T22:19:23.672725+0000","last_clean":"2021-03-15T22:19:23.672725+0000","last_became_active":"2021-03-15T22:17:55.986062+0000","last_became_peered":"2021-03-15T22:17:55.986062+0000","last_unstale":"2021-03-15T22:19:23.672725+0000","last_undegraded":"2021-03-15T22:19:23.672725+0000","last_fullsized":"2021-03-15T22:19:23.672725+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16,"ondisk_log_size":16,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1069,"num_read_kb":574,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":545,"ondisk_log_size":545,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":100,"num_read_kb":95,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299416,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6440,"kb_used_data":864,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738712,"statfs":{"total":95995035648,"available":95988441088,"internally_reserved":0,"allocated":884736,"data_stored":360759,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.376},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.58899999999999997}]},{"osd":1,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.55400000000000005}]},{"osd":2,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.51400000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.65100000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.61699999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.63200000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528348,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6472,"kb_used_data":896,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738680,"statfs":{"total":95995035648,"available":95988408320,"internally_reserved":0,"allocated":917504,"data_stored":361339,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48899999999999999,"15min":0.49299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63200000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.61899999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67500000000000004,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.61199999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.36099999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.48199999999999998,"15min":0.47799999999999998},"min":{"1min":0.40400000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.629,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.63400000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.501,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.68600000000000005,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.501,"5min":0.502,"15min":0.502},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.624,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.624,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69499999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.56000000000000005}]},{"osd":4,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.39300000000000002},"min":{"1min":0.30199999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.433,"15min":0.42699999999999999},"min":{"1min":0.32100000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.47399999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.441,"15min":0.433},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.498,"5min":0.46500000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.42899999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:18:52 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.66600000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574894,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7088,"kb_used_data":1000,"kb_used_omap":0,"kb_used_meta":6080,"kb_avail":93738064,"statfs":{"total":95995035648,"available":95987777536,"internally_reserved":0,"allocated":1024000,"data_stored":365735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6225920},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:13 2021","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38400000000000001,"15min":0.35099999999999998},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66400000000000003,"5min":3.387,"15min":3.387},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.377},"min":{"1min":0.23699999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.71899999999999997,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.39300000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:19:36 2021","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.42199999999999999,"15min":0.41699999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.78700000000000003,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41699999999999998,"15min":0.42099999999999999},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64400000000000002,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.30199999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:18:42 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.315},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.505,"15min":0.502},"min":{"1min":0.318,"5min":0.317,"15min":0.317},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.26400000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:19:00 2021","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.65300000000000002,"15min":0.63500000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":4.3499999999999996,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.48999999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:19:16 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.65200000000000002,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.94199999999999995,"5min":2.153,"15min":2.153},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.151,"15min":1.2450000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95099999999999996,"5min":15.472,"15min":15.472},"last":0.432}]},{"osd":6,"last update":"Mon Mar 15 22:19:36 2021","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.67700000000000005,"15min":0.69399999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.879,"5min":1.75,"15min":1.75},"last":0.5},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.64100000000000001,"15min":0.65300000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.87,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.48599999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:18:53 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":8.8100000000000005,"5min":8.8100000000000005,"15min":8.8100000000000005},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.51800000000000002}]}]},{"osd":0,"up_from":8,"seq":34359738416,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6764,"kb_used_data":868,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738388,"statfs":{"total":95995035648,"available":95988109312,"internally_reserved":0,"allocated":888832,"data_stored":361552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:19:12 2021","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.253,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.38200000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59299999999999997,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.35499999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:19:29 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.39700000000000002,"15min":0.40999999999999998},"min":{"1min":0.24199999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.42899999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.31900000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.39200000000000002,"15min":0.40400000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.442,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.26500000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:18:48 2021","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.432,"15min":0.442},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54700000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.442,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.53900000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.38}]},{"osd":4,"last update":"Mon Mar 15 22:19:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73599999999999999,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60499999999999998,"15min":0.61499999999999999},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72099999999999997,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.68700000000000006}]},{"osd":5,"last update":"Mon Mar 15 22:19:18 2021","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70199999999999996,"5min":0.871,"15min":0.871},"last":0.49099999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:19:31 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.754}]},{"osd":7,"last update":"Mon Mar 15 22:18:52 2021","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.66100000000000003}]}]},{"osd":2,"up_from":18,"seq":77309411370,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6508,"kb_used_data":804,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738644,"statfs":{"total":95995035648,"available":95988371456,"internally_reserved":0,"allocated":823296,"data_stored":358977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.35399999999999998,"15min":0.35999999999999999},"min":{"1min":0.20799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.377,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.191,"15min":0.191},"max":{"1min":0.53600000000000003,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.29199999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.38800000000000001},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55200000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.50700000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.36299999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:18:44 2021","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.501,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.309},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.55800000000000005,"5min":0.874,"15min":0.874},"last":0.34200000000000003}]},{"osd":4,"last update":"Mon Mar 15 22:18:59 2021","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.69199999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.56100000000000005,"15min":0.56899999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73399999999999999,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.45800000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:19:19 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69799999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.78000000000000003,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.52000000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.77300000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.56000000000000005}]}]},{"osd":3,"up_from":26,"seq":111669149734,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6796,"kb_used_data":964,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738356,"statfs":{"total":95995035648,"available":95988076544,"internally_reserved":0,"allocated":987136,"data_stored":364132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.36199999999999999,"15min":0.371},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.42499999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.38700000000000001,"15min":0.39700000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.47899999999999998,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.29899999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42599999999999999,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52300000000000002,"5min":0.621,"15min":0.621},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.42199999999999999,"15min":0.436},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.42699999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.318}]},{"osd":2,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.42699999999999999,"15min":0.44},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.49099999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39700000000000002,"15min":0.40500000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54100000000000004,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.40999999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:19:01 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.88600000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63800000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73899999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.752}]},{"osd":6,"last update":"Mon Mar 15 22:19:33 2021","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.871},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58499999999999996,"15min":0.58999999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.86699999999999999,"5min":0.88,"15min":0.88},"last":0.86699999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.71899999999999997}]}]},{"osd":4,"up_from":31,"seq":133143986211,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6692,"kb_used_data":924,"kb_used_omap":0,"kb_used_meta":5760,"kb_avail":93738460,"statfs":{"total":95995035648,"available":95988183040,"internally_reserved":0,"allocated":946176,"data_stored":362780,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5898240},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68600000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.497},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.47799999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51100000000000001,"15min":0.52100000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66000000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63400000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.67200000000000004}]},{"osd":2,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.64300000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.37}]},{"osd":3,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.67500000000000004,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.57499999999999996,"15min":0.58699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68100000000000005,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.45700000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:19:19 2021","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.436,"5min":0.40699999999999997,"15min":0.40200000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.42799999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.59099999999999997}]},{"osd":7,"last update":"Mon Mar 15 22:18:49 2021","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.56299999999999994}]}]},{"osd":5,"up_from":38,"seq":163208757280,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6372,"kb_used_data":796,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738780,"statfs":{"total":95995035648,"available":95988510720,"internally_reserved":0,"allocated":815104,"data_stored":357746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47799999999999998,"15min":0.47899999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59299999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.503},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.755,"15min":0.755},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73399999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48699999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69999999999999996,"5min":0.745,"15min":0.745},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.60399999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.376,"15min":0.36399999999999999},"min":{"1min":0.30099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.314,"5min":0.25,"15min":0.25},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.53300000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:19:35 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.44900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64000000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.40400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70199999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.55700000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.58299999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":12926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":5792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:19:40.304 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-03-15T22:19:40.304 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-03-15T22:19:40.305 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2021-03-15T22:19:40.305 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph health --format=json 2021-03-15T22:19:40.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[56514]: cluster 2021-03-15T22:19:38.767407+0000 osd.1 (osd.1) 21 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:19:40.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[56514]: 2021-03-15T22:19:40.330 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[56514]: cluster 2021-03-15T22:19:39.105062+0000 osd.3 (osd.3) 24 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:40.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[56514]: cluster 2021-03-15T22:19:39.105148+0000 osd.3 (osd.3) 25 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:40.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[56514]: cluster 2021-03-15T22:19:39.224193+0000 osd.5 (osd.5) 20 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:19:40.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[56514]: audit 2021-03-15T22:19:39.378879+0000 mgr.y (mgr.14144) 337 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:40.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[56514]: audit 2021-03-15T22:19:39.594944+0000 mgr.y (mgr.14144) 338 : audit [DBG] from='client.14637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:40.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[56514]: cluster 2021-03-15T22:19:39.681735+0000 osd.0 (osd.0) 25 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:19:40.332 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: cluster 2021-03-15T22:19:38.767407 2021-03-15T22:19:40.332 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 21 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:19:40.332 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: cluster 2021-03-15T22:19:39.105062+0000 osd.3 (osd.3) 24 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:40.333 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: cluster 2021-03-15T22:19:39.105148+0000 2021-03-15T22:19:40.333 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: osd.3 (osd.3) 25 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:40.333 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: cluster 2021-03-15T22:19: 2021-03-15T22:19:40.333 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: 39.224193+0000 osd.5 (osd.5) 20 2021-03-15T22:19:40.334 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: : cluster [DBG] 5.15 scrub ok 2021-03-15T22:19:40.334 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: audit 2021 2021-03-15T22:19:40.334 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: -03-15T22:19:39.378879+0000 mgr.y 2021-03-15T22:19:40.334 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: (mgr.14144) 337 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:40.334 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: audit 2021-03-15T22:19:39.594944+0000 mgr.y (mgr 2021-03-15T22:19:40.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: .14144) 338 : audit [DBG] from='client.14637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:40.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: cluster 2021-03-15T22:19: 2021-03-15T22:19:40.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: 39.681735+0000 osd.0 (osd.0) 25 : cluster [DBG] 2021-03-15T22:19:40.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:40 smithi076 conmon[62156]: 6.9 scrub ok 2021-03-15T22:19:40.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: cluster 2021-03-15T22:19:38.767407+0000 osd.1 (osd.1) 21 : cluster [DBG] 2021-03-15T22:19:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: 4.12 scrub ok 2021-03-15T22:19:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: cluster 2021-03-15T22:19:39.105062+0000 osd.3 (osd.3) 24 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: cluster 2021-03-15T22:19:39. 2021-03-15T22:19:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: 105148+0000 osd.3 (osd.3) 25 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: cluster 2021-03-15T22:19: 2021-03-15T22:19:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: 39.224193+0000 osd.5 (osd.5) 20 : 2021-03-15T22:19:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: cluster [DBG] 5.15 scrub ok 2021-03-15T22:19:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:19:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: -15T22:19:39.378879+0000 mgr.y ( 2021-03-15T22:19:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: mgr.14144) 337 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:19:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: 03-15T22:19:39.594944+0000 2021-03-15T22:19:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: mgr.y (mgr.14144) 338 : audit [DBG] from='client.14637 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: 2021-03-15T22:19:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: cluster 2021-03-15T22:19:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: 2021-03-15T22:19:39.681735+0000 osd.0 2021-03-15T22:19:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: (osd.0) 25 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:19:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:40 smithi125 conmon[56739]: 2021-03-15T22:19:40.586 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:41.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[56514]: cluster 2021-03-15T22:19:40.040929+0000 mon.a (mon.0) 585 2021-03-15T22:19:41.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[56514]: : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2021-03-15T22:19:41.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[56514]: cluster 2021-03-15T22:19:40.166697+0000 osd.4 (osd.4) 15 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:19:41.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[56514]: cluster 2021-03-15T22:19:40.240381+0000 osd.5 (osd.5) 21 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:19:41.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[56514]: cluster 2021-03-15T22:19:40.666506+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.1e deep-scrub ok 2021-03-15T22:19:41.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[56514]: cluster 2021-03-15T22:19:40.969970+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v247: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:19:41.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[62156]: cluster 2021-03-15T22:19:40.040929+0000 mon.a 2021-03-15T22:19:41.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[62156]: (mon.0) 585 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2021-03-15T22:19:41.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[62156]: cluster 2021-03-15T22:19:40.166697+0000 osd.4 (osd.4) 15 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:19:41.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[62156]: cluster 2021-03-15T22:19:40.240381+0000 osd.5 (osd.5) 21 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:19:41.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[62156]: cluster 2021-03-15T22:19:40.666506+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.1e deep-scrub ok 2021-03-15T22:19:41.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:41 smithi076 conmon[62156]: cluster 2021-03-15T22:19:40.969970+0000 mgr.y (mgr.14144) 339 : cluster [DBG] pgmap v247: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:19:41.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:41 smithi125 conmon[56739]: cluster 2021-03-15T22:19:40.040929+0000 mon.a (mon.0) 585 : cluster [DBG] osdmap e87: 8 total, 8 up, 8 in 2021-03-15T22:19:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:41 smithi125 conmon[56739]: 2021-03-15T22:19:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:41 smithi125 conmon[56739]: cluster 2021-03-15T22:19:40.166697+0000 osd.4 (osd.4) 15 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:19:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:41 smithi125 conmon[56739]: cluster 2021-03-15T22:19:40.240381+0000 osd.5 (osd.5) 21 : 2021-03-15T22:19:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:41 smithi125 conmon[56739]: cluster [DBG] 4.8 scrub ok 2021-03-15T22:19:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:41 smithi125 conmon[56739]: cluster 2021-03-15T22:19:40 2021-03-15T22:19:41.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:41 smithi125 conmon[56739]: .666506+0000 osd.0 (osd.0) 26 : cluster [DBG] 4.1e deep-scrub ok 2021-03-15T22:19:41.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:41 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:19:41.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:41 smithi125 conmon[56739]: 03-15T22:19:40.969970+0000 2021-03-15T22:19:41.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:41 smithi125 conmon[56739]: mgr.y (mgr.14144) 339 : cluster [DBG] 2021-03-15T22:19:41.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:41 smithi125 conmon[56739]: pgmap v247: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:19:42.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:42 smithi076 conmon[56514]: cluster 2021-03-15T22:19:41.212535+0000 osd.5 (osd.5) 22 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:19:42.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:42 smithi076 conmon[62156]: cluster 2021-03-15T22:19:41.212535+0000 osd.5 (osd.5) 22 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:19:42.415 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:19:42.415 INFO:teuthology.orchestra.run.smithi076.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-03-15T22:19:42.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:42 smithi125 conmon[56739]: cluster 2021-03-15T22:19:41.212535+0000 osd.5 (osd.5) 22 : cluster [DBG] 2021-03-15T22:19:42.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:42 smithi125 conmon[56739]: 3.5 scrub ok 2021-03-15T22:19:43.116 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2021-03-15T22:19:43.117 INFO:tasks.cephadm:Setup complete, yielding 2021-03-15T22:19:43.117 INFO:teuthology.run_tasks:Running task thrashosds... 2021-03-15T22:19:43.134 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2021-03-15T22:19:43.135 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'wip-sage-testing-2021-03-15-1507'}, 'ceph': {'conf': {'global': {'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': 'b658018acf46e32d2ec283d302076352e17476f5'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'sha1': 'b658018acf46e32d2ec283d302076352e17476f5'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'wip-sage-testing-2021-03-15-1507', 'sha1': 'b658018acf46e32d2ec283d302076352e17476f5'}} 2021-03-15T22:19:43.135 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2021-03-15T22:19:43.136 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2021-03-15T22:19:43.434 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:43.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[56514]: cluster 2021-03-15T22:19:42.258824+0000 2021-03-15T22:19:43.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[56514]: osd.5 (osd.5) 23 : cluster [DBG] 6.d scrub ok 2021-03-15T22:19:43.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[56514]: audit 2021-03-15T22:19:42.413766+0000 mon.a (mon.0) 586 : audit [DBG] from='client.? 172.21.15.76:0/3296215057' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-03-15T22:19:43.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[56514]: cluster 2021-03-15T22:19:42.970591+0000 mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v248: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:43.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: cluster 2021-03-15T22:19:43.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: 2021-03-15T22:19:43.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: 2021-03-15T22:19:42. 2021-03-15T22:19:43.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: 258824+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.d scrub ok 2021-03-15T22:19:43.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: audit 2021-03-15T22:19:43.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: 2021-03-15T22:19:42 2021-03-15T22:19:43.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: .413766+0000 mon.a (mon. 2021-03-15T22:19:43.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: 0) 586 : audit [DBG] from='client.? 172.21.15.76:0/3296215057' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-03-15T22:19:43.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: 2021-03-15T22:19:43.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:19:43.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: -03-15T22:19:42.970591+0000 2021-03-15T22:19:43.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: mgr.y (mgr.14144) 340 : cluster [DBG] pgmap v248: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:43.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:43 smithi076 conmon[62156]: 2021-03-15T22:19:43.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:43 smithi125 conmon[56739]: cluster 2021-03-15T22:19:42.258824+0000 osd.5 (osd.5) 23 : cluster [DBG] 6.d scrub ok 2021-03-15T22:19:43.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:43 smithi125 conmon[56739]: 2021-03-15T22:19:43.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:43 smithi125 conmon[56739]: audit 2021-03-15T22:19:42.413766+0000 mon.a (mon.0) 586 : audit [DBG] from='client.? 172.21.15.76:0/3296215057' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2021-03-15T22:19:43.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:43 smithi125 conmon[56739]: cluster 2021-03-15T22:19:42.970591+0000 mgr.y (mgr 2021-03-15T22:19:43.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:43 smithi125 conmon[56739]: .14144) 340 : cluster [DBG] pgmap v248: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:44.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:44 smithi076 conmon[62156]: cluster 2021-03-15T22:19:43.737122+0000 osd.1 (osd.1) 22 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:19:44.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:44 smithi076 conmon[56514]: cluster 2021-03-15T22:19:43.737122+0000 2021-03-15T22:19:44.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:44 smithi076 conmon[56514]: osd.1 (osd.1) 22 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:19:45.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:44 smithi125 conmon[56739]: cluster 2021-03-15T22:19:43.737122+0000 osd.1 (osd.1) 22 : cluster [DBG] 2021-03-15T22:19:45.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:44 smithi125 conmon[56739]: 4.2 scrub ok 2021-03-15T22:19:45.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: debug 2021-03-15T22:19:45.397+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:19:45.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:45.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:45.518 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:45.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:45.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:45.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:45.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:45.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:19:45.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: debug 2021-03-15T22:19:45.398+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:45.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:45.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:45.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:45.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:45.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:45.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:45.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:45.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:45.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: debug 2021-03-15T22:19:45.398+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:45.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:45.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:45.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:45.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:45.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:45.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:45.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:45.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:45.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: debug 2021-03-15T22:19:45.398+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:45.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:45.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:45.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:45.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:45.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:45.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:45.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:45.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:45.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: debug 2021-03-15T22:19:45.399+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:45.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:45.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:45.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:45.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:45.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:45.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:45.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:45.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:45.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: debug 2021-03-15T22:19:45.399+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:45.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:45.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:45.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:45.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:45.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:45.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:45.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:45.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:45.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: debug 2021-03-15T22:19:45.400+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:45.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:19:45.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:45.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:19:45.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:45.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:19:45.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:45.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:45.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:45.785 INFO:tasks.thrashosds:Beginning thrashosds... 2021-03-15T22:19:45.786 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2021-03-15T22:19:45.786 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:19:46.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: cluster 2021-03-15T22:19:43.915514+0000 osd.7 (osd.7) 18 : 2021-03-15T22:19:46.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: cluster [DBG] 3.13 scrub ok 2021-03-15T22:19:46.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: cluster 2021-03-15T22:19:44.585937+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:19:46.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: cluster 2021-03-15T22:19:46.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: 2021-03-15T22:19:44.971096+0000 mgr.y (mgr.14144) 341 : 2021-03-15T22:19:46.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: cluster [DBG] pgmap v249: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:46.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: audit 2021-03-15T22:19:46.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: 2021-03-15T22:19:45.112820+0000 mon.a ( 2021-03-15T22:19:46.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: mon.0) 587 : audit [INF] from='client.? 172.21.15.76:0/2305462783' entity='client.admin' 2021-03-15T22:19:46.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: 2021-03-15T22:19:46.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: audit 2021-03-15T22:19:46.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: 2021-03-15T22:19:45.116556+0000 2021-03-15T22:19:46.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: mon.a (mon.0) 588 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:19:46.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: audit 2021-03-15T22:19:45.398497+0000 2021-03-15T22:19:46.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[56514]: mon.a (mon.0) 589 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:46.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: cluster 2021-03-15T22:19:43.915514 2021-03-15T22:19:46.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: +0000 osd.7 (osd.7) 18 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:19:46.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:19:46.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: 22:19:44.585937+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:19:46.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: cluster 2021-03-15T22:19:44.971096 2021-03-15T22:19:46.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v249: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:46.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:19:46.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: T22:19:45.112820+0000 mon.a (mon.0) 587 : audit 2021-03-15T22:19:46.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/2305462783' entity='client.admin' 2021-03-15T22:19:46.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: audit 2021-03-15T22:19:46.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: 2021-03-15T22:19:45.116556+0000 mon.a ( 2021-03-15T22:19:46.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: mon.0) 588 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:19:46.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:19:46.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: 22:19:45.398497+0000 mon.a (mon.0) 589 : audit 2021-03-15T22:19:46.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:45 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:46.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:45 smithi125 conmon[56739]: cluster 2021-03-15T22:19:43.915514+0000 osd.7 (osd.7) 18 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:19:46.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:45 smithi125 conmon[56739]: cluster 2021-03-15T22:19:44.585937+0000 osd.0 (osd.0) 27 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:19:46.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:45 smithi125 conmon[56739]: cluster 2021-03-15T22:19:44.971096+0000 mgr.y (mgr.14144) 341 : cluster [DBG] pgmap v249: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:46.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:45 smithi125 conmon[56739]: audit 2021-03-15T22:19:45.112820+0000 mon.a (mon.0) 587 : audit [INF] from='client.? 172.21.15.76:0/2305462783' entity='client.admin' 2021-03-15T22:19:46.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:45 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:19:46.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:45 smithi125 conmon[56739]: :19:45.116556+0000 mon.a (mon.0) 588 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:19:46.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:45 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:19:46.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:45 smithi125 conmon[56739]: 22:19:45.398497+0000 mon.a ( 2021-03-15T22:19:46.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:45 smithi125 conmon[56739]: mon.0) 589 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:19:46.066 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:46.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: cluster 2021-03-15T22:19:44.915861+0000 osd.3 (osd. 2021-03-15T22:19:46.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: 3) 26 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:19:46.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:45.394914+0000 mgr.y (mgr.14144) 342 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:19:46.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:45.399336+0000 mgr.y (mgr.14144) 343 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:19:46.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:46.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:46.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:46.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:19:46.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:45.399816+0000 mgr.y (mgr.14144) 344 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:46.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:46.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:46.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:46.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:46.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:45.400273+0000 mgr.y (mgr.14144) 345 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:46.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:46.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:46.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:46.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: cluster 2021-03-15T22:19:44.915861+0000 osd.3 (osd.3) 2021-03-15T22:19:46.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: 26 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:19:46.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:45.394914+0000 mgr.y (mgr.14144) 342 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:19:46.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:45.399336+0000 mgr.y (mgr.14144) 343 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:19:46.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:46.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:46.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:46.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:19:46.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: cephadm 2021-03-15T 2021-03-15T22:19:46.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: 22:19:45.399816+0000 mgr.y (mgr.14144) 344 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:46.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:46.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:46.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:46.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:46.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:45.400273+0000 mgr.y (mgr.14144) 345 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:46.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:46.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:46.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:46.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:46.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:45.400643+0000 mgr.y (mgr.14144) 346 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:46.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:46.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:46.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:46.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:46.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:45.401029+0000 mgr.y (mgr.14144) 347 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:46.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:46.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:46.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:46.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:46.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:45.401398+0000 mgr.y (mgr.14144) 348 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:46.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:46.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:46.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:46.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:46.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:46.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:45.400643+0000 mgr.y (mgr.14144) 346 : cephadm 2021-03-15T22:19:46.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:46.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:46.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:46.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:46.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.791 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:46.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:45.401029+0000 mgr.y (mgr.14144) 347 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:46.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:46.792 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:46.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:46.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.793 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:46.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:19:45.401398+0000 mgr.y (mgr.14144) 348 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:46.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:46.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:46.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:46.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:46.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:19:45.401774+0000 mgr.y (mgr.14144) 349 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:46.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:19:46.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:19:46.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:19:46.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:46.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:19:46.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: 03-15T22:19:45.401774+0000 mgr.y (mgr.14144) 349 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:46.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:19:46.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:46.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:19:46.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:46.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:19:46.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:46.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:46.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:47.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: cluster 2021-03-15T22:19:44.915861+0000 osd.3 (osd.3) 26 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:19:47.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:45.394914+0000 mgr.y (mgr.14144) 342 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:19:47.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:45.399336+0000 mgr.y (mgr.14144) 343 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:19:47.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:47.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:47.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:47.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:47.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:47.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:47.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:47.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:19:47.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:45.399816+0000 mgr.y 2021-03-15T22:19:47.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: (mgr.14144) 344 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:19:47.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:47.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:47.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:47.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:47.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:47.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:47.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:47.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:19:47.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:45.400273+0000 mgr.y (mgr.14144) 345 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:19:47.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:47.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:47.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:47.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:47.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:47.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:47.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:47.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:19:47.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:45.400643+0000 mgr.y (mgr.14144) 346 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:19:47.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:47.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:47.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:47.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:47.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:47.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:47.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:47.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:19:47.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:45.401029+0000 mgr.y ( 2021-03-15T22:19:47.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: mgr.14144) 347 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:19:47.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:47.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:47.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:47.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:47.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:47.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:47.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:47.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:19:47.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:45.401398+0000 mgr.y (mgr.14144) 348 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:19:47.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:47.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:47.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:47.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:47.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:47.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:47.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:47.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:19:47.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:19:45.401774+0000 mgr.y (mgr.14144) 349 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:19:47.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:19:47.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:19:47.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:19:47.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:19:47.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:19:47.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:19:47.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:19:47.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:19:47.770 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:19:47.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:47 smithi076 conmon[56514]: cluster 2021-03-15T22:19:45.893905+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:19:47.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:47 smithi076 conmon[56514]: cluster 2021-03-15T22:19:46.971757+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v250: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:47.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:47 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:19:47.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:47 smithi076 conmon[62156]: :19:45.893905+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:19:47.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:47 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:19:47.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:47 smithi076 conmon[62156]: -03-15T22:19:46 2021-03-15T22:19:47.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:47 smithi076 conmon[62156]: .971757+0000 mgr.y (mgr. 2021-03-15T22:19:47.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:47 smithi076 conmon[62156]: 14144) 350 : cluster [DBG] pgmap v250: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:47.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:47 smithi076 conmon[62156]: 2021-03-15T22:19:47.775 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:19:48.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:47 smithi125 conmon[56739]: cluster 2021-03-15T22:19:45.893905+0000 osd.3 (osd.3) 27 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:19:48.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:47 smithi125 conmon[56739]: cluster 2021-03-15T22:19:46.971757+0000 mgr.y (mgr.14144) 350 : cluster [DBG] pgmap v250: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:48.402 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":250,"stamp":"2021-03-15T22:19:46.971345+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25236,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1183,"num_read_kb":683,"num_write":550,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":582,"ondisk_log_size":582,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":53356,"kb_used_data":7212,"kb_used_omap":0,"kb_used_meta":46080,"kb_avail":749907860,"statfs":{"total":767960285184,"available":767905648640,"internally_reserved":0,"allocated":7385088,"data_stored":2952372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47185920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002834"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994463+0000","last_change":"2021-03-15T22:19:28.122158+0000","last_active":"2021-03-15T22:19:30.994463+0000","last_peered":"2021-03-15T22:19:30.994463+0000","last_clean":"2021-03-15T22:19:30.994463+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:19:30.994463+0000","last_undegraded":"2021-03-15T22:19:30.994463+0000","last_fullsized":"2021-03-15T22:19:30.994463+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.122079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.122079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:37.782723+0000","last_change":"2021-03-15T22:19:37.782674+0000","last_active":"2021-03-15T22:19:37.782723+0000","last_peered":"2021-03-15T22:19:37.782723+0000","last_clean":"2021-03-15T22:19:37.782723+0000","last_became_active":"2021-03-15T22:17:58.014719+0000","last_became_peered":"2021-03-15T22:17:58.014719+0000","last_unstale":"2021-03-15T22:19:37.782723+0000","last_undegraded":"2021-03-15T22:19:37.782723+0000","last_fullsized":"2021-03-15T22:19:37.782723+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.782584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.782584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"46","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995957+0000","last_change":"2021-03-15T22:19:08.789028+0000","last_active":"2021-03-15T22:19:30.995957+0000","last_peered":"2021-03-15T22:19:30.995957+0000","last_clean":"2021-03-15T22:19:30.995957+0000","last_became_active":"2021-03-15T22:17:54.284855+0000","last_became_peered":"2021-03-15T22:17:54.284855+0000","last_unstale":"2021-03-15T22:19:30.995957+0000","last_undegraded":"2021-03-15T22:19:30.995957+0000","last_fullsized":"2021-03-15T22:19:30.995957+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:19:08.788964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:08.788964+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"70","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994360+0000","last_change":"2021-03-15T22:19:11.891528+0000","last_active":"2021-03-15T22:19:30.994360+0000","last_peered":"2021-03-15T22:19:30.994360+0000","last_clean":"2021-03-15T22:19:30.994360+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:19:30.994360+0000","last_undegraded":"2021-03-15T22:19:30.994360+0000","last_fullsized":"2021-03-15T22:19:30.994360+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:11.891465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.891465+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996834+0000","last_change":"2021-03-15T22:19:26.865961+0000","last_active":"2021-03-15T22:19:30.996834+0000","last_peered":"2021-03-15T22:19:30.996834+0000","last_clean":"2021-03-15T22:19:30.996834+0000","last_became_active":"2021-03-15T22:18:00.288598+0000","last_became_peered":"2021-03-15T22:18:00.288598+0000","last_unstale":"2021-03-15T22:19:30.996834+0000","last_undegraded":"2021-03-15T22:19:30.996834+0000","last_fullsized":"2021-03-15T22:19:30.996834+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.865895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.865895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"35","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994352+0000","last_change":"2021-03-15T22:17:58.011237+0000","last_active":"2021-03-15T22:19:30.994352+0000","last_peered":"2021-03-15T22:19:30.994352+0000","last_clean":"2021-03-15T22:19:30.994352+0000","last_became_active":"2021-03-15T22:17:58.010916+0000","last_became_peered":"2021-03-15T22:17:58.010916+0000","last_unstale":"2021-03-15T22:19:30.994352+0000","last_undegraded":"2021-03-15T22:19:30.994352+0000","last_fullsized":"2021-03-15T22:19:30.994352+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.997201+0000","last_change":"2021-03-15T22:19:10.851317+0000","last_active":"2021-03-15T22:19:30.997201+0000","last_peered":"2021-03-15T22:19:30.997201+0000","last_clean":"2021-03-15T22:19:30.997201+0000","last_became_active":"2021-03-15T22:17:54.283819+0000","last_became_peered":"2021-03-15T22:17:54.283819+0000","last_unstale":"2021-03-15T22:19:30.997201+0000","last_undegraded":"2021-03-15T22:19:30.997201+0000","last_fullsized":"2021-03-15T22:19:30.997201+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:10.851256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:10.851256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"77","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:44.915926+0000","last_change":"2021-03-15T22:19:44.915926+0000","last_active":"2021-03-15T22:19:44.915926+0000","last_peered":"2021-03-15T22:19:44.915926+0000","last_clean":"2021-03-15T22:19:44.915926+0000","last_became_active":"2021-03-15T22:17:55.991597+0000","last_became_peered":"2021-03-15T22:17:55.991597+0000","last_unstale":"2021-03-15T22:19:44.915926+0000","last_undegraded":"2021-03-15T22:19:44.915926+0000","last_fullsized":"2021-03-15T22:19:44.915926+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:19:44.915865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:44.915865+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996441+0000","last_change":"2021-03-15T22:19:11.832031+0000","last_active":"2021-03-15T22:19:30.996441+0000","last_peered":"2021-03-15T22:19:30.996441+0000","last_clean":"2021-03-15T22:19:30.996441+0000","last_became_active":"2021-03-15T22:18:00.015776+0000","last_became_peered":"2021-03-15T22:18:00.015776+0000","last_unstale":"2021-03-15T22:19:30.996441+0000","last_undegraded":"2021-03-15T22:19:30.996441+0000","last_fullsized":"2021-03-15T22:19:30.996441+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:11.831963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.831963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500493+0000","last_change":"2021-03-15T22:19:30.205991+0000","last_active":"2021-03-15T22:19:31.500493+0000","last_peered":"2021-03-15T22:19:31.500493+0000","last_clean":"2021-03-15T22:19:31.500493+0000","last_became_active":"2021-03-15T22:17:58.008828+0000","last_became_peered":"2021-03-15T22:17:58.008828+0000","last_unstale":"2021-03-15T22:19:31.500493+0000","last_undegraded":"2021-03-15T22:19:31.500493+0000","last_fullsized":"2021-03-15T22:19:31.500493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.205915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.205915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500577+0000","last_change":"2021-03-15T22:18:53.334542+0000","last_active":"2021-03-15T22:19:31.500577+0000","last_peered":"2021-03-15T22:19:31.500577+0000","last_clean":"2021-03-15T22:19:31.500577+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:19:31.500577+0000","last_undegraded":"2021-03-15T22:19:31.500577+0000","last_fullsized":"2021-03-15T22:19:31.500577+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:53.334478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:53.334478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"61","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994691+0000","last_change":"2021-03-15T22:17:55.990767+0000","last_active":"2021-03-15T22:19:30.994691+0000","last_peered":"2021-03-15T22:19:30.994691+0000","last_clean":"2021-03-15T22:19:30.994691+0000","last_became_active":"2021-03-15T22:17:55.990528+0000","last_became_peered":"2021-03-15T22:17:55.990528+0000","last_unstale":"2021-03-15T22:19:30.994691+0000","last_undegraded":"2021-03-15T22:19:30.994691+0000","last_fullsized":"2021-03-15T22:19:30.994691+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:37.268525+0000","last_change":"2021-03-15T22:19:37.268478+0000","last_active":"2021-03-15T22:19:37.268525+0000","last_peered":"2021-03-15T22:19:37.268525+0000","last_clean":"2021-03-15T22:19:37.268525+0000","last_became_active":"2021-03-15T22:18:00.019585+0000","last_became_peered":"2021-03-15T22:18:00.019585+0000","last_unstale":"2021-03-15T22:19:37.268525+0000","last_undegraded":"2021-03-15T22:19:37.268525+0000","last_fullsized":"2021-03-15T22:19:37.268525+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.268362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.268362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"35","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884909+0000","last_change":"2021-03-15T22:17:58.013492+0000","last_active":"2021-03-15T22:19:31.884909+0000","last_peered":"2021-03-15T22:19:31.884909+0000","last_clean":"2021-03-15T22:19:31.884909+0000","last_became_active":"2021-03-15T22:17:58.013236+0000","last_became_peered":"2021-03-15T22:17:58.013236+0000","last_unstale":"2021-03-15T22:19:31.884909+0000","last_undegraded":"2021-03-15T22:19:31.884909+0000","last_fullsized":"2021-03-15T22:19:31.884909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500507+0000","last_change":"2021-03-15T22:19:06.274483+0000","last_active":"2021-03-15T22:19:31.500507+0000","last_peered":"2021-03-15T22:19:31.500507+0000","last_clean":"2021-03-15T22:19:31.500507+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:19:31.500507+0000","last_undegraded":"2021-03-15T22:19:31.500507+0000","last_fullsized":"2021-03-15T22:19:31.500507+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:06.274416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.274416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'6","reported_seq":"71","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995078+0000","last_change":"2021-03-15T22:19:02.277922+0000","last_active":"2021-03-15T22:19:30.995078+0000","last_peered":"2021-03-15T22:19:30.995078+0000","last_clean":"2021-03-15T22:19:30.995078+0000","last_became_active":"2021-03-15T22:19:02.274105+0000","last_became_peered":"2021-03-15T22:19:02.274105+0000","last_unstale":"2021-03-15T22:19:30.995078+0000","last_undegraded":"2021-03-15T22:19:30.995078+0000","last_fullsized":"2021-03-15T22:19:30.995078+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2021-03-15T22:18:59.037504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.037504+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"33","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995329+0000","last_change":"2021-03-15T22:18:00.287135+0000","last_active":"2021-03-15T22:19:30.995329+0000","last_peered":"2021-03-15T22:19:30.995329+0000","last_clean":"2021-03-15T22:19:30.995329+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:19:30.995329+0000","last_undegraded":"2021-03-15T22:19:30.995329+0000","last_fullsized":"2021-03-15T22:19:30.995329+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:35.779241+0000","last_change":"2021-03-15T22:19:35.779203+0000","last_active":"2021-03-15T22:19:35.779241+0000","last_peered":"2021-03-15T22:19:35.779241+0000","last_clean":"2021-03-15T22:19:35.779241+0000","last_became_active":"2021-03-15T22:17:58.016067+0000","last_became_peered":"2021-03-15T22:17:58.016067+0000","last_unstale":"2021-03-15T22:19:35.779241+0000","last_undegraded":"2021-03-15T22:19:35.779241+0000","last_fullsized":"2021-03-15T22:19:35.779241+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:35.779105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.779105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"46","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996187+0000","last_change":"2021-03-15T22:19:14.859504+0000","last_active":"2021-03-15T22:19:30.996187+0000","last_peered":"2021-03-15T22:19:30.996187+0000","last_clean":"2021-03-15T22:19:30.996187+0000","last_became_active":"2021-03-15T22:17:53.976875+0000","last_became_peered":"2021-03-15T22:17:53.976875+0000","last_unstale":"2021-03-15T22:19:30.996187+0000","last_undegraded":"2021-03-15T22:19:30.996187+0000","last_fullsized":"2021-03-15T22:19:30.996187+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.859443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"67","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944940+0000","last_change":"2021-03-15T22:19:27.165212+0000","last_active":"2021-03-15T22:19:31.944940+0000","last_peered":"2021-03-15T22:19:31.944940+0000","last_clean":"2021-03-15T22:19:31.944940+0000","last_became_active":"2021-03-15T22:17:55.992247+0000","last_became_peered":"2021-03-15T22:17:55.992247+0000","last_unstale":"2021-03-15T22:19:31.944940+0000","last_undegraded":"2021-03-15T22:19:31.944940+0000","last_fullsized":"2021-03-15T22:19:31.944940+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:27.165119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.165119+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"38","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:45.893973+0000","last_change":"2021-03-15T22:19:45.893973+0000","last_active":"2021-03-15T22:19:45.893973+0000","last_peered":"2021-03-15T22:19:45.893973+0000","last_clean":"2021-03-15T22:19:45.893973+0000","last_became_active":"2021-03-15T22:18:00.286974+0000","last_became_peered":"2021-03-15T22:18:00.286974+0000","last_unstale":"2021-03-15T22:19:45.893973+0000","last_undegraded":"2021-03-15T22:19:45.893973+0000","last_fullsized":"2021-03-15T22:19:45.893973+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:45.893910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:45.893910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994924+0000","last_change":"2021-03-15T22:19:21.043341+0000","last_active":"2021-03-15T22:19:30.994924+0000","last_peered":"2021-03-15T22:19:30.994924+0000","last_clean":"2021-03-15T22:19:30.994924+0000","last_became_active":"2021-03-15T22:17:58.005477+0000","last_became_peered":"2021-03-15T22:17:58.005477+0000","last_unstale":"2021-03-15T22:19:30.994924+0000","last_undegraded":"2021-03-15T22:19:30.994924+0000","last_fullsized":"2021-03-15T22:19:30.994924+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.043261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.043261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:35.186077+0000","last_change":"2021-03-15T22:19:35.186018+0000","last_active":"2021-03-15T22:19:35.186077+0000","last_peered":"2021-03-15T22:19:35.186077+0000","last_clean":"2021-03-15T22:19:35.186077+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:19:35.186077+0000","last_undegraded":"2021-03-15T22:19:35.186077+0000","last_fullsized":"2021-03-15T22:19:35.186077+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:35.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.185891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"73","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:35.934619+0000","last_change":"2021-03-15T22:19:35.934561+0000","last_active":"2021-03-15T22:19:35.934619+0000","last_peered":"2021-03-15T22:19:35.934619+0000","last_clean":"2021-03-15T22:19:35.934619+0000","last_became_active":"2021-03-15T22:17:55.995747+0000","last_became_peered":"2021-03-15T22:17:55.995747+0000","last_unstale":"2021-03-15T22:19:35.934619+0000","last_undegraded":"2021-03-15T22:19:35.934619+0000","last_fullsized":"2021-03-15T22:19:35.934619+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:35.934475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.934475+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885511+0000","last_change":"2021-03-15T22:19:30.854972+0000","last_active":"2021-03-15T22:19:31.885511+0000","last_peered":"2021-03-15T22:19:31.885511+0000","last_clean":"2021-03-15T22:19:31.885511+0000","last_became_active":"2021-03-15T22:18:00.015889+0000","last_became_peered":"2021-03-15T22:18:00.015889+0000","last_unstale":"2021-03-15T22:19:31.885511+0000","last_undegraded":"2021-03-15T22:19:31.885511+0000","last_fullsized":"2021-03-15T22:19:31.885511+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.854899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.854899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"61'2","reported_seq":"62","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.165293+0000","last_change":"2021-03-15T22:19:21.656004+0000","last_active":"2021-03-15T22:19:40.165293+0000","last_peered":"2021-03-15T22:19:40.165293+0000","last_clean":"2021-03-15T22:19:40.165293+0000","last_became_active":"2021-03-15T22:17:58.013571+0000","last_became_peered":"2021-03-15T22:17:58.013571+0000","last_unstale":"2021-03-15T22:19:40.165293+0000","last_undegraded":"2021-03-15T22:19:40.165293+0000","last_fullsized":"2021-03-15T22:19:40.165293+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:21.655918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.655918+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944147+0000","last_change":"2021-03-15T22:19:16.638497+0000","last_active":"2021-03-15T22:19:31.944147+0000","last_peered":"2021-03-15T22:19:31.944147+0000","last_clean":"2021-03-15T22:19:31.944147+0000","last_became_active":"2021-03-15T22:17:53.975549+0000","last_became_peered":"2021-03-15T22:17:53.975549+0000","last_unstale":"2021-03-15T22:19:31.944147+0000","last_undegraded":"2021-03-15T22:19:31.944147+0000","last_fullsized":"2021-03-15T22:19:31.944147+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.638428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.638428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"79","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.666578+0000","last_change":"2021-03-15T22:19:40.666578+0000","last_active":"2021-03-15T22:19:40.666578+0000","last_peered":"2021-03-15T22:19:40.666578+0000","last_clean":"2021-03-15T22:19:40.666578+0000","last_became_active":"2021-03-15T22:17:56.288283+0000","last_became_peered":"2021-03-15T22:17:56.288283+0000","last_unstale":"2021-03-15T22:19:40.666578+0000","last_undegraded":"2021-03-15T22:19:40.666578+0000","last_fullsized":"2021-03-15T22:19:40.666578+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.666518+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944579+0000","last_change":"2021-03-15T22:19:07.544919+0000","last_active":"2021-03-15T22:19:31.944579+0000","last_peered":"2021-03-15T22:19:31.944579+0000","last_clean":"2021-03-15T22:19:31.944579+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:19:31.944579+0000","last_undegraded":"2021-03-15T22:19:31.944579+0000","last_fullsized":"2021-03-15T22:19:31.944579+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.544856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.544856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996391+0000","last_change":"2021-03-15T22:19:17.752562+0000","last_active":"2021-03-15T22:19:30.996391+0000","last_peered":"2021-03-15T22:19:30.996391+0000","last_clean":"2021-03-15T22:19:30.996391+0000","last_became_active":"2021-03-15T22:17:58.009184+0000","last_became_peered":"2021-03-15T22:17:58.009184+0000","last_unstale":"2021-03-15T22:19:30.996391+0000","last_undegraded":"2021-03-15T22:19:30.996391+0000","last_fullsized":"2021-03-15T22:19:30.996391+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.752495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.752495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996276+0000","last_change":"2021-03-15T22:19:21.900042+0000","last_active":"2021-03-15T22:19:30.996276+0000","last_peered":"2021-03-15T22:19:30.996276+0000","last_clean":"2021-03-15T22:19:30.996276+0000","last_became_active":"2021-03-15T22:17:53.967947+0000","last_became_peered":"2021-03-15T22:17:53.967947+0000","last_unstale":"2021-03-15T22:19:30.996276+0000","last_undegraded":"2021-03-15T22:19:30.996276+0000","last_fullsized":"2021-03-15T22:19:30.996276+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.899969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.899969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"72","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.886568+0000","last_change":"2021-03-15T22:19:13.525956+0000","last_active":"2021-03-15T22:19:31.886568+0000","last_peered":"2021-03-15T22:19:31.886568+0000","last_clean":"2021-03-15T22:19:31.886568+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:19:31.886568+0000","last_undegraded":"2021-03-15T22:19:31.886568+0000","last_fullsized":"2021-03-15T22:19:31.886568+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:19:13.525883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.525883+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884802+0000","last_change":"2021-03-15T22:19:25.759114+0000","last_active":"2021-03-15T22:19:31.884802+0000","last_peered":"2021-03-15T22:19:31.884802+0000","last_clean":"2021-03-15T22:19:31.884802+0000","last_became_active":"2021-03-15T22:18:00.015858+0000","last_became_peered":"2021-03-15T22:18:00.015858+0000","last_unstale":"2021-03-15T22:19:31.884802+0000","last_undegraded":"2021-03-15T22:19:31.884802+0000","last_fullsized":"2021-03-15T22:19:31.884802+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.759034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"41","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885584+0000","last_change":"2021-03-15T22:19:28.737870+0000","last_active":"2021-03-15T22:19:31.885584+0000","last_peered":"2021-03-15T22:19:31.885584+0000","last_clean":"2021-03-15T22:19:31.885584+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:19:31.885584+0000","last_undegraded":"2021-03-15T22:19:31.885584+0000","last_fullsized":"2021-03-15T22:19:31.885584+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.737796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.737796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996042+0000","last_change":"2021-03-15T22:19:01.740592+0000","last_active":"2021-03-15T22:19:30.996042+0000","last_peered":"2021-03-15T22:19:30.996042+0000","last_clean":"2021-03-15T22:19:30.996042+0000","last_became_active":"2021-03-15T22:17:53.973507+0000","last_became_peered":"2021-03-15T22:17:53.973507+0000","last_unstale":"2021-03-15T22:19:30.996042+0000","last_undegraded":"2021-03-15T22:19:30.996042+0000","last_fullsized":"2021-03-15T22:19:30.996042+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:01.740496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:01.740496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"59","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996360+0000","last_change":"2021-03-15T22:17:55.996994+0000","last_active":"2021-03-15T22:19:30.996360+0000","last_peered":"2021-03-15T22:19:30.996360+0000","last_clean":"2021-03-15T22:19:30.996360+0000","last_became_active":"2021-03-15T22:17:55.996707+0000","last_became_peered":"2021-03-15T22:17:55.996707+0000","last_unstale":"2021-03-15T22:19:30.996360+0000","last_undegraded":"2021-03-15T22:19:30.996360+0000","last_fullsized":"2021-03-15T22:19:30.996360+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996742+0000","last_change":"2021-03-15T22:19:14.850664+0000","last_active":"2021-03-15T22:19:30.996742+0000","last_peered":"2021-03-15T22:19:30.996742+0000","last_clean":"2021-03-15T22:19:30.996742+0000","last_became_active":"2021-03-15T22:18:00.287972+0000","last_became_peered":"2021-03-15T22:18:00.287972+0000","last_unstale":"2021-03-15T22:19:30.996742+0000","last_undegraded":"2021-03-15T22:19:30.996742+0000","last_fullsized":"2021-03-15T22:19:30.996742+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:14.850605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.850605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884873+0000","last_change":"2021-03-15T22:19:27.800200+0000","last_active":"2021-03-15T22:19:31.884873+0000","last_peered":"2021-03-15T22:19:31.884873+0000","last_clean":"2021-03-15T22:19:31.884873+0000","last_became_active":"2021-03-15T22:17:58.013174+0000","last_became_peered":"2021-03-15T22:17:58.013174+0000","last_unstale":"2021-03-15T22:19:31.884873+0000","last_undegraded":"2021-03-15T22:19:31.884873+0000","last_fullsized":"2021-03-15T22:19:31.884873+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:27.800106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.800106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500840+0000","last_change":"2021-03-15T22:19:17.295504+0000","last_active":"2021-03-15T22:19:31.500840+0000","last_peered":"2021-03-15T22:19:31.500840+0000","last_clean":"2021-03-15T22:19:31.500840+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:19:31.500840+0000","last_undegraded":"2021-03-15T22:19:31.500840+0000","last_fullsized":"2021-03-15T22:19:31.500840+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.295412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.295412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'14","reported_seq":"62","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996850+0000","last_change":"2021-03-15T22:17:56.289858+0000","last_active":"2021-03-15T22:19:30.996850+0000","last_peered":"2021-03-15T22:19:30.996850+0000","last_clean":"2021-03-15T22:19:30.996850+0000","last_became_active":"2021-03-15T22:17:56.289456+0000","last_became_peered":"2021-03-15T22:17:56.289456+0000","last_unstale":"2021-03-15T22:19:30.996850+0000","last_undegraded":"2021-03-15T22:19:30.996850+0000","last_fullsized":"2021-03-15T22:19:30.996850+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995910+0000","last_change":"2021-03-15T22:19:26.615442+0000","last_active":"2021-03-15T22:19:30.995910+0000","last_peered":"2021-03-15T22:19:30.995910+0000","last_clean":"2021-03-15T22:19:30.995910+0000","last_became_active":"2021-03-15T22:18:00.014112+0000","last_became_peered":"2021-03-15T22:18:00.014112+0000","last_unstale":"2021-03-15T22:19:30.995910+0000","last_undegraded":"2021-03-15T22:19:30.995910+0000","last_fullsized":"2021-03-15T22:19:30.995910+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.615380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.615380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996888+0000","last_change":"2021-03-15T22:19:03.699264+0000","last_active":"2021-03-15T22:19:30.996888+0000","last_peered":"2021-03-15T22:19:30.996888+0000","last_clean":"2021-03-15T22:19:30.996888+0000","last_became_active":"2021-03-15T22:17:58.009365+0000","last_became_peered":"2021-03-15T22:17:58.009365+0000","last_unstale":"2021-03-15T22:19:30.996888+0000","last_undegraded":"2021-03-15T22:19:30.996888+0000","last_fullsized":"2021-03-15T22:19:30.996888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_clean_scrub_stamp":"2021-03-15T22:19:03.699198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:36.927090+0000","last_change":"2021-03-15T22:19:36.927055+0000","last_active":"2021-03-15T22:19:36.927090+0000","last_peered":"2021-03-15T22:19:36.927090+0000","last_clean":"2021-03-15T22:19:36.927090+0000","last_became_active":"2021-03-15T22:17:53.979947+0000","last_became_peered":"2021-03-15T22:17:53.979947+0000","last_unstale":"2021-03-15T22:19:36.927090+0000","last_undegraded":"2021-03-15T22:19:36.927090+0000","last_fullsized":"2021-03-15T22:19:36.927090+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:36.926961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:36.926961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"70","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:38.767554+0000","last_change":"2021-03-15T22:19:38.767499+0000","last_active":"2021-03-15T22:19:38.767554+0000","last_peered":"2021-03-15T22:19:38.767554+0000","last_clean":"2021-03-15T22:19:38.767554+0000","last_became_active":"2021-03-15T22:17:55.991537+0000","last_became_peered":"2021-03-15T22:17:55.991537+0000","last_unstale":"2021-03-15T22:19:38.767554+0000","last_undegraded":"2021-03-15T22:19:38.767554+0000","last_fullsized":"2021-03-15T22:19:38.767554+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:38.767414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.767414+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"33","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996602+0000","last_change":"2021-03-15T22:18:00.018406+0000","last_active":"2021-03-15T22:19:30.996602+0000","last_peered":"2021-03-15T22:19:30.996602+0000","last_clean":"2021-03-15T22:19:30.996602+0000","last_became_active":"2021-03-15T22:18:00.018117+0000","last_became_peered":"2021-03-15T22:18:00.018117+0000","last_unstale":"2021-03-15T22:19:30.996602+0000","last_undegraded":"2021-03-15T22:19:30.996602+0000","last_fullsized":"2021-03-15T22:19:30.996602+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:32.759907+0000","last_change":"2021-03-15T22:19:32.759869+0000","last_active":"2021-03-15T22:19:32.759907+0000","last_peered":"2021-03-15T22:19:32.759907+0000","last_clean":"2021-03-15T22:19:32.759907+0000","last_became_active":"2021-03-15T22:17:58.014621+0000","last_became_peered":"2021-03-15T22:17:58.014621+0000","last_unstale":"2021-03-15T22:19:32.759907+0000","last_undegraded":"2021-03-15T22:19:32.759907+0000","last_fullsized":"2021-03-15T22:19:32.759907+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:32.759804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:32.759804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"44","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.166781+0000","last_change":"2021-03-15T22:19:40.166781+0000","last_active":"2021-03-15T22:19:40.166781+0000","last_peered":"2021-03-15T22:19:40.166781+0000","last_clean":"2021-03-15T22:19:40.166781+0000","last_became_active":"2021-03-15T22:17:53.975292+0000","last_became_peered":"2021-03-15T22:17:53.975292+0000","last_unstale":"2021-03-15T22:19:40.166781+0000","last_undegraded":"2021-03-15T22:19:40.166781+0000","last_fullsized":"2021-03-15T22:19:40.166781+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:40.166704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.166704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"69","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:38.156229+0000","last_change":"2021-03-15T22:19:38.156190+0000","last_active":"2021-03-15T22:19:38.156229+0000","last_peered":"2021-03-15T22:19:38.156229+0000","last_clean":"2021-03-15T22:19:38.156229+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:19:38.156229+0000","last_undegraded":"2021-03-15T22:19:38.156229+0000","last_fullsized":"2021-03-15T22:19:38.156229+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:38.156103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.156103+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"37","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:44.586030+0000","last_change":"2021-03-15T22:19:44.586030+0000","last_active":"2021-03-15T22:19:44.586030+0000","last_peered":"2021-03-15T22:19:44.586030+0000","last_clean":"2021-03-15T22:19:44.586030+0000","last_became_active":"2021-03-15T22:18:00.015034+0000","last_became_peered":"2021-03-15T22:18:00.015034+0000","last_unstale":"2021-03-15T22:19:44.586030+0000","last_undegraded":"2021-03-15T22:19:44.586030+0000","last_fullsized":"2021-03-15T22:19:44.586030+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:44.585947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:44.585947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"61'2","reported_seq":"62","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.239498+0000","last_change":"2021-03-15T22:19:39.224283+0000","last_active":"2021-03-15T22:19:40.239498+0000","last_peered":"2021-03-15T22:19:40.239498+0000","last_clean":"2021-03-15T22:19:40.239498+0000","last_became_active":"2021-03-15T22:17:58.010903+0000","last_became_peered":"2021-03-15T22:17:58.010903+0000","last_unstale":"2021-03-15T22:19:40.239498+0000","last_undegraded":"2021-03-15T22:19:40.239498+0000","last_fullsized":"2021-03-15T22:19:40.239498+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:39.224198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:39.224198+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"44","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:43.915597+0000","last_change":"2021-03-15T22:19:43.915597+0000","last_active":"2021-03-15T22:19:43.915597+0000","last_peered":"2021-03-15T22:19:43.915597+0000","last_clean":"2021-03-15T22:19:43.915597+0000","last_became_active":"2021-03-15T22:17:54.285851+0000","last_became_peered":"2021-03-15T22:17:54.285851+0000","last_unstale":"2021-03-15T22:19:43.915597+0000","last_undegraded":"2021-03-15T22:19:43.915597+0000","last_fullsized":"2021-03-15T22:19:43.915597+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:43.915518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:43.915518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"65","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.997197+0000","last_change":"2021-03-15T22:19:07.874527+0000","last_active":"2021-03-15T22:19:30.997197+0000","last_peered":"2021-03-15T22:19:30.997197+0000","last_clean":"2021-03-15T22:19:30.997197+0000","last_became_active":"2021-03-15T22:17:56.288102+0000","last_became_peered":"2021-03-15T22:17:56.288102+0000","last_unstale":"2021-03-15T22:19:30.997197+0000","last_undegraded":"2021-03-15T22:19:30.997197+0000","last_fullsized":"2021-03-15T22:19:30.997197+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:19:07.874462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.874462+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"33","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995545+0000","last_change":"2021-03-15T22:18:00.015238+0000","last_active":"2021-03-15T22:19:30.995545+0000","last_peered":"2021-03-15T22:19:30.995545+0000","last_clean":"2021-03-15T22:19:30.995545+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:19:30.995545+0000","last_undegraded":"2021-03-15T22:19:30.995545+0000","last_fullsized":"2021-03-15T22:19:30.995545+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"61'2","reported_seq":"63","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.661332+0000","last_change":"2021-03-15T22:19:15.850282+0000","last_active":"2021-03-15T22:19:40.661332+0000","last_peered":"2021-03-15T22:19:40.661332+0000","last_clean":"2021-03-15T22:19:40.661332+0000","last_became_active":"2021-03-15T22:17:58.003748+0000","last_became_peered":"2021-03-15T22:17:58.003748+0000","last_unstale":"2021-03-15T22:19:40.661332+0000","last_undegraded":"2021-03-15T22:19:40.661332+0000","last_fullsized":"2021-03-15T22:19:40.661332+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:15.850217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.850217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995608+0000","last_change":"2021-03-15T22:19:07.795140+0000","last_active":"2021-03-15T22:19:30.995608+0000","last_peered":"2021-03-15T22:19:30.995608+0000","last_clean":"2021-03-15T22:19:30.995608+0000","last_became_active":"2021-03-15T22:17:53.968736+0000","last_became_peered":"2021-03-15T22:17:53.968736+0000","last_unstale":"2021-03-15T22:19:30.995608+0000","last_undegraded":"2021-03-15T22:19:30.995608+0000","last_fullsized":"2021-03-15T22:19:30.995608+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.795072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.795072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"65","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:33.261379+0000","last_change":"2021-03-15T22:19:33.261320+0000","last_active":"2021-03-15T22:19:33.261379+0000","last_peered":"2021-03-15T22:19:33.261379+0000","last_clean":"2021-03-15T22:19:33.261379+0000","last_became_active":"2021-03-15T22:17:56.290563+0000","last_became_peered":"2021-03-15T22:17:56.290563+0000","last_unstale":"2021-03-15T22:19:33.261379+0000","last_undegraded":"2021-03-15T22:19:33.261379+0000","last_fullsized":"2021-03-15T22:19:33.261379+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:19:33.261213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:33.261213+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.945093+0000","last_change":"2021-03-15T22:19:16.224939+0000","last_active":"2021-03-15T22:19:31.945093+0000","last_peered":"2021-03-15T22:19:31.945093+0000","last_clean":"2021-03-15T22:19:31.945093+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:19:31.945093+0000","last_undegraded":"2021-03-15T22:19:31.945093+0000","last_fullsized":"2021-03-15T22:19:31.945093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.224872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:32.880488+0000","last_change":"2021-03-15T22:19:32.880446+0000","last_active":"2021-03-15T22:19:32.880488+0000","last_peered":"2021-03-15T22:19:32.880488+0000","last_clean":"2021-03-15T22:19:32.880488+0000","last_became_active":"2021-03-15T22:17:58.001340+0000","last_became_peered":"2021-03-15T22:17:58.001340+0000","last_unstale":"2021-03-15T22:19:32.880488+0000","last_undegraded":"2021-03-15T22:19:32.880488+0000","last_fullsized":"2021-03-15T22:19:32.880488+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:32.880362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:32.880362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"41","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884668+0000","last_change":"2021-03-15T22:19:13.870789+0000","last_active":"2021-03-15T22:19:31.884668+0000","last_peered":"2021-03-15T22:19:31.884668+0000","last_clean":"2021-03-15T22:19:31.884668+0000","last_became_active":"2021-03-15T22:17:53.976777+0000","last_became_peered":"2021-03-15T22:17:53.976777+0000","last_unstale":"2021-03-15T22:19:31.884668+0000","last_undegraded":"2021-03-15T22:19:31.884668+0000","last_fullsized":"2021-03-15T22:19:31.884668+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"71","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996311+0000","last_change":"2021-03-15T22:19:06.778692+0000","last_active":"2021-03-15T22:19:30.996311+0000","last_peered":"2021-03-15T22:19:30.996311+0000","last_clean":"2021-03-15T22:19:30.996311+0000","last_became_active":"2021-03-15T22:17:56.288678+0000","last_became_peered":"2021-03-15T22:17:56.288678+0000","last_unstale":"2021-03-15T22:19:30.996311+0000","last_undegraded":"2021-03-15T22:19:30.996311+0000","last_fullsized":"2021-03-15T22:19:30.996311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:06.778622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.778622+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885484+0000","last_change":"2021-03-15T22:19:17.797392+0000","last_active":"2021-03-15T22:19:31.885484+0000","last_peered":"2021-03-15T22:19:31.885484+0000","last_clean":"2021-03-15T22:19:31.885484+0000","last_became_active":"2021-03-15T22:18:00.288462+0000","last_became_peered":"2021-03-15T22:18:00.288462+0000","last_unstale":"2021-03-15T22:19:31.885484+0000","last_undegraded":"2021-03-15T22:19:31.885484+0000","last_fullsized":"2021-03-15T22:19:31.885484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.797291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.797291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500840+0000","last_change":"2021-03-15T22:19:29.179877+0000","last_active":"2021-03-15T22:19:31.500840+0000","last_peered":"2021-03-15T22:19:31.500840+0000","last_clean":"2021-03-15T22:19:31.500840+0000","last_became_active":"2021-03-15T22:17:58.015682+0000","last_became_peered":"2021-03-15T22:17:58.015682+0000","last_unstale":"2021-03-15T22:19:31.500840+0000","last_undegraded":"2021-03-15T22:19:31.500840+0000","last_fullsized":"2021-03-15T22:19:31.500840+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.179800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885817+0000","last_change":"2021-03-15T22:19:05.557588+0000","last_active":"2021-03-15T22:19:31.885817+0000","last_peered":"2021-03-15T22:19:31.885817+0000","last_clean":"2021-03-15T22:19:31.885817+0000","last_became_active":"2021-03-15T22:17:53.977470+0000","last_became_peered":"2021-03-15T22:17:53.977470+0000","last_unstale":"2021-03-15T22:19:31.885817+0000","last_undegraded":"2021-03-15T22:19:31.885817+0000","last_fullsized":"2021-03-15T22:19:31.885817+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:05.557534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:05.557534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'7","reported_seq":"48","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996676+0000","last_change":"2021-03-15T22:17:56.289796+0000","last_active":"2021-03-15T22:19:30.996676+0000","last_peered":"2021-03-15T22:19:30.996676+0000","last_clean":"2021-03-15T22:19:30.996676+0000","last_became_active":"2021-03-15T22:17:56.289454+0000","last_became_peered":"2021-03-15T22:17:56.289454+0000","last_unstale":"2021-03-15T22:19:30.996676+0000","last_undegraded":"2021-03-15T22:19:30.996676+0000","last_fullsized":"2021-03-15T22:19:30.996676+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500631+0000","last_change":"2021-03-15T22:19:11.318218+0000","last_active":"2021-03-15T22:19:31.500631+0000","last_peered":"2021-03-15T22:19:31.500631+0000","last_clean":"2021-03-15T22:19:31.500631+0000","last_became_active":"2021-03-15T22:18:00.286699+0000","last_became_peered":"2021-03-15T22:18:00.286699+0000","last_unstale":"2021-03-15T22:19:31.500631+0000","last_undegraded":"2021-03-15T22:19:31.500631+0000","last_fullsized":"2021-03-15T22:19:31.500631+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:11.318132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.318132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"47","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885799+0000","last_change":"2021-03-15T22:19:06.942939+0000","last_active":"2021-03-15T22:19:31.885799+0000","last_peered":"2021-03-15T22:19:31.885799+0000","last_clean":"2021-03-15T22:19:31.885799+0000","last_became_active":"2021-03-15T22:17:53.979246+0000","last_became_peered":"2021-03-15T22:17:53.979246+0000","last_unstale":"2021-03-15T22:19:31.885799+0000","last_undegraded":"2021-03-15T22:19:31.885799+0000","last_fullsized":"2021-03-15T22:19:31.885799+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:19:06.942870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.942870+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"72","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.240467+0000","last_change":"2021-03-15T22:19:40.240467+0000","last_active":"2021-03-15T22:19:40.240467+0000","last_peered":"2021-03-15T22:19:40.240467+0000","last_clean":"2021-03-15T22:19:40.240467+0000","last_became_active":"2021-03-15T22:17:56.289684+0000","last_became_peered":"2021-03-15T22:17:56.289684+0000","last_unstale":"2021-03-15T22:19:40.240467+0000","last_undegraded":"2021-03-15T22:19:40.240467+0000","last_fullsized":"2021-03-15T22:19:40.240467+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:19:40.240388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.240388+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"61'2","reported_seq":"58","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.164944+0000","last_change":"2021-03-15T22:17:58.015052+0000","last_active":"2021-03-15T22:19:40.164944+0000","last_peered":"2021-03-15T22:19:40.164944+0000","last_clean":"2021-03-15T22:19:40.164944+0000","last_became_active":"2021-03-15T22:17:58.013784+0000","last_became_peered":"2021-03-15T22:17:58.013784+0000","last_unstale":"2021-03-15T22:19:40.164944+0000","last_undegraded":"2021-03-15T22:19:40.164944+0000","last_fullsized":"2021-03-15T22:19:40.164944+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:37.905357+0000","last_change":"2021-03-15T22:19:37.905329+0000","last_active":"2021-03-15T22:19:37.905357+0000","last_peered":"2021-03-15T22:19:37.905357+0000","last_clean":"2021-03-15T22:19:37.905357+0000","last_became_active":"2021-03-15T22:18:00.019686+0000","last_became_peered":"2021-03-15T22:18:00.019686+0000","last_unstale":"2021-03-15T22:19:37.905357+0000","last_undegraded":"2021-03-15T22:19:37.905357+0000","last_fullsized":"2021-03-15T22:19:37.905357+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.905267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.905267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885831+0000","last_change":"2021-03-15T22:18:59.946642+0000","last_active":"2021-03-15T22:19:31.885831+0000","last_peered":"2021-03-15T22:19:31.885831+0000","last_clean":"2021-03-15T22:19:31.885831+0000","last_became_active":"2021-03-15T22:17:53.978907+0000","last_became_peered":"2021-03-15T22:17:53.978907+0000","last_unstale":"2021-03-15T22:19:31.885831+0000","last_undegraded":"2021-03-15T22:19:31.885831+0000","last_fullsized":"2021-03-15T22:19:31.885831+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:59.946561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.946561+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"66","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994629+0000","last_change":"2021-03-15T22:17:55.993082+0000","last_active":"2021-03-15T22:19:30.994629+0000","last_peered":"2021-03-15T22:19:30.994629+0000","last_clean":"2021-03-15T22:19:30.994629+0000","last_became_active":"2021-03-15T22:17:55.992386+0000","last_became_peered":"2021-03-15T22:17:55.992386+0000","last_unstale":"2021-03-15T22:19:30.994629+0000","last_undegraded":"2021-03-15T22:19:30.994629+0000","last_fullsized":"2021-03-15T22:19:30.994629+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.943950+0000","last_change":"2021-03-15T22:19:18.197284+0000","last_active":"2021-03-15T22:19:31.943950+0000","last_peered":"2021-03-15T22:19:31.943950+0000","last_clean":"2021-03-15T22:19:31.943950+0000","last_became_active":"2021-03-15T22:17:58.007916+0000","last_became_peered":"2021-03-15T22:17:58.007916+0000","last_unstale":"2021-03-15T22:19:31.943950+0000","last_undegraded":"2021-03-15T22:19:31.943950+0000","last_fullsized":"2021-03-15T22:19:31.943950+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.197224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.197224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884486+0000","last_change":"2021-03-15T22:19:09.989290+0000","last_active":"2021-03-15T22:19:31.884486+0000","last_peered":"2021-03-15T22:19:31.884486+0000","last_clean":"2021-03-15T22:19:31.884486+0000","last_became_active":"2021-03-15T22:18:00.016271+0000","last_became_peered":"2021-03-15T22:18:00.016271+0000","last_unstale":"2021-03-15T22:19:31.884486+0000","last_undegraded":"2021-03-15T22:19:31.884486+0000","last_fullsized":"2021-03-15T22:19:31.884486+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:09.989228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:09.989228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944648+0000","last_change":"2021-03-15T22:19:29.099735+0000","last_active":"2021-03-15T22:19:31.944648+0000","last_peered":"2021-03-15T22:19:31.944648+0000","last_clean":"2021-03-15T22:19:31.944648+0000","last_became_active":"2021-03-15T22:17:58.010511+0000","last_became_peered":"2021-03-15T22:17:58.010511+0000","last_unstale":"2021-03-15T22:19:31.944648+0000","last_undegraded":"2021-03-15T22:19:31.944648+0000","last_fullsized":"2021-03-15T22:19:31.944648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.099650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.099650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885676+0000","last_change":"2021-03-15T22:17:53.974424+0000","last_active":"2021-03-15T22:19:31.885676+0000","last_peered":"2021-03-15T22:19:31.885676+0000","last_clean":"2021-03-15T22:19:31.885676+0000","last_became_active":"2021-03-15T22:17:53.974108+0000","last_became_peered":"2021-03-15T22:17:53.974108+0000","last_unstale":"2021-03-15T22:19:31.885676+0000","last_undegraded":"2021-03-15T22:19:31.885676+0000","last_fullsized":"2021-03-15T22:19:31.885676+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"80","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885940+0000","last_change":"2021-03-15T22:19:22.700145+0000","last_active":"2021-03-15T22:19:31.885940+0000","last_peered":"2021-03-15T22:19:31.885940+0000","last_clean":"2021-03-15T22:19:31.885940+0000","last_became_active":"2021-03-15T22:17:56.288870+0000","last_became_peered":"2021-03-15T22:17:56.288870+0000","last_unstale":"2021-03-15T22:19:31.885940+0000","last_undegraded":"2021-03-15T22:19:31.885940+0000","last_fullsized":"2021-03-15T22:19:31.885940+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:19:22.700049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.700049+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:39.681830+0000","last_change":"2021-03-15T22:19:39.681802+0000","last_active":"2021-03-15T22:19:39.681830+0000","last_peered":"2021-03-15T22:19:39.681830+0000","last_clean":"2021-03-15T22:19:39.681830+0000","last_became_active":"2021-03-15T22:18:00.016982+0000","last_became_peered":"2021-03-15T22:18:00.016982+0000","last_unstale":"2021-03-15T22:19:39.681830+0000","last_undegraded":"2021-03-15T22:19:39.681830+0000","last_fullsized":"2021-03-15T22:19:39.681830+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:39.681741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:39.681741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944246+0000","last_change":"2021-03-15T22:19:01.036104+0000","last_active":"2021-03-15T22:19:31.944246+0000","last_peered":"2021-03-15T22:19:31.944246+0000","last_clean":"2021-03-15T22:19:31.944246+0000","last_became_active":"2021-03-15T22:17:58.001399+0000","last_became_peered":"2021-03-15T22:17:58.001399+0000","last_unstale":"2021-03-15T22:19:31.944246+0000","last_undegraded":"2021-03-15T22:19:31.944246+0000","last_fullsized":"2021-03-15T22:19:31.944246+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:01.036038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:01.036038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500687+0000","last_change":"2021-03-15T22:19:18.252532+0000","last_active":"2021-03-15T22:19:31.500687+0000","last_peered":"2021-03-15T22:19:31.500687+0000","last_clean":"2021-03-15T22:19:31.500687+0000","last_became_active":"2021-03-15T22:17:53.972006+0000","last_became_peered":"2021-03-15T22:17:53.972006+0000","last_unstale":"2021-03-15T22:19:31.500687+0000","last_undegraded":"2021-03-15T22:19:31.500687+0000","last_fullsized":"2021-03-15T22:19:31.500687+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.252456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.252456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"96","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996043+0000","last_change":"2021-03-15T22:19:28.581985+0000","last_active":"2021-03-15T22:19:30.996043+0000","last_peered":"2021-03-15T22:19:30.996043+0000","last_clean":"2021-03-15T22:19:30.996043+0000","last_became_active":"2021-03-15T22:19:02.093463+0000","last_became_peered":"2021-03-15T22:19:02.093463+0000","last_unstale":"2021-03-15T22:19:30.996043+0000","last_undegraded":"2021-03-15T22:19:30.996043+0000","last_fullsized":"2021-03-15T22:19:30.996043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:19:28.581920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.581920+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.997093+0000","last_change":"2021-03-15T22:19:19.908283+0000","last_active":"2021-03-15T22:19:30.997093+0000","last_peered":"2021-03-15T22:19:30.997093+0000","last_clean":"2021-03-15T22:19:30.997093+0000","last_became_active":"2021-03-15T22:18:00.020501+0000","last_became_peered":"2021-03-15T22:18:00.020501+0000","last_unstale":"2021-03-15T22:19:30.997093+0000","last_undegraded":"2021-03-15T22:19:30.997093+0000","last_fullsized":"2021-03-15T22:19:30.997093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.908218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.908218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"45","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500320+0000","last_change":"2021-03-15T22:19:08.268784+0000","last_active":"2021-03-15T22:19:31.500320+0000","last_peered":"2021-03-15T22:19:31.500320+0000","last_clean":"2021-03-15T22:19:31.500320+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:19:31.500320+0000","last_undegraded":"2021-03-15T22:19:31.500320+0000","last_fullsized":"2021-03-15T22:19:31.500320+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:19:08.268707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995028+0000","last_change":"2021-03-15T22:17:53.978932+0000","last_active":"2021-03-15T22:19:30.995028+0000","last_peered":"2021-03-15T22:19:30.995028+0000","last_clean":"2021-03-15T22:19:30.995028+0000","last_became_active":"2021-03-15T22:17:53.978521+0000","last_became_peered":"2021-03-15T22:17:53.978521+0000","last_unstale":"2021-03-15T22:19:30.995028+0000","last_undegraded":"2021-03-15T22:19:30.995028+0000","last_fullsized":"2021-03-15T22:19:30.995028+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"102","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944627+0000","last_change":"2021-03-15T22:19:17.650024+0000","last_active":"2021-03-15T22:19:31.944627+0000","last_peered":"2021-03-15T22:19:31.944627+0000","last_clean":"2021-03-15T22:19:31.944627+0000","last_became_active":"2021-03-15T22:17:55.995319+0000","last_became_peered":"2021-03-15T22:17:55.995319+0000","last_unstale":"2021-03-15T22:19:31.944627+0000","last_undegraded":"2021-03-15T22:19:31.944627+0000","last_fullsized":"2021-03-15T22:19:31.944627+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.649963+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995777+0000","last_change":"2021-03-15T22:19:09.821008+0000","last_active":"2021-03-15T22:19:30.995777+0000","last_peered":"2021-03-15T22:19:30.995777+0000","last_clean":"2021-03-15T22:19:30.995777+0000","last_became_active":"2021-03-15T22:17:58.017434+0000","last_became_peered":"2021-03-15T22:17:58.017434+0000","last_unstale":"2021-03-15T22:19:30.995777+0000","last_undegraded":"2021-03-15T22:19:30.995777+0000","last_fullsized":"2021-03-15T22:19:30.995777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:09.820947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:09.820947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885944+0000","last_change":"2021-03-15T22:19:31.885077+0000","last_active":"2021-03-15T22:19:31.885944+0000","last_peered":"2021-03-15T22:19:31.885944+0000","last_clean":"2021-03-15T22:19:31.885944+0000","last_became_active":"2021-03-15T22:18:00.287780+0000","last_became_peered":"2021-03-15T22:18:00.287780+0000","last_unstale":"2021-03-15T22:19:31.885944+0000","last_undegraded":"2021-03-15T22:19:31.885944+0000","last_fullsized":"2021-03-15T22:19:31.885944+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:31.884965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:31.884965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944218+0000","last_change":"2021-03-15T22:19:05.537563+0000","last_active":"2021-03-15T22:19:31.944218+0000","last_peered":"2021-03-15T22:19:31.944218+0000","last_clean":"2021-03-15T22:19:31.944218+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:19:31.944218+0000","last_undegraded":"2021-03-15T22:19:31.944218+0000","last_fullsized":"2021-03-15T22:19:31.944218+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:05.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:05.537497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"47","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944304+0000","last_change":"2021-03-15T22:19:24.142618+0000","last_active":"2021-03-15T22:19:31.944304+0000","last_peered":"2021-03-15T22:19:31.944304+0000","last_clean":"2021-03-15T22:19:31.944304+0000","last_became_active":"2021-03-15T22:17:51.631460+0000","last_became_peered":"2021-03-15T22:17:51.631460+0000","last_unstale":"2021-03-15T22:19:31.944304+0000","last_undegraded":"2021-03-15T22:19:31.944304+0000","last_fullsized":"2021-03-15T22:19:31.944304+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.142558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.142558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'15","reported_seq":"73","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944776+0000","last_change":"2021-03-15T22:19:24.648256+0000","last_active":"2021-03-15T22:19:31.944776+0000","last_peered":"2021-03-15T22:19:31.944776+0000","last_clean":"2021-03-15T22:19:31.944776+0000","last_became_active":"2021-03-15T22:17:55.992952+0000","last_became_peered":"2021-03-15T22:17:55.992952+0000","last_unstale":"2021-03-15T22:19:31.944776+0000","last_undegraded":"2021-03-15T22:19:31.944776+0000","last_fullsized":"2021-03-15T22:19:31.944776+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:24.648200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.648200+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944759+0000","last_change":"2021-03-15T22:19:15.249150+0000","last_active":"2021-03-15T22:19:31.944759+0000","last_peered":"2021-03-15T22:19:31.944759+0000","last_clean":"2021-03-15T22:19:31.944759+0000","last_became_active":"2021-03-15T22:17:58.014849+0000","last_became_peered":"2021-03-15T22:17:58.014849+0000","last_unstale":"2021-03-15T22:19:31.944759+0000","last_undegraded":"2021-03-15T22:19:31.944759+0000","last_fullsized":"2021-03-15T22:19:31.944759+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:15.249076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.249076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500903+0000","last_change":"2021-03-15T22:19:25.184338+0000","last_active":"2021-03-15T22:19:31.500903+0000","last_peered":"2021-03-15T22:19:31.500903+0000","last_clean":"2021-03-15T22:19:31.500903+0000","last_became_active":"2021-03-15T22:18:00.019363+0000","last_became_peered":"2021-03-15T22:18:00.019363+0000","last_unstale":"2021-03-15T22:19:31.500903+0000","last_undegraded":"2021-03-15T22:19:31.500903+0000","last_fullsized":"2021-03-15T22:19:31.500903+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.184260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.184260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"47","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884748+0000","last_change":"2021-03-15T22:19:23.843965+0000","last_active":"2021-03-15T22:19:31.884748+0000","last_peered":"2021-03-15T22:19:31.884748+0000","last_clean":"2021-03-15T22:19:31.884748+0000","last_became_active":"2021-03-15T22:17:51.044140+0000","last_became_peered":"2021-03-15T22:17:51.044140+0000","last_unstale":"2021-03-15T22:19:31.884748+0000","last_undegraded":"2021-03-15T22:19:31.884748+0000","last_fullsized":"2021-03-15T22:19:31.884748+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:23.843892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.843892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.997068+0000","last_change":"2021-03-15T22:19:20.880709+0000","last_active":"2021-03-15T22:19:30.997068+0000","last_peered":"2021-03-15T22:19:30.997068+0000","last_clean":"2021-03-15T22:19:30.997068+0000","last_became_active":"2021-03-15T22:17:53.972080+0000","last_became_peered":"2021-03-15T22:17:53.972080+0000","last_unstale":"2021-03-15T22:19:30.997068+0000","last_undegraded":"2021-03-15T22:19:30.997068+0000","last_fullsized":"2021-03-15T22:19:30.997068+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:20.880638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:20.880638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"65","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.886436+0000","last_change":"2021-03-15T22:19:23.732253+0000","last_active":"2021-03-15T22:19:31.886436+0000","last_peered":"2021-03-15T22:19:31.886436+0000","last_clean":"2021-03-15T22:19:31.886436+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:19:31.886436+0000","last_undegraded":"2021-03-15T22:19:31.886436+0000","last_fullsized":"2021-03-15T22:19:31.886436+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:23.732187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.732187+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885734+0000","last_change":"2021-03-15T22:19:07.981284+0000","last_active":"2021-03-15T22:19:31.885734+0000","last_peered":"2021-03-15T22:19:31.885734+0000","last_clean":"2021-03-15T22:19:31.885734+0000","last_became_active":"2021-03-15T22:17:58.011911+0000","last_became_peered":"2021-03-15T22:17:58.011911+0000","last_unstale":"2021-03-15T22:19:31.885734+0000","last_undegraded":"2021-03-15T22:19:31.885734+0000","last_fullsized":"2021-03-15T22:19:31.885734+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.981223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944334+0000","last_change":"2021-03-15T22:19:28.773808+0000","last_active":"2021-03-15T22:19:31.944334+0000","last_peered":"2021-03-15T22:19:31.944334+0000","last_clean":"2021-03-15T22:19:31.944334+0000","last_became_active":"2021-03-15T22:18:00.014336+0000","last_became_peered":"2021-03-15T22:18:00.014336+0000","last_unstale":"2021-03-15T22:19:31.944334+0000","last_undegraded":"2021-03-15T22:19:31.944334+0000","last_fullsized":"2021-03-15T22:19:31.944334+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.773747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.773747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:38.721626+0000","last_change":"2021-03-15T22:19:38.721599+0000","last_active":"2021-03-15T22:19:38.721626+0000","last_peered":"2021-03-15T22:19:38.721626+0000","last_clean":"2021-03-15T22:19:38.721626+0000","last_became_active":"2021-03-15T22:17:53.977311+0000","last_became_peered":"2021-03-15T22:17:53.977311+0000","last_unstale":"2021-03-15T22:19:38.721626+0000","last_undegraded":"2021-03-15T22:19:38.721626+0000","last_fullsized":"2021-03-15T22:19:38.721626+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:38.721534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.721534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"159","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:42.436120+0000","last_change":"2021-03-15T22:19:03.966872+0000","last_active":"2021-03-15T22:19:42.436120+0000","last_peered":"2021-03-15T22:19:42.436120+0000","last_clean":"2021-03-15T22:19:42.436120+0000","last_became_active":"2021-03-15T22:17:51.044139+0000","last_became_peered":"2021-03-15T22:17:51.044139+0000","last_unstale":"2021-03-15T22:19:42.436120+0000","last_undegraded":"2021-03-15T22:19:42.436120+0000","last_fullsized":"2021-03-15T22:19:42.436120+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:19:03.966803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:19:03.966803+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":110,"num_read_kb":105,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"67","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996188+0000","last_change":"2021-03-15T22:19:30.589143+0000","last_active":"2021-03-15T22:19:30.996188+0000","last_peered":"2021-03-15T22:19:30.996188+0000","last_clean":"2021-03-15T22:19:30.996188+0000","last_became_active":"2021-03-15T22:17:55.999564+0000","last_became_peered":"2021-03-15T22:17:55.999564+0000","last_unstale":"2021-03-15T22:19:30.996188+0000","last_undegraded":"2021-03-15T22:19:30.996188+0000","last_fullsized":"2021-03-15T22:19:30.996188+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:30.589062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.589062+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.501056+0000","last_change":"2021-03-15T22:19:00.305911+0000","last_active":"2021-03-15T22:19:31.501056+0000","last_peered":"2021-03-15T22:19:31.501056+0000","last_clean":"2021-03-15T22:19:31.501056+0000","last_became_active":"2021-03-15T22:17:58.010798+0000","last_became_peered":"2021-03-15T22:17:58.010798+0000","last_unstale":"2021-03-15T22:19:31.501056+0000","last_undegraded":"2021-03-15T22:19:31.501056+0000","last_fullsized":"2021-03-15T22:19:31.501056+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:00.305805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:00.305805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.943978+0000","last_change":"2021-03-15T22:19:15.593372+0000","last_active":"2021-03-15T22:19:31.943978+0000","last_peered":"2021-03-15T22:19:31.943978+0000","last_clean":"2021-03-15T22:19:31.943978+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:19:31.943978+0000","last_undegraded":"2021-03-15T22:19:31.943978+0000","last_fullsized":"2021-03-15T22:19:31.943978+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.593307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944530+0000","last_change":"2021-03-15T22:19:29.812510+0000","last_active":"2021-03-15T22:19:31.944530+0000","last_peered":"2021-03-15T22:19:31.944530+0000","last_clean":"2021-03-15T22:19:31.944530+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:19:31.944530+0000","last_undegraded":"2021-03-15T22:19:31.944530+0000","last_fullsized":"2021-03-15T22:19:31.944530+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.812446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.812446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"91","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996275+0000","last_change":"2021-03-15T22:18:56.832718+0000","last_active":"2021-03-15T22:19:30.996275+0000","last_peered":"2021-03-15T22:19:30.996275+0000","last_clean":"2021-03-15T22:19:30.996275+0000","last_became_active":"2021-03-15T22:17:56.288438+0000","last_became_peered":"2021-03-15T22:17:56.288438+0000","last_unstale":"2021-03-15T22:19:30.996275+0000","last_undegraded":"2021-03-15T22:19:30.996275+0000","last_fullsized":"2021-03-15T22:19:30.996275+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:18:56.832638+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.886070+0000","last_change":"2021-03-15T22:19:16.581584+0000","last_active":"2021-03-15T22:19:31.886070+0000","last_peered":"2021-03-15T22:19:31.886070+0000","last_clean":"2021-03-15T22:19:31.886070+0000","last_became_active":"2021-03-15T22:17:58.008731+0000","last_became_peered":"2021-03-15T22:17:58.008731+0000","last_unstale":"2021-03-15T22:19:31.886070+0000","last_undegraded":"2021-03-15T22:19:31.886070+0000","last_fullsized":"2021-03-15T22:19:31.886070+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.581514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996000+0000","last_change":"2021-03-15T22:19:19.751342+0000","last_active":"2021-03-15T22:19:30.996000+0000","last_peered":"2021-03-15T22:19:30.996000+0000","last_clean":"2021-03-15T22:19:30.996000+0000","last_became_active":"2021-03-15T22:18:00.015069+0000","last_became_peered":"2021-03-15T22:18:00.015069+0000","last_unstale":"2021-03-15T22:19:30.996000+0000","last_undegraded":"2021-03-15T22:19:30.996000+0000","last_fullsized":"2021-03-15T22:19:30.996000+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.751282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.751282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"43","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:41.212619+0000","last_change":"2021-03-15T22:19:41.212619+0000","last_active":"2021-03-15T22:19:41.212619+0000","last_peered":"2021-03-15T22:19:41.212619+0000","last_clean":"2021-03-15T22:19:41.212619+0000","last_became_active":"2021-03-15T22:17:54.284266+0000","last_became_peered":"2021-03-15T22:17:54.284266+0000","last_unstale":"2021-03-15T22:19:41.212619+0000","last_undegraded":"2021-03-15T22:19:41.212619+0000","last_fullsized":"2021-03-15T22:19:41.212619+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:41.212539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:41.212539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"66","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944471+0000","last_change":"2021-03-15T22:19:02.550860+0000","last_active":"2021-03-15T22:19:31.944471+0000","last_peered":"2021-03-15T22:19:31.944471+0000","last_clean":"2021-03-15T22:19:31.944471+0000","last_became_active":"2021-03-15T22:17:55.992257+0000","last_became_peered":"2021-03-15T22:17:55.992257+0000","last_unstale":"2021-03-15T22:19:31.944471+0000","last_undegraded":"2021-03-15T22:19:31.944471+0000","last_fullsized":"2021-03-15T22:19:31.944471+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"61'2","reported_seq":"62","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.659726+0000","last_change":"2021-03-15T22:19:25.604285+0000","last_active":"2021-03-15T22:19:40.659726+0000","last_peered":"2021-03-15T22:19:40.659726+0000","last_clean":"2021-03-15T22:19:40.659726+0000","last_became_active":"2021-03-15T22:17:58.008973+0000","last_became_peered":"2021-03-15T22:17:58.008973+0000","last_unstale":"2021-03-15T22:19:40.659726+0000","last_undegraded":"2021-03-15T22:19:40.659726+0000","last_fullsized":"2021-03-15T22:19:40.659726+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:25.604217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.604217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885406+0000","last_change":"2021-03-15T22:19:18.783378+0000","last_active":"2021-03-15T22:19:31.885406+0000","last_peered":"2021-03-15T22:19:31.885406+0000","last_clean":"2021-03-15T22:19:31.885406+0000","last_became_active":"2021-03-15T22:18:00.288441+0000","last_became_peered":"2021-03-15T22:18:00.288441+0000","last_unstale":"2021-03-15T22:19:31.885406+0000","last_undegraded":"2021-03-15T22:19:31.885406+0000","last_fullsized":"2021-03-15T22:19:31.885406+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.783297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.783297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996230+0000","last_change":"2021-03-15T22:19:13.870267+0000","last_active":"2021-03-15T22:19:30.996230+0000","last_peered":"2021-03-15T22:19:30.996230+0000","last_clean":"2021-03-15T22:19:30.996230+0000","last_became_active":"2021-03-15T22:17:53.976307+0000","last_became_peered":"2021-03-15T22:17:53.976307+0000","last_unstale":"2021-03-15T22:19:30.996230+0000","last_undegraded":"2021-03-15T22:19:30.996230+0000","last_fullsized":"2021-03-15T22:19:30.996230+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:13.870202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.870202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"75","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994627+0000","last_change":"2021-03-15T22:19:26.087704+0000","last_active":"2021-03-15T22:19:30.994627+0000","last_peered":"2021-03-15T22:19:30.994627+0000","last_clean":"2021-03-15T22:19:30.994627+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:19:30.994627+0000","last_undegraded":"2021-03-15T22:19:30.994627+0000","last_fullsized":"2021-03-15T22:19:30.994627+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:19:26.087622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.087622+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"61'2","reported_seq":"58","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.660942+0000","last_change":"2021-03-15T22:17:58.011617+0000","last_active":"2021-03-15T22:19:40.660942+0000","last_peered":"2021-03-15T22:19:40.660942+0000","last_clean":"2021-03-15T22:19:40.660942+0000","last_became_active":"2021-03-15T22:17:58.011279+0000","last_became_peered":"2021-03-15T22:17:58.011279+0000","last_unstale":"2021-03-15T22:19:40.660942+0000","last_undegraded":"2021-03-15T22:19:40.660942+0000","last_fullsized":"2021-03-15T22:19:40.660942+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:34.179549+0000","last_change":"2021-03-15T22:19:34.179513+0000","last_active":"2021-03-15T22:19:34.179549+0000","last_peered":"2021-03-15T22:19:34.179549+0000","last_clean":"2021-03-15T22:19:34.179549+0000","last_became_active":"2021-03-15T22:18:00.021088+0000","last_became_peered":"2021-03-15T22:18:00.021088+0000","last_unstale":"2021-03-15T22:19:34.179549+0000","last_undegraded":"2021-03-15T22:19:34.179549+0000","last_fullsized":"2021-03-15T22:19:34.179549+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:34.179433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:34.179433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996435+0000","last_change":"2021-03-15T22:18:59.667448+0000","last_active":"2021-03-15T22:19:30.996435+0000","last_peered":"2021-03-15T22:19:30.996435+0000","last_clean":"2021-03-15T22:19:30.996435+0000","last_became_active":"2021-03-15T22:17:53.970585+0000","last_became_peered":"2021-03-15T22:17:53.970585+0000","last_unstale":"2021-03-15T22:19:30.996435+0000","last_undegraded":"2021-03-15T22:19:30.996435+0000","last_fullsized":"2021-03-15T22:19:30.996435+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:59.667388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.667388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"94","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996578+0000","last_change":"2021-03-15T22:19:04.746329+0000","last_active":"2021-03-15T22:19:30.996578+0000","last_peered":"2021-03-15T22:19:30.996578+0000","last_clean":"2021-03-15T22:19:30.996578+0000","last_became_active":"2021-03-15T22:17:56.288319+0000","last_became_peered":"2021-03-15T22:17:56.288319+0000","last_unstale":"2021-03-15T22:19:30.996578+0000","last_undegraded":"2021-03-15T22:19:30.996578+0000","last_fullsized":"2021-03-15T22:19:30.996578+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:19:04.746224+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"35","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994437+0000","last_change":"2021-03-15T22:17:58.007412+0000","last_active":"2021-03-15T22:19:30.994437+0000","last_peered":"2021-03-15T22:19:30.994437+0000","last_clean":"2021-03-15T22:19:30.994437+0000","last_became_active":"2021-03-15T22:17:58.006706+0000","last_became_peered":"2021-03-15T22:17:58.006706+0000","last_unstale":"2021-03-15T22:19:30.994437+0000","last_undegraded":"2021-03-15T22:19:30.994437+0000","last_fullsized":"2021-03-15T22:19:30.994437+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"37","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:42.258910+0000","last_change":"2021-03-15T22:19:42.258910+0000","last_active":"2021-03-15T22:19:42.258910+0000","last_peered":"2021-03-15T22:19:42.258910+0000","last_clean":"2021-03-15T22:19:42.258910+0000","last_became_active":"2021-03-15T22:18:00.019457+0000","last_became_peered":"2021-03-15T22:18:00.019457+0000","last_unstale":"2021-03-15T22:19:42.258910+0000","last_undegraded":"2021-03-15T22:19:42.258910+0000","last_fullsized":"2021-03-15T22:19:42.258910+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:42.258829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:42.258829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"61'2","reported_seq":"57","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:37.025277+0000","last_change":"2021-03-15T22:17:58.010787+0000","last_active":"2021-03-15T22:19:37.025277+0000","last_peered":"2021-03-15T22:19:37.025277+0000","last_clean":"2021-03-15T22:19:37.025277+0000","last_became_active":"2021-03-15T22:17:58.010485+0000","last_became_peered":"2021-03-15T22:17:58.010485+0000","last_unstale":"2021-03-15T22:19:37.025277+0000","last_undegraded":"2021-03-15T22:19:37.025277+0000","last_fullsized":"2021-03-15T22:19:37.025277+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996494+0000","last_change":"2021-03-15T22:19:24.902865+0000","last_active":"2021-03-15T22:19:30.996494+0000","last_peered":"2021-03-15T22:19:30.996494+0000","last_clean":"2021-03-15T22:19:30.996494+0000","last_became_active":"2021-03-15T22:17:53.967568+0000","last_became_peered":"2021-03-15T22:17:53.967568+0000","last_unstale":"2021-03-15T22:19:30.996494+0000","last_undegraded":"2021-03-15T22:19:30.996494+0000","last_fullsized":"2021-03-15T22:19:30.996494+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.902804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.902804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.945166+0000","last_change":"2021-03-15T22:19:10.132572+0000","last_active":"2021-03-15T22:19:31.945166+0000","last_peered":"2021-03-15T22:19:31.945166+0000","last_clean":"2021-03-15T22:19:31.945166+0000","last_became_active":"2021-03-15T22:19:02.092703+0000","last_became_peered":"2021-03-15T22:19:02.092703+0000","last_unstale":"2021-03-15T22:19:31.945166+0000","last_undegraded":"2021-03-15T22:19:31.945166+0000","last_fullsized":"2021-03-15T22:19:31.945166+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:19:10.132514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:10.132514+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.997126+0000","last_change":"2021-03-15T22:19:25.878071+0000","last_active":"2021-03-15T22:19:30.997126+0000","last_peered":"2021-03-15T22:19:30.997126+0000","last_clean":"2021-03-15T22:19:30.997126+0000","last_became_active":"2021-03-15T22:18:00.287019+0000","last_became_peered":"2021-03-15T22:18:00.287019+0000","last_unstale":"2021-03-15T22:19:30.997126+0000","last_undegraded":"2021-03-15T22:19:30.997126+0000","last_fullsized":"2021-03-15T22:19:30.997126+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.878008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.501010+0000","last_change":"2021-03-15T22:18:58.347698+0000","last_active":"2021-03-15T22:19:31.501010+0000","last_peered":"2021-03-15T22:19:31.501010+0000","last_clean":"2021-03-15T22:19:31.501010+0000","last_became_active":"2021-03-15T22:17:58.008735+0000","last_became_peered":"2021-03-15T22:17:58.008735+0000","last_unstale":"2021-03-15T22:19:31.501010+0000","last_undegraded":"2021-03-15T22:19:31.501010+0000","last_fullsized":"2021-03-15T22:19:31.501010+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:58.347616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:18:58.347616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994474+0000","last_change":"2021-03-15T22:17:54.285942+0000","last_active":"2021-03-15T22:19:30.994474+0000","last_peered":"2021-03-15T22:19:30.994474+0000","last_clean":"2021-03-15T22:19:30.994474+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:19:30.994474+0000","last_undegraded":"2021-03-15T22:19:30.994474+0000","last_fullsized":"2021-03-15T22:19:30.994474+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"64","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995308+0000","last_change":"2021-03-15T22:17:55.992532+0000","last_active":"2021-03-15T22:19:30.995308+0000","last_peered":"2021-03-15T22:19:30.995308+0000","last_clean":"2021-03-15T22:19:30.995308+0000","last_became_active":"2021-03-15T22:17:55.992213+0000","last_became_peered":"2021-03-15T22:17:55.992213+0000","last_unstale":"2021-03-15T22:19:30.995308+0000","last_undegraded":"2021-03-15T22:19:30.995308+0000","last_fullsized":"2021-03-15T22:19:30.995308+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.945270+0000","last_change":"2021-03-15T22:19:22.172668+0000","last_active":"2021-03-15T22:19:31.945270+0000","last_peered":"2021-03-15T22:19:31.945270+0000","last_clean":"2021-03-15T22:19:31.945270+0000","last_became_active":"2021-03-15T22:18:00.019205+0000","last_became_peered":"2021-03-15T22:18:00.019205+0000","last_unstale":"2021-03-15T22:19:31.945270+0000","last_undegraded":"2021-03-15T22:19:31.945270+0000","last_fullsized":"2021-03-15T22:19:31.945270+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.172605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944782+0000","last_change":"2021-03-15T22:19:13.514504+0000","last_active":"2021-03-15T22:19:31.944782+0000","last_peered":"2021-03-15T22:19:31.944782+0000","last_clean":"2021-03-15T22:19:31.944782+0000","last_became_active":"2021-03-15T22:17:58.013512+0000","last_became_peered":"2021-03-15T22:17:58.013512+0000","last_unstale":"2021-03-15T22:19:31.944782+0000","last_undegraded":"2021-03-15T22:19:31.944782+0000","last_fullsized":"2021-03-15T22:19:31.944782+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:13.514438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.514438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885774+0000","last_change":"2021-03-15T22:19:18.591404+0000","last_active":"2021-03-15T22:19:31.885774+0000","last_peered":"2021-03-15T22:19:31.885774+0000","last_clean":"2021-03-15T22:19:31.885774+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:19:31.885774+0000","last_undegraded":"2021-03-15T22:19:31.885774+0000","last_fullsized":"2021-03-15T22:19:31.885774+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.591331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.591331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"65'21","reported_seq":"73","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995325+0000","last_change":"2021-03-15T22:17:55.992521+0000","last_active":"2021-03-15T22:19:30.995325+0000","last_peered":"2021-03-15T22:19:30.995325+0000","last_clean":"2021-03-15T22:19:30.995325+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:19:30.995325+0000","last_undegraded":"2021-03-15T22:19:30.995325+0000","last_fullsized":"2021-03-15T22:19:30.995325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"33","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995424+0000","last_change":"2021-03-15T22:18:00.019688+0000","last_active":"2021-03-15T22:19:30.995424+0000","last_peered":"2021-03-15T22:19:30.995424+0000","last_clean":"2021-03-15T22:19:30.995424+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:19:30.995424+0000","last_undegraded":"2021-03-15T22:19:30.995424+0000","last_fullsized":"2021-03-15T22:19:30.995424+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"61'2","reported_seq":"63","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.042461+0000","last_change":"2021-03-15T22:19:12.158924+0000","last_active":"2021-03-15T22:19:40.042461+0000","last_peered":"2021-03-15T22:19:40.042461+0000","last_clean":"2021-03-15T22:19:40.042461+0000","last_became_active":"2021-03-15T22:17:58.010539+0000","last_became_peered":"2021-03-15T22:17:58.010539+0000","last_unstale":"2021-03-15T22:19:40.042461+0000","last_undegraded":"2021-03-15T22:19:40.042461+0000","last_fullsized":"2021-03-15T22:19:40.042461+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:12.158857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:12.158857+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996941+0000","last_change":"2021-03-15T22:18:57.650089+0000","last_active":"2021-03-15T22:19:30.996941+0000","last_peered":"2021-03-15T22:19:30.996941+0000","last_clean":"2021-03-15T22:19:30.996941+0000","last_became_active":"2021-03-15T22:17:53.976045+0000","last_became_peered":"2021-03-15T22:17:53.976045+0000","last_unstale":"2021-03-15T22:19:30.996941+0000","last_undegraded":"2021-03-15T22:19:30.996941+0000","last_fullsized":"2021-03-15T22:19:30.996941+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:57.650003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:57.650003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"58","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994884+0000","last_change":"2021-03-15T22:17:55.986249+0000","last_active":"2021-03-15T22:19:30.994884+0000","last_peered":"2021-03-15T22:19:30.994884+0000","last_clean":"2021-03-15T22:19:30.994884+0000","last_became_active":"2021-03-15T22:17:55.986062+0000","last_became_peered":"2021-03-15T22:17:55.986062+0000","last_unstale":"2021-03-15T22:19:30.994884+0000","last_undegraded":"2021-03-15T22:19:30.994884+0000","last_fullsized":"2021-03-15T22:19:30.994884+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16,"ondisk_log_size":16,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1069,"num_read_kb":574,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":556,"ondisk_log_size":556,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":110,"num_read_kb":105,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299418,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6452,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738700,"statfs":{"total":95995035648,"available":95988428800,"internally_reserved":0,"allocated":897024,"data_stored":368178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.55400000000000005}]},{"osd":1,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.58699999999999997}]},{"osd":3,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.63700000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.80100000000000005},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.61299999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.68500000000000005},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.77000000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.66300000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528350,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6484,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738668,"statfs":{"total":95995035648,"available":95988396032,"internally_reserved":0,"allocated":929792,"data_stored":368758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48899999999999999,"15min":0.49299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63200000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67500000000000004,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47699999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.36099999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.48199999999999998,"15min":0.47799999999999998},"min":{"1min":0.40400000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.629,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.377}]},{"osd":2,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.501,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.68600000000000005,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.502,"15min":0.502},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.624,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.624,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69499999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.39300000000000002},"min":{"1min":0.30199999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.433,"15min":0.42699999999999999},"min":{"1min":0.32100000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.441,"15min":0.433},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.498,"5min":0.46500000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.621}]},{"osd":7,"last update":"Mon Mar 15 22:18:52 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.59799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574895,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7100,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":6080,"kb_avail":93738052,"statfs":{"total":95995035648,"available":95987765248,"internally_reserved":0,"allocated":1036288,"data_stored":373154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6225920},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:13 2021","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38400000000000001,"15min":0.35099999999999998},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66400000000000003,"5min":3.387,"15min":3.387},"last":0.439},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.377},"min":{"1min":0.23699999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.71899999999999997,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:19:36 2021","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.42199999999999999,"15min":0.41699999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.78700000000000003,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41699999999999998,"15min":0.42099999999999999},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64400000000000002,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.42699999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:18:42 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.45100000000000001,"15min":0.443},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":1.796,"5min":1.796,"15min":1.796},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.505,"15min":0.502},"min":{"1min":0.318,"5min":0.317,"15min":0.317},"max":{"1min":1.7150000000000001,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.47799999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:19:00 2021","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.628},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.65300000000000002,"15min":0.63500000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":4.3499999999999996,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.71599999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:19:16 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.65200000000000002,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.94199999999999995,"5min":2.153,"15min":2.153},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.151,"15min":1.2450000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95099999999999996,"5min":15.472,"15min":15.472},"last":0.70099999999999996}]},{"osd":6,"last update":"Mon Mar 15 22:19:36 2021","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.67700000000000005,"15min":0.69399999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.879,"5min":1.75,"15min":1.75},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.64100000000000001,"15min":0.65300000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.87,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.76600000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:18:53 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":8.8100000000000005,"5min":8.8100000000000005,"15min":8.8100000000000005},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.72999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6776,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738376,"statfs":{"total":95995035648,"available":95988097024,"internally_reserved":0,"allocated":901120,"data_stored":368971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:19:12 2021","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.253,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.38200000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59299999999999997,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.28199999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:19:29 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.39700000000000002,"15min":0.40999999999999998},"min":{"1min":0.24199999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.42899999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.39200000000000002,"15min":0.40400000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.442,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.247}]},{"osd":3,"last update":"Mon Mar 15 22:18:48 2021","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.432,"15min":0.442},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54700000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.442,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.53900000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.317}]},{"osd":4,"last update":"Mon Mar 15 22:19:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73599999999999999,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60499999999999998,"15min":0.61499999999999999},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72099999999999997,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:19:18 2021","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70199999999999996,"5min":0.871,"15min":0.871},"last":0.51900000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:19:31 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.437},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.53300000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:18:52 2021","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.45600000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6520,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738632,"statfs":{"total":95995035648,"available":95988359168,"internally_reserved":0,"allocated":835584,"data_stored":366396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.35399999999999998,"15min":0.35999999999999999},"min":{"1min":0.20799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.377,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.191,"15min":0.191},"max":{"1min":0.53600000000000003,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.27000000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.38800000000000001},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55200000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.50700000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:18:44 2021","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.501,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.55800000000000005,"5min":0.874,"15min":0.874},"last":0.314}]},{"osd":4,"last update":"Mon Mar 15 22:18:59 2021","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.69199999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.56100000000000005,"15min":0.56899999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73399999999999999,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.498}]},{"osd":5,"last update":"Mon Mar 15 22:19:19 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69799999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.78000000000000003,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.56699999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.77300000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.52300000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6872,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738280,"statfs":{"total":95995035648,"available":95987998720,"internally_reserved":0,"allocated":999424,"data_stored":371551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.36199999999999999,"15min":0.371},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.42499999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.38700000000000001,"15min":0.39700000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.47899999999999998,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.27800000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42599999999999999,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52300000000000002,"5min":0.621,"15min":0.621},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.42199999999999999,"15min":0.436},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.42699999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.42699999999999999,"15min":0.44},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.49099999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39700000000000002,"15min":0.40500000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54100000000000004,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.27000000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:19:01 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.66400000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63800000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.621},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73899999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.53400000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:19:33 2021","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58499999999999996,"15min":0.58999999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.86699999999999999,"5min":0.88,"15min":0.88},"last":0.56699999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.69399999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986212,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6768,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738384,"statfs":{"total":95995035648,"available":95988105216,"internally_reserved":0,"allocated":958464,"data_stored":370199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68600000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.497},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.376}]},{"osd":1,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51100000000000001,"15min":0.52100000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66000000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.439},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63400000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.45000000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.64300000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.499}]},{"osd":3,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.67500000000000004,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.502,"5min":0.57499999999999996,"15min":0.58699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68100000000000005,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.47299999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:19:19 2021","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.40699999999999997,"15min":0.40200000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.56399999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.625},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.61399999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:18:49 2021","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.67900000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757281,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6384,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":827392,"data_stored":365165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47799999999999998,"15min":0.47899999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59299999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.755,"15min":0.755},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73399999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.48899999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.52000000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69999999999999996,"5min":0.745,"15min":0.745},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.61799999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.376,"15min":0.36399999999999999},"min":{"1min":0.30099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.314,"5min":0.25,"15min":0.25},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.42199999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:19:35 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.44900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64000000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.40400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70199999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.65300000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.57899999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":12926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":5792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:19:48.411 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:19:48.693 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:48.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:48 smithi076 conmon[56514]: audit 2021-03-15T22:19:47.753195+0000 mgr.y (mgr.14144) 2021-03-15T22:19:48.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:48 smithi076 conmon[56514]: 351 : audit [DBG] from='client.14655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:48.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:48 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:19:48.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:48 smithi076 conmon[62156]: 19:47.753195+0000 mgr.y (mgr.14144) 351 : audit [DBG] from='client.14655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:49.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:48 smithi125 conmon[56739]: audit 2021-03-15T22:19:47.753195+0000 mgr.y (mgr.14144) 351 : audit [DBG] 2021-03-15T22:19:49.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:48 smithi125 conmon[56739]: from='client.14655 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:49.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:19:49.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: cluster 2021-03-15T22:19:49.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: 2021-03-15T22:19:47.931154+0000 osd.3 (osd. 2021-03-15T22:19:49.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: 3) 28 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:19:49.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:19:49.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: :19:47.956149+0000 osd.7 (osd.7) 19 : cluster [DBG] 2021-03-15T22:19:49.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: 3.d scrub ok 2021-03-15T22:19:49.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:19:49.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: -15T22:19:48. 2021-03-15T22:19:49.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: 154876+0000 osd.4 (osd.4) 16 : cluster 2021-03-15T22:19:49.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: [DBG] 4.c scrub ok 2021-03-15T22:19:49.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:19:49.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: 03-15T22:19:48. 2021-03-15T22:19:49.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:49 smithi125 conmon[56739]: 972251+0000 mgr.y (mgr.14144) 352 : cluster [DBG] pgmap v251: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:50.004 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[56514]: cluster 2021-03-15T22:19:47.931154+0000 osd.3 2021-03-15T22:19:50.004 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[56514]: (osd.3) 28 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:19:50.004 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[56514]: cluster 2021-03-15T22:19:47.956149+0000 osd.7 (osd.7) 19 : cluster [DBG] 3.d scrub ok 2021-03-15T22:19:50.005 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[56514]: cluster 2021-03-15T22:19:48.154876+0000 osd.4 (osd.4) 16 : cluster [DBG] 4.c scrub ok 2021-03-15T22:19:50.005 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[56514]: cluster 2021-03-15T22:19:48.972251+0000 mgr.y (mgr.14144) 352 : cluster [DBG] pgmap v251: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:50.005 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: cluster 2021-03-15T22:19:50.005 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: 2021-03-15T22:19: 2021-03-15T22:19:50.006 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: 47.931154+0000 osd.3 (osd.3) 28 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:19:50.006 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: cluster 2021-03-15T22:19:50.006 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: 2021-03-15T22:19:47.956149+0000 2021-03-15T22:19:50.006 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: osd.7 (osd.7) 19 : cluster [DBG] 3.d scrub ok 2021-03-15T22:19:50.007 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: 2021-03-15T22:19:50.007 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:19:50.007 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: 19:48.154876+0000 osd.4 (osd.4) 2021-03-15T22:19:50.007 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: 16 : cluster [DBG] 4.c scrub ok 2021-03-15T22:19:50.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: cluster 2021-03-15T22:19:50.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: 2021-03-15T22:19:48.972251 2021-03-15T22:19:50.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 352 : cluster [DBG] 2021-03-15T22:19:50.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:49 smithi076 conmon[62156]: pgmap v251: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:50.439 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:19:50.441 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:19:51.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:50 smithi076 conmon[56514]: audit 2021-03-15T22:19:49.389795+0000 mgr.y (mgr.14144) 353 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:51.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:50 smithi076 conmon[56514]: cluster 2021-03-15T22:19:49.447308+0000 osd.7 (osd.7) 20 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:51.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:50 smithi076 conmon[56514]: cluster 2021-03-15T22:19:49.447470+0000 osd.7 (osd.7) 21 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:51.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:50 smithi076 conmon[56514]: cluster 2021-03-15T22:19:49.580756+0000 mon.a (mon.0) 590 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2021-03-15T22:19:51.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:50 smithi076 conmon[62156]: audit 2021-03-15T22:19:49.389795+0000 mgr.y (mgr.14144) 353 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:51.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:50 smithi076 conmon[62156]: cluster 2021-03-15T22:19:49.447308+0000 osd.7 (osd.7) 20 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:51.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:50 smithi076 conmon[62156]: cluster 2021-03-15T22:19:49.447470+0000 osd.7 (osd.7) 21 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:51.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:50 smithi076 conmon[62156]: cluster 2021-03-15T22:19:49.580756+0000 mon.a (mon.0) 590 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2021-03-15T22:19:51.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:50 smithi125 conmon[56739]: audit 2021-03-15T22:19:49.389795+0000 mgr.y (mgr.14144) 353 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:19:51.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:50 smithi125 conmon[56739]: cluster 2021-03-15T22:19:49.447308+0000 osd.7 (osd.7) 20 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:51.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:50 smithi125 conmon[56739]: cluster 2021-03-15T22:19:49.447470+0000 osd.7 (osd.7) 21 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:51.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:50 smithi125 conmon[56739]: cluster 2021-03-15T22:19:49.580756+0000 mon.a (mon 2021-03-15T22:19:51.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:50 smithi125 conmon[56739]: .0) 590 : cluster [DBG] osdmap e88: 8 total, 8 up, 8 in 2021-03-15T22:19:51.178 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":252,"stamp":"2021-03-15T22:19:49.578151+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":25236,"num_objects":316,"num_object_clones":0,"num_object_copies":948,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":316,"num_whiteouts":0,"num_read":1188,"num_read_kb":688,"num_write":550,"num_write_kb":165,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":582,"ondisk_log_size":582,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":53356,"kb_used_data":7212,"kb_used_omap":0,"kb_used_meta":46080,"kb_avail":749907860,"statfs":{"total":767960285184,"available":767905648640,"internally_reserved":0,"allocated":7385088,"data_stored":2952372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47185920},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.539059"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994463+0000","last_change":"2021-03-15T22:19:28.122158+0000","last_active":"2021-03-15T22:19:30.994463+0000","last_peered":"2021-03-15T22:19:30.994463+0000","last_clean":"2021-03-15T22:19:30.994463+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:19:30.994463+0000","last_undegraded":"2021-03-15T22:19:30.994463+0000","last_fullsized":"2021-03-15T22:19:30.994463+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.122079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.122079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:37.782723+0000","last_change":"2021-03-15T22:19:37.782674+0000","last_active":"2021-03-15T22:19:37.782723+0000","last_peered":"2021-03-15T22:19:37.782723+0000","last_clean":"2021-03-15T22:19:37.782723+0000","last_became_active":"2021-03-15T22:17:58.014719+0000","last_became_peered":"2021-03-15T22:17:58.014719+0000","last_unstale":"2021-03-15T22:19:37.782723+0000","last_undegraded":"2021-03-15T22:19:37.782723+0000","last_fullsized":"2021-03-15T22:19:37.782723+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.782584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.782584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"46","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995957+0000","last_change":"2021-03-15T22:19:08.789028+0000","last_active":"2021-03-15T22:19:30.995957+0000","last_peered":"2021-03-15T22:19:30.995957+0000","last_clean":"2021-03-15T22:19:30.995957+0000","last_became_active":"2021-03-15T22:17:54.284855+0000","last_became_peered":"2021-03-15T22:17:54.284855+0000","last_unstale":"2021-03-15T22:19:30.995957+0000","last_undegraded":"2021-03-15T22:19:30.995957+0000","last_fullsized":"2021-03-15T22:19:30.995957+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:19:08.788964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:08.788964+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"70","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994360+0000","last_change":"2021-03-15T22:19:11.891528+0000","last_active":"2021-03-15T22:19:30.994360+0000","last_peered":"2021-03-15T22:19:30.994360+0000","last_clean":"2021-03-15T22:19:30.994360+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:19:30.994360+0000","last_undegraded":"2021-03-15T22:19:30.994360+0000","last_fullsized":"2021-03-15T22:19:30.994360+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:11.891465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.891465+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996834+0000","last_change":"2021-03-15T22:19:26.865961+0000","last_active":"2021-03-15T22:19:30.996834+0000","last_peered":"2021-03-15T22:19:30.996834+0000","last_clean":"2021-03-15T22:19:30.996834+0000","last_became_active":"2021-03-15T22:18:00.288598+0000","last_became_peered":"2021-03-15T22:18:00.288598+0000","last_unstale":"2021-03-15T22:19:30.996834+0000","last_undegraded":"2021-03-15T22:19:30.996834+0000","last_fullsized":"2021-03-15T22:19:30.996834+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.865895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.865895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"35","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994352+0000","last_change":"2021-03-15T22:17:58.011237+0000","last_active":"2021-03-15T22:19:30.994352+0000","last_peered":"2021-03-15T22:19:30.994352+0000","last_clean":"2021-03-15T22:19:30.994352+0000","last_became_active":"2021-03-15T22:17:58.010916+0000","last_became_peered":"2021-03-15T22:17:58.010916+0000","last_unstale":"2021-03-15T22:19:30.994352+0000","last_undegraded":"2021-03-15T22:19:30.994352+0000","last_fullsized":"2021-03-15T22:19:30.994352+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.997201+0000","last_change":"2021-03-15T22:19:10.851317+0000","last_active":"2021-03-15T22:19:30.997201+0000","last_peered":"2021-03-15T22:19:30.997201+0000","last_clean":"2021-03-15T22:19:30.997201+0000","last_became_active":"2021-03-15T22:17:54.283819+0000","last_became_peered":"2021-03-15T22:17:54.283819+0000","last_unstale":"2021-03-15T22:19:30.997201+0000","last_undegraded":"2021-03-15T22:19:30.997201+0000","last_fullsized":"2021-03-15T22:19:30.997201+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:10.851256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:10.851256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"77","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:44.915926+0000","last_change":"2021-03-15T22:19:44.915926+0000","last_active":"2021-03-15T22:19:44.915926+0000","last_peered":"2021-03-15T22:19:44.915926+0000","last_clean":"2021-03-15T22:19:44.915926+0000","last_became_active":"2021-03-15T22:17:55.991597+0000","last_became_peered":"2021-03-15T22:17:55.991597+0000","last_unstale":"2021-03-15T22:19:44.915926+0000","last_undegraded":"2021-03-15T22:19:44.915926+0000","last_fullsized":"2021-03-15T22:19:44.915926+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:19:44.915865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:44.915865+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996441+0000","last_change":"2021-03-15T22:19:11.832031+0000","last_active":"2021-03-15T22:19:30.996441+0000","last_peered":"2021-03-15T22:19:30.996441+0000","last_clean":"2021-03-15T22:19:30.996441+0000","last_became_active":"2021-03-15T22:18:00.015776+0000","last_became_peered":"2021-03-15T22:18:00.015776+0000","last_unstale":"2021-03-15T22:19:30.996441+0000","last_undegraded":"2021-03-15T22:19:30.996441+0000","last_fullsized":"2021-03-15T22:19:30.996441+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:11.831963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.831963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500493+0000","last_change":"2021-03-15T22:19:30.205991+0000","last_active":"2021-03-15T22:19:31.500493+0000","last_peered":"2021-03-15T22:19:31.500493+0000","last_clean":"2021-03-15T22:19:31.500493+0000","last_became_active":"2021-03-15T22:17:58.008828+0000","last_became_peered":"2021-03-15T22:17:58.008828+0000","last_unstale":"2021-03-15T22:19:31.500493+0000","last_undegraded":"2021-03-15T22:19:31.500493+0000","last_fullsized":"2021-03-15T22:19:31.500493+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.205915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.205915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500577+0000","last_change":"2021-03-15T22:18:53.334542+0000","last_active":"2021-03-15T22:19:31.500577+0000","last_peered":"2021-03-15T22:19:31.500577+0000","last_clean":"2021-03-15T22:19:31.500577+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:19:31.500577+0000","last_undegraded":"2021-03-15T22:19:31.500577+0000","last_fullsized":"2021-03-15T22:19:31.500577+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:53.334478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:53.334478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"61","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994691+0000","last_change":"2021-03-15T22:17:55.990767+0000","last_active":"2021-03-15T22:19:30.994691+0000","last_peered":"2021-03-15T22:19:30.994691+0000","last_clean":"2021-03-15T22:19:30.994691+0000","last_became_active":"2021-03-15T22:17:55.990528+0000","last_became_peered":"2021-03-15T22:17:55.990528+0000","last_unstale":"2021-03-15T22:19:30.994691+0000","last_undegraded":"2021-03-15T22:19:30.994691+0000","last_fullsized":"2021-03-15T22:19:30.994691+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:37.268525+0000","last_change":"2021-03-15T22:19:37.268478+0000","last_active":"2021-03-15T22:19:37.268525+0000","last_peered":"2021-03-15T22:19:37.268525+0000","last_clean":"2021-03-15T22:19:37.268525+0000","last_became_active":"2021-03-15T22:18:00.019585+0000","last_became_peered":"2021-03-15T22:18:00.019585+0000","last_unstale":"2021-03-15T22:19:37.268525+0000","last_undegraded":"2021-03-15T22:19:37.268525+0000","last_fullsized":"2021-03-15T22:19:37.268525+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.268362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.268362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"35","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884909+0000","last_change":"2021-03-15T22:17:58.013492+0000","last_active":"2021-03-15T22:19:31.884909+0000","last_peered":"2021-03-15T22:19:31.884909+0000","last_clean":"2021-03-15T22:19:31.884909+0000","last_became_active":"2021-03-15T22:17:58.013236+0000","last_became_peered":"2021-03-15T22:17:58.013236+0000","last_unstale":"2021-03-15T22:19:31.884909+0000","last_undegraded":"2021-03-15T22:19:31.884909+0000","last_fullsized":"2021-03-15T22:19:31.884909+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500507+0000","last_change":"2021-03-15T22:19:06.274483+0000","last_active":"2021-03-15T22:19:31.500507+0000","last_peered":"2021-03-15T22:19:31.500507+0000","last_clean":"2021-03-15T22:19:31.500507+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:19:31.500507+0000","last_undegraded":"2021-03-15T22:19:31.500507+0000","last_fullsized":"2021-03-15T22:19:31.500507+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:06.274416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.274416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'6","reported_seq":"71","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995078+0000","last_change":"2021-03-15T22:19:02.277922+0000","last_active":"2021-03-15T22:19:30.995078+0000","last_peered":"2021-03-15T22:19:30.995078+0000","last_clean":"2021-03-15T22:19:30.995078+0000","last_became_active":"2021-03-15T22:19:02.274105+0000","last_became_peered":"2021-03-15T22:19:02.274105+0000","last_unstale":"2021-03-15T22:19:30.995078+0000","last_undegraded":"2021-03-15T22:19:30.995078+0000","last_fullsized":"2021-03-15T22:19:30.995078+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2021-03-15T22:18:59.037504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.037504+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"33","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995329+0000","last_change":"2021-03-15T22:18:00.287135+0000","last_active":"2021-03-15T22:19:30.995329+0000","last_peered":"2021-03-15T22:19:30.995329+0000","last_clean":"2021-03-15T22:19:30.995329+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:19:30.995329+0000","last_undegraded":"2021-03-15T22:19:30.995329+0000","last_fullsized":"2021-03-15T22:19:30.995329+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:35.779241+0000","last_change":"2021-03-15T22:19:35.779203+0000","last_active":"2021-03-15T22:19:35.779241+0000","last_peered":"2021-03-15T22:19:35.779241+0000","last_clean":"2021-03-15T22:19:35.779241+0000","last_became_active":"2021-03-15T22:17:58.016067+0000","last_became_peered":"2021-03-15T22:17:58.016067+0000","last_unstale":"2021-03-15T22:19:35.779241+0000","last_undegraded":"2021-03-15T22:19:35.779241+0000","last_fullsized":"2021-03-15T22:19:35.779241+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:35.779105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.779105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"46","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996187+0000","last_change":"2021-03-15T22:19:14.859504+0000","last_active":"2021-03-15T22:19:30.996187+0000","last_peered":"2021-03-15T22:19:30.996187+0000","last_clean":"2021-03-15T22:19:30.996187+0000","last_became_active":"2021-03-15T22:17:53.976875+0000","last_became_peered":"2021-03-15T22:17:53.976875+0000","last_unstale":"2021-03-15T22:19:30.996187+0000","last_undegraded":"2021-03-15T22:19:30.996187+0000","last_fullsized":"2021-03-15T22:19:30.996187+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.859443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"67","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944940+0000","last_change":"2021-03-15T22:19:27.165212+0000","last_active":"2021-03-15T22:19:31.944940+0000","last_peered":"2021-03-15T22:19:31.944940+0000","last_clean":"2021-03-15T22:19:31.944940+0000","last_became_active":"2021-03-15T22:17:55.992247+0000","last_became_peered":"2021-03-15T22:17:55.992247+0000","last_unstale":"2021-03-15T22:19:31.944940+0000","last_undegraded":"2021-03-15T22:19:31.944940+0000","last_fullsized":"2021-03-15T22:19:31.944940+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:27.165119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.165119+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"38","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:45.893973+0000","last_change":"2021-03-15T22:19:45.893973+0000","last_active":"2021-03-15T22:19:45.893973+0000","last_peered":"2021-03-15T22:19:45.893973+0000","last_clean":"2021-03-15T22:19:45.893973+0000","last_became_active":"2021-03-15T22:18:00.286974+0000","last_became_peered":"2021-03-15T22:18:00.286974+0000","last_unstale":"2021-03-15T22:19:45.893973+0000","last_undegraded":"2021-03-15T22:19:45.893973+0000","last_fullsized":"2021-03-15T22:19:45.893973+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:45.893910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:45.893910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994924+0000","last_change":"2021-03-15T22:19:21.043341+0000","last_active":"2021-03-15T22:19:30.994924+0000","last_peered":"2021-03-15T22:19:30.994924+0000","last_clean":"2021-03-15T22:19:30.994924+0000","last_became_active":"2021-03-15T22:17:58.005477+0000","last_became_peered":"2021-03-15T22:17:58.005477+0000","last_unstale":"2021-03-15T22:19:30.994924+0000","last_undegraded":"2021-03-15T22:19:30.994924+0000","last_fullsized":"2021-03-15T22:19:30.994924+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.043261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.043261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:35.186077+0000","last_change":"2021-03-15T22:19:35.186018+0000","last_active":"2021-03-15T22:19:35.186077+0000","last_peered":"2021-03-15T22:19:35.186077+0000","last_clean":"2021-03-15T22:19:35.186077+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:19:35.186077+0000","last_undegraded":"2021-03-15T22:19:35.186077+0000","last_fullsized":"2021-03-15T22:19:35.186077+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:35.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.185891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"73","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:35.934619+0000","last_change":"2021-03-15T22:19:35.934561+0000","last_active":"2021-03-15T22:19:35.934619+0000","last_peered":"2021-03-15T22:19:35.934619+0000","last_clean":"2021-03-15T22:19:35.934619+0000","last_became_active":"2021-03-15T22:17:55.995747+0000","last_became_peered":"2021-03-15T22:17:55.995747+0000","last_unstale":"2021-03-15T22:19:35.934619+0000","last_undegraded":"2021-03-15T22:19:35.934619+0000","last_fullsized":"2021-03-15T22:19:35.934619+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:35.934475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.934475+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885511+0000","last_change":"2021-03-15T22:19:30.854972+0000","last_active":"2021-03-15T22:19:31.885511+0000","last_peered":"2021-03-15T22:19:31.885511+0000","last_clean":"2021-03-15T22:19:31.885511+0000","last_became_active":"2021-03-15T22:18:00.015889+0000","last_became_peered":"2021-03-15T22:18:00.015889+0000","last_unstale":"2021-03-15T22:19:31.885511+0000","last_undegraded":"2021-03-15T22:19:31.885511+0000","last_fullsized":"2021-03-15T22:19:31.885511+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.854899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.854899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"61'2","reported_seq":"62","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.165293+0000","last_change":"2021-03-15T22:19:21.656004+0000","last_active":"2021-03-15T22:19:40.165293+0000","last_peered":"2021-03-15T22:19:40.165293+0000","last_clean":"2021-03-15T22:19:40.165293+0000","last_became_active":"2021-03-15T22:17:58.013571+0000","last_became_peered":"2021-03-15T22:17:58.013571+0000","last_unstale":"2021-03-15T22:19:40.165293+0000","last_undegraded":"2021-03-15T22:19:40.165293+0000","last_fullsized":"2021-03-15T22:19:40.165293+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:21.655918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.655918+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944147+0000","last_change":"2021-03-15T22:19:16.638497+0000","last_active":"2021-03-15T22:19:31.944147+0000","last_peered":"2021-03-15T22:19:31.944147+0000","last_clean":"2021-03-15T22:19:31.944147+0000","last_became_active":"2021-03-15T22:17:53.975549+0000","last_became_peered":"2021-03-15T22:17:53.975549+0000","last_unstale":"2021-03-15T22:19:31.944147+0000","last_undegraded":"2021-03-15T22:19:31.944147+0000","last_fullsized":"2021-03-15T22:19:31.944147+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.638428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.638428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"79","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.666578+0000","last_change":"2021-03-15T22:19:40.666578+0000","last_active":"2021-03-15T22:19:40.666578+0000","last_peered":"2021-03-15T22:19:40.666578+0000","last_clean":"2021-03-15T22:19:40.666578+0000","last_became_active":"2021-03-15T22:17:56.288283+0000","last_became_peered":"2021-03-15T22:17:56.288283+0000","last_unstale":"2021-03-15T22:19:40.666578+0000","last_undegraded":"2021-03-15T22:19:40.666578+0000","last_fullsized":"2021-03-15T22:19:40.666578+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.666518+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944579+0000","last_change":"2021-03-15T22:19:07.544919+0000","last_active":"2021-03-15T22:19:31.944579+0000","last_peered":"2021-03-15T22:19:31.944579+0000","last_clean":"2021-03-15T22:19:31.944579+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:19:31.944579+0000","last_undegraded":"2021-03-15T22:19:31.944579+0000","last_fullsized":"2021-03-15T22:19:31.944579+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.544856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.544856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996391+0000","last_change":"2021-03-15T22:19:17.752562+0000","last_active":"2021-03-15T22:19:30.996391+0000","last_peered":"2021-03-15T22:19:30.996391+0000","last_clean":"2021-03-15T22:19:30.996391+0000","last_became_active":"2021-03-15T22:17:58.009184+0000","last_became_peered":"2021-03-15T22:17:58.009184+0000","last_unstale":"2021-03-15T22:19:30.996391+0000","last_undegraded":"2021-03-15T22:19:30.996391+0000","last_fullsized":"2021-03-15T22:19:30.996391+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.752495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.752495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996276+0000","last_change":"2021-03-15T22:19:21.900042+0000","last_active":"2021-03-15T22:19:30.996276+0000","last_peered":"2021-03-15T22:19:30.996276+0000","last_clean":"2021-03-15T22:19:30.996276+0000","last_became_active":"2021-03-15T22:17:53.967947+0000","last_became_peered":"2021-03-15T22:17:53.967947+0000","last_unstale":"2021-03-15T22:19:30.996276+0000","last_undegraded":"2021-03-15T22:19:30.996276+0000","last_fullsized":"2021-03-15T22:19:30.996276+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.899969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.899969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"72","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.886568+0000","last_change":"2021-03-15T22:19:13.525956+0000","last_active":"2021-03-15T22:19:31.886568+0000","last_peered":"2021-03-15T22:19:31.886568+0000","last_clean":"2021-03-15T22:19:31.886568+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:19:31.886568+0000","last_undegraded":"2021-03-15T22:19:31.886568+0000","last_fullsized":"2021-03-15T22:19:31.886568+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:19:13.525883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.525883+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884802+0000","last_change":"2021-03-15T22:19:25.759114+0000","last_active":"2021-03-15T22:19:31.884802+0000","last_peered":"2021-03-15T22:19:31.884802+0000","last_clean":"2021-03-15T22:19:31.884802+0000","last_became_active":"2021-03-15T22:18:00.015858+0000","last_became_peered":"2021-03-15T22:18:00.015858+0000","last_unstale":"2021-03-15T22:19:31.884802+0000","last_undegraded":"2021-03-15T22:19:31.884802+0000","last_fullsized":"2021-03-15T22:19:31.884802+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.759034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"41","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885584+0000","last_change":"2021-03-15T22:19:28.737870+0000","last_active":"2021-03-15T22:19:31.885584+0000","last_peered":"2021-03-15T22:19:31.885584+0000","last_clean":"2021-03-15T22:19:31.885584+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:19:31.885584+0000","last_undegraded":"2021-03-15T22:19:31.885584+0000","last_fullsized":"2021-03-15T22:19:31.885584+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.737796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.737796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996042+0000","last_change":"2021-03-15T22:19:01.740592+0000","last_active":"2021-03-15T22:19:30.996042+0000","last_peered":"2021-03-15T22:19:30.996042+0000","last_clean":"2021-03-15T22:19:30.996042+0000","last_became_active":"2021-03-15T22:17:53.973507+0000","last_became_peered":"2021-03-15T22:17:53.973507+0000","last_unstale":"2021-03-15T22:19:30.996042+0000","last_undegraded":"2021-03-15T22:19:30.996042+0000","last_fullsized":"2021-03-15T22:19:30.996042+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:01.740496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:01.740496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"59","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996360+0000","last_change":"2021-03-15T22:17:55.996994+0000","last_active":"2021-03-15T22:19:30.996360+0000","last_peered":"2021-03-15T22:19:30.996360+0000","last_clean":"2021-03-15T22:19:30.996360+0000","last_became_active":"2021-03-15T22:17:55.996707+0000","last_became_peered":"2021-03-15T22:17:55.996707+0000","last_unstale":"2021-03-15T22:19:30.996360+0000","last_undegraded":"2021-03-15T22:19:30.996360+0000","last_fullsized":"2021-03-15T22:19:30.996360+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996742+0000","last_change":"2021-03-15T22:19:14.850664+0000","last_active":"2021-03-15T22:19:30.996742+0000","last_peered":"2021-03-15T22:19:30.996742+0000","last_clean":"2021-03-15T22:19:30.996742+0000","last_became_active":"2021-03-15T22:18:00.287972+0000","last_became_peered":"2021-03-15T22:18:00.287972+0000","last_unstale":"2021-03-15T22:19:30.996742+0000","last_undegraded":"2021-03-15T22:19:30.996742+0000","last_fullsized":"2021-03-15T22:19:30.996742+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:14.850605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.850605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884873+0000","last_change":"2021-03-15T22:19:27.800200+0000","last_active":"2021-03-15T22:19:31.884873+0000","last_peered":"2021-03-15T22:19:31.884873+0000","last_clean":"2021-03-15T22:19:31.884873+0000","last_became_active":"2021-03-15T22:17:58.013174+0000","last_became_peered":"2021-03-15T22:17:58.013174+0000","last_unstale":"2021-03-15T22:19:31.884873+0000","last_undegraded":"2021-03-15T22:19:31.884873+0000","last_fullsized":"2021-03-15T22:19:31.884873+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:27.800106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.800106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500840+0000","last_change":"2021-03-15T22:19:17.295504+0000","last_active":"2021-03-15T22:19:31.500840+0000","last_peered":"2021-03-15T22:19:31.500840+0000","last_clean":"2021-03-15T22:19:31.500840+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:19:31.500840+0000","last_undegraded":"2021-03-15T22:19:31.500840+0000","last_fullsized":"2021-03-15T22:19:31.500840+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.295412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.295412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"65'14","reported_seq":"62","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996850+0000","last_change":"2021-03-15T22:17:56.289858+0000","last_active":"2021-03-15T22:19:30.996850+0000","last_peered":"2021-03-15T22:19:30.996850+0000","last_clean":"2021-03-15T22:19:30.996850+0000","last_became_active":"2021-03-15T22:17:56.289456+0000","last_became_peered":"2021-03-15T22:17:56.289456+0000","last_unstale":"2021-03-15T22:19:30.996850+0000","last_undegraded":"2021-03-15T22:19:30.996850+0000","last_fullsized":"2021-03-15T22:19:30.996850+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995910+0000","last_change":"2021-03-15T22:19:26.615442+0000","last_active":"2021-03-15T22:19:30.995910+0000","last_peered":"2021-03-15T22:19:30.995910+0000","last_clean":"2021-03-15T22:19:30.995910+0000","last_became_active":"2021-03-15T22:18:00.014112+0000","last_became_peered":"2021-03-15T22:18:00.014112+0000","last_unstale":"2021-03-15T22:19:30.995910+0000","last_undegraded":"2021-03-15T22:19:30.995910+0000","last_fullsized":"2021-03-15T22:19:30.995910+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.615380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.615380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996888+0000","last_change":"2021-03-15T22:19:03.699264+0000","last_active":"2021-03-15T22:19:30.996888+0000","last_peered":"2021-03-15T22:19:30.996888+0000","last_clean":"2021-03-15T22:19:30.996888+0000","last_became_active":"2021-03-15T22:17:58.009365+0000","last_became_peered":"2021-03-15T22:17:58.009365+0000","last_unstale":"2021-03-15T22:19:30.996888+0000","last_undegraded":"2021-03-15T22:19:30.996888+0000","last_fullsized":"2021-03-15T22:19:30.996888+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_clean_scrub_stamp":"2021-03-15T22:19:03.699198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:36.927090+0000","last_change":"2021-03-15T22:19:36.927055+0000","last_active":"2021-03-15T22:19:36.927090+0000","last_peered":"2021-03-15T22:19:36.927090+0000","last_clean":"2021-03-15T22:19:36.927090+0000","last_became_active":"2021-03-15T22:17:53.979947+0000","last_became_peered":"2021-03-15T22:17:53.979947+0000","last_unstale":"2021-03-15T22:19:36.927090+0000","last_undegraded":"2021-03-15T22:19:36.927090+0000","last_fullsized":"2021-03-15T22:19:36.927090+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:36.926961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:36.926961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"70","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:38.767554+0000","last_change":"2021-03-15T22:19:38.767499+0000","last_active":"2021-03-15T22:19:38.767554+0000","last_peered":"2021-03-15T22:19:38.767554+0000","last_clean":"2021-03-15T22:19:38.767554+0000","last_became_active":"2021-03-15T22:17:55.991537+0000","last_became_peered":"2021-03-15T22:17:55.991537+0000","last_unstale":"2021-03-15T22:19:38.767554+0000","last_undegraded":"2021-03-15T22:19:38.767554+0000","last_fullsized":"2021-03-15T22:19:38.767554+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:38.767414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.767414+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"33","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996602+0000","last_change":"2021-03-15T22:18:00.018406+0000","last_active":"2021-03-15T22:19:30.996602+0000","last_peered":"2021-03-15T22:19:30.996602+0000","last_clean":"2021-03-15T22:19:30.996602+0000","last_became_active":"2021-03-15T22:18:00.018117+0000","last_became_peered":"2021-03-15T22:18:00.018117+0000","last_unstale":"2021-03-15T22:19:30.996602+0000","last_undegraded":"2021-03-15T22:19:30.996602+0000","last_fullsized":"2021-03-15T22:19:30.996602+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:32.759907+0000","last_change":"2021-03-15T22:19:32.759869+0000","last_active":"2021-03-15T22:19:32.759907+0000","last_peered":"2021-03-15T22:19:32.759907+0000","last_clean":"2021-03-15T22:19:32.759907+0000","last_became_active":"2021-03-15T22:17:58.014621+0000","last_became_peered":"2021-03-15T22:17:58.014621+0000","last_unstale":"2021-03-15T22:19:32.759907+0000","last_undegraded":"2021-03-15T22:19:32.759907+0000","last_fullsized":"2021-03-15T22:19:32.759907+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:32.759804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:32.759804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"44","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.166781+0000","last_change":"2021-03-15T22:19:40.166781+0000","last_active":"2021-03-15T22:19:40.166781+0000","last_peered":"2021-03-15T22:19:40.166781+0000","last_clean":"2021-03-15T22:19:40.166781+0000","last_became_active":"2021-03-15T22:17:53.975292+0000","last_became_peered":"2021-03-15T22:17:53.975292+0000","last_unstale":"2021-03-15T22:19:40.166781+0000","last_undegraded":"2021-03-15T22:19:40.166781+0000","last_fullsized":"2021-03-15T22:19:40.166781+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:40.166704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.166704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"69","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:38.156229+0000","last_change":"2021-03-15T22:19:38.156190+0000","last_active":"2021-03-15T22:19:38.156229+0000","last_peered":"2021-03-15T22:19:38.156229+0000","last_clean":"2021-03-15T22:19:38.156229+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:19:38.156229+0000","last_undegraded":"2021-03-15T22:19:38.156229+0000","last_fullsized":"2021-03-15T22:19:38.156229+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:38.156103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.156103+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"37","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:44.586030+0000","last_change":"2021-03-15T22:19:44.586030+0000","last_active":"2021-03-15T22:19:44.586030+0000","last_peered":"2021-03-15T22:19:44.586030+0000","last_clean":"2021-03-15T22:19:44.586030+0000","last_became_active":"2021-03-15T22:18:00.015034+0000","last_became_peered":"2021-03-15T22:18:00.015034+0000","last_unstale":"2021-03-15T22:19:44.586030+0000","last_undegraded":"2021-03-15T22:19:44.586030+0000","last_fullsized":"2021-03-15T22:19:44.586030+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:44.585947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:44.585947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"61'2","reported_seq":"63","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:42.026423+0000","last_change":"2021-03-15T22:19:39.224283+0000","last_active":"2021-03-15T22:19:42.026423+0000","last_peered":"2021-03-15T22:19:42.026423+0000","last_clean":"2021-03-15T22:19:42.026423+0000","last_became_active":"2021-03-15T22:17:58.010903+0000","last_became_peered":"2021-03-15T22:17:58.010903+0000","last_unstale":"2021-03-15T22:19:42.026423+0000","last_undegraded":"2021-03-15T22:19:42.026423+0000","last_fullsized":"2021-03-15T22:19:42.026423+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:39.224198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:39.224198+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"44","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:43.915597+0000","last_change":"2021-03-15T22:19:43.915597+0000","last_active":"2021-03-15T22:19:43.915597+0000","last_peered":"2021-03-15T22:19:43.915597+0000","last_clean":"2021-03-15T22:19:43.915597+0000","last_became_active":"2021-03-15T22:17:54.285851+0000","last_became_peered":"2021-03-15T22:17:54.285851+0000","last_unstale":"2021-03-15T22:19:43.915597+0000","last_undegraded":"2021-03-15T22:19:43.915597+0000","last_fullsized":"2021-03-15T22:19:43.915597+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:43.915518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:43.915518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"65","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.997197+0000","last_change":"2021-03-15T22:19:07.874527+0000","last_active":"2021-03-15T22:19:30.997197+0000","last_peered":"2021-03-15T22:19:30.997197+0000","last_clean":"2021-03-15T22:19:30.997197+0000","last_became_active":"2021-03-15T22:17:56.288102+0000","last_became_peered":"2021-03-15T22:17:56.288102+0000","last_unstale":"2021-03-15T22:19:30.997197+0000","last_undegraded":"2021-03-15T22:19:30.997197+0000","last_fullsized":"2021-03-15T22:19:30.997197+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:19:07.874462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.874462+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"33","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995545+0000","last_change":"2021-03-15T22:18:00.015238+0000","last_active":"2021-03-15T22:19:30.995545+0000","last_peered":"2021-03-15T22:19:30.995545+0000","last_clean":"2021-03-15T22:19:30.995545+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:19:30.995545+0000","last_undegraded":"2021-03-15T22:19:30.995545+0000","last_fullsized":"2021-03-15T22:19:30.995545+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"61'2","reported_seq":"63","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.661332+0000","last_change":"2021-03-15T22:19:15.850282+0000","last_active":"2021-03-15T22:19:40.661332+0000","last_peered":"2021-03-15T22:19:40.661332+0000","last_clean":"2021-03-15T22:19:40.661332+0000","last_became_active":"2021-03-15T22:17:58.003748+0000","last_became_peered":"2021-03-15T22:17:58.003748+0000","last_unstale":"2021-03-15T22:19:40.661332+0000","last_undegraded":"2021-03-15T22:19:40.661332+0000","last_fullsized":"2021-03-15T22:19:40.661332+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:15.850217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.850217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995608+0000","last_change":"2021-03-15T22:19:07.795140+0000","last_active":"2021-03-15T22:19:30.995608+0000","last_peered":"2021-03-15T22:19:30.995608+0000","last_clean":"2021-03-15T22:19:30.995608+0000","last_became_active":"2021-03-15T22:17:53.968736+0000","last_became_peered":"2021-03-15T22:17:53.968736+0000","last_unstale":"2021-03-15T22:19:30.995608+0000","last_undegraded":"2021-03-15T22:19:30.995608+0000","last_fullsized":"2021-03-15T22:19:30.995608+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.795072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.795072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"65","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:33.261379+0000","last_change":"2021-03-15T22:19:33.261320+0000","last_active":"2021-03-15T22:19:33.261379+0000","last_peered":"2021-03-15T22:19:33.261379+0000","last_clean":"2021-03-15T22:19:33.261379+0000","last_became_active":"2021-03-15T22:17:56.290563+0000","last_became_peered":"2021-03-15T22:17:56.290563+0000","last_unstale":"2021-03-15T22:19:33.261379+0000","last_undegraded":"2021-03-15T22:19:33.261379+0000","last_fullsized":"2021-03-15T22:19:33.261379+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:19:33.261213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:33.261213+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.945093+0000","last_change":"2021-03-15T22:19:16.224939+0000","last_active":"2021-03-15T22:19:31.945093+0000","last_peered":"2021-03-15T22:19:31.945093+0000","last_clean":"2021-03-15T22:19:31.945093+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:19:31.945093+0000","last_undegraded":"2021-03-15T22:19:31.945093+0000","last_fullsized":"2021-03-15T22:19:31.945093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.224872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:32.880488+0000","last_change":"2021-03-15T22:19:32.880446+0000","last_active":"2021-03-15T22:19:32.880488+0000","last_peered":"2021-03-15T22:19:32.880488+0000","last_clean":"2021-03-15T22:19:32.880488+0000","last_became_active":"2021-03-15T22:17:58.001340+0000","last_became_peered":"2021-03-15T22:17:58.001340+0000","last_unstale":"2021-03-15T22:19:32.880488+0000","last_undegraded":"2021-03-15T22:19:32.880488+0000","last_fullsized":"2021-03-15T22:19:32.880488+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:32.880362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:32.880362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"41","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884668+0000","last_change":"2021-03-15T22:19:13.870789+0000","last_active":"2021-03-15T22:19:31.884668+0000","last_peered":"2021-03-15T22:19:31.884668+0000","last_clean":"2021-03-15T22:19:31.884668+0000","last_became_active":"2021-03-15T22:17:53.976777+0000","last_became_peered":"2021-03-15T22:17:53.976777+0000","last_unstale":"2021-03-15T22:19:31.884668+0000","last_undegraded":"2021-03-15T22:19:31.884668+0000","last_fullsized":"2021-03-15T22:19:31.884668+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"71","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996311+0000","last_change":"2021-03-15T22:19:06.778692+0000","last_active":"2021-03-15T22:19:30.996311+0000","last_peered":"2021-03-15T22:19:30.996311+0000","last_clean":"2021-03-15T22:19:30.996311+0000","last_became_active":"2021-03-15T22:17:56.288678+0000","last_became_peered":"2021-03-15T22:17:56.288678+0000","last_unstale":"2021-03-15T22:19:30.996311+0000","last_undegraded":"2021-03-15T22:19:30.996311+0000","last_fullsized":"2021-03-15T22:19:30.996311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:06.778622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.778622+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885484+0000","last_change":"2021-03-15T22:19:17.797392+0000","last_active":"2021-03-15T22:19:31.885484+0000","last_peered":"2021-03-15T22:19:31.885484+0000","last_clean":"2021-03-15T22:19:31.885484+0000","last_became_active":"2021-03-15T22:18:00.288462+0000","last_became_peered":"2021-03-15T22:18:00.288462+0000","last_unstale":"2021-03-15T22:19:31.885484+0000","last_undegraded":"2021-03-15T22:19:31.885484+0000","last_fullsized":"2021-03-15T22:19:31.885484+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.797291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.797291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500840+0000","last_change":"2021-03-15T22:19:29.179877+0000","last_active":"2021-03-15T22:19:31.500840+0000","last_peered":"2021-03-15T22:19:31.500840+0000","last_clean":"2021-03-15T22:19:31.500840+0000","last_became_active":"2021-03-15T22:17:58.015682+0000","last_became_peered":"2021-03-15T22:17:58.015682+0000","last_unstale":"2021-03-15T22:19:31.500840+0000","last_undegraded":"2021-03-15T22:19:31.500840+0000","last_fullsized":"2021-03-15T22:19:31.500840+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.179800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885817+0000","last_change":"2021-03-15T22:19:05.557588+0000","last_active":"2021-03-15T22:19:31.885817+0000","last_peered":"2021-03-15T22:19:31.885817+0000","last_clean":"2021-03-15T22:19:31.885817+0000","last_became_active":"2021-03-15T22:17:53.977470+0000","last_became_peered":"2021-03-15T22:17:53.977470+0000","last_unstale":"2021-03-15T22:19:31.885817+0000","last_undegraded":"2021-03-15T22:19:31.885817+0000","last_fullsized":"2021-03-15T22:19:31.885817+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:05.557534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:05.557534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'7","reported_seq":"48","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996676+0000","last_change":"2021-03-15T22:17:56.289796+0000","last_active":"2021-03-15T22:19:30.996676+0000","last_peered":"2021-03-15T22:19:30.996676+0000","last_clean":"2021-03-15T22:19:30.996676+0000","last_became_active":"2021-03-15T22:17:56.289454+0000","last_became_peered":"2021-03-15T22:17:56.289454+0000","last_unstale":"2021-03-15T22:19:30.996676+0000","last_undegraded":"2021-03-15T22:19:30.996676+0000","last_fullsized":"2021-03-15T22:19:30.996676+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500631+0000","last_change":"2021-03-15T22:19:11.318218+0000","last_active":"2021-03-15T22:19:31.500631+0000","last_peered":"2021-03-15T22:19:31.500631+0000","last_clean":"2021-03-15T22:19:31.500631+0000","last_became_active":"2021-03-15T22:18:00.286699+0000","last_became_peered":"2021-03-15T22:18:00.286699+0000","last_unstale":"2021-03-15T22:19:31.500631+0000","last_undegraded":"2021-03-15T22:19:31.500631+0000","last_fullsized":"2021-03-15T22:19:31.500631+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:11.318132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.318132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"47","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885799+0000","last_change":"2021-03-15T22:19:06.942939+0000","last_active":"2021-03-15T22:19:31.885799+0000","last_peered":"2021-03-15T22:19:31.885799+0000","last_clean":"2021-03-15T22:19:31.885799+0000","last_became_active":"2021-03-15T22:17:53.979246+0000","last_became_peered":"2021-03-15T22:17:53.979246+0000","last_unstale":"2021-03-15T22:19:31.885799+0000","last_undegraded":"2021-03-15T22:19:31.885799+0000","last_fullsized":"2021-03-15T22:19:31.885799+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:19:06.942870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.942870+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"72","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.240467+0000","last_change":"2021-03-15T22:19:40.240467+0000","last_active":"2021-03-15T22:19:40.240467+0000","last_peered":"2021-03-15T22:19:40.240467+0000","last_clean":"2021-03-15T22:19:40.240467+0000","last_became_active":"2021-03-15T22:17:56.289684+0000","last_became_peered":"2021-03-15T22:17:56.289684+0000","last_unstale":"2021-03-15T22:19:40.240467+0000","last_undegraded":"2021-03-15T22:19:40.240467+0000","last_fullsized":"2021-03-15T22:19:40.240467+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:19:40.240388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.240388+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"61'2","reported_seq":"59","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:42.026470+0000","last_change":"2021-03-15T22:17:58.015052+0000","last_active":"2021-03-15T22:19:42.026470+0000","last_peered":"2021-03-15T22:19:42.026470+0000","last_clean":"2021-03-15T22:19:42.026470+0000","last_became_active":"2021-03-15T22:17:58.013784+0000","last_became_peered":"2021-03-15T22:17:58.013784+0000","last_unstale":"2021-03-15T22:19:42.026470+0000","last_undegraded":"2021-03-15T22:19:42.026470+0000","last_fullsized":"2021-03-15T22:19:42.026470+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:37.905357+0000","last_change":"2021-03-15T22:19:37.905329+0000","last_active":"2021-03-15T22:19:37.905357+0000","last_peered":"2021-03-15T22:19:37.905357+0000","last_clean":"2021-03-15T22:19:37.905357+0000","last_became_active":"2021-03-15T22:18:00.019686+0000","last_became_peered":"2021-03-15T22:18:00.019686+0000","last_unstale":"2021-03-15T22:19:37.905357+0000","last_undegraded":"2021-03-15T22:19:37.905357+0000","last_fullsized":"2021-03-15T22:19:37.905357+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.905267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.905267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885831+0000","last_change":"2021-03-15T22:18:59.946642+0000","last_active":"2021-03-15T22:19:31.885831+0000","last_peered":"2021-03-15T22:19:31.885831+0000","last_clean":"2021-03-15T22:19:31.885831+0000","last_became_active":"2021-03-15T22:17:53.978907+0000","last_became_peered":"2021-03-15T22:17:53.978907+0000","last_unstale":"2021-03-15T22:19:31.885831+0000","last_undegraded":"2021-03-15T22:19:31.885831+0000","last_fullsized":"2021-03-15T22:19:31.885831+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:59.946561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.946561+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"66","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994629+0000","last_change":"2021-03-15T22:17:55.993082+0000","last_active":"2021-03-15T22:19:30.994629+0000","last_peered":"2021-03-15T22:19:30.994629+0000","last_clean":"2021-03-15T22:19:30.994629+0000","last_became_active":"2021-03-15T22:17:55.992386+0000","last_became_peered":"2021-03-15T22:17:55.992386+0000","last_unstale":"2021-03-15T22:19:30.994629+0000","last_undegraded":"2021-03-15T22:19:30.994629+0000","last_fullsized":"2021-03-15T22:19:30.994629+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.943950+0000","last_change":"2021-03-15T22:19:18.197284+0000","last_active":"2021-03-15T22:19:31.943950+0000","last_peered":"2021-03-15T22:19:31.943950+0000","last_clean":"2021-03-15T22:19:31.943950+0000","last_became_active":"2021-03-15T22:17:58.007916+0000","last_became_peered":"2021-03-15T22:17:58.007916+0000","last_unstale":"2021-03-15T22:19:31.943950+0000","last_undegraded":"2021-03-15T22:19:31.943950+0000","last_fullsized":"2021-03-15T22:19:31.943950+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.197224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.197224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884486+0000","last_change":"2021-03-15T22:19:09.989290+0000","last_active":"2021-03-15T22:19:31.884486+0000","last_peered":"2021-03-15T22:19:31.884486+0000","last_clean":"2021-03-15T22:19:31.884486+0000","last_became_active":"2021-03-15T22:18:00.016271+0000","last_became_peered":"2021-03-15T22:18:00.016271+0000","last_unstale":"2021-03-15T22:19:31.884486+0000","last_undegraded":"2021-03-15T22:19:31.884486+0000","last_fullsized":"2021-03-15T22:19:31.884486+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:09.989228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:09.989228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944648+0000","last_change":"2021-03-15T22:19:29.099735+0000","last_active":"2021-03-15T22:19:31.944648+0000","last_peered":"2021-03-15T22:19:31.944648+0000","last_clean":"2021-03-15T22:19:31.944648+0000","last_became_active":"2021-03-15T22:17:58.010511+0000","last_became_peered":"2021-03-15T22:17:58.010511+0000","last_unstale":"2021-03-15T22:19:31.944648+0000","last_undegraded":"2021-03-15T22:19:31.944648+0000","last_fullsized":"2021-03-15T22:19:31.944648+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.099650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.099650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"44","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:47.956238+0000","last_change":"2021-03-15T22:19:47.956238+0000","last_active":"2021-03-15T22:19:47.956238+0000","last_peered":"2021-03-15T22:19:47.956238+0000","last_clean":"2021-03-15T22:19:47.956238+0000","last_became_active":"2021-03-15T22:17:53.974108+0000","last_became_peered":"2021-03-15T22:17:53.974108+0000","last_unstale":"2021-03-15T22:19:47.956238+0000","last_undegraded":"2021-03-15T22:19:47.956238+0000","last_fullsized":"2021-03-15T22:19:47.956238+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:47.956156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:47.956156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"80","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885940+0000","last_change":"2021-03-15T22:19:22.700145+0000","last_active":"2021-03-15T22:19:31.885940+0000","last_peered":"2021-03-15T22:19:31.885940+0000","last_clean":"2021-03-15T22:19:31.885940+0000","last_became_active":"2021-03-15T22:17:56.288870+0000","last_became_peered":"2021-03-15T22:17:56.288870+0000","last_unstale":"2021-03-15T22:19:31.885940+0000","last_undegraded":"2021-03-15T22:19:31.885940+0000","last_fullsized":"2021-03-15T22:19:31.885940+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:19:22.700049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.700049+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:39.681830+0000","last_change":"2021-03-15T22:19:39.681802+0000","last_active":"2021-03-15T22:19:39.681830+0000","last_peered":"2021-03-15T22:19:39.681830+0000","last_clean":"2021-03-15T22:19:39.681830+0000","last_became_active":"2021-03-15T22:18:00.016982+0000","last_became_peered":"2021-03-15T22:18:00.016982+0000","last_unstale":"2021-03-15T22:19:39.681830+0000","last_undegraded":"2021-03-15T22:19:39.681830+0000","last_fullsized":"2021-03-15T22:19:39.681830+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:39.681741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:39.681741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944246+0000","last_change":"2021-03-15T22:19:01.036104+0000","last_active":"2021-03-15T22:19:31.944246+0000","last_peered":"2021-03-15T22:19:31.944246+0000","last_clean":"2021-03-15T22:19:31.944246+0000","last_became_active":"2021-03-15T22:17:58.001399+0000","last_became_peered":"2021-03-15T22:17:58.001399+0000","last_unstale":"2021-03-15T22:19:31.944246+0000","last_undegraded":"2021-03-15T22:19:31.944246+0000","last_fullsized":"2021-03-15T22:19:31.944246+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:01.036038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:01.036038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500687+0000","last_change":"2021-03-15T22:19:18.252532+0000","last_active":"2021-03-15T22:19:31.500687+0000","last_peered":"2021-03-15T22:19:31.500687+0000","last_clean":"2021-03-15T22:19:31.500687+0000","last_became_active":"2021-03-15T22:17:53.972006+0000","last_became_peered":"2021-03-15T22:17:53.972006+0000","last_unstale":"2021-03-15T22:19:31.500687+0000","last_undegraded":"2021-03-15T22:19:31.500687+0000","last_fullsized":"2021-03-15T22:19:31.500687+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.252456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.252456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"96","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996043+0000","last_change":"2021-03-15T22:19:28.581985+0000","last_active":"2021-03-15T22:19:30.996043+0000","last_peered":"2021-03-15T22:19:30.996043+0000","last_clean":"2021-03-15T22:19:30.996043+0000","last_became_active":"2021-03-15T22:19:02.093463+0000","last_became_peered":"2021-03-15T22:19:02.093463+0000","last_unstale":"2021-03-15T22:19:30.996043+0000","last_undegraded":"2021-03-15T22:19:30.996043+0000","last_fullsized":"2021-03-15T22:19:30.996043+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:19:28.581920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.581920+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.997093+0000","last_change":"2021-03-15T22:19:19.908283+0000","last_active":"2021-03-15T22:19:30.997093+0000","last_peered":"2021-03-15T22:19:30.997093+0000","last_clean":"2021-03-15T22:19:30.997093+0000","last_became_active":"2021-03-15T22:18:00.020501+0000","last_became_peered":"2021-03-15T22:18:00.020501+0000","last_unstale":"2021-03-15T22:19:30.997093+0000","last_undegraded":"2021-03-15T22:19:30.997093+0000","last_fullsized":"2021-03-15T22:19:30.997093+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.908218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.908218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"45","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500320+0000","last_change":"2021-03-15T22:19:08.268784+0000","last_active":"2021-03-15T22:19:31.500320+0000","last_peered":"2021-03-15T22:19:31.500320+0000","last_clean":"2021-03-15T22:19:31.500320+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:19:31.500320+0000","last_undegraded":"2021-03-15T22:19:31.500320+0000","last_fullsized":"2021-03-15T22:19:31.500320+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:19:08.268707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995028+0000","last_change":"2021-03-15T22:17:53.978932+0000","last_active":"2021-03-15T22:19:30.995028+0000","last_peered":"2021-03-15T22:19:30.995028+0000","last_clean":"2021-03-15T22:19:30.995028+0000","last_became_active":"2021-03-15T22:17:53.978521+0000","last_became_peered":"2021-03-15T22:17:53.978521+0000","last_unstale":"2021-03-15T22:19:30.995028+0000","last_undegraded":"2021-03-15T22:19:30.995028+0000","last_fullsized":"2021-03-15T22:19:30.995028+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"102","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944627+0000","last_change":"2021-03-15T22:19:17.650024+0000","last_active":"2021-03-15T22:19:31.944627+0000","last_peered":"2021-03-15T22:19:31.944627+0000","last_clean":"2021-03-15T22:19:31.944627+0000","last_became_active":"2021-03-15T22:17:55.995319+0000","last_became_peered":"2021-03-15T22:17:55.995319+0000","last_unstale":"2021-03-15T22:19:31.944627+0000","last_undegraded":"2021-03-15T22:19:31.944627+0000","last_fullsized":"2021-03-15T22:19:31.944627+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.649963+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995777+0000","last_change":"2021-03-15T22:19:09.821008+0000","last_active":"2021-03-15T22:19:30.995777+0000","last_peered":"2021-03-15T22:19:30.995777+0000","last_clean":"2021-03-15T22:19:30.995777+0000","last_became_active":"2021-03-15T22:17:58.017434+0000","last_became_peered":"2021-03-15T22:17:58.017434+0000","last_unstale":"2021-03-15T22:19:30.995777+0000","last_undegraded":"2021-03-15T22:19:30.995777+0000","last_fullsized":"2021-03-15T22:19:30.995777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:09.820947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:09.820947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885944+0000","last_change":"2021-03-15T22:19:31.885077+0000","last_active":"2021-03-15T22:19:31.885944+0000","last_peered":"2021-03-15T22:19:31.885944+0000","last_clean":"2021-03-15T22:19:31.885944+0000","last_became_active":"2021-03-15T22:18:00.287780+0000","last_became_peered":"2021-03-15T22:18:00.287780+0000","last_unstale":"2021-03-15T22:19:31.885944+0000","last_undegraded":"2021-03-15T22:19:31.885944+0000","last_fullsized":"2021-03-15T22:19:31.885944+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:31.884965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:31.884965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944218+0000","last_change":"2021-03-15T22:19:05.537563+0000","last_active":"2021-03-15T22:19:31.944218+0000","last_peered":"2021-03-15T22:19:31.944218+0000","last_clean":"2021-03-15T22:19:31.944218+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:19:31.944218+0000","last_undegraded":"2021-03-15T22:19:31.944218+0000","last_fullsized":"2021-03-15T22:19:31.944218+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:05.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:05.537497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"47","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944304+0000","last_change":"2021-03-15T22:19:24.142618+0000","last_active":"2021-03-15T22:19:31.944304+0000","last_peered":"2021-03-15T22:19:31.944304+0000","last_clean":"2021-03-15T22:19:31.944304+0000","last_became_active":"2021-03-15T22:17:51.631460+0000","last_became_peered":"2021-03-15T22:17:51.631460+0000","last_unstale":"2021-03-15T22:19:31.944304+0000","last_undegraded":"2021-03-15T22:19:31.944304+0000","last_fullsized":"2021-03-15T22:19:31.944304+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.142558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.142558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'15","reported_seq":"73","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944776+0000","last_change":"2021-03-15T22:19:24.648256+0000","last_active":"2021-03-15T22:19:31.944776+0000","last_peered":"2021-03-15T22:19:31.944776+0000","last_clean":"2021-03-15T22:19:31.944776+0000","last_became_active":"2021-03-15T22:17:55.992952+0000","last_became_peered":"2021-03-15T22:17:55.992952+0000","last_unstale":"2021-03-15T22:19:31.944776+0000","last_undegraded":"2021-03-15T22:19:31.944776+0000","last_fullsized":"2021-03-15T22:19:31.944776+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:24.648200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.648200+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944759+0000","last_change":"2021-03-15T22:19:15.249150+0000","last_active":"2021-03-15T22:19:31.944759+0000","last_peered":"2021-03-15T22:19:31.944759+0000","last_clean":"2021-03-15T22:19:31.944759+0000","last_became_active":"2021-03-15T22:17:58.014849+0000","last_became_peered":"2021-03-15T22:17:58.014849+0000","last_unstale":"2021-03-15T22:19:31.944759+0000","last_undegraded":"2021-03-15T22:19:31.944759+0000","last_fullsized":"2021-03-15T22:19:31.944759+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:15.249076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.249076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.500903+0000","last_change":"2021-03-15T22:19:25.184338+0000","last_active":"2021-03-15T22:19:31.500903+0000","last_peered":"2021-03-15T22:19:31.500903+0000","last_clean":"2021-03-15T22:19:31.500903+0000","last_became_active":"2021-03-15T22:18:00.019363+0000","last_became_peered":"2021-03-15T22:18:00.019363+0000","last_unstale":"2021-03-15T22:19:31.500903+0000","last_undegraded":"2021-03-15T22:19:31.500903+0000","last_fullsized":"2021-03-15T22:19:31.500903+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.184260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.184260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"47","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.884748+0000","last_change":"2021-03-15T22:19:23.843965+0000","last_active":"2021-03-15T22:19:31.884748+0000","last_peered":"2021-03-15T22:19:31.884748+0000","last_clean":"2021-03-15T22:19:31.884748+0000","last_became_active":"2021-03-15T22:17:51.044140+0000","last_became_peered":"2021-03-15T22:17:51.044140+0000","last_unstale":"2021-03-15T22:19:31.884748+0000","last_undegraded":"2021-03-15T22:19:31.884748+0000","last_fullsized":"2021-03-15T22:19:31.884748+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:23.843892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.843892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.997068+0000","last_change":"2021-03-15T22:19:20.880709+0000","last_active":"2021-03-15T22:19:30.997068+0000","last_peered":"2021-03-15T22:19:30.997068+0000","last_clean":"2021-03-15T22:19:30.997068+0000","last_became_active":"2021-03-15T22:17:53.972080+0000","last_became_peered":"2021-03-15T22:17:53.972080+0000","last_unstale":"2021-03-15T22:19:30.997068+0000","last_undegraded":"2021-03-15T22:19:30.997068+0000","last_fullsized":"2021-03-15T22:19:30.997068+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:20.880638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:20.880638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"65","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.886436+0000","last_change":"2021-03-15T22:19:23.732253+0000","last_active":"2021-03-15T22:19:31.886436+0000","last_peered":"2021-03-15T22:19:31.886436+0000","last_clean":"2021-03-15T22:19:31.886436+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:19:31.886436+0000","last_undegraded":"2021-03-15T22:19:31.886436+0000","last_fullsized":"2021-03-15T22:19:31.886436+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:23.732187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.732187+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885734+0000","last_change":"2021-03-15T22:19:07.981284+0000","last_active":"2021-03-15T22:19:31.885734+0000","last_peered":"2021-03-15T22:19:31.885734+0000","last_clean":"2021-03-15T22:19:31.885734+0000","last_became_active":"2021-03-15T22:17:58.011911+0000","last_became_peered":"2021-03-15T22:17:58.011911+0000","last_unstale":"2021-03-15T22:19:31.885734+0000","last_undegraded":"2021-03-15T22:19:31.885734+0000","last_fullsized":"2021-03-15T22:19:31.885734+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.981223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944334+0000","last_change":"2021-03-15T22:19:28.773808+0000","last_active":"2021-03-15T22:19:31.944334+0000","last_peered":"2021-03-15T22:19:31.944334+0000","last_clean":"2021-03-15T22:19:31.944334+0000","last_became_active":"2021-03-15T22:18:00.014336+0000","last_became_peered":"2021-03-15T22:18:00.014336+0000","last_unstale":"2021-03-15T22:19:31.944334+0000","last_undegraded":"2021-03-15T22:19:31.944334+0000","last_fullsized":"2021-03-15T22:19:31.944334+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.773747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.773747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:38.721626+0000","last_change":"2021-03-15T22:19:38.721599+0000","last_active":"2021-03-15T22:19:38.721626+0000","last_peered":"2021-03-15T22:19:38.721626+0000","last_clean":"2021-03-15T22:19:38.721626+0000","last_became_active":"2021-03-15T22:17:53.977311+0000","last_became_peered":"2021-03-15T22:17:53.977311+0000","last_unstale":"2021-03-15T22:19:38.721626+0000","last_undegraded":"2021-03-15T22:19:38.721626+0000","last_fullsized":"2021-03-15T22:19:38.721626+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:38.721534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.721534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"164","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:47.443644+0000","last_change":"2021-03-15T22:19:03.966872+0000","last_active":"2021-03-15T22:19:47.443644+0000","last_peered":"2021-03-15T22:19:47.443644+0000","last_clean":"2021-03-15T22:19:47.443644+0000","last_became_active":"2021-03-15T22:17:51.044139+0000","last_became_peered":"2021-03-15T22:17:51.044139+0000","last_unstale":"2021-03-15T22:19:47.443644+0000","last_undegraded":"2021-03-15T22:19:47.443644+0000","last_fullsized":"2021-03-15T22:19:47.443644+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:19:03.966803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:19:03.966803+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":115,"num_read_kb":110,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"67","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996188+0000","last_change":"2021-03-15T22:19:30.589143+0000","last_active":"2021-03-15T22:19:30.996188+0000","last_peered":"2021-03-15T22:19:30.996188+0000","last_clean":"2021-03-15T22:19:30.996188+0000","last_became_active":"2021-03-15T22:17:55.999564+0000","last_became_peered":"2021-03-15T22:17:55.999564+0000","last_unstale":"2021-03-15T22:19:30.996188+0000","last_undegraded":"2021-03-15T22:19:30.996188+0000","last_fullsized":"2021-03-15T22:19:30.996188+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:30.589062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.589062+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.501056+0000","last_change":"2021-03-15T22:19:00.305911+0000","last_active":"2021-03-15T22:19:31.501056+0000","last_peered":"2021-03-15T22:19:31.501056+0000","last_clean":"2021-03-15T22:19:31.501056+0000","last_became_active":"2021-03-15T22:17:58.010798+0000","last_became_peered":"2021-03-15T22:17:58.010798+0000","last_unstale":"2021-03-15T22:19:31.501056+0000","last_undegraded":"2021-03-15T22:19:31.501056+0000","last_fullsized":"2021-03-15T22:19:31.501056+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:00.305805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:00.305805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.943978+0000","last_change":"2021-03-15T22:19:15.593372+0000","last_active":"2021-03-15T22:19:31.943978+0000","last_peered":"2021-03-15T22:19:31.943978+0000","last_clean":"2021-03-15T22:19:31.943978+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:19:31.943978+0000","last_undegraded":"2021-03-15T22:19:31.943978+0000","last_fullsized":"2021-03-15T22:19:31.943978+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.593307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944530+0000","last_change":"2021-03-15T22:19:29.812510+0000","last_active":"2021-03-15T22:19:31.944530+0000","last_peered":"2021-03-15T22:19:31.944530+0000","last_clean":"2021-03-15T22:19:31.944530+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:19:31.944530+0000","last_undegraded":"2021-03-15T22:19:31.944530+0000","last_fullsized":"2021-03-15T22:19:31.944530+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.812446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.812446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"91","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996275+0000","last_change":"2021-03-15T22:18:56.832718+0000","last_active":"2021-03-15T22:19:30.996275+0000","last_peered":"2021-03-15T22:19:30.996275+0000","last_clean":"2021-03-15T22:19:30.996275+0000","last_became_active":"2021-03-15T22:17:56.288438+0000","last_became_peered":"2021-03-15T22:17:56.288438+0000","last_unstale":"2021-03-15T22:19:30.996275+0000","last_undegraded":"2021-03-15T22:19:30.996275+0000","last_fullsized":"2021-03-15T22:19:30.996275+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:18:56.832638+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.886070+0000","last_change":"2021-03-15T22:19:16.581584+0000","last_active":"2021-03-15T22:19:31.886070+0000","last_peered":"2021-03-15T22:19:31.886070+0000","last_clean":"2021-03-15T22:19:31.886070+0000","last_became_active":"2021-03-15T22:17:58.008731+0000","last_became_peered":"2021-03-15T22:17:58.008731+0000","last_unstale":"2021-03-15T22:19:31.886070+0000","last_undegraded":"2021-03-15T22:19:31.886070+0000","last_fullsized":"2021-03-15T22:19:31.886070+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.581514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"37","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996000+0000","last_change":"2021-03-15T22:19:19.751342+0000","last_active":"2021-03-15T22:19:30.996000+0000","last_peered":"2021-03-15T22:19:30.996000+0000","last_clean":"2021-03-15T22:19:30.996000+0000","last_became_active":"2021-03-15T22:18:00.015069+0000","last_became_peered":"2021-03-15T22:18:00.015069+0000","last_unstale":"2021-03-15T22:19:30.996000+0000","last_undegraded":"2021-03-15T22:19:30.996000+0000","last_fullsized":"2021-03-15T22:19:30.996000+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.751282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.751282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"43","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:41.212619+0000","last_change":"2021-03-15T22:19:41.212619+0000","last_active":"2021-03-15T22:19:41.212619+0000","last_peered":"2021-03-15T22:19:41.212619+0000","last_clean":"2021-03-15T22:19:41.212619+0000","last_became_active":"2021-03-15T22:17:54.284266+0000","last_became_peered":"2021-03-15T22:17:54.284266+0000","last_unstale":"2021-03-15T22:19:41.212619+0000","last_undegraded":"2021-03-15T22:19:41.212619+0000","last_fullsized":"2021-03-15T22:19:41.212619+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:41.212539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:41.212539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"71","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:43.737187+0000","last_change":"2021-03-15T22:19:43.737187+0000","last_active":"2021-03-15T22:19:43.737187+0000","last_peered":"2021-03-15T22:19:43.737187+0000","last_clean":"2021-03-15T22:19:43.737187+0000","last_became_active":"2021-03-15T22:17:55.992257+0000","last_became_peered":"2021-03-15T22:17:55.992257+0000","last_unstale":"2021-03-15T22:19:43.737187+0000","last_undegraded":"2021-03-15T22:19:43.737187+0000","last_fullsized":"2021-03-15T22:19:43.737187+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:43.737126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:43.737126+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"61'2","reported_seq":"62","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.659726+0000","last_change":"2021-03-15T22:19:25.604285+0000","last_active":"2021-03-15T22:19:40.659726+0000","last_peered":"2021-03-15T22:19:40.659726+0000","last_clean":"2021-03-15T22:19:40.659726+0000","last_became_active":"2021-03-15T22:17:58.008973+0000","last_became_peered":"2021-03-15T22:17:58.008973+0000","last_unstale":"2021-03-15T22:19:40.659726+0000","last_undegraded":"2021-03-15T22:19:40.659726+0000","last_fullsized":"2021-03-15T22:19:40.659726+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:25.604217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.604217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885406+0000","last_change":"2021-03-15T22:19:18.783378+0000","last_active":"2021-03-15T22:19:31.885406+0000","last_peered":"2021-03-15T22:19:31.885406+0000","last_clean":"2021-03-15T22:19:31.885406+0000","last_became_active":"2021-03-15T22:18:00.288441+0000","last_became_peered":"2021-03-15T22:18:00.288441+0000","last_unstale":"2021-03-15T22:19:31.885406+0000","last_undegraded":"2021-03-15T22:19:31.885406+0000","last_fullsized":"2021-03-15T22:19:31.885406+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.783297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.783297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996230+0000","last_change":"2021-03-15T22:19:13.870267+0000","last_active":"2021-03-15T22:19:30.996230+0000","last_peered":"2021-03-15T22:19:30.996230+0000","last_clean":"2021-03-15T22:19:30.996230+0000","last_became_active":"2021-03-15T22:17:53.976307+0000","last_became_peered":"2021-03-15T22:17:53.976307+0000","last_unstale":"2021-03-15T22:19:30.996230+0000","last_undegraded":"2021-03-15T22:19:30.996230+0000","last_fullsized":"2021-03-15T22:19:30.996230+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:13.870202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.870202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"75","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994627+0000","last_change":"2021-03-15T22:19:26.087704+0000","last_active":"2021-03-15T22:19:30.994627+0000","last_peered":"2021-03-15T22:19:30.994627+0000","last_clean":"2021-03-15T22:19:30.994627+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:19:30.994627+0000","last_undegraded":"2021-03-15T22:19:30.994627+0000","last_fullsized":"2021-03-15T22:19:30.994627+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:19:26.087622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.087622+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"61'2","reported_seq":"58","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.660942+0000","last_change":"2021-03-15T22:17:58.011617+0000","last_active":"2021-03-15T22:19:40.660942+0000","last_peered":"2021-03-15T22:19:40.660942+0000","last_clean":"2021-03-15T22:19:40.660942+0000","last_became_active":"2021-03-15T22:17:58.011279+0000","last_became_peered":"2021-03-15T22:17:58.011279+0000","last_unstale":"2021-03-15T22:19:40.660942+0000","last_undegraded":"2021-03-15T22:19:40.660942+0000","last_fullsized":"2021-03-15T22:19:40.660942+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:34.179549+0000","last_change":"2021-03-15T22:19:34.179513+0000","last_active":"2021-03-15T22:19:34.179549+0000","last_peered":"2021-03-15T22:19:34.179549+0000","last_clean":"2021-03-15T22:19:34.179549+0000","last_became_active":"2021-03-15T22:18:00.021088+0000","last_became_peered":"2021-03-15T22:18:00.021088+0000","last_unstale":"2021-03-15T22:19:34.179549+0000","last_undegraded":"2021-03-15T22:19:34.179549+0000","last_fullsized":"2021-03-15T22:19:34.179549+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:34.179433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:34.179433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996435+0000","last_change":"2021-03-15T22:18:59.667448+0000","last_active":"2021-03-15T22:19:30.996435+0000","last_peered":"2021-03-15T22:19:30.996435+0000","last_clean":"2021-03-15T22:19:30.996435+0000","last_became_active":"2021-03-15T22:17:53.970585+0000","last_became_peered":"2021-03-15T22:17:53.970585+0000","last_unstale":"2021-03-15T22:19:30.996435+0000","last_undegraded":"2021-03-15T22:19:30.996435+0000","last_fullsized":"2021-03-15T22:19:30.996435+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:59.667388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.667388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"94","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996578+0000","last_change":"2021-03-15T22:19:04.746329+0000","last_active":"2021-03-15T22:19:30.996578+0000","last_peered":"2021-03-15T22:19:30.996578+0000","last_clean":"2021-03-15T22:19:30.996578+0000","last_became_active":"2021-03-15T22:17:56.288319+0000","last_became_peered":"2021-03-15T22:17:56.288319+0000","last_unstale":"2021-03-15T22:19:30.996578+0000","last_undegraded":"2021-03-15T22:19:30.996578+0000","last_fullsized":"2021-03-15T22:19:30.996578+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:19:04.746224+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"35","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994437+0000","last_change":"2021-03-15T22:17:58.007412+0000","last_active":"2021-03-15T22:19:30.994437+0000","last_peered":"2021-03-15T22:19:30.994437+0000","last_clean":"2021-03-15T22:19:30.994437+0000","last_became_active":"2021-03-15T22:17:58.006706+0000","last_became_peered":"2021-03-15T22:17:58.006706+0000","last_unstale":"2021-03-15T22:19:30.994437+0000","last_undegraded":"2021-03-15T22:19:30.994437+0000","last_fullsized":"2021-03-15T22:19:30.994437+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"37","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:42.258910+0000","last_change":"2021-03-15T22:19:42.258910+0000","last_active":"2021-03-15T22:19:42.258910+0000","last_peered":"2021-03-15T22:19:42.258910+0000","last_clean":"2021-03-15T22:19:42.258910+0000","last_became_active":"2021-03-15T22:18:00.019457+0000","last_became_peered":"2021-03-15T22:18:00.019457+0000","last_unstale":"2021-03-15T22:19:42.258910+0000","last_undegraded":"2021-03-15T22:19:42.258910+0000","last_fullsized":"2021-03-15T22:19:42.258910+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:42.258829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:42.258829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"61'2","reported_seq":"58","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.040174+0000","last_change":"2021-03-15T22:17:58.010787+0000","last_active":"2021-03-15T22:19:40.040174+0000","last_peered":"2021-03-15T22:19:40.040174+0000","last_clean":"2021-03-15T22:19:40.040174+0000","last_became_active":"2021-03-15T22:17:58.010485+0000","last_became_peered":"2021-03-15T22:17:58.010485+0000","last_unstale":"2021-03-15T22:19:40.040174+0000","last_undegraded":"2021-03-15T22:19:40.040174+0000","last_fullsized":"2021-03-15T22:19:40.040174+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996494+0000","last_change":"2021-03-15T22:19:24.902865+0000","last_active":"2021-03-15T22:19:30.996494+0000","last_peered":"2021-03-15T22:19:30.996494+0000","last_clean":"2021-03-15T22:19:30.996494+0000","last_became_active":"2021-03-15T22:17:53.967568+0000","last_became_peered":"2021-03-15T22:17:53.967568+0000","last_unstale":"2021-03-15T22:19:30.996494+0000","last_undegraded":"2021-03-15T22:19:30.996494+0000","last_fullsized":"2021-03-15T22:19:30.996494+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.902804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.902804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.945166+0000","last_change":"2021-03-15T22:19:10.132572+0000","last_active":"2021-03-15T22:19:31.945166+0000","last_peered":"2021-03-15T22:19:31.945166+0000","last_clean":"2021-03-15T22:19:31.945166+0000","last_became_active":"2021-03-15T22:19:02.092703+0000","last_became_peered":"2021-03-15T22:19:02.092703+0000","last_unstale":"2021-03-15T22:19:31.945166+0000","last_undegraded":"2021-03-15T22:19:31.945166+0000","last_fullsized":"2021-03-15T22:19:31.945166+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:19:10.132514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:10.132514+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.997126+0000","last_change":"2021-03-15T22:19:25.878071+0000","last_active":"2021-03-15T22:19:30.997126+0000","last_peered":"2021-03-15T22:19:30.997126+0000","last_clean":"2021-03-15T22:19:30.997126+0000","last_became_active":"2021-03-15T22:18:00.287019+0000","last_became_peered":"2021-03-15T22:18:00.287019+0000","last_unstale":"2021-03-15T22:19:30.997126+0000","last_undegraded":"2021-03-15T22:19:30.997126+0000","last_fullsized":"2021-03-15T22:19:30.997126+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.878008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.501010+0000","last_change":"2021-03-15T22:18:58.347698+0000","last_active":"2021-03-15T22:19:31.501010+0000","last_peered":"2021-03-15T22:19:31.501010+0000","last_clean":"2021-03-15T22:19:31.501010+0000","last_became_active":"2021-03-15T22:17:58.008735+0000","last_became_peered":"2021-03-15T22:17:58.008735+0000","last_unstale":"2021-03-15T22:19:31.501010+0000","last_undegraded":"2021-03-15T22:19:31.501010+0000","last_fullsized":"2021-03-15T22:19:31.501010+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:58.347616+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:18:58.347616+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"39","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994474+0000","last_change":"2021-03-15T22:17:54.285942+0000","last_active":"2021-03-15T22:19:30.994474+0000","last_peered":"2021-03-15T22:19:30.994474+0000","last_clean":"2021-03-15T22:19:30.994474+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:19:30.994474+0000","last_undegraded":"2021-03-15T22:19:30.994474+0000","last_fullsized":"2021-03-15T22:19:30.994474+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:17:52.953223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"64","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995308+0000","last_change":"2021-03-15T22:17:55.992532+0000","last_active":"2021-03-15T22:19:30.995308+0000","last_peered":"2021-03-15T22:19:30.995308+0000","last_clean":"2021-03-15T22:19:30.995308+0000","last_became_active":"2021-03-15T22:17:55.992213+0000","last_became_peered":"2021-03-15T22:17:55.992213+0000","last_unstale":"2021-03-15T22:19:30.995308+0000","last_undegraded":"2021-03-15T22:19:30.995308+0000","last_fullsized":"2021-03-15T22:19:30.995308+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"38","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.945270+0000","last_change":"2021-03-15T22:19:22.172668+0000","last_active":"2021-03-15T22:19:31.945270+0000","last_peered":"2021-03-15T22:19:31.945270+0000","last_clean":"2021-03-15T22:19:31.945270+0000","last_became_active":"2021-03-15T22:18:00.019205+0000","last_became_peered":"2021-03-15T22:18:00.019205+0000","last_unstale":"2021-03-15T22:19:31.945270+0000","last_undegraded":"2021-03-15T22:19:31.945270+0000","last_fullsized":"2021-03-15T22:19:31.945270+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.172605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"40","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.944782+0000","last_change":"2021-03-15T22:19:13.514504+0000","last_active":"2021-03-15T22:19:31.944782+0000","last_peered":"2021-03-15T22:19:31.944782+0000","last_clean":"2021-03-15T22:19:31.944782+0000","last_became_active":"2021-03-15T22:17:58.013512+0000","last_became_peered":"2021-03-15T22:17:58.013512+0000","last_unstale":"2021-03-15T22:19:31.944782+0000","last_undegraded":"2021-03-15T22:19:31.944782+0000","last_fullsized":"2021-03-15T22:19:31.944782+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:13.514438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.514438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"43","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:31.885774+0000","last_change":"2021-03-15T22:19:18.591404+0000","last_active":"2021-03-15T22:19:31.885774+0000","last_peered":"2021-03-15T22:19:31.885774+0000","last_clean":"2021-03-15T22:19:31.885774+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:19:31.885774+0000","last_undegraded":"2021-03-15T22:19:31.885774+0000","last_fullsized":"2021-03-15T22:19:31.885774+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.591331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.591331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"65'21","reported_seq":"73","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995325+0000","last_change":"2021-03-15T22:17:55.992521+0000","last_active":"2021-03-15T22:19:30.995325+0000","last_peered":"2021-03-15T22:19:30.995325+0000","last_clean":"2021-03-15T22:19:30.995325+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:19:30.995325+0000","last_undegraded":"2021-03-15T22:19:30.995325+0000","last_fullsized":"2021-03-15T22:19:30.995325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"33","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.995424+0000","last_change":"2021-03-15T22:18:00.019688+0000","last_active":"2021-03-15T22:19:30.995424+0000","last_peered":"2021-03-15T22:19:30.995424+0000","last_clean":"2021-03-15T22:19:30.995424+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:19:30.995424+0000","last_undegraded":"2021-03-15T22:19:30.995424+0000","last_fullsized":"2021-03-15T22:19:30.995424+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"61'2","reported_seq":"63","reported_epoch":"87","state":"active+clean","last_fresh":"2021-03-15T22:19:40.042461+0000","last_change":"2021-03-15T22:19:12.158924+0000","last_active":"2021-03-15T22:19:40.042461+0000","last_peered":"2021-03-15T22:19:40.042461+0000","last_clean":"2021-03-15T22:19:40.042461+0000","last_became_active":"2021-03-15T22:17:58.010539+0000","last_became_peered":"2021-03-15T22:17:58.010539+0000","last_unstale":"2021-03-15T22:19:40.042461+0000","last_undegraded":"2021-03-15T22:19:40.042461+0000","last_fullsized":"2021-03-15T22:19:40.042461+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:12.158857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:12.158857+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"44","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.996941+0000","last_change":"2021-03-15T22:18:57.650089+0000","last_active":"2021-03-15T22:19:30.996941+0000","last_peered":"2021-03-15T22:19:30.996941+0000","last_clean":"2021-03-15T22:19:30.996941+0000","last_became_active":"2021-03-15T22:17:53.976045+0000","last_became_peered":"2021-03-15T22:17:53.976045+0000","last_unstale":"2021-03-15T22:19:30.996941+0000","last_undegraded":"2021-03-15T22:19:30.996941+0000","last_fullsized":"2021-03-15T22:19:30.996941+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:57.650003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:57.650003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"58","reported_epoch":"86","state":"active+clean","last_fresh":"2021-03-15T22:19:30.994884+0000","last_change":"2021-03-15T22:17:55.986249+0000","last_active":"2021-03-15T22:19:30.994884+0000","last_peered":"2021-03-15T22:19:30.994884+0000","last_clean":"2021-03-15T22:19:30.994884+0000","last_became_active":"2021-03-15T22:17:55.986062+0000","last_became_peered":"2021-03-15T22:17:55.986062+0000","last_unstale":"2021-03-15T22:19:30.994884+0000","last_undegraded":"2021-03-15T22:19:30.994884+0000","last_fullsized":"2021-03-15T22:19:30.994884+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16,"ondisk_log_size":16,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1069,"num_read_kb":574,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":556,"ondisk_log_size":556,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":115,"num_read_kb":110,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299419,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6452,"kb_used_data":876,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738700,"statfs":{"total":95995035648,"available":95988428800,"internally_reserved":0,"allocated":897024,"data_stored":368178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.70999999999999996,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.503,"5min":0.503,"15min":0.503},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.61799999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.60899999999999999,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.47899999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48399999999999999,"5min":0.48399999999999999,"15min":0.48399999999999999},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.48799999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":2.0339999999999998,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.497},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":2.1160000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.433}]},{"osd":5,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.59099999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.58199999999999996}]}]},{"osd":6,"up_from":45,"seq":193273528350,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6484,"kb_used_data":908,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738668,"statfs":{"total":95995035648,"available":95988396032,"internally_reserved":0,"allocated":929792,"data_stored":368758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48899999999999999,"15min":0.49299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63200000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.496,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67500000000000004,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.47699999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.36099999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.48199999999999998,"15min":0.47799999999999998},"min":{"1min":0.40400000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.629,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.377}]},{"osd":2,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.501,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.68600000000000005,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.502,"15min":0.502},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.624,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.624,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69499999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.45600000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.39300000000000002},"min":{"1min":0.30199999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.433,"15min":0.42699999999999999},"min":{"1min":0.32100000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.49399999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.441,"15min":0.433},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.498,"5min":0.46500000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.621}]},{"osd":7,"last update":"Mon Mar 15 22:18:52 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.75,"5min":0.75,"15min":0.75},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.52900000000000003,"15min":0.52900000000000003},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.73499999999999999,"5min":0.73499999999999999,"15min":0.73499999999999999},"last":0.59799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574896,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7100,"kb_used_data":1012,"kb_used_omap":0,"kb_used_meta":6080,"kb_avail":93738052,"statfs":{"total":95995035648,"available":95987765248,"internally_reserved":0,"allocated":1036288,"data_stored":373154,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6225920},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:13 2021","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38400000000000001,"15min":0.35099999999999998},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66400000000000003,"5min":3.387,"15min":3.387},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.377},"min":{"1min":0.23699999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.71899999999999997,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.37}]},{"osd":2,"last update":"Mon Mar 15 22:19:36 2021","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.42199999999999999,"15min":0.41699999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.78700000000000003,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41699999999999998,"15min":0.42099999999999999},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64400000000000002,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.314}]},{"osd":3,"last update":"Mon Mar 15 22:19:44 2021","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.434,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54100000000000004,"5min":1.796,"15min":1.796},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.48299999999999998,"15min":0.495},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.627,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.47699999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:19:00 2021","interfaces":[{"interface":"back","average":{"1min":0.59399999999999997,"5min":0.58599999999999997,"15min":0.58499999999999996},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":1.2789999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.76100000000000001,"5min":0.65300000000000002,"15min":0.63500000000000001},"min":{"1min":0.41899999999999998,"5min":0.41899999999999998,"15min":0.41899999999999998},"max":{"1min":4.3499999999999996,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.48599999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:19:16 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.65200000000000002,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.94199999999999995,"5min":2.153,"15min":2.153},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.151,"15min":1.2450000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95099999999999996,"5min":15.472,"15min":15.472},"last":0.59999999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:19:36 2021","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.67700000000000005,"15min":0.69399999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.879,"5min":1.75,"15min":1.75},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.64100000000000001,"15min":0.65300000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.87,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.54300000000000004}]},{"osd":7,"last update":"Mon Mar 15 22:18:53 2021","interfaces":[{"interface":"back","average":{"1min":1.0369999999999999,"5min":1.0369999999999999,"15min":1.0369999999999999},"min":{"1min":0.42699999999999999,"5min":0.42699999999999999,"15min":0.42699999999999999},"max":{"1min":8.8100000000000005,"5min":8.8100000000000005,"15min":8.8100000000000005},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.325,"5min":1.325,"15min":1.325},"last":0.51600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6776,"kb_used_data":880,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738376,"statfs":{"total":95995035648,"available":95988097024,"internally_reserved":0,"allocated":901120,"data_stored":368971,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:19:12 2021","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.253,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.38200000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59299999999999997,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.28199999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:19:29 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.39700000000000002,"15min":0.40999999999999998},"min":{"1min":0.24199999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.42899999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.39200000000000002,"15min":0.40400000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.442,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.247}]},{"osd":3,"last update":"Mon Mar 15 22:18:48 2021","interfaces":[{"interface":"back","average":{"1min":0.375,"5min":0.432,"15min":0.442},"min":{"1min":0.24199999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.54700000000000004,"5min":0.60099999999999998,"15min":0.60099999999999998},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.442,"15min":0.45000000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.53900000000000003,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.317}]},{"osd":4,"last update":"Mon Mar 15 22:19:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.55600000000000005,"15min":0.56899999999999995},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.73599999999999999,"5min":0.90700000000000003,"15min":0.90700000000000003},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.60499999999999998,"15min":0.61499999999999999},"min":{"1min":0.40400000000000003,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":0.72099999999999997,"5min":0.91700000000000004,"15min":0.91700000000000004},"last":0.50900000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:19:18 2021","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70199999999999996,"5min":0.871,"15min":0.871},"last":0.51900000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:19:31 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.437},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.53300000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:18:52 2021","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.57299999999999995,"15min":0.57299999999999995},"min":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.45600000000000002}]}]},{"osd":2,"up_from":18,"seq":77309411372,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6520,"kb_used_data":816,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93738632,"statfs":{"total":95995035648,"available":95988359168,"internally_reserved":0,"allocated":835584,"data_stored":366396,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.35399999999999998,"15min":0.35999999999999999},"min":{"1min":0.20799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.25700000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.377,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.191,"15min":0.191},"max":{"1min":0.53600000000000003,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.27000000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.38800000000000001},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55200000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.50700000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.29699999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:18:44 2021","interfaces":[{"interface":"back","average":{"1min":0.373,"5min":0.441,"15min":0.45200000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.501,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.55800000000000005,"5min":0.874,"15min":0.874},"last":0.314}]},{"osd":4,"last update":"Mon Mar 15 22:18:59 2021","interfaces":[{"interface":"back","average":{"1min":0.54600000000000004,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.69199999999999995,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.56100000000000005,"15min":0.56899999999999995},"min":{"1min":0.38400000000000001,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.73399999999999999,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.498}]},{"osd":5,"last update":"Mon Mar 15 22:19:19 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69799999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.78000000000000003,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.56699999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.77300000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.51200000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.54800000000000004},"min":{"1min":0.41399999999999998,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58099999999999996,"15min":0.58099999999999996},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.79100000000000004,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.52300000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149736,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6872,"kb_used_data":976,"kb_used_omap":0,"kb_used_meta":5888,"kb_avail":93738280,"statfs":{"total":95995035648,"available":95987998720,"internally_reserved":0,"allocated":999424,"data_stored":371551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6029312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.36199999999999999,"15min":0.371},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.42499999999999999,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.32400000000000001,"5min":0.38700000000000001,"15min":0.39700000000000002},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.47899999999999998,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.27800000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.36199999999999999,"5min":0.42599999999999999,"15min":0.436},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.52300000000000002,"5min":0.621,"15min":0.621},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.33800000000000002,"5min":0.42199999999999999,"15min":0.436},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.42699999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.33600000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:18:47 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.42699999999999999,"15min":0.44},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.49099999999999999,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.39700000000000002,"15min":0.40500000000000003},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54100000000000004,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.27000000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:19:01 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.50900000000000001,"15min":0.501},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":0.85299999999999998,"5min":0.85299999999999998,"15min":0.85299999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.84299999999999997,"5min":0.84299999999999997,"15min":0.84299999999999997},"last":0.66400000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63800000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.621},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73899999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.53400000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:19:33 2021","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58499999999999996,"15min":0.58999999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.86699999999999999,"5min":0.88,"15min":0.88},"last":0.56699999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.89800000000000002,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.377,"5min":0.377,"15min":0.377},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.69399999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986213,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6768,"kb_used_data":936,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93738384,"statfs":{"total":95995035648,"available":95988105216,"internally_reserved":0,"allocated":958464,"data_stored":370199,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.51800000000000002,"15min":0.52600000000000002},"min":{"1min":0.34100000000000003,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":0.68600000000000005,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.49399999999999999,"15min":0.497},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.47499999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.51100000000000001,"15min":0.52100000000000002},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66000000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.48999999999999999,"5min":0.52500000000000002,"15min":0.53100000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63400000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.49299999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.51700000000000002,"15min":0.52700000000000002},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.64300000000000002,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.50700000000000001,"15min":0.51200000000000001},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.502}]},{"osd":3,"last update":"Mon Mar 15 22:19:04 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.55900000000000005,"15min":0.56699999999999995},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.67500000000000004,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.57499999999999996,"15min":0.58699999999999997},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.68100000000000005,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.51100000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:19:19 2021","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.379},{"interface":"front","average":{"1min":0.436,"5min":0.40699999999999997,"15min":0.40200000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.34999999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52200000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:18:49 2021","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.73699999999999999,"5min":0.73699999999999999,"15min":0.73699999999999999},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.72799999999999998},"last":0.40100000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757282,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":6384,"kb_used_data":808,"kb_used_omap":0,"kb_used_meta":5568,"kb_avail":93738768,"statfs":{"total":95995035648,"available":95988498432,"internally_reserved":0,"allocated":827392,"data_stored":365165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5701632},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47799999999999998,"15min":0.47899999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59299999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.755,"15min":0.755},"last":0.48599999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73399999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.439}]},{"osd":2,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.50700000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69999999999999996,"5min":0.745,"15min":0.745},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.33200000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.376,"15min":0.36399999999999999},"min":{"1min":0.30099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.314,"5min":0.25,"15min":0.25},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.47199999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:19:35 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.44900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64000000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.40400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70199999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.54700000000000004}]},{"osd":7,"last update":"Mon Mar 15 22:18:50 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.51000000000000001,"15min":0.51000000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.52200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":12926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":5792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:19:51.186 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2021-03-15T22:19:51.187 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump 2021-03-15T22:19:51.467 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:51.862 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[56514]: cluster 2021-03-15T22:19:50.159699+0000 osd.0 (osd. 2021-03-15T22:19:51.863 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[56514]: 0) 28 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:51.863 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[56514]: cluster 2021-03-15T22:19:50.159772+0000 osd.0 (osd.0) 29 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:51.863 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[56514]: cluster 2021-03-15T22:19:50.171155+0000 osd.4 (osd.4) 17 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:19:51.863 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[56514]: audit 2021-03-15T22:19:50.424068+0000 mgr.y (mgr.14144) 354 : audit [DBG] from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:51.864 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[56514]: cluster 2021-03-15T22:19:50.972862+0000 mgr.y (mgr.14144) 355 : cluster [DBG] pgmap v253: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:51.864 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[56514]: cluster 2021-03-15T22:19:50.985957+0000 mon.a (mon. 2021-03-15T22:19:51.864 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[56514]: 0) 591 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2021-03-15T22:19:51.864 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[62156]: cluster 2021-03-15T22:19:50.159699+0000 osd.0 (osd.0 2021-03-15T22:19:51.865 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[62156]: ) 28 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:51.865 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[62156]: cluster 2021-03-15T22:19:50.159772+0000 osd.0 (osd.0) 29 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:51.865 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[62156]: cluster 2021-03-15T22:19:50.171155+0000 osd.4 (osd.4) 17 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:19:51.865 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[62156]: audit 2021-03-15T22:19:50.424068+0000 mgr.y (mgr.14144) 354 : audit [DBG] 2021-03-15T22:19:51.866 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[62156]: from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:51.866 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[62156]: cluster 2021-03-15T22:19:50.972862+0000 mgr.y (mgr.14144) 355 : cluster 2021-03-15T22:19:51.866 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[62156]: [DBG] pgmap v253: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:51.866 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[62156]: cluster 2021-03-15T22:19:51.866 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[62156]: 2021-03-15T22:19:50.985957+0000 mon.a (mon. 2021-03-15T22:19:51.867 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:51 smithi076 conmon[62156]: 0) 591 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2021-03-15T22:19:52.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: cluster 2021-03-15T22:19:50.159699+0000 osd.0 (osd.0) 28 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:52.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: cluster 2021-03-15T22:19:50.159772+0000 osd.0 (osd.0) 29 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:52.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: cluster 2021-03-15T22:19:50.171155+0000 osd.4 (osd. 2021-03-15T22:19:52.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: 4) 17 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:19:52.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: audit 2021-03-15T22:19:52.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: 2021-03-15T22:19:50.424068+0000 mgr.y (mgr. 2021-03-15T22:19:52.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: 14144) 354 : audit [DBG] from='client.14661 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:19:52.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:19:52.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: -15T22:19:50.972862+0000 mgr.y 2021-03-15T22:19:52.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: (mgr.14144) 355 : cluster [DBG] pgmap v253: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:19:52.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: cluster 2021-03-15T22:19:52.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: 2021-03-15 2021-03-15T22:19:52.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: T22:19:50.985957+0000 mon.a (mon 2021-03-15T22:19:52.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:51 smithi125 conmon[56739]: .0) 591 : cluster [DBG] osdmap e89: 8 total, 8 up, 8 in 2021-03-15T22:19:52.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:52 smithi076 conmon[56514]: cluster 2021-03-15T22:19:51.938120+0000 osd.7 (osd.7) 22 : 2021-03-15T22:19:52.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:52 smithi076 conmon[56514]: cluster [DBG] 5.9 deep-scrub ok 2021-03-15T22:19:52.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:52 smithi076 conmon[62156]: cluster 2021-03-15T22:19:51.938120+0000 osd.7 (osd.7) 22 : 2021-03-15T22:19:52.790 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:52 smithi076 conmon[62156]: cluster [DBG] 5.9 deep-scrub ok 2021-03-15T22:19:53.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:52 smithi125 conmon[56739]: cluster 2021-03-15T22:19:51.938120+0000 osd.7 (osd.7) 22 : cluster [DBG] 5.9 deep-scrub ok 2021-03-15T22:19:53.136 INFO:teuthology.orchestra.run.smithi076.stdout:epoch 89 2021-03-15T22:19:53.136 INFO:teuthology.orchestra.run.smithi076.stdout:fsid 8a572802-85db-11eb-9091-001a4aab830c 2021-03-15T22:19:53.136 INFO:teuthology.orchestra.run.smithi076.stdout:created 2021-03-15T22:13:20.370795+0000 2021-03-15T22:19:53.137 INFO:teuthology.orchestra.run.smithi076.stdout:modified 2021-03-15T22:19:50.980298+0000 2021-03-15T22:19:53.137 INFO:teuthology.orchestra.run.smithi076.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2021-03-15T22:19:53.137 INFO:teuthology.orchestra.run.smithi076.stdout:crush_version 18 2021-03-15T22:19:53.137 INFO:teuthology.orchestra.run.smithi076.stdout:full_ratio 0.95 2021-03-15T22:19:53.138 INFO:teuthology.orchestra.run.smithi076.stdout:backfillfull_ratio 0.9 2021-03-15T22:19:53.138 INFO:teuthology.orchestra.run.smithi076.stdout:nearfull_ratio 0.85 2021-03-15T22:19:53.138 INFO:teuthology.orchestra.run.smithi076.stdout:require_min_compat_client luminous 2021-03-15T22:19:53.138 INFO:teuthology.orchestra.run.smithi076.stdout:min_compat_client luminous 2021-03-15T22:19:53.139 INFO:teuthology.orchestra.run.smithi076.stdout:require_osd_release quincy 2021-03-15T22:19:53.139 INFO:teuthology.orchestra.run.smithi076.stdout:stretch_mode_enabled false 2021-03-15T22:19:53.139 INFO:teuthology.orchestra.run.smithi076.stdout:pool 1 'device_health_metrics' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 23 flags hashpspool stripe_width 0 pg_num_min 1 application mgr_devicehealth 2021-03-15T22:19:53.139 INFO:teuthology.orchestra.run.smithi076.stdout:pool 2 'iscsi' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 56 flags hashpspool stripe_width 0 application rbd 2021-03-15T22:19:53.139 INFO:teuthology.orchestra.run.smithi076.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 58 flags hashpspool stripe_width 0 application rgw 2021-03-15T22:19:53.140 INFO:teuthology.orchestra.run.smithi076.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 60 flags hashpspool stripe_width 0 application rgw 2021-03-15T22:19:53.140 INFO:teuthology.orchestra.run.smithi076.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 62 flags hashpspool stripe_width 0 application rgw 2021-03-15T22:19:53.140 INFO:teuthology.orchestra.run.smithi076.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 65 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 8 application rgw 2021-03-15T22:19:53.140 INFO:teuthology.orchestra.run.smithi076.stdout:max_osd 8 2021-03-15T22:19:53.140 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176] [v2:172.21.15.76:6804/188228176,v1:172.21.15.76:6805/188228176] exists,up aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd 2021-03-15T22:19:53.141 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 up in weight 1 up_from 13 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6810/3924955423,v1:172.21.15.76:6811/3924955423] [v2:172.21.15.76:6812/3924955423,v1:172.21.15.76:6813/3924955423] exists,up 97bd3ffc-15c8-4a6f-b37a-217813def526 2021-03-15T22:19:53.141 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050] [v2:172.21.15.76:6820/1401263050,v1:172.21.15.76:6821/1401263050] exists,up a7f5f115-e69e-47d9-b6c3-ff386ddfaa05 2021-03-15T22:19:53.141 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 up in weight 1 up_from 26 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6826/3960968636,v1:172.21.15.76:6827/3960968636] [v2:172.21.15.76:6828/3960968636,v1:172.21.15.76:6829/3960968636] exists,up a0bb7f88-0114-47cb-a82c-d8fb002aac56 2021-03-15T22:19:53.141 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4 up in weight 1 up_from 31 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.125:6800/37386122,v1:172.21.15.125:6801/37386122] [v2:172.21.15.125:6802/37386122,v1:172.21.15.125:6803/37386122] exists,up b650611c-6101-47e8-9ca9-94e02cf8b237 2021-03-15T22:19:53.142 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 up in weight 1 up_from 38 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.125:6808/3610838738,v1:172.21.15.125:6809/3610838738] [v2:172.21.15.125:6810/3610838738,v1:172.21.15.125:6811/3610838738] exists,up 7d6ceff2-7093-4ab9-8b96-ade448093bf2 2021-03-15T22:19:53.142 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6 up in weight 1 up_from 45 up_thru 60 down_at 0 last_clean_interval [0,0) [v2:172.21.15.125:6816/171378275,v1:172.21.15.125:6817/171378275] [v2:172.21.15.125:6818/171378275,v1:172.21.15.125:6819/171378275] exists,up 98b72022-a9a7-4ee2-89a1-cac037de7afe 2021-03-15T22:19:53.142 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7 up in weight 1 up_from 52 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.125:6824/652003948,v1:172.21.15.125:6825/652003948] [v2:172.21.15.125:6826/652003948,v1:172.21.15.125:6827/652003948] exists,up f99ea650-b2dc-4d87-a44c-56a897edd614 2021-03-15T22:19:53.142 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 4.b [1,2] 2021-03-15T22:19:53.143 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 4.f [1,2] 2021-03-15T22:19:53.143 INFO:teuthology.orchestra.run.smithi076.stdout:pg_upmap_items 4.1b [1,5] 2021-03-15T22:19:53.143 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6800/2604045245 expires 2021-03-16T22:14:00.771702+0000 2021-03-15T22:19:53.143 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/4087324200 expires 2021-03-16T22:14:00.771702+0000 2021-03-15T22:19:53.144 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/4141143304 expires 2021-03-16T22:14:00.771702+0000 2021-03-15T22:19:53.144 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6801/2604045245 expires 2021-03-16T22:14:00.771702+0000 2021-03-15T22:19:53.144 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/2779879991 expires 2021-03-16T22:13:41.603652+0000 2021-03-15T22:19:53.144 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6801/2764179881 expires 2021-03-16T22:13:41.603652+0000 2021-03-15T22:19:53.144 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:6800/2764179881 expires 2021-03-16T22:13:41.603652+0000 2021-03-15T22:19:53.145 INFO:teuthology.orchestra.run.smithi076.stdout:blocklist 172.21.15.76:0/2483102204 expires 2021-03-16T22:13:41.603652+0000 2021-03-15T22:19:53.826 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6802/188228176,v1:172.21.15.76:6803/188228176] [v2:172.21.15.76:6804/188228176,v1:172.21.15.76:6805/188228176] exists,up aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd', 'osd.1 up in weight 1 up_from 13 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6810/3924955423,v1:172.21.15.76:6811/3924955423] [v2:172.21.15.76:6812/3924955423,v1:172.21.15.76:6813/3924955423] exists,up 97bd3ffc-15c8-4a6f-b37a-217813def526', 'osd.2 up in weight 1 up_from 18 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6818/1401263050,v1:172.21.15.76:6819/1401263050] [v2:172.21.15.76:6820/1401263050,v1:172.21.15.76:6821/1401263050] exists,up a7f5f115-e69e-47d9-b6c3-ff386ddfaa05', 'osd.3 up in weight 1 up_from 26 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.76:6826/3960968636,v1:172.21.15.76:6827/3960968636] [v2:172.21.15.76:6828/3960968636,v1:172.21.15.76:6829/3960968636] exists,up a0bb7f88-0114-47cb-a82c-d8fb002aac56', 'osd.4 up in weight 1 up_from 31 up_thru 81 down_at 0 last_clean_interval [0,0) [v2:172.21.15.125:6800/37386122,v1:172.21.15.125:6801/37386122] [v2:172.21.15.125:6802/37386122,v1:172.21.15.125:6803/37386122] exists,up b650611c-6101-47e8-9ca9-94e02cf8b237', 'osd.5 up in weight 1 up_from 38 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.125:6808/3610838738,v1:172.21.15.125:6809/3610838738] [v2:172.21.15.125:6810/3610838738,v1:172.21.15.125:6811/3610838738] exists,up 7d6ceff2-7093-4ab9-8b96-ade448093bf2', 'osd.6 up in weight 1 up_from 45 up_thru 60 down_at 0 last_clean_interval [0,0) [v2:172.21.15.125:6816/171378275,v1:172.21.15.125:6817/171378275] [v2:172.21.15.125:6818/171378275,v1:172.21.15.125:6819/171378275] exists,up 98b72022-a9a7-4ee2-89a1-cac037de7afe', 'osd.7 up in weight 1 up_from 52 up_thru 62 down_at 0 last_clean_interval [0,0) [v2:172.21.15.125:6824/652003948,v1:172.21.15.125:6825/652003948] [v2:172.21.15.125:6826/652003948,v1:172.21.15.125:6827/652003948] exists,up f99ea650-b2dc-4d87-a44c-56a897edd614'] 2021-03-15T22:19:53.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon mon.a config show 2021-03-15T22:19:54.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:53 smithi076 conmon[56514]: cluster 2021-03-15T22:19:52.973331+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v255: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:19:54.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:53 smithi076 conmon[56514]: audit 2021-03-15T22:19:53.134670+0000 mon.a (mon.0) 592 : audit [DBG] from='client.? 172.21.15.76:0/3616942656' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-03-15T22:19:54.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:53 smithi076 conmon[56514]: cluster 2021-03-15T22:19:53.135001+0000 osd.4 (osd.4) 18 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:19:54.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:53 smithi076 conmon[62156]: cluster 2021-03-15T22:19:52.973331+0000 mgr.y ( 2021-03-15T22:19:54.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:53 smithi076 conmon[62156]: mgr.14144) 356 : cluster [DBG] pgmap v255: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:19:54.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:53 smithi076 conmon[62156]: audit 2021-03-15T22:19:53.134670+0000 2021-03-15T22:19:54.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:53 smithi076 conmon[62156]: mon.a (mon.0) 592 : audit [DBG] from='client.? 172.21.15.76:0/3616942656' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-03-15T22:19:54.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:53 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:19:54.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:53 smithi076 conmon[62156]: 19:53.135001+0000 osd.4 (osd.4 2021-03-15T22:19:54.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:53 smithi076 conmon[62156]: ) 18 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:19:54.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:53 smithi125 conmon[56739]: cluster 2021-03-15T22:19:52.973331+0000 mgr.y (mgr.14144) 356 : cluster [DBG] pgmap v255: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:19:54.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:53 smithi125 conmon[56739]: audit 2021-03-15T22:19:53.134670+0000 mon.a (mon.0) 592 : audit [DBG] from='client.? 172.21.15.76:0/3616942656' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2021-03-15T22:19:54.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:53 smithi125 conmon[56739]: cluster 2021-03-15T22:19:53.135001 2021-03-15T22:19:54.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:53 smithi125 conmon[56739]: +0000 osd.4 (osd.4) 18 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:19:54.107 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:55.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:54 smithi076 conmon[56514]: cluster 2021-03-15T22:19:53.845910+0000 osd.3 2021-03-15T22:19:55.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:54 smithi076 conmon[56514]: (osd.3) 29 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:19:55.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:54 smithi076 conmon[62156]: cluster 2021-03-15T22:19:53.845910+0000 osd.3 2021-03-15T22:19:55.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:54 smithi076 conmon[62156]: (osd.3) 29 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:19:55.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:54 smithi125 conmon[56739]: cluster 2021-03-15T22:19:53.845910+0000 osd.3 (osd.3) 29 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:19:55.656 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:19:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "name": "mon.a", 2021-03-15T22:19:55.656 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster": "ceph", 2021-03-15T22:19:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2021-03-15T22:19:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "admin_socket_mode": "", 2021-03-15T22:19:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_client_required": "cephx, none", 2021-03-15T22:19:55.657 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_cluster_required": "cephx", 2021-03-15T22:19:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_debug": "true", 2021-03-15T22:19:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_mon_ticket_ttl": "43200.000000", 2021-03-15T22:19:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_service_required": "cephx", 2021-03-15T22:19:55.658 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_service_ticket_ttl": "3600.000000", 2021-03-15T22:19:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "auth_supported": "", 2021-03-15T22:19:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio": "true", 2021-03-15T22:19:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio_max_queue_depth": "1024", 2021-03-15T22:19:55.659 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio_poll_ms": "250", 2021-03-15T22:19:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_aio_reap_max": "16", 2021-03-15T22:19:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_async_discard": "false", 2021-03-15T22:19:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_block_size": "4096", 2021-03-15T22:19:55.660 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_aio": "false", 2021-03-15T22:19:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_aio_log_age": "5.000000", 2021-03-15T22:19:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2021-03-15T22:19:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_debug_inflight_ios": "false", 2021-03-15T22:19:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_enable_discard": "false", 2021-03-15T22:19:55.661 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_flock_retry": "3", 2021-03-15T22:19:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_flock_retry_interval": "0.100000", 2021-03-15T22:19:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_inject_crash": "0", 2021-03-15T22:19:55.662 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_inject_crash_flush_delay": "2", 2021-03-15T22:19:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_ioring": "false", 2021-03-15T22:19:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_ioring_hipri": "false", 2021-03-15T22:19:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_ioring_sqthread_poll": "false", 2021-03-15T22:19:55.663 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_nvme_unbind_from_kernel": "false", 2021-03-15T22:19:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: "bdev_type": "", 2021-03-15T22:19:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_alloc_size": "1048576", 2021-03-15T22:19:55.664 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_allocator": "hybrid", 2021-03-15T22:19:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_buffered_io": "false", 2021-03-15T22:19:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_check_for_zeros": "false", 2021-03-15T22:19:55.666 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_compact_log_sync": "false", 2021-03-15T22:19:55.667 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2021-03-15T22:19:55.667 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_log_compact_min_size": "16777216", 2021-03-15T22:19:55.667 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_log_replay_check_allocations": "true", 2021-03-15T22:19:55.667 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_max_log_runway": "4194304", 2021-03-15T22:19:55.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_max_prefetch": "1048576", 2021-03-15T22:19:55.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_min_flush_size": "524288", 2021-03-15T22:19:55.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_min_log_runway": "1048576", 2021-03-15T22:19:55.668 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_replay_recovery": "false", 2021-03-15T22:19:55.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_replay_recovery_disable_compact": "false", 2021-03-15T22:19:55.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_shared_alloc_size": "65536", 2021-03-15T22:19:55.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluefs_sync_write": "false", 2021-03-15T22:19:55.669 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2021-03-15T22:19:55.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2021-03-15T22:19:55.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2021-03-15T22:19:55.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_allocator": "hybrid", 2021-03-15T22:19:55.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2021-03-15T22:19:55.670 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2021-03-15T22:19:55.671 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2021-03-15T22:19:55.671 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bitmapallocator_span_size": "1024", 2021-03-15T22:19:55.671 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_blobid_prealloc": "10240", 2021-03-15T22:19:55.671 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_create": "true", 2021-03-15T22:19:55.672 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_db_create": "false", 2021-03-15T22:19:55.672 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_db_path": "", 2021-03-15T22:19:55.672 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_db_size": "0", 2021-03-15T22:19:55.672 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_path": "", 2021-03-15T22:19:55.672 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_preallocate_file": "false", 2021-03-15T22:19:55.673 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_size": "107374182400", 2021-03-15T22:19:55.673 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_wal_create": "false", 2021-03-15T22:19:55.673 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_wal_path": "", 2021-03-15T22:19:55.673 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_block_wal_size": "100663296", 2021-03-15T22:19:55.673 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bluefs": "true", 2021-03-15T22:19:55.674 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2021-03-15T22:19:55.674 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_bluefs_env_mirror": "false", 2021-03-15T22:19:55.674 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_autotune": "true", 2021-03-15T22:19:55.674 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_autotune_interval": "5.000000", 2021-03-15T22:19:55.675 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2021-03-15T22:19:55.675 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_kv_ratio": "0.450000", 2021-03-15T22:19:55.675 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_meta_ratio": "0.450000", 2021-03-15T22:19:55.675 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_size": "0", 2021-03-15T22:19:55.676 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_size_hdd": "1073741824", 2021-03-15T22:19:55.676 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_size_ssd": "3221225472", 2021-03-15T22:19:55.676 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_trim_interval": "0.050000", 2021-03-15T22:19:55.676 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_trim_max_skip_pinned": "64", 2021-03-15T22:19:55.676 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_cache_type": "2q", 2021-03-15T22:19:55.677 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_clone_cow": "true", 2021-03-15T22:19:55.677 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_algorithm": "snappy", 2021-03-15T22:19:55.677 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_max_blob_size": "0", 2021-03-15T22:19:55.677 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2021-03-15T22:19:55.678 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2021-03-15T22:19:55.678 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_min_blob_size": "0", 2021-03-15T22:19:55.678 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2021-03-15T22:19:55.678 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_min_blob_size_ssd": "8192", 2021-03-15T22:19:55.679 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_mode": "none", 2021-03-15T22:19:55.679 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_compression_required_ratio": "0.875000", 2021-03-15T22:19:55.679 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_csum_type": "crc32c", 2021-03-15T22:19:55.679 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_enforce_settings": "default", 2021-03-15T22:19:55.680 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_freelist": "false", 2021-03-15T22:19:55.680 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_fsck_abort": "false", 2021-03-15T22:19:55.680 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_inject_bug21040": "false", 2021-03-15T22:19:55.680 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2021-03-15T22:19:55.680 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_inject_read_err": "false", 2021-03-15T22:19:55.681 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_no_reuse_blocks": "false", 2021-03-15T22:19:55.681 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_omit_block_device_write": "false", 2021-03-15T22:19:55.681 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_omit_kv_commit": "false", 2021-03-15T22:19:55.681 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2021-03-15T22:19:55.682 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_prefill": "0.000000", 2021-03-15T22:19:55.682 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_prefragment_max": "1048576", 2021-03-15T22:19:55.682 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_random_read_err": "0.000000", 2021-03-15T22:19:55.682 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2021-03-15T22:19:55.683 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_small_allocations": "0", 2021-03-15T22:19:55.683 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2021-03-15T22:19:55.683 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_default_buffered_read": "true", 2021-03-15T22:19:55.683 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_default_buffered_write": "false", 2021-03-15T22:19:55.684 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_deferred_batch_ops": "0", 2021-03-15T22:19:55.684 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2021-03-15T22:19:55.684 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2021-03-15T22:19:55.684 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2021-03-15T22:19:55.685 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_max_size": "1200", 2021-03-15T22:19:55.685 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_min_size": "150", 2021-03-15T22:19:55.685 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_target_size": "500", 2021-03-15T22:19:55.685 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2021-03-15T22:19:55.685 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_freelist_blocks_per_key": "128", 2021-03-15T22:19:55.686 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2021-03-15T22:19:55.686 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2021-03-15T22:19:55.686 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mkfs": "true", 2021-03-15T22:19:55.686 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2021-03-15T22:19:55.687 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mount": "false", 2021-03-15T22:19:55.687 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_mount_deep": "false", 2021-03-15T22:19:55.687 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_umount": "false", 2021-03-15T22:19:55.687 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_on_umount_deep": "false", 2021-03-15T22:19:55.687 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_quick_fix_on_mount": "true", 2021-03-15T22:19:55.688 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_quick_fix_threads": "2", 2021-03-15T22:19:55.688 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2021-03-15T22:19:55.688 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_gc_enable_blob_threshold": "0", 2021-03-15T22:19:55.688 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_gc_enable_total_threshold": "0", 2021-03-15T22:19:55.688 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2021-03-15T22:19:55.689 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_ignore_data_csum": "false", 2021-03-15T22:19:55.689 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2021-03-15T22:19:55.689 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_kvbackend": "rocksdb", 2021-03-15T22:19:55.689 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_log_collection_list_age": "60.000000", 2021-03-15T22:19:55.690 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2021-03-15T22:19:55.690 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_log_op_age": "5.000000", 2021-03-15T22:19:55.690 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_alloc_size": "0", 2021-03-15T22:19:55.690 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_blob_size": "0", 2021-03-15T22:19:55.691 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_blob_size_hdd": "65536", 2021-03-15T22:19:55.691 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_blob_size_ssd": "65536", 2021-03-15T22:19:55.691 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_defer_interval": "3.000000", 2021-03-15T22:19:55.692 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_max_deferred_txc": "32", 2021-03-15T22:19:55.692 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_min_alloc_size": "0", 2021-03-15T22:19:55.692 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_min_alloc_size_hdd": "4096", 2021-03-15T22:19:55.692 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_min_alloc_size_ssd": "4096", 2021-03-15T22:19:55.692 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_nid_prealloc": "1024", 2021-03-15T22:19:55.693 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_prefer_deferred_size": "0", 2021-03-15T22:19:55.693 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2021-03-15T22:19:55.693 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2021-03-15T22:19:55.693 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_retry_disk_reads": "3", 2021-03-15T22:19:55.693 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_cf": "true", 2021-03-15T22:19:55.694 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L P", 2021-03-15T22:19:55.694 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_options": "compression=kNoCompression,max_write_buffer_number=4,min_write_buffer_number_to_merge=1,recycle_log_file_num=4,write_buffer_size=268435456,writable_file_max_buffer_size=0,compaction_readahead_size=2097152,max_background_compactions=2,max_total_wal_size=1073741824", 2021-03-15T22:19:55.694 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_rocksdb_options_annex": "", 2021-03-15T22:19:55.695 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_coremask": "0x1", 2021-03-15T22:19:55.695 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_io_sleep": "5", 2021-03-15T22:19:55.695 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_max_io_completion": "0", 2021-03-15T22:19:55.695 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_spdk_mem": "512", 2021-03-15T22:19:55.696 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_sync_submit_transaction": "false", 2021-03-15T22:19:55.696 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_bytes": "67108864", 2021-03-15T22:19:55.697 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_cost_per_io": "0", 2021-03-15T22:19:55.698 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2021-03-15T22:19:55.698 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2021-03-15T22:19:55.698 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2021-03-15T22:19:55.699 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_throttle_trace_rate": "0.000000", 2021-03-15T22:19:55.699 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_tracing": "false", 2021-03-15T22:19:55.699 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2021-03-15T22:19:55.699 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_volume_selection_reserved": "0", 2021-03-15T22:19:55.700 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2021-03-15T22:19:55.700 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2021-03-15T22:19:55.700 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_legacy_statfs": "true", 2021-03-15T22:19:55.701 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2021-03-15T22:19:55.701 INFO:teuthology.orchestra.run.smithi076.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2021-03-15T22:19:55.702 INFO:teuthology.orchestra.run.smithi076.stdout: "cephadm_path": "/usr/sbin/cephadm", 2021-03-15T22:19:55.702 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_action_update_interval": "2", 2021-03-15T22:19:55.702 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_directory_scan_interval": "10", 2021-03-15T22:19:55.702 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2021-03-15T22:19:55.703 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2021-03-15T22:19:55.703 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2021-03-15T22:19:55.703 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2021-03-15T22:19:55.703 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2021-03-15T22:19:55.703 INFO:teuthology.orchestra.run.smithi076.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2021-03-15T22:19:55.704 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_cluster_require_signatures": "false", 2021-03-15T22:19:55.704 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_cluster_require_version": "1", 2021-03-15T22:19:55.704 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_require_signatures": "false", 2021-03-15T22:19:55.704 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_require_version": "1", 2021-03-15T22:19:55.705 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_service_require_signatures": "false", 2021-03-15T22:19:55.705 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_service_require_version": "1", 2021-03-15T22:19:55.705 INFO:teuthology.orchestra.run.smithi076.stdout: "cephx_sign_messages": "true", 2021-03-15T22:19:55.705 INFO:teuthology.orchestra.run.smithi076.stdout: "chdir": "", 2021-03-15T22:19:55.706 INFO:teuthology.orchestra.run.smithi076.stdout: "client_acl_type": "", 2021-03-15T22:19:55.706 INFO:teuthology.orchestra.run.smithi076.stdout: "client_asio_thread_count": "2", 2021-03-15T22:19:55.707 INFO:teuthology.orchestra.run.smithi076.stdout: "client_cache_mid": "0.750000", 2021-03-15T22:19:55.707 INFO:teuthology.orchestra.run.smithi076.stdout: "client_cache_size": "16384", 2021-03-15T22:19:55.707 INFO:teuthology.orchestra.run.smithi076.stdout: "client_caps_release_delay": "5", 2021-03-15T22:19:55.707 INFO:teuthology.orchestra.run.smithi076.stdout: "client_check_pool_perm": "true", 2021-03-15T22:19:55.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client_debug_force_sync_read": "false", 2021-03-15T22:19:55.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client_debug_getattr_caps": "false", 2021-03-15T22:19:55.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client_debug_inject_tick_delay": "0", 2021-03-15T22:19:55.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client_die_on_failed_dentry_invalidate": "true", 2021-03-15T22:19:55.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client_die_on_failed_remount": "false", 2021-03-15T22:19:55.710 INFO:teuthology.orchestra.run.smithi076.stdout: "client_dirsize_rbytes": "true", 2021-03-15T22:19:55.710 INFO:teuthology.orchestra.run.smithi076.stdout: "client_force_lazyio": "false", 2021-03-15T22:19:55.711 INFO:teuthology.orchestra.run.smithi076.stdout: "client_fs": "", 2021-03-15T22:19:55.711 INFO:teuthology.orchestra.run.smithi076.stdout: "client_inject_fixed_oldest_tid": "false", 2021-03-15T22:19:55.711 INFO:teuthology.orchestra.run.smithi076.stdout: "client_inject_release_failure": "false", 2021-03-15T22:19:55.711 INFO:teuthology.orchestra.run.smithi076.stdout: "client_max_inline_size": "4096", 2021-03-15T22:19:55.712 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mds_namespace": "", 2021-03-15T22:19:55.712 INFO:teuthology.orchestra.run.smithi076.stdout: "client_metadata": "", 2021-03-15T22:19:55.712 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mount_gid": "-1", 2021-03-15T22:19:55.712 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mount_timeout": "300.000000", 2021-03-15T22:19:55.712 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mount_uid": "-1", 2021-03-15T22:19:55.713 INFO:teuthology.orchestra.run.smithi076.stdout: "client_mountpoint": "/", 2021-03-15T22:19:55.713 INFO:teuthology.orchestra.run.smithi076.stdout: "client_notify_timeout": "10", 2021-03-15T22:19:55.713 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc": "true", 2021-03-15T22:19:55.713 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_max_dirty": "104857600", 2021-03-15T22:19:55.714 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_max_dirty_age": "5.000000", 2021-03-15T22:19:55.714 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_max_objects": "1000", 2021-03-15T22:19:55.714 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_size": "209715200", 2021-03-15T22:19:55.714 INFO:teuthology.orchestra.run.smithi076.stdout: "client_oc_target_dirty": "8388608", 2021-03-15T22:19:55.714 INFO:teuthology.orchestra.run.smithi076.stdout: "client_permissions": "true", 2021-03-15T22:19:55.715 INFO:teuthology.orchestra.run.smithi076.stdout: "client_quota_df": "true", 2021-03-15T22:19:55.715 INFO:teuthology.orchestra.run.smithi076.stdout: "client_readahead_max_bytes": "0", 2021-03-15T22:19:55.715 INFO:teuthology.orchestra.run.smithi076.stdout: "client_readahead_max_periods": "4", 2021-03-15T22:19:55.715 INFO:teuthology.orchestra.run.smithi076.stdout: "client_readahead_min": "131072", 2021-03-15T22:19:55.716 INFO:teuthology.orchestra.run.smithi076.stdout: "client_reconnect_stale": "false", 2021-03-15T22:19:55.716 INFO:teuthology.orchestra.run.smithi076.stdout: "client_shutdown_timeout": "30", 2021-03-15T22:19:55.716 INFO:teuthology.orchestra.run.smithi076.stdout: "client_snapdir": ".snap", 2021-03-15T22:19:55.716 INFO:teuthology.orchestra.run.smithi076.stdout: "client_tick_interval": "1", 2021-03-15T22:19:55.716 INFO:teuthology.orchestra.run.smithi076.stdout: "client_trace": "", 2021-03-15T22:19:55.717 INFO:teuthology.orchestra.run.smithi076.stdout: "client_try_dentry_invalidate": "false", 2021-03-15T22:19:55.717 INFO:teuthology.orchestra.run.smithi076.stdout: "client_use_faked_inos": "false", 2021-03-15T22:19:55.717 INFO:teuthology.orchestra.run.smithi076.stdout: "client_use_random_mds": "false", 2021-03-15T22:19:55.717 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_graylog": "false", 2021-03-15T22:19:55.717 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_graylog_host": "127.0.0.1", 2021-03-15T22:19:55.718 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_graylog_port": "12201", 2021-03-15T22:19:55.718 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_monitors": "default=true", 2021-03-15T22:19:55.718 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_syslog": "false", 2021-03-15T22:19:55.718 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2021-03-15T22:19:55.719 INFO:teuthology.orchestra.run.smithi076.stdout: "clog_to_syslog_level": "info", 2021-03-15T22:19:55.719 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "-", 2021-03-15T22:19:55.719 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_network": "", 2021-03-15T22:19:55.719 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_network_interface": "", 2021-03-15T22:19:55.719 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zlib_isal": "false", 2021-03-15T22:19:55.720 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zlib_level": "5", 2021-03-15T22:19:55.720 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zlib_winsize": "-15", 2021-03-15T22:19:55.720 INFO:teuthology.orchestra.run.smithi076.stdout: "compressor_zstd_level": "1", 2021-03-15T22:19:55.720 INFO:teuthology.orchestra.run.smithi076.stdout: "container_image": "quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd", 2021-03-15T22:19:55.720 INFO:teuthology.orchestra.run.smithi076.stdout: "crash_dir": "/var/lib/ceph/crash", 2021-03-15T22:19:55.721 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_alien_op_num_threads": "6", 2021-03-15T22:19:55.721 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_osd_obc_lru_size": "10", 2021-03-15T22:19:55.721 INFO:teuthology.orchestra.run.smithi076.stdout: "crimson_osd_scheduler_concurrency": "0", 2021-03-15T22:19:55.721 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location": "", 2021-03-15T22:19:55.721 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location_hook": "", 2021-03-15T22:19:55.722 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_location_hook_timeout": "10", 2021-03-15T22:19:55.722 INFO:teuthology.orchestra.run.smithi076.stdout: "daemonize": "false", 2021-03-15T22:19:55.722 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_allow_any_pool_priority": "false", 2021-03-15T22:19:55.722 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_asok": "1/5", 2021-03-15T22:19:55.723 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_asok_assert_abort": "false", 2021-03-15T22:19:55.723 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_asserts_on_shutdown": "true", 2021-03-15T22:19:55.723 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_auth": "1/5", 2021-03-15T22:19:55.723 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_bdev": "1/3", 2021-03-15T22:19:55.724 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_bluefs": "1/5", 2021-03-15T22:19:55.724 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_bluestore": "1/5", 2021-03-15T22:19:55.724 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_buffer": "0/1", 2021-03-15T22:19:55.724 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_cephfs_mirror": "0/5", 2021-03-15T22:19:55.724 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_civetweb": "1/10", 2021-03-15T22:19:55.725 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_client": "0/5", 2021-03-15T22:19:55.725 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_compressor": "1/5", 2021-03-15T22:19:55.725 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_context": "0/1", 2021-03-15T22:19:55.725 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_crush": "1/1", 2021-03-15T22:19:55.726 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_crypto": "1/5", 2021-03-15T22:19:55.726 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_deliberately_leak_memory": "false", 2021-03-15T22:19:55.726 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_disable_randomized_ping": "false", 2021-03-15T22:19:55.726 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_dpdk": "1/5", 2021-03-15T22:19:55.727 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_eventtrace": "1/5", 2021-03-15T22:19:55.727 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_filer": "0/1", 2021-03-15T22:19:55.727 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_filestore": "1/3", 2021-03-15T22:19:55.727 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_finisher": "1/1", 2021-03-15T22:19:55.727 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_fuse": "1/5", 2021-03-15T22:19:55.728 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_heartbeat_testing_span": "0", 2021-03-15T22:19:55.728 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_heartbeatmap": "1/5", 2021-03-15T22:19:55.728 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_immutable_obj_cache": "0/5", 2021-03-15T22:19:55.728 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_javaclient": "1/5", 2021-03-15T22:19:55.729 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_journal": "1/3", 2021-03-15T22:19:55.729 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_journaler": "0/5", 2021-03-15T22:19:55.729 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_kstore": "1/5", 2021-03-15T22:19:55.729 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_leveldb": "4/5", 2021-03-15T22:19:55.729 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_lockdep": "0/1", 2021-03-15T22:19:55.730 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds": "1/5", 2021-03-15T22:19:55.730 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_balancer": "1/5", 2021-03-15T22:19:55.730 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_locker": "1/5", 2021-03-15T22:19:55.730 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_log": "1/5", 2021-03-15T22:19:55.730 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_log_expire": "1/5", 2021-03-15T22:19:55.731 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mds_migrator": "1/5", 2021-03-15T22:19:55.731 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_memdb": "4/5", 2021-03-15T22:19:55.731 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mgr": "1/5", 2021-03-15T22:19:55.731 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mgrc": "1/5", 2021-03-15T22:19:55.732 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_mon": "20/20", 2021-03-15T22:19:55.732 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_monc": "0/10", 2021-03-15T22:19:55.732 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_ms": "1/1", 2021-03-15T22:19:55.732 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_none": "0/5", 2021-03-15T22:19:55.732 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_objclass": "0/5", 2021-03-15T22:19:55.733 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_objectcacher": "0/5", 2021-03-15T22:19:55.733 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_objecter": "0/1", 2021-03-15T22:19:55.733 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_optracker": "0/5", 2021-03-15T22:19:55.733 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_osd": "1/5", 2021-03-15T22:19:55.733 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_paxos": "20/20", 2021-03-15T22:19:55.734 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_perfcounter": "1/5", 2021-03-15T22:19:55.734 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_prioritycache": "1/5", 2021-03-15T22:19:55.734 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rados": "0/5", 2021-03-15T22:19:55.734 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd": "0/5", 2021-03-15T22:19:55.735 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd_mirror": "0/5", 2021-03-15T22:19:55.735 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd_pwl": "0/5", 2021-03-15T22:19:55.735 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rbd_replay": "0/5", 2021-03-15T22:19:55.735 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_refs": "0/0", 2021-03-15T22:19:55.735 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_reserver": "1/1", 2021-03-15T22:19:55.736 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw": "1/5", 2021-03-15T22:19:55.736 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rgw_sync": "1/5", 2021-03-15T22:19:55.736 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_rocksdb": "4/5", 2021-03-15T22:19:55.736 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_striper": "0/1", 2021-03-15T22:19:55.737 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_test": "0/5", 2021-03-15T22:19:55.737 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_throttle": "1/1", 2021-03-15T22:19:55.737 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_timer": "0/1", 2021-03-15T22:19:55.737 INFO:teuthology.orchestra.run.smithi076.stdout: "debug_tp": "0/5", 2021-03-15T22:19:55.737 INFO:teuthology.orchestra.run.smithi076.stdout: "device_failure_prediction_mode": "none", 2021-03-15T22:19:55.738 INFO:teuthology.orchestra.run.smithi076.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2021-03-15T22:19:55.738 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2021-03-15T22:19:55.738 INFO:teuthology.orchestra.run.smithi076.stdout: "err_to_graylog": "false", 2021-03-15T22:19:55.738 INFO:teuthology.orchestra.run.smithi076.stdout: "err_to_stderr": "true", 2021-03-15T22:19:55.738 INFO:teuthology.orchestra.run.smithi076.stdout: "err_to_syslog": "false", 2021-03-15T22:19:55.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event_tracing": "false", 2021-03-15T22:19:55.739 INFO:teuthology.orchestra.run.smithi076.stdout: "fake_statfs_for_testing": "0", 2021-03-15T22:19:55.739 INFO:teuthology.orchestra.run.smithi076.stdout: "fatal_signal_handlers": "true", 2021-03-15T22:19:55.739 INFO:teuthology.orchestra.run.smithi076.stdout: "filer_max_purge_ops": "10", 2021-03-15T22:19:55.740 INFO:teuthology.orchestra.run.smithi076.stdout: "filer_max_truncate_ops": "128", 2021-03-15T22:19:55.740 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_apply_finisher_threads": "1", 2021-03-15T22:19:55.740 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_blackhole": "false", 2021-03-15T22:19:55.740 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_btrfs_clone_range": "true", 2021-03-15T22:19:55.740 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_btrfs_snap": "true", 2021-03-15T22:19:55.741 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_caller_concurrency": "10", 2021-03-15T22:19:55.741 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_collect_device_partition_information": "true", 2021-03-15T22:19:55.741 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_commit_timeout": "600.000000", 2021-03-15T22:19:55.741 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_inject_read_err": "false", 2021-03-15T22:19:55.741 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_omap_check": "false", 2021-03-15T22:19:55.742 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_random_read_err": "0.000000", 2021-03-15T22:19:55.742 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_debug_verify_split": "false", 2021-03-15T22:19:55.742 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_dump_file": "", 2021-03-15T22:19:55.742 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2021-03-15T22:19:55.743 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_expected_throughput_ops": "200.000000", 2021-03-15T22:19:55.743 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fadvise": "true", 2021-03-15T22:19:55.743 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fail_eio": "true", 2021-03-15T22:19:55.743 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fd_cache_shards": "16", 2021-03-15T22:19:55.743 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fd_cache_size": "128", 2021-03-15T22:19:55.744 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fiemap": "false", 2021-03-15T22:19:55.744 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fiemap_threshold": "4096", 2021-03-15T22:19:55.744 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_fsync_flushes_journal_data": "false", 2021-03-15T22:19:55.744 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_index_retry_probability": "0.000000", 2021-03-15T22:19:55.744 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_inject_stall": "0", 2021-03-15T22:19:55.745 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_journal_parallel": "false", 2021-03-15T22:19:55.745 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_journal_trailing": "false", 2021-03-15T22:19:55.745 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_journal_writeahead": "false", 2021-03-15T22:19:55.745 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_kill_at": "0", 2021-03-15T22:19:55.746 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_alloc_hint_size": "1048576", 2021-03-15T22:19:55.746 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size": "0", 2021-03-15T22:19:55.746 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2021-03-15T22:19:55.746 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size_other": "512", 2021-03-15T22:19:55.747 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2021-03-15T22:19:55.747 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs": "0", 2021-03-15T22:19:55.747 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2021-03-15T22:19:55.747 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs_other": "2", 2021-03-15T22:19:55.747 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_inline_xattrs_xfs": "10", 2021-03-15T22:19:55.748 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_sync_interval": "5.000000", 2021-03-15T22:19:55.748 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size": "0", 2021-03-15T22:19:55.748 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2021-03-15T22:19:55.748 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size_other": "1024", 2021-03-15T22:19:55.748 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2021-03-15T22:19:55.749 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_merge_threshold": "-10", 2021-03-15T22:19:55.749 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_min_sync_interval": "0.010000", 2021-03-15T22:19:55.749 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_odsync_write": "false", 2021-03-15T22:19:55.749 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_omap_backend": "rocksdb", 2021-03-15T22:19:55.750 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_omap_backend_path": "", 2021-03-15T22:19:55.750 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_omap_header_cache_size": "1024", 2021-03-15T22:19:55.750 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_ondisk_finisher_threads": "1", 2021-03-15T22:19:55.750 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_op_thread_suicide_timeout": "180", 2021-03-15T22:19:55.751 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_op_thread_timeout": "60", 2021-03-15T22:19:55.751 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_op_threads": "2", 2021-03-15T22:19:55.751 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_punch_hole": "false", 2021-03-15T22:19:55.751 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2021-03-15T22:19:55.751 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2021-03-15T22:19:55.752 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2021-03-15T22:19:55.752 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_high_threshhold": "0.900000", 2021-03-15T22:19:55.752 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_low_threshhold": "0.300000", 2021-03-15T22:19:55.752 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_bytes": "104857600", 2021-03-15T22:19:55.753 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2021-03-15T22:19:55.753 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2021-03-15T22:19:55.753 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2021-03-15T22:19:55.753 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_queue_max_ops": "50", 2021-03-15T22:19:55.753 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2021-03-15T22:19:55.754 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_seek_data_hole": "false", 2021-03-15T22:19:55.754 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_sloppy_crc": "false", 2021-03-15T22:19:55.754 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_sloppy_crc_block_size": "65536", 2021-03-15T22:19:55.754 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_splice": "false", 2021-03-15T22:19:55.755 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_split_multiple": "2", 2021-03-15T22:19:55.755 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_split_rand_factor": "20", 2021-03-15T22:19:55.755 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_update_to": "1000", 2021-03-15T22:19:55.755 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2021-03-15T22:19:55.755 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2021-03-15T22:19:55.756 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2021-03-15T22:19:55.756 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2021-03-15T22:19:55.756 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2021-03-15T22:19:55.756 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2021-03-15T22:19:55.756 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_enable": "true", 2021-03-15T22:19:55.757 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2021-03-15T22:19:55.757 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2021-03-15T22:19:55.757 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2021-03-15T22:19:55.757 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2021-03-15T22:19:55.758 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2021-03-15T22:19:55.758 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2021-03-15T22:19:55.758 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_xfs_extsize": "false", 2021-03-15T22:19:55.758 INFO:teuthology.orchestra.run.smithi076.stdout: "filestore_zfs_snap": "false", 2021-03-15T22:19:55.759 INFO:teuthology.orchestra.run.smithi076.stdout: "fio_dir": "/tmp/fio", 2021-03-15T22:19:55.759 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:19:55.759 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_allow_other": "true", 2021-03-15T22:19:55.759 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_atomic_o_trunc": "true", 2021-03-15T22:19:55.759 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_big_writes": "true", 2021-03-15T22:19:55.760 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_debug": "false", 2021-03-15T22:19:55.760 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_default_permissions": "false", 2021-03-15T22:19:55.760 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_disable_pagecache": "false", 2021-03-15T22:19:55.760 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_max_write": "0", 2021-03-15T22:19:55.761 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_multithreaded": "true", 2021-03-15T22:19:55.761 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_require_active_mds": "true", 2021-03-15T22:19:55.761 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_set_user_groups": "true", 2021-03-15T22:19:55.761 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_splice_move": "true", 2021-03-15T22:19:55.761 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_splice_read": "true", 2021-03-15T22:19:55.762 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_splice_write": "true", 2021-03-15T22:19:55.762 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_syncfs_on_mksnap": "true", 2021-03-15T22:19:55.762 INFO:teuthology.orchestra.run.smithi076.stdout: "fuse_use_invalidate_cb": "true", 2021-03-15T22:19:55.762 INFO:teuthology.orchestra.run.smithi076.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2021-03-15T22:19:55.763 INFO:teuthology.orchestra.run.smithi076.stdout: "gss_target_name": "ceph", 2021-03-15T22:19:55.763 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_file": "", 2021-03-15T22:19:55.763 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_inject_failure": "0", 2021-03-15T22:19:55.763 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_interval": "5", 2021-03-15T22:19:55.763 INFO:teuthology.orchestra.run.smithi076.stdout: "host": "", 2021-03-15T22:19:55.764 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2021-03-15T22:19:55.764 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_max_inflight_ops": "128", 2021-03-15T22:19:55.764 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_max_size": "1073741824", 2021-03-15T22:19:55.764 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2021-03-15T22:19:55.764 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_bps_burst": "0", 2021-03-15T22:19:55.765 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2021-03-15T22:19:55.765 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_bps_limit": "0", 2021-03-15T22:19:55.765 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_iops_burst": "0", 2021-03-15T22:19:55.765 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2021-03-15T22:19:55.766 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_iops_limit": "0", 2021-03-15T22:19:55.766 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2021-03-15T22:19:55.766 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2021-03-15T22:19:55.766 INFO:teuthology.orchestra.run.smithi076.stdout: "immutable_object_cache_watermark": "0.900000", 2021-03-15T22:19:55.766 INFO:teuthology.orchestra.run.smithi076.stdout: "inject_early_sigterm": "false", 2021-03-15T22:19:55.767 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_aio": "true", 2021-03-15T22:19:55.767 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_align_min_size": "65536", 2021-03-15T22:19:55.767 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_block_align": "true", 2021-03-15T22:19:55.767 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_block_size": "4096", 2021-03-15T22:19:55.767 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_dio": "true", 2021-03-15T22:19:55.768 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_discard": "false", 2021-03-15T22:19:55.768 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_force_aio": "false", 2021-03-15T22:19:55.768 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_ignore_corruption": "false", 2021-03-15T22:19:55.768 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_max_write_bytes": "10485760", 2021-03-15T22:19:55.769 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_max_write_entries": "100", 2021-03-15T22:19:55.769 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_replay_from": "0", 2021-03-15T22:19:55.769 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_high_multiple": "0.000000", 2021-03-15T22:19:55.769 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_high_threshhold": "0.900000", 2021-03-15T22:19:55.770 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_low_threshhold": "0.600000", 2021-03-15T22:19:55.770 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_throttle_max_multiple": "0.000000", 2021-03-15T22:19:55.770 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_write_header_frequency": "0", 2021-03-15T22:19:55.770 INFO:teuthology.orchestra.run.smithi076.stdout: "journal_zero_on_create": "false", 2021-03-15T22:19:55.770 INFO:teuthology.orchestra.run.smithi076.stdout: "journaler_prefetch_periods": "10", 2021-03-15T22:19:55.771 INFO:teuthology.orchestra.run.smithi076.stdout: "journaler_prezero_periods": "5", 2021-03-15T22:19:55.771 INFO:teuthology.orchestra.run.smithi076.stdout: "journaler_write_head_interval": "15", 2021-03-15T22:19:55.771 INFO:teuthology.orchestra.run.smithi076.stdout: "key": "", 2021-03-15T22:19:55.771 INFO:teuthology.orchestra.run.smithi076.stdout: "keyfile": "", 2021-03-15T22:19:55.771 INFO:teuthology.orchestra.run.smithi076.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2021-03-15T22:19:55.772 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_backend": "rocksdb", 2021-03-15T22:19:55.772 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_default_stripe_size": "65536", 2021-03-15T22:19:55.772 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_fsck_on_mount": "false", 2021-03-15T22:19:55.772 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_fsck_on_mount_deep": "true", 2021-03-15T22:19:55.773 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_max_bytes": "67108864", 2021-03-15T22:19:55.773 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_max_ops": "512", 2021-03-15T22:19:55.773 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_nid_prealloc": "1024", 2021-03-15T22:19:55.773 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_onode_map_size": "1024", 2021-03-15T22:19:55.773 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2021-03-15T22:19:55.774 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_sync_submit_transaction": "false", 2021-03-15T22:19:55.774 INFO:teuthology.orchestra.run.smithi076.stdout: "kstore_sync_transaction": "false", 2021-03-15T22:19:55.774 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_block_size": "65536", 2021-03-15T22:19:55.774 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_bloom_size": "0", 2021-03-15T22:19:55.774 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_cache_size": "536870912", 2021-03-15T22:19:55.775 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_compact_on_mount": "false", 2021-03-15T22:19:55.775 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_compression": "false", 2021-03-15T22:19:55.775 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_log": "", 2021-03-15T22:19:55.775 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_log_to_ceph_log": "true", 2021-03-15T22:19:55.776 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_max_open_files": "0", 2021-03-15T22:19:55.776 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_paranoid": "false", 2021-03-15T22:19:55.776 INFO:teuthology.orchestra.run.smithi076.stdout: "leveldb_write_buffer_size": "33554432", 2021-03-15T22:19:55.776 INFO:teuthology.orchestra.run.smithi076.stdout: "librados_thread_count": "2", 2021-03-15T22:19:55.776 INFO:teuthology.orchestra.run.smithi076.stdout: "lockdep": "false", 2021-03-15T22:19:55.777 INFO:teuthology.orchestra.run.smithi076.stdout: "lockdep_force_backtrace": "false", 2021-03-15T22:19:55.777 INFO:teuthology.orchestra.run.smithi076.stdout: "log_coarse_timestamps": "true", 2021-03-15T22:19:55.777 INFO:teuthology.orchestra.run.smithi076.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2021-03-15T22:19:55.777 INFO:teuthology.orchestra.run.smithi076.stdout: "log_flush_on_exit": "false", 2021-03-15T22:19:55.778 INFO:teuthology.orchestra.run.smithi076.stdout: "log_graylog_host": "127.0.0.1", 2021-03-15T22:19:55.778 INFO:teuthology.orchestra.run.smithi076.stdout: "log_graylog_port": "12201", 2021-03-15T22:19:55.778 INFO:teuthology.orchestra.run.smithi076.stdout: "log_max_new": "1000", 2021-03-15T22:19:55.778 INFO:teuthology.orchestra.run.smithi076.stdout: "log_max_recent": "10000", 2021-03-15T22:19:55.778 INFO:teuthology.orchestra.run.smithi076.stdout: "log_stderr_prefix": "debug ", 2021-03-15T22:19:55.779 INFO:teuthology.orchestra.run.smithi076.stdout: "log_stop_at_utilization": "0.970000", 2021-03-15T22:19:55.779 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_file": "true", 2021-03-15T22:19:55.779 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_graylog": "false", 2021-03-15T22:19:55.779 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_stderr": "false", 2021-03-15T22:19:55.779 INFO:teuthology.orchestra.run.smithi076.stdout: "log_to_syslog": "false", 2021-03-15T22:19:55.780 INFO:teuthology.orchestra.run.smithi076.stdout: "max_rotating_auth_attempts": "10", 2021-03-15T22:19:55.780 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_action_on_write_error": "1", 2021-03-15T22:19:55.780 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_alternate_name_max": "8192", 2021-03-15T22:19:55.780 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_asio_thread_count": "2", 2021-03-15T22:19:55.781 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_export_pin": "true", 2021-03-15T22:19:55.781 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_dirs": "true", 2021-03-15T22:19:55.781 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2021-03-15T22:19:55.781 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_interval": "5", 2021-03-15T22:19:55.781 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_fragment_size_max": "100000", 2021-03-15T22:19:55.782 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_idle_threshold": "0.000000", 2021-03-15T22:19:55.782 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_interval": "10", 2021-03-15T22:19:55.782 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_max": "-1", 2021-03-15T22:19:55.782 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_max_until": "-1", 2021-03-15T22:19:55.782 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_merge_size": "50", 2021-03-15T22:19:55.783 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_midchunk": "0.300000", 2021-03-15T22:19:55.783 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_min_rebalance": "0.100000", 2021-03-15T22:19:55.783 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_min_start": "0.200000", 2021-03-15T22:19:55.783 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_minchunk": "0.001000", 2021-03-15T22:19:55.784 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_mode": "0", 2021-03-15T22:19:55.784 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_need_max": "1.200000", 2021-03-15T22:19:55.784 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_need_min": "0.800000", 2021-03-15T22:19:55.784 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_replicate_threshold": "8000.000000", 2021-03-15T22:19:55.784 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_sample_interval": "3.000000", 2021-03-15T22:19:55.785 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_bits": "3", 2021-03-15T22:19:55.785 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_rd": "25000.000000", 2021-03-15T22:19:55.785 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_size": "10000", 2021-03-15T22:19:55.785 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_split_wr": "10000.000000", 2021-03-15T22:19:55.786 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_target_decay": "10.000000", 2021-03-15T22:19:55.786 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2021-03-15T22:19:55.786 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_beacon_grace": "15.000000", 2021-03-15T22:19:55.786 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_beacon_interval": "4.000000", 2021-03-15T22:19:55.786 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_memory_limit": "4294967296", 2021-03-15T22:19:55.787 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_mid": "0.700000", 2021-03-15T22:19:55.787 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_release_free_interval": "10", 2021-03-15T22:19:55.787 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_reservation": "0.050000", 2021-03-15T22:19:55.787 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_trim_decay_rate": "1.000000", 2021-03-15T22:19:55.788 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_trim_interval": "1", 2021-03-15T22:19:55.788 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cache_trim_threshold": "262144", 2021-03-15T22:19:55.788 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2021-03-15T22:19:55.788 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2021-03-15T22:19:55.788 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_client_delegate_inos_pct": "50", 2021-03-15T22:19:55.789 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_client_prealloc_inos": "1000", 2021-03-15T22:19:55.789 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2021-03-15T22:19:55.789 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_damage_table_max_entries": "10000", 2021-03-15T22:19:55.789 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2021-03-15T22:19:55.789 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_auth_pins": "false", 2021-03-15T22:19:55.790 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_frag": "false", 2021-03-15T22:19:55.790 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_scatterstat": "false", 2021-03-15T22:19:55.790 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_debug_subtrees": "false", 2021-03-15T22:19:55.790 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_decay_halflife": "5.000000", 2021-03-15T22:19:55.790 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_default_dir_hash": "2", 2021-03-15T22:19:55.791 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_defer_session_stale": "true", 2021-03-15T22:19:55.791 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_deny_all_reconnect": "false", 2021-03-15T22:19:55.791 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dir_keys_per_op": "16384", 2021-03-15T22:19:55.791 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dir_max_commit_size": "10", 2021-03-15T22:19:55.792 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dirstat_min_interval": "1.000000", 2021-03-15T22:19:55.792 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_after_rejoin": "false", 2021-03-15T22:19:55.792 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_on_map": "false", 2021-03-15T22:19:55.792 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_threshold_file": "0", 2021-03-15T22:19:55.792 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2021-03-15T22:19:55.793 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_early_reply": "true", 2021-03-15T22:19:55.793 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_enable_op_tracker": "true", 2021-03-15T22:19:55.793 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_enforce_unique_name": "true", 2021-03-15T22:19:55.793 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_distributed": "true", 2021-03-15T22:19:55.793 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2021-03-15T22:19:55.794 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_random": "true", 2021-03-15T22:19:55.794 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_export_ephemeral_random_max": "0.010000", 2021-03-15T22:19:55.794 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_forward_all_requests_to_auth": "false", 2021-03-15T22:19:55.794 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_freeze_tree_timeout": "30.000000", 2021-03-15T22:19:55.795 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2021-03-15T22:19:55.795 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_health_cache_threshold": "1.500000", 2021-03-15T22:19:55.795 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_health_summarize_threshold": "10", 2021-03-15T22:19:55.795 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_heartbeat_grace": "15.000000", 2021-03-15T22:19:55.795 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_migrator_session_race": "false", 2021-03-15T22:19:55.796 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2021-03-15T22:19:55.796 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_join_fs": "", 2021-03-15T22:19:55.796 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_journal_format": "1", 2021-03-15T22:19:55.796 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_create_at": "0", 2021-03-15T22:19:55.796 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_export_at": "0", 2021-03-15T22:19:55.797 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_import_at": "0", 2021-03-15T22:19:55.797 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_journal_at": "0", 2021-03-15T22:19:55.797 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_journal_expire_at": "0", 2021-03-15T22:19:55.797 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_journal_replay_at": "0", 2021-03-15T22:19:55.798 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_link_at": "0", 2021-03-15T22:19:55.798 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_mdstable_at": "0", 2021-03-15T22:19:55.798 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_openc_at": "0", 2021-03-15T22:19:55.798 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_kill_rename_at": "0", 2021-03-15T22:19:55.798 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_events_per_segment": "1024", 2021-03-15T22:19:55.799 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_max_events": "-1", 2021-03-15T22:19:55.799 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_max_segments": "128", 2021-03-15T22:19:55.799 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_pause": "false", 2021-03-15T22:19:55.799 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_segment_size": "0", 2021-03-15T22:19:55.799 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_skip_corrupt_events": "false", 2021-03-15T22:19:55.800 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_log_warn_factor": "2.000000", 2021-03-15T22:19:55.800 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_caps_per_client": "1048576", 2021-03-15T22:19:55.800 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_completed_flushes": "100000", 2021-03-15T22:19:55.800 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_completed_requests": "100000", 2021-03-15T22:19:55.801 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_export_size": "20971520", 2021-03-15T22:19:55.801 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_file_recover": "32", 2021-03-15T22:19:55.801 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_purge_files": "64", 2021-03-15T22:19:55.801 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_purge_ops": "8192", 2021-03-15T22:19:55.801 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2021-03-15T22:19:55.802 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_retries_on_remount_failure": "5", 2021-03-15T22:19:55.802 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_scrub_ops_in_progress": "5", 2021-03-15T22:19:55.802 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_snaps_per_dir": "100", 2021-03-15T22:19:55.802 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_max_xattr_pairs_size": "65536", 2021-03-15T22:19:55.802 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_metrics_update_interval": "2", 2021-03-15T22:19:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_min_caps_per_client": "100", 2021-03-15T22:19:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_min_caps_working_set": "10000", 2021-03-15T22:19:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_mon_shutdown_timeout": "5.000000", 2021-03-15T22:19:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_numa_node": "-1", 2021-03-15T22:19:55.803 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_oft_prefetch_dirfrags": "true", 2021-03-15T22:19:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_complaint_time": "30.000000", 2021-03-15T22:19:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_history_duration": "600", 2021-03-15T22:19:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_history_size": "20", 2021-03-15T22:19:55.804 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_op_log_threshold": "5", 2021-03-15T22:19:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_ping_grace": "15", 2021-03-15T22:19:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_ping_interval": "5", 2021-03-15T22:19:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2021-03-15T22:19:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_global_max_decay_threshold": "131072", 2021-03-15T22:19:55.805 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_max_caps": "30000", 2021-03-15T22:19:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_max_decay_rate": "1.500000", 2021-03-15T22:19:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_max_decay_threshold": "131072", 2021-03-15T22:19:55.806 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_warning_decay_rate": "60.000000", 2021-03-15T22:19:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_recall_warning_threshold": "262144", 2021-03-15T22:19:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_reconnect_timeout": "45.000000", 2021-03-15T22:19:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_replay_interval": "1.000000", 2021-03-15T22:19:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_replay_unsafe_with_closed_session": "false", 2021-03-15T22:19:55.807 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_request_load_average_decay_rate": "60.000000", 2021-03-15T22:19:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_root_ino_gid": "0", 2021-03-15T22:19:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_root_ino_uid": "0", 2021-03-15T22:19:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_scatter_nudge_interval": "5.000000", 2021-03-15T22:19:55.808 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_blocklist_on_evict": "true", 2021-03-15T22:19:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_blocklist_on_timeout": "true", 2021-03-15T22:19:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2021-03-15T22:19:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cache_liveness_magnitude": "10", 2021-03-15T22:19:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cap_acquisition_decay_rate": "10.000000", 2021-03-15T22:19:55.809 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_cap_acquisition_throttle": "500000", 2021-03-15T22:19:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2021-03-15T22:19:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_sessionmap_keys_per_op": "1024", 2021-03-15T22:19:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_shutdown_check": "0", 2021-03-15T22:19:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_skip_ino": "0", 2021-03-15T22:19:55.810 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_snap_max_uid": "4294967294", 2021-03-15T22:19:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_snap_min_uid": "0", 2021-03-15T22:19:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_snap_rstat": "false", 2021-03-15T22:19:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_task_status_update_interval": "2.000000", 2021-03-15T22:19:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_thrash_exports": "0", 2021-03-15T22:19:55.811 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_thrash_fragments": "0", 2021-03-15T22:19:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_tick_interval": "5.000000", 2021-03-15T22:19:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_valgrind_exit": "false", 2021-03-15T22:19:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_verify_backtrace": "1", 2021-03-15T22:19:55.812 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_verify_scatter": "false", 2021-03-15T22:19:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_wipe_ino_prealloc": "false", 2021-03-15T22:19:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "mds_wipe_sessions": "false", 2021-03-15T22:19:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "mempool_debug": "false", 2021-03-15T22:19:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_debug_omit_block_device_write": "false", 2021-03-15T22:19:55.813 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_device_bytes": "1073741824", 2021-03-15T22:19:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_page_set": "false", 2021-03-15T22:19:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "memstore_page_size": "65536", 2021-03-15T22:19:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_client_bytes": "134217728", 2021-03-15T22:19:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_client_messages": "512", 2021-03-15T22:19:55.814 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2021-03-15T22:19:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_connect_retry_interval": "1.000000", 2021-03-15T22:19:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2021-03-15T22:19:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2021-03-15T22:19:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_disabled_modules": "", 2021-03-15T22:19:55.815 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_initial_modules": "restful iostat", 2021-03-15T22:19:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mds_bytes": "134217728", 2021-03-15T22:19:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mds_messages": "128", 2021-03-15T22:19:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2021-03-15T22:19:55.816 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mon_bytes": "134217728", 2021-03-15T22:19:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_mon_messages": "128", 2021-03-15T22:19:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_osd_bytes": "536870912", 2021-03-15T22:19:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_osd_messages": "8192", 2021-03-15T22:19:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_service_beacon_grace": "60.000000", 2021-03-15T22:19:55.817 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_stats_period": "5", 2021-03-15T22:19:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_stats_threshold": "5", 2021-03-15T22:19:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_tick_period": "2", 2021-03-15T22:19:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_accept_timeout_factor": "2.000000", 2021-03-15T22:19:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_allow_pool_delete": "true", 2021-03-15T22:19:55.818 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_allow_pool_size_one": "false", 2021-03-15T22:19:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_auth_validate_all_caps": "true", 2021-03-15T22:19:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2021-03-15T22:19:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2021-03-15T22:19:55.819 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_bytes": "104857600", 2021-03-15T22:19:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_directed_command_retry": "2", 2021-03-15T22:19:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval": "3.000000", 2021-03-15T22:19:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2021-03-15T22:19:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2021-03-15T22:19:55.820 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2021-03-15T22:19:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_hunt_parallel": "3", 2021-03-15T22:19:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_log_interval": "1.000000", 2021-03-15T22:19:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_max_log_entries_per_message": "1000", 2021-03-15T22:19:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_ping_interval": "10.000000", 2021-03-15T22:19:55.821 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_client_ping_timeout": "30.000000", 2021-03-15T22:19:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_clock_drift_allowed": "1.000000", 2021-03-15T22:19:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2021-03-15T22:19:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2021-03-15T22:19:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_file_level": "debug", 2021-03-15T22:19:55.822 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_file": "false", 2021-03-15T22:19:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_graylog": "false", 2021-03-15T22:19:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2021-03-15T22:19:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_graylog_port": "12201", 2021-03-15T22:19:55.823 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_stderr": "true", 2021-03-15T22:19:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_syslog": "default=false", 2021-03-15T22:19:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2021-03-15T22:19:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cluster_log_to_syslog_level": "info", 2021-03-15T22:19:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_compact_on_bootstrap": "false", 2021-03-15T22:19:55.824 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_compact_on_start": "false", 2021-03-15T22:19:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_compact_on_trim": "true", 2021-03-15T22:19:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_con_tracker_persist_interval": "10", 2021-03-15T22:19:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_con_tracker_score_halflife": "43200", 2021-03-15T22:19:55.825 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_config_key_max_entry_size": "65536", 2021-03-15T22:19:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_cpu_threads": "4", 2021-03-15T22:19:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_crush_min_required_version": "hammer", 2021-03-15T22:19:55.826 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_daemon_bytes": "419430400", 2021-03-15T22:19:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2021-03-15T22:19:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data_avail_crit": "5", 2021-03-15T22:19:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data_avail_warn": "5", 2021-03-15T22:19:55.827 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_data_size_warn": "16106127360", 2021-03-15T22:19:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_block_osdmap_trim": "false", 2021-03-15T22:19:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_deprecated_as_obsolete": "false", 2021-03-15T22:19:55.828 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_dump_json": "false", 2021-03-15T22:19:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2021-03-15T22:19:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_dump_transactions": "false", 2021-03-15T22:19:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_extra_checks": "false", 2021-03-15T22:19:55.829 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_initial_persistent_features": "false", 2021-03-15T22:19:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2021-03-15T22:19:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_require_pacific": "false", 2021-03-15T22:19:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_no_require_quincy": "false", 2021-03-15T22:19:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2021-03-15T22:19:55.830 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_delta_reset_interval": "10.000000", 2021-03-15T22:19:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_dns_srv_name": "ceph-mon", 2021-03-15T22:19:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_election_default_strategy": "1", 2021-03-15T22:19:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_election_timeout": "5.000000", 2021-03-15T22:19:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2021-03-15T22:19:55.831 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_elector_ping_divisor": "2", 2021-03-15T22:19:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_elector_ping_timeout": "2.000000", 2021-03-15T22:19:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_enable_op_tracker": "true", 2021-03-15T22:19:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_fake_pool_delete": "false", 2021-03-15T22:19:55.832 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_force_quorum_join": "false", 2021-03-15T22:19:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_globalid_prealloc": "10000", 2021-03-15T22:19:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_detail_to_clog": "true", 2021-03-15T22:19:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_log_update_period": "5", 2021-03-15T22:19:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_max_detail": "50", 2021-03-15T22:19:55.833 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_to_clog": "true", 2021-03-15T22:19:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_to_clog_interval": "600", 2021-03-15T22:19:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2021-03-15T22:19:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_host": "[v2:172.21.15.76:3300/0,v1:172.21.15.76:6789/0]", 2021-03-15T22:19:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_host_override": "", 2021-03-15T22:19:55.834 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_initial_members": "", 2021-03-15T22:19:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2021-03-15T22:19:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2021-03-15T22:19:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_transaction_delay_max": "10.000000", 2021-03-15T22:19:55.835 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2021-03-15T22:19:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_keyvaluedb": "rocksdb", 2021-03-15T22:19:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_lease": "5.000000", 2021-03-15T22:19:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2021-03-15T22:19:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_lease_renew_interval_factor": "0.600000", 2021-03-15T22:19:55.836 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_log_max_summary": "50", 2021-03-15T22:19:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_log_entries_per_event": "4096", 2021-03-15T22:19:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_log_epochs": "500", 2021-03-15T22:19:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_mdsmap_epochs": "500", 2021-03-15T22:19:55.837 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_mgrmap_epochs": "500", 2021-03-15T22:19:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_osd": "10000", 2021-03-15T22:19:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_pg_per_osd": "10000", 2021-03-15T22:19:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_pool_pg_num": "65536", 2021-03-15T22:19:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_max_snap_prune_per_epoch": "100", 2021-03-15T22:19:55.838 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mds_blocklist_interval": "86400.000000", 2021-03-15T22:19:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mds_force_trim_to": "0", 2021-03-15T22:19:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mds_skip_sanity": "false", 2021-03-15T22:19:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_memory_autotune": "true", 2021-03-15T22:19:55.839 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_memory_target": "2147483648", 2021-03-15T22:19:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_beacon_grace": "30", 2021-03-15T22:19:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2021-03-15T22:19:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_digest_period": "5", 2021-03-15T22:19:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_inactive_grace": "60", 2021-03-15T22:19:55.840 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_mkfs_grace": "240", 2021-03-15T22:19:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2021-03-15T22:19:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_min_osdmap_epochs": "50", 2021-03-15T22:19:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_complaint_time": "30", 2021-03-15T22:19:55.841 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_duration": "600", 2021-03-15T22:19:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_size": "20", 2021-03-15T22:19:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_slow_op_size": "20", 2021-03-15T22:19:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_history_slow_op_threshold": "10", 2021-03-15T22:19:55.842 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_op_log_threshold": "5", 2021-03-15T22:19:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_adjust_down_out_interval": "true", 2021-03-15T22:19:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2021-03-15T22:19:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2021-03-15T22:19:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_auto_mark_in": "false", 2021-03-15T22:19:55.843 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_auto_mark_new_in": "true", 2021-03-15T22:19:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2021-03-15T22:19:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2021-03-15T22:19:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_cache_size": "500", 2021-03-15T22:19:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_cache_size_min": "134217728", 2021-03-15T22:19:55.844 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_crush_smoke_test": "true", 2021-03-15T22:19:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_destroyed_out_interval": "600", 2021-03-15T22:19:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_down_out_interval": "600", 2021-03-15T22:19:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_down_out_subtree_limit": "rack", 2021-03-15T22:19:55.845 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2021-03-15T22:19:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_force_trim_to": "0", 2021-03-15T22:19:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_full_ratio": "0.950000", 2021-03-15T22:19:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2021-03-15T22:19:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_laggy_halflife": "3600", 2021-03-15T22:19:55.846 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_laggy_max_interval": "300", 2021-03-15T22:19:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_laggy_weight": "0.300000", 2021-03-15T22:19:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2021-03-15T22:19:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_max_creating_pgs": "1024", 2021-03-15T22:19:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_max_initial_pgs": "1024", 2021-03-15T22:19:55.847 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_min_down_reporters": "2", 2021-03-15T22:19:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_min_in_ratio": "0.750000", 2021-03-15T22:19:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_min_up_ratio": "0.300000", 2021-03-15T22:19:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_nearfull_ratio": "0.850000", 2021-03-15T22:19:55.848 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_prime_pg_temp": "true", 2021-03-15T22:19:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2021-03-15T22:19:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2021-03-15T22:19:55.849 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_report_timeout": "900", 2021-03-15T22:19:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_reporter_subtree_level": "osd", 2021-03-15T22:19:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2021-03-15T22:19:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_warn_num_repaired": "10", 2021-03-15T22:19:55.850 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osd_warn_op_age": "32.000000", 2021-03-15T22:19:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_enabled": "true", 2021-03-15T22:19:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_interval": "2", 2021-03-15T22:19:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_min": "15", 2021-03-15T22:19:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_osdmap_full_prune_txsize": "2", 2021-03-15T22:19:55.851 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2021-03-15T22:19:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_stuck_threshold": "60", 2021-03-15T22:19:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2021-03-15T22:19:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_min_objects": "10000", 2021-03-15T22:19:55.852 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_min_per_osd": "0", 2021-03-15T22:19:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pg_warn_min_pool_objects": "1000", 2021-03-15T22:19:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pool_quota_crit_threshold": "0", 2021-03-15T22:19:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_pool_quota_warn_threshold": "0", 2021-03-15T22:19:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_probe_timeout": "2.000000", 2021-03-15T22:19:55.853 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_max_change": "0.050000", 2021-03-15T22:19:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_max_osds": "4", 2021-03-15T22:19:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_min_bytes_per_osd": "10", 2021-03-15T22:19:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_reweight_min_pgs_per_osd": "4", 2021-03-15T22:19:55.854 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2021-03-15T22:19:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2021-03-15T22:19:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2021-03-15T22:19:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_interval": "86400", 2021-03-15T22:19:55.855 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_max_keys": "100", 2021-03-15T22:19:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_scrub_timeout": "300", 2021-03-15T22:19:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_session_timeout": "300", 2021-03-15T22:19:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_smart_report_timeout": "5", 2021-03-15T22:19:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stat_smooth_intervals": "6", 2021-03-15T22:19:55.856 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2021-03-15T22:19:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_pool_min_size": "2", 2021-03-15T22:19:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_pool_size": "4", 2021-03-15T22:19:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2021-03-15T22:19:55.857 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_subscribe_interval": "86400.000000", 2021-03-15T22:19:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_debug": "false", 2021-03-15T22:19:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_max_payload_keys": "2000", 2021-03-15T22:19:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_max_payload_size": "1048576", 2021-03-15T22:19:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_provider_kill_at": "0", 2021-03-15T22:19:55.858 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_requester_kill_at": "0", 2021-03-15T22:19:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_sync_timeout": "60.000000", 2021-03-15T22:19:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_target_pg_per_osd": "100", 2021-03-15T22:19:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_tick_interval": "5", 2021-03-15T22:19:55.859 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_timecheck_interval": "300.000000", 2021-03-15T22:19:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_timecheck_skew_interval": "30.000000", 2021-03-15T22:19:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_older_version_delay": "604800", 2021-03-15T22:19:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2021-03-15T22:19:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2021-03-15T22:19:55.860 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2021-03-15T22:19:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2021-03-15T22:19:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_misplaced": "false", 2021-03-15T22:19:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2021-03-15T22:19:55.861 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_older_version": "true", 2021-03-15T22:19:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2021-03-15T22:19:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_pool_no_app": "true", 2021-03-15T22:19:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_pool_no_redundancy": "true", 2021-03-15T22:19:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2021-03-15T22:19:55.862 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2021-03-15T22:19:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2021-03-15T22:19:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_on_too_few_osds": "false", 2021-03-15T22:19:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2021-03-15T22:19:55.863 INFO:teuthology.orchestra.run.smithi076.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2021-03-15T22:19:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "monmap": "", 2021-03-15T22:19:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_op_threads": "3", 2021-03-15T22:19:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_buffer_size": "131072", 2021-03-15T22:19:55.864 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_cm": "false", 2021-03-15T22:19:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_device_name": "", 2021-03-15T22:19:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_dscp": "96", 2021-03-15T22:19:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_enable_hugepage": "false", 2021-03-15T22:19:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_gid_idx": "0", 2021-03-15T22:19:55.865 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_local_gid": "", 2021-03-15T22:19:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_polling_us": "1000", 2021-03-15T22:19:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_port_num": "1", 2021-03-15T22:19:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_receive_buffers": "32768", 2021-03-15T22:19:55.866 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_receive_queue_len": "4096", 2021-03-15T22:19:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_roce_ver": "1", 2021-03-15T22:19:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_send_buffers": "1024", 2021-03-15T22:19:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_sl": "3", 2021-03-15T22:19:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_support_srq": "true", 2021-03-15T22:19:55.867 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_async_rdma_type": "ib", 2021-03-15T22:19:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_before_connect": "false", 2021-03-15T22:19:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_ipv4": "true", 2021-03-15T22:19:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_ipv6": "false", 2021-03-15T22:19:55.868 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_msgr1": "true", 2021-03-15T22:19:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_msgr2": "true", 2021-03-15T22:19:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_port_max": "7300", 2021-03-15T22:19:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_port_min": "6800", 2021-03-15T22:19:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_prefer_ipv4": "false", 2021-03-15T22:19:55.869 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_retry_count": "3", 2021-03-15T22:19:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_bind_retry_delay": "5", 2021-03-15T22:19:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_client": "false", 2021-03-15T22:19:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_mds": "false", 2021-03-15T22:19:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_mgr": "false", 2021-03-15T22:19:55.870 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_mon": "false", 2021-03-15T22:19:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_blackhole_osd": "false", 2021-03-15T22:19:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_client_mode": "crc secure", 2021-03-15T22:19:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_cluster_mode": "crc secure", 2021-03-15T22:19:55.871 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_cluster_type": "", 2021-03-15T22:19:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_connection_idle_timeout": "900", 2021-03-15T22:19:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_connection_ready_timeout": "10", 2021-03-15T22:19:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_crc_data": "true", 2021-03-15T22:19:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_crc_header": "true", 2021-03-15T22:19:55.872 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_bad_msg": "false", 2021-03-15T22:19:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_bug": "true", 2021-03-15T22:19:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_old_message": "true", 2021-03-15T22:19:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_skipped_message": "false", 2021-03-15T22:19:55.873 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_die_on_unhandled_msg": "false", 2021-03-15T22:19:55.874 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dispatch_throttle_bytes": "104857600", 2021-03-15T22:19:55.874 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_coremask": "0xF", 2021-03-15T22:19:55.874 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_debug_allow_loopback": "false", 2021-03-15T22:19:55.874 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2021-03-15T22:19:55.874 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_host_ipv4_addr": "", 2021-03-15T22:19:55.875 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_hugepages": "", 2021-03-15T22:19:55.875 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_hw_flow_control": "true", 2021-03-15T22:19:55.875 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2021-03-15T22:19:55.875 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_lro": "true", 2021-03-15T22:19:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_memory_channel": "4", 2021-03-15T22:19:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2021-03-15T22:19:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_pmd": "", 2021-03-15T22:19:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_port_id": "0", 2021-03-15T22:19:55.876 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2021-03-15T22:19:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dump_corrupt_message_level": "1", 2021-03-15T22:19:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_dump_on_send": "false", 2021-03-15T22:19:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_initial_backoff": "0.200000", 2021-03-15T22:19:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_delay_max": "1.000000", 2021-03-15T22:19:55.877 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_delay_probability": "0.000000", 2021-03-15T22:19:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_delay_type": "", 2021-03-15T22:19:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_internal_delays": "0.000000", 2021-03-15T22:19:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_inject_socket_failures": "0", 2021-03-15T22:19:55.878 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_learn_addr_from_peer": "true", 2021-03-15T22:19:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_max_accept_failures": "4", 2021-03-15T22:19:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_max_backoff": "15.000000", 2021-03-15T22:19:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_mon_client_mode": "secure crc", 2021-03-15T22:19:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_mon_cluster_mode": "secure crc", 2021-03-15T22:19:55.879 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_mon_service_mode": "secure crc", 2021-03-15T22:19:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2021-03-15T22:19:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_pq_min_cost": "65536", 2021-03-15T22:19:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_public_type": "", 2021-03-15T22:19:55.880 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_service_mode": "crc secure", 2021-03-15T22:19:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_listen_backlog": "512", 2021-03-15T22:19:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_nodelay": "true", 2021-03-15T22:19:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_prefetch_max_size": "4096", 2021-03-15T22:19:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_tcp_rcvbuf": "0", 2021-03-15T22:19:55.881 INFO:teuthology.orchestra.run.smithi076.stdout: "ms_type": "async+posix", 2021-03-15T22:19:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "no_config_file": "false", 2021-03-15T22:19:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_completion_locks_per_session": "32", 2021-03-15T22:19:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_debug_inject_relock_delay": "false", 2021-03-15T22:19:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_inflight_op_bytes": "104857600", 2021-03-15T22:19:55.882 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_inflight_ops": "1024", 2021-03-15T22:19:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_inject_no_watch_ping": "false", 2021-03-15T22:19:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_retry_writes_after_first_reply": "false", 2021-03-15T22:19:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_tick_interval": "5.000000", 2021-03-15T22:19:55.883 INFO:teuthology.orchestra.run.smithi076.stdout: "objecter_timeout": "10.000000", 2021-03-15T22:19:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "objectstore_blackhole": "false", 2021-03-15T22:19:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "openssl_engine_opts": "", 2021-03-15T22:19:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_delay_time": "5.000000", 2021-03-15T22:19:55.884 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_hist_halflife": "1000", 2021-03-15T22:19:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_max_low_ops": "2", 2021-03-15T22:19:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_max_ops": "4", 2021-03-15T22:19:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_min_evict_effort": "0.100000", 2021-03-15T22:19:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_quantize_effort": "0.100000", 2021-03-15T22:19:55.885 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_agent_slop": "0.020000", 2021-03-15T22:19:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_allow_recovery_below_min_size": "true", 2021-03-15T22:19:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_asio_thread_count": "2", 2021-03-15T22:19:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_async_recovery_min_cost": "100", 2021-03-15T22:19:55.886 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_auto_mark_unfound_lost": "false", 2021-03-15T22:19:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backfill_retry_interval": "30.000000", 2021-03-15T22:19:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backfill_scan_max": "512", 2021-03-15T22:19:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backfill_scan_min": "64", 2021-03-15T22:19:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backoff_on_degraded": "false", 2021-03-15T22:19:55.887 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backoff_on_peering": "false", 2021-03-15T22:19:55.888 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_backoff_on_unfound": "true", 2021-03-15T22:19:55.888 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_beacon_report_interval": "300", 2021-03-15T22:19:55.888 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_duration": "30", 2021-03-15T22:19:55.888 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_large_size_max_throughput": "104857600", 2021-03-15T22:19:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_max_block_size": "67108864", 2021-03-15T22:19:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_bench_small_size_max_iops": "100", 2021-03-15T22:19:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_blkin_trace_all": "false", 2021-03-15T22:19:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2021-03-15T22:19:55.889 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2021-03-15T22:19:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_check_for_log_corruption": "false", 2021-03-15T22:19:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_check_max_object_name_len_on_startup": "true", 2021-03-15T22:19:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2021-03-15T22:19:55.890 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2021-03-15T22:19:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2021-03-15T22:19:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_class_update_on_start": "true", 2021-03-15T22:19:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_message_cap": "0", 2021-03-15T22:19:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_message_size_cap": "524288000", 2021-03-15T22:19:55.891 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_op_priority": "63", 2021-03-15T22:19:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_client_watch_timeout": "30", 2021-03-15T22:19:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_command_max_records": "256", 2021-03-15T22:19:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_command_thread_suicide_timeout": "900", 2021-03-15T22:19:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_command_thread_timeout": "600", 2021-03-15T22:19:55.892 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_compact_on_start": "false", 2021-03-15T22:19:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_copyfrom_max_chunk": "8388608", 2021-03-15T22:19:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_chooseleaf_type": "1", 2021-03-15T22:19:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_initial_weight": "-1.000000", 2021-03-15T22:19:55.893 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_update_on_start": "true", 2021-03-15T22:19:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_crush_update_weight_set": "true", 2021-03-15T22:19:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2021-03-15T22:19:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2021-03-15T22:19:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2021-03-15T22:19:55.894 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_drop_ping_duration": "0", 2021-03-15T22:19:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_drop_ping_probability": "0.000000", 2021-03-15T22:19:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_feed_pullee": "-1", 2021-03-15T22:19:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_inject_copyfrom_error": "false", 2021-03-15T22:19:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2021-03-15T22:19:55.895 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2021-03-15T22:19:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_misdirected_ops": "false", 2021-03-15T22:19:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_no_acting_change": "false", 2021-03-15T22:19:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_no_purge_strays": "false", 2021-03-15T22:19:55.896 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_op_order": "false", 2021-03-15T22:19:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_pg_log_writeout": "false", 2021-03-15T22:19:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_pretend_recovery_active": "false", 2021-03-15T22:19:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_random_push_read_error": "0.000000", 2021-03-15T22:19:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2021-03-15T22:19:55.897 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_shutdown": "false", 2021-03-15T22:19:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2021-03-15T22:19:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2021-03-15T22:19:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_cached_snaps": "false", 2021-03-15T22:19:55.898 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_missing_on_start": "false", 2021-03-15T22:19:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_snaps": "false", 2021-03-15T22:19:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_debug_verify_stray_on_activate": "false", 2021-03-15T22:19:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_interval": "604800.000000", 2021-03-15T22:19:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_keys": "1024", 2021-03-15T22:19:55.899 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2021-03-15T22:19:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2021-03-15T22:19:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2021-03-15T22:19:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_stride": "524288", 2021-03-15T22:19:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2021-03-15T22:19:55.900 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_default_data_pool_replay_window": "45", 2021-03-15T22:19:55.901 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_default_notify_timeout": "30", 2021-03-15T22:19:55.901 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep": "0.000000", 2021-03-15T22:19:55.901 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep_hdd": "5.000000", 2021-03-15T22:19:55.901 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep_hybrid": "1.000000", 2021-03-15T22:19:55.901 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_delete_sleep_ssd": "1.000000", 2021-03-15T22:19:55.902 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_discard_disconnected_ops": "true", 2021-03-15T22:19:55.902 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_enable_op_tracker": "true", 2021-03-15T22:19:55.902 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_erasure_code_plugins": "jerasure lrc isa", 2021-03-15T22:19:55.902 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_failsafe_full_ratio": "0.970000", 2021-03-15T22:19:55.903 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_fail_on_connection_refused": "true", 2021-03-15T22:19:55.903 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_info": "true", 2021-03-15T22:19:55.903 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_shutdown": "true", 2021-03-15T22:19:55.903 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_fast_shutdown_notify_mon": "false", 2021-03-15T22:19:55.903 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_find_best_info_ignore_history_les": "false", 2021-03-15T22:19:55.904 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_force_auth_primary_missing_objects": "100", 2021-03-15T22:19:55.904 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2021-03-15T22:19:55.904 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_function_tracing": "false", 2021-03-15T22:19:55.904 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_grace": "20", 2021-03-15T22:19:55.904 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_interval": "6", 2021-03-15T22:19:55.905 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2021-03-15T22:19:55.905 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_min_peers": "10", 2021-03-15T22:19:55.905 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_min_size": "2000", 2021-03-15T22:19:55.905 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_stale": "600", 2021-03-15T22:19:55.906 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2021-03-15T22:19:55.906 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_hit_set_max_size": "100000", 2021-03-15T22:19:55.906 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_hit_set_min_size": "1000", 2021-03-15T22:19:55.906 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_hit_set_namespace": ".ceph-internal", 2021-03-15T22:19:55.906 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_ignore_stale_divergent_priors": "false", 2021-03-15T22:19:55.907 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2021-03-15T22:19:55.907 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_inject_failure_on_pg_removal": "false", 2021-03-15T22:19:55.907 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2021-03-15T22:19:55.907 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_journal_flush_on_shutdown": "true", 2021-03-15T22:19:55.907 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_journal_size": "5120", 2021-03-15T22:19:55.908 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_kill_backfill_at": "0", 2021-03-15T22:19:55.908 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_loop_before_reset_tphandle": "64", 2021-03-15T22:19:55.908 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_cache_size": "50", 2021-03-15T22:19:55.908 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_dedup": "true", 2021-03-15T22:19:55.908 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_message_max": "40", 2021-03-15T22:19:55.909 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_message_max_bytes": "10485760", 2021-03-15T22:19:55.909 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_map_share_max_epochs": "40", 2021-03-15T22:19:55.909 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_attr_name_len": "100", 2021-03-15T22:19:55.909 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_attr_size": "0", 2021-03-15T22:19:55.909 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_backfills": "1", 2021-03-15T22:19:55.910 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_markdown_count": "5", 2021-03-15T22:19:55.910 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_markdown_period": "600", 2021-03-15T22:19:55.910 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_object_name_len": "2048", 2021-03-15T22:19:55.910 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_object_namespace_len": "256", 2021-03-15T22:19:55.911 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_object_size": "134217728", 2021-03-15T22:19:55.911 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2021-03-15T22:19:55.911 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_omap_entries_per_request": "1024", 2021-03-15T22:19:55.911 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pg_blocked_by": "16", 2021-03-15T22:19:55.912 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pg_log_entries": "10000", 2021-03-15T22:19:55.912 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2021-03-15T22:19:55.912 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_pgls": "1024", 2021-03-15T22:19:55.912 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_push_cost": "8388608", 2021-03-15T22:19:55.912 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_push_objects": "10", 2021-03-15T22:19:55.913 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_scrubs": "1", 2021-03-15T22:19:55.913 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2021-03-15T22:19:55.913 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_trimming_pgs": "2", 2021-03-15T22:19:55.913 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_write_op_reply_len": "32", 2021-03-15T22:19:55.913 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_max_write_size": "90", 2021-03-15T22:19:55.914 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_cost_per_byte_usec": "0.000000", 2021-03-15T22:19:55.914 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_cost_per_byte_usec_hdd": "5.200000", 2021-03-15T22:19:55.914 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_cost_per_byte_usec_ssd": "0.011000", 2021-03-15T22:19:55.915 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_cost_per_io_usec": "0.000000", 2021-03-15T22:19:55.915 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_cost_per_io_usec_hdd": "25000.000000", 2021-03-15T22:19:55.915 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_cost_per_io_usec_ssd": "50.000000", 2021-03-15T22:19:55.915 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_max_capacity_iops": "0.000000", 2021-03-15T22:19:55.915 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2021-03-15T22:19:55.916 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2021-03-15T22:19:55.916 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_profile": "high_client_ops", 2021-03-15T22:19:55.916 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2021-03-15T22:19:55.916 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_best_effort_lim": "999999", 2021-03-15T22:19:55.917 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_best_effort_res": "1", 2021-03-15T22:19:55.917 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2021-03-15T22:19:55.917 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_recovery_lim": "999999", 2021-03-15T22:19:55.917 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_recovery_res": "1", 2021-03-15T22:19:55.917 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2021-03-15T22:19:55.918 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_client_lim": "999999", 2021-03-15T22:19:55.918 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_client_res": "1", 2021-03-15T22:19:55.918 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mclock_scheduler_client_wgt": "1", 2021-03-15T22:19:55.918 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_base": "805306368", 2021-03-15T22:19:55.918 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_cache_min": "134217728", 2021-03-15T22:19:55.919 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_cache_resize_interval": "1.000000", 2021-03-15T22:19:55.919 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_expected_fragmentation": "0.150000", 2021-03-15T22:19:55.919 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_target": "4294967296", 2021-03-15T22:19:55.919 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2021-03-15T22:19:55.920 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_min_pg_log_entries": "250", 2021-03-15T22:19:55.920 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_min_recovery_priority": "0", 2021-03-15T22:19:55.920 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_heartbeat_interval": "30", 2021-03-15T22:19:55.920 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2021-03-15T22:19:55.920 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_report_interval": "5", 2021-03-15T22:19:55.921 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_report_max_in_flight": "2", 2021-03-15T22:19:55.921 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_mon_shutdown_timeout": "5.000000", 2021-03-15T22:19:55.921 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_num_cache_shards": "32", 2021-03-15T22:19:55.921 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_num_op_tracker_shard": "32", 2021-03-15T22:19:55.922 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_numa_auto_affinity": "true", 2021-03-15T22:19:55.922 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_numa_node": "-1", 2021-03-15T22:19:55.922 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_numa_prefer_iface": "true", 2021-03-15T22:19:55.922 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_object_clean_region_max_num_intervals": "10", 2021-03-15T22:19:55.922 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objecter_finishers": "1", 2021-03-15T22:19:55.923 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objectstore": "bluestore", 2021-03-15T22:19:55.923 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objectstore_fuse": "false", 2021-03-15T22:19:55.923 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_objectstore_tracing": "false", 2021-03-15T22:19:55.923 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_complaint_time": "30.000000", 2021-03-15T22:19:55.924 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_duration": "600", 2021-03-15T22:19:55.924 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_size": "20", 2021-03-15T22:19:55.924 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_slow_op_size": "20", 2021-03-15T22:19:55.924 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2021-03-15T22:19:55.924 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_log_threshold": "5", 2021-03-15T22:19:55.925 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_shards": "0", 2021-03-15T22:19:55.925 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_shards_hdd": "5", 2021-03-15T22:19:55.925 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_shards_ssd": "8", 2021-03-15T22:19:55.925 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_threads_per_shard": "0", 2021-03-15T22:19:55.925 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2021-03-15T22:19:55.926 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2021-03-15T22:19:55.926 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2021-03-15T22:19:55.926 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_pq_min_cost": "65536", 2021-03-15T22:19:55.926 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_queue": "wpq", 2021-03-15T22:19:55.927 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_queue_cut_off": "high", 2021-03-15T22:19:55.927 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_thread_suicide_timeout": "150", 2021-03-15T22:19:55.927 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_op_thread_timeout": "15", 2021-03-15T22:19:55.927 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_open_classes_on_start": "true", 2021-03-15T22:19:55.927 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_os_flags": "0", 2021-03-15T22:19:55.928 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_peering_op_priority": "255", 2021-03-15T22:19:55.928 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_delete_cost": "1048576", 2021-03-15T22:19:55.928 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_delete_priority": "5", 2021-03-15T22:19:55.928 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2021-03-15T22:19:55.929 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2021-03-15T22:19:55.929 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_log_dups_tracked": "3000", 2021-03-15T22:19:55.929 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_log_trim_max": "10000", 2021-03-15T22:19:55.929 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_log_trim_min": "100", 2021-03-15T22:19:55.929 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2021-03-15T22:19:55.930 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_object_context_cache_count": "64", 2021-03-15T22:19:55.930 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pg_stat_report_interval_max": "500", 2021-03-15T22:19:55.930 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2021-03-15T22:19:55.930 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_min_evict_age": "0", 2021-03-15T22:19:55.931 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_min_flush_age": "0", 2021-03-15T22:19:55.931 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2021-03-15T22:19:55.931 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2021-03-15T22:19:55.931 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2021-03-15T22:19:55.931 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_crush_rule": "-1", 2021-03-15T22:19:55.932 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_ec_fast_read": "false", 2021-03-15T22:19:55.932 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd", 2021-03-15T22:19:55.932 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_hashpspool": "true", 2021-03-15T22:19:55.932 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_nodelete": "false", 2021-03-15T22:19:55.932 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_nopgchange": "false", 2021-03-15T22:19:55.933 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flag_nosizechange": "false", 2021-03-15T22:19:55.933 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_flags": "0", 2021-03-15T22:19:55.933 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2021-03-15T22:19:55.934 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_min_size": "0", 2021-03-15T22:19:55.934 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2021-03-15T22:19:55.934 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_pg_num": "32", 2021-03-15T22:19:55.934 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_pgp_num": "0", 2021-03-15T22:19:55.934 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2021-03-15T22:19:55.935 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_size": "3", 2021-03-15T22:19:55.935 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_default_type": "replicated", 2021-03-15T22:19:55.935 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2021-03-15T22:19:55.935 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_pool_use_gmt_hitset": "true", 2021-03-15T22:19:55.936 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_push_per_object_cost": "1000", 2021-03-15T22:19:55.936 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_read_ec_check_for_errors": "false", 2021-03-15T22:19:55.936 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recover_clone_overlap": "true", 2021-03-15T22:19:55.936 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recover_clone_overlap_limit": "10", 2021-03-15T22:19:55.936 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_cost": "20971520", 2021-03-15T22:19:55.937 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_delay_start": "0.000000", 2021-03-15T22:19:55.937 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_active": "0", 2021-03-15T22:19:55.937 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_active_hdd": "3", 2021-03-15T22:19:55.937 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_active_ssd": "10", 2021-03-15T22:19:55.938 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_chunk": "8388608", 2021-03-15T22:19:55.938 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2021-03-15T22:19:55.938 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_max_single_start": "1", 2021-03-15T22:19:55.938 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_op_priority": "3", 2021-03-15T22:19:55.938 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_op_warn_multiple": "16", 2021-03-15T22:19:55.939 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_priority": "5", 2021-03-15T22:19:55.939 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_retry_interval": "30.000000", 2021-03-15T22:19:55.939 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep": "0.000000", 2021-03-15T22:19:55.939 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep_hdd": "0.100000", 2021-03-15T22:19:55.939 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2021-03-15T22:19:55.940 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_recovery_sleep_ssd": "0.000000", 2021-03-15T22:19:55.940 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_repair_during_recovery": "false", 2021-03-15T22:19:55.940 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_requested_scrub_priority": "120", 2021-03-15T22:19:55.940 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_rollback_to_cluster_snap": "", 2021-03-15T22:19:55.941 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_auto_repair": "false", 2021-03-15T22:19:55.941 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_auto_repair_num_errors": "5", 2021-03-15T22:19:55.941 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_backoff_ratio": "0.660000", 2021-03-15T22:19:55.941 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_begin_hour": "0", 2021-03-15T22:19:55.941 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_begin_week_day": "0", 2021-03-15T22:19:55.942 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_chunk_max": "25", 2021-03-15T22:19:55.942 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_chunk_min": "5", 2021-03-15T22:19:55.942 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_cost": "52428800", 2021-03-15T22:19:55.942 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_during_recovery": "false", 2021-03-15T22:19:55.942 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_end_hour": "0", 2021-03-15T22:19:55.943 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_end_week_day": "0", 2021-03-15T22:19:55.943 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_extended_sleep": "0.000000", 2021-03-15T22:19:55.943 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2021-03-15T22:19:55.943 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_invalid_stats": "true", 2021-03-15T22:19:55.944 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_load_threshold": "0.500000", 2021-03-15T22:19:55.944 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_max_interval": "604800.000000", 2021-03-15T22:19:55.944 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_max_preemptions": "5", 2021-03-15T22:19:55.944 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_min_interval": "86400.000000", 2021-03-15T22:19:55.944 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_priority": "5", 2021-03-15T22:19:55.945 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_scrub_sleep": "0.000000", 2021-03-15T22:19:55.945 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_shutdown_pgref_assert": "false", 2021-03-15T22:19:55.945 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_skip_data_digest": "false", 2021-03-15T22:19:55.945 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_smart_report_timeout": "5", 2021-03-15T22:19:55.945 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_cost": "1048576", 2021-03-15T22:19:55.946 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_priority": "5", 2021-03-15T22:19:55.946 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep": "0.000000", 2021-03-15T22:19:55.946 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2021-03-15T22:19:55.946 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2021-03-15T22:19:55.946 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2021-03-15T22:19:55.947 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2021-03-15T22:19:55.947 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_target_transaction_size": "30", 2021-03-15T22:19:55.947 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_count": "4", 2021-03-15T22:19:55.947 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2021-03-15T22:19:55.947 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2021-03-15T22:19:55.948 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2021-03-15T22:19:55.948 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2021-03-15T22:19:55.948 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2021-03-15T22:19:55.948 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2021-03-15T22:19:55.948 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_default_cache_mode": "writeback", 2021-03-15T22:19:55.949 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2021-03-15T22:19:55.949 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tier_promote_max_objects_sec": "25", 2021-03-15T22:19:55.949 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_tracing": "false", 2021-03-15T22:19:55.949 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_use_stale_snap": "false", 2021-03-15T22:19:55.950 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2021-03-15T22:19:55.950 INFO:teuthology.orchestra.run.smithi076.stdout: "osdc_blkin_trace_all": "false", 2021-03-15T22:19:55.950 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_kill_at": "0", 2021-03-15T22:19:55.950 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_max_join_drift": "10", 2021-03-15T22:19:55.951 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_min": "500", 2021-03-15T22:19:55.951 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_min_wait": "0.050000", 2021-03-15T22:19:55.951 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_propose_interval": "1.000000", 2021-03-15T22:19:55.951 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_service_trim_max": "500", 2021-03-15T22:19:55.951 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_service_trim_min": "10", 2021-03-15T22:19:55.952 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_stash_full_interval": "25", 2021-03-15T22:19:55.952 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_trim_max": "500", 2021-03-15T22:19:55.952 INFO:teuthology.orchestra.run.smithi076.stdout: "paxos_trim_min": "250", 2021-03-15T22:19:55.952 INFO:teuthology.orchestra.run.smithi076.stdout: "perf": "true", 2021-03-15T22:19:55.952 INFO:teuthology.orchestra.run.smithi076.stdout: "pid_file": "", 2021-03-15T22:19:55.953 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin_crypto_accelerator": "crypto_isal", 2021-03-15T22:19:55.953 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin_dir": "/usr/lib64/ceph", 2021-03-15T22:19:55.953 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "-", 2021-03-15T22:19:55.953 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrv": "", 2021-03-15T22:19:55.953 INFO:teuthology.orchestra.run.smithi076.stdout: "public_bind_addr": "-", 2021-03-15T22:19:55.954 INFO:teuthology.orchestra.run.smithi076.stdout: "public_network": "", 2021-03-15T22:19:55.954 INFO:teuthology.orchestra.run.smithi076.stdout: "public_network_interface": "", 2021-03-15T22:19:55.954 INFO:teuthology.orchestra.run.smithi076.stdout: "qat_compressor_enabled": "false", 2021-03-15T22:19:55.954 INFO:teuthology.orchestra.run.smithi076.stdout: "rados_mon_op_timeout": "0", 2021-03-15T22:19:55.955 INFO:teuthology.orchestra.run.smithi076.stdout: "rados_osd_op_timeout": "0", 2021-03-15T22:19:55.955 INFO:teuthology.orchestra.run.smithi076.stdout: "rados_tracing": "false", 2021-03-15T22:19:55.955 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_atime_update_interval": "60", 2021-03-15T22:19:55.955 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2021-03-15T22:19:55.955 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_balance_parent_reads": "false", 2021-03-15T22:19:55.956 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_balance_snap_reads": "false", 2021-03-15T22:19:55.956 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_blkin_trace_all": "false", 2021-03-15T22:19:55.956 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_blocklist_expire_seconds": "0", 2021-03-15T22:19:55.956 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_blocklist_on_break_lock": "true", 2021-03-15T22:19:55.956 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache": "true", 2021-03-15T22:19:55.957 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_block_writes_upfront": "false", 2021-03-15T22:19:55.957 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_max_dirty": "25165824", 2021-03-15T22:19:55.957 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_max_dirty_age": "1.000000", 2021-03-15T22:19:55.957 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_max_dirty_object": "0", 2021-03-15T22:19:55.957 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_policy": "writearound", 2021-03-15T22:19:55.958 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_size": "33554432", 2021-03-15T22:19:55.958 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_target_dirty": "16777216", 2021-03-15T22:19:55.958 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_cache_writethrough_until_flush": "true", 2021-03-15T22:19:55.958 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_clone_copy_on_read": "false", 2021-03-15T22:19:55.959 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_compression_hint": "none", 2021-03-15T22:19:55.959 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_concurrent_management_ops": "10", 2021-03-15T22:19:55.959 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_config_pool_override_update_timestamp": "0", 2021-03-15T22:19:55.959 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_clone_format": "auto", 2021-03-15T22:19:55.959 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_data_pool": "", 2021-03-15T22:19:55.960 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_features": "61", 2021-03-15T22:19:55.960 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_format": "2", 2021-03-15T22:19:55.960 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_map_options": "", 2021-03-15T22:19:55.960 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_order": "22", 2021-03-15T22:19:55.960 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_pool": "rbd", 2021-03-15T22:19:55.961 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2021-03-15T22:19:55.961 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_stripe_count": "0", 2021-03-15T22:19:55.961 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_default_stripe_unit": "0", 2021-03-15T22:19:55.961 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_disable_zero_copy_writes": "true", 2021-03-15T22:19:55.962 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_discard_granularity_bytes": "65536", 2021-03-15T22:19:55.962 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_discard_on_zeroed_write_same": "true", 2021-03-15T22:19:55.962 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_enable_alloc_hint": "true", 2021-03-15T22:19:55.962 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2021-03-15T22:19:55.962 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_io_scheduler": "simple", 2021-03-15T22:19:55.963 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2021-03-15T22:19:55.963 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_commit_age": "5.000000", 2021-03-15T22:19:55.963 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2021-03-15T22:19:55.963 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_max_payload_bytes": "16384", 2021-03-15T22:19:55.964 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_flush_age": "0.000000", 2021-03-15T22:19:55.964 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_flush_bytes": "1048576", 2021-03-15T22:19:55.964 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_flush_interval": "0", 2021-03-15T22:19:55.964 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2021-03-15T22:19:55.964 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2021-03-15T22:19:55.965 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_order": "24", 2021-03-15T22:19:55.965 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_pool": "", 2021-03-15T22:19:55.965 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_journal_splay_width": "4", 2021-03-15T22:19:55.965 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_localize_parent_reads": "false", 2021-03-15T22:19:55.965 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_localize_snap_reads": "false", 2021-03-15T22:19:55.966 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2021-03-15T22:19:55.966 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2021-03-15T22:19:55.966 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2021-03-15T22:19:55.966 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2021-03-15T22:19:55.966 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2021-03-15T22:19:55.967 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2021-03-15T22:19:55.967 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_type": "simple", 2021-03-15T22:19:55.967 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2021-03-15T22:19:55.967 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_image_state_check_interval": "30", 2021-03-15T22:19:55.968 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2021-03-15T22:19:55.968 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2021-03-15T22:19:55.968 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2021-03-15T22:19:55.968 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2021-03-15T22:19:55.968 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2021-03-15T22:19:55.969 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_autotune": "true", 2021-03-15T22:19:55.969 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_base": "805306368", 2021-03-15T22:19:55.969 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2021-03-15T22:19:55.969 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_cache_min": "134217728", 2021-03-15T22:19:55.969 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2021-03-15T22:19:55.970 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2021-03-15T22:19:55.970 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_memory_target": "4294967296", 2021-03-15T22:19:55.970 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_perf_stats_prio": "5", 2021-03-15T22:19:55.970 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2021-03-15T22:19:55.970 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2021-03-15T22:19:55.971 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_delete_delay": "0", 2021-03-15T22:19:55.971 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_max_mirroring_snapshots": "3", 2021-03-15T22:19:55.971 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_replay_delay": "0", 2021-03-15T22:19:55.971 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2021-03-15T22:19:55.972 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2021-03-15T22:19:55.972 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_move_to_trash_on_remove": "false", 2021-03-15T22:19:55.972 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2021-03-15T22:19:55.972 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_mtime_update_interval": "60", 2021-03-15T22:19:55.972 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_non_blocking_aio": "true", 2021-03-15T22:19:55.973 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_op_thread_timeout": "60", 2021-03-15T22:19:55.973 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_op_threads": "1", 2021-03-15T22:19:55.973 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_parent_cache_enabled": "false", 2021-03-15T22:19:55.973 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_persistent_cache_log_periodic_stats": "false", 2021-03-15T22:19:55.973 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_persistent_cache_mode": "disabled", 2021-03-15T22:19:55.974 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_persistent_cache_path": "/tmp", 2021-03-15T22:19:55.974 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_persistent_cache_size": "1073741824", 2021-03-15T22:19:55.974 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_plugins": "", 2021-03-15T22:19:55.974 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_bps_burst": "0", 2021-03-15T22:19:55.974 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_bps_burst_seconds": "1", 2021-03-15T22:19:55.975 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_bps_limit": "0", 2021-03-15T22:19:55.975 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_exclude_ops": "0", 2021-03-15T22:19:55.975 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_iops_burst": "0", 2021-03-15T22:19:55.975 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_iops_burst_seconds": "1", 2021-03-15T22:19:55.975 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_iops_limit": "0", 2021-03-15T22:19:55.976 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_bps_burst": "0", 2021-03-15T22:19:55.976 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2021-03-15T22:19:55.976 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_bps_limit": "0", 2021-03-15T22:19:55.976 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_iops_burst": "0", 2021-03-15T22:19:55.977 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2021-03-15T22:19:55.977 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_read_iops_limit": "0", 2021-03-15T22:19:55.977 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_schedule_tick_min": "50", 2021-03-15T22:19:55.977 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_bps_burst": "0", 2021-03-15T22:19:55.977 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2021-03-15T22:19:55.977 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_bps_limit": "0", 2021-03-15T22:19:55.978 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_iops_burst": "0", 2021-03-15T22:19:55.978 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2021-03-15T22:19:55.978 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_qos_write_iops_limit": "0", 2021-03-15T22:19:55.978 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_quiesce_notification_attempts": "10", 2021-03-15T22:19:55.979 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_read_from_replica_policy": "default", 2021-03-15T22:19:55.979 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2021-03-15T22:19:55.979 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_readahead_max_bytes": "524288", 2021-03-15T22:19:55.979 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_readahead_trigger_requests": "10", 2021-03-15T22:19:55.979 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_request_timed_out_seconds": "30", 2021-03-15T22:19:55.980 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_skip_partial_discard": "true", 2021-03-15T22:19:55.980 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2021-03-15T22:19:55.980 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_tracing": "false", 2021-03-15T22:19:55.980 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_validate_names": "true", 2021-03-15T22:19:55.980 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd_validate_pool": "true", 2021-03-15T22:19:55.981 INFO:teuthology.orchestra.run.smithi076.stdout: "restapi_base_url": "", 2021-03-15T22:19:55.981 INFO:teuthology.orchestra.run.smithi076.stdout: "restapi_log_level": "", 2021-03-15T22:19:55.981 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_acl_grants_max_num": "100", 2021-03-15T22:19:55.981 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_admin_entry": "admin", 2021-03-15T22:19:55.982 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_barbican_url": "", 2021-03-15T22:19:55.982 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2021-03-15T22:19:55.982 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_default_quota_max_size": "-1", 2021-03-15T22:19:55.982 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_index_max_aio": "128", 2021-03-15T22:19:55.983 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_quota_cache_size": "10000", 2021-03-15T22:19:55.983 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_quota_soft_threshold": "0.950000", 2021-03-15T22:19:55.983 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_bucket_quota_ttl": "600", 2021-03-15T22:19:55.983 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cache_enabled": "true", 2021-03-15T22:19:55.983 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cache_expiry_interval": "900", 2021-03-15T22:19:55.984 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cache_lru_size": "10000", 2021-03-15T22:19:55.984 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_content_length_compat": "false", 2021-03-15T22:19:55.984 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_copy_obj_progress": "true", 2021-03-15T22:19:55.984 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2021-03-15T22:19:55.985 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cors_rules_max_num": "100", 2021-03-15T22:19:55.985 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_cross_domain_policy": "", 2021-03-15T22:19:55.985 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_default_encryption_key": "", 2021-03-15T22:19:55.985 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_addr": "", 2021-03-15T22:19:55.985 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_ca_path": "", 2021-03-15T22:19:55.986 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_client_cert": "", 2021-03-15T22:19:55.986 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_client_key": "", 2021-03-15T22:19:55.986 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_kms_key_template": "", 2021-03-15T22:19:55.986 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_password": "", 2021-03-15T22:19:55.987 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2021-03-15T22:19:55.987 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_kmip_username": "", 2021-03-15T22:19:55.987 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_require_ssl": "true", 2021-03-15T22:19:55.987 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2021-03-15T22:19:55.987 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2021-03-15T22:19:55.988 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_suppress_logs": "true", 2021-03-15T22:19:55.988 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_addr": "", 2021-03-15T22:19:55.988 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_auth": "token", 2021-03-15T22:19:55.988 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_namespace": "", 2021-03-15T22:19:55.988 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_prefix": "", 2021-03-15T22:19:55.989 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_secret_engine": "transit", 2021-03-15T22:19:55.989 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_crypt_vault_token_file": "", 2021-03-15T22:19:55.989 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_buffersize": "524288", 2021-03-15T22:19:55.989 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_low_speed_limit": "1024", 2021-03-15T22:19:55.989 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_low_speed_time": "300", 2021-03-15T22:19:55.990 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_curl_wait_timeout_ms": "1000", 2021-03-15T22:19:55.990 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2021-03-15T22:19:55.990 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_backing": "auto", 2021-03-15T22:19:55.990 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_changes_size": "1000", 2021-03-15T22:19:55.991 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_num_shards": "128", 2021-03-15T22:19:55.991 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_obj_prefix": "data_log", 2021-03-15T22:19:55.991 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_log_window": "30", 2021-03-15T22:19:55.991 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_data_notify_interval_msec": "200", 2021-03-15T22:19:55.991 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_realm_info_oid": "default.realm", 2021-03-15T22:19:55.992 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_region_info_oid": "default.region", 2021-03-15T22:19:55.992 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_zone_info_oid": "default.zone", 2021-03-15T22:19:55.992 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2021-03-15T22:19:55.992 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_defer_to_bucket_acls": "", 2021-03-15T22:19:55.993 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_delete_multi_obj_max_num": "1000", 2021-03-15T22:19:55.993 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_admin_lim": "0.000000", 2021-03-15T22:19:55.993 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_admin_res": "100.000000", 2021-03-15T22:19:55.993 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2021-03-15T22:19:55.994 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_auth_lim": "0.000000", 2021-03-15T22:19:55.994 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_auth_res": "200.000000", 2021-03-15T22:19:55.994 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2021-03-15T22:19:55.995 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_data_lim": "0.000000", 2021-03-15T22:19:55.995 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_data_res": "500.000000", 2021-03-15T22:19:55.995 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_data_wgt": "500.000000", 2021-03-15T22:19:55.995 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2021-03-15T22:19:55.996 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_metadata_res": "500.000000", 2021-03-15T22:19:55.996 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2021-03-15T22:19:55.996 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dns_name": "", 2021-03-15T22:19:55.997 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dns_s3website_name": "", 2021-03-15T22:19:55.997 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_dynamic_resharding": "true", 2021-03-15T22:19:55.997 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2021-03-15T22:19:55.997 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_gc_threads": "true", 2021-03-15T22:19:55.998 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_lc_threads": "true", 2021-03-15T22:19:55.998 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_ops_log": "false", 2021-03-15T22:19:55.998 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_quota_threads": "true", 2021-03-15T22:19:55.998 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_static_website": "false", 2021-03-15T22:19:55.998 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enable_usage_log": "false", 2021-03-15T22:19:55.999 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_enforce_swift_acls": "true", 2021-03-15T22:19:55.999 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_exit_timeout_secs": "120", 2021-03-15T22:19:55.999 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_expose_bucket": "false", 2021-03-15T22:19:55.999 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_extended_http_attrs": "", 2021-03-15T22:19:55.999 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_fcgi_socket_backlog": "1024", 2021-03-15T22:19:56.000 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2021-03-15T22:19:56.000 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_frontends": "beast port=7480", 2021-03-15T22:19:56.000 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_concurrent_io": "10", 2021-03-15T22:19:56.000 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_deferred": "50", 2021-03-15T22:19:56.000 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2021-03-15T22:19:56.001 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_objs": "32", 2021-03-15T22:19:56.001 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_queue_size": "134213632", 2021-03-15T22:19:56.001 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_max_trim_chunk": "16", 2021-03-15T22:19:56.001 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_obj_min_wait": "7200", 2021-03-15T22:19:56.002 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_processor_max_time": "3600", 2021-03-15T22:19:56.002 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_gc_processor_period": "3600", 2021-03-15T22:19:56.002 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_get_obj_max_req_size": "4194304", 2021-03-15T22:19:56.002 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_get_obj_window_size": "16777216", 2021-03-15T22:19:56.002 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_healthcheck_disabling_path": "", 2021-03-15T22:19:56.003 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_host": "", 2021-03-15T22:19:56.003 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ignore_get_invalid_range": "false", 2021-03-15T22:19:56.003 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_init_timeout": "300", 2021-03-15T22:19:56.004 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2021-03-15T22:19:56.004 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_accepted_admin_roles": "", 2021-03-15T22:19:56.005 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2021-03-15T22:19:56.005 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_domain": "", 2021-03-15T22:19:56.005 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_password": "", 2021-03-15T22:19:56.005 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_password_path": "", 2021-03-15T22:19:56.006 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_project": "", 2021-03-15T22:19:56.006 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_tenant": "", 2021-03-15T22:19:56.006 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_token": "", 2021-03-15T22:19:56.006 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_token_path": "", 2021-03-15T22:19:56.006 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_admin_user": "", 2021-03-15T22:19:56.007 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_api_version": "2", 2021-03-15T22:19:56.007 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_domain": "", 2021-03-15T22:19:56.007 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_password": "", 2021-03-15T22:19:56.007 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_project": "", 2021-03-15T22:19:56.008 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_tenant": "", 2021-03-15T22:19:56.008 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_barbican_user": "", 2021-03-15T22:19:56.008 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_implicit_tenants": "false", 2021-03-15T22:19:56.008 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_token_cache_size": "10000", 2021-03-15T22:19:56.008 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_url": "", 2021-03-15T22:19:56.009 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_keystone_verify_ssl": "true", 2021-03-15T22:19:56.009 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_debug_interval": "-1", 2021-03-15T22:19:56.009 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_lock_max_time": "90", 2021-03-15T22:19:56.009 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_objs": "32", 2021-03-15T22:19:56.010 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_rules": "1000", 2021-03-15T22:19:56.010 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_worker": "3", 2021-03-15T22:19:56.010 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_max_wp_worker": "3", 2021-03-15T22:19:56.010 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lc_thread_delay": "0", 2021-03-15T22:19:56.011 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2021-03-15T22:19:56.011 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_dnattr": "uid", 2021-03-15T22:19:56.011 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2021-03-15T22:19:56.011 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_searchfilter": "", 2021-03-15T22:19:56.011 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2021-03-15T22:19:56.012 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ldap_uri": "ldaps://", 2021-03-15T22:19:56.012 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2021-03-15T22:19:56.012 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_list_bucket_min_readahead": "1000", 2021-03-15T22:19:56.012 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_list_buckets_max_chunk": "1000", 2021-03-15T22:19:56.013 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_http_headers": "", 2021-03-15T22:19:56.013 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_nonexistent_bucket": "false", 2021-03-15T22:19:56.013 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2021-03-15T22:19:56.014 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_log_object_name_utc": "false", 2021-03-15T22:19:56.014 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_luarocks_location": "/tmp/luarocks", 2021-03-15T22:19:56.014 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_attr_name_len": "0", 2021-03-15T22:19:56.014 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_attr_size": "0", 2021-03-15T22:19:56.014 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_attrs_num_in_req": "0", 2021-03-15T22:19:56.015 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_chunk_size": "4194304", 2021-03-15T22:19:56.015 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_concurrent_requests": "1024", 2021-03-15T22:19:56.015 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_dynamic_shards": "1999", 2021-03-15T22:19:56.015 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_listing_results": "1000", 2021-03-15T22:19:56.016 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_notify_retries": "3", 2021-03-15T22:19:56.016 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_objs_per_shard": "100000", 2021-03-15T22:19:56.016 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_put_param_size": "1048576", 2021-03-15T22:19:56.016 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_put_size": "5368709120", 2021-03-15T22:19:56.016 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_max_slo_entries": "1000", 2021-03-15T22:19:56.017 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_md_log_max_shards": "64", 2021-03-15T22:19:56.017 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_md_notify_interval_msec": "200", 2021-03-15T22:19:56.017 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_mime_types_file": "/etc/mime.types", 2021-03-15T22:19:56.017 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_mp_lock_max_time": "600", 2021-03-15T22:19:56.018 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_multipart_min_part_size": "5242880", 2021-03-15T22:19:56.018 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_multipart_part_upload_limit": "10000", 2021-03-15T22:19:56.018 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_fhcache_partitions": "3", 2021-03-15T22:19:56.018 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_fhcache_size": "2017", 2021-03-15T22:19:56.018 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2021-03-15T22:19:56.019 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_lru_lanes": "5", 2021-03-15T22:19:56.019 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_max_gc": "300", 2021-03-15T22:19:56.020 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_namespace_expire_secs": "300", 2021-03-15T22:19:56.020 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_gc_threads": "false", 2021-03-15T22:19:56.020 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_lc_threads": "false", 2021-03-15T22:19:56.020 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_quota_threads": "false", 2021-03-15T22:19:56.021 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_run_sync_thread": "false", 2021-03-15T22:19:56.021 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_s3_fast_attrs": "false", 2021-03-15T22:19:56.021 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_nfs_write_completion_interval_s": "10", 2021-03-15T22:19:56.021 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_num_async_rados_threads": "32", 2021-03-15T22:19:56.021 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_num_control_oids": "8", 2021-03-15T22:19:56.022 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_num_rados_handles": "1", 2021-03-15T22:19:56.022 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_numa_node": "-1", 2021-03-15T22:19:56.022 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_obj_stripe_size": "4194304", 2021-03-15T22:19:56.022 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_obj_tombstone_cache_size": "1000", 2021-03-15T22:19:56.023 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_objexp_chunk_size": "100", 2021-03-15T22:19:56.023 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_objexp_gc_interval": "600", 2021-03-15T22:19:56.023 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_objexp_hints_num_shards": "127", 2021-03-15T22:19:56.023 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_olh_pending_timeout_sec": "3600", 2021-03-15T22:19:56.023 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_op_thread_suicide_timeout": "0", 2021-03-15T22:19:56.024 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_op_thread_timeout": "600", 2021-03-15T22:19:56.024 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_op_tracing": "false", 2021-03-15T22:19:56.024 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_opa_token": "", 2021-03-15T22:19:56.024 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_opa_url": "", 2021-03-15T22:19:56.024 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_opa_verify_ssl": "true", 2021-03-15T22:19:56.025 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ops_log_data_backlog": "5242880", 2021-03-15T22:19:56.025 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ops_log_rados": "true", 2021-03-15T22:19:56.025 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_ops_log_socket_path": "", 2021-03-15T22:19:56.025 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_override_bucket_index_max_shards": "0", 2021-03-15T22:19:56.026 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2021-03-15T22:19:56.026 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_push_interval": "2.000000", 2021-03-15T22:19:56.026 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_push_interval_max": "30.000000", 2021-03-15T22:19:56.026 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_period_root_pool": ".rgw.root", 2021-03-15T22:19:56.026 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_port": "", 2021-03-15T22:19:56.027 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_print_continue": "true", 2021-03-15T22:19:56.027 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_print_prohibited_content_length": "false", 2021-03-15T22:19:56.027 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_put_obj_max_window_size": "67108864", 2021-03-15T22:19:56.027 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_put_obj_min_window_size": "16777216", 2021-03-15T22:19:56.028 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2021-03-15T22:19:56.028 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rados_pool_pg_num_min": "8", 2021-03-15T22:19:56.028 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rados_pool_recovery_priority": "5", 2021-03-15T22:19:56.028 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rados_tracing": "false", 2021-03-15T22:19:56.028 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_realm": "", 2021-03-15T22:19:56.029 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_realm_root_pool": ".rgw.root", 2021-03-15T22:19:56.029 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_region": "", 2021-03-15T22:19:56.029 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_region_root_pool": ".rgw.root", 2021-03-15T22:19:56.029 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_relaxed_region_enforcement": "false", 2021-03-15T22:19:56.029 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_relaxed_s3_bucket_names": "false", 2021-03-15T22:19:56.030 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2021-03-15T22:19:56.030 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_request_uri": "", 2021-03-15T22:19:56.030 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_batch_size": "64", 2021-03-15T22:19:56.030 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_bucket_lock_duration": "360", 2021-03-15T22:19:56.031 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_max_aio": "128", 2021-03-15T22:19:56.031 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_num_logs": "16", 2021-03-15T22:19:56.031 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_reshard_thread_interval": "600", 2021-03-15T22:19:56.031 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_resolve_cname": "false", 2021-03-15T22:19:56.031 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_rest_getusage_op_compat": "false", 2021-03-15T22:19:56.032 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_run_sync_thread": "true", 2021-03-15T22:19:56.032 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_order": "sts, external, local", 2021-03-15T22:19:56.032 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_keystone": "false", 2021-03-15T22:19:56.032 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_ldap": "false", 2021-03-15T22:19:56.032 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_rados": "true", 2021-03-15T22:19:56.033 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_auth_use_sts": "false", 2021-03-15T22:19:56.033 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_s3_success_create_obj_status": "0", 2021-03-15T22:19:56.033 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_safe_max_objects_per_shard": "102400", 2021-03-15T22:19:56.033 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_scheduler_type": "throttler", 2021-03-15T22:19:56.033 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_script_uri": "", 2021-03-15T22:19:56.034 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_service_provider_name": "", 2021-03-15T22:19:56.034 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_shard_warning_threshold": "90.000000", 2021-03-15T22:19:56.034 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_socket_path": "", 2021-03-15T22:19:56.034 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_client_id": "", 2021-03-15T22:19:56.035 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_client_secret": "", 2021-03-15T22:19:56.035 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_entry": "sts", 2021-03-15T22:19:56.035 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_key": "sts", 2021-03-15T22:19:56.035 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_max_session_duration": "43200", 2021-03-15T22:19:56.035 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_min_session_duration": "900", 2021-03-15T22:19:56.036 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sts_token_introspection_url": "", 2021-03-15T22:19:56.036 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_account_in_url": "false", 2021-03-15T22:19:56.036 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_auth_entry": "auth", 2021-03-15T22:19:56.036 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_auth_url": "", 2021-03-15T22:19:56.037 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_custom_header": "", 2021-03-15T22:19:56.037 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_enforce_content_length": "false", 2021-03-15T22:19:56.037 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_need_stats": "true", 2021-03-15T22:19:56.037 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_tenant_name": "", 2021-03-15T22:19:56.037 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_token_expiration": "86400", 2021-03-15T22:19:56.038 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_url": "", 2021-03-15T22:19:56.038 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_url_prefix": "swift", 2021-03-15T22:19:56.038 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_swift_versioning_enabled": "false", 2021-03-15T22:19:56.038 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2021-03-15T22:19:56.038 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_lease_period": "120", 2021-03-15T22:19:56.039 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2021-03-15T22:19:56.039 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_interval": "1200", 2021-03-15T22:19:56.039 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_max_buckets": "16", 2021-03-15T22:19:56.039 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2021-03-15T22:19:56.040 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2021-03-15T22:19:56.040 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_obj_etag_verify": "false", 2021-03-15T22:19:56.040 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_trace_history_size": "4096", 2021-03-15T22:19:56.040 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_trace_per_node_log_size": "32", 2021-03-15T22:19:56.040 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2021-03-15T22:19:56.041 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_thread_pool_size": "512", 2021-03-15T22:19:56.041 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_comment": "", 2021-03-15T22:19:56.041 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_createby": "", 2021-03-15T22:19:56.041 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_encoding": "", 2021-03-15T22:19:56.042 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_flag": "false", 2021-03-15T22:19:56.042 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_origin": "", 2021-03-15T22:19:56.042 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_sha_unit": "524288", 2021-03-15T22:19:56.042 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_torrent_tracker": "", 2021-03-15T22:19:56.042 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_trust_forwarded_https": "false", 2021-03-15T22:19:56.043 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_log_flush_threshold": "1024", 2021-03-15T22:19:56.043 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_log_tick_interval": "30", 2021-03-15T22:19:56.043 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_max_shards": "32", 2021-03-15T22:19:56.043 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_usage_max_user_shards": "1", 2021-03-15T22:19:56.043 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_use_opa_authz": "false", 2021-03-15T22:19:56.044 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_default_quota_max_objects": "-1", 2021-03-15T22:19:56.044 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_default_quota_max_size": "-1", 2021-03-15T22:19:56.044 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_max_buckets": "1000", 2021-03-15T22:19:56.044 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2021-03-15T22:19:56.045 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_sync_idle_users": "false", 2021-03-15T22:19:56.045 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_sync_interval": "86400", 2021-03-15T22:19:56.045 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_quota_sync_wait_time": "86400", 2021-03-15T22:19:56.045 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_user_unique_email": "true", 2021-03-15T22:19:56.045 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_verify_ssl": "true", 2021-03-15T22:19:56.046 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_website_routing_rules_max_num": "50", 2021-03-15T22:19:56.046 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zone": "", 2021-03-15T22:19:56.046 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zone_root_pool": ".rgw.root", 2021-03-15T22:19:56.046 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zonegroup": "", 2021-03-15T22:19:56.047 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2021-03-15T22:19:56.047 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_block_size": "4096", 2021-03-15T22:19:56.047 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_bloom_bits_per_key": "20", 2021-03-15T22:19:56.047 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2021-03-15T22:19:56.047 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2021-03-15T22:19:56.048 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_row_ratio": "0.000000", 2021-03-15T22:19:56.048 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_shard_bits": "4", 2021-03-15T22:19:56.048 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_size": "536870912", 2021-03-15T22:19:56.048 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_cache_type": "binned_lru", 2021-03-15T22:19:56.048 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_collect_compaction_stats": "false", 2021-03-15T22:19:56.049 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_collect_extended_stats": "false", 2021-03-15T22:19:56.049 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_collect_memory_stats": "false", 2021-03-15T22:19:56.049 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_delete_range_threshold": "1048576", 2021-03-15T22:19:56.049 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_index_type": "binary_search", 2021-03-15T22:19:56.049 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_log_to_ceph_log": "true", 2021-03-15T22:19:56.050 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_metadata_block_size": "4096", 2021-03-15T22:19:56.050 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_partition_filters": "false", 2021-03-15T22:19:56.050 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_perf": "false", 2021-03-15T22:19:56.050 INFO:teuthology.orchestra.run.smithi076.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2021-03-15T22:19:56.050 INFO:teuthology.orchestra.run.smithi076.stdout: "rotating_keys_bootstrap_timeout": "30", 2021-03-15T22:19:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: "rotating_keys_renewal_timeout": "10", 2021-03-15T22:19:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: "run_dir": "/var/run/ceph", 2021-03-15T22:19:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: "setgroup": "ceph", 2021-03-15T22:19:56.051 INFO:teuthology.orchestra.run.smithi076.stdout: "setuser": "ceph", 2021-03-15T22:19:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "setuser_match_path": "", 2021-03-15T22:19:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_misplaced_ratio": "0.050000", 2021-03-15T22:19:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "thp": "false", 2021-03-15T22:19:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "threadpool_default_timeout": "60", 2021-03-15T22:19:56.052 INFO:teuthology.orchestra.run.smithi076.stdout: "threadpool_empty_queue_max_wait": "2", 2021-03-15T22:19:56.053 INFO:teuthology.orchestra.run.smithi076.stdout: "throttler_perf_counter": "true" 2021-03-15T22:19:56.053 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:19:56.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:55 smithi076 conmon[56514]: cluster 2021-03-15T22:19: 2021-03-15T22:19:56.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:55 smithi076 conmon[56514]: 54.973875+0000 mgr.y (mgr.14144) 357 : cluster [DBG] pgmap v256: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:56.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:55 smithi076 conmon[56514]: cluster 2021-03-15T22:19:55.098233+0000 osd.4 (osd.4) 19 : cluster [DBG] 3.9 deep-scrub ok 2021-03-15T22:19:56.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:55 smithi076 conmon[62156]: cluster 2021-03-15T22:19:54.973875+0000 mgr.y 2021-03-15T22:19:56.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:55 smithi076 conmon[62156]: (mgr.14144) 357 : cluster [DBG] pgmap v256: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:56.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:55 smithi076 conmon[62156]: cluster 2021-03-15T22:19:55.098233+0000 osd.4 (osd. 2021-03-15T22:19:56.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:55 smithi076 conmon[62156]: 4) 19 : cluster [DBG] 3.9 deep-scrub ok 2021-03-15T22:19:56.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:55 smithi125 conmon[56739]: cluster 2021-03-15T22:19:54.973875+0000 mgr.y (mgr.14144) 357 : cluster [DBG] 2021-03-15T22:19:56.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:55 smithi125 conmon[56739]: pgmap v256: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:56.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:55 smithi125 conmon[56739]: cluster 2021-03-15T22:19:55.098233+0000 osd.4 (osd.4) 19 : cluster [DBG] 3.9 deep-scrub ok 2021-03-15T22:19:56.394 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2021-03-15T22:19:56.675 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:57.136 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[56514]: cluster 2021-03-15T22:19:56.454989+0000 osd.4 (osd.4) 20 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:57.137 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[56514]: cluster 2021-03-15T22:19:56.455122+0000 osd.4 (osd.4) 21 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:57.137 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: cluster 2021-03-15T22:19:57.137 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: 2021-03-15T22:19:57.138 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: 2021 2021-03-15T22:19:57.138 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: - 2021-03-15T22:19:57.138 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: 03 2021-03-15T22:19:57.138 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: - 2021-03-15T22:19:57.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: 15 2021-03-15T22:19:57.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: T 2021-03-15T22:19:57.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: 22 2021-03-15T22:19:57.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: : 2021-03-15T22:19:57.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: 19 2021-03-15T22:19:57.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: : 2021-03-15T22:19:57.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: 56 2021-03-15T22:19:57.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: . 2021-03-15T22:19:57.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: 454989+0000 osd.4 (osd.4) 20 : 2021-03-15T22:19:57.141 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:57.141 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: cluster 2021-03-15T22:19:57.141 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: 2021-03-15T22:19:56.455122+0000 osd.4 ( 2021-03-15T22:19:57.141 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:56 smithi076 conmon[62156]: osd.4) 21 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:57.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:56 smithi125 conmon[56739]: cluster 2021-03-15T22:19:56.454989+0000 osd.4 (osd 2021-03-15T22:19:57.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:56 smithi125 conmon[56739]: .4) 20 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:19:57.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:56 smithi125 conmon[56739]: cluster 2021-03-15T22:19:56.455122+0000 osd.4 (osd.4) 21 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:19:58.064 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:57 smithi076 conmon[56514]: cluster 2021-03-15T22:19:56.974355 2021-03-15T22:19:58.065 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:57 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 358 : cluster [DBG] pgmap v257: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:58.065 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:57 smithi076 conmon[56514]: cluster 2021-03-15T22:19:56.990872+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-03-15T22:19:58.065 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:57 smithi076 conmon[56514]: cluster 2021-03-15T22:19:57.014330+0000 osd.7 (osd.7) 23 : cluster [DBG] 5.1a scrub ok 2021-03-15T22:19:58.065 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:57 smithi076 conmon[56514]: cluster 2021-03-15T22:19:57.135015+0000 osd.4 (osd.4) 22 : cluster [DBG] 5.e scrub ok 2021-03-15T22:19:58.066 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:57 smithi076 conmon[62156]: cluster 2021-03-15T22:19:56.974355+0000 mgr.y ( 2021-03-15T22:19:58.066 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:57 smithi076 conmon[62156]: mgr.14144) 358 : cluster [DBG] pgmap v257: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:58.066 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:57 smithi076 conmon[62156]: cluster 2021-03-15T22:19:56.990872+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-03-15T22:19:58.067 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:57 smithi076 conmon[62156]: cluster 2021-03-15T22:19:57.014330+0000 osd.7 (osd.7) 23 : cluster [DBG] 5.1a scrub ok 2021-03-15T22:19:58.067 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:57 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:19:58.067 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:57 smithi076 conmon[62156]: -15T22:19:57.135015+0000 osd.4 (osd.4) 22 : cluster [DBG] 5.e scrub ok 2021-03-15T22:19:58.257 INFO:teuthology.orchestra.run.smithi076.stdout:mon.a: {} 2021-03-15T22:19:58.266 INFO:teuthology.orchestra.run.smithi076.stdout:mon.c: {} 2021-03-15T22:19:58.275 INFO:teuthology.orchestra.run.smithi076.stdout:mon.b: {} 2021-03-15T22:19:58.276 INFO:teuthology.orchestra.run.smithi076.stderr:mon.a: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2021-03-15T22:19:58.276 INFO:teuthology.orchestra.run.smithi076.stderr:mon.c: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2021-03-15T22:19:58.277 INFO:teuthology.orchestra.run.smithi076.stderr:mon.b: mon_osd_down_out_interval = '0' (not observed, change may require restart) 2021-03-15T22:19:58.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:57 smithi125 conmon[56739]: cluster 2021-03-15T22:19:56.974355+0000 mgr.y (mgr.14144) 358 : cluster [DBG] 2021-03-15T22:19:58.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:57 smithi125 conmon[56739]: pgmap v257: 132 pgs: 132 active+clean; 25 KiB data, 52 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:19:58.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:57 smithi125 conmon[56739]: cluster 2021-03-15T22:19:56.990872+0000 mon.a (mon.0) 593 : cluster [DBG] osdmap e90: 8 total, 8 up, 8 in 2021-03-15T22:19:58.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:57 smithi125 conmon[56739]: cluster 2021-03-15T22:19:57.014330+0000 2021-03-15T22:19:58.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:57 smithi125 conmon[56739]: osd.7 (osd.7) 23 : cluster [DBG] 5.1a scrub ok 2021-03-15T22:19:58.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:57 smithi125 conmon[56739]: cluster 2021-03-15T22:19:58.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:57 smithi125 conmon[56739]: 2021-03-15T22:19:57.135015+0000 osd.4 (osd.4) 22 : 2021-03-15T22:19:58.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:57 smithi125 conmon[56739]: cluster [DBG] 5.e scrub ok 2021-03-15T22:19:58.926 INFO:teuthology.run_tasks:Running task rados... 2021-03-15T22:19:58.940 INFO:tasks.rados:Beginning rados... 2021-03-15T22:19:58.940 DEBUG:teuthology.run_tasks:Unwinding manager rados 2021-03-15T22:19:58.991 INFO:tasks.rados:joining rados 2021-03-15T22:19:58.992 INFO:tasks.thrashosds.thrasher:starting do_thrash 2021-03-15T22:19:58.992 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:19:58.992 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:19:58.993 INFO:tasks.thrashosds.thrasher:inject_pause on 1 2021-03-15T22:19:58.993 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-03-15T22:19:58.993 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-03-15T22:19:58.994 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 config set filestore_inject_stall 3 2021-03-15T22:19:58.994 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2021-03-15T22:19:58.995 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:19:58.995 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2021-03-15T22:19:58.996 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:19:58.996 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2021-03-15T22:19:58.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:19:58.997 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2021-03-15T22:19:58.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:19:58.998 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2021-03-15T22:19:58.998 INFO:tasks.rados:starting run 0 out of 1 2021-03-15T22:19:58.999 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2021-03-15T22:19:58.999 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool create unique_pool_0 16 2021-03-15T22:19:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:19:59.029 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:19:59.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:19:59.131 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:19:59.161 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:19:59.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:19:59.163 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:19:59.264 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:19:59.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:58 smithi076 conmon[56514]: audit 2021-03-15T22:19:58.246333+0000 mon.a (mon.0) 594 : audit [DBG] 2021-03-15T22:19:59.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:58 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/772724530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-15T22:19:59.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:58 smithi076 conmon[56514]: cluster 2021-03-15T22:19:58.974842+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v259: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:19:59.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:58 smithi076 conmon[62156]: audit 2021-03-15T22:19:58.246333+0000 2021-03-15T22:19:59.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:58 smithi076 conmon[62156]: mon.a (mon.0) 594 : audit [DBG] from='client.? 172.21.15.76:0/772724530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-15T22:19:59.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:58 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:19:59.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:58 smithi076 conmon[62156]: :19:58.974842+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v259: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:19:59.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:58 smithi125 conmon[56739]: audit 2021-03-15T22:19:58.246333+0000 mon.a (mon.0) 594 : audit [DBG] from='client.? 172.21.15.76:0/772724530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2021-03-15T22:19:59.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:58 smithi125 conmon[56739]: 2021-03-15T22:19:59.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:58 smithi125 conmon[56739]: cluster 2021-03-15T22:19:58.974842+0000 mgr.y (mgr.14144) 359 : cluster [DBG] pgmap v259: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:19:59.296 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:19:59.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:19:59.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:19:59.384 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:59.399 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:19:59.425 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:19:59.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:19:59.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:19:59.432 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:59.504 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:59.528 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:19:59.534 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:59.553 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:19:59.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:19:59.555 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:19:59.584 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:19:59.656 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:19:59.682 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:19:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:19:59.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:19:59.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:19:59.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:19:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:19:59.810 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:19:59.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:19:59.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:19:59.913 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:19:59.939 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:19:59.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:19:59.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:00.042 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:00.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:00.070 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:00.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:00.094 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:59 smithi076 conmon[56514]: cluster 2021-03-15T22:19:58.799539+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.11 deep-scrub ok 2021-03-15T22:20:00.095 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:59 smithi076 conmon[56514]: cluster 2021-03-15T22:19:59.243934+0000 osd.5 (osd.5) 24 : cluster [DBG] 5.f scrub ok 2021-03-15T22:20:00.095 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:19:59 smithi076 conmon[56514]: audit 2021-03-15T22:19:59.400888+0000 mgr.y (mgr.14144) 360 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:00.095 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:59 smithi076 conmon[62156]: cluster 2021-03-15T22:19:58.799539+0000 osd.3 (osd.3) 30 : cluster [DBG] 4.11 deep-scrub ok 2021-03-15T22:20:00.096 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:59 smithi076 conmon[62156]: cluster 2021-03-15T22:19:59.243934+0000 osd.5 (osd.5) 24 : cluster [DBG] 5.f scrub ok 2021-03-15T22:20:00.096 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:19:59 smithi076 conmon[62156]: audit 2021-03-15T22:19:59.400888+0000 mgr.y (mgr.14144) 360 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:00.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:00.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:00.196 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:00.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:00.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:59 smithi125 conmon[56739]: cluster 2021-03-15T22:19:58.799539+0000 osd.3 (osd.3) 30 : cluster 2021-03-15T22:20:00.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:59 smithi125 conmon[56739]: [DBG] 4.11 deep-scrub ok 2021-03-15T22:20:00.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:59 smithi125 conmon[56739]: cluster 2021-03-15T22:19:59.243934+0000 osd.5 (osd.5) 24 : cluster [DBG] 5.f scrub ok 2021-03-15T22:20:00.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:59 smithi125 conmon[56739]: audit 2021-03-15T22:19:59. 2021-03-15T22:20:00.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:19:59 smithi125 conmon[56739]: 400888+0000 mgr.y (mgr.14144) 360 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:00.297 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:00.319 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:00.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:00.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:00.422 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:00.449 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:00.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:00.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:00.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:00.578 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:00.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:00.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:00.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:00.706 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:00.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:00.708 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:00.808 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:00.836 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:00.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:00.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:00.939 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:00.963 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:00.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:00.964 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:01.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:01.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:01.088 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:01.089 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:01.190 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:01.215 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:01.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:01.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:01.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:01 smithi076 conmon[56514]: cluster 2021-03-15T22:20:00.000113+0000 mon.a 2021-03-15T22:20:01.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:01 smithi076 conmon[56514]: (mon.0) 595 : cluster [INF] overall HEALTH_OK 2021-03-15T22:20:01.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:01 smithi076 conmon[56514]: cluster 2021-03-15T22:20:00.206064+0000 osd.4 (osd.4) 23 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:20:01.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:01 smithi076 conmon[56514]: cluster 2021-03-15T22:20:00.975255+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v260: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2021-03-15T22:20:01.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:01 smithi076 conmon[62156]: cluster 2021-03-15T22:20:00.000113+0000 mon.a 2021-03-15T22:20:01.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:01 smithi076 conmon[62156]: (mon.0) 595 : cluster [INF] overall HEALTH_OK 2021-03-15T22:20:01.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:01 smithi076 conmon[62156]: cluster 2021-03-15T22:20:00.206064+0000 osd.4 2021-03-15T22:20:01.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:01 smithi076 conmon[62156]: (osd.4) 23 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:20:01.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:01 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:20:01.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:01 smithi076 conmon[62156]: :20:00.975255+0000 mgr.y (mgr.14144 2021-03-15T22:20:01.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:01 smithi076 conmon[62156]: ) 361 : cluster [DBG] pgmap v260: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2021-03-15T22:20:01.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:01 smithi125 conmon[56739]: cluster 2021-03-15T22:20:00.000113+0000 mon.a (mon.0) 595 : cluster [INF] overall HEALTH_OK 2021-03-15T22:20:01.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:01 smithi125 conmon[56739]: cluster 2021-03-15T22:20:00.206064+0000 osd.4 (osd.4) 23 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:20:01.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:01 smithi125 conmon[56739]: cluster 2021-03-15T22:20:00.975255+0000 mgr.y (mgr.14144) 361 : cluster [DBG] pgmap v260: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1 KiB/s rd, 1 op/s 2021-03-15T22:20:01.318 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:01.345 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:01.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:01.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:01.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:01.478 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:01.479 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:01.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:01.607 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:01.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:01.609 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:01.710 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:01.736 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:01.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:01.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:01.838 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:01.864 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:01.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:01.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:01.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:01.990 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:01.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:01.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:02.006 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:02 smithi076 conmon[56514]: cluster 2021-03-15T22:20:00.809064+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.0 scrub ok 2021-03-15T22:20:02.093 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:02.116 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:02.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:02.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:02.218 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:02.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:02.241 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:02.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:02.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:02 smithi076 conmon[56514]: audit 2021-03-15T22:20:01.039686+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:20:02.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:02 smithi076 conmon[56514]: audit 2021-03-15T22:20:01.040747+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:20:02.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:02 smithi076 conmon[56514]: cluster 2021-03-15T22:20:01.157851+0000 osd.4 (osd.4) 24 : cluster [DBG] 4.e scrub ok 2021-03-15T22:20:02.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:02 smithi076 conmon[62156]: cluster 2021-03-15T22:20:00.809064+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.0 scrub ok 2021-03-15T22:20:02.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:02 smithi076 conmon[62156]: audit 2021-03-15T22:20:01.039686+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:20:02.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:02 smithi076 conmon[62156]: audit 2021-03-15T22:20:01.040747+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:20:02.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:02 smithi076 conmon[62156]: cluster 2021-03-15T22:20:01.157851+0000 osd.4 (osd.4) 24 : cluster [DBG] 4.e scrub ok 2021-03-15T22:20:02.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:02 smithi125 conmon[56739]: cluster 2021-03-15T22:20:00.809064+0000 osd.3 (osd.3) 31 : cluster [DBG] 5.0 scrub ok 2021-03-15T22:20:02.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:02 smithi125 conmon[56739]: 2021-03-15T22:20:02.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:02 smithi125 conmon[56739]: audit 2021-03-15T22:20:01.039686+0000 mon.a (mon.0) 596 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:20:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:02 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:20:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:02 smithi125 conmon[56739]: :20:01.040747+0000 mon.a (mon.0) 597 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:20:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:02 smithi125 conmon[56739]: cluster 2021-03-15T22:20: 2021-03-15T22:20:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:02 smithi125 conmon[56739]: 01.157851+0000 osd.4 (osd.4) 24 : 2021-03-15T22:20:02.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:02 smithi125 conmon[56739]: cluster [DBG] 4.e scrub ok 2021-03-15T22:20:02.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:02.370 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:02.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:02.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:02.498 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:02.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:02.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:02.602 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:02.626 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:02.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:02.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:02.728 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:02.750 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:02.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:02.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:02.853 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:02.875 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:02.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:02.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:02.978 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:03.003 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:03.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:03.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:03.106 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:03.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:03.131 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:03.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:03.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:03 smithi076 conmon[62156]: cluster 2021-03-15T22:20:02. 2021-03-15T22:20:03.195 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:03 smithi076 conmon[62156]: 975756+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v261: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:20:03.195 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:03 smithi076 conmon[56514]: cluster 2021-03-15T22:20:02.975756 2021-03-15T22:20:03.195 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:03 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v261: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:20:03.232 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:03.258 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:03.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:03.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:03.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:03 smithi125 conmon[56739]: cluster 2021-03-15T22:20:02.975756+0000 mgr.y (mgr.14144) 362 : cluster [DBG] pgmap v261: 132 pgs: 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:20:03.362 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:03.389 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:03.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:03.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:03.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:03.514 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:03.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:03.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:03.640 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:03.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:03.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:03.711 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:20:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:20:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:20:03.712 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:20:03.742 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:03.769 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:03.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:03.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:03.872 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:03.895 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:03.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:03.897 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:03.998 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:04.025 INFO:teuthology.orchestra.run.smithi076.stderr:pool 'unique_pool_0' created 2021-03-15T22:20:04.027 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:04.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:04.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:04.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:04.156 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:04.157 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:04.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:04 smithi076 conmon[56514]: audit 2021-03-15T22:20:03.123983+0000 mon.a (mon.0) 598 : audit [INF] from='client.? 172.21.15.76:0/1992903637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-03-15T22:20:04.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:04 smithi076 conmon[56514]: cluster 2021-03-15T22:20:03.134328+0000 osd.7 (osd.7) 24 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:20:04.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:04 smithi076 conmon[62156]: audit 2021-03-15T22:20:03.123983+0000 mon.a (mon.0) 598 : audit [INF] from='client.? 172.21.15.76:0/1992903637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-03-15T22:20:04.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:04 smithi076 conmon[62156]: cluster 2021-03-15T22:20:03.134328+0000 osd.7 (osd.7) 24 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:20:04.258 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:04.283 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:04.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:04.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:04.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:04 smithi125 conmon[56739]: audit 2021-03-15T22:20:03.123983+0000 mon.a (mon.0) 598 : audit [INF] from='client.? 172.21.15.76:0/1992903637' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2021-03-15T22:20:04.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:04 smithi125 conmon[56739]: 2021-03-15T22:20:04.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:04 smithi125 conmon[56739]: cluster 2021-03-15T22:20:03.134328+0000 osd.7 (osd.7) 24 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:20:04.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:04.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:04.410 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:04.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:04.511 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:04.537 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:04.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:04.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:04.611 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:20:04.611 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-03-15T22:20:04.611 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:20:04.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:04.665 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:04.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:04.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:04.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:04.794 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:04.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:04.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:04.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:04.923 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:04.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:04.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:05.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:05.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:05.054 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:05.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:05.156 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:05.183 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:05.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:05.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:05.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:05 smithi076 conmon[56514]: audit 2021-03-15T22:20:04.018842+0000 mon.a (mon.0) 599 2021-03-15T22:20:05.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:05 smithi076 conmon[56514]: : audit [INF] from='client.? 172.21.15.76:0/1992903637' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2021-03-15T22:20:05.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:05 smithi076 conmon[56514]: cluster 2021-03-15T22:20:04.018913+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2021-03-15T22:20:05.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:05 smithi076 conmon[56514]: cluster 2021-03-15T22:20:04.976400+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v263: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:20:05.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:05 smithi076 conmon[62156]: audit 2021-03-15T22:20:04.018842+0000 mon.a (mon 2021-03-15T22:20:05.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:05 smithi076 conmon[62156]: .0) 599 : audit [INF] from='client.? 172.21.15.76:0/1992903637' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2021-03-15T22:20:05.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:05 smithi076 conmon[62156]: cluster 2021-03-15T22:20:04.018913+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2021-03-15T22:20:05.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:05 smithi076 conmon[62156]: cluster 2021-03-15T22:20:04.976400+0000 mgr.y (mgr 2021-03-15T22:20:05.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:05 smithi076 conmon[62156]: .14144) 363 : cluster [DBG] pgmap v263: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:20:05.285 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:05.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:05 smithi125 conmon[56739]: audit 2021-03-15T22:20:04.018842+0000 mon.a (mon.0) 599 : audit [INF] 2021-03-15T22:20:05.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:05 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1992903637' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2021-03-15T22:20:05.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:05 smithi125 conmon[56739]: cluster 2021-03-15T22:20:04.018913+0000 mon.a (mon.0) 600 : cluster [DBG] osdmap e91: 8 total, 8 up, 8 in 2021-03-15T22:20:05.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:05 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:20:05.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:05 smithi125 conmon[56739]: 20:04.976400+0000 mgr.y (mgr.14144) 363 : cluster [DBG] pgmap v263: 148 pgs: 3 creating+peering, 13 unknown, 132 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2021-03-15T22:20:05.312 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:05.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:05.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:05.415 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:05.437 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:05.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:05.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:05.540 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:05.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:05.563 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:05.564 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:05.666 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:05.692 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:05.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:05.694 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:05.775 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:20:05.795 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:05.839 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:05.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:05.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:05.943 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:05.968 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:05.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:06.027 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:06 smithi076 conmon[56514]: cluster 2021-03-15T22:20:05.025971+0000 mon.a (mon.0) 601 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2021-03-15T22:20:06.054 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2021-03-15T22:20:06.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:06.096 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:06.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:06.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:06.184 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:06.200 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:06.225 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:20:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:06.227 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:06.227 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:06.256 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:20:06.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:06 smithi125 conmon[56739]: cluster 2021-03-15T22:20:05.025971+0000 mon.a (mon.0) 601 : cluster [DBG] 2021-03-15T22:20:06.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:06 smithi125 conmon[56739]: osdmap e92: 8 total, 8 up, 8 in 2021-03-15T22:20:06.289 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:20:06.316 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:20:06.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:06.339 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:20:06.356 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:06.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:06.358 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:06.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:06.360 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:20:06.378 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:20:06.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:06 smithi076 conmon[62156]: cluster 2021-03-15T22:20:05.025971+0000 mon.a (mon. 2021-03-15T22:20:06.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:06 smithi076 conmon[62156]: 0) 601 : cluster [DBG] osdmap e92: 8 total, 8 up, 8 in 2021-03-15T22:20:06.404 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:20:06.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:20:06.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:20:06.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:20:06.406 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:20:06.406 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:20:06.406 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:20:06.406 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:20:06.407 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:20:06.460 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:06.485 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:06.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:06.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:06.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:06.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:06.611 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:06.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:06.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:06.739 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:06.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:06.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:06.841 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:06.867 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:06.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:06.969 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:06.992 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:06.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:07.050 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:20:07.095 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:07.121 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:07.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:07.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:07.192 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[56514]: cluster 2021-03-15T22:20:06.037536+0000 2021-03-15T22:20:07.193 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[56514]: mon.a (mon.0) 602 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-03-15T22:20:07.193 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[56514]: audit 2021-03-15T22:20:06.200787+0000 mon.c (mon.1) 13 : audit [DBG] from='client.? 172.21.15.76:0/2756159441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:07.193 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[56514]: audit 2021-03-15T22:20:06.308813+0000 mon.c (mon.1) 14 : audit [INF] from='client.? 172.21.15.76:0/462660668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:20:07.193 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[56514]: audit 2021-03-15T22:20:06.309150+0000 mon.a (mon.0) 603 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:20:07.194 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[56514]: cluster 2021-03-15T22:20:06.976836+0000 mgr.y (mgr.14144) 364 : cluster [DBG] pgmap v266: 148 pgs: 3 creating+peering, 3 unknown, 142 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:20:07.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: cluster 2021-03-15T22:20:07.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:20:07.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: :20:06.037536+0000 mon.a (mon.0) 602 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-03-15T22:20:07.195 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:20:07.195 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: 03-15T22:20:06.200787+0000 mon.c ( 2021-03-15T22:20:07.195 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: mon.1) 13 : audit [DBG] from='client.? 172.21.15.76:0/2756159441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:07.195 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:20:07.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: -15T22:20:06.308813+0000 mon.c (mon 2021-03-15T22:20:07.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: .1) 14 : audit [INF] from='client.? 172.21.15.76:0/462660668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:20:07.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:20:07.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: -15T22:20:06.309150+0000 mon.a ( 2021-03-15T22:20:07.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: mon.0) 603 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:20:07.197 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:20:07.197 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: -03-15T22:20:06.976836+0000 2021-03-15T22:20:07.197 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: mgr.y (mgr.14144) 364 : cluster [DBG] 2021-03-15T22:20:07.197 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:07 smithi076 conmon[62156]: pgmap v266: 148 pgs: 3 creating+peering, 3 unknown, 142 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:20:07.224 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:07.249 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:07.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:07.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:07.376 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:07.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:07.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:07.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:07.504 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:07.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:07.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:07.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: cluster 2021-03-15T22:20:07.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: 2021-03-15T22:20:06.037536+0000 mon.a (mon.0) 2021-03-15T22:20:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: 602 : cluster [DBG] osdmap e93: 8 total, 8 up, 8 in 2021-03-15T22:20:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: audit 2021-03-15T22:20:06.200787+0000 mon.c (mon.1) 13 : audit 2021-03-15T22:20:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/2756159441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:20:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: 22:20:06.308813+0000 mon.c (mon.1) 14 : audit [INF] from='client.? 172.21.15.76:0/462660668' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:20:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:20:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: 22:20:06.309150+0000 mon.a (mon.0) 603 : audit [INF] 2021-03-15T22:20:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:20:07.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:20:07.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: 20:06.976836+0000 mgr.y (mgr.14144) 2021-03-15T22:20:07.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:07 smithi125 conmon[56739]: 364 : cluster [DBG] pgmap v266: 148 pgs: 3 creating+peering, 3 unknown, 142 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:20:07.607 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:07.628 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:07.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:07.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:07.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:07.756 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:07.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:07.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:07.858 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:07.883 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:07.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:07.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:07.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:08.008 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:08.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:08.111 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:08.133 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:08.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:08.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:08.235 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:08.257 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:08.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:08.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[56514]: cluster 2021-03-15T22:20:06.751170+0000 osd.6 (osd.6) 16 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:20:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[56514]: cluster 2021-03-15T22:20:07.032349+0000 mon.a (mon.0) 604 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[56514]: audit 2021-03-15T22:20:07.044741+0000 mon.a (mon.0) 605 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:20:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[56514]: cluster 2021-03-15T22:20:07.044824+0000 mon.a (mon.0) 606 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2021-03-15T22:20:08.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: cluster 2021-03-15T22:20:06.751170 2021-03-15T22:20:08.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: +0000 osd.6 (osd.6) 16 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:20:08.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: cluster 2021-03-15T22:20:08.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: 2021-03-15T22:20:07.032349+0000 mon.a (mon.0) 604 2021-03-15T22:20:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:20:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: 03-15T22:20:07. 2021-03-15T22:20:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: 044741+0000 mon.a (mon.0) 605 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:20:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: 2021-03-15T22:20:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: cluster 2021-03-15T22:20:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: 2021-03-15T22:20: 2021-03-15T22:20:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: 07.044824+0000 mon.a (mon.0) 606 : cluster [DBG] 2021-03-15T22:20:08.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:08 smithi076 conmon[62156]: osdmap e94: 8 total, 8 up, 8 in 2021-03-15T22:20:08.360 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:08.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:08.383 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:08.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:08.485 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:08.507 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:08.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:08.508 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:08.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:08 smithi125 conmon[56739]: cluster 2021-03-15T22:20:06.751170+0000 osd.6 (osd.6) 16 : cluster [DBG] 2021-03-15T22:20:08.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:08 smithi125 conmon[56739]: 3.10 scrub ok 2021-03-15T22:20:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:08 smithi125 conmon[56739]: cluster 2021-03-15T22:20:07.032349+0000 mon.a (mon.0) 604 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:08 smithi125 conmon[56739]: audit 2021-03-15T22:20:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:08 smithi125 conmon[56739]: 2021-03-15T22:20:07.044741+0000 mon.a (mon.0) 605 : audit [INF] 2021-03-15T22:20:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:08 smithi125 conmon[56739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:20:08.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:08 smithi125 conmon[56739]: cluster 2021-03-15T22:20 2021-03-15T22:20:08.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:08 smithi125 conmon[56739]: :07.044824+0000 mon.a (mon.0) 2021-03-15T22:20:08.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:08 smithi125 conmon[56739]: 606 : cluster [DBG] osdmap e94: 8 total, 8 up, 8 in 2021-03-15T22:20:08.610 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:08.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:08.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:08.735 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:08.756 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:08.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:08.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:08.859 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:08.881 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:08.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:08.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:08.984 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:09.006 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:09.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:09.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:09.108 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:09.134 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:09.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:09.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:09.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[56514]: cluster 2021-03-15T22:20:08.400938+0000 osd.1 (osd.1 2021-03-15T22:20:09.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[56514]: ) 23 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:09.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[56514]: cluster 2021-03-15T22:20:08.401021+0000 osd.1 (osd.1) 24 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:09.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[56514]: cluster 2021-03-15T22:20:08.977418+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v268: 148 pgs: 3 creating+peering, 145 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:20:09.236 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[62156]: cluster 2021-03-15T22:20:08. 2021-03-15T22:20:09.236 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[62156]: 400938+0000 osd.1 (osd.1) 23 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:09.236 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[62156]: cluster 2021-03-15T22:20 2021-03-15T22:20:09.236 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[62156]: :08.401021+0000 osd.1 (osd.1) 24 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:09.237 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[62156]: 2021-03-15T22:20:09.237 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[62156]: cluster 2021-03-15T22:20:09.237 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[62156]: 2021-03-15T22:20:08.977418+0000 2021-03-15T22:20:09.237 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[62156]: mgr.y (mgr.14144) 365 : cluster 2021-03-15T22:20:09.238 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:09 smithi076 conmon[62156]: [DBG] pgmap v268: 148 pgs: 3 creating+peering, 145 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:20:09.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:09.264 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:09.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:09.265 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:09.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:09.390 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:09.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:09.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:09.415 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:09 smithi125 conmon[56739]: cluster 2021-03-15T22:20:08.400938+0000 osd.1 (osd. 2021-03-15T22:20:09.416 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:09 smithi125 conmon[56739]: 1) 23 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:09.416 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:09 smithi125 conmon[56739]: cluster 2021-03-15T22:20:08.401021+0000 osd.1 (osd.1) 24 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:09.416 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:09 smithi125 conmon[56739]: cluster 2021-03-15T22:20:08.977418+0000 mgr.y (mgr.14144) 365 : cluster [DBG] pgmap v268: 148 pgs: 3 creating+peering, 145 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:20:09.474 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:20:09.474 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:20:09.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:09.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:09.517 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:09.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:09.619 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:09.646 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:09.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:09.649 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:09.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:09.773 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:09.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:09.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:09.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:20:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:20:09.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:20:09.876 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:09.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:20:09.901 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:09.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:09.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:10.004 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:10.025 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:10.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:10.027 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:10.061 INFO:teuthology.orchestra.run.smithi076.stderr:enabled application 'rados' on pool 'unique_pool_0' 2021-03-15T22:20:10.128 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:10.154 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:10.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:10.156 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:10.257 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:10.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:10 smithi076 conmon[56514]: cluster 2021-03-15T22:20:09.049157+0000 mon.a (mon.0) 607 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2021-03-15T22:20:10.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:10 smithi076 conmon[56514]: audit 2021-03-15T22:20:09.411855+0000 mgr.y (mgr.14144) 366 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:10.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:10 smithi076 conmon[56514]: audit 2021-03-15T22:20:09.821771+0000 mon.a (mon.0) 608 : audit [INF] from='client.? 172.21.15.76:0/2937387817' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-03-15T22:20:10.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:10 smithi076 conmon[62156]: cluster 2021-03-15T22:20:09.049157+0000 mon.a (mon.0) 607 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2021-03-15T22:20:10.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:10 smithi076 conmon[62156]: audit 2021-03-15T22:20:09.411855+0000 mgr.y (mgr.14144) 366 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:10.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:10 smithi076 conmon[62156]: audit 2021-03-15T22:20:09.821771+0000 mon.a (mon.0) 608 : audit [INF] from='client.? 172.21.15.76:0/2937387817' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-03-15T22:20:10.283 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:10.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:10.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:10.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:10.410 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:10.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:10.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:10.513 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:10.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:10 smithi125 conmon[56739]: cluster 2021-03-15T22:20:09.049157+0000 mon.a ( 2021-03-15T22:20:10.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:10 smithi125 conmon[56739]: mon.0) 607 : cluster [DBG] osdmap e95: 8 total, 8 up, 8 in 2021-03-15T22:20:10.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:10 smithi125 conmon[56739]: audit 2021-03-15T22:20:09.411855+0000 mgr.y (mgr.14144) 366 : audit [DBG] 2021-03-15T22:20:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:10 smithi125 conmon[56739]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:10 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:20:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:10 smithi125 conmon[56739]: T22:20:09.821771+0000 mon.a (mon.0) 608 : audit [INF] 2021-03-15T22:20:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:10 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2937387817' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2021-03-15T22:20:10.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:10.539 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:10.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:10.641 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:10.665 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:10.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:10.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:10.729 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:20:10.730 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:20:10.767 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:10.791 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:10.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:10.793 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:10.821 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:20:10.858 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:10.895 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:10.904 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:10.919 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:10.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:10.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:11.022 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:11.045 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:11.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:11.046 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:11.148 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:11.170 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:11.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:11.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:11.273 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:11.298 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:11.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:11.300 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:11.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:11 smithi076 conmon[62156]: audit 2021-03-15T22:20:10. 2021-03-15T22:20:11.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:11 smithi076 conmon[62156]: 054912+0000 mon.a (mon.0) 609 : audit [INF] from='client.? 172.21.15.76:0/2937387817' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-03-15T22:20:11.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:11 smithi076 conmon[62156]: cluster 2021-03-15T22:20:10.054977+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2021-03-15T22:20:11.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:11 smithi076 conmon[62156]: cluster 2021-03-15T22:20:10.977726+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v271: 148 pgs: 148 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:20:11.311 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:11 smithi076 conmon[56514]: audit 2021-03-15T22:20 2021-03-15T22:20:11.311 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:11 smithi076 conmon[56514]: :10.054912+0000 mon.a (mon.0) 609 : audit [INF] from='client.? 172.21.15.76:0/2937387817' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-03-15T22:20:11.311 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:11 smithi076 conmon[56514]: cluster 2021-03-15T22:20:10.054977+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2021-03-15T22:20:11.312 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:11 smithi076 conmon[56514]: cluster 2021-03-15T22:20:10.977726+0000 mgr.y (mgr.14144) 367 : cluster [DBG] pgmap v271: 148 pgs: 148 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:20:11.384 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:11.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:11.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:11.432 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:11.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:11.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:11.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:11 smithi125 conmon[56739]: audit 2021-03-15T22:20:10.054912+0000 mon.a (mon.0) 609 2021-03-15T22:20:11.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:11 smithi125 conmon[56739]: : audit [INF] from='client.? 172.21.15.76:0/2937387817' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2021-03-15T22:20:11.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:11 smithi125 conmon[56739]: cluster 2021-03-15T22:20:10.054977+0000 mon.a (mon.0) 610 : cluster [DBG] osdmap e96: 8 total, 8 up, 8 in 2021-03-15T22:20:11.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:11 smithi125 conmon[56739]: 2021-03-15T22:20:11.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:11 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:20:11.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:11 smithi125 conmon[56739]: :20:10.977726+0000 mgr.y (mgr.14144) 367 : cluster [DBG] 2021-03-15T22:20:11.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:11 smithi125 conmon[56739]: pgmap v271: 148 pgs: 148 active+clean; 25 KiB data, 53 MiB used, 715 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:20:11.558 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:11.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:11.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:11.661 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:11.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:11.685 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:11.686 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:11.787 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:11.809 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:11.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:11.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:11.853 DEBUG:teuthology.orchestra.run.smithi076:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2021-03-15T22:20:11.855 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.meta 2021-03-15T22:20:11.855 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:20:11.912 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:11.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:11.935 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:11.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:12.036 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:12.062 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:12.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:12.064 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:12.085 INFO:tasks.rados.rados.0.smithi076.stdout:adding op weight read -> 100 2021-03-15T22:20:12.086 INFO:tasks.rados.rados.0.smithi076.stdout:adding op weight write -> 50 2021-03-15T22:20:12.086 INFO:tasks.rados.rados.0.smithi076.stdout:adding op weight delete -> 50 2021-03-15T22:20:12.086 INFO:tasks.rados.rados.0.smithi076.stdout:adding op weight snap_create -> 50 2021-03-15T22:20:12.086 INFO:tasks.rados.rados.0.smithi076.stdout:adding op weight snap_remove -> 50 2021-03-15T22:20:12.087 INFO:tasks.rados.rados.0.smithi076.stdout:adding op weight rollback -> 50 2021-03-15T22:20:12.087 INFO:tasks.rados.rados.0.smithi076.stdout:adding op weight copy_from -> 50 2021-03-15T22:20:12.087 INFO:tasks.rados.rados.0.smithi076.stdout:adding op weight write_excl -> 50 2021-03-15T22:20:12.087 INFO:tasks.rados.rados.0.smithi076.stdout:ceph version 17.0.0-2063-gb658018a (b658018acf46e32d2ec283d302076352e17476f5) quincy (dev) 2021-03-15T22:20:12.088 INFO:tasks.rados.rados.0.smithi076.stdout:Configuration: 2021-03-15T22:20:12.088 INFO:tasks.rados.rados.0.smithi076.stdout: Number of operations: 4000 2021-03-15T22:20:12.088 INFO:tasks.rados.rados.0.smithi076.stdout: Number of objects: 50 2021-03-15T22:20:12.088 INFO:tasks.rados.rados.0.smithi076.stdout: Max in flight operations: 16 2021-03-15T22:20:12.088 INFO:tasks.rados.rados.0.smithi076.stdout: Object size (in bytes): 4000000 2021-03-15T22:20:12.089 INFO:tasks.rados.rados.0.smithi076.stdout: Write stride min: 400000 2021-03-15T22:20:12.089 INFO:tasks.rados.rados.0.smithi076.stdout: Write stride max: 800000 2021-03-15T22:20:12.157 INFO:tasks.rados.rados.0.smithi076.stdout:1: write initial oid 1 2021-03-15T22:20:12.157 INFO:tasks.rados.rados.0.smithi076.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2021-03-15T22:20:12.162 INFO:tasks.rados.rados.0.smithi076.stdout:1: writing smithi07694878-1 from 416807 to 892056 tid 1 2021-03-15T22:20:12.165 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:12.169 INFO:tasks.rados.rados.0.smithi076.stdout:1: writing smithi07694878-1 from 1542129 to 2016807 tid 2 2021-03-15T22:20:12.170 INFO:tasks.rados.rados.0.smithi076.stdout:2: write initial oid 2 2021-03-15T22:20:12.171 INFO:tasks.rados.rados.0.smithi076.stdout:1: finishing write tid 1 to smithi07694878-1 2021-03-15T22:20:12.171 INFO:tasks.rados.rados.0.smithi076.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2021-03-15T22:20:12.175 INFO:tasks.rados.rados.0.smithi076.stdout:2: writing smithi07694878-2 from 416807 to 892056 tid 1 2021-03-15T22:20:12.182 INFO:tasks.rados.rados.0.smithi076.stdout:2: writing smithi07694878-2 from 1542129 to 2016807 tid 2 2021-03-15T22:20:12.182 INFO:tasks.rados.rados.0.smithi076.stdout:3: write initial oid 3 2021-03-15T22:20:12.182 INFO:tasks.rados.rados.0.smithi076.stdout:1: finishing write tid 2 to smithi07694878-1 2021-03-15T22:20:12.183 INFO:tasks.rados.rados.0.smithi076.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2021-03-15T22:20:12.187 INFO:tasks.rados.rados.0.smithi076.stdout:3: writing smithi07694878-3 from 433614 to 984112 tid 1 2021-03-15T22:20:12.190 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:12.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:12.194 INFO:tasks.rados.rados.0.smithi076.stdout:3: writing smithi07694878-3 from 1600611 to 2033614 tid 2 2021-03-15T22:20:12.194 INFO:tasks.rados.rados.0.smithi076.stdout:4: write initial oid 4 2021-03-15T22:20:12.194 INFO:tasks.rados.rados.0.smithi076.stdout:1: finishing write tid 3 to smithi07694878-1 2021-03-15T22:20:12.194 INFO:tasks.rados.rados.0.smithi076.stdout:1: finishing write tid 4 to smithi07694878-1 2021-03-15T22:20:12.195 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-03-15T22:20:12.195 INFO:tasks.rados.rados.0.smithi076.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2021-03-15T22:20:12.195 INFO:tasks.rados.rados.0.smithi076.stdout:2: finishing write tid 1 to smithi07694878-2 2021-03-15T22:20:12.196 INFO:tasks.rados.rados.0.smithi076.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2021-03-15T22:20:12.199 INFO:tasks.rados.rados.0.smithi076.stdout:4: writing smithi07694878-4 from 450421 to 1076168 tid 1 2021-03-15T22:20:12.205 INFO:tasks.rados.rados.0.smithi076.stdout:4: writing smithi07694878-4 from 1659093 to 2050421 tid 2 2021-03-15T22:20:12.205 INFO:tasks.rados.rados.0.smithi076.stdout:1: done (3 left) 2021-03-15T22:20:12.205 INFO:tasks.rados.rados.0.smithi076.stdout:5: write initial oid 5 2021-03-15T22:20:12.206 INFO:tasks.rados.rados.0.smithi076.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2021-03-15T22:20:12.211 INFO:tasks.rados.rados.0.smithi076.stdout:5: writing smithi07694878-5 from 467228 to 1168224 tid 1 2021-03-15T22:20:12.216 INFO:tasks.rados.rados.0.smithi076.stdout:5: writing smithi07694878-5 from 1717575 to 2067228 tid 2 2021-03-15T22:20:12.217 INFO:tasks.rados.rados.0.smithi076.stdout:6: write initial oid 6 2021-03-15T22:20:12.217 INFO:tasks.rados.rados.0.smithi076.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2021-03-15T22:20:12.224 INFO:tasks.rados.rados.0.smithi076.stdout:6: writing smithi07694878-6 from 484035 to 1260280 tid 1 2021-03-15T22:20:12.230 INFO:tasks.rados.rados.0.smithi076.stdout:6: writing smithi07694878-6 from 2059704 to 2084035 tid 2 2021-03-15T22:20:12.230 INFO:tasks.rados.rados.0.smithi076.stdout:2: finishing write tid 2 to smithi07694878-2 2021-03-15T22:20:12.230 INFO:tasks.rados.rados.0.smithi076.stdout:2: finishing write tid 3 to smithi07694878-2 2021-03-15T22:20:12.231 INFO:tasks.rados.rados.0.smithi076.stdout:2: finishing write tid 4 to smithi07694878-2 2021-03-15T22:20:12.231 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2021-03-15T22:20:12.231 INFO:tasks.rados.rados.0.smithi076.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2021-03-15T22:20:12.231 INFO:tasks.rados.rados.0.smithi076.stdout:3: finishing write tid 1 to smithi07694878-3 2021-03-15T22:20:12.232 INFO:tasks.rados.rados.0.smithi076.stdout:3: finishing write tid 2 to smithi07694878-3 2021-03-15T22:20:12.232 INFO:tasks.rados.rados.0.smithi076.stdout:3: finishing write tid 3 to smithi07694878-3 2021-03-15T22:20:12.232 INFO:tasks.rados.rados.0.smithi076.stdout:3: finishing write tid 4 to smithi07694878-3 2021-03-15T22:20:12.232 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2021-03-15T22:20:12.233 INFO:tasks.rados.rados.0.smithi076.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2021-03-15T22:20:12.233 INFO:tasks.rados.rados.0.smithi076.stdout:4: finishing write tid 1 to smithi07694878-4 2021-03-15T22:20:12.233 INFO:tasks.rados.rados.0.smithi076.stdout:4: finishing write tid 2 to smithi07694878-4 2021-03-15T22:20:12.233 INFO:tasks.rados.rados.0.smithi076.stdout:4: finishing write tid 3 to smithi07694878-4 2021-03-15T22:20:12.233 INFO:tasks.rados.rados.0.smithi076.stdout:4: finishing write tid 4 to smithi07694878-4 2021-03-15T22:20:12.234 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2021-03-15T22:20:12.234 INFO:tasks.rados.rados.0.smithi076.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2021-03-15T22:20:12.234 INFO:tasks.rados.rados.0.smithi076.stdout:5: finishing write tid 1 to smithi07694878-5 2021-03-15T22:20:12.235 INFO:tasks.rados.rados.0.smithi076.stdout:5: finishing write tid 2 to smithi07694878-5 2021-03-15T22:20:12.235 INFO:tasks.rados.rados.0.smithi076.stdout:5: finishing write tid 3 to smithi07694878-5 2021-03-15T22:20:12.235 INFO:tasks.rados.rados.0.smithi076.stdout:5: finishing write tid 4 to smithi07694878-5 2021-03-15T22:20:12.236 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2021-03-15T22:20:12.236 INFO:tasks.rados.rados.0.smithi076.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2021-03-15T22:20:12.236 INFO:tasks.rados.rados.0.smithi076.stdout:6: finishing write tid 1 to smithi07694878-6 2021-03-15T22:20:12.236 INFO:tasks.rados.rados.0.smithi076.stdout:2: done (4 left) 2021-03-15T22:20:12.237 INFO:tasks.rados.rados.0.smithi076.stdout:3: done (3 left) 2021-03-15T22:20:12.237 INFO:tasks.rados.rados.0.smithi076.stdout:4: done (2 left) 2021-03-15T22:20:12.237 INFO:tasks.rados.rados.0.smithi076.stdout:5: done (1 left) 2021-03-15T22:20:12.237 INFO:tasks.rados.rados.0.smithi076.stdout:7: write initial oid 7 2021-03-15T22:20:12.237 INFO:tasks.rados.rados.0.smithi076.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2021-03-15T22:20:12.238 INFO:tasks.rados.rados.0.smithi076.stdout:7: writing smithi07694878-7 from 500842 to 952336 tid 1 2021-03-15T22:20:12.241 INFO:tasks.rados.rados.0.smithi076.stdout:7: writing smithi07694878-7 from 1718186 to 2100842 tid 2 2021-03-15T22:20:12.241 INFO:tasks.rados.rados.0.smithi076.stdout:6: finishing write tid 2 to smithi07694878-6 2021-03-15T22:20:12.241 INFO:tasks.rados.rados.0.smithi076.stdout:6: finishing write tid 3 to smithi07694878-6 2021-03-15T22:20:12.241 INFO:tasks.rados.rados.0.smithi076.stdout:6: finishing write tid 4 to smithi07694878-6 2021-03-15T22:20:12.242 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2021-03-15T22:20:12.242 INFO:tasks.rados.rados.0.smithi076.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2021-03-15T22:20:12.242 INFO:tasks.rados.rados.0.smithi076.stdout:7: finishing write tid 1 to smithi07694878-7 2021-03-15T22:20:12.242 INFO:tasks.rados.rados.0.smithi076.stdout:6: done (1 left) 2021-03-15T22:20:12.243 INFO:tasks.rados.rados.0.smithi076.stdout:8: write initial oid 8 2021-03-15T22:20:12.243 INFO:tasks.rados.rados.0.smithi076.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2021-03-15T22:20:12.247 INFO:tasks.rados.rados.0.smithi076.stdout:8: writing smithi07694878-8 from 517649 to 1044392 tid 1 2021-03-15T22:20:12.254 INFO:tasks.rados.rados.0.smithi076.stdout:8: writing smithi07694878-8 from 1776668 to 2117649 tid 2 2021-03-15T22:20:12.254 INFO:tasks.rados.rados.0.smithi076.stdout:7: finishing write tid 2 to smithi07694878-7 2021-03-15T22:20:12.254 INFO:tasks.rados.rados.0.smithi076.stdout:7: finishing write tid 3 to smithi07694878-7 2021-03-15T22:20:12.254 INFO:tasks.rados.rados.0.smithi076.stdout:7: finishing write tid 4 to smithi07694878-7 2021-03-15T22:20:12.255 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:20:12.255 INFO:tasks.rados.rados.0.smithi076.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2021-03-15T22:20:12.255 INFO:tasks.rados.rados.0.smithi076.stdout:8: finishing write tid 1 to smithi07694878-8 2021-03-15T22:20:12.256 INFO:tasks.rados.rados.0.smithi076.stdout:7: done (1 left) 2021-03-15T22:20:12.256 INFO:tasks.rados.rados.0.smithi076.stdout:9: write initial oid 9 2021-03-15T22:20:12.256 INFO:tasks.rados.rados.0.smithi076.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2021-03-15T22:20:12.260 INFO:tasks.rados.rados.0.smithi076.stdout:9: writing smithi07694878-9 from 534456 to 1252801 tid 1 2021-03-15T22:20:12.266 INFO:tasks.rados.rados.0.smithi076.stdout:9: writing smithi07694878-9 from 1951503 to 2134456 tid 2 2021-03-15T22:20:12.267 INFO:tasks.rados.rados.0.smithi076.stdout:10: write initial oid 10 2021-03-15T22:20:12.267 INFO:tasks.rados.rados.0.smithi076.stdout:8: finishing write tid 2 to smithi07694878-8 2021-03-15T22:20:12.267 INFO:tasks.rados.rados.0.smithi076.stdout:8: finishing write tid 3 to smithi07694878-8 2021-03-15T22:20:12.267 INFO:tasks.rados.rados.0.smithi076.stdout:8: finishing write tid 4 to smithi07694878-8 2021-03-15T22:20:12.268 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 3 (ObjNum 7 snap 0 seq_num 7) dirty exists 2021-03-15T22:20:12.268 INFO:tasks.rados.rados.0.smithi076.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2021-03-15T22:20:12.268 INFO:tasks.rados.rados.0.smithi076.stdout:9: finishing write tid 1 to smithi07694878-9 2021-03-15T22:20:12.268 INFO:tasks.rados.rados.0.smithi076.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2021-03-15T22:20:12.273 INFO:tasks.rados.rados.0.smithi076.stdout:10: writing smithi07694878-10 from 551263 to 1344857 tid 1 2021-03-15T22:20:12.278 INFO:tasks.rados.rados.0.smithi076.stdout:10: writing smithi07694878-10 from 1893632 to 2151263 tid 2 2021-03-15T22:20:12.278 INFO:tasks.rados.rados.0.smithi076.stdout:8: done (2 left) 2021-03-15T22:20:12.279 INFO:tasks.rados.rados.0.smithi076.stdout:11: write initial oid 11 2021-03-15T22:20:12.279 INFO:tasks.rados.rados.0.smithi076.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2021-03-15T22:20:12.283 INFO:tasks.rados.rados.0.smithi076.stdout:11: writing smithi07694878-11 from 568070 to 1036913 tid 1 2021-03-15T22:20:12.291 INFO:tasks.rados.rados.0.smithi076.stdout:11: writing smithi07694878-11 from 1552114 to 2168070 tid 2 2021-03-15T22:20:12.291 INFO:tasks.rados.rados.0.smithi076.stdout:12: write initial oid 12 2021-03-15T22:20:12.291 INFO:tasks.rados.rados.0.smithi076.stdout:9: finishing write tid 2 to smithi07694878-9 2021-03-15T22:20:12.291 INFO:tasks.rados.rados.0.smithi076.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2021-03-15T22:20:12.293 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:12.297 INFO:tasks.rados.rados.0.smithi076.stdout:12: writing smithi07694878-12 from 584877 to 1128969 tid 1 2021-03-15T22:20:12.303 INFO:tasks.rados.rados.0.smithi076.stdout:12: writing smithi07694878-12 from 1610596 to 2172599 tid 2 2021-03-15T22:20:12.303 INFO:tasks.rados.rados.0.smithi076.stdout:13: write initial oid 13 2021-03-15T22:20:12.303 INFO:tasks.rados.rados.0.smithi076.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2021-03-15T22:20:12.309 INFO:tasks.rados.rados.0.smithi076.stdout:13: writing smithi07694878-13 from 601684 to 1221025 tid 1 2021-03-15T22:20:12.315 INFO:tasks.rados.rados.0.smithi076.stdout:13: writing smithi07694878-13 from 1669078 to 2201684 tid 2 2021-03-15T22:20:12.315 INFO:tasks.rados.rados.0.smithi076.stdout:14: write initial oid 14 2021-03-15T22:20:12.316 INFO:tasks.rados.rados.0.smithi076.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2021-03-15T22:20:12.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:12.318 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:12.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:12.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:12 smithi076 conmon[56514]: audit 2021-03-15T22:20:12.097433+0000 mon.c (mon.1) 15 : audit [INF] from='client.? 172.21.15.76:0/2794982104' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-03-15T22:20:12.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:12 smithi076 conmon[56514]: audit 2021-03-15T22:20:12.097811+0000 mon.a (mon.0) 611 : audit [INF] from='client.? ' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-03-15T22:20:12.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:12 smithi076 conmon[62156]: audit 2021-03-15T22:20:12.097433+0000 mon.c (mon.1) 15 : audit [INF] from='client.? 172.21.15.76:0/2794982104' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-03-15T22:20:12.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:12 smithi076 conmon[62156]: audit 2021-03-15T22:20:12.097811+0000 mon.a (mon.0) 611 : audit [INF] from='client.? ' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-03-15T22:20:12.323 INFO:tasks.rados.rados.0.smithi076.stdout:14: writing smithi07694878-14 from 618491 to 1313081 tid 1 2021-03-15T22:20:12.329 INFO:tasks.rados.rados.0.smithi076.stdout:14: writing smithi07694878-14 from 2011207 to 2218491 tid 2 2021-03-15T22:20:12.329 INFO:tasks.rados.rados.0.smithi076.stdout:15: write initial oid 15 2021-03-15T22:20:12.329 INFO:tasks.rados.rados.0.smithi076.stdout:9: finishing write tid 3 to smithi07694878-9 2021-03-15T22:20:12.329 INFO:tasks.rados.rados.0.smithi076.stdout:9: finishing write tid 4 to smithi07694878-9 2021-03-15T22:20:12.330 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 3 (ObjNum 8 snap 0 seq_num 8) dirty exists 2021-03-15T22:20:12.330 INFO:tasks.rados.rados.0.smithi076.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2021-03-15T22:20:12.330 INFO:tasks.rados.rados.0.smithi076.stdout:10: finishing write tid 1 to smithi07694878-10 2021-03-15T22:20:12.330 INFO:tasks.rados.rados.0.smithi076.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2021-03-15T22:20:12.336 INFO:tasks.rados.rados.0.smithi076.stdout:15: writing smithi07694878-15 from 635298 to 1405137 tid 1 2021-03-15T22:20:12.341 INFO:tasks.rados.rados.0.smithi076.stdout:15: writing smithi07694878-15 from 2069689 to 2235298 tid 2 2021-03-15T22:20:12.341 INFO:tasks.rados.rados.0.smithi076.stdout:9: done (6 left) 2021-03-15T22:20:12.341 INFO:tasks.rados.rados.0.smithi076.stdout:16: write initial oid 16 2021-03-15T22:20:12.341 INFO:tasks.rados.rados.0.smithi076.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2021-03-15T22:20:12.347 INFO:tasks.rados.rados.0.smithi076.stdout:16: writing smithi07694878-16 from 652105 to 1097193 tid 1 2021-03-15T22:20:12.353 INFO:tasks.rados.rados.0.smithi076.stdout:16: writing smithi07694878-16 from 1728171 to 2181159 tid 2 2021-03-15T22:20:12.353 INFO:tasks.rados.rados.0.smithi076.stdout:17: write initial oid 17 2021-03-15T22:20:12.353 INFO:tasks.rados.rados.0.smithi076.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2021-03-15T22:20:12.360 INFO:tasks.rados.rados.0.smithi076.stdout:17: writing smithi07694878-17 from 668912 to 1305602 tid 1 2021-03-15T22:20:12.365 INFO:tasks.rados.rados.0.smithi076.stdout:17: writing smithi07694878-17 from 1903006 to 2268912 tid 2 2021-03-15T22:20:12.366 INFO:tasks.rados.rados.0.smithi076.stdout:18: write initial oid 18 2021-03-15T22:20:12.366 INFO:tasks.rados.rados.0.smithi076.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2021-03-15T22:20:12.373 INFO:tasks.rados.rados.0.smithi076.stdout:18: writing smithi07694878-18 from 685719 to 1397658 tid 1 2021-03-15T22:20:12.378 INFO:tasks.rados.rados.0.smithi076.stdout:18: writing smithi07694878-18 from 1845135 to 2285719 tid 2 2021-03-15T22:20:12.379 INFO:tasks.rados.rados.0.smithi076.stdout:19: write initial oid 19 2021-03-15T22:20:12.379 INFO:tasks.rados.rados.0.smithi076.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2021-03-15T22:20:12.387 INFO:tasks.rados.rados.0.smithi076.stdout:19: writing smithi07694878-19 from 702526 to 1489714 tid 1 2021-03-15T22:20:12.392 INFO:tasks.rados.rados.0.smithi076.stdout:19: writing smithi07694878-19 from 1903617 to 2302526 tid 2 2021-03-15T22:20:12.392 INFO:tasks.rados.rados.0.smithi076.stdout:20: write initial oid 20 2021-03-15T22:20:12.392 INFO:tasks.rados.rados.0.smithi076.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2021-03-15T22:20:12.398 INFO:tasks.rados.rados.0.smithi076.stdout:20: writing smithi07694878-20 from 719333 to 1181770 tid 1 2021-03-15T22:20:12.404 INFO:tasks.rados.rados.0.smithi076.stdout:20: writing smithi07694878-20 from 1962099 to 2319333 tid 2 2021-03-15T22:20:12.405 INFO:tasks.rados.rados.0.smithi076.stdout:21: write initial oid 21 2021-03-15T22:20:12.405 INFO:tasks.rados.rados.0.smithi076.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2021-03-15T22:20:12.411 INFO:tasks.rados.rados.0.smithi076.stdout:21: writing smithi07694878-21 from 736140 to 1273826 tid 1 2021-03-15T22:20:12.417 INFO:tasks.rados.rados.0.smithi076.stdout:21: writing smithi07694878-21 from 2020581 to 2336140 tid 2 2021-03-15T22:20:12.418 INFO:tasks.rados.rados.0.smithi076.stdout:22: write initial oid 22 2021-03-15T22:20:12.418 INFO:tasks.rados.rados.0.smithi076.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2021-03-15T22:20:12.420 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:12.425 INFO:tasks.rados.rados.0.smithi076.stdout:22: writing smithi07694878-22 from 752947 to 1365882 tid 1 2021-03-15T22:20:12.431 INFO:tasks.rados.rados.0.smithi076.stdout:22: writing smithi07694878-22 from 1962710 to 2352947 tid 2 2021-03-15T22:20:12.431 INFO:tasks.rados.rados.0.smithi076.stdout:23: write initial oid 23 2021-03-15T22:20:12.431 INFO:tasks.rados.rados.0.smithi076.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2021-03-15T22:20:12.439 INFO:tasks.rados.rados.0.smithi076.stdout:23: writing smithi07694878-23 from 769754 to 1457938 tid 1 2021-03-15T22:20:12.441 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:12.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:12.443 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:12.444 INFO:tasks.rados.rados.0.smithi076.stdout:23: writing smithi07694878-23 from 2021192 to 2369754 tid 2 2021-03-15T22:20:12.444 INFO:tasks.rados.rados.0.smithi076.stdout:24: write initial oid 24 2021-03-15T22:20:12.445 INFO:tasks.rados.rados.0.smithi076.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2021-03-15T22:20:12.451 INFO:tasks.rados.rados.0.smithi076.stdout:24: writing smithi07694878-24 from 786561 to 1266347 tid 1 2021-03-15T22:20:12.457 INFO:tasks.rados.rados.0.smithi076.stdout:24: writing smithi07694878-24 from 1796027 to 2263691 tid 2 2021-03-15T22:20:12.457 INFO:tasks.rados.rados.0.smithi076.stdout:25: write initial oid 25 2021-03-15T22:20:12.457 INFO:tasks.rados.rados.0.smithi076.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2021-03-15T22:20:12.462 INFO:tasks.rados.rados.0.smithi076.stdout:25: writing smithi07694878-25 from 403368 to 958403 tid 1 2021-03-15T22:20:12.470 INFO:tasks.rados.rados.0.smithi076.stdout:25: writing smithi07694878-25 from 1454509 to 2182184 tid 2 2021-03-15T22:20:12.470 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.470 INFO:tasks.rados.rados.0.smithi076.stdout:10: finishing write tid 2 to smithi07694878-10 2021-03-15T22:20:12.470 INFO:tasks.rados.rados.0.smithi076.stdout:10: finishing write tid 3 to smithi07694878-10 2021-03-15T22:20:12.471 INFO:tasks.rados.rados.0.smithi076.stdout:10: finishing write tid 4 to smithi07694878-10 2021-03-15T22:20:12.471 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 6 (ObjNum 9 snap 0 seq_num 9) dirty exists 2021-03-15T22:20:12.471 INFO:tasks.rados.rados.0.smithi076.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2021-03-15T22:20:12.471 INFO:tasks.rados.rados.0.smithi076.stdout:11: finishing write tid 1 to smithi07694878-11 2021-03-15T22:20:12.472 INFO:tasks.rados.rados.0.smithi076.stdout:10: done (15 left) 2021-03-15T22:20:12.472 INFO:tasks.rados.rados.0.smithi076.stdout:26: write initial oid 26 2021-03-15T22:20:12.472 INFO:tasks.rados.rados.0.smithi076.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2021-03-15T22:20:12.475 INFO:tasks.rados.rados.0.smithi076.stdout:26: writing smithi07694878-26 from 420175 to 1050459 tid 1 2021-03-15T22:20:12.483 INFO:tasks.rados.rados.0.smithi076.stdout:26: writing smithi07694878-26 from 1796638 to 2267971 tid 2 2021-03-15T22:20:12.483 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.484 INFO:tasks.rados.rados.0.smithi076.stdout:11: finishing write tid 2 to smithi07694878-11 2021-03-15T22:20:12.484 INFO:tasks.rados.rados.0.smithi076.stdout:11: finishing write tid 3 to smithi07694878-11 2021-03-15T22:20:12.484 INFO:tasks.rados.rados.0.smithi076.stdout:11: finishing write tid 4 to smithi07694878-11 2021-03-15T22:20:12.484 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 6 (ObjNum 10 snap 0 seq_num 10) dirty exists 2021-03-15T22:20:12.485 INFO:tasks.rados.rados.0.smithi076.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2021-03-15T22:20:12.485 INFO:tasks.rados.rados.0.smithi076.stdout:12: finishing write tid 1 to smithi07694878-12 2021-03-15T22:20:12.485 INFO:tasks.rados.rados.0.smithi076.stdout:12: finishing write tid 2 to smithi07694878-12 2021-03-15T22:20:12.485 INFO:tasks.rados.rados.0.smithi076.stdout:12: finishing write tid 3 to smithi07694878-12 2021-03-15T22:20:12.486 INFO:tasks.rados.rados.0.smithi076.stdout:12: finishing write tid 4 to smithi07694878-12 2021-03-15T22:20:12.486 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 6 (ObjNum 11 snap 0 seq_num 11) dirty exists 2021-03-15T22:20:12.486 INFO:tasks.rados.rados.0.smithi076.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2021-03-15T22:20:12.486 INFO:tasks.rados.rados.0.smithi076.stdout:13: finishing write tid 1 to smithi07694878-13 2021-03-15T22:20:12.487 INFO:tasks.rados.rados.0.smithi076.stdout:11: done (15 left) 2021-03-15T22:20:12.487 INFO:tasks.rados.rados.0.smithi076.stdout:12: done (14 left) 2021-03-15T22:20:12.487 INFO:tasks.rados.rados.0.smithi076.stdout:27: write initial oid 27 2021-03-15T22:20:12.487 INFO:tasks.rados.rados.0.smithi076.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2021-03-15T22:20:12.489 INFO:tasks.rados.rados.0.smithi076.stdout:27: writing smithi07694878-27 from 436982 to 1142515 tid 1 2021-03-15T22:20:12.497 INFO:tasks.rados.rados.0.smithi076.stdout:27: writing smithi07694878-27 from 1855120 to 2436982 tid 2 2021-03-15T22:20:12.497 INFO:tasks.rados.rados.0.smithi076.stdout:28: write initial oid 28 2021-03-15T22:20:12.498 INFO:tasks.rados.rados.0.smithi076.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2021-03-15T22:20:12.504 INFO:tasks.rados.rados.0.smithi076.stdout:28: writing smithi07694878-28 from 453789 to 1234571 tid 1 2021-03-15T22:20:12.512 INFO:tasks.rados.rados.0.smithi076.stdout:28: writing smithi07694878-28 from 1913602 to 2388604 tid 2 2021-03-15T22:20:12.512 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.513 INFO:tasks.rados.rados.0.smithi076.stdout:13: finishing write tid 2 to smithi07694878-13 2021-03-15T22:20:12.513 INFO:tasks.rados.rados.0.smithi076.stdout:13: finishing write tid 3 to smithi07694878-13 2021-03-15T22:20:12.513 INFO:tasks.rados.rados.0.smithi076.stdout:13: finishing write tid 4 to smithi07694878-13 2021-03-15T22:20:12.513 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 9 (ObjNum 12 snap 0 seq_num 12) dirty exists 2021-03-15T22:20:12.514 INFO:tasks.rados.rados.0.smithi076.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2021-03-15T22:20:12.514 INFO:tasks.rados.rados.0.smithi076.stdout:14: finishing write tid 1 to smithi07694878-14 2021-03-15T22:20:12.514 INFO:tasks.rados.rados.0.smithi076.stdout:14: finishing write tid 2 to smithi07694878-14 2021-03-15T22:20:12.514 INFO:tasks.rados.rados.0.smithi076.stdout:14: finishing write tid 3 to smithi07694878-14 2021-03-15T22:20:12.515 INFO:tasks.rados.rados.0.smithi076.stdout:14: finishing write tid 4 to smithi07694878-14 2021-03-15T22:20:12.515 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2021-03-15T22:20:12.515 INFO:tasks.rados.rados.0.smithi076.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2021-03-15T22:20:12.515 INFO:tasks.rados.rados.0.smithi076.stdout:15: finishing write tid 1 to smithi07694878-15 2021-03-15T22:20:12.515 INFO:tasks.rados.rados.0.smithi076.stdout:15: finishing write tid 2 to smithi07694878-15 2021-03-15T22:20:12.516 INFO:tasks.rados.rados.0.smithi076.stdout:15: finishing write tid 3 to smithi07694878-15 2021-03-15T22:20:12.516 INFO:tasks.rados.rados.0.smithi076.stdout:13: done (15 left) 2021-03-15T22:20:12.516 INFO:tasks.rados.rados.0.smithi076.stdout:14: done (14 left) 2021-03-15T22:20:12.516 INFO:tasks.rados.rados.0.smithi076.stdout:29: write initial oid 29 2021-03-15T22:20:12.517 INFO:tasks.rados.rados.0.smithi076.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2021-03-15T22:20:12.517 INFO:tasks.rados.rados.0.smithi076.stdout:29: writing smithi07694878-29 from 470596 to 926627 tid 1 2021-03-15T22:20:12.523 INFO:tasks.rados.rados.0.smithi076.stdout:29: writing smithi07694878-29 from 1572084 to 2190744 tid 2 2021-03-15T22:20:12.523 INFO:tasks.rados.rados.0.smithi076.stdout:30: write initial oid 30 2021-03-15T22:20:12.523 INFO:tasks.rados.rados.0.smithi076.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2021-03-15T22:20:12.528 INFO:tasks.rados.rados.0.smithi076.stdout:30: writing smithi07694878-30 from 487403 to 1018683 tid 1 2021-03-15T22:20:12.534 INFO:tasks.rados.rados.0.smithi076.stdout:30: writing smithi07694878-30 from 1514213 to 1992884 tid 2 2021-03-15T22:20:12.535 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.535 INFO:tasks.rados.rados.0.smithi076.stdout:15: finishing write tid 4 to smithi07694878-15 2021-03-15T22:20:12.535 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 9 (ObjNum 14 snap 0 seq_num 14) dirty exists 2021-03-15T22:20:12.535 INFO:tasks.rados.rados.0.smithi076.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2021-03-15T22:20:12.536 INFO:tasks.rados.rados.0.smithi076.stdout:15: done (15 left) 2021-03-15T22:20:12.536 INFO:tasks.rados.rados.0.smithi076.stdout:31: write initial oid 31 2021-03-15T22:20:12.536 INFO:tasks.rados.rados.0.smithi076.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2021-03-15T22:20:12.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:12 smithi125 conmon[56739]: audit 2021-03-15T22:20:12.097433+0000 mon.c (mon.1) 15 : audit [INF] 2021-03-15T22:20:12.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:12 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2794982104' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-03-15T22:20:12.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:12 smithi125 conmon[56739]: audit 2021-03-15T22:20:12.097811+0000 mon.a (mon.0) 611 : audit [INF] from='client.? ' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2021-03-15T22:20:12.540 INFO:tasks.rados.rados.0.smithi076.stdout:31: writing smithi07694878-31 from 504210 to 1110739 tid 1 2021-03-15T22:20:12.544 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:12.547 INFO:tasks.rados.rados.0.smithi076.stdout:31: writing smithi07694878-31 from 1572695 to 2195024 tid 2 2021-03-15T22:20:12.548 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.548 INFO:tasks.rados.rados.0.smithi076.stdout:16: finishing write tid 1 to smithi07694878-16 2021-03-15T22:20:12.548 INFO:tasks.rados.rados.0.smithi076.stdout:16: finishing write tid 2 to smithi07694878-16 2021-03-15T22:20:12.548 INFO:tasks.rados.rados.0.smithi076.stdout:16: finishing write tid 3 to smithi07694878-16 2021-03-15T22:20:12.549 INFO:tasks.rados.rados.0.smithi076.stdout:16: finishing write tid 4 to smithi07694878-16 2021-03-15T22:20:12.549 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 12 (ObjNum 15 snap 0 seq_num 15) dirty exists 2021-03-15T22:20:12.549 INFO:tasks.rados.rados.0.smithi076.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2021-03-15T22:20:12.549 INFO:tasks.rados.rados.0.smithi076.stdout:17: finishing write tid 1 to smithi07694878-17 2021-03-15T22:20:12.549 INFO:tasks.rados.rados.0.smithi076.stdout:16: done (15 left) 2021-03-15T22:20:12.550 INFO:tasks.rados.rados.0.smithi076.stdout:32: write initial oid 32 2021-03-15T22:20:12.550 INFO:tasks.rados.rados.0.smithi076.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2021-03-15T22:20:12.554 INFO:tasks.rados.rados.0.smithi076.stdout:32: writing smithi07694878-32 from 521017 to 1319148 tid 1 2021-03-15T22:20:12.559 INFO:tasks.rados.rados.0.smithi076.stdout:32: writing smithi07694878-32 from 1747530 to 2229870 tid 2 2021-03-15T22:20:12.560 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.560 INFO:tasks.rados.rados.0.smithi076.stdout:17: finishing write tid 2 to smithi07694878-17 2021-03-15T22:20:12.560 INFO:tasks.rados.rados.0.smithi076.stdout:17: finishing write tid 3 to smithi07694878-17 2021-03-15T22:20:12.560 INFO:tasks.rados.rados.0.smithi076.stdout:17: finishing write tid 4 to smithi07694878-17 2021-03-15T22:20:12.561 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2021-03-15T22:20:12.561 INFO:tasks.rados.rados.0.smithi076.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2021-03-15T22:20:12.561 INFO:tasks.rados.rados.0.smithi076.stdout:18: finishing write tid 1 to smithi07694878-18 2021-03-15T22:20:12.561 INFO:tasks.rados.rados.0.smithi076.stdout:17: done (15 left) 2021-03-15T22:20:12.561 INFO:tasks.rados.rados.0.smithi076.stdout:33: write initial oid 33 2021-03-15T22:20:12.562 INFO:tasks.rados.rados.0.smithi076.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2021-03-15T22:20:12.565 INFO:tasks.rados.rados.0.smithi076.stdout:33: writing smithi07694878-33 from 537824 to 1011204 tid 1 2021-03-15T22:20:12.569 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:12.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:12.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:12.573 INFO:tasks.rados.rados.0.smithi076.stdout:33: writing smithi07694878-33 from 1806012 to 2432010 tid 2 2021-03-15T22:20:12.573 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.574 INFO:tasks.rados.rados.0.smithi076.stdout:18: finishing write tid 2 to smithi07694878-18 2021-03-15T22:20:12.574 INFO:tasks.rados.rados.0.smithi076.stdout:18: finishing write tid 3 to smithi07694878-18 2021-03-15T22:20:12.574 INFO:tasks.rados.rados.0.smithi076.stdout:18: finishing write tid 4 to smithi07694878-18 2021-03-15T22:20:12.574 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2021-03-15T22:20:12.574 INFO:tasks.rados.rados.0.smithi076.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2021-03-15T22:20:12.575 INFO:tasks.rados.rados.0.smithi076.stdout:19: finishing write tid 1 to smithi07694878-19 2021-03-15T22:20:12.575 INFO:tasks.rados.rados.0.smithi076.stdout:18: done (15 left) 2021-03-15T22:20:12.575 INFO:tasks.rados.rados.0.smithi076.stdout:34: write initial oid 34 2021-03-15T22:20:12.575 INFO:tasks.rados.rados.0.smithi076.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2021-03-15T22:20:12.579 INFO:tasks.rados.rados.0.smithi076.stdout:34: writing smithi07694878-34 from 554631 to 1103260 tid 1 2021-03-15T22:20:12.586 INFO:tasks.rados.rados.0.smithi076.stdout:34: writing smithi07694878-34 from 1748141 to 2234150 tid 2 2021-03-15T22:20:12.586 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.586 INFO:tasks.rados.rados.0.smithi076.stdout:19: finishing write tid 2 to smithi07694878-19 2021-03-15T22:20:12.586 INFO:tasks.rados.rados.0.smithi076.stdout:19: finishing write tid 3 to smithi07694878-19 2021-03-15T22:20:12.587 INFO:tasks.rados.rados.0.smithi076.stdout:19: finishing write tid 4 to smithi07694878-19 2021-03-15T22:20:12.587 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 6 (ObjNum 18 snap 0 seq_num 18) dirty exists 2021-03-15T22:20:12.587 INFO:tasks.rados.rados.0.smithi076.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2021-03-15T22:20:12.587 INFO:tasks.rados.rados.0.smithi076.stdout:20: finishing write tid 1 to smithi07694878-20 2021-03-15T22:20:12.587 INFO:tasks.rados.rados.0.smithi076.stdout:20: finishing write tid 2 to smithi07694878-20 2021-03-15T22:20:12.588 INFO:tasks.rados.rados.0.smithi076.stdout:20: finishing write tid 3 to smithi07694878-20 2021-03-15T22:20:12.588 INFO:tasks.rados.rados.0.smithi076.stdout:19: done (15 left) 2021-03-15T22:20:12.588 INFO:tasks.rados.rados.0.smithi076.stdout:35: write initial oid 35 2021-03-15T22:20:12.588 INFO:tasks.rados.rados.0.smithi076.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2021-03-15T22:20:12.592 INFO:tasks.rados.rados.0.smithi076.stdout:35: writing smithi07694878-35 from 571438 to 1195316 tid 1 2021-03-15T22:20:12.599 INFO:tasks.rados.rados.0.smithi076.stdout:35: writing smithi07694878-35 from 1806623 to 2436290 tid 2 2021-03-15T22:20:12.599 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.599 INFO:tasks.rados.rados.0.smithi076.stdout:20: finishing write tid 4 to smithi07694878-20 2021-03-15T22:20:12.600 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2021-03-15T22:20:12.600 INFO:tasks.rados.rados.0.smithi076.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2021-03-15T22:20:12.600 INFO:tasks.rados.rados.0.smithi076.stdout:21: finishing write tid 1 to smithi07694878-21 2021-03-15T22:20:12.600 INFO:tasks.rados.rados.0.smithi076.stdout:20: done (15 left) 2021-03-15T22:20:12.601 INFO:tasks.rados.rados.0.smithi076.stdout:36: write initial oid 36 2021-03-15T22:20:12.601 INFO:tasks.rados.rados.0.smithi076.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2021-03-15T22:20:12.606 INFO:tasks.rados.rados.0.smithi076.stdout:36: writing smithi07694878-36 from 588245 to 1287372 tid 1 2021-03-15T22:20:12.612 INFO:tasks.rados.rados.0.smithi076.stdout:36: writing smithi07694878-36 from 1865105 to 2354783 tid 2 2021-03-15T22:20:12.612 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.613 INFO:tasks.rados.rados.0.smithi076.stdout:21: finishing write tid 2 to smithi07694878-21 2021-03-15T22:20:12.613 INFO:tasks.rados.rados.0.smithi076.stdout:21: finishing write tid 3 to smithi07694878-21 2021-03-15T22:20:12.613 INFO:tasks.rados.rados.0.smithi076.stdout:21: finishing write tid 4 to smithi07694878-21 2021-03-15T22:20:12.613 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 3 (ObjNum 20 snap 0 seq_num 20) dirty exists 2021-03-15T22:20:12.614 INFO:tasks.rados.rados.0.smithi076.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2021-03-15T22:20:12.614 INFO:tasks.rados.rados.0.smithi076.stdout:22: finishing write tid 1 to smithi07694878-22 2021-03-15T22:20:12.614 INFO:tasks.rados.rados.0.smithi076.stdout:22: finishing write tid 2 to smithi07694878-22 2021-03-15T22:20:12.615 INFO:tasks.rados.rados.0.smithi076.stdout:22: finishing write tid 3 to smithi07694878-22 2021-03-15T22:20:12.615 INFO:tasks.rados.rados.0.smithi076.stdout:21: done (15 left) 2021-03-15T22:20:12.616 INFO:tasks.rados.rados.0.smithi076.stdout:37: write initial oid 37 2021-03-15T22:20:12.616 INFO:tasks.rados.rados.0.smithi076.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2021-03-15T22:20:12.619 INFO:tasks.rados.rados.0.smithi076.stdout:37: writing smithi07694878-37 from 605052 to 1379428 tid 1 2021-03-15T22:20:12.626 INFO:tasks.rados.rados.0.smithi076.stdout:37: writing smithi07694878-37 from 1923587 to 2556923 tid 2 2021-03-15T22:20:12.627 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.627 INFO:tasks.rados.rados.0.smithi076.stdout:22: finishing write tid 4 to smithi07694878-22 2021-03-15T22:20:12.627 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 15 (ObjNum 21 snap 0 seq_num 21) dirty exists 2021-03-15T22:20:12.627 INFO:tasks.rados.rados.0.smithi076.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2021-03-15T22:20:12.628 INFO:tasks.rados.rados.0.smithi076.stdout:23: finishing write tid 1 to smithi07694878-23 2021-03-15T22:20:12.628 INFO:tasks.rados.rados.0.smithi076.stdout:23: finishing write tid 2 to smithi07694878-23 2021-03-15T22:20:12.628 INFO:tasks.rados.rados.0.smithi076.stdout:22: done (15 left) 2021-03-15T22:20:12.628 INFO:tasks.rados.rados.0.smithi076.stdout:38: write initial oid 38 2021-03-15T22:20:12.628 INFO:tasks.rados.rados.0.smithi076.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2021-03-15T22:20:12.632 INFO:tasks.rados.rados.0.smithi076.stdout:38: writing smithi07694878-38 from 621859 to 1071484 tid 1 2021-03-15T22:20:12.641 INFO:tasks.rados.rados.0.smithi076.stdout:38: writing smithi07694878-38 from 1865716 to 2621859 tid 2 2021-03-15T22:20:12.641 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.641 INFO:tasks.rados.rados.0.smithi076.stdout:23: finishing write tid 3 to smithi07694878-23 2021-03-15T22:20:12.641 INFO:tasks.rados.rados.0.smithi076.stdout:23: finishing write tid 4 to smithi07694878-23 2021-03-15T22:20:12.642 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2021-03-15T22:20:12.642 INFO:tasks.rados.rados.0.smithi076.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2021-03-15T22:20:12.642 INFO:tasks.rados.rados.0.smithi076.stdout:24: finishing write tid 1 to smithi07694878-24 2021-03-15T22:20:12.642 INFO:tasks.rados.rados.0.smithi076.stdout:24: finishing write tid 2 to smithi07694878-24 2021-03-15T22:20:12.643 INFO:tasks.rados.rados.0.smithi076.stdout:23: done (15 left) 2021-03-15T22:20:12.643 INFO:tasks.rados.rados.0.smithi076.stdout:39: write initial oid 39 2021-03-15T22:20:12.643 INFO:tasks.rados.rados.0.smithi076.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2021-03-15T22:20:12.646 INFO:tasks.rados.rados.0.smithi076.stdout:39: writing smithi07694878-39 from 638666 to 1163540 tid 1 2021-03-15T22:20:12.655 INFO:tasks.rados.rados.0.smithi076.stdout:39: writing smithi07694878-39 from 1924198 to 2561203 tid 2 2021-03-15T22:20:12.656 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.656 INFO:tasks.rados.rados.0.smithi076.stdout:24: finishing write tid 3 to smithi07694878-24 2021-03-15T22:20:12.657 INFO:tasks.rados.rados.0.smithi076.stdout:24: finishing write tid 4 to smithi07694878-24 2021-03-15T22:20:12.657 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 3 (ObjNum 23 snap 0 seq_num 23) dirty exists 2021-03-15T22:20:12.657 INFO:tasks.rados.rados.0.smithi076.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2021-03-15T22:20:12.658 INFO:tasks.rados.rados.0.smithi076.stdout:25: finishing write tid 1 to smithi07694878-25 2021-03-15T22:20:12.658 INFO:tasks.rados.rados.0.smithi076.stdout:24: done (15 left) 2021-03-15T22:20:12.659 INFO:tasks.rados.rados.0.smithi076.stdout:40: write initial oid 40 2021-03-15T22:20:12.659 INFO:tasks.rados.rados.0.smithi076.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2021-03-15T22:20:12.661 INFO:tasks.rados.rados.0.smithi076.stdout:40: writing smithi07694878-40 from 655473 to 1371949 tid 1 2021-03-15T22:20:12.669 INFO:tasks.rados.rados.0.smithi076.stdout:40: writing smithi07694878-40 from 2099033 to 2655473 tid 2 2021-03-15T22:20:12.670 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.670 INFO:tasks.rados.rados.0.smithi076.stdout:25: finishing write tid 2 to smithi07694878-25 2021-03-15T22:20:12.671 INFO:tasks.rados.rados.0.smithi076.stdout:25: finishing write tid 3 to smithi07694878-25 2021-03-15T22:20:12.671 INFO:tasks.rados.rados.0.smithi076.stdout:25: finishing write tid 4 to smithi07694878-25 2021-03-15T22:20:12.671 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 9 (ObjNum 24 snap 0 seq_num 24) dirty exists 2021-03-15T22:20:12.672 INFO:tasks.rados.rados.0.smithi076.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2021-03-15T22:20:12.672 INFO:tasks.rados.rados.0.smithi076.stdout:26: finishing write tid 1 to smithi07694878-26 2021-03-15T22:20:12.673 INFO:tasks.rados.rados.0.smithi076.stdout:26: finishing write tid 2 to smithi07694878-26 2021-03-15T22:20:12.673 INFO:tasks.rados.rados.0.smithi076.stdout:26: finishing write tid 3 to smithi07694878-26 2021-03-15T22:20:12.674 INFO:tasks.rados.rados.0.smithi076.stdout:26: finishing write tid 4 to smithi07694878-26 2021-03-15T22:20:12.674 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 9 (ObjNum 25 snap 0 seq_num 25) dirty exists 2021-03-15T22:20:12.675 INFO:tasks.rados.rados.0.smithi076.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2021-03-15T22:20:12.675 INFO:tasks.rados.rados.0.smithi076.stdout:27: finishing write tid 1 to smithi07694878-27 2021-03-15T22:20:12.675 INFO:tasks.rados.rados.0.smithi076.stdout:27: finishing write tid 2 to smithi07694878-27 2021-03-15T22:20:12.676 INFO:tasks.rados.rados.0.smithi076.stdout:27: finishing write tid 3 to smithi07694878-27 2021-03-15T22:20:12.676 INFO:tasks.rados.rados.0.smithi076.stdout:27: finishing write tid 4 to smithi07694878-27 2021-03-15T22:20:12.677 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 3 (ObjNum 26 snap 0 seq_num 26) dirty exists 2021-03-15T22:20:12.677 INFO:tasks.rados.rados.0.smithi076.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2021-03-15T22:20:12.678 INFO:tasks.rados.rados.0.smithi076.stdout:28: finishing write tid 1 to smithi07694878-28 2021-03-15T22:20:12.678 INFO:tasks.rados.rados.0.smithi076.stdout:25: done (15 left) 2021-03-15T22:20:12.679 INFO:tasks.rados.rados.0.smithi076.stdout:26: done (14 left) 2021-03-15T22:20:12.679 INFO:tasks.rados.rados.0.smithi076.stdout:27: done (13 left) 2021-03-15T22:20:12.680 INFO:tasks.rados.rados.0.smithi076.stdout:41: write initial oid 41 2021-03-15T22:20:12.680 INFO:tasks.rados.rados.0.smithi076.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2021-03-15T22:20:12.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:12.682 INFO:tasks.rados.rados.0.smithi076.stdout:41: writing smithi07694878-41 from 672280 to 1464005 tid 1 2021-03-15T22:20:12.684 INFO:tasks.rados.rados.0.smithi076.stdout:41: writing smithi07694878-41 from 2157515 to 2672280 tid 2 2021-03-15T22:20:12.684 INFO:tasks.rados.rados.0.smithi076.stdout:42: write initial oid 42 2021-03-15T22:20:12.685 INFO:tasks.rados.rados.0.smithi076.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2021-03-15T22:20:12.690 INFO:tasks.rados.rados.0.smithi076.stdout:42: writing smithi07694878-42 from 689087 to 1156061 tid 1 2021-03-15T22:20:12.697 INFO:tasks.rados.rados.0.smithi076.stdout:42: writing smithi07694878-42 from 1699644 to 2483976 tid 2 2021-03-15T22:20:12.697 INFO:tasks.rados.rados.0.smithi076.stdout:43: write initial oid 43 2021-03-15T22:20:12.698 INFO:tasks.rados.rados.0.smithi076.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2021-03-15T22:20:12.704 INFO:tasks.rados.rados.0.smithi076.stdout:43: writing smithi07694878-43 from 705894 to 1248117 tid 1 2021-03-15T22:20:12.705 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:12.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:12.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:12.712 INFO:tasks.rados.rados.0.smithi076.stdout:43: writing smithi07694878-43 from 1758126 to 2402469 tid 2 2021-03-15T22:20:12.713 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 16 2021-03-15T22:20:12.713 INFO:tasks.rados.rados.0.smithi076.stdout:28: finishing write tid 2 to smithi07694878-28 2021-03-15T22:20:12.713 INFO:tasks.rados.rados.0.smithi076.stdout:28: finishing write tid 3 to smithi07694878-28 2021-03-15T22:20:12.713 INFO:tasks.rados.rados.0.smithi076.stdout:28: finishing write tid 4 to smithi07694878-28 2021-03-15T22:20:12.714 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2021-03-15T22:20:12.714 INFO:tasks.rados.rados.0.smithi076.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2021-03-15T22:20:12.714 INFO:tasks.rados.rados.0.smithi076.stdout:29: finishing write tid 1 to smithi07694878-29 2021-03-15T22:20:12.714 INFO:tasks.rados.rados.0.smithi076.stdout:29: finishing write tid 2 to smithi07694878-29 2021-03-15T22:20:12.715 INFO:tasks.rados.rados.0.smithi076.stdout:29: finishing write tid 3 to smithi07694878-29 2021-03-15T22:20:12.715 INFO:tasks.rados.rados.0.smithi076.stdout:29: finishing write tid 4 to smithi07694878-29 2021-03-15T22:20:12.715 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2021-03-15T22:20:12.715 INFO:tasks.rados.rados.0.smithi076.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2021-03-15T22:20:12.716 INFO:tasks.rados.rados.0.smithi076.stdout:30: finishing write tid 1 to smithi07694878-30 2021-03-15T22:20:12.716 INFO:tasks.rados.rados.0.smithi076.stdout:30: finishing write tid 2 to smithi07694878-30 2021-03-15T22:20:12.716 INFO:tasks.rados.rados.0.smithi076.stdout:30: finishing write tid 3 to smithi07694878-30 2021-03-15T22:20:12.716 INFO:tasks.rados.rados.0.smithi076.stdout:30: finishing write tid 4 to smithi07694878-30 2021-03-15T22:20:12.716 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 18 (ObjNum 29 snap 0 seq_num 29) dirty exists 2021-03-15T22:20:12.717 INFO:tasks.rados.rados.0.smithi076.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2021-03-15T22:20:12.717 INFO:tasks.rados.rados.0.smithi076.stdout:31: finishing write tid 1 to smithi07694878-31 2021-03-15T22:20:12.717 INFO:tasks.rados.rados.0.smithi076.stdout:31: finishing write tid 2 to smithi07694878-31 2021-03-15T22:20:12.717 INFO:tasks.rados.rados.0.smithi076.stdout:31: finishing write tid 3 to smithi07694878-31 2021-03-15T22:20:12.717 INFO:tasks.rados.rados.0.smithi076.stdout:31: finishing write tid 4 to smithi07694878-31 2021-03-15T22:20:12.718 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2021-03-15T22:20:12.718 INFO:tasks.rados.rados.0.smithi076.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2021-03-15T22:20:12.718 INFO:tasks.rados.rados.0.smithi076.stdout:32: finishing write tid 1 to smithi07694878-32 2021-03-15T22:20:12.718 INFO:tasks.rados.rados.0.smithi076.stdout:32: finishing write tid 2 to smithi07694878-32 2021-03-15T22:20:12.719 INFO:tasks.rados.rados.0.smithi076.stdout:32: finishing write tid 3 to smithi07694878-32 2021-03-15T22:20:12.719 INFO:tasks.rados.rados.0.smithi076.stdout:32: finishing write tid 4 to smithi07694878-32 2021-03-15T22:20:12.719 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 12 (ObjNum 31 snap 0 seq_num 31) dirty exists 2021-03-15T22:20:12.719 INFO:tasks.rados.rados.0.smithi076.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2021-03-15T22:20:12.719 INFO:tasks.rados.rados.0.smithi076.stdout:33: finishing write tid 1 to smithi07694878-33 2021-03-15T22:20:12.720 INFO:tasks.rados.rados.0.smithi076.stdout:33: finishing write tid 2 to smithi07694878-33 2021-03-15T22:20:12.720 INFO:tasks.rados.rados.0.smithi076.stdout:28: done (15 left) 2021-03-15T22:20:12.720 INFO:tasks.rados.rados.0.smithi076.stdout:29: done (14 left) 2021-03-15T22:20:12.720 INFO:tasks.rados.rados.0.smithi076.stdout:30: done (13 left) 2021-03-15T22:20:12.720 INFO:tasks.rados.rados.0.smithi076.stdout:31: done (12 left) 2021-03-15T22:20:12.721 INFO:tasks.rados.rados.0.smithi076.stdout:32: done (11 left) 2021-03-15T22:20:12.721 INFO:tasks.rados.rados.0.smithi076.stdout:44: write initial oid 44 2021-03-15T22:20:12.721 INFO:tasks.rados.rados.0.smithi076.stdout:33: finishing write tid 3 to smithi07694878-33 2021-03-15T22:20:12.721 INFO:tasks.rados.rados.0.smithi076.stdout:33: finishing write tid 4 to smithi07694878-33 2021-03-15T22:20:12.722 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 6 (ObjNum 32 snap 0 seq_num 32) dirty exists 2021-03-15T22:20:12.722 INFO:tasks.rados.rados.0.smithi076.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2021-03-15T22:20:12.722 INFO:tasks.rados.rados.0.smithi076.stdout:34: finishing write tid 1 to smithi07694878-34 2021-03-15T22:20:12.722 INFO:tasks.rados.rados.0.smithi076.stdout:34: finishing write tid 2 to smithi07694878-34 2021-03-15T22:20:12.722 INFO:tasks.rados.rados.0.smithi076.stdout:34: finishing write tid 3 to smithi07694878-34 2021-03-15T22:20:12.723 INFO:tasks.rados.rados.0.smithi076.stdout:34: finishing write tid 4 to smithi07694878-34 2021-03-15T22:20:12.723 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 9 (ObjNum 33 snap 0 seq_num 33) dirty exists 2021-03-15T22:20:12.723 INFO:tasks.rados.rados.0.smithi076.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2021-03-15T22:20:12.723 INFO:tasks.rados.rados.0.smithi076.stdout:35: finishing write tid 1 to smithi07694878-35 2021-03-15T22:20:12.723 INFO:tasks.rados.rados.0.smithi076.stdout:35: finishing write tid 2 to smithi07694878-35 2021-03-15T22:20:12.724 INFO:tasks.rados.rados.0.smithi076.stdout:35: finishing write tid 3 to smithi07694878-35 2021-03-15T22:20:12.724 INFO:tasks.rados.rados.0.smithi076.stdout:35: finishing write tid 4 to smithi07694878-35 2021-03-15T22:20:12.724 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2021-03-15T22:20:12.724 INFO:tasks.rados.rados.0.smithi076.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2021-03-15T22:20:12.725 INFO:tasks.rados.rados.0.smithi076.stdout:36: finishing write tid 1 to smithi07694878-36 2021-03-15T22:20:12.725 INFO:tasks.rados.rados.0.smithi076.stdout:36: finishing write tid 2 to smithi07694878-36 2021-03-15T22:20:12.725 INFO:tasks.rados.rados.0.smithi076.stdout:36: finishing write tid 3 to smithi07694878-36 2021-03-15T22:20:12.725 INFO:tasks.rados.rados.0.smithi076.stdout:37: finishing write tid 1 to smithi07694878-37 2021-03-15T22:20:12.725 INFO:tasks.rados.rados.0.smithi076.stdout:37: finishing write tid 2 to smithi07694878-37 2021-03-15T22:20:12.726 INFO:tasks.rados.rados.0.smithi076.stdout:37: finishing write tid 3 to smithi07694878-37 2021-03-15T22:20:12.726 INFO:tasks.rados.rados.0.smithi076.stdout:37: finishing write tid 4 to smithi07694878-37 2021-03-15T22:20:12.726 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 15 (ObjNum 36 snap 0 seq_num 36) dirty exists 2021-03-15T22:20:12.726 INFO:tasks.rados.rados.0.smithi076.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2021-03-15T22:20:12.726 INFO:tasks.rados.rados.0.smithi076.stdout:38: finishing write tid 1 to smithi07694878-38 2021-03-15T22:20:12.727 INFO:tasks.rados.rados.0.smithi076.stdout:38: finishing write tid 2 to smithi07694878-38 2021-03-15T22:20:12.727 INFO:tasks.rados.rados.0.smithi076.stdout:38: finishing write tid 3 to smithi07694878-38 2021-03-15T22:20:12.727 INFO:tasks.rados.rados.0.smithi076.stdout:38: finishing write tid 4 to smithi07694878-38 2021-03-15T22:20:12.727 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 12 (ObjNum 37 snap 0 seq_num 37) dirty exists 2021-03-15T22:20:12.727 INFO:tasks.rados.rados.0.smithi076.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2021-03-15T22:20:12.728 INFO:tasks.rados.rados.0.smithi076.stdout:39: finishing write tid 1 to smithi07694878-39 2021-03-15T22:20:12.728 INFO:tasks.rados.rados.0.smithi076.stdout:36: finishing write tid 4 to smithi07694878-36 2021-03-15T22:20:12.728 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 12 (ObjNum 35 snap 0 seq_num 35) dirty exists 2021-03-15T22:20:12.728 INFO:tasks.rados.rados.0.smithi076.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2021-03-15T22:20:12.729 INFO:tasks.rados.rados.0.smithi076.stdout:40: finishing write tid 1 to smithi07694878-40 2021-03-15T22:20:12.729 INFO:tasks.rados.rados.0.smithi076.stdout:39: finishing write tid 2 to smithi07694878-39 2021-03-15T22:20:12.729 INFO:tasks.rados.rados.0.smithi076.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2021-03-15T22:20:12.730 INFO:tasks.rados.rados.0.smithi076.stdout:44: writing smithi07694878-44 from 722701 to 1340173 tid 1 2021-03-15T22:20:12.731 INFO:tasks.rados.rados.0.smithi076.stdout:44: writing smithi07694878-44 from 1816608 to 2604609 tid 2 2021-03-15T22:20:12.731 INFO:tasks.rados.rados.0.smithi076.stdout:33: done (11 left) 2021-03-15T22:20:12.731 INFO:tasks.rados.rados.0.smithi076.stdout:34: done (10 left) 2021-03-15T22:20:12.731 INFO:tasks.rados.rados.0.smithi076.stdout:35: done (9 left) 2021-03-15T22:20:12.731 INFO:tasks.rados.rados.0.smithi076.stdout:36: done (8 left) 2021-03-15T22:20:12.732 INFO:tasks.rados.rados.0.smithi076.stdout:37: done (7 left) 2021-03-15T22:20:12.732 INFO:tasks.rados.rados.0.smithi076.stdout:38: done (6 left) 2021-03-15T22:20:12.732 INFO:tasks.rados.rados.0.smithi076.stdout:45: write initial oid 45 2021-03-15T22:20:12.732 INFO:tasks.rados.rados.0.smithi076.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2021-03-15T22:20:12.733 INFO:tasks.rados.rados.0.smithi076.stdout:45: writing smithi07694878-45 from 739508 to 1432229 tid 1 2021-03-15T22:20:12.739 INFO:tasks.rados.rados.0.smithi076.stdout:45: writing smithi07694878-45 from 1875090 to 2523102 tid 2 2021-03-15T22:20:12.740 INFO:tasks.rados.rados.0.smithi076.stdout:46: write initial oid 46 2021-03-15T22:20:12.740 INFO:tasks.rados.rados.0.smithi076.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2021-03-15T22:20:12.747 INFO:tasks.rados.rados.0.smithi076.stdout:46: writing smithi07694878-46 from 756315 to 1524285 tid 1 2021-03-15T22:20:12.755 INFO:tasks.rados.rados.0.smithi076.stdout:46: writing smithi07694878-46 from 1933572 to 2725242 tid 2 2021-03-15T22:20:12.755 INFO:tasks.rados.rados.0.smithi076.stdout:47: write initial oid 47 2021-03-15T22:20:12.755 INFO:tasks.rados.rados.0.smithi076.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2021-03-15T22:20:12.762 INFO:tasks.rados.rados.0.smithi076.stdout:47: writing smithi07694878-47 from 773122 to 1332694 tid 1 2021-03-15T22:20:12.770 INFO:tasks.rados.rados.0.smithi076.stdout:47: writing smithi07694878-47 from 1992054 to 2643735 tid 2 2021-03-15T22:20:12.771 INFO:tasks.rados.rados.0.smithi076.stdout:48: write initial oid 48 2021-03-15T22:20:12.771 INFO:tasks.rados.rados.0.smithi076.stdout:39: finishing write tid 3 to smithi07694878-39 2021-03-15T22:20:12.771 INFO:tasks.rados.rados.0.smithi076.stdout:39: finishing write tid 4 to smithi07694878-39 2021-03-15T22:20:12.771 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 15 (ObjNum 38 snap 0 seq_num 38) dirty exists 2021-03-15T22:20:12.772 INFO:tasks.rados.rados.0.smithi076.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2021-03-15T22:20:12.772 INFO:tasks.rados.rados.0.smithi076.stdout:40: finishing write tid 2 to smithi07694878-40 2021-03-15T22:20:12.772 INFO:tasks.rados.rados.0.smithi076.stdout:40: finishing write tid 3 to smithi07694878-40 2021-03-15T22:20:12.773 INFO:tasks.rados.rados.0.smithi076.stdout:40: finishing write tid 4 to smithi07694878-40 2021-03-15T22:20:12.773 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 18 (ObjNum 39 snap 0 seq_num 39) dirty exists 2021-03-15T22:20:12.773 INFO:tasks.rados.rados.0.smithi076.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2021-03-15T22:20:12.773 INFO:tasks.rados.rados.0.smithi076.stdout:41: finishing write tid 1 to smithi07694878-41 2021-03-15T22:20:12.773 INFO:tasks.rados.rados.0.smithi076.stdout:41: finishing write tid 2 to smithi07694878-41 2021-03-15T22:20:12.774 INFO:tasks.rados.rados.0.smithi076.stdout:41: finishing write tid 3 to smithi07694878-41 2021-03-15T22:20:12.774 INFO:tasks.rados.rados.0.smithi076.stdout:41: finishing write tid 4 to smithi07694878-41 2021-03-15T22:20:12.774 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2021-03-15T22:20:12.774 INFO:tasks.rados.rados.0.smithi076.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2021-03-15T22:20:12.775 INFO:tasks.rados.rados.0.smithi076.stdout:42: finishing write tid 1 to smithi07694878-42 2021-03-15T22:20:12.775 INFO:tasks.rados.rados.0.smithi076.stdout:42: finishing write tid 2 to smithi07694878-42 2021-03-15T22:20:12.775 INFO:tasks.rados.rados.0.smithi076.stdout:42: finishing write tid 3 to smithi07694878-42 2021-03-15T22:20:12.775 INFO:tasks.rados.rados.0.smithi076.stdout:42: finishing write tid 4 to smithi07694878-42 2021-03-15T22:20:12.775 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 21 (ObjNum 41 snap 0 seq_num 41) dirty exists 2021-03-15T22:20:12.776 INFO:tasks.rados.rados.0.smithi076.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2021-03-15T22:20:12.776 INFO:tasks.rados.rados.0.smithi076.stdout:43: finishing write tid 1 to smithi07694878-43 2021-03-15T22:20:12.776 INFO:tasks.rados.rados.0.smithi076.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2021-03-15T22:20:12.777 INFO:tasks.rados.rados.0.smithi076.stdout:48: writing smithi07694878-48 from 789929 to 1424750 tid 1 2021-03-15T22:20:12.785 INFO:tasks.rados.rados.0.smithi076.stdout:48: writing smithi07694878-48 from 2050536 to 2789929 tid 2 2021-03-15T22:20:12.785 INFO:tasks.rados.rados.0.smithi076.stdout:39: done (9 left) 2021-03-15T22:20:12.785 INFO:tasks.rados.rados.0.smithi076.stdout:40: done (8 left) 2021-03-15T22:20:12.786 INFO:tasks.rados.rados.0.smithi076.stdout:41: done (7 left) 2021-03-15T22:20:12.786 INFO:tasks.rados.rados.0.smithi076.stdout:42: done (6 left) 2021-03-15T22:20:12.786 INFO:tasks.rados.rados.0.smithi076.stdout:49: write initial oid 49 2021-03-15T22:20:12.786 INFO:tasks.rados.rados.0.smithi076.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2021-03-15T22:20:12.791 INFO:tasks.rados.rados.0.smithi076.stdout:49: writing smithi07694878-49 from 406736 to 1116806 tid 1 2021-03-15T22:20:12.798 INFO:tasks.rados.rados.0.smithi076.stdout:49: writing smithi07694878-49 from 1709018 to 2364368 tid 2 2021-03-15T22:20:12.798 INFO:tasks.rados.rados.0.smithi076.stdout:50: write initial oid 50 2021-03-15T22:20:12.798 INFO:tasks.rados.rados.0.smithi076.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2021-03-15T22:20:12.804 INFO:tasks.rados.rados.0.smithi076.stdout:50: writing smithi07694878-50 from 423543 to 1208862 tid 1 2021-03-15T22:20:12.807 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:12.813 INFO:tasks.rados.rados.0.smithi076.stdout:50: writing smithi07694878-50 from 1767500 to 2566508 tid 2 2021-03-15T22:20:12.813 INFO:tasks.rados.rados.0.smithi076.stdout:51: copy_from oid 13 from oid 18 current snap is 0 2021-03-15T22:20:12.813 INFO:tasks.rados.rados.0.smithi076.stdout:43: finishing write tid 2 to smithi07694878-43 2021-03-15T22:20:12.813 INFO:tasks.rados.rados.0.smithi076.stdout:43: finishing write tid 3 to smithi07694878-43 2021-03-15T22:20:12.814 INFO:tasks.rados.rados.0.smithi076.stdout:43: finishing write tid 4 to smithi07694878-43 2021-03-15T22:20:12.814 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2021-03-15T22:20:12.814 INFO:tasks.rados.rados.0.smithi076.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2021-03-15T22:20:12.814 INFO:tasks.rados.rados.0.smithi076.stdout:44: finishing write tid 1 to smithi07694878-44 2021-03-15T22:20:12.815 INFO:tasks.rados.rados.0.smithi076.stdout:44: finishing write tid 2 to smithi07694878-44 2021-03-15T22:20:12.815 INFO:tasks.rados.rados.0.smithi076.stdout:44: finishing write tid 3 to smithi07694878-44 2021-03-15T22:20:12.815 INFO:tasks.rados.rados.0.smithi076.stdout:44: finishing write tid 4 to smithi07694878-44 2021-03-15T22:20:12.815 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 6 (ObjNum 43 snap 0 seq_num 43) dirty exists 2021-03-15T22:20:12.815 INFO:tasks.rados.rados.0.smithi076.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2021-03-15T22:20:12.816 INFO:tasks.rados.rados.0.smithi076.stdout:45: finishing write tid 1 to smithi07694878-45 2021-03-15T22:20:12.816 INFO:tasks.rados.rados.0.smithi076.stdout:43: done (8 left) 2021-03-15T22:20:12.816 INFO:tasks.rados.rados.0.smithi076.stdout:44: done (7 left) 2021-03-15T22:20:12.816 INFO:tasks.rados.rados.0.smithi076.stdout:52: write oid 38 current snap is 0 2021-03-15T22:20:12.817 INFO:tasks.rados.rados.0.smithi076.stdout:45: finishing write tid 2 to smithi07694878-45 2021-03-15T22:20:12.817 INFO:tasks.rados.rados.0.smithi076.stdout:45: finishing write tid 3 to smithi07694878-45 2021-03-15T22:20:12.817 INFO:tasks.rados.rados.0.smithi076.stdout:45: finishing write tid 4 to smithi07694878-45 2021-03-15T22:20:12.817 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 6 (ObjNum 44 snap 0 seq_num 44) dirty exists 2021-03-15T22:20:12.817 INFO:tasks.rados.rados.0.smithi076.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2021-03-15T22:20:12.818 INFO:tasks.rados.rados.0.smithi076.stdout:46: finishing write tid 1 to smithi07694878-46 2021-03-15T22:20:12.818 INFO:tasks.rados.rados.0.smithi076.stdout:52: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2021-03-15T22:20:12.818 INFO:tasks.rados.rados.0.smithi076.stdout:52: writing smithi07694878-38 from 440350 to 900918 tid 1 2021-03-15T22:20:12.823 INFO:tasks.rados.rados.0.smithi076.stdout:52: writing smithi07694878-38 from 1309629 to 1852295 tid 2 2021-03-15T22:20:12.829 INFO:tasks.rados.rados.0.smithi076.stdout:52: writing smithi07694878-38 from 2323973 to 2840350 tid 3 2021-03-15T22:20:12.829 INFO:tasks.rados.rados.0.smithi076.stdout:45: done (7 left) 2021-03-15T22:20:12.829 INFO:tasks.rados.rados.0.smithi076.stdout:53: rollback oid 20 current snap is 0 2021-03-15T22:20:12.829 INFO:tasks.rados.rados.0.smithi076.stdout:53: done (7 left) 2021-03-15T22:20:12.830 INFO:tasks.rados.rados.0.smithi076.stdout:54: write (excl) oid 25 current snap is 0 2021-03-15T22:20:12.830 INFO:tasks.rados.rados.0.smithi076.stdout:46: finishing write tid 2 to smithi07694878-46 2021-03-15T22:20:12.830 INFO:tasks.rados.rados.0.smithi076.stdout:46: finishing write tid 3 to smithi07694878-46 2021-03-15T22:20:12.830 INFO:tasks.rados.rados.0.smithi076.stdout:46: finishing write tid 4 to smithi07694878-46 2021-03-15T22:20:12.831 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 3 (ObjNum 45 snap 0 seq_num 45) dirty exists 2021-03-15T22:20:12.831 INFO:tasks.rados.rados.0.smithi076.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2021-03-15T22:20:12.831 INFO:tasks.rados.rados.0.smithi076.stdout:47: finishing write tid 1 to smithi07694878-47 2021-03-15T22:20:12.831 INFO:tasks.rados.rados.0.smithi076.stdout:54: seq_num 51 ranges {457157=535817,1768111=402677} 2021-03-15T22:20:12.833 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:12.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:12.834 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:12.835 INFO:tasks.rados.rados.0.smithi076.stdout:54: writing smithi07694878-25 from 457157 to 992974 tid 1 2021-03-15T22:20:12.841 INFO:tasks.rados.rados.0.smithi076.stdout:54: writing smithi07694878-25 from 1768111 to 2170788 tid 2 2021-03-15T22:20:12.842 INFO:tasks.rados.rados.0.smithi076.stdout:46: done (7 left) 2021-03-15T22:20:12.842 INFO:tasks.rados.rados.0.smithi076.stdout:55: snap_create 2021-03-15T22:20:12.842 INFO:tasks.rados.rados.0.smithi076.stdout:47: finishing write tid 2 to smithi07694878-47 2021-03-15T22:20:12.842 INFO:tasks.rados.rados.0.smithi076.stdout:47: finishing write tid 3 to smithi07694878-47 2021-03-15T22:20:12.843 INFO:tasks.rados.rados.0.smithi076.stdout:47: finishing write tid 4 to smithi07694878-47 2021-03-15T22:20:12.843 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 9 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-03-15T22:20:12.843 INFO:tasks.rados.rados.0.smithi076.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2021-03-15T22:20:12.843 INFO:tasks.rados.rados.0.smithi076.stdout:48: finishing write tid 1 to smithi07694878-48 2021-03-15T22:20:12.844 INFO:tasks.rados.rados.0.smithi076.stdout:48: finishing write tid 2 to smithi07694878-48 2021-03-15T22:20:12.844 INFO:tasks.rados.rados.0.smithi076.stdout:48: finishing write tid 3 to smithi07694878-48 2021-03-15T22:20:12.844 INFO:tasks.rados.rados.0.smithi076.stdout:48: finishing write tid 4 to smithi07694878-48 2021-03-15T22:20:12.844 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 9 (ObjNum 47 snap 0 seq_num 47) dirty exists 2021-03-15T22:20:12.844 INFO:tasks.rados.rados.0.smithi076.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2021-03-15T22:20:12.845 INFO:tasks.rados.rados.0.smithi076.stdout:49: finishing write tid 1 to smithi07694878-49 2021-03-15T22:20:12.845 INFO:tasks.rados.rados.0.smithi076.stdout:49: finishing write tid 2 to smithi07694878-49 2021-03-15T22:20:12.845 INFO:tasks.rados.rados.0.smithi076.stdout:49: finishing write tid 3 to smithi07694878-49 2021-03-15T22:20:12.845 INFO:tasks.rados.rados.0.smithi076.stdout:49: finishing write tid 4 to smithi07694878-49 2021-03-15T22:20:12.845 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 21 (ObjNum 48 snap 0 seq_num 48) dirty exists 2021-03-15T22:20:12.846 INFO:tasks.rados.rados.0.smithi076.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2021-03-15T22:20:12.846 INFO:tasks.rados.rados.0.smithi076.stdout:50: finishing write tid 1 to smithi07694878-50 2021-03-15T22:20:12.846 INFO:tasks.rados.rados.0.smithi076.stdout:50: finishing write tid 2 to smithi07694878-50 2021-03-15T22:20:12.846 INFO:tasks.rados.rados.0.smithi076.stdout:50: finishing write tid 3 to smithi07694878-50 2021-03-15T22:20:12.847 INFO:tasks.rados.rados.0.smithi076.stdout:50: finishing write tid 4 to smithi07694878-50 2021-03-15T22:20:12.847 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 12 (ObjNum 49 snap 0 seq_num 49) dirty exists 2021-03-15T22:20:12.847 INFO:tasks.rados.rados.0.smithi076.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2021-03-15T22:20:12.847 INFO:tasks.rados.rados.0.smithi076.stdout:51: finishing copy_from to smithi07694878-13 2021-03-15T22:20:12.847 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2021-03-15T22:20:12.848 INFO:tasks.rados.rados.0.smithi076.stdout:51: finishing copy_from racing read to smithi07694878-13 2021-03-15T22:20:12.848 INFO:tasks.rados.rados.0.smithi076.stdout:52: finishing write tid 1 to smithi07694878-38 2021-03-15T22:20:12.848 INFO:tasks.rados.rados.0.smithi076.stdout:52: finishing write tid 2 to smithi07694878-38 2021-03-15T22:20:12.848 INFO:tasks.rados.rados.0.smithi076.stdout:52: finishing write tid 3 to smithi07694878-38 2021-03-15T22:20:12.849 INFO:tasks.rados.rados.0.smithi076.stdout:52: finishing write tid 4 to smithi07694878-38 2021-03-15T22:20:12.849 INFO:tasks.rados.rados.0.smithi076.stdout:52: finishing write tid 5 to smithi07694878-38 2021-03-15T22:20:12.849 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 19 (ObjNum 50 snap 0 seq_num 50) dirty exists 2021-03-15T22:20:12.849 INFO:tasks.rados.rados.0.smithi076.stdout:52: left oid 38 (ObjNum 50 snap 0 seq_num 50) 2021-03-15T22:20:12.849 INFO:tasks.rados.rados.0.smithi076.stdout:54: finishing write tid 1 to smithi07694878-25 2021-03-15T22:20:12.850 INFO:tasks.rados.rados.0.smithi076.stdout:54: finishing write tid 2 to smithi07694878-25 2021-03-15T22:20:12.851 INFO:tasks.rados.rados.0.smithi076.stdout:54: finishing write tid 3 to smithi07694878-25 2021-03-15T22:20:12.851 INFO:tasks.rados.rados.0.smithi076.stdout:54: finishing write tid 4 to smithi07694878-25 2021-03-15T22:20:12.851 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 12 (ObjNum 51 snap 0 seq_num 51) dirty exists 2021-03-15T22:20:12.851 INFO:tasks.rados.rados.0.smithi076.stdout:54: left oid 25 (ObjNum 51 snap 0 seq_num 51) 2021-03-15T22:20:12.935 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:12.958 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:12.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:12.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:13.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:13.082 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:13.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:13.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:13.170 INFO:tasks.rados.rados.0.smithi076.stdout:47: done (7 left) 2021-03-15T22:20:13.171 INFO:tasks.rados.rados.0.smithi076.stdout:48: done (6 left) 2021-03-15T22:20:13.171 INFO:tasks.rados.rados.0.smithi076.stdout:49: done (5 left) 2021-03-15T22:20:13.171 INFO:tasks.rados.rados.0.smithi076.stdout:50: done (4 left) 2021-03-15T22:20:13.172 INFO:tasks.rados.rados.0.smithi076.stdout:51: done (3 left) 2021-03-15T22:20:13.172 INFO:tasks.rados.rados.0.smithi076.stdout:52: done (2 left) 2021-03-15T22:20:13.172 INFO:tasks.rados.rados.0.smithi076.stdout:54: done (1 left) 2021-03-15T22:20:13.172 INFO:tasks.rados.rados.0.smithi076.stdout:55: done (0 left) 2021-03-15T22:20:13.172 INFO:tasks.rados.rados.0.smithi076.stdout:56: delete oid 34 current snap is 1 2021-03-15T22:20:13.173 INFO:tasks.rados.rados.0.smithi076.stdout:56: done (0 left) 2021-03-15T22:20:13.173 INFO:tasks.rados.rados.0.smithi076.stdout:57: snap_create 2021-03-15T22:20:13.185 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:13.207 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:13.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:13.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:13.309 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:13.332 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:13.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:13.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:13.349 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:13.435 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:13.456 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:13.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:13.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:13.459 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[56514]: audit 2021-03-15T22: 2021-03-15T22:20:13.459 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[56514]: 20:12.154616+0000 mon.a (mon.0) 612 : audit [INF] from='client.? ' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-03-15T22:20:13.460 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[56514]: cluster 2021-03-15T22:20:12.154670+0000 mon.a (mon.0) 613 : cluster [DBG] osdmap e97: 8 total, 8 up, 8 in 2021-03-15T22:20:13.460 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[56514]: cluster 2021-03-15T22:20:12.978127+0000 mgr.y (mgr.14144) 368 : cluster [DBG] pgmap v273: 148 pgs: 148 active+clean; 7.0 MiB data, 70 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 556 KiB/s wr, 3 op/s 2021-03-15T22:20:13.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[62156]: audit 2021-03-15T22:20:13.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:20:13.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[62156]: 20:12.154616+0000 mon.a (mon.0) 612 : audit [INF] from='client.? ' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-03-15T22:20:13.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:20:13.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[62156]: :20:12.154670+0000 mon.a (mon.0) 613 : cluster 2021-03-15T22:20:13.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[62156]: [DBG] osdmap e97: 8 total, 8 up, 8 in 2021-03-15T22:20:13.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[62156]: cluster 2021-03-15T22:20:13.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:20:13.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[62156]: 22:20:12.978127+0000 mgr.y (mgr.14144) 368 : cluster 2021-03-15T22:20:13.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:13 smithi076 conmon[62156]: [DBG] pgmap v273: 148 pgs: 148 active+clean; 7.0 MiB data, 70 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 556 KiB/s wr, 3 op/s 2021-03-15T22:20:13.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:13 smithi125 conmon[56739]: audit 2021-03-15T22:20:13.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:13 smithi125 conmon[56739]: 2021-03-15T22 2021-03-15T22:20:13.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:13 smithi125 conmon[56739]: :20:12.154616+0000 mon.a (mon.0) 612 : audit [INF] from='client.? ' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2021-03-15T22:20:13.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:13 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:20:13.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:13 smithi125 conmon[56739]: 03-15T22:20:12.154670+0000 mon.a ( 2021-03-15T22:20:13.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:13 smithi125 conmon[56739]: mon.0) 613 : cluster [DBG] 2021-03-15T22:20:13.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:13 smithi125 conmon[56739]: osdmap e97: 8 total, 8 up, 8 in 2021-03-15T22:20:13.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:13 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:20:13.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:13 smithi125 conmon[56739]: -03-15T22:20:12.978127+0000 2021-03-15T22:20:13.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:13 smithi125 conmon[56739]: mgr.y (mgr.14144) 368 : 2021-03-15T22:20:13.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:13 smithi125 conmon[56739]: cluster [DBG] pgmap v273: 148 pgs: 148 active+clean; 7.0 MiB data, 70 MiB used, 715 GiB / 715 GiB avail; 1.3 KiB/s rd, 556 KiB/s wr, 3 op/s 2021-03-15T22:20:13.559 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:13.583 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:13.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:13.686 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:13.708 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:13.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:13.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:13.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:13.832 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:13.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:13.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:13.935 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:13.962 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:13.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:13.963 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:14.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:14.090 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:14.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:14.092 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:14.177 INFO:tasks.rados.rados.0.smithi076.stdout:57: done (0 left) 2021-03-15T22:20:14.177 INFO:tasks.rados.rados.0.smithi076.stdout:58: write oid 33 current snap is 2 2021-03-15T22:20:14.178 INFO:tasks.rados.rados.0.smithi076.stdout:58: seq_num 53 ranges {490771=686315,1885075=406346} 2021-03-15T22:20:14.180 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:20:14.183 INFO:tasks.rados.rados.0.smithi076.stdout:58: writing smithi07694878-33 from 490771 to 1177086 tid 1 2021-03-15T22:20:14.190 INFO:tasks.rados.rados.0.smithi076.stdout:58: writing smithi07694878-33 from 1885075 to 2291421 tid 2 2021-03-15T22:20:14.190 INFO:tasks.rados.rados.0.smithi076.stdout:59: delete oid 3 current snap is 2 2021-03-15T22:20:14.190 INFO:tasks.rados.rados.0.smithi076.stdout:58: finishing write tid 1 to smithi07694878-33 2021-03-15T22:20:14.192 INFO:tasks.rados.rados.0.smithi076.stdout:59: done (1 left) 2021-03-15T22:20:14.192 INFO:tasks.rados.rados.0.smithi076.stdout:60: write oid 12 current snap is 2 2021-03-15T22:20:14.193 INFO:tasks.rados.rados.0.smithi076.stdout:60: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2021-03-15T22:20:14.194 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:14.198 INFO:tasks.rados.rados.0.smithi076.stdout:60: writing smithi07694878-12 from 524385 to 1077551 tid 1 2021-03-15T22:20:14.203 INFO:tasks.rados.rados.0.smithi076.stdout:60: writing smithi07694878-12 from 1602039 to 2012054 tid 2 2021-03-15T22:20:14.208 INFO:tasks.rados.rados.0.smithi076.stdout:60: writing smithi07694878-12 from 2577441 to 2924385 tid 3 2021-03-15T22:20:14.209 INFO:tasks.rados.rados.0.smithi076.stdout:61: rollback oid 47 current snap is 2 2021-03-15T22:20:14.209 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 47 to 1 2021-03-15T22:20:14.210 INFO:tasks.rados.rados.0.smithi076.stdout:58: finishing write tid 2 to smithi07694878-33 2021-03-15T22:20:14.210 INFO:tasks.rados.rados.0.smithi076.stdout:58: finishing write tid 3 to smithi07694878-33 2021-03-15T22:20:14.210 INFO:tasks.rados.rados.0.smithi076.stdout:58: finishing write tid 4 to smithi07694878-33 2021-03-15T22:20:14.210 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 15 (ObjNum 53 snap 2 seq_num 53) dirty exists 2021-03-15T22:20:14.211 INFO:tasks.rados.rados.0.smithi076.stdout:58: left oid 33 (ObjNum 53 snap 2 seq_num 53) 2021-03-15T22:20:14.211 INFO:tasks.rados.rados.0.smithi076.stdout:60: finishing write tid 1 to smithi07694878-12 2021-03-15T22:20:14.211 INFO:tasks.rados.rados.0.smithi076.stdout:60: finishing write tid 2 to smithi07694878-12 2021-03-15T22:20:14.212 INFO:tasks.rados.rados.0.smithi076.stdout:58: done (2 left) 2021-03-15T22:20:14.212 INFO:tasks.rados.rados.0.smithi076.stdout:62: snap_remove snap 0 2021-03-15T22:20:14.217 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:14.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:14.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:14.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:14 smithi076 conmon[56514]: cluster 2021-03-15T22:20:13.166590+0000 mon.a (mon.0) 614 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2021-03-15T22:20:14.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:14 smithi076 conmon[56514]: audit 2021-03-15T22:20:13.547016+0000 mon.c (mon.1) 16 : audit [INF] from='client.? 172.21.15.76:0/2852277535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:14.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:14 smithi076 conmon[56514]: audit 2021-03-15T22:20:13.547342+0000 mon.a (mon.0) 615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:14.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:14 smithi076 conmon[62156]: cluster 2021-03-15T22:20:13.166590+0000 mon.a (mon.0) 614 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2021-03-15T22:20:14.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:14 smithi076 conmon[62156]: audit 2021-03-15T22:20:13.547016+0000 mon.c (mon.1) 16 : audit [INF] from='client.? 172.21.15.76:0/2852277535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:14.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:14 smithi076 conmon[62156]: audit 2021-03-15T22:20:13.547342+0000 mon.a (mon.0) 615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:14.319 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:14.346 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:14.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:14.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:14.449 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:14.474 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:14.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:14.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:14.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:14 smithi125 conmon[56739]: cluster 2021-03-15T22:20:13.166590+0000 mon.a (mon.0) 614 : cluster [DBG] osdmap e98: 8 total, 8 up, 8 in 2021-03-15T22:20:14.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:14 smithi125 conmon[56739]: audit 2021-03-15T22:20:13.547016+0000 mon.c (mon.1) 16 : audit [INF] from='client.? 172.21.15.76:0/2852277535' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:14.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:14 smithi125 conmon[56739]: audit 2021-03-15T22:20:13.547342+0000 mon.a (mon.0) 615 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:14.577 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:14.600 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:14.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:14.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:14.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:14.726 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:14.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:14.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:14.828 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:14.851 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:14.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:14.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:14.954 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:14.978 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:14.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:14.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:14.989 INFO:tasks.rados.rados.0.smithi076.stdout:62: done (2 left) 2021-03-15T22:20:14.989 INFO:tasks.rados.rados.0.smithi076.stdout:63: read oid 28 snap -1 2021-03-15T22:20:14.989 INFO:tasks.rados.rados.0.smithi076.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2021-03-15T22:20:14.989 INFO:tasks.rados.rados.0.smithi076.stdout:64: read oid 33 snap -1 2021-03-15T22:20:14.990 INFO:tasks.rados.rados.0.smithi076.stdout:64: expect (ObjNum 53 snap 2 seq_num 53) 2021-03-15T22:20:14.990 INFO:tasks.rados.rados.0.smithi076.stdout:65: delete oid 2 current snap is 2 2021-03-15T22:20:14.990 INFO:tasks.rados.rados.0.smithi076.stdout:60: finishing write tid 3 to smithi07694878-12 2021-03-15T22:20:14.991 INFO:tasks.rados.rados.0.smithi076.stdout:60: finishing write tid 4 to smithi07694878-12 2021-03-15T22:20:14.991 INFO:tasks.rados.rados.0.smithi076.stdout:61: finishing rollback tid 0 to smithi07694878-47 2021-03-15T22:20:14.991 INFO:tasks.rados.rados.0.smithi076.stdout:61: finishing rollback tid 1 to smithi07694878-47 2021-03-15T22:20:14.991 INFO:tasks.rados.rados.0.smithi076.stdout:60: finishing write tid 5 to smithi07694878-12 2021-03-15T22:20:14.991 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 26 (ObjNum 55 snap 2 seq_num 55) dirty exists 2021-03-15T22:20:14.992 INFO:tasks.rados.rados.0.smithi076.stdout:60: left oid 12 (ObjNum 55 snap 2 seq_num 55) 2021-03-15T22:20:14.992 INFO:tasks.rados.rados.0.smithi076.stdout:61: finishing rollback tid 2 to smithi07694878-47 2021-03-15T22:20:14.992 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 13 (ObjNum 46 snap 0 seq_num 46) dirty exists 2021-03-15T22:20:14.995 INFO:tasks.rados.rados.0.smithi076.stdout:60: done (4 left) 2021-03-15T22:20:14.996 INFO:tasks.rados.rados.0.smithi076.stdout:61: done (3 left) 2021-03-15T22:20:14.996 INFO:tasks.rados.rados.0.smithi076.stdout:65: done (2 left) 2021-03-15T22:20:14.996 INFO:tasks.rados.rados.0.smithi076.stdout:66: copy_from oid 3 from oid 31 current snap is 2 2021-03-15T22:20:14.996 INFO:tasks.rados.rados.0.smithi076.stdout:67: snap_create 2021-03-15T22:20:14.999 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:20:14 smithi076 conmon[65511]: debug 2021-03-15T22:20:14.994+0000 7f1a20879700 -1 osd.0 pg_epoch: 100 pg[7.4( v 99'15 (0'0,99'15] local-lis/les=91/92 n=5 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [0,4,1] r=0 lpr=91 crt=99'15 lcod 99'14 mlcod 99'14 active+clean+snaptrim trimq=[2~1]] removing snap head 2021-03-15T22:20:14.999 INFO:tasks.rados.rados.0.smithi076.stdout:64: expect (ObjNum 53 snap 2 seq_num 53) 2021-03-15T22:20:15.080 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:15.105 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:15.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:15.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:15.145 INFO:tasks.rados.rados.0.smithi076.stdout:66: finishing copy_from to smithi07694878-3 2021-03-15T22:20:15.145 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 6 (ObjNum 30 snap 0 seq_num 30) dirty exists 2021-03-15T22:20:15.145 INFO:tasks.rados.rados.0.smithi076.stdout:66: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:20:15.145 INFO:tasks.rados.rados.0.smithi076.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2021-03-15T22:20:15.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:15.230 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:15.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:15.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:15.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[56514]: cluster 2021-03-15T22:20:14.163921+0000 mon.a (mon.0) 616 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:15.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[56514]: 2021-03-15T22:20:15.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[56514]: audit 2021-03-15T22:20:14.175431+0000 mon.a (mon.0) 617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:20:15.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[56514]: cluster 2021-03-15T22:20:14.175498+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-03-15T22:20:15.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[56514]: cluster 2021-03-15T22:20:14.686399+0000 osd.5 (osd.5) 25 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:15.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[56514]: cluster 2021-03-15T22:20:14.686545+0000 osd.5 (osd.5) 26 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:15.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[56514]: cluster 2021-03-15T22:20:14.978563+0000 mgr.y (mgr.14144) 369 : cluster [DBG] pgmap v276: 148 pgs: 148 active+clean; 38 MiB data, 113 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.6 MiB/s wr, 20 op/s 2021-03-15T22:20:15.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[56514]: cluster 2021-03-15T22:20:15.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[56514]: 2021-03-15T22:20:14.986722+0000 mon.a (mon.0) 619 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-03-15T22:20:15.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: cluster 2021-03-15T22:20: 2021-03-15T22:20:15.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: 14.163921+0000 mon.a (mon.0) 616 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:15.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: audit 2021 2021-03-15T22:20:15.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: -03-15T22:20:14. 2021-03-15T22:20:15.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: 175431+0000 mon.a (mon.0) 2021-03-15T22:20:15.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: 617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:20:15.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:20:15.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: -03-15T22:20:14 2021-03-15T22:20:15.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: .175498+0000 mon.a (mon.0 2021-03-15T22:20:15.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: ) 618 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-03-15T22:20:15.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:20:15.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: 22:20:14.686399+0000 osd.5 2021-03-15T22:20:15.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: (osd.5) 25 : cluster 2021-03-15T22:20:15.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: [DBG] purged_snaps scrub starts 2021-03-15T22:20:15.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:20:15.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: 03-15T22:20:14 2021-03-15T22:20:15.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: .686545+0000 osd.5 (osd. 2021-03-15T22:20:15.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: 5) 26 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:15.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: 2021-03-15T22:20:15.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:20:15.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: 15T22:20:14.978563+0000 2021-03-15T22:20:15.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: mgr.y (mgr.14144) 369 : cluster 2021-03-15T22:20:15.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: [DBG] pgmap v276: 148 pgs: 148 active+clean; 38 MiB data, 113 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.6 MiB/s wr, 20 op/s 2021-03-15T22:20:15.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:20:15.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: :20:14.986722+0000 mon.a (mon.0 2021-03-15T22:20:15.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:15 smithi076 conmon[62156]: ) 619 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-03-15T22:20:15.309 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:20:15.326 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:20:15.333 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:15.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:20:15.355 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:15.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:15.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:15.361 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:20:15.380 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:20:15.398 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:20:15.416 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:20:15.434 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:20:15.435 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:20:15.436 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:20:15.436 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:20:15.436 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:20:15.436 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:20:15.437 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:20:15.437 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:20:15.437 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:20:15.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:15.485 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:15.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:15.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: cluster 2021-03-15T22:20:14.163921+0000 mon.a (mon.0) 616 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:15.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: audit 2021-03-15T22:20:14.175431+0000 mon.a (mon.0) 617 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:20:15.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: cluster 2021-03-15T22:20:14.175498+0000 mon.a (mon.0) 618 : cluster [DBG] osdmap e99: 8 total, 8 up, 8 in 2021-03-15T22:20:15.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: cluster 2021-03-15T22:20:14.686399+0000 osd.5 (osd.5 2021-03-15T22:20:15.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: ) 25 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:15.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:20:15.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: -15T22:20:14.686545+0000 osd.5 2021-03-15T22:20:15.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: (osd.5) 26 : cluster [DBG] 2021-03-15T22:20:15.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: purged_snaps scrub ok 2021-03-15T22:20:15.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:20:15.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: 03-15T22:20:14.978563 2021-03-15T22:20:15.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 369 : 2021-03-15T22:20:15.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: cluster [DBG] pgmap v276: 148 pgs: 148 active+clean; 38 MiB data, 113 MiB used, 715 GiB / 715 GiB avail; 4.3 KiB/s rd, 3.6 MiB/s wr, 20 op/s 2021-03-15T22:20:15.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:20:15.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: -03-15T22:20:14 2021-03-15T22:20:15.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: .986722+0000 mon.a (mon. 2021-03-15T22:20:15.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: 0) 619 : cluster [DBG] osdmap e100: 8 total, 8 up, 8 in 2021-03-15T22:20:15.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:15 smithi125 conmon[56739]: 2021-03-15T22:20:15.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:15.612 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:15.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:15.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:15.737 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:15.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:15.739 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:15.840 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:15.866 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:15.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:15.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:15.969 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:15.991 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:15.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:15.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:15.996 INFO:tasks.rados.rados.0.smithi076.stdout:63: done (3 left) 2021-03-15T22:20:15.997 INFO:tasks.rados.rados.0.smithi076.stdout:64: done (2 left) 2021-03-15T22:20:15.997 INFO:tasks.rados.rados.0.smithi076.stdout:66: done (1 left) 2021-03-15T22:20:15.997 INFO:tasks.rados.rados.0.smithi076.stdout:67: done (0 left) 2021-03-15T22:20:15.997 INFO:tasks.rados.rados.0.smithi076.stdout:68: snap_remove snap 1 2021-03-15T22:20:16.094 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:16.120 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:16.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:16.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:16.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:16.249 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:16.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:16.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:16.351 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:16.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:16.379 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:16.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:16.384 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:20:16.384 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:20:16.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:20:16.385 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:20:16.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.385 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:83 3.1b 3:dece9eff:::zonegroup_info.9cdd70dc-2472-46c3-b39c-c4226f951217:head [read 0~358] snapc 0=[] ondisk+read+known_if_redirected e57)", 2021-03-15T22:20:16.385 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:54.317616+0000", 2021-03-15T22:20:16.386 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 141.69531144000001, 2021-03-15T22:20:16.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031629129999999998, 2021-03-15T22:20:16.386 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.386 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.387 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 83 2021-03-15T22:20:16.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.388 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:54.317616+0000", 2021-03-15T22:20:16.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:54.317616+0000", 2021-03-15T22:20:16.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5830000000000001e-06 2021-03-15T22:20:16.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:54.317619+0000", 2021-03-15T22:20:16.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2180000000000002e-06 2021-03-15T22:20:16.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:54.317621+0000", 2021-03-15T22:20:16.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1612e-05 2021-03-15T22:20:16.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:54.317633+0000", 2021-03-15T22:20:16.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7870000000000001e-06 2021-03-15T22:20:16.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:54.317637+0000", 2021-03-15T22:20:16.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5058000000000002e-05 2021-03-15T22:20:16.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:54.317672+0000", 2021-03-15T22:20:16.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1705999999999995e-05 2021-03-15T22:20:16.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:54.317734+0000", 2021-03-15T22:20:16.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030449489999999999 2021-03-15T22:20:16.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:54.320779+0000", 2021-03-15T22:20:16.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031629129999999998 2021-03-15T22:20:16.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.398 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.398 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.399 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:89 4.b 4:d467b91b:::bucket.sync-target-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected e59)", 2021-03-15T22:20:16.399 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:55.995553+0000", 2021-03-15T22:20:16.399 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 140.01737486499999, 2021-03-15T22:20:16.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003497427, 2021-03-15T22:20:16.399 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.400 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:20:16.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.400 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 89 2021-03-15T22:20:16.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.401 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:55.995553+0000", 2021-03-15T22:20:16.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:55.995553+0000", 2021-03-15T22:20:16.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1019999999999996e-06 2021-03-15T22:20:16.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:55.995558+0000", 2021-03-15T22:20:16.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8299999999999998e-06 2021-03-15T22:20:16.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:55.995562+0000", 2021-03-15T22:20:16.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8981e-05 2021-03-15T22:20:16.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:55.995581+0000", 2021-03-15T22:20:16.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2590000000000006e-06 2021-03-15T22:20:16.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:55.995590+0000", 2021-03-15T22:20:16.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6062e-05 2021-03-15T22:20:16.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:55.995646+0000", 2021-03-15T22:20:16.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6809999999999993e-06 2021-03-15T22:20:16.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:20:16.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:55.995656+0000", 2021-03-15T22:20:16.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003278432 2021-03-15T22:20:16.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:55.998934+0000", 2021-03-15T22:20:16.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011608 2021-03-15T22:20:16.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:55.999050+0000", 2021-03-15T22:20:16.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003497427 2021-03-15T22:20:16.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.412 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.413 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:374 4.1e 4:783e8286:::data_log.5:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.413 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.104706+0000", 2021-03-15T22:20:16.413 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.90822162500001, 2021-03-15T22:20:16.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025609220000000002, 2021-03-15T22:20:16.413 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.414 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.414 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.414 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.414 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.415 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 374 2021-03-15T22:20:16.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.415 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.104706+0000", 2021-03-15T22:20:16.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.104706+0000", 2021-03-15T22:20:16.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9130000000000001e-06 2021-03-15T22:20:16.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.104708+0000", 2021-03-15T22:20:16.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.871e-06 2021-03-15T22:20:16.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.104710+0000", 2021-03-15T22:20:16.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8300000000000008e-06 2021-03-15T22:20:16.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.104720+0000", 2021-03-15T22:20:16.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2046e-05 2021-03-15T22:20:16.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.104732+0000", 2021-03-15T22:20:16.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0354000000000001e-05 2021-03-15T22:20:16.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.104762+0000", 2021-03-15T22:20:16.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9512000000000003e-05 2021-03-15T22:20:16.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.104812+0000", 2021-03-15T22:20:16.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024553959999999999 2021-03-15T22:20:16.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.107267+0000", 2021-03-15T22:20:16.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025609220000000002 2021-03-15T22:20:16.425 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.425 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.425 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.426 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:380 4.1e 4:79810424:::data_log.8:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.426 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.117091+0000", 2021-03-15T22:20:16.426 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.895836202, 2021-03-15T22:20:16.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024535099999999999, 2021-03-15T22:20:16.427 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.427 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.427 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.427 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.428 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 380 2021-03-15T22:20:16.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.428 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.117091+0000", 2021-03-15T22:20:16.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.117091+0000", 2021-03-15T22:20:16.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9769999999999999e-06 2021-03-15T22:20:16.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.117093+0000", 2021-03-15T22:20:16.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8989999999999999e-06 2021-03-15T22:20:16.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.117095+0000", 2021-03-15T22:20:16.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0122000000000001e-05 2021-03-15T22:20:16.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.117105+0000", 2021-03-15T22:20:16.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1690000000000002e-06 2021-03-15T22:20:16.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.117110+0000", 2021-03-15T22:20:16.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9886000000000002e-05 2021-03-15T22:20:16.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.117140+0000", 2021-03-15T22:20:16.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2391e-05 2021-03-15T22:20:16.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.117192+0000", 2021-03-15T22:20:16.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002353066 2021-03-15T22:20:16.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.119545+0000", 2021-03-15T22:20:16.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024535099999999999 2021-03-15T22:20:16.438 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.438 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.439 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.439 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:392 4.3 4:c35aa124:::data_log.14:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.439 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.138491+0000", 2021-03-15T22:20:16.440 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.87443691300001, 2021-03-15T22:20:16.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032059139999999998, 2021-03-15T22:20:16.440 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.440 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.441 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.441 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 392 2021-03-15T22:20:16.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.442 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.138491+0000", 2021-03-15T22:20:16.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.138491+0000", 2021-03-15T22:20:16.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8140000000000001e-06 2021-03-15T22:20:16.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.138494+0000", 2021-03-15T22:20:16.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7880000000000001e-06 2021-03-15T22:20:16.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.138495+0000", 2021-03-15T22:20:16.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9020000000000005e-06 2021-03-15T22:20:16.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.138504+0000", 2021-03-15T22:20:16.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9580000000000001e-06 2021-03-15T22:20:16.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.138508+0000", 2021-03-15T22:20:16.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1902000000000001e-05 2021-03-15T22:20:16.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.138540+0000", 2021-03-15T22:20:16.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1136e-05 2021-03-15T22:20:16.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.138591+0000", 2021-03-15T22:20:16.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031054139999999999 2021-03-15T22:20:16.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.141697+0000", 2021-03-15T22:20:16.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032059139999999998 2021-03-15T22:20:16.451 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.452 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.453 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:524 4.1e 4:7d72aab8:::data_log.80:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.453 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.333414+0000", 2021-03-15T22:20:16.453 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.67951388899999, 2021-03-15T22:20:16.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030143119999999999, 2021-03-15T22:20:16.453 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.454 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.454 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.454 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.454 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.455 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 524 2021-03-15T22:20:16.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.455 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.333414+0000", 2021-03-15T22:20:16.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.333414+0000", 2021-03-15T22:20:16.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6899999999999999e-06 2021-03-15T22:20:16.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.333415+0000", 2021-03-15T22:20:16.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7e-06 2021-03-15T22:20:16.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.333417+0000", 2021-03-15T22:20:16.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6330000000000002e-06 2021-03-15T22:20:16.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.333426+0000", 2021-03-15T22:20:16.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7639999999999999e-06 2021-03-15T22:20:16.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.333430+0000", 2021-03-15T22:20:16.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0176999999999999e-05 2021-03-15T22:20:16.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.333460+0000", 2021-03-15T22:20:16.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1687999999999997e-05 2021-03-15T22:20:16.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.333511+0000", 2021-03-15T22:20:16.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029166600000000002 2021-03-15T22:20:16.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.336428+0000", 2021-03-15T22:20:16.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030143119999999999 2021-03-15T22:20:16.465 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.465 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.465 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.466 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:542 4.16 4:6b9e929a:::data_log.89:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.466 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.365786+0000", 2021-03-15T22:20:16.467 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.64714204399999, 2021-03-15T22:20:16.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029911899999999999, 2021-03-15T22:20:16.467 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.467 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.468 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.468 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.468 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 542 2021-03-15T22:20:16.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.468 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.365786+0000", 2021-03-15T22:20:16.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.365786+0000", 2021-03-15T22:20:16.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.824e-06 2021-03-15T22:20:16.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.365787+0000", 2021-03-15T22:20:16.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6670000000000001e-06 2021-03-15T22:20:16.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.365789+0000", 2021-03-15T22:20:16.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8540000000000003e-06 2021-03-15T22:20:16.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.365798+0000", 2021-03-15T22:20:16.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9230000000000004e-06 2021-03-15T22:20:16.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.365802+0000", 2021-03-15T22:20:16.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7470999999999998e-05 2021-03-15T22:20:16.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.365829+0000", 2021-03-15T22:20:16.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9973999999999999e-05 2021-03-15T22:20:16.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.365889+0000", 2021-03-15T22:20:16.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028874769999999998 2021-03-15T22:20:16.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.368777+0000", 2021-03-15T22:20:16.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029911899999999999 2021-03-15T22:20:16.478 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.479 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.479 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:544 4.1e 4:7dc404fa:::data_log.90:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.480 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.369869+0000", 2021-03-15T22:20:16.480 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.64305825100001, 2021-03-15T22:20:16.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028645839999999999, 2021-03-15T22:20:16.480 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.480 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.481 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.481 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.481 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.481 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 544 2021-03-15T22:20:16.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.482 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.369869+0000", 2021-03-15T22:20:16.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.369869+0000", 2021-03-15T22:20:16.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7e-06 2021-03-15T22:20:16.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.369871+0000", 2021-03-15T22:20:16.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6139999999999999e-06 2021-03-15T22:20:16.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.369873+0000", 2021-03-15T22:20:16.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6510000000000007e-06 2021-03-15T22:20:16.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.369881+0000", 2021-03-15T22:20:16.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9489999999999998e-06 2021-03-15T22:20:16.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.369885+0000", 2021-03-15T22:20:16.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5629000000000001e-05 2021-03-15T22:20:16.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.369911+0000", 2021-03-15T22:20:16.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7046999999999999e-05 2021-03-15T22:20:16.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.369958+0000", 2021-03-15T22:20:16.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027759939999999999 2021-03-15T22:20:16.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.372734+0000", 2021-03-15T22:20:16.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028645839999999999 2021-03-15T22:20:16.491 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.492 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.492 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:562 4.6 4:620d8b71:::data_log.99:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.493 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.396525+0000", 2021-03-15T22:20:16.493 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.616403021, 2021-03-15T22:20:16.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026815960000000001, 2021-03-15T22:20:16.493 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.493 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.494 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.494 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.494 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.494 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 562 2021-03-15T22:20:16.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.495 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.396525+0000", 2021-03-15T22:20:16.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.396525+0000", 2021-03-15T22:20:16.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.871e-06 2021-03-15T22:20:16.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.396527+0000", 2021-03-15T22:20:16.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.646e-06 2021-03-15T22:20:16.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.396528+0000", 2021-03-15T22:20:16.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1009999999999998e-06 2021-03-15T22:20:16.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.396537+0000", 2021-03-15T22:20:16.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8550000000000002e-06 2021-03-15T22:20:16.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.396545+0000", 2021-03-15T22:20:16.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4936e-05 2021-03-15T22:20:16.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.396570+0000", 2021-03-15T22:20:16.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7327999999999998e-05 2021-03-15T22:20:16.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.396617+0000", 2021-03-15T22:20:16.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025888590000000002 2021-03-15T22:20:16.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.399206+0000", 2021-03-15T22:20:16.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026815960000000001 2021-03-15T22:20:16.504 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.504 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.505 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.505 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:574 4.6 4:61dd9248:::data_log.105:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.505 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.415909+0000", 2021-03-15T22:20:16.506 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.59701846499999, 2021-03-15T22:20:16.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022744729999999999, 2021-03-15T22:20:16.506 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.506 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.507 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.507 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.507 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.507 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 574 2021-03-15T22:20:16.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.508 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.415909+0000", 2021-03-15T22:20:16.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.415909+0000", 2021-03-15T22:20:16.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4789999999999999e-06 2021-03-15T22:20:16.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.415911+0000", 2021-03-15T22:20:16.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.451e-06 2021-03-15T22:20:16.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.415912+0000", 2021-03-15T22:20:16.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0830000000000004e-06 2021-03-15T22:20:16.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.415920+0000", 2021-03-15T22:20:16.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5640000000000001e-06 2021-03-15T22:20:16.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.415924+0000", 2021-03-15T22:20:16.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4303e-05 2021-03-15T22:20:16.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.415958+0000", 2021-03-15T22:20:16.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7545e-05 2021-03-15T22:20:16.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.416006+0000", 2021-03-15T22:20:16.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021780480000000001 2021-03-15T22:20:16.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.418184+0000", 2021-03-15T22:20:16.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022744729999999999 2021-03-15T22:20:16.517 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.517 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.518 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.518 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:578 4.6 4:63b3b181:::data_log.107:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.519 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.422247+0000", 2021-03-15T22:20:16.519 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.59068083400001, 2021-03-15T22:20:16.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023239889999999998, 2021-03-15T22:20:16.519 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.519 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.520 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.520 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.520 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.520 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 578 2021-03-15T22:20:16.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.521 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.422247+0000", 2021-03-15T22:20:16.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.422247+0000", 2021-03-15T22:20:16.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1169999999999998e-06 2021-03-15T22:20:16.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.422249+0000", 2021-03-15T22:20:16.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8810000000000001e-06 2021-03-15T22:20:16.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.422251+0000", 2021-03-15T22:20:16.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6900000000000004e-06 2021-03-15T22:20:16.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.422261+0000", 2021-03-15T22:20:16.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9400000000000004e-06 2021-03-15T22:20:16.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.422264+0000", 2021-03-15T22:20:16.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8966e-05 2021-03-15T22:20:16.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.422293+0000", 2021-03-15T22:20:16.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5692000000000002e-05 2021-03-15T22:20:16.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.529 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.422339+0000", 2021-03-15T22:20:16.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022317029999999998 2021-03-15T22:20:16.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.424571+0000", 2021-03-15T22:20:16.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023239889999999998 2021-03-15T22:20:16.530 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.531 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.531 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.531 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:584 4.6 4:604d3639:::data_log.110:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.532 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.431952+0000", 2021-03-15T22:20:16.532 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.58097581999999, 2021-03-15T22:20:16.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024621980000000001, 2021-03-15T22:20:16.532 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.533 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.533 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.533 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.533 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.533 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 584 2021-03-15T22:20:16.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.534 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.431952+0000", 2021-03-15T22:20:16.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.535 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.431952+0000", 2021-03-15T22:20:16.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.905e-06 2021-03-15T22:20:16.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.431954+0000", 2021-03-15T22:20:16.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.897e-06 2021-03-15T22:20:16.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.431956+0000", 2021-03-15T22:20:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0399999999999998e-06 2021-03-15T22:20:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.431965+0000", 2021-03-15T22:20:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9990000000000002e-06 2021-03-15T22:20:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.431969+0000", 2021-03-15T22:20:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3971999999999999e-05 2021-03-15T22:20:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.432003+0000", 2021-03-15T22:20:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8084000000000002e-05 2021-03-15T22:20:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.432051+0000", 2021-03-15T22:20:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023633009999999999 2021-03-15T22:20:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.434414+0000", 2021-03-15T22:20:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024621980000000001 2021-03-15T22:20:16.543 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.544 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:594 4.b 4:d2db324f:::data_log.115:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.450598+0000", 2021-03-15T22:20:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.56232924, 2021-03-15T22:20:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029225560000000002, 2021-03-15T22:20:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.545 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.546 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 594 2021-03-15T22:20:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.450598+0000", 2021-03-15T22:20:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.450598+0000", 2021-03-15T22:20:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.995e-06 2021-03-15T22:20:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.450600+0000", 2021-03-15T22:20:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.885e-06 2021-03-15T22:20:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.450602+0000", 2021-03-15T22:20:16.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4879999999999996e-06 2021-03-15T22:20:16.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.450612+0000", 2021-03-15T22:20:16.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9009999999999996e-06 2021-03-15T22:20:16.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.553 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.450616+0000", 2021-03-15T22:20:16.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8161e-05 2021-03-15T22:20:16.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.450644+0000", 2021-03-15T22:20:16.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6175999999999999e-05 2021-03-15T22:20:16.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.450700+0000", 2021-03-15T22:20:16.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00282095 2021-03-15T22:20:16.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.453521+0000", 2021-03-15T22:20:16.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029225560000000002 2021-03-15T22:20:16.556 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.557 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.557 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.557 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:596 4.16 4:68d5aad3:::data_log.116:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.558 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.454831+0000", 2021-03-15T22:20:16.558 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.558096603, 2021-03-15T22:20:16.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003053704, 2021-03-15T22:20:16.558 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.559 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.559 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.559 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.559 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.559 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 596 2021-03-15T22:20:16.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.560 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.454831+0000", 2021-03-15T22:20:16.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.454831+0000", 2021-03-15T22:20:16.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.136e-06 2021-03-15T22:20:16.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.563 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.454833+0000", 2021-03-15T22:20:16.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.968e-06 2021-03-15T22:20:16.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.563 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.564 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.454835+0000", 2021-03-15T22:20:16.564 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0295000000000001e-05 2021-03-15T22:20:16.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.564 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.565 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.454845+0000", 2021-03-15T22:20:16.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2699999999999998e-06 2021-03-15T22:20:16.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.454850+0000", 2021-03-15T22:20:16.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1134999999999999e-05 2021-03-15T22:20:16.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.567 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.454881+0000", 2021-03-15T22:20:16.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0590000000000002e-05 2021-03-15T22:20:16.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.454931+0000", 2021-03-15T22:20:16.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029533099999999998 2021-03-15T22:20:16.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.569 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.457885+0000", 2021-03-15T22:20:16.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003053704 2021-03-15T22:20:16.569 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.569 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.570 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.570 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:598 4.b 4:d676cf7e:::data_log.117:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected e61)", 2021-03-15T22:20:16.571 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:17:58.458977+0000", 2021-03-15T22:20:16.571 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.55395100300001, 2021-03-15T22:20:16.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029174370000000002, 2021-03-15T22:20:16.571 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.571 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.572 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.572 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:20:16.572 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:20:16.572 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 598 2021-03-15T22:20:16.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.573 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.458977+0000", 2021-03-15T22:20:16.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.458977+0000", 2021-03-15T22:20:16.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9039999999999999e-06 2021-03-15T22:20:16.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.458979+0000", 2021-03-15T22:20:16.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.84e-06 2021-03-15T22:20:16.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.576 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.458980+0000", 2021-03-15T22:20:16.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8959999999999992e-06 2021-03-15T22:20:16.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.578 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.458989+0000", 2021-03-15T22:20:16.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6109999999999998e-06 2021-03-15T22:20:16.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.578 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.458993+0000", 2021-03-15T22:20:16.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1782999999999997e-05 2021-03-15T22:20:16.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.459025+0000", 2021-03-15T22:20:16.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0948000000000001e-05 2021-03-15T22:20:16.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.581 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.459076+0000", 2021-03-15T22:20:16.581 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0028184550000000001 2021-03-15T22:20:16.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.581 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.582 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:17:58.461894+0000", 2021-03-15T22:20:16.582 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029174370000000002 2021-03-15T22:20:16.582 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.582 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.583 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.583 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.583 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.583 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14427.0:4 4.b 4:d242335b:gc::gc.2:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected e65)", 2021-03-15T22:20:16.583 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:18:02.028432+0000", 2021-03-15T22:20:16.584 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 133.98449552100001, 2021-03-15T22:20:16.584 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029226230000000001, 2021-03-15T22:20:16.584 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.584 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:20:16.584 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:20:16.585 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14427", 2021-03-15T22:20:16.585 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2912955800", 2021-03-15T22:20:16.585 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4 2021-03-15T22:20:16.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.586 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.586 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.586 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.028432+0000", 2021-03-15T22:20:16.586 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.587 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.028432+0000", 2021-03-15T22:20:16.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2330000000000001e-06 2021-03-15T22:20:16.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.028434+0000", 2021-03-15T22:20:16.589 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3919999999999999e-06 2021-03-15T22:20:16.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.589 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.589 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.028438+0000", 2021-03-15T22:20:16.590 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0148e-05 2021-03-15T22:20:16.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.590 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.591 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.028448+0000", 2021-03-15T22:20:16.591 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8340000000000001e-06 2021-03-15T22:20:16.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.591 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.592 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.028455+0000", 2021-03-15T22:20:16.592 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7745999999999999e-05 2021-03-15T22:20:16.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.028512+0000", 2021-03-15T22:20:16.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7659999999999998e-06 2021-03-15T22:20:16.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:20:16.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.028520+0000", 2021-03-15T22:20:16.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016018009999999999 2021-03-15T22:20:16.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.030122+0000", 2021-03-15T22:20:16.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015074000000000001 2021-03-15T22:20:16.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.030273+0000", 2021-03-15T22:20:16.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024648500000000001 2021-03-15T22:20:16.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,4", 2021-03-15T22:20:16.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.030519+0000", 2021-03-15T22:20:16.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000462709 2021-03-15T22:20:16.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:20:16.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.030982+0000", 2021-03-15T22:20:16.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000253615 2021-03-15T22:20:16.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.599 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:20:16.599 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.031236+0000", 2021-03-15T22:20:16.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2113000000000002e-05 2021-03-15T22:20:16.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:20:16.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.031268+0000", 2021-03-15T22:20:16.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1121999999999997e-05 2021-03-15T22:20:16.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:20:16.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.031309+0000", 2021-03-15T22:20:16.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5918999999999999e-05 2021-03-15T22:20:16.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.602 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:18:02.031355+0000", 2021-03-15T22:20:16.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029226230000000001 2021-03-15T22:20:16.603 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.603 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.603 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.604 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:98 4.1e e87/58 ondisk, result = 0)", 2021-03-15T22:20:16.604 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:19:40.662966+0000", 2021-03-15T22:20:16.604 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.349961219000001, 2021-03-15T22:20:16.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023852220000000002, 2021-03-15T22:20:16.605 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.605 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.605 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.662966+0000", 2021-03-15T22:20:16.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.662966+0000", 2021-03-15T22:20:16.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.81e-06 2021-03-15T22:20:16.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.608 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.662968+0000", 2021-03-15T22:20:16.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1490000000000001e-06 2021-03-15T22:20:16.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.662970+0000", 2021-03-15T22:20:16.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1337e-05 2021-03-15T22:20:16.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.662982+0000", 2021-03-15T22:20:16.610 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.46e-06 2021-03-15T22:20:16.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.611 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.611 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.662988+0000", 2021-03-15T22:20:16.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022885499999999999 2021-03-15T22:20:16.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665277+0000", 2021-03-15T22:20:16.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2040000000000002e-06 2021-03-15T22:20:16.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.613 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665283+0000", 2021-03-15T22:20:16.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8712000000000002e-05 2021-03-15T22:20:16.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665352+0000", 2021-03-15T22:20:16.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023852220000000002 2021-03-15T22:20:16.615 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.615 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.615 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.616 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:98 4.1e e87/58 ondisk, result = 0)", 2021-03-15T22:20:16.616 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:19:40.663106+0000", 2021-03-15T22:20:16.616 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.349821403, 2021-03-15T22:20:16.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023190889999999999, 2021-03-15T22:20:16.616 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.617 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.617 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663106+0000", 2021-03-15T22:20:16.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663106+0000", 2021-03-15T22:20:16.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0269999999999998e-06 2021-03-15T22:20:16.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.619 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.619 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.620 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663108+0000", 2021-03-15T22:20:16.620 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1780000000000002e-06 2021-03-15T22:20:16.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.620 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.621 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663110+0000", 2021-03-15T22:20:16.621 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4620000000000002e-06 2021-03-15T22:20:16.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.621 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.622 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663120+0000", 2021-03-15T22:20:16.622 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3769999999999998e-06 2021-03-15T22:20:16.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663124+0000", 2021-03-15T22:20:16.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022406819999999999 2021-03-15T22:20:16.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.624 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665365+0000", 2021-03-15T22:20:16.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0189999999999997e-06 2021-03-15T22:20:16.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665370+0000", 2021-03-15T22:20:16.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5343999999999997e-05 2021-03-15T22:20:16.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665425+0000", 2021-03-15T22:20:16.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023190889999999999 2021-03-15T22:20:16.626 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.626 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.627 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.627 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:99 4.1e e87/58 ondisk, result = 0)", 2021-03-15T22:20:16.628 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:19:40.663163+0000", 2021-03-15T22:20:16.628 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.349764737000001, 2021-03-15T22:20:16.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023213520000000001, 2021-03-15T22:20:16.628 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.628 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.629 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663163+0000", 2021-03-15T22:20:16.629 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663163+0000", 2021-03-15T22:20:16.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7969999999999999e-06 2021-03-15T22:20:16.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663165+0000", 2021-03-15T22:20:16.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9209999999999999e-06 2021-03-15T22:20:16.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.632 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.632 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663167+0000", 2021-03-15T22:20:16.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5089999999999999e-06 2021-03-15T22:20:16.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.633 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.633 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663176+0000", 2021-03-15T22:20:16.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1680000000000001e-06 2021-03-15T22:20:16.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663182+0000", 2021-03-15T22:20:16.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002263948 2021-03-15T22:20:16.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665446+0000", 2021-03-15T22:20:16.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6789999999999998e-06 2021-03-15T22:20:16.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.637 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665450+0000", 2021-03-15T22:20:16.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4329999999999998e-05 2021-03-15T22:20:16.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.637 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.638 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665484+0000", 2021-03-15T22:20:16.638 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023213520000000001 2021-03-15T22:20:16.638 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.638 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.639 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.639 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:99 4.1e e87/58 ondisk, result = 0)", 2021-03-15T22:20:16.639 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:19:40.663279+0000", 2021-03-15T22:20:16.640 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.349648531, 2021-03-15T22:20:16.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022522530000000001, 2021-03-15T22:20:16.640 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:20:16.640 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:20:16.640 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:20:16.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.641 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:20:16.641 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663279+0000", 2021-03-15T22:20:16.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:20:16.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:20:16.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663279+0000", 2021-03-15T22:20:16.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4619999999999999e-06 2021-03-15T22:20:16.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.643 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:20:16.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663281+0000", 2021-03-15T22:20:16.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.823e-06 2021-03-15T22:20:16.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:20:16.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663282+0000", 2021-03-15T22:20:16.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4327e-05 2021-03-15T22:20:16.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:20:16.645 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663297+0000", 2021-03-15T22:20:16.645 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0305e-05 2021-03-15T22:20:16.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:20:16.646 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.663307+0000", 2021-03-15T22:20:16.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002186482 2021-03-15T22:20:16.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:20:16.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665494+0000", 2021-03-15T22:20:16.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8710000000000001e-06 2021-03-15T22:20:16.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:20:16.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665496+0000", 2021-03-15T22:20:16.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4983e-05 2021-03-15T22:20:16.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:20:16.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:20:16.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:20:16.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:19:40.665531+0000", 2021-03-15T22:20:16.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022522530000000001 2021-03-15T22:20:16.650 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.650 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.650 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.651 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:20:16.651 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:20:16.651 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:20:16.651 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:16.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:16 smithi125 conmon[56739]: audit 2021-03-15T22:20:15.289156+0000 mon.c (mon.1) 17 2021-03-15T22:20:16.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:16 smithi125 conmon[56739]: : audit [DBG] from='client.? 172.21.15.76:0/612125628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:16.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:16 smithi125 conmon[56739]: cluster 2021-03-15T22:20:15.994897+0000 mon.a 2021-03-15T22:20:16.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:16 smithi125 conmon[56739]: (mon.0) 620 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2021-03-15T22:20:16.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:16 smithi076 conmon[56514]: audit 2021-03-15T22:20:15.289156+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.76:0/612125628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:16.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:16 smithi076 conmon[56514]: cluster 2021-03-15T22:20:15.994897+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2021-03-15T22:20:16.654 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:16 smithi076 conmon[62156]: audit 2021-03-15T22:20:15.289156+0000 mon.c (mon.1) 17 : audit [DBG] from='client.? 172.21.15.76:0/612125628' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:16.655 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:16 smithi076 conmon[62156]: cluster 2021-03-15T22:20:15.994897+0000 mon.a (mon.0) 620 : cluster [DBG] osdmap e101: 8 total, 8 up, 8 in 2021-03-15T22:20:16.679 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:16.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:16.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:16.781 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:16.806 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:16.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:16.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:16.909 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:16.935 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:16.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:16.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:16.999 INFO:tasks.rados.rados.0.smithi076.stdout:68: done (0 left) 2021-03-15T22:20:17.000 INFO:tasks.rados.rados.0.smithi076.stdout:69: read oid 23 snap -1 2021-03-15T22:20:17.000 INFO:tasks.rados.rados.0.smithi076.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2021-03-15T22:20:17.000 INFO:tasks.rados.rados.0.smithi076.stdout:70: read oid 24 snap -1 2021-03-15T22:20:17.000 INFO:tasks.rados.rados.0.smithi076.stdout:70: expect (ObjNum 23 snap 0 seq_num 23) 2021-03-15T22:20:17.001 INFO:tasks.rados.rados.0.smithi076.stdout:71: read oid 41 snap -1 2021-03-15T22:20:17.001 INFO:tasks.rados.rados.0.smithi076.stdout:71: expect (ObjNum 40 snap 0 seq_num 40) 2021-03-15T22:20:17.001 INFO:tasks.rados.rados.0.smithi076.stdout:72: snap_remove snap 2 2021-03-15T22:20:17.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:17.064 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:17.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:17.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:17.100 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:20:17.131 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:20:17.166 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:17.207 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:17.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:17.209 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:17.286 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:20:16 smithi125 conmon[67289]: debug 2021-03-15T22:20:16.995+0000 7fd4d4829700 -1 osd.6 pg_epoch: 102 pg[7.7( v 100'14 (0'0,100'14] local-lis/les=91/92 n=4 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [6,0,3] r=0 lpr=91 crt=100'14 lcod 97'12 mlcod 97'12 active+clean+snaptrim trimq=[3~1] ps=[2~1]] removing snap head 2021-03-15T22:20:17.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:17 smithi125 conmon[56739]: cluster 2021-03-15T22:20:16.979096+0000 mgr.y (mgr.14144) 370 : cluster 2021-03-15T22:20:17.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:17 smithi125 conmon[56739]: [DBG] pgmap v279: 148 pgs: 148 active+clean; 117 MiB data, 200 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 15 MiB/s wr, 80 op/s 2021-03-15T22:20:17.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:17 smithi125 conmon[56739]: cluster 2021-03-15T22:20:16.995519+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2021-03-15T22:20:17.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:17 smithi076 conmon[56514]: cluster 2021-03-15T22:20:16.979096+0000 mgr.y (mgr.14144 2021-03-15T22:20:17.293 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:17 smithi076 conmon[56514]: ) 370 : cluster [DBG] pgmap v279: 148 pgs: 148 active+clean; 117 MiB data, 200 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 15 MiB/s wr, 80 op/s 2021-03-15T22:20:17.294 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:17 smithi076 conmon[56514]: cluster 2021-03-15T22:20:16.995519+0000 mon.a (mon.0) 621 : cluster [DBG] osdmap e102: 8 total, 8 up, 8 in 2021-03-15T22:20:17.294 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:17 smithi076 conmon[62156]: cluster 2021-03-15T22:20:16.979096 2021-03-15T22:20:17.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:17 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 370 : cluster [DBG] pgmap v279: 148 pgs: 148 active+clean; 117 MiB data, 200 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 15 MiB/s wr, 80 op/s 2021-03-15T22:20:17.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:17 smithi076 conmon[62156]: cluster 2021-03-15T22:20:16. 2021-03-15T22:20:17.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:17 smithi076 conmon[62156]: 995519+0000 mon.a (mon.0) 621 : cluster 2021-03-15T22:20:17.295 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:17 smithi076 conmon[62156]: [DBG] osdmap e102: 8 total, 8 up, 8 in 2021-03-15T22:20:17.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:17.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:17.339 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:17.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:17.441 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:17.452 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:17.465 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:17.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:17.466 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:17.568 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:17.593 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:17.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:17.667 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:20:17.670 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:20:17.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:17.732 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:17.733 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:17.834 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:17.860 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:17.862 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:17.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:17.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:17.992 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:17.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:17.999 INFO:tasks.rados.rados.0.smithi076.stdout:72: done (3 left) 2021-03-15T22:20:17.999 INFO:tasks.rados.rados.0.smithi076.stdout:73: write oid 47 current snap is 3 2021-03-15T22:20:17.999 INFO:tasks.rados.rados.0.smithi076.stdout:70: expect (ObjNum 23 snap 0 seq_num 23) 2021-03-15T22:20:18.077 INFO:tasks.rados.rados.0.smithi076.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2021-03-15T22:20:18.093 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:18.119 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:18.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:18.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:18.151 INFO:tasks.rados.rados.0.smithi076.stdout:71: expect (ObjNum 40 snap 0 seq_num 40) 2021-03-15T22:20:18.222 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:18.234 INFO:tasks.rados.rados.0.smithi076.stdout:73: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2021-03-15T22:20:18.241 INFO:tasks.rados.rados.0.smithi076.stdout:73: writing smithi07694878-47 from 557999 to 1261663 tid 1 2021-03-15T22:20:18.245 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:18.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:18.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:18.247 INFO:tasks.rados.rados.0.smithi076.stdout:73: writing smithi07694878-47 from 1719003 to 2132687 tid 2 2021-03-15T22:20:18.251 INFO:tasks.rados.rados.0.smithi076.stdout:73: writing smithi07694878-47 from 2632287 to 2957999 tid 3 2021-03-15T22:20:18.251 INFO:tasks.rados.rados.0.smithi076.stdout:69: done (3 left) 2021-03-15T22:20:18.251 INFO:tasks.rados.rados.0.smithi076.stdout:70: done (2 left) 2021-03-15T22:20:18.252 INFO:tasks.rados.rados.0.smithi076.stdout:71: done (1 left) 2021-03-15T22:20:18.252 INFO:tasks.rados.rados.0.smithi076.stdout:74: copy_from oid 20 from oid 12 current snap is 3 2021-03-15T22:20:18.252 INFO:tasks.rados.rados.0.smithi076.stdout:73: finishing write tid 1 to smithi07694878-47 2021-03-15T22:20:18.252 INFO:tasks.rados.rados.0.smithi076.stdout:73: finishing write tid 2 to smithi07694878-47 2021-03-15T22:20:18.253 INFO:tasks.rados.rados.0.smithi076.stdout:75: copy_from oid 26 from oid 1 current snap is 3 2021-03-15T22:20:18.253 INFO:tasks.rados.rados.0.smithi076.stdout:76: snap_create 2021-03-15T22:20:18.253 INFO:tasks.rados.rados.0.smithi076.stdout:73: finishing write tid 3 to smithi07694878-47 2021-03-15T22:20:18.254 INFO:tasks.rados.rados.0.smithi076.stdout:73: finishing write tid 4 to smithi07694878-47 2021-03-15T22:20:18.255 INFO:tasks.rados.rados.0.smithi076.stdout:73: finishing write tid 5 to smithi07694878-47 2021-03-15T22:20:18.255 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 21 (ObjNum 57 snap 3 seq_num 57) dirty exists 2021-03-15T22:20:18.255 INFO:tasks.rados.rados.0.smithi076.stdout:73: left oid 47 (ObjNum 57 snap 3 seq_num 57) 2021-03-15T22:20:18.265 INFO:tasks.rados.rados.0.smithi076.stdout:75: finishing copy_from to smithi07694878-26 2021-03-15T22:20:18.265 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-03-15T22:20:18.265 INFO:tasks.rados.rados.0.smithi076.stdout:75: finishing copy_from racing read to smithi07694878-26 2021-03-15T22:20:18.269 INFO:tasks.rados.rados.0.smithi076.stdout:74: finishing copy_from to smithi07694878-20 2021-03-15T22:20:18.269 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 26 (ObjNum 55 snap 2 seq_num 55) dirty exists 2021-03-15T22:20:18.270 INFO:tasks.rados.rados.0.smithi076.stdout:74: finishing copy_from racing read to smithi07694878-20 2021-03-15T22:20:18.348 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:18.360 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:20:18.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:18.373 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:18.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:18.475 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:18.497 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:18.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:18.499 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:18.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:18 smithi076 conmon[56514]: audit 2021-03-15T22:20:17.647971+0000 mgr.y (mgr.14144) 371 : audit [DBG] from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:20:18.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:18 smithi076 conmon[56514]: cluster 2021-03-15T22:20:17.998036+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2021-03-15T22:20:18.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:18 smithi076 conmon[62156]: audit 2021-03-15T22:20:17.647971+0000 mgr.y (mgr.14144) 371 : audit [DBG] from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:20:18.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:18 smithi076 conmon[62156]: cluster 2021-03-15T22:20:17.998036+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2021-03-15T22:20:18.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:18 smithi125 conmon[56739]: audit 2021-03-15T22:20:17.647971+0000 mgr.y (mgr.14144) 371 : audit [DBG] from='client.14727 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:20:18.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:18 smithi125 conmon[56739]: cluster 2021-03-15T22:20:17.998036+0000 mon.a (mon.0) 622 : cluster [DBG] osdmap e103: 8 total, 8 up, 8 in 2021-03-15T22:20:18.600 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:18.623 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:18.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:18.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:18.669 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":280,"stamp":"2021-03-15T22:20:16.993115+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":124484809,"num_objects":370,"num_object_clones":5,"num_object_copies":1110,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":370,"num_whiteouts":1,"num_read":1330,"num_read_kb":13349,"num_write":775,"num_write_kb":64546,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":777,"ondisk_log_size":777,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":220944,"kb_used_data":167024,"kb_used_omap":0,"kb_used_meta":52416,"kb_avail":749740272,"statfs":{"total":767960285184,"available":767734038528,"internally_reserved":0,"allocated":171032576,"data_stored":168350438,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53673984},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":99367052,"num_objects":44,"num_object_clones":5,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":1,"num_read":102,"num_read_kb":12631,"num_write":185,"num_write_kb":52872,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.829442"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"52","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989907+0000","last_change":"2021-03-15T22:19:28.122158+0000","last_active":"2021-03-15T22:20:14.989907+0000","last_peered":"2021-03-15T22:20:14.989907+0000","last_clean":"2021-03-15T22:20:14.989907+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:20:14.989907+0000","last_undegraded":"2021-03-15T22:20:14.989907+0000","last_fullsized":"2021-03-15T22:20:14.989907+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.122079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.122079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"49","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.205351+0000","last_change":"2021-03-15T22:19:37.782674+0000","last_active":"2021-03-15T22:20:10.205351+0000","last_peered":"2021-03-15T22:20:10.205351+0000","last_clean":"2021-03-15T22:20:10.205351+0000","last_became_active":"2021-03-15T22:17:58.014719+0000","last_became_peered":"2021-03-15T22:17:58.014719+0000","last_unstale":"2021-03-15T22:20:10.205351+0000","last_undegraded":"2021-03-15T22:20:10.205351+0000","last_fullsized":"2021-03-15T22:20:10.205351+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.782584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.782584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"60","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990940+0000","last_change":"2021-03-15T22:19:08.789028+0000","last_active":"2021-03-15T22:20:14.990940+0000","last_peered":"2021-03-15T22:20:14.990940+0000","last_clean":"2021-03-15T22:20:14.990940+0000","last_became_active":"2021-03-15T22:17:54.284855+0000","last_became_peered":"2021-03-15T22:17:54.284855+0000","last_unstale":"2021-03-15T22:20:14.990940+0000","last_undegraded":"2021-03-15T22:20:14.990940+0000","last_fullsized":"2021-03-15T22:20:14.990940+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:19:08.788964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:08.788964+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"84","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989066+0000","last_change":"2021-03-15T22:19:11.891528+0000","last_active":"2021-03-15T22:20:14.989066+0000","last_peered":"2021-03-15T22:20:14.989066+0000","last_clean":"2021-03-15T22:20:14.989066+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:20:14.989066+0000","last_undegraded":"2021-03-15T22:20:14.989066+0000","last_fullsized":"2021-03-15T22:20:14.989066+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:11.891465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.891465+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"51","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.177052+0000","last_change":"2021-03-15T22:19:26.865961+0000","last_active":"2021-03-15T22:20:14.177052+0000","last_peered":"2021-03-15T22:20:14.177052+0000","last_clean":"2021-03-15T22:20:14.177052+0000","last_became_active":"2021-03-15T22:18:00.288598+0000","last_became_peered":"2021-03-15T22:18:00.288598+0000","last_unstale":"2021-03-15T22:20:14.177052+0000","last_undegraded":"2021-03-15T22:20:14.177052+0000","last_fullsized":"2021-03-15T22:20:14.177052+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.865895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.865895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"49","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989224+0000","last_change":"2021-03-15T22:17:58.011237+0000","last_active":"2021-03-15T22:20:14.989224+0000","last_peered":"2021-03-15T22:20:14.989224+0000","last_clean":"2021-03-15T22:20:14.989224+0000","last_became_active":"2021-03-15T22:17:58.010916+0000","last_became_peered":"2021-03-15T22:17:58.010916+0000","last_unstale":"2021-03-15T22:20:14.989224+0000","last_undegraded":"2021-03-15T22:20:14.989224+0000","last_fullsized":"2021-03-15T22:20:14.989224+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:17:56.987106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"57","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.175923+0000","last_change":"2021-03-15T22:19:10.851317+0000","last_active":"2021-03-15T22:20:14.175923+0000","last_peered":"2021-03-15T22:20:14.175923+0000","last_clean":"2021-03-15T22:20:14.175923+0000","last_became_active":"2021-03-15T22:17:54.283819+0000","last_became_peered":"2021-03-15T22:17:54.283819+0000","last_unstale":"2021-03-15T22:20:14.175923+0000","last_undegraded":"2021-03-15T22:20:14.175923+0000","last_fullsized":"2021-03-15T22:20:14.175923+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:10.851256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:10.851256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"90","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.175762+0000","last_change":"2021-03-15T22:19:44.915926+0000","last_active":"2021-03-15T22:20:14.175762+0000","last_peered":"2021-03-15T22:20:14.175762+0000","last_clean":"2021-03-15T22:20:14.175762+0000","last_became_active":"2021-03-15T22:17:55.991597+0000","last_became_peered":"2021-03-15T22:17:55.991597+0000","last_unstale":"2021-03-15T22:20:14.175762+0000","last_undegraded":"2021-03-15T22:20:14.175762+0000","last_fullsized":"2021-03-15T22:20:14.175762+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:19:44.915865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:44.915865+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"51","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989821+0000","last_change":"2021-03-15T22:19:11.832031+0000","last_active":"2021-03-15T22:20:14.989821+0000","last_peered":"2021-03-15T22:20:14.989821+0000","last_clean":"2021-03-15T22:20:14.989821+0000","last_became_active":"2021-03-15T22:18:00.015776+0000","last_became_peered":"2021-03-15T22:18:00.015776+0000","last_unstale":"2021-03-15T22:20:14.989821+0000","last_undegraded":"2021-03-15T22:20:14.989821+0000","last_fullsized":"2021-03-15T22:20:14.989821+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:11.831963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.831963+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"49","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207137+0000","last_change":"2021-03-15T22:19:30.205991+0000","last_active":"2021-03-15T22:20:10.207137+0000","last_peered":"2021-03-15T22:20:10.207137+0000","last_clean":"2021-03-15T22:20:10.207137+0000","last_became_active":"2021-03-15T22:17:58.008828+0000","last_became_peered":"2021-03-15T22:17:58.008828+0000","last_unstale":"2021-03-15T22:20:10.207137+0000","last_undegraded":"2021-03-15T22:20:10.207137+0000","last_fullsized":"2021-03-15T22:20:10.207137+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.205915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.205915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207557+0000","last_change":"2021-03-15T22:18:53.334542+0000","last_active":"2021-03-15T22:20:10.207557+0000","last_peered":"2021-03-15T22:20:10.207557+0000","last_clean":"2021-03-15T22:20:10.207557+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:20:10.207557+0000","last_undegraded":"2021-03-15T22:20:10.207557+0000","last_fullsized":"2021-03-15T22:20:10.207557+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:53.334478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:53.334478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"80","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990186+0000","last_change":"2021-03-15T22:19:53.135106+0000","last_active":"2021-03-15T22:20:14.990186+0000","last_peered":"2021-03-15T22:20:14.990186+0000","last_clean":"2021-03-15T22:20:14.990186+0000","last_became_active":"2021-03-15T22:17:55.990528+0000","last_became_peered":"2021-03-15T22:17:55.990528+0000","last_unstale":"2021-03-15T22:20:14.990186+0000","last_undegraded":"2021-03-15T22:20:14.990186+0000","last_fullsized":"2021-03-15T22:20:14.990186+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:19:53.135007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:53.135007+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"47","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.205924+0000","last_change":"2021-03-15T22:19:37.268478+0000","last_active":"2021-03-15T22:20:10.205924+0000","last_peered":"2021-03-15T22:20:10.205924+0000","last_clean":"2021-03-15T22:20:10.205924+0000","last_became_active":"2021-03-15T22:18:00.019585+0000","last_became_peered":"2021-03-15T22:18:00.019585+0000","last_unstale":"2021-03-15T22:20:10.205924+0000","last_undegraded":"2021-03-15T22:20:10.205924+0000","last_fullsized":"2021-03-15T22:20:10.205924+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.268362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.268362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"51","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716775+0000","last_change":"2021-03-15T22:19:57.014430+0000","last_active":"2021-03-15T22:20:13.716775+0000","last_peered":"2021-03-15T22:20:13.716775+0000","last_clean":"2021-03-15T22:20:13.716775+0000","last_became_active":"2021-03-15T22:17:58.013236+0000","last_became_peered":"2021-03-15T22:17:58.013236+0000","last_unstale":"2021-03-15T22:20:13.716775+0000","last_undegraded":"2021-03-15T22:20:13.716775+0000","last_fullsized":"2021-03-15T22:20:13.716775+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:57.014336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:57.014336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207186+0000","last_change":"2021-03-15T22:19:06.274483+0000","last_active":"2021-03-15T22:20:10.207186+0000","last_peered":"2021-03-15T22:20:10.207186+0000","last_clean":"2021-03-15T22:20:10.207186+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:20:10.207186+0000","last_undegraded":"2021-03-15T22:20:10.207186+0000","last_fullsized":"2021-03-15T22:20:10.207186+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:06.274416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.274416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"65'6","reported_seq":"85","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990841+0000","last_change":"2021-03-15T22:19:02.277922+0000","last_active":"2021-03-15T22:20:14.990841+0000","last_peered":"2021-03-15T22:20:14.990841+0000","last_clean":"2021-03-15T22:20:14.990841+0000","last_became_active":"2021-03-15T22:19:02.274105+0000","last_became_peered":"2021-03-15T22:19:02.274105+0000","last_unstale":"2021-03-15T22:20:14.990841+0000","last_undegraded":"2021-03-15T22:20:14.990841+0000","last_fullsized":"2021-03-15T22:20:14.990841+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'6","last_scrub_stamp":"2021-03-15T22:18:59.037504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.037504+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"47","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990764+0000","last_change":"2021-03-15T22:18:00.287135+0000","last_active":"2021-03-15T22:20:14.990764+0000","last_peered":"2021-03-15T22:20:14.990764+0000","last_clean":"2021-03-15T22:20:14.990764+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:20:14.990764+0000","last_undegraded":"2021-03-15T22:20:14.990764+0000","last_fullsized":"2021-03-15T22:20:14.990764+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"49","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.205757+0000","last_change":"2021-03-15T22:19:35.779203+0000","last_active":"2021-03-15T22:20:10.205757+0000","last_peered":"2021-03-15T22:20:10.205757+0000","last_clean":"2021-03-15T22:20:10.205757+0000","last_became_active":"2021-03-15T22:17:58.016067+0000","last_became_peered":"2021-03-15T22:17:58.016067+0000","last_unstale":"2021-03-15T22:20:10.205757+0000","last_undegraded":"2021-03-15T22:20:10.205757+0000","last_fullsized":"2021-03-15T22:20:10.205757+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:35.779105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.779105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"60","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.991855+0000","last_change":"2021-03-15T22:19:14.859504+0000","last_active":"2021-03-15T22:20:14.991855+0000","last_peered":"2021-03-15T22:20:14.991855+0000","last_clean":"2021-03-15T22:20:14.991855+0000","last_became_active":"2021-03-15T22:17:53.976875+0000","last_became_peered":"2021-03-15T22:17:53.976875+0000","last_unstale":"2021-03-15T22:20:14.991855+0000","last_undegraded":"2021-03-15T22:20:14.991855+0000","last_fullsized":"2021-03-15T22:20:14.991855+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.859443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"79","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.715915+0000","last_change":"2021-03-15T22:19:27.165212+0000","last_active":"2021-03-15T22:20:13.715915+0000","last_peered":"2021-03-15T22:20:13.715915+0000","last_clean":"2021-03-15T22:20:13.715915+0000","last_became_active":"2021-03-15T22:17:55.992247+0000","last_became_peered":"2021-03-15T22:17:55.992247+0000","last_unstale":"2021-03-15T22:20:13.715915+0000","last_undegraded":"2021-03-15T22:20:13.715915+0000","last_fullsized":"2021-03-15T22:20:13.715915+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:27.165119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.165119+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"51","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.177830+0000","last_change":"2021-03-15T22:19:45.893973+0000","last_active":"2021-03-15T22:20:14.177830+0000","last_peered":"2021-03-15T22:20:14.177830+0000","last_clean":"2021-03-15T22:20:14.177830+0000","last_became_active":"2021-03-15T22:18:00.286974+0000","last_became_peered":"2021-03-15T22:18:00.286974+0000","last_unstale":"2021-03-15T22:20:14.177830+0000","last_undegraded":"2021-03-15T22:20:14.177830+0000","last_fullsized":"2021-03-15T22:20:14.177830+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:45.893910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:45.893910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"54","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990122+0000","last_change":"2021-03-15T22:19:21.043341+0000","last_active":"2021-03-15T22:20:14.990122+0000","last_peered":"2021-03-15T22:20:14.990122+0000","last_clean":"2021-03-15T22:20:14.990122+0000","last_became_active":"2021-03-15T22:17:58.005477+0000","last_became_peered":"2021-03-15T22:17:58.005477+0000","last_unstale":"2021-03-15T22:20:14.990122+0000","last_undegraded":"2021-03-15T22:20:14.990122+0000","last_fullsized":"2021-03-15T22:20:14.990122+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.043261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.043261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"58","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990275+0000","last_change":"2021-03-15T22:19:35.186018+0000","last_active":"2021-03-15T22:20:14.990275+0000","last_peered":"2021-03-15T22:20:14.990275+0000","last_clean":"2021-03-15T22:20:14.990275+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:20:14.990275+0000","last_undegraded":"2021-03-15T22:20:14.990275+0000","last_fullsized":"2021-03-15T22:20:14.990275+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:35.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.185891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"86","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.174383+0000","last_change":"2021-03-15T22:19:35.934561+0000","last_active":"2021-03-15T22:20:14.174383+0000","last_peered":"2021-03-15T22:20:14.174383+0000","last_clean":"2021-03-15T22:20:14.174383+0000","last_became_active":"2021-03-15T22:17:55.995747+0000","last_became_peered":"2021-03-15T22:17:55.995747+0000","last_unstale":"2021-03-15T22:20:14.174383+0000","last_undegraded":"2021-03-15T22:20:14.174383+0000","last_fullsized":"2021-03-15T22:20:14.174383+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:35.934475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.934475+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"49","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.717676+0000","last_change":"2021-03-15T22:19:30.854972+0000","last_active":"2021-03-15T22:20:13.717676+0000","last_peered":"2021-03-15T22:20:13.717676+0000","last_clean":"2021-03-15T22:20:13.717676+0000","last_became_active":"2021-03-15T22:18:00.015889+0000","last_became_peered":"2021-03-15T22:18:00.015889+0000","last_unstale":"2021-03-15T22:20:13.717676+0000","last_undegraded":"2021-03-15T22:20:13.717676+0000","last_fullsized":"2021-03-15T22:20:13.717676+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.854899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.854899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"61'2","reported_seq":"81","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.519423+0000","last_change":"2021-03-15T22:19:21.656004+0000","last_active":"2021-03-15T22:20:14.519423+0000","last_peered":"2021-03-15T22:20:14.519423+0000","last_clean":"2021-03-15T22:20:14.519423+0000","last_became_active":"2021-03-15T22:17:58.013571+0000","last_became_peered":"2021-03-15T22:17:58.013571+0000","last_unstale":"2021-03-15T22:20:14.519423+0000","last_undegraded":"2021-03-15T22:20:14.519423+0000","last_fullsized":"2021-03-15T22:20:14.519423+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:21.655918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.655918+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.205421+0000","last_change":"2021-03-15T22:19:16.638497+0000","last_active":"2021-03-15T22:20:10.205421+0000","last_peered":"2021-03-15T22:20:10.205421+0000","last_clean":"2021-03-15T22:20:10.205421+0000","last_became_active":"2021-03-15T22:17:53.975549+0000","last_became_peered":"2021-03-15T22:17:53.975549+0000","last_unstale":"2021-03-15T22:20:10.205421+0000","last_undegraded":"2021-03-15T22:20:10.205421+0000","last_fullsized":"2021-03-15T22:20:10.205421+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.638428+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.638428+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"93","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.993713+0000","last_change":"2021-03-15T22:19:40.666578+0000","last_active":"2021-03-15T22:20:14.993713+0000","last_peered":"2021-03-15T22:20:14.993713+0000","last_clean":"2021-03-15T22:20:14.993713+0000","last_became_active":"2021-03-15T22:17:56.288283+0000","last_became_peered":"2021-03-15T22:17:56.288283+0000","last_unstale":"2021-03-15T22:20:14.993713+0000","last_undegraded":"2021-03-15T22:20:14.993713+0000","last_fullsized":"2021-03-15T22:20:14.993713+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.666518+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"47","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.205969+0000","last_change":"2021-03-15T22:19:07.544919+0000","last_active":"2021-03-15T22:20:10.205969+0000","last_peered":"2021-03-15T22:20:10.205969+0000","last_clean":"2021-03-15T22:20:10.205969+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:20:10.205969+0000","last_undegraded":"2021-03-15T22:20:10.205969+0000","last_fullsized":"2021-03-15T22:20:10.205969+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.544856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.544856+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"53","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.994057+0000","last_change":"2021-03-15T22:19:17.752562+0000","last_active":"2021-03-15T22:20:14.994057+0000","last_peered":"2021-03-15T22:20:14.994057+0000","last_clean":"2021-03-15T22:20:14.994057+0000","last_became_active":"2021-03-15T22:17:58.009184+0000","last_became_peered":"2021-03-15T22:17:58.009184+0000","last_unstale":"2021-03-15T22:20:14.994057+0000","last_undegraded":"2021-03-15T22:20:14.994057+0000","last_fullsized":"2021-03-15T22:20:14.994057+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.752495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.752495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"57","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.176425+0000","last_change":"2021-03-15T22:19:21.900042+0000","last_active":"2021-03-15T22:20:14.176425+0000","last_peered":"2021-03-15T22:20:14.176425+0000","last_clean":"2021-03-15T22:20:14.176425+0000","last_became_active":"2021-03-15T22:17:53.967947+0000","last_became_peered":"2021-03-15T22:17:53.967947+0000","last_unstale":"2021-03-15T22:20:14.176425+0000","last_undegraded":"2021-03-15T22:20:14.176425+0000","last_fullsized":"2021-03-15T22:20:14.176425+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.899969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.899969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"85","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.519365+0000","last_change":"2021-03-15T22:19:13.525956+0000","last_active":"2021-03-15T22:20:14.519365+0000","last_peered":"2021-03-15T22:20:14.519365+0000","last_clean":"2021-03-15T22:20:14.519365+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:20:14.519365+0000","last_undegraded":"2021-03-15T22:20:14.519365+0000","last_fullsized":"2021-03-15T22:20:14.519365+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:19:13.525883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.525883+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"49","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716443+0000","last_change":"2021-03-15T22:19:25.759114+0000","last_active":"2021-03-15T22:20:13.716443+0000","last_peered":"2021-03-15T22:20:13.716443+0000","last_clean":"2021-03-15T22:20:13.716443+0000","last_became_active":"2021-03-15T22:18:00.015858+0000","last_became_peered":"2021-03-15T22:18:00.015858+0000","last_unstale":"2021-03-15T22:20:13.716443+0000","last_undegraded":"2021-03-15T22:20:13.716443+0000","last_fullsized":"2021-03-15T22:20:13.716443+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.759034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"54","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.518971+0000","last_change":"2021-03-15T22:19:28.737870+0000","last_active":"2021-03-15T22:20:14.518971+0000","last_peered":"2021-03-15T22:20:14.518971+0000","last_clean":"2021-03-15T22:20:14.518971+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:20:14.518971+0000","last_undegraded":"2021-03-15T22:20:14.518971+0000","last_fullsized":"2021-03-15T22:20:14.518971+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.737796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.737796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"57","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.991065+0000","last_change":"2021-03-15T22:19:01.740592+0000","last_active":"2021-03-15T22:20:14.991065+0000","last_peered":"2021-03-15T22:20:14.991065+0000","last_clean":"2021-03-15T22:20:14.991065+0000","last_became_active":"2021-03-15T22:17:53.973507+0000","last_became_peered":"2021-03-15T22:17:53.973507+0000","last_unstale":"2021-03-15T22:20:14.991065+0000","last_undegraded":"2021-03-15T22:20:14.991065+0000","last_fullsized":"2021-03-15T22:20:14.991065+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:01.740496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:01.740496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"77","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.176497+0000","last_change":"2021-03-15T22:19:53.846004+0000","last_active":"2021-03-15T22:20:14.176497+0000","last_peered":"2021-03-15T22:20:14.176497+0000","last_clean":"2021-03-15T22:20:14.176497+0000","last_became_active":"2021-03-15T22:17:55.996707+0000","last_became_peered":"2021-03-15T22:17:55.996707+0000","last_unstale":"2021-03-15T22:20:14.176497+0000","last_undegraded":"2021-03-15T22:20:14.176497+0000","last_fullsized":"2021-03-15T22:20:14.176497+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:19:53.845914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:53.845914+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"51","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.177275+0000","last_change":"2021-03-15T22:19:14.850664+0000","last_active":"2021-03-15T22:20:14.177275+0000","last_peered":"2021-03-15T22:20:14.177275+0000","last_clean":"2021-03-15T22:20:14.177275+0000","last_became_active":"2021-03-15T22:18:00.287972+0000","last_became_peered":"2021-03-15T22:18:00.287972+0000","last_unstale":"2021-03-15T22:20:14.177275+0000","last_undegraded":"2021-03-15T22:20:14.177275+0000","last_fullsized":"2021-03-15T22:20:14.177275+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:14.850605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.850605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"51","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716429+0000","last_change":"2021-03-15T22:19:27.800200+0000","last_active":"2021-03-15T22:20:13.716429+0000","last_peered":"2021-03-15T22:20:13.716429+0000","last_clean":"2021-03-15T22:20:13.716429+0000","last_became_active":"2021-03-15T22:17:58.013174+0000","last_became_peered":"2021-03-15T22:17:58.013174+0000","last_unstale":"2021-03-15T22:20:13.716429+0000","last_undegraded":"2021-03-15T22:20:13.716429+0000","last_fullsized":"2021-03-15T22:20:13.716429+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:27.800106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.800106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.206641+0000","last_change":"2021-03-15T22:19:17.295504+0000","last_active":"2021-03-15T22:20:10.206641+0000","last_peered":"2021-03-15T22:20:10.206641+0000","last_clean":"2021-03-15T22:20:10.206641+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:20:10.206641+0000","last_undegraded":"2021-03-15T22:20:10.206641+0000","last_fullsized":"2021-03-15T22:20:10.206641+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.295412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.295412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"90'22","reported_seq":"88","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.175676+0000","last_change":"2021-03-15T22:19:58.799601+0000","last_active":"2021-03-15T22:20:14.175676+0000","last_peered":"2021-03-15T22:20:14.175676+0000","last_clean":"2021-03-15T22:20:14.175676+0000","last_became_active":"2021-03-15T22:17:56.289456+0000","last_became_peered":"2021-03-15T22:17:56.289456+0000","last_unstale":"2021-03-15T22:20:14.175676+0000","last_undegraded":"2021-03-15T22:20:14.175676+0000","last_fullsized":"2021-03-15T22:20:14.175676+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:19:58.799544+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"51","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989486+0000","last_change":"2021-03-15T22:19:26.615442+0000","last_active":"2021-03-15T22:20:14.989486+0000","last_peered":"2021-03-15T22:20:14.989486+0000","last_clean":"2021-03-15T22:20:14.989486+0000","last_became_active":"2021-03-15T22:18:00.014112+0000","last_became_peered":"2021-03-15T22:18:00.014112+0000","last_unstale":"2021-03-15T22:20:14.989486+0000","last_undegraded":"2021-03-15T22:20:14.989486+0000","last_fullsized":"2021-03-15T22:20:14.989486+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.615380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.615380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"53","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.176983+0000","last_change":"2021-03-15T22:19:03.699264+0000","last_active":"2021-03-15T22:20:14.176983+0000","last_peered":"2021-03-15T22:20:14.176983+0000","last_clean":"2021-03-15T22:20:14.176983+0000","last_became_active":"2021-03-15T22:17:58.009365+0000","last_became_peered":"2021-03-15T22:17:58.009365+0000","last_unstale":"2021-03-15T22:20:14.176983+0000","last_undegraded":"2021-03-15T22:20:14.176983+0000","last_fullsized":"2021-03-15T22:20:14.176983+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_clean_scrub_stamp":"2021-03-15T22:19:03.699198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"55","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.717150+0000","last_change":"2021-03-15T22:19:36.927055+0000","last_active":"2021-03-15T22:20:13.717150+0000","last_peered":"2021-03-15T22:20:13.717150+0000","last_clean":"2021-03-15T22:20:13.717150+0000","last_became_active":"2021-03-15T22:17:53.979947+0000","last_became_peered":"2021-03-15T22:17:53.979947+0000","last_unstale":"2021-03-15T22:20:13.717150+0000","last_undegraded":"2021-03-15T22:20:13.717150+0000","last_fullsized":"2021-03-15T22:20:13.717150+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:36.926961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:36.926961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"79","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.206107+0000","last_change":"2021-03-15T22:19:38.767499+0000","last_active":"2021-03-15T22:20:10.206107+0000","last_peered":"2021-03-15T22:20:10.206107+0000","last_clean":"2021-03-15T22:20:10.206107+0000","last_became_active":"2021-03-15T22:17:55.991537+0000","last_became_peered":"2021-03-15T22:17:55.991537+0000","last_unstale":"2021-03-15T22:20:10.206107+0000","last_undegraded":"2021-03-15T22:20:10.206107+0000","last_fullsized":"2021-03-15T22:20:10.206107+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:38.767414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.767414+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"46","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.175390+0000","last_change":"2021-03-15T22:18:00.018406+0000","last_active":"2021-03-15T22:20:14.175390+0000","last_peered":"2021-03-15T22:20:14.175390+0000","last_clean":"2021-03-15T22:20:14.175390+0000","last_became_active":"2021-03-15T22:18:00.018117+0000","last_became_peered":"2021-03-15T22:18:00.018117+0000","last_unstale":"2021-03-15T22:20:14.175390+0000","last_undegraded":"2021-03-15T22:20:14.175390+0000","last_fullsized":"2021-03-15T22:20:14.175390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"49","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.206026+0000","last_change":"2021-03-15T22:19:32.759869+0000","last_active":"2021-03-15T22:20:10.206026+0000","last_peered":"2021-03-15T22:20:10.206026+0000","last_clean":"2021-03-15T22:20:10.206026+0000","last_became_active":"2021-03-15T22:17:58.014621+0000","last_became_peered":"2021-03-15T22:17:58.014621+0000","last_unstale":"2021-03-15T22:20:10.206026+0000","last_undegraded":"2021-03-15T22:20:10.206026+0000","last_fullsized":"2021-03-15T22:20:10.206026+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:32.759804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:32.759804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"58","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989869+0000","last_change":"2021-03-15T22:19:40.166781+0000","last_active":"2021-03-15T22:20:14.989869+0000","last_peered":"2021-03-15T22:20:14.989869+0000","last_clean":"2021-03-15T22:20:14.989869+0000","last_became_active":"2021-03-15T22:17:53.975292+0000","last_became_peered":"2021-03-15T22:17:53.975292+0000","last_unstale":"2021-03-15T22:20:14.989869+0000","last_undegraded":"2021-03-15T22:20:14.989869+0000","last_fullsized":"2021-03-15T22:20:14.989869+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:40.166704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.166704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"83","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990748+0000","last_change":"2021-03-15T22:19:38.156190+0000","last_active":"2021-03-15T22:20:14.990748+0000","last_peered":"2021-03-15T22:20:14.990748+0000","last_clean":"2021-03-15T22:20:14.990748+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:20:14.990748+0000","last_undegraded":"2021-03-15T22:20:14.990748+0000","last_fullsized":"2021-03-15T22:20:14.990748+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:38.156103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.156103+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"51","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.993457+0000","last_change":"2021-03-15T22:19:44.586030+0000","last_active":"2021-03-15T22:20:14.993457+0000","last_peered":"2021-03-15T22:20:14.993457+0000","last_clean":"2021-03-15T22:20:14.993457+0000","last_became_active":"2021-03-15T22:18:00.015034+0000","last_became_peered":"2021-03-15T22:18:00.015034+0000","last_unstale":"2021-03-15T22:20:14.993457+0000","last_undegraded":"2021-03-15T22:20:14.993457+0000","last_fullsized":"2021-03-15T22:20:14.993457+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:44.585947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:44.585947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"61'2","reported_seq":"78","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:12.031393+0000","last_change":"2021-03-15T22:19:39.224283+0000","last_active":"2021-03-15T22:20:12.031393+0000","last_peered":"2021-03-15T22:20:12.031393+0000","last_clean":"2021-03-15T22:20:12.031393+0000","last_became_active":"2021-03-15T22:17:58.010903+0000","last_became_peered":"2021-03-15T22:17:58.010903+0000","last_unstale":"2021-03-15T22:20:12.031393+0000","last_undegraded":"2021-03-15T22:20:12.031393+0000","last_fullsized":"2021-03-15T22:20:12.031393+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:39.224198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:39.224198+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"55","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716880+0000","last_change":"2021-03-15T22:19:43.915597+0000","last_active":"2021-03-15T22:20:13.716880+0000","last_peered":"2021-03-15T22:20:13.716880+0000","last_clean":"2021-03-15T22:20:13.716880+0000","last_became_active":"2021-03-15T22:17:54.285851+0000","last_became_peered":"2021-03-15T22:17:54.285851+0000","last_unstale":"2021-03-15T22:20:13.716880+0000","last_undegraded":"2021-03-15T22:20:13.716880+0000","last_fullsized":"2021-03-15T22:20:13.716880+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:43.915518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:43.915518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"78","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.174784+0000","last_change":"2021-03-15T22:19:07.874527+0000","last_active":"2021-03-15T22:20:14.174784+0000","last_peered":"2021-03-15T22:20:14.174784+0000","last_clean":"2021-03-15T22:20:14.174784+0000","last_became_active":"2021-03-15T22:17:56.288102+0000","last_became_peered":"2021-03-15T22:17:56.288102+0000","last_unstale":"2021-03-15T22:20:14.174784+0000","last_undegraded":"2021-03-15T22:20:14.174784+0000","last_fullsized":"2021-03-15T22:20:14.174784+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:19:07.874462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.874462+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"47","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990555+0000","last_change":"2021-03-15T22:18:00.015238+0000","last_active":"2021-03-15T22:20:14.990555+0000","last_peered":"2021-03-15T22:20:14.990555+0000","last_clean":"2021-03-15T22:20:14.990555+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:20:14.990555+0000","last_undegraded":"2021-03-15T22:20:14.990555+0000","last_fullsized":"2021-03-15T22:20:14.990555+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"61'2","reported_seq":"82","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.175092+0000","last_change":"2021-03-15T22:19:15.850282+0000","last_active":"2021-03-15T22:20:14.175092+0000","last_peered":"2021-03-15T22:20:14.175092+0000","last_clean":"2021-03-15T22:20:14.175092+0000","last_became_active":"2021-03-15T22:17:58.003748+0000","last_became_peered":"2021-03-15T22:17:58.003748+0000","last_unstale":"2021-03-15T22:20:14.175092+0000","last_undegraded":"2021-03-15T22:20:14.175092+0000","last_fullsized":"2021-03-15T22:20:14.175092+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:15.850217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.850217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"57","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990258+0000","last_change":"2021-03-15T22:19:07.795140+0000","last_active":"2021-03-15T22:20:14.990258+0000","last_peered":"2021-03-15T22:20:14.990258+0000","last_clean":"2021-03-15T22:20:14.990258+0000","last_became_active":"2021-03-15T22:17:53.968736+0000","last_became_peered":"2021-03-15T22:17:53.968736+0000","last_unstale":"2021-03-15T22:20:14.990258+0000","last_undegraded":"2021-03-15T22:20:14.990258+0000","last_fullsized":"2021-03-15T22:20:14.990258+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.795072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.795072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"75","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207814+0000","last_change":"2021-03-15T22:19:33.261320+0000","last_active":"2021-03-15T22:20:10.207814+0000","last_peered":"2021-03-15T22:20:10.207814+0000","last_clean":"2021-03-15T22:20:10.207814+0000","last_became_active":"2021-03-15T22:17:56.290563+0000","last_became_peered":"2021-03-15T22:17:56.290563+0000","last_unstale":"2021-03-15T22:20:10.207814+0000","last_undegraded":"2021-03-15T22:20:10.207814+0000","last_fullsized":"2021-03-15T22:20:10.207814+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:19:33.261213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:33.261213+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"50","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716085+0000","last_change":"2021-03-15T22:19:16.224939+0000","last_active":"2021-03-15T22:20:13.716085+0000","last_peered":"2021-03-15T22:20:13.716085+0000","last_clean":"2021-03-15T22:20:13.716085+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:20:13.716085+0000","last_undegraded":"2021-03-15T22:20:13.716085+0000","last_fullsized":"2021-03-15T22:20:13.716085+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.224872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"53","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.177693+0000","last_change":"2021-03-15T22:19:32.880446+0000","last_active":"2021-03-15T22:20:14.177693+0000","last_peered":"2021-03-15T22:20:14.177693+0000","last_clean":"2021-03-15T22:20:14.177693+0000","last_became_active":"2021-03-15T22:17:58.001340+0000","last_became_peered":"2021-03-15T22:17:58.001340+0000","last_unstale":"2021-03-15T22:20:14.177693+0000","last_undegraded":"2021-03-15T22:20:14.177693+0000","last_fullsized":"2021-03-15T22:20:14.177693+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:32.880362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:32.880362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"57","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716491+0000","last_change":"2021-03-15T22:20:03.134440+0000","last_active":"2021-03-15T22:20:13.716491+0000","last_peered":"2021-03-15T22:20:13.716491+0000","last_clean":"2021-03-15T22:20:13.716491+0000","last_became_active":"2021-03-15T22:17:53.976777+0000","last_became_peered":"2021-03-15T22:17:53.976777+0000","last_unstale":"2021-03-15T22:20:13.716491+0000","last_undegraded":"2021-03-15T22:20:13.716491+0000","last_fullsized":"2021-03-15T22:20:13.716491+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:03.134336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:03.134336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"85","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.993933+0000","last_change":"2021-03-15T22:19:06.778692+0000","last_active":"2021-03-15T22:20:14.993933+0000","last_peered":"2021-03-15T22:20:14.993933+0000","last_clean":"2021-03-15T22:20:14.993933+0000","last_became_active":"2021-03-15T22:17:56.288678+0000","last_became_peered":"2021-03-15T22:17:56.288678+0000","last_unstale":"2021-03-15T22:20:14.993933+0000","last_undegraded":"2021-03-15T22:20:14.993933+0000","last_fullsized":"2021-03-15T22:20:14.993933+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:06.778622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.778622+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"49","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.717072+0000","last_change":"2021-03-15T22:19:17.797392+0000","last_active":"2021-03-15T22:20:13.717072+0000","last_peered":"2021-03-15T22:20:13.717072+0000","last_clean":"2021-03-15T22:20:13.717072+0000","last_became_active":"2021-03-15T22:18:00.288462+0000","last_became_peered":"2021-03-15T22:18:00.288462+0000","last_unstale":"2021-03-15T22:20:13.717072+0000","last_undegraded":"2021-03-15T22:20:13.717072+0000","last_fullsized":"2021-03-15T22:20:13.717072+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.797291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.797291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"49","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.206718+0000","last_change":"2021-03-15T22:19:29.179877+0000","last_active":"2021-03-15T22:20:10.206718+0000","last_peered":"2021-03-15T22:20:10.206718+0000","last_clean":"2021-03-15T22:20:10.206718+0000","last_became_active":"2021-03-15T22:17:58.015682+0000","last_became_peered":"2021-03-15T22:17:58.015682+0000","last_unstale":"2021-03-15T22:20:10.206718+0000","last_undegraded":"2021-03-15T22:20:10.206718+0000","last_fullsized":"2021-03-15T22:20:10.206718+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.179800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"61","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.518648+0000","last_change":"2021-03-15T22:20:06.751237+0000","last_active":"2021-03-15T22:20:14.518648+0000","last_peered":"2021-03-15T22:20:14.518648+0000","last_clean":"2021-03-15T22:20:14.518648+0000","last_became_active":"2021-03-15T22:17:53.977470+0000","last_became_peered":"2021-03-15T22:17:53.977470+0000","last_unstale":"2021-03-15T22:20:14.518648+0000","last_undegraded":"2021-03-15T22:20:14.518648+0000","last_fullsized":"2021-03-15T22:20:14.518648+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:06.751174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:06.751174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'7","reported_seq":"66","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.177902+0000","last_change":"2021-03-15T22:19:47.931238+0000","last_active":"2021-03-15T22:20:14.177902+0000","last_peered":"2021-03-15T22:20:14.177902+0000","last_clean":"2021-03-15T22:20:14.177902+0000","last_became_active":"2021-03-15T22:17:56.289454+0000","last_became_peered":"2021-03-15T22:17:56.289454+0000","last_unstale":"2021-03-15T22:20:14.177902+0000","last_undegraded":"2021-03-15T22:20:14.177902+0000","last_fullsized":"2021-03-15T22:20:14.177902+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'7","last_scrub_stamp":"2021-03-15T22:19:47.931159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:47.931159+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"97'9","reported_seq":"26","reported_epoch":"97","state":"active+clean","last_fresh":"2021-03-15T22:20:12.713165+0000","last_change":"2021-03-15T22:20:05.038481+0000","last_active":"2021-03-15T22:20:12.713165+0000","last_peered":"2021-03-15T22:20:12.713165+0000","last_clean":"2021-03-15T22:20:12.713165+0000","last_became_active":"2021-03-15T22:20:05.038191+0000","last_became_peered":"2021-03-15T22:20:05.038191+0000","last_unstale":"2021-03-15T22:20:12.713165+0000","last_undegraded":"2021-03-15T22:20:12.713165+0000","last_fullsized":"2021-03-15T22:20:12.713165+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7361367,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3301,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"47","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207806+0000","last_change":"2021-03-15T22:19:11.318218+0000","last_active":"2021-03-15T22:20:10.207806+0000","last_peered":"2021-03-15T22:20:10.207806+0000","last_clean":"2021-03-15T22:20:10.207806+0000","last_became_active":"2021-03-15T22:18:00.286699+0000","last_became_peered":"2021-03-15T22:18:00.286699+0000","last_unstale":"2021-03-15T22:20:10.207806+0000","last_undegraded":"2021-03-15T22:20:10.207806+0000","last_fullsized":"2021-03-15T22:20:10.207806+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:11.318132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.318132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"58","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.717373+0000","last_change":"2021-03-15T22:19:06.942939+0000","last_active":"2021-03-15T22:20:13.717373+0000","last_peered":"2021-03-15T22:20:13.717373+0000","last_clean":"2021-03-15T22:20:13.717373+0000","last_became_active":"2021-03-15T22:17:53.979246+0000","last_became_peered":"2021-03-15T22:17:53.979246+0000","last_unstale":"2021-03-15T22:20:13.717373+0000","last_undegraded":"2021-03-15T22:20:13.717373+0000","last_fullsized":"2021-03-15T22:20:13.717373+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:19:06.942870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.942870+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"82","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.206854+0000","last_change":"2021-03-15T22:19:40.240467+0000","last_active":"2021-03-15T22:20:10.206854+0000","last_peered":"2021-03-15T22:20:10.206854+0000","last_clean":"2021-03-15T22:20:10.206854+0000","last_became_active":"2021-03-15T22:17:56.289684+0000","last_became_peered":"2021-03-15T22:17:56.289684+0000","last_unstale":"2021-03-15T22:20:10.206854+0000","last_undegraded":"2021-03-15T22:20:10.206854+0000","last_fullsized":"2021-03-15T22:20:10.206854+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:19:40.240388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.240388+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'3","reported_seq":"81","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716639+0000","last_change":"2021-03-15T22:19:51.938202+0000","last_active":"2021-03-15T22:20:13.716639+0000","last_peered":"2021-03-15T22:20:13.716639+0000","last_clean":"2021-03-15T22:20:13.716639+0000","last_became_active":"2021-03-15T22:17:58.013784+0000","last_became_peered":"2021-03-15T22:17:58.013784+0000","last_unstale":"2021-03-15T22:20:13.716639+0000","last_undegraded":"2021-03-15T22:20:13.716639+0000","last_fullsized":"2021-03-15T22:20:13.716639+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'3","last_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:19:51.938126+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"97'19","reported_seq":"46","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.994038+0000","last_change":"2021-03-15T22:20:14.994368+0000","last_active":"2021-03-15T22:20:14.994038+0000","last_peered":"2021-03-15T22:20:14.994038+0000","last_clean":"2021-03-15T22:20:14.994038+0000","last_became_active":"2021-03-15T22:20:05.034711+0000","last_became_peered":"2021-03-15T22:20:05.034711+0000","last_unstale":"2021-03-15T22:20:14.994038+0000","last_undegraded":"2021-03-15T22:20:14.994038+0000","last_fullsized":"2021-03-15T22:20:14.994038+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12453822,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":25,"num_write_kb":6999,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"51","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.177119+0000","last_change":"2021-03-15T22:19:37.905329+0000","last_active":"2021-03-15T22:20:14.177119+0000","last_peered":"2021-03-15T22:20:14.177119+0000","last_clean":"2021-03-15T22:20:14.177119+0000","last_became_active":"2021-03-15T22:18:00.019686+0000","last_became_peered":"2021-03-15T22:18:00.019686+0000","last_unstale":"2021-03-15T22:20:14.177119+0000","last_undegraded":"2021-03-15T22:20:14.177119+0000","last_fullsized":"2021-03-15T22:20:14.177119+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.905267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.905267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"55","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.717426+0000","last_change":"2021-03-15T22:18:59.946642+0000","last_active":"2021-03-15T22:20:13.717426+0000","last_peered":"2021-03-15T22:20:13.717426+0000","last_clean":"2021-03-15T22:20:13.717426+0000","last_became_active":"2021-03-15T22:17:53.978907+0000","last_became_peered":"2021-03-15T22:17:53.978907+0000","last_unstale":"2021-03-15T22:20:13.717426+0000","last_undegraded":"2021-03-15T22:20:13.717426+0000","last_fullsized":"2021-03-15T22:20:13.717426+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:59.946561+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.946561+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"80","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989807+0000","last_change":"2021-03-15T22:17:55.993082+0000","last_active":"2021-03-15T22:20:14.989807+0000","last_peered":"2021-03-15T22:20:14.989807+0000","last_clean":"2021-03-15T22:20:14.989807+0000","last_became_active":"2021-03-15T22:17:55.992386+0000","last_became_peered":"2021-03-15T22:17:55.992386+0000","last_unstale":"2021-03-15T22:20:14.989807+0000","last_undegraded":"2021-03-15T22:20:14.989807+0000","last_fullsized":"2021-03-15T22:20:14.989807+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"52","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.715713+0000","last_change":"2021-03-15T22:19:18.197284+0000","last_active":"2021-03-15T22:20:13.715713+0000","last_peered":"2021-03-15T22:20:13.715713+0000","last_clean":"2021-03-15T22:20:13.715713+0000","last_became_active":"2021-03-15T22:17:58.007916+0000","last_became_peered":"2021-03-15T22:17:58.007916+0000","last_unstale":"2021-03-15T22:20:13.715713+0000","last_undegraded":"2021-03-15T22:20:13.715713+0000","last_fullsized":"2021-03-15T22:20:13.715713+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.197224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.197224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"97'3","reported_seq":"26","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989870+0000","last_change":"2021-03-15T22:20:14.989870+0000","last_active":"2021-03-15T22:20:14.989870+0000","last_peered":"2021-03-15T22:20:14.989870+0000","last_clean":"2021-03-15T22:20:14.989870+0000","last_became_active":"2021-03-15T22:20:05.032301+0000","last_became_peered":"2021-03-15T22:20:05.032301+0000","last_unstale":"2021-03-15T22:20:14.989870+0000","last_undegraded":"2021-03-15T22:20:14.989870+0000","last_fullsized":"2021-03-15T22:20:14.989870+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2067228,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"49","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.715972+0000","last_change":"2021-03-15T22:19:09.989290+0000","last_active":"2021-03-15T22:20:13.715972+0000","last_peered":"2021-03-15T22:20:13.715972+0000","last_clean":"2021-03-15T22:20:13.715972+0000","last_became_active":"2021-03-15T22:18:00.016271+0000","last_became_peered":"2021-03-15T22:18:00.016271+0000","last_unstale":"2021-03-15T22:20:13.715972+0000","last_undegraded":"2021-03-15T22:20:13.715972+0000","last_fullsized":"2021-03-15T22:20:13.715972+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:09.989228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:09.989228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"52","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.715994+0000","last_change":"2021-03-15T22:19:29.099735+0000","last_active":"2021-03-15T22:20:13.715994+0000","last_peered":"2021-03-15T22:20:13.715994+0000","last_clean":"2021-03-15T22:20:13.715994+0000","last_became_active":"2021-03-15T22:17:58.010511+0000","last_became_peered":"2021-03-15T22:17:58.010511+0000","last_unstale":"2021-03-15T22:20:13.715994+0000","last_undegraded":"2021-03-15T22:20:13.715994+0000","last_fullsized":"2021-03-15T22:20:13.715994+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.099650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.099650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"55","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.717251+0000","last_change":"2021-03-15T22:19:47.956238+0000","last_active":"2021-03-15T22:20:13.717251+0000","last_peered":"2021-03-15T22:20:13.717251+0000","last_clean":"2021-03-15T22:20:13.717251+0000","last_became_active":"2021-03-15T22:17:53.974108+0000","last_became_peered":"2021-03-15T22:17:53.974108+0000","last_unstale":"2021-03-15T22:20:13.717251+0000","last_undegraded":"2021-03-15T22:20:13.717251+0000","last_fullsized":"2021-03-15T22:20:13.717251+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:47.956156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:47.956156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"93","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.519040+0000","last_change":"2021-03-15T22:19:22.700145+0000","last_active":"2021-03-15T22:20:14.519040+0000","last_peered":"2021-03-15T22:20:14.519040+0000","last_clean":"2021-03-15T22:20:14.519040+0000","last_became_active":"2021-03-15T22:17:56.288870+0000","last_became_peered":"2021-03-15T22:17:56.288870+0000","last_unstale":"2021-03-15T22:20:14.519040+0000","last_undegraded":"2021-03-15T22:20:14.519040+0000","last_fullsized":"2021-03-15T22:20:14.519040+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:19:22.700049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.700049+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"99'13","reported_seq":"31","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.209974+0000","last_change":"2021-03-15T22:20:05.032562+0000","last_active":"2021-03-15T22:20:14.209974+0000","last_peered":"2021-03-15T22:20:14.209974+0000","last_clean":"2021-03-15T22:20:14.209974+0000","last_became_active":"2021-03-15T22:20:05.032282+0000","last_became_peered":"2021-03-15T22:20:05.032282+0000","last_unstale":"2021-03-15T22:20:14.209974+0000","last_undegraded":"2021-03-15T22:20:14.209974+0000","last_fullsized":"2021-03-15T22:20:14.209974+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7932805,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":14,"num_write_kb":3815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"51","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990262+0000","last_change":"2021-03-15T22:19:39.681802+0000","last_active":"2021-03-15T22:20:14.990262+0000","last_peered":"2021-03-15T22:20:14.990262+0000","last_clean":"2021-03-15T22:20:14.990262+0000","last_became_active":"2021-03-15T22:18:00.016982+0000","last_became_peered":"2021-03-15T22:18:00.016982+0000","last_unstale":"2021-03-15T22:20:14.990262+0000","last_undegraded":"2021-03-15T22:20:14.990262+0000","last_fullsized":"2021-03-15T22:20:14.990262+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:39.681741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:39.681741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"52","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.715883+0000","last_change":"2021-03-15T22:19:01.036104+0000","last_active":"2021-03-15T22:20:13.715883+0000","last_peered":"2021-03-15T22:20:13.715883+0000","last_clean":"2021-03-15T22:20:13.715883+0000","last_became_active":"2021-03-15T22:17:58.001399+0000","last_became_peered":"2021-03-15T22:17:58.001399+0000","last_unstale":"2021-03-15T22:20:13.715883+0000","last_undegraded":"2021-03-15T22:20:13.715883+0000","last_fullsized":"2021-03-15T22:20:13.715883+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:01.036038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:01.036038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207336+0000","last_change":"2021-03-15T22:19:18.252532+0000","last_active":"2021-03-15T22:20:10.207336+0000","last_peered":"2021-03-15T22:20:10.207336+0000","last_clean":"2021-03-15T22:20:10.207336+0000","last_became_active":"2021-03-15T22:17:53.972006+0000","last_became_peered":"2021-03-15T22:17:53.972006+0000","last_unstale":"2021-03-15T22:20:10.207336+0000","last_undegraded":"2021-03-15T22:20:10.207336+0000","last_fullsized":"2021-03-15T22:20:10.207336+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.252456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.252456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"110","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.991173+0000","last_change":"2021-03-15T22:19:28.581985+0000","last_active":"2021-03-15T22:20:14.991173+0000","last_peered":"2021-03-15T22:20:14.991173+0000","last_clean":"2021-03-15T22:20:14.991173+0000","last_became_active":"2021-03-15T22:19:02.093463+0000","last_became_peered":"2021-03-15T22:19:02.093463+0000","last_unstale":"2021-03-15T22:20:14.991173+0000","last_undegraded":"2021-03-15T22:20:14.991173+0000","last_fullsized":"2021-03-15T22:20:14.991173+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:19:28.581920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.581920+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"100'14","reported_seq":"38","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990910+0000","last_change":"2021-03-15T22:20:14.990910+0000","last_active":"2021-03-15T22:20:14.990910+0000","last_peered":"2021-03-15T22:20:14.990910+0000","last_clean":"2021-03-15T22:20:14.990910+0000","last_became_active":"2021-03-15T22:20:05.036550+0000","last_became_peered":"2021-03-15T22:20:05.036550+0000","last_unstale":"2021-03-15T22:20:14.990910+0000","last_undegraded":"2021-03-15T22:20:14.990910+0000","last_fullsized":"2021-03-15T22:20:14.990910+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7025227,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":9,"num_read_kb":4,"num_write":17,"num_write_kb":4128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"51","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.176210+0000","last_change":"2021-03-15T22:19:19.908283+0000","last_active":"2021-03-15T22:20:14.176210+0000","last_peered":"2021-03-15T22:20:14.176210+0000","last_clean":"2021-03-15T22:20:14.176210+0000","last_became_active":"2021-03-15T22:18:00.020501+0000","last_became_peered":"2021-03-15T22:18:00.020501+0000","last_unstale":"2021-03-15T22:20:14.176210+0000","last_undegraded":"2021-03-15T22:20:14.176210+0000","last_fullsized":"2021-03-15T22:20:14.176210+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.908218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.908218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"55","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207596+0000","last_change":"2021-03-15T22:19:08.268784+0000","last_active":"2021-03-15T22:20:10.207596+0000","last_peered":"2021-03-15T22:20:10.207596+0000","last_clean":"2021-03-15T22:20:10.207596+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:20:10.207596+0000","last_undegraded":"2021-03-15T22:20:10.207596+0000","last_fullsized":"2021-03-15T22:20:10.207596+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:19:08.268707+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"58","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990635+0000","last_change":"2021-03-15T22:20:00.206145+0000","last_active":"2021-03-15T22:20:14.990635+0000","last_peered":"2021-03-15T22:20:14.990635+0000","last_clean":"2021-03-15T22:20:14.990635+0000","last_became_active":"2021-03-15T22:17:53.978521+0000","last_became_peered":"2021-03-15T22:17:53.978521+0000","last_unstale":"2021-03-15T22:20:14.990635+0000","last_undegraded":"2021-03-15T22:20:14.990635+0000","last_fullsized":"2021-03-15T22:20:14.990635+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:00.206068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:00.206068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"111","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.205929+0000","last_change":"2021-03-15T22:19:17.650024+0000","last_active":"2021-03-15T22:20:10.205929+0000","last_peered":"2021-03-15T22:20:10.205929+0000","last_clean":"2021-03-15T22:20:10.205929+0000","last_became_active":"2021-03-15T22:17:55.995319+0000","last_became_peered":"2021-03-15T22:17:55.995319+0000","last_unstale":"2021-03-15T22:20:10.205929+0000","last_undegraded":"2021-03-15T22:20:10.205929+0000","last_fullsized":"2021-03-15T22:20:10.205929+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.649963+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"53","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990736+0000","last_change":"2021-03-15T22:19:09.821008+0000","last_active":"2021-03-15T22:20:14.990736+0000","last_peered":"2021-03-15T22:20:14.990736+0000","last_clean":"2021-03-15T22:20:14.990736+0000","last_became_active":"2021-03-15T22:17:58.017434+0000","last_became_peered":"2021-03-15T22:17:58.017434+0000","last_unstale":"2021-03-15T22:20:14.990736+0000","last_undegraded":"2021-03-15T22:20:14.990736+0000","last_fullsized":"2021-03-15T22:20:14.990736+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:09.820947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:09.820947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"100'19","reported_seq":"45","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.998270+0000","last_change":"2021-03-15T22:20:14.998270+0000","last_active":"2021-03-15T22:20:14.998270+0000","last_peered":"2021-03-15T22:20:14.998270+0000","last_clean":"2021-03-15T22:20:14.998270+0000","last_became_active":"2021-03-15T22:20:05.038149+0000","last_became_peered":"2021-03-15T22:20:05.038149+0000","last_unstale":"2021-03-15T22:20:14.998270+0000","last_undegraded":"2021-03-15T22:20:14.998270+0000","last_fullsized":"2021-03-15T22:20:14.998270+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6302765,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":16,"num_read_kb":12586,"num_write":17,"num_write_kb":3955,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"49","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.717340+0000","last_change":"2021-03-15T22:19:31.885077+0000","last_active":"2021-03-15T22:20:13.717340+0000","last_peered":"2021-03-15T22:20:13.717340+0000","last_clean":"2021-03-15T22:20:13.717340+0000","last_became_active":"2021-03-15T22:18:00.287780+0000","last_became_peered":"2021-03-15T22:18:00.287780+0000","last_unstale":"2021-03-15T22:20:13.717340+0000","last_undegraded":"2021-03-15T22:20:13.717340+0000","last_fullsized":"2021-03-15T22:20:13.717340+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:31.884965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:31.884965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.205706+0000","last_change":"2021-03-15T22:19:05.537563+0000","last_active":"2021-03-15T22:20:10.205706+0000","last_peered":"2021-03-15T22:20:10.205706+0000","last_clean":"2021-03-15T22:20:10.205706+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:20:10.205706+0000","last_undegraded":"2021-03-15T22:20:10.205706+0000","last_fullsized":"2021-03-15T22:20:10.205706+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:05.537497+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:05.537497+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"59","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.715760+0000","last_change":"2021-03-15T22:19:24.142618+0000","last_active":"2021-03-15T22:20:13.715760+0000","last_peered":"2021-03-15T22:20:13.715760+0000","last_clean":"2021-03-15T22:20:13.715760+0000","last_became_active":"2021-03-15T22:17:51.631460+0000","last_became_peered":"2021-03-15T22:17:51.631460+0000","last_unstale":"2021-03-15T22:20:13.715760+0000","last_undegraded":"2021-03-15T22:20:13.715760+0000","last_fullsized":"2021-03-15T22:20:13.715760+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.142558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.142558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'15","reported_seq":"82","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.205605+0000","last_change":"2021-03-15T22:19:24.648256+0000","last_active":"2021-03-15T22:20:10.205605+0000","last_peered":"2021-03-15T22:20:10.205605+0000","last_clean":"2021-03-15T22:20:10.205605+0000","last_became_active":"2021-03-15T22:17:55.992952+0000","last_became_peered":"2021-03-15T22:17:55.992952+0000","last_unstale":"2021-03-15T22:20:10.205605+0000","last_undegraded":"2021-03-15T22:20:10.205605+0000","last_fullsized":"2021-03-15T22:20:10.205605+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:24.648200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.648200+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"52","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716204+0000","last_change":"2021-03-15T22:19:15.249150+0000","last_active":"2021-03-15T22:20:13.716204+0000","last_peered":"2021-03-15T22:20:13.716204+0000","last_clean":"2021-03-15T22:20:13.716204+0000","last_became_active":"2021-03-15T22:17:58.014849+0000","last_became_peered":"2021-03-15T22:17:58.014849+0000","last_unstale":"2021-03-15T22:20:13.716204+0000","last_undegraded":"2021-03-15T22:20:13.716204+0000","last_fullsized":"2021-03-15T22:20:13.716204+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:15.249076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.249076+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"100'28","reported_seq":"57","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.997407+0000","last_change":"2021-03-15T22:20:14.997407+0000","last_active":"2021-03-15T22:20:14.997407+0000","last_peered":"2021-03-15T22:20:14.997407+0000","last_clean":"2021-03-15T22:20:14.997407+0000","last_became_active":"2021-03-15T22:20:05.035885+0000","last_became_peered":"2021-03-15T22:20:05.035885+0000","last_unstale":"2021-03-15T22:20:14.997407+0000","last_undegraded":"2021-03-15T22:20:14.997407+0000","last_fullsized":"2021-03-15T22:20:14.997407+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18156511,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":15,"num_read_kb":8,"num_write":33,"num_write_kb":8337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"47","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207328+0000","last_change":"2021-03-15T22:19:25.184338+0000","last_active":"2021-03-15T22:20:10.207328+0000","last_peered":"2021-03-15T22:20:10.207328+0000","last_clean":"2021-03-15T22:20:10.207328+0000","last_became_active":"2021-03-15T22:18:00.019363+0000","last_became_peered":"2021-03-15T22:18:00.019363+0000","last_unstale":"2021-03-15T22:20:10.207328+0000","last_undegraded":"2021-03-15T22:20:10.207328+0000","last_fullsized":"2021-03-15T22:20:10.207328+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.184260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.184260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"58","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716156+0000","last_change":"2021-03-15T22:19:23.843965+0000","last_active":"2021-03-15T22:20:13.716156+0000","last_peered":"2021-03-15T22:20:13.716156+0000","last_clean":"2021-03-15T22:20:13.716156+0000","last_became_active":"2021-03-15T22:17:51.044140+0000","last_became_peered":"2021-03-15T22:17:51.044140+0000","last_unstale":"2021-03-15T22:20:13.716156+0000","last_undegraded":"2021-03-15T22:20:13.716156+0000","last_fullsized":"2021-03-15T22:20:13.716156+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:23.843892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.843892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"57","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.174506+0000","last_change":"2021-03-15T22:19:20.880709+0000","last_active":"2021-03-15T22:20:14.174506+0000","last_peered":"2021-03-15T22:20:14.174506+0000","last_clean":"2021-03-15T22:20:14.174506+0000","last_became_active":"2021-03-15T22:17:53.972080+0000","last_became_peered":"2021-03-15T22:17:53.972080+0000","last_unstale":"2021-03-15T22:20:14.174506+0000","last_undegraded":"2021-03-15T22:20:14.174506+0000","last_fullsized":"2021-03-15T22:20:14.174506+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:20.880638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:20.880638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"78","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.519339+0000","last_change":"2021-03-15T22:19:23.732253+0000","last_active":"2021-03-15T22:20:14.519339+0000","last_peered":"2021-03-15T22:20:14.519339+0000","last_clean":"2021-03-15T22:20:14.519339+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:20:14.519339+0000","last_undegraded":"2021-03-15T22:20:14.519339+0000","last_fullsized":"2021-03-15T22:20:14.519339+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:23.732187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.732187+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"51","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.717121+0000","last_change":"2021-03-15T22:19:07.981284+0000","last_active":"2021-03-15T22:20:13.717121+0000","last_peered":"2021-03-15T22:20:13.717121+0000","last_clean":"2021-03-15T22:20:13.717121+0000","last_became_active":"2021-03-15T22:17:58.011911+0000","last_became_peered":"2021-03-15T22:17:58.011911+0000","last_unstale":"2021-03-15T22:20:13.717121+0000","last_undegraded":"2021-03-15T22:20:13.717121+0000","last_fullsized":"2021-03-15T22:20:13.717121+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.981223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"97'6","reported_seq":"23","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.717044+0000","last_change":"2021-03-15T22:20:05.035960+0000","last_active":"2021-03-15T22:20:13.717044+0000","last_peered":"2021-03-15T22:20:13.717044+0000","last_clean":"2021-03-15T22:20:13.717044+0000","last_became_active":"2021-03-15T22:20:05.035608+0000","last_became_peered":"2021-03-15T22:20:05.035608+0000","last_unstale":"2021-03-15T22:20:13.717044+0000","last_undegraded":"2021-03-15T22:20:13.717044+0000","last_fullsized":"2021-03-15T22:20:13.717044+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4521017,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2130,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"47","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.205670+0000","last_change":"2021-03-15T22:19:28.773808+0000","last_active":"2021-03-15T22:20:10.205670+0000","last_peered":"2021-03-15T22:20:10.205670+0000","last_clean":"2021-03-15T22:20:10.205670+0000","last_became_active":"2021-03-15T22:18:00.014336+0000","last_became_peered":"2021-03-15T22:18:00.014336+0000","last_unstale":"2021-03-15T22:20:10.205670+0000","last_undegraded":"2021-03-15T22:20:10.205670+0000","last_fullsized":"2021-03-15T22:20:10.205670+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.773747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.773747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"57","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990499+0000","last_change":"2021-03-15T22:19:38.721599+0000","last_active":"2021-03-15T22:20:14.990499+0000","last_peered":"2021-03-15T22:20:14.990499+0000","last_clean":"2021-03-15T22:20:14.990499+0000","last_became_active":"2021-03-15T22:17:53.977311+0000","last_became_peered":"2021-03-15T22:17:53.977311+0000","last_unstale":"2021-03-15T22:20:14.990499+0000","last_undegraded":"2021-03-15T22:20:14.990499+0000","last_fullsized":"2021-03-15T22:20:14.990499+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:38.721534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.721534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"200","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716073+0000","last_change":"2021-03-15T22:19:03.966872+0000","last_active":"2021-03-15T22:20:13.716073+0000","last_peered":"2021-03-15T22:20:13.716073+0000","last_clean":"2021-03-15T22:20:13.716073+0000","last_became_active":"2021-03-15T22:17:51.044139+0000","last_became_peered":"2021-03-15T22:17:51.044139+0000","last_unstale":"2021-03-15T22:20:13.716073+0000","last_undegraded":"2021-03-15T22:20:13.716073+0000","last_fullsized":"2021-03-15T22:20:13.716073+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:19:03.966803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:19:03.966803+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":140,"num_read_kb":135,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"81","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.993540+0000","last_change":"2021-03-15T22:19:30.589143+0000","last_active":"2021-03-15T22:20:14.993540+0000","last_peered":"2021-03-15T22:20:14.993540+0000","last_clean":"2021-03-15T22:20:14.993540+0000","last_became_active":"2021-03-15T22:17:55.999564+0000","last_became_peered":"2021-03-15T22:17:55.999564+0000","last_unstale":"2021-03-15T22:20:14.993540+0000","last_undegraded":"2021-03-15T22:20:14.993540+0000","last_fullsized":"2021-03-15T22:20:14.993540+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:30.589062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.589062+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"49","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207677+0000","last_change":"2021-03-15T22:19:00.305911+0000","last_active":"2021-03-15T22:20:10.207677+0000","last_peered":"2021-03-15T22:20:10.207677+0000","last_clean":"2021-03-15T22:20:10.207677+0000","last_became_active":"2021-03-15T22:17:58.010798+0000","last_became_peered":"2021-03-15T22:17:58.010798+0000","last_unstale":"2021-03-15T22:20:10.207677+0000","last_undegraded":"2021-03-15T22:20:10.207677+0000","last_fullsized":"2021-03-15T22:20:10.207677+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:00.305805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:00.305805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"97'21","reported_seq":"42","reported_epoch":"97","state":"active+clean","last_fresh":"2021-03-15T22:20:12.700681+0000","last_change":"2021-03-15T22:20:05.039625+0000","last_active":"2021-03-15T22:20:12.700681+0000","last_peered":"2021-03-15T22:20:12.700681+0000","last_clean":"2021-03-15T22:20:12.700681+0000","last_became_active":"2021-03-15T22:20:05.039329+0000","last_became_peered":"2021-03-15T22:20:05.039329+0000","last_unstale":"2021-03-15T22:20:12.700681+0000","last_undegraded":"2021-03-15T22:20:12.700681+0000","last_fullsized":"2021-03-15T22:20:12.700681+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17731154,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":14,"num_read_kb":7,"num_write":28,"num_write_kb":8208,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"47","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.205279+0000","last_change":"2021-03-15T22:19:15.593372+0000","last_active":"2021-03-15T22:20:10.205279+0000","last_peered":"2021-03-15T22:20:10.205279+0000","last_clean":"2021-03-15T22:20:10.205279+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:20:10.205279+0000","last_undegraded":"2021-03-15T22:20:10.205279+0000","last_fullsized":"2021-03-15T22:20:10.205279+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.593307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.205900+0000","last_change":"2021-03-15T22:19:29.812510+0000","last_active":"2021-03-15T22:20:10.205900+0000","last_peered":"2021-03-15T22:20:10.205900+0000","last_clean":"2021-03-15T22:20:10.205900+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:20:10.205900+0000","last_undegraded":"2021-03-15T22:20:10.205900+0000","last_fullsized":"2021-03-15T22:20:10.205900+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.812446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.812446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"105","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.991414+0000","last_change":"2021-03-15T22:18:56.832718+0000","last_active":"2021-03-15T22:20:14.991414+0000","last_peered":"2021-03-15T22:20:14.991414+0000","last_clean":"2021-03-15T22:20:14.991414+0000","last_became_active":"2021-03-15T22:17:56.288438+0000","last_became_peered":"2021-03-15T22:17:56.288438+0000","last_unstale":"2021-03-15T22:20:14.991414+0000","last_undegraded":"2021-03-15T22:20:14.991414+0000","last_fullsized":"2021-03-15T22:20:14.991414+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:18:56.832638+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"52","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.519033+0000","last_change":"2021-03-15T22:19:16.581584+0000","last_active":"2021-03-15T22:20:14.519033+0000","last_peered":"2021-03-15T22:20:14.519033+0000","last_clean":"2021-03-15T22:20:14.519033+0000","last_became_active":"2021-03-15T22:17:58.008731+0000","last_became_peered":"2021-03-15T22:17:58.008731+0000","last_unstale":"2021-03-15T22:20:14.519033+0000","last_undegraded":"2021-03-15T22:20:14.519033+0000","last_fullsized":"2021-03-15T22:20:14.519033+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.581514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"97'3","reported_seq":"26","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990032+0000","last_change":"2021-03-15T22:20:14.990032+0000","last_active":"2021-03-15T22:20:14.990032+0000","last_peered":"2021-03-15T22:20:14.990032+0000","last_clean":"2021-03-15T22:20:14.990032+0000","last_became_active":"2021-03-15T22:20:05.033792+0000","last_became_peered":"2021-03-15T22:20:05.033792+0000","last_unstale":"2021-03-15T22:20:14.990032+0000","last_undegraded":"2021-03-15T22:20:14.990032+0000","last_fullsized":"2021-03-15T22:20:14.990032+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2268912,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"51","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989632+0000","last_change":"2021-03-15T22:19:19.751342+0000","last_active":"2021-03-15T22:20:14.989632+0000","last_peered":"2021-03-15T22:20:14.989632+0000","last_clean":"2021-03-15T22:20:14.989632+0000","last_became_active":"2021-03-15T22:18:00.015069+0000","last_became_peered":"2021-03-15T22:18:00.015069+0000","last_unstale":"2021-03-15T22:20:14.989632+0000","last_undegraded":"2021-03-15T22:20:14.989632+0000","last_fullsized":"2021-03-15T22:20:14.989632+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.751282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.751282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"53","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207668+0000","last_change":"2021-03-15T22:19:41.212619+0000","last_active":"2021-03-15T22:20:10.207668+0000","last_peered":"2021-03-15T22:20:10.207668+0000","last_clean":"2021-03-15T22:20:10.207668+0000","last_became_active":"2021-03-15T22:17:54.284266+0000","last_became_peered":"2021-03-15T22:17:54.284266+0000","last_unstale":"2021-03-15T22:20:10.207668+0000","last_undegraded":"2021-03-15T22:20:10.207668+0000","last_fullsized":"2021-03-15T22:20:10.207668+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:41.212539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:41.212539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"80","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.206061+0000","last_change":"2021-03-15T22:19:43.737187+0000","last_active":"2021-03-15T22:20:10.206061+0000","last_peered":"2021-03-15T22:20:10.206061+0000","last_clean":"2021-03-15T22:20:10.206061+0000","last_became_active":"2021-03-15T22:17:55.992257+0000","last_became_peered":"2021-03-15T22:17:55.992257+0000","last_unstale":"2021-03-15T22:20:10.206061+0000","last_undegraded":"2021-03-15T22:20:10.206061+0000","last_fullsized":"2021-03-15T22:20:10.206061+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:43.737126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:43.737126+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"61'2","reported_seq":"82","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.991081+0000","last_change":"2021-03-15T22:19:25.604285+0000","last_active":"2021-03-15T22:20:14.991081+0000","last_peered":"2021-03-15T22:20:14.991081+0000","last_clean":"2021-03-15T22:20:14.991081+0000","last_became_active":"2021-03-15T22:17:58.008973+0000","last_became_peered":"2021-03-15T22:17:58.008973+0000","last_unstale":"2021-03-15T22:20:14.991081+0000","last_undegraded":"2021-03-15T22:20:14.991081+0000","last_fullsized":"2021-03-15T22:20:14.991081+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:25.604217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.604217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"97'6","reported_seq":"29","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.993371+0000","last_change":"2021-03-15T22:20:14.994077+0000","last_active":"2021-03-15T22:20:14.993371+0000","last_peered":"2021-03-15T22:20:14.993371+0000","last_clean":"2021-03-15T22:20:14.993371+0000","last_became_active":"2021-03-15T22:20:05.039511+0000","last_became_peered":"2021-03-15T22:20:05.039511+0000","last_unstale":"2021-03-15T22:20:14.993371+0000","last_undegraded":"2021-03-15T22:20:14.993371+0000","last_fullsized":"2021-03-15T22:20:14.993371+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4823543,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":8,"num_write_kb":2093,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"49","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.717077+0000","last_change":"2021-03-15T22:19:18.783378+0000","last_active":"2021-03-15T22:20:13.717077+0000","last_peered":"2021-03-15T22:20:13.717077+0000","last_clean":"2021-03-15T22:20:13.717077+0000","last_became_active":"2021-03-15T22:18:00.288441+0000","last_became_peered":"2021-03-15T22:18:00.288441+0000","last_unstale":"2021-03-15T22:20:13.717077+0000","last_undegraded":"2021-03-15T22:20:13.717077+0000","last_fullsized":"2021-03-15T22:20:13.717077+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.783297+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.783297+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"57","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.993615+0000","last_change":"2021-03-15T22:19:13.870267+0000","last_active":"2021-03-15T22:20:14.993615+0000","last_peered":"2021-03-15T22:20:14.993615+0000","last_clean":"2021-03-15T22:20:14.993615+0000","last_became_active":"2021-03-15T22:17:53.976307+0000","last_became_peered":"2021-03-15T22:17:53.976307+0000","last_unstale":"2021-03-15T22:20:14.993615+0000","last_undegraded":"2021-03-15T22:20:14.993615+0000","last_fullsized":"2021-03-15T22:20:14.993615+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:13.870202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.870202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"89","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990920+0000","last_change":"2021-03-15T22:19:26.087704+0000","last_active":"2021-03-15T22:20:14.990920+0000","last_peered":"2021-03-15T22:20:14.990920+0000","last_clean":"2021-03-15T22:20:14.990920+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:20:14.990920+0000","last_undegraded":"2021-03-15T22:20:14.990920+0000","last_fullsized":"2021-03-15T22:20:14.990920+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:19:26.087622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.087622+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"61'2","reported_seq":"82","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.176774+0000","last_change":"2021-03-15T22:20:00.809142+0000","last_active":"2021-03-15T22:20:14.176774+0000","last_peered":"2021-03-15T22:20:14.176774+0000","last_clean":"2021-03-15T22:20:14.176774+0000","last_became_active":"2021-03-15T22:17:58.011279+0000","last_became_peered":"2021-03-15T22:17:58.011279+0000","last_unstale":"2021-03-15T22:20:14.176774+0000","last_undegraded":"2021-03-15T22:20:14.176774+0000","last_fullsized":"2021-03-15T22:20:14.176774+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:20:00.809069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:20:00.809069+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"100'17","reported_seq":"42","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:15.011716+0000","last_change":"2021-03-15T22:20:14.995375+0000","last_active":"2021-03-15T22:20:15.011716+0000","last_peered":"2021-03-15T22:20:15.011716+0000","last_clean":"2021-03-15T22:20:15.011716+0000","last_became_active":"2021-03-15T22:20:05.034804+0000","last_became_peered":"2021-03-15T22:20:05.034804+0000","last_unstale":"2021-03-15T22:20:15.011716+0000","last_undegraded":"2021-03-15T22:20:15.011716+0000","last_fullsized":"2021-03-15T22:20:15.011716+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12000033,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":18,"num_write_kb":6914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"52","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989999+0000","last_change":"2021-03-15T22:19:34.179513+0000","last_active":"2021-03-15T22:20:14.989999+0000","last_peered":"2021-03-15T22:20:14.989999+0000","last_clean":"2021-03-15T22:20:14.989999+0000","last_became_active":"2021-03-15T22:18:00.021088+0000","last_became_peered":"2021-03-15T22:18:00.021088+0000","last_unstale":"2021-03-15T22:20:14.989999+0000","last_undegraded":"2021-03-15T22:20:14.989999+0000","last_fullsized":"2021-03-15T22:20:14.989999+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:34.179433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:34.179433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"57","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.177526+0000","last_change":"2021-03-15T22:18:59.667448+0000","last_active":"2021-03-15T22:20:14.177526+0000","last_peered":"2021-03-15T22:20:14.177526+0000","last_clean":"2021-03-15T22:20:14.177526+0000","last_became_active":"2021-03-15T22:17:53.970585+0000","last_became_peered":"2021-03-15T22:17:53.970585+0000","last_unstale":"2021-03-15T22:20:14.177526+0000","last_undegraded":"2021-03-15T22:20:14.177526+0000","last_fullsized":"2021-03-15T22:20:14.177526+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:59.667388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:59.667388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"107","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.176696+0000","last_change":"2021-03-15T22:19:04.746329+0000","last_active":"2021-03-15T22:20:14.176696+0000","last_peered":"2021-03-15T22:20:14.176696+0000","last_clean":"2021-03-15T22:20:14.176696+0000","last_became_active":"2021-03-15T22:17:56.288319+0000","last_became_peered":"2021-03-15T22:17:56.288319+0000","last_unstale":"2021-03-15T22:20:14.176696+0000","last_undegraded":"2021-03-15T22:20:14.176696+0000","last_fullsized":"2021-03-15T22:20:14.176696+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:19:04.746224+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"54","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989549+0000","last_change":"2021-03-15T22:19:50.171242+0000","last_active":"2021-03-15T22:20:14.989549+0000","last_peered":"2021-03-15T22:20:14.989549+0000","last_clean":"2021-03-15T22:20:14.989549+0000","last_became_active":"2021-03-15T22:17:58.006706+0000","last_became_peered":"2021-03-15T22:17:58.006706+0000","last_unstale":"2021-03-15T22:20:14.989549+0000","last_undegraded":"2021-03-15T22:20:14.989549+0000","last_fullsized":"2021-03-15T22:20:14.989549+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:50.171159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:50.171159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"97'9","reported_seq":"33","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.987492+0000","last_change":"2021-03-15T22:20:14.987990+0000","last_active":"2021-03-15T22:20:14.987492+0000","last_peered":"2021-03-15T22:20:14.987492+0000","last_clean":"2021-03-15T22:20:14.987492+0000","last_became_active":"2021-03-15T22:20:05.035665+0000","last_became_peered":"2021-03-15T22:20:05.035665+0000","last_unstale":"2021-03-15T22:20:14.987492+0000","last_undegraded":"2021-03-15T22:20:14.987492+0000","last_fullsized":"2021-03-15T22:20:14.987492+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7579858,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":12,"num_write_kb":3468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"47","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207450+0000","last_change":"2021-03-15T22:19:42.258910+0000","last_active":"2021-03-15T22:20:10.207450+0000","last_peered":"2021-03-15T22:20:10.207450+0000","last_clean":"2021-03-15T22:20:10.207450+0000","last_became_active":"2021-03-15T22:18:00.019457+0000","last_became_peered":"2021-03-15T22:18:00.019457+0000","last_unstale":"2021-03-15T22:20:10.207450+0000","last_undegraded":"2021-03-15T22:20:10.207450+0000","last_fullsized":"2021-03-15T22:20:10.207450+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:42.258829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:42.258829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"61'2","reported_seq":"83","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990767+0000","last_change":"2021-03-15T22:19:57.135105+0000","last_active":"2021-03-15T22:20:14.990767+0000","last_peered":"2021-03-15T22:20:14.990767+0000","last_clean":"2021-03-15T22:20:14.990767+0000","last_became_active":"2021-03-15T22:17:58.010485+0000","last_became_peered":"2021-03-15T22:17:58.010485+0000","last_unstale":"2021-03-15T22:20:14.990767+0000","last_undegraded":"2021-03-15T22:20:14.990767+0000","last_fullsized":"2021-03-15T22:20:14.990767+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:57.135028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:57.135028+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"57","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.176612+0000","last_change":"2021-03-15T22:19:24.902865+0000","last_active":"2021-03-15T22:20:14.176612+0000","last_peered":"2021-03-15T22:20:14.176612+0000","last_clean":"2021-03-15T22:20:14.176612+0000","last_became_active":"2021-03-15T22:17:53.967568+0000","last_became_peered":"2021-03-15T22:17:53.967568+0000","last_unstale":"2021-03-15T22:20:14.176612+0000","last_undegraded":"2021-03-15T22:20:14.176612+0000","last_fullsized":"2021-03-15T22:20:14.176612+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.902804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.902804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"52","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716195+0000","last_change":"2021-03-15T22:19:10.132572+0000","last_active":"2021-03-15T22:20:13.716195+0000","last_peered":"2021-03-15T22:20:13.716195+0000","last_clean":"2021-03-15T22:20:13.716195+0000","last_became_active":"2021-03-15T22:19:02.092703+0000","last_became_peered":"2021-03-15T22:19:02.092703+0000","last_unstale":"2021-03-15T22:20:13.716195+0000","last_undegraded":"2021-03-15T22:20:13.716195+0000","last_fullsized":"2021-03-15T22:20:13.716195+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:19:10.132514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:10.132514+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"97'3","reported_seq":"25","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.987650+0000","last_change":"2021-03-15T22:20:14.988124+0000","last_active":"2021-03-15T22:20:14.987650+0000","last_peered":"2021-03-15T22:20:14.987650+0000","last_clean":"2021-03-15T22:20:14.987650+0000","last_became_active":"2021-03-15T22:20:05.039641+0000","last_became_peered":"2021-03-15T22:20:05.039641+0000","last_unstale":"2021-03-15T22:20:14.987650+0000","last_undegraded":"2021-03-15T22:20:14.987650+0000","last_fullsized":"2021-03-15T22:20:14.987650+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2756315,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":1524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"51","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.174703+0000","last_change":"2021-03-15T22:19:25.878071+0000","last_active":"2021-03-15T22:20:14.174703+0000","last_peered":"2021-03-15T22:20:14.174703+0000","last_clean":"2021-03-15T22:20:14.174703+0000","last_became_active":"2021-03-15T22:18:00.287019+0000","last_became_peered":"2021-03-15T22:18:00.287019+0000","last_unstale":"2021-03-15T22:20:14.174703+0000","last_undegraded":"2021-03-15T22:20:14.174703+0000","last_fullsized":"2021-03-15T22:20:14.174703+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.878008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"54","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.207544+0000","last_change":"2021-03-15T22:19:59.244041+0000","last_active":"2021-03-15T22:20:10.207544+0000","last_peered":"2021-03-15T22:20:10.207544+0000","last_clean":"2021-03-15T22:20:10.207544+0000","last_became_active":"2021-03-15T22:17:58.008735+0000","last_became_peered":"2021-03-15T22:17:58.008735+0000","last_unstale":"2021-03-15T22:20:10.207544+0000","last_undegraded":"2021-03-15T22:20:10.207544+0000","last_fullsized":"2021-03-15T22:20:10.207544+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:59.243939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:59.243939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"58","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.989661+0000","last_change":"2021-03-15T22:19:55.098323+0000","last_active":"2021-03-15T22:20:14.989661+0000","last_peered":"2021-03-15T22:20:14.989661+0000","last_clean":"2021-03-15T22:20:14.989661+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:20:14.989661+0000","last_undegraded":"2021-03-15T22:20:14.989661+0000","last_fullsized":"2021-03-15T22:20:14.989661+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:19:55.098239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"83","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990659+0000","last_change":"2021-03-15T22:20:01.157941+0000","last_active":"2021-03-15T22:20:14.990659+0000","last_peered":"2021-03-15T22:20:14.990659+0000","last_clean":"2021-03-15T22:20:14.990659+0000","last_became_active":"2021-03-15T22:17:55.992213+0000","last_became_peered":"2021-03-15T22:17:55.992213+0000","last_unstale":"2021-03-15T22:20:14.990659+0000","last_undegraded":"2021-03-15T22:20:14.990659+0000","last_fullsized":"2021-03-15T22:20:14.990659+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:20:01.157856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:20:01.157856+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"97'3","reported_seq":"20","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716318+0000","last_change":"2021-03-15T22:20:05.033922+0000","last_active":"2021-03-15T22:20:13.716318+0000","last_peered":"2021-03-15T22:20:13.716318+0000","last_clean":"2021-03-15T22:20:13.716318+0000","last_became_active":"2021-03-15T22:20:05.033636+0000","last_became_peered":"2021-03-15T22:20:05.033636+0000","last_unstale":"2021-03-15T22:20:13.716318+0000","last_undegraded":"2021-03-15T22:20:13.716318+0000","last_fullsized":"2021-03-15T22:20:13.716318+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2386561,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":4,"num_write_kb":926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"50","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.716245+0000","last_change":"2021-03-15T22:19:22.172668+0000","last_active":"2021-03-15T22:20:13.716245+0000","last_peered":"2021-03-15T22:20:13.716245+0000","last_clean":"2021-03-15T22:20:13.716245+0000","last_became_active":"2021-03-15T22:18:00.019205+0000","last_became_peered":"2021-03-15T22:18:00.019205+0000","last_unstale":"2021-03-15T22:20:13.716245+0000","last_undegraded":"2021-03-15T22:20:13.716245+0000","last_fullsized":"2021-03-15T22:20:13.716245+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.172605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"49","reported_epoch":"96","state":"active+clean","last_fresh":"2021-03-15T22:20:10.206031+0000","last_change":"2021-03-15T22:19:13.514504+0000","last_active":"2021-03-15T22:20:10.206031+0000","last_peered":"2021-03-15T22:20:10.206031+0000","last_clean":"2021-03-15T22:20:10.206031+0000","last_became_active":"2021-03-15T22:17:58.013512+0000","last_became_peered":"2021-03-15T22:17:58.013512+0000","last_unstale":"2021-03-15T22:20:10.206031+0000","last_undegraded":"2021-03-15T22:20:10.206031+0000","last_fullsized":"2021-03-15T22:20:10.206031+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:13.514438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.514438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"56","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.518865+0000","last_change":"2021-03-15T22:19:18.591404+0000","last_active":"2021-03-15T22:20:14.518865+0000","last_peered":"2021-03-15T22:20:14.518865+0000","last_clean":"2021-03-15T22:20:14.518865+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:20:14.518865+0000","last_undegraded":"2021-03-15T22:20:14.518865+0000","last_fullsized":"2021-03-15T22:20:14.518865+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.591331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.591331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"65'21","reported_seq":"87","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990397+0000","last_change":"2021-03-15T22:17:55.992521+0000","last_active":"2021-03-15T22:20:14.990397+0000","last_peered":"2021-03-15T22:20:14.990397+0000","last_clean":"2021-03-15T22:20:14.990397+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:20:14.990397+0000","last_undegraded":"2021-03-15T22:20:14.990397+0000","last_fullsized":"2021-03-15T22:20:14.990397+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:17:54.969195+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"97'13","reported_seq":"40","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.991498+0000","last_change":"2021-03-15T22:20:14.991498+0000","last_active":"2021-03-15T22:20:14.991498+0000","last_peered":"2021-03-15T22:20:14.991498+0000","last_clean":"2021-03-15T22:20:14.991498+0000","last_became_active":"2021-03-15T22:20:05.037884+0000","last_became_peered":"2021-03-15T22:20:05.037884+0000","last_unstale":"2021-03-15T22:20:14.991498+0000","last_undegraded":"2021-03-15T22:20:14.991498+0000","last_fullsized":"2021-03-15T22:20:14.991498+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9092455,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":4,"num_write":17,"num_write_kb":6576,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"47","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990860+0000","last_change":"2021-03-15T22:18:00.019688+0000","last_active":"2021-03-15T22:20:14.990860+0000","last_peered":"2021-03-15T22:20:14.990860+0000","last_clean":"2021-03-15T22:20:14.990860+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:20:14.990860+0000","last_undegraded":"2021-03-15T22:20:14.990860+0000","last_fullsized":"2021-03-15T22:20:14.990860+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:17:58.999017+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"61'2","reported_seq":"81","reported_epoch":"98","state":"active+clean","last_fresh":"2021-03-15T22:20:13.715997+0000","last_change":"2021-03-15T22:19:12.158924+0000","last_active":"2021-03-15T22:20:13.715997+0000","last_peered":"2021-03-15T22:20:13.715997+0000","last_clean":"2021-03-15T22:20:13.715997+0000","last_became_active":"2021-03-15T22:17:58.010539+0000","last_became_peered":"2021-03-15T22:17:58.010539+0000","last_unstale":"2021-03-15T22:20:13.715997+0000","last_undegraded":"2021-03-15T22:20:13.715997+0000","last_fullsized":"2021-03-15T22:20:13.715997+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:12.158857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:12.158857+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"57","reported_epoch":"99","state":"active+clean","last_fresh":"2021-03-15T22:20:14.176920+0000","last_change":"2021-03-15T22:18:57.650089+0000","last_active":"2021-03-15T22:20:14.176920+0000","last_peered":"2021-03-15T22:20:14.176920+0000","last_clean":"2021-03-15T22:20:14.176920+0000","last_became_active":"2021-03-15T22:17:53.976045+0000","last_became_peered":"2021-03-15T22:17:53.976045+0000","last_unstale":"2021-03-15T22:20:14.176920+0000","last_undegraded":"2021-03-15T22:20:14.176920+0000","last_fullsized":"2021-03-15T22:20:14.176920+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:57.650003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:57.650003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"77","reported_epoch":"100","state":"active+clean","last_fresh":"2021-03-15T22:20:14.990030+0000","last_change":"2021-03-15T22:19:48.154964+0000","last_active":"2021-03-15T22:20:14.990030+0000","last_peered":"2021-03-15T22:20:14.990030+0000","last_clean":"2021-03-15T22:20:14.990030+0000","last_became_active":"2021-03-15T22:17:55.986062+0000","last_became_peered":"2021-03-15T22:17:55.986062+0000","last_unstale":"2021-03-15T22:20:14.990030+0000","last_undegraded":"2021-03-15T22:20:14.990030+0000","last_fullsized":"2021-03-15T22:20:14.990030+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:19:48.154884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:48.154884+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":124459573,"num_objects":54,"num_object_clones":5,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":1,"num_read":117,"num_read_kb":12636,"num_write":225,"num_write_kb":64381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183988224,"data_stored":186015472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":186,"ondisk_log_size":186,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":140,"num_read_kb":135,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1069,"num_read_kb":574,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":564,"ondisk_log_size":564,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299424,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28892,"kb_used_data":23060,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93716260,"statfs":{"total":95995035648,"available":95965450240,"internally_reserved":0,"allocated":23613440,"data_stored":22881267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.33400000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.63600000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.61799999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.47299999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47399999999999998,"15min":0.47799999999999998},"min":{"1min":0.28999999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.56999999999999995,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.624,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.56999999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.502,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.39300000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64900000000000002,"5min":0.746,"15min":0.746},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.60899999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.55900000000000005,"15min":0.56599999999999995},"min":{"1min":0.39200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.63900000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.67100000000000004}]},{"osd":4,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.44,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.64200000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.433,"15min":0.42299999999999999},"min":{"1min":0.35599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.69699999999999995},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.49399999999999999,"15min":0.48999999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.56999999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.499,"15min":0.48999999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.39300000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.68500000000000005}]}]},{"osd":6,"up_from":45,"seq":193273528356,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":19488,"kb_used_data":13656,"kb_used_omap":0,"kb_used_meta":5824,"kb_avail":93725664,"statfs":{"total":95995035648,"available":95975079936,"internally_reserved":0,"allocated":13983744,"data_stored":13275869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5963776},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.48899999999999999,"15min":0.49299999999999999},"min":{"1min":0.30599999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63200000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.496,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.34000000000000002,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.67500000000000004,"5min":0.75700000000000001,"15min":0.75700000000000001},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51100000000000001,"15min":0.51500000000000001},"min":{"1min":0.36099999999999999,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.48199999999999998,"15min":0.47799999999999998},"min":{"1min":0.40400000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.629,"5min":0.69599999999999995,"15min":0.69599999999999995},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.501,"15min":0.497},"min":{"1min":0.35899999999999999,"5min":0.312,"15min":0.312},"max":{"1min":0.68600000000000005,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.441},{"interface":"front","average":{"1min":0.501,"5min":0.502,"15min":0.502},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.624,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.55700000000000005}]},{"osd":3,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.52700000000000002,"15min":0.53200000000000003},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.624,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.53000000000000003,"15min":0.52900000000000003},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.69499999999999995,"5min":0.72399999999999998,"15min":0.72399999999999998},"last":0.48599999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.40000000000000002,"15min":0.39300000000000002},"min":{"1min":0.30199999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.65300000000000002},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.433,"15min":0.42699999999999999},"min":{"1min":0.32100000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.51200000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:19:32 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.441,"15min":0.433},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.76400000000000001,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.498,"5min":0.46500000000000002,"15min":0.45900000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77300000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.65800000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:19:53 2021","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.502,"15min":0.496},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.67000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834574901,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":23720,"kb_used_data":17440,"kb_used_omap":0,"kb_used_meta":6272,"kb_avail":93721432,"statfs":{"total":95995035648,"available":95970746368,"internally_reserved":0,"allocated":17858560,"data_stored":17027849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6422528},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:13 2021","interfaces":[{"interface":"back","average":{"1min":0.38800000000000001,"5min":0.38400000000000001,"15min":0.35099999999999998},"min":{"1min":0.255,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.66400000000000003,"5min":3.387,"15min":3.387},"last":54.427999999999997},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.40300000000000002,"15min":0.377},"min":{"1min":0.23699999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.71899999999999997,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":54.356999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:19:36 2021","interfaces":[{"interface":"back","average":{"1min":0.35699999999999998,"5min":0.42199999999999999,"15min":0.41699999999999998},"min":{"1min":0.223,"5min":0.223,"15min":0.223},"max":{"1min":0.78700000000000003,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.41699999999999998,"15min":0.42099999999999999},"min":{"1min":0.26100000000000001,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64400000000000002,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.221}]},{"osd":3,"last update":"Mon Mar 15 22:19:44 2021","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.434,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54100000000000004,"5min":1.796,"15min":1.796},"last":0.373},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.48299999999999998,"15min":0.495},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.627,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.20999999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:20:01 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.74199999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71599999999999997,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.29999999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:19:16 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.65200000000000002,"15min":0.66900000000000004},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.94199999999999995,"5min":2.153,"15min":2.153},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":1.151,"15min":1.2450000000000001},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.95099999999999996,"5min":15.472,"15min":15.472},"last":0.35199999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:19:36 2021","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.67700000000000005,"15min":0.69399999999999995},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.879,"5min":1.75,"15min":1.75},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.64100000000000001,"15min":0.65300000000000002},"min":{"1min":0.34999999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.87,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.36399999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:19:53 2021","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.94299999999999995,"15min":1.0049999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78200000000000003,"5min":8.8100000000000005,"15min":8.8100000000000005},"last":0.316},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.629,"15min":0.64300000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72999999999999998,"5min":1.325,"15min":1.325},"last":0.38500000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738424,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":33928,"kb_used_data":26772,"kb_used_omap":0,"kb_used_meta":6272,"kb_avail":93711224,"statfs":{"total":95995035648,"available":95960293376,"internally_reserved":0,"allocated":27414528,"data_stored":27075415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6422528},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:19:12 2021","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.372,"15min":0.36799999999999999},"min":{"1min":0.253,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.24299999999999999},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38800000000000001,"15min":0.38200000000000001},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.59299999999999997,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.307}]},{"osd":2,"last update":"Mon Mar 15 22:19:29 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.39700000000000002,"15min":0.40999999999999998},"min":{"1min":0.24199999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.42899999999999999,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.32600000000000001,"5min":0.39200000000000002,"15min":0.40400000000000003},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.442,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.32900000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.42199999999999999,"15min":0.438},"min":{"1min":0.24399999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.433,"15min":0.44700000000000001},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.34399999999999997}]},{"osd":4,"last update":"Mon Mar 15 22:20:05 2021","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.309,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.45500000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.68799999999999994}]},{"osd":5,"last update":"Mon Mar 15 22:19:18 2021","interfaces":[{"interface":"back","average":{"1min":0.57799999999999996,"5min":0.61399999999999999,"15min":0.62},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.79000000000000004,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.57799999999999996,"15min":0.58899999999999997},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70199999999999996,"5min":0.871,"15min":0.871},"last":0.69599999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:19:31 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.57699999999999996,"15min":0.58099999999999996},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.63400000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"min":{"1min":0.433,"5min":0.433,"15min":0.433},"max":{"1min":1.0049999999999999,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.71899999999999997}]},{"osd":7,"last update":"Mon Mar 15 22:19:57 2021","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.44600000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.748},{"interface":"front","average":{"1min":0.625,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.58399999999999996}]}]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27540,"kb_used_data":20952,"kb_used_omap":0,"kb_used_meta":6016,"kb_avail":93717612,"statfs":{"total":95995035648,"available":95966834688,"internally_reserved":0,"allocated":21454848,"data_stored":21998709,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6160384},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.35399999999999998,"15min":0.35999999999999999},"min":{"1min":0.20799999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.68000000000000005,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.223},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.377,"15min":0.38300000000000001},"min":{"1min":0.245,"5min":0.191,"15min":0.191},"max":{"1min":0.53600000000000003,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":0.251}]},{"osd":1,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.38400000000000001,"15min":0.38800000000000001},"min":{"1min":0.27300000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.55200000000000005,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.438,"15min":0.45200000000000001},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.50700000000000001,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.33100000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:19:45 2021","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.41299999999999998,"15min":0.443},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54300000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.47499999999999998,"15min":0.51300000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59099999999999997,"5min":0.874,"15min":0.874},"last":0.35799999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:19:59 2021","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56799999999999995,"15min":0.58199999999999996},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86499999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.439},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55000000000000004,"15min":0.56499999999999995},"min":{"1min":0.39400000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82299999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.371}]},{"osd":5,"last update":"Mon Mar 15 22:19:19 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.57299999999999995,"15min":0.58299999999999996},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69799999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.55500000000000005,"15min":0.56200000000000006},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.78000000000000003,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.40300000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.57199999999999995,"5min":0.56200000000000006,"15min":0.56000000000000005},"min":{"1min":0.40300000000000002,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.56200000000000006,"15min":0.56599999999999995},"min":{"1min":0.39300000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.77300000000000002,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":0.51600000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:19:54 2021","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.42399999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.69299999999999995,"5min":0.873,"15min":0.873},"last":0.498},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.73399999999999999,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.45800000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149742,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":56968,"kb_used_data":46528,"kb_used_omap":0,"kb_used_meta":10432,"kb_avail":93688184,"statfs":{"total":95995035648,"available":95936700416,"internally_reserved":0,"allocated":47644672,"data_stored":46858201,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":10682368},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:52 2021","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.34699999999999998,"15min":0.36599999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.371,"15min":0.39200000000000002},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.49399999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.27900000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:19:52 2021","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.41099999999999998,"15min":0.43099999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.40600000000000003,"15min":0.43099999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61899999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.34899999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:19:52 2021","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.40799999999999997,"15min":0.434},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.505,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.38100000000000001,"15min":0.39900000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.435,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.441}]},{"osd":4,"last update":"Mon Mar 15 22:20:02 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53400000000000003,"15min":0.51800000000000002},"min":{"1min":0.35799999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.379}]},{"osd":5,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.51900000000000002,"15min":0.52100000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.63800000000000001,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55600000000000005,"15min":0.55900000000000005},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.73899999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.38600000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:19:33 2021","interfaces":[{"interface":"back","average":{"1min":0.60599999999999998,"5min":0.59799999999999998,"15min":0.59699999999999998},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.871,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58499999999999996,"15min":0.58999999999999997},"min":{"1min":0.371,"5min":0.371,"15min":0.371},"max":{"1min":0.86699999999999999,"5min":0.88,"15min":0.88},"last":0.501}]},{"osd":7,"last update":"Mon Mar 15 22:19:52 2021","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84099999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38900000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.40600000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986219,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":22956,"kb_used_data":16868,"kb_used_omap":0,"kb_used_meta":6080,"kb_avail":93722196,"statfs":{"total":95995035648,"available":95971528704,"internally_reserved":0,"allocated":17272832,"data_stored":17955990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6225920},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:20:05 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.505,"15min":0.52100000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.70999999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.5},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.72199999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:20:05 2021","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.67300000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.54100000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.53000000000000003},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.65100000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:20:05 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.625},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51100000000000001,"15min":0.51300000000000001},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.65700000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:20:05 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55400000000000005,"15min":0.56599999999999995},"min":{"1min":0.40100000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.92000000000000004,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.75},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56399999999999995,"15min":0.58399999999999996},"min":{"1min":0.36599999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.90300000000000002,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67200000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:19:19 2021","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.38800000000000001,"15min":0.38600000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.64500000000000002,"5min":0.64500000000000002,"15min":0.64500000000000002},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.40699999999999997,"15min":0.40200000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66600000000000004,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.432}]},{"osd":6,"last update":"Mon Mar 15 22:19:34 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.41899999999999998,"15min":0.41299999999999998},"min":{"1min":0.315,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.496,"15min":0.48999999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.84799999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.52900000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:19:49 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.47699999999999998,"15min":0.47299999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.56699999999999995}]}]},{"osd":5,"up_from":38,"seq":163208757287,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":7452,"kb_used_data":1748,"kb_used_omap":0,"kb_used_meta":5696,"kb_avail":93737700,"statfs":{"total":95995035648,"available":95987404800,"internally_reserved":0,"allocated":1789952,"data_stored":1277138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":5832704},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.47799999999999998,"15min":0.47899999999999998},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.59299999999999997,"5min":0.66200000000000003,"15min":0.66200000000000003},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.69399999999999995,"5min":0.755,"15min":0.755},"last":0.54000000000000004}]},{"osd":1,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.31900000000000001,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.73399999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.375},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.83999999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.40600000000000003}]},{"osd":2,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.505,"15min":0.50700000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.68600000000000005,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.52400000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.69999999999999996,"5min":0.745,"15min":0.745},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.52200000000000002,"15min":0.52400000000000002},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.44900000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:19:17 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.376,"15min":0.36399999999999999},"min":{"1min":0.30099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.40899999999999997,"15min":0.39700000000000002},"min":{"1min":0.314,"5min":0.25,"15min":0.25},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.61099999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:19:35 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.45100000000000001,"15min":0.44900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.64000000000000001,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.40400000000000003,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.70199999999999996,"5min":0.72899999999999998,"15min":0.72899999999999998},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:19:53 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.501,"15min":0.496},"min":{"1min":0.376,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.623},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72499999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.60099999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":12926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":5792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":26353664,"data_stored":26604265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":19615744,"data_stored":19477245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":20471808,"data_stored":21538342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":46485504,"data_stored":46384471,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16142336,"data_stored":17475510,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":19443712,"data_stored":19303515,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12894208,"data_stored":12804932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":22581248,"data_stored":22427192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:20:18.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool set default.rgw.meta pgp_num 32 2021-03-15T22:20:18.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:18.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:18.753 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:18.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:18.855 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:18.879 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:18.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:18.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:18.982 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:19.004 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:19.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:19.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:19.106 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:19.128 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:19.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:19.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:19.189 INFO:tasks.rados.rados.0.smithi076.stdout:73: done (3 left) 2021-03-15T22:20:19.190 INFO:tasks.rados.rados.0.smithi076.stdout:74: done (2 left) 2021-03-15T22:20:19.190 INFO:tasks.rados.rados.0.smithi076.stdout:75: done (1 left) 2021-03-15T22:20:19.190 INFO:tasks.rados.rados.0.smithi076.stdout:76: done (0 left) 2021-03-15T22:20:19.191 INFO:tasks.rados.rados.0.smithi076.stdout:77: copy_from oid 22 from oid 49 current snap is 4 2021-03-15T22:20:19.191 INFO:tasks.rados.rados.0.smithi076.stdout:78: read oid 39 snap 3 2021-03-15T22:20:19.191 INFO:tasks.rados.rados.0.smithi076.stdout:78: expect (ObjNum 38 snap 0 seq_num 38) 2021-03-15T22:20:19.191 INFO:tasks.rados.rados.0.smithi076.stdout:79: write oid 30 current snap is 4 2021-03-15T22:20:19.192 INFO:tasks.rados.rados.0.smithi076.stdout:79: seq_num 58 ranges {574806=778913,2061132=557342} 2021-03-15T22:20:19.196 INFO:tasks.rados.rados.0.smithi076.stdout:79: writing smithi07694878-30 from 574806 to 1353719 tid 1 2021-03-15T22:20:19.203 INFO:tasks.rados.rados.0.smithi076.stdout:79: writing smithi07694878-30 from 2061132 to 2618474 tid 2 2021-03-15T22:20:19.204 INFO:tasks.rados.rados.0.smithi076.stdout:80: write (excl) oid 29 current snap is 4 2021-03-15T22:20:19.204 INFO:tasks.rados.rados.0.smithi076.stdout:80: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2021-03-15T22:20:19.209 INFO:tasks.rados.rados.0.smithi076.stdout:80: writing smithi07694878-29 from 591613 to 1045775 tid 1 2021-03-15T22:20:19.216 INFO:tasks.rados.rados.0.smithi076.stdout:80: writing smithi07694878-29 from 1719614 to 2136967 tid 2 2021-03-15T22:20:19.221 INFO:tasks.rados.rados.0.smithi076.stdout:80: writing smithi07694878-29 from 2570780 to 2991613 tid 3 2021-03-15T22:20:19.221 INFO:tasks.rados.rados.0.smithi076.stdout:81: read oid 36 snap 3 2021-03-15T22:20:19.222 INFO:tasks.rados.rados.0.smithi076.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2021-03-15T22:20:19.222 INFO:tasks.rados.rados.0.smithi076.stdout:82: copy_from oid 42 from oid 28 current snap is 4 2021-03-15T22:20:19.222 INFO:tasks.rados.rados.0.smithi076.stdout:78: expect (ObjNum 38 snap 0 seq_num 38) 2021-03-15T22:20:19.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:19.256 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:19.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:19.258 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:19.307 INFO:tasks.rados.rados.0.smithi076.stdout:77: finishing copy_from to smithi07694878-22 2021-03-15T22:20:19.307 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 21 (ObjNum 48 snap 0 seq_num 48) dirty exists 2021-03-15T22:20:19.307 INFO:tasks.rados.rados.0.smithi076.stdout:77: finishing copy_from racing read to smithi07694878-22 2021-03-15T22:20:19.307 INFO:tasks.rados.rados.0.smithi076.stdout:77: done (5 left) 2021-03-15T22:20:19.308 INFO:tasks.rados.rados.0.smithi076.stdout:78: done (4 left) 2021-03-15T22:20:19.308 INFO:tasks.rados.rados.0.smithi076.stdout:83: rollback oid 4 current snap is 4 2021-03-15T22:20:19.308 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 4 to 3 2021-03-15T22:20:19.308 INFO:tasks.rados.rados.0.smithi076.stdout:79: finishing write tid 1 to smithi07694878-30 2021-03-15T22:20:19.309 INFO:tasks.rados.rados.0.smithi076.stdout:79: finishing write tid 2 to smithi07694878-30 2021-03-15T22:20:19.309 INFO:tasks.rados.rados.0.smithi076.stdout:79: finishing write tid 3 to smithi07694878-30 2021-03-15T22:20:19.309 INFO:tasks.rados.rados.0.smithi076.stdout:79: finishing write tid 4 to smithi07694878-30 2021-03-15T22:20:19.309 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 36 (ObjNum 58 snap 4 seq_num 58) dirty exists 2021-03-15T22:20:19.309 INFO:tasks.rados.rados.0.smithi076.stdout:79: left oid 30 (ObjNum 58 snap 4 seq_num 58) 2021-03-15T22:20:19.310 INFO:tasks.rados.rados.0.smithi076.stdout:80: finishing write tid 1 to smithi07694878-29 2021-03-15T22:20:19.310 INFO:tasks.rados.rados.0.smithi076.stdout:80: finishing write tid 2 to smithi07694878-29 2021-03-15T22:20:19.310 INFO:tasks.rados.rados.0.smithi076.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2021-03-15T22:20:19.359 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:19.386 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:19.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:19.388 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:19.395 INFO:tasks.rados.rados.0.smithi076.stdout:80: finishing write tid 3 to smithi07694878-29 2021-03-15T22:20:19.396 INFO:tasks.rados.rados.0.smithi076.stdout:80: finishing write tid 4 to smithi07694878-29 2021-03-15T22:20:19.396 INFO:tasks.rados.rados.0.smithi076.stdout:80: finishing write tid 5 to smithi07694878-29 2021-03-15T22:20:19.396 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 26 (ObjNum 59 snap 4 seq_num 59) dirty exists 2021-03-15T22:20:19.396 INFO:tasks.rados.rados.0.smithi076.stdout:80: left oid 29 (ObjNum 59 snap 4 seq_num 59) 2021-03-15T22:20:19.397 INFO:tasks.rados.rados.0.smithi076.stdout:82: finishing copy_from to smithi07694878-42 2021-03-15T22:20:19.397 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2021-03-15T22:20:19.397 INFO:tasks.rados.rados.0.smithi076.stdout:82: finishing copy_from racing read to smithi07694878-42 2021-03-15T22:20:19.397 INFO:tasks.rados.rados.0.smithi076.stdout:79: done (4 left) 2021-03-15T22:20:19.398 INFO:tasks.rados.rados.0.smithi076.stdout:80: done (3 left) 2021-03-15T22:20:19.398 INFO:tasks.rados.rados.0.smithi076.stdout:81: done (2 left) 2021-03-15T22:20:19.398 INFO:tasks.rados.rados.0.smithi076.stdout:82: done (1 left) 2021-03-15T22:20:19.398 INFO:tasks.rados.rados.0.smithi076.stdout:84: read oid 14 snap -1 2021-03-15T22:20:19.398 INFO:tasks.rados.rados.0.smithi076.stdout:84: expect (ObjNum 13 snap 0 seq_num 13) 2021-03-15T22:20:19.399 INFO:tasks.rados.rados.0.smithi076.stdout:85: read oid 29 snap -1 2021-03-15T22:20:19.399 INFO:tasks.rados.rados.0.smithi076.stdout:85: expect (ObjNum 59 snap 4 seq_num 59) 2021-03-15T22:20:19.399 INFO:tasks.rados.rados.0.smithi076.stdout:86: read oid 6 snap 3 2021-03-15T22:20:19.399 INFO:tasks.rados.rados.0.smithi076.stdout:86: expect (ObjNum 5 snap 0 seq_num 5) 2021-03-15T22:20:19.400 INFO:tasks.rados.rados.0.smithi076.stdout:87: read oid 24 snap -1 2021-03-15T22:20:19.400 INFO:tasks.rados.rados.0.smithi076.stdout:87: expect (ObjNum 23 snap 0 seq_num 23) 2021-03-15T22:20:19.400 INFO:tasks.rados.rados.0.smithi076.stdout:88: copy_from oid 21 from oid 26 current snap is 4 2021-03-15T22:20:19.400 INFO:tasks.rados.rados.0.smithi076.stdout:83: finishing rollback tid 0 to smithi07694878-4 2021-03-15T22:20:19.400 INFO:tasks.rados.rados.0.smithi076.stdout:83: finishing rollback tid 1 to smithi07694878-4 2021-03-15T22:20:19.401 INFO:tasks.rados.rados.0.smithi076.stdout:83: finishing rollback tid 2 to smithi07694878-4 2021-03-15T22:20:19.401 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 17 (ObjNum 3 snap 0 seq_num 3) dirty exists 2021-03-15T22:20:19.401 INFO:tasks.rados.rados.0.smithi076.stdout:83: done (5 left) 2021-03-15T22:20:19.401 INFO:tasks.rados.rados.0.smithi076.stdout:89: read oid 2 snap -1 2021-03-15T22:20:19.401 INFO:tasks.rados.rados.0.smithi076.stdout:89: expect deleted 2021-03-15T22:20:19.402 INFO:tasks.rados.rados.0.smithi076.stdout:90: read oid 37 snap -1 2021-03-15T22:20:19.402 INFO:tasks.rados.rados.0.smithi076.stdout:90: expect (ObjNum 36 snap 0 seq_num 36) 2021-03-15T22:20:19.402 INFO:tasks.rados.rados.0.smithi076.stdout:91: write oid 38 current snap is 4 2021-03-15T22:20:19.402 INFO:tasks.rados.rados.0.smithi076.stdout:91: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2021-03-15T22:20:19.403 INFO:tasks.rados.rados.0.smithi076.stdout:91: writing smithi07694878-38 from 608420 to 1137831 tid 1 2021-03-15T22:20:19.410 INFO:tasks.rados.rados.0.smithi076.stdout:91: writing smithi07694878-38 from 1778096 to 2339107 tid 2 2021-03-15T22:20:19.414 INFO:tasks.rados.rados.0.smithi076.stdout:91: writing smithi07694878-38 from 2881850 to 3008420 tid 3 2021-03-15T22:20:19.415 INFO:tasks.rados.rados.0.smithi076.stdout:92: read oid 17 snap -1 2021-03-15T22:20:19.415 INFO:tasks.rados.rados.0.smithi076.stdout:92: expect (ObjNum 16 snap 0 seq_num 16) 2021-03-15T22:20:19.415 INFO:tasks.rados.rados.0.smithi076.stdout:93: write (excl) oid 32 current snap is 4 2021-03-15T22:20:19.415 INFO:tasks.rados.rados.0.smithi076.stdout:93: seq_num 61 ranges {625227=721013,1952931=704669} 2021-03-15T22:20:19.422 INFO:tasks.rados.rados.0.smithi076.stdout:93: writing smithi07694878-32 from 625227 to 1346240 tid 1 2021-03-15T22:20:19.429 INFO:tasks.rados.rados.0.smithi076.stdout:93: writing smithi07694878-32 from 1952931 to 2657600 tid 2 2021-03-15T22:20:19.430 INFO:tasks.rados.rados.0.smithi076.stdout:94: write oid 45 current snap is 4 2021-03-15T22:20:19.430 INFO:tasks.rados.rados.0.smithi076.stdout:94: seq_num 62 ranges {642034=796262,1895060=564680} 2021-03-15T22:20:19.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:19 smithi076 conmon[62156]: cluster 2021-03-15T22:20:19.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:19 smithi076 conmon[62156]: 2021-03-15T22:20:19.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:19 smithi076 conmon[62156]: 2021 2021-03-15T22:20:19.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:19 smithi076 conmon[62156]: -03-15T22:20:18.979482+0000 2021-03-15T22:20:19.435 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:19 smithi076 conmon[62156]: mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v282: 148 pgs: 148 active+clean; 119 MiB data, 240 MiB used, 715 GiB / 715 GiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 65 op/s 2021-03-15T22:20:19.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:19 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:20:19.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:19 smithi076 conmon[56514]: :20:18.979482+0000 mgr.y (mgr.14144) 372 : cluster [DBG] pgmap v282: 148 pgs: 148 active+clean; 119 MiB data, 240 MiB used, 715 GiB / 715 GiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 65 op/s 2021-03-15T22:20:19.438 INFO:tasks.rados.rados.0.smithi076.stdout:94: writing smithi07694878-45 from 642034 to 1438296 tid 1 2021-03-15T22:20:19.444 INFO:tasks.rados.rados.0.smithi076.stdout:94: writing smithi07694878-45 from 1895060 to 2459740 tid 2 2021-03-15T22:20:19.444 INFO:tasks.rados.rados.0.smithi076.stdout:95: read oid 34 snap -1 2021-03-15T22:20:19.445 INFO:tasks.rados.rados.0.smithi076.stdout:95: expect deleted 2021-03-15T22:20:19.445 INFO:tasks.rados.rados.0.smithi076.stdout:96: copy_from oid 30 from oid 43 current snap is 4 2021-03-15T22:20:19.445 INFO:tasks.rados.rados.0.smithi076.stdout:89: done (12 left) 2021-03-15T22:20:19.445 INFO:tasks.rados.rados.0.smithi076.stdout:97: copy_from oid 44 from oid 13 current snap is 4 2021-03-15T22:20:19.445 INFO:tasks.rados.rados.0.smithi076.stdout:84: expect (ObjNum 13 snap 0 seq_num 13) 2021-03-15T22:20:19.489 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:19.519 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:19.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:19.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:19.525 INFO:tasks.rados.rados.0.smithi076.stdout:87: expect (ObjNum 23 snap 0 seq_num 23) 2021-03-15T22:20:19.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:19 smithi125 conmon[56739]: cluster 2021-03-15T22:20:18.979482+0000 mgr.y (mgr.14144) 372 : cluster [DBG] 2021-03-15T22:20:19.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:19 smithi125 conmon[56739]: pgmap v282: 148 pgs: 148 active+clean; 119 MiB data, 240 MiB used, 715 GiB / 715 GiB avail; 8.2 MiB/s rd, 11 MiB/s wr, 65 op/s 2021-03-15T22:20:19.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:20:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:20:19.606 INFO:tasks.rados.rados.0.smithi076.stdout:86: expect (ObjNum 5 snap 0 seq_num 5) 2021-03-15T22:20:19.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:19.645 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:19.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:19.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:19.675 INFO:tasks.rados.rados.0.smithi076.stdout:85: expect (ObjNum 59 snap 4 seq_num 59) 2021-03-15T22:20:19.748 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:19.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:19.772 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:19.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:19.809 INFO:tasks.rados.rados.0.smithi076.stdout:91: finishing write tid 1 to smithi07694878-38 2021-03-15T22:20:19.810 INFO:tasks.rados.rados.0.smithi076.stdout:84: done (12 left) 2021-03-15T22:20:19.810 INFO:tasks.rados.rados.0.smithi076.stdout:85: done (11 left) 2021-03-15T22:20:19.811 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:19.812 INFO:tasks.rados.rados.0.smithi076.stdout:86: done (10 left) 2021-03-15T22:20:19.813 INFO:tasks.rados.rados.0.smithi076.stdout:87: done (9 left) 2021-03-15T22:20:19.813 INFO:tasks.rados.rados.0.smithi076.stdout:98: read oid 46 snap -1 2021-03-15T22:20:19.813 INFO:tasks.rados.rados.0.smithi076.stdout:98: expect (ObjNum 45 snap 0 seq_num 45) 2021-03-15T22:20:19.814 INFO:tasks.rados.rados.0.smithi076.stdout:99: rollback oid 36 current snap is 4 2021-03-15T22:20:19.814 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 36 to 3 2021-03-15T22:20:19.814 INFO:tasks.rados.rados.0.smithi076.stdout:91: finishing write tid 2 to smithi07694878-38 2021-03-15T22:20:19.814 INFO:tasks.rados.rados.0.smithi076.stdout:91: finishing write tid 3 to smithi07694878-38 2021-03-15T22:20:19.815 INFO:tasks.rados.rados.0.smithi076.stdout:91: finishing write tid 4 to smithi07694878-38 2021-03-15T22:20:19.815 INFO:tasks.rados.rados.0.smithi076.stdout:91: finishing write tid 5 to smithi07694878-38 2021-03-15T22:20:19.815 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 24 (ObjNum 60 snap 4 seq_num 60) dirty exists 2021-03-15T22:20:19.815 INFO:tasks.rados.rados.0.smithi076.stdout:91: left oid 38 (ObjNum 60 snap 4 seq_num 60) 2021-03-15T22:20:19.815 INFO:tasks.rados.rados.0.smithi076.stdout:90: expect (ObjNum 36 snap 0 seq_num 36) 2021-03-15T22:20:19.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:19.894 INFO:tasks.rados.rados.0.smithi076.stdout:88: finishing copy_from to smithi07694878-21 2021-03-15T22:20:19.895 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-03-15T22:20:19.895 INFO:tasks.rados.rados.0.smithi076.stdout:88: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:20:19.895 INFO:tasks.rados.rados.0.smithi076.stdout:92: expect (ObjNum 16 snap 0 seq_num 16) 2021-03-15T22:20:19.898 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:19.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:19.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:19.971 INFO:tasks.rados.rados.0.smithi076.stdout:93: finishing write tid 1 to smithi07694878-32 2021-03-15T22:20:19.972 INFO:tasks.rados.rados.0.smithi076.stdout:93: finishing write tid 2 to smithi07694878-32 2021-03-15T22:20:19.972 INFO:tasks.rados.rados.0.smithi076.stdout:88: done (10 left) 2021-03-15T22:20:19.972 INFO:tasks.rados.rados.0.smithi076.stdout:90: done (9 left) 2021-03-15T22:20:19.973 INFO:tasks.rados.rados.0.smithi076.stdout:91: done (8 left) 2021-03-15T22:20:19.973 INFO:tasks.rados.rados.0.smithi076.stdout:92: done (7 left) 2021-03-15T22:20:19.973 INFO:tasks.rados.rados.0.smithi076.stdout:100: snap_create 2021-03-15T22:20:19.974 INFO:tasks.rados.rados.0.smithi076.stdout:93: finishing write tid 3 to smithi07694878-32 2021-03-15T22:20:19.974 INFO:tasks.rados.rados.0.smithi076.stdout:93: finishing write tid 4 to smithi07694878-32 2021-03-15T22:20:19.974 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 34 (ObjNum 61 snap 4 seq_num 61) dirty exists 2021-03-15T22:20:19.975 INFO:tasks.rados.rados.0.smithi076.stdout:93: left oid 32 (ObjNum 61 snap 4 seq_num 61) 2021-03-15T22:20:19.975 INFO:tasks.rados.rados.0.smithi076.stdout:94: finishing write tid 1 to smithi07694878-45 2021-03-15T22:20:19.975 INFO:tasks.rados.rados.0.smithi076.stdout:94: finishing write tid 2 to smithi07694878-45 2021-03-15T22:20:19.976 INFO:tasks.rados.rados.0.smithi076.stdout:94: finishing write tid 3 to smithi07694878-45 2021-03-15T22:20:19.976 INFO:tasks.rados.rados.0.smithi076.stdout:94: finishing write tid 4 to smithi07694878-45 2021-03-15T22:20:19.976 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 10 (ObjNum 62 snap 4 seq_num 62) dirty exists 2021-03-15T22:20:19.977 INFO:tasks.rados.rados.0.smithi076.stdout:94: left oid 45 (ObjNum 62 snap 4 seq_num 62) 2021-03-15T22:20:19.977 INFO:tasks.rados.rados.0.smithi076.stdout:96: finishing copy_from to smithi07694878-30 2021-03-15T22:20:19.977 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2021-03-15T22:20:19.978 INFO:tasks.rados.rados.0.smithi076.stdout:96: finishing copy_from racing read to smithi07694878-30 2021-03-15T22:20:19.978 INFO:tasks.rados.rados.0.smithi076.stdout:97: finishing copy_from to smithi07694878-44 2021-03-15T22:20:19.979 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 3 (ObjNum 17 snap 0 seq_num 17) dirty exists 2021-03-15T22:20:19.979 INFO:tasks.rados.rados.0.smithi076.stdout:97: finishing copy_from racing read to smithi07694878-44 2021-03-15T22:20:19.979 INFO:tasks.rados.rados.0.smithi076.stdout:99: finishing rollback tid 0 to smithi07694878-36 2021-03-15T22:20:19.979 INFO:tasks.rados.rados.0.smithi076.stdout:99: finishing rollback tid 1 to smithi07694878-36 2021-03-15T22:20:19.980 INFO:tasks.rados.rados.0.smithi076.stdout:99: finishing rollback tid 2 to smithi07694878-36 2021-03-15T22:20:19.980 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 21 (ObjNum 35 snap 0 seq_num 35) dirty exists 2021-03-15T22:20:19.980 INFO:tasks.rados.rados.0.smithi076.stdout:98: expect (ObjNum 45 snap 0 seq_num 45) 2021-03-15T22:20:19.987 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:20:19.987 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:20:19.988 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:20:19.988 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:20:20.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:20.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:20.030 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:20.030 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:20.063 INFO:tasks.rados.rados.0.smithi076.stdout:93: done (7 left) 2021-03-15T22:20:20.063 INFO:tasks.rados.rados.0.smithi076.stdout:94: done (6 left) 2021-03-15T22:20:20.063 INFO:tasks.rados.rados.0.smithi076.stdout:95: done (5 left) 2021-03-15T22:20:20.064 INFO:tasks.rados.rados.0.smithi076.stdout:96: done (4 left) 2021-03-15T22:20:20.064 INFO:tasks.rados.rados.0.smithi076.stdout:97: done (3 left) 2021-03-15T22:20:20.064 INFO:tasks.rados.rados.0.smithi076.stdout:98: done (2 left) 2021-03-15T22:20:20.064 INFO:tasks.rados.rados.0.smithi076.stdout:99: done (1 left) 2021-03-15T22:20:20.064 INFO:tasks.rados.rados.0.smithi076.stdout:100: done (0 left) 2021-03-15T22:20:20.065 INFO:tasks.rados.rados.0.smithi076.stdout:101: rollback oid 10 current snap is 5 2021-03-15T22:20:20.065 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 10 to 4 2021-03-15T22:20:20.065 INFO:tasks.rados.rados.0.smithi076.stdout:102: read oid 12 snap -1 2021-03-15T22:20:20.065 INFO:tasks.rados.rados.0.smithi076.stdout:102: expect (ObjNum 55 snap 2 seq_num 55) 2021-03-15T22:20:20.066 INFO:tasks.rados.rados.0.smithi076.stdout:103: snap_create 2021-03-15T22:20:20.066 INFO:tasks.rados.rados.0.smithi076.stdout:102: expect (ObjNum 55 snap 2 seq_num 55) 2021-03-15T22:20:20.131 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:20.154 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:20.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:20.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:20.189 INFO:tasks.rados.rados.0.smithi076.stdout:101: finishing rollback tid 0 to smithi07694878-10 2021-03-15T22:20:20.190 INFO:tasks.rados.rados.0.smithi076.stdout:101: finishing rollback tid 1 to smithi07694878-10 2021-03-15T22:20:20.190 INFO:tasks.rados.rados.0.smithi076.stdout:101: finishing rollback tid 2 to smithi07694878-10 2021-03-15T22:20:20.190 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 20 (ObjNum 9 snap 0 seq_num 9) dirty exists 2021-03-15T22:20:20.244 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:20.257 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:20.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:20.281 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:20.281 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:20.383 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:20.406 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:20.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:20.407 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:20.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:20.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:20 smithi076 conmon[56514]: cluster 2021-03-15T22:20:19.186343+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2021-03-15T22:20:20.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:20 smithi076 conmon[56514]: audit 2021-03-15T22:20:19.422541+0000 mgr.y (mgr.14144) 373 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:20.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:20 smithi076 conmon[56514]: cluster 2021-03-15T22:20:19.989280+0000 mon.a (mon.0) 624 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2021-03-15T22:20:20.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:20 smithi076 conmon[62156]: cluster 2021-03-15T22:20:19.186343+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2021-03-15T22:20:20.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:20 smithi076 conmon[62156]: audit 2021-03-15T22:20:19.422541+0000 mgr.y (mgr.14144) 373 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:20.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:20 smithi076 conmon[62156]: cluster 2021-03-15T22:20:19.989280+0000 mon.a (mon.0) 624 : cluster [DBG] osdmap e105: 8 total, 8 up, 8 in 2021-03-15T22:20:20.532 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:20.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:20.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:20.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:20 smithi125 conmon[56739]: cluster 2021-03-15T22:20:19.186343+0000 mon.a (mon.0) 623 : cluster [DBG] osdmap e104: 8 total, 8 up, 8 in 2021-03-15T22:20:20.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:20 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:20:20.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:20 smithi125 conmon[56739]: 20:19.422541+0000 mgr.y (mgr.14144) 373 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:20 smithi125 conmon[56739]: cluster 2021-03-15T22:20:19.989280+0000 mon.a (mon.0) 624 : cluster [DBG] 2021-03-15T22:20:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:20 smithi125 conmon[56739]: osdmap e105: 8 total, 8 up, 8 in 2021-03-15T22:20:20.635 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:20.657 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:20.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:20.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:20.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:20.782 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:20.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:20.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:20.884 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:20.910 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:20.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:20.911 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:21.000 INFO:tasks.rados.rados.0.smithi076.stdout:101: done (2 left) 2021-03-15T22:20:21.000 INFO:tasks.rados.rados.0.smithi076.stdout:102: done (1 left) 2021-03-15T22:20:21.000 INFO:tasks.rados.rados.0.smithi076.stdout:103: done (0 left) 2021-03-15T22:20:21.000 INFO:tasks.rados.rados.0.smithi076.stdout:104: write (excl) oid 43 current snap is 6 2021-03-15T22:20:21.001 INFO:tasks.rados.rados.0.smithi076.stdout:104: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2021-03-15T22:20:21.005 INFO:tasks.rados.rados.0.smithi076.stdout:104: writing smithi07694878-43 from 658841 to 1130352 tid 1 2021-03-15T22:20:21.011 INFO:tasks.rados.rados.0.smithi076.stdout:104: writing smithi07694878-43 from 1553542 to 2261880 tid 2 2021-03-15T22:20:21.012 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:21.016 INFO:tasks.rados.rados.0.smithi076.stdout:104: writing smithi07694878-43 from 2964119 to 3058841 tid 3 2021-03-15T22:20:21.017 INFO:tasks.rados.rados.0.smithi076.stdout:104: finishing write tid 1 to smithi07694878-43 2021-03-15T22:20:21.017 INFO:tasks.rados.rados.0.smithi076.stdout:105: copy_from oid 30 from oid 38 current snap is 6 2021-03-15T22:20:21.017 INFO:tasks.rados.rados.0.smithi076.stdout:104: finishing write tid 2 to smithi07694878-43 2021-03-15T22:20:21.017 INFO:tasks.rados.rados.0.smithi076.stdout:106: write oid 12 current snap is 6 2021-03-15T22:20:21.017 INFO:tasks.rados.rados.0.smithi076.stdout:106: seq_num 64 ranges {675648=546760,2012024=568349} 2021-03-15T22:20:21.023 INFO:tasks.rados.rados.0.smithi076.stdout:106: writing smithi07694878-12 from 675648 to 1222408 tid 1 2021-03-15T22:20:21.031 INFO:tasks.rados.rados.0.smithi076.stdout:106: writing smithi07694878-12 from 2012024 to 2580373 tid 2 2021-03-15T22:20:21.031 INFO:tasks.rados.rados.0.smithi076.stdout:107: copy_from oid 25 from oid 32 current snap is 6 2021-03-15T22:20:21.031 INFO:tasks.rados.rados.0.smithi076.stdout:104: finishing write tid 3 to smithi07694878-43 2021-03-15T22:20:21.031 INFO:tasks.rados.rados.0.smithi076.stdout:104: finishing write tid 4 to smithi07694878-43 2021-03-15T22:20:21.032 INFO:tasks.rados.rados.0.smithi076.stdout:104: finishing write tid 5 to smithi07694878-43 2021-03-15T22:20:21.032 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 14 (ObjNum 63 snap 6 seq_num 63) dirty exists 2021-03-15T22:20:21.032 INFO:tasks.rados.rados.0.smithi076.stdout:104: left oid 43 (ObjNum 63 snap 6 seq_num 63) 2021-03-15T22:20:21.032 INFO:tasks.rados.rados.0.smithi076.stdout:104: done (3 left) 2021-03-15T22:20:21.033 INFO:tasks.rados.rados.0.smithi076.stdout:108: snap_remove snap 4 2021-03-15T22:20:21.038 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:21.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:21.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:21.141 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:21.166 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:21.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:21.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:21.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:21 smithi076 conmon[56514]: cluster 2021-03-15T22:20:20.979906+0000 mgr.y 2021-03-15T22:20:21.935 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:21 smithi076 conmon[56514]: (mgr.14144) 374 : cluster [DBG] pgmap v285: 148 pgs: 148 active+clean; 121 MiB data, 249 MiB used, 715 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2021-03-15T22:20:21.935 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:21 smithi076 conmon[56514]: cluster 2021-03-15T22:20:20.997421+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2021-03-15T22:20:21.936 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:21 smithi076 conmon[62156]: cluster 2021-03-15T22:20:20.979906+0000 mgr.y ( 2021-03-15T22:20:21.936 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:21 smithi076 conmon[62156]: mgr.14144) 374 : cluster [DBG] pgmap v285: 148 pgs: 148 active+clean; 121 MiB data, 249 MiB used, 715 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2021-03-15T22:20:21.936 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:21 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:20:21.936 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:21 smithi076 conmon[62156]: :20:20.997421+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2021-03-15T22:20:21.938 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:20:21.938 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:21.939 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:21 smithi125 conmon[56739]: cluster 2021-03-15T22:20:20.979906+0000 mgr.y (mgr.14144) 374 : cluster 2021-03-15T22:20:21.939 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:21 smithi125 conmon[56739]: [DBG] pgmap v285: 148 pgs: 148 active+clean; 121 MiB data, 249 MiB used, 715 GiB / 715 GiB avail; 9.2 MiB/s rd, 1.1 MiB/s wr, 13 op/s 2021-03-15T22:20:21.939 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:21 smithi125 conmon[56739]: cluster 2021-03-15T22:20:20.997421+0000 mon.a (mon.0) 625 : cluster [DBG] osdmap e106: 8 total, 8 up, 8 in 2021-03-15T22:20:21.963 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:21.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:21.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:22.003 INFO:tasks.rados.rados.0.smithi076.stdout:108: done (3 left) 2021-03-15T22:20:22.004 INFO:tasks.rados.rados.0.smithi076.stdout:109: read oid 8 snap -1 2021-03-15T22:20:22.004 INFO:tasks.rados.rados.0.smithi076.stdout:109: expect (ObjNum 7 snap 0 seq_num 7) 2021-03-15T22:20:22.005 INFO:tasks.rados.rados.0.smithi076.stdout:110: snap_remove snap 3 2021-03-15T22:20:22.066 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:22.090 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:22.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:22.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:22.192 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:22.220 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:22.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:22.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:22.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:22.350 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:22.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:22.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:22.453 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:22.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:22.479 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:22.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:22.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:22.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:22.607 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:22.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:22.709 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:22.735 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:22.736 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:22.837 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:22.863 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:22.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:22.865 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:22.890 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:22.966 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:22.991 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:22.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:22.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:23.008 INFO:tasks.rados.rados.0.smithi076.stdout:110: done (4 left) 2021-03-15T22:20:23.008 INFO:tasks.rados.rados.0.smithi076.stdout:111: delete oid 1 current snap is 6 2021-03-15T22:20:23.009 INFO:tasks.rados.rados.0.smithi076.stdout:105: finishing copy_from to smithi07694878-30 2021-03-15T22:20:23.009 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 24 (ObjNum 60 snap 4 seq_num 60) dirty exists 2021-03-15T22:20:23.009 INFO:tasks.rados.rados.0.smithi076.stdout:105: finishing copy_from racing read to smithi07694878-30 2021-03-15T22:20:23.009 INFO:tasks.rados.rados.0.smithi076.stdout:106: finishing write tid 1 to smithi07694878-12 2021-03-15T22:20:23.010 INFO:tasks.rados.rados.0.smithi076.stdout:106: finishing write tid 2 to smithi07694878-12 2021-03-15T22:20:23.010 INFO:tasks.rados.rados.0.smithi076.stdout:106: finishing write tid 3 to smithi07694878-12 2021-03-15T22:20:23.010 INFO:tasks.rados.rados.0.smithi076.stdout:106: finishing write tid 4 to smithi07694878-12 2021-03-15T22:20:23.010 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 43 (ObjNum 64 snap 6 seq_num 64) dirty exists 2021-03-15T22:20:23.011 INFO:tasks.rados.rados.0.smithi076.stdout:106: left oid 12 (ObjNum 64 snap 6 seq_num 64) 2021-03-15T22:20:23.011 INFO:tasks.rados.rados.0.smithi076.stdout:107: finishing copy_from to smithi07694878-25 2021-03-15T22:20:23.011 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 34 (ObjNum 61 snap 4 seq_num 61) dirty exists 2021-03-15T22:20:23.011 INFO:tasks.rados.rados.0.smithi076.stdout:107: finishing copy_from racing read to smithi07694878-25 2021-03-15T22:20:23.012 INFO:tasks.rados.rados.0.smithi076.stdout:109: expect (ObjNum 7 snap 0 seq_num 7) 2021-03-15T22:20:23.020 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:20:23.088 INFO:tasks.rados.rados.0.smithi076.stdout:105: done (4 left) 2021-03-15T22:20:23.089 INFO:tasks.rados.rados.0.smithi076.stdout:106: done (3 left) 2021-03-15T22:20:23.089 INFO:tasks.rados.rados.0.smithi076.stdout:107: done (2 left) 2021-03-15T22:20:23.089 INFO:tasks.rados.rados.0.smithi076.stdout:109: done (1 left) 2021-03-15T22:20:23.090 INFO:tasks.rados.rados.0.smithi076.stdout:111: done (0 left) 2021-03-15T22:20:23.090 INFO:tasks.rados.rados.0.smithi076.stdout:112: write oid 25 current snap is 6 2021-03-15T22:20:23.090 INFO:tasks.rados.rados.0.smithi076.stdout:112: seq_num 66 ranges {709262=697258,2012635=572018,3046388=62874} 2021-03-15T22:20:23.094 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:23.096 INFO:tasks.rados.rados.0.smithi076.stdout:112: writing smithi07694878-25 from 709262 to 1406520 tid 1 2021-03-15T22:20:23.102 INFO:tasks.rados.rados.0.smithi076.stdout:112: writing smithi07694878-25 from 2012635 to 2584653 tid 2 2021-03-15T22:20:23.106 INFO:tasks.rados.rados.0.smithi076.stdout:112: writing smithi07694878-25 from 3046388 to 3109262 tid 3 2021-03-15T22:20:23.107 INFO:tasks.rados.rados.0.smithi076.stdout:113: rollback oid 3 current snap is 6 2021-03-15T22:20:23.107 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 5 2021-03-15T22:20:23.107 INFO:tasks.rados.rados.0.smithi076.stdout:112: finishing write tid 1 to smithi07694878-25 2021-03-15T22:20:23.107 INFO:tasks.rados.rados.0.smithi076.stdout:114: copy_from oid 32 from oid 35 current snap is 6 2021-03-15T22:20:23.108 INFO:tasks.rados.rados.0.smithi076.stdout:115: write oid 1 current snap is 6 2021-03-15T22:20:23.108 INFO:tasks.rados.rados.0.smithi076.stdout:115: seq_num 67 ranges {726069=772507,2071117=715676} 2021-03-15T22:20:23.114 INFO:tasks.rados.rados.0.smithi076.stdout:115: writing smithi07694878-1 from 726069 to 1498576 tid 1 2021-03-15T22:20:23.121 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:23.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:23.123 INFO:tasks.rados.rados.0.smithi076.stdout:115: writing smithi07694878-1 from 2071117 to 2786793 tid 2 2021-03-15T22:20:23.124 INFO:tasks.rados.rados.0.smithi076.stdout:116: copy_from oid 29 from oid 24 current snap is 6 2021-03-15T22:20:23.124 INFO:tasks.rados.rados.0.smithi076.stdout:112: finishing write tid 2 to smithi07694878-25 2021-03-15T22:20:23.124 INFO:tasks.rados.rados.0.smithi076.stdout:112: finishing write tid 3 to smithi07694878-25 2021-03-15T22:20:23.124 INFO:tasks.rados.rados.0.smithi076.stdout:112: finishing write tid 4 to smithi07694878-25 2021-03-15T22:20:23.125 INFO:tasks.rados.rados.0.smithi076.stdout:113: finishing rollback tid 0 to smithi07694878-3 2021-03-15T22:20:23.125 INFO:tasks.rados.rados.0.smithi076.stdout:113: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:20:23.125 INFO:tasks.rados.rados.0.smithi076.stdout:112: finishing write tid 5 to smithi07694878-25 2021-03-15T22:20:23.125 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 38 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:20:23.125 INFO:tasks.rados.rados.0.smithi076.stdout:112: left oid 25 (ObjNum 66 snap 6 seq_num 66) 2021-03-15T22:20:23.126 INFO:tasks.rados.rados.0.smithi076.stdout:113: finishing rollback tid 2 to smithi07694878-3 2021-03-15T22:20:23.126 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 24 (ObjNum 30 snap 0 seq_num 30) dirty exists 2021-03-15T22:20:23.126 INFO:tasks.rados.rados.0.smithi076.stdout:112: done (4 left) 2021-03-15T22:20:23.127 INFO:tasks.rados.rados.0.smithi076.stdout:113: done (3 left) 2021-03-15T22:20:23.127 INFO:tasks.rados.rados.0.smithi076.stdout:117: write (excl) oid 40 current snap is 6 2021-03-15T22:20:23.127 INFO:tasks.rados.rados.0.smithi076.stdout:117: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2021-03-15T22:20:23.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:23.128 INFO:tasks.rados.rados.0.smithi076.stdout:117: writing smithi07694878-40 from 742876 to 1190632 tid 1 2021-03-15T22:20:23.135 INFO:tasks.rados.rados.0.smithi076.stdout:117: writing smithi07694878-40 from 1729599 to 2305286 tid 2 2021-03-15T22:20:23.140 INFO:tasks.rados.rados.0.smithi076.stdout:117: writing smithi07694878-40 from 3101234 to 3142876 tid 3 2021-03-15T22:20:23.140 INFO:tasks.rados.rados.0.smithi076.stdout:118: snap_create 2021-03-15T22:20:23.140 INFO:tasks.rados.rados.0.smithi076.stdout:115: finishing write tid 1 to smithi07694878-1 2021-03-15T22:20:23.141 INFO:tasks.rados.rados.0.smithi076.stdout:114: finishing copy_from to smithi07694878-32 2021-03-15T22:20:23.141 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 9 (ObjNum 34 snap 0 seq_num 34) dirty exists 2021-03-15T22:20:23.141 INFO:tasks.rados.rados.0.smithi076.stdout:114: finishing copy_from racing read to smithi07694878-32 2021-03-15T22:20:23.141 INFO:tasks.rados.rados.0.smithi076.stdout:115: finishing write tid 2 to smithi07694878-1 2021-03-15T22:20:23.142 INFO:tasks.rados.rados.0.smithi076.stdout:115: finishing write tid 3 to smithi07694878-1 2021-03-15T22:20:23.142 INFO:tasks.rados.rados.0.smithi076.stdout:115: finishing write tid 4 to smithi07694878-1 2021-03-15T22:20:23.145 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 11 (ObjNum 67 snap 6 seq_num 67) dirty exists 2021-03-15T22:20:23.145 INFO:tasks.rados.rados.0.smithi076.stdout:115: left oid 1 (ObjNum 67 snap 6 seq_num 67) 2021-03-15T22:20:23.146 INFO:tasks.rados.rados.0.smithi076.stdout:117: finishing write tid 1 to smithi07694878-40 2021-03-15T22:20:23.147 INFO:tasks.rados.rados.0.smithi076.stdout:116: finishing copy_from to smithi07694878-29 2021-03-15T22:20:23.147 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 3 (ObjNum 23 snap 0 seq_num 23) dirty exists 2021-03-15T22:20:23.147 INFO:tasks.rados.rados.0.smithi076.stdout:116: finishing copy_from racing read to smithi07694878-29 2021-03-15T22:20:23.148 INFO:tasks.rados.rados.0.smithi076.stdout:117: finishing write tid 2 to smithi07694878-40 2021-03-15T22:20:23.148 INFO:tasks.rados.rados.0.smithi076.stdout:117: finishing write tid 3 to smithi07694878-40 2021-03-15T22:20:23.148 INFO:tasks.rados.rados.0.smithi076.stdout:117: finishing write tid 4 to smithi07694878-40 2021-03-15T22:20:23.148 INFO:tasks.rados.rados.0.smithi076.stdout:117: finishing write tid 5 to smithi07694878-40 2021-03-15T22:20:23.148 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 51 (ObjNum 68 snap 6 seq_num 68) dirty exists 2021-03-15T22:20:23.149 INFO:tasks.rados.rados.0.smithi076.stdout:117: left oid 40 (ObjNum 68 snap 6 seq_num 68) 2021-03-15T22:20:23.229 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:23.253 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:23.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:23.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:23.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[56514]: cluster 2021-03-15T22:20:23.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[56514]: 2021-03-15 2021-03-15T22:20:23.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[56514]: T22:20:22.001734+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2021-03-15T22:20:23.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:20:23.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[56514]: 15T22:20:22.980376+0000 mgr.y (mgr.14144 2021-03-15T22:20:23.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[56514]: ) 375 : cluster [DBG] pgmap v288: 148 pgs: 148 active+clean; 125 MiB data, 281 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 4.6 MiB/s wr, 32 op/s 2021-03-15T22:20:23.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[56514]: audit 2021-03-15T22:20:23.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[56514]: 2021-03-15T22:20:22.989529+0000 mon.c (mon.1) 18 : audit [INF] 2021-03-15T22:20:23.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/2739232825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:20:23.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[56514]: audit 2021-03-15T22:20:22.989816 2021-03-15T22:20:23.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[56514]: +0000 mon.a (mon.0) 627 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:20:23.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: cluster 2021-03-15T22:20:23.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: 2021-03-15T22:20:23.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: 2021 2021-03-15T22:20:23.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: - 2021-03-15T22:20:23.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: 03 2021-03-15T22:20:23.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: - 2021-03-15T22:20:23.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: 15 2021-03-15T22:20:23.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: T 2021-03-15T22:20:23.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: 22 2021-03-15T22:20:23.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: : 2021-03-15T22:20:23.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: 20:22.001734+0000 mon.a (mon 2021-03-15T22:20:23.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: .0) 626 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2021-03-15T22:20:23.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:20:23.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: 15T22:20:22.980376+0000 mgr.y (mgr.14144) 2021-03-15T22:20:23.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: 375 : cluster [DBG] pgmap v288: 148 pgs: 148 active+clean; 125 MiB data, 281 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 4.6 MiB/s wr, 32 op/s 2021-03-15T22:20:23.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:20:23.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: 20:22.989529+0000 mon.c (mon.1) 18 2021-03-15T22:20:23.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/2739232825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:20:23.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: audit 2021-03-15T22:20: 2021-03-15T22:20:23.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: 22.989816+0000 mon.a (mon.0) 627 : audit 2021-03-15T22:20:23.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:23 smithi076 conmon[62156]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:20:23.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:23 smithi125 conmon[56739]: cluster 2021-03-15T22:20:22.001734+0000 mon.a (mon.0) 626 : cluster [DBG] osdmap e107: 8 total, 8 up, 8 in 2021-03-15T22:20:23.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:23 smithi125 conmon[56739]: cluster 2021-03-15T22:20:22.980376+0000 mgr.y (mgr 2021-03-15T22:20:23.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:23 smithi125 conmon[56739]: .14144) 375 : cluster [DBG] pgmap v288: 148 pgs: 148 active+clean; 125 MiB data, 281 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 4.6 MiB/s wr, 32 op/s 2021-03-15T22:20:23.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:23 smithi125 conmon[56739]: audit 2021-03-15T22:20:22.989529+0000 mon.c (mon.1) 18 : audit [INF] from='client.? 172.21.15.76:0/2739232825' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:20:23.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:23 smithi125 conmon[56739]: audit 2021-03-15T22:20:22.989816+0000 mon.a (mon.0) 627 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:20:23.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:23.380 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:23.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:23.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:23.483 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:23.507 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:23.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:23.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:23.610 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:23.634 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:23.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:23.636 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:23.737 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:23.761 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:23.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:23.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:23.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:23.887 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:23.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:23.889 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:23.969 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:20:23.986 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:20:23.990 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:24.003 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:20:24.014 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:24.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:24.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:24.023 INFO:tasks.rados.rados.0.smithi076.stdout:114: done (4 left) 2021-03-15T22:20:24.023 INFO:tasks.rados.rados.0.smithi076.stdout:115: done (3 left) 2021-03-15T22:20:24.023 INFO:tasks.rados.rados.0.smithi076.stdout:116: done (2 left) 2021-03-15T22:20:24.023 INFO:tasks.rados.rados.0.smithi076.stdout:117: done (1 left) 2021-03-15T22:20:24.024 INFO:tasks.rados.rados.0.smithi076.stdout:118: done (0 left) 2021-03-15T22:20:24.024 INFO:tasks.rados.rados.0.smithi076.stdout:119: snap_create 2021-03-15T22:20:24.026 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:20:24.027 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 6 pgp_num to 32 2021-03-15T22:20:24.044 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:20:24.062 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:20:24.066 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: cluster 2021-03-15T22:20:22. 2021-03-15T22:20:24.066 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: 996249+0000 mon.a (mon.0) 628 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:24.066 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: audit 2021-03-15T22:20:24.067 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: 2021-03-15T22:20:23.003072+0000 mon.a (mon.0) 2021-03-15T22:20:24.067 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: 629 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:20:24.067 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: cluster 2021-03-15T22:20:24.067 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: 2021-03-15T22:20:23.003159 2021-03-15T22:20:24.067 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: +0000 mon.a (mon.0) 630 : cluster [DBG] 2021-03-15T22:20:24.068 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: osdmap e108: 8 total, 8 up, 8 in 2021-03-15T22:20:24.068 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: audit 2021-03-15T22:20:24.068 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: 2021-03-15T22:20: 2021-03-15T22:20:24.068 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: 23.406242+0000 mon.a (mon.0) 631 : audit 2021-03-15T22:20:24.069 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/436736130' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:20:24.069 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: audit 2021-03-15T22:20 2021-03-15T22:20:24.069 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[62156]: :23.949253+0000 mon.a (mon.0) 632 : audit [DBG] from='client.? 172.21.15.76:0/429731121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:24.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[56514]: cluster 2021-03-15T22:20:22.996249+0000 mon.a (mon.0) 628 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:24.070 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[56514]: audit 2021-03-15T22:20:23.003072+0000 mon.a (mon.0) 629 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:20:24.070 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[56514]: cluster 2021-03-15T22:20:23.003159+0000 mon.a (mon.0) 630 : cluster [DBG] osdmap e108: 8 total, 8 up, 8 in 2021-03-15T22:20:24.070 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[56514]: audit 2021-03-15T22:20:23.406242+0000 mon.a (mon.0) 631 : audit [INF] from='client.? 172.21.15.76:0/436736130' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:20:24.070 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:24 smithi076 conmon[56514]: audit 2021-03-15T22:20:23.949253+0000 mon.a (mon.0) 632 : audit [DBG] from='client.? 172.21.15.76:0/429731121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:24.082 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:20:24.101 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:20:24.102 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:20:24.103 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:20:24.103 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:20:24.103 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:20:24.103 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:20:24.103 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:20:24.104 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:20:24.104 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:20:24.117 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:24.140 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:24.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:24.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:24.243 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:24.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:24.266 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:24.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:24.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: cluster 2021-03-15T22:20:22.996249+0000 mon.a (mon.0) 628 : cluster 2021-03-15T22:20:24.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:24.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: audit 2021-03-15T22:20:23.003072+0000 mon.a (mon.0) 629 : audit [INF] 2021-03-15T22:20:24.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:20:24.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:20:24.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: -03-15T22:20:23.003159+0000 2021-03-15T22:20:24.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: mon.a (mon.0) 630 : cluster 2021-03-15T22:20:24.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: [DBG] osdmap e108: 8 total, 8 up, 8 in 2021-03-15T22:20:24.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: audit 2021 2021-03-15T22:20:24.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: -03-15T22:20: 2021-03-15T22:20:24.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: 23.406242+0000 mon.a (mon.0) 631 2021-03-15T22:20:24.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: : audit [INF] from='client.? 172.21.15.76:0/436736130' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:20:24.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:20:24.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: 03-15T22:20:23 2021-03-15T22:20:24.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: .949253+0000 mon.a (mon.0) 632 : 2021-03-15T22:20:24.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:24 smithi125 conmon[56739]: audit [DBG] from='client.? 172.21.15.76:0/429731121' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:24.368 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:24.389 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:24.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:24.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:24.513 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:24.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:24.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:24.617 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:24.643 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:24.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:24.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:24.746 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:24.772 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:24.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:24.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:24.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:24.901 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:24.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:24.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:24.996 INFO:tasks.rados.rados.0.smithi076.stdout:119: done (0 left) 2021-03-15T22:20:24.996 INFO:tasks.rados.rados.0.smithi076.stdout:120: delete oid 39 current snap is 8 2021-03-15T22:20:24.997 INFO:tasks.rados.rados.0.smithi076.stdout:120: done (0 left) 2021-03-15T22:20:24.998 INFO:tasks.rados.rados.0.smithi076.stdout:121: snap_remove snap 6 2021-03-15T22:20:25.004 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:25.030 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:25.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:25.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:25.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:25.161 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:25.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:25.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:25.264 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:25.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[56514]: cluster 2021-03-15T22:20:23.755254+0000 osd.1 2021-03-15T22:20:25.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[56514]: (osd.1) 25 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:20:25.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[56514]: cluster 2021-03-15T22:20:23.821446+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.11 scrub ok 2021-03-15T22:20:25.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[56514]: cluster 2021-03-15T22:20:23.864818+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.a scrub ok 2021-03-15T22:20:25.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[56514]: audit 2021-03-15T22:20:24.018946+0000 mon.a (mon.0) 633 : audit [INF] from='client.? 172.21.15.76:0/436736130' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:20:25.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[56514]: cluster 2021-03-15T22:20:24.019010+0000 mon.a (mon.0) 634 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2021-03-15T22:20:25.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[56514]: cluster 2021-03-15T22:20:24.980885+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v291: 148 pgs: 148 active+clean; 133 MiB data, 298 MiB used, 715 GiB / 715 GiB avail; 19 MiB/s rd, 5.8 MiB/s wr, 36 op/s 2021-03-15T22:20:25.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[56514]: cluster 2021-03-15T22:20:24.993406+0000 mon.a (mon.0) 635 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2021-03-15T22:20:25.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[62156]: cluster 2021-03-15T22:20:23.755254+0000 osd.1 (osd.1 2021-03-15T22:20:25.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[62156]: ) 25 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:20:25.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[62156]: cluster 2021-03-15T22:20:23.821446+0000 osd.3 (osd.3) 32 : cluster [DBG] 6.11 scrub ok 2021-03-15T22:20:25.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[62156]: cluster 2021-03-15T22:20:23.864818+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.a scrub ok 2021-03-15T22:20:25.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[62156]: audit 2021-03-15T22:20:24.018946+0000 mon.a (mon.0) 633 : audit [INF] from='client.? 172.21.15.76:0/436736130' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:20:25.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[62156]: cluster 2021-03-15T22:20:24.019010+0000 mon.a (mon.0) 634 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2021-03-15T22:20:25.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[62156]: cluster 2021-03-15T22:20:24.980885+0000 mgr.y (mgr.14144) 376 : cluster [DBG] pgmap v291: 148 pgs: 148 active+clean; 133 MiB data, 298 MiB used, 715 GiB / 715 GiB avail; 19 MiB/s rd, 5.8 MiB/s wr, 36 op/s 2021-03-15T22:20:25.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:25 smithi076 conmon[62156]: cluster 2021-03-15T22:20:24.993406+0000 mon.a (mon.0) 635 : cluster [DBG] osdmap e110: 8 total, 8 up, 8 in 2021-03-15T22:20:25.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:25.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: cluster 2021-03-15T22:20:23.755254+0000 osd.1 2021-03-15T22:20:25.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: (osd.1) 25 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:20:25.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:20:25.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: 20:23.821446+0000 osd.3 (osd.3) 32 : cluster 2021-03-15T22:20:25.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: [DBG] 6.11 scrub ok 2021-03-15T22:20:25.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:20:25.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: 03-15T22:20:23.864818 2021-03-15T22:20:25.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: +0000 osd.2 (osd.2) 19 2021-03-15T22:20:25.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: : cluster [DBG] 5.a scrub ok 2021-03-15T22:20:25.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: audit 2021 2021-03-15T22:20:25.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: -03-15T22:20:24 2021-03-15T22:20:25.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: .018946+0000 mon.a 2021-03-15T22:20:25.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: (mon.0) 633 : audit [INF] 2021-03-15T22:20:25.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/436736130' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:20:25.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:20:25.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: -03-15T22:20:24.019010+0000 2021-03-15T22:20:25.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: mon.a (mon.0) 634 : cluster [DBG] osdmap e109: 8 total, 8 up, 8 in 2021-03-15T22:20:25.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: 2021-03-15T22:20:25.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: cluster 2021-03-15T22:20:25.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:20:25.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: 20:24.980885+0000 mgr.y 2021-03-15T22:20:25.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: (mgr.14144) 376 : cluster 2021-03-15T22:20:25.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: [DBG] pgmap v291: 148 pgs: 148 active+clean; 133 MiB data, 298 MiB used, 715 GiB / 715 GiB avail; 19 MiB/s rd, 5.8 MiB/s wr, 36 op/s 2021-03-15T22:20:25.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:20:25.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: 03-15T22:20:24 2021-03-15T22:20:25.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: .993406+0000 mon.a (mon. 2021-03-15T22:20:25.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: 0) 635 : cluster [DBG] 2021-03-15T22:20:25.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:25 smithi125 conmon[56739]: osdmap e110: 8 total, 8 up, 8 in 2021-03-15T22:20:25.294 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:25.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:25.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:25.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:25.421 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:25.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:25.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:25.550 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:25.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:25.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:25.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:25.675 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:25.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:25.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:25.800 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:25.802 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:25.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:25.926 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:25.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:25.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:25.998 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:20:25.998 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:20:26.002 INFO:tasks.rados.rados.0.smithi076.stdout:121: done (0 left) 2021-03-15T22:20:26.002 INFO:tasks.rados.rados.0.smithi076.stdout:122: read oid 14 snap 5 2021-03-15T22:20:26.002 INFO:tasks.rados.rados.0.smithi076.stdout:122: expect (ObjNum 13 snap 0 seq_num 13) 2021-03-15T22:20:26.003 INFO:tasks.rados.rados.0.smithi076.stdout:123: write oid 47 current snap is 8 2021-03-15T22:20:26.003 INFO:tasks.rados.rados.0.smithi076.stdout:123: seq_num 70 ranges {776490=714607,2246563=579356} 2021-03-15T22:20:26.010 INFO:tasks.rados.rados.0.smithi076.stdout:123: writing smithi07694878-47 from 776490 to 1491097 tid 1 2021-03-15T22:20:26.017 INFO:tasks.rados.rados.0.smithi076.stdout:123: writing smithi07694878-47 from 2246563 to 2825919 tid 2 2021-03-15T22:20:26.018 INFO:tasks.rados.rados.0.smithi076.stdout:124: snap_remove snap 7 2021-03-15T22:20:26.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:26.053 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:26.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:26.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:26.077 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:20:26.080 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[56514]: cluster 2021-03-15T22:20:24 2021-03-15T22:20:26.080 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[56514]: .332904+0000 osd.2 (osd.2) 20 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:26.080 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[56514]: cluster 2021-03-15T22:20:24.332973+0000 osd.2 (osd.2) 21 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:26.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[56514]: cluster 2021-03-15T22:20:24.847224+0000 osd.2 (osd.2) 22 : cluster [DBG] 4.f scrub ok 2021-03-15T22:20:26.081 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[56514]: cluster 2021-03-15T22:20:26.000591+0000 mon.a (mon.0) 636 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2021-03-15T22:20:26.081 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[62156]: cluster 2021-03-15T22:20:24.332904+0000 osd.2 (osd 2021-03-15T22:20:26.082 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[62156]: .2) 20 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:26.083 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[62156]: cluster 2021-03-15T22:20:24.332973+0000 osd.2 ( 2021-03-15T22:20:26.084 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[62156]: osd.2) 21 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:26.084 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[62156]: cluster 2021-03-15T22:20:24 2021-03-15T22:20:26.084 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[62156]: .847224+0000 osd.2 (osd.2) 22 : cluster 2021-03-15T22:20:26.084 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[62156]: [DBG] 4.f scrub ok 2021-03-15T22:20:26.085 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[62156]: cluster 2021-03-15T22:20:26.085 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[62156]: 2021-03-15T22:20 2021-03-15T22:20:26.085 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[62156]: :26.000591+0000 mon.a (mon.0) 2021-03-15T22:20:26.085 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:26 smithi076 conmon[62156]: 636 : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2021-03-15T22:20:26.156 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:26.181 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:26.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:26.284 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:26 smithi125 conmon[56739]: cluster 2021-03-15T22:20:24.332904+0000 osd.2 (osd.2) 2021-03-15T22:20:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:26 smithi125 conmon[56739]: 20 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:26 smithi125 conmon[56739]: cluster 2021-03-15T22:20:24.332973+0000 osd.2 (osd.2) 21 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:26 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:20:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:26 smithi125 conmon[56739]: 15T22:20:24.847224+0000 osd.2 (osd.2) 22 : cluster [DBG] 4.f scrub ok 2021-03-15T22:20:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:26 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:20:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:26 smithi125 conmon[56739]: -03-15T22:20:26.000591+0000 mon.a (mon.0) 636 2021-03-15T22:20:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:26 smithi125 conmon[56739]: : cluster [DBG] osdmap e111: 8 total, 8 up, 8 in 2021-03-15T22:20:26.309 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:26.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:26.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:26.412 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:26.434 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:26.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:26.435 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:26.536 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:26.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:26.560 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:26.560 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:26.661 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:26.688 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:26.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:26.689 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:26.747 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:20:26.781 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:26.791 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:26.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:26.817 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:26.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:26.919 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:26.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:26.943 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:26.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:27.008 INFO:tasks.rados.rados.0.smithi076.stdout:124: done (2 left) 2021-03-15T22:20:27.008 INFO:tasks.rados.rados.0.smithi076.stdout:125: read oid 3 snap -1 2021-03-15T22:20:27.008 INFO:tasks.rados.rados.0.smithi076.stdout:125: expect (ObjNum 30 snap 0 seq_num 30) 2021-03-15T22:20:27.009 INFO:tasks.rados.rados.0.smithi076.stdout:126: read oid 5 snap 5 2021-03-15T22:20:27.009 INFO:tasks.rados.rados.0.smithi076.stdout:126: expect (ObjNum 4 snap 0 seq_num 4) 2021-03-15T22:20:27.009 INFO:tasks.rados.rados.0.smithi076.stdout:127: write (excl) oid 39 current snap is 8 2021-03-15T22:20:27.009 INFO:tasks.rados.rados.0.smithi076.stdout:127: seq_num 71 ranges {793297=789856,2305045=723014} 2021-03-15T22:20:27.019 INFO:tasks.rados.rados.0.smithi076.stdout:127: writing smithi07694878-39 from 793297 to 1583153 tid 1 2021-03-15T22:20:27.028 INFO:tasks.rados.rados.0.smithi076.stdout:127: writing smithi07694878-39 from 2305045 to 3028059 tid 2 2021-03-15T22:20:27.029 INFO:tasks.rados.rados.0.smithi076.stdout:128: write oid 48 current snap is 8 2021-03-15T22:20:27.029 INFO:tasks.rados.rados.0.smithi076.stdout:128: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2021-03-15T22:20:27.034 INFO:tasks.rados.rados.0.smithi076.stdout:128: writing smithi07694878-48 from 410104 to 875209 tid 1 2021-03-15T22:20:27.042 INFO:tasks.rados.rados.0.smithi076.stdout:128: writing smithi07694878-48 from 1563527 to 2146552 tid 2 2021-03-15T22:20:27.044 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:27.048 INFO:tasks.rados.rados.0.smithi076.stdout:128: writing smithi07694878-48 from 2810926 to 3210104 tid 3 2021-03-15T22:20:27.048 INFO:tasks.rados.rados.0.smithi076.stdout:129: rollback oid 42 current snap is 8 2021-03-15T22:20:27.049 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 42 to 5 2021-03-15T22:20:27.049 INFO:tasks.rados.rados.0.smithi076.stdout:122: expect (ObjNum 13 snap 0 seq_num 13) 2021-03-15T22:20:27.069 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:27.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:27.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:27.125 INFO:tasks.rados.rados.0.smithi076.stdout:123: finishing write tid 1 to smithi07694878-47 2021-03-15T22:20:27.125 INFO:tasks.rados.rados.0.smithi076.stdout:123: finishing write tid 2 to smithi07694878-47 2021-03-15T22:20:27.125 INFO:tasks.rados.rados.0.smithi076.stdout:122: done (6 left) 2021-03-15T22:20:27.126 INFO:tasks.rados.rados.0.smithi076.stdout:130: rollback oid 17 current snap is 8 2021-03-15T22:20:27.126 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 17 to 5 2021-03-15T22:20:27.126 INFO:tasks.rados.rados.0.smithi076.stdout:123: finishing write tid 3 to smithi07694878-47 2021-03-15T22:20:27.126 INFO:tasks.rados.rados.0.smithi076.stdout:123: finishing write tid 4 to smithi07694878-47 2021-03-15T22:20:27.126 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 29 (ObjNum 70 snap 8 seq_num 70) dirty exists 2021-03-15T22:20:27.127 INFO:tasks.rados.rados.0.smithi076.stdout:123: left oid 47 (ObjNum 70 snap 8 seq_num 70) 2021-03-15T22:20:27.127 INFO:tasks.rados.rados.0.smithi076.stdout:126: expect (ObjNum 4 snap 0 seq_num 4) 2021-03-15T22:20:27.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:27.189 INFO:tasks.rados.rados.0.smithi076.stdout:125: expect (ObjNum 30 snap 0 seq_num 30) 2021-03-15T22:20:27.193 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:27.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:27.195 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:27 smithi076 conmon[56514]: cluster 2021-03-15T22:20:25.877833+0000 osd.3 (osd.3) 33 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:20:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:27 smithi076 conmon[56514]: cluster 2021-03-15T22:20:26.981378+0000 mgr.y (mgr.14144) 377 : cluster [DBG] pgmap v294: 148 pgs: 148 active+clean; 133 MiB data, 319 MiB used, 715 GiB / 715 GiB avail; 7.1 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2021-03-15T22:20:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:27 smithi076 conmon[56514]: cluster 2021-03-15T22:20:27.005348+0000 mon.a (mon.0) 637 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2021-03-15T22:20:27.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:27 smithi076 conmon[62156]: cluster 2021-03-15T22:20:25.877833+0000 osd.3 (osd. 2021-03-15T22:20:27.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:27 smithi076 conmon[62156]: 3) 33 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:20:27.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:27 smithi076 conmon[62156]: cluster 2021-03-15T22:20:26.981378+0000 mgr.y (mgr.14144) 377 : cluster [DBG] pgmap v294: 148 pgs: 148 active+clean; 133 MiB data, 319 MiB used, 715 GiB / 715 GiB avail; 7.1 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2021-03-15T22:20:27.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:27 smithi076 conmon[62156]: cluster 2021-03-15T22:20:27.005348+0000 mon.a (mon.0) 637 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2021-03-15T22:20:27.269 INFO:tasks.rados.rados.0.smithi076.stdout:127: finishing write tid 1 to smithi07694878-39 2021-03-15T22:20:27.270 INFO:tasks.rados.rados.0.smithi076.stdout:127: finishing write tid 2 to smithi07694878-39 2021-03-15T22:20:27.270 INFO:tasks.rados.rados.0.smithi076.stdout:127: finishing write tid 3 to smithi07694878-39 2021-03-15T22:20:27.270 INFO:tasks.rados.rados.0.smithi076.stdout:127: finishing write tid 4 to smithi07694878-39 2021-03-15T22:20:27.270 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 35 (ObjNum 71 snap 8 seq_num 71) dirty exists 2021-03-15T22:20:27.271 INFO:tasks.rados.rados.0.smithi076.stdout:127: left oid 39 (ObjNum 71 snap 8 seq_num 71) 2021-03-15T22:20:27.271 INFO:tasks.rados.rados.0.smithi076.stdout:128: finishing write tid 1 to smithi07694878-48 2021-03-15T22:20:27.271 INFO:tasks.rados.rados.0.smithi076.stdout:128: finishing write tid 2 to smithi07694878-48 2021-03-15T22:20:27.271 INFO:tasks.rados.rados.0.smithi076.stdout:123: done (6 left) 2021-03-15T22:20:27.272 INFO:tasks.rados.rados.0.smithi076.stdout:125: done (5 left) 2021-03-15T22:20:27.272 INFO:tasks.rados.rados.0.smithi076.stdout:126: done (4 left) 2021-03-15T22:20:27.272 INFO:tasks.rados.rados.0.smithi076.stdout:127: done (3 left) 2021-03-15T22:20:27.272 INFO:tasks.rados.rados.0.smithi076.stdout:131: write (excl) oid 31 current snap is 8 2021-03-15T22:20:27.272 INFO:tasks.rados.rados.0.smithi076.stdout:131: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2021-03-15T22:20:27.274 INFO:tasks.rados.rados.0.smithi076.stdout:131: writing smithi07694878-31 from 426911 to 967265 tid 1 2021-03-15T22:20:27.282 INFO:tasks.rados.rados.0.smithi076.stdout:131: writing smithi07694878-31 from 1622009 to 2348692 tid 2 2021-03-15T22:20:27.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:27 smithi125 conmon[56739]: cluster 2021-03-15T22:20:25.877833+0000 osd.3 (osd.3) 33 : cluster [DBG] 2021-03-15T22:20:27.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:27 smithi125 conmon[56739]: 4.14 scrub ok 2021-03-15T22:20:27.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:27 smithi125 conmon[56739]: cluster 2021-03-15T22:20:26.981378+0000 mgr.y (mgr.14144) 377 : cluster [DBG] pgmap v294: 148 pgs: 148 active+clean; 133 MiB data, 319 MiB used, 715 GiB / 715 GiB avail; 7.1 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2021-03-15T22:20:27.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:27 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:20:27.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:27 smithi125 conmon[56739]: 20:27.005348+0000 mon.a (mon.0) 637 : cluster [DBG] osdmap e112: 8 total, 8 up, 8 in 2021-03-15T22:20:27.288 INFO:tasks.rados.rados.0.smithi076.stdout:131: writing smithi07694878-31 from 3121996 to 3226911 tid 3 2021-03-15T22:20:27.288 INFO:tasks.rados.rados.0.smithi076.stdout:132: copy_from oid 1 from oid 23 current snap is 8 2021-03-15T22:20:27.289 INFO:tasks.rados.rados.0.smithi076.stdout:129: finishing rollback tid 0 to smithi07694878-42 2021-03-15T22:20:27.289 INFO:tasks.rados.rados.0.smithi076.stdout:133: read oid 32 snap -1 2021-03-15T22:20:27.289 INFO:tasks.rados.rados.0.smithi076.stdout:133: expect (ObjNum 34 snap 0 seq_num 34) 2021-03-15T22:20:27.289 INFO:tasks.rados.rados.0.smithi076.stdout:134: write (excl) oid 15 current snap is 8 2021-03-15T22:20:27.290 INFO:tasks.rados.rados.0.smithi076.stdout:128: finishing write tid 3 to smithi07694878-48 2021-03-15T22:20:27.290 INFO:tasks.rados.rados.0.smithi076.stdout:134: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2021-03-15T22:20:27.294 INFO:tasks.rados.rados.0.smithi076.stdout:134: writing smithi07694878-15 from 443718 to 1059321 tid 1 2021-03-15T22:20:27.294 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:27.296 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:27.299 INFO:tasks.rados.rados.0.smithi076.stdout:134: writing smithi07694878-15 from 1564138 to 2034479 tid 2 2021-03-15T22:20:27.306 INFO:tasks.rados.rados.0.smithi076.stdout:134: writing smithi07694878-15 from 2633066 to 3243718 tid 3 2021-03-15T22:20:27.306 INFO:tasks.rados.rados.0.smithi076.stdout:135: snap_create 2021-03-15T22:20:27.307 INFO:tasks.rados.rados.0.smithi076.stdout:128: finishing write tid 4 to smithi07694878-48 2021-03-15T22:20:27.307 INFO:tasks.rados.rados.0.smithi076.stdout:129: finishing rollback tid 1 to smithi07694878-42 2021-03-15T22:20:27.307 INFO:tasks.rados.rados.0.smithi076.stdout:129: finishing rollback tid 2 to smithi07694878-42 2021-03-15T22:20:27.307 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 55 (ObjNum 27 snap 0 seq_num 27) dirty exists 2021-03-15T22:20:27.308 INFO:tasks.rados.rados.0.smithi076.stdout:128: finishing write tid 5 to smithi07694878-48 2021-03-15T22:20:27.308 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 14 (ObjNum 72 snap 8 seq_num 72) dirty exists 2021-03-15T22:20:27.308 INFO:tasks.rados.rados.0.smithi076.stdout:128: left oid 48 (ObjNum 72 snap 8 seq_num 72) 2021-03-15T22:20:27.308 INFO:tasks.rados.rados.0.smithi076.stdout:130: finishing rollback tid 0 to smithi07694878-17 2021-03-15T22:20:27.309 INFO:tasks.rados.rados.0.smithi076.stdout:130: finishing rollback tid 1 to smithi07694878-17 2021-03-15T22:20:27.309 INFO:tasks.rados.rados.0.smithi076.stdout:130: finishing rollback tid 2 to smithi07694878-17 2021-03-15T22:20:27.309 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 7 (ObjNum 16 snap 0 seq_num 16) dirty exists 2021-03-15T22:20:27.309 INFO:tasks.rados.rados.0.smithi076.stdout:131: finishing write tid 1 to smithi07694878-31 2021-03-15T22:20:27.309 INFO:tasks.rados.rados.0.smithi076.stdout:131: finishing write tid 2 to smithi07694878-31 2021-03-15T22:20:27.310 INFO:tasks.rados.rados.0.smithi076.stdout:131: finishing write tid 3 to smithi07694878-31 2021-03-15T22:20:27.310 INFO:tasks.rados.rados.0.smithi076.stdout:131: finishing write tid 4 to smithi07694878-31 2021-03-15T22:20:27.310 INFO:tasks.rados.rados.0.smithi076.stdout:131: finishing write tid 5 to smithi07694878-31 2021-03-15T22:20:27.310 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 16 (ObjNum 73 snap 8 seq_num 73) dirty exists 2021-03-15T22:20:27.311 INFO:tasks.rados.rados.0.smithi076.stdout:131: left oid 31 (ObjNum 73 snap 8 seq_num 73) 2021-03-15T22:20:27.311 INFO:tasks.rados.rados.0.smithi076.stdout:133: expect (ObjNum 34 snap 0 seq_num 34) 2021-03-15T22:20:27.321 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:27.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:27.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:27.396 INFO:tasks.rados.rados.0.smithi076.stdout:134: finishing write tid 1 to smithi07694878-15 2021-03-15T22:20:27.397 INFO:tasks.rados.rados.0.smithi076.stdout:132: finishing copy_from to smithi07694878-1 2021-03-15T22:20:27.397 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2021-03-15T22:20:27.397 INFO:tasks.rados.rados.0.smithi076.stdout:132: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:20:27.398 INFO:tasks.rados.rados.0.smithi076.stdout:134: finishing write tid 2 to smithi07694878-15 2021-03-15T22:20:27.398 INFO:tasks.rados.rados.0.smithi076.stdout:134: finishing write tid 3 to smithi07694878-15 2021-03-15T22:20:27.398 INFO:tasks.rados.rados.0.smithi076.stdout:134: finishing write tid 4 to smithi07694878-15 2021-03-15T22:20:27.398 INFO:tasks.rados.rados.0.smithi076.stdout:134: finishing write tid 5 to smithi07694878-15 2021-03-15T22:20:27.398 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 58 (ObjNum 74 snap 8 seq_num 74) dirty exists 2021-03-15T22:20:27.399 INFO:tasks.rados.rados.0.smithi076.stdout:134: left oid 15 (ObjNum 74 snap 8 seq_num 74) 2021-03-15T22:20:27.423 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:27.447 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:27.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:27.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:27.550 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:27.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:27.574 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:27.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:27.630 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:20:27.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:27.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:27.700 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:27.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:27.801 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:27.824 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:27.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:27.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:27.927 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:27.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:27.950 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:27.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:28.035 INFO:tasks.rados.rados.0.smithi076.stdout:128: done (7 left) 2021-03-15T22:20:28.036 INFO:tasks.rados.rados.0.smithi076.stdout:129: done (6 left) 2021-03-15T22:20:28.036 INFO:tasks.rados.rados.0.smithi076.stdout:130: done (5 left) 2021-03-15T22:20:28.036 INFO:tasks.rados.rados.0.smithi076.stdout:131: done (4 left) 2021-03-15T22:20:28.037 INFO:tasks.rados.rados.0.smithi076.stdout:132: done (3 left) 2021-03-15T22:20:28.037 INFO:tasks.rados.rados.0.smithi076.stdout:133: done (2 left) 2021-03-15T22:20:28.037 INFO:tasks.rados.rados.0.smithi076.stdout:134: done (1 left) 2021-03-15T22:20:28.038 INFO:tasks.rados.rados.0.smithi076.stdout:135: done (0 left) 2021-03-15T22:20:28.038 INFO:tasks.rados.rados.0.smithi076.stdout:136: snap_create 2021-03-15T22:20:28.052 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:28.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:28.075 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:28.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:28.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:28.200 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:28.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:28.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:28.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:28 smithi125 conmon[56739]: cluster 2021-03-15T22:20:26.830180+0000 osd.3 (osd.3) 34 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:20:28.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:28 smithi125 conmon[56739]: 2021-03-15T22:20:28.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:28 smithi125 conmon[56739]: cluster 2021-03-15T22:20:27.003237+0000 osd.7 (osd.7) 25 : cluster [DBG] 3.e scrub ok 2021-03-15T22:20:28.302 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:28.306 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:28 smithi076 conmon[56514]: cluster 2021-03-15T22:20:26.830180+0000 2021-03-15T22:20:28.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:28 smithi076 conmon[56514]: osd.3 (osd.3) 34 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:20:28.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:28 smithi076 conmon[56514]: cluster 2021-03-15T22:20:27.003237+0000 osd.7 (osd.7) 25 : cluster [DBG] 3.e scrub ok 2021-03-15T22:20:28.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:28 smithi076 conmon[62156]: cluster 2021-03-15T22:20:26.830180+0000 osd.3 (osd. 2021-03-15T22:20:28.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:28 smithi076 conmon[62156]: 3) 34 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:20:28.308 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:28 smithi076 conmon[62156]: cluster 2021-03-15T22:20:27.003237+0000 osd.7 (osd.7) 25 2021-03-15T22:20:28.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:28 smithi076 conmon[62156]: : cluster [DBG] 3.e scrub ok 2021-03-15T22:20:28.326 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:28.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:28.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:28.362 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:28.429 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:28.451 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:28.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:28.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:28.555 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:28.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:28.580 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:28.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:28.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:28.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:28.704 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:28.704 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:28.805 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:28.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:28.829 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:28.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:28.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:28.956 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:28.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:29.036 INFO:tasks.rados.rados.0.smithi076.stdout:136: done (0 left) 2021-03-15T22:20:29.037 INFO:tasks.rados.rados.0.smithi076.stdout:137: snap_remove snap 5 2021-03-15T22:20:29.059 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:29.084 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:29.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:29.087 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:29.188 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:29.210 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:29.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:29.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:29.310 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[56514]: cluster 2021-03-15T22:20:27.760748+0000 osd.1 (osd.1) 26 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:20:29.310 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[56514]: cluster 2021-03-15T22:20:28.025878+0000 osd.7 (osd.7) 26 : cluster [DBG] 3.f scrub ok 2021-03-15T22:20:29.311 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[56514]: cluster 2021-03-15T22:20:28.032602+0000 mon.a (mon.0) 638 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2021-03-15T22:20:29.311 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[56514]: cluster 2021-03-15T22:20:28.291289+0000 osd.4 (osd.4) 25 : cluster [DBG] 5.18 scrub ok 2021-03-15T22:20:29.311 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[56514]: cluster 2021-03-15T22:20:28.505563+0000 osd.0 (osd.0) 30 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:20:29.311 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[56514]: cluster 2021-03-15T22:20:28.981877+0000 mgr.y (mgr.14144) 378 : cluster [DBG] pgmap v297: 148 pgs: 148 active+clean; 132 MiB data, 323 MiB used, 715 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.5 MiB/s wr, 13 op/s 2021-03-15T22:20:29.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: cluster 2021-03-15T22:20:27.760748+0000 2021-03-15T22:20:29.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: osd.1 (osd.1) 26 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:20:29.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: cluster 2021-03-15T22:20:28. 2021-03-15T22:20:29.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: 025878+0000 osd.7 (osd.7) 26 : cluster [DBG] 3.f scrub ok 2021-03-15T22:20:29.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:20:29.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: 15T22:20:28.032602+0000 mon.a (mon 2021-03-15T22:20:29.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: .0) 638 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2021-03-15T22:20:29.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:20:29.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: -15T22:20:28.291289+0000 osd.4 (osd.4 2021-03-15T22:20:29.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: ) 25 : cluster [DBG] 5.18 scrub ok 2021-03-15T22:20:29.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: cluster 2021-03-15T22:20:29.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: 2021-03-15T22:20:28.505563 2021-03-15T22:20:29.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: +0000 osd.0 (osd.0) 30 : cluster [DBG] 2021-03-15T22:20:29.315 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: 4.3 scrub ok 2021-03-15T22:20:29.315 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:20:29.315 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: 22:20:28.981877+0000 mgr.y (mgr. 2021-03-15T22:20:29.316 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:29 smithi076 conmon[62156]: 14144) 378 : cluster [DBG] pgmap v297: 148 pgs: 148 active+clean; 132 MiB data, 323 MiB used, 715 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.5 MiB/s wr, 13 op/s 2021-03-15T22:20:29.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:29.339 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:29.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:29.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:29.427 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: cluster 2021-03-15T22:20:27.760748+0000 osd.1 (osd.1) 26 : cluster 2021-03-15T22:20:29.427 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: [DBG] 3.19 scrub ok 2021-03-15T22:20:29.428 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: cluster 2021-03-15T22:20:28.025878+0000 osd.7 (osd.7) 26 : cluster [DBG] 3.f scrub ok 2021-03-15T22:20:29.428 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: cluster 2021-03-15T22:20:28.032602+0000 mon.a (mon 2021-03-15T22:20:29.428 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: .0) 638 : cluster [DBG] osdmap e113: 8 total, 8 up, 8 in 2021-03-15T22:20:29.428 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:20:29.429 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: 22:20:28.291289+0000 osd.4 (osd.4) 25 2021-03-15T22:20:29.429 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: : cluster [DBG] 5.18 scrub ok 2021-03-15T22:20:29.429 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:20:29.429 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: 22:20:28.505563+0000 osd.0 (osd.0) 30 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:20:29.429 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:20:29.430 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: -15T22:20:28.981877 2021-03-15T22:20:29.430 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 378 : cluster 2021-03-15T22:20:29.430 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[56739]: [DBG] pgmap v297: 148 pgs: 148 active+clean; 132 MiB data, 323 MiB used, 715 GiB / 715 GiB avail; 5.2 MiB/s rd, 3.5 MiB/s wr, 13 op/s 2021-03-15T22:20:29.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:29.466 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:29.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:29.468 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:29.569 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:29.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:29.593 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:29.594 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:29.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:29.721 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:29.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:29.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:29.758 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:20:29.758 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:20:29.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:20:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:20:29.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:29.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:29.850 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:29.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:29.951 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:29.977 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:29.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:29.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:29.997 INFO:tasks.rados.rados.0.smithi076.stdout:137: done (0 left) 2021-03-15T22:20:29.997 INFO:tasks.rados.rados.0.smithi076.stdout:138: write oid 21 current snap is 10 2021-03-15T22:20:29.998 INFO:tasks.rados.rados.0.smithi076.stdout:138: seq_num 75 ranges {460525=690852,1622620=730352,3060489=200036} 2021-03-15T22:20:30.003 INFO:tasks.rados.rados.0.smithi076.stdout:138: writing smithi07694878-21 from 460525 to 1151377 tid 1 2021-03-15T22:20:30.010 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:20:30.010 INFO:tasks.rados.rados.0.smithi076.stdout:138: writing smithi07694878-21 from 1622620 to 2352972 tid 2 2021-03-15T22:20:30.017 INFO:tasks.rados.rados.0.smithi076.stdout:138: writing smithi07694878-21 from 3060489 to 3260525 tid 3 2021-03-15T22:20:30.017 INFO:tasks.rados.rados.0.smithi076.stdout:139: read oid 47 snap -1 2021-03-15T22:20:30.017 INFO:tasks.rados.rados.0.smithi076.stdout:139: expect (ObjNum 70 snap 8 seq_num 70) 2021-03-15T22:20:30.017 INFO:tasks.rados.rados.0.smithi076.stdout:140: delete oid 19 current snap is 10 2021-03-15T22:20:30.018 INFO:tasks.rados.rados.0.smithi076.stdout:138: finishing write tid 1 to smithi07694878-21 2021-03-15T22:20:30.018 INFO:tasks.rados.rados.0.smithi076.stdout:138: finishing write tid 2 to smithi07694878-21 2021-03-15T22:20:30.018 INFO:tasks.rados.rados.0.smithi076.stdout:140: done (2 left) 2021-03-15T22:20:30.019 INFO:tasks.rados.rados.0.smithi076.stdout:141: write oid 15 current snap is 10 2021-03-15T22:20:30.019 INFO:tasks.rados.rados.0.smithi076.stdout:141: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2021-03-15T22:20:30.024 INFO:tasks.rados.rados.0.smithi076.stdout:141: writing smithi07694878-15 from 494139 to 1051842 tid 1 2021-03-15T22:20:30.028 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:20:30.029 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:20:30.029 INFO:tasks.thrashosds.thrasher:Setting osd 3 primary_affinity to 1.000000 2021-03-15T22:20:30.029 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd primary-affinity 3 1 2021-03-15T22:20:30.030 INFO:tasks.rados.rados.0.smithi076.stdout:141: writing smithi07694878-15 from 1455937 to 2189958 tid 2 2021-03-15T22:20:30.037 INFO:tasks.rados.rados.0.smithi076.stdout:141: writing smithi07694878-15 from 2948041 to 3294139 tid 3 2021-03-15T22:20:30.037 INFO:tasks.rados.rados.0.smithi076.stdout:142: write (excl) oid 11 current snap is 10 2021-03-15T22:20:30.037 INFO:tasks.rados.rados.0.smithi076.stdout:138: finishing write tid 3 to smithi07694878-21 2021-03-15T22:20:30.038 INFO:tasks.rados.rados.0.smithi076.stdout:142: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2021-03-15T22:20:30.045 INFO:tasks.rados.rados.0.smithi076.stdout:142: writing smithi07694878-11 from 510946 to 1143898 tid 1 2021-03-15T22:20:30.050 INFO:tasks.rados.rados.0.smithi076.stdout:142: writing smithi07694878-11 from 1798066 to 2275745 tid 2 2021-03-15T22:20:30.055 INFO:tasks.rados.rados.0.smithi076.stdout:142: writing smithi07694878-11 from 2742758 to 3199975 tid 3 2021-03-15T22:20:30.055 INFO:tasks.rados.rados.0.smithi076.stdout:143: write (excl) oid 1 current snap is 10 2021-03-15T22:20:30.056 INFO:tasks.rados.rados.0.smithi076.stdout:138: finishing write tid 4 to smithi07694878-21 2021-03-15T22:20:30.056 INFO:tasks.rados.rados.0.smithi076.stdout:143: seq_num 79 ranges {527753=708201,1856548=737690,3286534=41219} 2021-03-15T22:20:30.062 INFO:tasks.rados.rados.0.smithi076.stdout:143: writing smithi07694878-1 from 527753 to 1235954 tid 1 2021-03-15T22:20:30.070 INFO:tasks.rados.rados.0.smithi076.stdout:143: writing smithi07694878-1 from 1856548 to 2594238 tid 2 2021-03-15T22:20:30.074 INFO:tasks.rados.rados.0.smithi076.stdout:143: writing smithi07694878-1 from 3286534 to 3327753 tid 3 2021-03-15T22:20:30.074 INFO:tasks.rados.rados.0.smithi076.stdout:144: delete oid 14 current snap is 10 2021-03-15T22:20:30.074 INFO:tasks.rados.rados.0.smithi076.stdout:138: finishing write tid 5 to smithi07694878-21 2021-03-15T22:20:30.075 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 68 (ObjNum 75 snap 10 seq_num 75) dirty exists 2021-03-15T22:20:30.075 INFO:tasks.rados.rados.0.smithi076.stdout:138: left oid 21 (ObjNum 75 snap 10 seq_num 75) 2021-03-15T22:20:30.075 INFO:tasks.rados.rados.0.smithi076.stdout:139: expect (ObjNum 70 snap 8 seq_num 70) 2021-03-15T22:20:30.080 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:30.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:30.107 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:30.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:30.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:30.232 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:30.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:30.233 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:30.263 INFO:tasks.rados.rados.0.smithi076.stdout:141: finishing write tid 1 to smithi07694878-15 2021-03-15T22:20:30.263 INFO:tasks.rados.rados.0.smithi076.stdout:141: finishing write tid 2 to smithi07694878-15 2021-03-15T22:20:30.263 INFO:tasks.rados.rados.0.smithi076.stdout:138: done (5 left) 2021-03-15T22:20:30.264 INFO:tasks.rados.rados.0.smithi076.stdout:139: done (4 left) 2021-03-15T22:20:30.264 INFO:tasks.rados.rados.0.smithi076.stdout:144: done (3 left) 2021-03-15T22:20:30.264 INFO:tasks.rados.rados.0.smithi076.stdout:145: rollback oid 28 current snap is 10 2021-03-15T22:20:30.264 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 8 2021-03-15T22:20:30.265 INFO:tasks.rados.rados.0.smithi076.stdout:141: finishing write tid 3 to smithi07694878-15 2021-03-15T22:20:30.265 INFO:tasks.rados.rados.0.smithi076.stdout:141: finishing write tid 4 to smithi07694878-15 2021-03-15T22:20:30.265 INFO:tasks.rados.rados.0.smithi076.stdout:141: finishing write tid 5 to smithi07694878-15 2021-03-15T22:20:30.265 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 69 (ObjNum 77 snap 10 seq_num 77) dirty exists 2021-03-15T22:20:30.265 INFO:tasks.rados.rados.0.smithi076.stdout:141: left oid 15 (ObjNum 77 snap 10 seq_num 77) 2021-03-15T22:20:30.266 INFO:tasks.rados.rados.0.smithi076.stdout:142: finishing write tid 1 to smithi07694878-11 2021-03-15T22:20:30.266 INFO:tasks.rados.rados.0.smithi076.stdout:142: finishing write tid 2 to smithi07694878-11 2021-03-15T22:20:30.266 INFO:tasks.rados.rados.0.smithi076.stdout:142: finishing write tid 3 to smithi07694878-11 2021-03-15T22:20:30.266 INFO:tasks.rados.rados.0.smithi076.stdout:142: finishing write tid 4 to smithi07694878-11 2021-03-15T22:20:30.267 INFO:tasks.rados.rados.0.smithi076.stdout:142: finishing write tid 5 to smithi07694878-11 2021-03-15T22:20:30.267 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 30 (ObjNum 78 snap 10 seq_num 78) dirty exists 2021-03-15T22:20:30.267 INFO:tasks.rados.rados.0.smithi076.stdout:142: left oid 11 (ObjNum 78 snap 10 seq_num 78) 2021-03-15T22:20:30.267 INFO:tasks.rados.rados.0.smithi076.stdout:143: finishing write tid 1 to smithi07694878-1 2021-03-15T22:20:30.267 INFO:tasks.rados.rados.0.smithi076.stdout:143: finishing write tid 2 to smithi07694878-1 2021-03-15T22:20:30.268 INFO:tasks.rados.rados.0.smithi076.stdout:143: finishing write tid 3 to smithi07694878-1 2021-03-15T22:20:30.268 INFO:tasks.rados.rados.0.smithi076.stdout:143: finishing write tid 4 to smithi07694878-1 2021-03-15T22:20:30.268 INFO:tasks.rados.rados.0.smithi076.stdout:143: finishing write tid 5 to smithi07694878-1 2021-03-15T22:20:30.268 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 26 (ObjNum 79 snap 10 seq_num 79) dirty exists 2021-03-15T22:20:30.268 INFO:tasks.rados.rados.0.smithi076.stdout:143: left oid 1 (ObjNum 79 snap 10 seq_num 79) 2021-03-15T22:20:30.269 INFO:tasks.rados.rados.0.smithi076.stdout:141: done (3 left) 2021-03-15T22:20:30.269 INFO:tasks.rados.rados.0.smithi076.stdout:142: done (2 left) 2021-03-15T22:20:30.269 INFO:tasks.rados.rados.0.smithi076.stdout:143: done (1 left) 2021-03-15T22:20:30.269 INFO:tasks.rados.rados.0.smithi076.stdout:146: snap_remove snap 9 2021-03-15T22:20:30.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[56514]: cluster 2021-03-15T22:20:29 2021-03-15T22:20:30.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[56514]: .033921+0000 mon.a (mon.0) 639 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2021-03-15T22:20:30.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[56514]: cluster 2021-03-15T22:20:29.220903+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:20:30.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[56514]: cluster 2021-03-15T22:20:29.251909+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:20:30.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[56514]: audit 2021-03-15T22:20:29.352294+0000 mon.a (mon.0) 640 : audit [INF] from='client.? 172.21.15.76:0/1666541464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:30.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[56514]: audit 2021-03-15T22:20:29.425696+0000 mgr.y (mgr.14144) 379 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:30.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[56514]: cluster 2021-03-15T22:20:29.988365+0000 mon.a (mon.0) 641 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:20:30.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[56514]: cluster 2021-03-15T22:20:29.988392+0000 mon.a (mon.0) 642 : cluster [INF] Cluster is now healthy 2021-03-15T22:20:30.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[56514]: audit 2021-03-15T22:20:29.994959+0000 mon.a (mon.0) 643 : audit [INF] from='client.? 172.21.15.76:0/1666541464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:20:30.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[56514]: cluster 2021-03-15T22:20:29.995020+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2021-03-15T22:20:30.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[62156]: cluster 2021-03-15T22:20:29.033921+0000 2021-03-15T22:20:30.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[62156]: mon.a (mon.0) 639 : cluster [DBG] osdmap e114: 8 total, 8 up, 8 in 2021-03-15T22:20:30.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[62156]: cluster 2021-03-15T22:20:29.220903+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:20:30.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[62156]: cluster 2021-03-15T22:20:29.251909+0000 osd.4 (osd.4) 26 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:20:30.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[62156]: audit 2021-03-15T22:20:29.352294+0000 mon.a (mon.0) 640 : audit [INF] from='client.? 172.21.15.76:0/1666541464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:30.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[62156]: audit 2021-03-15T22:20:29.425696+0000 mgr.y (mgr.14144) 379 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:30.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[62156]: cluster 2021-03-15T22:20:29.988365+0000 mon.a (mon.0) 641 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:20:30.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[62156]: cluster 2021-03-15T22:20:29.988392+0000 mon.a (mon.0) 642 : cluster [INF] Cluster is now healthy 2021-03-15T22:20:30.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[62156]: audit 2021-03-15T22:20:29.994959+0000 mon.a (mon.0) 643 : audit [INF] from='client.? 172.21.15.76:0/1666541464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:20:30.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:30 smithi076 conmon[62156]: cluster 2021-03-15T22:20:29.995020+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2021-03-15T22:20:30.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:30.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:30.359 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:30.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:30.461 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:30.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:30.485 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:30.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:30.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: cluster 2021-03-15T22:20:29.033921+0000 mon.a (mon.0) 639 : cluster [DBG] 2021-03-15T22:20:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: osdmap e114: 8 total, 8 up, 8 in 2021-03-15T22:20:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: cluster 2021-03-15T22:20:29.220903+0000 osd.5 (osd.5) 27 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:20:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: cluster 2021-03-15T22:20:29.251909+0000 osd.4 (osd.4) 26 : cluster [DBG] 2021-03-15T22:20:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: 6.1e scrub ok 2021-03-15T22:20:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: audit 2021-03-15T22:20:29.352294+0000 mon.a (mon.0) 640 : audit 2021-03-15T22:20:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/1666541464' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:20:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: 15T22:20:29.425696+0000 mgr.y (mgr.14144) 379 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: cluster 2021-03-15T22:20:29.988365 2021-03-15T22:20:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: +0000 mon.a (mon.0) 641 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:20:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:20:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: T22:20:29.988392+0000 mon.a (mon.0) 642 : cluster 2021-03-15T22:20:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: [INF] Cluster is now healthy 2021-03-15T22:20:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: audit 2021-03-15T22:20:29.994959+0000 mon.a ( 2021-03-15T22:20:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: mon.0) 643 : audit [INF] from='client.? 172.21.15.76:0/1666541464' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:20:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: cluster 2021-03-15T22:20 2021-03-15T22:20:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: :29.995020+0000 mon.a (mon.0) 644 : cluster [DBG] osdmap e115: 8 total, 8 up, 8 in 2021-03-15T22:20:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:30 smithi125 conmon[56739]: 2021-03-15T22:20:30.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:30.609 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:30.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:30.611 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:30.712 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:30.736 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:30.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:30.838 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:30.865 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:30.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:30.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:30.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:30.990 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:30.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:30.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:31.055 INFO:tasks.rados.rados.0.smithi076.stdout:146: done (1 left) 2021-03-15T22:20:31.056 INFO:tasks.rados.rados.0.smithi076.stdout:147: read oid 11 snap -1 2021-03-15T22:20:31.056 INFO:tasks.rados.rados.0.smithi076.stdout:147: expect (ObjNum 78 snap 10 seq_num 78) 2021-03-15T22:20:31.057 INFO:tasks.rados.rados.0.smithi076.stdout:148: write (excl) oid 21 current snap is 10 2021-03-15T22:20:31.057 INFO:tasks.rados.rados.0.smithi076.stdout:148: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2021-03-15T22:20:31.061 INFO:tasks.rados.rados.0.smithi076.stdout:148: writing smithi07694878-21 from 561367 to 1020066 tid 1 2021-03-15T22:20:31.069 INFO:tasks.rados.rados.0.smithi076.stdout:148: writing smithi07694878-21 from 1573512 to 2314871 tid 2 2021-03-15T22:20:31.075 INFO:tasks.rados.rados.0.smithi076.stdout:148: writing smithi07694878-21 from 2941380 to 3361367 tid 3 2021-03-15T22:20:31.075 INFO:tasks.rados.rados.0.smithi076.stdout:149: write oid 20 current snap is 10 2021-03-15T22:20:31.075 INFO:tasks.rados.rados.0.smithi076.stdout:149: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2021-03-15T22:20:31.080 INFO:tasks.rados.rados.0.smithi076.stdout:149: writing smithi07694878-20 from 578174 to 1112122 tid 1 2021-03-15T22:20:31.085 INFO:tasks.rados.rados.0.smithi076.stdout:149: writing smithi07694878-20 from 1515641 to 2000658 tid 2 2021-03-15T22:20:31.093 INFO:tasks.rados.rados.0.smithi076.stdout:149: writing smithi07694878-20 from 2736097 to 3238402 tid 3 2021-03-15T22:20:31.093 INFO:tasks.rados.rados.0.smithi076.stdout:150: snap_create 2021-03-15T22:20:31.093 INFO:tasks.rados.rados.0.smithi076.stdout:145: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:20:31.093 INFO:tasks.rados.rados.0.smithi076.stdout:145: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:20:31.094 INFO:tasks.rados.rados.0.smithi076.stdout:145: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:20:31.094 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 72 (ObjNum 27 snap 0 seq_num 27) dirty exists 2021-03-15T22:20:31.094 INFO:tasks.rados.rados.0.smithi076.stdout:147: expect (ObjNum 78 snap 10 seq_num 78) 2021-03-15T22:20:31.095 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:31.119 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:31.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:31.121 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:31.222 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:31.244 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:31.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:31.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:31.249 INFO:tasks.rados.rados.0.smithi076.stdout:148: finishing write tid 1 to smithi07694878-21 2021-03-15T22:20:31.249 INFO:tasks.rados.rados.0.smithi076.stdout:148: finishing write tid 2 to smithi07694878-21 2021-03-15T22:20:31.249 INFO:tasks.rados.rados.0.smithi076.stdout:148: finishing write tid 3 to smithi07694878-21 2021-03-15T22:20:31.250 INFO:tasks.rados.rados.0.smithi076.stdout:148: finishing write tid 4 to smithi07694878-21 2021-03-15T22:20:31.250 INFO:tasks.rados.rados.0.smithi076.stdout:148: finishing write tid 5 to smithi07694878-21 2021-03-15T22:20:31.250 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 80 (ObjNum 81 snap 10 seq_num 81) dirty exists 2021-03-15T22:20:31.250 INFO:tasks.rados.rados.0.smithi076.stdout:148: left oid 21 (ObjNum 81 snap 10 seq_num 81) 2021-03-15T22:20:31.251 INFO:tasks.rados.rados.0.smithi076.stdout:149: finishing write tid 1 to smithi07694878-20 2021-03-15T22:20:31.251 INFO:tasks.rados.rados.0.smithi076.stdout:149: finishing write tid 2 to smithi07694878-20 2021-03-15T22:20:31.251 INFO:tasks.rados.rados.0.smithi076.stdout:149: finishing write tid 3 to smithi07694878-20 2021-03-15T22:20:31.251 INFO:tasks.rados.rados.0.smithi076.stdout:149: finishing write tid 4 to smithi07694878-20 2021-03-15T22:20:31.251 INFO:tasks.rados.rados.0.smithi076.stdout:149: finishing write tid 5 to smithi07694878-20 2021-03-15T22:20:31.252 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 30 (ObjNum 82 snap 10 seq_num 82) dirty exists 2021-03-15T22:20:31.252 INFO:tasks.rados.rados.0.smithi076.stdout:149: left oid 20 (ObjNum 82 snap 10 seq_num 82) 2021-03-15T22:20:31.312 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:31 smithi076 conmon[56514]: cluster 2021-03-15T22:20:30.021406+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:20:31.313 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:31 smithi076 conmon[56514]: cluster 2021-03-15T22:20:30.227191+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.d deep-scrub ok 2021-03-15T22:20:31.313 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:31 smithi076 conmon[56514]: cluster 2021-03-15T22:20:30.254974+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.a scrub ok 2021-03-15T22:20:31.313 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:31 smithi076 conmon[56514]: cluster 2021-03-15T22:20:30.455422+0000 osd.0 (osd.0) 31 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:20:31.313 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:31 smithi076 conmon[56514]: cluster 2021-03-15T22:20:30.982210+0000 mgr.y (mgr.14144) 380 : cluster [DBG] pgmap v300: 148 pgs: 148 active+clean; 135 MiB data, 335 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 10 op/s 2021-03-15T22:20:31.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:31 smithi076 conmon[62156]: cluster 2021-03-15T22:20:30.021406+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:20:31.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:31 smithi076 conmon[62156]: cluster 2021-03-15T22:20:30.227191+0000 osd.4 (osd.4) 27 : cluster [DBG] 4.d deep-scrub ok 2021-03-15T22:20:31.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:31 smithi076 conmon[62156]: cluster 2021-03-15T22:20:30.254974+0000 osd.5 (osd.5) 28 : cluster [DBG] 6.a scrub ok 2021-03-15T22:20:31.315 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:31 smithi076 conmon[62156]: cluster 2021-03-15T22:20:30.455422+0000 osd.0 (osd.0) 31 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:20:31.315 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:31 smithi076 conmon[62156]: cluster 2021-03-15T22:20:30.982210+0000 mgr.y (mgr.14144) 380 : cluster [DBG] pgmap v300: 148 pgs: 148 active+clean; 135 MiB data, 335 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 10 op/s 2021-03-15T22:20:31.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-15T22:20:31.347 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:31.371 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:31.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:31.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:31.420 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:31.474 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:31.500 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:31.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:31.502 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:31.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: cluster 2021-03-15T22:20:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: 2021- 2021-03-15T22:20:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: 03-15T22:20:30.021406+0000 osd.7 (osd.7) 27 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:20:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:20:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: 03-15T22:20:30.227191+0000 osd.4 (osd.4) 27 : 2021-03-15T22:20:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: cluster [DBG] 4.d deep-scrub ok 2021-03-15T22:20:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: cluster 2021-03-15T22:20:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: 2021-03-15T22:20:30.254974+0000 osd.5 2021-03-15T22:20:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: (osd.5) 28 : cluster [DBG] 6.a scrub ok 2021-03-15T22:20:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: cluster 2021-03-15T22:20:31.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: 2021-03-15T22:20:30.455422+0000 osd.0 2021-03-15T22:20:31.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: (osd.0) 31 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:20:31.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:20:31.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: 03-15T22:20:30.982210+0000 mgr.y (mgr.14144) 380 : cluster 2021-03-15T22:20:31.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:31 smithi125 conmon[56739]: [DBG] pgmap v300: 148 pgs: 148 active+clean; 135 MiB data, 335 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.6 MiB/s wr, 10 op/s 2021-03-15T22:20:31.599 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:20:31.603 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:31.616 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:20:31.627 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:31.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:31.635 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:20:31.653 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:20:31.671 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:20:31.691 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:20:31.712 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:20:31.729 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:20:31.730 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:31.731 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:20:31.731 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:20:31.731 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:20:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:20:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:20:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:20:31.732 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:20:31.733 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:20:31.754 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:31.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:31.857 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:31.883 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:31.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:31.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:31.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:32.008 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:32.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:32.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:32.061 INFO:tasks.rados.rados.0.smithi076.stdout:145: done (4 left) 2021-03-15T22:20:32.061 INFO:tasks.rados.rados.0.smithi076.stdout:147: done (3 left) 2021-03-15T22:20:32.062 INFO:tasks.rados.rados.0.smithi076.stdout:148: done (2 left) 2021-03-15T22:20:32.062 INFO:tasks.rados.rados.0.smithi076.stdout:149: done (1 left) 2021-03-15T22:20:32.062 INFO:tasks.rados.rados.0.smithi076.stdout:150: done (0 left) 2021-03-15T22:20:32.062 INFO:tasks.rados.rados.0.smithi076.stdout:151: read oid 29 snap 10 2021-03-15T22:20:32.063 INFO:tasks.rados.rados.0.smithi076.stdout:151: expect (ObjNum 23 snap 0 seq_num 23) 2021-03-15T22:20:32.063 INFO:tasks.rados.rados.0.smithi076.stdout:152: write (excl) oid 21 current snap is 11 2021-03-15T22:20:32.063 INFO:tasks.rados.rados.0.smithi076.stdout:152: seq_num 83 ranges {594981=609197,1974123=745028,3279873=115108} 2021-03-15T22:20:32.067 INFO:tasks.rados.rados.0.smithi076.stdout:152: writing smithi07694878-21 from 594981 to 1204178 tid 1 2021-03-15T22:20:32.075 INFO:tasks.rados.rados.0.smithi076.stdout:152: writing smithi07694878-21 from 1974123 to 2719151 tid 2 2021-03-15T22:20:32.079 INFO:tasks.rados.rados.0.smithi076.stdout:152: writing smithi07694878-21 from 3279873 to 3394981 tid 3 2021-03-15T22:20:32.080 INFO:tasks.rados.rados.0.smithi076.stdout:153: delete oid 23 current snap is 11 2021-03-15T22:20:32.080 INFO:tasks.rados.rados.0.smithi076.stdout:151: expect (ObjNum 23 snap 0 seq_num 23) 2021-03-15T22:20:32.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:32.137 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:32.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:32.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:32.157 INFO:tasks.rados.rados.0.smithi076.stdout:152: finishing write tid 1 to smithi07694878-21 2021-03-15T22:20:32.157 INFO:tasks.rados.rados.0.smithi076.stdout:152: finishing write tid 2 to smithi07694878-21 2021-03-15T22:20:32.158 INFO:tasks.rados.rados.0.smithi076.stdout:152: finishing write tid 3 to smithi07694878-21 2021-03-15T22:20:32.158 INFO:tasks.rados.rados.0.smithi076.stdout:152: finishing write tid 4 to smithi07694878-21 2021-03-15T22:20:32.158 INFO:tasks.rados.rados.0.smithi076.stdout:152: finishing write tid 5 to smithi07694878-21 2021-03-15T22:20:32.158 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 85 (ObjNum 83 snap 11 seq_num 83) dirty exists 2021-03-15T22:20:32.159 INFO:tasks.rados.rados.0.smithi076.stdout:152: left oid 21 (ObjNum 83 snap 11 seq_num 83) 2021-03-15T22:20:32.159 INFO:tasks.rados.rados.0.smithi076.stdout:151: done (2 left) 2021-03-15T22:20:32.159 INFO:tasks.rados.rados.0.smithi076.stdout:152: done (1 left) 2021-03-15T22:20:32.159 INFO:tasks.rados.rados.0.smithi076.stdout:153: done (0 left) 2021-03-15T22:20:32.160 INFO:tasks.rados.rados.0.smithi076.stdout:154: read oid 28 snap 8 2021-03-15T22:20:32.160 INFO:tasks.rados.rados.0.smithi076.stdout:154: expect (ObjNum 27 snap 0 seq_num 27) 2021-03-15T22:20:32.160 INFO:tasks.rados.rados.0.smithi076.stdout:155: snap_remove snap 10 2021-03-15T22:20:32.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:32.262 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:32.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:32.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:32.365 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:32.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:32.390 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:32.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:32.453 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:32 smithi076 conmon[56514]: cluster 2021-03-15T22:20:31.050960+0000 osd.7 (osd.7) 28 : cluster [DBG] 5.4 scrub ok 2021-03-15T22:20:32.453 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:32 smithi076 conmon[56514]: cluster 2021-03-15T22:20:31.052404+0000 mon.a (mon.0) 645 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2021-03-15T22:20:32.453 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:32 smithi076 conmon[56514]: audit 2021-03-15T22:20:31.574021+0000 mon.a (mon.0) 646 : audit [DBG] from='client.? 172.21.15.76:0/3701311681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:32.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:32 smithi076 conmon[62156]: cluster 2021-03-15T22:20:31.050960+0000 osd.7 (osd.7) 28 : cluster [DBG] 5.4 scrub ok 2021-03-15T22:20:32.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:32 smithi076 conmon[62156]: cluster 2021-03-15T22:20:31.052404+0000 mon.a (mon.0) 645 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2021-03-15T22:20:32.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:32 smithi076 conmon[62156]: audit 2021-03-15T22:20:31.574021+0000 mon.a (mon.0) 646 : audit [DBG] from='client.? 172.21.15.76:0/3701311681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:32.478 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:32.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:32.516 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:32.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:32.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:32 smithi125 conmon[56739]: cluster 2021-03-15T22:20:31.050960+0000 osd.7 (osd.7) 28 : cluster [DBG] 2021-03-15T22:20:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:32 smithi125 conmon[56739]: 5.4 scrub ok 2021-03-15T22:20:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:32 smithi125 conmon[56739]: cluster 2021-03-15T22:20:31.052404+0000 mon.a (mon.0) 645 : cluster [DBG] osdmap e116: 8 total, 8 up, 8 in 2021-03-15T22:20:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:32 smithi125 conmon[56739]: audit 2021-03-15T22:20:31.574021+0000 mon.a (mon.0) 646 : audit [DBG] 2021-03-15T22:20:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:32 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/3701311681' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:32.619 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:32.645 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:32.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:32.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:32.747 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:32.771 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:32.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:32.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:32.874 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:32.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:32.897 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:32.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:32.998 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:33.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:33.025 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:33.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:33.064 INFO:tasks.rados.rados.0.smithi076.stdout:155: done (1 left) 2021-03-15T22:20:33.065 INFO:tasks.rados.rados.0.smithi076.stdout:156: write oid 44 current snap is 11 2021-03-15T22:20:33.065 INFO:tasks.rados.rados.0.smithi076.stdout:156: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2021-03-15T22:20:33.070 INFO:tasks.rados.rados.0.smithi076.stdout:156: writing smithi07694878-44 from 628595 to 1104643 tid 1 2021-03-15T22:20:33.078 INFO:tasks.rados.rados.0.smithi076.stdout:156: writing smithi07694878-44 from 1807440 to 2439784 tid 2 2021-03-15T22:20:33.084 INFO:tasks.rados.rados.0.smithi076.stdout:156: writing smithi07694878-44 from 2934719 to 3387193 tid 3 2021-03-15T22:20:33.084 INFO:tasks.rados.rados.0.smithi076.stdout:157: read oid 34 snap -1 2021-03-15T22:20:33.084 INFO:tasks.rados.rados.0.smithi076.stdout:157: expect deleted 2021-03-15T22:20:33.084 INFO:tasks.rados.rados.0.smithi076.stdout:158: snap_create 2021-03-15T22:20:33.085 INFO:tasks.rados.rados.0.smithi076.stdout:154: expect (ObjNum 27 snap 0 seq_num 27) 2021-03-15T22:20:33.126 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:33.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:33.149 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:33.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:33.161 INFO:tasks.rados.rados.0.smithi076.stdout:156: finishing write tid 1 to smithi07694878-44 2021-03-15T22:20:33.161 INFO:tasks.rados.rados.0.smithi076.stdout:156: finishing write tid 2 to smithi07694878-44 2021-03-15T22:20:33.161 INFO:tasks.rados.rados.0.smithi076.stdout:156: finishing write tid 3 to smithi07694878-44 2021-03-15T22:20:33.161 INFO:tasks.rados.rados.0.smithi076.stdout:156: finishing write tid 4 to smithi07694878-44 2021-03-15T22:20:33.162 INFO:tasks.rados.rados.0.smithi076.stdout:156: finishing write tid 5 to smithi07694878-44 2021-03-15T22:20:33.162 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 38 (ObjNum 85 snap 11 seq_num 85) dirty exists 2021-03-15T22:20:33.162 INFO:tasks.rados.rados.0.smithi076.stdout:156: left oid 44 (ObjNum 85 snap 11 seq_num 85) 2021-03-15T22:20:33.255 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:33.278 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:33.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:33.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:33.348 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[56514]: cluster 2021-03-15T22:20:31.826146+0000 2021-03-15T22:20:33.348 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[56514]: osd.1 (osd.1) 27 : cluster [DBG] 5.c scrub ok 2021-03-15T22:20:33.349 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[56514]: cluster 2021-03-15T22:20:31.843837+0000 osd.6 (osd.6) 17 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:33.349 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[56514]: cluster 2021-03-15T22:20:31.843910+0000 osd.6 (osd.6) 18 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:33.349 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[56514]: cluster 2021-03-15T22:20:32.006103+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:20:33.349 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[56514]: cluster 2021-03-15T22:20:32.056052+0000 mon.a (mon.0) 647 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2021-03-15T22:20:33.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[56514]: cluster 2021-03-15T22:20:32.982576+0000 mgr.y (mgr.14144) 381 : cluster [DBG] pgmap v303: 148 pgs: 148 active+clean; 130 MiB data, 334 MiB used, 715 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.2 MiB/s wr, 28 op/s 2021-03-15T22:20:33.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[62156]: cluster 2021-03-15T22:20:31.826146+0000 osd.1 (osd.1) 27 : cluster [DBG] 5.c scrub ok 2021-03-15T22:20:33.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[62156]: cluster 2021-03-15T22:20:31.843837+0000 osd.6 (osd.6) 17 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:33.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[62156]: cluster 2021-03-15T22:20:31.843910+0000 osd.6 (osd.6) 18 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:33.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[62156]: cluster 2021-03-15T22:20:32.006103+0000 osd.7 (osd.7) 29 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:20:33.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[62156]: cluster 2021-03-15T22:20:32.056052+0000 mon.a (mon.0) 647 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2021-03-15T22:20:33.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:33 smithi076 conmon[62156]: cluster 2021-03-15T22:20:32.982576+0000 mgr.y (mgr.14144) 381 : cluster [DBG] pgmap v303: 148 pgs: 148 active+clean; 130 MiB data, 334 MiB used, 715 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.2 MiB/s wr, 28 op/s 2021-03-15T22:20:33.381 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:33.405 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:33.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:33.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:33.446 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:20:33.507 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:33.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:33.535 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:33.536 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:33.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: cluster 2021-03-15T22:20:31.826146+0000 osd.1 (osd.1) 27 : 2021-03-15T22:20:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: cluster [DBG] 5.c scrub ok 2021-03-15T22:20:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: cluster 2021-03-15T22:20:31.843837+0000 osd.6 (osd.6) 17 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: cluster 2021-03-15T22:20:31.843910+0000 osd.6 (osd.6) 18 : cluster 2021-03-15T22:20:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: [DBG] purged_snaps scrub ok 2021-03-15T22:20:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:20:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: -15T22:20:32.006103+0000 osd.7 ( 2021-03-15T22:20:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: osd.7) 29 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:20:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:20:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: 03-15T22:20:32.056052+0000 mon.a ( 2021-03-15T22:20:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: mon.0) 647 : cluster [DBG] osdmap e117: 8 total, 8 up, 8 in 2021-03-15T22:20:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:20:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: 22:20:32.982576+0000 mgr.y (mgr.14144) 381 : cluster 2021-03-15T22:20:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:33 smithi125 conmon[56739]: [DBG] pgmap v303: 148 pgs: 148 active+clean; 130 MiB data, 334 MiB used, 715 GiB / 715 GiB avail; 9.4 MiB/s rd, 4.2 MiB/s wr, 28 op/s 2021-03-15T22:20:33.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:33.660 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:33.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:33.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:33.763 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:33.784 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:33.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:33.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:33.887 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:33.914 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:33.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:33.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:34.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:34.043 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:34.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:34.072 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:34.073 INFO:tasks.rados.rados.0.smithi076.stdout:154: done (3 left) 2021-03-15T22:20:34.074 INFO:tasks.rados.rados.0.smithi076.stdout:156: done (2 left) 2021-03-15T22:20:34.074 INFO:tasks.rados.rados.0.smithi076.stdout:157: done (1 left) 2021-03-15T22:20:34.074 INFO:tasks.rados.rados.0.smithi076.stdout:158: done (0 left) 2021-03-15T22:20:34.075 INFO:tasks.rados.rados.0.smithi076.stdout:159: rollback oid 38 current snap is 12 2021-03-15T22:20:34.075 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 38 to 11 2021-03-15T22:20:34.075 INFO:tasks.rados.rados.0.smithi076.stdout:160: write oid 33 current snap is 12 2021-03-15T22:20:34.075 INFO:tasks.rados.rados.0.smithi076.stdout:160: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2021-03-15T22:20:34.081 INFO:tasks.rados.rados.0.smithi076.stdout:160: writing smithi07694878-33 from 645402 to 1196699 tid 1 2021-03-15T22:20:34.087 INFO:tasks.rados.rados.0.smithi076.stdout:160: writing smithi07694878-33 from 1749569 to 2241924 tid 2 2021-03-15T22:20:34.095 INFO:tasks.rados.rados.0.smithi076.stdout:160: writing smithi07694878-33 from 2845789 to 3445402 tid 3 2021-03-15T22:20:34.095 INFO:tasks.rados.rados.0.smithi076.stdout:161: read oid 34 snap 11 2021-03-15T22:20:34.096 INFO:tasks.rados.rados.0.smithi076.stdout:161: expect deleted 2021-03-15T22:20:34.096 INFO:tasks.rados.rados.0.smithi076.stdout:159: finishing rollback tid 0 to smithi07694878-38 2021-03-15T22:20:34.096 INFO:tasks.rados.rados.0.smithi076.stdout:159: finishing rollback tid 1 to smithi07694878-38 2021-03-15T22:20:34.096 INFO:tasks.rados.rados.0.smithi076.stdout:159: finishing rollback tid 2 to smithi07694878-38 2021-03-15T22:20:34.096 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 45 (ObjNum 60 snap 4 seq_num 60) dirty exists 2021-03-15T22:20:34.097 INFO:tasks.rados.rados.0.smithi076.stdout:160: finishing write tid 1 to smithi07694878-33 2021-03-15T22:20:34.097 INFO:tasks.rados.rados.0.smithi076.stdout:160: finishing write tid 2 to smithi07694878-33 2021-03-15T22:20:34.097 INFO:tasks.rados.rados.0.smithi076.stdout:159: done (2 left) 2021-03-15T22:20:34.097 INFO:tasks.rados.rados.0.smithi076.stdout:162: write (excl) oid 16 current snap is 12 2021-03-15T22:20:34.098 INFO:tasks.rados.rados.0.smithi076.stdout:162: seq_num 87 ranges {662209=626546,1808051=636013,2873212=588997} 2021-03-15T22:20:34.102 INFO:tasks.rados.rados.0.smithi076.stdout:162: writing smithi07694878-16 from 662209 to 1288755 tid 1 2021-03-15T22:20:34.109 INFO:tasks.rados.rados.0.smithi076.stdout:162: writing smithi07694878-16 from 1808051 to 2444064 tid 2 2021-03-15T22:20:34.116 INFO:tasks.rados.rados.0.smithi076.stdout:162: writing smithi07694878-16 from 2873212 to 3462209 tid 3 2021-03-15T22:20:34.117 INFO:tasks.rados.rados.0.smithi076.stdout:163: read oid 31 snap -1 2021-03-15T22:20:34.117 INFO:tasks.rados.rados.0.smithi076.stdout:163: expect (ObjNum 73 snap 8 seq_num 73) 2021-03-15T22:20:34.117 INFO:tasks.rados.rados.0.smithi076.stdout:164: delete oid 49 current snap is 12 2021-03-15T22:20:34.117 INFO:tasks.rados.rados.0.smithi076.stdout:160: finishing write tid 3 to smithi07694878-33 2021-03-15T22:20:34.118 INFO:tasks.rados.rados.0.smithi076.stdout:160: finishing write tid 4 to smithi07694878-33 2021-03-15T22:20:34.118 INFO:tasks.rados.rados.0.smithi076.stdout:160: finishing write tid 5 to smithi07694878-33 2021-03-15T22:20:34.118 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 34 (ObjNum 86 snap 12 seq_num 86) dirty exists 2021-03-15T22:20:34.120 INFO:tasks.rados.rados.0.smithi076.stdout:160: left oid 33 (ObjNum 86 snap 12 seq_num 86) 2021-03-15T22:20:34.120 INFO:tasks.rados.rados.0.smithi076.stdout:162: finishing write tid 1 to smithi07694878-16 2021-03-15T22:20:34.120 INFO:tasks.rados.rados.0.smithi076.stdout:162: finishing write tid 2 to smithi07694878-16 2021-03-15T22:20:34.122 INFO:tasks.rados.rados.0.smithi076.stdout:162: finishing write tid 3 to smithi07694878-16 2021-03-15T22:20:34.122 INFO:tasks.rados.rados.0.smithi076.stdout:162: finishing write tid 4 to smithi07694878-16 2021-03-15T22:20:34.123 INFO:tasks.rados.rados.0.smithi076.stdout:162: finishing write tid 5 to smithi07694878-16 2021-03-15T22:20:34.123 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 76 (ObjNum 87 snap 12 seq_num 87) dirty exists 2021-03-15T22:20:34.123 INFO:tasks.rados.rados.0.smithi076.stdout:162: left oid 16 (ObjNum 87 snap 12 seq_num 87) 2021-03-15T22:20:34.124 INFO:tasks.rados.rados.0.smithi076.stdout:160: done (4 left) 2021-03-15T22:20:34.124 INFO:tasks.rados.rados.0.smithi076.stdout:161: done (3 left) 2021-03-15T22:20:34.124 INFO:tasks.rados.rados.0.smithi076.stdout:162: done (2 left) 2021-03-15T22:20:34.124 INFO:tasks.rados.rados.0.smithi076.stdout:164: done (1 left) 2021-03-15T22:20:34.126 INFO:tasks.rados.rados.0.smithi076.stdout:165: write (excl) oid 24 current snap is 12 2021-03-15T22:20:34.126 INFO:tasks.rados.rados.0.smithi076.stdout:165: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2021-03-15T22:20:34.128 INFO:tasks.rados.rados.0.smithi076.stdout:165: writing smithi07694878-24 from 695823 to 1472867 tid 1 2021-03-15T22:20:34.134 INFO:tasks.rados.rados.0.smithi076.stdout:165: writing smithi07694878-24 from 1925015 to 2564697 tid 2 2021-03-15T22:20:34.139 INFO:tasks.rados.rados.0.smithi076.stdout:165: writing smithi07694878-24 from 3328058 to 3495823 tid 3 2021-03-15T22:20:34.140 INFO:tasks.rados.rados.0.smithi076.stdout:166: snap_create 2021-03-15T22:20:34.140 INFO:tasks.rados.rados.0.smithi076.stdout:165: finishing write tid 1 to smithi07694878-24 2021-03-15T22:20:34.140 INFO:tasks.rados.rados.0.smithi076.stdout:163: expect (ObjNum 73 snap 8 seq_num 73) 2021-03-15T22:20:34.146 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:34.170 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:34.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:34 smithi076 conmon[56514]: cluster 2021-03-15T22:20:32.860147+0000 osd.2 (osd.2) 23 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:20:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:34 smithi076 conmon[56514]: cluster 2021-03-15T22:20:33.062615+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2021-03-15T22:20:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:34 smithi076 conmon[56514]: cluster 2021-03-15T22:20:33.236972+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:20:34.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:34 smithi076 conmon[56514]: cluster 2021-03-15T22:20:33.569263+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:20:34.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:34 smithi076 conmon[62156]: cluster 2021-03-15T22:20:32.860147+0000 2021-03-15T22:20:34.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:34 smithi076 conmon[62156]: osd.2 (osd.2) 23 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:20:34.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:34 smithi076 conmon[62156]: cluster 2021-03-15T22:20:33.062615+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2021-03-15T22:20:34.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:34 smithi076 conmon[62156]: cluster 2021-03-15T22:20:33.236972+0000 osd.4 (osd.4) 28 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:20:34.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:34 smithi076 conmon[62156]: cluster 2021-03-15T22:20:33.569263+0000 osd.0 (osd.0) 32 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:20:34.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:34.293 INFO:tasks.rados.rados.0.smithi076.stdout:165: finishing write tid 2 to smithi07694878-24 2021-03-15T22:20:34.293 INFO:tasks.rados.rados.0.smithi076.stdout:165: finishing write tid 3 to smithi07694878-24 2021-03-15T22:20:34.294 INFO:tasks.rados.rados.0.smithi076.stdout:165: finishing write tid 4 to smithi07694878-24 2021-03-15T22:20:34.294 INFO:tasks.rados.rados.0.smithi076.stdout:165: finishing write tid 5 to smithi07694878-24 2021-03-15T22:20:34.294 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 8 (ObjNum 89 snap 12 seq_num 89) dirty exists 2021-03-15T22:20:34.294 INFO:tasks.rados.rados.0.smithi076.stdout:165: left oid 24 (ObjNum 89 snap 12 seq_num 89) 2021-03-15T22:20:34.298 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:34.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:34.300 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:34.401 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:34.423 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:34.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:34.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:34.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:34.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:34 smithi125 conmon[56739]: cluster 2021-03-15T22:20:32.860147+0000 osd.2 (osd.2) 23 : cluster [DBG] 2021-03-15T22:20:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:34 smithi125 conmon[56739]: 5.6 scrub ok 2021-03-15T22:20:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:34 smithi125 conmon[56739]: cluster 2021-03-15T22:20:33.062615+0000 mon.a (mon.0) 648 : cluster [DBG] osdmap e118: 8 total, 8 up, 8 in 2021-03-15T22:20:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:34 smithi125 conmon[56739]: cluster 2021-03-15T22:20:33.236972+0000 osd.4 (osd.4) 2021-03-15T22:20:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:34 smithi125 conmon[56739]: 28 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:20:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:34 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:20:34.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:34 smithi125 conmon[56739]: -15T22:20:33.569263+0000 osd.0 2021-03-15T22:20:34.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:34 smithi125 conmon[56739]: (osd.0) 32 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:20:34.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:34 smithi125 conmon[56739]: 2021-03-15T22:20:34.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:34.552 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:34.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:34.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:20:34.654 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:34.681 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:34.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:34.784 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:34.809 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:34.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:34.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:34.912 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:34.915 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:20:34.915 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:20:34.915 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:20:34.915 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:20:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:34.936 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:34.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:34.998 INFO:tasks.rados.rados.0.smithi076.stdout:163: done (2 left) 2021-03-15T22:20:34.999 INFO:tasks.rados.rados.0.smithi076.stdout:165: done (1 left) 2021-03-15T22:20:34.999 INFO:tasks.rados.rados.0.smithi076.stdout:166: done (0 left) 2021-03-15T22:20:34.999 INFO:tasks.rados.rados.0.smithi076.stdout:167: snap_create 2021-03-15T22:20:35.000 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.3 primary-affinity to 1 (8655362) 2021-03-15T22:20:35.039 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:35.061 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:35.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:35.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:35.164 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:35.190 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:35.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:35.191 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:35.293 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:35.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:35.324 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:35.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:35.371 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[62156]: cluster 2021-03-15T22:20:33.849079+0000 osd.1 (osd.1) 2021-03-15T22:20:35.372 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[62156]: 28 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:20:35.372 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[62156]: cluster 2021-03-15T22:20:33.853962+0000 osd.2 (osd.2) 24 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:20:35.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[62156]: cluster 2021-03-15T22:20:34.071952+0000 mon.a (mon.0) 649 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2021-03-15T22:20:35.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[62156]: cluster 2021-03-15T22:20:34.242174+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:20:35.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[62156]: audit 2021-03-15T22:20:34.431737+0000 mon.a (mon.0) 650 : audit [INF] from='client.? 172.21.15.76:0/2564438946' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-15T22:20:35.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[62156]: cluster 2021-03-15T22:20:34.983001+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v306: 148 pgs: 148 active+clean; 132 MiB data, 338 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.1 MiB/s wr, 33 op/s 2021-03-15T22:20:35.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[62156]: audit 2021-03-15T22:20:34.994650+0000 mon.a (mon.0) 651 : audit [INF] from='client.? 172.21.15.76:0/2564438946' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2021-03-15T22:20:35.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[62156]: cluster 2021-03-15T22:20:34.994712+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2021-03-15T22:20:35.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[56514]: cluster 2021-03-15T22:20:33.849079+0000 osd.1 ( 2021-03-15T22:20:35.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[56514]: osd.1) 28 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:20:35.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[56514]: cluster 2021-03-15T22:20:33.853962+0000 osd.2 (osd.2) 24 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:20:35.378 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[56514]: cluster 2021-03-15T22:20:34.071952+0000 mon.a (mon.0) 649 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2021-03-15T22:20:35.378 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[56514]: cluster 2021-03-15T22:20:34.242174+0000 osd.4 (osd.4) 29 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:20:35.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[56514]: audit 2021-03-15T22:20:34.431737+0000 mon.a (mon.0) 650 : audit [INF] from='client.? 172.21.15.76:0/2564438946' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-15T22:20:35.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[56514]: cluster 2021-03-15T22:20:34.983001+0000 mgr.y (mgr.14144) 382 : cluster [DBG] pgmap v306: 148 pgs: 148 active+clean; 132 MiB data, 338 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.1 MiB/s wr, 33 op/s 2021-03-15T22:20:35.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[56514]: audit 2021-03-15T22:20:34.994650+0000 mon.a (mon.0) 651 : audit [INF] from='client.? 172.21.15.76:0/2564438946' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2021-03-15T22:20:35.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:35 smithi076 conmon[56514]: cluster 2021-03-15T22:20:34.994712+0000 mon.a (mon.0) 652 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2021-03-15T22:20:35.426 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:35.448 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:35.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:35.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:35.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: cluster 2021-03-15T22:20:33.849079+0000 osd.1 (osd.1) 28 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:20:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: cluster 2021-03-15T22:20:33.853962+0000 osd.2 (osd.2) 24 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:20:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: cluster 2021-03-15T22:20:34.071952+0000 mon.a (mon.0) 649 : cluster [DBG] osdmap e119: 8 total, 8 up, 8 in 2021-03-15T22:20:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: cluster 2021-03-15T22:20:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: 2021-03-15T22:20:34.242174+0000 osd.4 (osd.4) 29 : cluster 2021-03-15T22:20:35.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: [DBG] 6.17 scrub ok 2021-03-15T22:20:35.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: audit 2021-03-15T22:20: 2021-03-15T22:20:35.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: 34.431737+0000 mon.a (mon.0) 650 : 2021-03-15T22:20:35.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: audit [INF] from='client.? 172.21.15.76:0/2564438946' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]: dispatch 2021-03-15T22:20:35.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:20:35.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: 03-15T22:20:34.983001+0000 2021-03-15T22:20:35.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: mgr.y (mgr.14144) 382 : 2021-03-15T22:20:35.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: cluster [DBG] pgmap v306: 148 pgs: 148 active+clean; 132 MiB data, 338 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 4.1 MiB/s wr, 33 op/s 2021-03-15T22:20:35.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: audit 2021 2021-03-15T22:20:35.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: -03-15T22:20:34.994650 2021-03-15T22:20:35.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2021-03-15T22:20:35.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: 651 : audit [INF] from='client.? 172.21.15.76:0/2564438946' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 3, "weight": 1.0}]': finished 2021-03-15T22:20:35.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: cluster 2021-03-15T22:20:35.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: 2021-03-15T22:20:34 2021-03-15T22:20:35.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: .994712+0000 mon.a (mon.0 2021-03-15T22:20:35.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:35 smithi125 conmon[56739]: ) 652 : cluster [DBG] osdmap e120: 8 total, 8 up, 8 in 2021-03-15T22:20:35.551 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:35.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:35.574 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:35.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:35.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:35.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:35.699 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:35.700 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:35.801 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:35.827 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:35.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:35.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:35.930 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:35.955 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:35.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:35.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:36.005 INFO:tasks.rados.rados.0.smithi076.stdout:167: done (0 left) 2021-03-15T22:20:36.006 INFO:tasks.rados.rados.0.smithi076.stdout:168: write (excl) oid 5 current snap is 14 2021-03-15T22:20:36.006 INFO:tasks.rados.rados.0.smithi076.stdout:168: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2021-03-15T22:20:36.011 INFO:tasks.rados.rados.0.smithi076.stdout:168: writing smithi07694878-5 from 712630 to 1164923 tid 1 2021-03-15T22:20:36.016 INFO:tasks.rados.rados.0.smithi076.stdout:168: writing smithi07694878-5 from 1583497 to 2083190 tid 2 2021-03-15T22:20:36.022 INFO:tasks.rados.rados.0.smithi076.stdout:168: writing smithi07694878-5 from 2555481 to 2980668 tid 3 2021-03-15T22:20:36.022 INFO:tasks.rados.rados.0.smithi076.stdout:169: read oid 22 snap -1 2021-03-15T22:20:36.023 INFO:tasks.rados.rados.0.smithi076.stdout:169: expect (ObjNum 48 snap 0 seq_num 48) 2021-03-15T22:20:36.023 INFO:tasks.rados.rados.0.smithi076.stdout:170: snap_remove snap 8 2021-03-15T22:20:36.058 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:36.084 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:36.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:36.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:36.187 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:36.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:36.212 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:36.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:36.224 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:36.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-15T22:20:36.299 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:36 smithi076 conmon[56514]: cluster 2021-03-15T22:20:35.200932+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:20:36.300 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:36 smithi076 conmon[56514]: cluster 2021-03-15T22:20:35.270121+0000 osd.4 (osd.4) 30 : cluster [DBG] 6.e scrub ok 2021-03-15T22:20:36.300 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:36 smithi076 conmon[56514]: cluster 2021-03-15T22:20:36.000741+0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2021-03-15T22:20:36.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:36 smithi076 conmon[62156]: cluster 2021-03-15T22:20:35.200932+0000 osd.5 ( 2021-03-15T22:20:36.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:36 smithi076 conmon[62156]: osd.5) 29 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:20:36.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:36 smithi076 conmon[62156]: cluster 2021-03-15T22:20:35.270121+0000 osd.4 (osd.4) 30 : cluster 2021-03-15T22:20:36.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:36 smithi076 conmon[62156]: [DBG] 6.e scrub ok 2021-03-15T22:20:36.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:36 smithi076 conmon[62156]: cluster 2021-03-15T22:20:36. 2021-03-15T22:20:36.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:36 smithi076 conmon[62156]: 000741+0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2021-03-15T22:20:36.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:36.341 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:36.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:36.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:36.444 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:36.471 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:36.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:36.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:36 smithi125 conmon[56739]: cluster 2021-03-15T22:20:35.200932+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:20:36.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:36 smithi125 conmon[56739]: cluster 2021-03-15T22:20:35.270121+0000 osd.4 (osd.4) 30 : cluster [DBG] 6.e scrub ok 2021-03-15T22:20:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:36 smithi125 conmon[56739]: cluster 2021-03-15T22:20:36. 2021-03-15T22:20:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:36 smithi125 conmon[56739]: 000741+0000 mon.a (mon.0) 653 : cluster [DBG] osdmap e121: 8 total, 8 up, 8 in 2021-03-15T22:20:36.574 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:36.600 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:36.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:36.601 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:36.702 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:36.727 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:36.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:36.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:36.830 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:36.856 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:36.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:36.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:36.959 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:36.964 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:36.984 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:36.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:36.986 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:37.008 INFO:tasks.rados.rados.0.smithi076.stdout:170: done (2 left) 2021-03-15T22:20:37.009 INFO:tasks.rados.rados.0.smithi076.stdout:171: copy_from oid 47 from oid 43 current snap is 14 2021-03-15T22:20:37.009 INFO:tasks.rados.rados.0.smithi076.stdout:168: finishing write tid 1 to smithi07694878-5 2021-03-15T22:20:37.009 INFO:tasks.rados.rados.0.smithi076.stdout:168: finishing write tid 2 to smithi07694878-5 2021-03-15T22:20:37.009 INFO:tasks.rados.rados.0.smithi076.stdout:168: finishing write tid 3 to smithi07694878-5 2021-03-15T22:20:37.009 INFO:tasks.rados.rados.0.smithi076.stdout:168: finishing write tid 4 to smithi07694878-5 2021-03-15T22:20:37.010 INFO:tasks.rados.rados.0.smithi076.stdout:168: finishing write tid 5 to smithi07694878-5 2021-03-15T22:20:37.010 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 8 (ObjNum 90 snap 14 seq_num 90) dirty exists 2021-03-15T22:20:37.010 INFO:tasks.rados.rados.0.smithi076.stdout:168: left oid 5 (ObjNum 90 snap 14 seq_num 90) 2021-03-15T22:20:37.010 INFO:tasks.rados.rados.0.smithi076.stdout:169: expect (ObjNum 48 snap 0 seq_num 48) 2021-03-15T22:20:37.016 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:20:37.088 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:37.108 INFO:tasks.rados.rados.0.smithi076.stdout:168: done (2 left) 2021-03-15T22:20:37.109 INFO:tasks.rados.rados.0.smithi076.stdout:169: done (1 left) 2021-03-15T22:20:37.109 INFO:tasks.rados.rados.0.smithi076.stdout:172: snap_create 2021-03-15T22:20:37.109 INFO:tasks.rados.rados.0.smithi076.stdout:171: finishing copy_from to smithi07694878-47 2021-03-15T22:20:37.109 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 14 (ObjNum 63 snap 6 seq_num 63) dirty exists 2021-03-15T22:20:37.110 INFO:tasks.rados.rados.0.smithi076.stdout:171: finishing copy_from racing read to smithi07694878-47 2021-03-15T22:20:37.114 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:37.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:37.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:37.216 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:37.240 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:37.242 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:37.266 INFO:journalctl@ceph.osd.1.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[68886]: debug 2021-03-15T22:20:37.009+0000 7fae34dad700 -1 osd.1 pg_epoch: 122 pg[7.b( v 118'24 (0'0,118'24] local-lis/les=91/92 n=4 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [1,5,6] r=0 lpr=91 crt=118'24 lcod 117'22 mlcod 117'22 active+clean+snaptrim trimq=[11~1] ps=[a~1,12~1,15~1]] removing snap head 2021-03-15T22:20:37.267 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[75764]: debug 2021-03-15T22:20:37.014+0000 7f9fba7ae700 -1 osd.3 pg_epoch: 122 pg[7.3( v 116'30 (0'0,116'30] local-lis/les=91/92 n=5 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [3,2,4] r=0 lpr=91 crt=116'30 lcod 115'28 mlcod 115'28 active+clean+snaptrim trimq=[11~1] ps=[a~1,12~1,15~1]] removing snap head 2021-03-15T22:20:37.267 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[75764]: debug 2021-03-15T22:20:37.014+0000 7f9fbafaf700 -1 osd.3 pg_epoch: 122 pg[7.a( v 119'45 (0'0,119'45] local-lis/les=91/92 n=7 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [3,2,5] r=0 lpr=91 crt=119'45 lcod 119'44 mlcod 119'44 active+clean+snaptrim trimq=[11~1] ps=[a~1,12~1,15~1]] removing snap head 2021-03-15T22:20:37.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: cluster 2021-03-15T22:20:35.937757+0000 osd.1 2021-03-15T22:20:37.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: (osd.1) 29 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:20:37.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: cluster 2021-03-15T22:20:35.969898+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:20:37.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: cluster 2021-03-15T22:20:36.215032+0000 osd.5 (osd.5) 30 : cluster [DBG] 2021-03-15T22:20:37.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: 5.7 deep-scrub ok 2021-03-15T22:20:37.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: cluster 2021-03-15T22:20:36.258793+0000 osd.4 (osd.4) 31 : cluster 2021-03-15T22:20:37.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: [DBG] 4.1b deep-scrub ok 2021-03-15T22:20:37.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: audit 2021-03-15T22:20 2021-03-15T22:20:37.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: :36.836120+0000 mon.a (mon.0) 654 : audit [INF] from='client.? 172.21.15.76:0/1340941090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:20:37.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: cluster 2021-03-15T22:20: 2021-03-15T22:20:37.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: 36.983533+0000 mgr.y (mgr.14144) 383 : cluster [DBG] 2021-03-15T22:20:37.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: pgmap v309: 148 pgs: 148 active+clean; 139 MiB data, 348 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 21 op/s 2021-03-15T22:20:37.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:20:37.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: T22:20:36.998157+0000 mon.a (mon.0) 655 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:37.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: audit 2021-03-15T22 2021-03-15T22:20:37.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: :20:37.004813+0000 mon.a (mon.0) 656 : audit [INF] 2021-03-15T22:20:37.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/1340941090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:20:37.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:20:37.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: 03-15T22:20:37.004870+0000 mon.a (mon.0) 657 : cluster [DBG] 2021-03-15T22:20:37.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[56514]: osdmap e122: 8 total, 8 up, 8 in 2021-03-15T22:20:37.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: cluster 2021-03-15T22:20:35.937757 2021-03-15T22:20:37.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 29 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:20:37.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:20:37.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: 22:20:35.969898+0000 osd.7 (osd.7) 2021-03-15T22:20:37.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: 30 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:20:37.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: cluster 2021-03-15T22:20:37.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: 2021-03-15T22:20:36.215032 2021-03-15T22:20:37.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 30 : cluster [DBG] 5.7 deep-scrub ok 2021-03-15T22:20:37.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: 2021-03-15T22:20:37.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:20:37.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: 22:20:36.258793+0000 osd.4 (osd.4) 31 : cluster 2021-03-15T22:20:37.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: [DBG] 4.1b deep-scrub ok 2021-03-15T22:20:37.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:20:37.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: 03-15T22:20:36.836120+0000 2021-03-15T22:20:37.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: mon.a (mon.0) 654 : audit [INF] from='client.? 172.21.15.76:0/1340941090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:20:37.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: 2021-03-15T22:20:37.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:20:37.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: -03-15T22:20:36.983533+0000 mgr.y (mgr.14144) 2021-03-15T22:20:37.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: 383 : cluster [DBG] pgmap v309: 148 pgs: 148 active+clean; 139 MiB data, 348 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 21 op/s 2021-03-15T22:20:37.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:20:37.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: -15T22:20:36.998157+0000 mon.a 2021-03-15T22:20:37.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: (mon.0) 655 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:37.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:20:37.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: T22:20:37.004813+0000 mon.a (mon. 2021-03-15T22:20:37.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: 0) 656 : audit [INF] from='client.? 172.21.15.76:0/1340941090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:20:37.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:20:37.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: -03-15T22:20:37.004870+0000 2021-03-15T22:20:37.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: mon.a (mon.0) 657 : cluster [DBG] 2021-03-15T22:20:37.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:37 smithi076 conmon[62156]: osdmap e122: 8 total, 8 up, 8 in 2021-03-15T22:20:37.342 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:37.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:37.368 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:37.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:37.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:37.496 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:37.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:37.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: cluster 2021-03-15T22:20:35.937757+0000 osd.1 (osd.1) 29 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:20:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: cluster 2021-03-15T22:20:35.969898+0000 osd.7 (osd.7) 30 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:20:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: cluster 2021-03-15T22:20:36.215032+0000 osd.5 (osd.5) 30 : cluster [DBG] 2021-03-15T22:20:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 5.7 deep-scrub ok 2021-03-15T22:20:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: cluster 2021-03-15T22:20:37.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 2021-03-15T22:20:36. 2021-03-15T22:20:37.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 258793+0000 osd.4 (osd.4) 2021-03-15T22:20:37.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 31 : cluster [DBG] 4.1b deep-scrub ok 2021-03-15T22:20:37.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:20:37.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: :20:36.836120+0000 mon.a (mon. 2021-03-15T22:20:37.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 0) 654 : audit [INF] from='client.? 172.21.15.76:0/1340941090' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:20:37.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 2021-03-15T22:20:37.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:20:37.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: -03-15T22:20: 2021-03-15T22:20:37.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 36.983533+0000 mgr.y (mgr. 2021-03-15T22:20:37.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 14144) 383 : cluster [DBG] 2021-03-15T22:20:37.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: pgmap v309: 148 pgs: 148 active+clean; 139 MiB data, 348 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 21 op/s 2021-03-15T22:20:37.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:20:37.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: -03-15T22:20: 2021-03-15T22:20:37.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 36.998157+0000 mon.a (mon. 2021-03-15T22:20:37.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 0) 655 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:37.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 2021-03-15T22:20:37.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: audit 2021-03-15T22:20:37.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:20:37.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 20:37.004813+0000 mon.a ( 2021-03-15T22:20:37.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: mon.0) 656 : audit [INF] 2021-03-15T22:20:37.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1340941090' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:20:37.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:20:37.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: -03-15T22:20:37. 2021-03-15T22:20:37.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 004870+0000 mon.a (mon.0) 2021-03-15T22:20:37.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:37 smithi125 conmon[56739]: 657 : cluster [DBG] osdmap e122: 8 total, 8 up, 8 in 2021-03-15T22:20:37.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:37.627 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:37.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:37.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:37.730 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:37.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:37.755 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:37.755 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:37.857 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:37.882 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:37.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:37.884 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:37.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:38.011 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:38.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:38.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:38.089 INFO:tasks.rados.rados.0.smithi076.stdout:171: done (1 left) 2021-03-15T22:20:38.090 INFO:tasks.rados.rados.0.smithi076.stdout:172: done (0 left) 2021-03-15T22:20:38.090 INFO:tasks.rados.rados.0.smithi076.stdout:173: write (excl) oid 42 current snap is 15 2021-03-15T22:20:38.090 INFO:tasks.rados.rados.0.smithi076.stdout:173: seq_num 91 ranges {729437=527542,1925626=643351,3266551=262886} 2021-03-15T22:20:38.096 INFO:tasks.rados.rados.0.smithi076.stdout:173: writing smithi07694878-42 from 729437 to 1256979 tid 1 2021-03-15T22:20:38.103 INFO:tasks.rados.rados.0.smithi076.stdout:173: writing smithi07694878-42 from 1925626 to 2568977 tid 2 2021-03-15T22:20:38.109 INFO:tasks.rados.rados.0.smithi076.stdout:173: writing smithi07694878-42 from 3266551 to 3529437 tid 3 2021-03-15T22:20:38.110 INFO:tasks.rados.rados.0.smithi076.stdout:174: copy_from oid 6 from oid 31 current snap is 15 2021-03-15T22:20:38.110 INFO:tasks.rados.rados.0.smithi076.stdout:173: finishing write tid 1 to smithi07694878-42 2021-03-15T22:20:38.110 INFO:tasks.rados.rados.0.smithi076.stdout:173: finishing write tid 2 to smithi07694878-42 2021-03-15T22:20:38.110 INFO:tasks.rados.rados.0.smithi076.stdout:175: snap_remove snap 12 2021-03-15T22:20:38.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:38.138 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:38.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:38.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:38.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:38.263 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:38.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:38.265 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:38.366 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:38.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:38 smithi076 conmon[56514]: cluster 2021-03-15T22:20:36.923902+0000 osd.7 (osd.7) 31 : cluster 2021-03-15T22:20:38.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:38 smithi076 conmon[56514]: [DBG] 2.0 scrub ok 2021-03-15T22:20:38.381 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:38 smithi076 conmon[62156]: cluster 2021-03-15T22:20:36.923902+0000 osd.7 (osd.7) 2021-03-15T22:20:38.381 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:38 smithi076 conmon[62156]: 31 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:20:38.389 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:38.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:38.391 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:38.492 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:38.518 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:38.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:38.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:38.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:38 smithi125 conmon[56739]: cluster 2021-03-15T22:20:36.923902+0000 osd.7 (osd.7) 31 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:20:38.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:38.648 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:38.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:38.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:38.750 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:38.777 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:38.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:38.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:38.880 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:38.905 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:38.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:38.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:39.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:39.032 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:39.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:39.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:39.096 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:39 smithi076 conmon[62156]: cluster 2021-03-15T22:20:38.087332+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2021-03-15T22:20:39.096 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:39 smithi076 conmon[62156]: cluster 2021-03-15T22:20:38.983928+0000 mgr.y (mgr.14144) 384 : cluster [DBG] pgmap v312: 148 pgs: 148 active+clean; 135 MiB data, 351 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-03-15T22:20:39.098 INFO:tasks.rados.rados.0.smithi076.stdout:175: done (2 left) 2021-03-15T22:20:39.099 INFO:tasks.rados.rados.0.smithi076.stdout:176: copy_from oid 15 from oid 7 current snap is 15 2021-03-15T22:20:39.099 INFO:tasks.rados.rados.0.smithi076.stdout:173: finishing write tid 3 to smithi07694878-42 2021-03-15T22:20:39.099 INFO:tasks.rados.rados.0.smithi076.stdout:173: finishing write tid 4 to smithi07694878-42 2021-03-15T22:20:39.099 INFO:tasks.rados.rados.0.smithi076.stdout:173: finishing write tid 5 to smithi07694878-42 2021-03-15T22:20:39.100 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 96 (ObjNum 91 snap 15 seq_num 91) dirty exists 2021-03-15T22:20:39.100 INFO:tasks.rados.rados.0.smithi076.stdout:173: left oid 42 (ObjNum 91 snap 15 seq_num 91) 2021-03-15T22:20:39.100 INFO:tasks.rados.rados.0.smithi076.stdout:174: finishing copy_from to smithi07694878-6 2021-03-15T22:20:39.100 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 16 (ObjNum 73 snap 8 seq_num 73) dirty exists 2021-03-15T22:20:39.100 INFO:tasks.rados.rados.0.smithi076.stdout:174: finishing copy_from racing read to smithi07694878-6 2021-03-15T22:20:39.101 INFO:tasks.rados.rados.0.smithi076.stdout:173: done (2 left) 2021-03-15T22:20:39.101 INFO:tasks.rados.rados.0.smithi076.stdout:174: done (1 left) 2021-03-15T22:20:39.101 INFO:tasks.rados.rados.0.smithi076.stdout:177: delete oid 39 current snap is 15 2021-03-15T22:20:39.104 INFO:tasks.rados.rados.0.smithi076.stdout:177: done (1 left) 2021-03-15T22:20:39.104 INFO:tasks.rados.rados.0.smithi076.stdout:178: rollback oid 19 current snap is 15 2021-03-15T22:20:39.105 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 19 to 14 2021-03-15T22:20:39.105 INFO:tasks.rados.rados.0.smithi076.stdout:179: delete oid 11 current snap is 15 2021-03-15T22:20:39.105 INFO:tasks.rados.rados.0.smithi076.stdout:178: finishing rollback tid 1 to smithi07694878-19 2021-03-15T22:20:39.105 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 0 (ObjNum 0 snap 21896 seq_num 261855536) dirty dne 2021-03-15T22:20:39.107 INFO:tasks.rados.rados.0.smithi076.stdout:178: done (2 left) 2021-03-15T22:20:39.107 INFO:tasks.rados.rados.0.smithi076.stdout:179: done (1 left) 2021-03-15T22:20:39.107 INFO:tasks.rados.rados.0.smithi076.stdout:180: snap_create 2021-03-15T22:20:39.109 INFO:tasks.rados.rados.0.smithi076.stdout:176: finishing copy_from to smithi07694878-15 2021-03-15T22:20:39.109 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:20:39.110 INFO:tasks.rados.rados.0.smithi076.stdout:176: finishing copy_from racing read to smithi07694878-15 2021-03-15T22:20:39.134 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:39.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:39.159 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:39.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:39.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:39.286 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:39.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:39.287 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:39.356 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:39 smithi076 conmon[56514]: cluster 2021-03-15T22:20:38.087332+0000 mon.a (mon.0) 658 : cluster [DBG] osdmap e123: 8 total, 8 up, 8 in 2021-03-15T22:20:39.357 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:39 smithi076 conmon[56514]: cluster 2021-03-15T22:20:38.983928+0000 mgr.y (mgr.14144) 384 : cluster [DBG] pgmap v312: 148 pgs: 148 active+clean; 135 MiB data, 351 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-03-15T22:20:39.388 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:39.396 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:20:39.412 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:39.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:39.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:39.416 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:20:39.433 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:20:39.437 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:39 smithi125 conmon[56739]: cluster 2021-03-15T22:20:38.087332+0000 mon.a (mon.0) 658 : cluster [DBG] 2021-03-15T22:20:39.438 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:39 smithi125 conmon[56739]: osdmap e123: 8 total, 8 up, 8 in 2021-03-15T22:20:39.438 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:39 smithi125 conmon[56739]: cluster 2021-03-15T22:20:38.983928+0000 mgr.y (mgr.14144) 384 : cluster [DBG] pgmap v312: 148 pgs: 148 active+clean; 135 MiB data, 351 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-03-15T22:20:39.453 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:20:39.471 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:20:39.490 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:20:39.508 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:20:39.515 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:39.527 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:20:39.527 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:20:39.528 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:20:39.528 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:20:39.528 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:20:39.528 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:20:39.529 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:20:39.529 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:20:39.529 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:20:39.542 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:39.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:39.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:39.645 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:39.669 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:39.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:39.671 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:39.679 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:20:39.680 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:20:39.772 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:39.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:20:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:20:39.798 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:39.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:39.901 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:39.926 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:39.926 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:40.003 INFO:tasks.rados.rados.0.smithi076.stdout:176: done (1 left) 2021-03-15T22:20:40.003 INFO:tasks.rados.rados.0.smithi076.stdout:180: done (0 left) 2021-03-15T22:20:40.003 INFO:tasks.rados.rados.0.smithi076.stdout:181: write (excl) oid 12 current snap is 16 2021-03-15T22:20:40.004 INFO:tasks.rados.rados.0.smithi076.stdout:181: seq_num 94 ranges {779858=469642,1817425=507031,2781526=586628} 2021-03-15T22:20:40.009 INFO:tasks.rados.rados.0.smithi076.stdout:181: writing smithi07694878-12 from 779858 to 1249500 tid 1 2021-03-15T22:20:40.016 INFO:tasks.rados.rados.0.smithi076.stdout:181: writing smithi07694878-12 from 1817425 to 2324456 tid 2 2021-03-15T22:20:40.023 INFO:tasks.rados.rados.0.smithi076.stdout:181: writing smithi07694878-12 from 2781526 to 3368154 tid 3 2021-03-15T22:20:40.023 INFO:tasks.rados.rados.0.smithi076.stdout:182: snap_create 2021-03-15T22:20:40.023 INFO:tasks.rados.rados.0.smithi076.stdout:181: finishing write tid 1 to smithi07694878-12 2021-03-15T22:20:40.025 INFO:tasks.rados.rados.0.smithi076.stdout:181: finishing write tid 2 to smithi07694878-12 2021-03-15T22:20:40.026 INFO:tasks.rados.rados.0.smithi076.stdout:181: finishing write tid 3 to smithi07694878-12 2021-03-15T22:20:40.026 INFO:tasks.rados.rados.0.smithi076.stdout:181: finishing write tid 4 to smithi07694878-12 2021-03-15T22:20:40.027 INFO:tasks.rados.rados.0.smithi076.stdout:181: finishing write tid 5 to smithi07694878-12 2021-03-15T22:20:40.027 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 91 (ObjNum 94 snap 16 seq_num 94) dirty exists 2021-03-15T22:20:40.027 INFO:tasks.rados.rados.0.smithi076.stdout:181: left oid 12 (ObjNum 94 snap 16 seq_num 94) 2021-03-15T22:20:40.027 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:40.051 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:40.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:40.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:40.154 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:40.178 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:40.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:40.179 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:40.185 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:20:40.280 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:40.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:40.303 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:40.304 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:40.405 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:40.431 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:40.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:40.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:40.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[56514]: cluster 2021-03-15T22:20:39.096365+0000 mon.a (mon.0) 659 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2021-03-15T22:20:40.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[56514]: cluster 2021-03-15T22:20:39.289751+0000 osd.3 (osd.3) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:40.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[56514]: cluster 2021-03-15T22:20:39.289834+0000 osd.3 (osd.3) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:40.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[56514]: audit 2021-03-15T22:20:39.376197+0000 mon.a (mon.0) 660 : audit [DBG] from='client.? 172.21.15.76:0/2159513538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:40.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[56514]: audit 2021-03-15T22:20:39.436525+0000 mgr.y (mgr.14144) 385 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:40.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[56514]: cluster 2021-03-15T22:20:40.000353+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2021-03-15T22:20:40.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[62156]: cluster 2021-03-15T22:20:39.096365+0000 mon.a (mon.0) 659 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2021-03-15T22:20:40.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[62156]: cluster 2021-03-15T22:20:39.289751+0000 osd.3 (osd.3) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:40.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[62156]: cluster 2021-03-15T22:20:39.289834+0000 osd.3 (osd.3) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:40.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[62156]: audit 2021-03-15T22:20:39.376197+0000 mon.a (mon.0) 660 : audit [DBG] from='client.? 172.21.15.76:0/2159513538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:40.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[62156]: audit 2021-03-15T22:20:39.436525+0000 mgr.y (mgr.14144) 385 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:40.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:40 smithi076 conmon[62156]: cluster 2021-03-15T22:20:40.000353+0000 mon.a (mon.0) 661 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2021-03-15T22:20:40.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: cluster 2021-03-15T22:20:39.096365+0000 mon.a (mon.0) 659 : cluster [DBG] osdmap e124: 8 total, 8 up, 8 in 2021-03-15T22:20:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: cluster 2021-03-15T22:20: 2021-03-15T22:20:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: 39.289751+0000 osd.3 (osd.3) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: cluster 2021-03-15T22:20:39.289834+0000 osd.3 (osd.3) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: audit 2021-03-15T22:20:39.376197+0000 mon.a ( 2021-03-15T22:20:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: mon.0) 660 : audit [DBG] from='client.? 172.21.15.76:0/2159513538' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: audit 2021 2021-03-15T22:20:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: -03-15T22:20:39.436525 2021-03-15T22:20:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 385 : audit 2021-03-15T22:20:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:20:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: 22:20:40.000353+0000 mon.a ( 2021-03-15T22:20:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:40 smithi125 conmon[56739]: mon.0) 661 : cluster [DBG] osdmap e125: 8 total, 8 up, 8 in 2021-03-15T22:20:40.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:40.561 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:40.561 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:40.663 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:40.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:40.689 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:40.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:40.726 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-15T22:20:40.765 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:40.790 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:40.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:40.834 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:40.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:40.936 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:40.961 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:40.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:40.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:41.012 INFO:tasks.rados.rados.0.smithi076.stdout:181: done (1 left) 2021-03-15T22:20:41.013 INFO:tasks.rados.rados.0.smithi076.stdout:182: done (0 left) 2021-03-15T22:20:41.013 INFO:tasks.rados.rados.0.smithi076.stdout:183: delete oid 7 current snap is 17 2021-03-15T22:20:41.017 INFO:tasks.rados.rados.0.smithi076.stdout:183: done (0 left) 2021-03-15T22:20:41.018 INFO:tasks.rados.rados.0.smithi076.stdout:184: snap_create 2021-03-15T22:20:41.046 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:20:41.047 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:20:41.047 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.control 2021-03-15T22:20:41.048 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.control by 10 min 8 2021-03-15T22:20:41.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:20:41.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:41.089 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:41.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:41.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:41.192 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:41.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:41.219 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:41.219 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:41.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:41 smithi076 conmon[56514]: cluster 2021-03-15T22:20:40.984337+0000 2021-03-15T22:20:41.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:41 smithi076 conmon[56514]: mgr.y (mgr.14144) 386 : cluster [DBG] pgmap v315: 148 pgs: 148 active+clean; 135 MiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 747 KiB/s wr, 1 op/s 2021-03-15T22:20:41.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:41 smithi076 conmon[56514]: cluster 2021-03-15T22:20:41.009560+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2021-03-15T22:20:41.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:41 smithi076 conmon[62156]: cluster 2021-03-15T22:20: 2021-03-15T22:20:41.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:41 smithi076 conmon[62156]: 40.984337+0000 mgr.y (mgr.14144) 386 : cluster [DBG] pgmap v315: 148 pgs: 148 active+clean; 135 MiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 747 KiB/s wr, 1 op/s 2021-03-15T22:20:41.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:41 smithi076 conmon[62156]: cluster 2021-03-15T22:20:41.009560+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2021-03-15T22:20:41.256 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:41.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:41.344 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:41.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:41.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:41.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:41.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:41.473 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:41.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:41.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:41 smithi125 conmon[56739]: cluster 2021-03-15T22:20:40.984337+0000 mgr.y (mgr.14144) 386 : cluster [DBG] 2021-03-15T22:20:41.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:41 smithi125 conmon[56739]: pgmap v315: 148 pgs: 148 active+clean; 135 MiB data, 354 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 747 KiB/s wr, 1 op/s 2021-03-15T22:20:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:41 smithi125 conmon[56739]: cluster 2021-03-15T22:20:41.009560+0000 mon.a (mon.0) 662 : cluster [DBG] osdmap e126: 8 total, 8 up, 8 in 2021-03-15T22:20:41.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:41.602 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:41.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:41.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:41.705 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:41.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:41.729 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:41.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:41.830 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:41.863 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:41.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:41.966 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:41.993 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:41.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:41.994 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:42.012 INFO:tasks.rados.rados.0.smithi076.stdout:184: done (0 left) 2021-03-15T22:20:42.013 INFO:tasks.rados.rados.0.smithi076.stdout:185: read oid 32 snap 16 2021-03-15T22:20:42.014 INFO:tasks.rados.rados.0.smithi076.stdout:185: expect (ObjNum 34 snap 0 seq_num 34) 2021-03-15T22:20:42.015 INFO:tasks.rados.rados.0.smithi076.stdout:186: snap_create 2021-03-15T22:20:42.020 INFO:tasks.rados.rados.0.smithi076.stdout:185: expect (ObjNum 34 snap 0 seq_num 34) 2021-03-15T22:20:42.080 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:42.096 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:42.123 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:42.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:42.125 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:42.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:42.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:42.253 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:42.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:42.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:42.381 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:42.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:42.383 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:42.484 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:42.508 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:42.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:42.509 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:42.611 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:42.633 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:42.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:42.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:42.691 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:20:42.736 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:42.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:42.762 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:42.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:42.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:42.890 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:42.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:42.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:42.994 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:43.012 INFO:tasks.rados.rados.0.smithi076.stdout:185: done (1 left) 2021-03-15T22:20:43.012 INFO:tasks.rados.rados.0.smithi076.stdout:186: done (0 left) 2021-03-15T22:20:43.013 INFO:tasks.rados.rados.0.smithi076.stdout:187: copy_from oid 19 from oid 22 current snap is 19 2021-03-15T22:20:43.013 INFO:tasks.rados.rados.0.smithi076.stdout:188: read oid 21 snap 14 2021-03-15T22:20:43.013 INFO:tasks.rados.rados.0.smithi076.stdout:188: expect (ObjNum 83 snap 11 seq_num 83) 2021-03-15T22:20:43.013 INFO:tasks.rados.rados.0.smithi076.stdout:189: write oid 49 current snap is 19 2021-03-15T22:20:43.014 INFO:tasks.rados.rados.0.smithi076.stdout:189: seq_num 96 ranges {413472=620140,1818036=510700,3120019=493453} 2021-03-15T22:20:43.014 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:20:43.018 INFO:tasks.rados.rados.0.smithi076.stdout:189: writing smithi07694878-49 from 413472 to 1033612 tid 1 2021-03-15T22:20:43.021 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:43.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:43.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:43.026 INFO:tasks.rados.rados.0.smithi076.stdout:189: writing smithi07694878-49 from 1818036 to 2328736 tid 2 2021-03-15T22:20:43.034 INFO:tasks.rados.rados.0.smithi076.stdout:189: writing smithi07694878-49 from 3120019 to 3613472 tid 3 2021-03-15T22:20:43.034 INFO:tasks.rados.rados.0.smithi076.stdout:190: read oid 24 snap -1 2021-03-15T22:20:43.035 INFO:tasks.rados.rados.0.smithi076.stdout:190: expect (ObjNum 89 snap 12 seq_num 89) 2021-03-15T22:20:43.035 INFO:tasks.rados.rados.0.smithi076.stdout:191: delete oid 43 current snap is 19 2021-03-15T22:20:43.036 INFO:tasks.rados.rados.0.smithi076.stdout:189: finishing write tid 1 to smithi07694878-49 2021-03-15T22:20:43.036 INFO:tasks.rados.rados.0.smithi076.stdout:188: expect (ObjNum 83 snap 11 seq_num 83) 2021-03-15T22:20:43.124 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:43.148 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:43.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:43.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:43.235 INFO:tasks.rados.rados.0.smithi076.stdout:187: finishing copy_from to smithi07694878-19 2021-03-15T22:20:43.236 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 21 (ObjNum 48 snap 0 seq_num 48) dirty exists 2021-03-15T22:20:43.236 INFO:tasks.rados.rados.0.smithi076.stdout:189: finishing write tid 2 to smithi07694878-49 2021-03-15T22:20:43.236 INFO:tasks.rados.rados.0.smithi076.stdout:187: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:20:43.237 INFO:tasks.rados.rados.0.smithi076.stdout:187: done (4 left) 2021-03-15T22:20:43.237 INFO:tasks.rados.rados.0.smithi076.stdout:188: done (3 left) 2021-03-15T22:20:43.238 INFO:tasks.rados.rados.0.smithi076.stdout:191: done (2 left) 2021-03-15T22:20:43.238 INFO:tasks.rados.rados.0.smithi076.stdout:192: copy_from oid 9 from oid 5 current snap is 19 2021-03-15T22:20:43.238 INFO:tasks.rados.rados.0.smithi076.stdout:189: finishing write tid 3 to smithi07694878-49 2021-03-15T22:20:43.239 INFO:tasks.rados.rados.0.smithi076.stdout:189: finishing write tid 4 to smithi07694878-49 2021-03-15T22:20:43.239 INFO:tasks.rados.rados.0.smithi076.stdout:189: finishing write tid 5 to smithi07694878-49 2021-03-15T22:20:43.239 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 95 (ObjNum 96 snap 19 seq_num 96) dirty exists 2021-03-15T22:20:43.239 INFO:tasks.rados.rados.0.smithi076.stdout:189: left oid 49 (ObjNum 96 snap 19 seq_num 96) 2021-03-15T22:20:43.240 INFO:tasks.rados.rados.0.smithi076.stdout:190: expect (ObjNum 89 snap 12 seq_num 89) 2021-03-15T22:20:43.251 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:43.276 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:43.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:43.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:43.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:43 smithi125 conmon[56739]: cluster 2021-03-15T22:20:42.008423+0000 mon.a (mon.0 2021-03-15T22:20:43.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:43 smithi125 conmon[56739]: ) 663 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2021-03-15T22:20:43.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:43 smithi125 conmon[56739]: audit 2021-03-15T22:20:42.852238+0000 mon.a (mon.0) 664 : 2021-03-15T22:20:43.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:43 smithi125 conmon[56739]: audit [INF] from='client.? 172.21.15.76:0/3603971056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:43.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:43 smithi125 conmon[56739]: cluster 2021-03-15T22:20 2021-03-15T22:20:43.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:43 smithi125 conmon[56739]: :42.984788+0000 mgr.y (mgr.14144) 387 : cluster 2021-03-15T22:20:43.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:43 smithi125 conmon[56739]: [DBG] pgmap v318: 148 pgs: 148 active+clean; 136 MiB data, 363 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2021-03-15T22:20:43.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:43 smithi076 conmon[62156]: cluster 2021-03-15T22:20:42.008423+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2021-03-15T22:20:43.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:43 smithi076 conmon[62156]: audit 2021-03-15T22:20:42.852238+0000 mon.a (mon.0) 664 : audit [INF] from='client.? 172.21.15.76:0/3603971056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:43.297 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:43 smithi076 conmon[62156]: cluster 2021-03-15T22:20:42.984788+0000 mgr.y (mgr.14144) 387 : cluster [DBG] pgmap v318: 148 pgs: 148 active+clean; 136 MiB data, 363 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2021-03-15T22:20:43.298 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:43 smithi076 conmon[56514]: cluster 2021-03-15T22:20:42.008423+0000 mon.a (mon.0) 663 : cluster [DBG] osdmap e127: 8 total, 8 up, 8 in 2021-03-15T22:20:43.299 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:43 smithi076 conmon[56514]: audit 2021-03-15T22:20:42.852238+0000 mon.a (mon.0) 664 : audit [INF] from='client.? 172.21.15.76:0/3603971056' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:43.299 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:43 smithi076 conmon[56514]: cluster 2021-03-15T22:20:42.984788+0000 mgr.y (mgr.14144) 387 : cluster [DBG] pgmap v318: 148 pgs: 148 active+clean; 136 MiB data, 363 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 2.7 MiB/s wr, 9 op/s 2021-03-15T22:20:43.379 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:43.394 INFO:tasks.rados.rados.0.smithi076.stdout:189: done (2 left) 2021-03-15T22:20:43.394 INFO:tasks.rados.rados.0.smithi076.stdout:190: done (1 left) 2021-03-15T22:20:43.395 INFO:tasks.rados.rados.0.smithi076.stdout:193: snap_create 2021-03-15T22:20:43.395 INFO:tasks.rados.rados.0.smithi076.stdout:192: finishing copy_from to smithi07694878-9 2021-03-15T22:20:43.395 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 8 (ObjNum 90 snap 14 seq_num 90) dirty exists 2021-03-15T22:20:43.396 INFO:tasks.rados.rados.0.smithi076.stdout:192: finishing copy_from racing read to smithi07694878-9 2021-03-15T22:20:43.404 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:43.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:43.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:43.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:43.531 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:43.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:43.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:43.634 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:43.657 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:43.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:43.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:43.759 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:43.782 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:43.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:43.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:43.884 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:43.906 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:43.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:43.907 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:44.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:44.025 INFO:tasks.rados.rados.0.smithi076.stdout:192: done (1 left) 2021-03-15T22:20:44.025 INFO:tasks.rados.rados.0.smithi076.stdout:193: done (0 left) 2021-03-15T22:20:44.026 INFO:tasks.rados.rados.0.smithi076.stdout:194: copy_from oid 41 from oid 40 current snap is 20 2021-03-15T22:20:44.026 INFO:tasks.rados.rados.0.smithi076.stdout:195: copy_from oid 6 from oid 47 current snap is 20 2021-03-15T22:20:44.026 INFO:tasks.rados.rados.0.smithi076.stdout:196: read oid 3 snap -1 2021-03-15T22:20:44.026 INFO:tasks.rados.rados.0.smithi076.stdout:196: expect (ObjNum 30 snap 0 seq_num 30) 2021-03-15T22:20:44.027 INFO:tasks.rados.rados.0.smithi076.stdout:197: rollback oid 48 current snap is 20 2021-03-15T22:20:44.027 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 48 to 19 2021-03-15T22:20:44.027 INFO:tasks.rados.rados.0.smithi076.stdout:198: write (excl) oid 45 current snap is 20 2021-03-15T22:20:44.028 INFO:tasks.rados.rados.0.smithi076.stdout:198: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2021-03-15T22:20:44.033 INFO:tasks.rados.rados.0.smithi076.stdout:198: writing smithi07694878-45 from 447086 to 1217724 tid 1 2021-03-15T22:20:44.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:44.038 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:44.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:44.045 INFO:tasks.rados.rados.0.smithi076.stdout:198: writing smithi07694878-45 from 1935000 to 2733016 tid 2 2021-03-15T22:20:44.051 INFO:tasks.rados.rados.0.smithi076.stdout:198: writing smithi07694878-45 from 3458512 to 3647086 tid 3 2021-03-15T22:20:44.051 INFO:tasks.rados.rados.0.smithi076.stdout:199: read oid 50 snap -1 2021-03-15T22:20:44.051 INFO:tasks.rados.rados.0.smithi076.stdout:199: expect (ObjNum 49 snap 0 seq_num 49) 2021-03-15T22:20:44.052 INFO:tasks.rados.rados.0.smithi076.stdout:200: write (excl) oid 44 current snap is 20 2021-03-15T22:20:44.052 INFO:tasks.rados.rados.0.smithi076.stdout:200: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2021-03-15T22:20:44.056 INFO:tasks.rados.rados.0.smithi076.stdout:200: writing smithi07694878-44 from 463893 to 1026133 tid 1 2021-03-15T22:20:44.063 INFO:tasks.rados.rados.0.smithi076.stdout:200: writing smithi07694878-44 from 1593482 to 2251509 tid 2 2021-03-15T22:20:44.069 INFO:tasks.rados.rados.0.smithi076.stdout:200: writing smithi07694878-44 from 2685935 to 3245276 tid 3 2021-03-15T22:20:44.069 INFO:tasks.rados.rados.0.smithi076.stdout:201: copy_from oid 9 from oid 30 current snap is 20 2021-03-15T22:20:44.070 INFO:tasks.rados.rados.0.smithi076.stdout:196: expect (ObjNum 30 snap 0 seq_num 30) 2021-03-15T22:20:44.140 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:44.151 INFO:tasks.rados.rados.0.smithi076.stdout:197: finishing rollback tid 0 to smithi07694878-48 2021-03-15T22:20:44.151 INFO:tasks.rados.rados.0.smithi076.stdout:197: finishing rollback tid 1 to smithi07694878-48 2021-03-15T22:20:44.152 INFO:tasks.rados.rados.0.smithi076.stdout:197: finishing rollback tid 2 to smithi07694878-48 2021-03-15T22:20:44.152 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 20 (ObjNum 72 snap 8 seq_num 72) dirty exists 2021-03-15T22:20:44.152 INFO:tasks.rados.rados.0.smithi076.stdout:195: finishing copy_from to smithi07694878-6 2021-03-15T22:20:44.153 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 14 (ObjNum 63 snap 6 seq_num 63) dirty exists 2021-03-15T22:20:44.153 INFO:tasks.rados.rados.0.smithi076.stdout:194: finishing copy_from to smithi07694878-41 2021-03-15T22:20:44.154 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 51 (ObjNum 68 snap 6 seq_num 68) dirty exists 2021-03-15T22:20:44.154 INFO:tasks.rados.rados.0.smithi076.stdout:194: finishing copy_from racing read to smithi07694878-41 2021-03-15T22:20:44.155 INFO:tasks.rados.rados.0.smithi076.stdout:195: finishing copy_from racing read to smithi07694878-6 2021-03-15T22:20:44.155 INFO:tasks.rados.rados.0.smithi076.stdout:198: finishing write tid 1 to smithi07694878-45 2021-03-15T22:20:44.156 INFO:tasks.rados.rados.0.smithi076.stdout:198: finishing write tid 2 to smithi07694878-45 2021-03-15T22:20:44.156 INFO:tasks.rados.rados.0.smithi076.stdout:198: finishing write tid 3 to smithi07694878-45 2021-03-15T22:20:44.157 INFO:tasks.rados.rados.0.smithi076.stdout:198: finishing write tid 4 to smithi07694878-45 2021-03-15T22:20:44.157 INFO:tasks.rados.rados.0.smithi076.stdout:198: finishing write tid 5 to smithi07694878-45 2021-03-15T22:20:44.157 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 23 (ObjNum 98 snap 20 seq_num 98) dirty exists 2021-03-15T22:20:44.158 INFO:tasks.rados.rados.0.smithi076.stdout:198: left oid 45 (ObjNum 98 snap 20 seq_num 98) 2021-03-15T22:20:44.158 INFO:tasks.rados.rados.0.smithi076.stdout:194: done (7 left) 2021-03-15T22:20:44.159 INFO:tasks.rados.rados.0.smithi076.stdout:195: done (6 left) 2021-03-15T22:20:44.159 INFO:tasks.rados.rados.0.smithi076.stdout:196: done (5 left) 2021-03-15T22:20:44.159 INFO:tasks.rados.rados.0.smithi076.stdout:197: done (4 left) 2021-03-15T22:20:44.159 INFO:tasks.rados.rados.0.smithi076.stdout:198: done (3 left) 2021-03-15T22:20:44.160 INFO:tasks.rados.rados.0.smithi076.stdout:202: write (excl) oid 1 current snap is 20 2021-03-15T22:20:44.160 INFO:tasks.rados.rados.0.smithi076.stdout:199: expect (ObjNum 49 snap 0 seq_num 49) 2021-03-15T22:20:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:44.164 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:44.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:44.238 INFO:tasks.rados.rados.0.smithi076.stdout:200: finishing write tid 1 to smithi07694878-44 2021-03-15T22:20:44.238 INFO:tasks.rados.rados.0.smithi076.stdout:200: finishing write tid 2 to smithi07694878-44 2021-03-15T22:20:44.239 INFO:tasks.rados.rados.0.smithi076.stdout:200: finishing write tid 3 to smithi07694878-44 2021-03-15T22:20:44.239 INFO:tasks.rados.rados.0.smithi076.stdout:200: finishing write tid 4 to smithi07694878-44 2021-03-15T22:20:44.239 INFO:tasks.rados.rados.0.smithi076.stdout:200: finishing write tid 5 to smithi07694878-44 2021-03-15T22:20:44.239 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 49 (ObjNum 99 snap 20 seq_num 99) dirty exists 2021-03-15T22:20:44.240 INFO:tasks.rados.rados.0.smithi076.stdout:200: left oid 44 (ObjNum 99 snap 20 seq_num 99) 2021-03-15T22:20:44.240 INFO:tasks.rados.rados.0.smithi076.stdout:201: finishing copy_from to smithi07694878-9 2021-03-15T22:20:44.240 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 24 (ObjNum 60 snap 4 seq_num 60) dirty exists 2021-03-15T22:20:44.240 INFO:tasks.rados.rados.0.smithi076.stdout:201: finishing copy_from racing read to smithi07694878-9 2021-03-15T22:20:44.241 INFO:tasks.rados.rados.0.smithi076.stdout:202: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2021-03-15T22:20:44.244 INFO:tasks.rados.rados.0.smithi076.stdout:202: writing smithi07694878-1 from 480700 to 1118189 tid 1 2021-03-15T22:20:44.249 INFO:tasks.rados.rados.0.smithi076.stdout:202: writing smithi07694878-1 from 1651964 to 2053649 tid 2 2021-03-15T22:20:44.257 INFO:tasks.rados.rados.0.smithi076.stdout:202: writing smithi07694878-1 from 2713358 to 3483971 tid 3 2021-03-15T22:20:44.258 INFO:tasks.rados.rados.0.smithi076.stdout:199: done (3 left) 2021-03-15T22:20:44.258 INFO:tasks.rados.rados.0.smithi076.stdout:200: done (2 left) 2021-03-15T22:20:44.259 INFO:tasks.rados.rados.0.smithi076.stdout:201: done (1 left) 2021-03-15T22:20:44.259 INFO:tasks.rados.rados.0.smithi076.stdout:203: snap_create 2021-03-15T22:20:44.259 INFO:tasks.rados.rados.0.smithi076.stdout:202: finishing write tid 1 to smithi07694878-1 2021-03-15T22:20:44.259 INFO:tasks.rados.rados.0.smithi076.stdout:202: finishing write tid 2 to smithi07694878-1 2021-03-15T22:20:44.261 INFO:tasks.rados.rados.0.smithi076.stdout:202: finishing write tid 3 to smithi07694878-1 2021-03-15T22:20:44.262 INFO:tasks.rados.rados.0.smithi076.stdout:202: finishing write tid 4 to smithi07694878-1 2021-03-15T22:20:44.262 INFO:tasks.rados.rados.0.smithi076.stdout:202: finishing write tid 5 to smithi07694878-1 2021-03-15T22:20:44.262 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 35 (ObjNum 100 snap 20 seq_num 100) dirty exists 2021-03-15T22:20:44.262 INFO:tasks.rados.rados.0.smithi076.stdout:202: left oid 1 (ObjNum 100 snap 20 seq_num 100) 2021-03-15T22:20:44.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:44.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:44 smithi125 conmon[56739]: cluster 2021-03-15T22:20:43.005715+0000 mon.a (mon.0) 665 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:44.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:44 smithi125 conmon[56739]: audit 2021-03-15T22:20:43.009523+0000 mon.a (mon.0) 666 : audit [INF] from='client.? 172.21.15.76:0/3603971056' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:20:44.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:44 smithi125 conmon[56739]: cluster 2021-03-15T22:20:43.009586+0000 mon.a (mon.0) 667 : cluster [DBG] 2021-03-15T22:20:44.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:44 smithi125 conmon[56739]: osdmap e128: 8 total, 8 up, 8 in 2021-03-15T22:20:44.291 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:44.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:44.292 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:44.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[56514]: cluster 2021-03-15T22:20:43. 2021-03-15T22:20:44.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[56514]: 005715+0000 mon.a (mon.0) 665 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:44.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[56514]: audit 2021-03-15T22:20:43.009523+0000 mon.a (mon.0) 666 : audit [INF] from='client.? 172.21.15.76:0/3603971056' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:20:44.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[56514]: cluster 2021-03-15T22:20:43.009586+0000 mon.a (mon.0) 667 : cluster [DBG] osdmap e128: 8 total, 8 up, 8 in 2021-03-15T22:20:44.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[62156]: cluster 2021-03-15T22:20:43.005715 2021-03-15T22:20:44.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[62156]: +0000 mon.a (mon.0) 665 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:44.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:20:44.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[62156]: 03-15T22:20:43.009523+0000 mon.a (mon.0) 666 : audit [INF] from='client.? 172.21.15.76:0/3603971056' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:20:44.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[62156]: 2021-03-15T22:20:44.329 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[62156]: cluster 2021-03-15T22:20:43. 2021-03-15T22:20:44.329 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[62156]: 009586+0000 mon.a (mon.0) 667 : cluster 2021-03-15T22:20:44.329 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:44 smithi076 conmon[62156]: [DBG] osdmap e128: 8 total, 8 up, 8 in 2021-03-15T22:20:44.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:44.418 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:44.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:44.420 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:44.472 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:44.521 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:44.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:44.547 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:44.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:44.648 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:44.672 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:44.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:44.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:44.775 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:44.800 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:44.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:44.802 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:44.896 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:20:44.897 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:20:44.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:44.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:44.928 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:44.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:45.002 INFO:tasks.rados.rados.0.smithi076.stdout:202: done (1 left) 2021-03-15T22:20:45.002 INFO:tasks.rados.rados.0.smithi076.stdout:203: done (0 left) 2021-03-15T22:20:45.002 INFO:tasks.rados.rados.0.smithi076.stdout:204: write oid 27 current snap is 21 2021-03-15T22:20:45.003 INFO:tasks.rados.rados.0.smithi076.stdout:204: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2021-03-15T22:20:45.008 INFO:tasks.rados.rados.0.smithi076.stdout:204: writing smithi07694878-27 from 497507 to 1210245 tid 1 2021-03-15T22:20:45.014 INFO:tasks.rados.rados.0.smithi076.stdout:204: writing smithi07694878-27 from 1710446 to 2372142 tid 2 2021-03-15T22:20:45.022 INFO:tasks.rados.rados.0.smithi076.stdout:204: writing smithi07694878-27 from 3140781 to 3697507 tid 3 2021-03-15T22:20:45.023 INFO:tasks.rados.rados.0.smithi076.stdout:205: write (excl) oid 20 current snap is 21 2021-03-15T22:20:45.023 INFO:tasks.rados.rados.0.smithi076.stdout:204: finishing write tid 1 to smithi07694878-27 2021-03-15T22:20:45.023 INFO:tasks.rados.rados.0.smithi076.stdout:205: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2021-03-15T22:20:45.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:45.029 INFO:tasks.rados.rados.0.smithi076.stdout:205: writing smithi07694878-20 from 514314 to 1302301 tid 1 2021-03-15T22:20:45.034 INFO:tasks.rados.rados.0.smithi076.stdout:205: writing smithi07694878-20 from 1768928 to 2174282 tid 2 2021-03-15T22:20:45.041 INFO:tasks.rados.rados.0.smithi076.stdout:205: writing smithi07694878-20 from 2768204 to 3277714 tid 3 2021-03-15T22:20:45.041 INFO:tasks.rados.rados.0.smithi076.stdout:206: write (excl) oid 5 current snap is 21 2021-03-15T22:20:45.041 INFO:tasks.rados.rados.0.smithi076.stdout:204: finishing write tid 2 to smithi07694878-27 2021-03-15T22:20:45.041 INFO:tasks.rados.rados.0.smithi076.stdout:206: seq_num 103 ranges {531121=463236,1711057=665365,3079274=651847} 2021-03-15T22:20:45.046 INFO:tasks.rados.rados.0.smithi076.stdout:206: writing smithi07694878-5 from 531121 to 994357 tid 1 2021-03-15T22:20:45.053 INFO:tasks.rados.rados.0.smithi076.stdout:206: writing smithi07694878-5 from 1711057 to 2376422 tid 2 2021-03-15T22:20:45.055 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:45.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:45.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:45.061 INFO:tasks.rados.rados.0.smithi076.stdout:206: writing smithi07694878-5 from 3079274 to 3731121 tid 3 2021-03-15T22:20:45.062 INFO:tasks.rados.rados.0.smithi076.stdout:207: delete oid 49 current snap is 21 2021-03-15T22:20:45.062 INFO:tasks.rados.rados.0.smithi076.stdout:204: finishing write tid 3 to smithi07694878-27 2021-03-15T22:20:45.062 INFO:tasks.rados.rados.0.smithi076.stdout:204: finishing write tid 4 to smithi07694878-27 2021-03-15T22:20:45.062 INFO:tasks.rados.rados.0.smithi076.stdout:204: finishing write tid 5 to smithi07694878-27 2021-03-15T22:20:45.063 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 54 (ObjNum 101 snap 21 seq_num 101) dirty exists 2021-03-15T22:20:45.063 INFO:tasks.rados.rados.0.smithi076.stdout:204: left oid 27 (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:20:45.063 INFO:tasks.rados.rados.0.smithi076.stdout:205: finishing write tid 1 to smithi07694878-20 2021-03-15T22:20:45.063 INFO:tasks.rados.rados.0.smithi076.stdout:205: finishing write tid 2 to smithi07694878-20 2021-03-15T22:20:45.063 INFO:tasks.rados.rados.0.smithi076.stdout:205: finishing write tid 3 to smithi07694878-20 2021-03-15T22:20:45.064 INFO:tasks.rados.rados.0.smithi076.stdout:205: finishing write tid 4 to smithi07694878-20 2021-03-15T22:20:45.064 INFO:tasks.rados.rados.0.smithi076.stdout:205: finishing write tid 5 to smithi07694878-20 2021-03-15T22:20:45.064 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 41 (ObjNum 102 snap 21 seq_num 102) dirty exists 2021-03-15T22:20:45.064 INFO:tasks.rados.rados.0.smithi076.stdout:205: left oid 20 (ObjNum 102 snap 21 seq_num 102) 2021-03-15T22:20:45.065 INFO:tasks.rados.rados.0.smithi076.stdout:206: finishing write tid 1 to smithi07694878-5 2021-03-15T22:20:45.065 INFO:tasks.rados.rados.0.smithi076.stdout:206: finishing write tid 2 to smithi07694878-5 2021-03-15T22:20:45.065 INFO:tasks.rados.rados.0.smithi076.stdout:204: done (3 left) 2021-03-15T22:20:45.066 INFO:tasks.rados.rados.0.smithi076.stdout:205: done (2 left) 2021-03-15T22:20:45.066 INFO:tasks.rados.rados.0.smithi076.stdout:207: done (1 left) 2021-03-15T22:20:45.066 INFO:tasks.rados.rados.0.smithi076.stdout:208: snap_remove snap 15 2021-03-15T22:20:45.157 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:45.182 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:45.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:45.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:45.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56514]: cluster 2021-03-15T22:20:44.023602+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2021-03-15T22:20:45.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56514]: cluster 2021-03-15T22:20:44.985176+0000 mgr.y (mgr.14144) 388 : cluster [DBG] pgmap v321: 148 pgs: 148 active+clean; 138 MiB data, 374 MiB used, 715 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-03-15T22:20:45.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56514]: cluster 2021-03-15T22:20:44.997284+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2021-03-15T22:20:45.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[62156]: cluster 2021-03-15T22:20:44.023602+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2021-03-15T22:20:45.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[62156]: cluster 2021-03-15T22:20:44.985176+0000 mgr.y (mgr.14144) 388 : cluster [DBG] pgmap v321: 148 pgs: 148 active+clean; 138 MiB data, 374 MiB used, 715 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-03-15T22:20:45.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[62156]: cluster 2021-03-15T22:20:44.997284+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2021-03-15T22:20:45.284 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:45.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:45 smithi125 conmon[56739]: cluster 2021-03-15T22:20:44.023602+0000 mon.a (mon.0) 668 : cluster [DBG] osdmap e129: 8 total, 8 up, 8 in 2021-03-15T22:20:45.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:45 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:20:45.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:45 smithi125 conmon[56739]: 20:44.985176+0000 mgr.y (mgr.14144) 388 : cluster [DBG] pgmap v321: 148 pgs: 148 active+clean; 138 MiB data, 374 MiB used, 715 GiB / 715 GiB avail; 6.8 MiB/s rd, 2.4 MiB/s wr, 13 op/s 2021-03-15T22:20:45.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:45 smithi125 conmon[56739]: cluster 2021-03-15T22:20:44.997284+0000 mon.a (mon.0) 669 : cluster [DBG] osdmap e130: 8 total, 8 up, 8 in 2021-03-15T22:20:45.307 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:45.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:45.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:45.409 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:45.432 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:45.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:45.433 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:45.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:45.556 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:45.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:45.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:45.659 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:45.682 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:45.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:45.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:45.690 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: debug 2021-03-15T22:20:45.683+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:20:45.691 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:20:45.691 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:45.691 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:20:45.784 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:45.807 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:45.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:45.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:45.910 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:45.932 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:45.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:45.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:46.017 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:20:46.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:20:46.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: debug 2021-03-15T22:20:45.683+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:20:46.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:20:46.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:20:46.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:20:46.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:20:46.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: debug 2021-03-15T22:20:45.684+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:20:46.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:20:46.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:20:46.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:20:46.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:20:46.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: debug 2021-03-15T22:20:45.684+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:20:46.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:20:46.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:20:46.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:20:46.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.028 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.028 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:20:46.028 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: debug 2021-03-15T22:20:45.684+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:20:46.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:20:46.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.030 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:20:46.030 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:20:46.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.032 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:20:46.032 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: debug 2021-03-15T22:20:45.685+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:20:46.033 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:20:46.033 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.034 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:20:46.034 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.035 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:20:46.036 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.036 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.036 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:20:46.037 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: debug 2021-03-15T22:20:45.685+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:20:46.037 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:20:46.038 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.038 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:20:46.039 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.039 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:20:46.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:20:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:20:46.041 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:46.043 INFO:tasks.rados.rados.0.smithi076.stdout:208: done (1 left) 2021-03-15T22:20:46.044 INFO:tasks.rados.rados.0.smithi076.stdout:209: snap_remove snap 19 2021-03-15T22:20:46.063 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:46.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:46.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:46.099 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-15T22:20:46.166 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:46.188 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:46.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:46.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:46.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:46.317 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:46.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:46.319 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:46.364 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: audit 2021-03-15T22:20:45.402744+0000 2021-03-15T22:20:46.364 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: mon.a (mon.0) 670 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:20:46.365 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:20:45.680873+0000 mgr.y (mgr.14144) 389 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:20:46.365 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: audit 2021-03-15T22:20:45.684091+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:20:46.365 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:20:45.685018+0000 mgr.y (mgr.14144) 390 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:20:46.365 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:20:46.366 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.366 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:20:46.366 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.366 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:20:46.367 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.367 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.367 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:20:46.368 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:20:45.685457+0000 mgr.y (mgr.14144) 391 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:20:46.368 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:20:46.368 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.368 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:20:46.369 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.369 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:20:46.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:20:46.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:20:45.685927+0000 mgr.y (mgr.14144) 392 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:20:46.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:20:46.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.371 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: audit 2021-03-15T22:20:45.402744+0000 2021-03-15T22:20:46.372 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: mon.a (mon.0) 670 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:20:46.372 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: cephadm 2021-03-15T 2021-03-15T22:20:46.372 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: 22:20:45.680873+0000 mgr.y (mgr.14144) 389 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:20:46.372 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:20:46.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: 15T22:20:45.684091+0000 mon.a (mon.0) 671 : 2021-03-15T22:20:46.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:20:46.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:20:46.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: 2021-03-15T22:20:45.685018+0000 mgr.y (mgr.14144) 2021-03-15T22:20:46.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: 390 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:20:46.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:20:46.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:20:46.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:20:46.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:20:46.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:20:45.685457+0000 mgr.y (mgr.14144) 391 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:20:46.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:20:46.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:20:46.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.377 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:20:46.377 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.377 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.377 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:20:46.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:20:45.685927+0000 mgr.y (mgr.14144) 392 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:20:46.378 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:20:46.378 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:20:46.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:20:46.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: cephadm 2021-03- 2021-03-15T22:20:46.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: 15T22:20:45.686308+0000 mgr.y (mgr.14144) 393 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:20:46.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:20:46.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:20:46.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:20:46.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:20:46.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:20:45.686679+0000 mgr.y (mgr.14144) 394 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:20:46.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:20:46.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:20:46.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:20:46.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:20:46.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:20:45.687070+0000 mgr.y (mgr.14144) 395 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:20:46.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:20:46.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:20:46.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:20:46.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:20:46.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:20:45.687507+0000 mgr.y (mgr.14144) 396 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:20:46.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:20:46.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:20:46.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:20:46.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:20:46.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:20:46.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:20:46.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:20:46.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:20:46.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:20:45.686308+0000 mgr.y (mgr.14144) 393 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:20:46.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:20:46.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:20:46.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:20:46.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:20:46.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:20:45.686679+0000 mgr.y (mgr.14144) 394 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:20:46.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:20:46.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:20:46.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:20:46.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:20:46.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:20:45.687070+0000 mgr.y (mgr.14144) 395 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:20:46.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:20:46.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:20:46.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:20:46.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:20:46.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:20:45.687507+0000 mgr.y (mgr.14144) 396 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:20:46.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:20:46.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:20:46.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:20:46.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:20:46.420 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:46.436 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:20:46.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:46.442 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:46.443 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: audit 2021-03-15T22:20:45.402744+0000 mon.a 2021-03-15T22:20:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: (mon.0) 670 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:20:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:20:45.680873+0000 mgr.y (mgr.14144) 389 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:20:46.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: audit 2021-03-15T22:20:45.684091+0000 mon.a (mon.0) 671 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:20:46.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:20:45.685018+0000 mgr.y (mgr.14144) 390 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:20:46.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:20:46.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:20:46.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:20:46.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:20:46.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:20:45.685457+0000 mgr.y (mgr.14144) 391 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:20:46.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:20:46.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:20:46.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:20:46.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:20:46.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:20:45.685927+0000 mgr.y (mgr.14144) 392 : cephadm 2021-03-15T22:20:46.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:20:46.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:20:46.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:20:46.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:20:46.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:20:46.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:20:45.686308+0000 mgr.y (mgr.14144) 393 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:20:46.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:20:46.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:20:46.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:20:46.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:20:46.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:20:45.686679+0000 mgr.y (mgr.14144) 394 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:20:46.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:20:46.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:20:46.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:20:46.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:20:46.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:20:45.687070+0000 mgr.y (mgr.14144) 395 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:20:46.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:20:46.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:20:46.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:20:46.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:20:46.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: cephadm 2021-03 2021-03-15T22:20:46.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: -15T22:20:45.687507+0000 mgr.y (mgr.14144) 396 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:20:46.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:20:46.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:20:46.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:20:46.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:20:46.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:20:46.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:20:46.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:20:46.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:20:46.553 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:46.568 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:46.577 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:46.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:46.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:46.670 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:20:46.674 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:20:46.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:46.702 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:46.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:46.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:46.804 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:46.827 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:46.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:46.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:46.930 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:46.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:46.958 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:46.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:47.047 INFO:tasks.rados.rados.0.smithi076.stdout:209: done (1 left) 2021-03-15T22:20:47.048 INFO:tasks.rados.rados.0.smithi076.stdout:210: rollback oid 19 current snap is 21 2021-03-15T22:20:47.048 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 19 to 16 2021-03-15T22:20:47.048 INFO:tasks.rados.rados.0.smithi076.stdout:206: finishing write tid 3 to smithi07694878-5 2021-03-15T22:20:47.048 INFO:tasks.rados.rados.0.smithi076.stdout:206: finishing write tid 4 to smithi07694878-5 2021-03-15T22:20:47.049 INFO:tasks.rados.rados.0.smithi076.stdout:206: finishing write tid 5 to smithi07694878-5 2021-03-15T22:20:47.049 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 17 (ObjNum 103 snap 21 seq_num 103) dirty exists 2021-03-15T22:20:47.049 INFO:tasks.rados.rados.0.smithi076.stdout:206: left oid 5 (ObjNum 103 snap 21 seq_num 103) 2021-03-15T22:20:47.049 INFO:tasks.rados.rados.0.smithi076.stdout:206: done (1 left) 2021-03-15T22:20:47.050 INFO:tasks.rados.rados.0.smithi076.stdout:211: read oid 40 snap -1 2021-03-15T22:20:47.050 INFO:tasks.rados.rados.0.smithi076.stdout:211: expect (ObjNum 68 snap 6 seq_num 68) 2021-03-15T22:20:47.050 INFO:tasks.rados.rados.0.smithi076.stdout:212: write (excl) oid 4 current snap is 21 2021-03-15T22:20:47.050 INFO:tasks.rados.rados.0.smithi076.stdout:212: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2021-03-15T22:20:47.054 INFO:tasks.rados.rados.0.smithi076.stdout:212: writing smithi07694878-4 from 564735 to 1178469 tid 1 2021-03-15T22:20:47.062 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:47.063 INFO:tasks.rados.rados.0.smithi076.stdout:212: writing smithi07694878-4 from 1828021 to 2497055 tid 2 2021-03-15T22:20:47.069 INFO:tasks.rados.rados.0.smithi076.stdout:212: writing smithi07694878-4 from 3134120 to 3764735 tid 3 2021-03-15T22:20:47.070 INFO:tasks.rados.rados.0.smithi076.stdout:213: copy_from oid 36 from oid 23 current snap is 21 2021-03-15T22:20:47.070 INFO:tasks.rados.rados.0.smithi076.stdout:210: finishing rollback tid 0 to smithi07694878-19 2021-03-15T22:20:47.070 INFO:tasks.rados.rados.0.smithi076.stdout:210: finishing rollback tid 1 to smithi07694878-19 2021-03-15T22:20:47.071 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 59 (ObjNum 1 snap 32564 seq_num 261944624) dirty dne 2021-03-15T22:20:47.071 INFO:tasks.rados.rados.0.smithi076.stdout:212: finishing write tid 1 to smithi07694878-4 2021-03-15T22:20:47.071 INFO:tasks.rados.rados.0.smithi076.stdout:211: expect (ObjNum 68 snap 6 seq_num 68) 2021-03-15T22:20:47.088 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:47.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:47.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:47.191 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:47.211 INFO:tasks.rados.rados.0.smithi076.stdout:212: finishing write tid 2 to smithi07694878-4 2021-03-15T22:20:47.212 INFO:tasks.rados.rados.0.smithi076.stdout:213: finishing copy_from to smithi07694878-36 2021-03-15T22:20:47.212 INFO:tasks.rados.rados.0.smithi076.stdout:213: got expected ENOENT (src dne) 2021-03-15T22:20:47.212 INFO:tasks.rados.rados.0.smithi076.stdout:213: finishing copy_from racing read to smithi07694878-36 2021-03-15T22:20:47.212 INFO:tasks.rados.rados.0.smithi076.stdout:212: finishing write tid 3 to smithi07694878-4 2021-03-15T22:20:47.213 INFO:tasks.rados.rados.0.smithi076.stdout:210: done (3 left) 2021-03-15T22:20:47.213 INFO:tasks.rados.rados.0.smithi076.stdout:211: done (2 left) 2021-03-15T22:20:47.213 INFO:tasks.rados.rados.0.smithi076.stdout:213: done (1 left) 2021-03-15T22:20:47.213 INFO:tasks.rados.rados.0.smithi076.stdout:214: copy_from oid 14 from oid 49 current snap is 21 2021-03-15T22:20:47.213 INFO:tasks.rados.rados.0.smithi076.stdout:212: finishing write tid 4 to smithi07694878-4 2021-03-15T22:20:47.214 INFO:tasks.rados.rados.0.smithi076.stdout:212: finishing write tid 5 to smithi07694878-4 2021-03-15T22:20:47.214 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 41 (ObjNum 105 snap 21 seq_num 105) dirty exists 2021-03-15T22:20:47.214 INFO:tasks.rados.rados.0.smithi076.stdout:212: left oid 4 (ObjNum 105 snap 21 seq_num 105) 2021-03-15T22:20:47.214 INFO:tasks.rados.rados.0.smithi076.stdout:212: done (1 left) 2021-03-15T22:20:47.215 INFO:tasks.rados.rados.0.smithi076.stdout:215: snap_create 2021-03-15T22:20:47.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:47.216 INFO:tasks.rados.rados.0.smithi076.stdout:214: finishing copy_from to smithi07694878-14 2021-03-15T22:20:47.216 INFO:tasks.rados.rados.0.smithi076.stdout:214: got expected ENOENT (src dne) 2021-03-15T22:20:47.217 INFO:tasks.rados.rados.0.smithi076.stdout:214: finishing copy_from racing read to smithi07694878-14 2021-03-15T22:20:47.217 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:47.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:47.319 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:47.342 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:47.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:47.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:47.402 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:20:47.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:20:47.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:20:47.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:20:47.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:20:47.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:20:47.404 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:20:47.404 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:20:47.404 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:20:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:20:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:20:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:20:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:20:47.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:20:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:20:47.406 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:20:47.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[56514]: cluster 2021-03-15T22:20:46.033269+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2021-03-15T22:20:47.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[56514]: audit 2021-03-15T22:20:46.651639+0000 mgr.y (mgr.14144) 397 : audit [DBG] from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:20:47.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[56514]: cluster 2021-03-15T22:20:46.985613+0000 mgr.y (mgr.14144) 398 : cluster [DBG] pgmap v324: 148 pgs: 148 active+clean; 162 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 9.8 MiB/s rd, 5.8 MiB/s wr, 19 op/s 2021-03-15T22:20:47.407 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[62156]: cluster 2021-03-15T22:20:47.408 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:20:47.408 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[62156]: 15T22:20:46.033269+0000 mon.a (mon.0) 672 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2021-03-15T22:20:47.408 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[62156]: audit 2021-03-15T22:20:47.408 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[62156]: 2021-03 2021-03-15T22:20:47.408 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[62156]: -15T22:20:46.651639+0000 mgr.y (mgr.14144) 397 2021-03-15T22:20:47.409 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[62156]: : audit [DBG] from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:20:47.409 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[62156]: cluster 2021-03-15T22:20:47.409 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[62156]: 2021-03-15T22:20:46.985613+0000 2021-03-15T22:20:47.409 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[62156]: mgr.y (mgr.14144) 398 : cluster [DBG] 2021-03-15T22:20:47.410 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:47 smithi076 conmon[62156]: pgmap v324: 148 pgs: 148 active+clean; 162 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 9.8 MiB/s rd, 5.8 MiB/s wr, 19 op/s 2021-03-15T22:20:47.410 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:47.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:47.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:47.471 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:47.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:47.478 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":323,"stamp":"2021-03-15T22:20:46.030802+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":161999671,"num_objects":383,"num_object_clones":22,"num_object_copies":1149,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":383,"num_whiteouts":4,"num_read":1661,"num_read_kb":262585,"num_write":984,"num_write_kb":161398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1185,"ondisk_log_size":1185,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":399920,"kb_used_data":324272,"kb_used_omap":0,"kb_used_meta":75584,"kb_avail":749561296,"statfs":{"total":767960285184,"available":767550767104,"internally_reserved":0,"allocated":332054528,"data_stored":347415862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":77398016},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":21243096,"num_objects":9,"num_object_clones":9,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":36,"num_read_kb":27908,"num_write":31,"num_write_kb":17572,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.024868"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.997675+0000","last_change":"2021-03-15T22:19:28.122158+0000","last_active":"2021-03-15T22:20:39.997675+0000","last_peered":"2021-03-15T22:20:39.997675+0000","last_clean":"2021-03-15T22:20:39.997675+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:20:39.997675+0000","last_undegraded":"2021-03-15T22:20:39.997675+0000","last_fullsized":"2021-03-15T22:20:39.997675+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.122079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.122079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":"79","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.009366+0000","last_change":"2021-03-15T22:19:37.782674+0000","last_active":"2021-03-15T22:20:41.009366+0000","last_peered":"2021-03-15T22:20:41.009366+0000","last_clean":"2021-03-15T22:20:41.009366+0000","last_became_active":"2021-03-15T22:17:58.014719+0000","last_became_peered":"2021-03-15T22:17:58.014719+0000","last_unstale":"2021-03-15T22:20:41.009366+0000","last_undegraded":"2021-03-15T22:20:41.009366+0000","last_fullsized":"2021-03-15T22:20:41.009366+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.782584+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.782584+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"89","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027662+0000","last_change":"2021-03-15T22:19:08.789028+0000","last_active":"2021-03-15T22:20:44.027662+0000","last_peered":"2021-03-15T22:20:44.027662+0000","last_clean":"2021-03-15T22:20:44.027662+0000","last_became_active":"2021-03-15T22:17:54.284855+0000","last_became_peered":"2021-03-15T22:17:54.284855+0000","last_unstale":"2021-03-15T22:20:44.027662+0000","last_undegraded":"2021-03-15T22:20:44.027662+0000","last_fullsized":"2021-03-15T22:20:44.027662+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:19:08.788964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:08.788964+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"109","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.997465+0000","last_change":"2021-03-15T22:19:11.891528+0000","last_active":"2021-03-15T22:20:39.997465+0000","last_peered":"2021-03-15T22:20:39.997465+0000","last_clean":"2021-03-15T22:20:39.997465+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:20:39.997465+0000","last_undegraded":"2021-03-15T22:20:39.997465+0000","last_fullsized":"2021-03-15T22:20:39.997465+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:11.891465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:11.891465+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.016151+0000","last_change":"2021-03-15T22:19:26.865961+0000","last_active":"2021-03-15T22:20:40.016151+0000","last_peered":"2021-03-15T22:20:40.016151+0000","last_clean":"2021-03-15T22:20:40.016151+0000","last_became_active":"2021-03-15T22:18:00.288598+0000","last_became_peered":"2021-03-15T22:18:00.288598+0000","last_unstale":"2021-03-15T22:20:40.016151+0000","last_undegraded":"2021-03-15T22:20:40.016151+0000","last_fullsized":"2021-03-15T22:20:40.016151+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.865895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.865895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"79","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.997546+0000","last_change":"2021-03-15T22:20:28.291372+0000","last_active":"2021-03-15T22:20:39.997546+0000","last_peered":"2021-03-15T22:20:39.997546+0000","last_clean":"2021-03-15T22:20:39.997546+0000","last_became_active":"2021-03-15T22:17:58.010916+0000","last_became_peered":"2021-03-15T22:17:58.010916+0000","last_unstale":"2021-03-15T22:20:39.997546+0000","last_undegraded":"2021-03-15T22:20:39.997546+0000","last_fullsized":"2021-03-15T22:20:39.997546+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:28.291294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:20:28.291294+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"83","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.017371+0000","last_change":"2021-03-15T22:19:10.851317+0000","last_active":"2021-03-15T22:20:40.017371+0000","last_peered":"2021-03-15T22:20:40.017371+0000","last_clean":"2021-03-15T22:20:40.017371+0000","last_became_active":"2021-03-15T22:17:54.283819+0000","last_became_peered":"2021-03-15T22:17:54.283819+0000","last_unstale":"2021-03-15T22:20:40.017371+0000","last_undegraded":"2021-03-15T22:20:40.017371+0000","last_fullsized":"2021-03-15T22:20:40.017371+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:10.851256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:10.851256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"116","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.015670+0000","last_change":"2021-03-15T22:19:44.915926+0000","last_active":"2021-03-15T22:20:40.015670+0000","last_peered":"2021-03-15T22:20:40.015670+0000","last_clean":"2021-03-15T22:20:40.015670+0000","last_became_active":"2021-03-15T22:17:55.991597+0000","last_became_peered":"2021-03-15T22:17:55.991597+0000","last_unstale":"2021-03-15T22:20:40.015670+0000","last_undegraded":"2021-03-15T22:20:40.015670+0000","last_fullsized":"2021-03-15T22:20:40.015670+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:19:44.915865+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:44.915865+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":22,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"85","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.022486+0000","last_change":"2021-03-15T22:20:33.569323+0000","last_active":"2021-03-15T22:20:44.022486+0000","last_peered":"2021-03-15T22:20:44.022486+0000","last_clean":"2021-03-15T22:20:44.022486+0000","last_became_active":"2021-03-15T22:18:00.015776+0000","last_became_peered":"2021-03-15T22:18:00.015776+0000","last_unstale":"2021-03-15T22:20:44.022486+0000","last_undegraded":"2021-03-15T22:20:44.022486+0000","last_fullsized":"2021-03-15T22:20:44.022486+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:33.569266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:33.569266+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"79","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.015456+0000","last_change":"2021-03-15T22:19:30.205991+0000","last_active":"2021-03-15T22:20:41.015456+0000","last_peered":"2021-03-15T22:20:41.015456+0000","last_clean":"2021-03-15T22:20:41.015456+0000","last_became_active":"2021-03-15T22:17:58.008828+0000","last_became_peered":"2021-03-15T22:17:58.008828+0000","last_unstale":"2021-03-15T22:20:41.015456+0000","last_undegraded":"2021-03-15T22:20:41.015456+0000","last_fullsized":"2021-03-15T22:20:41.015456+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.205915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.205915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"88","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.015716+0000","last_change":"2021-03-15T22:20:35.201051+0000","last_active":"2021-03-15T22:20:41.015716+0000","last_peered":"2021-03-15T22:20:41.015716+0000","last_clean":"2021-03-15T22:20:41.015716+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:20:41.015716+0000","last_undegraded":"2021-03-15T22:20:41.015716+0000","last_fullsized":"2021-03-15T22:20:41.015716+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:35.200940+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:35.200940+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"105","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.997789+0000","last_change":"2021-03-15T22:19:53.135106+0000","last_active":"2021-03-15T22:20:39.997789+0000","last_peered":"2021-03-15T22:20:39.997789+0000","last_clean":"2021-03-15T22:20:39.997789+0000","last_became_active":"2021-03-15T22:17:55.990528+0000","last_became_peered":"2021-03-15T22:17:55.990528+0000","last_unstale":"2021-03-15T22:20:39.997789+0000","last_undegraded":"2021-03-15T22:20:39.997789+0000","last_fullsized":"2021-03-15T22:20:39.997789+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:19:53.135007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:53.135007+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"77","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.014543+0000","last_change":"2021-03-15T22:19:37.268478+0000","last_active":"2021-03-15T22:20:41.014543+0000","last_peered":"2021-03-15T22:20:41.014543+0000","last_clean":"2021-03-15T22:20:41.014543+0000","last_became_active":"2021-03-15T22:18:00.019585+0000","last_became_peered":"2021-03-15T22:18:00.019585+0000","last_unstale":"2021-03-15T22:20:41.014543+0000","last_undegraded":"2021-03-15T22:20:41.014543+0000","last_fullsized":"2021-03-15T22:20:41.014543+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.268362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.268362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"81","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022035+0000","last_change":"2021-03-15T22:19:57.014430+0000","last_active":"2021-03-15T22:20:43.022035+0000","last_peered":"2021-03-15T22:20:43.022035+0000","last_clean":"2021-03-15T22:20:43.022035+0000","last_became_active":"2021-03-15T22:17:58.013236+0000","last_became_peered":"2021-03-15T22:17:58.013236+0000","last_unstale":"2021-03-15T22:20:43.022035+0000","last_undegraded":"2021-03-15T22:20:43.022035+0000","last_fullsized":"2021-03-15T22:20:43.022035+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:57.014336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:57.014336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"83","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.014815+0000","last_change":"2021-03-15T22:19:06.274483+0000","last_active":"2021-03-15T22:20:41.014815+0000","last_peered":"2021-03-15T22:20:41.014815+0000","last_clean":"2021-03-15T22:20:41.014815+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:20:41.014815+0000","last_undegraded":"2021-03-15T22:20:41.014815+0000","last_fullsized":"2021-03-15T22:20:41.014815+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:06.274416+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.274416+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"121'9","reported_seq":"118","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998235+0000","last_change":"2021-03-15T22:20:36.258879+0000","last_active":"2021-03-15T22:20:39.998235+0000","last_peered":"2021-03-15T22:20:39.998235+0000","last_clean":"2021-03-15T22:20:39.998235+0000","last_became_active":"2021-03-15T22:19:02.274105+0000","last_became_peered":"2021-03-15T22:19:02.274105+0000","last_unstale":"2021-03-15T22:20:39.998235+0000","last_undegraded":"2021-03-15T22:20:39.998235+0000","last_fullsized":"2021-03-15T22:20:39.998235+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'9","last_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:20:36.258799+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998738+0000","last_change":"2021-03-15T22:20:29.251994+0000","last_active":"2021-03-15T22:20:39.998738+0000","last_peered":"2021-03-15T22:20:39.998738+0000","last_clean":"2021-03-15T22:20:39.998738+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:20:39.998738+0000","last_undegraded":"2021-03-15T22:20:39.998738+0000","last_fullsized":"2021-03-15T22:20:39.998738+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:29.251915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:29.251915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":"79","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.009824+0000","last_change":"2021-03-15T22:19:35.779203+0000","last_active":"2021-03-15T22:20:41.009824+0000","last_peered":"2021-03-15T22:20:41.009824+0000","last_clean":"2021-03-15T22:20:41.009824+0000","last_became_active":"2021-03-15T22:17:58.016067+0000","last_became_peered":"2021-03-15T22:17:58.016067+0000","last_unstale":"2021-03-15T22:20:41.009824+0000","last_undegraded":"2021-03-15T22:20:41.009824+0000","last_fullsized":"2021-03-15T22:20:41.009824+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:35.779105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.779105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"89","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027668+0000","last_change":"2021-03-15T22:19:14.859504+0000","last_active":"2021-03-15T22:20:44.027668+0000","last_peered":"2021-03-15T22:20:44.027668+0000","last_clean":"2021-03-15T22:20:44.027668+0000","last_became_active":"2021-03-15T22:17:53.976875+0000","last_became_peered":"2021-03-15T22:17:53.976875+0000","last_unstale":"2021-03-15T22:20:44.027668+0000","last_undegraded":"2021-03-15T22:20:44.027668+0000","last_fullsized":"2021-03-15T22:20:44.027668+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.859443+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"109","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.009061+0000","last_change":"2021-03-15T22:19:27.165212+0000","last_active":"2021-03-15T22:20:43.009061+0000","last_peered":"2021-03-15T22:20:43.009061+0000","last_clean":"2021-03-15T22:20:43.009061+0000","last_became_active":"2021-03-15T22:17:55.992247+0000","last_became_peered":"2021-03-15T22:17:55.992247+0000","last_unstale":"2021-03-15T22:20:43.009061+0000","last_undegraded":"2021-03-15T22:20:43.009061+0000","last_fullsized":"2021-03-15T22:20:43.009061+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:27.165119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.165119+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.018366+0000","last_change":"2021-03-15T22:19:45.893973+0000","last_active":"2021-03-15T22:20:40.018366+0000","last_peered":"2021-03-15T22:20:40.018366+0000","last_clean":"2021-03-15T22:20:40.018366+0000","last_became_active":"2021-03-15T22:18:00.286974+0000","last_became_peered":"2021-03-15T22:18:00.286974+0000","last_unstale":"2021-03-15T22:20:40.018366+0000","last_undegraded":"2021-03-15T22:20:40.018366+0000","last_fullsized":"2021-03-15T22:20:40.018366+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:45.893910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:45.893910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"79","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998359+0000","last_change":"2021-03-15T22:19:21.043341+0000","last_active":"2021-03-15T22:20:39.998359+0000","last_peered":"2021-03-15T22:20:39.998359+0000","last_clean":"2021-03-15T22:20:39.998359+0000","last_became_active":"2021-03-15T22:17:58.005477+0000","last_became_peered":"2021-03-15T22:17:58.005477+0000","last_unstale":"2021-03-15T22:20:39.998359+0000","last_undegraded":"2021-03-15T22:20:39.998359+0000","last_fullsized":"2021-03-15T22:20:39.998359+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.043261+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.043261+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"83","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.997950+0000","last_change":"2021-03-15T22:19:35.186018+0000","last_active":"2021-03-15T22:20:39.997950+0000","last_peered":"2021-03-15T22:20:39.997950+0000","last_clean":"2021-03-15T22:20:39.997950+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:20:39.997950+0000","last_undegraded":"2021-03-15T22:20:39.997950+0000","last_fullsized":"2021-03-15T22:20:39.997950+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:35.185891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.185891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"112","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.016915+0000","last_change":"2021-03-15T22:19:35.934561+0000","last_active":"2021-03-15T22:20:40.016915+0000","last_peered":"2021-03-15T22:20:40.016915+0000","last_clean":"2021-03-15T22:20:40.016915+0000","last_became_active":"2021-03-15T22:17:55.995747+0000","last_became_peered":"2021-03-15T22:17:55.995747+0000","last_unstale":"2021-03-15T22:20:40.016915+0000","last_undegraded":"2021-03-15T22:20:40.016915+0000","last_fullsized":"2021-03-15T22:20:40.016915+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:35.934475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:35.934475+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":20,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"79","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022467+0000","last_change":"2021-03-15T22:19:30.854972+0000","last_active":"2021-03-15T22:20:43.022467+0000","last_peered":"2021-03-15T22:20:43.022467+0000","last_clean":"2021-03-15T22:20:43.022467+0000","last_became_active":"2021-03-15T22:18:00.015889+0000","last_became_peered":"2021-03-15T22:18:00.015889+0000","last_unstale":"2021-03-15T22:20:43.022467+0000","last_undegraded":"2021-03-15T22:20:43.022467+0000","last_fullsized":"2021-03-15T22:20:43.022467+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:30.854899+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.854899+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1f","version":"61'2","reported_seq":"117","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.026489+0000","last_change":"2021-03-15T22:19:21.656004+0000","last_active":"2021-03-15T22:20:44.026489+0000","last_peered":"2021-03-15T22:20:44.026489+0000","last_clean":"2021-03-15T22:20:44.026489+0000","last_became_active":"2021-03-15T22:17:58.013571+0000","last_became_peered":"2021-03-15T22:17:58.013571+0000","last_unstale":"2021-03-15T22:20:44.026489+0000","last_undegraded":"2021-03-15T22:20:44.026489+0000","last_fullsized":"2021-03-15T22:20:44.026489+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:21.655918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.655918+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"88","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.009440+0000","last_change":"2021-03-15T22:20:27.760868+0000","last_active":"2021-03-15T22:20:41.009440+0000","last_peered":"2021-03-15T22:20:41.009440+0000","last_clean":"2021-03-15T22:20:41.009440+0000","last_became_active":"2021-03-15T22:17:53.975549+0000","last_became_peered":"2021-03-15T22:17:53.975549+0000","last_unstale":"2021-03-15T22:20:41.009440+0000","last_undegraded":"2021-03-15T22:20:41.009440+0000","last_fullsized":"2021-03-15T22:20:41.009440+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:27.760756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:27.760756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"122","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027599+0000","last_change":"2021-03-15T22:19:40.666578+0000","last_active":"2021-03-15T22:20:44.027599+0000","last_peered":"2021-03-15T22:20:44.027599+0000","last_clean":"2021-03-15T22:20:44.027599+0000","last_became_active":"2021-03-15T22:17:56.288283+0000","last_became_peered":"2021-03-15T22:17:56.288283+0000","last_unstale":"2021-03-15T22:20:44.027599+0000","last_undegraded":"2021-03-15T22:20:44.027599+0000","last_fullsized":"2021-03-15T22:20:44.027599+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.666518+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"82","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.009969+0000","last_change":"2021-03-15T22:20:23.755318+0000","last_active":"2021-03-15T22:20:41.009969+0000","last_peered":"2021-03-15T22:20:41.009969+0000","last_clean":"2021-03-15T22:20:41.009969+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:20:41.009969+0000","last_undegraded":"2021-03-15T22:20:41.009969+0000","last_fullsized":"2021-03-15T22:20:41.009969+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:23.755259+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:23.755259+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":"82","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027777+0000","last_change":"2021-03-15T22:19:17.752562+0000","last_active":"2021-03-15T22:20:44.027777+0000","last_peered":"2021-03-15T22:20:44.027777+0000","last_clean":"2021-03-15T22:20:44.027777+0000","last_became_active":"2021-03-15T22:17:58.009184+0000","last_became_peered":"2021-03-15T22:17:58.009184+0000","last_unstale":"2021-03-15T22:20:44.027777+0000","last_undegraded":"2021-03-15T22:20:44.027777+0000","last_fullsized":"2021-03-15T22:20:44.027777+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.752495+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.752495+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"83","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.015693+0000","last_change":"2021-03-15T22:19:21.900042+0000","last_active":"2021-03-15T22:20:40.015693+0000","last_peered":"2021-03-15T22:20:40.015693+0000","last_clean":"2021-03-15T22:20:40.015693+0000","last_became_active":"2021-03-15T22:17:53.967947+0000","last_became_peered":"2021-03-15T22:17:53.967947+0000","last_unstale":"2021-03-15T22:20:40.015693+0000","last_undegraded":"2021-03-15T22:20:40.015693+0000","last_fullsized":"2021-03-15T22:20:40.015693+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:21.899969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:21.899969+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"115","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.026455+0000","last_change":"2021-03-15T22:19:13.525956+0000","last_active":"2021-03-15T22:20:44.026455+0000","last_peered":"2021-03-15T22:20:44.026455+0000","last_clean":"2021-03-15T22:20:44.026455+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:20:44.026455+0000","last_undegraded":"2021-03-15T22:20:44.026455+0000","last_fullsized":"2021-03-15T22:20:44.026455+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:19:13.525883+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.525883+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"84","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.021864+0000","last_change":"2021-03-15T22:20:32.006187+0000","last_active":"2021-03-15T22:20:43.021864+0000","last_peered":"2021-03-15T22:20:43.021864+0000","last_clean":"2021-03-15T22:20:43.021864+0000","last_became_active":"2021-03-15T22:18:00.015858+0000","last_became_peered":"2021-03-15T22:18:00.015858+0000","last_unstale":"2021-03-15T22:20:43.021864+0000","last_undegraded":"2021-03-15T22:20:43.021864+0000","last_fullsized":"2021-03-15T22:20:43.021864+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:32.006110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:20:32.006110+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"84","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.025891+0000","last_change":"2021-03-15T22:19:28.737870+0000","last_active":"2021-03-15T22:20:44.025891+0000","last_peered":"2021-03-15T22:20:44.025891+0000","last_clean":"2021-03-15T22:20:44.025891+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:20:44.025891+0000","last_undegraded":"2021-03-15T22:20:44.025891+0000","last_fullsized":"2021-03-15T22:20:44.025891+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.737796+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.737796+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"86","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027872+0000","last_change":"2021-03-15T22:19:01.740592+0000","last_active":"2021-03-15T22:20:44.027872+0000","last_peered":"2021-03-15T22:20:44.027872+0000","last_clean":"2021-03-15T22:20:44.027872+0000","last_became_active":"2021-03-15T22:17:53.973507+0000","last_became_peered":"2021-03-15T22:17:53.973507+0000","last_unstale":"2021-03-15T22:20:44.027872+0000","last_undegraded":"2021-03-15T22:20:44.027872+0000","last_fullsized":"2021-03-15T22:20:44.027872+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:01.740496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:01.740496+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"103","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.015830+0000","last_change":"2021-03-15T22:19:53.846004+0000","last_active":"2021-03-15T22:20:40.015830+0000","last_peered":"2021-03-15T22:20:40.015830+0000","last_clean":"2021-03-15T22:20:40.015830+0000","last_became_active":"2021-03-15T22:17:55.996707+0000","last_became_peered":"2021-03-15T22:17:55.996707+0000","last_unstale":"2021-03-15T22:20:40.015830+0000","last_undegraded":"2021-03-15T22:20:40.015830+0000","last_fullsized":"2021-03-15T22:20:40.015830+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:19:53.845914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:53.845914+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":12,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.016508+0000","last_change":"2021-03-15T22:19:14.850664+0000","last_active":"2021-03-15T22:20:40.016508+0000","last_peered":"2021-03-15T22:20:40.016508+0000","last_clean":"2021-03-15T22:20:40.016508+0000","last_became_active":"2021-03-15T22:18:00.287972+0000","last_became_peered":"2021-03-15T22:18:00.287972+0000","last_unstale":"2021-03-15T22:20:40.016508+0000","last_undegraded":"2021-03-15T22:20:40.016508+0000","last_fullsized":"2021-03-15T22:20:40.016508+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:14.850605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:14.850605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"81","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022006+0000","last_change":"2021-03-15T22:19:27.800200+0000","last_active":"2021-03-15T22:20:43.022006+0000","last_peered":"2021-03-15T22:20:43.022006+0000","last_clean":"2021-03-15T22:20:43.022006+0000","last_became_active":"2021-03-15T22:17:58.013174+0000","last_became_peered":"2021-03-15T22:17:58.013174+0000","last_unstale":"2021-03-15T22:20:43.022006+0000","last_undegraded":"2021-03-15T22:20:43.022006+0000","last_fullsized":"2021-03-15T22:20:43.022006+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:27.800106+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:27.800106+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"83","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.016290+0000","last_change":"2021-03-15T22:19:17.295504+0000","last_active":"2021-03-15T22:20:41.016290+0000","last_peered":"2021-03-15T22:20:41.016290+0000","last_clean":"2021-03-15T22:20:41.016290+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:20:41.016290+0000","last_undegraded":"2021-03-15T22:20:41.016290+0000","last_fullsized":"2021-03-15T22:20:41.016290+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:17.295412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:17.295412+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"90'22","reported_seq":"114","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.015601+0000","last_change":"2021-03-15T22:19:58.799601+0000","last_active":"2021-03-15T22:20:40.015601+0000","last_peered":"2021-03-15T22:20:40.015601+0000","last_clean":"2021-03-15T22:20:40.015601+0000","last_became_active":"2021-03-15T22:17:56.289456+0000","last_became_peered":"2021-03-15T22:17:56.289456+0000","last_unstale":"2021-03-15T22:20:40.015601+0000","last_undegraded":"2021-03-15T22:20:40.015601+0000","last_fullsized":"2021-03-15T22:20:40.015601+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:19:58.799544+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"80","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.022252+0000","last_change":"2021-03-15T22:19:26.615442+0000","last_active":"2021-03-15T22:20:44.022252+0000","last_peered":"2021-03-15T22:20:44.022252+0000","last_clean":"2021-03-15T22:20:44.022252+0000","last_became_active":"2021-03-15T22:18:00.014112+0000","last_became_peered":"2021-03-15T22:18:00.014112+0000","last_unstale":"2021-03-15T22:20:44.022252+0000","last_undegraded":"2021-03-15T22:20:44.022252+0000","last_fullsized":"2021-03-15T22:20:44.022252+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:26.615380+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.615380+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"79","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.016069+0000","last_change":"2021-03-15T22:19:03.699264+0000","last_active":"2021-03-15T22:20:40.016069+0000","last_peered":"2021-03-15T22:20:40.016069+0000","last_clean":"2021-03-15T22:20:40.016069+0000","last_became_active":"2021-03-15T22:17:58.009365+0000","last_became_peered":"2021-03-15T22:17:58.009365+0000","last_unstale":"2021-03-15T22:20:40.016069+0000","last_undegraded":"2021-03-15T22:20:40.016069+0000","last_fullsized":"2021-03-15T22:20:40.016069+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_clean_scrub_stamp":"2021-03-15T22:19:03.699198+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"85","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022494+0000","last_change":"2021-03-15T22:19:36.927055+0000","last_active":"2021-03-15T22:20:43.022494+0000","last_peered":"2021-03-15T22:20:43.022494+0000","last_clean":"2021-03-15T22:20:43.022494+0000","last_became_active":"2021-03-15T22:17:53.979947+0000","last_became_peered":"2021-03-15T22:17:53.979947+0000","last_unstale":"2021-03-15T22:20:43.022494+0000","last_undegraded":"2021-03-15T22:20:43.022494+0000","last_fullsized":"2021-03-15T22:20:43.022494+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:36.926961+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:36.926961+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"109","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.010117+0000","last_change":"2021-03-15T22:19:38.767499+0000","last_active":"2021-03-15T22:20:41.010117+0000","last_peered":"2021-03-15T22:20:41.010117+0000","last_clean":"2021-03-15T22:20:41.010117+0000","last_became_active":"2021-03-15T22:17:55.991537+0000","last_became_peered":"2021-03-15T22:17:55.991537+0000","last_unstale":"2021-03-15T22:20:41.010117+0000","last_undegraded":"2021-03-15T22:20:41.010117+0000","last_fullsized":"2021-03-15T22:20:41.010117+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:38.767414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.767414+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.015376+0000","last_change":"2021-03-15T22:20:23.821510+0000","last_active":"2021-03-15T22:20:40.015376+0000","last_peered":"2021-03-15T22:20:40.015376+0000","last_clean":"2021-03-15T22:20:40.015376+0000","last_became_active":"2021-03-15T22:18:00.018117+0000","last_became_peered":"2021-03-15T22:18:00.018117+0000","last_unstale":"2021-03-15T22:20:40.015376+0000","last_undegraded":"2021-03-15T22:20:40.015376+0000","last_fullsized":"2021-03-15T22:20:40.015376+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:23.821451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:23.821451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"79","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.009972+0000","last_change":"2021-03-15T22:19:32.759869+0000","last_active":"2021-03-15T22:20:41.009972+0000","last_peered":"2021-03-15T22:20:41.009972+0000","last_clean":"2021-03-15T22:20:41.009972+0000","last_became_active":"2021-03-15T22:17:58.014621+0000","last_became_peered":"2021-03-15T22:17:58.014621+0000","last_unstale":"2021-03-15T22:20:41.009972+0000","last_undegraded":"2021-03-15T22:20:41.009972+0000","last_fullsized":"2021-03-15T22:20:41.009972+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:32.759804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:32.759804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"83","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998290+0000","last_change":"2021-03-15T22:19:40.166781+0000","last_active":"2021-03-15T22:20:39.998290+0000","last_peered":"2021-03-15T22:20:39.998290+0000","last_clean":"2021-03-15T22:20:39.998290+0000","last_became_active":"2021-03-15T22:17:53.975292+0000","last_became_peered":"2021-03-15T22:17:53.975292+0000","last_unstale":"2021-03-15T22:20:39.998290+0000","last_undegraded":"2021-03-15T22:20:39.998290+0000","last_fullsized":"2021-03-15T22:20:39.998290+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:40.166704+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.166704+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"108","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998157+0000","last_change":"2021-03-15T22:19:38.156190+0000","last_active":"2021-03-15T22:20:39.998157+0000","last_peered":"2021-03-15T22:20:39.998157+0000","last_clean":"2021-03-15T22:20:39.998157+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:20:39.998157+0000","last_undegraded":"2021-03-15T22:20:39.998157+0000","last_fullsized":"2021-03-15T22:20:39.998157+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:38.156103+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.156103+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"80","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027368+0000","last_change":"2021-03-15T22:19:44.586030+0000","last_active":"2021-03-15T22:20:44.027368+0000","last_peered":"2021-03-15T22:20:44.027368+0000","last_clean":"2021-03-15T22:20:44.027368+0000","last_became_active":"2021-03-15T22:18:00.015034+0000","last_became_peered":"2021-03-15T22:18:00.015034+0000","last_unstale":"2021-03-15T22:20:44.027368+0000","last_undegraded":"2021-03-15T22:20:44.027368+0000","last_fullsized":"2021-03-15T22:20:44.027368+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:44.585947+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:44.585947+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"61'2","reported_seq":"114","reported_epoch":"127","state":"active+clean","last_fresh":"2021-03-15T22:20:42.012938+0000","last_change":"2021-03-15T22:19:39.224283+0000","last_active":"2021-03-15T22:20:42.012938+0000","last_peered":"2021-03-15T22:20:42.012938+0000","last_clean":"2021-03-15T22:20:42.012938+0000","last_became_active":"2021-03-15T22:17:58.010903+0000","last_became_peered":"2021-03-15T22:17:58.010903+0000","last_unstale":"2021-03-15T22:20:42.012938+0000","last_undegraded":"2021-03-15T22:20:42.012938+0000","last_fullsized":"2021-03-15T22:20:42.012938+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:39.224198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:39.224198+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"85","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022233+0000","last_change":"2021-03-15T22:19:43.915597+0000","last_active":"2021-03-15T22:20:43.022233+0000","last_peered":"2021-03-15T22:20:43.022233+0000","last_clean":"2021-03-15T22:20:43.022233+0000","last_became_active":"2021-03-15T22:17:54.285851+0000","last_became_peered":"2021-03-15T22:17:54.285851+0000","last_unstale":"2021-03-15T22:20:43.022233+0000","last_undegraded":"2021-03-15T22:20:43.022233+0000","last_fullsized":"2021-03-15T22:20:43.022233+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:43.915518+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:43.915518+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"109","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.017102+0000","last_change":"2021-03-15T22:20:25.877953+0000","last_active":"2021-03-15T22:20:40.017102+0000","last_peered":"2021-03-15T22:20:40.017102+0000","last_clean":"2021-03-15T22:20:40.017102+0000","last_became_active":"2021-03-15T22:17:56.288102+0000","last_became_peered":"2021-03-15T22:17:56.288102+0000","last_unstale":"2021-03-15T22:20:40.017102+0000","last_undegraded":"2021-03-15T22:20:40.017102+0000","last_fullsized":"2021-03-15T22:20:40.017102+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:20:25.877843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:20:25.877843+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":14,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.999030+0000","last_change":"2021-03-15T22:20:34.242249+0000","last_active":"2021-03-15T22:20:39.999030+0000","last_peered":"2021-03-15T22:20:39.999030+0000","last_clean":"2021-03-15T22:20:39.999030+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:20:39.999030+0000","last_undegraded":"2021-03-15T22:20:39.999030+0000","last_fullsized":"2021-03-15T22:20:39.999030+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:34.242178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:34.242178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"61'2","reported_seq":"113","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.017319+0000","last_change":"2021-03-15T22:19:15.850282+0000","last_active":"2021-03-15T22:20:40.017319+0000","last_peered":"2021-03-15T22:20:40.017319+0000","last_clean":"2021-03-15T22:20:40.017319+0000","last_became_active":"2021-03-15T22:17:58.003748+0000","last_became_peered":"2021-03-15T22:17:58.003748+0000","last_unstale":"2021-03-15T22:20:40.017319+0000","last_undegraded":"2021-03-15T22:20:40.017319+0000","last_fullsized":"2021-03-15T22:20:40.017319+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:15.850217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.850217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"86","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027361+0000","last_change":"2021-03-15T22:19:07.795140+0000","last_active":"2021-03-15T22:20:44.027361+0000","last_peered":"2021-03-15T22:20:44.027361+0000","last_clean":"2021-03-15T22:20:44.027361+0000","last_became_active":"2021-03-15T22:17:53.968736+0000","last_became_peered":"2021-03-15T22:17:53.968736+0000","last_unstale":"2021-03-15T22:20:44.027361+0000","last_undegraded":"2021-03-15T22:20:44.027361+0000","last_fullsized":"2021-03-15T22:20:44.027361+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:07.795072+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:07.795072+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"105","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.015891+0000","last_change":"2021-03-15T22:19:33.261320+0000","last_active":"2021-03-15T22:20:41.015891+0000","last_peered":"2021-03-15T22:20:41.015891+0000","last_clean":"2021-03-15T22:20:41.015891+0000","last_became_active":"2021-03-15T22:17:56.290563+0000","last_became_peered":"2021-03-15T22:17:56.290563+0000","last_unstale":"2021-03-15T22:20:41.015891+0000","last_undegraded":"2021-03-15T22:20:41.015891+0000","last_fullsized":"2021-03-15T22:20:41.015891+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:19:33.261213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:33.261213+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"85","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.009363+0000","last_change":"2021-03-15T22:20:33.854023+0000","last_active":"2021-03-15T22:20:43.009363+0000","last_peered":"2021-03-15T22:20:43.009363+0000","last_clean":"2021-03-15T22:20:43.009363+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:20:43.009363+0000","last_undegraded":"2021-03-15T22:20:43.009363+0000","last_fullsized":"2021-03-15T22:20:43.009363+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:33.853966+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:20:33.853966+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"79","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.018270+0000","last_change":"2021-03-15T22:19:32.880446+0000","last_active":"2021-03-15T22:20:40.018270+0000","last_peered":"2021-03-15T22:20:40.018270+0000","last_clean":"2021-03-15T22:20:40.018270+0000","last_became_active":"2021-03-15T22:17:58.001340+0000","last_became_peered":"2021-03-15T22:17:58.001340+0000","last_unstale":"2021-03-15T22:20:40.018270+0000","last_undegraded":"2021-03-15T22:20:40.018270+0000","last_fullsized":"2021-03-15T22:20:40.018270+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:32.880362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:32.880362+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"87","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.021848+0000","last_change":"2021-03-15T22:20:03.134440+0000","last_active":"2021-03-15T22:20:43.021848+0000","last_peered":"2021-03-15T22:20:43.021848+0000","last_clean":"2021-03-15T22:20:43.021848+0000","last_became_active":"2021-03-15T22:17:53.976777+0000","last_became_peered":"2021-03-15T22:17:53.976777+0000","last_unstale":"2021-03-15T22:20:43.021848+0000","last_undegraded":"2021-03-15T22:20:43.021848+0000","last_fullsized":"2021-03-15T22:20:43.021848+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:03.134336+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:03.134336+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"114","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027692+0000","last_change":"2021-03-15T22:19:06.778692+0000","last_active":"2021-03-15T22:20:44.027692+0000","last_peered":"2021-03-15T22:20:44.027692+0000","last_clean":"2021-03-15T22:20:44.027692+0000","last_became_active":"2021-03-15T22:17:56.288678+0000","last_became_peered":"2021-03-15T22:17:56.288678+0000","last_unstale":"2021-03-15T22:20:44.027692+0000","last_undegraded":"2021-03-15T22:20:44.027692+0000","last_fullsized":"2021-03-15T22:20:44.027692+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:06.778622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:06.778622+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"84","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022413+0000","last_change":"2021-03-15T22:20:35.969982+0000","last_active":"2021-03-15T22:20:43.022413+0000","last_peered":"2021-03-15T22:20:43.022413+0000","last_clean":"2021-03-15T22:20:43.022413+0000","last_became_active":"2021-03-15T22:18:00.288462+0000","last_became_peered":"2021-03-15T22:18:00.288462+0000","last_unstale":"2021-03-15T22:20:43.022413+0000","last_undegraded":"2021-03-15T22:20:43.022413+0000","last_fullsized":"2021-03-15T22:20:43.022413+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:35.969903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:35.969903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"79","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.016339+0000","last_change":"2021-03-15T22:19:29.179877+0000","last_active":"2021-03-15T22:20:41.016339+0000","last_peered":"2021-03-15T22:20:41.016339+0000","last_clean":"2021-03-15T22:20:41.016339+0000","last_became_active":"2021-03-15T22:17:58.015682+0000","last_became_peered":"2021-03-15T22:17:58.015682+0000","last_unstale":"2021-03-15T22:20:41.016339+0000","last_undegraded":"2021-03-15T22:20:41.016339+0000","last_fullsized":"2021-03-15T22:20:41.016339+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:29.179800+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.179800+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"91","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.025517+0000","last_change":"2021-03-15T22:20:06.751237+0000","last_active":"2021-03-15T22:20:44.025517+0000","last_peered":"2021-03-15T22:20:44.025517+0000","last_clean":"2021-03-15T22:20:44.025517+0000","last_became_active":"2021-03-15T22:17:53.977470+0000","last_became_peered":"2021-03-15T22:17:53.977470+0000","last_unstale":"2021-03-15T22:20:44.025517+0000","last_undegraded":"2021-03-15T22:20:44.025517+0000","last_fullsized":"2021-03-15T22:20:44.025517+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:06.751174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:06.751174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'7","reported_seq":"92","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.018448+0000","last_change":"2021-03-15T22:19:47.931238+0000","last_active":"2021-03-15T22:20:40.018448+0000","last_peered":"2021-03-15T22:20:40.018448+0000","last_clean":"2021-03-15T22:20:40.018448+0000","last_became_active":"2021-03-15T22:17:56.289454+0000","last_became_peered":"2021-03-15T22:17:56.289454+0000","last_unstale":"2021-03-15T22:20:40.018448+0000","last_undegraded":"2021-03-15T22:20:40.018448+0000","last_fullsized":"2021-03-15T22:20:40.018448+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'7","last_scrub_stamp":"2021-03-15T22:19:47.931159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:47.931159+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":7,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"122'26","reported_seq":"130","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.008606+0000","last_change":"2021-03-15T22:20:39.098013+0000","last_active":"2021-03-15T22:20:41.008606+0000","last_peered":"2021-03-15T22:20:41.008606+0000","last_clean":"2021-03-15T22:20:41.008606+0000","last_became_active":"2021-03-15T22:20:05.038191+0000","last_became_peered":"2021-03-15T22:20:05.038191+0000","last_unstale":"2021-03-15T22:20:41.008606+0000","last_undegraded":"2021-03-15T22:20:41.008606+0000","last_fullsized":"2021-03-15T22:20:41.008606+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5344560,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":17,"num_read_kb":10281,"num_write":18,"num_write_kb":4547,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"11","length":"1"},{"start":"18","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"82","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.016332+0000","last_change":"2021-03-15T22:20:30.255052+0000","last_active":"2021-03-15T22:20:41.016332+0000","last_peered":"2021-03-15T22:20:41.016332+0000","last_clean":"2021-03-15T22:20:41.016332+0000","last_became_active":"2021-03-15T22:18:00.286699+0000","last_became_peered":"2021-03-15T22:18:00.286699+0000","last_unstale":"2021-03-15T22:20:41.016332+0000","last_undegraded":"2021-03-15T22:20:41.016332+0000","last_fullsized":"2021-03-15T22:20:41.016332+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:30.254978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:30.254978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"93","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.023009+0000","last_change":"2021-03-15T22:20:28.025972+0000","last_active":"2021-03-15T22:20:43.023009+0000","last_peered":"2021-03-15T22:20:43.023009+0000","last_clean":"2021-03-15T22:20:43.023009+0000","last_became_active":"2021-03-15T22:17:53.979246+0000","last_became_peered":"2021-03-15T22:17:53.979246+0000","last_unstale":"2021-03-15T22:20:43.023009+0000","last_undegraded":"2021-03-15T22:20:43.023009+0000","last_fullsized":"2021-03-15T22:20:43.023009+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:20:28.025885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:28.025885+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"112","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.014843+0000","last_change":"2021-03-15T22:19:40.240467+0000","last_active":"2021-03-15T22:20:41.014843+0000","last_peered":"2021-03-15T22:20:41.014843+0000","last_clean":"2021-03-15T22:20:41.014843+0000","last_became_active":"2021-03-15T22:17:56.289684+0000","last_became_peered":"2021-03-15T22:17:56.289684+0000","last_unstale":"2021-03-15T22:20:41.014843+0000","last_undegraded":"2021-03-15T22:20:41.014843+0000","last_fullsized":"2021-03-15T22:20:41.014843+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:19:40.240388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:40.240388+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"89'3","reported_seq":"117","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022001+0000","last_change":"2021-03-15T22:19:51.938202+0000","last_active":"2021-03-15T22:20:43.022001+0000","last_peered":"2021-03-15T22:20:43.022001+0000","last_clean":"2021-03-15T22:20:43.022001+0000","last_became_active":"2021-03-15T22:17:58.013784+0000","last_became_peered":"2021-03-15T22:17:58.013784+0000","last_unstale":"2021-03-15T22:20:43.022001+0000","last_undegraded":"2021-03-15T22:20:43.022001+0000","last_fullsized":"2021-03-15T22:20:43.022001+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'3","last_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:19:51.938126+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"126'53","reported_seq":"156","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.014746+0000","last_change":"2021-03-15T22:20:39.102939+0000","last_active":"2021-03-15T22:20:41.014746+0000","last_peered":"2021-03-15T22:20:41.014746+0000","last_clean":"2021-03-15T22:20:41.014746+0000","last_became_active":"2021-03-15T22:20:05.034711+0000","last_became_peered":"2021-03-15T22:20:05.034711+0000","last_unstale":"2021-03-15T22:20:41.014746+0000","last_undegraded":"2021-03-15T22:20:41.014746+0000","last_fullsized":"2021-03-15T22:20:41.014746+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":53,"ondisk_log_size":53,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10873997,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":2,"num_read":22,"num_read_kb":3417,"num_write":40,"num_write_kb":9668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"},{"start":"18","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.016220+0000","last_change":"2021-03-15T22:19:37.905329+0000","last_active":"2021-03-15T22:20:40.016220+0000","last_peered":"2021-03-15T22:20:40.016220+0000","last_clean":"2021-03-15T22:20:40.016220+0000","last_became_active":"2021-03-15T22:18:00.019686+0000","last_became_peered":"2021-03-15T22:18:00.019686+0000","last_unstale":"2021-03-15T22:20:40.016220+0000","last_undegraded":"2021-03-15T22:20:40.016220+0000","last_fullsized":"2021-03-15T22:20:40.016220+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:37.905267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:37.905267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"90","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022773+0000","last_change":"2021-03-15T22:20:27.003318+0000","last_active":"2021-03-15T22:20:43.022773+0000","last_peered":"2021-03-15T22:20:43.022773+0000","last_clean":"2021-03-15T22:20:43.022773+0000","last_became_active":"2021-03-15T22:17:53.978907+0000","last_became_peered":"2021-03-15T22:17:53.978907+0000","last_unstale":"2021-03-15T22:20:43.022773+0000","last_undegraded":"2021-03-15T22:20:43.022773+0000","last_fullsized":"2021-03-15T22:20:43.022773+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:27.003241+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:27.003241+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"110","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998364+0000","last_change":"2021-03-15T22:20:33.237095+0000","last_active":"2021-03-15T22:20:39.998364+0000","last_peered":"2021-03-15T22:20:39.998364+0000","last_clean":"2021-03-15T22:20:39.998364+0000","last_became_active":"2021-03-15T22:17:55.992386+0000","last_became_peered":"2021-03-15T22:17:55.992386+0000","last_unstale":"2021-03-15T22:20:39.998364+0000","last_undegraded":"2021-03-15T22:20:39.998364+0000","last_fullsized":"2021-03-15T22:20:39.998364+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:20:33.236981+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:20:33.236981+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"82","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.008252+0000","last_change":"2021-03-15T22:19:18.197284+0000","last_active":"2021-03-15T22:20:43.008252+0000","last_peered":"2021-03-15T22:20:43.008252+0000","last_clean":"2021-03-15T22:20:43.008252+0000","last_became_active":"2021-03-15T22:17:58.007916+0000","last_became_peered":"2021-03-15T22:17:58.007916+0000","last_unstale":"2021-03-15T22:20:43.008252+0000","last_undegraded":"2021-03-15T22:20:43.008252+0000","last_fullsized":"2021-03-15T22:20:43.008252+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.197224+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.197224+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"130'17","reported_seq":"122","reported_epoch":"130","state":"active+clean","last_fresh":"2021-03-15T22:20:45.063164+0000","last_change":"2021-03-15T22:20:39.100287+0000","last_active":"2021-03-15T22:20:45.063164+0000","last_peered":"2021-03-15T22:20:45.063164+0000","last_clean":"2021-03-15T22:20:45.063164+0000","last_became_active":"2021-03-15T22:20:05.032301+0000","last_became_peered":"2021-03-15T22:20:05.032301+0000","last_unstale":"2021-03-15T22:20:45.063164+0000","last_undegraded":"2021-03-15T22:20:45.063164+0000","last_fullsized":"2021-03-15T22:20:45.063164+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6229102,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":6099,"num_write":14,"num_write_kb":4113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"79","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.021759+0000","last_change":"2021-03-15T22:19:09.989290+0000","last_active":"2021-03-15T22:20:43.021759+0000","last_peered":"2021-03-15T22:20:43.021759+0000","last_clean":"2021-03-15T22:20:43.021759+0000","last_became_active":"2021-03-15T22:18:00.016271+0000","last_became_peered":"2021-03-15T22:18:00.016271+0000","last_unstale":"2021-03-15T22:20:43.021759+0000","last_undegraded":"2021-03-15T22:20:43.021759+0000","last_fullsized":"2021-03-15T22:20:43.021759+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:09.989228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:09.989228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"82","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.008739+0000","last_change":"2021-03-15T22:19:29.099735+0000","last_active":"2021-03-15T22:20:43.008739+0000","last_peered":"2021-03-15T22:20:43.008739+0000","last_clean":"2021-03-15T22:20:43.008739+0000","last_became_active":"2021-03-15T22:17:58.010511+0000","last_became_peered":"2021-03-15T22:17:58.010511+0000","last_unstale":"2021-03-15T22:20:43.008739+0000","last_undegraded":"2021-03-15T22:20:43.008739+0000","last_fullsized":"2021-03-15T22:20:43.008739+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.099650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.099650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"85","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022578+0000","last_change":"2021-03-15T22:19:47.956238+0000","last_active":"2021-03-15T22:20:43.022578+0000","last_peered":"2021-03-15T22:20:43.022578+0000","last_clean":"2021-03-15T22:20:43.022578+0000","last_became_active":"2021-03-15T22:17:53.974108+0000","last_became_peered":"2021-03-15T22:17:53.974108+0000","last_unstale":"2021-03-15T22:20:43.022578+0000","last_undegraded":"2021-03-15T22:20:43.022578+0000","last_fullsized":"2021-03-15T22:20:43.022578+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:47.956156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:47.956156+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"123","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.025846+0000","last_change":"2021-03-15T22:19:22.700145+0000","last_active":"2021-03-15T22:20:44.025846+0000","last_peered":"2021-03-15T22:20:44.025846+0000","last_clean":"2021-03-15T22:20:44.025846+0000","last_became_active":"2021-03-15T22:17:56.288870+0000","last_became_peered":"2021-03-15T22:17:56.288870+0000","last_unstale":"2021-03-15T22:20:44.025846+0000","last_undegraded":"2021-03-15T22:20:44.025846+0000","last_fullsized":"2021-03-15T22:20:44.025846+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:19:22.700049+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.700049+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"129'49","reported_seq":"154","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.071489+0000","last_change":"2021-03-15T22:20:39.103506+0000","last_active":"2021-03-15T22:20:44.071489+0000","last_peered":"2021-03-15T22:20:44.071489+0000","last_clean":"2021-03-15T22:20:44.071489+0000","last_became_active":"2021-03-15T22:20:05.032282+0000","last_became_peered":"2021-03-15T22:20:05.032282+0000","last_unstale":"2021-03-15T22:20:44.071489+0000","last_undegraded":"2021-03-15T22:20:44.071489+0000","last_fullsized":"2021-03-15T22:20:44.071489+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":49,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14115814,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":6377,"num_write":35,"num_write_kb":14976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"},{"start":"18","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"80","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027162+0000","last_change":"2021-03-15T22:19:39.681802+0000","last_active":"2021-03-15T22:20:44.027162+0000","last_peered":"2021-03-15T22:20:44.027162+0000","last_clean":"2021-03-15T22:20:44.027162+0000","last_became_active":"2021-03-15T22:18:00.016982+0000","last_became_peered":"2021-03-15T22:18:00.016982+0000","last_unstale":"2021-03-15T22:20:44.027162+0000","last_undegraded":"2021-03-15T22:20:44.027162+0000","last_fullsized":"2021-03-15T22:20:44.027162+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:39.681741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:39.681741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"87","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.009078+0000","last_change":"2021-03-15T22:20:23.864892+0000","last_active":"2021-03-15T22:20:43.009078+0000","last_peered":"2021-03-15T22:20:43.009078+0000","last_clean":"2021-03-15T22:20:43.009078+0000","last_became_active":"2021-03-15T22:17:58.001399+0000","last_became_peered":"2021-03-15T22:17:58.001399+0000","last_unstale":"2021-03-15T22:20:43.009078+0000","last_undegraded":"2021-03-15T22:20:43.009078+0000","last_fullsized":"2021-03-15T22:20:43.009078+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:23.864824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:20:23.864824+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"83","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.014868+0000","last_change":"2021-03-15T22:19:18.252532+0000","last_active":"2021-03-15T22:20:41.014868+0000","last_peered":"2021-03-15T22:20:41.014868+0000","last_clean":"2021-03-15T22:20:41.014868+0000","last_became_active":"2021-03-15T22:17:53.972006+0000","last_became_peered":"2021-03-15T22:17:53.972006+0000","last_unstale":"2021-03-15T22:20:41.014868+0000","last_undegraded":"2021-03-15T22:20:41.014868+0000","last_fullsized":"2021-03-15T22:20:41.014868+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.252456+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.252456+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"139","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027524+0000","last_change":"2021-03-15T22:19:28.581985+0000","last_active":"2021-03-15T22:20:44.027524+0000","last_peered":"2021-03-15T22:20:44.027524+0000","last_clean":"2021-03-15T22:20:44.027524+0000","last_became_active":"2021-03-15T22:19:02.093463+0000","last_became_peered":"2021-03-15T22:19:02.093463+0000","last_unstale":"2021-03-15T22:20:44.027524+0000","last_undegraded":"2021-03-15T22:20:44.027524+0000","last_fullsized":"2021-03-15T22:20:44.027524+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:19:28.581920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.581920+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"115'36","reported_seq":"137","reported_epoch":"130","state":"active+clean","last_fresh":"2021-03-15T22:20:45.330495+0000","last_change":"2021-03-15T22:20:39.524625+0000","last_active":"2021-03-15T22:20:45.330495+0000","last_peered":"2021-03-15T22:20:45.330495+0000","last_clean":"2021-03-15T22:20:45.330495+0000","last_became_active":"2021-03-15T22:20:05.036550+0000","last_became_peered":"2021-03-15T22:20:05.036550+0000","last_unstale":"2021-03-15T22:20:45.330495+0000","last_undegraded":"2021-03-15T22:20:45.330495+0000","last_fullsized":"2021-03-15T22:20:45.330495+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5260525,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":5,"num_write":25,"num_write_kb":8385,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.017843+0000","last_change":"2021-03-15T22:19:19.908283+0000","last_active":"2021-03-15T22:20:40.017843+0000","last_peered":"2021-03-15T22:20:40.017843+0000","last_clean":"2021-03-15T22:20:40.017843+0000","last_became_active":"2021-03-15T22:18:00.020501+0000","last_became_peered":"2021-03-15T22:18:00.020501+0000","last_unstale":"2021-03-15T22:20:40.017843+0000","last_undegraded":"2021-03-15T22:20:40.017843+0000","last_fullsized":"2021-03-15T22:20:40.017843+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.908218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.908218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"90","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.016130+0000","last_change":"2021-03-15T22:20:29.221010+0000","last_active":"2021-03-15T22:20:41.016130+0000","last_peered":"2021-03-15T22:20:41.016130+0000","last_clean":"2021-03-15T22:20:41.016130+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:20:41.016130+0000","last_undegraded":"2021-03-15T22:20:41.016130+0000","last_fullsized":"2021-03-15T22:20:41.016130+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:29.220910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:20:29.220910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"83","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998076+0000","last_change":"2021-03-15T22:20:00.206145+0000","last_active":"2021-03-15T22:20:39.998076+0000","last_peered":"2021-03-15T22:20:39.998076+0000","last_clean":"2021-03-15T22:20:39.998076+0000","last_became_active":"2021-03-15T22:17:53.978521+0000","last_became_peered":"2021-03-15T22:17:53.978521+0000","last_unstale":"2021-03-15T22:20:39.998076+0000","last_undegraded":"2021-03-15T22:20:39.998076+0000","last_fullsized":"2021-03-15T22:20:39.998076+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:00.206068+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:00.206068+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"146","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.010536+0000","last_change":"2021-03-15T22:20:35.937829+0000","last_active":"2021-03-15T22:20:41.010536+0000","last_peered":"2021-03-15T22:20:41.010536+0000","last_clean":"2021-03-15T22:20:41.010536+0000","last_became_active":"2021-03-15T22:17:55.995319+0000","last_became_peered":"2021-03-15T22:17:55.995319+0000","last_unstale":"2021-03-15T22:20:41.010536+0000","last_undegraded":"2021-03-15T22:20:41.010536+0000","last_fullsized":"2021-03-15T22:20:41.010536+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:20:35.937761+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:20:35.937761+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"87","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.028151+0000","last_change":"2021-03-15T22:20:30.455518+0000","last_active":"2021-03-15T22:20:44.028151+0000","last_peered":"2021-03-15T22:20:44.028151+0000","last_clean":"2021-03-15T22:20:44.028151+0000","last_became_active":"2021-03-15T22:17:58.017434+0000","last_became_peered":"2021-03-15T22:17:58.017434+0000","last_unstale":"2021-03-15T22:20:44.028151+0000","last_undegraded":"2021-03-15T22:20:44.028151+0000","last_fullsized":"2021-03-15T22:20:44.028151+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:30.455430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:20:30.455430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"130'41","reported_seq":"145","reported_epoch":"130","state":"active+clean","last_fresh":"2021-03-15T22:20:45.042343+0000","last_change":"2021-03-15T22:20:39.095995+0000","last_active":"2021-03-15T22:20:45.042343+0000","last_peered":"2021-03-15T22:20:45.042343+0000","last_clean":"2021-03-15T22:20:45.042343+0000","last_became_active":"2021-03-15T22:20:05.038149+0000","last_became_peered":"2021-03-15T22:20:05.038149+0000","last_unstale":"2021-03-15T22:20:45.042343+0000","last_undegraded":"2021-03-15T22:20:45.042343+0000","last_fullsized":"2021-03-15T22:20:45.042343+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9951201,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":21,"num_read_kb":12589,"num_write":33,"num_write_kb":11568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"79","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022705+0000","last_change":"2021-03-15T22:19:31.885077+0000","last_active":"2021-03-15T22:20:43.022705+0000","last_peered":"2021-03-15T22:20:43.022705+0000","last_clean":"2021-03-15T22:20:43.022705+0000","last_became_active":"2021-03-15T22:18:00.287780+0000","last_became_peered":"2021-03-15T22:18:00.287780+0000","last_unstale":"2021-03-15T22:20:43.022705+0000","last_undegraded":"2021-03-15T22:20:43.022705+0000","last_fullsized":"2021-03-15T22:20:43.022705+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:31.884965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:31.884965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"88","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.008701+0000","last_change":"2021-03-15T22:20:33.849147+0000","last_active":"2021-03-15T22:20:41.008701+0000","last_peered":"2021-03-15T22:20:41.008701+0000","last_clean":"2021-03-15T22:20:41.008701+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:20:41.008701+0000","last_undegraded":"2021-03-15T22:20:41.008701+0000","last_fullsized":"2021-03-15T22:20:41.008701+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:33.849083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:33.849083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"89","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.008870+0000","last_change":"2021-03-15T22:19:24.142618+0000","last_active":"2021-03-15T22:20:43.008870+0000","last_peered":"2021-03-15T22:20:43.008870+0000","last_clean":"2021-03-15T22:20:43.008870+0000","last_became_active":"2021-03-15T22:17:51.631460+0000","last_became_peered":"2021-03-15T22:17:51.631460+0000","last_unstale":"2021-03-15T22:20:43.008870+0000","last_undegraded":"2021-03-15T22:20:43.008870+0000","last_fullsized":"2021-03-15T22:20:43.008870+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.142558+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.142558+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"65'15","reported_seq":"112","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.008992+0000","last_change":"2021-03-15T22:19:24.648256+0000","last_active":"2021-03-15T22:20:41.008992+0000","last_peered":"2021-03-15T22:20:41.008992+0000","last_clean":"2021-03-15T22:20:41.008992+0000","last_became_active":"2021-03-15T22:17:55.992952+0000","last_became_peered":"2021-03-15T22:17:55.992952+0000","last_unstale":"2021-03-15T22:20:41.008992+0000","last_undegraded":"2021-03-15T22:20:41.008992+0000","last_fullsized":"2021-03-15T22:20:41.008992+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:24.648200+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.648200+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"87","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.009037+0000","last_change":"2021-03-15T22:20:32.860213+0000","last_active":"2021-03-15T22:20:43.009037+0000","last_peered":"2021-03-15T22:20:43.009037+0000","last_clean":"2021-03-15T22:20:43.009037+0000","last_became_active":"2021-03-15T22:17:58.014849+0000","last_became_peered":"2021-03-15T22:17:58.014849+0000","last_unstale":"2021-03-15T22:20:43.009037+0000","last_undegraded":"2021-03-15T22:20:43.009037+0000","last_fullsized":"2021-03-15T22:20:43.009037+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:32.860152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:20:32.860152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"130'101","reported_seq":"214","reported_epoch":"130","state":"active+clean","last_fresh":"2021-03-15T22:20:44.997381+0000","last_change":"2021-03-15T22:20:39.097244+0000","last_active":"2021-03-15T22:20:44.997381+0000","last_peered":"2021-03-15T22:20:44.997381+0000","last_clean":"2021-03-15T22:20:44.997381+0000","last_became_active":"2021-03-15T22:20:05.035885+0000","last_became_peered":"2021-03-15T22:20:05.035885+0000","last_unstale":"2021-03-15T22:20:44.997381+0000","last_undegraded":"2021-03-15T22:20:44.997381+0000","last_fullsized":"2021-03-15T22:20:44.997381+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":101,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35669313,"num_objects":14,"num_object_clones":7,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":48,"num_read_kb":19673,"num_write":74,"num_write_kb":35660,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"77","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.016354+0000","last_change":"2021-03-15T22:19:25.184338+0000","last_active":"2021-03-15T22:20:41.016354+0000","last_peered":"2021-03-15T22:20:41.016354+0000","last_clean":"2021-03-15T22:20:41.016354+0000","last_became_active":"2021-03-15T22:18:00.019363+0000","last_became_peered":"2021-03-15T22:18:00.019363+0000","last_unstale":"2021-03-15T22:20:41.016354+0000","last_undegraded":"2021-03-15T22:20:41.016354+0000","last_fullsized":"2021-03-15T22:20:41.016354+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.184260+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.184260+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"88","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.021852+0000","last_change":"2021-03-15T22:19:23.843965+0000","last_active":"2021-03-15T22:20:43.021852+0000","last_peered":"2021-03-15T22:20:43.021852+0000","last_clean":"2021-03-15T22:20:43.021852+0000","last_became_active":"2021-03-15T22:17:51.044140+0000","last_became_peered":"2021-03-15T22:17:51.044140+0000","last_unstale":"2021-03-15T22:20:43.021852+0000","last_undegraded":"2021-03-15T22:20:43.021852+0000","last_fullsized":"2021-03-15T22:20:43.021852+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:23.843892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.843892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"83","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.017937+0000","last_change":"2021-03-15T22:19:20.880709+0000","last_active":"2021-03-15T22:20:40.017937+0000","last_peered":"2021-03-15T22:20:40.017937+0000","last_clean":"2021-03-15T22:20:40.017937+0000","last_became_active":"2021-03-15T22:17:53.972080+0000","last_became_peered":"2021-03-15T22:17:53.972080+0000","last_unstale":"2021-03-15T22:20:40.017937+0000","last_undegraded":"2021-03-15T22:20:40.017937+0000","last_fullsized":"2021-03-15T22:20:40.017937+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:20.880638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:20.880638+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"108","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.026255+0000","last_change":"2021-03-15T22:19:23.732253+0000","last_active":"2021-03-15T22:20:44.026255+0000","last_peered":"2021-03-15T22:20:44.026255+0000","last_clean":"2021-03-15T22:20:44.026255+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:20:44.026255+0000","last_undegraded":"2021-03-15T22:20:44.026255+0000","last_fullsized":"2021-03-15T22:20:44.026255+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:19:23.732187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:23.732187+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"86","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022511+0000","last_change":"2021-03-15T22:20:31.051075+0000","last_active":"2021-03-15T22:20:43.022511+0000","last_peered":"2021-03-15T22:20:43.022511+0000","last_clean":"2021-03-15T22:20:43.022511+0000","last_became_active":"2021-03-15T22:17:58.011911+0000","last_became_peered":"2021-03-15T22:17:58.011911+0000","last_unstale":"2021-03-15T22:20:43.022511+0000","last_undegraded":"2021-03-15T22:20:43.022511+0000","last_fullsized":"2021-03-15T22:20:43.022511+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:31.050970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:31.050970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"122'30","reported_seq":"135","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022385+0000","last_change":"2021-03-15T22:20:39.105078+0000","last_active":"2021-03-15T22:20:43.022385+0000","last_peered":"2021-03-15T22:20:43.022385+0000","last_clean":"2021-03-15T22:20:43.022385+0000","last_became_active":"2021-03-15T22:20:05.035608+0000","last_became_peered":"2021-03-15T22:20:05.035608+0000","last_unstale":"2021-03-15T22:20:43.022385+0000","last_undegraded":"2021-03-15T22:20:43.022385+0000","last_fullsized":"2021-03-15T22:20:43.022385+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6554664,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":20,"num_read_kb":14146,"num_write":24,"num_write_kb":8694,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"11","length":"1"},{"start":"18","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"77","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.010278+0000","last_change":"2021-03-15T22:19:28.773808+0000","last_active":"2021-03-15T22:20:41.010278+0000","last_peered":"2021-03-15T22:20:41.010278+0000","last_clean":"2021-03-15T22:20:41.010278+0000","last_became_active":"2021-03-15T22:18:00.014336+0000","last_became_peered":"2021-03-15T22:18:00.014336+0000","last_unstale":"2021-03-15T22:20:41.010278+0000","last_undegraded":"2021-03-15T22:20:41.010278+0000","last_fullsized":"2021-03-15T22:20:41.010278+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:28.773747+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:28.773747+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"86","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027367+0000","last_change":"2021-03-15T22:19:38.721599+0000","last_active":"2021-03-15T22:20:44.027367+0000","last_peered":"2021-03-15T22:20:44.027367+0000","last_clean":"2021-03-15T22:20:44.027367+0000","last_became_active":"2021-03-15T22:17:53.977311+0000","last_became_peered":"2021-03-15T22:17:53.977311+0000","last_unstale":"2021-03-15T22:20:44.027367+0000","last_undegraded":"2021-03-15T22:20:44.027367+0000","last_fullsized":"2021-03-15T22:20:44.027367+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:38.721534+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:38.721534+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"265","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.532008+0000","last_change":"2021-03-15T22:20:36.923986+0000","last_active":"2021-03-15T22:20:43.532008+0000","last_peered":"2021-03-15T22:20:43.532008+0000","last_clean":"2021-03-15T22:20:43.532008+0000","last_became_active":"2021-03-15T22:17:51.044139+0000","last_became_peered":"2021-03-15T22:17:51.044139+0000","last_unstale":"2021-03-15T22:20:43.532008+0000","last_undegraded":"2021-03-15T22:20:43.532008+0000","last_fullsized":"2021-03-15T22:20:43.532008+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:20:36.923907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:20:36.923907+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":170,"num_read_kb":165,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"110","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027464+0000","last_change":"2021-03-15T22:19:30.589143+0000","last_active":"2021-03-15T22:20:44.027464+0000","last_peered":"2021-03-15T22:20:44.027464+0000","last_clean":"2021-03-15T22:20:44.027464+0000","last_became_active":"2021-03-15T22:17:55.999564+0000","last_became_peered":"2021-03-15T22:17:55.999564+0000","last_unstale":"2021-03-15T22:20:44.027464+0000","last_undegraded":"2021-03-15T22:20:44.027464+0000","last_fullsized":"2021-03-15T22:20:44.027464+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:30.589062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:30.589062+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"84","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.016581+0000","last_change":"2021-03-15T22:20:36.215145+0000","last_active":"2021-03-15T22:20:41.016581+0000","last_peered":"2021-03-15T22:20:41.016581+0000","last_clean":"2021-03-15T22:20:41.016581+0000","last_became_active":"2021-03-15T22:17:58.010798+0000","last_became_peered":"2021-03-15T22:17:58.010798+0000","last_unstale":"2021-03-15T22:20:41.016581+0000","last_undegraded":"2021-03-15T22:20:41.016581+0000","last_fullsized":"2021-03-15T22:20:41.016581+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:20:36.215040+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"124'98","reported_seq":"217","reported_epoch":"127","state":"active+clean","last_fresh":"2021-03-15T22:20:42.013231+0000","last_change":"2021-03-15T22:20:39.102932+0000","last_active":"2021-03-15T22:20:42.013231+0000","last_peered":"2021-03-15T22:20:42.013231+0000","last_clean":"2021-03-15T22:20:42.013231+0000","last_became_active":"2021-03-15T22:20:05.039329+0000","last_became_peered":"2021-03-15T22:20:05.039329+0000","last_unstale":"2021-03-15T22:20:42.013231+0000","last_undegraded":"2021-03-15T22:20:42.013231+0000","last_fullsized":"2021-03-15T22:20:42.013231+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":98,"ondisk_log_size":98,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21139839,"num_objects":8,"num_object_clones":1,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":93,"num_read_kb":78996,"num_write":70,"num_write_kb":27123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"11","length":"1"},{"start":"18","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"77","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.009292+0000","last_change":"2021-03-15T22:19:15.593372+0000","last_active":"2021-03-15T22:20:41.009292+0000","last_peered":"2021-03-15T22:20:41.009292+0000","last_clean":"2021-03-15T22:20:41.009292+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:20:41.009292+0000","last_undegraded":"2021-03-15T22:20:41.009292+0000","last_fullsized":"2021-03-15T22:20:41.009292+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:19:15.593307+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"83","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.010475+0000","last_change":"2021-03-15T22:19:29.812510+0000","last_active":"2021-03-15T22:20:41.010475+0000","last_peered":"2021-03-15T22:20:41.010475+0000","last_clean":"2021-03-15T22:20:41.010475+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:20:41.010475+0000","last_undegraded":"2021-03-15T22:20:41.010475+0000","last_fullsized":"2021-03-15T22:20:41.010475+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:29.812446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:29.812446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"139","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027817+0000","last_change":"2021-03-15T22:20:28.505643+0000","last_active":"2021-03-15T22:20:44.027817+0000","last_peered":"2021-03-15T22:20:44.027817+0000","last_clean":"2021-03-15T22:20:44.027817+0000","last_became_active":"2021-03-15T22:17:56.288438+0000","last_became_peered":"2021-03-15T22:17:56.288438+0000","last_unstale":"2021-03-15T22:20:44.027817+0000","last_undegraded":"2021-03-15T22:20:44.027817+0000","last_fullsized":"2021-03-15T22:20:44.027817+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:20:28.505568+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:20:28.505568+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"82","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.026019+0000","last_change":"2021-03-15T22:19:16.581584+0000","last_active":"2021-03-15T22:20:44.026019+0000","last_peered":"2021-03-15T22:20:44.026019+0000","last_clean":"2021-03-15T22:20:44.026019+0000","last_became_active":"2021-03-15T22:17:58.008731+0000","last_became_peered":"2021-03-15T22:17:58.008731+0000","last_unstale":"2021-03-15T22:20:44.026019+0000","last_undegraded":"2021-03-15T22:20:44.026019+0000","last_fullsized":"2021-03-15T22:20:44.026019+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:19:16.581514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"115'9","reported_seq":"107","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.997584+0000","last_change":"2021-03-15T22:20:39.096150+0000","last_active":"2021-03-15T22:20:39.997584+0000","last_peered":"2021-03-15T22:20:39.997584+0000","last_clean":"2021-03-15T22:20:39.997584+0000","last_became_active":"2021-03-15T22:20:05.033792+0000","last_became_peered":"2021-03-15T22:20:05.033792+0000","last_unstale":"2021-03-15T22:20:39.997584+0000","last_undegraded":"2021-03-15T22:20:39.997584+0000","last_fullsized":"2021-03-15T22:20:39.997584+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2268912,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":6397,"num_write":6,"num_write_kb":980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"11","length":"1"},{"start":"18","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"80","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.022397+0000","last_change":"2021-03-15T22:19:19.751342+0000","last_active":"2021-03-15T22:20:44.022397+0000","last_peered":"2021-03-15T22:20:44.022397+0000","last_clean":"2021-03-15T22:20:44.022397+0000","last_became_active":"2021-03-15T22:18:00.015069+0000","last_became_peered":"2021-03-15T22:18:00.015069+0000","last_unstale":"2021-03-15T22:20:44.022397+0000","last_undegraded":"2021-03-15T22:20:44.022397+0000","last_fullsized":"2021-03-15T22:20:44.022397+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:19.751282+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:19.751282+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"83","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.015813+0000","last_change":"2021-03-15T22:19:41.212619+0000","last_active":"2021-03-15T22:20:41.015813+0000","last_peered":"2021-03-15T22:20:41.015813+0000","last_clean":"2021-03-15T22:20:41.015813+0000","last_became_active":"2021-03-15T22:17:54.284266+0000","last_became_peered":"2021-03-15T22:17:54.284266+0000","last_unstale":"2021-03-15T22:20:41.015813+0000","last_undegraded":"2021-03-15T22:20:41.015813+0000","last_fullsized":"2021-03-15T22:20:41.015813+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:41.212539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:41.212539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"110","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.010017+0000","last_change":"2021-03-15T22:19:43.737187+0000","last_active":"2021-03-15T22:20:41.010017+0000","last_peered":"2021-03-15T22:20:41.010017+0000","last_clean":"2021-03-15T22:20:41.010017+0000","last_became_active":"2021-03-15T22:17:55.992257+0000","last_became_peered":"2021-03-15T22:17:55.992257+0000","last_unstale":"2021-03-15T22:20:41.010017+0000","last_undegraded":"2021-03-15T22:20:41.010017+0000","last_fullsized":"2021-03-15T22:20:41.010017+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:19:43.737126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:43.737126+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"61'2","reported_seq":"117","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027426+0000","last_change":"2021-03-15T22:19:25.604285+0000","last_active":"2021-03-15T22:20:44.027426+0000","last_peered":"2021-03-15T22:20:44.027426+0000","last_clean":"2021-03-15T22:20:44.027426+0000","last_became_active":"2021-03-15T22:17:58.008973+0000","last_became_peered":"2021-03-15T22:17:58.008973+0000","last_unstale":"2021-03-15T22:20:44.027426+0000","last_undegraded":"2021-03-15T22:20:44.027426+0000","last_fullsized":"2021-03-15T22:20:44.027426+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:25.604217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.604217+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"124'16","reported_seq":"116","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.018064+0000","last_change":"2021-03-15T22:20:39.103379+0000","last_active":"2021-03-15T22:20:40.018064+0000","last_peered":"2021-03-15T22:20:40.018064+0000","last_clean":"2021-03-15T22:20:40.018064+0000","last_became_active":"2021-03-15T22:20:05.039511+0000","last_became_peered":"2021-03-15T22:20:05.039511+0000","last_unstale":"2021-03-15T22:20:40.018064+0000","last_undegraded":"2021-03-15T22:20:40.018064+0000","last_fullsized":"2021-03-15T22:20:40.018064+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8352980,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":15,"num_read_kb":8933,"num_write":13,"num_write_kb":6575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"},{"start":"18","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"84","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.022461+0000","last_change":"2021-03-15T22:20:30.021482+0000","last_active":"2021-03-15T22:20:43.022461+0000","last_peered":"2021-03-15T22:20:43.022461+0000","last_clean":"2021-03-15T22:20:43.022461+0000","last_became_active":"2021-03-15T22:18:00.288441+0000","last_became_peered":"2021-03-15T22:18:00.288441+0000","last_unstale":"2021-03-15T22:20:43.022461+0000","last_undegraded":"2021-03-15T22:20:43.022461+0000","last_fullsized":"2021-03-15T22:20:43.022461+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:30.021411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:30.021411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"86","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.027516+0000","last_change":"2021-03-15T22:19:13.870267+0000","last_active":"2021-03-15T22:20:44.027516+0000","last_peered":"2021-03-15T22:20:44.027516+0000","last_clean":"2021-03-15T22:20:44.027516+0000","last_became_active":"2021-03-15T22:17:53.976307+0000","last_became_peered":"2021-03-15T22:17:53.976307+0000","last_unstale":"2021-03-15T22:20:44.027516+0000","last_undegraded":"2021-03-15T22:20:44.027516+0000","last_fullsized":"2021-03-15T22:20:44.027516+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:13.870202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:13.870202+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"114","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998235+0000","last_change":"2021-03-15T22:19:26.087704+0000","last_active":"2021-03-15T22:20:39.998235+0000","last_peered":"2021-03-15T22:20:39.998235+0000","last_clean":"2021-03-15T22:20:39.998235+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:20:39.998235+0000","last_undegraded":"2021-03-15T22:20:39.998235+0000","last_fullsized":"2021-03-15T22:20:39.998235+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:19:26.087622+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:26.087622+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"61'2","reported_seq":"113","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.016525+0000","last_change":"2021-03-15T22:20:00.809142+0000","last_active":"2021-03-15T22:20:40.016525+0000","last_peered":"2021-03-15T22:20:40.016525+0000","last_clean":"2021-03-15T22:20:40.016525+0000","last_became_active":"2021-03-15T22:17:58.011279+0000","last_became_peered":"2021-03-15T22:17:58.011279+0000","last_unstale":"2021-03-15T22:20:40.016525+0000","last_undegraded":"2021-03-15T22:20:40.016525+0000","last_fullsized":"2021-03-15T22:20:40.016525+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:20:00.809069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:20:00.809069+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"122'32","reported_seq":"140","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.016383+0000","last_change":"2021-03-15T22:20:39.098558+0000","last_active":"2021-03-15T22:20:40.016383+0000","last_peered":"2021-03-15T22:20:40.016383+0000","last_clean":"2021-03-15T22:20:40.016383+0000","last_became_active":"2021-03-15T22:20:05.034804+0000","last_became_peered":"2021-03-15T22:20:05.034804+0000","last_unstale":"2021-03-15T22:20:40.016383+0000","last_undegraded":"2021-03-15T22:20:40.016383+0000","last_fullsized":"2021-03-15T22:20:40.016383+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":21992,"num_write":22,"num_write_kb":8884,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"11","length":"1"},{"start":"18","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.997674+0000","last_change":"2021-03-15T22:19:34.179513+0000","last_active":"2021-03-15T22:20:39.997674+0000","last_peered":"2021-03-15T22:20:39.997674+0000","last_clean":"2021-03-15T22:20:39.997674+0000","last_became_active":"2021-03-15T22:18:00.021088+0000","last_became_peered":"2021-03-15T22:18:00.021088+0000","last_unstale":"2021-03-15T22:20:39.997674+0000","last_undegraded":"2021-03-15T22:20:39.997674+0000","last_fullsized":"2021-03-15T22:20:39.997674+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:34.179433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:34.179433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"88","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.018064+0000","last_change":"2021-03-15T22:20:26.830247+0000","last_active":"2021-03-15T22:20:40.018064+0000","last_peered":"2021-03-15T22:20:40.018064+0000","last_clean":"2021-03-15T22:20:40.018064+0000","last_became_active":"2021-03-15T22:17:53.970585+0000","last_became_peered":"2021-03-15T22:17:53.970585+0000","last_unstale":"2021-03-15T22:20:40.018064+0000","last_undegraded":"2021-03-15T22:20:40.018064+0000","last_fullsized":"2021-03-15T22:20:40.018064+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:26.830183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:26.830183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"133","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.016444+0000","last_change":"2021-03-15T22:19:04.746329+0000","last_active":"2021-03-15T22:20:40.016444+0000","last_peered":"2021-03-15T22:20:40.016444+0000","last_clean":"2021-03-15T22:20:40.016444+0000","last_became_active":"2021-03-15T22:17:56.288319+0000","last_became_peered":"2021-03-15T22:17:56.288319+0000","last_unstale":"2021-03-15T22:20:40.016444+0000","last_undegraded":"2021-03-15T22:20:40.016444+0000","last_fullsized":"2021-03-15T22:20:40.016444+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:19:04.746224+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"79","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.997856+0000","last_change":"2021-03-15T22:19:50.171242+0000","last_active":"2021-03-15T22:20:39.997856+0000","last_peered":"2021-03-15T22:20:39.997856+0000","last_clean":"2021-03-15T22:20:39.997856+0000","last_became_active":"2021-03-15T22:17:58.006706+0000","last_became_peered":"2021-03-15T22:17:58.006706+0000","last_unstale":"2021-03-15T22:20:39.997856+0000","last_undegraded":"2021-03-15T22:20:39.997856+0000","last_fullsized":"2021-03-15T22:20:39.997856+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:50.171159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:50.171159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"129'22","reported_seq":"130","reported_epoch":"130","state":"active+clean","last_fresh":"2021-03-15T22:20:45.330440+0000","last_change":"2021-03-15T22:20:39.523437+0000","last_active":"2021-03-15T22:20:45.330440+0000","last_peered":"2021-03-15T22:20:45.330440+0000","last_clean":"2021-03-15T22:20:45.330440+0000","last_became_active":"2021-03-15T22:20:05.035665+0000","last_became_peered":"2021-03-15T22:20:05.035665+0000","last_unstale":"2021-03-15T22:20:45.330440+0000","last_undegraded":"2021-03-15T22:20:45.330440+0000","last_fullsized":"2021-03-15T22:20:45.330440+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11142909,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":23,"num_read_kb":10328,"num_write":20,"num_write_kb":7953,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"77","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.015671+0000","last_change":"2021-03-15T22:19:42.258910+0000","last_active":"2021-03-15T22:20:41.015671+0000","last_peered":"2021-03-15T22:20:41.015671+0000","last_clean":"2021-03-15T22:20:41.015671+0000","last_became_active":"2021-03-15T22:18:00.019457+0000","last_became_peered":"2021-03-15T22:18:00.019457+0000","last_unstale":"2021-03-15T22:20:41.015671+0000","last_undegraded":"2021-03-15T22:20:41.015671+0000","last_fullsized":"2021-03-15T22:20:41.015671+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:42.258829+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:19:42.258829+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"61'2","reported_seq":"113","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998839+0000","last_change":"2021-03-15T22:19:57.135105+0000","last_active":"2021-03-15T22:20:39.998839+0000","last_peered":"2021-03-15T22:20:39.998839+0000","last_clean":"2021-03-15T22:20:39.998839+0000","last_became_active":"2021-03-15T22:17:58.010485+0000","last_became_peered":"2021-03-15T22:17:58.010485+0000","last_unstale":"2021-03-15T22:20:39.998839+0000","last_undegraded":"2021-03-15T22:20:39.998839+0000","last_fullsized":"2021-03-15T22:20:39.998839+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:57.135028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:57.135028+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"83","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.016412+0000","last_change":"2021-03-15T22:19:24.902865+0000","last_active":"2021-03-15T22:20:40.016412+0000","last_peered":"2021-03-15T22:20:40.016412+0000","last_clean":"2021-03-15T22:20:40.016412+0000","last_became_active":"2021-03-15T22:17:53.967568+0000","last_became_peered":"2021-03-15T22:17:53.967568+0000","last_unstale":"2021-03-15T22:20:40.016412+0000","last_undegraded":"2021-03-15T22:20:40.016412+0000","last_fullsized":"2021-03-15T22:20:40.016412+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:24.902804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:24.902804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"87","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.008160+0000","last_change":"2021-03-15T22:20:24.847308+0000","last_active":"2021-03-15T22:20:43.008160+0000","last_peered":"2021-03-15T22:20:43.008160+0000","last_clean":"2021-03-15T22:20:43.008160+0000","last_became_active":"2021-03-15T22:19:02.092703+0000","last_became_peered":"2021-03-15T22:19:02.092703+0000","last_unstale":"2021-03-15T22:20:43.008160+0000","last_undegraded":"2021-03-15T22:20:43.008160+0000","last_fullsized":"2021-03-15T22:20:43.008160+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:20:24.847230+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:20:24.847230+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":422,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"97'3","reported_seq":"107","reported_epoch":"130","state":"active+clean","last_fresh":"2021-03-15T22:20:45.330591+0000","last_change":"2021-03-15T22:20:39.524152+0000","last_active":"2021-03-15T22:20:45.330591+0000","last_peered":"2021-03-15T22:20:45.330591+0000","last_clean":"2021-03-15T22:20:45.330591+0000","last_became_active":"2021-03-15T22:20:05.039641+0000","last_became_peered":"2021-03-15T22:20:05.039641+0000","last_unstale":"2021-03-15T22:20:45.330591+0000","last_undegraded":"2021-03-15T22:20:45.330591+0000","last_fullsized":"2021-03-15T22:20:45.330591+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2756315,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":12297,"num_write":4,"num_write_kb":1524,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.017060+0000","last_change":"2021-03-15T22:19:25.878071+0000","last_active":"2021-03-15T22:20:40.017060+0000","last_peered":"2021-03-15T22:20:40.017060+0000","last_clean":"2021-03-15T22:20:40.017060+0000","last_became_active":"2021-03-15T22:18:00.287019+0000","last_became_peered":"2021-03-15T22:18:00.287019+0000","last_unstale":"2021-03-15T22:20:40.017060+0000","last_undegraded":"2021-03-15T22:20:40.017060+0000","last_fullsized":"2021-03-15T22:20:40.017060+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:19:25.878008+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":"84","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.016544+0000","last_change":"2021-03-15T22:19:59.244041+0000","last_active":"2021-03-15T22:20:41.016544+0000","last_peered":"2021-03-15T22:20:41.016544+0000","last_clean":"2021-03-15T22:20:41.016544+0000","last_became_active":"2021-03-15T22:17:58.008735+0000","last_became_peered":"2021-03-15T22:17:58.008735+0000","last_unstale":"2021-03-15T22:20:41.016544+0000","last_undegraded":"2021-03-15T22:20:41.016544+0000","last_fullsized":"2021-03-15T22:20:41.016544+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:59.243939+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:59.243939+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"83","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.997992+0000","last_change":"2021-03-15T22:19:55.098323+0000","last_active":"2021-03-15T22:20:39.997992+0000","last_peered":"2021-03-15T22:20:39.997992+0000","last_clean":"2021-03-15T22:20:39.997992+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:20:39.997992+0000","last_undegraded":"2021-03-15T22:20:39.997992+0000","last_fullsized":"2021-03-15T22:20:39.997992+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:19:55.098239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"108","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998689+0000","last_change":"2021-03-15T22:20:01.157941+0000","last_active":"2021-03-15T22:20:39.998689+0000","last_peered":"2021-03-15T22:20:39.998689+0000","last_clean":"2021-03-15T22:20:39.998689+0000","last_became_active":"2021-03-15T22:17:55.992213+0000","last_became_peered":"2021-03-15T22:17:55.992213+0000","last_unstale":"2021-03-15T22:20:39.998689+0000","last_undegraded":"2021-03-15T22:20:39.998689+0000","last_fullsized":"2021-03-15T22:20:39.998689+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:20:01.157856+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:20:01.157856+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"122'10","reported_seq":"119","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.037995+0000","last_change":"2021-03-15T22:20:39.102520+0000","last_active":"2021-03-15T22:20:43.037995+0000","last_peered":"2021-03-15T22:20:43.037995+0000","last_clean":"2021-03-15T22:20:43.037995+0000","last_became_active":"2021-03-15T22:20:05.033636+0000","last_became_peered":"2021-03-15T22:20:05.033636+0000","last_unstale":"2021-03-15T22:20:43.037995+0000","last_undegraded":"2021-03-15T22:20:43.037995+0000","last_fullsized":"2021-03-15T22:20:43.037995+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4734413,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":30,"num_read_kb":32108,"num_write":9,"num_write_kb":2474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"11","length":"1"},{"start":"18","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"80","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.008569+0000","last_change":"2021-03-15T22:19:22.172668+0000","last_active":"2021-03-15T22:20:43.008569+0000","last_peered":"2021-03-15T22:20:43.008569+0000","last_clean":"2021-03-15T22:20:43.008569+0000","last_became_active":"2021-03-15T22:18:00.019205+0000","last_became_peered":"2021-03-15T22:18:00.019205+0000","last_unstale":"2021-03-15T22:20:43.008569+0000","last_undegraded":"2021-03-15T22:20:43.008569+0000","last_fullsized":"2021-03-15T22:20:43.008569+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_clean_scrub_stamp":"2021-03-15T22:19:22.172605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"84","reported_epoch":"126","state":"active+clean","last_fresh":"2021-03-15T22:20:41.010628+0000","last_change":"2021-03-15T22:20:31.826217+0000","last_active":"2021-03-15T22:20:41.010628+0000","last_peered":"2021-03-15T22:20:41.010628+0000","last_clean":"2021-03-15T22:20:41.010628+0000","last_became_active":"2021-03-15T22:17:58.013512+0000","last_became_peered":"2021-03-15T22:17:58.013512+0000","last_unstale":"2021-03-15T22:20:41.010628+0000","last_undegraded":"2021-03-15T22:20:41.010628+0000","last_fullsized":"2021-03-15T22:20:41.010628+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:31.826150+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:20:31.826150+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"86","reported_epoch":"129","state":"active+clean","last_fresh":"2021-03-15T22:20:44.025724+0000","last_change":"2021-03-15T22:19:18.591404+0000","last_active":"2021-03-15T22:20:44.025724+0000","last_peered":"2021-03-15T22:20:44.025724+0000","last_clean":"2021-03-15T22:20:44.025724+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:20:44.025724+0000","last_undegraded":"2021-03-15T22:20:44.025724+0000","last_fullsized":"2021-03-15T22:20:44.025724+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:19:18.591331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:19:18.591331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"115'33","reported_seq":"129","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998707+0000","last_change":"2021-03-15T22:20:30.227263+0000","last_active":"2021-03-15T22:20:39.998707+0000","last_peered":"2021-03-15T22:20:39.998707+0000","last_clean":"2021-03-15T22:20:39.998707+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:20:39.998707+0000","last_undegraded":"2021-03-15T22:20:39.998707+0000","last_fullsized":"2021-03-15T22:20:39.998707+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'33","last_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:20:30.227196+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"124'36","reported_seq":"144","reported_epoch":"130","state":"active+clean","last_fresh":"2021-03-15T22:20:44.997214+0000","last_change":"2021-03-15T22:20:39.096407+0000","last_active":"2021-03-15T22:20:44.997214+0000","last_peered":"2021-03-15T22:20:44.997214+0000","last_clean":"2021-03-15T22:20:44.997214+0000","last_became_active":"2021-03-15T22:20:05.037884+0000","last_became_peered":"2021-03-15T22:20:05.037884+0000","last_unstale":"2021-03-15T22:20:44.997214+0000","last_undegraded":"2021-03-15T22:20:44.997214+0000","last_fullsized":"2021-03-15T22:20:44.997214+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":36,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10235331,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":28,"num_read_kb":18204,"num_write":27,"num_write_kb":8109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"77","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998850+0000","last_change":"2021-03-15T22:20:35.270198+0000","last_active":"2021-03-15T22:20:39.998850+0000","last_peered":"2021-03-15T22:20:39.998850+0000","last_clean":"2021-03-15T22:20:39.998850+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:20:39.998850+0000","last_undegraded":"2021-03-15T22:20:39.998850+0000","last_fullsized":"2021-03-15T22:20:39.998850+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:35.270125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:35.270125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"61'2","reported_seq":"117","reported_epoch":"128","state":"active+clean","last_fresh":"2021-03-15T22:20:43.008158+0000","last_change":"2021-03-15T22:19:12.158924+0000","last_active":"2021-03-15T22:20:43.008158+0000","last_peered":"2021-03-15T22:20:43.008158+0000","last_clean":"2021-03-15T22:20:43.008158+0000","last_became_active":"2021-03-15T22:17:58.010539+0000","last_became_peered":"2021-03-15T22:17:58.010539+0000","last_unstale":"2021-03-15T22:20:43.008158+0000","last_undegraded":"2021-03-15T22:20:43.008158+0000","last_fullsized":"2021-03-15T22:20:43.008158+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:19:12.158857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:19:12.158857+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"83","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:40.015997+0000","last_change":"2021-03-15T22:18:57.650089+0000","last_active":"2021-03-15T22:20:40.015997+0000","last_peered":"2021-03-15T22:20:40.015997+0000","last_clean":"2021-03-15T22:20:40.015997+0000","last_became_active":"2021-03-15T22:17:53.976045+0000","last_became_peered":"2021-03-15T22:17:53.976045+0000","last_unstale":"2021-03-15T22:20:40.015997+0000","last_undegraded":"2021-03-15T22:20:40.015997+0000","last_fullsized":"2021-03-15T22:20:40.015997+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:18:57.650003+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:18:57.650003+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"102","reported_epoch":"125","state":"active+clean","last_fresh":"2021-03-15T22:20:39.998382+0000","last_change":"2021-03-15T22:19:48.154964+0000","last_active":"2021-03-15T22:20:39.998382+0000","last_peered":"2021-03-15T22:20:39.998382+0000","last_clean":"2021-03-15T22:20:39.998382+0000","last_became_active":"2021-03-15T22:17:55.986062+0000","last_became_peered":"2021-03-15T22:17:55.986062+0000","last_unstale":"2021-03-15T22:20:39.998382+0000","last_undegraded":"2021-03-15T22:20:39.998382+0000","last_fullsized":"2021-03-15T22:20:39.998382+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:19:48.154884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:19:48.154884+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":161974435,"num_objects":67,"num_object_clones":22,"num_object_copies":201,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":67,"num_whiteouts":4,"num_read":418,"num_read_kb":261842,"num_write":434,"num_write_kb":161233,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":337047552,"data_stored":365362654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":579,"ondisk_log_size":579,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":170,"num_read_kb":165,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1069,"num_read_kb":574,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":579,"ondisk_log_size":579,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17,"ondisk_log_size":17,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299430,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":60116,"kb_used_data":49292,"kb_used_omap":0,"kb_used_meta":10816,"kb_avail":93685036,"statfs":{"total":95995035648,"available":95933476864,"internally_reserved":0,"allocated":50475008,"data_stored":50503016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":11075584},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.33400000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.63600000000000001,"5min":0.70999999999999996,"15min":0.70999999999999996},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.5},"min":{"1min":0.35499999999999998,"5min":0.309,"15min":0.309},"max":{"1min":0.61799999999999999,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.33800000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.47399999999999998,"15min":0.47799999999999998},"min":{"1min":0.28999999999999998,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.56999999999999995,"5min":0.60899999999999999,"15min":0.60899999999999999},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48999999999999999,"15min":0.49399999999999999},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.624,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.47399999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.47299999999999998,"5min":0.48099999999999998,"15min":0.48299999999999998},"min":{"1min":0.38800000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.56999999999999995,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.49099999999999999,"15min":0.48899999999999999},"min":{"1min":0.39300000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.64900000000000002,"5min":0.746,"15min":0.746},"last":0.498}]},{"osd":3,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.56299999999999994,"15min":0.57299999999999995},"min":{"1min":0.36699999999999999,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.60899999999999999,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.55900000000000005,"15min":0.56599999999999995},"min":{"1min":0.39200000000000002,"5min":0.307,"15min":0.307},"max":{"1min":0.63900000000000001,"5min":2.1160000000000001,"15min":2.1160000000000001},"last":0.55200000000000005}]},{"osd":4,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.45800000000000002,"5min":0.42499999999999999,"15min":0.41899999999999998},"min":{"1min":0.26100000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.80100000000000005,"5min":0.80100000000000005,"15min":0.80100000000000005},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.44,"15min":0.42799999999999999},"min":{"1min":0.307,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.60499999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.433,"15min":0.42299999999999999},"min":{"1min":0.35599999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.49399999999999999,"15min":0.48999999999999999},"min":{"1min":0.317,"5min":0.317,"15min":0.317},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.57999999999999996}]},{"osd":6,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.55300000000000005,"5min":0.499,"15min":0.48999999999999999},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.82099999999999995,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.61499999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.39300000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.754,"5min":0.754,"15min":0.754},"last":0.56499999999999995}]}]},{"osd":6,"up_from":45,"seq":193273528362,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":24868,"kb_used_data":18460,"kb_used_omap":0,"kb_used_meta":6400,"kb_avail":93720284,"statfs":{"total":95995035648,"available":95969570816,"internally_reserved":0,"allocated":18903040,"data_stored":20701161,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6553600},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:20:36 2021","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.309,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.73199999999999998,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.499,"15min":0.495},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.56699999999999995}]},{"osd":1,"last update":"Mon Mar 15 22:20:36 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.51900000000000002,"15min":0.51700000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.81599999999999995,"5min":0.81599999999999995,"15min":0.81599999999999995},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48899999999999999,"15min":0.48099999999999998},"min":{"1min":0.35899999999999999,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.437}]},{"osd":2,"last update":"Mon Mar 15 22:20:36 2021","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.51400000000000001,"15min":0.501},"min":{"1min":0.377,"5min":0.312,"15min":0.312},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.504,"15min":0.503},"min":{"1min":0.32200000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.39600000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:20:36 2021","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52900000000000003,"15min":0.53300000000000003},"min":{"1min":0.40200000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.76700000000000002,"5min":0.76700000000000002,"15min":0.76700000000000002},"last":0.376},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53400000000000003,"15min":0.53000000000000003},"min":{"1min":0.40799999999999997,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.77800000000000002,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.51600000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:20:36 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.41599999999999998,"15min":0.39800000000000002},"min":{"1min":0.29099999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.45200000000000001,"15min":0.433},"min":{"1min":0.314,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.54200000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:20:36 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.45600000000000002,"15min":0.438},"min":{"1min":0.32900000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.64800000000000002,"5min":0.76400000000000001,"15min":0.76400000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.48299999999999998,"15min":0.46500000000000002},"min":{"1min":0.34899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.77000000000000002,"5min":0.77300000000000002,"15min":0.77300000000000002},"last":0.60299999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:19:53 2021","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.502,"15min":0.496},"min":{"1min":0.34000000000000002,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.79900000000000004,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53600000000000003,"15min":0.53100000000000003},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.62}]}]},{"osd":1,"up_from":13,"seq":55834574907,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":46872,"kb_used_data":35600,"kb_used_omap":0,"kb_used_meta":11264,"kb_avail":93698280,"statfs":{"total":95995035648,"available":95947038720,"internally_reserved":0,"allocated":36454400,"data_stored":37392473,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":11534336},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:20:18 2021","interfaces":[{"interface":"back","average":{"1min":2.915,"5min":0.90000000000000002,"15min":0.52300000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":54.427999999999997,"5min":54.427999999999997,"15min":54.427999999999997},"last":0.25800000000000001},{"interface":"front","average":{"1min":2.903,"5min":0.91100000000000003,"15min":0.54700000000000004},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":54.356999999999999,"5min":54.356999999999999,"15min":54.356999999999999},"last":0.33200000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:20:37 2021","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.43099999999999999,"15min":0.41999999999999998},"min":{"1min":0.27200000000000002,"5min":0.223,"15min":0.223},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.40100000000000002},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.39800000000000002,"15min":0.41399999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":0.47899999999999998,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.29599999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:19:44 2021","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.434,"15min":0.438},"min":{"1min":0.27100000000000002,"5min":0.25600000000000001,"15min":0.25600000000000001},"max":{"1min":0.54100000000000004,"5min":1.796,"15min":1.796},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.48299999999999998,"15min":0.495},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.627,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.30299999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:20:01 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.57299999999999995,"15min":0.58099999999999996},"min":{"1min":0.38500000000000001,"5min":0.33600000000000002,"15min":0.33600000000000002},"max":{"1min":0.74199999999999999,"5min":1.2789999999999999,"15min":1.2789999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.63100000000000001,"15min":0.628},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.71599999999999997,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.59099999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:20:18 2021","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.627,"15min":0.66100000000000003},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.83199999999999996,"5min":2.153,"15min":2.153},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.53700000000000003,"5min":1,"15min":1.194},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70099999999999996,"5min":15.472,"15min":15.472},"last":0.54200000000000004}]},{"osd":6,"last update":"Mon Mar 15 22:20:37 2021","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.65300000000000002,"15min":0.68600000000000005},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.77500000000000002,"5min":1.75,"15min":1.75},"last":0.623},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.63200000000000001,"15min":0.65000000000000002},"min":{"1min":0.36399999999999999,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.91300000000000003,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.53300000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:19:53 2021","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.94299999999999995,"15min":1.0049999999999999},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.78200000000000003,"5min":8.8100000000000005,"15min":8.8100000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.629,"15min":0.64300000000000002},"min":{"1min":0.38700000000000001,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.72999999999999998,"5min":1.325,"15min":1.325},"last":0.51400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738430,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":71472,"kb_used_data":60072,"kb_used_omap":0,"kb_used_meta":11392,"kb_avail":93673680,"statfs":{"total":95995035648,"available":95921848320,"internally_reserved":0,"allocated":61513728,"data_stored":66092182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":11665408},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:20:17 2021","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.36599999999999999,"15min":0.36599999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.57299999999999995,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.38400000000000001,"15min":0.38},"min":{"1min":0.23499999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.58099999999999996,"5min":0.63800000000000001,"15min":0.63800000000000001},"last":0.29199999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:20:30 2021","interfaces":[{"interface":"back","average":{"1min":0.38300000000000001,"5min":0.39000000000000001,"15min":0.40799999999999997},"min":{"1min":0.248,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.64300000000000002,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.379,"15min":0.39900000000000002},"min":{"1min":0.22500000000000001,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.314}]},{"osd":3,"last update":"Mon Mar 15 22:19:50 2021","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.42199999999999999,"15min":0.438},"min":{"1min":0.24399999999999999,"5min":0.24199999999999999,"15min":0.24199999999999999},"max":{"1min":0.69699999999999995,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.27000000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.433,"15min":0.44700000000000001},"min":{"1min":0.30399999999999999,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.39300000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:20:05 2021","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.56100000000000005,"15min":0.57099999999999995},"min":{"1min":0.309,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.96799999999999997,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.60999999999999999,"15min":0.61699999999999999},"min":{"1min":0.45500000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"max":{"1min":1.034,"5min":1.034,"15min":1.034},"last":0.73799999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:20:20 2021","interfaces":[{"interface":"back","average":{"1min":0.59899999999999998,"5min":0.60899999999999999,"15min":0.61899999999999999},"min":{"1min":0.443,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.83199999999999996,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.57599999999999996,"15min":0.58799999999999997},"min":{"1min":0.376,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.0149999999999999,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.60699999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:20:33 2021","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.37,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.82499999999999996,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.629},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.627,"15min":0.63},"min":{"1min":0.434,"5min":0.433,"15min":0.433},"max":{"1min":0.80800000000000005,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.70199999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:19:57 2021","interfaces":[{"interface":"back","average":{"1min":0.63700000000000001,"5min":0.58799999999999997,"15min":0.57999999999999996},"min":{"1min":0.44600000000000001,"5min":0.373,"15min":0.373},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.625,"5min":0.58299999999999996,"15min":0.57599999999999996},"min":{"1min":0.39600000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"max":{"1min":0.99399999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67900000000000005}]}]},{"osd":2,"up_from":18,"seq":77309411384,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":43936,"kb_used_data":37144,"kb_used_omap":0,"kb_used_meta":6784,"kb_avail":93701216,"statfs":{"total":95995035648,"available":95950045184,"internally_reserved":0,"allocated":38035456,"data_stored":39671717,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6946816},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:20:36 2021","interfaces":[{"interface":"back","average":{"1min":0.29599999999999999,"5min":0.34000000000000002,"15min":0.35599999999999998},"min":{"1min":0.223,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.47199999999999998,"5min":0.68000000000000005,"15min":0.68000000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.30599999999999999,"5min":0.36099999999999999,"15min":0.377},"min":{"1min":0.23999999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.51500000000000001,"5min":0.64300000000000002,"15min":0.64300000000000002},"last":1.2889999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:20:36 2021","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.371,"15min":0.38300000000000001},"min":{"1min":0.26800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.40999999999999998,"5min":0.61499999999999999,"15min":0.61499999999999999},"last":0.89900000000000002},{"interface":"front","average":{"1min":0.378,"5min":0.42099999999999999,"15min":0.44600000000000001},"min":{"1min":0.26800000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.57899999999999996,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":1.052}]},{"osd":3,"last update":"Mon Mar 15 22:19:45 2021","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.41299999999999998,"15min":0.443},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.54300000000000004,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.86199999999999999},{"interface":"front","average":{"1min":0.35399999999999998,"5min":0.47499999999999998,"15min":0.51300000000000001},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.59099999999999997,"5min":0.874,"15min":0.874},"last":0.92600000000000005}]},{"osd":4,"last update":"Mon Mar 15 22:19:59 2021","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.56799999999999995,"15min":0.58199999999999996},"min":{"1min":0.41199999999999998,"5min":0.40799999999999997,"15min":0.40799999999999997},"max":{"1min":0.86499999999999999,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.97599999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.55000000000000004,"15min":0.56499999999999995},"min":{"1min":0.39400000000000002,"5min":0.38400000000000001,"15min":0.38400000000000001},"max":{"1min":0.82299999999999995,"5min":0.94599999999999995,"15min":0.94599999999999995},"last":0.67400000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:20:22 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.56200000000000006,"15min":0.57899999999999996},"min":{"1min":0.42999999999999999,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.69699999999999995,"5min":0.98499999999999999,"15min":0.98499999999999999},"last":0.79500000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.54800000000000004,"15min":0.56000000000000005},"min":{"1min":0.39100000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.77300000000000002,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":1.3280000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:20:36 2021","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.56799999999999995,"15min":0.56200000000000006},"min":{"1min":0.376,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71799999999999997,"5min":0.80500000000000005,"15min":0.80500000000000005},"last":0.71999999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56399999999999995,"15min":0.56599999999999995},"min":{"1min":0.46000000000000002,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.74099999999999999,"5min":0.90400000000000003,"15min":0.90400000000000003},"last":1.002}]},{"osd":7,"last update":"Mon Mar 15 22:19:54 2021","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54500000000000004,"15min":0.54700000000000004},"min":{"1min":0.42399999999999999,"5min":0.41399999999999998,"15min":0.41399999999999998},"max":{"1min":0.69299999999999995,"5min":0.873,"15min":0.873},"last":1.026},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57599999999999996,"15min":0.57899999999999996},"min":{"1min":0.436,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":0.73399999999999999,"5min":0.79100000000000004,"15min":0.79100000000000004},"last":0.82699999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149747,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":80840,"kb_used_data":69248,"kb_used_omap":0,"kb_used_meta":11584,"kb_avail":93664312,"statfs":{"total":95995035648,"available":95912255488,"internally_reserved":0,"allocated":70909952,"data_stored":74527258,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":11862016},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:19:52 2021","interfaces":[{"interface":"back","average":{"1min":0.30199999999999999,"5min":0.34699999999999998,"15min":0.36599999999999999},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.57699999999999996,"5min":0.57699999999999996,"15min":0.57699999999999996},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.371,"15min":0.39200000000000002},"min":{"1min":0.189,"5min":0.189,"15min":0.189},"max":{"1min":0.49399999999999999,"5min":0.54000000000000004,"15min":0.54000000000000004},"last":0.32400000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:19:52 2021","interfaces":[{"interface":"back","average":{"1min":0.371,"5min":0.41099999999999998,"15min":0.43099999999999999},"min":{"1min":0.24399999999999999,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.79400000000000004,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.40600000000000003,"15min":0.43099999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.61899999999999999,"5min":0.63500000000000001,"15min":0.63500000000000001},"last":0.33900000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:19:52 2021","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.40799999999999997,"15min":0.434},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.505,"5min":0.58999999999999997,"15min":0.58999999999999997},"last":0.372},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.38100000000000001,"15min":0.39900000000000002},"min":{"1min":0.23799999999999999,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.435,"5min":0.59399999999999997,"15min":0.59399999999999997},"last":0.38100000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:20:02 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51900000000000002,"15min":0.505},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.90600000000000003,"5min":0.90600000000000003,"15min":0.90600000000000003},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.53400000000000003,"15min":0.51800000000000002},"min":{"1min":0.35799999999999998,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.88600000000000001,"5min":0.88600000000000001,"15min":0.88600000000000001},"last":0.51000000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:20:19 2021","interfaces":[{"interface":"back","average":{"1min":0.61499999999999999,"5min":0.53800000000000003,"15min":0.52800000000000002},"min":{"1min":0.35699999999999998,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.56399999999999995,"15min":0.56200000000000006},"min":{"1min":0.38600000000000001,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.85899999999999999,"5min":0.92100000000000004,"15min":0.92100000000000004},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Mar 15 22:20:34 2021","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.60099999999999998,"15min":0.59799999999999998},"min":{"1min":0.42099999999999999,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.89100000000000001,"5min":0.89100000000000001,"15min":0.89100000000000001},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58099999999999996,"15min":0.58899999999999997},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.84899999999999998,"5min":0.88,"15min":0.88},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:19:52 2021","interfaces":[{"interface":"back","average":{"1min":0.56299999999999994,"5min":0.57299999999999995,"15min":0.57499999999999996},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.84099999999999997,"5min":0.89800000000000002,"15min":0.89800000000000002},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56999999999999995},"min":{"1min":0.38900000000000001,"5min":0.377,"15min":0.377},"max":{"1min":0.76300000000000001,"5min":0.82099999999999995,"15min":0.82099999999999995},"last":0.49199999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986224,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36860,"kb_used_data":26100,"kb_used_omap":0,"kb_used_meta":10752,"kb_avail":93708292,"statfs":{"total":95995035648,"available":95957291008,"internally_reserved":0,"allocated":26726400,"data_stored":26419156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":11010048},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:20:05 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.505,"15min":0.52100000000000002},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.70999999999999996,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.251,"5min":0.251,"15min":0.251},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:20:05 2021","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.23599999999999999,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.67300000000000004,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.53000000000000003},"min":{"1min":0.26800000000000002,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.39700000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:20:05 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.51100000000000001,"15min":0.52500000000000002},"min":{"1min":0.313,"5min":0.313,"15min":0.313},"max":{"1min":0.76100000000000001,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.51100000000000001,"15min":0.51300000000000001},"min":{"1min":0.37,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.1910000000000001,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.56000000000000005}]},{"osd":3,"last update":"Mon Mar 15 22:20:05 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.55400000000000005,"15min":0.56599999999999995},"min":{"1min":0.40100000000000002,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.92000000000000004,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56399999999999995,"15min":0.58399999999999996},"min":{"1min":0.36599999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.90300000000000002,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:20:20 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.41399999999999998,"15min":0.39500000000000002},"min":{"1min":0.34399999999999997,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.72299999999999998,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.42999999999999999,"15min":0.40999999999999998},"min":{"1min":0.34999999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.72799999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.59299999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:20:37 2021","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.435,"15min":0.41799999999999998},"min":{"1min":0.316,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68500000000000005,"5min":0.755,"15min":0.755},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.51600000000000001,"15min":0.497},"min":{"1min":0.42899999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78300000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.66200000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:19:49 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.47699999999999998,"15min":0.47299999999999998},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.80600000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.33700000000000002,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.74199999999999999,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.64300000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757293,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":34956,"kb_used_data":28356,"kb_used_omap":0,"kb_used_meta":6592,"kb_avail":93710196,"statfs":{"total":95995035648,"available":95959240704,"internally_reserved":0,"allocated":29036544,"data_stored":32108899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":6750208},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:20:22 2021","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.48299999999999998,"15min":0.48099999999999998},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.504,"15min":0.501},"min":{"1min":0.41799999999999998,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.73499999999999999,"5min":0.755,"15min":0.755},"last":0.498}]},{"osd":1,"last update":"Mon Mar 15 22:20:22 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.503,"15min":0.51200000000000001},"min":{"1min":0.313,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.80300000000000005,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51000000000000001,"15min":0.50800000000000001},"min":{"1min":0.39400000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.70599999999999996,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.42799999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:20:22 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.503,"15min":0.50600000000000001},"min":{"1min":0.34499999999999997,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.72599999999999998,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50700000000000001,"15min":0.50700000000000001},"min":{"1min":0.40300000000000002,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Mar 15 22:20:22 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.502,"15min":0.501},"min":{"1min":0.40999999999999998,"5min":0.308,"15min":0.308},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.82599999999999996,"5min":0.82599999999999996,"15min":0.82599999999999996},"last":0.57099999999999995}]},{"osd":4,"last update":"Mon Mar 15 22:20:22 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.39800000000000002,"15min":0.372},"min":{"1min":0.34399999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.68999999999999995,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.438,"15min":0.40699999999999997},"min":{"1min":0.38300000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.72999999999999998,"5min":0.79400000000000004,"15min":0.79400000000000004},"last":0.56000000000000005}]},{"osd":6,"last update":"Mon Mar 15 22:20:39 2021","interfaces":[{"interface":"back","average":{"1min":0.56699999999999995,"5min":0.47499999999999998,"15min":0.45700000000000002},"min":{"1min":0.40000000000000002,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.71099999999999997,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.61699999999999999,"5min":0.53400000000000003,"15min":0.51800000000000002},"min":{"1min":0.52100000000000002,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.79800000000000004,"5min":0.79800000000000004,"15min":0.79800000000000004},"last":0.40799999999999997}]},{"osd":7,"last update":"Mon Mar 15 22:19:53 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.501,"15min":0.496},"min":{"1min":0.376,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.83199999999999996,"5min":0.83199999999999996,"15min":0.83199999999999996},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.51700000000000002,"15min":0.51200000000000001},"min":{"1min":0.35099999999999998,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.72499999999999998,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.57999999999999996}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":12926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":159744,"data_stored":5684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":163840,"data_stored":5792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":60084224,"data_stored":65375360,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":34926592,"data_stored":36695962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":45121536,"data_stored":50568108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":69443584,"data_stored":73848650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":25288704,"data_stored":25733750,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":27717632,"data_stored":31420377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":17444864,"data_stored":19984552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":57020416,"data_stored":61735895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:20:47.488 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2021-03-15T22:20:47.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool set default.rgw.control pg_num 22 2021-03-15T22:20:47.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:47 smithi125 conmon[56739]: cluster 2021-03-15T22:20:47.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:47 smithi125 conmon[56739]: 2021-03-15T22:20:46.033269+0000 2021-03-15T22:20:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:47 smithi125 conmon[56739]: mon.a (mon.0) 672 : cluster [DBG] osdmap e131: 8 total, 8 up, 8 in 2021-03-15T22:20:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:47 smithi125 conmon[56739]: audit 2021-03-15T22:20:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:47 smithi125 conmon[56739]: 2021-03-15T22:20:46.651639+0000 mgr.y (mgr.14144 2021-03-15T22:20:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:47 smithi125 conmon[56739]: ) 397 : audit [DBG] from='client.14787 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:20:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:47 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:20:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:47 smithi125 conmon[56739]: :20:46.985613+0000 mgr.y (mgr.14144) 2021-03-15T22:20:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:47 smithi125 conmon[56739]: 398 : cluster [DBG] pgmap v324: 148 pgs: 148 active+clean; 162 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 9.8 MiB/s rd, 5.8 MiB/s wr, 19 op/s 2021-03-15T22:20:47.573 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:47.595 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:47.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:47.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:47.697 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:47.723 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:47.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:47.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:47.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:47.851 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:47.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:47.852 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:47.953 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:47.976 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:47.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:47.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:48.053 INFO:tasks.rados.rados.0.smithi076.stdout:214: done (1 left) 2021-03-15T22:20:48.053 INFO:tasks.rados.rados.0.smithi076.stdout:215: done (0 left) 2021-03-15T22:20:48.053 INFO:tasks.rados.rados.0.smithi076.stdout:216: write oid 24 current snap is 22 2021-03-15T22:20:48.054 INFO:tasks.rados.rados.0.smithi076.stdout:216: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2021-03-15T22:20:48.059 INFO:tasks.rados.rados.0.smithi076.stdout:216: writing smithi07694878-24 from 581542 to 1270525 tid 1 2021-03-15T22:20:48.065 INFO:tasks.rados.rados.0.smithi076.stdout:216: writing smithi07694878-24 from 1886503 to 2299195 tid 2 2021-03-15T22:20:48.072 INFO:tasks.rados.rados.0.smithi076.stdout:216: writing smithi07694878-24 from 2761543 to 3432494 tid 3 2021-03-15T22:20:48.072 INFO:tasks.rados.rados.0.smithi076.stdout:217: read oid 34 snap -1 2021-03-15T22:20:48.072 INFO:tasks.rados.rados.0.smithi076.stdout:217: expect deleted 2021-03-15T22:20:48.073 INFO:tasks.rados.rados.0.smithi076.stdout:218: rollback oid 36 current snap is 22 2021-03-15T22:20:48.073 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 36 to 11 2021-03-15T22:20:48.073 INFO:tasks.rados.rados.0.smithi076.stdout:216: finishing write tid 1 to smithi07694878-24 2021-03-15T22:20:48.073 INFO:tasks.rados.rados.0.smithi076.stdout:216: finishing write tid 2 to smithi07694878-24 2021-03-15T22:20:48.073 INFO:tasks.rados.rados.0.smithi076.stdout:219: snap_create 2021-03-15T22:20:48.075 INFO:tasks.rados.rados.0.smithi076.stdout:218: finishing rollback tid 0 to smithi07694878-36 2021-03-15T22:20:48.075 INFO:tasks.rados.rados.0.smithi076.stdout:216: finishing write tid 3 to smithi07694878-24 2021-03-15T22:20:48.076 INFO:tasks.rados.rados.0.smithi076.stdout:218: finishing rollback tid 1 to smithi07694878-36 2021-03-15T22:20:48.076 INFO:tasks.rados.rados.0.smithi076.stdout:216: finishing write tid 4 to smithi07694878-24 2021-03-15T22:20:48.076 INFO:tasks.rados.rados.0.smithi076.stdout:216: finishing write tid 5 to smithi07694878-24 2021-03-15T22:20:48.076 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 15 (ObjNum 106 snap 22 seq_num 106) dirty exists 2021-03-15T22:20:48.077 INFO:tasks.rados.rados.0.smithi076.stdout:216: left oid 24 (ObjNum 106 snap 22 seq_num 106) 2021-03-15T22:20:48.077 INFO:tasks.rados.rados.0.smithi076.stdout:218: finishing rollback tid 2 to smithi07694878-36 2021-03-15T22:20:48.077 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 45 (ObjNum 35 snap 0 seq_num 35) dirty exists 2021-03-15T22:20:48.079 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:48.100 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:48.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:48.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:48.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:48.206 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:48 smithi076 conmon[56514]: cluster 2021-03-15T22:20:47.044263+0000 mon.a (mon 2021-03-15T22:20:48.206 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:48 smithi076 conmon[56514]: .0) 673 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2021-03-15T22:20:48.206 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:48 smithi076 conmon[56514]: audit 2021-03-15T22:20:47.241671+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? 172.21.15.76:0/1919424412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:48.207 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:48 smithi076 conmon[62156]: cluster 2021-03-15T22:20:47.044263+0000 mon.a (mon. 2021-03-15T22:20:48.207 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:48 smithi076 conmon[62156]: 0) 673 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2021-03-15T22:20:48.207 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:48 smithi076 conmon[62156]: audit 2021-03-15T22:20:47.241671+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? 172.21.15.76:0/1919424412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:48.229 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:48.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:48.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:48.332 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:48.354 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:48.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:48.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:48.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:48.479 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:48.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:48.480 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:48.530 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:48.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:48 smithi125 conmon[56739]: cluster 2021-03-15T22:20:47.044263+0000 mon.a (mon.0) 673 : cluster [DBG] osdmap e132: 8 total, 8 up, 8 in 2021-03-15T22:20:48.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:48 smithi125 conmon[56739]: audit 2021-03-15T22:20 2021-03-15T22:20:48.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:48 smithi125 conmon[56739]: :47.241671+0000 mon.a (mon.0) 674 : audit [DBG] from='client.? 172.21.15.76:0/1919424412' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:48.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:48.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:48.604 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:48.605 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:48.706 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:48.732 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:48.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:48.734 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:48.835 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:48.856 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:48.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:48.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:48.959 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:48.980 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:48.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:48.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:49.059 INFO:tasks.rados.rados.0.smithi076.stdout:216: done (3 left) 2021-03-15T22:20:49.060 INFO:tasks.rados.rados.0.smithi076.stdout:217: done (2 left) 2021-03-15T22:20:49.060 INFO:tasks.rados.rados.0.smithi076.stdout:218: done (1 left) 2021-03-15T22:20:49.060 INFO:tasks.rados.rados.0.smithi076.stdout:219: done (0 left) 2021-03-15T22:20:49.060 INFO:tasks.rados.rados.0.smithi076.stdout:220: snap_remove snap 21 2021-03-15T22:20:49.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:49.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:49.108 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:49.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:49.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:49.235 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:49.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:49.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:49.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:49 smithi076 conmon[56514]: cluster 2021-03-15T22:20:48.050523+0000 mon.a (mon 2021-03-15T22:20:49.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:49 smithi076 conmon[56514]: .0) 675 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2021-03-15T22:20:49.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:49 smithi076 conmon[56514]: cluster 2021-03-15T22:20:48.985975+0000 mgr.y (mgr.14144) 399 : cluster [DBG] pgmap v327: 148 pgs: 148 active+clean; 162 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 18 op/s 2021-03-15T22:20:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:49 smithi076 conmon[62156]: cluster 2021-03-15T22:20:48.050523+0000 2021-03-15T22:20:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:49 smithi076 conmon[62156]: mon.a (mon.0) 675 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2021-03-15T22:20:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:49 smithi076 conmon[62156]: cluster 2021-03-15T22:20:48.985975+0000 mgr.y (mgr.14144) 399 : cluster [DBG] 2021-03-15T22:20:49.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:49 smithi076 conmon[62156]: pgmap v327: 148 pgs: 148 active+clean; 162 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 18 op/s 2021-03-15T22:20:49.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:49.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:49.365 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:49.366 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:49.448 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:49 smithi125 conmon[56739]: cluster 2021-03-15T22:20:48.050523+0000 mon.a (mon.0) 675 : cluster [DBG] osdmap e133: 8 total, 8 up, 8 in 2021-03-15T22:20:49.449 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:49 smithi125 conmon[56739]: cluster 2021-03-15T22:20:48.985975+0000 mgr.y (mgr.14144) 399 : cluster [DBG] pgmap v327: 148 pgs: 148 active+clean; 162 MiB data, 427 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 5.3 MiB/s wr, 18 op/s 2021-03-15T22:20:49.467 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:49.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:49.491 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:49.492 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:49.593 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:49.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:49.617 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:49.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:49.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:49.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:49.747 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:49.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:49.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:20:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:20:49.848 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:49.871 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:49.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:49.974 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:50.000 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:50.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:50.002 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:50.008 INFO:tasks.rados.rados.0.smithi076.stdout:220: done (0 left) 2021-03-15T22:20:50.009 INFO:tasks.rados.rados.0.smithi076.stdout:221: write oid 20 current snap is 23 2021-03-15T22:20:50.009 INFO:tasks.rados.rados.0.smithi076.stdout:221: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2021-03-15T22:20:50.013 INFO:tasks.rados.rados.0.smithi076.stdout:221: writing smithi07694878-20 from 598349 to 1078934 tid 1 2021-03-15T22:20:50.021 INFO:tasks.rados.rados.0.smithi076.stdout:221: writing smithi07694878-20 from 1544985 to 2217688 tid 2 2021-03-15T22:20:50.028 INFO:tasks.rados.rados.0.smithi076.stdout:221: writing smithi07694878-20 from 2905319 to 3387542 tid 3 2021-03-15T22:20:50.028 INFO:tasks.rados.rados.0.smithi076.stdout:222: write oid 32 current snap is 23 2021-03-15T22:20:50.028 INFO:tasks.rados.rados.0.smithi076.stdout:221: finishing write tid 1 to smithi07694878-20 2021-03-15T22:20:50.029 INFO:tasks.rados.rados.0.smithi076.stdout:222: seq_num 108 ranges {615156=555834,1603467=416361,2816389=693495} 2021-03-15T22:20:50.034 INFO:tasks.rados.rados.0.smithi076.stdout:222: writing smithi07694878-32 from 615156 to 1170990 tid 1 2021-03-15T22:20:50.039 INFO:tasks.rados.rados.0.smithi076.stdout:222: writing smithi07694878-32 from 1603467 to 2019828 tid 2 2021-03-15T22:20:50.048 INFO:tasks.rados.rados.0.smithi076.stdout:222: writing smithi07694878-32 from 2816389 to 3509884 tid 3 2021-03-15T22:20:50.048 INFO:tasks.rados.rados.0.smithi076.stdout:223: delete oid 17 current snap is 23 2021-03-15T22:20:50.048 INFO:tasks.rados.rados.0.smithi076.stdout:221: finishing write tid 2 to smithi07694878-20 2021-03-15T22:20:50.048 INFO:tasks.rados.rados.0.smithi076.stdout:221: finishing write tid 3 to smithi07694878-20 2021-03-15T22:20:50.049 INFO:tasks.rados.rados.0.smithi076.stdout:221: finishing write tid 4 to smithi07694878-20 2021-03-15T22:20:50.049 INFO:tasks.rados.rados.0.smithi076.stdout:221: finishing write tid 5 to smithi07694878-20 2021-03-15T22:20:50.049 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 50 (ObjNum 107 snap 23 seq_num 107) dirty exists 2021-03-15T22:20:50.049 INFO:tasks.rados.rados.0.smithi076.stdout:221: left oid 20 (ObjNum 107 snap 23 seq_num 107) 2021-03-15T22:20:50.049 INFO:tasks.rados.rados.0.smithi076.stdout:222: finishing write tid 1 to smithi07694878-32 2021-03-15T22:20:50.050 INFO:tasks.rados.rados.0.smithi076.stdout:222: finishing write tid 2 to smithi07694878-32 2021-03-15T22:20:50.050 INFO:tasks.rados.rados.0.smithi076.stdout:221: done (2 left) 2021-03-15T22:20:50.050 INFO:tasks.rados.rados.0.smithi076.stdout:223: done (1 left) 2021-03-15T22:20:50.051 INFO:tasks.rados.rados.0.smithi076.stdout:224: write (excl) oid 29 current snap is 23 2021-03-15T22:20:50.051 INFO:tasks.rados.rados.0.smithi076.stdout:224: seq_num 110 ranges {648770=706332,2120431=420030,3271235=432392} 2021-03-15T22:20:50.057 INFO:tasks.rados.rados.0.smithi076.stdout:224: writing smithi07694878-29 from 648770 to 1355102 tid 1 2021-03-15T22:20:50.064 INFO:tasks.rados.rados.0.smithi076.stdout:224: writing smithi07694878-29 from 2120431 to 2540461 tid 2 2021-03-15T22:20:50.070 INFO:tasks.rados.rados.0.smithi076.stdout:224: writing smithi07694878-29 from 3271235 to 3703627 tid 3 2021-03-15T22:20:50.071 INFO:tasks.rados.rados.0.smithi076.stdout:225: copy_from oid 34 from oid 7 current snap is 23 2021-03-15T22:20:50.071 INFO:tasks.rados.rados.0.smithi076.stdout:222: finishing write tid 3 to smithi07694878-32 2021-03-15T22:20:50.071 INFO:tasks.rados.rados.0.smithi076.stdout:222: finishing write tid 4 to smithi07694878-32 2021-03-15T22:20:50.072 INFO:tasks.rados.rados.0.smithi076.stdout:222: finishing write tid 5 to smithi07694878-32 2021-03-15T22:20:50.072 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 103 (ObjNum 108 snap 23 seq_num 108) dirty exists 2021-03-15T22:20:50.072 INFO:tasks.rados.rados.0.smithi076.stdout:222: left oid 32 (ObjNum 108 snap 23 seq_num 108) 2021-03-15T22:20:50.072 INFO:tasks.rados.rados.0.smithi076.stdout:224: finishing write tid 1 to smithi07694878-29 2021-03-15T22:20:50.073 INFO:tasks.rados.rados.0.smithi076.stdout:224: finishing write tid 2 to smithi07694878-29 2021-03-15T22:20:50.073 INFO:tasks.rados.rados.0.smithi076.stdout:222: done (2 left) 2021-03-15T22:20:50.073 INFO:tasks.rados.rados.0.smithi076.stdout:226: delete oid 48 current snap is 23 2021-03-15T22:20:50.074 INFO:tasks.rados.rados.0.smithi076.stdout:225: finishing copy_from to smithi07694878-34 2021-03-15T22:20:50.074 INFO:tasks.rados.rados.0.smithi076.stdout:225: got expected ENOENT (src dne) 2021-03-15T22:20:50.074 INFO:tasks.rados.rados.0.smithi076.stdout:225: finishing copy_from racing read to smithi07694878-34 2021-03-15T22:20:50.075 INFO:tasks.rados.rados.0.smithi076.stdout:225: done (2 left) 2021-03-15T22:20:50.075 INFO:tasks.rados.rados.0.smithi076.stdout:226: done (1 left) 2021-03-15T22:20:50.075 INFO:tasks.rados.rados.0.smithi076.stdout:227: snap_remove snap 13 2021-03-15T22:20:50.104 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:50.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:50.130 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:50.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:50.231 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:50.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:50.256 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:50.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:50.263 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:20:50.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:50 smithi076 conmon[56514]: cluster 2021-03-15T22:20:49.055343+0000 mon.a (mon.0) 676 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2021-03-15T22:20:50.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:50 smithi076 conmon[56514]: audit 2021-03-15T22:20:49.447356+0000 mgr.y (mgr.14144) 400 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:50.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:50 smithi076 conmon[56514]: cluster 2021-03-15T22:20:50.005988+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2021-03-15T22:20:50.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:50 smithi076 conmon[62156]: cluster 2021-03-15T22:20:49.055343+0000 mon.a (mon.0) 676 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2021-03-15T22:20:50.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:50 smithi076 conmon[62156]: audit 2021-03-15T22:20:49.447356+0000 mgr.y (mgr.14144) 400 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:50.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:50 smithi076 conmon[62156]: cluster 2021-03-15T22:20:50.005988+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2021-03-15T22:20:50.358 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:50.382 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:50.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:50.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:50.485 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:50.511 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:50.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:50.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:50 smithi125 conmon[56739]: cluster 2021-03-15T22:20:49.055343+0000 mon.a (mon.0) 676 : cluster [DBG] osdmap e134: 8 total, 8 up, 8 in 2021-03-15T22:20:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:50 smithi125 conmon[56739]: audit 2021-03-15T22:20:49. 2021-03-15T22:20:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:50 smithi125 conmon[56739]: 447356+0000 mgr.y (mgr.14144) 400 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:20:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:50 smithi125 conmon[56739]: cluster 2021-03-15T22:20:50.005988+0000 mon.a (mon.0) 677 : cluster [DBG] osdmap e135: 8 total, 8 up, 8 in 2021-03-15T22:20:50.614 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:50.643 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:50.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:50.745 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:50.771 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:50.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:50.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:50.873 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:50.896 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:50.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:50.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:50.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:51.022 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:51.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:51.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:51.072 INFO:tasks.rados.rados.0.smithi076.stdout:227: done (1 left) 2021-03-15T22:20:51.072 INFO:tasks.rados.rados.0.smithi076.stdout:228: rollback oid 26 current snap is 23 2021-03-15T22:20:51.072 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 22 2021-03-15T22:20:51.073 INFO:tasks.rados.rados.0.smithi076.stdout:224: finishing write tid 3 to smithi07694878-29 2021-03-15T22:20:51.073 INFO:tasks.rados.rados.0.smithi076.stdout:224: finishing write tid 4 to smithi07694878-29 2021-03-15T22:20:51.073 INFO:tasks.rados.rados.0.smithi076.stdout:224: finishing write tid 5 to smithi07694878-29 2021-03-15T22:20:51.073 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 108 (ObjNum 110 snap 23 seq_num 110) dirty exists 2021-03-15T22:20:51.074 INFO:tasks.rados.rados.0.smithi076.stdout:224: left oid 29 (ObjNum 110 snap 23 seq_num 110) 2021-03-15T22:20:51.074 INFO:tasks.rados.rados.0.smithi076.stdout:224: done (1 left) 2021-03-15T22:20:51.074 INFO:tasks.rados.rados.0.smithi076.stdout:229: snap_remove snap 20 2021-03-15T22:20:51.076 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:20:51.125 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:51.145 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:20:51.145 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:20:51.146 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:20:51.146 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:20:51.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:51.148 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:51.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:51.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:51.276 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:51.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:51.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:51.378 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:51.401 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:51.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:51.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:51.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[56514]: cluster 2021-03-15T22:20:50.252288+0000 osd.7 (osd.7) 32 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:51.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[56514]: cluster 2021-03-15T22:20:50.252439+0000 osd.7 (osd.7) 33 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:51.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[56514]: audit 2021-03-15T22:20:50.758114+0000 mon.a (mon.0) 678 : audit [INF] from='client.? 172.21.15.76:0/3338144072' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:20:51.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[56514]: cluster 2021-03-15T22:20:50.986326+0000 mgr.y (mgr.14144) 401 : cluster [DBG] pgmap v330: 148 pgs: 148 active+clean; 169 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2021-03-15T22:20:51.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[62156]: cluster 2021-03-15T22:20:50.252288+0000 2021-03-15T22:20:51.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[62156]: osd.7 (osd.7) 32 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:51.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[62156]: cluster 2021-03-15T22:20:50.252439+0000 2021-03-15T22:20:51.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[62156]: osd.7 (osd.7) 33 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:51.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[62156]: audit 2021-03-15T22:20:50.758114 2021-03-15T22:20:51.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[62156]: +0000 mon.a (mon.0) 678 : audit [INF] from='client.? 172.21.15.76:0/3338144072' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:20:51.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[62156]: cluster 2021-03-15T22:20:50.986326+0000 mgr.y (mgr.14144 2021-03-15T22:20:51.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:51 smithi076 conmon[62156]: ) 401 : cluster [DBG] pgmap v330: 148 pgs: 148 active+clean; 169 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2021-03-15T22:20:51.525 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:51.526 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:51.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:51 smithi125 conmon[56739]: cluster 2021-03-15T22:20:50.252288+0000 osd.7 (osd.7) 32 : 2021-03-15T22:20:51.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:51 smithi125 conmon[56739]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:51.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:51 smithi125 conmon[56739]: cluster 2021-03-15T22:20:50.252439+0000 osd.7 (osd.7) 33 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:51.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:51 smithi125 conmon[56739]: audit 2021-03-15T22:20:50. 2021-03-15T22:20:51.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:51 smithi125 conmon[56739]: 758114+0000 mon.a (mon.0) 678 : audit [INF] from='client.? 172.21.15.76:0/3338144072' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:20:51.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:51 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:20:51.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:51 smithi125 conmon[56739]: T22:20:50.986326+0000 mgr.y (mgr.14144) 401 : cluster [DBG] pgmap v330: 148 pgs: 148 active+clean; 169 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2021-03-15T22:20:51.627 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:51.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:51.655 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:51.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:51.757 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:51.779 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:51.781 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:51.882 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:51.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:51.907 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:51.907 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:52.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:52.031 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:52.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:52.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:52.068 INFO:tasks.rados.rados.0.smithi076.stdout:229: done (1 left) 2021-03-15T22:20:52.090 INFO:tasks.rados.rados.0.smithi076.stdout:230: read oid 31 snap -1 2021-03-15T22:20:52.091 INFO:tasks.rados.rados.0.smithi076.stdout:230: expect (ObjNum 73 snap 8 seq_num 73) 2021-03-15T22:20:52.091 INFO:tasks.rados.rados.0.smithi076.stdout:231: snap_create 2021-03-15T22:20:52.091 INFO:tasks.rados.rados.0.smithi076.stdout:228: finishing rollback tid 0 to smithi07694878-26 2021-03-15T22:20:52.091 INFO:tasks.rados.rados.0.smithi076.stdout:228: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:20:52.091 INFO:tasks.rados.rados.0.smithi076.stdout:228: finishing rollback tid 2 to smithi07694878-26 2021-03-15T22:20:52.092 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 36 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-03-15T22:20:52.092 INFO:tasks.rados.rados.0.smithi076.stdout:230: expect (ObjNum 73 snap 8 seq_num 73) 2021-03-15T22:20:52.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:52.160 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:52.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:52.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:52.263 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:52.289 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:52.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:52.291 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:52.336 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:52.361 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[75764]: debug 2021-03-15T22:20:52.362 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[75764]: 2021-03-15T22:20:52.072+0000 7f9fbafaf700 -1 osd.3 pg_epoch: 137 pg[7.a( v 136'63 (0'0,136'63] local-lis/les=91/92 n=9 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [3,2,5] r=0 lpr=91 crt=136'63 lcod 136'61 mlcod 136'61 active+clean+snaptrim trimq=[22~1] ps=[19~1,1d~1,21~1,25~1]] removing snap head 2021-03-15T22:20:52.362 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[75764]: 2021-03-15T22:20:52.363 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: cluster 2021-03-15T22:20:52.364 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: 2021-03-15T22:20:52.364 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: 2021 2021-03-15T22:20:52.364 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: -03-15T22:20:51. 2021-03-15T22:20:52.365 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: 037463+0000 osd.0 (osd.0) 33 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:52.366 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: cluster 2021-03-15T22:20 2021-03-15T22:20:52.366 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: :51.037639+0000 osd.0 (osd.0) 2021-03-15T22:20:52.367 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: 34 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:52.367 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:20:52.368 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: 15T22:20:51.056943+0000 mon.a ( 2021-03-15T22:20:52.368 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: mon.0) 679 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:52.369 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: audit 2021-03-15T22:20:51.065306+0000 mon.a ( 2021-03-15T22:20:52.369 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: mon.0) 680 : audit [INF] from='client.? 172.21.15.76:0/3338144072' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:20:52.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:20:52.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: 20:51.065362+0000 mon.a (mon.0) 681 2021-03-15T22:20:52.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2021-03-15T22:20:52.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:20:52.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: T22:20:51.316233+0000 osd.4 (osd. 2021-03-15T22:20:52.372 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[56514]: 4) 32 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:20:52.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: cluster 2021-03-15T22:20:52.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 2021-03 2021-03-15T22:20:52.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: -15T22:20:51.037463+0000 osd.0 (osd.0) 33 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:52.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: cluster 2021-03-15T22:20:52.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 2021-03-15T22:20:52.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 2021 2021-03-15T22:20:52.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: - 2021-03-15T22:20:52.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 03 2021-03-15T22:20:52.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: - 2021-03-15T22:20:52.377 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 15 2021-03-15T22:20:52.377 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: T 2021-03-15T22:20:52.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 22 2021-03-15T22:20:52.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: : 2021-03-15T22:20:52.379 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 20 2021-03-15T22:20:52.379 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: : 2021-03-15T22:20:52.380 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 51 2021-03-15T22:20:52.380 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: . 2021-03-15T22:20:52.381 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 037639 2021-03-15T22:20:52.381 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: +0000 2021-03-15T22:20:52.382 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 2021-03-15T22:20:52.382 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: osd.0 2021-03-15T22:20:52.383 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: ( 2021-03-15T22:20:52.383 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: osd 2021-03-15T22:20:52.383 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: . 2021-03-15T22:20:52.384 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 0 2021-03-15T22:20:52.384 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: ) 2021-03-15T22:20:52.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 34 2021-03-15T22:20:52.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: : 2021-03-15T22:20:52.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:52.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: cluster 2021-03-15T22:20:51.056943+0000 mon.a (mon.0) 679 2021-03-15T22:20:52.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:52.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:20:52.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 20:51.065306+0000 mon.a (mon. 2021-03-15T22:20:52.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 0) 680 : audit [INF] from='client.? 172.21.15.76:0/3338144072' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:20:52.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:20:52.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 03-15T22:20:51.065362 2021-03-15T22:20:52.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: +0000 mon.a (mon.0) 681 : cluster 2021-03-15T22:20:52.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: [DBG] osdmap e136: 8 total, 8 up, 8 in 2021-03-15T22:20:52.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:20:52.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: 22:20:51.316233+0000 osd.4 ( 2021-03-15T22:20:52.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:52 smithi076 conmon[62156]: osd.4) 32 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:20:52.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-15T22:20:52.394 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:52.421 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:52.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:52.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:52.525 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:52.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: cluster 2021-03-15T22:20:51.037463+0000 osd.0 (osd.0) 33 : cluster 2021-03-15T22:20:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: [DBG] purged_snaps scrub starts 2021-03-15T22:20:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: cluster 2021-03-15T22:20:51.037639+0000 osd.0 (osd.0) 34 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:20:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: 22:20:51.056943+0000 mon.a (mon.0) 679 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:20:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: audit 2021-03-15T22:20:51.065306+0000 mon.a 2021-03-15T22:20:52.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: (mon.0) 680 : audit [INF] from='client.? 172.21.15.76:0/3338144072' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:20:52.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:20:52.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: -03-15T22:20:51.065362+0000 mon.a 2021-03-15T22:20:52.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: (mon.0) 681 : cluster [DBG] osdmap e136: 8 total, 8 up, 8 in 2021-03-15T22:20:52.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: cluster 2021-03-15T22:20:52.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: 2021-03-15T22:20:51.316233+0000 osd.4 (osd.4 2021-03-15T22:20:52.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:52 smithi125 conmon[56739]: ) 32 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:20:52.549 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:52.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:52.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:52.652 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:52.678 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:52.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:52.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:52.780 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:52.806 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:52.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:52.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:52.909 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:52.934 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:52.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:52.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:52.982 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:53.036 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:53.059 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:53.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:53.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:53.077 INFO:tasks.rados.rados.0.smithi076.stdout:228: done (2 left) 2021-03-15T22:20:53.077 INFO:tasks.rados.rados.0.smithi076.stdout:230: done (1 left) 2021-03-15T22:20:53.077 INFO:tasks.rados.rados.0.smithi076.stdout:231: done (0 left) 2021-03-15T22:20:53.078 INFO:tasks.rados.rados.0.smithi076.stdout:232: write oid 18 current snap is 24 2021-03-15T22:20:53.078 INFO:tasks.rados.rados.0.smithi076.stdout:232: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2021-03-15T22:20:53.080 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 5 pg_num to 22 2021-03-15T22:20:53.081 INFO:tasks.rados.rados.0.smithi076.stdout:232: writing smithi07694878-18 from 682384 to 1139214 tid 1 2021-03-15T22:20:53.086 INFO:tasks.rados.rados.0.smithi076.stdout:232: writing smithi07694878-18 from 1721042 to 2144741 tid 2 2021-03-15T22:20:53.093 INFO:tasks.rados.rados.0.smithi076.stdout:232: writing smithi07694878-18 from 2809728 to 3264664 tid 3 2021-03-15T22:20:53.097 INFO:tasks.rados.rados.0.smithi076.stdout:232: writing smithi07694878-18 from 3731357 to 3882384 tid 4 2021-03-15T22:20:53.097 INFO:tasks.rados.rados.0.smithi076.stdout:233: read oid 27 snap -1 2021-03-15T22:20:53.098 INFO:tasks.rados.rados.0.smithi076.stdout:233: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:20:53.098 INFO:tasks.rados.rados.0.smithi076.stdout:234: snap_remove snap 22 2021-03-15T22:20:53.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:53.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:53.185 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:53.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:53.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[56514]: cluster 2021-03-15T22:20:52.000836+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.b scrub ok 2021-03-15T22:20:53.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[56514]: cluster 2021-03-15T22:20:52.014262+0000 osd.1 (osd.1) 30 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:20:53.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[56514]: cluster 2021-03-15T22:20:52.065733+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2021-03-15T22:20:53.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[56514]: audit 2021-03-15T22:20:52.842699+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.76:0/79297977' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2021-03-15T22:20:53.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[56514]: audit 2021-03-15T22:20:52.843059+0000 mon.a (mon.0) 683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2021-03-15T22:20:53.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[56514]: cluster 2021-03-15T22:20:52.986882+0000 mgr.y (mgr.14144) 402 : cluster [DBG] pgmap v333: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 174 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.9 MiB/s wr, 14 op/s 2021-03-15T22:20:53.218 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[62156]: cluster 2021-03-15T22:20:52.000836+0000 osd.2 (osd.2) 25 : cluster [DBG] 5.b scrub ok 2021-03-15T22:20:53.218 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[62156]: cluster 2021-03-15T22:20:52.014262+0000 osd.1 (osd.1) 30 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:20:53.218 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[62156]: cluster 2021-03-15T22:20:52.065733+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2021-03-15T22:20:53.218 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[62156]: audit 2021-03-15T22:20:52.842699+0000 mon.c (mon.1) 19 : audit [INF] from='client.? 172.21.15.76:0/79297977' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2021-03-15T22:20:53.219 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[62156]: audit 2021-03-15T22:20:52.843059+0000 mon.a (mon.0) 683 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2021-03-15T22:20:53.219 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:53 smithi076 conmon[62156]: cluster 2021-03-15T22:20:52.986882+0000 mgr.y (mgr.14144) 402 : cluster [DBG] pgmap v333: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 174 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.9 MiB/s wr, 14 op/s 2021-03-15T22:20:53.287 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:53.309 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:53.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:53.310 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:53.411 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:53.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:53.434 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:53.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:53.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:53.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: cluster 2021-03-15T22:20:52.000836+0000 osd.2 (osd.2) 25 : cluster 2021-03-15T22:20:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: [DBG] 5.b scrub ok 2021-03-15T22:20:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: cluster 2021-03-15T22:20:52.014262+0000 osd.1 (osd.1) 30 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:20:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:20:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: -15T22:20:52.065733+0000 mon.a (mon.0) 682 : cluster [DBG] osdmap e137: 8 total, 8 up, 8 in 2021-03-15T22:20:53.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: audit 2021-03-15T22:20:53.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: 2021-03-15T22:20:52.842699+0000 mon.c 2021-03-15T22:20:53.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: (mon.1) 19 : audit [INF] from='client.? 172.21.15.76:0/79297977' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2021-03-15T22:20:53.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: audit 2021-03-15T22:20:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: 2021-03-15T22:20:52.843059+0000 2021-03-15T22:20:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: mon.a (mon.0) 683 : audit [INF] 2021-03-15T22:20:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]: dispatch 2021-03-15T22:20:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: cluster 2021-03-15T22:20: 2021-03-15T22:20:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: 52.986882+0000 mgr.y (mgr.14144) 402 : 2021-03-15T22:20:53.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:53 smithi125 conmon[56739]: cluster [DBG] pgmap v333: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 174 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 3.0 KiB/s rd, 2.9 MiB/s wr, 14 op/s 2021-03-15T22:20:53.565 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:53.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:53.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:53.693 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:53.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:53.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:53.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:53.821 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:53.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:53.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:53.924 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:53.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:53.949 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:53.950 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:54.051 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:54.074 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:54.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:54.076 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:54.091 INFO:tasks.rados.rados.0.smithi076.stdout:234: done (2 left) 2021-03-15T22:20:54.091 INFO:tasks.rados.rados.0.smithi076.stdout:235: write (excl) oid 29 current snap is 24 2021-03-15T22:20:54.091 INFO:tasks.rados.rados.0.smithi076.stdout:235: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2021-03-15T22:20:54.097 INFO:tasks.rados.rados.0.smithi076.stdout:235: writing smithi07694878-29 from 699191 to 1231270 tid 1 2021-03-15T22:20:54.104 INFO:tasks.rados.rados.0.smithi076.stdout:235: writing smithi07694878-29 from 1779524 to 2346881 tid 2 2021-03-15T22:20:54.111 INFO:tasks.rados.rados.0.smithi076.stdout:235: writing smithi07694878-29 from 2837151 to 3503359 tid 3 2021-03-15T22:20:54.111 INFO:tasks.rados.rados.0.smithi076.stdout:236: write (excl) oid 41 current snap is 24 2021-03-15T22:20:54.111 INFO:tasks.rados.rados.0.smithi076.stdout:232: finishing write tid 1 to smithi07694878-18 2021-03-15T22:20:54.111 INFO:tasks.rados.rados.0.smithi076.stdout:236: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2021-03-15T22:20:54.117 INFO:tasks.rados.rados.0.smithi076.stdout:236: writing smithi07694878-41 from 715998 to 1323326 tid 1 2021-03-15T22:20:54.123 INFO:tasks.rados.rados.0.smithi076.stdout:236: writing smithi07694878-41 from 1838006 to 2265374 tid 2 2021-03-15T22:20:54.130 INFO:tasks.rados.rados.0.smithi076.stdout:236: writing smithi07694878-41 from 2864574 to 3342054 tid 3 2021-03-15T22:20:54.132 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:20:54.134 INFO:tasks.rados.rados.0.smithi076.stdout:236: writing smithi07694878-41 from 3863835 to 3915998 tid 4 2021-03-15T22:20:54.134 INFO:tasks.rados.rados.0.smithi076.stdout:237: read oid 46 snap -1 2021-03-15T22:20:54.135 INFO:tasks.rados.rados.0.smithi076.stdout:237: expect (ObjNum 45 snap 0 seq_num 45) 2021-03-15T22:20:54.135 INFO:tasks.rados.rados.0.smithi076.stdout:238: read oid 8 snap -1 2021-03-15T22:20:54.135 INFO:tasks.rados.rados.0.smithi076.stdout:238: expect (ObjNum 7 snap 0 seq_num 7) 2021-03-15T22:20:54.136 INFO:tasks.rados.rados.0.smithi076.stdout:239: snap_remove snap 14 2021-03-15T22:20:54.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:54.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:54.202 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:54.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:54.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:54.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[56514]: cluster 2021-03-15T22:20:52.775168+0000 osd.3 (osd.3) 37 2021-03-15T22:20:54.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[56514]: : cluster [DBG] 5.13 scrub ok 2021-03-15T22:20:54.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[56514]: cluster 2021-03-15T22:20:52.979259+0000 osd.1 (osd.1) 31 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:20:54.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[56514]: cluster 2021-03-15T22:20:53.009468+0000 osd.7 (osd.7) 34 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:20:54.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[56514]: cluster 2021-03-15T22:20:53.032325+0000 osd.2 (osd.2) 26 : cluster [DBG] 5.d scrub ok 2021-03-15T22:20:54.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[56514]: audit 2021-03-15T22:20:53.071792+0000 mon.a (mon.0) 684 : audit [INF] 2021-03-15T22:20:54.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[56514]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2021-03-15T22:20:54.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[56514]: cluster 2021-03-15T22:20:53.072054+0000 mon.a (mon.0) 685 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2021-03-15T22:20:54.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[56514]: cluster 2021-03-15T22:20:54.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[56514]: 2021-03-15T22:20:53.271056+0000 osd.4 (osd.4) 33 : 2021-03-15T22:20:54.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[56514]: cluster [DBG] 3.1a scrub ok 2021-03-15T22:20:54.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: cluster 2021-03-15T22:20:52.775168 2021-03-15T22:20:54.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: +0000 osd.3 (osd.3) 37 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:20:54.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: cluster 2021-03-15T22:20:52.979259 2021-03-15T22:20:54.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 31 : cluster [DBG] 2021-03-15T22:20:54.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: 6.1 scrub ok 2021-03-15T22:20:54.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:20:54.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: 20:53.009468+0000 osd.7 (osd.7 2021-03-15T22:20:54.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: ) 34 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:20:54.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: cluster 2021-03-15T22:20:54.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:20:54.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: T22:20:53.032325+0000 osd.2 ( 2021-03-15T22:20:54.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: osd.2) 26 : cluster [DBG] 5.d scrub ok 2021-03-15T22:20:54.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: 2021-03-15T22:20:54.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:20:54.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: T22:20:53.071792+0000 mon.a ( 2021-03-15T22:20:54.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: mon.0) 684 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2021-03-15T22:20:54.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:20:54.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: 20:53.072054+0000 mon.a (mon.0) 685 : 2021-03-15T22:20:54.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2021-03-15T22:20:54.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:20:54.329 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: -03-15T22:20:53.271056+0000 2021-03-15T22:20:54.329 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: osd.4 (osd.4) 33 : cluster [DBG] 2021-03-15T22:20:54.329 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:54 smithi076 conmon[62156]: 3.1a scrub ok 2021-03-15T22:20:54.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:54.331 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:54.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:54.432 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:54.458 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:54.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:54.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:54.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: cluster 2021-03-15T22:20:52.775168+0000 osd.3 (osd.3) 37 : 2021-03-15T22:20:54.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: cluster [DBG] 5.13 scrub ok 2021-03-15T22:20:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: cluster 2021-03-15T22:20:52.979259+0000 osd.1 (osd.1) 31 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:20:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: cluster 2021-03-15T22:20:53.009468+0000 osd.7 (osd.7) 2021-03-15T22:20:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: 34 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:20:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: cluster 2021-03-15T22:20:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: 2021-03-15T22:20:53.032325+0000 osd.2 2021-03-15T22:20:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: (osd.2) 26 : cluster [DBG] 5.d scrub ok 2021-03-15T22:20:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:20:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: 03-15T22:20:53.071792+0000 mon.a (mon.0) 684 2021-03-15T22:20:54.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "22"}]': finished 2021-03-15T22:20:54.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:20:54.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: -03-15T22:20:53.072054+0000 mon.a 2021-03-15T22:20:54.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: (mon.0) 685 : cluster [DBG] osdmap e138: 8 total, 8 up, 8 in 2021-03-15T22:20:54.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:20:54.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: T22:20:53.271056+0000 osd.4 (osd.4) 33 : cluster 2021-03-15T22:20:54.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:54 smithi125 conmon[56739]: [DBG] 3.1a scrub ok 2021-03-15T22:20:54.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:54.586 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:54.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:54.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:54.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:54.712 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:54.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:54.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:54.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:54.840 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:54.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:54.842 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:20:54.858 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:20:54.875 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:20:54.892 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:20:54.910 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:20:54.927 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:20:54.942 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:54.945 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:20:54.962 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:20:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:20:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:20:54.963 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:20:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:20:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:20:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:20:54.964 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:20:54.965 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:20:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:54.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:55.014 INFO:tasks.rados.rados.0.smithi076.stdout:239: done (6 left) 2021-03-15T22:20:55.014 INFO:tasks.rados.rados.0.smithi076.stdout:240: delete oid 23 current snap is 24 2021-03-15T22:20:55.015 INFO:tasks.rados.rados.0.smithi076.stdout:232: finishing write tid 2 to smithi07694878-18 2021-03-15T22:20:55.015 INFO:tasks.rados.rados.0.smithi076.stdout:232: finishing write tid 3 to smithi07694878-18 2021-03-15T22:20:55.015 INFO:tasks.rados.rados.0.smithi076.stdout:232: finishing write tid 4 to smithi07694878-18 2021-03-15T22:20:55.015 INFO:tasks.rados.rados.0.smithi076.stdout:232: finishing write tid 5 to smithi07694878-18 2021-03-15T22:20:55.016 INFO:tasks.rados.rados.0.smithi076.stdout:232: finishing write tid 6 to smithi07694878-18 2021-03-15T22:20:55.016 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 38 (ObjNum 112 snap 24 seq_num 112) dirty exists 2021-03-15T22:20:55.016 INFO:tasks.rados.rados.0.smithi076.stdout:232: left oid 18 (ObjNum 112 snap 24 seq_num 112) 2021-03-15T22:20:55.016 INFO:tasks.rados.rados.0.smithi076.stdout:233: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:20:55.068 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:55.092 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:55.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:55.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:55.169 INFO:tasks.rados.rados.0.smithi076.stdout:235: finishing write tid 1 to smithi07694878-29 2021-03-15T22:20:55.170 INFO:tasks.rados.rados.0.smithi076.stdout:235: finishing write tid 2 to smithi07694878-29 2021-03-15T22:20:55.170 INFO:tasks.rados.rados.0.smithi076.stdout:232: done (6 left) 2021-03-15T22:20:55.170 INFO:tasks.rados.rados.0.smithi076.stdout:233: done (5 left) 2021-03-15T22:20:55.171 INFO:tasks.rados.rados.0.smithi076.stdout:240: done (4 left) 2021-03-15T22:20:55.171 INFO:tasks.rados.rados.0.smithi076.stdout:241: snap_remove snap 17 2021-03-15T22:20:55.195 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:55.218 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:55.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:55.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:55.321 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:55.342 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:55.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:55.344 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:55.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:55.467 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:55.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:55.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:55.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[62156]: cluster 2021-03-15T22:20:54.007331+0000 osd.7 (osd.7) 35 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:20:55.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[62156]: cluster 2021-03-15T22:20:54.018624+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:20:55.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[62156]: cluster 2021-03-15T22:20:54.088052+0000 mon.a (mon.0) 686 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2021-03-15T22:20:55.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[62156]: cluster 2021-03-15T22:20:54.221643+0000 osd.4 (osd.4) 34 : cluster [DBG] 5.1c scrub ok 2021-03-15T22:20:55.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[62156]: audit 2021-03-15T22:20:54.822010+0000 mon.a (mon.0) 687 : audit 2021-03-15T22:20:55.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/439927883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:55.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[62156]: cluster 2021-03-15T22:20:54.987346+0000 mgr.y (mgr.14144) 403 : cluster [DBG] pgmap v336: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 174 MiB data, 466 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-03-15T22:20:55.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[62156]: audit 2021-03-15T22:20:54.988136+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-03-15T22:20:55.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[62156]: audit 2021-03-15T22:20:55.006660+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2021-03-15T22:20:55.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[62156]: cluster 2021-03-15T22:20:55.006883+0000 mon.a (mon.0) 690 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2021-03-15T22:20:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[56514]: cluster 2021-03-15T22:20:54.007331+0000 osd.7 (osd.7) 35 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:20:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[56514]: cluster 2021-03-15T22:20:54.018624+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:20:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[56514]: cluster 2021-03-15T22:20:54.088052+0000 mon.a (mon.0) 686 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2021-03-15T22:20:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[56514]: cluster 2021-03-15T22:20:54.221643+0000 osd.4 (osd.4) 34 : cluster [DBG] 5.1c scrub ok 2021-03-15T22:20:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[56514]: audit 2021-03-15T22:20:54.822010+0000 mon.a (mon.0) 687 : audit [DBG] from='client.? 172.21.15.76:0/439927883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:20:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[56514]: 22:20:54.987346+0000 mgr.y (mgr.14144) 403 : cluster [DBG] pgmap v336: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 174 MiB data, 466 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-03-15T22:20:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[56514]: audit 2021-03-15T22:20:54.988136+0000 mon.a (mon.0) 688 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-03-15T22:20:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[56514]: audit 2021-03-15T22:20:55.006660+0000 mon.a (mon.0) 689 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2021-03-15T22:20:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:55 smithi076 conmon[56514]: cluster 2021-03-15T22:20:55.006883+0000 mon.a (mon.0) 690 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2021-03-15T22:20:55.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: cluster 2021-03-15T22:20:54.007331+0000 osd.7 (osd.7) 35 : cluster [DBG] 2021-03-15T22:20:55.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: 1.0 scrub ok 2021-03-15T22:20:55.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: cluster 2021-03-15T22:20:54.018624+0000 osd.2 (osd.2) 27 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:20:55.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: cluster 2021-03-15T22:20:54.088052 2021-03-15T22:20:55.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: +0000 mon.a (mon.0) 686 : cluster [DBG] osdmap e139: 8 total, 8 up, 8 in 2021-03-15T22:20:55.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:20:55.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: :20:54.221643+0000 osd.4 (osd. 2021-03-15T22:20:55.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: 4) 34 : cluster [DBG] 5.1c scrub ok 2021-03-15T22:20:55.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: audit 2021 2021-03-15T22:20:55.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: -03-15T22:20:54.822010 2021-03-15T22:20:55.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: +0000 mon.a (mon.0) 687 : audit 2021-03-15T22:20:55.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/439927883' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:20:55.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:20:55.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: 20:54.987346+0000 mgr.y (mgr.14144 2021-03-15T22:20:55.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: ) 403 : cluster [DBG] pgmap v336: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 174 MiB data, 466 MiB used, 715 GiB / 715 GiB avail; 6.3 MiB/s rd, 1.6 MiB/s wr, 13 op/s 2021-03-15T22:20:55.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:20:55.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: 15T22:20:54.988136+0000 mon.a 2021-03-15T22:20:55.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: (mon.0) 688 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-03-15T22:20:55.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: 2021-03-15T22:20:55.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: audit 2021-03-15T22:20:55.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: 2021-03-15T22:20:55.006660+0000 2021-03-15T22:20:55.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: mon.a (mon.0) 689 : audit [INF] 2021-03-15T22:20:55.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "22"}]': finished 2021-03-15T22:20:55.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: cluster 2021-03-15T22:20:55.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: 2021-03-15T22:20:55.006883+0000 mon.a 2021-03-15T22:20:55.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: (mon.0) 690 : cluster [DBG] osdmap e140: 8 total, 8 up, 8 in 2021-03-15T22:20:55.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:55 smithi125 conmon[56739]: 2021-03-15T22:20:55.570 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:55.592 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:55.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:55.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:55.639 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:55.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:55.721 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:55.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:55.723 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:55.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:55.849 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:55.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:55.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:55.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:55.975 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:55.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:55.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:56.012 INFO:tasks.rados.rados.0.smithi076.stdout:241: done (4 left) 2021-03-15T22:20:56.013 INFO:tasks.rados.rados.0.smithi076.stdout:242: write oid 39 current snap is 24 2021-03-15T22:20:56.013 INFO:tasks.rados.rados.0.smithi076.stdout:242: seq_num 116 ranges {749612=474179,1954970=431037,2919420=616377} 2021-03-15T22:20:56.017 INFO:tasks.rados.rados.0.smithi076.stdout:242: writing smithi07694878-39 from 749612 to 1223791 tid 1 2021-03-15T22:20:56.024 INFO:tasks.rados.rados.0.smithi076.stdout:242: writing smithi07694878-39 from 1954970 to 2386007 tid 2 2021-03-15T22:20:56.031 INFO:tasks.rados.rados.0.smithi076.stdout:242: writing smithi07694878-39 from 2919420 to 3535797 tid 3 2021-03-15T22:20:56.032 INFO:tasks.rados.rados.0.smithi076.stdout:243: snap_remove snap 18 2021-03-15T22:20:56.078 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:56.102 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:56.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:56.205 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:56.211 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[56514]: cluster 2021-03-15T22:20:54.994172 2021-03-15T22:20:56.211 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[56514]: +0000 osd.7 (osd.7) 36 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:20:56.211 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[56514]: cluster 2021-03-15T22:20:55.151283+0000 osd.5 (osd.5) 31 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:20:56.212 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[56514]: cluster 2021-03-15T22:20:55.236480+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:20:56.212 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[56514]: cluster 2021-03-15T22:20:56.003359+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2021-03-15T22:20:56.212 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[62156]: cluster 2021-03-15T22:20:54.994172+0000 osd.7 2021-03-15T22:20:56.212 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[62156]: (osd.7) 36 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:20:56.213 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[62156]: cluster 2021-03-15T22:20:55.151283+0000 2021-03-15T22:20:56.213 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[62156]: osd.5 (osd.5) 31 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:20:56.213 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:20:56.213 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[62156]: :20:55.236480+0000 osd.4 (osd.4) 35 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:20:56.214 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[62156]: cluster 2021-03-15T22:20:56 2021-03-15T22:20:56.214 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:56 smithi076 conmon[62156]: .003359+0000 mon.a (mon.0) 691 : cluster [DBG] osdmap e141: 8 total, 8 up, 8 in 2021-03-15T22:20:56.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:56.232 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:56.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:56.334 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:56.344 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:20:56.344 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:20:56.358 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:56.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:56.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:56.485 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:56.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:56.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:56 smithi125 conmon[56739]: cluster 2021-03-15T22:20:54.994172+0000 osd.7 (osd.7) 36 : cluster 2021-03-15T22:20:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:56 smithi125 conmon[56739]: [DBG] 5.10 scrub ok 2021-03-15T22:20:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:56 smithi125 conmon[56739]: cluster 2021-03-15T22:20:55.151283+0000 osd.5 (osd.5) 31 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:20:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:56 smithi125 conmon[56739]: cluster 2021-03-15T22:20:55.236480+0000 2021-03-15T22:20:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:56 smithi125 conmon[56739]: osd.4 (osd.4) 35 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:20:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:56 smithi125 conmon[56739]: cluster 2021-03-15T22:20:56.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:56 smithi125 conmon[56739]: 2021-03-15T22:20:56.003359+0000 2021-03-15T22:20:56.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:56 smithi125 conmon[56739]: mon.a (mon.0) 691 : cluster [DBG] 2021-03-15T22:20:56.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:56 smithi125 conmon[56739]: osdmap e141: 8 total, 8 up, 8 in 2021-03-15T22:20:56.588 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:56.612 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:56.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:56.613 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:56.714 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:56.737 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:56.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:56.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:56.840 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:56.866 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:56.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:56.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:56.969 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:56.991 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:56.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:56.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:57.005 INFO:tasks.rados.rados.0.smithi076.stdout:243: done (5 left) 2021-03-15T22:20:57.005 INFO:tasks.rados.rados.0.smithi076.stdout:244: delete oid 36 current snap is 24 2021-03-15T22:20:57.013 INFO:tasks.rados.rados.0.smithi076.stdout:235: finishing write tid 3 to smithi07694878-29 2021-03-15T22:20:57.013 INFO:tasks.rados.rados.0.smithi076.stdout:235: finishing write tid 4 to smithi07694878-29 2021-03-15T22:20:57.014 INFO:tasks.rados.rados.0.smithi076.stdout:235: finishing write tid 5 to smithi07694878-29 2021-03-15T22:20:57.014 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 127 (ObjNum 113 snap 24 seq_num 113) dirty exists 2021-03-15T22:20:57.014 INFO:tasks.rados.rados.0.smithi076.stdout:235: left oid 29 (ObjNum 113 snap 24 seq_num 113) 2021-03-15T22:20:57.014 INFO:tasks.rados.rados.0.smithi076.stdout:236: finishing write tid 1 to smithi07694878-41 2021-03-15T22:20:57.015 INFO:tasks.rados.rados.0.smithi076.stdout:236: finishing write tid 2 to smithi07694878-41 2021-03-15T22:20:57.015 INFO:tasks.rados.rados.0.smithi076.stdout:236: finishing write tid 3 to smithi07694878-41 2021-03-15T22:20:57.015 INFO:tasks.rados.rados.0.smithi076.stdout:236: finishing write tid 4 to smithi07694878-41 2021-03-15T22:20:57.015 INFO:tasks.rados.rados.0.smithi076.stdout:236: finishing write tid 5 to smithi07694878-41 2021-03-15T22:20:57.016 INFO:tasks.rados.rados.0.smithi076.stdout:236: finishing write tid 6 to smithi07694878-41 2021-03-15T22:20:57.016 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 57 (ObjNum 114 snap 24 seq_num 114) dirty exists 2021-03-15T22:20:57.016 INFO:tasks.rados.rados.0.smithi076.stdout:236: left oid 41 (ObjNum 114 snap 24 seq_num 114) 2021-03-15T22:20:57.016 INFO:tasks.rados.rados.0.smithi076.stdout:237: expect (ObjNum 45 snap 0 seq_num 45) 2021-03-15T22:20:57.091 INFO:tasks.rados.rados.0.smithi076.stdout:238: expect (ObjNum 7 snap 0 seq_num 7) 2021-03-15T22:20:57.094 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:57.117 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:57.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:57.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:57.164 INFO:tasks.rados.rados.0.smithi076.stdout:242: finishing write tid 1 to smithi07694878-39 2021-03-15T22:20:57.165 INFO:tasks.rados.rados.0.smithi076.stdout:242: finishing write tid 2 to smithi07694878-39 2021-03-15T22:20:57.166 INFO:tasks.rados.rados.0.smithi076.stdout:242: finishing write tid 3 to smithi07694878-39 2021-03-15T22:20:57.166 INFO:tasks.rados.rados.0.smithi076.stdout:242: finishing write tid 4 to smithi07694878-39 2021-03-15T22:20:57.167 INFO:tasks.rados.rados.0.smithi076.stdout:242: finishing write tid 5 to smithi07694878-39 2021-03-15T22:20:57.167 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 73 (ObjNum 116 snap 24 seq_num 116) dirty exists 2021-03-15T22:20:57.167 INFO:tasks.rados.rados.0.smithi076.stdout:242: left oid 39 (ObjNum 116 snap 24 seq_num 116) 2021-03-15T22:20:57.168 INFO:tasks.rados.rados.0.smithi076.stdout:235: done (5 left) 2021-03-15T22:20:57.168 INFO:tasks.rados.rados.0.smithi076.stdout:236: done (4 left) 2021-03-15T22:20:57.169 INFO:tasks.rados.rados.0.smithi076.stdout:237: done (3 left) 2021-03-15T22:20:57.169 INFO:tasks.rados.rados.0.smithi076.stdout:238: done (2 left) 2021-03-15T22:20:57.170 INFO:tasks.rados.rados.0.smithi076.stdout:242: done (1 left) 2021-03-15T22:20:57.170 INFO:tasks.rados.rados.0.smithi076.stdout:244: done (0 left) 2021-03-15T22:20:57.171 INFO:tasks.rados.rados.0.smithi076.stdout:245: delete oid 20 current snap is 24 2021-03-15T22:20:57.173 INFO:tasks.rados.rados.0.smithi076.stdout:245: done (0 left) 2021-03-15T22:20:57.173 INFO:tasks.rados.rados.0.smithi076.stdout:246: copy_from oid 48 from oid 47 current snap is 24 2021-03-15T22:20:57.174 INFO:tasks.rados.rados.0.smithi076.stdout:247: copy_from oid 34 from oid 25 current snap is 24 2021-03-15T22:20:57.174 INFO:tasks.rados.rados.0.smithi076.stdout:248: copy_from oid 19 from oid 10 current snap is 24 2021-03-15T22:20:57.174 INFO:tasks.rados.rados.0.smithi076.stdout:249: read oid 9 snap -1 2021-03-15T22:20:57.175 INFO:tasks.rados.rados.0.smithi076.stdout:249: expect (ObjNum 60 snap 4 seq_num 60) 2021-03-15T22:20:57.175 INFO:tasks.rados.rados.0.smithi076.stdout:250: read oid 46 snap -1 2021-03-15T22:20:57.176 INFO:tasks.rados.rados.0.smithi076.stdout:250: expect (ObjNum 45 snap 0 seq_num 45) 2021-03-15T22:20:57.176 INFO:tasks.rados.rados.0.smithi076.stdout:251: snap_remove snap 16 2021-03-15T22:20:57.220 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:57.242 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:57.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:57.244 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:57.345 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:57.367 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:57.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:57.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:57.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: cluster 2021-03-15T22:20:57.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: 2021-03-15 2021-03-15T22:20:57.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: T22:20:55.778260+0000 osd.3 (osd.3) 38 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:20:57.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:20:57.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: -03-15T22:20:55 2021-03-15T22:20:57.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: .858310+0000 osd.1 (osd.1 2021-03-15T22:20:57.463 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: ) 32 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:20:57.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:20:57.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: 03-15T22:20:56 2021-03-15T22:20:57.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: .074742+0000 osd.2 (osd.2 2021-03-15T22:20:57.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: ) 28 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:20:57.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:20:57.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: -03-15T22:20:56.987968+0000 mgr.y (mgr.14144) 404 : cluster [DBG] pgmap v339: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 172 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 754 KiB/s wr, 13 op/s 2021-03-15T22:20:57.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: cluster 2021-03-15T22:20:57.004126+0000 mon.a (mon.0) 692 : cluster 2021-03-15T22:20:57.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[56514]: [DBG] osdmap e142: 8 total, 8 up, 8 in 2021-03-15T22:20:57.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: cluster 2021-03-15T22:20:55.778260 2021-03-15T22:20:57.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: +0000 osd.3 (osd.3) 38 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:20:57.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: cluster 2021-03-15T22:20:55.858310 2021-03-15T22:20:57.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 32 : cluster 2021-03-15T22:20:57.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: [DBG] 3.4 scrub ok 2021-03-15T22:20:57.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: cluster 2021-03-15T22:20:57.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:20:57.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: 20:56.074742+0000 osd.2 (osd.2 2021-03-15T22:20:57.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: ) 28 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:20:57.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:20:57.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: -03-15T22:20:56. 2021-03-15T22:20:57.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: 987968+0000 mgr.y (mgr.14144) 404 : cluster 2021-03-15T22:20:57.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: [DBG] pgmap v339: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 172 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 754 KiB/s wr, 13 op/s 2021-03-15T22:20:57.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: cluster 2021-03-15T22:20:57.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: 2021-03-15T22:20:57. 2021-03-15T22:20:57.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: 004126+0000 mon.a (mon.0) 692 : cluster 2021-03-15T22:20:57.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:57 smithi076 conmon[62156]: [DBG] osdmap e142: 8 total, 8 up, 8 in 2021-03-15T22:20:57.470 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:57.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-15T22:20:57.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:57.495 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:57.496 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:57.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:57 smithi125 conmon[56739]: cluster 2021-03-15T22:20:55.778260+0000 osd.3 (osd.3) 38 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:20:57.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:57 smithi125 conmon[56739]: cluster 2021-03-15T22:20:55.858310+0000 osd.1 (osd.1) 32 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:20:57.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:57 smithi125 conmon[56739]: cluster 2021-03-15T22:20:56.074742+0000 osd.2 (osd.2) 28 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:20:57.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:57 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:20:57.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:57 smithi125 conmon[56739]: 22:20:56.987968+0000 mgr.y (mgr. 2021-03-15T22:20:57.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:57 smithi125 conmon[56739]: 14144) 404 : cluster [DBG] pgmap v339: 148 pgs: 1 active+clean+scrubbing, 147 active+clean; 172 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 754 KiB/s wr, 13 op/s 2021-03-15T22:20:57.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:57 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:20:57.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:57 smithi125 conmon[56739]: 22:20:57.004126+0000 mon.a (mon 2021-03-15T22:20:57.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:57 smithi125 conmon[56739]: .0) 692 : cluster [DBG] osdmap e142: 8 total, 8 up, 8 in 2021-03-15T22:20:57.597 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:57.619 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:57.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:57.621 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:57.722 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:57.749 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:57.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:57.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:57.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:20:57.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:57.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:57.881 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:57.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:57.983 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:58.001 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:58.008 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:58.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:58.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:58.111 INFO:tasks.rados.rados.0.smithi076.stdout:251: done (5 left) 2021-03-15T22:20:58.111 INFO:tasks.rados.rados.0.smithi076.stdout:252: rollback oid 26 current snap is 24 2021-03-15T22:20:58.111 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 11 2021-03-15T22:20:58.112 INFO:tasks.rados.rados.0.smithi076.stdout:253: read oid 36 snap 11 2021-03-15T22:20:58.112 INFO:tasks.rados.rados.0.smithi076.stdout:253: expect (ObjNum 35 snap 0 seq_num 35) 2021-03-15T22:20:58.112 INFO:tasks.rados.rados.0.smithi076.stdout:254: copy_from oid 30 from oid 29 current snap is 24 2021-03-15T22:20:58.112 INFO:tasks.rados.rados.0.smithi076.stdout:255: delete oid 33 current snap is 24 2021-03-15T22:20:58.113 INFO:tasks.rados.rados.0.smithi076.stdout:249: expect (ObjNum 60 snap 4 seq_num 60) 2021-03-15T22:20:58.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:58.114 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[56514]: cluster 2021-03-15T22:20:56.799857+0000 osd.3 (osd.3) 39 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:20:58.114 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[56514]: cluster 2021-03-15T22:20:56.822500+0000 osd.1 (osd.1) 33 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:20:58.114 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[56514]: cluster 2021-03-15T22:20:57.165686+0000 osd.5 (osd.5) 32 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:20:58.115 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[56514]: cluster 2021-03-15T22:20:57.309326+0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1e scrub ok 2021-03-15T22:20:58.115 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[56514]: cluster 2021-03-15T22:20:57.378121+0000 osd.4 (osd.4) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:58.115 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[56514]: cluster 2021-03-15T22:20:57.378350+0000 osd.4 (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:58.142 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:58.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:58.144 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:58.157 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:20:58.244 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:58.255 INFO:tasks.rados.rados.0.smithi076.stdout:250: expect (ObjNum 45 snap 0 seq_num 45) 2021-03-15T22:20:58.270 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:58.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:58.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:20:58.344 INFO:tasks.rados.rados.0.smithi076.stdout:248: finishing copy_from to smithi07694878-19 2021-03-15T22:20:58.345 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 20 (ObjNum 9 snap 0 seq_num 9) dirty exists 2021-03-15T22:20:58.345 INFO:tasks.rados.rados.0.smithi076.stdout:248: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:20:58.345 INFO:tasks.rados.rados.0.smithi076.stdout:246: finishing copy_from to smithi07694878-48 2021-03-15T22:20:58.346 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 14 (ObjNum 63 snap 6 seq_num 63) dirty exists 2021-03-15T22:20:58.346 INFO:tasks.rados.rados.0.smithi076.stdout:246: finishing copy_from racing read to smithi07694878-48 2021-03-15T22:20:58.346 INFO:tasks.rados.rados.0.smithi076.stdout:247: finishing copy_from to smithi07694878-34 2021-03-15T22:20:58.346 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 38 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:20:58.347 INFO:tasks.rados.rados.0.smithi076.stdout:247: finishing copy_from racing read to smithi07694878-34 2021-03-15T22:20:58.347 INFO:tasks.rados.rados.0.smithi076.stdout:252: finishing rollback tid 0 to smithi07694878-26 2021-03-15T22:20:58.347 INFO:tasks.rados.rados.0.smithi076.stdout:252: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:20:58.347 INFO:tasks.rados.rados.0.smithi076.stdout:252: finishing rollback tid 2 to smithi07694878-26 2021-03-15T22:20:58.347 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 52 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-03-15T22:20:58.348 INFO:tasks.rados.rados.0.smithi076.stdout:246: done (8 left) 2021-03-15T22:20:58.348 INFO:tasks.rados.rados.0.smithi076.stdout:247: done (7 left) 2021-03-15T22:20:58.348 INFO:tasks.rados.rados.0.smithi076.stdout:248: done (6 left) 2021-03-15T22:20:58.348 INFO:tasks.rados.rados.0.smithi076.stdout:249: done (5 left) 2021-03-15T22:20:58.349 INFO:tasks.rados.rados.0.smithi076.stdout:250: done (4 left) 2021-03-15T22:20:58.349 INFO:tasks.rados.rados.0.smithi076.stdout:252: done (3 left) 2021-03-15T22:20:58.349 INFO:tasks.rados.rados.0.smithi076.stdout:255: done (2 left) 2021-03-15T22:20:58.349 INFO:tasks.rados.rados.0.smithi076.stdout:256: write oid 38 current snap is 24 2021-03-15T22:20:58.349 INFO:tasks.rados.rados.0.smithi076.stdout:253: expect (ObjNum 35 snap 0 seq_num 35) 2021-03-15T22:20:58.372 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:58.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: cluster 2021-03-15T22:20:56. 2021-03-15T22:20:58.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: 799857+0000 osd.3 (osd.3) 39 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:20:58.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: cluster 2021-03-15T22:20:56.822500+0000 osd.1 (osd.1) 33 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:20:58.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: cluster 2021-03-15T22:20:57.165686+0000 osd.5 (osd.5) 32 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:20:58.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: cluster 2021-03-15T22:20:57.309326 2021-03-15T22:20:58.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 36 : cluster [DBG] 5.1e scrub ok 2021-03-15T22:20:58.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:20:58.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: 03-15T22:20:57.378121+0000 osd.4 (osd 2021-03-15T22:20:58.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: .4) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:58.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: cluster 2021-03-15T22:20:58.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: 2021-03-15T22:20:57 2021-03-15T22:20:58.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: .378350+0000 osd.4 (osd.4) 38 2021-03-15T22:20:58.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:58 smithi076 conmon[62156]: : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:58.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:58.399 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:58.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:58.430 INFO:tasks.rados.rados.0.smithi076.stdout:254: finishing copy_from to smithi07694878-30 2021-03-15T22:20:58.431 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 127 (ObjNum 113 snap 24 seq_num 113) dirty exists 2021-03-15T22:20:58.431 INFO:tasks.rados.rados.0.smithi076.stdout:254: finishing copy_from racing read to smithi07694878-30 2021-03-15T22:20:58.431 INFO:tasks.rados.rados.0.smithi076.stdout:256: seq_num 120 ranges {416840=775175,1672545=344295} 2021-03-15T22:20:58.437 INFO:tasks.rados.rados.0.smithi076.stdout:256: writing smithi07694878-38 from 416840 to 1192015 tid 1 2021-03-15T22:20:58.442 INFO:tasks.rados.rados.0.smithi076.stdout:256: writing smithi07694878-38 from 1672545 to 2016840 tid 2 2021-03-15T22:20:58.442 INFO:tasks.rados.rados.0.smithi076.stdout:253: done (2 left) 2021-03-15T22:20:58.443 INFO:tasks.rados.rados.0.smithi076.stdout:254: done (1 left) 2021-03-15T22:20:58.443 INFO:tasks.rados.rados.0.smithi076.stdout:257: write oid 4 current snap is 24 2021-03-15T22:20:58.443 INFO:tasks.rados.rados.0.smithi076.stdout:257: seq_num 121 ranges {433647=450424,1331027=582033} 2021-03-15T22:20:58.447 INFO:tasks.rados.rados.0.smithi076.stdout:257: writing smithi07694878-4 from 433647 to 884071 tid 1 2021-03-15T22:20:58.453 INFO:tasks.rados.rados.0.smithi076.stdout:257: writing smithi07694878-4 from 1331027 to 1913060 tid 2 2021-03-15T22:20:58.453 INFO:tasks.rados.rados.0.smithi076.stdout:258: read oid 24 snap -1 2021-03-15T22:20:58.453 INFO:tasks.rados.rados.0.smithi076.stdout:258: expect (ObjNum 106 snap 22 seq_num 106) 2021-03-15T22:20:58.454 INFO:tasks.rados.rados.0.smithi076.stdout:259: delete oid 14 current snap is 24 2021-03-15T22:20:58.454 INFO:tasks.rados.rados.0.smithi076.stdout:256: finishing write tid 1 to smithi07694878-38 2021-03-15T22:20:58.454 INFO:tasks.rados.rados.0.smithi076.stdout:256: finishing write tid 2 to smithi07694878-38 2021-03-15T22:20:58.454 INFO:tasks.rados.rados.0.smithi076.stdout:256: finishing write tid 3 to smithi07694878-38 2021-03-15T22:20:58.455 INFO:tasks.rados.rados.0.smithi076.stdout:256: finishing write tid 4 to smithi07694878-38 2021-03-15T22:20:58.455 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 80 (ObjNum 120 snap 24 seq_num 120) dirty exists 2021-03-15T22:20:58.455 INFO:tasks.rados.rados.0.smithi076.stdout:256: left oid 38 (ObjNum 120 snap 24 seq_num 120) 2021-03-15T22:20:58.455 INFO:tasks.rados.rados.0.smithi076.stdout:257: finishing write tid 1 to smithi07694878-4 2021-03-15T22:20:58.456 INFO:tasks.rados.rados.0.smithi076.stdout:256: done (3 left) 2021-03-15T22:20:58.456 INFO:tasks.rados.rados.0.smithi076.stdout:259: done (2 left) 2021-03-15T22:20:58.456 INFO:tasks.rados.rados.0.smithi076.stdout:260: write (excl) oid 12 current snap is 24 2021-03-15T22:20:58.457 INFO:tasks.rados.rados.0.smithi076.stdout:260: seq_num 123 ranges {467261=717275,1847991=219270} 2021-03-15T22:20:58.461 INFO:tasks.rados.rados.0.smithi076.stdout:260: writing smithi07694878-12 from 467261 to 1184536 tid 1 2021-03-15T22:20:58.466 INFO:tasks.rados.rados.0.smithi076.stdout:260: writing smithi07694878-12 from 1847991 to 2067261 tid 2 2021-03-15T22:20:58.466 INFO:tasks.rados.rados.0.smithi076.stdout:261: rollback oid 22 current snap is 24 2021-03-15T22:20:58.466 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 22 to 23 2021-03-15T22:20:58.467 INFO:tasks.rados.rados.0.smithi076.stdout:257: finishing write tid 2 to smithi07694878-4 2021-03-15T22:20:58.467 INFO:tasks.rados.rados.0.smithi076.stdout:257: finishing write tid 3 to smithi07694878-4 2021-03-15T22:20:58.467 INFO:tasks.rados.rados.0.smithi076.stdout:257: finishing write tid 4 to smithi07694878-4 2021-03-15T22:20:58.467 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 81 (ObjNum 121 snap 24 seq_num 121) dirty exists 2021-03-15T22:20:58.468 INFO:tasks.rados.rados.0.smithi076.stdout:257: left oid 4 (ObjNum 121 snap 24 seq_num 121) 2021-03-15T22:20:58.468 INFO:tasks.rados.rados.0.smithi076.stdout:258: expect (ObjNum 106 snap 22 seq_num 106) 2021-03-15T22:20:58.501 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:58.526 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:58.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:58.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: cluster 2021-03-15T22:20:56.799857+0000 osd.3 (osd.3) 2021-03-15T22:20:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: 39 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:20:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: cluster 2021-03-15T22:20:56.822500+0000 osd.1 (osd.1) 33 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:20:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: cluster 2021-03-15T22:20:57.165686+0000 osd.5 (osd.5) 2021-03-15T22:20:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: 32 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:20:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:20:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: 22:20:57.309326+0000 osd.4 (osd.4) 36 : cluster 2021-03-15T22:20:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: [DBG] 5.1e scrub ok 2021-03-15T22:20:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:20:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: :20:57.378121+0000 osd.4 (osd. 2021-03-15T22:20:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: 4) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:20:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:20:58.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: 03-15T22:20:57.378350+0000 osd.4 2021-03-15T22:20:58.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:58 smithi125 conmon[56739]: (osd.4) 38 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:20:58.629 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:58.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:58.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:58.684 INFO:tasks.rados.rados.0.smithi076.stdout:260: finishing write tid 1 to smithi07694878-12 2021-03-15T22:20:58.684 INFO:tasks.rados.rados.0.smithi076.stdout:257: done (3 left) 2021-03-15T22:20:58.685 INFO:tasks.rados.rados.0.smithi076.stdout:258: done (2 left) 2021-03-15T22:20:58.685 INFO:tasks.rados.rados.0.smithi076.stdout:262: read oid 24 snap -1 2021-03-15T22:20:58.685 INFO:tasks.rados.rados.0.smithi076.stdout:262: expect (ObjNum 106 snap 22 seq_num 106) 2021-03-15T22:20:58.685 INFO:tasks.rados.rados.0.smithi076.stdout:263: write oid 23 current snap is 24 2021-03-15T22:20:58.685 INFO:tasks.rados.rados.0.smithi076.stdout:260: finishing write tid 2 to smithi07694878-12 2021-03-15T22:20:58.686 INFO:tasks.rados.rados.0.smithi076.stdout:260: finishing write tid 3 to smithi07694878-12 2021-03-15T22:20:58.686 INFO:tasks.rados.rados.0.smithi076.stdout:260: finishing write tid 4 to smithi07694878-12 2021-03-15T22:20:58.686 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 135 (ObjNum 123 snap 24 seq_num 123) dirty exists 2021-03-15T22:20:58.687 INFO:tasks.rados.rados.0.smithi076.stdout:260: left oid 12 (ObjNum 123 snap 24 seq_num 123) 2021-03-15T22:20:58.687 INFO:tasks.rados.rados.0.smithi076.stdout:261: finishing rollback tid 0 to smithi07694878-22 2021-03-15T22:20:58.687 INFO:tasks.rados.rados.0.smithi076.stdout:261: finishing rollback tid 1 to smithi07694878-22 2021-03-15T22:20:58.687 INFO:tasks.rados.rados.0.smithi076.stdout:261: finishing rollback tid 2 to smithi07694878-22 2021-03-15T22:20:58.687 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 139 (ObjNum 48 snap 0 seq_num 48) dirty exists 2021-03-15T22:20:58.688 INFO:tasks.rados.rados.0.smithi076.stdout:263: seq_num 124 ranges {484068=792524,1906473=177595} 2021-03-15T22:20:58.691 INFO:tasks.rados.rados.0.smithi076.stdout:263: writing smithi07694878-23 from 484068 to 1276592 tid 1 2021-03-15T22:20:58.696 INFO:tasks.rados.rados.0.smithi076.stdout:263: writing smithi07694878-23 from 1906473 to 2084068 tid 2 2021-03-15T22:20:58.697 INFO:tasks.rados.rados.0.smithi076.stdout:260: done (3 left) 2021-03-15T22:20:58.697 INFO:tasks.rados.rados.0.smithi076.stdout:261: done (2 left) 2021-03-15T22:20:58.697 INFO:tasks.rados.rados.0.smithi076.stdout:264: read oid 46 snap -1 2021-03-15T22:20:58.698 INFO:tasks.rados.rados.0.smithi076.stdout:264: expect (ObjNum 45 snap 0 seq_num 45) 2021-03-15T22:20:58.698 INFO:tasks.rados.rados.0.smithi076.stdout:265: read oid 38 snap -1 2021-03-15T22:20:58.698 INFO:tasks.rados.rados.0.smithi076.stdout:265: expect (ObjNum 120 snap 24 seq_num 120) 2021-03-15T22:20:58.698 INFO:tasks.rados.rados.0.smithi076.stdout:266: write (excl) oid 42 current snap is 24 2021-03-15T22:20:58.699 INFO:tasks.rados.rados.0.smithi076.stdout:266: seq_num 125 ranges {500875=467773,1564955=535920} 2021-03-15T22:20:58.702 INFO:tasks.rados.rados.0.smithi076.stdout:266: writing smithi07694878-42 from 500875 to 968648 tid 1 2021-03-15T22:20:58.709 INFO:tasks.rados.rados.0.smithi076.stdout:266: writing smithi07694878-42 from 1564955 to 2100875 tid 2 2021-03-15T22:20:58.709 INFO:tasks.rados.rados.0.smithi076.stdout:267: write oid 33 current snap is 24 2021-03-15T22:20:58.709 INFO:tasks.rados.rados.0.smithi076.stdout:267: seq_num 126 ranges {517682=543022,1623437=494245} 2021-03-15T22:20:58.715 INFO:tasks.rados.rados.0.smithi076.stdout:267: writing smithi07694878-33 from 517682 to 1060704 tid 1 2021-03-15T22:20:58.721 INFO:tasks.rados.rados.0.smithi076.stdout:267: writing smithi07694878-33 from 1623437 to 2117682 tid 2 2021-03-15T22:20:58.721 INFO:tasks.rados.rados.0.smithi076.stdout:268: write (excl) oid 18 current snap is 24 2021-03-15T22:20:58.721 INFO:tasks.rados.rados.0.smithi076.stdout:262: expect (ObjNum 106 snap 22 seq_num 106) 2021-03-15T22:20:58.755 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:20:58.777 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:58.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:58.778 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:20:58.880 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:20:58.906 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:58.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:58.908 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:20:58.930 INFO:tasks.rados.rados.0.smithi076.stdout:263: finishing write tid 1 to smithi07694878-23 2021-03-15T22:20:58.930 INFO:tasks.rados.rados.0.smithi076.stdout:263: finishing write tid 2 to smithi07694878-23 2021-03-15T22:20:58.931 INFO:tasks.rados.rados.0.smithi076.stdout:263: finishing write tid 3 to smithi07694878-23 2021-03-15T22:20:58.931 INFO:tasks.rados.rados.0.smithi076.stdout:265: expect (ObjNum 120 snap 24 seq_num 120) 2021-03-15T22:20:59.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:20:59.012 INFO:tasks.rados.rados.0.smithi076.stdout:263: finishing write tid 4 to smithi07694878-23 2021-03-15T22:20:59.012 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 60 (ObjNum 124 snap 24 seq_num 124) dirty exists 2021-03-15T22:20:59.012 INFO:tasks.rados.rados.0.smithi076.stdout:263: left oid 23 (ObjNum 124 snap 24 seq_num 124) 2021-03-15T22:20:59.012 INFO:tasks.rados.rados.0.smithi076.stdout:264: expect (ObjNum 45 snap 0 seq_num 45) 2021-03-15T22:20:59.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:59.031 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:59.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:20:59.097 INFO:tasks.rados.rados.0.smithi076.stdout:266: finishing write tid 1 to smithi07694878-42 2021-03-15T22:20:59.097 INFO:tasks.rados.rados.0.smithi076.stdout:266: finishing write tid 2 to smithi07694878-42 2021-03-15T22:20:59.097 INFO:tasks.rados.rados.0.smithi076.stdout:266: finishing write tid 3 to smithi07694878-42 2021-03-15T22:20:59.098 INFO:tasks.rados.rados.0.smithi076.stdout:266: finishing write tid 4 to smithi07694878-42 2021-03-15T22:20:59.098 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 149 (ObjNum 125 snap 24 seq_num 125) dirty exists 2021-03-15T22:20:59.098 INFO:tasks.rados.rados.0.smithi076.stdout:266: left oid 42 (ObjNum 125 snap 24 seq_num 125) 2021-03-15T22:20:59.098 INFO:tasks.rados.rados.0.smithi076.stdout:268: seq_num 127 ranges {534489=618271,1565566=568923} 2021-03-15T22:20:59.103 INFO:tasks.rados.rados.0.smithi076.stdout:268: writing smithi07694878-18 from 534489 to 1152760 tid 1 2021-03-15T22:20:59.109 INFO:tasks.rados.rados.0.smithi076.stdout:268: writing smithi07694878-18 from 1565566 to 2134489 tid 2 2021-03-15T22:20:59.110 INFO:tasks.rados.rados.0.smithi076.stdout:262: done (6 left) 2021-03-15T22:20:59.110 INFO:tasks.rados.rados.0.smithi076.stdout:263: done (5 left) 2021-03-15T22:20:59.110 INFO:tasks.rados.rados.0.smithi076.stdout:264: done (4 left) 2021-03-15T22:20:59.110 INFO:tasks.rados.rados.0.smithi076.stdout:265: done (3 left) 2021-03-15T22:20:59.111 INFO:tasks.rados.rados.0.smithi076.stdout:266: done (2 left) 2021-03-15T22:20:59.111 INFO:tasks.rados.rados.0.smithi076.stdout:269: write (excl) oid 49 current snap is 24 2021-03-15T22:20:59.111 INFO:tasks.rados.rados.0.smithi076.stdout:267: finishing write tid 1 to smithi07694878-33 2021-03-15T22:20:59.111 INFO:tasks.rados.rados.0.smithi076.stdout:267: finishing write tid 2 to smithi07694878-33 2021-03-15T22:20:59.111 INFO:tasks.rados.rados.0.smithi076.stdout:267: finishing write tid 3 to smithi07694878-33 2021-03-15T22:20:59.112 INFO:tasks.rados.rados.0.smithi076.stdout:267: finishing write tid 4 to smithi07694878-33 2021-03-15T22:20:59.112 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 72 (ObjNum 126 snap 24 seq_num 126) dirty exists 2021-03-15T22:20:59.112 INFO:tasks.rados.rados.0.smithi076.stdout:267: left oid 33 (ObjNum 126 snap 24 seq_num 126) 2021-03-15T22:20:59.112 INFO:tasks.rados.rados.0.smithi076.stdout:268: finishing write tid 1 to smithi07694878-18 2021-03-15T22:20:59.113 INFO:tasks.rados.rados.0.smithi076.stdout:269: seq_num 128 ranges {551296=693520,2024048=127248} 2021-03-15T22:20:59.116 INFO:tasks.rados.rados.0.smithi076.stdout:269: writing smithi07694878-49 from 551296 to 1244816 tid 1 2021-03-15T22:20:59.121 INFO:tasks.rados.rados.0.smithi076.stdout:269: writing smithi07694878-49 from 2024048 to 2151296 tid 2 2021-03-15T22:20:59.122 INFO:tasks.rados.rados.0.smithi076.stdout:267: done (2 left) 2021-03-15T22:20:59.122 INFO:tasks.rados.rados.0.smithi076.stdout:270: write (excl) oid 9 current snap is 24 2021-03-15T22:20:59.122 INFO:tasks.rados.rados.0.smithi076.stdout:270: seq_num 129 ranges {568103=768769,2082530=85573} 2021-03-15T22:20:59.130 INFO:tasks.rados.rados.0.smithi076.stdout:270: writing smithi07694878-9 from 568103 to 1336872 tid 1 2021-03-15T22:20:59.131 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:20:59.133 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:20:59.135 INFO:tasks.rados.rados.0.smithi076.stdout:270: writing smithi07694878-9 from 2082530 to 2168103 tid 2 2021-03-15T22:20:59.135 INFO:tasks.rados.rados.0.smithi076.stdout:271: read oid 42 snap -1 2021-03-15T22:20:59.135 INFO:tasks.rados.rados.0.smithi076.stdout:271: expect (ObjNum 125 snap 24 seq_num 125) 2021-03-15T22:20:59.136 INFO:tasks.rados.rados.0.smithi076.stdout:272: snap_create 2021-03-15T22:20:59.136 INFO:tasks.rados.rados.0.smithi076.stdout:268: finishing write tid 2 to smithi07694878-18 2021-03-15T22:20:59.136 INFO:tasks.rados.rados.0.smithi076.stdout:268: finishing write tid 3 to smithi07694878-18 2021-03-15T22:20:59.137 INFO:tasks.rados.rados.0.smithi076.stdout:268: finishing write tid 4 to smithi07694878-18 2021-03-15T22:20:59.137 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 63 (ObjNum 127 snap 24 seq_num 127) dirty exists 2021-03-15T22:20:59.137 INFO:tasks.rados.rados.0.smithi076.stdout:268: left oid 18 (ObjNum 127 snap 24 seq_num 127) 2021-03-15T22:20:59.137 INFO:tasks.rados.rados.0.smithi076.stdout:269: finishing write tid 1 to smithi07694878-49 2021-03-15T22:20:59.137 INFO:tasks.rados.rados.0.smithi076.stdout:269: finishing write tid 2 to smithi07694878-49 2021-03-15T22:20:59.138 INFO:tasks.rados.rados.0.smithi076.stdout:269: finishing write tid 3 to smithi07694878-49 2021-03-15T22:20:59.138 INFO:tasks.rados.rados.0.smithi076.stdout:269: finishing write tid 4 to smithi07694878-49 2021-03-15T22:20:59.138 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 142 (ObjNum 128 snap 24 seq_num 128) dirty exists 2021-03-15T22:20:59.138 INFO:tasks.rados.rados.0.smithi076.stdout:269: left oid 49 (ObjNum 128 snap 24 seq_num 128) 2021-03-15T22:20:59.139 INFO:tasks.rados.rados.0.smithi076.stdout:270: finishing write tid 1 to smithi07694878-9 2021-03-15T22:20:59.140 INFO:tasks.rados.rados.0.smithi076.stdout:270: finishing write tid 2 to smithi07694878-9 2021-03-15T22:20:59.140 INFO:tasks.rados.rados.0.smithi076.stdout:270: finishing write tid 3 to smithi07694878-9 2021-03-15T22:20:59.141 INFO:tasks.rados.rados.0.smithi076.stdout:270: finishing write tid 4 to smithi07694878-9 2021-03-15T22:20:59.141 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 146 (ObjNum 129 snap 24 seq_num 129) dirty exists 2021-03-15T22:20:59.141 INFO:tasks.rados.rados.0.smithi076.stdout:270: left oid 9 (ObjNum 129 snap 24 seq_num 129) 2021-03-15T22:20:59.143 INFO:tasks.rados.rados.0.smithi076.stdout:271: expect (ObjNum 125 snap 24 seq_num 125) 2021-03-15T22:20:59.158 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:59.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:59.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:20:59.260 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:20:59.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[56514]: cluster 2021-03-15T22:20:57.975694+0000 osd.7 2021-03-15T22:20:59.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[56514]: (osd.7) 37 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:20:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[56514]: cluster 2021-03-15T22:20:58.074322+0000 osd.2 (osd.2) 29 : cluster [DBG] 6.f scrub ok 2021-03-15T22:20:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[56514]: cluster 2021-03-15T22:20:58.105000+0000 mon.a (mon.0) 693 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2021-03-15T22:20:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[56514]: audit 2021-03-15T22:20:58.287683+0000 mon.a (mon.0) 694 : audit [INF] from='client.? 172.21.15.76:0/1558938028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[56514]: cluster 2021-03-15T22:20:58.988363+0000 mgr.y (mgr.14144) 405 : cluster [DBG] pgmap v342: 148 pgs: 148 active+clean; 176 MiB data, 479 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:20:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[56514]: audit 2021-03-15T22:20:58.989215+0000 mon.a (mon.0) 695 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-03-15T22:20:59.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: cluster 2021-03-15T22:20:59.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 2021-03-15T22:20:59.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 2021 2021-03-15T22:20:59.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: - 2021-03-15T22:20:59.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 03 2021-03-15T22:20:59.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: - 2021-03-15T22:20:59.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 15 2021-03-15T22:20:59.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: T 2021-03-15T22:20:59.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 22 2021-03-15T22:20:59.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: : 2021-03-15T22:20:59.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 20:57.975694 2021-03-15T22:20:59.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: +0000 osd.7 (osd.7) 37 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:20:59.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: cluster 2021-03-15T22:20:59.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 2021-03-15T22:20:59.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 2021 2021-03-15T22:20:59.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: - 2021-03-15T22:20:59.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 03 2021-03-15T22:20:59.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: - 2021-03-15T22:20:59.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 15 2021-03-15T22:20:59.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: T 2021-03-15T22:20:59.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 22 2021-03-15T22:20:59.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: : 2021-03-15T22:20:59.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 20 2021-03-15T22:20:59.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: : 2021-03-15T22:20:59.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 58 2021-03-15T22:20:59.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: . 2021-03-15T22:20:59.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 074322 2021-03-15T22:20:59.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: +0000 2021-03-15T22:20:59.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 2021-03-15T22:20:59.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: osd.2 2021-03-15T22:20:59.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: ( 2021-03-15T22:20:59.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: osd 2021-03-15T22:20:59.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: . 2021-03-15T22:20:59.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 2 2021-03-15T22:20:59.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: ) 2021-03-15T22:20:59.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 29 : cluster [DBG] 2021-03-15T22:20:59.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 6.f scrub ok 2021-03-15T22:20:59.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: cluster 2021-03-15T22:20: 2021-03-15T22:20:59.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 58.105000+0000 mon.a (mon.0) 693 2021-03-15T22:20:59.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2021-03-15T22:20:59.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:20:59.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 03-15T22:20:58.287683+0000 mon.a 2021-03-15T22:20:59.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: (mon.0) 694 : audit [INF] from='client.? 172.21.15.76:0/1558938028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:59.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:20:59.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 03-15T22:20:58.988363+0000 2021-03-15T22:20:59.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: mgr.y (mgr.14144) 405 : cluster [DBG] 2021-03-15T22:20:59.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: pgmap v342: 148 pgs: 148 active+clean; 176 MiB data, 479 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:20:59.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: audit 2021-03-15T22:20: 2021-03-15T22:20:59.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: 58.989215+0000 mon.a (mon.0) 695 2021-03-15T22:20:59.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:20:59 smithi076 conmon[62156]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-03-15T22:20:59.285 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:59.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:59.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:20:59.388 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:20:59.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:59.411 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:20:59.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:20:59.415 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:20:59.416 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:20:59.416 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump -f json-pretty 2021-03-15T22:20:59.459 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: cluster 2021-03-15T22:20:57.975694+0000 osd.7 (osd.7) 37 : 2021-03-15T22:20:59.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: cluster [DBG] 5.19 scrub ok 2021-03-15T22:20:59.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: cluster 2021-03-15T22:20:58.074322+0000 osd.2 (osd.2) 29 : cluster [DBG] 6.f scrub ok 2021-03-15T22:20:59.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: cluster 2021-03-15T22:20:58.105000+0000 mon.a (mon.0) 2021-03-15T22:20:59.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: 693 : cluster [DBG] osdmap e143: 8 total, 8 up, 8 in 2021-03-15T22:20:59.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: audit 2021-03-15T22:20:59.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: 2021-03-15T22:20:58. 2021-03-15T22:20:59.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: 287683+0000 mon.a (mon.0) 694 2021-03-15T22:20:59.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: : audit [INF] from='client.? 172.21.15.76:0/1558938028' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:20:59.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:20:59.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: -15T22:20:58.988363+0000 2021-03-15T22:20:59.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: mgr.y (mgr.14144) 405 : cluster 2021-03-15T22:20:59.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: [DBG] pgmap v342: 148 pgs: 148 active+clean; 176 MiB data, 479 MiB used, 715 GiB / 715 GiB avail; 4.4 MiB/s rd, 1.5 MiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:20:59.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:20:59.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: 03-15T22:20:58.989215 2021-03-15T22:20:59.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: +0000 mon.a (mon.0) 695 2021-03-15T22:20:59.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[56739]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-03-15T22:20:59.513 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:59.540 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:59.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:59.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:59.643 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:59.668 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:59.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:59.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:59.771 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:20:59.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:20:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:20:59.797 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:59.799 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:20:59.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:20:59.926 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:20:59.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:20:59.927 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:00.005 INFO:tasks.rados.rados.0.smithi076.stdout:268: done (4 left) 2021-03-15T22:21:00.006 INFO:tasks.rados.rados.0.smithi076.stdout:269: done (3 left) 2021-03-15T22:21:00.006 INFO:tasks.rados.rados.0.smithi076.stdout:270: done (2 left) 2021-03-15T22:21:00.006 INFO:tasks.rados.rados.0.smithi076.stdout:271: done (1 left) 2021-03-15T22:21:00.006 INFO:tasks.rados.rados.0.smithi076.stdout:272: done (0 left) 2021-03-15T22:21:00.007 INFO:tasks.rados.rados.0.smithi076.stdout:273: snap_remove snap 23 2021-03-15T22:21:00.028 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:00.052 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:00.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:00.053 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:00.154 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:00.179 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:00.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:00.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:00.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:00.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:00.305 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:00.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:00.407 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:00.432 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:00.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:00.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:00.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[56514]: cluster 2021-03-15T22:20:58.726436+0000 osd.6 (osd 2021-03-15T22:21:00.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[56514]: .6) 19 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:21:00.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[56514]: cluster 2021-03-15T22:20:58.817666+0000 osd.3 (osd.3) 40 : cluster [DBG] 3.b scrub ok 2021-03-15T22:21:00.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[56514]: cluster 2021-03-15T22:20:59.108885+0000 mon.a (mon.0) 696 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:21:00.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[56514]: cluster 2021-03-15T22:20:59.108915+0000 mon.a (mon.0) 697 : cluster [INF] Cluster is now healthy 2021-03-15T22:21:00.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[56514]: audit 2021-03-15T22:20:59.123970+0000 mon.a (mon.0) 698 : audit [INF] from='client.? 172.21.15.76:0/1558938028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:00.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[56514]: audit 2021-03-15T22:20:59.124034+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2021-03-15T22:21:00.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[56514]: cluster 2021-03-15T22:20:59.124090+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2021-03-15T22:21:00.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[56514]: audit 2021-03-15T22:20:59.458229+0000 mgr.y (mgr.14144) 406 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:00.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[56514]: cluster 2021-03-15T22:21:00.003222+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2021-03-15T22:21:00.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[62156]: cluster 2021-03-15T22:20:58.726436+0000 osd.6 (osd.6) 19 2021-03-15T22:21:00.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[62156]: : cluster [DBG] 5.11 scrub ok 2021-03-15T22:21:00.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[62156]: cluster 2021-03-15T22:20:58.817666+0000 osd.3 (osd.3) 40 : cluster [DBG] 3.b scrub ok 2021-03-15T22:21:00.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[62156]: cluster 2021-03-15T22:20:59.108885+0000 mon.a (mon.0) 696 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:21:00.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[62156]: cluster 2021-03-15T22:20:59.108915+0000 mon.a (mon.0) 697 : cluster [INF] Cluster is now healthy 2021-03-15T22:21:00.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[62156]: audit 2021-03-15T22:20:59.123970+0000 mon.a (mon.0) 698 : audit [INF] from='client.? 172.21.15.76:0/1558938028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:00.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[62156]: audit 2021-03-15T22:20:59.124034+0000 mon.a (mon.0) 699 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2021-03-15T22:21:00.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[62156]: cluster 2021-03-15T22:20:59.124090+0000 mon.a (mon.0) 700 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2021-03-15T22:21:00.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[62156]: audit 2021-03-15T22:20:59.458229+0000 mgr.y (mgr.14144) 406 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:00.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:00 smithi076 conmon[62156]: cluster 2021-03-15T22:21:00.003222+0000 mon.a (mon.0) 701 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2021-03-15T22:21:00.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:00.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: cluster 2021-03-15T22:20:58.726436+0000 osd.6 (osd.6) 19 2021-03-15T22:21:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: : cluster [DBG] 5.11 scrub ok 2021-03-15T22:21:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: cluster 2021-03-15T22:20:58.817666+0000 osd.3 (osd.3) 40 : cluster [DBG] 3.b scrub ok 2021-03-15T22:21:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: cluster 2021-03-15T22:20:59.108885+0000 mon.a ( 2021-03-15T22:21:00.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: mon.0) 696 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:21:00.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:21:00.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: -03-15T22:20:59.108915+0000 mon.a 2021-03-15T22:21:00.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: (mon.0) 697 : cluster [INF] Cluster is now healthy 2021-03-15T22:21:00.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:21:00.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: T22:20:59.123970+0000 mon.a (mon.0) 698 2021-03-15T22:21:00.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: : audit [INF] from='client.? 172.21.15.76:0/1558938028' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:00.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:21:00.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: 03-15T22:20:59.124034+0000 2021-03-15T22:21:00.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: mon.a (mon.0) 699 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "31"}]': finished 2021-03-15T22:21:00.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: 2021-03-15T22:21:00.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: cluster 2021-03-15T22:20 2021-03-15T22:21:00.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: :59.124090+0000 mon.a (mon.0) 2021-03-15T22:21:00.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: 700 : cluster [DBG] osdmap e144: 8 total, 8 up, 8 in 2021-03-15T22:21:00.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: audit 2021-03-15T22:21:00.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: 2021-03-15T22:20:59.458229+0000 2021-03-15T22:21:00.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: mgr.y (mgr.14144) 406 : audit [DBG] 2021-03-15T22:21:00.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:00.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:21:00.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: :21:00.003222+0000 mon.a (mon. 2021-03-15T22:21:00.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:00 smithi125 conmon[56739]: 0) 701 : cluster [DBG] osdmap e145: 8 total, 8 up, 8 in 2021-03-15T22:21:00.547 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:00.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:00.573 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:00.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:00.675 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:00.699 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:00.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:00.701 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:00.748 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:21:00.748 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:21:00.802 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:00.826 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:00.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:00.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:00.929 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:00.954 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:00.956 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:01.014 INFO:tasks.rados.rados.0.smithi076.stdout:273: done (0 left) 2021-03-15T22:21:01.014 INFO:tasks.rados.rados.0.smithi076.stdout:274: copy_from oid 12 from oid 49 current snap is 25 2021-03-15T22:21:01.014 INFO:tasks.rados.rados.0.smithi076.stdout:275: delete oid 18 current snap is 25 2021-03-15T22:21:01.019 INFO:tasks.rados.rados.0.smithi076.stdout:275: done (1 left) 2021-03-15T22:21:01.019 INFO:tasks.rados.rados.0.smithi076.stdout:276: delete oid 45 current snap is 25 2021-03-15T22:21:01.021 INFO:tasks.rados.rados.0.smithi076.stdout:276: done (1 left) 2021-03-15T22:21:01.022 INFO:tasks.rados.rados.0.smithi076.stdout:277: rollback oid 43 current snap is 25 2021-03-15T22:21:01.022 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 43 to 24 2021-03-15T22:21:01.022 INFO:tasks.rados.rados.0.smithi076.stdout:278: write (excl) oid 37 current snap is 25 2021-03-15T22:21:01.022 INFO:tasks.rados.rados.0.smithi076.stdout:278: seq_num 132 ranges {618524=710869,1857976=360548} 2021-03-15T22:21:01.029 INFO:tasks.rados.rados.0.smithi076.stdout:278: writing smithi07694878-37 from 618524 to 1329393 tid 1 2021-03-15T22:21:01.035 INFO:tasks.rados.rados.0.smithi076.stdout:278: writing smithi07694878-37 from 1857976 to 2218524 tid 2 2021-03-15T22:21:01.035 INFO:tasks.rados.rados.0.smithi076.stdout:279: read oid 36 snap 24 2021-03-15T22:21:01.035 INFO:tasks.rados.rados.0.smithi076.stdout:279: expect deleted 2021-03-15T22:21:01.036 INFO:tasks.rados.rados.0.smithi076.stdout:280: write oid 13 current snap is 25 2021-03-15T22:21:01.036 INFO:tasks.rados.rados.0.smithi076.stdout:277: finishing rollback tid 1 to smithi07694878-43 2021-03-15T22:21:01.036 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 28 (ObjNum 255155200 snap 21896 seq_num 4000000) dirty dne 2021-03-15T22:21:01.036 INFO:tasks.rados.rados.0.smithi076.stdout:280: seq_num 133 ranges {635331=786118,1916458=318873} 2021-03-15T22:21:01.043 INFO:tasks.rados.rados.0.smithi076.stdout:280: writing smithi07694878-13 from 635331 to 1421449 tid 1 2021-03-15T22:21:01.048 INFO:tasks.rados.rados.0.smithi076.stdout:280: writing smithi07694878-13 from 1916458 to 2235331 tid 2 2021-03-15T22:21:01.048 INFO:tasks.rados.rados.0.smithi076.stdout:277: done (4 left) 2021-03-15T22:21:01.049 INFO:tasks.rados.rados.0.smithi076.stdout:281: delete oid 41 current snap is 25 2021-03-15T22:21:01.049 INFO:tasks.rados.rados.0.smithi076.stdout:274: finishing copy_from to smithi07694878-12 2021-03-15T22:21:01.049 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 142 (ObjNum 128 snap 24 seq_num 128) dirty exists 2021-03-15T22:21:01.049 INFO:tasks.rados.rados.0.smithi076.stdout:274: finishing copy_from racing read to smithi07694878-12 2021-03-15T22:21:01.049 INFO:tasks.rados.rados.0.smithi076.stdout:278: finishing write tid 1 to smithi07694878-37 2021-03-15T22:21:01.050 INFO:tasks.rados.rados.0.smithi076.stdout:278: finishing write tid 2 to smithi07694878-37 2021-03-15T22:21:01.050 INFO:tasks.rados.rados.0.smithi076.stdout:278: finishing write tid 3 to smithi07694878-37 2021-03-15T22:21:01.050 INFO:tasks.rados.rados.0.smithi076.stdout:278: finishing write tid 4 to smithi07694878-37 2021-03-15T22:21:01.050 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 157 (ObjNum 132 snap 25 seq_num 132) dirty exists 2021-03-15T22:21:01.051 INFO:tasks.rados.rados.0.smithi076.stdout:278: left oid 37 (ObjNum 132 snap 25 seq_num 132) 2021-03-15T22:21:01.051 INFO:tasks.rados.rados.0.smithi076.stdout:280: finishing write tid 1 to smithi07694878-13 2021-03-15T22:21:01.052 INFO:tasks.rados.rados.0.smithi076.stdout:274: done (4 left) 2021-03-15T22:21:01.053 INFO:tasks.rados.rados.0.smithi076.stdout:278: done (3 left) 2021-03-15T22:21:01.053 INFO:tasks.rados.rados.0.smithi076.stdout:279: done (2 left) 2021-03-15T22:21:01.053 INFO:tasks.rados.rados.0.smithi076.stdout:281: done (1 left) 2021-03-15T22:21:01.054 INFO:tasks.rados.rados.0.smithi076.stdout:282: copy_from oid 45 from oid 37 current snap is 25 2021-03-15T22:21:01.054 INFO:tasks.rados.rados.0.smithi076.stdout:283: snap_create 2021-03-15T22:21:01.055 INFO:tasks.rados.rados.0.smithi076.stdout:280: finishing write tid 2 to smithi07694878-13 2021-03-15T22:21:01.055 INFO:tasks.rados.rados.0.smithi076.stdout:280: finishing write tid 3 to smithi07694878-13 2021-03-15T22:21:01.055 INFO:tasks.rados.rados.0.smithi076.stdout:280: finishing write tid 4 to smithi07694878-13 2021-03-15T22:21:01.055 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 89 (ObjNum 133 snap 25 seq_num 133) dirty exists 2021-03-15T22:21:01.056 INFO:tasks.rados.rados.0.smithi076.stdout:280: left oid 13 (ObjNum 133 snap 25 seq_num 133) 2021-03-15T22:21:01.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:01.065 INFO:tasks.rados.rados.0.smithi076.stdout:282: finishing copy_from to smithi07694878-45 2021-03-15T22:21:01.065 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 157 (ObjNum 132 snap 25 seq_num 132) dirty exists 2021-03-15T22:21:01.066 INFO:tasks.rados.rados.0.smithi076.stdout:282: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:21:01.083 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:01.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:01.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:01.186 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:01.211 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:01.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:01.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:01.315 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:01.337 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:01.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:01.338 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:01.439 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:01.461 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:01.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:01.463 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:01.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[56514]: cluster 2021-03-15T22:20:59.808075+0000 osd.3 (osd.3) 41 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:21:01.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[56514]: cluster 2021-03-15T22:20:59.975886+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:21:01.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[56514]: cluster 2021-03-15T22:21:00.088819+0000 osd.5 (osd.5) 33 : cluster [DBG] 3.c scrub ok 2021-03-15T22:21:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[56514]: cluster 2021-03-15T22:21:00.125311+0000 osd.2 (osd.2) 30 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:21:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[56514]: cluster 2021-03-15T22:21:00.367842+0000 osd.4 (osd.4) 39 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:21:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[56514]: cluster 2021-03-15T22:21:00.665221+0000 osd.0 (osd.0) 35 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:21:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[56514]: cluster 2021-03-15T22:21:00.988953+0000 mgr.y (mgr.14144) 407 : cluster [DBG] pgmap v345: 148 pgs: 148 active+clean; 178 MiB data, 495 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:21:01.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[56514]: cluster 2021-03-15T22:21:01.010784+0000 mon.a (mon.0) 702 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2021-03-15T22:21:01.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[56514]: audit 2021-03-15T22:21:01.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[56514]: 2021-03-15T22:21:01.042755+0000 mon.a (mon.0) 703 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:21:01.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[56514]: audit 2021-03-15T22:21:01.044070+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:21:01.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: cluster 2021-03-15T22:20:59.808075+0000 2021-03-15T22:21:01.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: osd.3 (osd.3) 41 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:21:01.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:21:01.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: :20:59.975886+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:21:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: cluster 2021-03-15T22:21:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: 2021-03-15T22:21:00.088819+0000 2021-03-15T22:21:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: osd.5 (osd.5) 33 : cluster [DBG] 3.c scrub ok 2021-03-15T22:21:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: 2021-03-15T22:21:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: cluster 2021-03-15T22:21: 2021-03-15T22:21:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: 00.125311+0000 osd.2 (osd.2) 30 2021-03-15T22:21:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: : cluster [DBG] 4.1c scrub ok 2021-03-15T22:21:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:21:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: -15T22:21:00.367842+0000 2021-03-15T22:21:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: osd.4 (osd.4) 39 : cluster [DBG] 2021-03-15T22:21:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: 6.2 scrub ok 2021-03-15T22:21:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:21:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: 21:00.665221+0000 osd.0 (osd.0 2021-03-15T22:21:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: ) 35 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:21:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: 03-15T22:21:00.988953+0000 2021-03-15T22:21:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: mgr.y (mgr.14144) 407 : cluster [DBG] 2021-03-15T22:21:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: pgmap v345: 148 pgs: 148 active+clean; 178 MiB data, 495 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:21:01.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: cluster 2021-03-15T22:21:01.010784+0000 2021-03-15T22:21:01.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: mon.a (mon.0) 702 : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2021-03-15T22:21:01.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: 2021-03-15T22:21:01.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:21:01.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: 21:01.042755+0000 mon.a (mon.0 2021-03-15T22:21:01.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: ) 703 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:21:01.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:21:01.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: 03-15T22:21:01.044070+0000 2021-03-15T22:21:01.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: mon.a (mon.0) 704 : audit [INF] 2021-03-15T22:21:01.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:01 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:21:01.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: cluster 2021-03-15T22:20:59.808075+0000 osd.3 (osd.3) 41 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:21:01.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: cluster 2021-03-15T22:20:59.975886+0000 osd.7 (osd.7) 38 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:21:01.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: cluster 2021-03-15T22:21:00.088819+0000 osd.5 (osd.5) 33 : cluster [DBG] 3.c scrub ok 2021-03-15T22:21:01.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: cluster 2021-03-15T22:21:00.125311+0000 2021-03-15T22:21:01.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: osd.2 (osd.2) 30 : cluster [DBG] 2021-03-15T22:21:01.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: 4.1c scrub ok 2021-03-15T22:21:01.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:21:01.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: :21:00.367842+0000 osd.4 (osd. 2021-03-15T22:21:01.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: 4) 39 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:21:01.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:21:01.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: 03-15T22:21:00.665221+0000 2021-03-15T22:21:01.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: osd.0 (osd.0) 35 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:21:01.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: 2021-03-15T22:21:01.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: cluster 2021-03-15T22:21:01.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: 2021-03-15T22:21:00.988953+0000 2021-03-15T22:21:01.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: mgr.y (mgr.14144) 407 : cluster [DBG] 2021-03-15T22:21:01.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: pgmap v345: 148 pgs: 148 active+clean; 178 MiB data, 495 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 1.5 MiB/s wr, 13 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:21:01.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: cluster 2021-03-15T22:21: 2021-03-15T22:21:01.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: 01.010784+0000 mon.a (mon.0) 702 2021-03-15T22:21:01.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: : cluster [DBG] osdmap e146: 8 total, 8 up, 8 in 2021-03-15T22:21:01.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:21:01.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: 15T22:21:01.042755+0000 mon.a ( 2021-03-15T22:21:01.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: mon.0) 703 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:21:01.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: audit 2021-03-15T22:21:01.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:01 smithi125 conmon[56739]: 2021-03-15T22:21:01.044070+0000 mon.a (mon.0) 704 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:21:01.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:01.569 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:21:01.569 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:21:01.569 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:21:01.570 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:21:01.570 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:21:01.570 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:21:01.570 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:21:01.571 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:21:01.571 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:21:01.571 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:21:01.571 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:21:01.572 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:21:01.572 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:21:01.572 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:21:01.574 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:21:01.574 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:21:01.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:01.588 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:01.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:01.690 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:01.717 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:01.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:01.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:01.820 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:01.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:01.845 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:01.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:01.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:01.974 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:01.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:01.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:02.015 INFO:tasks.rados.rados.0.smithi076.stdout:280: done (2 left) 2021-03-15T22:21:02.016 INFO:tasks.rados.rados.0.smithi076.stdout:282: done (1 left) 2021-03-15T22:21:02.016 INFO:tasks.rados.rados.0.smithi076.stdout:283: done (0 left) 2021-03-15T22:21:02.016 INFO:tasks.rados.rados.0.smithi076.stdout:284: delete oid 50 current snap is 26 2021-03-15T22:21:02.019 INFO:tasks.rados.rados.0.smithi076.stdout:284: done (0 left) 2021-03-15T22:21:02.019 INFO:tasks.rados.rados.0.smithi076.stdout:285: write (excl) oid 28 current snap is 26 2021-03-15T22:21:02.020 INFO:tasks.rados.rados.0.smithi076.stdout:285: seq_num 136 ranges {685752=611865,1975551=310201} 2021-03-15T22:21:02.026 INFO:tasks.rados.rados.0.smithi076.stdout:285: writing smithi07694878-28 from 685752 to 1297617 tid 1 2021-03-15T22:21:02.032 INFO:tasks.rados.rados.0.smithi076.stdout:285: writing smithi07694878-28 from 1975551 to 2285752 tid 2 2021-03-15T22:21:02.032 INFO:tasks.rados.rados.0.smithi076.stdout:286: delete oid 26 current snap is 26 2021-03-15T22:21:02.032 INFO:tasks.rados.rados.0.smithi076.stdout:285: finishing write tid 1 to smithi07694878-28 2021-03-15T22:21:02.033 INFO:tasks.rados.rados.0.smithi076.stdout:286: done (1 left) 2021-03-15T22:21:02.033 INFO:tasks.rados.rados.0.smithi076.stdout:287: snap_remove snap 11 2021-03-15T22:21:02.077 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:02.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:02.104 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:02.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:02.205 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:02.229 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:02.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:02.230 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:02.332 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:02.353 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:02.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:02.355 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:02.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:02 smithi076 conmon[56514]: audit 2021-03-15T22:21:01.416313 2021-03-15T22:21:02.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:02 smithi076 conmon[56514]: +0000 mon.a (mon.0) 705 : audit [DBG] from='client.? 172.21.15.76:0/1183979523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:02.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:02 smithi076 conmon[56514]: cluster 2021-03-15T22:21:02.011956+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2021-03-15T22:21:02.445 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:02 smithi076 conmon[62156]: audit 2021-03-15T22:21:01.416313+0000 mon.a (mon 2021-03-15T22:21:02.446 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:02 smithi076 conmon[62156]: .0) 705 : audit [DBG] from='client.? 172.21.15.76:0/1183979523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:02.446 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:02 smithi076 conmon[62156]: cluster 2021-03-15T22:21:02.011956+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2021-03-15T22:21:02.456 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:02.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-15T22:21:02.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:02.481 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:02.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:02.497 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:02 smithi125 conmon[56739]: audit 2021-03-15T22:21:01.416313+0000 mon.a (mon.0) 705 : audit [DBG] from='client.? 172.21.15.76:0/1183979523' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:02.497 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:02 smithi125 conmon[56739]: cluster 2021-03-15T22:21:02.011956+0000 mon.a (mon.0) 706 : cluster [DBG] osdmap e147: 8 total, 8 up, 8 in 2021-03-15T22:21:02.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:21:02.582 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:02.607 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:02.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:02.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:02.731 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:02.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:02.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:02.755 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:02.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:02.856 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:02.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:02.959 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:02.981 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:02.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:03.024 INFO:tasks.rados.rados.0.smithi076.stdout:287: done (1 left) 2021-03-15T22:21:03.024 INFO:tasks.rados.rados.0.smithi076.stdout:288: read oid 20 snap 24 2021-03-15T22:21:03.024 INFO:tasks.rados.rados.0.smithi076.stdout:288: expect deleted 2021-03-15T22:21:03.025 INFO:tasks.rados.rados.0.smithi076.stdout:289: snap_create 2021-03-15T22:21:03.025 INFO:tasks.rados.rados.0.smithi076.stdout:285: finishing write tid 2 to smithi07694878-28 2021-03-15T22:21:03.025 INFO:tasks.rados.rados.0.smithi076.stdout:285: finishing write tid 3 to smithi07694878-28 2021-03-15T22:21:03.025 INFO:tasks.rados.rados.0.smithi076.stdout:285: finishing write tid 4 to smithi07694878-28 2021-03-15T22:21:03.026 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 161 (ObjNum 136 snap 26 seq_num 136) dirty exists 2021-03-15T22:21:03.026 INFO:tasks.rados.rados.0.smithi076.stdout:285: left oid 28 (ObjNum 136 snap 26 seq_num 136) 2021-03-15T22:21:03.060 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:03.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:03.108 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:03.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:03.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:03.210 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:03.216 INFO:journalctl@ceph.osd.4.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[60581]: debug 2021-03-15T22:21:03.216 INFO:journalctl@ceph.osd.4.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[60581]: 2021-03-15T22:21:03.011+0000 7fe6acba5700 -1 osd.4 pg_epoch: 148 pg[7.1( v 143'25 (0'0,143'25] local-lis/les=91/92 n=2 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [4,3,6] r=0 lpr=91 crt=143'25 lcod 142'23 mlcod 142'23 active+clean+snaptrim trimq=[17~1] ps=[1b~1,1e~3,2a~1]] removing snap head 2021-03-15T22:21:03.216 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: cluster 2021-03-15T22:21:02.989570+0000 mgr.y ( 2021-03-15T22:21:03.216 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: mgr.14144) 408 : cluster [DBG] pgmap v348: 147 pgs: 1 active+clean+snaptrim, 146 active+clean; 185 MiB data, 520 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 6.6 MiB/s wr, 38 op/s 2021-03-15T22:21:03.217 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: audit 2021-03-15T22:21:02.990400+0000 mon.a (mon 2021-03-15T22:21:03.217 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: .0) 707 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-03-15T22:21:03.217 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: audit 2021-03-15T22:21: 2021-03-15T22:21:03.217 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: 03.018790+0000 mon.a (mon.0) 708 2021-03-15T22:21:03.218 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2021-03-15T22:21:03.218 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:21:03.218 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: 15T22:21:03.019051+0000 mon.a ( 2021-03-15T22:21:03.218 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: mon.0) 709 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2021-03-15T22:21:03.219 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:21:03.219 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: 21:03.040058+0000 osd.2 (osd.2) 31 : 2021-03-15T22:21:03.219 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:03 smithi125 conmon[56739]: cluster [DBG] 5.1b scrub ok 2021-03-15T22:21:03.233 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:03.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:03.234 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:03.267 INFO:journalctl@ceph.osd.1.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[68886]: debug 2021-03-15T22:21:03.022+0000 7fae34dad700 -1 osd.1 pg_epoch: 148 pg[7.b( v 148'70 (0'0,148'70] local-lis/les=91/92 n=5 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [1,5,6] r=0 lpr=91 luod=146'68 crt=148'70 lcod 146'67 mlcod 146'67 active+clean+snaptrim trimq=[17~1] ps=[1b~1,1e~3,2a~1]] removing snap head 2021-03-15T22:21:03.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: cluster 2021-03-15T22:21:02.989570+0000 mgr.y (mgr.14144) 408 2021-03-15T22:21:03.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: : cluster [DBG] pgmap v348: 147 pgs: 1 active+clean+snaptrim, 146 active+clean; 185 MiB data, 520 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 6.6 MiB/s wr, 38 op/s 2021-03-15T22:21:03.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: audit 2021-03-15T22:21:02.990400+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-03-15T22:21:03.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: audit 2021-03-15T22:21:03.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: 2021-03-15T22:21:03.018790+0000 mon.a (mon.0) 708 : audit [INF] 2021-03-15T22:21:03.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2021-03-15T22:21:03.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:21:03.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: :21:03.019051+0000 mon.a (mon.0 2021-03-15T22:21:03.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: ) 709 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2021-03-15T22:21:03.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:21:03.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: 22:21:03.040058+0000 osd.2 (osd.2) 31 : cluster [DBG] 2021-03-15T22:21:03.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[56514]: 5.1b scrub ok 2021-03-15T22:21:03.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[62156]: cluster 2021-03-15T22:21:02.989570+0000 2021-03-15T22:21:03.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[62156]: mgr.y (mgr.14144) 408 : cluster [DBG] pgmap v348: 147 pgs: 1 active+clean+snaptrim, 146 active+clean; 185 MiB data, 520 MiB used, 715 GiB / 715 GiB avail; 22 MiB/s rd, 6.6 MiB/s wr, 38 op/s 2021-03-15T22:21:03.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[62156]: audit 2021-03-15T22:21: 2021-03-15T22:21:03.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[62156]: 02.990400+0000 mon.a (mon.0) 707 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-03-15T22:21:03.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[62156]: audit 2021-03-15T22:21:03.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[62156]: 2021-03-15T22:21:03.018790+0000 mon.a (mon.0) 708 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "30"}]': finished 2021-03-15T22:21:03.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[62156]: 2021-03-15T22:21:03.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[62156]: cluster 2021-03-15T22:21 2021-03-15T22:21:03.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[62156]: :03.019051+0000 mon.a (mon.0) 709 : cluster [DBG] osdmap e148: 8 total, 8 up, 8 in 2021-03-15T22:21:03.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[62156]: cluster 2021-03-15T22:21:03. 2021-03-15T22:21:03.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[62156]: 040058+0000 osd.2 (osd.2) 31 : cluster [DBG] 5.1b scrub ok 2021-03-15T22:21:03.277 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[75764]: debug 2021-03-15T22:21:03.277 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[75764]: 2021-03-15T22:21:03.031+0000 7f9fbefb7700 -1 osd.3 pg_epoch: 148 pg[7.a( v 148'84 (0'0,148'84] local-lis/les=91/92 n=7 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [3,2,5] r=0 lpr=91 luod=146'82 crt=148'84 lcod 143'80 mlcod 143'80 active+clean+snaptrim trimq=[17~1] ps=[1b~1,1e~3,2a~1]] removing snap head 2021-03-15T22:21:03.278 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[75764]: 2021-03-15T22:21:03.278 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[65511]: debug 2021-03-15T22:21:03.279 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[65511]: 2021-03-15T22:21:03.020+0000 7f1a24881700 -1 osd.0 pg_epoch: 148 pg[7.4( v 148'78 (0'0,148'78] local-lis/les=91/92 n=4 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [0,4,1] r=0 lpr=91 luod=146'76 crt=148'78 lcod 146'74 mlcod 146'74 active+clean+snaptrim trimq=[17~1] ps=[1b~1,1e~3,2a~1]] removing snap head 2021-03-15T22:21:03.279 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[65511]: debug 2021-03-15T22:21:03.280 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[65511]: 2021-03-15T22:21:03.029+0000 7f1a1f076700 -1 osd.0 pg_epoch: 148 pg[7.f( v 148'91 (0'0,148'91] local-lis/les=91/92 n=7 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [0,3,7] r=0 lpr=91 luod=146'89 crt=148'91 lcod 146'88 mlcod 146'88 active+clean+snaptrim trimq=[17~1] ps=[1b~1,1e~3,2a~1]] removing snap head 2021-03-15T22:21:03.280 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[65511]: 2021-03-15T22:21:03.280 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[65511]: debug 2021-03-15T22:21:03.281 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[65511]: 2021-03-15T22:21:03.032+0000 7f1a2307e700 -1 osd.0 pg_epoch: 148 pg[7.f( v 148'93 (0'0,148'93] local-lis/les=91/92 n=5 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [0,3,7] r=0 lpr=91 crt=148'93 lcod 148'91 mlcod 148'91 active+clean+snaptrim trimq=[17~1] ps=[1b~1,1e~3,2a~1]] removing snap head 2021-03-15T22:21:03.281 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:03 smithi076 conmon[65511]: 2021-03-15T22:21:03.335 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:03.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:03.360 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:03.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:03.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:03.484 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:03.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:03.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:03.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:03.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:03.612 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:03.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:03.644 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:21:03.644 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:21:03.645 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 148, 2021-03-15T22:21:03.645 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:21:03.645 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2021-03-15T22:13:20.370795+0000", 2021-03-15T22:21:03.646 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2021-03-15T22:21:03.009228+0000", 2021-03-15T22:21:03.647 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2021-03-15T22:17:49.162889+0000", 2021-03-15T22:21:03.647 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2021-03-15T22:17:34.080423+0000", 2021-03-15T22:21:03.647 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-03-15T22:21:03.647 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5799936, 2021-03-15T22:21:03.648 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2021-03-15T22:21:03.648 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2021-03-15T22:21:03.648 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2021-03-15T22:21:03.648 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2021-03-15T22:21:03.649 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2021-03-15T22:21:03.649 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:03.649 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 18, 2021-03-15T22:21:03.649 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2021-03-15T22:21:03.650 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-03-15T22:21:03.650 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2021-03-15T22:21:03.650 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2021-03-15T22:21:03.650 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 7, 2021-03-15T22:21:03.651 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2021-03-15T22:21:03.651 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2021-03-15T22:21:03.651 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2021-03-15T22:21:03.652 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2021-03-15T22:21:03.652 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2021-03-15T22:21:03.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.652 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2021-03-15T22:21:03.653 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "device_health_metrics", 2021-03-15T22:21:03.653 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:16:34.817191+0000", 2021-03-15T22:21:03.653 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:03.653 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:03.654 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:03.654 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:03.654 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:03.654 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:03.655 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:03.655 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:03.655 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:03.655 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:03.656 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:03.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:03.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2021-03-15T22:21:03.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2021-03-15T22:21:03.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2021-03-15T22:21:03.657 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2021-03-15T22:21:03.657 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2021-03-15T22:21:03.657 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:03.657 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:03.658 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:03.658 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:03.658 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:03.658 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:03.659 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:03.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.659 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "23", 2021-03-15T22:21:03.659 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:03.660 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:03.660 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:03.660 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:03.660 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:03.661 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:03.661 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:03.661 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:03.661 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:03.662 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:03.662 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:03.662 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:03.662 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:03.663 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:03.663 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:03.663 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:03.663 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:03.663 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:03.664 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:03.664 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:03.664 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:03.664 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.666 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:03.666 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:03.666 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:03.667 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:03.667 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:03.667 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:03.667 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:03.667 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:03.668 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:03.668 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:03.668 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:03.668 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:21:03.669 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2021-03-15T22:21:03.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.669 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:03.669 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_devicehealth": {} 2021-03-15T22:21:03.669 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.670 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2021-03-15T22:21:03.670 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "iscsi", 2021-03-15T22:21:03.671 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:49.906400+0000", 2021-03-15T22:21:03.671 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:03.671 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:03.671 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:03.672 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:03.672 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:03.672 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:03.672 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:03.672 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:03.673 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:03.673 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:03.673 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:03.673 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:03.673 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 3, 2021-03-15T22:21:03.674 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 3, 2021-03-15T22:21:03.674 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 3, 2021-03-15T22:21:03.674 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 3, 2021-03-15T22:21:03.674 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 3, 2021-03-15T22:21:03.675 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:03.675 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:03.675 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:03.675 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:03.675 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:03.676 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:03.676 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:03.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.676 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "56", 2021-03-15T22:21:03.677 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:03.677 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:03.677 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:03.677 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:03.678 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:03.678 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:03.678 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:03.678 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:03.679 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:03.679 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:03.679 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:03.679 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:03.680 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:03.680 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:03.680 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:03.680 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:03.681 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:03.681 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:03.681 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:03.681 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:03.681 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:03.682 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:03.682 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:03.682 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:03.682 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:03.683 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:03.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.683 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:03.683 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:03.683 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:03.684 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:03.684 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:03.684 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:03.684 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:03.684 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:03.685 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:03.685 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:03.685 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:03.685 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:21:03.686 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:03.686 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2021-03-15T22:21:03.686 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.687 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2021-03-15T22:21:03.687 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".rgw.root", 2021-03-15T22:21:03.687 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:52.257970+0000", 2021-03-15T22:21:03.687 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:03.688 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:03.688 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:03.688 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:03.688 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:03.688 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:03.689 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:03.689 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:03.689 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:03.689 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:03.690 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:03.690 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:03.690 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:21:03.690 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:21:03.690 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:21:03.691 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:21:03.691 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:21:03.691 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:03.691 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:03.691 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:03.692 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:03.692 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:03.692 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:03.692 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:03.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.693 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "58", 2021-03-15T22:21:03.693 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:03.693 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:03.693 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:03.694 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:03.694 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:03.694 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:03.694 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:03.695 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:03.695 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:03.695 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:03.695 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:03.695 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:03.696 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:03.696 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:03.696 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:03.696 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:03.696 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:03.697 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:03.697 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:03.697 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:03.698 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:03.698 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:03.699 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:03.700 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:03.700 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:03.700 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:03.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.701 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:03.701 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:03.701 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:03.702 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:03.702 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:03.702 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:03.702 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:03.703 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:03.703 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:03.703 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:03.703 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:03.704 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:21:03.704 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:03.704 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:21:03.704 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.705 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2021-03-15T22:21:03.705 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.log", 2021-03-15T22:21:03.705 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:54.562498+0000", 2021-03-15T22:21:03.706 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:03.706 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:03.706 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:03.706 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:03.707 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:03.707 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:03.707 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:03.707 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:03.707 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:03.708 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:03.708 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:03.708 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:03.708 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:21:03.709 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:21:03.709 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:21:03.709 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:21:03.709 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:21:03.709 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "60", 2021-03-15T22:21:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:21:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:21:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 5, 2021-03-15T22:21:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.control", 2021-03-15T22:21:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:56.243806+0000", 2021-03-15T22:21:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 31, 2021-03-15T22:21:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 22, 2021-03-15T22:21:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 22, 2021-03-15T22:21:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 22, 2021-03-15T22:21:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 30, 2021-03-15T22:21:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "5.1f", 2021-03-15T22:21:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 146, 2021-03-15T22:21:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 145, 2021-03-15T22:21:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 145, 2021-03-15T22:21:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "142'3", 2021-03-15T22:21:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "148", 2021-03-15T22:21:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "148", 2021-03-15T22:21:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "148", 2021-03-15T22:21:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:21:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:21:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 6, 2021-03-15T22:21:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.meta", 2021-03-15T22:21:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:58.730534+0000", 2021-03-15T22:21:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:21:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:21:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:21:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:21:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:21:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "109", 2021-03-15T22:21:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:21:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_bias": 4, 2021-03-15T22:21:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 8 2021-03-15T22:21:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:21:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:21:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2021-03-15T22:21:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:20:03.377294+0000", 2021-03-15T22:21:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2021-03-15T22:21:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-03-15T22:21:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2021-03-15T22:21:03.760 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2021-03-15T22:21:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2021-03-15T22:21:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2021-03-15T22:21:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2021-03-15T22:21:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:03.761 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:03.762 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "148", 2021-03-15T22:21:03.763 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:03.764 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:03.765 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 51, 2021-03-15T22:21:03.766 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 148, 2021-03-15T22:21:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:03.767 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:03.768 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:03.769 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:03.770 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:03.771 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:03.772 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:03.773 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:03.774 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:21:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2021-03-15T22:21:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.775 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:21:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:21:03.776 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd", 2021-03-15T22:21:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:03.777 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:03.778 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 8, 2021-03-15T22:21:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 91, 2021-03-15T22:21:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:03.779 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.780 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2021-03-15T22:21:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.781 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2021-03-15T22:21:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.783 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2021-03-15T22:21:03.784 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.785 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2021-03-15T22:21:03.787 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.788 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2021-03-15T22:21:03.789 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.790 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2021-03-15T22:21:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.791 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2021-03-15T22:21:03.792 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.793 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2021-03-15T22:21:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.794 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/188228176", 2021-03-15T22:21:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/188228176", 2021-03-15T22:21:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/188228176", 2021-03-15T22:21:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/188228176", 2021-03-15T22:21:03.795 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:21:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526", 2021-03-15T22:21:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:03.797 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:03.798 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 13, 2021-03-15T22:21:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 140, 2021-03-15T22:21:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:03.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2021-03-15T22:21:03.800 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2021-03-15T22:21:03.801 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.802 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2021-03-15T22:21:03.803 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2021-03-15T22:21:03.804 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.805 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2021-03-15T22:21:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.807 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2021-03-15T22:21:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.809 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2021-03-15T22:21:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.810 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2021-03-15T22:21:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.811 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/3924955423", 2021-03-15T22:21:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/3924955423", 2021-03-15T22:21:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/3924955423", 2021-03-15T22:21:03.812 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/3924955423", 2021-03-15T22:21:03.814 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:21:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05", 2021-03-15T22:21:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:03.816 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:03.817 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 18, 2021-03-15T22:21:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 140, 2021-03-15T22:21:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:03.818 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.819 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2021-03-15T22:21:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2021-03-15T22:21:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.822 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2021-03-15T22:21:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2021-03-15T22:21:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:03.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.825 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.825 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:03.825 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.826 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.826 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2021-03-15T22:21:03.826 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:03.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.827 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.827 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2021-03-15T22:21:03.828 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:03.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.828 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.828 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:03.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2021-03-15T22:21:03.830 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:03.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.830 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2021-03-15T22:21:03.831 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:03.831 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.831 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.831 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/1401263050", 2021-03-15T22:21:03.832 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/1401263050", 2021-03-15T22:21:03.832 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/1401263050", 2021-03-15T22:21:03.832 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/1401263050", 2021-03-15T22:21:03.832 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:03.833 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:03.833 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:03.833 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.834 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:21:03.834 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56", 2021-03-15T22:21:03.835 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:03.835 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:03.835 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:03.836 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:03.836 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:03.836 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:03.836 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 26, 2021-03-15T22:21:03.837 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 91, 2021-03-15T22:21:03.838 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:03.838 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:03.839 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:03.839 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.840 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2021-03-15T22:21:03.841 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:03.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.842 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2021-03-15T22:21:03.842 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:03.842 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.843 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.846 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:03.847 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.847 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.847 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2021-03-15T22:21:03.847 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:03.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.848 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.848 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2021-03-15T22:21:03.849 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:03.849 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.849 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.849 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:03.850 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.851 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.851 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2021-03-15T22:21:03.851 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:03.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.853 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2021-03-15T22:21:03.853 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:03.853 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.853 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.854 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:03.854 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.855 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2021-03-15T22:21:03.855 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:03.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.856 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.856 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2021-03-15T22:21:03.857 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:03.857 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.857 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.858 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/3960968636", 2021-03-15T22:21:03.858 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/3960968636", 2021-03-15T22:21:03.858 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/3960968636", 2021-03-15T22:21:03.858 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/3960968636", 2021-03-15T22:21:03.859 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:03.859 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:03.859 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:03.860 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.860 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:21:03.861 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237", 2021-03-15T22:21:03.861 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:03.861 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:03.861 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:03.861 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:03.862 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:03.862 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:03.863 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 31, 2021-03-15T22:21:03.863 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 140, 2021-03-15T22:21:03.864 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:03.864 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:03.865 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:03.865 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.866 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.867 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6800", 2021-03-15T22:21:03.867 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:03.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6801", 2021-03-15T22:21:03.869 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:03.870 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.870 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.871 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:03.872 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.873 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6802", 2021-03-15T22:21:03.874 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:03.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.875 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.875 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6803", 2021-03-15T22:21:03.876 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:03.876 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.877 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.878 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:03.878 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.880 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6806", 2021-03-15T22:21:03.880 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:03.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.882 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6807", 2021-03-15T22:21:03.883 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:03.883 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.883 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.884 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:03.884 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.884 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.885 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6804", 2021-03-15T22:21:03.885 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:03.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.885 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.886 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6805", 2021-03-15T22:21:03.886 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:03.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.886 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.887 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6801/37386122", 2021-03-15T22:21:03.887 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6803/37386122", 2021-03-15T22:21:03.887 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6807/37386122", 2021-03-15T22:21:03.887 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6805/37386122", 2021-03-15T22:21:03.888 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:03.888 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:03.888 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:03.888 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.889 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:21:03.889 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2", 2021-03-15T22:21:03.889 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:03.890 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:03.890 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:03.890 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:03.890 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:03.891 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:03.891 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 38, 2021-03-15T22:21:03.891 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 146, 2021-03-15T22:21:03.891 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:03.891 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:03.892 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:03.892 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.892 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.892 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6808", 2021-03-15T22:21:03.893 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:03.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.893 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.894 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6809", 2021-03-15T22:21:03.894 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:03.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.894 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.895 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:03.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6810", 2021-03-15T22:21:03.896 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:03.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.897 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6811", 2021-03-15T22:21:03.897 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:03.897 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.897 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.898 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:03.898 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.899 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6814", 2021-03-15T22:21:03.899 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:03.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.900 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.900 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6815", 2021-03-15T22:21:03.900 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:03.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.900 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.901 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:03.901 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.901 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.902 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6812", 2021-03-15T22:21:03.902 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:03.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.903 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.904 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6813", 2021-03-15T22:21:03.904 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:03.905 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.905 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.905 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6809/3610838738", 2021-03-15T22:21:03.906 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6811/3610838738", 2021-03-15T22:21:03.906 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6815/3610838738", 2021-03-15T22:21:03.906 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6813/3610838738", 2021-03-15T22:21:03.906 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:03.906 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:03.907 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:03.907 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.908 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:21:03.908 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe", 2021-03-15T22:21:03.908 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:03.908 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:03.908 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:03.909 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:03.909 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:03.909 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:03.909 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 45, 2021-03-15T22:21:03.909 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 91, 2021-03-15T22:21:03.910 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:03.910 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:03.910 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:03.910 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.911 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.911 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6816", 2021-03-15T22:21:03.911 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:03.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.912 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.912 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6817", 2021-03-15T22:21:03.912 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:03.912 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.913 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.913 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:03.913 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.914 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.914 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6818", 2021-03-15T22:21:03.914 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:03.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.915 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6819", 2021-03-15T22:21:03.915 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:03.915 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.916 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.916 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:03.916 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.917 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.917 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6822", 2021-03-15T22:21:03.917 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:03.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.918 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6823", 2021-03-15T22:21:03.918 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:03.918 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.919 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.919 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:03.919 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.920 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.920 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6820", 2021-03-15T22:21:03.920 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:03.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.921 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6821", 2021-03-15T22:21:03.921 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:03.921 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.922 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.922 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6817/171378275", 2021-03-15T22:21:03.922 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6819/171378275", 2021-03-15T22:21:03.923 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6823/171378275", 2021-03-15T22:21:03.923 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6821/171378275", 2021-03-15T22:21:03.923 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:03.923 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:03.923 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:03.924 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.924 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:21:03.925 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614", 2021-03-15T22:21:03.925 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:03.925 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:03.925 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:03.926 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:03.926 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:03.926 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:03.926 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 52, 2021-03-15T22:21:03.927 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 140, 2021-03-15T22:21:03.927 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:03.927 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:03.927 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:03.927 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.928 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.928 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6824", 2021-03-15T22:21:03.928 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:03.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.929 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.929 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6825", 2021-03-15T22:21:03.929 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:03.930 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.930 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.930 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:03.930 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.931 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.931 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6826", 2021-03-15T22:21:03.931 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:03.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.932 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.932 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6827", 2021-03-15T22:21:03.932 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:03.933 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.933 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.933 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:03.933 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.934 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.934 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6830", 2021-03-15T22:21:03.934 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:03.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.935 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.935 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6831", 2021-03-15T22:21:03.935 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:03.936 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.936 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.936 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:03.936 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:03.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:03.937 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6828", 2021-03-15T22:21:03.937 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:03.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:03.938 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6829", 2021-03-15T22:21:03.939 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:03.939 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.939 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.939 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6825/652003948", 2021-03-15T22:21:03.940 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6827/652003948", 2021-03-15T22:21:03.940 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6831/652003948", 2021-03-15T22:21:03.940 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6829/652003948", 2021-03-15T22:21:03.940 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:03.940 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:03.941 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:03.941 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.941 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.941 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:03.942 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2021-03-15T22:21:03.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.942 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:21:03.942 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:03.943 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:03.943 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:03.943 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:03.943 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:03.943 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:51.037646+0000", 2021-03-15T22:21:03.944 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:03.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.944 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:21:03.945 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:03.945 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:03.945 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:03.945 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:03.945 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:03.946 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:08.401024+0000", 2021-03-15T22:21:03.946 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:03.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.946 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:21:03.947 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:03.947 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:03.947 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:03.947 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:03.947 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:03.948 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:24.332975+0000", 2021-03-15T22:21:03.948 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:03.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.949 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:21:03.949 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:03.949 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:03.949 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:03.949 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:03.950 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:03.950 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:39.289836+0000", 2021-03-15T22:21:03.950 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:03.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.951 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:21:03.951 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:03.951 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:03.951 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:03.952 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:03.952 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:03.952 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:57.378352+0000", 2021-03-15T22:21:03.952 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:03.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.953 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:21:03.953 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:03.953 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:03.953 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:03.954 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:03.954 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:03.954 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:14.686550+0000", 2021-03-15T22:21:03.954 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:03.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.955 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:21:03.955 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:03.955 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:03.956 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:03.956 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:03.956 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:03.956 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:31.843914+0000", 2021-03-15T22:21:03.956 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:03.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.957 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:21:03.957 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:03.958 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:03.958 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:03.958 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:03.958 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:03.958 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:50.252442+0000", 2021-03-15T22:21:03.959 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:03.959 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.959 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:03.959 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [], 2021-03-15T22:21:03.959 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2021-03-15T22:21:03.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.960 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.b", 2021-03-15T22:21:03.960 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:21:03.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.960 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:21:03.961 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:21:03.961 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.961 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.962 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2021-03-15T22:21:03.962 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:21:03.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.962 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:21:03.963 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:21:03.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.963 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.964 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.1b", 2021-03-15T22:21:03.964 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:21:03.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.964 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:21:03.964 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 5 2021-03-15T22:21:03.965 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.965 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.966 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.966 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:03.966 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2021-03-15T22:21:03.966 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2021-03-15T22:21:03.967 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": { 2021-03-15T22:21:03.967 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:21:03.967 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4087324200": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:21:03.967 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4141143304": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:21:03.967 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:21:03.968 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2779879991": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:21:03.968 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:21:03.968 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:21:03.968 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2483102204": "2021-03-16T22:13:41.603652+0000" 2021-03-15T22:21:03.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.969 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2021-03-15T22:21:03.969 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2021-03-15T22:21:03.969 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2021-03-15T22:21:03.969 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2021-03-15T22:21:03.970 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2021-03-15T22:21:03.970 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2021-03-15T22:21:03.970 INFO:teuthology.orchestra.run.smithi076.stdout: "ruleset-failure-domain": "osd", 2021-03-15T22:21:03.970 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2021-03-15T22:21:03.970 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.971 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2021-03-15T22:21:03.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.971 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:21:03.971 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:21:03.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.972 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 23, 2021-03-15T22:21:03.972 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:21:03.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.973 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 27, 2021-03-15T22:21:03.973 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:21:03.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.973 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 30, 2021-03-15T22:21:03.974 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 3 2021-03-15T22:21:03.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:03.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.974 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 42, 2021-03-15T22:21:03.975 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:21:03.975 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.975 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.975 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.975 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:03.976 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [ 2021-03-15T22:21:03.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.976 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:21:03.976 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:21:03.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:03.977 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 23, 2021-03-15T22:21:03.977 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:21:03.977 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.977 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:03.977 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.978 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:03.978 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2021-03-15T22:21:03.978 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2021-03-15T22:21:03.978 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2021-03-15T22:21:03.979 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2021-03-15T22:21:03.979 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2021-03-15T22:21:03.979 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2021-03-15T22:21:03.979 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2021-03-15T22:21:03.979 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2021-03-15T22:21:03.980 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2021-03-15T22:21:03.980 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:03.980 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:21:03.980 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:04.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:04.004 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:04.004 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:04.018 INFO:tasks.rados.rados.0.smithi076.stdout:285: done (2 left) 2021-03-15T22:21:04.019 INFO:tasks.rados.rados.0.smithi076.stdout:288: done (1 left) 2021-03-15T22:21:04.019 INFO:tasks.rados.rados.0.smithi076.stdout:289: done (0 left) 2021-03-15T22:21:04.019 INFO:tasks.rados.rados.0.smithi076.stdout:290: read oid 39 snap 25 2021-03-15T22:21:04.020 INFO:tasks.rados.rados.0.smithi076.stdout:290: expect (ObjNum 116 snap 24 seq_num 116) 2021-03-15T22:21:04.020 INFO:tasks.rados.rados.0.smithi076.stdout:291: read oid 38 snap -1 2021-03-15T22:21:04.020 INFO:tasks.rados.rados.0.smithi076.stdout:291: expect (ObjNum 120 snap 24 seq_num 120) 2021-03-15T22:21:04.021 INFO:tasks.rados.rados.0.smithi076.stdout:292: copy_from oid 3 from oid 30 current snap is 27 2021-03-15T22:21:04.021 INFO:tasks.rados.rados.0.smithi076.stdout:293: write (excl) oid 35 current snap is 27 2021-03-15T22:21:04.021 INFO:tasks.rados.rados.0.smithi076.stdout:293: seq_num 138 ranges {719366=478716,1808868=510498} 2021-03-15T22:21:04.025 INFO:tasks.rados.rados.0.smithi076.stdout:293: writing smithi07694878-35 from 719366 to 1198082 tid 1 2021-03-15T22:21:04.033 INFO:tasks.rados.rados.0.smithi076.stdout:293: writing smithi07694878-35 from 1808868 to 2319366 tid 2 2021-03-15T22:21:04.033 INFO:tasks.rados.rados.0.smithi076.stdout:294: read oid 4 snap -1 2021-03-15T22:21:04.034 INFO:tasks.rados.rados.0.smithi076.stdout:294: expect (ObjNum 121 snap 24 seq_num 121) 2021-03-15T22:21:04.034 INFO:tasks.rados.rados.0.smithi076.stdout:295: rollback oid 42 current snap is 27 2021-03-15T22:21:04.034 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 42 to 24 2021-03-15T22:21:04.034 INFO:tasks.rados.rados.0.smithi076.stdout:296: copy_from oid 15 from oid 18 current snap is 27 2021-03-15T22:21:04.035 INFO:tasks.rados.rados.0.smithi076.stdout:297: delete oid 49 current snap is 27 2021-03-15T22:21:04.040 INFO:tasks.rados.rados.0.smithi076.stdout:290: expect (ObjNum 116 snap 24 seq_num 116) 2021-03-15T22:21:04.105 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:04.128 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:04.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:04.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:04.167 INFO:tasks.rados.rados.0.smithi076.stdout:290: done (7 left) 2021-03-15T22:21:04.167 INFO:tasks.rados.rados.0.smithi076.stdout:297: done (6 left) 2021-03-15T22:21:04.168 INFO:tasks.rados.rados.0.smithi076.stdout:298: snap_create 2021-03-15T22:21:04.168 INFO:tasks.rados.rados.0.smithi076.stdout:291: expect (ObjNum 120 snap 24 seq_num 120) 2021-03-15T22:21:04.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:04.252 INFO:tasks.rados.rados.0.smithi076.stdout:296: finishing copy_from to smithi07694878-15 2021-03-15T22:21:04.252 INFO:tasks.rados.rados.0.smithi076.stdout:296: got expected ENOENT (src dne) 2021-03-15T22:21:04.252 INFO:tasks.rados.rados.0.smithi076.stdout:296: finishing copy_from racing read to smithi07694878-15 2021-03-15T22:21:04.253 INFO:tasks.rados.rados.0.smithi076.stdout:295: finishing rollback tid 0 to smithi07694878-42 2021-03-15T22:21:04.253 INFO:tasks.rados.rados.0.smithi076.stdout:295: finishing rollback tid 1 to smithi07694878-42 2021-03-15T22:21:04.253 INFO:tasks.rados.rados.0.smithi076.stdout:295: finishing rollback tid 2 to smithi07694878-42 2021-03-15T22:21:04.253 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 175 (ObjNum 125 snap 24 seq_num 125) dirty exists 2021-03-15T22:21:04.253 INFO:tasks.rados.rados.0.smithi076.stdout:294: expect (ObjNum 121 snap 24 seq_num 121) 2021-03-15T22:21:04.254 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:04.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:04.256 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:04.341 INFO:tasks.rados.rados.0.smithi076.stdout:293: finishing write tid 1 to smithi07694878-35 2021-03-15T22:21:04.342 INFO:tasks.rados.rados.0.smithi076.stdout:292: finishing copy_from to smithi07694878-3 2021-03-15T22:21:04.342 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 127 (ObjNum 113 snap 24 seq_num 113) dirty exists 2021-03-15T22:21:04.342 INFO:tasks.rados.rados.0.smithi076.stdout:292: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:21:04.343 INFO:tasks.rados.rados.0.smithi076.stdout:293: finishing write tid 2 to smithi07694878-35 2021-03-15T22:21:04.343 INFO:tasks.rados.rados.0.smithi076.stdout:293: finishing write tid 3 to smithi07694878-35 2021-03-15T22:21:04.343 INFO:tasks.rados.rados.0.smithi076.stdout:293: finishing write tid 4 to smithi07694878-35 2021-03-15T22:21:04.343 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 92 (ObjNum 138 snap 27 seq_num 138) dirty exists 2021-03-15T22:21:04.344 INFO:tasks.rados.rados.0.smithi076.stdout:293: left oid 35 (ObjNum 138 snap 27 seq_num 138) 2021-03-15T22:21:04.356 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:04.379 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:04.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:04.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:04.397 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:04 smithi125 conmon[56739]: cluster 2021-03-15T22:21:02.904674+0000 osd.3 (osd.3) 42 : cluster [DBG] 6.c scrub ok 2021-03-15T22:21:04.397 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:04 smithi125 conmon[56739]: cluster 2021-03-15T22:21:03.144802+0000 osd.5 (osd.5) 34 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:21:04.397 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:04 smithi125 conmon[56739]: cluster 2021-03-15T22:21:03.362017+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:21:04.398 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:04 smithi125 conmon[56739]: audit 2021-03-15T22:21:03.640409+0000 mon.a (mon.0) 710 : audit [DBG] from='client.? 172.21.15.76:0/1126586842' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:21:04.398 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:04 smithi125 conmon[56739]: cluster 2021-03-15T22:21:04.017066+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2021-03-15T22:21:04.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[56514]: cluster 2021-03-15T22:21:02.904674+0000 osd.3 (osd.3) 42 : cluster [DBG] 6.c scrub ok 2021-03-15T22:21:04.436 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[56514]: cluster 2021-03-15T22:21:03.144802+0000 osd.5 (osd.5) 34 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:21:04.436 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[56514]: cluster 2021-03-15T22:21:03.362017+0000 osd.4 (osd.4) 40 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:21:04.436 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[56514]: audit 2021-03-15T22:21:03.640409+0000 mon.a (mon.0) 710 : audit [DBG] from='client.? 172.21.15.76:0/1126586842' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:21:04.437 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[56514]: cluster 2021-03-15T22:21:04.017066+0000 mon.a (mon.0) 711 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2021-03-15T22:21:04.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: cluster 2021-03-15T22:21:02.904674+0000 2021-03-15T22:21:04.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: osd.3 (osd.3) 42 : cluster [DBG] 6.c scrub ok 2021-03-15T22:21:04.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: cluster 2021-03-15T22:21:04.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: 2021-03-15T22:21:03.144802 2021-03-15T22:21:04.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 2021-03-15T22:21:04.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: 34 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:21:04.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: cluster 2021-03-15T22:21:04.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: 2021-03-15T22:21: 2021-03-15T22:21:04.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: 03.362017+0000 osd.4 (osd.4) 40 : cluster 2021-03-15T22:21:04.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: [DBG] 4.1 scrub ok 2021-03-15T22:21:04.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: audit 2021-03-15T22:21:04.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: 2021-03-15T22:21 2021-03-15T22:21:04.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: :03.640409+0000 mon.a (mon 2021-03-15T22:21:04.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: .0) 710 : audit [DBG] 2021-03-15T22:21:04.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/1126586842' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:21:04.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:04.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: 03-15T22:21:04 2021-03-15T22:21:04.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: .017066+0000 mon.a (mon. 2021-03-15T22:21:04.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: 0) 711 : cluster [DBG] osdmap e149: 8 total, 8 up, 8 in 2021-03-15T22:21:04.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:04 smithi076 conmon[62156]: 2021-03-15T22:21:04.457 INFO:tasks.thrashosds.thrasher:j is {'epoch': 148, 'fsid': '8a572802-85db-11eb-9091-001a4aab830c', 'created': '2021-03-15T22:13:20.370795+0000', 'modified': '2021-03-15T22:21:03.009228+0000', 'last_up_change': '2021-03-15T22:17:49.162889+0000', 'last_in_change': '2021-03-15T22:17:34.080423+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-15T22:16:34.817191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-15T22:17:49.906400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-15T22:17:52.257970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-15T22:17:54.562498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-15T22:17:56.243806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 31, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 30, 'last_pg_merge_meta': {'source_pgid': '5.1f', 'ready_epoch': 146, 'last_epoch_started': 145, 'last_epoch_clean': 145, 'source_version': "142'3", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '148', 'last_force_op_resend_preluminous': '148', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-15T22:17:58.730534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-03-15T22:20:03.377294+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 51, 'snap_epoch': 148, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 188228176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 188228176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 188228176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 188228176}]}, 'public_addr': '172.21.15.76:6803/188228176', 'cluster_addr': '172.21.15.76:6805/188228176', 'heartbeat_back_addr': '172.21.15.76:6809/188228176', 'heartbeat_front_addr': '172.21.15.76:6807/188228176', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '97bd3ffc-15c8-4a6f-b37a-217813def526', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 3924955423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 3924955423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 3924955423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 3924955423}]}, 'public_addr': '172.21.15.76:6811/3924955423', 'cluster_addr': '172.21.15.76:6813/3924955423', 'heartbeat_back_addr': '172.21.15.76:6817/3924955423', 'heartbeat_front_addr': '172.21.15.76:6815/3924955423', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a7f5f115-e69e-47d9-b6c3-ff386ddfaa05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 1401263050}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 1401263050}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 1401263050}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 1401263050}]}, 'public_addr': '172.21.15.76:6819/1401263050', 'cluster_addr': '172.21.15.76:6821/1401263050', 'heartbeat_back_addr': '172.21.15.76:6825/1401263050', 'heartbeat_front_addr': '172.21.15.76:6823/1401263050', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a0bb7f88-0114-47cb-a82c-d8fb002aac56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 3960968636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 3960968636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 3960968636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 3960968636}]}, 'public_addr': '172.21.15.76:6827/3960968636', 'cluster_addr': '172.21.15.76:6829/3960968636', 'heartbeat_back_addr': '172.21.15.76:6833/3960968636', 'heartbeat_front_addr': '172.21.15.76:6831/3960968636', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b650611c-6101-47e8-9ca9-94e02cf8b237', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6800', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6801', 'nonce': 37386122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6802', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6803', 'nonce': 37386122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6806', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6807', 'nonce': 37386122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6804', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6805', 'nonce': 37386122}]}, 'public_addr': '172.21.15.125:6801/37386122', 'cluster_addr': '172.21.15.125:6803/37386122', 'heartbeat_back_addr': '172.21.15.125:6807/37386122', 'heartbeat_front_addr': '172.21.15.125:6805/37386122', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7d6ceff2-7093-4ab9-8b96-ade448093bf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6808', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6809', 'nonce': 3610838738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6810', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6811', 'nonce': 3610838738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6814', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6815', 'nonce': 3610838738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6812', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6813', 'nonce': 3610838738}]}, 'public_addr': '172.21.15.125:6809/3610838738', 'cluster_addr': '172.21.15.125:6811/3610838738', 'heartbeat_back_addr': '172.21.15.125:6815/3610838738', 'heartbeat_front_addr': '172.21.15.125:6813/3610838738', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98b72022-a9a7-4ee2-89a1-cac037de7afe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6816', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6817', 'nonce': 171378275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6818', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6819', 'nonce': 171378275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6822', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6823', 'nonce': 171378275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6820', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6821', 'nonce': 171378275}]}, 'public_addr': '172.21.15.125:6817/171378275', 'cluster_addr': '172.21.15.125:6819/171378275', 'heartbeat_back_addr': '172.21.15.125:6823/171378275', 'heartbeat_front_addr': '172.21.15.125:6821/171378275', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f99ea650-b2dc-4d87-a44c-56a897edd614', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6824', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6825', 'nonce': 652003948}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6826', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6827', 'nonce': 652003948}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6830', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6831', 'nonce': 652003948}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6828', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6829', 'nonce': 652003948}]}, 'public_addr': '172.21.15.125:6825/652003948', 'cluster_addr': '172.21.15.125:6827/652003948', 'heartbeat_back_addr': '172.21.15.125:6831/652003948', 'heartbeat_front_addr': '172.21.15.125:6829/652003948', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:51.037646+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:08.401024+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:24.332975+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:39.289836+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:57.378352+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:14.686550+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:31.843914+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:50.252442+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.76:6800/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4087324200': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4141143304': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:6801/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/2779879991': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6801/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6800/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:0/2483102204': '2021-03-16T22:13:41.603652+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 23, 'length': 1}, {'begin': 27, 'length': 1}, {'begin': 30, 'length': 3}, {'begin': 42, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 23, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-03-15T22:21:04.458 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2021-03-15T22:21:04.482 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:04.504 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:04.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:04.505 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:04.544 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:21:04.569 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:21:04.570 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:21:04.570 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:21:04.570 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:21:04.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:04.632 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:04.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:04.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:04.735 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:04.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:04.758 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:04.759 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:04.860 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:04.882 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:04.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:04.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:04.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:05.006 INFO:tasks.rados.rados.0.smithi076.stdout:291: done (6 left) 2021-03-15T22:21:05.007 INFO:tasks.rados.rados.0.smithi076.stdout:292: done (5 left) 2021-03-15T22:21:05.007 INFO:tasks.rados.rados.0.smithi076.stdout:293: done (4 left) 2021-03-15T22:21:05.008 INFO:tasks.rados.rados.0.smithi076.stdout:294: done (3 left) 2021-03-15T22:21:05.008 INFO:tasks.rados.rados.0.smithi076.stdout:295: done (2 left) 2021-03-15T22:21:05.009 INFO:tasks.rados.rados.0.smithi076.stdout:296: done (1 left) 2021-03-15T22:21:05.009 INFO:tasks.rados.rados.0.smithi076.stdout:298: done (0 left) 2021-03-15T22:21:05.010 INFO:tasks.rados.rados.0.smithi076.stdout:299: snap_remove snap 27 2021-03-15T22:21:05.012 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:05.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:05.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:05.115 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:05.137 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:05.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:05.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:05.221 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-15T22:21:05.224 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:05.241 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:05.266 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:05.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:05.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:05.368 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:05.390 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:05.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:05.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:05.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[56514]: cluster 2021-03-15T22:21:03.982274+0000 osd.7 (osd.7) 39 : cluster 2021-03-15T22:21:05.427 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[56514]: [DBG] 6.1c scrub ok 2021-03-15T22:21:05.427 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[56514]: cluster 2021-03-15T22:21:04.375638+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.c scrub ok 2021-03-15T22:21:05.427 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[56514]: cluster 2021-03-15T22:21:04.990086+0000 mgr.y (mgr.14144) 409 : cluster [DBG] pgmap v351: 147 pgs: 1 active+clean+snaptrim, 146 active+clean; 176 MiB data, 491 MiB used, 715 GiB / 715 GiB avail; 9.1 MiB/s rd, 5.8 MiB/s wr, 29 op/s 2021-03-15T22:21:05.427 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[56514]: cluster 2021-03-15T22:21:05.004427+0000 mon.a (mon.0) 712 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2021-03-15T22:21:05.428 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[56514]: cluster 2021-03-15T22:21:05.101152+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.16 scrub ok 2021-03-15T22:21:05.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: cluster 2021-03-15T22:21:03.982274+0000 2021-03-15T22:21:05.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: osd.7 (osd.7) 39 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:21:05.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:21:05.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: :21:04.375638+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.c scrub ok 2021-03-15T22:21:05.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: 2021-03-15T22:21:05.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: cluster 2021-03-15T22:21: 2021-03-15T22:21:05.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: 04.990086+0000 mgr.y (mgr.14144) 409 2021-03-15T22:21:05.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: : cluster [DBG] pgmap v351: 147 pgs: 1 active+clean+snaptrim, 146 active+clean; 176 MiB data, 491 MiB used, 715 GiB / 715 GiB avail; 9.1 MiB/s rd, 5.8 MiB/s wr, 29 op/s 2021-03-15T22:21:05.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:21:05.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: 15T22:21:05.004427+0000 mon.a 2021-03-15T22:21:05.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: (mon.0) 712 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2021-03-15T22:21:05.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: 2021-03-15T22:21:05.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: cluster 2021-03-15T22:21:05.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: 2021-03-15T22:21:05.101152+0000 2021-03-15T22:21:05.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: osd.2 (osd.2) 32 : cluster [DBG] 5.16 scrub ok 2021-03-15T22:21:05.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:05 smithi076 conmon[62156]: 2021-03-15T22:21:05.492 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:05.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:05.515 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:05.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:05.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:05.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:05 smithi125 conmon[56739]: cluster 2021-03-15T22:21:03.982274+0000 osd.7 (osd.7 2021-03-15T22:21:05.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:05 smithi125 conmon[56739]: ) 39 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:21:05.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:05 smithi125 conmon[56739]: cluster 2021-03-15T22:21:04.375638+0000 osd.4 (osd.4) 41 : cluster [DBG] 4.c scrub ok 2021-03-15T22:21:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:05 smithi125 conmon[56739]: cluster 2021-03-15T22:21:04.990086+0000 mgr.y (mgr.14144) 409 : cluster [DBG] pgmap v351: 147 pgs: 1 active+clean+snaptrim, 146 active+clean; 176 MiB data, 491 MiB used, 715 GiB / 715 GiB avail; 9.1 MiB/s rd, 5.8 MiB/s wr, 29 op/s 2021-03-15T22:21:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:05 smithi125 conmon[56739]: cluster 2021-03-15T22:21:05.004427+0000 mon.a (mon.0) 712 : cluster [DBG] osdmap e150: 8 total, 8 up, 8 in 2021-03-15T22:21:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:05 smithi125 conmon[56739]: cluster 2021-03-15T22:21:05.101152+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.16 scrub ok 2021-03-15T22:21:05.618 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:05.642 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:05.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:05.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:05.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:05.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:05.769 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:05.769 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:05.871 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:05.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:05.894 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:05.894 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:05.995 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:06.016 INFO:tasks.rados.rados.0.smithi076.stdout:299: done (0 left) 2021-03-15T22:21:06.017 INFO:tasks.rados.rados.0.smithi076.stdout:300: delete oid 20 current snap is 28 2021-03-15T22:21:06.021 INFO:tasks.rados.rados.0.smithi076.stdout:300: done (0 left) 2021-03-15T22:21:06.022 INFO:tasks.rados.rados.0.smithi076.stdout:301: snap_remove snap 25 2021-03-15T22:21:06.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:06.023 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:06.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:06.125 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:06.148 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:06.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:06.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:06.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:06.272 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:06.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:06.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:06.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:06 smithi125 conmon[56739]: cluster 2021-03-15T22:21:04.762287+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.7 deep-scrub ok 2021-03-15T22:21:06.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:06 smithi125 conmon[56739]: cluster 2021-03-15T22:21:04.885369+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:21:06.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:06 smithi125 conmon[56739]: cluster 2021-03-15T22:21:05.103657+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.17 scrub ok 2021-03-15T22:21:06.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:06 smithi125 conmon[56739]: cluster 2021-03-15T22:21:05.736766+0000 osd.0 (osd.0) 36 : cluster [DBG] 3.17 deep-scrub ok 2021-03-15T22:21:06.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:06 smithi125 conmon[56739]: cluster 2021-03-15T22:21:06.013027+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2021-03-15T22:21:06.375 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:06.397 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:06.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:06.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:06.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:06.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:06 smithi076 conmon[62156]: cluster 2021-03-15T22:21:04.762287+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.7 deep-scrub ok 2021-03-15T22:21:06.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:06 smithi076 conmon[62156]: cluster 2021-03-15T22:21:04.885369+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:21:06.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:06 smithi076 conmon[62156]: cluster 2021-03-15T22:21:05.103657+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.17 scrub ok 2021-03-15T22:21:06.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:06 smithi076 conmon[62156]: cluster 2021-03-15T22:21:05.736766+0000 osd.0 (osd.0) 36 : cluster [DBG] 3.17 deep-scrub ok 2021-03-15T22:21:06.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:06 smithi076 conmon[62156]: cluster 2021-03-15T22:21:06.013027+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2021-03-15T22:21:06.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:06 smithi076 conmon[56514]: cluster 2021-03-15T22:21:04.762287+0000 osd.1 (osd.1) 34 : cluster [DBG] 4.7 deep-scrub ok 2021-03-15T22:21:06.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:06 smithi076 conmon[56514]: cluster 2021-03-15T22:21:04.885369+0000 osd.3 (osd.3) 43 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:21:06.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:06 smithi076 conmon[56514]: cluster 2021-03-15T22:21:05.103657+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.17 scrub ok 2021-03-15T22:21:06.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:06 smithi076 conmon[56514]: cluster 2021-03-15T22:21:05.736766+0000 osd.0 (osd.0) 36 : cluster [DBG] 3.17 deep-scrub ok 2021-03-15T22:21:06.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:06 smithi076 conmon[56514]: cluster 2021-03-15T22:21:06.013027+0000 mon.a (mon.0) 713 : cluster [DBG] osdmap e151: 8 total, 8 up, 8 in 2021-03-15T22:21:06.521 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:06.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:06.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:06.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:06.648 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:06.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:06.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:06.774 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:06.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:06.776 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:06.877 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:06.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:06.900 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:06.901 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:07.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:07.018 INFO:tasks.rados.rados.0.smithi076.stdout:301: done (0 left) 2021-03-15T22:21:07.019 INFO:tasks.rados.rados.0.smithi076.stdout:302: rollback oid 31 current snap is 28 2021-03-15T22:21:07.019 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 31 to 24 2021-03-15T22:21:07.019 INFO:tasks.rados.rados.0.smithi076.stdout:303: rollback oid 19 current snap is 28 2021-03-15T22:21:07.019 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 19 to 26 2021-03-15T22:21:07.020 INFO:tasks.rados.rados.0.smithi076.stdout:304: write (excl) oid 26 current snap is 28 2021-03-15T22:21:07.020 INFO:tasks.rados.rados.0.smithi076.stdout:304: seq_num 141 ranges {769787=704463,2267961=101826} 2021-03-15T22:21:07.022 INFO:tasks.rados.rados.0.smithi076.stdout:304: writing smithi07694878-26 from 769787 to 1474250 tid 1 2021-03-15T22:21:07.026 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:21:07.029 INFO:tasks.rados.rados.0.smithi076.stdout:304: writing smithi07694878-26 from 2267961 to 2369787 tid 2 2021-03-15T22:21:07.029 INFO:tasks.rados.rados.0.smithi076.stdout:305: read oid 4 snap -1 2021-03-15T22:21:07.029 INFO:tasks.rados.rados.0.smithi076.stdout:305: expect (ObjNum 121 snap 24 seq_num 121) 2021-03-15T22:21:07.029 INFO:tasks.rados.rados.0.smithi076.stdout:306: copy_from oid 22 from oid 30 current snap is 28 2021-03-15T22:21:07.030 INFO:tasks.rados.rados.0.smithi076.stdout:302: finishing rollback tid 0 to smithi07694878-31 2021-03-15T22:21:07.030 INFO:tasks.rados.rados.0.smithi076.stdout:302: finishing rollback tid 1 to smithi07694878-31 2021-03-15T22:21:07.031 INFO:tasks.rados.rados.0.smithi076.stdout:302: finishing rollback tid 2 to smithi07694878-31 2021-03-15T22:21:07.031 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 55 (ObjNum 73 snap 8 seq_num 73) dirty exists 2021-03-15T22:21:07.032 INFO:tasks.rados.rados.0.smithi076.stdout:302: done (4 left) 2021-03-15T22:21:07.032 INFO:tasks.rados.rados.0.smithi076.stdout:307: rollback oid 11 current snap is 28 2021-03-15T22:21:07.032 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 11 to 26 2021-03-15T22:21:07.032 INFO:tasks.rados.rados.0.smithi076.stdout:303: finishing rollback tid 0 to smithi07694878-19 2021-03-15T22:21:07.032 INFO:tasks.rados.rados.0.smithi076.stdout:303: finishing rollback tid 1 to smithi07694878-19 2021-03-15T22:21:07.033 INFO:tasks.rados.rados.0.smithi076.stdout:303: finishing rollback tid 2 to smithi07694878-19 2021-03-15T22:21:07.033 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 98 (ObjNum 9 snap 0 seq_num 9) dirty exists 2021-03-15T22:21:07.033 INFO:tasks.rados.rados.0.smithi076.stdout:303: done (4 left) 2021-03-15T22:21:07.033 INFO:tasks.rados.rados.0.smithi076.stdout:308: delete oid 37 current snap is 28 2021-03-15T22:21:07.034 INFO:tasks.rados.rados.0.smithi076.stdout:304: finishing write tid 1 to smithi07694878-26 2021-03-15T22:21:07.034 INFO:tasks.rados.rados.0.smithi076.stdout:305: expect (ObjNum 121 snap 24 seq_num 121) 2021-03-15T22:21:07.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:07.037 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:07.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:07.067 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:21:07.072 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:21:07.072 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:21:07.072 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:21:07.073 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:21:07.073 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:21:07.084 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:21:07.101 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:21:07.118 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:21:07.129 INFO:tasks.rados.rados.0.smithi076.stdout:305: done (4 left) 2021-03-15T22:21:07.129 INFO:tasks.rados.rados.0.smithi076.stdout:308: done (3 left) 2021-03-15T22:21:07.129 INFO:tasks.rados.rados.0.smithi076.stdout:309: write oid 21 current snap is 28 2021-03-15T22:21:07.130 INFO:tasks.rados.rados.0.smithi076.stdout:307: finishing rollback tid 1 to smithi07694878-11 2021-03-15T22:21:07.130 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 0 (ObjNum 1 snap 21896 seq_num 261578544) dirty dne 2021-03-15T22:21:07.130 INFO:tasks.rados.rados.0.smithi076.stdout:304: finishing write tid 2 to smithi07694878-26 2021-03-15T22:21:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:304: finishing write tid 3 to smithi07694878-26 2021-03-15T22:21:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:304: finishing write tid 4 to smithi07694878-26 2021-03-15T22:21:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 130 (ObjNum 141 snap 28 seq_num 141) dirty exists 2021-03-15T22:21:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:304: left oid 26 (ObjNum 141 snap 28 seq_num 141) 2021-03-15T22:21:07.132 INFO:tasks.rados.rados.0.smithi076.stdout:306: finishing copy_from to smithi07694878-22 2021-03-15T22:21:07.132 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 127 (ObjNum 113 snap 24 seq_num 113) dirty exists 2021-03-15T22:21:07.132 INFO:tasks.rados.rados.0.smithi076.stdout:306: finishing copy_from racing read to smithi07694878-22 2021-03-15T22:21:07.133 INFO:tasks.rados.rados.0.smithi076.stdout:309: seq_num 143 ranges {403401=454961,1584925=506039} 2021-03-15T22:21:07.133 INFO:tasks.rados.rados.0.smithi076.stdout:309: writing smithi07694878-21 from 403401 to 858362 tid 1 2021-03-15T22:21:07.138 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:07.140 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:21:07.141 INFO:tasks.rados.rados.0.smithi076.stdout:309: writing smithi07694878-21 from 1584925 to 2090964 tid 2 2021-03-15T22:21:07.142 INFO:tasks.rados.rados.0.smithi076.stdout:304: done (3 left) 2021-03-15T22:21:07.142 INFO:tasks.rados.rados.0.smithi076.stdout:306: done (2 left) 2021-03-15T22:21:07.142 INFO:tasks.rados.rados.0.smithi076.stdout:307: done (1 left) 2021-03-15T22:21:07.142 INFO:tasks.rados.rados.0.smithi076.stdout:310: snap_create 2021-03-15T22:21:07.143 INFO:tasks.rados.rados.0.smithi076.stdout:309: finishing write tid 1 to smithi07694878-21 2021-03-15T22:21:07.149 INFO:tasks.rados.rados.0.smithi076.stdout:309: finishing write tid 2 to smithi07694878-21 2021-03-15T22:21:07.150 INFO:tasks.rados.rados.0.smithi076.stdout:309: finishing write tid 3 to smithi07694878-21 2021-03-15T22:21:07.150 INFO:tasks.rados.rados.0.smithi076.stdout:309: finishing write tid 4 to smithi07694878-21 2021-03-15T22:21:07.150 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 185 (ObjNum 143 snap 28 seq_num 143) dirty exists 2021-03-15T22:21:07.150 INFO:tasks.rados.rados.0.smithi076.stdout:309: left oid 21 (ObjNum 143 snap 28 seq_num 143) 2021-03-15T22:21:07.162 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:21:07.165 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:07.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:07.166 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:07.180 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:21:07.199 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:21:07.200 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:21:07.201 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:21:07.201 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:21:07.202 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:21:07.202 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:21:07.203 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:21:07.203 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:21:07.203 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:21:07.269 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:07.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:07.293 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:07.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:07.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:07.418 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:07.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:07.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:07.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: cluster 2021-03-15T22:21:06.006082+0000 2021-03-15T22:21:07.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: osd.7 (osd.7) 40 : cluster [DBG] 3.d deep-scrub ok 2021-03-15T22:21:07.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: audit 2021-03-15T22:21 2021-03-15T22:21:07.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: :06.147389+0000 mon.a (mon.0) 714 : audit [INF] from='client.? 172.21.15.76:0/2659202478' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:21:07.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:21:07.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: T22:21:06.404278+0000 osd.4 (osd 2021-03-15T22:21:07.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: .4) 42 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:21:07.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:07.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: 03-15T22:21:06.725495+0000 2021-03-15T22:21:07.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: osd.0 (osd.0) 37 : cluster [DBG] 2021-03-15T22:21:07.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: 6.0 scrub ok 2021-03-15T22:21:07.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: cluster 2021-03-15T22:21:06 2021-03-15T22:21:07.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: .990600+0000 mgr.y (mgr.14144) 410 : 2021-03-15T22:21:07.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: cluster [DBG] pgmap v354: 146 pgs: 146 active+clean; 137 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.4 MiB/s wr, 12 op/s 2021-03-15T22:21:07.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:21:07.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: 21:06.991351+0000 mon.a (mon.0 2021-03-15T22:21:07.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: ) 715 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-03-15T22:21:07.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:21:07.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: 15T22:21:07.007281+0000 mon.a ( 2021-03-15T22:21:07.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: mon.0) 716 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:07.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:21:07.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: 03-15T22:21:07.010869+0000 2021-03-15T22:21:07.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: mon.a (mon.0) 717 : audit [INF] from='client.? 172.21.15.76:0/2659202478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:21:07.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: 2021-03-15T22:21:07.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: audit 2021 2021-03-15T22:21:07.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: -03-15T22:21:07.010938+0000 mon.a (mon 2021-03-15T22:21:07.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: .0) 718 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2021-03-15T22:21:07.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:07.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: 03-15T22:21:07.011090+0000 2021-03-15T22:21:07.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: mon.a (mon.0) 719 : cluster [DBG] 2021-03-15T22:21:07.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: osdmap e152: 8 total, 8 up, 8 in 2021-03-15T22:21:07.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: audit 2021-03-15T22:21:07 2021-03-15T22:21:07.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: .047484+0000 mon.a (mon.0) 720 : 2021-03-15T22:21:07.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[62156]: audit [DBG] from='client.? 172.21.15.76:0/416263334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:07.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: cluster 2021-03-15T22:21:07.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: 2021-03 2021-03-15T22:21:07.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: -15T22:21:06.006082+0000 osd.7 (osd.7) 40 : cluster [DBG] 3.d deep-scrub ok 2021-03-15T22:21:07.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: audit 2021-03-15T22:21:07.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: 2021-03 2021-03-15T22:21:07.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: -15T22:21:06.147389+0000 mon.a (mon. 2021-03-15T22:21:07.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: 0) 714 : audit [INF] from='client.? 172.21.15.76:0/2659202478' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:21:07.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:21:07.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: 21:06.404278+0000 osd.4 (osd.4) 42 2021-03-15T22:21:07.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: : cluster [DBG] 6.1a scrub ok 2021-03-15T22:21:07.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:21:07.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: 15T22:21:06.725495+0000 osd.0 ( 2021-03-15T22:21:07.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: osd.0) 37 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:21:07.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:21:07.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: -03-15T22:21:06.990600+0000 2021-03-15T22:21:07.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: mgr.y (mgr.14144) 410 : cluster [DBG] 2021-03-15T22:21:07.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: pgmap v354: 146 pgs: 146 active+clean; 137 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.4 MiB/s wr, 12 op/s 2021-03-15T22:21:07.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: audit 2021-03-15T22:21:06. 2021-03-15T22:21:07.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: 991351+0000 mon.a (mon.0) 715 : audit 2021-03-15T22:21:07.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-03-15T22:21:07.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: cluster 2021-03-15T22:21: 2021-03-15T22:21:07.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: 07.007281+0000 mon.a (mon.0) 716 : 2021-03-15T22:21:07.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:07.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: audit 2021-03-15T22 2021-03-15T22:21:07.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: :21:07.010869+0000 mon.a (mon.0 2021-03-15T22:21:07.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: ) 717 : audit [INF] from='client.? 172.21.15.76:0/2659202478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:21:07.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:21:07.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: 15T22:21:07.010938+0000 mon.a (mon 2021-03-15T22:21:07.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: .0) 718 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2021-03-15T22:21:07.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:21:07.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: -15T22:21:07.011090+0000 mon.a 2021-03-15T22:21:07.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: (mon.0) 719 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2021-03-15T22:21:07.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:21:07.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: 03-15T22:21:07.047484+0000 mon.a 2021-03-15T22:21:07.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: (mon.0) 720 : audit [DBG] from='client.? 172.21.15.76:0/416263334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:07.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:07 smithi076 conmon[56514]: 2021-03-15T22:21:07.533 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:07.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: cluster 2021-03-15T22:21:06.006082+0000 2021-03-15T22:21:07.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: osd.7 (osd.7) 40 : cluster [DBG] 3.d deep-scrub ok 2021-03-15T22:21:07.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: audit 2021-03-15T22:21:06.147389+0000 mon.a (mon.0) 714 : audit [INF] from='client.? 172.21.15.76:0/2659202478' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:21:07.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: cluster 2021-03-15T22:21:06.404278+0000 osd.4 (osd.4) 42 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:21:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: cluster 2021-03-15T22:21:06.725495+0000 osd.0 (osd.0) 37 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:21:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: cluster 2021-03-15T22:21:06.990600+0000 mgr.y (mgr.14144) 410 : cluster [DBG] pgmap v354: 146 pgs: 146 active+clean; 137 MiB data, 432 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.4 MiB/s wr, 12 op/s 2021-03-15T22:21:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: audit 2021-03-15T22:21:06.991351+0000 mon.a (mon.0) 715 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-03-15T22:21:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: cluster 2021-03-15T22:21:07.007281+0000 mon.a (mon.0) 716 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: audit 2021-03-15T22:21:07.010869+0000 mon.a (mon.0) 717 : audit [INF] from='client.? 172.21.15.76:0/2659202478' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:21:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: audit 2021-03-15T22:21:07.010938+0000 mon.a (mon.0) 718 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "29"}]': finished 2021-03-15T22:21:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: cluster 2021-03-15T22:21:07.011090+0000 mon.a (mon.0) 719 : cluster [DBG] osdmap e152: 8 total, 8 up, 8 in 2021-03-15T22:21:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:07 smithi125 conmon[56739]: audit 2021-03-15T22:21:07.047484+0000 mon.a (mon.0) 720 : audit [DBG] from='client.? 172.21.15.76:0/416263334' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:07.555 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:07.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:07.557 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:07.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:07.680 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:07.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:07.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:07.703 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-15T22:21:07.783 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:07.808 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:07.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:07.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:07.911 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:07.933 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:07.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:07.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:08.016 INFO:tasks.rados.rados.0.smithi076.stdout:309: done (1 left) 2021-03-15T22:21:08.017 INFO:tasks.rados.rados.0.smithi076.stdout:310: done (0 left) 2021-03-15T22:21:08.017 INFO:tasks.rados.rados.0.smithi076.stdout:311: rollback oid 8 current snap is 29 2021-03-15T22:21:08.017 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 24 2021-03-15T22:21:08.017 INFO:tasks.rados.rados.0.smithi076.stdout:312: snap_create 2021-03-15T22:21:08.022 INFO:tasks.rados.rados.0.smithi076.stdout:311: finishing rollback tid 0 to smithi07694878-8 2021-03-15T22:21:08.023 INFO:tasks.rados.rados.0.smithi076.stdout:311: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:21:08.023 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:08.024 INFO:tasks.rados.rados.0.smithi076.stdout:311: finishing rollback tid 2 to smithi07694878-8 2021-03-15T22:21:08.024 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 75 (ObjNum 7 snap 0 seq_num 7) dirty exists 2021-03-15T22:21:08.036 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:08.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:08.059 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:08.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:08.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:08.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:08.184 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:08.184 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:08.286 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:08.307 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:08.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:08.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:08.406 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:08 smithi125 conmon[56739]: cluster 2021-03-15T22:21:06.781513+0000 osd.6 (osd.6) 2021-03-15T22:21:08.406 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:08 smithi125 conmon[56739]: 20 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:21:08.406 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:08 smithi125 conmon[56739]: cluster 2021-03-15T22:21:08.014320+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2021-03-15T22:21:08.410 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:08.433 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:08.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:08.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:08 smithi076 conmon[56514]: cluster 2021-03-15T22:21:06.781513+0000 2021-03-15T22:21:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:08 smithi076 conmon[56514]: osd.6 (osd.6) 20 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:21:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:08 smithi076 conmon[56514]: cluster 2021-03-15T22:21:08.014320+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2021-03-15T22:21:08.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:08 smithi076 conmon[62156]: cluster 2021-03-15T22:21:06.781513+0000 2021-03-15T22:21:08.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:08 smithi076 conmon[62156]: osd.6 (osd.6) 20 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:21:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:08 smithi076 conmon[62156]: cluster 2021-03-15T22:21:08.014320+0000 mon.a (mon.0) 721 : cluster [DBG] osdmap e153: 8 total, 8 up, 8 in 2021-03-15T22:21:08.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:08.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:08.558 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:08.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:08.660 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:08.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:08.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:08.786 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:08.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:08.809 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:08.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:08.910 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:08.933 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:08.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:09.016 INFO:tasks.rados.rados.0.smithi076.stdout:311: done (1 left) 2021-03-15T22:21:09.016 INFO:tasks.rados.rados.0.smithi076.stdout:312: done (0 left) 2021-03-15T22:21:09.017 INFO:tasks.rados.rados.0.smithi076.stdout:313: snap_remove snap 29 2021-03-15T22:21:09.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:09.057 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:09.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:09.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:09.160 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:09.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:09.183 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:09.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:09.285 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:09.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:09.309 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:09.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:09.334 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:09 smithi125 conmon[56739]: cluster 2021-03-15T22:21:08.991056+0000 mgr.y (mgr.14144) 411 : cluster [DBG] pgmap v357: 146 pgs: 146 active+clean; 128 MiB data, 402 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-03-15T22:21:09.334 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:09 smithi125 conmon[56739]: cluster 2021-03-15T22:21:09.013670+0000 mon.a (mon.0) 722 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2021-03-15T22:21:09.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:09.432 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:09.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:09.434 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:09.459 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:21:09.460 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:21:09.460 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump -f json-pretty 2021-03-15T22:21:09.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:09 smithi076 conmon[56514]: cluster 2021-03-15T22:21:08.991056 2021-03-15T22:21:09.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:09 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 411 : cluster [DBG] pgmap v357: 146 pgs: 146 active+clean; 128 MiB data, 402 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-03-15T22:21:09.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:09 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:21:09.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:09 smithi076 conmon[56514]: -03-15T22:21:09.013670+0000 mon.a (mon.0) 2021-03-15T22:21:09.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:09 smithi076 conmon[56514]: 722 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2021-03-15T22:21:09.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:09 smithi076 conmon[62156]: cluster 2021-03-15T22:21: 2021-03-15T22:21:09.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:09 smithi076 conmon[62156]: 08.991056+0000 mgr.y (mgr.14144) 411 : cluster [DBG] pgmap v357: 146 pgs: 146 active+clean; 128 MiB data, 402 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.7 MiB/s wr, 13 op/s 2021-03-15T22:21:09.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:09 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:21:09.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:09 smithi076 conmon[62156]: 21:09.013670+0000 mon.a (mon. 2021-03-15T22:21:09.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:09 smithi076 conmon[62156]: 0) 722 : cluster [DBG] osdmap e154: 8 total, 8 up, 8 in 2021-03-15T22:21:09.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:09 smithi076 conmon[62156]: 2021-03-15T22:21:09.522 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:21:09.523 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:21:09.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:09.557 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:09.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:09.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:09.660 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:09.682 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:09.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:09.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:09.743 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:09.785 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:09.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:21:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:21:09.808 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:09.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:09.810 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:09.911 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:09.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:09.935 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:09.935 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:10.011 INFO:tasks.rados.rados.0.smithi076.stdout:313: done (0 left) 2021-03-15T22:21:10.012 INFO:tasks.rados.rados.0.smithi076.stdout:314: copy_from oid 29 from oid 45 current snap is 30 2021-03-15T22:21:10.017 INFO:tasks.rados.rados.0.smithi076.stdout:315: rollback oid 34 current snap is 30 2021-03-15T22:21:10.017 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 34 to 24 2021-03-15T22:21:10.018 INFO:tasks.rados.rados.0.smithi076.stdout:316: read oid 13 snap -1 2021-03-15T22:21:10.018 INFO:tasks.rados.rados.0.smithi076.stdout:316: expect (ObjNum 133 snap 25 seq_num 133) 2021-03-15T22:21:10.018 INFO:tasks.rados.rados.0.smithi076.stdout:317: read oid 26 snap -1 2021-03-15T22:21:10.019 INFO:tasks.rados.rados.0.smithi076.stdout:317: expect (ObjNum 141 snap 28 seq_num 141) 2021-03-15T22:21:10.019 INFO:tasks.rados.rados.0.smithi076.stdout:318: write oid 17 current snap is 30 2021-03-15T22:21:10.019 INFO:tasks.rados.rados.0.smithi076.stdout:318: seq_num 144 ranges {420208=530210,1527054=766050} 2021-03-15T22:21:10.019 INFO:tasks.rados.rados.0.smithi076.stdout:318: writing smithi07694878-17 from 420208 to 950418 tid 1 2021-03-15T22:21:10.026 INFO:tasks.rados.rados.0.smithi076.stdout:318: writing smithi07694878-17 from 1527054 to 2293104 tid 2 2021-03-15T22:21:10.026 INFO:tasks.rados.rados.0.smithi076.stdout:319: snap_remove snap 26 2021-03-15T22:21:10.037 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:10.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:10.137 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:21:10.163 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:10.201 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:10.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:10.203 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:10.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:21:10.275 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:21:10.304 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:10.341 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:10.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:10.413 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:10.444 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:10.470 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:10.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:10.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:10.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:10 smithi076 conmon[56514]: cluster 2021-03-15T22:21:08.603196+0000 osd.1 (osd.1) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:10 smithi076 conmon[56514]: cluster 2021-03-15T22:21:08.603369+0000 osd.1 (osd.1) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:10 smithi076 conmon[56514]: audit 2021-03-15T22:21:09.468957+0000 mgr.y (mgr.14144) 412 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:10 smithi076 conmon[56514]: cluster 2021-03-15T22:21:10.005646+0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2021-03-15T22:21:10.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:10 smithi076 conmon[62156]: cluster 2021-03-15T22:21:08.603196+0000 osd.1 (osd.1) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:10.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:10 smithi076 conmon[62156]: cluster 2021-03-15T22:21:08.603369+0000 osd.1 (osd.1) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:10.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:10 smithi076 conmon[62156]: audit 2021-03-15T22:21:09 2021-03-15T22:21:10.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:10 smithi076 conmon[62156]: .468957+0000 mgr.y (mgr.14144) 412 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:10.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:10 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:21:10.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:10 smithi076 conmon[62156]: 15T22:21:10.005646+0000 mon.a (mon. 2021-03-15T22:21:10.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:10 smithi076 conmon[62156]: 0) 723 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2021-03-15T22:21:10.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:10 smithi125 conmon[56739]: cluster 2021-03-15T22:21:08.603196+0000 osd.1 (osd. 2021-03-15T22:21:10.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:10 smithi125 conmon[56739]: 1) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:10 smithi125 conmon[56739]: cluster 2021-03-15T22:21:08.603369+0000 osd.1 (osd.1) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:10 smithi125 conmon[56739]: audit 2021-03-15T22:21:09.468957+0000 mgr.y (mgr.14144) 412 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:10 smithi125 conmon[56739]: cluster 2021-03-15T22:21:10.005646 2021-03-15T22:21:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:10 smithi125 conmon[56739]: +0000 mon.a (mon.0) 723 : cluster [DBG] osdmap e155: 8 total, 8 up, 8 in 2021-03-15T22:21:10.573 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:10.595 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:10.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:10.597 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:10.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:10.720 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:10.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:10.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:10.823 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:10.847 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:10.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:10.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:10.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:10.967 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:10.972 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:10.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:10.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:11.017 INFO:tasks.rados.rados.0.smithi076.stdout:319: done (5 left) 2021-03-15T22:21:11.018 INFO:tasks.rados.rados.0.smithi076.stdout:320: read oid 39 snap 24 2021-03-15T22:21:11.018 INFO:tasks.rados.rados.0.smithi076.stdout:320: expect (ObjNum 116 snap 24 seq_num 116) 2021-03-15T22:21:11.018 INFO:tasks.rados.rados.0.smithi076.stdout:321: delete oid 31 current snap is 30 2021-03-15T22:21:11.025 INFO:tasks.rados.rados.0.smithi076.stdout:315: finishing rollback tid 0 to smithi07694878-34 2021-03-15T22:21:11.026 INFO:tasks.rados.rados.0.smithi076.stdout:316: expect (ObjNum 133 snap 25 seq_num 133) 2021-03-15T22:21:11.033 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:11.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:11.098 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:11.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:11.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:11.139 INFO:tasks.rados.rados.0.smithi076.stdout:315: finishing rollback tid 1 to smithi07694878-34 2021-03-15T22:21:11.139 INFO:tasks.rados.rados.0.smithi076.stdout:315: finishing rollback tid 2 to smithi07694878-34 2021-03-15T22:21:11.139 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 85 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:21:11.140 INFO:tasks.rados.rados.0.smithi076.stdout:318: finishing write tid 1 to smithi07694878-17 2021-03-15T22:21:11.140 INFO:tasks.rados.rados.0.smithi076.stdout:317: expect (ObjNum 141 snap 28 seq_num 141) 2021-03-15T22:21:11.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:11 smithi076 conmon[56514]: cluster 2021-03-15T22:21:10.991484 2021-03-15T22:21:11.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:11 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 413 : cluster [DBG] pgmap v360: 145 pgs: 1 peering, 144 active+clean; 128 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 235 KiB/s wr, 4 op/s 2021-03-15T22:21:11.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:11 smithi076 conmon[56514]: audit 2021-03-15T22:21:10.992311+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-03-15T22:21:11.166 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:11 smithi076 conmon[56514]: audit 2021-03-15T22:21:11.015636+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2021-03-15T22:21:11.166 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:11 smithi076 conmon[56514]: cluster 2021-03-15T22:21:11.015701+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2021-03-15T22:21:11.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:11.223 INFO:tasks.rados.rados.0.smithi076.stdout:318: finishing write tid 2 to smithi07694878-17 2021-03-15T22:21:11.223 INFO:tasks.rados.rados.0.smithi076.stdout:318: finishing write tid 3 to smithi07694878-17 2021-03-15T22:21:11.224 INFO:tasks.rados.rados.0.smithi076.stdout:314: finishing copy_from to smithi07694878-29 2021-03-15T22:21:11.224 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 157 (ObjNum 132 snap 25 seq_num 132) dirty exists 2021-03-15T22:21:11.224 INFO:tasks.rados.rados.0.smithi076.stdout:318: finishing write tid 4 to smithi07694878-17 2021-03-15T22:21:11.224 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 30 (ObjNum 144 snap 30 seq_num 144) dirty exists 2021-03-15T22:21:11.225 INFO:tasks.rados.rados.0.smithi076.stdout:318: left oid 17 (ObjNum 144 snap 30 seq_num 144) 2021-03-15T22:21:11.225 INFO:tasks.rados.rados.0.smithi076.stdout:314: finishing copy_from racing read to smithi07694878-29 2021-03-15T22:21:11.225 INFO:tasks.rados.rados.0.smithi076.stdout:314: done (6 left) 2021-03-15T22:21:11.225 INFO:tasks.rados.rados.0.smithi076.stdout:315: done (5 left) 2021-03-15T22:21:11.226 INFO:tasks.rados.rados.0.smithi076.stdout:316: done (4 left) 2021-03-15T22:21:11.226 INFO:tasks.rados.rados.0.smithi076.stdout:317: done (3 left) 2021-03-15T22:21:11.226 INFO:tasks.rados.rados.0.smithi076.stdout:318: done (2 left) 2021-03-15T22:21:11.226 INFO:tasks.rados.rados.0.smithi076.stdout:321: done (1 left) 2021-03-15T22:21:11.227 INFO:tasks.rados.rados.0.smithi076.stdout:322: snap_remove snap 28 2021-03-15T22:21:11.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:11.228 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:11.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:11.330 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:11.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:11.354 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:11.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:11.456 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:11.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:11 smithi125 conmon[56739]: cluster 2021-03-15T22:21:10.991484+0000 mgr.y (mgr.14144) 413 : cluster [DBG] pgmap v360: 145 pgs: 1 peering, 144 active+clean; 128 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 235 KiB/s wr, 4 op/s 2021-03-15T22:21:11.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:11 smithi125 conmon[56739]: audit 2021-03-15T22:21:10.992311+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-03-15T22:21:11.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:11 smithi125 conmon[56739]: audit 2021-03-15T22:21:11.015636+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2021-03-15T22:21:11.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:11 smithi125 conmon[56739]: cluster 2021-03-15T22:21:11.015701+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2021-03-15T22:21:11.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:11 smithi076 conmon[62156]: cluster 2021-03-15T22:21:10.991484+0000 mgr.y (mgr.14144) 413 : cluster [DBG] pgmap v360: 145 pgs: 1 peering, 144 active+clean; 128 MiB data, 409 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 235 KiB/s wr, 4 op/s 2021-03-15T22:21:11.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:11 smithi076 conmon[62156]: audit 2021-03-15T22:21:10.992311+0000 mon.a (mon.0) 724 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-03-15T22:21:11.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:11 smithi076 conmon[62156]: audit 2021-03-15T22:21:11.015636+0000 mon.a (mon.0) 725 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "28"}]': finished 2021-03-15T22:21:11.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:11 smithi076 conmon[62156]: cluster 2021-03-15T22:21:11.015701+0000 mon.a (mon.0) 726 : cluster [DBG] osdmap e156: 8 total, 8 up, 8 in 2021-03-15T22:21:11.479 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:11.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:11.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:11.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:11.604 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:11.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:11.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:11.708 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:11.731 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:11.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:11.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:11.834 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:11.857 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:11.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:11.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:11.919 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:21:11.920 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:21:11.959 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:11.982 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:11.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:11.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:12.017 INFO:tasks.rados.rados.0.smithi076.stdout:322: done (1 left) 2021-03-15T22:21:12.017 INFO:tasks.rados.rados.0.smithi076.stdout:323: delete oid 35 current snap is 30 2021-03-15T22:21:12.018 INFO:tasks.rados.rados.0.smithi076.stdout:320: expect (ObjNum 116 snap 24 seq_num 116) 2021-03-15T22:21:12.084 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:12.106 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:12.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:12.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:12.153 INFO:tasks.rados.rados.0.smithi076.stdout:320: done (1 left) 2021-03-15T22:21:12.153 INFO:tasks.rados.rados.0.smithi076.stdout:323: done (0 left) 2021-03-15T22:21:12.153 INFO:tasks.rados.rados.0.smithi076.stdout:324: snap_create 2021-03-15T22:21:12.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:12.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:12.232 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:12.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:12.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:12.356 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:12.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:12.358 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:12.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:12.484 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:12.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:12.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:12.550 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:21:12.550 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:21:12.550 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 157, 2021-03-15T22:21:12.551 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:21:12.551 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2021-03-15T22:13:20.370795+0000", 2021-03-15T22:21:12.551 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2021-03-15T22:21:12.011933+0000", 2021-03-15T22:21:12.551 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2021-03-15T22:17:49.162889+0000", 2021-03-15T22:21:12.552 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2021-03-15T22:17:34.080423+0000", 2021-03-15T22:21:12.552 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-03-15T22:21:12.552 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5801984, 2021-03-15T22:21:12.552 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2021-03-15T22:21:12.552 INFO:teuthology.orchestra.run.smithi076.stdout: "noscrub", 2021-03-15T22:21:12.553 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2021-03-15T22:21:12.553 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2021-03-15T22:21:12.553 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2021-03-15T22:21:12.553 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2021-03-15T22:21:12.553 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:12.554 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 18, 2021-03-15T22:21:12.554 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2021-03-15T22:21:12.554 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-03-15T22:21:12.554 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2021-03-15T22:21:12.555 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2021-03-15T22:21:12.555 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 7, 2021-03-15T22:21:12.555 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2021-03-15T22:21:12.555 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2021-03-15T22:21:12.556 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2021-03-15T22:21:12.556 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2021-03-15T22:21:12.556 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2021-03-15T22:21:12.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.557 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2021-03-15T22:21:12.557 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "device_health_metrics", 2021-03-15T22:21:12.557 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:16:34.817191+0000", 2021-03-15T22:21:12.557 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:12.558 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:12.558 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:12.558 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:12.558 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:12.559 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:12.559 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:12.559 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:12.559 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:12.559 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:12.560 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:12.560 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:12.560 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2021-03-15T22:21:12.560 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2021-03-15T22:21:12.561 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2021-03-15T22:21:12.561 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2021-03-15T22:21:12.561 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2021-03-15T22:21:12.561 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:12.562 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:12.562 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:12.562 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:12.562 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:12.563 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:12.563 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:12.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.564 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "23", 2021-03-15T22:21:12.564 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:12.564 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:12.564 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:12.564 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:12.565 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:12.565 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:12.565 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:12.565 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:12.565 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:12.566 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:12.566 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:12.566 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:12.566 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:12.566 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:12.567 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:12.567 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:12.567 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:12.567 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:12.568 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:12.568 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:12.568 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:12.568 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:12.569 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:12.569 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:12.569 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:12.569 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:12.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.570 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:12.570 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:12.570 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:12.570 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:12.571 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:12.571 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:12.571 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:12.571 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:12.571 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:12.572 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:12.572 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:12.572 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:21:12.572 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2021-03-15T22:21:12.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.573 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:12.573 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_devicehealth": {} 2021-03-15T22:21:12.573 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.574 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2021-03-15T22:21:12.574 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "iscsi", 2021-03-15T22:21:12.574 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:49.906400+0000", 2021-03-15T22:21:12.574 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:12.575 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:12.575 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:12.575 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:12.575 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:12.575 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:12.576 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:12.576 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:12.576 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:12.576 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:12.577 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:12.577 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:12.577 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 3, 2021-03-15T22:21:12.577 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 3, 2021-03-15T22:21:12.577 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 3, 2021-03-15T22:21:12.578 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 3, 2021-03-15T22:21:12.578 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 3, 2021-03-15T22:21:12.578 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:12.578 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:12.579 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:12.579 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:12.579 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:12.579 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:12.580 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:12.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.580 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "56", 2021-03-15T22:21:12.580 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:12.580 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:12.581 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:12.581 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:12.581 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:12.581 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:12.582 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:12.582 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:12.582 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:12.582 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:12.582 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:12.583 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:12.583 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:12.583 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:12.583 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:12.584 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:12.584 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:12.584 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:12.584 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:12.585 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:12.585 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:12.585 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:12.585 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:12.585 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:12.586 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:12.586 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:12.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.586 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:12.586 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:12.587 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:12.587 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:12.587 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:12.587 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:12.587 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:12.588 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:12.588 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:12.588 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:12.588 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:12.589 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:21:12.589 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:12.589 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2021-03-15T22:21:12.589 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.590 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2021-03-15T22:21:12.590 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".rgw.root", 2021-03-15T22:21:12.590 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:52.257970+0000", 2021-03-15T22:21:12.590 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:12.591 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:12.591 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:12.591 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:12.591 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:12.592 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:12.592 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:12.592 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:12.592 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:12.592 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:12.593 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:12.593 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:12.593 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:21:12.593 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:21:12.593 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:21:12.594 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:21:12.594 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:21:12.594 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:12.594 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:12.595 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:12.595 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:12.595 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:12.595 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.596 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "58", 2021-03-15T22:21:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:12.597 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:12.598 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:12.599 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:12.600 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:12.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:12.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:12.603 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:12.604 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:21:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:12.605 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:21:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.606 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2021-03-15T22:21:12.607 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.log", 2021-03-15T22:21:12.607 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:54.562498+0000", 2021-03-15T22:21:12.607 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:12.607 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:12.607 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:12.608 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:12.608 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:12.608 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:12.608 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:12.609 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:12.609 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:12.609 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:12.609 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:12.609 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:12.610 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:21:12.610 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:21:12.610 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:21:12.610 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:21:12.611 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:21:12.611 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:12.611 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:12.611 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:12.612 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:12.612 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:12.612 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:12.612 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:12.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.613 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "60", 2021-03-15T22:21:12.613 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:12.613 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:12.613 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:12.614 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:12.614 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:12.614 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:12.614 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:12.615 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:12.615 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:12.615 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:12.615 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:12.615 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:12.616 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:12.616 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:12.616 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:12.616 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:12.616 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:12.617 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:12.617 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:12.617 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:12.617 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:12.617 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:12.618 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:12.618 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:12.618 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:12.618 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:12.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.619 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:12.619 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:12.619 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:12.619 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:12.620 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:12.620 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:12.620 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:12.620 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:12.620 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:12.621 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:12.621 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:12.621 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:21:12.621 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:12.621 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:21:12.622 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.622 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 5, 2021-03-15T22:21:12.622 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.control", 2021-03-15T22:21:12.623 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:56.243806+0000", 2021-03-15T22:21:12.623 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:12.623 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:12.623 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:12.624 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:12.624 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:12.624 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:12.624 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:12.624 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:12.625 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:12.625 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:12.625 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:12.625 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:12.625 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 29, 2021-03-15T22:21:12.626 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 22, 2021-03-15T22:21:12.626 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 22, 2021-03-15T22:21:12.626 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 22, 2021-03-15T22:21:12.626 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 28, 2021-03-15T22:21:12.626 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:12.627 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "5.1d", 2021-03-15T22:21:12.627 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 154, 2021-03-15T22:21:12.627 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 153, 2021-03-15T22:21:12.627 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 153, 2021-03-15T22:21:12.627 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:12.628 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "61'2" 2021-03-15T22:21:12.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.628 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "156", 2021-03-15T22:21:12.628 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:12.629 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "156", 2021-03-15T22:21:12.629 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "156", 2021-03-15T22:21:12.629 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:12.629 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:12.629 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:12.630 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:12.630 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:12.630 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:12.630 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:12.630 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:12.631 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:12.631 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:12.631 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:12.631 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:12.632 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:12.632 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:12.632 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:12.633 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:12.634 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:12.634 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:12.634 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:12.635 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:12.635 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:12.635 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:12.635 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:12.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.636 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:12.636 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:12.636 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:12.636 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:12.637 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:12.637 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:12.637 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:12.637 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:12.637 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:12.638 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:12.638 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:12.638 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:21:12.638 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:12.638 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:21:12.639 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.639 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 6, 2021-03-15T22:21:12.640 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.meta", 2021-03-15T22:21:12.640 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:58.730534+0000", 2021-03-15T22:21:12.640 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:21:12.640 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:21:12.640 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:12.641 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:12.641 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:12.641 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:12.641 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:12.641 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:12.642 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:12.642 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:12.642 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:12.642 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:12.642 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:21:12.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:21:12.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:21:12.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:21:12.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:21:12.643 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:12.644 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:12.644 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:12.644 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:12.644 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:12.645 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:12.645 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:12.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.645 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "109", 2021-03-15T22:21:12.645 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:12.646 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:12.646 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:12.646 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:12.646 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:12.646 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:21:12.647 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:21:12.647 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:12.647 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:12.647 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:12.647 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:12.648 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:12.648 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:12.648 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:12.648 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:12.648 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:12.649 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:12.649 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:12.649 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:12.649 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:12.650 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:12.650 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:12.650 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:12.650 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:12.650 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:12.651 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:12.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.651 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:12.651 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:12.651 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:12.652 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:12.652 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:12.652 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:12.652 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:12.652 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:12.653 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:12.653 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:12.653 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:12.653 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:21:12.654 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_bias": 4, 2021-03-15T22:21:12.654 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 8 2021-03-15T22:21:12.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.654 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:12.654 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:21:12.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.655 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:21:12.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2021-03-15T22:21:12.656 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:20:03.377294+0000", 2021-03-15T22:21:12.656 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2021-03-15T22:21:12.656 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-03-15T22:21:12.656 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:21:12.657 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:21:12.657 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:21:12.657 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:21:12.657 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:21:12.658 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:21:12.658 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:21:12.658 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:21:12.658 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:21:12.658 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:21:12.659 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2021-03-15T22:21:12.659 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2021-03-15T22:21:12.659 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2021-03-15T22:21:12.659 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2021-03-15T22:21:12.659 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2021-03-15T22:21:12.660 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:21:12.660 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:21:12.660 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:21:12.660 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:21:12.660 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:21:12.661 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:21:12.661 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:21:12.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.661 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "157", 2021-03-15T22:21:12.661 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:21:12.662 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:21:12.662 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:21:12.662 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:21:12.662 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:21:12.663 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 60, 2021-03-15T22:21:12.663 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 157, 2021-03-15T22:21:12.663 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:21:12.663 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:21:12.663 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:21:12.664 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:21:12.664 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:21:12.664 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:21:12.664 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:21:12.664 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:21:12.665 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:21:12.665 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:21:12.665 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:21:12.665 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:21:12.666 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:21:12.666 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:21:12.666 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:21:12.666 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:21:12.666 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:21:12.667 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:21:12.667 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:21:12.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.667 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:21:12.667 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:21:12.668 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:21:12.668 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:21:12.668 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:21:12.668 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:21:12.669 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:21:12.669 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:21:12.669 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:21:12.669 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:21:12.669 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:21:12.670 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:21:12.670 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:21:12.670 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2021-03-15T22:21:12.670 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.670 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.671 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:12.671 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:21:12.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.671 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:21:12.671 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd", 2021-03-15T22:21:12.672 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:12.672 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:12.672 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:12.672 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:12.672 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:12.673 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:12.673 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 8, 2021-03-15T22:21:12.673 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 91, 2021-03-15T22:21:12.673 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:12.673 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:12.674 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:12.674 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.674 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.675 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2021-03-15T22:21:12.675 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:12.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.675 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.676 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2021-03-15T22:21:12.676 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:12.676 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.676 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.677 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:12.677 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.677 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.677 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2021-03-15T22:21:12.678 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:12.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.678 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.679 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2021-03-15T22:21:12.679 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:12.679 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.679 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.680 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:12.680 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.680 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.681 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2021-03-15T22:21:12.681 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:12.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.681 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.682 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2021-03-15T22:21:12.682 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:12.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.682 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.683 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:12.683 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.683 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.684 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2021-03-15T22:21:12.684 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:12.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.684 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2021-03-15T22:21:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:21:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/188228176", 2021-03-15T22:21:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/188228176", 2021-03-15T22:21:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/188228176", 2021-03-15T22:21:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/188228176", 2021-03-15T22:21:12.686 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:21:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526", 2021-03-15T22:21:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:12.688 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:12.689 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 13, 2021-03-15T22:21:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 156, 2021-03-15T22:21:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:12.690 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2021-03-15T22:21:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:12.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2021-03-15T22:21:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:12.692 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2021-03-15T22:21:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:12.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2021-03-15T22:21:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:12.695 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.696 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.696 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:12.696 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.697 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.697 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2021-03-15T22:21:12.697 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:12.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.698 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.698 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2021-03-15T22:21:12.698 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:12.698 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.698 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.699 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:12.699 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.700 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.700 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2021-03-15T22:21:12.700 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:12.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.701 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.701 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2021-03-15T22:21:12.701 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:21:12.701 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.701 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.702 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/3924955423", 2021-03-15T22:21:12.702 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/3924955423", 2021-03-15T22:21:12.702 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/3924955423", 2021-03-15T22:21:12.702 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/3924955423", 2021-03-15T22:21:12.703 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:12.703 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:12.703 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:12.703 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.704 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:21:12.704 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05", 2021-03-15T22:21:12.704 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:12.705 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:12.705 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:12.705 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:12.705 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:12.705 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:12.706 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 18, 2021-03-15T22:21:12.706 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 154, 2021-03-15T22:21:12.706 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:12.706 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:12.706 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:12.707 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.707 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.707 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2021-03-15T22:21:12.707 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:12.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.708 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.708 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2021-03-15T22:21:12.708 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:12.709 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.709 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.709 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:12.710 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.710 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.710 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2021-03-15T22:21:12.710 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:12.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.711 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.711 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2021-03-15T22:21:12.711 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:12.712 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.712 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.712 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:12.712 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.713 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.713 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2021-03-15T22:21:12.713 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:12.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.714 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.714 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2021-03-15T22:21:12.714 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:12.715 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.715 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.715 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:12.715 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.716 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.716 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2021-03-15T22:21:12.716 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:12.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.717 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.717 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2021-03-15T22:21:12.717 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:21:12.717 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.718 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.718 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/1401263050", 2021-03-15T22:21:12.718 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/1401263050", 2021-03-15T22:21:12.719 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/1401263050", 2021-03-15T22:21:12.719 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/1401263050", 2021-03-15T22:21:12.719 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:12.719 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:12.719 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:12.720 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.720 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:21:12.720 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56", 2021-03-15T22:21:12.721 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:12.721 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:12.721 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:12.721 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:12.721 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:12.722 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:12.722 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 26, 2021-03-15T22:21:12.722 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 91, 2021-03-15T22:21:12.722 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:12.723 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:12.723 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:12.723 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.723 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.724 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2021-03-15T22:21:12.724 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:12.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.724 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.725 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2021-03-15T22:21:12.725 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:12.725 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.725 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.726 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:12.726 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.726 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.727 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2021-03-15T22:21:12.727 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:12.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.727 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.728 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2021-03-15T22:21:12.728 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:12.728 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.728 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.729 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:12.729 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.729 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.729 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2021-03-15T22:21:12.730 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:12.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.730 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.730 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2021-03-15T22:21:12.731 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:12.731 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.731 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.732 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:12.732 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.732 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.732 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2021-03-15T22:21:12.733 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:12.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.733 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.733 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2021-03-15T22:21:12.734 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:21:12.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.734 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.734 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/3960968636", 2021-03-15T22:21:12.735 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/3960968636", 2021-03-15T22:21:12.735 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/3960968636", 2021-03-15T22:21:12.735 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/3960968636", 2021-03-15T22:21:12.735 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:12.736 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:12.736 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:12.736 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.737 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:21:12.737 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237", 2021-03-15T22:21:12.737 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:12.737 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:12.737 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:12.738 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:12.738 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:12.738 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:12.738 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 31, 2021-03-15T22:21:12.738 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 150, 2021-03-15T22:21:12.739 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:12.739 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:12.739 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:12.739 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.740 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.740 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6800", 2021-03-15T22:21:12.740 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:12.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.741 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.741 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6801", 2021-03-15T22:21:12.741 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:12.741 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.741 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.742 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:12.742 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.743 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.743 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6802", 2021-03-15T22:21:12.743 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:12.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.744 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.744 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6803", 2021-03-15T22:21:12.744 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:12.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.744 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.745 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:12.745 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.746 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6806", 2021-03-15T22:21:12.746 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:12.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.747 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.747 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6807", 2021-03-15T22:21:12.747 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:12.747 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.747 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.748 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:12.748 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.748 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.749 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6804", 2021-03-15T22:21:12.749 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:12.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.750 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6805", 2021-03-15T22:21:12.750 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:21:12.750 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.750 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.751 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6801/37386122", 2021-03-15T22:21:12.751 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6803/37386122", 2021-03-15T22:21:12.751 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6807/37386122", 2021-03-15T22:21:12.751 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6805/37386122", 2021-03-15T22:21:12.752 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:12.752 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:12.752 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:12.752 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.753 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:21:12.753 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2", 2021-03-15T22:21:12.753 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:12.753 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:12.754 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:12.754 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:12.754 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:12.754 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:12.754 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 38, 2021-03-15T22:21:12.755 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 146, 2021-03-15T22:21:12.755 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:12.755 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:12.755 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:12.755 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.756 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.756 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6808", 2021-03-15T22:21:12.756 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:12.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.757 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.757 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6809", 2021-03-15T22:21:12.757 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:12.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.758 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.758 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:12.758 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.759 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6810", 2021-03-15T22:21:12.759 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:12.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6811", 2021-03-15T22:21:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:12.760 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.761 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.761 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:12.761 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.762 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6814", 2021-03-15T22:21:12.762 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:12.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.763 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6815", 2021-03-15T22:21:12.763 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:12.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.764 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.764 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:12.764 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.765 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.765 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6812", 2021-03-15T22:21:12.765 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:12.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.766 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.766 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6813", 2021-03-15T22:21:12.766 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:21:12.766 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.767 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.767 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6809/3610838738", 2021-03-15T22:21:12.767 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6811/3610838738", 2021-03-15T22:21:12.767 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6815/3610838738", 2021-03-15T22:21:12.768 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6813/3610838738", 2021-03-15T22:21:12.768 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:12.768 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:12.768 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:12.768 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.769 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:21:12.769 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe", 2021-03-15T22:21:12.769 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:12.770 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:12.770 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:12.770 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:12.770 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:12.770 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:12.771 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 45, 2021-03-15T22:21:12.771 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 91, 2021-03-15T22:21:12.771 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:12.771 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:12.772 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:12.772 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.772 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.772 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6816", 2021-03-15T22:21:12.773 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:12.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.773 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.773 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6817", 2021-03-15T22:21:12.774 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:12.774 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.774 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.774 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:12.775 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.775 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.775 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6818", 2021-03-15T22:21:12.775 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:12.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.776 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.776 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6819", 2021-03-15T22:21:12.777 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:12.777 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.777 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.777 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:12.778 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.778 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.778 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6822", 2021-03-15T22:21:12.778 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:12.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.779 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.779 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6823", 2021-03-15T22:21:12.779 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:12.780 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.780 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.780 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:12.780 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.781 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6820", 2021-03-15T22:21:12.781 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:12.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.782 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.782 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6821", 2021-03-15T22:21:12.782 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:21:12.783 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.783 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.783 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6817/171378275", 2021-03-15T22:21:12.783 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6819/171378275", 2021-03-15T22:21:12.784 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6823/171378275", 2021-03-15T22:21:12.784 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6821/171378275", 2021-03-15T22:21:12.784 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:12.784 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:12.784 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:12.785 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.785 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:21:12.786 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614", 2021-03-15T22:21:12.786 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:21:12.786 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:21:12.786 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:21:12.786 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:21:12.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:21:12.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:21:12.787 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 52, 2021-03-15T22:21:12.787 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 140, 2021-03-15T22:21:12.787 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:21:12.788 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:21:12.788 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:21:12.788 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.788 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.789 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6824", 2021-03-15T22:21:12.789 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:12.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.789 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.790 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6825", 2021-03-15T22:21:12.790 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:12.790 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.790 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.791 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:21:12.791 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.791 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.792 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6826", 2021-03-15T22:21:12.792 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:12.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.792 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.793 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6827", 2021-03-15T22:21:12.793 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:12.793 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.793 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.794 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:21:12.794 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.794 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.794 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6830", 2021-03-15T22:21:12.795 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:12.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.795 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.796 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6831", 2021-03-15T22:21:12.796 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:12.796 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.796 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.797 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:21:12.797 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:21:12.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:21:12.797 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6828", 2021-03-15T22:21:12.798 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:12.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.798 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:21:12.798 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6829", 2021-03-15T22:21:12.799 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:21:12.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.799 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6825/652003948", 2021-03-15T22:21:12.800 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6827/652003948", 2021-03-15T22:21:12.800 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6831/652003948", 2021-03-15T22:21:12.800 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6829/652003948", 2021-03-15T22:21:12.800 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:21:12.801 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:21:12.801 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:21:12.801 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.801 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:12.802 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2021-03-15T22:21:12.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.802 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:21:12.802 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:12.802 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:12.803 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:12.803 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:12.803 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:12.803 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:51.037646+0000", 2021-03-15T22:21:12.803 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:12.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.804 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:21:12.804 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:12.804 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:12.805 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:12.805 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:12.805 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:12.805 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:21:08.603372+0000", 2021-03-15T22:21:12.805 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:12.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.806 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:21:12.806 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:12.807 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:12.807 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:12.807 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:12.807 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:12.807 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:24.332975+0000", 2021-03-15T22:21:12.808 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:12.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.808 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:21:12.808 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:12.809 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:12.809 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:12.809 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:12.809 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:12.809 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:39.289836+0000", 2021-03-15T22:21:12.810 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:12.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.810 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:21:12.810 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:12.811 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:12.811 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:12.811 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:12.811 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:12.812 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:57.378352+0000", 2021-03-15T22:21:12.812 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:12.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.812 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:21:12.813 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:12.813 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:12.813 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:12.813 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:12.813 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:12.814 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:14.686550+0000", 2021-03-15T22:21:12.814 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:12.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.814 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:21:12.815 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:12.815 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:12.815 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:12.815 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:12.816 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:12.816 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:31.843914+0000", 2021-03-15T22:21:12.816 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:12.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.817 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:21:12.817 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:21:12.817 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:21:12.817 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:21:12.817 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:21:12.818 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:21:12.818 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:20:50.252442+0000", 2021-03-15T22:21:12.818 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:21:12.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.818 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:12.819 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [], 2021-03-15T22:21:12.819 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2021-03-15T22:21:12.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.819 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.b", 2021-03-15T22:21:12.819 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:21:12.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.820 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:21:12.820 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:21:12.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.821 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.821 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2021-03-15T22:21:12.821 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:21:12.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.822 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:21:12.822 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:21:12.822 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.822 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.823 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.1b", 2021-03-15T22:21:12.823 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:21:12.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.824 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:21:12.824 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 5 2021-03-15T22:21:12.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.824 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.825 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:12.825 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2021-03-15T22:21:12.825 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2021-03-15T22:21:12.825 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": { 2021-03-15T22:21:12.825 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:21:12.826 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4087324200": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:21:12.826 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4141143304": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:21:12.826 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:21:12.826 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2779879991": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:21:12.826 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:21:12.827 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:21:12.827 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2483102204": "2021-03-16T22:13:41.603652+0000" 2021-03-15T22:21:12.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.827 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2021-03-15T22:21:12.828 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2021-03-15T22:21:12.828 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2021-03-15T22:21:12.828 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2021-03-15T22:21:12.828 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2021-03-15T22:21:12.828 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2021-03-15T22:21:12.829 INFO:teuthology.orchestra.run.smithi076.stdout: "ruleset-failure-domain": "osd", 2021-03-15T22:21:12.829 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2021-03-15T22:21:12.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.829 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2021-03-15T22:21:12.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.830 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:21:12.830 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:21:12.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.830 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 50, 2021-03-15T22:21:12.831 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:21:12.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.831 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 52, 2021-03-15T22:21:12.831 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:21:12.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:12.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.832 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 56, 2021-03-15T22:21:12.832 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:21:12.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.833 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.833 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.833 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:12.833 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [ 2021-03-15T22:21:12.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.834 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:21:12.834 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:21:12.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:12.834 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 56, 2021-03-15T22:21:12.835 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:21:12.835 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.835 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:12.835 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.835 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:21:12.836 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2021-03-15T22:21:12.836 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2021-03-15T22:21:12.836 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2021-03-15T22:21:12.836 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2021-03-15T22:21:12.836 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2021-03-15T22:21:12.837 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2021-03-15T22:21:12.837 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2021-03-15T22:21:12.837 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2021-03-15T22:21:12.837 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2021-03-15T22:21:12.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:12.838 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:21:12.838 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:12.840 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-15T22:21:12.861 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:12.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:12.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:12.963 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:12.985 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:12.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:12.987 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:13.029 INFO:tasks.rados.rados.0.smithi076.stdout:324: done (0 left) 2021-03-15T22:21:13.029 INFO:tasks.rados.rados.0.smithi076.stdout:325: read oid 12 snap 30 2021-03-15T22:21:13.029 INFO:tasks.rados.rados.0.smithi076.stdout:325: expect (ObjNum 128 snap 24 seq_num 128) 2021-03-15T22:21:13.030 INFO:tasks.rados.rados.0.smithi076.stdout:326: read oid 35 snap -1 2021-03-15T22:21:13.030 INFO:tasks.rados.rados.0.smithi076.stdout:326: expect deleted 2021-03-15T22:21:13.030 INFO:tasks.rados.rados.0.smithi076.stdout:327: read oid 19 snap 24 2021-03-15T22:21:13.030 INFO:tasks.rados.rados.0.smithi076.stdout:327: expect (ObjNum 9 snap 0 seq_num 9) 2021-03-15T22:21:13.031 INFO:tasks.rados.rados.0.smithi076.stdout:328: read oid 24 snap -1 2021-03-15T22:21:13.031 INFO:tasks.rados.rados.0.smithi076.stdout:328: expect (ObjNum 106 snap 22 seq_num 106) 2021-03-15T22:21:13.031 INFO:tasks.rados.rados.0.smithi076.stdout:329: read oid 40 snap -1 2021-03-15T22:21:13.031 INFO:tasks.rados.rados.0.smithi076.stdout:329: expect (ObjNum 68 snap 6 seq_num 68) 2021-03-15T22:21:13.032 INFO:tasks.rados.rados.0.smithi076.stdout:330: write oid 28 current snap is 31 2021-03-15T22:21:13.032 INFO:tasks.rados.rados.0.smithi076.stdout:330: seq_num 147 ranges {470629=472310,1418853=513377} 2021-03-15T22:21:13.035 INFO:tasks.rados.rados.0.smithi076.stdout:330: writing smithi07694878-28 from 470629 to 942939 tid 1 2021-03-15T22:21:13.042 INFO:tasks.rados.rados.0.smithi076.stdout:330: writing smithi07694878-28 from 1418853 to 1932230 tid 2 2021-03-15T22:21:13.043 INFO:tasks.rados.rados.0.smithi076.stdout:331: read oid 26 snap 24 2021-03-15T22:21:13.043 INFO:tasks.rados.rados.0.smithi076.stdout:331: expect (ObjNum 0 snap 0 seq_num 0) 2021-03-15T22:21:13.043 INFO:tasks.rados.rados.0.smithi076.stdout:332: copy_from oid 17 from oid 44 current snap is 31 2021-03-15T22:21:13.043 INFO:tasks.rados.rados.0.smithi076.stdout:325: expect (ObjNum 128 snap 24 seq_num 128) 2021-03-15T22:21:13.088 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:13.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:13.112 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:13.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:13.116 INFO:tasks.rados.rados.0.smithi076.stdout:327: expect (ObjNum 9 snap 0 seq_num 9) 2021-03-15T22:21:13.122 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:13.191 INFO:tasks.rados.rados.0.smithi076.stdout:328: expect (ObjNum 106 snap 22 seq_num 106) 2021-03-15T22:21:13.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:13.236 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:13.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:13.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:13.240 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:13 smithi076 conmon[62156]: cluster 2021-03-15T22:21:12 2021-03-15T22:21:13.240 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:13 smithi076 conmon[62156]: .015653+0000 mon.a (mon.0) 727 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2021-03-15T22:21:13.240 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:13 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:21:13.240 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:13 smithi076 conmon[62156]: :21:12.546454+0000 mon.a (mon.0 2021-03-15T22:21:13.241 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:13 smithi076 conmon[62156]: ) 728 : audit [DBG] from='client.? 172.21.15.76:0/331021168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:21:13.241 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:13 smithi076 conmon[62156]: cluster 2021-03-15T22:21:12.992056 2021-03-15T22:21:13.241 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:13 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v363: 145 pgs: 2 clean+premerge+peered, 1 peering, 142 active+clean; 137 MiB data, 426 MiB used, 715 GiB / 715 GiB avail; 9.4 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2021-03-15T22:21:13.241 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:13 smithi076 conmon[62156]: 2021-03-15T22:21:13.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:13 smithi076 conmon[56514]: cluster 2021-03-15T22:21:12.015653+0000 mon.a (mon.0) 727 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2021-03-15T22:21:13.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:13 smithi076 conmon[56514]: audit 2021-03-15T22:21:12.546454+0000 mon.a (mon.0) 728 : audit [DBG] from='client.? 172.21.15.76:0/331021168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:21:13.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:13 smithi076 conmon[56514]: cluster 2021-03-15T22:21:12.992056+0000 mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v363: 145 pgs: 2 clean+premerge+peered, 1 peering, 142 active+clean; 137 MiB data, 426 MiB used, 715 GiB / 715 GiB avail; 9.4 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2021-03-15T22:21:13.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:13 smithi125 conmon[56739]: cluster 2021-03-15T22:21:13.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:13 smithi125 conmon[56739]: 2021 2021-03-15T22:21:13.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:13 smithi125 conmon[56739]: -03-15T22:21:12.015653+0000 mon.a (mon.0 2021-03-15T22:21:13.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:13 smithi125 conmon[56739]: ) 727 : cluster [DBG] osdmap e157: 8 total, 8 up, 8 in 2021-03-15T22:21:13.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:13 smithi125 conmon[56739]: audit 2021-03-15T22:21:13.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:13 smithi125 conmon[56739]: 2021-03-15T22:21:12.546454+0000 mon.a (mon 2021-03-15T22:21:13.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:13 smithi125 conmon[56739]: .0) 728 : audit [DBG] from='client.? 172.21.15.76:0/331021168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:21:13.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:13 smithi125 conmon[56739]: cluster 2021-03-15T22:21:12.992056+0000 2021-03-15T22:21:13.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:13 smithi125 conmon[56739]: mgr.y (mgr.14144) 414 : cluster [DBG] pgmap v363: 145 pgs: 2 clean+premerge+peered, 1 peering, 142 active+clean; 137 MiB data, 426 MiB used, 715 GiB / 715 GiB avail; 9.4 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2021-03-15T22:21:13.339 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:13.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:13.363 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:13.364 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:13.406 INFO:tasks.rados.rados.0.smithi076.stdout:329: expect (ObjNum 68 snap 6 seq_num 68) 2021-03-15T22:21:13.465 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:13.486 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:13.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:13.544 INFO:tasks.rados.rados.0.smithi076.stdout:330: finishing write tid 1 to smithi07694878-28 2021-03-15T22:21:13.545 INFO:tasks.rados.rados.0.smithi076.stdout:331: expect (ObjNum 0 snap 0 seq_num 0) 2021-03-15T22:21:13.590 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:13.608 INFO:tasks.rados.rados.0.smithi076.stdout:330: finishing write tid 2 to smithi07694878-28 2021-03-15T22:21:13.609 INFO:tasks.rados.rados.0.smithi076.stdout:330: finishing write tid 3 to smithi07694878-28 2021-03-15T22:21:13.609 INFO:tasks.rados.rados.0.smithi076.stdout:330: finishing write tid 4 to smithi07694878-28 2021-03-15T22:21:13.609 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 201 (ObjNum 147 snap 31 seq_num 147) dirty exists 2021-03-15T22:21:13.609 INFO:tasks.rados.rados.0.smithi076.stdout:330: left oid 28 (ObjNum 147 snap 31 seq_num 147) 2021-03-15T22:21:13.609 INFO:tasks.rados.rados.0.smithi076.stdout:332: finishing copy_from to smithi07694878-17 2021-03-15T22:21:13.610 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 49 (ObjNum 99 snap 20 seq_num 99) dirty exists 2021-03-15T22:21:13.610 INFO:tasks.rados.rados.0.smithi076.stdout:332: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:21:13.610 INFO:tasks.rados.rados.0.smithi076.stdout:325: done (7 left) 2021-03-15T22:21:13.610 INFO:tasks.rados.rados.0.smithi076.stdout:326: done (6 left) 2021-03-15T22:21:13.611 INFO:tasks.rados.rados.0.smithi076.stdout:327: done (5 left) 2021-03-15T22:21:13.611 INFO:tasks.rados.rados.0.smithi076.stdout:328: done (4 left) 2021-03-15T22:21:13.611 INFO:tasks.rados.rados.0.smithi076.stdout:329: done (3 left) 2021-03-15T22:21:13.611 INFO:tasks.rados.rados.0.smithi076.stdout:330: done (2 left) 2021-03-15T22:21:13.611 INFO:tasks.rados.rados.0.smithi076.stdout:331: done (1 left) 2021-03-15T22:21:13.612 INFO:tasks.rados.rados.0.smithi076.stdout:332: done (0 left) 2021-03-15T22:21:13.612 INFO:tasks.rados.rados.0.smithi076.stdout:333: write oid 33 current snap is 31 2021-03-15T22:21:13.612 INFO:tasks.rados.rados.0.smithi076.stdout:333: seq_num 148 ranges {487436=547559,1760982=657035} 2021-03-15T22:21:13.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:13.613 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:13.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:13.614 INFO:tasks.rados.rados.0.smithi076.stdout:333: writing smithi07694878-33 from 487436 to 1034995 tid 1 2021-03-15T22:21:13.622 INFO:tasks.rados.rados.0.smithi076.stdout:333: writing smithi07694878-33 from 1760982 to 2418017 tid 2 2021-03-15T22:21:13.622 INFO:tasks.rados.rados.0.smithi076.stdout:334: write (excl) oid 30 current snap is 31 2021-03-15T22:21:13.622 INFO:tasks.rados.rados.0.smithi076.stdout:334: seq_num 149 ranges {504243=622808,1819464=517046} 2021-03-15T22:21:13.628 INFO:tasks.rados.rados.0.smithi076.stdout:334: writing smithi07694878-30 from 504243 to 1127051 tid 1 2021-03-15T22:21:13.636 INFO:tasks.rados.rados.0.smithi076.stdout:334: writing smithi07694878-30 from 1819464 to 2336510 tid 2 2021-03-15T22:21:13.636 INFO:tasks.rados.rados.0.smithi076.stdout:335: read oid 29 snap 30 2021-03-15T22:21:13.636 INFO:tasks.rados.rados.0.smithi076.stdout:335: expect (ObjNum 132 snap 25 seq_num 132) 2021-03-15T22:21:13.637 INFO:tasks.rados.rados.0.smithi076.stdout:336: rollback oid 49 current snap is 31 2021-03-15T22:21:13.637 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 49 to 30 2021-03-15T22:21:13.637 INFO:tasks.rados.rados.0.smithi076.stdout:333: finishing write tid 1 to smithi07694878-33 2021-03-15T22:21:13.637 INFO:tasks.rados.rados.0.smithi076.stdout:337: copy_from oid 19 from oid 27 current snap is 31 2021-03-15T22:21:13.638 INFO:tasks.rados.rados.0.smithi076.stdout:333: finishing write tid 2 to smithi07694878-33 2021-03-15T22:21:13.638 INFO:tasks.rados.rados.0.smithi076.stdout:333: finishing write tid 3 to smithi07694878-33 2021-03-15T22:21:13.638 INFO:tasks.rados.rados.0.smithi076.stdout:333: finishing write tid 4 to smithi07694878-33 2021-03-15T22:21:13.638 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 93 (ObjNum 148 snap 31 seq_num 148) dirty exists 2021-03-15T22:21:13.638 INFO:tasks.rados.rados.0.smithi076.stdout:333: left oid 33 (ObjNum 148 snap 31 seq_num 148) 2021-03-15T22:21:13.639 INFO:tasks.rados.rados.0.smithi076.stdout:334: finishing write tid 1 to smithi07694878-30 2021-03-15T22:21:13.639 INFO:tasks.rados.rados.0.smithi076.stdout:333: done (4 left) 2021-03-15T22:21:13.639 INFO:tasks.rados.rados.0.smithi076.stdout:338: read oid 2 snap 30 2021-03-15T22:21:13.639 INFO:tasks.rados.rados.0.smithi076.stdout:338: expect deleted 2021-03-15T22:21:13.639 INFO:tasks.rados.rados.0.smithi076.stdout:339: rollback oid 26 current snap is 31 2021-03-15T22:21:13.640 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 24 2021-03-15T22:21:13.640 INFO:tasks.rados.rados.0.smithi076.stdout:340: snap_create 2021-03-15T22:21:13.641 INFO:tasks.rados.rados.0.smithi076.stdout:339: finishing rollback tid 0 to smithi07694878-26 2021-03-15T22:21:13.641 INFO:tasks.rados.rados.0.smithi076.stdout:339: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:21:13.642 INFO:tasks.rados.rados.0.smithi076.stdout:339: finishing rollback tid 2 to smithi07694878-26 2021-03-15T22:21:13.643 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 133 (ObjNum 0 snap 0 seq_num 0) dirty exists 2021-03-15T22:21:13.643 INFO:tasks.rados.rados.0.smithi076.stdout:334: finishing write tid 2 to smithi07694878-30 2021-03-15T22:21:13.643 INFO:tasks.rados.rados.0.smithi076.stdout:334: finishing write tid 3 to smithi07694878-30 2021-03-15T22:21:13.643 INFO:tasks.rados.rados.0.smithi076.stdout:336: finishing rollback tid 1 to smithi07694878-49 2021-03-15T22:21:13.643 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 172 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:21:13.644 INFO:tasks.rados.rados.0.smithi076.stdout:334: finishing write tid 4 to smithi07694878-30 2021-03-15T22:21:13.644 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 184 (ObjNum 149 snap 31 seq_num 149) dirty exists 2021-03-15T22:21:13.644 INFO:tasks.rados.rados.0.smithi076.stdout:334: left oid 30 (ObjNum 149 snap 31 seq_num 149) 2021-03-15T22:21:13.645 INFO:tasks.rados.rados.0.smithi076.stdout:335: expect (ObjNum 132 snap 25 seq_num 132) 2021-03-15T22:21:13.715 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:13.737 INFO:tasks.rados.rados.0.smithi076.stdout:337: finishing copy_from to smithi07694878-19 2021-03-15T22:21:13.737 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 54 (ObjNum 101 snap 21 seq_num 101) dirty exists 2021-03-15T22:21:13.738 INFO:tasks.rados.rados.0.smithi076.stdout:337: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:21:13.739 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:13.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:13.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:13.754 INFO:tasks.thrashosds.thrasher:j is {'epoch': 157, 'fsid': '8a572802-85db-11eb-9091-001a4aab830c', 'created': '2021-03-15T22:13:20.370795+0000', 'modified': '2021-03-15T22:21:12.011933+0000', 'last_up_change': '2021-03-15T22:17:49.162889+0000', 'last_in_change': '2021-03-15T22:17:34.080423+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-15T22:16:34.817191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-15T22:17:49.906400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-15T22:17:52.257970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-15T22:17:54.562498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-15T22:17:56.243806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 29, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 28, 'last_pg_merge_meta': {'source_pgid': '5.1d', 'ready_epoch': 154, 'last_epoch_started': 153, 'last_epoch_clean': 153, 'source_version': "0'0", 'target_version': "61'2"}, 'last_change': '156', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '156', 'last_force_op_resend_preluminous': '156', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-15T22:17:58.730534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-03-15T22:20:03.377294+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '157', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 60, 'snap_epoch': 157, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 188228176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 188228176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 188228176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 188228176}]}, 'public_addr': '172.21.15.76:6803/188228176', 'cluster_addr': '172.21.15.76:6805/188228176', 'heartbeat_back_addr': '172.21.15.76:6809/188228176', 'heartbeat_front_addr': '172.21.15.76:6807/188228176', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '97bd3ffc-15c8-4a6f-b37a-217813def526', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 156, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 3924955423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 3924955423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 3924955423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 3924955423}]}, 'public_addr': '172.21.15.76:6811/3924955423', 'cluster_addr': '172.21.15.76:6813/3924955423', 'heartbeat_back_addr': '172.21.15.76:6817/3924955423', 'heartbeat_front_addr': '172.21.15.76:6815/3924955423', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a7f5f115-e69e-47d9-b6c3-ff386ddfaa05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 154, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 1401263050}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 1401263050}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 1401263050}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 1401263050}]}, 'public_addr': '172.21.15.76:6819/1401263050', 'cluster_addr': '172.21.15.76:6821/1401263050', 'heartbeat_back_addr': '172.21.15.76:6825/1401263050', 'heartbeat_front_addr': '172.21.15.76:6823/1401263050', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a0bb7f88-0114-47cb-a82c-d8fb002aac56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 3960968636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 3960968636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 3960968636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 3960968636}]}, 'public_addr': '172.21.15.76:6827/3960968636', 'cluster_addr': '172.21.15.76:6829/3960968636', 'heartbeat_back_addr': '172.21.15.76:6833/3960968636', 'heartbeat_front_addr': '172.21.15.76:6831/3960968636', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b650611c-6101-47e8-9ca9-94e02cf8b237', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 150, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6800', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6801', 'nonce': 37386122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6802', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6803', 'nonce': 37386122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6806', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6807', 'nonce': 37386122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6804', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6805', 'nonce': 37386122}]}, 'public_addr': '172.21.15.125:6801/37386122', 'cluster_addr': '172.21.15.125:6803/37386122', 'heartbeat_back_addr': '172.21.15.125:6807/37386122', 'heartbeat_front_addr': '172.21.15.125:6805/37386122', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7d6ceff2-7093-4ab9-8b96-ade448093bf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 146, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6808', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6809', 'nonce': 3610838738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6810', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6811', 'nonce': 3610838738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6814', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6815', 'nonce': 3610838738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6812', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6813', 'nonce': 3610838738}]}, 'public_addr': '172.21.15.125:6809/3610838738', 'cluster_addr': '172.21.15.125:6811/3610838738', 'heartbeat_back_addr': '172.21.15.125:6815/3610838738', 'heartbeat_front_addr': '172.21.15.125:6813/3610838738', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98b72022-a9a7-4ee2-89a1-cac037de7afe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 91, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6816', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6817', 'nonce': 171378275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6818', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6819', 'nonce': 171378275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6822', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6823', 'nonce': 171378275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6820', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6821', 'nonce': 171378275}]}, 'public_addr': '172.21.15.125:6817/171378275', 'cluster_addr': '172.21.15.125:6819/171378275', 'heartbeat_back_addr': '172.21.15.125:6823/171378275', 'heartbeat_front_addr': '172.21.15.125:6821/171378275', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f99ea650-b2dc-4d87-a44c-56a897edd614', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 140, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6824', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6825', 'nonce': 652003948}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6826', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6827', 'nonce': 652003948}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6830', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6831', 'nonce': 652003948}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6828', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6829', 'nonce': 652003948}]}, 'public_addr': '172.21.15.125:6825/652003948', 'cluster_addr': '172.21.15.125:6827/652003948', 'heartbeat_back_addr': '172.21.15.125:6831/652003948', 'heartbeat_front_addr': '172.21.15.125:6829/652003948', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:51.037646+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:21:08.603372+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:24.332975+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:39.289836+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:57.378352+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:14.686550+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:31.843914+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:20:50.252442+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.76:6800/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4087324200': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4141143304': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:6801/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/2779879991': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6801/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6800/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:0/2483102204': '2021-03-16T22:13:41.603652+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 50, 'length': 1}, {'begin': 52, 'length': 2}, {'begin': 56, 'length': 2}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 56, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-03-15T22:21:13.755 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2021-03-15T22:21:13.842 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:13.864 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:13.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:13.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:13.966 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:13.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:13.991 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:13.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:14.032 INFO:tasks.rados.rados.0.smithi076.stdout:334: done (6 left) 2021-03-15T22:21:14.033 INFO:tasks.rados.rados.0.smithi076.stdout:335: done (5 left) 2021-03-15T22:21:14.033 INFO:tasks.rados.rados.0.smithi076.stdout:336: done (4 left) 2021-03-15T22:21:14.033 INFO:tasks.rados.rados.0.smithi076.stdout:337: done (3 left) 2021-03-15T22:21:14.033 INFO:tasks.rados.rados.0.smithi076.stdout:338: done (2 left) 2021-03-15T22:21:14.034 INFO:tasks.rados.rados.0.smithi076.stdout:339: done (1 left) 2021-03-15T22:21:14.034 INFO:tasks.rados.rados.0.smithi076.stdout:340: done (0 left) 2021-03-15T22:21:14.034 INFO:tasks.rados.rados.0.smithi076.stdout:341: rollback oid 7 current snap is 32 2021-03-15T22:21:14.034 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 7 to 24 2021-03-15T22:21:14.035 INFO:tasks.rados.rados.0.smithi076.stdout:342: write oid 13 current snap is 32 2021-03-15T22:21:14.035 INFO:tasks.rados.rados.0.smithi076.stdout:342: seq_num 150 ranges {521050=698057,1877946=643104} 2021-03-15T22:21:14.037 INFO:tasks.rados.rados.0.smithi076.stdout:342: writing smithi07694878-13 from 521050 to 1219107 tid 1 2021-03-15T22:21:14.038 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:21:14.045 INFO:tasks.rados.rados.0.smithi076.stdout:342: writing smithi07694878-13 from 1877946 to 2521050 tid 2 2021-03-15T22:21:14.045 INFO:tasks.rados.rados.0.smithi076.stdout:343: copy_from oid 35 from oid 6 current snap is 32 2021-03-15T22:21:14.045 INFO:tasks.rados.rados.0.smithi076.stdout:341: finishing rollback tid 1 to smithi07694878-7 2021-03-15T22:21:14.045 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 0 (ObjNum 261868800 snap 21896 seq_num 260042768) dirty dne 2021-03-15T22:21:14.046 INFO:tasks.rados.rados.0.smithi076.stdout:341: done (2 left) 2021-03-15T22:21:14.046 INFO:tasks.rados.rados.0.smithi076.stdout:344: snap_create 2021-03-15T22:21:14.046 INFO:tasks.rados.rados.0.smithi076.stdout:342: finishing write tid 1 to smithi07694878-13 2021-03-15T22:21:14.048 INFO:tasks.rados.rados.0.smithi076.stdout:342: finishing write tid 2 to smithi07694878-13 2021-03-15T22:21:14.048 INFO:tasks.rados.rados.0.smithi076.stdout:342: finishing write tid 3 to smithi07694878-13 2021-03-15T22:21:14.048 INFO:tasks.rados.rados.0.smithi076.stdout:342: finishing write tid 4 to smithi07694878-13 2021-03-15T22:21:14.049 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 101 (ObjNum 150 snap 32 seq_num 150) dirty exists 2021-03-15T22:21:14.049 INFO:tasks.rados.rados.0.smithi076.stdout:342: left oid 13 (ObjNum 150 snap 32 seq_num 150) 2021-03-15T22:21:14.058 INFO:tasks.rados.rados.0.smithi076.stdout:343: finishing copy_from to smithi07694878-35 2021-03-15T22:21:14.059 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 14 (ObjNum 63 snap 6 seq_num 63) dirty exists 2021-03-15T22:21:14.059 INFO:tasks.rados.rados.0.smithi076.stdout:343: finishing copy_from racing read to smithi07694878-35 2021-03-15T22:21:14.093 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:14.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:14.117 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:14.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:14.218 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:14.242 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:14.243 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:21:14.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:14.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:14.259 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:21:14.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:14 smithi076 conmon[56514]: cluster 2021-03-15T22:21:13.024463+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2021-03-15T22:21:14.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:14 smithi076 conmon[56514]: audit 2021-03-15T22:21:13.984794+0000 mon.a (mon.0) 730 : audit [INF] from='client.? 172.21.15.76:0/2511661088' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:14.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:14 smithi076 conmon[62156]: cluster 2021-03-15T22:21:13.024463+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2021-03-15T22:21:14.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:14 smithi076 conmon[62156]: audit 2021-03-15T22:21:13.984794+0000 mon.a (mon.0) 730 : audit [INF] from='client.? 172.21.15.76:0/2511661088' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:14.277 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:21:14.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:14 smithi125 conmon[56739]: cluster 2021-03-15T22:21:13.024463+0000 mon.a (mon.0) 729 : cluster [DBG] osdmap e158: 8 total, 8 up, 8 in 2021-03-15T22:21:14.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:14 smithi125 conmon[56739]: audit 2021-03-15T22:21:13.984794+0000 mon.a (mon.0) 730 : audit [INF] from='client.? 172.21.15.76:0/2511661088' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:14.298 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:21:14.311 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:21:14.328 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:21:14.345 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:14.346 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:21:14.362 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:21:14.364 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:21:14.364 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:21:14.364 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:21:14.364 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:21:14.365 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:21:14.365 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:21:14.365 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:21:14.365 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:21:14.368 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:14.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:14.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:14.471 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:14.493 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:14.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:14.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:14.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:14.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:14.621 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:14.622 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:14.638 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:21:14.639 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:21:14.639 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:21:14.639 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:21:14.639 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:21:14.723 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:14.746 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:14.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:14.747 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:14.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:14.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:14.873 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:14.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:14.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:14.996 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:14.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:14.998 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:15.010 INFO:tasks.rados.rados.0.smithi076.stdout:342: done (2 left) 2021-03-15T22:21:15.010 INFO:tasks.rados.rados.0.smithi076.stdout:343: done (1 left) 2021-03-15T22:21:15.010 INFO:tasks.rados.rados.0.smithi076.stdout:344: done (0 left) 2021-03-15T22:21:15.011 INFO:tasks.rados.rados.0.smithi076.stdout:345: delete oid 45 current snap is 33 2021-03-15T22:21:15.015 INFO:tasks.rados.rados.0.smithi076.stdout:345: done (0 left) 2021-03-15T22:21:15.015 INFO:tasks.rados.rados.0.smithi076.stdout:346: write oid 19 current snap is 33 2021-03-15T22:21:15.015 INFO:tasks.rados.rados.0.smithi076.stdout:346: seq_num 152 ranges {554664=448555,1478557=664373} 2021-03-15T22:21:15.020 INFO:tasks.rados.rados.0.smithi076.stdout:346: writing smithi07694878-19 from 554664 to 1003219 tid 1 2021-03-15T22:21:15.028 INFO:tasks.rados.rados.0.smithi076.stdout:346: writing smithi07694878-19 from 1478557 to 2142930 tid 2 2021-03-15T22:21:15.028 INFO:tasks.rados.rados.0.smithi076.stdout:347: snap_create 2021-03-15T22:21:15.028 INFO:tasks.rados.rados.0.smithi076.stdout:346: finishing write tid 1 to smithi07694878-19 2021-03-15T22:21:15.030 INFO:tasks.rados.rados.0.smithi076.stdout:346: finishing write tid 2 to smithi07694878-19 2021-03-15T22:21:15.030 INFO:tasks.rados.rados.0.smithi076.stdout:346: finishing write tid 3 to smithi07694878-19 2021-03-15T22:21:15.032 INFO:tasks.rados.rados.0.smithi076.stdout:346: finishing write tid 4 to smithi07694878-19 2021-03-15T22:21:15.033 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 110 (ObjNum 152 snap 33 seq_num 152) dirty exists 2021-03-15T22:21:15.033 INFO:tasks.rados.rados.0.smithi076.stdout:346: left oid 19 (ObjNum 152 snap 33 seq_num 152) 2021-03-15T22:21:15.100 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:15.122 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:15.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:15.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:15.225 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:15.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:15.248 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:15.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:15.280 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[56514]: cluster 2021-03-15T22:21:14.019849+0000 mon.a 2021-03-15T22:21:15.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[56514]: (mon.0) 731 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:15.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[56514]: audit 2021-03-15T22:21:14.028663+0000 mon.a (mon.0) 732 : audit [INF] from='client.? 172.21.15.76:0/2511661088' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:15.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[56514]: cluster 2021-03-15T22:21:14.028726+0000 mon.a (mon.0) 733 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2021-03-15T22:21:15.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[56514]: audit 2021-03-15T22:21:14.223789+0000 mon.a (mon.0) 734 : audit [DBG] from='client.? 172.21.15.76:0/2842665606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:15.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[56514]: cluster 2021-03-15T22:21:14.992506+0000 mgr.y (mgr.14144) 415 : cluster [DBG] pgmap v366: 144 pgs: 1 clean+premerge+peered, 143 active+clean; 137 MiB data, 433 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2021-03-15T22:21:15.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[56514]: audit 2021-03-15T22:21:14.993278+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-03-15T22:21:15.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[56514]: audit 2021-03-15T22:21:15.007667+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2021-03-15T22:21:15.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[56514]: cluster 2021-03-15T22:21:15.007724+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2021-03-15T22:21:15.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: cluster 2021-03-15T22:21:15.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:21:15.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: 21:14.019849+0000 mon.a (mon.0) 731 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:15.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: audit 2021-03-15T22:21 2021-03-15T22:21:15.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: :14.028663+0000 mon.a (mon.0) 732 : 2021-03-15T22:21:15.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/2511661088' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:15.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: cluster 2021-03-15T22:21:15.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: 2021-03-15T22:21: 2021-03-15T22:21:15.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: 14.028726+0000 mon.a (mon 2021-03-15T22:21:15.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: .0) 733 : cluster [DBG] 2021-03-15T22:21:15.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: osdmap e159: 8 total, 8 up, 8 in 2021-03-15T22:21:15.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: audit 2021-03-15T22:21:15.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: 2021-03-15T22:21 2021-03-15T22:21:15.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: :14.223789+0000 mon.a ( 2021-03-15T22:21:15.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: mon.0) 734 : audit 2021-03-15T22:21:15.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/2842665606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:15.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: cluster 2021-03-15T22:21:15.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: 2021-03-15T22:21 2021-03-15T22:21:15.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: :14.992506+0000 mgr.y (mgr 2021-03-15T22:21:15.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: .14144) 415 : cluster [DBG] 2021-03-15T22:21:15.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: pgmap v366: 144 pgs: 1 clean+premerge+peered, 143 active+clean; 137 MiB data, 433 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2021-03-15T22:21:15.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: audit 2021-03-15T22:21:15.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: 2021-03-15T22:21 2021-03-15T22:21:15.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: :14.993278+0000 mon.a ( 2021-03-15T22:21:15.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: mon.0) 735 : audit 2021-03-15T22:21:15.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-03-15T22:21:15.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: audit 2021-03-15T22:21:15.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: 2021-03-15T22:21 2021-03-15T22:21:15.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: :15.007667+0000 mon.a ( 2021-03-15T22:21:15.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: mon.0) 736 : audit 2021-03-15T22:21:15.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2021-03-15T22:21:15.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: cluster 2021-03-15T22:21:15.290 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: 2021-03-15T22:21: 2021-03-15T22:21:15.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: 15.007724+0000 mon.a (mon 2021-03-15T22:21:15.291 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:15 smithi076 conmon[62156]: .0) 737 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2021-03-15T22:21:15.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:15 smithi125 conmon[56739]: cluster 2021-03-15T22:21:14.019849+0000 mon.a (mon 2021-03-15T22:21:15.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:15 smithi125 conmon[56739]: .0) 731 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:15.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:15 smithi125 conmon[56739]: audit 2021-03-15T22:21:14.028663+0000 mon.a (mon.0) 732 : audit [INF] from='client.? 172.21.15.76:0/2511661088' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:15.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:15 smithi125 conmon[56739]: cluster 2021-03-15T22:21:14.028726+0000 mon.a (mon.0) 733 : cluster [DBG] osdmap e159: 8 total, 8 up, 8 in 2021-03-15T22:21:15.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:15 smithi125 conmon[56739]: audit 2021-03-15T22:21:14.223789+0000 mon.a (mon.0) 734 : audit [DBG] from='client.? 172.21.15.76:0/2842665606' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:15.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:15 smithi125 conmon[56739]: cluster 2021-03-15T22:21:14.992506+0000 mgr.y (mgr.14144) 415 : cluster [DBG] pgmap v366: 144 pgs: 1 clean+premerge+peered, 143 active+clean; 137 MiB data, 433 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 16 op/s 2021-03-15T22:21:15.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:15 smithi125 conmon[56739]: audit 2021-03-15T22:21:14.993278+0000 mon.a (mon.0) 735 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-03-15T22:21:15.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:15 smithi125 conmon[56739]: audit 2021-03-15T22:21:15.007667+0000 mon.a (mon.0) 736 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "27"}]': finished 2021-03-15T22:21:15.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:15 smithi125 conmon[56739]: cluster 2021-03-15T22:21:15.007724+0000 mon.a (mon.0) 737 : cluster [DBG] osdmap e160: 8 total, 8 up, 8 in 2021-03-15T22:21:15.344 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-15T22:21:15.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:15.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:15.373 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:15.374 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:15.475 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:15.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:15.498 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:15.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:15.599 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:15.619 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:15.621 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:15.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:15.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:15.724 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:15.746 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:15.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:15.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:15.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:15.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:15.874 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:15.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:15.976 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:15.999 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:16.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:16.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:16.024 INFO:tasks.rados.rados.0.smithi076.stdout:346: done (1 left) 2021-03-15T22:21:16.024 INFO:tasks.rados.rados.0.smithi076.stdout:347: done (0 left) 2021-03-15T22:21:16.024 INFO:tasks.rados.rados.0.smithi076.stdout:348: write (excl) oid 46 current snap is 34 2021-03-15T22:21:16.025 INFO:tasks.rados.rados.0.smithi076.stdout:348: seq_num 153 ranges {571471=640157,1653392=524384} 2021-03-15T22:21:16.030 INFO:tasks.rados.rados.0.smithi076.stdout:348: writing smithi07694878-46 from 571471 to 1211628 tid 1 2021-03-15T22:21:16.037 INFO:tasks.rados.rados.0.smithi076.stdout:348: writing smithi07694878-46 from 1653392 to 2177776 tid 2 2021-03-15T22:21:16.037 INFO:tasks.rados.rados.0.smithi076.stdout:349: copy_from oid 40 from oid 14 current snap is 34 2021-03-15T22:21:16.037 INFO:tasks.rados.rados.0.smithi076.stdout:350: read oid 6 snap -1 2021-03-15T22:21:16.038 INFO:tasks.rados.rados.0.smithi076.stdout:350: expect (ObjNum 63 snap 6 seq_num 63) 2021-03-15T22:21:16.038 INFO:tasks.rados.rados.0.smithi076.stdout:351: rollback oid 39 current snap is 34 2021-03-15T22:21:16.038 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 39 to 30 2021-03-15T22:21:16.038 INFO:tasks.rados.rados.0.smithi076.stdout:352: delete oid 42 current snap is 34 2021-03-15T22:21:16.039 INFO:tasks.rados.rados.0.smithi076.stdout:348: finishing write tid 1 to smithi07694878-46 2021-03-15T22:21:16.040 INFO:tasks.rados.rados.0.smithi076.stdout:349: finishing copy_from to smithi07694878-40 2021-03-15T22:21:16.040 INFO:tasks.rados.rados.0.smithi076.stdout:349: got expected ENOENT (src dne) 2021-03-15T22:21:16.040 INFO:tasks.rados.rados.0.smithi076.stdout:349: finishing copy_from racing read to smithi07694878-40 2021-03-15T22:21:16.041 INFO:tasks.rados.rados.0.smithi076.stdout:349: done (4 left) 2021-03-15T22:21:16.041 INFO:tasks.rados.rados.0.smithi076.stdout:352: done (3 left) 2021-03-15T22:21:16.041 INFO:tasks.rados.rados.0.smithi076.stdout:353: snap_create 2021-03-15T22:21:16.043 INFO:tasks.rados.rados.0.smithi076.stdout:348: finishing write tid 2 to smithi07694878-46 2021-03-15T22:21:16.043 INFO:tasks.rados.rados.0.smithi076.stdout:348: finishing write tid 3 to smithi07694878-46 2021-03-15T22:21:16.044 INFO:tasks.rados.rados.0.smithi076.stdout:348: finishing write tid 4 to smithi07694878-46 2021-03-15T22:21:16.044 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 7 (ObjNum 153 snap 34 seq_num 153) dirty exists 2021-03-15T22:21:16.044 INFO:tasks.rados.rados.0.smithi076.stdout:348: left oid 46 (ObjNum 153 snap 34 seq_num 153) 2021-03-15T22:21:16.046 INFO:tasks.rados.rados.0.smithi076.stdout:350: expect (ObjNum 63 snap 6 seq_num 63) 2021-03-15T22:21:16.097 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:16 smithi125 conmon[56739]: cluster 2021-03-15T22:21:15.426642+0000 osd.5 (osd.5) 36 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:16.097 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:16 smithi125 conmon[56739]: cluster 2021-03-15T22:21:15.426840+0000 osd.5 (osd.5) 37 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:16.098 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:16 smithi125 conmon[56739]: cluster 2021-03-15T22:21:16.021992+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2021-03-15T22:21:16.101 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:16.125 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:16.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:16.127 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:16.180 INFO:tasks.rados.rados.0.smithi076.stdout:351: finishing rollback tid 0 to smithi07694878-39 2021-03-15T22:21:16.180 INFO:tasks.rados.rados.0.smithi076.stdout:351: finishing rollback tid 1 to smithi07694878-39 2021-03-15T22:21:16.180 INFO:tasks.rados.rados.0.smithi076.stdout:351: finishing rollback tid 2 to smithi07694878-39 2021-03-15T22:21:16.181 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 114 (ObjNum 116 snap 24 seq_num 116) dirty exists 2021-03-15T22:21:16.228 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:16.250 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:16.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:16.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:16.353 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:16.374 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:16.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:16.376 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:16.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:16.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:16.500 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:16.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:16.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[56514]: cluster 2021-03-15T22:21:15.426642+0000 2021-03-15T22:21:16.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[56514]: osd.5 (osd.5) 36 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:16.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[56514]: cluster 2021-03-15T22:21:15.426840+0000 osd.5 (osd.5) 37 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:16.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:21:16.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[56514]: 15T22:21:16.021992+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2021-03-15T22:21:16.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[62156]: cluster 2021-03-15T22:21:15.426642+0000 osd.5 (osd.5 2021-03-15T22:21:16.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[62156]: ) 36 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:16.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[62156]: cluster 2021-03-15T22:21:15.426840+0000 osd.5 (osd.5 2021-03-15T22:21:16.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[62156]: ) 37 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:16.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:16.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[62156]: 03-15T22:21:16.021992+0000 mon.a (mon.0) 738 : cluster [DBG] osdmap e161: 8 total, 8 up, 8 in 2021-03-15T22:21:16.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:16 smithi076 conmon[62156]: 2021-03-15T22:21:16.601 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:16.623 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:16.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:16.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:16.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:16.751 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:16.752 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:16.854 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:16.878 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:16.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:16.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:16.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:17.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:17.004 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:17.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:17.039 INFO:tasks.rados.rados.0.smithi076.stdout:348: done (3 left) 2021-03-15T22:21:17.039 INFO:tasks.rados.rados.0.smithi076.stdout:350: done (2 left) 2021-03-15T22:21:17.039 INFO:tasks.rados.rados.0.smithi076.stdout:351: done (1 left) 2021-03-15T22:21:17.039 INFO:tasks.rados.rados.0.smithi076.stdout:353: done (0 left) 2021-03-15T22:21:17.040 INFO:tasks.rados.rados.0.smithi076.stdout:354: read oid 33 snap -1 2021-03-15T22:21:17.040 INFO:tasks.rados.rados.0.smithi076.stdout:354: expect (ObjNum 148 snap 31 seq_num 148) 2021-03-15T22:21:17.040 INFO:tasks.rados.rados.0.smithi076.stdout:355: snap_remove snap 31 2021-03-15T22:21:17.105 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:17.115 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:21:17.123 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:21:17.124 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:21:17.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:21:17.124 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:21:17.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.125 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:269 7.0 7:054fa7db:::smithi07694878-29:head [write 591613~454162,stat] snapc 8=[8] ondisk+write+known_if_redirected e104)", 2021-03-15T22:21:17.125 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.208860+0000", 2021-03-15T22:21:17.125 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 57.910682107, 2021-03-15T22:21:17.125 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0058167749999999997, 2021-03-15T22:21:17.125 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.126 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.126 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.126 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.126 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.127 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 269 2021-03-15T22:21:17.127 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.127 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.127 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.128 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.128 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.208860+0000", 2021-03-15T22:21:17.128 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.128 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.129 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.129 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.208860+0000", 2021-03-15T22:21:17.129 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047850900000000001 2021-03-15T22:21:17.129 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.130 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.130 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.130 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.209339+0000", 2021-03-15T22:21:17.130 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5449999999999997e-06 2021-03-15T22:21:17.131 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.131 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.209343+0000", 2021-03-15T22:21:17.131 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2719999999999999e-05 2021-03-15T22:21:17.132 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.132 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.132 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.132 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.209366+0000", 2021-03-15T22:21:17.133 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2139999999999999e-06 2021-03-15T22:21:17.133 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.133 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.133 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.209373+0000", 2021-03-15T22:21:17.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010061200000000001 2021-03-15T22:21:17.134 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.134 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.134 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.135 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.209474+0000", 2021-03-15T22:21:17.135 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1929e-05 2021-03-15T22:21:17.135 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.135 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.135 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:21:17.136 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.209486+0000", 2021-03-15T22:21:17.136 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013434510000000001 2021-03-15T22:21:17.136 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.136 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.136 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.137 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.210829+0000", 2021-03-15T22:21:17.137 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027986500000000002 2021-03-15T22:21:17.137 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.137 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.138 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.138 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.211109+0000", 2021-03-15T22:21:17.138 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025489499999999999 2021-03-15T22:21:17.138 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.139 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.211364+0000", 2021-03-15T22:21:17.139 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012911890000000001 2021-03-15T22:21:17.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.140 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.140 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.212655+0000", 2021-03-15T22:21:17.140 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011903389999999999 2021-03-15T22:21:17.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.141 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.141 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.213845+0000", 2021-03-15T22:21:17.142 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00071967799999999996 2021-03-15T22:21:17.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.142 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.142 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.142 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.214565+0000", 2021-03-15T22:21:17.143 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3714999999999997e-05 2021-03-15T22:21:17.143 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.143 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.143 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.144 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.214609+0000", 2021-03-15T22:21:17.144 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8114000000000005e-05 2021-03-15T22:21:17.144 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.144 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.144 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.145 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.214677+0000", 2021-03-15T22:21:17.145 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0058167749999999997 2021-03-15T22:21:17.145 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.145 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.146 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.146 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.146 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.146 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:277 7.0 7:08414d91:::smithi07694878-42:head [copy-from ver 6] snapc 8=[8] ondisk+write+known_if_redirected e104)", 2021-03-15T22:21:17.146 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.220857+0000", 2021-03-15T22:21:17.147 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 57.898685616000002, 2021-03-15T22:21:17.147 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01426084, 2021-03-15T22:21:17.147 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.147 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.147 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.148 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.148 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.148 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 277 2021-03-15T22:21:17.148 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.149 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.149 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.149 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.149 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220857+0000", 2021-03-15T22:21:17.151 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.151 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.151 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.152 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220857+0000", 2021-03-15T22:21:17.152 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.73e-06 2021-03-15T22:21:17.152 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.152 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.153 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.153 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220858+0000", 2021-03-15T22:21:17.153 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.277e-06 2021-03-15T22:21:17.153 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.153 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220860+0000", 2021-03-15T22:21:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8489999999999995e-06 2021-03-15T22:21:17.154 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.155 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.155 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.155 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220868+0000", 2021-03-15T22:21:17.155 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5480000000000003e-06 2021-03-15T22:21:17.155 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.156 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.156 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.156 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220873+0000", 2021-03-15T22:21:17.156 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001357249 2021-03-15T22:21:17.156 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.157 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.157 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.157 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222230+0000", 2021-03-15T22:21:17.157 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4117999999999998e-05 2021-03-15T22:21:17.158 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.158 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.158 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.158 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222294+0000", 2021-03-15T22:21:17.158 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035570179999999999 2021-03-15T22:21:17.159 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.159 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.225851+0000", 2021-03-15T22:21:17.159 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038919620000000001 2021-03-15T22:21:17.160 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.160 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.160 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.160 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.229743+0000", 2021-03-15T22:21:17.161 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024645940000000001 2021-03-15T22:21:17.161 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.161 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.161 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.161 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.232208+0000", 2021-03-15T22:21:17.162 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002799171 2021-03-15T22:21:17.162 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.162 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.162 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.162 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235007+0000", 2021-03-15T22:21:17.163 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3729e-05 2021-03-15T22:21:17.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.163 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.164 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235061+0000", 2021-03-15T22:21:17.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6595000000000003e-05 2021-03-15T22:21:17.164 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.164 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.164 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.165 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235117+0000", 2021-03-15T22:21:17.165 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01426084 2021-03-15T22:21:17.165 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.165 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.165 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.166 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.166 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:278 7.0 7:08414d91:::smithi07694878-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e104)", 2021-03-15T22:21:17.166 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.220878+0000", 2021-03-15T22:21:17.167 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 57.898664183000001, 2021-03-15T22:21:17.167 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014453041999999999, 2021-03-15T22:21:17.167 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.167 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:21:17.167 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.168 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.168 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.168 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 278 2021-03-15T22:21:17.168 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.169 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.169 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.169 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.169 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220878+0000", 2021-03-15T22:21:17.169 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.170 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.170 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.170 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220878+0000", 2021-03-15T22:21:17.170 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.037e-06 2021-03-15T22:21:17.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.171 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.171 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.171 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220879+0000", 2021-03-15T22:21:17.172 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.187e-06 2021-03-15T22:21:17.172 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.172 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.172 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.172 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220880+0000", 2021-03-15T22:21:17.173 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.879e-06 2021-03-15T22:21:17.173 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.173 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.173 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.174 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220886+0000", 2021-03-15T22:21:17.174 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0269999999999997e-06 2021-03-15T22:21:17.174 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.174 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.174 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.175 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220890+0000", 2021-03-15T22:21:17.175 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001548542 2021-03-15T22:21:17.175 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.175 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.175 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.176 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222439+0000", 2021-03-15T22:21:17.176 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1516000000000001e-05 2021-03-15T22:21:17.176 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.176 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.177 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:17.177 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222470+0000", 2021-03-15T22:21:17.177 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066482809999999998 2021-03-15T22:21:17.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.177 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.178 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.178 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.229118+0000", 2021-03-15T22:21:17.178 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6447999999999997e-05 2021-03-15T22:21:17.178 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.179 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:17.179 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.229195+0000", 2021-03-15T22:21:17.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0059693100000000002 2021-03-15T22:21:17.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.180 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.180 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235164+0000", 2021-03-15T22:21:17.180 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4153999999999994e-05 2021-03-15T22:21:17.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.181 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.181 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.181 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235248+0000", 2021-03-15T22:21:17.181 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2661e-05 2021-03-15T22:21:17.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.182 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.182 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235331+0000", 2021-03-15T22:21:17.182 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014453041999999999 2021-03-15T22:21:17.183 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.183 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.183 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.183 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.184 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:294 7.0 7:0cc37e3d:::smithi07694878-21:head [copy-from ver 3] snapc 8=[8] ondisk+write+known_if_redirected e104)", 2021-03-15T22:21:17.184 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.394807+0000", 2021-03-15T22:21:17.184 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 57.724735072000001, 2021-03-15T22:21:17.184 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018445559, 2021-03-15T22:21:17.185 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.185 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.185 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.185 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.185 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.186 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 294 2021-03-15T22:21:17.186 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.186 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.186 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.186 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.187 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394807+0000", 2021-03-15T22:21:17.187 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.187 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.187 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.188 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.188 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394807+0000", 2021-03-15T22:21:17.188 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9659999999999999e-06 2021-03-15T22:21:17.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.188 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.189 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.189 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394809+0000", 2021-03-15T22:21:17.189 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2610000000000001e-06 2021-03-15T22:21:17.189 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.189 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.190 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.190 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394810+0000", 2021-03-15T22:21:17.190 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0881e-05 2021-03-15T22:21:17.190 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.191 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.191 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.191 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394821+0000", 2021-03-15T22:21:17.191 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8509999999999994e-06 2021-03-15T22:21:17.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.192 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.192 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.192 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394831+0000", 2021-03-15T22:21:17.192 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035723980000000001 2021-03-15T22:21:17.192 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.193 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.193 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.193 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398403+0000", 2021-03-15T22:21:17.193 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5474000000000004e-05 2021-03-15T22:21:17.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.194 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.194 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.194 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398469+0000", 2021-03-15T22:21:17.194 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0069378080000000002 2021-03-15T22:21:17.195 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.195 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.195 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.195 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.405407+0000", 2021-03-15T22:21:17.195 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033175689999999998 2021-03-15T22:21:17.196 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.196 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.196 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.196 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.408724+0000", 2021-03-15T22:21:17.197 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042654010000000003 2021-03-15T22:21:17.197 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.197 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.197 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.412990+0000", 2021-03-15T22:21:17.198 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016210500000000001 2021-03-15T22:21:17.198 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.198 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.198 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.198 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413152+0000", 2021-03-15T22:21:17.199 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0185999999999997e-05 2021-03-15T22:21:17.199 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.199 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.199 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.200 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413202+0000", 2021-03-15T22:21:17.200 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0658999999999997e-05 2021-03-15T22:21:17.200 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.200 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.200 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.201 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413253+0000", 2021-03-15T22:21:17.201 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018445559 2021-03-15T22:21:17.201 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.201 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.201 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.202 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.202 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.202 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:295 7.0 7:0cc37e3d:::smithi07694878-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e104)", 2021-03-15T22:21:17.202 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.394836+0000", 2021-03-15T22:21:17.203 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 57.724706216999998, 2021-03-15T22:21:17.203 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018595192999999999, 2021-03-15T22:21:17.203 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.203 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:21:17.203 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.204 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.204 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.204 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 295 2021-03-15T22:21:17.204 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.204 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.205 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.205 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.205 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394836+0000", 2021-03-15T22:21:17.205 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.206 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.206 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.206 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.206 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394836+0000", 2021-03-15T22:21:17.206 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4300000000000002e-07 2021-03-15T22:21:17.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.207 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.207 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394837+0000", 2021-03-15T22:21:17.207 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5799999999999998e-07 2021-03-15T22:21:17.208 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.208 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.208 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394838+0000", 2021-03-15T22:21:17.209 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9940000000000002e-06 2021-03-15T22:21:17.209 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.209 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.209 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394842+0000", 2021-03-15T22:21:17.210 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.339e-06 2021-03-15T22:21:17.210 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.210 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.210 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.210 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394844+0000", 2021-03-15T22:21:17.211 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003783809 2021-03-15T22:21:17.211 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.211 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.211 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.212 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398628+0000", 2021-03-15T22:21:17.212 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0221e-05 2021-03-15T22:21:17.212 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.212 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.212 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:17.213 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398668+0000", 2021-03-15T22:21:17.213 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0094639070000000006 2021-03-15T22:21:17.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.213 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.214 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.408132+0000", 2021-03-15T22:21:17.214 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0552999999999999e-05 2021-03-15T22:21:17.214 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.214 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:17.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.408212+0000", 2021-03-15T22:21:17.215 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050811019999999997 2021-03-15T22:21:17.215 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.215 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.216 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.216 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413294+0000", 2021-03-15T22:21:17.216 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9265999999999997e-05 2021-03-15T22:21:17.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.217 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.217 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413363+0000", 2021-03-15T22:21:17.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8300999999999995e-05 2021-03-15T22:21:17.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.217 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.218 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.218 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413431+0000", 2021-03-15T22:21:17.218 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018595192999999999 2021-03-15T22:21:17.218 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.219 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.219 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.219 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.219 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.219 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:300 7.0 7:071f7a04:::smithi07694878-37:head [read 0~2605052,checksum] snapc 0=[] ondisk+read+known_if_redirected e104)", 2021-03-15T22:21:17.220 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.394856+0000", 2021-03-15T22:21:17.220 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 57.724685827999998, 2021-03-15T22:21:17.220 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0062890660000000003, 2021-03-15T22:21:17.220 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.220 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:21:17.221 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.221 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.221 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.221 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 300 2021-03-15T22:21:17.222 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.222 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.222 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.222 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.222 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394856+0000", 2021-03-15T22:21:17.223 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.223 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.223 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.223 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.223 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394856+0000", 2021-03-15T22:21:17.224 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.19e-06 2021-03-15T22:21:17.224 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.224 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.224 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.224 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394858+0000", 2021-03-15T22:21:17.225 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6700000000000002e-07 2021-03-15T22:21:17.225 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.225 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.225 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.226 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394858+0000", 2021-03-15T22:21:17.226 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7980000000000001e-06 2021-03-15T22:21:17.226 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.226 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.226 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.227 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394861+0000", 2021-03-15T22:21:17.227 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1019999999999999e-06 2021-03-15T22:21:17.227 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.227 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.227 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.228 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394863+0000", 2021-03-15T22:21:17.228 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050499840000000004 2021-03-15T22:21:17.228 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.228 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.229 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.229 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.399913+0000", 2021-03-15T22:21:17.229 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0797000000000001e-05 2021-03-15T22:21:17.229 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.229 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.230 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.230 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.399974+0000", 2021-03-15T22:21:17.230 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011713279999999999 2021-03-15T22:21:17.230 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.230 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.231 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.231 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.401145+0000", 2021-03-15T22:21:17.231 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0062890660000000003 2021-03-15T22:21:17.231 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.232 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.232 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.232 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.232 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.232 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:301 7.0 7:071f7a04:::smithi07694878-37:head [read 0~2605052,checksum] snapc 0=[] ondisk+read+known_if_redirected e104)", 2021-03-15T22:21:17.233 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.394865+0000", 2021-03-15T22:21:17.233 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 57.724676676999998, 2021-03-15T22:21:17.233 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0072225989999999997, 2021-03-15T22:21:17.233 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.233 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:21:17.234 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.234 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.234 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.234 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 301 2021-03-15T22:21:17.235 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.235 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.235 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.235 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.235 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394865+0000", 2021-03-15T22:21:17.236 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.236 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.236 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.236 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.236 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394865+0000", 2021-03-15T22:21:17.237 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3540000000000001e-06 2021-03-15T22:21:17.237 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.237 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.237 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.237 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394867+0000", 2021-03-15T22:21:17.238 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8100000000000002e-07 2021-03-15T22:21:17.238 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.238 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.238 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.239 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394868+0000", 2021-03-15T22:21:17.239 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9000000000000002e-06 2021-03-15T22:21:17.239 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.239 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.239 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.240 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394871+0000", 2021-03-15T22:21:17.240 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0889999999999998e-06 2021-03-15T22:21:17.240 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.240 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394873+0000", 2021-03-15T22:21:17.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.006295864 2021-03-15T22:21:17.241 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.241 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.242 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.242 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.401168+0000", 2021-03-15T22:21:17.242 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9645000000000002e-05 2021-03-15T22:21:17.242 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.242 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.243 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.243 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.401228+0000", 2021-03-15T22:21:17.243 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00085996599999999999 2021-03-15T22:21:17.243 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.243 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.244 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.244 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.402088+0000", 2021-03-15T22:21:17.244 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0072225989999999997 2021-03-15T22:21:17.244 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.244 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.245 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.245 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.245 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.245 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:366 7.0 7:0732c451:::smithi07694878-32:head [copy-from ver 9] snapc a=[a] ondisk+write+known_if_redirected e108)", 2021-03-15T22:21:17.246 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.105457+0000", 2021-03-15T22:21:17.246 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 54.014085541, 2021-03-15T22:21:17.246 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018098547, 2021-03-15T22:21:17.246 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.246 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.247 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.247 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.247 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.247 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 366 2021-03-15T22:21:17.247 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.248 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.248 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.248 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105457+0000", 2021-03-15T22:21:17.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.249 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.249 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105457+0000", 2021-03-15T22:21:17.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9000000000000003e-06 2021-03-15T22:21:17.250 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.250 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.250 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.250 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105463+0000", 2021-03-15T22:21:17.251 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1069999999999998e-06 2021-03-15T22:21:17.251 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.251 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.251 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.252 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105467+0000", 2021-03-15T22:21:17.252 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0707999999999999e-05 2021-03-15T22:21:17.252 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.252 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.253 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105487+0000", 2021-03-15T22:21:17.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4830000000000005e-06 2021-03-15T22:21:17.253 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.253 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.254 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105497+0000", 2021-03-15T22:21:17.254 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9896999999999998e-05 2021-03-15T22:21:17.254 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.254 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105577+0000", 2021-03-15T22:21:17.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000138195 2021-03-15T22:21:17.255 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.256 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105715+0000", 2021-03-15T22:21:17.256 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0074524730000000003 2021-03-15T22:21:17.256 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.257 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.257 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.113167+0000", 2021-03-15T22:21:17.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053675119999999996 2021-03-15T22:21:17.257 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.258 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.258 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.258 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.118535+0000", 2021-03-15T22:21:17.258 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002009102 2021-03-15T22:21:17.258 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.259 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.259 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.120544+0000", 2021-03-15T22:21:17.259 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028954580000000001 2021-03-15T22:21:17.259 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123439+0000", 2021-03-15T22:21:17.260 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6292999999999999e-05 2021-03-15T22:21:17.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.261 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.261 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123486+0000", 2021-03-15T22:21:17.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9418999999999994e-05 2021-03-15T22:21:17.262 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.262 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.262 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.262 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123555+0000", 2021-03-15T22:21:17.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018098547 2021-03-15T22:21:17.263 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.263 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.263 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.264 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.264 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:367 7.0 7:0732c451:::smithi07694878-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e108)", 2021-03-15T22:21:17.264 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.105520+0000", 2021-03-15T22:21:17.264 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 54.014021683000003, 2021-03-15T22:21:17.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018220706, 2021-03-15T22:21:17.265 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.265 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:21:17.265 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.265 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.265 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.266 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 367 2021-03-15T22:21:17.266 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.266 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.266 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.267 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.267 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105520+0000", 2021-03-15T22:21:17.267 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.267 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.267 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.268 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.268 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105520+0000", 2021-03-15T22:21:17.268 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.68e-06 2021-03-15T22:21:17.268 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.268 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.269 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.269 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105522+0000", 2021-03-15T22:21:17.269 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0650000000000001e-06 2021-03-15T22:21:17.269 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.269 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.270 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.270 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105524+0000", 2021-03-15T22:21:17.270 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9020000000000005e-06 2021-03-15T22:21:17.270 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.271 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.271 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105533+0000", 2021-03-15T22:21:17.271 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1390000000000003e-06 2021-03-15T22:21:17.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.272 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.272 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.272 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105538+0000", 2021-03-15T22:21:17.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034772699999999999 2021-03-15T22:21:17.272 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105886+0000", 2021-03-15T22:21:17.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1764000000000001e-05 2021-03-15T22:21:17.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:17.274 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105938+0000", 2021-03-15T22:21:17.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011904672999999999 2021-03-15T22:21:17.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.117842+0000", 2021-03-15T22:21:17.276 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4134999999999994e-05 2021-03-15T22:21:17.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.276 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:17.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.117917+0000", 2021-03-15T22:21:17.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056846789999999998 2021-03-15T22:21:17.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.277 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123601+0000", 2021-03-15T22:21:17.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1218999999999999e-05 2021-03-15T22:21:17.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.278 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.278 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.279 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123652+0000", 2021-03-15T22:21:17.279 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8722999999999997e-05 2021-03-15T22:21:17.279 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.279 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.279 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.280 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123741+0000", 2021-03-15T22:21:17.280 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018220706 2021-03-15T22:21:17.280 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.280 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.280 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.281 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.281 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:372 7.0 7:054fa7db:::smithi07694878-29:head [copy-from ver 3] snapc a=[a] ondisk+write+known_if_redirected e108)", 2021-03-15T22:21:17.281 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.121689+0000", 2021-03-15T22:21:17.281 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 53.997852858000002, 2021-03-15T22:21:17.282 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018844218999999999, 2021-03-15T22:21:17.282 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.282 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.282 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.283 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.283 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.283 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 372 2021-03-15T22:21:17.283 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.283 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.284 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.284 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.284 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121689+0000", 2021-03-15T22:21:17.284 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.284 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.285 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.285 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.285 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121689+0000", 2021-03-15T22:21:17.285 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5499999999999996e-06 2021-03-15T22:21:17.285 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.286 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.286 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.286 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121694+0000", 2021-03-15T22:21:17.286 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0380000000000001e-06 2021-03-15T22:21:17.287 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.287 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.287 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.287 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121698+0000", 2021-03-15T22:21:17.287 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9465000000000001e-05 2021-03-15T22:21:17.288 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.288 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.288 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.288 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121717+0000", 2021-03-15T22:21:17.288 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7600000000000001e-05 2021-03-15T22:21:17.289 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.289 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.289 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.289 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121735+0000", 2021-03-15T22:21:17.290 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1116000000000003e-05 2021-03-15T22:21:17.290 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.290 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.290 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.290 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121786+0000", 2021-03-15T22:21:17.291 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9568999999999996e-05 2021-03-15T22:21:17.291 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.291 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.291 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.292 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121876+0000", 2021-03-15T22:21:17.292 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085690019999999992 2021-03-15T22:21:17.292 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.292 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.292 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.293 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.130445+0000", 2021-03-15T22:21:17.293 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0025600380000000002 2021-03-15T22:21:17.293 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.293 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.293 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.294 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.133005+0000", 2021-03-15T22:21:17.294 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041963060000000003 2021-03-15T22:21:17.294 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.294 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.295 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.295 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.137201+0000", 2021-03-15T22:21:17.295 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003222258 2021-03-15T22:21:17.295 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.295 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.296 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.296 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140423+0000", 2021-03-15T22:21:17.296 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6202999999999997e-05 2021-03-15T22:21:17.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.297 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.297 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.297 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140479+0000", 2021-03-15T22:21:17.297 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4073999999999998e-05 2021-03-15T22:21:17.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.298 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.298 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.298 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140534+0000", 2021-03-15T22:21:17.298 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018844218999999999 2021-03-15T22:21:17.298 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.299 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.299 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.299 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.299 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.300 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:373 7.0 7:054fa7db:::smithi07694878-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e108)", 2021-03-15T22:21:17.300 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.121751+0000", 2021-03-15T22:21:17.300 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 53.997791249999999, 2021-03-15T22:21:17.300 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018944425000000001, 2021-03-15T22:21:17.300 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.301 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:21:17.301 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.301 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.301 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.301 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 373 2021-03-15T22:21:17.302 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.302 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.302 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.302 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.302 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121751+0000", 2021-03-15T22:21:17.303 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.303 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.303 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.303 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.304 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121751+0000", 2021-03-15T22:21:17.304 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.443e-06 2021-03-15T22:21:17.304 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.304 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.304 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.305 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121752+0000", 2021-03-15T22:21:17.305 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.057e-06 2021-03-15T22:21:17.305 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.305 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121754+0000", 2021-03-15T22:21:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2910000000000007e-06 2021-03-15T22:21:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.306 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.307 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.307 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121763+0000", 2021-03-15T22:21:17.307 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9339999999999997e-06 2021-03-15T22:21:17.307 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.308 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.308 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121768+0000", 2021-03-15T22:21:17.308 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024470200000000002 2021-03-15T22:21:17.308 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.308 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.309 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.309 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122012+0000", 2021-03-15T22:21:17.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3108000000000001e-05 2021-03-15T22:21:17.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.309 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:17.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122055+0000", 2021-03-15T22:21:17.310 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010285216 2021-03-15T22:21:17.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.311 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.311 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.311 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.132341+0000", 2021-03-15T22:21:17.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4697999999999994e-05 2021-03-15T22:21:17.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.312 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.312 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:17.312 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.132425+0000", 2021-03-15T22:21:17.312 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081476789999999997 2021-03-15T22:21:17.312 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.313 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.313 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.313 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140573+0000", 2021-03-15T22:21:17.313 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1277999999999999e-05 2021-03-15T22:21:17.313 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.314 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.314 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140624+0000", 2021-03-15T22:21:17.314 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1019000000000006e-05 2021-03-15T22:21:17.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.315 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.315 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.315 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140695+0000", 2021-03-15T22:21:17.315 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018944425000000001 2021-03-15T22:21:17.316 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.316 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.316 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.316 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.317 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:375 7.0 7:00c03349:::smithi07694878-40:head [write 1729599~575687] snapc a=[a] ondisk+write+known_if_redirected e108)", 2021-03-15T22:21:17.317 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.134705+0000", 2021-03-15T22:21:17.317 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 53.984837382999999, 2021-03-15T22:21:17.317 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061277609999999998, 2021-03-15T22:21:17.318 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.318 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.318 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.318 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.318 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.319 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 375 2021-03-15T22:21:17.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.319 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.319 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.319 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.320 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.134705+0000", 2021-03-15T22:21:17.320 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.134705+0000", 2021-03-15T22:21:17.321 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00063177600000000002 2021-03-15T22:21:17.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.322 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.322 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.135337+0000", 2021-03-15T22:21:17.322 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.134e-06 2021-03-15T22:21:17.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.323 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.135340+0000", 2021-03-15T22:21:17.323 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6033e-05 2021-03-15T22:21:17.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.324 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.324 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.324 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.135356+0000", 2021-03-15T22:21:17.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0569999999999998e-06 2021-03-15T22:21:17.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.325 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.325 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.135362+0000", 2021-03-15T22:21:17.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1449000000000002e-05 2021-03-15T22:21:17.325 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.326 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.326 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.326 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.135403+0000", 2021-03-15T22:21:17.326 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7115999999999997e-05 2021-03-15T22:21:17.327 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.327 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.327 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.135490+0000", 2021-03-15T22:21:17.327 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013703899999999999 2021-03-15T22:21:17.328 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.328 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.328 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.328 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.135627+0000", 2021-03-15T22:21:17.328 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00083323100000000003 2021-03-15T22:21:17.329 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.329 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.329 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.329 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.136461+0000", 2021-03-15T22:21:17.329 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00099209800000000007 2021-03-15T22:21:17.330 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.330 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.330 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.330 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.137453+0000", 2021-03-15T22:21:17.331 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0032921500000000002 2021-03-15T22:21:17.331 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.331 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.331 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140745+0000", 2021-03-15T22:21:17.332 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6645999999999999e-05 2021-03-15T22:21:17.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.332 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.333 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140802+0000", 2021-03-15T22:21:17.333 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1031999999999997e-05 2021-03-15T22:21:17.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.333 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.334 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140833+0000", 2021-03-15T22:21:17.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061277609999999998 2021-03-15T22:21:17.334 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.334 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.334 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.335 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.335 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.335 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:476 7.0 7:0cc37e3d:::smithi07694878-21:head [write 594981~609197,stat] snapc 15=[15,11] ondisk+write+known_if_redirected e117)", 2021-03-15T22:21:17.335 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:32.066860+0000", 2021-03-15T22:21:17.335 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 45.052682466999997, 2021-03-15T22:21:17.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0059583960000000004, 2021-03-15T22:21:17.336 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.336 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.336 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.337 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.337 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.337 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 476 2021-03-15T22:21:17.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.337 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.338 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.338 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.066860+0000", 2021-03-15T22:21:17.338 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.339 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.339 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.066860+0000", 2021-03-15T22:21:17.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014187480000000001 2021-03-15T22:21:17.339 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.340 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.340 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.068278+0000", 2021-03-15T22:21:17.340 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9620000000000001e-06 2021-03-15T22:21:17.341 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.341 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.341 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.341 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.068283+0000", 2021-03-15T22:21:17.341 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.234e-05 2021-03-15T22:21:17.342 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.342 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.342 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.342 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.068306+0000", 2021-03-15T22:21:17.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6070000000000002e-06 2021-03-15T22:21:17.343 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.343 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.343 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.068313+0000", 2021-03-15T22:21:17.344 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2556000000000002e-05 2021-03-15T22:21:17.344 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.344 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.344 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.344 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.068376+0000", 2021-03-15T22:21:17.345 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2849000000000007e-05 2021-03-15T22:21:17.345 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.345 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.345 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.345 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.068449+0000", 2021-03-15T22:21:17.346 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017991799999999999 2021-03-15T22:21:17.346 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.346 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.346 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.068629+0000", 2021-03-15T22:21:17.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022308319999999999 2021-03-15T22:21:17.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.347 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.070860+0000", 2021-03-15T22:21:17.348 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001647917 2021-03-15T22:21:17.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.348 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.349 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.072507+0000", 2021-03-15T22:21:17.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000228259 2021-03-15T22:21:17.349 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.350 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.072736+0000", 2021-03-15T22:21:17.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6267999999999998e-05 2021-03-15T22:21:17.350 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.072782+0000", 2021-03-15T22:21:17.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6140000000000003e-05 2021-03-15T22:21:17.351 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.351 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.352 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.352 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.072818+0000", 2021-03-15T22:21:17.352 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0059583960000000004 2021-03-15T22:21:17.352 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.352 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.353 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.353 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.353 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.353 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:477 7.0 7:0cc37e3d:::smithi07694878-21:head [write 1974123~745028] snapc 15=[15,11] ondisk+write+known_if_redirected e117)", 2021-03-15T22:21:17.353 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:32.075392+0000", 2021-03-15T22:21:17.354 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 45.044150469999998, 2021-03-15T22:21:17.354 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066045670000000004, 2021-03-15T22:21:17.354 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.354 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.355 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.355 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.355 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.355 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 477 2021-03-15T22:21:17.355 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.356 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.356 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.075392+0000", 2021-03-15T22:21:17.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.357 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.357 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.357 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.357 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.075392+0000", 2021-03-15T22:21:17.358 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00066662600000000002 2021-03-15T22:21:17.358 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.358 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.358 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.076058+0000", 2021-03-15T22:21:17.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.287e-06 2021-03-15T22:21:17.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.359 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.359 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.076062+0000", 2021-03-15T22:21:17.360 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6682000000000001e-05 2021-03-15T22:21:17.360 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.360 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.360 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.361 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.076078+0000", 2021-03-15T22:21:17.361 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9820000000000002e-06 2021-03-15T22:21:17.361 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.361 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.361 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.362 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.076085+0000", 2021-03-15T22:21:17.362 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7732999999999997e-05 2021-03-15T22:21:17.362 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.362 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.363 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.076153+0000", 2021-03-15T22:21:17.363 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011012 2021-03-15T22:21:17.363 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.363 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.364 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.364 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.076263+0000", 2021-03-15T22:21:17.364 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021697900000000001 2021-03-15T22:21:17.364 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.364 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.365 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.365 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.076480+0000", 2021-03-15T22:21:17.365 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001451908 2021-03-15T22:21:17.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.366 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.366 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.077932+0000", 2021-03-15T22:21:17.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013484289999999999 2021-03-15T22:21:17.366 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.367 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.367 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.367 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.079280+0000", 2021-03-15T22:21:17.367 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002611014 2021-03-15T22:21:17.367 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.368 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.368 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.368 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.081891+0000", 2021-03-15T22:21:17.368 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2488999999999998e-05 2021-03-15T22:21:17.368 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.369 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.369 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.369 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.081944+0000", 2021-03-15T22:21:17.369 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2318000000000003e-05 2021-03-15T22:21:17.370 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.370 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.370 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.370 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:32.081996+0000", 2021-03-15T22:21:17.370 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066045670000000004 2021-03-15T22:21:17.371 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.371 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.371 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.371 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.371 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.372 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:534 7.0 7:08414d91:::smithi07694878-42:head [write 729437~527542,stat] snapc 1b=[1b,19,18,17] ondisk+write+known_if_redirected e123)", 2021-03-15T22:21:17.372 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:38.095834+0000", 2021-03-15T22:21:17.372 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 39.023707788999999, 2021-03-15T22:21:17.372 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0057273699999999999, 2021-03-15T22:21:17.373 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.373 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.373 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.373 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.373 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.374 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 534 2021-03-15T22:21:17.374 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.374 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.374 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.375 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.375 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.095834+0000", 2021-03-15T22:21:17.375 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.375 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.376 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.376 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.095834+0000", 2021-03-15T22:21:17.376 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00071756999999999995 2021-03-15T22:21:17.376 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.376 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.377 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.377 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.096552+0000", 2021-03-15T22:21:17.377 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5770000000000004e-06 2021-03-15T22:21:17.377 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.377 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.378 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.096557+0000", 2021-03-15T22:21:17.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0613999999999999e-05 2021-03-15T22:21:17.378 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.379 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.096577+0000", 2021-03-15T22:21:17.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6629999999999993e-06 2021-03-15T22:21:17.379 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.380 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.380 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.096585+0000", 2021-03-15T22:21:17.380 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1542e-05 2021-03-15T22:21:17.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.381 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.381 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.381 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.096656+0000", 2021-03-15T22:21:17.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0030999999999999e-05 2021-03-15T22:21:17.382 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.382 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.382 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:21:17.382 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.096666+0000", 2021-03-15T22:21:17.382 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001188342 2021-03-15T22:21:17.383 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.383 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.383 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.097855+0000", 2021-03-15T22:21:17.383 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000107487 2021-03-15T22:21:17.384 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.384 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.097962+0000", 2021-03-15T22:21:17.385 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00037487300000000001 2021-03-15T22:21:17.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.385 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.385 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.098337+0000", 2021-03-15T22:21:17.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001449669 2021-03-15T22:21:17.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.386 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.386 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.099787+0000", 2021-03-15T22:21:17.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001621518 2021-03-15T22:21:17.387 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.388 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.101408+0000", 2021-03-15T22:21:17.388 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8832999999999996e-05 2021-03-15T22:21:17.388 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.388 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.388 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.389 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.101487+0000", 2021-03-15T22:21:17.389 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2710999999999998e-05 2021-03-15T22:21:17.389 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.389 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.389 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.390 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.101530+0000", 2021-03-15T22:21:17.390 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1940000000000003e-05 2021-03-15T22:21:17.390 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.390 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.391 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:38.101562+0000", 2021-03-15T22:21:17.391 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0057273699999999999 2021-03-15T22:21:17.391 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.391 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.392 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.392 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.392 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.392 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:692 7.0 7:054fa7db:::smithi07694878-29:head [write 699191~532079,stat] snapc 2a=[2a,20,1f,1e,1b,17] ondisk+write+known_if_redirected e139)", 2021-03-15T22:21:17.392 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:54.096860+0000", 2021-03-15T22:21:17.393 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 23.022682295999999, 2021-03-15T22:21:17.393 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0077367729999999997, 2021-03-15T22:21:17.393 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.393 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.394 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.394 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.394 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.394 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 692 2021-03-15T22:21:17.394 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.395 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.395 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.395 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.395 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.096860+0000", 2021-03-15T22:21:17.395 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.396 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.396 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.396 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.396 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.096860+0000", 2021-03-15T22:21:17.396 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00059679000000000004 2021-03-15T22:21:17.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.397 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.397 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.397 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097457+0000", 2021-03-15T22:21:17.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3510000000000002e-06 2021-03-15T22:21:17.398 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.398 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.398 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.398 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097460+0000", 2021-03-15T22:21:17.399 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8439000000000001e-05 2021-03-15T22:21:17.399 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097478+0000", 2021-03-15T22:21:17.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6081e-05 2021-03-15T22:21:17.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.401 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097505+0000", 2021-03-15T22:21:17.401 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8774000000000003e-05 2021-03-15T22:21:17.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.402 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097573+0000", 2021-03-15T22:21:17.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3470000000000001e-06 2021-03-15T22:21:17.402 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.402 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:21:17.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097581+0000", 2021-03-15T22:21:17.403 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001269776 2021-03-15T22:21:17.403 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.403 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.403 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.404 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.098850+0000", 2021-03-15T22:21:17.404 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000112728 2021-03-15T22:21:17.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.405 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.405 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.098963+0000", 2021-03-15T22:21:17.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00042357400000000002 2021-03-15T22:21:17.405 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.405 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.406 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.406 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.099387+0000", 2021-03-15T22:21:17.406 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0016555210000000001 2021-03-15T22:21:17.406 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.407 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.101042+0000", 2021-03-15T22:21:17.407 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013926629999999999 2021-03-15T22:21:17.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.408 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.408 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.408 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.102435+0000", 2021-03-15T22:21:17.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002038644 2021-03-15T22:21:17.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.409 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.104474+0000", 2021-03-15T22:21:17.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6980000000000001e-05 2021-03-15T22:21:17.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.410 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.104521+0000", 2021-03-15T22:21:17.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6105000000000004e-05 2021-03-15T22:21:17.411 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.104597+0000", 2021-03-15T22:21:17.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0077367729999999997 2021-03-15T22:21:17.412 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.412 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.412 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.413 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.413 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:847 7.0 7:08414d91:::smithi07694878-42:head [append 0~0] snapc 34=[34,32,30] ondisk+write+known_if_redirected e149)", 2021-03-15T22:21:17.413 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:04.032816+0000", 2021-03-15T22:21:17.413 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 13.086725752, 2021-03-15T22:21:17.414 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053991739999999996, 2021-03-15T22:21:17.414 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.414 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.414 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.414 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.415 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.415 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 847 2021-03-15T22:21:17.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.415 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.416 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.416 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.032816+0000", 2021-03-15T22:21:17.416 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.416 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.417 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.417 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.032816+0000", 2021-03-15T22:21:17.417 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9100000000000002e-07 2021-03-15T22:21:17.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.418 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.418 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.418 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.032817+0000", 2021-03-15T22:21:17.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9740000000000004e-06 2021-03-15T22:21:17.419 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.419 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.419 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.419 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.032821+0000", 2021-03-15T22:21:17.419 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2740000000000001e-06 2021-03-15T22:21:17.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.420 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.420 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.032826+0000", 2021-03-15T22:21:17.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6960000000000001e-06 2021-03-15T22:21:17.421 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.421 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.421 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.032828+0000", 2021-03-15T22:21:17.422 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014207410000000001 2021-03-15T22:21:17.422 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.422 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.422 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.422 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.034249+0000", 2021-03-15T22:21:17.423 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9246999999999995e-05 2021-03-15T22:21:17.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.423 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.423 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.034328+0000", 2021-03-15T22:21:17.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000148298 2021-03-15T22:21:17.424 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.425 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.034477+0000", 2021-03-15T22:21:17.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036401800000000002 2021-03-15T22:21:17.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.425 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.425 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.426 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.034841+0000", 2021-03-15T22:21:17.426 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001103702 2021-03-15T22:21:17.426 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.426 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.427 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.035944+0000", 2021-03-15T22:21:17.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021560619999999998 2021-03-15T22:21:17.427 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.427 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.428 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.428 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.038100+0000", 2021-03-15T22:21:17.428 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9609000000000004e-05 2021-03-15T22:21:17.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.428 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.429 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.429 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.038170+0000", 2021-03-15T22:21:17.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5661999999999998e-05 2021-03-15T22:21:17.429 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.429 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.430 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:04.038216+0000", 2021-03-15T22:21:17.430 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053991739999999996 2021-03-15T22:21:17.430 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.431 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.431 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.431 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.431 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.431 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:873 7.0 7:0cc37e3d:::smithi07694878-21:head [write 1584925~506039] snapc 35=[34,30] ondisk+write+known_if_redirected e152)", 2021-03-15T22:21:17.432 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:07.140779+0000", 2021-03-15T22:21:17.432 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 9.9787633299999996, 2021-03-15T22:21:17.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0080121819999999996, 2021-03-15T22:21:17.432 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.432 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.433 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.433 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.433 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.433 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 873 2021-03-15T22:21:17.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.434 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.434 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.434 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.434 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.140779+0000", 2021-03-15T22:21:17.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.435 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.435 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.140779+0000", 2021-03-15T22:21:17.436 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00045175199999999998 2021-03-15T22:21:17.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.436 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.436 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141231+0000", 2021-03-15T22:21:17.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5019999999999999e-06 2021-03-15T22:21:17.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.437 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.437 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.438 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141233+0000", 2021-03-15T22:21:17.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1036000000000001e-05 2021-03-15T22:21:17.438 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.438 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.439 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141244+0000", 2021-03-15T22:21:17.439 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7910000000000004e-06 2021-03-15T22:21:17.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141249+0000", 2021-03-15T22:21:17.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9688000000000003e-05 2021-03-15T22:21:17.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.440 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.440 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.441 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141299+0000", 2021-03-15T22:21:17.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7974999999999999e-05 2021-03-15T22:21:17.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.441 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.442 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141367+0000", 2021-03-15T22:21:17.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106593 2021-03-15T22:21:17.442 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.443 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141473+0000", 2021-03-15T22:21:17.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013605780000000001 2021-03-15T22:21:17.443 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142834+0000", 2021-03-15T22:21:17.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028287149999999999 2021-03-15T22:21:17.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.445 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.145662+0000", 2021-03-15T22:21:17.445 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003004652 2021-03-15T22:21:17.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.446 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.446 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.446 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148667+0000", 2021-03-15T22:21:17.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5068999999999997e-05 2021-03-15T22:21:17.447 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.447 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.447 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148732+0000", 2021-03-15T22:21:17.447 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8831e-05 2021-03-15T22:21:17.448 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.448 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.448 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148791+0000", 2021-03-15T22:21:17.448 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0080121819999999996 2021-03-15T22:21:17.449 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.449 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.449 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.449 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.449 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.450 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:874 7.0 7:0cc37e3d:::smithi07694878-21:head [setxattr _header (57) in=7b,truncate 2403401] snapc 35=[34,30] ondisk+write+known_if_redirected e152)", 2021-03-15T22:21:17.450 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:07.141260+0000", 2021-03-15T22:21:17.450 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 9.9782823950000008, 2021-03-15T22:21:17.450 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076387740000000001, 2021-03-15T22:21:17.451 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.451 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:17.451 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.451 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.451 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.452 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 874 2021-03-15T22:21:17.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.452 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.453 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141260+0000", 2021-03-15T22:21:17.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.453 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.454 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.454 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141260+0000", 2021-03-15T22:21:17.454 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.201e-06 2021-03-15T22:21:17.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.455 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.455 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141261+0000", 2021-03-15T22:21:17.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9399999999999993e-07 2021-03-15T22:21:17.455 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.456 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141262+0000", 2021-03-15T22:21:17.456 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9730000000000003e-06 2021-03-15T22:21:17.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.457 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141271+0000", 2021-03-15T22:21:17.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6380000000000002e-06 2021-03-15T22:21:17.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141275+0000", 2021-03-15T22:21:17.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001038693 2021-03-15T22:21:17.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.459 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.459 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142313+0000", 2021-03-15T22:21:17.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6582999999999996e-05 2021-03-15T22:21:17.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.460 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142380+0000", 2021-03-15T22:21:17.460 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000198116 2021-03-15T22:21:17.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.461 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.461 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:21:17.461 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142578+0000", 2021-03-15T22:21:17.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000434752 2021-03-15T22:21:17.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.462 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:17.462 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.143013+0000", 2021-03-15T22:21:17.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027017529999999999 2021-03-15T22:21:17.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.463 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.463 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.463 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.145714+0000", 2021-03-15T22:21:17.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031088420000000001 2021-03-15T22:21:17.464 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.464 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.464 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:17.465 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148823+0000", 2021-03-15T22:21:17.465 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3077999999999997e-05 2021-03-15T22:21:17.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.465 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:17.466 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148856+0000", 2021-03-15T22:21:17.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2150999999999997e-05 2021-03-15T22:21:17.466 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.466 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.466 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.467 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148899+0000", 2021-03-15T22:21:17.467 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076387740000000001 2021-03-15T22:21:17.467 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.467 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.468 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.468 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.468 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.468 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:875 7.0 7:0cc37e3d:::smithi07694878-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e152)", 2021-03-15T22:21:17.468 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:07.141294+0000", 2021-03-15T22:21:17.469 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 9.9782480549999999, 2021-03-15T22:21:17.469 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0079780700000000003, 2021-03-15T22:21:17.469 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:17.469 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:21:17.469 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:17.470 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:17.470 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:17.470 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 875 2021-03-15T22:21:17.470 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.471 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:17.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.471 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:17.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141294+0000", 2021-03-15T22:21:17.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:17.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:17.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141294+0000", 2021-03-15T22:21:17.473 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6599999999999996e-07 2021-03-15T22:21:17.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.473 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:17.473 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141295+0000", 2021-03-15T22:21:17.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1800000000000005e-07 2021-03-15T22:21:17.474 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.474 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.474 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:17.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141295+0000", 2021-03-15T22:21:17.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7010000000000004e-06 2021-03-15T22:21:17.475 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:17.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141300+0000", 2021-03-15T22:21:17.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6419999999999999e-06 2021-03-15T22:21:17.476 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:17.477 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141303+0000", 2021-03-15T22:21:17.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015483140000000001 2021-03-15T22:21:17.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.477 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.477 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.478 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142851+0000", 2021-03-15T22:21:17.478 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8303000000000001e-05 2021-03-15T22:21:17.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:17.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142909+0000", 2021-03-15T22:21:17.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060387089999999997 2021-03-15T22:21:17.479 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:17.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148948+0000", 2021-03-15T22:21:17.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7832999999999997e-05 2021-03-15T22:21:17.480 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:17.481 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.149046+0000", 2021-03-15T22:21:17.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022618399999999999 2021-03-15T22:21:17.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:17.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:17.482 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:17.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.149272+0000", 2021-03-15T22:21:17.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0079780700000000003 2021-03-15T22:21:17.482 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.483 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.483 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.483 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:17.483 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:17.483 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:21:17.484 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:21:17.485 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:17.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:17.487 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:17.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:17.487 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:17 smithi125 conmon[56739]: cluster 2021-03-15T22:21:16.993038+0000 mgr.y (mgr.14144) 416 : cluster [DBG] pgmap v369: 144 pgs: 1 clean+premerge+peered, 143 active+clean; 153 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 20 op/s 2021-03-15T22:21:17.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:17 smithi076 conmon[56514]: cluster 2021-03-15T22:21:16.993038+0000 mgr.y (mgr. 2021-03-15T22:21:17.508 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:17 smithi076 conmon[56514]: 14144) 416 : cluster [DBG] pgmap v369: 144 pgs: 1 clean+premerge+peered, 143 active+clean; 153 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 20 op/s 2021-03-15T22:21:17.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:17 smithi076 conmon[62156]: cluster 2021-03-15T22:21:16.993038+0000 2021-03-15T22:21:17.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:17 smithi076 conmon[62156]: mgr.y (mgr.14144) 416 : cluster [DBG] pgmap v369: 144 pgs: 1 clean+premerge+peered, 143 active+clean; 153 MiB data, 458 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 3.6 MiB/s wr, 20 op/s 2021-03-15T22:21:17.588 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:17.614 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:17.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:17.716 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:17.739 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:17.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:17.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:17.786 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-15T22:21:17.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:17.866 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:17.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:17.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:17.879 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:17.968 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:17.991 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:17.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:17.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:18.045 INFO:tasks.rados.rados.0.smithi076.stdout:355: done (1 left) 2021-03-15T22:21:18.045 INFO:tasks.rados.rados.0.smithi076.stdout:356: copy_from oid 47 from oid 36 current snap is 35 2021-03-15T22:21:18.045 INFO:tasks.rados.rados.0.smithi076.stdout:357: read oid 12 snap -1 2021-03-15T22:21:18.045 INFO:tasks.rados.rados.0.smithi076.stdout:357: expect (ObjNum 128 snap 24 seq_num 128) 2021-03-15T22:21:18.046 INFO:tasks.rados.rados.0.smithi076.stdout:358: write (excl) oid 26 current snap is 35 2021-03-15T22:21:18.046 INFO:tasks.rados.rados.0.smithi076.stdout:358: seq_num 155 ranges {605085=790655,2170356=434729} 2021-03-15T22:21:18.052 INFO:tasks.rados.rados.0.smithi076.stdout:358: writing smithi07694878-26 from 605085 to 1395740 tid 1 2021-03-15T22:21:18.059 INFO:tasks.rados.rados.0.smithi076.stdout:358: writing smithi07694878-26 from 2170356 to 2605085 tid 2 2021-03-15T22:21:18.059 INFO:tasks.rados.rados.0.smithi076.stdout:359: write oid 48 current snap is 35 2021-03-15T22:21:18.059 INFO:tasks.rados.rados.0.smithi076.stdout:359: seq_num 156 ranges {621892=465904,1712485=671711} 2021-03-15T22:21:18.064 INFO:tasks.rados.rados.0.smithi076.stdout:359: writing smithi07694878-48 from 621892 to 1087796 tid 1 2021-03-15T22:21:18.068 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:18.072 INFO:tasks.rados.rados.0.smithi076.stdout:359: writing smithi07694878-48 from 1712485 to 2384196 tid 2 2021-03-15T22:21:18.072 INFO:tasks.rados.rados.0.smithi076.stdout:360: write oid 19 current snap is 35 2021-03-15T22:21:18.072 INFO:tasks.rados.rados.0.smithi076.stdout:360: seq_num 157 ranges {638699=541153,1770967=531722} 2021-03-15T22:21:18.078 INFO:tasks.rados.rados.0.smithi076.stdout:360: writing smithi07694878-19 from 638699 to 1179852 tid 1 2021-03-15T22:21:18.084 INFO:tasks.rados.rados.0.smithi076.stdout:360: writing smithi07694878-19 from 1770967 to 2302689 tid 2 2021-03-15T22:21:18.085 INFO:tasks.rados.rados.0.smithi076.stdout:361: snap_create 2021-03-15T22:21:18.085 INFO:tasks.rados.rados.0.smithi076.stdout:354: expect (ObjNum 148 snap 31 seq_num 148) 2021-03-15T22:21:18.093 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:18.116 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:18.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:18.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:18.197 INFO:tasks.rados.rados.0.smithi076.stdout:356: finishing copy_from to smithi07694878-47 2021-03-15T22:21:18.197 INFO:tasks.rados.rados.0.smithi076.stdout:356: got expected ENOENT (src dne) 2021-03-15T22:21:18.197 INFO:tasks.rados.rados.0.smithi076.stdout:356: finishing copy_from racing read to smithi07694878-47 2021-03-15T22:21:18.197 INFO:tasks.rados.rados.0.smithi076.stdout:357: expect (ObjNum 128 snap 24 seq_num 128) 2021-03-15T22:21:18.220 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:18.241 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:18.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:18.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:18.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:18 smithi076 conmon[56514]: cluster 2021-03-15T22:21:17.036637+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2021-03-15T22:21:18.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:18 smithi076 conmon[62156]: cluster 2021-03-15T22:21:17 2021-03-15T22:21:18.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:18 smithi076 conmon[62156]: .036637+0000 mon.a (mon.0) 739 : cluster [DBG] osdmap e162: 8 total, 8 up, 8 in 2021-03-15T22:21:18.268 INFO:tasks.rados.rados.0.smithi076.stdout:358: finishing write tid 1 to smithi07694878-26 2021-03-15T22:21:18.270 INFO:tasks.rados.rados.0.smithi076.stdout:358: finishing write tid 2 to smithi07694878-26 2021-03-15T22:21:18.270 INFO:tasks.rados.rados.0.smithi076.stdout:358: finishing write tid 3 to smithi07694878-26 2021-03-15T22:21:18.270 INFO:tasks.rados.rados.0.smithi076.stdout:358: finishing write tid 4 to smithi07694878-26 2021-03-15T22:21:18.270 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 136 (ObjNum 155 snap 35 seq_num 155) dirty exists 2021-03-15T22:21:18.270 INFO:tasks.rados.rados.0.smithi076.stdout:358: left oid 26 (ObjNum 155 snap 35 seq_num 155) 2021-03-15T22:21:18.271 INFO:tasks.rados.rados.0.smithi076.stdout:359: finishing write tid 1 to smithi07694878-48 2021-03-15T22:21:18.271 INFO:tasks.rados.rados.0.smithi076.stdout:359: finishing write tid 2 to smithi07694878-48 2021-03-15T22:21:18.271 INFO:tasks.rados.rados.0.smithi076.stdout:359: finishing write tid 3 to smithi07694878-48 2021-03-15T22:21:18.271 INFO:tasks.rados.rados.0.smithi076.stdout:359: finishing write tid 4 to smithi07694878-48 2021-03-15T22:21:18.272 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 83 (ObjNum 156 snap 35 seq_num 156) dirty exists 2021-03-15T22:21:18.272 INFO:tasks.rados.rados.0.smithi076.stdout:359: left oid 48 (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:21:18.273 INFO:tasks.rados.rados.0.smithi076.stdout:360: finishing write tid 1 to smithi07694878-19 2021-03-15T22:21:18.273 INFO:tasks.rados.rados.0.smithi076.stdout:360: finishing write tid 2 to smithi07694878-19 2021-03-15T22:21:18.273 INFO:tasks.rados.rados.0.smithi076.stdout:360: finishing write tid 3 to smithi07694878-19 2021-03-15T22:21:18.273 INFO:tasks.rados.rados.0.smithi076.stdout:360: finishing write tid 4 to smithi07694878-19 2021-03-15T22:21:18.274 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 122 (ObjNum 157 snap 35 seq_num 157) dirty exists 2021-03-15T22:21:18.274 INFO:tasks.rados.rados.0.smithi076.stdout:360: left oid 19 (ObjNum 157 snap 35 seq_num 157) 2021-03-15T22:21:18.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:18 smithi125 conmon[56739]: cluster 2021-03-15T22:21:17.036637+0000 mon.a (mon.0) 739 : cluster [DBG] 2021-03-15T22:21:18.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:18 smithi125 conmon[56739]: osdmap e162: 8 total, 8 up, 8 in 2021-03-15T22:21:18.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:18.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:18.368 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:18.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:18.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:18.491 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:18.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:18.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:18.593 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:18.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:18.617 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:18.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:18.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:18.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:18.743 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:18.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:18.756 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:21:18.757 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:21:18.757 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2021-03-15T22:21:18.758 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd test-reweight-by-utilization 1000 3.0 2021-03-15T22:21:18.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:18.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:18.868 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:18.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:18.970 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:18.992 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:18.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:18.994 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:19.049 INFO:tasks.rados.rados.0.smithi076.stdout:354: done (6 left) 2021-03-15T22:21:19.050 INFO:tasks.rados.rados.0.smithi076.stdout:356: done (5 left) 2021-03-15T22:21:19.050 INFO:tasks.rados.rados.0.smithi076.stdout:357: done (4 left) 2021-03-15T22:21:19.050 INFO:tasks.rados.rados.0.smithi076.stdout:358: done (3 left) 2021-03-15T22:21:19.051 INFO:tasks.rados.rados.0.smithi076.stdout:359: done (2 left) 2021-03-15T22:21:19.051 INFO:tasks.rados.rados.0.smithi076.stdout:360: done (1 left) 2021-03-15T22:21:19.051 INFO:tasks.rados.rados.0.smithi076.stdout:361: done (0 left) 2021-03-15T22:21:19.051 INFO:tasks.rados.rados.0.smithi076.stdout:362: read oid 45 snap -1 2021-03-15T22:21:19.052 INFO:tasks.rados.rados.0.smithi076.stdout:362: expect deleted 2021-03-15T22:21:19.052 INFO:tasks.rados.rados.0.smithi076.stdout:363: delete oid 42 current snap is 36 2021-03-15T22:21:19.054 INFO:tasks.rados.rados.0.smithi076.stdout:362: done (1 left) 2021-03-15T22:21:19.054 INFO:tasks.rados.rados.0.smithi076.stdout:363: done (0 left) 2021-03-15T22:21:19.054 INFO:tasks.rados.rados.0.smithi076.stdout:364: rollback oid 3 current snap is 36 2021-03-15T22:21:19.054 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 30 2021-03-15T22:21:19.055 INFO:tasks.rados.rados.0.smithi076.stdout:365: write (excl) oid 38 current snap is 36 2021-03-15T22:21:19.055 INFO:tasks.rados.rados.0.smithi076.stdout:365: seq_num 159 ranges {672313=691651,1887931=419038} 2021-03-15T22:21:19.059 INFO:tasks.rados.rados.0.smithi076.stdout:365: writing smithi07694878-38 from 672313 to 1363964 tid 1 2021-03-15T22:21:19.065 INFO:tasks.rados.rados.0.smithi076.stdout:365: writing smithi07694878-38 from 1887931 to 2306969 tid 2 2021-03-15T22:21:19.065 INFO:tasks.rados.rados.0.smithi076.stdout:366: write oid 22 current snap is 36 2021-03-15T22:21:19.065 INFO:tasks.rados.rados.0.smithi076.stdout:364: finishing rollback tid 0 to smithi07694878-3 2021-03-15T22:21:19.066 INFO:tasks.rados.rados.0.smithi076.stdout:366: seq_num 160 ranges {689120=483253,1946413=679049} 2021-03-15T22:21:19.071 INFO:tasks.rados.rados.0.smithi076.stdout:366: writing smithi07694878-22 from 689120 to 1172373 tid 1 2021-03-15T22:21:19.080 INFO:tasks.rados.rados.0.smithi076.stdout:366: writing smithi07694878-22 from 1946413 to 2625462 tid 2 2021-03-15T22:21:19.080 INFO:tasks.rados.rados.0.smithi076.stdout:367: write (excl) oid 8 current snap is 36 2021-03-15T22:21:19.080 INFO:tasks.rados.rados.0.smithi076.stdout:364: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:21:19.081 INFO:tasks.rados.rados.0.smithi076.stdout:364: finishing rollback tid 2 to smithi07694878-3 2021-03-15T22:21:19.081 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 139 (ObjNum 113 snap 24 seq_num 113) dirty exists 2021-03-15T22:21:19.081 INFO:tasks.rados.rados.0.smithi076.stdout:365: finishing write tid 1 to smithi07694878-38 2021-03-15T22:21:19.082 INFO:tasks.rados.rados.0.smithi076.stdout:367: seq_num 161 ranges {705927=558502,2004895=422707} 2021-03-15T22:21:19.089 INFO:tasks.rados.rados.0.smithi076.stdout:367: writing smithi07694878-8 from 705927 to 1264429 tid 1 2021-03-15T22:21:19.094 INFO:tasks.rados.rados.0.smithi076.stdout:367: writing smithi07694878-8 from 2004895 to 2427602 tid 2 2021-03-15T22:21:19.095 INFO:tasks.rados.rados.0.smithi076.stdout:364: done (3 left) 2021-03-15T22:21:19.095 INFO:tasks.rados.rados.0.smithi076.stdout:368: snap_create 2021-03-15T22:21:19.095 INFO:tasks.rados.rados.0.smithi076.stdout:365: finishing write tid 2 to smithi07694878-38 2021-03-15T22:21:19.096 INFO:tasks.rados.rados.0.smithi076.stdout:365: finishing write tid 3 to smithi07694878-38 2021-03-15T22:21:19.096 INFO:tasks.rados.rados.0.smithi076.stdout:365: finishing write tid 4 to smithi07694878-38 2021-03-15T22:21:19.096 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 126 (ObjNum 159 snap 36 seq_num 159) dirty exists 2021-03-15T22:21:19.096 INFO:tasks.rados.rados.0.smithi076.stdout:365: left oid 38 (ObjNum 159 snap 36 seq_num 159) 2021-03-15T22:21:19.097 INFO:tasks.rados.rados.0.smithi076.stdout:366: finishing write tid 1 to smithi07694878-22 2021-03-15T22:21:19.097 INFO:tasks.rados.rados.0.smithi076.stdout:366: finishing write tid 2 to smithi07694878-22 2021-03-15T22:21:19.097 INFO:tasks.rados.rados.0.smithi076.stdout:366: finishing write tid 3 to smithi07694878-22 2021-03-15T22:21:19.097 INFO:tasks.rados.rados.0.smithi076.stdout:366: finishing write tid 4 to smithi07694878-22 2021-03-15T22:21:19.098 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 189 (ObjNum 160 snap 36 seq_num 160) dirty exists 2021-03-15T22:21:19.098 INFO:tasks.rados.rados.0.smithi076.stdout:366: left oid 22 (ObjNum 160 snap 36 seq_num 160) 2021-03-15T22:21:19.098 INFO:tasks.rados.rados.0.smithi076.stdout:367: finishing write tid 1 to smithi07694878-8 2021-03-15T22:21:19.099 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:19.100 INFO:tasks.rados.rados.0.smithi076.stdout:367: finishing write tid 2 to smithi07694878-8 2021-03-15T22:21:19.100 INFO:tasks.rados.rados.0.smithi076.stdout:367: finishing write tid 3 to smithi07694878-8 2021-03-15T22:21:19.101 INFO:tasks.rados.rados.0.smithi076.stdout:367: finishing write tid 4 to smithi07694878-8 2021-03-15T22:21:19.101 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 87 (ObjNum 161 snap 36 seq_num 161) dirty exists 2021-03-15T22:21:19.101 INFO:tasks.rados.rados.0.smithi076.stdout:367: left oid 8 (ObjNum 161 snap 36 seq_num 161) 2021-03-15T22:21:19.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:19.125 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:19.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:19.227 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:19.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:19.250 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:19.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:19.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:19 smithi076 conmon[56514]: cluster 2021-03-15T22:21:18. 2021-03-15T22:21:19.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:19 smithi076 conmon[56514]: 041441+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2021-03-15T22:21:19.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:19 smithi076 conmon[56514]: cluster 2021-03-15T22:21:18.993414+0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v372: 143 pgs: 143 active+clean; 154 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.9 MiB/s wr, 23 op/s 2021-03-15T22:21:19.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:19 smithi076 conmon[56514]: audit 2021-03-15T22:21:18.994349+0000 mon.a (mon.0) 741 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-15T22:21:19.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:19 smithi076 conmon[62156]: cluster 2021-03-15T22:21:18.041441+0000 mon.a 2021-03-15T22:21:19.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:19 smithi076 conmon[62156]: (mon.0) 740 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2021-03-15T22:21:19.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:19 smithi076 conmon[62156]: cluster 2021-03-15T22:21:18.993414+0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v372: 143 pgs: 143 active+clean; 154 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.9 MiB/s wr, 23 op/s 2021-03-15T22:21:19.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:19 smithi076 conmon[62156]: audit 2021-03-15T22:21:18.994349+0000 mon.a (mon.0) 741 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-15T22:21:19.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:19.363 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:19 smithi125 conmon[56739]: cluster 2021-03-15T22:21:18.041441+0000 mon.a (mon.0) 740 : cluster [DBG] osdmap e163: 8 total, 8 up, 8 in 2021-03-15T22:21:19.363 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:19 smithi125 conmon[56739]: cluster 2021-03-15T22:21:18.993414+0000 mgr.y (mgr.14144) 417 : cluster [DBG] pgmap v372: 143 pgs: 143 active+clean; 154 MiB data, 470 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 3.9 MiB/s wr, 23 op/s 2021-03-15T22:21:19.363 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:19 smithi125 conmon[56739]: audit 2021-03-15T22:21:18.994349+0000 mon.a (mon.0) 741 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-15T22:21:19.374 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:19.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:19.376 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:19.477 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:19.499 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:19.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:19.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:19.602 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:19.625 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:19.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:19.635 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:21:19.635 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:21:19.636 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:21:19.636 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:21:19.711 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:21:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:21:19.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:19.730 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:19.751 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:19.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:19.752 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:19.853 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:19.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:19.878 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:19.879 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:19.980 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:20.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:20.003 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:20.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:20.013 INFO:tasks.rados.rados.0.smithi076.stdout:365: done (3 left) 2021-03-15T22:21:20.014 INFO:tasks.rados.rados.0.smithi076.stdout:366: done (2 left) 2021-03-15T22:21:20.014 INFO:tasks.rados.rados.0.smithi076.stdout:367: done (1 left) 2021-03-15T22:21:20.015 INFO:tasks.rados.rados.0.smithi076.stdout:368: done (0 left) 2021-03-15T22:21:20.015 INFO:tasks.rados.rados.0.smithi076.stdout:369: delete oid 30 current snap is 37 2021-03-15T22:21:20.018 INFO:tasks.rados.rados.0.smithi076.stdout:369: done (0 left) 2021-03-15T22:21:20.018 INFO:tasks.rados.rados.0.smithi076.stdout:370: snap_remove snap 36 2021-03-15T22:21:20.024 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:21:20.104 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:20.127 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:20.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:20.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:20.183 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:21:20.201 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:21:20.218 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:21:20.229 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:20.234 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:21:20.250 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:20.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:20.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:20.253 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:21:20.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[56514]: audit 2021-03-15T22:21:19.047361+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-15T22:21:20.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[56514]: cluster 2021-03-15T22:21:19.047456+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2021-03-15T22:21:20.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[56514]: audit 2021-03-15T22:21:19.477458+0000 mgr.y (mgr.14144) 418 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:20.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[56514]: audit 2021-03-15T22:21:19.710266+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.76:0/616879231' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:21:20.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[56514]: cluster 2021-03-15T22:21:20.005116+0000 mon.a (mon.0) 745 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:20.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[56514]: audit 2021-03-15T22:21:20.009993+0000 mon.a (mon.0) 746 : audit [INF] from='client.? 172.21.15.76:0/616879231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:21:20.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[56514]: cluster 2021-03-15T22:21:20.010042+0000 mon.a (mon.0) 747 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2021-03-15T22:21:20.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[62156]: audit 2021-03-15T22:21:19.047361+0000 mon.a (mon.0) 742 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-15T22:21:20.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[62156]: cluster 2021-03-15T22:21:19.047456+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2021-03-15T22:21:20.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[62156]: audit 2021-03-15T22:21:19.477458+0000 mgr.y (mgr.14144) 418 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:20.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[62156]: audit 2021-03-15T22:21:19.710266+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.76:0/616879231' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:21:20.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[62156]: cluster 2021-03-15T22:21:20.005116+0000 mon.a (mon.0) 745 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:20.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[62156]: audit 2021 2021-03-15T22:21:20.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[62156]: -03-15T22:21:20.009993+0000 mon.a (mon.0) 746 : audit [INF] from='client.? 172.21.15.76:0/616879231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:21:20.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:20 smithi076 conmon[62156]: cluster 2021-03-15T22:21:20.010042+0000 mon.a (mon.0) 747 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2021-03-15T22:21:20.271 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:21:20.289 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:21:20.308 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:21:20.309 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:21:20.310 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:21:20.310 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:21:20.310 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:21:20.310 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:21:20.311 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:21:20.311 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:21:20.311 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:21:20.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-15T22:21:20.353 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:20.378 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:20.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:20.480 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:20.502 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:20.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:20.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:20.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: audit 2021-03-15T22:21:19.047361+0000 mon.a ( 2021-03-15T22:21:20.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: mon.0) 742 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-15T22:21:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: cluster 2021-03-15T22:21:19.047456+0000 mon.a (mon.0) 743 : cluster [DBG] osdmap e164: 8 total, 8 up, 8 in 2021-03-15T22:21:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: audit 2021-03-15T22:21:19.477458+0000 mgr.y (mgr.14144) 418 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: audit 2021-03-15T22:21:19.710266+0000 mon.a (mon.0) 744 : audit [INF] from='client.? 172.21.15.76:0/616879231' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:21:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:21:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: -15T22:21:20.005116+0000 mon.a (mon.0) 745 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: audit 2021-03-15T22:21:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: 2021-03-15T22:21:20.009993 2021-03-15T22:21:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: +0000 mon.a (mon.0) 746 : audit 2021-03-15T22:21:20.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/616879231' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:21:20.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:21:20.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: T22:21:20.010042+0000 mon.a ( 2021-03-15T22:21:20.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:20 smithi125 conmon[56739]: mon.0) 747 : cluster [DBG] osdmap e165: 8 total, 8 up, 8 in 2021-03-15T22:21:20.589 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:20.605 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:20.628 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:20.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:20.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:20.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:20.752 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:20.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:20.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:20.855 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:20.878 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:20.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:20.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:20.980 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:21.002 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:21.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:21.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:21.025 INFO:tasks.rados.rados.0.smithi076.stdout:370: done (0 left) 2021-03-15T22:21:21.025 INFO:tasks.rados.rados.0.smithi076.stdout:371: copy_from oid 46 from oid 21 current snap is 37 2021-03-15T22:21:21.026 INFO:tasks.rados.rados.0.smithi076.stdout:372: read oid 29 snap -1 2021-03-15T22:21:21.026 INFO:tasks.rados.rados.0.smithi076.stdout:372: expect (ObjNum 132 snap 25 seq_num 132) 2021-03-15T22:21:21.026 INFO:tasks.rados.rados.0.smithi076.stdout:373: write (excl) oid 33 current snap is 37 2021-03-15T22:21:21.026 INFO:tasks.rados.rados.0.smithi076.stdout:373: seq_num 163 ranges {739541=709000,2121859=426376} 2021-03-15T22:21:21.033 INFO:tasks.rados.rados.0.smithi076.stdout:373: writing smithi07694878-33 from 739541 to 1448541 tid 1 2021-03-15T22:21:21.041 INFO:tasks.rados.rados.0.smithi076.stdout:373: writing smithi07694878-33 from 2121859 to 2548235 tid 2 2021-03-15T22:21:21.041 INFO:tasks.rados.rados.0.smithi076.stdout:374: read oid 27 snap -1 2021-03-15T22:21:21.041 INFO:tasks.rados.rados.0.smithi076.stdout:374: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:21:21.041 INFO:tasks.rados.rados.0.smithi076.stdout:375: write oid 41 current snap is 37 2021-03-15T22:21:21.041 INFO:tasks.rados.rados.0.smithi076.stdout:375: seq_num 164 ranges {756348=784249,2063988=686387} 2021-03-15T22:21:21.049 INFO:tasks.rados.rados.0.smithi076.stdout:375: writing smithi07694878-41 from 756348 to 1540597 tid 1 2021-03-15T22:21:21.056 INFO:tasks.rados.rados.0.smithi076.stdout:375: writing smithi07694878-41 from 2063988 to 2750375 tid 2 2021-03-15T22:21:21.056 INFO:tasks.rados.rados.0.smithi076.stdout:376: copy_from oid 18 from oid 14 current snap is 37 2021-03-15T22:21:21.056 INFO:tasks.rados.rados.0.smithi076.stdout:372: expect (ObjNum 132 snap 25 seq_num 132) 2021-03-15T22:21:21.105 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:21.118 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[56514]: audit 2021-03-15T22:21:21.118 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[56514]: 2021-03 2021-03-15T22:21:21.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[56514]: -15T22:21:20.164612+0000 mon.a (mon.0) 748 : audit [DBG] from='client.? 172.21.15.76:0/1995802362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:21.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[56514]: 2021-03-15T22:21:21.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:21:21.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[56514]: -03-15T22:21:20.994075 2021-03-15T22:21:21.120 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 419 : cluster [DBG] 2021-03-15T22:21:21.120 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[56514]: pgmap v375: 143 pgs: 2 peering, 141 active+clean; 157 MiB data, 481 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2021-03-15T22:21:21.120 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[56514]: cluster 2021-03-15T22:21:21. 2021-03-15T22:21:21.120 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[56514]: 022554+0000 mon.a (mon.0) 749 : cluster 2021-03-15T22:21:21.120 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[56514]: [DBG] osdmap e166: 8 total, 8 up, 8 in 2021-03-15T22:21:21.121 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[62156]: audit 2021-03-15T22:21:20.164612 2021-03-15T22:21:21.121 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[62156]: +0000 mon.a (mon.0) 748 : audit [DBG] from='client.? 172.21.15.76:0/1995802362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:21.121 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[62156]: cluster 2021-03-15T22:21:21.122 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[62156]: 2021-03-15T22:21:20.994075+0000 mgr.y (mgr.14144) 419 : 2021-03-15T22:21:21.122 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[62156]: cluster [DBG] pgmap v375: 143 pgs: 2 peering, 141 active+clean; 157 MiB data, 481 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2021-03-15T22:21:21.122 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:21:21.122 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[62156]: 22:21:21.022554+0000 mon.a (mon. 2021-03-15T22:21:21.123 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:21 smithi076 conmon[62156]: 0) 749 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2021-03-15T22:21:21.131 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:21.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:21.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:21.152 INFO:tasks.rados.rados.0.smithi076.stdout:373: finishing write tid 1 to smithi07694878-33 2021-03-15T22:21:21.152 INFO:tasks.rados.rados.0.smithi076.stdout:373: finishing write tid 2 to smithi07694878-33 2021-03-15T22:21:21.153 INFO:tasks.rados.rados.0.smithi076.stdout:373: finishing write tid 3 to smithi07694878-33 2021-03-15T22:21:21.153 INFO:tasks.rados.rados.0.smithi076.stdout:372: done (5 left) 2021-03-15T22:21:21.153 INFO:tasks.rados.rados.0.smithi076.stdout:377: delete oid 31 current snap is 37 2021-03-15T22:21:21.153 INFO:tasks.rados.rados.0.smithi076.stdout:374: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:21:21.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:21.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:21.256 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:21.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:21.303 INFO:tasks.rados.rados.0.smithi076.stdout:371: finishing copy_from to smithi07694878-46 2021-03-15T22:21:21.304 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 185 (ObjNum 143 snap 28 seq_num 143) dirty exists 2021-03-15T22:21:21.304 INFO:tasks.rados.rados.0.smithi076.stdout:373: finishing write tid 4 to smithi07694878-33 2021-03-15T22:21:21.304 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 97 (ObjNum 163 snap 37 seq_num 163) dirty exists 2021-03-15T22:21:21.304 INFO:tasks.rados.rados.0.smithi076.stdout:373: left oid 33 (ObjNum 163 snap 37 seq_num 163) 2021-03-15T22:21:21.305 INFO:tasks.rados.rados.0.smithi076.stdout:373: done (5 left) 2021-03-15T22:21:21.305 INFO:tasks.rados.rados.0.smithi076.stdout:374: done (4 left) 2021-03-15T22:21:21.305 INFO:tasks.rados.rados.0.smithi076.stdout:377: done (3 left) 2021-03-15T22:21:21.305 INFO:tasks.rados.rados.0.smithi076.stdout:378: read oid 22 snap -1 2021-03-15T22:21:21.306 INFO:tasks.rados.rados.0.smithi076.stdout:378: expect (ObjNum 160 snap 36 seq_num 160) 2021-03-15T22:21:21.306 INFO:tasks.rados.rados.0.smithi076.stdout:379: delete oid 45 current snap is 37 2021-03-15T22:21:21.306 INFO:tasks.rados.rados.0.smithi076.stdout:371: finishing copy_from racing read to smithi07694878-46 2021-03-15T22:21:21.306 INFO:tasks.rados.rados.0.smithi076.stdout:375: finishing write tid 1 to smithi07694878-41 2021-03-15T22:21:21.306 INFO:tasks.rados.rados.0.smithi076.stdout:376: finishing copy_from to smithi07694878-18 2021-03-15T22:21:21.307 INFO:tasks.rados.rados.0.smithi076.stdout:376: got expected ENOENT (src dne) 2021-03-15T22:21:21.307 INFO:tasks.rados.rados.0.smithi076.stdout:376: finishing copy_from racing read to smithi07694878-18 2021-03-15T22:21:21.307 INFO:tasks.rados.rados.0.smithi076.stdout:375: finishing write tid 2 to smithi07694878-41 2021-03-15T22:21:21.307 INFO:tasks.rados.rados.0.smithi076.stdout:375: finishing write tid 3 to smithi07694878-41 2021-03-15T22:21:21.308 INFO:tasks.rados.rados.0.smithi076.stdout:375: finishing write tid 4 to smithi07694878-41 2021-03-15T22:21:21.308 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 90 (ObjNum 164 snap 37 seq_num 164) dirty exists 2021-03-15T22:21:21.308 INFO:tasks.rados.rados.0.smithi076.stdout:375: left oid 41 (ObjNum 164 snap 37 seq_num 164) 2021-03-15T22:21:21.309 INFO:tasks.rados.rados.0.smithi076.stdout:371: done (4 left) 2021-03-15T22:21:21.309 INFO:tasks.rados.rados.0.smithi076.stdout:375: done (3 left) 2021-03-15T22:21:21.310 INFO:tasks.rados.rados.0.smithi076.stdout:376: done (2 left) 2021-03-15T22:21:21.310 INFO:tasks.rados.rados.0.smithi076.stdout:379: done (1 left) 2021-03-15T22:21:21.310 INFO:tasks.rados.rados.0.smithi076.stdout:380: rollback oid 39 current snap is 37 2021-03-15T22:21:21.311 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 39 to 30 2021-03-15T22:21:21.311 INFO:tasks.rados.rados.0.smithi076.stdout:381: rollback oid 34 current snap is 37 2021-03-15T22:21:21.311 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 34 to 35 2021-03-15T22:21:21.311 INFO:tasks.rados.rados.0.smithi076.stdout:382: write oid 11 current snap is 37 2021-03-15T22:21:21.312 INFO:tasks.rados.rados.0.smithi076.stdout:382: seq_num 167 ranges {406769=609996,1439434=433714,2303522=503247} 2021-03-15T22:21:21.312 INFO:tasks.rados.rados.0.smithi076.stdout:382: writing smithi07694878-11 from 406769 to 1016765 tid 1 2021-03-15T22:21:21.315 INFO:tasks.rados.rados.0.smithi076.stdout:382: writing smithi07694878-11 from 1439434 to 1873148 tid 2 2021-03-15T22:21:21.320 INFO:tasks.rados.rados.0.smithi076.stdout:382: writing smithi07694878-11 from 2303522 to 2806769 tid 3 2021-03-15T22:21:21.321 INFO:tasks.rados.rados.0.smithi076.stdout:383: read oid 12 snap 35 2021-03-15T22:21:21.321 INFO:tasks.rados.rados.0.smithi076.stdout:383: expect (ObjNum 128 snap 24 seq_num 128) 2021-03-15T22:21:21.321 INFO:tasks.rados.rados.0.smithi076.stdout:384: copy_from oid 21 from oid 33 current snap is 37 2021-03-15T22:21:21.321 INFO:tasks.rados.rados.0.smithi076.stdout:380: finishing rollback tid 0 to smithi07694878-39 2021-03-15T22:21:21.322 INFO:tasks.rados.rados.0.smithi076.stdout:380: finishing rollback tid 1 to smithi07694878-39 2021-03-15T22:21:21.322 INFO:tasks.rados.rados.0.smithi076.stdout:380: finishing rollback tid 2 to smithi07694878-39 2021-03-15T22:21:21.322 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 130 (ObjNum 116 snap 24 seq_num 116) dirty exists 2021-03-15T22:21:21.322 INFO:tasks.rados.rados.0.smithi076.stdout:378: expect (ObjNum 160 snap 36 seq_num 160) 2021-03-15T22:21:21.358 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:21.379 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:21.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:21.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:21.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:21.499 INFO:tasks.rados.rados.0.smithi076.stdout:381: finishing rollback tid 0 to smithi07694878-34 2021-03-15T22:21:21.499 INFO:tasks.rados.rados.0.smithi076.stdout:381: finishing rollback tid 1 to smithi07694878-34 2021-03-15T22:21:21.499 INFO:tasks.rados.rados.0.smithi076.stdout:381: finishing rollback tid 2 to smithi07694878-34 2021-03-15T22:21:21.499 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 101 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:21:21.500 INFO:tasks.rados.rados.0.smithi076.stdout:382: finishing write tid 1 to smithi07694878-11 2021-03-15T22:21:21.500 INFO:tasks.rados.rados.0.smithi076.stdout:378: done (5 left) 2021-03-15T22:21:21.500 INFO:tasks.rados.rados.0.smithi076.stdout:380: done (4 left) 2021-03-15T22:21:21.500 INFO:tasks.rados.rados.0.smithi076.stdout:381: done (3 left) 2021-03-15T22:21:21.501 INFO:tasks.rados.rados.0.smithi076.stdout:385: snap_remove snap 30 2021-03-15T22:21:21.504 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:21.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:21.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:21 smithi125 conmon[56739]: audit 2021-03-15T22:21:20.164612+0000 mon.a (mon.0) 748 : audit [DBG] from='client.? 172.21.15.76:0/1995802362' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:21.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:21 smithi125 conmon[56739]: cluster 2021-03-15T22:21:20.994075+0000 mgr.y (mgr.14144) 419 : cluster [DBG] pgmap v375: 143 pgs: 2 peering, 141 active+clean; 157 MiB data, 481 MiB used, 715 GiB / 715 GiB avail; 6.6 MiB/s rd, 1.0 MiB/s wr, 11 op/s 2021-03-15T22:21:21.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:21 smithi125 conmon[56739]: cluster 2021-03-15T22:21:21.022554+0000 mon.a (mon.0) 749 : cluster [DBG] osdmap e166: 8 total, 8 up, 8 in 2021-03-15T22:21:21.606 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:21.630 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:21.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:21.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:21.732 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:21.754 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:21.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:21.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:21.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:21.880 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:21.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:21.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:22.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:22.006 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:22.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:22.027 INFO:tasks.rados.rados.0.smithi076.stdout:385: done (3 left) 2021-03-15T22:21:22.028 INFO:tasks.rados.rados.0.smithi076.stdout:386: delete oid 6 current snap is 37 2021-03-15T22:21:22.028 INFO:tasks.rados.rados.0.smithi076.stdout:382: finishing write tid 2 to smithi07694878-11 2021-03-15T22:21:22.028 INFO:tasks.rados.rados.0.smithi076.stdout:383: expect (ObjNum 128 snap 24 seq_num 128) 2021-03-15T22:21:22.084 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:21:22.085 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:21:22.104 INFO:tasks.rados.rados.0.smithi076.stdout:382: finishing write tid 3 to smithi07694878-11 2021-03-15T22:21:22.104 INFO:tasks.rados.rados.0.smithi076.stdout:382: finishing write tid 4 to smithi07694878-11 2021-03-15T22:21:22.104 INFO:tasks.rados.rados.0.smithi076.stdout:382: finishing write tid 5 to smithi07694878-11 2021-03-15T22:21:22.105 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 107 (ObjNum 167 snap 37 seq_num 167) dirty exists 2021-03-15T22:21:22.105 INFO:tasks.rados.rados.0.smithi076.stdout:382: left oid 11 (ObjNum 167 snap 37 seq_num 167) 2021-03-15T22:21:22.105 INFO:tasks.rados.rados.0.smithi076.stdout:384: finishing copy_from to smithi07694878-21 2021-03-15T22:21:22.105 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 97 (ObjNum 163 snap 37 seq_num 163) dirty exists 2021-03-15T22:21:22.106 INFO:tasks.rados.rados.0.smithi076.stdout:384: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:21:22.106 INFO:tasks.rados.rados.0.smithi076.stdout:382: done (3 left) 2021-03-15T22:21:22.106 INFO:tasks.rados.rados.0.smithi076.stdout:383: done (2 left) 2021-03-15T22:21:22.106 INFO:tasks.rados.rados.0.smithi076.stdout:384: done (1 left) 2021-03-15T22:21:22.107 INFO:tasks.rados.rados.0.smithi076.stdout:386: done (0 left) 2021-03-15T22:21:22.107 INFO:tasks.rados.rados.0.smithi076.stdout:387: snap_remove snap 24 2021-03-15T22:21:22.107 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:22.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:22.132 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:22.132 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:22.234 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:22.256 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:22.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:22.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:22.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:22.380 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:22.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:22.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:22 smithi076 conmon[56514]: cluster 2021-03-15T22:21:20.673598+0000 osd.7 2021-03-15T22:21:22.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:22 smithi076 conmon[56514]: (osd.7) 41 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:21:22.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:22 smithi076 conmon[56514]: cluster 2021-03-15T22:21:20.983184+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:21:22.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:22 smithi076 conmon[56514]: cluster 2021-03-15T22:21:22.020535+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2021-03-15T22:21:22.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:22 smithi076 conmon[62156]: cluster 2021-03-15T22:21:20.673598+0000 osd.7 (osd 2021-03-15T22:21:22.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:22 smithi076 conmon[62156]: .7) 41 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:21:22.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:22 smithi076 conmon[62156]: cluster 2021-03-15T22:21:20.983184 2021-03-15T22:21:22.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:22 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 38 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:21:22.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:22 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:21:22.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:22 smithi076 conmon[62156]: :21:22.020535+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2021-03-15T22:21:22.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:22 smithi076 conmon[62156]: 2021-03-15T22:21:22.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:22.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:22.505 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:22.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:22 smithi125 conmon[56739]: cluster 2021-03-15T22:21:20.673598+0000 osd.7 (osd.7) 41 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:21:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:22 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:21:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:22 smithi125 conmon[56739]: 22:21:20.983184+0000 osd.5 (osd.5) 38 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:21:22.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:22 smithi125 conmon[56739]: cluster 2021-03-15T22:21:22.020535+0000 mon.a (mon.0) 750 : cluster [DBG] osdmap e167: 8 total, 8 up, 8 in 2021-03-15T22:21:22.597 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 426 (0%) 2021-03-15T22:21:22.598 INFO:teuthology.orchestra.run.smithi076.stdout:avg 53.25 2021-03-15T22:21:22.598 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 5.56215 -> 5.56215 (expected baseline 6.82596) 2021-03-15T22:21:22.598 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.2 with 44 -> 44 pgs (0.826291 -> 0.826291 * mean) 2021-03-15T22:21:22.598 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.3 with 62 -> 62 pgs (1.16432 -> 1.16432 * mean) 2021-03-15T22:21:22.599 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:21:22.599 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:21:22.599 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 3 2021-03-15T22:21:22.599 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:21:22.600 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0007 2021-03-15T22:21:22.600 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0067 2021-03-15T22:21:22.600 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:21:22.606 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:22.628 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:22.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:22.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:22.754 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:22.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:22.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:22.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-15T22:21:22.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:22.893 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:22.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:22.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:22.996 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:23.019 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:23.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:23.020 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:23.022 INFO:tasks.rados.rados.0.smithi076.stdout:387: done (0 left) 2021-03-15T22:21:23.023 INFO:tasks.rados.rados.0.smithi076.stdout:388: rollback oid 18 current snap is 37 2021-03-15T22:21:23.023 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 18 to 35 2021-03-15T22:21:23.023 INFO:tasks.rados.rados.0.smithi076.stdout:389: delete oid 40 current snap is 37 2021-03-15T22:21:23.023 INFO:tasks.rados.rados.0.smithi076.stdout:388: finishing rollback tid 1 to smithi07694878-18 2021-03-15T22:21:23.024 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 0 (ObjNum 262038896 snap 21896 seq_num 261938160) dirty dne 2021-03-15T22:21:23.027 INFO:journalctl@ceph.osd.7.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[70700]: debug 2021-03-15T22:21:23.023+0000 7f3807536700 -1 osd.7 pg_epoch: 168 pg[7.5( v 166'62 (0'0,166'62] local-lis/les=91/92 n=3 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [7,2,0] r=0 lpr=91 crt=157'61 lcod 157'61 mlcod 157'61 active+clean+snaptrim trimq=[30~1] ps=[3d~2,44~1]] removing snap head 2021-03-15T22:21:23.037 INFO:tasks.rados.rados.0.smithi076.stdout:388: done (1 left) 2021-03-15T22:21:23.037 INFO:tasks.rados.rados.0.smithi076.stdout:389: done (0 left) 2021-03-15T22:21:23.037 INFO:tasks.rados.rados.0.smithi076.stdout:390: write oid 50 current snap is 37 2021-03-15T22:21:23.038 INFO:tasks.rados.rados.0.smithi076.stdout:390: seq_num 170 ranges {457190=552096,1614880=581041,2785791=71399} 2021-03-15T22:21:23.038 INFO:tasks.rados.rados.0.smithi076.stdout:390: writing smithi07694878-50 from 457190 to 1009286 tid 1 2021-03-15T22:21:23.045 INFO:tasks.rados.rados.0.smithi076.stdout:390: writing smithi07694878-50 from 1614880 to 2195921 tid 2 2021-03-15T22:21:23.049 INFO:tasks.rados.rados.0.smithi076.stdout:390: writing smithi07694878-50 from 2785791 to 2857190 tid 3 2021-03-15T22:21:23.049 INFO:tasks.rados.rados.0.smithi076.stdout:391: rollback oid 32 current snap is 37 2021-03-15T22:21:23.050 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 32 to 33 2021-03-15T22:21:23.050 INFO:tasks.rados.rados.0.smithi076.stdout:390: finishing write tid 1 to smithi07694878-50 2021-03-15T22:21:23.050 INFO:tasks.rados.rados.0.smithi076.stdout:390: finishing write tid 2 to smithi07694878-50 2021-03-15T22:21:23.050 INFO:tasks.rados.rados.0.smithi076.stdout:392: rollback oid 29 current snap is 37 2021-03-15T22:21:23.051 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 29 to 35 2021-03-15T22:21:23.051 INFO:tasks.rados.rados.0.smithi076.stdout:393: read oid 37 snap 32 2021-03-15T22:21:23.051 INFO:tasks.rados.rados.0.smithi076.stdout:393: expect deleted 2021-03-15T22:21:23.051 INFO:tasks.rados.rados.0.smithi076.stdout:394: copy_from oid 7 from oid 15 current snap is 37 2021-03-15T22:21:23.052 INFO:tasks.rados.rados.0.smithi076.stdout:395: rollback oid 42 current snap is 37 2021-03-15T22:21:23.052 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 42 to 33 2021-03-15T22:21:23.052 INFO:tasks.rados.rados.0.smithi076.stdout:396: delete oid 10 current snap is 37 2021-03-15T22:21:23.054 INFO:tasks.rados.rados.0.smithi076.stdout:390: finishing write tid 3 to smithi07694878-50 2021-03-15T22:21:23.054 INFO:tasks.rados.rados.0.smithi076.stdout:396: done (6 left) 2021-03-15T22:21:23.055 INFO:tasks.rados.rados.0.smithi076.stdout:397: read oid 33 snap 34 2021-03-15T22:21:23.055 INFO:tasks.rados.rados.0.smithi076.stdout:397: expect (ObjNum 148 snap 31 seq_num 148) 2021-03-15T22:21:23.055 INFO:tasks.rados.rados.0.smithi076.stdout:398: read oid 43 snap -1 2021-03-15T22:21:23.055 INFO:tasks.rados.rados.0.smithi076.stdout:398: expect deleted 2021-03-15T22:21:23.056 INFO:tasks.rados.rados.0.smithi076.stdout:399: read oid 19 snap 35 2021-03-15T22:21:23.056 INFO:tasks.rados.rados.0.smithi076.stdout:399: expect (ObjNum 157 snap 35 seq_num 157) 2021-03-15T22:21:23.056 INFO:tasks.rados.rados.0.smithi076.stdout:400: rollback oid 5 current snap is 37 2021-03-15T22:21:23.056 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 33 2021-03-15T22:21:23.056 INFO:tasks.rados.rados.0.smithi076.stdout:390: finishing write tid 4 to smithi07694878-50 2021-03-15T22:21:23.057 INFO:tasks.rados.rados.0.smithi076.stdout:391: finishing rollback tid 0 to smithi07694878-32 2021-03-15T22:21:23.057 INFO:tasks.rados.rados.0.smithi076.stdout:390: finishing write tid 5 to smithi07694878-50 2021-03-15T22:21:23.057 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 143 (ObjNum 170 snap 37 seq_num 170) dirty exists 2021-03-15T22:21:23.057 INFO:tasks.rados.rados.0.smithi076.stdout:390: left oid 50 (ObjNum 170 snap 37 seq_num 170) 2021-03-15T22:21:23.058 INFO:tasks.rados.rados.0.smithi076.stdout:390: done (9 left) 2021-03-15T22:21:23.058 INFO:tasks.rados.rados.0.smithi076.stdout:401: rollback oid 39 current snap is 37 2021-03-15T22:21:23.058 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 39 to 35 2021-03-15T22:21:23.058 INFO:tasks.rados.rados.0.smithi076.stdout:402: read oid 13 snap 33 2021-03-15T22:21:23.058 INFO:tasks.rados.rados.0.smithi076.stdout:402: expect (ObjNum 150 snap 32 seq_num 150) 2021-03-15T22:21:23.059 INFO:tasks.rados.rados.0.smithi076.stdout:403: read oid 31 snap -1 2021-03-15T22:21:23.059 INFO:tasks.rados.rados.0.smithi076.stdout:403: expect deleted 2021-03-15T22:21:23.059 INFO:tasks.rados.rados.0.smithi076.stdout:404: delete oid 49 current snap is 37 2021-03-15T22:21:23.059 INFO:tasks.rados.rados.0.smithi076.stdout:391: finishing rollback tid 1 to smithi07694878-32 2021-03-15T22:21:23.059 INFO:tasks.rados.rados.0.smithi076.stdout:391: finishing rollback tid 2 to smithi07694878-32 2021-03-15T22:21:23.060 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 230 (ObjNum 108 snap 23 seq_num 108) dirty exists 2021-03-15T22:21:23.060 INFO:tasks.rados.rados.0.smithi076.stdout:392: finishing rollback tid 0 to smithi07694878-29 2021-03-15T22:21:23.060 INFO:tasks.rados.rados.0.smithi076.stdout:397: expect (ObjNum 148 snap 31 seq_num 148) 2021-03-15T22:21:23.117 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:23.121 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:23.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:23.145 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:23.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:23.172 INFO:tasks.rados.rados.0.smithi076.stdout:392: finishing rollback tid 1 to smithi07694878-29 2021-03-15T22:21:23.173 INFO:tasks.rados.rados.0.smithi076.stdout:392: finishing rollback tid 2 to smithi07694878-29 2021-03-15T22:21:23.173 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 234 (ObjNum 132 snap 25 seq_num 132) dirty exists 2021-03-15T22:21:23.173 INFO:tasks.rados.rados.0.smithi076.stdout:395: finishing rollback tid 1 to smithi07694878-42 2021-03-15T22:21:23.173 INFO:tasks.rados.rados.0.smithi076.stdout:395: finishing rollback tid 2 to smithi07694878-42 2021-03-15T22:21:23.174 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 236 (ObjNum 125 snap 24 seq_num 125) dirty exists 2021-03-15T22:21:23.174 INFO:tasks.rados.rados.0.smithi076.stdout:391: done (12 left) 2021-03-15T22:21:23.174 INFO:tasks.rados.rados.0.smithi076.stdout:392: done (11 left) 2021-03-15T22:21:23.174 INFO:tasks.rados.rados.0.smithi076.stdout:393: done (10 left) 2021-03-15T22:21:23.174 INFO:tasks.rados.rados.0.smithi076.stdout:395: done (9 left) 2021-03-15T22:21:23.175 INFO:tasks.rados.rados.0.smithi076.stdout:397: done (8 left) 2021-03-15T22:21:23.175 INFO:tasks.rados.rados.0.smithi076.stdout:398: done (7 left) 2021-03-15T22:21:23.175 INFO:tasks.rados.rados.0.smithi076.stdout:403: done (6 left) 2021-03-15T22:21:23.175 INFO:tasks.rados.rados.0.smithi076.stdout:404: done (5 left) 2021-03-15T22:21:23.176 INFO:tasks.rados.rados.0.smithi076.stdout:405: rollback oid 24 current snap is 37 2021-03-15T22:21:23.176 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 34 2021-03-15T22:21:23.176 INFO:tasks.rados.rados.0.smithi076.stdout:400: finishing rollback tid 0 to smithi07694878-5 2021-03-15T22:21:23.176 INFO:tasks.rados.rados.0.smithi076.stdout:402: expect (ObjNum 150 snap 32 seq_num 150) 2021-03-15T22:21:23.246 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:23.268 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:23.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:23.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:23.286 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[63905]: debug 2021-03-15T22:21:23.038+0000 7fce00c31700 -1 osd.5 pg_epoch: 168 pg[7.0( v 168'224 (0'0,168'224] local-lis/les=91/92 n=11 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [5,1,3] r=0 lpr=91 crt=168'224 lcod 168'222 mlcod 168'222 active+clean+snaptrim trimq=[30~1] ps=[3d~2,44~1]] removing snap head 2021-03-15T22:21:23.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: cluster 2021-03-15T22:21:21.773158+0000 osd.1 (osd.1) 37 : 2021-03-15T22:21:23.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: cluster [DBG] 4.2 scrub ok 2021-03-15T22:21:23.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: cluster 2021-03-15T22:21:21.929716+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:21:23.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: cluster 2021-03-15T22:21:22.029833+0000 osd.5 ( 2021-03-15T22:21:23.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: osd.5) 39 : cluster [DBG] 6.d scrub ok 2021-03-15T22:21:23.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: cluster 2021-03-15T22:21 2021-03-15T22:21:23.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: :22.324832+0000 osd.4 (osd.4) 43 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:21:23.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:21:23.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: :21:22.596132+0000 mgr.y (mgr.14144) 420 : audit 2021-03-15T22:21:23.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: [DBG] from='client.14877 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:23.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: cluster 2021-03-15T22:21:23.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: 2021-03-15T22:21:22.794334+0000 osd.0 (osd.0) 38 : cluster [DBG] 2021-03-15T22:21:23.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: 3.1f scrub ok 2021-03-15T22:21:23.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:21:23.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: -15T22:21:22.994503+0000 mgr.y (mgr.14144) 421 : cluster [DBG] pgmap v378: 142 pgs: 1 peering, 141 active+clean; 164 MiB data, 503 MiB used, 715 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.5 MiB/s wr, 19 op/s 2021-03-15T22:21:23.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: 2021-03-15T22:21:23.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:21:23.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: T22:21:22.995215+0000 mon.a (mon.0) 751 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-03-15T22:21:23.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:21:23.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: 03-15T22:21:23.020469+0000 mon.a (mon.0 2021-03-15T22:21:23.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: ) 752 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2021-03-15T22:21:23.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: cluster 2021-03-15T22:21:23.020628+0000 2021-03-15T22:21:23.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:23 smithi125 conmon[56739]: mon.a (mon.0) 753 : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2021-03-15T22:21:23.298 INFO:tasks.rados.rados.0.smithi076.stdout:400: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:21:23.298 INFO:tasks.rados.rados.0.smithi076.stdout:400: finishing rollback tid 2 to smithi07694878-5 2021-03-15T22:21:23.299 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 39 (ObjNum 103 snap 21 seq_num 103) dirty exists 2021-03-15T22:21:23.299 INFO:tasks.rados.rados.0.smithi076.stdout:399: expect (ObjNum 157 snap 35 seq_num 157) 2021-03-15T22:21:23.308 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[65511]: debug 2021-03-15T22:21:23.308 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[65511]: 2021-03-15T22:21:23.032+0000 7f1a1f877700 -1 osd.0 pg_epoch: 168 pg[7.6( v 168'201 (0'0,168'201] local-lis/les=91/92 n=11 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [0,7,3] r=0 lpr=91 crt=168'201 lcod 168'199 mlcod 168'199 active+clean+snaptrim trimq=[30~1] ps=[3d~2,44~1]] removing snap head 2021-03-15T22:21:23.309 INFO:journalctl@ceph.osd.1.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[68886]: debug 2021-03-15T22:21:23.019+0000 7fae34dad700 -1 osd.1 pg_epoch: 168 pg[7.b( v 148'72 (0'0,148'72] local-lis/les=91/92 n=3 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [1,5,6] r=0 lpr=91 crt=148'72 lcod 148'70 mlcod 148'70 active+clean+snaptrim trimq=[30~1] ps=[3d~2,44~1]] removing snap head 2021-03-15T22:21:23.309 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[75764]: debug 2021-03-15T22:21:23.035+0000 7f9fba7ae700 -1 osd.3 pg_epoch: 168 pg[7.3( v 167'92 (0'0,167'92] local-lis/les=91/92 n=8 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [3,2,4] r=0 lpr=91 crt=167'92 lcod 167'90 mlcod 167'90 active+clean+snaptrim trimq=[30~1] ps=[3d~2,44~1]] removing snap head 2021-03-15T22:21:23.309 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: cluster 2021-03-15T22:21:23.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:21:23.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 22:21:21.773158+0000 osd.1 (osd.1) 37 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:21:23.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:21:23.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: -03-15T22:21:21.929716+0000 2021-03-15T22:21:23.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: osd.6 (osd.6) 21 : cluster 2021-03-15T22:21:23.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: [DBG] 5.2 scrub ok 2021-03-15T22:21:23.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:23.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 03-15T22:21:22.029833 2021-03-15T22:21:23.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 2021-03-15T22:21:23.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 39 : cluster [DBG] 6.d scrub ok 2021-03-15T22:21:23.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 2021-03-15T22:21:23.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:21:23.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: -03-15T22:21:22.324832 2021-03-15T22:21:23.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 43 : 2021-03-15T22:21:23.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: cluster [DBG] 3.14 scrub ok 2021-03-15T22:21:23.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:21:23.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 03-15T22:21:22.596132+0000 mgr.y 2021-03-15T22:21:23.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: (mgr.14144) 420 : audit 2021-03-15T22:21:23.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: [DBG] from='client.14877 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:23.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: cluster 2021-03-15T22:21:23.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 2021-03-15T22:21:22.794334+0000 2021-03-15T22:21:23.315 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: osd.0 (osd.0) 38 : cluster 2021-03-15T22:21:23.315 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: [DBG] 3.1f scrub ok 2021-03-15T22:21:23.315 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:21:23.315 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: T22:21:22.994503+0000 mgr.y (mgr 2021-03-15T22:21:23.315 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: .14144) 421 : cluster [DBG] pgmap v378: 142 pgs: 1 peering, 141 active+clean; 164 MiB data, 503 MiB used, 715 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.5 MiB/s wr, 19 op/s 2021-03-15T22:21:23.316 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 2021-03-15T22:21:23.316 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: audit 2021-03-15T22:21:23.316 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 2021-03-15T22:21:22.995215 2021-03-15T22:21:23.316 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2021-03-15T22:21:23.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 751 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-03-15T22:21:23.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:21:23.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 03-15T22:21:23.020469+0000 2021-03-15T22:21:23.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: mon.a (mon.0) 752 : 2021-03-15T22:21:23.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2021-03-15T22:21:23.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: cluster 2021-03-15T22:21:23.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:21:23.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: 15T22:21:23.020628+0000 mon.a ( 2021-03-15T22:21:23.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: mon.0) 753 : cluster [DBG] 2021-03-15T22:21:23.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[62156]: osdmap e168: 8 total, 8 up, 8 in 2021-03-15T22:21:23.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: cluster 2021-03-15T22:21:21.773158+0000 osd.1 (osd.1 2021-03-15T22:21:23.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: ) 37 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:21:23.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: cluster 2021-03-15T22:21:21.929716+0000 osd.6 (osd.6) 21 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:21:23.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: cluster 2021-03-15T22:21:23.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: 2021-03-15T22:21 2021-03-15T22:21:23.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: :22.029833+0000 osd.5 2021-03-15T22:21:23.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: (osd.5) 39 : cluster 2021-03-15T22:21:23.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: [DBG] 6.d scrub ok 2021-03-15T22:21:23.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:21:23.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: -03-15T22:21: 2021-03-15T22:21:23.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: 22.324832+0000 osd.4 (osd 2021-03-15T22:21:23.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: .4) 43 : cluster [DBG] 2021-03-15T22:21:23.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: 3.14 scrub ok 2021-03-15T22:21:23.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: audit 2021 2021-03-15T22:21:23.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: -03-15T22:21: 2021-03-15T22:21:23.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: 22.596132+0000 mgr.y (mgr 2021-03-15T22:21:23.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: .14144) 420 : audit [DBG] 2021-03-15T22:21:23.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: from='client.14877 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:23.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:21:23.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: :21:22.794334+0000 osd.0 (osd. 2021-03-15T22:21:23.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: 0) 38 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:21:23.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:21:23.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: -03-15T22:21: 2021-03-15T22:21:23.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: 22.994503+0000 mgr.y (mgr. 2021-03-15T22:21:23.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: 14144) 421 : cluster [DBG] 2021-03-15T22:21:23.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: pgmap v378: 142 pgs: 1 peering, 141 active+clean; 164 MiB data, 503 MiB used, 715 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.5 MiB/s wr, 19 op/s 2021-03-15T22:21:23.325 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:21:23.325 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: -15T22:21:22.995215+0000 2021-03-15T22:21:23.325 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: mon.a (mon.0) 751 : audit [INF] 2021-03-15T22:21:23.325 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-03-15T22:21:23.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: audit 2021-03-15T22:21:23.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: 2021-03-15T22:21:23.020469 2021-03-15T22:21:23.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: +0000 mon.a (mon.0) 752 : audit [INF] 2021-03-15T22:21:23.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "25"}]': finished 2021-03-15T22:21:23.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: cluster 2021-03-15T22:21: 2021-03-15T22:21:23.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: 23.020628+0000 mon.a (mon.0) 753 2021-03-15T22:21:23.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:23 smithi076 conmon[56514]: : cluster [DBG] osdmap e168: 8 total, 8 up, 8 in 2021-03-15T22:21:23.331 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2021-03-15T22:21:23.331 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight-by-utilization 110 3.0 2021-03-15T22:21:23.372 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:23.395 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:23.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:23.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:23.409 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:21:23.498 INFO:tasks.rados.rados.0.smithi076.stdout:394: finishing copy_from to smithi07694878-7 2021-03-15T22:21:23.498 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:21:23.498 INFO:tasks.rados.rados.0.smithi076.stdout:394: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:21:23.499 INFO:tasks.rados.rados.0.smithi076.stdout:401: finishing rollback tid 0 to smithi07694878-39 2021-03-15T22:21:23.499 INFO:tasks.rados.rados.0.smithi076.stdout:401: finishing rollback tid 1 to smithi07694878-39 2021-03-15T22:21:23.499 INFO:tasks.rados.rados.0.smithi076.stdout:401: finishing rollback tid 2 to smithi07694878-39 2021-03-15T22:21:23.499 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 148 (ObjNum 116 snap 24 seq_num 116) dirty exists 2021-03-15T22:21:23.499 INFO:tasks.rados.rados.0.smithi076.stdout:405: finishing rollback tid 0 to smithi07694878-24 2021-03-15T22:21:23.500 INFO:tasks.rados.rados.0.smithi076.stdout:405: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:21:23.500 INFO:tasks.rados.rados.0.smithi076.stdout:394: done (5 left) 2021-03-15T22:21:23.500 INFO:tasks.rados.rados.0.smithi076.stdout:399: done (4 left) 2021-03-15T22:21:23.500 INFO:tasks.rados.rados.0.smithi076.stdout:400: done (3 left) 2021-03-15T22:21:23.501 INFO:tasks.rados.rados.0.smithi076.stdout:401: done (2 left) 2021-03-15T22:21:23.501 INFO:tasks.rados.rados.0.smithi076.stdout:402: done (1 left) 2021-03-15T22:21:23.501 INFO:tasks.rados.rados.0.smithi076.stdout:406: read oid 23 snap -1 2021-03-15T22:21:23.501 INFO:tasks.rados.rados.0.smithi076.stdout:406: expect (ObjNum 124 snap 24 seq_num 124) 2021-03-15T22:21:23.501 INFO:tasks.rados.rados.0.smithi076.stdout:407: delete oid 22 current snap is 37 2021-03-15T22:21:23.502 INFO:tasks.rados.rados.0.smithi076.stdout:405: finishing rollback tid 2 to smithi07694878-24 2021-03-15T22:21:23.502 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 35 (ObjNum 106 snap 22 seq_num 106) dirty exists 2021-03-15T22:21:23.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:23.503 INFO:tasks.rados.rados.0.smithi076.stdout:405: done (2 left) 2021-03-15T22:21:23.504 INFO:tasks.rados.rados.0.smithi076.stdout:407: done (1 left) 2021-03-15T22:21:23.504 INFO:tasks.rados.rados.0.smithi076.stdout:408: write (excl) oid 49 current snap is 37 2021-03-15T22:21:23.504 INFO:tasks.rados.rados.0.smithi076.stdout:408: seq_num 174 ranges {524418=453092,1732455=588379,2779130=145288} 2021-03-15T22:21:23.505 INFO:tasks.rados.rados.0.smithi076.stdout:408: writing smithi07694878-49 from 524418 to 977510 tid 1 2021-03-15T22:21:23.512 INFO:tasks.rados.rados.0.smithi076.stdout:408: writing smithi07694878-49 from 1732455 to 2320834 tid 2 2021-03-15T22:21:23.515 INFO:tasks.rados.rados.0.smithi076.stdout:408: writing smithi07694878-49 from 2779130 to 2924418 tid 3 2021-03-15T22:21:23.516 INFO:tasks.rados.rados.0.smithi076.stdout:409: delete oid 20 current snap is 37 2021-03-15T22:21:23.516 INFO:tasks.rados.rados.0.smithi076.stdout:406: expect (ObjNum 124 snap 24 seq_num 124) 2021-03-15T22:21:23.527 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:23.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:23.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:23.592 INFO:tasks.rados.rados.0.smithi076.stdout:408: finishing write tid 1 to smithi07694878-49 2021-03-15T22:21:23.592 INFO:tasks.rados.rados.0.smithi076.stdout:408: finishing write tid 2 to smithi07694878-49 2021-03-15T22:21:23.593 INFO:tasks.rados.rados.0.smithi076.stdout:406: done (2 left) 2021-03-15T22:21:23.593 INFO:tasks.rados.rados.0.smithi076.stdout:409: done (1 left) 2021-03-15T22:21:23.593 INFO:tasks.rados.rados.0.smithi076.stdout:410: rollback oid 6 current snap is 37 2021-03-15T22:21:23.593 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 35 2021-03-15T22:21:23.594 INFO:tasks.rados.rados.0.smithi076.stdout:408: finishing write tid 3 to smithi07694878-49 2021-03-15T22:21:23.594 INFO:tasks.rados.rados.0.smithi076.stdout:408: finishing write tid 4 to smithi07694878-49 2021-03-15T22:21:23.594 INFO:tasks.rados.rados.0.smithi076.stdout:408: finishing write tid 5 to smithi07694878-49 2021-03-15T22:21:23.594 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 211 (ObjNum 174 snap 37 seq_num 174) dirty exists 2021-03-15T22:21:23.594 INFO:tasks.rados.rados.0.smithi076.stdout:408: left oid 49 (ObjNum 174 snap 37 seq_num 174) 2021-03-15T22:21:23.595 INFO:tasks.rados.rados.0.smithi076.stdout:408: done (1 left) 2021-03-15T22:21:23.595 INFO:tasks.rados.rados.0.smithi076.stdout:411: write oid 5 current snap is 37 2021-03-15T22:21:23.595 INFO:tasks.rados.rados.0.smithi076.stdout:411: seq_num 176 ranges {558032=719943,1849419=592048} 2021-03-15T22:21:23.599 INFO:tasks.rados.rados.0.smithi076.stdout:411: writing smithi07694878-5 from 558032 to 1277975 tid 1 2021-03-15T22:21:23.607 INFO:tasks.rados.rados.0.smithi076.stdout:411: writing smithi07694878-5 from 1849419 to 2441467 tid 2 2021-03-15T22:21:23.607 INFO:tasks.rados.rados.0.smithi076.stdout:410: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:21:23.607 INFO:tasks.rados.rados.0.smithi076.stdout:410: finishing rollback tid 2 to smithi07694878-6 2021-03-15T22:21:23.608 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 161 (ObjNum 63 snap 6 seq_num 63) dirty exists 2021-03-15T22:21:23.608 INFO:tasks.rados.rados.0.smithi076.stdout:411: finishing write tid 1 to smithi07694878-5 2021-03-15T22:21:23.608 INFO:tasks.rados.rados.0.smithi076.stdout:410: done (1 left) 2021-03-15T22:21:23.608 INFO:tasks.rados.rados.0.smithi076.stdout:412: read oid 50 snap -1 2021-03-15T22:21:23.609 INFO:tasks.rados.rados.0.smithi076.stdout:412: expect (ObjNum 170 snap 37 seq_num 170) 2021-03-15T22:21:23.609 INFO:tasks.rados.rados.0.smithi076.stdout:413: read oid 46 snap -1 2021-03-15T22:21:23.609 INFO:tasks.rados.rados.0.smithi076.stdout:413: expect (ObjNum 143 snap 28 seq_num 143) 2021-03-15T22:21:23.609 INFO:tasks.rados.rados.0.smithi076.stdout:414: read oid 32 snap -1 2021-03-15T22:21:23.609 INFO:tasks.rados.rados.0.smithi076.stdout:414: expect (ObjNum 108 snap 23 seq_num 108) 2021-03-15T22:21:23.610 INFO:tasks.rados.rados.0.smithi076.stdout:415: read oid 43 snap -1 2021-03-15T22:21:23.610 INFO:tasks.rados.rados.0.smithi076.stdout:415: expect deleted 2021-03-15T22:21:23.610 INFO:tasks.rados.rados.0.smithi076.stdout:416: copy_from oid 19 from oid 29 current snap is 37 2021-03-15T22:21:23.610 INFO:tasks.rados.rados.0.smithi076.stdout:417: copy_from oid 16 from oid 2 current snap is 37 2021-03-15T22:21:23.610 INFO:tasks.rados.rados.0.smithi076.stdout:418: snap_remove snap 35 2021-03-15T22:21:23.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:23.653 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:23.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:23.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:23.694 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:23.719 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:23.755 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:23.779 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:23.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:23.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:23.810 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:21:23.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:23.904 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:23.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:23.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:24.007 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:24.033 INFO:tasks.rados.rados.0.smithi076.stdout:418: done (7 left) 2021-03-15T22:21:24.034 INFO:tasks.rados.rados.0.smithi076.stdout:419: write (excl) oid 4 current snap is 37 2021-03-15T22:21:24.034 INFO:tasks.rados.rados.0.smithi076.stdout:419: seq_num 177 ranges {574839=795192,1907901=452059} 2021-03-15T22:21:24.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:24.035 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:24.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:24.043 INFO:tasks.rados.rados.0.smithi076.stdout:419: writing smithi07694878-4 from 574839 to 1370031 tid 1 2021-03-15T22:21:24.050 INFO:tasks.rados.rados.0.smithi076.stdout:419: writing smithi07694878-4 from 1907901 to 2359960 tid 2 2021-03-15T22:21:24.050 INFO:tasks.rados.rados.0.smithi076.stdout:420: snap_create 2021-03-15T22:21:24.050 INFO:tasks.rados.rados.0.smithi076.stdout:417: finishing copy_from to smithi07694878-16 2021-03-15T22:21:24.050 INFO:tasks.rados.rados.0.smithi076.stdout:417: got expected ENOENT (src dne) 2021-03-15T22:21:24.051 INFO:tasks.rados.rados.0.smithi076.stdout:417: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:21:24.051 INFO:tasks.rados.rados.0.smithi076.stdout:411: finishing write tid 2 to smithi07694878-5 2021-03-15T22:21:24.051 INFO:tasks.rados.rados.0.smithi076.stdout:411: finishing write tid 3 to smithi07694878-5 2021-03-15T22:21:24.051 INFO:tasks.rados.rados.0.smithi076.stdout:411: finishing write tid 4 to smithi07694878-5 2021-03-15T22:21:24.051 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 42 (ObjNum 176 snap 37 seq_num 176) dirty exists 2021-03-15T22:21:24.052 INFO:tasks.rados.rados.0.smithi076.stdout:411: left oid 5 (ObjNum 176 snap 37 seq_num 176) 2021-03-15T22:21:24.052 INFO:tasks.rados.rados.0.smithi076.stdout:412: expect (ObjNum 170 snap 37 seq_num 170) 2021-03-15T22:21:24.125 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: cluster 2021-03-15T22:21:24.126 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[56514]: cluster 2021-03-15T22:21:22.710903+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:21:24.126 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[56514]: cluster 2021-03-15T22:21:22.904951+0000 osd.6 (osd.6) 22 : cluster [DBG] 3.a scrub ok 2021-03-15T22:21:24.127 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[56514]: cluster 2021-03-15T22:21:23.309320+0000 osd.4 (osd.4) 44 : cluster [DBG] 5.e scrub ok 2021-03-15T22:21:24.127 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[56514]: cluster 2021-03-15T22:21:23.838978+0000 osd.0 (osd.0) 39 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:21:24.127 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[56514]: cluster 2021-03-15T22:21:24.021905+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2021-03-15T22:21:24.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:24.159 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:24.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:24.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:24.172 INFO:tasks.rados.rados.0.smithi076.stdout:413: expect (ObjNum 143 snap 28 seq_num 143) 2021-03-15T22:21:24.230 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:24.261 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:24.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:24.289 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:24.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:24.340 INFO:tasks.rados.rados.0.smithi076.stdout:414: expect (ObjNum 108 snap 23 seq_num 108) 2021-03-15T22:21:24.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:24.414 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:24.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:24.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:24.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:24 smithi125 conmon[56739]: cluster 2021-03-15T22:21:22.710903+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:21:24.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:24 smithi125 conmon[56739]: cluster 2021-03-15T22:21:22.904951+0000 osd.6 (osd.6) 22 : cluster [DBG] 3.a scrub ok 2021-03-15T22:21:24.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:24 smithi125 conmon[56739]: cluster 2021-03-15T22:21:23.309320+0000 osd.4 (osd.4) 44 : cluster [DBG] 5.e scrub ok 2021-03-15T22:21:24.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:24 smithi125 conmon[56739]: cluster 2021-03-15T22:21:23.838978+0000 osd.0 (osd.0) 39 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:21:24.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:24 smithi125 conmon[56739]: cluster 2021-03-15T22:21:24.021905+0000 mon.a (mon.0) 754 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2021-03-15T22:21:24.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: 2021-03-15T22:21:22.710903+0000 osd.7 (osd.7) 42 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:21:24.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: cluster 2021-03-15T22:21:22.904951+0000 osd.6 (osd.6) 22 : cluster [DBG] 3.a scrub ok 2021-03-15T22:21:24.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: cluster 2021-03-15T22:21:23.309320+0000 osd.4 (osd 2021-03-15T22:21:24.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: .4) 44 : cluster [DBG] 5.e scrub ok 2021-03-15T22:21:24.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: cluster 2021-03-15T22:21:24.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: 2021 2021-03-15T22:21:24.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: -03-15T22:21:23 2021-03-15T22:21:24.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: .838978+0000 osd.0 (osd.0) 39 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:21:24.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: cluster 2021-03-15T22:21:24.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: 2021-03-15T22:21:24.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: 2021-03-15T22:21:24.021905+0000 mon.a 2021-03-15T22:21:24.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:24 smithi076 conmon[62156]: (mon.0) 754 : cluster [DBG] osdmap e169: 8 total, 8 up, 8 in 2021-03-15T22:21:24.516 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:24.517 INFO:tasks.rados.rados.0.smithi076.stdout:416: finishing copy_from to smithi07694878-19 2021-03-15T22:21:24.518 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 234 (ObjNum 132 snap 25 seq_num 132) dirty exists 2021-03-15T22:21:24.518 INFO:tasks.rados.rados.0.smithi076.stdout:416: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:21:24.518 INFO:tasks.rados.rados.0.smithi076.stdout:419: finishing write tid 1 to smithi07694878-4 2021-03-15T22:21:24.519 INFO:tasks.rados.rados.0.smithi076.stdout:419: finishing write tid 2 to smithi07694878-4 2021-03-15T22:21:24.519 INFO:tasks.rados.rados.0.smithi076.stdout:419: finishing write tid 3 to smithi07694878-4 2021-03-15T22:21:24.519 INFO:tasks.rados.rados.0.smithi076.stdout:419: finishing write tid 4 to smithi07694878-4 2021-03-15T22:21:24.519 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 115 (ObjNum 177 snap 37 seq_num 177) dirty exists 2021-03-15T22:21:24.519 INFO:tasks.rados.rados.0.smithi076.stdout:419: left oid 4 (ObjNum 177 snap 37 seq_num 177) 2021-03-15T22:21:24.542 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:24.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:24.644 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:24.666 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:24.668 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:24.721 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:21:24.721 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:21:24.769 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:24.792 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:24.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:24.793 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:24.894 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:24.917 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:24.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:25.017 INFO:tasks.rados.rados.0.smithi076.stdout:411: done (8 left) 2021-03-15T22:21:25.018 INFO:tasks.rados.rados.0.smithi076.stdout:412: done (7 left) 2021-03-15T22:21:25.018 INFO:tasks.rados.rados.0.smithi076.stdout:413: done (6 left) 2021-03-15T22:21:25.018 INFO:tasks.rados.rados.0.smithi076.stdout:414: done (5 left) 2021-03-15T22:21:25.019 INFO:tasks.rados.rados.0.smithi076.stdout:415: done (4 left) 2021-03-15T22:21:25.019 INFO:tasks.rados.rados.0.smithi076.stdout:416: done (3 left) 2021-03-15T22:21:25.019 INFO:tasks.rados.rados.0.smithi076.stdout:417: done (2 left) 2021-03-15T22:21:25.019 INFO:tasks.rados.rados.0.smithi076.stdout:419: done (1 left) 2021-03-15T22:21:25.020 INFO:tasks.rados.rados.0.smithi076.stdout:420: done (0 left) 2021-03-15T22:21:25.020 INFO:tasks.rados.rados.0.smithi076.stdout:421: write oid 13 current snap is 38 2021-03-15T22:21:25.020 INFO:tasks.rados.rados.0.smithi076.stdout:421: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2021-03-15T22:21:25.021 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:25.024 INFO:tasks.rados.rados.0.smithi076.stdout:421: writing smithi07694878-13 from 591646 to 1062087 tid 1 2021-03-15T22:21:25.030 INFO:tasks.rados.rados.0.smithi076.stdout:421: writing smithi07694878-13 from 1566383 to 2162100 tid 2 2021-03-15T22:21:25.035 INFO:tasks.rados.rados.0.smithi076.stdout:421: writing smithi07694878-13 from 2888822 to 2991646 tid 3 2021-03-15T22:21:25.035 INFO:tasks.rados.rados.0.smithi076.stdout:422: snap_remove snap 33 2021-03-15T22:21:25.047 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:25.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:25.049 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:25.151 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:25.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:25.174 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:25.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:25.275 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:25.297 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:25.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:25.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:25.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:25.401 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:25 smithi125 conmon[56739]: cluster 2021-03-15T22:21:24.493632+0000 osd.2 (osd.2) 33 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:25.401 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:25 smithi125 conmon[56739]: cluster 2021-03-15T22:21:24. 2021-03-15T22:21:25.401 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:25 smithi125 conmon[56739]: 493785+0000 osd.2 (osd.2) 34 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:25.402 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:25 smithi125 conmon[56739]: cluster 2021-03-15T22:21:24.995195+0000 mgr.y (mgr.14144) 422 : cluster [DBG] pgmap v381: 142 pgs: 2 clean+premerge+peered, 140 active+clean; 161 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 8.9 MiB/s rd, 1.7 MiB/s wr, 17 op/s 2021-03-15T22:21:25.402 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:25 smithi125 conmon[56739]: cluster 2021-03-15T22:21:25.015135+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2021-03-15T22:21:25.403 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-15T22:21:25.421 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:25.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:25.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:25.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:25 smithi076 conmon[56514]: cluster 2021-03-15T22:21:24.493632+0000 osd.2 (osd. 2021-03-15T22:21:25.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:25 smithi076 conmon[56514]: 2) 33 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:25.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:25 smithi076 conmon[56514]: cluster 2021-03-15T22:21:24.493785+0000 osd.2 (osd.2) 34 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:25.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:25 smithi076 conmon[56514]: cluster 2021-03-15T22:21:24.995195+0000 mgr.y (mgr.14144) 422 : cluster [DBG] pgmap v381: 142 pgs: 2 clean+premerge+peered, 140 active+clean; 161 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 8.9 MiB/s rd, 1.7 MiB/s wr, 17 op/s 2021-03-15T22:21:25.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:25 smithi076 conmon[56514]: cluster 2021-03-15T22:21:25.015135+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2021-03-15T22:21:25.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:25 smithi076 conmon[62156]: cluster 2021-03-15T22:21:24.493632+0000 osd.2 (osd 2021-03-15T22:21:25.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:25 smithi076 conmon[62156]: .2) 33 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:25.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:25 smithi076 conmon[62156]: cluster 2021-03-15T22:21:24.493785+0000 osd.2 (osd.2) 34 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:25.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:25 smithi076 conmon[62156]: cluster 2021-03-15T22:21:24.995195+0000 mgr.y (mgr.14144) 422 : cluster [DBG] pgmap v381: 142 pgs: 2 clean+premerge+peered, 140 active+clean; 161 MiB data, 488 MiB used, 715 GiB / 715 GiB avail; 8.9 MiB/s rd, 1.7 MiB/s wr, 17 op/s 2021-03-15T22:21:25.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:25 smithi076 conmon[62156]: cluster 2021-03-15T22:21:25.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:25 smithi076 conmon[62156]: 2021-03-15T22:21:25.015135+0000 mon.a (mon.0) 755 : cluster [DBG] osdmap e170: 8 total, 8 up, 8 in 2021-03-15T22:21:25.524 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:25.549 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:25.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:25.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:25.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:25.675 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:25.675 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:25.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:25.677 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:25.778 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:25.800 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:25.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:25.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:25.903 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:25.925 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:25.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:25.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:26.027 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:26.032 INFO:tasks.rados.rados.0.smithi076.stdout:422: done (1 left) 2021-03-15T22:21:26.032 INFO:tasks.rados.rados.0.smithi076.stdout:423: write oid 25 current snap is 38 2021-03-15T22:21:26.033 INFO:tasks.rados.rados.0.smithi076.stdout:423: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2021-03-15T22:21:26.036 INFO:tasks.rados.rados.0.smithi076.stdout:423: writing smithi07694878-25 from 608453 to 1154143 tid 1 2021-03-15T22:21:26.041 INFO:tasks.rados.rados.0.smithi076.stdout:423: writing smithi07694878-25 from 1624865 to 2080593 tid 2 2021-03-15T22:21:26.046 INFO:tasks.rados.rados.0.smithi076.stdout:423: writing smithi07694878-25 from 2632598 to 3008453 tid 3 2021-03-15T22:21:26.047 INFO:tasks.rados.rados.0.smithi076.stdout:424: snap_remove snap 37 2021-03-15T22:21:26.052 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:26.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:26.054 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:26.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:26.181 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:26.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:26.183 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:26.284 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:26.307 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:26.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:26.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:26.410 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:26.424 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:26 smithi076 conmon[56514]: cluster 2021-03-15T22:21:24.789521+0000 osd.1 (osd.1) 38 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:21:26.424 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:26 smithi076 conmon[56514]: cluster 2021-03-15T22:21:24.823784+0000 osd.6 (osd.6) 23 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:21:26.425 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:26 smithi076 conmon[56514]: cluster 2021-03-15T22:21:25.785976+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:21:26.425 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:26 smithi076 conmon[56514]: cluster 2021-03-15T22:21:26.027118+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2021-03-15T22:21:26.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:26 smithi076 conmon[62156]: cluster 2021-03-15T22:21:24.789521+0000 2021-03-15T22:21:26.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:26 smithi076 conmon[62156]: osd.1 (osd.1) 38 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:21:26.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:26 smithi076 conmon[62156]: cluster 2021-03-15T22:21:24.823784+0000 osd.6 (osd.6) 23 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:21:26.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:26 smithi076 conmon[62156]: cluster 2021-03-15T22:21:25.785976+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:21:26.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:26 smithi076 conmon[62156]: cluster 2021-03-15T22:21:26.027118+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2021-03-15T22:21:26.433 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:26.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:26.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:26.536 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:26 smithi125 conmon[56739]: cluster 2021-03-15T22:21:24.789521+0000 osd.1 (osd.1) 38 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:21:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:26 smithi125 conmon[56739]: cluster 2021-03-15T22:21:24.823784+0000 osd.6 (osd.6) 23 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:21:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:26 smithi125 conmon[56739]: cluster 2021-03-15T22:21:25.785976+0000 osd.0 (osd.0) 40 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:21:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:26 smithi125 conmon[56739]: cluster 2021-03-15T22:21:26.027118+0000 mon.a (mon.0) 756 : cluster [DBG] osdmap e171: 8 total, 8 up, 8 in 2021-03-15T22:21:26.559 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:26.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:26.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:26.661 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:26.684 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:26.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:26.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:26.789 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:26.812 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:26.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:26.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:26.914 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:26.937 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:26.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:26.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:27.036 INFO:teuthology.orchestra.run.smithi076.stdout:moved 37 / 423 (8.74704%) 2021-03-15T22:21:27.036 INFO:teuthology.orchestra.run.smithi076.stdout:avg 52.875 2021-03-15T22:21:27.036 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 5.57758 -> 11.4612 (expected baseline 6.80188) 2021-03-15T22:21:27.037 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.2 with 44 -> 50 pgs (0.832151 -> 0.945626 * mean) 2021-03-15T22:21:27.037 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.3 with 62 -> 25 pgs (1.17258 -> 0.472813 * mean) 2021-03-15T22:21:27.037 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:21:27.037 INFO:teuthology.orchestra.run.smithi076.stdout:oload 110 2021-03-15T22:21:27.037 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 3 2021-03-15T22:21:27.038 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:21:27.038 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0007 2021-03-15T22:21:27.038 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0007 2021-03-15T22:21:27.038 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 weight 1.0000 -> 0.5226 2021-03-15T22:21:27.039 INFO:tasks.rados.rados.0.smithi076.stdout:424: done (2 left) 2021-03-15T22:21:27.039 INFO:tasks.rados.rados.0.smithi076.stdout:425: snap_remove snap 34 2021-03-15T22:21:27.040 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:21:27.040 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:27.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:27.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:27.169 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:27.191 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:27.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:27.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:27.223 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:21:27.239 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:21:27.246 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:21:27.246 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:21:27.256 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:21:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: cluster 2021-03-15T22:21:25.834717+0000 osd.1 (osd.1) 39 : cluster [DBG] 7.b scrub ok 2021-03-15T22:21:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: audit 2021-03-15T22:21:26.338767+0000 mon.a (mon.0) 757 : audit [INF] from='client.? 172.21.15.76:0/3903331860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: audit 2021-03-15T22:21:26.690035+0000 mgr.y (mgr.14144) 423 : audit [DBG] from='client.24607 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:27.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: audit 2021-03-15T22:21:26.690761+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'34251'}"}]: dispatch 2021-03-15T22:21:27.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: cluster 2021-03-15T22:21:26.995955+0000 mgr.y (mgr.14144) 424 : cluster [DBG] pgmap v384: 141 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 clean+premerge+peered, 136 active+clean; 150 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 21 MiB/s rd, 4.1 MiB/s wr, 34 op/s 2021-03-15T22:21:27.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: audit 2021-03-15T22:21:26.996938+0000 2021-03-15T22:21:27.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: mon.a (mon.0) 759 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-03-15T22:21:27.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: cluster 2021-03-15T22:21:27.022182+0000 mon.a (mon.0) 760 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:21:27.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: cluster 2021-03-15T22:21:27.022206+0000 mon.a (mon.0) 761 : cluster [INF] Cluster is now healthy 2021-03-15T22:21:27.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: audit 2021-03-15T22:21:27.026541+0000 mon.a (mon.0) 762 : audit [INF] from='client.? 172.21.15.76:0/3903331860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:27.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: audit 2021-03-15T22:21:27.026581+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'34251'}"}]': finished 2021-03-15T22:21:27.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: audit 2021-03-15T22:21:27.026609+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2021-03-15T22:21:27.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[56514]: cluster 2021-03-15T22:21:27.026634+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2021-03-15T22:21:27.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: cluster 2021-03-15T22:21:25.834717+0000 osd.1 ( 2021-03-15T22:21:27.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: osd.1) 39 : cluster [DBG] 7.b scrub ok 2021-03-15T22:21:27.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: audit 2021-03-15T22:21:26.338767+0000 mon.a (mon.0) 757 : audit [INF] from='client.? 172.21.15.76:0/3903331860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:27.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: audit 2021-03-15T22:21:26.690035+0000 mgr.y (mgr.14144) 423 : audit [DBG] from='client.24607 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:27.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: audit 2021-03-15T22:21:26.690761+0000 mon.a (mon.0) 2021-03-15T22:21:27.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: 758 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'34251'}"}]: dispatch 2021-03-15T22:21:27.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: cluster 2021-03-15T22:21:26.995955+0000 mgr.y ( 2021-03-15T22:21:27.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: mgr.14144) 424 : cluster [DBG] pgmap v384: 141 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 clean+premerge+peered, 136 active+clean; 150 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 21 MiB/s rd, 4.1 MiB/s wr, 34 op/s 2021-03-15T22:21:27.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: audit 2021-03-15T22:21:27.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: 2021-03-15T22:21:26.996938+0000 mon.a (mon.0) 759 2021-03-15T22:21:27.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-03-15T22:21:27.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: cluster 2021-03-15T22:21:27.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: 2021-03-15T22:21:27.022182+0000 2021-03-15T22:21:27.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: mon.a (mon.0) 760 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:21:27.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:21:27.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: 21:27.022206+0000 mon.a (mon.0) 761 : cluster [INF] Cluster is now healthy 2021-03-15T22:21:27.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:21:27.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: 21:27.026541+0000 mon.a (mon.0) 762 : 2021-03-15T22:21:27.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/3903331860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:27.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:21:27.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: T22:21:27.026581+0000 mon.a (mon. 2021-03-15T22:21:27.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: 0) 763 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'34251'}"}]': finished 2021-03-15T22:21:27.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: audit 2021-03-15T22:21:27.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: 2021-03-15T22:21:27.026609+0000 2021-03-15T22:21:27.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: mon.a (mon.0) 764 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2021-03-15T22:21:27.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:27.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: 03-15T22:21:27.026634+0000 mon.a (mon. 2021-03-15T22:21:27.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:27 smithi076 conmon[62156]: 0) 765 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2021-03-15T22:21:27.277 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:21:27.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: cluster 2021-03-15T22:21:25.834717+0000 osd.1 (osd.1) 39 : cluster [DBG] 7.b scrub ok 2021-03-15T22:21:27.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: audit 2021-03-15T22:21:27.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: 2021-03-15T22:21:26.338767+0000 mon.a (mon.0) 757 : audit [INF] from='client.? 172.21.15.76:0/3903331860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:27.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: audit 2021-03-15T22:21:26.690035+0000 mgr.y (mgr.14144) 423 : audit [DBG] from='client.24607 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:27.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: audit 2021-03-15T22:21:26.690761+0000 mon.a (mon.0) 758 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'34251'}"}]: dispatch 2021-03-15T22:21:27.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: cluster 2021-03-15T22:21:26.995955+0000 mgr.y (mgr.14144) 424 : cluster [DBG] pgmap v384: 141 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 1 clean+premerge+peered, 136 active+clean; 150 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 21 MiB/s rd, 4.1 MiB/s wr, 34 op/s 2021-03-15T22:21:27.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: audit 2021-03-15T22:21:26.996938+0000 mon.a (mon.0) 759 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-03-15T22:21:27.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: cluster 2021-03-15T22:21:27.022182+0000 mon.a (mon.0) 760 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:21:27.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: cluster 2021-03-15T22:21:27.022206+0000 mon.a (mon.0) 761 : cluster [INF] Cluster is now healthy 2021-03-15T22:21:27.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: audit 2021-03-15T22:21:27.026541+0000 mon.a (mon.0) 762 : audit [INF] from='client.? 172.21.15.76:0/3903331860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:27.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: audit 2021-03-15T22:21:27.026581+0000 mon.a (mon.0) 763 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'34251'}"}]': finished 2021-03-15T22:21:27.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: audit 2021-03-15T22:21:27.026609+0000 mon.a (mon.0) 764 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "24"}]': finished 2021-03-15T22:21:27.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:27 smithi125 conmon[56739]: cluster 2021-03-15T22:21:27.026634+0000 mon.a (mon.0) 765 : cluster [DBG] osdmap e172: 8 total, 8 up, 8 in 2021-03-15T22:21:27.293 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:21:27.294 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:27.312 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:21:27.316 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:27.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:27.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:27.330 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:21:27.418 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:27.439 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:27.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:27.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:27.471 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:21:27.471 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:21:27.471 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:21:27.471 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:21:27.472 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:21:27.472 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:21:27.472 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:21:27.472 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:21:27.473 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:21:27.541 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:27.563 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:27.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:27.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:27.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:27.688 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:27.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:27.689 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:27.790 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:27.814 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:27.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:27.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:27.917 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-15T22:21:27.917 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:27.939 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:27.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:27.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:28.039 INFO:tasks.rados.rados.0.smithi076.stdout:425: done (2 left) 2021-03-15T22:21:28.039 INFO:tasks.rados.rados.0.smithi076.stdout:426: snap_create 2021-03-15T22:21:28.039 INFO:tasks.rados.rados.0.smithi076.stdout:421: finishing write tid 1 to smithi07694878-13 2021-03-15T22:21:28.039 INFO:tasks.rados.rados.0.smithi076.stdout:421: finishing write tid 2 to smithi07694878-13 2021-03-15T22:21:28.040 INFO:tasks.rados.rados.0.smithi076.stdout:421: finishing write tid 3 to smithi07694878-13 2021-03-15T22:21:28.040 INFO:tasks.rados.rados.0.smithi076.stdout:421: finishing write tid 4 to smithi07694878-13 2021-03-15T22:21:28.040 INFO:tasks.rados.rados.0.smithi076.stdout:421: finishing write tid 5 to smithi07694878-13 2021-03-15T22:21:28.040 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 120 (ObjNum 178 snap 38 seq_num 178) dirty exists 2021-03-15T22:21:28.041 INFO:tasks.rados.rados.0.smithi076.stdout:421: left oid 13 (ObjNum 178 snap 38 seq_num 178) 2021-03-15T22:21:28.041 INFO:tasks.rados.rados.0.smithi076.stdout:423: finishing write tid 1 to smithi07694878-25 2021-03-15T22:21:28.041 INFO:tasks.rados.rados.0.smithi076.stdout:423: finishing write tid 2 to smithi07694878-25 2021-03-15T22:21:28.041 INFO:tasks.rados.rados.0.smithi076.stdout:423: finishing write tid 3 to smithi07694878-25 2021-03-15T22:21:28.041 INFO:tasks.rados.rados.0.smithi076.stdout:423: finishing write tid 4 to smithi07694878-25 2021-03-15T22:21:28.042 INFO:tasks.rados.rados.0.smithi076.stdout:423: finishing write tid 5 to smithi07694878-25 2021-03-15T22:21:28.042 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 47 (ObjNum 179 snap 38 seq_num 179) dirty exists 2021-03-15T22:21:28.042 INFO:tasks.rados.rados.0.smithi076.stdout:423: left oid 25 (ObjNum 179 snap 38 seq_num 179) 2021-03-15T22:21:28.043 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:28.071 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:28.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:28.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:28.140 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:28 smithi125 conmon[56739]: cluster 2021-03-15T22:21:26.679032+0000 2021-03-15T22:21:28.140 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:28 smithi125 conmon[56739]: osd.7 (osd.7) 43 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:21:28.140 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:28 smithi125 conmon[56739]: audit 2021-03-15T22:21:27.203888+0000 mon.a (mon.0) 766 : audit [DBG] from='client.? 172.21.15.76:0/3048865790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:28.141 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:28 smithi125 conmon[56739]: cluster 2021-03-15T22:21:27.231258+0000 osd.4 (osd.4) 45 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:21:28.141 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:28 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:21:28.141 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:28 smithi125 conmon[56739]: T22:21:28.030099+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2021-03-15T22:21:28.174 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:28.195 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:28.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:28.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:28.229 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:28.298 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:28.321 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:28.323 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:28.424 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:28.447 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:28.447 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[56514]: cluster 2021-03-15T22:21:26.679032+0000 osd.7 (osd. 2021-03-15T22:21:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[56514]: 7) 43 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:21:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[56514]: audit 2021-03-15T22:21:27.203888+0000 mon.a (mon.0) 766 : audit [DBG] from='client.? 172.21.15.76:0/3048865790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[56514]: cluster 2021-03-15T22:21:27.231258+0000 osd.4 (osd.4) 45 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:21:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[56514]: cluster 2021-03-15T22:21:28.030099+0000 mon.a (mon.0) 767 : cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2021-03-15T22:21:28.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[62156]: cluster 2021-03-15T22:21:26.679032+0000 osd.7 (osd.7 2021-03-15T22:21:28.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[62156]: ) 43 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:21:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[62156]: audit 2021-03-15T22:21:27.203888+0000 mon.a (mon.0 2021-03-15T22:21:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[62156]: ) 766 : audit [DBG] from='client.? 172.21.15.76:0/3048865790' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[62156]: cluster 2021-03-15T22:21:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[62156]: 2021-03-15T22:21:27.231258+0000 osd.4 (osd.4) 45 2021-03-15T22:21:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[62156]: : cluster [DBG] 3.3 scrub ok 2021-03-15T22:21:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[62156]: cluster 2021-03-15T22:21 2021-03-15T22:21:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[62156]: :28.030099+0000 mon.a (mon.0) 767 : 2021-03-15T22:21:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:28 smithi076 conmon[62156]: cluster [DBG] osdmap e173: 8 total, 8 up, 8 in 2021-03-15T22:21:28.548 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:28.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:28.571 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:28.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:28.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:28.697 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:28.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:28.699 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:28.767 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2021-03-15T22:21:28.767 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd test-reweight-by-utilization 1000 1.0 2021-03-15T22:21:28.800 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:28.822 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:28.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:28.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:28.925 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:28.946 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:28.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:28.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:29.039 INFO:tasks.rados.rados.0.smithi076.stdout:421: done (2 left) 2021-03-15T22:21:29.039 INFO:tasks.rados.rados.0.smithi076.stdout:423: done (1 left) 2021-03-15T22:21:29.039 INFO:tasks.rados.rados.0.smithi076.stdout:426: done (0 left) 2021-03-15T22:21:29.039 INFO:tasks.rados.rados.0.smithi076.stdout:427: write (excl) oid 36 current snap is 39 2021-03-15T22:21:29.040 INFO:tasks.rados.rados.0.smithi076.stdout:427: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2021-03-15T22:21:29.040 INFO:tasks.rados.rados.0.smithi076.stdout:427: writing smithi07694878-36 from 625260 to 1246199 tid 1 2021-03-15T22:21:29.045 INFO:tasks.rados.rados.0.smithi076.stdout:427: writing smithi07694878-36 from 1683347 to 2282733 tid 2 2021-03-15T22:21:29.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:29.051 INFO:tasks.rados.rados.0.smithi076.stdout:427: writing smithi07694878-36 from 2943668 to 3025260 tid 3 2021-03-15T22:21:29.052 INFO:tasks.rados.rados.0.smithi076.stdout:428: write (excl) oid 47 current snap is 39 2021-03-15T22:21:29.052 INFO:tasks.rados.rados.0.smithi076.stdout:428: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2021-03-15T22:21:29.058 INFO:tasks.rados.rados.0.smithi076.stdout:428: writing smithi07694878-47 from 642067 to 1338255 tid 1 2021-03-15T22:21:29.065 INFO:tasks.rados.rados.0.smithi076.stdout:428: writing smithi07694878-47 from 2025476 to 2484873 tid 2 2021-03-15T22:21:29.069 INFO:tasks.rados.rados.0.smithi076.stdout:428: writing smithi07694878-47 from 2971091 to 3042067 tid 3 2021-03-15T22:21:29.069 INFO:tasks.rados.rados.0.smithi076.stdout:427: finishing write tid 1 to smithi07694878-36 2021-03-15T22:21:29.070 INFO:tasks.rados.rados.0.smithi076.stdout:427: finishing write tid 2 to smithi07694878-36 2021-03-15T22:21:29.070 INFO:tasks.rados.rados.0.smithi076.stdout:427: finishing write tid 3 to smithi07694878-36 2021-03-15T22:21:29.070 INFO:tasks.rados.rados.0.smithi076.stdout:427: finishing write tid 4 to smithi07694878-36 2021-03-15T22:21:29.070 INFO:tasks.rados.rados.0.smithi076.stdout:427: finishing write tid 5 to smithi07694878-36 2021-03-15T22:21:29.071 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 134 (ObjNum 180 snap 39 seq_num 180) dirty exists 2021-03-15T22:21:29.071 INFO:tasks.rados.rados.0.smithi076.stdout:427: left oid 36 (ObjNum 180 snap 39 seq_num 180) 2021-03-15T22:21:29.071 INFO:tasks.rados.rados.0.smithi076.stdout:428: finishing write tid 1 to smithi07694878-47 2021-03-15T22:21:29.071 INFO:tasks.rados.rados.0.smithi076.stdout:427: done (1 left) 2021-03-15T22:21:29.072 INFO:tasks.rados.rados.0.smithi076.stdout:429: snap_create 2021-03-15T22:21:29.072 INFO:tasks.rados.rados.0.smithi076.stdout:428: finishing write tid 2 to smithi07694878-47 2021-03-15T22:21:29.073 INFO:tasks.rados.rados.0.smithi076.stdout:428: finishing write tid 3 to smithi07694878-47 2021-03-15T22:21:29.073 INFO:tasks.rados.rados.0.smithi076.stdout:428: finishing write tid 4 to smithi07694878-47 2021-03-15T22:21:29.074 INFO:tasks.rados.rados.0.smithi076.stdout:428: finishing write tid 5 to smithi07694878-47 2021-03-15T22:21:29.074 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 97 (ObjNum 181 snap 39 seq_num 181) dirty exists 2021-03-15T22:21:29.074 INFO:tasks.rados.rados.0.smithi076.stdout:428: left oid 47 (ObjNum 181 snap 39 seq_num 181) 2021-03-15T22:21:29.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:29.078 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:29.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:29.120 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:29.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:29.202 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:29.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:29.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:29.305 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:29.327 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:29.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:29.329 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:29.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:29.453 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:29.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:29.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:29.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:29 smithi125 conmon[56739]: cluster 2021-03-15T22:21:27.654561+0000 osd.7 (osd.7) 44 : cluster [DBG] 7.5 deep-scrub ok 2021-03-15T22:21:29.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:29 smithi125 conmon[56739]: cluster 2021-03-15T22:21:28.996562+0000 mgr.y (mgr.14144) 425 : cluster [DBG] pgmap v387: 141 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 10 peering, 1 clean+premerge+peered, 126 active+clean; 140 MiB data, 446 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2021-03-15T22:21:29.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:29 smithi125 conmon[56739]: cluster 2021-03-15T22:21:29.029006+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2021-03-15T22:21:29.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[56514]: cluster 2021-03-15T22:21:27.654561+0000 osd.7 ( 2021-03-15T22:21:29.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[56514]: osd.7) 44 : cluster [DBG] 7.5 deep-scrub ok 2021-03-15T22:21:29.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[56514]: cluster 2021-03-15T22:21:28.996562+0000 mgr.y (mgr.14144) 425 : cluster [DBG] pgmap v387: 141 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 10 peering, 1 clean+premerge+peered, 126 active+clean; 140 MiB data, 446 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2021-03-15T22:21:29.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[56514]: cluster 2021-03-15T22:21:29.029006+0000 mon.a (mon.0) 768 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2021-03-15T22:21:29.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[62156]: cluster 2021-03-15T22:21:27. 2021-03-15T22:21:29.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[62156]: 654561+0000 osd.7 (osd.7) 44 : cluster [DBG] 7.5 deep-scrub ok 2021-03-15T22:21:29.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:21:29.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[62156]: 15T22:21:28.996562+0000 mgr.y (mgr.14144) 425 : cluster [DBG] 2021-03-15T22:21:29.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[62156]: pgmap v387: 141 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 10 peering, 1 clean+premerge+peered, 126 active+clean; 140 MiB data, 446 MiB used, 715 GiB / 715 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 37 op/s 2021-03-15T22:21:29.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[62156]: cluster 2021-03-15T22:21:29.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[62156]: 2021-03-15T22:21:29.029006+0000 mon.a (mon.0 2021-03-15T22:21:29.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:29 smithi076 conmon[62156]: ) 768 : cluster [DBG] osdmap e174: 8 total, 8 up, 8 in 2021-03-15T22:21:29.556 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:29.578 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:29.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:29.580 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:29.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:29.703 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:29.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:29.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:29.716 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:21:29.716 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:21:29.716 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:21:29.716 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:21:29.717 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:21:29.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:21:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:21:29.806 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:29.827 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:29.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:29.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:29.929 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:29.952 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:29.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:29.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:30.025 INFO:tasks.rados.rados.0.smithi076.stdout:428: done (1 left) 2021-03-15T22:21:30.025 INFO:tasks.rados.rados.0.smithi076.stdout:429: done (0 left) 2021-03-15T22:21:30.026 INFO:tasks.rados.rados.0.smithi076.stdout:430: write oid 45 current snap is 40 2021-03-15T22:21:30.026 INFO:tasks.rados.rados.0.smithi076.stdout:430: seq_num 182 ranges {658874=771437,2083958=603055} 2021-03-15T22:21:30.029 INFO:tasks.rados.rados.0.smithi076.stdout:430: writing smithi07694878-45 from 658874 to 1430311 tid 1 2021-03-15T22:21:30.037 INFO:tasks.rados.rados.0.smithi076.stdout:430: writing smithi07694878-45 from 2083958 to 2687013 tid 2 2021-03-15T22:21:30.038 INFO:tasks.rados.rados.0.smithi076.stdout:431: read oid 2 snap -1 2021-03-15T22:21:30.038 INFO:tasks.rados.rados.0.smithi076.stdout:431: expect deleted 2021-03-15T22:21:30.038 INFO:tasks.rados.rados.0.smithi076.stdout:432: copy_from oid 5 from oid 18 current snap is 40 2021-03-15T22:21:30.038 INFO:tasks.rados.rados.0.smithi076.stdout:430: finishing write tid 1 to smithi07694878-45 2021-03-15T22:21:30.038 INFO:tasks.rados.rados.0.smithi076.stdout:433: write oid 47 current snap is 40 2021-03-15T22:21:30.039 INFO:tasks.rados.rados.0.smithi076.stdout:433: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2021-03-15T22:21:30.044 INFO:tasks.rados.rados.0.smithi076.stdout:433: writing smithi07694878-47 from 675681 to 1238720 tid 1 2021-03-15T22:21:30.054 INFO:tasks.rados.rados.0.smithi076.stdout:433: writing smithi07694878-47 from 1858793 to 2605506 tid 2 2021-03-15T22:21:30.054 INFO:tasks.rados.rados.0.smithi076.stdout:433: writing smithi07694878-47 from 3025937 to 3075681 tid 3 2021-03-15T22:21:30.055 INFO:tasks.rados.rados.0.smithi076.stdout:434: write (excl) oid 40 current snap is 40 2021-03-15T22:21:30.055 INFO:tasks.rados.rados.0.smithi076.stdout:434: seq_num 184 ranges {692488=638288,1917275=606724} 2021-03-15T22:21:30.055 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:30.061 INFO:tasks.rados.rados.0.smithi076.stdout:434: writing smithi07694878-40 from 692488 to 1330776 tid 1 2021-03-15T22:21:30.069 INFO:tasks.rados.rados.0.smithi076.stdout:434: writing smithi07694878-40 from 1917275 to 2523999 tid 2 2021-03-15T22:21:30.069 INFO:tasks.rados.rados.0.smithi076.stdout:435: copy_from oid 16 from oid 34 current snap is 40 2021-03-15T22:21:30.070 INFO:tasks.rados.rados.0.smithi076.stdout:432: finishing copy_from to smithi07694878-5 2021-03-15T22:21:30.070 INFO:tasks.rados.rados.0.smithi076.stdout:432: got expected ENOENT (src dne) 2021-03-15T22:21:30.070 INFO:tasks.rados.rados.0.smithi076.stdout:432: finishing copy_from racing read to smithi07694878-5 2021-03-15T22:21:30.070 INFO:tasks.rados.rados.0.smithi076.stdout:430: finishing write tid 2 to smithi07694878-45 2021-03-15T22:21:30.071 INFO:tasks.rados.rados.0.smithi076.stdout:430: finishing write tid 3 to smithi07694878-45 2021-03-15T22:21:30.071 INFO:tasks.rados.rados.0.smithi076.stdout:430: finishing write tid 4 to smithi07694878-45 2021-03-15T22:21:30.071 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 164 (ObjNum 182 snap 40 seq_num 182) dirty exists 2021-03-15T22:21:30.071 INFO:tasks.rados.rados.0.smithi076.stdout:430: left oid 45 (ObjNum 182 snap 40 seq_num 182) 2021-03-15T22:21:30.071 INFO:tasks.rados.rados.0.smithi076.stdout:433: finishing write tid 1 to smithi07694878-47 2021-03-15T22:21:30.072 INFO:tasks.rados.rados.0.smithi076.stdout:430: done (5 left) 2021-03-15T22:21:30.072 INFO:tasks.rados.rados.0.smithi076.stdout:431: done (4 left) 2021-03-15T22:21:30.072 INFO:tasks.rados.rados.0.smithi076.stdout:432: done (3 left) 2021-03-15T22:21:30.073 INFO:tasks.rados.rados.0.smithi076.stdout:436: write oid 33 current snap is 40 2021-03-15T22:21:30.073 INFO:tasks.rados.rados.0.smithi076.stdout:433: finishing write tid 2 to smithi07694878-47 2021-03-15T22:21:30.073 INFO:tasks.rados.rados.0.smithi076.stdout:433: finishing write tid 3 to smithi07694878-47 2021-03-15T22:21:30.073 INFO:tasks.rados.rados.0.smithi076.stdout:433: finishing write tid 4 to smithi07694878-47 2021-03-15T22:21:30.073 INFO:tasks.rados.rados.0.smithi076.stdout:436: seq_num 185 ranges {709295=713537,1859404=750382} 2021-03-15T22:21:30.076 INFO:tasks.rados.rados.0.smithi076.stdout:436: writing smithi07694878-33 from 709295 to 1422832 tid 1 2021-03-15T22:21:30.078 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:30.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:30.080 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:30.083 INFO:tasks.rados.rados.0.smithi076.stdout:436: writing smithi07694878-33 from 1859404 to 2609786 tid 2 2021-03-15T22:21:30.083 INFO:tasks.rados.rados.0.smithi076.stdout:437: write oid 13 current snap is 40 2021-03-15T22:21:30.083 INFO:tasks.rados.rados.0.smithi076.stdout:433: finishing write tid 5 to smithi07694878-47 2021-03-15T22:21:30.084 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 102 (ObjNum 183 snap 40 seq_num 183) dirty exists 2021-03-15T22:21:30.084 INFO:tasks.rados.rados.0.smithi076.stdout:433: left oid 47 (ObjNum 183 snap 40 seq_num 183) 2021-03-15T22:21:30.084 INFO:tasks.rados.rados.0.smithi076.stdout:434: finishing write tid 1 to smithi07694878-40 2021-03-15T22:21:30.085 INFO:tasks.rados.rados.0.smithi076.stdout:437: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2021-03-15T22:21:30.090 INFO:tasks.rados.rados.0.smithi076.stdout:437: writing smithi07694878-13 from 726102 to 1514888 tid 1 2021-03-15T22:21:30.096 INFO:tasks.rados.rados.0.smithi076.stdout:437: writing smithi07694878-13 from 1917886 to 2528279 tid 2 2021-03-15T22:21:30.100 INFO:tasks.rados.rados.0.smithi076.stdout:437: writing smithi07694878-13 from 3108206 to 3126102 tid 3 2021-03-15T22:21:30.100 INFO:tasks.rados.rados.0.smithi076.stdout:433: done (4 left) 2021-03-15T22:21:30.101 INFO:tasks.rados.rados.0.smithi076.stdout:438: copy_from oid 4 from oid 29 current snap is 40 2021-03-15T22:21:30.101 INFO:tasks.rados.rados.0.smithi076.stdout:434: finishing write tid 2 to smithi07694878-40 2021-03-15T22:21:30.101 INFO:tasks.rados.rados.0.smithi076.stdout:434: finishing write tid 3 to smithi07694878-40 2021-03-15T22:21:30.101 INFO:tasks.rados.rados.0.smithi076.stdout:434: finishing write tid 4 to smithi07694878-40 2021-03-15T22:21:30.102 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 265 (ObjNum 184 snap 40 seq_num 184) dirty exists 2021-03-15T22:21:30.102 INFO:tasks.rados.rados.0.smithi076.stdout:434: left oid 40 (ObjNum 184 snap 40 seq_num 184) 2021-03-15T22:21:30.102 INFO:tasks.rados.rados.0.smithi076.stdout:436: finishing write tid 1 to smithi07694878-33 2021-03-15T22:21:30.102 INFO:tasks.rados.rados.0.smithi076.stdout:434: done (4 left) 2021-03-15T22:21:30.103 INFO:tasks.rados.rados.0.smithi076.stdout:439: snap_remove snap 38 2021-03-15T22:21:30.181 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:30.203 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:30.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:30.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:30.306 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:30.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:30.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:30.408 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:30 smithi125 conmon[56739]: cluster 2021-03-15T22:21:29.136211+0000 mon.a (mon.0) 769 2021-03-15T22:21:30.408 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:30 smithi125 conmon[56739]: : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2021-03-15T22:21:30.408 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:30 smithi125 conmon[56739]: audit 2021-03-15T22:21:29.483449+0000 mgr.y (mgr.14144) 426 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:30.408 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:30 smithi125 conmon[56739]: cluster 2021-03-15T22:21:30.017244+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2021-03-15T22:21:30.410 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-15T22:21:30.431 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:30.455 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:30.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:30.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:30.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[56514]: cluster 2021-03-15T22:21:29.136211+0000 mon.a (mon.0) 769 : 2021-03-15T22:21:30.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[56514]: cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2021-03-15T22:21:30.508 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[56514]: audit 2021-03-15T22:21:29.483449+0000 mgr.y (mgr.14144) 426 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:30.508 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[56514]: cluster 2021-03-15T22:21:30.017244+0000 mon.a (mon.0) 770 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2021-03-15T22:21:30.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[62156]: cluster 2021-03-15T22:21:29.136211+0000 2021-03-15T22:21:30.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[62156]: mon.a (mon.0) 769 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2021-03-15T22:21:30.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[62156]: audit 2021 2021-03-15T22:21:30.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[62156]: -03-15T22:21:29.483449+0000 mgr.y (mgr 2021-03-15T22:21:30.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[62156]: .14144) 426 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:30.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[62156]: cluster 2021-03-15T22:21:30.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:21:30.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[62156]: 21:30.017244+0000 mon.a (mon.0 2021-03-15T22:21:30.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:30 smithi076 conmon[62156]: ) 770 : cluster [DBG] osdmap e175: 8 total, 8 up, 8 in 2021-03-15T22:21:30.558 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:30.580 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:30.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:30.582 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:30.683 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:30.686 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:21:30.706 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:30.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:30.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:30.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:21:30.738 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:21:30.809 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:30.832 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:30.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:30.834 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:30.901 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 420 (0%) 2021-03-15T22:21:30.902 INFO:teuthology.orchestra.run.smithi076.stdout:avg 52.5 2021-03-15T22:21:30.902 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 11.3137 -> 11.3137 (expected baseline 6.77772) 2021-03-15T22:21:30.902 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 25 -> 25 pgs (0.47619 -> 0.47619 * mean) 2021-03-15T22:21:30.902 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 65 -> 65 pgs (1.2381 -> 1.2381 * mean) 2021-03-15T22:21:30.903 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:21:30.903 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:21:30.903 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 1 2021-03-15T22:21:30.903 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:21:30.904 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0006 2021-03-15T22:21:30.904 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0062 2021-03-15T22:21:30.905 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:21:30.935 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:30.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:30.958 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:30.959 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:31.022 INFO:tasks.rados.rados.0.smithi076.stdout:439: done (4 left) 2021-03-15T22:21:31.022 INFO:tasks.rados.rados.0.smithi076.stdout:440: snap_create 2021-03-15T22:21:31.022 INFO:tasks.rados.rados.0.smithi076.stdout:436: finishing write tid 2 to smithi07694878-33 2021-03-15T22:21:31.023 INFO:tasks.rados.rados.0.smithi076.stdout:436: finishing write tid 3 to smithi07694878-33 2021-03-15T22:21:31.023 INFO:tasks.rados.rados.0.smithi076.stdout:436: finishing write tid 4 to smithi07694878-33 2021-03-15T22:21:31.023 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 126 (ObjNum 185 snap 40 seq_num 185) dirty exists 2021-03-15T22:21:31.023 INFO:tasks.rados.rados.0.smithi076.stdout:436: left oid 33 (ObjNum 185 snap 40 seq_num 185) 2021-03-15T22:21:31.024 INFO:tasks.rados.rados.0.smithi076.stdout:437: finishing write tid 1 to smithi07694878-13 2021-03-15T22:21:31.024 INFO:tasks.rados.rados.0.smithi076.stdout:437: finishing write tid 2 to smithi07694878-13 2021-03-15T22:21:31.024 INFO:tasks.rados.rados.0.smithi076.stdout:437: finishing write tid 3 to smithi07694878-13 2021-03-15T22:21:31.024 INFO:tasks.rados.rados.0.smithi076.stdout:437: finishing write tid 4 to smithi07694878-13 2021-03-15T22:21:31.024 INFO:tasks.rados.rados.0.smithi076.stdout:437: finishing write tid 5 to smithi07694878-13 2021-03-15T22:21:31.025 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 139 (ObjNum 186 snap 40 seq_num 186) dirty exists 2021-03-15T22:21:31.025 INFO:tasks.rados.rados.0.smithi076.stdout:437: left oid 13 (ObjNum 186 snap 40 seq_num 186) 2021-03-15T22:21:31.025 INFO:tasks.rados.rados.0.smithi076.stdout:438: finishing copy_from to smithi07694878-4 2021-03-15T22:21:31.025 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 234 (ObjNum 132 snap 25 seq_num 132) dirty exists 2021-03-15T22:21:31.026 INFO:tasks.rados.rados.0.smithi076.stdout:438: finishing copy_from racing read to smithi07694878-4 2021-03-15T22:21:31.046 INFO:tasks.rados.rados.0.smithi076.stdout:435: finishing copy_from to smithi07694878-16 2021-03-15T22:21:31.047 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 101 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:21:31.047 INFO:tasks.rados.rados.0.smithi076.stdout:435: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:21:31.060 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:31.081 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:31.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:31.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:31.184 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:31.209 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:31.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:31.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:31.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:31.335 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:31.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:31.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:31.438 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:31.459 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:31.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:31.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:31.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[56514]: cluster 2021-03-15T22:21:30.473567+0000 osd.2 (osd.2) 35 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:21:31.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[56514]: cluster 2021-03-15T22:21:30.812508+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.b scrub ok 2021-03-15T22:21:31.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[56514]: audit 2021-03-15T22:21:30.899908+0000 mgr.y (mgr.14144) 427 : audit [DBG] from='client.14901 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:31.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[56514]: audit 2021-03-15T22:21:30.997775+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-03-15T22:21:31.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[56514]: audit 2021-03-15T22:21:31.017028+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2021-03-15T22:21:31.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[56514]: cluster 2021-03-15T22:21:31.017101+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2021-03-15T22:21:31.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[62156]: cluster 2021-03-15T22:21:30.473567+0000 osd.2 (osd.2) 35 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:21:31.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[62156]: cluster 2021-03-15T22:21:30.812508+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.b scrub ok 2021-03-15T22:21:31.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[62156]: audit 2021-03-15T22:21:30.899908+0000 mgr.y (mgr.14144) 427 : audit [DBG] from='client.14901 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:31.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[62156]: audit 2021-03-15T22:21:30.997775+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-03-15T22:21:31.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:21:31.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[62156]: 22:21:31.017028+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2021-03-15T22:21:31.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:31 smithi076 conmon[62156]: cluster 2021-03-15T22:21:31.017101+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2021-03-15T22:21:31.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:31 smithi125 conmon[56739]: cluster 2021-03-15T22:21:30.473567+0000 osd.2 (osd.2) 35 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:21:31.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:31 smithi125 conmon[56739]: cluster 2021-03-15T22:21:30.812508+0000 osd.0 (osd.0) 41 : cluster [DBG] 4.b scrub ok 2021-03-15T22:21:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:31 smithi125 conmon[56739]: audit 2021-03-15T22:21:30.899908+0000 mgr.y (mgr.14144) 427 : audit [DBG] from='client.14901 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:31 smithi125 conmon[56739]: audit 2021-03-15T22:21:30.997775+0000 mon.a (mon.0) 771 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-03-15T22:21:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:31 smithi125 conmon[56739]: audit 2021-03-15T22:21:31.017028+0000 mon.a (mon.0) 772 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "23"}]': finished 2021-03-15T22:21:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:31 smithi125 conmon[56739]: cluster 2021-03-15T22:21:31.017101+0000 mon.a (mon.0) 773 : cluster [DBG] osdmap e176: 8 total, 8 up, 8 in 2021-03-15T22:21:31.562 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:31.586 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:31.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:31.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:31.645 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:31.667 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2021-03-15T22:21:31.668 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd test-reweight-by-utilization 1000 0.05 2021-03-15T22:21:31.688 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:31.696 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:31.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:31.713 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:31.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:31.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:31.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:31.839 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:31.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:31.940 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:31.962 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:31.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:31.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:32.024 INFO:tasks.rados.rados.0.smithi076.stdout:435: done (4 left) 2021-03-15T22:21:32.024 INFO:tasks.rados.rados.0.smithi076.stdout:436: done (3 left) 2021-03-15T22:21:32.025 INFO:tasks.rados.rados.0.smithi076.stdout:437: done (2 left) 2021-03-15T22:21:32.025 INFO:tasks.rados.rados.0.smithi076.stdout:438: done (1 left) 2021-03-15T22:21:32.025 INFO:tasks.rados.rados.0.smithi076.stdout:440: done (0 left) 2021-03-15T22:21:32.025 INFO:tasks.rados.rados.0.smithi076.stdout:441: read oid 20 snap -1 2021-03-15T22:21:32.026 INFO:tasks.rados.rados.0.smithi076.stdout:441: expect deleted 2021-03-15T22:21:32.026 INFO:tasks.rados.rados.0.smithi076.stdout:442: rollback oid 43 current snap is 41 2021-03-15T22:21:32.026 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 43 to 39 2021-03-15T22:21:32.026 INFO:tasks.rados.rados.0.smithi076.stdout:443: delete oid 42 current snap is 41 2021-03-15T22:21:32.027 INFO:tasks.rados.rados.0.smithi076.stdout:442: finishing rollback tid 1 to smithi07694878-43 2021-03-15T22:21:32.027 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 0 (ObjNum 261650672 snap 21896 seq_num 260042768) dirty dne 2021-03-15T22:21:32.029 INFO:tasks.rados.rados.0.smithi076.stdout:441: done (2 left) 2021-03-15T22:21:32.030 INFO:tasks.rados.rados.0.smithi076.stdout:442: done (1 left) 2021-03-15T22:21:32.030 INFO:tasks.rados.rados.0.smithi076.stdout:443: done (0 left) 2021-03-15T22:21:32.030 INFO:tasks.rados.rados.0.smithi076.stdout:444: snap_remove snap 39 2021-03-15T22:21:32.065 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:32.087 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:32.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:32.089 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:32.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:32.200 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:21:32.200 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:21:32.201 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:21:32.201 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:21:32.201 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.201 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:3 3.13 3:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e57)", 2021-03-15T22:21:32.202 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:17:53.975558+0000", 2021-03-15T22:21:32.202 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 218.22169386100001, 2021-03-15T22:21:32.202 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.31057361700000002, 2021-03-15T22:21:32.202 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.203 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-15T22:21:32.203 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:32.203 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:21:32.203 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:21:32.203 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3 2021-03-15T22:21:32.204 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.204 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.204 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.204 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.204 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975558+0000", 2021-03-15T22:21:32.205 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.205 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.205 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.205 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.205 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975558+0000", 2021-03-15T22:21:32.206 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3149999999999999e-06 2021-03-15T22:21:32.206 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.206 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.206 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.207 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975560+0000", 2021-03-15T22:21:32.207 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7670000000000001e-06 2021-03-15T22:21:32.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.207 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.208 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975564+0000", 2021-03-15T22:21:32.208 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0937e-05 2021-03-15T22:21:32.208 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.208 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.209 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975575+0000", 2021-03-15T22:21:32.209 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1800000000000002e-06 2021-03-15T22:21:32.209 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.209 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.212 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975584+0000", 2021-03-15T22:21:32.212 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4805000000000001e-05 2021-03-15T22:21:32.212 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.213 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.213 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975639+0000", 2021-03-15T22:21:32.213 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2069999999999995e-06 2021-03-15T22:21:32.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.214 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.214 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:21:32.214 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975647+0000", 2021-03-15T22:21:32.214 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.31038343600000001 2021-03-15T22:21:32.215 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.215 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:54.286030+0000", 2021-03-15T22:21:32.215 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010097 2021-03-15T22:21:32.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.217 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.217 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:54.286131+0000", 2021-03-15T22:21:32.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.31057361700000002 2021-03-15T22:21:32.217 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.217 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.218 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.218 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.218 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.218 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:260 7.6 e104/91 7:63693c38:::smithi07694878-22:head v 104'32, mlcod=100'28)", 2021-03-15T22:21:32.218 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.195529+0000", 2021-03-15T22:21:32.219 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 73.001722607000005, 2021-03-15T22:21:32.219 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066965699999999998, 2021-03-15T22:21:32.219 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.219 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.220 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.220 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.220 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.220 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.195529+0000", 2021-03-15T22:21:32.220 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.221 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.221 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.221 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.221 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.195529+0000", 2021-03-15T22:21:32.221 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0026948739999999999 2021-03-15T22:21:32.222 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.222 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.222 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.222 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198224+0000", 2021-03-15T22:21:32.223 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1399999999999999e-06 2021-03-15T22:21:32.223 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.223 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.223 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.223 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198229+0000", 2021-03-15T22:21:32.224 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2414e-05 2021-03-15T22:21:32.224 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.224 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.224 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.224 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198241+0000", 2021-03-15T22:21:32.225 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0659999999999998e-06 2021-03-15T22:21:32.225 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.225 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.225 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.226 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198248+0000", 2021-03-15T22:21:32.226 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6831e-05 2021-03-15T22:21:32.226 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.226 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.227 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.227 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198285+0000", 2021-03-15T22:21:32.227 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7632e-05 2021-03-15T22:21:32.227 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.227 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.228 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.228 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198303+0000", 2021-03-15T22:21:32.228 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038786379999999998 2021-03-15T22:21:32.228 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.228 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.229 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.229 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.202182+0000", 2021-03-15T22:21:32.229 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3974999999999998e-05 2021-03-15T22:21:32.229 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.230 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.230 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.230 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.202226+0000", 2021-03-15T22:21:32.230 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066965699999999998 2021-03-15T22:21:32.230 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.231 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.231 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.231 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.231 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.231 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:265 7.6 e104/91 7:6283974b:::smithi07694878-30:head v 104'34, mlcod=100'28)", 2021-03-15T22:21:32.232 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.198708+0000", 2021-03-15T22:21:32.232 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 72.998543789999999, 2021-03-15T22:21:32.232 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050519500000000004, 2021-03-15T22:21:32.232 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.232 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.233 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.233 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.233 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.233 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198708+0000", 2021-03-15T22:21:32.234 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.234 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.234 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.234 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.234 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198708+0000", 2021-03-15T22:21:32.235 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00075425600000000002 2021-03-15T22:21:32.235 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.235 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.235 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.236 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.199462+0000", 2021-03-15T22:21:32.236 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0170000000000001e-06 2021-03-15T22:21:32.236 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.236 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.236 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.237 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.199465+0000", 2021-03-15T22:21:32.237 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3107e-05 2021-03-15T22:21:32.237 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.237 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.237 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.238 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.199478+0000", 2021-03-15T22:21:32.238 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6520000000000003e-06 2021-03-15T22:21:32.238 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.238 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.238 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.239 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.199484+0000", 2021-03-15T22:21:32.239 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013140490000000001 2021-03-15T22:21:32.239 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.239 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.240 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.240 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.200798+0000", 2021-03-15T22:21:32.240 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6419999999999999e-05 2021-03-15T22:21:32.240 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.241 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.200814+0000", 2021-03-15T22:21:32.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028916110000000001 2021-03-15T22:21:32.241 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.241 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.242 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.242 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.203706+0000", 2021-03-15T22:21:32.242 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3838000000000002e-05 2021-03-15T22:21:32.242 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.242 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.243 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.243 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.203760+0000", 2021-03-15T22:21:32.243 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050519500000000004 2021-03-15T22:21:32.243 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.244 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.244 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.244 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.244 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.244 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:321 7.6 e104/91 7:6283974b:::smithi07694878-30:head v 104'37, mlcod=104'35)", 2021-03-15T22:21:32.245 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.448113+0000", 2021-03-15T22:21:32.245 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 72.749138156000001, 2021-03-15T22:21:32.245 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046484220000000001, 2021-03-15T22:21:32.245 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.245 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.246 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.246 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.246 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.246 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.448113+0000", 2021-03-15T22:21:32.247 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.247 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.247 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.247 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.247 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.448113+0000", 2021-03-15T22:21:32.248 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023519470000000001 2021-03-15T22:21:32.248 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.248 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.248 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.450465+0000", 2021-03-15T22:21:32.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6929999999999998e-06 2021-03-15T22:21:32.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.249 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.250 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.450470+0000", 2021-03-15T22:21:32.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9466e-05 2021-03-15T22:21:32.250 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.250 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.250 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.251 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.450490+0000", 2021-03-15T22:21:32.251 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4710000000000007e-06 2021-03-15T22:21:32.251 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.251 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.252 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.252 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.450498+0000", 2021-03-15T22:21:32.252 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3998999999999994e-05 2021-03-15T22:21:32.252 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.253 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.253 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.450572+0000", 2021-03-15T22:21:32.253 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4173e-05 2021-03-15T22:21:32.253 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.253 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.254 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.450596+0000", 2021-03-15T22:21:32.254 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021226589999999998 2021-03-15T22:21:32.254 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.452719+0000", 2021-03-15T22:21:32.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3013999999999998e-05 2021-03-15T22:21:32.255 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.256 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.452762+0000", 2021-03-15T22:21:32.256 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046484220000000001 2021-03-15T22:21:32.257 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.257 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.257 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.257 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.258 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.258 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:323 7.8 e104/91 7:1d74a1c3:::smithi07694878-44:head v 104'23, mlcod=103'20)", 2021-03-15T22:21:32.258 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.452200+0000", 2021-03-15T22:21:32.258 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 72.745051497000006, 2021-03-15T22:21:32.258 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0064996999999999997, 2021-03-15T22:21:32.259 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.259 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.259 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.452200+0000", 2021-03-15T22:21:32.260 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.261 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.261 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.452200+0000", 2021-03-15T22:21:32.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039340390000000003 2021-03-15T22:21:32.261 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.262 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.262 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.262 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456134+0000", 2021-03-15T22:21:32.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.729e-06 2021-03-15T22:21:32.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.263 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.263 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.263 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456139+0000", 2021-03-15T22:21:32.263 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6049000000000002e-05 2021-03-15T22:21:32.264 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.264 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.264 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.264 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456155+0000", 2021-03-15T22:21:32.265 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6189999999999996e-06 2021-03-15T22:21:32.265 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.265 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.265 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.265 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456162+0000", 2021-03-15T22:21:32.266 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8286999999999999e-05 2021-03-15T22:21:32.266 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.266 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.266 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.267 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456220+0000", 2021-03-15T22:21:32.267 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8587e-05 2021-03-15T22:21:32.267 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.267 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.267 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.268 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456238+0000", 2021-03-15T22:21:32.268 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024070160000000001 2021-03-15T22:21:32.268 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.268 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.268 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.269 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.458645+0000", 2021-03-15T22:21:32.269 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4373999999999998e-05 2021-03-15T22:21:32.269 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.269 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.269 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.270 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.458700+0000", 2021-03-15T22:21:32.270 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0064996999999999997 2021-03-15T22:21:32.270 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.270 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.271 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.271 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:344 7.6 e106/91 7:6283974b:::smithi07694878-30:head v 106'39, mlcod=104'36)", 2021-03-15T22:21:32.271 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:21.021046+0000", 2021-03-15T22:21:32.272 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 71.176205167999996, 2021-03-15T22:21:32.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0068779380000000001, 2021-03-15T22:21:32.272 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.272 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.272 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.021046+0000", 2021-03-15T22:21:32.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.274 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.021046+0000", 2021-03-15T22:21:32.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029106470000000001 2021-03-15T22:21:32.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.023957+0000", 2021-03-15T22:21:32.276 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1580000000000002e-06 2021-03-15T22:21:32.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.276 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.023963+0000", 2021-03-15T22:21:32.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0822000000000002e-05 2021-03-15T22:21:32.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.277 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.023984+0000", 2021-03-15T22:21:32.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1955e-05 2021-03-15T22:21:32.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.278 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.278 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.279 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.023996+0000", 2021-03-15T22:21:32.279 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0494000000000001e-05 2021-03-15T22:21:32.279 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.279 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.279 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.280 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.024056+0000", 2021-03-15T22:21:32.281 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7806000000000001e-05 2021-03-15T22:21:32.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.281 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.282 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.282 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.024084+0000", 2021-03-15T22:21:32.282 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037723679999999999 2021-03-15T22:21:32.282 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.282 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.283 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.283 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.027857+0000", 2021-03-15T22:21:32.283 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7688000000000007e-05 2021-03-15T22:21:32.283 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.284 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.284 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.284 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.027924+0000", 2021-03-15T22:21:32.284 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0068779380000000001 2021-03-15T22:21:32.284 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.285 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.285 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.285 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.285 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.285 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:346 7.6 e106/91 7:608078b0:::smithi07694878-12:head v 106'41, mlcod=104'36)", 2021-03-15T22:21:32.286 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:21.025337+0000", 2021-03-15T22:21:32.286 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 71.171914443000006, 2021-03-15T22:21:32.286 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042614209999999996, 2021-03-15T22:21:32.286 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.286 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.287 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.287 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.287 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.287 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.025337+0000", 2021-03-15T22:21:32.288 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.288 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.288 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.288 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.289 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.025337+0000", 2021-03-15T22:21:32.289 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00053107600000000001 2021-03-15T22:21:32.289 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.289 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.289 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.290 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.025868+0000", 2021-03-15T22:21:32.290 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2549999999999997e-06 2021-03-15T22:21:32.290 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.290 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.290 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.291 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.025873+0000", 2021-03-15T22:21:32.291 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5904000000000001e-05 2021-03-15T22:21:32.291 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.291 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.291 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.292 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.025899+0000", 2021-03-15T22:21:32.292 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2110000000000004e-06 2021-03-15T22:21:32.292 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.292 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.293 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.293 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.025909+0000", 2021-03-15T22:21:32.293 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048556100000000002 2021-03-15T22:21:32.293 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.293 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.294 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.294 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.026394+0000", 2021-03-15T22:21:32.294 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2433999999999999e-05 2021-03-15T22:21:32.294 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.294 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.295 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.295 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.026417+0000", 2021-03-15T22:21:32.295 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003151278 2021-03-15T22:21:32.295 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.295 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.296 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.296 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.029568+0000", 2021-03-15T22:21:32.296 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0701999999999998e-05 2021-03-15T22:21:32.296 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.296 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.297 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.297 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.029599+0000", 2021-03-15T22:21:32.297 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042614209999999996 2021-03-15T22:21:32.297 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.298 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.298 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.298 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.298 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.298 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:368 7.5 7:a9545d62:::smithi07694878-1:head [write 726069~772507] snapc a=[a] ondisk+write+known_if_redirected e108)", 2021-03-15T22:21:32.299 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.113968+0000", 2021-03-15T22:21:32.299 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 69.083283078999997, 2021-03-15T22:21:32.299 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078723230000000005, 2021-03-15T22:21:32.299 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.299 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.300 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:32.300 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:32.300 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:32.300 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 368 2021-03-15T22:21:32.300 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.301 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.301 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.301 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.301 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.113968+0000", 2021-03-15T22:21:32.302 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.302 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.302 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.302 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.302 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.113968+0000", 2021-03-15T22:21:32.303 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0025141629999999998 2021-03-15T22:21:32.303 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.303 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.303 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.303 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116483+0000", 2021-03-15T22:21:32.304 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3490000000000002e-06 2021-03-15T22:21:32.304 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.304 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.304 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.305 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116487+0000", 2021-03-15T22:21:32.305 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6874000000000001e-05 2021-03-15T22:21:32.305 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.305 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.306 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116504+0000", 2021-03-15T22:21:32.306 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7606000000000001e-05 2021-03-15T22:21:32.306 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.306 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.306 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.307 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116521+0000", 2021-03-15T22:21:32.307 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3022999999999993e-05 2021-03-15T22:21:32.307 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.307 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.308 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116615+0000", 2021-03-15T22:21:32.308 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013848800000000001 2021-03-15T22:21:32.308 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.308 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.308 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.309 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116753+0000", 2021-03-15T22:21:32.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024625100000000001 2021-03-15T22:21:32.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.309 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:21:32.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116999+0000", 2021-03-15T22:21:32.310 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0016196349999999999 2021-03-15T22:21:32.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.310 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.311 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:32.311 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.118619+0000", 2021-03-15T22:21:32.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0019812649999999998 2021-03-15T22:21:32.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.311 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.312 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:32.312 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.120600+0000", 2021-03-15T22:21:32.313 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011110619999999999 2021-03-15T22:21:32.313 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.313 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.314 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:32.314 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121711+0000", 2021-03-15T22:21:32.314 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6941999999999997e-05 2021-03-15T22:21:32.314 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.315 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.315 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121778+0000", 2021-03-15T22:21:32.315 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2664999999999997e-05 2021-03-15T22:21:32.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.316 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.316 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121841+0000", 2021-03-15T22:21:32.316 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078723230000000005 2021-03-15T22:21:32.316 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.317 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.317 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.317 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.317 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.318 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:369 7.5 7:a9545d62:::smithi07694878-1:head [write 2071117~715676] snapc a=[a] ondisk+write+known_if_redirected e108)", 2021-03-15T22:21:32.318 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.121469+0000", 2021-03-15T22:21:32.318 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 69.075782337999996, 2021-03-15T22:21:32.318 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004600008, 2021-03-15T22:21:32.318 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.319 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.319 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:32.319 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:32.319 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:32.320 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 369 2021-03-15T22:21:32.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.320 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121469+0000", 2021-03-15T22:21:32.321 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.321 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.322 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121469+0000", 2021-03-15T22:21:32.322 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000675576 2021-03-15T22:21:32.322 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.323 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122145+0000", 2021-03-15T22:21:32.323 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6129999999999998e-06 2021-03-15T22:21:32.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.324 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.324 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122147+0000", 2021-03-15T22:21:32.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2361000000000001e-05 2021-03-15T22:21:32.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.325 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.325 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122160+0000", 2021-03-15T22:21:32.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.006e-06 2021-03-15T22:21:32.325 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.326 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.326 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.326 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122165+0000", 2021-03-15T22:21:32.326 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0769000000000002e-05 2021-03-15T22:21:32.326 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.327 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.327 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122206+0000", 2021-03-15T22:21:32.327 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3020000000000003e-05 2021-03-15T22:21:32.327 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.328 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.328 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.328 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122269+0000", 2021-03-15T22:21:32.328 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012716999999999999 2021-03-15T22:21:32.328 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.329 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.329 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:21:32.329 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122396+0000", 2021-03-15T22:21:32.329 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010923630000000001 2021-03-15T22:21:32.330 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.330 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.330 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:32.330 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123488+0000", 2021-03-15T22:21:32.330 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002123434 2021-03-15T22:21:32.331 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.331 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.331 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:32.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.125612+0000", 2021-03-15T22:21:32.332 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00038163200000000002 2021-03-15T22:21:32.332 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.332 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:32.332 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.125993+0000", 2021-03-15T22:21:32.333 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5639000000000001e-05 2021-03-15T22:21:32.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.333 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.333 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.126039+0000", 2021-03-15T22:21:32.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0425000000000001e-05 2021-03-15T22:21:32.334 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.334 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.335 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.335 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.126069+0000", 2021-03-15T22:21:32.336 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004600008 2021-03-15T22:21:32.336 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.336 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.336 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.337 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:371 7.5 7:a9545d62:::smithi07694878-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e108)", 2021-03-15T22:21:32.337 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.122203+0000", 2021-03-15T22:21:32.337 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 69.075048510000002, 2021-03-15T22:21:32.338 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041800309999999999, 2021-03-15T22:21:32.338 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.338 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:21:32.338 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:32.339 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:32.339 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:32.339 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 371 2021-03-15T22:21:32.339 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.340 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.340 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.340 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122203+0000", 2021-03-15T22:21:32.340 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.341 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.341 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.341 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.341 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122203+0000", 2021-03-15T22:21:32.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.057e-06 2021-03-15T22:21:32.342 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.342 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.342 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.342 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122204+0000", 2021-03-15T22:21:32.343 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3179999999999999e-06 2021-03-15T22:21:32.343 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.343 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.343 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122205+0000", 2021-03-15T22:21:32.344 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.437e-06 2021-03-15T22:21:32.344 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.344 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.344 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.345 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122210+0000", 2021-03-15T22:21:32.345 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6259999999999999e-06 2021-03-15T22:21:32.345 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.345 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.345 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.346 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122213+0000", 2021-03-15T22:21:32.346 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001310819 2021-03-15T22:21:32.346 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.347 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.347 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123523+0000", 2021-03-15T22:21:32.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7159999999999998e-05 2021-03-15T22:21:32.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:32.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123620+0000", 2021-03-15T22:21:32.348 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0026345140000000001 2021-03-15T22:21:32.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.349 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.126255+0000", 2021-03-15T22:21:32.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3170000000000001e-05 2021-03-15T22:21:32.350 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.126308+0000", 2021-03-15T22:21:32.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4930000000000003e-05 2021-03-15T22:21:32.351 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.351 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.352 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.352 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.126383+0000", 2021-03-15T22:21:32.352 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041800309999999999 2021-03-15T22:21:32.352 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.352 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.353 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.353 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.353 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.353 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:513 7.5 7:ac84e656:::smithi07694878-31:head [read 0~3226911,checksum] snapc 0=[] ondisk+read+known_if_redirected e119)", 2021-03-15T22:21:32.353 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:34.115828+0000", 2021-03-15T22:21:32.354 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 58.081423479000001, 2021-03-15T22:21:32.354 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041075640000000002, 2021-03-15T22:21:32.354 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.354 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:21:32.355 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:32.355 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:32.355 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:32.355 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 513 2021-03-15T22:21:32.355 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.356 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.356 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:34.115828+0000", 2021-03-15T22:21:32.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.357 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.357 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.357 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.358 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:34.115828+0000", 2021-03-15T22:21:32.358 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1150000000000001e-06 2021-03-15T22:21:32.358 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.358 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:34.115830+0000", 2021-03-15T22:21:32.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7069999999999999e-06 2021-03-15T22:21:32.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.359 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.360 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.360 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:34.115832+0000", 2021-03-15T22:21:32.360 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6669999999999996e-06 2021-03-15T22:21:32.360 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.360 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.361 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.361 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:34.115840+0000", 2021-03-15T22:21:32.361 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4109999999999998e-06 2021-03-15T22:21:32.361 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.362 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.362 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.362 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:34.115844+0000", 2021-03-15T22:21:32.362 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029848769999999999 2021-03-15T22:21:32.362 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.363 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.363 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:34.118829+0000", 2021-03-15T22:21:32.363 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000119008 2021-03-15T22:21:32.364 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.364 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.364 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.364 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:34.118948+0000", 2021-03-15T22:21:32.364 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00098777900000000004 2021-03-15T22:21:32.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.365 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.365 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:34.119936+0000", 2021-03-15T22:21:32.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041075640000000002 2021-03-15T22:21:32.366 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.366 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.366 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.366 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.367 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.367 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:531 7.8 e122/91 7:1822d391:::smithi07694878-47:head v 122'42, mlcod=118'38)", 2021-03-15T22:21:32.367 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:37.026000+0000", 2021-03-15T22:21:32.367 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 55.171251988999998, 2021-03-15T22:21:32.368 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011542325000000001, 2021-03-15T22:21:32.368 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.368 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.368 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.368 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.369 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.369 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.026000+0000", 2021-03-15T22:21:32.369 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.369 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.369 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.370 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.370 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.026000+0000", 2021-03-15T22:21:32.370 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053171069999999997 2021-03-15T22:21:32.370 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.371 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.371 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.371 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031317+0000", 2021-03-15T22:21:32.371 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8739999999999997e-06 2021-03-15T22:21:32.371 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.372 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.372 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.372 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031321+0000", 2021-03-15T22:21:32.372 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3403e-05 2021-03-15T22:21:32.372 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.373 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.373 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.373 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031334+0000", 2021-03-15T22:21:32.373 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1370000000000001e-06 2021-03-15T22:21:32.374 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.374 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.374 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.374 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031339+0000", 2021-03-15T22:21:32.375 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3701000000000001e-05 2021-03-15T22:21:32.375 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.375 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.375 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031383+0000", 2021-03-15T22:21:32.376 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0068000000000002e-05 2021-03-15T22:21:32.376 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.376 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.376 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.376 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031403+0000", 2021-03-15T22:21:32.377 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061123979999999998 2021-03-15T22:21:32.377 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.377 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.377 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.378 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.037515+0000", 2021-03-15T22:21:32.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6636999999999999e-05 2021-03-15T22:21:32.378 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.379 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.037542+0000", 2021-03-15T22:21:32.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011542325000000001 2021-03-15T22:21:32.379 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.379 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.380 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.380 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:680 7.5 7:ac84e656:::smithi07694878-31:head [read 0~3226911,checksum] snapc 0=[] ondisk+read+known_if_redirected e137)", 2021-03-15T22:21:32.380 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:52.069145+0000", 2021-03-15T22:21:32.381 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 40.128106889000001, 2021-03-15T22:21:32.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056771690000000001, 2021-03-15T22:21:32.381 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.381 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:21:32.381 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:32.382 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:32.382 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:32.382 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 680 2021-03-15T22:21:32.382 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.383 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.383 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.383 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069145+0000", 2021-03-15T22:21:32.384 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.384 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.384 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069145+0000", 2021-03-15T22:21:32.385 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5859999999999999e-06 2021-03-15T22:21:32.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.385 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.385 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069147+0000", 2021-03-15T22:21:32.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7990000000000001e-06 2021-03-15T22:21:32.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.386 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.386 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069149+0000", 2021-03-15T22:21:32.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9809999999999997e-06 2021-03-15T22:21:32.387 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.388 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069156+0000", 2021-03-15T22:21:32.388 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2540000000000003e-06 2021-03-15T22:21:32.388 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.388 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.388 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.389 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069160+0000", 2021-03-15T22:21:32.389 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033422629999999998 2021-03-15T22:21:32.389 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.389 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.390 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.390 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.072503+0000", 2021-03-15T22:21:32.390 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9270999999999994e-05 2021-03-15T22:21:32.390 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.390 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.391 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.072582+0000", 2021-03-15T22:21:32.391 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022400150000000001 2021-03-15T22:21:32.391 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.392 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.392 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.392 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.074822+0000", 2021-03-15T22:21:32.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056771690000000001 2021-03-15T22:21:32.392 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.393 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.393 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.393 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.393 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.394 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:855 7.5 7:ac84e656:::smithi07694878-31:head [append 0~0] snapc 35=[34,30] ondisk+write+known_if_redirected e152)", 2021-03-15T22:21:32.394 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:07.018148+0000", 2021-03-15T22:21:32.394 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 25.179103452, 2021-03-15T22:21:32.394 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046879779999999998, 2021-03-15T22:21:32.395 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.395 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.395 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:32.395 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:32.395 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:32.396 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 855 2021-03-15T22:21:32.396 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.396 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.396 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.397 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.397 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018148+0000", 2021-03-15T22:21:32.397 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.397 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.398 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.398 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018148+0000", 2021-03-15T22:21:32.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0260000000000002e-06 2021-03-15T22:21:32.398 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018150+0000", 2021-03-15T22:21:32.399 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7800000000000001e-06 2021-03-15T22:21:32.399 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.400 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018153+0000", 2021-03-15T22:21:32.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2996000000000001e-05 2021-03-15T22:21:32.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.401 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018166+0000", 2021-03-15T22:21:32.401 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6180000000000003e-06 2021-03-15T22:21:32.402 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.402 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.402 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018172+0000", 2021-03-15T22:21:32.403 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6759000000000003e-05 2021-03-15T22:21:32.403 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.403 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.403 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018228+0000", 2021-03-15T22:21:32.404 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011022 2021-03-15T22:21:32.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.404 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.405 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018339+0000", 2021-03-15T22:21:32.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027672900000000003 2021-03-15T22:21:32.405 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.405 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.405 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:21:32.406 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018615+0000", 2021-03-15T22:21:32.406 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00152736 2021-03-15T22:21:32.406 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.406 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:32.407 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.020143+0000", 2021-03-15T22:21:32.407 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012256039999999999 2021-03-15T22:21:32.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.408 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:32.408 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.021368+0000", 2021-03-15T22:21:32.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013614650000000001 2021-03-15T22:21:32.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.409 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:32.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.022730+0000", 2021-03-15T22:21:32.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2139999999999998e-05 2021-03-15T22:21:32.409 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.410 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.022792+0000", 2021-03-15T22:21:32.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4280999999999998e-05 2021-03-15T22:21:32.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.022836+0000", 2021-03-15T22:21:32.411 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046879779999999998 2021-03-15T22:21:32.412 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.412 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.412 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.413 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:856 7.5 7:ac84e656:::smithi07694878-31:head [rollback 30] snapc 35=[34,30] ondisk+write+known_if_redirected e152)", 2021-03-15T22:21:32.413 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:07.018183+0000", 2021-03-15T22:21:32.413 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 25.179068167, 2021-03-15T22:21:32.413 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048534429999999998, 2021-03-15T22:21:32.414 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.414 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.414 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:32.414 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:32.414 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:32.415 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 856 2021-03-15T22:21:32.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.415 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.415 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.416 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018183+0000", 2021-03-15T22:21:32.416 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.416 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.417 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.417 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018183+0000", 2021-03-15T22:21:32.417 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6599999999999995e-07 2021-03-15T22:21:32.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.418 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.418 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.418 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018184+0000", 2021-03-15T22:21:32.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1500000000000003e-07 2021-03-15T22:21:32.419 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.419 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.419 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.419 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018185+0000", 2021-03-15T22:21:32.419 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1830000000000002e-06 2021-03-15T22:21:32.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.420 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.420 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018190+0000", 2021-03-15T22:21:32.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6809999999999998e-06 2021-03-15T22:21:32.421 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.421 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.422 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018193+0000", 2021-03-15T22:21:32.422 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00092436700000000005 2021-03-15T22:21:32.422 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.422 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.423 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.423 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.019117+0000", 2021-03-15T22:21:32.423 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9268000000000003e-05 2021-03-15T22:21:32.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.424 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.019166+0000", 2021-03-15T22:21:32.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016772600000000001 2021-03-15T22:21:32.424 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.425 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:21:32.425 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.019334+0000", 2021-03-15T22:21:32.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00081421599999999998 2021-03-15T22:21:32.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.426 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:32.426 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.020148+0000", 2021-03-15T22:21:32.426 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012727999999999999 2021-03-15T22:21:32.426 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.427 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.427 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:32.427 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.021421+0000", 2021-03-15T22:21:32.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001503573 2021-03-15T22:21:32.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.428 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.428 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:32.428 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.022925+0000", 2021-03-15T22:21:32.428 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5167999999999998e-05 2021-03-15T22:21:32.429 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.429 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.429 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.429 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.022980+0000", 2021-03-15T22:21:32.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6879999999999998e-05 2021-03-15T22:21:32.430 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.430 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.430 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.023037+0000", 2021-03-15T22:21:32.431 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048534429999999998 2021-03-15T22:21:32.431 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.431 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.431 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.431 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.432 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:857 7.5 7:ac84e656:::smithi07694878-31:head [append 0~0] snapc 35=[34,30] ondisk+write+known_if_redirected e152)", 2021-03-15T22:21:32.432 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:07.018215+0000", 2021-03-15T22:21:32.432 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 25.179036573000001, 2021-03-15T22:21:32.433 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049281749999999999, 2021-03-15T22:21:32.433 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.433 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.433 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:21:32.433 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:21:32.434 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:32.434 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 857 2021-03-15T22:21:32.434 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.434 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.435 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018215+0000", 2021-03-15T22:21:32.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.435 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.436 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.436 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018215+0000", 2021-03-15T22:21:32.436 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4099999999999998e-07 2021-03-15T22:21:32.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.437 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.437 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.437 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018216+0000", 2021-03-15T22:21:32.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.54e-07 2021-03-15T22:21:32.438 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.438 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.438 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018216+0000", 2021-03-15T22:21:32.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5770000000000001e-06 2021-03-15T22:21:32.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.439 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018222+0000", 2021-03-15T22:21:32.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.763e-06 2021-03-15T22:21:32.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.440 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.440 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.018230+0000", 2021-03-15T22:21:32.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001457582 2021-03-15T22:21:32.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.441 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.019687+0000", 2021-03-15T22:21:32.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0978999999999998e-05 2021-03-15T22:21:32.442 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.442 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.019768+0000", 2021-03-15T22:21:32.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016400599999999999 2021-03-15T22:21:32.443 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:21:32.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.019932+0000", 2021-03-15T22:21:32.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020119109999999999 2021-03-15T22:21:32.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:21:32.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.021944+0000", 2021-03-15T22:21:32.445 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00094588099999999998 2021-03-15T22:21:32.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.445 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.446 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:32.446 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.022890+0000", 2021-03-15T22:21:32.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018295900000000001 2021-03-15T22:21:32.446 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.447 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:32.447 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.023073+0000", 2021-03-15T22:21:32.447 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0374000000000001e-05 2021-03-15T22:21:32.447 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.448 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.448 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.023104+0000", 2021-03-15T22:21:32.448 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9647999999999997e-05 2021-03-15T22:21:32.449 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.449 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.449 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.449 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.023143+0000", 2021-03-15T22:21:32.449 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0049281749999999999 2021-03-15T22:21:32.450 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.450 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.450 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.451 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.451 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:868 7.6 e152/91 7:63693c38:::smithi07694878-22:head v 152'176, mlcod=149'172)", 2021-03-15T22:21:32.451 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:07.032095+0000", 2021-03-15T22:21:32.451 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 25.165156347, 2021-03-15T22:21:32.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0089994459999999995, 2021-03-15T22:21:32.452 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.452 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.452 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.032095+0000", 2021-03-15T22:21:32.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.453 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.454 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.454 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.032095+0000", 2021-03-15T22:21:32.454 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044590350000000001 2021-03-15T22:21:32.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.455 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.455 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036554+0000", 2021-03-15T22:21:32.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4150000000000001e-06 2021-03-15T22:21:32.455 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.455 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.456 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036559+0000", 2021-03-15T22:21:32.456 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5784e-05 2021-03-15T22:21:32.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.457 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036574+0000", 2021-03-15T22:21:32.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9740000000000002e-06 2021-03-15T22:21:32.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036582+0000", 2021-03-15T22:21:32.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4443999999999995e-05 2021-03-15T22:21:32.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.459 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.459 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036647+0000", 2021-03-15T22:21:32.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1956999999999999e-05 2021-03-15T22:21:32.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.460 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036679+0000", 2021-03-15T22:21:32.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004329612 2021-03-15T22:21:32.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.461 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.461 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.461 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.041008+0000", 2021-03-15T22:21:32.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6224999999999995e-05 2021-03-15T22:21:32.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.462 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.463 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.041095+0000", 2021-03-15T22:21:32.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0089994459999999995 2021-03-15T22:21:32.463 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.463 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.463 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.464 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.464 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.464 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:1202 7.f e175/91 7:fc7cd984:::smithi07694878-4:head v 175'141, mlcod=175'138)", 2021-03-15T22:21:32.464 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:30.106191+0000", 2021-03-15T22:21:32.465 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 2.091060395, 2021-03-15T22:21:32.465 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004542028, 2021-03-15T22:21:32.465 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.465 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.465 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.466 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.466 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.466 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:30.106191+0000", 2021-03-15T22:21:32.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.467 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.467 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:30.106191+0000", 2021-03-15T22:21:32.467 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023359409999999998 2021-03-15T22:21:32.468 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.468 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.468 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:30.108527+0000", 2021-03-15T22:21:32.469 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9749999999999999e-06 2021-03-15T22:21:32.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.469 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.469 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:30.108530+0000", 2021-03-15T22:21:32.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3217000000000001e-05 2021-03-15T22:21:32.470 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.470 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.470 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:30.108543+0000", 2021-03-15T22:21:32.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0379999999999999e-06 2021-03-15T22:21:32.471 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:30.108548+0000", 2021-03-15T22:21:32.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5760999999999996e-05 2021-03-15T22:21:32.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.473 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.473 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:30.108614+0000", 2021-03-15T22:21:32.473 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6686e-05 2021-03-15T22:21:32.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.474 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.474 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:30.108661+0000", 2021-03-15T22:21:32.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020456290000000002 2021-03-15T22:21:32.474 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:30.110706+0000", 2021-03-15T22:21:32.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6781e-05 2021-03-15T22:21:32.476 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:30.110733+0000", 2021-03-15T22:21:32.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004542028 2021-03-15T22:21:32.477 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.477 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.477 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.478 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.2.0:112 7.8 e176/91 7:1822d391:::smithi07694878-47:4d v 176'104, mlcod=175'101)", 2021-03-15T22:21:32.478 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:31.018380+0000", 2021-03-15T22:21:32.478 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.178871835, 2021-03-15T22:21:32.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055471119999999999, 2021-03-15T22:21:32.479 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.479 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.479 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018380+0000", 2021-03-15T22:21:32.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.480 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.481 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018380+0000", 2021-03-15T22:21:32.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4490000000000002e-06 2021-03-15T22:21:32.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.482 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.482 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018382+0000", 2021-03-15T22:21:32.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7350000000000001e-06 2021-03-15T22:21:32.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.483 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.483 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018385+0000", 2021-03-15T22:21:32.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4560000000000006e-06 2021-03-15T22:21:32.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.484 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.484 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.484 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018394+0000", 2021-03-15T22:21:32.484 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6590000000000001e-06 2021-03-15T22:21:32.485 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.485 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018401+0000", 2021-03-15T22:21:32.486 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7095000000000006e-05 2021-03-15T22:21:32.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.486 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.487 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018478+0000", 2021-03-15T22:21:32.487 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3722000000000001e-05 2021-03-15T22:21:32.487 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.487 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.487 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.488 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018502+0000", 2021-03-15T22:21:32.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053930260000000004 2021-03-15T22:21:32.488 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.488 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.489 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.489 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.023895+0000", 2021-03-15T22:21:32.489 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1970000000000001e-05 2021-03-15T22:21:32.489 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.489 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.490 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.490 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.023927+0000", 2021-03-15T22:21:32.490 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055471119999999999 2021-03-15T22:21:32.490 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.491 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.491 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.491 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.491 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:1191 7.6 e176/175 7:61e0cc05:::smithi07694878-16:head v 176'225, mlcod=176'221)", 2021-03-15T22:21:32.492 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:31.032485+0000", 2021-03-15T22:21:32.492 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.1647665970000001, 2021-03-15T22:21:32.492 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013396315000000001, 2021-03-15T22:21:32.492 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:21:32.493 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:32.493 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:21:32.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.493 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:21:32.493 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.032485+0000", 2021-03-15T22:21:32.494 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:21:32.494 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.494 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:21:32.495 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.032485+0000", 2021-03-15T22:21:32.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055944209999999996 2021-03-15T22:21:32.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.495 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.495 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:21:32.496 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038079+0000", 2021-03-15T22:21:32.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.549e-06 2021-03-15T22:21:32.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.496 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:21:32.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038084+0000", 2021-03-15T22:21:32.497 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6657999999999999e-05 2021-03-15T22:21:32.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.497 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.497 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:21:32.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038101+0000", 2021-03-15T22:21:32.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9969999999999998e-06 2021-03-15T22:21:32.498 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:21:32.499 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038109+0000", 2021-03-15T22:21:32.499 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2562999999999998e-05 2021-03-15T22:21:32.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.500 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:21:32.500 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038161+0000", 2021-03-15T22:21:32.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.62e-05 2021-03-15T22:21:32.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.501 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:21:32.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038187+0000", 2021-03-15T22:21:32.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076488060000000002 2021-03-15T22:21:32.501 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.502 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:21:32.502 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.045836+0000", 2021-03-15T22:21:32.502 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5120999999999997e-05 2021-03-15T22:21:32.503 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:21:32.503 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:21:32.503 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:21:32.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.045881+0000", 2021-03-15T22:21:32.504 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013396315000000001 2021-03-15T22:21:32.504 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.504 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.504 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.504 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:21:32.505 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:21:32.505 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:21:32.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:32 smithi076 conmon[56514]: cluster 2021-03-15T22:21:30.997009+0000 mgr.y (mgr.14144) 428 : cluster [DBG] 2021-03-15T22:21:32.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:32 smithi076 conmon[56514]: pgmap v390: 140 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 12 peering, 124 active+clean; 141 MiB data, 456 MiB used, 715 GiB / 715 GiB avail; 6.9 MiB/s rd, 636 KiB/s wr, 9 op/s; 3.1 MiB/s, 13 objects/s recovering 2021-03-15T22:21:32.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:32 smithi076 conmon[56514]: cluster 2021-03-15T22:21:31.442399+0000 osd.2 (osd.2) 36 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:21:32.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:32 smithi076 conmon[56514]: cluster 2021-03-15T22:21:32.022557+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2021-03-15T22:21:32.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:32.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:32 smithi076 conmon[62156]: cluster 2021-03-15T22:21:30.997009+0000 mgr.y (mgr.14144) 428 : cluster [DBG] pgmap v390: 140 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 12 peering, 124 active+clean; 141 MiB data, 456 MiB used, 715 GiB / 715 GiB avail; 6.9 MiB/s rd, 636 KiB/s wr, 9 op/s; 3.1 MiB/s, 13 objects/s recovering 2021-03-15T22:21:32.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:32 smithi076 conmon[62156]: cluster 2021-03-15T22:21:31.442399+0000 osd.2 (osd.2) 36 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:21:32.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:32 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:21:32.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:32 smithi076 conmon[62156]: -03-15T22:21: 2021-03-15T22:21:32.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:32 smithi076 conmon[62156]: 32.022557+0000 mon.a (mon. 2021-03-15T22:21:32.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:32 smithi076 conmon[62156]: 0) 774 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2021-03-15T22:21:32.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:32 smithi076 conmon[62156]: 2021-03-15T22:21:32.510 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:32.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:32.511 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:32 smithi125 conmon[56739]: cluster 2021-03-15T22:21:30.997009+0000 mgr.y (mgr.14144) 428 : cluster [DBG] pgmap v390: 140 pgs: 2 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 12 peering, 124 active+clean; 141 MiB data, 456 MiB used, 715 GiB / 715 GiB avail; 6.9 MiB/s rd, 636 KiB/s wr, 9 op/s; 3.1 MiB/s, 13 objects/s recovering 2021-03-15T22:21:32.511 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:32 smithi125 conmon[56739]: cluster 2021-03-15T22:21:31.442399+0000 osd.2 (osd.2) 36 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:21:32.511 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:32 smithi125 conmon[56739]: cluster 2021-03-15T22:21:32.022557+0000 mon.a (mon.0) 774 : cluster [DBG] osdmap e177: 8 total, 8 up, 8 in 2021-03-15T22:21:32.611 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:32.634 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:32.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:32.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:32.736 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:32.758 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:32.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:32.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:32.859 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:21:32.860 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:32.882 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:32.883 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:32.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:33.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:33.010 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:33.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:33.022 INFO:tasks.rados.rados.0.smithi076.stdout:444: done (0 left) 2021-03-15T22:21:33.023 INFO:tasks.rados.rados.0.smithi076.stdout:445: rollback oid 43 current snap is 41 2021-03-15T22:21:33.023 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 43 to 32 2021-03-15T22:21:33.023 INFO:tasks.rados.rados.0.smithi076.stdout:446: snap_remove snap 40 2021-03-15T22:21:33.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:33.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:33.135 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:33.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:33.237 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:33.260 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:33.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:33.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:33.291 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:33.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:33.366 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:33.385 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:33.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:33.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:33 smithi076 conmon[62156]: cluster 2021-03-15T22:21:31.688949+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:21:33.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:33 smithi076 conmon[62156]: cluster 2021-03-15T22:21:31.842178+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.b scrub ok 2021-03-15T22:21:33.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:33 smithi076 conmon[62156]: cluster 2021-03-15T22:21:32.017803+0000 osd.5 (osd.5) 40 : cluster [DBG] 3.5 deep-scrub ok 2021-03-15T22:21:33.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:33 smithi076 conmon[62156]: cluster 2021-03-15T22:21:32.290875+0000 osd.4 (osd.4) 46 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:21:33.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:33 smithi076 conmon[62156]: cluster 2021-03-15T22:21:33.018275+0000 mon.a (mon.0) 775 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2021-03-15T22:21:33.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:33 smithi076 conmon[56514]: cluster 2021-03-15T22:21:31.688949+0000 osd.7 (osd.7) 45 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:21:33.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:33 smithi076 conmon[56514]: cluster 2021-03-15T22:21:31.842178+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.b scrub ok 2021-03-15T22:21:33.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:33 smithi076 conmon[56514]: cluster 2021-03-15T22:21:32.017803+0000 osd.5 (osd.5) 40 : cluster [DBG] 3.5 deep-scrub ok 2021-03-15T22:21:33.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:33 smithi076 conmon[56514]: cluster 2021-03-15T22:21:32.290875+0000 osd.4 (osd.4) 46 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:21:33.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:33 smithi076 conmon[56514]: cluster 2021-03-15T22:21:33.018275+0000 mon.a (mon.0) 775 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2021-03-15T22:21:33.487 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:33.511 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:33.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:33.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:33.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:33 smithi125 conmon[56739]: cluster 2021-03-15T22:21:31.688949+0000 osd.7 (osd. 2021-03-15T22:21:33.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:33 smithi125 conmon[56739]: 7) 45 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:21:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:33 smithi125 conmon[56739]: cluster 2021-03-15T22:21:31.842178+0000 osd.3 (osd.3) 44 : cluster [DBG] 6.b scrub ok 2021-03-15T22:21:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:33 smithi125 conmon[56739]: cluster 2021-03-15T22:21:32.017803+0000 osd.5 (osd.5) 40 : cluster [DBG] 3.5 deep-scrub ok 2021-03-15T22:21:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:33 smithi125 conmon[56739]: cluster 2021-03-15T22:21:32.290875+0000 osd.4 (osd.4) 46 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:21:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:33 smithi125 conmon[56739]: cluster 2021-03-15T22:21:33.018275+0000 mon.a (mon.0) 2021-03-15T22:21:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:33 smithi125 conmon[56739]: 775 : cluster [DBG] osdmap e178: 8 total, 8 up, 8 in 2021-03-15T22:21:33.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:33.637 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:33.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:33.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:33.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:33.763 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:33.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:33.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:33.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:33.890 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:33.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:33.891 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:33.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:34.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:34.020 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:34.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:34.024 INFO:tasks.rados.rados.0.smithi076.stdout:446: done (1 left) 2021-03-15T22:21:34.025 INFO:tasks.rados.rados.0.smithi076.stdout:447: delete oid 6 current snap is 41 2021-03-15T22:21:34.025 INFO:tasks.rados.rados.0.smithi076.stdout:445: finishing rollback tid 1 to smithi07694878-43 2021-03-15T22:21:34.025 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 0 (ObjNum 261650656 snap 21896 seq_num 261817184) dirty dne 2021-03-15T22:21:34.028 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:21:34.032 INFO:tasks.rados.rados.0.smithi076.stdout:445: done (1 left) 2021-03-15T22:21:34.032 INFO:tasks.rados.rados.0.smithi076.stdout:447: done (0 left) 2021-03-15T22:21:34.032 INFO:tasks.rados.rados.0.smithi076.stdout:448: copy_from oid 2 from oid 16 current snap is 41 2021-03-15T22:21:34.032 INFO:tasks.rados.rados.0.smithi076.stdout:449: read oid 50 snap 32 2021-03-15T22:21:34.033 INFO:tasks.rados.rados.0.smithi076.stdout:449: expect deleted 2021-03-15T22:21:34.033 INFO:tasks.rados.rados.0.smithi076.stdout:450: snap_create 2021-03-15T22:21:34.060 INFO:tasks.rados.rados.0.smithi076.stdout:448: finishing copy_from to smithi07694878-2 2021-03-15T22:21:34.060 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 101 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:21:34.061 INFO:tasks.rados.rados.0.smithi076.stdout:448: finishing copy_from racing read to smithi07694878-2 2021-03-15T22:21:34.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:34.147 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:34.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:34.149 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:34.250 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:34.274 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:34.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:34.276 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:34.377 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:34.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:34.401 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:34.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:34.503 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:34.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: cluster 2021-03-15T22:21:32.704784+0000 osd.6 (osd.6) 24 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:34.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: cluster 2021-03-15T22:21:32.704889+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:34.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: cluster 2021-03-15T22:21:32.997581+0000 mgr.y (mgr.14144) 429 : cluster [DBG] pgmap v393: 140 pgs: 2 clean+premerge+peered, 2 peering, 136 active+clean; 154 MiB data, 497 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.8 MiB/s wr, 12 op/s; 7.8 MiB/s, 33 objects/s recovering 2021-03-15T22:21:34.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: cluster 2021-03-15T22:21:33.152751+0000 mon.a (mon.0) 776 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2021-03-15T22:21:34.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: cluster 2021-03-15T22:21:33.152781+0000 mon.a (mon.0) 777 : cluster [INF] Cluster is now healthy 2021-03-15T22:21:34.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: cluster 2021-03-15T22:21:33.258379+0000 osd.4 (osd.4) 47 : cluster [DBG] 4.e scrub ok 2021-03-15T22:21:34.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: cluster 2021-03-15T22:21:33.451813+0000 osd.2 (osd.2) 37 : cluster [DBG] 7.e scrub ok 2021-03-15T22:21:34.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: cluster 2021-03-15T22:21:34.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: 2021-03-15T22:21:33.796944+0000 osd.0 (osd.0) 42 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:21:34.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: audit 2021-03-15T22:21:33.827056+0000 mon.a (mon.0) 778 : audit [INF] from='client.? 172.21.15.76:0/2108859592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:21:34.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: cluster 2021-03-15T22:21:34.015884+0000 mon.a (mon.0) 779 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:34.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: audit 2021-03-15T22:21:34.020987+0000 mon.a (mon.0) 780 : audit [INF] from='client.? 172.21.15.76:0/2108859592' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:21:34.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[56514]: cluster 2021-03-15T22:21:34.021042+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2021-03-15T22:21:34.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: cluster 2021-03-15T22:21:32.704784+0000 osd.6 (osd.6) 24 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:34.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: cluster 2021-03-15T22:21:32.704889+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:34.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: cluster 2021-03-15T22:21:32.997581+0000 mgr.y (mgr.14144) 429 : cluster [DBG] pgmap v393: 140 pgs: 2 clean+premerge+peered, 2 peering, 136 active+clean; 154 MiB data, 497 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.8 MiB/s wr, 12 op/s; 7.8 MiB/s, 33 objects/s recovering 2021-03-15T22:21:34.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: cluster 2021-03-15T22:21:33.152751+0000 mon.a (mon.0) 776 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2021-03-15T22:21:34.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: cluster 2021-03-15T22:21:33.152781+0000 mon.a (mon.0) 777 : cluster [INF] Cluster is now healthy 2021-03-15T22:21:34.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: cluster 2021-03-15T22:21:33.258379+0000 osd.4 (osd.4) 47 : cluster [DBG] 4.e scrub ok 2021-03-15T22:21:34.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: cluster 2021-03-15T22:21:33.451813+0000 osd.2 (osd.2) 37 : cluster [DBG] 7.e scrub ok 2021-03-15T22:21:34.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: cluster 2021-03-15T22:21:33.796944+0000 osd.0 (osd.0) 42 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:21:34.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: audit 2021-03-15T22:21:33.827056+0000 mon.a (mon.0) 778 : audit [INF] from='client.? 172.21.15.76:0/2108859592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:21:34.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: cluster 2021-03-15T22:21:34.015884+0000 mon.a (mon.0) 779 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:34.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: audit 2021-03-15T22:21:34.020987+0000 mon.a (mon.0) 780 : audit [INF] from='client.? 172.21.15.76:0/2108859592' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:21:34.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:34 smithi076 conmon[62156]: cluster 2021-03-15T22:21:34.021042+0000 mon.a (mon.0) 781 : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2021-03-15T22:21:34.527 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:34.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:34.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:34.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: cluster 2021-03-15T22:21:32.704784+0000 osd.6 (osd.6) 24 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:34.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:21:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: 15T22:21:32.704889+0000 osd.6 (osd.6) 25 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: cluster 2021-03-15T22:21:32.997581+0000 mgr.y (mgr.14144) 429 : cluster [DBG] 2021-03-15T22:21:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: pgmap v393: 140 pgs: 2 clean+premerge+peered, 2 peering, 136 active+clean; 154 MiB data, 497 MiB used, 715 GiB / 715 GiB avail; 2.2 KiB/s rd, 2.8 MiB/s wr, 12 op/s; 7.8 MiB/s, 33 objects/s recovering 2021-03-15T22:21:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:21:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: -15T22:21:33.152751+0000 mon.a (mon.0) 776 2021-03-15T22:21:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 4 pgs peering) 2021-03-15T22:21:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:21:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: 15T22:21:33.152781+0000 mon.a 2021-03-15T22:21:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: (mon.0) 777 : cluster [INF] Cluster is now healthy 2021-03-15T22:21:34.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: 2021-03-15T22:21:34.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: cluster 2021-03-15T22:21:34.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: 2021-03-15T22:21:33 2021-03-15T22:21:34.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: .258379+0000 osd.4 (osd.4) 47 : 2021-03-15T22:21:34.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: cluster [DBG] 4.e scrub ok 2021-03-15T22:21:34.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:21:34.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: 03-15T22:21:33.451813+0000 2021-03-15T22:21:34.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: osd.2 (osd.2) 37 : cluster [DBG] 2021-03-15T22:21:34.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: 7.e scrub ok 2021-03-15T22:21:34.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:21:34.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: :21:33.796944+0000 osd.0 (osd.0 2021-03-15T22:21:34.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: ) 42 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:21:34.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: audit 2021 2021-03-15T22:21:34.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: -03-15T22:21:33.827056 2021-03-15T22:21:34.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: +0000 mon.a (mon.0) 778 : audit 2021-03-15T22:21:34.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/2108859592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:21:34.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:21:34.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: 22:21:34.015884+0000 mon.a (mon 2021-03-15T22:21:34.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: .0) 779 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:34.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:21:34.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: 03-15T22:21:34.020987+0000 2021-03-15T22:21:34.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: mon.a (mon.0) 780 : audit [INF] 2021-03-15T22:21:34.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2108859592' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:21:34.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: cluster 2021-03-15T22:21 2021-03-15T22:21:34.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: :34.021042+0000 mon.a (mon.0) 781 2021-03-15T22:21:34.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:34 smithi125 conmon[56739]: : cluster [DBG] osdmap e179: 8 total, 8 up, 8 in 2021-03-15T22:21:34.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:34.652 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:34.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:34.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:34.755 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:34.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:34.779 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:34.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:34.881 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:34.906 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:34.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:34.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:35.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:35.020 INFO:tasks.rados.rados.0.smithi076.stdout:448: done (2 left) 2021-03-15T22:21:35.021 INFO:tasks.rados.rados.0.smithi076.stdout:449: done (1 left) 2021-03-15T22:21:35.021 INFO:tasks.rados.rados.0.smithi076.stdout:450: done (0 left) 2021-03-15T22:21:35.021 INFO:tasks.rados.rados.0.smithi076.stdout:451: snap_remove snap 41 2021-03-15T22:21:35.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:35.038 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:35.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:35.140 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:35.162 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:35.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:35.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:35.265 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:21:35.265 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:21:35.266 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:21:35.266 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:21:35.266 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:21:35.266 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:21:35.266 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:21:35.267 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:21:35.267 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:21:35.267 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:21:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:21:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:21:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:21:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:21:35.269 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:21:35.269 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:21:35.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[56514]: cluster 2021-03-15T22:21:33.792441 2021-03-15T22:21:35.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[56514]: +0000 osd.6 (osd.6) 26 : cluster [DBG] 4.a scrub ok 2021-03-15T22:21:35.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[56514]: audit 2021-03-15T22:21:34.932786+0000 mon.a (mon.0) 782 : audit [DBG] from='client.? 172.21.15.76:0/1185622930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:35.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[56514]: audit 2021-03-15T22:21:34.998914+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-03-15T22:21:35.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[56514]: audit 2021-03-15T22:21:35.016872+0000 mon.a (mon.0) 784 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2021-03-15T22:21:35.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[56514]: cluster 2021-03-15T22:21:35.016936+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2021-03-15T22:21:35.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[62156]: cluster 2021-03-15T22:21:33.792441+0000 2021-03-15T22:21:35.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[62156]: osd.6 (osd.6) 26 : cluster [DBG] 4.a scrub ok 2021-03-15T22:21:35.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[62156]: audit 2021-03-15T22:21 2021-03-15T22:21:35.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[62156]: :34.932786+0000 mon.a (mon.0) 782 : audit [DBG] from='client.? 172.21.15.76:0/1185622930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:35.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[62156]: audit 2021-03-15T22:21:35.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[62156]: 2021-03-15T22:21:34.998914+0000 mon.a (mon.0) 783 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-03-15T22:21:35.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:21:35.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[62156]: 22:21:35.016872+0000 mon.a (mon.0) 784 : audit [INF] 2021-03-15T22:21:35.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2021-03-15T22:21:35.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:21:35.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:35 smithi076 conmon[62156]: 15T22:21:35.016936+0000 mon.a (mon.0) 785 : cluster [DBG] osdmap e180: 8 total, 8 up, 8 in 2021-03-15T22:21:35.274 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:35.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:35.298 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:35.298 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:35.399 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:35.426 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:35.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:35.529 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:35.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: cluster 2021-03-15T22:21:33.792441+0000 osd.6 (osd.6) 26 : cluster [DBG] 2021-03-15T22:21:35.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: 4.a scrub ok 2021-03-15T22:21:35.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: audit 2021-03-15T22:21:34.932786+0000 mon.a (mon.0) 782 : audit [DBG] from='client.? 172.21.15.76:0/1185622930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: audit 2021-03-15T22:21:34.998914 2021-03-15T22:21:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: +0000 mon.a (mon.0) 783 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-03-15T22:21:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: audit 2021-03-15T22:21:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: 2021-03-15T22:21:35.016872 2021-03-15T22:21:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: +0000 mon.a (mon.0) 784 : audit [INF] 2021-03-15T22:21:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "22"}]': finished 2021-03-15T22:21:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: cluster 2021-03-15T22:21:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: 2021-03-15T22:21:35.016936+0000 2021-03-15T22:21:35.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: mon.a (mon.0) 785 : cluster [DBG] 2021-03-15T22:21:35.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:35 smithi125 conmon[56739]: osdmap e180: 8 total, 8 up, 8 in 2021-03-15T22:21:35.551 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:35.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:35.654 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:35.679 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:35.681 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:35.782 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:35.804 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:35.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:35.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:35.906 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:35.928 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:35.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:36.020 INFO:tasks.rados.rados.0.smithi076.stdout:451: done (0 left) 2021-03-15T22:21:36.020 INFO:tasks.rados.rados.0.smithi076.stdout:452: read oid 35 snap -1 2021-03-15T22:21:36.020 INFO:tasks.rados.rados.0.smithi076.stdout:452: expect (ObjNum 63 snap 6 seq_num 63) 2021-03-15T22:21:36.021 INFO:tasks.rados.rados.0.smithi076.stdout:453: read oid 23 snap -1 2021-03-15T22:21:36.021 INFO:tasks.rados.rados.0.smithi076.stdout:453: expect (ObjNum 124 snap 24 seq_num 124) 2021-03-15T22:21:36.021 INFO:tasks.rados.rados.0.smithi076.stdout:454: snap_remove snap 32 2021-03-15T22:21:36.031 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:36.057 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:36.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:36.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:36.146 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:21:36.147 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:21:36.160 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:36.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:36 smithi076 conmon[56514]: cluster 2021-03-15T22:21:34.998157+0000 mgr.y ( 2021-03-15T22:21:36.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:36 smithi076 conmon[56514]: mgr.14144) 430 : cluster [DBG] pgmap v396: 139 pgs: 1 clean+premerge+peered, 1 peering, 137 active+clean; 155 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 MiB/s wr, 11 op/s; 4.7 MiB/s, 19 objects/s recovering 2021-03-15T22:21:36.168 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:36 smithi076 conmon[56514]: cluster 2021-03-15T22:21:36.016526+0000 mon.a (mon.0) 786 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2021-03-15T22:21:36.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:36.187 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:36.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:36.288 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:36.315 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:36.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:36.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:36.443 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:36.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:36.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:36.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:36 smithi076 conmon[62156]: cluster 2021-03-15T22:21:34.998157 2021-03-15T22:21:36.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:36 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 430 : cluster [DBG] pgmap v396: 139 pgs: 1 clean+premerge+peered, 1 peering, 137 active+clean; 155 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 MiB/s wr, 11 op/s; 4.7 MiB/s, 19 objects/s recovering 2021-03-15T22:21:36.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:36 smithi076 conmon[62156]: cluster 2021-03-15T22:21:36.016526+0000 mon.a (mon.0) 786 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2021-03-15T22:21:36.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:36 smithi125 conmon[56739]: cluster 2021-03-15T22:21:34.998157+0000 mgr.y (mgr.14144) 430 : cluster [DBG] 2021-03-15T22:21:36.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:36 smithi125 conmon[56739]: pgmap v396: 139 pgs: 1 clean+premerge+peered, 1 peering, 137 active+clean; 155 MiB data, 507 MiB used, 715 GiB / 715 GiB avail; 2.7 KiB/s rd, 2.5 MiB/s wr, 11 op/s; 4.7 MiB/s, 19 objects/s recovering 2021-03-15T22:21:36.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:36 smithi125 conmon[56739]: cluster 2021-03-15T22:21:36.016526+0000 mon.a (mon.0) 786 : cluster [DBG] osdmap e181: 8 total, 8 up, 8 in 2021-03-15T22:21:36.545 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:36.567 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:36.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:36.669 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:36.695 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:36.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:36.798 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:36.821 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:36.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:36.923 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:36.945 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:36.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:36.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:37.027 INFO:tasks.rados.rados.0.smithi076.stdout:454: done (2 left) 2021-03-15T22:21:37.028 INFO:tasks.rados.rados.0.smithi076.stdout:455: write oid 6 current snap is 42 2021-03-15T22:21:37.028 INFO:tasks.rados.rados.0.smithi076.stdout:455: seq_num 189 ranges {776523=614533,1976979=757720} 2021-03-15T22:21:37.035 INFO:tasks.rados.rados.0.smithi076.stdout:455: writing smithi07694878-6 from 776523 to 1391056 tid 1 2021-03-15T22:21:37.042 INFO:tasks.rados.rados.0.smithi076.stdout:455: writing smithi07694878-6 from 1976979 to 2734699 tid 2 2021-03-15T22:21:37.043 INFO:tasks.rados.rados.0.smithi076.stdout:456: write (excl) oid 17 current snap is 42 2021-03-15T22:21:37.043 INFO:tasks.rados.rados.0.smithi076.stdout:456: seq_num 190 ranges {793330=689782,2035461=617731,3101545=91785} 2021-03-15T22:21:37.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:37.050 INFO:tasks.rados.rados.0.smithi076.stdout:456: writing smithi07694878-17 from 793330 to 1483112 tid 1 2021-03-15T22:21:37.057 INFO:tasks.rados.rados.0.smithi076.stdout:456: writing smithi07694878-17 from 2035461 to 2653192 tid 2 2021-03-15T22:21:37.061 INFO:tasks.rados.rados.0.smithi076.stdout:456: writing smithi07694878-17 from 3101545 to 3193330 tid 3 2021-03-15T22:21:37.061 INFO:tasks.rados.rados.0.smithi076.stdout:457: read oid 46 snap -1 2021-03-15T22:21:37.062 INFO:tasks.rados.rados.0.smithi076.stdout:457: expect (ObjNum 143 snap 28 seq_num 143) 2021-03-15T22:21:37.062 INFO:tasks.rados.rados.0.smithi076.stdout:458: read oid 49 snap -1 2021-03-15T22:21:37.062 INFO:tasks.rados.rados.0.smithi076.stdout:458: expect (ObjNum 174 snap 37 seq_num 174) 2021-03-15T22:21:37.063 INFO:tasks.rados.rados.0.smithi076.stdout:459: write oid 19 current snap is 42 2021-03-15T22:21:37.063 INFO:tasks.rados.rados.0.smithi076.stdout:459: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2021-03-15T22:21:37.066 INFO:tasks.rados.rados.0.smithi076.stdout:459: writing smithi07694878-19 from 410137 to 891521 tid 1 2021-03-15T22:21:37.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:37.075 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:37.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:37.076 INFO:tasks.rados.rados.0.smithi076.stdout:459: writing smithi07694878-19 from 1410296 to 2171685 tid 2 2021-03-15T22:21:37.079 INFO:tasks.rados.rados.0.smithi076.stdout:459: writing smithi07694878-19 from 2728968 to 3210137 tid 3 2021-03-15T22:21:37.079 INFO:tasks.rados.rados.0.smithi076.stdout:460: rollback oid 33 current snap is 42 2021-03-15T22:21:37.080 INFO:tasks.rados.rados.0.smithi076.stdout:460: done (7 left) 2021-03-15T22:21:37.080 INFO:tasks.rados.rados.0.smithi076.stdout:461: snap_create 2021-03-15T22:21:37.080 INFO:tasks.rados.rados.0.smithi076.stdout:453: expect (ObjNum 124 snap 24 seq_num 124) 2021-03-15T22:21:37.148 INFO:tasks.rados.rados.0.smithi076.stdout:452: expect (ObjNum 63 snap 6 seq_num 63) 2021-03-15T22:21:37.177 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:37.202 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:37.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:37.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:37.267 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:21:37 smithi076 conmon[75764]: debug 2021-03-15T22:21:37.023+0000 7f9fbafaf700 -1 osd.3 pg_epoch: 182 pg[7.2( v 182'85 (0'0,182'85] local-lis/les=91/92 n=3 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [3,1,2] r=0 lpr=91 luod=179'83 crt=182'85 lcod 175'82 mlcod 175'82 active+clean+snaptrim trimq=[3f~1] ps=[41~1,49~1,4d~2,50~1,53~1]] removing snap head 2021-03-15T22:21:37.267 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:37 smithi076 conmon[65511]: debug 2021-03-15T22:21:37.267 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:37 smithi076 conmon[65511]: 2021-03-15T22:21:37.029+0000 7f1a2387f700 -1 osd.0 pg_epoch: 182 pg[7.6( v 178'227 (0'0,178'227] local-lis/les=175/176 n=10 ec=91/91 lis/c=175/175 les/c/f=176/176/0 sis=175) [0,7,5] r=0 lpr=175 crt=178'227 lcod 176'225 mlcod 176'225 active+clean+snaptrim trimq=[3f~1] ps=[41~1,49~1,4d~2,50~1,53~1]] removing snap head 2021-03-15T22:21:37.268 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:37 smithi076 conmon[65511]: 2021-03-15T22:21:37.268 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:37 smithi076 conmon[65511]: debug 2021-03-15T22:21:37.268 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:37 smithi076 conmon[65511]: 2021-03-15T22:21:37.032+0000 7f1a1f877700 -1 osd.0 pg_epoch: 182 pg[7.6( v 182'231 (0'0,182'231] local-lis/les=175/176 n=7 ec=91/91 lis/c=175/175 les/c/f=176/176/0 sis=175) [0,7,5] r=0 lpr=175 crt=182'231 lcod 182'229 mlcod 182'229 active+clean+snaptrim trimq=[3f~1] ps=[41~1,49~1,4d~2,50~1,53~1]] removing snap head 2021-03-15T22:21:37.268 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:37 smithi076 conmon[65511]: 2021-03-15T22:21:37.286 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:21:37 smithi125 conmon[67289]: debug 2021-03-15T22:21:37.055+0000 7fd4d8831700 -1 osd.6 pg_epoch: 182 pg[7.7( v 179'52 (0'0,179'52] local-lis/les=172/173 n=5 ec=91/91 lis/c=172/172 les/c/f=173/173/0 sis=172) [6,0,1] r=0 lpr=172 crt=179'52 lcod 173'51 mlcod 173'51 active+clean+snaptrim trimq=[3f~1] ps=[41~1,49~1,4d~2,50~1,53~1]] removing snap head 2021-03-15T22:21:37.287 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:21:37 smithi125 conmon[63905]: debug 2021-03-15T22:21:37.287 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:21:37 smithi125 conmon[63905]: 2021-03-15T22:21:37.028+0000 7fce00c31700 -1 osd.5 pg_epoch: 182 pg[7.0( v 182'273 (0'0,182'273] local-lis/les=91/92 n=10 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [5,1,3] r=0 lpr=91 luod=179'271 crt=182'273 lcod 178'269 mlcod 178'269 active+clean+snaptrim trimq=[3f~1] ps=[41~1,49~1,4d~2,50~1,53~1]] removing snap head 2021-03-15T22:21:37.287 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:21:37 smithi125 conmon[63905]: 2021-03-15T22:21:37.290 INFO:tasks.rados.rados.0.smithi076.stdout:455: finishing write tid 1 to smithi07694878-6 2021-03-15T22:21:37.290 INFO:tasks.rados.rados.0.smithi076.stdout:455: finishing write tid 2 to smithi07694878-6 2021-03-15T22:21:37.290 INFO:tasks.rados.rados.0.smithi076.stdout:455: finishing write tid 3 to smithi07694878-6 2021-03-15T22:21:37.291 INFO:tasks.rados.rados.0.smithi076.stdout:455: finishing write tid 4 to smithi07694878-6 2021-03-15T22:21:37.291 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 168 (ObjNum 189 snap 42 seq_num 189) dirty exists 2021-03-15T22:21:37.291 INFO:tasks.rados.rados.0.smithi076.stdout:455: left oid 6 (ObjNum 189 snap 42 seq_num 189) 2021-03-15T22:21:37.291 INFO:tasks.rados.rados.0.smithi076.stdout:456: finishing write tid 1 to smithi07694878-17 2021-03-15T22:21:37.292 INFO:tasks.rados.rados.0.smithi076.stdout:456: finishing write tid 2 to smithi07694878-17 2021-03-15T22:21:37.292 INFO:tasks.rados.rados.0.smithi076.stdout:456: finishing write tid 3 to smithi07694878-17 2021-03-15T22:21:37.292 INFO:tasks.rados.rados.0.smithi076.stdout:456: finishing write tid 4 to smithi07694878-17 2021-03-15T22:21:37.292 INFO:tasks.rados.rados.0.smithi076.stdout:456: finishing write tid 5 to smithi07694878-17 2021-03-15T22:21:37.293 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 53 (ObjNum 190 snap 42 seq_num 190) dirty exists 2021-03-15T22:21:37.293 INFO:tasks.rados.rados.0.smithi076.stdout:456: left oid 17 (ObjNum 190 snap 42 seq_num 190) 2021-03-15T22:21:37.293 INFO:tasks.rados.rados.0.smithi076.stdout:458: expect (ObjNum 174 snap 37 seq_num 174) 2021-03-15T22:21:37.304 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:37.330 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:37.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:37.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:37.391 INFO:tasks.rados.rados.0.smithi076.stdout:459: finishing write tid 1 to smithi07694878-19 2021-03-15T22:21:37.391 INFO:tasks.rados.rados.0.smithi076.stdout:457: expect (ObjNum 143 snap 28 seq_num 143) 2021-03-15T22:21:37.433 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:37.457 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:37.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:37.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:37.473 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:21:37.553 INFO:tasks.rados.rados.0.smithi076.stdout:459: finishing write tid 2 to smithi07694878-19 2021-03-15T22:21:37.553 INFO:tasks.rados.rados.0.smithi076.stdout:459: finishing write tid 3 to smithi07694878-19 2021-03-15T22:21:37.553 INFO:tasks.rados.rados.0.smithi076.stdout:459: finishing write tid 4 to smithi07694878-19 2021-03-15T22:21:37.554 INFO:tasks.rados.rados.0.smithi076.stdout:459: finishing write tid 5 to smithi07694878-19 2021-03-15T22:21:37.554 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 238 (ObjNum 191 snap 42 seq_num 191) dirty exists 2021-03-15T22:21:37.554 INFO:tasks.rados.rados.0.smithi076.stdout:459: left oid 19 (ObjNum 191 snap 42 seq_num 191) 2021-03-15T22:21:37.560 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:37.581 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:37.583 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:37.602 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 414 (0%) 2021-03-15T22:21:37.602 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:21:37.603 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 11.1327 -> 11.1327 (expected baseline 6.72913) 2021-03-15T22:21:37.603 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 25 -> 25 pgs (0.483092 -> 0.483092 * mean) 2021-03-15T22:21:37.603 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 64 -> 64 pgs (1.23671 -> 1.23671 * mean) 2021-03-15T22:21:37.603 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:21:37.604 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:21:37.604 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 0.05 2021-03-15T22:21:37.604 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:21:37.604 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0007 2021-03-15T22:21:37.604 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0067 2021-03-15T22:21:37.605 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:21:37.684 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:37.709 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:37.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:37.711 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:37.812 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:37.834 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:37.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:37.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:37.900 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:21:37.937 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:37.965 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:37.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:37.967 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:38.028 INFO:tasks.rados.rados.0.smithi076.stdout:452: done (7 left) 2021-03-15T22:21:38.029 INFO:tasks.rados.rados.0.smithi076.stdout:453: done (6 left) 2021-03-15T22:21:38.029 INFO:tasks.rados.rados.0.smithi076.stdout:455: done (5 left) 2021-03-15T22:21:38.029 INFO:tasks.rados.rados.0.smithi076.stdout:456: done (4 left) 2021-03-15T22:21:38.029 INFO:tasks.rados.rados.0.smithi076.stdout:457: done (3 left) 2021-03-15T22:21:38.030 INFO:tasks.rados.rados.0.smithi076.stdout:458: done (2 left) 2021-03-15T22:21:38.030 INFO:tasks.rados.rados.0.smithi076.stdout:459: done (1 left) 2021-03-15T22:21:38.030 INFO:tasks.rados.rados.0.smithi076.stdout:461: done (0 left) 2021-03-15T22:21:38.030 INFO:tasks.rados.rados.0.smithi076.stdout:462: read oid 23 snap 42 2021-03-15T22:21:38.030 INFO:tasks.rados.rados.0.smithi076.stdout:462: expect (ObjNum 124 snap 24 seq_num 124) 2021-03-15T22:21:38.031 INFO:tasks.rados.rados.0.smithi076.stdout:463: write oid 18 current snap is 43 2021-03-15T22:21:38.031 INFO:tasks.rados.rados.0.smithi076.stdout:463: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2021-03-15T22:21:38.033 INFO:tasks.rados.rados.0.smithi076.stdout:463: writing smithi07694878-18 from 426944 to 983577 tid 1 2021-03-15T22:21:38.039 INFO:tasks.rados.rados.0.smithi076.stdout:463: writing smithi07694878-18 from 1468778 to 2090178 tid 2 2021-03-15T22:21:38.046 INFO:tasks.rados.rados.0.smithi076.stdout:463: writing smithi07694878-18 from 2872744 to 3226944 tid 3 2021-03-15T22:21:38.046 INFO:tasks.rados.rados.0.smithi076.stdout:464: rollback oid 34 current snap is 43 2021-03-15T22:21:38.046 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 34 to 42 2021-03-15T22:21:38.047 INFO:tasks.rados.rados.0.smithi076.stdout:462: expect (ObjNum 124 snap 24 seq_num 124) 2021-03-15T22:21:38.068 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:38.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:38.091 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:38.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:38.119 INFO:tasks.rados.rados.0.smithi076.stdout:463: finishing write tid 1 to smithi07694878-18 2021-03-15T22:21:38.119 INFO:tasks.rados.rados.0.smithi076.stdout:463: finishing write tid 2 to smithi07694878-18 2021-03-15T22:21:38.119 INFO:tasks.rados.rados.0.smithi076.stdout:462: done (2 left) 2021-03-15T22:21:38.119 INFO:tasks.rados.rados.0.smithi076.stdout:465: write oid 6 current snap is 43 2021-03-15T22:21:38.120 INFO:tasks.rados.rados.0.smithi076.stdout:465: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2021-03-15T22:21:38.124 INFO:tasks.rados.rados.0.smithi076.stdout:465: writing smithi07694878-6 from 443751 to 1075633 tid 1 2021-03-15T22:21:38.132 INFO:tasks.rados.rados.0.smithi076.stdout:465: writing smithi07694878-6 from 1810907 to 2575965 tid 2 2021-03-15T22:21:38.137 INFO:tasks.rados.rados.0.smithi076.stdout:465: writing smithi07694878-6 from 3067461 to 3243751 tid 3 2021-03-15T22:21:38.137 INFO:tasks.rados.rados.0.smithi076.stdout:466: read oid 32 snap 42 2021-03-15T22:21:38.137 INFO:tasks.rados.rados.0.smithi076.stdout:466: expect (ObjNum 108 snap 23 seq_num 108) 2021-03-15T22:21:38.137 INFO:tasks.rados.rados.0.smithi076.stdout:467: write (excl) oid 12 current snap is 43 2021-03-15T22:21:38.138 INFO:tasks.rados.rados.0.smithi076.stdout:463: finishing write tid 3 to smithi07694878-18 2021-03-15T22:21:38.138 INFO:tasks.rados.rados.0.smithi076.stdout:467: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2021-03-15T22:21:38.143 INFO:tasks.rados.rados.0.smithi076.stdout:467: writing smithi07694878-12 from 460558 to 1167689 tid 1 2021-03-15T22:21:38.150 INFO:tasks.rados.rados.0.smithi076.stdout:467: writing smithi07694878-12 from 1869389 to 2378105 tid 2 2021-03-15T22:21:38.155 INFO:tasks.rados.rados.0.smithi076.stdout:467: writing smithi07694878-12 from 3094884 to 3260558 tid 3 2021-03-15T22:21:38.156 INFO:tasks.rados.rados.0.smithi076.stdout:468: read oid 43 snap 42 2021-03-15T22:21:38.156 INFO:tasks.rados.rados.0.smithi076.stdout:468: expect deleted 2021-03-15T22:21:38.156 INFO:tasks.rados.rados.0.smithi076.stdout:469: write (excl) oid 5 current snap is 43 2021-03-15T22:21:38.156 INFO:tasks.rados.rados.0.smithi076.stdout:463: finishing write tid 4 to smithi07694878-18 2021-03-15T22:21:38.157 INFO:tasks.rados.rados.0.smithi076.stdout:463: finishing write tid 5 to smithi07694878-18 2021-03-15T22:21:38.157 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 79 (ObjNum 192 snap 43 seq_num 192) dirty exists 2021-03-15T22:21:38.157 INFO:tasks.rados.rados.0.smithi076.stdout:463: left oid 18 (ObjNum 192 snap 43 seq_num 192) 2021-03-15T22:21:38.157 INFO:tasks.rados.rados.0.smithi076.stdout:464: finishing rollback tid 0 to smithi07694878-34 2021-03-15T22:21:38.157 INFO:tasks.rados.rados.0.smithi076.stdout:464: finishing rollback tid 1 to smithi07694878-34 2021-03-15T22:21:38.158 INFO:tasks.rados.rados.0.smithi076.stdout:464: finishing rollback tid 2 to smithi07694878-34 2021-03-15T22:21:38.158 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 138 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:21:38.158 INFO:tasks.rados.rados.0.smithi076.stdout:465: finishing write tid 1 to smithi07694878-6 2021-03-15T22:21:38.158 INFO:tasks.rados.rados.0.smithi076.stdout:465: finishing write tid 2 to smithi07694878-6 2021-03-15T22:21:38.159 INFO:tasks.rados.rados.0.smithi076.stdout:465: finishing write tid 3 to smithi07694878-6 2021-03-15T22:21:38.159 INFO:tasks.rados.rados.0.smithi076.stdout:465: finishing write tid 4 to smithi07694878-6 2021-03-15T22:21:38.159 INFO:tasks.rados.rados.0.smithi076.stdout:465: finishing write tid 5 to smithi07694878-6 2021-03-15T22:21:38.159 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 172 (ObjNum 193 snap 43 seq_num 193) dirty exists 2021-03-15T22:21:38.159 INFO:tasks.rados.rados.0.smithi076.stdout:465: left oid 6 (ObjNum 193 snap 43 seq_num 193) 2021-03-15T22:21:38.160 INFO:tasks.rados.rados.0.smithi076.stdout:467: finishing write tid 1 to smithi07694878-12 2021-03-15T22:21:38.160 INFO:tasks.rados.rados.0.smithi076.stdout:466: expect (ObjNum 108 snap 23 seq_num 108) 2021-03-15T22:21:38.192 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:38.218 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:38.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:38.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:38.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:38 smithi076 conmon[56514]: cluster 2021-03-15T22:21:36.998834 2021-03-15T22:21:38.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:38 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 431 : cluster [DBG] pgmap v399: 139 pgs: 1 clean+premerge+peered, 138 active+clean; 154 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 5 op/s 2021-03-15T22:21:38.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:38 smithi076 conmon[56514]: cluster 2021-03-15T22:21:37.022673+0000 mon.a (mon.0) 787 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2021-03-15T22:21:38.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:38 smithi076 conmon[56514]: audit 2021-03-15T22:21:37.600818+0000 mgr.y (mgr.14144) 432 : audit [DBG] from='client.14919 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:38.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:38 smithi076 conmon[62156]: cluster 2021-03-15T22:21:36.998834+0000 mgr.y (mgr.14144 2021-03-15T22:21:38.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:38 smithi076 conmon[62156]: ) 431 : cluster [DBG] pgmap v399: 139 pgs: 1 clean+premerge+peered, 138 active+clean; 154 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 5 op/s 2021-03-15T22:21:38.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:38 smithi076 conmon[62156]: cluster 2021-03-15T22:21:37.022673+0000 mon.a (mon.0) 787 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2021-03-15T22:21:38.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:38 smithi076 conmon[62156]: audit 2021-03-15T22:21:37.600818+0000 mgr.y (mgr.14144) 432 : audit [DBG] from='client.14919 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:38.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:38 smithi125 conmon[56739]: cluster 2021-03-15T22:21:36.998834+0000 mgr.y (mgr.14144) 431 : cluster [DBG] pgmap v399: 139 pgs: 1 clean+premerge+peered, 138 active+clean; 154 MiB data, 487 MiB used, 715 GiB / 715 GiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 5 op/s 2021-03-15T22:21:38.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:38 smithi125 conmon[56739]: cluster 2021-03-15T22:21:37.022673+0000 mon.a (mon.0) 787 : cluster [DBG] osdmap e182: 8 total, 8 up, 8 in 2021-03-15T22:21:38.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:38 smithi125 conmon[56739]: audit 2021-03-15T22:21:37.600818+0000 mgr.y (mgr.14144) 432 : 2021-03-15T22:21:38.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:38 smithi125 conmon[56739]: audit [DBG] from='client.14919 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:38.321 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:38.327 INFO:tasks.rados.rados.0.smithi076.stdout:467: finishing write tid 2 to smithi07694878-12 2021-03-15T22:21:38.327 INFO:tasks.rados.rados.0.smithi076.stdout:469: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2021-03-15T22:21:38.333 INFO:tasks.rados.rados.0.smithi076.stdout:469: writing smithi07694878-5 from 477365 to 1259745 tid 1 2021-03-15T22:21:38.341 INFO:tasks.rados.rados.0.smithi076.stdout:469: writing smithi07694878-5 from 1927871 to 2696598 tid 2 2021-03-15T22:21:38.344 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:38.345 INFO:tasks.rados.rados.0.smithi076.stdout:469: writing smithi07694878-5 from 3122307 to 3277365 tid 3 2021-03-15T22:21:38.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:38.346 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:38.347 INFO:tasks.rados.rados.0.smithi076.stdout:463: done (6 left) 2021-03-15T22:21:38.347 INFO:tasks.rados.rados.0.smithi076.stdout:464: done (5 left) 2021-03-15T22:21:38.348 INFO:tasks.rados.rados.0.smithi076.stdout:465: done (4 left) 2021-03-15T22:21:38.348 INFO:tasks.rados.rados.0.smithi076.stdout:466: done (3 left) 2021-03-15T22:21:38.348 INFO:tasks.rados.rados.0.smithi076.stdout:470: delete oid 14 current snap is 43 2021-03-15T22:21:38.348 INFO:tasks.rados.rados.0.smithi076.stdout:467: finishing write tid 3 to smithi07694878-12 2021-03-15T22:21:38.348 INFO:tasks.rados.rados.0.smithi076.stdout:467: finishing write tid 4 to smithi07694878-12 2021-03-15T22:21:38.349 INFO:tasks.rados.rados.0.smithi076.stdout:467: finishing write tid 5 to smithi07694878-12 2021-03-15T22:21:38.349 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 238 (ObjNum 194 snap 43 seq_num 194) dirty exists 2021-03-15T22:21:38.349 INFO:tasks.rados.rados.0.smithi076.stdout:467: left oid 12 (ObjNum 194 snap 43 seq_num 194) 2021-03-15T22:21:38.349 INFO:tasks.rados.rados.0.smithi076.stdout:469: finishing write tid 1 to smithi07694878-5 2021-03-15T22:21:38.350 INFO:tasks.rados.rados.0.smithi076.stdout:469: finishing write tid 2 to smithi07694878-5 2021-03-15T22:21:38.350 INFO:tasks.rados.rados.0.smithi076.stdout:467: done (3 left) 2021-03-15T22:21:38.350 INFO:tasks.rados.rados.0.smithi076.stdout:468: done (2 left) 2021-03-15T22:21:38.350 INFO:tasks.rados.rados.0.smithi076.stdout:470: done (1 left) 2021-03-15T22:21:38.350 INFO:tasks.rados.rados.0.smithi076.stdout:471: write oid 8 current snap is 43 2021-03-15T22:21:38.351 INFO:tasks.rados.rados.0.smithi076.stdout:471: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2021-03-15T22:21:38.352 INFO:tasks.rados.rados.0.smithi076.stdout:471: writing smithi07694878-8 from 510979 to 1043857 tid 1 2021-03-15T22:21:38.359 INFO:tasks.rados.rados.0.smithi076.stdout:471: writing smithi07694878-8 from 1528482 to 2300878 tid 2 2021-03-15T22:21:38.365 INFO:tasks.rados.rados.0.smithi076.stdout:471: writing smithi07694878-8 from 3060800 to 3310979 tid 3 2021-03-15T22:21:38.365 INFO:tasks.rados.rados.0.smithi076.stdout:472: copy_from oid 9 from oid 14 current snap is 43 2021-03-15T22:21:38.366 INFO:tasks.rados.rados.0.smithi076.stdout:469: finishing write tid 3 to smithi07694878-5 2021-03-15T22:21:38.366 INFO:tasks.rados.rados.0.smithi076.stdout:473: rollback oid 32 current snap is 43 2021-03-15T22:21:38.366 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 32 to 42 2021-03-15T22:21:38.366 INFO:tasks.rados.rados.0.smithi076.stdout:469: finishing write tid 4 to smithi07694878-5 2021-03-15T22:21:38.367 INFO:tasks.rados.rados.0.smithi076.stdout:469: finishing write tid 5 to smithi07694878-5 2021-03-15T22:21:38.367 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 55 (ObjNum 195 snap 43 seq_num 195) dirty exists 2021-03-15T22:21:38.367 INFO:tasks.rados.rados.0.smithi076.stdout:469: left oid 5 (ObjNum 195 snap 43 seq_num 195) 2021-03-15T22:21:38.367 INFO:tasks.rados.rados.0.smithi076.stdout:471: finishing write tid 1 to smithi07694878-8 2021-03-15T22:21:38.367 INFO:tasks.rados.rados.0.smithi076.stdout:471: finishing write tid 2 to smithi07694878-8 2021-03-15T22:21:38.368 INFO:tasks.rados.rados.0.smithi076.stdout:469: done (3 left) 2021-03-15T22:21:38.368 INFO:tasks.rados.rados.0.smithi076.stdout:474: read oid 27 snap -1 2021-03-15T22:21:38.368 INFO:tasks.rados.rados.0.smithi076.stdout:474: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:21:38.368 INFO:tasks.rados.rados.0.smithi076.stdout:475: write (excl) oid 20 current snap is 43 2021-03-15T22:21:38.369 INFO:tasks.rados.rados.0.smithi076.stdout:475: seq_num 198 ranges {527786=724480,1703317=516054,2804576=435035} 2021-03-15T22:21:38.372 INFO:tasks.rados.rados.0.smithi076.stdout:475: writing smithi07694878-20 from 527786 to 1252266 tid 1 2021-03-15T22:21:38.377 INFO:tasks.rados.rados.0.smithi076.stdout:475: writing smithi07694878-20 from 1703317 to 2219371 tid 2 2021-03-15T22:21:38.383 INFO:tasks.rados.rados.0.smithi076.stdout:475: writing smithi07694878-20 from 2804576 to 3239611 tid 3 2021-03-15T22:21:38.384 INFO:tasks.rados.rados.0.smithi076.stdout:476: write oid 12 current snap is 43 2021-03-15T22:21:38.384 INFO:tasks.rados.rados.0.smithi076.stdout:472: finishing copy_from to smithi07694878-9 2021-03-15T22:21:38.384 INFO:tasks.rados.rados.0.smithi076.stdout:472: got expected ENOENT (src dne) 2021-03-15T22:21:38.384 INFO:tasks.rados.rados.0.smithi076.stdout:476: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2021-03-15T22:21:38.391 INFO:tasks.rados.rados.0.smithi076.stdout:476: writing smithi07694878-12 from 544593 to 1344322 tid 1 2021-03-15T22:21:38.397 INFO:tasks.rados.rados.0.smithi076.stdout:476: writing smithi07694878-12 from 1761799 to 2537864 tid 2 2021-03-15T22:21:38.403 INFO:tasks.rados.rados.0.smithi076.stdout:476: writing smithi07694878-12 from 2948352 to 3344593 tid 3 2021-03-15T22:21:38.403 INFO:tasks.rados.rados.0.smithi076.stdout:477: rollback oid 47 current snap is 43 2021-03-15T22:21:38.403 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 47 to 42 2021-03-15T22:21:38.403 INFO:tasks.rados.rados.0.smithi076.stdout:472: finishing copy_from racing read to smithi07694878-9 2021-03-15T22:21:38.404 INFO:tasks.rados.rados.0.smithi076.stdout:472: done (6 left) 2021-03-15T22:21:38.404 INFO:tasks.rados.rados.0.smithi076.stdout:478: write oid 23 current snap is 43 2021-03-15T22:21:38.404 INFO:tasks.rados.rados.0.smithi076.stdout:473: finishing rollback tid 0 to smithi07694878-32 2021-03-15T22:21:38.404 INFO:tasks.rados.rados.0.smithi076.stdout:473: finishing rollback tid 1 to smithi07694878-32 2021-03-15T22:21:38.405 INFO:tasks.rados.rados.0.smithi076.stdout:473: finishing rollback tid 2 to smithi07694878-32 2021-03-15T22:21:38.405 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 285 (ObjNum 108 snap 23 seq_num 108) dirty exists 2021-03-15T22:21:38.405 INFO:tasks.rados.rados.0.smithi076.stdout:478: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2021-03-15T22:21:38.408 INFO:tasks.rados.rados.0.smithi076.stdout:478: writing smithi07694878-23 from 561400 to 1036378 tid 1 2021-03-15T22:21:38.412 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:38.415 INFO:tasks.rados.rados.0.smithi076.stdout:478: writing smithi07694878-23 from 1820281 to 2340004 tid 2 2021-03-15T22:21:38.422 INFO:tasks.rados.rados.0.smithi076.stdout:478: writing smithi07694878-23 from 2859422 to 3317001 tid 3 2021-03-15T22:21:38.422 INFO:tasks.rados.rados.0.smithi076.stdout:473: done (6 left) 2021-03-15T22:21:38.423 INFO:tasks.rados.rados.0.smithi076.stdout:479: read oid 44 snap -1 2021-03-15T22:21:38.423 INFO:tasks.rados.rados.0.smithi076.stdout:479: expect (ObjNum 99 snap 20 seq_num 99) 2021-03-15T22:21:38.423 INFO:tasks.rados.rados.0.smithi076.stdout:480: read oid 50 snap 42 2021-03-15T22:21:38.423 INFO:tasks.rados.rados.0.smithi076.stdout:480: expect (ObjNum 170 snap 37 seq_num 170) 2021-03-15T22:21:38.424 INFO:tasks.rados.rados.0.smithi076.stdout:481: write (excl) oid 10 current snap is 43 2021-03-15T22:21:38.424 INFO:tasks.rados.rados.0.smithi076.stdout:481: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2021-03-15T22:21:38.427 INFO:tasks.rados.rados.0.smithi076.stdout:481: writing smithi07694878-10 from 578207 to 1128434 tid 1 2021-03-15T22:21:38.436 INFO:tasks.rados.rados.0.smithi076.stdout:481: writing smithi07694878-10 from 1762410 to 2542144 tid 2 2021-03-15T22:21:38.441 INFO:tasks.rados.rados.0.smithi076.stdout:481: writing smithi07694878-10 from 3286845 to 3378207 tid 3 2021-03-15T22:21:38.441 INFO:tasks.rados.rados.0.smithi076.stdout:482: snap_create 2021-03-15T22:21:38.442 INFO:tasks.rados.rados.0.smithi076.stdout:474: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:21:38.444 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2021-03-15T22:21:38.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd test-reweight-by-utilization 1000 1.0 2021-03-15T22:21:38.448 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:38.451 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:38.478 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:38.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:38.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:38.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:38.604 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:38.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:38.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:38.610 INFO:tasks.rados.rados.0.smithi076.stdout:471: finishing write tid 3 to smithi07694878-8 2021-03-15T22:21:38.610 INFO:tasks.rados.rados.0.smithi076.stdout:471: finishing write tid 4 to smithi07694878-8 2021-03-15T22:21:38.611 INFO:tasks.rados.rados.0.smithi076.stdout:471: finishing write tid 5 to smithi07694878-8 2021-03-15T22:21:38.611 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 119 (ObjNum 197 snap 43 seq_num 197) dirty exists 2021-03-15T22:21:38.611 INFO:tasks.rados.rados.0.smithi076.stdout:471: left oid 8 (ObjNum 197 snap 43 seq_num 197) 2021-03-15T22:21:38.611 INFO:tasks.rados.rados.0.smithi076.stdout:475: finishing write tid 1 to smithi07694878-20 2021-03-15T22:21:38.612 INFO:tasks.rados.rados.0.smithi076.stdout:475: finishing write tid 2 to smithi07694878-20 2021-03-15T22:21:38.612 INFO:tasks.rados.rados.0.smithi076.stdout:475: finishing write tid 3 to smithi07694878-20 2021-03-15T22:21:38.612 INFO:tasks.rados.rados.0.smithi076.stdout:475: finishing write tid 4 to smithi07694878-20 2021-03-15T22:21:38.612 INFO:tasks.rados.rados.0.smithi076.stdout:475: finishing write tid 5 to smithi07694878-20 2021-03-15T22:21:38.612 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 142 (ObjNum 198 snap 43 seq_num 198) dirty exists 2021-03-15T22:21:38.613 INFO:tasks.rados.rados.0.smithi076.stdout:475: left oid 20 (ObjNum 198 snap 43 seq_num 198) 2021-03-15T22:21:38.613 INFO:tasks.rados.rados.0.smithi076.stdout:476: finishing write tid 1 to smithi07694878-12 2021-03-15T22:21:38.613 INFO:tasks.rados.rados.0.smithi076.stdout:476: finishing write tid 2 to smithi07694878-12 2021-03-15T22:21:38.613 INFO:tasks.rados.rados.0.smithi076.stdout:477: finishing rollback tid 0 to smithi07694878-47 2021-03-15T22:21:38.614 INFO:tasks.rados.rados.0.smithi076.stdout:477: finishing rollback tid 1 to smithi07694878-47 2021-03-15T22:21:38.614 INFO:tasks.rados.rados.0.smithi076.stdout:476: finishing write tid 3 to smithi07694878-12 2021-03-15T22:21:38.614 INFO:tasks.rados.rados.0.smithi076.stdout:477: finishing rollback tid 2 to smithi07694878-47 2021-03-15T22:21:38.614 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 112 (ObjNum 183 snap 40 seq_num 183) dirty exists 2021-03-15T22:21:38.614 INFO:tasks.rados.rados.0.smithi076.stdout:476: finishing write tid 4 to smithi07694878-12 2021-03-15T22:21:38.615 INFO:tasks.rados.rados.0.smithi076.stdout:476: finishing write tid 5 to smithi07694878-12 2021-03-15T22:21:38.615 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 242 (ObjNum 199 snap 43 seq_num 199) dirty exists 2021-03-15T22:21:38.615 INFO:tasks.rados.rados.0.smithi076.stdout:476: left oid 12 (ObjNum 199 snap 43 seq_num 199) 2021-03-15T22:21:38.615 INFO:tasks.rados.rados.0.smithi076.stdout:478: finishing write tid 1 to smithi07694878-23 2021-03-15T22:21:38.616 INFO:tasks.rados.rados.0.smithi076.stdout:478: finishing write tid 2 to smithi07694878-23 2021-03-15T22:21:38.616 INFO:tasks.rados.rados.0.smithi076.stdout:480: expect (ObjNum 170 snap 37 seq_num 170) 2021-03-15T22:21:38.707 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:38.713 INFO:tasks.rados.rados.0.smithi076.stdout:478: finishing write tid 3 to smithi07694878-23 2021-03-15T22:21:38.714 INFO:tasks.rados.rados.0.smithi076.stdout:478: finishing write tid 4 to smithi07694878-23 2021-03-15T22:21:38.714 INFO:tasks.rados.rados.0.smithi076.stdout:478: finishing write tid 5 to smithi07694878-23 2021-03-15T22:21:38.714 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 84 (ObjNum 200 snap 43 seq_num 200) dirty exists 2021-03-15T22:21:38.714 INFO:tasks.rados.rados.0.smithi076.stdout:478: left oid 23 (ObjNum 200 snap 43 seq_num 200) 2021-03-15T22:21:38.715 INFO:tasks.rados.rados.0.smithi076.stdout:479: expect (ObjNum 99 snap 20 seq_num 99) 2021-03-15T22:21:38.729 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:38.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:38.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:38.832 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:38.858 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:38.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:38.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:38.940 INFO:tasks.rados.rados.0.smithi076.stdout:481: finishing write tid 1 to smithi07694878-10 2021-03-15T22:21:38.941 INFO:tasks.rados.rados.0.smithi076.stdout:481: finishing write tid 2 to smithi07694878-10 2021-03-15T22:21:38.941 INFO:tasks.rados.rados.0.smithi076.stdout:481: finishing write tid 3 to smithi07694878-10 2021-03-15T22:21:38.941 INFO:tasks.rados.rados.0.smithi076.stdout:481: finishing write tid 4 to smithi07694878-10 2021-03-15T22:21:38.941 INFO:tasks.rados.rados.0.smithi076.stdout:481: finishing write tid 5 to smithi07694878-10 2021-03-15T22:21:38.942 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 105 (ObjNum 201 snap 43 seq_num 201) dirty exists 2021-03-15T22:21:38.942 INFO:tasks.rados.rados.0.smithi076.stdout:481: left oid 10 (ObjNum 201 snap 43 seq_num 201) 2021-03-15T22:21:38.961 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:38.984 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:38.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:38.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:39.030 INFO:tasks.rados.rados.0.smithi076.stdout:471: done (9 left) 2021-03-15T22:21:39.030 INFO:tasks.rados.rados.0.smithi076.stdout:474: done (8 left) 2021-03-15T22:21:39.031 INFO:tasks.rados.rados.0.smithi076.stdout:475: done (7 left) 2021-03-15T22:21:39.031 INFO:tasks.rados.rados.0.smithi076.stdout:476: done (6 left) 2021-03-15T22:21:39.031 INFO:tasks.rados.rados.0.smithi076.stdout:477: done (5 left) 2021-03-15T22:21:39.031 INFO:tasks.rados.rados.0.smithi076.stdout:478: done (4 left) 2021-03-15T22:21:39.032 INFO:tasks.rados.rados.0.smithi076.stdout:479: done (3 left) 2021-03-15T22:21:39.032 INFO:tasks.rados.rados.0.smithi076.stdout:480: done (2 left) 2021-03-15T22:21:39.032 INFO:tasks.rados.rados.0.smithi076.stdout:481: done (1 left) 2021-03-15T22:21:39.032 INFO:tasks.rados.rados.0.smithi076.stdout:482: done (0 left) 2021-03-15T22:21:39.032 INFO:tasks.rados.rados.0.smithi076.stdout:483: delete oid 24 current snap is 44 2021-03-15T22:21:39.033 INFO:tasks.rados.rados.0.smithi076.stdout:483: done (0 left) 2021-03-15T22:21:39.034 INFO:tasks.rados.rados.0.smithi076.stdout:484: write oid 5 current snap is 44 2021-03-15T22:21:39.034 INFO:tasks.rados.rados.0.smithi076.stdout:484: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2021-03-15T22:21:39.039 INFO:tasks.rados.rados.0.smithi076.stdout:484: writing smithi07694878-5 from 611821 to 1312546 tid 1 2021-03-15T22:21:39.047 INFO:tasks.rados.rados.0.smithi076.stdout:484: writing smithi07694878-5 from 1879374 to 2662777 tid 2 2021-03-15T22:21:39.052 INFO:tasks.rados.rados.0.smithi076.stdout:484: writing smithi07694878-5 from 3341691 to 3411821 tid 3 2021-03-15T22:21:39.052 INFO:tasks.rados.rados.0.smithi076.stdout:485: read oid 17 snap -1 2021-03-15T22:21:39.052 INFO:tasks.rados.rados.0.smithi076.stdout:485: expect (ObjNum 190 snap 42 seq_num 190) 2021-03-15T22:21:39.052 INFO:tasks.rados.rados.0.smithi076.stdout:486: read oid 6 snap -1 2021-03-15T22:21:39.053 INFO:tasks.rados.rados.0.smithi076.stdout:486: expect (ObjNum 193 snap 43 seq_num 193) 2021-03-15T22:21:39.053 INFO:tasks.rados.rados.0.smithi076.stdout:487: read oid 48 snap -1 2021-03-15T22:21:39.053 INFO:tasks.rados.rados.0.smithi076.stdout:487: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:21:39.053 INFO:tasks.rados.rados.0.smithi076.stdout:484: finishing write tid 1 to smithi07694878-5 2021-03-15T22:21:39.054 INFO:tasks.rados.rados.0.smithi076.stdout:484: finishing write tid 2 to smithi07694878-5 2021-03-15T22:21:39.054 INFO:tasks.rados.rados.0.smithi076.stdout:488: write oid 10 current snap is 44 2021-03-15T22:21:39.054 INFO:tasks.rados.rados.0.smithi076.stdout:488: seq_num 204 ranges {628628=775974,1937856=527061,3252761=175867} 2021-03-15T22:21:39.060 INFO:tasks.rados.rados.0.smithi076.stdout:488: writing smithi07694878-10 from 628628 to 1404602 tid 1 2021-03-15T22:21:39.066 INFO:tasks.rados.rados.0.smithi076.stdout:488: writing smithi07694878-10 from 1937856 to 2464917 tid 2 2021-03-15T22:21:39.072 INFO:tasks.rados.rados.0.smithi076.stdout:488: writing smithi07694878-10 from 3252761 to 3428628 tid 3 2021-03-15T22:21:39.073 INFO:tasks.rados.rados.0.smithi076.stdout:489: read oid 43 snap 42 2021-03-15T22:21:39.073 INFO:tasks.rados.rados.0.smithi076.stdout:489: expect deleted 2021-03-15T22:21:39.073 INFO:tasks.rados.rados.0.smithi076.stdout:490: read oid 1 snap -1 2021-03-15T22:21:39.073 INFO:tasks.rados.rados.0.smithi076.stdout:490: expect (ObjNum 100 snap 20 seq_num 100) 2021-03-15T22:21:39.073 INFO:tasks.rados.rados.0.smithi076.stdout:491: rollback oid 3 current snap is 44 2021-03-15T22:21:39.074 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 42 2021-03-15T22:21:39.074 INFO:tasks.rados.rados.0.smithi076.stdout:484: finishing write tid 3 to smithi07694878-5 2021-03-15T22:21:39.074 INFO:tasks.rados.rados.0.smithi076.stdout:484: finishing write tid 4 to smithi07694878-5 2021-03-15T22:21:39.074 INFO:tasks.rados.rados.0.smithi076.stdout:484: finishing write tid 5 to smithi07694878-5 2021-03-15T22:21:39.075 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 60 (ObjNum 203 snap 44 seq_num 203) dirty exists 2021-03-15T22:21:39.075 INFO:tasks.rados.rados.0.smithi076.stdout:484: left oid 5 (ObjNum 203 snap 44 seq_num 203) 2021-03-15T22:21:39.075 INFO:tasks.rados.rados.0.smithi076.stdout:486: expect (ObjNum 193 snap 43 seq_num 193) 2021-03-15T22:21:39.087 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:39.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:39.110 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:39.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:39.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:39.219 INFO:tasks.rados.rados.0.smithi076.stdout:487: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:21:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:39.234 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:39.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:39.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:39 smithi076 conmon[56514]: cluster 2021-03-15T22:21:38.023039+0000 mon.a (mon.0) 788 : cluster 2021-03-15T22:21:39.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:39 smithi076 conmon[56514]: [DBG] osdmap e183: 8 total, 8 up, 8 in 2021-03-15T22:21:39.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:39 smithi076 conmon[62156]: cluster 2021-03-15T22:21:38.023039 2021-03-15T22:21:39.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:39 smithi076 conmon[62156]: +0000 mon.a (mon.0) 788 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2021-03-15T22:21:39.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:39 smithi125 conmon[56739]: cluster 2021-03-15T22:21:38.023039+0000 mon.a (mon.0) 2021-03-15T22:21:39.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:39 smithi125 conmon[56739]: 788 : cluster [DBG] osdmap e183: 8 total, 8 up, 8 in 2021-03-15T22:21:39.321 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:21:39.336 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:39.363 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:39.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:39.364 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:39.366 INFO:tasks.rados.rados.0.smithi076.stdout:488: finishing write tid 1 to smithi07694878-10 2021-03-15T22:21:39.367 INFO:tasks.rados.rados.0.smithi076.stdout:484: done (7 left) 2021-03-15T22:21:39.367 INFO:tasks.rados.rados.0.smithi076.stdout:486: done (6 left) 2021-03-15T22:21:39.367 INFO:tasks.rados.rados.0.smithi076.stdout:487: done (5 left) 2021-03-15T22:21:39.367 INFO:tasks.rados.rados.0.smithi076.stdout:492: snap_remove snap 43 2021-03-15T22:21:39.465 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:39.488 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:39.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:39.489 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:39.590 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:39.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:39.613 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:39.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:39.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:39.739 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:39.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:39.741 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:39.768 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:39.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:21:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:21:39.842 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:39.869 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:39.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:39.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:39.973 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:39.999 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:39.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:40.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:40.018 INFO:tasks.rados.rados.0.smithi076.stdout:492: done (5 left) 2021-03-15T22:21:40.018 INFO:tasks.rados.rados.0.smithi076.stdout:493: write (excl) oid 25 current snap is 44 2021-03-15T22:21:40.019 INFO:tasks.rados.rados.0.smithi076.stdout:488: finishing write tid 2 to smithi07694878-10 2021-03-15T22:21:40.019 INFO:tasks.rados.rados.0.smithi076.stdout:493: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2021-03-15T22:21:40.024 INFO:tasks.rados.rados.0.smithi076.stdout:493: writing smithi07694878-25 from 645435 to 1096658 tid 1 2021-03-15T22:21:40.034 INFO:tasks.rados.rados.0.smithi076.stdout:493: writing smithi07694878-25 from 1879985 to 2667057 tid 2 2021-03-15T22:21:40.038 INFO:tasks.rados.rados.0.smithi076.stdout:493: writing smithi07694878-25 from 3280184 to 3445435 tid 3 2021-03-15T22:21:40.039 INFO:tasks.rados.rados.0.smithi076.stdout:494: delete oid 30 current snap is 44 2021-03-15T22:21:40.039 INFO:tasks.rados.rados.0.smithi076.stdout:488: finishing write tid 3 to smithi07694878-10 2021-03-15T22:21:40.039 INFO:tasks.rados.rados.0.smithi076.stdout:485: expect (ObjNum 190 snap 42 seq_num 190) 2021-03-15T22:21:40.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:40.123 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:40.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:40.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:40.221 INFO:tasks.rados.rados.0.smithi076.stdout:488: finishing write tid 4 to smithi07694878-10 2021-03-15T22:21:40.221 INFO:tasks.rados.rados.0.smithi076.stdout:488: finishing write tid 5 to smithi07694878-10 2021-03-15T22:21:40.222 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 109 (ObjNum 204 snap 44 seq_num 204) dirty exists 2021-03-15T22:21:40.222 INFO:tasks.rados.rados.0.smithi076.stdout:488: left oid 10 (ObjNum 204 snap 44 seq_num 204) 2021-03-15T22:21:40.222 INFO:tasks.rados.rados.0.smithi076.stdout:491: finishing rollback tid 0 to smithi07694878-3 2021-03-15T22:21:40.222 INFO:tasks.rados.rados.0.smithi076.stdout:491: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:21:40.223 INFO:tasks.rados.rados.0.smithi076.stdout:485: done (6 left) 2021-03-15T22:21:40.223 INFO:tasks.rados.rados.0.smithi076.stdout:488: done (5 left) 2021-03-15T22:21:40.223 INFO:tasks.rados.rados.0.smithi076.stdout:489: done (4 left) 2021-03-15T22:21:40.223 INFO:tasks.rados.rados.0.smithi076.stdout:494: done (3 left) 2021-03-15T22:21:40.223 INFO:tasks.rados.rados.0.smithi076.stdout:495: snap_create 2021-03-15T22:21:40.224 INFO:tasks.rados.rados.0.smithi076.stdout:491: finishing rollback tid 2 to smithi07694878-3 2021-03-15T22:21:40.224 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 146 (ObjNum 113 snap 24 seq_num 113) dirty exists 2021-03-15T22:21:40.224 INFO:tasks.rados.rados.0.smithi076.stdout:490: expect (ObjNum 100 snap 20 seq_num 100) 2021-03-15T22:21:40.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:40.247 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:40.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:40.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:40.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: cluster 2021-03-15T22:21:40.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 2021-03-15T22:21:40.332 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 2021 2021-03-15T22:21:40.333 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: - 2021-03-15T22:21:40.333 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 03 2021-03-15T22:21:40.333 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: - 2021-03-15T22:21:40.333 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 15 2021-03-15T22:21:40.333 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: T 2021-03-15T22:21:40.334 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 22 2021-03-15T22:21:40.334 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: : 2021-03-15T22:21:40.334 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 21 2021-03-15T22:21:40.334 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: : 2021-03-15T22:21:40.335 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 38 2021-03-15T22:21:40.335 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: . 2021-03-15T22:21:40.335 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 999156 2021-03-15T22:21:40.335 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 433 2021-03-15T22:21:40.335 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: : cluster [DBG] pgmap v402: 138 pgs: 138 active+clean; 145 MiB data, 475 MiB used, 715 GiB / 715 GiB avail; 5.1 MiB/s rd, 1.5 MiB/s wr, 5 op/s 2021-03-15T22:21:40.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:21:40.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 03-15T22:21:39.028010+0000 mon.a (mon.0 2021-03-15T22:21:40.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: ) 789 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2021-03-15T22:21:40.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:21:40.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 15T22:21:39.320245+0000 osd.3 (osd 2021-03-15T22:21:40.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: .3) 45 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:40.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:21:40.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: -03-15T22: 2021-03-15T22:21:40.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 21:39.320384+0000 osd.3 (osd.3 2021-03-15T22:21:40.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: ) 46 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:40.338 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: audit 2021 2021-03-15T22:21:40.338 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: -03-15T22:21:39.488711+0000 2021-03-15T22:21:40.338 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: mgr.y (mgr.14144) 434 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:40.338 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 2021-03-15T22:21:40.339 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: cluster 2021-03-15T22:21:40.339 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 2021-03-15T22:21:40.016301+0000 mon.a (mon. 2021-03-15T22:21:40.339 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[56514]: 0) 790 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2021-03-15T22:21:40.339 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: cluster 2021-03-15T22:21:40.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: 2021-03-15T22:21:40.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: 2021-03-15T22:21:38.999156+0000 mgr.y (mgr.14144) 433 2021-03-15T22:21:40.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: : cluster [DBG] pgmap v402: 138 pgs: 138 active+clean; 145 MiB data, 475 MiB used, 715 GiB / 715 GiB avail; 5.1 MiB/s rd, 1.5 MiB/s wr, 5 op/s 2021-03-15T22:21:40.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:40.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: 03-15T22:21:39.028010+0000 mon.a (mon. 2021-03-15T22:21:40.341 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: 0) 789 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2021-03-15T22:21:40.341 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:40.341 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: 03-15T22:21:39.320245+0000 osd.3 2021-03-15T22:21:40.341 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: (osd.3) 45 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:40.341 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: 2021-03-15T22:21:40.342 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: cluster 2021-03-15T22:21:40.342 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: 2021-03-15T22:21:39.320384+0000 osd.3 2021-03-15T22:21:40.342 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: (osd.3) 46 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:40.342 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:21:40.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: 03-15T22:21:39.488711+0000 mgr.y (mgr 2021-03-15T22:21:40.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: .14144) 434 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:40.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:21:40.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: -03-15T22:21:40.016301+0000 2021-03-15T22:21:40.344 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:40 smithi076 conmon[62156]: mon.a (mon.0) 790 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2021-03-15T22:21:40.350 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:40.375 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:40.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:40.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:40.430 INFO:tasks.rados.rados.0.smithi076.stdout:493: finishing write tid 1 to smithi07694878-25 2021-03-15T22:21:40.431 INFO:tasks.rados.rados.0.smithi076.stdout:493: finishing write tid 2 to smithi07694878-25 2021-03-15T22:21:40.431 INFO:tasks.rados.rados.0.smithi076.stdout:493: finishing write tid 3 to smithi07694878-25 2021-03-15T22:21:40.431 INFO:tasks.rados.rados.0.smithi076.stdout:493: finishing write tid 4 to smithi07694878-25 2021-03-15T22:21:40.431 INFO:tasks.rados.rados.0.smithi076.stdout:493: finishing write tid 5 to smithi07694878-25 2021-03-15T22:21:40.431 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 113 (ObjNum 205 snap 44 seq_num 205) dirty exists 2021-03-15T22:21:40.432 INFO:tasks.rados.rados.0.smithi076.stdout:493: left oid 25 (ObjNum 205 snap 44 seq_num 205) 2021-03-15T22:21:40.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:40.503 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:40.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:40.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:40.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: cluster 2021-03-15T22:21:38.999156+0000 mgr.y (mgr.14144) 433 : cluster 2021-03-15T22:21:40.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: [DBG] pgmap v402: 138 pgs: 138 active+clean; 145 MiB data, 475 MiB used, 715 GiB / 715 GiB avail; 5.1 MiB/s rd, 1.5 MiB/s wr, 5 op/s 2021-03-15T22:21:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: cluster 2021-03-15T22:21:39.028010+0000 mon.a (mon.0) 789 : cluster [DBG] osdmap e184: 8 total, 8 up, 8 in 2021-03-15T22:21:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: cluster 2021-03-15T22:21:39.320245+0000 osd.3 (osd 2021-03-15T22:21:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: .3) 45 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:21:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: 15T22:21:39.320384+0000 osd.3 (osd.3) 46 : 2021-03-15T22:21:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:21:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: :21:39.488711+0000 mgr.y (mgr.14144 2021-03-15T22:21:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: ) 434 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:21:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: -15T22:21:40.016301+0000 2021-03-15T22:21:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: mon.a (mon.0) 790 : cluster [DBG] osdmap e185: 8 total, 8 up, 8 in 2021-03-15T22:21:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:40 smithi125 conmon[56739]: 2021-03-15T22:21:40.571 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:40.605 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:40.607 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:21:40.607 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:21:40.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:40.632 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:40.632 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:40.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:40.759 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:40.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:40.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:40.862 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:40.886 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:40.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:40.988 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:41.011 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:41.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:41.046 INFO:tasks.rados.rados.0.smithi076.stdout:490: done (3 left) 2021-03-15T22:21:41.046 INFO:tasks.rados.rados.0.smithi076.stdout:491: done (2 left) 2021-03-15T22:21:41.046 INFO:tasks.rados.rados.0.smithi076.stdout:493: done (1 left) 2021-03-15T22:21:41.047 INFO:tasks.rados.rados.0.smithi076.stdout:495: done (0 left) 2021-03-15T22:21:41.047 INFO:tasks.rados.rados.0.smithi076.stdout:496: delete oid 11 current snap is 45 2021-03-15T22:21:41.047 INFO:tasks.rados.rados.0.smithi076.stdout:496: done (0 left) 2021-03-15T22:21:41.048 INFO:tasks.rados.rados.0.smithi076.stdout:497: snap_create 2021-03-15T22:21:41.114 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:41.136 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:41.138 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:41.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:41.260 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:41.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:41.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:41.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:41.390 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:41.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:41.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:41.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:41.516 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:41.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:41.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:41.643 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:21:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:41.645 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:41.645 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:41.746 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:41.769 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:41.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:41.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:41.872 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:41.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:41.899 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:41.900 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:42.001 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:42.026 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:42.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:42.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:42.048 INFO:tasks.rados.rados.0.smithi076.stdout:497: done (0 left) 2021-03-15T22:21:42.049 INFO:tasks.rados.rados.0.smithi076.stdout:498: snap_create 2021-03-15T22:21:42.117 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:42 smithi076 conmon[62156]: cluster 2021-03-15T22:21:40.999527+0000 mgr.y (mgr.14144) 435 : cluster [DBG] pgmap v405: 138 pgs: 138 active+clean; 142 MiB data, 460 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 21 op/s 2021-03-15T22:21:42.117 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:42 smithi076 conmon[62156]: cluster 2021-03-15T22:21:41.044258+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2021-03-15T22:21:42.117 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:42 smithi076 conmon[56514]: cluster 2021-03-15T22:21:40.999527+0000 mgr.y (mgr.14144) 435 : cluster [DBG] pgmap v405: 138 pgs: 138 active+clean; 142 MiB data, 460 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 21 op/s 2021-03-15T22:21:42.118 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:42 smithi076 conmon[56514]: cluster 2021-03-15T22:21:41.044258+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2021-03-15T22:21:42.130 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:42.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:42.156 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:42.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:42.257 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:42.283 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:42.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:42.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:42.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:42.411 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:42.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:42.514 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:42.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:42 smithi125 conmon[56739]: cluster 2021-03-15T22:21:40.999527+0000 mgr.y (mgr.14144) 435 : cluster [DBG] pgmap v405: 138 pgs: 138 active+clean; 142 MiB data, 460 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 21 op/s 2021-03-15T22:21:42.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:42 smithi125 conmon[56739]: 2021-03-15T22:21:42.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:42 smithi125 conmon[56739]: cluster 2021-03-15T22:21:41.044258+0000 mon.a (mon.0) 791 : cluster [DBG] osdmap e186: 8 total, 8 up, 8 in 2021-03-15T22:21:42.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:42.540 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:42.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:42.641 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:42.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:42.668 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:42.669 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:42.771 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:42.798 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:42.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:42.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:42.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:42.927 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:42.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:42.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:43.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:43.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:43.057 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:43.058 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:43.066 INFO:tasks.rados.rados.0.smithi076.stdout:498: done (0 left) 2021-03-15T22:21:43.066 INFO:tasks.rados.rados.0.smithi076.stdout:499: write (excl) oid 16 current snap is 47 2021-03-15T22:21:43.066 INFO:tasks.rados.rados.0.smithi076.stdout:499: seq_num 208 ranges {695856=793323,2171784=534399,3362453=133403} 2021-03-15T22:21:43.067 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:21:43.072 INFO:tasks.rados.rados.0.smithi076.stdout:499: writing smithi07694878-16 from 695856 to 1489179 tid 1 2021-03-15T22:21:43.078 INFO:tasks.rados.rados.0.smithi076.stdout:499: writing smithi07694878-16 from 2171784 to 2706183 tid 2 2021-03-15T22:21:43.084 INFO:tasks.rados.rados.0.smithi076.stdout:499: writing smithi07694878-16 from 3362453 to 3495856 tid 3 2021-03-15T22:21:43.085 INFO:tasks.rados.rados.0.smithi076.stdout:500: write oid 40 current snap is 47 2021-03-15T22:21:43.085 INFO:tasks.rados.rados.0.smithi076.stdout:499: finishing write tid 1 to smithi07694878-16 2021-03-15T22:21:43.086 INFO:tasks.rados.rados.0.smithi076.stdout:500: seq_num 209 ranges {712663=468572,1713913=678057,2873523=591733} 2021-03-15T22:21:43.090 INFO:tasks.rados.rados.0.smithi076.stdout:500: writing smithi07694878-40 from 712663 to 1181235 tid 1 2021-03-15T22:21:43.097 INFO:tasks.rados.rados.0.smithi076.stdout:500: writing smithi07694878-40 from 1713913 to 2391970 tid 2 2021-03-15T22:21:43.098 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:43.103 INFO:tasks.rados.rados.0.smithi076.stdout:500: writing smithi07694878-40 from 2873523 to 3465256 tid 3 2021-03-15T22:21:43.104 INFO:tasks.rados.rados.0.smithi076.stdout:501: snap_remove snap 46 2021-03-15T22:21:43.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:43.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:43.186 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:43.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:43.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:43.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:43.315 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:43.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:43.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:43.442 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:43.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:43.444 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:43.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:43 smithi076 conmon[56514]: cluster 2021-03-15T22:21:42.046147+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2021-03-15T22:21:43.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:43 smithi076 conmon[56514]: audit 2021-03-15T22:21:42.171315+0000 mon.a (mon.0) 793 : audit [INF] from='client.? 172.21.15.76:0/1190683080' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:43.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:43 smithi076 conmon[62156]: cluster 2021-03-15T22:21:42.046147+0000 mon.a (mon.0) 792 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2021-03-15T22:21:43.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:43 smithi076 conmon[62156]: audit 2021-03-15T22:21:42.171315+0000 mon.a (mon.0) 793 : audit [INF] from='client.? 172.21.15.76:0/1190683080' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:43.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:43 smithi125 conmon[56739]: cluster 2021-03-15T22:21:42.046147+0000 mon.a ( 2021-03-15T22:21:43.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:43 smithi125 conmon[56739]: mon.0) 792 : cluster [DBG] osdmap e187: 8 total, 8 up, 8 in 2021-03-15T22:21:43.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:43 smithi125 conmon[56739]: audit 2021-03-15T22:21:42.171315+0000 mon.a (mon.0) 793 : audit [INF] from='client.? 172.21.15.76:0/1190683080' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:43.545 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:43.571 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:43.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:43.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:43.674 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:43.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:43.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:43.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:43.830 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:43.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:43.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:43.896 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:21:43.913 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:21:43.930 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:21:43.932 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:43.948 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:21:43.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:43.956 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:43.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:43.966 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:21:43.984 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:21:44.003 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:21:44.022 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:21:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:21:44.023 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:21:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:21:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:21:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:21:44.024 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:21:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:21:44.025 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:21:44.058 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:44.072 INFO:tasks.rados.rados.0.smithi076.stdout:501: done (2 left) 2021-03-15T22:21:44.072 INFO:tasks.rados.rados.0.smithi076.stdout:502: snap_remove snap 44 2021-03-15T22:21:44.084 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:44.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:44.086 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:44.187 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:44.215 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:44.316 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:44.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[56514]: cluster 2021-03-15T22:21:42.999971 2021-03-15T22:21:44.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 436 : cluster [DBG] pgmap v408: 138 pgs: 138 active+clean; 133 MiB data, 431 MiB used, 715 GiB / 715 GiB avail; 37 MiB/s rd, 5.5 MiB/s wr, 51 op/s 2021-03-15T22:21:44.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[56514]: cluster 2021-03-15T22:21:43.050858+0000 mon.a (mon.0) 794 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:44.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[56514]: audit 2021-03-15T22:21:43.059569+0000 mon.a (mon.0) 795 : audit [INF] from='client.? 172.21.15.76:0/1190683080' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:44.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[56514]: cluster 2021-03-15T22:21:43.059614+0000 mon.a (mon.0) 796 : cluster [DBG] osdmap e188: 8 total, 8 up, 8 in 2021-03-15T22:21:44.323 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[56514]: audit 2021-03-15T22:21:43.877121+0000 mon.a (mon.0) 797 : audit [DBG] from='client.? 172.21.15.76:0/2839914780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:44.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: cluster 2021-03-15T22:21:42. 2021-03-15T22:21:44.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: 999971+0000 mgr.y (mgr.14144) 436 : cluster [DBG] pgmap v408: 138 pgs: 138 active+clean; 133 MiB data, 431 MiB used, 715 GiB / 715 GiB avail; 37 MiB/s rd, 5.5 MiB/s wr, 51 op/s 2021-03-15T22:21:44.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: cluster 2021-03-15T22:21:43. 2021-03-15T22:21:44.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: 050858+0000 mon.a (mon.0) 794 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:44.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:21:44.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: 15T22:21:43.059569+0000 mon.a (mon 2021-03-15T22:21:44.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: .0) 795 : audit [INF] from='client.? 172.21.15.76:0/1190683080' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:44.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:21:44.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: -03-15T22:21:43.059614 2021-03-15T22:21:44.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: +0000 mon.a (mon.0) 796 : cluster [DBG] 2021-03-15T22:21:44.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: osdmap e188: 8 total, 8 up, 8 in 2021-03-15T22:21:44.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: audit 2021-03-15T22:21:44.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: 2021-03-15T22:21:43.877121+0000 mon.a ( 2021-03-15T22:21:44.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:44 smithi076 conmon[62156]: mon.0) 797 : audit [DBG] from='client.? 172.21.15.76:0/2839914780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:44.342 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:44.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:44.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:44.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:44.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:44.470 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:44.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:44.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:44 smithi125 conmon[56739]: cluster 2021-03-15T22:21:42.999971+0000 mgr.y (mgr.14144) 436 : cluster 2021-03-15T22:21:44.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:44 smithi125 conmon[56739]: [DBG] pgmap v408: 138 pgs: 138 active+clean; 133 MiB data, 431 MiB used, 715 GiB / 715 GiB avail; 37 MiB/s rd, 5.5 MiB/s wr, 51 op/s 2021-03-15T22:21:44.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:44 smithi125 conmon[56739]: cluster 2021-03-15T22:21:43.050858+0000 mon.a (mon.0) 794 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:44.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:44 smithi125 conmon[56739]: audit 2021-03-15T22:21:43.059569+0000 mon.a (mon.0) 795 : audit [INF] from='client.? 172.21.15.76:0/1190683080' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:44.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:44 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:21:44.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:44 smithi125 conmon[56739]: -03-15T22:21:43.059614+0000 2021-03-15T22:21:44.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:44 smithi125 conmon[56739]: mon.a (mon.0) 796 : cluster [DBG] 2021-03-15T22:21:44.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:44 smithi125 conmon[56739]: osdmap e188: 8 total, 8 up, 8 in 2021-03-15T22:21:44.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:44 smithi125 conmon[56739]: audit 2021-03-15T22:21 2021-03-15T22:21:44.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:44 smithi125 conmon[56739]: :43.877121+0000 mon.a (mon.0) 2021-03-15T22:21:44.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:44 smithi125 conmon[56739]: 797 : audit [DBG] from='client.? 172.21.15.76:0/2839914780' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:44.573 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:44.600 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:44.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:44.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:44.703 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:44.727 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:44.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:44.729 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:44.830 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:44.856 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:44.858 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:44.959 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:44.984 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:44.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:44.986 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:45.026 INFO:tasks.rados.rados.0.smithi076.stdout:502: done (2 left) 2021-03-15T22:21:45.027 INFO:tasks.rados.rados.0.smithi076.stdout:503: copy_from oid 35 from oid 15 current snap is 47 2021-03-15T22:21:45.027 INFO:tasks.rados.rados.0.smithi076.stdout:504: write oid 49 current snap is 47 2021-03-15T22:21:45.027 INFO:tasks.rados.rados.0.smithi076.stdout:504: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2021-03-15T22:21:45.031 INFO:tasks.rados.rados.0.smithi076.stdout:504: writing smithi07694878-49 from 729470 to 1273291 tid 1 2021-03-15T22:21:45.038 INFO:tasks.rados.rados.0.smithi076.stdout:504: writing smithi07694878-49 from 1772395 to 2310463 tid 2 2021-03-15T22:21:45.044 INFO:tasks.rados.rados.0.smithi076.stdout:504: writing smithi07694878-49 from 2900946 to 3303951 tid 3 2021-03-15T22:21:45.044 INFO:tasks.rados.rados.0.smithi076.stdout:505: write (excl) oid 23 current snap is 47 2021-03-15T22:21:45.045 INFO:tasks.rados.rados.0.smithi076.stdout:499: finishing write tid 2 to smithi07694878-16 2021-03-15T22:21:45.045 INFO:tasks.rados.rados.0.smithi076.stdout:505: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2021-03-15T22:21:45.051 INFO:tasks.rados.rados.0.smithi076.stdout:505: writing smithi07694878-23 from 746277 to 1365347 tid 1 2021-03-15T22:21:45.058 INFO:tasks.rados.rados.0.smithi076.stdout:505: writing smithi07694878-23 from 1830877 to 2512603 tid 2 2021-03-15T22:21:45.064 INFO:tasks.rados.rados.0.smithi076.stdout:505: writing smithi07694878-23 from 2928369 to 3546277 tid 3 2021-03-15T22:21:45.065 INFO:tasks.rados.rados.0.smithi076.stdout:506: delete oid 38 current snap is 47 2021-03-15T22:21:45.065 INFO:tasks.rados.rados.0.smithi076.stdout:499: finishing write tid 3 to smithi07694878-16 2021-03-15T22:21:45.065 INFO:tasks.rados.rados.0.smithi076.stdout:499: finishing write tid 4 to smithi07694878-16 2021-03-15T22:21:45.065 INFO:tasks.rados.rados.0.smithi076.stdout:499: finishing write tid 5 to smithi07694878-16 2021-03-15T22:21:45.065 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 248 (ObjNum 208 snap 47 seq_num 208) dirty exists 2021-03-15T22:21:45.066 INFO:tasks.rados.rados.0.smithi076.stdout:499: left oid 16 (ObjNum 208 snap 47 seq_num 208) 2021-03-15T22:21:45.066 INFO:tasks.rados.rados.0.smithi076.stdout:500: finishing write tid 1 to smithi07694878-40 2021-03-15T22:21:45.066 INFO:tasks.rados.rados.0.smithi076.stdout:500: finishing write tid 2 to smithi07694878-40 2021-03-15T22:21:45.066 INFO:tasks.rados.rados.0.smithi076.stdout:500: finishing write tid 3 to smithi07694878-40 2021-03-15T22:21:45.067 INFO:tasks.rados.rados.0.smithi076.stdout:500: finishing write tid 4 to smithi07694878-40 2021-03-15T22:21:45.067 INFO:tasks.rados.rados.0.smithi076.stdout:500: finishing write tid 5 to smithi07694878-40 2021-03-15T22:21:45.067 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 290 (ObjNum 209 snap 47 seq_num 209) dirty exists 2021-03-15T22:21:45.067 INFO:tasks.rados.rados.0.smithi076.stdout:500: left oid 40 (ObjNum 209 snap 47 seq_num 209) 2021-03-15T22:21:45.067 INFO:tasks.rados.rados.0.smithi076.stdout:504: finishing write tid 1 to smithi07694878-49 2021-03-15T22:21:45.068 INFO:tasks.rados.rados.0.smithi076.stdout:504: finishing write tid 2 to smithi07694878-49 2021-03-15T22:21:45.068 INFO:tasks.rados.rados.0.smithi076.stdout:504: finishing write tid 3 to smithi07694878-49 2021-03-15T22:21:45.068 INFO:tasks.rados.rados.0.smithi076.stdout:504: finishing write tid 4 to smithi07694878-49 2021-03-15T22:21:45.068 INFO:tasks.rados.rados.0.smithi076.stdout:503: finishing copy_from to smithi07694878-35 2021-03-15T22:21:45.068 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:21:45.069 INFO:tasks.rados.rados.0.smithi076.stdout:504: finishing write tid 5 to smithi07694878-49 2021-03-15T22:21:45.069 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 257 (ObjNum 210 snap 47 seq_num 210) dirty exists 2021-03-15T22:21:45.069 INFO:tasks.rados.rados.0.smithi076.stdout:504: left oid 49 (ObjNum 210 snap 47 seq_num 210) 2021-03-15T22:21:45.069 INFO:tasks.rados.rados.0.smithi076.stdout:503: finishing copy_from racing read to smithi07694878-35 2021-03-15T22:21:45.069 INFO:tasks.rados.rados.0.smithi076.stdout:505: finishing write tid 1 to smithi07694878-23 2021-03-15T22:21:45.070 INFO:tasks.rados.rados.0.smithi076.stdout:505: finishing write tid 2 to smithi07694878-23 2021-03-15T22:21:45.071 INFO:tasks.rados.rados.0.smithi076.stdout:499: done (5 left) 2021-03-15T22:21:45.071 INFO:tasks.rados.rados.0.smithi076.stdout:500: done (4 left) 2021-03-15T22:21:45.072 INFO:tasks.rados.rados.0.smithi076.stdout:503: done (3 left) 2021-03-15T22:21:45.072 INFO:tasks.rados.rados.0.smithi076.stdout:504: done (2 left) 2021-03-15T22:21:45.072 INFO:tasks.rados.rados.0.smithi076.stdout:506: done (1 left) 2021-03-15T22:21:45.072 INFO:tasks.rados.rados.0.smithi076.stdout:507: delete oid 10 current snap is 47 2021-03-15T22:21:45.073 INFO:tasks.rados.rados.0.smithi076.stdout:505: finishing write tid 3 to smithi07694878-23 2021-03-15T22:21:45.073 INFO:tasks.rados.rados.0.smithi076.stdout:505: finishing write tid 4 to smithi07694878-23 2021-03-15T22:21:45.073 INFO:tasks.rados.rados.0.smithi076.stdout:505: finishing write tid 5 to smithi07694878-23 2021-03-15T22:21:45.073 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 89 (ObjNum 211 snap 47 seq_num 211) dirty exists 2021-03-15T22:21:45.073 INFO:tasks.rados.rados.0.smithi076.stdout:505: left oid 23 (ObjNum 211 snap 47 seq_num 211) 2021-03-15T22:21:45.074 INFO:tasks.rados.rados.0.smithi076.stdout:505: done (1 left) 2021-03-15T22:21:45.074 INFO:tasks.rados.rados.0.smithi076.stdout:507: done (0 left) 2021-03-15T22:21:45.074 INFO:tasks.rados.rados.0.smithi076.stdout:508: copy_from oid 28 from oid 37 current snap is 47 2021-03-15T22:21:45.075 INFO:tasks.rados.rados.0.smithi076.stdout:509: snap_create 2021-03-15T22:21:45.075 INFO:tasks.rados.rados.0.smithi076.stdout:508: finishing copy_from to smithi07694878-28 2021-03-15T22:21:45.075 INFO:tasks.rados.rados.0.smithi076.stdout:508: got expected ENOENT (src dne) 2021-03-15T22:21:45.075 INFO:tasks.rados.rados.0.smithi076.stdout:508: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:21:45.087 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:45.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:45.112 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:45.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:45.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:45.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:45.238 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:45.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:45.321 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 414 (0%) 2021-03-15T22:21:45.322 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:21:45.322 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 11.1327 -> 11.1327 (expected baseline 6.72913) 2021-03-15T22:21:45.322 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 25 -> 25 pgs (0.483092 -> 0.483092 * mean) 2021-03-15T22:21:45.323 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 64 -> 64 pgs (1.23671 -> 1.23671 * mean) 2021-03-15T22:21:45.323 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:21:45.323 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:21:45.323 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 1 2021-03-15T22:21:45.324 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:21:45.324 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0006 2021-03-15T22:21:45.324 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0059 2021-03-15T22:21:45.325 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:21:45.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[62156]: cluster 2021-03-15T22:21:44.070296+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2021-03-15T22:21:45.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[62156]: cluster 2021-03-15T22:21:45.019744+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2021-03-15T22:21:45.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56514]: cluster 2021-03-15T22:21:44.070296+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2021-03-15T22:21:45.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56514]: cluster 2021-03-15T22:21:45.019744+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2021-03-15T22:21:45.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:45.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:45.365 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:45.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:45.466 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:45.489 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:45.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:45.491 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:45.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:45 smithi125 conmon[56739]: cluster 2021-03-15T22:21:44.070296+0000 mon.a (mon.0) 798 : cluster [DBG] osdmap e189: 8 total, 8 up, 8 in 2021-03-15T22:21:45.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:45 smithi125 conmon[56739]: cluster 2021-03-15T22:21:45.019744+0000 mon.a (mon.0) 799 : cluster [DBG] osdmap e190: 8 total, 8 up, 8 in 2021-03-15T22:21:45.592 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:45.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:45.616 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:45.617 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:45.718 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:45.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:45.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:45.826 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:21:45.845 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:45.892 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:45.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:45.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:46.017 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:46.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:46.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:46.072 INFO:tasks.rados.rados.0.smithi076.stdout:508: done (1 left) 2021-03-15T22:21:46.073 INFO:tasks.rados.rados.0.smithi076.stdout:509: done (0 left) 2021-03-15T22:21:46.073 INFO:tasks.rados.rados.0.smithi076.stdout:510: write (excl) oid 26 current snap is 48 2021-03-15T22:21:46.073 INFO:tasks.rados.rados.0.smithi076.stdout:510: seq_num 214 ranges {796698=561170,2006323=545406,3126991=469707} 2021-03-15T22:21:46.081 INFO:tasks.rados.rados.0.smithi076.stdout:510: writing smithi07694878-26 from 796698 to 1357868 tid 1 2021-03-15T22:21:46.088 INFO:tasks.rados.rados.0.smithi076.stdout:510: writing smithi07694878-26 from 2006323 to 2551729 tid 2 2021-03-15T22:21:46.094 INFO:tasks.rados.rados.0.smithi076.stdout:510: writing smithi07694878-26 from 3126991 to 3596698 tid 3 2021-03-15T22:21:46.095 INFO:tasks.rados.rados.0.smithi076.stdout:510: finishing write tid 1 to smithi07694878-26 2021-03-15T22:21:46.096 INFO:tasks.rados.rados.0.smithi076.stdout:510: finishing write tid 2 to smithi07694878-26 2021-03-15T22:21:46.096 INFO:tasks.rados.rados.0.smithi076.stdout:511: read oid 36 snap -1 2021-03-15T22:21:46.096 INFO:tasks.rados.rados.0.smithi076.stdout:511: expect (ObjNum 180 snap 39 seq_num 180) 2021-03-15T22:21:46.096 INFO:tasks.rados.rados.0.smithi076.stdout:512: read oid 8 snap -1 2021-03-15T22:21:46.096 INFO:tasks.rados.rados.0.smithi076.stdout:512: expect (ObjNum 197 snap 43 seq_num 197) 2021-03-15T22:21:46.097 INFO:tasks.rados.rados.0.smithi076.stdout:513: snap_remove snap 45 2021-03-15T22:21:46.116 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: debug 2021-03-15T22:21:45.969+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:21:46.116 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:21:46.116 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.117 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:21:46.117 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.117 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:21:46.117 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.118 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.118 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:21:46.118 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: debug 2021-03-15T22:21:45.969+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:21:46.118 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:21:46.118 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.119 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:21:46.119 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.119 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:21:46.119 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.120 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.120 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:21:46.120 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: debug 2021-03-15T22:21:45.969+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:21:46.121 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:21:46.121 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.121 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:21:46.121 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.121 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:21:46.122 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.122 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.122 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:21:46.123 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: cluster 2021-03-15T22:21:45.000421+0000 mgr.y (mgr.14144) 437 : cluster [DBG] pgmap v411: 138 pgs: 138 active+clean; 133 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 21 MiB/s rd, 4.1 MiB/s wr, 33 op/s 2021-03-15T22:21:46.123 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: audit 2021-03-15T22:21:45.082023+0000 mgr.y (mgr.14144) 438 : audit [DBG] from='client.14937 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:46.123 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: audit 2021-03-15T22:21:45.688544+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:21:46.123 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:21:45.966351+0000 mgr.y (mgr.14144) 439 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:21:46.124 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: audit 2021-03-15T22:21:45.969967+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:21:46.124 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:21:45.970923+0000 mgr.y (mgr.14144) 440 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:21:46.124 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:21:46.124 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.124 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:21:46.125 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.125 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:21:46.125 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.125 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.125 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:21:46.126 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:21:45.971320+0000 mgr.y (mgr.14144) 441 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:21:46.126 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:21:46.126 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.126 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:21:46.127 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.127 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:21:46.127 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.127 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.127 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:21:46.128 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: cluster 2021-03-15T22:21:45.000421+0000 2021-03-15T22:21:46.128 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: mgr.y (mgr.14144) 437 : cluster [DBG] pgmap v411: 138 pgs: 138 active+clean; 133 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 21 MiB/s rd, 4.1 MiB/s wr, 33 op/s 2021-03-15T22:21:46.128 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: audit 2021-03-15T22:21:45.082023+0000 mgr.y (mgr.14144) 438 : audit [DBG] from='client.14937 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:46.129 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: audit 2021-03-15T22:21:45.688544+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:21:46.129 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:21:46.129 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: -15T22:21:45.966351+0000 mgr.y (mgr.14144) 439 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:21:46.129 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:21:46.129 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: -15T22:21:45.969967+0000 mon.a (mon. 2021-03-15T22:21:46.130 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: 0) 801 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:21:46.130 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:21:46.130 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: 2021-03-15T 2021-03-15T22:21:46.130 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: 22:21:45.970923+0000 mgr.y (mgr.14144) 440 : 2021-03-15T22:21:46.131 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:21:46.131 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:21:46.131 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.131 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:21:46.131 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.132 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:21:46.132 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.132 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.132 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:21:46.132 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:21:45.971320+0000 mgr.y (mgr.14144) 441 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:21:46.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:21:46.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:21:46.133 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.134 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:21:46.134 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.134 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.134 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:21:46.135 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:21:45.971674+0000 mgr.y (mgr.14144) 442 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:21:46.136 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:21:46.136 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.136 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:21:46.137 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.137 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:21:46.137 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.137 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.138 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:21:46.138 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:21:45.972016+0000 mgr.y (mgr.14144) 443 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:21:46.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:21:46.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:21:46.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:21:46.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.141 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:21:46.141 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:21:45.972363+0000 mgr.y (mgr.14144) 444 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:21:46.141 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:21:46.141 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.142 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:21:46.142 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.142 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:21:46.143 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.143 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.143 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:21:46.143 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:21:45.972702+0000 mgr.y (mgr.14144) 445 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:21:46.144 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:21:46.145 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: debug 2021-03-15T22:21:45.970+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:21:46.145 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:21:46.146 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.147 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:21:46.147 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.147 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:21:46.148 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.148 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.148 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:21:46.148 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: debug 2021-03-15T22:21:45.970+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:21:46.149 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:21:46.149 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.149 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:21:46.150 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.150 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:21:46.150 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.150 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.151 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:21:46.151 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: debug 2021-03-15T22:21:45.971+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:21:46.151 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:21:46.152 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.152 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:21:46.152 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.153 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:21:46.153 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.153 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.154 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:21:46.154 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: debug 2021-03-15T22:21:45.971+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:21:46.154 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:21:46.155 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.155 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:21:46.155 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.156 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:21:46.156 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.156 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.157 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:21:45 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:21:46.158 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.158 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:21:46.158 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.159 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:21:46.159 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.159 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.159 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:21:46.159 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: cephadm 2021-03-15T22:21:45.973034+0000 mgr.y (mgr.14144) 446 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:21:46.160 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:21:46.160 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.160 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:21:46.160 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.161 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:21:46.161 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.161 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.161 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:21:46.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:21:45.971674+0000 mgr.y (mgr.14144) 442 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:21:46.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:21:46.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:21:46.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:21:46.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:21:46.164 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: cephadm 2021-03-15T22: 2021-03-15T22:21:46.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: 21:45.972016+0000 mgr.y (mgr.14144) 443 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:21:46.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:21:46.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:21:46.165 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.166 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:21:46.166 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.166 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.166 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:21:46.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:21:45.972363+0000 mgr.y (mgr.14144) 444 2021-03-15T22:21:46.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:21:46.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:21:46.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.167 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:21:46.168 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.168 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:21:46.168 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.168 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.168 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:21:46.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: cephadm 2021-03-15T22:21:45.972702+0000 mgr.y (mgr.14144) 445 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:21:46.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:21:46.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:21:46.170 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.170 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:21:46.170 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.170 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.170 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:21:46.171 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: cephadm 2021-03-15T22 2021-03-15T22:21:46.171 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: :21:45.973034+0000 mgr.y (mgr.14144) 446 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:21:46.171 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:21:46.171 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.171 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:21:46.172 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.172 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:21:46.172 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.172 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:46 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:21:46.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:46.195 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:46.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:46.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:46.222 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:46.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:46.316 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:21:46.318 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:21:46.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:21:46.318 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:21:46.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.319 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:255 7.4 7:26337d05:::smithi07694878-20:head [copy-from ver 26] snapc 5=[] ondisk+write+known_if_redirected e103)", 2021-03-15T22:21:46.319 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:18.250702+0000", 2021-03-15T22:21:46.319 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 88.062159528999999, 2021-03-15T22:21:46.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018134056999999999, 2021-03-15T22:21:46.319 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.320 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.320 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.320 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.320 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.321 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 255 2021-03-15T22:21:46.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.321 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250702+0000", 2021-03-15T22:21:46.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250702+0000", 2021-03-15T22:21:46.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5500000000000002e-06 2021-03-15T22:21:46.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250707+0000", 2021-03-15T22:21:46.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8340000000000001e-06 2021-03-15T22:21:46.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250711+0000", 2021-03-15T22:21:46.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2194000000000001e-05 2021-03-15T22:21:46.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250733+0000", 2021-03-15T22:21:46.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4460000000000001e-06 2021-03-15T22:21:46.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250742+0000", 2021-03-15T22:21:46.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6552000000000003e-05 2021-03-15T22:21:46.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250798+0000", 2021-03-15T22:21:46.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7729999999999999e-06 2021-03-15T22:21:46.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:21:46.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250806+0000", 2021-03-15T22:21:46.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034413170000000002 2021-03-15T22:21:46.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.254247+0000", 2021-03-15T22:21:46.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5164999999999998e-05 2021-03-15T22:21:46.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.254302+0000", 2021-03-15T22:21:46.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075848039999999997 2021-03-15T22:21:46.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,4", 2021-03-15T22:21:46.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.261887+0000", 2021-03-15T22:21:46.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060765710000000002 2021-03-15T22:21:46.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.267964+0000", 2021-03-15T22:21:46.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00052121800000000005 2021-03-15T22:21:46.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:21:46.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.268485+0000", 2021-03-15T22:21:46.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026508199999999998 2021-03-15T22:21:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.268750+0000", 2021-03-15T22:21:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4768000000000002e-05 2021-03-15T22:21:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.268795+0000", 2021-03-15T22:21:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0782999999999999e-05 2021-03-15T22:21:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.268836+0000", 2021-03-15T22:21:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018134056999999999 2021-03-15T22:21:46.340 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:256 7.4 7:26337d05:::smithi07694878-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e103)", 2021-03-15T22:21:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:18.250761+0000", 2021-03-15T22:21:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 88.062100240999996, 2021-03-15T22:21:46.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018285802, 2021-03-15T22:21:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:21:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.343 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 256 2021-03-15T22:21:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.344 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250761+0000", 2021-03-15T22:21:46.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250761+0000", 2021-03-15T22:21:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.381e-06 2021-03-15T22:21:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250762+0000", 2021-03-15T22:21:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.894e-06 2021-03-15T22:21:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250764+0000", 2021-03-15T22:21:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4440000000000002e-06 2021-03-15T22:21:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250772+0000", 2021-03-15T22:21:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8600000000000001e-06 2021-03-15T22:21:46.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250776+0000", 2021-03-15T22:21:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4781e-05 2021-03-15T22:21:46.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250861+0000", 2021-03-15T22:21:46.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2269999999999998e-06 2021-03-15T22:21:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:21:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250867+0000", 2021-03-15T22:21:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035623220000000001 2021-03-15T22:21:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.254430+0000", 2021-03-15T22:21:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5142000000000002e-05 2021-03-15T22:21:46.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.254475+0000", 2021-03-15T22:21:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010549012 2021-03-15T22:21:46.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.265024+0000", 2021-03-15T22:21:46.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6784999999999999e-05 2021-03-15T22:21:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:46.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.265081+0000", 2021-03-15T22:21:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0037962320000000001 2021-03-15T22:21:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.268877+0000", 2021-03-15T22:21:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8145000000000005e-05 2021-03-15T22:21:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.268955+0000", 2021-03-15T22:21:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1576999999999994e-05 2021-03-15T22:21:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.269047+0000", 2021-03-15T22:21:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018285802 2021-03-15T22:21:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.362 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.363 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:260 7.6 7:63693c38:::smithi07694878-22:head [copy-from ver 21] snapc 8=[8] ondisk+write+known_if_redirected e104)", 2021-03-15T22:21:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:19.188718+0000", 2021-03-15T22:21:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.124142845999998, 2021-03-15T22:21:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013718614000000001, 2021-03-15T22:21:46.364 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.365 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 260 2021-03-15T22:21:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188718+0000", 2021-03-15T22:21:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188718+0000", 2021-03-15T22:21:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.383e-06 2021-03-15T22:21:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188724+0000", 2021-03-15T22:21:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4479999999999996e-06 2021-03-15T22:21:46.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188728+0000", 2021-03-15T22:21:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3241999999999999e-05 2021-03-15T22:21:46.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188751+0000", 2021-03-15T22:21:46.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5650000000000002e-06 2021-03-15T22:21:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188760+0000", 2021-03-15T22:21:46.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6980999999999999e-05 2021-03-15T22:21:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188857+0000", 2021-03-15T22:21:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0844e-05 2021-03-15T22:21:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:21:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188868+0000", 2021-03-15T22:21:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002419128 2021-03-15T22:21:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.191287+0000", 2021-03-15T22:21:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9197999999999995e-05 2021-03-15T22:21:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.191376+0000", 2021-03-15T22:21:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038822370000000002 2021-03-15T22:21:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:21:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.195258+0000", 2021-03-15T22:21:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0042875389999999999 2021-03-15T22:21:46.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:21:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.199546+0000", 2021-03-15T22:21:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001117807 2021-03-15T22:21:46.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.200664+0000", 2021-03-15T22:21:46.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016892369999999999 2021-03-15T22:21:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.202353+0000", 2021-03-15T22:21:46.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0448999999999998e-05 2021-03-15T22:21:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.202393+0000", 2021-03-15T22:21:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3556000000000001e-05 2021-03-15T22:21:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.202437+0000", 2021-03-15T22:21:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013718614000000001 2021-03-15T22:21:46.384 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:261 7.6 7:63693c38:::smithi07694878-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e104)", 2021-03-15T22:21:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:19.188775+0000", 2021-03-15T22:21:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.124086496999993, 2021-03-15T22:21:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013796008, 2021-03-15T22:21:46.386 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:21:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.387 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 261 2021-03-15T22:21:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188775+0000", 2021-03-15T22:21:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188775+0000", 2021-03-15T22:21:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2890000000000001e-06 2021-03-15T22:21:46.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188776+0000", 2021-03-15T22:21:46.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.522e-06 2021-03-15T22:21:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188777+0000", 2021-03-15T22:21:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0870000000000001e-06 2021-03-15T22:21:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188784+0000", 2021-03-15T22:21:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3590000000000001e-06 2021-03-15T22:21:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188789+0000", 2021-03-15T22:21:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000161612 2021-03-15T22:21:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188950+0000", 2021-03-15T22:21:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3630000000000004e-06 2021-03-15T22:21:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:21:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.188955+0000", 2021-03-15T22:21:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002553844 2021-03-15T22:21:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.191509+0000", 2021-03-15T22:21:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9162e-05 2021-03-15T22:21:46.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.191538+0000", 2021-03-15T22:21:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067428829999999999 2021-03-15T22:21:46.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.198281+0000", 2021-03-15T22:21:46.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4543999999999997e-05 2021-03-15T22:21:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:46.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.198345+0000", 2021-03-15T22:21:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004115222 2021-03-15T22:21:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.202460+0000", 2021-03-15T22:21:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6681999999999998e-05 2021-03-15T22:21:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.202507+0000", 2021-03-15T22:21:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3439e-05 2021-03-15T22:21:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.202571+0000", 2021-03-15T22:21:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013796008 2021-03-15T22:21:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.405 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.406 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:321 7.6 7:6283974b:::smithi07694878-30:head [copy-from ver 9] snapc 8=[8] ondisk+write+known_if_redirected e104)", 2021-03-15T22:21:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:19.443648+0000", 2021-03-15T22:21:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.869213212999995, 2021-03-15T22:21:46.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014592526, 2021-03-15T22:21:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.408 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 321 2021-03-15T22:21:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.409 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443648+0000", 2021-03-15T22:21:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443648+0000", 2021-03-15T22:21:46.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1179999999999999e-06 2021-03-15T22:21:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443650+0000", 2021-03-15T22:21:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0169999999999998e-06 2021-03-15T22:21:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443654+0000", 2021-03-15T22:21:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1824000000000001e-05 2021-03-15T22:21:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443666+0000", 2021-03-15T22:21:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7679999999999999e-06 2021-03-15T22:21:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443671+0000", 2021-03-15T22:21:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8227999999999999e-05 2021-03-15T22:21:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443729+0000", 2021-03-15T22:21:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2294999999999996e-05 2021-03-15T22:21:46.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443821+0000", 2021-03-15T22:21:46.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039947050000000003 2021-03-15T22:21:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:21:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.447816+0000", 2021-03-15T22:21:46.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0046883640000000004 2021-03-15T22:21:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:21:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.452504+0000", 2021-03-15T22:21:46.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000427339 2021-03-15T22:21:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.452931+0000", 2021-03-15T22:21:46.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005203231 2021-03-15T22:21:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.458135+0000", 2021-03-15T22:21:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7593000000000003e-05 2021-03-15T22:21:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.458182+0000", 2021-03-15T22:21:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8044000000000002e-05 2021-03-15T22:21:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.458240+0000", 2021-03-15T22:21:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014592526 2021-03-15T22:21:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.426 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.427 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:322 7.6 7:6283974b:::smithi07694878-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e104)", 2021-03-15T22:21:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:19.443682+0000", 2021-03-15T22:21:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 86.869179095000007, 2021-03-15T22:21:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014715953, 2021-03-15T22:21:46.428 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:21:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.429 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 322 2021-03-15T22:21:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443682+0000", 2021-03-15T22:21:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443682+0000", 2021-03-15T22:21:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.249e-06 2021-03-15T22:21:46.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443683+0000", 2021-03-15T22:21:46.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.392e-06 2021-03-15T22:21:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443685+0000", 2021-03-15T22:21:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4587999999999999e-05 2021-03-15T22:21:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443709+0000", 2021-03-15T22:21:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8559999999999998e-06 2021-03-15T22:21:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443714+0000", 2021-03-15T22:21:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024507799999999997 2021-03-15T22:21:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443959+0000", 2021-03-15T22:21:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2342999999999998e-05 2021-03-15T22:21:46.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.444001+0000", 2021-03-15T22:21:46.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077478909999999998 2021-03-15T22:21:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.451749+0000", 2021-03-15T22:21:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4707000000000002e-05 2021-03-15T22:21:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.451814+0000", 2021-03-15T22:21:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006467446 2021-03-15T22:21:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.458282+0000", 2021-03-15T22:21:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0368000000000002e-05 2021-03-15T22:21:46.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.458342+0000", 2021-03-15T22:21:46.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6035000000000001e-05 2021-03-15T22:21:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.458398+0000", 2021-03-15T22:21:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014715953 2021-03-15T22:21:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.446 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:344 7.6 7:6283974b:::smithi07694878-30:head [copy-from ver 24] snapc a=[a,9,8] ondisk+write+known_if_redirected e106)", 2021-03-15T22:21:46.447 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:21.015806+0000", 2021-03-15T22:21:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 85.297054884999994, 2021-03-15T22:21:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016163374000000001, 2021-03-15T22:21:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.448 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 344 2021-03-15T22:21:46.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015806+0000", 2021-03-15T22:21:46.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015806+0000", 2021-03-15T22:21:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0839999999999997e-06 2021-03-15T22:21:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015812+0000", 2021-03-15T22:21:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1999999999999996e-06 2021-03-15T22:21:46.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015816+0000", 2021-03-15T22:21:46.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1724000000000001e-05 2021-03-15T22:21:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015838+0000", 2021-03-15T22:21:46.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7160000000000003e-06 2021-03-15T22:21:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015846+0000", 2021-03-15T22:21:46.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8777999999999998e-05 2021-03-15T22:21:46.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015915+0000", 2021-03-15T22:21:46.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000103766 2021-03-15T22:21:46.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.016018+0000", 2021-03-15T22:21:46.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048164510000000002 2021-03-15T22:21:46.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:21:46.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.020835+0000", 2021-03-15T22:21:46.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078504999999999998 2021-03-15T22:21:46.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.028685+0000", 2021-03-15T22:21:46.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4822e-05 2021-03-15T22:21:46.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:21:46.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.028700+0000", 2021-03-15T22:21:46.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031992040000000002 2021-03-15T22:21:46.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.031899+0000", 2021-03-15T22:21:46.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2533999999999997e-05 2021-03-15T22:21:46.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.031932+0000", 2021-03-15T22:21:46.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7595000000000002e-05 2021-03-15T22:21:46.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.031970+0000", 2021-03-15T22:21:46.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016163374000000001 2021-03-15T22:21:46.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.466 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.467 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.467 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:345 7.6 7:6283974b:::smithi07694878-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e106)", 2021-03-15T22:21:46.468 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:21.015859+0000", 2021-03-15T22:21:46.468 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 85.297002222000003, 2021-03-15T22:21:46.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016254362000000001, 2021-03-15T22:21:46.468 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.468 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:21:46.469 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.469 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.469 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.469 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 345 2021-03-15T22:21:46.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.470 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015859+0000", 2021-03-15T22:21:46.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015859+0000", 2021-03-15T22:21:46.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.254e-06 2021-03-15T22:21:46.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015860+0000", 2021-03-15T22:21:46.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.618e-06 2021-03-15T22:21:46.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015862+0000", 2021-03-15T22:21:46.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.312e-06 2021-03-15T22:21:46.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015871+0000", 2021-03-15T22:21:46.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.288e-06 2021-03-15T22:21:46.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.015876+0000", 2021-03-15T22:21:46.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033248899999999999 2021-03-15T22:21:46.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.016209+0000", 2021-03-15T22:21:46.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9957999999999998e-05 2021-03-15T22:21:46.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:46.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.016269+0000", 2021-03-15T22:21:46.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084883059999999993 2021-03-15T22:21:46.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.024757+0000", 2021-03-15T22:21:46.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1354999999999995e-05 2021-03-15T22:21:46.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:46.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.024848+0000", 2021-03-15T22:21:46.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071526460000000004 2021-03-15T22:21:46.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.032001+0000", 2021-03-15T22:21:46.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6621e-05 2021-03-15T22:21:46.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.032048+0000", 2021-03-15T22:21:46.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5514999999999999e-05 2021-03-15T22:21:46.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.032113+0000", 2021-03-15T22:21:46.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016254362000000001 2021-03-15T22:21:46.485 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.485 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.486 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.486 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:346 7.6 7:608078b0:::smithi07694878-12:head [write 675648~546760] snapc a=[a,9,8] ondisk+write+known_if_redirected e106)", 2021-03-15T22:21:46.487 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:21.022802+0000", 2021-03-15T22:21:46.487 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 85.290059534999997, 2021-03-15T22:21:46.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093995009999999993, 2021-03-15T22:21:46.487 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.487 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.488 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 346 2021-03-15T22:21:46.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.489 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.022802+0000", 2021-03-15T22:21:46.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.022802+0000", 2021-03-15T22:21:46.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000281871 2021-03-15T22:21:46.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.023083+0000", 2021-03-15T22:21:46.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4600000000000002e-06 2021-03-15T22:21:46.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.023089+0000", 2021-03-15T22:21:46.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9238e-05 2021-03-15T22:21:46.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.023118+0000", 2021-03-15T22:21:46.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0264e-05 2021-03-15T22:21:46.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.023128+0000", 2021-03-15T22:21:46.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017556519999999999 2021-03-15T22:21:46.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.024884+0000", 2021-03-15T22:21:46.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4562999999999996e-05 2021-03-15T22:21:46.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.024979+0000", 2021-03-15T22:21:46.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000225398 2021-03-15T22:21:46.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:21:46.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.025204+0000", 2021-03-15T22:21:46.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004552164 2021-03-15T22:21:46.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.029756+0000", 2021-03-15T22:21:46.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023551930000000002 2021-03-15T22:21:46.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:21:46.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.032111+0000", 2021-03-15T22:21:46.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0448999999999999e-05 2021-03-15T22:21:46.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.032142+0000", 2021-03-15T22:21:46.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0654000000000002e-05 2021-03-15T22:21:46.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.032172+0000", 2021-03-15T22:21:46.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8595e-05 2021-03-15T22:21:46.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.032201+0000", 2021-03-15T22:21:46.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093995009999999993 2021-03-15T22:21:46.507 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.507 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.507 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.508 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:350 7.7 e106/91 7:ecc0cb20:::smithi07694878-25:head v 106'22, mlcod=105'19)", 2021-03-15T22:21:46.508 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:21.035878+0000", 2021-03-15T22:21:46.508 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 85.276982571000005, 2021-03-15T22:21:46.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012613607000000001, 2021-03-15T22:21:46.509 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.509 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.509 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.035878+0000", 2021-03-15T22:21:46.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.035878+0000", 2021-03-15T22:21:46.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056062330000000004 2021-03-15T22:21:46.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.041485+0000", 2021-03-15T22:21:46.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8489999999999999e-06 2021-03-15T22:21:46.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.041489+0000", 2021-03-15T22:21:46.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5766000000000001e-05 2021-03-15T22:21:46.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.041504+0000", 2021-03-15T22:21:46.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4700000000000001e-06 2021-03-15T22:21:46.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.041510+0000", 2021-03-15T22:21:46.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9842000000000002e-05 2021-03-15T22:21:46.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.041550+0000", 2021-03-15T22:21:46.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5928999999999999e-05 2021-03-15T22:21:46.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.041566+0000", 2021-03-15T22:21:46.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0068974589999999999 2021-03-15T22:21:46.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.048463+0000", 2021-03-15T22:21:46.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9059000000000001e-05 2021-03-15T22:21:46.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.048492+0000", 2021-03-15T22:21:46.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012613607000000001 2021-03-15T22:21:46.525 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.526 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.526 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.528 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:542 7.6 7:6de8b070:::smithi07694878-15:head [copy-from ver 3] snapc 1b=[1b,19,17] ondisk+write+known_if_redirected e124)", 2021-03-15T22:21:46.528 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:39.097751+0000", 2021-03-15T22:21:46.529 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.215110022999994, 2021-03-15T22:21:46.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011118423000000001, 2021-03-15T22:21:46.529 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.530 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.530 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.530 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.530 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.531 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 542 2021-03-15T22:21:46.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.531 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097751+0000", 2021-03-15T22:21:46.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097751+0000", 2021-03-15T22:21:46.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8380000000000004e-06 2021-03-15T22:21:46.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097755+0000", 2021-03-15T22:21:46.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.253e-06 2021-03-15T22:21:46.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.535 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097758+0000", 2021-03-15T22:21:46.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2435e-05 2021-03-15T22:21:46.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097771+0000", 2021-03-15T22:21:46.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2220000000000001e-06 2021-03-15T22:21:46.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097776+0000", 2021-03-15T22:21:46.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9986e-05 2021-03-15T22:21:46.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097816+0000", 2021-03-15T22:21:46.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8435999999999993e-05 2021-03-15T22:21:46.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097884+0000", 2021-03-15T22:21:46.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064682309999999996 2021-03-15T22:21:46.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:21:46.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.104352+0000", 2021-03-15T22:21:46.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040355640000000002 2021-03-15T22:21:46.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:21:46.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.108388+0000", 2021-03-15T22:21:46.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026528499999999998 2021-03-15T22:21:46.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.108653+0000", 2021-03-15T22:21:46.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012215500000000001 2021-03-15T22:21:46.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.108775+0000", 2021-03-15T22:21:46.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1476000000000001e-05 2021-03-15T22:21:46.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.108817+0000", 2021-03-15T22:21:46.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2541999999999999e-05 2021-03-15T22:21:46.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.108869+0000", 2021-03-15T22:21:46.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011118423000000001 2021-03-15T22:21:46.552 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.553 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.553 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.554 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:543 7.6 7:6de8b070:::smithi07694878-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e124)", 2021-03-15T22:21:46.554 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:39.097796+0000", 2021-03-15T22:21:46.555 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.215065065999994, 2021-03-15T22:21:46.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011237281, 2021-03-15T22:21:46.555 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.555 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:21:46.556 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.556 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.556 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.556 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 543 2021-03-15T22:21:46.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.557 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097796+0000", 2021-03-15T22:21:46.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097796+0000", 2021-03-15T22:21:46.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7409999999999999e-06 2021-03-15T22:21:46.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097798+0000", 2021-03-15T22:21:46.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.001e-06 2021-03-15T22:21:46.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097800+0000", 2021-03-15T22:21:46.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.029e-06 2021-03-15T22:21:46.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.563 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.563 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097807+0000", 2021-03-15T22:21:46.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4599999999999996e-06 2021-03-15T22:21:46.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.564 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.564 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097811+0000", 2021-03-15T22:21:46.564 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000184694 2021-03-15T22:21:46.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.565 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.097996+0000", 2021-03-15T22:21:46.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8198000000000003e-05 2021-03-15T22:21:46.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:46.567 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.098034+0000", 2021-03-15T22:21:46.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093328070000000003 2021-03-15T22:21:46.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.107367+0000", 2021-03-15T22:21:46.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110036 2021-03-15T22:21:46.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.569 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:46.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.107477+0000", 2021-03-15T22:21:46.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001435076 2021-03-15T22:21:46.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.570 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.108912+0000", 2021-03-15T22:21:46.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2460999999999998e-05 2021-03-15T22:21:46.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.108964+0000", 2021-03-15T22:21:46.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8777999999999998e-05 2021-03-15T22:21:46.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:39.109033+0000", 2021-03-15T22:21:46.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011237281 2021-03-15T22:21:46.573 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.573 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.574 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.574 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:868 7.6 7:63693c38:::smithi07694878-22:head [copy-from ver 127] snapc 35=[34,30] ondisk+write+known_if_redirected e152)", 2021-03-15T22:21:46.575 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:07.027861+0000", 2021-03-15T22:21:46.575 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 39.285000470999996, 2021-03-15T22:21:46.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016322204, 2021-03-15T22:21:46.576 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.576 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.576 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.577 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.577 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.577 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 868 2021-03-15T22:21:46.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.578 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027861+0000", 2021-03-15T22:21:46.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.581 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.581 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027861+0000", 2021-03-15T22:21:46.581 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4750000000000001e-06 2021-03-15T22:21:46.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.582 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.582 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.583 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027862+0000", 2021-03-15T22:21:46.583 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0100000000000003e-07 2021-03-15T22:21:46.583 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.584 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.584 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027863+0000", 2021-03-15T22:21:46.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.36e-06 2021-03-15T22:21:46.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.586 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.586 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027866+0000", 2021-03-15T22:21:46.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0140000000000001e-06 2021-03-15T22:21:46.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027868+0000", 2021-03-15T22:21:46.589 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4653e-05 2021-03-15T22:21:46.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.589 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027923+0000", 2021-03-15T22:21:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6051999999999994e-05 2021-03-15T22:21:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.028019+0000", 2021-03-15T22:21:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039365119999999997 2021-03-15T22:21:46.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:21:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.031956+0000", 2021-03-15T22:21:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092519019999999993 2021-03-15T22:21:46.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.041207+0000", 2021-03-15T22:21:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002453804 2021-03-15T22:21:46.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:21:46.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.043661+0000", 2021-03-15T22:21:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037677199999999999 2021-03-15T22:21:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.044038+0000", 2021-03-15T22:21:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5184999999999993e-05 2021-03-15T22:21:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.044103+0000", 2021-03-15T22:21:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9573999999999994e-05 2021-03-15T22:21:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.044183+0000", 2021-03-15T22:21:46.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016322204 2021-03-15T22:21:46.601 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.601 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.601 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.602 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:869 7.6 7:63693c38:::smithi07694878-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e152)", 2021-03-15T22:21:46.602 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:07.027877+0000", 2021-03-15T22:21:46.603 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 39.284984516999998, 2021-03-15T22:21:46.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016550925000000001, 2021-03-15T22:21:46.603 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.603 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:21:46.603 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.604 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.604 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.604 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 869 2021-03-15T22:21:46.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.605 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027877+0000", 2021-03-15T22:21:46.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027877+0000", 2021-03-15T22:21:46.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7000000000000003e-07 2021-03-15T22:21:46.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027877+0000", 2021-03-15T22:21:46.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0100000000000001e-06 2021-03-15T22:21:46.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027878+0000", 2021-03-15T22:21:46.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3778e-05 2021-03-15T22:21:46.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027922+0000", 2021-03-15T22:21:46.610 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7060000000000003e-06 2021-03-15T22:21:46.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.611 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.611 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.027927+0000", 2021-03-15T22:21:46.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067313500000000005 2021-03-15T22:21:46.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.028600+0000", 2021-03-15T22:21:46.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2176999999999999e-05 2021-03-15T22:21:46.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:46.613 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.028652+0000", 2021-03-15T22:21:46.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089212289999999993 2021-03-15T22:21:46.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.037573+0000", 2021-03-15T22:21:46.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8059000000000005e-05 2021-03-15T22:21:46.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.615 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:46.616 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.037661+0000", 2021-03-15T22:21:46.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065709399999999999 2021-03-15T22:21:46.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.616 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.044232+0000", 2021-03-15T22:21:46.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3751000000000004e-05 2021-03-15T22:21:46.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.044326+0000", 2021-03-15T22:21:46.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010127 2021-03-15T22:21:46.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.619 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.044427+0000", 2021-03-15T22:21:46.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016550925000000001 2021-03-15T22:21:46.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.619 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.620 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.620 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1191 7.6 7:61e0cc05:::smithi07694878-16:head [copy-from ver 101] snapc 4e=[4e,4d,3f] ondisk+write+known_if_redirected e175)", 2021-03-15T22:21:46.620 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:21:46.621 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.245120651000001, 2021-03-15T22:21:46.621 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.97834980800000004, 2021-03-15T22:21:46.621 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.621 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.622 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.622 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.622 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.622 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1191 2021-03-15T22:21:46.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.623 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.623 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:21:46.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:21:46.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0440000000000003e-06 2021-03-15T22:21:46.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067745+0000", 2021-03-15T22:21:46.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4340000000000001e-06 2021-03-15T22:21:46.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067749+0000", 2021-03-15T22:21:46.627 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9241000000000001e-05 2021-03-15T22:21:46.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.628 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.628 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067768+0000", 2021-03-15T22:21:46.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3010000000000007e-06 2021-03-15T22:21:46.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067776+0000", 2021-03-15T22:21:46.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5014e-05 2021-03-15T22:21:46.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067821+0000", 2021-03-15T22:21:46.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9779999999999994e-06 2021-03-15T22:21:46.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:21:46.632 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067830+0000", 2021-03-15T22:21:46.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.95797833899999996 2021-03-15T22:21:46.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.632 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.633 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.025809+0000", 2021-03-15T22:21:46.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138412 2021-03-15T22:21:46.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.633 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.025947+0000", 2021-03-15T22:21:46.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006355744 2021-03-15T22:21:46.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,7", 2021-03-15T22:21:46.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.032303+0000", 2021-03-15T22:21:46.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043109439999999997 2021-03-15T22:21:46.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:21:46.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.036614+0000", 2021-03-15T22:21:46.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092878839999999997 2021-03-15T22:21:46.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.637 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.637 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.045902+0000", 2021-03-15T22:21:46.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8556999999999999e-05 2021-03-15T22:21:46.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.638 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.638 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.045990+0000", 2021-03-15T22:21:46.638 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6665999999999997e-05 2021-03-15T22:21:46.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.639 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.639 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046037+0000", 2021-03-15T22:21:46.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.325e-05 2021-03-15T22:21:46.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046090+0000", 2021-03-15T22:21:46.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.97834980800000004 2021-03-15T22:21:46.641 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.641 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.641 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.642 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1192 7.6 7:61e0cc05:::smithi07694878-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e175)", 2021-03-15T22:21:46.642 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:21:46.642 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.245071331999998, 2021-03-15T22:21:46.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.978448604, 2021-03-15T22:21:46.642 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.643 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:21:46.643 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.643 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.643 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.644 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1192 2021-03-15T22:21:46.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.644 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.646 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:21:46.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:21:46.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.401e-06 2021-03-15T22:21:46.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067791+0000", 2021-03-15T22:21:46.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.083e-06 2021-03-15T22:21:46.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067794+0000", 2021-03-15T22:21:46.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4919999999999996e-06 2021-03-15T22:21:46.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067802+0000", 2021-03-15T22:21:46.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.383e-06 2021-03-15T22:21:46.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067807+0000", 2021-03-15T22:21:46.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8674000000000002e-05 2021-03-15T22:21:46.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067856+0000", 2021-03-15T22:21:46.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5890000000000003e-06 2021-03-15T22:21:46.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:21:46.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067864+0000", 2021-03-15T22:21:46.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.95935619400000005 2021-03-15T22:21:46.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.027221+0000", 2021-03-15T22:21:46.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9426999999999998e-05 2021-03-15T22:21:46.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:46.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.027290+0000", 2021-03-15T22:21:46.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085250369999999992 2021-03-15T22:21:46.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.035815+0000", 2021-03-15T22:21:46.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9997999999999997e-05 2021-03-15T22:21:46.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:46.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.035875+0000", 2021-03-15T22:21:46.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010246089 2021-03-15T22:21:46.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046121+0000", 2021-03-15T22:21:46.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3338999999999998e-05 2021-03-15T22:21:46.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046184+0000", 2021-03-15T22:21:46.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3897999999999998e-05 2021-03-15T22:21:46.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046238+0000", 2021-03-15T22:21:46.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.978448604 2021-03-15T22:21:46.664 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.664 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.664 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.665 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1202 7.f 7:fc7cd984:::smithi07694878-4:head [copy-from ver 234] snapc 4e=[4e,4d,3f] ondisk+write+known_if_redirected e175)", 2021-03-15T22:21:46.665 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.099464+0000", 2021-03-15T22:21:46.665 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.213396830000001, 2021-03-15T22:21:46.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012678567, 2021-03-15T22:21:46.666 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.666 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.666 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.666 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.667 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.667 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1202 2021-03-15T22:21:46.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.667 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099464+0000", 2021-03-15T22:21:46.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099464+0000", 2021-03-15T22:21:46.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.116e-06 2021-03-15T22:21:46.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099465+0000", 2021-03-15T22:21:46.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4700000000000001e-07 2021-03-15T22:21:46.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099466+0000", 2021-03-15T22:21:46.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2629999999999999e-06 2021-03-15T22:21:46.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099469+0000", 2021-03-15T22:21:46.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1739999999999999e-06 2021-03-15T22:21:46.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099472+0000", 2021-03-15T22:21:46.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011399629999999999 2021-03-15T22:21:46.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.100611+0000", 2021-03-15T22:21:46.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5240000000000006e-05 2021-03-15T22:21:46.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.100677+0000", 2021-03-15T22:21:46.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053840700000000003 2021-03-15T22:21:46.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:21:46.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.106061+0000", 2021-03-15T22:21:46.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047980050000000002 2021-03-15T22:21:46.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.110859+0000", 2021-03-15T22:21:46.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00094032100000000004 2021-03-15T22:21:46.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:21:46.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.111799+0000", 2021-03-15T22:21:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025784400000000002 2021-03-15T22:21:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.112057+0000", 2021-03-15T22:21:46.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5563999999999999e-05 2021-03-15T22:21:46.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.112103+0000", 2021-03-15T22:21:46.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0259999999999997e-05 2021-03-15T22:21:46.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.112143+0000", 2021-03-15T22:21:46.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012678567 2021-03-15T22:21:46.684 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.684 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.684 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.685 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1203 7.f 7:fc7cd984:::smithi07694878-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e175)", 2021-03-15T22:21:46.685 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.099474+0000", 2021-03-15T22:21:46.686 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.213387408999999, 2021-03-15T22:21:46.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012795605999999999, 2021-03-15T22:21:46.686 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.686 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:21:46.686 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.687 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.687 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.687 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1203 2021-03-15T22:21:46.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.688 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099474+0000", 2021-03-15T22:21:46.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099474+0000", 2021-03-15T22:21:46.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3000000000000001e-07 2021-03-15T22:21:46.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099474+0000", 2021-03-15T22:21:46.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9599999999999999e-07 2021-03-15T22:21:46.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099475+0000", 2021-03-15T22:21:46.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.089e-06 2021-03-15T22:21:46.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099478+0000", 2021-03-15T22:21:46.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.95e-06 2021-03-15T22:21:46.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.099480+0000", 2021-03-15T22:21:46.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001334011 2021-03-15T22:21:46.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.100814+0000", 2021-03-15T22:21:46.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3042e-05 2021-03-15T22:21:46.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:21:46.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.100847+0000", 2021-03-15T22:21:46.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077078240000000003 2021-03-15T22:21:46.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.108555+0000", 2021-03-15T22:21:46.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4970999999999997e-05 2021-03-15T22:21:46.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:21:46.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.108640+0000", 2021-03-15T22:21:46.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035253350000000001 2021-03-15T22:21:46.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.112165+0000", 2021-03-15T22:21:46.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6783999999999998e-05 2021-03-15T22:21:46.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.112212+0000", 2021-03-15T22:21:46.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7374000000000003e-05 2021-03-15T22:21:46.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.112269+0000", 2021-03-15T22:21:46.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012795605999999999 2021-03-15T22:21:46.703 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.703 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.703 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.704 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:1215 7.7 e179/172 7:ed4a01d2:::smithi07694878-2:head v 179'52, mlcod=173'49)", 2021-03-15T22:21:46.704 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:34.045926+0000", 2021-03-15T22:21:46.704 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.266934711999999, 2021-03-15T22:21:46.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012963952000000001, 2021-03-15T22:21:46.705 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.705 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.705 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.045926+0000", 2021-03-15T22:21:46.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.045926+0000", 2021-03-15T22:21:46.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053973620000000002 2021-03-15T22:21:46.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051324+0000", 2021-03-15T22:21:46.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2089999999999999e-06 2021-03-15T22:21:46.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051331+0000", 2021-03-15T22:21:46.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7379e-05 2021-03-15T22:21:46.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051358+0000", 2021-03-15T22:21:46.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1982e-05 2021-03-15T22:21:46.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051370+0000", 2021-03-15T22:21:46.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0495999999999998e-05 2021-03-15T22:21:46.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051431+0000", 2021-03-15T22:21:46.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4309999999999999e-05 2021-03-15T22:21:46.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051455+0000", 2021-03-15T22:21:46.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073871450000000003 2021-03-15T22:21:46.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.058842+0000", 2021-03-15T22:21:46.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8068999999999997e-05 2021-03-15T22:21:46.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.058890+0000", 2021-03-15T22:21:46.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012963952000000001 2021-03-15T22:21:46.717 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.717 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.717 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.718 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1380 7.6 7:6a64397b:::smithi07694878-49:head [write 729470~543821] snapc 5b=[5a,56] ondisk+write+known_if_redirected e190)", 2021-03-15T22:21:46.718 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:45.031013+0000", 2021-03-15T22:21:46.718 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.2818478310000001, 2021-03-15T22:21:46.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012472901, 2021-03-15T22:21:46.719 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:21:46.719 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:21:46.719 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:21:46.719 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:21:46.720 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:21:46.720 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1380 2021-03-15T22:21:46.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.720 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:21:46.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:21:46.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.031013+0000", 2021-03-15T22:21:46.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:21:46.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:21:46.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.031013+0000", 2021-03-15T22:21:46.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000230499 2021-03-15T22:21:46.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:21:46.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.031244+0000", 2021-03-15T22:21:46.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7519999999999998e-06 2021-03-15T22:21:46.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:21:46.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.031247+0000", 2021-03-15T22:21:46.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0707999999999999e-05 2021-03-15T22:21:46.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:21:46.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.031268+0000", 2021-03-15T22:21:46.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.893e-06 2021-03-15T22:21:46.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:21:46.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.031278+0000", 2021-03-15T22:21:46.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0183999999999998e-05 2021-03-15T22:21:46.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.031338+0000", 2021-03-15T22:21:46.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5079999999999999e-06 2021-03-15T22:21:46.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:21:46.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.031346+0000", 2021-03-15T22:21:46.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079167769999999998 2021-03-15T22:21:46.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:21:46.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.039263+0000", 2021-03-15T22:21:46.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011364100000000001 2021-03-15T22:21:46.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:21:46.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.039376+0000", 2021-03-15T22:21:46.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025580500000000001 2021-03-15T22:21:46.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,7", 2021-03-15T22:21:46.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.039632+0000", 2021-03-15T22:21:46.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036047509999999998 2021-03-15T22:21:46.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:21:46.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.043237+0000", 2021-03-15T22:21:46.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6051000000000003e-05 2021-03-15T22:21:46.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.043293+0000", 2021-03-15T22:21:46.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4086999999999998e-05 2021-03-15T22:21:46.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:21:46.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.043357+0000", 2021-03-15T22:21:46.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9528999999999999e-05 2021-03-15T22:21:46.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:21:46.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.043416+0000", 2021-03-15T22:21:46.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9715999999999995e-05 2021-03-15T22:21:46.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:21:46.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:21:46.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:21:46.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.043486+0000", 2021-03-15T22:21:46.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012472901 2021-03-15T22:21:46.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.739 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:21:46.739 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:21:46.740 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:21:46.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:46.742 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:46.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:46.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: cluster 2021-03-15T22:21:45.000421+0000 mgr.y (mgr.14144) 437 : cluster [DBG] pgmap v411: 138 pgs: 138 active+clean; 133 MiB data, 430 MiB used, 715 GiB / 715 GiB avail; 21 MiB/s rd, 4.1 MiB/s wr, 33 op/s 2021-03-15T22:21:46.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: audit 2021-03-15T22:21:45.082023+0000 mgr.y (mgr.14144) 438 : audit [DBG] from='client.14937 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:21:46.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: audit 2021-03-15T22:21:45.688544+0000 mon.a (mon.0) 800 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:21:46.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:21:45.966351+0000 mgr.y (mgr.14144) 439 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:21:46.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: audit 2021-03-15T22:21:45.969967+0000 mon.a (mon.0) 801 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:21:46.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:21:45.970923+0000 mgr.y (mgr.14144) 440 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:21:46.745 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:21:46.745 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.745 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:21:46.745 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:21:46.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:21:46.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:21:45.971320+0000 mgr.y (mgr.14144) 441 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:21:46.747 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:21:46.747 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.747 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:21:46.747 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.748 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:21:46.748 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.748 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.748 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:21:46.748 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:21:45.971674+0000 mgr.y (mgr.14144) 442 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:21:46.749 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:21:46.749 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.749 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:21:46.749 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.750 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:21:46.750 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.750 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.750 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:21:46.751 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:21:45.972016+0000 mgr.y (mgr.14144) 443 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:21:46.751 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:21:46.751 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.751 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:21:46.751 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.752 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:21:46.752 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.752 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.752 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:21:46.753 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:21:45.972363+0000 mgr.y (mgr.14144) 444 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:21:46.753 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:21:46.753 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.753 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:21:46.753 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.754 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:21:46.754 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.754 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.754 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:21:46.755 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:21:45.972702+0000 mgr.y (mgr.14144) 445 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:21:46.755 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:21:46.755 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.755 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:21:46.755 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.756 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:21:46.756 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.756 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.756 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:21:46.757 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: cephadm 2021-03-15T22:21:45.973034+0000 mgr.y (mgr.14144) 446 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:21:46.757 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:21:46.757 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:21:46.757 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:21:46.757 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:21:46.758 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:21:46.758 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:21:46.758 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:21:46.758 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:46 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:21:46.843 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:46.870 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:46.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:46.972 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:46.993 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:46.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:46.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:47.086 INFO:tasks.rados.rados.0.smithi076.stdout:513: done (3 left) 2021-03-15T22:21:47.086 INFO:tasks.rados.rados.0.smithi076.stdout:514: read oid 9 snap 47 2021-03-15T22:21:47.086 INFO:tasks.rados.rados.0.smithi076.stdout:514: expect (ObjNum 129 snap 24 seq_num 129) 2021-03-15T22:21:47.086 INFO:tasks.rados.rados.0.smithi076.stdout:515: snap_remove snap 42 2021-03-15T22:21:47.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:47.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:47.122 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:47.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:47.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:47.249 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:47.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:47.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:47.331 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:47 smithi076 conmon[56514]: cluster 2021-03-15T22:21:46.070530+0000 mon.a (mon.0) 802 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2021-03-15T22:21:47.332 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:47 smithi076 conmon[62156]: cluster 2021-03-15T22:21:47.332 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:47 smithi076 conmon[62156]: 2021-03-15T22:21:47.332 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:47 smithi076 conmon[62156]: 2021-03-15T22:21:46.070530+0000 mon.a (mon.0 2021-03-15T22:21:47.333 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:47 smithi076 conmon[62156]: ) 802 : cluster [DBG] osdmap e191: 8 total, 8 up, 8 in 2021-03-15T22:21:47.351 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:47.375 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:47.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:47.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:47.380 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:21:47.454 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:21:47.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:47.503 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:47.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:47.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:47.536 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:21:47 smithi125 conmon[67289]: debug 2021-03-15T22:21:47.087+0000 7fd4d8831700 -1 osd.6 pg_epoch: 192 pg[7.7( v 190'74 (0'0,190'74] local-lis/les=172/173 n=5 ec=91/91 lis/c=172/172 les/c/f=173/173/0 sis=172) [6,0,1] r=0 lpr=172 crt=190'74 lcod 185'72 mlcod 185'72 active+clean+snaptrim trimq=[5a~1] ps=[57~1,59~1,5b~1]] removing snap head 2021-03-15T22:21:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:47 smithi125 conmon[56739]: cluster 2021-03-15T22:21:46.070530+0000 mon.a (mon.0) 802 : cluster 2021-03-15T22:21:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:47 smithi125 conmon[56739]: [DBG] osdmap e191: 8 total, 8 up, 8 in 2021-03-15T22:21:47.606 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:47.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:47.633 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:47.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:47.735 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:47.762 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:47.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:47.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:47.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:47.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:47.890 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:47.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:47.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:48.019 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:48.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:48.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:48.062 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:48.093 INFO:tasks.rados.rados.0.smithi076.stdout:515: done (4 left) 2021-03-15T22:21:48.094 INFO:tasks.rados.rados.0.smithi076.stdout:516: read oid 18 snap 47 2021-03-15T22:21:48.094 INFO:tasks.rados.rados.0.smithi076.stdout:516: expect (ObjNum 192 snap 43 seq_num 192) 2021-03-15T22:21:48.094 INFO:tasks.rados.rados.0.smithi076.stdout:517: snap_create 2021-03-15T22:21:48.100 INFO:tasks.rados.rados.0.smithi076.stdout:510: finishing write tid 3 to smithi07694878-26 2021-03-15T22:21:48.100 INFO:tasks.rados.rados.0.smithi076.stdout:510: finishing write tid 4 to smithi07694878-26 2021-03-15T22:21:48.101 INFO:tasks.rados.rados.0.smithi076.stdout:510: finishing write tid 5 to smithi07694878-26 2021-03-15T22:21:48.101 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 150 (ObjNum 214 snap 48 seq_num 214) dirty exists 2021-03-15T22:21:48.101 INFO:tasks.rados.rados.0.smithi076.stdout:510: left oid 26 (ObjNum 214 snap 48 seq_num 214) 2021-03-15T22:21:48.101 INFO:tasks.rados.rados.0.smithi076.stdout:511: expect (ObjNum 180 snap 39 seq_num 180) 2021-03-15T22:21:48.114 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:48.122 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:48.154 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:48.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:48.156 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:48.214 INFO:tasks.rados.rados.0.smithi076.stdout:512: expect (ObjNum 197 snap 43 seq_num 197) 2021-03-15T22:21:48.257 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:48.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:48.280 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:48.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:48.314 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[56514]: cluster 2021-03-15T22:21:47.001128+0000 mgr.y (mgr. 2021-03-15T22:21:48.314 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[56514]: 14144) 447 : cluster [DBG] pgmap v414: 138 pgs: 138 active+clean; 140 MiB data, 444 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.6 MiB/s wr, 7 op/s 2021-03-15T22:21:48.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[56514]: cluster 2021-03-15T22:21:47.082031+0000 mon.a (mon.0) 803 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2021-03-15T22:21:48.315 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[75764]: debug 2021-03-15T22:21:48.093+0000 7f9fbafaf700 -1 osd.3 pg_epoch: 193 pg[7.a( v 193'188 (0'0,193'188] local-lis/les=91/92 n=6 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [3,2,5] r=0 lpr=91 luod=192'186 crt=193'188 lcod 192'184 mlcod 192'184 active+clean+snaptrim trimq=[56~1] ps=[57~1,59~3]] removing snap head 2021-03-15T22:21:48.316 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[62156]: cluster 2021-03-15T22:21:47.001128 2021-03-15T22:21:48.316 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 447 : cluster [DBG] pgmap v414: 138 pgs: 138 active+clean; 140 MiB data, 444 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.6 MiB/s wr, 7 op/s 2021-03-15T22:21:48.316 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[62156]: cluster 2021-03-15T22:21:48.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[62156]: 2021-03-15T22:21:47.082031+0000 mon.a (mon 2021-03-15T22:21:48.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[62156]: .0) 803 : cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2021-03-15T22:21:48.317 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[72291]: debug 2021-03-15T22:21:48.087+0000 7fb92f173700 -1 osd.2 pg_epoch: 193 pg[7.e( v 185'47 (0'0,185'47] local-lis/les=91/92 n=2 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [2,5,0] r=0 lpr=91 crt=185'47 lcod 184'45 mlcod 184'45 active+clean+snaptrim trimq=[56~1] ps=[57~1,59~3]] removing snap head 2021-03-15T22:21:48.318 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[65511]: debug 2021-03-15T22:21:48.318 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[65511]: 2021-03-15T22:21:48.097+0000 7f1a2307e700 -1 osd.0 pg_epoch: 193 pg[7.f( v 190'157 (0'0,190'157] local-lis/les=91/92 n=5 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [0,3,7] r=0 lpr=91 crt=190'157 lcod 186'155 mlcod 186'155 active+clean+snaptrim trimq=[56~1] ps=[57~1,59~3]] removing snap head 2021-03-15T22:21:48.318 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:48 smithi076 conmon[65511]: 2021-03-15T22:21:48.382 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:48.408 INFO:tasks.rados.rados.0.smithi076.stdout:514: expect (ObjNum 129 snap 24 seq_num 129) 2021-03-15T22:21:48.411 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:48.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:48.513 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:48.529 INFO:tasks.rados.rados.0.smithi076.stdout:516: expect (ObjNum 192 snap 43 seq_num 192) 2021-03-15T22:21:48.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:48 smithi125 conmon[56739]: cluster 2021-03-15T22:21:47.001128+0000 mgr.y (mgr.14144) 447 : cluster [DBG] pgmap v414: 138 pgs: 138 active+clean; 140 MiB data, 444 MiB used, 715 GiB / 715 GiB avail; 2.0 KiB/s rd, 1.6 MiB/s wr, 7 op/s 2021-03-15T22:21:48.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:48 smithi125 conmon[56739]: 2021-03-15T22:21:48.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:48 smithi125 conmon[56739]: cluster 2021-03-15T22:21:47.082031+0000 mon.a (mon.0) 803 : 2021-03-15T22:21:48.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:48 smithi125 conmon[56739]: cluster [DBG] osdmap e192: 8 total, 8 up, 8 in 2021-03-15T22:21:48.540 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:48.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:48.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:48.643 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:48.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:48.672 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:48.673 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:48.775 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:48.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:48.801 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:48.801 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:48.903 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:48.929 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:48.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:48.930 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:49.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:49.057 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:49.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:49.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:49.097 INFO:tasks.rados.rados.0.smithi076.stdout:510: done (5 left) 2021-03-15T22:21:49.098 INFO:tasks.rados.rados.0.smithi076.stdout:511: done (4 left) 2021-03-15T22:21:49.098 INFO:tasks.rados.rados.0.smithi076.stdout:512: done (3 left) 2021-03-15T22:21:49.098 INFO:tasks.rados.rados.0.smithi076.stdout:514: done (2 left) 2021-03-15T22:21:49.099 INFO:tasks.rados.rados.0.smithi076.stdout:516: done (1 left) 2021-03-15T22:21:49.099 INFO:tasks.rados.rados.0.smithi076.stdout:517: done (0 left) 2021-03-15T22:21:49.099 INFO:tasks.rados.rados.0.smithi076.stdout:518: write oid 13 current snap is 49 2021-03-15T22:21:49.099 INFO:tasks.rados.rados.0.smithi076.stdout:518: seq_num 215 ranges {413505=636419,1664805=689064,3038061=492071} 2021-03-15T22:21:49.100 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:21:49.103 INFO:tasks.rados.rados.0.smithi076.stdout:518: writing smithi07694878-13 from 413505 to 1049924 tid 1 2021-03-15T22:21:49.110 INFO:tasks.rados.rados.0.smithi076.stdout:518: writing smithi07694878-13 from 1664805 to 2353869 tid 2 2021-03-15T22:21:49.117 INFO:tasks.rados.rados.0.smithi076.stdout:518: writing smithi07694878-13 from 3038061 to 3530132 tid 3 2021-03-15T22:21:49.117 INFO:tasks.rados.rados.0.smithi076.stdout:519: snap_remove snap 47 2021-03-15T22:21:49.160 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:49.185 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:49.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:49.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:49 smithi076 conmon[56514]: cluster 2021-03-15T22:21:48.086616+0000 mon.a (mon.0) 804 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2021-03-15T22:21:49.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:49 smithi076 conmon[56514]: audit 2021-03-15T22:21:49.066520+0000 mon.a (mon.0) 805 : audit [INF] from='client.? 172.21.15.76:0/249254730' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:21:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:49 smithi076 conmon[62156]: cluster 2021-03-15T22:21:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:49 smithi076 conmon[62156]: 2021-03-15T22:21:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:49 smithi076 conmon[62156]: 2021-03-15T22:21:48. 2021-03-15T22:21:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:49 smithi076 conmon[62156]: 086616+0000 mon.a (mon.0) 804 : cluster [DBG] osdmap e193: 8 total, 8 up, 8 in 2021-03-15T22:21:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:49 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:21:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:49 smithi076 conmon[62156]: 21:49.066520+0000 mon.a (mon.0) 805 : audit [INF] 2021-03-15T22:21:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:49 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/249254730' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:21:49.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:49.312 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:49.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:49.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:49.415 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:49.438 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:49.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:49.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:49 smithi125 conmon[56739]: cluster 2021-03-15T22:21:48.086616+0000 mon.a (mon.0) 804 : cluster 2021-03-15T22:21:49.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:49 smithi125 conmon[56739]: [DBG] osdmap e193: 8 total, 8 up, 8 in 2021-03-15T22:21:49.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:49 smithi125 conmon[56739]: audit 2021-03-15T22:21:49.066520+0000 mon.a (mon.0 2021-03-15T22:21:49.502 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:49 smithi125 conmon[56739]: ) 805 : audit [INF] from='client.? 172.21.15.76:0/249254730' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:21:49.539 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:49.565 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:49.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:49.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:49.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:49.694 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:49.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:49.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:49.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:21:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:21:49.797 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:49.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:49.825 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:49.826 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:49.927 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:49.953 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:49.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:49.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:50.025 INFO:tasks.rados.rados.0.smithi076.stdout:519: done (1 left) 2021-03-15T22:21:50.025 INFO:tasks.rados.rados.0.smithi076.stdout:520: read oid 35 snap -1 2021-03-15T22:21:50.026 INFO:tasks.rados.rados.0.smithi076.stdout:520: expect (ObjNum 6 snap 0 seq_num 6) 2021-03-15T22:21:50.026 INFO:tasks.rados.rados.0.smithi076.stdout:521: snap_remove snap 48 2021-03-15T22:21:50.056 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:50.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:50.084 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:50.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:50.186 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:50.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:50.216 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:50.216 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:50.318 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:50.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:50.342 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:50.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:50.445 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:50.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:50.474 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:50.475 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:50.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: cluster 2021-03-15T22:21:49.001472+0000 2021-03-15T22:21:50.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: mgr.y (mgr.14144) 448 : cluster [DBG] pgmap v417: 138 pgs: 138 active+clean; 142 MiB data, 454 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-03-15T22:21:50.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:21:50.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: :21:49.085005+0000 mon.a (mon.0 2021-03-15T22:21:50.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: ) 806 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:50.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:21:50.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: 03-15T22:21:49.093190+0000 mon.a ( 2021-03-15T22:21:50.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: mon.0) 807 : audit [INF] from='client.? 172.21.15.76:0/249254730' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:21:50.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: cluster 2021-03-15T22:21:50.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:21:50.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: T22:21:49.093234+0000 mon.a (mon.0 2021-03-15T22:21:50.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: ) 808 : cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2021-03-15T22:21:50.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: audit 2021-03-15T22:21:50.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: 2021-03-15T22:21:49 2021-03-15T22:21:50.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: .499616+0000 mgr.y (mgr.14144) 449 : audit [DBG] 2021-03-15T22:21:50.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:50.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: cluster 2021-03-15T22:21:50.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: 2021-03-15T22:21:50.023043+0000 mon.a (mon.0) 809 : 2021-03-15T22:21:50.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[62156]: cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2021-03-15T22:21:50.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: cluster 2021-03-15T22:21:49.001472+0000 mgr.y (mgr.14144) 448 : 2021-03-15T22:21:50.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: cluster [DBG] pgmap v417: 138 pgs: 138 active+clean; 142 MiB data, 454 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-03-15T22:21:50.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: cluster 2021-03-15T22:21:49.085005+0000 mon.a (mon.0) 806 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:50.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: audit 2021-03-15T22:21:49.093190+0000 mon.a (mon.0) 807 : audit [INF] from='client.? 172.21.15.76:0/249254730' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:21:50.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: 2021-03-15T22:21:50.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: cluster 2021-03-15T22:21:49.093234+0000 mon.a (mon.0) 808 : cluster [DBG] 2021-03-15T22:21:50.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: osdmap e194: 8 total, 8 up, 8 in 2021-03-15T22:21:50.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: audit 2021-03-15 2021-03-15T22:21:50.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: T22:21:49.499616+0000 mgr.y (mgr.14144) 2021-03-15T22:21:50.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: 449 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:50.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:21:50.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: 03-15T22:21:50 2021-03-15T22:21:50.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: .023043+0000 mon.a (mon.0 2021-03-15T22:21:50.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:50 smithi076 conmon[56514]: ) 809 : cluster [DBG] osdmap e195: 8 total, 8 up, 8 in 2021-03-15T22:21:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: cluster 2021-03-15T22:21:49.001472+0000 mgr.y (mgr 2021-03-15T22:21:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: .14144) 448 : cluster [DBG] pgmap v417: 138 pgs: 138 active+clean; 142 MiB data, 454 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-03-15T22:21:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: cluster 2021-03-15T22:21:49.085005+0000 mon.a (mon.0 2021-03-15T22:21:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: ) 806 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:21:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: audit 2021-03-15T22:21:49.093190+0000 2021-03-15T22:21:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: mon.a (mon.0) 807 : audit [INF] from='client.? 172.21.15.76:0/249254730' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:21:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: 2021-03-15T22:21:50.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: cluster 2021-03-15T22:21:49 2021-03-15T22:21:50.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: .093234+0000 mon.a (mon.0) 808 : 2021-03-15T22:21:50.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: cluster [DBG] osdmap e194: 8 total, 8 up, 8 in 2021-03-15T22:21:50.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:21:50.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: 03-15T22:21:49.499616+0000 mgr.y ( 2021-03-15T22:21:50.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: mgr.14144) 449 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:21:50.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: cluster 2021-03-15T22:21:50.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: 2021-03-15T22:21:50.023043+0000 2021-03-15T22:21:50.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: mon.a (mon.0) 809 : cluster [DBG] 2021-03-15T22:21:50.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:50 smithi125 conmon[56739]: osdmap e195: 8 total, 8 up, 8 in 2021-03-15T22:21:50.576 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:50.603 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:50.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:50.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:50.705 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:50.732 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:50.733 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:50.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:50.859 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:50.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:50.961 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:50.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:50.986 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:50.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:51.007 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:21:51.008 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:21:51.008 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:21:51.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd out 3 2021-03-15T22:21:51.034 INFO:tasks.rados.rados.0.smithi076.stdout:521: done (2 left) 2021-03-15T22:21:51.034 INFO:tasks.rados.rados.0.smithi076.stdout:522: copy_from oid 25 from oid 7 current snap is 49 2021-03-15T22:21:51.034 INFO:tasks.rados.rados.0.smithi076.stdout:518: finishing write tid 1 to smithi07694878-13 2021-03-15T22:21:51.035 INFO:tasks.rados.rados.0.smithi076.stdout:518: finishing write tid 2 to smithi07694878-13 2021-03-15T22:21:51.035 INFO:tasks.rados.rados.0.smithi076.stdout:518: finishing write tid 3 to smithi07694878-13 2021-03-15T22:21:51.035 INFO:tasks.rados.rados.0.smithi076.stdout:518: finishing write tid 4 to smithi07694878-13 2021-03-15T22:21:51.035 INFO:tasks.rados.rados.0.smithi076.stdout:518: finishing write tid 5 to smithi07694878-13 2021-03-15T22:21:51.036 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 239 (ObjNum 215 snap 49 seq_num 215) dirty exists 2021-03-15T22:21:51.036 INFO:tasks.rados.rados.0.smithi076.stdout:518: left oid 13 (ObjNum 215 snap 49 seq_num 215) 2021-03-15T22:21:51.036 INFO:tasks.rados.rados.0.smithi076.stdout:520: expect (ObjNum 6 snap 0 seq_num 6) 2021-03-15T22:21:51.088 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:51.110 INFO:tasks.rados.rados.0.smithi076.stdout:522: finishing copy_from to smithi07694878-25 2021-03-15T22:21:51.110 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:21:51.110 INFO:tasks.rados.rados.0.smithi076.stdout:522: finishing copy_from racing read to smithi07694878-25 2021-03-15T22:21:51.111 INFO:tasks.rados.rados.0.smithi076.stdout:518: done (2 left) 2021-03-15T22:21:51.111 INFO:tasks.rados.rados.0.smithi076.stdout:520: done (1 left) 2021-03-15T22:21:51.111 INFO:tasks.rados.rados.0.smithi076.stdout:522: done (0 left) 2021-03-15T22:21:51.111 INFO:tasks.rados.rados.0.smithi076.stdout:523: snap_create 2021-03-15T22:21:51.115 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:51.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:51.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:51.184 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:21:51.184 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:21:51.184 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:21:51.185 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:21:51.218 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:51.247 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:51.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:51.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:51.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:51 smithi076 conmon[56514]: cluster 2021-03-15T22:21:49.846387+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:21:51.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:51 smithi076 conmon[56514]: cluster 2021-03-15T22:21:49.930590+0000 osd.1 (osd.1) 40 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:21:51.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:51 smithi076 conmon[56514]: cluster 2021-03-15T22:21:51.031402+0000 mon.a (mon.0) 810 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2021-03-15T22:21:51.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:51 smithi076 conmon[62156]: cluster 2021-03-15T22:21:49.846387+0000 osd.7 (osd.7) 46 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:21:51.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:51 smithi076 conmon[62156]: cluster 2021-03-15T22:21:49.930590+0000 osd.1 (osd.1) 40 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:21:51.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:51 smithi076 conmon[62156]: cluster 2021-03-15T22:21:51.031402+0000 mon.a (mon.0) 810 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2021-03-15T22:21:51.291 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:21:51.311 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:21:51.328 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:21:51.346 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:21:51.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:51.365 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:21:51.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:51.377 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:51.377 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:51.388 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:21:51.405 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:21:51.426 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:21:51.427 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:21:51.427 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:21:51.427 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:21:51.427 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:21:51.428 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:21:51.428 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:21:51.428 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:21:51.428 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:21:51.479 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:51.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:51.503 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:51.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:51.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:51 smithi125 conmon[56739]: cluster 2021-03-15T22:21:49.846387+0000 osd.7 (osd.7) 46 : cluster 2021-03-15T22:21:51.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:51 smithi125 conmon[56739]: [DBG] 6.1b scrub ok 2021-03-15T22:21:51.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:51 smithi125 conmon[56739]: cluster 2021-03-15T22:21:49.930590+0000 osd.1 (osd.1) 40 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:21:51.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:51 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:21:51.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:51 smithi125 conmon[56739]: 15T22:21:51.031402+0000 mon.a (mon.0) 810 : cluster [DBG] osdmap e196: 8 total, 8 up, 8 in 2021-03-15T22:21:51.604 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:51.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:51.627 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:51.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:51.729 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:51.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:51.753 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:51.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:51.855 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:51.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:51.882 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:51.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:51.984 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:52.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:52.009 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:52.010 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:52.111 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:52.115 INFO:tasks.rados.rados.0.smithi076.stdout:523: done (0 left) 2021-03-15T22:21:52.116 INFO:tasks.rados.rados.0.smithi076.stdout:524: snap_remove snap 49 2021-03-15T22:21:52.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:52.135 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:52.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:52.237 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:52.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:52.259 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:52.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:52.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:52.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[56514]: cluster 2021-03-15T22:21:50.930567+0000 osd.1 (osd.1) 41 : cluster 2021-03-15T22:21:52.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[56514]: [DBG] 3.8 scrub ok 2021-03-15T22:21:52.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[56514]: cluster 2021-03-15T22:21:51.001850+0000 mgr.y (mgr.14144) 450 : cluster [DBG] pgmap v420: 138 pgs: 138 active+clean; 133 MiB data, 439 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 893 KiB/s wr, 6 op/s 2021-03-15T22:21:52.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[56514]: cluster 2021-03-15T22:21:51.132168+0000 osd.7 (osd.7) 47 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:52.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[56514]: cluster 2021-03-15T22:21:51.132358+0000 osd.7 (osd.7) 48 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:52.389 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[56514]: cluster 2021-03-15T22:21:51.156634+0000 osd.5 (osd.5) 41 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:21:52.389 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[56514]: audit 2021-03-15T22:21:51.270364+0000 mon.a (mon.0) 811 : audit [DBG] from='client.? 172.21.15.76:0/1133201057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:52.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: cluster 2021-03-15T22:21:50.930567+0000 2021-03-15T22:21:52.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: osd.1 (osd.1) 41 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:21:52.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:21:52.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: :21:51.001850+0000 mgr.y (mgr.14144) 450 : cluster [DBG] pgmap v420: 138 pgs: 138 active+clean; 133 MiB data, 439 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 893 KiB/s wr, 6 op/s 2021-03-15T22:21:52.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:21:52.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: -03-15T22:21:51.132168+0000 2021-03-15T22:21:52.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: osd.7 (osd.7) 47 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:52.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: 2021-03-15T22:21:52.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:21:52.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: 21:51.132358+0000 osd.7 (osd.7 2021-03-15T22:21:52.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: ) 48 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:52.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:21:52.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: -03-15T22:21:51.156634+0000 2021-03-15T22:21:52.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: osd.5 (osd.5) 41 : cluster [DBG] 2021-03-15T22:21:52.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: 5.15 scrub ok 2021-03-15T22:21:52.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: audit 2021-03-15T22:21:52.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: 2021-03-15T22:21: 2021-03-15T22:21:52.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: 51.270364+0000 mon.a (mon.0) 2021-03-15T22:21:52.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:52 smithi076 conmon[62156]: 811 : audit [DBG] from='client.? 172.21.15.76:0/1133201057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:52.395 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:52.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:52.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:52.412 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:21:52.477 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:52.498 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:52.521 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:52.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:52.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:52.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: cluster 2021-03-15T22:21:50.930567+0000 osd.1 (osd.1) 41 : cluster [DBG] 2021-03-15T22:21:52.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: 3.8 scrub ok 2021-03-15T22:21:52.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: cluster 2021-03-15T22:21:51.001850+0000 mgr.y (mgr.14144) 450 : cluster [DBG] pgmap v420: 138 pgs: 138 active+clean; 133 MiB data, 439 MiB used, 715 GiB / 715 GiB avail; 2.5 MiB/s rd, 893 KiB/s wr, 6 op/s 2021-03-15T22:21:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: cluster 2021-03-15T22:21:51.132168 2021-03-15T22:21:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: +0000 osd.7 (osd.7) 47 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: cluster 2021-03-15T22:21:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: 2021-03-15T22:21:51.132358+0000 osd.7 2021-03-15T22:21:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: (osd.7) 48 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: 2021-03-15T22:21:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: cluster 2021-03-15T22:21:51. 2021-03-15T22:21:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: 156634+0000 osd.5 (osd.5) 41 : cluster 2021-03-15T22:21:52.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: [DBG] 5.15 scrub ok 2021-03-15T22:21:52.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:21:52.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: :21:51.270364+0000 mon.a (mon. 2021-03-15T22:21:52.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:52 smithi125 conmon[56739]: 0) 811 : audit [DBG] from='client.? 172.21.15.76:0/1133201057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:52.624 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:52.647 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:52.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:52.749 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:52.772 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:52.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:52.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:52.776 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:21:52.874 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:52.897 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:52.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:52.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:52.956 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:53.000 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:53.027 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:53.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:53.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:53.118 INFO:tasks.rados.rados.0.smithi076.stdout:524: done (0 left) 2021-03-15T22:21:53.118 INFO:tasks.rados.rados.0.smithi076.stdout:525: read oid 7 snap -1 2021-03-15T22:21:53.119 INFO:tasks.rados.rados.0.smithi076.stdout:525: expect (ObjNum 6 snap 0 seq_num 6) 2021-03-15T22:21:53.119 INFO:tasks.rados.rados.0.smithi076.stdout:526: rollback oid 4 current snap is 50 2021-03-15T22:21:53.119 INFO:tasks.rados.rados.0.smithi076.stdout:526: done (1 left) 2021-03-15T22:21:53.119 INFO:tasks.rados.rados.0.smithi076.stdout:527: read oid 39 snap -1 2021-03-15T22:21:53.120 INFO:tasks.rados.rados.0.smithi076.stdout:527: expect (ObjNum 116 snap 24 seq_num 116) 2021-03-15T22:21:53.120 INFO:tasks.rados.rados.0.smithi076.stdout:528: delete oid 43 current snap is 50 2021-03-15T22:21:53.124 INFO:tasks.rados.rados.0.smithi076.stdout:525: expect (ObjNum 6 snap 0 seq_num 6) 2021-03-15T22:21:53.130 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:53.151 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:53.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:53.153 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:53.195 INFO:tasks.rados.rados.0.smithi076.stdout:527: expect (ObjNum 116 snap 24 seq_num 116) 2021-03-15T22:21:53.254 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:53.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[56514]: cluster 2021-03-15T22:21:51.578521+0000 osd.2 2021-03-15T22:21:53.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[56514]: (osd.2) 38 : cluster [DBG] 5.a scrub ok 2021-03-15T22:21:53.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[56514]: cluster 2021-03-15T22:21:51.825075+0000 osd.6 (osd.6) 27 : cluster [DBG] 7.c scrub ok 2021-03-15T22:21:53.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[56514]: cluster 2021-03-15T22:21:51.997732+0000 osd.0 (osd.0) 43 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:53.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[56514]: cluster 2021-03-15T22:21:51.997859+0000 osd.0 (osd.0) 44 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:53.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[56514]: cluster 2021-03-15T22:21:52.113257+0000 mon.a (mon.0) 812 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2021-03-15T22:21:53.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[56514]: cluster 2021-03-15T22:21:52.835986+0000 osd.0 (osd.0) 45 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:21:53.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[62156]: cluster 2021-03-15T22:21:51.578521+0000 osd.2 (osd.2) 38 : cluster [DBG] 5.a scrub ok 2021-03-15T22:21:53.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[62156]: cluster 2021-03-15T22:21:51.825075+0000 osd.6 (osd.6) 27 : cluster [DBG] 7.c scrub ok 2021-03-15T22:21:53.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[62156]: cluster 2021-03-15T22:21:51.997732+0000 osd.0 (osd.0) 43 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:53.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[62156]: cluster 2021-03-15T22:21:51.997859+0000 osd.0 (osd.0) 44 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:53.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[62156]: cluster 2021-03-15T22:21:52.113257+0000 mon.a (mon.0) 812 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2021-03-15T22:21:53.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:53 smithi076 conmon[62156]: cluster 2021-03-15T22:21:52.835986+0000 osd.0 (osd.0) 45 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:21:53.281 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:53.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:53.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:53.321 INFO:tasks.rados.rados.0.smithi076.stdout:525: done (2 left) 2021-03-15T22:21:53.322 INFO:tasks.rados.rados.0.smithi076.stdout:527: done (1 left) 2021-03-15T22:21:53.322 INFO:tasks.rados.rados.0.smithi076.stdout:528: done (0 left) 2021-03-15T22:21:53.322 INFO:tasks.rados.rados.0.smithi076.stdout:529: rollback oid 27 current snap is 50 2021-03-15T22:21:53.322 INFO:tasks.rados.rados.0.smithi076.stdout:529: done (0 left) 2021-03-15T22:21:53.323 INFO:tasks.rados.rados.0.smithi076.stdout:530: write oid 18 current snap is 50 2021-03-15T22:21:53.323 INFO:tasks.rados.rados.0.smithi076.stdout:530: seq_num 217 ranges {447119=786917,1665416=692733,2976554=514615} 2021-03-15T22:21:53.327 INFO:tasks.rados.rados.0.smithi076.stdout:530: writing smithi07694878-18 from 447119 to 1234036 tid 1 2021-03-15T22:21:53.334 INFO:tasks.rados.rados.0.smithi076.stdout:530: writing smithi07694878-18 from 1665416 to 2358149 tid 2 2021-03-15T22:21:53.340 INFO:tasks.rados.rados.0.smithi076.stdout:530: writing smithi07694878-18 from 2976554 to 3491169 tid 3 2021-03-15T22:21:53.341 INFO:tasks.rados.rados.0.smithi076.stdout:531: copy_from oid 4 from oid 41 current snap is 50 2021-03-15T22:21:53.341 INFO:tasks.rados.rados.0.smithi076.stdout:530: finishing write tid 1 to smithi07694878-18 2021-03-15T22:21:53.341 INFO:tasks.rados.rados.0.smithi076.stdout:530: finishing write tid 2 to smithi07694878-18 2021-03-15T22:21:53.341 INFO:tasks.rados.rados.0.smithi076.stdout:532: write oid 25 current snap is 50 2021-03-15T22:21:53.342 INFO:tasks.rados.rados.0.smithi076.stdout:532: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2021-03-15T22:21:53.345 INFO:tasks.rados.rados.0.smithi076.stdout:532: writing smithi07694878-25 from 463926 to 926092 tid 1 2021-03-15T22:21:53.352 INFO:tasks.rados.rados.0.smithi076.stdout:532: writing smithi07694878-25 from 1723898 to 2160289 tid 2 2021-03-15T22:21:53.359 INFO:tasks.rados.rados.0.smithi076.stdout:532: writing smithi07694878-25 from 2603977 to 3329864 tid 3 2021-03-15T22:21:53.360 INFO:tasks.rados.rados.0.smithi076.stdout:533: delete oid 5 current snap is 50 2021-03-15T22:21:53.360 INFO:tasks.rados.rados.0.smithi076.stdout:530: finishing write tid 3 to smithi07694878-18 2021-03-15T22:21:53.360 INFO:tasks.rados.rados.0.smithi076.stdout:530: finishing write tid 4 to smithi07694878-18 2021-03-15T22:21:53.360 INFO:tasks.rados.rados.0.smithi076.stdout:530: finishing write tid 5 to smithi07694878-18 2021-03-15T22:21:53.360 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 98 (ObjNum 217 snap 50 seq_num 217) dirty exists 2021-03-15T22:21:53.361 INFO:tasks.rados.rados.0.smithi076.stdout:530: left oid 18 (ObjNum 217 snap 50 seq_num 217) 2021-03-15T22:21:53.361 INFO:tasks.rados.rados.0.smithi076.stdout:532: finishing write tid 1 to smithi07694878-25 2021-03-15T22:21:53.362 INFO:tasks.rados.rados.0.smithi076.stdout:531: finishing copy_from to smithi07694878-4 2021-03-15T22:21:53.362 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 90 (ObjNum 164 snap 37 seq_num 164) dirty exists 2021-03-15T22:21:53.362 INFO:tasks.rados.rados.0.smithi076.stdout:531: finishing copy_from racing read to smithi07694878-4 2021-03-15T22:21:53.363 INFO:tasks.rados.rados.0.smithi076.stdout:530: done (3 left) 2021-03-15T22:21:53.363 INFO:tasks.rados.rados.0.smithi076.stdout:531: done (2 left) 2021-03-15T22:21:53.363 INFO:tasks.rados.rados.0.smithi076.stdout:533: done (1 left) 2021-03-15T22:21:53.363 INFO:tasks.rados.rados.0.smithi076.stdout:534: read oid 32 snap -1 2021-03-15T22:21:53.364 INFO:tasks.rados.rados.0.smithi076.stdout:534: expect (ObjNum 108 snap 23 seq_num 108) 2021-03-15T22:21:53.364 INFO:tasks.rados.rados.0.smithi076.stdout:535: write oid 17 current snap is 50 2021-03-15T22:21:53.364 INFO:tasks.rados.rados.0.smithi076.stdout:535: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2021-03-15T22:21:53.367 INFO:tasks.rados.rados.0.smithi076.stdout:535: writing smithi07694878-17 from 497540 to 1110204 tid 1 2021-03-15T22:21:53.373 INFO:tasks.rados.rados.0.smithi076.stdout:535: writing smithi07694878-17 from 1840862 to 2280922 tid 2 2021-03-15T22:21:53.380 INFO:tasks.rados.rados.0.smithi076.stdout:535: writing smithi07694878-17 from 3058823 to 3523607 tid 3 2021-03-15T22:21:53.380 INFO:tasks.rados.rados.0.smithi076.stdout:536: snap_create 2021-03-15T22:21:53.381 INFO:tasks.rados.rados.0.smithi076.stdout:532: finishing write tid 2 to smithi07694878-25 2021-03-15T22:21:53.381 INFO:tasks.rados.rados.0.smithi076.stdout:532: finishing write tid 3 to smithi07694878-25 2021-03-15T22:21:53.381 INFO:tasks.rados.rados.0.smithi076.stdout:532: finishing write tid 4 to smithi07694878-25 2021-03-15T22:21:53.381 INFO:tasks.rados.rados.0.smithi076.stdout:532: finishing write tid 5 to smithi07694878-25 2021-03-15T22:21:53.382 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 118 (ObjNum 218 snap 50 seq_num 218) dirty exists 2021-03-15T22:21:53.382 INFO:tasks.rados.rados.0.smithi076.stdout:532: left oid 25 (ObjNum 218 snap 50 seq_num 218) 2021-03-15T22:21:53.382 INFO:tasks.rados.rados.0.smithi076.stdout:534: expect (ObjNum 108 snap 23 seq_num 108) 2021-03-15T22:21:53.384 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:53.408 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:53.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:53.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:53.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:53.566 INFO:tasks.rados.rados.0.smithi076.stdout:535: finishing write tid 1 to smithi07694878-17 2021-03-15T22:21:53.566 INFO:tasks.rados.rados.0.smithi076.stdout:535: finishing write tid 2 to smithi07694878-17 2021-03-15T22:21:53.566 INFO:tasks.rados.rados.0.smithi076.stdout:535: finishing write tid 3 to smithi07694878-17 2021-03-15T22:21:53.567 INFO:tasks.rados.rados.0.smithi076.stdout:535: finishing write tid 4 to smithi07694878-17 2021-03-15T22:21:53.567 INFO:tasks.rados.rados.0.smithi076.stdout:535: finishing write tid 5 to smithi07694878-17 2021-03-15T22:21:53.567 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 57 (ObjNum 220 snap 50 seq_num 220) dirty exists 2021-03-15T22:21:53.567 INFO:tasks.rados.rados.0.smithi076.stdout:535: left oid 17 (ObjNum 220 snap 50 seq_num 220) 2021-03-15T22:21:53.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: cluster 2021-03-15T22:21:51.578521+0000 osd.2 (osd.2) 38 : cluster [DBG] 5.a scrub ok 2021-03-15T22:21:53.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: cluster 2021-03-15T22:21:51.825075+0000 osd.6 (osd.6) 27 : cluster [DBG] 7.c scrub ok 2021-03-15T22:21:53.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: cluster 2021-03-15T22:21:51.997732+0000 osd.0 (osd.0) 43 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:53.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: cluster 2021-03-15T22:21 2021-03-15T22:21:53.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: :51.997859+0000 osd.0 (osd.0) 44 2021-03-15T22:21:53.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:53.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:21:53.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: 15T22:21:52.113257+0000 mon.a 2021-03-15T22:21:53.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: (mon.0) 812 : cluster [DBG] osdmap e197: 8 total, 8 up, 8 in 2021-03-15T22:21:53.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: 2021-03-15T22:21:53.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: cluster 2021-03-15T22:21:52 2021-03-15T22:21:53.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: .835986+0000 osd.0 (osd.0) 45 : cluster 2021-03-15T22:21:53.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:53 smithi125 conmon[56739]: [DBG] 3.6 scrub ok 2021-03-15T22:21:53.588 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:53.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:53.691 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:53.717 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:53.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:53.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:53.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:53.843 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:53.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:53.845 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:53.946 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:53.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:53.974 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:53.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:54.035 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:54.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:54.099 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:54.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:54.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:54.131 INFO:tasks.rados.rados.0.smithi076.stdout:532: done (3 left) 2021-03-15T22:21:54.131 INFO:tasks.rados.rados.0.smithi076.stdout:534: done (2 left) 2021-03-15T22:21:54.132 INFO:tasks.rados.rados.0.smithi076.stdout:535: done (1 left) 2021-03-15T22:21:54.132 INFO:tasks.rados.rados.0.smithi076.stdout:536: done (0 left) 2021-03-15T22:21:54.132 INFO:tasks.rados.rados.0.smithi076.stdout:537: snap_remove snap 50 2021-03-15T22:21:54.202 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:54.225 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:54.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:54.226 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:54.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:54.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:54.357 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:54.357 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:54.446 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:21:54.458 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:54.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[56514]: cluster 2021-03-15T22:21:52.622744+0000 osd.2 (osd.2) 39 : cluster [DBG] 4.f scrub ok 2021-03-15T22:21:54.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[56514]: cluster 2021-03-15T22:21:53.002453+0000 mgr.y (mgr.14144) 451 : cluster [DBG] pgmap v423: 138 pgs: 138 active+clean; 117 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 444 KiB/s wr, 13 op/s 2021-03-15T22:21:54.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[56514]: cluster 2021-03-15T22:21:53.115172+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2021-03-15T22:21:54.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[56514]: cluster 2021-03-15T22:21:53.179147+0000 osd.5 (osd.5) 42 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:21:54.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[56514]: cluster 2021-03-15T22:21:53.852822+0000 osd.0 (osd.0) 46 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:21:54.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: cluster 2021-03-15T22:21: 2021-03-15T22:21:54.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: 52.622744+0000 osd.2 (osd.2) 39 : cluster [DBG] 4.f scrub ok 2021-03-15T22:21:54.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: cluster 2021-03-15T22:21:54.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: 2021-03-15T22:21:53.002453+0000 mgr.y (mgr.14144 2021-03-15T22:21:54.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: ) 451 : cluster [DBG] pgmap v423: 138 pgs: 138 active+clean; 117 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 444 KiB/s wr, 13 op/s 2021-03-15T22:21:54.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: cluster 2021-03-15T22:21:54.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:21:54.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: :21:53.115172+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2021-03-15T22:21:54.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:21:54.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: -03-15T22:21:53. 2021-03-15T22:21:54.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: 179147+0000 osd.5 (osd.5) 2021-03-15T22:21:54.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: 42 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:21:54.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: cluster 2021-03-15T22:21:54.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: 2021-03-15T22:21:54.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: 2021 2021-03-15T22:21:54.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: - 2021-03-15T22:21:54.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: 03 2021-03-15T22:21:54.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: - 2021-03-15T22:21:54.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: 15 2021-03-15T22:21:54.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: T22:21:53.852822 2021-03-15T22:21:54.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[62156]: +0000 osd.0 (osd.0) 46 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:21:54.497 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:54.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:54.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:54.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:54 smithi125 conmon[56739]: cluster 2021-03-15T22:21:52.622744+0000 osd.2 (osd.2) 39 : cluster [DBG] 4.f scrub ok 2021-03-15T22:21:54.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:54 smithi125 conmon[56739]: cluster 2021-03-15T22:21:53.002453+0000 mgr.y (mgr.14144) 451 : cluster [DBG] pgmap v423: 138 pgs: 138 active+clean; 117 MiB data, 406 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 444 KiB/s wr, 13 op/s 2021-03-15T22:21:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:54 smithi125 conmon[56739]: cluster 2021-03-15T22:21: 2021-03-15T22:21:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:54 smithi125 conmon[56739]: 53.115172+0000 mon.a (mon.0) 813 : cluster [DBG] osdmap e198: 8 total, 8 up, 8 in 2021-03-15T22:21:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:54 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:21:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:54 smithi125 conmon[56739]: -15T22:21:53.179147+0000 osd.5 2021-03-15T22:21:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:54 smithi125 conmon[56739]: (osd.5) 42 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:21:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:54 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:21:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:54 smithi125 conmon[56739]: -03-15T22:21:53.852822+0000 2021-03-15T22:21:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:54 smithi125 conmon[56739]: osd.0 (osd.0) 46 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:21:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:54 smithi125 conmon[56739]: 2021-03-15T22:21:54.600 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:54.623 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:54.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:54.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:54.726 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:54.749 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:54.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:54.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:54.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:54.858 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:54.878 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:54.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:54.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:54.980 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:55.006 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:55.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:55.008 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:55.016 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:54 smithi076 conmon[56514]: debug 2021-03-15T22:21:54.891+0000 7f0537c88700 -1 mon.a@0(leader).osd e199 definitely_dead 0 2021-03-15T22:21:55.041 INFO:tasks.rados.rados.0.smithi076.stdout:537: done (0 left) 2021-03-15T22:21:55.041 INFO:tasks.rados.rados.0.smithi076.stdout:538: write (excl) oid 28 current snap is 51 2021-03-15T22:21:55.042 INFO:tasks.rados.rados.0.smithi076.stdout:538: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2021-03-15T22:21:55.042 INFO:tasks.rados.rados.0.smithi076.stdout:538: writing smithi07694878-28 from 514347 to 918613 tid 1 2021-03-15T22:21:55.042 INFO:teuthology.orchestra.run.smithi076.stderr:marked out osd.3. 2021-03-15T22:21:55.047 INFO:tasks.rados.rados.0.smithi076.stdout:538: writing smithi07694878-28 from 1499344 to 2199415 tid 2 2021-03-15T22:21:55.054 INFO:tasks.rados.rados.0.smithi076.stdout:538: writing smithi07694878-28 from 2686246 to 3362302 tid 3 2021-03-15T22:21:55.054 INFO:tasks.rados.rados.0.smithi076.stdout:539: delete oid 25 current snap is 51 2021-03-15T22:21:55.056 INFO:tasks.rados.rados.0.smithi076.stdout:539: done (1 left) 2021-03-15T22:21:55.056 INFO:tasks.rados.rados.0.smithi076.stdout:540: copy_from oid 3 from oid 11 current snap is 51 2021-03-15T22:21:55.056 INFO:tasks.rados.rados.0.smithi076.stdout:541: rollback oid 46 current snap is 51 2021-03-15T22:21:55.057 INFO:tasks.rados.rados.0.smithi076.stdout:541: done (2 left) 2021-03-15T22:21:55.057 INFO:tasks.rados.rados.0.smithi076.stdout:542: copy_from oid 21 from oid 47 current snap is 51 2021-03-15T22:21:55.057 INFO:tasks.rados.rados.0.smithi076.stdout:543: write oid 5 current snap is 51 2021-03-15T22:21:55.057 INFO:tasks.rados.rados.0.smithi076.stdout:543: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2021-03-15T22:21:55.062 INFO:tasks.rados.rados.0.smithi076.stdout:543: writing smithi07694878-5 from 547961 to 1102725 tid 1 2021-03-15T22:21:55.068 INFO:tasks.rados.rados.0.smithi076.stdout:543: writing smithi07694878-5 from 1616308 to 2320048 tid 2 2021-03-15T22:21:55.075 INFO:tasks.rados.rados.0.smithi076.stdout:543: writing smithi07694878-5 from 2741092 to 3439692 tid 3 2021-03-15T22:21:55.075 INFO:tasks.rados.rados.0.smithi076.stdout:544: write (excl) oid 37 current snap is 51 2021-03-15T22:21:55.076 INFO:tasks.rados.rados.0.smithi076.stdout:543: finishing write tid 1 to smithi07694878-5 2021-03-15T22:21:55.076 INFO:tasks.rados.rados.0.smithi076.stdout:543: finishing write tid 2 to smithi07694878-5 2021-03-15T22:21:55.076 INFO:tasks.rados.rados.0.smithi076.stdout:544: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2021-03-15T22:21:55.082 INFO:tasks.rados.rados.0.smithi076.stdout:544: writing smithi07694878-37 from 564768 to 1194781 tid 1 2021-03-15T22:21:55.087 INFO:tasks.rados.rados.0.smithi076.stdout:544: writing smithi07694878-37 from 1674790 to 2122188 tid 2 2021-03-15T22:21:55.094 INFO:tasks.rados.rados.0.smithi076.stdout:544: writing smithi07694878-37 from 2768515 to 3394740 tid 3 2021-03-15T22:21:55.094 INFO:tasks.rados.rados.0.smithi076.stdout:545: write (excl) oid 23 current snap is 51 2021-03-15T22:21:55.095 INFO:tasks.rados.rados.0.smithi076.stdout:545: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2021-03-15T22:21:55.101 INFO:tasks.rados.rados.0.smithi076.stdout:545: writing smithi07694878-23 from 581575 to 1286837 tid 1 2021-03-15T22:21:55.108 INFO:tasks.rados.rados.0.smithi076.stdout:545: writing smithi07694878-23 from 1733272 to 2440681 tid 2 2021-03-15T22:21:55.108 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:55.115 INFO:tasks.rados.rados.0.smithi076.stdout:545: writing smithi07694878-23 from 3195938 to 3633435 tid 3 2021-03-15T22:21:55.115 INFO:tasks.rados.rados.0.smithi076.stdout:546: delete oid 6 current snap is 51 2021-03-15T22:21:55.115 INFO:tasks.rados.rados.0.smithi076.stdout:543: finishing write tid 3 to smithi07694878-5 2021-03-15T22:21:55.116 INFO:tasks.rados.rados.0.smithi076.stdout:543: finishing write tid 4 to smithi07694878-5 2021-03-15T22:21:55.116 INFO:tasks.rados.rados.0.smithi076.stdout:543: finishing write tid 5 to smithi07694878-5 2021-03-15T22:21:55.116 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 69 (ObjNum 223 snap 51 seq_num 223) dirty exists 2021-03-15T22:21:55.116 INFO:tasks.rados.rados.0.smithi076.stdout:543: left oid 5 (ObjNum 223 snap 51 seq_num 223) 2021-03-15T22:21:55.116 INFO:tasks.rados.rados.0.smithi076.stdout:545: finishing write tid 1 to smithi07694878-23 2021-03-15T22:21:55.117 INFO:tasks.rados.rados.0.smithi076.stdout:545: finishing write tid 2 to smithi07694878-23 2021-03-15T22:21:55.117 INFO:tasks.rados.rados.0.smithi076.stdout:545: finishing write tid 3 to smithi07694878-23 2021-03-15T22:21:55.118 INFO:tasks.rados.rados.0.smithi076.stdout:545: finishing write tid 4 to smithi07694878-23 2021-03-15T22:21:55.118 INFO:tasks.rados.rados.0.smithi076.stdout:545: finishing write tid 5 to smithi07694878-23 2021-03-15T22:21:55.118 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 102 (ObjNum 225 snap 51 seq_num 225) dirty exists 2021-03-15T22:21:55.118 INFO:tasks.rados.rados.0.smithi076.stdout:545: left oid 23 (ObjNum 225 snap 51 seq_num 225) 2021-03-15T22:21:55.132 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:55.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:55.134 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:55.235 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:55.258 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:55.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:55.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:55.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:55.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:55.386 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:55.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:55.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[56514]: cluster 2021-03-15T22:21:54.127606+0000 mon.a (mon.0) 814 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2021-03-15T22:21:55.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[56514]: cluster 2021-03-15T22:21:54.579676+0000 osd.3 (osd.3) 47 : cluster [DBG] 7.2 scrub ok 2021-03-15T22:21:55.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[56514]: audit 2021-03-15T22:21:54.893608+0000 mon.a (mon.0) 815 : audit [INF] from='client.? 172.21.15.76:0/3286227382' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2021-03-15T22:21:55.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[56514]: cluster 2021-03-15T22:21:54.893699+0000 mon.a (mon.0) 816 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2021-03-15T22:21:55.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[56514]: audit 2021-03-15T22:21:55.021735+0000 mon.a (mon.0) 817 : audit [INF] from='client.? 172.21.15.76:0/3286227382' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2021-03-15T22:21:55.388 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[56514]: cluster 2021-03-15T22:21:55.021899+0000 mon.a (mon.0) 818 : cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2021-03-15T22:21:55.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: cluster 2021-03-15T22:21:54.127606+0000 2021-03-15T22:21:55.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: mon.a (mon.0) 814 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2021-03-15T22:21:55.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:21:55.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: 15T22:21:54.579676+0000 osd.3 (osd.3) 47 : cluster [DBG] 7.2 scrub ok 2021-03-15T22:21:55.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: audit 2021-03-15T22:21:55.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: 2021-03-15T22:21:54.893608+0000 2021-03-15T22:21:55.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: mon.a (mon.0) 815 : audit [INF] from='client.? 172.21.15.76:0/3286227382' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2021-03-15T22:21:55.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:55.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: 03-15T22:21:54.893699+0000 mon.a (mon.0) 816 2021-03-15T22:21:55.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2021-03-15T22:21:55.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:21:55.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: 03-15T22:21:55.021735+0000 mon.a (mon.0) 2021-03-15T22:21:55.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: 817 : audit [INF] from='client.? 172.21.15.76:0/3286227382' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2021-03-15T22:21:55.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:21:55.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: 21:55.021899+0000 mon.a (mon.0) 818 : 2021-03-15T22:21:55.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:55 smithi076 conmon[62156]: cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2021-03-15T22:21:55.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:55.511 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:55.512 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:55.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: cluster 2021-03-15T22:21:54.127606+0000 mon.a (mon.0 2021-03-15T22:21:55.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: ) 814 : cluster [DBG] osdmap e199: 8 total, 8 up, 8 in 2021-03-15T22:21:55.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: cluster 2021-03-15T22:21:54.579676+0000 osd.3 (osd.3) 47 : 2021-03-15T22:21:55.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: cluster [DBG] 7.2 scrub ok 2021-03-15T22:21:55.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: audit 2021-03-15T22:21:55.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: 2021-03-15T22:21:54.893608+0000 mon.a ( 2021-03-15T22:21:55.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: mon.0) 815 : audit [INF] from='client.? 172.21.15.76:0/3286227382' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2021-03-15T22:21:55.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:21:55.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: 03-15T22:21:54.893699+0000 mon.a (mon. 2021-03-15T22:21:55.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: 0) 816 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2021-03-15T22:21:55.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: audit 2021-03-15T22:21:55.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: 2021-03-15T22:21:55.021735+0000 mon.a (mon 2021-03-15T22:21:55.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: .0) 817 : audit [INF] from='client.? 172.21.15.76:0/3286227382' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2021-03-15T22:21:55.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:21:55.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: 03-15T22:21:55.021899+0000 mon.a (mon 2021-03-15T22:21:55.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:55 smithi125 conmon[56739]: .0) 818 : cluster [DBG] osdmap e200: 8 total, 8 up, 7 in 2021-03-15T22:21:55.613 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:55.637 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:55.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:55.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:55.740 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:55.764 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:55.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:55.766 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:55.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:55.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:55.890 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:55.890 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:55.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:56.014 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:56.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:56.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:56.056 INFO:tasks.rados.rados.0.smithi076.stdout:543: done (6 left) 2021-03-15T22:21:56.056 INFO:tasks.rados.rados.0.smithi076.stdout:545: done (5 left) 2021-03-15T22:21:56.056 INFO:tasks.rados.rados.0.smithi076.stdout:546: done (4 left) 2021-03-15T22:21:56.057 INFO:tasks.rados.rados.0.smithi076.stdout:547: copy_from oid 45 from oid 10 current snap is 51 2021-03-15T22:21:56.057 INFO:tasks.rados.rados.0.smithi076.stdout:548: read oid 9 snap -1 2021-03-15T22:21:56.057 INFO:tasks.rados.rados.0.smithi076.stdout:548: expect (ObjNum 129 snap 24 seq_num 129) 2021-03-15T22:21:56.057 INFO:tasks.rados.rados.0.smithi076.stdout:549: read oid 33 snap -1 2021-03-15T22:21:56.057 INFO:tasks.rados.rados.0.smithi076.stdout:549: expect (ObjNum 185 snap 40 seq_num 185) 2021-03-15T22:21:56.058 INFO:tasks.rados.rados.0.smithi076.stdout:550: snap_create 2021-03-15T22:21:56.058 INFO:tasks.rados.rados.0.smithi076.stdout:548: expect (ObjNum 129 snap 24 seq_num 129) 2021-03-15T22:21:56.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:56.139 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:56.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:56.141 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:56.195 INFO:tasks.rados.rados.0.smithi076.stdout:549: expect (ObjNum 185 snap 40 seq_num 185) 2021-03-15T22:21:56.237 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:21:56.237 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:21:56.242 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:56.263 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:56.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:56.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:56.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:56.387 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:56.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:56.388 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:56.406 INFO:tasks.rados.rados.0.smithi076.stdout:547: finishing copy_from to smithi07694878-45 2021-03-15T22:21:56.406 INFO:tasks.rados.rados.0.smithi076.stdout:547: got expected ENOENT (src dne) 2021-03-15T22:21:56.406 INFO:tasks.rados.rados.0.smithi076.stdout:547: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:21:56.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:56.510 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:56.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:56.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:56.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[56514]: cluster 2021-03-15T22:21:54.720361+0000 2021-03-15T22:21:56.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[56514]: osd.6 (osd.6) 28 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:21:56.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[56514]: cluster 2021-03-15T22:21:54.890497+0000 osd.1 (osd.1) 42 : cluster [DBG] 5.0 scrub ok 2021-03-15T22:21:56.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[56514]: cluster 2021-03-15T22:21:55.002890+0000 mgr.y (mgr.14144) 452 : cluster [DBG] pgmap v426: 138 pgs: 138 active+clean; 117 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 444 KiB/s wr, 11 op/s 2021-03-15T22:21:56.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[56514]: cluster 2021-03-15T22:21:55.110122+0000 osd.5 (osd.5) 43 : cluster [DBG] 5.f scrub ok 2021-03-15T22:21:56.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[56514]: cluster 2021-03-15T22:21:55.222481+0000 osd.4 (osd.4) 48 : cluster [DBG] 3.9 scrub ok 2021-03-15T22:21:56.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[56514]: cluster 2021-03-15T22:21:56.028321+0000 mon.a (mon.0) 819 : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2021-03-15T22:21:56.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: cluster 2021-03-15T22:21:56.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: 2021-03-15T22:21:56.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: 2021 2021-03-15T22:21:56.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: - 2021-03-15T22:21:56.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: 03-15T22: 2021-03-15T22:21:56.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: 21:54.720361+0000 osd.6 (osd.6) 28 : cluster [DBG] 2021-03-15T22:21:56.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: 3.10 scrub ok 2021-03-15T22:21:56.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:21:56.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: 22:21:54.890497+0000 osd.1 (osd.1) 42 2021-03-15T22:21:56.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: : cluster [DBG] 5.0 scrub ok 2021-03-15T22:21:56.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:21:56.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: -03-15T22:21:55.002890 2021-03-15T22:21:56.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 452 : cluster 2021-03-15T22:21:56.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: [DBG] pgmap v426: 138 pgs: 138 active+clean; 117 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 444 KiB/s wr, 11 op/s 2021-03-15T22:21:56.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:21:56.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: 21:55.110122+0000 osd.5 (osd.5 2021-03-15T22:21:56.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: ) 43 : cluster [DBG] 5.f scrub ok 2021-03-15T22:21:56.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:56.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: 03-15T22:21:55.222481+0000 2021-03-15T22:21:56.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: osd.4 (osd.4) 48 : cluster [DBG] 2021-03-15T22:21:56.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: 3.9 scrub ok 2021-03-15T22:21:56.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: cluster 2021-03-15T22:21: 2021-03-15T22:21:56.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: 56.028321+0000 mon.a (mon.0) 819 2021-03-15T22:21:56.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:56 smithi076 conmon[62156]: : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2021-03-15T22:21:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: cluster 2021-03-15T22:21:54.720361+0000 osd.6 (osd.6) 28 : 2021-03-15T22:21:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: cluster [DBG] 3.10 scrub ok 2021-03-15T22:21:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: cluster 2021-03-15T22:21:54.890497+0000 osd.1 (osd.1) 42 : cluster [DBG] 5.0 scrub ok 2021-03-15T22:21:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: cluster 2021-03-15T22:21:55. 2021-03-15T22:21:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: 002890+0000 mgr.y (mgr.14144) 452 : cluster [DBG] pgmap v426: 138 pgs: 138 active+clean; 117 MiB data, 407 MiB used, 715 GiB / 715 GiB avail; 11 MiB/s rd, 444 KiB/s wr, 11 op/s 2021-03-15T22:21:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:21:56.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: -15T22:21:55.110122+0000 osd.5 (osd. 2021-03-15T22:21:56.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: 5) 43 : cluster [DBG] 5.f scrub ok 2021-03-15T22:21:56.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:21:56.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: :21:55.222481+0000 osd.4 (osd.4) 48 : cluster 2021-03-15T22:21:56.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: [DBG] 3.9 scrub ok 2021-03-15T22:21:56.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: cluster 2021-03-15T22:21:56.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: 2021-03-15T22:21:56.028321+0000 mon.a (mon. 2021-03-15T22:21:56.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:56 smithi125 conmon[56739]: 0) 819 : cluster [DBG] osdmap e201: 8 total, 8 up, 7 in 2021-03-15T22:21:56.613 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:56.642 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:56.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:56.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:56.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:56.767 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:56.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:56.768 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:56.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:56.891 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:56.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:56.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:56.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:57.016 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:57.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:57.026 INFO:tasks.rados.rados.0.smithi076.stdout:547: done (7 left) 2021-03-15T22:21:57.026 INFO:tasks.rados.rados.0.smithi076.stdout:548: done (6 left) 2021-03-15T22:21:57.027 INFO:tasks.rados.rados.0.smithi076.stdout:549: done (5 left) 2021-03-15T22:21:57.027 INFO:tasks.rados.rados.0.smithi076.stdout:550: done (4 left) 2021-03-15T22:21:57.027 INFO:tasks.rados.rados.0.smithi076.stdout:551: read oid 14 snap 51 2021-03-15T22:21:57.027 INFO:tasks.rados.rados.0.smithi076.stdout:551: expect deleted 2021-03-15T22:21:57.028 INFO:tasks.rados.rados.0.smithi076.stdout:552: read oid 22 snap -1 2021-03-15T22:21:57.028 INFO:tasks.rados.rados.0.smithi076.stdout:552: expect deleted 2021-03-15T22:21:57.029 INFO:tasks.rados.rados.0.smithi076.stdout:553: copy_from oid 36 from oid 24 current snap is 52 2021-03-15T22:21:57.029 INFO:tasks.rados.rados.0.smithi076.stdout:554: read oid 20 snap -1 2021-03-15T22:21:57.029 INFO:tasks.rados.rados.0.smithi076.stdout:554: expect (ObjNum 198 snap 43 seq_num 198) 2021-03-15T22:21:57.029 INFO:tasks.rados.rados.0.smithi076.stdout:555: copy_from oid 39 from oid 17 current snap is 52 2021-03-15T22:21:57.030 INFO:tasks.rados.rados.0.smithi076.stdout:556: read oid 27 snap -1 2021-03-15T22:21:57.030 INFO:tasks.rados.rados.0.smithi076.stdout:556: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:21:57.030 INFO:tasks.rados.rados.0.smithi076.stdout:557: write oid 46 current snap is 52 2021-03-15T22:21:57.030 INFO:tasks.rados.rados.0.smithi076.stdout:557: seq_num 227 ranges {615189=455760,1733883=711078,3134431=460041} 2021-03-15T22:21:57.032 INFO:tasks.rados.rados.0.smithi076.stdout:557: writing smithi07694878-46 from 615189 to 1070949 tid 1 2021-03-15T22:21:57.041 INFO:tasks.rados.rados.0.smithi076.stdout:557: writing smithi07694878-46 from 1733883 to 2444961 tid 2 2021-03-15T22:21:57.049 INFO:tasks.rados.rados.0.smithi076.stdout:557: writing smithi07694878-46 from 3134431 to 3594472 tid 3 2021-03-15T22:21:57.049 INFO:tasks.rados.rados.0.smithi076.stdout:558: write oid 13 current snap is 52 2021-03-15T22:21:57.050 INFO:tasks.rados.rados.0.smithi076.stdout:553: finishing copy_from to smithi07694878-36 2021-03-15T22:21:57.050 INFO:tasks.rados.rados.0.smithi076.stdout:553: got expected ENOENT (src dne) 2021-03-15T22:21:57.050 INFO:tasks.rados.rados.0.smithi076.stdout:553: finishing copy_from racing read to smithi07694878-36 2021-03-15T22:21:57.051 INFO:tasks.rados.rados.0.smithi076.stdout:556: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:21:57.118 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:57.143 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:57.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:57.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:57.216 INFO:tasks.rados.rados.0.smithi076.stdout:557: finishing write tid 1 to smithi07694878-46 2021-03-15T22:21:57.217 INFO:tasks.rados.rados.0.smithi076.stdout:554: expect (ObjNum 198 snap 43 seq_num 198) 2021-03-15T22:21:57.246 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:57.267 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:57.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:57.269 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:57.326 INFO:tasks.rados.rados.0.smithi076.stdout:538: finishing write tid 1 to smithi07694878-28 2021-03-15T22:21:57.327 INFO:tasks.rados.rados.0.smithi076.stdout:540: finishing copy_from to smithi07694878-3 2021-03-15T22:21:57.327 INFO:tasks.rados.rados.0.smithi076.stdout:540: got expected ENOENT (src dne) 2021-03-15T22:21:57.327 INFO:tasks.rados.rados.0.smithi076.stdout:540: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:21:57.327 INFO:tasks.rados.rados.0.smithi076.stdout:558: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2021-03-15T22:21:57.332 INFO:tasks.rados.rados.0.smithi076.stdout:558: writing smithi07694878-13 from 631996 to 1163005 tid 1 2021-03-15T22:21:57.338 INFO:tasks.rados.rados.0.smithi076.stdout:558: writing smithi07694878-13 from 1792365 to 2247101 tid 2 2021-03-15T22:21:57.345 INFO:tasks.rados.rados.0.smithi076.stdout:558: writing smithi07694878-13 from 2761854 to 3433167 tid 3 2021-03-15T22:21:57.345 INFO:tasks.rados.rados.0.smithi076.stdout:540: done (11 left) 2021-03-15T22:21:57.346 INFO:tasks.rados.rados.0.smithi076.stdout:552: done (10 left) 2021-03-15T22:21:57.346 INFO:tasks.rados.rados.0.smithi076.stdout:553: done (9 left) 2021-03-15T22:21:57.346 INFO:tasks.rados.rados.0.smithi076.stdout:554: done (8 left) 2021-03-15T22:21:57.346 INFO:tasks.rados.rados.0.smithi076.stdout:556: done (7 left) 2021-03-15T22:21:57.347 INFO:tasks.rados.rados.0.smithi076.stdout:559: write (excl) oid 1 current snap is 52 2021-03-15T22:21:57.347 INFO:tasks.rados.rados.0.smithi076.stdout:559: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2021-03-15T22:21:57.352 INFO:tasks.rados.rados.0.smithi076.stdout:559: writing smithi07694878-1 from 648803 to 1371414 tid 1 2021-03-15T22:21:57.360 INFO:tasks.rados.rados.0.smithi076.stdout:559: writing smithi07694878-1 from 1967200 to 2681947 tid 2 2021-03-15T22:21:57.368 INFO:tasks.rados.rados.0.smithi076.stdout:559: writing smithi07694878-1 from 3421983 to 3848803 tid 3 2021-03-15T22:21:57.368 INFO:tasks.rados.rados.0.smithi076.stdout:560: delete oid 11 current snap is 52 2021-03-15T22:21:57.369 INFO:tasks.rados.rados.0.smithi076.stdout:557: finishing write tid 2 to smithi07694878-46 2021-03-15T22:21:57.369 INFO:tasks.rados.rados.0.smithi076.stdout:538: finishing write tid 2 to smithi07694878-28 2021-03-15T22:21:57.369 INFO:tasks.rados.rados.0.smithi076.stdout:538: finishing write tid 3 to smithi07694878-28 2021-03-15T22:21:57.369 INFO:tasks.rados.rados.0.smithi076.stdout:538: finishing write tid 4 to smithi07694878-28 2021-03-15T22:21:57.369 INFO:tasks.rados.rados.0.smithi076.stdout:538: finishing write tid 5 to smithi07694878-28 2021-03-15T22:21:57.370 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 304 (ObjNum 221 snap 51 seq_num 221) dirty exists 2021-03-15T22:21:57.370 INFO:tasks.rados.rados.0.smithi076.stdout:538: left oid 28 (ObjNum 221 snap 51 seq_num 221) 2021-03-15T22:21:57.370 INFO:tasks.rados.rados.0.smithi076.stdout:542: finishing copy_from to smithi07694878-21 2021-03-15T22:21:57.370 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 112 (ObjNum 183 snap 40 seq_num 183) dirty exists 2021-03-15T22:21:57.371 INFO:tasks.rados.rados.0.smithi076.stdout:542: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:21:57.371 INFO:tasks.rados.rados.0.smithi076.stdout:544: finishing write tid 1 to smithi07694878-37 2021-03-15T22:21:57.371 INFO:tasks.rados.rados.0.smithi076.stdout:544: finishing write tid 2 to smithi07694878-37 2021-03-15T22:21:57.371 INFO:tasks.rados.rados.0.smithi076.stdout:557: finishing write tid 3 to smithi07694878-46 2021-03-15T22:21:57.371 INFO:tasks.rados.rados.0.smithi076.stdout:557: finishing write tid 4 to smithi07694878-46 2021-03-15T22:21:57.372 INFO:tasks.rados.rados.0.smithi076.stdout:557: finishing write tid 5 to smithi07694878-46 2021-03-15T22:21:57.372 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 189 (ObjNum 227 snap 52 seq_num 227) dirty exists 2021-03-15T22:21:57.372 INFO:tasks.rados.rados.0.smithi076.stdout:557: left oid 46 (ObjNum 227 snap 52 seq_num 227) 2021-03-15T22:21:57.372 INFO:tasks.rados.rados.0.smithi076.stdout:544: finishing write tid 3 to smithi07694878-37 2021-03-15T22:21:57.373 INFO:tasks.rados.rados.0.smithi076.stdout:544: finishing write tid 4 to smithi07694878-37 2021-03-15T22:21:57.373 INFO:tasks.rados.rados.0.smithi076.stdout:544: finishing write tid 5 to smithi07694878-37 2021-03-15T22:21:57.373 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 309 (ObjNum 224 snap 51 seq_num 224) dirty exists 2021-03-15T22:21:57.373 INFO:tasks.rados.rados.0.smithi076.stdout:544: left oid 37 (ObjNum 224 snap 51 seq_num 224) 2021-03-15T22:21:57.373 INFO:tasks.rados.rados.0.smithi076.stdout:555: finishing copy_from to smithi07694878-39 2021-03-15T22:21:57.374 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 57 (ObjNum 220 snap 50 seq_num 220) dirty exists 2021-03-15T22:21:57.374 INFO:tasks.rados.rados.0.smithi076.stdout:555: finishing copy_from racing read to smithi07694878-39 2021-03-15T22:21:57.374 INFO:tasks.rados.rados.0.smithi076.stdout:558: finishing write tid 1 to smithi07694878-13 2021-03-15T22:21:57.374 INFO:tasks.rados.rados.0.smithi076.stdout:558: finishing write tid 2 to smithi07694878-13 2021-03-15T22:21:57.375 INFO:tasks.rados.rados.0.smithi076.stdout:558: finishing write tid 3 to smithi07694878-13 2021-03-15T22:21:57.375 INFO:tasks.rados.rados.0.smithi076.stdout:558: finishing write tid 4 to smithi07694878-13 2021-03-15T22:21:57.375 INFO:tasks.rados.rados.0.smithi076.stdout:558: finishing write tid 5 to smithi07694878-13 2021-03-15T22:21:57.375 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 243 (ObjNum 228 snap 52 seq_num 228) dirty exists 2021-03-15T22:21:57.376 INFO:tasks.rados.rados.0.smithi076.stdout:558: left oid 13 (ObjNum 228 snap 52 seq_num 228) 2021-03-15T22:21:57.376 INFO:tasks.rados.rados.0.smithi076.stdout:559: finishing write tid 1 to smithi07694878-1 2021-03-15T22:21:57.376 INFO:tasks.rados.rados.0.smithi076.stdout:559: finishing write tid 2 to smithi07694878-1 2021-03-15T22:21:57.376 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:57.378 INFO:tasks.rados.rados.0.smithi076.stdout:538: done (8 left) 2021-03-15T22:21:57.378 INFO:tasks.rados.rados.0.smithi076.stdout:542: done (7 left) 2021-03-15T22:21:57.378 INFO:tasks.rados.rados.0.smithi076.stdout:544: done (6 left) 2021-03-15T22:21:57.378 INFO:tasks.rados.rados.0.smithi076.stdout:551: done (5 left) 2021-03-15T22:21:57.379 INFO:tasks.rados.rados.0.smithi076.stdout:555: done (4 left) 2021-03-15T22:21:57.379 INFO:tasks.rados.rados.0.smithi076.stdout:557: done (3 left) 2021-03-15T22:21:57.379 INFO:tasks.rados.rados.0.smithi076.stdout:558: done (2 left) 2021-03-15T22:21:57.379 INFO:tasks.rados.rados.0.smithi076.stdout:560: done (1 left) 2021-03-15T22:21:57.380 INFO:tasks.rados.rados.0.smithi076.stdout:561: write oid 40 current snap is 52 2021-03-15T22:21:57.380 INFO:tasks.rados.rados.0.smithi076.stdout:561: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2021-03-15T22:21:57.380 INFO:tasks.rados.rados.0.smithi076.stdout:561: writing smithi07694878-40 from 682417 to 1155526 tid 1 2021-03-15T22:21:57.381 INFO:tasks.rados.rados.0.smithi076.stdout:561: writing smithi07694878-40 from 1567811 to 2169874 tid 2 2021-03-15T22:21:57.389 INFO:tasks.rados.rados.0.smithi076.stdout:561: writing smithi07694878-40 from 2844123 to 3465605 tid 3 2021-03-15T22:21:57.389 INFO:tasks.rados.rados.0.smithi076.stdout:562: snap_create 2021-03-15T22:21:57.389 INFO:tasks.rados.rados.0.smithi076.stdout:559: finishing write tid 3 to smithi07694878-1 2021-03-15T22:21:57.389 INFO:tasks.rados.rados.0.smithi076.stdout:559: finishing write tid 4 to smithi07694878-1 2021-03-15T22:21:57.390 INFO:tasks.rados.rados.0.smithi076.stdout:559: finishing write tid 5 to smithi07694878-1 2021-03-15T22:21:57.390 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 70 (ObjNum 229 snap 52 seq_num 229) dirty exists 2021-03-15T22:21:57.390 INFO:tasks.rados.rados.0.smithi076.stdout:559: left oid 1 (ObjNum 229 snap 52 seq_num 229) 2021-03-15T22:21:57.390 INFO:tasks.rados.rados.0.smithi076.stdout:561: finishing write tid 1 to smithi07694878-40 2021-03-15T22:21:57.391 INFO:tasks.rados.rados.0.smithi076.stdout:561: finishing write tid 2 to smithi07694878-40 2021-03-15T22:21:57.391 INFO:tasks.rados.rados.0.smithi076.stdout:561: finishing write tid 3 to smithi07694878-40 2021-03-15T22:21:57.392 INFO:tasks.rados.rados.0.smithi076.stdout:561: finishing write tid 4 to smithi07694878-40 2021-03-15T22:21:57.392 INFO:tasks.rados.rados.0.smithi076.stdout:561: finishing write tid 5 to smithi07694878-40 2021-03-15T22:21:57.392 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 314 (ObjNum 231 snap 52 seq_num 231) dirty exists 2021-03-15T22:21:57.393 INFO:tasks.rados.rados.0.smithi076.stdout:561: left oid 40 (ObjNum 231 snap 52 seq_num 231) 2021-03-15T22:21:57.401 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:57.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:57.504 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:57.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:57 smithi076 conmon[56514]: cluster 2021-03-15T22:21:55.867504+0000 2021-03-15T22:21:57.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:57 smithi076 conmon[56514]: osd.1 (osd.1) 43 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:21:57.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:57 smithi076 conmon[56514]: cluster 2021-03-15T22:21:57.025005+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2021-03-15T22:21:57.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:57 smithi076 conmon[62156]: cluster 2021-03-15T22:21:55.867504+0000 osd.1 (osd. 2021-03-15T22:21:57.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:57 smithi076 conmon[62156]: 1) 43 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:21:57.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:57 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:21:57.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:57 smithi076 conmon[62156]: 21:57.025005+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2021-03-15T22:21:57.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:57 smithi076 conmon[62156]: 2021-03-15T22:21:57.526 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:57.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:57.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:57 smithi125 conmon[56739]: cluster 2021-03-15T22:21:55.867504+0000 osd.1 (osd.1 2021-03-15T22:21:57.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:57 smithi125 conmon[56739]: ) 43 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:21:57.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:57 smithi125 conmon[56739]: cluster 2021-03-15T22:21:57.025005+0000 mon.a (mon.0) 820 : cluster [DBG] osdmap e202: 8 total, 8 up, 7 in 2021-03-15T22:21:57.629 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:57.651 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:57.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:57.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:57.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:57.775 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:57.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:57.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:57.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:57.899 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:57.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:57.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:58.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:58.024 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:58.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:58.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:58.030 INFO:tasks.rados.rados.0.smithi076.stdout:559: done (2 left) 2021-03-15T22:21:58.030 INFO:tasks.rados.rados.0.smithi076.stdout:561: done (1 left) 2021-03-15T22:21:58.030 INFO:tasks.rados.rados.0.smithi076.stdout:562: done (0 left) 2021-03-15T22:21:58.031 INFO:tasks.rados.rados.0.smithi076.stdout:563: read oid 22 snap 52 2021-03-15T22:21:58.031 INFO:tasks.rados.rados.0.smithi076.stdout:563: expect deleted 2021-03-15T22:21:58.031 INFO:tasks.rados.rados.0.smithi076.stdout:564: rollback oid 7 current snap is 53 2021-03-15T22:21:58.031 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 7 to 52 2021-03-15T22:21:58.034 INFO:tasks.rados.rados.0.smithi076.stdout:565: copy_from oid 28 from oid 40 current snap is 53 2021-03-15T22:21:58.034 INFO:tasks.rados.rados.0.smithi076.stdout:566: copy_from oid 26 from oid 2 current snap is 53 2021-03-15T22:21:58.034 INFO:tasks.rados.rados.0.smithi076.stdout:567: read oid 17 snap -1 2021-03-15T22:21:58.035 INFO:tasks.rados.rados.0.smithi076.stdout:567: expect (ObjNum 220 snap 50 seq_num 220) 2021-03-15T22:21:58.035 INFO:tasks.rados.rados.0.smithi076.stdout:568: rollback oid 50 current snap is 53 2021-03-15T22:21:58.035 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 50 to 52 2021-03-15T22:21:58.035 INFO:tasks.rados.rados.0.smithi076.stdout:569: write (excl) oid 49 current snap is 53 2021-03-15T22:21:58.036 INFO:tasks.rados.rados.0.smithi076.stdout:569: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2021-03-15T22:21:58.038 INFO:tasks.rados.rados.0.smithi076.stdout:569: writing smithi07694878-49 from 699224 to 1247582 tid 1 2021-03-15T22:21:58.045 INFO:tasks.rados.rados.0.smithi076.stdout:569: writing smithi07694878-49 from 2026293 to 2488367 tid 2 2021-03-15T22:21:58.052 INFO:tasks.rados.rados.0.smithi076.stdout:569: writing smithi07694878-49 from 3271546 to 3704300 tid 3 2021-03-15T22:21:58.053 INFO:tasks.rados.rados.0.smithi076.stdout:570: snap_create 2021-03-15T22:21:58.053 INFO:tasks.rados.rados.0.smithi076.stdout:568: finishing rollback tid 0 to smithi07694878-50 2021-03-15T22:21:58.053 INFO:tasks.rados.rados.0.smithi076.stdout:564: finishing rollback tid 0 to smithi07694878-7 2021-03-15T22:21:58.053 INFO:tasks.rados.rados.0.smithi076.stdout:568: finishing rollback tid 1 to smithi07694878-50 2021-03-15T22:21:58.054 INFO:tasks.rados.rados.0.smithi076.stdout:564: finishing rollback tid 1 to smithi07694878-7 2021-03-15T22:21:58.054 INFO:tasks.rados.rados.0.smithi076.stdout:564: finishing rollback tid 2 to smithi07694878-7 2021-03-15T22:21:58.054 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 242 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:21:58.054 INFO:tasks.rados.rados.0.smithi076.stdout:568: finishing rollback tid 2 to smithi07694878-50 2021-03-15T22:21:58.055 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 153 (ObjNum 170 snap 37 seq_num 170) dirty exists 2021-03-15T22:21:58.055 INFO:tasks.rados.rados.0.smithi076.stdout:569: finishing write tid 1 to smithi07694878-49 2021-03-15T22:21:58.055 INFO:tasks.rados.rados.0.smithi076.stdout:569: finishing write tid 2 to smithi07694878-49 2021-03-15T22:21:58.056 INFO:tasks.rados.rados.0.smithi076.stdout:569: finishing write tid 3 to smithi07694878-49 2021-03-15T22:21:58.056 INFO:tasks.rados.rados.0.smithi076.stdout:569: finishing write tid 4 to smithi07694878-49 2021-03-15T22:21:58.056 INFO:tasks.rados.rados.0.smithi076.stdout:569: finishing write tid 5 to smithi07694878-49 2021-03-15T22:21:58.057 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 272 (ObjNum 232 snap 53 seq_num 232) dirty exists 2021-03-15T22:21:58.057 INFO:tasks.rados.rados.0.smithi076.stdout:569: left oid 49 (ObjNum 232 snap 53 seq_num 232) 2021-03-15T22:21:58.057 INFO:tasks.rados.rados.0.smithi076.stdout:567: expect (ObjNum 220 snap 50 seq_num 220) 2021-03-15T22:21:58.126 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:58.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:21:58.152 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:58.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:58.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:58.255 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:21:58.287 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:58.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:58.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:21:58.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[56514]: cluster 2021-03-15T22:21:57 2021-03-15T22:21:58.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[56514]: .003392+0000 mgr.y (mgr.14144) 453 : cluster [DBG] pgmap v429: 138 pgs: 1 remapped+peering, 2 activating, 1 peering, 1 active+recovery_wait+degraded, 133 active+clean; 113 MiB data, 349 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.4 MiB/s wr, 7 op/s; 11/1062 objects degraded (1.036%); 366 B/s, 4 objects/s recovering 2021-03-15T22:21:58.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[56514]: cluster 2021-03-15T22:21:57.130807+0000 mon.a (mon.0) 821 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:21:58.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[56514]: cluster 2021-03-15T22:21:57.130848+0000 mon.a (mon.0) 822 : cluster [WRN] Health check failed: Degraded data redundancy: 11/1062 objects degraded (1.036%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:21:58.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[56514]: cluster 2021-03-15T22:21:57.510653+0000 osd.4 (osd.4) 49 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:58.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[56514]: cluster 2021-03-15T22:21:57.510790+0000 osd.4 (osd.4) 50 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:58.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[56514]: cluster 2021-03-15T22:21:58.027587+0000 mon.a (mon.0) 823 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2021-03-15T22:21:58.323 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: cluster 2021-03-15T22:21:57.003392+0000 2021-03-15T22:21:58.323 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: mgr.y (mgr.14144) 453 : cluster [DBG] pgmap v429: 138 pgs: 1 remapped+peering, 2 activating, 1 peering, 1 active+recovery_wait+degraded, 133 active+clean; 113 MiB data, 349 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.4 MiB/s wr, 7 op/s; 11/1062 objects degraded (1.036%); 366 B/s, 4 objects/s recovering 2021-03-15T22:21:58.323 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: cluster 2021-03-15T22:21:58.323 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: 2021-03-15T22:21:57.130807+0000 mon.a (mon.0) 821 : cluster 2021-03-15T22:21:58.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:21:58.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: cluster 2021-03-15T22:21 2021-03-15T22:21:58.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: :57.130848+0000 mon.a (mon.0) 2021-03-15T22:21:58.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: 822 : cluster [WRN] Health check failed: Degraded data redundancy: 11/1062 objects degraded (1.036%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:21:58.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:21:58.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: T22:21:57.510653+0000 osd.4 (osd 2021-03-15T22:21:58.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: .4) 49 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:58.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:21:58.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: 03-15T22:21:57.510790+0000 osd.4 2021-03-15T22:21:58.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: (osd.4) 50 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:58.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: 2021-03-15T22:21:58.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: cluster 2021-03-15T22:21:58.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: 2021-03-15T22:21:58.027587+0000 mon.a 2021-03-15T22:21:58.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: (mon.0) 823 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2021-03-15T22:21:58.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[62156]: 2021-03-15T22:21:58.345 INFO:tasks.rados.rados.0.smithi076.stdout:565: finishing copy_from to smithi07694878-28 2021-03-15T22:21:58.345 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 314 (ObjNum 231 snap 52 seq_num 231) dirty exists 2021-03-15T22:21:58.346 INFO:tasks.rados.rados.0.smithi076.stdout:565: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:21:58.346 INFO:tasks.rados.rados.0.smithi076.stdout:566: finishing copy_from to smithi07694878-26 2021-03-15T22:21:58.346 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 101 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:21:58.347 INFO:tasks.rados.rados.0.smithi076.stdout:566: finishing copy_from racing read to smithi07694878-26 2021-03-15T22:21:58.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:58.412 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:58.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:58.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:58.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:21:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: cluster 2021-03-15T22:21:57.003392+0000 mgr.y (mgr 2021-03-15T22:21:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: .14144) 453 : cluster [DBG] pgmap v429: 138 pgs: 1 remapped+peering, 2 activating, 1 peering, 1 active+recovery_wait+degraded, 133 active+clean; 113 MiB data, 349 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.4 MiB/s wr, 7 op/s; 11/1062 objects degraded (1.036%); 366 B/s, 4 objects/s recovering 2021-03-15T22:21:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: cluster 2021-03-15T22:21:57.130807+0000 mon.a (mon.0) 821 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:21:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: cluster 2021-03-15T22:21:57.130848+0000 mon.a 2021-03-15T22:21:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: (mon.0) 822 : cluster [WRN] Health check failed: Degraded data redundancy: 11/1062 objects degraded (1.036%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:21:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:21:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: 03-15T22:21:57.510653+0000 osd.4 (osd 2021-03-15T22:21:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: .4) 49 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:21:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:21:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: -15T22:21:57.510790+0000 osd.4 (osd 2021-03-15T22:21:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: .4) 50 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:21:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:21:58.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: 03-15T22:21:58.027587+0000 mon.a ( 2021-03-15T22:21:58.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:58 smithi125 conmon[56739]: mon.0) 823 : cluster [DBG] osdmap e203: 8 total, 8 up, 7 in 2021-03-15T22:21:58.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:58.540 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:58.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:21:58.543 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:21:58.605 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:21:58.635 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:21:58.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:21:58.656 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:21:58.663 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:58.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:58.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:21:58.672 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:21:58.691 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:21:58.708 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:21:58.725 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:21:58.741 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:21:58.743 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:21:58.743 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:21:58.743 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:21:58.743 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:21:58.744 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:21:58.744 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:21:58.744 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:21:58.744 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:21:58.766 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:58.791 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:58.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:58.892 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:58.915 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:58.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:58.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:59.016 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:21:58 smithi076 conmon[65511]: debug 2021-03-15T22:21:58.812+0000 7f1a26084700 -1 osd.0 pg_epoch: 203 pg[6.9( empty local-lis/les=62/63 n=0 ec=62/62 lis/c=62/62 les/c/f=63/63/0 sis=62) [0,7,2] r=0 lpr=62 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.9) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x557aca09ab00) 2021-03-15T22:21:59.017 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:59.030 INFO:tasks.rados.rados.0.smithi076.stdout:563: done (7 left) 2021-03-15T22:21:59.031 INFO:tasks.rados.rados.0.smithi076.stdout:564: done (6 left) 2021-03-15T22:21:59.031 INFO:tasks.rados.rados.0.smithi076.stdout:565: done (5 left) 2021-03-15T22:21:59.031 INFO:tasks.rados.rados.0.smithi076.stdout:566: done (4 left) 2021-03-15T22:21:59.031 INFO:tasks.rados.rados.0.smithi076.stdout:567: done (3 left) 2021-03-15T22:21:59.032 INFO:tasks.rados.rados.0.smithi076.stdout:568: done (2 left) 2021-03-15T22:21:59.032 INFO:tasks.rados.rados.0.smithi076.stdout:569: done (1 left) 2021-03-15T22:21:59.032 INFO:tasks.rados.rados.0.smithi076.stdout:570: done (0 left) 2021-03-15T22:21:59.032 INFO:tasks.rados.rados.0.smithi076.stdout:571: rollback oid 25 current snap is 54 2021-03-15T22:21:59.032 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 52 2021-03-15T22:21:59.033 INFO:tasks.rados.rados.0.smithi076.stdout:572: copy_from oid 49 from oid 36 current snap is 54 2021-03-15T22:21:59.033 INFO:tasks.rados.rados.0.smithi076.stdout:573: delete oid 9 current snap is 54 2021-03-15T22:21:59.040 INFO:tasks.rados.rados.0.smithi076.stdout:573: done (2 left) 2021-03-15T22:21:59.040 INFO:tasks.rados.rados.0.smithi076.stdout:574: write oid 43 current snap is 54 2021-03-15T22:21:59.040 INFO:tasks.rados.rados.0.smithi076.stdout:574: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2021-03-15T22:21:59.042 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:21:59.045 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:59.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:59.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:21:59.047 INFO:tasks.rados.rados.0.smithi076.stdout:574: writing smithi07694878-43 from 732838 to 1431694 tid 1 2021-03-15T22:21:59.052 INFO:tasks.rados.rados.0.smithi076.stdout:574: writing smithi07694878-43 from 2026904 to 2492647 tid 2 2021-03-15T22:21:59.060 INFO:tasks.rados.rados.0.smithi076.stdout:574: writing smithi07694878-43 from 3210039 to 3781690 tid 3 2021-03-15T22:21:59.060 INFO:tasks.rados.rados.0.smithi076.stdout:575: copy_from oid 11 from oid 12 current snap is 54 2021-03-15T22:21:59.061 INFO:tasks.rados.rados.0.smithi076.stdout:571: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:21:59.061 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 0 (ObjNum 255155200 snap 21896 seq_num 4000000) dirty dne 2021-03-15T22:21:59.061 INFO:tasks.rados.rados.0.smithi076.stdout:574: finishing write tid 1 to smithi07694878-43 2021-03-15T22:21:59.061 INFO:tasks.rados.rados.0.smithi076.stdout:572: finishing copy_from to smithi07694878-49 2021-03-15T22:21:59.062 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 134 (ObjNum 180 snap 39 seq_num 180) dirty exists 2021-03-15T22:21:59.062 INFO:tasks.rados.rados.0.smithi076.stdout:574: finishing write tid 2 to smithi07694878-43 2021-03-15T22:21:59.062 INFO:tasks.rados.rados.0.smithi076.stdout:572: finishing copy_from racing read to smithi07694878-49 2021-03-15T22:21:59.062 INFO:tasks.rados.rados.0.smithi076.stdout:571: done (3 left) 2021-03-15T22:21:59.062 INFO:tasks.rados.rados.0.smithi076.stdout:572: done (2 left) 2021-03-15T22:21:59.063 INFO:tasks.rados.rados.0.smithi076.stdout:576: rollback oid 25 current snap is 54 2021-03-15T22:21:59.063 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 52 2021-03-15T22:21:59.063 INFO:tasks.rados.rados.0.smithi076.stdout:577: read oid 22 snap 51 2021-03-15T22:21:59.063 INFO:tasks.rados.rados.0.smithi076.stdout:577: expect deleted 2021-03-15T22:21:59.063 INFO:tasks.rados.rados.0.smithi076.stdout:578: read oid 28 snap -1 2021-03-15T22:21:59.064 INFO:tasks.rados.rados.0.smithi076.stdout:578: expect (ObjNum 231 snap 52 seq_num 231) 2021-03-15T22:21:59.064 INFO:tasks.rados.rados.0.smithi076.stdout:579: read oid 48 snap -1 2021-03-15T22:21:59.064 INFO:tasks.rados.rados.0.smithi076.stdout:579: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:21:59.065 INFO:tasks.rados.rados.0.smithi076.stdout:580: delete oid 38 current snap is 54 2021-03-15T22:21:59.065 INFO:tasks.rados.rados.0.smithi076.stdout:574: finishing write tid 3 to smithi07694878-43 2021-03-15T22:21:59.067 INFO:tasks.rados.rados.0.smithi076.stdout:574: finishing write tid 4 to smithi07694878-43 2021-03-15T22:21:59.067 INFO:tasks.rados.rados.0.smithi076.stdout:574: finishing write tid 5 to smithi07694878-43 2021-03-15T22:21:59.067 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 106 (ObjNum 234 snap 54 seq_num 234) dirty exists 2021-03-15T22:21:59.067 INFO:tasks.rados.rados.0.smithi076.stdout:574: left oid 43 (ObjNum 234 snap 54 seq_num 234) 2021-03-15T22:21:59.068 INFO:tasks.rados.rados.0.smithi076.stdout:574: done (6 left) 2021-03-15T22:21:59.068 INFO:tasks.rados.rados.0.smithi076.stdout:577: done (5 left) 2021-03-15T22:21:59.068 INFO:tasks.rados.rados.0.smithi076.stdout:580: done (4 left) 2021-03-15T22:21:59.068 INFO:tasks.rados.rados.0.smithi076.stdout:581: read oid 3 snap 53 2021-03-15T22:21:59.069 INFO:tasks.rados.rados.0.smithi076.stdout:581: expect (ObjNum 113 snap 24 seq_num 113) 2021-03-15T22:21:59.069 INFO:tasks.rados.rados.0.smithi076.stdout:582: snap_remove snap 51 2021-03-15T22:21:59.148 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:59.169 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:59.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:59.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:21:59.294 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:59.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:59.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:21:59.397 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:21:59.419 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:59.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:59.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:21:59.511 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: audit 2021-03-15T22:21:58.569311+0000 mon.a (mon.0) 824 : audit [DBG] 2021-03-15T22:21:59.512 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2433854930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:59.512 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: audit 2021-03-15T22:21:58.618346+0000 mon.c (mon.1) 20 : audit [INF] from='client.? 172.21.15.76:0/4011107711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:59.512 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: audit 2021-03-15T22:21:58.618641+0000 mon.a (mon.0) 825 : 2021-03-15T22:21:59.512 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:59.513 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: cluster 2021-03-15T22:21:59.022880+0000 mon.a (mon.0) 826 2021-03-15T22:21:59.513 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:21:59.513 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:21:59.513 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: -15T22:21:59.028216+0000 mon.a (mon.0) 827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:59.513 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: 2021-03-15T22:21:59.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:21:59.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: -03-15T22:21:59.028257+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2021-03-15T22:21:59.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[56739]: 2021-03-15T22:21:59.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[56514]: audit 2021-03-15T22:21:58.569311+0000 mon.a (mon.0) 824 : audit [DBG] from='client.? 172.21.15.76:0/2433854930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:59.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[56514]: audit 2021-03-15T22:21:58.618346+0000 mon.c (mon.1) 20 : audit [INF] from='client.? 172.21.15.76:0/4011107711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:59.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[56514]: audit 2021-03-15T22:21:58.618641+0000 mon.a (mon.0) 825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:59.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[56514]: cluster 2021-03-15T22:21:59.022880+0000 mon.a (mon.0) 826 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:21:59.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[56514]: audit 2021-03-15T22:21:59.028216+0000 mon.a (mon.0) 827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:59.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[56514]: cluster 2021-03-15T22:21:59.028257+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2021-03-15T22:21:59.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[62156]: audit 2021-03-15T22:21:58.569311+0000 mon.a (mon.0) 824 : audit [DBG] from='client.? 172.21.15.76:0/2433854930' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:21:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[62156]: audit 2021-03-15T22:21:58.618346+0000 mon.c (mon.1) 20 : audit [INF] from='client.? 172.21.15.76:0/4011107711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[62156]: audit 2021-03-15T22:21:58.618641+0000 mon.a (mon.0) 825 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:21:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[62156]: cluster 2021-03-15T22:21:59.022880+0000 mon.a (mon.0) 826 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:21:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[62156]: audit 2021-03-15T22:21:59.028216+0000 mon.a (mon.0) 827 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:21:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:21:59 smithi076 conmon[62156]: cluster 2021-03-15T22:21:59.028257+0000 mon.a (mon.0) 828 : cluster [DBG] osdmap e204: 8 total, 8 up, 7 in 2021-03-15T22:21:59.521 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:59.546 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:59.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:59.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:59.648 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:21:59.670 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:21:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:59.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:21:59.773 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:21:59.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:21:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:21:59.795 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:59.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:59.796 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:21:59.897 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:21:59.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:21:59.920 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:21:59.921 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:00.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:00.031 INFO:tasks.rados.rados.0.smithi076.stdout:582: done (5 left) 2021-03-15T22:22:00.031 INFO:tasks.rados.rados.0.smithi076.stdout:583: snap_create 2021-03-15T22:22:00.031 INFO:tasks.rados.rados.0.smithi076.stdout:576: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:22:00.032 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 0 (ObjNum 255155200 snap 21896 seq_num 4000000) dirty dne 2021-03-15T22:22:00.032 INFO:tasks.rados.rados.0.smithi076.stdout:579: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:22:00.045 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:00.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:00.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:00.148 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:00.169 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:00.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:00.171 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:00.182 INFO:tasks.rados.rados.0.smithi076.stdout:578: expect (ObjNum 231 snap 52 seq_num 231) 2021-03-15T22:22:00.272 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:00.293 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:00.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:00.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:00.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:00.409 INFO:tasks.rados.rados.0.smithi076.stdout:575: finishing copy_from to smithi07694878-11 2021-03-15T22:22:00.410 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 242 (ObjNum 199 snap 43 seq_num 199) dirty exists 2021-03-15T22:22:00.410 INFO:tasks.rados.rados.0.smithi076.stdout:575: finishing copy_from racing read to smithi07694878-11 2021-03-15T22:22:00.410 INFO:tasks.rados.rados.0.smithi076.stdout:581: expect (ObjNum 113 snap 24 seq_num 113) 2021-03-15T22:22:00.421 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:00.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:00.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:00.495 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:00 smithi076 conmon[56514]: cluster 2021-03-15T22:21 2021-03-15T22:22:00.495 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:00 smithi076 conmon[56514]: :59.003809+0000 mgr.y (mgr.14144) 454 : cluster [DBG] pgmap v432: 138 pgs: 1 remapped+peering, 2 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 132 active+clean; 118 MiB data, 367 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 5.3 MiB/s wr, 16 op/s; 15/1065 objects degraded (1.408%); 2/1065 objects misplaced (0.188%); 836 KiB/s, 16 objects/s recovering 2021-03-15T22:22:00.496 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:00 smithi076 conmon[56514]: cluster 2021-03-15T22:21:59.228446+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.1 deep-scrub ok 2021-03-15T22:22:00.496 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:00 smithi076 conmon[56514]: audit 2021-03-15T22:21:59.510516+0000 mgr.y (mgr.14144) 455 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:00.496 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:00 smithi076 conmon[56514]: cluster 2021-03-15T22:22:00.026376+0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2021-03-15T22:22:00.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:00 smithi076 conmon[62156]: cluster 2021-03-15T22:21:59 2021-03-15T22:22:00.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:00 smithi076 conmon[62156]: .003809+0000 mgr.y (mgr.14144) 454 : cluster [DBG] pgmap v432: 138 pgs: 1 remapped+peering, 2 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 132 active+clean; 118 MiB data, 367 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 5.3 MiB/s wr, 16 op/s; 15/1065 objects degraded (1.408%); 2/1065 objects misplaced (0.188%); 836 KiB/s, 16 objects/s recovering 2021-03-15T22:22:00.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:00 smithi076 conmon[62156]: cluster 2021-03-15T22:21:59.228446+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.1 deep-scrub ok 2021-03-15T22:22:00.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:00 smithi076 conmon[62156]: audit 2021-03-15T22:21:59.510516+0000 mgr.y (mgr.14144) 455 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:00.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:00 smithi076 conmon[62156]: cluster 2021-03-15T22:22:00.026376+0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2021-03-15T22:22:00.523 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:00.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:00 smithi125 conmon[56739]: cluster 2021-03-15T22:21:59.003809+0000 mgr.y (mgr.14144) 454 : cluster [DBG] pgmap v432: 138 pgs: 1 remapped+peering, 2 activating, 1 peering, 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+degraded, 132 active+clean; 118 MiB data, 367 MiB used, 625 GiB / 626 GiB avail; 3.4 MiB/s rd, 5.3 MiB/s wr, 16 op/s; 15/1065 objects degraded (1.408%); 2/1065 objects misplaced (0.188%); 836 KiB/s, 16 objects/s recovering 2021-03-15T22:22:00.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:00 smithi125 conmon[56739]: cluster 2021-03-15T22:21 2021-03-15T22:22:00.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:00 smithi125 conmon[56739]: :59.228446+0000 osd.4 (osd.4) 51 : cluster [DBG] 5.1 deep-scrub ok 2021-03-15T22:22:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:00 smithi125 conmon[56739]: audit 2021-03-15T22:21:59.510516+0000 mgr.y (mgr.14144) 455 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:00 smithi125 conmon[56739]: cluster 2021-03-15T22:22:00.026376 2021-03-15T22:22:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:00 smithi125 conmon[56739]: +0000 mon.a (mon.0) 829 : cluster [DBG] osdmap e205: 8 total, 8 up, 7 in 2021-03-15T22:22:00.546 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:00.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:00.547 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:00.648 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:00.671 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:00.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:00.672 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:00.773 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:00.798 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:00.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:00.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:00.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:00.915 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:22:00.916 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:22:00.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:22:00.916 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:22:00.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.917 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:242 7.b 7:d980be1f:::smithi07694878-23:head [read 0~2369754,checksum] snapc 0=[] ondisk+read+known_if_redirected e102)", 2021-03-15T22:22:00.917 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:16.998381+0000", 2021-03-15T22:22:00.917 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 103.913649617, 2021-03-15T22:22:00.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063374360000000001, 2021-03-15T22:22:00.918 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:00.918 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:00.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:00.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:00.919 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:00.919 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 242 2021-03-15T22:22:00.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.919 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:00.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:00.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:16.998381+0000", 2021-03-15T22:22:00.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:00.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:00.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:16.998381+0000", 2021-03-15T22:22:00.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2390000000000001e-06 2021-03-15T22:22:00.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:00.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:16.998384+0000", 2021-03-15T22:22:00.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.539e-06 2021-03-15T22:22:00.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:00.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:16.998385+0000", 2021-03-15T22:22:00.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9830000000000001e-06 2021-03-15T22:22:00.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:00.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:16.998391+0000", 2021-03-15T22:22:00.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7790000000000002e-06 2021-03-15T22:22:00.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:00.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:16.998395+0000", 2021-03-15T22:22:00.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054384630000000002 2021-03-15T22:22:00.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:00.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:17.003833+0000", 2021-03-15T22:22:00.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7662000000000003e-05 2021-03-15T22:22:00.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:00.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:17.003911+0000", 2021-03-15T22:22:00.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00080777099999999999 2021-03-15T22:22:00.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:00.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:17.004719+0000", 2021-03-15T22:22:00.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063374360000000001 2021-03-15T22:22:00.929 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.930 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:00.930 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.931 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:255 7.4 e103/91 7:26337d05:::smithi07694878-20:head v 103'22, mlcod=100'19)", 2021-03-15T22:22:00.931 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:18.262016+0000", 2021-03-15T22:22:00.931 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 102.650014885, 2021-03-15T22:22:00.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066155169999999996, 2021-03-15T22:22:00.931 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:00.932 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:00.932 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:00.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:00.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.262016+0000", 2021-03-15T22:22:00.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:00.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:00.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.262016+0000", 2021-03-15T22:22:00.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012543529999999999 2021-03-15T22:22:00.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:00.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.263270+0000", 2021-03-15T22:22:00.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7870000000000001e-06 2021-03-15T22:22:00.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:00.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.263275+0000", 2021-03-15T22:22:00.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3642000000000002e-05 2021-03-15T22:22:00.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:00.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.263299+0000", 2021-03-15T22:22:00.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8550000000000002e-06 2021-03-15T22:22:00.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:00.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.263307+0000", 2021-03-15T22:22:00.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6878999999999994e-05 2021-03-15T22:22:00.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:00.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.263374+0000", 2021-03-15T22:22:00.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7889e-05 2021-03-15T22:22:00.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:00.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.263402+0000", 2021-03-15T22:22:00.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051914040000000002 2021-03-15T22:22:00.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:00.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.268593+0000", 2021-03-15T22:22:00.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8708000000000003e-05 2021-03-15T22:22:00.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:00.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.268632+0000", 2021-03-15T22:22:00.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066155169999999996 2021-03-15T22:22:00.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.943 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:00.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.944 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:294 7.0 e104/91 7:0cc37e3d:::smithi07694878-21:head v 104'30, mlcod=104'26)", 2021-03-15T22:22:00.944 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:19.405564+0000", 2021-03-15T22:22:00.944 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 101.506467378, 2021-03-15T22:22:00.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072789500000000002, 2021-03-15T22:22:00.945 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:00.945 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:00.945 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:00.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:00.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.405564+0000", 2021-03-15T22:22:00.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:00.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:00.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.405564+0000", 2021-03-15T22:22:00.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.004506543 2021-03-15T22:22:00.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:00.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.410070+0000", 2021-03-15T22:22:00.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2629999999999997e-06 2021-03-15T22:22:00.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:00.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.410077+0000", 2021-03-15T22:22:00.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2640999999999999e-05 2021-03-15T22:22:00.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:00.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.410100+0000", 2021-03-15T22:22:00.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8690000000000004e-06 2021-03-15T22:22:00.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:00.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.410108+0000", 2021-03-15T22:22:00.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6956999999999998e-05 2021-03-15T22:22:00.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:00.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.410155+0000", 2021-03-15T22:22:00.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0819999999999998e-05 2021-03-15T22:22:00.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:00.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.410206+0000", 2021-03-15T22:22:00.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025863510000000002 2021-03-15T22:22:00.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:00.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.412792+0000", 2021-03-15T22:22:00.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0506000000000001e-05 2021-03-15T22:22:00.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:00.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.412843+0000", 2021-03-15T22:22:00.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072789500000000002 2021-03-15T22:22:00.956 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.956 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:00.957 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.957 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.5.0:63 7.0 e108/91 7:08414d91:::smithi07694878-42:8 v 108'36, mlcod=104'33)", 2021-03-15T22:22:00.958 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:23.006509+0000", 2021-03-15T22:22:00.958 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 97.905521848999996, 2021-03-15T22:22:00.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008624853, 2021-03-15T22:22:00.958 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:00.958 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:00.959 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:00.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:00.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006509+0000", 2021-03-15T22:22:00.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:00.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:00.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006509+0000", 2021-03-15T22:22:00.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8430000000000003e-06 2021-03-15T22:22:00.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:00.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006513+0000", 2021-03-15T22:22:00.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999998e-06 2021-03-15T22:22:00.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006516+0000", 2021-03-15T22:22:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1904e-05 2021-03-15T22:22:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:00.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006528+0000", 2021-03-15T22:22:00.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6889999999999998e-06 2021-03-15T22:22:00.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:00.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006535+0000", 2021-03-15T22:22:00.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1385999999999997e-05 2021-03-15T22:22:00.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:00.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006606+0000", 2021-03-15T22:22:00.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4292000000000001e-05 2021-03-15T22:22:00.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:00.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006630+0000", 2021-03-15T22:22:00.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084716670000000004 2021-03-15T22:22:00.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:00.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.015102+0000", 2021-03-15T22:22:00.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2125999999999997e-05 2021-03-15T22:22:00.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:00.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.015134+0000", 2021-03-15T22:22:00.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008624853 2021-03-15T22:22:00.971 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.971 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:00.971 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.972 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.5.0:64 7.0 e108/91 7:0cc37e3d:::smithi07694878-21:8 v 108'38, mlcod=104'33)", 2021-03-15T22:22:00.972 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:23.007846+0000", 2021-03-15T22:22:00.972 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 97.904184634000003, 2021-03-15T22:22:00.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073126550000000004, 2021-03-15T22:22:00.973 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:00.973 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:00.973 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:00.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:00.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.007846+0000", 2021-03-15T22:22:00.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:00.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:00.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.007846+0000", 2021-03-15T22:22:00.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9639999999999997e-06 2021-03-15T22:22:00.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:00.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.007850+0000", 2021-03-15T22:22:00.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4209999999999999e-06 2021-03-15T22:22:00.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:00.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.007854+0000", 2021-03-15T22:22:00.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0661e-05 2021-03-15T22:22:00.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:00.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.007864+0000", 2021-03-15T22:22:00.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9000000000000003e-06 2021-03-15T22:22:00.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:00.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.007870+0000", 2021-03-15T22:22:00.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006604469 2021-03-15T22:22:00.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:00.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.014475+0000", 2021-03-15T22:22:00.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2901999999999999e-05 2021-03-15T22:22:00.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:00.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.014498+0000", 2021-03-15T22:22:00.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00064214800000000005 2021-03-15T22:22:00.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:00.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.015140+0000", 2021-03-15T22:22:00.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9190000000000001e-05 2021-03-15T22:22:00.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:00.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.015159+0000", 2021-03-15T22:22:00.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073126550000000004 2021-03-15T22:22:00.984 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.984 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:00.984 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.985 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:366 7.0 e108/91 7:0732c451:::smithi07694878-32:head v 108'44, mlcod=108'40)", 2021-03-15T22:22:00.985 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:23.113297+0000", 2021-03-15T22:22:00.985 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 97.798734374000006, 2021-03-15T22:22:00.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070584910000000001, 2021-03-15T22:22:00.985 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:00.986 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:00.986 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:00.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:00.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.113297+0000", 2021-03-15T22:22:00.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:00.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:00.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.113297+0000", 2021-03-15T22:22:00.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038174860000000001 2021-03-15T22:22:00.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:00.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.117114+0000", 2021-03-15T22:22:00.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7069999999999998e-06 2021-03-15T22:22:00.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:00.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.117118+0000", 2021-03-15T22:22:00.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0689e-05 2021-03-15T22:22:00.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:00.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.117129+0000", 2021-03-15T22:22:00.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7039999999999999e-06 2021-03-15T22:22:00.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:00.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.117134+0000", 2021-03-15T22:22:00.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7185999999999998e-05 2021-03-15T22:22:00.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:00.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.117191+0000", 2021-03-15T22:22:00.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0871000000000001e-05 2021-03-15T22:22:00.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:00.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.117212+0000", 2021-03-15T22:22:00.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0031030340000000002 2021-03-15T22:22:00.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:00.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.120315+0000", 2021-03-15T22:22:00.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9814000000000002e-05 2021-03-15T22:22:00.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:00.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.120355+0000", 2021-03-15T22:22:00.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070584910000000001 2021-03-15T22:22:00.997 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.997 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:00.997 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:00.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:00.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:00.998 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:372 7.0 e108/91 7:054fa7db:::smithi07694878-29:head v 108'48, mlcod=108'42)", 2021-03-15T22:22:00.998 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:23.130569+0000", 2021-03-15T22:22:00.998 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 97.781461602999997, 2021-03-15T22:22:00.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065251220000000004, 2021-03-15T22:22:00.999 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:00.999 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:00.999 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:00.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.130569+0000", 2021-03-15T22:22:01.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.130569+0000", 2021-03-15T22:22:01.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0030677999999999999 2021-03-15T22:22:01.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.133637+0000", 2021-03-15T22:22:01.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6080000000000003e-06 2021-03-15T22:22:01.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.133643+0000", 2021-03-15T22:22:01.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3811e-05 2021-03-15T22:22:01.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.133667+0000", 2021-03-15T22:22:01.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9890000000000003e-06 2021-03-15T22:22:01.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.133674+0000", 2021-03-15T22:22:01.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6975000000000003e-05 2021-03-15T22:22:01.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.133731+0000", 2021-03-15T22:22:01.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9700999999999999e-05 2021-03-15T22:22:01.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.133760+0000", 2021-03-15T22:22:01.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032838479999999998 2021-03-15T22:22:01.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:01.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.137044+0000", 2021-03-15T22:22:01.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0389999999999997e-05 2021-03-15T22:22:01.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.137095+0000", 2021-03-15T22:22:01.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065251220000000004 2021-03-15T22:22:01.010 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.010 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.010 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.011 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:539 7.2 e123/91 7:45ee0f2f:::smithi07694878-6:head v 123'14, mlcod=104'10)", 2021-03-15T22:22:01.011 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:38.116343+0000", 2021-03-15T22:22:01.011 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 82.795687821000001, 2021-03-15T22:22:01.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01064652, 2021-03-15T22:22:01.012 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.012 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:01.012 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.116343+0000", 2021-03-15T22:22:01.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.116343+0000", 2021-03-15T22:22:01.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017858069999999999 2021-03-15T22:22:01.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118129+0000", 2021-03-15T22:22:01.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5420000000000002e-06 2021-03-15T22:22:01.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.016 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118136+0000", 2021-03-15T22:22:01.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9451999999999999e-05 2021-03-15T22:22:01.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118165+0000", 2021-03-15T22:22:01.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3950000000000007e-06 2021-03-15T22:22:01.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118174+0000", 2021-03-15T22:22:01.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5652000000000002e-05 2021-03-15T22:22:01.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118230+0000", 2021-03-15T22:22:01.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1316999999999999e-05 2021-03-15T22:22:01.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118251+0000", 2021-03-15T22:22:01.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086919890000000007 2021-03-15T22:22:01.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:01.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.126943+0000", 2021-03-15T22:22:01.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6366000000000003e-05 2021-03-15T22:22:01.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.126990+0000", 2021-03-15T22:22:01.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01064652 2021-03-15T22:22:01.023 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.023 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.023 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.024 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.5.0:115 7.0 e136/91 7:0732c451:::smithi07694878-32:26 v 136'114, mlcod=136'110)", 2021-03-15T22:22:01.024 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:51.073130+0000", 2021-03-15T22:22:01.024 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 69.838901438999997, 2021-03-15T22:22:01.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076143579999999999, 2021-03-15T22:22:01.025 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.025 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:01.026 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.073130+0000", 2021-03-15T22:22:01.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.073130+0000", 2021-03-15T22:22:01.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4419999999999998e-06 2021-03-15T22:22:01.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.073133+0000", 2021-03-15T22:22:01.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7750000000000001e-06 2021-03-15T22:22:01.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.073136+0000", 2021-03-15T22:22:01.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3963e-05 2021-03-15T22:22:01.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.073150+0000", 2021-03-15T22:22:01.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1380000000000004e-06 2021-03-15T22:22:01.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.073156+0000", 2021-03-15T22:22:01.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001385126 2021-03-15T22:22:01.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.074541+0000", 2021-03-15T22:22:01.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5429000000000002e-05 2021-03-15T22:22:01.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.074576+0000", 2021-03-15T22:22:01.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061296509999999998 2021-03-15T22:22:01.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:01.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.080706+0000", 2021-03-15T22:22:01.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7833999999999997e-05 2021-03-15T22:22:01.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.080744+0000", 2021-03-15T22:22:01.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076143579999999999 2021-03-15T22:22:01.037 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.037 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.037 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.038 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.0.0:443 7.4 e173/91 7:20074f29:::smithi07694878-34:44 v 173'120, mlcod=171'116)", 2021-03-15T22:22:01.038 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:28.038322+0000", 2021-03-15T22:22:01.039 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.87370876, 2021-03-15T22:22:01.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077940370000000002, 2021-03-15T22:22:01.039 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.039 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:01.039 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038322+0000", 2021-03-15T22:22:01.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038322+0000", 2021-03-15T22:22:01.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8400000000000007e-06 2021-03-15T22:22:01.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038332+0000", 2021-03-15T22:22:01.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.305e-06 2021-03-15T22:22:01.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038334+0000", 2021-03-15T22:22:01.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3090000000000001e-06 2021-03-15T22:22:01.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038344+0000", 2021-03-15T22:22:01.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1000000000000003e-06 2021-03-15T22:22:01.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038349+0000", 2021-03-15T22:22:01.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057226669999999999 2021-03-15T22:22:01.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.044071+0000", 2021-03-15T22:22:01.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5285999999999999e-05 2021-03-15T22:22:01.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.044087+0000", 2021-03-15T22:22:01.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019995740000000001 2021-03-15T22:22:01.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:01.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.046086+0000", 2021-03-15T22:22:01.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9955999999999999e-05 2021-03-15T22:22:01.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.046116+0000", 2021-03-15T22:22:01.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077940370000000002 2021-03-15T22:22:01.050 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.051 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.051 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.0.0:444 7.4 e173/91 7:24f33681:::smithi07694878-33:44 v 173'122, mlcod=171'116)", 2021-03-15T22:22:01.052 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:28.038436+0000", 2021-03-15T22:22:01.052 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.873594992000001, 2021-03-15T22:22:01.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014810036, 2021-03-15T22:22:01.052 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.053 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:01.053 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038436+0000", 2021-03-15T22:22:01.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038436+0000", 2021-03-15T22:22:01.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3089999999999998e-06 2021-03-15T22:22:01.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038438+0000", 2021-03-15T22:22:01.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2021-03-15T22:22:01.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038440+0000", 2021-03-15T22:22:01.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.81e-06 2021-03-15T22:22:01.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038447+0000", 2021-03-15T22:22:01.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4270000000000001e-06 2021-03-15T22:22:01.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038452+0000", 2021-03-15T22:22:01.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0058912499999999998 2021-03-15T22:22:01.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.044343+0000", 2021-03-15T22:22:01.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0003000000000001e-05 2021-03-15T22:22:01.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.044363+0000", 2021-03-15T22:22:01.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088552849999999992 2021-03-15T22:22:01.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:01.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.053218+0000", 2021-03-15T22:22:01.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8017999999999998e-05 2021-03-15T22:22:01.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.053246+0000", 2021-03-15T22:22:01.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014810036 2021-03-15T22:22:01.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.064 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.065 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.5.0:201 7.0 e173/91 7:00c03349:::smithi07694878-40:44 v 173'256, mlcod=171'252)", 2021-03-15T22:22:01.065 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:28.043898+0000", 2021-03-15T22:22:01.065 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.868133473999997, 2021-03-15T22:22:01.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008587819, 2021-03-15T22:22:01.066 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.066 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:01.066 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043898+0000", 2021-03-15T22:22:01.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043898+0000", 2021-03-15T22:22:01.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2150000000000001e-06 2021-03-15T22:22:01.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043901+0000", 2021-03-15T22:22:01.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.542e-06 2021-03-15T22:22:01.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043903+0000", 2021-03-15T22:22:01.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8909999999999995e-06 2021-03-15T22:22:01.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043911+0000", 2021-03-15T22:22:01.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1120000000000004e-06 2021-03-15T22:22:01.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043916+0000", 2021-03-15T22:22:01.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138269 2021-03-15T22:22:01.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.044055+0000", 2021-03-15T22:22:01.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0902000000000001e-05 2021-03-15T22:22:01.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.044075+0000", 2021-03-15T22:22:01.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083778680000000001 2021-03-15T22:22:01.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:01.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.052453+0000", 2021-03-15T22:22:01.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2020000000000002e-05 2021-03-15T22:22:01.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.076 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.076 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.052485+0000", 2021-03-15T22:22:01.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008587819 2021-03-15T22:22:01.077 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.077 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.077 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.078 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.5.0:202 7.0 e173/91 7:0cc37e3d:::smithi07694878-21:44 v 173'258, mlcod=171'252)", 2021-03-15T22:22:01.078 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:28.043926+0000", 2021-03-15T22:22:01.078 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.868105221, 2021-03-15T22:22:01.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085792720000000006, 2021-03-15T22:22:01.079 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.079 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:01.079 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043926+0000", 2021-03-15T22:22:01.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043926+0000", 2021-03-15T22:22:01.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7020000000000001e-06 2021-03-15T22:22:01.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.082 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043933+0000", 2021-03-15T22:22:01.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1050000000000002e-06 2021-03-15T22:22:01.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043936+0000", 2021-03-15T22:22:01.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2140000000000001e-06 2021-03-15T22:22:01.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043942+0000", 2021-03-15T22:22:01.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8210000000000003e-06 2021-03-15T22:22:01.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043946+0000", 2021-03-15T22:22:01.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024850649999999998 2021-03-15T22:22:01.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.046431+0000", 2021-03-15T22:22:01.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9768000000000001e-05 2021-03-15T22:22:01.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.088 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.046450+0000", 2021-03-15T22:22:01.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060373780000000004 2021-03-15T22:22:01.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:01.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.052488+0000", 2021-03-15T22:22:01.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7218999999999999e-05 2021-03-15T22:22:01.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.052505+0000", 2021-03-15T22:22:01.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085792720000000006 2021-03-15T22:22:01.090 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.090 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.091 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.091 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:1215 7.7 e179/172 7:ed4a01d2:::smithi07694878-2:head v 179'52, mlcod=173'49)", 2021-03-15T22:22:01.091 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:34.045906+0000", 2021-03-15T22:22:01.092 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 26.86612457, 2021-03-15T22:22:01.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013063455999999999, 2021-03-15T22:22:01.092 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.092 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:01.092 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.045906+0000", 2021-03-15T22:22:01.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.045906+0000", 2021-03-15T22:22:01.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056384349999999998 2021-03-15T22:22:01.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051545+0000", 2021-03-15T22:22:01.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5680000000000001e-06 2021-03-15T22:22:01.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051549+0000", 2021-03-15T22:22:01.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6931000000000001e-05 2021-03-15T22:22:01.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051566+0000", 2021-03-15T22:22:01.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1999999999999999e-06 2021-03-15T22:22:01.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.098 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051573+0000", 2021-03-15T22:22:01.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8409999999999998e-05 2021-03-15T22:22:01.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051631+0000", 2021-03-15T22:22:01.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7115999999999999e-05 2021-03-15T22:22:01.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051658+0000", 2021-03-15T22:22:01.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072775619999999996 2021-03-15T22:22:01.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:01.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.058936+0000", 2021-03-15T22:22:01.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4233999999999998e-05 2021-03-15T22:22:01.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.058970+0000", 2021-03-15T22:22:01.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013063455999999999 2021-03-15T22:22:01.103 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.104 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.104 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.104 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1412 7.b 7:d4368cae:::smithi07694878-18:5e [read 0~3226944,checksum] snapc 0=[] ondisk+read+known_if_redirected e193)", 2021-03-15T22:22:01.105 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:48.096961+0000", 2021-03-15T22:22:01.105 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.815070260000001, 2021-03-15T22:22:01.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070763129999999999, 2021-03-15T22:22:01.105 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.106 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:01.106 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:01.106 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:01.106 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:01.106 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1412 2021-03-15T22:22:01.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.107 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096961+0000", 2021-03-15T22:22:01.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096961+0000", 2021-03-15T22:22:01.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9489999999999999e-06 2021-03-15T22:22:01.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.110 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096963+0000", 2021-03-15T22:22:01.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8980000000000001e-06 2021-03-15T22:22:01.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096965+0000", 2021-03-15T22:22:01.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1999999999999997e-06 2021-03-15T22:22:01.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096972+0000", 2021-03-15T22:22:01.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5689999999999998e-06 2021-03-15T22:22:01.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096976+0000", 2021-03-15T22:22:01.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.005626191 2021-03-15T22:22:01.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.102603+0000", 2021-03-15T22:22:01.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0926000000000004e-05 2021-03-15T22:22:01.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.102683+0000", 2021-03-15T22:22:01.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013535800000000001 2021-03-15T22:22:01.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.104037+0000", 2021-03-15T22:22:01.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070763129999999999 2021-03-15T22:22:01.117 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.117 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.117 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.118 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1413 7.b 7:d4368cae:::smithi07694878-18:5e [read 0~3226944,checksum] snapc 0=[] ondisk+read+known_if_redirected e193)", 2021-03-15T22:22:01.118 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:48.096981+0000", 2021-03-15T22:22:01.118 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 12.815049643, 2021-03-15T22:22:01.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079882589999999993, 2021-03-15T22:22:01.119 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.119 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:01.119 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:01.120 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:01.120 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:01.120 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1413 2021-03-15T22:22:01.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.120 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096981+0000", 2021-03-15T22:22:01.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.122 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.122 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096981+0000", 2021-03-15T22:22:01.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5689999999999999e-06 2021-03-15T22:22:01.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096983+0000", 2021-03-15T22:22:01.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1790000000000001e-06 2021-03-15T22:22:01.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096984+0000", 2021-03-15T22:22:01.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0320000000000003e-06 2021-03-15T22:22:01.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096989+0000", 2021-03-15T22:22:01.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5549999999999998e-06 2021-03-15T22:22:01.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.096993+0000", 2021-03-15T22:22:01.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071195030000000001 2021-03-15T22:22:01.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.104112+0000", 2021-03-15T22:22:01.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7724000000000002e-05 2021-03-15T22:22:01.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.104210+0000", 2021-03-15T22:22:01.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00075969700000000004 2021-03-15T22:22:01.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.104970+0000", 2021-03-15T22:22:01.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079882589999999993 2021-03-15T22:22:01.131 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.131 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.131 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.132 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDScrubReserve(6.1b REQUEST e194)", 2021-03-15T22:22:01.132 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:22:01.132 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.128939316, 2021-03-15T22:22:01.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.061386018000000001, 2021-03-15T22:22:01.132 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.133 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:01.133 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:22:01.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:22:01.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6899999999999999e-06 2021-03-15T22:22:01.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783093+0000", 2021-03-15T22:22:01.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5640000000000001e-06 2021-03-15T22:22:01.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783097+0000", 2021-03-15T22:22:01.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7742e-05 2021-03-15T22:22:01.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783125+0000", 2021-03-15T22:22:01.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3793e-05 2021-03-15T22:22:01.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783138+0000", 2021-03-15T22:22:01.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8529999999999993e-05 2021-03-15T22:22:01.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783237+0000", 2021-03-15T22:22:01.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.241e-05 2021-03-15T22:22:01.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:22:01.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783249+0000", 2021-03-15T22:22:01.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013943790000000001 2021-03-15T22:22:01.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.784644+0000", 2021-03-15T22:22:01.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2398000000000001e-05 2021-03-15T22:22:01.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.143 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.784656+0000", 2021-03-15T22:22:01.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.059821512 2021-03-15T22:22:01.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.844478+0000", 2021-03-15T22:22:01.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.061386018000000001 2021-03-15T22:22:01.145 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.145 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.145 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.146 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1558 7.b 7:d10ba14d:::smithi07694878-43:head [write 732838~698856] snapc 6a=[6a,69,68] ondisk+write+known_if_redirected e204)", 2021-03-15T22:22:01.146 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.046384+0000", 2021-03-15T22:22:01.146 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.8656471100000001, 2021-03-15T22:22:01.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083510470000000003, 2021-03-15T22:22:01.147 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.147 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:01.147 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:01.147 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:01.148 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:01.148 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1558 2021-03-15T22:22:01.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.148 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046384+0000", 2021-03-15T22:22:01.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046384+0000", 2021-03-15T22:22:01.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00036469999999999997 2021-03-15T22:22:01.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046749+0000", 2021-03-15T22:22:01.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1568e-05 2021-03-15T22:22:01.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046760+0000", 2021-03-15T22:22:01.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4433000000000001e-05 2021-03-15T22:22:01.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046785+0000", 2021-03-15T22:22:01.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5650000000000007e-06 2021-03-15T22:22:01.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046794+0000", 2021-03-15T22:22:01.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3423999999999994e-05 2021-03-15T22:22:01.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046868+0000", 2021-03-15T22:22:01.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000115204 2021-03-15T22:22:01.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046983+0000", 2021-03-15T22:22:01.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000209953 2021-03-15T22:22:01.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:22:01.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.047193+0000", 2021-03-15T22:22:01.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011912909999999999 2021-03-15T22:22:01.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:22:01.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.048384+0000", 2021-03-15T22:22:01.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061578550000000003 2021-03-15T22:22:01.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:01.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.054542+0000", 2021-03-15T22:22:01.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1480000000000003e-05 2021-03-15T22:22:01.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:01.161 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.054583+0000", 2021-03-15T22:22:01.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4589000000000001e-05 2021-03-15T22:22:01.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.161 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:01.162 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.054648+0000", 2021-03-15T22:22:01.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6984999999999995e-05 2021-03-15T22:22:01.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.054735+0000", 2021-03-15T22:22:01.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083510470000000003 2021-03-15T22:22:01.163 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.163 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.164 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.164 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1562 7.b 7:d10ba14d:::smithi07694878-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e204)", 2021-03-15T22:22:01.164 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.059556+0000", 2021-03-15T22:22:01.165 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 1.8524752069999999, 2021-03-15T22:22:01.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064757119999999998, 2021-03-15T22:22:01.165 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.165 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:01.165 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:01.166 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:01.166 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:01.166 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1562 2021-03-15T22:22:01.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.167 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059556+0000", 2021-03-15T22:22:01.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059556+0000", 2021-03-15T22:22:01.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7320000000000003e-06 2021-03-15T22:22:01.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059562+0000", 2021-03-15T22:22:01.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5040000000000003e-05 2021-03-15T22:22:01.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059597+0000", 2021-03-15T22:22:01.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1148000000000001e-05 2021-03-15T22:22:01.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059608+0000", 2021-03-15T22:22:01.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2299999999999996e-06 2021-03-15T22:22:01.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059614+0000", 2021-03-15T22:22:01.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001475858 2021-03-15T22:22:01.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.061090+0000", 2021-03-15T22:22:01.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013587000000000001 2021-03-15T22:22:01.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:01.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.061226+0000", 2021-03-15T22:22:01.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045935209999999997 2021-03-15T22:22:01.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.065819+0000", 2021-03-15T22:22:01.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000114881 2021-03-15T22:22:01.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.065934+0000", 2021-03-15T22:22:01.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7431999999999998e-05 2021-03-15T22:22:01.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066032+0000", 2021-03-15T22:22:01.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064757119999999998 2021-03-15T22:22:01.179 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.179 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.179 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.180 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.6.0:214 7.d e205/91 7:bd183e1b:::smithi07694878-46:68 v 205'30, mlcod=202'27)", 2021-03-15T22:22:01.180 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:22:01.180 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.82291630000000004, 2021-03-15T22:22:01.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036819649000000003, 2021-03-15T22:22:01.181 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:01.181 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:01.181 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:01.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:01.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:22:01.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:01.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:01.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:22:01.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2879999999999999e-06 2021-03-15T22:22:01.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:01.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089117+0000", 2021-03-15T22:22:01.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4639e-05 2021-03-15T22:22:01.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:01.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089132+0000", 2021-03-15T22:22:01.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0759e-05 2021-03-15T22:22:01.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:01.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089142+0000", 2021-03-15T22:22:01.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8793000000000001e-05 2021-03-15T22:22:01.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:01.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089161+0000", 2021-03-15T22:22:01.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3105999999999998e-05 2021-03-15T22:22:01.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:01.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089194+0000", 2021-03-15T22:22:01.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3468e-05 2021-03-15T22:22:01.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:01.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089208+0000", 2021-03-15T22:22:01.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036675487 2021-03-15T22:22:01.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:01.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125883+0000", 2021-03-15T22:22:01.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1109000000000001e-05 2021-03-15T22:22:01.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:01.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:01.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:01.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125934+0000", 2021-03-15T22:22:01.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036819649000000003 2021-03-15T22:22:01.195 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.195 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.195 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.196 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:01.196 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:01.196 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:22:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:01.200 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:01.201 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:01.201 INFO:tasks.rados.rados.0.smithi076.stdout:575: done (5 left) 2021-03-15T22:22:01.201 INFO:tasks.rados.rados.0.smithi076.stdout:576: done (4 left) 2021-03-15T22:22:01.202 INFO:tasks.rados.rados.0.smithi076.stdout:578: done (3 left) 2021-03-15T22:22:01.202 INFO:tasks.rados.rados.0.smithi076.stdout:579: done (2 left) 2021-03-15T22:22:01.202 INFO:tasks.rados.rados.0.smithi076.stdout:581: done (1 left) 2021-03-15T22:22:01.202 INFO:tasks.rados.rados.0.smithi076.stdout:583: done (0 left) 2021-03-15T22:22:01.203 INFO:tasks.rados.rados.0.smithi076.stdout:584: rollback oid 20 current snap is 55 2021-03-15T22:22:01.203 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 20 to 53 2021-03-15T22:22:01.203 INFO:tasks.rados.rados.0.smithi076.stdout:585: copy_from oid 5 from oid 34 current snap is 55 2021-03-15T22:22:01.203 INFO:tasks.rados.rados.0.smithi076.stdout:586: read oid 26 snap -1 2021-03-15T22:22:01.203 INFO:tasks.rados.rados.0.smithi076.stdout:586: expect (ObjNum 66 snap 6 seq_num 66) 2021-03-15T22:22:01.204 INFO:tasks.rados.rados.0.smithi076.stdout:587: write (excl) oid 40 current snap is 55 2021-03-15T22:22:01.204 INFO:tasks.rados.rados.0.smithi076.stdout:587: seq_num 236 ranges {766452=565707,1860221=469412,2981238=594195} 2021-03-15T22:22:01.204 INFO:tasks.rados.rados.0.smithi076.stdout:587: writing smithi07694878-40 from 766452 to 1332159 tid 1 2021-03-15T22:22:01.204 INFO:tasks.rados.rados.0.smithi076.stdout:587: writing smithi07694878-40 from 1860221 to 2329633 tid 2 2021-03-15T22:22:01.205 INFO:tasks.rados.rados.0.smithi076.stdout:587: writing smithi07694878-40 from 2981238 to 3575433 tid 3 2021-03-15T22:22:01.205 INFO:tasks.rados.rados.0.smithi076.stdout:588: delete oid 46 current snap is 55 2021-03-15T22:22:01.205 INFO:tasks.rados.rados.0.smithi076.stdout:584: finishing rollback tid 0 to smithi07694878-20 2021-03-15T22:22:01.205 INFO:tasks.rados.rados.0.smithi076.stdout:584: finishing rollback tid 1 to smithi07694878-20 2021-03-15T22:22:01.205 INFO:tasks.rados.rados.0.smithi076.stdout:586: expect (ObjNum 66 snap 6 seq_num 66) 2021-03-15T22:22:01.244 INFO:tasks.rados.rados.0.smithi076.stdout:584: finishing rollback tid 2 to smithi07694878-20 2021-03-15T22:22:01.244 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 148 (ObjNum 198 snap 43 seq_num 198) dirty exists 2021-03-15T22:22:01.244 INFO:tasks.rados.rados.0.smithi076.stdout:585: finishing copy_from to smithi07694878-5 2021-03-15T22:22:01.245 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 138 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:22:01.245 INFO:tasks.rados.rados.0.smithi076.stdout:585: finishing copy_from racing read to smithi07694878-5 2021-03-15T22:22:01.245 INFO:tasks.rados.rados.0.smithi076.stdout:584: done (4 left) 2021-03-15T22:22:01.245 INFO:tasks.rados.rados.0.smithi076.stdout:585: done (3 left) 2021-03-15T22:22:01.246 INFO:tasks.rados.rados.0.smithi076.stdout:586: done (2 left) 2021-03-15T22:22:01.246 INFO:tasks.rados.rados.0.smithi076.stdout:588: done (1 left) 2021-03-15T22:22:01.246 INFO:tasks.rados.rados.0.smithi076.stdout:589: write oid 1 current snap is 55 2021-03-15T22:22:01.246 INFO:tasks.rados.rados.0.smithi076.stdout:589: seq_num 238 ranges {400066=716205,1860832=139234} 2021-03-15T22:22:01.250 INFO:tasks.rados.rados.0.smithi076.stdout:589: writing smithi07694878-1 from 400066 to 1116271 tid 1 2021-03-15T22:22:01.255 INFO:tasks.rados.rados.0.smithi076.stdout:589: writing smithi07694878-1 from 1860832 to 2000066 tid 2 2021-03-15T22:22:01.255 INFO:tasks.rados.rados.0.smithi076.stdout:590: rollback oid 16 current snap is 55 2021-03-15T22:22:01.255 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 53 2021-03-15T22:22:01.255 INFO:tasks.rados.rados.0.smithi076.stdout:591: snap_create 2021-03-15T22:22:01.256 INFO:tasks.rados.rados.0.smithi076.stdout:589: finishing write tid 1 to smithi07694878-1 2021-03-15T22:22:01.257 INFO:tasks.rados.rados.0.smithi076.stdout:589: finishing write tid 2 to smithi07694878-1 2021-03-15T22:22:01.257 INFO:tasks.rados.rados.0.smithi076.stdout:589: finishing write tid 3 to smithi07694878-1 2021-03-15T22:22:01.258 INFO:tasks.rados.rados.0.smithi076.stdout:590: finishing rollback tid 0 to smithi07694878-16 2021-03-15T22:22:01.258 INFO:tasks.rados.rados.0.smithi076.stdout:589: finishing write tid 4 to smithi07694878-1 2021-03-15T22:22:01.258 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 76 (ObjNum 238 snap 55 seq_num 238) dirty exists 2021-03-15T22:22:01.258 INFO:tasks.rados.rados.0.smithi076.stdout:589: left oid 1 (ObjNum 238 snap 55 seq_num 238) 2021-03-15T22:22:01.259 INFO:tasks.rados.rados.0.smithi076.stdout:590: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:22:01.259 INFO:tasks.rados.rados.0.smithi076.stdout:590: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:22:01.259 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 284 (ObjNum 208 snap 47 seq_num 208) dirty exists 2021-03-15T22:22:01.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:01.326 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:01.328 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:01.429 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:01.453 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:01.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:01.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:01.556 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:01.580 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:01.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:01.582 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:01.622 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:22:01.683 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:01.707 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:01.708 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:01.752 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-15T22:22:01.809 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:01.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:01.836 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:01.836 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:01.918 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:01.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:01.962 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:01.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:02.037 INFO:tasks.rados.rados.0.smithi076.stdout:589: done (3 left) 2021-03-15T22:22:02.037 INFO:tasks.rados.rados.0.smithi076.stdout:590: done (2 left) 2021-03-15T22:22:02.037 INFO:tasks.rados.rados.0.smithi076.stdout:591: done (1 left) 2021-03-15T22:22:02.038 INFO:tasks.rados.rados.0.smithi076.stdout:592: snap_create 2021-03-15T22:22:02.046 INFO:tasks.rados.rados.0.smithi076.stdout:587: finishing write tid 1 to smithi07694878-40 2021-03-15T22:22:02.050 INFO:tasks.rados.rados.0.smithi076.stdout:587: finishing write tid 2 to smithi07694878-40 2021-03-15T22:22:02.051 INFO:tasks.rados.rados.0.smithi076.stdout:587: finishing write tid 3 to smithi07694878-40 2021-03-15T22:22:02.051 INFO:tasks.rados.rados.0.smithi076.stdout:587: finishing write tid 4 to smithi07694878-40 2021-03-15T22:22:02.051 INFO:tasks.rados.rados.0.smithi076.stdout:587: finishing write tid 5 to smithi07694878-40 2021-03-15T22:22:02.052 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 321 (ObjNum 236 snap 55 seq_num 236) dirty exists 2021-03-15T22:22:02.052 INFO:tasks.rados.rados.0.smithi076.stdout:587: left oid 40 (ObjNum 236 snap 55 seq_num 236) 2021-03-15T22:22:02.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:02.070 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:02.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:02.089 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:02.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:02.191 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:02.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:02.214 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:02.214 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:02.245 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7] out_osds: [3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:22:02.246 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 0 1 2021-03-15T22:22:02.315 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:02.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[56514]: cluster 2021-03-15T22:22:01.004277+0000 mgr.y (mgr.14144) 456 : cluster [DBG] pgmap v435: 138 pgs: 131 active+clean, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 activating, 1 remapped+peering; 128 MiB data, 390 MiB used, 625 GiB / 626 GiB avail; 8.5 MiB/s rd, 5.3 MiB/s wr, 19 op/s; 15/1080 objects degraded (1.389%); 2/1080 objects misplaced (0.185%); 6.4 MiB/s, 27 objects/s recovering 2021-03-15T22:22:02.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[56514]: cluster 2021-03-15T22:22:01.030942+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2021-03-15T22:22:02.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[56514]: audit 2021-03-15T22:22:01.049585+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:22:02.328 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[56514]: audit 2021-03-15T22:22:01.049840+0000 mon.a (mon.0) 832 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:22:02.329 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[62156]: cluster 2021-03-15T22:22:01.004277+0000 mgr.y (mgr.14144) 2021-03-15T22:22:02.329 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[62156]: 456 : cluster [DBG] pgmap v435: 138 pgs: 131 active+clean, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 activating, 1 remapped+peering; 128 MiB data, 390 MiB used, 625 GiB / 626 GiB avail; 8.5 MiB/s rd, 5.3 MiB/s wr, 19 op/s; 15/1080 objects degraded (1.389%); 2/1080 objects misplaced (0.185%); 6.4 MiB/s, 27 objects/s recovering 2021-03-15T22:22:02.330 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[62156]: cluster 2021-03-15T22:22:01.030942+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2021-03-15T22:22:02.330 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[62156]: audit 2021-03-15T22:22:01.049585+0000 mon.a (mon.0) 831 : audit [INF] 2021-03-15T22:22:02.330 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:22:02.330 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[62156]: audit 2021-03-15T22:22:02.331 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[62156]: 2021-03-15T22:22:01.049840+0000 2021-03-15T22:22:02.331 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[62156]: mon.a (mon.0) 832 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:22:02.331 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:02 smithi076 conmon[62156]: 2021-03-15T22:22:02.342 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:02.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:02.343 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:02.444 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:02.471 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:02.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:02.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:02.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:02 smithi125 conmon[56739]: cluster 2021-03-15T22:22:01.004277+0000 mgr.y (mgr.14144) 456 : cluster 2021-03-15T22:22:02.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:02 smithi125 conmon[56739]: [DBG] pgmap v435: 138 pgs: 131 active+clean, 1 active+recovery_wait+degraded, 1 active+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 peering, 2 activating, 1 remapped+peering; 128 MiB data, 390 MiB used, 625 GiB / 626 GiB avail; 8.5 MiB/s rd, 5.3 MiB/s wr, 19 op/s; 15/1080 objects degraded (1.389%); 2/1080 objects misplaced (0.185%); 6.4 MiB/s, 27 objects/s recovering 2021-03-15T22:22:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:02 smithi125 conmon[56739]: cluster 2021-03-15T22:22:01.030942+0000 mon.a (mon.0) 830 : cluster [DBG] osdmap e206: 8 total, 8 up, 7 in 2021-03-15T22:22:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:02 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:22:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:02 smithi125 conmon[56739]: :22:01.049585+0000 mon.a (mon.0) 831 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:22:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:02 smithi125 conmon[56739]: audit 2021-03-15T22:22:02.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:02 smithi125 conmon[56739]: 2021-03-15T22:22:01.049840+0000 mon.a (mon 2021-03-15T22:22:02.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:02 smithi125 conmon[56739]: .0) 832 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:22:02.566 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:22:02.575 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:02.630 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:02.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:02.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:02.733 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:02.755 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:02.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:02.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:02.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:02.879 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:02.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:02.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:02.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:03.008 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:03.010 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:03.039 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:03.051 INFO:tasks.rados.rados.0.smithi076.stdout:587: done (1 left) 2021-03-15T22:22:03.051 INFO:tasks.rados.rados.0.smithi076.stdout:592: done (0 left) 2021-03-15T22:22:03.052 INFO:tasks.rados.rados.0.smithi076.stdout:593: rollback oid 7 current snap is 57 2021-03-15T22:22:03.052 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 7 to 56 2021-03-15T22:22:03.053 INFO:tasks.rados.rados.0.smithi076.stdout:594: write oid 4 current snap is 57 2021-03-15T22:22:03.053 INFO:tasks.rados.rados.0.smithi076.stdout:594: seq_num 239 ranges {416873=791454,1919314=97559} 2021-03-15T22:22:03.058 INFO:tasks.rados.rados.0.smithi076.stdout:594: writing smithi07694878-4 from 416873 to 1208327 tid 1 2021-03-15T22:22:03.063 INFO:tasks.rados.rados.0.smithi076.stdout:594: writing smithi07694878-4 from 1919314 to 2016873 tid 2 2021-03-15T22:22:03.063 INFO:tasks.rados.rados.0.smithi076.stdout:595: rollback oid 25 current snap is 57 2021-03-15T22:22:03.064 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 54 2021-03-15T22:22:03.064 INFO:tasks.rados.rados.0.smithi076.stdout:593: finishing rollback tid 0 to smithi07694878-7 2021-03-15T22:22:03.064 INFO:tasks.rados.rados.0.smithi076.stdout:593: finishing rollback tid 1 to smithi07694878-7 2021-03-15T22:22:03.064 INFO:tasks.rados.rados.0.smithi076.stdout:593: finishing rollback tid 2 to smithi07694878-7 2021-03-15T22:22:03.065 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 245 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:22:03.065 INFO:tasks.rados.rados.0.smithi076.stdout:593: done (2 left) 2021-03-15T22:22:03.065 INFO:tasks.rados.rados.0.smithi076.stdout:596: snap_remove snap 53 2021-03-15T22:22:03.111 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:03.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:03.135 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:03.135 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:03.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:03.259 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:03.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:03.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:03.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[56514]: cluster 2021-03-15T22:22:01.769277+0000 2021-03-15T22:22:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[56514]: osd.7 (osd.7) 49 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:22:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[56514]: cluster 2021-03-15T22:22:01.885515+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:22:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[56514]: cluster 2021-03-15T22:22:02.032300+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e207: 8 total, 8 up, 7 in 2021-03-15T22:22:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[56514]: cluster 2021-03-15T22:22:02.316259+0000 osd.4 (osd.4) 52 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:22:03.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[62156]: cluster 2021-03-15T22:22:03.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[62156]: 2021-03-15T22:22:01.769277+0000 osd.7 (osd.7) 49 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:22:03.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[62156]: cluster 2021-03-15T22:22:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[62156]: 2021 2021-03-15T22:22:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[62156]: - 2021-03-15T22:22:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[62156]: 03-15T22:22:01.885515+0000 osd.1 (osd.1) 44 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:22:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[62156]: cluster 2021-03-15T22:22:02.032300+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e207: 8 total, 8 up, 7 in 2021-03-15T22:22:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[62156]: cluster 2021-03-15T22:22:03.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[62156]: 2021-03-15T22:22:02.316259+0000 2021-03-15T22:22:03.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[62156]: osd.4 (osd.4) 52 : cluster 2021-03-15T22:22:03.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:03 smithi076 conmon[62156]: [DBG] 4.1a scrub ok 2021-03-15T22:22:03.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:03.388 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:03.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:03.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:03.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:03.516 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:03.517 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:03.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:03 smithi125 conmon[56739]: cluster 2021-03-15T22:22:01.769277+0000 osd.7 (osd. 2021-03-15T22:22:03.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:03 smithi125 conmon[56739]: 7) 49 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:22:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:03 smithi125 conmon[56739]: cluster 2021-03-15T22:22:01.885515+0000 osd.1 (osd.1) 44 : cluster [DBG] 2021-03-15T22:22:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:03 smithi125 conmon[56739]: 4.10 scrub ok 2021-03-15T22:22:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:03 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:22:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:03 smithi125 conmon[56739]: 22:02.032300+0000 mon.a (mon.0) 833 : cluster [DBG] osdmap e207: 8 total, 8 up, 7 in 2021-03-15T22:22:03.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:03 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:22:03.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:03 smithi125 conmon[56739]: 22:02.316259+0000 osd.4 (osd.4 2021-03-15T22:22:03.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:03 smithi125 conmon[56739]: ) 52 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:22:03.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:03.644 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:03.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:03.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:03.746 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:03.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:03.774 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:03.774 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:03.875 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:03.914 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:03.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:03.916 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:03.980 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:04.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:04.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[56514]: cluster 2021-03-15T22:22:02.880779+0000 osd.1 (osd.1) 45 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:22:04.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[56514]: cluster 2021-03-15T22:22:03.004981+0000 mgr.y (mgr.14144) 457 : cluster [DBG] pgmap v438: 138 pgs: 136 active+clean, 1 active+undersized+remapped, 1 peering; 143 MiB data, 436 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 6.4 MiB/s wr, 29 op/s; 12 MiB/s, 29 objects/s recovering 2021-03-15T22:22:04.050 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[56514]: cluster 2021-03-15T22:22:03.036901+0000 mon.a (mon.0) 834 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2021-03-15T22:22:04.051 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: cluster 2021-03-15T22:22:02.880779+0000 2021-03-15T22:22:04.052 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: osd.1 (osd.1) 45 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:22:04.052 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:22:04.053 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: 22:22:03.004981+0000 mgr.y (mgr.14144) 457 : cluster [DBG] pgmap v438: 138 pgs: 136 active+clean, 1 active+undersized+remapped, 1 peering; 143 MiB data, 436 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 6.4 MiB/s wr, 29 op/s; 12 MiB/s, 29 objects/s recovering 2021-03-15T22:22:04.053 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:22:04.053 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: -15T22:22:03.036901+0000 mon.a (mon 2021-03-15T22:22:04.053 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: .0) 834 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2021-03-15T22:22:04.054 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:22:04.054 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: -15T22:22:03.036935+0000 2021-03-15T22:22:04.054 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: mon.a (mon.0) 835 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/1080 objects degraded (1.389%), 2 pgs degraded) 2021-03-15T22:22:04.054 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: 2021-03-15T22:22:04.055 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: cluster 2021-03-15T22:22:04.055 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: 2021-03-15T22:22:03.036955+0000 mon.a 2021-03-15T22:22:04.055 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: (mon.0) 836 : cluster [INF] Cluster is now healthy 2021-03-15T22:22:04.055 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:22:04.055 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: 03-15T22:22:03.049476+0000 2021-03-15T22:22:04.056 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: mon.a (mon.0) 837 : cluster [DBG] osdmap e208: 8 total, 8 up, 7 in 2021-03-15T22:22:04.056 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: 2021-03-15T22:22:04.056 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: cluster 2021-03-15T22:22:04.056 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: 2021-03-15T22:22:03.126573+0000 osd.5 ( 2021-03-15T22:22:04.057 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[62156]: osd.5) 44 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:22:04.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:04.058 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:04.058 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:04.059 INFO:tasks.rados.rados.0.smithi076.stdout:596: done (2 left) 2021-03-15T22:22:04.059 INFO:tasks.rados.rados.0.smithi076.stdout:597: snap_create 2021-03-15T22:22:04.060 INFO:tasks.rados.rados.0.smithi076.stdout:594: finishing write tid 1 to smithi07694878-4 2021-03-15T22:22:04.060 INFO:tasks.rados.rados.0.smithi076.stdout:594: finishing write tid 2 to smithi07694878-4 2021-03-15T22:22:04.060 INFO:tasks.rados.rados.0.smithi076.stdout:594: finishing write tid 3 to smithi07694878-4 2021-03-15T22:22:04.060 INFO:tasks.rados.rados.0.smithi076.stdout:594: finishing write tid 4 to smithi07694878-4 2021-03-15T22:22:04.061 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 246 (ObjNum 239 snap 57 seq_num 239) dirty exists 2021-03-15T22:22:04.061 INFO:tasks.rados.rados.0.smithi076.stdout:594: left oid 4 (ObjNum 239 snap 57 seq_num 239) 2021-03-15T22:22:04.061 INFO:tasks.rados.rados.0.smithi076.stdout:595: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:22:04.061 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 0 (ObjNum 261882656 snap 21896 seq_num 261960784) dirty dne 2021-03-15T22:22:04.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[56514]: cluster 2021-03-15T22:22:03.036935+0000 mon.a (mon.0) 835 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/1080 objects degraded (1.389%), 2 pgs degraded) 2021-03-15T22:22:04.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[56514]: cluster 2021-03-15T22:22:03.036955+0000 mon.a (mon.0) 836 : cluster [INF] Cluster is now healthy 2021-03-15T22:22:04.062 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[56514]: cluster 2021-03-15T22:22:03.049476+0000 mon.a (mon.0) 837 : cluster [DBG] osdmap e208: 8 total, 8 up, 7 in 2021-03-15T22:22:04.063 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:04 smithi076 conmon[56514]: cluster 2021-03-15T22:22:03.126573+0000 osd.5 (osd.5) 44 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:22:04.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:04.186 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:04.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:04.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:04.288 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:04.314 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:04.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:04.316 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:04.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:04.442 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:04.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:04.444 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:04.484 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:22:04.515 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:22:04.533 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:22:04.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: cluster 2021-03-15T22:22:02.880779+0000 osd.1 (osd.1) 45 : cluster 2021-03-15T22:22:04.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: [DBG] 3.19 scrub ok 2021-03-15T22:22:04.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: cluster 2021-03-15T22:22:03.004981+0000 mgr.y (mgr.14144) 457 : cluster [DBG] pgmap v438: 138 pgs: 136 active+clean, 1 active+undersized+remapped, 1 peering; 143 MiB data, 436 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 6.4 MiB/s wr, 29 op/s; 12 MiB/s, 29 objects/s recovering 2021-03-15T22:22:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:22:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: -15T22:22:03.036901 2021-03-15T22:22:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: +0000 mon.a (mon.0) 834 2021-03-15T22:22:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2021-03-15T22:22:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:22:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: 22:22:03.036935+0000 mon.a (mon.0) 2021-03-15T22:22:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: 835 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 15/1080 objects degraded (1.389%), 2 pgs degraded) 2021-03-15T22:22:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: cluster 2021-03-15T22:22:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:22:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: 22:03.036955+0000 mon.a (mon. 2021-03-15T22:22:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: 0) 836 : cluster [INF] Cluster is now healthy 2021-03-15T22:22:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: cluster 2021-03-15T22:22:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: 2021-03-15T22:22:03 2021-03-15T22:22:04.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: .049476+0000 mon.a (mon.0) 837 : cluster 2021-03-15T22:22:04.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: [DBG] osdmap e208: 8 total, 8 up, 7 in 2021-03-15T22:22:04.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:22:04.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: 03-15T22:22:03.126573 2021-03-15T22:22:04.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:04 smithi125 conmon[56739]: +0000 osd.5 (osd.5) 44 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:22:04.545 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:04.551 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:22:04.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:04.571 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:04.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:04.572 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:22:04.589 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:22:04.610 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:22:04.631 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:22:04.633 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:22:04.633 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:22:04.633 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:22:04.633 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:22:04.634 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:22:04.634 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:22:04.634 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:22:04.634 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:22:04.672 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:04.695 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:04.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:04.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:04.798 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:04.824 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:04.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:04.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:04.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:04.950 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:04.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:05.028 INFO:tasks.rados.rados.0.smithi076.stdout:594: done (2 left) 2021-03-15T22:22:05.029 INFO:tasks.rados.rados.0.smithi076.stdout:595: done (1 left) 2021-03-15T22:22:05.029 INFO:tasks.rados.rados.0.smithi076.stdout:597: done (0 left) 2021-03-15T22:22:05.029 INFO:tasks.rados.rados.0.smithi076.stdout:598: read oid 21 snap -1 2021-03-15T22:22:05.030 INFO:tasks.rados.rados.0.smithi076.stdout:598: expect (ObjNum 183 snap 40 seq_num 183) 2021-03-15T22:22:05.030 INFO:tasks.rados.rados.0.smithi076.stdout:599: write (excl) oid 29 current snap is 58 2021-03-15T22:22:05.030 INFO:tasks.rados.rados.0.smithi076.stdout:599: seq_num 240 ranges {433680=466703,1577796=455884} 2021-03-15T22:22:05.033 INFO:tasks.rados.rados.0.smithi076.stdout:599: writing smithi07694878-29 from 433680 to 900383 tid 1 2021-03-15T22:22:05.039 INFO:tasks.rados.rados.0.smithi076.stdout:599: writing smithi07694878-29 from 1577796 to 2033680 tid 2 2021-03-15T22:22:05.040 INFO:tasks.rados.rados.0.smithi076.stdout:600: snap_remove snap 52 2021-03-15T22:22:05.052 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:05.061 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:22:05.062 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:22:05.076 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:05.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:05.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:05.178 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:05.200 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:05.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:05.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:05.328 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:05.330 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:05.431 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:05.456 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:05.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:05.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[56514]: cluster 2021-03-15T22:22:03.775119+0000 osd.7 (osd 2021-03-15T22:22:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[56514]: .7) 50 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:22:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[56514]: cluster 2021-03-15T22:22:03.851042+0000 osd.1 (osd.1) 46 : cluster [DBG] 5.c deep-scrub ok 2021-03-15T22:22:05.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[56514]: cluster 2021-03-15T22:22:04.054367+0000 mon.a (mon.0) 838 : cluster [DBG] osdmap e209: 8 total, 8 up, 7 in 2021-03-15T22:22:05.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[56514]: audit 2021-03-15T22:22:04.452448+0000 mon.a (mon.0) 839 : audit [DBG] from='client.? 172.21.15.76:0/4279310959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:05.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[56514]: cluster 2021-03-15T22:22:05.024127+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2021-03-15T22:22:05.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[62156]: cluster 2021-03-15T22:22:03.775119+0000 osd.7 (osd. 2021-03-15T22:22:05.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[62156]: 7) 50 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:22:05.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[62156]: cluster 2021-03-15T22:22:03.851042+0000 osd.1 (osd.1) 46 : cluster [DBG] 5.c deep-scrub ok 2021-03-15T22:22:05.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[62156]: cluster 2021-03-15T22:22:04.054367+0000 mon.a (mon.0) 838 : cluster [DBG] osdmap e209: 8 total, 8 up, 7 in 2021-03-15T22:22:05.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[62156]: audit 2021-03-15T22:22:04.452448+0000 mon.a (mon.0) 839 : audit [DBG] from='client.? 172.21.15.76:0/4279310959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:05.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:05 smithi076 conmon[62156]: cluster 2021-03-15T22:22:05.024127+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2021-03-15T22:22:05.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:05 smithi125 conmon[56739]: cluster 2021-03-15T22:22:03.775119+0000 osd.7 (osd.7) 50 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:22:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:05 smithi125 conmon[56739]: cluster 2021-03-15T22:22:03.851042+0000 osd.1 (osd. 2021-03-15T22:22:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:05 smithi125 conmon[56739]: 1) 46 : cluster [DBG] 5.c deep-scrub ok 2021-03-15T22:22:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:05 smithi125 conmon[56739]: cluster 2021-03-15T22:22:04.054367+0000 mon.a (mon.0) 838 : cluster [DBG] osdmap e209: 8 total, 8 up, 7 in 2021-03-15T22:22:05.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:05 smithi125 conmon[56739]: audit 2021-03-15T22:22:04.452448+0000 mon.a (mon.0) 839 : audit 2021-03-15T22:22:05.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:05 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/4279310959' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:05.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:05 smithi125 conmon[56739]: cluster 2021-03-15T22:22: 2021-03-15T22:22:05.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:05 smithi125 conmon[56739]: 05.024127+0000 mon.a (mon.0) 840 : cluster [DBG] osdmap e210: 8 total, 8 up, 7 in 2021-03-15T22:22:05.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:05 smithi125 conmon[56739]: 2021-03-15T22:22:05.560 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:05.582 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:05.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:05.584 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:05.685 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:05.711 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:05.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:05.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:05.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:05.840 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:05.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:05.841 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:05.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:05.968 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:05.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:05.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:06.035 INFO:tasks.rados.rados.0.smithi076.stdout:600: done (2 left) 2021-03-15T22:22:06.035 INFO:tasks.rados.rados.0.smithi076.stdout:601: delete oid 43 current snap is 58 2021-03-15T22:22:06.036 INFO:tasks.rados.rados.0.smithi076.stdout:598: expect (ObjNum 183 snap 40 seq_num 183) 2021-03-15T22:22:06.071 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:06.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:06.095 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:06.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:06.197 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:06.224 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:06.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:06.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:06.267 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[65511]: debug 2021-03-15T22:22:06.267 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[65511]: 2021-03-15T22:22:06.041+0000 7f1a2387f700 -1 osd.0 pg_epoch: 211 pg[7.6( v 209'290 (0'0,209'290] local-lis/les=175/176 n=8 ec=91/91 lis/c=175/175 les/c/f=176/176/0 sis=175) [0,7,5] r=0 lpr=175 crt=209'290 lcod 209'288 mlcod 209'288 active+clean+snaptrim trimq=[69~1] ps=[66~1,68~1,6a~1]] removing snap head 2021-03-15T22:22:06.267 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[65511]: 2021-03-15T22:22:06.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: cluster 2021-03-15T22:22:04.618669+0000 osd.6 (osd.6 2021-03-15T22:22:06.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: ) 29 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:22:06.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: cluster 2021-03-15T22:22:04.792547+0000 osd.7 (osd.7) 51 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:22:06.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: cluster 2021-03-15T22:22:04.890762 2021-03-15T22:22:06.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: +0000 osd.1 (osd.1) 47 : cluster [DBG] 2021-03-15T22:22:06.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: 6.1d scrub ok 2021-03-15T22:22:06.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:22:06.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: -15T22:22:05.005408 2021-03-15T22:22:06.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 458 2021-03-15T22:22:06.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: : cluster [DBG] pgmap v441: 138 pgs: 138 active+clean; 138 MiB data, 459 MiB used, 625 GiB / 626 GiB avail; 26 MiB/s rd, 4.2 MiB/s wr, 27 op/s; 6.1 MiB/s, 14 objects/s recovering 2021-03-15T22:22:06.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:22:06.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: -03-15T22:22:05 2021-03-15T22:22:06.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: .296885+0000 osd.4 (osd.4) 53 : 2021-03-15T22:22:06.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: cluster [DBG] 7.1 scrub ok 2021-03-15T22:22:06.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:22:06.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: :22:05.738929+0000 osd.0 2021-03-15T22:22:06.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: (osd.0) 47 : cluster 2021-03-15T22:22:06.272 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: [DBG] 4.6 scrub ok 2021-03-15T22:22:06.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:22:06.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: -03-15T22:22:06 2021-03-15T22:22:06.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: .033072+0000 mon.a (mon.0 2021-03-15T22:22:06.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: ) 841 : cluster [DBG] osdmap e211: 8 total, 8 up, 7 in 2021-03-15T22:22:06.273 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[56514]: 2021-03-15T22:22:06.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: cluster 2021-03-15T22:22:04.618669+0000 osd.6 (osd.6 2021-03-15T22:22:06.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: ) 29 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:22:06.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: cluster 2021-03-15T22:22:04.792547+0000 osd.7 (osd.7) 51 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:22:06.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: 2021-03-15T22:22:06.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:22:06.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: -03-15T22:22:04.890762+0000 osd.1 ( 2021-03-15T22:22:06.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: osd.1) 47 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:22:06.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:22:06.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: 22:05.005408+0000 mgr.y (mgr.14144) 458 : cluster [DBG] pgmap v441: 138 pgs: 138 active+clean; 138 MiB data, 459 MiB used, 625 GiB / 626 GiB avail; 26 MiB/s rd, 4.2 MiB/s wr, 27 op/s; 6.1 MiB/s, 14 objects/s recovering 2021-03-15T22:22:06.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: cluster 2021-03-15T22:22 2021-03-15T22:22:06.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: :05.296885+0000 osd.4 (osd.4) 53 : 2021-03-15T22:22:06.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: cluster [DBG] 7.1 scrub ok 2021-03-15T22:22:06.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:22:06.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: 22:22:05.738929+0000 osd.0 (osd 2021-03-15T22:22:06.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: .0) 47 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:22:06.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: cluster 2021-03-15T22:22:06.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:22:06.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: T22:22:06.033072+0000 mon.a (mon.0) 841 : 2021-03-15T22:22:06.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:06 smithi076 conmon[62156]: cluster [DBG] osdmap e211: 8 total, 8 up, 7 in 2021-03-15T22:22:06.280 INFO:tasks.rados.rados.0.smithi076.stdout:599: finishing write tid 1 to smithi07694878-29 2021-03-15T22:22:06.280 INFO:tasks.rados.rados.0.smithi076.stdout:599: finishing write tid 2 to smithi07694878-29 2021-03-15T22:22:06.281 INFO:tasks.rados.rados.0.smithi076.stdout:599: finishing write tid 3 to smithi07694878-29 2021-03-15T22:22:06.281 INFO:tasks.rados.rados.0.smithi076.stdout:599: finishing write tid 4 to smithi07694878-29 2021-03-15T22:22:06.281 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 331 (ObjNum 240 snap 58 seq_num 240) dirty exists 2021-03-15T22:22:06.281 INFO:tasks.rados.rados.0.smithi076.stdout:599: left oid 29 (ObjNum 240 snap 58 seq_num 240) 2021-03-15T22:22:06.282 INFO:tasks.rados.rados.0.smithi076.stdout:598: done (2 left) 2021-03-15T22:22:06.282 INFO:tasks.rados.rados.0.smithi076.stdout:599: done (1 left) 2021-03-15T22:22:06.282 INFO:tasks.rados.rados.0.smithi076.stdout:601: done (0 left) 2021-03-15T22:22:06.282 INFO:tasks.rados.rados.0.smithi076.stdout:602: rollback oid 28 current snap is 58 2021-03-15T22:22:06.283 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 54 2021-03-15T22:22:06.283 INFO:tasks.rados.rados.0.smithi076.stdout:603: read oid 21 snap 54 2021-03-15T22:22:06.283 INFO:tasks.rados.rados.0.smithi076.stdout:603: expect (ObjNum 183 snap 40 seq_num 183) 2021-03-15T22:22:06.283 INFO:tasks.rados.rados.0.smithi076.stdout:604: write oid 39 current snap is 58 2021-03-15T22:22:06.284 INFO:tasks.rados.rados.0.smithi076.stdout:604: seq_num 242 ranges {467294=617201,1578407=488887} 2021-03-15T22:22:06.284 INFO:tasks.rados.rados.0.smithi076.stdout:604: writing smithi07694878-39 from 467294 to 1084495 tid 1 2021-03-15T22:22:06.284 INFO:tasks.rados.rados.0.smithi076.stdout:604: writing smithi07694878-39 from 1578407 to 2067294 tid 2 2021-03-15T22:22:06.284 INFO:tasks.rados.rados.0.smithi076.stdout:605: snap_remove snap 56 2021-03-15T22:22:06.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:06.353 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:06.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:06.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:06.455 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:06.479 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:06.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:06.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:06.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: cluster 2021-03-15T22:22:04.618669+0000 osd.6 (osd.6) 29 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:22:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: cluster 2021-03-15T22:22:04.792547+0000 osd.7 (osd.7) 51 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:22:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: cluster 2021-03-15T22:22:04.890762+0000 osd.1 (osd.1) 47 : cluster [DBG] 2021-03-15T22:22:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: 6.1d scrub ok 2021-03-15T22:22:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:22:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: 03-15T22:22:05.005408+0000 mgr.y (mgr.14144 2021-03-15T22:22:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: ) 458 : cluster [DBG] pgmap v441: 138 pgs: 138 active+clean; 138 MiB data, 459 MiB used, 625 GiB / 626 GiB avail; 26 MiB/s rd, 4.2 MiB/s wr, 27 op/s; 6.1 MiB/s, 14 objects/s recovering 2021-03-15T22:22:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:22:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: 15T22:22:05.296885+0000 osd.4 (osd.4) 53 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:22:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: cluster 2021-03-15T22:22 2021-03-15T22:22:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: :05.738929+0000 osd.0 (osd.0) 2021-03-15T22:22:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: 47 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:22:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:22:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: -03-15T22:22:06.033072 2021-03-15T22:22:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: +0000 mon.a (mon.0) 841 : cluster [DBG] 2021-03-15T22:22:06.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:06 smithi125 conmon[56739]: osdmap e211: 8 total, 8 up, 7 in 2021-03-15T22:22:06.582 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:06.603 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-15T22:22:06.607 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:06.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:06.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:06.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:06.733 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:06.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:06.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:06.837 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:06.863 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:06.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:06.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:06.966 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:06.992 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:06.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:07.067 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: cluster 2021-03-15T22:22:07.067 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: 2021-03-15T22:22:07.067 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: 2021 2021-03-15T22:22:07.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: -03-15T22:22:05 2021-03-15T22:22:07.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: .628754+0000 osd.6 (osd.6) 30 : cluster [DBG] 7.d scrub ok 2021-03-15T22:22:07.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:22:07.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: -03-15T22:22:06.142030+0000 2021-03-15T22:22:07.068 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: osd.5 (osd.5) 45 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:22:07.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: 2021-03-15T22:22:07.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: cluster 2021-03-15T22:22:07.069 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: 2021-03-15T22:22:06.723747+0000 osd.0 2021-03-15T22:22:07.070 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[56514]: (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2021-03-15T22:22:07.070 INFO:tasks.rados.rados.0.smithi076.stdout:605: done (3 left) 2021-03-15T22:22:07.071 INFO:tasks.rados.rados.0.smithi076.stdout:606: copy_from oid 3 from oid 11 current snap is 58 2021-03-15T22:22:07.071 INFO:tasks.rados.rados.0.smithi076.stdout:602: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:22:07.071 INFO:tasks.rados.rados.0.smithi076.stdout:602: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:22:07.071 INFO:tasks.rados.rados.0.smithi076.stdout:602: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:22:07.072 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 341 (ObjNum 231 snap 52 seq_num 231) dirty exists 2021-03-15T22:22:07.072 INFO:tasks.rados.rados.0.smithi076.stdout:604: finishing write tid 1 to smithi07694878-39 2021-03-15T22:22:07.072 INFO:tasks.rados.rados.0.smithi076.stdout:603: expect (ObjNum 183 snap 40 seq_num 183) 2021-03-15T22:22:07.094 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:07.116 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:07.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:07.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:07.219 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:07.240 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:07.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:07.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:07.300 INFO:tasks.rados.rados.0.smithi076.stdout:604: finishing write tid 2 to smithi07694878-39 2021-03-15T22:22:07.300 INFO:tasks.rados.rados.0.smithi076.stdout:604: finishing write tid 3 to smithi07694878-39 2021-03-15T22:22:07.300 INFO:tasks.rados.rados.0.smithi076.stdout:604: finishing write tid 4 to smithi07694878-39 2021-03-15T22:22:07.301 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 248 (ObjNum 242 snap 58 seq_num 242) dirty exists 2021-03-15T22:22:07.301 INFO:tasks.rados.rados.0.smithi076.stdout:604: left oid 39 (ObjNum 242 snap 58 seq_num 242) 2021-03-15T22:22:07.301 INFO:tasks.rados.rados.0.smithi076.stdout:606: finishing copy_from to smithi07694878-3 2021-03-15T22:22:07.301 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 242 (ObjNum 199 snap 43 seq_num 199) dirty exists 2021-03-15T22:22:07.302 INFO:tasks.rados.rados.0.smithi076.stdout:606: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:22:07.302 INFO:tasks.rados.rados.0.smithi076.stdout:602: done (3 left) 2021-03-15T22:22:07.302 INFO:tasks.rados.rados.0.smithi076.stdout:603: done (2 left) 2021-03-15T22:22:07.302 INFO:tasks.rados.rados.0.smithi076.stdout:604: done (1 left) 2021-03-15T22:22:07.303 INFO:tasks.rados.rados.0.smithi076.stdout:606: done (0 left) 2021-03-15T22:22:07.303 INFO:tasks.rados.rados.0.smithi076.stdout:607: snap_remove snap 54 2021-03-15T22:22:07.342 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:07.368 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:07.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:07.370 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:07.435 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[62156]: cluster 2021-03-15T22:22:05.628754+0000 osd.6 (osd.6) 30 : cluster [DBG] 7.d scrub ok 2021-03-15T22:22:07.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[62156]: cluster 2021-03-15T22:22:06.142030+0000 osd.5 (osd.5) 45 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:22:07.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:07 smithi076 conmon[62156]: cluster 2021-03-15T22:22:06.723747+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2021-03-15T22:22:07.442 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:07.471 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:07.496 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:07.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:07.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:07.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:07 smithi125 conmon[56739]: cluster 2021-03-15T22:22:05.628754+0000 osd.6 ( 2021-03-15T22:22:07.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:07 smithi125 conmon[56739]: osd.6) 30 : cluster [DBG] 7.d scrub ok 2021-03-15T22:22:07.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:07 smithi125 conmon[56739]: cluster 2021-03-15T22:22:06.142030+0000 osd.5 (osd.5) 45 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:22:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:07 smithi125 conmon[56739]: cluster 2021-03-15T22:22: 2021-03-15T22:22:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:07 smithi125 conmon[56739]: 06.723747+0000 osd.0 (osd.0) 48 : cluster [DBG] 6.10 scrub ok 2021-03-15T22:22:07.599 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:07.622 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:07.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:07.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:07.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:07.751 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:07.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:07.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:07.854 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:07.880 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:07.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:07.983 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:08.009 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:08.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:08.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:08.069 INFO:tasks.rados.rados.0.smithi076.stdout:607: done (0 left) 2021-03-15T22:22:08.069 INFO:tasks.rados.rados.0.smithi076.stdout:608: delete oid 43 current snap is 58 2021-03-15T22:22:08.082 INFO:tasks.rados.rados.0.smithi076.stdout:608: done (0 left) 2021-03-15T22:22:08.082 INFO:tasks.rados.rados.0.smithi076.stdout:609: write oid 29 current snap is 58 2021-03-15T22:22:08.082 INFO:tasks.rados.rados.0.smithi076.stdout:609: seq_num 244 ranges {500908=484052,1411724=689184} 2021-03-15T22:22:08.084 INFO:tasks.rados.rados.0.smithi076.stdout:609: writing smithi07694878-29 from 500908 to 984960 tid 1 2021-03-15T22:22:08.086 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.0 to 1 (10000) 2021-03-15T22:22:08.092 INFO:tasks.rados.rados.0.smithi076.stdout:609: writing smithi07694878-29 from 1411724 to 2100908 tid 2 2021-03-15T22:22:08.092 INFO:tasks.rados.rados.0.smithi076.stdout:609: finishing write tid 1 to smithi07694878-29 2021-03-15T22:22:08.093 INFO:tasks.rados.rados.0.smithi076.stdout:610: read oid 28 snap -1 2021-03-15T22:22:08.093 INFO:tasks.rados.rados.0.smithi076.stdout:610: expect (ObjNum 231 snap 52 seq_num 231) 2021-03-15T22:22:08.093 INFO:tasks.rados.rados.0.smithi076.stdout:611: delete oid 34 current snap is 58 2021-03-15T22:22:08.094 INFO:tasks.rados.rados.0.smithi076.stdout:611: done (2 left) 2021-03-15T22:22:08.094 INFO:tasks.rados.rados.0.smithi076.stdout:612: read oid 4 snap -1 2021-03-15T22:22:08.095 INFO:tasks.rados.rados.0.smithi076.stdout:612: expect (ObjNum 239 snap 57 seq_num 239) 2021-03-15T22:22:08.095 INFO:tasks.rados.rados.0.smithi076.stdout:613: delete oid 11 current snap is 58 2021-03-15T22:22:08.098 INFO:tasks.rados.rados.0.smithi076.stdout:612: expect (ObjNum 239 snap 57 seq_num 239) 2021-03-15T22:22:08.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:08.135 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:08.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:08.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:08.190 INFO:tasks.rados.rados.0.smithi076.stdout:609: finishing write tid 2 to smithi07694878-29 2021-03-15T22:22:08.191 INFO:tasks.rados.rados.0.smithi076.stdout:609: finishing write tid 3 to smithi07694878-29 2021-03-15T22:22:08.191 INFO:tasks.rados.rados.0.smithi076.stdout:609: finishing write tid 4 to smithi07694878-29 2021-03-15T22:22:08.191 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 354 (ObjNum 244 snap 58 seq_num 244) dirty exists 2021-03-15T22:22:08.191 INFO:tasks.rados.rados.0.smithi076.stdout:609: left oid 29 (ObjNum 244 snap 58 seq_num 244) 2021-03-15T22:22:08.191 INFO:tasks.rados.rados.0.smithi076.stdout:610: expect (ObjNum 231 snap 52 seq_num 231) 2021-03-15T22:22:08.237 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:08.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:08.264 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:08.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:08 smithi076 conmon[56514]: cluster 2021-03-15T22:22:07.005855+0000 mgr.y (mgr.14144) 459 : cluster [DBG] pgmap v444: 138 pgs: 138 active+clean; 141 MiB data, 461 MiB used, 625 GiB / 626 GiB avail; 8.2 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2021-03-15T22:22:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:08 smithi076 conmon[56514]: cluster 2021-03-15T22:22:07.066594+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e212: 8 total, 8 up, 7 in 2021-03-15T22:22:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:08 smithi076 conmon[56514]: audit 2021-03-15T22:22:07.107966+0000 mon.a (mon.0) 843 : audit [INF] from='client.? 172.21.15.76:0/1939037667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-15T22:22:08.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:08 smithi076 conmon[56514]: cluster 2021-03-15T22:22:07.679355+0000 osd.0 (osd.0) 49 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:22:08.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:08 smithi076 conmon[62156]: cluster 2021-03-15T22:22:07.005855+0000 2021-03-15T22:22:08.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:08 smithi076 conmon[62156]: mgr.y (mgr.14144) 459 : cluster [DBG] pgmap v444: 138 pgs: 138 active+clean; 141 MiB data, 461 MiB used, 625 GiB / 626 GiB avail; 8.2 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2021-03-15T22:22:08.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:08 smithi076 conmon[62156]: cluster 2021-03-15T22:22:07.066594+0000 mon.a (mon.0) 842 : cluster [DBG] osdmap e212: 8 total, 8 up, 7 in 2021-03-15T22:22:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:08 smithi076 conmon[62156]: audit 2021-03-15T22:22:07.107966+0000 mon.a (mon.0) 843 : audit [INF] from='client.? 172.21.15.76:0/1939037667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-15T22:22:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:08 smithi076 conmon[62156]: cluster 2021-03-15T22:22:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:08 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:22:08.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:08 smithi076 conmon[62156]: 22:07.679355+0000 osd.0 (osd.0) 49 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:22:08.365 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:08.387 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:08.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:08.389 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:08.422 INFO:tasks.rados.rados.0.smithi076.stdout:609: done (3 left) 2021-03-15T22:22:08.423 INFO:tasks.rados.rados.0.smithi076.stdout:610: done (2 left) 2021-03-15T22:22:08.423 INFO:tasks.rados.rados.0.smithi076.stdout:612: done (1 left) 2021-03-15T22:22:08.423 INFO:tasks.rados.rados.0.smithi076.stdout:613: done (0 left) 2021-03-15T22:22:08.424 INFO:tasks.rados.rados.0.smithi076.stdout:614: copy_from oid 17 from oid 14 current snap is 58 2021-03-15T22:22:08.424 INFO:tasks.rados.rados.0.smithi076.stdout:615: copy_from oid 30 from oid 7 current snap is 58 2021-03-15T22:22:08.424 INFO:tasks.rados.rados.0.smithi076.stdout:616: snap_remove snap 57 2021-03-15T22:22:08.490 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:08.512 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:08.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:08.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:08.536 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[67289]: debug 2021-03-15T22:22:08.536 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[67289]: 2021-03-15T22:22:08.072+0000 7fd4d9833700 -1 osd.6 pg_epoch: 213 pg[7.d( v 211'36 (0'0,211'36] local-lis/les=91/92 n=2 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [6,1,0] r=0 lpr=91 crt=211'36 lcod 209'34 mlcod 209'34 active+clean+snaptrim trimq=[6c~1] ps=[66~1,68~3,6e~1]] removing snap head 2021-03-15T22:22:08.537 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[67289]: 2021-03-15T22:22:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[56739]: cluster 2021-03-15T22:22:07. 2021-03-15T22:22:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[56739]: 005855+0000 mgr.y (mgr.14144) 459 : cluster [DBG] pgmap v444: 138 pgs: 138 active+clean; 141 MiB data, 461 MiB used, 625 GiB / 626 GiB avail; 8.2 MiB/s rd, 1.2 MiB/s wr, 10 op/s 2021-03-15T22:22:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[56739]: cluster 2021-03-15T22:22:07. 2021-03-15T22:22:08.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[56739]: 066594+0000 mon.a (mon.0) 842 : cluster [DBG] 2021-03-15T22:22:08.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[56739]: osdmap e212: 8 total, 8 up, 7 in 2021-03-15T22:22:08.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[56739]: audit 2021-03-15T22:22:08.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[56739]: 2021-03-15T22:22:07.107966+0000 mon.a ( 2021-03-15T22:22:08.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[56739]: mon.0) 843 : audit [INF] from='client.? 172.21.15.76:0/1939037667' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-15T22:22:08.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[56739]: cluster 2021-03-15T22:22: 2021-03-15T22:22:08.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:08 smithi125 conmon[56739]: 07.679355+0000 osd.0 (osd.0) 49 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:22:08.541 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:22:08.615 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:08.640 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:08.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:08.642 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:08.743 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:08.769 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:08.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:08.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:08.872 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:08.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:08.896 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:08.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:08.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:09.020 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:09.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:09.022 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:09.085 INFO:tasks.rados.rados.0.smithi076.stdout:616: done (2 left) 2021-03-15T22:22:09.086 INFO:tasks.rados.rados.0.smithi076.stdout:617: snap_create 2021-03-15T22:22:09.086 INFO:tasks.rados.rados.0.smithi076.stdout:614: finishing copy_from to smithi07694878-17 2021-03-15T22:22:09.086 INFO:tasks.rados.rados.0.smithi076.stdout:614: got expected ENOENT (src dne) 2021-03-15T22:22:09.087 INFO:tasks.rados.rados.0.smithi076.stdout:614: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:22:09.087 INFO:tasks.rados.rados.0.smithi076.stdout:615: finishing copy_from to smithi07694878-30 2021-03-15T22:22:09.087 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 245 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:22:09.087 INFO:tasks.rados.rados.0.smithi076.stdout:615: finishing copy_from racing read to smithi07694878-30 2021-03-15T22:22:09.089 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:22:09.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:09.149 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:09.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:09.150 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:09.251 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:09.277 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:09.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:09.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:09.349 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[56514]: cluster 2021-03-15T22:22:07.676462+0000 osd.2 (osd.2) 2021-03-15T22:22:09.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[56514]: 40 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:22:09.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[56514]: cluster 2021-03-15T22:22:07.718405+0000 osd.6 (osd.6) 31 : cluster [DBG] 7.7 deep-scrub ok 2021-03-15T22:22:09.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[56514]: cluster 2021-03-15T22:22:07.968913+0000 osd.1 (osd.1) 48 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:22:09.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[56514]: audit 2021-03-15T22:22:08.066887+0000 mon.a (mon.0) 844 : audit [INF] from='client.? 172.21.15.76:0/1939037667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-15T22:22:09.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[56514]: cluster 2021-03-15T22:22:08.066973+0000 mon.a (mon.0) 845 : cluster [DBG] osdmap e213: 8 total, 8 up, 7 in 2021-03-15T22:22:09.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[56514]: audit 2021-03-15T22:22:08.218164+0000 mon.c (mon.1) 21 : audit [INF] from='client.? 172.21.15.76:0/339047808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:22:09.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[56514]: audit 2021-03-15T22:22:08.218529+0000 mon.a (mon.0) 846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:22:09.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[56514]: cluster 2021-03-15T22:22:08.294848+0000 osd.4 (osd.4) 54 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:22:09.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: cluster 2021-03-15T22:22:07.676462 2021-03-15T22:22:09.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: +0000 osd.2 (osd.2) 40 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:22:09.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: cluster 2021-03-15T22:22:07.718405+0000 osd.6 (osd.6) 2021-03-15T22:22:09.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: 31 : cluster [DBG] 7.7 deep-scrub ok 2021-03-15T22:22:09.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: cluster 2021-03-15T22:22:09.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: 2021-03-15T22:22:07 2021-03-15T22:22:09.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: .968913+0000 osd.1 (osd 2021-03-15T22:22:09.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: .1) 48 : cluster [DBG] 2021-03-15T22:22:09.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: 4.14 scrub ok 2021-03-15T22:22:09.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: audit 2021-03-15T22:22:09.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:22:09.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: 22:08.066887+0000 2021-03-15T22:22:09.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: mon.a (mon.0) 844 : audit 2021-03-15T22:22:09.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/1939037667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-15T22:22:09.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:22:09.356 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: -03-15T22:22: 2021-03-15T22:22:09.356 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: 08.066973+0000 mon.a ( 2021-03-15T22:22:09.356 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: mon.0) 845 : cluster 2021-03-15T22:22:09.357 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: [DBG] osdmap e213: 8 total, 8 up, 7 in 2021-03-15T22:22:09.357 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: audit 2021-03-15T22:22:09.357 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:22:09.357 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: :22:08.218164+0000 2021-03-15T22:22:09.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: mon.c (mon.1) 21 2021-03-15T22:22:09.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/339047808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:22:09.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:22:09.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: 15T22:22:08.218529 2021-03-15T22:22:09.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2021-03-15T22:22:09.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: 846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:22:09.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:22:09.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: -03-15T22:22: 2021-03-15T22:22:09.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: 08.294848+0000 osd.4 (osd 2021-03-15T22:22:09.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: .4) 54 : cluster [DBG] 2021-03-15T22:22:09.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:09 smithi076 conmon[62156]: 6.1e scrub ok 2021-03-15T22:22:09.379 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:09.403 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:09.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:09.404 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:09.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 1 1 2021-03-15T22:22:09.506 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:09.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: cluster 2021-03-15T22:22:07.676462+0000 osd.2 (osd.2) 40 2021-03-15T22:22:09.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: : cluster [DBG] 7.3 scrub ok 2021-03-15T22:22:09.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: cluster 2021-03-15T22:22:07.718405+0000 osd.6 (osd.6) 31 : cluster [DBG] 7.7 deep-scrub ok 2021-03-15T22:22:09.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: cluster 2021-03-15T22:22:07.968913+0000 osd.1 (osd 2021-03-15T22:22:09.523 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: .1) 48 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:22:09.523 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: 2021-03-15T22:22:09.523 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: audit 2021-03-15T22:22:08 2021-03-15T22:22:09.523 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: .066887+0000 mon.a (mon.0) 844 : audit [INF] from='client.? 172.21.15.76:0/1939037667' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-15T22:22:09.523 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: cluster 2021-03-15T22:22:08.066973+0000 mon.a (mon 2021-03-15T22:22:09.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: .0) 845 : cluster [DBG] osdmap e213: 8 total, 8 up, 7 in 2021-03-15T22:22:09.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:22:09.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: :22:08.218164+0000 mon.c (mon.1) 21 : audit 2021-03-15T22:22:09.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/339047808' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:22:09.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:22:09.525 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: 03-15T22:22:08.218529+0000 2021-03-15T22:22:09.525 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: mon.a (mon.0) 846 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:22:09.525 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: 2021-03-15T22:22:09.525 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: cluster 2021-03-15T22:22 2021-03-15T22:22:09.526 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: :08.294848+0000 osd.4 (osd.4) 2021-03-15T22:22:09.526 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[56739]: 54 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:22:09.531 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:09.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:09.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:09.628 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:22:09.628 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:22:09.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:09.658 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:09.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:09.660 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:09.761 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:09.785 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:09.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:09.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:22:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:22:09.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:09.908 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:09.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:09.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:10.011 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:10.030 INFO:tasks.rados.rados.0.smithi076.stdout:614: done (2 left) 2021-03-15T22:22:10.030 INFO:tasks.rados.rados.0.smithi076.stdout:615: done (1 left) 2021-03-15T22:22:10.031 INFO:tasks.rados.rados.0.smithi076.stdout:617: done (0 left) 2021-03-15T22:22:10.031 INFO:tasks.rados.rados.0.smithi076.stdout:618: delete oid 39 current snap is 59 2021-03-15T22:22:10.035 INFO:tasks.rados.rados.0.smithi076.stdout:618: done (0 left) 2021-03-15T22:22:10.036 INFO:tasks.rados.rados.0.smithi076.stdout:619: snap_create 2021-03-15T22:22:10.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:10.037 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:10.037 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:10.109 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:10.139 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:10.161 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:10.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:10.162 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:10.263 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:10.289 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:10.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:10.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:10.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:10.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:10.418 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:10.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[56514]: cluster 2021-03-15T22:22:08.682237+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:22:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[56514]: cluster 2021-03-15T22:22:08.753078+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:22:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[56514]: cluster 2021-03-15T22:22:09.006192+0000 mgr.y (mgr.14144) 460 : cluster [DBG] pgmap v447: 138 pgs: 138 active+clean; 138 MiB data, 456 MiB used, 625 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-03-15T22:22:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[56514]: cluster 2021-03-15T22:22:09.072856+0000 mon.a (mon.0) 847 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[56514]: audit 2021-03-15T22:22:09.080763+0000 mon.a (mon.0) 848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:22:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[56514]: cluster 2021-03-15T22:22:09.080859+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2021-03-15T22:22:10.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[56514]: audit 2021-03-15T22:22:09.520379+0000 mgr.y (mgr.14144) 461 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:10.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[56514]: cluster 2021-03-15T22:22:10.028017+0000 mon.a (mon.0) 850 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2021-03-15T22:22:10.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[62156]: cluster 2021-03-15T22:22:08.682237+0000 osd.6 (osd.6) 32 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:22:10.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[62156]: cluster 2021-03-15T22:22:08.753078+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:22:10.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[62156]: cluster 2021-03-15T22:22:09.006192+0000 mgr.y (mgr.14144) 460 : cluster [DBG] pgmap v447: 138 pgs: 138 active+clean; 138 MiB data, 456 MiB used, 625 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-03-15T22:22:10.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[62156]: cluster 2021-03-15T22:22:09.072856+0000 mon.a (mon.0) 847 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:10.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[62156]: audit 2021-03-15T22:22:09.080763+0000 mon.a (mon.0) 848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:22:10.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[62156]: cluster 2021-03-15T22:22:09.080859+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2021-03-15T22:22:10.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[62156]: audit 2021-03-15T22:22:09.520379+0000 mgr.y (mgr.14144) 461 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:10.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:10 smithi076 conmon[62156]: cluster 2021-03-15T22:22:10.028017+0000 mon.a (mon.0) 850 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2021-03-15T22:22:10.521 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:10.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: cluster 2021-03-15T22:22:08.682237+0000 osd.6 (osd.6) 32 : cluster [DBG] 2021-03-15T22:22:10.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: 5.11 scrub ok 2021-03-15T22:22:10.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: cluster 2021-03-15T22:22:08.753078+0000 osd.7 (osd.7) 52 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:22:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: cluster 2021-03-15T22:22:09.006192+0000 mgr.y (mgr.14144) 460 : cluster [DBG] pgmap v447: 138 pgs: 138 active+clean; 138 MiB data, 456 MiB used, 625 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-03-15T22:22:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: cluster 2021-03-15T22:22:09.072856+0000 mon.a (mon.0) 847 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: audit 2021-03-15T22:22:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: 2021-03-15T22:22:09.080763+0000 mon.a (mon.0) 848 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:22:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: cluster 2021-03-15T22:22: 2021-03-15T22:22:10.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: 09.080859+0000 mon.a (mon.0) 849 : cluster [DBG] osdmap e214: 8 total, 8 up, 7 in 2021-03-15T22:22:10.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:22:10.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: 22:22:09.520379+0000 mgr.y (mgr.14144) 461 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:10.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: 2021-03-15T22:22:10.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:22:10.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:10 smithi125 conmon[56739]: -15T22:22:10.028017+0000 mon.a (mon.0) 850 : cluster [DBG] osdmap e215: 8 total, 8 up, 7 in 2021-03-15T22:22:10.543 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:10.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:10.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:10.646 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:10.673 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:10.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:10.674 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:10.775 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:10.798 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:10.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:10.800 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:10.901 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:10.924 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:10.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:10.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:11.027 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:11.038 INFO:tasks.rados.rados.0.smithi076.stdout:619: done (0 left) 2021-03-15T22:22:11.038 INFO:tasks.rados.rados.0.smithi076.stdout:620: copy_from oid 1 from oid 20 current snap is 60 2021-03-15T22:22:11.039 INFO:tasks.rados.rados.0.smithi076.stdout:621: read oid 48 snap 58 2021-03-15T22:22:11.039 INFO:tasks.rados.rados.0.smithi076.stdout:621: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:22:11.039 INFO:tasks.rados.rados.0.smithi076.stdout:622: copy_from oid 45 from oid 30 current snap is 60 2021-03-15T22:22:11.039 INFO:tasks.rados.rados.0.smithi076.stdout:623: rollback oid 14 current snap is 60 2021-03-15T22:22:11.040 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 14 to 58 2021-03-15T22:22:11.040 INFO:tasks.rados.rados.0.smithi076.stdout:624: snap_remove snap 55 2021-03-15T22:22:11.059 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:11.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:11.060 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:11.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:11.187 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:11.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:11.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[56514]: cluster 2021-03-15T22:22:09.544033+0000 2021-03-15T22:22:11.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[56514]: osd.1 (osd.1) 49 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:11.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[56514]: cluster 2021-03-15T22:22:09.544281+0000 osd.1 (osd.1) 50 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:11.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[56514]: cluster 2021-03-15T22:22:11.035596+0000 mon.a (mon.0) 851 : cluster [DBG] osdmap e216: 8 total, 8 up, 7 in 2021-03-15T22:22:11.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[56514]: audit 2021-03-15T22:22:11.040174+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:22:11.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[62156]: cluster 2021-03-15T22:22:09.544033 2021-03-15T22:22:11.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 49 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:11.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:22:11.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[62156]: :22:09.544281+0000 osd.1 (osd.1) 50 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:11.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[62156]: cluster 2021-03-15T22:22:11.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[62156]: 2021-03-15T22:22:11.035596+0000 2021-03-15T22:22:11.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[62156]: mon.a (mon.0) 851 : cluster [DBG] 2021-03-15T22:22:11.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[62156]: osdmap e216: 8 total, 8 up, 7 in 2021-03-15T22:22:11.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:22:11.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[62156]: 22:11.040174+0000 mon.a (mon.0 2021-03-15T22:22:11.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:11 smithi076 conmon[62156]: ) 852 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:22:11.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:11.312 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:11.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:11.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:11.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-15T22:22:11.415 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:11.450 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:11.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:11.451 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:11.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:11 smithi125 conmon[56739]: cluster 2021-03-15T22:22:09.544033+0000 osd.1 (osd.1) 49 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:11.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:11 smithi125 conmon[56739]: cluster 2021-03-15T22:22:09.544281+0000 2021-03-15T22:22:11.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:11 smithi125 conmon[56739]: osd.1 (osd.1) 50 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:11.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:11 smithi125 conmon[56739]: cluster 2021-03-15T22:22:11.035596+0000 mon.a (mon.0) 851 : cluster [DBG] osdmap e216: 8 total, 8 up, 7 in 2021-03-15T22:22:11.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:11 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:22:11.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:11 smithi125 conmon[56739]: 22:11.040174+0000 mon.a (mon.0) 852 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:22:11.552 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:11.573 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:11.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:11.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:11.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:11.705 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:11.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:11.706 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:11.807 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:11.830 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:11.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:11.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:11.933 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:11.949 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:11.957 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:11.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:11.958 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:12.046 INFO:tasks.rados.rados.0.smithi076.stdout:624: done (4 left) 2021-03-15T22:22:12.047 INFO:tasks.rados.rados.0.smithi076.stdout:625: read oid 19 snap -1 2021-03-15T22:22:12.047 INFO:tasks.rados.rados.0.smithi076.stdout:625: expect (ObjNum 191 snap 42 seq_num 191) 2021-03-15T22:22:12.047 INFO:tasks.rados.rados.0.smithi076.stdout:626: copy_from oid 21 from oid 39 current snap is 60 2021-03-15T22:22:12.047 INFO:tasks.rados.rados.0.smithi076.stdout:627: snap_create 2021-03-15T22:22:12.048 INFO:tasks.rados.rados.0.smithi076.stdout:623: finishing rollback tid 1 to smithi07694878-14 2021-03-15T22:22:12.048 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 0 (ObjNum 1902431520 snap 32564 seq_num 261630592) dirty dne 2021-03-15T22:22:12.048 INFO:tasks.rados.rados.0.smithi076.stdout:621: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:22:12.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:12.090 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:12.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:12.193 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:12.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[62156]: cluster 2021-03-15T22:22:11 2021-03-15T22:22:12.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[62156]: .006609+0000 mgr.y (mgr.14144) 462 : cluster [DBG] pgmap v450: 138 pgs: 138 active+clean; 133 MiB data, 445 MiB used, 625 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2021-03-15T22:22:12.194 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[62156]: cluster 2021-03-15T22:22:12.041932+0000 mon.a (mon.0) 853 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2021-03-15T22:22:12.195 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[56514]: cluster 2021-03-15T22:22:12.195 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:22:12.195 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[56514]: 22:11.006609+0000 mgr.y (mgr.14144) 462 : cluster [DBG] pgmap v450: 138 pgs: 138 active+clean; 133 MiB data, 445 MiB used, 625 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2021-03-15T22:22:12.196 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:22:12.196 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[56514]: -15T22:22:12.041932+0000 mon.a (mon.0 2021-03-15T22:22:12.196 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[56514]: ) 853 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2021-03-15T22:22:12.196 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[65511]: debug 2021-03-15T22:22:12.197 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[65511]: 2021-03-15T22:22:12.055+0000 7f1a20879700 -1 osd.0 pg_epoch: 217 pg[7.4( v 214'158 (0'0,214'158] local-lis/les=91/92 n=4 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [0,4,1] r=0 lpr=91 crt=214'158 lcod 213'156 mlcod 213'156 active+clean+snaptrim trimq=[6d~1] ps=[69~1,6c~1,6e~1,70~1]] removing snap head 2021-03-15T22:22:12.197 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[65511]: debug 2021-03-15T22:22:12.197 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[65511]: 2021-03-15T22:22:12.057+0000 7f1a2307e700 -1 osd.0 pg_epoch: 217 pg[7.f( v 217'196 (0'0,217'196] local-lis/les=203/204 n=5 ec=91/91 lis/c=203/203 les/c/f=204/204/0 sis=203) [0,7,4] r=0 lpr=203 luod=214'194 crt=217'196 lcod 213'192 mlcod 213'192 active+clean+snaptrim trimq=[6d~1] ps=[69~1,6c~1,6e~1,70~1]] removing snap head 2021-03-15T22:22:12.197 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[65511]: 2021-03-15T22:22:12.198 INFO:journalctl@ceph.osd.1.smithi076.stdout:Mar 15 22:22:12 smithi076 conmon[68886]: debug 2021-03-15T22:22:12.043+0000 7fae34dad700 -1 osd.1 pg_epoch: 217 pg[7.b( v 214'115 (0'0,214'115] local-lis/les=91/92 n=4 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [1,5,6] r=0 lpr=91 crt=214'115 lcod 213'113 mlcod 213'113 active+clean+snaptrim trimq=[6d~1] ps=[69~1,6c~1,6e~1,70~1]] removing snap head 2021-03-15T22:22:12.200 INFO:tasks.rados.rados.0.smithi076.stdout:622: finishing copy_from to smithi07694878-45 2021-03-15T22:22:12.200 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 245 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:22:12.200 INFO:tasks.rados.rados.0.smithi076.stdout:622: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:22:12.201 INFO:tasks.rados.rados.0.smithi076.stdout:620: finishing copy_from to smithi07694878-1 2021-03-15T22:22:12.201 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 148 (ObjNum 198 snap 43 seq_num 198) dirty exists 2021-03-15T22:22:12.201 INFO:tasks.rados.rados.0.smithi076.stdout:620: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:22:12.201 INFO:tasks.rados.rados.0.smithi076.stdout:625: expect (ObjNum 191 snap 42 seq_num 191) 2021-03-15T22:22:12.223 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:12.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:12.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:12.320 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:22:12.325 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:12.337 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:22:12.348 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:12.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:12.349 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:12.354 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:22:12.354 INFO:tasks.rados.rados.0.smithi076.stdout:626: finishing copy_from to smithi07694878-21 2021-03-15T22:22:12.355 INFO:tasks.rados.rados.0.smithi076.stdout:626: got expected ENOENT (src dne) 2021-03-15T22:22:12.355 INFO:tasks.rados.rados.0.smithi076.stdout:626: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:22:12.371 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:22:12.390 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:22:12.407 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:22:12.425 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:22:12.443 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:22:12.444 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:22:12.444 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:22:12.444 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:22:12.445 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:22:12.445 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:22:12.445 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:22:12.445 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:22:12.445 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:22:12.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:12.476 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:12.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:12.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:12.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:12 smithi125 conmon[56739]: cluster 2021-03-15T22:22:11.006609+0000 mgr.y (mgr.14144) 462 : cluster [DBG] pgmap v450: 138 pgs: 138 active+clean; 133 MiB data, 445 MiB used, 625 GiB / 626 GiB avail; 6.6 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2021-03-15T22:22:12.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:12 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:22:12.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:12 smithi125 conmon[56739]: 22:22:12.041932+0000 mon.a (mon.0) 853 : cluster [DBG] osdmap e217: 8 total, 8 up, 7 in 2021-03-15T22:22:12.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:12.601 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:12.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:12.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:12.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:12.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:12.729 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:12.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:12.795 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:22:12.831 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:12.859 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:12.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:12.861 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:12.962 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:12.987 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:12.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:12.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:13.047 INFO:tasks.rados.rados.0.smithi076.stdout:620: done (6 left) 2021-03-15T22:22:13.048 INFO:tasks.rados.rados.0.smithi076.stdout:621: done (5 left) 2021-03-15T22:22:13.048 INFO:tasks.rados.rados.0.smithi076.stdout:622: done (4 left) 2021-03-15T22:22:13.048 INFO:tasks.rados.rados.0.smithi076.stdout:623: done (3 left) 2021-03-15T22:22:13.048 INFO:tasks.rados.rados.0.smithi076.stdout:625: done (2 left) 2021-03-15T22:22:13.049 INFO:tasks.rados.rados.0.smithi076.stdout:626: done (1 left) 2021-03-15T22:22:13.049 INFO:tasks.rados.rados.0.smithi076.stdout:627: done (0 left) 2021-03-15T22:22:13.049 INFO:tasks.rados.rados.0.smithi076.stdout:628: read oid 44 snap 59 2021-03-15T22:22:13.049 INFO:tasks.rados.rados.0.smithi076.stdout:628: expect (ObjNum 99 snap 20 seq_num 99) 2021-03-15T22:22:13.049 INFO:tasks.rados.rados.0.smithi076.stdout:629: snap_create 2021-03-15T22:22:13.053 INFO:tasks.rados.rados.0.smithi076.stdout:628: expect (ObjNum 99 snap 20 seq_num 99) 2021-03-15T22:22:13.090 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:13.118 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:13.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:13.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:13.221 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:13.242 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:13.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:13.244 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:13.345 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:13.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:13 smithi076 conmon[62156]: audit 2021-03-15T22:22:12.300546+0000 mon.a (mon. 2021-03-15T22:22:13.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:13 smithi076 conmon[62156]: 0) 854 : audit [DBG] from='client.? 172.21.15.76:0/1736717921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:13.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:13 smithi076 conmon[62156]: cluster 2021-03-15T22:22:13.043967+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e218: 8 total, 8 up, 7 in 2021-03-15T22:22:13.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:13 smithi076 conmon[56514]: audit 2021-03-15T22:22:13.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:13 smithi076 conmon[56514]: 2021-03-15T22:22:13.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:13 smithi076 conmon[56514]: 2021-03-15T22:22:12.300546+0000 mon.a 2021-03-15T22:22:13.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:13 smithi076 conmon[56514]: (mon.0) 854 : audit [DBG] from='client.? 172.21.15.76:0/1736717921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:13.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:13 smithi076 conmon[56514]: cluster 2021-03-15T22:22:13.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:13 smithi076 conmon[56514]: 2021-03-15T22:22:13.043967+0000 mon.a (mon.0) 855 : cluster [DBG] 2021-03-15T22:22:13.353 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:13 smithi076 conmon[56514]: osdmap e218: 8 total, 8 up, 7 in 2021-03-15T22:22:13.371 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:13.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:13.373 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:13.474 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:13.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:13.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:13.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:13 smithi125 conmon[56739]: audit 2021-03-15T22:22:12.300546+0000 mon.a (mon.0) 854 : audit 2021-03-15T22:22:13.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:13 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/1736717921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:13.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:13 smithi125 conmon[56739]: cluster 2021-03-15T22:22:13.043967+0000 mon.a (mon.0) 855 : cluster [DBG] osdmap e218: 8 total, 8 up, 7 in 2021-03-15T22:22:13.599 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:13.621 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:13.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:13.623 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:13.724 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:13.750 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:13.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:13.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:13.852 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:13.878 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:13.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:13.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:13.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:14.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:14.005 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:14.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:14.050 INFO:tasks.rados.rados.0.smithi076.stdout:628: done (1 left) 2021-03-15T22:22:14.050 INFO:tasks.rados.rados.0.smithi076.stdout:629: done (0 left) 2021-03-15T22:22:14.050 INFO:tasks.rados.rados.0.smithi076.stdout:630: delete oid 2 current snap is 62 2021-03-15T22:22:14.052 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.1 to 1 (10000) 2021-03-15T22:22:14.054 INFO:tasks.rados.rados.0.smithi076.stdout:630: done (0 left) 2021-03-15T22:22:14.054 INFO:tasks.rados.rados.0.smithi076.stdout:631: write oid 37 current snap is 62 2021-03-15T22:22:14.055 INFO:tasks.rados.rados.0.smithi076.stdout:631: seq_num 249 ranges {584943=460297,1587781=597162} 2021-03-15T22:22:14.060 INFO:tasks.rados.rados.0.smithi076.stdout:631: writing smithi07694878-37 from 584943 to 1045240 tid 1 2021-03-15T22:22:14.067 INFO:tasks.rados.rados.0.smithi076.stdout:631: writing smithi07694878-37 from 1587781 to 2184943 tid 2 2021-03-15T22:22:14.067 INFO:tasks.rados.rados.0.smithi076.stdout:632: snap_create 2021-03-15T22:22:14.067 INFO:tasks.rados.rados.0.smithi076.stdout:631: finishing write tid 1 to smithi07694878-37 2021-03-15T22:22:14.071 INFO:tasks.rados.rados.0.smithi076.stdout:631: finishing write tid 2 to smithi07694878-37 2021-03-15T22:22:14.071 INFO:tasks.rados.rados.0.smithi076.stdout:631: finishing write tid 3 to smithi07694878-37 2021-03-15T22:22:14.071 INFO:tasks.rados.rados.0.smithi076.stdout:631: finishing write tid 4 to smithi07694878-37 2021-03-15T22:22:14.071 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 366 (ObjNum 249 snap 62 seq_num 249) dirty exists 2021-03-15T22:22:14.071 INFO:tasks.rados.rados.0.smithi076.stdout:631: left oid 37 (ObjNum 249 snap 62 seq_num 249) 2021-03-15T22:22:14.107 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:14.136 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:14.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:14.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:14.239 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:14.265 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:14.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:14.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:14.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:14.389 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:14.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:14.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:14.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:14 smithi076 conmon[56514]: cluster 2021-03-15T22:22:13.007018+0000 mgr.y (mgr.14144) 463 : cluster [DBG] pgmap v453: 138 pgs: 138 active+clean; 121 MiB data, 429 MiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2021-03-15T22:22:14.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:14 smithi076 conmon[56514]: audit 2021-03-15T22:22:13.756361+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.76:0/377888240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:22:14.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:14 smithi076 conmon[56514]: audit 2021-03-15T22:22:13.756660+0000 mon.a (mon.0) 856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:22:14.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:14 smithi076 conmon[56514]: audit 2021-03-15T22:22:14.047936+0000 mon.a (mon.0) 857 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-15T22:22:14.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:14 smithi076 conmon[56514]: cluster 2021-03-15T22:22:14.048024+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2021-03-15T22:22:14.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:14 smithi076 conmon[62156]: cluster 2021-03-15T22:22:13.007018+0000 mgr.y (mgr.14144) 463 : cluster [DBG] pgmap v453: 138 pgs: 138 active+clean; 121 MiB data, 429 MiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2021-03-15T22:22:14.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:14 smithi076 conmon[62156]: audit 2021-03-15T22:22:13.756361+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.76:0/377888240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:22:14.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:14 smithi076 conmon[62156]: audit 2021-03-15T22:22:13.756660+0000 mon.a (mon.0) 856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:22:14.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:14 smithi076 conmon[62156]: audit 2021-03-15T22:22:14.047936+0000 mon.a (mon.0) 857 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-15T22:22:14.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:14 smithi076 conmon[62156]: cluster 2021-03-15T22:22:14.048024+0000 mon.a (mon.0) 858 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2021-03-15T22:22:14.481 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:14.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:14.518 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:14.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:14.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:14 smithi125 conmon[56739]: cluster 2021-03-15T22:22:13.007018+0000 mgr.y (mgr.14144) 463 : cluster [DBG] 2021-03-15T22:22:14.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:14 smithi125 conmon[56739]: pgmap v453: 138 pgs: 138 active+clean; 121 MiB data, 429 MiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2021-03-15T22:22:14.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:14 smithi125 conmon[56739]: audit 2021-03-15T22:22:13.756361+0000 mon.c (mon.1) 22 : audit [INF] from='client.? 172.21.15.76:0/377888240' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:22:14.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:14 smithi125 conmon[56739]: audit 2021-03-15T22:22:13.756660+0000 mon.a (mon.0) 2021-03-15T22:22:14.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:14 smithi125 conmon[56739]: 856 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:22:14.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:14 smithi125 conmon[56739]: audit 2021-03-15T22:22 2021-03-15T22:22:14.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:14 smithi125 conmon[56739]: :14.047936+0000 mon.a (mon.0) 857 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-15T22:22:14.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:14 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:22:14.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:14 smithi125 conmon[56739]: 15T22:22:14.048024+0000 mon.a (mon. 2021-03-15T22:22:14.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:14 smithi125 conmon[56739]: 0) 858 : cluster [DBG] osdmap e219: 8 total, 8 up, 7 in 2021-03-15T22:22:14.621 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:14.646 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:14.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:14.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:14.749 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:14.770 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:14.772 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:14.873 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:14.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:14.897 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:14.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:14.999 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:15.025 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:15.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:15.027 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:15.032 INFO:tasks.rados.rados.0.smithi076.stdout:631: done (1 left) 2021-03-15T22:22:15.032 INFO:tasks.rados.rados.0.smithi076.stdout:632: done (0 left) 2021-03-15T22:22:15.033 INFO:tasks.rados.rados.0.smithi076.stdout:633: delete oid 28 current snap is 63 2021-03-15T22:22:15.040 INFO:tasks.rados.rados.0.smithi076.stdout:633: done (0 left) 2021-03-15T22:22:15.040 INFO:tasks.rados.rados.0.smithi076.stdout:634: copy_from oid 44 from oid 48 current snap is 63 2021-03-15T22:22:15.040 INFO:tasks.rados.rados.0.smithi076.stdout:635: copy_from oid 11 from oid 3 current snap is 63 2021-03-15T22:22:15.040 INFO:tasks.rados.rados.0.smithi076.stdout:636: read oid 9 snap 62 2021-03-15T22:22:15.041 INFO:tasks.rados.rados.0.smithi076.stdout:636: expect deleted 2021-03-15T22:22:15.041 INFO:tasks.rados.rados.0.smithi076.stdout:637: delete oid 15 current snap is 63 2021-03-15T22:22:15.042 INFO:tasks.rados.rados.0.smithi076.stdout:636: done (3 left) 2021-03-15T22:22:15.043 INFO:tasks.rados.rados.0.smithi076.stdout:637: done (2 left) 2021-03-15T22:22:15.043 INFO:tasks.rados.rados.0.smithi076.stdout:638: snap_remove snap 60 2021-03-15T22:22:15.127 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:15.150 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:15.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:15.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:15.182 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 2 1 2021-03-15T22:22:15.252 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:15.275 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:15.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:15.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:15.377 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:15.400 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:15.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:15.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:15.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:15.525 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:15.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:15.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:15.622 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:22:15.622 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:22:15.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:22:15.623 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:22:15.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.623 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:323 7.8 7:1d74a1c3:::smithi07694878-44:head [copy-from ver 3] snapc 8=[8] ondisk+write+known_if_redirected e104)", 2021-03-15T22:22:15.624 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:19.443283+0000", 2021-03-15T22:22:15.624 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 116.175259575, 2021-03-15T22:22:15.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015830329000000001, 2021-03-15T22:22:15.624 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.625 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:15.625 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.625 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.625 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.625 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 323 2021-03-15T22:22:15.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.626 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443283+0000", 2021-03-15T22:22:15.627 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.628 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.628 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443283+0000", 2021-03-15T22:22:15.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1140000000000002e-06 2021-03-15T22:22:15.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443286+0000", 2021-03-15T22:22:15.629 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2950000000000004e-06 2021-03-15T22:22:15.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443290+0000", 2021-03-15T22:22:15.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4205999999999999e-05 2021-03-15T22:22:15.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443304+0000", 2021-03-15T22:22:15.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5860000000000001e-06 2021-03-15T22:22:15.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.632 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.632 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443311+0000", 2021-03-15T22:22:15.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3111000000000001e-05 2021-03-15T22:22:15.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.633 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443354+0000", 2021-03-15T22:22:15.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2724000000000003e-05 2021-03-15T22:22:15.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443437+0000", 2021-03-15T22:22:15.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00847615 2021-03-15T22:22:15.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:22:15.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.451913+0000", 2021-03-15T22:22:15.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062090670000000004 2021-03-15T22:22:15.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.637 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:22:15.637 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.458122+0000", 2021-03-15T22:22:15.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073570700000000005 2021-03-15T22:22:15.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.638 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:15.638 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.458858+0000", 2021-03-15T22:22:15.638 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000171175 2021-03-15T22:22:15.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.639 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:15.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459029+0000", 2021-03-15T22:22:15.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2721e-05 2021-03-15T22:22:15.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:15.641 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459072+0000", 2021-03-15T22:22:15.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1473000000000001e-05 2021-03-15T22:22:15.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459113+0000", 2021-03-15T22:22:15.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015830329000000001 2021-03-15T22:22:15.642 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.643 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.643 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.643 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:324 7.8 7:1d74a1c3:::smithi07694878-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e104)", 2021-03-15T22:22:15.644 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:19.443323+0000", 2021-03-15T22:22:15.644 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 116.175219005, 2021-03-15T22:22:15.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01594897, 2021-03-15T22:22:15.644 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.645 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.645 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.645 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.645 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.645 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 324 2021-03-15T22:22:15.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.646 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443323+0000", 2021-03-15T22:22:15.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443323+0000", 2021-03-15T22:22:15.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2889999999999997e-06 2021-03-15T22:22:15.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443329+0000", 2021-03-15T22:22:15.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6029999999999999e-06 2021-03-15T22:22:15.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443330+0000", 2021-03-15T22:22:15.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2021-03-15T22:22:15.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443337+0000", 2021-03-15T22:22:15.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2849999999999996e-06 2021-03-15T22:22:15.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443342+0000", 2021-03-15T22:22:15.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025426499999999999 2021-03-15T22:22:15.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443596+0000", 2021-03-15T22:22:15.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7543999999999997e-05 2021-03-15T22:22:15.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:15.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443643+0000", 2021-03-15T22:22:15.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010718 2021-03-15T22:22:15.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.454361+0000", 2021-03-15T22:22:15.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000111568 2021-03-15T22:22:15.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:15.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.454473+0000", 2021-03-15T22:22:15.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0046729909999999996 2021-03-15T22:22:15.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459146+0000", 2021-03-15T22:22:15.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3789000000000003e-05 2021-03-15T22:22:15.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459200+0000", 2021-03-15T22:22:15.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2533999999999994e-05 2021-03-15T22:22:15.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459272+0000", 2021-03-15T22:22:15.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01594897 2021-03-15T22:22:15.660 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.661 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.661 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.662 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:531 7.8 7:1822d391:::smithi07694878-47:head [copy-from ver 14] snapc 19=[19,18,17] ondisk+write+known_if_redirected e122)", 2021-03-15T22:22:15.662 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:37.008160+0000", 2021-03-15T22:22:15.662 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 98.610382329999993, 2021-03-15T22:22:15.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029706733999999999, 2021-03-15T22:22:15.662 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.663 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:15.663 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.663 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.663 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.664 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 531 2021-03-15T22:22:15.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.664 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008160+0000", 2021-03-15T22:22:15.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008160+0000", 2021-03-15T22:22:15.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4279999999999999e-06 2021-03-15T22:22:15.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008162+0000", 2021-03-15T22:22:15.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9100000000000001e-06 2021-03-15T22:22:15.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008165+0000", 2021-03-15T22:22:15.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.204e-05 2021-03-15T22:22:15.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008177+0000", 2021-03-15T22:22:15.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2616000000000001e-05 2021-03-15T22:22:15.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008190+0000", 2021-03-15T22:22:15.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000128796 2021-03-15T22:22:15.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008319+0000", 2021-03-15T22:22:15.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000103112 2021-03-15T22:22:15.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008422+0000", 2021-03-15T22:22:15.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016673884 2021-03-15T22:22:15.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:22:15.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.025096+0000", 2021-03-15T22:22:15.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053861949999999999 2021-03-15T22:22:15.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:22:15.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.030482+0000", 2021-03-15T22:22:15.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071629290000000002 2021-03-15T22:22:15.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:15.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.037645+0000", 2021-03-15T22:22:15.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012967600000000001 2021-03-15T22:22:15.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:15.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.037775+0000", 2021-03-15T22:22:15.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9626999999999999e-05 2021-03-15T22:22:15.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:15.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.037814+0000", 2021-03-15T22:22:15.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2521000000000001e-05 2021-03-15T22:22:15.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.037867+0000", 2021-03-15T22:22:15.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029706733999999999 2021-03-15T22:22:15.680 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.680 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.680 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.681 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:532 7.8 7:1822d391:::smithi07694878-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e122)", 2021-03-15T22:22:15.681 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:37.008263+0000", 2021-03-15T22:22:15.682 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 98.610279285999994, 2021-03-15T22:22:15.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029847118999999998, 2021-03-15T22:22:15.682 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.682 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.682 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.683 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.683 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.683 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 532 2021-03-15T22:22:15.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.684 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008263+0000", 2021-03-15T22:22:15.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008263+0000", 2021-03-15T22:22:15.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.542e-06 2021-03-15T22:22:15.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008265+0000", 2021-03-15T22:22:15.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8369999999999999e-06 2021-03-15T22:22:15.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008266+0000", 2021-03-15T22:22:15.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0428000000000001e-05 2021-03-15T22:22:15.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008277+0000", 2021-03-15T22:22:15.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7709e-05 2021-03-15T22:22:15.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008325+0000", 2021-03-15T22:22:15.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025799800000000001 2021-03-15T22:22:15.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008583+0000", 2021-03-15T22:22:15.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8608e-05 2021-03-15T22:22:15.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:15.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008621+0000", 2021-03-15T22:22:15.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.021177707 2021-03-15T22:22:15.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.029799+0000", 2021-03-15T22:22:15.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4729999999999996e-05 2021-03-15T22:22:15.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:15.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.029894+0000", 2021-03-15T22:22:15.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080190699999999997 2021-03-15T22:22:15.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.037913+0000", 2021-03-15T22:22:15.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7991999999999994e-05 2021-03-15T22:22:15.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.038001+0000", 2021-03-15T22:22:15.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109498 2021-03-15T22:22:15.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.038110+0000", 2021-03-15T22:22:15.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029847118999999998 2021-03-15T22:22:15.698 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.698 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.698 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.699 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:539 7.2 e123/91 7:45ee0f2f:::smithi07694878-6:head v 123'14, mlcod=104'10)", 2021-03-15T22:22:15.699 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:38.116403+0000", 2021-03-15T22:22:15.699 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 97.502139618000001, 2021-03-15T22:22:15.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010556181, 2021-03-15T22:22:15.700 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.700 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:15.700 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.116403+0000", 2021-03-15T22:22:15.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.116403+0000", 2021-03-15T22:22:15.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001998086 2021-03-15T22:22:15.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118401+0000", 2021-03-15T22:22:15.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4809999999999997e-06 2021-03-15T22:22:15.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118406+0000", 2021-03-15T22:22:15.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3042999999999999e-05 2021-03-15T22:22:15.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118429+0000", 2021-03-15T22:22:15.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0678e-05 2021-03-15T22:22:15.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118440+0000", 2021-03-15T22:22:15.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4852999999999997e-05 2021-03-15T22:22:15.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118505+0000", 2021-03-15T22:22:15.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6299999999999999e-05 2021-03-15T22:22:15.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118531+0000", 2021-03-15T22:22:15.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083770519999999994 2021-03-15T22:22:15.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:15.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.126908+0000", 2021-03-15T22:22:15.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0688e-05 2021-03-15T22:22:15.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.126959+0000", 2021-03-15T22:22:15.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010556181 2021-03-15T22:22:15.711 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.712 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.712 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.712 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.3.0:218 7.2 e136/91 7:42f18a95:::smithi07694878-45:21 v 136'33, mlcod=132'29)", 2021-03-15T22:22:15.713 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:51.068004+0000", 2021-03-15T22:22:15.713 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 84.550538302000007, 2021-03-15T22:22:15.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070510140000000004, 2021-03-15T22:22:15.713 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.714 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:15.714 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.068004+0000", 2021-03-15T22:22:15.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.068004+0000", 2021-03-15T22:22:15.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.117e-06 2021-03-15T22:22:15.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.068007+0000", 2021-03-15T22:22:15.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.153e-06 2021-03-15T22:22:15.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.068014+0000", 2021-03-15T22:22:15.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8868999999999998e-05 2021-03-15T22:22:15.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.068033+0000", 2021-03-15T22:22:15.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0779999999999998e-06 2021-03-15T22:22:15.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.068040+0000", 2021-03-15T22:22:15.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0038999999999999e-05 2021-03-15T22:22:15.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.068100+0000", 2021-03-15T22:22:15.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5675000000000001e-05 2021-03-15T22:22:15.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.068126+0000", 2021-03-15T22:22:15.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069065669999999997 2021-03-15T22:22:15.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:15.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.075032+0000", 2021-03-15T22:22:15.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2515999999999999e-05 2021-03-15T22:22:15.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:51.075055+0000", 2021-03-15T22:22:15.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070510140000000004 2021-03-15T22:22:15.725 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.725 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.725 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.726 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:688 7.8 7:1ed4caeb:::smithi07694878-27:head [read 0~3697507,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e138)", 2021-03-15T22:22:15.726 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:53.096551+0000", 2021-03-15T22:22:15.726 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 82.521991619000005, 2021-03-15T22:22:15.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067351859999999998, 2021-03-15T22:22:15.727 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.727 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.727 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.727 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.728 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.728 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 688 2021-03-15T22:22:15.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.728 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096551+0000", 2021-03-15T22:22:15.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096551+0000", 2021-03-15T22:22:15.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.164e-06 2021-03-15T22:22:15.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096556+0000", 2021-03-15T22:22:15.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6440000000000002e-06 2021-03-15T22:22:15.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096559+0000", 2021-03-15T22:22:15.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0886e-05 2021-03-15T22:22:15.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096580+0000", 2021-03-15T22:22:15.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8960000000000003e-06 2021-03-15T22:22:15.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096588+0000", 2021-03-15T22:22:15.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9917000000000002e-05 2021-03-15T22:22:15.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096648+0000", 2021-03-15T22:22:15.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0319999999999992e-06 2021-03-15T22:22:15.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:22:15.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096657+0000", 2021-03-15T22:22:15.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018901090000000001 2021-03-15T22:22:15.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.098547+0000", 2021-03-15T22:22:15.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2558000000000005e-05 2021-03-15T22:22:15.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.098630+0000", 2021-03-15T22:22:15.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0046559799999999997 2021-03-15T22:22:15.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.103286+0000", 2021-03-15T22:22:15.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067351859999999998 2021-03-15T22:22:15.741 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.741 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.741 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.742 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:689 7.8 7:1ed4caeb:::smithi07694878-27:head [read 0~3697507,checksum] snapc 0=[] ondisk+read+known_if_redirected e138)", 2021-03-15T22:22:15.742 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:53.096603+0000", 2021-03-15T22:22:15.742 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 82.521938882000001, 2021-03-15T22:22:15.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084282479999999993, 2021-03-15T22:22:15.743 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.743 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.743 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.743 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.744 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.744 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 689 2021-03-15T22:22:15.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.744 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096603+0000", 2021-03-15T22:22:15.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096603+0000", 2021-03-15T22:22:15.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0040000000000002e-06 2021-03-15T22:22:15.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096605+0000", 2021-03-15T22:22:15.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.666e-06 2021-03-15T22:22:15.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096607+0000", 2021-03-15T22:22:15.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8990000000000004e-06 2021-03-15T22:22:15.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096614+0000", 2021-03-15T22:22:15.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0249999999999996e-06 2021-03-15T22:22:15.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096618+0000", 2021-03-15T22:22:15.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000115988 2021-03-15T22:22:15.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096734+0000", 2021-03-15T22:22:15.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5939999999999999e-06 2021-03-15T22:22:15.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:22:15.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096741+0000", 2021-03-15T22:22:15.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065763059999999996 2021-03-15T22:22:15.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.103317+0000", 2021-03-15T22:22:15.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4918000000000002e-05 2021-03-15T22:22:15.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.103402+0000", 2021-03-15T22:22:15.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016298479999999999 2021-03-15T22:22:15.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.105032+0000", 2021-03-15T22:22:15.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084282479999999993 2021-03-15T22:22:15.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.757 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.758 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:690 7.8 7:1ed4caeb:::smithi07694878-27:head [sparse-read 0~2056525] snapc 0=[] ondisk+read+known_if_redirected e138)", 2021-03-15T22:22:15.758 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:53.096635+0000", 2021-03-15T22:22:15.758 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 82.521907541000004, 2021-03-15T22:22:15.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086221779999999994, 2021-03-15T22:22:15.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.759 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.760 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.760 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 690 2021-03-15T22:22:15.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.760 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096635+0000", 2021-03-15T22:22:15.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096635+0000", 2021-03-15T22:22:15.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.09e-07 2021-03-15T22:22:15.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096636+0000", 2021-03-15T22:22:15.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0130000000000001e-06 2021-03-15T22:22:15.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096637+0000", 2021-03-15T22:22:15.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9890000000000003e-06 2021-03-15T22:22:15.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096641+0000", 2021-03-15T22:22:15.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5299999999999999e-06 2021-03-15T22:22:15.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096643+0000", 2021-03-15T22:22:15.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000120708 2021-03-15T22:22:15.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096764+0000", 2021-03-15T22:22:15.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2979999999999997e-06 2021-03-15T22:22:15.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:22:15.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.096770+0000", 2021-03-15T22:22:15.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0082927569999999996 2021-03-15T22:22:15.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.105063+0000", 2021-03-15T22:22:15.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8811000000000001e-05 2021-03-15T22:22:15.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.105142+0000", 2021-03-15T22:22:15.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011516299999999999 2021-03-15T22:22:15.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:53.105257+0000", 2021-03-15T22:22:15.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086221779999999994 2021-03-15T22:22:15.773 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.773 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.773 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.774 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:836 7.3 e149/91 7:c458c800:::smithi07694878-3:head v 149'65, mlcod=148'61)", 2021-03-15T22:22:15.774 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.030244+0000", 2021-03-15T22:22:15.774 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 71.588298566999995, 2021-03-15T22:22:15.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013182968, 2021-03-15T22:22:15.775 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.775 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:15.775 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.030244+0000", 2021-03-15T22:22:15.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.030244+0000", 2021-03-15T22:22:15.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036198279999999998 2021-03-15T22:22:15.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.033864+0000", 2021-03-15T22:22:15.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4490000000000002e-06 2021-03-15T22:22:15.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.033871+0000", 2021-03-15T22:22:15.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2189999999999999e-05 2021-03-15T22:22:15.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.033893+0000", 2021-03-15T22:22:15.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.91e-06 2021-03-15T22:22:15.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.033903+0000", 2021-03-15T22:22:15.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00096714999999999995 2021-03-15T22:22:15.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.034870+0000", 2021-03-15T22:22:15.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1061999999999999e-05 2021-03-15T22:22:15.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.034891+0000", 2021-03-15T22:22:15.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085107499999999992 2021-03-15T22:22:15.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:15.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043402+0000", 2021-03-15T22:22:15.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4629000000000001e-05 2021-03-15T22:22:15.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043427+0000", 2021-03-15T22:22:15.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013182968 2021-03-15T22:22:15.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.787 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.788 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:982 7.8 7:1822d391:::smithi07694878-47:head [copy-from ver 0] snapc 41=[41,40,3f,3d,30] ondisk+write+known_if_redirected e163)", 2021-03-15T22:22:15.788 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:18.043280+0000", 2021-03-15T22:22:15.788 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.575262506999998, 2021-03-15T22:22:15.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071198980000000004, 2021-03-15T22:22:15.789 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.789 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.789 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.789 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.789 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.790 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 982 2021-03-15T22:22:15.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.790 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.043280+0000", 2021-03-15T22:22:15.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.043280+0000", 2021-03-15T22:22:15.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8370000000000002e-06 2021-03-15T22:22:15.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.043286+0000", 2021-03-15T22:22:15.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7350000000000001e-06 2021-03-15T22:22:15.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.043288+0000", 2021-03-15T22:22:15.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2397e-05 2021-03-15T22:22:15.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.043301+0000", 2021-03-15T22:22:15.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7000000000000008e-06 2021-03-15T22:22:15.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.043308+0000", 2021-03-15T22:22:15.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034930300000000003 2021-03-15T22:22:15.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.043658+0000", 2021-03-15T22:22:15.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00020889199999999999 2021-03-15T22:22:15.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.043867+0000", 2021-03-15T22:22:15.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065330340000000001 2021-03-15T22:22:15.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.050400+0000", 2021-03-15T22:22:15.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071198980000000004 2021-03-15T22:22:15.800 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.800 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.801 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:983 7.8 7:1822d391:::smithi07694878-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e163)", 2021-03-15T22:22:15.801 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:18.043996+0000", 2021-03-15T22:22:15.802 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.574546400999999, 2021-03-15T22:22:15.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066464410000000003, 2021-03-15T22:22:15.802 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.802 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.803 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.804 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.804 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 983 2021-03-15T22:22:15.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.804 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.043996+0000", 2021-03-15T22:22:15.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.043996+0000", 2021-03-15T22:22:15.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6959999999999999e-06 2021-03-15T22:22:15.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.043998+0000", 2021-03-15T22:22:15.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.295e-06 2021-03-15T22:22:15.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.044000+0000", 2021-03-15T22:22:15.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0509999999999997e-06 2021-03-15T22:22:15.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.044008+0000", 2021-03-15T22:22:15.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3310000000000003e-06 2021-03-15T22:22:15.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.044013+0000", 2021-03-15T22:22:15.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2532e-05 2021-03-15T22:22:15.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.044046+0000", 2021-03-15T22:22:15.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5104e-05 2021-03-15T22:22:15.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:15.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.044091+0000", 2021-03-15T22:22:15.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063876669999999997 2021-03-15T22:22:15.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.050479+0000", 2021-03-15T22:22:15.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0270000000000001e-05 2021-03-15T22:22:15.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.050569+0000", 2021-03-15T22:22:15.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3495e-05 2021-03-15T22:22:15.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:18.050642+0000", 2021-03-15T22:22:15.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066464410000000003 2021-03-15T22:22:15.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.819 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.3.0:398 7.a e193/91 7:53dee572:::smithi07694878-35:5b v 193'188, mlcod=192'184)", 2021-03-15T22:22:15.819 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:48.094866+0000", 2021-03-15T22:22:15.819 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.523675815000001, 2021-03-15T22:22:15.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065923570000000001, 2021-03-15T22:22:15.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.820 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:15.820 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.094866+0000", 2021-03-15T22:22:15.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.094866+0000", 2021-03-15T22:22:15.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.446e-06 2021-03-15T22:22:15.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.094869+0000", 2021-03-15T22:22:15.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6110000000000001e-06 2021-03-15T22:22:15.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.094872+0000", 2021-03-15T22:22:15.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0508e-05 2021-03-15T22:22:15.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.094882+0000", 2021-03-15T22:22:15.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9649999999999996e-06 2021-03-15T22:22:15.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.094888+0000", 2021-03-15T22:22:15.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043010160000000004 2021-03-15T22:22:15.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.099189+0000", 2021-03-15T22:22:15.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6808000000000002e-05 2021-03-15T22:22:15.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.099216+0000", 2021-03-15T22:22:15.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022155270000000001 2021-03-15T22:22:15.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:15.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.101431+0000", 2021-03-15T22:22:15.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7475999999999999e-05 2021-03-15T22:22:15.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:48.101459+0000", 2021-03-15T22:22:15.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065923570000000001 2021-03-15T22:22:15.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.832 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.833 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1576 7.3 7:c458c800:::smithi07694878-3:6a [sparse-read 0~3899191,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e204)", 2021-03-15T22:22:15.833 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.066256+0000", 2021-03-15T22:22:15.833 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.552286119000001, 2021-03-15T22:22:15.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.05921051, 2021-03-15T22:22:15.834 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.834 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.834 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.834 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.835 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.835 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1576 2021-03-15T22:22:15.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.835 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066256+0000", 2021-03-15T22:22:15.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066256+0000", 2021-03-15T22:22:15.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8909999999999997e-06 2021-03-15T22:22:15.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066260+0000", 2021-03-15T22:22:15.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9709999999999998e-06 2021-03-15T22:22:15.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066264+0000", 2021-03-15T22:22:15.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6415000000000001e-05 2021-03-15T22:22:15.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066280+0000", 2021-03-15T22:22:15.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5329999999999997e-06 2021-03-15T22:22:15.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066288+0000", 2021-03-15T22:22:15.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6039999999999999e-05 2021-03-15T22:22:15.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066344+0000", 2021-03-15T22:22:15.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6260000000000007e-06 2021-03-15T22:22:15.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:22:15.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066354+0000", 2021-03-15T22:22:15.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0541787069999999 2021-03-15T22:22:15.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.120532+0000", 2021-03-15T22:22:15.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014794499999999999 2021-03-15T22:22:15.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.120680+0000", 2021-03-15T22:22:15.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047863819999999996 2021-03-15T22:22:15.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125467+0000", 2021-03-15T22:22:15.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.05921051 2021-03-15T22:22:15.848 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.848 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.848 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.849 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1577 7.3 7:c458c800:::smithi07694878-3:6a [read 0~3899191,checksum] snapc 0=[] ondisk+read+known_if_redirected e204)", 2021-03-15T22:22:15.849 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.066317+0000", 2021-03-15T22:22:15.849 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.552225391, 2021-03-15T22:22:15.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0607751919999999, 2021-03-15T22:22:15.850 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.850 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.851 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.851 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1577 2021-03-15T22:22:15.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.851 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066317+0000", 2021-03-15T22:22:15.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066317+0000", 2021-03-15T22:22:15.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5050000000000002e-06 2021-03-15T22:22:15.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066319+0000", 2021-03-15T22:22:15.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6270000000000001e-06 2021-03-15T22:22:15.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066321+0000", 2021-03-15T22:22:15.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2679999999999997e-06 2021-03-15T22:22:15.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066328+0000", 2021-03-15T22:22:15.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.313e-06 2021-03-15T22:22:15.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066333+0000", 2021-03-15T22:22:15.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9864000000000002e-05 2021-03-15T22:22:15.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066382+0000", 2021-03-15T22:22:15.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2140000000000001e-06 2021-03-15T22:22:15.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:22:15.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066389+0000", 2021-03-15T22:22:15.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0591109620000001 2021-03-15T22:22:15.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125500+0000", 2021-03-15T22:22:15.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074530899999999999 2021-03-15T22:22:15.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.126245+0000", 2021-03-15T22:22:15.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084712999999999998 2021-03-15T22:22:15.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127092+0000", 2021-03-15T22:22:15.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0607751919999999 2021-03-15T22:22:15.863 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.864 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.864 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1578 7.3 7:c458c800:::smithi07694878-3:6a [sparse-read 0~3899191] snapc 0=[] ondisk+read+known_if_redirected e204)", 2021-03-15T22:22:15.865 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.066339+0000", 2021-03-15T22:22:15.865 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.552203737999999, 2021-03-15T22:22:15.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0611799630000001, 2021-03-15T22:22:15.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.866 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.867 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1578 2021-03-15T22:22:15.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.867 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066339+0000", 2021-03-15T22:22:15.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066339+0000", 2021-03-15T22:22:15.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.356e-06 2021-03-15T22:22:15.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066340+0000", 2021-03-15T22:22:15.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1090000000000001e-06 2021-03-15T22:22:15.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066341+0000", 2021-03-15T22:22:15.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2199999999999997e-06 2021-03-15T22:22:15.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066347+0000", 2021-03-15T22:22:15.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2209999999999996e-06 2021-03-15T22:22:15.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066352+0000", 2021-03-15T22:22:15.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4710999999999998e-05 2021-03-15T22:22:15.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066407+0000", 2021-03-15T22:22:15.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0290000000000005e-06 2021-03-15T22:22:15.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:22:15.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066415+0000", 2021-03-15T22:22:15.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.060719011 2021-03-15T22:22:15.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127134+0000", 2021-03-15T22:22:15.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013284599999999999 2021-03-15T22:22:15.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127267+0000", 2021-03-15T22:22:15.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025146000000000001 2021-03-15T22:22:15.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127519+0000", 2021-03-15T22:22:15.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0611799630000001 2021-03-15T22:22:15.879 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.879 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.880 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:61 7.5 e205/91 7:a9545d62:::smithi07694878-1:68 v 205'72, mlcod=202'69)", 2021-03-15T22:22:15.880 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.090032+0000", 2021-03-15T22:22:15.881 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.528509901, 2021-03-15T22:22:15.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034649245000000002, 2021-03-15T22:22:15.881 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.881 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:15.882 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090032+0000", 2021-03-15T22:22:15.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090032+0000", 2021-03-15T22:22:15.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.835e-06 2021-03-15T22:22:15.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090034+0000", 2021-03-15T22:22:15.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.75e-06 2021-03-15T22:22:15.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090036+0000", 2021-03-15T22:22:15.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3450000000000002e-06 2021-03-15T22:22:15.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090042+0000", 2021-03-15T22:22:15.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0460000000000002e-06 2021-03-15T22:22:15.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090048+0000", 2021-03-15T22:22:15.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4586999999999999e-05 2021-03-15T22:22:15.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090083+0000", 2021-03-15T22:22:15.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4909000000000001e-05 2021-03-15T22:22:15.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090098+0000", 2021-03-15T22:22:15.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034543644999999998 2021-03-15T22:22:15.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:15.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.124641+0000", 2021-03-15T22:22:15.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0127999999999999e-05 2021-03-15T22:22:15.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.124682+0000", 2021-03-15T22:22:15.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034649245000000002 2021-03-15T22:22:15.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.893 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.894 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1588 7.3 7:c7825e8f:::smithi07694878-26:head [read 0~3109262,checksum] snapc 0=[] ondisk+read+known_if_redirected e206)", 2021-03-15T22:22:15.894 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:01.035526+0000", 2021-03-15T22:22:15.895 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 14.583016377, 2021-03-15T22:22:15.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065659949999999998, 2021-03-15T22:22:15.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.895 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.896 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.896 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.896 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1588 2021-03-15T22:22:15.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.897 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.035526+0000", 2021-03-15T22:22:15.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.035526+0000", 2021-03-15T22:22:15.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.733e-06 2021-03-15T22:22:15.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.035528+0000", 2021-03-15T22:22:15.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.313e-06 2021-03-15T22:22:15.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.035529+0000", 2021-03-15T22:22:15.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6269999999999997e-06 2021-03-15T22:22:15.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.035535+0000", 2021-03-15T22:22:15.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8970000000000001e-06 2021-03-15T22:22:15.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.035538+0000", 2021-03-15T22:22:15.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057300090000000003 2021-03-15T22:22:15.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.041268+0000", 2021-03-15T22:22:15.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104354 2021-03-15T22:22:15.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.041373+0000", 2021-03-15T22:22:15.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071906200000000002 2021-03-15T22:22:15.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.042092+0000", 2021-03-15T22:22:15.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0065659949999999998 2021-03-15T22:22:15.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.907 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.908 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1684 7.8 7:1d74a1c3:::smithi07694878-44:head [copy-from ver 83] snapc 7a=[7a,79,78,76,75] ondisk+write+known_if_redirected e220)", 2021-03-15T22:22:15.908 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:15.039221+0000", 2021-03-15T22:22:15.908 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.57932122200000002, 2021-03-15T22:22:15.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026953068, 2021-03-15T22:22:15.909 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.909 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:15.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.910 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1684 2021-03-15T22:22:15.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.910 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039221+0000", 2021-03-15T22:22:15.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039221+0000", 2021-03-15T22:22:15.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7960000000000001e-06 2021-03-15T22:22:15.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039227+0000", 2021-03-15T22:22:15.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1250000000000001e-06 2021-03-15T22:22:15.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039232+0000", 2021-03-15T22:22:15.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8726999999999999e-05 2021-03-15T22:22:15.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039251+0000", 2021-03-15T22:22:15.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6390000000000004e-06 2021-03-15T22:22:15.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039260+0000", 2021-03-15T22:22:15.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2811000000000005e-05 2021-03-15T22:22:15.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039333+0000", 2021-03-15T22:22:15.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8609000000000004e-05 2021-03-15T22:22:15.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039432+0000", 2021-03-15T22:22:15.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013565288 2021-03-15T22:22:15.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:22:15.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.052997+0000", 2021-03-15T22:22:15.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050078329999999997 2021-03-15T22:22:15.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:22:15.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.058005+0000", 2021-03-15T22:22:15.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080478589999999992 2021-03-15T22:22:15.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:15.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066053+0000", 2021-03-15T22:22:15.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.239e-05 2021-03-15T22:22:15.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:15.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066085+0000", 2021-03-15T22:22:15.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8930999999999997e-05 2021-03-15T22:22:15.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:15.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066124+0000", 2021-03-15T22:22:15.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0059999999999998e-05 2021-03-15T22:22:15.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066174+0000", 2021-03-15T22:22:15.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026953068 2021-03-15T22:22:15.926 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.926 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.926 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.927 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1685 7.8 7:1d74a1c3:::smithi07694878-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e220)", 2021-03-15T22:22:15.927 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:15.039276+0000", 2021-03-15T22:22:15.927 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 0.57926587600000001, 2021-03-15T22:22:15.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027064135999999999, 2021-03-15T22:22:15.928 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:15.928 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:15.928 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:15.928 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:15.929 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:15.929 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1685 2021-03-15T22:22:15.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.929 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:15.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:15.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039276+0000", 2021-03-15T22:22:15.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:15.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:15.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039276+0000", 2021-03-15T22:22:15.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4589999999999999e-06 2021-03-15T22:22:15.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:15.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039278+0000", 2021-03-15T22:22:15.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5710000000000001e-06 2021-03-15T22:22:15.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:15.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039279+0000", 2021-03-15T22:22:15.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2021-03-15T22:22:15.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:15.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039285+0000", 2021-03-15T22:22:15.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.236e-06 2021-03-15T22:22:15.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:15.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039288+0000", 2021-03-15T22:22:15.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028498599999999998 2021-03-15T22:22:15.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039573+0000", 2021-03-15T22:22:15.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8646000000000001e-05 2021-03-15T22:22:15.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:15.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039622+0000", 2021-03-15T22:22:15.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017618592999999998 2021-03-15T22:22:15.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.057240+0000", 2021-03-15T22:22:15.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2138999999999995e-05 2021-03-15T22:22:15.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:15.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.057312+0000", 2021-03-15T22:22:15.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088990739999999999 2021-03-15T22:22:15.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:15.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066211+0000", 2021-03-15T22:22:15.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9696999999999994e-05 2021-03-15T22:22:15.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:15.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066281+0000", 2021-03-15T22:22:15.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9481999999999997e-05 2021-03-15T22:22:15.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:15.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:15.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:15.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066341+0000", 2021-03-15T22:22:15.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027064135999999999 2021-03-15T22:22:15.945 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.945 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.945 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.946 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:15.946 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:15.946 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:22:15.946 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:15.972 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:15.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:15.974 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:16.046 INFO:tasks.rados.rados.0.smithi076.stdout:638: done (2 left) 2021-03-15T22:22:16.046 INFO:tasks.rados.rados.0.smithi076.stdout:639: delete oid 10 current snap is 63 2021-03-15T22:22:16.047 INFO:tasks.rados.rados.0.smithi076.stdout:635: finishing copy_from to smithi07694878-11 2021-03-15T22:22:16.047 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 242 (ObjNum 199 snap 43 seq_num 199) dirty exists 2021-03-15T22:22:16.047 INFO:tasks.rados.rados.0.smithi076.stdout:634: finishing copy_from to smithi07694878-44 2021-03-15T22:22:16.047 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 83 (ObjNum 156 snap 35 seq_num 156) dirty exists 2021-03-15T22:22:16.048 INFO:tasks.rados.rados.0.smithi076.stdout:635: finishing copy_from racing read to smithi07694878-11 2021-03-15T22:22:16.048 INFO:tasks.rados.rados.0.smithi076.stdout:634: finishing copy_from racing read to smithi07694878-44 2021-03-15T22:22:16.052 INFO:tasks.rados.rados.0.smithi076.stdout:634: done (2 left) 2021-03-15T22:22:16.052 INFO:tasks.rados.rados.0.smithi076.stdout:635: done (1 left) 2021-03-15T22:22:16.052 INFO:tasks.rados.rados.0.smithi076.stdout:639: done (0 left) 2021-03-15T22:22:16.053 INFO:tasks.rados.rados.0.smithi076.stdout:640: write oid 48 current snap is 63 2021-03-15T22:22:16.053 INFO:tasks.rados.rados.0.smithi076.stdout:640: seq_num 253 ranges {652171=477646,1821709=430462} 2021-03-15T22:22:16.056 INFO:tasks.rados.rados.0.smithi076.stdout:640: writing smithi07694878-48 from 652171 to 1129817 tid 1 2021-03-15T22:22:16.063 INFO:tasks.rados.rados.0.smithi076.stdout:640: writing smithi07694878-48 from 1821709 to 2252171 tid 2 2021-03-15T22:22:16.063 INFO:tasks.rados.rados.0.smithi076.stdout:641: write oid 19 current snap is 63 2021-03-15T22:22:16.064 INFO:tasks.rados.rados.0.smithi076.stdout:640: finishing write tid 1 to smithi07694878-48 2021-03-15T22:22:16.064 INFO:tasks.rados.rados.0.smithi076.stdout:641: seq_num 254 ranges {668978=552895,1763838=505140} 2021-03-15T22:22:16.070 INFO:tasks.rados.rados.0.smithi076.stdout:641: writing smithi07694878-19 from 668978 to 1221873 tid 1 2021-03-15T22:22:16.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:16.076 INFO:tasks.rados.rados.0.smithi076.stdout:641: writing smithi07694878-19 from 1763838 to 2268978 tid 2 2021-03-15T22:22:16.077 INFO:tasks.rados.rados.0.smithi076.stdout:642: snap_remove snap 61 2021-03-15T22:22:16.098 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:16.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:16.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:16.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:16.226 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:16.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:16.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:16.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:16 smithi125 conmon[56739]: cluster 2021-03-15T22:22:15.007622+0000 mgr.y (mgr.14144) 464 : cluster [DBG] 2021-03-15T22:22:16.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:16 smithi125 conmon[56739]: pgmap v456: 138 pgs: 138 active+clean; 118 MiB data, 432 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-03-15T22:22:16.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:16 smithi125 conmon[56739]: cluster 2021-03-15T22:22:15.030458+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2021-03-15T22:22:16.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:16 smithi125 conmon[56739]: cluster 2021-03-15T22:22:16.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:16 smithi125 conmon[56739]: 2021-03-15T22:22:15.524934+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:16.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:16 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:22:16.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:16 smithi125 conmon[56739]: -15T22:22:15.525151+0000 osd.5 (osd.5) 47 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:16.329 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:16.350 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:16.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:16.352 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:16.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:16.478 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:16.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:16.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:16.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:16 smithi076 conmon[56514]: cluster 2021-03-15T22:22:15.007622+0000 mgr.y (mgr.14144) 464 : cluster [DBG] pgmap v456: 138 pgs: 138 active+clean; 118 MiB data, 432 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-03-15T22:22:16.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:16 smithi076 conmon[56514]: cluster 2021-03-15T22:22:15.030458+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2021-03-15T22:22:16.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:16 smithi076 conmon[56514]: cluster 2021-03-15T22:22:15.524934+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:16.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:16 smithi076 conmon[56514]: cluster 2021-03-15T22:22:15.525151+0000 osd.5 (osd.5) 47 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:16.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:16 smithi076 conmon[62156]: cluster 2021-03-15T22:22:15.007622+0000 mgr.y (mgr.14144) 464 : cluster [DBG] pgmap v456: 138 pgs: 138 active+clean; 118 MiB data, 432 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-03-15T22:22:16.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:16 smithi076 conmon[62156]: cluster 2021-03-15T22:22:15.030458+0000 mon.a (mon.0) 859 : cluster [DBG] osdmap e220: 8 total, 8 up, 7 in 2021-03-15T22:22:16.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:16 smithi076 conmon[62156]: cluster 2021-03-15T22:22:15.524934+0000 osd.5 (osd.5) 46 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:16.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:16 smithi076 conmon[62156]: cluster 2021-03-15T22:22:15.525151+0000 osd.5 (osd.5) 47 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:16.492 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:16.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:22:16.550 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-15T22:22:16.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:16.606 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:16.608 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:16.709 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:16.738 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:16.739 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:16.840 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:16.862 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:16.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:16.864 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:16.965 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:16.991 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:16.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:16.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:17.050 INFO:tasks.rados.rados.0.smithi076.stdout:642: done (2 left) 2021-03-15T22:22:17.050 INFO:tasks.rados.rados.0.smithi076.stdout:643: snap_create 2021-03-15T22:22:17.050 INFO:tasks.rados.rados.0.smithi076.stdout:640: finishing write tid 2 to smithi07694878-48 2021-03-15T22:22:17.051 INFO:tasks.rados.rados.0.smithi076.stdout:640: finishing write tid 3 to smithi07694878-48 2021-03-15T22:22:17.051 INFO:tasks.rados.rados.0.smithi076.stdout:640: finishing write tid 4 to smithi07694878-48 2021-03-15T22:22:17.051 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 125 (ObjNum 253 snap 63 seq_num 253) dirty exists 2021-03-15T22:22:17.051 INFO:tasks.rados.rados.0.smithi076.stdout:640: left oid 48 (ObjNum 253 snap 63 seq_num 253) 2021-03-15T22:22:17.052 INFO:tasks.rados.rados.0.smithi076.stdout:641: finishing write tid 1 to smithi07694878-19 2021-03-15T22:22:17.052 INFO:tasks.rados.rados.0.smithi076.stdout:641: finishing write tid 2 to smithi07694878-19 2021-03-15T22:22:17.052 INFO:tasks.rados.rados.0.smithi076.stdout:641: finishing write tid 3 to smithi07694878-19 2021-03-15T22:22:17.052 INFO:tasks.rados.rados.0.smithi076.stdout:641: finishing write tid 4 to smithi07694878-19 2021-03-15T22:22:17.052 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 252 (ObjNum 254 snap 63 seq_num 254) dirty exists 2021-03-15T22:22:17.053 INFO:tasks.rados.rados.0.smithi076.stdout:641: left oid 19 (ObjNum 254 snap 63 seq_num 254) 2021-03-15T22:22:17.094 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:17.115 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:17.118 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:17.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:17.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:17.220 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:17.243 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:17.246 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:17.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:17.248 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:17.300 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:17 smithi076 conmon[56514]: cluster 2021-03-15T22:22:16.041868+0000 mon.a (mon.0) 860 : cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2021-03-15T22:22:17.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:17 smithi076 conmon[62156]: cluster 2021-03-15T22:22:17.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:17 smithi076 conmon[62156]: 2021-03-15T22:22:17.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:17 smithi076 conmon[62156]: 2021 2021-03-15T22:22:17.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:17 smithi076 conmon[62156]: - 2021-03-15T22:22:17.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:17 smithi076 conmon[62156]: 03 2021-03-15T22:22:17.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:17 smithi076 conmon[62156]: -15T22:22:16.041868+0000 mon.a 2021-03-15T22:22:17.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:17 smithi076 conmon[62156]: (mon.0) 860 : cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2021-03-15T22:22:17.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:17.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:17.375 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:17.376 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:17.477 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:17.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:17.502 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:17.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:17.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:17 smithi125 conmon[56739]: cluster 2021-03-15T22:22:16.041868+0000 mon.a (mon.0) 860 : 2021-03-15T22:22:17.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:17 smithi125 conmon[56739]: cluster [DBG] osdmap e221: 8 total, 8 up, 7 in 2021-03-15T22:22:17.604 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:17.631 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:17.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:17.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:17.734 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:17.760 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:17.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:17.761 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:17.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:17.888 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:17.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:17.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:17.991 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:18.017 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:18.019 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:18.053 INFO:tasks.rados.rados.0.smithi076.stdout:640: done (2 left) 2021-03-15T22:22:18.054 INFO:tasks.rados.rados.0.smithi076.stdout:641: done (1 left) 2021-03-15T22:22:18.054 INFO:tasks.rados.rados.0.smithi076.stdout:643: done (0 left) 2021-03-15T22:22:18.055 INFO:tasks.rados.rados.0.smithi076.stdout:644: read oid 10 snap 58 2021-03-15T22:22:18.055 INFO:tasks.rados.rados.0.smithi076.stdout:644: expect deleted 2021-03-15T22:22:18.055 INFO:tasks.rados.rados.0.smithi076.stdout:645: snap_remove snap 62 2021-03-15T22:22:18.059 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:22:18.119 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:18.145 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:18.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:18.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[56514]: cluster 2021-03-15T22:22:17.008040+0000 2021-03-15T22:22:18.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[56514]: mgr.y (mgr.14144) 465 : cluster [DBG] pgmap v459: 138 pgs: 138 active+clean; 114 MiB data, 420 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2021-03-15T22:22:18.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[56514]: cluster 2021-03-15T22:22:17.047554+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2021-03-15T22:22:18.198 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[56514]: audit 2021-03-15T22:22:17.348451+0000 mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.76:0/2048544192' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:18.199 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[56514]: audit 2021-03-15T22:22:17.348804+0000 mon.a (mon.0) 862 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:18.199 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[62156]: cluster 2021-03-15T22:22:17.008040 2021-03-15T22:22:18.200 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 465 : cluster [DBG] pgmap v459: 138 pgs: 138 active+clean; 114 MiB data, 420 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2021-03-15T22:22:18.200 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:22:18.200 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[62156]: 03-15T22:22:17.047554+0000 mon.a (mon.0) 861 : 2021-03-15T22:22:18.200 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[62156]: cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2021-03-15T22:22:18.201 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:22:18.201 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[62156]: -15T22:22:17.348451+0000 2021-03-15T22:22:18.201 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[62156]: mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.76:0/2048544192' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:18.202 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[62156]: audit 2021-03-15T22:22:17.348804 2021-03-15T22:22:18.202 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[62156]: +0000 mon.a (mon.0) 862 : audit 2021-03-15T22:22:18.202 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:18 smithi076 conmon[62156]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:18.248 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:18.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:18.272 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:18.273 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:18.374 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:18.401 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:18.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:18.504 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:18.530 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:18.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:18.532 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:18.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:18 smithi125 conmon[56739]: cluster 2021-03-15T22:22:17.008040+0000 mgr.y (mgr.14144) 465 : cluster [DBG] pgmap v459: 138 pgs: 138 active+clean; 114 MiB data, 420 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2021-03-15T22:22:18.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:18 smithi125 conmon[56739]: cluster 2021-03-15T22:22:17.047554+0000 mon.a (mon.0) 861 : cluster [DBG] osdmap e222: 8 total, 8 up, 7 in 2021-03-15T22:22:18.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:18 smithi125 conmon[56739]: audit 2021-03-15T22:22:17.348451+0000 mon.c (mon.1) 23 : audit [INF] from='client.? 172.21.15.76:0/2048544192' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:18.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:18 smithi125 conmon[56739]: audit 2021-03-15T22:22:17.348804+0000 mon.a (mon 2021-03-15T22:22:18.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:18 smithi125 conmon[56739]: .0) 862 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:18.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:18.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:18.658 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:18.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:18.760 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:18.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:18.790 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:18.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:18.892 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:18.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:18.921 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:18.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:19.023 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:19.047 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:19.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:19.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:19.064 INFO:tasks.rados.rados.0.smithi076.stdout:645: done (1 left) 2021-03-15T22:22:19.064 INFO:tasks.rados.rados.0.smithi076.stdout:646: delete oid 7 current snap is 64 2021-03-15T22:22:19.067 INFO:tasks.rados.rados.0.smithi076.stdout:644: done (1 left) 2021-03-15T22:22:19.067 INFO:tasks.rados.rados.0.smithi076.stdout:646: done (0 left) 2021-03-15T22:22:19.068 INFO:tasks.rados.rados.0.smithi076.stdout:647: write oid 7 current snap is 64 2021-03-15T22:22:19.068 INFO:tasks.rados.rados.0.smithi076.stdout:647: seq_num 256 ranges {702592=703393,1880802=421790} 2021-03-15T22:22:19.075 INFO:tasks.rados.rados.0.smithi076.stdout:647: writing smithi07694878-7 from 702592 to 1405985 tid 1 2021-03-15T22:22:19.080 INFO:tasks.rados.rados.0.smithi076.stdout:647: writing smithi07694878-7 from 1880802 to 2302592 tid 2 2021-03-15T22:22:19.081 INFO:tasks.rados.rados.0.smithi076.stdout:648: rollback oid 25 current snap is 64 2021-03-15T22:22:19.081 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 59 2021-03-15T22:22:19.081 INFO:tasks.rados.rados.0.smithi076.stdout:647: finishing write tid 1 to smithi07694878-7 2021-03-15T22:22:19.081 INFO:tasks.rados.rados.0.smithi076.stdout:649: read oid 38 snap -1 2021-03-15T22:22:19.082 INFO:tasks.rados.rados.0.smithi076.stdout:649: expect deleted 2021-03-15T22:22:19.082 INFO:tasks.rados.rados.0.smithi076.stdout:650: copy_from oid 21 from oid 2 current snap is 64 2021-03-15T22:22:19.082 INFO:tasks.rados.rados.0.smithi076.stdout:651: copy_from oid 39 from oid 3 current snap is 64 2021-03-15T22:22:19.082 INFO:tasks.rados.rados.0.smithi076.stdout:652: snap_create 2021-03-15T22:22:19.083 INFO:tasks.rados.rados.0.smithi076.stdout:647: finishing write tid 2 to smithi07694878-7 2021-03-15T22:22:19.084 INFO:tasks.rados.rados.0.smithi076.stdout:647: finishing write tid 3 to smithi07694878-7 2021-03-15T22:22:19.084 INFO:tasks.rados.rados.0.smithi076.stdout:647: finishing write tid 4 to smithi07694878-7 2021-03-15T22:22:19.085 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 256 (ObjNum 256 snap 64 seq_num 256) dirty exists 2021-03-15T22:22:19.085 INFO:tasks.rados.rados.0.smithi076.stdout:647: left oid 7 (ObjNum 256 snap 64 seq_num 256) 2021-03-15T22:22:19.086 INFO:tasks.rados.rados.0.smithi076.stdout:650: finishing copy_from to smithi07694878-21 2021-03-15T22:22:19.086 INFO:tasks.rados.rados.0.smithi076.stdout:650: got expected ENOENT (src dne) 2021-03-15T22:22:19.086 INFO:tasks.rados.rados.0.smithi076.stdout:650: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:22:19.087 INFO:tasks.rados.rados.0.smithi076.stdout:648: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:22:19.087 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 0 (ObjNum 261755696 snap 21896 seq_num 261965408) dirty dne 2021-03-15T22:22:19.095 INFO:tasks.rados.rados.0.smithi076.stdout:651: finishing copy_from to smithi07694878-39 2021-03-15T22:22:19.095 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 242 (ObjNum 199 snap 43 seq_num 199) dirty exists 2021-03-15T22:22:19.096 INFO:tasks.rados.rados.0.smithi076.stdout:651: finishing copy_from racing read to smithi07694878-39 2021-03-15T22:22:19.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:19.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:19.183 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:19.183 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:19.259 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:19 smithi076 conmon[56514]: cluster 2021-03-15T22:22:18.044754+0000 mon.a (mon.0) 863 2021-03-15T22:22:19.259 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:19 smithi076 conmon[56514]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:19.260 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:19 smithi076 conmon[56514]: audit 2021-03-15T22:22:18.048680+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:22:19.260 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:19 smithi076 conmon[56514]: cluster 2021-03-15T22:22:18.048728+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2021-03-15T22:22:19.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:19 smithi076 conmon[62156]: cluster 2021-03-15T22:22:18.044754+0000 mon.a (mon 2021-03-15T22:22:19.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:19 smithi076 conmon[62156]: .0) 863 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:19.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:19 smithi076 conmon[62156]: audit 2021-03-15T22:22:18.048680+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:22:19.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:19 smithi076 conmon[62156]: cluster 2021-03-15T22:22:18.048728+0000 mon.a (mon.0) 865 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2021-03-15T22:22:19.285 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:19.315 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:19.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:19.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:19.418 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:19.448 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:19.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:19.533 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:19 smithi125 conmon[56739]: cluster 2021-03-15T22:22:18.044754+0000 mon.a (mon.0) 863 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:19.533 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:19 smithi125 conmon[56739]: audit 2021-03-15T22:22:18.048680+0000 mon.a (mon.0) 864 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:22:19.535 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:19 smithi125 conmon[56739]: cluster 2021-03-15T22:22:18.048728+0000 2021-03-15T22:22:19.535 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:19 smithi125 conmon[56739]: mon.a (mon.0) 865 : cluster [DBG] osdmap e223: 8 total, 8 up, 7 in 2021-03-15T22:22:19.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:19.585 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:19.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:19.586 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:19.687 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:19.715 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:19.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:19.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:19.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:22:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:22:19.818 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:19.844 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:19.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:19.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:19.947 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:19.969 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:19.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:19.971 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:20.041 INFO:tasks.rados.rados.0.smithi076.stdout:647: done (5 left) 2021-03-15T22:22:20.041 INFO:tasks.rados.rados.0.smithi076.stdout:648: done (4 left) 2021-03-15T22:22:20.041 INFO:tasks.rados.rados.0.smithi076.stdout:649: done (3 left) 2021-03-15T22:22:20.041 INFO:tasks.rados.rados.0.smithi076.stdout:650: done (2 left) 2021-03-15T22:22:20.042 INFO:tasks.rados.rados.0.smithi076.stdout:651: done (1 left) 2021-03-15T22:22:20.042 INFO:tasks.rados.rados.0.smithi076.stdout:652: done (0 left) 2021-03-15T22:22:20.042 INFO:tasks.rados.rados.0.smithi076.stdout:653: copy_from oid 1 from oid 42 current snap is 65 2021-03-15T22:22:20.042 INFO:tasks.rados.rados.0.smithi076.stdout:654: read oid 44 snap -1 2021-03-15T22:22:20.043 INFO:tasks.rados.rados.0.smithi076.stdout:654: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:22:20.043 INFO:tasks.rados.rados.0.smithi076.stdout:655: snap_create 2021-03-15T22:22:20.046 INFO:tasks.rados.rados.0.smithi076.stdout:653: finishing copy_from to smithi07694878-1 2021-03-15T22:22:20.046 INFO:tasks.rados.rados.0.smithi076.stdout:653: got expected ENOENT (src dne) 2021-03-15T22:22:20.046 INFO:tasks.rados.rados.0.smithi076.stdout:653: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:22:20.048 INFO:tasks.rados.rados.0.smithi076.stdout:654: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:22:20.072 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:20.096 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:20.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:20.097 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:20.199 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:20.219 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:22:20.219 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:22:20.219 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:22:20.220 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:22:20.223 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:20.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:20.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:20.326 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:20.348 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:20.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:20.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:20.451 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:20.480 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:20.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:20.482 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:20.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[56514]: cluster 2021-03-15T22:22:19.008440+0000 mgr.y (mgr.14144) 466 : cluster [DBG] pgmap v462: 138 pgs: 138 active+clean; 115 MiB data, 424 MiB used, 625 GiB / 626 GiB avail; 3.8 MiB/s rd, 1.0 MiB/s wr, 4 op/s 2021-03-15T22:22:20.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[56514]: cluster 2021-03-15T22:22:19.061085+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e224: 8 total, 8 up, 7 in 2021-03-15T22:22:20.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[56514]: audit 2021-03-15T22:22:19.531321+0000 mgr.y (mgr.14144) 467 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:20.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[56514]: cluster 2021-03-15T22:22:20.029396+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2021-03-15T22:22:20.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: cluster 2021-03-15T22:22:20.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021 2021-03-15T22:22:20.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: - 2021-03-15T22:22:20.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 03 2021-03-15T22:22:20.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: - 2021-03-15T22:22:20.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 15 2021-03-15T22:22:20.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: T 2021-03-15T22:22:20.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 22 2021-03-15T22:22:20.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: : 2021-03-15T22:22:20.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 22 2021-03-15T22:22:20.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: : 2021-03-15T22:22:20.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 19 2021-03-15T22:22:20.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: . 2021-03-15T22:22:20.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 008440 2021-03-15T22:22:20.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: +0000 2021-03-15T22:22:20.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: mgr.y 2021-03-15T22:22:20.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: ( 2021-03-15T22:22:20.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: mgr 2021-03-15T22:22:20.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: . 2021-03-15T22:22:20.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 14144 2021-03-15T22:22:20.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: ) 2021-03-15T22:22:20.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 466 2021-03-15T22:22:20.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: : 2021-03-15T22:22:20.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: cluster 2021-03-15T22:22:20.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: [DBG] 2021-03-15T22:22:20.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: pgmap v462: 138 pgs: 138 active+clean; 115 MiB data, 424 MiB used, 625 GiB / 626 GiB avail; 3.8 MiB/s rd, 1.0 MiB/s wr, 4 op/s 2021-03-15T22:22:20.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: cluster 2021-03-15T22:22:20.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021 2021-03-15T22:22:20.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: - 2021-03-15T22:22:20.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 03 2021-03-15T22:22:20.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: - 2021-03-15T22:22:20.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 15 2021-03-15T22:22:20.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: T 2021-03-15T22:22:20.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 22 2021-03-15T22:22:20.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: : 2021-03-15T22:22:20.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 22 2021-03-15T22:22:20.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: : 2021-03-15T22:22:20.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 19 2021-03-15T22:22:20.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: . 2021-03-15T22:22:20.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 061085 2021-03-15T22:22:20.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: +0000 2021-03-15T22:22:20.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: mon.a 2021-03-15T22:22:20.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: ( 2021-03-15T22:22:20.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: mon 2021-03-15T22:22:20.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: . 2021-03-15T22:22:20.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 0 2021-03-15T22:22:20.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: ) 2021-03-15T22:22:20.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 866 2021-03-15T22:22:20.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: : 2021-03-15T22:22:20.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: cluster 2021-03-15T22:22:20.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: [DBG] 2021-03-15T22:22:20.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: osdmap e224: 8 total, 8 up, 7 in 2021-03-15T22:22:20.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: audit 2021-03-15T22:22:20.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021 2021-03-15T22:22:20.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: - 2021-03-15T22:22:20.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 03 2021-03-15T22:22:20.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: - 2021-03-15T22:22:20.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 15 2021-03-15T22:22:20.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: T 2021-03-15T22:22:20.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 22 2021-03-15T22:22:20.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: : 2021-03-15T22:22:20.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 22 2021-03-15T22:22:20.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: : 2021-03-15T22:22:20.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 19 2021-03-15T22:22:20.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: . 2021-03-15T22:22:20.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 531321 2021-03-15T22:22:20.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: +0000 2021-03-15T22:22:20.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 2021-03-15T22:22:20.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: mgr.y 2021-03-15T22:22:20.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: ( 2021-03-15T22:22:20.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: mgr 2021-03-15T22:22:20.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: . 2021-03-15T22:22:20.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 14144 2021-03-15T22:22:20.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: ) 2021-03-15T22:22:20.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: 467 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:20.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: cluster 2021-03-15T22:22:20.029396+0000 mon.a ( 2021-03-15T22:22:20.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:20 smithi076 conmon[62156]: mon.0) 867 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2021-03-15T22:22:20.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:20 smithi125 conmon[56739]: cluster 2021-03-15T22:22:19.008440+0000 mgr.y (mgr.14144) 466 : cluster [DBG] 2021-03-15T22:22:20.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:20 smithi125 conmon[56739]: pgmap v462: 138 pgs: 138 active+clean; 115 MiB data, 424 MiB used, 625 GiB / 626 GiB avail; 3.8 MiB/s rd, 1.0 MiB/s wr, 4 op/s 2021-03-15T22:22:20.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:20 smithi125 conmon[56739]: cluster 2021-03-15T22:22:19.061085+0000 mon.a (mon.0) 866 : cluster [DBG] osdmap e224: 8 total, 8 up, 7 in 2021-03-15T22:22:20.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:20 smithi125 conmon[56739]: audit 2021-03-15T22:22:19.531321+0000 mgr.y (mgr.14144) 467 : audit 2021-03-15T22:22:20.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:20 smithi125 conmon[56739]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:20.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:20 smithi125 conmon[56739]: cluster 2021-03-15T22:22:20 2021-03-15T22:22:20.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:20 smithi125 conmon[56739]: .029396+0000 mon.a (mon.0) 867 : cluster [DBG] osdmap e225: 8 total, 8 up, 7 in 2021-03-15T22:22:20.583 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:20.606 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:20.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:20.607 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:20.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:20.735 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:20.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:20.838 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:20.865 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:20.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:20.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:20.989 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:20.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:21.038 INFO:tasks.rados.rados.0.smithi076.stdout:653: done (2 left) 2021-03-15T22:22:21.039 INFO:tasks.rados.rados.0.smithi076.stdout:654: done (1 left) 2021-03-15T22:22:21.039 INFO:tasks.rados.rados.0.smithi076.stdout:655: done (0 left) 2021-03-15T22:22:21.039 INFO:tasks.rados.rados.0.smithi076.stdout:656: snap_create 2021-03-15T22:22:21.041 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.2 to 1 (10000) 2021-03-15T22:22:21.067 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-15T22:22:21.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:21.118 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:21.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:21.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:21.221 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:21.222 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:22:21.240 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:22:21.244 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:21.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:21.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:21.259 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:22:21.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:21 smithi076 conmon[56514]: audit 2021-03-15T22:22:20.373752+0000 mon.a (mon.0) 868 : 2021-03-15T22:22:21.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:21 smithi076 conmon[56514]: audit [INF] from='client.? 172.21.15.76:0/3837086209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-15T22:22:21.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:21 smithi076 conmon[56514]: audit 2021-03-15T22:22:21.036863+0000 mon.a (mon.0) 869 : audit [INF] from='client.? 172.21.15.76:0/3837086209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-15T22:22:21.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:21 smithi076 conmon[56514]: cluster 2021-03-15T22:22:21.036927+0000 mon.a (mon.0) 870 : cluster [DBG] osdmap e226: 8 total, 8 up, 7 in 2021-03-15T22:22:21.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:21 smithi076 conmon[62156]: audit 2021-03-15T22:22:20. 2021-03-15T22:22:21.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:21 smithi076 conmon[62156]: 373752+0000 mon.a (mon.0) 868 : audit [INF] from='client.? 172.21.15.76:0/3837086209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-15T22:22:21.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:21 smithi076 conmon[62156]: audit 2021-03-15T22:22:21.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:21 smithi076 conmon[62156]: 2021-03-15T22:22:21.036863+0000 mon.a (mon.0) 869 : audit [INF] from='client.? 172.21.15.76:0/3837086209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-15T22:22:21.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:21 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:22:21.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:21 smithi076 conmon[62156]: -03-15T22:22:21.036927+0000 mon.a 2021-03-15T22:22:21.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:21 smithi076 conmon[62156]: (mon.0) 870 : cluster [DBG] osdmap e226: 8 total, 8 up, 7 in 2021-03-15T22:22:21.277 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:22:21.296 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:22:21.316 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:22:21.334 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:22:21.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:21.353 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:22:21.355 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:22:21.356 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:22:21.356 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:22:21.356 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:22:21.356 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:22:21.356 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:22:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:22:21.357 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:22:21.370 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:21.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:21.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:21.473 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:21.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:21.497 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:21.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:21.532 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:22:21.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:21 smithi125 conmon[56739]: audit 2021-03-15T22:22:20.373752+0000 mon.a (mon.0) 868 : audit [INF] from='client.? 172.21.15.76:0/3837086209' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-15T22:22:21.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:21 smithi125 conmon[56739]: 2021-03-15T22:22:21.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:21 smithi125 conmon[56739]: audit 2021-03-15T22:22:21.036863+0000 mon.a (mon.0) 869 : audit [INF] from='client.? 172.21.15.76:0/3837086209' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-15T22:22:21.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:21 smithi125 conmon[56739]: cluster 2021-03-15T22:22:21.036927+0000 mon.a (mon.0) 870 : cluster 2021-03-15T22:22:21.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:21 smithi125 conmon[56739]: [DBG] osdmap e226: 8 total, 8 up, 7 in 2021-03-15T22:22:21.598 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:21.625 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:21.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:21.626 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:21.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:21.754 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:21.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:21.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:21.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:21.880 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:21.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:21.881 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:21.883 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:21.911 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 4 1 2021-03-15T22:22:21.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:22.008 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:22.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:22.010 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:22.040 INFO:tasks.rados.rados.0.smithi076.stdout:656: done (0 left) 2021-03-15T22:22:22.041 INFO:tasks.rados.rados.0.smithi076.stdout:657: read oid 27 snap -1 2021-03-15T22:22:22.041 INFO:tasks.rados.rados.0.smithi076.stdout:657: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:22:22.041 INFO:tasks.rados.rados.0.smithi076.stdout:658: write oid 49 current snap is 67 2021-03-15T22:22:22.041 INFO:tasks.rados.rados.0.smithi076.stdout:658: seq_num 257 ranges {719399=778642,1939284=380115} 2021-03-15T22:22:22.048 INFO:tasks.rados.rados.0.smithi076.stdout:658: writing smithi07694878-49 from 719399 to 1498041 tid 1 2021-03-15T22:22:22.053 INFO:tasks.rados.rados.0.smithi076.stdout:658: writing smithi07694878-49 from 1939284 to 2319399 tid 2 2021-03-15T22:22:22.054 INFO:tasks.rados.rados.0.smithi076.stdout:659: delete oid 10 current snap is 67 2021-03-15T22:22:22.054 INFO:tasks.rados.rados.0.smithi076.stdout:657: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:22:22.110 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:22.132 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:22.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:22.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:22.223 INFO:tasks.rados.rados.0.smithi076.stdout:658: finishing write tid 1 to smithi07694878-49 2021-03-15T22:22:22.224 INFO:tasks.rados.rados.0.smithi076.stdout:658: finishing write tid 2 to smithi07694878-49 2021-03-15T22:22:22.224 INFO:tasks.rados.rados.0.smithi076.stdout:658: finishing write tid 3 to smithi07694878-49 2021-03-15T22:22:22.225 INFO:tasks.rados.rados.0.smithi076.stdout:658: finishing write tid 4 to smithi07694878-49 2021-03-15T22:22:22.225 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 311 (ObjNum 257 snap 67 seq_num 257) dirty exists 2021-03-15T22:22:22.226 INFO:tasks.rados.rados.0.smithi076.stdout:658: left oid 49 (ObjNum 257 snap 67 seq_num 257) 2021-03-15T22:22:22.227 INFO:tasks.rados.rados.0.smithi076.stdout:657: done (2 left) 2021-03-15T22:22:22.227 INFO:tasks.rados.rados.0.smithi076.stdout:658: done (1 left) 2021-03-15T22:22:22.228 INFO:tasks.rados.rados.0.smithi076.stdout:659: done (0 left) 2021-03-15T22:22:22.228 INFO:tasks.rados.rados.0.smithi076.stdout:660: rollback oid 17 current snap is 67 2021-03-15T22:22:22.228 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 17 to 66 2021-03-15T22:22:22.228 INFO:tasks.rados.rados.0.smithi076.stdout:661: snap_remove snap 64 2021-03-15T22:22:22.234 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:22.256 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:22.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:22.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:22.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:22 smithi076 conmon[56514]: cluster 2021-03-15T22:22:21.008824+0000 mgr.y (mgr.14144) 468 : cluster [DBG] pgmap v465: 138 pgs: 138 active+clean; 124 MiB data, 441 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.4 MiB/s wr, 7 op/s 2021-03-15T22:22:22.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:22 smithi076 conmon[56514]: audit 2021-03-15T22:22:21.202877+0000 mon.a (mon.0) 871 : audit [DBG] from='client.? 172.21.15.76:0/1652183966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:22.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:22 smithi076 conmon[56514]: cluster 2021-03-15T22:22:22.038293+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e227: 8 total, 8 up, 7 in 2021-03-15T22:22:22.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:22 smithi076 conmon[62156]: cluster 2021-03-15T22:22:21.008824+0000 mgr.y (mgr.14144) 468 : cluster [DBG] pgmap v465: 138 pgs: 138 active+clean; 124 MiB data, 441 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.4 MiB/s wr, 7 op/s 2021-03-15T22:22:22.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:22 smithi076 conmon[62156]: audit 2021-03-15T22:22:21.202877+0000 mon.a (mon.0) 871 : audit [DBG] from='client.? 172.21.15.76:0/1652183966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:22.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:22 smithi076 conmon[62156]: cluster 2021-03-15T22:22:22.038293+0000 mon.a (mon.0) 872 : cluster [DBG] osdmap e227: 8 total, 8 up, 7 in 2021-03-15T22:22:22.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:22.380 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:22.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:22.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:22.483 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:22.510 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:22.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:22.511 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:22 smithi125 conmon[56739]: cluster 2021-03-15T22:22:21.008824+0000 mgr.y (mgr.14144) 468 : cluster [DBG] pgmap v465: 138 pgs: 138 active+clean; 124 MiB data, 441 MiB used, 625 GiB / 626 GiB avail; 2.2 KiB/s rd, 2.4 MiB/s wr, 7 op/s 2021-03-15T22:22:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:22 smithi125 conmon[56739]: audit 2021-03-15T22:22:21.202877+0000 mon.a (mon.0) 871 : audit [DBG] from='client.? 172.21.15.76:0/1652183966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:22 smithi125 conmon[56739]: cluster 2021-03-15T22:22:22.038293+0000 mon.a (mon.0) 872 : cluster 2021-03-15T22:22:22.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:22 smithi125 conmon[56739]: [DBG] osdmap e227: 8 total, 8 up, 7 in 2021-03-15T22:22:22.565 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:22.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:22.615 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:22.638 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:22.640 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:22.741 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:22.764 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:22.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:22.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:22.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:22.892 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:22.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:22.995 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:23.018 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:23.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:23.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:23.080 INFO:tasks.rados.rados.0.smithi076.stdout:661: done (1 left) 2021-03-15T22:22:23.080 INFO:tasks.rados.rados.0.smithi076.stdout:662: read oid 13 snap -1 2021-03-15T22:22:23.081 INFO:tasks.rados.rados.0.smithi076.stdout:662: expect (ObjNum 228 snap 52 seq_num 228) 2021-03-15T22:22:23.081 INFO:tasks.rados.rados.0.smithi076.stdout:663: snap_remove snap 59 2021-03-15T22:22:23.121 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:23.147 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:23.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:23.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:23.249 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:23.271 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:23.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:23.273 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:23.374 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:23.400 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:23.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:23.401 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:23.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:23.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:23.525 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:23.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:23.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:23.649 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:23.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:23.650 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:23.751 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:23.778 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:23.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:23.779 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:23.880 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:23.902 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:23.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:23.904 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:24.005 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:24.027 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:24.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:24.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:24.087 INFO:tasks.rados.rados.0.smithi076.stdout:663: done (2 left) 2021-03-15T22:22:24.088 INFO:tasks.rados.rados.0.smithi076.stdout:664: read oid 27 snap -1 2021-03-15T22:22:24.088 INFO:tasks.rados.rados.0.smithi076.stdout:664: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:22:24.088 INFO:tasks.rados.rados.0.smithi076.stdout:665: copy_from oid 26 from oid 8 current snap is 67 2021-03-15T22:22:24.089 INFO:tasks.rados.rados.0.smithi076.stdout:660: finishing rollback tid 0 to smithi07694878-17 2021-03-15T22:22:24.089 INFO:tasks.rados.rados.0.smithi076.stdout:660: finishing rollback tid 1 to smithi07694878-17 2021-03-15T22:22:24.089 INFO:tasks.rados.rados.0.smithi076.stdout:660: finishing rollback tid 2 to smithi07694878-17 2021-03-15T22:22:24.089 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 60 (ObjNum 220 snap 50 seq_num 220) dirty exists 2021-03-15T22:22:24.089 INFO:tasks.rados.rados.0.smithi076.stdout:662: expect (ObjNum 228 snap 52 seq_num 228) 2021-03-15T22:22:24.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:24.154 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:24.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:24.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:24.257 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:24.278 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:24.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:24.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:24.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:24.389 INFO:tasks.rados.rados.0.smithi076.stdout:664: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:22:24.406 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:24.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:24.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:24.431 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:22:24.456 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:24 smithi076 conmon[56514]: cluster 2021-03-15T22:22:23.009275+0000 mgr.y (mgr. 2021-03-15T22:22:24.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:24 smithi076 conmon[56514]: 14144) 469 : cluster [DBG] pgmap v468: 138 pgs: 138 active+clean; 124 MiB data, 451 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2021-03-15T22:22:24.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:24 smithi076 conmon[56514]: cluster 2021-03-15T22:22:23.077429+0000 mon.a (mon.0) 873 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2021-03-15T22:22:24.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:24 smithi076 conmon[62156]: cluster 2021-03-15T22:22: 2021-03-15T22:22:24.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:24 smithi076 conmon[62156]: 23.009275+0000 mgr.y (mgr.14144) 469 : cluster [DBG] pgmap v468: 138 pgs: 138 active+clean; 124 MiB data, 451 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2021-03-15T22:22:24.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:24 smithi076 conmon[62156]: cluster 2021-03-15T22:22:23.077429+0000 mon.a (mon.0) 873 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2021-03-15T22:22:24.525 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:24.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:24 smithi125 conmon[56739]: cluster 2021-03-15T22:22:23.009275+0000 mgr.y (mgr.14144) 469 : cluster [DBG] 2021-03-15T22:22:24.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:24 smithi125 conmon[56739]: pgmap v468: 138 pgs: 138 active+clean; 124 MiB data, 451 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 2.2 MiB/s wr, 6 op/s 2021-03-15T22:22:24.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:24 smithi125 conmon[56739]: cluster 2021-03-15T22:22:23.077429+0000 mon.a (mon.0) 873 : cluster [DBG] osdmap e228: 8 total, 8 up, 7 in 2021-03-15T22:22:24.548 INFO:tasks.rados.rados.0.smithi076.stdout:665: finishing copy_from to smithi07694878-26 2021-03-15T22:22:24.548 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 119 (ObjNum 197 snap 43 seq_num 197) dirty exists 2021-03-15T22:22:24.549 INFO:tasks.rados.rados.0.smithi076.stdout:665: finishing copy_from racing read to smithi07694878-26 2021-03-15T22:22:24.549 INFO:tasks.rados.rados.0.smithi076.stdout:660: done (3 left) 2021-03-15T22:22:24.549 INFO:tasks.rados.rados.0.smithi076.stdout:662: done (2 left) 2021-03-15T22:22:24.549 INFO:tasks.rados.rados.0.smithi076.stdout:664: done (1 left) 2021-03-15T22:22:24.550 INFO:tasks.rados.rados.0.smithi076.stdout:665: done (0 left) 2021-03-15T22:22:24.550 INFO:tasks.rados.rados.0.smithi076.stdout:666: copy_from oid 19 from oid 45 current snap is 67 2021-03-15T22:22:24.550 INFO:tasks.rados.rados.0.smithi076.stdout:667: copy_from oid 9 from oid 34 current snap is 67 2021-03-15T22:22:24.550 INFO:tasks.rados.rados.0.smithi076.stdout:668: write (excl) oid 47 current snap is 67 2021-03-15T22:22:24.551 INFO:tasks.rados.rados.0.smithi076.stdout:668: seq_num 259 ranges {753013=645493,2056248=296765} 2021-03-15T22:22:24.551 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:24.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:24.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:24.556 INFO:tasks.rados.rados.0.smithi076.stdout:668: writing smithi07694878-47 from 753013 to 1398506 tid 1 2021-03-15T22:22:24.561 INFO:tasks.rados.rados.0.smithi076.stdout:668: writing smithi07694878-47 from 2056248 to 2353013 tid 2 2021-03-15T22:22:24.562 INFO:tasks.rados.rados.0.smithi076.stdout:669: snap_create 2021-03-15T22:22:24.562 INFO:tasks.rados.rados.0.smithi076.stdout:667: finishing copy_from to smithi07694878-9 2021-03-15T22:22:24.562 INFO:tasks.rados.rados.0.smithi076.stdout:667: got expected ENOENT (src dne) 2021-03-15T22:22:24.562 INFO:tasks.rados.rados.0.smithi076.stdout:667: finishing copy_from racing read to smithi07694878-9 2021-03-15T22:22:24.563 INFO:tasks.rados.rados.0.smithi076.stdout:666: finishing copy_from to smithi07694878-19 2021-03-15T22:22:24.563 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 245 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:22:24.563 INFO:tasks.rados.rados.0.smithi076.stdout:666: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:22:24.563 INFO:tasks.rados.rados.0.smithi076.stdout:668: finishing write tid 1 to smithi07694878-47 2021-03-15T22:22:24.564 INFO:tasks.rados.rados.0.smithi076.stdout:668: finishing write tid 2 to smithi07694878-47 2021-03-15T22:22:24.565 INFO:tasks.rados.rados.0.smithi076.stdout:668: finishing write tid 3 to smithi07694878-47 2021-03-15T22:22:24.565 INFO:tasks.rados.rados.0.smithi076.stdout:668: finishing write tid 4 to smithi07694878-47 2021-03-15T22:22:24.565 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 128 (ObjNum 259 snap 67 seq_num 259) dirty exists 2021-03-15T22:22:24.565 INFO:tasks.rados.rados.0.smithi076.stdout:668: left oid 47 (ObjNum 259 snap 67 seq_num 259) 2021-03-15T22:22:24.654 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:24.677 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:24.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:24.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:24.780 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:24.806 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:24.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:24.807 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:24.908 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:24.930 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:24.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:24.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:24.976 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:25.034 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:25.036 INFO:tasks.rados.rados.0.smithi076.stdout:666: done (3 left) 2021-03-15T22:22:25.036 INFO:tasks.rados.rados.0.smithi076.stdout:667: done (2 left) 2021-03-15T22:22:25.036 INFO:tasks.rados.rados.0.smithi076.stdout:668: done (1 left) 2021-03-15T22:22:25.037 INFO:tasks.rados.rados.0.smithi076.stdout:669: done (0 left) 2021-03-15T22:22:25.037 INFO:tasks.rados.rados.0.smithi076.stdout:670: write (excl) oid 24 current snap is 68 2021-03-15T22:22:25.038 INFO:tasks.rados.rados.0.smithi076.stdout:670: seq_num 260 ranges {769820=720742,2114730=255090} 2021-03-15T22:22:25.042 INFO:tasks.rados.rados.0.smithi076.stdout:670: writing smithi07694878-24 from 769820 to 1490562 tid 1 2021-03-15T22:22:25.051 INFO:tasks.rados.rados.0.smithi076.stdout:670: writing smithi07694878-24 from 2114730 to 2369820 tid 2 2021-03-15T22:22:25.052 INFO:tasks.rados.rados.0.smithi076.stdout:671: snap_remove snap 58 2021-03-15T22:22:25.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:25.065 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:25.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:25.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:25.192 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:25.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:25.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:25.203 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:25 smithi076 conmon[56514]: cluster 2021-03-15T22:22:24.085137+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2021-03-15T22:22:25.204 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:25 smithi076 conmon[56514]: cluster 2021-03-15T22:22:25.032580+0000 mon.a (mon.0) 875 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2021-03-15T22:22:25.204 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:25 smithi076 conmon[62156]: cluster 2021-03-15T22:22:25.205 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:25 smithi076 conmon[62156]: 2021-03-15T22:22:25.205 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:25 smithi076 conmon[62156]: 2021-03-15T22:22:24. 2021-03-15T22:22:25.206 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:25 smithi076 conmon[62156]: 085137+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2021-03-15T22:22:25.206 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:25 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:22:25.206 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:25 smithi076 conmon[62156]: 03-15T22:22:25.032580+0000 mon.a 2021-03-15T22:22:25.207 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:25 smithi076 conmon[62156]: (mon.0) 875 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2021-03-15T22:22:25.295 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:25.318 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:25.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:25.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:25.339 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:22:25.339 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:22:25.422 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:25.449 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:25.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:25.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:25.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:25 smithi125 conmon[56739]: cluster 2021-03-15T22:22:24.085137+0000 mon.a (mon.0) 874 : cluster [DBG] osdmap e229: 8 total, 8 up, 7 in 2021-03-15T22:22:25.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:25 smithi125 conmon[56739]: cluster 2021-03-15T22:22:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:25 smithi125 conmon[56739]: 2021-03-15T22:22:25.032580+0000 mon.a (mon.0) 875 : cluster [DBG] osdmap e230: 8 total, 8 up, 7 in 2021-03-15T22:22:25.552 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:25.575 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:25.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:25.577 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:25.678 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:25.699 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:25.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:25.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:25.802 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:25.824 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:25.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:25.825 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:25.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:25.949 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:25.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:25.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:26.043 INFO:tasks.rados.rados.0.smithi076.stdout:671: done (1 left) 2021-03-15T22:22:26.044 INFO:tasks.rados.rados.0.smithi076.stdout:672: write oid 29 current snap is 68 2021-03-15T22:22:26.044 INFO:tasks.rados.rados.0.smithi076.stdout:672: seq_num 261 ranges {786627=795991,2173212=213415} 2021-03-15T22:22:26.046 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.4 to 1 (10000) 2021-03-15T22:22:26.051 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:26.052 INFO:tasks.rados.rados.0.smithi076.stdout:672: writing smithi07694878-29 from 786627 to 1582618 tid 1 2021-03-15T22:22:26.056 INFO:tasks.rados.rados.0.smithi076.stdout:672: writing smithi07694878-29 from 2173212 to 2386627 tid 2 2021-03-15T22:22:26.057 INFO:tasks.rados.rados.0.smithi076.stdout:673: delete oid 16 current snap is 68 2021-03-15T22:22:26.057 INFO:tasks.rados.rados.0.smithi076.stdout:670: finishing write tid 1 to smithi07694878-24 2021-03-15T22:22:26.057 INFO:tasks.rados.rados.0.smithi076.stdout:670: finishing write tid 2 to smithi07694878-24 2021-03-15T22:22:26.057 INFO:tasks.rados.rados.0.smithi076.stdout:670: finishing write tid 3 to smithi07694878-24 2021-03-15T22:22:26.058 INFO:tasks.rados.rados.0.smithi076.stdout:670: finishing write tid 4 to smithi07694878-24 2021-03-15T22:22:26.058 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 52 (ObjNum 260 snap 68 seq_num 260) dirty exists 2021-03-15T22:22:26.058 INFO:tasks.rados.rados.0.smithi076.stdout:670: left oid 24 (ObjNum 260 snap 68 seq_num 260) 2021-03-15T22:22:26.059 INFO:tasks.rados.rados.0.smithi076.stdout:672: finishing write tid 1 to smithi07694878-29 2021-03-15T22:22:26.059 INFO:tasks.rados.rados.0.smithi076.stdout:670: done (2 left) 2021-03-15T22:22:26.059 INFO:tasks.rados.rados.0.smithi076.stdout:673: done (1 left) 2021-03-15T22:22:26.060 INFO:tasks.rados.rados.0.smithi076.stdout:674: rollback oid 32 current snap is 68 2021-03-15T22:22:26.060 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 32 to 66 2021-03-15T22:22:26.060 INFO:tasks.rados.rados.0.smithi076.stdout:675: write oid 21 current snap is 68 2021-03-15T22:22:26.060 INFO:tasks.rados.rados.0.smithi076.stdout:675: seq_num 263 ranges {420241=546489,1373823=544414} 2021-03-15T22:22:26.064 INFO:tasks.rados.rados.0.smithi076.stdout:675: writing smithi07694878-21 from 420241 to 966730 tid 1 2021-03-15T22:22:26.070 INFO:tasks.rados.rados.0.smithi076.stdout:675: writing smithi07694878-21 from 1373823 to 1918237 tid 2 2021-03-15T22:22:26.070 INFO:tasks.rados.rados.0.smithi076.stdout:676: snap_create 2021-03-15T22:22:26.070 INFO:tasks.rados.rados.0.smithi076.stdout:672: finishing write tid 2 to smithi07694878-29 2021-03-15T22:22:26.071 INFO:tasks.rados.rados.0.smithi076.stdout:672: finishing write tid 3 to smithi07694878-29 2021-03-15T22:22:26.071 INFO:tasks.rados.rados.0.smithi076.stdout:672: finishing write tid 4 to smithi07694878-29 2021-03-15T22:22:26.071 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 390 (ObjNum 261 snap 68 seq_num 261) dirty exists 2021-03-15T22:22:26.071 INFO:tasks.rados.rados.0.smithi076.stdout:672: left oid 29 (ObjNum 261 snap 68 seq_num 261) 2021-03-15T22:22:26.071 INFO:tasks.rados.rados.0.smithi076.stdout:674: finishing rollback tid 0 to smithi07694878-32 2021-03-15T22:22:26.072 INFO:tasks.rados.rados.0.smithi076.stdout:674: finishing rollback tid 1 to smithi07694878-32 2021-03-15T22:22:26.072 INFO:tasks.rados.rados.0.smithi076.stdout:674: finishing rollback tid 2 to smithi07694878-32 2021-03-15T22:22:26.072 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 394 (ObjNum 108 snap 23 seq_num 108) dirty exists 2021-03-15T22:22:26.072 INFO:tasks.rados.rados.0.smithi076.stdout:675: finishing write tid 1 to smithi07694878-21 2021-03-15T22:22:26.073 INFO:tasks.rados.rados.0.smithi076.stdout:675: finishing write tid 2 to smithi07694878-21 2021-03-15T22:22:26.074 INFO:tasks.rados.rados.0.smithi076.stdout:675: finishing write tid 3 to smithi07694878-21 2021-03-15T22:22:26.074 INFO:tasks.rados.rados.0.smithi076.stdout:675: finishing write tid 4 to smithi07694878-21 2021-03-15T22:22:26.074 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 398 (ObjNum 263 snap 68 seq_num 263) dirty exists 2021-03-15T22:22:26.074 INFO:tasks.rados.rados.0.smithi076.stdout:675: left oid 21 (ObjNum 263 snap 68 seq_num 263) 2021-03-15T22:22:26.078 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:26.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:26.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:26.181 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:26.203 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:26.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:26.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:26.306 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:26.328 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:26.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:26.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:26.431 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:26.453 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:26.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:26.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:26.517 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[65511]: debug 2021-03-15T22:22:26.518 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[65511]: 2021-03-15T22:22:26.048+0000 7f1a1f877700 -1 osd.0 pg_epoch: 231 pg[7.6( v 231'319 (0'0,231'319] local-lis/les=175/176 n=7 ec=91/91 lis/c=175/175 les/c/f=176/176/0 sis=175) [0,7,5] r=0 lpr=175 luod=229'317 crt=231'319 lcod 229'315 mlcod 229'315 active+clean+snaptrim trimq=[75~1] ps=[76~1,7f~1]] removing snap head 2021-03-15T22:22:26.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: cluster 2021-03-15T22:22:24.667801 2021-03-15T22:22:26.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: +0000 osd.2 (osd.2) 41 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:26.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: cluster 2021-03-15T22:22:24.667966+0000 osd.2 (osd.2) 42 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:26.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: cluster 2021-03-15T22:22:25.009758+0000 mgr.y (mgr.14144) 470 : cluster [DBG] 2021-03-15T22:22:26.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: pgmap v471: 138 pgs: 138 active+clean; 130 MiB data, 459 MiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-03-15T22:22:26.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: audit 2021-03-15T22:22:26.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: 2021-03-15T22:22:25.735006+0000 2021-03-15T22:22:26.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: mon.a (mon.0) 876 : audit 2021-03-15T22:22:26.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: [INF] from='client.? 172.21.15.76:0/3053777089' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:22:26.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: audit 2021-03-15T22:22:26.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: 2021-03-15T22:22:26.038661 2021-03-15T22:22:26.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: +0000 mon.a (mon.0) 877 2021-03-15T22:22:26.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: : audit [INF] from='client.? 172.21.15.76:0/3053777089' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-15T22:22:26.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:22:26.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: -15T22:22:26.038729+0000 mon.a ( 2021-03-15T22:22:26.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: mon.0) 878 : cluster 2021-03-15T22:22:26.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[56514]: [DBG] osdmap e231: 8 total, 8 up, 7 in 2021-03-15T22:22:26.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: cluster 2021-03-15T22:22:24.667801 2021-03-15T22:22:26.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: +0000 osd.2 (osd.2) 41 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:26.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:22:26.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: -15T22:22:24.667966+0000 osd.2 (osd.2) 42 : cluster [DBG] 2021-03-15T22:22:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: purged_snaps scrub ok 2021-03-15T22:22:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:22:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: 03-15T22:22:25.009758+0000 2021-03-15T22:22:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: mgr.y (mgr.14144) 470 : cluster [DBG] pgmap v471: 138 pgs: 138 active+clean; 130 MiB data, 459 MiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-03-15T22:22:26.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:22:26.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: -15T22:22:25.735006+0000 mon.a (mon 2021-03-15T22:22:26.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: .0) 876 : audit [INF] from='client.? 172.21.15.76:0/3053777089' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:22:26.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: audit 2021-03-15T22:22:26.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:22:26.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: 22:26.038661+0000 mon.a (mon.0) 2021-03-15T22:22:26.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: 877 : audit [INF] from='client.? 172.21.15.76:0/3053777089' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-15T22:22:26.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: cluster 2021-03-15T22:22:26.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: 2021-03-15T22:22:26.038729 2021-03-15T22:22:26.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: +0000 mon.a (mon.0) 878 : cluster 2021-03-15T22:22:26.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:26 smithi076 conmon[62156]: [DBG] osdmap e231: 8 total, 8 up, 7 in 2021-03-15T22:22:26.536 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[67289]: debug 2021-03-15T22:22:26.044+0000 7fd4d8831700 -1 osd.6 pg_epoch: 231 pg[7.7( v 229'99 (0'0,229'99] local-lis/les=172/173 n=2 ec=91/91 lis/c=172/172 les/c/f=173/173/0 sis=172) [6,0,1] r=0 lpr=172 crt=229'99 lcod 227'97 mlcod 227'97 active+clean+snaptrim trimq=[75~1] ps=[76~1,7f~1]] removing snap head 2021-03-15T22:22:26.536 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[63905]: debug 2021-03-15T22:22:26.537 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[63905]: 2021-03-15T22:22:26.044+0000 7fce04c39700 -1 osd.5 pg_epoch: 231 pg[7.0( v 229'382 (0'0,229'382] local-lis/les=206/207 n=8 ec=91/91 lis/c=206/206 les/c/f=207/207/0 sis=206) [5,1,2] r=0 lpr=206 crt=229'382 lcod 229'380 mlcod 229'380 active+clean+snaptrim trimq=[75~1] ps=[76~1,7f~1]] removing snap head 2021-03-15T22:22:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: cluster 2021-03-15T22:22:24.667801+0000 osd.2 (osd.2) 41 : cluster [DBG] 2021-03-15T22:22:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: purged_snaps scrub starts 2021-03-15T22:22:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: cluster 2021-03-15T22:22:24.667966+0000 osd.2 (osd.2) 42 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: cluster 2021-03-15T22:22:25.009758+0000 mgr.y 2021-03-15T22:22:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: (mgr.14144) 470 : cluster [DBG] pgmap v471: 138 pgs: 138 active+clean; 130 MiB data, 459 MiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-03-15T22:22:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:22:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: 03-15T22:22:25.735006+0000 2021-03-15T22:22:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: mon.a (mon.0) 876 : audit [INF] 2021-03-15T22:22:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/3053777089' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:22:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: audit 2021-03-15T22:22:26 2021-03-15T22:22:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: .038661+0000 mon.a (mon.0) 877 : 2021-03-15T22:22:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: audit [INF] from='client.? 172.21.15.76:0/3053777089' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-15T22:22:26.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: cluster 2021-03-15T22:22:26.038729+0000 mon.a ( 2021-03-15T22:22:26.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:26 smithi125 conmon[56739]: mon.0) 878 : cluster [DBG] osdmap e231: 8 total, 8 up, 7 in 2021-03-15T22:22:26.556 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:26.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:26.584 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:26.584 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:26.686 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:26.711 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:26.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:26.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:26.757 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-15T22:22:26.813 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:26.835 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:26.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:26.837 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:26.938 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:26.962 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:26.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:26.963 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:27.045 INFO:tasks.rados.rados.0.smithi076.stdout:672: done (3 left) 2021-03-15T22:22:27.046 INFO:tasks.rados.rados.0.smithi076.stdout:674: done (2 left) 2021-03-15T22:22:27.046 INFO:tasks.rados.rados.0.smithi076.stdout:675: done (1 left) 2021-03-15T22:22:27.046 INFO:tasks.rados.rados.0.smithi076.stdout:676: done (0 left) 2021-03-15T22:22:27.046 INFO:tasks.rados.rados.0.smithi076.stdout:677: read oid 1 snap -1 2021-03-15T22:22:27.047 INFO:tasks.rados.rados.0.smithi076.stdout:677: expect (ObjNum 198 snap 43 seq_num 198) 2021-03-15T22:22:27.047 INFO:tasks.rados.rados.0.smithi076.stdout:678: snap_create 2021-03-15T22:22:27.047 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:22:27.060 INFO:tasks.rados.rados.0.smithi076.stdout:677: expect (ObjNum 198 snap 43 seq_num 198) 2021-03-15T22:22:27.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:27.087 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:27.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:27.088 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:27.189 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:27.216 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:27.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:27.218 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:27.319 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:27.344 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:27.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:27.346 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:27.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:27.473 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:27.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:27.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:27.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[56514]: audit 2021-03-15T22:22: 2021-03-15T22:22:27.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[56514]: 26.811418+0000 mon.a (mon.0) 879 : audit [INF] from='client.? 172.21.15.76:0/3531797383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:22:27.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[56514]: cluster 2021-03-15T22:22:27.036062+0000 mon.a (mon.0) 880 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:27.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[56514]: audit 2021-03-15T22:22:27.042339+0000 mon.a (mon.0) 881 : audit [INF] from='client.? 172.21.15.76:0/3531797383' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:22:27.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[56514]: cluster 2021-03-15T22:22:27.042540+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2021-03-15T22:22:27.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: audit 2021-03-15T22:22:26. 2021-03-15T22:22:27.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: 811418+0000 mon.a (mon.0) 879 : audit [INF] from='client.? 172.21.15.76:0/3531797383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:22:27.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: cluster 2021-03-15T22:22:27.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: 2021-03-15T22:22:27.036062+0000 mon.a ( 2021-03-15T22:22:27.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: mon.0) 880 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:27.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: 2021-03-15T22:22:27.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:22:27.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: 22:27.042339+0000 mon.a (mon. 2021-03-15T22:22:27.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: 0) 881 : audit [INF] from='client.? 172.21.15.76:0/3531797383' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:22:27.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:22:27.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: 03-15T22:22:27. 2021-03-15T22:22:27.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: 042540+0000 mon.a (mon.0) 882 2021-03-15T22:22:27.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:27 smithi076 conmon[62156]: : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2021-03-15T22:22:27.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:27 smithi125 conmon[56739]: audit 2021-03-15T22:22:26.811418+0000 mon.a (mon.0) 879 : audit [INF] from='client.? 172.21.15.76:0/3531797383' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:22:27.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:27 smithi125 conmon[56739]: cluster 2021-03-15T22:22:27.036062+0000 mon.a (mon. 2021-03-15T22:22:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:27 smithi125 conmon[56739]: 0) 880 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:27 smithi125 conmon[56739]: audit 2021-03-15T22:22:27.042339+0000 mon.a (mon.0) 881 : audit [INF] from='client.? 172.21.15.76:0/3531797383' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:22:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:27 smithi125 conmon[56739]: cluster 2021-03-15T22:22:27.042540+0000 mon.a (mon.0) 882 : cluster [DBG] osdmap e232: 8 total, 8 up, 7 in 2021-03-15T22:22:27.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:27.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:27.608 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:27.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:27.710 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:27.737 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:27.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:27.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:27.747 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 5 1 2021-03-15T22:22:27.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:27.865 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:27.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:27.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:27.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:27.991 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:27.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:27.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:28.045 INFO:tasks.rados.rados.0.smithi076.stdout:677: done (1 left) 2021-03-15T22:22:28.045 INFO:tasks.rados.rados.0.smithi076.stdout:678: done (0 left) 2021-03-15T22:22:28.046 INFO:tasks.rados.rados.0.smithi076.stdout:679: snap_create 2021-03-15T22:22:28.094 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:28.118 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:28.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:28.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:28.221 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:28.248 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:28.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:28.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:28.352 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:28.376 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:28.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:28.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:28.504 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:28.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:28.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:28 smithi076 conmon[56514]: cluster 2021-03-15T22:22:27.010185+0000 mgr.y (mgr.14144) 471 : 2021-03-15T22:22:28.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:28 smithi076 conmon[56514]: cluster [DBG] pgmap v474: 138 pgs: 138 active+clean; 132 MiB data, 466 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2021-03-15T22:22:28.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:28 smithi076 conmon[56514]: cluster 2021-03-15T22:22:28.043643+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2021-03-15T22:22:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:28 smithi076 conmon[62156]: cluster 2021-03-15T22:22:27.010185+0000 mgr.y (mgr.14144) 471 : cluster [DBG] pgmap v474: 138 pgs: 138 active+clean; 132 MiB data, 466 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2021-03-15T22:22:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:28 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:22:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:28 smithi076 conmon[62156]: T22:22:28.043643+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2021-03-15T22:22:28.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:28 smithi125 conmon[56739]: cluster 2021-03-15T22:22:27.010185+0000 mgr.y (mgr.14144) 471 : cluster [DBG] pgmap v474: 138 pgs: 138 active+clean; 132 MiB data, 466 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2021-03-15T22:22:28.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:28 smithi125 conmon[56739]: cluster 2021-03-15T22:22:28.043643+0000 mon.a (mon.0) 883 : cluster [DBG] osdmap e233: 8 total, 8 up, 7 in 2021-03-15T22:22:28.604 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:28.606 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:28.632 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:28.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:28.633 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:28.718 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:28.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:28.759 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:28.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:28.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:28.778 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:22:28.796 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:22:28.814 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:22:28.833 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:22:28.851 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:22:28.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:28.870 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:22:28.888 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:22:28.888 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:28.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:28.905 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:22:28.907 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:22:28.907 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:22:28.907 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:22:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:22:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:22:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:22:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:22:28.908 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:22:28.991 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:29.017 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:29.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:29.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:29.048 INFO:tasks.rados.rados.0.smithi076.stdout:679: done (0 left) 2021-03-15T22:22:29.048 INFO:tasks.rados.rados.0.smithi076.stdout:680: snap_create 2021-03-15T22:22:29.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:29.146 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:29.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:29.248 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:29.271 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:29.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:29.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:29.375 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:29.397 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:29.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:29.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:29 smithi076 conmon[56514]: cluster 2021-03-15T22:22:27.911219+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.d scrub ok 2021-03-15T22:22:29.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:29 smithi076 conmon[56514]: cluster 2021-03-15T22:22:28.062744+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.a scrub ok 2021-03-15T22:22:29.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:29 smithi076 conmon[56514]: cluster 2021-03-15T22:22:28.711741+0000 osd.0 (osd.0) 50 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:22:29.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:29 smithi076 conmon[56514]: audit 2021-03-15T22:22:28.759023+0000 mon.a (mon.0) 884 : audit [DBG] from='client.? 172.21.15.76:0/893896474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:29.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:29 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:22:29.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:29 smithi076 conmon[56514]: :22:29.045064+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2021-03-15T22:22:29.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:29 smithi076 conmon[62156]: cluster 2021-03-15T22:22:27.911219+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.d scrub ok 2021-03-15T22:22:29.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:29 smithi076 conmon[62156]: cluster 2021-03-15T22:22:28.062744+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.a scrub ok 2021-03-15T22:22:29.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:29 smithi076 conmon[62156]: cluster 2021-03-15T22:22:28.711741+0000 osd.0 (osd.0) 50 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:22:29.488 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:29 smithi076 conmon[62156]: audit 2021-03-15T22:22:28.759023+0000 mon.a (mon.0) 884 : audit [DBG] from='client.? 172.21.15.76:0/893896474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:29.488 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:29 smithi076 conmon[62156]: cluster 2021-03-15T22:22:29.045064+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2021-03-15T22:22:29.499 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:29.525 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:29.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:29.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:29 smithi125 conmon[56739]: cluster 2021-03-15T22:22:27.911219+0000 osd.4 (osd.4) 55 : cluster [DBG] 4.d scrub ok 2021-03-15T22:22:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:29 smithi125 conmon[56739]: cluster 2021-03-15T22:22:28.062744+0000 osd.5 (osd.5) 48 : cluster [DBG] 6.a scrub ok 2021-03-15T22:22:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:29 smithi125 conmon[56739]: cluster 2021-03-15T22:22:28.711741+0000 osd.0 (osd.0) 50 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:22:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:29 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:22:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:29 smithi125 conmon[56739]: 03-15T22:22:28.759023+0000 mon.a (mon.0) 884 : audit [DBG] from='client.? 172.21.15.76:0/893896474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:29 smithi125 conmon[56739]: cluster 2021-03-15T22:22:29 2021-03-15T22:22:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:29 smithi125 conmon[56739]: .045064+0000 mon.a (mon.0) 885 : cluster [DBG] osdmap e234: 8 total, 8 up, 7 in 2021-03-15T22:22:29.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:29.650 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:29.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:29.752 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:29.779 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:29.781 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:29.882 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:29.904 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:29.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:29.906 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:30.007 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:30.034 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:30.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:30.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:22:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:22:30.039 INFO:tasks.rados.rados.0.smithi076.stdout:680: done (0 left) 2021-03-15T22:22:30.039 INFO:tasks.rados.rados.0.smithi076.stdout:681: snap_remove snap 71 2021-03-15T22:22:30.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:30.158 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:30.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:30.261 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:30.283 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:30.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:30.385 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:30.408 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:30.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:30.511 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:30.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[56514]: cluster 2021-03-15T22:22 2021-03-15T22:22:30.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[56514]: :28.717525+0000 osd.7 (osd.7) 53 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:22:30.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[56514]: cluster 2021-03-15T22:22:28.887584+0000 osd.4 (osd.4) 56 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:22:30.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[56514]: cluster 2021-03-15T22:22:29.010570+0000 mgr.y (mgr.14144) 472 : cluster [DBG] pgmap v477: 138 pgs: 138 active+clean; 124 MiB data, 452 MiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 796 KiB/s wr, 6 op/s 2021-03-15T22:22:30.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[56514]: audit 2021-03-15T22:22:29.541513+0000 mgr.y (mgr.14144) 473 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[56514]: cluster 2021-03-15T22:22:30.034835+0000 mon.a (mon.0) 886 : cluster [DBG] osdmap e235: 8 total, 8 up, 7 in 2021-03-15T22:22:30.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: cluster 2021-03-15T22:22:28.717525+0000 osd.7 (osd.7) 2021-03-15T22:22:30.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: 53 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:22:30.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: cluster 2021-03-15T22:22:28.887584+0000 osd.4 (osd.4) 56 2021-03-15T22:22:30.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: : cluster [DBG] 4.9 scrub ok 2021-03-15T22:22:30.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: cluster 2021-03-15T22:22:30.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: 2021-03-15T22:22:29.010570+0000 mgr.y (mgr. 2021-03-15T22:22:30.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: 14144) 472 : cluster [DBG] pgmap v477: 138 pgs: 138 active+clean; 124 MiB data, 452 MiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 796 KiB/s wr, 6 op/s 2021-03-15T22:22:30.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:22:30.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: -15T22:22:29.541513+0000 mgr.y (mgr. 2021-03-15T22:22:30.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: 14144) 473 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:30.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: cluster 2021-03-15T22:22:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: 2021-03-15T22:22:30. 2021-03-15T22:22:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: 034835+0000 mon.a (mon.0) 886 : cluster 2021-03-15T22:22:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:30 smithi076 conmon[62156]: [DBG] osdmap e235: 8 total, 8 up, 7 in 2021-03-15T22:22:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:30.534 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:30.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:30 smithi125 conmon[56739]: cluster 2021-03-15T22:22:28.717525+0000 osd.7 (osd.7) 53 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:22:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:30 smithi125 conmon[56739]: cluster 2021-03-15T22:22:28.887584+0000 osd.4 2021-03-15T22:22:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:30 smithi125 conmon[56739]: (osd.4) 56 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:22:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:30 smithi125 conmon[56739]: cluster 2021-03-15T22:22:29.010570+0000 mgr.y (mgr.14144) 472 : cluster [DBG] pgmap v477: 138 pgs: 138 active+clean; 124 MiB data, 452 MiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 796 KiB/s wr, 6 op/s 2021-03-15T22:22:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:30 smithi125 conmon[56739]: audit 2021-03-15T22:22:29.541513+0000 2021-03-15T22:22:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:30 smithi125 conmon[56739]: mgr.y (mgr.14144) 473 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:30 smithi125 conmon[56739]: cluster 2021-03-15T22:22:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:30 smithi125 conmon[56739]: 2021-03-15T22:22:30.034835+0000 mon.a (mon.0 2021-03-15T22:22:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:30 smithi125 conmon[56739]: ) 886 : cluster [DBG] osdmap e235: 8 total, 8 up, 7 in 2021-03-15T22:22:30.630 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:22:30.635 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:30.727 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:30.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:30.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:30.830 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:30.854 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:30.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:30.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:30.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:30.984 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:30.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:30.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:31.049 INFO:tasks.rados.rados.0.smithi076.stdout:681: done (0 left) 2021-03-15T22:22:31.050 INFO:tasks.rados.rados.0.smithi076.stdout:682: rollback oid 33 current snap is 72 2021-03-15T22:22:31.050 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 33 to 63 2021-03-15T22:22:31.050 INFO:tasks.rados.rados.0.smithi076.stdout:683: copy_from oid 7 from oid 16 current snap is 72 2021-03-15T22:22:31.050 INFO:tasks.rados.rados.0.smithi076.stdout:684: write oid 38 current snap is 72 2021-03-15T22:22:31.051 INFO:tasks.rados.rados.0.smithi076.stdout:684: seq_num 264 ranges {437048=621738,1832305=404425} 2021-03-15T22:22:31.055 INFO:tasks.rados.rados.0.smithi076.stdout:684: writing smithi07694878-38 from 437048 to 1058786 tid 1 2021-03-15T22:22:31.062 INFO:tasks.rados.rados.0.smithi076.stdout:684: writing smithi07694878-38 from 1832305 to 2236730 tid 2 2021-03-15T22:22:31.063 INFO:tasks.rados.rados.0.smithi076.stdout:685: copy_from oid 41 from oid 15 current snap is 72 2021-03-15T22:22:31.063 INFO:tasks.rados.rados.0.smithi076.stdout:683: finishing copy_from to smithi07694878-7 2021-03-15T22:22:31.063 INFO:tasks.rados.rados.0.smithi076.stdout:683: got expected ENOENT (src dne) 2021-03-15T22:22:31.063 INFO:tasks.rados.rados.0.smithi076.stdout:683: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:22:31.064 INFO:tasks.rados.rados.0.smithi076.stdout:682: finishing rollback tid 0 to smithi07694878-33 2021-03-15T22:22:31.064 INFO:tasks.rados.rados.0.smithi076.stdout:683: done (3 left) 2021-03-15T22:22:31.064 INFO:tasks.rados.rados.0.smithi076.stdout:686: write (excl) oid 32 current snap is 72 2021-03-15T22:22:31.064 INFO:tasks.rados.rados.0.smithi076.stdout:682: finishing rollback tid 1 to smithi07694878-33 2021-03-15T22:22:31.065 INFO:tasks.rados.rados.0.smithi076.stdout:686: seq_num 265 ranges {453855=696987,1890787=548083} 2021-03-15T22:22:31.069 INFO:tasks.rados.rados.0.smithi076.stdout:686: writing smithi07694878-32 from 453855 to 1150842 tid 1 2021-03-15T22:22:31.077 INFO:tasks.rados.rados.0.smithi076.stdout:686: writing smithi07694878-32 from 1890787 to 2438870 tid 2 2021-03-15T22:22:31.078 INFO:tasks.rados.rados.0.smithi076.stdout:687: read oid 36 snap 68 2021-03-15T22:22:31.078 INFO:tasks.rados.rados.0.smithi076.stdout:687: expect (ObjNum 180 snap 39 seq_num 180) 2021-03-15T22:22:31.079 INFO:tasks.rados.rados.0.smithi076.stdout:688: rollback oid 29 current snap is 72 2021-03-15T22:22:31.079 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 29 to 69 2021-03-15T22:22:31.079 INFO:tasks.rados.rados.0.smithi076.stdout:682: finishing rollback tid 2 to smithi07694878-33 2021-03-15T22:22:31.080 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 164 (ObjNum 185 snap 40 seq_num 185) dirty exists 2021-03-15T22:22:31.080 INFO:tasks.rados.rados.0.smithi076.stdout:684: finishing write tid 1 to smithi07694878-38 2021-03-15T22:22:31.080 INFO:tasks.rados.rados.0.smithi076.stdout:684: finishing write tid 2 to smithi07694878-38 2021-03-15T22:22:31.081 INFO:tasks.rados.rados.0.smithi076.stdout:684: finishing write tid 3 to smithi07694878-38 2021-03-15T22:22:31.081 INFO:tasks.rados.rados.0.smithi076.stdout:684: finishing write tid 4 to smithi07694878-38 2021-03-15T22:22:31.082 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 259 (ObjNum 264 snap 72 seq_num 264) dirty exists 2021-03-15T22:22:31.082 INFO:tasks.rados.rados.0.smithi076.stdout:684: left oid 38 (ObjNum 264 snap 72 seq_num 264) 2021-03-15T22:22:31.082 INFO:tasks.rados.rados.0.smithi076.stdout:685: finishing copy_from to smithi07694878-41 2021-03-15T22:22:31.083 INFO:tasks.rados.rados.0.smithi076.stdout:685: got expected ENOENT (src dne) 2021-03-15T22:22:31.083 INFO:tasks.rados.rados.0.smithi076.stdout:685: finishing copy_from racing read to smithi07694878-41 2021-03-15T22:22:31.084 INFO:tasks.rados.rados.0.smithi076.stdout:686: finishing write tid 1 to smithi07694878-32 2021-03-15T22:22:31.084 INFO:tasks.rados.rados.0.smithi076.stdout:682: done (5 left) 2021-03-15T22:22:31.084 INFO:tasks.rados.rados.0.smithi076.stdout:684: done (4 left) 2021-03-15T22:22:31.085 INFO:tasks.rados.rados.0.smithi076.stdout:685: done (3 left) 2021-03-15T22:22:31.085 INFO:tasks.rados.rados.0.smithi076.stdout:689: delete oid 27 current snap is 72 2021-03-15T22:22:31.086 INFO:tasks.rados.rados.0.smithi076.stdout:689: done (3 left) 2021-03-15T22:22:31.087 INFO:tasks.rados.rados.0.smithi076.stdout:690: read oid 20 snap -1 2021-03-15T22:22:31.087 INFO:tasks.rados.rados.0.smithi076.stdout:690: expect (ObjNum 198 snap 43 seq_num 198) 2021-03-15T22:22:31.088 INFO:tasks.rados.rados.0.smithi076.stdout:691: write oid 22 current snap is 72 2021-03-15T22:22:31.088 INFO:tasks.rados.rados.0.smithi076.stdout:691: seq_num 267 ranges {487469=563838,1607751=551752} 2021-03-15T22:22:31.089 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:31.089 INFO:tasks.rados.rados.0.smithi076.stdout:691: writing smithi07694878-22 from 487469 to 1051307 tid 1 2021-03-15T22:22:31.090 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:31.096 INFO:tasks.rados.rados.0.smithi076.stdout:691: writing smithi07694878-22 from 1607751 to 2159503 tid 2 2021-03-15T22:22:31.096 INFO:tasks.rados.rados.0.smithi076.stdout:692: read oid 15 snap -1 2021-03-15T22:22:31.097 INFO:tasks.rados.rados.0.smithi076.stdout:692: expect deleted 2021-03-15T22:22:31.097 INFO:tasks.rados.rados.0.smithi076.stdout:693: read oid 45 snap -1 2021-03-15T22:22:31.097 INFO:tasks.rados.rados.0.smithi076.stdout:693: expect (ObjNum 6 snap 0 seq_num 6) 2021-03-15T22:22:31.098 INFO:tasks.rados.rados.0.smithi076.stdout:694: delete oid 11 current snap is 72 2021-03-15T22:22:31.098 INFO:tasks.rados.rados.0.smithi076.stdout:686: finishing write tid 2 to smithi07694878-32 2021-03-15T22:22:31.098 INFO:tasks.rados.rados.0.smithi076.stdout:686: finishing write tid 3 to smithi07694878-32 2021-03-15T22:22:31.099 INFO:tasks.rados.rados.0.smithi076.stdout:686: finishing write tid 4 to smithi07694878-32 2021-03-15T22:22:31.099 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 402 (ObjNum 265 snap 72 seq_num 265) dirty exists 2021-03-15T22:22:31.100 INFO:tasks.rados.rados.0.smithi076.stdout:686: left oid 32 (ObjNum 265 snap 72 seq_num 265) 2021-03-15T22:22:31.100 INFO:tasks.rados.rados.0.smithi076.stdout:688: finishing rollback tid 0 to smithi07694878-29 2021-03-15T22:22:31.100 INFO:tasks.rados.rados.0.smithi076.stdout:688: finishing rollback tid 1 to smithi07694878-29 2021-03-15T22:22:31.101 INFO:tasks.rados.rados.0.smithi076.stdout:687: expect (ObjNum 180 snap 39 seq_num 180) 2021-03-15T22:22:31.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:31.115 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:31.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[56514]: cluster 2021-03-15T22:22:29.689306+0000 osd.2 (osd.2) 43 : cluster [DBG] 7.a scrub ok 2021-03-15T22:22:31.162 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[56514]: cluster 2021-03-15T22:22:30.800419+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:22:31.163 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[56514]: cluster 2021-03-15T22:22:31.045351+0000 mon.a (mon.0) 887 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2021-03-15T22:22:31.163 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[62156]: cluster 2021-03-15T22:22:31.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[62156]: 2021-03-15T22:22 2021-03-15T22:22:31.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[62156]: :29.689306+0000 osd.2 (osd.2) 43 : cluster [DBG] 7.a scrub ok 2021-03-15T22:22:31.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:22:31.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[62156]: 15T22:22:30.800419+0000 osd.0 (osd.0) 51 2021-03-15T22:22:31.165 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[62156]: : cluster [DBG] 6.16 scrub ok 2021-03-15T22:22:31.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:22:31.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[62156]: 03-15T22:22:31.045351+0000 mon.a 2021-03-15T22:22:31.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:31 smithi076 conmon[62156]: (mon.0) 887 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2021-03-15T22:22:31.217 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:31.222 INFO:tasks.rados.rados.0.smithi076.stdout:688: finishing rollback tid 2 to smithi07694878-29 2021-03-15T22:22:31.222 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 406 (ObjNum 261 snap 68 seq_num 261) dirty exists 2021-03-15T22:22:31.222 INFO:tasks.rados.rados.0.smithi076.stdout:686: done (7 left) 2021-03-15T22:22:31.222 INFO:tasks.rados.rados.0.smithi076.stdout:687: done (6 left) 2021-03-15T22:22:31.223 INFO:tasks.rados.rados.0.smithi076.stdout:688: done (5 left) 2021-03-15T22:22:31.223 INFO:tasks.rados.rados.0.smithi076.stdout:694: done (4 left) 2021-03-15T22:22:31.223 INFO:tasks.rados.rados.0.smithi076.stdout:695: snap_remove snap 68 2021-03-15T22:22:31.242 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:31.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:31.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:31.344 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:31.366 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:31.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:31.375 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:22:31.375 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:22:31.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:22:31.376 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:22:31.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.377 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:257 7.3 7:c7825e8f:::smithi07694878-26:head [copy-from ver 3] snapc 5=[] ondisk+write+known_if_redirected e103)", 2021-03-15T22:22:31.377 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:18.250698+0000", 2021-03-15T22:22:31.377 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 133.12077444100001, 2021-03-15T22:22:31.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013491402, 2021-03-15T22:22:31.377 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.378 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:31.378 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.379 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 257 2021-03-15T22:22:31.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.380 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250698+0000", 2021-03-15T22:22:31.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250698+0000", 2021-03-15T22:22:31.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8069999999999999e-06 2021-03-15T22:22:31.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250703+0000", 2021-03-15T22:22:31.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2169999999999997e-06 2021-03-15T22:22:31.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250710+0000", 2021-03-15T22:22:31.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9879999999999999e-05 2021-03-15T22:22:31.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250730+0000", 2021-03-15T22:22:31.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2840000000000006e-06 2021-03-15T22:22:31.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250739+0000", 2021-03-15T22:22:31.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9257000000000005e-05 2021-03-15T22:22:31.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250808+0000", 2021-03-15T22:22:31.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8200000000000003e-06 2021-03-15T22:22:31.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:22:31.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250817+0000", 2021-03-15T22:22:31.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001712873 2021-03-15T22:22:31.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.252529+0000", 2021-03-15T22:22:31.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2012e-05 2021-03-15T22:22:31.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.252612+0000", 2021-03-15T22:22:31.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055665840000000003 2021-03-15T22:22:31.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:22:31.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.258178+0000", 2021-03-15T22:22:31.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052418129999999997 2021-03-15T22:22:31.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:22:31.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.263420+0000", 2021-03-15T22:22:31.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029105599999999999 2021-03-15T22:22:31.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.263711+0000", 2021-03-15T22:22:31.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039116800000000001 2021-03-15T22:22:31.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264102+0000", 2021-03-15T22:22:31.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3337000000000002e-05 2021-03-15T22:22:31.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:31.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264145+0000", 2021-03-15T22:22:31.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4294e-05 2021-03-15T22:22:31.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264190+0000", 2021-03-15T22:22:31.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013491402 2021-03-15T22:22:31.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.400 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.401 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:258 7.3 7:c7825e8f:::smithi07694878-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e103)", 2021-03-15T22:22:31.401 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:18.250762+0000", 2021-03-15T22:22:31.401 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 133.12071098800001, 2021-03-15T22:22:31.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013601281999999999, 2021-03-15T22:22:31.402 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.402 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:31.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.404 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 258 2021-03-15T22:22:31.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.404 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250762+0000", 2021-03-15T22:22:31.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250762+0000", 2021-03-15T22:22:31.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.528e-06 2021-03-15T22:22:31.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250763+0000", 2021-03-15T22:22:31.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8560000000000001e-06 2021-03-15T22:22:31.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250765+0000", 2021-03-15T22:22:31.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9169999999999995e-06 2021-03-15T22:22:31.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250774+0000", 2021-03-15T22:22:31.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9060000000000001e-06 2021-03-15T22:22:31.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250779+0000", 2021-03-15T22:22:31.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9030999999999995e-05 2021-03-15T22:22:31.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250868+0000", 2021-03-15T22:22:31.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1899999999999998e-06 2021-03-15T22:22:31.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:22:31.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250875+0000", 2021-03-15T22:22:31.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018793950000000001 2021-03-15T22:22:31.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.252755+0000", 2021-03-15T22:22:31.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0155999999999999e-05 2021-03-15T22:22:31.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:31.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.252805+0000", 2021-03-15T22:22:31.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077066119999999998 2021-03-15T22:22:31.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.260511+0000", 2021-03-15T22:22:31.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8893000000000006e-05 2021-03-15T22:22:31.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:31.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.260580+0000", 2021-03-15T22:22:31.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036392949999999999 2021-03-15T22:22:31.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264220+0000", 2021-03-15T22:22:31.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8607999999999999e-05 2021-03-15T22:22:31.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264278+0000", 2021-03-15T22:22:31.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4895000000000006e-05 2021-03-15T22:22:31.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264363+0000", 2021-03-15T22:22:31.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013601281999999999 2021-03-15T22:22:31.425 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.426 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.426 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.427 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:350 7.7 e106/91 7:ecc0cb20:::smithi07694878-25:head v 106'22, mlcod=105'19)", 2021-03-15T22:22:31.428 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:21.035952+0000", 2021-03-15T22:22:31.428 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 130.33552051199999, 2021-03-15T22:22:31.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010302982, 2021-03-15T22:22:31.429 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.430 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:31.430 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.035952+0000", 2021-03-15T22:22:31.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.035952+0000", 2021-03-15T22:22:31.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048549860000000004 2021-03-15T22:22:31.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.040807+0000", 2021-03-15T22:22:31.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5889999999999999e-06 2021-03-15T22:22:31.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.040811+0000", 2021-03-15T22:22:31.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2366e-05 2021-03-15T22:22:31.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.040823+0000", 2021-03-15T22:22:31.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6690000000000003e-06 2021-03-15T22:22:31.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.040829+0000", 2021-03-15T22:22:31.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9060000000000001e-05 2021-03-15T22:22:31.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.040878+0000", 2021-03-15T22:22:31.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6973000000000001e-05 2021-03-15T22:22:31.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.040905+0000", 2021-03-15T22:22:31.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053090780000000001 2021-03-15T22:22:31.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:31.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.046214+0000", 2021-03-15T22:22:31.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1260999999999997e-05 2021-03-15T22:22:31.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:21.046255+0000", 2021-03-15T22:22:31.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010302982 2021-03-15T22:22:31.451 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.452 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.452 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.453 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:366 7.0 e108/91 7:0732c451:::smithi07694878-32:head v 108'44, mlcod=108'40)", 2021-03-15T22:22:31.454 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:23.113325+0000", 2021-03-15T22:22:31.454 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 128.25814747199999, 2021-03-15T22:22:31.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099697889999999997, 2021-03-15T22:22:31.455 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.455 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:31.456 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.113325+0000", 2021-03-15T22:22:31.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.113325+0000", 2021-03-15T22:22:31.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0057349940000000002 2021-03-15T22:22:31.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.119060+0000", 2021-03-15T22:22:31.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9639999999999997e-06 2021-03-15T22:22:31.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.119067+0000", 2021-03-15T22:22:31.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4348000000000001e-05 2021-03-15T22:22:31.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.119092+0000", 2021-03-15T22:22:31.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7860000000000002e-06 2021-03-15T22:22:31.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.119100+0000", 2021-03-15T22:22:31.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073482400000000002 2021-03-15T22:22:31.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.119835+0000", 2021-03-15T22:22:31.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1395000000000001e-05 2021-03-15T22:22:31.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.119857+0000", 2021-03-15T22:22:31.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003392038 2021-03-15T22:22:31.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:31.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.123249+0000", 2021-03-15T22:22:31.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6440000000000003e-05 2021-03-15T22:22:31.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.123295+0000", 2021-03-15T22:22:31.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099697889999999997 2021-03-15T22:22:31.468 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.469 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.469 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.470 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:539 7.2 7:45ee0f2f:::smithi07694878-6:head [copy-from ver 16] snapc 1b=[1b,19,18,17] ondisk+write+known_if_redirected e123)", 2021-03-15T22:22:31.470 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:38.108937+0000", 2021-03-15T22:22:31.470 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 113.262535304, 2021-03-15T22:22:31.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018433483000000001, 2021-03-15T22:22:31.471 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.471 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:31.471 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.472 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 539 2021-03-15T22:22:31.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.473 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108937+0000", 2021-03-15T22:22:31.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108937+0000", 2021-03-15T22:22:31.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.977e-06 2021-03-15T22:22:31.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108940+0000", 2021-03-15T22:22:31.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7489999999999999e-06 2021-03-15T22:22:31.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108943+0000", 2021-03-15T22:22:31.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.348e-05 2021-03-15T22:22:31.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108957+0000", 2021-03-15T22:22:31.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0370000000000002e-06 2021-03-15T22:22:31.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108962+0000", 2021-03-15T22:22:31.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5223999999999999e-05 2021-03-15T22:22:31.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.109007+0000", 2021-03-15T22:22:31.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7335999999999994e-05 2021-03-15T22:22:31.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.109074+0000", 2021-03-15T22:22:31.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007106991 2021-03-15T22:22:31.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:22:31.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.116181+0000", 2021-03-15T22:22:31.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010935861999999999 2021-03-15T22:22:31.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127117+0000", 2021-03-15T22:22:31.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4037e-05 2021-03-15T22:22:31.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127151+0000", 2021-03-15T22:22:31.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138885 2021-03-15T22:22:31.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:22:31.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127290+0000", 2021-03-15T22:22:31.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6146000000000003e-05 2021-03-15T22:22:31.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:31.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127326+0000", 2021-03-15T22:22:31.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4759000000000003e-05 2021-03-15T22:22:31.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127371+0000", 2021-03-15T22:22:31.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018433483000000001 2021-03-15T22:22:31.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.493 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.494 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:540 7.2 7:45ee0f2f:::smithi07694878-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e123)", 2021-03-15T22:22:31.494 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:38.108970+0000", 2021-03-15T22:22:31.494 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 113.262502545, 2021-03-15T22:22:31.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018548431000000001, 2021-03-15T22:22:31.495 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.495 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:31.496 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.496 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.496 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.496 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 540 2021-03-15T22:22:31.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.497 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108970+0000", 2021-03-15T22:22:31.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108970+0000", 2021-03-15T22:22:31.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2299999999999999e-07 2021-03-15T22:22:31.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108971+0000", 2021-03-15T22:22:31.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.35e-06 2021-03-15T22:22:31.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108973+0000", 2021-03-15T22:22:31.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9750000000000001e-06 2021-03-15T22:22:31.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108976+0000", 2021-03-15T22:22:31.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5330000000000001e-06 2021-03-15T22:22:31.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108979+0000", 2021-03-15T22:22:31.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000204456 2021-03-15T22:22:31.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.109183+0000", 2021-03-15T22:22:31.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3643000000000003e-05 2021-03-15T22:22:31.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:31.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.109217+0000", 2021-03-15T22:22:31.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011687242 2021-03-15T22:22:31.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.120904+0000", 2021-03-15T22:22:31.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010511299999999999 2021-03-15T22:22:31.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:31.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.121009+0000", 2021-03-15T22:22:31.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063902330000000004 2021-03-15T22:22:31.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127400+0000", 2021-03-15T22:22:31.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3569000000000002e-05 2021-03-15T22:22:31.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127453+0000", 2021-03-15T22:22:31.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5394000000000004e-05 2021-03-15T22:22:31.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127519+0000", 2021-03-15T22:22:31.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018548431000000001 2021-03-15T22:22:31.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.512 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.513 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:830 7.a 7:58400cda:::smithi07694878-39:32 [read 0~3949612,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e149)", 2021-03-15T22:22:31.513 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018803+0000", 2021-03-15T22:22:31.513 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.352670265, 2021-03-15T22:22:31.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099842969999999996, 2021-03-15T22:22:31.514 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.514 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:31.514 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.514 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.515 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.515 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 830 2021-03-15T22:22:31.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.515 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018803+0000", 2021-03-15T22:22:31.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018803+0000", 2021-03-15T22:22:31.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9749999999999998e-06 2021-03-15T22:22:31.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018808+0000", 2021-03-15T22:22:31.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0980000000000004e-06 2021-03-15T22:22:31.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018812+0000", 2021-03-15T22:22:31.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8664999999999998e-05 2021-03-15T22:22:31.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018830+0000", 2021-03-15T22:22:31.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0960000000000003e-06 2021-03-15T22:22:31.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018837+0000", 2021-03-15T22:22:31.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7676000000000001e-05 2021-03-15T22:22:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018895+0000", 2021-03-15T22:22:31.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3160000000000003e-06 2021-03-15T22:22:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:22:31.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018904+0000", 2021-03-15T22:22:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003536656 2021-03-15T22:22:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.022441+0000", 2021-03-15T22:22:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3789999999999998e-05 2021-03-15T22:22:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.022495+0000", 2021-03-15T22:22:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062920249999999997 2021-03-15T22:22:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.028787+0000", 2021-03-15T22:22:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099842969999999996 2021-03-15T22:22:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.528 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:831 7.a 7:58400cda:::smithi07694878-39:32 [sparse-read 0~1521593] snapc 0=[] ondisk+read+known_if_redirected e149)", 2021-03-15T22:22:31.529 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018849+0000", 2021-03-15T22:22:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.352623632999993, 2021-03-15T22:22:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010236998000000001, 2021-03-15T22:22:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.530 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 831 2021-03-15T22:22:31.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018849+0000", 2021-03-15T22:22:31.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018849+0000", 2021-03-15T22:22:31.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2500000000000001e-06 2021-03-15T22:22:31.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018850+0000", 2021-03-15T22:22:31.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4890000000000001e-06 2021-03-15T22:22:31.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.535 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018852+0000", 2021-03-15T22:22:31.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7270000000000004e-06 2021-03-15T22:22:31.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018858+0000", 2021-03-15T22:22:31.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2420000000000002e-06 2021-03-15T22:22:31.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018862+0000", 2021-03-15T22:22:31.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6997000000000007e-05 2021-03-15T22:22:31.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018959+0000", 2021-03-15T22:22:31.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5729999999999996e-06 2021-03-15T22:22:31.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:22:31.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018965+0000", 2021-03-15T22:22:31.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098575079999999992 2021-03-15T22:22:31.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.028823+0000", 2021-03-15T22:22:31.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000110031 2021-03-15T22:22:31.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.028933+0000", 2021-03-15T22:22:31.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015318100000000001 2021-03-15T22:22:31.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.029086+0000", 2021-03-15T22:22:31.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010236998000000001 2021-03-15T22:22:31.544 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.544 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.544 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.545 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:832 7.a 7:58400cda:::smithi07694878-39:32 [sparse-read 0~1521593] snapc 0=[] ondisk+read+known_if_redirected e149)", 2021-03-15T22:22:31.545 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018866+0000", 2021-03-15T22:22:31.545 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.352606979000001, 2021-03-15T22:22:31.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010483065999999999, 2021-03-15T22:22:31.546 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.546 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:31.546 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.547 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.547 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.547 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 832 2021-03-15T22:22:31.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.547 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018866+0000", 2021-03-15T22:22:31.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018866+0000", 2021-03-15T22:22:31.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2200000000000002e-07 2021-03-15T22:22:31.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018867+0000", 2021-03-15T22:22:31.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.062e-06 2021-03-15T22:22:31.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018868+0000", 2021-03-15T22:22:31.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.147e-06 2021-03-15T22:22:31.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018873+0000", 2021-03-15T22:22:31.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8260000000000001e-06 2021-03-15T22:22:31.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018876+0000", 2021-03-15T22:22:31.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109102 2021-03-15T22:22:31.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018985+0000", 2021-03-15T22:22:31.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9869999999999996e-06 2021-03-15T22:22:31.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:22:31.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018991+0000", 2021-03-15T22:22:31.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010138061 2021-03-15T22:22:31.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.029129+0000", 2021-03-15T22:22:31.557 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6847000000000004e-05 2021-03-15T22:22:31.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.029216+0000", 2021-03-15T22:22:31.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013311200000000001 2021-03-15T22:22:31.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.029349+0000", 2021-03-15T22:22:31.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010483065999999999 2021-03-15T22:22:31.560 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.560 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.560 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.561 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:833 7.a 7:5bc7f316:::smithi07694878-38:head [sparse-read 0~1482386,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e149)", 2021-03-15T22:22:31.562 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018879+0000", 2021-03-15T22:22:31.562 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.352594206999996, 2021-03-15T22:22:31.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010936191, 2021-03-15T22:22:31.563 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.564 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:31.564 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.564 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.565 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.565 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 833 2021-03-15T22:22:31.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.565 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.566 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018879+0000", 2021-03-15T22:22:31.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.567 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018879+0000", 2021-03-15T22:22:31.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3540000000000001e-06 2021-03-15T22:22:31.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018880+0000", 2021-03-15T22:22:31.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7999999999999993e-07 2021-03-15T22:22:31.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.569 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018881+0000", 2021-03-15T22:22:31.570 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9739999999999998e-06 2021-03-15T22:22:31.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.570 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018884+0000", 2021-03-15T22:22:31.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1160000000000002e-06 2021-03-15T22:22:31.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018886+0000", 2021-03-15T22:22:31.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012289399999999999 2021-03-15T22:22:31.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.019009+0000", 2021-03-15T22:22:31.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9490000000000001e-06 2021-03-15T22:22:31.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:22:31.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.019015+0000", 2021-03-15T22:22:31.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010395936999999999 2021-03-15T22:22:31.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.029411+0000", 2021-03-15T22:22:31.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013028599999999999 2021-03-15T22:22:31.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.576 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.029541+0000", 2021-03-15T22:22:31.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000273701 2021-03-15T22:22:31.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.577 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.029815+0000", 2021-03-15T22:22:31.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010936191 2021-03-15T22:22:31.578 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.578 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.578 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.579 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:834 7.a 7:5bc7f316:::smithi07694878-38:head [read 0~2016840,checksum] snapc 0=[] ondisk+read+known_if_redirected e149)", 2021-03-15T22:22:31.579 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018888+0000", 2021-03-15T22:22:31.579 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.352584386000004, 2021-03-15T22:22:31.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011790652, 2021-03-15T22:22:31.580 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.580 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:31.580 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.581 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.581 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.581 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 834 2021-03-15T22:22:31.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.582 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.582 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.582 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.582 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018888+0000", 2021-03-15T22:22:31.583 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.583 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.583 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.583 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.583 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018888+0000", 2021-03-15T22:22:31.584 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1629999999999999e-06 2021-03-15T22:22:31.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.584 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.585 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018890+0000", 2021-03-15T22:22:31.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8000000000000005e-07 2021-03-15T22:22:31.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.585 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.585 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.586 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018890+0000", 2021-03-15T22:22:31.586 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9790000000000002e-06 2021-03-15T22:22:31.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.587 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018893+0000", 2021-03-15T22:22:31.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0449999999999999e-06 2021-03-15T22:22:31.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018895+0000", 2021-03-15T22:22:31.588 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000136415 2021-03-15T22:22:31.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.589 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.590 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.019032+0000", 2021-03-15T22:22:31.590 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1510000000000001e-06 2021-03-15T22:22:31.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.591 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:22:31.591 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.019038+0000", 2021-03-15T22:22:31.591 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010808144 2021-03-15T22:22:31.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.592 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.029846+0000", 2021-03-15T22:22:31.592 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2358e-05 2021-03-15T22:22:31.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.029928+0000", 2021-03-15T22:22:31.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00075061699999999997 2021-03-15T22:22:31.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.030679+0000", 2021-03-15T22:22:31.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011790652 2021-03-15T22:22:31.595 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.595 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.595 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.596 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:835 7.a 7:5bc7f316:::smithi07694878-38:head [sparse-read 0~1482386] snapc 0=[] ondisk+read+known_if_redirected e149)", 2021-03-15T22:22:31.596 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018897+0000", 2021-03-15T22:22:31.596 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.352575328, 2021-03-15T22:22:31.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012112934000000001, 2021-03-15T22:22:31.597 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.597 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:31.597 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.597 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.598 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.598 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 835 2021-03-15T22:22:31.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.598 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.599 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.599 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018897+0000", 2021-03-15T22:22:31.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018897+0000", 2021-03-15T22:22:31.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8299999999999997e-07 2021-03-15T22:22:31.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018898+0000", 2021-03-15T22:22:31.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1999999999999999e-07 2021-03-15T22:22:31.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.602 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018899+0000", 2021-03-15T22:22:31.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8559999999999998e-06 2021-03-15T22:22:31.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018902+0000", 2021-03-15T22:22:31.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4490000000000002e-06 2021-03-15T22:22:31.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018904+0000", 2021-03-15T22:22:31.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015147500000000001 2021-03-15T22:22:31.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.019056+0000", 2021-03-15T22:22:31.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6529999999999999e-06 2021-03-15T22:22:31.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:22:31.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.019061+0000", 2021-03-15T22:22:31.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011683427999999999 2021-03-15T22:22:31.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.030745+0000", 2021-03-15T22:22:31.610 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011034799999999999 2021-03-15T22:22:31.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.611 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.611 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.030855+0000", 2021-03-15T22:22:31.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000155422 2021-03-15T22:22:31.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.031010+0000", 2021-03-15T22:22:31.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012112934000000001 2021-03-15T22:22:31.612 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.613 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.613 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.614 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:836 7.3 7:c458c800:::smithi07694878-3:head [copy-from ver 127] snapc 34=[34,32,30] ondisk+write+known_if_redirected e149)", 2021-03-15T22:22:31.614 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018906+0000", 2021-03-15T22:22:31.614 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.352566413000005, 2021-03-15T22:22:31.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024704783000000001, 2021-03-15T22:22:31.614 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.615 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:31.615 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.615 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.615 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.616 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 836 2021-03-15T22:22:31.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.616 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.616 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018906+0000", 2021-03-15T22:22:31.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018906+0000", 2021-03-15T22:22:31.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.209e-06 2021-03-15T22:22:31.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.619 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018908+0000", 2021-03-15T22:22:31.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9100000000000003e-07 2021-03-15T22:22:31.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.620 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.621 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018908+0000", 2021-03-15T22:22:31.621 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2780000000000002e-06 2021-03-15T22:22:31.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018912+0000", 2021-03-15T22:22:31.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9469999999999999e-06 2021-03-15T22:22:31.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.624 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018915+0000", 2021-03-15T22:22:31.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0328000000000003e-05 2021-03-15T22:22:31.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018965+0000", 2021-03-15T22:22:31.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7919999999999993e-06 2021-03-15T22:22:31.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:22:31.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018973+0000", 2021-03-15T22:22:31.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025313699999999998 2021-03-15T22:22:31.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021504+0000", 2021-03-15T22:22:31.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4449000000000006e-05 2021-03-15T22:22:31.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.628 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021568+0000", 2021-03-15T22:22:31.629 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071715939999999999 2021-03-15T22:22:31.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:22:31.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.028740+0000", 2021-03-15T22:22:31.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012489007 2021-03-15T22:22:31.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.632 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.633 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.041229+0000", 2021-03-15T22:22:31.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001956981 2021-03-15T22:22:31.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:22:31.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043186+0000", 2021-03-15T22:22:31.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000330864 2021-03-15T22:22:31.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043517+0000", 2021-03-15T22:22:31.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8581000000000001e-05 2021-03-15T22:22:31.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:31.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043565+0000", 2021-03-15T22:22:31.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5692000000000002e-05 2021-03-15T22:22:31.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.637 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.637 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043611+0000", 2021-03-15T22:22:31.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024704783000000001 2021-03-15T22:22:31.638 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.638 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.638 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.639 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:837 7.3 7:c458c800:::smithi07694878-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e149)", 2021-03-15T22:22:31.639 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018929+0000", 2021-03-15T22:22:31.639 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.352543861000001, 2021-03-15T22:22:31.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024813472, 2021-03-15T22:22:31.640 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.640 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:31.640 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.640 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.641 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.641 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 837 2021-03-15T22:22:31.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.641 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018929+0000", 2021-03-15T22:22:31.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.643 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018929+0000", 2021-03-15T22:22:31.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0210000000000001e-06 2021-03-15T22:22:31.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018930+0000", 2021-03-15T22:22:31.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1789999999999997e-06 2021-03-15T22:22:31.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.645 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018939+0000", 2021-03-15T22:22:31.645 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7000000000000003e-06 2021-03-15T22:22:31.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.646 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018949+0000", 2021-03-15T22:22:31.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0043e-05 2021-03-15T22:22:31.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018959+0000", 2021-03-15T22:22:31.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2956000000000001e-05 2021-03-15T22:22:31.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018992+0000", 2021-03-15T22:22:31.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6950000000000002e-06 2021-03-15T22:22:31.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:22:31.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018996+0000", 2021-03-15T22:22:31.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026765769999999999 2021-03-15T22:22:31.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021672+0000", 2021-03-15T22:22:31.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1886999999999999e-05 2021-03-15T22:22:31.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:31.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021734+0000", 2021-03-15T22:22:31.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015444232 2021-03-15T22:22:31.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.037178+0000", 2021-03-15T22:22:31.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8273999999999998e-05 2021-03-15T22:22:31.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:31.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.037236+0000", 2021-03-15T22:22:31.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064013719999999998 2021-03-15T22:22:31.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043638+0000", 2021-03-15T22:22:31.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4591e-05 2021-03-15T22:22:31.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043682+0000", 2021-03-15T22:22:31.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9945000000000003e-05 2021-03-15T22:22:31.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043742+0000", 2021-03-15T22:22:31.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024813472 2021-03-15T22:22:31.658 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.658 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.658 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.659 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:838 7.a 7:53dee572:::smithi07694878-35:head [write 719366~478716,stat] snapc 34=[34,32,30] ondisk+write+known_if_redirected e149)", 2021-03-15T22:22:31.659 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.025398+0000", 2021-03-15T22:22:31.659 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.346075103000004, 2021-03-15T22:22:31.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018168215000000001, 2021-03-15T22:22:31.660 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.660 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:31.660 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.661 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.661 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.661 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 838 2021-03-15T22:22:31.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.661 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025398+0000", 2021-03-15T22:22:31.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025398+0000", 2021-03-15T22:22:31.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024823199999999998 2021-03-15T22:22:31.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025646+0000", 2021-03-15T22:22:31.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8279999999999999e-06 2021-03-15T22:22:31.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025649+0000", 2021-03-15T22:22:31.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2312e-05 2021-03-15T22:22:31.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025661+0000", 2021-03-15T22:22:31.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3240000000000002e-06 2021-03-15T22:22:31.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025666+0000", 2021-03-15T22:22:31.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053758139999999996 2021-03-15T22:22:31.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.031042+0000", 2021-03-15T22:22:31.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5798999999999997e-05 2021-03-15T22:22:31.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.031118+0000", 2021-03-15T22:22:31.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027348699999999999 2021-03-15T22:22:31.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:22:31.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.031391+0000", 2021-03-15T22:22:31.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023824789999999998 2021-03-15T22:22:31.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:22:31.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.033774+0000", 2021-03-15T22:22:31.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024423019999999999 2021-03-15T22:22:31.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.036216+0000", 2021-03-15T22:22:31.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071806320000000002 2021-03-15T22:22:31.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043397+0000", 2021-03-15T22:22:31.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1420000000000005e-05 2021-03-15T22:22:31.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:31.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043458+0000", 2021-03-15T22:22:31.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107586 2021-03-15T22:22:31.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043566+0000", 2021-03-15T22:22:31.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018168215000000001 2021-03-15T22:22:31.677 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.678 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.678 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.678 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:839 7.a 7:53dee572:::smithi07694878-35:head [write 1808868~510498] snapc 34=[34,32,30] ondisk+write+known_if_redirected e149)", 2021-03-15T22:22:31.679 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.035536+0000", 2021-03-15T22:22:31.679 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.335936904999997, 2021-03-15T22:22:31.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099430850000000008, 2021-03-15T22:22:31.679 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.680 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:31.680 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.680 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.680 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.681 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 839 2021-03-15T22:22:31.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.681 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035536+0000", 2021-03-15T22:22:31.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035536+0000", 2021-03-15T22:22:31.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017081799999999999 2021-03-15T22:22:31.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035707+0000", 2021-03-15T22:22:31.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6659999999999999e-06 2021-03-15T22:22:31.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035711+0000", 2021-03-15T22:22:31.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3878000000000001e-05 2021-03-15T22:22:31.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035735+0000", 2021-03-15T22:22:31.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2210000000000003e-06 2021-03-15T22:22:31.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035744+0000", 2021-03-15T22:22:31.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016437999999999999 2021-03-15T22:22:31.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035909+0000", 2021-03-15T22:22:31.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2387999999999997e-05 2021-03-15T22:22:31.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:31.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.036001+0000", 2021-03-15T22:22:31.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0076107579999999996 2021-03-15T22:22:31.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043612+0000", 2021-03-15T22:22:31.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4861000000000002e-05 2021-03-15T22:22:31.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043667+0000", 2021-03-15T22:22:31.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013138000000000001 2021-03-15T22:22:31.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:22:31.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043798+0000", 2021-03-15T22:22:31.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001029929 2021-03-15T22:22:31.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:22:31.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.044828+0000", 2021-03-15T22:22:31.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026279999999999999 2021-03-15T22:22:31.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.045091+0000", 2021-03-15T22:22:31.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00031650999999999999 2021-03-15T22:22:31.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.045407+0000", 2021-03-15T22:22:31.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2944999999999997e-05 2021-03-15T22:22:31.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:31.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.045440+0000", 2021-03-15T22:22:31.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8550999999999997e-05 2021-03-15T22:22:31.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.045479+0000", 2021-03-15T22:22:31.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099430850000000008 2021-03-15T22:22:31.699 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.700 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.700 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.700 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:841 7.a 7:53dee572:::smithi07694878-35:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e149)", 2021-03-15T22:22:31.701 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.035780+0000", 2021-03-15T22:22:31.701 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 87.335692734999995, 2021-03-15T22:22:31.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099394189999999997, 2021-03-15T22:22:31.701 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.702 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:31.702 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.702 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.702 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.703 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 841 2021-03-15T22:22:31.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.703 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035780+0000", 2021-03-15T22:22:31.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035780+0000", 2021-03-15T22:22:31.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1429999999999999e-06 2021-03-15T22:22:31.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035781+0000", 2021-03-15T22:22:31.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4449999999999999e-06 2021-03-15T22:22:31.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035783+0000", 2021-03-15T22:22:31.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.914e-05 2021-03-15T22:22:31.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035802+0000", 2021-03-15T22:22:31.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.194e-06 2021-03-15T22:22:31.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.035806+0000", 2021-03-15T22:22:31.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030803500000000001 2021-03-15T22:22:31.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.036114+0000", 2021-03-15T22:22:31.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7397999999999994e-05 2021-03-15T22:22:31.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:31.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.036181+0000", 2021-03-15T22:22:31.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086821280000000008 2021-03-15T22:22:31.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.044864+0000", 2021-03-15T22:22:31.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4041000000000005e-05 2021-03-15T22:22:31.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:31.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.044938+0000", 2021-03-15T22:22:31.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067060700000000004 2021-03-15T22:22:31.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.045608+0000", 2021-03-15T22:22:31.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4339999999999999e-05 2021-03-15T22:22:31.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.045653+0000", 2021-03-15T22:22:31.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6947999999999997e-05 2021-03-15T22:22:31.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.045719+0000", 2021-03-15T22:22:31.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099394189999999997 2021-03-15T22:22:31.718 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.718 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.719 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.720 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:868 7.6 e152/91 7:63693c38:::smithi07694878-22:head v 152'176, mlcod=149'172)", 2021-03-15T22:22:31.720 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:07.032048+0000", 2021-03-15T22:22:31.720 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 84.339425199000004, 2021-03-15T22:22:31.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011856909000000001, 2021-03-15T22:22:31.721 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.721 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:31.721 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.032048+0000", 2021-03-15T22:22:31.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.032048+0000", 2021-03-15T22:22:31.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002353267 2021-03-15T22:22:31.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.034401+0000", 2021-03-15T22:22:31.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0580000000000004e-06 2021-03-15T22:22:31.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.034410+0000", 2021-03-15T22:22:31.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2496999999999997e-05 2021-03-15T22:22:31.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.034442+0000", 2021-03-15T22:22:31.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1139e-05 2021-03-15T22:22:31.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.034454+0000", 2021-03-15T22:22:31.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4712e-05 2021-03-15T22:22:31.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.034518+0000", 2021-03-15T22:22:31.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8148000000000001e-05 2021-03-15T22:22:31.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.034556+0000", 2021-03-15T22:22:31.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0093118790000000003 2021-03-15T22:22:31.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:31.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.043868+0000", 2021-03-15T22:22:31.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6208999999999998e-05 2021-03-15T22:22:31.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:07.043905+0000", 2021-03-15T22:22:31.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011856909000000001 2021-03-15T22:22:31.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.734 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.735 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1378 7.a 7:53dee572:::smithi07694878-35:head [copy-from ver 3] snapc 5b=[5a,56] ondisk+write+known_if_redirected e190)", 2021-03-15T22:22:31.736 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:45.023845+0000", 2021-03-15T22:22:31.736 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.347628168999996, 2021-03-15T22:22:31.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023497436, 2021-03-15T22:22:31.737 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.737 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:31.737 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.737 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.738 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.738 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1378 2021-03-15T22:22:31.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.738 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023845+0000", 2021-03-15T22:22:31.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023845+0000", 2021-03-15T22:22:31.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5330000000000002e-06 2021-03-15T22:22:31.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023850+0000", 2021-03-15T22:22:31.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6090000000000003e-06 2021-03-15T22:22:31.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023855+0000", 2021-03-15T22:22:31.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9403e-05 2021-03-15T22:22:31.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023874+0000", 2021-03-15T22:22:31.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6260000000000003e-06 2021-03-15T22:22:31.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023882+0000", 2021-03-15T22:22:31.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3671999999999997e-05 2021-03-15T22:22:31.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023935+0000", 2021-03-15T22:22:31.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2021-03-15T22:22:31.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:22:31.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023945+0000", 2021-03-15T22:22:31.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086676960000000008 2021-03-15T22:22:31.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.032613+0000", 2021-03-15T22:22:31.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2980999999999996e-05 2021-03-15T22:22:31.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.032706+0000", 2021-03-15T22:22:31.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080630400000000005 2021-03-15T22:22:31.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:22:31.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.040769+0000", 2021-03-15T22:22:31.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035132739999999998 2021-03-15T22:22:31.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:22:31.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.044282+0000", 2021-03-15T22:22:31.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002886647 2021-03-15T22:22:31.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047169+0000", 2021-03-15T22:22:31.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9790999999999998e-05 2021-03-15T22:22:31.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:31.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047219+0000", 2021-03-15T22:22:31.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9148999999999999e-05 2021-03-15T22:22:31.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:22:31.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047278+0000", 2021-03-15T22:22:31.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3989e-05 2021-03-15T22:22:31.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047342+0000", 2021-03-15T22:22:31.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023497436 2021-03-15T22:22:31.756 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.756 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.757 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1379 7.a 7:53dee572:::smithi07694878-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e190)", 2021-03-15T22:22:31.757 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:45.023893+0000", 2021-03-15T22:22:31.758 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.347580239000003, 2021-03-15T22:22:31.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023671298, 2021-03-15T22:22:31.758 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:22:31.758 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:22:31.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:22:31.759 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:22:31.760 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:31.760 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1379 2021-03-15T22:22:31.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.761 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:22:31.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:22:31.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023893+0000", 2021-03-15T22:22:31.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:22:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:22:31.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023893+0000", 2021-03-15T22:22:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4050000000000001e-06 2021-03-15T22:22:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:22:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023894+0000", 2021-03-15T22:22:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.657e-06 2021-03-15T22:22:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:22:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023896+0000", 2021-03-15T22:22:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6059999999999999e-06 2021-03-15T22:22:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:22:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023902+0000", 2021-03-15T22:22:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4329999999999997e-06 2021-03-15T22:22:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:22:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023907+0000", 2021-03-15T22:22:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6927000000000004e-05 2021-03-15T22:22:31.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023984+0000", 2021-03-15T22:22:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.563e-06 2021-03-15T22:22:31.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:22:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023987+0000", 2021-03-15T22:22:31.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0094145919999999994 2021-03-15T22:22:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.033402+0000", 2021-03-15T22:22:31.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8862999999999997e-05 2021-03-15T22:22:31.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:31.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.033441+0000", 2021-03-15T22:22:31.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010337974999999999 2021-03-15T22:22:31.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.043779+0000", 2021-03-15T22:22:31.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3967000000000003e-05 2021-03-15T22:22:31.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:31.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.043873+0000", 2021-03-15T22:22:31.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035103690000000002 2021-03-15T22:22:31.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:22:31.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047383+0000", 2021-03-15T22:22:31.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9901000000000005e-05 2021-03-15T22:22:31.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:22:31.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047473+0000", 2021-03-15T22:22:31.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1039999999999996e-05 2021-03-15T22:22:31.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:22:31.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:22:31.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:22:31.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047564+0000", 2021-03-15T22:22:31.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023671298 2021-03-15T22:22:31.779 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.779 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.779 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.779 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:22:31.779 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:22:31.780 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:22:31.780 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:31.781 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:31 smithi125 conmon[56739]: cluster 2021-03-15T22:22:29.689306+0000 osd.2 (osd.2) 2021-03-15T22:22:31.782 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:31 smithi125 conmon[56739]: 43 : cluster [DBG] 7.a scrub ok 2021-03-15T22:22:31.782 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:31 smithi125 conmon[56739]: cluster 2021-03-15T22:22:30.800419+0000 osd.0 (osd.0) 51 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:22:31.782 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:31 smithi125 conmon[56739]: cluster 2021-03-15T22:22:31.045351+0000 mon.a (mon.0 2021-03-15T22:22:31.782 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:31 smithi125 conmon[56739]: ) 887 : cluster [DBG] osdmap e236: 8 total, 8 up, 7 in 2021-03-15T22:22:31.803 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:31.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:31.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:31.906 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:31.928 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:31.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:31.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:32.030 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:32.053 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:32.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:32.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:32.133 INFO:tasks.rados.rados.0.smithi076.stdout:695: done (4 left) 2021-03-15T22:22:32.133 INFO:tasks.rados.rados.0.smithi076.stdout:696: write (excl) oid 9 current snap is 72 2021-03-15T22:22:32.134 INFO:tasks.rados.rados.0.smithi076.stdout:696: seq_num 269 ranges {521083=714336,1724715=555421} 2021-03-15T22:22:32.136 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.5 to 1 (10000) 2021-03-15T22:22:32.138 INFO:tasks.rados.rados.0.smithi076.stdout:696: writing smithi07694878-9 from 521083 to 1235419 tid 1 2021-03-15T22:22:32.145 INFO:tasks.rados.rados.0.smithi076.stdout:696: writing smithi07694878-9 from 1724715 to 2280136 tid 2 2021-03-15T22:22:32.145 INFO:tasks.rados.rados.0.smithi076.stdout:697: delete oid 31 current snap is 72 2021-03-15T22:22:32.145 INFO:tasks.rados.rados.0.smithi076.stdout:690: expect (ObjNum 198 snap 43 seq_num 198) 2021-03-15T22:22:32.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:32.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:32.178 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:32.178 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:32.256 INFO:tasks.rados.rados.0.smithi076.stdout:691: finishing write tid 1 to smithi07694878-22 2021-03-15T22:22:32.256 INFO:tasks.rados.rados.0.smithi076.stdout:693: expect (ObjNum 6 snap 0 seq_num 6) 2021-03-15T22:22:32.279 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:32.302 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:32.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:32.304 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:32.330 INFO:tasks.rados.rados.0.smithi076.stdout:691: finishing write tid 2 to smithi07694878-22 2021-03-15T22:22:32.330 INFO:tasks.rados.rados.0.smithi076.stdout:691: finishing write tid 3 to smithi07694878-22 2021-03-15T22:22:32.330 INFO:tasks.rados.rados.0.smithi076.stdout:691: finishing write tid 4 to smithi07694878-22 2021-03-15T22:22:32.331 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 326 (ObjNum 267 snap 72 seq_num 267) dirty exists 2021-03-15T22:22:32.331 INFO:tasks.rados.rados.0.smithi076.stdout:691: left oid 22 (ObjNum 267 snap 72 seq_num 267) 2021-03-15T22:22:32.331 INFO:tasks.rados.rados.0.smithi076.stdout:696: finishing write tid 1 to smithi07694878-9 2021-03-15T22:22:32.331 INFO:tasks.rados.rados.0.smithi076.stdout:696: finishing write tid 2 to smithi07694878-9 2021-03-15T22:22:32.332 INFO:tasks.rados.rados.0.smithi076.stdout:696: finishing write tid 3 to smithi07694878-9 2021-03-15T22:22:32.332 INFO:tasks.rados.rados.0.smithi076.stdout:696: finishing write tid 4 to smithi07694878-9 2021-03-15T22:22:32.332 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 329 (ObjNum 269 snap 72 seq_num 269) dirty exists 2021-03-15T22:22:32.332 INFO:tasks.rados.rados.0.smithi076.stdout:696: left oid 9 (ObjNum 269 snap 72 seq_num 269) 2021-03-15T22:22:32.333 INFO:tasks.rados.rados.0.smithi076.stdout:690: done (5 left) 2021-03-15T22:22:32.333 INFO:tasks.rados.rados.0.smithi076.stdout:691: done (4 left) 2021-03-15T22:22:32.333 INFO:tasks.rados.rados.0.smithi076.stdout:692: done (3 left) 2021-03-15T22:22:32.333 INFO:tasks.rados.rados.0.smithi076.stdout:693: done (2 left) 2021-03-15T22:22:32.333 INFO:tasks.rados.rados.0.smithi076.stdout:696: done (1 left) 2021-03-15T22:22:32.334 INFO:tasks.rados.rados.0.smithi076.stdout:697: done (0 left) 2021-03-15T22:22:32.334 INFO:tasks.rados.rados.0.smithi076.stdout:698: read oid 18 snap -1 2021-03-15T22:22:32.334 INFO:tasks.rados.rados.0.smithi076.stdout:698: expect (ObjNum 217 snap 50 seq_num 217) 2021-03-15T22:22:32.334 INFO:tasks.rados.rados.0.smithi076.stdout:699: write (excl) oid 7 current snap is 72 2021-03-15T22:22:32.335 INFO:tasks.rados.rados.0.smithi076.stdout:699: seq_num 271 ranges {554697=464834,1725326=559090} 2021-03-15T22:22:32.336 INFO:tasks.rados.rados.0.smithi076.stdout:699: writing smithi07694878-7 from 554697 to 1019531 tid 1 2021-03-15T22:22:32.343 INFO:tasks.rados.rados.0.smithi076.stdout:699: writing smithi07694878-7 from 1725326 to 2284416 tid 2 2021-03-15T22:22:32.343 INFO:tasks.rados.rados.0.smithi076.stdout:700: snap_remove snap 66 2021-03-15T22:22:32.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[56514]: cluster 2021-03-15T22:22:30.930248+0000 osd.1 (osd 2021-03-15T22:22:32.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[56514]: .1) 51 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:22:32.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[56514]: cluster 2021-03-15T22:22:31.010976+0000 mgr.y (mgr.14144) 474 : cluster [DBG] pgmap v480: 138 pgs: 138 active+clean; 113 MiB data, 426 MiB used, 625 GiB / 626 GiB avail; 2.4 MiB/s rd, 752 KiB/s wr, 7 op/s 2021-03-15T22:22:32.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[56514]: audit 2021-03-15T22:22:31.339647+0000 mon.a (mon.0) 888 : audit [INF] from='client.? 172.21.15.76:0/3436534215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:22:32.383 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[62156]: cluster 2021-03-15T22:22:30.930248 2021-03-15T22:22:32.383 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 51 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:22:32.384 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:22:32.384 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[62156]: -03-15T22:22:31.010976+0000 mgr.y (mgr.14144) 474 : cluster 2021-03-15T22:22:32.384 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[62156]: [DBG] pgmap v480: 138 pgs: 138 active+clean; 113 MiB data, 426 MiB used, 625 GiB / 626 GiB avail; 2.4 MiB/s rd, 752 KiB/s wr, 7 op/s 2021-03-15T22:22:32.384 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:22:32.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[62156]: T22:22:31.339647+0000 mon.a (mon 2021-03-15T22:22:32.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:32 smithi076 conmon[62156]: .0) 888 : audit [INF] from='client.? 172.21.15.76:0/3436534215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:22:32.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:32.426 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:32.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:32.427 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:32.528 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:32 smithi125 conmon[56739]: cluster 2021-03-15T22:22:30.930248+0000 osd.1 (osd.1) 51 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:22:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:32 smithi125 conmon[56739]: 2021-03-15T22:22:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:32 smithi125 conmon[56739]: cluster 2021-03-15T22:22:31.010976+0000 mgr.y (mgr.14144) 474 : cluster [DBG] pgmap v480: 138 pgs: 138 active+clean; 113 MiB data, 426 MiB used, 625 GiB / 626 GiB avail; 2.4 MiB/s rd, 752 KiB/s wr, 7 op/s 2021-03-15T22:22:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:32 smithi125 conmon[56739]: audit 2021-03-15T22:22:31. 2021-03-15T22:22:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:32 smithi125 conmon[56739]: 339647+0000 mon.a (mon.0) 888 : audit [INF] from='client.? 172.21.15.76:0/3436534215' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:22:32.551 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:32.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:32.552 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:32.653 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:32.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:22:32.680 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:32.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:32.681 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:32.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-15T22:22:32.783 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:32.808 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:32.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:32.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:32.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:32.933 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:32.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:33.020 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:22:33.036 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:33.058 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:33.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:33.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:33.136 INFO:tasks.rados.rados.0.smithi076.stdout:700: done (2 left) 2021-03-15T22:22:33.137 INFO:tasks.rados.rados.0.smithi076.stdout:701: write (excl) oid 45 current snap is 72 2021-03-15T22:22:33.137 INFO:tasks.rados.rados.0.smithi076.stdout:701: seq_num 272 ranges {571504=540083,1783808=702748} 2021-03-15T22:22:33.143 INFO:tasks.rados.rados.0.smithi076.stdout:701: writing smithi07694878-45 from 571504 to 1111587 tid 1 2021-03-15T22:22:33.151 INFO:tasks.rados.rados.0.smithi076.stdout:701: writing smithi07694878-45 from 1783808 to 2486556 tid 2 2021-03-15T22:22:33.152 INFO:tasks.rados.rados.0.smithi076.stdout:702: read oid 17 snap 69 2021-03-15T22:22:33.152 INFO:tasks.rados.rados.0.smithi076.stdout:702: expect (ObjNum 220 snap 50 seq_num 220) 2021-03-15T22:22:33.152 INFO:tasks.rados.rados.0.smithi076.stdout:703: write (excl) oid 19 current snap is 72 2021-03-15T22:22:33.152 INFO:tasks.rados.rados.0.smithi076.stdout:703: seq_num 273 ranges {588311=615332,1842290=562759} 2021-03-15T22:22:33.158 INFO:tasks.rados.rados.0.smithi076.stdout:703: writing smithi07694878-19 from 588311 to 1203643 tid 1 2021-03-15T22:22:33.160 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:33.165 INFO:tasks.rados.rados.0.smithi076.stdout:703: writing smithi07694878-19 from 1842290 to 2405049 tid 2 2021-03-15T22:22:33.165 INFO:tasks.rados.rados.0.smithi076.stdout:704: write (excl) oid 3 current snap is 72 2021-03-15T22:22:33.165 INFO:tasks.rados.rados.0.smithi076.stdout:704: seq_num 274 ranges {605118=406934,1617125=706417} 2021-03-15T22:22:33.170 INFO:tasks.rados.rados.0.smithi076.stdout:704: writing smithi07694878-3 from 605118 to 1012052 tid 1 2021-03-15T22:22:33.178 INFO:tasks.rados.rados.0.smithi076.stdout:704: writing smithi07694878-3 from 1617125 to 2323542 tid 2 2021-03-15T22:22:33.178 INFO:tasks.rados.rados.0.smithi076.stdout:705: snap_create 2021-03-15T22:22:33.178 INFO:tasks.rados.rados.0.smithi076.stdout:698: expect (ObjNum 217 snap 50 seq_num 217) 2021-03-15T22:22:33.183 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:33.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:33.184 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:33.274 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 6 1 2021-03-15T22:22:33.285 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:33.308 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:33.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:33.309 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:33.355 INFO:tasks.rados.rados.0.smithi076.stdout:699: finishing write tid 1 to smithi07694878-7 2021-03-15T22:22:33.355 INFO:tasks.rados.rados.0.smithi076.stdout:699: finishing write tid 2 to smithi07694878-7 2021-03-15T22:22:33.356 INFO:tasks.rados.rados.0.smithi076.stdout:699: finishing write tid 3 to smithi07694878-7 2021-03-15T22:22:33.356 INFO:tasks.rados.rados.0.smithi076.stdout:699: finishing write tid 4 to smithi07694878-7 2021-03-15T22:22:33.356 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 262 (ObjNum 271 snap 72 seq_num 271) dirty exists 2021-03-15T22:22:33.356 INFO:tasks.rados.rados.0.smithi076.stdout:699: left oid 7 (ObjNum 271 snap 72 seq_num 271) 2021-03-15T22:22:33.357 INFO:tasks.rados.rados.0.smithi076.stdout:701: finishing write tid 1 to smithi07694878-45 2021-03-15T22:22:33.357 INFO:tasks.rados.rados.0.smithi076.stdout:701: finishing write tid 2 to smithi07694878-45 2021-03-15T22:22:33.357 INFO:tasks.rados.rados.0.smithi076.stdout:701: finishing write tid 3 to smithi07694878-45 2021-03-15T22:22:33.357 INFO:tasks.rados.rados.0.smithi076.stdout:701: finishing write tid 4 to smithi07694878-45 2021-03-15T22:22:33.358 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 248 (ObjNum 272 snap 72 seq_num 272) dirty exists 2021-03-15T22:22:33.358 INFO:tasks.rados.rados.0.smithi076.stdout:701: left oid 45 (ObjNum 272 snap 72 seq_num 272) 2021-03-15T22:22:33.358 INFO:tasks.rados.rados.0.smithi076.stdout:703: finishing write tid 1 to smithi07694878-19 2021-03-15T22:22:33.358 INFO:tasks.rados.rados.0.smithi076.stdout:702: expect (ObjNum 220 snap 50 seq_num 220) 2021-03-15T22:22:33.408 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: cluster 2021-03-15T22:22:31.635827+0000 osd.6 2021-03-15T22:22:33.408 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: (osd.6) 33 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:22:33.409 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: cluster 2021-03-15T22:22:31.872454+0000 osd.4 (osd.4 2021-03-15T22:22:33.409 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: ) 57 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:22:33.409 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:22:33.409 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: 15T22:22:32.129712+0000 2021-03-15T22:22:33.410 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: mon.a (mon.0) 889 2021-03-15T22:22:33.410 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: : audit [INF] from='client.? 172.21.15.76:0/3436534215' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-15T22:22:33.410 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: cluster 2021-03-15T22:22:33.410 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: 2021-03-15 2021-03-15T22:22:33.411 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: T22:22:32.129773 2021-03-15T22:22:33.411 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2021-03-15T22:22:33.411 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: 890 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2021-03-15T22:22:33.411 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:22:33.411 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: -15T22:22:32.668822 2021-03-15T22:22:33.412 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: +0000 osd.2 (osd.2) 2021-03-15T22:22:33.412 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:33 smithi125 conmon[56739]: 44 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:22:33.412 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:33.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:33 smithi076 conmon[62156]: cluster 2021-03-15T22:22:31.635827+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:22:33.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:33 smithi076 conmon[62156]: cluster 2021-03-15T22:22:31.872454+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:22:33.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:33 smithi076 conmon[62156]: audit 2021-03-15T22:22:32.129712+0000 mon.a (mon.0) 889 : audit [INF] from='client.? 172.21.15.76:0/3436534215' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-15T22:22:33.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:33 smithi076 conmon[62156]: cluster 2021-03-15T22:22:32.129773+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2021-03-15T22:22:33.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:33 smithi076 conmon[62156]: cluster 2021-03-15T22:22:32.668822+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:22:33.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:33 smithi076 conmon[56514]: cluster 2021-03-15T22:22:31.635827+0000 osd.6 (osd.6) 33 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:22:33.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:33 smithi076 conmon[56514]: cluster 2021-03-15T22:22:31.872454+0000 osd.4 (osd.4) 57 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:22:33.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:33 smithi076 conmon[56514]: audit 2021-03-15T22:22:32.129712+0000 mon.a (mon.0) 889 : audit [INF] from='client.? 172.21.15.76:0/3436534215' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-15T22:22:33.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:33 smithi076 conmon[56514]: cluster 2021-03-15T22:22:32.129773+0000 mon.a (mon.0) 890 : cluster [DBG] osdmap e237: 8 total, 8 up, 7 in 2021-03-15T22:22:33.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:33 smithi076 conmon[56514]: cluster 2021-03-15T22:22:32.668822+0000 osd.2 (osd.2) 44 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:22:33.420 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:33.444 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:33.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:33.445 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:33.546 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:33.568 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:33.569 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:33.583 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:33.628 INFO:tasks.rados.rados.0.smithi076.stdout:703: finishing write tid 2 to smithi07694878-19 2021-03-15T22:22:33.628 INFO:tasks.rados.rados.0.smithi076.stdout:703: finishing write tid 3 to smithi07694878-19 2021-03-15T22:22:33.628 INFO:tasks.rados.rados.0.smithi076.stdout:703: finishing write tid 4 to smithi07694878-19 2021-03-15T22:22:33.628 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 270 (ObjNum 273 snap 72 seq_num 273) dirty exists 2021-03-15T22:22:33.629 INFO:tasks.rados.rados.0.smithi076.stdout:703: left oid 19 (ObjNum 273 snap 72 seq_num 273) 2021-03-15T22:22:33.629 INFO:tasks.rados.rados.0.smithi076.stdout:704: finishing write tid 1 to smithi07694878-3 2021-03-15T22:22:33.629 INFO:tasks.rados.rados.0.smithi076.stdout:704: finishing write tid 2 to smithi07694878-3 2021-03-15T22:22:33.629 INFO:tasks.rados.rados.0.smithi076.stdout:704: finishing write tid 3 to smithi07694878-3 2021-03-15T22:22:33.630 INFO:tasks.rados.rados.0.smithi076.stdout:704: finishing write tid 4 to smithi07694878-3 2021-03-15T22:22:33.630 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 245 (ObjNum 274 snap 72 seq_num 274) dirty exists 2021-03-15T22:22:33.630 INFO:tasks.rados.rados.0.smithi076.stdout:704: left oid 3 (ObjNum 274 snap 72 seq_num 274) 2021-03-15T22:22:33.671 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:33.692 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:33.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:33.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:33.795 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:33.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:33.820 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:33.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:33.922 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:33.945 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:33.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:34.047 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:34.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:34.072 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:34.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:34.147 INFO:tasks.rados.rados.0.smithi076.stdout:698: done (6 left) 2021-03-15T22:22:34.147 INFO:tasks.rados.rados.0.smithi076.stdout:699: done (5 left) 2021-03-15T22:22:34.147 INFO:tasks.rados.rados.0.smithi076.stdout:701: done (4 left) 2021-03-15T22:22:34.148 INFO:tasks.rados.rados.0.smithi076.stdout:702: done (3 left) 2021-03-15T22:22:34.148 INFO:tasks.rados.rados.0.smithi076.stdout:703: done (2 left) 2021-03-15T22:22:34.148 INFO:tasks.rados.rados.0.smithi076.stdout:704: done (1 left) 2021-03-15T22:22:34.149 INFO:tasks.rados.rados.0.smithi076.stdout:705: done (0 left) 2021-03-15T22:22:34.149 INFO:tasks.rados.rados.0.smithi076.stdout:706: write oid 14 current snap is 73 2021-03-15T22:22:34.149 INFO:tasks.rados.rados.0.smithi076.stdout:706: seq_num 275 ranges {621925=482183,1559254=566428} 2021-03-15T22:22:34.152 INFO:tasks.rados.rados.0.smithi076.stdout:706: writing smithi07694878-14 from 621925 to 1104108 tid 1 2021-03-15T22:22:34.159 INFO:tasks.rados.rados.0.smithi076.stdout:706: writing smithi07694878-14 from 1559254 to 2125682 tid 2 2021-03-15T22:22:34.159 INFO:tasks.rados.rados.0.smithi076.stdout:707: read oid 32 snap -1 2021-03-15T22:22:34.159 INFO:tasks.rados.rados.0.smithi076.stdout:707: expect (ObjNum 265 snap 72 seq_num 265) 2021-03-15T22:22:34.160 INFO:tasks.rados.rados.0.smithi076.stdout:708: read oid 39 snap -1 2021-03-15T22:22:34.160 INFO:tasks.rados.rados.0.smithi076.stdout:708: expect (ObjNum 199 snap 43 seq_num 199) 2021-03-15T22:22:34.160 INFO:tasks.rados.rados.0.smithi076.stdout:709: delete oid 26 current snap is 73 2021-03-15T22:22:34.160 INFO:tasks.rados.rados.0.smithi076.stdout:706: finishing write tid 1 to smithi07694878-14 2021-03-15T22:22:34.164 INFO:tasks.rados.rados.0.smithi076.stdout:708: expect (ObjNum 199 snap 43 seq_num 199) 2021-03-15T22:22:34.174 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:34.195 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:34.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:34.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:34.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:34.317 INFO:tasks.rados.rados.0.smithi076.stdout:706: finishing write tid 2 to smithi07694878-14 2021-03-15T22:22:34.317 INFO:tasks.rados.rados.0.smithi076.stdout:706: finishing write tid 3 to smithi07694878-14 2021-03-15T22:22:34.317 INFO:tasks.rados.rados.0.smithi076.stdout:706: finishing write tid 4 to smithi07694878-14 2021-03-15T22:22:34.318 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 248 (ObjNum 275 snap 73 seq_num 275) dirty exists 2021-03-15T22:22:34.318 INFO:tasks.rados.rados.0.smithi076.stdout:706: left oid 14 (ObjNum 275 snap 73 seq_num 275) 2021-03-15T22:22:34.318 INFO:tasks.rados.rados.0.smithi076.stdout:707: expect (ObjNum 265 snap 72 seq_num 265) 2021-03-15T22:22:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:34.322 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:34.322 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:34.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[56514]: cluster 2021-03-15T22:22:32 2021-03-15T22:22:34.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[56514]: .631426+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:22:34.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[56514]: cluster 2021-03-15T22:22:33.011432+0000 mgr.y (mgr.14144) 475 : cluster [DBG] pgmap v483: 138 pgs: 138 active+clean; 112 MiB data, 426 MiB used, 625 GiB / 626 GiB avail; 7.7 MiB/s rd, 543 KiB/s wr, 11 op/s 2021-03-15T22:22:34.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[56514]: cluster 2021-03-15T22:22:33.106086+0000 osd.6 (osd.6) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:34.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[56514]: cluster 2021-03-15T22:22:33.106213+0000 osd.6 (osd.6) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:34.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[56514]: cluster 2021-03-15T22:22:33.134112+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e238: 8 total, 8 up, 7 in 2021-03-15T22:22:34.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: cluster 2021-03-15T22:22:32.631426+0000 osd.6 (osd 2021-03-15T22:22:34.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: .6) 34 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:22:34.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: cluster 2021-03-15T22:22:33.011432+0000 mgr.y (mgr.14144) 475 : cluster 2021-03-15T22:22:34.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: [DBG] pgmap v483: 138 pgs: 138 active+clean; 112 MiB data, 426 MiB used, 625 GiB / 626 GiB avail; 7.7 MiB/s rd, 543 KiB/s wr, 11 op/s 2021-03-15T22:22:34.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: cluster 2021-03-15T22:22:33.106086+0000 2021-03-15T22:22:34.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: osd.6 (osd.6) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:34.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: cluster 2021-03-15T22:22:34.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: 2021-03-15T22:22:33.106213+0000 osd.6 (osd.6) 36 : cluster [DBG] 2021-03-15T22:22:34.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: purged_snaps scrub ok 2021-03-15T22:22:34.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: cluster 2021-03-15T22:22:34.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: 2021-03-15T22:22:33.134112+0000 mon.a (mon.0) 891 : cluster 2021-03-15T22:22:34.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:34 smithi076 conmon[62156]: [DBG] osdmap e238: 8 total, 8 up, 7 in 2021-03-15T22:22:34.423 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:34.438 INFO:tasks.rados.rados.0.smithi076.stdout:706: done (3 left) 2021-03-15T22:22:34.438 INFO:tasks.rados.rados.0.smithi076.stdout:707: done (2 left) 2021-03-15T22:22:34.438 INFO:tasks.rados.rados.0.smithi076.stdout:708: done (1 left) 2021-03-15T22:22:34.438 INFO:tasks.rados.rados.0.smithi076.stdout:709: done (0 left) 2021-03-15T22:22:34.439 INFO:tasks.rados.rados.0.smithi076.stdout:710: read oid 11 snap 72 2021-03-15T22:22:34.439 INFO:tasks.rados.rados.0.smithi076.stdout:710: expect deleted 2021-03-15T22:22:34.439 INFO:tasks.rados.rados.0.smithi076.stdout:711: read oid 7 snap -1 2021-03-15T22:22:34.440 INFO:tasks.rados.rados.0.smithi076.stdout:711: expect (ObjNum 271 snap 72 seq_num 271) 2021-03-15T22:22:34.440 INFO:tasks.rados.rados.0.smithi076.stdout:712: snap_create 2021-03-15T22:22:34.441 INFO:tasks.rados.rados.0.smithi076.stdout:711: expect (ObjNum 271 snap 72 seq_num 271) 2021-03-15T22:22:34.446 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:34.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:34.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:34.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:34 smithi125 conmon[56739]: cluster 2021-03-15T22:22:32.631426+0000 osd.6 (osd.6) 34 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:22:34.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:34 smithi125 conmon[56739]: cluster 2021-03-15T22:22:33.011432+0000 mgr.y (mgr.14144) 475 : cluster [DBG] pgmap v483: 138 pgs: 138 active+clean; 112 MiB data, 426 MiB used, 625 GiB / 626 GiB avail; 7.7 MiB/s rd, 543 KiB/s wr, 11 op/s 2021-03-15T22:22:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:34 smithi125 conmon[56739]: cluster 2021-03-15T22:22:33.106086+0000 osd.6 (osd.6) 35 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:34 smithi125 conmon[56739]: cluster 2021-03-15T22:22:33.106213+0000 osd.6 (osd.6) 36 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:34 smithi125 conmon[56739]: cluster 2021-03-15T22:22:33.134112+0000 mon.a (mon.0) 891 : cluster [DBG] osdmap e238: 8 total, 8 up, 7 in 2021-03-15T22:22:34.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:34.571 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:34.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:34.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:34.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:34.695 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:34.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:34.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:34.798 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:34.804 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:22:34.805 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:22:34.805 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:22:34.806 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:22:34.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:34.822 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:34.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:34.924 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:34.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:34.949 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:34.949 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:35.047 INFO:tasks.rados.rados.0.smithi076.stdout:710: done (2 left) 2021-03-15T22:22:35.048 INFO:tasks.rados.rados.0.smithi076.stdout:711: done (1 left) 2021-03-15T22:22:35.048 INFO:tasks.rados.rados.0.smithi076.stdout:712: done (0 left) 2021-03-15T22:22:35.048 INFO:tasks.rados.rados.0.smithi076.stdout:713: read oid 18 snap -1 2021-03-15T22:22:35.048 INFO:tasks.rados.rados.0.smithi076.stdout:713: expect (ObjNum 217 snap 50 seq_num 217) 2021-03-15T22:22:35.049 INFO:tasks.rados.rados.0.smithi076.stdout:714: write oid 16 current snap is 74 2021-03-15T22:22:35.049 INFO:tasks.rados.rados.0.smithi076.stdout:714: seq_num 277 ranges {655539=632681,2076218=570097} 2021-03-15T22:22:35.049 INFO:tasks.rados.rados.0.smithi076.stdout:714: writing smithi07694878-16 from 655539 to 1288220 tid 1 2021-03-15T22:22:35.049 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:22:35.050 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:35.056 INFO:tasks.rados.rados.0.smithi076.stdout:714: writing smithi07694878-16 from 2076218 to 2646315 tid 2 2021-03-15T22:22:35.056 INFO:tasks.rados.rados.0.smithi076.stdout:715: write oid 23 current snap is 74 2021-03-15T22:22:35.056 INFO:tasks.rados.rados.0.smithi076.stdout:715: seq_num 278 ranges {672346=707930,2134700=537646} 2021-03-15T22:22:35.063 INFO:tasks.rados.rados.0.smithi076.stdout:715: writing smithi07694878-23 from 672346 to 1380276 tid 1 2021-03-15T22:22:35.071 INFO:tasks.rados.rados.0.smithi076.stdout:715: writing smithi07694878-23 from 2134700 to 2672346 tid 2 2021-03-15T22:22:35.071 INFO:tasks.rados.rados.0.smithi076.stdout:716: delete oid 19 current snap is 74 2021-03-15T22:22:35.071 INFO:tasks.rados.rados.0.smithi076.stdout:713: expect (ObjNum 217 snap 50 seq_num 217) 2021-03-15T22:22:35.074 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:35.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:35.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:35.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:35.201 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:35.203 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:35.248 INFO:tasks.rados.rados.0.smithi076.stdout:714: finishing write tid 1 to smithi07694878-16 2021-03-15T22:22:35.248 INFO:tasks.rados.rados.0.smithi076.stdout:714: finishing write tid 2 to smithi07694878-16 2021-03-15T22:22:35.248 INFO:tasks.rados.rados.0.smithi076.stdout:714: finishing write tid 3 to smithi07694878-16 2021-03-15T22:22:35.249 INFO:tasks.rados.rados.0.smithi076.stdout:714: finishing write tid 4 to smithi07694878-16 2021-03-15T22:22:35.249 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 336 (ObjNum 277 snap 74 seq_num 277) dirty exists 2021-03-15T22:22:35.249 INFO:tasks.rados.rados.0.smithi076.stdout:714: left oid 16 (ObjNum 277 snap 74 seq_num 277) 2021-03-15T22:22:35.249 INFO:tasks.rados.rados.0.smithi076.stdout:715: finishing write tid 1 to smithi07694878-23 2021-03-15T22:22:35.250 INFO:tasks.rados.rados.0.smithi076.stdout:715: finishing write tid 2 to smithi07694878-23 2021-03-15T22:22:35.250 INFO:tasks.rados.rados.0.smithi076.stdout:713: done (3 left) 2021-03-15T22:22:35.250 INFO:tasks.rados.rados.0.smithi076.stdout:714: done (2 left) 2021-03-15T22:22:35.250 INFO:tasks.rados.rados.0.smithi076.stdout:716: done (1 left) 2021-03-15T22:22:35.250 INFO:tasks.rados.rados.0.smithi076.stdout:717: read oid 18 snap -1 2021-03-15T22:22:35.251 INFO:tasks.rados.rados.0.smithi076.stdout:717: expect (ObjNum 217 snap 50 seq_num 217) 2021-03-15T22:22:35.251 INFO:tasks.rados.rados.0.smithi076.stdout:718: snap_remove snap 73 2021-03-15T22:22:35.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:35.329 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:35.329 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:35.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:35.452 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:35.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:35.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:35.466 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:35 smithi125 conmon[56739]: cluster 2021-03-15T22:22:33.947353+0000 osd.1 (osd.1) 52 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:22:35.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:35 smithi125 conmon[56739]: cluster 2021-03-15T22:22:35.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:35 smithi125 conmon[56739]: 2021-03-15T22:22:34.142357+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e239: 8 total, 8 up, 7 in 2021-03-15T22:22:35.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:35 smithi125 conmon[56739]: cluster 2021-03-15T22:22:34.656436+0000 osd.2 (osd.2) 45 : cluster [DBG] 5.b scrub ok 2021-03-15T22:22:35.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:35 smithi125 conmon[56739]: audit 2021-03-15T22:22:34.803673+0000 mon.a (mon.0) 893 : audit [INF] from='client.? 172.21.15.76:0/1646938363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:35.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:35 smithi125 conmon[56739]: cluster 2021-03-15T22:22:35.030570+0000 mon.a (mon.0) 894 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:22:35.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:35 smithi125 conmon[56739]: cluster 2021-03-15T22:22:35.030593+0000 mon.a (mon.0) 895 : cluster [INF] Cluster is now healthy 2021-03-15T22:22:35.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:35 smithi125 conmon[56739]: audit 2021-03-15T22:22:35.036285+0000 mon.a (mon.0) 896 : audit [INF] from='client.? 172.21.15.76:0/1646938363' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:22:35.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:35 smithi125 conmon[56739]: cluster 2021-03-15T22:22:35.036354+0000 mon.a (mon.0) 897 : cluster [DBG] osdmap e240: 8 total, 8 up, 7 in 2021-03-15T22:22:35.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-15T22:22:35.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[56514]: cluster 2021-03-15T22:22:33.947353+0000 osd.1 (osd.1) 52 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:22:35.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[56514]: cluster 2021-03-15T22:22:34.142357+0000 mon.a (mon.0) 892 : cluster [DBG] osdmap e239: 8 total, 8 up, 7 in 2021-03-15T22:22:35.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[56514]: cluster 2021-03-15T22:22:34.656436+0000 osd.2 (osd.2) 45 : cluster [DBG] 5.b scrub ok 2021-03-15T22:22:35.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[56514]: audit 2021-03-15T22:22:34.803673+0000 mon.a (mon.0) 893 : audit [INF] from='client.? 172.21.15.76:0/1646938363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:35.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[56514]: cluster 2021-03-15T22:22:35.030570+0000 mon.a (mon.0) 894 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:22:35.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[56514]: cluster 2021-03-15T22:22:35.030593+0000 mon.a (mon.0) 895 : cluster [INF] Cluster is now healthy 2021-03-15T22:22:35.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[56514]: audit 2021-03-15T22:22:35.036285+0000 mon.a (mon.0) 896 : audit [INF] from='client.? 172.21.15.76:0/1646938363' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:22:35.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[56514]: cluster 2021-03-15T22:22:35.036354+0000 mon.a (mon.0) 897 : cluster [DBG] osdmap e240: 8 total, 8 up, 7 in 2021-03-15T22:22:35.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: cluster 2021-03-15T22:22: 2021-03-15T22:22:35.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: 33.947353+0000 osd.1 (osd.1) 52 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:22:35.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:22:35.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: -15T22:22:34.142357+0000 mon.a (mon.0) 892 : cluster 2021-03-15T22:22:35.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: [DBG] osdmap e239: 8 total, 8 up, 7 in 2021-03-15T22:22:35.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:22:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: -15T22:22:34.656436 2021-03-15T22:22:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: +0000 osd.2 (osd.2) 45 : cluster 2021-03-15T22:22:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: [DBG] 5.b scrub ok 2021-03-15T22:22:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: audit 2021 2021-03-15T22:22:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: -03-15T22:22:34.803673 2021-03-15T22:22:35.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: +0000 mon.a (mon.0) 893 : 2021-03-15T22:22:35.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/1646938363' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:35.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:22:35.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: -03-15T22:22:35. 2021-03-15T22:22:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: 030570+0000 mon.a (mon.0) 894 : 2021-03-15T22:22:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:22:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:22:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: 03-15T22:22:35.030593+0000 2021-03-15T22:22:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: mon.a (mon.0) 895 : cluster 2021-03-15T22:22:35.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: [INF] Cluster is now healthy 2021-03-15T22:22:35.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: audit 2021-03-15T22:22:35.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: 2021-03-15T22:22: 2021-03-15T22:22:35.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: 35.036285+0000 mon.a (mon.0 2021-03-15T22:22:35.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: ) 896 : audit [INF] from='client.? 172.21.15.76:0/1646938363' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:22:35.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:22:35.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: -15T22:22:35.036354+0000 2021-03-15T22:22:35.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: mon.a (mon.0) 897 : cluster [DBG] 2021-03-15T22:22:35.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:35 smithi076 conmon[62156]: osdmap e240: 8 total, 8 up, 7 in 2021-03-15T22:22:35.555 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:35.578 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:35.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:35.580 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:35.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:35.705 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:35.705 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:35.801 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:22:35.806 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:35.829 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:35.830 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:35.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:35.953 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:35.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:35.954 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:36.055 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:36.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:36.078 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:36.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:36.160 INFO:tasks.rados.rados.0.smithi076.stdout:718: done (2 left) 2021-03-15T22:22:36.161 INFO:tasks.rados.rados.0.smithi076.stdout:719: snap_remove snap 69 2021-03-15T22:22:36.179 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:36.202 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:36.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:36.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:36.253 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:36 smithi125 conmon[56739]: cluster 2021-03-15T22:22:34.938476+0000 osd.1 2021-03-15T22:22:36.253 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:36 smithi125 conmon[56739]: (osd.1) 53 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:22:36.253 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:36 smithi125 conmon[56739]: cluster 2021-03-15T22:22:35.011869+0000 mgr.y (mgr.14144) 476 : cluster [DBG] pgmap v486: 138 pgs: 138 active+clean; 121 MiB data, 438 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 19 op/s 2021-03-15T22:22:36.254 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:36 smithi125 conmon[56739]: cluster 2021-03-15T22:22:35.649327+0000 osd.2 (osd.2) 46 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:22:36.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:36.327 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:36.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:36.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:36.356 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:36 smithi076 conmon[56514]: cluster 2021-03-15T22:22:34.938476+0000 2021-03-15T22:22:36.357 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:36 smithi076 conmon[56514]: osd.1 (osd.1) 53 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:22:36.357 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:36 smithi076 conmon[56514]: cluster 2021-03-15T22:22:35.011869+0000 mgr.y (mgr.14144) 476 : cluster [DBG] pgmap v486: 138 pgs: 138 active+clean; 121 MiB data, 438 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 19 op/s 2021-03-15T22:22:36.357 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:36 smithi076 conmon[56514]: cluster 2021-03-15T22:22:35.649327+0000 osd.2 (osd.2) 46 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:22:36.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:36 smithi076 conmon[62156]: cluster 2021-03-15T22:22:34.938476+0000 osd.1 ( 2021-03-15T22:22:36.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:36 smithi076 conmon[62156]: osd.1) 53 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:22:36.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:36 smithi076 conmon[62156]: cluster 2021-03-15T22:22:35.011869+0000 mgr.y (mgr.14144 2021-03-15T22:22:36.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:36 smithi076 conmon[62156]: ) 476 : cluster [DBG] pgmap v486: 138 pgs: 138 active+clean; 121 MiB data, 438 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 19 op/s 2021-03-15T22:22:36.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:36 smithi076 conmon[62156]: cluster 2021-03-15T22:22:36.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:36 smithi076 conmon[62156]: 2021-03-15T22:22:35.649327+0000 osd.2 (osd.2 2021-03-15T22:22:36.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:36 smithi076 conmon[62156]: ) 46 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:22:36.407 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:22:36.423 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:22:36.429 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:36.442 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:22:36.452 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:36.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:36.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:36.463 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:22:36.482 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:22:36.501 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:22:36.519 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:22:36.537 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:22:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:22:36.538 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:22:36.539 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:22:36.539 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:22:36.539 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:22:36.539 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:22:36.540 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:22:36.540 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:22:36.556 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:36.580 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:36.582 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:36.683 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:36.705 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:36.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:36.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:36.808 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:36.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:36.831 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:36.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:36.933 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:36.955 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:36.957 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:37.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:37.080 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:37.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:37.082 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:37.168 INFO:tasks.rados.rados.0.smithi076.stdout:719: done (2 left) 2021-03-15T22:22:37.168 INFO:tasks.rados.rados.0.smithi076.stdout:720: rollback oid 39 current snap is 74 2021-03-15T22:22:37.169 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 39 to 72 2021-03-15T22:22:37.169 INFO:tasks.rados.rados.0.smithi076.stdout:721: write oid 1 current snap is 74 2021-03-15T22:22:37.169 INFO:tasks.rados.rados.0.smithi076.stdout:721: seq_num 280 ranges {705960=458428,1735311=717424} 2021-03-15T22:22:37.170 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.6 to 1 (10000) 2021-03-15T22:22:37.174 INFO:tasks.rados.rados.0.smithi076.stdout:721: writing smithi07694878-1 from 705960 to 1164388 tid 1 2021-03-15T22:22:37.182 INFO:tasks.rados.rados.0.smithi076.stdout:721: writing smithi07694878-1 from 1735311 to 2452735 tid 2 2021-03-15T22:22:37.182 INFO:tasks.rados.rados.0.smithi076.stdout:722: write (excl) oid 37 current snap is 74 2021-03-15T22:22:37.183 INFO:tasks.rados.rados.0.smithi076.stdout:715: finishing write tid 3 to smithi07694878-23 2021-03-15T22:22:37.183 INFO:tasks.rados.rados.0.smithi076.stdout:722: seq_num 281 ranges {722767=533677,1793793=461082} 2021-03-15T22:22:37.183 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:37.189 INFO:tasks.rados.rados.0.smithi076.stdout:722: writing smithi07694878-37 from 722767 to 1256444 tid 1 2021-03-15T22:22:37.195 INFO:tasks.rados.rados.0.smithi076.stdout:722: writing smithi07694878-37 from 1793793 to 2254875 tid 2 2021-03-15T22:22:37.195 INFO:tasks.rados.rados.0.smithi076.stdout:723: delete oid 2 current snap is 74 2021-03-15T22:22:37.196 INFO:tasks.rados.rados.0.smithi076.stdout:715: finishing write tid 4 to smithi07694878-23 2021-03-15T22:22:37.196 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 121 (ObjNum 278 snap 74 seq_num 278) dirty exists 2021-03-15T22:22:37.196 INFO:tasks.rados.rados.0.smithi076.stdout:715: left oid 23 (ObjNum 278 snap 74 seq_num 278) 2021-03-15T22:22:37.196 INFO:tasks.rados.rados.0.smithi076.stdout:717: expect (ObjNum 217 snap 50 seq_num 217) 2021-03-15T22:22:37.205 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:37.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:37.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:37.271 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:22:37.271 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:22:37.308 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:37.325 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:37 smithi125 conmon[56739]: cluster 2021-03-15T22:22:35.877567+0000 osd.4 (osd.4) 58 : cluster [DBG] 6.e scrub ok 2021-03-15T22:22:37.325 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:37 smithi125 conmon[56739]: cluster 2021-03-15T22:22:36.156191+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2021-03-15T22:22:37.326 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:37 smithi125 conmon[56739]: audit 2021-03-15T22:22:36.387442+0000 mon.a (mon.0) 899 : audit [DBG] from='client.? 172.21.15.76:0/30590070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:37.326 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:37 smithi125 conmon[56739]: audit 2021-03-15T22:22:36.846597+0000 mon.a (mon.0) 900 : audit [INF] from='client.? 172.21.15.76:0/185486720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-15T22:22:37.332 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:37.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:37.364 INFO:tasks.rados.rados.0.smithi076.stdout:720: finishing rollback tid 0 to smithi07694878-39 2021-03-15T22:22:37.365 INFO:tasks.rados.rados.0.smithi076.stdout:720: finishing rollback tid 1 to smithi07694878-39 2021-03-15T22:22:37.365 INFO:tasks.rados.rados.0.smithi076.stdout:720: finishing rollback tid 2 to smithi07694878-39 2021-03-15T22:22:37.365 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 282 (ObjNum 199 snap 43 seq_num 199) dirty exists 2021-03-15T22:22:37.365 INFO:tasks.rados.rados.0.smithi076.stdout:721: finishing write tid 1 to smithi07694878-1 2021-03-15T22:22:37.366 INFO:tasks.rados.rados.0.smithi076.stdout:715: done (5 left) 2021-03-15T22:22:37.366 INFO:tasks.rados.rados.0.smithi076.stdout:717: done (4 left) 2021-03-15T22:22:37.366 INFO:tasks.rados.rados.0.smithi076.stdout:720: done (3 left) 2021-03-15T22:22:37.366 INFO:tasks.rados.rados.0.smithi076.stdout:723: done (2 left) 2021-03-15T22:22:37.366 INFO:tasks.rados.rados.0.smithi076.stdout:724: write (excl) oid 18 current snap is 74 2021-03-15T22:22:37.367 INFO:tasks.rados.rados.0.smithi076.stdout:721: finishing write tid 2 to smithi07694878-1 2021-03-15T22:22:37.367 INFO:tasks.rados.rados.0.smithi076.stdout:721: finishing write tid 3 to smithi07694878-1 2021-03-15T22:22:37.367 INFO:tasks.rados.rados.0.smithi076.stdout:721: finishing write tid 4 to smithi07694878-1 2021-03-15T22:22:37.367 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 151 (ObjNum 280 snap 74 seq_num 280) dirty exists 2021-03-15T22:22:37.368 INFO:tasks.rados.rados.0.smithi076.stdout:721: left oid 1 (ObjNum 280 snap 74 seq_num 280) 2021-03-15T22:22:37.368 INFO:tasks.rados.rados.0.smithi076.stdout:722: finishing write tid 1 to smithi07694878-37 2021-03-15T22:22:37.368 INFO:tasks.rados.rados.0.smithi076.stdout:724: seq_num 283 ranges {756381=400528,1910757=464751} 2021-03-15T22:22:37.370 INFO:tasks.rados.rados.0.smithi076.stdout:724: writing smithi07694878-18 from 756381 to 1156909 tid 1 2021-03-15T22:22:37.378 INFO:tasks.rados.rados.0.smithi076.stdout:724: writing smithi07694878-18 from 1910757 to 2375508 tid 2 2021-03-15T22:22:37.378 INFO:tasks.rados.rados.0.smithi076.stdout:721: done (2 left) 2021-03-15T22:22:37.378 INFO:tasks.rados.rados.0.smithi076.stdout:725: read oid 27 snap 65 2021-03-15T22:22:37.379 INFO:tasks.rados.rados.0.smithi076.stdout:725: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:22:37.379 INFO:tasks.rados.rados.0.smithi076.stdout:726: read oid 17 snap -1 2021-03-15T22:22:37.379 INFO:tasks.rados.rados.0.smithi076.stdout:726: expect (ObjNum 220 snap 50 seq_num 220) 2021-03-15T22:22:37.379 INFO:tasks.rados.rados.0.smithi076.stdout:727: snap_create 2021-03-15T22:22:37.379 INFO:tasks.rados.rados.0.smithi076.stdout:722: finishing write tid 2 to smithi07694878-37 2021-03-15T22:22:37.380 INFO:tasks.rados.rados.0.smithi076.stdout:722: finishing write tid 3 to smithi07694878-37 2021-03-15T22:22:37.380 INFO:tasks.rados.rados.0.smithi076.stdout:722: finishing write tid 4 to smithi07694878-37 2021-03-15T22:22:37.380 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 424 (ObjNum 281 snap 74 seq_num 281) dirty exists 2021-03-15T22:22:37.380 INFO:tasks.rados.rados.0.smithi076.stdout:722: left oid 37 (ObjNum 281 snap 74 seq_num 281) 2021-03-15T22:22:37.381 INFO:tasks.rados.rados.0.smithi076.stdout:724: finishing write tid 1 to smithi07694878-18 2021-03-15T22:22:37.382 INFO:tasks.rados.rados.0.smithi076.stdout:724: finishing write tid 2 to smithi07694878-18 2021-03-15T22:22:37.382 INFO:tasks.rados.rados.0.smithi076.stdout:724: finishing write tid 3 to smithi07694878-18 2021-03-15T22:22:37.382 INFO:tasks.rados.rados.0.smithi076.stdout:724: finishing write tid 4 to smithi07694878-18 2021-03-15T22:22:37.382 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 129 (ObjNum 283 snap 74 seq_num 283) dirty exists 2021-03-15T22:22:37.383 INFO:tasks.rados.rados.0.smithi076.stdout:724: left oid 18 (ObjNum 283 snap 74 seq_num 283) 2021-03-15T22:22:37.383 INFO:tasks.rados.rados.0.smithi076.stdout:725: expect (ObjNum 101 snap 21 seq_num 101) 2021-03-15T22:22:37.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:37.456 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:37 smithi076 conmon[56514]: cluster 2021-03-15T22:22:35.877567+0000 osd.4 (osd.4) 58 : cluster [DBG] 6.e scrub ok 2021-03-15T22:22:37.457 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:37 smithi076 conmon[56514]: cluster 2021-03-15T22:22:36.156191+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2021-03-15T22:22:37.457 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:37 smithi076 conmon[56514]: audit 2021-03-15T22:22:36.387442+0000 mon.a (mon.0) 899 : audit [DBG] from='client.? 172.21.15.76:0/30590070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:37.457 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:37 smithi076 conmon[56514]: audit 2021-03-15T22:22:36.846597+0000 mon.a (mon.0) 900 : audit [INF] from='client.? 172.21.15.76:0/185486720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-15T22:22:37.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:37 smithi076 conmon[62156]: cluster 2021-03-15T22:22:35.877567+0000 osd.4 (osd.4) 58 : cluster [DBG] 6.e scrub ok 2021-03-15T22:22:37.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:37 smithi076 conmon[62156]: cluster 2021-03-15T22:22:36.156191+0000 mon.a (mon.0) 898 : cluster [DBG] osdmap e241: 8 total, 8 up, 7 in 2021-03-15T22:22:37.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:37 smithi076 conmon[62156]: audit 2021-03-15T22:22:36.387442+0000 mon.a (mon.0) 899 : audit [DBG] from='client.? 172.21.15.76:0/30590070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:37.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:37 smithi076 conmon[62156]: audit 2021-03-15T22:22:36.846597+0000 mon.a (mon.0) 900 : audit [INF] from='client.? 172.21.15.76:0/185486720' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-15T22:22:37.459 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:37.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:37.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:37.540 INFO:tasks.rados.rados.0.smithi076.stdout:726: expect (ObjNum 220 snap 50 seq_num 220) 2021-03-15T22:22:37.562 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:37.584 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:37.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:37.686 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:37.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:37.710 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:37.710 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:37.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:37.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:37.835 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:37.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:37.905 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-15T22:22:37.936 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:37.970 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 7 1 2021-03-15T22:22:37.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:37.972 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:37.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:38.073 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:38.095 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:38.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:38.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:38.177 INFO:tasks.rados.rados.0.smithi076.stdout:722: done (4 left) 2021-03-15T22:22:38.178 INFO:tasks.rados.rados.0.smithi076.stdout:724: done (3 left) 2021-03-15T22:22:38.178 INFO:tasks.rados.rados.0.smithi076.stdout:725: done (2 left) 2021-03-15T22:22:38.178 INFO:tasks.rados.rados.0.smithi076.stdout:726: done (1 left) 2021-03-15T22:22:38.178 INFO:tasks.rados.rados.0.smithi076.stdout:727: done (0 left) 2021-03-15T22:22:38.179 INFO:tasks.rados.rados.0.smithi076.stdout:728: write (excl) oid 30 current snap is 75 2021-03-15T22:22:38.179 INFO:tasks.rados.rados.0.smithi076.stdout:728: seq_num 284 ranges {773188=475777,1969239=724762} 2021-03-15T22:22:38.184 INFO:tasks.rados.rados.0.smithi076.stdout:728: writing smithi07694878-30 from 773188 to 1248965 tid 1 2021-03-15T22:22:38.186 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:22:38.187 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:22:38.192 INFO:tasks.rados.rados.0.smithi076.stdout:728: writing smithi07694878-30 from 1969239 to 2694001 tid 2 2021-03-15T22:22:38.193 INFO:tasks.rados.rados.0.smithi076.stdout:729: write (excl) oid 3 current snap is 75 2021-03-15T22:22:38.193 INFO:tasks.rados.rados.0.smithi076.stdout:728: finishing write tid 1 to smithi07694878-30 2021-03-15T22:22:38.193 INFO:tasks.rados.rados.0.smithi076.stdout:729: seq_num 285 ranges {789995=551026,2027721=468420} 2021-03-15T22:22:38.198 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:38.199 INFO:tasks.rados.rados.0.smithi076.stdout:729: writing smithi07694878-3 from 789995 to 1341021 tid 1 2021-03-15T22:22:38.207 INFO:tasks.rados.rados.0.smithi076.stdout:729: writing smithi07694878-3 from 2027721 to 2496141 tid 2 2021-03-15T22:22:38.207 INFO:tasks.rados.rados.0.smithi076.stdout:730: delete oid 10 current snap is 75 2021-03-15T22:22:38.207 INFO:tasks.rados.rados.0.smithi076.stdout:728: finishing write tid 2 to smithi07694878-30 2021-03-15T22:22:38.208 INFO:tasks.rados.rados.0.smithi076.stdout:728: finishing write tid 3 to smithi07694878-30 2021-03-15T22:22:38.208 INFO:tasks.rados.rados.0.smithi076.stdout:728: finishing write tid 4 to smithi07694878-30 2021-03-15T22:22:38.208 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 340 (ObjNum 284 snap 75 seq_num 284) dirty exists 2021-03-15T22:22:38.208 INFO:tasks.rados.rados.0.smithi076.stdout:728: left oid 30 (ObjNum 284 snap 75 seq_num 284) 2021-03-15T22:22:38.208 INFO:tasks.rados.rados.0.smithi076.stdout:729: finishing write tid 1 to smithi07694878-3 2021-03-15T22:22:38.210 INFO:tasks.rados.rados.0.smithi076.stdout:728: done (2 left) 2021-03-15T22:22:38.210 INFO:tasks.rados.rados.0.smithi076.stdout:730: done (1 left) 2021-03-15T22:22:38.211 INFO:tasks.rados.rados.0.smithi076.stdout:731: read oid 17 snap -1 2021-03-15T22:22:38.211 INFO:tasks.rados.rados.0.smithi076.stdout:731: expect (ObjNum 220 snap 50 seq_num 220) 2021-03-15T22:22:38.211 INFO:tasks.rados.rados.0.smithi076.stdout:732: copy_from oid 50 from oid 42 current snap is 75 2021-03-15T22:22:38.211 INFO:tasks.rados.rados.0.smithi076.stdout:733: rollback oid 24 current snap is 75 2021-03-15T22:22:38.212 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 67 2021-03-15T22:22:38.212 INFO:tasks.rados.rados.0.smithi076.stdout:734: read oid 27 snap -1 2021-03-15T22:22:38.212 INFO:tasks.rados.rados.0.smithi076.stdout:734: expect deleted 2021-03-15T22:22:38.212 INFO:tasks.rados.rados.0.smithi076.stdout:735: snap_create 2021-03-15T22:22:38.212 INFO:tasks.rados.rados.0.smithi076.stdout:729: finishing write tid 2 to smithi07694878-3 2021-03-15T22:22:38.213 INFO:tasks.rados.rados.0.smithi076.stdout:729: finishing write tid 3 to smithi07694878-3 2021-03-15T22:22:38.214 INFO:tasks.rados.rados.0.smithi076.stdout:729: finishing write tid 4 to smithi07694878-3 2021-03-15T22:22:38.214 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 252 (ObjNum 285 snap 75 seq_num 285) dirty exists 2021-03-15T22:22:38.214 INFO:tasks.rados.rados.0.smithi076.stdout:729: left oid 3 (ObjNum 285 snap 75 seq_num 285) 2021-03-15T22:22:38.214 INFO:tasks.rados.rados.0.smithi076.stdout:733: finishing rollback tid 0 to smithi07694878-24 2021-03-15T22:22:38.214 INFO:tasks.rados.rados.0.smithi076.stdout:732: finishing copy_from to smithi07694878-50 2021-03-15T22:22:38.215 INFO:tasks.rados.rados.0.smithi076.stdout:732: got expected ENOENT (src dne) 2021-03-15T22:22:38.215 INFO:tasks.rados.rados.0.smithi076.stdout:733: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:22:38.215 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 55 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:22:38.215 INFO:tasks.rados.rados.0.smithi076.stdout:732: finishing copy_from racing read to smithi07694878-50 2021-03-15T22:22:38.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[56514]: cluster 2021-03-15T22:22:36 2021-03-15T22:22:38.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[56514]: .796223+0000 osd.7 (osd.7) 54 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:22:38.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[56514]: cluster 2021-03-15T22:22:37.012246+0000 mgr.y (mgr.14144) 477 : cluster [DBG] pgmap v489: 138 pgs: 138 active+clean; 128 MiB data, 447 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.1 MiB/s wr, 24 op/s 2021-03-15T22:22:38.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[56514]: audit 2021-03-15T22:22:37.163523+0000 mon.a (mon.0) 901 : audit [INF] from='client.? 172.21.15.76:0/185486720' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-15T22:22:38.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[56514]: cluster 2021-03-15T22:22:37.163608+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2021-03-15T22:22:38.217 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: cluster 2021-03-15T22:22:36.796223 2021-03-15T22:22:38.217 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: +0000 osd.7 (osd.7) 54 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:22:38.218 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:22:38.218 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: 22:37.012246+0000 mgr.y (mgr.14144) 477 : cluster [DBG] pgmap v489: 138 pgs: 138 active+clean; 128 MiB data, 447 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.1 MiB/s wr, 24 op/s 2021-03-15T22:22:38.218 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:22:38.218 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: 03-15T22:22:37.163523+0000 2021-03-15T22:22:38.219 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: mon.a (mon.0) 901 : audit [INF] 2021-03-15T22:22:38.219 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/185486720' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-15T22:22:38.219 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:22:38.219 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: 15T22:22:37.163608 2021-03-15T22:22:38.219 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2021-03-15T22:22:38.220 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:38 smithi076 conmon[62156]: 902 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2021-03-15T22:22:38.220 INFO:tasks.rados.rados.0.smithi076.stdout:731: expect (ObjNum 220 snap 50 seq_num 220) 2021-03-15T22:22:38.222 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:38.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:38.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:38.267 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:38.324 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:38.347 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:38.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:38.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:38.415 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:38 smithi125 conmon[56739]: cluster 2021-03-15T22:22:36.796223+0000 osd.7 (osd 2021-03-15T22:22:38.415 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:38 smithi125 conmon[56739]: .7) 54 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:22:38.416 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:38 smithi125 conmon[56739]: cluster 2021-03-15T22:22:37.012246+0000 mgr.y (mgr.14144) 477 : cluster [DBG] pgmap v489: 138 pgs: 138 active+clean; 128 MiB data, 447 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.1 MiB/s wr, 24 op/s 2021-03-15T22:22:38.416 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:38 smithi125 conmon[56739]: audit 2021-03-15T22:22:37.163523+0000 mon.a (mon.0) 901 : audit [INF] from='client.? 172.21.15.76:0/185486720' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-15T22:22:38.416 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:38 smithi125 conmon[56739]: cluster 2021-03-15T22:22:37.163608+0000 mon.a (mon.0) 902 : cluster [DBG] osdmap e242: 8 total, 8 up, 7 in 2021-03-15T22:22:38.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:38.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:38.474 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:38.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:38.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:38.599 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:38.600 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:38.701 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:38.723 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:38.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:38.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:38.751 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:38.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:38.849 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:38.850 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:38.951 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:38.974 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:38.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:38.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:39.077 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:39.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:39.100 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:39.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:39.181 INFO:tasks.rados.rados.0.smithi076.stdout:729: done (5 left) 2021-03-15T22:22:39.181 INFO:tasks.rados.rados.0.smithi076.stdout:731: done (4 left) 2021-03-15T22:22:39.181 INFO:tasks.rados.rados.0.smithi076.stdout:732: done (3 left) 2021-03-15T22:22:39.182 INFO:tasks.rados.rados.0.smithi076.stdout:733: done (2 left) 2021-03-15T22:22:39.182 INFO:tasks.rados.rados.0.smithi076.stdout:734: done (1 left) 2021-03-15T22:22:39.182 INFO:tasks.rados.rados.0.smithi076.stdout:735: done (0 left) 2021-03-15T22:22:39.182 INFO:tasks.rados.rados.0.smithi076.stdout:736: copy_from oid 6 from oid 24 current snap is 76 2021-03-15T22:22:39.183 INFO:tasks.rados.rados.0.smithi076.stdout:737: snap_create 2021-03-15T22:22:39.183 INFO:tasks.rados.rados.0.smithi076.stdout:736: finishing copy_from to smithi07694878-6 2021-03-15T22:22:39.184 INFO:tasks.rados.rados.0.smithi076.stdout:736: got expected ENOENT (src dne) 2021-03-15T22:22:39.184 INFO:tasks.rados.rados.0.smithi076.stdout:736: finishing copy_from racing read to smithi07694878-6 2021-03-15T22:22:39.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:39.225 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:39.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:39.226 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:39.327 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:39.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:39.351 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:39.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:39.438 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[56514]: cluster 2021-03-15T22:22:37.642034+0000 osd.6 (osd 2021-03-15T22:22:39.438 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[56514]: .6) 37 : cluster [DBG] 6.c scrub ok 2021-03-15T22:22:39.439 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[56514]: cluster 2021-03-15T22:22:37.982709+0000 osd.0 (osd.0) 52 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:22:39.439 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[56514]: cluster 2021-03-15T22:22:38.101638+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:22:39.439 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[56514]: cluster 2021-03-15T22:22:38.173994+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2021-03-15T22:22:39.439 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[56514]: cluster 2021-03-15T22:22:38.649029+0000 osd.2 (osd.2) 47 : cluster [DBG] 5.d scrub ok 2021-03-15T22:22:39.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: cluster 2021-03-15T22:22:37.642034 2021-03-15T22:22:39.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: +0000 osd.6 (osd.6) 37 : cluster [DBG] 6.c scrub ok 2021-03-15T22:22:39.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:22:39.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: 03-15T22:22:37.982709+0000 osd.0 (osd.0) 52 : cluster [DBG] 2021-03-15T22:22:39.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: 7.9 scrub ok 2021-03-15T22:22:39.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:22:39.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: 22:38.101638+0000 osd.5 (osd.5 2021-03-15T22:22:39.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: ) 49 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:22:39.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:22:39.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: T22:22:38.173994+0000 mon.a ( 2021-03-15T22:22:39.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: mon.0) 903 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2021-03-15T22:22:39.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:22:39.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: -03-15T22:22:38.649029+0000 2021-03-15T22:22:39.443 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: osd.2 (osd.2) 47 : cluster [DBG] 2021-03-15T22:22:39.443 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:39 smithi076 conmon[62156]: 5.d scrub ok 2021-03-15T22:22:39.450 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:39 smithi125 conmon[56739]: cluster 2021-03-15T22:22:37.642034+0000 osd.6 (osd.6) 37 : cluster [DBG] 6.c scrub ok 2021-03-15T22:22:39.450 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:39 smithi125 conmon[56739]: cluster 2021-03-15T22:22:37.982709+0000 osd.0 (osd.0) 52 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:22:39.450 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:39 smithi125 conmon[56739]: cluster 2021-03-15T22:22:38.101638+0000 osd.5 (osd.5) 49 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:22:39.450 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:39 smithi125 conmon[56739]: cluster 2021-03-15T22:22:38.173994+0000 mon.a (mon.0) 903 : cluster [DBG] osdmap e243: 8 total, 8 up, 7 in 2021-03-15T22:22:39.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:39 smithi125 conmon[56739]: cluster 2021-03-15T22:22:38.649029+0000 osd.2 (osd.2) 47 : cluster [DBG] 5.d scrub ok 2021-03-15T22:22:39.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:39.474 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:39.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:39.475 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:39.479 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:22:39.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:39.601 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:39.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:39.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:39.704 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:39.728 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:39.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:39.742 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:22:39.742 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:22:39.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:22:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:22:39.830 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:39.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:39.853 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:39.853 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:39.954 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:39.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:39.979 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:39.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:40.044 INFO:tasks.rados.rados.0.smithi076.stdout:736: done (1 left) 2021-03-15T22:22:40.044 INFO:tasks.rados.rados.0.smithi076.stdout:737: done (0 left) 2021-03-15T22:22:40.044 INFO:tasks.rados.rados.0.smithi076.stdout:738: write (excl) oid 13 current snap is 77 2021-03-15T22:22:40.045 INFO:tasks.rados.rados.0.smithi076.stdout:738: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2021-03-15T22:22:40.050 INFO:tasks.rados.rados.0.smithi076.stdout:738: writing smithi07694878-13 from 423609 to 1125133 tid 1 2021-03-15T22:22:40.056 INFO:tasks.rados.rados.0.smithi076.stdout:738: writing smithi07694878-13 from 1628332 to 2100421 tid 2 2021-03-15T22:22:40.061 INFO:tasks.rados.rados.0.smithi076.stdout:738: writing smithi07694878-13 from 2648987 to 2823609 tid 3 2021-03-15T22:22:40.061 INFO:tasks.rados.rados.0.smithi076.stdout:739: rollback oid 12 current snap is 77 2021-03-15T22:22:40.061 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 12 to 67 2021-03-15T22:22:40.062 INFO:tasks.rados.rados.0.smithi076.stdout:738: finishing write tid 1 to smithi07694878-13 2021-03-15T22:22:40.062 INFO:tasks.rados.rados.0.smithi076.stdout:740: delete oid 49 current snap is 77 2021-03-15T22:22:40.063 INFO:tasks.rados.rados.0.smithi076.stdout:738: finishing write tid 2 to smithi07694878-13 2021-03-15T22:22:40.064 INFO:tasks.rados.rados.0.smithi076.stdout:738: finishing write tid 3 to smithi07694878-13 2021-03-15T22:22:40.065 INFO:tasks.rados.rados.0.smithi076.stdout:738: finishing write tid 4 to smithi07694878-13 2021-03-15T22:22:40.065 INFO:tasks.rados.rados.0.smithi076.stdout:738: finishing write tid 5 to smithi07694878-13 2021-03-15T22:22:40.065 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 252 (ObjNum 287 snap 77 seq_num 287) dirty exists 2021-03-15T22:22:40.065 INFO:tasks.rados.rados.0.smithi076.stdout:738: left oid 13 (ObjNum 287 snap 77 seq_num 287) 2021-03-15T22:22:40.065 INFO:tasks.rados.rados.0.smithi076.stdout:739: finishing rollback tid 0 to smithi07694878-12 2021-03-15T22:22:40.066 INFO:tasks.rados.rados.0.smithi076.stdout:739: finishing rollback tid 1 to smithi07694878-12 2021-03-15T22:22:40.067 INFO:tasks.rados.rados.0.smithi076.stdout:739: finishing rollback tid 2 to smithi07694878-12 2021-03-15T22:22:40.067 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 344 (ObjNum 199 snap 43 seq_num 199) dirty exists 2021-03-15T22:22:40.067 INFO:tasks.rados.rados.0.smithi076.stdout:738: done (2 left) 2021-03-15T22:22:40.068 INFO:tasks.rados.rados.0.smithi076.stdout:739: done (1 left) 2021-03-15T22:22:40.068 INFO:tasks.rados.rados.0.smithi076.stdout:740: done (0 left) 2021-03-15T22:22:40.068 INFO:tasks.rados.rados.0.smithi076.stdout:741: delete oid 14 current snap is 77 2021-03-15T22:22:40.069 INFO:tasks.rados.rados.0.smithi076.stdout:741: done (0 left) 2021-03-15T22:22:40.069 INFO:tasks.rados.rados.0.smithi076.stdout:742: read oid 5 snap -1 2021-03-15T22:22:40.070 INFO:tasks.rados.rados.0.smithi076.stdout:742: expect (ObjNum 66 snap 6 seq_num 66) 2021-03-15T22:22:40.071 INFO:tasks.rados.rados.0.smithi076.stdout:743: read oid 48 snap 70 2021-03-15T22:22:40.071 INFO:tasks.rados.rados.0.smithi076.stdout:743: expect (ObjNum 253 snap 63 seq_num 253) 2021-03-15T22:22:40.071 INFO:tasks.rados.rados.0.smithi076.stdout:744: write oid 41 current snap is 77 2021-03-15T22:22:40.072 INFO:tasks.rados.rados.0.smithi076.stdout:744: seq_num 290 ranges {474030=643624,1520131=735769} 2021-03-15T22:22:40.074 INFO:tasks.rados.rados.0.smithi076.stdout:744: writing smithi07694878-41 from 474030 to 1117654 tid 1 2021-03-15T22:22:40.081 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:40.082 INFO:tasks.rados.rados.0.smithi076.stdout:744: writing smithi07694878-41 from 1520131 to 2255900 tid 2 2021-03-15T22:22:40.082 INFO:tasks.rados.rados.0.smithi076.stdout:745: snap_remove snap 67 2021-03-15T22:22:40.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:40.105 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:40.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:40.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:40.230 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:40.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:40.232 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:40.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:40.356 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:40.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:40.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:22:40.430 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: cluster 2021-03-15T22:22:38.664620+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2021-03-15T22:22:40.430 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: cluster 2021-03-15T22:22:38.810389+0000 osd.7 (osd.7) 55 : cluster [DBG] 3.e scrub ok 2021-03-15T22:22:40.430 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: cluster 2021-03-15T22:22:39.012833+0000 mgr.y (mgr.14144) 478 : cluster [DBG] pgmap v492: 138 pgs: 138 active+clean; 133 MiB data, 457 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 1.9 MiB/s wr, 23 op/s 2021-03-15T22:22:40.431 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: cluster 2021-03-15T22:22:39.054149+0000 osd.1 (osd.1) 54 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:22:40.431 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:22:40.431 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: T22:22:39.140413+0000 osd.5 (osd.5) 50 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:22:40.431 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:22:40.432 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: T22:22:39.177863+0000 mon.a ( 2021-03-15T22:22:40.432 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: mon.0) 904 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2021-03-15T22:22:40.432 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: cluster 2021-03-15T22:22:40.433 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: 2021-03-15T22:22:39.379018 2021-03-15T22:22:40.433 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: +0000 osd.3 (osd.3) 48 : cluster 2021-03-15T22:22:40.433 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: [DBG] purged_snaps scrub starts 2021-03-15T22:22:40.433 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: cluster 2021-03-15T22:22 2021-03-15T22:22:40.434 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: : 2021-03-15T22:22:40.434 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: 39 2021-03-15T22:22:40.434 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: .379298+0000 osd.3 (osd.3) 49 2021-03-15T22:22:40.434 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:40.434 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: audit 2021-03-15T22:22:40.435 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: 2021-03-15T22:22:39.551410+0000 mgr.y (mgr.14144) 2021-03-15T22:22:40.435 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: 479 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:40.435 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: cluster 2021-03-15T22:22:40.036842+0000 mon.a ( 2021-03-15T22:22:40.436 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:40 smithi125 conmon[56739]: mon.0) 905 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2021-03-15T22:22:40.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:40.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:40.483 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:40.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:40.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[56514]: cluster 2021-03-15T22:22:38.664620+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2021-03-15T22:22:40.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[56514]: cluster 2021-03-15T22:22:38.810389+0000 osd.7 (osd.7) 55 : cluster [DBG] 3.e scrub ok 2021-03-15T22:22:40.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[56514]: cluster 2021-03-15T22:22:39.012833+0000 mgr.y (mgr.14144) 478 : cluster [DBG] pgmap v492: 138 pgs: 138 active+clean; 133 MiB data, 457 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 1.9 MiB/s wr, 23 op/s 2021-03-15T22:22:40.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[56514]: cluster 2021-03-15T22:22:39.054149+0000 osd.1 (osd.1) 54 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:22:40.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[56514]: cluster 2021-03-15T22:22:39.140413+0000 osd.5 (osd.5) 50 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:22:40.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[56514]: cluster 2021-03-15T22:22:39.177863+0000 mon.a (mon.0) 904 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2021-03-15T22:22:40.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[56514]: cluster 2021-03-15T22:22:39.379018+0000 osd.3 (osd.3) 48 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:40.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[56514]: cluster 2021-03-15T22:22:39.379298+0000 osd.3 (osd.3) 49 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:40.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[56514]: audit 2021-03-15T22:22:39.551410+0000 mgr.y (mgr.14144) 479 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:40.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[56514]: cluster 2021-03-15T22:22:40.036842+0000 mon.a (mon.0) 905 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2021-03-15T22:22:40.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[62156]: cluster 2021-03-15T22:22:38.664620+0000 osd.6 (osd.6) 38 : cluster [DBG] 3.a scrub ok 2021-03-15T22:22:40.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[62156]: cluster 2021-03-15T22:22:38.810389+0000 osd.7 (osd.7) 55 : cluster [DBG] 3.e scrub ok 2021-03-15T22:22:40.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[62156]: cluster 2021-03-15T22:22:39.012833+0000 mgr.y (mgr.14144) 478 : cluster [DBG] pgmap v492: 138 pgs: 138 active+clean; 133 MiB data, 457 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 1.9 MiB/s wr, 23 op/s 2021-03-15T22:22:40.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[62156]: cluster 2021-03-15T22:22:39.054149+0000 osd.1 (osd.1) 54 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:22:40.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[62156]: cluster 2021-03-15T22:22:39.140413+0000 osd.5 (osd.5) 50 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:22:40.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[62156]: cluster 2021-03-15T22:22:39.177863+0000 mon.a (mon.0) 904 : cluster [DBG] osdmap e244: 8 total, 8 up, 7 in 2021-03-15T22:22:40.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[62156]: cluster 2021-03-15T22:22:39.379018+0000 osd.3 (osd.3) 48 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:40.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[62156]: cluster 2021-03-15T22:22:39.379298+0000 osd.3 (osd.3) 49 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:40.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[62156]: audit 2021-03-15T22:22:39.551410+0000 mgr.y (mgr.14144) 479 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:40.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:40 smithi076 conmon[62156]: cluster 2021-03-15T22:22:40.036842+0000 mon.a (mon.0) 905 : cluster [DBG] osdmap e245: 8 total, 8 up, 7 in 2021-03-15T22:22:40.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:40.602 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:40.607 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:40.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:40.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:40.681 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:22:40.709 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:40.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:40.735 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:40.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:40.836 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:40.859 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:40.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:40.861 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:40.962 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:40.986 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:40.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:40.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:41.045 INFO:tasks.rados.rados.0.smithi076.stdout:745: done (3 left) 2021-03-15T22:22:41.045 INFO:tasks.rados.rados.0.smithi076.stdout:746: snap_create 2021-03-15T22:22:41.046 INFO:tasks.rados.rados.0.smithi076.stdout:742: expect (ObjNum 66 snap 6 seq_num 66) 2021-03-15T22:22:41.058 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.7 to 1 (10000) 2021-03-15T22:22:41.089 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:41.111 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:41.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:41.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:41.187 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[56514]: cluster 2021-03-15T22:22:39.712175+0000 osd.6 (osd.6) 2021-03-15T22:22:41.188 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[56514]: 39 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:22:41.188 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[56514]: cluster 2021-03-15T22:22:39.928345+0000 osd.0 (osd.0) 53 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:22:41.188 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[56514]: cluster 2021-03-15T22:22:40.038180+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:22:41.188 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[56514]: cluster 2021-03-15T22:22:40.167660+0000 osd.5 (osd.5) 51 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:22:41.189 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[56514]: audit 2021-03-15T22:22:40.543458+0000 mon.a (mon.0) 906 : audit [INF] from='client.? 172.21.15.76:0/2877032303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:22:41.189 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[56514]: audit 2021-03-15T22:22:41.043337+0000 mon.a (mon.0) 907 : audit [INF] from='client.? 172.21.15.76:0/2877032303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:22:41.189 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[56514]: cluster 2021-03-15T22:22:41.043395+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2021-03-15T22:22:41.189 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[62156]: cluster 2021-03-15T22:22:41.190 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[62156]: 2021-03-15T22:22 2021-03-15T22:22:41.190 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[62156]: :39.712175+0000 osd.6 (osd.6) 39 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:22:41.190 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:22:41.190 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[62156]: 03-15T22:22:39.928345+0000 osd.0 (osd. 2021-03-15T22:22:41.191 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[62156]: 0) 53 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:22:41.191 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[62156]: cluster 2021-03-15T22:22:40.038180+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:22:41.191 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[62156]: cluster 2021-03-15T22:22:40.167660+0000 osd.5 (osd.5) 51 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:22:41.191 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[62156]: audit 2021-03-15T22:22:40.543458+0000 mon.a (mon.0) 906 : audit [INF] from='client.? 172.21.15.76:0/2877032303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:22:41.191 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[62156]: audit 2021-03-15T22:22:41.043337+0000 mon.a (mon.0) 907 : audit [INF] from='client.? 172.21.15.76:0/2877032303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:22:41.192 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:41 smithi076 conmon[62156]: cluster 2021-03-15T22:22:41.043395+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2021-03-15T22:22:41.214 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:41.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:41.237 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:41.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:41.241 INFO:tasks.rados.rados.0.smithi076.stdout:743: expect (ObjNum 253 snap 63 seq_num 253) 2021-03-15T22:22:41.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:41.363 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:41.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:41.364 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:41.387 INFO:tasks.rados.rados.0.smithi076.stdout:744: finishing write tid 1 to smithi07694878-41 2021-03-15T22:22:41.388 INFO:tasks.rados.rados.0.smithi076.stdout:744: finishing write tid 2 to smithi07694878-41 2021-03-15T22:22:41.388 INFO:tasks.rados.rados.0.smithi076.stdout:744: finishing write tid 3 to smithi07694878-41 2021-03-15T22:22:41.388 INFO:tasks.rados.rados.0.smithi076.stdout:744: finishing write tid 4 to smithi07694878-41 2021-03-15T22:22:41.388 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 137 (ObjNum 290 snap 77 seq_num 290) dirty exists 2021-03-15T22:22:41.388 INFO:tasks.rados.rados.0.smithi076.stdout:744: left oid 41 (ObjNum 290 snap 77 seq_num 290) 2021-03-15T22:22:41.465 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:41.487 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:41.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:41.489 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:41.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:41 smithi125 conmon[56739]: cluster 2021-03-15T22:22:39.712175+0000 osd.6 (osd.6) 39 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:22:41.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:41 smithi125 conmon[56739]: cluster 2021-03-15T22:22:39.928345+0000 osd.0 (osd.0) 53 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:22:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:41 smithi125 conmon[56739]: cluster 2021-03-15T22:22:40.038180+0000 osd.1 (osd.1) 55 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:22:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:41 smithi125 conmon[56739]: cluster 2021-03-15T22:22:40.167660+0000 osd.5 (osd.5) 51 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:22:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:41 smithi125 conmon[56739]: audit 2021-03-15T22:22:40.543458+0000 mon.a (mon.0) 906 : audit [INF] from='client.? 172.21.15.76:0/2877032303' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:22:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:41 smithi125 conmon[56739]: audit 2021-03-15T22:22:41.043337+0000 mon.a (mon.0) 907 : audit [INF] from='client.? 172.21.15.76:0/2877032303' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:22:41.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:41 smithi125 conmon[56739]: cluster 2021-03-15T22:22:41.043395+0000 mon.a (mon.0) 908 : cluster [DBG] osdmap e246: 8 total, 8 up, 7 in 2021-03-15T22:22:41.590 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:41.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:41.614 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:41.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:41.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:41.738 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:41.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:41.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:41.840 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:41.863 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:41.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:41.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:41.961 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2021-03-15T22:22:41.961 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:22:41.965 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:41.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:41.989 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:41.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:42.050 INFO:tasks.rados.rados.0.smithi076.stdout:742: done (3 left) 2021-03-15T22:22:42.050 INFO:tasks.rados.rados.0.smithi076.stdout:743: done (2 left) 2021-03-15T22:22:42.050 INFO:tasks.rados.rados.0.smithi076.stdout:744: done (1 left) 2021-03-15T22:22:42.051 INFO:tasks.rados.rados.0.smithi076.stdout:746: done (0 left) 2021-03-15T22:22:42.051 INFO:tasks.rados.rados.0.smithi076.stdout:747: read oid 10 snap 63 2021-03-15T22:22:42.051 INFO:tasks.rados.rados.0.smithi076.stdout:747: expect deleted 2021-03-15T22:22:42.051 INFO:tasks.rados.rados.0.smithi076.stdout:748: write oid 26 current snap is 78 2021-03-15T22:22:42.052 INFO:tasks.rados.rados.0.smithi076.stdout:748: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2021-03-15T22:22:42.054 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:22:42.057 INFO:tasks.rados.rados.0.smithi076.stdout:748: writing smithi07694878-26 from 490837 to 1209710 tid 1 2021-03-15T22:22:42.064 INFO:tasks.rados.rados.0.smithi076.stdout:748: writing smithi07694878-26 from 1862260 to 2341687 tid 2 2021-03-15T22:22:42.068 INFO:tasks.rados.rados.0.smithi076.stdout:748: writing smithi07694878-26 from 2875032 to 2890837 tid 3 2021-03-15T22:22:42.068 INFO:tasks.rados.rados.0.smithi076.stdout:749: rollback oid 48 current snap is 78 2021-03-15T22:22:42.068 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 48 to 63 2021-03-15T22:22:42.069 INFO:tasks.rados.rados.0.smithi076.stdout:748: finishing write tid 1 to smithi07694878-26 2021-03-15T22:22:42.069 INFO:tasks.rados.rados.0.smithi076.stdout:748: finishing write tid 2 to smithi07694878-26 2021-03-15T22:22:42.069 INFO:tasks.rados.rados.0.smithi076.stdout:747: done (2 left) 2021-03-15T22:22:42.069 INFO:tasks.rados.rados.0.smithi076.stdout:750: rollback oid 42 current snap is 78 2021-03-15T22:22:42.070 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 42 to 63 2021-03-15T22:22:42.070 INFO:tasks.rados.rados.0.smithi076.stdout:751: delete oid 12 current snap is 78 2021-03-15T22:22:42.070 INFO:tasks.rados.rados.0.smithi076.stdout:748: finishing write tid 3 to smithi07694878-26 2021-03-15T22:22:42.071 INFO:tasks.rados.rados.0.smithi076.stdout:749: finishing rollback tid 0 to smithi07694878-48 2021-03-15T22:22:42.071 INFO:tasks.rados.rados.0.smithi076.stdout:749: finishing rollback tid 1 to smithi07694878-48 2021-03-15T22:22:42.072 INFO:tasks.rados.rados.0.smithi076.stdout:749: finishing rollback tid 2 to smithi07694878-48 2021-03-15T22:22:42.072 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 143 (ObjNum 253 snap 63 seq_num 253) dirty exists 2021-03-15T22:22:42.072 INFO:tasks.rados.rados.0.smithi076.stdout:750: finishing rollback tid 1 to smithi07694878-42 2021-03-15T22:22:42.072 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:22:42.073 INFO:tasks.rados.rados.0.smithi076.stdout:748: finishing write tid 4 to smithi07694878-26 2021-03-15T22:22:42.074 INFO:tasks.rados.rados.0.smithi076.stdout:748: finishing write tid 5 to smithi07694878-26 2021-03-15T22:22:42.074 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 257 (ObjNum 291 snap 78 seq_num 291) dirty exists 2021-03-15T22:22:42.074 INFO:tasks.rados.rados.0.smithi076.stdout:748: left oid 26 (ObjNum 291 snap 78 seq_num 291) 2021-03-15T22:22:42.075 INFO:tasks.rados.rados.0.smithi076.stdout:748: done (3 left) 2021-03-15T22:22:42.075 INFO:tasks.rados.rados.0.smithi076.stdout:749: done (2 left) 2021-03-15T22:22:42.075 INFO:tasks.rados.rados.0.smithi076.stdout:750: done (1 left) 2021-03-15T22:22:42.075 INFO:tasks.rados.rados.0.smithi076.stdout:751: done (0 left) 2021-03-15T22:22:42.076 INFO:tasks.rados.rados.0.smithi076.stdout:752: rollback oid 7 current snap is 78 2021-03-15T22:22:42.076 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 7 to 72 2021-03-15T22:22:42.076 INFO:tasks.rados.rados.0.smithi076.stdout:753: delete oid 18 current snap is 78 2021-03-15T22:22:42.076 INFO:tasks.rados.rados.0.smithi076.stdout:753: done (1 left) 2021-03-15T22:22:42.077 INFO:tasks.rados.rados.0.smithi076.stdout:754: write oid 10 current snap is 78 2021-03-15T22:22:42.077 INFO:tasks.rados.rados.0.smithi076.stdout:754: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2021-03-15T22:22:42.079 INFO:tasks.rados.rados.0.smithi076.stdout:754: writing smithi07694878-10 from 541258 to 1085878 tid 1 2021-03-15T22:22:42.086 INFO:tasks.rados.rados.0.smithi076.stdout:754: writing smithi07694878-10 from 1637706 to 2264460 tid 2 2021-03-15T22:22:42.090 INFO:tasks.rados.rados.0.smithi076.stdout:754: writing smithi07694878-10 from 2840948 to 2941258 tid 3 2021-03-15T22:22:42.090 INFO:tasks.rados.rados.0.smithi076.stdout:755: read oid 44 snap -1 2021-03-15T22:22:42.091 INFO:tasks.rados.rados.0.smithi076.stdout:755: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:22:42.091 INFO:tasks.rados.rados.0.smithi076.stdout:756: write (excl) oid 2 current snap is 78 2021-03-15T22:22:42.091 INFO:tasks.rados.rados.0.smithi076.stdout:752: finishing rollback tid 0 to smithi07694878-7 2021-03-15T22:22:42.091 INFO:tasks.rados.rados.0.smithi076.stdout:756: seq_num 295 ranges {558065=619869,1579835=486765,2468371=489694} 2021-03-15T22:22:42.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:42.096 INFO:tasks.rados.rados.0.smithi076.stdout:756: writing smithi07694878-2 from 558065 to 1177934 tid 1 2021-03-15T22:22:42.102 INFO:tasks.rados.rados.0.smithi076.stdout:756: writing smithi07694878-2 from 1579835 to 2066600 tid 2 2021-03-15T22:22:42.107 INFO:tasks.rados.rados.0.smithi076.stdout:756: writing smithi07694878-2 from 2468371 to 2958065 tid 3 2021-03-15T22:22:42.107 INFO:tasks.rados.rados.0.smithi076.stdout:757: snap_remove snap 77 2021-03-15T22:22:42.115 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:42.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:42.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:42.217 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:42.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:42.241 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:42.242 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:42.259 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:22:42.259 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:22:42.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:42 smithi125 conmon[56739]: cluster 2021-03-15T22:22:40.995785+0000 osd.0 (osd.0) 54 : cluster [DBG] 7.6 deep-scrub ok 2021-03-15T22:22:42.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:42 smithi125 conmon[56739]: cluster 2021-03-15T22:22:41.013268+0000 mgr.y (mgr.14144) 480 : cluster [DBG] pgmap v495: 138 pgs: 138 active+clean; 137 MiB data, 467 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 23 op/s 2021-03-15T22:22:42.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:42 smithi125 conmon[56739]: cluster 2021-03-15T22:22:41.086177+0000 osd.1 (osd.1) 56 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:22:42.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:42 smithi125 conmon[56739]: cluster 2021-03-15T22:22:41.168880+0000 osd.5 (osd.5) 52 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:22:42.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:42 smithi125 conmon[56739]: audit 2021-03-15T22:22:41.363973+0000 mon.a (mon.0) 909 : 2021-03-15T22:22:42.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:42 smithi125 conmon[56739]: audit [INF] from='client.? 172.21.15.76:0/3902773027' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:22:42.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:42 smithi125 conmon[56739]: cluster 2021-03-15T22:22:41.652184+0000 osd.2 (osd.2) 48 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:22:42.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:42 smithi125 conmon[56739]: cluster 2021-03-15T22:22:42.040877+0000 mon.a (mon.0) 910 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:42.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:42 smithi125 conmon[56739]: audit 2021-03-15T22:22:42.047220+0000 mon.a (mon.0) 911 : audit [INF] from='client.? 172.21.15.76:0/3902773027' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:22:42.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:42 smithi125 conmon[56739]: cluster 2021-03-15T22:22:42.047280+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2021-03-15T22:22:42.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:42.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:42.367 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:42.368 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:42.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:42.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:42.492 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:42.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:42.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[56514]: cluster 2021-03-15T22:22: 2021-03-15T22:22:42.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[56514]: 40.995785+0000 osd.0 (osd.0) 54 : cluster [DBG] 7.6 deep-scrub ok 2021-03-15T22:22:42.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[56514]: cluster 2021-03-15T22:22:41.013268+0000 mgr.y (mgr.14144) 480 : cluster [DBG] pgmap v495: 138 pgs: 138 active+clean; 137 MiB data, 467 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 23 op/s 2021-03-15T22:22:42.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[56514]: cluster 2021-03-15T22:22:41.086177+0000 osd.1 (osd.1) 56 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:22:42.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[56514]: cluster 2021-03-15T22:22:41.168880+0000 osd.5 (osd.5) 52 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:22:42.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[56514]: audit 2021-03-15T22:22:41.363973+0000 mon.a (mon.0) 909 : audit [INF] from='client.? 172.21.15.76:0/3902773027' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:22:42.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[56514]: cluster 2021-03-15T22:22:41.652184+0000 osd.2 (osd.2) 48 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:22:42.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[56514]: cluster 2021-03-15T22:22:42.040877+0000 mon.a (mon.0) 910 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:42.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[56514]: audit 2021-03-15T22:22:42.047220+0000 mon.a (mon.0) 911 : audit [INF] from='client.? 172.21.15.76:0/3902773027' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:22:42.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[56514]: cluster 2021-03-15T22:22:42.047280+0000 mon.a (mon.0) 912 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2021-03-15T22:22:42.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: cluster 2021-03-15T22:22:40.995785+0000 osd.0 2021-03-15T22:22:42.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: (osd.0) 54 : cluster [DBG] 7.6 deep-scrub ok 2021-03-15T22:22:42.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:22:42.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: 15T22:22:41.013268+0000 mgr.y (mgr.14144) 480 : cluster [DBG] 2021-03-15T22:22:42.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: pgmap v495: 138 pgs: 138 active+clean; 137 MiB data, 467 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 1.9 MiB/s wr, 23 op/s 2021-03-15T22:22:42.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:22:42.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: 15T22:22:41.086177+0000 osd.1 (osd 2021-03-15T22:22:42.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: .1) 56 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:22:42.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:22:42.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: 03-15T22:22:41.168880+0000 osd.5 2021-03-15T22:22:42.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: (osd.5) 52 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:22:42.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:22:42.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: 03-15T22:22:41.363973+0000 mon.a 2021-03-15T22:22:42.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: (mon.0) 909 : audit [INF] from='client.? 172.21.15.76:0/3902773027' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:22:42.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: cluster 2021-03-15T22:22 2021-03-15T22:22:42.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: :41.652184+0000 osd.2 (osd.2) 48 : 2021-03-15T22:22:42.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: cluster [DBG] 6.14 scrub ok 2021-03-15T22:22:42.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:22:42.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: 22:42.040877+0000 mon.a (mon.0) 910 : 2021-03-15T22:22:42.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:42.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: audit 2021-03-15T22:22: 2021-03-15T22:22:42.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: 42.047220+0000 mon.a (mon.0) 911 : audit 2021-03-15T22:22:42.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/3902773027' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:22:42.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: cluster 2021-03-15T22:22:42.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: 2021-03-15T22:22:42.047280+0000 mon.a (mon.0 2021-03-15T22:22:42.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:42 smithi076 conmon[62156]: ) 912 : cluster [DBG] osdmap e247: 8 total, 8 up, 7 in 2021-03-15T22:22:42.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:42.619 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:42.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:42.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:42.722 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:42.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:42.747 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:42.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:42.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:42.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:42.873 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:42.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:42.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-15T22:22:42.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:42.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:43.000 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:43.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:43.051 INFO:tasks.rados.rados.0.smithi076.stdout:757: done (4 left) 2021-03-15T22:22:43.052 INFO:tasks.rados.rados.0.smithi076.stdout:758: write oid 19 current snap is 78 2021-03-15T22:22:43.053 INFO:tasks.rados.rados.0.smithi076.stdout:758: seq_num 296 ranges {574872=695118,2038317=630423} 2021-03-15T22:22:43.058 INFO:tasks.rados.rados.0.smithi076.stdout:758: writing smithi07694878-19 from 574872 to 1269990 tid 1 2021-03-15T22:22:43.067 INFO:tasks.rados.rados.0.smithi076.stdout:758: writing smithi07694878-19 from 2038317 to 2668740 tid 2 2021-03-15T22:22:43.067 INFO:tasks.rados.rados.0.smithi076.stdout:759: delete oid 3 current snap is 78 2021-03-15T22:22:43.067 INFO:tasks.rados.rados.0.smithi076.stdout:752: finishing rollback tid 1 to smithi07694878-7 2021-03-15T22:22:43.068 INFO:tasks.rados.rados.0.smithi076.stdout:752: finishing rollback tid 2 to smithi07694878-7 2021-03-15T22:22:43.068 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 292 (ObjNum 271 snap 72 seq_num 271) dirty exists 2021-03-15T22:22:43.068 INFO:tasks.rados.rados.0.smithi076.stdout:754: finishing write tid 1 to smithi07694878-10 2021-03-15T22:22:43.068 INFO:tasks.rados.rados.0.smithi076.stdout:754: finishing write tid 2 to smithi07694878-10 2021-03-15T22:22:43.069 INFO:tasks.rados.rados.0.smithi076.stdout:755: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:22:43.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:43.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:43.125 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:43.126 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:43.163 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:43.217 INFO:tasks.rados.rados.0.smithi076.stdout:754: finishing write tid 3 to smithi07694878-10 2021-03-15T22:22:43.218 INFO:tasks.rados.rados.0.smithi076.stdout:754: finishing write tid 4 to smithi07694878-10 2021-03-15T22:22:43.218 INFO:tasks.rados.rados.0.smithi076.stdout:754: finishing write tid 5 to smithi07694878-10 2021-03-15T22:22:43.218 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 124 (ObjNum 294 snap 78 seq_num 294) dirty exists 2021-03-15T22:22:43.219 INFO:tasks.rados.rados.0.smithi076.stdout:754: left oid 10 (ObjNum 294 snap 78 seq_num 294) 2021-03-15T22:22:43.219 INFO:tasks.rados.rados.0.smithi076.stdout:756: finishing write tid 1 to smithi07694878-2 2021-03-15T22:22:43.219 INFO:tasks.rados.rados.0.smithi076.stdout:752: done (5 left) 2021-03-15T22:22:43.219 INFO:tasks.rados.rados.0.smithi076.stdout:754: done (4 left) 2021-03-15T22:22:43.220 INFO:tasks.rados.rados.0.smithi076.stdout:755: done (3 left) 2021-03-15T22:22:43.220 INFO:tasks.rados.rados.0.smithi076.stdout:759: done (2 left) 2021-03-15T22:22:43.220 INFO:tasks.rados.rados.0.smithi076.stdout:760: write (excl) oid 20 current snap is 78 2021-03-15T22:22:43.220 INFO:tasks.rados.rados.0.smithi076.stdout:756: finishing write tid 2 to smithi07694878-2 2021-03-15T22:22:43.220 INFO:tasks.rados.rados.0.smithi076.stdout:756: finishing write tid 3 to smithi07694878-2 2021-03-15T22:22:43.221 INFO:tasks.rados.rados.0.smithi076.stdout:760: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2021-03-15T22:22:43.223 INFO:tasks.rados.rados.0.smithi076.stdout:760: writing smithi07694878-20 from 608486 to 1170455 tid 1 2021-03-15T22:22:43.227 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:43.228 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:22:43.240 INFO:tasks.rados.rados.0.smithi076.stdout:760: writing smithi07694878-20 from 1871634 to 2505726 tid 2 2021-03-15T22:22:43.241 INFO:tasks.rados.rados.0.smithi076.stdout:760: writing smithi07694878-20 from 2950640 to 3008486 tid 3 2021-03-15T22:22:43.241 INFO:tasks.rados.rados.0.smithi076.stdout:761: delete oid 11 current snap is 78 2021-03-15T22:22:43.241 INFO:tasks.rados.rados.0.smithi076.stdout:756: finishing write tid 4 to smithi07694878-2 2021-03-15T22:22:43.241 INFO:tasks.rados.rados.0.smithi076.stdout:756: finishing write tid 5 to smithi07694878-2 2021-03-15T22:22:43.242 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 128 (ObjNum 295 snap 78 seq_num 295) dirty exists 2021-03-15T22:22:43.242 INFO:tasks.rados.rados.0.smithi076.stdout:756: left oid 2 (ObjNum 295 snap 78 seq_num 295) 2021-03-15T22:22:43.242 INFO:tasks.rados.rados.0.smithi076.stdout:758: finishing write tid 1 to smithi07694878-19 2021-03-15T22:22:43.243 INFO:tasks.rados.rados.0.smithi076.stdout:758: finishing write tid 2 to smithi07694878-19 2021-03-15T22:22:43.243 INFO:tasks.rados.rados.0.smithi076.stdout:758: finishing write tid 3 to smithi07694878-19 2021-03-15T22:22:43.243 INFO:tasks.rados.rados.0.smithi076.stdout:758: finishing write tid 4 to smithi07694878-19 2021-03-15T22:22:43.243 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 297 (ObjNum 296 snap 78 seq_num 296) dirty exists 2021-03-15T22:22:43.243 INFO:tasks.rados.rados.0.smithi076.stdout:758: left oid 19 (ObjNum 296 snap 78 seq_num 296) 2021-03-15T22:22:43.244 INFO:tasks.rados.rados.0.smithi076.stdout:760: finishing write tid 1 to smithi07694878-20 2021-03-15T22:22:43.244 INFO:tasks.rados.rados.0.smithi076.stdout:760: finishing write tid 2 to smithi07694878-20 2021-03-15T22:22:43.244 INFO:tasks.rados.rados.0.smithi076.stdout:756: done (3 left) 2021-03-15T22:22:43.244 INFO:tasks.rados.rados.0.smithi076.stdout:758: done (2 left) 2021-03-15T22:22:43.245 INFO:tasks.rados.rados.0.smithi076.stdout:761: done (1 left) 2021-03-15T22:22:43.245 INFO:tasks.rados.rados.0.smithi076.stdout:762: read oid 7 snap -1 2021-03-15T22:22:43.245 INFO:tasks.rados.rados.0.smithi076.stdout:762: expect (ObjNum 271 snap 72 seq_num 271) 2021-03-15T22:22:43.245 INFO:tasks.rados.rados.0.smithi076.stdout:763: delete oid 27 current snap is 78 2021-03-15T22:22:43.246 INFO:tasks.rados.rados.0.smithi076.stdout:760: finishing write tid 3 to smithi07694878-20 2021-03-15T22:22:43.246 INFO:tasks.rados.rados.0.smithi076.stdout:760: finishing write tid 4 to smithi07694878-20 2021-03-15T22:22:43.246 INFO:tasks.rados.rados.0.smithi076.stdout:760: finishing write tid 5 to smithi07694878-20 2021-03-15T22:22:43.247 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 177 (ObjNum 298 snap 78 seq_num 298) dirty exists 2021-03-15T22:22:43.247 INFO:tasks.rados.rados.0.smithi076.stdout:760: left oid 20 (ObjNum 298 snap 78 seq_num 298) 2021-03-15T22:22:43.247 INFO:tasks.rados.rados.0.smithi076.stdout:760: done (2 left) 2021-03-15T22:22:43.247 INFO:tasks.rados.rados.0.smithi076.stdout:763: done (1 left) 2021-03-15T22:22:43.247 INFO:tasks.rados.rados.0.smithi076.stdout:764: rollback oid 3 current snap is 78 2021-03-15T22:22:43.248 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 72 2021-03-15T22:22:43.248 INFO:tasks.rados.rados.0.smithi076.stdout:765: read oid 38 snap -1 2021-03-15T22:22:43.248 INFO:tasks.rados.rados.0.smithi076.stdout:765: expect (ObjNum 264 snap 72 seq_num 264) 2021-03-15T22:22:43.248 INFO:tasks.rados.rados.0.smithi076.stdout:766: delete oid 48 current snap is 78 2021-03-15T22:22:43.249 INFO:tasks.rados.rados.0.smithi076.stdout:762: expect (ObjNum 271 snap 72 seq_num 271) 2021-03-15T22:22:43.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:43.257 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:43.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:43.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:43.364 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:43 smithi076 conmon[56514]: cluster 2021-03-15T22:22:41.941851+0000 osd.4 (osd.4) 59 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:22:43.364 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:43 smithi076 conmon[56514]: cluster 2021-03-15T22:22:43.364 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:43 smithi076 conmon[56514]: 2021-03-15T22:22:43. 2021-03-15T22:22:43.364 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:43 smithi076 conmon[56514]: 048536+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e248: 8 total, 8 up, 7 in 2021-03-15T22:22:43.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:43 smithi076 conmon[62156]: cluster 2021-03-15T22:22:41.941851+0000 2021-03-15T22:22:43.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:43 smithi076 conmon[62156]: osd.4 (osd.4) 59 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:22:43.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:43 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:22:43.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:43 smithi076 conmon[62156]: -03-15T22:22:43.048536+0000 mon.a (mon.0) 913 : cluster [DBG] 2021-03-15T22:22:43.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:43 smithi076 conmon[62156]: osdmap e248: 8 total, 8 up, 7 in 2021-03-15T22:22:43.367 INFO:tasks.rados.rados.0.smithi076.stdout:764: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:22:43.367 INFO:tasks.rados.rados.0.smithi076.stdout:764: finishing rollback tid 2 to smithi07694878-3 2021-03-15T22:22:43.367 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 260 (ObjNum 274 snap 72 seq_num 274) dirty exists 2021-03-15T22:22:43.367 INFO:tasks.rados.rados.0.smithi076.stdout:765: expect (ObjNum 264 snap 72 seq_num 264) 2021-03-15T22:22:43.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:43.382 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:43.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:43.441 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:22:43.445 INFO:tasks.rados.rados.0.smithi076.stdout:762: done (3 left) 2021-03-15T22:22:43.446 INFO:tasks.rados.rados.0.smithi076.stdout:764: done (2 left) 2021-03-15T22:22:43.446 INFO:tasks.rados.rados.0.smithi076.stdout:765: done (1 left) 2021-03-15T22:22:43.446 INFO:tasks.rados.rados.0.smithi076.stdout:766: done (0 left) 2021-03-15T22:22:43.446 INFO:tasks.rados.rados.0.smithi076.stdout:767: copy_from oid 19 from oid 11 current snap is 78 2021-03-15T22:22:43.447 INFO:tasks.rados.rados.0.smithi076.stdout:768: rollback oid 44 current snap is 78 2021-03-15T22:22:43.447 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 44 to 76 2021-03-15T22:22:43.447 INFO:tasks.rados.rados.0.smithi076.stdout:769: write (excl) oid 14 current snap is 78 2021-03-15T22:22:43.447 INFO:tasks.rados.rados.0.smithi076.stdout:769: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2021-03-15T22:22:43.451 INFO:tasks.rados.rados.0.smithi076.stdout:769: writing smithi07694878-14 from 675714 to 1138679 tid 1 2021-03-15T22:22:43.456 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:43 smithi125 conmon[56739]: cluster 2021-03-15T22:22:41.941851 2021-03-15T22:22:43.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:43 smithi125 conmon[56739]: +0000 osd.4 (osd.4) 59 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:22:43.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:43 smithi125 conmon[56739]: cluster 2021-03-15T22:22:43.048536+0000 mon.a (mon.0) 913 : cluster [DBG] osdmap e248: 8 total, 8 up, 7 in 2021-03-15T22:22:43.458 INFO:tasks.rados.rados.0.smithi076.stdout:769: writing smithi07694878-14 from 1589209 to 2230639 tid 2 2021-03-15T22:22:43.459 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:22:43.462 INFO:tasks.rados.rados.0.smithi076.stdout:769: writing smithi07694878-14 from 2943979 to 3075714 tid 3 2021-03-15T22:22:43.463 INFO:tasks.rados.rados.0.smithi076.stdout:770: write (excl) oid 22 current snap is 78 2021-03-15T22:22:43.463 INFO:tasks.rados.rados.0.smithi076.stdout:767: finishing copy_from to smithi07694878-19 2021-03-15T22:22:43.463 INFO:tasks.rados.rados.0.smithi076.stdout:767: got expected ENOENT (src dne) 2021-03-15T22:22:43.463 INFO:tasks.rados.rados.0.smithi076.stdout:767: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:22:43.464 INFO:tasks.rados.rados.0.smithi076.stdout:770: seq_num 303 ranges {692521=538214,1931338=785088} 2021-03-15T22:22:43.469 INFO:tasks.rados.rados.0.smithi076.stdout:770: writing smithi07694878-22 from 692521 to 1230735 tid 1 2021-03-15T22:22:43.479 INFO:tasks.rados.rados.0.smithi076.stdout:770: writing smithi07694878-22 from 1931338 to 2716426 tid 2 2021-03-15T22:22:43.479 INFO:tasks.rados.rados.0.smithi076.stdout:767: done (3 left) 2021-03-15T22:22:43.479 INFO:tasks.rados.rados.0.smithi076.stdout:771: copy_from oid 18 from oid 15 current snap is 78 2021-03-15T22:22:43.479 INFO:tasks.rados.rados.0.smithi076.stdout:768: finishing rollback tid 0 to smithi07694878-44 2021-03-15T22:22:43.480 INFO:tasks.rados.rados.0.smithi076.stdout:768: finishing rollback tid 1 to smithi07694878-44 2021-03-15T22:22:43.480 INFO:tasks.rados.rados.0.smithi076.stdout:768: finishing rollback tid 2 to smithi07694878-44 2021-03-15T22:22:43.480 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 149 (ObjNum 156 snap 35 seq_num 156) dirty exists 2021-03-15T22:22:43.480 INFO:tasks.rados.rados.0.smithi076.stdout:769: finishing write tid 1 to smithi07694878-14 2021-03-15T22:22:43.481 INFO:tasks.rados.rados.0.smithi076.stdout:769: finishing write tid 2 to smithi07694878-14 2021-03-15T22:22:43.481 INFO:tasks.rados.rados.0.smithi076.stdout:769: finishing write tid 3 to smithi07694878-14 2021-03-15T22:22:43.481 INFO:tasks.rados.rados.0.smithi076.stdout:769: finishing write tid 4 to smithi07694878-14 2021-03-15T22:22:43.481 INFO:tasks.rados.rados.0.smithi076.stdout:769: finishing write tid 5 to smithi07694878-14 2021-03-15T22:22:43.482 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 264 (ObjNum 302 snap 78 seq_num 302) dirty exists 2021-03-15T22:22:43.482 INFO:tasks.rados.rados.0.smithi076.stdout:769: left oid 14 (ObjNum 302 snap 78 seq_num 302) 2021-03-15T22:22:43.482 INFO:tasks.rados.rados.0.smithi076.stdout:770: finishing write tid 1 to smithi07694878-22 2021-03-15T22:22:43.482 INFO:tasks.rados.rados.0.smithi076.stdout:768: done (3 left) 2021-03-15T22:22:43.483 INFO:tasks.rados.rados.0.smithi076.stdout:769: done (2 left) 2021-03-15T22:22:43.483 INFO:tasks.rados.rados.0.smithi076.stdout:772: rollback oid 30 current snap is 78 2021-03-15T22:22:43.483 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 30 to 72 2021-03-15T22:22:43.483 INFO:tasks.rados.rados.0.smithi076.stdout:773: delete oid 45 current snap is 78 2021-03-15T22:22:43.484 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:43.485 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:22:43.485 INFO:tasks.rados.rados.0.smithi076.stdout:773: done (3 left) 2021-03-15T22:22:43.485 INFO:tasks.rados.rados.0.smithi076.stdout:774: write oid 40 current snap is 78 2021-03-15T22:22:43.486 INFO:tasks.rados.rados.0.smithi076.stdout:774: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2021-03-15T22:22:43.486 INFO:tasks.rados.rados.0.smithi076.stdout:774: writing smithi07694878-40 from 726135 to 1131200 tid 1 2021-03-15T22:22:43.497 INFO:tasks.rados.rados.0.smithi076.stdout:774: writing smithi07694878-40 from 1764655 to 2553412 tid 2 2021-03-15T22:22:43.501 INFO:tasks.rados.rados.0.smithi076.stdout:774: writing smithi07694878-40 from 3026248 to 3126135 tid 3 2021-03-15T22:22:43.501 INFO:tasks.rados.rados.0.smithi076.stdout:771: finishing copy_from to smithi07694878-18 2021-03-15T22:22:43.502 INFO:tasks.rados.rados.0.smithi076.stdout:771: got expected ENOENT (src dne) 2021-03-15T22:22:43.502 INFO:tasks.rados.rados.0.smithi076.stdout:771: finishing copy_from racing read to smithi07694878-18 2021-03-15T22:22:43.502 INFO:tasks.rados.rados.0.smithi076.stdout:770: finishing write tid 2 to smithi07694878-22 2021-03-15T22:22:43.502 INFO:tasks.rados.rados.0.smithi076.stdout:770: finishing write tid 3 to smithi07694878-22 2021-03-15T22:22:43.503 INFO:tasks.rados.rados.0.smithi076.stdout:770: finishing write tid 4 to smithi07694878-22 2021-03-15T22:22:43.503 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 366 (ObjNum 303 snap 78 seq_num 303) dirty exists 2021-03-15T22:22:43.503 INFO:tasks.rados.rados.0.smithi076.stdout:770: left oid 22 (ObjNum 303 snap 78 seq_num 303) 2021-03-15T22:22:43.503 INFO:tasks.rados.rados.0.smithi076.stdout:772: finishing rollback tid 0 to smithi07694878-30 2021-03-15T22:22:43.504 INFO:tasks.rados.rados.0.smithi076.stdout:772: finishing rollback tid 1 to smithi07694878-30 2021-03-15T22:22:43.504 INFO:tasks.rados.rados.0.smithi076.stdout:772: finishing rollback tid 2 to smithi07694878-30 2021-03-15T22:22:43.504 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 370 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:22:43.504 INFO:tasks.rados.rados.0.smithi076.stdout:774: finishing write tid 1 to smithi07694878-40 2021-03-15T22:22:43.505 INFO:tasks.rados.rados.0.smithi076.stdout:770: done (3 left) 2021-03-15T22:22:43.505 INFO:tasks.rados.rados.0.smithi076.stdout:771: done (2 left) 2021-03-15T22:22:43.505 INFO:tasks.rados.rados.0.smithi076.stdout:772: done (1 left) 2021-03-15T22:22:43.505 INFO:tasks.rados.rados.0.smithi076.stdout:775: read oid 32 snap -1 2021-03-15T22:22:43.505 INFO:tasks.rados.rados.0.smithi076.stdout:775: expect (ObjNum 265 snap 72 seq_num 265) 2021-03-15T22:22:43.506 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:22:43.507 INFO:tasks.rados.rados.0.smithi076.stdout:776: read oid 9 snap 70 2021-03-15T22:22:43.507 INFO:tasks.rados.rados.0.smithi076.stdout:776: expect deleted 2021-03-15T22:22:43.507 INFO:tasks.rados.rados.0.smithi076.stdout:777: read oid 22 snap -1 2021-03-15T22:22:43.507 INFO:tasks.rados.rados.0.smithi076.stdout:777: expect (ObjNum 303 snap 78 seq_num 303) 2021-03-15T22:22:43.508 INFO:tasks.rados.rados.0.smithi076.stdout:778: read oid 27 snap -1 2021-03-15T22:22:43.508 INFO:tasks.rados.rados.0.smithi076.stdout:778: expect deleted 2021-03-15T22:22:43.508 INFO:tasks.rados.rados.0.smithi076.stdout:779: snap_create 2021-03-15T22:22:43.508 INFO:tasks.rados.rados.0.smithi076.stdout:774: finishing write tid 2 to smithi07694878-40 2021-03-15T22:22:43.509 INFO:tasks.rados.rados.0.smithi076.stdout:777: expect (ObjNum 303 snap 78 seq_num 303) 2021-03-15T22:22:43.512 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:43.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:43.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:43.526 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:22:43.544 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:22:43.572 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:22:43.593 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:22:43.595 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:22:43.595 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:22:43.595 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:22:43.595 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:22:43.596 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:22:43.596 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:22:43.596 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:22:43.596 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:22:43.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:43.636 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:43.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:43.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:43.649 INFO:tasks.rados.rados.0.smithi076.stdout:775: expect (ObjNum 265 snap 72 seq_num 265) 2021-03-15T22:22:43.739 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:43.762 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:43.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:43.764 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:43.777 INFO:tasks.rados.rados.0.smithi076.stdout:774: finishing write tid 3 to smithi07694878-40 2021-03-15T22:22:43.777 INFO:tasks.rados.rados.0.smithi076.stdout:774: finishing write tid 4 to smithi07694878-40 2021-03-15T22:22:43.777 INFO:tasks.rados.rados.0.smithi076.stdout:774: finishing write tid 5 to smithi07694878-40 2021-03-15T22:22:43.778 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 438 (ObjNum 305 snap 78 seq_num 305) dirty exists 2021-03-15T22:22:43.778 INFO:tasks.rados.rados.0.smithi076.stdout:774: left oid 40 (ObjNum 305 snap 78 seq_num 305) 2021-03-15T22:22:43.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:43.886 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:43.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:43.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:43.989 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:44.011 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:44.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:44.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:44.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:44.135 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:44.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:44.200 INFO:tasks.rados.rados.0.smithi076.stdout:774: done (5 left) 2021-03-15T22:22:44.201 INFO:tasks.rados.rados.0.smithi076.stdout:775: done (4 left) 2021-03-15T22:22:44.201 INFO:tasks.rados.rados.0.smithi076.stdout:776: done (3 left) 2021-03-15T22:22:44.201 INFO:tasks.rados.rados.0.smithi076.stdout:777: done (2 left) 2021-03-15T22:22:44.201 INFO:tasks.rados.rados.0.smithi076.stdout:778: done (1 left) 2021-03-15T22:22:44.202 INFO:tasks.rados.rados.0.smithi076.stdout:779: done (0 left) 2021-03-15T22:22:44.202 INFO:tasks.rados.rados.0.smithi076.stdout:780: delete oid 5 current snap is 79 2021-03-15T22:22:44.203 INFO:tasks.rados.rados.0.smithi076.stdout:780: done (0 left) 2021-03-15T22:22:44.203 INFO:tasks.rados.rados.0.smithi076.stdout:781: rollback oid 43 current snap is 79 2021-03-15T22:22:44.203 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 43 to 76 2021-03-15T22:22:44.203 INFO:tasks.rados.rados.0.smithi076.stdout:782: write (excl) oid 28 current snap is 79 2021-03-15T22:22:44.204 INFO:tasks.rados.rados.0.smithi076.stdout:782: seq_num 307 ranges {759749=555563,1765266=792426,2964741=195008} 2021-03-15T22:22:44.210 INFO:tasks.rados.rados.0.smithi076.stdout:782: writing smithi07694878-28 from 759749 to 1315312 tid 1 2021-03-15T22:22:44.217 INFO:tasks.rados.rados.0.smithi076.stdout:782: writing smithi07694878-28 from 1765266 to 2557692 tid 2 2021-03-15T22:22:44.221 INFO:tasks.rados.rados.0.smithi076.stdout:782: writing smithi07694878-28 from 2964741 to 3159749 tid 3 2021-03-15T22:22:44.221 INFO:tasks.rados.rados.0.smithi076.stdout:783: snap_create 2021-03-15T22:22:44.221 INFO:tasks.rados.rados.0.smithi076.stdout:781: finishing rollback tid 1 to smithi07694878-43 2021-03-15T22:22:44.221 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 0 (ObjNum 0 snap 21896 seq_num 261842912) dirty dne 2021-03-15T22:22:44.222 INFO:tasks.rados.rados.0.smithi076.stdout:782: finishing write tid 1 to smithi07694878-28 2021-03-15T22:22:44.223 INFO:tasks.rados.rados.0.smithi076.stdout:782: finishing write tid 2 to smithi07694878-28 2021-03-15T22:22:44.223 INFO:tasks.rados.rados.0.smithi076.stdout:782: finishing write tid 3 to smithi07694878-28 2021-03-15T22:22:44.223 INFO:tasks.rados.rados.0.smithi076.stdout:782: finishing write tid 4 to smithi07694878-28 2021-03-15T22:22:44.223 INFO:tasks.rados.rados.0.smithi076.stdout:782: finishing write tid 5 to smithi07694878-28 2021-03-15T22:22:44.224 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 442 (ObjNum 307 snap 79 seq_num 307) dirty exists 2021-03-15T22:22:44.224 INFO:tasks.rados.rados.0.smithi076.stdout:782: left oid 28 (ObjNum 307 snap 79 seq_num 307) 2021-03-15T22:22:44.239 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:44.260 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:44.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:44.262 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:44.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:44.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:44.388 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:44.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:44.458 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:44 smithi076 conmon[56514]: cluster 2021-03-15T22:22:43.013930+0000 mgr.y (mgr.14144) 481 : cluster [DBG] pgmap v498: 138 pgs: 138 active+clean; 139 MiB data, 482 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 1.5 MiB/s wr, 21 op/s 2021-03-15T22:22:44.458 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:44 smithi076 conmon[56514]: audit 2021-03-15T22:22:43.422111+0000 mon.a (mon.0) 914 : audit [DBG] from='client.? 172.21.15.76:0/3857916726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:44.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:44 smithi076 conmon[62156]: cluster 2021-03-15T22:22:43.013930+0000 mgr.y (mgr.14144) 481 : cluster [DBG] pgmap v498: 138 pgs: 138 active+clean; 139 MiB data, 482 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 1.5 MiB/s wr, 21 op/s 2021-03-15T22:22:44.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:44 smithi076 conmon[62156]: audit 2021-03-15T22:22:43.422111+0000 mon.a (mon.0) 914 : audit [DBG] from='client.? 172.21.15.76:0/3857916726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:44.489 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:44.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:44.512 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:44.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:44.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:44 smithi125 conmon[56739]: cluster 2021-03-15T22:22:43.013930+0000 mgr.y (mgr.14144) 481 : cluster [DBG] pgmap v498: 138 pgs: 138 active+clean; 139 MiB data, 482 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 1.5 MiB/s wr, 21 op/s 2021-03-15T22:22:44.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:44 smithi125 conmon[56739]: audit 2021-03-15T22:22:43.422111+0000 mon.a (mon.0) 914 : audit [DBG] from='client.? 172.21.15.76:0/3857916726' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:44.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:44.636 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:44.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:44.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:44.739 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:44.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:44.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:44.781 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:22:44.782 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:22:44.782 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:22:44.783 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:22:44.783 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:22:44.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:44.890 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:44.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:44.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:44.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:45.016 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:45.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:45.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:45.046 INFO:tasks.rados.rados.0.smithi076.stdout:781: done (2 left) 2021-03-15T22:22:45.046 INFO:tasks.rados.rados.0.smithi076.stdout:782: done (1 left) 2021-03-15T22:22:45.047 INFO:tasks.rados.rados.0.smithi076.stdout:783: done (0 left) 2021-03-15T22:22:45.047 INFO:tasks.rados.rados.0.smithi076.stdout:784: read oid 47 snap 75 2021-03-15T22:22:45.047 INFO:tasks.rados.rados.0.smithi076.stdout:784: expect (ObjNum 259 snap 67 seq_num 259) 2021-03-15T22:22:45.048 INFO:tasks.rados.rados.0.smithi076.stdout:785: write (excl) oid 12 current snap is 80 2021-03-15T22:22:45.048 INFO:tasks.rados.rados.0.smithi076.stdout:785: seq_num 308 ranges {776556=630812,1823748=652437,3108517=68039} 2021-03-15T22:22:45.053 INFO:tasks.rados.rados.0.smithi076.stdout:785: writing smithi07694878-12 from 776556 to 1407368 tid 1 2021-03-15T22:22:45.058 INFO:tasks.rados.rados.0.smithi076.stdout:785: writing smithi07694878-12 from 1823748 to 2476185 tid 2 2021-03-15T22:22:45.063 INFO:tasks.rados.rados.0.smithi076.stdout:785: writing smithi07694878-12 from 3108517 to 3176556 tid 3 2021-03-15T22:22:45.064 INFO:tasks.rados.rados.0.smithi076.stdout:786: read oid 29 snap -1 2021-03-15T22:22:45.064 INFO:tasks.rados.rados.0.smithi076.stdout:786: expect (ObjNum 261 snap 68 seq_num 261) 2021-03-15T22:22:45.064 INFO:tasks.rados.rados.0.smithi076.stdout:787: read oid 39 snap 63 2021-03-15T22:22:45.064 INFO:tasks.rados.rados.0.smithi076.stdout:787: expect deleted 2021-03-15T22:22:45.065 INFO:tasks.rados.rados.0.smithi076.stdout:788: snap_remove snap 79 2021-03-15T22:22:45.120 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:45.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:45.143 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:45.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:45.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:22:45.245 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:45.269 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:45.269 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:45.371 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:45.395 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:45.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:45.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-15T22:22:45.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:45 smithi125 conmon[56739]: cluster 2021-03-15T22:22:44.198601+0000 mon.a (mon.0) 915 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2021-03-15T22:22:45.469 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:45 smithi125 conmon[56739]: cluster 2021-03-15T22:22 2021-03-15T22:22:45.469 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:45 smithi125 conmon[56739]: :45.041059+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2021-03-15T22:22:45.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:45 smithi076 conmon[56514]: cluster 2021-03-15T22:22:44.198601+0000 mon.a (mon 2021-03-15T22:22:45.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:45 smithi076 conmon[56514]: .0) 915 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2021-03-15T22:22:45.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:45 smithi076 conmon[56514]: cluster 2021-03-15T22:22:45.041059+0000 mon.a (mon.0) 916 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2021-03-15T22:22:45.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:45 smithi076 conmon[62156]: cluster 2021-03-15T22:22:44.198601+0000 mon.a (mon. 2021-03-15T22:22:45.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:45 smithi076 conmon[62156]: 0) 915 : cluster [DBG] osdmap e249: 8 total, 8 up, 7 in 2021-03-15T22:22:45.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:45 smithi076 conmon[62156]: cluster 2021-03-15T22:22:45.041059+0000 mon.a 2021-03-15T22:22:45.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:45 smithi076 conmon[62156]: (mon.0) 916 : cluster [DBG] osdmap e250: 8 total, 8 up, 7 in 2021-03-15T22:22:45.498 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:45.521 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:45.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:45.610 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:45.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:45.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:45.651 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:45.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:45.721 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:22:45.752 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:45.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:45.776 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:45.777 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:22:45.852 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:22:45.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:45.901 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:45.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:46.003 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:46.027 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:46.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:46.051 INFO:tasks.rados.rados.0.smithi076.stdout:788: done (4 left) 2021-03-15T22:22:46.051 INFO:tasks.rados.rados.0.smithi076.stdout:789: rollback oid 4 current snap is 80 2021-03-15T22:22:46.052 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 4 to 75 2021-03-15T22:22:46.055 INFO:tasks.rados.rados.0.smithi076.stdout:790: read oid 15 snap -1 2021-03-15T22:22:46.055 INFO:tasks.rados.rados.0.smithi076.stdout:790: expect deleted 2021-03-15T22:22:46.056 INFO:tasks.rados.rados.0.smithi076.stdout:791: write oid 36 current snap is 80 2021-03-15T22:22:46.056 INFO:tasks.rados.rados.0.smithi076.stdout:791: seq_num 309 ranges {793363=706061,2282230=796095} 2021-03-15T22:22:46.059 INFO:tasks.rados.rados.0.smithi076.stdout:791: writing smithi07694878-36 from 793363 to 1499424 tid 1 2021-03-15T22:22:46.069 INFO:tasks.rados.rados.0.smithi076.stdout:791: writing smithi07694878-36 from 2282230 to 3078325 tid 2 2021-03-15T22:22:46.069 INFO:tasks.rados.rados.0.smithi076.stdout:792: read oid 6 snap -1 2021-03-15T22:22:46.069 INFO:tasks.rados.rados.0.smithi076.stdout:792: expect deleted 2021-03-15T22:22:46.070 INFO:tasks.rados.rados.0.smithi076.stdout:793: delete oid 38 current snap is 80 2021-03-15T22:22:46.070 INFO:tasks.rados.rados.0.smithi076.stdout:784: expect (ObjNum 259 snap 67 seq_num 259) 2021-03-15T22:22:46.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:46.152 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:46.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:46.226 INFO:tasks.rados.rados.0.smithi076.stdout:785: finishing write tid 1 to smithi07694878-12 2021-03-15T22:22:46.227 INFO:tasks.rados.rados.0.smithi076.stdout:785: finishing write tid 2 to smithi07694878-12 2021-03-15T22:22:46.227 INFO:tasks.rados.rados.0.smithi076.stdout:785: finishing write tid 3 to smithi07694878-12 2021-03-15T22:22:46.227 INFO:tasks.rados.rados.0.smithi076.stdout:784: done (8 left) 2021-03-15T22:22:46.227 INFO:tasks.rados.rados.0.smithi076.stdout:787: done (7 left) 2021-03-15T22:22:46.228 INFO:tasks.rados.rados.0.smithi076.stdout:793: done (6 left) 2021-03-15T22:22:46.228 INFO:tasks.rados.rados.0.smithi076.stdout:794: snap_create 2021-03-15T22:22:46.228 INFO:tasks.rados.rados.0.smithi076.stdout:785: finishing write tid 4 to smithi07694878-12 2021-03-15T22:22:46.228 INFO:tasks.rados.rados.0.smithi076.stdout:785: finishing write tid 5 to smithi07694878-12 2021-03-15T22:22:46.229 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 374 (ObjNum 308 snap 80 seq_num 308) dirty exists 2021-03-15T22:22:46.229 INFO:tasks.rados.rados.0.smithi076.stdout:785: left oid 12 (ObjNum 308 snap 80 seq_num 308) 2021-03-15T22:22:46.229 INFO:tasks.rados.rados.0.smithi076.stdout:786: expect (ObjNum 261 snap 68 seq_num 261) 2021-03-15T22:22:46.255 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:46.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:46.282 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:46.283 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:46.335 INFO:tasks.rados.rados.0.smithi076.stdout:789: finishing rollback tid 0 to smithi07694878-4 2021-03-15T22:22:46.335 INFO:tasks.rados.rados.0.smithi076.stdout:789: finishing rollback tid 1 to smithi07694878-4 2021-03-15T22:22:46.335 INFO:tasks.rados.rados.0.smithi076.stdout:789: finishing rollback tid 2 to smithi07694878-4 2021-03-15T22:22:46.336 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 255 (ObjNum 239 snap 57 seq_num 239) dirty exists 2021-03-15T22:22:46.336 INFO:tasks.rados.rados.0.smithi076.stdout:791: finishing write tid 1 to smithi07694878-36 2021-03-15T22:22:46.336 INFO:tasks.rados.rados.0.smithi076.stdout:791: finishing write tid 2 to smithi07694878-36 2021-03-15T22:22:46.336 INFO:tasks.rados.rados.0.smithi076.stdout:791: finishing write tid 3 to smithi07694878-36 2021-03-15T22:22:46.337 INFO:tasks.rados.rados.0.smithi076.stdout:791: finishing write tid 4 to smithi07694878-36 2021-03-15T22:22:46.337 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 258 (ObjNum 309 snap 80 seq_num 309) dirty exists 2021-03-15T22:22:46.337 INFO:tasks.rados.rados.0.smithi076.stdout:791: left oid 36 (ObjNum 309 snap 80 seq_num 309) 2021-03-15T22:22:46.384 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:46.407 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:46.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:46.510 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:46.515 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:22:46.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56514]: cluster 2021-03-15T22:22:45.014398+0000 mgr.y (mgr.14144) 482 : cluster [DBG] pgmap v501: 138 pgs: 138 active+clean; 150 MiB data, 494 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 1.5 MiB/s wr, 25 op/s 2021-03-15T22:22:46.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56514]: audit 2021-03-15T22:22:45.830370+0000 mgr.y (mgr.14144) 483 : audit [DBG] from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:22:46.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56514]: audit 2021-03-15T22:22:45.973953+0000 mon.a (mon.0) 917 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:22:46.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56514]: cluster 2021-03-15T22:22:46.048882+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2021-03-15T22:22:46.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[62156]: cluster 2021-03-15T22:22:45.014398+0000 mgr.y (mgr.14144) 482 : cluster [DBG] pgmap v501: 138 pgs: 138 active+clean; 150 MiB data, 494 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 1.5 MiB/s wr, 25 op/s 2021-03-15T22:22:46.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[62156]: audit 2021-03-15T22:22:45.830370+0000 mgr.y (mgr.14144) 483 : audit [DBG] from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:22:46.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[62156]: audit 2021-03-15T22:22:45.973953+0000 mon.a (mon.0) 917 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:22:46.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[62156]: cluster 2021-03-15T22:22:46.048882+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2021-03-15T22:22:46.519 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: debug 2021-03-15T22:22:46.263+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:22:46.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:22:46.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:46.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:22:46.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:46.520 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:22:46.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:46.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:46.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:22:46.521 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: debug 2021-03-15T22:22:46.264+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:22:46.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:22:46.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:46.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:22:46.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:46.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:22:46.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:46.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:46.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:22:46.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: debug 2021-03-15T22:22:46.264+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:22:46.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:22:46.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:46.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:22:46.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:46.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:22:46.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:46.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:46.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:22:46.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: debug 2021-03-15T22:22:46.264+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:22:46.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:22:46.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:46.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:22:46.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:46.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:22:46.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:46.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:46.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:22:46.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: debug 2021-03-15T22:22:46.265+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:22:46.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:22:46.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:46.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:22:46.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:46.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:22:46.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:46.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:46.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:22:46.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: debug 2021-03-15T22:22:46.265+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:22:46.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:22:46.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:46.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:22:46.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:46.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:22:46.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:46.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:46.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:22:46.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: debug 2021-03-15T22:22:46.266+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:22:46.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:22:46.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:46.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:22:46.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:46.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:22:46.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:46.533 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:46.533 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:22:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:22:46.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:46.534 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:46.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:46.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:46 smithi125 conmon[56739]: cluster 2021-03-15T22:22:45.014398+0000 mgr.y (mgr.14144) 482 : cluster [DBG] pgmap v501: 138 pgs: 138 active+clean; 150 MiB data, 494 MiB used, 625 GiB / 626 GiB avail; 19 MiB/s rd, 1.5 MiB/s wr, 25 op/s 2021-03-15T22:22:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:46 smithi125 conmon[56739]: audit 2021-03-15T22:22:45.830370+0000 mgr.y (mgr.14144) 483 : audit [DBG] from='client.15081 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:22:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:46 smithi125 conmon[56739]: audit 2021-03-15T22:22:45.973953+0000 mon.a (mon.0) 917 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:22:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:46 smithi125 conmon[56739]: cluster 2021-03-15T22:22:46.048882+0000 mon.a (mon.0) 918 : cluster [DBG] osdmap e251: 8 total, 8 up, 7 in 2021-03-15T22:22:46.636 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:46.663 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:46.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:46.751 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":502,"stamp":"2021-03-15T22:22:45.038602+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":157325497,"num_objects":393,"num_object_clones":38,"num_object_copies":1179,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":393,"num_whiteouts":5,"num_read":2896,"num_read_kb":1493762,"num_write":1719,"num_write_kb":438707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":85238491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3235,"ondisk_log_size":3235,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":505808,"kb_used_data":375192,"kb_used_omap":0,"kb_used_meta":130560,"kb_avail":655710256,"statfs":{"total":671965249536,"available":671447302144,"internally_reserved":0,"allocated":384196608,"data_stored":496285559,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":133693440},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14174502,"num_objects":9,"num_object_clones":9,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":68,"num_read_kb":78135,"num_write":34,"num_write_kb":6184,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.997657"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"202","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054512+0000","last_change":"2021-03-15T22:21:06.404346+0000","last_active":"2021-03-15T22:22:40.054512+0000","last_peered":"2021-03-15T22:22:40.054512+0000","last_clean":"2021-03-15T22:22:40.054512+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:22:40.054512+0000","last_undegraded":"2021-03-15T22:22:40.054512+0000","last_fullsized":"2021-03-15T22:22:40.054512+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:06.404281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:06.404281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"209","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177905+0000","last_change":"2021-03-15T22:21:22.794410+0000","last_active":"2021-03-15T22:22:39.177905+0000","last_peered":"2021-03-15T22:22:39.177905+0000","last_clean":"2021-03-15T22:22:39.177905+0000","last_became_active":"2021-03-15T22:17:54.284855+0000","last_became_peered":"2021-03-15T22:17:54.284855+0000","last_unstale":"2021-03-15T22:22:39.177905+0000","last_undegraded":"2021-03-15T22:22:39.177905+0000","last_fullsized":"2021-03-15T22:22:39.177905+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:21:22.794340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:22.794340+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"234","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054437+0000","last_change":"2021-03-15T22:20:51.316325+0000","last_active":"2021-03-15T22:22:40.054437+0000","last_peered":"2021-03-15T22:22:40.054437+0000","last_clean":"2021-03-15T22:22:40.054437+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:22:40.054437+0000","last_undegraded":"2021-03-15T22:22:40.054437+0000","last_fullsized":"2021-03-15T22:22:40.054437+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:20:51.316237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:20:51.316237+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"91","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.043592+0000","last_change":"2021-03-15T22:21:49.846475+0000","last_active":"2021-03-15T22:22:42.043592+0000","last_peered":"2021-03-15T22:22:42.043592+0000","last_clean":"2021-03-15T22:22:42.043592+0000","last_became_active":"2021-03-15T22:21:28.039958+0000","last_became_peered":"2021-03-15T22:21:28.039958+0000","last_unstale":"2021-03-15T22:22:42.043592+0000","last_undegraded":"2021-03-15T22:22:42.043592+0000","last_fullsized":"2021-03-15T22:22:42.043592+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:49.846392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:49.846392+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"88","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.532802+0000","last_change":"2021-03-15T22:22:32.631519+0000","last_active":"2021-03-15T22:22:39.532802+0000","last_peered":"2021-03-15T22:22:39.532802+0000","last_clean":"2021-03-15T22:22:39.532802+0000","last_became_active":"2021-03-15T22:21:28.043528+0000","last_became_peered":"2021-03-15T22:21:28.043528+0000","last_unstale":"2021-03-15T22:22:39.532802+0000","last_undegraded":"2021-03-15T22:22:39.532802+0000","last_fullsized":"2021-03-15T22:22:39.532802+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:32.631430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:32.631430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"166","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.047381+0000","last_change":"2021-03-15T22:21:30.473634+0000","last_active":"2021-03-15T22:22:43.047381+0000","last_peered":"2021-03-15T22:22:43.047381+0000","last_clean":"2021-03-15T22:22:43.047381+0000","last_became_active":"2021-03-15T22:21:28.049801+0000","last_became_peered":"2021-03-15T22:21:28.049801+0000","last_unstale":"2021-03-15T22:22:43.047381+0000","last_undegraded":"2021-03-15T22:22:43.047381+0000","last_fullsized":"2021-03-15T22:22:43.047381+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:21:30.473571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:30.473571+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":21,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"200","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.176520+0000","last_change":"2021-03-15T22:20:33.569323+0000","last_active":"2021-03-15T22:22:39.176520+0000","last_peered":"2021-03-15T22:22:39.176520+0000","last_clean":"2021-03-15T22:22:39.176520+0000","last_became_active":"2021-03-15T22:18:00.015776+0000","last_became_peered":"2021-03-15T22:18:00.015776+0000","last_unstale":"2021-03-15T22:22:39.176520+0000","last_undegraded":"2021-03-15T22:22:39.176520+0000","last_fullsized":"2021-03-15T22:22:39.176520+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:33.569266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:33.569266+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"211","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.048357+0000","last_change":"2021-03-15T22:22:40.167726+0000","last_active":"2021-03-15T22:22:41.048357+0000","last_peered":"2021-03-15T22:22:41.048357+0000","last_clean":"2021-03-15T22:22:41.048357+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:22:41.048357+0000","last_undegraded":"2021-03-15T22:22:41.048357+0000","last_fullsized":"2021-03-15T22:22:41.048357+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:40.167663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:40.167663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"257","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054603+0000","last_change":"2021-03-15T22:22:02.316373+0000","last_active":"2021-03-15T22:22:40.054603+0000","last_peered":"2021-03-15T22:22:40.054603+0000","last_clean":"2021-03-15T22:22:40.054603+0000","last_became_active":"2021-03-15T22:21:56.030245+0000","last_became_peered":"2021-03-15T22:21:56.030245+0000","last_unstale":"2021-03-15T22:22:40.054603+0000","last_undegraded":"2021-03-15T22:22:40.054603+0000","last_fullsized":"2021-03-15T22:22:40.054603+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:22:02.316263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:02.316263+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"209","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.047511+0000","last_change":"2021-03-15T22:21:53.179241+0000","last_active":"2021-03-15T22:22:41.047511+0000","last_peered":"2021-03-15T22:22:41.047511+0000","last_clean":"2021-03-15T22:22:41.047511+0000","last_became_active":"2021-03-15T22:21:28.045904+0000","last_became_peered":"2021-03-15T22:21:28.045904+0000","last_unstale":"2021-03-15T22:22:41.047511+0000","last_undegraded":"2021-03-15T22:22:41.047511+0000","last_fullsized":"2021-03-15T22:22:41.047511+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:53.179151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:53.179151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"211","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.168972+0000","last_change":"2021-03-15T22:22:41.168947+0000","last_active":"2021-03-15T22:22:41.168972+0000","last_peered":"2021-03-15T22:22:41.168972+0000","last_clean":"2021-03-15T22:22:41.168972+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:22:41.168972+0000","last_undegraded":"2021-03-15T22:22:41.168972+0000","last_fullsized":"2021-03-15T22:22:41.168972+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:41.168884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:41.168884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"121'9","reported_seq":"260","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054896+0000","last_change":"2021-03-15T22:22:31.872540+0000","last_active":"2021-03-15T22:22:40.054896+0000","last_peered":"2021-03-15T22:22:40.054896+0000","last_clean":"2021-03-15T22:22:40.054896+0000","last_became_active":"2021-03-15T22:21:56.035207+0000","last_became_peered":"2021-03-15T22:21:56.035207+0000","last_unstale":"2021-03-15T22:22:40.054896+0000","last_undegraded":"2021-03-15T22:22:40.054896+0000","last_fullsized":"2021-03-15T22:22:40.054896+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'9","last_scrub_stamp":"2021-03-15T22:22:31.872458+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:22:31.872458+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"202","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.055821+0000","last_change":"2021-03-15T22:22:08.294925+0000","last_active":"2021-03-15T22:22:40.055821+0000","last_peered":"2021-03-15T22:22:40.055821+0000","last_clean":"2021-03-15T22:22:40.055821+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:22:40.055821+0000","last_undegraded":"2021-03-15T22:22:40.055821+0000","last_fullsized":"2021-03-15T22:22:40.055821+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:08.294855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:08.294855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"209","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177422+0000","last_change":"2021-03-15T22:21:25.786047+0000","last_active":"2021-03-15T22:22:39.177422+0000","last_peered":"2021-03-15T22:22:39.177422+0000","last_clean":"2021-03-15T22:22:39.177422+0000","last_became_active":"2021-03-15T22:17:53.976875+0000","last_became_peered":"2021-03-15T22:17:53.976875+0000","last_unstale":"2021-03-15T22:22:39.177422+0000","last_undegraded":"2021-03-15T22:22:39.177422+0000","last_fullsized":"2021-03-15T22:22:39.177422+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:21:25.785981+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:21:25.785981+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"258","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.048871+0000","last_change":"2021-03-15T22:21:29.556475+0000","last_active":"2021-03-15T22:22:43.048871+0000","last_peered":"2021-03-15T22:22:43.048871+0000","last_clean":"2021-03-15T22:22:43.048871+0000","last_became_active":"2021-03-15T22:21:28.049386+0000","last_became_peered":"2021-03-15T22:21:28.049386+0000","last_unstale":"2021-03-15T22:22:43.048871+0000","last_undegraded":"2021-03-15T22:22:43.048871+0000","last_fullsized":"2021-03-15T22:22:43.048871+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:21:00.125316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:00.125316+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"61","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.532906+0000","last_change":"2021-03-15T22:22:04.618748+0000","last_active":"2021-03-15T22:22:39.532906+0000","last_peered":"2021-03-15T22:22:39.532906+0000","last_clean":"2021-03-15T22:22:39.532906+0000","last_became_active":"2021-03-15T22:21:56.035432+0000","last_became_peered":"2021-03-15T22:21:56.035432+0000","last_unstale":"2021-03-15T22:22:39.532906+0000","last_undegraded":"2021-03-15T22:22:39.532906+0000","last_fullsized":"2021-03-15T22:22:39.532906+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:04.618673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:04.618673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"208","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054606+0000","last_change":"2021-03-15T22:20:53.271135+0000","last_active":"2021-03-15T22:22:40.054606+0000","last_peered":"2021-03-15T22:22:40.054606+0000","last_clean":"2021-03-15T22:22:40.054606+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:22:40.054606+0000","last_undegraded":"2021-03-15T22:22:40.054606+0000","last_fullsized":"2021-03-15T22:22:40.054606+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:53.271063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:53.271063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"154","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.048610+0000","last_change":"2021-03-15T22:21:49.930676+0000","last_active":"2021-03-15T22:22:41.048610+0000","last_peered":"2021-03-15T22:22:41.048610+0000","last_clean":"2021-03-15T22:22:41.048610+0000","last_became_active":"2021-03-15T22:21:28.045170+0000","last_became_peered":"2021-03-15T22:21:28.045170+0000","last_unstale":"2021-03-15T22:22:41.048610+0000","last_undegraded":"2021-03-15T22:22:41.048610+0000","last_fullsized":"2021-03-15T22:22:41.048610+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:21:49.930593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:49.930593+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":36,"num_read_kb":19,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"201","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.044014+0000","last_change":"2021-03-15T22:21:03.982348+0000","last_active":"2021-03-15T22:22:42.044014+0000","last_peered":"2021-03-15T22:22:42.044014+0000","last_clean":"2021-03-15T22:22:42.044014+0000","last_became_active":"2021-03-15T22:18:00.015889+0000","last_became_peered":"2021-03-15T22:18:00.015889+0000","last_unstale":"2021-03-15T22:22:42.044014+0000","last_undegraded":"2021-03-15T22:22:42.044014+0000","last_fullsized":"2021-03-15T22:22:42.044014+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:03.982279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:03.982279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"221","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.048341+0000","last_change":"2021-03-15T22:22:02.880850+0000","last_active":"2021-03-15T22:22:41.048341+0000","last_peered":"2021-03-15T22:22:41.048341+0000","last_clean":"2021-03-15T22:22:41.048341+0000","last_became_active":"2021-03-15T22:21:56.037322+0000","last_became_peered":"2021-03-15T22:21:56.037322+0000","last_unstale":"2021-03-15T22:22:41.048341+0000","last_undegraded":"2021-03-15T22:22:41.048341+0000","last_fullsized":"2021-03-15T22:22:41.048341+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:02.880783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:02.880783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"275","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177779+0000","last_change":"2021-03-15T22:22:07.679421+0000","last_active":"2021-03-15T22:22:39.177779+0000","last_peered":"2021-03-15T22:22:39.177779+0000","last_clean":"2021-03-15T22:22:39.177779+0000","last_became_active":"2021-03-15T22:21:28.051429+0000","last_became_peered":"2021-03-15T22:21:28.051429+0000","last_unstale":"2021-03-15T22:22:39.177779+0000","last_undegraded":"2021-03-15T22:22:39.177779+0000","last_fullsized":"2021-03-15T22:22:39.177779+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:22:07.679358+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:22:07.679358+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"206","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.049075+0000","last_change":"2021-03-15T22:22:04.890835+0000","last_active":"2021-03-15T22:22:41.049075+0000","last_peered":"2021-03-15T22:22:41.049075+0000","last_clean":"2021-03-15T22:22:41.049075+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:22:41.049075+0000","last_undegraded":"2021-03-15T22:22:41.049075+0000","last_fullsized":"2021-03-15T22:22:41.049075+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:04.890767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:04.890767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"89","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.176484+0000","last_change":"2021-03-15T22:21:53.852914+0000","last_active":"2021-03-15T22:22:39.176484+0000","last_peered":"2021-03-15T22:22:39.176484+0000","last_clean":"2021-03-15T22:22:39.176484+0000","last_became_active":"2021-03-15T22:21:28.044842+0000","last_became_peered":"2021-03-15T22:21:28.044842+0000","last_unstale":"2021-03-15T22:22:39.176484+0000","last_undegraded":"2021-03-15T22:22:39.176484+0000","last_fullsized":"2021-03-15T22:22:39.176484+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:53.852830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:53.852830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"233","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.533177+0000","last_change":"2021-03-15T22:21:24.823853+0000","last_active":"2021-03-15T22:22:39.533177+0000","last_peered":"2021-03-15T22:22:39.533177+0000","last_clean":"2021-03-15T22:22:39.533177+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:22:39.533177+0000","last_undegraded":"2021-03-15T22:22:39.533177+0000","last_fullsized":"2021-03-15T22:22:39.533177+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:21:24.823789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:24.823789+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"206","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.043605+0000","last_change":"2021-03-15T22:22:08.753158+0000","last_active":"2021-03-15T22:22:42.043605+0000","last_peered":"2021-03-15T22:22:42.043605+0000","last_clean":"2021-03-15T22:22:42.043605+0000","last_became_active":"2021-03-15T22:18:00.015858+0000","last_became_peered":"2021-03-15T22:18:00.015858+0000","last_unstale":"2021-03-15T22:22:42.043605+0000","last_undegraded":"2021-03-15T22:22:42.043605+0000","last_fullsized":"2021-03-15T22:22:42.043605+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:08.753083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:22:08.753083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"207","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.531772+0000","last_change":"2021-03-15T22:22:08.682317+0000","last_active":"2021-03-15T22:22:39.531772+0000","last_peered":"2021-03-15T22:22:39.531772+0000","last_clean":"2021-03-15T22:22:39.531772+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:22:39.531772+0000","last_undegraded":"2021-03-15T22:22:39.531772+0000","last_fullsized":"2021-03-15T22:22:39.531772+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:08.682244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:08.682244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"215","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.178034+0000","last_change":"2021-03-15T22:21:28.044437+0000","last_active":"2021-03-15T22:22:39.178034+0000","last_peered":"2021-03-15T22:22:39.178034+0000","last_clean":"2021-03-15T22:22:39.178034+0000","last_became_active":"2021-03-15T22:21:28.044155+0000","last_became_peered":"2021-03-15T22:21:28.044155+0000","last_unstale":"2021-03-15T22:22:39.178034+0000","last_undegraded":"2021-03-15T22:22:39.178034+0000","last_fullsized":"2021-03-15T22:22:39.178034+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:21:05.736770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"119","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.052277+0000","last_change":"2021-03-15T22:22:01.885586+0000","last_active":"2021-03-15T22:22:41.052277+0000","last_peered":"2021-03-15T22:22:41.052277+0000","last_clean":"2021-03-15T22:22:41.052277+0000","last_became_active":"2021-03-15T22:21:56.039344+0000","last_became_peered":"2021-03-15T22:21:56.039344+0000","last_unstale":"2021-03-15T22:22:41.052277+0000","last_undegraded":"2021-03-15T22:22:41.052277+0000","last_fullsized":"2021-03-15T22:22:41.052277+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:22:01.885519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:01.885519+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":11,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"84","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177172+0000","last_change":"2021-03-15T22:21:28.048155+0000","last_active":"2021-03-15T22:22:39.177172+0000","last_peered":"2021-03-15T22:22:39.177172+0000","last_clean":"2021-03-15T22:22:39.177172+0000","last_became_active":"2021-03-15T22:21:28.047835+0000","last_became_peered":"2021-03-15T22:21:28.047835+0000","last_unstale":"2021-03-15T22:22:39.177172+0000","last_undegraded":"2021-03-15T22:22:39.177172+0000","last_fullsized":"2021-03-15T22:22:39.177172+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:56.799862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:56.799862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"203","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.043815+0000","last_change":"2021-03-15T22:20:54.994251+0000","last_active":"2021-03-15T22:22:42.043815+0000","last_peered":"2021-03-15T22:22:42.043815+0000","last_clean":"2021-03-15T22:22:42.043815+0000","last_became_active":"2021-03-15T22:17:58.013174+0000","last_became_peered":"2021-03-15T22:17:58.013174+0000","last_unstale":"2021-03-15T22:22:42.043815+0000","last_undegraded":"2021-03-15T22:22:42.043815+0000","last_fullsized":"2021-03-15T22:22:42.043815+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:54.994178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:20:54.994178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"206","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.049387+0000","last_change":"2021-03-15T22:20:57.165778+0000","last_active":"2021-03-15T22:22:41.049387+0000","last_peered":"2021-03-15T22:22:41.049387+0000","last_clean":"2021-03-15T22:22:41.049387+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:22:41.049387+0000","last_undegraded":"2021-03-15T22:22:41.049387+0000","last_fullsized":"2021-03-15T22:22:41.049387+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:57.165693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:57.165693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"90'22","reported_seq":"148","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.043614+0000","last_change":"2021-03-15T22:22:01.769364+0000","last_active":"2021-03-15T22:22:42.043614+0000","last_peered":"2021-03-15T22:22:42.043614+0000","last_clean":"2021-03-15T22:22:42.043614+0000","last_became_active":"2021-03-15T22:21:56.034049+0000","last_became_peered":"2021-03-15T22:21:56.034049+0000","last_unstale":"2021-03-15T22:22:42.043614+0000","last_undegraded":"2021-03-15T22:22:42.043614+0000","last_fullsized":"2021-03-15T22:22:42.043614+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:22:01.769284+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:22:01.769284+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"200","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.176344+0000","last_change":"2021-03-15T22:22:06.723810+0000","last_active":"2021-03-15T22:22:39.176344+0000","last_peered":"2021-03-15T22:22:39.176344+0000","last_clean":"2021-03-15T22:22:39.176344+0000","last_became_active":"2021-03-15T22:18:00.014112+0000","last_became_peered":"2021-03-15T22:18:00.014112+0000","last_unstale":"2021-03-15T22:22:39.176344+0000","last_undegraded":"2021-03-15T22:22:39.176344+0000","last_fullsized":"2021-03-15T22:22:39.176344+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:06.723750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:06.723750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"56","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177213+0000","last_change":"2021-03-15T22:21:56.037408+0000","last_active":"2021-03-15T22:22:39.177213+0000","last_peered":"2021-03-15T22:22:39.177213+0000","last_clean":"2021-03-15T22:22:39.177213+0000","last_became_active":"2021-03-15T22:21:56.037065+0000","last_became_peered":"2021-03-15T22:21:56.037065+0000","last_unstale":"2021-03-15T22:22:39.177213+0000","last_undegraded":"2021-03-15T22:22:39.177213+0000","last_fullsized":"2021-03-15T22:22:39.177213+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:52.775171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_clean_scrub_stamp":"2021-03-15T22:20:52.775171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"216","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.044187+0000","last_change":"2021-03-15T22:21:56.030382+0000","last_active":"2021-03-15T22:22:42.044187+0000","last_peered":"2021-03-15T22:22:42.044187+0000","last_clean":"2021-03-15T22:22:42.044187+0000","last_became_active":"2021-03-15T22:21:56.029986+0000","last_became_peered":"2021-03-15T22:21:56.029986+0000","last_unstale":"2021-03-15T22:22:42.044187+0000","last_undegraded":"2021-03-15T22:22:42.044187+0000","last_fullsized":"2021-03-15T22:22:42.044187+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:20.673603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:20.673603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"233","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.051329+0000","last_change":"2021-03-15T22:21:24.789596+0000","last_active":"2021-03-15T22:22:41.051329+0000","last_peered":"2021-03-15T22:22:41.051329+0000","last_clean":"2021-03-15T22:22:41.051329+0000","last_became_active":"2021-03-15T22:17:55.991537+0000","last_became_peered":"2021-03-15T22:17:55.991537+0000","last_unstale":"2021-03-15T22:22:41.051329+0000","last_undegraded":"2021-03-15T22:22:41.051329+0000","last_fullsized":"2021-03-15T22:22:41.051329+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:21:24.789526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:24.789526+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"84","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.176917+0000","last_change":"2021-03-15T22:21:28.042915+0000","last_active":"2021-03-15T22:22:39.176917+0000","last_peered":"2021-03-15T22:22:39.176917+0000","last_clean":"2021-03-15T22:22:39.176917+0000","last_became_active":"2021-03-15T22:21:28.042578+0000","last_became_peered":"2021-03-15T22:21:28.042578+0000","last_unstale":"2021-03-15T22:22:39.176917+0000","last_undegraded":"2021-03-15T22:22:39.176917+0000","last_fullsized":"2021-03-15T22:22:39.176917+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:23.821451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:23.821451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"217","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.049709+0000","last_change":"2021-03-15T22:22:39.054218+0000","last_active":"2021-03-15T22:22:41.049709+0000","last_peered":"2021-03-15T22:22:41.049709+0000","last_clean":"2021-03-15T22:22:41.049709+0000","last_became_active":"2021-03-15T22:21:28.047856+0000","last_became_peered":"2021-03-15T22:21:28.047856+0000","last_unstale":"2021-03-15T22:22:41.049709+0000","last_undegraded":"2021-03-15T22:22:41.049709+0000","last_fullsized":"2021-03-15T22:22:41.049709+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:39.054153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:39.054153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"208","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054654+0000","last_change":"2021-03-15T22:21:22.324910+0000","last_active":"2021-03-15T22:22:40.054654+0000","last_peered":"2021-03-15T22:22:40.054654+0000","last_clean":"2021-03-15T22:22:40.054654+0000","last_became_active":"2021-03-15T22:17:53.975292+0000","last_became_peered":"2021-03-15T22:17:53.975292+0000","last_unstale":"2021-03-15T22:22:40.054654+0000","last_undegraded":"2021-03-15T22:22:40.054654+0000","last_fullsized":"2021-03-15T22:22:40.054654+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:22.324836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:22.324836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"233","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054832+0000","last_change":"2021-03-15T22:20:55.236593+0000","last_active":"2021-03-15T22:22:40.054832+0000","last_peered":"2021-03-15T22:22:40.054832+0000","last_clean":"2021-03-15T22:22:40.054832+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:22:40.054832+0000","last_undegraded":"2021-03-15T22:22:40.054832+0000","last_fullsized":"2021-03-15T22:22:40.054832+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:20:55.236487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:20:55.236487+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"209","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177477+0000","last_change":"2021-03-15T22:22:30.800481+0000","last_active":"2021-03-15T22:22:39.177477+0000","last_peered":"2021-03-15T22:22:39.177477+0000","last_clean":"2021-03-15T22:22:39.177477+0000","last_became_active":"2021-03-15T22:21:28.046227+0000","last_became_peered":"2021-03-15T22:21:28.046227+0000","last_unstale":"2021-03-15T22:22:39.177477+0000","last_undegraded":"2021-03-15T22:22:39.177477+0000","last_fullsized":"2021-03-15T22:22:39.177477+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:30.800426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:30.800426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"61'2","reported_seq":"261","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:42.056423+0000","last_change":"2021-03-15T22:21:51.156735+0000","last_active":"2021-03-15T22:22:42.056423+0000","last_peered":"2021-03-15T22:22:42.056423+0000","last_clean":"2021-03-15T22:22:42.056423+0000","last_became_active":"2021-03-15T22:17:58.010903+0000","last_became_peered":"2021-03-15T22:17:58.010903+0000","last_unstale":"2021-03-15T22:22:42.056423+0000","last_undegraded":"2021-03-15T22:22:42.056423+0000","last_fullsized":"2021-03-15T22:22:42.056423+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:21:51.156644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:51.156644+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"207","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.043684+0000","last_change":"2021-03-15T22:21:22.710972+0000","last_active":"2021-03-15T22:22:42.043684+0000","last_peered":"2021-03-15T22:22:42.043684+0000","last_clean":"2021-03-15T22:22:42.043684+0000","last_became_active":"2021-03-15T22:17:54.285851+0000","last_became_peered":"2021-03-15T22:17:54.285851+0000","last_unstale":"2021-03-15T22:22:42.043684+0000","last_undegraded":"2021-03-15T22:22:42.043684+0000","last_fullsized":"2021-03-15T22:22:42.043684+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:22.710908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:22.710908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"118","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.048245+0000","last_change":"2021-03-15T22:22:07.968987+0000","last_active":"2021-03-15T22:22:41.048245+0000","last_peered":"2021-03-15T22:22:41.048245+0000","last_clean":"2021-03-15T22:22:41.048245+0000","last_became_active":"2021-03-15T22:21:56.039170+0000","last_became_peered":"2021-03-15T22:21:56.039170+0000","last_unstale":"2021-03-15T22:22:41.048245+0000","last_undegraded":"2021-03-15T22:22:41.048245+0000","last_fullsized":"2021-03-15T22:22:41.048245+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:22:07.968919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:07.968919+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":13,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"202","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.941923+0000","last_change":"2021-03-15T22:22:41.941923+0000","last_active":"2021-03-15T22:22:41.941923+0000","last_peered":"2021-03-15T22:22:41.941923+0000","last_clean":"2021-03-15T22:22:41.941923+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:22:41.941923+0000","last_undegraded":"2021-03-15T22:22:41.941923+0000","last_fullsized":"2021-03-15T22:22:41.941923+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:41.941855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:41.941855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"173'3","reported_seq":"120","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.056366+0000","last_change":"2021-03-15T22:21:31.689038+0000","last_active":"2021-03-15T22:22:42.056366+0000","last_peered":"2021-03-15T22:22:42.056366+0000","last_clean":"2021-03-15T22:22:42.056366+0000","last_became_active":"2021-03-15T22:21:28.048472+0000","last_became_peered":"2021-03-15T22:21:28.048472+0000","last_unstale":"2021-03-15T22:22:42.056366+0000","last_undegraded":"2021-03-15T22:22:42.056366+0000","last_fullsized":"2021-03-15T22:22:42.056366+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'3","last_scrub_stamp":"2021-03-15T22:21:31.688955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:31.688955+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"215","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177068+0000","last_change":"2021-03-15T22:21:28.045090+0000","last_active":"2021-03-15T22:22:39.177068+0000","last_peered":"2021-03-15T22:22:39.177068+0000","last_clean":"2021-03-15T22:22:39.177068+0000","last_became_active":"2021-03-15T22:21:28.044742+0000","last_became_peered":"2021-03-15T22:21:28.044742+0000","last_unstale":"2021-03-15T22:22:39.177068+0000","last_undegraded":"2021-03-15T22:22:39.177068+0000","last_fullsized":"2021-03-15T22:22:39.177068+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:00.665226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:00.665226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"255","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.048703+0000","last_change":"2021-03-15T22:22:03.126656+0000","last_active":"2021-03-15T22:22:41.048703+0000","last_peered":"2021-03-15T22:22:41.048703+0000","last_clean":"2021-03-15T22:22:41.048703+0000","last_became_active":"2021-03-15T22:21:56.032898+0000","last_became_peered":"2021-03-15T22:21:56.032898+0000","last_unstale":"2021-03-15T22:22:41.048703+0000","last_undegraded":"2021-03-15T22:22:41.048703+0000","last_fullsized":"2021-03-15T22:22:41.048703+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:22:03.126578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:03.126578+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"209","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.049110+0000","last_change":"2021-03-15T22:22:41.652274+0000","last_active":"2021-03-15T22:22:43.049110+0000","last_peered":"2021-03-15T22:22:43.049110+0000","last_clean":"2021-03-15T22:22:43.049110+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:22:43.049110+0000","last_undegraded":"2021-03-15T22:22:43.049110+0000","last_fullsized":"2021-03-15T22:22:43.049110+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:41.652191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:22:41.652191+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"209","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.043523+0000","last_change":"2021-03-15T22:22:03.775204+0000","last_active":"2021-03-15T22:22:42.043523+0000","last_peered":"2021-03-15T22:22:42.043523+0000","last_clean":"2021-03-15T22:22:42.043523+0000","last_became_active":"2021-03-15T22:17:53.976777+0000","last_became_peered":"2021-03-15T22:17:53.976777+0000","last_unstale":"2021-03-15T22:22:42.043523+0000","last_undegraded":"2021-03-15T22:22:42.043523+0000","last_fullsized":"2021-03-15T22:22:42.043523+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:03.775128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:03.775128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"267","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177920+0000","last_change":"2021-03-15T22:21:56.058518+0000","last_active":"2021-03-15T22:22:39.177920+0000","last_peered":"2021-03-15T22:22:39.177920+0000","last_clean":"2021-03-15T22:22:39.177920+0000","last_became_active":"2021-03-15T22:21:56.036234+0000","last_became_peered":"2021-03-15T22:21:56.036234+0000","last_unstale":"2021-03-15T22:22:39.177920+0000","last_undegraded":"2021-03-15T22:22:39.177920+0000","last_fullsized":"2021-03-15T22:22:39.177920+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:21:23.838983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:23.838983+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"201","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.044207+0000","last_change":"2021-03-15T22:20:35.969982+0000","last_active":"2021-03-15T22:22:42.044207+0000","last_peered":"2021-03-15T22:22:42.044207+0000","last_clean":"2021-03-15T22:22:42.044207+0000","last_became_active":"2021-03-15T22:18:00.288462+0000","last_became_peered":"2021-03-15T22:18:00.288462+0000","last_unstale":"2021-03-15T22:22:42.044207+0000","last_undegraded":"2021-03-15T22:22:42.044207+0000","last_fullsized":"2021-03-15T22:22:42.044207+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:35.969903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:35.969903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"209","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.530948+0000","last_change":"2021-03-15T22:21:54.720472+0000","last_active":"2021-03-15T22:22:39.530948+0000","last_peered":"2021-03-15T22:22:39.530948+0000","last_clean":"2021-03-15T22:22:39.530948+0000","last_became_active":"2021-03-15T22:17:53.977470+0000","last_became_peered":"2021-03-15T22:17:53.977470+0000","last_unstale":"2021-03-15T22:22:39.530948+0000","last_undegraded":"2021-03-15T22:22:39.530948+0000","last_fullsized":"2021-03-15T22:22:39.530948+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:54.720368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:54.720368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'7","reported_seq":"127","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.044385+0000","last_change":"2021-03-15T22:22:28.717607+0000","last_active":"2021-03-15T22:22:42.044385+0000","last_peered":"2021-03-15T22:22:42.044385+0000","last_clean":"2021-03-15T22:22:42.044385+0000","last_became_active":"2021-03-15T22:21:28.040048+0000","last_became_peered":"2021-03-15T22:21:28.040048+0000","last_unstale":"2021-03-15T22:22:42.044385+0000","last_undegraded":"2021-03-15T22:22:42.044385+0000","last_fullsized":"2021-03-15T22:22:42.044385+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'7","last_scrub_stamp":"2021-03-15T22:22:28.717530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:28.717530+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"247'135","reported_seq":"639","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.049090+0000","last_change":"2021-03-15T22:22:41.053327+0000","last_active":"2021-03-15T22:22:42.049090+0000","last_peered":"2021-03-15T22:22:42.049090+0000","last_clean":"2021-03-15T22:22:42.049090+0000","last_became_active":"2021-03-15T22:20:05.038191+0000","last_became_peered":"2021-03-15T22:20:05.038191+0000","last_unstale":"2021-03-15T22:22:42.049090+0000","last_undegraded":"2021-03-15T22:22:42.049090+0000","last_fullsized":"2021-03-15T22:22:42.049090+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'74","last_scrub_stamp":"2021-03-15T22:21:25.834723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:21:25.834723+0000","log_size":135,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9539549,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":98,"num_read_kb":106035,"num_write":74,"num_write_kb":20418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"84","length":"1"},{"start":"87","length":"1"},{"start":"8e","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"205","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.048106+0000","last_change":"2021-03-15T22:22:28.062846+0000","last_active":"2021-03-15T22:22:41.048106+0000","last_peered":"2021-03-15T22:22:41.048106+0000","last_clean":"2021-03-15T22:22:41.048106+0000","last_became_active":"2021-03-15T22:18:00.286699+0000","last_became_peered":"2021-03-15T22:18:00.286699+0000","last_unstale":"2021-03-15T22:22:41.048106+0000","last_undegraded":"2021-03-15T22:22:41.048106+0000","last_fullsized":"2021-03-15T22:22:41.048106+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:28.062751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:28.062751+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"210","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.044467+0000","last_change":"2021-03-15T22:20:28.025972+0000","last_active":"2021-03-15T22:22:42.044467+0000","last_peered":"2021-03-15T22:22:42.044467+0000","last_clean":"2021-03-15T22:22:42.044467+0000","last_became_active":"2021-03-15T22:17:53.979246+0000","last_became_peered":"2021-03-15T22:17:53.979246+0000","last_unstale":"2021-03-15T22:22:42.044467+0000","last_undegraded":"2021-03-15T22:22:42.044467+0000","last_fullsized":"2021-03-15T22:22:42.044467+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:20:28.025885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:28.025885+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"235","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.047496+0000","last_change":"2021-03-15T22:21:03.144874+0000","last_active":"2021-03-15T22:22:41.047496+0000","last_peered":"2021-03-15T22:22:41.047496+0000","last_clean":"2021-03-15T22:22:41.047496+0000","last_became_active":"2021-03-15T22:17:56.289684+0000","last_became_peered":"2021-03-15T22:17:56.289684+0000","last_unstale":"2021-03-15T22:22:41.047496+0000","last_undegraded":"2021-03-15T22:22:41.047496+0000","last_fullsized":"2021-03-15T22:22:41.047496+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:21:03.144807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:03.144807+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"172'5","reported_seq":"282","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.056374+0000","last_change":"2021-03-15T22:21:26.679105+0000","last_active":"2021-03-15T22:22:42.056374+0000","last_peered":"2021-03-15T22:22:42.056374+0000","last_clean":"2021-03-15T22:22:42.056374+0000","last_became_active":"2021-03-15T22:21:26.034534+0000","last_became_peered":"2021-03-15T22:21:26.034534+0000","last_unstale":"2021-03-15T22:22:42.056374+0000","last_undegraded":"2021-03-15T22:22:42.056374+0000","last_fullsized":"2021-03-15T22:22:42.056374+0000","mapping_epoch":60,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'3","last_scrub_stamp":"2021-03-15T22:21:26.679039+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:21:26.679039+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"248'297","reported_seq":"845","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.445826+0000","last_change":"2021-03-15T22:22:43.053877+0000","last_active":"2021-03-15T22:22:43.445826+0000","last_peered":"2021-03-15T22:22:43.445826+0000","last_clean":"2021-03-15T22:22:43.445826+0000","last_became_active":"2021-03-15T22:22:01.038626+0000","last_became_peered":"2021-03-15T22:22:01.038626+0000","last_unstale":"2021-03-15T22:22:43.445826+0000","last_undegraded":"2021-03-15T22:22:43.445826+0000","last_fullsized":"2021-03-15T22:22:43.445826+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'255","last_scrub_stamp":"2021-03-15T22:22:29.689315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:29.689315+0000","log_size":297,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22267991,"num_objects":12,"num_object_clones":7,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":157,"num_read_kb":140275,"num_write":131,"num_write_kb":52430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12120510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"84","length":"1"},{"start":"87","length":"1"},{"start":"8e","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"59","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.043784+0000","last_change":"2021-03-15T22:21:56.037894+0000","last_active":"2021-03-15T22:22:42.043784+0000","last_peered":"2021-03-15T22:22:42.043784+0000","last_clean":"2021-03-15T22:22:42.043784+0000","last_became_active":"2021-03-15T22:21:56.037602+0000","last_became_peered":"2021-03-15T22:21:56.037602+0000","last_unstale":"2021-03-15T22:22:42.043784+0000","last_undegraded":"2021-03-15T22:22:42.043784+0000","last_fullsized":"2021-03-15T22:22:42.043784+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:31.842182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:31.842182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"212","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.044372+0000","last_change":"2021-03-15T22:22:38.810490+0000","last_active":"2021-03-15T22:22:42.044372+0000","last_peered":"2021-03-15T22:22:42.044372+0000","last_clean":"2021-03-15T22:22:42.044372+0000","last_became_active":"2021-03-15T22:17:53.978907+0000","last_became_peered":"2021-03-15T22:17:53.978907+0000","last_unstale":"2021-03-15T22:22:42.044372+0000","last_undegraded":"2021-03-15T22:22:42.044372+0000","last_fullsized":"2021-03-15T22:22:42.044372+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:38.810397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.810397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"262","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.055608+0000","last_change":"2021-03-15T22:22:28.887668+0000","last_active":"2021-03-15T22:22:40.055608+0000","last_peered":"2021-03-15T22:22:40.055608+0000","last_clean":"2021-03-15T22:22:40.055608+0000","last_became_active":"2021-03-15T22:21:28.045396+0000","last_became_peered":"2021-03-15T22:21:28.045396+0000","last_unstale":"2021-03-15T22:22:40.055608+0000","last_undegraded":"2021-03-15T22:22:40.055608+0000","last_fullsized":"2021-03-15T22:22:40.055608+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:22:28.887588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:28.887588+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"229","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.047232+0000","last_change":"2021-03-15T22:22:35.649394+0000","last_active":"2021-03-15T22:22:43.047232+0000","last_peered":"2021-03-15T22:22:43.047232+0000","last_clean":"2021-03-15T22:22:43.047232+0000","last_became_active":"2021-03-15T22:21:30.034044+0000","last_became_peered":"2021-03-15T22:21:30.034044+0000","last_unstale":"2021-03-15T22:22:43.047232+0000","last_undegraded":"2021-03-15T22:22:43.047232+0000","last_fullsized":"2021-03-15T22:22:43.047232+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:35.649331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:35.649331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"213'77","reported_seq":"557","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.069913+0000","last_change":"2021-03-15T22:22:37.982790+0000","last_active":"2021-03-15T22:22:40.069913+0000","last_peered":"2021-03-15T22:22:40.069913+0000","last_clean":"2021-03-15T22:22:40.069913+0000","last_became_active":"2021-03-15T22:20:05.032301+0000","last_became_peered":"2021-03-15T22:20:05.032301+0000","last_unstale":"2021-03-15T22:22:40.069913+0000","last_undegraded":"2021-03-15T22:22:40.069913+0000","last_fullsized":"2021-03-15T22:22:40.069913+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'77","last_scrub_stamp":"2021-03-15T22:22:37.982715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:37.982715+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3109262,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":30,"num_read_kb":21292,"num_write":37,"num_write_kb":13534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"87","length":"1"},{"start":"8e","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"210","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.043433+0000","last_change":"2021-03-15T22:21:56.033485+0000","last_active":"2021-03-15T22:22:42.043433+0000","last_peered":"2021-03-15T22:22:42.043433+0000","last_clean":"2021-03-15T22:22:42.043433+0000","last_became_active":"2021-03-15T22:21:56.033006+0000","last_became_peered":"2021-03-15T22:21:56.033006+0000","last_unstale":"2021-03-15T22:22:42.043433+0000","last_undegraded":"2021-03-15T22:22:42.043433+0000","last_fullsized":"2021-03-15T22:22:42.043433+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:53.009472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:53.009472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"229","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.049041+0000","last_change":"2021-03-15T22:22:34.656502+0000","last_active":"2021-03-15T22:22:43.049041+0000","last_peered":"2021-03-15T22:22:43.049041+0000","last_clean":"2021-03-15T22:22:43.049041+0000","last_became_active":"2021-03-15T22:21:18.047758+0000","last_became_peered":"2021-03-15T22:21:18.047758+0000","last_unstale":"2021-03-15T22:22:43.049041+0000","last_undegraded":"2021-03-15T22:22:43.049041+0000","last_fullsized":"2021-03-15T22:22:43.049041+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:34.656440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:34.656440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"207","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.044303+0000","last_change":"2021-03-15T22:21:06.006147+0000","last_active":"2021-03-15T22:22:42.044303+0000","last_peered":"2021-03-15T22:22:42.044303+0000","last_clean":"2021-03-15T22:22:42.044303+0000","last_became_active":"2021-03-15T22:17:53.974108+0000","last_became_peered":"2021-03-15T22:17:53.974108+0000","last_unstale":"2021-03-15T22:22:42.044303+0000","last_undegraded":"2021-03-15T22:22:42.044303+0000","last_fullsized":"2021-03-15T22:22:42.044303+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:21:06.006086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"241","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.532269+0000","last_change":"2021-03-15T22:21:33.792545+0000","last_active":"2021-03-15T22:22:39.532269+0000","last_peered":"2021-03-15T22:22:39.532269+0000","last_clean":"2021-03-15T22:22:39.532269+0000","last_became_active":"2021-03-15T22:17:56.288870+0000","last_became_peered":"2021-03-15T22:17:56.288870+0000","last_unstale":"2021-03-15T22:22:39.532269+0000","last_undegraded":"2021-03-15T22:22:39.532269+0000","last_fullsized":"2021-03-15T22:22:39.532269+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:21:33.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:33.792449+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"248'149","reported_seq":"662","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.446909+0000","last_change":"2021-03-15T22:22:43.051246+0000","last_active":"2021-03-15T22:22:43.446909+0000","last_peered":"2021-03-15T22:22:43.446909+0000","last_clean":"2021-03-15T22:22:43.446909+0000","last_became_active":"2021-03-15T22:20:05.032282+0000","last_became_peered":"2021-03-15T22:20:05.032282+0000","last_unstale":"2021-03-15T22:22:43.446909+0000","last_undegraded":"2021-03-15T22:22:43.446909+0000","last_fullsized":"2021-03-15T22:22:43.446909+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'104","last_scrub_stamp":"2021-03-15T22:21:31.442405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:21:31.442405+0000","log_size":149,"ondisk_log_size":149,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10337338,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":121,"num_read_kb":155594,"num_write":60,"num_write_kb":22874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"84","length":"1"},{"start":"87","length":"1"},{"start":"8e","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"202","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177049+0000","last_change":"2021-03-15T22:22:28.711806+0000","last_active":"2021-03-15T22:22:39.177049+0000","last_peered":"2021-03-15T22:22:39.177049+0000","last_clean":"2021-03-15T22:22:39.177049+0000","last_became_active":"2021-03-15T22:18:00.016982+0000","last_became_peered":"2021-03-15T22:18:00.016982+0000","last_unstale":"2021-03-15T22:22:39.177049+0000","last_undegraded":"2021-03-15T22:22:39.177049+0000","last_fullsized":"2021-03-15T22:22:39.177049+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:28.711745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:28.711745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"238","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.049916+0000","last_change":"2021-03-15T22:21:51.578587+0000","last_active":"2021-03-15T22:22:43.049916+0000","last_peered":"2021-03-15T22:22:43.049916+0000","last_clean":"2021-03-15T22:22:43.049916+0000","last_became_active":"2021-03-15T22:21:28.049417+0000","last_became_peered":"2021-03-15T22:21:28.049417+0000","last_unstale":"2021-03-15T22:22:43.049916+0000","last_undegraded":"2021-03-15T22:22:43.049916+0000","last_fullsized":"2021-03-15T22:22:43.049916+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:51.578525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:51.578525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"215","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.048132+0000","last_change":"2021-03-15T22:21:28.047052+0000","last_active":"2021-03-15T22:22:41.048132+0000","last_peered":"2021-03-15T22:22:41.048132+0000","last_clean":"2021-03-15T22:22:41.048132+0000","last_became_active":"2021-03-15T22:21:28.046734+0000","last_became_peered":"2021-03-15T22:21:28.046734+0000","last_unstale":"2021-03-15T22:22:41.048132+0000","last_undegraded":"2021-03-15T22:22:41.048132+0000","last_fullsized":"2021-03-15T22:22:41.048132+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:00.088823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:00.088823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"259","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177350+0000","last_change":"2021-03-15T22:21:30.812577+0000","last_active":"2021-03-15T22:22:39.177350+0000","last_peered":"2021-03-15T22:22:39.177350+0000","last_clean":"2021-03-15T22:22:39.177350+0000","last_became_active":"2021-03-15T22:19:02.093463+0000","last_became_peered":"2021-03-15T22:19:02.093463+0000","last_unstale":"2021-03-15T22:22:39.177350+0000","last_undegraded":"2021-03-15T22:22:39.177350+0000","last_fullsized":"2021-03-15T22:22:39.177350+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:21:30.812512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:30.812512+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"243'103","reported_seq":"601","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.067762+0000","last_change":"2021-03-15T22:22:37.173881+0000","last_active":"2021-03-15T22:22:40.067762+0000","last_peered":"2021-03-15T22:22:40.067762+0000","last_clean":"2021-03-15T22:22:40.067762+0000","last_became_active":"2021-03-15T22:21:28.041958+0000","last_became_peered":"2021-03-15T22:21:28.041958+0000","last_unstale":"2021-03-15T22:22:40.067762+0000","last_undegraded":"2021-03-15T22:22:40.067762+0000","last_fullsized":"2021-03-15T22:22:40.067762+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'88","last_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:22:07.718413+0000","log_size":103,"ondisk_log_size":103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":9,"num_write":56,"num_write_kb":20616,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":5516599,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"87","length":"1"},{"start":"8e","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"90","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054331+0000","last_change":"2021-03-15T22:21:32.290962+0000","last_active":"2021-03-15T22:22:40.054331+0000","last_peered":"2021-03-15T22:22:40.054331+0000","last_clean":"2021-03-15T22:22:40.054331+0000","last_became_active":"2021-03-15T22:21:28.047359+0000","last_became_peered":"2021-03-15T22:21:28.047359+0000","last_unstale":"2021-03-15T22:22:40.054331+0000","last_undegraded":"2021-03-15T22:22:40.054331+0000","last_fullsized":"2021-03-15T22:22:40.054331+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:32.290882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:32.290882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"213","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.047908+0000","last_change":"2021-03-15T22:22:39.140480+0000","last_active":"2021-03-15T22:22:41.047908+0000","last_peered":"2021-03-15T22:22:41.047908+0000","last_clean":"2021-03-15T22:22:41.047908+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:22:41.047908+0000","last_undegraded":"2021-03-15T22:22:41.047908+0000","last_fullsized":"2021-03-15T22:22:41.047908+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:39.140419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:22:39.140419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"208","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054751+0000","last_change":"2021-03-15T22:21:27.231350+0000","last_active":"2021-03-15T22:22:40.054751+0000","last_peered":"2021-03-15T22:22:40.054751+0000","last_clean":"2021-03-15T22:22:40.054751+0000","last_became_active":"2021-03-15T22:17:53.978521+0000","last_became_peered":"2021-03-15T22:17:53.978521+0000","last_unstale":"2021-03-15T22:22:40.054751+0000","last_undegraded":"2021-03-15T22:22:40.054751+0000","last_fullsized":"2021-03-15T22:22:40.054751+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:27.231266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:27.231266+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"307","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.047453+0000","last_change":"2021-03-15T22:21:55.867576+0000","last_active":"2021-03-15T22:22:41.047453+0000","last_peered":"2021-03-15T22:22:41.047453+0000","last_clean":"2021-03-15T22:22:41.047453+0000","last_became_active":"2021-03-15T22:21:28.052494+0000","last_became_peered":"2021-03-15T22:21:28.052494+0000","last_unstale":"2021-03-15T22:22:41.047453+0000","last_undegraded":"2021-03-15T22:22:41.047453+0000","last_fullsized":"2021-03-15T22:22:41.047453+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:21:55.867509+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:21:55.867509+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"202","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177271+0000","last_change":"2021-03-15T22:20:30.455518+0000","last_active":"2021-03-15T22:22:39.177271+0000","last_peered":"2021-03-15T22:22:39.177271+0000","last_clean":"2021-03-15T22:22:39.177271+0000","last_became_active":"2021-03-15T22:17:58.017434+0000","last_became_peered":"2021-03-15T22:17:58.017434+0000","last_unstale":"2021-03-15T22:22:39.177271+0000","last_undegraded":"2021-03-15T22:22:39.177271+0000","last_fullsized":"2021-03-15T22:22:39.177271+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:30.455430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:20:30.455430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"242'170","reported_seq":"645","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.052641+0000","last_change":"2021-03-15T22:22:37.180838+0000","last_active":"2021-03-15T22:22:40.052641+0000","last_peered":"2021-03-15T22:22:40.052641+0000","last_clean":"2021-03-15T22:22:40.052641+0000","last_became_active":"2021-03-15T22:20:05.038149+0000","last_became_peered":"2021-03-15T22:20:05.038149+0000","last_unstale":"2021-03-15T22:22:40.052641+0000","last_undegraded":"2021-03-15T22:22:40.052641+0000","last_fullsized":"2021-03-15T22:22:40.052641+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":170,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6437081,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":72,"num_read_kb":68170,"num_write":73,"num_write_kb":22571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"87","length":"1"},{"start":"8e","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"210","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.044390+0000","last_change":"2021-03-15T22:21:56.031698+0000","last_active":"2021-03-15T22:22:42.044390+0000","last_peered":"2021-03-15T22:22:42.044390+0000","last_clean":"2021-03-15T22:22:42.044390+0000","last_became_active":"2021-03-15T22:21:56.031350+0000","last_became_peered":"2021-03-15T22:21:56.031350+0000","last_unstale":"2021-03-15T22:22:42.044390+0000","last_undegraded":"2021-03-15T22:22:42.044390+0000","last_fullsized":"2021-03-15T22:22:42.044390+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:59.975891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:59.975891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"212","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.052159+0000","last_change":"2021-03-15T22:22:30.930369+0000","last_active":"2021-03-15T22:22:41.052159+0000","last_peered":"2021-03-15T22:22:41.052159+0000","last_clean":"2021-03-15T22:22:41.052159+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:22:41.052159+0000","last_undegraded":"2021-03-15T22:22:41.052159+0000","last_fullsized":"2021-03-15T22:22:41.052159+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:30.930256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:30.930256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"222","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.047278+0000","last_change":"2021-03-15T22:21:28.050859+0000","last_active":"2021-03-15T22:22:43.047278+0000","last_peered":"2021-03-15T22:22:43.047278+0000","last_clean":"2021-03-15T22:22:43.047278+0000","last_became_active":"2021-03-15T22:21:28.050532+0000","last_became_peered":"2021-03-15T22:21:28.050532+0000","last_unstale":"2021-03-15T22:22:43.047278+0000","last_undegraded":"2021-03-15T22:22:43.047278+0000","last_fullsized":"2021-03-15T22:22:43.047278+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:54.018631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:20:54.018631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"149'22","reported_seq":"248","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.050305+0000","last_change":"2021-03-15T22:22:40.038242+0000","last_active":"2021-03-15T22:22:41.050305+0000","last_peered":"2021-03-15T22:22:41.050305+0000","last_clean":"2021-03-15T22:22:41.050305+0000","last_became_active":"2021-03-15T22:17:55.992952+0000","last_became_peered":"2021-03-15T22:17:55.992952+0000","last_unstale":"2021-03-15T22:22:41.050305+0000","last_undegraded":"2021-03-15T22:22:41.050305+0000","last_fullsized":"2021-03-15T22:22:41.050305+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'22","last_scrub_stamp":"2021-03-15T22:22:40.038183+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:22:40.038183+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"229","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.048295+0000","last_change":"2021-03-15T22:22:32.668891+0000","last_active":"2021-03-15T22:22:43.048295+0000","last_peered":"2021-03-15T22:22:43.048295+0000","last_clean":"2021-03-15T22:22:43.048295+0000","last_became_active":"2021-03-15T22:21:38.147364+0000","last_became_peered":"2021-03-15T22:21:38.147364+0000","last_unstale":"2021-03-15T22:22:43.048295+0000","last_undegraded":"2021-03-15T22:22:43.048295+0000","last_fullsized":"2021-03-15T22:22:43.048295+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:32.668827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:32.668827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"245'350","reported_seq":"921","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.995849+0000","last_change":"2021-03-15T22:22:40.995849+0000","last_active":"2021-03-15T22:22:40.995849+0000","last_peered":"2021-03-15T22:22:40.995849+0000","last_clean":"2021-03-15T22:22:40.995849+0000","last_became_active":"2021-03-15T22:21:31.025186+0000","last_became_peered":"2021-03-15T22:21:31.025186+0000","last_unstale":"2021-03-15T22:22:40.995849+0000","last_undegraded":"2021-03-15T22:22:40.995849+0000","last_fullsized":"2021-03-15T22:22:40.995849+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'350","last_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:22:40.995789+0000","log_size":350,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22069125,"num_objects":11,"num_object_clones":5,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":158,"num_read_kb":113158,"num_write":163,"num_write_kb":72764,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":15951974,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"87","length":"1"},{"start":"8e","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"209","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.049278+0000","last_change":"2021-03-15T22:21:56.040261+0000","last_active":"2021-03-15T22:22:41.049278+0000","last_peered":"2021-03-15T22:22:41.049278+0000","last_clean":"2021-03-15T22:22:41.049278+0000","last_became_active":"2021-03-15T22:21:56.039902+0000","last_became_peered":"2021-03-15T22:21:56.039902+0000","last_unstale":"2021-03-15T22:22:41.049278+0000","last_undegraded":"2021-03-15T22:22:41.049278+0000","last_fullsized":"2021-03-15T22:22:41.049278+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:20.983187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:20.983187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"210","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.043508+0000","last_change":"2021-03-15T22:20:54.007423+0000","last_active":"2021-03-15T22:22:42.043508+0000","last_peered":"2021-03-15T22:22:42.043508+0000","last_clean":"2021-03-15T22:22:42.043508+0000","last_became_active":"2021-03-15T22:17:51.044140+0000","last_became_peered":"2021-03-15T22:17:51.044140+0000","last_unstale":"2021-03-15T22:22:42.043508+0000","last_undegraded":"2021-03-15T22:22:42.043508+0000","last_fullsized":"2021-03-15T22:22:42.043508+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:54.007335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:20:54.007335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"85","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.047853+0000","last_change":"2021-03-15T22:21:28.039449+0000","last_active":"2021-03-15T22:22:41.047853+0000","last_peered":"2021-03-15T22:22:41.047853+0000","last_clean":"2021-03-15T22:22:41.047853+0000","last_became_active":"2021-03-15T22:21:28.039041+0000","last_became_peered":"2021-03-15T22:21:28.039041+0000","last_unstale":"2021-03-15T22:22:41.047853+0000","last_undegraded":"2021-03-15T22:22:41.047853+0000","last_fullsized":"2021-03-15T22:22:41.047853+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:04.885373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:04.885373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"231","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.533476+0000","last_change":"2021-03-15T22:22:31.635919+0000","last_active":"2021-03-15T22:22:39.533476+0000","last_peered":"2021-03-15T22:22:39.533476+0000","last_clean":"2021-03-15T22:22:39.533476+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:22:39.533476+0000","last_undegraded":"2021-03-15T22:22:39.533476+0000","last_fullsized":"2021-03-15T22:22:39.533476+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:22:31.635832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:31.635832+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"203","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.044086+0000","last_change":"2021-03-15T22:20:31.051075+0000","last_active":"2021-03-15T22:22:42.044086+0000","last_peered":"2021-03-15T22:22:42.044086+0000","last_clean":"2021-03-15T22:22:42.044086+0000","last_became_active":"2021-03-15T22:17:58.011911+0000","last_became_peered":"2021-03-15T22:17:58.011911+0000","last_unstale":"2021-03-15T22:22:42.044086+0000","last_undegraded":"2021-03-15T22:22:42.044086+0000","last_fullsized":"2021-03-15T22:22:42.044086+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:31.050970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:31.050970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"246'97","reported_seq":"591","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.055803+0000","last_change":"2021-03-15T22:22:43.056006+0000","last_active":"2021-03-15T22:22:43.055803+0000","last_peered":"2021-03-15T22:22:43.055803+0000","last_clean":"2021-03-15T22:22:43.055803+0000","last_became_active":"2021-03-15T22:20:05.035608+0000","last_became_peered":"2021-03-15T22:20:05.035608+0000","last_unstale":"2021-03-15T22:22:43.055803+0000","last_undegraded":"2021-03-15T22:22:43.055803+0000","last_fullsized":"2021-03-15T22:22:43.055803+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'65","last_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:21:27.654567+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4503638,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":47036,"num_write":46,"num_write_kb":17519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"84","length":"1"},{"start":"87","length":"1"},{"start":"8e","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"215","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.086264+0000","last_change":"2021-03-15T22:22:41.086239+0000","last_active":"2021-03-15T22:22:41.086264+0000","last_peered":"2021-03-15T22:22:41.086264+0000","last_clean":"2021-03-15T22:22:41.086264+0000","last_became_active":"2021-03-15T22:21:28.046898+0000","last_became_peered":"2021-03-15T22:21:28.046898+0000","last_unstale":"2021-03-15T22:22:41.086264+0000","last_undegraded":"2021-03-15T22:22:41.086264+0000","last_fullsized":"2021-03-15T22:22:41.086264+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:41.086181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:41.086181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"215","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.928453+0000","last_change":"2021-03-15T22:22:39.928411+0000","last_active":"2021-03-15T22:22:39.928453+0000","last_peered":"2021-03-15T22:22:39.928453+0000","last_clean":"2021-03-15T22:22:39.928453+0000","last_became_active":"2021-03-15T22:21:28.047113+0000","last_became_peered":"2021-03-15T22:21:28.047113+0000","last_unstale":"2021-03-15T22:22:39.928453+0000","last_undegraded":"2021-03-15T22:22:39.928453+0000","last_fullsized":"2021-03-15T22:22:39.928453+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:39.928349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:39.928349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"502","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.055000+0000","last_change":"2021-03-15T22:20:36.923986+0000","last_active":"2021-03-15T22:22:43.055000+0000","last_peered":"2021-03-15T22:22:43.055000+0000","last_clean":"2021-03-15T22:22:43.055000+0000","last_became_active":"2021-03-15T22:17:51.044139+0000","last_became_peered":"2021-03-15T22:17:51.044139+0000","last_unstale":"2021-03-15T22:22:43.055000+0000","last_undegraded":"2021-03-15T22:22:43.055000+0000","last_fullsized":"2021-03-15T22:22:43.055000+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:20:36.923907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:20:36.923907+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":290,"num_read_kb":285,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"230","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177544+0000","last_change":"2021-03-15T22:22:05.739037+0000","last_active":"2021-03-15T22:22:39.177544+0000","last_peered":"2021-03-15T22:22:39.177544+0000","last_clean":"2021-03-15T22:22:39.177544+0000","last_became_active":"2021-03-15T22:17:55.999564+0000","last_became_peered":"2021-03-15T22:17:55.999564+0000","last_unstale":"2021-03-15T22:22:39.177544+0000","last_undegraded":"2021-03-15T22:22:39.177544+0000","last_fullsized":"2021-03-15T22:22:39.177544+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:22:05.738937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:05.738937+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"225","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.049553+0000","last_change":"2021-03-15T22:22:38.101741+0000","last_active":"2021-03-15T22:22:41.049553+0000","last_peered":"2021-03-15T22:22:41.049553+0000","last_clean":"2021-03-15T22:22:41.049553+0000","last_became_active":"2021-03-15T22:21:34.036372+0000","last_became_peered":"2021-03-15T22:21:34.036372+0000","last_unstale":"2021-03-15T22:22:41.049553+0000","last_undegraded":"2021-03-15T22:22:41.049553+0000","last_fullsized":"2021-03-15T22:22:41.049553+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:38.101646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.101646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"247'433","reported_seq":"1037","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.068761+0000","last_change":"2021-03-15T22:22:41.058971+0000","last_active":"2021-03-15T22:22:42.068761+0000","last_peered":"2021-03-15T22:22:42.068761+0000","last_clean":"2021-03-15T22:22:42.068761+0000","last_became_active":"2021-03-15T22:22:02.040514+0000","last_became_peered":"2021-03-15T22:22:02.040514+0000","last_unstale":"2021-03-15T22:22:42.068761+0000","last_undegraded":"2021-03-15T22:22:42.068761+0000","last_fullsized":"2021-03-15T22:22:42.068761+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'337","last_scrub_stamp":"2021-03-15T22:22:06.142035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:06.142035+0000","log_size":433,"ondisk_log_size":433,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20023474,"num_objects":11,"num_object_clones":6,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":254,"num_read_kb":261119,"num_write":186,"num_write_kb":65372,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":18565099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"84","length":"1"},{"start":"87","length":"1"},{"start":"8e","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"206","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.047929+0000","last_change":"2021-03-15T22:22:33.947421+0000","last_active":"2021-03-15T22:22:41.047929+0000","last_peered":"2021-03-15T22:22:41.047929+0000","last_clean":"2021-03-15T22:22:41.047929+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:22:41.047929+0000","last_undegraded":"2021-03-15T22:22:41.047929+0000","last_fullsized":"2021-03-15T22:22:41.047929+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:33.947357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:22:33.947357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"212","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.048167+0000","last_change":"2021-03-15T22:22:34.938551+0000","last_active":"2021-03-15T22:22:41.048167+0000","last_peered":"2021-03-15T22:22:41.048167+0000","last_clean":"2021-03-15T22:22:41.048167+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:22:41.048167+0000","last_undegraded":"2021-03-15T22:22:41.048167+0000","last_fullsized":"2021-03-15T22:22:41.048167+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:34.938482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:34.938482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"254","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177473+0000","last_change":"2021-03-15T22:20:28.505643+0000","last_active":"2021-03-15T22:22:39.177473+0000","last_peered":"2021-03-15T22:22:39.177473+0000","last_clean":"2021-03-15T22:22:39.177473+0000","last_became_active":"2021-03-15T22:17:56.288438+0000","last_became_peered":"2021-03-15T22:17:56.288438+0000","last_unstale":"2021-03-15T22:22:39.177473+0000","last_undegraded":"2021-03-15T22:22:39.177473+0000","last_fullsized":"2021-03-15T22:22:39.177473+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:20:28.505568+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:20:28.505568+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"205","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.712336+0000","last_change":"2021-03-15T22:22:39.712285+0000","last_active":"2021-03-15T22:22:39.712336+0000","last_peered":"2021-03-15T22:22:39.712336+0000","last_clean":"2021-03-15T22:22:39.712336+0000","last_became_active":"2021-03-15T22:17:58.008731+0000","last_became_peered":"2021-03-15T22:17:58.008731+0000","last_unstale":"2021-03-15T22:22:39.712336+0000","last_undegraded":"2021-03-15T22:22:39.712336+0000","last_fullsized":"2021-03-15T22:22:39.712336+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:39.712183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:22:39.712183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"238'54","reported_seq":"569","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.045067+0000","last_change":"2021-03-15T22:22:41.045315+0000","last_active":"2021-03-15T22:22:41.045067+0000","last_peered":"2021-03-15T22:22:41.045067+0000","last_clean":"2021-03-15T22:22:41.045067+0000","last_became_active":"2021-03-15T22:21:28.045150+0000","last_became_peered":"2021-03-15T22:21:28.045150+0000","last_unstale":"2021-03-15T22:22:41.045067+0000","last_undegraded":"2021-03-15T22:22:41.045067+0000","last_fullsized":"2021-03-15T22:22:41.045067+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'42","last_scrub_stamp":"2021-03-15T22:22:05.296900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:05.296900+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3697540,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":82219,"num_write":24,"num_write_kb":8677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3663893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"84","length":"1"},{"start":"87","length":"1"},{"start":"8e","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"209","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.176428+0000","last_change":"2021-03-15T22:21:56.040799+0000","last_active":"2021-03-15T22:22:39.176428+0000","last_peered":"2021-03-15T22:22:39.176428+0000","last_clean":"2021-03-15T22:22:39.176428+0000","last_became_active":"2021-03-15T22:21:56.040442+0000","last_became_peered":"2021-03-15T22:21:56.040442+0000","last_unstale":"2021-03-15T22:22:39.176428+0000","last_undegraded":"2021-03-15T22:22:39.176428+0000","last_fullsized":"2021-03-15T22:22:39.176428+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:06.725499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:06.725499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"215","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.048522+0000","last_change":"2021-03-15T22:21:32.017870+0000","last_active":"2021-03-15T22:22:41.048522+0000","last_peered":"2021-03-15T22:22:41.048522+0000","last_clean":"2021-03-15T22:22:41.048522+0000","last_became_active":"2021-03-15T22:21:28.040070+0000","last_became_peered":"2021-03-15T22:21:28.040070+0000","last_unstale":"2021-03-15T22:22:41.048522+0000","last_undegraded":"2021-03-15T22:22:41.048522+0000","last_fullsized":"2021-03-15T22:22:41.048522+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:21:32.017806+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"234","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.050610+0000","last_change":"2021-03-15T22:21:21.773230+0000","last_active":"2021-03-15T22:22:41.050610+0000","last_peered":"2021-03-15T22:22:41.050610+0000","last_clean":"2021-03-15T22:22:41.050610+0000","last_became_active":"2021-03-15T22:17:55.992257+0000","last_became_peered":"2021-03-15T22:17:55.992257+0000","last_unstale":"2021-03-15T22:22:41.050610+0000","last_undegraded":"2021-03-15T22:22:41.050610+0000","last_fullsized":"2021-03-15T22:22:41.050610+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:21:21.773164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:21.773164+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"61'2","reported_seq":"260","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177259+0000","last_change":"2021-03-15T22:21:33.797042+0000","last_active":"2021-03-15T22:22:39.177259+0000","last_peered":"2021-03-15T22:22:39.177259+0000","last_clean":"2021-03-15T22:22:39.177259+0000","last_became_active":"2021-03-15T22:17:58.008973+0000","last_became_peered":"2021-03-15T22:17:58.008973+0000","last_unstale":"2021-03-15T22:22:39.177259+0000","last_undegraded":"2021-03-15T22:22:39.177259+0000","last_fullsized":"2021-03-15T22:22:39.177259+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:21:33.796953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:33.796953+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"246'114","reported_seq":"600","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.055483+0000","last_change":"2021-03-15T22:22:41.055483+0000","last_active":"2021-03-15T22:22:41.055483+0000","last_peered":"2021-03-15T22:22:41.055483+0000","last_clean":"2021-03-15T22:22:41.055483+0000","last_became_active":"2021-03-15T22:21:56.039222+0000","last_became_peered":"2021-03-15T22:21:56.039222+0000","last_unstale":"2021-03-15T22:22:41.055483+0000","last_undegraded":"2021-03-15T22:22:41.055483+0000","last_fullsized":"2021-03-15T22:22:41.055483+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'97","last_scrub_stamp":"2021-03-15T22:21:54.579693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:21:54.579693+0000","log_size":114,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3428621,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":52,"num_read_kb":40150,"num_write":44,"num_write_kb":20938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3421879,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"84","length":"1"},{"start":"87","length":"1"},{"start":"8e","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"206","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.043931+0000","last_change":"2021-03-15T22:22:36.796288+0000","last_active":"2021-03-15T22:22:42.043931+0000","last_peered":"2021-03-15T22:22:42.043931+0000","last_clean":"2021-03-15T22:22:42.043931+0000","last_became_active":"2021-03-15T22:18:00.288441+0000","last_became_peered":"2021-03-15T22:18:00.288441+0000","last_unstale":"2021-03-15T22:22:42.043931+0000","last_undegraded":"2021-03-15T22:22:42.043931+0000","last_fullsized":"2021-03-15T22:22:42.043931+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:36.796228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:36.796228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"206","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177606+0000","last_change":"2021-03-15T22:21:52.836071+0000","last_active":"2021-03-15T22:22:39.177606+0000","last_peered":"2021-03-15T22:22:39.177606+0000","last_clean":"2021-03-15T22:22:39.177606+0000","last_became_active":"2021-03-15T22:17:53.976307+0000","last_became_peered":"2021-03-15T22:17:53.976307+0000","last_unstale":"2021-03-15T22:22:39.177606+0000","last_undegraded":"2021-03-15T22:22:39.177606+0000","last_fullsized":"2021-03-15T22:22:39.177606+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:52.835994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:52.835994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"239","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.055414+0000","last_change":"2021-03-15T22:21:03.362110+0000","last_active":"2021-03-15T22:22:40.055414+0000","last_peered":"2021-03-15T22:22:40.055414+0000","last_clean":"2021-03-15T22:22:40.055414+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:22:40.055414+0000","last_undegraded":"2021-03-15T22:22:40.055414+0000","last_fullsized":"2021-03-15T22:22:40.055414+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:21:03.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:03.362028+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"173'3","reported_seq":"119","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.045914+0000","last_change":"2021-03-15T22:21:54.890590+0000","last_active":"2021-03-15T22:22:42.045914+0000","last_peered":"2021-03-15T22:22:42.045914+0000","last_clean":"2021-03-15T22:22:42.045914+0000","last_became_active":"2021-03-15T22:21:28.048810+0000","last_became_peered":"2021-03-15T22:21:28.048810+0000","last_unstale":"2021-03-15T22:22:42.045914+0000","last_undegraded":"2021-03-15T22:22:42.045914+0000","last_fullsized":"2021-03-15T22:22:42.045914+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'3","last_scrub_stamp":"2021-03-15T22:21:54.890506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:54.890506+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"248'200","reported_seq":"742","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.463897+0000","last_change":"2021-03-15T22:22:43.049903+0000","last_active":"2021-03-15T22:22:43.463897+0000","last_peered":"2021-03-15T22:22:43.463897+0000","last_clean":"2021-03-15T22:22:43.463897+0000","last_became_active":"2021-03-15T22:22:00.120085+0000","last_became_peered":"2021-03-15T22:22:00.120085+0000","last_unstale":"2021-03-15T22:22:43.463897+0000","last_undegraded":"2021-03-15T22:22:43.463897+0000","last_fullsized":"2021-03-15T22:22:43.463897+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'152","last_scrub_stamp":"2021-03-15T22:22:07.676468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:07.676468+0000","log_size":200,"ondisk_log_size":200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26133292,"num_objects":10,"num_object_clones":6,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":130,"num_read_kb":121035,"num_write":84,"num_write_kb":32442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":11287781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"84","length":"1"},{"start":"87","length":"1"},{"start":"8e","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"211","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054190+0000","last_change":"2021-03-15T22:21:28.038681+0000","last_active":"2021-03-15T22:22:40.054190+0000","last_peered":"2021-03-15T22:22:40.054190+0000","last_clean":"2021-03-15T22:22:40.054190+0000","last_became_active":"2021-03-15T22:21:28.038391+0000","last_became_peered":"2021-03-15T22:21:28.038391+0000","last_unstale":"2021-03-15T22:22:40.054190+0000","last_undegraded":"2021-03-15T22:22:40.054190+0000","last_fullsized":"2021-03-15T22:22:40.054190+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:00.367847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:00.367847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"64","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.044497+0000","last_change":"2021-03-15T22:22:04.792632+0000","last_active":"2021-03-15T22:22:42.044497+0000","last_peered":"2021-03-15T22:22:42.044497+0000","last_clean":"2021-03-15T22:22:42.044497+0000","last_became_active":"2021-03-15T22:21:56.036359+0000","last_became_peered":"2021-03-15T22:21:56.036359+0000","last_unstale":"2021-03-15T22:22:42.044497+0000","last_undegraded":"2021-03-15T22:22:42.044497+0000","last_fullsized":"2021-03-15T22:22:42.044497+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:04.792551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:04.792551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"185","reported_epoch":"247","state":"active+clean","last_fresh":"2021-03-15T22:22:42.043736+0000","last_change":"2021-03-15T22:21:29.174708+0000","last_active":"2021-03-15T22:22:42.043736+0000","last_peered":"2021-03-15T22:22:42.043736+0000","last_clean":"2021-03-15T22:22:42.043736+0000","last_became_active":"2021-03-15T22:21:28.047566+0000","last_became_peered":"2021-03-15T22:21:28.047566+0000","last_unstale":"2021-03-15T22:22:42.043736+0000","last_undegraded":"2021-03-15T22:22:42.043736+0000","last_fullsized":"2021-03-15T22:22:42.043736+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:20:55.778268+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:20:55.778268+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"213","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054314+0000","last_change":"2021-03-15T22:21:59.228515+0000","last_active":"2021-03-15T22:22:40.054314+0000","last_peered":"2021-03-15T22:22:40.054314+0000","last_clean":"2021-03-15T22:22:40.054314+0000","last_became_active":"2021-03-15T22:21:28.041912+0000","last_became_peered":"2021-03-15T22:21:28.041912+0000","last_unstale":"2021-03-15T22:22:40.054314+0000","last_undegraded":"2021-03-15T22:22:40.054314+0000","last_fullsized":"2021-03-15T22:22:40.054314+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:21:59.228452+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"245'137","reported_seq":"663","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.147540+0000","last_change":"2021-03-15T22:22:37.173924+0000","last_active":"2021-03-15T22:22:40.147540+0000","last_peered":"2021-03-15T22:22:40.147540+0000","last_clean":"2021-03-15T22:22:40.147540+0000","last_became_active":"2021-03-15T22:21:31.022979+0000","last_became_peered":"2021-03-15T22:21:31.022979+0000","last_unstale":"2021-03-15T22:22:40.147540+0000","last_undegraded":"2021-03-15T22:22:40.147540+0000","last_fullsized":"2021-03-15T22:22:40.147540+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'121","last_scrub_stamp":"2021-03-15T22:21:51.825081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:21:51.825081+0000","log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9816573,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":88,"num_read_kb":76155,"num_write":56,"num_write_kb":19735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":7516025,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"87","length":"1"},{"start":"8e","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"200","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.048281+0000","last_change":"2021-03-15T22:21:22.029900+0000","last_active":"2021-03-15T22:22:41.048281+0000","last_peered":"2021-03-15T22:22:41.048281+0000","last_clean":"2021-03-15T22:22:41.048281+0000","last_became_active":"2021-03-15T22:18:00.019457+0000","last_became_peered":"2021-03-15T22:18:00.019457+0000","last_unstale":"2021-03-15T22:22:41.048281+0000","last_undegraded":"2021-03-15T22:22:41.048281+0000","last_fullsized":"2021-03-15T22:22:41.048281+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:22.029836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:22.029836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"152'4","reported_seq":"281","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.055870+0000","last_change":"2021-03-15T22:21:23.309388+0000","last_active":"2021-03-15T22:22:40.055870+0000","last_peered":"2021-03-15T22:22:40.055870+0000","last_clean":"2021-03-15T22:22:40.055870+0000","last_became_active":"2021-03-15T22:21:06.019218+0000","last_became_peered":"2021-03-15T22:21:06.019218+0000","last_unstale":"2021-03-15T22:22:40.055870+0000","last_undegraded":"2021-03-15T22:22:40.055870+0000","last_fullsized":"2021-03-15T22:22:40.055870+0000","mapping_epoch":60,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'4","last_scrub_stamp":"2021-03-15T22:21:23.309323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:23.309323+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"90","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.050995+0000","last_change":"2021-03-15T22:21:50.930666+0000","last_active":"2021-03-15T22:22:41.050995+0000","last_peered":"2021-03-15T22:22:41.050995+0000","last_clean":"2021-03-15T22:22:41.050995+0000","last_became_active":"2021-03-15T22:21:28.048995+0000","last_became_peered":"2021-03-15T22:21:28.048995+0000","last_unstale":"2021-03-15T22:22:41.050995+0000","last_undegraded":"2021-03-15T22:22:41.050995+0000","last_fullsized":"2021-03-15T22:22:41.050995+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:50.930571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:50.930571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"240","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.048392+0000","last_change":"2021-03-15T22:21:56.811802+0000","last_active":"2021-03-15T22:22:43.048392+0000","last_peered":"2021-03-15T22:22:43.048392+0000","last_clean":"2021-03-15T22:22:43.048392+0000","last_became_active":"2021-03-15T22:21:56.030199+0000","last_became_peered":"2021-03-15T22:21:56.030199+0000","last_unstale":"2021-03-15T22:22:43.048392+0000","last_undegraded":"2021-03-15T22:22:43.048392+0000","last_fullsized":"2021-03-15T22:22:43.048392+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:21:52.622751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:52.622751+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"213'38","reported_seq":"524","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.067867+0000","last_change":"2021-03-15T22:22:37.174445+0000","last_active":"2021-03-15T22:22:40.067867+0000","last_peered":"2021-03-15T22:22:40.067867+0000","last_clean":"2021-03-15T22:22:40.067867+0000","last_became_active":"2021-03-15T22:20:05.039641+0000","last_became_peered":"2021-03-15T22:20:05.039641+0000","last_unstale":"2021-03-15T22:22:40.067867+0000","last_undegraded":"2021-03-15T22:22:40.067867+0000","last_fullsized":"2021-03-15T22:22:40.067867+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'34","last_scrub_stamp":"2021-03-15T22:22:05.628759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:05.628759+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":62607,"num_write":15,"num_write_kb":6602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"87","length":"1"},{"start":"8e","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"88","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.532461+0000","last_change":"2021-03-15T22:22:37.642100+0000","last_active":"2021-03-15T22:22:39.532461+0000","last_peered":"2021-03-15T22:22:39.532461+0000","last_clean":"2021-03-15T22:22:39.532461+0000","last_became_active":"2021-03-15T22:21:28.044508+0000","last_became_peered":"2021-03-15T22:21:28.044508+0000","last_unstale":"2021-03-15T22:22:39.532461+0000","last_undegraded":"2021-03-15T22:22:39.532461+0000","last_fullsized":"2021-03-15T22:22:39.532461+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:37.642038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:22:37.642038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"147'5","reported_seq":"247","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:42.056353+0000","last_change":"2021-03-15T22:21:55.110240+0000","last_active":"2021-03-15T22:22:42.056353+0000","last_peered":"2021-03-15T22:22:42.056353+0000","last_clean":"2021-03-15T22:22:42.056353+0000","last_became_active":"2021-03-15T22:21:02.017066+0000","last_became_peered":"2021-03-15T22:21:02.017066+0000","last_unstale":"2021-03-15T22:22:42.056353+0000","last_undegraded":"2021-03-15T22:22:42.056353+0000","last_fullsized":"2021-03-15T22:22:42.056353+0000","mapping_epoch":60,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'5","last_scrub_stamp":"2021-03-15T22:21:55.110127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:55.110127+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"208","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.055221+0000","last_change":"2021-03-15T22:21:55.222551+0000","last_active":"2021-03-15T22:22:40.055221+0000","last_peered":"2021-03-15T22:22:40.055221+0000","last_clean":"2021-03-15T22:22:40.055221+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:22:40.055221+0000","last_undegraded":"2021-03-15T22:22:40.055221+0000","last_fullsized":"2021-03-15T22:22:40.055221+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:55.222484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:21:55.222484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"233","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.055788+0000","last_change":"2021-03-15T22:21:33.258467+0000","last_active":"2021-03-15T22:22:40.055788+0000","last_peered":"2021-03-15T22:22:40.055788+0000","last_clean":"2021-03-15T22:22:40.055788+0000","last_became_active":"2021-03-15T22:17:55.992213+0000","last_became_peered":"2021-03-15T22:17:55.992213+0000","last_unstale":"2021-03-15T22:22:40.055788+0000","last_undegraded":"2021-03-15T22:22:40.055788+0000","last_fullsized":"2021-03-15T22:22:40.055788+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:21:33.258382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:33.258382+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"243'55","reported_seq":"554","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.048740+0000","last_change":"2021-03-15T22:22:43.048740+0000","last_active":"2021-03-15T22:22:43.048740+0000","last_peered":"2021-03-15T22:22:43.048740+0000","last_clean":"2021-03-15T22:22:43.048740+0000","last_became_active":"2021-03-15T22:20:05.033636+0000","last_became_peered":"2021-03-15T22:20:05.033636+0000","last_unstale":"2021-03-15T22:22:43.048740+0000","last_undegraded":"2021-03-15T22:22:43.048740+0000","last_fullsized":"2021-03-15T22:22:43.048740+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'41","last_scrub_stamp":"2021-03-15T22:21:33.451818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:21:33.451818+0000","log_size":55,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2369820,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":61,"num_read_kb":83455,"num_write":23,"num_write_kb":5161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"84","length":"1"},{"start":"87","length":"1"},{"start":"8e","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"213","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.048477+0000","last_change":"2021-03-15T22:21:28.050618+0000","last_active":"2021-03-15T22:22:43.048477+0000","last_peered":"2021-03-15T22:22:43.048477+0000","last_clean":"2021-03-15T22:22:43.048477+0000","last_became_active":"2021-03-15T22:21:28.050311+0000","last_became_peered":"2021-03-15T22:21:28.050311+0000","last_unstale":"2021-03-15T22:22:43.048477+0000","last_undegraded":"2021-03-15T22:22:43.048477+0000","last_fullsized":"2021-03-15T22:22:43.048477+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:58.074327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_clean_scrub_stamp":"2021-03-15T22:20:58.074327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"226","reported_epoch":"246","state":"active+clean","last_fresh":"2021-03-15T22:22:41.048328+0000","last_change":"2021-03-15T22:22:03.851108+0000","last_active":"2021-03-15T22:22:41.048328+0000","last_peered":"2021-03-15T22:22:41.048328+0000","last_clean":"2021-03-15T22:22:41.048328+0000","last_became_active":"2021-03-15T22:21:14.032519+0000","last_became_peered":"2021-03-15T22:21:14.032519+0000","last_unstale":"2021-03-15T22:22:41.048328+0000","last_undegraded":"2021-03-15T22:22:41.048328+0000","last_fullsized":"2021-03-15T22:22:41.048328+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:22:03.851047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"209","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.531952+0000","last_change":"2021-03-15T22:22:38.664690+0000","last_active":"2021-03-15T22:22:39.531952+0000","last_peered":"2021-03-15T22:22:39.531952+0000","last_clean":"2021-03-15T22:22:39.531952+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:22:39.531952+0000","last_undegraded":"2021-03-15T22:22:39.531952+0000","last_fullsized":"2021-03-15T22:22:39.531952+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:38.664626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.664626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"115'33","reported_seq":"254","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.055540+0000","last_change":"2021-03-15T22:22:27.911298+0000","last_active":"2021-03-15T22:22:40.055540+0000","last_peered":"2021-03-15T22:22:40.055540+0000","last_clean":"2021-03-15T22:22:40.055540+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:22:40.055540+0000","last_undegraded":"2021-03-15T22:22:40.055540+0000","last_fullsized":"2021-03-15T22:22:40.055540+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'33","last_scrub_stamp":"2021-03-15T22:22:27.911223+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:22:27.911223+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"245'212","reported_seq":"761","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.063757+0000","last_change":"2021-03-15T22:22:37.180972+0000","last_active":"2021-03-15T22:22:40.063757+0000","last_peered":"2021-03-15T22:22:40.063757+0000","last_clean":"2021-03-15T22:22:40.063757+0000","last_became_active":"2021-03-15T22:21:59.034617+0000","last_became_peered":"2021-03-15T22:21:59.034617+0000","last_unstale":"2021-03-15T22:22:40.063757+0000","last_undegraded":"2021-03-15T22:22:40.063757+0000","last_fullsized":"2021-03-15T22:22:40.063757+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":212,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13566957,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":130,"num_read_kb":114595,"num_write":97,"num_write_kb":36889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":7181439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"87","length":"1"},{"start":"8e","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"202","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.055901+0000","last_change":"2021-03-15T22:22:35.877634+0000","last_active":"2021-03-15T22:22:40.055901+0000","last_peered":"2021-03-15T22:22:40.055901+0000","last_clean":"2021-03-15T22:22:40.055901+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:22:40.055901+0000","last_undegraded":"2021-03-15T22:22:40.055901+0000","last_fullsized":"2021-03-15T22:22:40.055901+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:35.877570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:35.877570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"155'4","reported_seq":"289","reported_epoch":"248","state":"active+clean","last_fresh":"2021-03-15T22:22:43.047767+0000","last_change":"2021-03-15T22:22:38.649104+0000","last_active":"2021-03-15T22:22:43.047767+0000","last_peered":"2021-03-15T22:22:43.047767+0000","last_clean":"2021-03-15T22:22:43.047767+0000","last_became_active":"2021-03-15T22:21:10.013838+0000","last_became_peered":"2021-03-15T22:21:10.013838+0000","last_unstale":"2021-03-15T22:22:43.047767+0000","last_undegraded":"2021-03-15T22:22:43.047767+0000","last_fullsized":"2021-03-15T22:22:43.047767+0000","mapping_epoch":60,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'4","last_scrub_stamp":"2021-03-15T22:22:38.649035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.649035+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"56","reported_epoch":"244","state":"active+clean","last_fresh":"2021-03-15T22:22:39.177304+0000","last_change":"2021-03-15T22:21:56.038426+0000","last_active":"2021-03-15T22:22:39.177304+0000","last_peered":"2021-03-15T22:22:39.177304+0000","last_clean":"2021-03-15T22:22:39.177304+0000","last_became_active":"2021-03-15T22:21:56.038093+0000","last_became_peered":"2021-03-15T22:21:56.038093+0000","last_unstale":"2021-03-15T22:22:39.177304+0000","last_undegraded":"2021-03-15T22:22:39.177304+0000","last_fullsized":"2021-03-15T22:22:39.177304+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:58.817670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:58.817670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"254","reported_epoch":"245","state":"active+clean","last_fresh":"2021-03-15T22:22:40.054724+0000","last_change":"2021-03-15T22:21:56.052881+0000","last_active":"2021-03-15T22:22:40.054724+0000","last_peered":"2021-03-15T22:22:40.054724+0000","last_clean":"2021-03-15T22:22:40.054724+0000","last_became_active":"2021-03-15T22:21:56.038572+0000","last_became_peered":"2021-03-15T22:21:56.038572+0000","last_unstale":"2021-03-15T22:22:40.054724+0000","last_undegraded":"2021-03-15T22:22:40.054724+0000","last_fullsized":"2021-03-15T22:22:40.054724+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:21:04.375645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:04.375645+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":157300261,"num_objects":77,"num_object_clones":38,"num_object_copies":231,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":77,"num_whiteouts":5,"num_read":1538,"num_read_kb":1492904,"num_write":1169,"num_write_kb":438542,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":85225199,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":375791616,"data_stored":496448625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2621,"ondisk_log_size":2621,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":290,"num_read_kb":285,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1064,"num_read_kb":569,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":13292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":73560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299454,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":74332,"kb_used_data":55316,"kb_used_omap":0,"kb_used_meta":19008,"kb_avail":93670820,"statfs":{"total":95995035648,"available":95918919680,"internally_reserved":0,"allocated":56643584,"data_stored":69468126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19464192},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.505,"15min":0.501},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.499},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52000000000000002,"15min":0.50800000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.8799999999999999,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.38100000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.49399999999999999,"15min":0.48499999999999999},"min":{"1min":0.30299999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.498,"15min":0.496},"min":{"1min":0.32900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.48699999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47899999999999998,"15min":0.48199999999999998},"min":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.51700000000000002,"15min":0.498},"min":{"1min":0.33700000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.44900000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.53000000000000003,"15min":0.56200000000000006},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.95999999999999996,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.35999999999999999,"5min":0.307,"15min":0.307},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.46400000000000002,"15min":0.432},"min":{"1min":0.32800000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.316},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.48399999999999999,"15min":0.442},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.29599999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.48799999999999999,"15min":0.442},"min":{"1min":0.30299999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.5110000000000001,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52800000000000002,"15min":0.501},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.255}]},{"osd":6,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.53800000000000003,"15min":0.503},"min":{"1min":0.379,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.502},"min":{"1min":0.38800000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.34300000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528385,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":47424,"kb_used_data":33848,"kb_used_omap":0,"kb_used_meta":13568,"kb_avail":93697728,"statfs":{"total":95995035648,"available":95946473472,"internally_reserved":0,"allocated":34660352,"data_stored":42150752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":13893632},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:21:38 2021","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.49399999999999999,"15min":0.49399999999999999},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.995,"5min":0.995,"15min":0.995},"last":0.438},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.49299999999999999,"15min":0.49299999999999999},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.1899999999999999,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.44600000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:21:38 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51500000000000001,"15min":0.51600000000000001},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":1.05,"5min":1.05,"15min":1.05},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.48299999999999998,"15min":0.47899999999999998},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.72199999999999998,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.45700000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:21:38 2021","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.50800000000000001,"15min":0.499},"min":{"1min":0.33800000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.62,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.5,"15min":0.502},"min":{"1min":0.34899999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":1.2170000000000001,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.40799999999999997}]},{"osd":3,"last update":"Mon Mar 15 22:21:38 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.52000000000000002,"15min":0.53000000000000003},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.026},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.52700000000000002,"15min":0.52800000000000002},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":0.76500000000000001,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.42499999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:21:38 2021","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.41599999999999998,"15min":0.39800000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.52400000000000002,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.46500000000000002,"15min":0.437},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.30499999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:21:38 2021","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.46800000000000003,"15min":0.442},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":1.135,"5min":1.135,"15min":1.135},"last":0.497},{"interface":"front","average":{"1min":0.502,"5min":0.49199999999999999,"15min":0.46800000000000003},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.35099999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:21:55 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.752,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54900000000000004,"15min":0.53500000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.46899999999999997}]}]},{"osd":1,"up_from":13,"seq":55834574931,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78400,"kb_used_data":57912,"kb_used_omap":0,"kb_used_meta":20480,"kb_avail":93666752,"statfs":{"total":95995035648,"available":95914754048,"internally_reserved":0,"allocated":59301888,"data_stored":81982174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20971520},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:22:20 2021","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.89900000000000002,"15min":0.52300000000000002},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.191},"max":{"1min":0.49199999999999999,"5min":54.427999999999997,"15min":54.427999999999997},"last":0.315},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.91000000000000003,"15min":0.54600000000000004},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.23,"5min":54.356999999999999,"15min":54.356999999999999},"last":0.30199999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:21:39 2021","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.433,"15min":0.42099999999999999},"min":{"1min":0.23400000000000001,"5min":0.223,"15min":0.223},"max":{"1min":1.1220000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.33400000000000002,"5min":0.38,"15min":0.40799999999999997},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":0.60499999999999998,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.28699999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:21:51 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41899999999999998,"15min":0.433},"min":{"1min":0.27000000000000002,"5min":0.222,"15min":0.222},"max":{"1min":1.393,"5min":1.796,"15min":1.796},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.44700000000000001,"15min":0.48299999999999998},"min":{"1min":0.23000000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.135,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.33300000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:22:03 2021","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.57899999999999996,"15min":0.58299999999999996},"min":{"1min":0.35799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.3439999999999999,"5min":3.3439999999999999,"15min":3.3439999999999999},"last":0.55600000000000005},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.65300000000000002,"15min":0.63500000000000001},"min":{"1min":0.39500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"last":0.61799999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:22:20 2021","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58499999999999996,"15min":0.64700000000000002},"min":{"1min":0.42899999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81999999999999995,"5min":2.153,"15min":2.153},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.71999999999999997,"15min":1.101},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1950000000000001,"5min":15.472,"15min":15.472},"last":0.63900000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:21:39 2021","interfaces":[{"interface":"back","average":{"1min":0.58599999999999997,"5min":0.63,"15min":0.67800000000000005},"min":{"1min":0.434,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.101,"5min":1.75,"15min":1.75},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.60699999999999998,"15min":0.64100000000000001},"min":{"1min":0.40799999999999997,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.75,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.60699999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:21:57 2021","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.77500000000000002,"15min":0.94899999999999995},"min":{"1min":0.38900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":1.081,"5min":8.8100000000000005,"15min":8.8100000000000005},"last":0.71099999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59999999999999998,"15min":0.63300000000000001},"min":{"1min":0.41899999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82699999999999996,"5min":1.325,"15min":1.325},"last":0.58399999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":62668,"kb_used_data":42820,"kb_used_omap":0,"kb_used_meta":19840,"kb_avail":93682484,"statfs":{"total":95995035648,"available":95930863616,"internally_reserved":0,"allocated":43847680,"data_stored":52862564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20316160},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:22:19 2021","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34000000000000002,"15min":0.35699999999999998},"min":{"1min":0.23799999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.47099999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36899999999999999,"15min":0.375},"min":{"1min":0.24299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.54200000000000004,"5min":0.63500000000000001,"15min":0.63800000000000001},"last":0.22700000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:22:31 2021","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.36899999999999999,"15min":0.40100000000000002},"min":{"1min":0.29199999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.58699999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.34899999999999998,"15min":0.38900000000000001},"min":{"1min":0.26900000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.625,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.28599999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:21:57 2021","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38100000000000001,"15min":0.42499999999999999},"min":{"1min":0.249,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51800000000000002,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.40200000000000002,"15min":0.437},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.31900000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:22:08 2021","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.56399999999999995},"min":{"1min":0.314,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70399999999999996,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69099999999999995,"5min":1.034,"15min":1.034},"last":0.63700000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:22:23 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57699999999999996,"15min":0.60799999999999998},"min":{"1min":0.36899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72999999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.58299999999999996},"min":{"1min":0.39400000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70899999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.53500000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:22:36 2021","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.39700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76400000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.45800000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78200000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.71399999999999997}]},{"osd":7,"last update":"Mon Mar 15 22:22:02 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79300000000000004,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.66200000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58499999999999996,"15min":0.57699999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.67300000000000004}]}]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99324,"kb_used_data":79412,"kb_used_omap":0,"kb_used_meta":19904,"kb_avail":93645828,"statfs":{"total":95995035648,"available":95893327872,"internally_reserved":0,"allocated":81317888,"data_stored":109808939,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20381696},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:22:37 2021","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.35099999999999998,"15min":0.35899999999999999},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.442,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.39000000000000001,"15min":0.38700000000000001},"min":{"1min":0.23599999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.54900000000000004,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.21099999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:22:37 2021","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.44,"15min":0.40600000000000003},"min":{"1min":0.28299999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.47499999999999998,"5min":5.4489999999999998,"15min":5.4489999999999998},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.52200000000000002,"15min":0.47999999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.495,"5min":11.659000000000001,"15min":11.659000000000001},"last":0.28899999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:21:48 2021","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.40999999999999998,"15min":0.442},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.443,"15min":0.502},"min":{"1min":0.27300000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:22:05 2021","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55600000000000005,"15min":0.56699999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.34100000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:22:32 2021","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73799999999999999,"5min":1.954,"15min":1.954},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.57399999999999995},"min":{"1min":0.38200000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82099999999999995,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.27600000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:22:37 2021","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59299999999999997,"15min":0.57099999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77000000000000002,"5min":1.875,"15min":1.875},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57499999999999996,"15min":0.56999999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.76300000000000001,"5min":1.609,"15min":1.609},"last":0.38300000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:22:01 2021","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.57299999999999995,"15min":0.55600000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59699999999999998,"15min":0.58599999999999997},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":0.41099999999999998}]}]},{"osd":3,"up_from":0,"seq":111669149771,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986248,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":64972,"kb_used_data":46468,"kb_used_omap":0,"kb_used_meta":18496,"kb_avail":93680180,"statfs":{"total":95995035648,"available":95928504320,"internally_reserved":0,"allocated":47583232,"data_stored":57407626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18939904},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:22:10 2021","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47099999999999997,"15min":0.51000000000000001},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.372},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.65500000000000003,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.497}]},{"osd":1,"last update":"Mon Mar 15 22:22:10 2021","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.48599999999999999,"15min":0.51200000000000001},"min":{"1min":0.376,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.69499999999999995,"5min":1.631,"15min":1.631},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.36199999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67800000000000005,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.38300000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:22:10 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48999999999999999,"15min":0.51800000000000002},"min":{"1min":0.36599999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.624,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.439},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.34599999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.66400000000000003,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.35899999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:22:10 2021","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54200000000000004,"15min":0.56200000000000006},"min":{"1min":0.42299999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70299999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53800000000000003,"15min":0.57499999999999996},"min":{"1min":0.41599999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69199999999999995,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.48799999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.441,"15min":0.40400000000000003},"min":{"1min":0.32100000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67000000000000004,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.46800000000000003,"15min":0.42199999999999999},"min":{"1min":0.313,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.42599999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:21:39 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.443,"15min":0.42099999999999999},"min":{"1min":0.30599999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.76800000000000002,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.499},"min":{"1min":0.38900000000000001,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.76500000000000001,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.50900000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:21:53 2021","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50600000000000001,"15min":0.48299999999999998},"min":{"1min":0.39200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.73799999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.53500000000000003,"15min":0.51900000000000002},"min":{"1min":0.38600000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.46100000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757317,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":78688,"kb_used_data":59416,"kb_used_omap":0,"kb_used_meta":19264,"kb_avail":93666464,"statfs":{"total":95995035648,"available":95914459136,"internally_reserved":0,"allocated":60841984,"data_stored":82605378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19726336},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.67800000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.50600000000000001,"15min":0.502},"min":{"1min":0.28299999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.72199999999999998,"5min":0.755,"15min":0.755},"last":0.5}]},{"osd":1,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.51200000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.68700000000000006,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.36399999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72099999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.497,"15min":0.504},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67600000000000005,"5min":1.006,"15min":1.006},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52800000000000002,"15min":0.51400000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.746,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.53600000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.38,"5min":0.308,"15min":0.308},"max":{"1min":0.74199999999999999,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54700000000000004,"15min":0.53200000000000003},"min":{"1min":0.34100000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.51900000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44500000000000001,"15min":0.38700000000000001},"min":{"1min":0.28199999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.42199999999999999},"min":{"1min":0.35199999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.65500000000000003,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.46500000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:22:42 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.49399999999999999,"15min":0.46400000000000002},"min":{"1min":0.34100000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72799999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56100000000000005,"15min":0.52700000000000002},"min":{"1min":0.435,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71599999999999997,"5min":1.327,"15min":1.327},"last":0.56200000000000006}]},{"osd":7,"last update":"Mon Mar 15 22:21:55 2021","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53200000000000003,"15min":0.50600000000000001},"min":{"1min":0.34699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54400000000000004,"15min":0.52100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.59599999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":7530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":15174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":8670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":307200,"data_stored":11162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":7262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":12852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":7922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":41000960,"data_stored":51190660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":56238080,"data_stored":80287553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":78409728,"data_stored":108103918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12046336,"data_stored":11973832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":44638208,"data_stored":55725122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":58007552,"data_stored":80919524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":31694848,"data_stored":40475947,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":53755904,"data_stored":67772069,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:22:46.762 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:22:46.764 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:46.787 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:46.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:46.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:46.889 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:46.911 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:46.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:47.014 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:47.038 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:47.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:47.040 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:47.140 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:47.163 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:47.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:47.216 INFO:tasks.rados.rados.0.smithi076.stdout:785: done (6 left) 2021-03-15T22:22:47.216 INFO:tasks.rados.rados.0.smithi076.stdout:786: done (5 left) 2021-03-15T22:22:47.216 INFO:tasks.rados.rados.0.smithi076.stdout:789: done (4 left) 2021-03-15T22:22:47.217 INFO:tasks.rados.rados.0.smithi076.stdout:790: done (3 left) 2021-03-15T22:22:47.217 INFO:tasks.rados.rados.0.smithi076.stdout:791: done (2 left) 2021-03-15T22:22:47.217 INFO:tasks.rados.rados.0.smithi076.stdout:792: done (1 left) 2021-03-15T22:22:47.217 INFO:tasks.rados.rados.0.smithi076.stdout:794: done (0 left) 2021-03-15T22:22:47.217 INFO:tasks.rados.rados.0.smithi076.stdout:795: read oid 19 snap -1 2021-03-15T22:22:47.218 INFO:tasks.rados.rados.0.smithi076.stdout:795: expect (ObjNum 296 snap 78 seq_num 296) 2021-03-15T22:22:47.218 INFO:tasks.rados.rados.0.smithi076.stdout:796: read oid 42 snap -1 2021-03-15T22:22:47.218 INFO:tasks.rados.rados.0.smithi076.stdout:796: expect deleted 2021-03-15T22:22:47.218 INFO:tasks.rados.rados.0.smithi076.stdout:797: read oid 26 snap 78 2021-03-15T22:22:47.219 INFO:tasks.rados.rados.0.smithi076.stdout:797: expect (ObjNum 291 snap 78 seq_num 291) 2021-03-15T22:22:47.219 INFO:tasks.rados.rados.0.smithi076.stdout:798: rollback oid 3 current snap is 81 2021-03-15T22:22:47.219 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 65 2021-03-15T22:22:47.219 INFO:tasks.rados.rados.0.smithi076.stdout:799: copy_from oid 33 from oid 43 current snap is 81 2021-03-15T22:22:47.219 INFO:tasks.rados.rados.0.smithi076.stdout:800: copy_from oid 36 from oid 38 current snap is 81 2021-03-15T22:22:47.220 INFO:tasks.rados.rados.0.smithi076.stdout:801: write oid 17 current snap is 81 2021-03-15T22:22:47.220 INFO:tasks.rados.rados.0.smithi076.stdout:801: seq_num 311 ranges {426977=456559,1482841=799764,2958080=268897} 2021-03-15T22:22:47.220 INFO:tasks.rados.rados.0.smithi076.stdout:801: writing smithi07694878-17 from 426977 to 883536 tid 1 2021-03-15T22:22:47.228 INFO:tasks.rados.rados.0.smithi076.stdout:801: writing smithi07694878-17 from 1482841 to 2282605 tid 2 2021-03-15T22:22:47.234 INFO:tasks.rados.rados.0.smithi076.stdout:801: writing smithi07694878-17 from 2958080 to 3226977 tid 3 2021-03-15T22:22:47.234 INFO:tasks.rados.rados.0.smithi076.stdout:802: snap_remove snap 74 2021-03-15T22:22:47.258 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:22:47.259 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:22:47.259 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:22:47.259 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:22:47.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.260 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:277 7.0 7:08414d91:::smithi07694878-42:head [copy-from ver 6] snapc 8=[8] ondisk+write+known_if_redirected e104)", 2021-03-15T22:22:47.260 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.220857+0000", 2021-03-15T22:22:47.260 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 148.034122473, 2021-03-15T22:22:47.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01426084, 2021-03-15T22:22:47.261 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.261 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.261 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.261 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.262 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.262 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 277 2021-03-15T22:22:47.262 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.262 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.263 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.263 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.263 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220857+0000", 2021-03-15T22:22:47.263 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.264 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.264 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.264 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220857+0000", 2021-03-15T22:22:47.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.73e-06 2021-03-15T22:22:47.265 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.265 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.265 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.265 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220858+0000", 2021-03-15T22:22:47.265 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.277e-06 2021-03-15T22:22:47.266 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.266 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.266 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.266 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220860+0000", 2021-03-15T22:22:47.267 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8489999999999995e-06 2021-03-15T22:22:47.267 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.267 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.267 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.267 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220868+0000", 2021-03-15T22:22:47.268 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5480000000000003e-06 2021-03-15T22:22:47.268 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.268 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.268 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.269 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220873+0000", 2021-03-15T22:22:47.269 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001357249 2021-03-15T22:22:47.269 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.269 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.270 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.270 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222230+0000", 2021-03-15T22:22:47.270 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4117999999999998e-05 2021-03-15T22:22:47.270 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.270 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.271 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.271 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222294+0000", 2021-03-15T22:22:47.271 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035570179999999999 2021-03-15T22:22:47.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.272 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:22:47.272 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.225851+0000", 2021-03-15T22:22:47.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038919620000000001 2021-03-15T22:22:47.272 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.229743+0000", 2021-03-15T22:22:47.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024645940000000001 2021-03-15T22:22:47.273 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.274 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.274 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.232208+0000", 2021-03-15T22:22:47.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002799171 2021-03-15T22:22:47.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235007+0000", 2021-03-15T22:22:47.276 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3729e-05 2021-03-15T22:22:47.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.276 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235061+0000", 2021-03-15T22:22:47.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6595000000000003e-05 2021-03-15T22:22:47.277 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.277 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.278 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235117+0000", 2021-03-15T22:22:47.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01426084 2021-03-15T22:22:47.278 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.278 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.278 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.279 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.279 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.279 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:278 7.0 7:08414d91:::smithi07694878-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e104)", 2021-03-15T22:22:47.279 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.220878+0000", 2021-03-15T22:22:47.279 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 148.03410104, 2021-03-15T22:22:47.280 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014453041999999999, 2021-03-15T22:22:47.280 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.280 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:22:47.280 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.281 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.281 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.281 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 278 2021-03-15T22:22:47.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.282 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.282 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.282 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.282 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220878+0000", 2021-03-15T22:22:47.282 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.283 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.283 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.283 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.283 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220878+0000", 2021-03-15T22:22:47.284 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.037e-06 2021-03-15T22:22:47.284 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.284 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.284 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.284 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220879+0000", 2021-03-15T22:22:47.285 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.187e-06 2021-03-15T22:22:47.285 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.285 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.285 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.286 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220880+0000", 2021-03-15T22:22:47.286 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.879e-06 2021-03-15T22:22:47.286 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.286 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.286 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.287 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220886+0000", 2021-03-15T22:22:47.287 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0269999999999997e-06 2021-03-15T22:22:47.287 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.287 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.288 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.288 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220890+0000", 2021-03-15T22:22:47.288 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001548542 2021-03-15T22:22:47.288 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.288 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.289 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.289 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222439+0000", 2021-03-15T22:22:47.289 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1516000000000001e-05 2021-03-15T22:22:47.289 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.290 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.290 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:47.290 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222470+0000", 2021-03-15T22:22:47.290 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066482809999999998 2021-03-15T22:22:47.290 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.291 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.291 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.291 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.229118+0000", 2021-03-15T22:22:47.291 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6447999999999997e-05 2021-03-15T22:22:47.292 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.292 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.292 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:47.292 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.229195+0000", 2021-03-15T22:22:47.292 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0059693100000000002 2021-03-15T22:22:47.293 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.293 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.293 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.293 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235164+0000", 2021-03-15T22:22:47.294 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4153999999999994e-05 2021-03-15T22:22:47.294 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.294 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.294 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.294 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235248+0000", 2021-03-15T22:22:47.295 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2661e-05 2021-03-15T22:22:47.295 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.295 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.295 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.296 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235331+0000", 2021-03-15T22:22:47.296 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014453041999999999 2021-03-15T22:22:47.296 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.296 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.296 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.297 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.297 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.297 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:294 7.0 7:0cc37e3d:::smithi07694878-21:head [copy-from ver 3] snapc 8=[8] ondisk+write+known_if_redirected e104)", 2021-03-15T22:22:47.297 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.394807+0000", 2021-03-15T22:22:47.297 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 147.86017192899999, 2021-03-15T22:22:47.298 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018445559, 2021-03-15T22:22:47.298 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.298 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.298 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.299 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.299 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.299 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 294 2021-03-15T22:22:47.299 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.299 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.300 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.300 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.300 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394807+0000", 2021-03-15T22:22:47.300 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.301 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.301 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.301 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.301 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394807+0000", 2021-03-15T22:22:47.301 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9659999999999999e-06 2021-03-15T22:22:47.302 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.302 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.302 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.302 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394809+0000", 2021-03-15T22:22:47.303 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2610000000000001e-06 2021-03-15T22:22:47.303 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.303 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.303 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.303 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394810+0000", 2021-03-15T22:22:47.304 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0881e-05 2021-03-15T22:22:47.304 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.304 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.304 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.304 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394821+0000", 2021-03-15T22:22:47.305 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8509999999999994e-06 2021-03-15T22:22:47.305 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.305 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.305 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.305 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394831+0000", 2021-03-15T22:22:47.306 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035723980000000001 2021-03-15T22:22:47.306 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.306 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.306 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.307 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398403+0000", 2021-03-15T22:22:47.307 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5474000000000004e-05 2021-03-15T22:22:47.307 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.307 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.307 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.308 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398469+0000", 2021-03-15T22:22:47.308 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0069378080000000002 2021-03-15T22:22:47.308 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.308 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.309 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:22:47.309 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.405407+0000", 2021-03-15T22:22:47.309 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033175689999999998 2021-03-15T22:22:47.309 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.309 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.310 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.310 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.408724+0000", 2021-03-15T22:22:47.310 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042654010000000003 2021-03-15T22:22:47.310 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.310 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.311 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.311 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.412990+0000", 2021-03-15T22:22:47.311 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016210500000000001 2021-03-15T22:22:47.311 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.312 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.312 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.312 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413152+0000", 2021-03-15T22:22:47.312 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0185999999999997e-05 2021-03-15T22:22:47.312 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.313 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.313 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.313 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413202+0000", 2021-03-15T22:22:47.313 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0658999999999997e-05 2021-03-15T22:22:47.314 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.314 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.314 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.314 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413253+0000", 2021-03-15T22:22:47.314 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018445559 2021-03-15T22:22:47.315 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.315 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.315 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.315 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.316 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.316 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:295 7.0 7:0cc37e3d:::smithi07694878-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e104)", 2021-03-15T22:22:47.316 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.394836+0000", 2021-03-15T22:22:47.316 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 147.86014307400001, 2021-03-15T22:22:47.316 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018595192999999999, 2021-03-15T22:22:47.317 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.317 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:22:47.317 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.317 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.318 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.318 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 295 2021-03-15T22:22:47.318 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.318 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.318 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.319 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.319 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394836+0000", 2021-03-15T22:22:47.319 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.319 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.320 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.320 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.320 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394836+0000", 2021-03-15T22:22:47.320 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4300000000000002e-07 2021-03-15T22:22:47.320 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.321 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.321 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.321 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394837+0000", 2021-03-15T22:22:47.321 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5799999999999998e-07 2021-03-15T22:22:47.321 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.322 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.322 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.322 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394838+0000", 2021-03-15T22:22:47.322 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9940000000000002e-06 2021-03-15T22:22:47.323 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.323 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.323 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.323 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394842+0000", 2021-03-15T22:22:47.324 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.339e-06 2021-03-15T22:22:47.324 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.324 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.324 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.324 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394844+0000", 2021-03-15T22:22:47.325 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003783809 2021-03-15T22:22:47.325 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.325 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.325 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.325 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398628+0000", 2021-03-15T22:22:47.326 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0221e-05 2021-03-15T22:22:47.326 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.326 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.326 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:47.327 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398668+0000", 2021-03-15T22:22:47.327 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0094639070000000006 2021-03-15T22:22:47.327 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.327 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.327 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.328 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.408132+0000", 2021-03-15T22:22:47.328 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0552999999999999e-05 2021-03-15T22:22:47.328 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.328 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.328 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:47.329 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.408212+0000", 2021-03-15T22:22:47.329 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050811019999999997 2021-03-15T22:22:47.329 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.329 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.329 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.330 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413294+0000", 2021-03-15T22:22:47.330 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9265999999999997e-05 2021-03-15T22:22:47.330 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.330 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.331 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.331 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413363+0000", 2021-03-15T22:22:47.331 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8300999999999995e-05 2021-03-15T22:22:47.331 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.332 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.332 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.332 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413431+0000", 2021-03-15T22:22:47.332 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018595192999999999 2021-03-15T22:22:47.332 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.333 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.333 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.333 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.333 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.333 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:366 7.0 7:0732c451:::smithi07694878-32:head [copy-from ver 9] snapc a=[a] ondisk+write+known_if_redirected e108)", 2021-03-15T22:22:47.334 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.105457+0000", 2021-03-15T22:22:47.334 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 144.14952239799999, 2021-03-15T22:22:47.334 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018098547, 2021-03-15T22:22:47.334 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.334 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.335 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.335 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.335 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.335 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 366 2021-03-15T22:22:47.336 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.336 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.336 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.336 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.336 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105457+0000", 2021-03-15T22:22:47.337 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.337 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.337 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.337 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.338 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105457+0000", 2021-03-15T22:22:47.338 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9000000000000003e-06 2021-03-15T22:22:47.338 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.338 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.338 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.339 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105463+0000", 2021-03-15T22:22:47.339 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1069999999999998e-06 2021-03-15T22:22:47.339 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.339 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.340 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.340 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105467+0000", 2021-03-15T22:22:47.340 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0707999999999999e-05 2021-03-15T22:22:47.340 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.340 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.341 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.341 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105487+0000", 2021-03-15T22:22:47.341 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4830000000000005e-06 2021-03-15T22:22:47.341 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.342 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.342 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.342 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105497+0000", 2021-03-15T22:22:47.342 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9896999999999998e-05 2021-03-15T22:22:47.342 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.343 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.343 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.343 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105577+0000", 2021-03-15T22:22:47.343 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000138195 2021-03-15T22:22:47.343 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.344 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.344 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.344 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105715+0000", 2021-03-15T22:22:47.344 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0074524730000000003 2021-03-15T22:22:47.345 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.345 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.345 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:22:47.345 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.113167+0000", 2021-03-15T22:22:47.345 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053675119999999996 2021-03-15T22:22:47.346 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.346 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.346 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.118535+0000", 2021-03-15T22:22:47.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002009102 2021-03-15T22:22:47.347 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.347 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.347 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.347 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.120544+0000", 2021-03-15T22:22:47.348 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028954580000000001 2021-03-15T22:22:47.348 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.348 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.348 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.348 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123439+0000", 2021-03-15T22:22:47.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6292999999999999e-05 2021-03-15T22:22:47.349 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.350 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123486+0000", 2021-03-15T22:22:47.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9418999999999994e-05 2021-03-15T22:22:47.350 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.350 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123555+0000", 2021-03-15T22:22:47.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018098547 2021-03-15T22:22:47.351 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.351 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.351 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.352 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:367 7.0 7:0732c451:::smithi07694878-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e108)", 2021-03-15T22:22:47.352 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.105520+0000", 2021-03-15T22:22:47.352 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 144.14945854000001, 2021-03-15T22:22:47.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018220706, 2021-03-15T22:22:47.353 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.353 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:22:47.353 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.354 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.354 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.354 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 367 2021-03-15T22:22:47.354 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.354 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.355 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.355 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.355 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105520+0000", 2021-03-15T22:22:47.355 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.356 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.356 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105520+0000", 2021-03-15T22:22:47.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.68e-06 2021-03-15T22:22:47.357 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.357 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.357 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.357 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105522+0000", 2021-03-15T22:22:47.358 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0650000000000001e-06 2021-03-15T22:22:47.358 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.358 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105524+0000", 2021-03-15T22:22:47.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9020000000000005e-06 2021-03-15T22:22:47.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.359 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.359 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.360 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105533+0000", 2021-03-15T22:22:47.360 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1390000000000003e-06 2021-03-15T22:22:47.360 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.360 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.361 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.361 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105538+0000", 2021-03-15T22:22:47.361 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034772699999999999 2021-03-15T22:22:47.361 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.361 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.362 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.362 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105886+0000", 2021-03-15T22:22:47.362 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1764000000000001e-05 2021-03-15T22:22:47.362 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.363 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:47.363 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105938+0000", 2021-03-15T22:22:47.363 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011904672999999999 2021-03-15T22:22:47.363 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.364 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.364 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.364 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.117842+0000", 2021-03-15T22:22:47.364 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4134999999999994e-05 2021-03-15T22:22:47.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.365 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:47.365 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.117917+0000", 2021-03-15T22:22:47.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056846789999999998 2021-03-15T22:22:47.366 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.366 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.366 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.367 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123601+0000", 2021-03-15T22:22:47.367 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1218999999999999e-05 2021-03-15T22:22:47.367 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.367 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.367 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.368 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123652+0000", 2021-03-15T22:22:47.368 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8722999999999997e-05 2021-03-15T22:22:47.368 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.368 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.369 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.369 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123741+0000", 2021-03-15T22:22:47.369 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018220706 2021-03-15T22:22:47.369 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.369 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.370 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.370 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.370 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.370 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:372 7.0 7:054fa7db:::smithi07694878-29:head [copy-from ver 3] snapc a=[a] ondisk+write+known_if_redirected e108)", 2021-03-15T22:22:47.370 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.121689+0000", 2021-03-15T22:22:47.371 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 144.13328971499999, 2021-03-15T22:22:47.371 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018844218999999999, 2021-03-15T22:22:47.371 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.371 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.372 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.372 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.372 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.372 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 372 2021-03-15T22:22:47.372 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.373 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.373 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.373 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.373 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121689+0000", 2021-03-15T22:22:47.373 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.374 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.374 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.374 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.374 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121689+0000", 2021-03-15T22:22:47.375 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5499999999999996e-06 2021-03-15T22:22:47.375 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.375 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.375 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121694+0000", 2021-03-15T22:22:47.376 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0380000000000001e-06 2021-03-15T22:22:47.376 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.376 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.376 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.376 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121698+0000", 2021-03-15T22:22:47.377 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9465000000000001e-05 2021-03-15T22:22:47.377 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.377 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.377 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.377 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121717+0000", 2021-03-15T22:22:47.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7600000000000001e-05 2021-03-15T22:22:47.378 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.379 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121735+0000", 2021-03-15T22:22:47.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1116000000000003e-05 2021-03-15T22:22:47.379 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.380 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121786+0000", 2021-03-15T22:22:47.380 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9568999999999996e-05 2021-03-15T22:22:47.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.380 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.381 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121876+0000", 2021-03-15T22:22:47.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085690019999999992 2021-03-15T22:22:47.381 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.381 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.382 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:22:47.382 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.130445+0000", 2021-03-15T22:22:47.382 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0025600380000000002 2021-03-15T22:22:47.382 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.382 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.383 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.383 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.133005+0000", 2021-03-15T22:22:47.383 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041963060000000003 2021-03-15T22:22:47.383 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.137201+0000", 2021-03-15T22:22:47.384 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003222258 2021-03-15T22:22:47.384 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.384 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.385 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.385 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140423+0000", 2021-03-15T22:22:47.385 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6202999999999997e-05 2021-03-15T22:22:47.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.386 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.386 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140479+0000", 2021-03-15T22:22:47.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4073999999999998e-05 2021-03-15T22:22:47.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140534+0000", 2021-03-15T22:22:47.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018844218999999999 2021-03-15T22:22:47.387 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.388 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.388 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.388 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.388 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.389 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:373 7.0 7:054fa7db:::smithi07694878-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e108)", 2021-03-15T22:22:47.389 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.121751+0000", 2021-03-15T22:22:47.389 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 144.13322810700001, 2021-03-15T22:22:47.389 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018944425000000001, 2021-03-15T22:22:47.389 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.390 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:22:47.390 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.390 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.390 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.390 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 373 2021-03-15T22:22:47.391 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.391 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.391 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.391 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121751+0000", 2021-03-15T22:22:47.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.392 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.392 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.392 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.393 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121751+0000", 2021-03-15T22:22:47.393 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.443e-06 2021-03-15T22:22:47.393 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.393 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.393 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.394 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121752+0000", 2021-03-15T22:22:47.394 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.057e-06 2021-03-15T22:22:47.394 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.394 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.394 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.395 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121754+0000", 2021-03-15T22:22:47.395 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2910000000000007e-06 2021-03-15T22:22:47.395 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.395 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.395 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.396 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121763+0000", 2021-03-15T22:22:47.396 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9339999999999997e-06 2021-03-15T22:22:47.396 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.396 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.397 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.397 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121768+0000", 2021-03-15T22:22:47.397 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024470200000000002 2021-03-15T22:22:47.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.397 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.398 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.398 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122012+0000", 2021-03-15T22:22:47.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3108000000000001e-05 2021-03-15T22:22:47.398 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:22:47.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122055+0000", 2021-03-15T22:22:47.399 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010285216 2021-03-15T22:22:47.399 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.400 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.132341+0000", 2021-03-15T22:22:47.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4697999999999994e-05 2021-03-15T22:22:47.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:47.401 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.132425+0000", 2021-03-15T22:22:47.401 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081476789999999997 2021-03-15T22:22:47.402 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.402 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.402 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140573+0000", 2021-03-15T22:22:47.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1277999999999999e-05 2021-03-15T22:22:47.403 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.403 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.403 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140624+0000", 2021-03-15T22:22:47.404 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1019000000000006e-05 2021-03-15T22:22:47.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.404 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.404 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140695+0000", 2021-03-15T22:22:47.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018944425000000001 2021-03-15T22:22:47.405 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.405 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.405 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.405 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.406 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.406 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:692 7.0 7:054fa7db:::smithi07694878-29:head [write 699191~532079,stat] snapc 2a=[2a,20,1f,1e,1b,17] ondisk+write+known_if_redirected e139)", 2021-03-15T22:22:47.406 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:54.096860+0000", 2021-03-15T22:22:47.406 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 113.158119153, 2021-03-15T22:22:47.407 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0077367729999999997, 2021-03-15T22:22:47.407 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.407 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.407 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.407 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.408 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.408 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 692 2021-03-15T22:22:47.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.408 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.409 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.096860+0000", 2021-03-15T22:22:47.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.409 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.410 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.096860+0000", 2021-03-15T22:22:47.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00059679000000000004 2021-03-15T22:22:47.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097457+0000", 2021-03-15T22:22:47.411 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3510000000000002e-06 2021-03-15T22:22:47.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.412 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.412 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097460+0000", 2021-03-15T22:22:47.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8439000000000001e-05 2021-03-15T22:22:47.413 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.413 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.413 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.413 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097478+0000", 2021-03-15T22:22:47.414 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6081e-05 2021-03-15T22:22:47.414 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.414 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.414 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.414 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097505+0000", 2021-03-15T22:22:47.415 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8774000000000003e-05 2021-03-15T22:22:47.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.415 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.415 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.415 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097573+0000", 2021-03-15T22:22:47.416 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3470000000000001e-06 2021-03-15T22:22:47.416 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.416 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:22:47.417 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.097581+0000", 2021-03-15T22:22:47.417 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001269776 2021-03-15T22:22:47.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.417 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.418 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.098850+0000", 2021-03-15T22:22:47.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000112728 2021-03-15T22:22:47.418 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.418 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.419 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.419 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.098963+0000", 2021-03-15T22:22:47.419 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00042357400000000002 2021-03-15T22:22:47.419 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.420 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:22:47.420 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.099387+0000", 2021-03-15T22:22:47.420 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0016555210000000001 2021-03-15T22:22:47.421 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.421 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.421 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.101042+0000", 2021-03-15T22:22:47.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013926629999999999 2021-03-15T22:22:47.422 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.422 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.422 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.422 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.102435+0000", 2021-03-15T22:22:47.423 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002038644 2021-03-15T22:22:47.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.423 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.424 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.104474+0000", 2021-03-15T22:22:47.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6980000000000001e-05 2021-03-15T22:22:47.424 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.425 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.104521+0000", 2021-03-15T22:22:47.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6105000000000004e-05 2021-03-15T22:22:47.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.425 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.425 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.426 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:54.104597+0000", 2021-03-15T22:22:47.426 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0077367729999999997 2021-03-15T22:22:47.426 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.426 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.427 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.427 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.427 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.427 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:873 7.0 7:0cc37e3d:::smithi07694878-21:head [write 1584925~506039] snapc 35=[34,30] ondisk+write+known_if_redirected e152)", 2021-03-15T22:22:47.427 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:07.140779+0000", 2021-03-15T22:22:47.428 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 100.11420018699999, 2021-03-15T22:22:47.428 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0080121819999999996, 2021-03-15T22:22:47.428 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.428 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.429 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.429 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.429 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.429 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 873 2021-03-15T22:22:47.429 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.430 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.140779+0000", 2021-03-15T22:22:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.431 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.140779+0000", 2021-03-15T22:22:47.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00045175199999999998 2021-03-15T22:22:47.432 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.432 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.432 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141231+0000", 2021-03-15T22:22:47.433 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5019999999999999e-06 2021-03-15T22:22:47.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.433 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.433 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.434 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141233+0000", 2021-03-15T22:22:47.434 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1036000000000001e-05 2021-03-15T22:22:47.434 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.434 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.434 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.435 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141244+0000", 2021-03-15T22:22:47.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7910000000000004e-06 2021-03-15T22:22:47.435 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.436 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141249+0000", 2021-03-15T22:22:47.436 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9688000000000003e-05 2021-03-15T22:22:47.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.437 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.437 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141299+0000", 2021-03-15T22:22:47.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7974999999999999e-05 2021-03-15T22:22:47.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.437 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.438 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.438 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141367+0000", 2021-03-15T22:22:47.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000106593 2021-03-15T22:22:47.438 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:22:47.439 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141473+0000", 2021-03-15T22:22:47.439 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013605780000000001 2021-03-15T22:22:47.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.440 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.440 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142834+0000", 2021-03-15T22:22:47.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028287149999999999 2021-03-15T22:22:47.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.441 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.441 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.145662+0000", 2021-03-15T22:22:47.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003004652 2021-03-15T22:22:47.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.442 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148667+0000", 2021-03-15T22:22:47.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5068999999999997e-05 2021-03-15T22:22:47.443 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.443 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148732+0000", 2021-03-15T22:22:47.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8831e-05 2021-03-15T22:22:47.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148791+0000", 2021-03-15T22:22:47.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0080121819999999996 2021-03-15T22:22:47.445 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.445 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.445 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.446 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.446 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.446 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:874 7.0 7:0cc37e3d:::smithi07694878-21:head [setxattr _header (57) in=7b,truncate 2403401] snapc 35=[34,30] ondisk+write+known_if_redirected e152)", 2021-03-15T22:22:47.447 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:07.141260+0000", 2021-03-15T22:22:47.447 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 100.113719252, 2021-03-15T22:22:47.447 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076387740000000001, 2021-03-15T22:22:47.447 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.447 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.448 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.448 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.448 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.448 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 874 2021-03-15T22:22:47.448 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.449 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.449 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.449 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.449 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141260+0000", 2021-03-15T22:22:47.450 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.450 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.450 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141260+0000", 2021-03-15T22:22:47.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.201e-06 2021-03-15T22:22:47.451 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.451 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.452 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141261+0000", 2021-03-15T22:22:47.452 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9399999999999993e-07 2021-03-15T22:22:47.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141262+0000", 2021-03-15T22:22:47.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9730000000000003e-06 2021-03-15T22:22:47.453 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.454 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.454 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141271+0000", 2021-03-15T22:22:47.454 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6380000000000002e-06 2021-03-15T22:22:47.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.455 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.455 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141275+0000", 2021-03-15T22:22:47.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001038693 2021-03-15T22:22:47.455 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.456 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142313+0000", 2021-03-15T22:22:47.456 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6582999999999996e-05 2021-03-15T22:22:47.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.457 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142380+0000", 2021-03-15T22:22:47.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000198116 2021-03-15T22:22:47.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:22:47.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142578+0000", 2021-03-15T22:22:47.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000434752 2021-03-15T22:22:47.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.459 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.459 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.143013+0000", 2021-03-15T22:22:47.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027017529999999999 2021-03-15T22:22:47.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.460 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.145714+0000", 2021-03-15T22:22:47.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031088420000000001 2021-03-15T22:22:47.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.461 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.461 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.461 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148823+0000", 2021-03-15T22:22:47.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3077999999999997e-05 2021-03-15T22:22:47.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.462 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.462 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148856+0000", 2021-03-15T22:22:47.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2150999999999997e-05 2021-03-15T22:22:47.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.463 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.463 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.464 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148899+0000", 2021-03-15T22:22:47.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076387740000000001 2021-03-15T22:22:47.464 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.464 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.464 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.465 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:875 7.0 7:0cc37e3d:::smithi07694878-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e152)", 2021-03-15T22:22:47.465 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:07.141294+0000", 2021-03-15T22:22:47.465 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 100.113684912, 2021-03-15T22:22:47.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0079780700000000003, 2021-03-15T22:22:47.466 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.466 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:22:47.466 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.467 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.467 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.467 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 875 2021-03-15T22:22:47.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.468 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.468 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.468 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141294+0000", 2021-03-15T22:22:47.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.469 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.469 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141294+0000", 2021-03-15T22:22:47.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6599999999999996e-07 2021-03-15T22:22:47.470 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.470 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.470 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141295+0000", 2021-03-15T22:22:47.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1800000000000005e-07 2021-03-15T22:22:47.471 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141295+0000", 2021-03-15T22:22:47.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7010000000000004e-06 2021-03-15T22:22:47.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.473 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.473 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141300+0000", 2021-03-15T22:22:47.473 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6419999999999999e-06 2021-03-15T22:22:47.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.474 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.474 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.141303+0000", 2021-03-15T22:22:47.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015483140000000001 2021-03-15T22:22:47.474 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142851+0000", 2021-03-15T22:22:47.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8303000000000001e-05 2021-03-15T22:22:47.475 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:47.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.142909+0000", 2021-03-15T22:22:47.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060387089999999997 2021-03-15T22:22:47.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.477 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.477 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.477 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.148948+0000", 2021-03-15T22:22:47.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7832999999999997e-05 2021-03-15T22:22:47.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.478 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.149046+0000", 2021-03-15T22:22:47.478 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022618399999999999 2021-03-15T22:22:47.479 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.479 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.149272+0000", 2021-03-15T22:22:47.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0079780700000000003 2021-03-15T22:22:47.480 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.480 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.480 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.480 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.481 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:1191 7.6 e176/175 7:61e0cc05:::smithi07694878-16:head v 176'225, mlcod=176'221)", 2021-03-15T22:22:47.481 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:31.032483+0000", 2021-03-15T22:22:47.481 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 76.222495746000007, 2021-03-15T22:22:47.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013234634, 2021-03-15T22:22:47.482 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.482 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.482 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.482 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.483 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.483 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.032483+0000", 2021-03-15T22:22:47.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.484 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.484 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.032483+0000", 2021-03-15T22:22:47.484 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054095030000000004 2021-03-15T22:22:47.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.485 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.037893+0000", 2021-03-15T22:22:47.485 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.989e-06 2021-03-15T22:22:47.485 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.486 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.486 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.037898+0000", 2021-03-15T22:22:47.486 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2257999999999999e-05 2021-03-15T22:22:47.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.487 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.487 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.487 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.037920+0000", 2021-03-15T22:22:47.487 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2199999999999992e-06 2021-03-15T22:22:47.488 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.488 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.488 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.488 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.037928+0000", 2021-03-15T22:22:47.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8709999999999998e-05 2021-03-15T22:22:47.489 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.489 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.489 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.489 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.037997+0000", 2021-03-15T22:22:47.489 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2061000000000003e-05 2021-03-15T22:22:47.490 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.490 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.490 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.490 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038039+0000", 2021-03-15T22:22:47.491 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076352579999999998 2021-03-15T22:22:47.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.491 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.491 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.491 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.045674+0000", 2021-03-15T22:22:47.492 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3634999999999998e-05 2021-03-15T22:22:47.492 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.492 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.492 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.493 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.045718+0000", 2021-03-15T22:22:47.493 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013234634 2021-03-15T22:22:47.493 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.493 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.493 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.494 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.494 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:1555 7.6 e204/175 7:6a64397b:::smithi07694878-49:head v 204'276, mlcod=203'272)", 2021-03-15T22:22:47.494 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:59.040478+0000", 2021-03-15T22:22:47.495 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 48.214500889, 2021-03-15T22:22:47.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078201050000000008, 2021-03-15T22:22:47.495 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.495 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.495 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.496 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.496 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.040478+0000", 2021-03-15T22:22:47.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.497 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.497 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.040478+0000", 2021-03-15T22:22:47.497 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045571819999999999 2021-03-15T22:22:47.498 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.045035+0000", 2021-03-15T22:22:47.499 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7899999999999999e-06 2021-03-15T22:22:47.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.499 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.499 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.045040+0000", 2021-03-15T22:22:47.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8082000000000001e-05 2021-03-15T22:22:47.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.500 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.045058+0000", 2021-03-15T22:22:47.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0720000000000002e-06 2021-03-15T22:22:47.501 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.501 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.502 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.045065+0000", 2021-03-15T22:22:47.502 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8667e-05 2021-03-15T22:22:47.502 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.503 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.045114+0000", 2021-03-15T22:22:47.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9737000000000001e-05 2021-03-15T22:22:47.503 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.503 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.504 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.504 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.045134+0000", 2021-03-15T22:22:47.504 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031298020000000001 2021-03-15T22:22:47.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.505 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.505 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.505 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.048263+0000", 2021-03-15T22:22:47.505 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4773000000000001e-05 2021-03-15T22:22:47.506 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.506 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.506 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.506 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.048298+0000", 2021-03-15T22:22:47.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078201050000000008 2021-03-15T22:22:47.507 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.507 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.507 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.508 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1589 7.0 7:00c03349:::smithi07694878-40:head [write 766452~565707,stat] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:22:47.508 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.044128+0000", 2021-03-15T22:22:47.509 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 46.210850565999998, 2021-03-15T22:22:47.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.002067477, 2021-03-15T22:22:47.509 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.509 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.510 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.510 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.510 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.510 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1589 2021-03-15T22:22:47.510 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.511 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.511 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.511 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.511 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044128+0000", 2021-03-15T22:22:47.511 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.512 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.512 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044128+0000", 2021-03-15T22:22:47.513 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00075323800000000004 2021-03-15T22:22:47.513 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.513 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.513 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.513 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044882+0000", 2021-03-15T22:22:47.514 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3730000000000001e-06 2021-03-15T22:22:47.514 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.514 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.514 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.515 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044885+0000", 2021-03-15T22:22:47.515 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3543999999999999e-05 2021-03-15T22:22:47.515 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.515 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.515 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.516 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044899+0000", 2021-03-15T22:22:47.516 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2700000000000001e-06 2021-03-15T22:22:47.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.516 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.517 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044905+0000", 2021-03-15T22:22:47.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1384000000000001e-05 2021-03-15T22:22:47.517 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.517 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.518 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044956+0000", 2021-03-15T22:22:47.518 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1709999999999996e-06 2021-03-15T22:22:47.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.518 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:22:47.519 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044963+0000", 2021-03-15T22:22:47.519 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99623219900000004 2021-03-15T22:22:47.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.520 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.041196+0000", 2021-03-15T22:22:47.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000118097 2021-03-15T22:22:47.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.521 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.041314+0000", 2021-03-15T22:22:47.521 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018693400000000001 2021-03-15T22:22:47.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:22:47.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.041501+0000", 2021-03-15T22:22:47.522 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029280589999999998 2021-03-15T22:22:47.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.044429+0000", 2021-03-15T22:22:47.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000135282 2021-03-15T22:22:47.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.524 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.524 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.044564+0000", 2021-03-15T22:22:47.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001471113 2021-03-15T22:22:47.525 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.525 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.526 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046035+0000", 2021-03-15T22:22:47.526 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9391999999999997e-05 2021-03-15T22:22:47.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.526 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.527 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046085+0000", 2021-03-15T22:22:47.527 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111421 2021-03-15T22:22:47.527 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.527 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.527 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.528 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046196+0000", 2021-03-15T22:22:47.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.002067477 2021-03-15T22:22:47.528 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.528 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.528 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.529 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.529 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.529 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1590 7.0 7:00c03349:::smithi07694878-40:head [write 1860221~469412] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:22:47.529 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.049091+0000", 2021-03-15T22:22:47.530 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 46.205888090000002, 2021-03-15T22:22:47.530 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0004167690000001, 2021-03-15T22:22:47.530 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.530 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.530 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.531 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.531 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.531 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1590 2021-03-15T22:22:47.531 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.531 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.532 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.532 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.532 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049091+0000", 2021-03-15T22:22:47.532 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.532 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.533 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.533 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049091+0000", 2021-03-15T22:22:47.533 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036754099999999998 2021-03-15T22:22:47.534 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.534 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.534 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.534 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049458+0000", 2021-03-15T22:22:47.534 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2369999999999999e-06 2021-03-15T22:22:47.535 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.535 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.535 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.535 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049461+0000", 2021-03-15T22:22:47.536 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3468e-05 2021-03-15T22:22:47.536 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.536 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.536 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.536 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049474+0000", 2021-03-15T22:22:47.537 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7660000000000001e-06 2021-03-15T22:22:47.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.537 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.537 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.538 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049481+0000", 2021-03-15T22:22:47.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0846000000000006e-05 2021-03-15T22:22:47.538 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.538 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049552+0000", 2021-03-15T22:22:47.539 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4910000000000005e-06 2021-03-15T22:22:47.539 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.539 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.539 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:22:47.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049560+0000", 2021-03-15T22:22:47.540 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99323641500000004 2021-03-15T22:22:47.540 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.540 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.541 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.042797+0000", 2021-03-15T22:22:47.541 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4145000000000002e-05 2021-03-15T22:22:47.541 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.541 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.542 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:47.542 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.042881+0000", 2021-03-15T22:22:47.542 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033443700000000002 2021-03-15T22:22:47.542 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.543 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.543 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.543 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046225+0000", 2021-03-15T22:22:47.543 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1945000000000004e-05 2021-03-15T22:22:47.543 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.544 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.544 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.544 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046287+0000", 2021-03-15T22:22:47.544 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013465800000000001 2021-03-15T22:22:47.544 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.545 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:22:47.545 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046422+0000", 2021-03-15T22:22:47.545 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0025218760000000002 2021-03-15T22:22:47.546 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.546 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.546 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.546 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048944+0000", 2021-03-15T22:22:47.546 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036462099999999998 2021-03-15T22:22:47.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.547 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.547 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049308+0000", 2021-03-15T22:22:47.548 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013490400000000001 2021-03-15T22:22:47.548 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.548 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.548 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.548 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049443+0000", 2021-03-15T22:22:47.549 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4076999999999999e-05 2021-03-15T22:22:47.549 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.549 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.549 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049477+0000", 2021-03-15T22:22:47.550 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0409e-05 2021-03-15T22:22:47.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.551 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049508+0000", 2021-03-15T22:22:47.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0004167690000001 2021-03-15T22:22:47.551 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.551 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.552 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.552 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.552 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.552 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1591 7.0 7:00c03349:::smithi07694878-40:head [write 2981238~594195] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:22:47.552 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.056067+0000", 2021-03-15T22:22:47.553 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 46.198912382000003, 2021-03-15T22:22:47.553 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99404281400000005, 2021-03-15T22:22:47.553 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.553 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.554 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.554 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.554 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.554 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1591 2021-03-15T22:22:47.554 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.555 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.555 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.555 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.555 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056067+0000", 2021-03-15T22:22:47.555 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.556 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056067+0000", 2021-03-15T22:22:47.557 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00055381400000000002 2021-03-15T22:22:47.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.557 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.557 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056620+0000", 2021-03-15T22:22:47.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5569999999999998e-06 2021-03-15T22:22:47.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.558 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056623+0000", 2021-03-15T22:22:47.559 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3344e-05 2021-03-15T22:22:47.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.560 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056636+0000", 2021-03-15T22:22:47.560 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2089999999999996e-06 2021-03-15T22:22:47.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.560 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.561 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056642+0000", 2021-03-15T22:22:47.561 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4657000000000003e-05 2021-03-15T22:22:47.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.562 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.562 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056686+0000", 2021-03-15T22:22:47.562 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0719999999999996e-06 2021-03-15T22:22:47.562 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.563 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:22:47.563 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056692+0000", 2021-03-15T22:22:47.563 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98623406700000005 2021-03-15T22:22:47.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.564 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.564 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.564 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.042926+0000", 2021-03-15T22:22:47.564 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101845 2021-03-15T22:22:47.564 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.565 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:47.565 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043028+0000", 2021-03-15T22:22:47.565 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042733099999999998 2021-03-15T22:22:47.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.566 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.566 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.566 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.047302+0000", 2021-03-15T22:22:47.566 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8188000000000004e-05 2021-03-15T22:22:47.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.567 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.567 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.047380+0000", 2021-03-15T22:22:47.567 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013972 2021-03-15T22:22:47.568 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.568 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.568 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:22:47.568 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.047519+0000", 2021-03-15T22:22:47.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020016600000000002 2021-03-15T22:22:47.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.569 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.569 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.569 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049521+0000", 2021-03-15T22:22:47.569 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034165299999999998 2021-03-15T22:22:47.570 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.570 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.570 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049863+0000", 2021-03-15T22:22:47.571 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015579400000000001 2021-03-15T22:22:47.571 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.571 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.571 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.571 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050019+0000", 2021-03-15T22:22:47.572 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1232e-05 2021-03-15T22:22:47.572 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.572 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.572 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.572 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050070+0000", 2021-03-15T22:22:47.573 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9691999999999999e-05 2021-03-15T22:22:47.573 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.573 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.573 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.573 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050109+0000", 2021-03-15T22:22:47.574 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99404281400000005 2021-03-15T22:22:47.574 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.574 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.574 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.575 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.575 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1592 7.0 7:00c03349:::smithi07694878-40:head [setxattr _header (57) in=7b,truncate 3966452] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:22:47.575 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.056653+0000", 2021-03-15T22:22:47.575 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 46.198325697999998, 2021-03-15T22:22:47.576 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99360381200000003, 2021-03-15T22:22:47.576 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.576 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.576 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.576 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.577 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.577 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1592 2021-03-15T22:22:47.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.577 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.578 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056653+0000", 2021-03-15T22:22:47.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.579 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.579 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056653+0000", 2021-03-15T22:22:47.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1789999999999998e-06 2021-03-15T22:22:47.579 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.580 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.580 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.580 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056655+0000", 2021-03-15T22:22:47.580 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.689e-06 2021-03-15T22:22:47.580 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.581 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.581 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.581 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056657+0000", 2021-03-15T22:22:47.581 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5310000000000001e-06 2021-03-15T22:22:47.581 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.582 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.582 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056663+0000", 2021-03-15T22:22:47.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3579999999999996e-06 2021-03-15T22:22:47.583 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.583 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.583 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.583 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056667+0000", 2021-03-15T22:22:47.583 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8161000000000002e-05 2021-03-15T22:22:47.584 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.584 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.584 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.584 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056715+0000", 2021-03-15T22:22:47.584 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2899999999999999e-06 2021-03-15T22:22:47.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.585 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:22:47.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056722+0000", 2021-03-15T22:22:47.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98635435299999996 2021-03-15T22:22:47.586 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.586 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043076+0000", 2021-03-15T22:22:47.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100571 2021-03-15T22:22:47.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:47.587 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043176+0000", 2021-03-15T22:22:47.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0051726330000000003 2021-03-15T22:22:47.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.588 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048349+0000", 2021-03-15T22:22:47.589 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010839 2021-03-15T22:22:47.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.589 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.589 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.590 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048457+0000", 2021-03-15T22:22:47.590 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0001582 2021-03-15T22:22:47.590 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.590 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:22:47.591 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048616+0000", 2021-03-15T22:22:47.591 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00090888500000000001 2021-03-15T22:22:47.591 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.591 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.592 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.592 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049525+0000", 2021-03-15T22:22:47.592 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00045732299999999999 2021-03-15T22:22:47.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.592 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.593 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.593 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049982+0000", 2021-03-15T22:22:47.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016349299999999999 2021-03-15T22:22:47.593 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.593 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.594 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050145+0000", 2021-03-15T22:22:47.594 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9777e-05 2021-03-15T22:22:47.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050175+0000", 2021-03-15T22:22:47.595 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1978999999999996e-05 2021-03-15T22:22:47.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.596 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.596 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050257+0000", 2021-03-15T22:22:47.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99360381200000003 2021-03-15T22:22:47.597 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.597 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.597 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.597 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.598 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1593 7.0 7:00c03349:::smithi07694878-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e206)", 2021-03-15T22:22:47.598 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.056680+0000", 2021-03-15T22:22:47.598 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 46.198299339999998, 2021-03-15T22:22:47.598 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99375757600000003, 2021-03-15T22:22:47.598 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.599 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:22:47.599 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.599 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.599 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.600 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1593 2021-03-15T22:22:47.600 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.600 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.600 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.601 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056680+0000", 2021-03-15T22:22:47.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.601 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.601 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.602 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056680+0000", 2021-03-15T22:22:47.602 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9000000000000005e-07 2021-03-15T22:22:47.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.602 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.603 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.603 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056681+0000", 2021-03-15T22:22:47.603 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.243e-06 2021-03-15T22:22:47.603 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.604 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056682+0000", 2021-03-15T22:22:47.604 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5610000000000004e-06 2021-03-15T22:22:47.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.604 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.605 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.605 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056688+0000", 2021-03-15T22:22:47.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4100000000000001e-06 2021-03-15T22:22:47.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056693+0000", 2021-03-15T22:22:47.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.562e-05 2021-03-15T22:22:47.606 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.607 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.607 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.607 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056738+0000", 2021-03-15T22:22:47.607 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6179999999999999e-06 2021-03-15T22:22:47.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.608 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:22:47.608 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056742+0000", 2021-03-15T22:22:47.608 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98647171499999997 2021-03-15T22:22:47.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.609 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.609 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.609 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043214+0000", 2021-03-15T22:22:47.609 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9062999999999998e-05 2021-03-15T22:22:47.609 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.610 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.610 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:47.610 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043303+0000", 2021-03-15T22:22:47.610 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055719380000000002 2021-03-15T22:22:47.611 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.611 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.611 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.611 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048875+0000", 2021-03-15T22:22:47.611 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6421000000000001e-05 2021-03-15T22:22:47.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.612 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.612 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:22:47.612 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048931+0000", 2021-03-15T22:22:47.613 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013764700000000001 2021-03-15T22:22:47.613 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.613 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.613 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.613 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050308+0000", 2021-03-15T22:22:47.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6387000000000001e-05 2021-03-15T22:22:47.614 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.614 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.615 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050364+0000", 2021-03-15T22:22:47.615 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3139999999999994e-05 2021-03-15T22:22:47.615 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.615 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.615 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.616 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050437+0000", 2021-03-15T22:22:47.616 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99375757600000003 2021-03-15T22:22:47.616 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.616 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.617 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.617 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1804 7.0 7:054fa7db:::smithi07694878-29:head [append 0~0] snapc 89=[88,87,86,84,81,80,7d] ondisk+write+known_if_redirected e236)", 2021-03-15T22:22:47.617 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:31.076303+0000", 2021-03-15T22:22:47.618 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 16.178675609999999, 2021-03-15T22:22:47.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078352590000000007, 2021-03-15T22:22:47.618 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:22:47.618 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:22:47.618 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:22:47.619 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:22:47.619 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:22:47.619 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1804 2021-03-15T22:22:47.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.620 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:22:47.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.620 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:22:47.620 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.076303+0000", 2021-03-15T22:22:47.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:22:47.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:22:47.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.076303+0000", 2021-03-15T22:22:47.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8700000000000005e-07 2021-03-15T22:22:47.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:22:47.622 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.076304+0000", 2021-03-15T22:22:47.622 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3780000000000007e-06 2021-03-15T22:22:47.623 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.623 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.623 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:22:47.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.076313+0000", 2021-03-15T22:22:47.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6480000000000001e-06 2021-03-15T22:22:47.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:22:47.624 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.076317+0000", 2021-03-15T22:22:47.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7140000000000002e-06 2021-03-15T22:22:47.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:22:47.626 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.076320+0000", 2021-03-15T22:22:47.626 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0036356880000000002 2021-03-15T22:22:47.626 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.626 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.626 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:22:47.627 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.079956+0000", 2021-03-15T22:22:47.627 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010786200000000001 2021-03-15T22:22:47.627 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.627 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.627 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:22:47.628 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.080063+0000", 2021-03-15T22:22:47.628 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016608200000000001 2021-03-15T22:22:47.628 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.628 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.629 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:22:47.629 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.080229+0000", 2021-03-15T22:22:47.629 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00091307300000000003 2021-03-15T22:22:47.629 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.630 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:22:47.630 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.081143+0000", 2021-03-15T22:22:47.630 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9128999999999998e-05 2021-03-15T22:22:47.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.631 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.631 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.081172+0000", 2021-03-15T22:22:47.631 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029116070000000001 2021-03-15T22:22:47.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.632 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:22:47.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.084083+0000", 2021-03-15T22:22:47.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7747999999999999e-05 2021-03-15T22:22:47.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:22:47.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.084111+0000", 2021-03-15T22:22:47.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7642999999999999e-05 2021-03-15T22:22:47.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:22:47.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:22:47.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:22:47.634 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:31.084139+0000", 2021-03-15T22:22:47.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078352590000000007 2021-03-15T22:22:47.635 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.635 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.635 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.635 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:22:47.636 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:22:47.636 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:22:47.636 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:47.637 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:47.638 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:47.639 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:22:46.260766+0000 mgr.y (mgr.14144) 2021-03-15T22:22:47.639 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: 484 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:22:47.639 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: audit 2021-03-15T22:22:46.264345+0000 mon.a (mon.0) 919 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:22:47.640 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:22:46.265301+0000 mgr.y (mgr.14144) 485 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:22:47.640 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:22:47.640 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.640 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:22:47.641 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.641 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:22:47.641 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.641 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.641 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:22:47.642 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:22:46.265716+0000 mgr.y (mgr.14144) 486 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:22:47.642 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:22:47.642 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.642 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:22:47.643 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.643 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:22:47.643 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.643 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.643 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:22:47.644 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:22:46.266102+0000 mgr.y (mgr.14144) 487 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:22:47.644 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:22:47.644 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.644 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:22:47.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:22:47.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:22:47.646 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:22:46.266474+0000 mgr.y (mgr 2021-03-15T22:22:47.646 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: .14144) 488 : 2021-03-15T22:22:47.646 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:22:47.646 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:22:47.647 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.647 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:22:47.647 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.647 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:22:47.647 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.648 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.648 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:22:47.648 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:22:46.266967+0000 mgr.y (mgr.14144) 489 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:22:47.648 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:22:47.648 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:22:47.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:22:47.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:22:47.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: cephadm 2021-03-15T 2021-03-15T22:22:47.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: 22:22:46.267509+0000 mgr.y (mgr.14144) 490 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:22:47.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:22:47.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:22:47.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:22:47.652 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.652 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.652 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:22:47.652 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:22:46.267941 2021-03-15T22:22:47.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 491 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:22:47.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:22:47.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:22:47.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.654 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:22:47.654 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.654 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.654 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:22:47.655 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: cluster 2021-03-15T22:22:47.213741+0000 mon.a (mon.0) 2021-03-15T22:22:47.655 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:47 smithi125 conmon[56739]: 920 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2021-03-15T22:22:47.663 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:47.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:47.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:47.697 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:22:46.260766+0000 mgr.y (mgr. 2021-03-15T22:22:47.697 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: 14144) 484 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:22:47.697 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: audit 2021-03-15T22:22:46.264345+0000 mon.a (mon.0) 919 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:22:47.697 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:22:46.265301+0000 mgr.y (mgr.14144) 485 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:22:47.698 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:22:47.698 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.698 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:22:47.698 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:22:47.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:22:47.700 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:22:46.265716+0000 mgr.y (mgr.14144) 486 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:22:47.700 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:22:47.700 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.700 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:22:47.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:22:47.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:22:47.702 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:22:46.266102+0000 mgr.y (mgr.14144) 487 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:22:47.702 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:22:47.702 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.702 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:22:47.703 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.703 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:22:47.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:22:46 2021-03-15T22:22:47.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: .260766+0000 mgr.y (mgr.14144) 484 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:22:47.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: audit 2021-03-15T22:22:47.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: 2021-03-15T22:22:46.264345+0000 mon.a ( 2021-03-15T22:22:47.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: mon.0) 919 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:22:47.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:22:47.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: -03-15T22:22:46. 2021-03-15T22:22:47.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: 265301+0000 mgr.y (mgr.14144) 485 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:22:47.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:22:47.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:22:47.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:22:47.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:22:47.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:22:46.265716+0000 mgr.y (mgr.14144) 486 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:22:47.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:22:47.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:22:47.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:22:47.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:22:47.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: cephadm 2021-03-15 2021-03-15T22:22:47.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: T22:22:46.266102+0000 mgr.y (mgr.14144) 487 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:22:47.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:22:47.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:22:47.711 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.711 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.712 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:22:47.712 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:22:46.266474+0000 mgr.y (mgr.14144) 488 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:22:47.712 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:22:47.712 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:22:47.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:22:47.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.714 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:22:47.714 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:22:46.266967+0000 mgr.y (mgr.14144) 489 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:22:47.714 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:22:47.714 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.715 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:22:47.715 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.715 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:22:47.715 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.716 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.716 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:22:47.716 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:22:46.267509+0000 mgr.y (mgr.14144) 490 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:22:47.716 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:22:47.716 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:22:47.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:22:47.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:22:47.718 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:22:47.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:22:47.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:22:46.266474+0000 mgr.y (mgr.14144) 488 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:22:47.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:22:47.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:22:47.721 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.721 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:22:47.721 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.721 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.721 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:22:47.722 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:22:47.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: 2021-03-15T22:22:46.266967+0000 mgr.y (mgr.14144) 489 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:22:47.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:22:47.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:22:47.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:22:47.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:22:47.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:22:46.267509+0000 mgr.y (mgr.14144) 490 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:22:47.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:22:47.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:22:47.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:22:47.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:22:47.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:22:46.267941+0000 mgr.y (mgr.14144) 491 : cephadm 2021-03-15T22:22:47.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:22:47.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:22:47.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:22:47.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:22:47.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:22:47.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[62156]: cluster 2021-03-15T22:22:47.213741+0000 mon.a (mon.0) 920 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2021-03-15T22:22:47.730 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:22:46.267941+0000 mgr.y (mgr.14144) 491 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:22:47.730 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:22:47.730 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:22:47.730 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:22:47.731 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:22:47.731 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:22:47.731 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:22:47.731 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:22:47.732 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:22:47.732 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:47 smithi076 conmon[56514]: cluster 2021-03-15T22:22:47.213741+0000 mon.a (mon.0) 920 : cluster [DBG] osdmap e252: 8 total, 8 up, 7 in 2021-03-15T22:22:47.765 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:47.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:47.789 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:47.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:47.890 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:47.895 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-15T22:22:47.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:47.915 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:47.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:48.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:48.040 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:48.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:48.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:48.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:48.167 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:48.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:48.191 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:22:48.212 INFO:tasks.rados.rados.0.smithi076.stdout:802: done (7 left) 2021-03-15T22:22:48.213 INFO:tasks.rados.rados.0.smithi076.stdout:803: copy_from oid 11 from oid 21 current snap is 81 2021-03-15T22:22:48.213 INFO:tasks.rados.rados.0.smithi076.stdout:804: read oid 4 snap 72 2021-03-15T22:22:48.213 INFO:tasks.rados.rados.0.smithi076.stdout:804: expect (ObjNum 239 snap 57 seq_num 239) 2021-03-15T22:22:48.213 INFO:tasks.rados.rados.0.smithi076.stdout:805: snap_create 2021-03-15T22:22:48.214 INFO:tasks.rados.rados.0.smithi076.stdout:800: finishing copy_from to smithi07694878-36 2021-03-15T22:22:48.214 INFO:tasks.rados.rados.0.smithi076.stdout:800: got expected ENOENT (src dne) 2021-03-15T22:22:48.214 INFO:tasks.rados.rados.0.smithi076.stdout:800: finishing copy_from racing read to smithi07694878-36 2021-03-15T22:22:48.214 INFO:tasks.rados.rados.0.smithi076.stdout:795: expect (ObjNum 296 snap 78 seq_num 296) 2021-03-15T22:22:48.224 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:22:48.268 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:48.303 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:48.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:48.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:48.329 INFO:tasks.rados.rados.0.smithi076.stdout:799: finishing copy_from to smithi07694878-33 2021-03-15T22:22:48.329 INFO:tasks.rados.rados.0.smithi076.stdout:799: got expected ENOENT (src dne) 2021-03-15T22:22:48.329 INFO:tasks.rados.rados.0.smithi076.stdout:799: finishing copy_from racing read to smithi07694878-33 2021-03-15T22:22:48.330 INFO:tasks.rados.rados.0.smithi076.stdout:797: expect (ObjNum 291 snap 78 seq_num 291) 2021-03-15T22:22:48.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:48.427 INFO:tasks.rados.rados.0.smithi076.stdout:798: finishing rollback tid 0 to smithi07694878-3 2021-03-15T22:22:48.427 INFO:tasks.rados.rados.0.smithi076.stdout:798: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:22:48.427 INFO:tasks.rados.rados.0.smithi076.stdout:798: finishing rollback tid 2 to smithi07694878-3 2021-03-15T22:22:48.427 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 267 (ObjNum 199 snap 43 seq_num 199) dirty exists 2021-03-15T22:22:48.428 INFO:tasks.rados.rados.0.smithi076.stdout:801: finishing write tid 1 to smithi07694878-17 2021-03-15T22:22:48.428 INFO:tasks.rados.rados.0.smithi076.stdout:801: finishing write tid 2 to smithi07694878-17 2021-03-15T22:22:48.428 INFO:tasks.rados.rados.0.smithi076.stdout:801: finishing write tid 3 to smithi07694878-17 2021-03-15T22:22:48.428 INFO:tasks.rados.rados.0.smithi076.stdout:801: finishing write tid 4 to smithi07694878-17 2021-03-15T22:22:48.429 INFO:tasks.rados.rados.0.smithi076.stdout:801: finishing write tid 5 to smithi07694878-17 2021-03-15T22:22:48.429 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 64 (ObjNum 311 snap 81 seq_num 311) dirty exists 2021-03-15T22:22:48.429 INFO:tasks.rados.rados.0.smithi076.stdout:801: left oid 17 (ObjNum 311 snap 81 seq_num 311) 2021-03-15T22:22:48.429 INFO:tasks.rados.rados.0.smithi076.stdout:804: expect (ObjNum 239 snap 57 seq_num 239) 2021-03-15T22:22:48.431 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:48.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:48.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:48.509 INFO:tasks.rados.rados.0.smithi076.stdout:803: finishing copy_from to smithi07694878-11 2021-03-15T22:22:48.510 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 398 (ObjNum 263 snap 68 seq_num 263) dirty exists 2021-03-15T22:22:48.510 INFO:tasks.rados.rados.0.smithi076.stdout:803: finishing copy_from racing read to smithi07694878-11 2021-03-15T22:22:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[56514]: cluster 2021-03-15T22:22:47.014840+0000 mgr.y (mgr.14144) 492 : cluster [DBG] pgmap v504: 138 pgs: 138 active+clean; 162 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2021-03-15T22:22:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[56514]: audit 2021-03-15T22:22:48.119723+0000 mon.a (mon.0) 921 : audit [INF] from='client.? 172.21.15.76:0/3803980776' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[56514]: cluster 2021-03-15T22:22:48.206206+0000 mon.a (mon.0) 922 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[56514]: audit 2021-03-15T22:22:48.210201+0000 mon.a (mon.0) 923 : audit [INF] from='client.? 172.21.15.76:0/3803980776' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:22:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[56514]: cluster 2021-03-15T22:22:48.210254+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2021-03-15T22:22:48.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: cluster 2021-03-15T22:22:47.014840+0000 2021-03-15T22:22:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: mgr.y (mgr.14144) 492 : cluster [DBG] pgmap v504: 138 pgs: 138 active+clean; 162 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2021-03-15T22:22:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:22:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: 22:48.119723+0000 mon.a (mon.0) 921 : audit [INF] from='client.? 172.21.15.76:0/3803980776' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:22:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: T22:22:48.206206+0000 mon.a (mon 2021-03-15T22:22:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: .0) 922 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:22:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: 03-15T22:22:48.210201+0000 2021-03-15T22:22:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: mon.a (mon.0) 923 : audit [INF] 2021-03-15T22:22:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3803980776' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:22:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:22:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: 22:22:48.210254+0000 mon.a (mon 2021-03-15T22:22:48.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:48 smithi076 conmon[62156]: .0) 924 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2021-03-15T22:22:48.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:48.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:48 smithi125 conmon[56739]: cluster 2021-03-15T22:22:47.014840+0000 mgr.y (mgr.14144) 492 : cluster [DBG] pgmap v504: 138 pgs: 138 active+clean; 162 MiB data, 515 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.5 MiB/s wr, 27 op/s 2021-03-15T22:22:48.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:48 smithi125 conmon[56739]: audit 2021-03-15T22:22:48.119723+0000 mon.a (mon.0) 921 : audit [INF] from='client.? 172.21.15.76:0/3803980776' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:22:48.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:48 smithi125 conmon[56739]: cluster 2021-03-15T22:22:48.206206+0000 mon.a (mon.0) 922 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:48.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:48 smithi125 conmon[56739]: audit 2021-03-15T22:22:48.210201+0000 mon.a (mon.0) 923 : audit [INF] from='client.? 172.21.15.76:0/3803980776' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:22:48.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:48 smithi125 conmon[56739]: cluster 2021-03-15T22:22:48.210254+0000 mon.a (mon.0) 924 : cluster [DBG] osdmap e253: 8 total, 8 up, 7 in 2021-03-15T22:22:48.559 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:48.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:48.560 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:48.662 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:48.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:48.685 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:48.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:48.787 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:48.809 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:48.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:48.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:48.912 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:48.934 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:48.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:48.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:49.037 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:49.058 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:49.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:49.060 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:49.161 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:49.185 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:49.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:49.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:49.215 INFO:tasks.rados.rados.0.smithi076.stdout:795: done (9 left) 2021-03-15T22:22:49.216 INFO:tasks.rados.rados.0.smithi076.stdout:796: done (8 left) 2021-03-15T22:22:49.216 INFO:tasks.rados.rados.0.smithi076.stdout:797: done (7 left) 2021-03-15T22:22:49.216 INFO:tasks.rados.rados.0.smithi076.stdout:798: done (6 left) 2021-03-15T22:22:49.216 INFO:tasks.rados.rados.0.smithi076.stdout:799: done (5 left) 2021-03-15T22:22:49.216 INFO:tasks.rados.rados.0.smithi076.stdout:800: done (4 left) 2021-03-15T22:22:49.217 INFO:tasks.rados.rados.0.smithi076.stdout:801: done (3 left) 2021-03-15T22:22:49.217 INFO:tasks.rados.rados.0.smithi076.stdout:803: done (2 left) 2021-03-15T22:22:49.217 INFO:tasks.rados.rados.0.smithi076.stdout:804: done (1 left) 2021-03-15T22:22:49.217 INFO:tasks.rados.rados.0.smithi076.stdout:805: done (0 left) 2021-03-15T22:22:49.218 INFO:tasks.rados.rados.0.smithi076.stdout:806: read oid 38 snap -1 2021-03-15T22:22:49.218 INFO:tasks.rados.rados.0.smithi076.stdout:806: expect deleted 2021-03-15T22:22:49.218 INFO:tasks.rados.rados.0.smithi076.stdout:807: snap_remove snap 80 2021-03-15T22:22:49.288 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:49.311 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:49.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:49.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:49.414 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:49.436 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:49.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:49.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:49.539 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:49.560 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:49.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:49.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:49.663 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:49.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:49.686 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:49.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:49.764 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:22:49.765 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:22:49.765 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:22:49.765 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:22:49.787 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:49.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:49.811 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:49.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:49.816 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:22:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:22:49.912 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:49.934 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:49.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:49.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:50.036 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:50.046 INFO:tasks.rados.rados.0.smithi076.stdout:807: done (1 left) 2021-03-15T22:22:50.046 INFO:tasks.rados.rados.0.smithi076.stdout:808: read oid 24 snap -1 2021-03-15T22:22:50.047 INFO:tasks.rados.rados.0.smithi076.stdout:808: expect deleted 2021-03-15T22:22:50.047 INFO:tasks.rados.rados.0.smithi076.stdout:809: snap_create 2021-03-15T22:22:50.059 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:50.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:50.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:50.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:50.200 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:50.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:50.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:50.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:50.336 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:50.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:50.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:50.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-15T22:22:50.410 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:22:50.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:50 smithi076 conmon[56514]: cluster 2021-03-15T22:22:49.015200+0000 mgr.y (mgr.14144) 493 : cluster [DBG] pgmap v507: 138 pgs: 138 active+clean; 166 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 8.6 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2021-03-15T22:22:50.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:50 smithi076 conmon[56514]: cluster 2021-03-15T22:22:49.213789+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2021-03-15T22:22:50.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:50 smithi076 conmon[56514]: audit 2021-03-15T22:22:49.561518+0000 mgr.y (mgr.14144) 494 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:50.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:50 smithi076 conmon[56514]: cluster 2021-03-15T22:22:50.042768+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2021-03-15T22:22:50.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:50 smithi076 conmon[62156]: cluster 2021-03-15T22:22:49.015200+0000 mgr.y (mgr 2021-03-15T22:22:50.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:50 smithi076 conmon[62156]: .14144) 493 : cluster [DBG] pgmap v507: 138 pgs: 138 active+clean; 166 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 8.6 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2021-03-15T22:22:50.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:50 smithi076 conmon[62156]: cluster 2021-03-15T22:22:49.213789+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2021-03-15T22:22:50.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:50 smithi076 conmon[62156]: audit 2021-03-15T22:22:49.561518+0000 mgr.y (mgr.14144) 494 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:50.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:50 smithi076 conmon[62156]: cluster 2021-03-15T22:22:50.042768+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2021-03-15T22:22:50.430 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:22:50.439 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:50.450 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:22:50.468 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:22:50.485 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:50.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:50.487 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:22:50.503 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:22:50.521 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:22:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:50 smithi125 conmon[56739]: cluster 2021-03-15T22:22:49.015200+0000 mgr.y (mgr.14144 2021-03-15T22:22:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:50 smithi125 conmon[56739]: ) 493 : cluster [DBG] pgmap v507: 138 pgs: 138 active+clean; 166 MiB data, 528 MiB used, 625 GiB / 626 GiB avail; 8.6 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2021-03-15T22:22:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:50 smithi125 conmon[56739]: cluster 2021-03-15T22:22:49.213789+0000 mon.a (mon.0) 925 : cluster [DBG] osdmap e254: 8 total, 8 up, 7 in 2021-03-15T22:22:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:50 smithi125 conmon[56739]: audit 2021-03-15T22:22:49.561518+0000 mgr.y (mgr.14144) 494 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:22:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:50 smithi125 conmon[56739]: cluster 2021-03-15T22:22:50.042768+0000 mon.a (mon.0) 926 : cluster [DBG] osdmap e255: 8 total, 8 up, 7 in 2021-03-15T22:22:50.538 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:22:50.538 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:22:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:22:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:22:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:22:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:22:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:22:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:22:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:22:50.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:50.612 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:50.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:50.614 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:50.687 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:22:50.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:50.738 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:50.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:50.740 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:50.830 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:22:50.833 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:22:50.841 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:50.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:50.867 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:50.867 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:50.968 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:50.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:50.991 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:50.991 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:51.046 INFO:tasks.rados.rados.0.smithi076.stdout:806: done (2 left) 2021-03-15T22:22:51.046 INFO:tasks.rados.rados.0.smithi076.stdout:808: done (1 left) 2021-03-15T22:22:51.046 INFO:tasks.rados.rados.0.smithi076.stdout:809: done (0 left) 2021-03-15T22:22:51.046 INFO:tasks.rados.rados.0.smithi076.stdout:810: rollback oid 25 current snap is 83 2021-03-15T22:22:51.047 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 76 2021-03-15T22:22:51.047 INFO:tasks.rados.rados.0.smithi076.stdout:811: rollback oid 19 current snap is 83 2021-03-15T22:22:51.047 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 19 to 72 2021-03-15T22:22:51.047 INFO:tasks.rados.rados.0.smithi076.stdout:812: read oid 39 snap 76 2021-03-15T22:22:51.047 INFO:tasks.rados.rados.0.smithi076.stdout:812: expect (ObjNum 199 snap 43 seq_num 199) 2021-03-15T22:22:51.048 INFO:tasks.rados.rados.0.smithi076.stdout:813: rollback oid 37 current snap is 83 2021-03-15T22:22:51.048 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 37 to 63 2021-03-15T22:22:51.048 INFO:tasks.rados.rados.0.smithi076.stdout:814: read oid 16 snap 75 2021-03-15T22:22:51.048 INFO:tasks.rados.rados.0.smithi076.stdout:814: expect (ObjNum 277 snap 74 seq_num 277) 2021-03-15T22:22:51.049 INFO:tasks.rados.rados.0.smithi076.stdout:815: snap_remove snap 78 2021-03-15T22:22:51.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:51.116 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:51.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:51.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:51.219 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:51.240 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:51.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:51.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:51.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:51.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:51.368 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:51.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:51.470 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:51.492 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:51.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:51.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:51.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:51 smithi076 conmon[56514]: audit 2021-03-15T22:22:50.390527+0000 mon.a 2021-03-15T22:22:51.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:51 smithi076 conmon[56514]: (mon.0) 927 : audit [DBG] from='client.? 172.21.15.76:0/2807210735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:51.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:51 smithi076 conmon[56514]: audit 2021-03-15T22:22:50.804410+0000 mgr.y (mgr.14144) 495 : audit [DBG] from='client.15099 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:22:51.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:51 smithi076 conmon[56514]: cluster 2021-03-15T22:22:51.041427+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2021-03-15T22:22:51.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:51 smithi076 conmon[62156]: audit 2021-03-15T22:22:50.390527+0000 mon.a (mon.0) 927 2021-03-15T22:22:51.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:51 smithi076 conmon[62156]: : audit [DBG] from='client.? 172.21.15.76:0/2807210735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:51.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:51 smithi076 conmon[62156]: audit 2021-03-15T22:22:50.804410+0000 mgr.y (mgr 2021-03-15T22:22:51.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:51 smithi076 conmon[62156]: .14144) 495 : audit [DBG] from='client.15099 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:22:51.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:51 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:22:51.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:51 smithi076 conmon[62156]: :22:51.041427+0000 mon.a (mon. 2021-03-15T22:22:51.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:51 smithi076 conmon[62156]: 0) 928 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2021-03-15T22:22:51.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:51 smithi125 conmon[56739]: audit 2021-03-15T22:22:50.390527+0000 mon.a (mon.0) 927 : audit [DBG] from='client.? 172.21.15.76:0/2807210735' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:51.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:51 smithi125 conmon[56739]: audit 2021-03-15T22:22:50.804410+0000 mgr.y (mgr.14144) 495 : audit [DBG] from='client.15099 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:22:51.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:51 smithi125 conmon[56739]: cluster 2021-03-15T22:22:51.041427+0000 mon.a (mon.0) 928 : cluster [DBG] osdmap e256: 8 total, 8 up, 7 in 2021-03-15T22:22:51.595 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:51.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:51.619 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:51.620 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:51.721 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:51.743 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:51.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:51.744 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:51.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:51.853 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":509,"stamp":"2021-03-15T22:22:50.040246+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":177760111,"num_objects":405,"num_object_clones":47,"num_object_copies":1215,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":405,"num_whiteouts":9,"num_read":2968,"num_read_kb":1565755,"num_write":1764,"num_write_kb":448125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":342,"num_bytes_recovered":85238491,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3321,"ondisk_log_size":3321,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":545784,"kb_used_data":409920,"kb_used_omap":0,"kb_used_meta":135808,"kb_avail":655670280,"statfs":{"total":671965249536,"available":671406366720,"internally_reserved":0,"allocated":419758080,"data_stored":550217454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":139067392},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7798051,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":53,"num_read_kb":58091,"num_write":14,"num_write_kb":2773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.993797"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"207","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.039888+0000","last_change":"2021-03-15T22:21:06.404346+0000","last_active":"2021-03-15T22:22:45.039888+0000","last_peered":"2021-03-15T22:22:45.039888+0000","last_clean":"2021-03-15T22:22:45.039888+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:22:45.039888+0000","last_undegraded":"2021-03-15T22:22:45.039888+0000","last_fullsized":"2021-03-15T22:22:45.039888+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:06.404281+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:06.404281+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"214","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.199037+0000","last_change":"2021-03-15T22:21:22.794410+0000","last_active":"2021-03-15T22:22:44.199037+0000","last_peered":"2021-03-15T22:22:44.199037+0000","last_clean":"2021-03-15T22:22:44.199037+0000","last_became_active":"2021-03-15T22:17:54.284855+0000","last_became_peered":"2021-03-15T22:17:54.284855+0000","last_unstale":"2021-03-15T22:22:44.199037+0000","last_undegraded":"2021-03-15T22:22:44.199037+0000","last_fullsized":"2021-03-15T22:22:44.199037+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:21:22.794340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:22.794340+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"239","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.039830+0000","last_change":"2021-03-15T22:20:51.316325+0000","last_active":"2021-03-15T22:22:45.039830+0000","last_peered":"2021-03-15T22:22:45.039830+0000","last_clean":"2021-03-15T22:22:45.039830+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:22:45.039830+0000","last_undegraded":"2021-03-15T22:22:45.039830+0000","last_fullsized":"2021-03-15T22:22:45.039830+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:20:51.316237+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:20:51.316237+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"96","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375166+0000","last_change":"2021-03-15T22:21:49.846475+0000","last_active":"2021-03-15T22:22:47.375166+0000","last_peered":"2021-03-15T22:22:47.375166+0000","last_clean":"2021-03-15T22:22:47.375166+0000","last_became_active":"2021-03-15T22:21:28.039958+0000","last_became_peered":"2021-03-15T22:21:28.039958+0000","last_unstale":"2021-03-15T22:22:47.375166+0000","last_undegraded":"2021-03-15T22:22:47.375166+0000","last_fullsized":"2021-03-15T22:22:47.375166+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:49.846392+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:49.846392+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"93","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.672159+0000","last_change":"2021-03-15T22:22:32.631519+0000","last_active":"2021-03-15T22:22:44.672159+0000","last_peered":"2021-03-15T22:22:44.672159+0000","last_clean":"2021-03-15T22:22:44.672159+0000","last_became_active":"2021-03-15T22:21:28.043528+0000","last_became_peered":"2021-03-15T22:21:28.043528+0000","last_unstale":"2021-03-15T22:22:44.672159+0000","last_undegraded":"2021-03-15T22:22:44.672159+0000","last_fullsized":"2021-03-15T22:22:44.672159+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:32.631430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:32.631430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"171","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.208937+0000","last_change":"2021-03-15T22:21:30.473634+0000","last_active":"2021-03-15T22:22:48.208937+0000","last_peered":"2021-03-15T22:22:48.208937+0000","last_clean":"2021-03-15T22:22:48.208937+0000","last_became_active":"2021-03-15T22:21:28.049801+0000","last_became_peered":"2021-03-15T22:21:28.049801+0000","last_unstale":"2021-03-15T22:22:48.208937+0000","last_undegraded":"2021-03-15T22:22:48.208937+0000","last_fullsized":"2021-03-15T22:22:48.208937+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:21:30.473571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:30.473571+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":21,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"205","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.197752+0000","last_change":"2021-03-15T22:20:33.569323+0000","last_active":"2021-03-15T22:22:44.197752+0000","last_peered":"2021-03-15T22:22:44.197752+0000","last_clean":"2021-03-15T22:22:44.197752+0000","last_became_active":"2021-03-15T22:18:00.015776+0000","last_became_peered":"2021-03-15T22:18:00.015776+0000","last_unstale":"2021-03-15T22:22:44.197752+0000","last_undegraded":"2021-03-15T22:22:44.197752+0000","last_fullsized":"2021-03-15T22:22:44.197752+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:33.569266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:33.569266+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"216","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.048411+0000","last_change":"2021-03-15T22:22:40.167726+0000","last_active":"2021-03-15T22:22:46.048411+0000","last_peered":"2021-03-15T22:22:46.048411+0000","last_clean":"2021-03-15T22:22:46.048411+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:22:46.048411+0000","last_undegraded":"2021-03-15T22:22:46.048411+0000","last_fullsized":"2021-03-15T22:22:46.048411+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:40.167663+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:40.167663+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"262","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040031+0000","last_change":"2021-03-15T22:22:02.316373+0000","last_active":"2021-03-15T22:22:45.040031+0000","last_peered":"2021-03-15T22:22:45.040031+0000","last_clean":"2021-03-15T22:22:45.040031+0000","last_became_active":"2021-03-15T22:21:56.030245+0000","last_became_peered":"2021-03-15T22:21:56.030245+0000","last_unstale":"2021-03-15T22:22:45.040031+0000","last_undegraded":"2021-03-15T22:22:45.040031+0000","last_fullsized":"2021-03-15T22:22:45.040031+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:22:02.316263+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:02.316263+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,2],"acting":[4,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"214","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.047374+0000","last_change":"2021-03-15T22:21:53.179241+0000","last_active":"2021-03-15T22:22:46.047374+0000","last_peered":"2021-03-15T22:22:46.047374+0000","last_clean":"2021-03-15T22:22:46.047374+0000","last_became_active":"2021-03-15T22:21:28.045904+0000","last_became_peered":"2021-03-15T22:21:28.045904+0000","last_unstale":"2021-03-15T22:22:46.047374+0000","last_undegraded":"2021-03-15T22:22:46.047374+0000","last_fullsized":"2021-03-15T22:22:46.047374+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:53.179151+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:53.179151+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"216","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.047909+0000","last_change":"2021-03-15T22:22:41.168947+0000","last_active":"2021-03-15T22:22:46.047909+0000","last_peered":"2021-03-15T22:22:46.047909+0000","last_clean":"2021-03-15T22:22:46.047909+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:22:46.047909+0000","last_undegraded":"2021-03-15T22:22:46.047909+0000","last_fullsized":"2021-03-15T22:22:46.047909+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:41.168884+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:41.168884+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"121'9","reported_seq":"265","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040498+0000","last_change":"2021-03-15T22:22:31.872540+0000","last_active":"2021-03-15T22:22:45.040498+0000","last_peered":"2021-03-15T22:22:45.040498+0000","last_clean":"2021-03-15T22:22:45.040498+0000","last_became_active":"2021-03-15T22:21:56.035207+0000","last_became_peered":"2021-03-15T22:21:56.035207+0000","last_unstale":"2021-03-15T22:22:45.040498+0000","last_undegraded":"2021-03-15T22:22:45.040498+0000","last_fullsized":"2021-03-15T22:22:45.040498+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'9","last_scrub_stamp":"2021-03-15T22:22:31.872458+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:22:31.872458+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"207","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040544+0000","last_change":"2021-03-15T22:22:08.294925+0000","last_active":"2021-03-15T22:22:45.040544+0000","last_peered":"2021-03-15T22:22:45.040544+0000","last_clean":"2021-03-15T22:22:45.040544+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:22:45.040544+0000","last_undegraded":"2021-03-15T22:22:45.040544+0000","last_fullsized":"2021-03-15T22:22:45.040544+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:08.294855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:08.294855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"214","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198409+0000","last_change":"2021-03-15T22:21:25.786047+0000","last_active":"2021-03-15T22:22:44.198409+0000","last_peered":"2021-03-15T22:22:44.198409+0000","last_clean":"2021-03-15T22:22:44.198409+0000","last_became_active":"2021-03-15T22:17:53.976875+0000","last_became_peered":"2021-03-15T22:17:53.976875+0000","last_unstale":"2021-03-15T22:22:44.198409+0000","last_undegraded":"2021-03-15T22:22:44.198409+0000","last_fullsized":"2021-03-15T22:22:44.198409+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:21:25.785981+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:21:25.785981+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"263","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.209091+0000","last_change":"2021-03-15T22:21:29.556475+0000","last_active":"2021-03-15T22:22:48.209091+0000","last_peered":"2021-03-15T22:22:48.209091+0000","last_clean":"2021-03-15T22:22:48.209091+0000","last_became_active":"2021-03-15T22:21:28.049386+0000","last_became_peered":"2021-03-15T22:21:28.049386+0000","last_unstale":"2021-03-15T22:22:48.209091+0000","last_undegraded":"2021-03-15T22:22:48.209091+0000","last_fullsized":"2021-03-15T22:22:48.209091+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:21:00.125316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:00.125316+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"66","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.672334+0000","last_change":"2021-03-15T22:22:04.618748+0000","last_active":"2021-03-15T22:22:44.672334+0000","last_peered":"2021-03-15T22:22:44.672334+0000","last_clean":"2021-03-15T22:22:44.672334+0000","last_became_active":"2021-03-15T22:21:56.035432+0000","last_became_peered":"2021-03-15T22:21:56.035432+0000","last_unstale":"2021-03-15T22:22:44.672334+0000","last_undegraded":"2021-03-15T22:22:44.672334+0000","last_fullsized":"2021-03-15T22:22:44.672334+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:04.618673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:04.618673+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"213","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040061+0000","last_change":"2021-03-15T22:20:53.271135+0000","last_active":"2021-03-15T22:22:45.040061+0000","last_peered":"2021-03-15T22:22:45.040061+0000","last_clean":"2021-03-15T22:22:45.040061+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:22:45.040061+0000","last_undegraded":"2021-03-15T22:22:45.040061+0000","last_fullsized":"2021-03-15T22:22:45.040061+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:53.271063+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:53.271063+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"159","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.050780+0000","last_change":"2021-03-15T22:21:49.930676+0000","last_active":"2021-03-15T22:22:46.050780+0000","last_peered":"2021-03-15T22:22:46.050780+0000","last_clean":"2021-03-15T22:22:46.050780+0000","last_became_active":"2021-03-15T22:21:28.045170+0000","last_became_peered":"2021-03-15T22:21:28.045170+0000","last_unstale":"2021-03-15T22:22:46.050780+0000","last_undegraded":"2021-03-15T22:22:46.050780+0000","last_fullsized":"2021-03-15T22:22:46.050780+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:21:49.930593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:49.930593+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":36,"num_read_kb":19,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"206","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375342+0000","last_change":"2021-03-15T22:21:03.982348+0000","last_active":"2021-03-15T22:22:47.375342+0000","last_peered":"2021-03-15T22:22:47.375342+0000","last_clean":"2021-03-15T22:22:47.375342+0000","last_became_active":"2021-03-15T22:18:00.015889+0000","last_became_peered":"2021-03-15T22:18:00.015889+0000","last_unstale":"2021-03-15T22:22:47.375342+0000","last_undegraded":"2021-03-15T22:22:47.375342+0000","last_fullsized":"2021-03-15T22:22:47.375342+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:03.982279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:03.982279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"226","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.049893+0000","last_change":"2021-03-15T22:22:02.880850+0000","last_active":"2021-03-15T22:22:46.049893+0000","last_peered":"2021-03-15T22:22:46.049893+0000","last_clean":"2021-03-15T22:22:46.049893+0000","last_became_active":"2021-03-15T22:21:56.037322+0000","last_became_peered":"2021-03-15T22:21:56.037322+0000","last_unstale":"2021-03-15T22:22:46.049893+0000","last_undegraded":"2021-03-15T22:22:46.049893+0000","last_fullsized":"2021-03-15T22:22:46.049893+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:02.880783+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:02.880783+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"280","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.199000+0000","last_change":"2021-03-15T22:22:07.679421+0000","last_active":"2021-03-15T22:22:44.199000+0000","last_peered":"2021-03-15T22:22:44.199000+0000","last_clean":"2021-03-15T22:22:44.199000+0000","last_became_active":"2021-03-15T22:21:28.051429+0000","last_became_peered":"2021-03-15T22:21:28.051429+0000","last_unstale":"2021-03-15T22:22:44.199000+0000","last_undegraded":"2021-03-15T22:22:44.199000+0000","last_fullsized":"2021-03-15T22:22:44.199000+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:22:07.679358+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:22:07.679358+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"211","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.051113+0000","last_change":"2021-03-15T22:22:04.890835+0000","last_active":"2021-03-15T22:22:46.051113+0000","last_peered":"2021-03-15T22:22:46.051113+0000","last_clean":"2021-03-15T22:22:46.051113+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:22:46.051113+0000","last_undegraded":"2021-03-15T22:22:46.051113+0000","last_fullsized":"2021-03-15T22:22:46.051113+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:04.890767+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:04.890767+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"94","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.197643+0000","last_change":"2021-03-15T22:21:53.852914+0000","last_active":"2021-03-15T22:22:44.197643+0000","last_peered":"2021-03-15T22:22:44.197643+0000","last_clean":"2021-03-15T22:22:44.197643+0000","last_became_active":"2021-03-15T22:21:28.044842+0000","last_became_peered":"2021-03-15T22:21:28.044842+0000","last_unstale":"2021-03-15T22:22:44.197643+0000","last_undegraded":"2021-03-15T22:22:44.197643+0000","last_fullsized":"2021-03-15T22:22:44.197643+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:53.852830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:53.852830+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"238","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.672437+0000","last_change":"2021-03-15T22:21:24.823853+0000","last_active":"2021-03-15T22:22:44.672437+0000","last_peered":"2021-03-15T22:22:44.672437+0000","last_clean":"2021-03-15T22:22:44.672437+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:22:44.672437+0000","last_undegraded":"2021-03-15T22:22:44.672437+0000","last_fullsized":"2021-03-15T22:22:44.672437+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:21:24.823789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:24.823789+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"211","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375194+0000","last_change":"2021-03-15T22:22:08.753158+0000","last_active":"2021-03-15T22:22:47.375194+0000","last_peered":"2021-03-15T22:22:47.375194+0000","last_clean":"2021-03-15T22:22:47.375194+0000","last_became_active":"2021-03-15T22:18:00.015858+0000","last_became_peered":"2021-03-15T22:18:00.015858+0000","last_unstale":"2021-03-15T22:22:47.375194+0000","last_undegraded":"2021-03-15T22:22:47.375194+0000","last_fullsized":"2021-03-15T22:22:47.375194+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:08.753083+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:22:08.753083+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"212","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.671567+0000","last_change":"2021-03-15T22:22:08.682317+0000","last_active":"2021-03-15T22:22:44.671567+0000","last_peered":"2021-03-15T22:22:44.671567+0000","last_clean":"2021-03-15T22:22:44.671567+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:22:44.671567+0000","last_undegraded":"2021-03-15T22:22:44.671567+0000","last_fullsized":"2021-03-15T22:22:44.671567+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:08.682244+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:08.682244+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"220","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.199147+0000","last_change":"2021-03-15T22:21:28.044437+0000","last_active":"2021-03-15T22:22:44.199147+0000","last_peered":"2021-03-15T22:22:44.199147+0000","last_clean":"2021-03-15T22:22:44.199147+0000","last_became_active":"2021-03-15T22:21:28.044155+0000","last_became_peered":"2021-03-15T22:21:28.044155+0000","last_unstale":"2021-03-15T22:22:44.199147+0000","last_undegraded":"2021-03-15T22:22:44.199147+0000","last_fullsized":"2021-03-15T22:22:44.199147+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:21:05.736770+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"124","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.052477+0000","last_change":"2021-03-15T22:22:01.885586+0000","last_active":"2021-03-15T22:22:46.052477+0000","last_peered":"2021-03-15T22:22:46.052477+0000","last_clean":"2021-03-15T22:22:46.052477+0000","last_became_active":"2021-03-15T22:21:56.039344+0000","last_became_peered":"2021-03-15T22:21:56.039344+0000","last_unstale":"2021-03-15T22:22:46.052477+0000","last_undegraded":"2021-03-15T22:22:46.052477+0000","last_fullsized":"2021-03-15T22:22:46.052477+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:22:01.885519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:01.885519+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":11,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"89","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198100+0000","last_change":"2021-03-15T22:21:28.048155+0000","last_active":"2021-03-15T22:22:44.198100+0000","last_peered":"2021-03-15T22:22:44.198100+0000","last_clean":"2021-03-15T22:22:44.198100+0000","last_became_active":"2021-03-15T22:21:28.047835+0000","last_became_peered":"2021-03-15T22:21:28.047835+0000","last_unstale":"2021-03-15T22:22:44.198100+0000","last_undegraded":"2021-03-15T22:22:44.198100+0000","last_fullsized":"2021-03-15T22:22:44.198100+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:56.799862+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:56.799862+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,2],"acting":[0,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"208","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375400+0000","last_change":"2021-03-15T22:20:54.994251+0000","last_active":"2021-03-15T22:22:47.375400+0000","last_peered":"2021-03-15T22:22:47.375400+0000","last_clean":"2021-03-15T22:22:47.375400+0000","last_became_active":"2021-03-15T22:17:58.013174+0000","last_became_peered":"2021-03-15T22:17:58.013174+0000","last_unstale":"2021-03-15T22:22:47.375400+0000","last_undegraded":"2021-03-15T22:22:47.375400+0000","last_fullsized":"2021-03-15T22:22:47.375400+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:54.994178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:20:54.994178+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"211","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.047653+0000","last_change":"2021-03-15T22:20:57.165778+0000","last_active":"2021-03-15T22:22:46.047653+0000","last_peered":"2021-03-15T22:22:46.047653+0000","last_clean":"2021-03-15T22:22:46.047653+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:22:46.047653+0000","last_undegraded":"2021-03-15T22:22:46.047653+0000","last_fullsized":"2021-03-15T22:22:46.047653+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:57.165693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:57.165693+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"90'22","reported_seq":"153","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375051+0000","last_change":"2021-03-15T22:22:01.769364+0000","last_active":"2021-03-15T22:22:47.375051+0000","last_peered":"2021-03-15T22:22:47.375051+0000","last_clean":"2021-03-15T22:22:47.375051+0000","last_became_active":"2021-03-15T22:21:56.034049+0000","last_became_peered":"2021-03-15T22:21:56.034049+0000","last_unstale":"2021-03-15T22:22:47.375051+0000","last_undegraded":"2021-03-15T22:22:47.375051+0000","last_fullsized":"2021-03-15T22:22:47.375051+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:22:01.769284+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:22:01.769284+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,5],"acting":[7,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"205","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.197394+0000","last_change":"2021-03-15T22:22:06.723810+0000","last_active":"2021-03-15T22:22:44.197394+0000","last_peered":"2021-03-15T22:22:44.197394+0000","last_clean":"2021-03-15T22:22:44.197394+0000","last_became_active":"2021-03-15T22:18:00.014112+0000","last_became_peered":"2021-03-15T22:18:00.014112+0000","last_unstale":"2021-03-15T22:22:44.197394+0000","last_undegraded":"2021-03-15T22:22:44.197394+0000","last_fullsized":"2021-03-15T22:22:44.197394+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:06.723750+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:06.723750+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"61","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198156+0000","last_change":"2021-03-15T22:21:56.037408+0000","last_active":"2021-03-15T22:22:44.198156+0000","last_peered":"2021-03-15T22:22:44.198156+0000","last_clean":"2021-03-15T22:22:44.198156+0000","last_became_active":"2021-03-15T22:21:56.037065+0000","last_became_peered":"2021-03-15T22:21:56.037065+0000","last_unstale":"2021-03-15T22:22:44.198156+0000","last_undegraded":"2021-03-15T22:22:44.198156+0000","last_fullsized":"2021-03-15T22:22:44.198156+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:52.775171+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:03.699198+0000","last_clean_scrub_stamp":"2021-03-15T22:20:52.775171+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"221","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375432+0000","last_change":"2021-03-15T22:21:56.030382+0000","last_active":"2021-03-15T22:22:47.375432+0000","last_peered":"2021-03-15T22:22:47.375432+0000","last_clean":"2021-03-15T22:22:47.375432+0000","last_became_active":"2021-03-15T22:21:56.029986+0000","last_became_peered":"2021-03-15T22:21:56.029986+0000","last_unstale":"2021-03-15T22:22:47.375432+0000","last_undegraded":"2021-03-15T22:22:47.375432+0000","last_fullsized":"2021-03-15T22:22:47.375432+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:20.673603+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:20.673603+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"238","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.052953+0000","last_change":"2021-03-15T22:21:24.789596+0000","last_active":"2021-03-15T22:22:46.052953+0000","last_peered":"2021-03-15T22:22:46.052953+0000","last_clean":"2021-03-15T22:22:46.052953+0000","last_became_active":"2021-03-15T22:17:55.991537+0000","last_became_peered":"2021-03-15T22:17:55.991537+0000","last_unstale":"2021-03-15T22:22:46.052953+0000","last_undegraded":"2021-03-15T22:22:46.052953+0000","last_fullsized":"2021-03-15T22:22:46.052953+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:21:24.789526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:24.789526+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"89","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.197846+0000","last_change":"2021-03-15T22:21:28.042915+0000","last_active":"2021-03-15T22:22:44.197846+0000","last_peered":"2021-03-15T22:22:44.197846+0000","last_clean":"2021-03-15T22:22:44.197846+0000","last_became_active":"2021-03-15T22:21:28.042578+0000","last_became_peered":"2021-03-15T22:21:28.042578+0000","last_unstale":"2021-03-15T22:22:44.197846+0000","last_undegraded":"2021-03-15T22:22:44.197846+0000","last_fullsized":"2021-03-15T22:22:44.197846+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:23.821451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:23.821451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"222","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.052807+0000","last_change":"2021-03-15T22:22:39.054218+0000","last_active":"2021-03-15T22:22:46.052807+0000","last_peered":"2021-03-15T22:22:46.052807+0000","last_clean":"2021-03-15T22:22:46.052807+0000","last_became_active":"2021-03-15T22:21:28.047856+0000","last_became_peered":"2021-03-15T22:21:28.047856+0000","last_unstale":"2021-03-15T22:22:46.052807+0000","last_undegraded":"2021-03-15T22:22:46.052807+0000","last_fullsized":"2021-03-15T22:22:46.052807+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:39.054153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:39.054153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"213","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040167+0000","last_change":"2021-03-15T22:21:22.324910+0000","last_active":"2021-03-15T22:22:45.040167+0000","last_peered":"2021-03-15T22:22:45.040167+0000","last_clean":"2021-03-15T22:22:45.040167+0000","last_became_active":"2021-03-15T22:17:53.975292+0000","last_became_peered":"2021-03-15T22:17:53.975292+0000","last_unstale":"2021-03-15T22:22:45.040167+0000","last_undegraded":"2021-03-15T22:22:45.040167+0000","last_fullsized":"2021-03-15T22:22:45.040167+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:22.324836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:22.324836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"238","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040375+0000","last_change":"2021-03-15T22:20:55.236593+0000","last_active":"2021-03-15T22:22:45.040375+0000","last_peered":"2021-03-15T22:22:45.040375+0000","last_clean":"2021-03-15T22:22:45.040375+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:22:45.040375+0000","last_undegraded":"2021-03-15T22:22:45.040375+0000","last_fullsized":"2021-03-15T22:22:45.040375+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:20:55.236487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:20:55.236487+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"214","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198830+0000","last_change":"2021-03-15T22:22:30.800481+0000","last_active":"2021-03-15T22:22:44.198830+0000","last_peered":"2021-03-15T22:22:44.198830+0000","last_clean":"2021-03-15T22:22:44.198830+0000","last_became_active":"2021-03-15T22:21:28.046227+0000","last_became_peered":"2021-03-15T22:21:28.046227+0000","last_unstale":"2021-03-15T22:22:44.198830+0000","last_undegraded":"2021-03-15T22:22:44.198830+0000","last_fullsized":"2021-03-15T22:22:44.198830+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:30.800426+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:30.800426+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"61'2","reported_seq":"267","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:47.057737+0000","last_change":"2021-03-15T22:21:51.156735+0000","last_active":"2021-03-15T22:22:47.057737+0000","last_peered":"2021-03-15T22:22:47.057737+0000","last_clean":"2021-03-15T22:22:47.057737+0000","last_became_active":"2021-03-15T22:17:58.010903+0000","last_became_peered":"2021-03-15T22:17:58.010903+0000","last_unstale":"2021-03-15T22:22:47.057737+0000","last_undegraded":"2021-03-15T22:22:47.057737+0000","last_fullsized":"2021-03-15T22:22:47.057737+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:21:51.156644+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:51.156644+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"212","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375060+0000","last_change":"2021-03-15T22:21:22.710972+0000","last_active":"2021-03-15T22:22:47.375060+0000","last_peered":"2021-03-15T22:22:47.375060+0000","last_clean":"2021-03-15T22:22:47.375060+0000","last_became_active":"2021-03-15T22:17:54.285851+0000","last_became_peered":"2021-03-15T22:17:54.285851+0000","last_unstale":"2021-03-15T22:22:47.375060+0000","last_undegraded":"2021-03-15T22:22:47.375060+0000","last_fullsized":"2021-03-15T22:22:47.375060+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:22.710908+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:22.710908+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"123","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.051871+0000","last_change":"2021-03-15T22:22:07.968987+0000","last_active":"2021-03-15T22:22:46.051871+0000","last_peered":"2021-03-15T22:22:46.051871+0000","last_clean":"2021-03-15T22:22:46.051871+0000","last_became_active":"2021-03-15T22:21:56.039170+0000","last_became_peered":"2021-03-15T22:21:56.039170+0000","last_unstale":"2021-03-15T22:22:46.051871+0000","last_undegraded":"2021-03-15T22:22:46.051871+0000","last_fullsized":"2021-03-15T22:22:46.051871+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:22:07.968919+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:07.968919+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":13,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":624,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"207","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040569+0000","last_change":"2021-03-15T22:22:41.941923+0000","last_active":"2021-03-15T22:22:45.040569+0000","last_peered":"2021-03-15T22:22:45.040569+0000","last_clean":"2021-03-15T22:22:45.040569+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:22:45.040569+0000","last_undegraded":"2021-03-15T22:22:45.040569+0000","last_fullsized":"2021-03-15T22:22:45.040569+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:41.941855+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:41.941855+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"173'3","reported_seq":"126","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375318+0000","last_change":"2021-03-15T22:21:31.689038+0000","last_active":"2021-03-15T22:22:47.375318+0000","last_peered":"2021-03-15T22:22:47.375318+0000","last_clean":"2021-03-15T22:22:47.375318+0000","last_became_active":"2021-03-15T22:21:28.048472+0000","last_became_peered":"2021-03-15T22:21:28.048472+0000","last_unstale":"2021-03-15T22:22:47.375318+0000","last_undegraded":"2021-03-15T22:22:47.375318+0000","last_fullsized":"2021-03-15T22:22:47.375318+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'3","last_scrub_stamp":"2021-03-15T22:21:31.688955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:31.688955+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"220","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198087+0000","last_change":"2021-03-15T22:21:28.045090+0000","last_active":"2021-03-15T22:22:44.198087+0000","last_peered":"2021-03-15T22:22:44.198087+0000","last_clean":"2021-03-15T22:22:44.198087+0000","last_became_active":"2021-03-15T22:21:28.044742+0000","last_became_peered":"2021-03-15T22:21:28.044742+0000","last_unstale":"2021-03-15T22:22:44.198087+0000","last_undegraded":"2021-03-15T22:22:44.198087+0000","last_fullsized":"2021-03-15T22:22:44.198087+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:00.665226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:00.665226+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"260","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.048737+0000","last_change":"2021-03-15T22:22:03.126656+0000","last_active":"2021-03-15T22:22:46.048737+0000","last_peered":"2021-03-15T22:22:46.048737+0000","last_clean":"2021-03-15T22:22:46.048737+0000","last_became_active":"2021-03-15T22:21:56.032898+0000","last_became_peered":"2021-03-15T22:21:56.032898+0000","last_unstale":"2021-03-15T22:22:46.048737+0000","last_undegraded":"2021-03-15T22:22:46.048737+0000","last_fullsized":"2021-03-15T22:22:46.048737+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:22:03.126578+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:03.126578+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"214","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.209243+0000","last_change":"2021-03-15T22:22:41.652274+0000","last_active":"2021-03-15T22:22:48.209243+0000","last_peered":"2021-03-15T22:22:48.209243+0000","last_clean":"2021-03-15T22:22:48.209243+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:22:48.209243+0000","last_undegraded":"2021-03-15T22:22:48.209243+0000","last_fullsized":"2021-03-15T22:22:48.209243+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:41.652191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:22:41.652191+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"214","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.374986+0000","last_change":"2021-03-15T22:22:03.775204+0000","last_active":"2021-03-15T22:22:47.374986+0000","last_peered":"2021-03-15T22:22:47.374986+0000","last_clean":"2021-03-15T22:22:47.374986+0000","last_became_active":"2021-03-15T22:17:53.976777+0000","last_became_peered":"2021-03-15T22:17:53.976777+0000","last_unstale":"2021-03-15T22:22:47.374986+0000","last_undegraded":"2021-03-15T22:22:47.374986+0000","last_fullsized":"2021-03-15T22:22:47.374986+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:03.775128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:03.775128+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"272","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.199088+0000","last_change":"2021-03-15T22:21:56.058518+0000","last_active":"2021-03-15T22:22:44.199088+0000","last_peered":"2021-03-15T22:22:44.199088+0000","last_clean":"2021-03-15T22:22:44.199088+0000","last_became_active":"2021-03-15T22:21:56.036234+0000","last_became_peered":"2021-03-15T22:21:56.036234+0000","last_unstale":"2021-03-15T22:22:44.199088+0000","last_undegraded":"2021-03-15T22:22:44.199088+0000","last_fullsized":"2021-03-15T22:22:44.199088+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:21:23.838983+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:23.838983+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":1206,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"206","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375342+0000","last_change":"2021-03-15T22:20:35.969982+0000","last_active":"2021-03-15T22:22:47.375342+0000","last_peered":"2021-03-15T22:22:47.375342+0000","last_clean":"2021-03-15T22:22:47.375342+0000","last_became_active":"2021-03-15T22:18:00.288462+0000","last_became_peered":"2021-03-15T22:18:00.288462+0000","last_unstale":"2021-03-15T22:22:47.375342+0000","last_undegraded":"2021-03-15T22:22:47.375342+0000","last_fullsized":"2021-03-15T22:22:47.375342+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:35.969903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:35.969903+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"214","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.671245+0000","last_change":"2021-03-15T22:21:54.720472+0000","last_active":"2021-03-15T22:22:44.671245+0000","last_peered":"2021-03-15T22:22:44.671245+0000","last_clean":"2021-03-15T22:22:44.671245+0000","last_became_active":"2021-03-15T22:17:53.977470+0000","last_became_peered":"2021-03-15T22:17:53.977470+0000","last_unstale":"2021-03-15T22:22:44.671245+0000","last_undegraded":"2021-03-15T22:22:44.671245+0000","last_fullsized":"2021-03-15T22:22:44.671245+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:54.720368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:54.720368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"65'7","reported_seq":"132","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375559+0000","last_change":"2021-03-15T22:22:28.717607+0000","last_active":"2021-03-15T22:22:47.375559+0000","last_peered":"2021-03-15T22:22:47.375559+0000","last_clean":"2021-03-15T22:22:47.375559+0000","last_became_active":"2021-03-15T22:21:28.040048+0000","last_became_peered":"2021-03-15T22:21:28.040048+0000","last_unstale":"2021-03-15T22:22:47.375559+0000","last_undegraded":"2021-03-15T22:22:47.375559+0000","last_fullsized":"2021-03-15T22:22:47.375559+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'7","last_scrub_stamp":"2021-03-15T22:22:28.717530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:28.717530+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":158,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"248'137","reported_seq":"656","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.052379+0000","last_change":"2021-03-15T22:22:46.052379+0000","last_active":"2021-03-15T22:22:46.052379+0000","last_peered":"2021-03-15T22:22:46.052379+0000","last_clean":"2021-03-15T22:22:46.052379+0000","last_became_active":"2021-03-15T22:20:05.038191+0000","last_became_peered":"2021-03-15T22:20:05.038191+0000","last_unstale":"2021-03-15T22:22:46.052379+0000","last_undegraded":"2021-03-15T22:22:46.052379+0000","last_fullsized":"2021-03-15T22:22:46.052379+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"168'74","last_scrub_stamp":"2021-03-15T22:21:25.834723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:21:25.834723+0000","log_size":137,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9539549,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":98,"num_read_kb":106035,"num_write":74,"num_write_kb":20418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"84","length":"1"},{"start":"95","length":"1"},{"start":"98","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"210","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.048412+0000","last_change":"2021-03-15T22:22:28.062846+0000","last_active":"2021-03-15T22:22:46.048412+0000","last_peered":"2021-03-15T22:22:46.048412+0000","last_clean":"2021-03-15T22:22:46.048412+0000","last_became_active":"2021-03-15T22:18:00.286699+0000","last_became_peered":"2021-03-15T22:18:00.286699+0000","last_unstale":"2021-03-15T22:22:46.048412+0000","last_undegraded":"2021-03-15T22:22:46.048412+0000","last_fullsized":"2021-03-15T22:22:46.048412+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:28.062751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:28.062751+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"215","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375609+0000","last_change":"2021-03-15T22:20:28.025972+0000","last_active":"2021-03-15T22:22:47.375609+0000","last_peered":"2021-03-15T22:22:47.375609+0000","last_clean":"2021-03-15T22:22:47.375609+0000","last_became_active":"2021-03-15T22:17:53.979246+0000","last_became_peered":"2021-03-15T22:17:53.979246+0000","last_unstale":"2021-03-15T22:22:47.375609+0000","last_undegraded":"2021-03-15T22:22:47.375609+0000","last_fullsized":"2021-03-15T22:22:47.375609+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:20:28.025885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:28.025885+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"240","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.049084+0000","last_change":"2021-03-15T22:21:03.144874+0000","last_active":"2021-03-15T22:22:46.049084+0000","last_peered":"2021-03-15T22:22:46.049084+0000","last_clean":"2021-03-15T22:22:46.049084+0000","last_became_active":"2021-03-15T22:17:56.289684+0000","last_became_peered":"2021-03-15T22:17:56.289684+0000","last_unstale":"2021-03-15T22:22:46.049084+0000","last_undegraded":"2021-03-15T22:22:46.049084+0000","last_fullsized":"2021-03-15T22:22:46.049084+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:21:03.144807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:03.144807+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":20,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"172'5","reported_seq":"288","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375091+0000","last_change":"2021-03-15T22:21:26.679105+0000","last_active":"2021-03-15T22:22:47.375091+0000","last_peered":"2021-03-15T22:22:47.375091+0000","last_clean":"2021-03-15T22:22:47.375091+0000","last_became_active":"2021-03-15T22:21:26.034534+0000","last_became_peered":"2021-03-15T22:21:26.034534+0000","last_unstale":"2021-03-15T22:22:47.375091+0000","last_undegraded":"2021-03-15T22:22:47.375091+0000","last_fullsized":"2021-03-15T22:22:47.375091+0000","mapping_epoch":60,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":171,"parent":"0.0","parent_split_bits":0,"last_scrub":"89'3","last_scrub_stamp":"2021-03-15T22:21:26.679039+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:21:26.679039+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"253'303","reported_seq":"866","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.227152+0000","last_change":"2021-03-15T22:22:48.227152+0000","last_active":"2021-03-15T22:22:48.227152+0000","last_peered":"2021-03-15T22:22:48.227152+0000","last_clean":"2021-03-15T22:22:48.227152+0000","last_became_active":"2021-03-15T22:22:01.038626+0000","last_became_peered":"2021-03-15T22:22:01.038626+0000","last_unstale":"2021-03-15T22:22:48.227152+0000","last_undegraded":"2021-03-15T22:22:48.227152+0000","last_fullsized":"2021-03-15T22:22:48.227152+0000","mapping_epoch":205,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":206,"parent":"0.0","parent_split_bits":0,"last_scrub":"231'255","last_scrub_stamp":"2021-03-15T22:22:29.689315+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:29.689315+0000","log_size":303,"ondisk_log_size":303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22267991,"num_objects":13,"num_object_clones":8,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":167,"num_read_kb":153198,"num_write":132,"num_write_kb":52430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":12120510,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"84","length":"1"},{"start":"91","length":"1"},{"start":"95","length":"1"},{"start":"98","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"64","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375162+0000","last_change":"2021-03-15T22:21:56.037894+0000","last_active":"2021-03-15T22:22:47.375162+0000","last_peered":"2021-03-15T22:22:47.375162+0000","last_clean":"2021-03-15T22:22:47.375162+0000","last_became_active":"2021-03-15T22:21:56.037602+0000","last_became_peered":"2021-03-15T22:21:56.037602+0000","last_unstale":"2021-03-15T22:22:47.375162+0000","last_undegraded":"2021-03-15T22:22:47.375162+0000","last_fullsized":"2021-03-15T22:22:47.375162+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:31.842182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:31.842182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"217","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375540+0000","last_change":"2021-03-15T22:22:38.810490+0000","last_active":"2021-03-15T22:22:47.375540+0000","last_peered":"2021-03-15T22:22:47.375540+0000","last_clean":"2021-03-15T22:22:47.375540+0000","last_became_active":"2021-03-15T22:17:53.978907+0000","last_became_peered":"2021-03-15T22:17:53.978907+0000","last_unstale":"2021-03-15T22:22:47.375540+0000","last_undegraded":"2021-03-15T22:22:47.375540+0000","last_fullsized":"2021-03-15T22:22:47.375540+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:38.810397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.810397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"267","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040828+0000","last_change":"2021-03-15T22:22:28.887668+0000","last_active":"2021-03-15T22:22:45.040828+0000","last_peered":"2021-03-15T22:22:45.040828+0000","last_clean":"2021-03-15T22:22:45.040828+0000","last_became_active":"2021-03-15T22:21:28.045396+0000","last_became_peered":"2021-03-15T22:21:28.045396+0000","last_unstale":"2021-03-15T22:22:45.040828+0000","last_undegraded":"2021-03-15T22:22:45.040828+0000","last_fullsized":"2021-03-15T22:22:45.040828+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:22:28.887588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:28.887588+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":692,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"234","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.208809+0000","last_change":"2021-03-15T22:22:35.649394+0000","last_active":"2021-03-15T22:22:48.208809+0000","last_peered":"2021-03-15T22:22:48.208809+0000","last_clean":"2021-03-15T22:22:48.208809+0000","last_became_active":"2021-03-15T22:21:30.034044+0000","last_became_peered":"2021-03-15T22:21:30.034044+0000","last_unstale":"2021-03-15T22:22:48.208809+0000","last_undegraded":"2021-03-15T22:22:48.208809+0000","last_fullsized":"2021-03-15T22:22:48.208809+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":175,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:35.649331+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:35.649331+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"249'79","reported_seq":"572","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.054776+0000","last_change":"2021-03-15T22:22:43.208149+0000","last_active":"2021-03-15T22:22:45.054776+0000","last_peered":"2021-03-15T22:22:45.054776+0000","last_clean":"2021-03-15T22:22:45.054776+0000","last_became_active":"2021-03-15T22:20:05.032301+0000","last_became_peered":"2021-03-15T22:20:05.032301+0000","last_unstale":"2021-03-15T22:22:45.054776+0000","last_undegraded":"2021-03-15T22:22:45.054776+0000","last_fullsized":"2021-03-15T22:22:45.054776+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"213'77","last_scrub_stamp":"2021-03-15T22:22:37.982715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:37.982715+0000","log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3109262,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":31,"num_read_kb":21292,"num_write":38,"num_write_kb":13534,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"84","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"215","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.374944+0000","last_change":"2021-03-15T22:21:56.033485+0000","last_active":"2021-03-15T22:22:47.374944+0000","last_peered":"2021-03-15T22:22:47.374944+0000","last_clean":"2021-03-15T22:22:47.374944+0000","last_became_active":"2021-03-15T22:21:56.033006+0000","last_became_peered":"2021-03-15T22:21:56.033006+0000","last_unstale":"2021-03-15T22:22:47.374944+0000","last_undegraded":"2021-03-15T22:22:47.374944+0000","last_fullsized":"2021-03-15T22:22:47.374944+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:53.009472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:53.009472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,6],"acting":[7,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"234","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.209158+0000","last_change":"2021-03-15T22:22:34.656502+0000","last_active":"2021-03-15T22:22:48.209158+0000","last_peered":"2021-03-15T22:22:48.209158+0000","last_clean":"2021-03-15T22:22:48.209158+0000","last_became_active":"2021-03-15T22:21:18.047758+0000","last_became_peered":"2021-03-15T22:21:18.047758+0000","last_unstale":"2021-03-15T22:22:48.209158+0000","last_undegraded":"2021-03-15T22:22:48.209158+0000","last_fullsized":"2021-03-15T22:22:48.209158+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":163,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:34.656440+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:34.656440+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"212","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375561+0000","last_change":"2021-03-15T22:21:06.006147+0000","last_active":"2021-03-15T22:22:47.375561+0000","last_peered":"2021-03-15T22:22:47.375561+0000","last_clean":"2021-03-15T22:22:47.375561+0000","last_became_active":"2021-03-15T22:17:53.974108+0000","last_became_peered":"2021-03-15T22:17:53.974108+0000","last_unstale":"2021-03-15T22:22:47.375561+0000","last_undegraded":"2021-03-15T22:22:47.375561+0000","last_fullsized":"2021-03-15T22:22:47.375561+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:21:06.006086+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"246","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.671105+0000","last_change":"2021-03-15T22:21:33.792545+0000","last_active":"2021-03-15T22:22:44.671105+0000","last_peered":"2021-03-15T22:22:44.671105+0000","last_clean":"2021-03-15T22:22:44.671105+0000","last_became_active":"2021-03-15T22:17:56.288870+0000","last_became_peered":"2021-03-15T22:17:56.288870+0000","last_unstale":"2021-03-15T22:22:44.671105+0000","last_undegraded":"2021-03-15T22:22:44.671105+0000","last_fullsized":"2021-03-15T22:22:44.671105+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:21:33.792449+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:33.792449+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"253'151","reported_seq":"681","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.217028+0000","last_change":"2021-03-15T22:22:48.217028+0000","last_active":"2021-03-15T22:22:48.217028+0000","last_peered":"2021-03-15T22:22:48.217028+0000","last_clean":"2021-03-15T22:22:48.217028+0000","last_became_active":"2021-03-15T22:20:05.032282+0000","last_became_peered":"2021-03-15T22:20:05.032282+0000","last_unstale":"2021-03-15T22:22:48.217028+0000","last_undegraded":"2021-03-15T22:22:48.217028+0000","last_fullsized":"2021-03-15T22:22:48.217028+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'104","last_scrub_stamp":"2021-03-15T22:21:31.442405+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:21:31.442405+0000","log_size":151,"ondisk_log_size":151,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10337338,"num_objects":6,"num_object_clones":3,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":130,"num_read_kb":167088,"num_write":60,"num_write_kb":22874,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"84","length":"1"},{"start":"91","length":"1"},{"start":"95","length":"1"},{"start":"98","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"207","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.197983+0000","last_change":"2021-03-15T22:22:28.711806+0000","last_active":"2021-03-15T22:22:44.197983+0000","last_peered":"2021-03-15T22:22:44.197983+0000","last_clean":"2021-03-15T22:22:44.197983+0000","last_became_active":"2021-03-15T22:18:00.016982+0000","last_became_peered":"2021-03-15T22:18:00.016982+0000","last_unstale":"2021-03-15T22:22:44.197983+0000","last_undegraded":"2021-03-15T22:22:44.197983+0000","last_fullsized":"2021-03-15T22:22:44.197983+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:28.711745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:28.711745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"243","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.210712+0000","last_change":"2021-03-15T22:21:51.578587+0000","last_active":"2021-03-15T22:22:48.210712+0000","last_peered":"2021-03-15T22:22:48.210712+0000","last_clean":"2021-03-15T22:22:48.210712+0000","last_became_active":"2021-03-15T22:21:28.049417+0000","last_became_peered":"2021-03-15T22:21:28.049417+0000","last_unstale":"2021-03-15T22:22:48.210712+0000","last_undegraded":"2021-03-15T22:22:48.210712+0000","last_fullsized":"2021-03-15T22:22:48.210712+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:51.578525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:51.578525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,0],"acting":[2,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"220","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.048044+0000","last_change":"2021-03-15T22:21:28.047052+0000","last_active":"2021-03-15T22:22:46.048044+0000","last_peered":"2021-03-15T22:22:46.048044+0000","last_clean":"2021-03-15T22:22:46.048044+0000","last_became_active":"2021-03-15T22:21:28.046734+0000","last_became_peered":"2021-03-15T22:21:28.046734+0000","last_unstale":"2021-03-15T22:22:46.048044+0000","last_undegraded":"2021-03-15T22:22:46.048044+0000","last_fullsized":"2021-03-15T22:22:46.048044+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:00.088823+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:00.088823+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,1],"acting":[5,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"264","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198320+0000","last_change":"2021-03-15T22:21:30.812577+0000","last_active":"2021-03-15T22:22:44.198320+0000","last_peered":"2021-03-15T22:22:44.198320+0000","last_clean":"2021-03-15T22:22:44.198320+0000","last_became_active":"2021-03-15T22:19:02.093463+0000","last_became_peered":"2021-03-15T22:19:02.093463+0000","last_unstale":"2021-03-15T22:22:44.198320+0000","last_undegraded":"2021-03-15T22:22:44.198320+0000","last_fullsized":"2021-03-15T22:22:44.198320+0000","mapping_epoch":81,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":82,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:21:30.812512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:30.812512+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":894,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"247'111","reported_seq":"625","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.331119+0000","last_change":"2021-03-15T22:22:43.053483+0000","last_active":"2021-03-15T22:22:45.331119+0000","last_peered":"2021-03-15T22:22:45.331119+0000","last_clean":"2021-03-15T22:22:45.331119+0000","last_became_active":"2021-03-15T22:21:28.041958+0000","last_became_peered":"2021-03-15T22:21:28.041958+0000","last_unstale":"2021-03-15T22:22:45.331119+0000","last_undegraded":"2021-03-15T22:22:45.331119+0000","last_fullsized":"2021-03-15T22:22:45.331119+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'88","last_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:22:07.718413+0000","log_size":111,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5899323,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":11,"num_write":66,"num_write_kb":23420,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":5516599,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"84","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"95","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040354+0000","last_change":"2021-03-15T22:21:32.290962+0000","last_active":"2021-03-15T22:22:45.040354+0000","last_peered":"2021-03-15T22:22:45.040354+0000","last_clean":"2021-03-15T22:22:45.040354+0000","last_became_active":"2021-03-15T22:21:28.047359+0000","last_became_peered":"2021-03-15T22:21:28.047359+0000","last_unstale":"2021-03-15T22:22:45.040354+0000","last_undegraded":"2021-03-15T22:22:45.040354+0000","last_fullsized":"2021-03-15T22:22:45.040354+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:32.290882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:32.290882+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"218","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.048277+0000","last_change":"2021-03-15T22:22:39.140480+0000","last_active":"2021-03-15T22:22:46.048277+0000","last_peered":"2021-03-15T22:22:46.048277+0000","last_clean":"2021-03-15T22:22:46.048277+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:22:46.048277+0000","last_undegraded":"2021-03-15T22:22:46.048277+0000","last_fullsized":"2021-03-15T22:22:46.048277+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:39.140419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:22:39.140419+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"213","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040237+0000","last_change":"2021-03-15T22:21:27.231350+0000","last_active":"2021-03-15T22:22:45.040237+0000","last_peered":"2021-03-15T22:22:45.040237+0000","last_clean":"2021-03-15T22:22:45.040237+0000","last_became_active":"2021-03-15T22:17:53.978521+0000","last_became_peered":"2021-03-15T22:17:53.978521+0000","last_unstale":"2021-03-15T22:22:45.040237+0000","last_undegraded":"2021-03-15T22:22:45.040237+0000","last_fullsized":"2021-03-15T22:22:45.040237+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:27.231266+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:27.231266+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"312","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.051749+0000","last_change":"2021-03-15T22:21:55.867576+0000","last_active":"2021-03-15T22:22:46.051749+0000","last_peered":"2021-03-15T22:22:46.051749+0000","last_clean":"2021-03-15T22:22:46.051749+0000","last_became_active":"2021-03-15T22:21:28.052494+0000","last_became_peered":"2021-03-15T22:21:28.052494+0000","last_unstale":"2021-03-15T22:22:46.051749+0000","last_undegraded":"2021-03-15T22:22:46.051749+0000","last_fullsized":"2021-03-15T22:22:46.051749+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:21:55.867509+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:21:55.867509+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":954,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"207","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198830+0000","last_change":"2021-03-15T22:20:30.455518+0000","last_active":"2021-03-15T22:22:44.198830+0000","last_peered":"2021-03-15T22:22:44.198830+0000","last_clean":"2021-03-15T22:22:44.198830+0000","last_became_active":"2021-03-15T22:17:58.017434+0000","last_became_peered":"2021-03-15T22:17:58.017434+0000","last_unstale":"2021-03-15T22:22:44.198830+0000","last_undegraded":"2021-03-15T22:22:44.198830+0000","last_fullsized":"2021-03-15T22:22:44.198830+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:30.455430+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:20:30.455430+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"248'177","reported_seq":"666","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.053850+0000","last_change":"2021-03-15T22:22:43.208782+0000","last_active":"2021-03-15T22:22:45.053850+0000","last_peered":"2021-03-15T22:22:45.053850+0000","last_clean":"2021-03-15T22:22:45.053850+0000","last_became_active":"2021-03-15T22:20:05.038149+0000","last_became_peered":"2021-03-15T22:20:05.038149+0000","last_unstale":"2021-03-15T22:22:45.053850+0000","last_undegraded":"2021-03-15T22:22:45.053850+0000","last_fullsized":"2021-03-15T22:22:45.053850+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":177,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7690988,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":74,"num_read_kb":68171,"num_write":78,"num_write_kb":23797,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"84","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"215","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375647+0000","last_change":"2021-03-15T22:21:56.031698+0000","last_active":"2021-03-15T22:22:47.375647+0000","last_peered":"2021-03-15T22:22:47.375647+0000","last_clean":"2021-03-15T22:22:47.375647+0000","last_became_active":"2021-03-15T22:21:56.031350+0000","last_became_peered":"2021-03-15T22:21:56.031350+0000","last_unstale":"2021-03-15T22:22:47.375647+0000","last_undegraded":"2021-03-15T22:22:47.375647+0000","last_fullsized":"2021-03-15T22:22:47.375647+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:59.975891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:20:59.975891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"217","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.052423+0000","last_change":"2021-03-15T22:22:30.930369+0000","last_active":"2021-03-15T22:22:46.052423+0000","last_peered":"2021-03-15T22:22:46.052423+0000","last_clean":"2021-03-15T22:22:46.052423+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:22:46.052423+0000","last_undegraded":"2021-03-15T22:22:46.052423+0000","last_fullsized":"2021-03-15T22:22:46.052423+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:30.930256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:30.930256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"227","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.208851+0000","last_change":"2021-03-15T22:21:28.050859+0000","last_active":"2021-03-15T22:22:48.208851+0000","last_peered":"2021-03-15T22:22:48.208851+0000","last_clean":"2021-03-15T22:22:48.208851+0000","last_became_active":"2021-03-15T22:21:28.050532+0000","last_became_peered":"2021-03-15T22:21:28.050532+0000","last_unstale":"2021-03-15T22:22:48.208851+0000","last_undegraded":"2021-03-15T22:22:48.208851+0000","last_fullsized":"2021-03-15T22:22:48.208851+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:54.018631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:20:54.018631+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"149'22","reported_seq":"253","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.050268+0000","last_change":"2021-03-15T22:22:40.038242+0000","last_active":"2021-03-15T22:22:46.050268+0000","last_peered":"2021-03-15T22:22:46.050268+0000","last_clean":"2021-03-15T22:22:46.050268+0000","last_became_active":"2021-03-15T22:17:55.992952+0000","last_became_peered":"2021-03-15T22:17:55.992952+0000","last_unstale":"2021-03-15T22:22:46.050268+0000","last_undegraded":"2021-03-15T22:22:46.050268+0000","last_fullsized":"2021-03-15T22:22:46.050268+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'22","last_scrub_stamp":"2021-03-15T22:22:40.038183+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:22:40.038183+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"234","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.209808+0000","last_change":"2021-03-15T22:22:32.668891+0000","last_active":"2021-03-15T22:22:48.209808+0000","last_peered":"2021-03-15T22:22:48.209808+0000","last_clean":"2021-03-15T22:22:48.209808+0000","last_became_active":"2021-03-15T22:21:38.147364+0000","last_became_peered":"2021-03-15T22:21:38.147364+0000","last_unstale":"2021-03-15T22:22:48.209808+0000","last_undegraded":"2021-03-15T22:22:48.209808+0000","last_fullsized":"2021-03-15T22:22:48.209808+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:32.668827+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:32.668827+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"250'374","reported_seq":"959","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.064447+0000","last_change":"2021-03-15T22:22:43.210955+0000","last_active":"2021-03-15T22:22:45.064447+0000","last_peered":"2021-03-15T22:22:45.064447+0000","last_clean":"2021-03-15T22:22:45.064447+0000","last_became_active":"2021-03-15T22:21:31.025186+0000","last_became_peered":"2021-03-15T22:21:31.025186+0000","last_unstale":"2021-03-15T22:22:45.064447+0000","last_undegraded":"2021-03-15T22:22:45.064447+0000","last_fullsized":"2021-03-15T22:22:45.064447+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'350","last_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:22:40.995789+0000","log_size":374,"ondisk_log_size":374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27552458,"num_objects":13,"num_object_clones":7,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":171,"num_read_kb":127057,"num_write":175,"num_write_kb":75379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":15951974,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"84","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"214","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.047771+0000","last_change":"2021-03-15T22:21:56.040261+0000","last_active":"2021-03-15T22:22:46.047771+0000","last_peered":"2021-03-15T22:22:46.047771+0000","last_clean":"2021-03-15T22:22:46.047771+0000","last_became_active":"2021-03-15T22:21:56.039902+0000","last_became_peered":"2021-03-15T22:21:56.039902+0000","last_unstale":"2021-03-15T22:22:46.047771+0000","last_undegraded":"2021-03-15T22:22:46.047771+0000","last_fullsized":"2021-03-15T22:22:46.047771+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:20.983187+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:20.983187+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"215","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375078+0000","last_change":"2021-03-15T22:20:54.007423+0000","last_active":"2021-03-15T22:22:47.375078+0000","last_peered":"2021-03-15T22:22:47.375078+0000","last_clean":"2021-03-15T22:22:47.375078+0000","last_became_active":"2021-03-15T22:17:51.044140+0000","last_became_peered":"2021-03-15T22:17:51.044140+0000","last_unstale":"2021-03-15T22:22:47.375078+0000","last_undegraded":"2021-03-15T22:22:47.375078+0000","last_fullsized":"2021-03-15T22:22:47.375078+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:54.007335+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:20:54.007335+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"90","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.048233+0000","last_change":"2021-03-15T22:21:28.039449+0000","last_active":"2021-03-15T22:22:46.048233+0000","last_peered":"2021-03-15T22:22:46.048233+0000","last_clean":"2021-03-15T22:22:46.048233+0000","last_became_active":"2021-03-15T22:21:28.039041+0000","last_became_peered":"2021-03-15T22:21:28.039041+0000","last_unstale":"2021-03-15T22:22:46.048233+0000","last_undegraded":"2021-03-15T22:22:46.048233+0000","last_fullsized":"2021-03-15T22:22:46.048233+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:04.885373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:04.885373+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"236","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.671139+0000","last_change":"2021-03-15T22:22:31.635919+0000","last_active":"2021-03-15T22:22:44.671139+0000","last_peered":"2021-03-15T22:22:44.671139+0000","last_clean":"2021-03-15T22:22:44.671139+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:22:44.671139+0000","last_undegraded":"2021-03-15T22:22:44.671139+0000","last_fullsized":"2021-03-15T22:22:44.671139+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:22:31.635832+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:31.635832+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"208","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375422+0000","last_change":"2021-03-15T22:20:31.051075+0000","last_active":"2021-03-15T22:22:47.375422+0000","last_peered":"2021-03-15T22:22:47.375422+0000","last_clean":"2021-03-15T22:22:47.375422+0000","last_became_active":"2021-03-15T22:17:58.011911+0000","last_became_peered":"2021-03-15T22:17:58.011911+0000","last_unstale":"2021-03-15T22:22:47.375422+0000","last_undegraded":"2021-03-15T22:22:47.375422+0000","last_fullsized":"2021-03-15T22:22:47.375422+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:31.050970+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:31.050970+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"246'97","reported_seq":"605","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.214730+0000","last_change":"2021-03-15T22:22:48.215225+0000","last_active":"2021-03-15T22:22:48.214730+0000","last_peered":"2021-03-15T22:22:48.214730+0000","last_clean":"2021-03-15T22:22:48.214730+0000","last_became_active":"2021-03-15T22:20:05.035608+0000","last_became_peered":"2021-03-15T22:20:05.035608+0000","last_unstale":"2021-03-15T22:22:48.214730+0000","last_undegraded":"2021-03-15T22:22:48.214730+0000","last_fullsized":"2021-03-15T22:22:48.214730+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"172'65","last_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:21:27.654567+0000","log_size":97,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":4503638,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":47036,"num_write":46,"num_write_kb":17519,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"84","length":"1"},{"start":"91","length":"1"},{"start":"95","length":"1"},{"start":"98","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"220","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.051411+0000","last_change":"2021-03-15T22:22:41.086239+0000","last_active":"2021-03-15T22:22:46.051411+0000","last_peered":"2021-03-15T22:22:46.051411+0000","last_clean":"2021-03-15T22:22:46.051411+0000","last_became_active":"2021-03-15T22:21:28.046898+0000","last_became_peered":"2021-03-15T22:21:28.046898+0000","last_unstale":"2021-03-15T22:22:46.051411+0000","last_undegraded":"2021-03-15T22:22:46.051411+0000","last_fullsized":"2021-03-15T22:22:46.051411+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:41.086181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:41.086181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"220","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.197823+0000","last_change":"2021-03-15T22:22:39.928411+0000","last_active":"2021-03-15T22:22:44.197823+0000","last_peered":"2021-03-15T22:22:44.197823+0000","last_clean":"2021-03-15T22:22:44.197823+0000","last_became_active":"2021-03-15T22:21:28.047113+0000","last_became_peered":"2021-03-15T22:21:28.047113+0000","last_unstale":"2021-03-15T22:22:44.197823+0000","last_undegraded":"2021-03-15T22:22:44.197823+0000","last_fullsized":"2021-03-15T22:22:44.197823+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:39.928349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:39.928349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"512","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.212059+0000","last_change":"2021-03-15T22:20:36.923986+0000","last_active":"2021-03-15T22:22:48.212059+0000","last_peered":"2021-03-15T22:22:48.212059+0000","last_clean":"2021-03-15T22:22:48.212059+0000","last_became_active":"2021-03-15T22:17:51.044139+0000","last_became_peered":"2021-03-15T22:17:51.044139+0000","last_unstale":"2021-03-15T22:22:48.212059+0000","last_undegraded":"2021-03-15T22:22:48.212059+0000","last_fullsized":"2021-03-15T22:22:48.212059+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:20:36.923907+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:20:36.923907+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":290,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"235","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198901+0000","last_change":"2021-03-15T22:22:05.739037+0000","last_active":"2021-03-15T22:22:44.198901+0000","last_peered":"2021-03-15T22:22:44.198901+0000","last_clean":"2021-03-15T22:22:44.198901+0000","last_became_active":"2021-03-15T22:17:55.999564+0000","last_became_peered":"2021-03-15T22:17:55.999564+0000","last_unstale":"2021-03-15T22:22:44.198901+0000","last_undegraded":"2021-03-15T22:22:44.198901+0000","last_fullsized":"2021-03-15T22:22:44.198901+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:22:05.738937+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:22:05.738937+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":30,"num_read_kb":16,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"230","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.048195+0000","last_change":"2021-03-15T22:22:38.101741+0000","last_active":"2021-03-15T22:22:46.048195+0000","last_peered":"2021-03-15T22:22:46.048195+0000","last_clean":"2021-03-15T22:22:46.048195+0000","last_became_active":"2021-03-15T22:21:34.036372+0000","last_became_peered":"2021-03-15T22:21:34.036372+0000","last_unstale":"2021-03-15T22:22:46.048195+0000","last_undegraded":"2021-03-15T22:22:46.048195+0000","last_fullsized":"2021-03-15T22:22:46.048195+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":179,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:38.101646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.101646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"249'442","reported_seq":"1067","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.048426+0000","last_change":"2021-03-15T22:22:46.048426+0000","last_active":"2021-03-15T22:22:46.048426+0000","last_peered":"2021-03-15T22:22:46.048426+0000","last_clean":"2021-03-15T22:22:46.048426+0000","last_became_active":"2021-03-15T22:22:02.040514+0000","last_became_peered":"2021-03-15T22:22:02.040514+0000","last_unstale":"2021-03-15T22:22:46.048426+0000","last_undegraded":"2021-03-15T22:22:46.048426+0000","last_fullsized":"2021-03-15T22:22:46.048426+0000","mapping_epoch":206,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":207,"parent":"0.0","parent_split_bits":0,"last_scrub":"211'337","last_scrub_stamp":"2021-03-15T22:22:06.142035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:06.142035+0000","log_size":442,"ondisk_log_size":442,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24476932,"num_objects":13,"num_object_clones":7,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":274,"num_read_kb":282302,"num_write":196,"num_write_kb":68145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":18565099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"84","length":"1"},{"start":"95","length":"1"},{"start":"98","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"211","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.049976+0000","last_change":"2021-03-15T22:22:33.947421+0000","last_active":"2021-03-15T22:22:46.049976+0000","last_peered":"2021-03-15T22:22:46.049976+0000","last_clean":"2021-03-15T22:22:46.049976+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:22:46.049976+0000","last_undegraded":"2021-03-15T22:22:46.049976+0000","last_fullsized":"2021-03-15T22:22:46.049976+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:33.947357+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:22:33.947357+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"217","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.051597+0000","last_change":"2021-03-15T22:22:34.938551+0000","last_active":"2021-03-15T22:22:46.051597+0000","last_peered":"2021-03-15T22:22:46.051597+0000","last_clean":"2021-03-15T22:22:46.051597+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:22:46.051597+0000","last_undegraded":"2021-03-15T22:22:46.051597+0000","last_fullsized":"2021-03-15T22:22:46.051597+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:34.938482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:34.938482+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"259","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198452+0000","last_change":"2021-03-15T22:20:28.505643+0000","last_active":"2021-03-15T22:22:44.198452+0000","last_peered":"2021-03-15T22:22:44.198452+0000","last_clean":"2021-03-15T22:22:44.198452+0000","last_became_active":"2021-03-15T22:17:56.288438+0000","last_became_peered":"2021-03-15T22:17:56.288438+0000","last_unstale":"2021-03-15T22:22:44.198452+0000","last_undegraded":"2021-03-15T22:22:44.198452+0000","last_fullsized":"2021-03-15T22:22:44.198452+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:20:28.505568+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:20:28.505568+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"210","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.671722+0000","last_change":"2021-03-15T22:22:39.712285+0000","last_active":"2021-03-15T22:22:44.671722+0000","last_peered":"2021-03-15T22:22:44.671722+0000","last_clean":"2021-03-15T22:22:44.671722+0000","last_became_active":"2021-03-15T22:17:58.008731+0000","last_became_peered":"2021-03-15T22:17:58.008731+0000","last_unstale":"2021-03-15T22:22:44.671722+0000","last_undegraded":"2021-03-15T22:22:44.671722+0000","last_fullsized":"2021-03-15T22:22:44.671722+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:39.712183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:22:39.712183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"238'54","reported_seq":"583","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.056613+0000","last_change":"2021-03-15T22:22:46.057135+0000","last_active":"2021-03-15T22:22:46.056613+0000","last_peered":"2021-03-15T22:22:46.056613+0000","last_clean":"2021-03-15T22:22:46.056613+0000","last_became_active":"2021-03-15T22:21:28.045150+0000","last_became_peered":"2021-03-15T22:21:28.045150+0000","last_unstale":"2021-03-15T22:22:46.056613+0000","last_undegraded":"2021-03-15T22:22:46.056613+0000","last_fullsized":"2021-03-15T22:22:46.056613+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'42","last_scrub_stamp":"2021-03-15T22:22:05.296900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:05.296900+0000","log_size":54,"ondisk_log_size":54,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3697540,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":58,"num_read_kb":82219,"num_write":24,"num_write_kb":8677,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3663893,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"84","length":"1"},{"start":"95","length":"1"},{"start":"98","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"214","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.197492+0000","last_change":"2021-03-15T22:21:56.040799+0000","last_active":"2021-03-15T22:22:44.197492+0000","last_peered":"2021-03-15T22:22:44.197492+0000","last_clean":"2021-03-15T22:22:44.197492+0000","last_became_active":"2021-03-15T22:21:56.040442+0000","last_became_peered":"2021-03-15T22:21:56.040442+0000","last_unstale":"2021-03-15T22:22:44.197492+0000","last_undegraded":"2021-03-15T22:22:44.197492+0000","last_fullsized":"2021-03-15T22:22:44.197492+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:06.725499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:06.725499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"220","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.048571+0000","last_change":"2021-03-15T22:21:32.017870+0000","last_active":"2021-03-15T22:22:46.048571+0000","last_peered":"2021-03-15T22:22:46.048571+0000","last_clean":"2021-03-15T22:22:46.048571+0000","last_became_active":"2021-03-15T22:21:28.040070+0000","last_became_peered":"2021-03-15T22:21:28.040070+0000","last_unstale":"2021-03-15T22:22:46.048571+0000","last_undegraded":"2021-03-15T22:22:46.048571+0000","last_fullsized":"2021-03-15T22:22:46.048571+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:21:32.017806+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"239","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.052867+0000","last_change":"2021-03-15T22:21:21.773230+0000","last_active":"2021-03-15T22:22:46.052867+0000","last_peered":"2021-03-15T22:22:46.052867+0000","last_clean":"2021-03-15T22:22:46.052867+0000","last_became_active":"2021-03-15T22:17:55.992257+0000","last_became_peered":"2021-03-15T22:17:55.992257+0000","last_unstale":"2021-03-15T22:22:46.052867+0000","last_undegraded":"2021-03-15T22:22:46.052867+0000","last_fullsized":"2021-03-15T22:22:46.052867+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:21:21.773164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:21.773164+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"61'2","reported_seq":"266","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198206+0000","last_change":"2021-03-15T22:21:33.797042+0000","last_active":"2021-03-15T22:22:44.198206+0000","last_peered":"2021-03-15T22:22:44.198206+0000","last_clean":"2021-03-15T22:22:44.198206+0000","last_became_active":"2021-03-15T22:17:58.008973+0000","last_became_peered":"2021-03-15T22:17:58.008973+0000","last_unstale":"2021-03-15T22:22:44.198206+0000","last_undegraded":"2021-03-15T22:22:44.198206+0000","last_fullsized":"2021-03-15T22:22:44.198206+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:21:33.796953+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:33.796953+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"248'116","reported_seq":"615","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.053166+0000","last_change":"2021-03-15T22:22:46.053166+0000","last_active":"2021-03-15T22:22:46.053166+0000","last_peered":"2021-03-15T22:22:46.053166+0000","last_clean":"2021-03-15T22:22:46.053166+0000","last_became_active":"2021-03-15T22:21:56.039222+0000","last_became_peered":"2021-03-15T22:21:56.039222+0000","last_unstale":"2021-03-15T22:22:46.053166+0000","last_undegraded":"2021-03-15T22:22:46.053166+0000","last_fullsized":"2021-03-15T22:22:46.053166+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'97","last_scrub_stamp":"2021-03-15T22:21:54.579693+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:21:54.579693+0000","log_size":116,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":3428621,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":52,"num_read_kb":40150,"num_write":45,"num_write_kb":20938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3421879,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"84","length":"1"},{"start":"95","length":"1"},{"start":"98","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"211","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375313+0000","last_change":"2021-03-15T22:22:36.796288+0000","last_active":"2021-03-15T22:22:47.375313+0000","last_peered":"2021-03-15T22:22:47.375313+0000","last_clean":"2021-03-15T22:22:47.375313+0000","last_became_active":"2021-03-15T22:18:00.288441+0000","last_became_peered":"2021-03-15T22:18:00.288441+0000","last_unstale":"2021-03-15T22:22:47.375313+0000","last_undegraded":"2021-03-15T22:22:47.375313+0000","last_fullsized":"2021-03-15T22:22:47.375313+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:36.796228+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:36.796228+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"211","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198916+0000","last_change":"2021-03-15T22:21:52.836071+0000","last_active":"2021-03-15T22:22:44.198916+0000","last_peered":"2021-03-15T22:22:44.198916+0000","last_clean":"2021-03-15T22:22:44.198916+0000","last_became_active":"2021-03-15T22:17:53.976307+0000","last_became_peered":"2021-03-15T22:17:53.976307+0000","last_unstale":"2021-03-15T22:22:44.198916+0000","last_undegraded":"2021-03-15T22:22:44.198916+0000","last_fullsized":"2021-03-15T22:22:44.198916+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:52.835994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:52.835994+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"244","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040755+0000","last_change":"2021-03-15T22:21:03.362110+0000","last_active":"2021-03-15T22:22:45.040755+0000","last_peered":"2021-03-15T22:22:45.040755+0000","last_clean":"2021-03-15T22:22:45.040755+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:22:45.040755+0000","last_undegraded":"2021-03-15T22:22:45.040755+0000","last_fullsized":"2021-03-15T22:22:45.040755+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:21:03.362028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:03.362028+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"173'3","reported_seq":"125","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:47.057727+0000","last_change":"2021-03-15T22:21:54.890590+0000","last_active":"2021-03-15T22:22:47.057727+0000","last_peered":"2021-03-15T22:22:47.057727+0000","last_clean":"2021-03-15T22:22:47.057727+0000","last_became_active":"2021-03-15T22:21:28.048810+0000","last_became_peered":"2021-03-15T22:21:28.048810+0000","last_unstale":"2021-03-15T22:22:47.057727+0000","last_undegraded":"2021-03-15T22:22:47.057727+0000","last_fullsized":"2021-03-15T22:22:47.057727+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'3","last_scrub_stamp":"2021-03-15T22:21:54.890506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:54.890506+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.3","version":"253'208","reported_seq":"765","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.221833+0000","last_change":"2021-03-15T22:22:48.221833+0000","last_active":"2021-03-15T22:22:48.221833+0000","last_peered":"2021-03-15T22:22:48.221833+0000","last_clean":"2021-03-15T22:22:48.221833+0000","last_became_active":"2021-03-15T22:22:00.120085+0000","last_became_peered":"2021-03-15T22:22:00.120085+0000","last_unstale":"2021-03-15T22:22:48.221833+0000","last_undegraded":"2021-03-15T22:22:48.221833+0000","last_fullsized":"2021-03-15T22:22:48.221833+0000","mapping_epoch":204,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":205,"parent":"0.0","parent_split_bits":0,"last_scrub":"212'152","last_scrub_stamp":"2021-03-15T22:22:07.676468+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:07.676468+0000","log_size":208,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29477885,"num_objects":11,"num_object_clones":7,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":139,"num_read_kb":133521,"num_write":86,"num_write_kb":32442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":11287781,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"84","length":"1"},{"start":"91","length":"1"},{"start":"95","length":"1"},{"start":"98","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"216","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.039973+0000","last_change":"2021-03-15T22:21:28.038681+0000","last_active":"2021-03-15T22:22:45.039973+0000","last_peered":"2021-03-15T22:22:45.039973+0000","last_clean":"2021-03-15T22:22:45.039973+0000","last_became_active":"2021-03-15T22:21:28.038391+0000","last_became_peered":"2021-03-15T22:21:28.038391+0000","last_unstale":"2021-03-15T22:22:45.039973+0000","last_undegraded":"2021-03-15T22:22:45.039973+0000","last_fullsized":"2021-03-15T22:22:45.039973+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:00.367847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:00.367847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"69","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375675+0000","last_change":"2021-03-15T22:22:04.792632+0000","last_active":"2021-03-15T22:22:47.375675+0000","last_peered":"2021-03-15T22:22:47.375675+0000","last_clean":"2021-03-15T22:22:47.375675+0000","last_became_active":"2021-03-15T22:21:56.036359+0000","last_became_peered":"2021-03-15T22:21:56.036359+0000","last_unstale":"2021-03-15T22:22:47.375675+0000","last_undegraded":"2021-03-15T22:22:47.375675+0000","last_fullsized":"2021-03-15T22:22:47.375675+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:04.792551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:04.792551+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"190","reported_epoch":"252","state":"active+clean","last_fresh":"2021-03-15T22:22:47.375354+0000","last_change":"2021-03-15T22:21:29.174708+0000","last_active":"2021-03-15T22:22:47.375354+0000","last_peered":"2021-03-15T22:22:47.375354+0000","last_clean":"2021-03-15T22:22:47.375354+0000","last_became_active":"2021-03-15T22:21:28.047566+0000","last_became_peered":"2021-03-15T22:21:28.047566+0000","last_unstale":"2021-03-15T22:22:47.375354+0000","last_undegraded":"2021-03-15T22:22:47.375354+0000","last_fullsized":"2021-03-15T22:22:47.375354+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":174,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:20:55.778268+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:20:55.778268+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"218","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040624+0000","last_change":"2021-03-15T22:21:59.228515+0000","last_active":"2021-03-15T22:22:45.040624+0000","last_peered":"2021-03-15T22:22:45.040624+0000","last_clean":"2021-03-15T22:22:45.040624+0000","last_became_active":"2021-03-15T22:21:28.041912+0000","last_became_peered":"2021-03-15T22:21:28.041912+0000","last_unstale":"2021-03-15T22:22:45.040624+0000","last_undegraded":"2021-03-15T22:22:45.040624+0000","last_fullsized":"2021-03-15T22:22:45.040624+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:21:59.228452+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"248'146","reported_seq":"685","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.331070+0000","last_change":"2021-03-15T22:22:43.054882+0000","last_active":"2021-03-15T22:22:45.331070+0000","last_peered":"2021-03-15T22:22:45.331070+0000","last_clean":"2021-03-15T22:22:45.331070+0000","last_became_active":"2021-03-15T22:21:31.022979+0000","last_became_peered":"2021-03-15T22:21:31.022979+0000","last_unstale":"2021-03-15T22:22:45.331070+0000","last_undegraded":"2021-03-15T22:22:45.331070+0000","last_fullsized":"2021-03-15T22:22:45.331070+0000","mapping_epoch":175,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"193'121","last_scrub_stamp":"2021-03-15T22:21:51.825081+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:21:51.825081+0000","log_size":146,"ondisk_log_size":146,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9816573,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":88,"num_read_kb":76155,"num_write":59,"num_write_kb":19735,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":7516025,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"84","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"205","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.048338+0000","last_change":"2021-03-15T22:21:22.029900+0000","last_active":"2021-03-15T22:22:46.048338+0000","last_peered":"2021-03-15T22:22:46.048338+0000","last_clean":"2021-03-15T22:22:46.048338+0000","last_became_active":"2021-03-15T22:18:00.019457+0000","last_became_peered":"2021-03-15T22:18:00.019457+0000","last_unstale":"2021-03-15T22:22:46.048338+0000","last_undegraded":"2021-03-15T22:22:46.048338+0000","last_fullsized":"2021-03-15T22:22:46.048338+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:22.029836+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:21:22.029836+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"152'4","reported_seq":"287","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040599+0000","last_change":"2021-03-15T22:21:23.309388+0000","last_active":"2021-03-15T22:22:45.040599+0000","last_peered":"2021-03-15T22:22:45.040599+0000","last_clean":"2021-03-15T22:22:45.040599+0000","last_became_active":"2021-03-15T22:21:06.019218+0000","last_became_peered":"2021-03-15T22:21:06.019218+0000","last_unstale":"2021-03-15T22:22:45.040599+0000","last_undegraded":"2021-03-15T22:22:45.040599+0000","last_fullsized":"2021-03-15T22:22:45.040599+0000","mapping_epoch":60,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":151,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'4","last_scrub_stamp":"2021-03-15T22:21:23.309323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:23.309323+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"95","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.052525+0000","last_change":"2021-03-15T22:21:50.930666+0000","last_active":"2021-03-15T22:22:46.052525+0000","last_peered":"2021-03-15T22:22:46.052525+0000","last_clean":"2021-03-15T22:22:46.052525+0000","last_became_active":"2021-03-15T22:21:28.048995+0000","last_became_peered":"2021-03-15T22:21:28.048995+0000","last_unstale":"2021-03-15T22:22:46.052525+0000","last_undegraded":"2021-03-15T22:22:46.052525+0000","last_fullsized":"2021-03-15T22:22:46.052525+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:50.930571+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:21:50.930571+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"245","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.209563+0000","last_change":"2021-03-15T22:21:56.811802+0000","last_active":"2021-03-15T22:22:48.209563+0000","last_peered":"2021-03-15T22:22:48.209563+0000","last_clean":"2021-03-15T22:22:48.209563+0000","last_became_active":"2021-03-15T22:21:56.030199+0000","last_became_peered":"2021-03-15T22:21:56.030199+0000","last_unstale":"2021-03-15T22:22:48.209563+0000","last_undegraded":"2021-03-15T22:22:48.209563+0000","last_fullsized":"2021-03-15T22:22:48.209563+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:21:52.622751+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:52.622751+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"213'38","reported_seq":"538","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.331176+0000","last_change":"2021-03-15T22:22:43.053301+0000","last_active":"2021-03-15T22:22:45.331176+0000","last_peered":"2021-03-15T22:22:45.331176+0000","last_clean":"2021-03-15T22:22:45.331176+0000","last_became_active":"2021-03-15T22:20:05.039641+0000","last_became_peered":"2021-03-15T22:20:05.039641+0000","last_unstale":"2021-03-15T22:22:45.331176+0000","last_undegraded":"2021-03-15T22:22:45.331176+0000","last_fullsized":"2021-03-15T22:22:45.331176+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"209'34","last_scrub_stamp":"2021-03-15T22:22:05.628759+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:22:05.628759+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":58,"num_read_kb":62607,"num_write":15,"num_write_kb":6602,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"84","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"93","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.671916+0000","last_change":"2021-03-15T22:22:37.642100+0000","last_active":"2021-03-15T22:22:44.671916+0000","last_peered":"2021-03-15T22:22:44.671916+0000","last_clean":"2021-03-15T22:22:44.671916+0000","last_became_active":"2021-03-15T22:21:28.044508+0000","last_became_peered":"2021-03-15T22:21:28.044508+0000","last_unstale":"2021-03-15T22:22:44.671916+0000","last_undegraded":"2021-03-15T22:22:44.671916+0000","last_fullsized":"2021-03-15T22:22:44.671916+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:37.642038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:22:37.642038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"147'5","reported_seq":"253","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:47.057584+0000","last_change":"2021-03-15T22:21:55.110240+0000","last_active":"2021-03-15T22:22:47.057584+0000","last_peered":"2021-03-15T22:22:47.057584+0000","last_clean":"2021-03-15T22:22:47.057584+0000","last_became_active":"2021-03-15T22:21:02.017066+0000","last_became_peered":"2021-03-15T22:21:02.017066+0000","last_unstale":"2021-03-15T22:22:47.057584+0000","last_undegraded":"2021-03-15T22:22:47.057584+0000","last_fullsized":"2021-03-15T22:22:47.057584+0000","mapping_epoch":60,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'5","last_scrub_stamp":"2021-03-15T22:21:55.110127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:21:55.110127+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"213","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040696+0000","last_change":"2021-03-15T22:21:55.222551+0000","last_active":"2021-03-15T22:22:45.040696+0000","last_peered":"2021-03-15T22:22:45.040696+0000","last_clean":"2021-03-15T22:22:45.040696+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:22:45.040696+0000","last_undegraded":"2021-03-15T22:22:45.040696+0000","last_fullsized":"2021-03-15T22:22:45.040696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:21:55.222484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:21:55.222484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"238","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040504+0000","last_change":"2021-03-15T22:21:33.258467+0000","last_active":"2021-03-15T22:22:45.040504+0000","last_peered":"2021-03-15T22:22:45.040504+0000","last_clean":"2021-03-15T22:22:45.040504+0000","last_became_active":"2021-03-15T22:17:55.992213+0000","last_became_peered":"2021-03-15T22:17:55.992213+0000","last_unstale":"2021-03-15T22:22:45.040504+0000","last_undegraded":"2021-03-15T22:22:45.040504+0000","last_fullsized":"2021-03-15T22:22:45.040504+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:21:33.258382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:33.258382+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"253'57","reported_seq":"570","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.225059+0000","last_change":"2021-03-15T22:22:48.225059+0000","last_active":"2021-03-15T22:22:48.225059+0000","last_peered":"2021-03-15T22:22:48.225059+0000","last_clean":"2021-03-15T22:22:48.225059+0000","last_became_active":"2021-03-15T22:20:05.033636+0000","last_became_peered":"2021-03-15T22:20:05.033636+0000","last_unstale":"2021-03-15T22:22:48.225059+0000","last_undegraded":"2021-03-15T22:22:48.225059+0000","last_fullsized":"2021-03-15T22:22:48.225059+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"173'41","last_scrub_stamp":"2021-03-15T22:21:33.451818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:21:33.451818+0000","log_size":57,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2369820,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":61,"num_read_kb":83455,"num_write":23,"num_write_kb":5161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"84","length":"1"},{"start":"91","length":"1"},{"start":"95","length":"1"},{"start":"98","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"218","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.209668+0000","last_change":"2021-03-15T22:21:28.050618+0000","last_active":"2021-03-15T22:22:48.209668+0000","last_peered":"2021-03-15T22:22:48.209668+0000","last_clean":"2021-03-15T22:22:48.209668+0000","last_became_active":"2021-03-15T22:21:28.050311+0000","last_became_peered":"2021-03-15T22:21:28.050311+0000","last_unstale":"2021-03-15T22:22:48.209668+0000","last_undegraded":"2021-03-15T22:22:48.209668+0000","last_fullsized":"2021-03-15T22:22:48.209668+0000","mapping_epoch":172,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:58.074327+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:22.172605+0000","last_clean_scrub_stamp":"2021-03-15T22:20:58.074327+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,6],"acting":[2,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"231","reported_epoch":"251","state":"active+clean","last_fresh":"2021-03-15T22:22:46.051890+0000","last_change":"2021-03-15T22:22:03.851108+0000","last_active":"2021-03-15T22:22:46.051890+0000","last_peered":"2021-03-15T22:22:46.051890+0000","last_clean":"2021-03-15T22:22:46.051890+0000","last_became_active":"2021-03-15T22:21:14.032519+0000","last_became_peered":"2021-03-15T22:21:14.032519+0000","last_unstale":"2021-03-15T22:22:46.051890+0000","last_undegraded":"2021-03-15T22:22:46.051890+0000","last_fullsized":"2021-03-15T22:22:46.051890+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":159,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:22:03.851047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"214","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.671637+0000","last_change":"2021-03-15T22:22:38.664690+0000","last_active":"2021-03-15T22:22:44.671637+0000","last_peered":"2021-03-15T22:22:44.671637+0000","last_clean":"2021-03-15T22:22:44.671637+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:22:44.671637+0000","last_undegraded":"2021-03-15T22:22:44.671637+0000","last_fullsized":"2021-03-15T22:22:44.671637+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:38.664626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.664626+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"115'33","reported_seq":"259","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040706+0000","last_change":"2021-03-15T22:22:27.911298+0000","last_active":"2021-03-15T22:22:45.040706+0000","last_peered":"2021-03-15T22:22:45.040706+0000","last_clean":"2021-03-15T22:22:45.040706+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:22:45.040706+0000","last_undegraded":"2021-03-15T22:22:45.040706+0000","last_fullsized":"2021-03-15T22:22:45.040706+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'33","last_scrub_stamp":"2021-03-15T22:22:27.911223+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:22:27.911223+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"248'217","reported_seq":"779","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.054587+0000","last_change":"2021-03-15T22:22:43.209379+0000","last_active":"2021-03-15T22:22:45.054587+0000","last_peered":"2021-03-15T22:22:45.054587+0000","last_clean":"2021-03-15T22:22:45.054587+0000","last_became_active":"2021-03-15T22:21:59.034617+0000","last_became_peered":"2021-03-15T22:21:59.034617+0000","last_unstale":"2021-03-15T22:22:45.054587+0000","last_undegraded":"2021-03-15T22:22:45.054587+0000","last_fullsized":"2021-03-15T22:22:45.054587+0000","mapping_epoch":203,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":204,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:20:04.009088+0000","log_size":217,"ondisk_log_size":217,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13566957,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":130,"num_read_kb":114595,"num_write":97,"num_write_kb":36889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":7181439,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"84","length":"1"},{"start":"95","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"207","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040625+0000","last_change":"2021-03-15T22:22:35.877634+0000","last_active":"2021-03-15T22:22:45.040625+0000","last_peered":"2021-03-15T22:22:45.040625+0000","last_clean":"2021-03-15T22:22:45.040625+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:22:45.040625+0000","last_undegraded":"2021-03-15T22:22:45.040625+0000","last_fullsized":"2021-03-15T22:22:45.040625+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:35.877570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:22:35.877570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"155'4","reported_seq":"295","reported_epoch":"253","state":"active+clean","last_fresh":"2021-03-15T22:22:48.208895+0000","last_change":"2021-03-15T22:22:38.649104+0000","last_active":"2021-03-15T22:22:48.208895+0000","last_peered":"2021-03-15T22:22:48.208895+0000","last_clean":"2021-03-15T22:22:48.208895+0000","last_became_active":"2021-03-15T22:21:10.013838+0000","last_became_peered":"2021-03-15T22:21:10.013838+0000","last_unstale":"2021-03-15T22:22:48.208895+0000","last_undegraded":"2021-03-15T22:22:48.208895+0000","last_fullsized":"2021-03-15T22:22:48.208895+0000","mapping_epoch":60,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'4","last_scrub_stamp":"2021-03-15T22:22:38.649035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.649035+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"61","reported_epoch":"249","state":"active+clean","last_fresh":"2021-03-15T22:22:44.198267+0000","last_change":"2021-03-15T22:21:56.038426+0000","last_active":"2021-03-15T22:22:44.198267+0000","last_peered":"2021-03-15T22:22:44.198267+0000","last_clean":"2021-03-15T22:22:44.198267+0000","last_became_active":"2021-03-15T22:21:56.038093+0000","last_became_peered":"2021-03-15T22:21:56.038093+0000","last_unstale":"2021-03-15T22:22:44.198267+0000","last_undegraded":"2021-03-15T22:22:44.198267+0000","last_fullsized":"2021-03-15T22:22:44.198267+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:20:58.817670+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:20:58.817670+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"259","reported_epoch":"250","state":"active+clean","last_fresh":"2021-03-15T22:22:45.040316+0000","last_change":"2021-03-15T22:21:56.052881+0000","last_active":"2021-03-15T22:22:45.040316+0000","last_peered":"2021-03-15T22:22:45.040316+0000","last_clean":"2021-03-15T22:22:45.040316+0000","last_became_active":"2021-03-15T22:21:56.038572+0000","last_became_peered":"2021-03-15T22:21:56.038572+0000","last_unstale":"2021-03-15T22:22:45.040316+0000","last_undegraded":"2021-03-15T22:22:45.040316+0000","last_fullsized":"2021-03-15T22:22:45.040316+0000","mapping_epoch":200,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":201,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:21:04.375645+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:21:04.375645+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":177734875,"num_objects":89,"num_object_clones":47,"num_object_copies":267,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":89,"num_whiteouts":9,"num_read":1605,"num_read_kb":1564892,"num_write":1214,"num_write_kb":447960,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":85225199,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":412479488,"data_stored":553336491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2707,"ondisk_log_size":2707,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":295,"num_read_kb":290,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1064,"num_read_kb":569,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":171,"num_bytes_recovered":13292,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2048000,"data_stored":73560,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":586,"ondisk_log_size":586,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18,"ondisk_log_size":18,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299455,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":79756,"kb_used_data":60548,"kb_used_omap":0,"kb_used_meta":19200,"kb_avail":93665396,"statfs":{"total":95995035648,"available":95913365504,"internally_reserved":0,"allocated":62001152,"data_stored":77566716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19660800},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.505,"15min":0.501},"min":{"1min":0.26800000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.73,"5min":2.73,"15min":2.73},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.63500000000000001,"5min":0.52000000000000002,"15min":0.50800000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.8799999999999999,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.41599999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.58999999999999997,"5min":0.49399999999999999,"15min":0.48499999999999999},"min":{"1min":0.30299999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":2.6859999999999999,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.498,"15min":0.496},"min":{"1min":0.32900000000000001,"5min":0.307,"15min":0.307},"max":{"1min":1.877,"5min":1.877,"15min":1.877},"last":0.373}]},{"osd":2,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.47899999999999998,"15min":0.48199999999999998},"min":{"1min":0.32900000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":1.0029999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.51700000000000002,"15min":0.498},"min":{"1min":0.33700000000000002,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":2.5630000000000002,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.42299999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.53000000000000003,"15min":0.56200000000000006},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.95999999999999996,"5min":2.0339999999999998,"15min":2.0339999999999998},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.61499999999999999,"5min":0.55500000000000005,"15min":0.56499999999999995},"min":{"1min":0.35999999999999999,"5min":0.307,"15min":0.307},"max":{"1min":2.7949999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.46400000000000002,"15min":0.432},"min":{"1min":0.32800000000000001,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":1.835,"5min":1.835,"15min":1.835},"last":0.312},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.48399999999999999,"15min":0.442},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":2.5339999999999998,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.29799999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.48799999999999999,"15min":0.442},"min":{"1min":0.30299999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":2.5110000000000001,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52800000000000002,"15min":0.501},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.9870000000000001,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.34899999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:21:56 2021","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.53800000000000003,"15min":0.503},"min":{"1min":0.379,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.52900000000000003,"15min":0.502},"min":{"1min":0.38800000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.2250000000000001,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.36599999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528386,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":57020,"kb_used_data":39156,"kb_used_omap":0,"kb_used_meta":17856,"kb_avail":93688132,"statfs":{"total":95995035648,"available":95936647168,"internally_reserved":0,"allocated":40095744,"data_stored":50115171,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18284544},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:22:41 2021","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.48999999999999999,"15min":0.49299999999999999},"min":{"1min":0.30099999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.65500000000000003,"5min":0.995,"15min":0.995},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.48799999999999999,"15min":0.49099999999999999},"min":{"1min":0.29499999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.63200000000000001,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.27100000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:22:41 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.67200000000000004,"5min":1.05,"15min":1.05},"last":0.254},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.48599999999999999,"15min":0.47999999999999998},"min":{"1min":0.29799999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.70199999999999996,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.313}]},{"osd":2,"last update":"Mon Mar 15 22:22:41 2021","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51300000000000001,"15min":0.501},"min":{"1min":0.32000000000000001,"5min":0.312,"15min":0.312},"max":{"1min":0.72799999999999998,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.497,"15min":0.501},"min":{"1min":0.32800000000000001,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.77600000000000002,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.32900000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:22:41 2021","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.51400000000000001,"15min":0.52800000000000002},"min":{"1min":0.34499999999999997,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70699999999999996,"5min":1.026,"15min":1.026},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.52500000000000002,"15min":0.52700000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.71899999999999997,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.38300000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:22:41 2021","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.42999999999999999,"15min":0.40300000000000002},"min":{"1min":0.28000000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.68700000000000006,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.47899999999999998,"15min":0.442},"min":{"1min":0.26500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.754,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.36799999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:22:41 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.48399999999999999,"15min":0.44800000000000001},"min":{"1min":0.32300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.74299999999999999,"5min":1.135,"15min":1.135},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.50800000000000001,"15min":0.47399999999999998},"min":{"1min":0.35099999999999998,"5min":0.311,"15min":0.311},"max":{"1min":0.79000000000000004,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.39900000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:21:55 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.50700000000000001,"15min":0.498},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.752,"5min":0.79900000000000004,"15min":0.79900000000000004},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.54900000000000004,"15min":0.53500000000000003},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.33000000000000002},"max":{"1min":1.1659999999999999,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.44800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574932,"num_pgs":69,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":83920,"kb_used_data":63304,"kb_used_omap":0,"kb_used_meta":20608,"kb_avail":93661232,"statfs":{"total":95995035648,"available":95909101568,"internally_reserved":0,"allocated":64823296,"data_stored":88531746,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21102592},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:22:20 2021","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.89900000000000002,"15min":0.52300000000000002},"min":{"1min":0.19400000000000001,"5min":0.191,"15min":0.191},"max":{"1min":0.49199999999999999,"5min":54.427999999999997,"15min":54.427999999999997},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.91000000000000003,"15min":0.54600000000000004},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":1.23,"5min":54.356999999999999,"15min":54.356999999999999},"last":0.30499999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:22:43 2021","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.435,"15min":0.42099999999999999},"min":{"1min":0.29299999999999998,"5min":0.223,"15min":0.223},"max":{"1min":1.087,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.309,"5min":0.35699999999999998,"15min":0.40100000000000002},"min":{"1min":0.24399999999999999,"5min":0.215,"15min":0.215},"max":{"1min":0.48999999999999999,"5min":2.0760000000000001,"15min":2.0760000000000001},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:21:51 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41899999999999998,"15min":0.433},"min":{"1min":0.27000000000000002,"5min":0.222,"15min":0.222},"max":{"1min":1.393,"5min":1.796,"15min":1.796},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.433,"5min":0.44700000000000001,"15min":0.48299999999999998},"min":{"1min":0.23000000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":1.135,"5min":1.7150000000000001,"15min":1.7150000000000001},"last":0.371}]},{"osd":4,"last update":"Mon Mar 15 22:22:03 2021","interfaces":[{"interface":"back","average":{"1min":0.66400000000000003,"5min":0.57899999999999996,"15min":0.58299999999999996},"min":{"1min":0.35799999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":3.3439999999999999,"5min":3.3439999999999999,"15min":3.3439999999999999},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.80600000000000005,"5min":0.65300000000000002,"15min":0.63500000000000001},"min":{"1min":0.39500000000000002,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":4.6139999999999999,"5min":4.6139999999999999,"15min":4.6139999999999999},"last":0.59599999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:22:20 2021","interfaces":[{"interface":"back","average":{"1min":0.58399999999999996,"5min":0.58499999999999996,"15min":0.64700000000000002},"min":{"1min":0.42899999999999999,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.81999999999999995,"5min":2.153,"15min":2.153},"last":0.66500000000000004},{"interface":"front","average":{"1min":0.60699999999999998,"5min":0.71999999999999997,"15min":1.101},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":1.1950000000000001,"5min":15.472,"15min":15.472},"last":0.67500000000000004}]},{"osd":6,"last update":"Mon Mar 15 22:22:43 2021","interfaces":[{"interface":"back","average":{"1min":0.65700000000000003,"5min":0.621,"15min":0.67500000000000004},"min":{"1min":0.377,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.276,"5min":1.75,"15min":1.75},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.59699999999999998,"15min":0.63800000000000001},"min":{"1min":0.41599999999999998,"5min":0.34999999999999998,"15min":0.34999999999999998},"max":{"1min":0.80900000000000005,"5min":1.3580000000000001,"15min":1.3580000000000001},"last":0.69799999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:21:57 2021","interfaces":[{"interface":"back","average":{"1min":0.621,"5min":0.77500000000000002,"15min":0.94899999999999995},"min":{"1min":0.38900000000000001,"5min":0.316,"15min":0.316},"max":{"1min":1.081,"5min":8.8100000000000005,"15min":8.8100000000000005},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59999999999999998,"15min":0.63300000000000001},"min":{"1min":0.41899999999999998,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.82699999999999996,"5min":1.325,"15min":1.325},"last":0.73499999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":69616,"kb_used_data":49576,"kb_used_omap":0,"kb_used_meta":20032,"kb_avail":93675536,"statfs":{"total":95995035648,"available":95923748864,"internally_reserved":0,"allocated":50765824,"data_stored":59872395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20512768},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:22:19 2021","interfaces":[{"interface":"back","average":{"1min":0.32400000000000001,"5min":0.34000000000000002,"15min":0.35699999999999998},"min":{"1min":0.23799999999999999,"5min":0.188,"15min":0.188},"max":{"1min":0.47099999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.36899999999999999,"15min":0.375},"min":{"1min":0.24299999999999999,"5min":0.217,"15min":0.217},"max":{"1min":0.54200000000000004,"5min":0.63500000000000001,"15min":0.63800000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:22:31 2021","interfaces":[{"interface":"back","average":{"1min":0.38600000000000001,"5min":0.36899999999999999,"15min":0.40100000000000002},"min":{"1min":0.29199999999999998,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.58699999999999997,"5min":0.71299999999999997,"15min":0.71299999999999997},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.34899999999999998,"15min":0.38900000000000001},"min":{"1min":0.26900000000000002,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.625,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.26100000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:21:57 2021","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.38100000000000001,"15min":0.42499999999999999},"min":{"1min":0.249,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.51800000000000002,"5min":0.69699999999999995,"15min":0.69699999999999995},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39200000000000002,"5min":0.40200000000000002,"15min":0.437},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.57999999999999996,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33900000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:22:08 2021","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.54000000000000004,"15min":0.56399999999999995},"min":{"1min":0.314,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.70399999999999996,"5min":0.96799999999999997,"15min":0.96799999999999997},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.58799999999999997,"15min":0.60999999999999999},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.69099999999999995,"5min":1.034,"15min":1.034},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Mar 15 22:22:23 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.57699999999999996,"15min":0.60799999999999998},"min":{"1min":0.36899999999999999,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.72999999999999998,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.626},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.55900000000000005,"15min":0.58299999999999996},"min":{"1min":0.39400000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.70899999999999996,"5min":1.0149999999999999,"15min":1.0149999999999999},"last":0.68999999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:22:36 2021","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.56799999999999995,"15min":0.57799999999999996},"min":{"1min":0.39700000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.76400000000000001,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.61199999999999999,"15min":0.625},"min":{"1min":0.45800000000000002,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.78200000000000003,"5min":1.0049999999999999,"15min":1.0049999999999999},"last":0.70499999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:22:02 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.58699999999999997,"15min":0.57899999999999996},"min":{"1min":0.36199999999999999,"5min":0.36199999999999999,"15min":0.36199999999999999},"max":{"1min":0.79300000000000004,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.66600000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58499999999999996,"15min":0.57699999999999996},"min":{"1min":0.38,"5min":0.38,"15min":0.38},"max":{"1min":0.73999999999999999,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.57099999999999995}]}]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":99512,"kb_used_data":79472,"kb_used_omap":0,"kb_used_meta":20032,"kb_avail":93645640,"statfs":{"total":95995035648,"available":95893135360,"internally_reserved":0,"allocated":81379328,"data_stored":113194578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20512768},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:22:37 2021","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.35099999999999998,"15min":0.35899999999999999},"min":{"1min":0.193,"5min":0.189,"15min":0.189},"max":{"1min":0.442,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.39000000000000001,"15min":0.38700000000000001},"min":{"1min":0.23599999999999999,"5min":0.191,"15min":0.191},"max":{"1min":0.54900000000000004,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.39600000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:22:37 2021","interfaces":[{"interface":"back","average":{"1min":0.34399999999999997,"5min":0.44,"15min":0.40600000000000003},"min":{"1min":0.28299999999999997,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.47499999999999998,"5min":5.4489999999999998,"15min":5.4489999999999998},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.36099999999999999,"5min":0.52200000000000002,"15min":0.47999999999999998},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.495,"5min":11.659000000000001,"15min":11.659000000000001},"last":0.38}]},{"osd":3,"last update":"Mon Mar 15 22:21:48 2021","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.40999999999999998,"15min":0.442},"min":{"1min":0.23499999999999999,"5min":0.22,"15min":0.22},"max":{"1min":2.0640000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.443,"15min":0.502},"min":{"1min":0.27300000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:22:05 2021","interfaces":[{"interface":"back","average":{"1min":0.63900000000000001,"5min":0.57999999999999996,"15min":0.58599999999999997},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.9850000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.505},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.55600000000000005,"15min":0.56699999999999995},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.34899999999999998},"max":{"1min":1.8100000000000001,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.495}]},{"osd":5,"last update":"Mon Mar 15 22:22:32 2021","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.57499999999999996,"15min":0.58299999999999996},"min":{"1min":0.35399999999999998,"5min":0.31900000000000001,"15min":0.31900000000000001},"max":{"1min":0.73799999999999999,"5min":1.954,"15min":1.954},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.59099999999999997,"15min":0.57399999999999995},"min":{"1min":0.38200000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.82099999999999995,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.48899999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:22:37 2021","interfaces":[{"interface":"back","average":{"1min":0.57499999999999996,"5min":0.59299999999999997,"15min":0.57099999999999995},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77000000000000002,"5min":1.875,"15min":1.875},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57499999999999996,"15min":0.56999999999999995},"min":{"1min":0.372,"5min":0.372,"15min":0.372},"max":{"1min":0.76300000000000001,"5min":1.609,"15min":1.609},"last":0.53700000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:22:01 2021","interfaces":[{"interface":"back","average":{"1min":0.63100000000000001,"5min":0.57299999999999995,"15min":0.55600000000000005},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":1.7190000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.621},{"interface":"front","average":{"1min":0.64600000000000002,"5min":0.59699999999999998,"15min":0.58599999999999997},"min":{"1min":0.441,"5min":0.38700000000000001,"15min":0.38700000000000001},"max":{"1min":1.923,"5min":1.923,"15min":1.923},"last":0.55500000000000005}]}]},{"osd":3,"up_from":0,"seq":111669149772,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":4,"up_from":31,"seq":133143986249,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":70356,"kb_used_data":51660,"kb_used_omap":0,"kb_used_meta":18688,"kb_avail":93674796,"statfs":{"total":95995035648,"available":95922991104,"internally_reserved":0,"allocated":52899840,"data_stored":71142731,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19136512},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:22:10 2021","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.47099999999999997,"15min":0.51000000000000001},"min":{"1min":0.33500000000000002,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.74399999999999999,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32900000000000001,"5min":0.251,"15min":0.251},"max":{"1min":0.65500000000000003,"5min":0.74199999999999999,"15min":0.74199999999999999},"last":0.61799999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:22:10 2021","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.48599999999999999,"15min":0.51200000000000001},"min":{"1min":0.376,"5min":0.23599999999999999,"15min":0.23599999999999999},"max":{"1min":0.69499999999999995,"5min":1.631,"15min":1.631},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52400000000000002,"15min":0.53000000000000003},"min":{"1min":0.36199999999999999,"5min":0.26800000000000002,"15min":0.26800000000000002},"max":{"1min":0.67800000000000005,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Mar 15 22:22:10 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.48999999999999999,"15min":0.51800000000000002},"min":{"1min":0.36599999999999999,"5min":0.313,"15min":0.313},"max":{"1min":0.624,"5min":0.76100000000000001,"15min":0.76100000000000001},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.502,"15min":0.51000000000000001},"min":{"1min":0.34599999999999997,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.66400000000000003,"5min":1.1910000000000001,"15min":1.1910000000000001},"last":0.55200000000000005}]},{"osd":3,"last update":"Mon Mar 15 22:22:10 2021","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.54200000000000004,"15min":0.56200000000000006},"min":{"1min":0.42299999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70299999999999996,"5min":0.98799999999999999,"15min":0.98799999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.53800000000000003,"15min":0.57499999999999996},"min":{"1min":0.41599999999999998,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69199999999999995,"5min":0.99399999999999999,"15min":0.99399999999999999},"last":0.627}]},{"osd":5,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.441,"15min":0.40400000000000003},"min":{"1min":0.32100000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67000000000000004,"5min":0.72299999999999998,"15min":0.72299999999999998},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.46800000000000003,"15min":0.42199999999999999},"min":{"1min":0.313,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.79200000000000004,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.37}]},{"osd":6,"last update":"Mon Mar 15 22:22:43 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.46400000000000002,"15min":0.42799999999999999},"min":{"1min":0.35899999999999999,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.72599999999999998,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54200000000000004,"15min":0.50600000000000001},"min":{"1min":0.443,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78400000000000003,"5min":0.84799999999999998,"15min":0.84799999999999998},"last":0.46600000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:21:53 2021","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.50600000000000001,"15min":0.48299999999999998},"min":{"1min":0.39200000000000002,"5min":0.316,"15min":0.316},"max":{"1min":0.73799999999999999,"5min":0.80600000000000005,"15min":0.80600000000000005},"last":0.443},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.53500000000000003,"15min":0.51900000000000002},"min":{"1min":0.38600000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":0.81200000000000006,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.43099999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757318,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85604,"kb_used_data":66204,"kb_used_omap":0,"kb_used_meta":19392,"kb_avail":93659548,"statfs":{"total":95995035648,"available":95907377152,"internally_reserved":0,"allocated":67792896,"data_stored":89794117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19857408},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48499999999999999,"15min":0.48099999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.67800000000000005,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.38},{"interface":"front","average":{"1min":0.53600000000000003,"5min":0.50600000000000001,"15min":0.502},"min":{"1min":0.28299999999999997,"5min":0.26100000000000001,"15min":0.26100000000000001},"max":{"1min":0.72199999999999998,"5min":0.755,"15min":0.755},"last":0.59699999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.51200000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.68700000000000006,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.50900000000000001,"15min":0.50800000000000001},"min":{"1min":0.36399999999999999,"5min":0.317,"15min":0.317},"max":{"1min":0.72099999999999997,"5min":0.83999999999999997,"15min":0.83999999999999997},"last":0.55700000000000005}]},{"osd":2,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.497,"15min":0.504},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.67600000000000005,"5min":1.006,"15min":1.006},"last":0.505},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.52800000000000002,"15min":0.51400000000000001},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.746,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.51600000000000001,"15min":0.50600000000000001},"min":{"1min":0.38,"5min":0.308,"15min":0.308},"max":{"1min":0.74199999999999999,"5min":0.97199999999999998,"15min":0.97199999999999998},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.54700000000000004,"15min":0.53200000000000003},"min":{"1min":0.34100000000000003,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.76900000000000002,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.52200000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:22:26 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.44500000000000001,"15min":0.38700000000000001},"min":{"1min":0.28199999999999997,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.73099999999999998,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.48199999999999998,"15min":0.42199999999999999},"min":{"1min":0.35199999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.65500000000000003,"5min":1.0269999999999999,"15min":1.0269999999999999},"last":0.54000000000000004}]},{"osd":6,"last update":"Mon Mar 15 22:22:42 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.49399999999999999,"15min":0.46400000000000002},"min":{"1min":0.34100000000000003,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.72799999999999998,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.57899999999999996,"5min":0.56100000000000005,"15min":0.52700000000000002},"min":{"1min":0.435,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.71599999999999997,"5min":1.327,"15min":1.327},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:21:55 2021","interfaces":[{"interface":"back","average":{"1min":0.56899999999999995,"5min":0.53200000000000003,"15min":0.50600000000000001},"min":{"1min":0.34699999999999998,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.3,"5min":1.3,"15min":1.3},"last":0.61399999999999999},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54400000000000004,"15min":0.52100000000000002},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.879,"5min":0.879,"15min":0.879},"last":0.623}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":7530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":15174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":241664,"data_stored":8670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":307200,"data_stored":11162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":204800,"data_stored":7262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":12852,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":221184,"data_stored":7922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":47857664,"data_stored":58159437,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":63242240,"data_stored":90031222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":78409728,"data_stored":111448511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12046336,"data_stored":11973832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49893376,"data_stored":69419181,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":64909312,"data_stored":88075383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":37068800,"data_stored":48399312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":59052032,"data_stored":75829613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:22:51.861 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2021-03-15T22:22:51.861 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:22:51.861 INFO:tasks.thrashosds.thrasher:Adding osd 3 2021-03-15T22:22:51.862 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd in 3 2021-03-15T22:22:51.869 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:51.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:51.871 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:51.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:51.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:51.994 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:51.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:51.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:22:52.054 INFO:tasks.rados.rados.0.smithi076.stdout:815: done (5 left) 2021-03-15T22:22:52.055 INFO:tasks.rados.rados.0.smithi076.stdout:816: write oid 40 current snap is 83 2021-03-15T22:22:52.055 INFO:tasks.rados.rados.0.smithi076.stdout:816: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2021-03-15T22:22:52.055 INFO:tasks.rados.rados.0.smithi076.stdout:816: writing smithi07694878-40 from 443784 to 1091945 tid 1 2021-03-15T22:22:52.062 INFO:tasks.rados.rados.0.smithi076.stdout:816: writing smithi07694878-40 from 1657676 to 2317451 tid 2 2021-03-15T22:22:52.068 INFO:tasks.rados.rados.0.smithi076.stdout:816: writing smithi07694878-40 from 2818209 to 3243784 tid 3 2021-03-15T22:22:52.068 INFO:tasks.rados.rados.0.smithi076.stdout:817: snap_create 2021-03-15T22:22:52.069 INFO:tasks.rados.rados.0.smithi076.stdout:812: expect (ObjNum 199 snap 43 seq_num 199) 2021-03-15T22:22:52.095 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:52.117 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:52.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:52.119 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:52.165 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:52.220 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:52.236 INFO:tasks.rados.rados.0.smithi076.stdout:811: finishing rollback tid 0 to smithi07694878-19 2021-03-15T22:22:52.236 INFO:tasks.rados.rados.0.smithi076.stdout:810: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:22:52.236 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 0 (ObjNum 1902433216 snap 32564 seq_num 1902433216) dirty dne 2021-03-15T22:22:52.236 INFO:tasks.rados.rados.0.smithi076.stdout:811: finishing rollback tid 1 to smithi07694878-19 2021-03-15T22:22:52.237 INFO:tasks.rados.rados.0.smithi076.stdout:811: finishing rollback tid 2 to smithi07694878-19 2021-03-15T22:22:52.237 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 307 (ObjNum 273 snap 72 seq_num 273) dirty exists 2021-03-15T22:22:52.237 INFO:tasks.rados.rados.0.smithi076.stdout:813: finishing rollback tid 0 to smithi07694878-37 2021-03-15T22:22:52.237 INFO:tasks.rados.rados.0.smithi076.stdout:813: finishing rollback tid 1 to smithi07694878-37 2021-03-15T22:22:52.237 INFO:tasks.rados.rados.0.smithi076.stdout:813: finishing rollback tid 2 to smithi07694878-37 2021-03-15T22:22:52.238 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 448 (ObjNum 249 snap 62 seq_num 249) dirty exists 2021-03-15T22:22:52.238 INFO:tasks.rados.rados.0.smithi076.stdout:814: expect (ObjNum 277 snap 74 seq_num 277) 2021-03-15T22:22:52.246 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:52.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:52.248 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:52.283 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:22:52.283 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:22:52.330 INFO:tasks.rados.rados.0.smithi076.stdout:816: finishing write tid 1 to smithi07694878-40 2021-03-15T22:22:52.331 INFO:tasks.rados.rados.0.smithi076.stdout:816: finishing write tid 2 to smithi07694878-40 2021-03-15T22:22:52.331 INFO:tasks.rados.rados.0.smithi076.stdout:816: finishing write tid 3 to smithi07694878-40 2021-03-15T22:22:52.331 INFO:tasks.rados.rados.0.smithi076.stdout:816: finishing write tid 4 to smithi07694878-40 2021-03-15T22:22:52.331 INFO:tasks.rados.rados.0.smithi076.stdout:816: finishing write tid 5 to smithi07694878-40 2021-03-15T22:22:52.332 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 455 (ObjNum 312 snap 83 seq_num 312) dirty exists 2021-03-15T22:22:52.332 INFO:tasks.rados.rados.0.smithi076.stdout:816: left oid 40 (ObjNum 312 snap 83 seq_num 312) 2021-03-15T22:22:52.350 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:52.359 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:52 smithi125 conmon[56739]: cluster 2021-03-15T22:22:51.015579+0000 2021-03-15T22:22:52.359 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:52 smithi125 conmon[56739]: mgr.y (mgr.14144) 496 : cluster [DBG] pgmap v510: 138 pgs: 138 active+clean; 170 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 693 KiB/s wr, 16 op/s 2021-03-15T22:22:52.360 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:52 smithi125 conmon[56739]: cluster 2021-03-15T22:22:51.251515+0000 osd.7 (osd.7) 56 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:52.360 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:52 smithi125 conmon[56739]: cluster 2021-03-15T22:22:51.252154+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:52.360 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:52 smithi125 conmon[56739]: cluster 2021-03-15T22:22:52.043017+0000 mon.a (mon.0) 929 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2021-03-15T22:22:52.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:52.374 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:52.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:52.375 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:52.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[56514]: cluster 2021-03-15T22:22:51.015579+0000 2021-03-15T22:22:52.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[56514]: mgr.y (mgr.14144) 496 : cluster [DBG] pgmap v510: 138 pgs: 138 active+clean; 170 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 693 KiB/s wr, 16 op/s 2021-03-15T22:22:52.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[56514]: cluster 2021-03-15T22:22:51.251515+0000 osd.7 (osd.7) 56 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:52.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[56514]: cluster 2021-03-15T22:22:51.252154+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:52.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[56514]: cluster 2021-03-15T22:22:52.043017+0000 mon.a (mon.0) 929 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2021-03-15T22:22:52.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[62156]: cluster 2021-03-15T22:22:51.015579+0000 mgr.y (mgr.14144) 496 2021-03-15T22:22:52.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[62156]: : cluster [DBG] pgmap v510: 138 pgs: 138 active+clean; 170 MiB data, 535 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 693 KiB/s wr, 16 op/s 2021-03-15T22:22:52.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[62156]: cluster 2021-03-15T22:22:51.251515+0000 osd.7 (osd.7) 56 : cluster [DBG] 2021-03-15T22:22:52.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[62156]: purged_snaps scrub starts 2021-03-15T22:22:52.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:22:52.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[62156]: 22:51.252154+0000 osd.7 (osd.7) 57 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:52.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:22:52.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[62156]: 15T22:22:52.043017+0000 mon.a 2021-03-15T22:22:52.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:52 smithi076 conmon[62156]: (mon.0) 929 : cluster [DBG] osdmap e257: 8 total, 8 up, 7 in 2021-03-15T22:22:52.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:52.497 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:52.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:52.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:52.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:52.622 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:52.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:52.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:52.724 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:52.746 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:52.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:52.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:52.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:52.869 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:52.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:52.871 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:52.946 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-15T22:22:52.973 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:52.996 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:52.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:52.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:53.045 INFO:tasks.rados.rados.0.smithi076.stdout:810: done (6 left) 2021-03-15T22:22:53.046 INFO:tasks.rados.rados.0.smithi076.stdout:811: done (5 left) 2021-03-15T22:22:53.046 INFO:tasks.rados.rados.0.smithi076.stdout:812: done (4 left) 2021-03-15T22:22:53.046 INFO:tasks.rados.rados.0.smithi076.stdout:813: done (3 left) 2021-03-15T22:22:53.046 INFO:tasks.rados.rados.0.smithi076.stdout:814: done (2 left) 2021-03-15T22:22:53.047 INFO:tasks.rados.rados.0.smithi076.stdout:816: done (1 left) 2021-03-15T22:22:53.047 INFO:tasks.rados.rados.0.smithi076.stdout:817: done (0 left) 2021-03-15T22:22:53.047 INFO:tasks.rados.rados.0.smithi076.stdout:818: rollback oid 15 current snap is 84 2021-03-15T22:22:53.047 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 15 to 83 2021-03-15T22:22:53.048 INFO:tasks.rados.rados.0.smithi076.stdout:819: write oid 31 current snap is 84 2021-03-15T22:22:53.048 INFO:tasks.rados.rados.0.smithi076.stdout:819: seq_num 313 ranges {460591=723410,1716158=403433,2729279=440140} 2021-03-15T22:22:53.051 INFO:tasks.rados.rados.0.smithi076.stdout:819: writing smithi07694878-31 from 460591 to 1184001 tid 1 2021-03-15T22:22:53.057 INFO:tasks.rados.rados.0.smithi076.stdout:819: writing smithi07694878-31 from 1716158 to 2119591 tid 2 2021-03-15T22:22:53.063 INFO:tasks.rados.rados.0.smithi076.stdout:819: writing smithi07694878-31 from 2729279 to 3169419 tid 3 2021-03-15T22:22:53.063 INFO:tasks.rados.rados.0.smithi076.stdout:820: write (excl) oid 10 current snap is 84 2021-03-15T22:22:53.063 INFO:tasks.rados.rados.0.smithi076.stdout:820: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2021-03-15T22:22:53.070 INFO:tasks.rados.rados.0.smithi076.stdout:820: writing smithi07694878-10 from 477398 to 1276057 tid 1 2021-03-15T22:22:53.077 INFO:tasks.rados.rados.0.smithi076.stdout:820: writing smithi07694878-10 from 1774640 to 2438084 tid 2 2021-03-15T22:22:53.082 INFO:tasks.rados.rados.0.smithi076.stdout:820: writing smithi07694878-10 from 2873055 to 3277398 tid 3 2021-03-15T22:22:53.083 INFO:tasks.rados.rados.0.smithi076.stdout:821: rollback oid 46 current snap is 84 2021-03-15T22:22:53.083 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 46 to 70 2021-03-15T22:22:53.083 INFO:tasks.rados.rados.0.smithi076.stdout:818: finishing rollback tid 1 to smithi07694878-15 2021-03-15T22:22:53.083 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 0 (ObjNum 1902431744 snap 32564 seq_num 1902431744) dirty dne 2021-03-15T22:22:53.083 INFO:tasks.rados.rados.0.smithi076.stdout:819: finishing write tid 1 to smithi07694878-31 2021-03-15T22:22:53.084 INFO:tasks.rados.rados.0.smithi076.stdout:819: finishing write tid 2 to smithi07694878-31 2021-03-15T22:22:53.084 INFO:tasks.rados.rados.0.smithi076.stdout:819: finishing write tid 3 to smithi07694878-31 2021-03-15T22:22:53.084 INFO:tasks.rados.rados.0.smithi076.stdout:818: done (3 left) 2021-03-15T22:22:53.084 INFO:tasks.rados.rados.0.smithi076.stdout:822: read oid 8 snap -1 2021-03-15T22:22:53.085 INFO:tasks.rados.rados.0.smithi076.stdout:822: expect (ObjNum 197 snap 43 seq_num 197) 2021-03-15T22:22:53.085 INFO:tasks.rados.rados.0.smithi076.stdout:819: finishing write tid 4 to smithi07694878-31 2021-03-15T22:22:53.085 INFO:tasks.rados.rados.0.smithi076.stdout:819: finishing write tid 5 to smithi07694878-31 2021-03-15T22:22:53.085 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 155 (ObjNum 313 snap 84 seq_num 313) dirty exists 2021-03-15T22:22:53.085 INFO:tasks.rados.rados.0.smithi076.stdout:819: left oid 31 (ObjNum 313 snap 84 seq_num 313) 2021-03-15T22:22:53.086 INFO:tasks.rados.rados.0.smithi076.stdout:820: finishing write tid 1 to smithi07694878-10 2021-03-15T22:22:53.086 INFO:tasks.rados.rados.0.smithi076.stdout:820: finishing write tid 2 to smithi07694878-10 2021-03-15T22:22:53.086 INFO:tasks.rados.rados.0.smithi076.stdout:823: read oid 24 snap 82 2021-03-15T22:22:53.086 INFO:tasks.rados.rados.0.smithi076.stdout:823: expect deleted 2021-03-15T22:22:53.087 INFO:tasks.rados.rados.0.smithi076.stdout:819: done (4 left) 2021-03-15T22:22:53.087 INFO:tasks.rados.rados.0.smithi076.stdout:824: delete oid 42 current snap is 84 2021-03-15T22:22:53.087 INFO:tasks.rados.rados.0.smithi076.stdout:821: finishing rollback tid 1 to smithi07694878-46 2021-03-15T22:22:53.087 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:22:53.088 INFO:tasks.rados.rados.0.smithi076.stdout:821: done (4 left) 2021-03-15T22:22:53.088 INFO:tasks.rados.rados.0.smithi076.stdout:823: done (3 left) 2021-03-15T22:22:53.088 INFO:tasks.rados.rados.0.smithi076.stdout:824: done (2 left) 2021-03-15T22:22:53.089 INFO:tasks.rados.rados.0.smithi076.stdout:825: read oid 28 snap -1 2021-03-15T22:22:53.089 INFO:tasks.rados.rados.0.smithi076.stdout:825: expect (ObjNum 307 snap 79 seq_num 307) 2021-03-15T22:22:53.089 INFO:tasks.rados.rados.0.smithi076.stdout:826: snap_remove snap 75 2021-03-15T22:22:53.099 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:53.120 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:53.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:53.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:53.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:53.228 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:22:53.247 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:53.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:53.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:53.349 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:53.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:53.374 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:53.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:53.475 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:53.480 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:53 smithi125 conmon[56739]: cluster 2021-03-15T22:22:52.759079+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:53.481 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:53 smithi125 conmon[56739]: cluster 2021-03-15T22:22:52.759366+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:53.481 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:53 smithi125 conmon[56739]: cluster 2021-03-15T22:22:53.043597+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2021-03-15T22:22:53.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:53 smithi076 conmon[62156]: cluster 2021-03-15T22:22:52.759079+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:53.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:53 smithi076 conmon[62156]: cluster 2021-03-15T22:22:52.759366+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:53.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:53 smithi076 conmon[62156]: cluster 2021-03-15T22:22:53.043597+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2021-03-15T22:22:53.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:53 smithi076 conmon[56514]: cluster 2021-03-15T22:22:52.759079+0000 osd.0 (osd.0) 55 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:53.498 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:53 smithi076 conmon[56514]: cluster 2021-03-15T22:22:52.759366+0000 osd.0 (osd.0) 56 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:53.498 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:53 smithi076 conmon[56514]: cluster 2021-03-15T22:22:53.043597+0000 mon.a (mon.0) 930 : cluster [DBG] osdmap e258: 8 total, 8 up, 7 in 2021-03-15T22:22:53.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:53.500 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:53.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:53.601 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:53.624 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:53.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:53.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:22:53.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:53.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:53.750 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:53.751 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:53.852 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:53.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:53.877 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:53.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:53.979 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:54.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:54.003 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:54.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:54.049 INFO:tasks.rados.rados.0.smithi076.stdout:826: done (3 left) 2021-03-15T22:22:54.049 INFO:tasks.rados.rados.0.smithi076.stdout:827: snap_create 2021-03-15T22:22:54.050 INFO:tasks.rados.rados.0.smithi076.stdout:822: expect (ObjNum 197 snap 43 seq_num 197) 2021-03-15T22:22:54.105 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:54.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:54.130 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:54.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:54.217 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:54.232 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:54.240 INFO:tasks.rados.rados.0.smithi076.stdout:820: finishing write tid 3 to smithi07694878-10 2021-03-15T22:22:54.241 INFO:tasks.rados.rados.0.smithi076.stdout:820: finishing write tid 4 to smithi07694878-10 2021-03-15T22:22:54.241 INFO:tasks.rados.rados.0.smithi076.stdout:820: finishing write tid 5 to smithi07694878-10 2021-03-15T22:22:54.241 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 132 (ObjNum 314 snap 84 seq_num 314) dirty exists 2021-03-15T22:22:54.241 INFO:tasks.rados.rados.0.smithi076.stdout:820: left oid 10 (ObjNum 314 snap 84 seq_num 314) 2021-03-15T22:22:54.241 INFO:tasks.rados.rados.0.smithi076.stdout:825: expect (ObjNum 307 snap 79 seq_num 307) 2021-03-15T22:22:54.254 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:54.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:54.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:54.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:54.384 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:54.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:54.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:54 smithi076 conmon[56514]: cluster 2021-03-15T22:22:53.016093+0000 mgr.y (mgr.14144) 497 : 2021-03-15T22:22:54.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:54 smithi076 conmon[56514]: cluster [DBG] pgmap v513: 138 pgs: 138 active+clean; 177 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 17 op/s 2021-03-15T22:22:54.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:54 smithi076 conmon[56514]: cluster 2021-03-15T22:22:54.045314+0000 mon.a (mon.0) 931 : cluster [DBG] osdmap e259: 8 total, 8 up, 7 in 2021-03-15T22:22:54.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:54 smithi076 conmon[62156]: cluster 2021-03-15T22:22:53 2021-03-15T22:22:54.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:54 smithi076 conmon[62156]: .016093+0000 mgr.y (mgr.14144) 497 : cluster [DBG] pgmap v513: 138 pgs: 138 active+clean; 177 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 17 op/s 2021-03-15T22:22:54.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:54 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:22:54.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:54 smithi076 conmon[62156]: -03-15T22:22:54.045314 2021-03-15T22:22:54.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:54 smithi076 conmon[62156]: +0000 mon.a (mon.0) 931 : 2021-03-15T22:22:54.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:54 smithi076 conmon[62156]: cluster [DBG] osdmap e259: 8 total, 8 up, 7 in 2021-03-15T22:22:54.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:54.514 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:54.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:54.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:54.530 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:54 smithi125 conmon[56739]: cluster 2021-03-15T22:22:53.016093+0000 mgr.y (mgr.14144) 2021-03-15T22:22:54.530 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:54 smithi125 conmon[56739]: 497 : cluster [DBG] pgmap v513: 138 pgs: 138 active+clean; 177 MiB data, 548 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 17 op/s 2021-03-15T22:22:54.531 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:54 smithi125 conmon[56739]: cluster 2021-03-15T22:22:54.045314+0000 mon.a (mon.0) 931 : cluster [DBG] osdmap e259: 8 total, 8 up, 7 in 2021-03-15T22:22:54.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:54.639 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:54.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:54.640 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:54.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:54.759 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:22:54.759 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:22:54.765 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:54.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:54.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:54.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:54.891 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:54.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:54.941 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:54 smithi076 conmon[56514]: debug 2021-03-15T22:22:54.678+0000 7f0537c88700 -1 mon.a@0(leader).osd e259 definitely_dead 0 2021-03-15T22:22:54.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:55.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:55.015 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:55.015 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:55.052 INFO:tasks.rados.rados.0.smithi076.stdout:820: done (3 left) 2021-03-15T22:22:55.053 INFO:tasks.rados.rados.0.smithi076.stdout:822: done (2 left) 2021-03-15T22:22:55.053 INFO:tasks.rados.rados.0.smithi076.stdout:825: done (1 left) 2021-03-15T22:22:55.053 INFO:tasks.rados.rados.0.smithi076.stdout:827: done (0 left) 2021-03-15T22:22:55.054 INFO:tasks.rados.rados.0.smithi076.stdout:828: write (excl) oid 46 current snap is 85 2021-03-15T22:22:55.054 INFO:tasks.rados.rados.0.smithi076.stdout:828: seq_num 316 ranges {511012=549157,1775251=550760,3095195=215817} 2021-03-15T22:22:55.056 INFO:tasks.rados.rados.0.smithi076.stdout:828: writing smithi07694878-46 from 511012 to 1060169 tid 1 2021-03-15T22:22:55.061 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:22:55.064 INFO:tasks.rados.rados.0.smithi076.stdout:828: writing smithi07694878-46 from 1775251 to 2326011 tid 2 2021-03-15T22:22:55.071 INFO:tasks.rados.rados.0.smithi076.stdout:828: writing smithi07694878-46 from 3095195 to 3311012 tid 3 2021-03-15T22:22:55.071 INFO:tasks.rados.rados.0.smithi076.stdout:829: copy_from oid 23 from oid 45 current snap is 85 2021-03-15T22:22:55.071 INFO:tasks.rados.rados.0.smithi076.stdout:828: finishing write tid 1 to smithi07694878-46 2021-03-15T22:22:55.072 INFO:tasks.rados.rados.0.smithi076.stdout:828: finishing write tid 2 to smithi07694878-46 2021-03-15T22:22:55.072 INFO:tasks.rados.rados.0.smithi076.stdout:830: snap_remove snap 65 2021-03-15T22:22:55.073 INFO:teuthology.orchestra.run.smithi076.stderr:marked in osd.3. 2021-03-15T22:22:55.117 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:55.139 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:55.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:55.140 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:55.241 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:55.263 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:55.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:55.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:55.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:55.387 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:55.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:55.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:55.403 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-15T22:22:55.490 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: audit 2021-03-15T22:22:54.679141+0000 2021-03-15T22:22:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: mon.a (mon.0) 932 : audit [DBG] from='client.? 172.21.15.76:0/817811409' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2021-03-15T22:22:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: audit 2021-03-15T22:22:54.909152+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.76:0/3777667015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:22:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: audit 2021-03-15T22:22:54.909507+0000 mon.a (mon.0) 933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:22:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: cluster 2021-03-15T22:22:55.037628+0000 mon.a (mon.0) 934 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: audit 2021-03-15T22:22:55. 2021-03-15T22:22:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: 041812+0000 mon.a (mon.0) 935 : audit [INF] from='client.? 172.21.15.76:0/817811409' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2021-03-15T22:22:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: audit 2021-03-15T22:22:55.041882+0000 mon.a (mon. 2021-03-15T22:22:55.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: 0) 936 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:22:55.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: cluster 2021-03-15T22:22:55.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: 2021-03-15T22:22:55.041934+0000 mon.a (mon. 2021-03-15T22:22:55.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[56514]: 0) 937 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2021-03-15T22:22:55.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: audit 2021-03-15T22:22:54.679141+0000 mon.a ( 2021-03-15T22:22:55.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: mon.0) 932 : audit [DBG] from='client.? 172.21.15.76:0/817811409' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2021-03-15T22:22:55.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: audit 2021-03-15T22:22:54.909152+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.76:0/3777667015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:22:55.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: audit 2021-03-15T22:22:54.909507+0000 mon.a (mon.0) 933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:22:55.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:22:55.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: 15T22:22:55.037628+0000 mon.a (mon.0) 934 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:55.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:22:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: 22:55.041812+0000 mon.a (mon.0) 2021-03-15T22:22:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: 935 : audit [INF] from='client.? 172.21.15.76:0/817811409' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2021-03-15T22:22:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:22:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: 22:55.041882+0000 mon.a (mon.0 2021-03-15T22:22:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: ) 936 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:22:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:22:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: 22:22:55.041934+0000 mon.a (mon. 2021-03-15T22:22:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:55 smithi076 conmon[62156]: 0) 937 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2021-03-15T22:22:55.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:55 smithi125 conmon[56739]: audit 2021-03-15T22:22:54.679141+0000 mon.a (mon.0) 932 : audit [DBG] from='client.? 172.21.15.76:0/817811409' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2021-03-15T22:22:55.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:55 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:22:55.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:55 smithi125 conmon[56739]: 15T22:22:54.909152+0000 mon.c (mon.1) 24 : audit [INF] from='client.? 172.21.15.76:0/3777667015' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:22:55.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:55 smithi125 conmon[56739]: audit 2021-03-15T22:22:54.909507+0000 mon.a (mon.0) 933 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:22:55.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:55 smithi125 conmon[56739]: cluster 2021-03-15T22:22:55.037628+0000 mon.a (mon.0) 934 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:22:55.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:55 smithi125 conmon[56739]: audit 2021-03-15T22:22:55.041812+0000 mon.a (mon.0) 935 : audit [INF] from='client.? 172.21.15.76:0/817811409' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2021-03-15T22:22:55.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:55 smithi125 conmon[56739]: audit 2021-03-15T22:22:55.041882+0000 mon.a (mon.0) 936 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:22:55.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:55 smithi125 conmon[56739]: cluster 2021-03-15T22:22:55.041934+0000 mon.a (mon.0) 937 : cluster [DBG] osdmap e260: 8 total, 8 up, 8 in 2021-03-15T22:22:55.561 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:55.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:55.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:55.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:55.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:55.692 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:55.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:55.717 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:22:55.794 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:55.816 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:55.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:55.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:55.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:55.941 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:55.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:55.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:56.044 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:56.058 INFO:tasks.rados.rados.0.smithi076.stdout:830: done (2 left) 2021-03-15T22:22:56.059 INFO:tasks.rados.rados.0.smithi076.stdout:831: rollback oid 10 current snap is 85 2021-03-15T22:22:56.059 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 10 to 72 2021-03-15T22:22:56.059 INFO:tasks.rados.rados.0.smithi076.stdout:832: write (excl) oid 21 current snap is 85 2021-03-15T22:22:56.059 INFO:tasks.rados.rados.0.smithi076.stdout:832: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2021-03-15T22:22:56.061 INFO:tasks.rados.rados.0.smithi076.stdout:832: writing smithi07694878-21 from 527819 to 1152225 tid 1 2021-03-15T22:22:56.068 INFO:tasks.rados.rados.0.smithi076.stdout:832: writing smithi07694878-21 from 1833733 to 2244504 tid 2 2021-03-15T22:22:56.077 INFO:tasks.rados.rados.0.smithi076.stdout:832: writing smithi07694878-21 from 2722618 to 3324199 tid 3 2021-03-15T22:22:56.077 INFO:tasks.rados.rados.0.smithi076.stdout:828: finishing write tid 3 to smithi07694878-46 2021-03-15T22:22:56.078 INFO:tasks.rados.rados.0.smithi076.stdout:828: finishing write tid 4 to smithi07694878-46 2021-03-15T22:22:56.078 INFO:tasks.rados.rados.0.smithi076.stdout:828: finishing write tid 5 to smithi07694878-46 2021-03-15T22:22:56.078 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 194 (ObjNum 316 snap 85 seq_num 316) dirty exists 2021-03-15T22:22:56.078 INFO:tasks.rados.rados.0.smithi076.stdout:828: left oid 46 (ObjNum 316 snap 85 seq_num 316) 2021-03-15T22:22:56.079 INFO:tasks.rados.rados.0.smithi076.stdout:831: finishing rollback tid 0 to smithi07694878-10 2021-03-15T22:22:56.080 INFO:tasks.rados.rados.0.smithi076.stdout:831: finishing rollback tid 1 to smithi07694878-10 2021-03-15T22:22:56.080 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 134 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:22:56.081 INFO:tasks.rados.rados.0.smithi076.stdout:828: done (3 left) 2021-03-15T22:22:56.081 INFO:tasks.rados.rados.0.smithi076.stdout:831: done (2 left) 2021-03-15T22:22:56.081 INFO:tasks.rados.rados.0.smithi076.stdout:833: copy_from oid 49 from oid 15 current snap is 85 2021-03-15T22:22:56.081 INFO:tasks.rados.rados.0.smithi076.stdout:834: read oid 46 snap 84 2021-03-15T22:22:56.081 INFO:tasks.rados.rados.0.smithi076.stdout:834: expect deleted 2021-03-15T22:22:56.082 INFO:tasks.rados.rados.0.smithi076.stdout:835: rollback oid 48 current snap is 85 2021-03-15T22:22:56.082 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 48 to 76 2021-03-15T22:22:56.082 INFO:tasks.rados.rados.0.smithi076.stdout:836: read oid 20 snap -1 2021-03-15T22:22:56.083 INFO:tasks.rados.rados.0.smithi076.stdout:836: expect (ObjNum 298 snap 78 seq_num 298) 2021-03-15T22:22:56.084 INFO:tasks.rados.rados.0.smithi076.stdout:837: write (excl) oid 36 current snap is 85 2021-03-15T22:22:56.084 INFO:tasks.rados.rados.0.smithi076.stdout:837: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2021-03-15T22:22:56.085 INFO:tasks.rados.rados.0.smithi076.stdout:837: writing smithi07694878-36 from 544626 to 1244281 tid 1 2021-03-15T22:22:56.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:56.086 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:56.087 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:56.093 INFO:tasks.rados.rados.0.smithi076.stdout:837: writing smithi07694878-36 from 1892215 to 2446644 tid 2 2021-03-15T22:22:56.098 INFO:tasks.rados.rados.0.smithi076.stdout:837: writing smithi07694878-36 from 3150041 to 3344626 tid 3 2021-03-15T22:22:56.099 INFO:tasks.rados.rados.0.smithi076.stdout:838: write oid 43 current snap is 85 2021-03-15T22:22:56.099 INFO:tasks.rados.rados.0.smithi076.stdout:838: seq_num 319 ranges {561433=774904,1950697=414440,2777464=583969} 2021-03-15T22:22:56.106 INFO:tasks.rados.rados.0.smithi076.stdout:838: writing smithi07694878-43 from 561433 to 1336337 tid 1 2021-03-15T22:22:56.111 INFO:tasks.rados.rados.0.smithi076.stdout:838: writing smithi07694878-43 from 1950697 to 2365137 tid 2 2021-03-15T22:22:56.117 INFO:tasks.rados.rados.0.smithi076.stdout:838: writing smithi07694878-43 from 2777464 to 3361433 tid 3 2021-03-15T22:22:56.118 INFO:tasks.rados.rados.0.smithi076.stdout:839: snap_create 2021-03-15T22:22:56.118 INFO:tasks.rados.rados.0.smithi076.stdout:833: finishing copy_from to smithi07694878-49 2021-03-15T22:22:56.118 INFO:tasks.rados.rados.0.smithi076.stdout:833: got expected ENOENT (src dne) 2021-03-15T22:22:56.118 INFO:tasks.rados.rados.0.smithi076.stdout:833: finishing copy_from racing read to smithi07694878-49 2021-03-15T22:22:56.119 INFO:tasks.rados.rados.0.smithi076.stdout:835: finishing rollback tid 1 to smithi07694878-48 2021-03-15T22:22:56.119 INFO:tasks.rados.rados.0.smithi076.stdout:835: finishing rollback tid 2 to smithi07694878-48 2021-03-15T22:22:56.119 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 160 (ObjNum 253 snap 63 seq_num 253) dirty exists 2021-03-15T22:22:56.119 INFO:tasks.rados.rados.0.smithi076.stdout:836: expect (ObjNum 298 snap 78 seq_num 298) 2021-03-15T22:22:56.188 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:56.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:56.211 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:56.211 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:56.267 INFO:tasks.rados.rados.0.smithi076.stdout:838: finishing write tid 1 to smithi07694878-43 2021-03-15T22:22:56.267 INFO:tasks.rados.rados.0.smithi076.stdout:838: finishing write tid 2 to smithi07694878-43 2021-03-15T22:22:56.267 INFO:tasks.rados.rados.0.smithi076.stdout:838: finishing write tid 3 to smithi07694878-43 2021-03-15T22:22:56.267 INFO:tasks.rados.rados.0.smithi076.stdout:838: finishing write tid 4 to smithi07694878-43 2021-03-15T22:22:56.268 INFO:tasks.rados.rados.0.smithi076.stdout:838: finishing write tid 5 to smithi07694878-43 2021-03-15T22:22:56.268 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 149 (ObjNum 319 snap 85 seq_num 319) dirty exists 2021-03-15T22:22:56.268 INFO:tasks.rados.rados.0.smithi076.stdout:838: left oid 43 (ObjNum 319 snap 85 seq_num 319) 2021-03-15T22:22:56.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[56514]: cluster 2021-03-15T22:22:56.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[56514]: 2021-03-15T22:22:55.016863+0000 mgr.y (mgr.14144) 498 : cluster [DBG] pgmap v516: 138 pgs: 138 active+clean; 180 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 5.9 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-03-15T22:22:56.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[56514]: cluster 2021-03-15T22:22:56.046006+0000 2021-03-15T22:22:56.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[56514]: mon.a (mon.0) 938 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2021-03-15T22:22:56.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:22:56.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[62156]: 22:55.016863+0000 mgr.y (mgr.14144) 498 : cluster [DBG] pgmap v516: 138 pgs: 138 active+clean; 180 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 5.9 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-03-15T22:22:56.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[62156]: cluster 2021-03-15T22:22:56.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[62156]: 2021-03 2021-03-15T22:22:56.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[62156]: -15T22:22:56 2021-03-15T22:22:56.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[62156]: .046006+0000 mon.a (mon. 2021-03-15T22:22:56.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[62156]: 0) 938 : cluster [DBG] 2021-03-15T22:22:56.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:56 smithi076 conmon[62156]: osdmap e261: 8 total, 8 up, 8 in 2021-03-15T22:22:56.313 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:56.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:56.337 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:56.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:56.438 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:56.460 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:56.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:56.462 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:56 smithi125 conmon[56739]: cluster 2021-03-15T22:22:55.016863+0000 mgr.y (mgr.14144) 498 : cluster [DBG] pgmap v516: 138 pgs: 138 active+clean; 180 MiB data, 553 MiB used, 625 GiB / 626 GiB avail; 5.9 MiB/s rd, 2.1 MiB/s wr, 13 op/s 2021-03-15T22:22:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:56 smithi125 conmon[56739]: cluster 2021-03-15T22:22:56.046006+0000 mon.a (mon.0) 938 : cluster [DBG] osdmap e261: 8 total, 8 up, 8 in 2021-03-15T22:22:56.563 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:56.584 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:56.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:56.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:56.687 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:56.713 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:56.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:56.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:56.768 INFO:tasks.thrashosds.thrasher:Added osd 3 2021-03-15T22:22:56.816 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:56.839 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:56.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:56.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:56.942 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:56.965 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:56.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:56.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:57.050 INFO:tasks.rados.rados.0.smithi076.stdout:833: done (8 left) 2021-03-15T22:22:57.051 INFO:tasks.rados.rados.0.smithi076.stdout:834: done (7 left) 2021-03-15T22:22:57.051 INFO:tasks.rados.rados.0.smithi076.stdout:835: done (6 left) 2021-03-15T22:22:57.051 INFO:tasks.rados.rados.0.smithi076.stdout:836: done (5 left) 2021-03-15T22:22:57.052 INFO:tasks.rados.rados.0.smithi076.stdout:838: done (4 left) 2021-03-15T22:22:57.052 INFO:tasks.rados.rados.0.smithi076.stdout:839: done (3 left) 2021-03-15T22:22:57.052 INFO:tasks.rados.rados.0.smithi076.stdout:840: read oid 1 snap -1 2021-03-15T22:22:57.052 INFO:tasks.rados.rados.0.smithi076.stdout:840: expect (ObjNum 280 snap 74 seq_num 280) 2021-03-15T22:22:57.052 INFO:tasks.rados.rados.0.smithi076.stdout:841: read oid 25 snap 63 2021-03-15T22:22:57.053 INFO:tasks.rados.rados.0.smithi076.stdout:841: expect deleted 2021-03-15T22:22:57.053 INFO:tasks.rados.rados.0.smithi076.stdout:842: read oid 8 snap -1 2021-03-15T22:22:57.053 INFO:tasks.rados.rados.0.smithi076.stdout:842: expect (ObjNum 197 snap 43 seq_num 197) 2021-03-15T22:22:57.053 INFO:tasks.rados.rados.0.smithi076.stdout:843: copy_from oid 9 from oid 2 current snap is 86 2021-03-15T22:22:57.054 INFO:tasks.rados.rados.0.smithi076.stdout:844: rollback oid 7 current snap is 86 2021-03-15T22:22:57.054 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 7 to 72 2021-03-15T22:22:57.054 INFO:tasks.rados.rados.0.smithi076.stdout:845: read oid 27 snap -1 2021-03-15T22:22:57.054 INFO:tasks.rados.rados.0.smithi076.stdout:845: expect deleted 2021-03-15T22:22:57.055 INFO:tasks.rados.rados.0.smithi076.stdout:846: snap_remove snap 85 2021-03-15T22:22:57.068 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:57.092 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:57.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:57.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:57.164 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:22:57.180 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:22:57.194 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:57.198 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:22:57.214 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:22:57.216 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:57.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:57.218 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:57.232 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:22:57.251 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:22:57.269 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:22:57.287 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:22:57.288 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:22:57.288 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:22:57.288 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:22:57.289 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:22:57.289 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:22:57.289 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:22:57.289 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:22:57.289 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:22:57.319 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:57.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:57.343 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:57.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:57.359 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:22:57.360 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:22:57.360 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:22:57.360 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:22:57.409 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:57 smithi125 conmon[56739]: cluster 2021-03-15T22:22:55.743154+0000 osd.6 (osd.6 2021-03-15T22:22:57.409 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:57 smithi125 conmon[56739]: ) 40 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:22:57.410 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:57 smithi125 conmon[56739]: cluster 2021-03-15T22:22:55.893644+0000 osd.4 (osd.4) 60 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:22:57.410 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:57 smithi125 conmon[56739]: cluster 2021-03-15T22:22:57.046151+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2021-03-15T22:22:57.410 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:57 smithi125 conmon[56739]: audit 2021-03-15T22:22:57.144929+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.76:0/471198645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:57.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:57.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:57.468 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:57.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:57.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:57 smithi076 conmon[62156]: cluster 2021-03-15T22:22:55.743154+0000 osd.6 (osd.6) 2021-03-15T22:22:57.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:57 smithi076 conmon[62156]: 40 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:22:57.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:57 smithi076 conmon[62156]: cluster 2021-03-15T22:22:55.893644+0000 osd.4 (osd.4) 60 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:22:57.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:57 smithi076 conmon[62156]: cluster 2021-03-15T22:22:57.046151+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2021-03-15T22:22:57.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:57 smithi076 conmon[62156]: audit 2021-03-15T22:22:57.144929+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.76:0/471198645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:57.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:57 smithi076 conmon[56514]: cluster 2021-03-15T22:22:55.743154+0000 osd.6 ( 2021-03-15T22:22:57.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:57 smithi076 conmon[56514]: osd.6) 40 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:22:57.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:57 smithi076 conmon[56514]: cluster 2021-03-15T22:22:55.893644+0000 osd.4 (osd.4) 60 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:22:57.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:57 smithi076 conmon[56514]: cluster 2021-03-15T22:22:57.046151+0000 mon.a (mon.0) 939 : cluster [DBG] osdmap e262: 8 total, 8 up, 8 in 2021-03-15T22:22:57.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:57 smithi076 conmon[56514]: audit 2021-03-15T22:22:57.144929+0000 mon.c (mon.1) 25 : audit [DBG] from='client.? 172.21.15.76:0/471198645' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:22:57.570 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:57.592 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:57.594 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:57.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:57.717 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:57.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:57.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:57.820 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:57.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:57.846 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:57.847 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:57.948 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:57.971 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-15T22:22:57.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:57.972 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:57.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:58.050 INFO:tasks.rados.rados.0.smithi076.stdout:846: done (9 left) 2021-03-15T22:22:58.051 INFO:tasks.rados.rados.0.smithi076.stdout:847: rollback oid 40 current snap is 86 2021-03-15T22:22:58.051 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 40 to 81 2021-03-15T22:22:58.051 INFO:tasks.rados.rados.0.smithi076.stdout:848: rollback oid 50 current snap is 86 2021-03-15T22:22:58.051 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 50 to 70 2021-03-15T22:22:58.052 INFO:tasks.rados.rados.0.smithi076.stdout:849: read oid 13 snap -1 2021-03-15T22:22:58.052 INFO:tasks.rados.rados.0.smithi076.stdout:849: expect (ObjNum 287 snap 77 seq_num 287) 2021-03-15T22:22:58.052 INFO:tasks.rados.rados.0.smithi076.stdout:850: write oid 24 current snap is 86 2021-03-15T22:22:58.053 INFO:tasks.rados.rados.0.smithi076.stdout:850: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2021-03-15T22:22:58.057 INFO:tasks.rados.rados.0.smithi076.stdout:850: writing smithi07694878-24 from 578240 to 1144746 tid 1 2021-03-15T22:22:58.063 INFO:tasks.rados.rados.0.smithi076.stdout:850: writing smithi07694878-24 from 1609179 to 2167277 tid 2 2021-03-15T22:22:58.071 INFO:tasks.rados.rados.0.smithi076.stdout:850: writing smithi07694878-24 from 2804887 to 3356637 tid 3 2021-03-15T22:22:58.072 INFO:tasks.rados.rados.0.smithi076.stdout:851: read oid 43 snap -1 2021-03-15T22:22:58.072 INFO:tasks.rados.rados.0.smithi076.stdout:851: expect (ObjNum 319 snap 85 seq_num 319) 2021-03-15T22:22:58.072 INFO:tasks.rados.rados.0.smithi076.stdout:852: read oid 44 snap 63 2021-03-15T22:22:58.073 INFO:tasks.rados.rados.0.smithi076.stdout:852: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:22:58.073 INFO:tasks.rados.rados.0.smithi076.stdout:845: done (14 left) 2021-03-15T22:22:58.073 INFO:tasks.rados.rados.0.smithi076.stdout:853: snap_create 2021-03-15T22:22:58.074 INFO:tasks.rados.rados.0.smithi076.stdout:842: expect (ObjNum 197 snap 43 seq_num 197) 2021-03-15T22:22:58.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:58.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:58.098 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:58.099 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:58.200 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:58.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:58.222 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:58.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:58.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:58 smithi076 conmon[56514]: cluster 2021-03-15T22:22:56.701069+0000 osd.6 (osd.6) 41 : cluster [DBG] 4.a scrub ok 2021-03-15T22:22:58.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:58 smithi076 conmon[56514]: cluster 2021-03-15T22:22:57.017314+0000 mgr.y (mgr.14144) 499 : cluster [DBG] pgmap v519: 138 pgs: 3 peering, 135 active+clean; 185 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 835 B/s, 9 objects/s recovering 2021-03-15T22:22:58.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:58 smithi076 conmon[56514]: cluster 2021-03-15T22:22:58.046194+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2021-03-15T22:22:58.243 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:58 smithi125 conmon[56739]: cluster 2021-03-15T22:22:56.701069+0000 osd.6 (osd.6) 41 : cluster [DBG] 4.a scrub ok 2021-03-15T22:22:58.243 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:58 smithi125 conmon[56739]: cluster 2021-03-15T22:22:57.017314+0000 mgr.y (mgr.14144) 499 : cluster [DBG] pgmap v519: 138 pgs: 3 peering, 135 active+clean; 185 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 835 B/s, 9 objects/s recovering 2021-03-15T22:22:58.243 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:58 smithi125 conmon[56739]: cluster 2021-03-15T22:22:58.046194+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2021-03-15T22:22:58.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:58 smithi076 conmon[62156]: cluster 2021-03-15T22:22:56.701069+0000 osd.6 (osd.6) 41 : cluster [DBG] 4.a scrub ok 2021-03-15T22:22:58.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:58 smithi076 conmon[62156]: cluster 2021-03-15T22:22:57.017314+0000 mgr.y (mgr.14144) 499 : cluster [DBG] pgmap v519: 138 pgs: 3 peering, 135 active+clean; 185 MiB data, 584 MiB used, 715 GiB / 715 GiB avail; 8.2 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 835 B/s, 9 objects/s recovering 2021-03-15T22:22:58.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:58 smithi076 conmon[62156]: cluster 2021-03-15T22:22:58.046194+0000 mon.a (mon.0) 940 : cluster [DBG] osdmap e263: 8 total, 8 up, 8 in 2021-03-15T22:22:58.259 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:22:58.259 INFO:tasks.rados.rados.0.smithi076.stdout:840: expect (ObjNum 280 snap 74 seq_num 280) 2021-03-15T22:22:58.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:58.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:58.347 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:58.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:58.376 INFO:tasks.rados.rados.0.smithi076.stdout:843: finishing copy_from to smithi07694878-9 2021-03-15T22:22:58.377 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 128 (ObjNum 295 snap 78 seq_num 295) dirty exists 2021-03-15T22:22:58.377 INFO:tasks.rados.rados.0.smithi076.stdout:843: finishing copy_from racing read to smithi07694878-9 2021-03-15T22:22:58.377 INFO:tasks.rados.rados.0.smithi076.stdout:837: finishing write tid 1 to smithi07694878-36 2021-03-15T22:22:58.377 INFO:tasks.rados.rados.0.smithi076.stdout:829: finishing copy_from to smithi07694878-23 2021-03-15T22:22:58.378 INFO:tasks.rados.rados.0.smithi076.stdout:829: got expected ENOENT (src dne) 2021-03-15T22:22:58.378 INFO:tasks.rados.rados.0.smithi076.stdout:837: finishing write tid 2 to smithi07694878-36 2021-03-15T22:22:58.378 INFO:tasks.rados.rados.0.smithi076.stdout:829: finishing copy_from racing read to smithi07694878-23 2021-03-15T22:22:58.378 INFO:tasks.rados.rados.0.smithi076.stdout:837: finishing write tid 3 to smithi07694878-36 2021-03-15T22:22:58.379 INFO:tasks.rados.rados.0.smithi076.stdout:837: finishing write tid 4 to smithi07694878-36 2021-03-15T22:22:58.379 INFO:tasks.rados.rados.0.smithi076.stdout:837: finishing write tid 5 to smithi07694878-36 2021-03-15T22:22:58.379 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 402 (ObjNum 318 snap 85 seq_num 318) dirty exists 2021-03-15T22:22:58.379 INFO:tasks.rados.rados.0.smithi076.stdout:837: left oid 36 (ObjNum 318 snap 85 seq_num 318) 2021-03-15T22:22:58.380 INFO:tasks.rados.rados.0.smithi076.stdout:832: finishing write tid 1 to smithi07694878-21 2021-03-15T22:22:58.380 INFO:tasks.rados.rados.0.smithi076.stdout:844: finishing rollback tid 0 to smithi07694878-7 2021-03-15T22:22:58.380 INFO:tasks.rados.rados.0.smithi076.stdout:832: finishing write tid 2 to smithi07694878-21 2021-03-15T22:22:58.380 INFO:tasks.rados.rados.0.smithi076.stdout:832: finishing write tid 3 to smithi07694878-21 2021-03-15T22:22:58.380 INFO:tasks.rados.rados.0.smithi076.stdout:832: finishing write tid 4 to smithi07694878-21 2021-03-15T22:22:58.381 INFO:tasks.rados.rados.0.smithi076.stdout:832: finishing write tid 5 to smithi07694878-21 2021-03-15T22:22:58.381 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 465 (ObjNum 317 snap 85 seq_num 317) dirty exists 2021-03-15T22:22:58.381 INFO:tasks.rados.rados.0.smithi076.stdout:832: left oid 21 (ObjNum 317 snap 85 seq_num 317) 2021-03-15T22:22:58.381 INFO:tasks.rados.rados.0.smithi076.stdout:844: finishing rollback tid 1 to smithi07694878-7 2021-03-15T22:22:58.381 INFO:tasks.rados.rados.0.smithi076.stdout:844: finishing rollback tid 2 to smithi07694878-7 2021-03-15T22:22:58.382 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 319 (ObjNum 271 snap 72 seq_num 271) dirty exists 2021-03-15T22:22:58.382 INFO:tasks.rados.rados.0.smithi076.stdout:848: finishing rollback tid 0 to smithi07694878-50 2021-03-15T22:22:58.382 INFO:tasks.rados.rados.0.smithi076.stdout:847: finishing rollback tid 0 to smithi07694878-40 2021-03-15T22:22:58.382 INFO:tasks.rados.rados.0.smithi076.stdout:850: finishing write tid 1 to smithi07694878-24 2021-03-15T22:22:58.383 INFO:tasks.rados.rados.0.smithi076.stdout:850: finishing write tid 2 to smithi07694878-24 2021-03-15T22:22:58.383 INFO:tasks.rados.rados.0.smithi076.stdout:847: finishing rollback tid 1 to smithi07694878-40 2021-03-15T22:22:58.383 INFO:tasks.rados.rados.0.smithi076.stdout:847: finishing rollback tid 2 to smithi07694878-40 2021-03-15T22:22:58.383 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 469 (ObjNum 305 snap 78 seq_num 305) dirty exists 2021-03-15T22:22:58.383 INFO:tasks.rados.rados.0.smithi076.stdout:852: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:22:58.449 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:58.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:58.472 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:58.472 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:58.524 INFO:tasks.rados.rados.0.smithi076.stdout:850: finishing write tid 3 to smithi07694878-24 2021-03-15T22:22:58.525 INFO:tasks.rados.rados.0.smithi076.stdout:850: finishing write tid 4 to smithi07694878-24 2021-03-15T22:22:58.525 INFO:tasks.rados.rados.0.smithi076.stdout:850: finishing write tid 5 to smithi07694878-24 2021-03-15T22:22:58.525 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 61 (ObjNum 320 snap 86 seq_num 320) dirty exists 2021-03-15T22:22:58.526 INFO:tasks.rados.rados.0.smithi076.stdout:850: left oid 24 (ObjNum 320 snap 86 seq_num 320) 2021-03-15T22:22:58.526 INFO:tasks.rados.rados.0.smithi076.stdout:851: expect (ObjNum 319 snap 85 seq_num 319) 2021-03-15T22:22:58.574 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:22:58.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:58.598 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:58.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:22:58.699 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:22:58.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:58.723 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:58.723 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:22:58.733 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:22:58.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:22:58.846 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:58.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:58.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:22:58.949 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:58.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:58.974 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:58.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:59.050 INFO:tasks.rados.rados.0.smithi076.stdout:829: done (14 left) 2021-03-15T22:22:59.050 INFO:tasks.rados.rados.0.smithi076.stdout:832: done (13 left) 2021-03-15T22:22:59.051 INFO:tasks.rados.rados.0.smithi076.stdout:837: done (12 left) 2021-03-15T22:22:59.051 INFO:tasks.rados.rados.0.smithi076.stdout:840: done (11 left) 2021-03-15T22:22:59.051 INFO:tasks.rados.rados.0.smithi076.stdout:841: done (10 left) 2021-03-15T22:22:59.051 INFO:tasks.rados.rados.0.smithi076.stdout:842: done (9 left) 2021-03-15T22:22:59.051 INFO:tasks.rados.rados.0.smithi076.stdout:843: done (8 left) 2021-03-15T22:22:59.052 INFO:tasks.rados.rados.0.smithi076.stdout:844: done (7 left) 2021-03-15T22:22:59.052 INFO:tasks.rados.rados.0.smithi076.stdout:847: done (6 left) 2021-03-15T22:22:59.052 INFO:tasks.rados.rados.0.smithi076.stdout:850: done (5 left) 2021-03-15T22:22:59.052 INFO:tasks.rados.rados.0.smithi076.stdout:851: done (4 left) 2021-03-15T22:22:59.053 INFO:tasks.rados.rados.0.smithi076.stdout:852: done (3 left) 2021-03-15T22:22:59.053 INFO:tasks.rados.rados.0.smithi076.stdout:853: done (2 left) 2021-03-15T22:22:59.053 INFO:tasks.rados.rados.0.smithi076.stdout:854: read oid 12 snap -1 2021-03-15T22:22:59.053 INFO:tasks.rados.rados.0.smithi076.stdout:854: expect (ObjNum 308 snap 80 seq_num 308) 2021-03-15T22:22:59.054 INFO:tasks.rados.rados.0.smithi076.stdout:855: read oid 35 snap 70 2021-03-15T22:22:59.054 INFO:tasks.rados.rados.0.smithi076.stdout:855: expect (ObjNum 6 snap 0 seq_num 6) 2021-03-15T22:22:59.054 INFO:tasks.rados.rados.0.smithi076.stdout:856: delete oid 33 current snap is 87 2021-03-15T22:22:59.059 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:22:59.060 INFO:tasks.rados.rados.0.smithi076.stdout:854: expect (ObjNum 308 snap 80 seq_num 308) 2021-03-15T22:22:59.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:22:59.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:59.100 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:59.101 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:22:59.176 INFO:tasks.rados.rados.0.smithi076.stdout:854: done (4 left) 2021-03-15T22:22:59.177 INFO:tasks.rados.rados.0.smithi076.stdout:856: done (3 left) 2021-03-15T22:22:59.177 INFO:tasks.rados.rados.0.smithi076.stdout:857: rollback oid 17 current snap is 87 2021-03-15T22:22:59.177 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 17 to 84 2021-03-15T22:22:59.178 INFO:tasks.rados.rados.0.smithi076.stdout:849: expect (ObjNum 287 snap 77 seq_num 287) 2021-03-15T22:22:59.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:59.223 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:59.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:59.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:22:59.326 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:59.351 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:59.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:59.454 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:59.455 INFO:tasks.rados.rados.0.smithi076.stdout:855: expect (ObjNum 6 snap 0 seq_num 6) 2021-03-15T22:22:59.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:59.478 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:59.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:59.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[62156]: cluster 2021-03-15T22:22:57.626104+0000 osd.1 (osd.1) 57 : cluster [DBG] 7.b scrub ok 2021-03-15T22:22:59.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[62156]: cluster 2021-03-15T22:22:58.472701+0000 osd.4 (osd.4) 61 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:59.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[62156]: cluster 2021-03-15T22:22:58.472978+0000 osd.4 (osd.4) 62 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:59.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[62156]: cluster 2021-03-15T22:22:59.047180+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2021-03-15T22:22:59.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: cluster 2021-03-15T22:22:57.626104+0000 2021-03-15T22:22:59.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: osd.1 (osd.1) 57 : cluster [DBG] 7.b scrub ok 2021-03-15T22:22:59.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:22:59.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: -15T22:22:58 2021-03-15T22:22:59.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: .472701+0000 osd.4 (osd.4) 61 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:59.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:22:59.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: 03-15T22:22:58.472978+0000 2021-03-15T22:22:59.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: osd.4 (osd.4) 62 : cluster [DBG] 2021-03-15T22:22:59.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: purged_snaps scrub ok 2021-03-15T22:22:59.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: cluster 2021-03-15T22:22 2021-03-15T22:22:59.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: :59.047180+0000 mon.a (mon.0) 941 2021-03-15T22:22:59.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:22:59 smithi076 conmon[56514]: : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2021-03-15T22:22:59.528 INFO:tasks.rados.rados.0.smithi076.stdout:857: finishing rollback tid 0 to smithi07694878-17 2021-03-15T22:22:59.528 INFO:tasks.rados.rados.0.smithi076.stdout:857: finishing rollback tid 1 to smithi07694878-17 2021-03-15T22:22:59.529 INFO:tasks.rados.rados.0.smithi076.stdout:857: finishing rollback tid 2 to smithi07694878-17 2021-03-15T22:22:59.529 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 73 (ObjNum 311 snap 81 seq_num 311) dirty exists 2021-03-15T22:22:59.529 INFO:tasks.rados.rados.0.smithi076.stdout:849: done (3 left) 2021-03-15T22:22:59.529 INFO:tasks.rados.rados.0.smithi076.stdout:855: done (2 left) 2021-03-15T22:22:59.529 INFO:tasks.rados.rados.0.smithi076.stdout:857: done (1 left) 2021-03-15T22:22:59.530 INFO:tasks.rados.rados.0.smithi076.stdout:858: read oid 19 snap 70 2021-03-15T22:22:59.530 INFO:tasks.rados.rados.0.smithi076.stdout:858: expect (ObjNum 6 snap 0 seq_num 6) 2021-03-15T22:22:59.530 INFO:tasks.rados.rados.0.smithi076.stdout:859: read oid 34 snap -1 2021-03-15T22:22:59.530 INFO:tasks.rados.rados.0.smithi076.stdout:859: expect deleted 2021-03-15T22:22:59.531 INFO:tasks.rados.rados.0.smithi076.stdout:860: copy_from oid 31 from oid 5 current snap is 87 2021-03-15T22:22:59.531 INFO:tasks.rados.rados.0.smithi076.stdout:861: rollback oid 7 current snap is 87 2021-03-15T22:22:59.531 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 7 to 76 2021-03-15T22:22:59.531 INFO:tasks.rados.rados.0.smithi076.stdout:862: read oid 27 snap -1 2021-03-15T22:22:59.531 INFO:tasks.rados.rados.0.smithi076.stdout:862: expect deleted 2021-03-15T22:22:59.532 INFO:tasks.rados.rados.0.smithi076.stdout:863: read oid 45 snap 72 2021-03-15T22:22:59.532 INFO:tasks.rados.rados.0.smithi076.stdout:863: expect (ObjNum 272 snap 72 seq_num 272) 2021-03-15T22:22:59.532 INFO:tasks.rados.rados.0.smithi076.stdout:864: write oid 26 current snap is 87 2021-03-15T22:22:59.532 INFO:tasks.rados.rados.0.smithi076.stdout:864: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2021-03-15T22:22:59.535 INFO:tasks.rados.rados.0.smithi076.stdout:864: writing smithi07694878-26 from 611854 to 1328858 tid 1 2021-03-15T22:22:59.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:59 smithi125 conmon[56739]: cluster 2021-03-15T22:22:57.626104+0000 osd.1 (osd.1) 2021-03-15T22:22:59.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:59 smithi125 conmon[56739]: 57 : cluster [DBG] 7.b scrub ok 2021-03-15T22:22:59.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:59 smithi125 conmon[56739]: cluster 2021-03-15T22:22:58.472701+0000 osd.4 (osd.4) 61 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:22:59.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:59 smithi125 conmon[56739]: cluster 2021-03-15T22:22:58.472978+0000 osd.4 (osd.4) 62 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:22:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:22:59 smithi125 conmon[56739]: cluster 2021-03-15T22:22:59.047180+0000 mon.a (mon.0) 941 : cluster [DBG] osdmap e264: 8 total, 8 up, 8 in 2021-03-15T22:22:59.543 INFO:tasks.rados.rados.0.smithi076.stdout:864: writing smithi07694878-26 from 2126143 to 2687910 tid 2 2021-03-15T22:22:59.547 INFO:tasks.rados.rados.0.smithi076.stdout:864: writing smithi07694878-26 from 3259733 to 3411854 tid 3 2021-03-15T22:22:59.547 INFO:tasks.rados.rados.0.smithi076.stdout:865: write (excl) oid 8 current snap is 87 2021-03-15T22:22:59.548 INFO:tasks.rados.rados.0.smithi076.stdout:865: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2021-03-15T22:22:59.555 INFO:tasks.rados.rados.0.smithi076.stdout:865: writing smithi07694878-8 from 628661 to 1420914 tid 1 2021-03-15T22:22:59.561 INFO:tasks.rados.rados.0.smithi076.stdout:865: writing smithi07694878-8 from 2184625 to 2606403 tid 2 2021-03-15T22:22:59.566 INFO:tasks.rados.rados.0.smithi076.stdout:865: writing smithi07694878-8 from 3403509 to 3428661 tid 3 2021-03-15T22:22:59.566 INFO:tasks.rados.rados.0.smithi076.stdout:866: write oid 47 current snap is 87 2021-03-15T22:22:59.567 INFO:tasks.rados.rados.0.smithi076.stdout:860: finishing copy_from to smithi07694878-31 2021-03-15T22:22:59.567 INFO:tasks.rados.rados.0.smithi076.stdout:860: got expected ENOENT (src dne) 2021-03-15T22:22:59.567 INFO:tasks.rados.rados.0.smithi076.stdout:860: finishing copy_from racing read to smithi07694878-31 2021-03-15T22:22:59.567 INFO:tasks.rados.rados.0.smithi076.stdout:866: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2021-03-15T22:22:59.572 INFO:tasks.rados.rados.0.smithi076.stdout:866: writing smithi07694878-47 from 645468 to 1112970 tid 1 2021-03-15T22:22:59.579 INFO:tasks.rados.rados.0.smithi076.stdout:866: writing smithi07694878-47 from 1726754 to 2292190 tid 2 2021-03-15T22:22:59.579 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:22:59.585 INFO:tasks.rados.rados.0.smithi076.stdout:866: writing smithi07694878-47 from 2798226 to 3395064 tid 3 2021-03-15T22:22:59.585 INFO:tasks.rados.rados.0.smithi076.stdout:859: done (9 left) 2021-03-15T22:22:59.585 INFO:tasks.rados.rados.0.smithi076.stdout:860: done (8 left) 2021-03-15T22:22:59.586 INFO:tasks.rados.rados.0.smithi076.stdout:862: done (7 left) 2021-03-15T22:22:59.586 INFO:tasks.rados.rados.0.smithi076.stdout:867: copy_from oid 43 from oid 4 current snap is 87 2021-03-15T22:22:59.586 INFO:tasks.rados.rados.0.smithi076.stdout:858: expect (ObjNum 6 snap 0 seq_num 6) 2021-03-15T22:22:59.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:59.609 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:59.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:22:59.687 INFO:tasks.rados.rados.0.smithi076.stdout:861: finishing rollback tid 0 to smithi07694878-7 2021-03-15T22:22:59.688 INFO:tasks.rados.rados.0.smithi076.stdout:861: finishing rollback tid 1 to smithi07694878-7 2021-03-15T22:22:59.688 INFO:tasks.rados.rados.0.smithi076.stdout:861: finishing rollback tid 2 to smithi07694878-7 2021-03-15T22:22:59.688 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 323 (ObjNum 271 snap 72 seq_num 271) dirty exists 2021-03-15T22:22:59.688 INFO:tasks.rados.rados.0.smithi076.stdout:864: finishing write tid 1 to smithi07694878-26 2021-03-15T22:22:59.689 INFO:tasks.rados.rados.0.smithi076.stdout:864: finishing write tid 2 to smithi07694878-26 2021-03-15T22:22:59.689 INFO:tasks.rados.rados.0.smithi076.stdout:858: done (7 left) 2021-03-15T22:22:59.689 INFO:tasks.rados.rados.0.smithi076.stdout:861: done (6 left) 2021-03-15T22:22:59.689 INFO:tasks.rados.rados.0.smithi076.stdout:868: read oid 37 snap -1 2021-03-15T22:22:59.690 INFO:tasks.rados.rados.0.smithi076.stdout:868: expect (ObjNum 249 snap 62 seq_num 249) 2021-03-15T22:22:59.690 INFO:tasks.rados.rados.0.smithi076.stdout:869: read oid 31 snap -1 2021-03-15T22:22:59.690 INFO:tasks.rados.rados.0.smithi076.stdout:869: expect (ObjNum 313 snap 84 seq_num 313) 2021-03-15T22:22:59.690 INFO:tasks.rados.rados.0.smithi076.stdout:870: read oid 40 snap 83 2021-03-15T22:22:59.691 INFO:tasks.rados.rados.0.smithi076.stdout:870: expect (ObjNum 312 snap 83 seq_num 312) 2021-03-15T22:22:59.691 INFO:tasks.rados.rados.0.smithi076.stdout:864: finishing write tid 3 to smithi07694878-26 2021-03-15T22:22:59.691 INFO:tasks.rados.rados.0.smithi076.stdout:864: finishing write tid 4 to smithi07694878-26 2021-03-15T22:22:59.691 INFO:tasks.rados.rados.0.smithi076.stdout:864: finishing write tid 5 to smithi07694878-26 2021-03-15T22:22:59.691 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 272 (ObjNum 322 snap 87 seq_num 322) dirty exists 2021-03-15T22:22:59.692 INFO:tasks.rados.rados.0.smithi076.stdout:864: left oid 26 (ObjNum 322 snap 87 seq_num 322) 2021-03-15T22:22:59.692 INFO:tasks.rados.rados.0.smithi076.stdout:865: finishing write tid 1 to smithi07694878-8 2021-03-15T22:22:59.692 INFO:tasks.rados.rados.0.smithi076.stdout:865: finishing write tid 2 to smithi07694878-8 2021-03-15T22:22:59.692 INFO:tasks.rados.rados.0.smithi076.stdout:865: finishing write tid 3 to smithi07694878-8 2021-03-15T22:22:59.693 INFO:tasks.rados.rados.0.smithi076.stdout:865: finishing write tid 4 to smithi07694878-8 2021-03-15T22:22:59.693 INFO:tasks.rados.rados.0.smithi076.stdout:865: finishing write tid 5 to smithi07694878-8 2021-03-15T22:22:59.693 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 165 (ObjNum 323 snap 87 seq_num 323) dirty exists 2021-03-15T22:22:59.694 INFO:tasks.rados.rados.0.smithi076.stdout:865: left oid 8 (ObjNum 323 snap 87 seq_num 323) 2021-03-15T22:22:59.694 INFO:tasks.rados.rados.0.smithi076.stdout:866: finishing write tid 1 to smithi07694878-47 2021-03-15T22:22:59.694 INFO:tasks.rados.rados.0.smithi076.stdout:866: finishing write tid 2 to smithi07694878-47 2021-03-15T22:22:59.694 INFO:tasks.rados.rados.0.smithi076.stdout:866: finishing write tid 3 to smithi07694878-47 2021-03-15T22:22:59.695 INFO:tasks.rados.rados.0.smithi076.stdout:871: read oid 32 snap -1 2021-03-15T22:22:59.695 INFO:tasks.rados.rados.0.smithi076.stdout:871: expect (ObjNum 265 snap 72 seq_num 265) 2021-03-15T22:22:59.695 INFO:tasks.rados.rados.0.smithi076.stdout:864: done (9 left) 2021-03-15T22:22:59.695 INFO:tasks.rados.rados.0.smithi076.stdout:865: done (8 left) 2021-03-15T22:22:59.695 INFO:tasks.rados.rados.0.smithi076.stdout:872: read oid 49 snap -1 2021-03-15T22:22:59.696 INFO:tasks.rados.rados.0.smithi076.stdout:872: expect deleted 2021-03-15T22:22:59.696 INFO:tasks.rados.rados.0.smithi076.stdout:873: write (excl) oid 21 current snap is 87 2021-03-15T22:22:59.696 INFO:tasks.rados.rados.0.smithi076.stdout:866: finishing write tid 4 to smithi07694878-47 2021-03-15T22:22:59.696 INFO:tasks.rados.rados.0.smithi076.stdout:873: seq_num 325 ranges {662275=542751,1785236=425447,2942002=520273} 2021-03-15T22:22:59.697 INFO:tasks.rados.rados.0.smithi076.stdout:873: writing smithi07694878-21 from 662275 to 1205026 tid 1 2021-03-15T22:22:59.700 INFO:tasks.rados.rados.0.smithi076.stdout:873: writing smithi07694878-21 from 1785236 to 2210683 tid 2 2021-03-15T22:22:59.708 INFO:tasks.rados.rados.0.smithi076.stdout:873: writing smithi07694878-21 from 2942002 to 3462275 tid 3 2021-03-15T22:22:59.708 INFO:tasks.rados.rados.0.smithi076.stdout:874: write oid 42 current snap is 87 2021-03-15T22:22:59.709 INFO:tasks.rados.rados.0.smithi076.stdout:866: finishing write tid 5 to smithi07694878-47 2021-03-15T22:22:59.709 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 164 (ObjNum 324 snap 87 seq_num 324) dirty exists 2021-03-15T22:22:59.709 INFO:tasks.rados.rados.0.smithi076.stdout:866: left oid 47 (ObjNum 324 snap 87 seq_num 324) 2021-03-15T22:22:59.709 INFO:tasks.rados.rados.0.smithi076.stdout:867: finishing copy_from to smithi07694878-43 2021-03-15T22:22:59.710 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 255 (ObjNum 239 snap 57 seq_num 239) dirty exists 2021-03-15T22:22:59.710 INFO:tasks.rados.rados.0.smithi076.stdout:867: finishing copy_from racing read to smithi07694878-43 2021-03-15T22:22:59.710 INFO:tasks.rados.rados.0.smithi076.stdout:874: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2021-03-15T22:22:59.711 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:22:59.716 INFO:tasks.rados.rados.0.smithi076.stdout:874: writing smithi07694878-42 from 679082 to 1297082 tid 1 2021-03-15T22:22:59.724 INFO:tasks.rados.rados.0.smithi076.stdout:874: writing smithi07694878-42 from 1843718 to 2412823 tid 2 2021-03-15T22:22:59.732 INFO:tasks.rados.rados.0.smithi076.stdout:874: writing smithi07694878-42 from 2853072 to 3479082 tid 3 2021-03-15T22:22:59.732 INFO:tasks.rados.rados.0.smithi076.stdout:866: done (10 left) 2021-03-15T22:22:59.732 INFO:tasks.rados.rados.0.smithi076.stdout:867: done (9 left) 2021-03-15T22:22:59.732 INFO:tasks.rados.rados.0.smithi076.stdout:875: copy_from oid 16 from oid 34 current snap is 87 2021-03-15T22:22:59.733 INFO:tasks.rados.rados.0.smithi076.stdout:869: expect (ObjNum 313 snap 84 seq_num 313) 2021-03-15T22:22:59.735 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:22:59.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:59.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:22:59.837 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:22:59.848 INFO:tasks.rados.rados.0.smithi076.stdout:868: expect (ObjNum 249 snap 62 seq_num 249) 2021-03-15T22:22:59.853 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:22:59.853 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:22:59.853 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:22:59.854 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:22:59.854 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:22:59.861 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:59.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:59.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:22:59.908 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:22:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:22:59.937 INFO:tasks.rados.rados.0.smithi076.stdout:870: expect (ObjNum 312 snap 83 seq_num 312) 2021-03-15T22:22:59.957 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:22:59.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:22:59.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:22:59.988 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:22:59.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:00.090 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:00.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:00.114 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:00.115 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:00.159 INFO:tasks.rados.rados.0.smithi076.stdout:871: expect (ObjNum 265 snap 72 seq_num 265) 2021-03-15T22:23:00.217 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:00.239 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:00.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:00.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:00.296 INFO:tasks.rados.rados.0.smithi076.stdout:873: finishing write tid 1 to smithi07694878-21 2021-03-15T22:23:00.297 INFO:tasks.rados.rados.0.smithi076.stdout:873: finishing write tid 2 to smithi07694878-21 2021-03-15T22:23:00.297 INFO:tasks.rados.rados.0.smithi076.stdout:873: finishing write tid 3 to smithi07694878-21 2021-03-15T22:23:00.297 INFO:tasks.rados.rados.0.smithi076.stdout:873: finishing write tid 4 to smithi07694878-21 2021-03-15T22:23:00.297 INFO:tasks.rados.rados.0.smithi076.stdout:873: finishing write tid 5 to smithi07694878-21 2021-03-15T22:23:00.298 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 474 (ObjNum 325 snap 87 seq_num 325) dirty exists 2021-03-15T22:23:00.298 INFO:tasks.rados.rados.0.smithi076.stdout:873: left oid 21 (ObjNum 325 snap 87 seq_num 325) 2021-03-15T22:23:00.298 INFO:tasks.rados.rados.0.smithi076.stdout:874: finishing write tid 1 to smithi07694878-42 2021-03-15T22:23:00.298 INFO:tasks.rados.rados.0.smithi076.stdout:874: finishing write tid 2 to smithi07694878-42 2021-03-15T22:23:00.299 INFO:tasks.rados.rados.0.smithi076.stdout:875: finishing copy_from to smithi07694878-16 2021-03-15T22:23:00.299 INFO:tasks.rados.rados.0.smithi076.stdout:875: got expected ENOENT (src dne) 2021-03-15T22:23:00.299 INFO:tasks.rados.rados.0.smithi076.stdout:875: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:23:00.299 INFO:tasks.rados.rados.0.smithi076.stdout:868: done (9 left) 2021-03-15T22:23:00.299 INFO:tasks.rados.rados.0.smithi076.stdout:869: done (8 left) 2021-03-15T22:23:00.300 INFO:tasks.rados.rados.0.smithi076.stdout:870: done (7 left) 2021-03-15T22:23:00.300 INFO:tasks.rados.rados.0.smithi076.stdout:871: done (6 left) 2021-03-15T22:23:00.300 INFO:tasks.rados.rados.0.smithi076.stdout:872: done (5 left) 2021-03-15T22:23:00.300 INFO:tasks.rados.rados.0.smithi076.stdout:873: done (4 left) 2021-03-15T22:23:00.301 INFO:tasks.rados.rados.0.smithi076.stdout:875: done (3 left) 2021-03-15T22:23:00.301 INFO:tasks.rados.rados.0.smithi076.stdout:876: copy_from oid 4 from oid 46 current snap is 87 2021-03-15T22:23:00.301 INFO:tasks.rados.rados.0.smithi076.stdout:874: finishing write tid 3 to smithi07694878-42 2021-03-15T22:23:00.301 INFO:tasks.rados.rados.0.smithi076.stdout:874: finishing write tid 4 to smithi07694878-42 2021-03-15T22:23:00.301 INFO:tasks.rados.rados.0.smithi076.stdout:874: finishing write tid 5 to smithi07694878-42 2021-03-15T22:23:00.302 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 478 (ObjNum 326 snap 87 seq_num 326) dirty exists 2021-03-15T22:23:00.302 INFO:tasks.rados.rados.0.smithi076.stdout:874: left oid 42 (ObjNum 326 snap 87 seq_num 326) 2021-03-15T22:23:00.302 INFO:tasks.rados.rados.0.smithi076.stdout:848: finishing rollback tid 1 to smithi07694878-50 2021-03-15T22:23:00.302 INFO:tasks.rados.rados.0.smithi076.stdout:848: finishing rollback tid 2 to smithi07694878-50 2021-03-15T22:23:00.303 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 274 (ObjNum 170 snap 37 seq_num 170) dirty exists 2021-03-15T22:23:00.303 INFO:tasks.rados.rados.0.smithi076.stdout:863: expect (ObjNum 272 snap 72 seq_num 272) 2021-03-15T22:23:00.342 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:00.366 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:00.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:00.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:00.403 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:00.427 INFO:tasks.rados.rados.0.smithi076.stdout:848: done (3 left) 2021-03-15T22:23:00.427 INFO:tasks.rados.rados.0.smithi076.stdout:863: done (2 left) 2021-03-15T22:23:00.427 INFO:tasks.rados.rados.0.smithi076.stdout:874: done (1 left) 2021-03-15T22:23:00.427 INFO:tasks.rados.rados.0.smithi076.stdout:877: snap_remove snap 70 2021-03-15T22:23:00.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:00.492 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:00.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:00.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:00.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:00 smithi125 conmon[56739]: cluster 2021-03-15T22:22:59.017697+0000 mgr.y 2021-03-15T22:23:00.515 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:00 smithi125 conmon[56739]: (mgr.14144) 500 : cluster [DBG] pgmap v522: 138 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 peering, 132 active+clean; 190 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 2/1248 objects degraded (0.160%); 56/1248 objects misplaced (4.487%); 350 KiB/s, 13 objects/s recovering 2021-03-15T22:23:00.515 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:00 smithi125 conmon[56739]: cluster 2021-03-15T22:22:59.238977+0000 mon.a (mon.0) 942 : cluster [WRN] Health check failed: Degraded data redundancy: 2/1248 objects degraded (0.160%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:23:00.515 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:00 smithi125 conmon[56739]: audit 2021-03-15T22:22:59.572231+0000 mgr.y (mgr.14144) 501 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:00.516 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:00 smithi125 conmon[56739]: cluster 2021-03-15T22:23:00.047868+0000 mon.a (mon.0) 943 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2021-03-15T22:23:00.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[56514]: cluster 2021-03-15T22:22: 2021-03-15T22:23:00.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[56514]: 59.017697+0000 mgr.y (mgr.14144) 500 : cluster [DBG] pgmap v522: 138 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 peering, 132 active+clean; 190 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 2/1248 objects degraded (0.160%); 56/1248 objects misplaced (4.487%); 350 KiB/s, 13 objects/s recovering 2021-03-15T22:23:00.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[56514]: cluster 2021-03-15T22:22:59.238977+0000 mon.a (mon.0) 942 : cluster [WRN] Health check failed: Degraded data redundancy: 2/1248 objects degraded (0.160%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:23:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[56514]: audit 2021-03-15T22:22:59.572231+0000 mgr.y (mgr.14144) 501 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[56514]: cluster 2021-03-15T22:23:00.047868+0000 mon.a (mon.0) 943 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2021-03-15T22:23:00.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[62156]: cluster 2021-03-15T22:22:59.017697+0000 2021-03-15T22:23:00.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[62156]: mgr.y (mgr.14144) 500 : cluster [DBG] pgmap v522: 138 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 3 peering, 132 active+clean; 190 MiB data, 593 MiB used, 715 GiB / 715 GiB avail; 5.3 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 2/1248 objects degraded (0.160%); 56/1248 objects misplaced (4.487%); 350 KiB/s, 13 objects/s recovering 2021-03-15T22:23:00.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[62156]: cluster 2021-03-15T22:22: 2021-03-15T22:23:00.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[62156]: 59.238977+0000 mon.a (mon.0) 942 : cluster [WRN] Health check failed: Degraded data redundancy: 2/1248 objects degraded (0.160%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:23:00.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:23:00.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[62156]: 22:22:59.572231+0000 mgr.y (mgr.14144) 501 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:00.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:23:00.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[62156]: 23:00.047868+0000 mon.a (mon.0) 943 : cluster [DBG] osdmap e265: 8 total, 8 up, 8 in 2021-03-15T22:23:00.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:00 smithi076 conmon[62156]: 2021-03-15T22:23:00.537 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-15T22:23:00.595 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:00.641 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:00.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:00.643 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:00.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:00.767 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:00.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:00.861 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:23:00.870 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:00.896 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:00.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:00.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:01.023 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:01.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:01.061 INFO:tasks.rados.rados.0.smithi076.stdout:877: done (1 left) 2021-03-15T22:23:01.062 INFO:tasks.rados.rados.0.smithi076.stdout:878: read oid 19 snap -1 2021-03-15T22:23:01.062 INFO:tasks.rados.rados.0.smithi076.stdout:878: expect (ObjNum 273 snap 72 seq_num 273) 2021-03-15T22:23:01.062 INFO:tasks.rados.rados.0.smithi076.stdout:879: copy_from oid 31 from oid 1 current snap is 87 2021-03-15T22:23:01.062 INFO:tasks.rados.rados.0.smithi076.stdout:880: rollback oid 16 current snap is 87 2021-03-15T22:23:01.063 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 63 2021-03-15T22:23:01.063 INFO:tasks.rados.rados.0.smithi076.stdout:876: finishing copy_from to smithi07694878-4 2021-03-15T22:23:01.063 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 194 (ObjNum 316 snap 85 seq_num 316) dirty exists 2021-03-15T22:23:01.063 INFO:tasks.rados.rados.0.smithi076.stdout:876: finishing copy_from racing read to smithi07694878-4 2021-03-15T22:23:01.063 INFO:tasks.rados.rados.0.smithi076.stdout:876: done (3 left) 2021-03-15T22:23:01.064 INFO:tasks.rados.rados.0.smithi076.stdout:881: read oid 12 snap -1 2021-03-15T22:23:01.064 INFO:tasks.rados.rados.0.smithi076.stdout:881: expect (ObjNum 308 snap 80 seq_num 308) 2021-03-15T22:23:01.064 INFO:tasks.rados.rados.0.smithi076.stdout:882: snap_create 2021-03-15T22:23:01.070 INFO:tasks.rados.rados.0.smithi076.stdout:878: expect (ObjNum 273 snap 72 seq_num 273) 2021-03-15T22:23:01.071 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:23:01.126 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:01.147 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:01.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:01.199 INFO:tasks.rados.rados.0.smithi076.stdout:880: finishing rollback tid 0 to smithi07694878-16 2021-03-15T22:23:01.199 INFO:tasks.rados.rados.0.smithi076.stdout:881: expect (ObjNum 308 snap 80 seq_num 308) 2021-03-15T22:23:01.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:01.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:01.272 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:01.272 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:01.304 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:01 smithi125 conmon[56739]: audit 2021-03-15T22:23:00.998630+0000 mon.a (mon.0) 944 : audit [INF] from='client.? 172.21.15.76:0/3696471509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:01.304 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:01 smithi125 conmon[56739]: cluster 2021-03-15T22:23:01.052154+0000 mon.a (mon.0) 945 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:23:01.304 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:01 smithi125 conmon[56739]: audit 2021-03-15T22:23:01.056480+0000 mon.a (mon.0) 946 : audit [INF] from='client.? 172.21.15.76:0/3696471509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:01.304 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:01 smithi125 conmon[56739]: cluster 2021-03-15T22:23:01.056519+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2021-03-15T22:23:01.305 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:01 smithi125 conmon[56739]: audit 2021-03-15T22:23:01.059420+0000 mon.a (mon.0) 948 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:23:01.305 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:01 smithi125 conmon[56739]: audit 2021-03-15T22:23:01.059740+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:23:01.311 INFO:tasks.rados.rados.0.smithi076.stdout:880: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:23:01.311 INFO:tasks.rados.rados.0.smithi076.stdout:880: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:23:01.312 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 406 (ObjNum 208 snap 47 seq_num 208) dirty exists 2021-03-15T22:23:01.312 INFO:tasks.rados.rados.0.smithi076.stdout:879: finishing copy_from to smithi07694878-31 2021-03-15T22:23:01.312 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 151 (ObjNum 280 snap 74 seq_num 280) dirty exists 2021-03-15T22:23:01.312 INFO:tasks.rados.rados.0.smithi076.stdout:879: finishing copy_from racing read to smithi07694878-31 2021-03-15T22:23:01.373 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:01.396 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:01.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:01.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:01.498 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:01.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[56514]: audit 2021-03-15T22:23:00.998630+0000 mon.a (mon.0) 944 : audit [INF] from='client.? 172.21.15.76:0/3696471509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:01.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[56514]: cluster 2021-03-15T22:23:01.052154+0000 mon.a (mon.0) 945 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:23:01.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[56514]: audit 2021-03-15T22:23:01.056480+0000 mon.a (mon.0) 946 : audit [INF] from='client.? 172.21.15.76:0/3696471509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:01.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[56514]: cluster 2021-03-15T22:23:01.056519+0000 mon.a (mon.0) 947 : cluster [DBG] osdmap e266: 8 total, 8 up, 8 in 2021-03-15T22:23:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[56514]: audit 2021-03-15T22:23:01.059420+0000 mon.a (mon. 2021-03-15T22:23:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[56514]: 0) 948 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:23:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[56514]: audit 2021-03-15T22:23:01.059740+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:23:01.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: audit 2021-03-15T22:23:00. 2021-03-15T22:23:01.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 998630+0000 mon.a (mon.0) 944 : audit [INF] from='client.? 172.21.15.76:0/3696471509' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:01.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:23:01.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 03-15T22:23:01.052154+0000 mon.a (mon.0) 945 : cluster 2021-03-15T22:23:01.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:23:01.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: audit 2021-03-15T22:23:01.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:23:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: T22:23:01.056480+0000 mon.a (mon.0) 946 2021-03-15T22:23:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/3696471509' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: cluster 2021-03-15T22:23:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021-03-15T22:23:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021 2021-03-15T22:23:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: - 2021-03-15T22:23:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 03 2021-03-15T22:23:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: - 2021-03-15T22:23:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 15 2021-03-15T22:23:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: T 2021-03-15T22:23:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 22 2021-03-15T22:23:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: : 2021-03-15T22:23:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 23 2021-03-15T22:23:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: : 2021-03-15T22:23:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 01 2021-03-15T22:23:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: . 2021-03-15T22:23:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 056519 2021-03-15T22:23:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: +0000 2021-03-15T22:23:01.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021-03-15T22:23:01.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: mon.a 2021-03-15T22:23:01.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: ( 2021-03-15T22:23:01.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: mon 2021-03-15T22:23:01.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: . 2021-03-15T22:23:01.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 0 2021-03-15T22:23:01.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: ) 2021-03-15T22:23:01.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 947 2021-03-15T22:23:01.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: : 2021-03-15T22:23:01.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: cluster 2021-03-15T22:23:01.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021-03-15T22:23:01.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: [DBG] 2021-03-15T22:23:01.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021-03-15T22:23:01.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: osdmap e266: 8 total, 8 up, 8 in 2021-03-15T22:23:01.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021-03-15T22:23:01.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: audit 2021-03-15T22:23:01.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021-03-15T22:23:01.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021 2021-03-15T22:23:01.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: - 2021-03-15T22:23:01.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 03 2021-03-15T22:23:01.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: - 2021-03-15T22:23:01.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 15 2021-03-15T22:23:01.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: T 2021-03-15T22:23:01.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 22 2021-03-15T22:23:01.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: : 2021-03-15T22:23:01.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 23 2021-03-15T22:23:01.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: : 2021-03-15T22:23:01.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 01 2021-03-15T22:23:01.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: . 2021-03-15T22:23:01.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 059420 2021-03-15T22:23:01.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: +0000 2021-03-15T22:23:01.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021-03-15T22:23:01.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: mon.a 2021-03-15T22:23:01.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: ( 2021-03-15T22:23:01.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: mon 2021-03-15T22:23:01.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: . 2021-03-15T22:23:01.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 0 2021-03-15T22:23:01.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: ) 2021-03-15T22:23:01.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 948 2021-03-15T22:23:01.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: : 2021-03-15T22:23:01.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: audit 2021-03-15T22:23:01.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021-03-15T22:23:01.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: [INF] 2021-03-15T22:23:01.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021-03-15T22:23:01.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:23:01.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: 2021-03-15T22:23:01.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:23:01.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[62156]: :23:01.059740+0000 mon.a (mon.0) 949 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:23:01.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:01.539 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:01.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:01.641 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:01.663 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:01.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:01.765 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:01.768 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:23:01.769 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:23:01.769 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2021-03-15T22:23:01.769 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight-by-utilization 1000 3.0 2021-03-15T22:23:01.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:01.788 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:01.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:01.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:01.912 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:01.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:01.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:02.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:02.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:02.039 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:02.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:02.061 INFO:tasks.rados.rados.0.smithi076.stdout:878: done (4 left) 2021-03-15T22:23:02.061 INFO:tasks.rados.rados.0.smithi076.stdout:879: done (3 left) 2021-03-15T22:23:02.062 INFO:tasks.rados.rados.0.smithi076.stdout:880: done (2 left) 2021-03-15T22:23:02.062 INFO:tasks.rados.rados.0.smithi076.stdout:881: done (1 left) 2021-03-15T22:23:02.062 INFO:tasks.rados.rados.0.smithi076.stdout:882: done (0 left) 2021-03-15T22:23:02.062 INFO:tasks.rados.rados.0.smithi076.stdout:883: rollback oid 15 current snap is 88 2021-03-15T22:23:02.063 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 15 to 82 2021-03-15T22:23:02.063 INFO:tasks.rados.rados.0.smithi076.stdout:884: delete oid 24 current snap is 88 2021-03-15T22:23:02.064 INFO:tasks.rados.rados.0.smithi076.stdout:883: finishing rollback tid 1 to smithi07694878-15 2021-03-15T22:23:02.064 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 0 (ObjNum 261879888 snap 21896 seq_num 0) dirty dne 2021-03-15T22:23:02.065 INFO:tasks.rados.rados.0.smithi076.stdout:883: done (1 left) 2021-03-15T22:23:02.065 INFO:tasks.rados.rados.0.smithi076.stdout:884: done (0 left) 2021-03-15T22:23:02.065 INFO:tasks.rados.rados.0.smithi076.stdout:885: write oid 11 current snap is 88 2021-03-15T22:23:02.066 INFO:tasks.rados.rados.0.smithi076.stdout:885: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2021-03-15T22:23:02.072 INFO:tasks.rados.rados.0.smithi076.stdout:885: writing smithi07694878-11 from 712696 to 1197547 tid 1 2021-03-15T22:23:02.079 INFO:tasks.rados.rados.0.smithi076.stdout:885: writing smithi07694878-11 from 1960682 to 2533456 tid 2 2021-03-15T22:23:02.085 INFO:tasks.rados.rados.0.smithi076.stdout:885: writing smithi07694878-11 from 3307918 to 3512696 tid 3 2021-03-15T22:23:02.085 INFO:tasks.rados.rados.0.smithi076.stdout:886: read oid 44 snap -1 2021-03-15T22:23:02.086 INFO:tasks.rados.rados.0.smithi076.stdout:886: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:23:02.086 INFO:tasks.rados.rados.0.smithi076.stdout:887: write (excl) oid 49 current snap is 88 2021-03-15T22:23:02.086 INFO:tasks.rados.rados.0.smithi076.stdout:885: finishing write tid 1 to smithi07694878-11 2021-03-15T22:23:02.087 INFO:tasks.rados.rados.0.smithi076.stdout:885: finishing write tid 2 to smithi07694878-11 2021-03-15T22:23:02.087 INFO:tasks.rados.rados.0.smithi076.stdout:887: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2021-03-15T22:23:02.092 INFO:tasks.rados.rados.0.smithi076.stdout:887: writing smithi07694878-49 from 729503 to 1289603 tid 1 2021-03-15T22:23:02.100 INFO:tasks.rados.rados.0.smithi076.stdout:887: writing smithi07694878-49 from 2019164 to 2735596 tid 2 2021-03-15T22:23:02.104 INFO:tasks.rados.rados.0.smithi076.stdout:887: writing smithi07694878-49 from 3335341 to 3529503 tid 3 2021-03-15T22:23:02.104 INFO:tasks.rados.rados.0.smithi076.stdout:888: snap_remove snap 76 2021-03-15T22:23:02.140 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:02.162 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:02.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:02.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:02.263 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:23:01 smithi076 conmon[65511]: debug 2021-03-15T22:23:01.763+0000 7f1a2307e700 -1 osd.0 pg_epoch: 266 pg[7.f( v 266'265 (0'0,266'265] local-lis/les=263/264 n=10 ec=91/91 lis/c=263/263 les/c/f=264/264/0 sis=263) [0,3,7] r=0 lpr=263 crt=266'265 lcod 266'263 mlcod 266'263 active+clean ps=[80~1,88~1,a4~1]] scrubber pg(7.f) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x557ad6fba9a0) 2021-03-15T22:23:02.265 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:02.288 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:02.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:02.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:02.344 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:23:02.345 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:23:02.345 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:23:02.345 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:23:02.346 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.346 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:3 3.13 3:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e57)", 2021-03-15T22:23:02.346 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:17:53.975558+0000", 2021-03-15T22:23:02.346 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 308.36590046700002, 2021-03-15T22:23:02.347 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.31057361700000002, 2021-03-15T22:23:02.347 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.347 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-15T22:23:02.347 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:23:02.348 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:23:02.348 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:23:02.348 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3 2021-03-15T22:23:02.349 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.349 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.349 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.350 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975558+0000", 2021-03-15T22:23:02.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.350 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.350 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.351 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.351 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975558+0000", 2021-03-15T22:23:02.351 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3149999999999999e-06 2021-03-15T22:23:02.351 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.352 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.352 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975560+0000", 2021-03-15T22:23:02.352 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7670000000000001e-06 2021-03-15T22:23:02.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.353 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.353 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.353 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975564+0000", 2021-03-15T22:23:02.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0937e-05 2021-03-15T22:23:02.354 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.354 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.354 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.354 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975575+0000", 2021-03-15T22:23:02.355 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1800000000000002e-06 2021-03-15T22:23:02.355 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.355 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.355 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.355 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975584+0000", 2021-03-15T22:23:02.356 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4805000000000001e-05 2021-03-15T22:23:02.356 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.357 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975639+0000", 2021-03-15T22:23:02.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2069999999999995e-06 2021-03-15T22:23:02.357 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.357 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:23:02.358 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975647+0000", 2021-03-15T22:23:02.358 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.31038343600000001 2021-03-15T22:23:02.358 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.358 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.359 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:54.286030+0000", 2021-03-15T22:23:02.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010097 2021-03-15T22:23:02.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.360 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.360 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.360 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:54.286131+0000", 2021-03-15T22:23:02.360 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.31057361700000002 2021-03-15T22:23:02.360 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.361 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.361 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.361 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.361 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.362 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:260 7.6 e104/91 7:63693c38:::smithi07694878-22:head v 104'32, mlcod=100'28)", 2021-03-15T22:23:02.362 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.195529+0000", 2021-03-15T22:23:02.362 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 163.14592921299999, 2021-03-15T22:23:02.362 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066965699999999998, 2021-03-15T22:23:02.363 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.363 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.363 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.363 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.363 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.364 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.195529+0000", 2021-03-15T22:23:02.364 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.364 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.364 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.365 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.365 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.195529+0000", 2021-03-15T22:23:02.365 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0026948739999999999 2021-03-15T22:23:02.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.366 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.366 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.366 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198224+0000", 2021-03-15T22:23:02.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1399999999999999e-06 2021-03-15T22:23:02.367 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.367 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.367 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.367 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198229+0000", 2021-03-15T22:23:02.367 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2414e-05 2021-03-15T22:23:02.368 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.368 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.368 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.368 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198241+0000", 2021-03-15T22:23:02.369 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0659999999999998e-06 2021-03-15T22:23:02.369 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.369 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.369 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.369 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198248+0000", 2021-03-15T22:23:02.370 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6831e-05 2021-03-15T22:23:02.370 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.370 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.370 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.371 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198285+0000", 2021-03-15T22:23:02.371 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7632e-05 2021-03-15T22:23:02.371 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.371 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.371 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.372 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198303+0000", 2021-03-15T22:23:02.372 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038786379999999998 2021-03-15T22:23:02.372 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.372 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.373 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.373 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.202182+0000", 2021-03-15T22:23:02.373 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3974999999999998e-05 2021-03-15T22:23:02.373 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.374 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.374 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.374 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.202226+0000", 2021-03-15T22:23:02.374 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066965699999999998 2021-03-15T22:23:02.375 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.375 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.375 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.375 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.376 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:265 7.6 e104/91 7:6283974b:::smithi07694878-30:head v 104'34, mlcod=100'28)", 2021-03-15T22:23:02.376 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.198708+0000", 2021-03-15T22:23:02.376 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 163.142750396, 2021-03-15T22:23:02.376 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050519500000000004, 2021-03-15T22:23:02.377 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.377 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.377 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.377 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.378 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198708+0000", 2021-03-15T22:23:02.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.378 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.379 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.198708+0000", 2021-03-15T22:23:02.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00075425600000000002 2021-03-15T22:23:02.379 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.380 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.380 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.199462+0000", 2021-03-15T22:23:02.380 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0170000000000001e-06 2021-03-15T22:23:02.381 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.381 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.381 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.381 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.199465+0000", 2021-03-15T22:23:02.382 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3107e-05 2021-03-15T22:23:02.382 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.382 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.382 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.382 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.199478+0000", 2021-03-15T22:23:02.383 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6520000000000003e-06 2021-03-15T22:23:02.383 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.383 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.199484+0000", 2021-03-15T22:23:02.384 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013140490000000001 2021-03-15T22:23:02.384 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.384 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.385 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.200798+0000", 2021-03-15T22:23:02.385 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6419999999999999e-05 2021-03-15T22:23:02.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.386 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.200814+0000", 2021-03-15T22:23:02.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028916110000000001 2021-03-15T22:23:02.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.203706+0000", 2021-03-15T22:23:02.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3838000000000002e-05 2021-03-15T22:23:02.387 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.388 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.388 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.388 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.203760+0000", 2021-03-15T22:23:02.388 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050519500000000004 2021-03-15T22:23:02.389 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.389 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.389 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.389 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.390 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.390 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:323 7.8 e104/91 7:1d74a1c3:::smithi07694878-44:head v 104'23, mlcod=103'20)", 2021-03-15T22:23:02.390 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.452200+0000", 2021-03-15T22:23:02.390 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 162.889258103, 2021-03-15T22:23:02.391 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0064996999999999997, 2021-03-15T22:23:02.391 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.391 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.391 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.391 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.392 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.392 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.452200+0000", 2021-03-15T22:23:02.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.392 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.393 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.393 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.393 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.452200+0000", 2021-03-15T22:23:02.393 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0039340390000000003 2021-03-15T22:23:02.393 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.394 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.394 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.394 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456134+0000", 2021-03-15T22:23:02.394 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.729e-06 2021-03-15T22:23:02.395 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.395 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.395 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.395 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456139+0000", 2021-03-15T22:23:02.395 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6049000000000002e-05 2021-03-15T22:23:02.396 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.396 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.396 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.396 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456155+0000", 2021-03-15T22:23:02.397 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6189999999999996e-06 2021-03-15T22:23:02.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.397 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.397 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.398 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456162+0000", 2021-03-15T22:23:02.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8286999999999999e-05 2021-03-15T22:23:02.398 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.398 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.399 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.399 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456220+0000", 2021-03-15T22:23:02.399 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8587e-05 2021-03-15T22:23:02.399 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.399 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.400 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.456238+0000", 2021-03-15T22:23:02.400 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024070160000000001 2021-03-15T22:23:02.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.401 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.458645+0000", 2021-03-15T22:23:02.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4373999999999998e-05 2021-03-15T22:23:02.402 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.402 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.458700+0000", 2021-03-15T22:23:02.403 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0064996999999999997 2021-03-15T22:23:02.403 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.403 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.404 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.404 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:344 7.6 e106/91 7:6283974b:::smithi07694878-30:head v 106'39, mlcod=104'36)", 2021-03-15T22:23:02.404 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:21.021046+0000", 2021-03-15T22:23:02.405 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 161.32041177400001, 2021-03-15T22:23:02.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0068779380000000001, 2021-03-15T22:23:02.405 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.405 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.406 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.406 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.406 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.406 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.021046+0000", 2021-03-15T22:23:02.407 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.408 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.021046+0000", 2021-03-15T22:23:02.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029106470000000001 2021-03-15T22:23:02.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.408 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.408 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.023957+0000", 2021-03-15T22:23:02.409 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1580000000000002e-06 2021-03-15T22:23:02.409 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.410 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.023963+0000", 2021-03-15T22:23:02.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0822000000000002e-05 2021-03-15T22:23:02.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.023984+0000", 2021-03-15T22:23:02.411 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1955e-05 2021-03-15T22:23:02.411 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.412 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.413 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.023996+0000", 2021-03-15T22:23:02.413 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0494000000000001e-05 2021-03-15T22:23:02.413 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.413 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.414 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.415 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.024056+0000", 2021-03-15T22:23:02.415 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7806000000000001e-05 2021-03-15T22:23:02.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.416 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.416 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.024084+0000", 2021-03-15T22:23:02.416 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037723679999999999 2021-03-15T22:23:02.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.417 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.417 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.027857+0000", 2021-03-15T22:23:02.417 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7688000000000007e-05 2021-03-15T22:23:02.418 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.418 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.418 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.418 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.027924+0000", 2021-03-15T22:23:02.419 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0068779380000000001 2021-03-15T22:23:02.419 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.419 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.419 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.420 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:368 7.5 7:a9545d62:::smithi07694878-1:head [write 726069~772507] snapc a=[a] ondisk+write+known_if_redirected e108)", 2021-03-15T22:23:02.420 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.113968+0000", 2021-03-15T22:23:02.421 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 159.22748968499999, 2021-03-15T22:23:02.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078723230000000005, 2021-03-15T22:23:02.421 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.421 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.422 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:23:02.422 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:23:02.422 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:02.422 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 368 2021-03-15T22:23:02.422 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.423 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.423 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.423 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.113968+0000", 2021-03-15T22:23:02.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.424 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.113968+0000", 2021-03-15T22:23:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0025141629999999998 2021-03-15T22:23:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.425 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.426 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116483+0000", 2021-03-15T22:23:02.426 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3490000000000002e-06 2021-03-15T22:23:02.426 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.427 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.427 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116487+0000", 2021-03-15T22:23:02.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6874000000000001e-05 2021-03-15T22:23:02.427 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.428 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.428 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.428 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116504+0000", 2021-03-15T22:23:02.428 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7606000000000001e-05 2021-03-15T22:23:02.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.429 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.429 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.429 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116521+0000", 2021-03-15T22:23:02.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3022999999999993e-05 2021-03-15T22:23:02.430 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.430 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.430 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116615+0000", 2021-03-15T22:23:02.431 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013848800000000001 2021-03-15T22:23:02.431 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.431 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.431 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.431 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116753+0000", 2021-03-15T22:23:02.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024625100000000001 2021-03-15T22:23:02.432 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.432 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.432 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:23:02.433 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.116999+0000", 2021-03-15T22:23:02.433 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0016196349999999999 2021-03-15T22:23:02.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.433 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.434 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:23:02.434 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.118619+0000", 2021-03-15T22:23:02.434 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0019812649999999998 2021-03-15T22:23:02.434 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.435 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.435 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:02.435 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.120600+0000", 2021-03-15T22:23:02.435 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011110619999999999 2021-03-15T22:23:02.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.436 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:02.436 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121711+0000", 2021-03-15T22:23:02.436 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6941999999999997e-05 2021-03-15T22:23:02.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.437 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.437 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.437 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121778+0000", 2021-03-15T22:23:02.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2664999999999997e-05 2021-03-15T22:23:02.438 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.438 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.439 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121841+0000", 2021-03-15T22:23:02.439 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078723230000000005 2021-03-15T22:23:02.439 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.439 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.440 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.440 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.440 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:531 7.8 e122/91 7:1822d391:::smithi07694878-47:head v 122'42, mlcod=118'38)", 2021-03-15T22:23:02.441 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:37.026000+0000", 2021-03-15T22:23:02.441 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 145.315458595, 2021-03-15T22:23:02.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011542325000000001, 2021-03-15T22:23:02.441 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.441 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.442 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.442 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.026000+0000", 2021-03-15T22:23:02.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.443 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.444 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.026000+0000", 2021-03-15T22:23:02.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053171069999999997 2021-03-15T22:23:02.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031317+0000", 2021-03-15T22:23:02.445 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8739999999999997e-06 2021-03-15T22:23:02.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.446 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.446 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.446 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031321+0000", 2021-03-15T22:23:02.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3403e-05 2021-03-15T22:23:02.446 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.447 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.447 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031334+0000", 2021-03-15T22:23:02.447 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1370000000000001e-06 2021-03-15T22:23:02.448 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.448 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.448 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031339+0000", 2021-03-15T22:23:02.449 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3701000000000001e-05 2021-03-15T22:23:02.449 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.449 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.449 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.449 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031383+0000", 2021-03-15T22:23:02.450 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0068000000000002e-05 2021-03-15T22:23:02.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.450 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.031403+0000", 2021-03-15T22:23:02.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061123979999999998 2021-03-15T22:23:02.451 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.451 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.452 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.037515+0000", 2021-03-15T22:23:02.452 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6636999999999999e-05 2021-03-15T22:23:02.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.453 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:37.037542+0000", 2021-03-15T22:23:02.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011542325000000001 2021-03-15T22:23:02.453 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.454 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.454 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.455 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:680 7.5 7:ac84e656:::smithi07694878-31:head [read 0~3226911,checksum] snapc 0=[] ondisk+read+known_if_redirected e137)", 2021-03-15T22:23:02.455 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:52.069145+0000", 2021-03-15T22:23:02.455 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 130.27231349499999, 2021-03-15T22:23:02.455 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056771690000000001, 2021-03-15T22:23:02.456 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.456 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:23:02.456 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:23:02.456 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:23:02.457 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:02.457 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 680 2021-03-15T22:23:02.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.457 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069145+0000", 2021-03-15T22:23:02.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.459 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.459 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069145+0000", 2021-03-15T22:23:02.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5859999999999999e-06 2021-03-15T22:23:02.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.460 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069147+0000", 2021-03-15T22:23:02.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7990000000000001e-06 2021-03-15T22:23:02.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.461 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.461 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.462 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069149+0000", 2021-03-15T22:23:02.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9809999999999997e-06 2021-03-15T22:23:02.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.463 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.463 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069156+0000", 2021-03-15T22:23:02.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2540000000000003e-06 2021-03-15T22:23:02.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.463 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.464 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.464 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.069160+0000", 2021-03-15T22:23:02.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033422629999999998 2021-03-15T22:23:02.464 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.465 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.465 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.072503+0000", 2021-03-15T22:23:02.465 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9270999999999994e-05 2021-03-15T22:23:02.466 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.466 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.466 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.466 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.072582+0000", 2021-03-15T22:23:02.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022400150000000001 2021-03-15T22:23:02.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.467 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.467 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:52.074822+0000", 2021-03-15T22:23:02.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056771690000000001 2021-03-15T22:23:02.468 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.468 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.468 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.468 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.469 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:868 7.6 e152/91 7:63693c38:::smithi07694878-22:head v 152'176, mlcod=149'172)", 2021-03-15T22:23:02.469 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:07.032095+0000", 2021-03-15T22:23:02.469 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 115.309362953, 2021-03-15T22:23:02.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0089994459999999995, 2021-03-15T22:23:02.470 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.470 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.470 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.471 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.032095+0000", 2021-03-15T22:23:02.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.032095+0000", 2021-03-15T22:23:02.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044590350000000001 2021-03-15T22:23:02.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.473 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.473 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036554+0000", 2021-03-15T22:23:02.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4150000000000001e-06 2021-03-15T22:23:02.474 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.474 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.474 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036559+0000", 2021-03-15T22:23:02.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5784e-05 2021-03-15T22:23:02.475 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036574+0000", 2021-03-15T22:23:02.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9740000000000002e-06 2021-03-15T22:23:02.476 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.477 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.477 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036582+0000", 2021-03-15T22:23:02.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4443999999999995e-05 2021-03-15T22:23:02.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.478 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036647+0000", 2021-03-15T22:23:02.478 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1956999999999999e-05 2021-03-15T22:23:02.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.479 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.036679+0000", 2021-03-15T22:23:02.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004329612 2021-03-15T22:23:02.480 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.041008+0000", 2021-03-15T22:23:02.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6224999999999995e-05 2021-03-15T22:23:02.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.481 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:07.041095+0000", 2021-03-15T22:23:02.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0089994459999999995 2021-03-15T22:23:02.482 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.482 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.482 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.483 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.2.0:112 7.8 e176/91 7:1822d391:::smithi07694878-47:4d v 176'104, mlcod=175'101)", 2021-03-15T22:23:02.483 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:31.018380+0000", 2021-03-15T22:23:02.484 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 91.323078441000007, 2021-03-15T22:23:02.484 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055471119999999999, 2021-03-15T22:23:02.484 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.484 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.485 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.485 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018380+0000", 2021-03-15T22:23:02.486 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.486 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.487 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018380+0000", 2021-03-15T22:23:02.487 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4490000000000002e-06 2021-03-15T22:23:02.487 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.487 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.487 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.488 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018382+0000", 2021-03-15T22:23:02.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7350000000000001e-06 2021-03-15T22:23:02.488 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.488 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.489 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.489 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018385+0000", 2021-03-15T22:23:02.489 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4560000000000006e-06 2021-03-15T22:23:02.489 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.490 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.490 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.490 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018394+0000", 2021-03-15T22:23:02.490 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6590000000000001e-06 2021-03-15T22:23:02.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.491 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.491 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.491 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018401+0000", 2021-03-15T22:23:02.491 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7095000000000006e-05 2021-03-15T22:23:02.492 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.492 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.492 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.492 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018478+0000", 2021-03-15T22:23:02.493 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3722000000000001e-05 2021-03-15T22:23:02.493 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.493 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.494 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.018502+0000", 2021-03-15T22:23:02.494 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053930260000000004 2021-03-15T22:23:02.494 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.494 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.495 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.023895+0000", 2021-03-15T22:23:02.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1970000000000001e-05 2021-03-15T22:23:02.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.496 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.496 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.023927+0000", 2021-03-15T22:23:02.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055471119999999999 2021-03-15T22:23:02.497 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.497 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.497 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.497 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.498 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:1191 7.6 e176/175 7:61e0cc05:::smithi07694878-16:head v 176'225, mlcod=176'221)", 2021-03-15T22:23:02.498 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:31.032485+0000", 2021-03-15T22:23:02.498 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 91.308973202999994, 2021-03-15T22:23:02.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013396315000000001, 2021-03-15T22:23:02.499 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.499 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.499 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.499 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.500 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.500 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.032485+0000", 2021-03-15T22:23:02.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.501 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.032485+0000", 2021-03-15T22:23:02.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055944209999999996 2021-03-15T22:23:02.501 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.502 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.502 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038079+0000", 2021-03-15T22:23:02.502 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.549e-06 2021-03-15T22:23:02.503 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.503 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.503 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038084+0000", 2021-03-15T22:23:02.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6657999999999999e-05 2021-03-15T22:23:02.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.504 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.504 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.504 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038101+0000", 2021-03-15T22:23:02.505 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9969999999999998e-06 2021-03-15T22:23:02.505 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.505 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.505 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.505 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038109+0000", 2021-03-15T22:23:02.506 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2562999999999998e-05 2021-03-15T22:23:02.506 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.506 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.506 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.507 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038161+0000", 2021-03-15T22:23:02.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.62e-05 2021-03-15T22:23:02.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.507 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.508 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.508 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038187+0000", 2021-03-15T22:23:02.508 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076488060000000002 2021-03-15T22:23:02.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.509 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.509 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.509 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.045836+0000", 2021-03-15T22:23:02.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5120999999999997e-05 2021-03-15T22:23:02.509 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.510 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.510 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.510 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.045881+0000", 2021-03-15T22:23:02.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013396315000000001 2021-03-15T22:23:02.511 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.511 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.511 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.511 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.512 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:1555 7.6 e204/175 7:6a64397b:::smithi07694878-49:head v 204'276, mlcod=203'272)", 2021-03-15T22:23:02.512 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:59.040507+0000", 2021-03-15T22:23:02.512 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 63.300951484999999, 2021-03-15T22:23:02.512 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013965283, 2021-03-15T22:23:02.513 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.513 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.513 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.513 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.514 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.514 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.040507+0000", 2021-03-15T22:23:02.514 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.514 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.514 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.515 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.515 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.040507+0000", 2021-03-15T22:23:02.515 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.006416934 2021-03-15T22:23:02.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.516 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.516 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.046924+0000", 2021-03-15T22:23:02.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2510000000000002e-06 2021-03-15T22:23:02.517 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.517 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.517 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.046929+0000", 2021-03-15T22:23:02.518 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6387999999999999e-05 2021-03-15T22:23:02.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.518 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.518 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.519 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.046955+0000", 2021-03-15T22:23:02.519 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8990000000000007e-06 2021-03-15T22:23:02.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.520 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.046964+0000", 2021-03-15T22:23:02.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5887000000000001e-05 2021-03-15T22:23:02.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.521 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.047030+0000", 2021-03-15T22:23:02.521 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6874000000000001e-05 2021-03-15T22:23:02.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.047057+0000", 2021-03-15T22:23:02.522 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0073782819999999999 2021-03-15T22:23:02.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.054435+0000", 2021-03-15T22:23:02.523 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6767999999999998e-05 2021-03-15T22:23:02.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.524 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.524 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.054472+0000", 2021-03-15T22:23:02.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013965283 2021-03-15T22:23:02.525 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.525 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.525 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.526 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:1563 7.f e204/203 7:fee9f235:::smithi07694878-11:head v 204'176, mlcod=0'0)", 2021-03-15T22:23:02.526 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:59.068892+0000", 2021-03-15T22:23:02.526 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 63.272566454, 2021-03-15T22:23:02.527 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017115345000000001, 2021-03-15T22:23:02.527 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.527 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.527 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.528 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.528 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.068892+0000", 2021-03-15T22:23:02.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.529 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.529 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.529 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.529 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.068892+0000", 2021-03-15T22:23:02.530 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.007026872 2021-03-15T22:23:02.530 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.530 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.530 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.531 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.075919+0000", 2021-03-15T22:23:02.531 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8240000000000006e-06 2021-03-15T22:23:02.531 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.531 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.532 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.075927+0000", 2021-03-15T22:23:02.532 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4705000000000001e-05 2021-03-15T22:23:02.532 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.532 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.533 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.075962+0000", 2021-03-15T22:23:02.533 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1775e-05 2021-03-15T22:23:02.533 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.534 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.534 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.534 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.075974+0000", 2021-03-15T22:23:02.534 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0989999999999995e-05 2021-03-15T22:23:02.535 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.535 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.535 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.535 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.076055+0000", 2021-03-15T22:23:02.535 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3104000000000001e-05 2021-03-15T22:23:02.536 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.536 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.536 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.536 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.076078+0000", 2021-03-15T22:23:02.536 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0098892359999999992 2021-03-15T22:23:02.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.537 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.537 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.537 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.085967+0000", 2021-03-15T22:23:02.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9839000000000003e-05 2021-03-15T22:23:02.538 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.538 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:59.086007+0000", 2021-03-15T22:23:02.539 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017115345000000001 2021-03-15T22:23:02.539 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.539 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.540 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.540 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.540 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1595 7.5 7:a9545d62:::smithi07694878-1:head [write 400066~716205] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:23:02.541 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.249476+0000", 2021-03-15T22:23:02.541 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 61.091982567999999, 2021-03-15T22:23:02.541 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0064056670000000003, 2021-03-15T22:23:02.541 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.542 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.542 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:23:02.542 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:23:02.542 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:02.543 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1595 2021-03-15T22:23:02.543 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.543 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.543 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.544 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.544 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.249476+0000", 2021-03-15T22:23:02.544 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.544 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.545 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.545 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.249476+0000", 2021-03-15T22:23:02.545 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00080617099999999995 2021-03-15T22:23:02.546 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.546 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.546 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.546 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.250282+0000", 2021-03-15T22:23:02.546 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.924e-06 2021-03-15T22:23:02.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.547 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.547 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.250286+0000", 2021-03-15T22:23:02.548 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7218999999999999e-05 2021-03-15T22:23:02.548 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.548 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.548 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.549 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.250303+0000", 2021-03-15T22:23:02.549 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6789999999999994e-06 2021-03-15T22:23:02.549 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.549 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.549 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.250312+0000", 2021-03-15T22:23:02.550 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2695000000000006e-05 2021-03-15T22:23:02.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.551 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.250404+0000", 2021-03-15T22:23:02.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1489999999999994e-06 2021-03-15T22:23:02.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.552 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.552 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:23:02.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.250412+0000", 2021-03-15T22:23:02.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001723893 2021-03-15T22:23:02.553 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.553 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.553 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.252136+0000", 2021-03-15T22:23:02.554 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7341000000000002e-05 2021-03-15T22:23:02.554 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.554 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.554 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.554 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.252234+0000", 2021-03-15T22:23:02.555 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021959200000000001 2021-03-15T22:23:02.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.555 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.555 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:23:02.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.252453+0000", 2021-03-15T22:23:02.556 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001752717 2021-03-15T22:23:02.556 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.557 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:23:02.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.254206+0000", 2021-03-15T22:23:02.557 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001549 2021-03-15T22:23:02.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.558 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:02.558 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.255755+0000", 2021-03-15T22:23:02.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5952999999999999e-05 2021-03-15T22:23:02.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:02.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.255801+0000", 2021-03-15T22:23:02.559 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6021e-05 2021-03-15T22:23:02.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.560 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.560 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.255837+0000", 2021-03-15T22:23:02.561 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4313000000000001e-05 2021-03-15T22:23:02.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.561 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.561 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.255881+0000", 2021-03-15T22:23:02.562 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0064056670000000003 2021-03-15T22:23:02.562 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.562 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.562 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.563 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:1686 7.f e220/203 7:fee9f235:::smithi07694878-11:head v 220'199, mlcod=217'196)", 2021-03-15T22:23:02.563 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:15.046367+0000", 2021-03-15T22:23:02.563 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 47.295091548000002, 2021-03-15T22:23:02.564 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019333758999999999, 2021-03-15T22:23:02.564 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.564 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.565 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.566 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.566 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.046367+0000", 2021-03-15T22:23:02.567 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.567 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.568 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.046367+0000", 2021-03-15T22:23:02.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060220710000000004 2021-03-15T22:23:02.568 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.568 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.568 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.569 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.052389+0000", 2021-03-15T22:23:02.569 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.289e-06 2021-03-15T22:23:02.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.569 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.571 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.052396+0000", 2021-03-15T22:23:02.571 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6400000000000001e-05 2021-03-15T22:23:02.571 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.571 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.572 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.572 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.052422+0000", 2021-03-15T22:23:02.572 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0732e-05 2021-03-15T22:23:02.572 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.572 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.573 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.573 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.052433+0000", 2021-03-15T22:23:02.573 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9537999999999996e-05 2021-03-15T22:23:02.573 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.573 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.574 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.574 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.052523+0000", 2021-03-15T22:23:02.575 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5561000000000001e-05 2021-03-15T22:23:02.575 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.575 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.575 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.052548+0000", 2021-03-15T22:23:02.576 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013095804000000001 2021-03-15T22:23:02.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.577 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.577 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.065644+0000", 2021-03-15T22:23:02.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6363999999999998e-05 2021-03-15T22:23:02.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.578 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.578 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.065700+0000", 2021-03-15T22:23:02.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019333758999999999 2021-03-15T22:23:02.579 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.579 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.579 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.580 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.580 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.580 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:1684 7.8 e220/91 7:1d74a1c3:::smithi07694878-44:head v 220'116, mlcod=183'112)", 2021-03-15T22:23:02.580 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:15.053350+0000", 2021-03-15T22:23:02.580 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 47.288107781999997, 2021-03-15T22:23:02.581 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012562525999999999, 2021-03-15T22:23:02.581 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.581 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.581 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.582 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.582 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.053350+0000", 2021-03-15T22:23:02.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.582 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.583 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.583 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.583 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.053350+0000", 2021-03-15T22:23:02.583 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047345360000000001 2021-03-15T22:23:02.583 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.584 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.584 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.584 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.058085+0000", 2021-03-15T22:23:02.584 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1070000000000001e-06 2021-03-15T22:23:02.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.586 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.058091+0000", 2021-03-15T22:23:02.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4290999999999998e-05 2021-03-15T22:23:02.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.587 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.058115+0000", 2021-03-15T22:23:02.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.101e-05 2021-03-15T22:23:02.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.588 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.058126+0000", 2021-03-15T22:23:02.589 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6981000000000002e-05 2021-03-15T22:23:02.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.589 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.589 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.590 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.058183+0000", 2021-03-15T22:23:02.590 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4841000000000002e-05 2021-03-15T22:23:02.590 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.590 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.591 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.058208+0000", 2021-03-15T22:23:02.591 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076592830000000002 2021-03-15T22:23:02.591 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.591 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.592 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.592 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.065867+0000", 2021-03-15T22:23:02.592 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5476999999999998e-05 2021-03-15T22:23:02.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.592 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.593 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.593 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:15.065913+0000", 2021-03-15T22:23:02.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012562525999999999 2021-03-15T22:23:02.593 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.594 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.595 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.596 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1723 7.5 7:a9545d62:::smithi07694878-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e225)", 2021-03-15T22:23:02.596 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:20.040053+0000", 2021-03-15T22:23:02.596 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 42.301405228999997, 2021-03-15T22:23:02.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053817090000000001, 2021-03-15T22:23:02.597 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.598 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:23:02.598 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:23:02.599 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:23:02.599 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:02.599 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1723 2021-03-15T22:23:02.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.600 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.600 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.600 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.040053+0000", 2021-03-15T22:23:02.600 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.601 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.601 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.601 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.040053+0000", 2021-03-15T22:23:02.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3700000000000002e-06 2021-03-15T22:23:02.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.602 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.602 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.602 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.040055+0000", 2021-03-15T22:23:02.603 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2330000000000001e-06 2021-03-15T22:23:02.603 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.603 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.603 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.040058+0000", 2021-03-15T22:23:02.604 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5534999999999999e-05 2021-03-15T22:23:02.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.604 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.604 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.040083+0000", 2021-03-15T22:23:02.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4449999999999999e-06 2021-03-15T22:23:02.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.605 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.040091+0000", 2021-03-15T22:23:02.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5917e-05 2021-03-15T22:23:02.606 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.606 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.607 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.040146+0000", 2021-03-15T22:23:02.607 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2310000000000002e-06 2021-03-15T22:23:02.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.607 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.607 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:23:02.608 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.040156+0000", 2021-03-15T22:23:02.608 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029418629999999999 2021-03-15T22:23:02.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.609 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.609 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.043098+0000", 2021-03-15T22:23:02.609 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0404000000000001e-05 2021-03-15T22:23:02.609 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.609 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.610 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:23:02.610 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.043138+0000", 2021-03-15T22:23:02.610 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002190362 2021-03-15T22:23:02.610 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.611 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.611 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.611 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.045328+0000", 2021-03-15T22:23:02.611 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3567999999999999e-05 2021-03-15T22:23:02.611 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.612 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.612 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.612 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.045382+0000", 2021-03-15T22:23:02.612 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2781000000000002e-05 2021-03-15T22:23:02.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.613 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.613 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.613 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:20.045435+0000", 2021-03-15T22:23:02.613 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053817090000000001 2021-03-15T22:23:02.613 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.614 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.614 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.614 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.615 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:2307 7.f e265/263 7:fc7cd984:::smithi07694878-4:head v 265'257, mlcod=264'253)", 2021-03-15T22:23:02.615 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:00.304187+0000", 2021-03-15T22:23:02.615 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 2.0372710089999999, 2021-03-15T22:23:02.615 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070446609999999998, 2021-03-15T22:23:02.615 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.616 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.616 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.616 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.616 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.616 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:00.304187+0000", 2021-03-15T22:23:02.617 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.617 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.617 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:00.304187+0000", 2021-03-15T22:23:02.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00388879 2021-03-15T22:23:02.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.618 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.618 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.619 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:00.308076+0000", 2021-03-15T22:23:02.619 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6859999999999997e-06 2021-03-15T22:23:02.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.620 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.620 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:00.308081+0000", 2021-03-15T22:23:02.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3842999999999999e-05 2021-03-15T22:23:02.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:00.308094+0000", 2021-03-15T22:23:02.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.255e-06 2021-03-15T22:23:02.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.622 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:00.308102+0000", 2021-03-15T22:23:02.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5806999999999998e-05 2021-03-15T22:23:02.623 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.623 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.623 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:00.308138+0000", 2021-03-15T22:23:02.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6415000000000001e-05 2021-03-15T22:23:02.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.624 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:00.308154+0000", 2021-03-15T22:23:02.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003047386 2021-03-15T22:23:02.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.625 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:00.311201+0000", 2021-03-15T22:23:02.626 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0479000000000001e-05 2021-03-15T22:23:02.626 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.626 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.626 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.627 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:00.311232+0000", 2021-03-15T22:23:02.627 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070446609999999998 2021-03-15T22:23:02.627 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.627 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.627 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.628 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.628 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.628 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:2313 7.5 7:ac84e656:::smithi07694878-31:head [copy-from ver 151] snapc a6=[a6,a2,a0,9e,9c,93,8d,7d] ondisk+write+known_if_redirected e266)", 2021-03-15T22:23:02.628 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:01.061013+0000", 2021-03-15T22:23:02.628 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.2804451210000001, 2021-03-15T22:23:02.629 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025357358, 2021-03-15T22:23:02.629 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.629 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:02.629 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:23:02.629 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:23:02.630 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:02.630 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2313 2021-03-15T22:23:02.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.630 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.631 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.631 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061013+0000", 2021-03-15T22:23:02.631 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.632 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061013+0000", 2021-03-15T22:23:02.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0879999999999999e-06 2021-03-15T22:23:02.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061016+0000", 2021-03-15T22:23:02.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.993e-06 2021-03-15T22:23:02.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.634 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061019+0000", 2021-03-15T22:23:02.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6614000000000001e-05 2021-03-15T22:23:02.635 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061036+0000", 2021-03-15T22:23:02.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0029999999999997e-06 2021-03-15T22:23:02.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.636 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061043+0000", 2021-03-15T22:23:02.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00042638000000000002 2021-03-15T22:23:02.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.637 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.637 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061469+0000", 2021-03-15T22:23:02.638 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000110006 2021-03-15T22:23:02.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.638 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.638 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061579+0000", 2021-03-15T22:23:02.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005380045 2021-03-15T22:23:02.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.639 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.639 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:23:02.639 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.066959+0000", 2021-03-15T22:23:02.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042905870000000002 2021-03-15T22:23:02.640 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:23:02.641 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.071250+0000", 2021-03-15T22:23:02.641 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011425196 2021-03-15T22:23:02.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.641 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:02.642 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.082675+0000", 2021-03-15T22:23:02.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003569727 2021-03-15T22:23:02.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.642 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.642 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:02.643 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.086245+0000", 2021-03-15T22:23:02.643 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4766999999999998e-05 2021-03-15T22:23:02.643 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.643 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.644 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:23:02.644 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.086309+0000", 2021-03-15T22:23:02.644 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0952000000000002e-05 2021-03-15T22:23:02.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.644 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.645 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.086370+0000", 2021-03-15T22:23:02.645 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025357358 2021-03-15T22:23:02.645 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.645 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.646 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.646 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.646 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:2314 7.5 7:ac84e656:::smithi07694878-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e266)", 2021-03-15T22:23:02.647 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:01.061444+0000", 2021-03-15T22:23:02.647 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.280014531, 2021-03-15T22:23:02.647 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025096124000000001, 2021-03-15T22:23:02.647 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:23:02.647 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:23:02.648 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:23:02.648 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:23:02.648 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:02.648 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2314 2021-03-15T22:23:02.648 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.649 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:23:02.649 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.649 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:23:02.649 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061444+0000", 2021-03-15T22:23:02.649 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:23:02.650 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.650 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:23:02.650 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061444+0000", 2021-03-15T22:23:02.651 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6819999999999999e-06 2021-03-15T22:23:02.651 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.651 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.651 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:23:02.651 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061445+0000", 2021-03-15T22:23:02.652 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1969999999999999e-06 2021-03-15T22:23:02.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:23:02.652 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061448+0000", 2021-03-15T22:23:02.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0861e-05 2021-03-15T22:23:02.653 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.653 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.653 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:23:02.654 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061458+0000", 2021-03-15T22:23:02.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3199999999999999e-06 2021-03-15T22:23:02.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:23:02.655 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061464+0000", 2021-03-15T22:23:02.655 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00026995600000000002 2021-03-15T22:23:02.655 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.655 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.656 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.656 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061734+0000", 2021-03-15T22:23:02.656 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9314999999999999e-05 2021-03-15T22:23:02.656 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.657 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:23:02.657 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.061773+0000", 2021-03-15T22:23:02.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0075660459999999999 2021-03-15T22:23:02.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.658 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.069339+0000", 2021-03-15T22:23:02.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4104999999999998e-05 2021-03-15T22:23:02.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.659 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:23:02.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.069393+0000", 2021-03-15T22:23:02.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017000189999999998 2021-03-15T22:23:02.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.660 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:23:02.660 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.086393+0000", 2021-03-15T22:23:02.660 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7005000000000001e-05 2021-03-15T22:23:02.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:23:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.086450+0000", 2021-03-15T22:23:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9446999999999999e-05 2021-03-15T22:23:02.661 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:23:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:23:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:23:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:01.086540+0000", 2021-03-15T22:23:02.662 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025096124000000001 2021-03-15T22:23:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:23:02.663 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:23:02.664 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:23:02.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:02.665 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:02 smithi125 conmon[56739]: cluster 2021-03-15T22:23:01.018096+0000 mgr.y ( 2021-03-15T22:23:02.666 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:02 smithi125 conmon[56739]: mgr.14144) 502 : cluster [DBG] pgmap v525: 138 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 peering, 131 active+clean; 201 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.2 MiB/s wr, 19 op/s; 2/1263 objects degraded (0.158%); 56/1263 objects misplaced (4.434%); 5.1 MiB/s, 13 objects/s recovering 2021-03-15T22:23:02.666 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:02 smithi125 conmon[56739]: cluster 2021-03-15T22:23:02.058659+0000 mon.a (mon.0) 950 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2021-03-15T22:23:02.667 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:02.668 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:02 smithi076 conmon[56514]: cluster 2021-03-15T22:23:01. 2021-03-15T22:23:02.668 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:02 smithi076 conmon[56514]: 018096+0000 mgr.y (mgr.14144) 502 : cluster [DBG] pgmap v525: 138 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 peering, 131 active+clean; 201 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.2 MiB/s wr, 19 op/s; 2/1263 objects degraded (0.158%); 56/1263 objects misplaced (4.434%); 5.1 MiB/s, 13 objects/s recovering 2021-03-15T22:23:02.668 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:02 smithi076 conmon[56514]: cluster 2021-03-15T22:23:02.058659+0000 mon.a (mon.0) 950 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2021-03-15T22:23:02.668 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:02 smithi076 conmon[62156]: cluster 2021-03-15T22:23:01.018096+0000 mgr.y (mgr.14144 2021-03-15T22:23:02.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:02 smithi076 conmon[62156]: ) 502 : cluster [DBG] pgmap v525: 138 pgs: 1 active+recovery_wait+remapped, 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 3 peering, 131 active+clean; 201 MiB data, 605 MiB used, 715 GiB / 715 GiB avail; 6.9 MiB/s rd, 2.2 MiB/s wr, 19 op/s; 2/1263 objects degraded (0.158%); 56/1263 objects misplaced (4.434%); 5.1 MiB/s, 13 objects/s recovering 2021-03-15T22:23:02.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:02 smithi076 conmon[62156]: cluster 2021-03-15T22:23:02.058659+0000 mon.a (mon. 2021-03-15T22:23:02.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:02 smithi076 conmon[62156]: 0) 950 : cluster [DBG] osdmap e267: 8 total, 8 up, 8 in 2021-03-15T22:23:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:02.690 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:02.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:02.792 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:02.816 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:02.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:02.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:02.879 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:23:02.898 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:23:02.919 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:02.923 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:23:02.941 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:23:02.942 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:02.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:02.944 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:02.958 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:23:02.976 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:23:02.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:23:02.994 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:23:03.011 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:23:03.012 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:23:03.012 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:23:03.012 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:23:03.013 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:23:03.013 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:23:03.013 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:23:03.013 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:23:03.014 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:23:03.045 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:03.063 INFO:tasks.rados.rados.0.smithi076.stdout:888: done (3 left) 2021-03-15T22:23:03.064 INFO:tasks.rados.rados.0.smithi076.stdout:889: read oid 18 snap 72 2021-03-15T22:23:03.064 INFO:tasks.rados.rados.0.smithi076.stdout:889: expect (ObjNum 217 snap 50 seq_num 217) 2021-03-15T22:23:03.064 INFO:tasks.rados.rados.0.smithi076.stdout:890: read oid 2 snap -1 2021-03-15T22:23:03.064 INFO:tasks.rados.rados.0.smithi076.stdout:890: expect (ObjNum 295 snap 78 seq_num 295) 2021-03-15T22:23:03.064 INFO:tasks.rados.rados.0.smithi076.stdout:891: delete oid 21 current snap is 88 2021-03-15T22:23:03.065 INFO:tasks.rados.rados.0.smithi076.stdout:885: finishing write tid 3 to smithi07694878-11 2021-03-15T22:23:03.065 INFO:tasks.rados.rados.0.smithi076.stdout:885: finishing write tid 4 to smithi07694878-11 2021-03-15T22:23:03.065 INFO:tasks.rados.rados.0.smithi076.stdout:886: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:23:03.072 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:03.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:03.174 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:03.196 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:03.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:03.222 INFO:tasks.rados.rados.0.smithi076.stdout:885: finishing write tid 5 to smithi07694878-11 2021-03-15T22:23:03.223 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 406 (ObjNum 328 snap 88 seq_num 328) dirty exists 2021-03-15T22:23:03.223 INFO:tasks.rados.rados.0.smithi076.stdout:885: left oid 11 (ObjNum 328 snap 88 seq_num 328) 2021-03-15T22:23:03.223 INFO:tasks.rados.rados.0.smithi076.stdout:887: finishing write tid 1 to smithi07694878-49 2021-03-15T22:23:03.223 INFO:tasks.rados.rados.0.smithi076.stdout:887: finishing write tid 2 to smithi07694878-49 2021-03-15T22:23:03.223 INFO:tasks.rados.rados.0.smithi076.stdout:887: finishing write tid 3 to smithi07694878-49 2021-03-15T22:23:03.224 INFO:tasks.rados.rados.0.smithi076.stdout:887: finishing write tid 4 to smithi07694878-49 2021-03-15T22:23:03.224 INFO:tasks.rados.rados.0.smithi076.stdout:887: finishing write tid 5 to smithi07694878-49 2021-03-15T22:23:03.224 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 417 (ObjNum 329 snap 88 seq_num 329) dirty exists 2021-03-15T22:23:03.224 INFO:tasks.rados.rados.0.smithi076.stdout:887: left oid 49 (ObjNum 329 snap 88 seq_num 329) 2021-03-15T22:23:03.225 INFO:tasks.rados.rados.0.smithi076.stdout:890: expect (ObjNum 295 snap 78 seq_num 295) 2021-03-15T22:23:03.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:03.319 INFO:tasks.rados.rados.0.smithi076.stdout:889: expect (ObjNum 217 snap 50 seq_num 217) 2021-03-15T22:23:03.321 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:03.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:03.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:03.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:03.446 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:03.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:03.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:03.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:03 smithi076 conmon[56514]: cluster 2021-03-15T22:23:02.474227+0000 osd.2 (osd.2) 49 : cluster [DBG] 6.f deep-scrub ok 2021-03-15T22:23:03.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:03 smithi076 conmon[56514]: audit 2021-03-15T22:23:02.859189+0000 mon.a (mon.0) 951 : audit [DBG] from='client.? 172.21.15.76:0/3623084968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:03.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:03 smithi076 conmon[56514]: cluster 2021-03-15T22:23:03.061746+0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2021-03-15T22:23:03.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:03 smithi076 conmon[62156]: cluster 2021-03-15T22:23:02.474227+0000 osd.2 (osd.2) 49 : cluster [DBG] 6.f deep-scrub ok 2021-03-15T22:23:03.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:03 smithi076 conmon[62156]: audit 2021-03-15T22:23:02.859189+0000 mon.a (mon.0) 951 : audit [DBG] from='client.? 172.21.15.76:0/3623084968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:03.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:03 smithi076 conmon[62156]: cluster 2021-03-15T22:23:03.061746+0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2021-03-15T22:23:03.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:03 smithi125 conmon[56739]: cluster 2021-03-15T22:23:02.474227+0000 osd.2 (osd.2) 49 : cluster [DBG] 6.f deep-scrub ok 2021-03-15T22:23:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:03 smithi125 conmon[56739]: audit 2021-03-15T22:23:02.859189+0000 mon.a (mon.0) 951 : audit [DBG] from='client.? 172.21.15.76:0/3623084968' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:03 smithi125 conmon[56739]: cluster 2021-03-15T22:23:03.061746+0000 mon.a (mon.0) 952 : cluster [DBG] osdmap e268: 8 total, 8 up, 8 in 2021-03-15T22:23:03.549 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:03.574 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:03.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:03.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:03.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:03.699 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:03.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:03.801 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:03.823 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:03.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:03.925 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:03.947 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:03.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:03.949 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:04.050 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:04.074 INFO:tasks.rados.rados.0.smithi076.stdout:885: done (5 left) 2021-03-15T22:23:04.074 INFO:tasks.rados.rados.0.smithi076.stdout:886: done (4 left) 2021-03-15T22:23:04.074 INFO:tasks.rados.rados.0.smithi076.stdout:887: done (3 left) 2021-03-15T22:23:04.075 INFO:tasks.rados.rados.0.smithi076.stdout:889: done (2 left) 2021-03-15T22:23:04.075 INFO:tasks.rados.rados.0.smithi076.stdout:890: done (1 left) 2021-03-15T22:23:04.075 INFO:tasks.rados.rados.0.smithi076.stdout:891: done (0 left) 2021-03-15T22:23:04.075 INFO:tasks.rados.rados.0.smithi076.stdout:892: copy_from oid 37 from oid 11 current snap is 88 2021-03-15T22:23:04.075 INFO:tasks.rados.rados.0.smithi076.stdout:893: write (excl) oid 25 current snap is 88 2021-03-15T22:23:04.076 INFO:tasks.rados.rados.0.smithi076.stdout:893: seq_num 331 ranges {763117=710598,2136128=720101,3390187=172930} 2021-03-15T22:23:04.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:04.077 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:04.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:04.082 INFO:tasks.rados.rados.0.smithi076.stdout:893: writing smithi07694878-25 from 763117 to 1473715 tid 1 2021-03-15T22:23:04.090 INFO:tasks.rados.rados.0.smithi076.stdout:893: writing smithi07694878-25 from 2136128 to 2856229 tid 2 2021-03-15T22:23:04.095 INFO:tasks.rados.rados.0.smithi076.stdout:893: writing smithi07694878-25 from 3390187 to 3563117 tid 3 2021-03-15T22:23:04.095 INFO:tasks.rados.rados.0.smithi076.stdout:894: copy_from oid 48 from oid 27 current snap is 88 2021-03-15T22:23:04.095 INFO:tasks.rados.rados.0.smithi076.stdout:893: finishing write tid 1 to smithi07694878-25 2021-03-15T22:23:04.096 INFO:tasks.rados.rados.0.smithi076.stdout:895: rollback oid 46 current snap is 88 2021-03-15T22:23:04.096 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 46 to 63 2021-03-15T22:23:04.096 INFO:tasks.rados.rados.0.smithi076.stdout:896: write oid 45 current snap is 88 2021-03-15T22:23:04.096 INFO:tasks.rados.rados.0.smithi076.stdout:896: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2021-03-15T22:23:04.103 INFO:tasks.rados.rados.0.smithi076.stdout:896: writing smithi07694878-45 from 779924 to 1565771 tid 1 2021-03-15T22:23:04.109 INFO:tasks.rados.rados.0.smithi076.stdout:896: writing smithi07694878-45 from 2078257 to 2658369 tid 2 2021-03-15T22:23:04.115 INFO:tasks.rados.rados.0.smithi076.stdout:896: writing smithi07694878-45 from 3301257 to 3579924 tid 3 2021-03-15T22:23:04.116 INFO:tasks.rados.rados.0.smithi076.stdout:897: delete oid 17 current snap is 88 2021-03-15T22:23:04.116 INFO:tasks.rados.rados.0.smithi076.stdout:894: finishing copy_from to smithi07694878-48 2021-03-15T22:23:04.116 INFO:tasks.rados.rados.0.smithi076.stdout:894: got expected ENOENT (src dne) 2021-03-15T22:23:04.116 INFO:tasks.rados.rados.0.smithi076.stdout:894: finishing copy_from racing read to smithi07694878-48 2021-03-15T22:23:04.117 INFO:tasks.rados.rados.0.smithi076.stdout:895: finishing rollback tid 0 to smithi07694878-46 2021-03-15T22:23:04.117 INFO:tasks.rados.rados.0.smithi076.stdout:895: finishing rollback tid 1 to smithi07694878-46 2021-03-15T22:23:04.117 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 196 (ObjNum 261676496 snap 21896 seq_num 261694160) dirty dne 2021-03-15T22:23:04.117 INFO:tasks.rados.rados.0.smithi076.stdout:893: finishing write tid 2 to smithi07694878-25 2021-03-15T22:23:04.118 INFO:tasks.rados.rados.0.smithi076.stdout:893: finishing write tid 3 to smithi07694878-25 2021-03-15T22:23:04.118 INFO:tasks.rados.rados.0.smithi076.stdout:893: finishing write tid 4 to smithi07694878-25 2021-03-15T22:23:04.118 INFO:tasks.rados.rados.0.smithi076.stdout:893: finishing write tid 5 to smithi07694878-25 2021-03-15T22:23:04.118 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 138 (ObjNum 331 snap 88 seq_num 331) dirty exists 2021-03-15T22:23:04.118 INFO:tasks.rados.rados.0.smithi076.stdout:893: left oid 25 (ObjNum 331 snap 88 seq_num 331) 2021-03-15T22:23:04.119 INFO:tasks.rados.rados.0.smithi076.stdout:892: finishing copy_from to smithi07694878-37 2021-03-15T22:23:04.119 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 406 (ObjNum 328 snap 88 seq_num 328) dirty exists 2021-03-15T22:23:04.119 INFO:tasks.rados.rados.0.smithi076.stdout:892: finishing copy_from racing read to smithi07694878-37 2021-03-15T22:23:04.119 INFO:tasks.rados.rados.0.smithi076.stdout:896: finishing write tid 1 to smithi07694878-45 2021-03-15T22:23:04.119 INFO:tasks.rados.rados.0.smithi076.stdout:896: finishing write tid 2 to smithi07694878-45 2021-03-15T22:23:04.120 INFO:tasks.rados.rados.0.smithi076.stdout:896: finishing write tid 3 to smithi07694878-45 2021-03-15T22:23:04.121 INFO:tasks.rados.rados.0.smithi076.stdout:896: finishing write tid 4 to smithi07694878-45 2021-03-15T22:23:04.121 INFO:tasks.rados.rados.0.smithi076.stdout:896: finishing write tid 5 to smithi07694878-45 2021-03-15T22:23:04.121 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 253 (ObjNum 332 snap 88 seq_num 332) dirty exists 2021-03-15T22:23:04.121 INFO:tasks.rados.rados.0.smithi076.stdout:896: left oid 45 (ObjNum 332 snap 88 seq_num 332) 2021-03-15T22:23:04.122 INFO:tasks.rados.rados.0.smithi076.stdout:892: done (5 left) 2021-03-15T22:23:04.122 INFO:tasks.rados.rados.0.smithi076.stdout:893: done (4 left) 2021-03-15T22:23:04.122 INFO:tasks.rados.rados.0.smithi076.stdout:894: done (3 left) 2021-03-15T22:23:04.122 INFO:tasks.rados.rados.0.smithi076.stdout:895: done (2 left) 2021-03-15T22:23:04.123 INFO:tasks.rados.rados.0.smithi076.stdout:896: done (1 left) 2021-03-15T22:23:04.123 INFO:tasks.rados.rados.0.smithi076.stdout:897: done (0 left) 2021-03-15T22:23:04.123 INFO:tasks.rados.rados.0.smithi076.stdout:898: snap_remove snap 84 2021-03-15T22:23:04.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:04.202 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:04.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:04.204 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:04.305 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:04.327 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:04.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:04.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:04.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:04.452 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:04.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:04.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:04.489 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: cluster 2021-03-15T22:23:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:23:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: 23:03.018550+0000 mgr.y (mgr.14144) 503 : cluster [DBG] pgmap v528: 138 pgs: 1 active+undersized+remapped, 1 active+clean+wait, 1 active+remapped, 135 active+clean; 216 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 37 MiB/s rd, 4.5 MiB/s wr, 46 op/s; 20 MiB/s, 44 objects/s recovering 2021-03-15T22:23:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: cluster 2021-03-15T22:23:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: 2021-03-15T22:23:03. 2021-03-15T22:23:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: 245437+0000 osd.1 (osd.1) 58 : cluster 2021-03-15T22:23:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: [DBG] 4.12 scrub ok 2021-03-15T22:23:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:23:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: -15T22:23:03.261786+0000 mon.a (mon.0) 953 : cluster 2021-03-15T22:23:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/1263 objects degraded (0.158%), 2 pgs degraded) 2021-03-15T22:23:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: cluster 2021-03-15T22:23:04.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: 2021-03-15T22:23:03.261814 2021-03-15T22:23:04.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: +0000 mon.a (mon.0) 954 2021-03-15T22:23:04.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: : cluster [INF] Cluster is now healthy 2021-03-15T22:23:04.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:23:04.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: 15T22:23:03.482302+0000 osd.2 (osd. 2021-03-15T22:23:04.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: 2) 50 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:23:04.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: 2021-03-15T22:23:04.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: cluster 2021-03-15T22:23:04.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: 2021-03-15T22:23:04.065298+0000 2021-03-15T22:23:04.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: mon.a (mon.0) 955 : cluster [DBG] 2021-03-15T22:23:04.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:04 smithi125 conmon[56739]: osdmap e269: 8 total, 8 up, 8 in 2021-03-15T22:23:04.555 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:04.577 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:04.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:04.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:04.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[56514]: cluster 2021-03-15T22:23:03.018550+0000 2021-03-15T22:23:04.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[56514]: mgr.y (mgr.14144) 503 : cluster [DBG] pgmap v528: 138 pgs: 1 active+undersized+remapped, 1 active+clean+wait, 1 active+remapped, 135 active+clean; 216 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 37 MiB/s rd, 4.5 MiB/s wr, 46 op/s; 20 MiB/s, 44 objects/s recovering 2021-03-15T22:23:04.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[56514]: cluster 2021-03-15T22:23:03.245437+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:23:04.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[56514]: cluster 2021-03-15T22:23:03.261786+0000 mon.a (mon.0) 953 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/1263 objects degraded (0.158%), 2 pgs degraded) 2021-03-15T22:23:04.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[56514]: cluster 2021-03-15T22:23:03.261814+0000 mon.a (mon.0) 954 : cluster [INF] Cluster is now healthy 2021-03-15T22:23:04.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[56514]: cluster 2021-03-15T22:23:03.482302+0000 osd.2 (osd.2) 50 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:23:04.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[56514]: cluster 2021-03-15T22:23:04.065298+0000 mon.a (mon.0) 955 : cluster [DBG] osdmap e269: 8 total, 8 up, 8 in 2021-03-15T22:23:04.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: cluster 2021-03-15T22:23:03.018550+0000 2021-03-15T22:23:04.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: mgr.y (mgr.14144) 503 : cluster [DBG] pgmap v528: 138 pgs: 1 active+undersized+remapped, 1 active+clean+wait, 1 active+remapped, 135 active+clean; 216 MiB data, 693 MiB used, 715 GiB / 715 GiB avail; 37 MiB/s rd, 4.5 MiB/s wr, 46 op/s; 20 MiB/s, 44 objects/s recovering 2021-03-15T22:23:04.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: cluster 2021-03-15T22:23:03 2021-03-15T22:23:04.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: .245437+0000 osd.1 (osd.1) 58 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:23:04.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:23:04.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: :23:03.261786+0000 mon.a (mon.0) 953 : cluster 2021-03-15T22:23:04.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/1263 objects degraded (0.158%), 2 pgs degraded) 2021-03-15T22:23:04.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: cluster 2021-03-15T22:23:04.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: 2021-03-15T22:23:03.261814+0000 mon.a (mon 2021-03-15T22:23:04.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: .0) 954 : cluster [INF] Cluster is now healthy 2021-03-15T22:23:04.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: cluster 2021-03-15T22:23:04.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:23:04.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: 22:23:03.482302+0000 osd.2 ( 2021-03-15T22:23:04.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: osd.2) 50 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:23:04.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: 2021-03-15T22:23:04.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: cluster 2021-03-15T22:23:04.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: 2021-03-15T22:23:04.065298+0000 mon.a 2021-03-15T22:23:04.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: (mon.0) 955 : cluster [DBG] 2021-03-15T22:23:04.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:04 smithi076 conmon[62156]: osdmap e269: 8 total, 8 up, 8 in 2021-03-15T22:23:04.668 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:23:04.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:04.701 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:04.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:04.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:04.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:04.826 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:04.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:04.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:04.929 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:04.952 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:04.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:05.054 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:05.079 INFO:tasks.rados.rados.0.smithi076.stdout:898: done (0 left) 2021-03-15T22:23:05.079 INFO:tasks.rados.rados.0.smithi076.stdout:899: write oid 8 current snap is 88 2021-03-15T22:23:05.079 INFO:tasks.rados.rados.0.smithi076.stdout:899: seq_num 334 ranges {413538=536345,1395221=583781,2556103=658617} 2021-03-15T22:23:05.084 INFO:tasks.rados.rados.0.smithi076.stdout:899: writing smithi07694878-8 from 413538 to 949883 tid 1 2021-03-15T22:23:05.090 INFO:tasks.rados.rados.0.smithi076.stdout:899: writing smithi07694878-8 from 1395221 to 1979002 tid 2 2021-03-15T22:23:05.091 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:05.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:05.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:05.098 INFO:tasks.rados.rados.0.smithi076.stdout:899: writing smithi07694878-8 from 2556103 to 3214720 tid 3 2021-03-15T22:23:05.098 INFO:tasks.rados.rados.0.smithi076.stdout:900: read oid 21 snap 81 2021-03-15T22:23:05.098 INFO:tasks.rados.rados.0.smithi076.stdout:900: expect (ObjNum 263 snap 68 seq_num 263) 2021-03-15T22:23:05.098 INFO:tasks.rados.rados.0.smithi076.stdout:901: snap_remove snap 86 2021-03-15T22:23:05.193 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:05.216 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:05.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:05.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:05.318 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:05.333 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:05.340 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:05.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:05.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:05.355 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 414 (0%) 2021-03-15T22:23:05.355 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:23:05.356 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 11.1327 -> 11.1327 (expected baseline 6.72913) 2021-03-15T22:23:05.356 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 25 -> 25 pgs (0.483092 -> 0.483092 * mean) 2021-03-15T22:23:05.356 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 64 -> 64 pgs (1.23671 -> 1.23671 * mean) 2021-03-15T22:23:05.356 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:23:05.357 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:23:05.357 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 3 2021-03-15T22:23:05.357 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:23:05.357 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0009 2021-03-15T22:23:05.357 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0094 2021-03-15T22:23:05.358 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:23:05.443 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:05.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:05.468 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:05.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:05 smithi076 conmon[56514]: cluster 2021-03-15T22:23:03.888758+0000 osd.7 (osd.7) 58 : cluster [DBG] 3.f scrub ok 2021-03-15T22:23:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:05 smithi076 conmon[56514]: cluster 2021-03-15T22:23:04.268974+0000 osd.1 (osd.1) 59 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:23:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:05 smithi076 conmon[56514]: cluster 2021-03-15T22:23:04.612996+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.13 deep-scrub ok 2021-03-15T22:23:05.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:05 smithi076 conmon[56514]: cluster 2021-03-15T22:23:05.066916+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2021-03-15T22:23:05.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:05 smithi076 conmon[62156]: cluster 2021-03-15T22:23:03.888758+0000 2021-03-15T22:23:05.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:05 smithi076 conmon[62156]: osd.7 (osd.7) 58 : cluster [DBG] 3.f scrub ok 2021-03-15T22:23:05.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:05 smithi076 conmon[62156]: cluster 2021-03-15T22:23:04.268974+0000 osd.1 (osd.1) 59 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:23:05.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:05 smithi076 conmon[62156]: cluster 2021-03-15T22:23:04.612996+0000 osd.3 (osd.3) 50 : cluster [DBG] 5.13 deep-scrub ok 2021-03-15T22:23:05.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:05 smithi076 conmon[62156]: cluster 2021-03-15T22:23:05.066916+0000 mon.a (mon.0) 956 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2021-03-15T22:23:05.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: cluster 2021-03-15T22:23:03.888758+0000 osd.7 (osd. 2021-03-15T22:23:05.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: 7) 58 : cluster [DBG] 3.f scrub ok 2021-03-15T22:23:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: cluster 2021-03-15T22:23:04.268974+0000 osd.1 (osd.1) 59 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:23:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:23:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: -15T22:23:04. 2021-03-15T22:23:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: 612996+0000 osd.3 (osd.3) 2021-03-15T22:23:05.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: 50 : cluster [DBG] 5.13 deep-scrub ok 2021-03-15T22:23:05.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:23:05.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: -03-15T22:23:05 2021-03-15T22:23:05.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: .066916+0000 mon.a (mon. 2021-03-15T22:23:05.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: 0) 956 : cluster [DBG] osdmap e270: 8 total, 8 up, 8 in 2021-03-15T22:23:05.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:05 smithi125 conmon[56739]: 2021-03-15T22:23:05.569 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:05.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:05.593 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:05.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:05.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:05.717 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:05.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:05.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:05.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:05.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:05.842 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:05.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:05.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:05.965 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:05.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:05.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:05.984 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:23:06.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:06.069 INFO:tasks.rados.rados.0.smithi076.stdout:901: done (2 left) 2021-03-15T22:23:06.069 INFO:tasks.rados.rados.0.smithi076.stdout:902: snap_remove snap 83 2021-03-15T22:23:06.093 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:06.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:06.094 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:06.195 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:06.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:06.219 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:06.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:06.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:23:06.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[56514]: 23:05.018988+0000 mgr.y (mgr.14144) 504 : cluster [DBG] pgmap v531: 138 pgs: 1 active+undersized+remapped, 1 active+clean+wait, 1 active+remapped, 135 active+clean; 219 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 36 MiB/s rd, 3.8 MiB/s wr, 38 op/s; 15 MiB/s, 35 objects/s recovering 2021-03-15T22:23:06.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[56514]: audit 2021-03-15T22:23:05.353085+0000 mgr.y (mgr.14144) 505 : audit [DBG] from='client.15135 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:06.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[56514]: cluster 2021-03-15T22:23:05.519540+0000 osd.6 (osd.6) 42 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:23:06.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[56514]: cluster 2021-03-15T22:23:05.541380+0000 osd.2 (osd.2) 51 2021-03-15T22:23:06.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[56514]: : cluster [DBG] 7.8 scrub ok 2021-03-15T22:23:06.282 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[56514]: cluster 2021-03-15T22:23 2021-03-15T22:23:06.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[56514]: :06.065729+0000 mon.a (mon.0) 957 : cluster [DBG] 2021-03-15T22:23:06.283 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[56514]: osdmap e271: 8 total, 8 up, 8 in 2021-03-15T22:23:06.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: cluster 2021-03-15T22:23:05.018988 2021-03-15T22:23:06.283 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 504 : cluster [DBG] pgmap v531: 138 pgs: 1 active+undersized+remapped, 1 active+clean+wait, 1 active+remapped, 135 active+clean; 219 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 36 MiB/s rd, 3.8 MiB/s wr, 38 op/s; 15 MiB/s, 35 objects/s recovering 2021-03-15T22:23:06.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: audit 2021-03-15T22:23:05.353085+0000 mgr.y (mgr.14144) 505 : audit [DBG] from='client.15135 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:06.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: cluster 2021-03-15T22:23:06.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: 2021-03-15T22:23:05.519540+0000 osd.6 (osd.6) 42 : cluster 2021-03-15T22:23:06.284 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: [DBG] 3.10 scrub ok 2021-03-15T22:23:06.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:23:06.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: :23:05.541380+0000 osd.2 (osd. 2021-03-15T22:23:06.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: 2) 51 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:23:06.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:23:06.285 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: 15T22:23:06.065729+0000 mon.a (mon. 2021-03-15T22:23:06.286 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:06 smithi076 conmon[62156]: 0) 957 : cluster [DBG] osdmap e271: 8 total, 8 up, 8 in 2021-03-15T22:23:06.320 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:06.344 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:06.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:06.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:06.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:06.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:06.471 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:06.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:06.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: cluster 2021-03-15T22:23:05.018988+0000 mgr.y (mgr.14144) 504 : 2021-03-15T22:23:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: cluster [DBG] pgmap v531: 138 pgs: 1 active+undersized+remapped, 1 active+clean+wait, 1 active+remapped, 135 active+clean; 219 MiB data, 685 MiB used, 715 GiB / 715 GiB avail; 36 MiB/s rd, 3.8 MiB/s wr, 38 op/s; 15 MiB/s, 35 objects/s recovering 2021-03-15T22:23:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: audit 2021-03-15T22:23:05.353085+0000 mgr.y (mgr.14144) 505 : audit [DBG] from='client.15135 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: cluster 2021-03-15T22:23:05.519540+0000 osd.6 ( 2021-03-15T22:23:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: osd.6) 42 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:23:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:23:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: -15T22:23:05.541380+0000 osd.2 ( 2021-03-15T22:23:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: osd.2) 51 : cluster [DBG] 2021-03-15T22:23:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: 7.8 scrub ok 2021-03-15T22:23:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:23:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: -15T22:23:06.065729+0000 mon.a 2021-03-15T22:23:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: (mon.0) 957 : cluster 2021-03-15T22:23:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:06 smithi125 conmon[56739]: [DBG] osdmap e271: 8 total, 8 up, 8 in 2021-03-15T22:23:06.573 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:06.596 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:06.598 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:06.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:06.724 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:06.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:06.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:06.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:06.843 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2021-03-15T22:23:06.843 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight-by-utilization 1000 1.0 2021-03-15T22:23:06.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:06.851 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:06.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:06.903 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:06.953 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:06.975 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:06.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:06.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:07.074 INFO:tasks.rados.rados.0.smithi076.stdout:902: done (2 left) 2021-03-15T22:23:07.074 INFO:tasks.rados.rados.0.smithi076.stdout:903: snap_create 2021-03-15T22:23:07.075 INFO:tasks.rados.rados.0.smithi076.stdout:900: expect (ObjNum 263 snap 68 seq_num 263) 2021-03-15T22:23:07.078 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:07.102 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:07.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:07.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:07.205 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:07.213 INFO:tasks.rados.rados.0.smithi076.stdout:899: finishing write tid 1 to smithi07694878-8 2021-03-15T22:23:07.214 INFO:tasks.rados.rados.0.smithi076.stdout:899: finishing write tid 2 to smithi07694878-8 2021-03-15T22:23:07.214 INFO:tasks.rados.rados.0.smithi076.stdout:899: finishing write tid 3 to smithi07694878-8 2021-03-15T22:23:07.214 INFO:tasks.rados.rados.0.smithi076.stdout:899: finishing write tid 4 to smithi07694878-8 2021-03-15T22:23:07.214 INFO:tasks.rados.rados.0.smithi076.stdout:899: finishing write tid 5 to smithi07694878-8 2021-03-15T22:23:07.215 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 184 (ObjNum 334 snap 88 seq_num 334) dirty exists 2021-03-15T22:23:07.215 INFO:tasks.rados.rados.0.smithi076.stdout:899: left oid 8 (ObjNum 334 snap 88 seq_num 334) 2021-03-15T22:23:07.230 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:07.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:07.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:07.333 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:07.356 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:07.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:07.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:07.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:07.486 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:07.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:07.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:07.552 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[56514]: cluster 2021-03-15T22:23:05.791840+0000 2021-03-15T22:23:07.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[56514]: osd.4 (osd.4) 63 : cluster [DBG] 4.c scrub ok 2021-03-15T22:23:07.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[56514]: cluster 2021-03-15T22:23:06.147447+0000 osd.5 (osd.5) 53 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:23:07.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[56514]: cluster 2021-03-15T22:23:06.500713+0000 osd.2 (osd.2) 52 : cluster [DBG] 7.e scrub ok 2021-03-15T22:23:07.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[56514]: cluster 2021-03-15T22:23:06.573174+0000 osd.3 (osd.3) 51 : cluster [DBG] 3.b deep-scrub ok 2021-03-15T22:23:07.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[56514]: cluster 2021-03-15T22:23:07.068606+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2021-03-15T22:23:07.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[62156]: cluster 2021-03-15T22:23:05.791840+0000 2021-03-15T22:23:07.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[62156]: osd.4 (osd.4) 63 : cluster [DBG] 4.c scrub ok 2021-03-15T22:23:07.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[62156]: cluster 2021-03-15T22:23:06.147447+0000 osd.5 (osd.5 2021-03-15T22:23:07.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[62156]: ) 53 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:23:07.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:23:07.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[62156]: 22:23:06.500713+0000 osd.2 (osd.2) 52 : cluster [DBG] 7.e scrub ok 2021-03-15T22:23:07.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:23:07.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[62156]: 22:23:06.573174+0000 osd.3 (osd.3) 51 : cluster 2021-03-15T22:23:07.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[62156]: [DBG] 3.b deep-scrub ok 2021-03-15T22:23:07.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[62156]: cluster 2021-03-15T22:23:07.068606+0000 mon.a 2021-03-15T22:23:07.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:07 smithi076 conmon[62156]: (mon.0) 958 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2021-03-15T22:23:07.588 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:07.610 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:07.614 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:07.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:07.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:07.717 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:07.742 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:07.744 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:07.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:07 smithi125 conmon[56739]: cluster 2021-03-15T22:23:05.791840+0000 osd.4 (osd.4) 63 : cluster [DBG] 2021-03-15T22:23:07.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:07 smithi125 conmon[56739]: 4.c scrub ok 2021-03-15T22:23:07.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:07 smithi125 conmon[56739]: cluster 2021-03-15T22:23:06.147447+0000 osd.5 (osd.5) 53 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:23:07.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:07 smithi125 conmon[56739]: cluster 2021-03-15T22:23:06.500713+0000 osd.2 (osd.2) 52 : cluster [DBG] 7.e scrub ok 2021-03-15T22:23:07.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:07 smithi125 conmon[56739]: cluster 2021-03-15T22:23:06.573174+0000 osd.3 (osd.3) 51 : cluster [DBG] 3.b deep-scrub ok 2021-03-15T22:23:07.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:07 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:23:07.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:07 smithi125 conmon[56739]: :23:07.068606+0000 mon.a (mon.0) 958 : cluster [DBG] osdmap e272: 8 total, 8 up, 8 in 2021-03-15T22:23:07.846 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:07.871 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:07.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:07.873 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:07.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:07.977 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:23:07.977 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:23:07.997 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:07.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:07.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:08.072 INFO:tasks.rados.rados.0.smithi076.stdout:899: done (2 left) 2021-03-15T22:23:08.072 INFO:tasks.rados.rados.0.smithi076.stdout:900: done (1 left) 2021-03-15T22:23:08.072 INFO:tasks.rados.rados.0.smithi076.stdout:903: done (0 left) 2021-03-15T22:23:08.073 INFO:tasks.rados.rados.0.smithi076.stdout:904: read oid 33 snap -1 2021-03-15T22:23:08.073 INFO:tasks.rados.rados.0.smithi076.stdout:904: expect deleted 2021-03-15T22:23:08.073 INFO:tasks.rados.rados.0.smithi076.stdout:905: read oid 14 snap -1 2021-03-15T22:23:08.073 INFO:tasks.rados.rados.0.smithi076.stdout:905: expect (ObjNum 302 snap 78 seq_num 302) 2021-03-15T22:23:08.073 INFO:tasks.rados.rados.0.smithi076.stdout:906: write (excl) oid 21 current snap is 89 2021-03-15T22:23:08.074 INFO:tasks.rados.rados.0.smithi076.stdout:906: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2021-03-15T22:23:08.078 INFO:tasks.rados.rados.0.smithi076.stdout:906: writing smithi07694878-21 from 430345 to 1158292 tid 1 2021-03-15T22:23:08.085 INFO:tasks.rados.rados.0.smithi076.stdout:906: writing smithi07694878-21 from 1570056 to 2297495 tid 2 2021-03-15T22:23:08.091 INFO:tasks.rados.rados.0.smithi076.stdout:906: writing smithi07694878-21 from 2699879 to 3169768 tid 3 2021-03-15T22:23:08.091 INFO:tasks.rados.rados.0.smithi076.stdout:906: finishing write tid 1 to smithi07694878-21 2021-03-15T22:23:08.092 INFO:tasks.rados.rados.0.smithi076.stdout:904: done (2 left) 2021-03-15T22:23:08.092 INFO:tasks.rados.rados.0.smithi076.stdout:907: rollback oid 5 current snap is 89 2021-03-15T22:23:08.092 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 63 2021-03-15T22:23:08.093 INFO:tasks.rados.rados.0.smithi076.stdout:908: read oid 19 snap 72 2021-03-15T22:23:08.093 INFO:tasks.rados.rados.0.smithi076.stdout:908: expect (ObjNum 273 snap 72 seq_num 273) 2021-03-15T22:23:08.093 INFO:tasks.rados.rados.0.smithi076.stdout:909: rollback oid 37 current snap is 89 2021-03-15T22:23:08.093 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 37 to 82 2021-03-15T22:23:08.094 INFO:tasks.rados.rados.0.smithi076.stdout:910: snap_create 2021-03-15T22:23:08.094 INFO:tasks.rados.rados.0.smithi076.stdout:906: finishing write tid 2 to smithi07694878-21 2021-03-15T22:23:08.094 INFO:tasks.rados.rados.0.smithi076.stdout:907: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:23:08.095 INFO:tasks.rados.rados.0.smithi076.stdout:907: finishing rollback tid 2 to smithi07694878-5 2021-03-15T22:23:08.095 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 141 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:23:08.096 INFO:tasks.rados.rados.0.smithi076.stdout:906: finishing write tid 3 to smithi07694878-21 2021-03-15T22:23:08.096 INFO:tasks.rados.rados.0.smithi076.stdout:906: finishing write tid 4 to smithi07694878-21 2021-03-15T22:23:08.096 INFO:tasks.rados.rados.0.smithi076.stdout:906: finishing write tid 5 to smithi07694878-21 2021-03-15T22:23:08.096 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 520 (ObjNum 335 snap 89 seq_num 335) dirty exists 2021-03-15T22:23:08.097 INFO:tasks.rados.rados.0.smithi076.stdout:906: left oid 21 (ObjNum 335 snap 89 seq_num 335) 2021-03-15T22:23:08.099 INFO:tasks.rados.rados.0.smithi076.stdout:908: expect (ObjNum 273 snap 72 seq_num 273) 2021-03-15T22:23:08.099 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:08.124 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:08.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:08.125 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:08.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:08.227 INFO:tasks.rados.rados.0.smithi076.stdout:909: finishing rollback tid 0 to smithi07694878-37 2021-03-15T22:23:08.228 INFO:tasks.rados.rados.0.smithi076.stdout:909: finishing rollback tid 1 to smithi07694878-37 2021-03-15T22:23:08.228 INFO:tasks.rados.rados.0.smithi076.stdout:909: finishing rollback tid 2 to smithi07694878-37 2021-03-15T22:23:08.228 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 524 (ObjNum 281 snap 74 seq_num 281) dirty exists 2021-03-15T22:23:08.252 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:08.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:08.253 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:08.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:08.383 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:08.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:08.385 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:08.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:08 smithi076 conmon[62156]: cluster 2021-03-15T22:23:06.891900+0000 osd.0 (osd.0) 57 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:23:08.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:08 smithi076 conmon[62156]: cluster 2021-03-15T22:23:07.019458+0000 mgr.y (mgr.14144) 506 : cluster [DBG] pgmap v534: 138 pgs: 1 active+clean+wait, 1 active+undersized+remapped, 136 active+clean; 228 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 9.1 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2021-03-15T22:23:08.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:08 smithi076 conmon[62156]: cluster 2021-03-15T22:23:07.259745+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:23:08.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:08 smithi076 conmon[62156]: cluster 2021-03-15T22:23:07.578765+0000 osd.3 (osd.3) 52 : cluster [DBG] 6.b scrub ok 2021-03-15T22:23:08.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:08 smithi076 conmon[62156]: cluster 2021-03-15T22:23:08.069941+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2021-03-15T22:23:08.455 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:08 smithi076 conmon[56514]: cluster 2021-03-15T22:23:06.891900+0000 osd.0 (osd.0) 57 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:23:08.456 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:08 smithi076 conmon[56514]: cluster 2021-03-15T22:23:07.019458+0000 mgr.y (mgr.14144) 506 : cluster [DBG] pgmap v534: 138 pgs: 1 active+clean+wait, 1 active+undersized+remapped, 136 active+clean; 228 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 9.1 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2021-03-15T22:23:08.456 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:08 smithi076 conmon[56514]: cluster 2021-03-15T22:23:07.259745+0000 osd.1 (osd.1) 60 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:23:08.456 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:08 smithi076 conmon[56514]: cluster 2021-03-15T22:23:07.578765+0000 osd.3 (osd.3) 52 : cluster [DBG] 6.b scrub ok 2021-03-15T22:23:08.456 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:08 smithi076 conmon[56514]: cluster 2021-03-15T22:23:08.069941+0000 mon.a (mon.0) 959 : cluster [DBG] osdmap e273: 8 total, 8 up, 8 in 2021-03-15T22:23:08.486 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:08.511 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:08.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:08.512 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:08.613 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:08.638 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:08.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:08.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:08.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:08.766 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:08.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:08.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:08.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: cluster 2021-03-15T22:23:06.891900+0000 osd.0 (osd.0) 57 : 2021-03-15T22:23:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: cluster [DBG] 7.4 scrub ok 2021-03-15T22:23:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: cluster 2021-03-15T22:23:07.019458+0000 mgr.y (mgr.14144) 506 : cluster [DBG] pgmap v534: 138 pgs: 1 active+clean+wait, 1 active+undersized+remapped, 136 active+clean; 228 MiB data, 695 MiB used, 715 GiB / 715 GiB avail; 9.1 MiB/s rd, 2.5 MiB/s wr, 20 op/s 2021-03-15T22:23:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: cluster 2021-03-15T22:23:07.259745+0000 2021-03-15T22:23:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: osd.1 (osd.1) 60 : cluster [DBG] 2021-03-15T22:23:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: 4.4 scrub ok 2021-03-15T22:23:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:23:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: :23:07.578765+0000 osd.3 (osd. 2021-03-15T22:23:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: 3) 52 : cluster [DBG] 6.b scrub ok 2021-03-15T22:23:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: cluster 2021-03-15T22:23:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: 2021-03-15T22:23:08. 2021-03-15T22:23:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: 069941+0000 mon.a (mon.0) 959 : cluster 2021-03-15T22:23:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:08 smithi125 conmon[56739]: [DBG] osdmap e273: 8 total, 8 up, 8 in 2021-03-15T22:23:08.869 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:08.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:08.895 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:08.895 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:08.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:09.024 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:09.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:09.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:09.075 INFO:tasks.rados.rados.0.smithi076.stdout:906: done (5 left) 2021-03-15T22:23:09.075 INFO:tasks.rados.rados.0.smithi076.stdout:907: done (4 left) 2021-03-15T22:23:09.075 INFO:tasks.rados.rados.0.smithi076.stdout:908: done (3 left) 2021-03-15T22:23:09.076 INFO:tasks.rados.rados.0.smithi076.stdout:909: done (2 left) 2021-03-15T22:23:09.076 INFO:tasks.rados.rados.0.smithi076.stdout:910: done (1 left) 2021-03-15T22:23:09.076 INFO:tasks.rados.rados.0.smithi076.stdout:911: snap_create 2021-03-15T22:23:09.077 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:23:09.126 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:09.153 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:09.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:09.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:09.256 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:09.282 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:09.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:09.385 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:09.410 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:23:09.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:09.413 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:09.413 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:09.515 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:09.540 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:09.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:09.541 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:09.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: cluster 2021-03-15T22:23:07.762332+0000 osd.4 (osd.4) 64 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:23:09.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: cluster 2021-03-15T22:23:08.105273+0000 osd.5 2021-03-15T22:23:09.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: (osd.5) 54 : cluster [DBG] 3.c scrub ok 2021-03-15T22:23:09.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: audit 2021-03-15T22:23:08.498444+0000 mon.a (mon.0) 960 : audit [INF] from='client.? 172.21.15.76:0/2995316122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:23:09.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: cluster 2021-03-15T22:23 2021-03-15T22:23:09.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: :08.578819+0000 osd.3 (osd.3) 53 : cluster [DBG] 7.2 scrub ok 2021-03-15T22:23:09.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: cluster 2021-03-15T22:23 2021-03-15T22:23:09.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: :09.064778+0000 mon.a (mon.0) 961 2021-03-15T22:23:09.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:09.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:23:09.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: 15T22:23:09.070602+0000 mon.a 2021-03-15T22:23:09.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: (mon.0) 962 : audit [INF] from='client.? 172.21.15.76:0/2995316122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:23:09.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:23:09.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: 03-15T22:23:09.070656+0000 2021-03-15T22:23:09.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: mon.a (mon.0) 963 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2021-03-15T22:23:09.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[56739]: 2021-03-15T22:23:09.645 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:09.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:09.670 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:09.671 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:09.745 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[56514]: cluster 2021-03-15T22:23:07.762332 2021-03-15T22:23:09.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[56514]: +0000 osd.4 (osd.4) 64 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:23:09.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[56514]: cluster 2021-03-15T22:23:08.105273+0000 osd.5 (osd.5) 54 : cluster [DBG] 3.c scrub ok 2021-03-15T22:23:09.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[56514]: audit 2021-03-15T22:23:08.498444+0000 mon.a (mon.0) 960 : audit [INF] from='client.? 172.21.15.76:0/2995316122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:23:09.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[56514]: cluster 2021-03-15T22:23:08.578819+0000 osd.3 (osd.3) 53 : cluster [DBG] 7.2 scrub ok 2021-03-15T22:23:09.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[56514]: cluster 2021-03-15T22:23:09.064778+0000 mon.a (mon.0) 961 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:09.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[56514]: audit 2021-03-15T22:23:09.070602+0000 mon.a (mon.0) 962 : audit [INF] from='client.? 172.21.15.76:0/2995316122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:23:09.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[56514]: cluster 2021-03-15T22:23:09.070656+0000 mon.a (mon.0) 963 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2021-03-15T22:23:09.748 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: cluster 2021-03-15T22:23:07.762332+0000 osd.4 ( 2021-03-15T22:23:09.748 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: osd.4) 64 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:23:09.748 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: cluster 2021-03-15T22:23:08.105273+0000 osd.5 (osd.5) 54 : cluster [DBG] 2021-03-15T22:23:09.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: 3.c scrub ok 2021-03-15T22:23:09.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: audit 2021-03-15T22:23:08.498444+0000 mon.a (mon 2021-03-15T22:23:09.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: .0) 960 : audit [INF] from='client.? 172.21.15.76:0/2995316122' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:23:09.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: cluster 2021-03-15T22:23:08.578819+0000 osd.3 ( 2021-03-15T22:23:09.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: osd.3) 53 : cluster [DBG] 7.2 scrub ok 2021-03-15T22:23:09.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:23:09.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: 22:23:09.064778+0000 mon.a (mon.0) 961 : cluster [WRN] 2021-03-15T22:23:09.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:09.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: audit 2021-03-15T22:23:09.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: 2021-03-15T22:23:09.070602+0000 mon.a (mon.0) 962 : audit 2021-03-15T22:23:09.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/2995316122' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:23:09.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: cluster 2021-03-15T22:23:09.070656+0000 mon.a ( 2021-03-15T22:23:09.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:09 smithi076 conmon[62156]: mon.0) 963 : cluster [DBG] osdmap e274: 8 total, 8 up, 8 in 2021-03-15T22:23:09.772 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:09.799 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:09.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:09.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:09.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:09.928 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:09.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:09.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:10.030 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:10.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:23:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:23:10.051 INFO:tasks.rados.rados.0.smithi076.stdout:911: done (1 left) 2021-03-15T22:23:10.051 INFO:tasks.rados.rados.0.smithi076.stdout:912: write oid 16 current snap is 91 2021-03-15T22:23:10.051 INFO:tasks.rados.rados.0.smithi076.stdout:912: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2021-03-15T22:23:10.055 INFO:tasks.rados.rados.0.smithi076.stdout:912: writing smithi07694878-16 from 447152 to 850348 tid 1 2021-03-15T22:23:10.060 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:10.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:10.062 INFO:tasks.rados.rados.0.smithi076.stdout:912: writing smithi07694878-16 from 1512185 to 2099635 tid 2 2021-03-15T22:23:10.070 INFO:tasks.rados.rados.0.smithi076.stdout:912: writing smithi07694878-16 from 2727302 to 3408463 tid 3 2021-03-15T22:23:10.070 INFO:tasks.rados.rados.0.smithi076.stdout:913: snap_remove snap 90 2021-03-15T22:23:10.163 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:10.185 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:10.186 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:10.287 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:10.311 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:10.312 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:10.413 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:10.436 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:10.436 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:10.538 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:10.561 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:10.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:10.563 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:10.577 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:10 smithi076 conmon[56514]: cluster 2021-03-15T22:23:08.898274+0000 osd.0 (osd.0) 58 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:23:10.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:10 smithi076 conmon[56514]: cluster 2021-03-15T22:23:09.019855+0000 mgr.y (mgr.14144) 507 : cluster [DBG] pgmap v537: 138 pgs: 1 active+clean+wait, 137 active+clean; 224 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.7 MiB/s wr, 18 op/s 2021-03-15T22:23:10.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:10 smithi076 conmon[56514]: audit 2021-03-15T22:23:09.583208+0000 mgr.y (mgr.14144) 508 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:10.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:10 smithi076 conmon[56514]: cluster 2021-03-15T22:23:10.047122+0000 mon.a (mon.0) 964 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2021-03-15T22:23:10.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:10 smithi076 conmon[62156]: cluster 2021-03-15T22:23:08.898274+0000 osd.0 (osd.0) 58 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:23:10.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:10 smithi076 conmon[62156]: cluster 2021-03-15T22:23:09.019855+0000 mgr.y (mgr.14144) 507 : cluster [DBG] pgmap v537: 138 pgs: 1 active+clean+wait, 137 active+clean; 224 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.7 MiB/s wr, 18 op/s 2021-03-15T22:23:10.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:10 smithi076 conmon[62156]: audit 2021-03-15T22:23:09.583208+0000 mgr.y (mgr.14144) 508 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:10.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:10 smithi076 conmon[62156]: cluster 2021-03-15T22:23:10.047122+0000 mon.a (mon.0) 964 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2021-03-15T22:23:10.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:10.685 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:10.690 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:10.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:10.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:10.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:10 smithi125 conmon[56739]: cluster 2021-03-15T22:23:08.898274+0000 osd.0 (osd.0) 58 : 2021-03-15T22:23:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:10 smithi125 conmon[56739]: cluster [DBG] 5.5 scrub ok 2021-03-15T22:23:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:10 smithi125 conmon[56739]: cluster 2021-03-15T22:23:09.019855+0000 mgr.y (mgr.14144) 507 : cluster [DBG] pgmap v537: 138 pgs: 1 active+clean+wait, 137 active+clean; 224 MiB data, 677 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.7 MiB/s wr, 18 op/s 2021-03-15T22:23:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:10 smithi125 conmon[56739]: audit 2021-03-15T22:23:09.583208+0000 mgr.y (mgr.14144 2021-03-15T22:23:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:10 smithi125 conmon[56739]: ) 508 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:10 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:23:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:10 smithi125 conmon[56739]: 03-15T22:23:10.047122+0000 mon.a ( 2021-03-15T22:23:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:10 smithi125 conmon[56739]: mon.0) 964 : cluster [DBG] osdmap e275: 8 total, 8 up, 8 in 2021-03-15T22:23:10.794 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:10.820 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:10.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:10.822 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:10.923 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:10.949 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:10.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:10.974 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:23:10.991 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:23:11.009 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:23:11.031 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:23:11.051 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:23:11.052 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:11.062 INFO:tasks.rados.rados.0.smithi076.stdout:913: done (2 left) 2021-03-15T22:23:11.062 INFO:tasks.rados.rados.0.smithi076.stdout:914: write (excl) oid 12 current snap is 91 2021-03-15T22:23:11.062 INFO:tasks.rados.rados.0.smithi076.stdout:912: finishing write tid 1 to smithi07694878-16 2021-03-15T22:23:11.063 INFO:tasks.rados.rados.0.smithi076.stdout:914: seq_num 337 ranges {463959=478445,1570667=731108,3038372=492433} 2021-03-15T22:23:11.066 INFO:tasks.rados.rados.0.smithi076.stdout:914: writing smithi07694878-12 from 463959 to 942404 tid 1 2021-03-15T22:23:11.072 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:23:11.074 INFO:tasks.rados.rados.0.smithi076.stdout:914: writing smithi07694878-12 from 1570667 to 2301775 tid 2 2021-03-15T22:23:11.078 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:11.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:11.079 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:11.081 INFO:tasks.rados.rados.0.smithi076.stdout:914: writing smithi07694878-12 from 3038372 to 3530805 tid 3 2021-03-15T22:23:11.081 INFO:tasks.rados.rados.0.smithi076.stdout:915: write (excl) oid 1 current snap is 91 2021-03-15T22:23:11.082 INFO:tasks.rados.rados.0.smithi076.stdout:915: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2021-03-15T22:23:11.087 INFO:tasks.rados.rados.0.smithi076.stdout:915: writing smithi07694878-1 from 480766 to 1034460 tid 1 2021-03-15T22:23:11.094 INFO:tasks.rados.rados.0.smithi076.stdout:915: writing smithi07694878-1 from 1629149 to 2220268 tid 2 2021-03-15T22:23:11.099 INFO:tasks.rados.rados.0.smithi076.stdout:915: writing smithi07694878-1 from 2782148 to 3202206 tid 3 2021-03-15T22:23:11.099 INFO:tasks.rados.rados.0.smithi076.stdout:916: snap_remove snap 82 2021-03-15T22:23:11.102 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:23:11.113 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:23:11.114 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:23:11.114 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:23:11.114 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:23:11.115 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:23:11.115 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:23:11.115 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:23:11.115 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:23:11.116 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:23:11.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:11.204 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:11.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:11.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:11.306 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:11.330 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:11.331 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:11.384 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 414 (0%) 2021-03-15T22:23:11.384 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:23:11.385 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 11.1327 -> 11.1327 (expected baseline 6.72913) 2021-03-15T22:23:11.385 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 25 -> 25 pgs (0.483092 -> 0.483092 * mean) 2021-03-15T22:23:11.385 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 64 -> 64 pgs (1.23671 -> 1.23671 * mean) 2021-03-15T22:23:11.385 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:23:11.386 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:23:11.386 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 1 2021-03-15T22:23:11.386 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:23:11.386 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0009 2021-03-15T22:23:11.386 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0092 2021-03-15T22:23:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:23:11.433 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:11.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[56514]: cluster 2021-03-15T22:23:10.377575+0000 2021-03-15T22:23:11.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[56514]: osd.1 (osd.1) 61 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:11.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[56514]: cluster 2021-03-15T22:23:10.377857+0000 osd.1 (osd.1) 62 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:11.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[56514]: audit 2021-03-15T22:23:10.954302+0000 mon.a (mon.0) 965 : audit [DBG] from='client.? 172.21.15.76:0/3981401317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:11.446 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[56514]: audit 2021-03-15T22:23:11.050300+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:23:11.446 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[56514]: cluster 2021-03-15T22:23:11.057939+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2021-03-15T22:23:11.447 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[62156]: cluster 2021-03-15T22:23:10.377575+0000 osd.1 (osd. 2021-03-15T22:23:11.447 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[62156]: 1) 61 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:11.447 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[62156]: cluster 2021-03-15T22:23:10.377857+0000 osd.1 (osd.1) 62 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:11.447 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[62156]: audit 2021-03-15T22:23:10.954302+0000 mon.a (mon.0) 965 : audit [DBG] from='client.? 172.21.15.76:0/3981401317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:11.448 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[62156]: audit 2021-03-15T22:23:11.050300+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:23:11.448 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:11 smithi076 conmon[62156]: cluster 2021-03-15T22:23:11.057939+0000 mon.a (mon.0) 967 : cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2021-03-15T22:23:11.458 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:11.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:11.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:11.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:11.585 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:11.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:11.586 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:11.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:11.710 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:11.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:11.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:11.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:11 smithi125 conmon[56739]: cluster 2021-03-15T22:23:10.377575+0000 osd.1 (osd.1) 61 2021-03-15T22:23:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:11 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:11 smithi125 conmon[56739]: cluster 2021-03-15T22:23:10.377857+0000 osd.1 (osd.1) 62 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:11 smithi125 conmon[56739]: audit 2021-03-15T22:23:10.954302+0000 mon.a (mon.0) 965 : audit [DBG] from='client.? 172.21.15.76:0/3981401317' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:11 smithi125 conmon[56739]: audit 2021-03-15T22:23:11.050300+0000 mon.a (mon.0) 966 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:23:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:11 smithi125 conmon[56739]: cluster 2021-03-15T22:23:11.057939+0000 mon.a (mon.0) 967 : 2021-03-15T22:23:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:11 smithi125 conmon[56739]: cluster [DBG] osdmap e276: 8 total, 8 up, 8 in 2021-03-15T22:23:11.813 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:11.834 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:11.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:11.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:11.937 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:11.959 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:11.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:11.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:12.062 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:12.065 INFO:tasks.rados.rados.0.smithi076.stdout:916: done (4 left) 2021-03-15T22:23:12.065 INFO:tasks.rados.rados.0.smithi076.stdout:917: read oid 45 snap -1 2021-03-15T22:23:12.066 INFO:tasks.rados.rados.0.smithi076.stdout:917: expect (ObjNum 332 snap 88 seq_num 332) 2021-03-15T22:23:12.066 INFO:tasks.rados.rados.0.smithi076.stdout:918: snap_remove snap 63 2021-03-15T22:23:12.090 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:12.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:12.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:12.193 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:12.215 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:12.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:12.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:12.317 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:12.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:12.340 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:12.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:12.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:12.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:12.470 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:12.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:12.499 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2021-03-15T22:23:12.500 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd test-reweight-by-utilization 110 1.0 2021-03-15T22:23:12.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:12 smithi076 conmon[56514]: cluster 2021-03-15T22:23:11.020260+0000 mgr.y (mgr.14144) 509 : cluster [DBG] pgmap v540: 138 pgs: 1 active+clean+wait, 137 active+clean; 221 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 858 KiB/s wr, 6 op/s 2021-03-15T22:23:12.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:12 smithi076 conmon[56514]: audit 2021-03-15T22:23:11.382236+0000 mgr.y (mgr.14144) 510 : audit [DBG] from='client.15153 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:12.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:12 smithi076 conmon[56514]: cluster 2021-03-15T22:23:12.060154+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2021-03-15T22:23:12.563 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:12 smithi076 conmon[62156]: cluster 2021-03-15T22:23:11.020260+0000 mgr.y (mgr.14144) 509 : cluster [DBG] pgmap v540: 138 pgs: 1 active+clean+wait, 137 active+clean; 221 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 858 KiB/s wr, 6 op/s 2021-03-15T22:23:12.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:12 smithi076 conmon[62156]: audit 2021-03-15T22:23:11.382236+0000 mgr.y (mgr.14144) 510 : audit [DBG] from='client.15153 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:12.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:12 smithi076 conmon[62156]: cluster 2021-03-15T22:23:12.060154+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2021-03-15T22:23:12.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:12.594 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:12.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:12.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:12.687 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:23:12.696 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:12.723 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:12.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:12.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:12.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:12 smithi125 conmon[56739]: cluster 2021-03-15T22:23:11.020260+0000 mgr.y (mgr.14144) 509 : cluster [DBG] pgmap v540: 138 pgs: 1 active+clean+wait, 137 active+clean; 221 MiB data, 674 MiB used, 715 GiB / 715 GiB avail; 5.7 MiB/s rd, 858 KiB/s wr, 6 op/s 2021-03-15T22:23:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:12 smithi125 conmon[56739]: audit 2021-03-15T22:23:11.382236+0000 mgr.y (mgr.14144) 510 : audit [DBG] from='client.15153 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:12 smithi125 conmon[56739]: cluster 2021-03-15T22:23:12 2021-03-15T22:23:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:12 smithi125 conmon[56739]: .060154+0000 mon.a (mon.0) 968 : cluster [DBG] osdmap e277: 8 total, 8 up, 8 in 2021-03-15T22:23:12.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:12.847 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:12.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:12.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:12.972 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:12.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:12.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:13.065 INFO:tasks.rados.rados.0.smithi076.stdout:918: done (5 left) 2021-03-15T22:23:13.065 INFO:tasks.rados.rados.0.smithi076.stdout:919: write oid 6 current snap is 91 2021-03-15T22:23:13.065 INFO:tasks.rados.rados.0.smithi076.stdout:919: seq_num 339 ranges {497573=628943,1687631=734777,3093218=604355} 2021-03-15T22:23:13.071 INFO:tasks.rados.rados.0.smithi076.stdout:919: writing smithi07694878-6 from 497573 to 1126516 tid 1 2021-03-15T22:23:13.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:13.079 INFO:tasks.rados.rados.0.smithi076.stdout:919: writing smithi07694878-6 from 1687631 to 2422408 tid 2 2021-03-15T22:23:13.087 INFO:tasks.rados.rados.0.smithi076.stdout:919: writing smithi07694878-6 from 3093218 to 3697573 tid 3 2021-03-15T22:23:13.087 INFO:tasks.rados.rados.0.smithi076.stdout:920: snap_remove snap 89 2021-03-15T22:23:13.100 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:13.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:13.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:13.137 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:13.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:13.229 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:13.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:13.231 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:13.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:13.343 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:23:13 smithi076 conmon[72291]: debug 2021-03-15T22:23:13.066+0000 7fb936181700 -1 osd.2 pg_epoch: 278 pg[7.8( v 278'184 (0'0,278'184] local-lis/les=91/92 n=6 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [2,7,4] r=0 lpr=91 luod=277'182 crt=278'184 lcod 272'180 mlcod 272'180 active+clean+snaptrim trimq=[7d~1] ps=[9e~1,a0~1,a2~1,a6~1,af~1]] removing snap head 2021-03-15T22:23:13.359 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:13.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:13.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:13.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:13.487 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:13.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:13.488 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:13.590 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:13.616 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:13.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:13.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:13.718 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:13.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:13.748 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:13.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:13.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:13.875 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:13.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:13.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:13.884 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:23:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:23:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:23:13.885 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:23:13.885 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:23:13.978 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:14.000 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:14.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:14.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:14.072 INFO:tasks.rados.rados.0.smithi076.stdout:920: done (6 left) 2021-03-15T22:23:14.072 INFO:tasks.rados.rados.0.smithi076.stdout:921: rollback oid 48 current snap is 91 2021-03-15T22:23:14.073 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 48 to 88 2021-03-15T22:23:14.073 INFO:tasks.rados.rados.0.smithi076.stdout:922: read oid 15 snap -1 2021-03-15T22:23:14.073 INFO:tasks.rados.rados.0.smithi076.stdout:922: expect deleted 2021-03-15T22:23:14.073 INFO:tasks.rados.rados.0.smithi076.stdout:923: write oid 33 current snap is 91 2021-03-15T22:23:14.073 INFO:tasks.rados.rados.0.smithi076.stdout:923: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2021-03-15T22:23:14.079 INFO:tasks.rados.rados.0.smithi076.stdout:923: writing smithi07694878-33 from 514380 to 1218572 tid 1 2021-03-15T22:23:14.084 INFO:tasks.rados.rados.0.smithi076.stdout:923: writing smithi07694878-33 from 1629760 to 2108195 tid 2 2021-03-15T22:23:14.089 INFO:tasks.rados.rados.0.smithi076.stdout:923: writing smithi07694878-33 from 2604288 to 3046890 tid 3 2021-03-15T22:23:14.089 INFO:tasks.rados.rados.0.smithi076.stdout:924: snap_remove snap 87 2021-03-15T22:23:14.103 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:14.124 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:14.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:14.126 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:14.143 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:14 smithi076 conmon[56514]: cluster 2021-03-15T22:23:13.020772+0000 mgr.y (mgr.14144) 511 : cluster [DBG] pgmap v543: 138 pgs: 1 active+clean+wait, 137 active+clean; 232 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 879 KiB/s wr, 6 op/s 2021-03-15T22:23:14.143 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:14 smithi076 conmon[56514]: cluster 2021-03-15T22:23:13.060055+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2021-03-15T22:23:14.143 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:14 smithi076 conmon[62156]: cluster 2021-03-15T22:23:13.020772+0000 mgr.y 2021-03-15T22:23:14.144 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:14 smithi076 conmon[62156]: (mgr.14144) 511 : cluster [DBG] pgmap v543: 138 pgs: 1 active+clean+wait, 137 active+clean; 232 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 879 KiB/s wr, 6 op/s 2021-03-15T22:23:14.144 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:14 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:23:14.144 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:14 smithi076 conmon[62156]: T22:23:13.060055+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2021-03-15T22:23:14.227 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:14.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:14.251 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:14.251 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:14.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:14.374 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:14.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:14.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:14.469 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:23:14.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:14.503 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:14.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:14.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:14.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:14 smithi125 conmon[56739]: cluster 2021-03-15T22:23:13.020772+0000 mgr.y (mgr.14144) 511 : cluster [DBG] pgmap v543: 138 pgs: 1 active+clean+wait, 137 active+clean; 232 MiB data, 680 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 879 KiB/s wr, 6 op/s 2021-03-15T22:23:14.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:14 smithi125 conmon[56739]: cluster 2021-03-15T22:23:13.060055+0000 mon.a (mon.0) 969 : cluster [DBG] osdmap e278: 8 total, 8 up, 8 in 2021-03-15T22:23:14.606 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:14.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:14.631 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:14.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:14.732 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:14.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:14.761 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:14.761 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:14.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:14.888 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:14.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:14.991 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:15.016 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:15.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:15.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:15.052 INFO:tasks.rados.rados.0.smithi076.stdout:924: done (9 left) 2021-03-15T22:23:15.052 INFO:tasks.rados.rados.0.smithi076.stdout:925: copy_from oid 43 from oid 17 current snap is 91 2021-03-15T22:23:15.053 INFO:tasks.rados.rados.0.smithi076.stdout:926: write oid 4 current snap is 91 2021-03-15T22:23:15.053 INFO:tasks.rados.rados.0.smithi076.stdout:926: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2021-03-15T22:23:15.062 INFO:tasks.rados.rados.0.smithi076.stdout:926: writing smithi07694878-4 from 531187 to 1310628 tid 1 2021-03-15T22:23:15.070 INFO:tasks.rados.rados.0.smithi076.stdout:926: writing smithi07694878-4 from 2088242 to 2826688 tid 2 2021-03-15T22:23:15.076 INFO:tasks.rados.rados.0.smithi076.stdout:926: writing smithi07694878-4 from 3431711 to 3731187 tid 3 2021-03-15T22:23:15.077 INFO:tasks.rados.rados.0.smithi076.stdout:927: delete oid 8 current snap is 91 2021-03-15T22:23:15.077 INFO:tasks.rados.rados.0.smithi076.stdout:912: finishing write tid 2 to smithi07694878-16 2021-03-15T22:23:15.077 INFO:tasks.rados.rados.0.smithi076.stdout:912: finishing write tid 3 to smithi07694878-16 2021-03-15T22:23:15.077 INFO:tasks.rados.rados.0.smithi076.stdout:912: finishing write tid 4 to smithi07694878-16 2021-03-15T22:23:15.078 INFO:tasks.rados.rados.0.smithi076.stdout:912: finishing write tid 5 to smithi07694878-16 2021-03-15T22:23:15.078 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 444 (ObjNum 336 snap 91 seq_num 336) dirty exists 2021-03-15T22:23:15.078 INFO:tasks.rados.rados.0.smithi076.stdout:912: left oid 16 (ObjNum 336 snap 91 seq_num 336) 2021-03-15T22:23:15.079 INFO:tasks.rados.rados.0.smithi076.stdout:914: finishing write tid 1 to smithi07694878-12 2021-03-15T22:23:15.079 INFO:tasks.rados.rados.0.smithi076.stdout:914: finishing write tid 2 to smithi07694878-12 2021-03-15T22:23:15.079 INFO:tasks.rados.rados.0.smithi076.stdout:914: finishing write tid 3 to smithi07694878-12 2021-03-15T22:23:15.080 INFO:tasks.rados.rados.0.smithi076.stdout:914: finishing write tid 4 to smithi07694878-12 2021-03-15T22:23:15.080 INFO:tasks.rados.rados.0.smithi076.stdout:914: finishing write tid 5 to smithi07694878-12 2021-03-15T22:23:15.080 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 451 (ObjNum 337 snap 91 seq_num 337) dirty exists 2021-03-15T22:23:15.080 INFO:tasks.rados.rados.0.smithi076.stdout:914: left oid 12 (ObjNum 337 snap 91 seq_num 337) 2021-03-15T22:23:15.081 INFO:tasks.rados.rados.0.smithi076.stdout:915: finishing write tid 1 to smithi07694878-1 2021-03-15T22:23:15.081 INFO:tasks.rados.rados.0.smithi076.stdout:915: finishing write tid 2 to smithi07694878-1 2021-03-15T22:23:15.081 INFO:tasks.rados.rados.0.smithi076.stdout:915: finishing write tid 3 to smithi07694878-1 2021-03-15T22:23:15.081 INFO:tasks.rados.rados.0.smithi076.stdout:915: finishing write tid 4 to smithi07694878-1 2021-03-15T22:23:15.081 INFO:tasks.rados.rados.0.smithi076.stdout:915: finishing write tid 5 to smithi07694878-1 2021-03-15T22:23:15.082 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 159 (ObjNum 338 snap 91 seq_num 338) dirty exists 2021-03-15T22:23:15.082 INFO:tasks.rados.rados.0.smithi076.stdout:915: left oid 1 (ObjNum 338 snap 91 seq_num 338) 2021-03-15T22:23:15.082 INFO:tasks.rados.rados.0.smithi076.stdout:917: expect (ObjNum 332 snap 88 seq_num 332) 2021-03-15T22:23:15.119 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:15.143 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:15.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:15.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:15.200 INFO:tasks.rados.rados.0.smithi076.stdout:919: finishing write tid 1 to smithi07694878-6 2021-03-15T22:23:15.200 INFO:tasks.rados.rados.0.smithi076.stdout:919: finishing write tid 2 to smithi07694878-6 2021-03-15T22:23:15.200 INFO:tasks.rados.rados.0.smithi076.stdout:919: finishing write tid 3 to smithi07694878-6 2021-03-15T22:23:15.200 INFO:tasks.rados.rados.0.smithi076.stdout:919: finishing write tid 4 to smithi07694878-6 2021-03-15T22:23:15.201 INFO:tasks.rados.rados.0.smithi076.stdout:919: finishing write tid 5 to smithi07694878-6 2021-03-15T22:23:15.201 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 257 (ObjNum 339 snap 91 seq_num 339) dirty exists 2021-03-15T22:23:15.201 INFO:tasks.rados.rados.0.smithi076.stdout:919: left oid 6 (ObjNum 339 snap 91 seq_num 339) 2021-03-15T22:23:15.201 INFO:tasks.rados.rados.0.smithi076.stdout:912: done (11 left) 2021-03-15T22:23:15.202 INFO:tasks.rados.rados.0.smithi076.stdout:914: done (10 left) 2021-03-15T22:23:15.202 INFO:tasks.rados.rados.0.smithi076.stdout:915: done (9 left) 2021-03-15T22:23:15.202 INFO:tasks.rados.rados.0.smithi076.stdout:917: done (8 left) 2021-03-15T22:23:15.202 INFO:tasks.rados.rados.0.smithi076.stdout:919: done (7 left) 2021-03-15T22:23:15.202 INFO:tasks.rados.rados.0.smithi076.stdout:927: done (6 left) 2021-03-15T22:23:15.203 INFO:tasks.rados.rados.0.smithi076.stdout:928: read oid 2 snap 72 2021-03-15T22:23:15.203 INFO:tasks.rados.rados.0.smithi076.stdout:928: expect deleted 2021-03-15T22:23:15.203 INFO:tasks.rados.rados.0.smithi076.stdout:929: read oid 44 snap -1 2021-03-15T22:23:15.204 INFO:tasks.rados.rados.0.smithi076.stdout:929: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:23:15.204 INFO:tasks.rados.rados.0.smithi076.stdout:930: read oid 41 snap -1 2021-03-15T22:23:15.204 INFO:tasks.rados.rados.0.smithi076.stdout:930: expect (ObjNum 290 snap 77 seq_num 290) 2021-03-15T22:23:15.204 INFO:tasks.rados.rados.0.smithi076.stdout:931: snap_create 2021-03-15T22:23:15.205 INFO:tasks.rados.rados.0.smithi076.stdout:905: expect (ObjNum 302 snap 78 seq_num 302) 2021-03-15T22:23:15.245 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:15.271 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:15.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:15.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:15.301 INFO:tasks.rados.rados.0.smithi076.stdout:921: finishing rollback tid 0 to smithi07694878-48 2021-03-15T22:23:15.301 INFO:tasks.rados.rados.0.smithi076.stdout:921: finishing rollback tid 1 to smithi07694878-48 2021-03-15T22:23:15.302 INFO:tasks.rados.rados.0.smithi076.stdout:921: finishing rollback tid 2 to smithi07694878-48 2021-03-15T22:23:15.302 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 200 (ObjNum 253 snap 63 seq_num 253) dirty exists 2021-03-15T22:23:15.302 INFO:tasks.rados.rados.0.smithi076.stdout:923: finishing write tid 1 to smithi07694878-33 2021-03-15T22:23:15.302 INFO:tasks.rados.rados.0.smithi076.stdout:923: finishing write tid 2 to smithi07694878-33 2021-03-15T22:23:15.303 INFO:tasks.rados.rados.0.smithi076.stdout:923: finishing write tid 3 to smithi07694878-33 2021-03-15T22:23:15.303 INFO:tasks.rados.rados.0.smithi076.stdout:923: finishing write tid 4 to smithi07694878-33 2021-03-15T22:23:15.303 INFO:tasks.rados.rados.0.smithi076.stdout:923: finishing write tid 5 to smithi07694878-33 2021-03-15T22:23:15.303 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 211 (ObjNum 340 snap 91 seq_num 340) dirty exists 2021-03-15T22:23:15.303 INFO:tasks.rados.rados.0.smithi076.stdout:923: left oid 33 (ObjNum 340 snap 91 seq_num 340) 2021-03-15T22:23:15.304 INFO:tasks.rados.rados.0.smithi076.stdout:925: finishing copy_from to smithi07694878-43 2021-03-15T22:23:15.304 INFO:tasks.rados.rados.0.smithi076.stdout:925: got expected ENOENT (src dne) 2021-03-15T22:23:15.304 INFO:tasks.rados.rados.0.smithi076.stdout:925: finishing copy_from racing read to smithi07694878-43 2021-03-15T22:23:15.304 INFO:tasks.rados.rados.0.smithi076.stdout:926: finishing write tid 1 to smithi07694878-4 2021-03-15T22:23:15.305 INFO:tasks.rados.rados.0.smithi076.stdout:926: finishing write tid 2 to smithi07694878-4 2021-03-15T22:23:15.305 INFO:tasks.rados.rados.0.smithi076.stdout:926: finishing write tid 3 to smithi07694878-4 2021-03-15T22:23:15.305 INFO:tasks.rados.rados.0.smithi076.stdout:926: finishing write tid 4 to smithi07694878-4 2021-03-15T22:23:15.305 INFO:tasks.rados.rados.0.smithi076.stdout:926: finishing write tid 5 to smithi07694878-4 2021-03-15T22:23:15.305 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 410 (ObjNum 341 snap 91 seq_num 341) dirty exists 2021-03-15T22:23:15.306 INFO:tasks.rados.rados.0.smithi076.stdout:926: left oid 4 (ObjNum 341 snap 91 seq_num 341) 2021-03-15T22:23:15.306 INFO:tasks.rados.rados.0.smithi076.stdout:929: expect (ObjNum 156 snap 35 seq_num 156) 2021-03-15T22:23:15.374 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:15.399 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:15.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:15.400 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:15.443 INFO:tasks.rados.rados.0.smithi076.stdout:930: expect (ObjNum 290 snap 77 seq_num 290) 2021-03-15T22:23:15.501 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:15.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:15 smithi076 conmon[56514]: cluster 2021-03-15T22:23:14.069292 2021-03-15T22:23:15.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:15 smithi076 conmon[56514]: +0000 mon.a (mon.0) 970 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2021-03-15T22:23:15.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:15 smithi076 conmon[56514]: cluster 2021-03-15T22:23:15.049727+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2021-03-15T22:23:15.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:15 smithi076 conmon[62156]: cluster 2021-03-15T22:23:14.069292+0000 2021-03-15T22:23:15.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:15 smithi076 conmon[62156]: mon.a (mon.0) 970 : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2021-03-15T22:23:15.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:15 smithi076 conmon[62156]: cluster 2021-03-15T22:23: 2021-03-15T22:23:15.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:15 smithi076 conmon[62156]: 15.049727+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2021-03-15T22:23:15.524 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:15.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:15.536 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:23:15 smithi125 conmon[67289]: debug 2021-03-15T22:23:15.060+0000 7fd4d9833700 -1 osd.6 pg_epoch: 280 pg[7.d( v 271'47 (0'0,271'47] local-lis/les=91/92 n=2 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [6,1,0] r=0 lpr=91 crt=271'47 lcod 269'45 mlcod 269'45 active+clean+snaptrim trimq=[a8~1] ps=[7d~1,9e~1,ae~2]] removing snap head 2021-03-15T22:23:15.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:15 smithi125 conmon[56739]: cluster 2021-03-15T22:23:14.069292+0000 mon.a (mon.0) 970 2021-03-15T22:23:15.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:15 smithi125 conmon[56739]: : cluster [DBG] osdmap e279: 8 total, 8 up, 8 in 2021-03-15T22:23:15.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:15 smithi125 conmon[56739]: cluster 2021-03-15T22:23:15.049727+0000 mon.a (mon.0) 971 : cluster [DBG] osdmap e280: 8 total, 8 up, 8 in 2021-03-15T22:23:15.627 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:15.653 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:15.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:15.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:15.689 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:23:15.755 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:15.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:15.779 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:15.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:15.865 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:15.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:15.910 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:15.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:15.912 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:16.013 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:16.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:16.038 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:16.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:16.087 INFO:tasks.rados.rados.0.smithi076.stdout:905: done (9 left) 2021-03-15T22:23:16.087 INFO:tasks.rados.rados.0.smithi076.stdout:921: done (8 left) 2021-03-15T22:23:16.087 INFO:tasks.rados.rados.0.smithi076.stdout:922: done (7 left) 2021-03-15T22:23:16.087 INFO:tasks.rados.rados.0.smithi076.stdout:923: done (6 left) 2021-03-15T22:23:16.088 INFO:tasks.rados.rados.0.smithi076.stdout:925: done (5 left) 2021-03-15T22:23:16.088 INFO:tasks.rados.rados.0.smithi076.stdout:926: done (4 left) 2021-03-15T22:23:16.088 INFO:tasks.rados.rados.0.smithi076.stdout:928: done (3 left) 2021-03-15T22:23:16.088 INFO:tasks.rados.rados.0.smithi076.stdout:929: done (2 left) 2021-03-15T22:23:16.089 INFO:tasks.rados.rados.0.smithi076.stdout:930: done (1 left) 2021-03-15T22:23:16.089 INFO:tasks.rados.rados.0.smithi076.stdout:931: done (0 left) 2021-03-15T22:23:16.089 INFO:tasks.rados.rados.0.smithi076.stdout:932: write (excl) oid 16 current snap is 92 2021-03-15T22:23:16.089 INFO:tasks.rados.rados.0.smithi076.stdout:932: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2021-03-15T22:23:16.093 INFO:tasks.rados.rados.0.smithi076.stdout:932: writing smithi07694878-16 from 564801 to 1211093 tid 1 2021-03-15T22:23:16.101 INFO:tasks.rados.rados.0.smithi076.stdout:932: writing smithi07694878-16 from 1921559 to 2663674 tid 2 2021-03-15T22:23:16.108 INFO:tasks.rados.rados.0.smithi076.stdout:932: writing smithi07694878-16 from 3202910 to 3764801 tid 3 2021-03-15T22:23:16.109 INFO:tasks.rados.rados.0.smithi076.stdout:933: write (excl) oid 25 current snap is 92 2021-03-15T22:23:16.109 INFO:tasks.rados.rados.0.smithi076.stdout:932: finishing write tid 1 to smithi07694878-16 2021-03-15T22:23:16.109 INFO:tasks.rados.rados.0.smithi076.stdout:933: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2021-03-15T22:23:16.115 INFO:tasks.rados.rados.0.smithi076.stdout:933: writing smithi07694878-25 from 581608 to 1303149 tid 1 2021-03-15T22:23:16.122 INFO:tasks.rados.rados.0.smithi076.stdout:933: writing smithi07694878-25 from 1863688 to 2349461 tid 2 2021-03-15T22:23:16.130 INFO:tasks.rados.rados.0.smithi076.stdout:933: writing smithi07694878-25 from 3113980 to 3718023 tid 3 2021-03-15T22:23:16.131 INFO:tasks.rados.rados.0.smithi076.stdout:934: delete oid 29 current snap is 92 2021-03-15T22:23:16.131 INFO:tasks.rados.rados.0.smithi076.stdout:932: finishing write tid 2 to smithi07694878-16 2021-03-15T22:23:16.131 INFO:tasks.rados.rados.0.smithi076.stdout:932: finishing write tid 3 to smithi07694878-16 2021-03-15T22:23:16.132 INFO:tasks.rados.rados.0.smithi076.stdout:932: finishing write tid 4 to smithi07694878-16 2021-03-15T22:23:16.132 INFO:tasks.rados.rados.0.smithi076.stdout:932: finishing write tid 5 to smithi07694878-16 2021-03-15T22:23:16.132 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 478 (ObjNum 343 snap 92 seq_num 343) dirty exists 2021-03-15T22:23:16.132 INFO:tasks.rados.rados.0.smithi076.stdout:932: left oid 16 (ObjNum 343 snap 92 seq_num 343) 2021-03-15T22:23:16.133 INFO:tasks.rados.rados.0.smithi076.stdout:933: finishing write tid 1 to smithi07694878-25 2021-03-15T22:23:16.133 INFO:tasks.rados.rados.0.smithi076.stdout:933: finishing write tid 2 to smithi07694878-25 2021-03-15T22:23:16.133 INFO:tasks.rados.rados.0.smithi076.stdout:932: done (2 left) 2021-03-15T22:23:16.133 INFO:tasks.rados.rados.0.smithi076.stdout:934: done (1 left) 2021-03-15T22:23:16.134 INFO:tasks.rados.rados.0.smithi076.stdout:935: snap_create 2021-03-15T22:23:16.135 INFO:tasks.rados.rados.0.smithi076.stdout:933: finishing write tid 3 to smithi07694878-25 2021-03-15T22:23:16.135 INFO:tasks.rados.rados.0.smithi076.stdout:933: finishing write tid 4 to smithi07694878-25 2021-03-15T22:23:16.135 INFO:tasks.rados.rados.0.smithi076.stdout:933: finishing write tid 5 to smithi07694878-25 2021-03-15T22:23:16.135 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 142 (ObjNum 344 snap 92 seq_num 344) dirty exists 2021-03-15T22:23:16.136 INFO:tasks.rados.rados.0.smithi076.stdout:933: left oid 25 (ObjNum 344 snap 92 seq_num 344) 2021-03-15T22:23:16.140 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:16.162 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:16.164 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:16.177 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:16 smithi076 conmon[56514]: cluster 2021-03-15T22:23:15.021228+0000 mgr.y (mgr.14144) 512 : cluster [DBG] pgmap v546: 138 pgs: 1 active+clean+wait, 137 active+clean; 229 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-03-15T22:23:16.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:16 smithi076 conmon[56514]: cluster 2021-03-15T22:23:15.530860+0000 osd.5 (osd.5) 55 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:16.178 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:16 smithi076 conmon[56514]: cluster 2021-03-15T22:23:15.531230+0000 osd.5 (osd.5) 56 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:16.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:16 smithi076 conmon[62156]: cluster 2021-03-15T22:23:15.021228+0000 mgr.y (mgr.14144) 512 : cluster [DBG] pgmap v546: 138 pgs: 1 active+clean+wait, 137 active+clean; 229 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-03-15T22:23:16.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:16 smithi076 conmon[62156]: cluster 2021-03-15T22:23:15.530860+0000 osd.5 (osd.5) 55 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:16.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:16 smithi076 conmon[62156]: cluster 2021-03-15T22:23:15.531230+0000 osd.5 (osd.5) 56 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:16.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:16.292 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:16.292 INFO:teuthology.orchestra.run.smithi076.stdout:moved 14 / 414 (3.38164%) 2021-03-15T22:23:16.292 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:23:16.293 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 11.1327 -> 14.5838 (expected baseline 6.72913) 2021-03-15T22:23:16.293 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 25 -> 17 pgs (0.483092 -> 0.328502 * mean) 2021-03-15T22:23:16.293 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 64 -> 68 pgs (1.23671 -> 1.31401 * mean) 2021-03-15T22:23:16.293 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:23:16.293 INFO:teuthology.orchestra.run.smithi076.stdout:oload 110 2021-03-15T22:23:16.294 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 1 2021-03-15T22:23:16.294 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:23:16.294 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0009 2021-03-15T22:23:16.294 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0010 2021-03-15T22:23:16.295 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 weight 0.5226 -> 0.3672 2021-03-15T22:23:16.295 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 weight 1.0000 -> 0.8285 2021-03-15T22:23:16.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:16.296 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:23:16.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:16.398 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:16.425 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:16.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:16.426 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:16.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:16 smithi076 conmon[56844]: debug 2021-03-15T22:23:16.288+0000 7fc67af46700 -1 mgr.server reply reply (2) No such file or directory no change 2021-03-15T22:23:16.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:16.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:16 smithi125 conmon[56739]: cluster 2021-03-15T22:23:15.021228+0000 mgr.y (mgr.14144) 512 : cluster [DBG] pgmap v546: 138 pgs: 1 active+clean+wait, 137 active+clean; 229 MiB data, 669 MiB used, 715 GiB / 715 GiB avail; 1.7 MiB/s rd, 1.2 MiB/s wr, 8 op/s 2021-03-15T22:23:16.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:16 smithi125 conmon[56739]: cluster 2021-03-15T22:23:15.530860+0000 osd.5 (osd.5) 55 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:16.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:16 smithi125 conmon[56739]: cluster 2021-03-15T22:23:15.531230+0000 osd.5 (osd.5) 56 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:16.550 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:16.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:16.551 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:16.652 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:16.678 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:16.680 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:16.782 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:16.808 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:16.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:16.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:16.911 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:16.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:16.935 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:16.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:17.037 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:17.060 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:17.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:17.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:17.102 INFO:tasks.rados.rados.0.smithi076.stdout:933: done (1 left) 2021-03-15T22:23:17.102 INFO:tasks.rados.rados.0.smithi076.stdout:935: done (0 left) 2021-03-15T22:23:17.102 INFO:tasks.rados.rados.0.smithi076.stdout:936: read oid 2 snap -1 2021-03-15T22:23:17.103 INFO:tasks.rados.rados.0.smithi076.stdout:936: expect (ObjNum 295 snap 78 seq_num 295) 2021-03-15T22:23:17.103 INFO:tasks.rados.rados.0.smithi076.stdout:937: delete oid 20 current snap is 93 2021-03-15T22:23:17.104 INFO:tasks.rados.rados.0.smithi076.stdout:937: done (1 left) 2021-03-15T22:23:17.105 INFO:tasks.rados.rados.0.smithi076.stdout:938: copy_from oid 38 from oid 19 current snap is 93 2021-03-15T22:23:17.105 INFO:tasks.rados.rados.0.smithi076.stdout:939: read oid 20 snap -1 2021-03-15T22:23:17.105 INFO:tasks.rados.rados.0.smithi076.stdout:939: expect deleted 2021-03-15T22:23:17.105 INFO:tasks.rados.rados.0.smithi076.stdout:940: read oid 6 snap 92 2021-03-15T22:23:17.106 INFO:tasks.rados.rados.0.smithi076.stdout:940: expect (ObjNum 339 snap 91 seq_num 339) 2021-03-15T22:23:17.106 INFO:tasks.rados.rados.0.smithi076.stdout:941: delete oid 4 current snap is 93 2021-03-15T22:23:17.106 INFO:tasks.rados.rados.0.smithi076.stdout:936: expect (ObjNum 295 snap 78 seq_num 295) 2021-03-15T22:23:17.123 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:17.163 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:17.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:17.188 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:17.188 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:17.208 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2021-03-15T22:23:17.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd test-reweight-by-utilization 1000 3.0 2021-03-15T22:23:17.209 INFO:tasks.rados.rados.0.smithi076.stdout:940: expect (ObjNum 339 snap 91 seq_num 339) 2021-03-15T22:23:17.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:17.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:17.315 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:17.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:17.330 INFO:tasks.rados.rados.0.smithi076.stdout:938: finishing copy_from to smithi07694878-38 2021-03-15T22:23:17.330 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 307 (ObjNum 273 snap 72 seq_num 273) dirty exists 2021-03-15T22:23:17.330 INFO:tasks.rados.rados.0.smithi076.stdout:938: finishing copy_from racing read to smithi07694878-38 2021-03-15T22:23:17.331 INFO:tasks.rados.rados.0.smithi076.stdout:936: done (4 left) 2021-03-15T22:23:17.331 INFO:tasks.rados.rados.0.smithi076.stdout:938: done (3 left) 2021-03-15T22:23:17.331 INFO:tasks.rados.rados.0.smithi076.stdout:939: done (2 left) 2021-03-15T22:23:17.331 INFO:tasks.rados.rados.0.smithi076.stdout:940: done (1 left) 2021-03-15T22:23:17.332 INFO:tasks.rados.rados.0.smithi076.stdout:941: done (0 left) 2021-03-15T22:23:17.332 INFO:tasks.rados.rados.0.smithi076.stdout:942: read oid 19 snap -1 2021-03-15T22:23:17.332 INFO:tasks.rados.rados.0.smithi076.stdout:942: expect (ObjNum 273 snap 72 seq_num 273) 2021-03-15T22:23:17.333 INFO:tasks.rados.rados.0.smithi076.stdout:943: snap_remove snap 92 2021-03-15T22:23:17.416 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:17.443 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:17.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:17.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:17.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[56514]: cluster 2021-03-15T22:23:17.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[56514]: 2021-03-15T22:23:17.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[56514]: 2021-03-15T22:23:16. 2021-03-15T22:23:17.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[56514]: 083110+0000 mon.a (mon.0) 972 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2021-03-15T22:23:17.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[56514]: audit 2021-03-15T22 2021-03-15T22:23:17.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[56514]: :23:16.289515+0000 mgr.y (mgr.14144) 513 : audit 2021-03-15T22:23:17.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[56514]: [DBG] from='client.24742 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:17.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: cluster 2021-03-15T22:23:17.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: 2021-03-15T22:23:17.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: 2021 2021-03-15T22:23:17.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: - 2021-03-15T22:23:17.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: 03 2021-03-15T22:23:17.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: - 2021-03-15T22:23:17.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: 15 2021-03-15T22:23:17.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: T 2021-03-15T22:23:17.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: 22 2021-03-15T22:23:17.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: : 2021-03-15T22:23:17.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: 23 2021-03-15T22:23:17.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: : 2021-03-15T22:23:17.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: 16 2021-03-15T22:23:17.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: . 2021-03-15T22:23:17.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: 083110 2021-03-15T22:23:17.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: +0000 mon.a (mon.0) 972 : cluster 2021-03-15T22:23:17.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: [DBG] osdmap e281: 8 total, 8 up, 8 in 2021-03-15T22:23:17.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: audit 2021-03-15T22:23:17.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: 2021-03-15T22:23:16.289515+0000 mgr.y (mgr.14144) 513 2021-03-15T22:23:17.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:17 smithi076 conmon[62156]: : audit [DBG] from='client.24742 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:17.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:17 smithi125 conmon[56739]: cluster 2021-03-15T22:23:16.083110+0000 mon.a (mon.0) 2021-03-15T22:23:17.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:17 smithi125 conmon[56739]: 972 : cluster [DBG] osdmap e281: 8 total, 8 up, 8 in 2021-03-15T22:23:17.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:17 smithi125 conmon[56739]: audit 2021-03-15T22:23:16.289515+0000 mgr.y 2021-03-15T22:23:17.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:17 smithi125 conmon[56739]: (mgr.14144) 513 : audit [DBG] from='client.24742 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:17.545 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:17.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:17.569 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:17.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:17.671 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:17.692 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:17.694 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:17.795 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:17.821 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:17.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:17.823 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:17.924 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:17.947 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:17.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:17.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:18.049 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:18.072 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:18.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:18.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:18.109 INFO:tasks.rados.rados.0.smithi076.stdout:943: done (1 left) 2021-03-15T22:23:18.109 INFO:tasks.rados.rados.0.smithi076.stdout:944: write oid 9 current snap is 93 2021-03-15T22:23:18.109 INFO:tasks.rados.rados.0.smithi076.stdout:944: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2021-03-15T22:23:18.112 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:23:18.116 INFO:tasks.rados.rados.0.smithi076.stdout:944: writing smithi07694878-9 from 648836 to 1271373 tid 1 2021-03-15T22:23:18.125 INFO:tasks.rados.rados.0.smithi076.stdout:944: writing smithi07694878-9 from 1981263 to 2474374 tid 2 2021-03-15T22:23:18.132 INFO:tasks.rados.rados.0.smithi076.stdout:944: writing smithi07694878-9 from 3107319 to 3848836 tid 3 2021-03-15T22:23:18.133 INFO:tasks.rados.rados.0.smithi076.stdout:945: read oid 39 snap -1 2021-03-15T22:23:18.133 INFO:tasks.rados.rados.0.smithi076.stdout:945: expect (ObjNum 199 snap 43 seq_num 199) 2021-03-15T22:23:18.133 INFO:tasks.rados.rados.0.smithi076.stdout:942: expect (ObjNum 273 snap 72 seq_num 273) 2021-03-15T22:23:18.175 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:18.200 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:18.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:18.234 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:18.259 INFO:tasks.rados.rados.0.smithi076.stdout:942: done (2 left) 2021-03-15T22:23:18.260 INFO:tasks.rados.rados.0.smithi076.stdout:946: rollback oid 45 current snap is 93 2021-03-15T22:23:18.260 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 45 to 91 2021-03-15T22:23:18.260 INFO:tasks.rados.rados.0.smithi076.stdout:944: finishing write tid 1 to smithi07694878-9 2021-03-15T22:23:18.260 INFO:tasks.rados.rados.0.smithi076.stdout:944: finishing write tid 2 to smithi07694878-9 2021-03-15T22:23:18.261 INFO:tasks.rados.rados.0.smithi076.stdout:944: finishing write tid 3 to smithi07694878-9 2021-03-15T22:23:18.261 INFO:tasks.rados.rados.0.smithi076.stdout:944: finishing write tid 4 to smithi07694878-9 2021-03-15T22:23:18.265 INFO:tasks.rados.rados.0.smithi076.stdout:944: finishing write tid 5 to smithi07694878-9 2021-03-15T22:23:18.265 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 483 (ObjNum 348 snap 93 seq_num 348) dirty exists 2021-03-15T22:23:18.266 INFO:tasks.rados.rados.0.smithi076.stdout:944: left oid 9 (ObjNum 348 snap 93 seq_num 348) 2021-03-15T22:23:18.266 INFO:tasks.rados.rados.0.smithi076.stdout:945: expect (ObjNum 199 snap 43 seq_num 199) 2021-03-15T22:23:18.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:18.330 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:18.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:18.331 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:18.357 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:18 smithi076 conmon[56514]: cluster 2021-03-15T22:23:17.021675+0000 mgr.y (mgr.14144) 514 : cluster [DBG] pgmap v549: 138 pgs: 138 active+clean; 218 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2021-03-15T22:23:18.358 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:18 smithi076 conmon[56514]: cluster 2021-03-15T22:23:17.098927+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2021-03-15T22:23:18.358 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:18 smithi076 conmon[56514]: audit 2021-03-15T22:23:17.359859+0000 mon.a (mon.0) 974 : audit [INF] from='client.? 172.21.15.76:0/3521625101' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:18.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:18 smithi076 conmon[62156]: cluster 2021-03-15T22:23:17.021675+0000 mgr.y (mgr.14144) 514 : cluster [DBG] pgmap v549: 138 pgs: 138 active+clean; 218 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2021-03-15T22:23:18.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:18 smithi076 conmon[62156]: cluster 2021-03-15T22:23:17.098927+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2021-03-15T22:23:18.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:18 smithi076 conmon[62156]: audit 2021-03-15T22:23:17.359859+0000 mon.a (mon.0) 974 : audit [INF] from='client.? 172.21.15.76:0/3521625101' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:18.423 INFO:tasks.rados.rados.0.smithi076.stdout:944: done (2 left) 2021-03-15T22:23:18.423 INFO:tasks.rados.rados.0.smithi076.stdout:945: done (1 left) 2021-03-15T22:23:18.423 INFO:tasks.rados.rados.0.smithi076.stdout:947: read oid 24 snap -1 2021-03-15T22:23:18.424 INFO:tasks.rados.rados.0.smithi076.stdout:947: expect deleted 2021-03-15T22:23:18.424 INFO:tasks.rados.rados.0.smithi076.stdout:948: write (excl) oid 22 current snap is 93 2021-03-15T22:23:18.424 INFO:tasks.rados.rados.0.smithi076.stdout:946: finishing rollback tid 0 to smithi07694878-45 2021-03-15T22:23:18.424 INFO:tasks.rados.rados.0.smithi076.stdout:946: finishing rollback tid 1 to smithi07694878-45 2021-03-15T22:23:18.424 INFO:tasks.rados.rados.0.smithi076.stdout:946: finishing rollback tid 2 to smithi07694878-45 2021-03-15T22:23:18.425 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 260 (ObjNum 332 snap 88 seq_num 332) dirty exists 2021-03-15T22:23:18.425 INFO:tasks.rados.rados.0.smithi076.stdout:948: seq_num 349 ranges {665643=697786,2039745=753122,3534742=330901} 2021-03-15T22:23:18.430 INFO:tasks.rados.rados.0.smithi076.stdout:948: writing smithi07694878-22 from 665643 to 1363429 tid 1 2021-03-15T22:23:18.433 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:18.438 INFO:tasks.rados.rados.0.smithi076.stdout:948: writing smithi07694878-22 from 2039745 to 2792867 tid 2 2021-03-15T22:23:18.444 INFO:tasks.rados.rados.0.smithi076.stdout:948: writing smithi07694878-22 from 3534742 to 3865643 tid 3 2021-03-15T22:23:18.444 INFO:tasks.rados.rados.0.smithi076.stdout:946: done (2 left) 2021-03-15T22:23:18.444 INFO:tasks.rados.rados.0.smithi076.stdout:949: delete oid 13 current snap is 93 2021-03-15T22:23:18.445 INFO:tasks.rados.rados.0.smithi076.stdout:948: finishing write tid 1 to smithi07694878-22 2021-03-15T22:23:18.445 INFO:tasks.rados.rados.0.smithi076.stdout:948: finishing write tid 2 to smithi07694878-22 2021-03-15T22:23:18.447 INFO:tasks.rados.rados.0.smithi076.stdout:947: done (2 left) 2021-03-15T22:23:18.447 INFO:tasks.rados.rados.0.smithi076.stdout:949: done (1 left) 2021-03-15T22:23:18.447 INFO:tasks.rados.rados.0.smithi076.stdout:950: rollback oid 20 current snap is 93 2021-03-15T22:23:18.448 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 20 to 91 2021-03-15T22:23:18.448 INFO:tasks.rados.rados.0.smithi076.stdout:948: finishing write tid 3 to smithi07694878-22 2021-03-15T22:23:18.448 INFO:tasks.rados.rados.0.smithi076.stdout:951: snap_remove snap 88 2021-03-15T22:23:18.459 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:18.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:18.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:18.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:18 smithi125 conmon[56739]: cluster 2021-03-15T22:23:17.021675+0000 mgr.y (mgr.14144) 514 : cluster [DBG] pgmap v549: 138 pgs: 138 active+clean; 218 MiB data, 646 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 2.1 MiB/s wr, 16 op/s 2021-03-15T22:23:18.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:18 smithi125 conmon[56739]: cluster 2021-03-15T22:23:17.098927+0000 mon.a (mon.0) 973 : cluster [DBG] osdmap e282: 8 total, 8 up, 8 in 2021-03-15T22:23:18.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:18 smithi125 conmon[56739]: audit 2021-03-15T22:23:17.359859+0000 mon.a (mon.0) 974 : audit [INF] 2021-03-15T22:23:18.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:18 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/3521625101' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:18.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:18.585 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:18.586 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:18.687 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:18.714 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:18.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:18.715 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:18.816 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:18.838 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:18.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:18.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:18.941 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:18.963 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:18.965 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:19.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:19.092 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:19.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:19.094 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:19.112 INFO:tasks.rados.rados.0.smithi076.stdout:951: done (2 left) 2021-03-15T22:23:19.112 INFO:tasks.rados.rados.0.smithi076.stdout:952: rollback oid 25 current snap is 93 2021-03-15T22:23:19.113 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 91 2021-03-15T22:23:19.113 INFO:tasks.rados.rados.0.smithi076.stdout:953: write oid 16 current snap is 93 2021-03-15T22:23:19.113 INFO:tasks.rados.rados.0.smithi076.stdout:953: seq_num 351 ranges {699257=564637,1873062=640438,3305941=593316} 2021-03-15T22:23:19.118 INFO:tasks.rados.rados.0.smithi076.stdout:953: writing smithi07694878-16 from 699257 to 1263894 tid 1 2021-03-15T22:23:19.126 INFO:tasks.rados.rados.0.smithi076.stdout:953: writing smithi07694878-16 from 1873062 to 2513500 tid 2 2021-03-15T22:23:19.134 INFO:tasks.rados.rados.0.smithi076.stdout:953: writing smithi07694878-16 from 3305941 to 3899257 tid 3 2021-03-15T22:23:19.135 INFO:tasks.rados.rados.0.smithi076.stdout:954: snap_create 2021-03-15T22:23:19.135 INFO:tasks.rados.rados.0.smithi076.stdout:948: finishing write tid 4 to smithi07694878-22 2021-03-15T22:23:19.135 INFO:tasks.rados.rados.0.smithi076.stdout:948: finishing write tid 5 to smithi07694878-22 2021-03-15T22:23:19.136 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 488 (ObjNum 349 snap 93 seq_num 349) dirty exists 2021-03-15T22:23:19.136 INFO:tasks.rados.rados.0.smithi076.stdout:948: left oid 22 (ObjNum 349 snap 93 seq_num 349) 2021-03-15T22:23:19.136 INFO:tasks.rados.rados.0.smithi076.stdout:950: finishing rollback tid 1 to smithi07694878-20 2021-03-15T22:23:19.136 INFO:tasks.rados.rados.0.smithi076.stdout:950: finishing rollback tid 2 to smithi07694878-20 2021-03-15T22:23:19.136 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 217 (ObjNum 298 snap 78 seq_num 298) dirty exists 2021-03-15T22:23:19.137 INFO:tasks.rados.rados.0.smithi076.stdout:952: finishing rollback tid 0 to smithi07694878-25 2021-03-15T22:23:19.137 INFO:tasks.rados.rados.0.smithi076.stdout:952: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:23:19.137 INFO:tasks.rados.rados.0.smithi076.stdout:952: finishing rollback tid 2 to smithi07694878-25 2021-03-15T22:23:19.137 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 145 (ObjNum 331 snap 88 seq_num 331) dirty exists 2021-03-15T22:23:19.138 INFO:tasks.rados.rados.0.smithi076.stdout:953: finishing write tid 1 to smithi07694878-16 2021-03-15T22:23:19.138 INFO:tasks.rados.rados.0.smithi076.stdout:953: finishing write tid 2 to smithi07694878-16 2021-03-15T22:23:19.139 INFO:tasks.rados.rados.0.smithi076.stdout:953: finishing write tid 3 to smithi07694878-16 2021-03-15T22:23:19.139 INFO:tasks.rados.rados.0.smithi076.stdout:953: finishing write tid 4 to smithi07694878-16 2021-03-15T22:23:19.139 INFO:tasks.rados.rados.0.smithi076.stdout:953: finishing write tid 5 to smithi07694878-16 2021-03-15T22:23:19.139 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 500 (ObjNum 351 snap 93 seq_num 351) dirty exists 2021-03-15T22:23:19.140 INFO:tasks.rados.rados.0.smithi076.stdout:953: left oid 16 (ObjNum 351 snap 93 seq_num 351) 2021-03-15T22:23:19.195 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:19.199 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:23:19.216 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:23:19.218 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:19.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:19.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:19.234 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:23:19.251 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:23:19.269 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:23:19.286 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:23:19.303 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:23:19.321 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:19.322 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:23:19.324 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:23:19.324 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:23:19.325 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:23:19.325 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:23:19.325 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:23:19.325 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:23:19.325 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:23:19.326 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:23:19.343 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:19.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:19.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:19.410 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:19 smithi076 conmon[62156]: cluster 2021-03-15T22:23:18.093802+0000 2021-03-15T22:23:19.410 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:19 smithi076 conmon[62156]: mon.a (mon.0) 975 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:19.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:19 smithi076 conmon[62156]: audit 2021-03-15T22:23:18.106955+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.76:0/3521625101' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:19.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:19 smithi076 conmon[62156]: cluster 2021-03-15T22:23:18.107016+0000 mon.a (mon.0) 977 : cluster 2021-03-15T22:23:19.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:19 smithi076 conmon[62156]: [DBG] osdmap e283: 8 total, 8 up, 8 in 2021-03-15T22:23:19.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:19 smithi076 conmon[56514]: cluster 2021-03-15T22:23:18.093802+0000 mon.a 2021-03-15T22:23:19.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:19 smithi076 conmon[56514]: (mon.0) 975 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:19.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:19 smithi076 conmon[56514]: audit 2021-03-15T22:23:18.106955+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.76:0/3521625101' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:19.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:19 smithi076 conmon[56514]: cluster 2021-03-15T22:23:18.107016+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2021-03-15T22:23:19.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:19.472 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:19.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:19.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:19.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:19 smithi125 conmon[56739]: cluster 2021-03-15T22:23:18.093802+0000 mon.a (mon.0) 975 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:19.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:19 smithi125 conmon[56739]: audit 2021-03-15T22:23: 2021-03-15T22:23:19.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:19 smithi125 conmon[56739]: 18.106955+0000 mon.a (mon.0) 976 : audit [INF] from='client.? 172.21.15.76:0/3521625101' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:19.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:19 smithi125 conmon[56739]: cluster 2021-03-15T22:23:18.107016+0000 mon.a (mon.0) 977 : cluster [DBG] osdmap e283: 8 total, 8 up, 8 in 2021-03-15T22:23:19.575 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:19.597 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:19.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:19.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:19.700 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:19.722 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:19.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:19.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:19.825 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:19.851 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:19.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:19.852 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:19.953 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:19.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:19.978 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:19.979 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:20.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:23:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:23:20.054 INFO:tasks.rados.rados.0.smithi076.stdout:948: done (4 left) 2021-03-15T22:23:20.054 INFO:tasks.rados.rados.0.smithi076.stdout:950: done (3 left) 2021-03-15T22:23:20.055 INFO:tasks.rados.rados.0.smithi076.stdout:952: done (2 left) 2021-03-15T22:23:20.055 INFO:tasks.rados.rados.0.smithi076.stdout:953: done (1 left) 2021-03-15T22:23:20.056 INFO:tasks.rados.rados.0.smithi076.stdout:954: done (0 left) 2021-03-15T22:23:20.056 INFO:tasks.rados.rados.0.smithi076.stdout:955: delete oid 8 current snap is 94 2021-03-15T22:23:20.065 INFO:tasks.rados.rados.0.smithi076.stdout:955: done (0 left) 2021-03-15T22:23:20.065 INFO:tasks.rados.rados.0.smithi076.stdout:956: delete oid 12 current snap is 94 2021-03-15T22:23:20.071 INFO:tasks.rados.rados.0.smithi076.stdout:956: done (0 left) 2021-03-15T22:23:20.071 INFO:tasks.rados.rados.0.smithi076.stdout:957: delete oid 7 current snap is 94 2021-03-15T22:23:20.074 INFO:tasks.rados.rados.0.smithi076.stdout:957: done (0 left) 2021-03-15T22:23:20.074 INFO:tasks.rados.rados.0.smithi076.stdout:958: snap_create 2021-03-15T22:23:20.079 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:20.103 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:20.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:20.105 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:20.206 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:20.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:20.229 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:20.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:20.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[56514]: cluster 2021-03-15T22:23:19.022149+0000 mgr.y (mgr.14144) 515 2021-03-15T22:23:20.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[56514]: : cluster [DBG] pgmap v552: 138 pgs: 138 active+clean; 210 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-03-15T22:23:20.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[56514]: cluster 2021-03-15T22:23:19.109002+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2021-03-15T22:23:20.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[56514]: audit 2021-03-15T22:23:19.179214+0000 mon.a (mon.0) 979 : audit [DBG] from='client.? 172.21.15.76:0/715667357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:20.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[56514]: audit 2021-03-15T22:23:19.584490+0000 mgr.y (mgr.14144) 516 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:20.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[56514]: cluster 2021-03-15T22:23:20.050212+0000 mon.a (mon.0) 980 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2021-03-15T22:23:20.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: cluster 2021-03-15T22:23: 2021-03-15T22:23:20.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: 19.022149+0000 mgr.y (mgr.14144) 515 : cluster [DBG] pgmap v552: 138 pgs: 138 active+clean; 210 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-03-15T22:23:20.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: cluster 2021-03-15T22:23:20.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:23:20.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: 23:19.109002+0000 mon.a 2021-03-15T22:23:20.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: (mon.0) 978 : cluster 2021-03-15T22:23:20.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: [DBG] osdmap e284: 8 total, 8 up, 8 in 2021-03-15T22:23:20.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: audit 2021-03-15T22:23:20.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:23:20.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: 22:23:19.179214+0000 2021-03-15T22:23:20.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: mon.a (mon.0) 979 : 2021-03-15T22:23:20.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: audit [DBG] from='client.? 172.21.15.76:0/715667357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:20.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: audit 2021-03-15T22:23:20.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:23:20.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: 22:23:19.584490+0000 2021-03-15T22:23:20.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: mgr.y (mgr.14144) 516 : 2021-03-15T22:23:20.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:20.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:23:20.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: -15T22:23:20. 2021-03-15T22:23:20.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: 050212+0000 mon.a (mon.0 2021-03-15T22:23:20.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:20 smithi076 conmon[62156]: ) 980 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2021-03-15T22:23:20.331 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:20.357 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:20.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:20.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:20.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:20.486 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:20.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:20.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:20 smithi125 conmon[56739]: cluster 2021-03-15T22:23:19.022149+0000 mgr.y (mgr.14144) 515 : cluster [DBG] pgmap v552: 138 pgs: 138 active+clean; 210 MiB data, 631 MiB used, 715 GiB / 715 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-03-15T22:23:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:20 smithi125 conmon[56739]: cluster 2021-03-15T22:23:19.109002+0000 mon.a (mon.0) 978 : cluster [DBG] osdmap e284: 8 total, 8 up, 8 in 2021-03-15T22:23:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:20 smithi125 conmon[56739]: audit 2021-03-15T22:23:19.179214+0000 mon.a (mon.0) 979 : audit [DBG] from='client.? 172.21.15.76:0/715667357' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:20 smithi125 conmon[56739]: audit 2021-03-15T22:23:19.584490+0000 mgr.y ( 2021-03-15T22:23:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:20 smithi125 conmon[56739]: mgr.14144) 516 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:20 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:23:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:20 smithi125 conmon[56739]: -15T22:23:20.050212+0000 2021-03-15T22:23:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:20 smithi125 conmon[56739]: mon.a (mon.0) 980 : cluster [DBG] osdmap e285: 8 total, 8 up, 8 in 2021-03-15T22:23:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:20 smithi125 conmon[56739]: 2021-03-15T22:23:20.558 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:23:20.559 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:23:20.589 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:20.612 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:20.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:20.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:20.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:20.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:20.739 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:20.740 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:20.841 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:20.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:20.865 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:20.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:20.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:20.989 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:20.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:20.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:21.061 INFO:tasks.rados.rados.0.smithi076.stdout:958: done (0 left) 2021-03-15T22:23:21.062 INFO:tasks.rados.rados.0.smithi076.stdout:959: copy_from oid 22 from oid 47 current snap is 95 2021-03-15T22:23:21.062 INFO:tasks.rados.rados.0.smithi076.stdout:960: read oid 27 snap 94 2021-03-15T22:23:21.062 INFO:tasks.rados.rados.0.smithi076.stdout:960: expect deleted 2021-03-15T22:23:21.062 INFO:tasks.rados.rados.0.smithi076.stdout:961: write (excl) oid 44 current snap is 95 2021-03-15T22:23:21.063 INFO:tasks.rados.rados.0.smithi076.stdout:961: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2021-03-15T22:23:21.068 INFO:tasks.rados.rados.0.smithi076.stdout:961: writing smithi07694878-44 from 766485 to 1232118 tid 1 2021-03-15T22:23:21.076 INFO:tasks.rados.rados.0.smithi076.stdout:961: writing smithi07694878-44 from 1990637 to 2638413 tid 2 2021-03-15T22:23:21.084 INFO:tasks.rados.rados.0.smithi076.stdout:961: writing smithi07694878-44 from 3299280 to 3966485 tid 3 2021-03-15T22:23:21.085 INFO:tasks.rados.rados.0.smithi076.stdout:962: write oid 10 current snap is 95 2021-03-15T22:23:21.085 INFO:tasks.rados.rados.0.smithi076.stdout:962: seq_num 356 ranges {783292=540882,1932766=507787,3210350=572013} 2021-03-15T22:23:21.090 INFO:tasks.rados.rados.0.smithi076.stdout:962: writing smithi07694878-10 from 783292 to 1324174 tid 1 2021-03-15T22:23:21.092 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:21.097 INFO:tasks.rados.rados.0.smithi076.stdout:962: writing smithi07694878-10 from 1932766 to 2440553 tid 2 2021-03-15T22:23:21.104 INFO:tasks.rados.rados.0.smithi076.stdout:962: writing smithi07694878-10 from 3210350 to 3782363 tid 3 2021-03-15T22:23:21.104 INFO:tasks.rados.rados.0.smithi076.stdout:963: snap_remove snap 91 2021-03-15T22:23:21.116 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:21.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:21.118 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:21.219 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:21.244 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:21.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:21.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:21.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:21.371 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:21.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:21.423 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:23:21.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:23:21.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:21.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:21.508 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:21.508 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:21.610 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:21.638 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:21.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:21.639 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:21.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:21.772 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:21.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:21.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:21.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:21.906 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:21.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:21.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:21.966 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:22.012 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:22.036 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:22.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:22.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:22.071 INFO:tasks.rados.rados.0.smithi076.stdout:963: done (4 left) 2021-03-15T22:23:22.071 INFO:tasks.rados.rados.0.smithi076.stdout:964: snap_remove snap 72 2021-03-15T22:23:22.072 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:22.139 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:22.162 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:22.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:22.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:22.186 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:22 smithi076 conmon[56514]: cluster 2021-03-15T22:23:21.022878+0000 mgr.y (mgr.14144) 517 : cluster [DBG] pgmap v555: 138 pgs: 138 active+clean; 206 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 3.6 MiB/s rd, 442 KiB/s wr, 7 op/s 2021-03-15T22:23:22.187 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:22 smithi076 conmon[56514]: cluster 2021-03-15T22:23:21.056464+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2021-03-15T22:23:22.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:22 smithi076 conmon[62156]: cluster 2021-03-15T22:23:21.022878+0000 mgr.y (mgr.14144) 517 : cluster [DBG] pgmap v555: 138 pgs: 138 active+clean; 206 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 3.6 MiB/s rd, 442 KiB/s wr, 7 op/s 2021-03-15T22:23:22.188 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:22 smithi076 conmon[62156]: cluster 2021-03-15T22:23:21.056464+0000 mon.a (mon.0) 981 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2021-03-15T22:23:22.265 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:22.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:22.290 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:22.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:22.392 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:22.416 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:22.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:22.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:22.519 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:22.525 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:23:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:22 smithi125 conmon[56739]: cluster 2021-03-15T22:23:21.022878+0000 mgr.y (mgr.14144) 517 : cluster [DBG] pgmap v555: 138 pgs: 138 active+clean; 206 MiB data, 634 MiB used, 715 GiB / 715 GiB avail; 3.6 MiB/s rd, 442 KiB/s wr, 7 op/s 2021-03-15T22:23:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:22 smithi125 conmon[56739]: cluster 2021-03-15T22:23:21.056464+0000 2021-03-15T22:23:22.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:22 smithi125 conmon[56739]: mon.a (mon.0) 981 : cluster [DBG] osdmap e286: 8 total, 8 up, 8 in 2021-03-15T22:23:22.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:22.541 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:22.542 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:22.543 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 414 (0%) 2021-03-15T22:23:22.543 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:23:22.543 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 11.1327 -> 11.1327 (expected baseline 6.72913) 2021-03-15T22:23:22.543 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 25 -> 25 pgs (0.483092 -> 0.483092 * mean) 2021-03-15T22:23:22.544 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 64 -> 64 pgs (1.23671 -> 1.23671 * mean) 2021-03-15T22:23:22.544 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:23:22.544 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:23:22.544 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 3 2021-03-15T22:23:22.545 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:23:22.545 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0009 2021-03-15T22:23:22.545 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0088 2021-03-15T22:23:22.546 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:23:22.643 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:22.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:22.669 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:22.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:22.771 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:22.798 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:22.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:22.800 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:22.901 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:22.924 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:22.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:22.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:23.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:23.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:23.050 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:23.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:23.075 INFO:tasks.rados.rados.0.smithi076.stdout:964: done (4 left) 2021-03-15T22:23:23.076 INFO:tasks.rados.rados.0.smithi076.stdout:965: snap_create 2021-03-15T22:23:23.076 INFO:tasks.rados.rados.0.smithi076.stdout:961: finishing write tid 1 to smithi07694878-44 2021-03-15T22:23:23.076 INFO:tasks.rados.rados.0.smithi076.stdout:961: finishing write tid 2 to smithi07694878-44 2021-03-15T22:23:23.076 INFO:tasks.rados.rados.0.smithi076.stdout:959: finishing copy_from to smithi07694878-22 2021-03-15T22:23:23.077 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 164 (ObjNum 324 snap 87 seq_num 324) dirty exists 2021-03-15T22:23:23.077 INFO:tasks.rados.rados.0.smithi076.stdout:959: finishing copy_from racing read to smithi07694878-22 2021-03-15T22:23:23.077 INFO:tasks.rados.rados.0.smithi076.stdout:961: finishing write tid 3 to smithi07694878-44 2021-03-15T22:23:23.077 INFO:tasks.rados.rados.0.smithi076.stdout:961: finishing write tid 4 to smithi07694878-44 2021-03-15T22:23:23.078 INFO:tasks.rados.rados.0.smithi076.stdout:961: finishing write tid 5 to smithi07694878-44 2021-03-15T22:23:23.078 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 193 (ObjNum 355 snap 95 seq_num 355) dirty exists 2021-03-15T22:23:23.078 INFO:tasks.rados.rados.0.smithi076.stdout:961: left oid 44 (ObjNum 355 snap 95 seq_num 355) 2021-03-15T22:23:23.078 INFO:tasks.rados.rados.0.smithi076.stdout:962: finishing write tid 1 to smithi07694878-10 2021-03-15T22:23:23.079 INFO:tasks.rados.rados.0.smithi076.stdout:962: finishing write tid 2 to smithi07694878-10 2021-03-15T22:23:23.079 INFO:tasks.rados.rados.0.smithi076.stdout:962: finishing write tid 3 to smithi07694878-10 2021-03-15T22:23:23.079 INFO:tasks.rados.rados.0.smithi076.stdout:962: finishing write tid 4 to smithi07694878-10 2021-03-15T22:23:23.079 INFO:tasks.rados.rados.0.smithi076.stdout:962: finishing write tid 5 to smithi07694878-10 2021-03-15T22:23:23.079 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 149 (ObjNum 356 snap 95 seq_num 356) dirty exists 2021-03-15T22:23:23.080 INFO:tasks.rados.rados.0.smithi076.stdout:962: left oid 10 (ObjNum 356 snap 95 seq_num 356) 2021-03-15T22:23:23.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:23.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:23.179 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:23.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:23.270 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:23:23 smithi076 conmon[72291]: debug 2021-03-15T22:23:23.076+0000 7fb92f173700 -1 osd.2 pg_epoch: 288 pg[7.e( v 280'69 (0'0,280'69] local-lis/les=91/92 n=2 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [2,5,0] r=0 lpr=91 crt=280'69 lcod 271'67 mlcod 271'67 active+clean+snaptrim trimq=[8d~1] ps=[a8~1,ad~1,b5~2]] removing snap head 2021-03-15T22:23:23.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:23 smithi076 conmon[56514]: cluster 2021-03-15T22:23:22.067399+0000 mon.a (mon.0) 982 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2021-03-15T22:23:23.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:23 smithi076 conmon[56514]: audit 2021-03-15T22:23:22.541200+0000 mgr.y (mgr.14144) 518 : audit [DBG] from='client.15174 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:23.272 INFO:journalctl@ceph.osd.1.smithi076.stdout:Mar 15 22:23:23 smithi076 conmon[68886]: debug 2021-03-15T22:23:23.272 INFO:journalctl@ceph.osd.1.smithi076.stdout:Mar 15 22:23:23 smithi076 conmon[68886]: 2021-03-15T22:23:23.073+0000 7fae30da5700 -1 osd.1 pg_epoch: 288 pg[7.b( v 278'163 (0'0,278'163] local-lis/les=91/92 n=5 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [1,5,6] r=0 lpr=91 crt=278'163 lcod 278'161 mlcod 278'161 active+clean+snaptrim trimq=[8d~1] ps=[a8~1,ad~1,b5~2]] removing snap head 2021-03-15T22:23:23.272 INFO:journalctl@ceph.osd.1.smithi076.stdout:Mar 15 22:23:23 smithi076 conmon[68886]: 2021-03-15T22:23:23.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:23 smithi076 conmon[62156]: cluster 2021-03-15T22:23:22.067399+0000 2021-03-15T22:23:23.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:23 smithi076 conmon[62156]: mon.a (mon.0) 982 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2021-03-15T22:23:23.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:23 smithi076 conmon[62156]: audit 2021-03-15T22:23:22.541200+0000 mgr.y (mgr.14144) 518 : audit [DBG] from='client.15174 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:23.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:23.305 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:23.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:23.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:23.408 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:23.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:23.435 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:23.436 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:23.505 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2021-03-15T22:23:23.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd test-reweight-by-utilization 110 3.0 2021-03-15T22:23:23.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:23 smithi125 conmon[56739]: cluster 2021-03-15T22:23:22.067399+0000 mon.a (mon. 2021-03-15T22:23:23.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:23 smithi125 conmon[56739]: 0) 982 : cluster [DBG] osdmap e287: 8 total, 8 up, 8 in 2021-03-15T22:23:23.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:23 smithi125 conmon[56739]: audit 2021-03-15T22:23:22.541200+0000 mgr.y (mgr.14144) 518 : audit [DBG] from='client.15174 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:23.537 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:23.563 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:23.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:23.565 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:23.666 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:23.691 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:23.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:23.693 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:23.794 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:23.817 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:23.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:23.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:23.919 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:23.942 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:23.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:23.943 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:24.044 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:24.071 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:24.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:24.073 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:24.079 INFO:tasks.rados.rados.0.smithi076.stdout:959: done (4 left) 2021-03-15T22:23:24.080 INFO:tasks.rados.rados.0.smithi076.stdout:960: done (3 left) 2021-03-15T22:23:24.081 INFO:tasks.rados.rados.0.smithi076.stdout:961: done (2 left) 2021-03-15T22:23:24.081 INFO:tasks.rados.rados.0.smithi076.stdout:962: done (1 left) 2021-03-15T22:23:24.081 INFO:tasks.rados.rados.0.smithi076.stdout:965: done (0 left) 2021-03-15T22:23:24.081 INFO:tasks.rados.rados.0.smithi076.stdout:966: write (excl) oid 15 current snap is 96 2021-03-15T22:23:24.081 INFO:tasks.rados.rados.0.smithi076.stdout:966: seq_num 357 ranges {400099=616131,1591248=408851} 2021-03-15T22:23:24.085 INFO:tasks.rados.rados.0.smithi076.stdout:966: writing smithi07694878-15 from 400099 to 1016230 tid 1 2021-03-15T22:23:24.091 INFO:tasks.rados.rados.0.smithi076.stdout:966: writing smithi07694878-15 from 1591248 to 2000099 tid 2 2021-03-15T22:23:24.091 INFO:tasks.rados.rados.0.smithi076.stdout:967: read oid 42 snap -1 2021-03-15T22:23:24.092 INFO:tasks.rados.rados.0.smithi076.stdout:967: expect (ObjNum 326 snap 87 seq_num 326) 2021-03-15T22:23:24.092 INFO:tasks.rados.rados.0.smithi076.stdout:968: write oid 41 current snap is 96 2021-03-15T22:23:24.092 INFO:tasks.rados.rados.0.smithi076.stdout:968: seq_num 358 ranges {416906=407733,1366083=511456} 2021-03-15T22:23:24.096 INFO:tasks.rados.rados.0.smithi076.stdout:968: writing smithi07694878-41 from 416906 to 824639 tid 1 2021-03-15T22:23:24.103 INFO:tasks.rados.rados.0.smithi076.stdout:968: writing smithi07694878-41 from 1366083 to 1877539 tid 2 2021-03-15T22:23:24.103 INFO:tasks.rados.rados.0.smithi076.stdout:969: read oid 43 snap -1 2021-03-15T22:23:24.103 INFO:tasks.rados.rados.0.smithi076.stdout:969: expect (ObjNum 239 snap 57 seq_num 239) 2021-03-15T22:23:24.103 INFO:tasks.rados.rados.0.smithi076.stdout:970: write oid 49 current snap is 96 2021-03-15T22:23:24.104 INFO:tasks.rados.rados.0.smithi076.stdout:966: finishing write tid 1 to smithi07694878-15 2021-03-15T22:23:24.104 INFO:tasks.rados.rados.0.smithi076.stdout:970: seq_num 359 ranges {433713=482982,1424565=609148} 2021-03-15T22:23:24.107 INFO:tasks.rados.rados.0.smithi076.stdout:970: writing smithi07694878-49 from 433713 to 916695 tid 1 2021-03-15T22:23:24.115 INFO:tasks.rados.rados.0.smithi076.stdout:970: writing smithi07694878-49 from 1424565 to 2033713 tid 2 2021-03-15T22:23:24.115 INFO:tasks.rados.rados.0.smithi076.stdout:971: read oid 37 snap -1 2021-03-15T22:23:24.115 INFO:tasks.rados.rados.0.smithi076.stdout:971: expect (ObjNum 281 snap 74 seq_num 281) 2021-03-15T22:23:24.115 INFO:tasks.rados.rados.0.smithi076.stdout:972: rollback oid 27 current snap is 96 2021-03-15T22:23:24.116 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 94 2021-03-15T22:23:24.116 INFO:tasks.rados.rados.0.smithi076.stdout:966: finishing write tid 2 to smithi07694878-15 2021-03-15T22:23:24.116 INFO:tasks.rados.rados.0.smithi076.stdout:966: finishing write tid 3 to smithi07694878-15 2021-03-15T22:23:24.116 INFO:tasks.rados.rados.0.smithi076.stdout:966: finishing write tid 4 to smithi07694878-15 2021-03-15T22:23:24.116 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 525 (ObjNum 357 snap 96 seq_num 357) dirty exists 2021-03-15T22:23:24.117 INFO:tasks.rados.rados.0.smithi076.stdout:966: left oid 15 (ObjNum 357 snap 96 seq_num 357) 2021-03-15T22:23:24.117 INFO:tasks.rados.rados.0.smithi076.stdout:966: done (6 left) 2021-03-15T22:23:24.117 INFO:tasks.rados.rados.0.smithi076.stdout:973: copy_from oid 9 from oid 22 current snap is 96 2021-03-15T22:23:24.117 INFO:tasks.rados.rados.0.smithi076.stdout:968: finishing write tid 1 to smithi07694878-41 2021-03-15T22:23:24.118 INFO:tasks.rados.rados.0.smithi076.stdout:968: finishing write tid 2 to smithi07694878-41 2021-03-15T22:23:24.118 INFO:tasks.rados.rados.0.smithi076.stdout:967: expect (ObjNum 326 snap 87 seq_num 326) 2021-03-15T22:23:24.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:24.196 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:24.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:24.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:24.225 INFO:tasks.rados.rados.0.smithi076.stdout:968: finishing write tid 3 to smithi07694878-41 2021-03-15T22:23:24.226 INFO:tasks.rados.rados.0.smithi076.stdout:968: finishing write tid 4 to smithi07694878-41 2021-03-15T22:23:24.226 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 221 (ObjNum 358 snap 96 seq_num 358) dirty exists 2021-03-15T22:23:24.226 INFO:tasks.rados.rados.0.smithi076.stdout:968: left oid 41 (ObjNum 358 snap 96 seq_num 358) 2021-03-15T22:23:24.226 INFO:tasks.rados.rados.0.smithi076.stdout:967: done (6 left) 2021-03-15T22:23:24.227 INFO:tasks.rados.rados.0.smithi076.stdout:968: done (5 left) 2021-03-15T22:23:24.227 INFO:tasks.rados.rados.0.smithi076.stdout:974: delete oid 44 current snap is 96 2021-03-15T22:23:24.227 INFO:tasks.rados.rados.0.smithi076.stdout:969: expect (ObjNum 239 snap 57 seq_num 239) 2021-03-15T22:23:24.299 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:24.311 INFO:tasks.rados.rados.0.smithi076.stdout:970: finishing write tid 1 to smithi07694878-49 2021-03-15T22:23:24.312 INFO:tasks.rados.rados.0.smithi076.stdout:969: done (5 left) 2021-03-15T22:23:24.312 INFO:tasks.rados.rados.0.smithi076.stdout:974: done (4 left) 2021-03-15T22:23:24.312 INFO:tasks.rados.rados.0.smithi076.stdout:975: snap_remove snap 81 2021-03-15T22:23:24.321 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:24.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:24.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:24.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:24.448 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:24.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:24.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:24.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:24 smithi076 conmon[56514]: cluster 2021-03-15T22:23:23.023310+0000 mgr.y 2021-03-15T22:23:24.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:24 smithi076 conmon[56514]: (mgr.14144) 519 : cluster [DBG] pgmap v558: 138 pgs: 138 active+clean; 211 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.8 MiB/s wr, 23 op/s 2021-03-15T22:23:24.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:24 smithi076 conmon[56514]: cluster 2021-03-15T22:23:23.073313+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2021-03-15T22:23:24.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:24 smithi076 conmon[62156]: cluster 2021-03-15T22:23:23.023310 2021-03-15T22:23:24.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:24 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 519 : cluster [DBG] pgmap v558: 138 pgs: 138 active+clean; 211 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.8 MiB/s wr, 23 op/s 2021-03-15T22:23:24.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:24 smithi076 conmon[62156]: cluster 2021-03-15T22:23:23.073313+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2021-03-15T22:23:24.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:24 smithi125 conmon[56739]: cluster 2021-03-15T22:23:23.023310+0000 mgr.y (mgr.14144) 519 : cluster [DBG] pgmap v558: 138 pgs: 138 active+clean; 211 MiB data, 643 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 2.8 MiB/s wr, 23 op/s 2021-03-15T22:23:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:24 smithi125 conmon[56739]: cluster 2021-03-15T22:23:23.073313+0000 mon.a (mon.0) 983 : cluster [DBG] osdmap e288: 8 total, 8 up, 8 in 2021-03-15T22:23:24.550 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:24.574 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:24.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:24.647 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:24.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:24.701 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:24.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:24.702 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:24.741 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:24.803 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:24.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:24.827 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:24.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:24.929 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:24.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:24.956 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:24.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:25.054 INFO:tasks.rados.rados.0.smithi076.stdout:975: done (4 left) 2021-03-15T22:23:25.055 INFO:tasks.rados.rados.0.smithi076.stdout:976: read oid 41 snap -1 2021-03-15T22:23:25.055 INFO:tasks.rados.rados.0.smithi076.stdout:976: expect (ObjNum 358 snap 96 seq_num 358) 2021-03-15T22:23:25.055 INFO:tasks.rados.rados.0.smithi076.stdout:977: delete oid 50 current snap is 96 2021-03-15T22:23:25.055 INFO:tasks.rados.rados.0.smithi076.stdout:972: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:23:25.056 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:23:25.056 INFO:tasks.rados.rados.0.smithi076.stdout:970: finishing write tid 2 to smithi07694878-49 2021-03-15T22:23:25.056 INFO:tasks.rados.rados.0.smithi076.stdout:970: finishing write tid 3 to smithi07694878-49 2021-03-15T22:23:25.056 INFO:tasks.rados.rados.0.smithi076.stdout:970: finishing write tid 4 to smithi07694878-49 2021-03-15T22:23:25.057 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 529 (ObjNum 359 snap 96 seq_num 359) dirty exists 2021-03-15T22:23:25.057 INFO:tasks.rados.rados.0.smithi076.stdout:970: left oid 49 (ObjNum 359 snap 96 seq_num 359) 2021-03-15T22:23:25.057 INFO:tasks.rados.rados.0.smithi076.stdout:971: expect (ObjNum 281 snap 74 seq_num 281) 2021-03-15T22:23:25.057 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:25.083 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:23:25 smithi076 conmon[65511]: debug 2021-03-15T22:23:25.078+0000 7f1a1f076700 -1 osd.0 pg_epoch: 290 pg[7.f( v 288'333 (0'0,288'333] local-lis/les=263/264 n=8 ec=91/91 lis/c=263/263 les/c/f=264/264/0 sis=263) [0,3,7] r=0 lpr=263 crt=288'333 lcod 288'331 mlcod 288'331 active+clean+snaptrim trimq=[9c~1] ps=[8d~1,b5~1]] removing snap head 2021-03-15T22:23:25.085 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:25.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:25.087 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:25.188 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:25.198 INFO:tasks.rados.rados.0.smithi076.stdout:973: finishing copy_from to smithi07694878-9 2021-03-15T22:23:25.199 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 164 (ObjNum 324 snap 87 seq_num 324) dirty exists 2021-03-15T22:23:25.199 INFO:tasks.rados.rados.0.smithi076.stdout:973: finishing copy_from racing read to smithi07694878-9 2021-03-15T22:23:25.199 INFO:tasks.rados.rados.0.smithi076.stdout:970: done (5 left) 2021-03-15T22:23:25.200 INFO:tasks.rados.rados.0.smithi076.stdout:971: done (4 left) 2021-03-15T22:23:25.200 INFO:tasks.rados.rados.0.smithi076.stdout:972: done (3 left) 2021-03-15T22:23:25.200 INFO:tasks.rados.rados.0.smithi076.stdout:973: done (2 left) 2021-03-15T22:23:25.200 INFO:tasks.rados.rados.0.smithi076.stdout:977: done (1 left) 2021-03-15T22:23:25.200 INFO:tasks.rados.rados.0.smithi076.stdout:978: snap_remove snap 94 2021-03-15T22:23:25.213 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:25.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:25.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:25.316 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:25.339 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:25 smithi076 conmon[56514]: cluster 2021-03-15T22:23:24.075246+0000 mon.a 2021-03-15T22:23:25.339 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:25 smithi076 conmon[56514]: (mon.0) 984 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2021-03-15T22:23:25.339 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:25 smithi076 conmon[56514]: cluster 2021-03-15T22:23:25.051863+0000 mon.a ( 2021-03-15T22:23:25.339 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:25 smithi076 conmon[56514]: mon.0) 985 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2021-03-15T22:23:25.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:25 smithi076 conmon[62156]: cluster 2021-03-15T22:23:24.075246 2021-03-15T22:23:25.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:25 smithi076 conmon[62156]: +0000 mon.a (mon.0) 984 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2021-03-15T22:23:25.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:25 smithi076 conmon[62156]: cluster 2021-03-15T22:23:25.051863+0000 2021-03-15T22:23:25.341 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:25 smithi076 conmon[62156]: mon.a (mon.0) 985 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2021-03-15T22:23:25.341 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:23:25 smithi076 conmon[65511]: debug 2021-03-15T22:23:25.341 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:23:25 smithi076 conmon[65511]: 2021-03-15T22:23:25.087+0000 7f1a1f076700 -1 osd.0 pg_epoch: 290 pg[7.f( v 290'339 (0'0,290'339] local-lis/les=263/264 n=4 ec=91/91 lis/c=263/263 les/c/f=264/264/0 sis=263) [0,3,7] r=0 lpr=263 luod=290'337 crt=290'339 lcod 290'335 mlcod 290'335 active+clean+snaptrim trimq=[9c~1] ps=[8d~1,b5~1]] removing snap head 2021-03-15T22:23:25.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:25.343 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:25.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:25.445 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:25.470 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:25.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:25.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:25.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:25 smithi125 conmon[56739]: cluster 2021-03-15T22:23:24.075246+0000 mon.a (mon.0 2021-03-15T22:23:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:25 smithi125 conmon[56739]: ) 984 : cluster [DBG] osdmap e289: 8 total, 8 up, 8 in 2021-03-15T22:23:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:25 smithi125 conmon[56739]: cluster 2021-03-15T22:23:25.051863+0000 mon.a (mon.0) 985 : cluster [DBG] osdmap e290: 8 total, 8 up, 8 in 2021-03-15T22:23:25.537 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:23:25 smithi125 conmon[63905]: debug 2021-03-15T22:23:25.090+0000 7fce04c39700 -1 osd.5 pg_epoch: 290 pg[7.0( v 290'564 (0'0,290'564] local-lis/les=268/269 n=9 ec=91/91 lis/c=268/268 les/c/f=269/269/0 sis=268) [5,1,3] r=0 lpr=268 crt=290'564 lcod 290'562 mlcod 290'562 active+clean+snaptrim trimq=[9c~1] ps=[8d~1,b5~1]] removing snap head 2021-03-15T22:23:25.538 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:23:25 smithi125 conmon[67289]: debug 2021-03-15T22:23:25.538 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:23:25 smithi125 conmon[67289]: 2021-03-15T22:23:25.060+0000 7fd4da034700 -1 osd.6 pg_epoch: 290 pg[7.c( v 289'221 (0'0,289'221] local-lis/les=175/176 n=5 ec=91/91 lis/c=175/175 les/c/f=176/176/0 sis=175) [6,7,2] r=0 lpr=175 crt=289'221 lcod 289'220 mlcod 289'220 active+clean+snaptrim trimq=[9c~1] ps=[8d~1,b5~1]] removing snap head 2021-03-15T22:23:25.539 INFO:journalctl@ceph.osd.4.smithi125.stdout:Mar 15 22:23:25 smithi125 conmon[60581]: debug 2021-03-15T22:23:25.051+0000 7fe6a8b9d700 -1 osd.4 pg_epoch: 290 pg[7.1( v 288'93 (0'0,288'93] local-lis/les=172/173 n=2 ec=91/91 lis/c=172/172 les/c/f=173/173/0 sis=172) [4,6,1] r=0 lpr=172 crt=288'93 lcod 280'91 mlcod 280'91 active+clean+snaptrim trimq=[9c~1] ps=[8d~1,b5~1]] removing snap head 2021-03-15T22:23:25.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:25.598 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:25.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:25.600 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:25.701 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:25.727 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:25.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:25.729 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:25.758 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:23:25.759 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:23:25.830 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:25.856 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:25.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:25.857 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:25.958 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:25.984 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:25.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:25.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:26.087 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:26.108 INFO:tasks.rados.rados.0.smithi076.stdout:978: done (1 left) 2021-03-15T22:23:26.108 INFO:tasks.rados.rados.0.smithi076.stdout:979: delete oid 11 current snap is 96 2021-03-15T22:23:26.109 INFO:tasks.rados.rados.0.smithi076.stdout:976: expect (ObjNum 358 snap 96 seq_num 358) 2021-03-15T22:23:26.114 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:23:26.116 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:26.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:26.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:26.218 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:26.220 INFO:tasks.rados.rados.0.smithi076.stdout:976: done (1 left) 2021-03-15T22:23:26.220 INFO:tasks.rados.rados.0.smithi076.stdout:979: done (0 left) 2021-03-15T22:23:26.220 INFO:tasks.rados.rados.0.smithi076.stdout:980: snap_create 2021-03-15T22:23:26.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:26.243 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:26.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:26.344 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:26.366 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:26.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:26.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:26.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:26.582 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:26 smithi076 conmon[56514]: cluster 2021-03-15T22:23:25.023796+0000 mgr.y (mgr. 2021-03-15T22:23:26.582 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:26 smithi076 conmon[56514]: 14144) 520 : cluster [DBG] pgmap v561: 138 pgs: 138 active+clean; 209 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 8.4 MiB/s rd, 2.8 MiB/s wr, 19 op/s 2021-03-15T22:23:26.582 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:26 smithi076 conmon[56514]: audit 2021-03-15T22:23:25.249658+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.76:0/3442650908' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:23:26.583 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:26 smithi076 conmon[56514]: audit 2021-03-15T22:23:25.249985+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:23:26.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:26 smithi076 conmon[62156]: cluster 2021-03-15T22:23:25.023796+0000 mgr.y 2021-03-15T22:23:26.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:26 smithi076 conmon[62156]: (mgr.14144) 520 : cluster [DBG] pgmap v561: 138 pgs: 138 active+clean; 209 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 8.4 MiB/s rd, 2.8 MiB/s wr, 19 op/s 2021-03-15T22:23:26.584 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:26 smithi076 conmon[62156]: audit 2021-03-15T22:23:25.249658+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.76:0/3442650908' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:23:26.584 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:26 smithi076 conmon[62156]: audit 2021-03-15T22:23:25.249985+0000 mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:23:26.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:26 smithi125 conmon[56739]: cluster 2021-03-15T22:23:25.023796+0000 mgr.y (mgr.14144) 520 : cluster 2021-03-15T22:23:26.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:26 smithi125 conmon[56739]: [DBG] pgmap v561: 138 pgs: 138 active+clean; 209 MiB data, 619 MiB used, 715 GiB / 715 GiB avail; 8.4 MiB/s rd, 2.8 MiB/s wr, 19 op/s 2021-03-15T22:23:26.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:26 smithi125 conmon[56739]: audit 2021-03-15T22:23:25.249658+0000 mon.c (mon.1) 26 : audit [INF] from='client.? 172.21.15.76:0/3442650908' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:23:26.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:26 smithi125 conmon[56739]: audit 2021-03-15T22:23:25.249985+0000 2021-03-15T22:23:26.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:26 smithi125 conmon[56739]: mon.a (mon.0) 986 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:23:26.611 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:26.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:26.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:26.714 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:26.741 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:26.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:26.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:26.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:26.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:26.872 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:26.872 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:26.974 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:27.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:27.002 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:27.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:27.105 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:27.108 INFO:tasks.rados.rados.0.smithi076.stdout:980: done (0 left) 2021-03-15T22:23:27.108 INFO:tasks.rados.rados.0.smithi076.stdout:981: copy_from oid 17 from oid 11 current snap is 97 2021-03-15T22:23:27.109 INFO:tasks.rados.rados.0.smithi076.stdout:982: delete oid 3 current snap is 97 2021-03-15T22:23:27.112 INFO:tasks.rados.rados.0.smithi076.stdout:981: finishing copy_from to smithi07694878-17 2021-03-15T22:23:27.112 INFO:tasks.rados.rados.0.smithi076.stdout:981: got expected ENOENT (src dne) 2021-03-15T22:23:27.113 INFO:tasks.rados.rados.0.smithi076.stdout:981: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:23:27.113 INFO:tasks.rados.rados.0.smithi076.stdout:981: done (1 left) 2021-03-15T22:23:27.113 INFO:tasks.rados.rados.0.smithi076.stdout:982: done (0 left) 2021-03-15T22:23:27.113 INFO:tasks.rados.rados.0.smithi076.stdout:983: write (excl) oid 28 current snap is 97 2021-03-15T22:23:27.114 INFO:tasks.rados.rados.0.smithi076.stdout:983: seq_num 364 ranges {517748=459227,1600622=406110} 2021-03-15T22:23:27.118 INFO:tasks.rados.rados.0.smithi076.stdout:983: writing smithi07694878-28 from 517748 to 976975 tid 1 2021-03-15T22:23:27.124 INFO:tasks.rados.rados.0.smithi076.stdout:983: writing smithi07694878-28 from 1600622 to 2006732 tid 2 2021-03-15T22:23:27.124 INFO:tasks.rados.rados.0.smithi076.stdout:984: copy_from oid 9 from oid 5 current snap is 97 2021-03-15T22:23:27.124 INFO:tasks.rados.rados.0.smithi076.stdout:983: finishing write tid 1 to smithi07694878-28 2021-03-15T22:23:27.125 INFO:tasks.rados.rados.0.smithi076.stdout:985: snap_create 2021-03-15T22:23:27.128 INFO:tasks.rados.rados.0.smithi076.stdout:983: finishing write tid 2 to smithi07694878-28 2021-03-15T22:23:27.129 INFO:tasks.rados.rados.0.smithi076.stdout:983: finishing write tid 3 to smithi07694878-28 2021-03-15T22:23:27.129 INFO:tasks.rados.rados.0.smithi076.stdout:983: finishing write tid 4 to smithi07694878-28 2021-03-15T22:23:27.130 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 572 (ObjNum 364 snap 97 seq_num 364) dirty exists 2021-03-15T22:23:27.130 INFO:tasks.rados.rados.0.smithi076.stdout:983: left oid 28 (ObjNum 364 snap 97 seq_num 364) 2021-03-15T22:23:27.134 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:27.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:27.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:27.140 INFO:tasks.rados.rados.0.smithi076.stdout:984: finishing copy_from to smithi07694878-9 2021-03-15T22:23:27.140 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 141 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:23:27.141 INFO:tasks.rados.rados.0.smithi076.stdout:984: finishing copy_from racing read to smithi07694878-9 2021-03-15T22:23:27.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:27.260 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:27.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:27.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:27.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:27.367 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: cluster 2021-03-15T22:23:25.549756+0000 osd.2 (osd.2) 2021-03-15T22:23:27.367 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: 53 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:27.367 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: cluster 2021-03-15T22:23:25.549952+0000 osd.2 (osd.2) 54 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:27.368 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: cluster 2021-03-15T22:23 2021-03-15T22:23:27.368 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: :26.093670+0000 mon.a (mon.0) 987 2021-03-15T22:23:27.369 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:27.369 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:23:27.369 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: 03-15T22:23:26. 2021-03-15T22:23:27.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: 103597+0000 mon.a (mon.0) 988 : audit 2021-03-15T22:23:27.370 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:23:27.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:23:27.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: 15T22:23:26.103677+0000 2021-03-15T22:23:27.372 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: mon.a (mon.0) 989 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2021-03-15T22:23:27.372 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: 2021-03-15T22:23:27.372 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: cluster 2021-03-15T22:23:27.372 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: 2021-03-15T22:23:26. 2021-03-15T22:23:27.373 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: 791034+0000 osd.0 (osd.0) 59 : cluster 2021-03-15T22:23:27.373 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[56514]: [DBG] 7.f scrub ok 2021-03-15T22:23:27.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: cluster 2021-03-15T22:23:25.549756 2021-03-15T22:23:27.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: +0000 osd.2 (osd.2) 53 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:27.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:23:27.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: -03-15T22:23:25.549952+0000 2021-03-15T22:23:27.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: osd.2 (osd.2) 54 : cluster [DBG] 2021-03-15T22:23:27.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: purged_snaps scrub ok 2021-03-15T22:23:27.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:23:27.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: :23:26.093670+0000 mon.a (mon. 2021-03-15T22:23:27.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: 0) 987 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:27.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: audit 2021-03-15T22:23:27.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: 2021-03-15T22:23 2021-03-15T22:23:27.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: :26.103597+0000 mon.a ( 2021-03-15T22:23:27.376 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: mon.0) 988 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:23:27.377 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: 2021-03-15T22:23:27.377 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:23:27.377 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: 22:23:26.103677+0000 mon.a ( 2021-03-15T22:23:27.377 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: mon.0) 989 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2021-03-15T22:23:27.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: cluster 2021-03-15T22:23:27.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: 2021-03-15T22:23: 2021-03-15T22:23:27.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:27 smithi076 conmon[62156]: 26.791034+0000 osd.0 (osd.0) 59 : cluster [DBG] 7.f scrub ok 2021-03-15T22:23:27.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:27.389 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:27.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:27.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:27.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:27.516 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:27.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:27.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:27 smithi125 conmon[56739]: cluster 2021-03-15T22:23:25.549756+0000 osd.2 (osd.2) 53 : cluster [DBG] 2021-03-15T22:23:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:27 smithi125 conmon[56739]: purged_snaps scrub starts 2021-03-15T22:23:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:27 smithi125 conmon[56739]: cluster 2021-03-15T22:23:25.549952+0000 osd.2 (osd.2) 54 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:27 smithi125 conmon[56739]: cluster 2021-03-15T22:23 2021-03-15T22:23:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:27 smithi125 conmon[56739]: :26.093670+0000 mon.a (mon.0) 987 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:27.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:27 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:23:27.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:27 smithi125 conmon[56739]: 23:26.103597+0000 mon.a (mon.0) 988 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:23:27.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:27 smithi125 conmon[56739]: cluster 2021-03-15T22:23:26.103677+0000 mon.a ( 2021-03-15T22:23:27.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:27 smithi125 conmon[56739]: mon.0) 989 : cluster [DBG] osdmap e291: 8 total, 8 up, 8 in 2021-03-15T22:23:27.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:27 smithi125 conmon[56739]: cluster 2021-03-15T22:23:26.791034+0000 osd.0 2021-03-15T22:23:27.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:27 smithi125 conmon[56739]: (osd.0) 59 : cluster [DBG] 7.f scrub ok 2021-03-15T22:23:27.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:27.644 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:27.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:27.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:27.701 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:23:27.746 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:27.772 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:27.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:27.774 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:27.875 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:27.899 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:27.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:27.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:28.002 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:28.024 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:28.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:28.026 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:28.116 INFO:tasks.rados.rados.0.smithi076.stdout:983: done (2 left) 2021-03-15T22:23:28.116 INFO:tasks.rados.rados.0.smithi076.stdout:984: done (1 left) 2021-03-15T22:23:28.117 INFO:tasks.rados.rados.0.smithi076.stdout:985: done (0 left) 2021-03-15T22:23:28.117 INFO:tasks.rados.rados.0.smithi076.stdout:986: copy_from oid 37 from oid 31 current snap is 98 2021-03-15T22:23:28.117 INFO:tasks.rados.rados.0.smithi076.stdout:987: read oid 19 snap -1 2021-03-15T22:23:28.117 INFO:tasks.rados.rados.0.smithi076.stdout:987: expect (ObjNum 273 snap 72 seq_num 273) 2021-03-15T22:23:28.117 INFO:tasks.rados.rados.0.smithi076.stdout:988: snap_create 2021-03-15T22:23:28.119 INFO:tasks.rados.rados.0.smithi076.stdout:987: expect (ObjNum 273 snap 72 seq_num 273) 2021-03-15T22:23:28.127 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:28.151 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:28.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:28.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:28.221 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:28 smithi076 conmon[56514]: cluster 2021-03-15T22:23:26.743805+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:23:28.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:28 smithi076 conmon[56514]: cluster 2021-03-15T22:23:26.748479+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.4 scrub ok 2021-03-15T22:23:28.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:28 smithi076 conmon[56514]: cluster 2021-03-15T22:23:27.024215+0000 mgr.y (mgr.14144) 521 : cluster [DBG] pgmap v564: 138 pgs: 138 active+clean; 149 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 12 op/s 2021-03-15T22:23:28.222 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:28 smithi076 conmon[56514]: cluster 2021-03-15T22:23:27.106381+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2021-03-15T22:23:28.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:28 smithi076 conmon[62156]: cluster 2021-03-15T22:23:26.743805+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:23:28.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:28 smithi076 conmon[62156]: cluster 2021-03-15T22:23:26.748479+0000 osd.7 (osd.7) 59 : cluster [DBG] 5.4 scrub ok 2021-03-15T22:23:28.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:28 smithi076 conmon[62156]: cluster 2021-03-15T22:23:27.024215+0000 mgr.y (mgr.14144) 521 : cluster [DBG] pgmap v564: 138 pgs: 138 active+clean; 149 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 12 op/s 2021-03-15T22:23:28.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:28 smithi076 conmon[62156]: cluster 2021-03-15T22:23:27.106381+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2021-03-15T22:23:28.253 INFO:tasks.rados.rados.0.smithi076.stdout:986: finishing copy_from to smithi07694878-37 2021-03-15T22:23:28.253 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 151 (ObjNum 280 snap 74 seq_num 280) dirty exists 2021-03-15T22:23:28.254 INFO:tasks.rados.rados.0.smithi076.stdout:986: finishing copy_from racing read to smithi07694878-37 2021-03-15T22:23:28.254 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:28.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:28.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:28.301 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:28.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:28.409 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:28.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:28.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:28.413 INFO:teuthology.orchestra.run.smithi076.stdout:moved 34 / 414 (8.21256%) 2021-03-15T22:23:28.413 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:23:28.413 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 11.1327 -> 13.2264 (expected baseline 6.72913) 2021-03-15T22:23:28.414 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 25 -> 25 pgs (0.483092 -> 0.483092 * mean) 2021-03-15T22:23:28.414 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 64 -> 67 pgs (1.23671 -> 1.29469 * mean) 2021-03-15T22:23:28.414 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:23:28.415 INFO:teuthology.orchestra.run.smithi076.stdout:oload 110 2021-03-15T22:23:28.416 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 3 2021-03-15T22:23:28.416 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:23:28.416 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0007 2021-03-15T22:23:28.417 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0008 2021-03-15T22:23:28.417 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7 weight 1.0000 -> 0.7996 2021-03-15T22:23:28.418 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 weight 1.0000 -> 0.8168 2021-03-15T22:23:28.418 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0 weight 1.0000 -> 0.8522 2021-03-15T22:23:28.419 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2 weight 1.0000 -> 0.8986 2021-03-15T22:23:28.420 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:23:28.453 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:23:28.470 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:23:28.486 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:28 smithi076 conmon[56844]: debug 2021-03-15T22:23:28.410+0000 7fc67af46700 -1 mgr.server reply reply (4) Interrupted system call no change 2021-03-15T22:23:28.487 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:23:28.505 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:23:28.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:28.528 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:23:28.535 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:28.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:28.536 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:28 smithi125 conmon[56739]: cluster 2021-03-15T22:23:26.743805+0000 osd.4 (osd.4) 65 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:23:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:28 smithi125 conmon[56739]: cluster 2021-03-15T22:23:26.748479+0000 osd.7 (osd.7) 2021-03-15T22:23:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:28 smithi125 conmon[56739]: 59 : cluster [DBG] 5.4 scrub ok 2021-03-15T22:23:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:28 smithi125 conmon[56739]: cluster 2021-03-15T22:23:27.024215+0000 mgr.y (mgr.14144) 521 : cluster [DBG] pgmap v564: 138 pgs: 138 active+clean; 149 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 2.3 MiB/s rd, 3.2 MiB/s wr, 12 op/s 2021-03-15T22:23:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:28 smithi125 conmon[56739]: cluster 2021-03-15T22:23:27.106381+0000 mon.a (mon.0) 990 : cluster [DBG] osdmap e292: 8 total, 8 up, 8 in 2021-03-15T22:23:28.547 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:23:28.566 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:23:28.584 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:23:28.585 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:23:28.586 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:23:28.586 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:23:28.586 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:23:28.586 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:23:28.587 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:23:28.587 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:23:28.587 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:23:28.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:28.663 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:28.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:28.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:28.765 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:28.791 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:28.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:28.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:28.894 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:28.916 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:28.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:28.918 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:29.019 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:29.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:29.047 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:29.048 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:29.127 INFO:tasks.rados.rados.0.smithi076.stdout:986: done (2 left) 2021-03-15T22:23:29.128 INFO:tasks.rados.rados.0.smithi076.stdout:987: done (1 left) 2021-03-15T22:23:29.128 INFO:tasks.rados.rados.0.smithi076.stdout:988: done (0 left) 2021-03-15T22:23:29.128 INFO:tasks.rados.rados.0.smithi076.stdout:989: rollback oid 27 current snap is 99 2021-03-15T22:23:29.128 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 95 2021-03-15T22:23:29.128 INFO:tasks.rados.rados.0.smithi076.stdout:990: snap_remove snap 98 2021-03-15T22:23:29.148 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:29.170 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:29.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:29.171 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:29.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:29.299 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:29.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:29.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:29.346 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[56514]: cluster 2021-03-15T22:23:27.790033+0000 osd.7 (osd.7) 60 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:23:29.346 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[56514]: cluster 2021-03-15T22:23:27.994273+0000 osd.5 (osd.5) 57 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:23:29.347 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[56514]: cluster 2021-03-15T22:23:28.113684+0000 mon.a (mon.0) 991 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2021-03-15T22:23:29.347 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[56514]: audit 2021-03-15T22:23:28.410669+0000 mgr.y (mgr.14144) 522 : audit [DBG] from='client.24757 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:29.347 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[56514]: audit 2021-03-15T22:23:28.433890+0000 mon.a (mon.0) 992 : audit [DBG] from='client.? 172.21.15.76:0/1082512610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:29.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: cluster 2021-03-15T22:23:27.790033+0000 2021-03-15T22:23:29.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: osd.7 (osd.7) 60 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:23:29.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:23:29.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: 22:23:27.994273+0000 osd.5 (osd.5) 57 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:23:29.349 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: cluster 2021-03-15T22:23:28.113684+0000 mon.a 2021-03-15T22:23:29.349 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: (mon.0) 991 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2021-03-15T22:23:29.349 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: audit 2021-03-15T22:23:28. 2021-03-15T22:23:29.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: 410669+0000 mgr.y (mgr.14144) 522 : audit 2021-03-15T22:23:29.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: [DBG] from='client.24757 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:29.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: audit 2021-03-15T22:23: 2021-03-15T22:23:29.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: 28.433890+0000 mon.a (mon.0) 992 : audit 2021-03-15T22:23:29.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:29 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/1082512610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:29.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:29.427 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:29.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:29.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:29.530 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:29 smithi125 conmon[56739]: cluster 2021-03-15T22:23:27.790033+0000 osd.7 ( 2021-03-15T22:23:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:29 smithi125 conmon[56739]: osd.7) 60 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:23:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:29 smithi125 conmon[56739]: cluster 2021-03-15T22:23:27.994273+0000 osd.5 (osd.5) 57 : cluster [DBG] 2021-03-15T22:23:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:29 smithi125 conmon[56739]: 3.2 scrub ok 2021-03-15T22:23:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:29 smithi125 conmon[56739]: cluster 2021-03-15T22:23:28.113684+0000 mon.a (mon 2021-03-15T22:23:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:29 smithi125 conmon[56739]: .0) 991 : cluster [DBG] osdmap e293: 8 total, 8 up, 8 in 2021-03-15T22:23:29.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:29 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:23:29.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:29 smithi125 conmon[56739]: 22:23:28.410669+0000 mgr.y (mgr.14144) 522 : audit [DBG] from='client.24757 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:23:29.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:29 smithi125 conmon[56739]: audit 2021-03-15T22:23:28.433890+0000 mon.a ( 2021-03-15T22:23:29.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:29 smithi125 conmon[56739]: mon.0) 992 : audit [DBG] from='client.? 172.21.15.76:0/1082512610' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:29.553 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:29.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:29.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:29.656 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:29.677 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:29.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:29.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:29.780 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:29.806 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:29.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:29.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:29.899 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:23:29.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:29.933 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:29.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:29.935 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:30.036 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:30.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:23:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:23:30.059 INFO:tasks.rados.rados.0.smithi076.stdout:990: done (1 left) 2021-03-15T22:23:30.059 INFO:tasks.rados.rados.0.smithi076.stdout:991: rollback oid 35 current snap is 99 2021-03-15T22:23:30.060 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 35 to 97 2021-03-15T22:23:30.060 INFO:tasks.rados.rados.0.smithi076.stdout:992: rollback oid 24 current snap is 99 2021-03-15T22:23:30.060 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 97 2021-03-15T22:23:30.060 INFO:tasks.rados.rados.0.smithi076.stdout:993: write oid 25 current snap is 99 2021-03-15T22:23:30.060 INFO:tasks.rados.rados.0.smithi076.stdout:993: seq_num 365 ranges {534555=650829,1659104=475451} 2021-03-15T22:23:30.062 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:30.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:30.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:30.065 INFO:tasks.rados.rados.0.smithi076.stdout:993: writing smithi07694878-25 from 534555 to 1185384 tid 1 2021-03-15T22:23:30.072 INFO:tasks.rados.rados.0.smithi076.stdout:993: writing smithi07694878-25 from 1659104 to 2134555 tid 2 2021-03-15T22:23:30.072 INFO:tasks.rados.rados.0.smithi076.stdout:989: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:23:30.072 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:23:30.072 INFO:tasks.rados.rados.0.smithi076.stdout:992: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:23:30.073 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 0 (ObjNum 262020640 snap 21896 seq_num 261798016) dirty dne 2021-03-15T22:23:30.073 INFO:tasks.rados.rados.0.smithi076.stdout:991: finishing rollback tid 0 to smithi07694878-35 2021-03-15T22:23:30.073 INFO:tasks.rados.rados.0.smithi076.stdout:991: finishing rollback tid 1 to smithi07694878-35 2021-03-15T22:23:30.073 INFO:tasks.rados.rados.0.smithi076.stdout:991: finishing rollback tid 2 to smithi07694878-35 2021-03-15T22:23:30.073 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 372 (ObjNum 6 snap 0 seq_num 6) dirty exists 2021-03-15T22:23:30.074 INFO:tasks.rados.rados.0.smithi076.stdout:989: done (3 left) 2021-03-15T22:23:30.074 INFO:tasks.rados.rados.0.smithi076.stdout:991: done (2 left) 2021-03-15T22:23:30.074 INFO:tasks.rados.rados.0.smithi076.stdout:992: done (1 left) 2021-03-15T22:23:30.074 INFO:tasks.rados.rados.0.smithi076.stdout:994: snap_remove snap 96 2021-03-15T22:23:30.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:30.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:30.189 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:30.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:30.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[56514]: cluster 2021-03-15T22:23:28.808867+0000 osd.7 (osd.7) 2021-03-15T22:23:30.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[56514]: 61 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:23:30.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[56514]: cluster 2021-03-15T22:23:29.014591+0000 osd.1 (osd.1) 63 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:23:30.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[56514]: cluster 2021-03-15T22:23:29.024599+0000 mgr.y (mgr.14144) 523 : cluster [DBG] pgmap v567: 138 pgs: 138 active+clean; 132 MiB data, 500 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 17 op/s 2021-03-15T22:23:30.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[56514]: cluster 2021-03-15T22:23:29.125397+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2021-03-15T22:23:30.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[56514]: cluster 2021-03-15T22:23:29.431916+0000 osd.6 (osd.6) 43 : cluster [DBG] 7.c scrub ok 2021-03-15T22:23:30.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[56514]: audit 2021-03-15T22:23:29.595605+0000 mgr.y (mgr.14144) 524 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:30.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[56514]: cluster 2021-03-15T22:23:30.056052+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2021-03-15T22:23:30.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: cluster 2021-03-15T22:23:28.808867+0000 osd.7 (osd.7) 61 : cluster [DBG] 2021-03-15T22:23:30.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: 6.15 scrub ok 2021-03-15T22:23:30.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: cluster 2021-03-15T22:23:29.014591+0000 osd.1 (osd.1) 63 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:23:30.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: cluster 2021-03-15T22:23:29.024599+0000 mgr.y (mgr.14144) 523 : cluster 2021-03-15T22:23:30.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: [DBG] pgmap v567: 138 pgs: 138 active+clean; 132 MiB data, 500 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 17 op/s 2021-03-15T22:23:30.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: cluster 2021-03-15T22:23:29. 2021-03-15T22:23:30.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: 125397+0000 mon.a (mon.0) 993 : cluster [DBG] osdmap e294: 8 total, 8 up, 8 in 2021-03-15T22:23:30.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:23:30.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: 22:23:29.431916+0000 osd.6 (osd. 2021-03-15T22:23:30.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: 6) 43 : cluster [DBG] 7.c scrub ok 2021-03-15T22:23:30.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:23:30.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: -15T22:23:29.595605+0000 2021-03-15T22:23:30.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: mgr.y (mgr.14144) 524 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:30.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: 2021-03-15T22:23:30.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: cluster 2021-03-15T22:23:30 2021-03-15T22:23:30.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:30 smithi076 conmon[62156]: .056052+0000 mon.a (mon.0) 994 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2021-03-15T22:23:30.290 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:30.317 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:30.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:30.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:30.406 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:30.419 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:30.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:30.449 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:30.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:30.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: cluster 2021-03-15T22:23:28.808867+0000 osd.7 (osd.7) 61 : cluster [DBG] 2021-03-15T22:23:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: 6.15 scrub ok 2021-03-15T22:23:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: cluster 2021-03-15T22:23:29.014591+0000 osd.1 (osd.1) 63 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:23:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: cluster 2021-03-15T22:23:29. 2021-03-15T22:23:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: 024599+0000 mgr.y (mgr.14144) 523 : cluster [DBG] pgmap v567: 138 pgs: 138 active+clean; 132 MiB data, 500 MiB used, 715 GiB / 715 GiB avail; 12 MiB/s rd, 3.0 MiB/s wr, 17 op/s 2021-03-15T22:23:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: cluster 2021-03-15T22:23:29. 2021-03-15T22:23:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: 125397+0000 mon.a (mon.0) 993 : cluster 2021-03-15T22:23:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: [DBG] osdmap e294: 8 total, 8 up, 8 in 2021-03-15T22:23:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:23:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: T22:23:29.431916+0000 osd.6 (osd 2021-03-15T22:23:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: .6) 43 : cluster [DBG] 7.c scrub ok 2021-03-15T22:23:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: audit 2021-03-15T22:23:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: 2021-03-15T22:23:29. 2021-03-15T22:23:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: 595605+0000 mgr.y (mgr.14144) 524 : audit 2021-03-15T22:23:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:23:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: :23:30.056052+0000 mon.a (mon. 2021-03-15T22:23:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:30 smithi125 conmon[56739]: 0) 994 : cluster [DBG] osdmap e295: 8 total, 8 up, 8 in 2021-03-15T22:23:30.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:30.577 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:30.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:30.579 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:30.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:30.705 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:30.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:30.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:30.808 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:30.835 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:30.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:30.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:30.935 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:23:30.935 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:23:30.935 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:23:30.936 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:23:30.936 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:23:30.937 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:30.962 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:30.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:30.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:31.065 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:31.072 INFO:tasks.rados.rados.0.smithi076.stdout:994: done (1 left) 2021-03-15T22:23:31.072 INFO:tasks.rados.rados.0.smithi076.stdout:995: delete oid 28 current snap is 99 2021-03-15T22:23:31.072 INFO:tasks.rados.rados.0.smithi076.stdout:993: finishing write tid 1 to smithi07694878-25 2021-03-15T22:23:31.073 INFO:tasks.rados.rados.0.smithi076.stdout:993: finishing write tid 2 to smithi07694878-25 2021-03-15T22:23:31.073 INFO:tasks.rados.rados.0.smithi076.stdout:993: finishing write tid 3 to smithi07694878-25 2021-03-15T22:23:31.073 INFO:tasks.rados.rados.0.smithi076.stdout:993: finishing write tid 4 to smithi07694878-25 2021-03-15T22:23:31.073 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 152 (ObjNum 365 snap 99 seq_num 365) dirty exists 2021-03-15T22:23:31.073 INFO:tasks.rados.rados.0.smithi076.stdout:993: left oid 25 (ObjNum 365 snap 99 seq_num 365) 2021-03-15T22:23:31.075 INFO:tasks.rados.rados.0.smithi076.stdout:993: done (1 left) 2021-03-15T22:23:31.076 INFO:tasks.rados.rados.0.smithi076.stdout:995: done (0 left) 2021-03-15T22:23:31.076 INFO:tasks.rados.rados.0.smithi076.stdout:996: snap_remove snap 93 2021-03-15T22:23:31.092 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:31.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:31.094 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:31.195 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:31.221 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:31.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:31.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:31.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:31.350 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:31.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:31.351 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:31.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:31.479 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:31.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:31.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:31.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:31 smithi076 conmon[56514]: cluster 2021-03-15T22:23:29.266168+0000 osd.2 (osd.2) 55 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:23:31.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:31 smithi076 conmon[56514]: cluster 2021-03-15T22:23:29.709670+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:23:31.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:31 smithi076 conmon[56514]: cluster 2021-03-15T22:23:29.845433+0000 osd.7 (osd.7) 62 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:23:31.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:31 smithi076 conmon[56514]: cluster 2021-03-15T22:23:30.005785+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:23:31.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:31 smithi076 conmon[56514]: cluster 2021-03-15T22:23:31.067677+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2021-03-15T22:23:31.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:31 smithi076 conmon[62156]: cluster 2021-03-15T22:23:29.266168+0000 osd.2 (osd.2) 55 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:23:31.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:31 smithi076 conmon[62156]: cluster 2021-03-15T22:23:29.709670+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:23:31.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:31 smithi076 conmon[62156]: cluster 2021-03-15T22:23:29.845433+0000 osd.7 (osd.7) 62 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:23:31.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:31 smithi076 conmon[62156]: cluster 2021-03-15T22:23:30.005785+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:23:31.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:31 smithi076 conmon[62156]: cluster 2021-03-15T22:23:31.067677+0000 mon.a (mon.0) 995 : cluster [DBG] osdmap e296: 8 total, 8 up, 8 in 2021-03-15T22:23:31.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: cluster 2021-03-15T22:23:29.266168+0000 osd.2 (osd.2) 55 : cluster 2021-03-15T22:23:31.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: [DBG] 2.1 scrub ok 2021-03-15T22:23:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: cluster 2021-03-15T22:23:29.709670+0000 osd.4 (osd.4) 66 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:23:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: cluster 2021-03-15T22:23 2021-03-15T22:23:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: :29.845433+0000 osd.7 (osd.7) 62 2021-03-15T22:23:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: : cluster [DBG] 6.8 scrub ok 2021-03-15T22:23:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:23:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: 15T22:23:30.005785+0000 osd.5 2021-03-15T22:23:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: (osd.5) 58 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:23:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: cluster 2021-03-15T22:23:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: 2021-03-15T22:23:31.067677 2021-03-15T22:23:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: +0000 mon.a (mon.0) 995 : cluster [DBG] 2021-03-15T22:23:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:31 smithi125 conmon[56739]: osdmap e296: 8 total, 8 up, 8 in 2021-03-15T22:23:31.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:31.605 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:31.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:31.696 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:23:31.707 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:31.761 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:31.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:31.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:31.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:31.890 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:31.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:31.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:31.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:32.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:32.017 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:32.018 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:32.066 INFO:tasks.rados.rados.0.smithi076.stdout:996: done (0 left) 2021-03-15T22:23:32.067 INFO:tasks.rados.rados.0.smithi076.stdout:997: snap_create 2021-03-15T22:23:32.119 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:32.140 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:32.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:32.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:32.243 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:32.269 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:32.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:32.271 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:32.372 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:32.375 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:32.397 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:32.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:32.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:32.406 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:23:32.422 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[56514]: cluster 2021-03-15T22:23:30.831594+0000 2021-03-15T22:23:32.422 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[56514]: osd.7 (osd.7) 63 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:23:32.422 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[56514]: cluster 2021-03-15T22:23:31.003230+0000 osd.1 (osd.1) 64 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:23:32.422 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[56514]: cluster 2021-03-15T22:23:31.024933+0000 mgr.y (mgr.14144) 525 : cluster [DBG] pgmap v570: 138 pgs: 138 active+clean; 131 MiB data, 490 MiB used, 715 GiB / 715 GiB avail; 9.6 MiB/s rd, 487 KiB/s wr, 10 op/s 2021-03-15T22:23:32.423 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[56514]: cluster 2021-03-15T22:23:31.445604+0000 osd.6 (osd.6) 44 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:23:32.423 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[56514]: cluster 2021-03-15T22:23:32.063803+0000 2021-03-15T22:23:32.423 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[56514]: mon.a (mon.0) 996 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2021-03-15T22:23:32.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: cluster 2021-03-15T22:23:30.831594 2021-03-15T22:23:32.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: +0000 osd.7 (osd.7) 63 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:23:32.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: cluster 2021-03-15T22:23:31.003230+0000 2021-03-15T22:23:32.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: osd.1 (osd.1) 64 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:23:32.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: 2021-03-15T22:23:32.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:23:32.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: :23:31.024933+0000 mgr.y (mgr. 2021-03-15T22:23:32.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: 14144) 525 : cluster [DBG] pgmap v570: 138 pgs: 138 active+clean; 131 MiB data, 490 MiB used, 715 GiB / 715 GiB avail; 9.6 MiB/s rd, 487 KiB/s wr, 10 op/s 2021-03-15T22:23:32.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:23:32.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: -03-15T22:23:31.445604+0000 2021-03-15T22:23:32.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: osd.6 (osd.6) 44 : cluster [DBG] 2021-03-15T22:23:32.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: 7.7 scrub ok 2021-03-15T22:23:32.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:23:32.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: 22:23:32.063803+0000 mon.a (mon 2021-03-15T22:23:32.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[62156]: .0) 996 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2021-03-15T22:23:32.428 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[65511]: debug 2021-03-15T22:23:32.417+0000 7f1a1f877700 -1 osd.0 pg_epoch: 297 pg[7.6( v 296'549 (0'0,296'549] local-lis/les=175/176 n=11 ec=91/91 lis/c=175/175 les/c/f=176/176/0 sis=175) [0,7,5] r=0 lpr=175 crt=296'549 lcod 292'547 mlcod 292'547 active+clean+snaptrim trimq=[b9~1] ps=[9c~1,ba~1,c0~1,c2~1]] removing snap head 2021-03-15T22:23:32.499 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:32.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:32.523 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:32.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:32 smithi125 conmon[56739]: cluster 2021-03-15T22:23:30.831594+0000 osd.7 (osd.7) 63 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:23:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:32 smithi125 conmon[56739]: 2021-03-15T22:23:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:32 smithi125 conmon[56739]: cluster 2021-03-15T22:23:31.003230+0000 osd.1 (osd.1) 64 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:23:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:32 smithi125 conmon[56739]: cluster 2021-03-15T22:23:31.024933+0000 mgr.y (mgr 2021-03-15T22:23:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:32 smithi125 conmon[56739]: .14144) 525 : cluster [DBG] pgmap v570: 138 pgs: 138 active+clean; 131 MiB data, 490 MiB used, 715 GiB / 715 GiB avail; 9.6 MiB/s rd, 487 KiB/s wr, 10 op/s 2021-03-15T22:23:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:32 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:23:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:32 smithi125 conmon[56739]: -03-15T22:23:31.445604+0000 osd.6 (osd.6) 44 : cluster 2021-03-15T22:23:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:32 smithi125 conmon[56739]: [DBG] 7.7 scrub ok 2021-03-15T22:23:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:32 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:23:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:32 smithi125 conmon[56739]: T22:23:32.063803+0000 mon.a ( 2021-03-15T22:23:32.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:32 smithi125 conmon[56739]: mon.0) 996 : cluster [DBG] osdmap e297: 8 total, 8 up, 8 in 2021-03-15T22:23:32.624 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:32.648 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:32.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:32.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:32.732 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:23:32 smithi076 conmon[75764]: debug 2021-03-15T22:23:32.420+0000 7f9fbefb7700 -1 osd.3 pg_epoch: 297 pg[7.a( v 296'374 (0'0,296'374] local-lis/les=268/269 n=7 ec=91/91 lis/c=268/268 les/c/f=269/269/0 sis=268) [3,2,5] r=0 lpr=268 crt=296'374 lcod 295'372 mlcod 295'372 active+clean+snaptrim trimq=[b9~1] ps=[9c~1,ba~1,c0~1,c2~1]] removing snap head 2021-03-15T22:23:32.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:32.772 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:32.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:32.773 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:32.874 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:32.901 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:32.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:32.903 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:33.004 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:33.012 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:33.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:33.031 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:33.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:33.071 INFO:tasks.rados.rados.0.smithi076.stdout:997: done (0 left) 2021-03-15T22:23:33.071 INFO:tasks.rados.rados.0.smithi076.stdout:998: delete oid 14 current snap is 100 2021-03-15T22:23:33.073 INFO:tasks.rados.rados.0.smithi076.stdout:998: done (0 left) 2021-03-15T22:23:33.073 INFO:tasks.rados.rados.0.smithi076.stdout:999: delete oid 43 current snap is 100 2021-03-15T22:23:33.078 INFO:tasks.rados.rados.0.smithi076.stdout:999: done (0 left) 2021-03-15T22:23:33.078 INFO:tasks.rados.rados.0.smithi076.stdout:1000: snap_remove snap 99 2021-03-15T22:23:33.133 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:33.157 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:33.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:33.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:33.260 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:33.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:33 smithi076 conmon[56514]: cluster 2021-03-15T22:23:32.049039+0000 osd.1 (osd.1) 65 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:23:33.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:33 smithi076 conmon[56514]: cluster 2021-03-15T22:23:32.414522+0000 osd.6 (osd.6) 45 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:23:33.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:33 smithi076 conmon[56514]: cluster 2021-03-15T22:23:33.067641+0000 mon.a (mon.0) 997 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2021-03-15T22:23:33.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:33 smithi076 conmon[62156]: cluster 2021-03-15T22:23:32.049039+0000 osd.1 2021-03-15T22:23:33.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:33 smithi076 conmon[62156]: (osd.1) 65 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:23:33.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:33 smithi076 conmon[62156]: cluster 2021-03-15T22:23:32.414522+0000 2021-03-15T22:23:33.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:33 smithi076 conmon[62156]: osd.6 (osd.6) 45 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:23:33.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:33 smithi076 conmon[62156]: cluster 2021-03-15T22:23:33.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:33 smithi076 conmon[62156]: 2021-03-15T22:23 2021-03-15T22:23:33.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:33 smithi076 conmon[62156]: :33.067641+0000 mon.a (mon.0 2021-03-15T22:23:33.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:33 smithi076 conmon[62156]: ) 997 : cluster [DBG] osdmap e298: 8 total, 8 up, 8 in 2021-03-15T22:23:33.283 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:33.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:33.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:33.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:33.408 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:33.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:33.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:33.511 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:33.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:33 smithi125 conmon[56739]: cluster 2021-03-15T22:23:32.049039+0000 osd.1 (osd.1) 65 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:23:33.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:33 smithi125 conmon[56739]: cluster 2021-03-15T22:23:32.414522+0000 osd.6 (osd.6) 45 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:23:33.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:33 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:23:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:33 smithi125 conmon[56739]: :23:33.067641+0000 mon.a ( 2021-03-15T22:23:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:33 smithi125 conmon[56739]: mon.0) 997 : cluster [DBG] 2021-03-15T22:23:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:33 smithi125 conmon[56739]: osdmap e298: 8 total, 8 up, 8 in 2021-03-15T22:23:33.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:33.539 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:33.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:33.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:33.666 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:33.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:33.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:33.795 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:33.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:33.797 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:33.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:33.924 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:33.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:33.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:34.027 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:34.051 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:34.053 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:34.073 INFO:tasks.rados.rados.0.smithi076.stdout:1000: done (0 left) 2021-03-15T22:23:34.073 INFO:tasks.rados.rados.0.smithi076.stdout:1001: write (excl) oid 35 current snap is 100 2021-03-15T22:23:34.074 INFO:tasks.rados.rados.0.smithi076.stdout:1001: seq_num 369 ranges {601783=551825,1776679=425104} 2021-03-15T22:23:34.075 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:23:34.078 INFO:tasks.rados.rados.0.smithi076.stdout:1001: writing smithi07694878-35 from 601783 to 1153608 tid 1 2021-03-15T22:23:34.085 INFO:tasks.rados.rados.0.smithi076.stdout:1001: writing smithi07694878-35 from 1776679 to 2201783 tid 2 2021-03-15T22:23:34.085 INFO:tasks.rados.rados.0.smithi076.stdout:1002: snap_create 2021-03-15T22:23:34.086 INFO:tasks.rados.rados.0.smithi076.stdout:1001: finishing write tid 1 to smithi07694878-35 2021-03-15T22:23:34.090 INFO:tasks.rados.rados.0.smithi076.stdout:1001: finishing write tid 2 to smithi07694878-35 2021-03-15T22:23:34.090 INFO:tasks.rados.rados.0.smithi076.stdout:1001: finishing write tid 3 to smithi07694878-35 2021-03-15T22:23:34.091 INFO:tasks.rados.rados.0.smithi076.stdout:1001: finishing write tid 4 to smithi07694878-35 2021-03-15T22:23:34.091 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 381 (ObjNum 369 snap 100 seq_num 369) dirty exists 2021-03-15T22:23:34.091 INFO:tasks.rados.rados.0.smithi076.stdout:1001: left oid 35 (ObjNum 369 snap 100 seq_num 369) 2021-03-15T22:23:34.154 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:34.181 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:34.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:34.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:34.283 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:34.310 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:34.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:34.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:34.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[56514]: cluster 2021-03-15T22:23:32.751652+0000 osd.4 ( 2021-03-15T22:23:34.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[56514]: osd.4) 67 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:23:34.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[56514]: cluster 2021-03-15T22:23:33.025508+0000 mgr.y (mgr.14144) 526 : cluster [DBG] pgmap v573: 138 pgs: 138 active+clean; 137 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2021-03-15T22:23:34.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[56514]: cluster 2021-03-15T22:23:33.120680+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:34.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[56514]: cluster 2021-03-15T22:23:33.121048+0000 osd.6 (osd.6) 47 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:34.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[56514]: audit 2021-03-15T22:23:33.177514+0000 mon.a (mon.0) 998 : audit [INF] from='client.? 172.21.15.76:0/747281392' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:34.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[56514]: cluster 2021-03-15T22:23:33.717511+0000 osd.0 (osd.0) 60 : cluster [DBG] 6.11 scrub ok 2021-03-15T22:23:34.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[56514]: cluster 2021-03-15T22:23:34.062857+0000 mon.a (mon.0) 999 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:23:34.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[56514]: cluster 2021-03-15T22:23:34.062885+0000 mon.a (mon.0) 1000 : cluster [INF] Cluster is now healthy 2021-03-15T22:23:34.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[56514]: audit 2021-03-15T22:23:34.068814+0000 mon.a (mon.0) 1001 : audit [INF] from='client.? 172.21.15.76:0/747281392' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:34.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[56514]: cluster 2021-03-15T22:23:34.068880+0000 mon.a (mon.0) 1002 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2021-03-15T22:23:34.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: cluster 2021-03-15T22:23:32 2021-03-15T22:23:34.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: .751652+0000 osd.4 (osd.4) 67 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:23:34.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: cluster 2021-03-15T22:23:33.025508+0000 mgr.y (mgr.14144) 526 : cluster [DBG] pgmap v573: 138 pgs: 138 active+clean; 137 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2021-03-15T22:23:34.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: cluster 2021-03-15T22:23:33.120680+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:34.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: cluster 2021-03-15T22:23:33.121048+0000 osd.6 (osd.6) 47 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:34.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: audit 2021-03-15T22:23:33.177514+0000 mon.a (mon.0) 998 : audit [INF] from='client.? 172.21.15.76:0/747281392' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:34.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: cluster 2021-03-15T22:23:33.717511+0000 osd.0 (osd.0) 60 : cluster [DBG] 2021-03-15T22:23:34.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: 6.11 scrub ok 2021-03-15T22:23:34.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: cluster 2021-03-15T22:23:34.062857+0000 mon.a (mon.0) 999 : cluster [INF] 2021-03-15T22:23:34.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:23:34.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:23:34.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: 22:23:34.062885+0000 mon.a (mon.0) 1000 2021-03-15T22:23:34.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: : cluster [INF] Cluster is now healthy 2021-03-15T22:23:34.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:23:34.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: 22:23:34.068814+0000 mon.a (mon 2021-03-15T22:23:34.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: .0) 1001 : audit [INF] from='client.? 172.21.15.76:0/747281392' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:34.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:23:34.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: -03-15T22:23:34. 2021-03-15T22:23:34.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: 068880+0000 mon.a (mon.0) 1002 : cluster 2021-03-15T22:23:34.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:34 smithi076 conmon[62156]: [DBG] osdmap e299: 8 total, 8 up, 8 in 2021-03-15T22:23:34.413 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:34.427 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:23:34.428 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:23:34.428 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.000000 2021-03-15T22:23:34.429 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd primary-affinity 7 0 2021-03-15T22:23:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:34.437 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:34.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:34.449 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:23:34.449 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:23:34.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:23:34.450 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:23:34.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.450 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.5.0:63 7.0 e108/91 7:08414d91:::smithi07694878-42:8 v 108'36, mlcod=104'33)", 2021-03-15T22:23:34.450 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:23.006509+0000", 2021-03-15T22:23:34.451 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 191.43837922700001, 2021-03-15T22:23:34.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008624853, 2021-03-15T22:23:34.452 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.452 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.452 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006509+0000", 2021-03-15T22:23:34.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006509+0000", 2021-03-15T22:23:34.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8430000000000003e-06 2021-03-15T22:23:34.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006513+0000", 2021-03-15T22:23:34.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9459999999999998e-06 2021-03-15T22:23:34.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006516+0000", 2021-03-15T22:23:34.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1904e-05 2021-03-15T22:23:34.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006528+0000", 2021-03-15T22:23:34.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6889999999999998e-06 2021-03-15T22:23:34.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006535+0000", 2021-03-15T22:23:34.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1385999999999997e-05 2021-03-15T22:23:34.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006606+0000", 2021-03-15T22:23:34.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4292000000000001e-05 2021-03-15T22:23:34.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.006630+0000", 2021-03-15T22:23:34.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0084716670000000004 2021-03-15T22:23:34.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.015102+0000", 2021-03-15T22:23:34.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2125999999999997e-05 2021-03-15T22:23:34.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:23.015134+0000", 2021-03-15T22:23:34.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008624853 2021-03-15T22:23:34.465 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.465 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.466 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.466 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:539 7.2 e123/91 7:45ee0f2f:::smithi07694878-6:head v 123'14, mlcod=104'10)", 2021-03-15T22:23:34.466 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:38.116343+0000", 2021-03-15T22:23:34.467 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 176.32854519899999, 2021-03-15T22:23:34.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01064652, 2021-03-15T22:23:34.467 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.467 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.468 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.116343+0000", 2021-03-15T22:23:34.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.116343+0000", 2021-03-15T22:23:34.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017858069999999999 2021-03-15T22:23:34.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118129+0000", 2021-03-15T22:23:34.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5420000000000002e-06 2021-03-15T22:23:34.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118136+0000", 2021-03-15T22:23:34.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9451999999999999e-05 2021-03-15T22:23:34.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118165+0000", 2021-03-15T22:23:34.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3950000000000007e-06 2021-03-15T22:23:34.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118174+0000", 2021-03-15T22:23:34.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5652000000000002e-05 2021-03-15T22:23:34.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118230+0000", 2021-03-15T22:23:34.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1316999999999999e-05 2021-03-15T22:23:34.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118251+0000", 2021-03-15T22:23:34.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086919890000000007 2021-03-15T22:23:34.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.126943+0000", 2021-03-15T22:23:34.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6366000000000003e-05 2021-03-15T22:23:34.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.126990+0000", 2021-03-15T22:23:34.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01064652 2021-03-15T22:23:34.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.480 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.480 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.481 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.0.0:444 7.4 e173/91 7:24f33681:::smithi07694878-33:44 v 173'122, mlcod=171'116)", 2021-03-15T22:23:34.481 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:28.038436+0000", 2021-03-15T22:23:34.481 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 126.40645237, 2021-03-15T22:23:34.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014810036, 2021-03-15T22:23:34.482 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.482 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.482 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038436+0000", 2021-03-15T22:23:34.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038436+0000", 2021-03-15T22:23:34.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3089999999999998e-06 2021-03-15T22:23:34.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038438+0000", 2021-03-15T22:23:34.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.934e-06 2021-03-15T22:23:34.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038440+0000", 2021-03-15T22:23:34.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.81e-06 2021-03-15T22:23:34.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038447+0000", 2021-03-15T22:23:34.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4270000000000001e-06 2021-03-15T22:23:34.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.038452+0000", 2021-03-15T22:23:34.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0058912499999999998 2021-03-15T22:23:34.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.044343+0000", 2021-03-15T22:23:34.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0003000000000001e-05 2021-03-15T22:23:34.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.044363+0000", 2021-03-15T22:23:34.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088552849999999992 2021-03-15T22:23:34.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.053218+0000", 2021-03-15T22:23:34.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8017999999999998e-05 2021-03-15T22:23:34.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.053246+0000", 2021-03-15T22:23:34.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014810036 2021-03-15T22:23:34.495 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.496 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.496 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.497 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.5.0:201 7.0 e173/91 7:00c03349:::smithi07694878-40:44 v 173'256, mlcod=171'252)", 2021-03-15T22:23:34.497 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:28.043898+0000", 2021-03-15T22:23:34.497 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 126.40099085200001, 2021-03-15T22:23:34.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008587819, 2021-03-15T22:23:34.498 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.498 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.499 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043898+0000", 2021-03-15T22:23:34.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043898+0000", 2021-03-15T22:23:34.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2150000000000001e-06 2021-03-15T22:23:34.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043901+0000", 2021-03-15T22:23:34.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.542e-06 2021-03-15T22:23:34.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043903+0000", 2021-03-15T22:23:34.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8909999999999995e-06 2021-03-15T22:23:34.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043911+0000", 2021-03-15T22:23:34.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1120000000000004e-06 2021-03-15T22:23:34.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043916+0000", 2021-03-15T22:23:34.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138269 2021-03-15T22:23:34.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.044055+0000", 2021-03-15T22:23:34.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0902000000000001e-05 2021-03-15T22:23:34.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.044075+0000", 2021-03-15T22:23:34.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083778680000000001 2021-03-15T22:23:34.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.052453+0000", 2021-03-15T22:23:34.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2020000000000002e-05 2021-03-15T22:23:34.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.052485+0000", 2021-03-15T22:23:34.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008587819 2021-03-15T22:23:34.513 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.513 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.514 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.514 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.5.0:202 7.0 e173/91 7:0cc37e3d:::smithi07694878-21:44 v 173'258, mlcod=171'252)", 2021-03-15T22:23:34.515 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:28.043926+0000", 2021-03-15T22:23:34.515 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 126.400962599, 2021-03-15T22:23:34.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085792720000000006, 2021-03-15T22:23:34.515 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.516 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.516 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043926+0000", 2021-03-15T22:23:34.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043926+0000", 2021-03-15T22:23:34.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7020000000000001e-06 2021-03-15T22:23:34.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043933+0000", 2021-03-15T22:23:34.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1050000000000002e-06 2021-03-15T22:23:34.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043936+0000", 2021-03-15T22:23:34.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2140000000000001e-06 2021-03-15T22:23:34.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043942+0000", 2021-03-15T22:23:34.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8210000000000003e-06 2021-03-15T22:23:34.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.043946+0000", 2021-03-15T22:23:34.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024850649999999998 2021-03-15T22:23:34.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.046431+0000", 2021-03-15T22:23:34.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9768000000000001e-05 2021-03-15T22:23:34.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.046450+0000", 2021-03-15T22:23:34.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060373780000000004 2021-03-15T22:23:34.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.052488+0000", 2021-03-15T22:23:34.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7218999999999999e-05 2021-03-15T22:23:34.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:28.052505+0000", 2021-03-15T22:23:34.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085792720000000006 2021-03-15T22:23:34.527 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.528 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.528 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.528 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:1215 7.7 e179/172 7:ed4a01d2:::smithi07694878-2:head v 179'52, mlcod=173'49)", 2021-03-15T22:23:34.529 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:34.045906+0000", 2021-03-15T22:23:34.529 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 120.398981948, 2021-03-15T22:23:34.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013063455999999999, 2021-03-15T22:23:34.529 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.530 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.530 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.045906+0000", 2021-03-15T22:23:34.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.045906+0000", 2021-03-15T22:23:34.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056384349999999998 2021-03-15T22:23:34.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051545+0000", 2021-03-15T22:23:34.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5680000000000001e-06 2021-03-15T22:23:34.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051549+0000", 2021-03-15T22:23:34.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6931000000000001e-05 2021-03-15T22:23:34.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.535 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051566+0000", 2021-03-15T22:23:34.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1999999999999999e-06 2021-03-15T22:23:34.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051573+0000", 2021-03-15T22:23:34.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8409999999999998e-05 2021-03-15T22:23:34.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051631+0000", 2021-03-15T22:23:34.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7115999999999999e-05 2021-03-15T22:23:34.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.051658+0000", 2021-03-15T22:23:34.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072775619999999996 2021-03-15T22:23:34.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.058936+0000", 2021-03-15T22:23:34.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4233999999999998e-05 2021-03-15T22:23:34.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:34.058970+0000", 2021-03-15T22:23:34.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013063455999999999 2021-03-15T22:23:34.541 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.541 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.541 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.542 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDScrubReserve(6.1b REQUEST e194)", 2021-03-15T22:23:34.542 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:23:34.542 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 104.661796694, 2021-03-15T22:23:34.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.061386018000000001, 2021-03-15T22:23:34.543 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.543 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:34.543 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:23:34.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:23:34.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6899999999999999e-06 2021-03-15T22:23:34.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783093+0000", 2021-03-15T22:23:34.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5640000000000001e-06 2021-03-15T22:23:34.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783097+0000", 2021-03-15T22:23:34.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7742e-05 2021-03-15T22:23:34.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783125+0000", 2021-03-15T22:23:34.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3793e-05 2021-03-15T22:23:34.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783138+0000", 2021-03-15T22:23:34.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8529999999999993e-05 2021-03-15T22:23:34.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783237+0000", 2021-03-15T22:23:34.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.241e-05 2021-03-15T22:23:34.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:23:34.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783249+0000", 2021-03-15T22:23:34.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013943790000000001 2021-03-15T22:23:34.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.553 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.784644+0000", 2021-03-15T22:23:34.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2398000000000001e-05 2021-03-15T22:23:34.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.784656+0000", 2021-03-15T22:23:34.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.059821512 2021-03-15T22:23:34.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.844478+0000", 2021-03-15T22:23:34.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.061386018000000001 2021-03-15T22:23:34.556 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.556 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.556 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.557 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.6.0:214 7.d e205/91 7:bd183e1b:::smithi07694878-46:68 v 205'30, mlcod=202'27)", 2021-03-15T22:23:34.557 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:23:34.557 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 94.355773678000006, 2021-03-15T22:23:34.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036819649000000003, 2021-03-15T22:23:34.558 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.558 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.558 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:23:34.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:23:34.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2879999999999999e-06 2021-03-15T22:23:34.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089117+0000", 2021-03-15T22:23:34.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4639e-05 2021-03-15T22:23:34.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089132+0000", 2021-03-15T22:23:34.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0759e-05 2021-03-15T22:23:34.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.563 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.563 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089142+0000", 2021-03-15T22:23:34.564 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8793000000000001e-05 2021-03-15T22:23:34.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.564 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.564 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089161+0000", 2021-03-15T22:23:34.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3105999999999998e-05 2021-03-15T22:23:34.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089194+0000", 2021-03-15T22:23:34.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3468e-05 2021-03-15T22:23:34.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.566 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.567 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089208+0000", 2021-03-15T22:23:34.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036675487 2021-03-15T22:23:34.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125883+0000", 2021-03-15T22:23:34.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1109000000000001e-05 2021-03-15T22:23:34.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.569 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125934+0000", 2021-03-15T22:23:34.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036819649000000003 2021-03-15T22:23:34.569 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.570 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.570 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.571 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:1584 7.9 e206/91 7:91d654eb:::smithi07694878-5:head v 206'71, mlcod=200'68)", 2021-03-15T22:23:34.571 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:01.039948+0000", 2021-03-15T22:23:34.571 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 93.404940667000005, 2021-03-15T22:23:34.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010112703000000001, 2021-03-15T22:23:34.571 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.572 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.572 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.039948+0000", 2021-03-15T22:23:34.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.039948+0000", 2021-03-15T22:23:34.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029256239999999999 2021-03-15T22:23:34.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.042873+0000", 2021-03-15T22:23:34.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1178999999999999e-05 2021-03-15T22:23:34.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.576 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.042885+0000", 2021-03-15T22:23:34.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3067e-05 2021-03-15T22:23:34.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.577 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.042918+0000", 2021-03-15T22:23:34.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1666e-05 2021-03-15T22:23:34.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.578 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.578 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.042929+0000", 2021-03-15T22:23:34.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8371e-05 2021-03-15T22:23:34.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.042998+0000", 2021-03-15T22:23:34.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8467999999999999e-05 2021-03-15T22:23:34.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.043026+0000", 2021-03-15T22:23:34.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069885019999999997 2021-03-15T22:23:34.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.581 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.581 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.050015+0000", 2021-03-15T22:23:34.582 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5825999999999998e-05 2021-03-15T22:23:34.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.582 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.582 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.582 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:01.050060+0000", 2021-03-15T22:23:34.583 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010112703000000001 2021-03-15T22:23:34.583 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.583 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.583 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.583 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.584 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2241 7.a 7:53dee572:::smithi07694878-35:88 [sparse-read 0~2100842] snapc 0=[] ondisk+read+known_if_redirected e264)", 2021-03-15T22:23:34.584 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.049191+0000", 2021-03-15T22:23:34.584 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.395697497999997, 2021-03-15T22:23:34.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091979829999999999, 2021-03-15T22:23:34.585 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.585 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:34.585 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:34.585 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:34.586 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:34.586 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2241 2021-03-15T22:23:34.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.586 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.587 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.049191+0000", 2021-03-15T22:23:34.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.049191+0000", 2021-03-15T22:23:34.588 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.429e-06 2021-03-15T22:23:34.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.589 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.589 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.049192+0000", 2021-03-15T22:23:34.589 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.88e-06 2021-03-15T22:23:34.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.590 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.590 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.049194+0000", 2021-03-15T22:23:34.591 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1100999999999999e-05 2021-03-15T22:23:34.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.591 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.591 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.049205+0000", 2021-03-15T22:23:34.592 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.356e-06 2021-03-15T22:23:34.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.049211+0000", 2021-03-15T22:23:34.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7328e-05 2021-03-15T22:23:34.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.049238+0000", 2021-03-15T22:23:34.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3200000000000006e-06 2021-03-15T22:23:34.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:23:34.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.049247+0000", 2021-03-15T22:23:34.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087037780000000006 2021-03-15T22:23:34.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.057951+0000", 2021-03-15T22:23:34.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015570499999999999 2021-03-15T22:23:34.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.058107+0000", 2021-03-15T22:23:34.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028208600000000002 2021-03-15T22:23:34.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.058389+0000", 2021-03-15T22:23:34.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091979829999999999 2021-03-15T22:23:34.598 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.599 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.599 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.600 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2278 7.b 7:d10ba14d:::smithi07694878-43:head [copy-from ver 255] snapc a6=[a6,a2,a0,9e,9c,93,8d,88,7d] ondisk+write+known_if_redirected e264)", 2021-03-15T22:23:34.600 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.584545+0000", 2021-03-15T22:23:34.600 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 34.860343319000002, 2021-03-15T22:23:34.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011979576, 2021-03-15T22:23:34.600 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.601 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.601 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:34.601 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:34.601 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:34.601 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2278 2021-03-15T22:23:34.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.602 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584545+0000", 2021-03-15T22:23:34.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584545+0000", 2021-03-15T22:23:34.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4460000000000007e-06 2021-03-15T22:23:34.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584555+0000", 2021-03-15T22:23:34.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9109999999999998e-06 2021-03-15T22:23:34.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584560+0000", 2021-03-15T22:23:34.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0239e-05 2021-03-15T22:23:34.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584591+0000", 2021-03-15T22:23:34.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2563e-05 2021-03-15T22:23:34.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.608 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584603+0000", 2021-03-15T22:23:34.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9865999999999996e-05 2021-03-15T22:23:34.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584703+0000", 2021-03-15T22:23:34.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108904 2021-03-15T22:23:34.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584812+0000", 2021-03-15T22:23:34.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0049508219999999997 2021-03-15T22:23:34.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.611 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:23:34.611 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.589763+0000", 2021-03-15T22:23:34.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002915118 2021-03-15T22:23:34.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:23:34.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.592678+0000", 2021-03-15T22:23:34.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003564967 2021-03-15T22:23:34.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:34.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.596243+0000", 2021-03-15T22:23:34.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012299899999999999 2021-03-15T22:23:34.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:34.615 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.596366+0000", 2021-03-15T22:23:34.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3763000000000005e-05 2021-03-15T22:23:34.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.616 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.616 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.596440+0000", 2021-03-15T22:23:34.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4977999999999999e-05 2021-03-15T22:23:34.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.596525+0000", 2021-03-15T22:23:34.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011979576 2021-03-15T22:23:34.617 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.618 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.618 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.618 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2279 7.b 7:d10ba14d:::smithi07694878-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e264)", 2021-03-15T22:23:34.619 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.584621+0000", 2021-03-15T22:23:34.619 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 34.860267190999998, 2021-03-15T22:23:34.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012146788, 2021-03-15T22:23:34.619 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.620 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:34.620 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:34.620 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:34.620 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:34.620 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2279 2021-03-15T22:23:34.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.621 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.621 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.622 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584621+0000", 2021-03-15T22:23:34.622 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584621+0000", 2021-03-15T22:23:34.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.139e-06 2021-03-15T22:23:34.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.624 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584622+0000", 2021-03-15T22:23:34.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0824e-05 2021-03-15T22:23:34.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584633+0000", 2021-03-15T22:23:34.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1335000000000001e-05 2021-03-15T22:23:34.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584645+0000", 2021-03-15T22:23:34.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0390000000000004e-06 2021-03-15T22:23:34.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584650+0000", 2021-03-15T22:23:34.627 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00032992699999999999 2021-03-15T22:23:34.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.628 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.628 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.584979+0000", 2021-03-15T22:23:34.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2979000000000003e-05 2021-03-15T22:23:34.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:23:34.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.585022+0000", 2021-03-15T22:23:34.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070916690000000001 2021-03-15T22:23:34.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.592114+0000", 2021-03-15T22:23:34.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7596999999999994e-05 2021-03-15T22:23:34.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:23:34.632 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.592212+0000", 2021-03-15T22:23:34.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043513989999999997 2021-03-15T22:23:34.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.633 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.633 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.596563+0000", 2021-03-15T22:23:34.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7328000000000001e-05 2021-03-15T22:23:34.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.596660+0000", 2021-03-15T22:23:34.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107552 2021-03-15T22:23:34.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.596768+0000", 2021-03-15T22:23:34.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012146788 2021-03-15T22:23:34.635 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.636 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.636 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.637 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2312 7.a 7:5e93ebe0:::smithi07694878-19:head [sparse-read 0~2405049] snapc 0=[] ondisk+read+known_if_redirected e266)", 2021-03-15T22:23:34.637 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.061143+0000", 2021-03-15T22:23:34.637 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.383745144999999, 2021-03-15T22:23:34.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085684760000000002, 2021-03-15T22:23:34.637 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.638 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:34.638 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:34.638 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:34.638 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:34.639 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2312 2021-03-15T22:23:34.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.639 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.061143+0000", 2021-03-15T22:23:34.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.641 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.641 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.061143+0000", 2021-03-15T22:23:34.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.31e-06 2021-03-15T22:23:34.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.061145+0000", 2021-03-15T22:23:34.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.536e-06 2021-03-15T22:23:34.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.643 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.061146+0000", 2021-03-15T22:23:34.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3690000000000003e-06 2021-03-15T22:23:34.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.061152+0000", 2021-03-15T22:23:34.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8010000000000001e-06 2021-03-15T22:23:34.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.645 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.061156+0000", 2021-03-15T22:23:34.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0074720639999999996 2021-03-15T22:23:34.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.068628+0000", 2021-03-15T22:23:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000136198 2021-03-15T22:23:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.068765+0000", 2021-03-15T22:23:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00094719800000000001 2021-03-15T22:23:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.069712+0000", 2021-03-15T22:23:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085684760000000002 2021-03-15T22:23:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.649 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.0.0:708 7.4 e266/91 7:26337d05:::smithi07694878-20:95 v 266'191, mlcod=261'185)", 2021-03-15T22:23:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.063309+0000", 2021-03-15T22:23:34.650 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.381579021999997, 2021-03-15T22:23:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016260241000000002, 2021-03-15T22:23:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063309+0000", 2021-03-15T22:23:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063309+0000", 2021-03-15T22:23:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3489999999999999e-06 2021-03-15T22:23:34.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063315+0000", 2021-03-15T22:23:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.779e-06 2021-03-15T22:23:34.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063316+0000", 2021-03-15T22:23:34.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7019999999999998e-06 2021-03-15T22:23:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063322+0000", 2021-03-15T22:23:34.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0029999999999996e-06 2021-03-15T22:23:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063326+0000", 2021-03-15T22:23:34.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028956799999999998 2021-03-15T22:23:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063616+0000", 2021-03-15T22:23:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6943e-05 2021-03-15T22:23:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063633+0000", 2021-03-15T22:23:34.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0158838 2021-03-15T22:23:34.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.079517+0000", 2021-03-15T22:23:34.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3096999999999997e-05 2021-03-15T22:23:34.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.079570+0000", 2021-03-15T22:23:34.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016260241000000002 2021-03-15T22:23:34.662 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.663 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.663 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.663 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.0.0:709 7.9 e266/91 7:91d654eb:::smithi07694878-5:97 v 266'89, mlcod=259'85)", 2021-03-15T22:23:34.664 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.064203+0000", 2021-03-15T22:23:34.664 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.380684975000001, 2021-03-15T22:23:34.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017427448000000002, 2021-03-15T22:23:34.664 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.665 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.665 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064203+0000", 2021-03-15T22:23:34.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064203+0000", 2021-03-15T22:23:34.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.922e-06 2021-03-15T22:23:34.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064210+0000", 2021-03-15T22:23:34.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.678e-06 2021-03-15T22:23:34.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064213+0000", 2021-03-15T22:23:34.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0438000000000001e-05 2021-03-15T22:23:34.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064223+0000", 2021-03-15T22:23:34.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5829999999999997e-06 2021-03-15T22:23:34.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064229+0000", 2021-03-15T22:23:34.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054566599999999999 2021-03-15T22:23:34.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064775+0000", 2021-03-15T22:23:34.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2643e-05 2021-03-15T22:23:34.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064797+0000", 2021-03-15T22:23:34.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016787641999999998 2021-03-15T22:23:34.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.081585+0000", 2021-03-15T22:23:34.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5875999999999999e-05 2021-03-15T22:23:34.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.081631+0000", 2021-03-15T22:23:34.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017427448000000002 2021-03-15T22:23:34.676 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.676 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.677 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.677 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.1.0:233 7.b e266/91 ondisk, result = 0)", 2021-03-15T22:23:34.677 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.065176+0000", 2021-03-15T22:23:34.678 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.379712515999998, 2021-03-15T22:23:34.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014869920999999999, 2021-03-15T22:23:34.678 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.678 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:34.679 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065176+0000", 2021-03-15T22:23:34.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065176+0000", 2021-03-15T22:23:34.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5919999999999999e-06 2021-03-15T22:23:34.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065178+0000", 2021-03-15T22:23:34.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.519e-06 2021-03-15T22:23:34.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065184+0000", 2021-03-15T22:23:34.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5899999999999997e-06 2021-03-15T22:23:34.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065194+0000", 2021-03-15T22:23:34.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6010000000000002e-06 2021-03-15T22:23:34.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065199+0000", 2021-03-15T22:23:34.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014821381999999999 2021-03-15T22:23:34.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080021+0000", 2021-03-15T22:23:34.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0757000000000001e-05 2021-03-15T22:23:34.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080031+0000", 2021-03-15T22:23:34.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.448e-05 2021-03-15T22:23:34.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080046+0000", 2021-03-15T22:23:34.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014869920999999999 2021-03-15T22:23:34.689 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.689 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.689 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.690 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.1.0:234 7.b e266/91 ondisk, result = 0)", 2021-03-15T22:23:34.690 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.067434+0000", 2021-03-15T22:23:34.690 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.377454102999998, 2021-03-15T22:23:34.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012746100999999999, 2021-03-15T22:23:34.691 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.691 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:34.691 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067434+0000", 2021-03-15T22:23:34.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067434+0000", 2021-03-15T22:23:34.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7949999999999999e-06 2021-03-15T22:23:34.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067437+0000", 2021-03-15T22:23:34.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3489999999999999e-06 2021-03-15T22:23:34.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067439+0000", 2021-03-15T22:23:34.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1042e-05 2021-03-15T22:23:34.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067450+0000", 2021-03-15T22:23:34.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4003e-05 2021-03-15T22:23:34.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067464+0000", 2021-03-15T22:23:34.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012700516 2021-03-15T22:23:34.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080165+0000", 2021-03-15T22:23:34.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0770000000000002e-06 2021-03-15T22:23:34.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080172+0000", 2021-03-15T22:23:34.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3189999999999995e-06 2021-03-15T22:23:34.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080180+0000", 2021-03-15T22:23:34.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012746100999999999 2021-03-15T22:23:34.701 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.701 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.702 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.702 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.3.0:404 7.2 e266/264 7:42f18a95:::smithi07694878-45:89 v 266'124, mlcod=259'120)", 2021-03-15T22:23:34.702 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.067481+0000", 2021-03-15T22:23:34.703 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.377407245999997, 2021-03-15T22:23:34.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01461748, 2021-03-15T22:23:34.703 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.703 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:34.704 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067481+0000", 2021-03-15T22:23:34.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067481+0000", 2021-03-15T22:23:34.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1746000000000001e-05 2021-03-15T22:23:34.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067493+0000", 2021-03-15T22:23:34.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2649999999999999e-06 2021-03-15T22:23:34.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067495+0000", 2021-03-15T22:23:34.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8780000000000004e-06 2021-03-15T22:23:34.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067504+0000", 2021-03-15T22:23:34.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6289999999999998e-06 2021-03-15T22:23:34.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067510+0000", 2021-03-15T22:23:34.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00047436700000000001 2021-03-15T22:23:34.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.067984+0000", 2021-03-15T22:23:34.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.912e-05 2021-03-15T22:23:34.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.068003+0000", 2021-03-15T22:23:34.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014025299999999999 2021-03-15T22:23:34.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:34.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.082028+0000", 2021-03-15T22:23:34.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0174999999999998e-05 2021-03-15T22:23:34.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.082099+0000", 2021-03-15T22:23:34.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01461748 2021-03-15T22:23:34.715 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.715 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.715 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.716 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.1.0:233 7.b e266/91 ondisk, result = 0)", 2021-03-15T22:23:34.716 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.070346+0000", 2021-03-15T22:23:34.716 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.374541929000003, 2021-03-15T22:23:34.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098729179999999996, 2021-03-15T22:23:34.717 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.717 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:34.717 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070346+0000", 2021-03-15T22:23:34.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070346+0000", 2021-03-15T22:23:34.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9640000000000002e-06 2021-03-15T22:23:34.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070348+0000", 2021-03-15T22:23:34.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8439999999999998e-06 2021-03-15T22:23:34.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070351+0000", 2021-03-15T22:23:34.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2099999999999999e-06 2021-03-15T22:23:34.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070360+0000", 2021-03-15T22:23:34.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1360000000000002e-06 2021-03-15T22:23:34.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070367+0000", 2021-03-15T22:23:34.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098382820000000003 2021-03-15T22:23:34.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080205+0000", 2021-03-15T22:23:34.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6599999999999998e-06 2021-03-15T22:23:34.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080212+0000", 2021-03-15T22:23:34.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8220000000000007e-06 2021-03-15T22:23:34.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080219+0000", 2021-03-15T22:23:34.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098729179999999996 2021-03-15T22:23:34.727 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.727 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.727 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.728 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.1.0:234 7.b e266/91 ondisk, result = 0)", 2021-03-15T22:23:34.728 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.070372+0000", 2021-03-15T22:23:34.729 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.374516069000002, 2021-03-15T22:23:34.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099714539999999994, 2021-03-15T22:23:34.729 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:34.729 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:34.729 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:34.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:34.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070372+0000", 2021-03-15T22:23:34.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:34.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:34.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070372+0000", 2021-03-15T22:23:34.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0070000000000001e-06 2021-03-15T22:23:34.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:34.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070373+0000", 2021-03-15T22:23:34.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6099999999999999e-07 2021-03-15T22:23:34.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:34.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070374+0000", 2021-03-15T22:23:34.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4489999999999998e-06 2021-03-15T22:23:34.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:34.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070380+0000", 2021-03-15T22:23:34.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5300000000000001e-06 2021-03-15T22:23:34.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:34.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.070383+0000", 2021-03-15T22:23:34.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099357200000000003 2021-03-15T22:23:34.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:34.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080319+0000", 2021-03-15T22:23:34.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4209999999999995e-06 2021-03-15T22:23:34.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:34.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080327+0000", 2021-03-15T22:23:34.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6365999999999999e-05 2021-03-15T22:23:34.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:34.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:34.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:34.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080344+0000", 2021-03-15T22:23:34.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099714539999999994 2021-03-15T22:23:34.739 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.739 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.740 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.740 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:34.740 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:34.740 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:23:34.741 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:34.742 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: cluster 2021-03-15T22:23:32.751652+0000 osd.4 (osd.4) 67 : cluster [DBG] 2021-03-15T22:23:34.742 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: 6.2 scrub ok 2021-03-15T22:23:34.742 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: cluster 2021-03-15T22:23:33.025508+0000 mgr.y (mgr.14144) 526 : cluster [DBG] pgmap v573: 138 pgs: 138 active+clean; 137 MiB data, 496 MiB used, 715 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.7 MiB/s wr, 7 op/s 2021-03-15T22:23:34.742 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: cluster 2021-03-15T22:23:33. 2021-03-15T22:23:34.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: 120680+0000 osd.6 (osd.6) 46 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:34.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:23:34.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: T22:23:33.121048+0000 osd.6 (osd.6) 47 : cluster [DBG] 2021-03-15T22:23:34.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: purged_snaps scrub ok 2021-03-15T22:23:34.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: audit 2021 2021-03-15T22:23:34.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: -03-15T22:23:33.177514+0000 mon.a ( 2021-03-15T22:23:34.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: mon.0) 998 : audit [INF] from='client.? 172.21.15.76:0/747281392' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:34.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: cluster 2021-03-15T22:23:34.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: 2021-03-15T22:23:33.717511+0000 2021-03-15T22:23:34.745 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: osd.0 (osd.0) 60 : cluster [DBG] 6.11 scrub ok 2021-03-15T22:23:34.745 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:23:34.745 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: T22:23:34.062857+0000 mon.a (mon.0) 999 : cluster 2021-03-15T22:23:34.745 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:23:34.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:23:34.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: T22:23:34.062885+0000 mon.a (mon.0) 2021-03-15T22:23:34.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: 1000 : cluster [INF] Cluster is now healthy 2021-03-15T22:23:34.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: audit 2021-03-15T22:23:34.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: 2021-03-15T 2021-03-15T22:23:34.747 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: 22:23:34.068814+0000 mon.a (mon.0 2021-03-15T22:23:34.747 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: ) 1001 : audit [INF] from='client.? 172.21.15.76:0/747281392' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:34.747 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: cluster 2021-03-15T22:23:34.747 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: 2021-03-15T22:23:34.068880+0000 2021-03-15T22:23:34.748 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:34 smithi125 conmon[56739]: mon.a (mon.0) 1002 : cluster [DBG] osdmap e299: 8 total, 8 up, 8 in 2021-03-15T22:23:34.769 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:34.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:34.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:34.872 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:34.893 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:34.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:34.996 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:35.021 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:35.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:35.023 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:35.061 INFO:tasks.rados.rados.0.smithi076.stdout:1001: done (1 left) 2021-03-15T22:23:35.062 INFO:tasks.rados.rados.0.smithi076.stdout:1002: done (0 left) 2021-03-15T22:23:35.062 INFO:tasks.rados.rados.0.smithi076.stdout:1003: snap_create 2021-03-15T22:23:35.124 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:35.146 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:35.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:35.148 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:35.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:35.275 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:35.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:35.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:35.378 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:35.403 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:35.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:35.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:35.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:35.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:35 smithi076 conmon[56514]: cluster 2021-03-15T22:23:33.728924+0000 osd.4 (osd.4) 68 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:23:35.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:35 smithi076 conmon[56514]: cluster 2021-03-15T22:23:35.058299+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2021-03-15T22:23:35.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:35 smithi076 conmon[62156]: cluster 2021-03-15T22:23:33.728924+0000 osd.4 (osd.4) 68 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:23:35.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:35 smithi076 conmon[62156]: cluster 2021-03-15T22:23:35.058299+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2021-03-15T22:23:35.529 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:35.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:35.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:35.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:35 smithi125 conmon[56739]: cluster 2021-03-15T22:23:33.728924+0000 osd.4 (osd.4) 68 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:23:35.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:35 smithi125 conmon[56739]: cluster 2021-03-15T22:23:35.058299+0000 mon.a (mon.0) 1003 : cluster [DBG] osdmap e300: 8 total, 8 up, 8 in 2021-03-15T22:23:35.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-15T22:23:35.631 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:35.658 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:35.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:35.659 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:35.760 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:35.787 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:35.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:35.789 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:35.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:35.911 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:35.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:35.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:36.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:36.041 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:36.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:36.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:36.070 INFO:tasks.rados.rados.0.smithi076.stdout:1003: done (0 left) 2021-03-15T22:23:36.070 INFO:tasks.rados.rados.0.smithi076.stdout:1004: write oid 41 current snap is 102 2021-03-15T22:23:36.070 INFO:tasks.rados.rados.0.smithi076.stdout:1004: seq_num 370 ranges {618590=627074,1835161=383429} 2021-03-15T22:23:36.077 INFO:tasks.rados.rados.0.smithi076.stdout:1004: writing smithi07694878-41 from 618590 to 1245664 tid 1 2021-03-15T22:23:36.082 INFO:tasks.rados.rados.0.smithi076.stdout:1004: writing smithi07694878-41 from 1835161 to 2218590 tid 2 2021-03-15T22:23:36.083 INFO:tasks.rados.rados.0.smithi076.stdout:1005: write (excl) oid 32 current snap is 102 2021-03-15T22:23:36.083 INFO:tasks.rados.rados.0.smithi076.stdout:1005: seq_num 371 ranges {635397=702323,1893643=341754} 2021-03-15T22:23:36.090 INFO:tasks.rados.rados.0.smithi076.stdout:1005: writing smithi07694878-32 from 635397 to 1337720 tid 1 2021-03-15T22:23:36.095 INFO:tasks.rados.rados.0.smithi076.stdout:1005: writing smithi07694878-32 from 1893643 to 2235397 tid 2 2021-03-15T22:23:36.095 INFO:tasks.rados.rados.0.smithi076.stdout:1006: read oid 7 snap -1 2021-03-15T22:23:36.096 INFO:tasks.rados.rados.0.smithi076.stdout:1006: expect deleted 2021-03-15T22:23:36.096 INFO:tasks.rados.rados.0.smithi076.stdout:1007: snap_remove snap 101 2021-03-15T22:23:36.109 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:36.129 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:36.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:36.168 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:36.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:36.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:36.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:36.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:36.297 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:36.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:36.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[62156]: cluster 2021-03-15T22:23:34.763319+0000 osd.4 (osd.4) 69 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:23:36.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[62156]: cluster 2021-03-15T22:23:34.782218+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:23:36.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[62156]: cluster 2021-03-15T22:23:35.026028+0000 mgr.y (mgr.14144) 527 : cluster [DBG] pgmap v576: 138 pgs: 138 active+clean; 136 MiB data, 490 MiB used, 715 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-03-15T22:23:36.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[62156]: cluster 2021-03-15T22:23:35.100073+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.0 deep-scrub ok 2021-03-15T22:23:36.361 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[62156]: cluster 2021-03-15T22:23:35.677435+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.18 deep-scrub ok 2021-03-15T22:23:36.361 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[62156]: cluster 2021-03-15T22:23:36.065932+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2021-03-15T22:23:36.361 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:23:36.362 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[56514]: 23:34.763319+0000 osd.4 (osd.4) 69 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:23:36.362 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[56514]: cluster 2021-03-15T22:23:34.782218+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:23:36.362 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[56514]: cluster 2021-03-15T22:23:35.026028+0000 mgr.y (mgr.14144) 527 : cluster [DBG] pgmap v576: 138 pgs: 138 active+clean; 136 MiB data, 490 MiB used, 715 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-03-15T22:23:36.362 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[56514]: cluster 2021-03-15T22:23:35.100073+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.0 deep-scrub ok 2021-03-15T22:23:36.363 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[56514]: cluster 2021-03-15T22:23:35.677435+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.18 deep-scrub ok 2021-03-15T22:23:36.363 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:36 smithi076 conmon[56514]: cluster 2021-03-15T22:23:36.065932+0000 mon.a (mon.0) 1004 : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2021-03-15T22:23:36.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:36.424 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:36.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:36.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:36.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:36.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:36 smithi125 conmon[56739]: cluster 2021-03-15T22:23:34.763319+0000 osd.4 (osd.4) 69 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:23:36.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:36 smithi125 conmon[56739]: 2021-03-15T22:23:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:36 smithi125 conmon[56739]: cluster 2021-03-15T22:23:34.782218+0000 osd.7 (osd.7) 64 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:23:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:36 smithi125 conmon[56739]: cluster 2021-03-15T22:23:35.026028+0000 mgr.y (mgr.14144) 527 : cluster [DBG] pgmap v576: 138 pgs: 138 active+clean; 136 MiB data, 490 MiB used, 715 GiB / 715 GiB avail; 3.7 MiB/s rd, 1.4 MiB/s wr, 6 op/s 2021-03-15T22:23:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:36 smithi125 conmon[56739]: cluster 2021-03-15T22:23:35.100073+0000 osd.1 (osd.1) 66 : cluster [DBG] 5.0 deep-scrub ok 2021-03-15T22:23:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:36 smithi125 conmon[56739]: cluster 2021-03-15T22:23:35.677435+0000 osd.0 (osd.0) 61 : cluster [DBG] 6.18 deep-scrub ok 2021-03-15T22:23:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:36 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:23:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:36 smithi125 conmon[56739]: 23:36.065932+0000 mon.a (mon.0) 1004 2021-03-15T22:23:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:36 smithi125 conmon[56739]: : cluster [DBG] osdmap e301: 8 total, 8 up, 8 in 2021-03-15T22:23:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:36.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:36.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:36.601 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:23:36.617 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:23:36.634 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:23:36.652 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:23:36.653 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:36.670 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:23:36.680 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:36.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:36.681 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:36.690 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:23:36.709 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:23:36.726 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:23:36.727 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:23:36.728 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:23:36.728 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:23:36.728 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:23:36.728 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:23:36.728 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:23:36.729 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:23:36.729 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:23:36.782 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:36.809 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:36.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:36.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:36.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:36.937 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:36.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:36.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:37.040 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:37.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:37.065 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:37.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:37.071 INFO:tasks.rados.rados.0.smithi076.stdout:1007: done (3 left) 2021-03-15T22:23:37.072 INFO:tasks.rados.rados.0.smithi076.stdout:1008: read oid 42 snap 97 2021-03-15T22:23:37.072 INFO:tasks.rados.rados.0.smithi076.stdout:1008: expect (ObjNum 326 snap 87 seq_num 326) 2021-03-15T22:23:37.073 INFO:tasks.rados.rados.0.smithi076.stdout:1009: snap_remove snap 95 2021-03-15T22:23:37.167 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:37.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:37.196 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:37.197 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:37.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:37.320 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:37.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:37.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:37.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:23:37.392 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[56514]: cluster 2021-03-15T22:23:35.793428+0000 osd.4 ( 2021-03-15T22:23:37.392 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[56514]: osd.4) 70 : cluster [DBG] 5.e scrub ok 2021-03-15T22:23:37.392 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[56514]: cluster 2021-03-15T22:23:36.438270+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:23:37.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[56514]: cluster 2021-03-15T22:23:36.496848+0000 osd.6 (osd.6) 48 : cluster [DBG] 7.d scrub ok 2021-03-15T22:23:37.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[56514]: audit 2021-03-15T22:23:36.581365+0000 mon.a (mon.0) 1005 : audit [DBG] from='client.? 172.21.15.76:0/192592767' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:37.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[56514]: cluster 2021-03-15T22:23:36.678180+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:23:37.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[56514]: cluster 2021-03-15T22:23:37.067748+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2021-03-15T22:23:37.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[62156]: cluster 2021-03-15T22:23:35.793428+0000 osd.4 (osd 2021-03-15T22:23:37.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[62156]: .4) 70 : cluster [DBG] 5.e scrub ok 2021-03-15T22:23:37.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[62156]: cluster 2021-03-15T22:23:36.438270+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:23:37.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[62156]: cluster 2021-03-15T22:23:36.496848+0000 osd.6 (osd.6) 48 : cluster [DBG] 7.d scrub ok 2021-03-15T22:23:37.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[62156]: audit 2021-03-15T22:23:36.581365+0000 mon.a (mon.0) 1005 : audit [DBG] from='client.? 172.21.15.76:0/192592767' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:37.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[62156]: cluster 2021-03-15T22:23:36.678180+0000 osd.0 (osd.0) 62 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:23:37.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:37 smithi076 conmon[62156]: cluster 2021-03-15T22:23:37.067748+0000 mon.a (mon.0) 1006 : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2021-03-15T22:23:37.424 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:37.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:37.448 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:37.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:37.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: cluster 2021-03-15T22:23:35.793428+0000 osd.4 (osd.4) 70 : cluster [DBG] 5.e scrub ok 2021-03-15T22:23:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: cluster 2021-03-15T22:23:36.438270+0000 osd.3 (osd.3) 54 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:23:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: cluster 2021-03-15T22:23:36.496848+0000 osd.6 (osd.6) 48 : cluster [DBG] 7.d scrub ok 2021-03-15T22:23:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:23:37.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: 03-15T22:23:36.581365+0000 mon.a (mon.0) 1005 : audit 2021-03-15T22:23:37.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/192592767' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:37.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: cluster 2021-03-15T22:23:37.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:23:37.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: 23:36.678180+0000 osd.0 (osd 2021-03-15T22:23:37.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: .0) 62 : cluster [DBG] 2021-03-15T22:23:37.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: 4.3 scrub ok 2021-03-15T22:23:37.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: cluster 2021-03-15T22:23:37.067748+0000 mon.a (mon.0) 1006 2021-03-15T22:23:37.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:37 smithi125 conmon[56739]: : cluster [DBG] osdmap e302: 8 total, 8 up, 8 in 2021-03-15T22:23:37.549 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:37.571 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:37.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:37.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:37.675 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:37.702 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:37.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:37.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:37.805 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:37.831 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:37.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:37.933 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:37.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:37.961 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:37.961 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:38.062 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:38.077 INFO:tasks.rados.rados.0.smithi076.stdout:1009: done (4 left) 2021-03-15T22:23:38.078 INFO:tasks.rados.rados.0.smithi076.stdout:1010: delete oid 14 current snap is 102 2021-03-15T22:23:38.078 INFO:tasks.rados.rados.0.smithi076.stdout:1004: finishing write tid 1 to smithi07694878-41 2021-03-15T22:23:38.078 INFO:tasks.rados.rados.0.smithi076.stdout:1004: finishing write tid 2 to smithi07694878-41 2021-03-15T22:23:38.078 INFO:tasks.rados.rados.0.smithi076.stdout:1004: finishing write tid 3 to smithi07694878-41 2021-03-15T22:23:38.079 INFO:tasks.rados.rados.0.smithi076.stdout:1004: finishing write tid 4 to smithi07694878-41 2021-03-15T22:23:38.079 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 233 (ObjNum 370 snap 102 seq_num 370) dirty exists 2021-03-15T22:23:38.079 INFO:tasks.rados.rados.0.smithi076.stdout:1004: left oid 41 (ObjNum 370 snap 102 seq_num 370) 2021-03-15T22:23:38.079 INFO:tasks.rados.rados.0.smithi076.stdout:1005: finishing write tid 1 to smithi07694878-32 2021-03-15T22:23:38.080 INFO:tasks.rados.rados.0.smithi076.stdout:1005: finishing write tid 2 to smithi07694878-32 2021-03-15T22:23:38.080 INFO:tasks.rados.rados.0.smithi076.stdout:1005: finishing write tid 3 to smithi07694878-32 2021-03-15T22:23:38.080 INFO:tasks.rados.rados.0.smithi076.stdout:1005: finishing write tid 4 to smithi07694878-32 2021-03-15T22:23:38.080 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 588 (ObjNum 371 snap 102 seq_num 371) dirty exists 2021-03-15T22:23:38.080 INFO:tasks.rados.rados.0.smithi076.stdout:1005: left oid 32 (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:23:38.081 INFO:tasks.rados.rados.0.smithi076.stdout:1008: expect (ObjNum 326 snap 87 seq_num 326) 2021-03-15T22:23:38.084 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[72291]: debug 2021-03-15T22:23:38.072+0000 7fb932179700 -1 osd.2 pg_epoch: 303 pg[7.8( v 297'210 (0'0,297'210] local-lis/les=91/92 n=3 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [2,7,4] r=0 lpr=91 crt=297'210 lcod 294'208 mlcod 294'208 active+clean+snaptrim trimq=[bd~1] ps=[b9~1,c0~1,c2~1,c6~1,c9~1]] removing snap head 2021-03-15T22:23:38.085 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[75764]: debug 2021-03-15T22:23:38.079+0000 7f9fba7ae700 -1 osd.3 pg_epoch: 303 pg[7.3( v 303'286 (0'0,303'286] local-lis/les=265/266 n=7 ec=91/91 lis/c=265/265 les/c/f=266/266/0 sis=265) [3,2,4] r=0 lpr=265 luod=299'285 crt=299'285 lcod 298'283 mlcod 298'283 active+clean+snaptrim trimq=[bd~1] ps=[b9~1,c0~1,c2~1,c6~1,c9~1]] removing snap head 2021-03-15T22:23:38.085 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[65511]: debug 2021-03-15T22:23:38.077+0000 7f1a1f076700 -1 osd.0 pg_epoch: 303 pg[7.f( v 297'345 (0'0,297'345] local-lis/les=263/264 n=3 ec=91/91 lis/c=263/263 les/c/f=264/264/0 sis=263) [0,3,7] r=0 lpr=263 crt=297'345 lcod 291'343 mlcod 291'343 active+clean+snaptrim trimq=[bd~1] ps=[b9~1,c0~1,c2~1,c6~1,c9~1]] removing snap head 2021-03-15T22:23:38.091 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:38.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:38.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:38.186 INFO:tasks.rados.rados.0.smithi076.stdout:1004: done (4 left) 2021-03-15T22:23:38.186 INFO:tasks.rados.rados.0.smithi076.stdout:1005: done (3 left) 2021-03-15T22:23:38.187 INFO:tasks.rados.rados.0.smithi076.stdout:1006: done (2 left) 2021-03-15T22:23:38.187 INFO:tasks.rados.rados.0.smithi076.stdout:1008: done (1 left) 2021-03-15T22:23:38.187 INFO:tasks.rados.rados.0.smithi076.stdout:1010: done (0 left) 2021-03-15T22:23:38.187 INFO:tasks.rados.rados.0.smithi076.stdout:1011: write oid 31 current snap is 102 2021-03-15T22:23:38.187 INFO:tasks.rados.rados.0.smithi076.stdout:1011: seq_num 373 ranges {669011=569174,2010607=258404} 2021-03-15T22:23:38.192 INFO:tasks.rados.rados.0.smithi076.stdout:1011: writing smithi07694878-31 from 669011 to 1238185 tid 1 2021-03-15T22:23:38.194 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:38.198 INFO:tasks.rados.rados.0.smithi076.stdout:1011: writing smithi07694878-31 from 2010607 to 2269011 tid 2 2021-03-15T22:23:38.198 INFO:tasks.rados.rados.0.smithi076.stdout:1012: copy_from oid 29 from oid 21 current snap is 102 2021-03-15T22:23:38.199 INFO:tasks.rados.rados.0.smithi076.stdout:1011: finishing write tid 1 to smithi07694878-31 2021-03-15T22:23:38.199 INFO:tasks.rados.rados.0.smithi076.stdout:1013: snap_remove snap 97 2021-03-15T22:23:38.217 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:38.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:38.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:38.319 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:38.341 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:38.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:38.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:38.356 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[75764]: debug 2021-03-15T22:23:38.080+0000 7f9fba7ae700 -1 osd.3 pg_epoch: 303 pg[7.3( v 303'288 (0'0,303'288] local-lis/les=265/266 n=5 ec=91/91 lis/c=265/265 les/c/f=266/266/0 sis=265) [3,2,4] r=0 lpr=265 luod=299'285 crt=303'288 lcod 298'283 mlcod 298'283 active+clean+snaptrim trimq=[bd~1] ps=[b9~1,c0~1,c2~1,c6~1,c9~1]] removing snap head 2021-03-15T22:23:38.357 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[56514]: cluster 2021-03-15T22:23:37.026625+0000 mgr.y (mgr. 2021-03-15T22:23:38.357 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[56514]: 14144) 528 : cluster [DBG] pgmap v579: 138 pgs: 138 active+clean; 124 MiB data, 482 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 239 KiB/s wr, 3 op/s 2021-03-15T22:23:38.357 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[56514]: cluster 2021-03-15T22:23:37.441712+0000 osd.3 (osd.3) 55 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:23:38.358 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:23:38.358 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[56514]: T22:23:38.073544+0000 2021-03-15T22:23:38.358 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[56514]: mon.a (mon.0) 1007 : cluster 2021-03-15T22:23:38.358 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[56514]: [DBG] osdmap e303: 8 total, 8 up, 8 in 2021-03-15T22:23:38.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[62156]: cluster 2021-03-15T22:23:37.026625+0000 2021-03-15T22:23:38.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[62156]: mgr.y (mgr.14144) 528 : cluster [DBG] pgmap v579: 138 pgs: 138 active+clean; 124 MiB data, 482 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 239 KiB/s wr, 3 op/s 2021-03-15T22:23:38.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:23:38.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[62156]: 03-15T22:23:37.441712+0000 osd.3 (osd.3) 55 2021-03-15T22:23:38.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[62156]: : cluster [DBG] 4.11 scrub ok 2021-03-15T22:23:38.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:23:38.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[62156]: 22:23:38.073544+0000 mon.a (mon 2021-03-15T22:23:38.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:38 smithi076 conmon[62156]: .0) 1007 : cluster [DBG] osdmap e303: 8 total, 8 up, 8 in 2021-03-15T22:23:38.444 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:38.470 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:38.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:38.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:38.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:38 smithi125 conmon[56739]: cluster 2021-03-15T22:23:37.026625+0000 mgr.y (mgr.14144) 528 : cluster [DBG] pgmap v579: 138 pgs: 138 active+clean; 124 MiB data, 482 MiB used, 715 GiB / 715 GiB avail; 1.5 KiB/s rd, 239 KiB/s wr, 3 op/s 2021-03-15T22:23:38.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:38 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:23:38.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:38 smithi125 conmon[56739]: 15T22:23:37.441712+0000 osd.3 (osd.3) 55 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:23:38.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:38 smithi125 conmon[56739]: cluster 2021-03-15T22:23:38.073544+0000 mon.a (mon.0) 1007 : cluster [DBG] 2021-03-15T22:23:38.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:38 smithi125 conmon[56739]: osdmap e303: 8 total, 8 up, 8 in 2021-03-15T22:23:38.573 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:38.599 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:38.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:38.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:38.702 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:38.724 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:38.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:38.726 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:38.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:38.849 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:38.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:38.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:38.920 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:38.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:38.975 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:38.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:38.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:39.077 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:39.104 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:39.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:39.106 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:39.162 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: cluster 2021-03-15T22:23:39.163 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: 2021-03-15T22:23:39.163 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: 2021 2021-03-15T22:23:39.163 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: - 2021-03-15T22:23:39.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: 03 2021-03-15T22:23:39.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: - 2021-03-15T22:23:39.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: 15 2021-03-15T22:23:39.164 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: T 2021-03-15T22:23:39.165 INFO:tasks.rados.rados.0.smithi076.stdout:1013: done (2 left) 2021-03-15T22:23:39.165 INFO:tasks.rados.rados.0.smithi076.stdout:1014: read oid 34 snap -1 2021-03-15T22:23:39.166 INFO:tasks.rados.rados.0.smithi076.stdout:1014: expect deleted 2021-03-15T22:23:39.166 INFO:tasks.rados.rados.0.smithi076.stdout:1015: write (excl) oid 33 current snap is 102 2021-03-15T22:23:39.166 INFO:tasks.rados.rados.0.smithi076.stdout:1011: finishing write tid 2 to smithi07694878-31 2021-03-15T22:23:39.166 INFO:tasks.rados.rados.0.smithi076.stdout:1015: seq_num 374 ranges {685818=644423,2069089=216729} 2021-03-15T22:23:39.173 INFO:tasks.rados.rados.0.smithi076.stdout:1015: writing smithi07694878-33 from 685818 to 1330241 tid 1 2021-03-15T22:23:39.179 INFO:tasks.rados.rados.0.smithi076.stdout:1015: writing smithi07694878-33 from 2069089 to 2285818 tid 2 2021-03-15T22:23:39.180 INFO:tasks.rados.rados.0.smithi076.stdout:1016: snap_create 2021-03-15T22:23:39.180 INFO:tasks.rados.rados.0.smithi076.stdout:1011: finishing write tid 3 to smithi07694878-31 2021-03-15T22:23:39.180 INFO:tasks.rados.rados.0.smithi076.stdout:1011: finishing write tid 4 to smithi07694878-31 2021-03-15T22:23:39.180 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 162 (ObjNum 373 snap 102 seq_num 373) dirty exists 2021-03-15T22:23:39.180 INFO:tasks.rados.rados.0.smithi076.stdout:1011: left oid 31 (ObjNum 373 snap 102 seq_num 373) 2021-03-15T22:23:39.181 INFO:tasks.rados.rados.0.smithi076.stdout:1012: finishing copy_from to smithi07694878-29 2021-03-15T22:23:39.181 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 520 (ObjNum 335 snap 89 seq_num 335) dirty exists 2021-03-15T22:23:39.181 INFO:tasks.rados.rados.0.smithi076.stdout:1012: finishing copy_from racing read to smithi07694878-29 2021-03-15T22:23:39.181 INFO:tasks.rados.rados.0.smithi076.stdout:1015: finishing write tid 1 to smithi07694878-33 2021-03-15T22:23:39.182 INFO:tasks.rados.rados.0.smithi076.stdout:1015: finishing write tid 2 to smithi07694878-33 2021-03-15T22:23:39.182 INFO:tasks.rados.rados.0.smithi076.stdout:1015: finishing write tid 3 to smithi07694878-33 2021-03-15T22:23:39.183 INFO:tasks.rados.rados.0.smithi076.stdout:1015: finishing write tid 4 to smithi07694878-33 2021-03-15T22:23:39.183 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 233 (ObjNum 374 snap 102 seq_num 374) dirty exists 2021-03-15T22:23:39.183 INFO:tasks.rados.rados.0.smithi076.stdout:1015: left oid 33 (ObjNum 374 snap 102 seq_num 374) 2021-03-15T22:23:39.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:39.231 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:39.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:39.232 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:39.333 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:39.357 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:39.358 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:39.417 INFO:journalctl@ceph.osd.1.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[68886]: debug 2021-03-15T22:23:39.169+0000 7fae34dad700 -1 osd.1 pg_epoch: 304 pg[7.b( v 303'173 (0'0,303'173] local-lis/les=91/92 n=3 ec=91/91 lis/c=91/91 les/c/f=92/92/0 sis=91) [1,5,6] r=0 lpr=91 crt=303'173 lcod 299'171 mlcod 299'171 active+clean+snaptrim trimq=[c1~1] ps=[b9~1,bd~1,c0~1,c2~1,c6~1,c9~1]] removing snap head 2021-03-15T22:23:39.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[56514]: cluster 2021-03-15T22:23:38.487731+0000 osd.3 (osd.3 2021-03-15T22:23:39.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[56514]: ) 56 : cluster [DBG] 3.7 deep-scrub ok 2021-03-15T22:23:39.419 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[75764]: debug 2021-03-15T22:23:39.162+0000 7f9fbe7b6700 -1 osd.3 pg_epoch: 304 pg[7.3( v 303'292 (0'0,303'292] local-lis/les=265/266 n=3 ec=91/91 lis/c=265/265 les/c/f=266/266/0 sis=265) [3,2,4] r=0 lpr=265 crt=303'292 lcod 303'290 mlcod 303'290 active+clean+snaptrim trimq=[c1~1] ps=[b9~1,bd~1,c0~1,c2~1,c6~1,c9~1]] removing snap head 2021-03-15T22:23:39.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: 22 2021-03-15T22:23:39.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: : 2021-03-15T22:23:39.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: 23 2021-03-15T22:23:39.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: : 2021-03-15T22:23:39.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: 38 2021-03-15T22:23:39.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: . 2021-03-15T22:23:39.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: 487731 2021-03-15T22:23:39.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: +0000 2021-03-15T22:23:39.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: 2021-03-15T22:23:39.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: osd.3 2021-03-15T22:23:39.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: ( 2021-03-15T22:23:39.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: osd 2021-03-15T22:23:39.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: . 2021-03-15T22:23:39.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: 3 2021-03-15T22:23:39.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: ) 2021-03-15T22:23:39.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: 56 2021-03-15T22:23:39.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: : 2021-03-15T22:23:39.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: cluster 2021-03-15T22:23:39.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:39 smithi076 conmon[62156]: [DBG] 3.7 deep-scrub ok 2021-03-15T22:23:39.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:39.485 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:39.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:39.486 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:39.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:39 smithi125 conmon[56739]: cluster 2021-03-15T22:23:38.487731+0000 osd.3 (osd.3) 56 : cluster 2021-03-15T22:23:39.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:39 smithi125 conmon[56739]: [DBG] 3.7 deep-scrub ok 2021-03-15T22:23:39.537 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:23:39 smithi125 conmon[63905]: debug 2021-03-15T22:23:39.165+0000 7fce00c31700 -1 osd.5 pg_epoch: 304 pg[7.0( v 303'597 (0'0,303'597] local-lis/les=268/269 n=10 ec=91/91 lis/c=268/268 les/c/f=269/269/0 sis=268) [5,1,3] r=0 lpr=268 crt=303'597 lcod 303'596 mlcod 303'596 active+clean+snaptrim trimq=[c1~1] ps=[b9~1,bd~1,c0~1,c2~1,c6~1,c9~1]] removing snap head 2021-03-15T22:23:39.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:39.601 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:23:39.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:39.612 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:39.613 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:39.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:39.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:39.738 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:39.738 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:39.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:39.866 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:39.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:39.867 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:39.968 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:39.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:39.991 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:39.992 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:39.997 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:23:39.997 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:23:40.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:23:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:23:40.067 INFO:tasks.rados.rados.0.smithi076.stdout:1011: done (4 left) 2021-03-15T22:23:40.068 INFO:tasks.rados.rados.0.smithi076.stdout:1012: done (3 left) 2021-03-15T22:23:40.068 INFO:tasks.rados.rados.0.smithi076.stdout:1014: done (2 left) 2021-03-15T22:23:40.068 INFO:tasks.rados.rados.0.smithi076.stdout:1015: done (1 left) 2021-03-15T22:23:40.069 INFO:tasks.rados.rados.0.smithi076.stdout:1016: done (0 left) 2021-03-15T22:23:40.069 INFO:tasks.rados.rados.0.smithi076.stdout:1017: rollback oid 37 current snap is 103 2021-03-15T22:23:40.070 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 37 to 100 2021-03-15T22:23:40.070 INFO:tasks.rados.rados.0.smithi076.stdout:1018: read oid 32 snap -1 2021-03-15T22:23:40.071 INFO:tasks.rados.rados.0.smithi076.stdout:1018: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:23:40.071 INFO:tasks.rados.rados.0.smithi076.stdout:1019: rollback oid 25 current snap is 103 2021-03-15T22:23:40.071 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 100 2021-03-15T22:23:40.072 INFO:tasks.rados.rados.0.smithi076.stdout:1020: write (excl) oid 29 current snap is 103 2021-03-15T22:23:40.072 INFO:tasks.rados.rados.0.smithi076.stdout:1020: seq_num 375 ranges {702625=719672,2127571=175054} 2021-03-15T22:23:40.075 INFO:tasks.rados.rados.0.smithi076.stdout:1020: writing smithi07694878-29 from 702625 to 1422297 tid 1 2021-03-15T22:23:40.075 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.7 primary-affinity to 0 (802) 2021-03-15T22:23:40.079 INFO:tasks.rados.rados.0.smithi076.stdout:1020: writing smithi07694878-29 from 2127571 to 2302625 tid 2 2021-03-15T22:23:40.080 INFO:tasks.rados.rados.0.smithi076.stdout:1021: write oid 20 current snap is 103 2021-03-15T22:23:40.080 INFO:tasks.rados.rados.0.smithi076.stdout:1019: finishing rollback tid 0 to smithi07694878-25 2021-03-15T22:23:40.081 INFO:tasks.rados.rados.0.smithi076.stdout:1019: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:23:40.081 INFO:tasks.rados.rados.0.smithi076.stdout:1019: finishing rollback tid 2 to smithi07694878-25 2021-03-15T22:23:40.081 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 164 (ObjNum 365 snap 99 seq_num 365) dirty exists 2021-03-15T22:23:40.081 INFO:tasks.rados.rados.0.smithi076.stdout:1021: seq_num 376 ranges {719432=794921,2186053=133379} 2021-03-15T22:23:40.088 INFO:tasks.rados.rados.0.smithi076.stdout:1021: writing smithi07694878-20 from 719432 to 1514353 tid 1 2021-03-15T22:23:40.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:40.093 INFO:tasks.rados.rados.0.smithi076.stdout:1021: writing smithi07694878-20 from 2186053 to 2319432 tid 2 2021-03-15T22:23:40.094 INFO:tasks.rados.rados.0.smithi076.stdout:1019: done (4 left) 2021-03-15T22:23:40.094 INFO:tasks.rados.rados.0.smithi076.stdout:1022: delete oid 16 current snap is 103 2021-03-15T22:23:40.094 INFO:tasks.rados.rados.0.smithi076.stdout:1017: finishing rollback tid 0 to smithi07694878-37 2021-03-15T22:23:40.095 INFO:tasks.rados.rados.0.smithi076.stdout:1017: finishing rollback tid 1 to smithi07694878-37 2021-03-15T22:23:40.095 INFO:tasks.rados.rados.0.smithi076.stdout:1017: finishing rollback tid 2 to smithi07694878-37 2021-03-15T22:23:40.095 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 607 (ObjNum 280 snap 74 seq_num 280) dirty exists 2021-03-15T22:23:40.095 INFO:tasks.rados.rados.0.smithi076.stdout:1018: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:23:40.118 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:40.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:40.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:40.221 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:40.225 INFO:tasks.rados.rados.0.smithi076.stdout:1020: finishing write tid 1 to smithi07694878-29 2021-03-15T22:23:40.225 INFO:tasks.rados.rados.0.smithi076.stdout:1020: finishing write tid 2 to smithi07694878-29 2021-03-15T22:23:40.226 INFO:tasks.rados.rados.0.smithi076.stdout:1020: finishing write tid 3 to smithi07694878-29 2021-03-15T22:23:40.226 INFO:tasks.rados.rados.0.smithi076.stdout:1020: finishing write tid 4 to smithi07694878-29 2021-03-15T22:23:40.226 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 611 (ObjNum 375 snap 103 seq_num 375) dirty exists 2021-03-15T22:23:40.227 INFO:tasks.rados.rados.0.smithi076.stdout:1020: left oid 29 (ObjNum 375 snap 103 seq_num 375) 2021-03-15T22:23:40.227 INFO:tasks.rados.rados.0.smithi076.stdout:1021: finishing write tid 1 to smithi07694878-20 2021-03-15T22:23:40.227 INFO:tasks.rados.rados.0.smithi076.stdout:1021: finishing write tid 2 to smithi07694878-20 2021-03-15T22:23:40.228 INFO:tasks.rados.rados.0.smithi076.stdout:1021: finishing write tid 3 to smithi07694878-20 2021-03-15T22:23:40.228 INFO:tasks.rados.rados.0.smithi076.stdout:1021: finishing write tid 4 to smithi07694878-20 2021-03-15T22:23:40.228 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 237 (ObjNum 376 snap 103 seq_num 376) dirty exists 2021-03-15T22:23:40.228 INFO:tasks.rados.rados.0.smithi076.stdout:1021: left oid 20 (ObjNum 376 snap 103 seq_num 376) 2021-03-15T22:23:40.228 INFO:tasks.rados.rados.0.smithi076.stdout:1017: done (4 left) 2021-03-15T22:23:40.229 INFO:tasks.rados.rados.0.smithi076.stdout:1018: done (3 left) 2021-03-15T22:23:40.229 INFO:tasks.rados.rados.0.smithi076.stdout:1020: done (2 left) 2021-03-15T22:23:40.229 INFO:tasks.rados.rados.0.smithi076.stdout:1021: done (1 left) 2021-03-15T22:23:40.229 INFO:tasks.rados.rados.0.smithi076.stdout:1022: done (0 left) 2021-03-15T22:23:40.230 INFO:tasks.rados.rados.0.smithi076.stdout:1023: write oid 30 current snap is 103 2021-03-15T22:23:40.230 INFO:tasks.rados.rados.0.smithi076.stdout:1023: seq_num 378 ranges {753046=545419,1786664=431793} 2021-03-15T22:23:40.233 INFO:tasks.rados.rados.0.smithi076.stdout:1023: writing smithi07694878-30 from 753046 to 1298465 tid 1 2021-03-15T22:23:40.241 INFO:tasks.rados.rados.0.smithi076.stdout:1023: writing smithi07694878-30 from 1786664 to 2218457 tid 2 2021-03-15T22:23:40.241 INFO:tasks.rados.rados.0.smithi076.stdout:1024: copy_from oid 11 from oid 4 current snap is 103 2021-03-15T22:23:40.241 INFO:tasks.rados.rados.0.smithi076.stdout:1023: finishing write tid 1 to smithi07694878-30 2021-03-15T22:23:40.242 INFO:tasks.rados.rados.0.smithi076.stdout:1025: copy_from oid 17 from oid 50 current snap is 103 2021-03-15T22:23:40.242 INFO:tasks.rados.rados.0.smithi076.stdout:1026: rollback oid 29 current snap is 103 2021-03-15T22:23:40.242 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 29 to 102 2021-03-15T22:23:40.242 INFO:tasks.rados.rados.0.smithi076.stdout:1027: write (excl) oid 10 current snap is 103 2021-03-15T22:23:40.243 INFO:tasks.rados.rados.0.smithi076.stdout:1027: seq_num 379 ranges {769853=620668,1845146=524707} 2021-03-15T22:23:40.245 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:40.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:40.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:40.247 INFO:tasks.rados.rados.0.smithi076.stdout:1027: writing smithi07694878-10 from 769853 to 1390521 tid 1 2021-03-15T22:23:40.253 INFO:tasks.rados.rados.0.smithi076.stdout:1027: writing smithi07694878-10 from 1845146 to 2369853 tid 2 2021-03-15T22:23:40.253 INFO:tasks.rados.rados.0.smithi076.stdout:1028: rollback oid 45 current snap is 103 2021-03-15T22:23:40.253 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 45 to 102 2021-03-15T22:23:40.254 INFO:tasks.rados.rados.0.smithi076.stdout:1025: finishing copy_from to smithi07694878-17 2021-03-15T22:23:40.254 INFO:tasks.rados.rados.0.smithi076.stdout:1025: got expected ENOENT (src dne) 2021-03-15T22:23:40.254 INFO:tasks.rados.rados.0.smithi076.stdout:1025: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:23:40.254 INFO:tasks.rados.rados.0.smithi076.stdout:1023: finishing write tid 2 to smithi07694878-30 2021-03-15T22:23:40.255 INFO:tasks.rados.rados.0.smithi076.stdout:1023: finishing write tid 3 to smithi07694878-30 2021-03-15T22:23:40.255 INFO:tasks.rados.rados.0.smithi076.stdout:1026: finishing rollback tid 0 to smithi07694878-29 2021-03-15T22:23:40.255 INFO:tasks.rados.rados.0.smithi076.stdout:1024: finishing copy_from to smithi07694878-11 2021-03-15T22:23:40.255 INFO:tasks.rados.rados.0.smithi076.stdout:1024: got expected ENOENT (src dne) 2021-03-15T22:23:40.256 INFO:tasks.rados.rados.0.smithi076.stdout:1026: finishing rollback tid 1 to smithi07694878-29 2021-03-15T22:23:40.256 INFO:tasks.rados.rados.0.smithi076.stdout:1026: finishing rollback tid 2 to smithi07694878-29 2021-03-15T22:23:40.257 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 614 (ObjNum 335 snap 89 seq_num 335) dirty exists 2021-03-15T22:23:40.257 INFO:tasks.rados.rados.0.smithi076.stdout:1023: finishing write tid 4 to smithi07694878-30 2021-03-15T22:23:40.257 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 567 (ObjNum 378 snap 103 seq_num 378) dirty exists 2021-03-15T22:23:40.257 INFO:tasks.rados.rados.0.smithi076.stdout:1023: left oid 30 (ObjNum 378 snap 103 seq_num 378) 2021-03-15T22:23:40.258 INFO:tasks.rados.rados.0.smithi076.stdout:1024: finishing copy_from racing read to smithi07694878-11 2021-03-15T22:23:40.258 INFO:tasks.rados.rados.0.smithi076.stdout:1027: finishing write tid 1 to smithi07694878-10 2021-03-15T22:23:40.258 INFO:tasks.rados.rados.0.smithi076.stdout:1023: done (5 left) 2021-03-15T22:23:40.258 INFO:tasks.rados.rados.0.smithi076.stdout:1024: done (4 left) 2021-03-15T22:23:40.258 INFO:tasks.rados.rados.0.smithi076.stdout:1025: done (3 left) 2021-03-15T22:23:40.259 INFO:tasks.rados.rados.0.smithi076.stdout:1026: done (2 left) 2021-03-15T22:23:40.259 INFO:tasks.rados.rados.0.smithi076.stdout:1029: read oid 47 snap -1 2021-03-15T22:23:40.259 INFO:tasks.rados.rados.0.smithi076.stdout:1029: expect (ObjNum 324 snap 87 seq_num 324) 2021-03-15T22:23:40.259 INFO:tasks.rados.rados.0.smithi076.stdout:1030: rollback oid 6 current snap is 103 2021-03-15T22:23:40.260 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 102 2021-03-15T22:23:40.261 INFO:tasks.rados.rados.0.smithi076.stdout:1031: read oid 11 snap -1 2021-03-15T22:23:40.261 INFO:tasks.rados.rados.0.smithi076.stdout:1031: expect deleted 2021-03-15T22:23:40.261 INFO:tasks.rados.rados.0.smithi076.stdout:1032: read oid 13 snap 102 2021-03-15T22:23:40.261 INFO:tasks.rados.rados.0.smithi076.stdout:1032: expect deleted 2021-03-15T22:23:40.262 INFO:tasks.rados.rados.0.smithi076.stdout:1033: copy_from oid 3 from oid 46 current snap is 103 2021-03-15T22:23:40.262 INFO:tasks.rados.rados.0.smithi076.stdout:1034: rollback oid 38 current snap is 103 2021-03-15T22:23:40.262 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 38 to 102 2021-03-15T22:23:40.262 INFO:tasks.rados.rados.0.smithi076.stdout:1035: snap_remove snap 100 2021-03-15T22:23:40.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[56514]: cluster 2021-03-15T22:23:38.685701+0000 osd.4 (osd.4) 71 : cluster [DBG] 4.e scrub ok 2021-03-15T22:23:40.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[56514]: cluster 2021-03-15T22:23:38.967929+0000 osd.5 (osd.5) 59 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:23:40.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[56514]: cluster 2021-03-15T22:23:39.026997+0000 mgr.y (mgr.14144) 529 : cluster [DBG] pgmap v582: 138 pgs: 138 active+clean; 125 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 494 KiB/s wr, 5 op/s 2021-03-15T22:23:40.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[56514]: cluster 2021-03-15T22:23:39.161386+0000 mon.a (mon.0) 1008 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2021-03-15T22:23:40.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[56514]: audit 2021-03-15T22:23:39.398777+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? 172.21.15.76:0/3219801465' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2021-03-15T22:23:40.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[56514]: cluster 2021-03-15T22:23:39.525282+0000 osd.3 (osd.3) 57 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:40.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[56514]: cluster 2021-03-15T22:23:39.525506+0000 osd.3 (osd.3) 58 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:40.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[56514]: audit 2021-03-15T22:23:39.599908+0000 mgr.y (mgr.14144) 530 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:40.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[56514]: cluster 2021-03-15T22:23:39.695661+0000 osd.0 (osd.0) 63 : cluster [DBG] 3.12 deep-scrub ok 2021-03-15T22:23:40.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[56514]: audit 2021-03-15T22:23:40.058067+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? 172.21.15.76:0/3219801465' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2021-03-15T22:23:40.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[56514]: cluster 2021-03-15T22:23:40.058129+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2021-03-15T22:23:40.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: cluster 2021-03-15T22:23:38.685701+0000 osd.4 (osd.4) 71 : cluster [DBG] 4.e scrub ok 2021-03-15T22:23:40.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: cluster 2021-03-15T22:23:38.967929+0000 osd.5 (osd.5) 59 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:23:40.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: cluster 2021-03-15T22:23:39.026997+0000 mgr.y (mgr.14144) 529 : cluster [DBG] pgmap v582: 138 pgs: 138 active+clean; 125 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 494 KiB/s wr, 5 op/s 2021-03-15T22:23:40.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: cluster 2021-03-15T22:23:39.161386+0000 mon.a (mon.0) 1008 : cluster [DBG] osdmap e304: 8 total, 8 up, 8 in 2021-03-15T22:23:40.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:23:40.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: 22:23:39.398777+0000 mon.a (mon.0) 1009 : audit [INF] from='client.? 172.21.15.76:0/3219801465' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2021-03-15T22:23:40.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: cluster 2021-03-15T22:23:39.525282+0000 osd.3 (osd.3) 57 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:40.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: cluster 2021-03-15T22:23:39.525506+0000 osd.3 (osd.3) 58 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:40.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: audit 2021-03-15T22:23:39.599908+0000 mgr.y (mgr.14144) 530 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:40.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: cluster 2021-03-15T22:23:39.695661+0000 osd.0 (osd.0) 63 : cluster [DBG] 3.12 deep-scrub ok 2021-03-15T22:23:40.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: audit 2021-03-15T22:23:40.058067+0000 mon.a (mon.0) 1010 : audit [INF] from='client.? 172.21.15.76:0/3219801465' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2021-03-15T22:23:40.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:40 smithi076 conmon[62156]: cluster 2021-03-15T22:23:40.058129+0000 mon.a (mon.0) 1011 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2021-03-15T22:23:40.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:40.371 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:40.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:40.372 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:40.473 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:40.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:40.498 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:40.498 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:40.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: cluster 2021-03-15T22:23:38.685701+0000 osd.4 (osd.4) 71 : 2021-03-15T22:23:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: cluster [DBG] 4.e scrub ok 2021-03-15T22:23:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: cluster 2021-03-15T22:23:38.967929+0000 osd.5 (osd.5) 59 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:23:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:23:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: T22:23:39.026997+0000 2021-03-15T22:23:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: mgr.y (mgr.14144) 529 : cluster 2021-03-15T22:23:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: [DBG] pgmap v582: 138 pgs: 138 active+clean; 125 MiB data, 485 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 494 KiB/s wr, 5 op/s 2021-03-15T22:23:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:23:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: T22:23:39.161386+0000 mon.a 2021-03-15T22:23:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: (mon.0) 1008 : cluster 2021-03-15T22:23:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: [DBG] osdmap e304: 8 total, 8 up, 8 in 2021-03-15T22:23:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: audit 2021 2021-03-15T22:23:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: -03-15T22:23:39 2021-03-15T22:23:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: .398777+0000 mon.a (mon. 2021-03-15T22:23:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: 0) 1009 : audit [INF] 2021-03-15T22:23:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/3219801465' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]: dispatch 2021-03-15T22:23:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:23:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: -15T22:23:39.525282 2021-03-15T22:23:40.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: +0000 osd.3 (osd.3) 2021-03-15T22:23:40.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: 57 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:40.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:23:40.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: -03-15T22:23:39 2021-03-15T22:23:40.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: .525506+0000 osd.3 (osd. 2021-03-15T22:23:40.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: 3) 58 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:40.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: 2021-03-15T22:23:40.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: audit 2021-03-15T22:23:40.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: 2021-03-15T22:23:39 2021-03-15T22:23:40.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: .599908+0000 mgr.y (mgr.14144 2021-03-15T22:23:40.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: ) 530 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:40.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:23:40.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: -03-15T22:23:39 2021-03-15T22:23:40.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: .695661+0000 osd.0 (osd. 2021-03-15T22:23:40.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: 0) 63 : cluster [DBG] 3.12 deep-scrub ok 2021-03-15T22:23:40.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: 2021-03-15T22:23:40.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: audit 2021-03-15T22:23:40.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: 2021-03-15T22:23 2021-03-15T22:23:40.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: :40.058067+0000 mon.a ( 2021-03-15T22:23:40.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: mon.0) 1010 : audit 2021-03-15T22:23:40.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/3219801465' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.0}]': finished 2021-03-15T22:23:40.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:23:40.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: -03-15T22:23: 2021-03-15T22:23:40.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: 40.058129+0000 mon.a (mon. 2021-03-15T22:23:40.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: 0) 1011 : cluster [DBG] osdmap e305: 8 total, 8 up, 8 in 2021-03-15T22:23:40.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:40 smithi125 conmon[56739]: 2021-03-15T22:23:40.599 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:40.622 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:40.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:40.623 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:40.724 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:40.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:40.750 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:40.751 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:40.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:40.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:40.878 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:40.879 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:40.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:41.003 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:41.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:41.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:41.072 INFO:tasks.rados.rados.0.smithi076.stdout:1035: done (8 left) 2021-03-15T22:23:41.073 INFO:tasks.rados.rados.0.smithi076.stdout:1036: rollback oid 34 current snap is 103 2021-03-15T22:23:41.073 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 34 to 102 2021-03-15T22:23:41.073 INFO:tasks.rados.rados.0.smithi076.stdout:1037: write (excl) oid 23 current snap is 103 2021-03-15T22:23:41.073 INFO:tasks.rados.rados.0.smithi076.stdout:1037: seq_num 380 ranges {786660=695917,1903628=435462} 2021-03-15T22:23:41.081 INFO:tasks.rados.rados.0.smithi076.stdout:1037: writing smithi07694878-23 from 786660 to 1482577 tid 1 2021-03-15T22:23:41.086 INFO:tasks.rados.rados.0.smithi076.stdout:1037: writing smithi07694878-23 from 1903628 to 2339090 tid 2 2021-03-15T22:23:41.086 INFO:tasks.rados.rados.0.smithi076.stdout:1038: read oid 20 snap -1 2021-03-15T22:23:41.087 INFO:tasks.rados.rados.0.smithi076.stdout:1038: expect (ObjNum 376 snap 103 seq_num 376) 2021-03-15T22:23:41.087 INFO:tasks.rados.rados.0.smithi076.stdout:1039: write (excl) oid 35 current snap is 103 2021-03-15T22:23:41.087 INFO:tasks.rados.rados.0.smithi076.stdout:1039: seq_num 381 ranges {403467=487519,1562110=579120} 2021-03-15T22:23:41.091 INFO:tasks.rados.rados.0.smithi076.stdout:1039: writing smithi07694878-35 from 403467 to 890986 tid 1 2021-03-15T22:23:41.098 INFO:tasks.rados.rados.0.smithi076.stdout:1039: writing smithi07694878-35 from 1562110 to 2141230 tid 2 2021-03-15T22:23:41.098 INFO:tasks.rados.rados.0.smithi076.stdout:1040: rollback oid 24 current snap is 103 2021-03-15T22:23:41.099 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 102 2021-03-15T22:23:41.099 INFO:tasks.rados.rados.0.smithi076.stdout:1029: expect (ObjNum 324 snap 87 seq_num 324) 2021-03-15T22:23:41.105 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:41.128 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:41.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:41.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:41.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:41.254 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-15T22:23:41.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:41.258 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:41.259 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:41.280 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:41.302 INFO:tasks.rados.rados.0.smithi076.stdout:1029: done (12 left) 2021-03-15T22:23:41.302 INFO:tasks.rados.rados.0.smithi076.stdout:1031: done (11 left) 2021-03-15T22:23:41.302 INFO:tasks.rados.rados.0.smithi076.stdout:1041: write (excl) oid 40 current snap is 103 2021-03-15T22:23:41.303 INFO:tasks.rados.rados.0.smithi076.stdout:1033: finishing copy_from to smithi07694878-3 2021-03-15T22:23:41.303 INFO:tasks.rados.rados.0.smithi076.stdout:1033: got expected ENOENT (src dne) 2021-03-15T22:23:41.303 INFO:tasks.rados.rados.0.smithi076.stdout:1033: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:23:41.303 INFO:tasks.rados.rados.0.smithi076.stdout:1027: finishing write tid 2 to smithi07694878-10 2021-03-15T22:23:41.304 INFO:tasks.rados.rados.0.smithi076.stdout:1027: finishing write tid 3 to smithi07694878-10 2021-03-15T22:23:41.304 INFO:tasks.rados.rados.0.smithi076.stdout:1027: finishing write tid 4 to smithi07694878-10 2021-03-15T22:23:41.304 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 168 (ObjNum 379 snap 103 seq_num 379) dirty exists 2021-03-15T22:23:41.304 INFO:tasks.rados.rados.0.smithi076.stdout:1027: left oid 10 (ObjNum 379 snap 103 seq_num 379) 2021-03-15T22:23:41.305 INFO:tasks.rados.rados.0.smithi076.stdout:1028: finishing rollback tid 0 to smithi07694878-45 2021-03-15T22:23:41.305 INFO:tasks.rados.rados.0.smithi076.stdout:1028: finishing rollback tid 1 to smithi07694878-45 2021-03-15T22:23:41.305 INFO:tasks.rados.rados.0.smithi076.stdout:1028: finishing rollback tid 2 to smithi07694878-45 2021-03-15T22:23:41.305 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 263 (ObjNum 332 snap 88 seq_num 332) dirty exists 2021-03-15T22:23:41.306 INFO:tasks.rados.rados.0.smithi076.stdout:1030: finishing rollback tid 0 to smithi07694878-6 2021-03-15T22:23:41.306 INFO:tasks.rados.rados.0.smithi076.stdout:1034: finishing rollback tid 0 to smithi07694878-38 2021-03-15T22:23:41.306 INFO:tasks.rados.rados.0.smithi076.stdout:1030: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:23:41.306 INFO:tasks.rados.rados.0.smithi076.stdout:1030: finishing rollback tid 2 to smithi07694878-6 2021-03-15T22:23:41.306 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 266 (ObjNum 339 snap 91 seq_num 339) dirty exists 2021-03-15T22:23:41.307 INFO:tasks.rados.rados.0.smithi076.stdout:1034: finishing rollback tid 1 to smithi07694878-38 2021-03-15T22:23:41.307 INFO:tasks.rados.rados.0.smithi076.stdout:1034: finishing rollback tid 2 to smithi07694878-38 2021-03-15T22:23:41.307 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 389 (ObjNum 273 snap 72 seq_num 273) dirty exists 2021-03-15T22:23:41.307 INFO:tasks.rados.rados.0.smithi076.stdout:1036: finishing rollback tid 1 to smithi07694878-34 2021-03-15T22:23:41.308 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 0 (ObjNum 261598160 snap 21896 seq_num 261982784) dirty dne 2021-03-15T22:23:41.308 INFO:tasks.rados.rados.0.smithi076.stdout:1037: finishing write tid 1 to smithi07694878-23 2021-03-15T22:23:41.308 INFO:tasks.rados.rados.0.smithi076.stdout:1041: seq_num 382 ranges {420274=562768,1620592=439131} 2021-03-15T22:23:41.309 INFO:tasks.rados.rados.0.smithi076.stdout:1041: writing smithi07694878-40 from 420274 to 983042 tid 1 2021-03-15T22:23:41.314 INFO:tasks.rados.rados.0.smithi076.stdout:1041: writing smithi07694878-40 from 1620592 to 2059723 tid 2 2021-03-15T22:23:41.314 INFO:tasks.rados.rados.0.smithi076.stdout:1027: done (11 left) 2021-03-15T22:23:41.315 INFO:tasks.rados.rados.0.smithi076.stdout:1028: done (10 left) 2021-03-15T22:23:41.315 INFO:tasks.rados.rados.0.smithi076.stdout:1030: done (9 left) 2021-03-15T22:23:41.315 INFO:tasks.rados.rados.0.smithi076.stdout:1032: done (8 left) 2021-03-15T22:23:41.315 INFO:tasks.rados.rados.0.smithi076.stdout:1033: done (7 left) 2021-03-15T22:23:41.316 INFO:tasks.rados.rados.0.smithi076.stdout:1034: done (6 left) 2021-03-15T22:23:41.316 INFO:tasks.rados.rados.0.smithi076.stdout:1036: done (5 left) 2021-03-15T22:23:41.316 INFO:tasks.rados.rados.0.smithi076.stdout:1042: write (excl) oid 2 current snap is 103 2021-03-15T22:23:41.316 INFO:tasks.rados.rados.0.smithi076.stdout:1042: seq_num 383 ranges {437081=638017,1679074=582789} 2021-03-15T22:23:41.319 INFO:tasks.rados.rados.0.smithi076.stdout:1042: writing smithi07694878-2 from 437081 to 1075098 tid 1 2021-03-15T22:23:41.327 INFO:tasks.rados.rados.0.smithi076.stdout:1042: writing smithi07694878-2 from 1679074 to 2261863 tid 2 2021-03-15T22:23:41.327 INFO:tasks.rados.rados.0.smithi076.stdout:1043: snap_create 2021-03-15T22:23:41.328 INFO:tasks.rados.rados.0.smithi076.stdout:1038: expect (ObjNum 376 snap 103 seq_num 376) 2021-03-15T22:23:41.360 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:41.390 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:41.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:41.432 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:41 smithi076 conmon[56514]: cluster 2021-03-15T22:23:40.199092+0000 osd.1 (osd 2021-03-15T22:23:41.432 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:41 smithi076 conmon[56514]: .1) 67 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:23:41.432 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:41 smithi076 conmon[56514]: cluster 2021-03-15T22:23:40.670670+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.13 deep-scrub ok 2021-03-15T22:23:41.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:41 smithi076 conmon[56514]: cluster 2021-03-15T22:23:41.063905+0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2021-03-15T22:23:41.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:41 smithi076 conmon[62156]: cluster 2021-03-15T22:23:40.199092+0000 osd.1 2021-03-15T22:23:41.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:41 smithi076 conmon[62156]: (osd.1) 67 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:23:41.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:41 smithi076 conmon[62156]: cluster 2021-03-15T22:23:40.670670+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.13 deep-scrub ok 2021-03-15T22:23:41.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:41 smithi076 conmon[62156]: cluster 2021-03-15T22:23:41.063905+0000 mon.a (mon.0) 1012 : cluster [DBG] osdmap e306: 8 total, 8 up, 8 in 2021-03-15T22:23:41.472 INFO:tasks.rados.rados.0.smithi076.stdout:1037: finishing write tid 2 to smithi07694878-23 2021-03-15T22:23:41.473 INFO:tasks.rados.rados.0.smithi076.stdout:1037: finishing write tid 3 to smithi07694878-23 2021-03-15T22:23:41.473 INFO:tasks.rados.rados.0.smithi076.stdout:1037: finishing write tid 4 to smithi07694878-23 2021-03-15T22:23:41.473 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 259 (ObjNum 380 snap 103 seq_num 380) dirty exists 2021-03-15T22:23:41.473 INFO:tasks.rados.rados.0.smithi076.stdout:1037: left oid 23 (ObjNum 380 snap 103 seq_num 380) 2021-03-15T22:23:41.474 INFO:tasks.rados.rados.0.smithi076.stdout:1039: finishing write tid 1 to smithi07694878-35 2021-03-15T22:23:41.474 INFO:tasks.rados.rados.0.smithi076.stdout:1040: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:23:41.474 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 0 (ObjNum 261765952 snap 21896 seq_num 0) dirty dne 2021-03-15T22:23:41.474 INFO:tasks.rados.rados.0.smithi076.stdout:1039: finishing write tid 2 to smithi07694878-35 2021-03-15T22:23:41.475 INFO:tasks.rados.rados.0.smithi076.stdout:1039: finishing write tid 3 to smithi07694878-35 2021-03-15T22:23:41.475 INFO:tasks.rados.rados.0.smithi076.stdout:1039: finishing write tid 4 to smithi07694878-35 2021-03-15T22:23:41.475 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 395 (ObjNum 381 snap 103 seq_num 381) dirty exists 2021-03-15T22:23:41.475 INFO:tasks.rados.rados.0.smithi076.stdout:1039: left oid 35 (ObjNum 381 snap 103 seq_num 381) 2021-03-15T22:23:41.476 INFO:tasks.rados.rados.0.smithi076.stdout:1041: finishing write tid 1 to smithi07694878-40 2021-03-15T22:23:41.476 INFO:tasks.rados.rados.0.smithi076.stdout:1041: finishing write tid 2 to smithi07694878-40 2021-03-15T22:23:41.476 INFO:tasks.rados.rados.0.smithi076.stdout:1041: finishing write tid 3 to smithi07694878-40 2021-03-15T22:23:41.476 INFO:tasks.rados.rados.0.smithi076.stdout:1041: finishing write tid 4 to smithi07694878-40 2021-03-15T22:23:41.476 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 622 (ObjNum 382 snap 103 seq_num 382) dirty exists 2021-03-15T22:23:41.478 INFO:tasks.rados.rados.0.smithi076.stdout:1041: left oid 40 (ObjNum 382 snap 103 seq_num 382) 2021-03-15T22:23:41.478 INFO:tasks.rados.rados.0.smithi076.stdout:1042: finishing write tid 1 to smithi07694878-2 2021-03-15T22:23:41.478 INFO:tasks.rados.rados.0.smithi076.stdout:1042: finishing write tid 2 to smithi07694878-2 2021-03-15T22:23:41.479 INFO:tasks.rados.rados.0.smithi076.stdout:1042: finishing write tid 3 to smithi07694878-2 2021-03-15T22:23:41.479 INFO:tasks.rados.rados.0.smithi076.stdout:1042: finishing write tid 4 to smithi07694878-2 2021-03-15T22:23:41.479 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 176 (ObjNum 383 snap 103 seq_num 383) dirty exists 2021-03-15T22:23:41.479 INFO:tasks.rados.rados.0.smithi076.stdout:1042: left oid 2 (ObjNum 383 snap 103 seq_num 383) 2021-03-15T22:23:41.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:41.518 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:41.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:41.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:41.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:41 smithi125 conmon[56739]: cluster 2021-03-15T22:23:40.199092+0000 osd.1 (osd.1) 67 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:23:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:41 smithi125 conmon[56739]: cluster 2021-03-15T22:23:40.670670+0000 osd.0 (osd.0) 64 : cluster [DBG] 6.13 deep-scrub ok 2021-03-15T22:23:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:41 smithi125 conmon[56739]: cluster 2021-03-15T22:23:41.063905+0000 mon.a (mon.0) 1012 : cluster 2021-03-15T22:23:41.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:41 smithi125 conmon[56739]: [DBG] osdmap e306: 8 total, 8 up, 8 in 2021-03-15T22:23:41.621 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:41.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:41.648 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:41.649 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:41.716 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:41.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:41.774 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:41.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:41.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:41.876 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:41.902 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:41.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:41.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:42.005 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:42.031 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:42.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:42.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:42.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:42.161 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:42.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:42.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:42.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[62156]: cluster 2021-03-15T22:23:41.027450+0000 mgr.y (mgr.14144) 531 : cluster [DBG] pgmap v585: 138 pgs: 5 peering, 133 active+clean; 120 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 984 KiB/s wr, 9 op/s 2021-03-15T22:23:42.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[62156]: cluster 2021-03-15T22:23:41.141213+0000 osd.2 (osd.2) 56 : cluster [DBG] 7.5 scrub ok 2021-03-15T22:23:42.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[62156]: cluster 2021-03-15T22:23:41.183105+0000 mon.a (mon.0) 1013 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2021-03-15T22:23:42.196 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[62156]: cluster 2021-03-15T22:23:41.449469+0000 osd.3 (osd.3) 59 : cluster [DBG] 3.15 deep-scrub ok 2021-03-15T22:23:42.197 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[62156]: audit 2021-03-15T22:23:41.818850+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.76:0/2178161705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:23:42.197 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[62156]: audit 2021-03-15T22:23:41.819189+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:23:42.202 INFO:tasks.rados.rados.0.smithi076.stdout:1037: done (6 left) 2021-03-15T22:23:42.203 INFO:tasks.rados.rados.0.smithi076.stdout:1038: done (5 left) 2021-03-15T22:23:42.203 INFO:tasks.rados.rados.0.smithi076.stdout:1039: done (4 left) 2021-03-15T22:23:42.203 INFO:tasks.rados.rados.0.smithi076.stdout:1040: done (3 left) 2021-03-15T22:23:42.203 INFO:tasks.rados.rados.0.smithi076.stdout:1041: done (2 left) 2021-03-15T22:23:42.204 INFO:tasks.rados.rados.0.smithi076.stdout:1042: done (1 left) 2021-03-15T22:23:42.204 INFO:tasks.rados.rados.0.smithi076.stdout:1043: done (0 left) 2021-03-15T22:23:42.204 INFO:tasks.rados.rados.0.smithi076.stdout:1044: write (excl) oid 23 current snap is 104 2021-03-15T22:23:42.204 INFO:tasks.rados.rados.0.smithi076.stdout:1044: seq_num 384 ranges {453888=713266,1737556=442800} 2021-03-15T22:23:42.208 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:23:42.209 INFO:tasks.rados.rados.0.smithi076.stdout:1044: writing smithi07694878-23 from 453888 to 1167154 tid 1 2021-03-15T22:23:42.214 INFO:tasks.rados.rados.0.smithi076.stdout:1044: writing smithi07694878-23 from 1737556 to 2180356 tid 2 2021-03-15T22:23:42.215 INFO:tasks.rados.rados.0.smithi076.stdout:1045: write oid 22 current snap is 104 2021-03-15T22:23:42.215 INFO:tasks.rados.rados.0.smithi076.stdout:1045: seq_num 385 ranges {470695=788515,1679685=586458} 2021-03-15T22:23:42.221 INFO:tasks.rados.rados.0.smithi076.stdout:1045: writing smithi07694878-22 from 470695 to 1259210 tid 1 2021-03-15T22:23:42.228 INFO:tasks.rados.rados.0.smithi076.stdout:1045: writing smithi07694878-22 from 1679685 to 2266143 tid 2 2021-03-15T22:23:42.228 INFO:tasks.rados.rados.0.smithi076.stdout:1046: read oid 10 snap 103 2021-03-15T22:23:42.228 INFO:tasks.rados.rados.0.smithi076.stdout:1046: expect (ObjNum 379 snap 103 seq_num 379) 2021-03-15T22:23:42.229 INFO:tasks.rados.rados.0.smithi076.stdout:1047: rollback oid 5 current snap is 104 2021-03-15T22:23:42.229 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 103 2021-03-15T22:23:42.229 INFO:tasks.rados.rados.0.smithi076.stdout:1044: finishing write tid 1 to smithi07694878-23 2021-03-15T22:23:42.229 INFO:tasks.rados.rados.0.smithi076.stdout:1044: finishing write tid 2 to smithi07694878-23 2021-03-15T22:23:42.229 INFO:tasks.rados.rados.0.smithi076.stdout:1044: finishing write tid 3 to smithi07694878-23 2021-03-15T22:23:42.230 INFO:tasks.rados.rados.0.smithi076.stdout:1044: finishing write tid 4 to smithi07694878-23 2021-03-15T22:23:42.230 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 262 (ObjNum 384 snap 104 seq_num 384) dirty exists 2021-03-15T22:23:42.230 INFO:tasks.rados.rados.0.smithi076.stdout:1044: left oid 23 (ObjNum 384 snap 104 seq_num 384) 2021-03-15T22:23:42.230 INFO:tasks.rados.rados.0.smithi076.stdout:1045: finishing write tid 1 to smithi07694878-22 2021-03-15T22:23:42.231 INFO:tasks.rados.rados.0.smithi076.stdout:1044: done (3 left) 2021-03-15T22:23:42.231 INFO:tasks.rados.rados.0.smithi076.stdout:1048: delete oid 15 current snap is 104 2021-03-15T22:23:42.232 INFO:tasks.rados.rados.0.smithi076.stdout:1047: finishing rollback tid 0 to smithi07694878-5 2021-03-15T22:23:42.232 INFO:tasks.rados.rados.0.smithi076.stdout:1047: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:23:42.233 INFO:tasks.rados.rados.0.smithi076.stdout:1047: finishing rollback tid 2 to smithi07694878-5 2021-03-15T22:23:42.233 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 144 (ObjNum 66 snap 6 seq_num 66) dirty exists 2021-03-15T22:23:42.233 INFO:tasks.rados.rados.0.smithi076.stdout:1045: finishing write tid 2 to smithi07694878-22 2021-03-15T22:23:42.233 INFO:tasks.rados.rados.0.smithi076.stdout:1045: finishing write tid 3 to smithi07694878-22 2021-03-15T22:23:42.234 INFO:tasks.rados.rados.0.smithi076.stdout:1045: finishing write tid 4 to smithi07694878-22 2021-03-15T22:23:42.234 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 575 (ObjNum 385 snap 104 seq_num 385) dirty exists 2021-03-15T22:23:42.235 INFO:tasks.rados.rados.0.smithi076.stdout:1045: left oid 22 (ObjNum 385 snap 104 seq_num 385) 2021-03-15T22:23:42.235 INFO:tasks.rados.rados.0.smithi076.stdout:1045: done (3 left) 2021-03-15T22:23:42.235 INFO:tasks.rados.rados.0.smithi076.stdout:1047: done (2 left) 2021-03-15T22:23:42.235 INFO:tasks.rados.rados.0.smithi076.stdout:1048: done (1 left) 2021-03-15T22:23:42.235 INFO:tasks.rados.rados.0.smithi076.stdout:1049: write oid 19 current snap is 104 2021-03-15T22:23:42.236 INFO:tasks.rados.rados.0.smithi076.stdout:1049: seq_num 387 ranges {504309=539013,1796649=590127} 2021-03-15T22:23:42.238 INFO:tasks.rados.rados.0.smithi076.stdout:1049: writing smithi07694878-19 from 504309 to 1043322 tid 1 2021-03-15T22:23:42.245 INFO:tasks.rados.rados.0.smithi076.stdout:1049: writing smithi07694878-19 from 1796649 to 2386776 tid 2 2021-03-15T22:23:42.246 INFO:tasks.rados.rados.0.smithi076.stdout:1050: snap_remove snap 102 2021-03-15T22:23:42.263 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:42.287 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:42.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:42.389 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:42.411 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:42.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:42.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:42.480 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[56514]: cluster 2021-03-15T22:23:41.027450+0000 mgr.y (mgr.14144) 531 : cluster [DBG] pgmap v585: 138 pgs: 5 peering, 133 active+clean; 120 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 984 KiB/s wr, 9 op/s 2021-03-15T22:23:42.480 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[56514]: cluster 2021-03-15T22:23:41.141213+0000 osd.2 (osd.2) 56 : cluster [DBG] 7.5 scrub ok 2021-03-15T22:23:42.480 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[56514]: cluster 2021-03-15T22:23:41.183105+0000 mon.a (mon.0) 1013 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2021-03-15T22:23:42.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[56514]: cluster 2021-03-15T22:23:41.449469+0000 osd.3 (osd.3) 59 : cluster [DBG] 3.15 deep-scrub ok 2021-03-15T22:23:42.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[56514]: audit 2021-03-15T22:23:41.818850+0000 mon.c (mon.1) 27 : audit [INF] from='client.? 172.21.15.76:0/2178161705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:23:42.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:42 smithi076 conmon[56514]: audit 2021-03-15T22:23:41.819189+0000 mon.a (mon.0) 1014 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:23:42.514 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:42.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:42.538 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:42.538 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:42.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: cluster 2021-03-15T22:23:41.027450+0000 mgr.y (mgr.14144) 531 : cluster [DBG] 2021-03-15T22:23:42.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: pgmap v585: 138 pgs: 5 peering, 133 active+clean; 120 MiB data, 464 MiB used, 715 GiB / 715 GiB avail; 3.8 MiB/s rd, 984 KiB/s wr, 9 op/s 2021-03-15T22:23:42.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: cluster 2021-03-15T22:23:41.141213+0000 osd.2 (osd.2) 56 : cluster [DBG] 7.5 scrub ok 2021-03-15T22:23:42.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: cluster 2021-03-15T22:23:41 2021-03-15T22:23:42.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: .183105+0000 mon.a (mon.0) 2021-03-15T22:23:42.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: 1013 : cluster [WRN] Health check failed: Reduced data availability: 3 pgs inactive, 4 pgs peering (PG_AVAILABILITY) 2021-03-15T22:23:42.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:23:42.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: 03-15T22:23:41.449469+0000 2021-03-15T22:23:42.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: osd.3 (osd.3) 59 : 2021-03-15T22:23:42.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: cluster [DBG] 3.15 deep-scrub ok 2021-03-15T22:23:42.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: audit 2021 2021-03-15T22:23:42.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: -03-15T22:23:41. 2021-03-15T22:23:42.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: 818850+0000 mon.c (mon.1 2021-03-15T22:23:42.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: ) 27 : audit [INF] from='client.? 172.21.15.76:0/2178161705' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:23:42.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: 2021-03-15T22:23:42.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:23:42.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: 03-15T22:23:41.819189 2021-03-15T22:23:42.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1014 2021-03-15T22:23:42.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:42 smithi125 conmon[56739]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:23:42.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:42.666 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:42.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:42.669 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:42.770 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:42.796 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:42.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:42.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:42.899 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:42.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:42.923 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:42.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:43.025 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:43.051 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:43.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:43.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:43.154 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:43.176 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:43.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:43.178 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:43.209 INFO:tasks.rados.rados.0.smithi076.stdout:1050: done (2 left) 2021-03-15T22:23:43.209 INFO:tasks.rados.rados.0.smithi076.stdout:1051: rollback oid 6 current snap is 104 2021-03-15T22:23:43.209 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 103 2021-03-15T22:23:43.209 INFO:tasks.rados.rados.0.smithi076.stdout:1046: expect (ObjNum 379 snap 103 seq_num 379) 2021-03-15T22:23:43.280 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:43.288 INFO:tasks.rados.rados.0.smithi076.stdout:1049: finishing write tid 1 to smithi07694878-19 2021-03-15T22:23:43.289 INFO:tasks.rados.rados.0.smithi076.stdout:1049: finishing write tid 2 to smithi07694878-19 2021-03-15T22:23:43.289 INFO:tasks.rados.rados.0.smithi076.stdout:1049: finishing write tid 3 to smithi07694878-19 2021-03-15T22:23:43.289 INFO:tasks.rados.rados.0.smithi076.stdout:1049: finishing write tid 4 to smithi07694878-19 2021-03-15T22:23:43.289 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 399 (ObjNum 387 snap 104 seq_num 387) dirty exists 2021-03-15T22:23:43.290 INFO:tasks.rados.rados.0.smithi076.stdout:1049: left oid 19 (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:23:43.290 INFO:tasks.rados.rados.0.smithi076.stdout:1046: done (2 left) 2021-03-15T22:23:43.290 INFO:tasks.rados.rados.0.smithi076.stdout:1049: done (1 left) 2021-03-15T22:23:43.290 INFO:tasks.rados.rados.0.smithi076.stdout:1052: write (excl) oid 25 current snap is 104 2021-03-15T22:23:43.290 INFO:tasks.rados.rados.0.smithi076.stdout:1051: finishing rollback tid 0 to smithi07694878-6 2021-03-15T22:23:43.291 INFO:tasks.rados.rados.0.smithi076.stdout:1051: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:23:43.291 INFO:tasks.rados.rados.0.smithi076.stdout:1051: finishing rollback tid 2 to smithi07694878-6 2021-03-15T22:23:43.291 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 269 (ObjNum 339 snap 91 seq_num 339) dirty exists 2021-03-15T22:23:43.291 INFO:tasks.rados.rados.0.smithi076.stdout:1052: seq_num 388 ranges {521116=730615,1971484=549632} 2021-03-15T22:23:43.294 INFO:tasks.rados.rados.0.smithi076.stdout:1052: writing smithi07694878-25 from 521116 to 1251731 tid 1 2021-03-15T22:23:43.303 INFO:tasks.rados.rados.0.smithi076.stdout:1052: writing smithi07694878-25 from 1971484 to 2521116 tid 2 2021-03-15T22:23:43.303 INFO:tasks.rados.rados.0.smithi076.stdout:1051: done (1 left) 2021-03-15T22:23:43.303 INFO:tasks.rados.rados.0.smithi076.stdout:1053: read oid 30 snap -1 2021-03-15T22:23:43.303 INFO:tasks.rados.rados.0.smithi076.stdout:1053: expect (ObjNum 378 snap 103 seq_num 378) 2021-03-15T22:23:43.304 INFO:tasks.rados.rados.0.smithi076.stdout:1054: write oid 10 current snap is 104 2021-03-15T22:23:43.304 INFO:tasks.rados.rados.0.smithi076.stdout:1052: finishing write tid 1 to smithi07694878-25 2021-03-15T22:23:43.304 INFO:tasks.rados.rados.0.smithi076.stdout:1054: seq_num 389 ranges {537923=405864,1513613=593796} 2021-03-15T22:23:43.310 INFO:tasks.rados.rados.0.smithi076.stdout:1054: writing smithi07694878-10 from 537923 to 943787 tid 1 2021-03-15T22:23:43.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:43.313 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:43.313 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:43.315 INFO:tasks.rados.rados.0.smithi076.stdout:1054: writing smithi07694878-10 from 1513613 to 2107409 tid 2 2021-03-15T22:23:43.315 INFO:tasks.rados.rados.0.smithi076.stdout:1055: copy_from oid 5 from oid 32 current snap is 104 2021-03-15T22:23:43.316 INFO:tasks.rados.rados.0.smithi076.stdout:1052: finishing write tid 2 to smithi07694878-25 2021-03-15T22:23:43.316 INFO:tasks.rados.rados.0.smithi076.stdout:1052: finishing write tid 3 to smithi07694878-25 2021-03-15T22:23:43.316 INFO:tasks.rados.rados.0.smithi076.stdout:1053: expect (ObjNum 378 snap 103 seq_num 378) 2021-03-15T22:23:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[56514]: cluster 2021-03-15T22:23:41.710650+0000 2021-03-15T22:23:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[56514]: osd.4 (osd.4) 72 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:23:43.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[56514]: cluster 2021-03-15T22:23:42.144817+0000 osd.2 (osd.2) 57 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:23:43.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[56514]: cluster 2021-03-15T22:23:42.177415+0000 osd.1 (osd.1) 68 : cluster [DBG] 5.c scrub ok 2021-03-15T22:23:43.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[56514]: cluster 2021-03-15T22:23:42.191528+0000 mon.a (mon.0) 1015 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:43.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[56514]: audit 2021-03-15T22:23:42.200349+0000 mon.a (mon.0) 1016 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:23:43.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[56514]: cluster 2021-03-15T22:23: 2021-03-15T22:23:43.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[56514]: 42.200417+0000 mon.a (mon.0) 1017 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2021-03-15T22:23:43.383 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[65511]: debug 2021-03-15T22:23:43.220+0000 7f1a1f877700 -1 osd.0 pg_epoch: 308 pg[7.6( v 308'579 (0'0,308'579] local-lis/les=175/176 n=9 ec=91/91 lis/c=175/175 les/c/f=176/176/0 sis=175) [0,7,5] r=0 lpr=175 luod=307'577 crt=308'579 lcod 307'575 mlcod 307'575 active+clean+snaptrim trimq=[cd~1] ps=[bd~1,c1~1,c8~2]] removing snap head 2021-03-15T22:23:43.383 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: cluster 2021-03-15T22:23:41. 2021-03-15T22:23:43.384 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: 710650+0000 osd.4 (osd.4) 72 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:23:43.384 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:23:43.384 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: 15T22:23:42.144817+0000 osd.2 (osd.2) 57 : cluster [DBG] 2021-03-15T22:23:43.384 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: 4.19 scrub ok 2021-03-15T22:23:43.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: cluster 2021-03-15T22:23:42 2021-03-15T22:23:43.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: .177415+0000 osd.1 (osd.1) 68 : cluster 2021-03-15T22:23:43.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: [DBG] 5.c scrub ok 2021-03-15T22:23:43.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:23:43.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: -03-15T22:23 2021-03-15T22:23:43.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: :42.191528+0000 mon.a ( 2021-03-15T22:23:43.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: mon.0) 1015 : cluster [WRN] 2021-03-15T22:23:43.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:43.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: audit 2021-03-15T22:23:43.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: 2021-03-15T22:23 2021-03-15T22:23:43.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: :42.200349+0000 mon.a (mon. 2021-03-15T22:23:43.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: 0) 1016 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:23:43.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: cluster 2021-03-15T22:23:43.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: 2021-03-15T22:23 2021-03-15T22:23:43.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: :42.200417+0000 mon.a (mon.0) 2021-03-15T22:23:43.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:43 smithi076 conmon[62156]: 1017 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2021-03-15T22:23:43.415 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:43.433 INFO:tasks.rados.rados.0.smithi076.stdout:1052: finishing write tid 4 to smithi07694878-25 2021-03-15T22:23:43.433 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 186 (ObjNum 388 snap 104 seq_num 388) dirty exists 2021-03-15T22:23:43.433 INFO:tasks.rados.rados.0.smithi076.stdout:1052: left oid 25 (ObjNum 388 snap 104 seq_num 388) 2021-03-15T22:23:43.433 INFO:tasks.rados.rados.0.smithi076.stdout:1054: finishing write tid 1 to smithi07694878-10 2021-03-15T22:23:43.434 INFO:tasks.rados.rados.0.smithi076.stdout:1054: finishing write tid 2 to smithi07694878-10 2021-03-15T22:23:43.434 INFO:tasks.rados.rados.0.smithi076.stdout:1054: finishing write tid 3 to smithi07694878-10 2021-03-15T22:23:43.434 INFO:tasks.rados.rados.0.smithi076.stdout:1054: finishing write tid 4 to smithi07694878-10 2021-03-15T22:23:43.434 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 190 (ObjNum 389 snap 104 seq_num 389) dirty exists 2021-03-15T22:23:43.434 INFO:tasks.rados.rados.0.smithi076.stdout:1054: left oid 10 (ObjNum 389 snap 104 seq_num 389) 2021-03-15T22:23:43.435 INFO:tasks.rados.rados.0.smithi076.stdout:1055: finishing copy_from to smithi07694878-5 2021-03-15T22:23:43.435 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 588 (ObjNum 371 snap 102 seq_num 371) dirty exists 2021-03-15T22:23:43.435 INFO:tasks.rados.rados.0.smithi076.stdout:1055: finishing copy_from racing read to smithi07694878-5 2021-03-15T22:23:43.435 INFO:tasks.rados.rados.0.smithi076.stdout:1052: done (3 left) 2021-03-15T22:23:43.436 INFO:tasks.rados.rados.0.smithi076.stdout:1053: done (2 left) 2021-03-15T22:23:43.436 INFO:tasks.rados.rados.0.smithi076.stdout:1054: done (1 left) 2021-03-15T22:23:43.436 INFO:tasks.rados.rados.0.smithi076.stdout:1055: done (0 left) 2021-03-15T22:23:43.436 INFO:tasks.rados.rados.0.smithi076.stdout:1056: read oid 30 snap -1 2021-03-15T22:23:43.437 INFO:tasks.rados.rados.0.smithi076.stdout:1056: expect (ObjNum 378 snap 103 seq_num 378) 2021-03-15T22:23:43.437 INFO:tasks.rados.rados.0.smithi076.stdout:1057: rollback oid 4 current snap is 104 2021-03-15T22:23:43.437 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 4 to 103 2021-03-15T22:23:43.437 INFO:tasks.rados.rados.0.smithi076.stdout:1058: snap_create 2021-03-15T22:23:43.437 INFO:tasks.rados.rados.0.smithi076.stdout:1057: finishing rollback tid 1 to smithi07694878-4 2021-03-15T22:23:43.438 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 0 (ObjNum 261995296 snap 21896 seq_num 260042768) dirty dne 2021-03-15T22:23:43.438 INFO:tasks.rados.rados.0.smithi076.stdout:1056: expect (ObjNum 378 snap 103 seq_num 378) 2021-03-15T22:23:43.441 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:43.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:43.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:43.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: cluster 2021-03-15T22:23:41.710650+0000 osd.4 (osd.4) 72 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:23:43.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: 2021-03-15T22:23:43.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: cluster 2021-03-15T22:23:42.144817+0000 osd.2 (osd.2) 57 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:23:43.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: cluster 2021-03-15T22:23:43.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: 2021-03-15T22:23:42.177415+0000 osd.1 (osd.1) 68 : cluster 2021-03-15T22:23:43.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: [DBG] 5.c scrub ok 2021-03-15T22:23:43.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:23:43.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: 22:23:42.191528+0000 mon.a (mon 2021-03-15T22:23:43.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: .0) 1015 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:43.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:23:43.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: 15T22:23:42.200349+0000 mon.a (mon.0) 2021-03-15T22:23:43.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: 1016 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:23:43.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:23:43.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: 15T22:23:42.200417+0000 mon.a (mon 2021-03-15T22:23:43.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:43 smithi125 conmon[56739]: .0) 1017 : cluster [DBG] osdmap e307: 8 total, 8 up, 8 in 2021-03-15T22:23:43.543 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:43.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:43.571 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:43.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:43.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:43.696 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:43.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:43.799 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:43.824 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:43.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:43.927 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:43.952 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:43.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:43.954 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:44.055 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:44.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:44.083 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:44.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:44.184 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:44.210 INFO:tasks.rados.rados.0.smithi076.stdout:1056: done (2 left) 2021-03-15T22:23:44.211 INFO:tasks.rados.rados.0.smithi076.stdout:1057: done (1 left) 2021-03-15T22:23:44.211 INFO:tasks.rados.rados.0.smithi076.stdout:1058: done (0 left) 2021-03-15T22:23:44.211 INFO:tasks.rados.rados.0.smithi076.stdout:1059: snap_create 2021-03-15T22:23:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:44.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:44.222 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:23:44.239 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:23:44.256 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:23:44.273 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:23:44.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:44 smithi076 conmon[56514]: cluster 2021-03-15T22:23:43.028061+0000 mgr.y (mgr.14144) 532 : cluster [DBG] pgmap v588: 138 pgs: 7 peering, 131 active+clean; 104 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s wr, 14 op/s 2021-03-15T22:23:44.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:44 smithi076 conmon[56514]: cluster 2021-03-15T22:23:43.205586+0000 mon.a (mon.0) 1018 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2021-03-15T22:23:44.281 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:44 smithi076 conmon[56514]: audit 2021-03-15T22:23:44.198772+0000 mon.a (mon.0) 1019 : audit [DBG] from='client.? 172.21.15.76:0/2605837752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:44.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:44 smithi076 conmon[62156]: cluster 2021-03-15T22:23:43.028061+0000 mgr.y (mgr.14144) 532 : cluster [DBG] pgmap v588: 138 pgs: 7 peering, 131 active+clean; 104 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s wr, 14 op/s 2021-03-15T22:23:44.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:44 smithi076 conmon[62156]: cluster 2021-03-15T22:23:43.205586+0000 mon.a (mon.0) 1018 : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2021-03-15T22:23:44.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:44 smithi076 conmon[62156]: audit 2021-03-15T22:23:44.198772+0000 mon.a (mon.0) 1019 : audit [DBG] from='client.? 172.21.15.76:0/2605837752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:44.293 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:23:44.312 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:23:44.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:44.333 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:23:44.340 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:44.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:44.341 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:44.352 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:23:44.353 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:23:44.353 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:23:44.353 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:23:44.354 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:23:44.354 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:23:44.354 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:23:44.354 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:23:44.354 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:23:44.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:44.469 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:44.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:44.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:44 smithi125 conmon[56739]: cluster 2021-03-15T22:23:43.028061+0000 mgr.y 2021-03-15T22:23:44.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:44 smithi125 conmon[56739]: (mgr.14144) 532 : cluster [DBG] pgmap v588: 138 pgs: 7 peering, 131 active+clean; 104 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 3.5 MiB/s wr, 14 op/s 2021-03-15T22:23:44.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:44 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:23:44.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:44 smithi125 conmon[56739]: T22:23:43.205586+0000 mon.a (mon.0) 1018 2021-03-15T22:23:44.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:44 smithi125 conmon[56739]: : cluster [DBG] osdmap e308: 8 total, 8 up, 8 in 2021-03-15T22:23:44.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:44 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:23:44.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:44 smithi125 conmon[56739]: 03-15T22:23:44.198772 2021-03-15T22:23:44.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:44 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1019 2021-03-15T22:23:44.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:44 smithi125 conmon[56739]: : audit [DBG] from='client.? 172.21.15.76:0/2605837752' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:44.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:44.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:44.599 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:44.599 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:44.700 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:44.722 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:44.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:44.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:44.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:44.847 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:44.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:44.849 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:44.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:44.953 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:23:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:23:44.953 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:23:44.954 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:23:44.954 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:23:44.971 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:44.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:44.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:45.066 INFO:tasks.rados.rados.0.smithi076.stdout:1059: done (0 left) 2021-03-15T22:23:45.066 INFO:tasks.rados.rados.0.smithi076.stdout:1060: write oid 14 current snap is 106 2021-03-15T22:23:45.067 INFO:tasks.rados.rados.0.smithi076.stdout:1060: seq_num 390 ranges {554730=481113,1572095=737454} 2021-03-15T22:23:45.071 INFO:tasks.rados.rados.0.smithi076.stdout:1060: writing smithi07694878-14 from 554730 to 1035843 tid 1 2021-03-15T22:23:45.074 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:45.079 INFO:tasks.rados.rados.0.smithi076.stdout:1060: writing smithi07694878-14 from 1572095 to 2309549 tid 2 2021-03-15T22:23:45.079 INFO:tasks.rados.rados.0.smithi076.stdout:1061: write (excl) oid 31 current snap is 106 2021-03-15T22:23:45.079 INFO:tasks.rados.rados.0.smithi076.stdout:1061: seq_num 391 ranges {571537=556362,1630577=597465} 2021-03-15T22:23:45.085 INFO:tasks.rados.rados.0.smithi076.stdout:1061: writing smithi07694878-31 from 571537 to 1127899 tid 1 2021-03-15T22:23:45.092 INFO:tasks.rados.rados.0.smithi076.stdout:1061: writing smithi07694878-31 from 1630577 to 2228042 tid 2 2021-03-15T22:23:45.092 INFO:tasks.rados.rados.0.smithi076.stdout:1062: copy_from oid 41 from oid 17 current snap is 106 2021-03-15T22:23:45.092 INFO:tasks.rados.rados.0.smithi076.stdout:1060: finishing write tid 1 to smithi07694878-14 2021-03-15T22:23:45.092 INFO:tasks.rados.rados.0.smithi076.stdout:1060: finishing write tid 2 to smithi07694878-14 2021-03-15T22:23:45.092 INFO:tasks.rados.rados.0.smithi076.stdout:1060: finishing write tid 3 to smithi07694878-14 2021-03-15T22:23:45.093 INFO:tasks.rados.rados.0.smithi076.stdout:1060: finishing write tid 4 to smithi07694878-14 2021-03-15T22:23:45.093 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 297 (ObjNum 390 snap 106 seq_num 390) dirty exists 2021-03-15T22:23:45.093 INFO:tasks.rados.rados.0.smithi076.stdout:1060: left oid 14 (ObjNum 390 snap 106 seq_num 390) 2021-03-15T22:23:45.093 INFO:tasks.rados.rados.0.smithi076.stdout:1061: finishing write tid 1 to smithi07694878-31 2021-03-15T22:23:45.094 INFO:tasks.rados.rados.0.smithi076.stdout:1060: done (2 left) 2021-03-15T22:23:45.094 INFO:tasks.rados.rados.0.smithi076.stdout:1063: write oid 43 current snap is 106 2021-03-15T22:23:45.094 INFO:tasks.rados.rados.0.smithi076.stdout:1063: seq_num 392 ranges {588344=631611,1689059=741123} 2021-03-15T22:23:45.098 INFO:tasks.rados.rados.0.smithi076.stdout:1063: writing smithi07694878-43 from 588344 to 1219955 tid 1 2021-03-15T22:23:45.103 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:45.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:45.106 INFO:tasks.rados.rados.0.smithi076.stdout:1063: writing smithi07694878-43 from 1689059 to 2430182 tid 2 2021-03-15T22:23:45.106 INFO:tasks.rados.rados.0.smithi076.stdout:1064: read oid 38 snap -1 2021-03-15T22:23:45.106 INFO:tasks.rados.rados.0.smithi076.stdout:1064: expect (ObjNum 273 snap 72 seq_num 273) 2021-03-15T22:23:45.107 INFO:tasks.rados.rados.0.smithi076.stdout:1065: snap_remove snap 105 2021-03-15T22:23:45.205 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:45.230 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:45.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:45.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:45.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:45.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:45.358 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:45.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:45.406 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:23:45.460 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:45.484 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:45.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:45.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:45.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:45 smithi076 conmon[56514]: cluster 2021-03-15T22:23:44.207526+0000 mon.a (mon.0) 1020 : 2021-03-15T22:23:45.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:45 smithi076 conmon[56514]: cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2021-03-15T22:23:45.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:45 smithi076 conmon[56514]: cluster 2021-03-15T22:23:45.062254+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2021-03-15T22:23:45.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:45 smithi076 conmon[62156]: cluster 2021-03-15T22:23:44. 2021-03-15T22:23:45.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:45 smithi076 conmon[62156]: 207526+0000 mon.a (mon.0) 1020 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2021-03-15T22:23:45.488 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:45 smithi076 conmon[62156]: cluster 2021-03-15T22:23:45.062254+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2021-03-15T22:23:45.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:45 smithi125 conmon[56739]: cluster 2021-03-15T22:23:44.207526+0000 2021-03-15T22:23:45.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:45 smithi125 conmon[56739]: mon.a (mon.0) 1020 : cluster [DBG] osdmap e309: 8 total, 8 up, 8 in 2021-03-15T22:23:45.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:45 smithi125 conmon[56739]: cluster 2021-03-15T22:23:45.062254+0000 mon.a (mon.0) 1021 : cluster [DBG] osdmap e310: 8 total, 8 up, 8 in 2021-03-15T22:23:45.586 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:45.611 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:45.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:45.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:45.692 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-15T22:23:45.694 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:45.714 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:45.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:45.742 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:45.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:45.844 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:45.867 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:45.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:45.869 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:45.970 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:45.992 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:45.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:45.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:46.076 INFO:tasks.rados.rados.0.smithi076.stdout:1065: done (4 left) 2021-03-15T22:23:46.077 INFO:tasks.rados.rados.0.smithi076.stdout:1066: write oid 15 current snap is 106 2021-03-15T22:23:46.077 INFO:tasks.rados.rados.0.smithi076.stdout:1061: finishing write tid 2 to smithi07694878-31 2021-03-15T22:23:46.077 INFO:tasks.rados.rados.0.smithi076.stdout:1066: seq_num 393 ranges {605151=706860,2031188=573963} 2021-03-15T22:23:46.083 INFO:tasks.rados.rados.0.smithi076.stdout:1066: writing smithi07694878-15 from 605151 to 1312011 tid 1 2021-03-15T22:23:46.091 INFO:tasks.rados.rados.0.smithi076.stdout:1066: writing smithi07694878-15 from 2031188 to 2605151 tid 2 2021-03-15T22:23:46.091 INFO:tasks.rados.rados.0.smithi076.stdout:1067: rollback oid 49 current snap is 106 2021-03-15T22:23:46.092 INFO:tasks.rados.rados.0.smithi076.stdout:1061: finishing write tid 3 to smithi07694878-31 2021-03-15T22:23:46.092 INFO:tasks.rados.rados.0.smithi076.stdout:1061: finishing write tid 4 to smithi07694878-31 2021-03-15T22:23:46.092 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 165 (ObjNum 391 snap 106 seq_num 391) dirty exists 2021-03-15T22:23:46.092 INFO:tasks.rados.rados.0.smithi076.stdout:1061: left oid 31 (ObjNum 391 snap 106 seq_num 391) 2021-03-15T22:23:46.092 INFO:tasks.rados.rados.0.smithi076.stdout:1062: finishing copy_from to smithi07694878-41 2021-03-15T22:23:46.093 INFO:tasks.rados.rados.0.smithi076.stdout:1062: got expected ENOENT (src dne) 2021-03-15T22:23:46.093 INFO:tasks.rados.rados.0.smithi076.stdout:1062: finishing copy_from racing read to smithi07694878-41 2021-03-15T22:23:46.093 INFO:tasks.rados.rados.0.smithi076.stdout:1063: finishing write tid 1 to smithi07694878-43 2021-03-15T22:23:46.093 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 49 to 104 2021-03-15T22:23:46.094 INFO:tasks.rados.rados.0.smithi076.stdout:1064: expect (ObjNum 273 snap 72 seq_num 273) 2021-03-15T22:23:46.095 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:46.116 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:46.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:46.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:46.213 INFO:tasks.rados.rados.0.smithi076.stdout:1061: done (5 left) 2021-03-15T22:23:46.213 INFO:tasks.rados.rados.0.smithi076.stdout:1062: done (4 left) 2021-03-15T22:23:46.214 INFO:tasks.rados.rados.0.smithi076.stdout:1064: done (3 left) 2021-03-15T22:23:46.214 INFO:tasks.rados.rados.0.smithi076.stdout:1068: delete oid 50 current snap is 106 2021-03-15T22:23:46.214 INFO:tasks.rados.rados.0.smithi076.stdout:1063: finishing write tid 2 to smithi07694878-43 2021-03-15T22:23:46.214 INFO:tasks.rados.rados.0.smithi076.stdout:1063: finishing write tid 3 to smithi07694878-43 2021-03-15T22:23:46.215 INFO:tasks.rados.rados.0.smithi076.stdout:1063: finishing write tid 4 to smithi07694878-43 2021-03-15T22:23:46.215 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 265 (ObjNum 392 snap 106 seq_num 392) dirty exists 2021-03-15T22:23:46.215 INFO:tasks.rados.rados.0.smithi076.stdout:1063: left oid 43 (ObjNum 392 snap 106 seq_num 392) 2021-03-15T22:23:46.215 INFO:tasks.rados.rados.0.smithi076.stdout:1066: finishing write tid 1 to smithi07694878-15 2021-03-15T22:23:46.215 INFO:tasks.rados.rados.0.smithi076.stdout:1066: finishing write tid 2 to smithi07694878-15 2021-03-15T22:23:46.216 INFO:tasks.rados.rados.0.smithi076.stdout:1066: finishing write tid 3 to smithi07694878-15 2021-03-15T22:23:46.216 INFO:tasks.rados.rados.0.smithi076.stdout:1066: finishing write tid 4 to smithi07694878-15 2021-03-15T22:23:46.216 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 588 (ObjNum 393 snap 106 seq_num 393) dirty exists 2021-03-15T22:23:46.216 INFO:tasks.rados.rados.0.smithi076.stdout:1066: left oid 15 (ObjNum 393 snap 106 seq_num 393) 2021-03-15T22:23:46.216 INFO:tasks.rados.rados.0.smithi076.stdout:1067: finishing rollback tid 0 to smithi07694878-49 2021-03-15T22:23:46.217 INFO:tasks.rados.rados.0.smithi076.stdout:1067: finishing rollback tid 1 to smithi07694878-49 2021-03-15T22:23:46.217 INFO:tasks.rados.rados.0.smithi076.stdout:1067: finishing rollback tid 2 to smithi07694878-49 2021-03-15T22:23:46.217 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 592 (ObjNum 359 snap 96 seq_num 359) dirty exists 2021-03-15T22:23:46.218 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:46.218 INFO:tasks.rados.rados.0.smithi076.stdout:1063: done (3 left) 2021-03-15T22:23:46.219 INFO:tasks.rados.rados.0.smithi076.stdout:1066: done (2 left) 2021-03-15T22:23:46.219 INFO:tasks.rados.rados.0.smithi076.stdout:1067: done (1 left) 2021-03-15T22:23:46.219 INFO:tasks.rados.rados.0.smithi076.stdout:1068: done (0 left) 2021-03-15T22:23:46.219 INFO:tasks.rados.rados.0.smithi076.stdout:1069: copy_from oid 48 from oid 45 current snap is 106 2021-03-15T22:23:46.220 INFO:tasks.rados.rados.0.smithi076.stdout:1070: read oid 50 snap -1 2021-03-15T22:23:46.220 INFO:tasks.rados.rados.0.smithi076.stdout:1070: expect deleted 2021-03-15T22:23:46.220 INFO:tasks.rados.rados.0.smithi076.stdout:1071: snap_remove snap 103 2021-03-15T22:23:46.220 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:46.247 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:46.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:46.249 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:46.310 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:23:46.311 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:23:46.311 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 1.000000 2021-03-15T22:23:46.312 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd primary-affinity 7 1 2021-03-15T22:23:46.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:46.382 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:46.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:46.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:46.485 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:46.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:46.508 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:46.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:46.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56514]: cluster 2021-03-15T22:23:45.028510+0000 mgr.y (mgr.14144) 533 : cluster [DBG] pgmap v591: 138 pgs: 7 peering, 131 active+clean; 102 MiB data, 436 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s wr, 10 op/s 2021-03-15T22:23:46.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56514]: cluster 2021-03-15T22:23:46.074954+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2021-03-15T22:23:46.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[62156]: cluster 2021-03-15T22:23:45.028510+0000 mgr.y (mgr.14144) 533 : cluster [DBG] pgmap v591: 138 pgs: 7 peering, 131 active+clean; 102 MiB data, 436 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s wr, 10 op/s 2021-03-15T22:23:46.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[62156]: cluster 2021-03-15T22:23:46.074954+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2021-03-15T22:23:46.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:46 smithi125 conmon[56739]: cluster 2021-03-15T22:23:45.028510+0000 mgr.y (mgr.14144) 533 : cluster [DBG] pgmap v591: 138 pgs: 7 peering, 131 active+clean; 102 MiB data, 436 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s wr, 10 op/s 2021-03-15T22:23:46.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:46 smithi125 conmon[56739]: cluster 2021-03-15T22:23:46.074954+0000 mon.a (mon.0) 1022 : cluster [DBG] osdmap e311: 8 total, 8 up, 8 in 2021-03-15T22:23:46.610 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:46.632 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:46.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:46.634 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:46.735 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:46.756 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:46.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:46.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:46.842 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:46.859 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:46.872 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: debug 2021-03-15T22:23:46.565+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:23:46.873 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:23:46.873 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:46.873 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:23:46.873 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:46.874 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:23:46.874 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:46.874 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:46.874 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:23:46.875 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: debug 2021-03-15T22:23:46.566+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:23:46.875 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:23:46.875 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:46.875 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:23:46.876 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:46.876 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:23:46.876 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:46.876 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:46.877 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:23:46.877 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: debug 2021-03-15T22:23:46.566+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:23:46.877 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:23:46.878 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:46.878 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:23:46.878 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:46.878 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:23:46.879 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:46.879 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:46.879 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:23:46.879 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: debug 2021-03-15T22:23:46.566+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:23:46.880 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:23:46.880 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:46.880 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:23:46.880 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:46.880 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:23:46.881 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:46.881 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:46.881 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:23:46.881 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: debug 2021-03-15T22:23:46.567+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:23:46.882 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:23:46.882 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:46.882 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:23:46.883 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:46.883 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:23:46.883 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:46.884 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:46.885 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:23:46.885 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: debug 2021-03-15T22:23:46.567+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:23:46.885 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:23:46.886 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:46.886 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:23:46.886 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:46.886 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:23:46.887 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:46.887 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:46.887 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:23:46.887 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: debug 2021-03-15T22:23:46.568+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:23:46.888 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:23:46.888 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:46.888 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:23:46.888 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:46.889 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:23:46.889 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:46.889 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:46.889 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:23:46 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:23:46.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:46.892 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:46.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:46.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:47.016 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:47.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:47.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:47.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:47.141 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:47.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:47.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:47.236 INFO:tasks.rados.rados.0.smithi076.stdout:1071: done (2 left) 2021-03-15T22:23:47.237 INFO:tasks.rados.rados.0.smithi076.stdout:1072: read oid 4 snap -1 2021-03-15T22:23:47.237 INFO:tasks.rados.rados.0.smithi076.stdout:1072: expect deleted 2021-03-15T22:23:47.237 INFO:tasks.rados.rados.0.smithi076.stdout:1073: write (excl) oid 44 current snap is 106 2021-03-15T22:23:47.237 INFO:tasks.rados.rados.0.smithi076.stdout:1069: finishing copy_from to smithi07694878-48 2021-03-15T22:23:47.238 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 263 (ObjNum 332 snap 88 seq_num 332) dirty exists 2021-03-15T22:23:47.238 INFO:tasks.rados.rados.0.smithi076.stdout:1069: finishing copy_from racing read to smithi07694878-48 2021-03-15T22:23:47.238 INFO:tasks.rados.rados.0.smithi076.stdout:1073: seq_num 395 ranges {638765=457358,1748152=604803} 2021-03-15T22:23:47.242 INFO:tasks.rados.rados.0.smithi076.stdout:1073: writing smithi07694878-44 from 638765 to 1096123 tid 1 2021-03-15T22:23:47.248 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:47.251 INFO:tasks.rados.rados.0.smithi076.stdout:1073: writing smithi07694878-44 from 1748152 to 2352955 tid 2 2021-03-15T22:23:47.252 INFO:tasks.rados.rados.0.smithi076.stdout:1069: done (3 left) 2021-03-15T22:23:47.252 INFO:tasks.rados.rados.0.smithi076.stdout:1070: done (2 left) 2021-03-15T22:23:47.252 INFO:tasks.rados.rados.0.smithi076.stdout:1074: write oid 7 current snap is 106 2021-03-15T22:23:47.252 INFO:tasks.rados.rados.0.smithi076.stdout:1074: seq_num 396 ranges {655572=648960,1922987=732585} 2021-03-15T22:23:47.256 INFO:tasks.rados.rados.0.smithi076.stdout:1074: writing smithi07694878-7 from 655572 to 1304532 tid 1 2021-03-15T22:23:47.266 INFO:tasks.rados.rados.0.smithi076.stdout:1074: writing smithi07694878-7 from 1922987 to 2655572 tid 2 2021-03-15T22:23:47.266 INFO:tasks.rados.rados.0.smithi076.stdout:1075: read oid 43 snap -1 2021-03-15T22:23:47.267 INFO:tasks.rados.rados.0.smithi076.stdout:1075: expect (ObjNum 392 snap 106 seq_num 392) 2021-03-15T22:23:47.268 INFO:tasks.rados.rados.0.smithi076.stdout:1076: snap_create 2021-03-15T22:23:47.268 INFO:tasks.rados.rados.0.smithi076.stdout:1073: finishing write tid 1 to smithi07694878-44 2021-03-15T22:23:47.268 INFO:tasks.rados.rados.0.smithi076.stdout:1073: finishing write tid 2 to smithi07694878-44 2021-03-15T22:23:47.268 INFO:tasks.rados.rados.0.smithi076.stdout:1073: finishing write tid 3 to smithi07694878-44 2021-03-15T22:23:47.269 INFO:tasks.rados.rados.0.smithi076.stdout:1073: finishing write tid 4 to smithi07694878-44 2021-03-15T22:23:47.269 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 215 (ObjNum 395 snap 106 seq_num 395) dirty exists 2021-03-15T22:23:47.269 INFO:tasks.rados.rados.0.smithi076.stdout:1073: left oid 44 (ObjNum 395 snap 106 seq_num 395) 2021-03-15T22:23:47.270 INFO:tasks.rados.rados.0.smithi076.stdout:1074: finishing write tid 1 to smithi07694878-7 2021-03-15T22:23:47.271 INFO:tasks.rados.rados.0.smithi076.stdout:1074: finishing write tid 2 to smithi07694878-7 2021-03-15T22:23:47.271 INFO:tasks.rados.rados.0.smithi076.stdout:1074: finishing write tid 3 to smithi07694878-7 2021-03-15T22:23:47.272 INFO:tasks.rados.rados.0.smithi076.stdout:1074: finishing write tid 4 to smithi07694878-7 2021-03-15T22:23:47.272 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 410 (ObjNum 396 snap 106 seq_num 396) dirty exists 2021-03-15T22:23:47.272 INFO:tasks.rados.rados.0.smithi076.stdout:1074: left oid 7 (ObjNum 396 snap 106 seq_num 396) 2021-03-15T22:23:47.272 INFO:tasks.rados.rados.0.smithi076.stdout:1075: expect (ObjNum 392 snap 106 seq_num 392) 2021-03-15T22:23:47.279 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:47.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:47.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:47.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:47.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:47.407 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:47.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:47.430 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: audit 2021-03-15T22:23:46.268878+0000 mon.a ( 2021-03-15T22:23:47.431 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: mon.0) 1023 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:23:47.431 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:23:46.561794+0000 mgr.y (mgr.14144) 534 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:23:47.431 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: audit 2021-03-15T22:23:46.566211+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:23:47.432 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:23:46.567264+0000 mgr.y (mgr.14144) 535 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:23:47.432 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:23:47.432 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.432 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:23:47.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:23:47.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:23:47.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:23:46.567727+0000 mgr.y (mgr.14144) 536 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:23:47.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:23:47.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:23:47.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:23:47.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.436 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:23:47.436 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:23:46.568171+0000 mgr.y (mgr.14144) 537 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:23:47.436 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:23:47.436 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: audit 2021-03-15T22:23:47.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:23:47.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: 23:46.268878+0000 mon.a (mon.0) 1023 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:23:47.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: cephadm 2021-03-15T 2021-03-15T22:23:47.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: 22:23:46.561794+0000 mgr.y (mgr.14144 2021-03-15T22:23:47.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: ) 534 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:23:47.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: audit 2021-03-15T22:23:47.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: 2021-03-15T22:23:46.566211 2021-03-15T22:23:47.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1024 : audit [INF] 2021-03-15T22:23:47.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:23:47.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:23:46.567264+0000 mgr.y (mgr.14144) 535 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:23:47.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:23:47.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:23:47.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:23:47.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:23:47.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:23:46.567727+0000 mgr.y (mgr.14144) 536 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:23:47.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:23:47.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:23:47.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:23:47.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.443 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.443 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:23:47.443 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:23:46.568171+0000 mgr.y ( 2021-03-15T22:23:47.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:23:47.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:23:47.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:23:47.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:23:46.568648+0000 mgr.y (mgr.14144) 538 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:23:47.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:23:47.445 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.446 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:23:47.446 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.446 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:23:47.446 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:23:47.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:23:46.569044+0000 mgr.y (mgr.14144) 539 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:23:47.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:23:47.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:23:47.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:23:47.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.449 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:23:47.449 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:23:46.569388+0000 mgr.y (mgr.14144) 540 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:23:47.449 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:23:47.449 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:23:47.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:23:47.450 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.451 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: mgr.14144) 537 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:23:47.451 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:23:47.451 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.452 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:23:47.452 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.452 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:23:47.452 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.452 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.453 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:23:47.453 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:23:46.568648+0000 mgr.y (mgr.14144) 538 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:23:47.453 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:23:47.453 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:23:47.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:23:47.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:23:47.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:23: 2021-03-15T22:23:47.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: 46.569044+0000 mgr.y (mgr.14144) 539 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:23:47.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:23:47.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:23:47.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:23:47.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:23:47.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:23:46.569388+0000 mgr.y (mgr.14144) 540 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:23:47.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:23:47.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:23:47.458 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.459 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:23:47.459 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: cephadm 2021-03-15T22:23:46.569728 2021-03-15T22:23:47.459 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 541 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:23:47.459 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:23:47.460 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.460 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:23:47.460 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.460 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:23:47.460 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.461 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.461 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:23:47.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:23:47.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:23:47.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: cephadm 2021-03-15T22:23:46.569728+0000 mgr.y (mgr.14144) 541 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:23:47.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:23:47.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:23:47.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:23:47.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:47 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:23:47.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:23:47.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: audit 2021-03-15T22:23:46.268878+0000 mon.a (mon.0) 1023 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:23:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:23:46.561794+0000 mgr.y (mgr.14144) 534 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:23:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:23:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: 03-15T22:23:46.566211+0000 mon.a (mon.0) 1024 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:23:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: cephadm 2021-03-15T22: 2021-03-15T22:23:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: 23:46.567264+0000 mgr.y (mgr.14144) 535 : cephadm [ERR] 2021-03-15T22:23:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:23:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:23:47.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:23:47.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:23:47.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:23:47.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:23:47.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: 2021-03-15T22:23:46.567727+0000 mgr.y (mgr.14144) 536 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:23:47.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:23:47.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:23:47.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:23:47.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:23:47.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:23:46.568171+0000 mgr.y (mgr.14144) 537 : cephadm [ERR] 2021-03-15T22:23:47.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:23:47.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:23:47.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:23:47.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:23:47.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:23:47.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:23:46.568648+0000 mgr.y (mgr.14144) 538 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:23:47.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:23:47.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:23:47.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:23:47.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:23:47.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:23:46.569044+0000 2021-03-15T22:23:47.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: mgr.y (mgr.14144) 539 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:23:47.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:23:47.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:23:47.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:23:47.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:23:47.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:23:46.569388+0000 mgr.y (mgr.14144) 540 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:23:47.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:23:47.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:23:47.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:23:47.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:23:47.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: cephadm 2021-03-15T22:23:46.569728+0000 mgr.y (mgr.14144) 541 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:23:47.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:23:47.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:23:47.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:23:47.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:23:47.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:23:47.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:23:47.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:23:47.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:47 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:23:47.557 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:47.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:47.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:47.660 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:47.682 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:47.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:47.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:47.785 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:47.807 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:47.809 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:47.910 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:47.931 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:47.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:47.933 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:48.034 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:48.055 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:48.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:48.158 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:48.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:48.183 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:48.238 INFO:tasks.rados.rados.0.smithi076.stdout:1072: done (4 left) 2021-03-15T22:23:48.238 INFO:tasks.rados.rados.0.smithi076.stdout:1073: done (3 left) 2021-03-15T22:23:48.239 INFO:tasks.rados.rados.0.smithi076.stdout:1074: done (2 left) 2021-03-15T22:23:48.239 INFO:tasks.rados.rados.0.smithi076.stdout:1075: done (1 left) 2021-03-15T22:23:48.239 INFO:tasks.rados.rados.0.smithi076.stdout:1076: done (0 left) 2021-03-15T22:23:48.239 INFO:tasks.rados.rados.0.smithi076.stdout:1077: snap_remove snap 104 2021-03-15T22:23:48.284 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:48.315 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:48.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:48.317 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:48.418 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:48.444 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:48.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:48.445 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:48 smithi076 conmon[56514]: cluster 2021-03-15T22:23:47.028965+0000 mgr.y (mgr.14144) 542 : cluster [DBG] pgmap v594: 138 pgs: 138 active+clean; 98 MiB data, 439 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2021-03-15T22:23:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:48 smithi076 conmon[56514]: cluster 2021-03-15T22:23:47.228422+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2021-03-15T22:23:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:48 smithi076 conmon[56514]: cluster 2021-03-15T22:23:47.563582+0000 mon.a (mon.0) 1026 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2021-03-15T22:23:48.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:48 smithi076 conmon[62156]: cluster 2021-03-15T22:23:47.028965+0000 mgr.y (mgr.14144) 542 : cluster [DBG] pgmap v594: 138 pgs: 138 active+clean; 98 MiB data, 439 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2021-03-15T22:23:48.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:48 smithi076 conmon[62156]: cluster 2021-03-15T22:23:47.228422+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2021-03-15T22:23:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:48 smithi076 conmon[62156]: cluster 2021-03-15T22:23:47.563582+0000 mon.a (mon.0) 1026 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2021-03-15T22:23:48.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:48 smithi125 conmon[56739]: cluster 2021-03-15T22:23:47.028965+0000 mgr.y (mgr.14144) 542 : cluster [DBG] pgmap v594: 138 pgs: 138 active+clean; 98 MiB data, 439 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 22 op/s 2021-03-15T22:23:48.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:48 smithi125 conmon[56739]: 2021-03-15T22:23:48.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:48 smithi125 conmon[56739]: cluster 2021-03-15T22:23:47.228422+0000 mon.a (mon.0) 1025 : cluster [DBG] osdmap e312: 8 total, 8 up, 8 in 2021-03-15T22:23:48.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:48 smithi125 conmon[56739]: cluster 2021-03-15T22:23:47.563582+0000 mon.a (mon.0) 1026 : cluster [INF] 2021-03-15T22:23:48.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:48 smithi125 conmon[56739]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 6 pgs peering) 2021-03-15T22:23:48.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:48.573 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:48.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:48.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:48.675 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:48.702 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:48.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:48.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:48.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:48.830 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:48.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:48.927 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:48.932 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:48.957 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:48.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:48.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:49.060 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:49.086 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:49.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:49.088 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:49.189 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:49.216 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:49.216 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:49.244 INFO:tasks.rados.rados.0.smithi076.stdout:1077: done (0 left) 2021-03-15T22:23:49.244 INFO:tasks.rados.rados.0.smithi076.stdout:1078: snap_create 2021-03-15T22:23:49.246 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:23:49.257 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:23:49.258 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:23:49.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:23:49.258 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:23:49.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.259 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:323 7.8 7:1d74a1c3:::smithi07694878-44:head [copy-from ver 3] snapc 8=[8] ondisk+write+known_if_redirected e104)", 2021-03-15T22:23:49.259 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:19.443283+0000", 2021-03-15T22:23:49.259 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 209.80990101899999, 2021-03-15T22:23:49.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015830329000000001, 2021-03-15T22:23:49.260 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.260 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.260 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.260 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.260 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.261 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 323 2021-03-15T22:23:49.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.261 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443283+0000", 2021-03-15T22:23:49.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443283+0000", 2021-03-15T22:23:49.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1140000000000002e-06 2021-03-15T22:23:49.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443286+0000", 2021-03-15T22:23:49.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2950000000000004e-06 2021-03-15T22:23:49.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443290+0000", 2021-03-15T22:23:49.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4205999999999999e-05 2021-03-15T22:23:49.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443304+0000", 2021-03-15T22:23:49.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5860000000000001e-06 2021-03-15T22:23:49.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443311+0000", 2021-03-15T22:23:49.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3111000000000001e-05 2021-03-15T22:23:49.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443354+0000", 2021-03-15T22:23:49.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2724000000000003e-05 2021-03-15T22:23:49.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443437+0000", 2021-03-15T22:23:49.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00847615 2021-03-15T22:23:49.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:23:49.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.451913+0000", 2021-03-15T22:23:49.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0062090670000000004 2021-03-15T22:23:49.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:23:49.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.458122+0000", 2021-03-15T22:23:49.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00073570700000000005 2021-03-15T22:23:49.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.458858+0000", 2021-03-15T22:23:49.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000171175 2021-03-15T22:23:49.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459029+0000", 2021-03-15T22:23:49.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2721e-05 2021-03-15T22:23:49.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459072+0000", 2021-03-15T22:23:49.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1473000000000001e-05 2021-03-15T22:23:49.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459113+0000", 2021-03-15T22:23:49.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015830329000000001 2021-03-15T22:23:49.277 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.277 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.278 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.278 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:324 7.8 7:1d74a1c3:::smithi07694878-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e104)", 2021-03-15T22:23:49.278 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:19.443323+0000", 2021-03-15T22:23:49.279 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 209.80986044900001, 2021-03-15T22:23:49.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01594897, 2021-03-15T22:23:49.279 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.279 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:49.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.280 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 324 2021-03-15T22:23:49.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.281 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443323+0000", 2021-03-15T22:23:49.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443323+0000", 2021-03-15T22:23:49.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2889999999999997e-06 2021-03-15T22:23:49.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443329+0000", 2021-03-15T22:23:49.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6029999999999999e-06 2021-03-15T22:23:49.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443330+0000", 2021-03-15T22:23:49.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1019999999999999e-06 2021-03-15T22:23:49.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443337+0000", 2021-03-15T22:23:49.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2849999999999996e-06 2021-03-15T22:23:49.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443342+0000", 2021-03-15T22:23:49.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025426499999999999 2021-03-15T22:23:49.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443596+0000", 2021-03-15T22:23:49.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7543999999999997e-05 2021-03-15T22:23:49.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:23:49.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.443643+0000", 2021-03-15T22:23:49.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010718 2021-03-15T22:23:49.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.454361+0000", 2021-03-15T22:23:49.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000111568 2021-03-15T22:23:49.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:23:49.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.454473+0000", 2021-03-15T22:23:49.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0046729909999999996 2021-03-15T22:23:49.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459146+0000", 2021-03-15T22:23:49.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3789000000000003e-05 2021-03-15T22:23:49.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459200+0000", 2021-03-15T22:23:49.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2533999999999994e-05 2021-03-15T22:23:49.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:19.459272+0000", 2021-03-15T22:23:49.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01594897 2021-03-15T22:23:49.296 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.297 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.297 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.297 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:531 7.8 7:1822d391:::smithi07694878-47:head [copy-from ver 14] snapc 19=[19,18,17] ondisk+write+known_if_redirected e122)", 2021-03-15T22:23:49.298 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:37.008160+0000", 2021-03-15T22:23:49.298 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 192.245023774, 2021-03-15T22:23:49.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029706733999999999, 2021-03-15T22:23:49.298 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.299 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.299 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.299 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.299 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.299 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 531 2021-03-15T22:23:49.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.300 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.301 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008160+0000", 2021-03-15T22:23:49.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.301 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008160+0000", 2021-03-15T22:23:49.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4279999999999999e-06 2021-03-15T22:23:49.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008162+0000", 2021-03-15T22:23:49.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9100000000000001e-06 2021-03-15T22:23:49.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008165+0000", 2021-03-15T22:23:49.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.204e-05 2021-03-15T22:23:49.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.306 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.306 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008177+0000", 2021-03-15T22:23:49.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2616000000000001e-05 2021-03-15T22:23:49.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008190+0000", 2021-03-15T22:23:49.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000128796 2021-03-15T22:23:49.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008319+0000", 2021-03-15T22:23:49.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000103112 2021-03-15T22:23:49.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.310 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.310 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008422+0000", 2021-03-15T22:23:49.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016673884 2021-03-15T22:23:49.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:23:49.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.025096+0000", 2021-03-15T22:23:49.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053861949999999999 2021-03-15T22:23:49.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:23:49.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.030482+0000", 2021-03-15T22:23:49.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071629290000000002 2021-03-15T22:23:49.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.313 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.037645+0000", 2021-03-15T22:23:49.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012967600000000001 2021-03-15T22:23:49.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.037775+0000", 2021-03-15T22:23:49.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9626999999999999e-05 2021-03-15T22:23:49.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.037814+0000", 2021-03-15T22:23:49.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2521000000000001e-05 2021-03-15T22:23:49.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.037867+0000", 2021-03-15T22:23:49.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029706733999999999 2021-03-15T22:23:49.318 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.318 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.319 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.319 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:532 7.8 7:1822d391:::smithi07694878-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e122)", 2021-03-15T22:23:49.320 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:37.008263+0000", 2021-03-15T22:23:49.320 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 192.24492072999999, 2021-03-15T22:23:49.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029847118999999998, 2021-03-15T22:23:49.321 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.321 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:49.321 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.321 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.322 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.322 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 532 2021-03-15T22:23:49.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.322 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008263+0000", 2021-03-15T22:23:49.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008263+0000", 2021-03-15T22:23:49.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.542e-06 2021-03-15T22:23:49.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008265+0000", 2021-03-15T22:23:49.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8369999999999999e-06 2021-03-15T22:23:49.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008266+0000", 2021-03-15T22:23:49.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0428000000000001e-05 2021-03-15T22:23:49.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008277+0000", 2021-03-15T22:23:49.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7709e-05 2021-03-15T22:23:49.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008325+0000", 2021-03-15T22:23:49.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025799800000000001 2021-03-15T22:23:49.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008583+0000", 2021-03-15T22:23:49.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8608e-05 2021-03-15T22:23:49.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:23:49.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.008621+0000", 2021-03-15T22:23:49.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.021177707 2021-03-15T22:23:49.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.029799+0000", 2021-03-15T22:23:49.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4729999999999996e-05 2021-03-15T22:23:49.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:23:49.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.029894+0000", 2021-03-15T22:23:49.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080190699999999997 2021-03-15T22:23:49.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.037913+0000", 2021-03-15T22:23:49.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7991999999999994e-05 2021-03-15T22:23:49.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.038001+0000", 2021-03-15T22:23:49.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000109498 2021-03-15T22:23:49.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:37.038110+0000", 2021-03-15T22:23:49.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029847118999999998 2021-03-15T22:23:49.337 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.337 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.337 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.338 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:539 7.2 e123/91 7:45ee0f2f:::smithi07694878-6:head v 123'14, mlcod=104'10)", 2021-03-15T22:23:49.338 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:38.116403+0000", 2021-03-15T22:23:49.338 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 191.13678106200001, 2021-03-15T22:23:49.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010556181, 2021-03-15T22:23:49.339 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.339 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.339 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.116403+0000", 2021-03-15T22:23:49.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.116403+0000", 2021-03-15T22:23:49.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001998086 2021-03-15T22:23:49.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118401+0000", 2021-03-15T22:23:49.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4809999999999997e-06 2021-03-15T22:23:49.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118406+0000", 2021-03-15T22:23:49.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3042999999999999e-05 2021-03-15T22:23:49.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118429+0000", 2021-03-15T22:23:49.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0678e-05 2021-03-15T22:23:49.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118440+0000", 2021-03-15T22:23:49.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4852999999999997e-05 2021-03-15T22:23:49.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118505+0000", 2021-03-15T22:23:49.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6299999999999999e-05 2021-03-15T22:23:49.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.118531+0000", 2021-03-15T22:23:49.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0083770519999999994 2021-03-15T22:23:49.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.126908+0000", 2021-03-15T22:23:49.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0688e-05 2021-03-15T22:23:49.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.126959+0000", 2021-03-15T22:23:49.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010556181 2021-03-15T22:23:49.352 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.352 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.352 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.353 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:836 7.3 e149/91 7:c458c800:::smithi07694878-3:head v 149'65, mlcod=148'61)", 2021-03-15T22:23:49.353 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.030244+0000", 2021-03-15T22:23:49.354 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 165.22294001099999, 2021-03-15T22:23:49.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013182968, 2021-03-15T22:23:49.354 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.354 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.355 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.030244+0000", 2021-03-15T22:23:49.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.030244+0000", 2021-03-15T22:23:49.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036198279999999998 2021-03-15T22:23:49.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.033864+0000", 2021-03-15T22:23:49.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4490000000000002e-06 2021-03-15T22:23:49.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.033871+0000", 2021-03-15T22:23:49.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2189999999999999e-05 2021-03-15T22:23:49.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.033893+0000", 2021-03-15T22:23:49.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.91e-06 2021-03-15T22:23:49.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.033903+0000", 2021-03-15T22:23:49.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00096714999999999995 2021-03-15T22:23:49.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.034870+0000", 2021-03-15T22:23:49.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1061999999999999e-05 2021-03-15T22:23:49.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.034891+0000", 2021-03-15T22:23:49.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085107499999999992 2021-03-15T22:23:49.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043402+0000", 2021-03-15T22:23:49.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4629000000000001e-05 2021-03-15T22:23:49.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043427+0000", 2021-03-15T22:23:49.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013182968 2021-03-15T22:23:49.366 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.366 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.366 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.367 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1576 7.3 7:c458c800:::smithi07694878-3:6a [sparse-read 0~3899191,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e204)", 2021-03-15T22:23:49.367 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.066256+0000", 2021-03-15T22:23:49.367 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 110.186927563, 2021-03-15T22:23:49.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.05921051, 2021-03-15T22:23:49.368 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.368 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:49.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.369 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.369 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1576 2021-03-15T22:23:49.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.369 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066256+0000", 2021-03-15T22:23:49.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066256+0000", 2021-03-15T22:23:49.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8909999999999997e-06 2021-03-15T22:23:49.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066260+0000", 2021-03-15T22:23:49.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9709999999999998e-06 2021-03-15T22:23:49.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066264+0000", 2021-03-15T22:23:49.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6415000000000001e-05 2021-03-15T22:23:49.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066280+0000", 2021-03-15T22:23:49.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5329999999999997e-06 2021-03-15T22:23:49.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066288+0000", 2021-03-15T22:23:49.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6039999999999999e-05 2021-03-15T22:23:49.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066344+0000", 2021-03-15T22:23:49.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6260000000000007e-06 2021-03-15T22:23:49.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:23:49.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066354+0000", 2021-03-15T22:23:49.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0541787069999999 2021-03-15T22:23:49.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.120532+0000", 2021-03-15T22:23:49.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014794499999999999 2021-03-15T22:23:49.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.120680+0000", 2021-03-15T22:23:49.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0047863819999999996 2021-03-15T22:23:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125467+0000", 2021-03-15T22:23:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.05921051 2021-03-15T22:23:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1577 7.3 7:c458c800:::smithi07694878-3:6a [read 0~3899191,checksum] snapc 0=[] ondisk+read+known_if_redirected e204)", 2021-03-15T22:23:49.384 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.066317+0000", 2021-03-15T22:23:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 110.186866835, 2021-03-15T22:23:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0607751919999999, 2021-03-15T22:23:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.385 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1577 2021-03-15T22:23:49.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066317+0000", 2021-03-15T22:23:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066317+0000", 2021-03-15T22:23:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5050000000000002e-06 2021-03-15T22:23:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066319+0000", 2021-03-15T22:23:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6270000000000001e-06 2021-03-15T22:23:49.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066321+0000", 2021-03-15T22:23:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2679999999999997e-06 2021-03-15T22:23:49.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066328+0000", 2021-03-15T22:23:49.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.313e-06 2021-03-15T22:23:49.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066333+0000", 2021-03-15T22:23:49.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9864000000000002e-05 2021-03-15T22:23:49.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066382+0000", 2021-03-15T22:23:49.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2140000000000001e-06 2021-03-15T22:23:49.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:23:49.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066389+0000", 2021-03-15T22:23:49.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0591109620000001 2021-03-15T22:23:49.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125500+0000", 2021-03-15T22:23:49.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00074530899999999999 2021-03-15T22:23:49.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.126245+0000", 2021-03-15T22:23:49.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084712999999999998 2021-03-15T22:23:49.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127092+0000", 2021-03-15T22:23:49.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0607751919999999 2021-03-15T22:23:49.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.401 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.401 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.402 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1578 7.3 7:c458c800:::smithi07694878-3:6a [sparse-read 0~3899191] snapc 0=[] ondisk+read+known_if_redirected e204)", 2021-03-15T22:23:49.402 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.066339+0000", 2021-03-15T22:23:49.402 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 110.186845182, 2021-03-15T22:23:49.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0611799630000001, 2021-03-15T22:23:49.402 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.403 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:49.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.403 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.404 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1578 2021-03-15T22:23:49.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.404 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066339+0000", 2021-03-15T22:23:49.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066339+0000", 2021-03-15T22:23:49.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.356e-06 2021-03-15T22:23:49.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066340+0000", 2021-03-15T22:23:49.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1090000000000001e-06 2021-03-15T22:23:49.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066341+0000", 2021-03-15T22:23:49.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2199999999999997e-06 2021-03-15T22:23:49.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066347+0000", 2021-03-15T22:23:49.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2209999999999996e-06 2021-03-15T22:23:49.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066352+0000", 2021-03-15T22:23:49.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4710999999999998e-05 2021-03-15T22:23:49.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066407+0000", 2021-03-15T22:23:49.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0290000000000005e-06 2021-03-15T22:23:49.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:23:49.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.066415+0000", 2021-03-15T22:23:49.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.060719011 2021-03-15T22:23:49.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127134+0000", 2021-03-15T22:23:49.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013284599999999999 2021-03-15T22:23:49.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127267+0000", 2021-03-15T22:23:49.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025146000000000001 2021-03-15T22:23:49.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127519+0000", 2021-03-15T22:23:49.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0611799630000001 2021-03-15T22:23:49.418 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.419 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.419 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.420 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:61 7.5 e205/91 7:a9545d62:::smithi07694878-1:68 v 205'72, mlcod=202'69)", 2021-03-15T22:23:49.420 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.090032+0000", 2021-03-15T22:23:49.420 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 109.163151345, 2021-03-15T22:23:49.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034649245000000002, 2021-03-15T22:23:49.420 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.421 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.421 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090032+0000", 2021-03-15T22:23:49.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090032+0000", 2021-03-15T22:23:49.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.835e-06 2021-03-15T22:23:49.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.424 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.424 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090034+0000", 2021-03-15T22:23:49.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.75e-06 2021-03-15T22:23:49.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.425 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090036+0000", 2021-03-15T22:23:49.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3450000000000002e-06 2021-03-15T22:23:49.425 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.426 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090042+0000", 2021-03-15T22:23:49.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0460000000000002e-06 2021-03-15T22:23:49.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090048+0000", 2021-03-15T22:23:49.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4586999999999999e-05 2021-03-15T22:23:49.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090083+0000", 2021-03-15T22:23:49.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4909000000000001e-05 2021-03-15T22:23:49.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090098+0000", 2021-03-15T22:23:49.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034543644999999998 2021-03-15T22:23:49.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.124641+0000", 2021-03-15T22:23:49.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0127999999999999e-05 2021-03-15T22:23:49.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.124682+0000", 2021-03-15T22:23:49.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034649245000000002 2021-03-15T22:23:49.435 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.435 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.435 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.436 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1684 7.8 7:1d74a1c3:::smithi07694878-44:head [copy-from ver 83] snapc 7a=[7a,79,78,76,75] ondisk+write+known_if_redirected e220)", 2021-03-15T22:23:49.436 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:15.039221+0000", 2021-03-15T22:23:49.436 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 94.213962666, 2021-03-15T22:23:49.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026953068, 2021-03-15T22:23:49.437 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.437 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.437 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.437 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.437 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.438 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1684 2021-03-15T22:23:49.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.438 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039221+0000", 2021-03-15T22:23:49.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039221+0000", 2021-03-15T22:23:49.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7960000000000001e-06 2021-03-15T22:23:49.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039227+0000", 2021-03-15T22:23:49.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1250000000000001e-06 2021-03-15T22:23:49.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039232+0000", 2021-03-15T22:23:49.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8726999999999999e-05 2021-03-15T22:23:49.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039251+0000", 2021-03-15T22:23:49.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6390000000000004e-06 2021-03-15T22:23:49.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039260+0000", 2021-03-15T22:23:49.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2811000000000005e-05 2021-03-15T22:23:49.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039333+0000", 2021-03-15T22:23:49.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8609000000000004e-05 2021-03-15T22:23:49.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039432+0000", 2021-03-15T22:23:49.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013565288 2021-03-15T22:23:49.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:23:49.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.052997+0000", 2021-03-15T22:23:49.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050078329999999997 2021-03-15T22:23:49.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:23:49.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.058005+0000", 2021-03-15T22:23:49.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080478589999999992 2021-03-15T22:23:49.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066053+0000", 2021-03-15T22:23:49.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.239e-05 2021-03-15T22:23:49.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066085+0000", 2021-03-15T22:23:49.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8930999999999997e-05 2021-03-15T22:23:49.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066124+0000", 2021-03-15T22:23:49.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0059999999999998e-05 2021-03-15T22:23:49.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066174+0000", 2021-03-15T22:23:49.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026953068 2021-03-15T22:23:49.453 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.453 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.454 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.454 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1685 7.8 7:1d74a1c3:::smithi07694878-44:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e220)", 2021-03-15T22:23:49.454 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:15.039276+0000", 2021-03-15T22:23:49.455 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 94.213907320000004, 2021-03-15T22:23:49.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027064135999999999, 2021-03-15T22:23:49.455 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.456 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:49.457 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.457 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.457 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.458 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1685 2021-03-15T22:23:49.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.458 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039276+0000", 2021-03-15T22:23:49.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039276+0000", 2021-03-15T22:23:49.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4589999999999999e-06 2021-03-15T22:23:49.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039278+0000", 2021-03-15T22:23:49.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5710000000000001e-06 2021-03-15T22:23:49.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039279+0000", 2021-03-15T22:23:49.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2530000000000004e-06 2021-03-15T22:23:49.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039285+0000", 2021-03-15T22:23:49.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.236e-06 2021-03-15T22:23:49.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039288+0000", 2021-03-15T22:23:49.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028498599999999998 2021-03-15T22:23:49.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039573+0000", 2021-03-15T22:23:49.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8646000000000001e-05 2021-03-15T22:23:49.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:23:49.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039622+0000", 2021-03-15T22:23:49.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017618592999999998 2021-03-15T22:23:49.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.057240+0000", 2021-03-15T22:23:49.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2138999999999995e-05 2021-03-15T22:23:49.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:23:49.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.057312+0000", 2021-03-15T22:23:49.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088990739999999999 2021-03-15T22:23:49.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066211+0000", 2021-03-15T22:23:49.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9696999999999994e-05 2021-03-15T22:23:49.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066281+0000", 2021-03-15T22:23:49.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9481999999999997e-05 2021-03-15T22:23:49.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066341+0000", 2021-03-15T22:23:49.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027064135999999999 2021-03-15T22:23:49.474 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.475 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.475 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1719 7.a 7:58400cda:::smithi07694878-39:head [copy-from ver 242] snapc 7d=[7d,76,75] ondisk+write+known_if_redirected e224)", 2021-03-15T22:23:49.476 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:19.079943+0000", 2021-03-15T22:23:49.476 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 90.173240402000005, 2021-03-15T22:23:49.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014513622, 2021-03-15T22:23:49.476 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.476 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.477 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.477 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.477 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.477 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1719 2021-03-15T22:23:49.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.478 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079943+0000", 2021-03-15T22:23:49.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079943+0000", 2021-03-15T22:23:49.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.669e-06 2021-03-15T22:23:49.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079945+0000", 2021-03-15T22:23:49.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3740000000000001e-06 2021-03-15T22:23:49.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079946+0000", 2021-03-15T22:23:49.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7610000000000007e-06 2021-03-15T22:23:49.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079954+0000", 2021-03-15T22:23:49.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1049999999999997e-06 2021-03-15T22:23:49.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079958+0000", 2021-03-15T22:23:49.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017872400000000001 2021-03-15T22:23:49.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.081745+0000", 2021-03-15T22:23:49.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1965000000000001e-05 2021-03-15T22:23:49.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.081797+0000", 2021-03-15T22:23:49.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026574139999999999 2021-03-15T22:23:49.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,5", 2021-03-15T22:23:49.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.084455+0000", 2021-03-15T22:23:49.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092122469999999998 2021-03-15T22:23:49.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:23:49.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.093667+0000", 2021-03-15T22:23:49.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025727399999999999 2021-03-15T22:23:49.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.093924+0000", 2021-03-15T22:23:49.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000441159 2021-03-15T22:23:49.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.094366+0000", 2021-03-15T22:23:49.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2379000000000002e-05 2021-03-15T22:23:49.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.094408+0000", 2021-03-15T22:23:49.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9035000000000001e-05 2021-03-15T22:23:49.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.094457+0000", 2021-03-15T22:23:49.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014513622 2021-03-15T22:23:49.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.493 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.493 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.494 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1720 7.a 7:58400cda:::smithi07694878-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e224)", 2021-03-15T22:23:49.494 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:19.079963+0000", 2021-03-15T22:23:49.495 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 90.173220649000001, 2021-03-15T22:23:49.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014649005, 2021-03-15T22:23:49.495 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.495 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:23:49.496 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.496 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.496 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.496 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1720 2021-03-15T22:23:49.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.497 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079963+0000", 2021-03-15T22:23:49.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079963+0000", 2021-03-15T22:23:49.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5099999999999998e-07 2021-03-15T22:23:49.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079964+0000", 2021-03-15T22:23:49.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1510000000000001e-06 2021-03-15T22:23:49.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079965+0000", 2021-03-15T22:23:49.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9059999999999996e-06 2021-03-15T22:23:49.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079972+0000", 2021-03-15T22:23:49.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2910000000000001e-06 2021-03-15T22:23:49.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.079976+0000", 2021-03-15T22:23:49.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019319649999999999 2021-03-15T22:23:49.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.081908+0000", 2021-03-15T22:23:49.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6480000000000003e-05 2021-03-15T22:23:49.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:23:49.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.081945+0000", 2021-03-15T22:23:49.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063457019999999999 2021-03-15T22:23:49.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.088291+0000", 2021-03-15T22:23:49.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6920000000000003e-05 2021-03-15T22:23:49.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:23:49.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.088357+0000", 2021-03-15T22:23:49.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061291200000000001 2021-03-15T22:23:49.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.094487+0000", 2021-03-15T22:23:49.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9545e-05 2021-03-15T22:23:49.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.094546+0000", 2021-03-15T22:23:49.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5974000000000002e-05 2021-03-15T22:23:49.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:19.094612+0000", 2021-03-15T22:23:49.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014649005 2021-03-15T22:23:49.511 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.511 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.511 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.512 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2219 7.3 7:c00fa989:::smithi07694878-50:head [rollback 88] snapc a4=[a2,a0,9e,9c,93,8d,88,7d] ondisk+write+known_if_redirected e263)", 2021-03-15T22:23:49.512 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.049377+0000", 2021-03-15T22:23:49.512 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 51.203806325999999, 2021-03-15T22:23:49.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.812799944, 2021-03-15T22:23:49.513 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.513 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.513 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.513 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.514 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.514 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2219 2021-03-15T22:23:49.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.514 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049377+0000", 2021-03-15T22:23:49.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049377+0000", 2021-03-15T22:23:49.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.187e-06 2021-03-15T22:23:49.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049379+0000", 2021-03-15T22:23:49.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2899999999999999e-06 2021-03-15T22:23:49.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049380+0000", 2021-03-15T22:23:49.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9710000000000001e-06 2021-03-15T22:23:49.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049386+0000", 2021-03-15T22:23:49.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1799999999999998e-06 2021-03-15T22:23:49.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049390+0000", 2021-03-15T22:23:49.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00081442199999999998 2021-03-15T22:23:49.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050204+0000", 2021-03-15T22:23:49.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010783 2021-03-15T22:23:49.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050312+0000", 2021-03-15T22:23:49.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3077000000000003e-05 2021-03-15T22:23:49.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for degraded object", 2021-03-15T22:23:49.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050405+0000", 2021-03-15T22:23:49.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.810425357 2021-03-15T22:23:49.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.860831+0000", 2021-03-15T22:23:49.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8446000000000001e-05 2021-03-15T22:23:49.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.860909+0000", 2021-03-15T22:23:49.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017099300000000001 2021-03-15T22:23:49.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,4", 2021-03-15T22:23:49.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.861080+0000", 2021-03-15T22:23:49.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000866646 2021-03-15T22:23:49.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.861947+0000", 2021-03-15T22:23:49.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3630000000000002e-05 2021-03-15T22:23:49.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.529 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.861980+0000", 2021-03-15T22:23:49.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6860000000000006e-06 2021-03-15T22:23:49.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:23:49.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.861988+0000", 2021-03-15T22:23:49.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4487999999999996e-05 2021-03-15T22:23:49.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.862053+0000", 2021-03-15T22:23:49.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012474099999999999 2021-03-15T22:23:49.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.862177+0000", 2021-03-15T22:23:49.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.812799944 2021-03-15T22:23:49.532 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.533 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.533 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.533 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2220 7.3 7:c00fa989:::smithi07694878-50:head [append 0~0] snapc a4=[a2,a0,9e,9c,93,8d,88,7d] ondisk+write+known_if_redirected e263)", 2021-03-15T22:23:49.534 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.049402+0000", 2021-03-15T22:23:49.534 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 51.203782160000003, 2021-03-15T22:23:49.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.813034257, 2021-03-15T22:23:49.534 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.535 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.535 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:23:49.535 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:23:49.535 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:23:49.535 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2220 2021-03-15T22:23:49.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.536 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049402+0000", 2021-03-15T22:23:49.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049402+0000", 2021-03-15T22:23:49.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0419999999999999e-06 2021-03-15T22:23:49.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049403+0000", 2021-03-15T22:23:49.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.325e-06 2021-03-15T22:23:49.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049404+0000", 2021-03-15T22:23:49.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6149999999999997e-06 2021-03-15T22:23:49.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049413+0000", 2021-03-15T22:23:49.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0169999999999998e-06 2021-03-15T22:23:49.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.049417+0000", 2021-03-15T22:23:49.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00104385 2021-03-15T22:23:49.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050460+0000", 2021-03-15T22:23:49.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2867999999999994e-05 2021-03-15T22:23:49.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for degraded object", 2021-03-15T22:23:49.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050533+0000", 2021-03-15T22:23:49.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8109451110000001 2021-03-15T22:23:49.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.861478+0000", 2021-03-15T22:23:49.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9259999999999998e-05 2021-03-15T22:23:49.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.861548+0000", 2021-03-15T22:23:49.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018158900000000001 2021-03-15T22:23:49.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,4", 2021-03-15T22:23:49.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.861729+0000", 2021-03-15T22:23:49.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00046329200000000002 2021-03-15T22:23:49.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:23:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.862193+0000", 2021-03-15T22:23:49.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0434999999999999e-05 2021-03-15T22:23:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.862263+0000", 2021-03-15T22:23:49.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6708999999999997e-05 2021-03-15T22:23:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:23:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.862300+0000", 2021-03-15T22:23:49.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0667999999999998e-05 2021-03-15T22:23:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.862340+0000", 2021-03-15T22:23:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5476000000000005e-05 2021-03-15T22:23:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.862436+0000", 2021-03-15T22:23:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.813034257 2021-03-15T22:23:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.3.0:403 7.2 e265/264 7:42f18a95:::smithi07694878-45:89 v 265'122, mlcod=0'0)", 2021-03-15T22:23:49.554 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:00.067459+0000", 2021-03-15T22:23:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 49.185724309000001, 2021-03-15T22:23:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011973859999999999, 2021-03-15T22:23:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.555 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.067459+0000", 2021-03-15T22:23:49.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.067459+0000", 2021-03-15T22:23:49.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4420000000000001e-06 2021-03-15T22:23:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.067467+0000", 2021-03-15T22:23:49.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.091e-06 2021-03-15T22:23:49.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.067472+0000", 2021-03-15T22:23:49.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3249000000000001e-05 2021-03-15T22:23:49.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.067495+0000", 2021-03-15T22:23:49.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8909999999999995e-06 2021-03-15T22:23:49.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.563 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.563 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.067503+0000", 2021-03-15T22:23:49.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011288919999999999 2021-03-15T22:23:49.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.564 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.564 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.078792+0000", 2021-03-15T22:23:49.564 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3769999999999997e-05 2021-03-15T22:23:49.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.565 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.078826+0000", 2021-03-15T22:23:49.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00056356099999999997 2021-03-15T22:23:49.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.566 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.079389+0000", 2021-03-15T22:23:49.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3936000000000001e-05 2021-03-15T22:23:49.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.567 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.079433+0000", 2021-03-15T22:23:49.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011973859999999999 2021-03-15T22:23:49.568 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.568 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.568 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.569 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:2313 7.5 e266/91 7:ac84e656:::smithi07694878-31:head v 266'107, mlcod=261'103)", 2021-03-15T22:23:49.569 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.069549+0000", 2021-03-15T22:23:49.569 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 48.183634339999998, 2021-03-15T22:23:49.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012906040000000001, 2021-03-15T22:23:49.570 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.570 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.570 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.069549+0000", 2021-03-15T22:23:49.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.069549+0000", 2021-03-15T22:23:49.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063343640000000003 2021-03-15T22:23:49.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.075884+0000", 2021-03-15T22:23:49.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.658e-06 2021-03-15T22:23:49.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.075887+0000", 2021-03-15T22:23:49.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8571e-05 2021-03-15T22:23:49.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.075936+0000", 2021-03-15T22:23:49.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7749999999999998e-06 2021-03-15T22:23:49.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.576 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.075942+0000", 2021-03-15T22:23:49.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2556000000000003e-05 2021-03-15T22:23:49.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.577 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.075994+0000", 2021-03-15T22:23:49.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6999e-05 2021-03-15T22:23:49.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.578 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.578 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.076021+0000", 2021-03-15T22:23:49.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063726989999999999 2021-03-15T22:23:49.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.082394+0000", 2021-03-15T22:23:49.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1418e-05 2021-03-15T22:23:49.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.082455+0000", 2021-03-15T22:23:49.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012906040000000001 2021-03-15T22:23:49.581 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.581 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.581 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.582 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.582 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:2406 7.5 e276/91 7:a9545d62:::smithi07694878-1:head v 276'113, mlcod=270'109)", 2021-03-15T22:23:49.582 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:11.087737+0000", 2021-03-15T22:23:49.582 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 38.165446934999999, 2021-03-15T22:23:49.582 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013664020000000001, 2021-03-15T22:23:49.583 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.583 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.583 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.583 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.583 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.584 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:11.087737+0000", 2021-03-15T22:23:49.584 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.585 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.585 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:11.087737+0000", 2021-03-15T22:23:49.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010095200000000001 2021-03-15T22:23:49.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.585 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.586 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.586 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:11.088746+0000", 2021-03-15T22:23:49.586 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3799999999999998e-06 2021-03-15T22:23:49.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.587 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:11.088750+0000", 2021-03-15T22:23:49.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0538e-05 2021-03-15T22:23:49.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:11.088760+0000", 2021-03-15T22:23:49.588 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0449999999999997e-06 2021-03-15T22:23:49.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.590 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.590 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:11.088766+0000", 2021-03-15T22:23:49.590 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4097999999999995e-05 2021-03-15T22:23:49.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.591 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.591 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:11.088830+0000", 2021-03-15T22:23:49.591 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0205000000000001e-05 2021-03-15T22:23:49.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.592 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:11.088851+0000", 2021-03-15T22:23:49.592 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011070489999999999 2021-03-15T22:23:49.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:11.089958+0000", 2021-03-15T22:23:49.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011443185 2021-03-15T22:23:49.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:11.101401+0000", 2021-03-15T22:23:49.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013664020000000001 2021-03-15T22:23:49.595 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.595 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.596 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.596 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:2783 7.c e311/175 7:37eb566e:::smithi07694878-48:head v 311'243, mlcod=304'239)", 2021-03-15T22:23:49.596 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:46.222335+0000", 2021-03-15T22:23:49.597 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.030848416, 2021-03-15T22:23:49.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010620465000000001, 2021-03-15T22:23:49.597 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:23:49.597 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:23:49.598 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:23:49.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:23:49.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:46.222335+0000", 2021-03-15T22:23:49.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:23:49.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.599 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:23:49.599 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:46.222335+0000", 2021-03-15T22:23:49.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066548550000000003 2021-03-15T22:23:49.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:23:49.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:46.228990+0000", 2021-03-15T22:23:49.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.329e-06 2021-03-15T22:23:49.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:23:49.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:46.228999+0000", 2021-03-15T22:23:49.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2097000000000001e-05 2021-03-15T22:23:49.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:23:49.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:46.229032+0000", 2021-03-15T22:23:49.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1629e-05 2021-03-15T22:23:49.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:23:49.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:46.229043+0000", 2021-03-15T22:23:49.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0951000000000006e-05 2021-03-15T22:23:49.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:23:49.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:46.229114+0000", 2021-03-15T22:23:49.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7847e-05 2021-03-15T22:23:49.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:23:49.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:46.229142+0000", 2021-03-15T22:23:49.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003766985 2021-03-15T22:23:49.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:23:49.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:46.232909+0000", 2021-03-15T22:23:49.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6771999999999999e-05 2021-03-15T22:23:49.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:23:49.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:23:49.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:23:49.608 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:46.232956+0000", 2021-03-15T22:23:49.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010620465000000001 2021-03-15T22:23:49.608 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.609 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.609 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.609 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:23:49.609 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:23:49.609 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:23:49.610 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:49.611 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:49 smithi125 conmon[56739]: cluster 2021-03-15T22:23:48.233632+0000 mon.a (mon.0) 2021-03-15T22:23:49.611 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:49 smithi125 conmon[56739]: 1027 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2021-03-15T22:23:49.612 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:49 smithi125 conmon[56739]: audit 2021-03-15T22:23:48.550292+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? 172.21.15.76:0/2617508269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:49.612 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:49 smithi076 conmon[56514]: cluster 2021-03-15T22:23:48.233632+0000 mon.a (mon.0) 1027 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2021-03-15T22:23:49.612 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:49 smithi076 conmon[56514]: audit 2021-03-15T22:23:48.550292+0000 mon.a (mon.0) 1028 : audit [INF] from='client.? 172.21.15.76:0/2617508269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:49.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:49 smithi076 conmon[62156]: cluster 2021-03-15T22:23:48.233632+0000 2021-03-15T22:23:49.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:49 smithi076 conmon[62156]: mon.a (mon.0) 1027 : cluster [DBG] osdmap e313: 8 total, 8 up, 8 in 2021-03-15T22:23:49.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:49 smithi076 conmon[62156]: audit 2021-03-15T22:23:48.550292+0000 mon.a (mon.0) 1028 : audit [INF] 2021-03-15T22:23:49.614 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:49 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2617508269' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:23:49.637 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:49.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:49.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:49.740 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:49.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:49.765 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:49.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:49.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:49.893 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:49.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:49.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:49.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:50.020 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:50.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:50.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:50.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:23:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:23:50.065 INFO:tasks.rados.rados.0.smithi076.stdout:1078: done (0 left) 2021-03-15T22:23:50.065 INFO:tasks.rados.rados.0.smithi076.stdout:1079: rollback oid 17 current snap is 108 2021-03-15T22:23:50.065 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 17 to 107 2021-03-15T22:23:50.065 INFO:tasks.rados.rados.0.smithi076.stdout:1080: read oid 42 snap -1 2021-03-15T22:23:50.066 INFO:tasks.rados.rados.0.smithi076.stdout:1080: expect (ObjNum 326 snap 87 seq_num 326) 2021-03-15T22:23:50.066 INFO:tasks.rados.rados.0.smithi076.stdout:1081: write oid 29 current snap is 108 2021-03-15T22:23:50.066 INFO:tasks.rados.rados.0.smithi076.stdout:1081: seq_num 397 ranges {672379=724209,1865116=608472} 2021-03-15T22:23:50.072 INFO:tasks.rados.rados.0.smithi076.stdout:1081: writing smithi07694878-29 from 672379 to 1396588 tid 1 2021-03-15T22:23:50.078 INFO:tasks.rados.rados.0.smithi076.stdout:1081: writing smithi07694878-29 from 1865116 to 2473588 tid 2 2021-03-15T22:23:50.079 INFO:tasks.rados.rados.0.smithi076.stdout:1082: write (excl) oid 46 current snap is 108 2021-03-15T22:23:50.079 INFO:tasks.rados.rados.0.smithi076.stdout:1082: seq_num 398 ranges {689186=799458,1923598=752130} 2021-03-15T22:23:50.086 INFO:tasks.rados.rados.0.smithi076.stdout:1082: writing smithi07694878-46 from 689186 to 1488644 tid 1 2021-03-15T22:23:50.094 INFO:tasks.rados.rados.0.smithi076.stdout:1082: writing smithi07694878-46 from 1923598 to 2675728 tid 2 2021-03-15T22:23:50.094 INFO:tasks.rados.rados.0.smithi076.stdout:1083: read oid 36 snap -1 2021-03-15T22:23:50.095 INFO:tasks.rados.rados.0.smithi076.stdout:1083: expect (ObjNum 318 snap 85 seq_num 318) 2021-03-15T22:23:50.095 INFO:tasks.rados.rados.0.smithi076.stdout:1084: write oid 8 current snap is 108 2021-03-15T22:23:50.095 INFO:tasks.rados.rados.0.smithi076.stdout:1079: finishing rollback tid 1 to smithi07694878-17 2021-03-15T22:23:50.095 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:23:50.096 INFO:tasks.rados.rados.0.smithi076.stdout:1080: expect (ObjNum 326 snap 87 seq_num 326) 2021-03-15T22:23:50.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:50.147 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:50.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:50.149 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:50.212 INFO:tasks.rados.rados.0.smithi076.stdout:1081: finishing write tid 1 to smithi07694878-29 2021-03-15T22:23:50.212 INFO:tasks.rados.rados.0.smithi076.stdout:1081: finishing write tid 2 to smithi07694878-29 2021-03-15T22:23:50.213 INFO:tasks.rados.rados.0.smithi076.stdout:1081: finishing write tid 3 to smithi07694878-29 2021-03-15T22:23:50.213 INFO:tasks.rados.rados.0.smithi076.stdout:1081: finishing write tid 4 to smithi07694878-29 2021-03-15T22:23:50.213 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 632 (ObjNum 397 snap 108 seq_num 397) dirty exists 2021-03-15T22:23:50.213 INFO:tasks.rados.rados.0.smithi076.stdout:1081: left oid 29 (ObjNum 397 snap 108 seq_num 397) 2021-03-15T22:23:50.213 INFO:tasks.rados.rados.0.smithi076.stdout:1082: finishing write tid 1 to smithi07694878-46 2021-03-15T22:23:50.214 INFO:tasks.rados.rados.0.smithi076.stdout:1083: expect (ObjNum 318 snap 85 seq_num 318) 2021-03-15T22:23:50.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:50.274 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:50.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:50.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:50.296 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[56514]: cluster 2021-03-15T22:23:49.029298 2021-03-15T22:23:50.297 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 543 : cluster [DBG] pgmap v597: 138 pgs: 138 active+clean; 95 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 25 op/s 2021-03-15T22:23:50.297 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[56514]: cluster 2021-03-15T22:23:49.229796+0000 mon.a (mon.0) 1029 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:50.297 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[56514]: audit 2021-03-15T22:23:49.239595+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? 172.21.15.76:0/2617508269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:50.298 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[56514]: cluster 2021-03-15T22:23:49.239660+0000 mon.a (mon.0) 1031 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2021-03-15T22:23:50.298 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[56514]: audit 2021-03-15T22:23:49.609475+0000 mgr.y (mgr.14144) 544 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:50.298 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[56514]: cluster 2021-03-15T22:23:50.062242+0000 mon.a (mon.0) 1032 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2021-03-15T22:23:50.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: cluster 2021-03-15T22:23:49.029298+0000 mgr.y (mgr 2021-03-15T22:23:50.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: .14144) 543 : cluster [DBG] pgmap v597: 138 pgs: 138 active+clean; 95 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 25 op/s 2021-03-15T22:23:50.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: cluster 2021-03-15T22:23:49.229796+0000 mon.a (mon. 2021-03-15T22:23:50.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: 0) 1029 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:50.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:23:50.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: 15T22:23:49.239595+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? 172.21.15.76:0/2617508269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:50.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:23:50.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: 15T22:23:49.239660+0000 mon.a ( 2021-03-15T22:23:50.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: mon.0) 1031 : cluster [DBG] osdmap e314: 8 total, 8 up, 8 in 2021-03-15T22:23:50.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:23:50.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: -15T22:23:49.609475+0000 2021-03-15T22:23:50.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: mgr.y (mgr.14144) 544 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:50.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:23:50.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: 03-15T22:23:50.062242+0000 2021-03-15T22:23:50.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:50 smithi076 conmon[62156]: mon.a (mon.0) 1032 : cluster [DBG] osdmap e315: 8 total, 8 up, 8 in 2021-03-15T22:23:50.318 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-15T22:23:50.376 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:50.402 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:50.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:50.404 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:50.409 INFO:tasks.rados.rados.0.smithi076.stdout:1082: finishing write tid 2 to smithi07694878-46 2021-03-15T22:23:50.409 INFO:tasks.rados.rados.0.smithi076.stdout:1082: finishing write tid 3 to smithi07694878-46 2021-03-15T22:23:50.410 INFO:tasks.rados.rados.0.smithi076.stdout:1082: finishing write tid 4 to smithi07694878-46 2021-03-15T22:23:50.410 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 199 (ObjNum 398 snap 108 seq_num 398) dirty exists 2021-03-15T22:23:50.410 INFO:tasks.rados.rados.0.smithi076.stdout:1082: left oid 46 (ObjNum 398 snap 108 seq_num 398) 2021-03-15T22:23:50.411 INFO:tasks.rados.rados.0.smithi076.stdout:1084: seq_num 399 ranges {705993=474707,1582080=612141} 2021-03-15T22:23:50.416 INFO:tasks.rados.rados.0.smithi076.stdout:1084: writing smithi07694878-8 from 705993 to 1180700 tid 1 2021-03-15T22:23:50.421 INFO:tasks.rados.rados.0.smithi076.stdout:1084: writing smithi07694878-8 from 1582080 to 2194221 tid 2 2021-03-15T22:23:50.422 INFO:tasks.rados.rados.0.smithi076.stdout:1079: done (5 left) 2021-03-15T22:23:50.422 INFO:tasks.rados.rados.0.smithi076.stdout:1080: done (4 left) 2021-03-15T22:23:50.422 INFO:tasks.rados.rados.0.smithi076.stdout:1081: done (3 left) 2021-03-15T22:23:50.422 INFO:tasks.rados.rados.0.smithi076.stdout:1082: done (2 left) 2021-03-15T22:23:50.423 INFO:tasks.rados.rados.0.smithi076.stdout:1083: done (1 left) 2021-03-15T22:23:50.423 INFO:tasks.rados.rados.0.smithi076.stdout:1085: snap_remove snap 106 2021-03-15T22:23:50.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:50.530 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:50.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: cluster 2021-03-15T22:23:49.029298+0000 mgr.y (mgr.14144) 543 : cluster [DBG] pgmap v597: 138 pgs: 138 active+clean; 95 MiB data, 442 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 2.6 MiB/s wr, 25 op/s 2021-03-15T22:23:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:23:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: 22:23:49.229796+0000 mon.a (mon.0) 1029 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: audit 2021-03-15T22:23:49.239595+0000 mon.a (mon.0) 1030 : audit [INF] from='client.? 172.21.15.76:0/2617508269' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:23:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: cluster 2021-03-15T22:23:49.239660 2021-03-15T22:23:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1031 : cluster [DBG] 2021-03-15T22:23:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: osdmap e314: 8 total, 8 up, 8 in 2021-03-15T22:23:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:23:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: :23:49.609475+0000 mgr.y (mgr. 2021-03-15T22:23:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: 14144) 544 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:23:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:23:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: -03-15T22:23:50.062242 2021-03-15T22:23:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1032 : cluster 2021-03-15T22:23:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:50 smithi125 conmon[56739]: [DBG] osdmap e315: 8 total, 8 up, 8 in 2021-03-15T22:23:50.631 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:50.653 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:50.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:50.655 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:50.757 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:50.783 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:50.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:50.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:50.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:50.909 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:50.909 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:51.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:51.036 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:51.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:51.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:51.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:51.162 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:51.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:51.164 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:51.250 INFO:tasks.rados.rados.0.smithi076.stdout:1085: done (1 left) 2021-03-15T22:23:51.251 INFO:tasks.rados.rados.0.smithi076.stdout:1086: snap_remove snap 107 2021-03-15T22:23:51.253 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.7 primary-affinity to 1 (8655362) 2021-03-15T22:23:51.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:51.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:51 smithi076 conmon[56514]: audit 2021-03-15T22:23:50.880940+0000 mon.a (mon.0) 2021-03-15T22:23:51.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:51 smithi076 conmon[56514]: 1033 : audit [INF] from='client.? 172.21.15.76:0/2116314666' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:23:51.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:51 smithi076 conmon[62156]: audit 2021-03-15T22:23:50.880940+0000 2021-03-15T22:23:51.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:51 smithi076 conmon[62156]: mon.a (mon.0) 1033 : audit [INF] from='client.? 172.21.15.76:0/2116314666' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:23:51.289 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:51.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:51.391 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:51.415 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:51.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:51.517 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:51.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:51 smithi125 conmon[56739]: audit 2021-03-15T22:23:50.880940+0000 mon.a (mon.0 2021-03-15T22:23:51.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:51 smithi125 conmon[56739]: ) 1033 : audit [INF] from='client.? 172.21.15.76:0/2116314666' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:23:51.540 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:51.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:51.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:51.643 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:51.666 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:51.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:51.667 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:51.768 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:51.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:51.793 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:51.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:51.895 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:51.919 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:51.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:51.921 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:51.990 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:52.022 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:52.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:52.049 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:52.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:52.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:52.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:52.180 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:52.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:52.254 INFO:tasks.rados.rados.0.smithi076.stdout:1086: done (1 left) 2021-03-15T22:23:52.254 INFO:tasks.rados.rados.0.smithi076.stdout:1087: copy_from oid 39 from oid 44 current snap is 108 2021-03-15T22:23:52.254 INFO:tasks.rados.rados.0.smithi076.stdout:1084: finishing write tid 1 to smithi07694878-8 2021-03-15T22:23:52.255 INFO:tasks.rados.rados.0.smithi076.stdout:1084: finishing write tid 2 to smithi07694878-8 2021-03-15T22:23:52.255 INFO:tasks.rados.rados.0.smithi076.stdout:1084: finishing write tid 3 to smithi07694878-8 2021-03-15T22:23:52.255 INFO:tasks.rados.rados.0.smithi076.stdout:1084: finishing write tid 4 to smithi07694878-8 2021-03-15T22:23:52.255 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 266 (ObjNum 399 snap 108 seq_num 399) dirty exists 2021-03-15T22:23:52.255 INFO:tasks.rados.rados.0.smithi076.stdout:1084: left oid 8 (ObjNum 399 snap 108 seq_num 399) 2021-03-15T22:23:52.256 INFO:tasks.rados.rados.0.smithi076.stdout:1084: done (1 left) 2021-03-15T22:23:52.256 INFO:tasks.rados.rados.0.smithi076.stdout:1088: rollback oid 26 current snap is 108 2021-03-15T22:23:52.256 INFO:tasks.rados.rados.0.smithi076.stdout:1088: done (1 left) 2021-03-15T22:23:52.256 INFO:tasks.rados.rados.0.smithi076.stdout:1089: write (excl) oid 16 current snap is 108 2021-03-15T22:23:52.257 INFO:tasks.rados.rados.0.smithi076.stdout:1089: seq_num 400 ranges {722800=549956,2040562=682238} 2021-03-15T22:23:52.261 INFO:tasks.rados.rados.0.smithi076.stdout:1089: writing smithi07694878-16 from 722800 to 1272756 tid 1 2021-03-15T22:23:52.271 INFO:tasks.rados.rados.0.smithi076.stdout:1089: writing smithi07694878-16 from 2040562 to 2722800 tid 2 2021-03-15T22:23:52.272 INFO:tasks.rados.rados.0.smithi076.stdout:1090: copy_from oid 35 from oid 34 current snap is 108 2021-03-15T22:23:52.272 INFO:tasks.rados.rados.0.smithi076.stdout:1091: write (excl) oid 27 current snap is 108 2021-03-15T22:23:52.272 INFO:tasks.rados.rados.0.smithi076.stdout:1091: seq_num 401 ranges {739607=625205,1982691=499457} 2021-03-15T22:23:52.280 INFO:tasks.rados.rados.0.smithi076.stdout:1091: writing smithi07694878-27 from 739607 to 1364812 tid 1 2021-03-15T22:23:52.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:52.287 INFO:tasks.rados.rados.0.smithi076.stdout:1091: writing smithi07694878-27 from 1982691 to 2482148 tid 2 2021-03-15T22:23:52.287 INFO:tasks.rados.rados.0.smithi076.stdout:1092: write (excl) oid 18 current snap is 108 2021-03-15T22:23:52.288 INFO:tasks.rados.rados.0.smithi076.stdout:1092: seq_num 402 ranges {756414=700454,2041173=715241} 2021-03-15T22:23:52.296 INFO:tasks.rados.rados.0.smithi076.stdout:1092: writing smithi07694878-18 from 756414 to 1456868 tid 1 2021-03-15T22:23:52.305 INFO:tasks.rados.rados.0.smithi076.stdout:1092: writing smithi07694878-18 from 2041173 to 2756414 tid 2 2021-03-15T22:23:52.306 INFO:tasks.rados.rados.0.smithi076.stdout:1093: snap_create 2021-03-15T22:23:52.306 INFO:tasks.rados.rados.0.smithi076.stdout:1090: finishing copy_from to smithi07694878-35 2021-03-15T22:23:52.306 INFO:tasks.rados.rados.0.smithi076.stdout:1090: got expected ENOENT (src dne) 2021-03-15T22:23:52.306 INFO:tasks.rados.rados.0.smithi076.stdout:1090: finishing copy_from racing read to smithi07694878-35 2021-03-15T22:23:52.307 INFO:tasks.rados.rados.0.smithi076.stdout:1089: finishing write tid 1 to smithi07694878-16 2021-03-15T22:23:52.307 INFO:tasks.rados.rados.0.smithi076.stdout:1087: finishing copy_from to smithi07694878-39 2021-03-15T22:23:52.307 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 215 (ObjNum 395 snap 106 seq_num 395) dirty exists 2021-03-15T22:23:52.307 INFO:tasks.rados.rados.0.smithi076.stdout:1087: finishing copy_from racing read to smithi07694878-39 2021-03-15T22:23:52.307 INFO:tasks.rados.rados.0.smithi076.stdout:1089: finishing write tid 2 to smithi07694878-16 2021-03-15T22:23:52.308 INFO:tasks.rados.rados.0.smithi076.stdout:1089: finishing write tid 3 to smithi07694878-16 2021-03-15T22:23:52.308 INFO:tasks.rados.rados.0.smithi076.stdout:1089: finishing write tid 4 to smithi07694878-16 2021-03-15T22:23:52.308 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 603 (ObjNum 400 snap 108 seq_num 400) dirty exists 2021-03-15T22:23:52.309 INFO:tasks.rados.rados.0.smithi076.stdout:1089: left oid 16 (ObjNum 400 snap 108 seq_num 400) 2021-03-15T22:23:52.309 INFO:tasks.rados.rados.0.smithi076.stdout:1091: finishing write tid 1 to smithi07694878-27 2021-03-15T22:23:52.309 INFO:tasks.rados.rados.0.smithi076.stdout:1091: finishing write tid 2 to smithi07694878-27 2021-03-15T22:23:52.309 INFO:tasks.rados.rados.0.smithi076.stdout:1091: finishing write tid 3 to smithi07694878-27 2021-03-15T22:23:52.310 INFO:tasks.rados.rados.0.smithi076.stdout:1091: finishing write tid 4 to smithi07694878-27 2021-03-15T22:23:52.310 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 218 (ObjNum 401 snap 108 seq_num 401) dirty exists 2021-03-15T22:23:52.310 INFO:tasks.rados.rados.0.smithi076.stdout:1091: left oid 27 (ObjNum 401 snap 108 seq_num 401) 2021-03-15T22:23:52.310 INFO:tasks.rados.rados.0.smithi076.stdout:1092: finishing write tid 1 to smithi07694878-18 2021-03-15T22:23:52.312 INFO:tasks.rados.rados.0.smithi076.stdout:1092: finishing write tid 2 to smithi07694878-18 2021-03-15T22:23:52.312 INFO:tasks.rados.rados.0.smithi076.stdout:1092: finishing write tid 3 to smithi07694878-18 2021-03-15T22:23:52.312 INFO:tasks.rados.rados.0.smithi076.stdout:1092: finishing write tid 4 to smithi07694878-18 2021-03-15T22:23:52.313 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 268 (ObjNum 402 snap 108 seq_num 402) dirty exists 2021-03-15T22:23:52.313 INFO:tasks.rados.rados.0.smithi076.stdout:1092: left oid 18 (ObjNum 402 snap 108 seq_num 402) 2021-03-15T22:23:52.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:52.315 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:52.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:52.357 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:23:52.374 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:23:52.391 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:23:52.408 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:23:52.417 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:52.421 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:52 smithi076 conmon[56514]: cluster 2021-03-15T22:23:51.029852+0000 mgr.y (mgr.14144) 545 : cluster [DBG] pgmap v600: 138 pgs: 138 active+clean; 102 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2021-03-15T22:23:52.421 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:52 smithi076 conmon[56514]: audit 2021-03-15T22:23:51.247739+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? 172.21.15.76:0/2116314666' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:23:52.422 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:52 smithi076 conmon[56514]: cluster 2021-03-15T22:23:51.247794+0000 mon.a (mon.0) 1035 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2021-03-15T22:23:52.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:52 smithi076 conmon[62156]: cluster 2021-03-15T22:23:51.029852+0000 mgr.y (mgr.14144) 545 : cluster [DBG] pgmap v600: 138 pgs: 138 active+clean; 102 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2021-03-15T22:23:52.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:52 smithi076 conmon[62156]: audit 2021-03-15T22:23:51.247739+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? 172.21.15.76:0/2116314666' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:23:52.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:52 smithi076 conmon[62156]: cluster 2021-03-15T22:23:51.247794+0000 mon.a (mon.0) 1035 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2021-03-15T22:23:52.426 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:23:52.444 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:52.445 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:23:52.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:52.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:52.463 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:23:52.482 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:23:52.484 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:23:52.485 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:23:52.485 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:23:52.486 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:23:52.486 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:23:52.486 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:23:52.487 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:23:52.487 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:23:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:52 smithi125 conmon[56739]: cluster 2021-03-15T22:23:51.029852+0000 mgr.y (mgr.14144) 545 : cluster [DBG] pgmap v600: 138 pgs: 138 active+clean; 102 MiB data, 453 MiB used, 715 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 10 op/s 2021-03-15T22:23:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:52 smithi125 conmon[56739]: audit 2021-03-15T22:23:51.247739+0000 mon.a (mon.0) 1034 : audit [INF] from='client.? 172.21.15.76:0/2116314666' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:23:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:52 smithi125 conmon[56739]: cluster 2021-03-15T22:23:51.247794+0000 mon.a (mon.0) 1035 : cluster [DBG] osdmap e316: 8 total, 8 up, 8 in 2021-03-15T22:23:52.547 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:52.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:52.577 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:52.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:52.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:52.702 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:52.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:52.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:52.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:52.833 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:52.833 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:52.935 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:52.959 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:52.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:52.961 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:53.062 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:53.083 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:53.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:53.085 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:53.128 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:23:53.186 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:53.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:53.217 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:53.218 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:53.265 INFO:tasks.rados.rados.0.smithi076.stdout:1087: done (5 left) 2021-03-15T22:23:53.265 INFO:tasks.rados.rados.0.smithi076.stdout:1089: done (4 left) 2021-03-15T22:23:53.266 INFO:tasks.rados.rados.0.smithi076.stdout:1090: done (3 left) 2021-03-15T22:23:53.266 INFO:tasks.rados.rados.0.smithi076.stdout:1091: done (2 left) 2021-03-15T22:23:53.266 INFO:tasks.rados.rados.0.smithi076.stdout:1092: done (1 left) 2021-03-15T22:23:53.266 INFO:tasks.rados.rados.0.smithi076.stdout:1093: done (0 left) 2021-03-15T22:23:53.267 INFO:tasks.rados.rados.0.smithi076.stdout:1094: rollback oid 44 current snap is 109 2021-03-15T22:23:53.267 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 44 to 108 2021-03-15T22:23:53.267 INFO:tasks.rados.rados.0.smithi076.stdout:1095: read oid 19 snap 108 2021-03-15T22:23:53.267 INFO:tasks.rados.rados.0.smithi076.stdout:1095: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:23:53.267 INFO:tasks.rados.rados.0.smithi076.stdout:1096: read oid 2 snap -1 2021-03-15T22:23:53.268 INFO:tasks.rados.rados.0.smithi076.stdout:1096: expect (ObjNum 383 snap 103 seq_num 383) 2021-03-15T22:23:53.268 INFO:tasks.rados.rados.0.smithi076.stdout:1097: rollback oid 14 current snap is 109 2021-03-15T22:23:53.268 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 14 to 108 2021-03-15T22:23:53.268 INFO:tasks.rados.rados.0.smithi076.stdout:1098: snap_create 2021-03-15T22:23:53.269 INFO:tasks.rados.rados.0.smithi076.stdout:1095: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:23:53.319 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:53.344 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:53.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:53.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:53.429 INFO:tasks.rados.rados.0.smithi076.stdout:1096: expect (ObjNum 383 snap 103 seq_num 383) 2021-03-15T22:23:53.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:53.472 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:53.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:53.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:53.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[56514]: cluster 2021-03-15T22:23:51.861600+0000 osd.7 (osd.7) 65 : cluster [DBG] 2021-03-15T22:23:53.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[56514]: purged_snaps scrub starts 2021-03-15T22:23:53.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[56514]: cluster 2021-03-15T22:23:51.861898+0000 osd.7 (osd.7) 66 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:53.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[56514]: cluster 2021-03-15T22:23:52.251331+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2021-03-15T22:23:53.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[56514]: audit 2021-03-15T22:23:52.338429+0000 mon.a (mon.0) 1037 : audit [DBG] from='client.? 172.21.15.76:0/3327825602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:53.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[56514]: cluster 2021-03-15T22:23:52.837218+0000 osd.0 (osd.0) 65 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:53.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[56514]: cluster 2021-03-15T22:23:52.837440+0000 osd.0 (osd.0) 66 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:53.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: cluster 2021-03-15T22:23:53.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: 2021-03-15T22:23:53.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: 2021 2021-03-15T22:23:53.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: - 2021-03-15T22:23:53.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: 03 2021-03-15T22:23:53.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: - 2021-03-15T22:23:53.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: 15 2021-03-15T22:23:53.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: T22:23:51. 2021-03-15T22:23:53.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: 861600+0000 osd.7 (osd.7) 65 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:53.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:23:53.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: -15T22:23:51.861898+0000 osd.7 (osd. 2021-03-15T22:23:53.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: 7) 66 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:53.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:23:53.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: -03-15T22:23:52.251331+0000 2021-03-15T22:23:53.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: mon.a (mon.0) 1036 : cluster [DBG] 2021-03-15T22:23:53.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: osdmap e317: 8 total, 8 up, 8 in 2021-03-15T22:23:53.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: audit 2021-03-15T22:23:52 2021-03-15T22:23:53.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: .338429+0000 mon.a (mon.0 2021-03-15T22:23:53.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: ) 1037 : audit [DBG] from='client.? 172.21.15.76:0/3327825602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:53.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:23:53.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: -15T22:23:52.837218+0000 osd.0 2021-03-15T22:23:53.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: (osd.0) 65 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:53.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: 2021-03-15T22:23:53.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: cluster 2021-03-15T22:23:53.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: 2021-03-15T22:23:52.837440+0000 osd.0 2021-03-15T22:23:53.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: (osd.0) 66 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:53.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:53 smithi076 conmon[62156]: 2021-03-15T22:23:53.527 INFO:tasks.rados.rados.0.smithi076.stdout:1094: finishing rollback tid 0 to smithi07694878-44 2021-03-15T22:23:53.528 INFO:tasks.rados.rados.0.smithi076.stdout:1094: finishing rollback tid 1 to smithi07694878-44 2021-03-15T22:23:53.529 INFO:tasks.rados.rados.0.smithi076.stdout:1097: finishing rollback tid 0 to smithi07694878-14 2021-03-15T22:23:53.529 INFO:tasks.rados.rados.0.smithi076.stdout:1094: finishing rollback tid 2 to smithi07694878-44 2021-03-15T22:23:53.529 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 222 (ObjNum 395 snap 106 seq_num 395) dirty exists 2021-03-15T22:23:53.530 INFO:tasks.rados.rados.0.smithi076.stdout:1097: finishing rollback tid 1 to smithi07694878-14 2021-03-15T22:23:53.530 INFO:tasks.rados.rados.0.smithi076.stdout:1097: finishing rollback tid 2 to smithi07694878-14 2021-03-15T22:23:53.530 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 302 (ObjNum 390 snap 106 seq_num 390) dirty exists 2021-03-15T22:23:53.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: cluster 2021-03-15T22:23:51.861600+0000 osd.7 (osd 2021-03-15T22:23:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: .7) 65 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:23:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: cluster 2021-03-15T22:23:51.861898+0000 osd.7 (osd.7) 66 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:53.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:23:53.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: :23:52.251331+0000 mon.a (mon.0) 1036 : cluster [DBG] osdmap e317: 8 total, 8 up, 8 in 2021-03-15T22:23:53.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:23:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: 22:23:52.338429+0000 mon.a (mon.0) 1037 : audit [DBG] from='client.? 172.21.15.76:0/3327825602' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:23:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: cluster 2021-03-15T22:23:52 2021-03-15T22:23:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: .837218+0000 osd.0 (osd.0) 65 : cluster 2021-03-15T22:23:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: [DBG] purged_snaps scrub starts 2021-03-15T22:23:53.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:23:53.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: 15T22:23:52.837440+0000 osd.0 2021-03-15T22:23:53.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:53 smithi125 conmon[56739]: (osd.0) 66 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:23:53.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:53.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:53.603 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:53.603 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:53.705 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:53.728 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:53.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:53.730 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:53.788 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:53.832 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:53.856 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:53.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:53.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:53.959 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:53.982 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:53.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:53.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:54.084 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:54.110 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:54.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:54.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:54.159 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:23:54.160 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:23:54.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:54.237 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:54.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:54.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:54.268 INFO:tasks.rados.rados.0.smithi076.stdout:1094: done (4 left) 2021-03-15T22:23:54.269 INFO:tasks.rados.rados.0.smithi076.stdout:1095: done (3 left) 2021-03-15T22:23:54.269 INFO:tasks.rados.rados.0.smithi076.stdout:1096: done (2 left) 2021-03-15T22:23:54.269 INFO:tasks.rados.rados.0.smithi076.stdout:1097: done (1 left) 2021-03-15T22:23:54.269 INFO:tasks.rados.rados.0.smithi076.stdout:1098: done (0 left) 2021-03-15T22:23:54.270 INFO:tasks.rados.rados.0.smithi076.stdout:1099: read oid 28 snap -1 2021-03-15T22:23:54.270 INFO:tasks.rados.rados.0.smithi076.stdout:1099: expect deleted 2021-03-15T22:23:54.270 INFO:tasks.rados.rados.0.smithi076.stdout:1100: delete oid 35 current snap is 110 2021-03-15T22:23:54.271 INFO:tasks.rados.rados.0.smithi076.stdout:1099: done (1 left) 2021-03-15T22:23:54.272 INFO:tasks.rados.rados.0.smithi076.stdout:1100: done (0 left) 2021-03-15T22:23:54.272 INFO:tasks.rados.rados.0.smithi076.stdout:1101: rollback oid 32 current snap is 110 2021-03-15T22:23:54.272 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 32 to 108 2021-03-15T22:23:54.272 INFO:tasks.rados.rados.0.smithi076.stdout:1102: delete oid 39 current snap is 110 2021-03-15T22:23:54.273 INFO:tasks.rados.rados.0.smithi076.stdout:1102: done (1 left) 2021-03-15T22:23:54.273 INFO:tasks.rados.rados.0.smithi076.stdout:1103: write oid 42 current snap is 110 2021-03-15T22:23:54.273 INFO:tasks.rados.rados.0.smithi076.stdout:1103: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2021-03-15T22:23:54.277 INFO:tasks.rados.rados.0.smithi076.stdout:1103: writing smithi07694878-42 from 406835 to 1049389 tid 1 2021-03-15T22:23:54.283 INFO:tasks.rados.rados.0.smithi076.stdout:1103: writing smithi07694878-42 from 1532972 to 2039767 tid 2 2021-03-15T22:23:54.289 INFO:tasks.rados.rados.0.smithi076.stdout:1103: writing smithi07694878-42 from 2772312 to 2806835 tid 3 2021-03-15T22:23:54.289 INFO:tasks.rados.rados.0.smithi076.stdout:1104: read oid 43 snap 108 2021-03-15T22:23:54.289 INFO:tasks.rados.rados.0.smithi076.stdout:1104: expect (ObjNum 392 snap 106 seq_num 392) 2021-03-15T22:23:54.290 INFO:tasks.rados.rados.0.smithi076.stdout:1105: copy_from oid 25 from oid 34 current snap is 110 2021-03-15T22:23:54.290 INFO:tasks.rados.rados.0.smithi076.stdout:1101: finishing rollback tid 0 to smithi07694878-32 2021-03-15T22:23:54.291 INFO:tasks.rados.rados.0.smithi076.stdout:1101: finishing rollback tid 1 to smithi07694878-32 2021-03-15T22:23:54.291 INFO:tasks.rados.rados.0.smithi076.stdout:1101: finishing rollback tid 2 to smithi07694878-32 2021-03-15T22:23:54.292 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 640 (ObjNum 371 snap 102 seq_num 371) dirty exists 2021-03-15T22:23:54.292 INFO:tasks.rados.rados.0.smithi076.stdout:1103: finishing write tid 1 to smithi07694878-42 2021-03-15T22:23:54.293 INFO:tasks.rados.rados.0.smithi076.stdout:1103: finishing write tid 2 to smithi07694878-42 2021-03-15T22:23:54.293 INFO:tasks.rados.rados.0.smithi076.stdout:1101: done (3 left) 2021-03-15T22:23:54.293 INFO:tasks.rados.rados.0.smithi076.stdout:1106: rollback oid 16 current snap is 110 2021-03-15T22:23:54.294 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 108 2021-03-15T22:23:54.294 INFO:tasks.rados.rados.0.smithi076.stdout:1107: snap_create 2021-03-15T22:23:54.295 INFO:tasks.rados.rados.0.smithi076.stdout:1104: expect (ObjNum 392 snap 106 seq_num 392) 2021-03-15T22:23:54.340 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:54.362 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:54.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:54.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:54.375 INFO:tasks.rados.rados.0.smithi076.stdout:1103: finishing write tid 3 to smithi07694878-42 2021-03-15T22:23:54.376 INFO:tasks.rados.rados.0.smithi076.stdout:1103: finishing write tid 4 to smithi07694878-42 2021-03-15T22:23:54.376 INFO:tasks.rados.rados.0.smithi076.stdout:1103: finishing write tid 5 to smithi07694878-42 2021-03-15T22:23:54.377 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 645 (ObjNum 405 snap 110 seq_num 405) dirty exists 2021-03-15T22:23:54.377 INFO:tasks.rados.rados.0.smithi076.stdout:1103: left oid 42 (ObjNum 405 snap 110 seq_num 405) 2021-03-15T22:23:54.378 INFO:tasks.rados.rados.0.smithi076.stdout:1105: finishing copy_from to smithi07694878-25 2021-03-15T22:23:54.378 INFO:tasks.rados.rados.0.smithi076.stdout:1105: got expected ENOENT (src dne) 2021-03-15T22:23:54.379 INFO:tasks.rados.rados.0.smithi076.stdout:1105: finishing copy_from racing read to smithi07694878-25 2021-03-15T22:23:54.379 INFO:tasks.rados.rados.0.smithi076.stdout:1106: finishing rollback tid 0 to smithi07694878-16 2021-03-15T22:23:54.379 INFO:tasks.rados.rados.0.smithi076.stdout:1106: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:23:54.380 INFO:tasks.rados.rados.0.smithi076.stdout:1106: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:23:54.381 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 607 (ObjNum 400 snap 108 seq_num 400) dirty exists 2021-03-15T22:23:54.465 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:54.490 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:54.492 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:54.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:54 smithi076 conmon[56514]: cluster 2021-03-15T22:23:53.030306+0000 mgr.y (mgr.14144) 546 : cluster [DBG] pgmap v603: 138 pgs: 138 active+clean; 101 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2021-03-15T22:23:54.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:54 smithi076 conmon[56514]: cluster 2021-03-15T22:23:53.261165+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2021-03-15T22:23:54.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:54 smithi076 conmon[62156]: cluster 2021-03-15T22:23:53.030306+0000 mgr.y (mgr.14144 2021-03-15T22:23:54.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:54 smithi076 conmon[62156]: ) 546 : cluster [DBG] pgmap v603: 138 pgs: 138 active+clean; 101 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2021-03-15T22:23:54.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:54 smithi076 conmon[62156]: cluster 2021-03-15T22:23:53.261165+0000 mon.a ( 2021-03-15T22:23:54.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:54 smithi076 conmon[62156]: mon.0) 1038 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2021-03-15T22:23:54.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:54 smithi125 conmon[56739]: cluster 2021-03-15T22:23:53.030306+0000 mgr.y (mgr.14144) 546 : cluster [DBG] pgmap v603: 138 pgs: 138 active+clean; 101 MiB data, 441 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s rd, 3.3 MiB/s wr, 15 op/s 2021-03-15T22:23:54.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:54 smithi125 conmon[56739]: 2021-03-15T22:23:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:54 smithi125 conmon[56739]: cluster 2021-03-15T22:23:53.261165+0000 mon.a (mon.0) 1038 : cluster [DBG] osdmap e318: 8 total, 8 up, 8 in 2021-03-15T22:23:54.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:54.619 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:54.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:54.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:54.722 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:54.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:54.749 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:54.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:54.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:54.876 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:54.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:54.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:54.979 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:54.981 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-15T22:23:55.041 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:55.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:55.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:55.064 INFO:tasks.rados.rados.0.smithi076.stdout:1103: done (4 left) 2021-03-15T22:23:55.064 INFO:tasks.rados.rados.0.smithi076.stdout:1104: done (3 left) 2021-03-15T22:23:55.064 INFO:tasks.rados.rados.0.smithi076.stdout:1105: done (2 left) 2021-03-15T22:23:55.065 INFO:tasks.rados.rados.0.smithi076.stdout:1106: done (1 left) 2021-03-15T22:23:55.065 INFO:tasks.rados.rados.0.smithi076.stdout:1107: done (0 left) 2021-03-15T22:23:55.065 INFO:tasks.rados.rados.0.smithi076.stdout:1108: snap_create 2021-03-15T22:23:55.144 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:55.168 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:55.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:55.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:55.271 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:55.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:55.295 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:55.295 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:55.396 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:55.420 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:55.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:55.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:55.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:55.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:55 smithi125 conmon[56739]: cluster 2021-03-15T22:23:54.259172+0000 mon.a (mon.0) 1039 : cluster [DBG] 2021-03-15T22:23:55.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:55 smithi125 conmon[56739]: osdmap e319: 8 total, 8 up, 8 in 2021-03-15T22:23:55.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:55 smithi125 conmon[56739]: cluster 2021-03-15T22:23:55.062815+0000 mon.a (mon.0) 1040 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2021-03-15T22:23:55.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:55 smithi076 conmon[56514]: cluster 2021-03-15T22:23:54.259172 2021-03-15T22:23:55.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:55 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1039 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2021-03-15T22:23:55.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:55 smithi076 conmon[56514]: cluster 2021-03-15T22:23:55.062815+0000 mon.a (mon.0) 1040 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2021-03-15T22:23:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:55 smithi076 conmon[62156]: cluster 2021-03-15T22:23:54.259172+0000 2021-03-15T22:23:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:55 smithi076 conmon[62156]: mon.a (mon.0) 1039 : cluster [DBG] osdmap e319: 8 total, 8 up, 8 in 2021-03-15T22:23:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:55 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:23:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:55 smithi076 conmon[62156]: 23:55.062815+0000 mon.a (mon.0) 1040 : cluster [DBG] osdmap e320: 8 total, 8 up, 8 in 2021-03-15T22:23:55.544 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:55.549 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:55.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:55.551 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:55.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:55.674 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:55.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:55.676 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:55.767 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:23:55.777 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:55.832 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:55.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:55.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:55.935 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:55.960 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:55.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:55.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:56.063 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:56.068 INFO:tasks.rados.rados.0.smithi076.stdout:1108: done (0 left) 2021-03-15T22:23:56.068 INFO:tasks.rados.rados.0.smithi076.stdout:1109: rollback oid 43 current snap is 112 2021-03-15T22:23:56.068 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 43 to 110 2021-03-15T22:23:56.068 INFO:tasks.rados.rados.0.smithi076.stdout:1110: delete oid 9 current snap is 112 2021-03-15T22:23:56.075 INFO:tasks.rados.rados.0.smithi076.stdout:1109: finishing rollback tid 0 to smithi07694878-43 2021-03-15T22:23:56.075 INFO:tasks.rados.rados.0.smithi076.stdout:1109: finishing rollback tid 1 to smithi07694878-43 2021-03-15T22:23:56.076 INFO:tasks.rados.rados.0.smithi076.stdout:1109: finishing rollback tid 2 to smithi07694878-43 2021-03-15T22:23:56.076 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 271 (ObjNum 392 snap 106 seq_num 392) dirty exists 2021-03-15T22:23:56.077 INFO:tasks.rados.rados.0.smithi076.stdout:1109: done (1 left) 2021-03-15T22:23:56.077 INFO:tasks.rados.rados.0.smithi076.stdout:1110: done (0 left) 2021-03-15T22:23:56.077 INFO:tasks.rados.rados.0.smithi076.stdout:1111: write (excl) oid 13 current snap is 112 2021-03-15T22:23:56.078 INFO:tasks.rados.rados.0.smithi076.stdout:1111: seq_num 407 ranges {440449=793052,1933583=510464} 2021-03-15T22:23:56.081 INFO:tasks.rados.rados.0.smithi076.stdout:1111: writing smithi07694878-13 from 440449 to 1233501 tid 1 2021-03-15T22:23:56.089 INFO:tasks.rados.rados.0.smithi076.stdout:1111: writing smithi07694878-13 from 1933583 to 2444047 tid 2 2021-03-15T22:23:56.089 INFO:tasks.rados.rados.0.smithi076.stdout:1112: read oid 50 snap 111 2021-03-15T22:23:56.089 INFO:tasks.rados.rados.0.smithi076.stdout:1112: expect deleted 2021-03-15T22:23:56.089 INFO:tasks.rados.rados.0.smithi076.stdout:1113: read oid 35 snap 108 2021-03-15T22:23:56.090 INFO:tasks.rados.rados.0.smithi076.stdout:1113: expect (ObjNum 381 snap 103 seq_num 381) 2021-03-15T22:23:56.090 INFO:tasks.rados.rados.0.smithi076.stdout:1114: write oid 37 current snap is 112 2021-03-15T22:23:56.090 INFO:tasks.rados.rados.0.smithi076.stdout:1111: finishing write tid 1 to smithi07694878-13 2021-03-15T22:23:56.090 INFO:tasks.rados.rados.0.smithi076.stdout:1114: seq_num 408 ranges {457256=468301,1592065=770475,2854581=2675} 2021-03-15T22:23:56.093 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:56.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:56.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:56.095 INFO:tasks.rados.rados.0.smithi076.stdout:1114: writing smithi07694878-37 from 457256 to 925557 tid 1 2021-03-15T22:23:56.101 INFO:tasks.rados.rados.0.smithi076.stdout:1114: writing smithi07694878-37 from 1592065 to 2362540 tid 2 2021-03-15T22:23:56.104 INFO:tasks.rados.rados.0.smithi076.stdout:1114: writing smithi07694878-37 from 2854581 to 2857256 tid 3 2021-03-15T22:23:56.105 INFO:tasks.rados.rados.0.smithi076.stdout:1115: delete oid 25 current snap is 112 2021-03-15T22:23:56.105 INFO:tasks.rados.rados.0.smithi076.stdout:1111: finishing write tid 2 to smithi07694878-13 2021-03-15T22:23:56.105 INFO:tasks.rados.rados.0.smithi076.stdout:1111: finishing write tid 3 to smithi07694878-13 2021-03-15T22:23:56.105 INFO:tasks.rados.rados.0.smithi076.stdout:1111: finishing write tid 4 to smithi07694878-13 2021-03-15T22:23:56.106 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 416 (ObjNum 407 snap 112 seq_num 407) dirty exists 2021-03-15T22:23:56.106 INFO:tasks.rados.rados.0.smithi076.stdout:1111: left oid 13 (ObjNum 407 snap 112 seq_num 407) 2021-03-15T22:23:56.106 INFO:tasks.rados.rados.0.smithi076.stdout:1113: expect (ObjNum 381 snap 103 seq_num 381) 2021-03-15T22:23:56.195 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:56.221 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:56.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:56.228 INFO:tasks.rados.rados.0.smithi076.stdout:1114: finishing write tid 1 to smithi07694878-37 2021-03-15T22:23:56.229 INFO:tasks.rados.rados.0.smithi076.stdout:1114: finishing write tid 2 to smithi07694878-37 2021-03-15T22:23:56.229 INFO:tasks.rados.rados.0.smithi076.stdout:1111: done (4 left) 2021-03-15T22:23:56.229 INFO:tasks.rados.rados.0.smithi076.stdout:1112: done (3 left) 2021-03-15T22:23:56.229 INFO:tasks.rados.rados.0.smithi076.stdout:1113: done (2 left) 2021-03-15T22:23:56.230 INFO:tasks.rados.rados.0.smithi076.stdout:1115: done (1 left) 2021-03-15T22:23:56.230 INFO:tasks.rados.rados.0.smithi076.stdout:1116: read oid 8 snap -1 2021-03-15T22:23:56.230 INFO:tasks.rados.rados.0.smithi076.stdout:1116: expect (ObjNum 399 snap 108 seq_num 399) 2021-03-15T22:23:56.230 INFO:tasks.rados.rados.0.smithi076.stdout:1117: write (excl) oid 23 current snap is 112 2021-03-15T22:23:56.231 INFO:tasks.rados.rados.0.smithi076.stdout:1114: finishing write tid 3 to smithi07694878-37 2021-03-15T22:23:56.231 INFO:tasks.rados.rados.0.smithi076.stdout:1114: finishing write tid 4 to smithi07694878-37 2021-03-15T22:23:56.231 INFO:tasks.rados.rados.0.smithi076.stdout:1117: seq_num 410 ranges {490870=618799,1592676=774144,2793074=97796} 2021-03-15T22:23:56.236 INFO:tasks.rados.rados.0.smithi076.stdout:1117: writing smithi07694878-23 from 490870 to 1109669 tid 1 2021-03-15T22:23:56.244 INFO:tasks.rados.rados.0.smithi076.stdout:1117: writing smithi07694878-23 from 1592676 to 2366820 tid 2 2021-03-15T22:23:56.248 INFO:tasks.rados.rados.0.smithi076.stdout:1117: writing smithi07694878-23 from 2793074 to 2890870 tid 3 2021-03-15T22:23:56.249 INFO:tasks.rados.rados.0.smithi076.stdout:1118: read oid 10 snap -1 2021-03-15T22:23:56.249 INFO:tasks.rados.rados.0.smithi076.stdout:1118: expect (ObjNum 389 snap 104 seq_num 389) 2021-03-15T22:23:56.249 INFO:tasks.rados.rados.0.smithi076.stdout:1119: write (excl) oid 17 current snap is 112 2021-03-15T22:23:56.249 INFO:tasks.rados.rados.0.smithi076.stdout:1114: finishing write tid 5 to smithi07694878-37 2021-03-15T22:23:56.250 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 650 (ObjNum 408 snap 112 seq_num 408) dirty exists 2021-03-15T22:23:56.250 INFO:tasks.rados.rados.0.smithi076.stdout:1114: left oid 37 (ObjNum 408 snap 112 seq_num 408) 2021-03-15T22:23:56.250 INFO:tasks.rados.rados.0.smithi076.stdout:1119: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2021-03-15T22:23:56.254 INFO:tasks.rados.rados.0.smithi076.stdout:1119: writing smithi07694878-17 from 507677 to 918078 tid 1 2021-03-15T22:23:56.261 INFO:tasks.rados.rados.0.smithi076.stdout:1119: writing smithi07694878-17 from 1367511 to 1885313 tid 2 2021-03-15T22:23:56.268 INFO:tasks.rados.rados.0.smithi076.stdout:1119: writing smithi07694878-17 from 2420497 to 2907677 tid 3 2021-03-15T22:23:56.268 INFO:tasks.rados.rados.0.smithi076.stdout:1114: done (4 left) 2021-03-15T22:23:56.268 INFO:tasks.rados.rados.0.smithi076.stdout:1120: write (excl) oid 30 current snap is 112 2021-03-15T22:23:56.269 INFO:tasks.rados.rados.0.smithi076.stdout:1120: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2021-03-15T22:23:56.274 INFO:tasks.rados.rados.0.smithi076.stdout:1120: writing smithi07694878-30 from 524484 to 1010134 tid 1 2021-03-15T22:23:56.282 INFO:tasks.rados.rados.0.smithi076.stdout:1120: writing smithi07694878-30 from 1425993 to 2087453 tid 2 2021-03-15T22:23:56.287 INFO:tasks.rados.rados.0.smithi076.stdout:1120: writing smithi07694878-30 from 2847920 to 2924484 tid 3 2021-03-15T22:23:56.287 INFO:tasks.rados.rados.0.smithi076.stdout:1121: write (excl) oid 40 current snap is 112 2021-03-15T22:23:56.288 INFO:tasks.rados.rados.0.smithi076.stdout:1121: seq_num 413 ranges {541291=560899,1884475=521471} 2021-03-15T22:23:56.293 INFO:tasks.rados.rados.0.smithi076.stdout:1121: writing smithi07694878-40 from 541291 to 1102190 tid 1 2021-03-15T22:23:56.301 INFO:tasks.rados.rados.0.smithi076.stdout:1121: writing smithi07694878-40 from 1884475 to 2405946 tid 2 2021-03-15T22:23:56.301 INFO:tasks.rados.rados.0.smithi076.stdout:1122: copy_from oid 36 from oid 19 current snap is 112 2021-03-15T22:23:56.302 INFO:tasks.rados.rados.0.smithi076.stdout:1123: read oid 43 snap -1 2021-03-15T22:23:56.302 INFO:tasks.rados.rados.0.smithi076.stdout:1123: expect (ObjNum 392 snap 106 seq_num 392) 2021-03-15T22:23:56.302 INFO:tasks.rados.rados.0.smithi076.stdout:1124: copy_from oid 42 from oid 45 current snap is 112 2021-03-15T22:23:56.302 INFO:tasks.rados.rados.0.smithi076.stdout:1125: write oid 24 current snap is 112 2021-03-15T22:23:56.303 INFO:tasks.rados.rados.0.smithi076.stdout:1125: seq_num 414 ranges {558098=636148,1826604=665129} 2021-03-15T22:23:56.307 INFO:tasks.rados.rados.0.smithi076.stdout:1125: writing smithi07694878-24 from 558098 to 1194246 tid 1 2021-03-15T22:23:56.315 INFO:tasks.rados.rados.0.smithi076.stdout:1125: writing smithi07694878-24 from 1826604 to 2491733 tid 2 2021-03-15T22:23:56.316 INFO:tasks.rados.rados.0.smithi076.stdout:1126: rollback oid 1 current snap is 112 2021-03-15T22:23:56.316 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 1 to 109 2021-03-15T22:23:56.316 INFO:tasks.rados.rados.0.smithi076.stdout:1116: expect (ObjNum 399 snap 108 seq_num 399) 2021-03-15T22:23:56.324 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:56.350 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:56.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:56.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:56.411 INFO:tasks.rados.rados.0.smithi076.stdout:1117: finishing write tid 1 to smithi07694878-23 2021-03-15T22:23:56.411 INFO:tasks.rados.rados.0.smithi076.stdout:1117: finishing write tid 2 to smithi07694878-23 2021-03-15T22:23:56.411 INFO:tasks.rados.rados.0.smithi076.stdout:1117: finishing write tid 3 to smithi07694878-23 2021-03-15T22:23:56.411 INFO:tasks.rados.rados.0.smithi076.stdout:1117: finishing write tid 4 to smithi07694878-23 2021-03-15T22:23:56.412 INFO:tasks.rados.rados.0.smithi076.stdout:1117: finishing write tid 5 to smithi07694878-23 2021-03-15T22:23:56.412 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 275 (ObjNum 410 snap 112 seq_num 410) dirty exists 2021-03-15T22:23:56.412 INFO:tasks.rados.rados.0.smithi076.stdout:1117: left oid 23 (ObjNum 410 snap 112 seq_num 410) 2021-03-15T22:23:56.412 INFO:tasks.rados.rados.0.smithi076.stdout:1116: done (10 left) 2021-03-15T22:23:56.413 INFO:tasks.rados.rados.0.smithi076.stdout:1117: done (9 left) 2021-03-15T22:23:56.413 INFO:tasks.rados.rados.0.smithi076.stdout:1127: write oid 22 current snap is 112 2021-03-15T22:23:56.413 INFO:tasks.rados.rados.0.smithi076.stdout:1118: expect (ObjNum 389 snap 104 seq_num 389) 2021-03-15T22:23:56.452 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:56.473 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:56.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:56.475 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:56.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:56 smithi076 conmon[56514]: cluster 2021-03-15T22:23:55.030721+0000 mgr.y (mgr.14144) 547 : cluster [DBG] pgmap v606: 138 pgs: 138 active+clean; 103 MiB data, 444 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 94 op/s; 250 B/s, 4 objects/s recovering 2021-03-15T22:23:56.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:56 smithi076 conmon[56514]: cluster 2021-03-15T22:23:56.064817+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2021-03-15T22:23:56.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:56 smithi076 conmon[62156]: cluster 2021-03-15T22:23:55.030721+0000 mgr.y (mgr.14144) 547 : cluster [DBG] pgmap v606: 138 pgs: 138 active+clean; 103 MiB data, 444 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 94 op/s; 250 B/s, 4 objects/s recovering 2021-03-15T22:23:56.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:56 smithi076 conmon[62156]: cluster 2021-03-15T22:23:56.064817+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2021-03-15T22:23:56.521 INFO:tasks.rados.rados.0.smithi076.stdout:1119: finishing write tid 1 to smithi07694878-17 2021-03-15T22:23:56.522 INFO:tasks.rados.rados.0.smithi076.stdout:1119: finishing write tid 2 to smithi07694878-17 2021-03-15T22:23:56.522 INFO:tasks.rados.rados.0.smithi076.stdout:1119: finishing write tid 3 to smithi07694878-17 2021-03-15T22:23:56.522 INFO:tasks.rados.rados.0.smithi076.stdout:1119: finishing write tid 4 to smithi07694878-17 2021-03-15T22:23:56.522 INFO:tasks.rados.rados.0.smithi076.stdout:1127: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2021-03-15T22:23:56.528 INFO:tasks.rados.rados.0.smithi076.stdout:1127: writing smithi07694878-22 from 574905 to 1286302 tid 1 2021-03-15T22:23:56.535 INFO:tasks.rados.rados.0.smithi076.stdout:1127: writing smithi07694878-22 from 1885086 to 2410226 tid 2 2021-03-15T22:23:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:56 smithi125 conmon[56739]: cluster 2021-03-15T22:23:55.030721+0000 mgr.y (mgr.14144) 547 : cluster [DBG] pgmap v606: 138 pgs: 138 active+clean; 103 MiB data, 444 MiB used, 715 GiB / 715 GiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 94 op/s; 250 B/s, 4 objects/s recovering 2021-03-15T22:23:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:56 smithi125 conmon[56739]: 2021-03-15T22:23:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:56 smithi125 conmon[56739]: cluster 2021-03-15T22:23:56.064817+0000 mon.a (mon.0) 1041 : cluster [DBG] osdmap e321: 8 total, 8 up, 8 in 2021-03-15T22:23:56.538 INFO:tasks.rados.rados.0.smithi076.stdout:1127: writing smithi07694878-22 from 2930189 to 2974905 tid 3 2021-03-15T22:23:56.539 INFO:tasks.rados.rados.0.smithi076.stdout:1118: done (9 left) 2021-03-15T22:23:56.539 INFO:tasks.rados.rados.0.smithi076.stdout:1128: copy_from oid 48 from oid 41 current snap is 112 2021-03-15T22:23:56.539 INFO:tasks.rados.rados.0.smithi076.stdout:1119: finishing write tid 5 to smithi07694878-17 2021-03-15T22:23:56.539 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 99 (ObjNum 411 snap 112 seq_num 411) dirty exists 2021-03-15T22:23:56.540 INFO:tasks.rados.rados.0.smithi076.stdout:1119: left oid 17 (ObjNum 411 snap 112 seq_num 411) 2021-03-15T22:23:56.540 INFO:tasks.rados.rados.0.smithi076.stdout:1120: finishing write tid 1 to smithi07694878-30 2021-03-15T22:23:56.540 INFO:tasks.rados.rados.0.smithi076.stdout:1119: done (9 left) 2021-03-15T22:23:56.540 INFO:tasks.rados.rados.0.smithi076.stdout:1129: write oid 27 current snap is 112 2021-03-15T22:23:56.541 INFO:tasks.rados.rados.0.smithi076.stdout:1120: finishing write tid 2 to smithi07694878-30 2021-03-15T22:23:56.541 INFO:tasks.rados.rados.0.smithi076.stdout:1120: finishing write tid 3 to smithi07694878-30 2021-03-15T22:23:56.541 INFO:tasks.rados.rados.0.smithi076.stdout:1129: seq_num 416 ranges {591712=786646,1943568=668798} 2021-03-15T22:23:56.545 INFO:tasks.rados.rados.0.smithi076.stdout:1129: writing smithi07694878-27 from 591712 to 1378358 tid 1 2021-03-15T22:23:56.553 INFO:tasks.rados.rados.0.smithi076.stdout:1129: writing smithi07694878-27 from 1943568 to 2612366 tid 2 2021-03-15T22:23:56.553 INFO:tasks.rados.rados.0.smithi076.stdout:1130: write (excl) oid 7 current snap is 112 2021-03-15T22:23:56.553 INFO:tasks.rados.rados.0.smithi076.stdout:1120: finishing write tid 4 to smithi07694878-30 2021-03-15T22:23:56.553 INFO:tasks.rados.rados.0.smithi076.stdout:1120: finishing write tid 5 to smithi07694878-30 2021-03-15T22:23:56.554 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 614 (ObjNum 412 snap 112 seq_num 412) dirty exists 2021-03-15T22:23:56.554 INFO:tasks.rados.rados.0.smithi076.stdout:1120: left oid 30 (ObjNum 412 snap 112 seq_num 412) 2021-03-15T22:23:56.554 INFO:tasks.rados.rados.0.smithi076.stdout:1121: finishing write tid 1 to smithi07694878-40 2021-03-15T22:23:56.554 INFO:tasks.rados.rados.0.smithi076.stdout:1130: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2021-03-15T22:23:56.558 INFO:tasks.rados.rados.0.smithi076.stdout:1130: writing smithi07694878-7 from 608519 to 1070414 tid 1 2021-03-15T22:23:56.564 INFO:tasks.rados.rados.0.smithi076.stdout:1130: writing smithi07694878-7 from 1602050 to 2130859 tid 2 2021-03-15T22:23:56.569 INFO:tasks.rados.rados.0.smithi076.stdout:1130: writing smithi07694878-7 from 2585035 to 3008519 tid 3 2021-03-15T22:23:56.570 INFO:tasks.rados.rados.0.smithi076.stdout:1120: done (10 left) 2021-03-15T22:23:56.570 INFO:tasks.rados.rados.0.smithi076.stdout:1131: read oid 21 snap 111 2021-03-15T22:23:56.570 INFO:tasks.rados.rados.0.smithi076.stdout:1131: expect (ObjNum 335 snap 89 seq_num 335) 2021-03-15T22:23:56.570 INFO:tasks.rados.rados.0.smithi076.stdout:1132: copy_from oid 38 from oid 3 current snap is 112 2021-03-15T22:23:56.571 INFO:tasks.rados.rados.0.smithi076.stdout:1133: snap_create 2021-03-15T22:23:56.571 INFO:tasks.rados.rados.0.smithi076.stdout:1123: expect (ObjNum 392 snap 106 seq_num 392) 2021-03-15T22:23:56.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:56.598 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:56.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:56.600 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:56.649 INFO:tasks.rados.rados.0.smithi076.stdout:1121: finishing write tid 2 to smithi07694878-40 2021-03-15T22:23:56.650 INFO:tasks.rados.rados.0.smithi076.stdout:1121: finishing write tid 3 to smithi07694878-40 2021-03-15T22:23:56.650 INFO:tasks.rados.rados.0.smithi076.stdout:1121: finishing write tid 4 to smithi07694878-40 2021-03-15T22:23:56.650 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 654 (ObjNum 413 snap 112 seq_num 413) dirty exists 2021-03-15T22:23:56.650 INFO:tasks.rados.rados.0.smithi076.stdout:1121: left oid 40 (ObjNum 413 snap 112 seq_num 413) 2021-03-15T22:23:56.651 INFO:tasks.rados.rados.0.smithi076.stdout:1122: finishing copy_from to smithi07694878-36 2021-03-15T22:23:56.651 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 399 (ObjNum 387 snap 104 seq_num 387) dirty exists 2021-03-15T22:23:56.651 INFO:tasks.rados.rados.0.smithi076.stdout:1122: finishing copy_from racing read to smithi07694878-36 2021-03-15T22:23:56.651 INFO:tasks.rados.rados.0.smithi076.stdout:1125: finishing write tid 1 to smithi07694878-24 2021-03-15T22:23:56.651 INFO:tasks.rados.rados.0.smithi076.stdout:1126: finishing rollback tid 0 to smithi07694878-1 2021-03-15T22:23:56.652 INFO:tasks.rados.rados.0.smithi076.stdout:1126: finishing rollback tid 1 to smithi07694878-1 2021-03-15T22:23:56.652 INFO:tasks.rados.rados.0.smithi076.stdout:1125: finishing write tid 2 to smithi07694878-24 2021-03-15T22:23:56.652 INFO:tasks.rados.rados.0.smithi076.stdout:1125: finishing write tid 3 to smithi07694878-24 2021-03-15T22:23:56.652 INFO:tasks.rados.rados.0.smithi076.stdout:1125: finishing write tid 4 to smithi07694878-24 2021-03-15T22:23:56.653 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 75 (ObjNum 414 snap 112 seq_num 414) dirty exists 2021-03-15T22:23:56.653 INFO:tasks.rados.rados.0.smithi076.stdout:1125: left oid 24 (ObjNum 414 snap 112 seq_num 414) 2021-03-15T22:23:56.653 INFO:tasks.rados.rados.0.smithi076.stdout:1124: finishing copy_from to smithi07694878-42 2021-03-15T22:23:56.653 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 263 (ObjNum 332 snap 88 seq_num 332) dirty exists 2021-03-15T22:23:56.654 INFO:tasks.rados.rados.0.smithi076.stdout:1126: finishing rollback tid 2 to smithi07694878-1 2021-03-15T22:23:56.654 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 168 (ObjNum 338 snap 91 seq_num 338) dirty exists 2021-03-15T22:23:56.654 INFO:tasks.rados.rados.0.smithi076.stdout:1124: finishing copy_from racing read to smithi07694878-42 2021-03-15T22:23:56.654 INFO:tasks.rados.rados.0.smithi076.stdout:1127: finishing write tid 1 to smithi07694878-22 2021-03-15T22:23:56.655 INFO:tasks.rados.rados.0.smithi076.stdout:1127: finishing write tid 2 to smithi07694878-22 2021-03-15T22:23:56.655 INFO:tasks.rados.rados.0.smithi076.stdout:1127: finishing write tid 3 to smithi07694878-22 2021-03-15T22:23:56.655 INFO:tasks.rados.rados.0.smithi076.stdout:1127: finishing write tid 4 to smithi07694878-22 2021-03-15T22:23:56.655 INFO:tasks.rados.rados.0.smithi076.stdout:1127: finishing write tid 5 to smithi07694878-22 2021-03-15T22:23:56.655 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 619 (ObjNum 415 snap 112 seq_num 415) dirty exists 2021-03-15T22:23:56.656 INFO:tasks.rados.rados.0.smithi076.stdout:1127: left oid 22 (ObjNum 415 snap 112 seq_num 415) 2021-03-15T22:23:56.656 INFO:tasks.rados.rados.0.smithi076.stdout:1128: finishing copy_from to smithi07694878-48 2021-03-15T22:23:56.656 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 233 (ObjNum 370 snap 102 seq_num 370) dirty exists 2021-03-15T22:23:56.656 INFO:tasks.rados.rados.0.smithi076.stdout:1128: finishing copy_from racing read to smithi07694878-48 2021-03-15T22:23:56.656 INFO:tasks.rados.rados.0.smithi076.stdout:1129: finishing write tid 1 to smithi07694878-27 2021-03-15T22:23:56.657 INFO:tasks.rados.rados.0.smithi076.stdout:1129: finishing write tid 2 to smithi07694878-27 2021-03-15T22:23:56.657 INFO:tasks.rados.rados.0.smithi076.stdout:1129: finishing write tid 3 to smithi07694878-27 2021-03-15T22:23:56.657 INFO:tasks.rados.rados.0.smithi076.stdout:1129: finishing write tid 4 to smithi07694878-27 2021-03-15T22:23:56.657 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 226 (ObjNum 416 snap 112 seq_num 416) dirty exists 2021-03-15T22:23:56.658 INFO:tasks.rados.rados.0.smithi076.stdout:1129: left oid 27 (ObjNum 416 snap 112 seq_num 416) 2021-03-15T22:23:56.658 INFO:tasks.rados.rados.0.smithi076.stdout:1130: finishing write tid 1 to smithi07694878-7 2021-03-15T22:23:56.658 INFO:tasks.rados.rados.0.smithi076.stdout:1130: finishing write tid 2 to smithi07694878-7 2021-03-15T22:23:56.658 INFO:tasks.rados.rados.0.smithi076.stdout:1130: finishing write tid 3 to smithi07694878-7 2021-03-15T22:23:56.659 INFO:tasks.rados.rados.0.smithi076.stdout:1130: finishing write tid 4 to smithi07694878-7 2021-03-15T22:23:56.659 INFO:tasks.rados.rados.0.smithi076.stdout:1130: finishing write tid 5 to smithi07694878-7 2021-03-15T22:23:56.659 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 420 (ObjNum 417 snap 112 seq_num 417) dirty exists 2021-03-15T22:23:56.659 INFO:tasks.rados.rados.0.smithi076.stdout:1130: left oid 7 (ObjNum 417 snap 112 seq_num 417) 2021-03-15T22:23:56.659 INFO:tasks.rados.rados.0.smithi076.stdout:1132: finishing copy_from to smithi07694878-38 2021-03-15T22:23:56.660 INFO:tasks.rados.rados.0.smithi076.stdout:1132: got expected ENOENT (src dne) 2021-03-15T22:23:56.660 INFO:tasks.rados.rados.0.smithi076.stdout:1132: finishing copy_from racing read to smithi07694878-38 2021-03-15T22:23:56.660 INFO:tasks.rados.rados.0.smithi076.stdout:1131: expect (ObjNum 335 snap 89 seq_num 335) 2021-03-15T22:23:56.701 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:56.723 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:56.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:56.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:56.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:56.848 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:56.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:56.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:56.868 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:56.922 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:23:56.923 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:23:56.923 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2021-03-15T22:23:56.923 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight-by-utilization 110 3.0 2021-03-15T22:23:56.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:56.972 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:56.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:56.974 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:57.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:57.104 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:57.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:57.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:57.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:57.229 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:57.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:57.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:57.288 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:23:57.289 INFO:tasks.rados.rados.0.smithi076.stdout:1121: done (12 left) 2021-03-15T22:23:57.289 INFO:tasks.rados.rados.0.smithi076.stdout:1122: done (11 left) 2021-03-15T22:23:57.289 INFO:tasks.rados.rados.0.smithi076.stdout:1123: done (10 left) 2021-03-15T22:23:57.289 INFO:tasks.rados.rados.0.smithi076.stdout:1124: done (9 left) 2021-03-15T22:23:57.290 INFO:tasks.rados.rados.0.smithi076.stdout:1125: done (8 left) 2021-03-15T22:23:57.290 INFO:tasks.rados.rados.0.smithi076.stdout:1126: done (7 left) 2021-03-15T22:23:57.290 INFO:tasks.rados.rados.0.smithi076.stdout:1127: done (6 left) 2021-03-15T22:23:57.290 INFO:tasks.rados.rados.0.smithi076.stdout:1128: done (5 left) 2021-03-15T22:23:57.290 INFO:tasks.rados.rados.0.smithi076.stdout:1129: done (4 left) 2021-03-15T22:23:57.291 INFO:tasks.rados.rados.0.smithi076.stdout:1130: done (3 left) 2021-03-15T22:23:57.291 INFO:tasks.rados.rados.0.smithi076.stdout:1131: done (2 left) 2021-03-15T22:23:57.291 INFO:tasks.rados.rados.0.smithi076.stdout:1132: done (1 left) 2021-03-15T22:23:57.291 INFO:tasks.rados.rados.0.smithi076.stdout:1133: done (0 left) 2021-03-15T22:23:57.292 INFO:tasks.rados.rados.0.smithi076.stdout:1134: delete oid 49 current snap is 113 2021-03-15T22:23:57.292 INFO:tasks.rados.rados.0.smithi076.stdout:1134: done (0 left) 2021-03-15T22:23:57.292 INFO:tasks.rados.rados.0.smithi076.stdout:1135: snap_create 2021-03-15T22:23:57.332 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:57.355 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:57.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:57.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:57.428 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:23:57.437 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:23:57.457 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:57.481 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:57.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:57.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:57.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:57 smithi076 conmon[56514]: audit 2021-03-15T22:23:56.409554+0000 mon.a (mon.0) 2021-03-15T22:23:57.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:57 smithi076 conmon[56514]: 1042 : audit [INF] from='client.? 172.21.15.76:0/2211652745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:23:57.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:57 smithi076 conmon[62156]: audit 2021-03-15T22:23:56.409554+0000 2021-03-15T22:23:57.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:57 smithi076 conmon[62156]: mon.a (mon.0) 1042 : audit [INF] from='client.? 172.21.15.76:0/2211652745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:23:57.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:57 smithi125 conmon[56739]: audit 2021-03-15T22:23:56.409554+0000 mon.a (mon.0) 1042 : 2021-03-15T22:23:57.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:57 smithi125 conmon[56739]: audit [INF] from='client.? 172.21.15.76:0/2211652745' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:23:57.585 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:57.609 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:57.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:57.612 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:57.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:57.740 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:57.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:57.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:57.842 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:57.868 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:57.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:57.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:57.971 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:57.996 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:57.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:57.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:58.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:58.121 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:58.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:58.122 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:58.223 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:58.245 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:58.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:58.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:58.288 INFO:tasks.rados.rados.0.smithi076.stdout:1135: done (0 left) 2021-03-15T22:23:58.289 INFO:tasks.rados.rados.0.smithi076.stdout:1136: snap_create 2021-03-15T22:23:58.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:58.355 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-15T22:23:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:58.372 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:58.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:58.474 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:58.497 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:58.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:58.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: cluster 2021-03-15T22:23:57.031159+0000 mgr.y (mgr.14144) 548 : cluster [DBG] pgmap v609: 138 pgs: 138 active+clean; 109 MiB data, 460 MiB used, 715 GiB / 715 GiB avail; 7.7 MiB/s rd, 1.9 MiB/s wr, 100 op/s; 250 B/s, 4 objects/s recovering 2021-03-15T22:23:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: 2021-03-15T22:23:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: cluster 2021-03-15T22:23:57.270617+0000 mon.a (mon.0) 1043 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: audit 2021-03-15T22:23:57.281954+0000 mon.a 2021-03-15T22:23:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: (mon.0) 1044 : audit [INF] from='client.? 172.21.15.76:0/2211652745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:23:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: cluster 2021-03-15T22:23:57.282006+0000 2021-03-15T22:23:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: mon.a (mon.0) 1045 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2021-03-15T22:23:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:23:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: 15T22:23:57.444594+0000 osd.0 (osd.0) 2021-03-15T22:23:58.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: 67 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:23:58.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:23:58.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: 15T22:23:57.460213+0000 osd.6 (osd. 2021-03-15T22:23:58.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: 6) 49 : cluster [DBG] 2021-03-15T22:23:58.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:58 smithi125 conmon[56739]: 4.5 scrub ok 2021-03-15T22:23:58.587 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[56514]: cluster 2021-03-15T22:23:57.031159+0000 mgr.y (mgr.14144 2021-03-15T22:23:58.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[56514]: ) 548 : cluster [DBG] pgmap v609: 138 pgs: 138 active+clean; 109 MiB data, 460 MiB used, 715 GiB / 715 GiB avail; 7.7 MiB/s rd, 1.9 MiB/s wr, 100 op/s; 250 B/s, 4 objects/s recovering 2021-03-15T22:23:58.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[56514]: cluster 2021-03-15T22:23:57.270617+0000 mon.a (mon.0) 1043 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:58.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[56514]: audit 2021-03-15T22:23:57.281954+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? 172.21.15.76:0/2211652745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:23:58.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[56514]: cluster 2021-03-15T22:23:57.282006+0000 mon.a (mon.0) 1045 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2021-03-15T22:23:58.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[56514]: cluster 2021-03-15T22:23:57.444594+0000 osd.0 (osd.0) 67 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:23:58.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[56514]: cluster 2021-03-15T22:23:57.460213+0000 osd.6 (osd.6) 49 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:23:58.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[62156]: cluster 2021-03-15T22:23:57.031159 2021-03-15T22:23:58.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 548 : cluster [DBG] pgmap v609: 138 pgs: 138 active+clean; 109 MiB data, 460 MiB used, 715 GiB / 715 GiB avail; 7.7 MiB/s rd, 1.9 MiB/s wr, 100 op/s; 250 B/s, 4 objects/s recovering 2021-03-15T22:23:58.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[62156]: cluster 2021-03-15T22:23:57.270617+0000 mon.a (mon.0) 1043 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:23:58.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[62156]: audit 2021-03-15T22:23:57.281954+0000 mon.a (mon.0) 1044 : audit [INF] from='client.? 172.21.15.76:0/2211652745' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:23:58.591 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[62156]: cluster 2021-03-15T22:23:57.282006+0000 mon.a (mon.0) 1045 : cluster [DBG] osdmap e322: 8 total, 8 up, 8 in 2021-03-15T22:23:58.591 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[62156]: cluster 2021-03-15T22:23:57.444594+0000 osd.0 (osd.0) 67 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:23:58.591 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:58 smithi076 conmon[62156]: cluster 2021-03-15T22:23:57.460213+0000 osd.6 (osd.6) 49 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:23:58.594 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:58.600 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:23:58.628 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:58.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:58.629 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:23:58.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:23:58.756 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:58.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:58.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:23:58.858 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:58.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:58.882 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:58.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:58.984 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:23:59.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:59.007 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:59.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:23:59.109 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:59.130 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:59.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:59.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:59.233 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:59.264 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:59.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:59.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:59.296 INFO:tasks.rados.rados.0.smithi076.stdout:1136: done (0 left) 2021-03-15T22:23:59.296 INFO:tasks.rados.rados.0.smithi076.stdout:1137: read oid 40 snap -1 2021-03-15T22:23:59.297 INFO:tasks.rados.rados.0.smithi076.stdout:1137: expect (ObjNum 413 snap 112 seq_num 413) 2021-03-15T22:23:59.297 INFO:tasks.rados.rados.0.smithi076.stdout:1138: write (excl) oid 10 current snap is 115 2021-03-15T22:23:59.297 INFO:tasks.rados.rados.0.smithi076.stdout:1138: seq_num 419 ranges {642133=728746,2119014=532478} 2021-03-15T22:23:59.303 INFO:tasks.rados.rados.0.smithi076.stdout:1138: writing smithi07694878-10 from 642133 to 1370879 tid 1 2021-03-15T22:23:59.311 INFO:tasks.rados.rados.0.smithi076.stdout:1138: writing smithi07694878-10 from 2119014 to 2651492 tid 2 2021-03-15T22:23:59.311 INFO:tasks.rados.rados.0.smithi076.stdout:1139: snap_create 2021-03-15T22:23:59.311 INFO:tasks.rados.rados.0.smithi076.stdout:1137: expect (ObjNum 413 snap 112 seq_num 413) 2021-03-15T22:23:59.357 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:23:59.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:23:59.393 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:59.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:59.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:23:59.495 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:23:59.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:59 smithi076 conmon[56514]: cluster 2021-03-15T22:23:58.133374+0000 osd.2 2021-03-15T22:23:59.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:59 smithi076 conmon[56514]: (osd.2) 58 : cluster [DBG] 4.f scrub ok 2021-03-15T22:23:59.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:59 smithi076 conmon[56514]: cluster 2021-03-15T22:23:58.285570+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2021-03-15T22:23:59.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:59 smithi076 conmon[56514]: cluster 2021-03-15T22:23:58.480271+0000 osd.0 (osd.0) 68 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:23:59.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:23:59 smithi076 conmon[56514]: cluster 2021-03-15T22:23:58.936734+0000 osd.5 (osd.5) 60 : cluster [DBG] 6.d scrub ok 2021-03-15T22:23:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:59 smithi076 conmon[62156]: cluster 2021-03-15T22:23:58.133374+0000 osd.2 (osd.2) 58 : cluster [DBG] 4.f scrub ok 2021-03-15T22:23:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:59 smithi076 conmon[62156]: cluster 2021-03-15T22:23:58.285570+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2021-03-15T22:23:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:59 smithi076 conmon[62156]: cluster 2021-03-15T22:23:58.480271+0000 osd.0 (osd.0) 68 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:23:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:23:59 smithi076 conmon[62156]: cluster 2021-03-15T22:23:58.936734+0000 osd.5 (osd.5) 60 : cluster [DBG] 6.d scrub ok 2021-03-15T22:23:59.521 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:59.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:59.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:23:59.553 INFO:tasks.rados.rados.0.smithi076.stdout:1138: finishing write tid 1 to smithi07694878-10 2021-03-15T22:23:59.553 INFO:tasks.rados.rados.0.smithi076.stdout:1138: finishing write tid 2 to smithi07694878-10 2021-03-15T22:23:59.553 INFO:tasks.rados.rados.0.smithi076.stdout:1138: finishing write tid 3 to smithi07694878-10 2021-03-15T22:23:59.554 INFO:tasks.rados.rados.0.smithi076.stdout:1138: finishing write tid 4 to smithi07694878-10 2021-03-15T22:23:59.554 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 200 (ObjNum 419 snap 115 seq_num 419) dirty exists 2021-03-15T22:23:59.554 INFO:tasks.rados.rados.0.smithi076.stdout:1138: left oid 10 (ObjNum 419 snap 115 seq_num 419) 2021-03-15T22:23:59.622 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:59 smithi125 conmon[56739]: cluster 2021-03-15T22:23:58.133374+0000 osd.2 (osd.2) 2021-03-15T22:23:59.622 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:59 smithi125 conmon[56739]: 58 : cluster [DBG] 4.f scrub ok 2021-03-15T22:23:59.622 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:59 smithi125 conmon[56739]: cluster 2021-03-15T22:23:58.285570+0000 mon.a (mon.0) 1046 : cluster [DBG] osdmap e323: 8 total, 8 up, 8 in 2021-03-15T22:23:59.622 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:59 smithi125 conmon[56739]: cluster 2021-03-15T22:23:58.480271+0000 osd.0 (osd. 2021-03-15T22:23:59.623 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:59 smithi125 conmon[56739]: 0) 68 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:23:59.623 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:59 smithi125 conmon[56739]: cluster 2021-03-15T22:23:59.623 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:59 smithi125 conmon[56739]: 2021-03-15T22 2021-03-15T22:23:59.623 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:59 smithi125 conmon[56739]: :23:58.936734+0000 osd.5 (osd.5) 60 : cluster [DBG] 2021-03-15T22:23:59.624 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:23:59 smithi125 conmon[56739]: 6.d scrub ok 2021-03-15T22:23:59.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:23:59.648 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:23:59.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:59.649 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:23:59.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:23:59.776 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:59.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:59.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:23:59.792 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:23:59.809 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:23:59.826 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:23:59.843 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:23:59.861 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:23:59.878 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:23:59.879 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:23:59.897 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:23:59.900 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:23:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:23:59.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:23:59.915 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:23:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:23:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:23:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:23:59.917 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:23:59.918 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:23:59.918 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:23:59.918 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:23:59.918 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:24:00.002 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:00.029 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:00.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:00.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:00.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:23:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:24:00.069 INFO:tasks.rados.rados.0.smithi076.stdout:1137: done (2 left) 2021-03-15T22:24:00.069 INFO:tasks.rados.rados.0.smithi076.stdout:1138: done (1 left) 2021-03-15T22:24:00.069 INFO:tasks.rados.rados.0.smithi076.stdout:1139: done (0 left) 2021-03-15T22:24:00.070 INFO:tasks.rados.rados.0.smithi076.stdout:1140: write (excl) oid 48 current snap is 116 2021-03-15T22:24:00.070 INFO:tasks.rados.rados.0.smithi076.stdout:1140: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2021-03-15T22:24:00.074 INFO:tasks.rados.rados.0.smithi076.stdout:1140: writing smithi07694878-48 from 658940 to 1062935 tid 1 2021-03-15T22:24:00.082 INFO:tasks.rados.rados.0.smithi076.stdout:1140: writing smithi07694878-48 from 1777496 to 2453632 tid 2 2021-03-15T22:24:00.087 INFO:tasks.rados.rados.0.smithi076.stdout:1140: writing smithi07694878-48 from 2950951 to 3058940 tid 3 2021-03-15T22:24:00.087 INFO:tasks.rados.rados.0.smithi076.stdout:1141: read oid 20 snap 109 2021-03-15T22:24:00.088 INFO:tasks.rados.rados.0.smithi076.stdout:1141: expect (ObjNum 376 snap 103 seq_num 376) 2021-03-15T22:24:00.088 INFO:tasks.rados.rados.0.smithi076.stdout:1142: read oid 41 snap -1 2021-03-15T22:24:00.089 INFO:tasks.rados.rados.0.smithi076.stdout:1142: expect (ObjNum 370 snap 102 seq_num 370) 2021-03-15T22:24:00.089 INFO:tasks.rados.rados.0.smithi076.stdout:1143: write (excl) oid 28 current snap is 116 2021-03-15T22:24:00.089 INFO:tasks.rados.rados.0.smithi076.stdout:1143: seq_num 421 ranges {675747=479244,1835978=536147} 2021-03-15T22:24:00.093 INFO:tasks.rados.rados.0.smithi076.stdout:1143: writing smithi07694878-28 from 675747 to 1154991 tid 1 2021-03-15T22:24:00.101 INFO:tasks.rados.rados.0.smithi076.stdout:1143: writing smithi07694878-28 from 1835978 to 2372125 tid 2 2021-03-15T22:24:00.101 INFO:tasks.rados.rados.0.smithi076.stdout:1144: write oid 17 current snap is 116 2021-03-15T22:24:00.101 INFO:tasks.rados.rados.0.smithi076.stdout:1140: finishing write tid 1 to smithi07694878-48 2021-03-15T22:24:00.101 INFO:tasks.rados.rados.0.smithi076.stdout:1142: expect (ObjNum 370 snap 102 seq_num 370) 2021-03-15T22:24:00.132 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:00.157 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:00.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:00.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:00.224 INFO:tasks.rados.rados.0.smithi076.stdout:1140: finishing write tid 2 to smithi07694878-48 2021-03-15T22:24:00.224 INFO:tasks.rados.rados.0.smithi076.stdout:1144: seq_num 422 ranges {692554=554493,1778107=679805,2889444=203110} 2021-03-15T22:24:00.230 INFO:tasks.rados.rados.0.smithi076.stdout:1144: writing smithi07694878-17 from 692554 to 1247047 tid 1 2021-03-15T22:24:00.236 INFO:tasks.rados.rados.0.smithi076.stdout:1144: writing smithi07694878-17 from 1778107 to 2457912 tid 2 2021-03-15T22:24:00.241 INFO:tasks.rados.rados.0.smithi076.stdout:1144: writing smithi07694878-17 from 2889444 to 3092554 tid 3 2021-03-15T22:24:00.241 INFO:tasks.rados.rados.0.smithi076.stdout:1142: done (4 left) 2021-03-15T22:24:00.241 INFO:tasks.rados.rados.0.smithi076.stdout:1145: read oid 13 snap 113 2021-03-15T22:24:00.241 INFO:tasks.rados.rados.0.smithi076.stdout:1145: expect (ObjNum 407 snap 112 seq_num 407) 2021-03-15T22:24:00.242 INFO:tasks.rados.rados.0.smithi076.stdout:1146: read oid 36 snap -1 2021-03-15T22:24:00.242 INFO:tasks.rados.rados.0.smithi076.stdout:1146: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:24:00.242 INFO:tasks.rados.rados.0.smithi076.stdout:1147: snap_remove snap 108 2021-03-15T22:24:00.261 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:00.283 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:00.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:00.285 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:00.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:00.410 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:00.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:00.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:00.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:00.535 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:00.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:00.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:00.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:24:00.638 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:00.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[56514]: cluster 2021-03-15T22:23:58.872018+0000 osd.7 (osd.7) 67 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:24:00.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[56514]: cluster 2021-03-15T22:23:59.031529+0000 mgr.y (mgr.14144) 549 : cluster [DBG] pgmap v612: 138 pgs: 138 active+clean; 118 MiB data, 483 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2021-03-15T22:24:00.652 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[56514]: cluster 2021-03-15T22:23:59.165638+0000 osd.4 (osd.4) 73 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:00.652 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[56514]: cluster 2021-03-15T22:23:59.165897+0000 osd.4 (osd.4) 74 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:00.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[56514]: cluster 2021-03-15T22:23:59.294286+0000 mon.a (mon.0) 1047 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2021-03-15T22:24:00.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[56514]: cluster 2021-03-15T22:23:59.467757+0000 osd.3 (osd.3) 60 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:24:00.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[56514]: cluster 2021-03-15T22:23:59.500673+0000 osd.0 (osd.0) 69 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:24:00.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[56514]: audit 2021-03-15T22:23:59.620395+0000 mgr.y (mgr.14144) 550 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:00.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[56514]: audit 2021-03-15T22:23:59.771992+0000 mon.a (mon. 2021-03-15T22:24:00.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[56514]: 0) 1048 : audit [DBG] from='client.? 172.21.15.76:0/2400077757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:00.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[56514]: cluster 2021-03-15T22:24:00.064357+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2021-03-15T22:24:00.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[62156]: cluster 2021-03-15T22:23:58.872018+0000 osd.7 (osd.7) 67 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:24:00.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[62156]: cluster 2021-03-15T22:23:59.031529+0000 mgr.y (mgr.14144) 549 : cluster [DBG] pgmap v612: 138 pgs: 138 active+clean; 118 MiB data, 483 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2021-03-15T22:24:00.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[62156]: cluster 2021-03-15T22:23:59.165638+0000 osd.4 (osd.4) 73 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:00.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[62156]: cluster 2021-03-15T22:23:59.165897+0000 osd.4 (osd.4) 74 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:00.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[62156]: cluster 2021-03-15T22:23:59.294286+0000 mon.a (mon.0) 1047 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2021-03-15T22:24:00.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[62156]: cluster 2021-03-15T22:23:59.467757+0000 osd.3 (osd.3) 60 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:24:00.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[62156]: cluster 2021-03-15T22:23:59.500673+0000 osd.0 (osd.0) 69 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:24:00.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[62156]: audit 2021-03-15T22:23:59.620395+0000 mgr.y (mgr.14144) 550 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:00.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[62156]: audit 2021-03-15T22:23 2021-03-15T22:24:00.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[62156]: :59.771992+0000 mon.a (mon.0) 1048 : audit [DBG] from='client.? 172.21.15.76:0/2400077757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:00.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:00 smithi076 conmon[62156]: cluster 2021-03-15T22:24:00.064357+0000 mon.a (mon.0) 1049 : cluster [DBG] osdmap e325: 8 total, 8 up, 8 in 2021-03-15T22:24:00.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:00.683 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:00.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:00.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:00.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: cluster 2021-03-15T22:23:58.872018+0000 osd.7 (osd.7) 67 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:24:00.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: cluster 2021-03-15T22:23:59.031529+0000 mgr.y (mgr.14144) 549 : cluster [DBG] pgmap v612: 138 pgs: 138 active+clean; 118 MiB data, 483 MiB used, 715 GiB / 715 GiB avail; 14 MiB/s rd, 3.6 MiB/s wr, 28 op/s 2021-03-15T22:24:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: cluster 2021-03-15T22:23:59.165638+0000 osd.4 (osd.4) 73 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: 2021-03-15T22:24:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: cluster 2021-03-15T22:23:59 2021-03-15T22:24:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: .165897+0000 osd.4 (osd.4) 74 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: 2021-03-15T22:24:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:24:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: -03-15T22:23:59.294286+0000 mon.a ( 2021-03-15T22:24:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: mon.0) 1047 : cluster [DBG] osdmap e324: 8 total, 8 up, 8 in 2021-03-15T22:24:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:24:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: -03-15T22:23:59.467757+0000 2021-03-15T22:24:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: osd.3 (osd.3) 60 : cluster [DBG] 2021-03-15T22:24:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: 7.3 scrub ok 2021-03-15T22:24:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:24:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: 23:59.500673+0000 osd.0 (osd.0 2021-03-15T22:24:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: ) 69 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:24:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:24:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: 23:59.620395+0000 mgr.y (mgr.14144) 550 : 2021-03-15T22:24:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:24:00.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: 22:23:59.771992+0000 mon.a (mon 2021-03-15T22:24:00.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: .0) 1048 : audit [DBG] from='client.? 172.21.15.76:0/2400077757' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:00.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:24:00.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: 03-15T22:24:00.064357+0000 2021-03-15T22:24:00.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: mon.a (mon.0) 1049 : cluster [DBG] 2021-03-15T22:24:00.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:00 smithi125 conmon[56739]: osdmap e325: 8 total, 8 up, 8 in 2021-03-15T22:24:00.816 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:00.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:00.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:00.919 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:00.944 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:00.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:00.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:01.046 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:01.070 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:01.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:01.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:01.074 INFO:tasks.rados.rados.0.smithi076.stdout:1147: done (6 left) 2021-03-15T22:24:01.074 INFO:tasks.rados.rados.0.smithi076.stdout:1148: delete oid 5 current snap is 116 2021-03-15T22:24:01.075 INFO:tasks.rados.rados.0.smithi076.stdout:1140: finishing write tid 3 to smithi07694878-48 2021-03-15T22:24:01.075 INFO:tasks.rados.rados.0.smithi076.stdout:1140: finishing write tid 4 to smithi07694878-48 2021-03-15T22:24:01.075 INFO:tasks.rados.rados.0.smithi076.stdout:1140: finishing write tid 5 to smithi07694878-48 2021-03-15T22:24:01.075 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 270 (ObjNum 420 snap 116 seq_num 420) dirty exists 2021-03-15T22:24:01.076 INFO:tasks.rados.rados.0.smithi076.stdout:1140: left oid 48 (ObjNum 420 snap 116 seq_num 420) 2021-03-15T22:24:01.076 INFO:tasks.rados.rados.0.smithi076.stdout:1141: expect (ObjNum 376 snap 103 seq_num 376) 2021-03-15T22:24:01.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:01.196 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:01.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:01.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:01.214 INFO:tasks.rados.rados.0.smithi076.stdout:1143: finishing write tid 1 to smithi07694878-28 2021-03-15T22:24:01.215 INFO:tasks.rados.rados.0.smithi076.stdout:1143: finishing write tid 2 to smithi07694878-28 2021-03-15T22:24:01.215 INFO:tasks.rados.rados.0.smithi076.stdout:1143: finishing write tid 3 to smithi07694878-28 2021-03-15T22:24:01.216 INFO:tasks.rados.rados.0.smithi076.stdout:1143: finishing write tid 4 to smithi07694878-28 2021-03-15T22:24:01.216 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 659 (ObjNum 421 snap 116 seq_num 421) dirty exists 2021-03-15T22:24:01.217 INFO:tasks.rados.rados.0.smithi076.stdout:1143: left oid 28 (ObjNum 421 snap 116 seq_num 421) 2021-03-15T22:24:01.217 INFO:tasks.rados.rados.0.smithi076.stdout:1144: finishing write tid 1 to smithi07694878-17 2021-03-15T22:24:01.217 INFO:tasks.rados.rados.0.smithi076.stdout:1140: done (6 left) 2021-03-15T22:24:01.218 INFO:tasks.rados.rados.0.smithi076.stdout:1141: done (5 left) 2021-03-15T22:24:01.218 INFO:tasks.rados.rados.0.smithi076.stdout:1143: done (4 left) 2021-03-15T22:24:01.218 INFO:tasks.rados.rados.0.smithi076.stdout:1148: done (3 left) 2021-03-15T22:24:01.219 INFO:tasks.rados.rados.0.smithi076.stdout:1149: copy_from oid 37 from oid 38 current snap is 116 2021-03-15T22:24:01.219 INFO:tasks.rados.rados.0.smithi076.stdout:1144: finishing write tid 2 to smithi07694878-17 2021-03-15T22:24:01.219 INFO:tasks.rados.rados.0.smithi076.stdout:1144: finishing write tid 3 to smithi07694878-17 2021-03-15T22:24:01.219 INFO:tasks.rados.rados.0.smithi076.stdout:1144: finishing write tid 4 to smithi07694878-17 2021-03-15T22:24:01.220 INFO:tasks.rados.rados.0.smithi076.stdout:1144: finishing write tid 5 to smithi07694878-17 2021-03-15T22:24:01.220 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 104 (ObjNum 422 snap 116 seq_num 422) dirty exists 2021-03-15T22:24:01.220 INFO:tasks.rados.rados.0.smithi076.stdout:1144: left oid 17 (ObjNum 422 snap 116 seq_num 422) 2021-03-15T22:24:01.220 INFO:tasks.rados.rados.0.smithi076.stdout:1145: expect (ObjNum 407 snap 112 seq_num 407) 2021-03-15T22:24:01.300 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:01.316 INFO:tasks.rados.rados.0.smithi076.stdout:1146: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:24:01.324 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:01.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:01.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:01.427 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:01.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:01.453 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:01.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:01.468 INFO:tasks.rados.rados.0.smithi076.stdout:1144: done (3 left) 2021-03-15T22:24:01.468 INFO:tasks.rados.rados.0.smithi076.stdout:1145: done (2 left) 2021-03-15T22:24:01.468 INFO:tasks.rados.rados.0.smithi076.stdout:1146: done (1 left) 2021-03-15T22:24:01.468 INFO:tasks.rados.rados.0.smithi076.stdout:1150: read oid 17 snap 111 2021-03-15T22:24:01.469 INFO:tasks.rados.rados.0.smithi076.stdout:1150: expect deleted 2021-03-15T22:24:01.469 INFO:tasks.rados.rados.0.smithi076.stdout:1151: snap_create 2021-03-15T22:24:01.469 INFO:tasks.rados.rados.0.smithi076.stdout:1149: finishing copy_from to smithi07694878-37 2021-03-15T22:24:01.469 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 389 (ObjNum 273 snap 72 seq_num 273) dirty exists 2021-03-15T22:24:01.470 INFO:tasks.rados.rados.0.smithi076.stdout:1149: finishing copy_from racing read to smithi07694878-37 2021-03-15T22:24:01.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[56514]: cluster 2021-03-15T22:24:00.480940+0000 osd.3 2021-03-15T22:24:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[56514]: (osd.3) 61 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:24:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[56514]: cluster 2021-03-15T22:24:01.072312+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2021-03-15T22:24:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[56514]: audit 2021-03-15T22:24:01.086347+0000 mon.a (mon.0) 1051 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:24:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[56514]: audit 2021-03-15T22:24:01.089004+0000 mon.a (mon.0) 1052 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:24:01.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[56514]: audit 2021-03-15T22:24:01.098336+0000 mon.a (mon.0) 1053 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2021-03-15T22:24:01.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[56514]: audit 2021-03-15T22:24:01.098543+0000 mon.a (mon.0) 1054 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 2]}]: dispatch 2021-03-15T22:24:01.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[56514]: audit 2021-03-15T22:24:01.098703+0000 mon.a (mon.0) 1055 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2021-03-15T22:24:01.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[56514]: audit 2021-03-15T22:24:01.098854+0000 mon.a (mon.0) 1056 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2021-03-15T22:24:01.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: cluster 2021-03-15T22:24:00.480940+0000 2021-03-15T22:24:01.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: osd.3 (osd.3) 61 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:24:01.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: cluster 2021-03-15T22:24: 2021-03-15T22:24:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: 01.072312+0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2021-03-15T22:24:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: audit 2021 2021-03-15T22:24:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: -03-15T22:24:01.086347 2021-03-15T22:24:01.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1051 : audit [INF] 2021-03-15T22:24:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:24:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: audit 2021-03-15T22:24: 2021-03-15T22:24:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: 01.089004+0000 mon.a (mon.0) 1052 : 2021-03-15T22:24:01.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:24:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:24:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: 24:01.098336+0000 mon.a (mon.0) 2021-03-15T22:24:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: 1053 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2021-03-15T22:24:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:24:01.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: 22:24:01.098543+0000 mon.a (mon. 2021-03-15T22:24:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: 0) 1054 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 2]}]: dispatch 2021-03-15T22:24:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:24:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: 03-15T22:24:01.098703+0000 2021-03-15T22:24:01.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: mon.a (mon.0) 1055 : audit [INF] 2021-03-15T22:24:01.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2021-03-15T22:24:01.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: audit 2021 2021-03-15T22:24:01.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: -03-15T22:24:01.098854 2021-03-15T22:24:01.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1056 : audit 2021-03-15T22:24:01.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:01 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2021-03-15T22:24:01.555 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:01.579 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:01.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:01.581 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:01.682 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:01.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:01.707 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:01.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:01.745 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:01.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: cluster 2021-03-15T22:24:00.480940+0000 osd.3 (osd.3) 61 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:24:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: cluster 2021-03-15T22:24:01.072312 2021-03-15T22:24:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1050 : cluster [DBG] osdmap e326: 8 total, 8 up, 8 in 2021-03-15T22:24:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: audit 2021-03-15T22:24:01.086347+0000 mon.a (mon.0) 1051 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:24:01.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:24:01.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: 24:01.089004+0000 mon.a (mon.0) 1052 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:24:01.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: 2021-03-15T22:24:01.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:24:01.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: 03-15T22:24:01 2021-03-15T22:24:01.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: .098336+0000 mon.a (mon.0 2021-03-15T22:24:01.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: ) 1053 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]: dispatch 2021-03-15T22:24:01.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:24:01.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: 03-15T22:24:01.098543+0000 2021-03-15T22:24:01.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: mon.a (mon.0) 1054 : 2021-03-15T22:24:01.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 2]}]: dispatch 2021-03-15T22:24:01.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: audit 2021 2021-03-15T22:24:01.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: -03-15T22:24:01.098703+0000 2021-03-15T22:24:01.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: mon.a (mon.0) 1055 : 2021-03-15T22:24:01.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]: dispatch 2021-03-15T22:24:01.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: audit 2021 2021-03-15T22:24:01.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: -03-15T22:24:01. 2021-03-15T22:24:01.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: 098854+0000 mon.a (mon.0 2021-03-15T22:24:01.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:01 smithi125 conmon[56739]: ) 1056 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]: dispatch 2021-03-15T22:24:01.808 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:01.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:01.837 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:01.837 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:01.939 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:01.962 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:01.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:01.964 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:02.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:02.076 INFO:tasks.rados.rados.0.smithi076.stdout:1149: done (2 left) 2021-03-15T22:24:02.076 INFO:tasks.rados.rados.0.smithi076.stdout:1150: done (1 left) 2021-03-15T22:24:02.077 INFO:tasks.rados.rados.0.smithi076.stdout:1151: done (0 left) 2021-03-15T22:24:02.077 INFO:tasks.rados.rados.0.smithi076.stdout:1152: rollback oid 37 current snap is 117 2021-03-15T22:24:02.077 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 37 to 111 2021-03-15T22:24:02.077 INFO:tasks.rados.rados.0.smithi076.stdout:1153: snap_remove snap 116 2021-03-15T22:24:02.094 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:02.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:02.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:02.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:02.219 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:02.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:02.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:02.321 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:02.347 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:02.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:02.349 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:02.450 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:02.473 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:02.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:02.475 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:02.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[62156]: cluster 2021-03-15T22:24:00.938805+0000 osd.4 (osd.4) 75 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:24:02.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[62156]: cluster 2021-03-15T22:24:01.031975+0000 mgr.y (mgr.14144) 551 : cluster [DBG] pgmap v615: 138 pgs: 138 active+clean; 128 MiB data, 508 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 28 op/s 2021-03-15T22:24:02.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[62156]: cluster 2021-03-15T22:24:01.499094+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.b scrub ok 2021-03-15T22:24:02.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[62156]: audit 2021-03-15T22:24:02.073882+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]': finished 2021-03-15T22:24:02.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[62156]: audit 2021-03-15T22:24:02.074014+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 2]}]': finished 2021-03-15T22:24:02.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[62156]: audit 2021-03-15T22:24:02.074092+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2021-03-15T22:24:02.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[62156]: audit 2021-03-15T22:24:02.074237+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2021-03-15T22:24:02.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[62156]: cluster 2021-03-15T22:24:02.074335+0000 mon.a (mon.0) 1061 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2021-03-15T22:24:02.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[56514]: cluster 2021-03-15T22:24:00.938805+0000 osd.4 (osd.4) 75 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:24:02.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[56514]: cluster 2021-03-15T22:24:01.031975+0000 mgr.y (mgr.14144) 551 : cluster [DBG] pgmap v615: 138 pgs: 138 active+clean; 128 MiB data, 508 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 28 op/s 2021-03-15T22:24:02.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[56514]: cluster 2021-03-15T22:24:01.499094+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.b scrub ok 2021-03-15T22:24:02.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[56514]: audit 2021-03-15T22:24:02.073882+0000 mon.a (mon.0) 1057 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]': finished 2021-03-15T22:24:02.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[56514]: audit 2021-03-15T 2021-03-15T22:24:02.552 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[56514]: 22:24:02.074014+0000 mon.a (mon.0) 1058 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 2]}]': finished 2021-03-15T22:24:02.552 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[56514]: audit 2021-03-15T22:24:02.074092+0000 mon.a (mon.0) 1059 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2021-03-15T22:24:02.552 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[56514]: audit 2021-03-15T22:24:02.074237+0000 mon.a (mon.0) 1060 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2021-03-15T22:24:02.552 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:02 smithi076 conmon[56514]: cluster 2021-03-15T22:24:02.074335+0000 mon.a (mon.0) 1061 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2021-03-15T22:24:02.576 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:02.602 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:02.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:02.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:02.704 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:02.727 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:02.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:02.729 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:02.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: cluster 2021-03-15T22:24:00.938805+0000 osd.4 (osd.4) 75 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:24:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: cluster 2021-03-15T22:24:01.031975+0000 mgr.y (mgr.14144) 551 : cluster [DBG] pgmap v615: 138 pgs: 138 active+clean; 128 MiB data, 508 MiB used, 715 GiB / 715 GiB avail; 15 MiB/s rd, 3.8 MiB/s wr, 28 op/s 2021-03-15T22:24:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: cluster 2021-03-15T22:24:01.499094+0000 osd.0 (osd.0) 70 : cluster [DBG] 4.b scrub ok 2021-03-15T22:24:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:24:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: 03-15T22:24:02. 2021-03-15T22:24:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: 073882+0000 mon.a (mon.0 2021-03-15T22:24:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: ) 1057 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 2]}]': finished 2021-03-15T22:24:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: audit 2021-03-15T22:24:02.074014+0000 mon.a (mon 2021-03-15T22:24:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: .0) 1058 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 2]}]': finished 2021-03-15T22:24:02.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:24:02.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: 03-15T22:24:02.074092+0000 2021-03-15T22:24:02.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: mon.a (mon.0) 1059 : audit 2021-03-15T22:24:02.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.a", "id": [1, 2]}]': finished 2021-03-15T22:24:02.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: audit 2021 2021-03-15T22:24:02.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: -03-15T22:24:02 2021-03-15T22:24:02.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: .074237+0000 mon.a (mon. 2021-03-15T22:24:02.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: 0) 1060 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [1, 5]}]': finished 2021-03-15T22:24:02.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: 2021-03-15T22:24:02.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:24:02.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: 03-15T22:24:02. 2021-03-15T22:24:02.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: 074335+0000 mon.a (mon.0) 2021-03-15T22:24:02.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:02 smithi125 conmon[56739]: 1061 : cluster [DBG] osdmap e327: 8 total, 8 up, 8 in 2021-03-15T22:24:02.829 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:02.856 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:02.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:02.858 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:02.924 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:24:02.925 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:24:02.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:24:02.925 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:24:02.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.926 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:257 7.3 7:c7825e8f:::smithi07694878-26:head [copy-from ver 3] snapc 5=[] ondisk+write+known_if_redirected e103)", 2021-03-15T22:24:02.926 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:18.250698+0000", 2021-03-15T22:24:02.926 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 224.33877998400001, 2021-03-15T22:24:02.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013491402, 2021-03-15T22:24:02.927 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:02.927 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:02.927 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:02.928 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:02.928 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:02.928 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 257 2021-03-15T22:24:02.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.928 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:02.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:02.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250698+0000", 2021-03-15T22:24:02.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:02.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:02.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250698+0000", 2021-03-15T22:24:02.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8069999999999999e-06 2021-03-15T22:24:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250703+0000", 2021-03-15T22:24:02.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2169999999999997e-06 2021-03-15T22:24:02.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:02.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250710+0000", 2021-03-15T22:24:02.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9879999999999999e-05 2021-03-15T22:24:02.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:02.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250730+0000", 2021-03-15T22:24:02.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2840000000000006e-06 2021-03-15T22:24:02.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:02.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250739+0000", 2021-03-15T22:24:02.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9257000000000005e-05 2021-03-15T22:24:02.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:02.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250808+0000", 2021-03-15T22:24:02.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8200000000000003e-06 2021-03-15T22:24:02.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:24:02.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250817+0000", 2021-03-15T22:24:02.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001712873 2021-03-15T22:24:02.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:02.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.252529+0000", 2021-03-15T22:24:02.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2012e-05 2021-03-15T22:24:02.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:02.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.252612+0000", 2021-03-15T22:24:02.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055665840000000003 2021-03-15T22:24:02.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:24:02.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.258178+0000", 2021-03-15T22:24:02.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052418129999999997 2021-03-15T22:24:02.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:02.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.263420+0000", 2021-03-15T22:24:02.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029105599999999999 2021-03-15T22:24:02.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:02.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.263711+0000", 2021-03-15T22:24:02.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00039116800000000001 2021-03-15T22:24:02.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:02.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264102+0000", 2021-03-15T22:24:02.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3337000000000002e-05 2021-03-15T22:24:02.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:02.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264145+0000", 2021-03-15T22:24:02.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4294e-05 2021-03-15T22:24:02.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:02.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264190+0000", 2021-03-15T22:24:02.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013491402 2021-03-15T22:24:02.950 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:02.950 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:02.950 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:02.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.951 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:258 7.3 7:c7825e8f:::smithi07694878-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e103)", 2021-03-15T22:24:02.951 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:18.250762+0000", 2021-03-15T22:24:02.951 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 224.33871653099999, 2021-03-15T22:24:02.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013601281999999999, 2021-03-15T22:24:02.952 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:02.952 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:02.952 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:02.953 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:02.953 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:02.953 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 258 2021-03-15T22:24:02.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.954 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:02.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:02.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250762+0000", 2021-03-15T22:24:02.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:02.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:02.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250762+0000", 2021-03-15T22:24:02.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.528e-06 2021-03-15T22:24:02.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:02.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250763+0000", 2021-03-15T22:24:02.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8560000000000001e-06 2021-03-15T22:24:02.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:02.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250765+0000", 2021-03-15T22:24:02.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9169999999999995e-06 2021-03-15T22:24:02.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:02.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250774+0000", 2021-03-15T22:24:02.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9060000000000001e-06 2021-03-15T22:24:02.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:02.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250779+0000", 2021-03-15T22:24:02.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9030999999999995e-05 2021-03-15T22:24:02.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:02.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250868+0000", 2021-03-15T22:24:02.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1899999999999998e-06 2021-03-15T22:24:02.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:24:02.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.250875+0000", 2021-03-15T22:24:02.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018793950000000001 2021-03-15T22:24:02.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:02.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.252755+0000", 2021-03-15T22:24:02.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0155999999999999e-05 2021-03-15T22:24:02.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:02.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.252805+0000", 2021-03-15T22:24:02.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0077066119999999998 2021-03-15T22:24:02.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:02.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.260511+0000", 2021-03-15T22:24:02.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8893000000000006e-05 2021-03-15T22:24:02.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:02.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.260580+0000", 2021-03-15T22:24:02.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036392949999999999 2021-03-15T22:24:02.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:02.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264220+0000", 2021-03-15T22:24:02.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8607999999999999e-05 2021-03-15T22:24:02.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:02.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264278+0000", 2021-03-15T22:24:02.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4895000000000006e-05 2021-03-15T22:24:02.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:02.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:18.264363+0000", 2021-03-15T22:24:02.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013601281999999999 2021-03-15T22:24:02.976 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:02.976 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:02.976 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:02.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.977 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:539 7.2 7:45ee0f2f:::smithi07694878-6:head [copy-from ver 16] snapc 1b=[1b,19,18,17] ondisk+write+known_if_redirected e123)", 2021-03-15T22:24:02.977 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:38.108937+0000", 2021-03-15T22:24:02.978 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 204.48054084699999, 2021-03-15T22:24:02.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018433483000000001, 2021-03-15T22:24:02.978 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:02.978 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:02.979 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:02.979 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:02.979 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:02.979 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 539 2021-03-15T22:24:02.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.980 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:02.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:02.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108937+0000", 2021-03-15T22:24:02.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:02.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:02.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108937+0000", 2021-03-15T22:24:02.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.977e-06 2021-03-15T22:24:02.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:02.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108940+0000", 2021-03-15T22:24:02.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7489999999999999e-06 2021-03-15T22:24:02.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:02.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108943+0000", 2021-03-15T22:24:02.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.348e-05 2021-03-15T22:24:02.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:02.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108957+0000", 2021-03-15T22:24:02.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0370000000000002e-06 2021-03-15T22:24:02.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:02.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108962+0000", 2021-03-15T22:24:02.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5223999999999999e-05 2021-03-15T22:24:02.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:02.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.109007+0000", 2021-03-15T22:24:02.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7335999999999994e-05 2021-03-15T22:24:02.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:02.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.109074+0000", 2021-03-15T22:24:02.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007106991 2021-03-15T22:24:02.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:24:02.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.116181+0000", 2021-03-15T22:24:02.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010935861999999999 2021-03-15T22:24:02.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:02.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127117+0000", 2021-03-15T22:24:02.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4037e-05 2021-03-15T22:24:02.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:02.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127151+0000", 2021-03-15T22:24:02.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138885 2021-03-15T22:24:02.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:02.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127290+0000", 2021-03-15T22:24:02.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6146000000000003e-05 2021-03-15T22:24:02.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:02.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127326+0000", 2021-03-15T22:24:02.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4759000000000003e-05 2021-03-15T22:24:02.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:02.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127371+0000", 2021-03-15T22:24:02.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018433483000000001 2021-03-15T22:24:02.996 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:02.997 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:02.997 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:02.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:02.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:02.998 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:540 7.2 7:45ee0f2f:::smithi07694878-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e123)", 2021-03-15T22:24:02.998 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:20:38.108970+0000", 2021-03-15T22:24:02.999 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 204.48050808799999, 2021-03-15T22:24:02.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018548431000000001, 2021-03-15T22:24:02.999 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:02.999 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:02.999 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.000 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.000 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.000 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 540 2021-03-15T22:24:03.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.001 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108970+0000", 2021-03-15T22:24:03.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108970+0000", 2021-03-15T22:24:03.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2299999999999999e-07 2021-03-15T22:24:03.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108971+0000", 2021-03-15T22:24:03.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.35e-06 2021-03-15T22:24:03.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108973+0000", 2021-03-15T22:24:03.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9750000000000001e-06 2021-03-15T22:24:03.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108976+0000", 2021-03-15T22:24:03.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5330000000000001e-06 2021-03-15T22:24:03.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.108979+0000", 2021-03-15T22:24:03.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000204456 2021-03-15T22:24:03.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.109183+0000", 2021-03-15T22:24:03.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3643000000000003e-05 2021-03-15T22:24:03.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:03.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.109217+0000", 2021-03-15T22:24:03.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011687242 2021-03-15T22:24:03.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.120904+0000", 2021-03-15T22:24:03.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010511299999999999 2021-03-15T22:24:03.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:03.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.121009+0000", 2021-03-15T22:24:03.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0063902330000000004 2021-03-15T22:24:03.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127400+0000", 2021-03-15T22:24:03.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3569000000000002e-05 2021-03-15T22:24:03.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127453+0000", 2021-03-15T22:24:03.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5394000000000004e-05 2021-03-15T22:24:03.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:20:38.127519+0000", 2021-03-15T22:24:03.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018548431000000001 2021-03-15T22:24:03.016 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.016 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.016 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.017 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:835 7.a 7:5bc7f316:::smithi07694878-38:head [sparse-read 0~1482386] snapc 0=[] ondisk+read+known_if_redirected e149)", 2021-03-15T22:24:03.017 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018897+0000", 2021-03-15T22:24:03.017 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 178.570580871, 2021-03-15T22:24:03.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012112934000000001, 2021-03-15T22:24:03.018 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.018 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:03.018 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.019 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.019 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.019 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 835 2021-03-15T22:24:03.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.020 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018897+0000", 2021-03-15T22:24:03.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018897+0000", 2021-03-15T22:24:03.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8299999999999997e-07 2021-03-15T22:24:03.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018898+0000", 2021-03-15T22:24:03.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1999999999999999e-07 2021-03-15T22:24:03.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018899+0000", 2021-03-15T22:24:03.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8559999999999998e-06 2021-03-15T22:24:03.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018902+0000", 2021-03-15T22:24:03.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4490000000000002e-06 2021-03-15T22:24:03.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018904+0000", 2021-03-15T22:24:03.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015147500000000001 2021-03-15T22:24:03.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.019056+0000", 2021-03-15T22:24:03.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6529999999999999e-06 2021-03-15T22:24:03.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:24:03.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.019061+0000", 2021-03-15T22:24:03.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011683427999999999 2021-03-15T22:24:03.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.030745+0000", 2021-03-15T22:24:03.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011034799999999999 2021-03-15T22:24:03.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.030855+0000", 2021-03-15T22:24:03.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000155422 2021-03-15T22:24:03.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.031010+0000", 2021-03-15T22:24:03.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012112934000000001 2021-03-15T22:24:03.032 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.032 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.033 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.033 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:836 7.3 7:c458c800:::smithi07694878-3:head [copy-from ver 127] snapc 34=[34,32,30] ondisk+write+known_if_redirected e149)", 2021-03-15T22:24:03.034 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018906+0000", 2021-03-15T22:24:03.034 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 178.57057195600001, 2021-03-15T22:24:03.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024704783000000001, 2021-03-15T22:24:03.034 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.034 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:03.035 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.035 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.035 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.035 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 836 2021-03-15T22:24:03.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.036 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018906+0000", 2021-03-15T22:24:03.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018906+0000", 2021-03-15T22:24:03.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.209e-06 2021-03-15T22:24:03.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.038 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018908+0000", 2021-03-15T22:24:03.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9100000000000003e-07 2021-03-15T22:24:03.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018908+0000", 2021-03-15T22:24:03.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2780000000000002e-06 2021-03-15T22:24:03.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018912+0000", 2021-03-15T22:24:03.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9469999999999999e-06 2021-03-15T22:24:03.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018915+0000", 2021-03-15T22:24:03.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0328000000000003e-05 2021-03-15T22:24:03.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018965+0000", 2021-03-15T22:24:03.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7919999999999993e-06 2021-03-15T22:24:03.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:24:03.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018973+0000", 2021-03-15T22:24:03.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025313699999999998 2021-03-15T22:24:03.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021504+0000", 2021-03-15T22:24:03.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4449000000000006e-05 2021-03-15T22:24:03.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021568+0000", 2021-03-15T22:24:03.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071715939999999999 2021-03-15T22:24:03.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:24:03.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.028740+0000", 2021-03-15T22:24:03.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012489007 2021-03-15T22:24:03.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:03.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.041229+0000", 2021-03-15T22:24:03.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001956981 2021-03-15T22:24:03.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:03.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043186+0000", 2021-03-15T22:24:03.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000330864 2021-03-15T22:24:03.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:03.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043517+0000", 2021-03-15T22:24:03.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8581000000000001e-05 2021-03-15T22:24:03.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:03.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043565+0000", 2021-03-15T22:24:03.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5692000000000002e-05 2021-03-15T22:24:03.052 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043611+0000", 2021-03-15T22:24:03.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024704783000000001 2021-03-15T22:24:03.053 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.053 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.054 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.054 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:837 7.3 7:c458c800:::smithi07694878-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e149)", 2021-03-15T22:24:03.054 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018929+0000", 2021-03-15T22:24:03.055 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 178.57054940399999, 2021-03-15T22:24:03.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024813472, 2021-03-15T22:24:03.055 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.055 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:03.055 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.056 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.056 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.056 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 837 2021-03-15T22:24:03.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.057 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018929+0000", 2021-03-15T22:24:03.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018929+0000", 2021-03-15T22:24:03.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0210000000000001e-06 2021-03-15T22:24:03.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018930+0000", 2021-03-15T22:24:03.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1789999999999997e-06 2021-03-15T22:24:03.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018939+0000", 2021-03-15T22:24:03.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7000000000000003e-06 2021-03-15T22:24:03.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018949+0000", 2021-03-15T22:24:03.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0043e-05 2021-03-15T22:24:03.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.064 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018959+0000", 2021-03-15T22:24:03.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2956000000000001e-05 2021-03-15T22:24:03.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.065 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018992+0000", 2021-03-15T22:24:03.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6950000000000002e-06 2021-03-15T22:24:03.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:24:03.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018996+0000", 2021-03-15T22:24:03.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026765769999999999 2021-03-15T22:24:03.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021672+0000", 2021-03-15T22:24:03.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1886999999999999e-05 2021-03-15T22:24:03.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:03.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021734+0000", 2021-03-15T22:24:03.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015444232 2021-03-15T22:24:03.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.037178+0000", 2021-03-15T22:24:03.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8273999999999998e-05 2021-03-15T22:24:03.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:03.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.037236+0000", 2021-03-15T22:24:03.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064013719999999998 2021-03-15T22:24:03.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043638+0000", 2021-03-15T22:24:03.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4591e-05 2021-03-15T22:24:03.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043682+0000", 2021-03-15T22:24:03.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9945000000000003e-05 2021-03-15T22:24:03.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043742+0000", 2021-03-15T22:24:03.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024813472 2021-03-15T22:24:03.075 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.076 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.076 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.077 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:838 7.a 7:53dee572:::smithi07694878-35:head [write 719366~478716,stat] snapc 34=[34,32,30] ondisk+write+known_if_redirected e149)", 2021-03-15T22:24:03.077 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.025398+0000", 2021-03-15T22:24:03.077 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 178.56408064600001, 2021-03-15T22:24:03.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018168215000000001, 2021-03-15T22:24:03.077 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.078 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:03.078 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.078 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.078 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.079 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 838 2021-03-15T22:24:03.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.079 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025398+0000", 2021-03-15T22:24:03.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025398+0000", 2021-03-15T22:24:03.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024823199999999998 2021-03-15T22:24:03.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.082 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025646+0000", 2021-03-15T22:24:03.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8279999999999999e-06 2021-03-15T22:24:03.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025649+0000", 2021-03-15T22:24:03.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2312e-05 2021-03-15T22:24:03.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025661+0000", 2021-03-15T22:24:03.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3240000000000002e-06 2021-03-15T22:24:03.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.025666+0000", 2021-03-15T22:24:03.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053758139999999996 2021-03-15T22:24:03.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.031042+0000", 2021-03-15T22:24:03.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5798999999999997e-05 2021-03-15T22:24:03.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.031118+0000", 2021-03-15T22:24:03.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027348699999999999 2021-03-15T22:24:03.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:24:03.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.031391+0000", 2021-03-15T22:24:03.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023824789999999998 2021-03-15T22:24:03.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:03.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.033774+0000", 2021-03-15T22:24:03.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024423019999999999 2021-03-15T22:24:03.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:03.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.036216+0000", 2021-03-15T22:24:03.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071806320000000002 2021-03-15T22:24:03.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:03.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043397+0000", 2021-03-15T22:24:03.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1420000000000005e-05 2021-03-15T22:24:03.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:03.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043458+0000", 2021-03-15T22:24:03.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000107586 2021-03-15T22:24:03.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043566+0000", 2021-03-15T22:24:03.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018168215000000001 2021-03-15T22:24:03.097 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.097 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.097 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.098 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1378 7.a 7:53dee572:::smithi07694878-35:head [copy-from ver 3] snapc 5b=[5a,56] ondisk+write+known_if_redirected e190)", 2021-03-15T22:24:03.098 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:45.023845+0000", 2021-03-15T22:24:03.098 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.56563371199999, 2021-03-15T22:24:03.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023497436, 2021-03-15T22:24:03.099 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.099 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:03.099 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.099 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.100 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.100 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1378 2021-03-15T22:24:03.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.100 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023845+0000", 2021-03-15T22:24:03.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023845+0000", 2021-03-15T22:24:03.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5330000000000002e-06 2021-03-15T22:24:03.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023850+0000", 2021-03-15T22:24:03.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6090000000000003e-06 2021-03-15T22:24:03.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023855+0000", 2021-03-15T22:24:03.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9403e-05 2021-03-15T22:24:03.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023874+0000", 2021-03-15T22:24:03.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6260000000000003e-06 2021-03-15T22:24:03.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023882+0000", 2021-03-15T22:24:03.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3671999999999997e-05 2021-03-15T22:24:03.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023935+0000", 2021-03-15T22:24:03.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2021-03-15T22:24:03.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.110 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:24:03.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023945+0000", 2021-03-15T22:24:03.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086676960000000008 2021-03-15T22:24:03.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.032613+0000", 2021-03-15T22:24:03.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2980999999999996e-05 2021-03-15T22:24:03.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.032706+0000", 2021-03-15T22:24:03.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080630400000000005 2021-03-15T22:24:03.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:24:03.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.040769+0000", 2021-03-15T22:24:03.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035132739999999998 2021-03-15T22:24:03.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:03.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.044282+0000", 2021-03-15T22:24:03.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002886647 2021-03-15T22:24:03.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:03.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047169+0000", 2021-03-15T22:24:03.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9790999999999998e-05 2021-03-15T22:24:03.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:03.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047219+0000", 2021-03-15T22:24:03.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9148999999999999e-05 2021-03-15T22:24:03.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:03.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047278+0000", 2021-03-15T22:24:03.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3989e-05 2021-03-15T22:24:03.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047342+0000", 2021-03-15T22:24:03.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023497436 2021-03-15T22:24:03.120 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.120 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.121 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1379 7.a 7:53dee572:::smithi07694878-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e190)", 2021-03-15T22:24:03.122 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:45.023893+0000", 2021-03-15T22:24:03.122 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 137.565585782, 2021-03-15T22:24:03.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023671298, 2021-03-15T22:24:03.123 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.123 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:03.123 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.123 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.124 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.124 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1379 2021-03-15T22:24:03.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.124 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023893+0000", 2021-03-15T22:24:03.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023893+0000", 2021-03-15T22:24:03.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4050000000000001e-06 2021-03-15T22:24:03.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023894+0000", 2021-03-15T22:24:03.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.657e-06 2021-03-15T22:24:03.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023896+0000", 2021-03-15T22:24:03.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6059999999999999e-06 2021-03-15T22:24:03.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023902+0000", 2021-03-15T22:24:03.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4329999999999997e-06 2021-03-15T22:24:03.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023907+0000", 2021-03-15T22:24:03.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6927000000000004e-05 2021-03-15T22:24:03.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023984+0000", 2021-03-15T22:24:03.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.563e-06 2021-03-15T22:24:03.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:24:03.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023987+0000", 2021-03-15T22:24:03.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0094145919999999994 2021-03-15T22:24:03.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.033402+0000", 2021-03-15T22:24:03.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8862999999999997e-05 2021-03-15T22:24:03.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:03.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.033441+0000", 2021-03-15T22:24:03.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010337974999999999 2021-03-15T22:24:03.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.043779+0000", 2021-03-15T22:24:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3967000000000003e-05 2021-03-15T22:24:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:03.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.043873+0000", 2021-03-15T22:24:03.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035103690000000002 2021-03-15T22:24:03.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047383+0000", 2021-03-15T22:24:03.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9901000000000005e-05 2021-03-15T22:24:03.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047473+0000", 2021-03-15T22:24:03.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1039999999999996e-05 2021-03-15T22:24:03.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047564+0000", 2021-03-15T22:24:03.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023671298 2021-03-15T22:24:03.142 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.142 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.143 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.143 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2260 7.2 7:42f18a95:::smithi07694878-45:8d [sparse-read 0~2486556,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e264)", 2021-03-15T22:24:03.144 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.527310+0000", 2021-03-15T22:24:03.144 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.062168669999998, 2021-03-15T22:24:03.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53810241999999997, 2021-03-15T22:24:03.144 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.145 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:03.145 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.145 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.145 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.145 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2260 2021-03-15T22:24:03.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.146 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527310+0000", 2021-03-15T22:24:03.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527310+0000", 2021-03-15T22:24:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4089999999999996e-06 2021-03-15T22:24:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527314+0000", 2021-03-15T22:24:03.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2021-03-15T22:24:03.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527317+0000", 2021-03-15T22:24:03.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7672999999999998e-05 2021-03-15T22:24:03.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527335+0000", 2021-03-15T22:24:03.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6059999999999999e-06 2021-03-15T22:24:03.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527342+0000", 2021-03-15T22:24:03.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2244999999999996e-05 2021-03-15T22:24:03.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527424+0000", 2021-03-15T22:24:03.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.431e-06 2021-03-15T22:24:03.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:24:03.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527433+0000", 2021-03-15T22:24:03.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53064125699999998 2021-03-15T22:24:03.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.058075+0000", 2021-03-15T22:24:03.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015664499999999999 2021-03-15T22:24:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.058231+0000", 2021-03-15T22:24:03.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071808219999999999 2021-03-15T22:24:03.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.065412+0000", 2021-03-15T22:24:03.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53810241999999997 2021-03-15T22:24:03.159 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.159 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.160 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.160 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2261 7.2 7:42f18a95:::smithi07694878-45:8d [sparse-read 0~2486556] snapc 0=[] ondisk+read+known_if_redirected e264)", 2021-03-15T22:24:03.160 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.527444+0000", 2021-03-15T22:24:03.161 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.062034439000001, 2021-03-15T22:24:03.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53896513499999998, 2021-03-15T22:24:03.161 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.161 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:03.162 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.162 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.162 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.162 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2261 2021-03-15T22:24:03.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.163 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527444+0000", 2021-03-15T22:24:03.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527444+0000", 2021-03-15T22:24:03.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.581e-06 2021-03-15T22:24:03.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527445+0000", 2021-03-15T22:24:03.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0140000000000001e-06 2021-03-15T22:24:03.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527447+0000", 2021-03-15T22:24:03.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.644e-06 2021-03-15T22:24:03.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527455+0000", 2021-03-15T22:24:03.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.42e-06 2021-03-15T22:24:03.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527460+0000", 2021-03-15T22:24:03.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6919000000000003e-05 2021-03-15T22:24:03.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527506+0000", 2021-03-15T22:24:03.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6880000000000005e-06 2021-03-15T22:24:03.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:24:03.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527515+0000", 2021-03-15T22:24:03.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53819720400000004 2021-03-15T22:24:03.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.065712+0000", 2021-03-15T22:24:03.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118725 2021-03-15T22:24:03.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.065831+0000", 2021-03-15T22:24:03.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057793999999999999 2021-03-15T22:24:03.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066409+0000", 2021-03-15T22:24:03.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53896513499999998 2021-03-15T22:24:03.177 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.177 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.178 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.179 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2262 7.2 7:42f18a95:::smithi07694878-45:8d [sparse-read 0~2486556] snapc 0=[] ondisk+read+known_if_redirected e264)", 2021-03-15T22:24:03.179 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.527470+0000", 2021-03-15T22:24:03.179 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 63.062008306000003, 2021-03-15T22:24:03.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53929365200000001, 2021-03-15T22:24:03.180 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.180 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:03.180 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.180 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.180 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.181 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2262 2021-03-15T22:24:03.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.181 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527470+0000", 2021-03-15T22:24:03.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527470+0000", 2021-03-15T22:24:03.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1170000000000001e-06 2021-03-15T22:24:03.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527471+0000", 2021-03-15T22:24:03.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.468e-06 2021-03-15T22:24:03.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527473+0000", 2021-03-15T22:24:03.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1550000000000004e-06 2021-03-15T22:24:03.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527479+0000", 2021-03-15T22:24:03.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1910000000000002e-06 2021-03-15T22:24:03.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527483+0000", 2021-03-15T22:24:03.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104373 2021-03-15T22:24:03.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.189 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527587+0000", 2021-03-15T22:24:03.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3989999999999993e-06 2021-03-15T22:24:03.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:24:03.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527597+0000", 2021-03-15T22:24:03.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53885792499999996 2021-03-15T22:24:03.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066455+0000", 2021-03-15T22:24:03.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000101804 2021-03-15T22:24:03.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066556+0000", 2021-03-15T22:24:03.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00020722000000000001 2021-03-15T22:24:03.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066764+0000", 2021-03-15T22:24:03.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53929365200000001 2021-03-15T22:24:03.194 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.195 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.195 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.196 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2468 7.a 7:5bc7f316:::smithi07694878-38:head [copy-from ver 307] snapc b6=[b6,b5,ad,9c,8d] ondisk+write+known_if_redirected e282)", 2021-03-15T22:24:03.196 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:17.103884+0000", 2021-03-15T22:24:03.196 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.485594761999998, 2021-03-15T22:24:03.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017222021000000001, 2021-03-15T22:24:03.196 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.197 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:03.197 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.197 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.197 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.198 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2468 2021-03-15T22:24:03.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.198 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103884+0000", 2021-03-15T22:24:03.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103884+0000", 2021-03-15T22:24:03.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7590000000000003e-06 2021-03-15T22:24:03.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.201 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103889+0000", 2021-03-15T22:24:03.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4449999999999998e-06 2021-03-15T22:24:03.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103894+0000", 2021-03-15T22:24:03.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5321999999999999e-05 2021-03-15T22:24:03.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103919+0000", 2021-03-15T22:24:03.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5699999999999999e-06 2021-03-15T22:24:03.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103929+0000", 2021-03-15T22:24:03.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0724999999999997e-05 2021-03-15T22:24:03.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.104009+0000", 2021-03-15T22:24:03.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0499999999999997e-06 2021-03-15T22:24:03.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:24:03.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.104018+0000", 2021-03-15T22:24:03.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043887350000000004 2021-03-15T22:24:03.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.108407+0000", 2021-03-15T22:24:03.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9679000000000007e-05 2021-03-15T22:24:03.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.108497+0000", 2021-03-15T22:24:03.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0069025570000000001 2021-03-15T22:24:03.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:24:03.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.115399+0000", 2021-03-15T22:24:03.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050972789999999997 2021-03-15T22:24:03.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:03.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.120497+0000", 2021-03-15T22:24:03.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033331700000000002 2021-03-15T22:24:03.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:03.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.120830+0000", 2021-03-15T22:24:03.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000105122 2021-03-15T22:24:03.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:03.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.120935+0000", 2021-03-15T22:24:03.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7670999999999994e-05 2021-03-15T22:24:03.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:03.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.121033+0000", 2021-03-15T22:24:03.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2789999999999999e-05 2021-03-15T22:24:03.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.121106+0000", 2021-03-15T22:24:03.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017222021000000001 2021-03-15T22:24:03.217 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.218 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.218 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.219 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2469 7.a 7:5bc7f316:::smithi07694878-38:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e282)", 2021-03-15T22:24:03.219 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:17.103944+0000", 2021-03-15T22:24:03.219 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 45.485534829999999, 2021-03-15T22:24:03.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01743428, 2021-03-15T22:24:03.220 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.220 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:03.220 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.220 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.221 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.221 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2469 2021-03-15T22:24:03.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.221 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103944+0000", 2021-03-15T22:24:03.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103944+0000", 2021-03-15T22:24:03.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6500000000000001e-06 2021-03-15T22:24:03.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103945+0000", 2021-03-15T22:24:03.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7439999999999999e-06 2021-03-15T22:24:03.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103947+0000", 2021-03-15T22:24:03.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4410000000000004e-06 2021-03-15T22:24:03.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103954+0000", 2021-03-15T22:24:03.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.921e-06 2021-03-15T22:24:03.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.103959+0000", 2021-03-15T22:24:03.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011199100000000001 2021-03-15T22:24:03.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.230 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.104071+0000", 2021-03-15T22:24:03.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7750000000000004e-06 2021-03-15T22:24:03.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.231 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:24:03.231 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.104078+0000", 2021-03-15T22:24:03.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045710500000000001 2021-03-15T22:24:03.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.108649+0000", 2021-03-15T22:24:03.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7557000000000004e-05 2021-03-15T22:24:03.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:03.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.108717+0000", 2021-03-15T22:24:03.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010250125000000001 2021-03-15T22:24:03.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.118967+0000", 2021-03-15T22:24:03.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6789e-05 2021-03-15T22:24:03.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:03.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.119044+0000", 2021-03-15T22:24:03.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021147710000000001 2021-03-15T22:24:03.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.121158+0000", 2021-03-15T22:24:03.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7219000000000005e-05 2021-03-15T22:24:03.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.121256+0000", 2021-03-15T22:24:03.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000122247 2021-03-15T22:24:03.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:17.121378+0000", 2021-03-15T22:24:03.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01743428 2021-03-15T22:24:03.240 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.240 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.240 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.241 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2771 7.a 7:5bc7f316:::smithi07694878-38:head [sparse-read 0~2588311,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e310)", 2021-03-15T22:24:03.241 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105078+0000", 2021-03-15T22:24:03.241 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.484400121, 2021-03-15T22:24:03.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013224962, 2021-03-15T22:24:03.242 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.242 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:03.242 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.242 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.243 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.243 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2771 2021-03-15T22:24:03.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.243 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105078+0000", 2021-03-15T22:24:03.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105078+0000", 2021-03-15T22:24:03.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.161e-06 2021-03-15T22:24:03.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105084+0000", 2021-03-15T22:24:03.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8579999999999999e-06 2021-03-15T22:24:03.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105089+0000", 2021-03-15T22:24:03.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2830999999999999e-05 2021-03-15T22:24:03.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105112+0000", 2021-03-15T22:24:03.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1390000000000001e-06 2021-03-15T22:24:03.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105121+0000", 2021-03-15T22:24:03.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6487000000000001e-05 2021-03-15T22:24:03.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105208+0000", 2021-03-15T22:24:03.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5445000000000005e-05 2021-03-15T22:24:03.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105303+0000", 2021-03-15T22:24:03.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013000041 2021-03-15T22:24:03.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.118303+0000", 2021-03-15T22:24:03.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013224962 2021-03-15T22:24:03.254 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.254 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.254 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.255 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2772 7.a 7:5bc7f316:::smithi07694878-38:head [sparse-read 0~2588311] snapc 0=[] ondisk+read+known_if_redirected e310)", 2021-03-15T22:24:03.255 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105135+0000", 2021-03-15T22:24:03.255 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.484342998999999, 2021-03-15T22:24:03.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013364589, 2021-03-15T22:24:03.256 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.256 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:03.256 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.257 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.257 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.257 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2772 2021-03-15T22:24:03.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.257 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105135+0000", 2021-03-15T22:24:03.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105135+0000", 2021-03-15T22:24:03.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3480000000000001e-06 2021-03-15T22:24:03.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105137+0000", 2021-03-15T22:24:03.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4959999999999999e-06 2021-03-15T22:24:03.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105138+0000", 2021-03-15T22:24:03.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.942e-06 2021-03-15T22:24:03.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105144+0000", 2021-03-15T22:24:03.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2019999999999997e-06 2021-03-15T22:24:03.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105148+0000", 2021-03-15T22:24:03.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013182332999999999 2021-03-15T22:24:03.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.118331+0000", 2021-03-15T22:24:03.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.05e-05 2021-03-15T22:24:03.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.118391+0000", 2021-03-15T22:24:03.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108768 2021-03-15T22:24:03.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.118500+0000", 2021-03-15T22:24:03.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013364589 2021-03-15T22:24:03.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.268 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.268 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.269 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2773 7.a 7:5bc7f316:::smithi07694878-38:head [sparse-read 0~2588311] snapc 0=[] ondisk+read+known_if_redirected e310)", 2021-03-15T22:24:03.269 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105152+0000", 2021-03-15T22:24:03.270 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 17.484325900999998, 2021-03-15T22:24:03.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01360392, 2021-03-15T22:24:03.270 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.270 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:03.271 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.275 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.275 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.275 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2773 2021-03-15T22:24:03.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.276 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105152+0000", 2021-03-15T22:24:03.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105152+0000", 2021-03-15T22:24:03.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3799999999999996e-07 2021-03-15T22:24:03.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105153+0000", 2021-03-15T22:24:03.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.178e-06 2021-03-15T22:24:03.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105155+0000", 2021-03-15T22:24:03.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0610000000000003e-06 2021-03-15T22:24:03.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105160+0000", 2021-03-15T22:24:03.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6780000000000002e-06 2021-03-15T22:24:03.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105163+0000", 2021-03-15T22:24:03.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013373383000000001 2021-03-15T22:24:03.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.118537+0000", 2021-03-15T22:24:03.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4452999999999999e-05 2021-03-15T22:24:03.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.118621+0000", 2021-03-15T22:24:03.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000135229 2021-03-15T22:24:03.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.118756+0000", 2021-03-15T22:24:03.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01360392 2021-03-15T22:24:03.286 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.286 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.286 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.287 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2827 7.a 7:58400cda:::smithi07694878-39:head [copy-from ver 215] snapc d7=[] ondisk+write+known_if_redirected e317)", 2021-03-15T22:24:03.287 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:52.252765+0000", 2021-03-15T22:24:03.287 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.336713669, 2021-03-15T22:24:03.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023253293000000001, 2021-03-15T22:24:03.288 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.288 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:03.288 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.288 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.288 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.289 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2827 2021-03-15T22:24:03.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.289 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252765+0000", 2021-03-15T22:24:03.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252765+0000", 2021-03-15T22:24:03.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5710000000000003e-06 2021-03-15T22:24:03.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252771+0000", 2021-03-15T22:24:03.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4500000000000003e-06 2021-03-15T22:24:03.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252777+0000", 2021-03-15T22:24:03.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1381000000000001e-05 2021-03-15T22:24:03.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252798+0000", 2021-03-15T22:24:03.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.747e-06 2021-03-15T22:24:03.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252808+0000", 2021-03-15T22:24:03.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1598000000000002e-05 2021-03-15T22:24:03.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252869+0000", 2021-03-15T22:24:03.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.02e-06 2021-03-15T22:24:03.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:24:03.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252875+0000", 2021-03-15T22:24:03.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0074058630000000004 2021-03-15T22:24:03.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.260281+0000", 2021-03-15T22:24:03.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0491e-05 2021-03-15T22:24:03.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.260352+0000", 2021-03-15T22:24:03.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051586779999999999 2021-03-15T22:24:03.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.301 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:24:03.301 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.265510+0000", 2021-03-15T22:24:03.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088347449999999998 2021-03-15T22:24:03.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:03.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.274345+0000", 2021-03-15T22:24:03.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015050790000000001 2021-03-15T22:24:03.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:03.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.275850+0000", 2021-03-15T22:24:03.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8770000000000002e-05 2021-03-15T22:24:03.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:03.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.275869+0000", 2021-03-15T22:24:03.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0575999999999996e-05 2021-03-15T22:24:03.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:03.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.275950+0000", 2021-03-15T22:24:03.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8324000000000005e-05 2021-03-15T22:24:03.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.306 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.306 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.276018+0000", 2021-03-15T22:24:03.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023253293000000001 2021-03-15T22:24:03.307 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.307 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.307 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.308 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2828 7.a 7:58400cda:::smithi07694878-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e317)", 2021-03-15T22:24:03.308 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:52.252824+0000", 2021-03-15T22:24:03.308 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 10.336654617000001, 2021-03-15T22:24:03.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023410568, 2021-03-15T22:24:03.309 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:03.309 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:03.309 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:03.309 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:03.310 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:03.310 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2828 2021-03-15T22:24:03.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.310 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:03.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:03.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252824+0000", 2021-03-15T22:24:03.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:03.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:03.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252824+0000", 2021-03-15T22:24:03.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2021-03-15T22:24:03.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.313 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:03.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252825+0000", 2021-03-15T22:24:03.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.812e-06 2021-03-15T22:24:03.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:03.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252827+0000", 2021-03-15T22:24:03.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2676e-05 2021-03-15T22:24:03.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:03.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252840+0000", 2021-03-15T22:24:03.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9070000000000002e-06 2021-03-15T22:24:03.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:03.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252850+0000", 2021-03-15T22:24:03.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.97e-05 2021-03-15T22:24:03.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252919+0000", 2021-03-15T22:24:03.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8239999999999996e-06 2021-03-15T22:24:03.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:24:03.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252925+0000", 2021-03-15T22:24:03.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075364489999999998 2021-03-15T22:24:03.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.260462+0000", 2021-03-15T22:24:03.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6307999999999998e-05 2021-03-15T22:24:03.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:03.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.260508+0000", 2021-03-15T22:24:03.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089375010000000005 2021-03-15T22:24:03.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.269445+0000", 2021-03-15T22:24:03.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6283999999999998e-05 2021-03-15T22:24:03.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:03.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.269502+0000", 2021-03-15T22:24:03.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006549722 2021-03-15T22:24:03.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:03.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.276051+0000", 2021-03-15T22:24:03.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0444000000000004e-05 2021-03-15T22:24:03.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:03.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.276132+0000", 2021-03-15T22:24:03.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102391 2021-03-15T22:24:03.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:03.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:03.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:03.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.276234+0000", 2021-03-15T22:24:03.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023410568 2021-03-15T22:24:03.327 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.327 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.327 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.327 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:03.328 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:03.328 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:24:03.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:03.329 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:24:03.331 INFO:tasks.rados.rados.0.smithi076.stdout:1153: done (1 left) 2021-03-15T22:24:03.331 INFO:tasks.rados.rados.0.smithi076.stdout:1154: write (excl) oid 23 current snap is 117 2021-03-15T22:24:03.331 INFO:tasks.rados.rados.0.smithi076.stdout:1152: finishing rollback tid 0 to smithi07694878-37 2021-03-15T22:24:03.331 INFO:tasks.rados.rados.0.smithi076.stdout:1152: finishing rollback tid 1 to smithi07694878-37 2021-03-15T22:24:03.332 INFO:tasks.rados.rados.0.smithi076.stdout:1152: finishing rollback tid 2 to smithi07694878-37 2021-03-15T22:24:03.332 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 673 (ObjNum 280 snap 74 seq_num 280) dirty exists 2021-03-15T22:24:03.332 INFO:tasks.rados.rados.0.smithi076.stdout:1154: seq_num 424 ranges {726168=704991,1895071=683474} 2021-03-15T22:24:03.332 INFO:teuthology.orchestra.run.smithi076.stdout:moved 11 / 414 (2.657%) 2021-03-15T22:24:03.333 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:24:03.333 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 10.6154 -> 12.3466 (expected baseline 6.72913) 2021-03-15T22:24:03.333 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 25 -> 22 pgs (0.483092 -> 0.425121 * mean) 2021-03-15T22:24:03.333 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 62 -> 63 pgs (1.19807 -> 1.21739 * mean) 2021-03-15T22:24:03.334 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:24:03.334 INFO:teuthology.orchestra.run.smithi076.stdout:oload 110 2021-03-15T22:24:03.334 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 3 2021-03-15T22:24:03.334 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:24:03.335 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0007 2021-03-15T22:24:03.335 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0008 2021-03-15T22:24:03.335 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 weight 1.0000 -> 0.8093 2021-03-15T22:24:03.335 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 weight 0.5226 -> 0.4398 2021-03-15T22:24:03.336 INFO:tasks.rados.rados.0.smithi076.stdout:1154: writing smithi07694878-23 from 726168 to 1431159 tid 1 2021-03-15T22:24:03.336 INFO:tasks.rados.rados.0.smithi076.stdout:1154: writing smithi07694878-23 from 1895071 to 2578545 tid 2 2021-03-15T22:24:03.336 INFO:tasks.rados.rados.0.smithi076.stdout:1152: done (1 left) 2021-03-15T22:24:03.337 INFO:tasks.rados.rados.0.smithi076.stdout:1155: snap_remove snap 110 2021-03-15T22:24:03.351 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:03.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:03.352 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:03.453 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:03.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:03.477 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:03.477 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:03.579 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:03.600 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:03.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:03.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:03.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:03.725 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:03.727 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:03.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[56514]: cluster 2021-03-15T22:24:01.896984+0000 osd.7 (osd.7) 68 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:24:03.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[56514]: audit 2021-03-15T22:24:02.306577+0000 mgr.y (mgr.14144) 552 : audit [DBG] from='client.15258 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:03.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[56514]: audit 2021-03-15T22:24:02.307296+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]: dispatch 2021-03-15T22:24:03.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[56514]: cluster 2021-03-15T22:24:02.448218+0000 osd.6 (osd.6) 50 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:24:03.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[56514]: cluster 2021-03-15T22:24:02.507320+0000 osd.0 (osd.0) 71 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:24:03.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[56514]: audit 2021-03-15T22:24:03.075343+0000 mon.a (mon.0) 1063 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]': finished 2021-03-15T22:24:03.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[56514]: cluster 2021-03-15T22:24:03.075397+0000 mon.a (mon.0) 1064 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2021-03-15T22:24:03.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: cluster 2021-03-15T22:24:01.896984+0000 2021-03-15T22:24:03.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: osd.7 (osd.7) 68 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:24:03.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: audit 2021-03-15T22:24:03.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: 2021-03-15T22:24:02.306577+0000 mgr.y (mgr.14144 2021-03-15T22:24:03.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: ) 552 : audit [DBG] from='client.15258 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:03.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:24:03.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: -15T22:24:02.307296+0000 mon.a 2021-03-15T22:24:03.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: (mon.0) 1062 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]: dispatch 2021-03-15T22:24:03.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: 2021-03-15T22:24:03.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:24:03.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: 03-15T22:24:02.448218+0000 osd.6 (osd. 2021-03-15T22:24:03.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: 6) 50 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:24:03.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:24:03.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: 03-15T22:24:02.507320+0000 2021-03-15T22:24:03.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: osd.0 (osd.0) 71 : cluster [DBG] 2021-03-15T22:24:03.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: 5.3 scrub ok 2021-03-15T22:24:03.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: audit 2021-03-15T22:24:03 2021-03-15T22:24:03.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: .075343+0000 mon.a (mon.0) 1063 : 2021-03-15T22:24:03.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]': finished 2021-03-15T22:24:03.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: cluster 2021-03-15T22:24:03.075397+0000 2021-03-15T22:24:03.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: mon.a (mon.0) 1064 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2021-03-15T22:24:03.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:03 smithi076 conmon[62156]: 2021-03-15T22:24:03.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: cluster 2021-03-15T22:24:01.896984+0000 osd.7 (osd.7) 2021-03-15T22:24:03.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: 68 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:24:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: audit 2021-03-15T22:24:02.306577+0000 mgr.y (mgr.14144) 552 : audit [DBG] from='client.15258 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: audit 2021 2021-03-15T22:24:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: -03-15T22:24:02.307296+0000 mon.a (mon.0) 1062 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]: dispatch 2021-03-15T22:24:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:24:03.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: -03-15T22:24:02.448218 2021-03-15T22:24:03.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: +0000 osd.6 (osd.6) 50 : cluster 2021-03-15T22:24:03.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: [DBG] 3.1e scrub ok 2021-03-15T22:24:03.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:24:03.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: 24:02.507320+0000 osd.0 (osd.0 2021-03-15T22:24:03.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: ) 71 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:24:03.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:24:03.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: -15T22:24:03.075343+0000 2021-03-15T22:24:03.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: mon.a (mon.0) 1063 : audit [INF] 2021-03-15T22:24:03.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]': finished 2021-03-15T22:24:03.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: cluster 2021-03-15T22:24:03.075397+0000 mon.a (mon.0 2021-03-15T22:24:03.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:03 smithi125 conmon[56739]: ) 1064 : cluster [DBG] osdmap e328: 8 total, 8 up, 8 in 2021-03-15T22:24:03.828 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:03.850 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:03.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:03.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:03.953 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:03.974 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:03.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:03.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:04.026 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '110', 'type': 'reweight-by-utilization'} 2021-03-15T22:24:04.027 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight-by-utilization 110 0.05 2021-03-15T22:24:04.077 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:04.081 INFO:tasks.rados.rados.0.smithi076.stdout:1155: done (1 left) 2021-03-15T22:24:04.081 INFO:tasks.rados.rados.0.smithi076.stdout:1156: write oid 41 current snap is 117 2021-03-15T22:24:04.081 INFO:tasks.rados.rados.0.smithi076.stdout:1156: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2021-03-15T22:24:04.090 INFO:tasks.rados.rados.0.smithi076.stdout:1156: writing smithi07694878-41 from 742975 to 1523215 tid 1 2021-03-15T22:24:04.095 INFO:tasks.rados.rados.0.smithi076.stdout:1156: writing smithi07694878-41 from 1953553 to 2380685 tid 2 2021-03-15T22:24:04.100 INFO:tasks.rados.rados.0.smithi076.stdout:1156: writing smithi07694878-41 from 2971713 to 3142975 tid 3 2021-03-15T22:24:04.100 INFO:tasks.rados.rados.0.smithi076.stdout:1157: snap_remove snap 114 2021-03-15T22:24:04.105 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:04.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:04.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:04.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:04.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:04.233 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:04.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:04.335 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:04.337 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-15T22:24:04.338 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:04.366 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:04.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:04.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:04.434 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:04.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:04.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:04.492 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:04.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:04 smithi076 conmon[56514]: cluster 2021-03-15T22:24:02.949598+0000 2021-03-15T22:24:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:04 smithi076 conmon[56514]: osd.4 (osd.4) 76 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:24:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:04 smithi076 conmon[56514]: cluster 2021-03-15T22:24:03.032398+0000 mgr.y (mgr.14144) 553 : cluster [DBG] pgmap v618: 138 pgs: 138 active+clean; 146 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 17 MiB/s rd, 4.5 MiB/s wr, 32 op/s 2021-03-15T22:24:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:04 smithi076 conmon[56514]: cluster 2021-03-15T22:24:04.077061+0000 mon.a (mon.0) 1065 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2021-03-15T22:24:04.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:04 smithi076 conmon[62156]: cluster 2021-03-15T22:24:02. 2021-03-15T22:24:04.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:04 smithi076 conmon[62156]: 949598+0000 osd.4 (osd.4) 76 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:24:04.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:04 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:24:04.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:04 smithi076 conmon[62156]: 22:24:03.032398+0000 mgr.y (mgr.14144) 553 : cluster [DBG] pgmap v618: 138 pgs: 138 active+clean; 146 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 17 MiB/s rd, 4.5 MiB/s wr, 32 op/s 2021-03-15T22:24:04.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:04 smithi076 conmon[62156]: cluster 2021-03-15T22:24:04.077061+0000 mon.a (mon 2021-03-15T22:24:04.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:04 smithi076 conmon[62156]: .0) 1065 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2021-03-15T22:24:04.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:04.618 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:04.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:04.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:04.621 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:04.721 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:04.743 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:04.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:04.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:04.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:04 smithi125 conmon[56739]: cluster 2021-03-15T22:24:02.949598+0000 osd.4 (osd.4 2021-03-15T22:24:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:04 smithi125 conmon[56739]: ) 76 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:24:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:04 smithi125 conmon[56739]: cluster 2021-03-15T22:24:03.032398+0000 mgr.y (mgr.14144) 553 : cluster [DBG] pgmap v618: 138 pgs: 138 active+clean; 146 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 17 MiB/s rd, 4.5 MiB/s wr, 32 op/s 2021-03-15T22:24:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:04 smithi125 conmon[56739]: cluster 2021-03-15T22:24:04.077061+0000 mon.a ( 2021-03-15T22:24:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:04 smithi125 conmon[56739]: mon.0) 1065 : cluster [DBG] osdmap e329: 8 total, 8 up, 8 in 2021-03-15T22:24:04.846 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:04.869 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:04.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:04.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:04.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:04.994 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:04.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:04.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:05.083 INFO:tasks.rados.rados.0.smithi076.stdout:1157: done (2 left) 2021-03-15T22:24:05.083 INFO:tasks.rados.rados.0.smithi076.stdout:1158: copy_from oid 15 from oid 32 current snap is 117 2021-03-15T22:24:05.084 INFO:tasks.rados.rados.0.smithi076.stdout:1159: delete oid 45 current snap is 117 2021-03-15T22:24:05.084 INFO:tasks.rados.rados.0.smithi076.stdout:1156: finishing write tid 1 to smithi07694878-41 2021-03-15T22:24:05.084 INFO:tasks.rados.rados.0.smithi076.stdout:1156: finishing write tid 2 to smithi07694878-41 2021-03-15T22:24:05.084 INFO:tasks.rados.rados.0.smithi076.stdout:1156: finishing write tid 3 to smithi07694878-41 2021-03-15T22:24:05.085 INFO:tasks.rados.rados.0.smithi076.stdout:1156: finishing write tid 4 to smithi07694878-41 2021-03-15T22:24:05.085 INFO:tasks.rados.rados.0.smithi076.stdout:1156: finishing write tid 5 to smithi07694878-41 2021-03-15T22:24:05.085 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 274 (ObjNum 425 snap 117 seq_num 425) dirty exists 2021-03-15T22:24:05.085 INFO:tasks.rados.rados.0.smithi076.stdout:1156: left oid 41 (ObjNum 425 snap 117 seq_num 425) 2021-03-15T22:24:05.090 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:24:05.097 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:05.106 INFO:tasks.rados.rados.0.smithi076.stdout:1156: done (3 left) 2021-03-15T22:24:05.106 INFO:tasks.rados.rados.0.smithi076.stdout:1159: done (2 left) 2021-03-15T22:24:05.106 INFO:tasks.rados.rados.0.smithi076.stdout:1160: snap_create 2021-03-15T22:24:05.132 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:05.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:05.136 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:05.144 INFO:tasks.rados.rados.0.smithi076.stdout:1154: finishing write tid 1 to smithi07694878-23 2021-03-15T22:24:05.191 INFO:tasks.rados.rados.0.smithi076.stdout:1154: finishing write tid 2 to smithi07694878-23 2021-03-15T22:24:05.191 INFO:tasks.rados.rados.0.smithi076.stdout:1154: finishing write tid 3 to smithi07694878-23 2021-03-15T22:24:05.192 INFO:tasks.rados.rados.0.smithi076.stdout:1154: finishing write tid 4 to smithi07694878-23 2021-03-15T22:24:05.192 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 278 (ObjNum 424 snap 117 seq_num 424) dirty exists 2021-03-15T22:24:05.192 INFO:tasks.rados.rados.0.smithi076.stdout:1154: left oid 23 (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:24:05.208 INFO:tasks.rados.rados.0.smithi076.stdout:1158: finishing copy_from to smithi07694878-15 2021-03-15T22:24:05.208 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 640 (ObjNum 371 snap 102 seq_num 371) dirty exists 2021-03-15T22:24:05.208 INFO:tasks.rados.rados.0.smithi076.stdout:1158: finishing copy_from racing read to smithi07694878-15 2021-03-15T22:24:05.237 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:05.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:05.262 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:05.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:05.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[56514]: audit 2021-03-15T22:24:04.901667+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.76:0/150184386' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:05.316 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[56514]: audit 2021-03-15T22:24:04.902066+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:05.317 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[56514]: cluster 2021-03-15T22:24:05.073969+0000 mon.a (mon.0) 1067 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:24:05.317 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[56514]: cluster 2021-03-15T22:24:05.074013+0000 mon.a (mon.0) 1068 : cluster [INF] Cluster is now healthy 2021-03-15T22:24:05.317 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[56514]: audit 2021-03-15T22:24:05.078808+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:05.365 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:05.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:05.388 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:05.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:05.490 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:05.511 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:05.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:05.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:05.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:05 smithi125 conmon[56739]: audit 2021-03-15T22:24:04.901667+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.76:0/150184386' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:05 smithi125 conmon[56739]: audit 2021-03-15T22:24:04.902066+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:05 smithi125 conmon[56739]: cluster 2021-03-15T22:24:05.073969+0000 mon.a (mon.0) 1067 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:24:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:05 smithi125 conmon[56739]: cluster 2021-03-15T22:24:05.074013+0000 mon.a (mon.0) 1068 : cluster [INF] Cluster is now healthy 2021-03-15T22:24:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:05 smithi125 conmon[56739]: audit 2021-03-15T22:24:05.078808+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:05.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:05 smithi125 conmon[56739]: cluster 2021-03-15T22:24:05.078874+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2021-03-15T22:24:05.614 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:05.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:05.637 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:05.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:05.738 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:05.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:05.761 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:05.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:05.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[56514]: cluster 2021-03-15T22:24:05.078874+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2021-03-15T22:24:05.767 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[65511]: debug 2021-03-15T22:24:05.478+0000 7f1a1f877700 -1 osd.0 pg_epoch: 330 pg[4.16( v 65'16 (0'0,65'16] local-lis/les=328/329 n=11 ec=58/58 lis/c=328/328 les/c/f=329/329/0 sis=328) [0,7,2] r=0 lpr=328 crt=65'16 lcod 65'15 mlcod 0'0 active+clean TIME_FOR_DEEP] scrubber pg(4.16) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 7 (0x557aca0edce0) 2021-03-15T22:24:05.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[62156]: audit 2021-03-15T22:24:04.901667+0000 mon.c (mon.1) 28 : audit [INF] from='client.? 172.21.15.76:0/150184386' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:05.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[62156]: audit 2021-03-15T22:24:04.902066+0000 mon.a (mon.0) 1066 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:05.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[62156]: cluster 2021-03-15T22:24:05.073969+0000 mon.a (mon.0) 1067 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:24:05.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[62156]: cluster 2021-03-15T22:24:05.074013+0000 mon.a (mon.0) 1068 : cluster [INF] Cluster is now healthy 2021-03-15T22:24:05.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[62156]: audit 2021-03-15T22:24:05.078808+0000 mon.a (mon.0) 1069 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:05.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:05 smithi076 conmon[62156]: cluster 2021-03-15T22:24:05.078874+0000 mon.a (mon.0) 1070 : cluster [DBG] osdmap e330: 8 total, 8 up, 8 in 2021-03-15T22:24:05.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:05.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:05.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:05.989 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:06.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:06.013 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:06.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:06.085 INFO:tasks.rados.rados.0.smithi076.stdout:1154: done (2 left) 2021-03-15T22:24:06.085 INFO:tasks.rados.rados.0.smithi076.stdout:1158: done (1 left) 2021-03-15T22:24:06.085 INFO:tasks.rados.rados.0.smithi076.stdout:1160: done (0 left) 2021-03-15T22:24:06.086 INFO:tasks.rados.rados.0.smithi076.stdout:1161: delete oid 37 current snap is 118 2021-03-15T22:24:06.087 INFO:tasks.rados.rados.0.smithi076.stdout:1161: done (0 left) 2021-03-15T22:24:06.087 INFO:tasks.rados.rados.0.smithi076.stdout:1162: read oid 28 snap 111 2021-03-15T22:24:06.088 INFO:tasks.rados.rados.0.smithi076.stdout:1162: expect deleted 2021-03-15T22:24:06.088 INFO:tasks.rados.rados.0.smithi076.stdout:1163: snap_create 2021-03-15T22:24:06.115 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:06.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:06.138 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:06.139 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:06.240 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:06.264 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:06.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:06.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:06.367 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:06.376 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:24:06.376 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:24:06.377 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:24:06.377 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:24:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:06.390 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:06.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:06.491 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:06.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:06.514 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:06.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:06.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:06 smithi076 conmon[56514]: cluster 2021-03-15T22:24:05.032959+0000 mgr.y (mgr.14144) 554 : 2021-03-15T22:24:06.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:06 smithi076 conmon[56514]: cluster [DBG] pgmap v621: 138 pgs: 1 active+recovering, 137 active+clean; 146 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 6/1155 objects misplaced (0.519%) 2021-03-15T22:24:06.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:06 smithi076 conmon[56514]: cluster 2021-03-15T22:24:06.082806+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2021-03-15T22:24:06.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:06 smithi076 conmon[56514]: 2021-03-15T22:24:06.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:06 smithi076 conmon[62156]: cluster 2021-03-15T22:24:05.032959+0000 mgr.y (mgr.14144 2021-03-15T22:24:06.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:06 smithi076 conmon[62156]: ) 554 : cluster [DBG] pgmap v621: 138 pgs: 1 active+recovering, 137 active+clean; 146 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 6/1155 objects misplaced (0.519%) 2021-03-15T22:24:06.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:06 smithi076 conmon[62156]: cluster 2021-03-15T22:24:06.082806+0000 mon.a (mon.0) 1071 : cluster 2021-03-15T22:24:06.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:06 smithi076 conmon[62156]: [DBG] osdmap e331: 8 total, 8 up, 8 in 2021-03-15T22:24:06.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:06 smithi125 conmon[56739]: cluster 2021-03-15T22:24:05.032959 2021-03-15T22:24:06.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:06 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 554 : cluster [DBG] pgmap v621: 138 pgs: 1 active+recovering, 137 active+clean; 146 MiB data, 536 MiB used, 715 GiB / 715 GiB avail; 8.6 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 6/1155 objects misplaced (0.519%) 2021-03-15T22:24:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:06 smithi125 conmon[56739]: cluster 2021-03-15T22:24:06.082806+0000 mon.a (mon.0) 1071 : cluster [DBG] osdmap e331: 8 total, 8 up, 8 in 2021-03-15T22:24:06.615 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:06.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:06.639 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:06.639 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:06.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:06.762 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:06.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:06.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:06.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:06.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:06.888 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:06.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:06.990 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:07.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-15T22:24:07.012 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:07.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:07.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:07.086 INFO:tasks.rados.rados.0.smithi076.stdout:1162: done (1 left) 2021-03-15T22:24:07.087 INFO:tasks.rados.rados.0.smithi076.stdout:1163: done (0 left) 2021-03-15T22:24:07.087 INFO:tasks.rados.rados.0.smithi076.stdout:1164: delete oid 44 current snap is 119 2021-03-15T22:24:07.090 INFO:tasks.rados.rados.0.smithi076.stdout:1164: done (0 left) 2021-03-15T22:24:07.090 INFO:tasks.rados.rados.0.smithi076.stdout:1165: rollback oid 32 current snap is 119 2021-03-15T22:24:07.090 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 32 to 118 2021-03-15T22:24:07.090 INFO:tasks.rados.rados.0.smithi076.stdout:1166: write (excl) oid 24 current snap is 119 2021-03-15T22:24:07.091 INFO:tasks.rados.rados.0.smithi076.stdout:1166: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2021-03-15T22:24:07.097 INFO:tasks.rados.rados.0.smithi076.stdout:1166: writing smithi07694878-24 from 410203 to 1207792 tid 1 2021-03-15T22:24:07.102 INFO:tasks.rados.rados.0.smithi076.stdout:1166: writing smithi07694878-24 from 1787481 to 2221951 tid 2 2021-03-15T22:24:07.109 INFO:tasks.rados.rados.0.smithi076.stdout:1166: writing smithi07694878-24 from 2681405 to 3210203 tid 3 2021-03-15T22:24:07.109 INFO:tasks.rados.rados.0.smithi076.stdout:1167: write (excl) oid 12 current snap is 119 2021-03-15T22:24:07.109 INFO:tasks.rados.rados.0.smithi076.stdout:1165: finishing rollback tid 0 to smithi07694878-32 2021-03-15T22:24:07.109 INFO:tasks.rados.rados.0.smithi076.stdout:1167: seq_num 430 ranges {427010=472838,1329610=694481,2708828=518182} 2021-03-15T22:24:07.114 INFO:tasks.rados.rados.0.smithi076.stdout:1167: writing smithi07694878-12 from 427010 to 899848 tid 1 2021-03-15T22:24:07.114 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:07.120 INFO:tasks.rados.rados.0.smithi076.stdout:1167: writing smithi07694878-12 from 1329610 to 2024091 tid 2 2021-03-15T22:24:07.128 INFO:tasks.rados.rados.0.smithi076.stdout:1167: writing smithi07694878-12 from 2708828 to 3227010 tid 3 2021-03-15T22:24:07.128 INFO:tasks.rados.rados.0.smithi076.stdout:1168: copy_from oid 37 from oid 3 current snap is 119 2021-03-15T22:24:07.129 INFO:tasks.rados.rados.0.smithi076.stdout:1166: finishing write tid 1 to smithi07694878-24 2021-03-15T22:24:07.129 INFO:tasks.rados.rados.0.smithi076.stdout:1166: finishing write tid 2 to smithi07694878-24 2021-03-15T22:24:07.129 INFO:tasks.rados.rados.0.smithi076.stdout:1166: finishing write tid 3 to smithi07694878-24 2021-03-15T22:24:07.129 INFO:tasks.rados.rados.0.smithi076.stdout:1166: finishing write tid 4 to smithi07694878-24 2021-03-15T22:24:07.130 INFO:tasks.rados.rados.0.smithi076.stdout:1166: finishing write tid 5 to smithi07694878-24 2021-03-15T22:24:07.130 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 80 (ObjNum 429 snap 119 seq_num 429) dirty exists 2021-03-15T22:24:07.130 INFO:tasks.rados.rados.0.smithi076.stdout:1166: left oid 24 (ObjNum 429 snap 119 seq_num 429) 2021-03-15T22:24:07.130 INFO:tasks.rados.rados.0.smithi076.stdout:1167: finishing write tid 1 to smithi07694878-12 2021-03-15T22:24:07.130 INFO:tasks.rados.rados.0.smithi076.stdout:1166: done (3 left) 2021-03-15T22:24:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:1169: rollback oid 13 current snap is 119 2021-03-15T22:24:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 13 to 111 2021-03-15T22:24:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:1167: finishing write tid 2 to smithi07694878-12 2021-03-15T22:24:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:1170: write oid 38 current snap is 119 2021-03-15T22:24:07.132 INFO:tasks.rados.rados.0.smithi076.stdout:1170: seq_num 431 ranges {443817=548087,1788092=438139,3019898=223919} 2021-03-15T22:24:07.132 INFO:tasks.rados.rados.0.smithi076.stdout:1170: writing smithi07694878-38 from 443817 to 991904 tid 1 2021-03-15T22:24:07.139 INFO:tasks.rados.rados.0.smithi076.stdout:1170: writing smithi07694878-38 from 1788092 to 2226231 tid 2 2021-03-15T22:24:07.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:07.141 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:07.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:07.145 INFO:tasks.rados.rados.0.smithi076.stdout:1170: writing smithi07694878-38 from 3019898 to 3243817 tid 3 2021-03-15T22:24:07.145 INFO:tasks.rados.rados.0.smithi076.stdout:1171: write oid 6 current snap is 119 2021-03-15T22:24:07.146 INFO:tasks.rados.rados.0.smithi076.stdout:1168: finishing copy_from to smithi07694878-37 2021-03-15T22:24:07.146 INFO:tasks.rados.rados.0.smithi076.stdout:1168: got expected ENOENT (src dne) 2021-03-15T22:24:07.146 INFO:tasks.rados.rados.0.smithi076.stdout:1171: seq_num 432 ranges {460624=623336,1846574=698150,3163674=96950} 2021-03-15T22:24:07.151 INFO:tasks.rados.rados.0.smithi076.stdout:1171: writing smithi07694878-6 from 460624 to 1083960 tid 1 2021-03-15T22:24:07.160 INFO:tasks.rados.rados.0.smithi076.stdout:1171: writing smithi07694878-6 from 1846574 to 2544724 tid 2 2021-03-15T22:24:07.164 INFO:tasks.rados.rados.0.smithi076.stdout:1171: writing smithi07694878-6 from 3163674 to 3260624 tid 3 2021-03-15T22:24:07.164 INFO:tasks.rados.rados.0.smithi076.stdout:1172: snap_create 2021-03-15T22:24:07.165 INFO:tasks.rados.rados.0.smithi076.stdout:1169: finishing rollback tid 0 to smithi07694878-13 2021-03-15T22:24:07.165 INFO:tasks.rados.rados.0.smithi076.stdout:1169: finishing rollback tid 1 to smithi07694878-13 2021-03-15T22:24:07.165 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 418 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:24:07.165 INFO:tasks.rados.rados.0.smithi076.stdout:1168: finishing copy_from racing read to smithi07694878-37 2021-03-15T22:24:07.166 INFO:tasks.rados.rados.0.smithi076.stdout:1167: finishing write tid 3 to smithi07694878-12 2021-03-15T22:24:07.166 INFO:tasks.rados.rados.0.smithi076.stdout:1167: finishing write tid 4 to smithi07694878-12 2021-03-15T22:24:07.166 INFO:tasks.rados.rados.0.smithi076.stdout:1167: finishing write tid 5 to smithi07694878-12 2021-03-15T22:24:07.166 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 645 (ObjNum 430 snap 119 seq_num 430) dirty exists 2021-03-15T22:24:07.167 INFO:tasks.rados.rados.0.smithi076.stdout:1167: left oid 12 (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:24:07.167 INFO:tasks.rados.rados.0.smithi076.stdout:1170: finishing write tid 1 to smithi07694878-38 2021-03-15T22:24:07.167 INFO:tasks.rados.rados.0.smithi076.stdout:1170: finishing write tid 2 to smithi07694878-38 2021-03-15T22:24:07.167 INFO:tasks.rados.rados.0.smithi076.stdout:1170: finishing write tid 3 to smithi07694878-38 2021-03-15T22:24:07.167 INFO:tasks.rados.rados.0.smithi076.stdout:1170: finishing write tid 4 to smithi07694878-38 2021-03-15T22:24:07.168 INFO:tasks.rados.rados.0.smithi076.stdout:1170: finishing write tid 5 to smithi07694878-38 2021-03-15T22:24:07.168 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 433 (ObjNum 431 snap 119 seq_num 431) dirty exists 2021-03-15T22:24:07.168 INFO:tasks.rados.rados.0.smithi076.stdout:1170: left oid 38 (ObjNum 431 snap 119 seq_num 431) 2021-03-15T22:24:07.168 INFO:tasks.rados.rados.0.smithi076.stdout:1171: finishing write tid 1 to smithi07694878-6 2021-03-15T22:24:07.168 INFO:tasks.rados.rados.0.smithi076.stdout:1171: finishing write tid 2 to smithi07694878-6 2021-03-15T22:24:07.169 INFO:tasks.rados.rados.0.smithi076.stdout:1171: finishing write tid 3 to smithi07694878-6 2021-03-15T22:24:07.169 INFO:tasks.rados.rados.0.smithi076.stdout:1171: finishing write tid 4 to smithi07694878-6 2021-03-15T22:24:07.170 INFO:tasks.rados.rados.0.smithi076.stdout:1171: finishing write tid 5 to smithi07694878-6 2021-03-15T22:24:07.170 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 274 (ObjNum 432 snap 119 seq_num 432) dirty exists 2021-03-15T22:24:07.170 INFO:tasks.rados.rados.0.smithi076.stdout:1171: left oid 6 (ObjNum 432 snap 119 seq_num 432) 2021-03-15T22:24:07.237 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:24:07.242 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:07.255 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:24:07.268 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:07.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:07.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:07.275 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:24:07.295 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:24:07.304 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:07.314 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:24:07.333 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:24:07.350 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:24:07.370 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:24:07.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:24:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:24:07.372 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:24:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:24:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:24:07.373 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:24:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:24:07.374 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:24:07.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:07.399 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:07.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:07.501 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:07.524 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:07.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:07.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:07.626 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:07.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:07.649 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:07.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:07.751 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:07.775 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:07.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:07.876 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:07.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:07.902 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:07.902 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:08.004 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:08.026 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:08.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:08.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:08.036 INFO:journalctl@ceph.osd.4.smithi125.stdout:Mar 15 22:24:07 smithi125 conmon[60581]: debug 2021-03-15T22:24:07.878+0000 7fe6a6398700 -1 osd.4 pg_epoch: 332 pg[6.1e( empty local-lis/les=62/63 n=0 ec=62/62 lis/c=62/62 les/c/f=63/63/0 sis=62) [4,6,5] r=0 lpr=62 crt=0'0 mlcod 0'0 active+clean] scrubber pg(6.1e) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 6 (0x555de53c6dc0) 2021-03-15T22:24:08.092 INFO:teuthology.orchestra.run.smithi076.stdout:moved 11 / 414 (2.657%) 2021-03-15T22:24:08.092 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:24:08.092 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 12.3466 -> 13.809 (expected baseline 6.72913) 2021-03-15T22:24:08.093 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 22 -> 19 pgs (0.425121 -> 0.36715 * mean) 2021-03-15T22:24:08.093 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 63 -> 62 pgs (1.21739 -> 1.19807 * mean) 2021-03-15T22:24:08.093 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:24:08.093 INFO:teuthology.orchestra.run.smithi076.stdout:oload 110 2021-03-15T22:24:08.094 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 0.05 2021-03-15T22:24:08.094 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:24:08.094 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0008 2021-03-15T22:24:08.094 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0008 2021-03-15T22:24:08.094 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 weight 0.8093 -> 0.7593 2021-03-15T22:24:08.095 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 weight 0.4398 -> 0.3898 2021-03-15T22:24:08.095 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 weight 1.0000 -> 0.9500 2021-03-15T22:24:08.095 INFO:tasks.rados.rados.0.smithi076.stdout:1167: done (6 left) 2021-03-15T22:24:08.096 INFO:tasks.rados.rados.0.smithi076.stdout:1168: done (5 left) 2021-03-15T22:24:08.096 INFO:tasks.rados.rados.0.smithi076.stdout:1169: done (4 left) 2021-03-15T22:24:08.096 INFO:tasks.rados.rados.0.smithi076.stdout:1170: done (3 left) 2021-03-15T22:24:08.096 INFO:tasks.rados.rados.0.smithi076.stdout:1171: done (2 left) 2021-03-15T22:24:08.097 INFO:tasks.rados.rados.0.smithi076.stdout:1172: done (1 left) 2021-03-15T22:24:08.097 INFO:tasks.rados.rados.0.smithi076.stdout:1173: copy_from oid 41 from oid 43 current snap is 120 2021-03-15T22:24:08.097 INFO:tasks.rados.rados.0.smithi076.stdout:1174: snap_remove snap 111 2021-03-15T22:24:08.129 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:08.152 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:08.152 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:08.253 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:08.276 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:08.277 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:08.378 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:08.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:08.403 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:08.403 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:08.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:08.509 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:08 smithi125 conmon[56739]: cluster 2021-03-15T22:24:07.033464+0000 mgr.y (mgr.14144 2021-03-15T22:24:08.509 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:08 smithi125 conmon[56739]: ) 555 : cluster [DBG] pgmap v624: 138 pgs: 1 active+undersized+remapped, 1 active+recovering, 136 active+clean; 150 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 883 KiB/s wr, 3 op/s; 6/1167 objects misplaced (0.514%); 1.5 MiB/s, 12 objects/s recovering 2021-03-15T22:24:08.509 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:08 smithi125 conmon[56739]: cluster 2021-03-15T22:24:07.083730+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2021-03-15T22:24:08.510 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:08 smithi125 conmon[56739]: audit 2021-03-15T22:24:07.217668+0000 mon.a (mon.0) 1073 : audit [DBG] from='client.? 172.21.15.76:0/4129044425' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:08.510 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:08 smithi125 conmon[56739]: audit 2021-03-15T22:24:07.638066+0000 mgr.y (mgr.14144) 556 : audit [DBG] from='client.15276 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:08.510 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:08 smithi125 conmon[56739]: audit 2021-03-15T22:24:07.638844+0000 mon.a (mon. 2021-03-15T22:24:08.510 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:08 smithi125 conmon[56739]: 0) 1074 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'25548','5':'49764'}"}]: dispatch 2021-03-15T22:24:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:08 smithi076 conmon[56514]: cluster 2021-03-15T22:24:07.033464+0000 mgr.y (mgr.14144) 555 : cluster [DBG] pgmap v624: 138 pgs: 1 active+undersized+remapped, 1 active+recovering, 136 active+clean; 150 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 883 KiB/s wr, 3 op/s; 6/1167 objects misplaced (0.514%); 1.5 MiB/s, 12 objects/s recovering 2021-03-15T22:24:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:08 smithi076 conmon[56514]: cluster 2021-03-15T22:24:07.083730+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2021-03-15T22:24:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:08 smithi076 conmon[56514]: audit 2021-03-15T22:24:07.217668+0000 mon.a (mon.0) 1073 : audit [DBG] from='client.? 172.21.15.76:0/4129044425' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:08 smithi076 conmon[56514]: audit 2021-03-15T22:24:07.638066+0000 mgr.y (mgr.14144) 556 : audit [DBG] from='client.15276 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:08 smithi076 conmon[56514]: audit 2021-03-15T22:24:07.638844+0000 mon.a (mon.0) 1074 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'25548','5':'49764'}"}]: dispatch 2021-03-15T22:24:08.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:08 smithi076 conmon[62156]: cluster 2021-03-15T22:24:07.033464+0000 mgr.y (mgr.14144) 555 : cluster [DBG] pgmap v624: 138 pgs: 1 active+undersized+remapped, 1 active+recovering, 136 active+clean; 150 MiB data, 552 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 883 KiB/s wr, 3 op/s; 6/1167 objects misplaced (0.514%); 1.5 MiB/s, 12 objects/s recovering 2021-03-15T22:24:08.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:08 smithi076 conmon[62156]: cluster 2021-03-15T22:24:07.083730+0000 mon.a (mon.0) 1072 : cluster [DBG] osdmap e332: 8 total, 8 up, 8 in 2021-03-15T22:24:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:08 smithi076 conmon[62156]: audit 2021-03-15T22:24:07.217668+0000 mon.a (mon.0) 1073 : audit [DBG] from='client.? 172.21.15.76:0/4129044425' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:08 smithi076 conmon[62156]: audit 2021-03-15T22:24:07.638066+0000 mgr.y (mgr.14144) 556 : audit [DBG] from='client.15276 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:08 smithi076 conmon[62156]: audit 2021-03-15T22:24:07.638844+0000 mon.a (mon.0) 1074 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'25548','5':'49764'}"}]: dispatch 2021-03-15T22:24:08.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:08.529 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:08.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:08.631 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:08.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:08.655 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:08.655 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:08.756 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:08.757 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:24:08.765 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:24:08.765 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:24:08.778 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:08.780 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:08.785 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2021-03-15T22:24:08.785 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd test-reweight-by-utilization 1000 1.0 2021-03-15T22:24:08.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:08.904 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:08.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:08.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:09.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:09.031 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:09.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:09.059 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:09.108 INFO:tasks.rados.rados.0.smithi076.stdout:1174: done (2 left) 2021-03-15T22:24:09.108 INFO:tasks.rados.rados.0.smithi076.stdout:1175: snap_remove snap 113 2021-03-15T22:24:09.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:09.156 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:09.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:09.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:09.198 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:09.258 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:09.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:09.282 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:09.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:09.384 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:09.407 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:09.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:09.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:09.442 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:09 smithi125 conmon[56739]: audit 2021-03-15T22:24:08.089472+0000 mon.a 2021-03-15T22:24:09.442 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:09 smithi125 conmon[56739]: (mon.0) 1075 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'25548','5':'49764'}"}]': finished 2021-03-15T22:24:09.443 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:09 smithi125 conmon[56739]: cluster 2021-03-15T22:24:08.089530+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2021-03-15T22:24:09.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:09 smithi076 conmon[56514]: audit 2021-03-15T22:24:08.089472+0000 mon.a 2021-03-15T22:24:09.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:09 smithi076 conmon[56514]: (mon.0) 1075 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'25548','5':'49764'}"}]': finished 2021-03-15T22:24:09.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:09 smithi076 conmon[56514]: cluster 2021-03-15T22:24:08.089530+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2021-03-15T22:24:09.444 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:09 smithi076 conmon[62156]: audit 2021-03-15T22:24:08.089472+0000 mon.a (mon. 2021-03-15T22:24:09.445 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:09 smithi076 conmon[62156]: 0) 1075 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'62260','3':'25548','5':'49764'}"}]': finished 2021-03-15T22:24:09.445 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:09 smithi076 conmon[62156]: cluster 2021-03-15T22:24:08.089530+0000 mon.a (mon.0) 1076 : cluster [DBG] osdmap e333: 8 total, 8 up, 8 in 2021-03-15T22:24:09.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-15T22:24:09.510 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:09.532 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:09.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:09.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:09.635 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:09.657 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:09.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:09.659 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:09.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:09.762 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:09.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:09.785 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:09.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:09.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:24:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:24:09.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:09.908 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:09.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:09.910 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:10.011 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:10.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:10.034 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:10.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:10.073 INFO:tasks.rados.rados.0.smithi076.stdout:1175: done (2 left) 2021-03-15T22:24:10.073 INFO:tasks.rados.rados.0.smithi076.stdout:1176: read oid 25 snap -1 2021-03-15T22:24:10.073 INFO:tasks.rados.rados.0.smithi076.stdout:1176: expect deleted 2021-03-15T22:24:10.074 INFO:tasks.rados.rados.0.smithi076.stdout:1177: snap_remove snap 109 2021-03-15T22:24:10.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:24:10.136 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:10.166 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:10.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:10.168 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:10.225 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:10 smithi125 conmon[56739]: cluster 2021-03-15T22:24:09.034032+0000 mgr.y (mgr.14144) 557 : cluster [DBG] pgmap v627: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 135 active+clean; 151 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.2 MiB/s wr, 5 op/s; 4/1176 objects degraded (0.340%); 18/1176 objects misplaced (1.531%); 3.3 MiB/s, 30 objects/s recovering 2021-03-15T22:24:10.226 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:10 smithi125 conmon[56739]: cluster 2021-03-15T22:24:09.086573+0000 mon.a (mon.0) 1077 : cluster [WRN] Health check failed: Degraded data redundancy: 4/1176 objects degraded (0.340%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:24:10.226 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:10 smithi125 conmon[56739]: cluster 2021-03-15T22:24:09.096714+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2021-03-15T22:24:10.226 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:10 smithi125 conmon[56739]: audit 2021-03-15T22:24:09.630328+0000 mgr.y (mgr.14144) 558 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:10.226 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:10 smithi125 conmon[56739]: cluster 2021-03-15T22:24:10.070556+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2021-03-15T22:24:10.269 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:10.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[56514]: cluster 2021-03-15T22:24:09.034032+0000 mgr.y (mgr.14144) 557 : cluster [DBG] pgmap v627: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 135 active+clean; 151 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.2 MiB/s wr, 5 op/s; 4/1176 objects degraded (0.340%); 18/1176 objects misplaced (1.531%); 3.3 MiB/s, 30 objects/s recovering 2021-03-15T22:24:10.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[56514]: cluster 2021-03-15T22:24:09.086573+0000 mon.a (mon.0) 1077 : cluster [WRN] Health check failed: Degraded data redundancy: 4/1176 objects degraded (0.340%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:24:10.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[56514]: cluster 2021-03-15T22:24:09.096714+0000 mon.a (mon.0) 1078 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2021-03-15T22:24:10.286 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[56514]: audit 2021-03-15T22:24:09.630328+0000 mgr.y (mgr.14144) 558 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:10.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[56514]: cluster 2021-03-15T22:24:10.070556+0000 mon.a (mon.0) 1079 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2021-03-15T22:24:10.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[62156]: cluster 2021-03-15T22:24:10.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:24:10.287 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[62156]: 24:09.034032+0000 mgr.y (mgr.14144) 557 : cluster [DBG] pgmap v627: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 135 active+clean; 151 MiB data, 556 MiB used, 715 GiB / 715 GiB avail; 511 B/s rd, 1.2 MiB/s wr, 5 op/s; 4/1176 objects degraded (0.340%); 18/1176 objects misplaced (1.531%); 3.3 MiB/s, 30 objects/s recovering 2021-03-15T22:24:10.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[62156]: cluster 2021-03-15T22:24:10.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[62156]: 2021-03-15T22:24:09.086573+0000 mon.a (mon.0) 1077 : cluster [WRN] Health check failed: Degraded data redundancy: 4/1176 objects degraded (0.340%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:24:10.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[62156]: cluster 2021-03-15T22:24:09.096714+0000 2021-03-15T22:24:10.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[62156]: mon.a (mon.0) 1078 : cluster [DBG] osdmap e334: 8 total, 8 up, 8 in 2021-03-15T22:24:10.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[62156]: audit 2021-03-15T22:24:09.630328+0000 mgr.y 2021-03-15T22:24:10.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[62156]: (mgr.14144) 558 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:10.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[62156]: cluster 2021-03-15T22:24:10.070556+0000 mon.a (mon 2021-03-15T22:24:10.289 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:10 smithi076 conmon[62156]: .0) 1079 : cluster [DBG] osdmap e335: 8 total, 8 up, 8 in 2021-03-15T22:24:10.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:10.294 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:10.295 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:10.396 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:10.418 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:10.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:10.419 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:10.520 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:10.542 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:10.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:10.544 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:10.645 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:10.668 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:10.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:10.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:10.770 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:10.794 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:10.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:10.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:10.896 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:10.918 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:10.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:10.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:11.020 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:11.042 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:11.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:11.044 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:11.075 INFO:tasks.rados.rados.0.smithi076.stdout:1177: done (3 left) 2021-03-15T22:24:11.076 INFO:tasks.rados.rados.0.smithi076.stdout:1178: read oid 35 snap 118 2021-03-15T22:24:11.076 INFO:tasks.rados.rados.0.smithi076.stdout:1178: expect deleted 2021-03-15T22:24:11.077 INFO:tasks.rados.rados.0.smithi076.stdout:1179: write oid 48 current snap is 120 2021-03-15T22:24:11.077 INFO:tasks.rados.rados.0.smithi076.stdout:1179: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2021-03-15T22:24:11.082 INFO:tasks.rados.rados.0.smithi076.stdout:1179: writing smithi07694878-48 from 477431 to 1176016 tid 1 2021-03-15T22:24:11.089 INFO:tasks.rados.rados.0.smithi076.stdout:1179: writing smithi07694878-48 from 1905056 to 2346864 tid 2 2021-03-15T22:24:11.096 INFO:tasks.rados.rados.0.smithi076.stdout:1179: writing smithi07694878-48 from 3074744 to 3277431 tid 3 2021-03-15T22:24:11.096 INFO:tasks.rados.rados.0.smithi076.stdout:1180: rollback oid 19 current snap is 120 2021-03-15T22:24:11.097 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 19 to 118 2021-03-15T22:24:11.097 INFO:tasks.rados.rados.0.smithi076.stdout:1173: finishing copy_from to smithi07694878-41 2021-03-15T22:24:11.097 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 271 (ObjNum 392 snap 106 seq_num 392) dirty exists 2021-03-15T22:24:11.097 INFO:tasks.rados.rados.0.smithi076.stdout:1173: finishing copy_from racing read to smithi07694878-41 2021-03-15T22:24:11.098 INFO:tasks.rados.rados.0.smithi076.stdout:1179: finishing write tid 1 to smithi07694878-48 2021-03-15T22:24:11.098 INFO:tasks.rados.rados.0.smithi076.stdout:1179: finishing write tid 2 to smithi07694878-48 2021-03-15T22:24:11.098 INFO:tasks.rados.rados.0.smithi076.stdout:1173: done (5 left) 2021-03-15T22:24:11.099 INFO:tasks.rados.rados.0.smithi076.stdout:1176: done (4 left) 2021-03-15T22:24:11.099 INFO:tasks.rados.rados.0.smithi076.stdout:1178: done (3 left) 2021-03-15T22:24:11.099 INFO:tasks.rados.rados.0.smithi076.stdout:1181: read oid 17 snap -1 2021-03-15T22:24:11.100 INFO:tasks.rados.rados.0.smithi076.stdout:1181: expect (ObjNum 422 snap 116 seq_num 422) 2021-03-15T22:24:11.100 INFO:tasks.rados.rados.0.smithi076.stdout:1182: copy_from oid 22 from oid 12 current snap is 120 2021-03-15T22:24:11.101 INFO:tasks.rados.rados.0.smithi076.stdout:1183: read oid 29 snap -1 2021-03-15T22:24:11.101 INFO:tasks.rados.rados.0.smithi076.stdout:1183: expect (ObjNum 397 snap 108 seq_num 397) 2021-03-15T22:24:11.102 INFO:tasks.rados.rados.0.smithi076.stdout:1184: snap_create 2021-03-15T22:24:11.103 INFO:tasks.rados.rados.0.smithi076.stdout:1180: finishing rollback tid 0 to smithi07694878-19 2021-03-15T22:24:11.103 INFO:tasks.rados.rados.0.smithi076.stdout:1179: finishing write tid 3 to smithi07694878-48 2021-03-15T22:24:11.104 INFO:tasks.rados.rados.0.smithi076.stdout:1179: finishing write tid 4 to smithi07694878-48 2021-03-15T22:24:11.104 INFO:tasks.rados.rados.0.smithi076.stdout:1180: finishing rollback tid 1 to smithi07694878-19 2021-03-15T22:24:11.105 INFO:tasks.rados.rados.0.smithi076.stdout:1179: finishing write tid 5 to smithi07694878-48 2021-03-15T22:24:11.106 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 289 (ObjNum 433 snap 120 seq_num 433) dirty exists 2021-03-15T22:24:11.106 INFO:tasks.rados.rados.0.smithi076.stdout:1179: left oid 48 (ObjNum 433 snap 120 seq_num 433) 2021-03-15T22:24:11.106 INFO:tasks.rados.rados.0.smithi076.stdout:1180: finishing rollback tid 2 to smithi07694878-19 2021-03-15T22:24:11.106 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 451 (ObjNum 387 snap 104 seq_num 387) dirty exists 2021-03-15T22:24:11.109 INFO:tasks.rados.rados.0.smithi076.stdout:1183: expect (ObjNum 397 snap 108 seq_num 397) 2021-03-15T22:24:11.145 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:11.168 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:11.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:11.170 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:11.187 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:24:11 smithi125 conmon[67289]: debug 2021-03-15T22:24:11.078+0000 7fd4d8831700 -1 osd.6 pg_epoch: 336 pg[7.7( v 336'218 (0'0,336'218] local-lis/les=172/173 n=5 ec=91/91 lis/c=172/172 les/c/f=173/173/0 sis=172) [6,0,1] r=0 lpr=172 luod=335'216 crt=336'218 lcod 334'214 mlcod 334'214 active+clean+snaptrim trimq=[db~1] ps=[dc~2,df~2,e3~1]] removing snap head 2021-03-15T22:24:11.187 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:11 smithi125 conmon[56739]: cluster 2021-03-15T22:24:10.412079+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:11.188 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:11 smithi125 conmon[56739]: cluster 2021-03-15T22:24:10.412298+0000 osd.1 (osd.1) 70 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:11.188 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:11 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:24:11.188 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:11 smithi125 conmon[56739]: 24:11.069889+0000 mon.a (mon. 2021-03-15T22:24:11.188 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:11 smithi125 conmon[56739]: 0) 1080 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2021-03-15T22:24:11.219 INFO:tasks.rados.rados.0.smithi076.stdout:1181: expect (ObjNum 422 snap 116 seq_num 422) 2021-03-15T22:24:11.271 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:11.292 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:11.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:11.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[56514]: cluster 2021-03-15T22:24:10.412079+0000 osd.1 (osd.1) 69 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:11.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[56514]: 2021-03-15T22:24:11.321 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[56514]: cluster 2021-03-15T22:24:10.412298+0000 osd.1 (osd.1) 70 : cluster [DBG] 2021-03-15T22:24:11.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[56514]: purged_snaps scrub ok 2021-03-15T22:24:11.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:24:11.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[56514]: 03-15T22:24:11.069889+0000 mon.a (mon.0 2021-03-15T22:24:11.322 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[56514]: ) 1080 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2021-03-15T22:24:11.323 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[75764]: debug 2021-03-15T22:24:11.323 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[75764]: 2021-03-15T22:24:11.069+0000 7f9fbafaf700 -1 osd.3 pg_epoch: 336 pg[7.a( v 335'439 (0'0,335'439] local-lis/les=268/269 n=9 ec=91/91 lis/c=268/268 les/c/f=269/269/0 sis=268) [3,2,5] r=0 lpr=268 crt=335'439 lcod 334'437 mlcod 334'437 active+clean+snaptrim trimq=[db~1] ps=[dc~2,df~2,e3~1]] removing snap head 2021-03-15T22:24:11.323 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[75764]: 2021-03-15T22:24:11.323 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[75764]: debug 2021-03-15T22:24:11.324 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[75764]: 2021-03-15T22:24:11.077+0000 7f9fbafaf700 -1 osd.3 pg_epoch: 336 pg[7.a( v 336'443 (0'0,336'443] local-lis/les=268/269 n=6 ec=91/91 lis/c=268/268 les/c/f=269/269/0 sis=268) [3,2,5] r=0 lpr=268 crt=336'443 lcod 336'441 mlcod 336'441 active+clean+snaptrim trimq=[db~1] ps=[dc~2,df~2,e3~1]] removing snap head 2021-03-15T22:24:11.324 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[75764]: 2021-03-15T22:24:11.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[62156]: cluster 2021-03-15T22:24:10.412079+0000 osd.1 (osd. 2021-03-15T22:24:11.324 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[62156]: 1) 69 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:11.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[62156]: cluster 2021-03-15T22:24:10.412298+0000 osd.1 (osd.1) 70 2021-03-15T22:24:11.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[62156]: : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:11.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:24:11.325 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[62156]: 03-15T22:24:11.069889+0000 mon.a (mon.0) 1080 : cluster [DBG] osdmap e336: 8 total, 8 up, 8 in 2021-03-15T22:24:11.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:11 smithi076 conmon[62156]: 2021-03-15T22:24:11.354 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:24:11.355 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:24:11.373 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:11.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:11.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:11.421 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:11.421 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:11.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:11.545 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:11.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:11.546 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:11.647 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:11.670 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:11.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:11.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:11.732 INFO:tasks.rados.rados.0.smithi076.stdout:1165: finishing rollback tid 1 to smithi07694878-32 2021-03-15T22:24:11.733 INFO:tasks.rados.rados.0.smithi076.stdout:1165: finishing rollback tid 2 to smithi07694878-32 2021-03-15T22:24:11.733 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 679 (ObjNum 371 snap 102 seq_num 371) dirty exists 2021-03-15T22:24:11.773 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:11.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:11.796 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:11.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:11.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:11.919 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:11.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:11.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:12.005 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:24:12.022 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:12.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:12.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:12.081 INFO:tasks.rados.rados.0.smithi076.stdout:1165: done (6 left) 2021-03-15T22:24:12.082 INFO:tasks.rados.rados.0.smithi076.stdout:1179: done (5 left) 2021-03-15T22:24:12.082 INFO:tasks.rados.rados.0.smithi076.stdout:1180: done (4 left) 2021-03-15T22:24:12.082 INFO:tasks.rados.rados.0.smithi076.stdout:1181: done (3 left) 2021-03-15T22:24:12.082 INFO:tasks.rados.rados.0.smithi076.stdout:1183: done (2 left) 2021-03-15T22:24:12.083 INFO:tasks.rados.rados.0.smithi076.stdout:1184: done (1 left) 2021-03-15T22:24:12.083 INFO:tasks.rados.rados.0.smithi076.stdout:1185: snap_create 2021-03-15T22:24:12.084 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:24:12.100 INFO:tasks.rados.rados.0.smithi076.stdout:1182: finishing copy_from to smithi07694878-22 2021-03-15T22:24:12.100 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 645 (ObjNum 430 snap 119 seq_num 430) dirty exists 2021-03-15T22:24:12.100 INFO:tasks.rados.rados.0.smithi076.stdout:1182: finishing copy_from racing read to smithi07694878-22 2021-03-15T22:24:12.148 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:12.170 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:12.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:12.171 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:12.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: cluster 2021-03-15T22:24:10.979313+0000 2021-03-15T22:24:12.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: osd.4 (osd.4) 77 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:24:12.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:24:12.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: 24:11.034473+0000 mgr.y (mgr.14144) 559 : cluster [DBG] pgmap v630: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 135 active+clean; 156 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.4 MiB/s wr, 5 op/s; 4/1185 objects degraded (0.338%); 20/1185 objects misplaced (1.688%); 2.5 MiB/s, 22 objects/s recovering 2021-03-15T22:24:12.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: audit 2021-03-15T22 2021-03-15T22:24:12.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: :24:11.349090+0000 mon.a (mon.0) 1081 : audit 2021-03-15T22:24:12.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: [INF] from='client.? 172.21.15.76:0/3630401811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:24:12.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: cluster 2021-03-15T22:24:12.066411+0000 mon.a 2021-03-15T22:24:12.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: (mon.0) 1082 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:12.175 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:24:12.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: -15T22:24:12.072727+0000 mon.a (mon.0) 2021-03-15T22:24:12.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: 1083 : audit [INF] from='client.? 172.21.15.76:0/3630401811' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:24:12.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: cluster 2021-03-15T22:24:12. 2021-03-15T22:24:12.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: 072798+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2021-03-15T22:24:12.176 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56514]: 2021-03-15T22:24:12.177 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[65511]: debug 2021-03-15T22:24:12.177 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[65511]: 2021-03-15T22:24:12.086+0000 7f1a1f877700 -1 osd.0 pg_epoch: 337 pg[7.6( v 332'645 (0'0,332'645] local-lis/les=336/337 n=13 ec=91/91 lis/c=336/336 les/c/f=337/337/0 sis=336) [0,7,2] r=0 lpr=336 crt=332'645 lcod 332'644 mlcod 0'0 active+clean+snaptrim trimq=[db~1,dd~1,df~1] ps=[dc~1,e0~1,e3~1]] removing snap head 2021-03-15T22:24:12.177 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[65511]: 2021-03-15T22:24:12.178 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[56844]: debug 2021-03-15T22:24:12.171+0000 7fc67af46700 -1 mgr.server reply reply (1) Operation not permitted no change 2021-03-15T22:24:12.178 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: cluster 2021-03-15T22:24:10.979313+0000 2021-03-15T22:24:12.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: osd.4 (osd.4) 77 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:24:12.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:24:12.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: 24:11.034473+0000 mgr.y (mgr.14144) 559 : cluster [DBG] pgmap v630: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 135 active+clean; 156 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.4 MiB/s wr, 5 op/s; 4/1185 objects degraded (0.338%); 20/1185 objects misplaced (1.688%); 2.5 MiB/s, 22 objects/s recovering 2021-03-15T22:24:12.179 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:24:12.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: 15T22:24:11.349090+0000 mon.a (mon 2021-03-15T22:24:12.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: .0) 1081 : audit [INF] from='client.? 172.21.15.76:0/3630401811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:24:12.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:24:12.180 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: 03-15T22:24:12.066411+0000 2021-03-15T22:24:12.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: mon.a (mon.0) 1082 : cluster 2021-03-15T22:24:12.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:12.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:24:12.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: 22:24:12.072727+0000 mon.a (mon. 2021-03-15T22:24:12.181 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: 0) 1083 : audit [INF] from='client.? 172.21.15.76:0/3630401811' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:24:12.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:24:12.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: 15T22:24:12.072798+0000 mon.a 2021-03-15T22:24:12.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: (mon.0) 1084 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2021-03-15T22:24:12.182 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:12 smithi076 conmon[62156]: 2021-03-15T22:24:12.183 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 414 (0%) 2021-03-15T22:24:12.183 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:24:12.183 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 13.809 -> 13.809 (expected baseline 6.72913) 2021-03-15T22:24:12.184 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 19 -> 19 pgs (0.36715 -> 0.36715 * mean) 2021-03-15T22:24:12.184 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.4 with 64 -> 64 pgs (1.23671 -> 1.23671 * mean) 2021-03-15T22:24:12.184 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:24:12.184 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:24:12.185 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 1 2021-03-15T22:24:12.185 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:24:12.185 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0008 2021-03-15T22:24:12.186 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0078 2021-03-15T22:24:12.186 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 weight 0.3898 -> 0.4044 2021-03-15T22:24:12.186 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:24:12.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:12.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:12 smithi125 conmon[56739]: cluster 2021-03-15T22:24:10.979313+0000 osd.4 (osd.4) 77 : cluster [DBG] 2021-03-15T22:24:12.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:12 smithi125 conmon[56739]: 4.1a scrub ok 2021-03-15T22:24:12.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:12 smithi125 conmon[56739]: cluster 2021-03-15T22:24:11.034473+0000 mgr.y (mgr.14144) 559 : cluster [DBG] pgmap v630: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 135 active+clean; 156 MiB data, 582 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 1.4 MiB/s wr, 5 op/s; 4/1185 objects degraded (0.338%); 20/1185 objects misplaced (1.688%); 2.5 MiB/s, 22 objects/s recovering 2021-03-15T22:24:12.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:12 smithi125 conmon[56739]: audit 2021-03-15T22:24:11.349090+0000 mon.a (mon.0) 1081 : audit [INF] from='client.? 172.21.15.76:0/3630401811' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:24:12.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:12 smithi125 conmon[56739]: cluster 2021-03-15T22:24:12.066411+0000 mon.a (mon.0) 1082 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:12.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:12 smithi125 conmon[56739]: audit 2021-03-15T22:24:12.072727+0000 mon.a (mon.0) 1083 : audit [INF] from='client.? 172.21.15.76:0/3630401811' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:24:12.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:12 smithi125 conmon[56739]: cluster 2021-03-15T22:24:12.072798+0000 mon.a (mon.0) 1084 : cluster [DBG] osdmap e337: 8 total, 8 up, 8 in 2021-03-15T22:24:12.288 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:12.295 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:12.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:12.296 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:12.397 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:12.420 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:12.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:12.422 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:12.523 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:12.545 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:12.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:12.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:12.648 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:12.670 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:12.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:12.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:12.772 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:12.795 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:12.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:12.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:12.898 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:12.919 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:12.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:12.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:13.022 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:13.044 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:13.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:13.075 INFO:tasks.rados.rados.0.smithi076.stdout:1182: done (1 left) 2021-03-15T22:24:13.075 INFO:tasks.rados.rados.0.smithi076.stdout:1185: done (0 left) 2021-03-15T22:24:13.076 INFO:tasks.rados.rados.0.smithi076.stdout:1186: rollback oid 32 current snap is 122 2021-03-15T22:24:13.076 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 32 to 117 2021-03-15T22:24:13.076 INFO:tasks.rados.rados.0.smithi076.stdout:1187: read oid 19 snap -1 2021-03-15T22:24:13.076 INFO:tasks.rados.rados.0.smithi076.stdout:1187: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:24:13.077 INFO:tasks.rados.rados.0.smithi076.stdout:1188: copy_from oid 10 from oid 11 current snap is 122 2021-03-15T22:24:13.077 INFO:tasks.rados.rados.0.smithi076.stdout:1189: snap_remove snap 121 2021-03-15T22:24:13.147 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:13.170 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:13.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:13.272 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:13.295 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:13.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:13.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:13.348 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '110', 'type': 'test-reweight-by-utilization'} 2021-03-15T22:24:13.349 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd test-reweight-by-utilization 110 1.0 2021-03-15T22:24:13.398 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:13.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[62156]: cluster 2021-03-15T22:24:12.017092+0000 osd.2 ( 2021-03-15T22:24:13.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[62156]: osd.2) 59 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:24:13.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[62156]: audit 2021-03-15T22:24:12.171238+0000 2021-03-15T22:24:13.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[62156]: mgr.y (mgr.14144) 560 : audit [DBG] from='client.24808 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:13.412 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[62156]: cluster 2021-03-15T22:24: 2021-03-15T22:24:13.412 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[62156]: 13.071765+0000 mon.a (mon.0) 1085 2021-03-15T22:24:13.412 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[62156]: : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2021-03-15T22:24:13.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[56514]: cluster 2021-03-15T22:24:12.017092+0000 osd.2 (osd. 2021-03-15T22:24:13.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[56514]: 2) 59 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:24:13.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[56514]: audit 2021-03-15T22:24:12. 2021-03-15T22:24:13.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[56514]: 171238+0000 mgr.y (mgr.14144) 560 : audit [DBG] from='client.24808 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:13.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:24:13.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[56514]: -03-15T22:24:13.071765+0000 mon.a 2021-03-15T22:24:13.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:13 smithi076 conmon[56514]: (mon.0) 1085 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2021-03-15T22:24:13.435 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:13.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:13.483 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:13 smithi125 conmon[56739]: cluster 2021-03-15T22:24:12.017092+0000 osd.2 (osd.2) 59 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:24:13.484 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:13 smithi125 conmon[56739]: audit 2021-03-15T22:24:12.171238+0000 mgr.y (mgr.14144) 560 : audit [DBG] from='client.24808 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:13.484 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:13 smithi125 conmon[56739]: cluster 2021-03-15T22:24:13.071765+0000 mon.a (mon.0) 1085 : cluster [DBG] osdmap e338: 8 total, 8 up, 8 in 2021-03-15T22:24:13.538 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:13.561 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:13.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:13.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:13.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:13.688 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:13.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:13.690 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:13.725 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:13.726 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:24:13.726 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:24:13.791 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:13.813 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:13.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:13.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:13.915 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:13.941 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:13.941 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:13.986 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:24:14.004 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:24:14.021 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:24:14.041 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:24:14.043 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:14.060 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:24:14.065 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:14.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:14.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:14.075 INFO:tasks.rados.rados.0.smithi076.stdout:1189: done (3 left) 2021-03-15T22:24:14.076 INFO:tasks.rados.rados.0.smithi076.stdout:1190: write oid 41 current snap is 122 2021-03-15T22:24:14.076 INFO:tasks.rados.rados.0.smithi076.stdout:1190: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2021-03-15T22:24:14.081 INFO:tasks.rados.rados.0.smithi076.stdout:1190: writing smithi07694878-41 from 494238 to 984425 tid 1 2021-03-15T22:24:14.083 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:24:14.089 INFO:tasks.rados.rados.0.smithi076.stdout:1190: writing smithi07694878-41 from 1563538 to 2265357 tid 2 2021-03-15T22:24:14.094 INFO:tasks.rados.rados.0.smithi076.stdout:1190: writing smithi07694878-41 from 2818520 to 3294238 tid 3 2021-03-15T22:24:14.094 INFO:tasks.rados.rados.0.smithi076.stdout:1191: write (excl) oid 30 current snap is 122 2021-03-15T22:24:14.095 INFO:tasks.rados.rados.0.smithi076.stdout:1191: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2021-03-15T22:24:14.100 INFO:tasks.rados.rados.0.smithi076.stdout:1191: writing smithi07694878-30 from 511045 to 1076481 tid 1 2021-03-15T22:24:14.103 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:24:14.106 INFO:tasks.rados.rados.0.smithi076.stdout:1191: writing smithi07694878-30 from 1622020 to 2067497 tid 2 2021-03-15T22:24:14.112 INFO:tasks.rados.rados.0.smithi076.stdout:1191: writing smithi07694878-30 from 2729590 to 3286445 tid 3 2021-03-15T22:24:14.113 INFO:tasks.rados.rados.0.smithi076.stdout:1192: snap_remove snap 120 2021-03-15T22:24:14.124 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:24:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:24:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:24:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:24:14.126 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:24:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:24:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:24:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:24:14.127 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:24:14.167 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:14.188 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:14.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:14.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:14.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:14.312 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:14.314 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:14.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-15T22:24:14.399 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:14 smithi125 conmon[56739]: cluster 2021-03-15T22:24:13.034918+0000 mgr.y (mgr.14144) 561 : cluster [DBG] pgmap v633: 138 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering, 135 active+clean; 154 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.1 MiB/s wr, 13 op/s; 8/1179 objects misplaced (0.679%); 11 MiB/s, 39 objects/s recovering 2021-03-15T22:24:14.400 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:14 smithi125 conmon[56739]: cluster 2021-03-15T22:24:13.107314+0000 mon.a (mon.0) 1086 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/1185 objects degraded (0.338%), 1 pg degraded) 2021-03-15T22:24:14.400 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:14 smithi125 conmon[56739]: audit 2021-03-15T22:24:13.966873+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.76:0/3815313328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:14.400 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:14 smithi125 conmon[56739]: cluster 2021-03-15T22:24:14.073403+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2021-03-15T22:24:14.415 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:14.438 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:14.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:14.440 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:14.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:14 smithi076 conmon[56514]: cluster 2021-03-15T22:24:13.034918+0000 mgr.y (mgr.14144) 561 : cluster [DBG] pgmap v633: 138 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering, 135 active+clean; 154 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.1 MiB/s wr, 13 op/s; 8/1179 objects misplaced (0.679%); 11 MiB/s, 39 objects/s recovering 2021-03-15T22:24:14.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:14 smithi076 conmon[56514]: cluster 2021-03-15T22:24:13.107314+0000 mon.a (mon.0) 1086 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/1185 objects degraded (0.338%), 1 pg degraded) 2021-03-15T22:24:14.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:14 smithi076 conmon[56514]: audit 2021-03-15T22:24:13.966873+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.76:0/3815313328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:14.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:14 smithi076 conmon[56514]: cluster 2021-03-15T22:24:14.073403+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2021-03-15T22:24:14.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:14 smithi076 conmon[62156]: cluster 2021-03-15T22:24:13.034918+0000 mgr.y (mgr.14144) 561 : cluster [DBG] pgmap v633: 138 pgs: 1 peering, 1 active+undersized+remapped, 1 active+recovering, 135 active+clean; 154 MiB data, 549 MiB used, 715 GiB / 715 GiB avail; 7.0 MiB/s rd, 2.1 MiB/s wr, 13 op/s; 8/1179 objects misplaced (0.679%); 11 MiB/s, 39 objects/s recovering 2021-03-15T22:24:14.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:14 smithi076 conmon[62156]: cluster 2021-03-15T22:24:13.107314+0000 mon.a (mon.0) 1086 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/1185 objects degraded (0.338%), 1 pg degraded) 2021-03-15T22:24:14.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:14 smithi076 conmon[62156]: audit 2021-03-15T22:24:13.966873+0000 mon.c (mon.1) 29 : audit [DBG] from='client.? 172.21.15.76:0/3815313328' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:14.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:14 smithi076 conmon[62156]: cluster 2021-03-15T22:24:14.073403+0000 mon.a (mon.0) 1087 : cluster [DBG] osdmap e339: 8 total, 8 up, 8 in 2021-03-15T22:24:14.540 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:14.563 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:14.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:14.564 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:14.651 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:14.665 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:14.687 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:14.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:14.689 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:14.790 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:14.812 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:14.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:14.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:14.914 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:14.936 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:14.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:14.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:15.039 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:15.061 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:15.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:15.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:15.075 INFO:tasks.rados.rados.0.smithi076.stdout:1192: done (5 left) 2021-03-15T22:24:15.075 INFO:tasks.rados.rados.0.smithi076.stdout:1193: snap_remove snap 118 2021-03-15T22:24:15.163 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:15.185 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:15.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:15.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:15.288 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:15.310 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:15.310 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:24:15.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:15.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:15.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:15.437 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:15.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:15.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:15.540 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:15.562 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:15.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:15.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:15.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:15.687 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:15.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:15.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:15.770 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:15.789 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:15.813 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:15.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:15.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:15.916 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:15.939 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:15.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:15.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:16.042 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:16.066 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:16.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:16.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:16.082 INFO:tasks.rados.rados.0.smithi076.stdout:1193: done (5 left) 2021-03-15T22:24:16.082 INFO:tasks.rados.rados.0.smithi076.stdout:1194: write oid 25 current snap is 122 2021-03-15T22:24:16.082 INFO:tasks.rados.rados.0.smithi076.stdout:1194: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2021-03-15T22:24:16.089 INFO:tasks.rados.rados.0.smithi076.stdout:1194: writing smithi07694878-25 from 527852 to 1168537 tid 1 2021-03-15T22:24:16.096 INFO:tasks.rados.rados.0.smithi076.stdout:1194: writing smithi07694878-25 from 1680502 to 2269637 tid 2 2021-03-15T22:24:16.102 INFO:tasks.rados.rados.0.smithi076.stdout:1194: writing smithi07694878-25 from 2757013 to 3327852 tid 3 2021-03-15T22:24:16.103 INFO:tasks.rados.rados.0.smithi076.stdout:1195: copy_from oid 3 from oid 22 current snap is 122 2021-03-15T22:24:16.103 INFO:tasks.rados.rados.0.smithi076.stdout:1186: finishing rollback tid 0 to smithi07694878-32 2021-03-15T22:24:16.103 INFO:tasks.rados.rados.0.smithi076.stdout:1187: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:24:16.169 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:16.189 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:24:16.189 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:24:16.189 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:24:16.189 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:24:16.190 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:24:16.195 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:16.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:16.196 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:16.263 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[56514]: cluster 2021-03-15T22:24:15.035384+0000 2021-03-15T22:24:16.263 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[56514]: mgr.y (mgr.14144) 562 : cluster [DBG] pgmap v636: 138 pgs: 1 peering, 1 active+undersized+remapped, 136 active+clean; 153 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s; 11 MiB/s, 34 objects/s recovering 2021-03-15T22:24:16.264 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[56514]: cluster 2021-03-15T22:24:15.073800+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2021-03-15T22:24:16.264 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: cluster 2021-03-15T22:24:16.264 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: 2021-03-15T22:24:16.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: 2021 2021-03-15T22:24:16.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: - 2021-03-15T22:24:16.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: 03 2021-03-15T22:24:16.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: - 2021-03-15T22:24:16.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: 15 2021-03-15T22:24:16.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: T22:24:15. 2021-03-15T22:24:16.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: 035384+0000 mgr.y (mgr.14144) 562 : cluster [DBG] pgmap v636: 138 pgs: 1 peering, 1 active+undersized+remapped, 136 active+clean; 153 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s; 11 MiB/s, 34 objects/s recovering 2021-03-15T22:24:16.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:24:16.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: -03-15T22:24:15.073800+0000 mon.a (mon 2021-03-15T22:24:16.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[62156]: .0) 1088 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2021-03-15T22:24:16.268 INFO:tasks.rados.rados.0.smithi076.stdout:1186: finishing rollback tid 1 to smithi07694878-32 2021-03-15T22:24:16.268 INFO:tasks.rados.rados.0.smithi076.stdout:1186: finishing rollback tid 2 to smithi07694878-32 2021-03-15T22:24:16.269 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 683 (ObjNum 371 snap 102 seq_num 371) dirty exists 2021-03-15T22:24:16.269 INFO:tasks.rados.rados.0.smithi076.stdout:1188: finishing copy_from to smithi07694878-10 2021-03-15T22:24:16.269 INFO:tasks.rados.rados.0.smithi076.stdout:1188: got expected ENOENT (src dne) 2021-03-15T22:24:16.269 INFO:tasks.rados.rados.0.smithi076.stdout:1188: finishing copy_from racing read to smithi07694878-10 2021-03-15T22:24:16.270 INFO:tasks.rados.rados.0.smithi076.stdout:1190: finishing write tid 1 to smithi07694878-41 2021-03-15T22:24:16.270 INFO:tasks.rados.rados.0.smithi076.stdout:1190: finishing write tid 2 to smithi07694878-41 2021-03-15T22:24:16.270 INFO:tasks.rados.rados.0.smithi076.stdout:1190: finishing write tid 3 to smithi07694878-41 2021-03-15T22:24:16.270 INFO:tasks.rados.rados.0.smithi076.stdout:1186: done (6 left) 2021-03-15T22:24:16.270 INFO:tasks.rados.rados.0.smithi076.stdout:1187: done (5 left) 2021-03-15T22:24:16.271 INFO:tasks.rados.rados.0.smithi076.stdout:1188: done (4 left) 2021-03-15T22:24:16.272 INFO:tasks.rados.rados.0.smithi076.stdout:1196: delete oid 1 current snap is 122 2021-03-15T22:24:16.272 INFO:tasks.rados.rados.0.smithi076.stdout:1190: finishing write tid 4 to smithi07694878-41 2021-03-15T22:24:16.272 INFO:tasks.rados.rados.0.smithi076.stdout:1190: finishing write tid 5 to smithi07694878-41 2021-03-15T22:24:16.273 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 294 (ObjNum 434 snap 122 seq_num 434) dirty exists 2021-03-15T22:24:16.273 INFO:tasks.rados.rados.0.smithi076.stdout:1190: left oid 41 (ObjNum 434 snap 122 seq_num 434) 2021-03-15T22:24:16.273 INFO:tasks.rados.rados.0.smithi076.stdout:1191: finishing write tid 1 to smithi07694878-30 2021-03-15T22:24:16.273 INFO:tasks.rados.rados.0.smithi076.stdout:1191: finishing write tid 2 to smithi07694878-30 2021-03-15T22:24:16.273 INFO:tasks.rados.rados.0.smithi076.stdout:1191: finishing write tid 3 to smithi07694878-30 2021-03-15T22:24:16.274 INFO:tasks.rados.rados.0.smithi076.stdout:1191: finishing write tid 4 to smithi07694878-30 2021-03-15T22:24:16.274 INFO:tasks.rados.rados.0.smithi076.stdout:1191: finishing write tid 5 to smithi07694878-30 2021-03-15T22:24:16.274 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 664 (ObjNum 435 snap 122 seq_num 435) dirty exists 2021-03-15T22:24:16.274 INFO:tasks.rados.rados.0.smithi076.stdout:1191: left oid 30 (ObjNum 435 snap 122 seq_num 435) 2021-03-15T22:24:16.275 INFO:tasks.rados.rados.0.smithi076.stdout:1194: finishing write tid 1 to smithi07694878-25 2021-03-15T22:24:16.275 INFO:tasks.rados.rados.0.smithi076.stdout:1194: finishing write tid 2 to smithi07694878-25 2021-03-15T22:24:16.275 INFO:tasks.rados.rados.0.smithi076.stdout:1194: finishing write tid 3 to smithi07694878-25 2021-03-15T22:24:16.276 INFO:tasks.rados.rados.0.smithi076.stdout:1194: finishing write tid 4 to smithi07694878-25 2021-03-15T22:24:16.276 INFO:tasks.rados.rados.0.smithi076.stdout:1194: finishing write tid 5 to smithi07694878-25 2021-03-15T22:24:16.277 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 224 (ObjNum 436 snap 122 seq_num 436) dirty exists 2021-03-15T22:24:16.277 INFO:tasks.rados.rados.0.smithi076.stdout:1194: left oid 25 (ObjNum 436 snap 122 seq_num 436) 2021-03-15T22:24:16.277 INFO:tasks.rados.rados.0.smithi076.stdout:1195: finishing copy_from to smithi07694878-3 2021-03-15T22:24:16.278 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 645 (ObjNum 430 snap 119 seq_num 430) dirty exists 2021-03-15T22:24:16.278 INFO:tasks.rados.rados.0.smithi076.stdout:1195: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:24:16.279 INFO:tasks.rados.rados.0.smithi076.stdout:1190: done (4 left) 2021-03-15T22:24:16.279 INFO:tasks.rados.rados.0.smithi076.stdout:1191: done (3 left) 2021-03-15T22:24:16.279 INFO:tasks.rados.rados.0.smithi076.stdout:1194: done (2 left) 2021-03-15T22:24:16.280 INFO:tasks.rados.rados.0.smithi076.stdout:1195: done (1 left) 2021-03-15T22:24:16.280 INFO:tasks.rados.rados.0.smithi076.stdout:1196: done (0 left) 2021-03-15T22:24:16.281 INFO:tasks.rados.rados.0.smithi076.stdout:1197: write oid 41 current snap is 122 2021-03-15T22:24:16.281 INFO:tasks.rados.rados.0.smithi076.stdout:1197: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2021-03-15T22:24:16.282 INFO:tasks.rados.rados.0.smithi076.stdout:1197: writing smithi07694878-41 from 561466 to 1352649 tid 1 2021-03-15T22:24:16.283 INFO:tasks.rados.rados.0.smithi076.stdout:1197: writing smithi07694878-41 from 2081113 to 2673917 tid 2 2021-03-15T22:24:16.287 INFO:tasks.rados.rados.0.smithi076.stdout:1197: writing smithi07694878-41 from 3095506 to 3361466 tid 3 2021-03-15T22:24:16.287 INFO:tasks.rados.rados.0.smithi076.stdout:1198: snap_create 2021-03-15T22:24:16.288 INFO:tasks.rados.rados.0.smithi076.stdout:1197: finishing write tid 1 to smithi07694878-41 2021-03-15T22:24:16.288 INFO:tasks.rados.rados.0.smithi076.stdout:1197: finishing write tid 2 to smithi07694878-41 2021-03-15T22:24:16.289 INFO:tasks.rados.rados.0.smithi076.stdout:1197: finishing write tid 3 to smithi07694878-41 2021-03-15T22:24:16.289 INFO:tasks.rados.rados.0.smithi076.stdout:1197: finishing write tid 4 to smithi07694878-41 2021-03-15T22:24:16.290 INFO:tasks.rados.rados.0.smithi076.stdout:1197: finishing write tid 5 to smithi07694878-41 2021-03-15T22:24:16.290 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 304 (ObjNum 438 snap 122 seq_num 438) dirty exists 2021-03-15T22:24:16.290 INFO:tasks.rados.rados.0.smithi076.stdout:1197: left oid 41 (ObjNum 438 snap 122 seq_num 438) 2021-03-15T22:24:16.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:16.321 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:16.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:16.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:16.393 INFO:teuthology.orchestra.run.smithi076.stdout:moved 35 / 414 (8.45411%) 2021-03-15T22:24:16.393 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:24:16.393 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 13.809 -> 13.8992 (expected baseline 6.72913) 2021-03-15T22:24:16.394 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 19 -> 21 pgs (0.36715 -> 0.405797 * mean) 2021-03-15T22:24:16.394 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.4 with 64 -> 67 pgs (1.23671 -> 1.29469 * mean) 2021-03-15T22:24:16.394 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:24:16.394 INFO:teuthology.orchestra.run.smithi076.stdout:oload 110 2021-03-15T22:24:16.395 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 1 2021-03-15T22:24:16.395 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:24:16.395 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0008 2021-03-15T22:24:16.395 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0008 2021-03-15T22:24:16.396 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2 weight 1.0000 -> 0.6168 2021-03-15T22:24:16.396 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 weight 0.9500 -> 0.8235 2021-03-15T22:24:16.396 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 weight 0.7593 -> 0.8856 2021-03-15T22:24:16.396 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 weight 0.3898 -> 0.3954 2021-03-15T22:24:16.397 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:24:16.424 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:16.446 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:16.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:16.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:16.517 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:16 smithi076 conmon[56844]: debug 2021-03-15T22:24:16.389+0000 7fc67af46700 -1 mgr.server reply reply (4) Interrupted system call no change 2021-03-15T22:24:16.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:16 smithi125 conmon[56739]: cluster 2021-03-15T22:24:15.035384+0000 mgr.y (mgr.14144) 562 : cluster [DBG] pgmap v636: 138 pgs: 1 peering, 1 active+undersized+remapped, 136 active+clean; 153 MiB data, 569 MiB used, 715 GiB / 715 GiB avail; 7.0 MiB/s rd, 1.0 MiB/s wr, 11 op/s; 11 MiB/s, 34 objects/s recovering 2021-03-15T22:24:16.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:16 smithi125 conmon[56739]: cluster 2021-03-15T22:24:15.073800+0000 mon.a (mon.0) 1088 : cluster [DBG] osdmap e340: 8 total, 8 up, 8 in 2021-03-15T22:24:16.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:16.571 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:16.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:16.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:16.674 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:16.696 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:16.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:16.697 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:16.798 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:16.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:16.821 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:16.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:16.913 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-15T22:24:16.923 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:16.945 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:16.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:16.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:17.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:24:17.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:17.070 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:17.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:17.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:17.100 INFO:tasks.rados.rados.0.smithi076.stdout:1197: done (1 left) 2021-03-15T22:24:17.101 INFO:tasks.rados.rados.0.smithi076.stdout:1198: done (0 left) 2021-03-15T22:24:17.101 INFO:tasks.rados.rados.0.smithi076.stdout:1199: read oid 37 snap -1 2021-03-15T22:24:17.101 INFO:tasks.rados.rados.0.smithi076.stdout:1199: expect deleted 2021-03-15T22:24:17.101 INFO:tasks.rados.rados.0.smithi076.stdout:1200: read oid 48 snap 117 2021-03-15T22:24:17.101 INFO:tasks.rados.rados.0.smithi076.stdout:1200: expect (ObjNum 420 snap 116 seq_num 420) 2021-03-15T22:24:17.102 INFO:tasks.rados.rados.0.smithi076.stdout:1201: delete oid 44 current snap is 123 2021-03-15T22:24:17.103 INFO:tasks.rados.rados.0.smithi076.stdout:1201: done (2 left) 2021-03-15T22:24:17.103 INFO:tasks.rados.rados.0.smithi076.stdout:1202: read oid 2 snap -1 2021-03-15T22:24:17.103 INFO:tasks.rados.rados.0.smithi076.stdout:1202: expect (ObjNum 383 snap 103 seq_num 383) 2021-03-15T22:24:17.103 INFO:tasks.rados.rados.0.smithi076.stdout:1203: write oid 22 current snap is 123 2021-03-15T22:24:17.104 INFO:tasks.rados.rados.0.smithi076.stdout:1203: seq_num 440 ranges {595080=541681,1798077=596473,3150352=244728} 2021-03-15T22:24:17.109 INFO:tasks.rados.rados.0.smithi076.stdout:1203: writing smithi07694878-22 from 595080 to 1136761 tid 1 2021-03-15T22:24:17.118 INFO:tasks.rados.rados.0.smithi076.stdout:1203: writing smithi07694878-22 from 1798077 to 2394550 tid 2 2021-03-15T22:24:17.123 INFO:tasks.rados.rados.0.smithi076.stdout:1203: writing smithi07694878-22 from 3150352 to 3395080 tid 3 2021-03-15T22:24:17.123 INFO:tasks.rados.rados.0.smithi076.stdout:1204: copy_from oid 16 from oid 34 current snap is 123 2021-03-15T22:24:17.124 INFO:tasks.rados.rados.0.smithi076.stdout:1202: expect (ObjNum 383 snap 103 seq_num 383) 2021-03-15T22:24:17.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:17.191 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:17.194 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:17.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:17.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:17.226 INFO:tasks.rados.rados.0.smithi076.stdout:1199: done (4 left) 2021-03-15T22:24:17.226 INFO:tasks.rados.rados.0.smithi076.stdout:1202: done (3 left) 2021-03-15T22:24:17.227 INFO:tasks.rados.rados.0.smithi076.stdout:1205: copy_from oid 26 from oid 13 current snap is 123 2021-03-15T22:24:17.227 INFO:tasks.rados.rados.0.smithi076.stdout:1203: finishing write tid 1 to smithi07694878-22 2021-03-15T22:24:17.227 INFO:tasks.rados.rados.0.smithi076.stdout:1200: expect (ObjNum 420 snap 116 seq_num 420) 2021-03-15T22:24:17.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:17 smithi125 conmon[56739]: cluster 2021-03-15T22:24:16.080162+0000 mon.a ( 2021-03-15T22:24:17.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:17 smithi125 conmon[56739]: mon.0) 1089 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2021-03-15T22:24:17.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:17 smithi125 conmon[56739]: cluster 2021-03-15T22:24:16.165163+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:17.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:17 smithi125 conmon[56739]: cluster 2021-03-15T22:24:16.165420+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:17.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:17 smithi125 conmon[56739]: audit 2021-03-15T22:24:16.390343+0000 mgr.y (mgr 2021-03-15T22:24:17.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:17 smithi125 conmon[56739]: .14144) 563 : audit [DBG] from='client.24814 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:17.297 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:17.319 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:17.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:17.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:17.404 INFO:tasks.rados.rados.0.smithi076.stdout:1203: finishing write tid 2 to smithi07694878-22 2021-03-15T22:24:17.405 INFO:tasks.rados.rados.0.smithi076.stdout:1203: finishing write tid 3 to smithi07694878-22 2021-03-15T22:24:17.405 INFO:tasks.rados.rados.0.smithi076.stdout:1203: finishing write tid 4 to smithi07694878-22 2021-03-15T22:24:17.405 INFO:tasks.rados.rados.0.smithi076.stdout:1203: finishing write tid 5 to smithi07694878-22 2021-03-15T22:24:17.405 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 675 (ObjNum 440 snap 123 seq_num 440) dirty exists 2021-03-15T22:24:17.406 INFO:tasks.rados.rados.0.smithi076.stdout:1203: left oid 22 (ObjNum 440 snap 123 seq_num 440) 2021-03-15T22:24:17.406 INFO:tasks.rados.rados.0.smithi076.stdout:1204: finishing copy_from to smithi07694878-16 2021-03-15T22:24:17.406 INFO:tasks.rados.rados.0.smithi076.stdout:1204: got expected ENOENT (src dne) 2021-03-15T22:24:17.407 INFO:tasks.rados.rados.0.smithi076.stdout:1204: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:24:17.407 INFO:tasks.rados.rados.0.smithi076.stdout:1205: finishing copy_from to smithi07694878-26 2021-03-15T22:24:17.407 INFO:tasks.rados.rados.0.smithi076.stdout:1205: got expected ENOENT (src dne) 2021-03-15T22:24:17.407 INFO:tasks.rados.rados.0.smithi076.stdout:1205: finishing copy_from racing read to smithi07694878-26 2021-03-15T22:24:17.408 INFO:tasks.rados.rados.0.smithi076.stdout:1200: done (3 left) 2021-03-15T22:24:17.408 INFO:tasks.rados.rados.0.smithi076.stdout:1203: done (2 left) 2021-03-15T22:24:17.408 INFO:tasks.rados.rados.0.smithi076.stdout:1204: done (1 left) 2021-03-15T22:24:17.408 INFO:tasks.rados.rados.0.smithi076.stdout:1205: done (0 left) 2021-03-15T22:24:17.409 INFO:tasks.rados.rados.0.smithi076.stdout:1206: snap_create 2021-03-15T22:24:17.423 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:17.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:17.447 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:17.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:17.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[56514]: cluster 2021-03-15T22:24:16.080162+0000 mon.a ( 2021-03-15T22:24:17.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[56514]: mon.0) 1089 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2021-03-15T22:24:17.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[56514]: cluster 2021-03-15T22:24:16.165163+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:17.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[56514]: cluster 2021-03-15T22:24:16.165420+0000 osd.5 (osd.5) 62 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:17.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[56514]: audit 2021-03-15T22:24:16.390343+0000 mgr.y (mgr.14144) 563 : audit [DBG] from='client.24814 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:17.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[62156]: cluster 2021-03-15T22:24:16.080162+0000 2021-03-15T22:24:17.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[62156]: mon.a (mon.0) 1089 : cluster [DBG] osdmap e341: 8 total, 8 up, 8 in 2021-03-15T22:24:17.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:24:17.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[62156]: 22:24:16.165163+0000 osd.5 (osd.5) 61 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:17.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:24:17.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[62156]: 03-15T22:24:16.165420+0000 osd.5 (osd. 2021-03-15T22:24:17.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[62156]: 5) 62 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:17.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[62156]: audit 2021-03-15T22:24:17.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[62156]: 2021 2021-03-15T22:24:17.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[62156]: -03-15T22:24:16 2021-03-15T22:24:17.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:17 smithi076 conmon[62156]: .390343+0000 mgr.y (mgr.14144) 563 : audit [DBG] from='client.24814 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 110, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:17.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:17.571 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:17.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:17.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:17.674 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:17.697 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:17.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:17.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:17.800 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:17.822 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:17.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:17.824 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'reweight-by-utilization'} 2021-03-15T22:24:17.824 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight-by-utilization 1000 0.05 2021-03-15T22:24:17.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:17.883 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:17.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:17.949 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:17.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:17.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:18.052 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:18.074 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:18.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:18.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:18.109 INFO:tasks.rados.rados.0.smithi076.stdout:1206: done (0 left) 2021-03-15T22:24:18.109 INFO:tasks.rados.rados.0.smithi076.stdout:1207: write (excl) oid 21 current snap is 124 2021-03-15T22:24:18.109 INFO:tasks.rados.rados.0.smithi076.stdout:1207: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2021-03-15T22:24:18.116 INFO:tasks.rados.rados.0.smithi076.stdout:1207: writing smithi07694878-21 from 611887 to 1345170 tid 1 2021-03-15T22:24:18.125 INFO:tasks.rados.rados.0.smithi076.stdout:1207: writing smithi07694878-21 from 1972912 to 2429396 tid 2 2021-03-15T22:24:18.131 INFO:tasks.rados.rados.0.smithi076.stdout:1207: writing smithi07694878-21 from 2894128 to 3411887 tid 3 2021-03-15T22:24:18.131 INFO:tasks.rados.rados.0.smithi076.stdout:1208: write (excl) oid 42 current snap is 124 2021-03-15T22:24:18.131 INFO:tasks.rados.rados.0.smithi076.stdout:1208: seq_num 442 ranges {628694=408532,1515041=600142,2805198=552112} 2021-03-15T22:24:18.137 INFO:tasks.rados.rados.0.smithi076.stdout:1208: writing smithi07694878-42 from 628694 to 1037226 tid 1 2021-03-15T22:24:18.143 INFO:tasks.rados.rados.0.smithi076.stdout:1208: writing smithi07694878-42 from 1515041 to 2115183 tid 2 2021-03-15T22:24:18.150 INFO:tasks.rados.rados.0.smithi076.stdout:1208: writing smithi07694878-42 from 2805198 to 3357310 tid 3 2021-03-15T22:24:18.150 INFO:tasks.rados.rados.0.smithi076.stdout:1209: write oid 18 current snap is 124 2021-03-15T22:24:18.151 INFO:tasks.rados.rados.0.smithi076.stdout:1207: finishing write tid 1 to smithi07694878-21 2021-03-15T22:24:18.151 INFO:tasks.rados.rados.0.smithi076.stdout:1207: finishing write tid 2 to smithi07694878-21 2021-03-15T22:24:18.151 INFO:tasks.rados.rados.0.smithi076.stdout:1207: finishing write tid 3 to smithi07694878-21 2021-03-15T22:24:18.151 INFO:tasks.rados.rados.0.smithi076.stdout:1209: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2021-03-15T22:24:18.156 INFO:tasks.rados.rados.0.smithi076.stdout:1209: writing smithi07694878-18 from 645501 to 1129282 tid 1 2021-03-15T22:24:18.161 INFO:tasks.rados.rados.0.smithi076.stdout:1209: writing smithi07694878-18 from 1573523 to 2033676 tid 2 2021-03-15T22:24:18.169 INFO:tasks.rados.rados.0.smithi076.stdout:1209: writing smithi07694878-18 from 2548974 to 3312358 tid 3 2021-03-15T22:24:18.169 INFO:tasks.rados.rados.0.smithi076.stdout:1210: read oid 27 snap -1 2021-03-15T22:24:18.170 INFO:tasks.rados.rados.0.smithi076.stdout:1210: expect (ObjNum 416 snap 112 seq_num 416) 2021-03-15T22:24:18.170 INFO:tasks.rados.rados.0.smithi076.stdout:1211: snap_create 2021-03-15T22:24:18.170 INFO:tasks.rados.rados.0.smithi076.stdout:1207: finishing write tid 4 to smithi07694878-21 2021-03-15T22:24:18.170 INFO:tasks.rados.rados.0.smithi076.stdout:1207: finishing write tid 5 to smithi07694878-21 2021-03-15T22:24:18.171 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 710 (ObjNum 441 snap 124 seq_num 441) dirty exists 2021-03-15T22:24:18.171 INFO:tasks.rados.rados.0.smithi076.stdout:1207: left oid 21 (ObjNum 441 snap 124 seq_num 441) 2021-03-15T22:24:18.171 INFO:tasks.rados.rados.0.smithi076.stdout:1208: finishing write tid 1 to smithi07694878-42 2021-03-15T22:24:18.171 INFO:tasks.rados.rados.0.smithi076.stdout:1208: finishing write tid 2 to smithi07694878-42 2021-03-15T22:24:18.172 INFO:tasks.rados.rados.0.smithi076.stdout:1208: finishing write tid 3 to smithi07694878-42 2021-03-15T22:24:18.172 INFO:tasks.rados.rados.0.smithi076.stdout:1208: finishing write tid 4 to smithi07694878-42 2021-03-15T22:24:18.172 INFO:tasks.rados.rados.0.smithi076.stdout:1208: finishing write tid 5 to smithi07694878-42 2021-03-15T22:24:18.172 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 715 (ObjNum 442 snap 124 seq_num 442) dirty exists 2021-03-15T22:24:18.172 INFO:tasks.rados.rados.0.smithi076.stdout:1208: left oid 42 (ObjNum 442 snap 124 seq_num 442) 2021-03-15T22:24:18.173 INFO:tasks.rados.rados.0.smithi076.stdout:1209: finishing write tid 1 to smithi07694878-18 2021-03-15T22:24:18.173 INFO:tasks.rados.rados.0.smithi076.stdout:1209: finishing write tid 2 to smithi07694878-18 2021-03-15T22:24:18.174 INFO:tasks.rados.rados.0.smithi076.stdout:1209: finishing write tid 3 to smithi07694878-18 2021-03-15T22:24:18.174 INFO:tasks.rados.rados.0.smithi076.stdout:1209: finishing write tid 4 to smithi07694878-18 2021-03-15T22:24:18.174 INFO:tasks.rados.rados.0.smithi076.stdout:1209: finishing write tid 5 to smithi07694878-18 2021-03-15T22:24:18.174 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 282 (ObjNum 443 snap 124 seq_num 443) dirty exists 2021-03-15T22:24:18.175 INFO:tasks.rados.rados.0.smithi076.stdout:1209: left oid 18 (ObjNum 443 snap 124 seq_num 443) 2021-03-15T22:24:18.175 INFO:tasks.rados.rados.0.smithi076.stdout:1210: expect (ObjNum 416 snap 112 seq_num 416) 2021-03-15T22:24:18.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:18.200 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:18.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:18.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:18.210 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:18.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:18 smithi076 conmon[62156]: cluster 2021-03-15T22:24:17.035803+0000 mgr.y ( 2021-03-15T22:24:18.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:18 smithi076 conmon[62156]: mgr.14144) 564 : cluster [DBG] pgmap v639: 138 pgs: 1 peering, 137 active+clean; 148 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 9 op/s; 66 B/s, 1 objects/s recovering 2021-03-15T22:24:18.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:18 smithi076 conmon[62156]: cluster 2021-03-15T22:24:17.095396+0000 2021-03-15T22:24:18.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:18 smithi076 conmon[62156]: mon.a (mon.0) 1090 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2021-03-15T22:24:18.261 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:18 smithi076 conmon[56514]: cluster 2021-03-15T22:24:17.035803+0000 mgr.y (mgr.14144) 564 : cluster [DBG] pgmap v639: 138 pgs: 1 peering, 137 active+clean; 148 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 9 op/s; 66 B/s, 1 objects/s recovering 2021-03-15T22:24:18.261 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:18 smithi076 conmon[56514]: cluster 2021-03-15T22:24:17.095396+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2021-03-15T22:24:18.303 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:18.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:18.329 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:18.330 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:18.431 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:18.440 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:18 smithi125 conmon[56739]: cluster 2021-03-15T22:24:17.035803+0000 mgr.y (mgr.14144) 564 : cluster [DBG] pgmap v639: 138 pgs: 1 peering, 137 active+clean; 148 MiB data, 545 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 9 op/s; 66 B/s, 1 objects/s recovering 2021-03-15T22:24:18.441 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:18 smithi125 conmon[56739]: cluster 2021-03-15T22:24:17.095396+0000 mon.a (mon.0) 1090 : cluster [DBG] osdmap e342: 8 total, 8 up, 8 in 2021-03-15T22:24:18.455 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:18.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:18.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:18.557 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:18.580 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:18.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:18.581 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:18.682 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:18.693 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:24:18.693 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:24:18.693 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:24:18.694 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:24:18.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.694 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:277 7.0 7:08414d91:::smithi07694878-42:head [copy-from ver 6] snapc 8=[8] ondisk+write+known_if_redirected e104)", 2021-03-15T22:24:18.694 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.220857+0000", 2021-03-15T22:24:18.695 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 239.46800489200001, 2021-03-15T22:24:18.695 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01426084, 2021-03-15T22:24:18.695 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.695 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:18.695 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.696 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.696 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.696 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 277 2021-03-15T22:24:18.696 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.697 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.697 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.697 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.697 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220857+0000", 2021-03-15T22:24:18.697 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.698 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.698 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.698 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220857+0000", 2021-03-15T22:24:18.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.73e-06 2021-03-15T22:24:18.699 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.699 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.699 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.699 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220858+0000", 2021-03-15T22:24:18.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.277e-06 2021-03-15T22:24:18.700 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220860+0000", 2021-03-15T22:24:18.701 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8489999999999995e-06 2021-03-15T22:24:18.701 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.701 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.701 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.701 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220868+0000", 2021-03-15T22:24:18.702 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5480000000000003e-06 2021-03-15T22:24:18.702 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.702 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.702 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.703 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220873+0000", 2021-03-15T22:24:18.703 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001357249 2021-03-15T22:24:18.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.703 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.703 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.704 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222230+0000", 2021-03-15T22:24:18.704 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4117999999999998e-05 2021-03-15T22:24:18.704 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.704 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.704 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.705 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222294+0000", 2021-03-15T22:24:18.705 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035570179999999999 2021-03-15T22:24:18.705 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.706 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:24:18.706 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.225851+0000", 2021-03-15T22:24:18.706 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038919620000000001 2021-03-15T22:24:18.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.706 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.707 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:24:18.707 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.229743+0000", 2021-03-15T22:24:18.707 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024645940000000001 2021-03-15T22:24:18.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.707 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.708 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.708 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.232208+0000", 2021-03-15T22:24:18.708 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002799171 2021-03-15T22:24:18.708 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.709 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.709 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.709 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235007+0000", 2021-03-15T22:24:18.709 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3729e-05 2021-03-15T22:24:18.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.710 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.710 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:18.710 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235061+0000", 2021-03-15T22:24:18.710 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6595000000000003e-05 2021-03-15T22:24:18.710 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.711 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.711 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.711 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235117+0000", 2021-03-15T22:24:18.711 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01426084 2021-03-15T22:24:18.712 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.712 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.712 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.712 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.712 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.713 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:278 7.0 7:08414d91:::smithi07694878-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e104)", 2021-03-15T22:24:18.713 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.220878+0000", 2021-03-15T22:24:18.713 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 239.46798345900001, 2021-03-15T22:24:18.713 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014453041999999999, 2021-03-15T22:24:18.713 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.714 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:24:18.714 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.714 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.714 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.715 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 278 2021-03-15T22:24:18.715 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.715 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.715 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.715 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.716 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220878+0000", 2021-03-15T22:24:18.716 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.716 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.716 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.716 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.717 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220878+0000", 2021-03-15T22:24:18.717 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.037e-06 2021-03-15T22:24:18.717 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.717 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.718 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.718 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220879+0000", 2021-03-15T22:24:18.718 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.187e-06 2021-03-15T22:24:18.718 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.718 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.719 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.719 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220880+0000", 2021-03-15T22:24:18.719 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.879e-06 2021-03-15T22:24:18.719 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.719 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.720 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.720 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220886+0000", 2021-03-15T22:24:18.720 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0269999999999997e-06 2021-03-15T22:24:18.720 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.721 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.721 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.721 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.220890+0000", 2021-03-15T22:24:18.721 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001548542 2021-03-15T22:24:18.721 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.722 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.722 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.722 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222439+0000", 2021-03-15T22:24:18.722 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1516000000000001e-05 2021-03-15T22:24:18.723 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.723 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.723 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:18.723 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.222470+0000", 2021-03-15T22:24:18.724 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066482809999999998 2021-03-15T22:24:18.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.724 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.724 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.724 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.229118+0000", 2021-03-15T22:24:18.725 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6447999999999997e-05 2021-03-15T22:24:18.725 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.725 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.725 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:18.726 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.229195+0000", 2021-03-15T22:24:18.726 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0059693100000000002 2021-03-15T22:24:18.726 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.726 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.726 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.727 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235164+0000", 2021-03-15T22:24:18.727 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4153999999999994e-05 2021-03-15T22:24:18.727 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.727 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.727 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.728 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235248+0000", 2021-03-15T22:24:18.728 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2661e-05 2021-03-15T22:24:18.728 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.728 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.728 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.729 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.235331+0000", 2021-03-15T22:24:18.729 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014453041999999999 2021-03-15T22:24:18.729 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.729 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.730 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.730 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.730 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.730 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:294 7.0 7:0cc37e3d:::smithi07694878-21:head [copy-from ver 3] snapc 8=[8] ondisk+write+known_if_redirected e104)", 2021-03-15T22:24:18.730 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.394807+0000", 2021-03-15T22:24:18.731 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 239.294054348, 2021-03-15T22:24:18.731 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018445559, 2021-03-15T22:24:18.731 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.731 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:18.731 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.732 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.732 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.732 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 294 2021-03-15T22:24:18.732 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.733 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.733 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.733 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.733 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394807+0000", 2021-03-15T22:24:18.733 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.734 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.734 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.734 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.734 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394807+0000", 2021-03-15T22:24:18.734 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9659999999999999e-06 2021-03-15T22:24:18.735 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.735 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.735 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.735 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394809+0000", 2021-03-15T22:24:18.736 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2610000000000001e-06 2021-03-15T22:24:18.736 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.736 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.736 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.737 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394810+0000", 2021-03-15T22:24:18.737 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0881e-05 2021-03-15T22:24:18.737 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.737 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.737 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.738 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394821+0000", 2021-03-15T22:24:18.738 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8509999999999994e-06 2021-03-15T22:24:18.738 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.738 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.739 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.739 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394831+0000", 2021-03-15T22:24:18.739 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035723980000000001 2021-03-15T22:24:18.739 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.739 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.740 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.740 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398403+0000", 2021-03-15T22:24:18.740 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5474000000000004e-05 2021-03-15T22:24:18.740 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.740 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.741 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.741 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398469+0000", 2021-03-15T22:24:18.741 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0069378080000000002 2021-03-15T22:24:18.741 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.742 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.742 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:24:18.742 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.405407+0000", 2021-03-15T22:24:18.742 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033175689999999998 2021-03-15T22:24:18.742 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.743 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.743 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:24:18.743 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.408724+0000", 2021-03-15T22:24:18.743 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042654010000000003 2021-03-15T22:24:18.743 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.744 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.744 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.744 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.412990+0000", 2021-03-15T22:24:18.744 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016210500000000001 2021-03-15T22:24:18.745 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.745 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.745 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.745 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413152+0000", 2021-03-15T22:24:18.746 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0185999999999997e-05 2021-03-15T22:24:18.746 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.746 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.746 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:18.746 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413202+0000", 2021-03-15T22:24:18.747 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0658999999999997e-05 2021-03-15T22:24:18.747 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.747 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.747 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.748 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413253+0000", 2021-03-15T22:24:18.748 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018445559 2021-03-15T22:24:18.748 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.748 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.748 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.749 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.749 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.749 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:295 7.0 7:0cc37e3d:::smithi07694878-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e104)", 2021-03-15T22:24:18.749 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:19.394836+0000", 2021-03-15T22:24:18.750 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 239.29402549299999, 2021-03-15T22:24:18.750 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018595192999999999, 2021-03-15T22:24:18.750 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.750 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:24:18.750 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.751 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.751 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.751 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 295 2021-03-15T22:24:18.751 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.751 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.752 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.752 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.752 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394836+0000", 2021-03-15T22:24:18.752 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.753 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.753 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.753 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.753 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394836+0000", 2021-03-15T22:24:18.754 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4300000000000002e-07 2021-03-15T22:24:18.754 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.754 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.754 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.754 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394837+0000", 2021-03-15T22:24:18.755 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5799999999999998e-07 2021-03-15T22:24:18.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.755 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.755 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394838+0000", 2021-03-15T22:24:18.756 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9940000000000002e-06 2021-03-15T22:24:18.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.756 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.757 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394842+0000", 2021-03-15T22:24:18.757 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.339e-06 2021-03-15T22:24:18.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.757 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.758 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.394844+0000", 2021-03-15T22:24:18.758 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003783809 2021-03-15T22:24:18.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.758 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.759 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398628+0000", 2021-03-15T22:24:18.759 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0221e-05 2021-03-15T22:24:18.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.760 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:18.760 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.398668+0000", 2021-03-15T22:24:18.760 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0094639070000000006 2021-03-15T22:24:18.760 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.760 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.761 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.761 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.408132+0000", 2021-03-15T22:24:18.761 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0552999999999999e-05 2021-03-15T22:24:18.761 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.761 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.762 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:18.762 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.408212+0000", 2021-03-15T22:24:18.762 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050811019999999997 2021-03-15T22:24:18.762 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413294+0000", 2021-03-15T22:24:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9265999999999997e-05 2021-03-15T22:24:18.763 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.764 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.764 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.764 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413363+0000", 2021-03-15T22:24:18.764 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8300999999999995e-05 2021-03-15T22:24:18.764 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.765 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.765 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.765 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:19.413431+0000", 2021-03-15T22:24:18.765 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018595192999999999 2021-03-15T22:24:18.766 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.766 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.766 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.766 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.766 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.767 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:366 7.0 7:0732c451:::smithi07694878-32:head [copy-from ver 9] snapc a=[a] ondisk+write+known_if_redirected e108)", 2021-03-15T22:24:18.767 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.105457+0000", 2021-03-15T22:24:18.767 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 235.583404817, 2021-03-15T22:24:18.768 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018098547, 2021-03-15T22:24:18.768 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.768 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:18.768 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.768 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.769 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.769 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 366 2021-03-15T22:24:18.769 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.769 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.770 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.771 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.771 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105457+0000", 2021-03-15T22:24:18.771 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.771 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.772 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.772 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.772 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105457+0000", 2021-03-15T22:24:18.772 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9000000000000003e-06 2021-03-15T22:24:18.772 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.773 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.773 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.773 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105463+0000", 2021-03-15T22:24:18.773 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1069999999999998e-06 2021-03-15T22:24:18.773 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.774 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.774 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.774 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105467+0000", 2021-03-15T22:24:18.774 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0707999999999999e-05 2021-03-15T22:24:18.774 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.775 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.775 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.775 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105487+0000", 2021-03-15T22:24:18.775 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4830000000000005e-06 2021-03-15T22:24:18.776 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.776 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.776 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.776 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105497+0000", 2021-03-15T22:24:18.776 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9896999999999998e-05 2021-03-15T22:24:18.777 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.777 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.777 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.777 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105577+0000", 2021-03-15T22:24:18.777 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000138195 2021-03-15T22:24:18.778 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.778 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.778 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.778 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105715+0000", 2021-03-15T22:24:18.779 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0074524730000000003 2021-03-15T22:24:18.779 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.779 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.779 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:24:18.779 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.113167+0000", 2021-03-15T22:24:18.780 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053675119999999996 2021-03-15T22:24:18.780 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.780 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.780 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:24:18.781 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.118535+0000", 2021-03-15T22:24:18.781 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002009102 2021-03-15T22:24:18.781 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.781 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.781 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.782 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.120544+0000", 2021-03-15T22:24:18.782 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028954580000000001 2021-03-15T22:24:18.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.782 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.783 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123439+0000", 2021-03-15T22:24:18.783 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6292999999999999e-05 2021-03-15T22:24:18.783 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.783 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.784 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:18.784 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123486+0000", 2021-03-15T22:24:18.784 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9418999999999994e-05 2021-03-15T22:24:18.784 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.784 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.785 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.785 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123555+0000", 2021-03-15T22:24:18.785 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018098547 2021-03-15T22:24:18.785 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.786 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.786 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.786 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.786 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.786 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:367 7.0 7:0732c451:::smithi07694878-32:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e108)", 2021-03-15T22:24:18.787 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.105520+0000", 2021-03-15T22:24:18.787 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 235.583340959, 2021-03-15T22:24:18.787 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018220706, 2021-03-15T22:24:18.787 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.787 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:24:18.788 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.788 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.788 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.788 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 367 2021-03-15T22:24:18.789 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.789 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.789 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.789 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.790 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105520+0000", 2021-03-15T22:24:18.790 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.790 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.790 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.791 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.791 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105520+0000", 2021-03-15T22:24:18.791 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.68e-06 2021-03-15T22:24:18.791 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.792 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.792 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.792 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105522+0000", 2021-03-15T22:24:18.792 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0650000000000001e-06 2021-03-15T22:24:18.792 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.793 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.793 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.793 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105524+0000", 2021-03-15T22:24:18.793 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9020000000000005e-06 2021-03-15T22:24:18.793 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.794 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.794 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.794 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105533+0000", 2021-03-15T22:24:18.794 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1390000000000003e-06 2021-03-15T22:24:18.795 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.795 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.795 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.795 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105538+0000", 2021-03-15T22:24:18.795 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034772699999999999 2021-03-15T22:24:18.796 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.796 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.796 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.796 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105886+0000", 2021-03-15T22:24:18.796 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1764000000000001e-05 2021-03-15T22:24:18.797 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.797 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.797 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:18.797 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.105938+0000", 2021-03-15T22:24:18.797 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011904672999999999 2021-03-15T22:24:18.798 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.798 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.798 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.798 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.117842+0000", 2021-03-15T22:24:18.799 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4134999999999994e-05 2021-03-15T22:24:18.799 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.799 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.799 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:18.799 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.117917+0000", 2021-03-15T22:24:18.800 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056846789999999998 2021-03-15T22:24:18.800 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.800 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.800 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.801 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123601+0000", 2021-03-15T22:24:18.801 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1218999999999999e-05 2021-03-15T22:24:18.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.801 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.802 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123652+0000", 2021-03-15T22:24:18.802 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8722999999999997e-05 2021-03-15T22:24:18.802 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.802 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.802 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.803 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.123741+0000", 2021-03-15T22:24:18.803 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018220706 2021-03-15T22:24:18.803 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.803 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.803 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.804 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.804 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.804 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:372 7.0 7:054fa7db:::smithi07694878-29:head [copy-from ver 3] snapc a=[a] ondisk+write+known_if_redirected e108)", 2021-03-15T22:24:18.804 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.121689+0000", 2021-03-15T22:24:18.805 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 235.567172134, 2021-03-15T22:24:18.805 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018844218999999999, 2021-03-15T22:24:18.805 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.805 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:18.805 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.806 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.806 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.806 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 372 2021-03-15T22:24:18.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.806 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.807 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.807 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.807 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121689+0000", 2021-03-15T22:24:18.807 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.808 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.808 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.808 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.808 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121689+0000", 2021-03-15T22:24:18.808 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5499999999999996e-06 2021-03-15T22:24:18.809 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.809 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.809 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.809 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121694+0000", 2021-03-15T22:24:18.809 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0380000000000001e-06 2021-03-15T22:24:18.810 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.810 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.810 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121698+0000", 2021-03-15T22:24:18.810 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9465000000000001e-05 2021-03-15T22:24:18.811 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.811 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.811 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.811 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121717+0000", 2021-03-15T22:24:18.812 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7600000000000001e-05 2021-03-15T22:24:18.812 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.812 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.812 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121735+0000", 2021-03-15T22:24:18.813 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1116000000000003e-05 2021-03-15T22:24:18.813 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.813 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.813 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.813 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121786+0000", 2021-03-15T22:24:18.814 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9568999999999996e-05 2021-03-15T22:24:18.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.814 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.815 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121876+0000", 2021-03-15T22:24:18.815 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085690019999999992 2021-03-15T22:24:18.815 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.815 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:24:18.816 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.130445+0000", 2021-03-15T22:24:18.816 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0025600380000000002 2021-03-15T22:24:18.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.816 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:24:18.817 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.133005+0000", 2021-03-15T22:24:18.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041963060000000003 2021-03-15T22:24:18.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.817 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.137201+0000", 2021-03-15T22:24:18.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003222258 2021-03-15T22:24:18.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.818 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.819 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140423+0000", 2021-03-15T22:24:18.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6202999999999997e-05 2021-03-15T22:24:18.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.820 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:18.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140479+0000", 2021-03-15T22:24:18.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4073999999999998e-05 2021-03-15T22:24:18.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.821 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.821 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140534+0000", 2021-03-15T22:24:18.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018844218999999999 2021-03-15T22:24:18.822 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.822 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.822 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.822 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.823 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:373 7.0 7:054fa7db:::smithi07694878-29:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e108)", 2021-03-15T22:24:18.823 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:23.121751+0000", 2021-03-15T22:24:18.823 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 235.56711052599999, 2021-03-15T22:24:18.823 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018944425000000001, 2021-03-15T22:24:18.824 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.824 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:24:18.824 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.824 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.824 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.825 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 373 2021-03-15T22:24:18.825 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.825 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.826 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.826 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121751+0000", 2021-03-15T22:24:18.826 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.826 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.827 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.827 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121751+0000", 2021-03-15T22:24:18.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.443e-06 2021-03-15T22:24:18.827 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.828 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.828 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.828 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121752+0000", 2021-03-15T22:24:18.828 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.057e-06 2021-03-15T22:24:18.828 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.829 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.829 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.829 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121754+0000", 2021-03-15T22:24:18.829 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2910000000000007e-06 2021-03-15T22:24:18.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.830 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.830 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.830 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121763+0000", 2021-03-15T22:24:18.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9339999999999997e-06 2021-03-15T22:24:18.831 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.831 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.831 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.831 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.121768+0000", 2021-03-15T22:24:18.832 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024470200000000002 2021-03-15T22:24:18.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.833 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122012+0000", 2021-03-15T22:24:18.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3108000000000001e-05 2021-03-15T22:24:18.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:18.834 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.122055+0000", 2021-03-15T22:24:18.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010285216 2021-03-15T22:24:18.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.834 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.835 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.835 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.132341+0000", 2021-03-15T22:24:18.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4697999999999994e-05 2021-03-15T22:24:18.835 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.836 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:18.836 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.132425+0000", 2021-03-15T22:24:18.836 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081476789999999997 2021-03-15T22:24:18.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.837 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140573+0000", 2021-03-15T22:24:18.837 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1277999999999999e-05 2021-03-15T22:24:18.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.838 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140624+0000", 2021-03-15T22:24:18.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1019000000000006e-05 2021-03-15T22:24:18.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:23.140695+0000", 2021-03-15T22:24:18.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018944425000000001 2021-03-15T22:24:18.840 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.840 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.840 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.841 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:1191 7.6 e176/175 7:61e0cc05:::smithi07694878-16:head v 176'225, mlcod=176'221)", 2021-03-15T22:24:18.841 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:31.032483+0000", 2021-03-15T22:24:18.841 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 167.65637816500001, 2021-03-15T22:24:18.842 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013234634, 2021-03-15T22:24:18.842 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.842 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:18.842 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.842 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.843 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.843 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.032483+0000", 2021-03-15T22:24:18.843 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.844 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.844 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.032483+0000", 2021-03-15T22:24:18.844 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054095030000000004 2021-03-15T22:24:18.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.845 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.845 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.845 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.037893+0000", 2021-03-15T22:24:18.845 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.989e-06 2021-03-15T22:24:18.845 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.846 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.846 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.037898+0000", 2021-03-15T22:24:18.846 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2257999999999999e-05 2021-03-15T22:24:18.846 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.037920+0000", 2021-03-15T22:24:18.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2199999999999992e-06 2021-03-15T22:24:18.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.848 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.037928+0000", 2021-03-15T22:24:18.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8709999999999998e-05 2021-03-15T22:24:18.849 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.849 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.849 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.037997+0000", 2021-03-15T22:24:18.849 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2061000000000003e-05 2021-03-15T22:24:18.850 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.850 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.850 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.850 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.038039+0000", 2021-03-15T22:24:18.850 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076352579999999998 2021-03-15T22:24:18.851 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.851 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.851 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:18.851 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.045674+0000", 2021-03-15T22:24:18.852 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3634999999999998e-05 2021-03-15T22:24:18.852 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.852 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.852 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.852 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:31.045718+0000", 2021-03-15T22:24:18.853 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013234634 2021-03-15T22:24:18.853 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.853 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.853 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.853 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.854 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.854 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1589 7.0 7:00c03349:::smithi07694878-40:head [write 766452~565707,stat] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:24:18.854 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.044128+0000", 2021-03-15T22:24:18.854 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 137.64473298499999, 2021-03-15T22:24:18.855 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.002067477, 2021-03-15T22:24:18.855 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.855 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:18.855 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.855 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.856 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.856 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1589 2021-03-15T22:24:18.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.856 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.857 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.857 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044128+0000", 2021-03-15T22:24:18.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.857 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.858 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.858 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.858 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044128+0000", 2021-03-15T22:24:18.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00075323800000000004 2021-03-15T22:24:18.858 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.859 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.859 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.859 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044882+0000", 2021-03-15T22:24:18.859 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3730000000000001e-06 2021-03-15T22:24:18.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.860 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.860 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.860 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044885+0000", 2021-03-15T22:24:18.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3543999999999999e-05 2021-03-15T22:24:18.861 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.861 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044899+0000", 2021-03-15T22:24:18.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2700000000000001e-06 2021-03-15T22:24:18.862 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.862 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044905+0000", 2021-03-15T22:24:18.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1384000000000001e-05 2021-03-15T22:24:18.863 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.863 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.863 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.864 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044956+0000", 2021-03-15T22:24:18.864 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1709999999999996e-06 2021-03-15T22:24:18.864 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.864 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.864 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:24:18.865 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044963+0000", 2021-03-15T22:24:18.865 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99623219900000004 2021-03-15T22:24:18.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.865 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.866 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.866 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.041196+0000", 2021-03-15T22:24:18.866 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000118097 2021-03-15T22:24:18.866 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.867 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.867 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.041314+0000", 2021-03-15T22:24:18.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018693400000000001 2021-03-15T22:24:18.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.867 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.868 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:24:18.868 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.041501+0000", 2021-03-15T22:24:18.868 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029280589999999998 2021-03-15T22:24:18.868 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.869 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.869 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.869 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.044429+0000", 2021-03-15T22:24:18.869 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000135282 2021-03-15T22:24:18.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.870 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.870 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.870 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.044564+0000", 2021-03-15T22:24:18.870 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001471113 2021-03-15T22:24:18.870 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.871 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:24:18.871 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046035+0000", 2021-03-15T22:24:18.871 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9391999999999997e-05 2021-03-15T22:24:18.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.872 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.872 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:18.872 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046085+0000", 2021-03-15T22:24:18.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111421 2021-03-15T22:24:18.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.873 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.873 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046196+0000", 2021-03-15T22:24:18.873 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.002067477 2021-03-15T22:24:18.874 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.874 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.874 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.874 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.875 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.875 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1590 7.0 7:00c03349:::smithi07694878-40:head [write 1860221~469412] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:24:18.875 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.049091+0000", 2021-03-15T22:24:18.875 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 137.63977050899999, 2021-03-15T22:24:18.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0004167690000001, 2021-03-15T22:24:18.876 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.876 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:18.876 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.876 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.876 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.877 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1590 2021-03-15T22:24:18.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.877 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.878 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.878 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049091+0000", 2021-03-15T22:24:18.878 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.878 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.878 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.879 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.879 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049091+0000", 2021-03-15T22:24:18.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036754099999999998 2021-03-15T22:24:18.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.880 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.880 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049458+0000", 2021-03-15T22:24:18.880 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2369999999999999e-06 2021-03-15T22:24:18.880 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.881 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049461+0000", 2021-03-15T22:24:18.881 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3468e-05 2021-03-15T22:24:18.881 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.882 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049474+0000", 2021-03-15T22:24:18.882 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7660000000000001e-06 2021-03-15T22:24:18.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.883 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.883 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.883 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049481+0000", 2021-03-15T22:24:18.883 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0846000000000006e-05 2021-03-15T22:24:18.884 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.884 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.884 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.884 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049552+0000", 2021-03-15T22:24:18.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4910000000000005e-06 2021-03-15T22:24:18.885 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.885 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.885 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:24:18.885 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049560+0000", 2021-03-15T22:24:18.886 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99323641500000004 2021-03-15T22:24:18.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.886 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.886 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.886 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.042797+0000", 2021-03-15T22:24:18.887 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4145000000000002e-05 2021-03-15T22:24:18.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.887 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.887 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:18.887 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.042881+0000", 2021-03-15T22:24:18.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033443700000000002 2021-03-15T22:24:18.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.888 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.888 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046225+0000", 2021-03-15T22:24:18.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1945000000000004e-05 2021-03-15T22:24:18.889 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.889 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.889 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.890 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046287+0000", 2021-03-15T22:24:18.890 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013465800000000001 2021-03-15T22:24:18.890 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.890 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.890 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:24:18.891 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046422+0000", 2021-03-15T22:24:18.891 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0025218760000000002 2021-03-15T22:24:18.891 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.891 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.891 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:24:18.892 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048944+0000", 2021-03-15T22:24:18.892 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036462099999999998 2021-03-15T22:24:18.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.893 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.893 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049308+0000", 2021-03-15T22:24:18.893 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013490400000000001 2021-03-15T22:24:18.893 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.893 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.894 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.894 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049443+0000", 2021-03-15T22:24:18.894 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4076999999999999e-05 2021-03-15T22:24:18.894 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.894 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.895 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:18.895 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049477+0000", 2021-03-15T22:24:18.895 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0409e-05 2021-03-15T22:24:18.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.895 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049508+0000", 2021-03-15T22:24:18.896 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0004167690000001 2021-03-15T22:24:18.896 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.897 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.897 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.897 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.897 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1591 7.0 7:00c03349:::smithi07694878-40:head [write 2981238~594195] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:24:18.898 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.056067+0000", 2021-03-15T22:24:18.898 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 137.63279480099999, 2021-03-15T22:24:18.898 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99404281400000005, 2021-03-15T22:24:18.898 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.899 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:18.899 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.899 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.899 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.899 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1591 2021-03-15T22:24:18.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.900 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.900 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.900 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.901 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056067+0000", 2021-03-15T22:24:18.901 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.901 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.901 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.901 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.902 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056067+0000", 2021-03-15T22:24:18.902 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00055381400000000002 2021-03-15T22:24:18.902 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.902 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.902 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.903 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056620+0000", 2021-03-15T22:24:18.903 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5569999999999998e-06 2021-03-15T22:24:18.903 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.903 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.904 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.904 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056623+0000", 2021-03-15T22:24:18.904 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3344e-05 2021-03-15T22:24:18.904 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.904 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.905 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.905 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056636+0000", 2021-03-15T22:24:18.905 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2089999999999996e-06 2021-03-15T22:24:18.905 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.905 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.906 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.906 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056642+0000", 2021-03-15T22:24:18.906 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4657000000000003e-05 2021-03-15T22:24:18.906 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.907 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.907 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.907 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056686+0000", 2021-03-15T22:24:18.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0719999999999996e-06 2021-03-15T22:24:18.907 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.908 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.908 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:24:18.908 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056692+0000", 2021-03-15T22:24:18.908 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98623406700000005 2021-03-15T22:24:18.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.042926+0000", 2021-03-15T22:24:18.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101845 2021-03-15T22:24:18.909 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.910 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.910 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:18.910 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043028+0000", 2021-03-15T22:24:18.910 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042733099999999998 2021-03-15T22:24:18.911 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.911 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.911 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.911 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.047302+0000", 2021-03-15T22:24:18.911 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8188000000000004e-05 2021-03-15T22:24:18.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.912 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.912 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.912 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.047380+0000", 2021-03-15T22:24:18.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013972 2021-03-15T22:24:18.913 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.913 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:24:18.913 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.047519+0000", 2021-03-15T22:24:18.914 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020016600000000002 2021-03-15T22:24:18.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.914 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.914 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:24:18.914 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049521+0000", 2021-03-15T22:24:18.915 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034165299999999998 2021-03-15T22:24:18.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.915 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049863+0000", 2021-03-15T22:24:18.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015579400000000001 2021-03-15T22:24:18.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.916 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.917 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050019+0000", 2021-03-15T22:24:18.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1232e-05 2021-03-15T22:24:18.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.917 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:18.918 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050070+0000", 2021-03-15T22:24:18.918 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9691999999999999e-05 2021-03-15T22:24:18.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.918 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.919 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050109+0000", 2021-03-15T22:24:18.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99404281400000005 2021-03-15T22:24:18.919 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.919 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.920 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.920 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1592 7.0 7:00c03349:::smithi07694878-40:head [setxattr _header (57) in=7b,truncate 3966452] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:24:18.920 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.056653+0000", 2021-03-15T22:24:18.921 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 137.632208117, 2021-03-15T22:24:18.921 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99360381200000003, 2021-03-15T22:24:18.921 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.921 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:18.921 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.922 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.922 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.922 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1592 2021-03-15T22:24:18.922 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.923 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.923 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.923 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056653+0000", 2021-03-15T22:24:18.923 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.924 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.924 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056653+0000", 2021-03-15T22:24:18.925 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1789999999999998e-06 2021-03-15T22:24:18.925 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.925 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.925 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.925 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056655+0000", 2021-03-15T22:24:18.926 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.689e-06 2021-03-15T22:24:18.926 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.926 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.926 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.926 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056657+0000", 2021-03-15T22:24:18.927 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5310000000000001e-06 2021-03-15T22:24:18.927 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.927 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056663+0000", 2021-03-15T22:24:18.928 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3579999999999996e-06 2021-03-15T22:24:18.928 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.928 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.928 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056667+0000", 2021-03-15T22:24:18.929 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8161000000000002e-05 2021-03-15T22:24:18.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.929 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.930 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056715+0000", 2021-03-15T22:24:18.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2899999999999999e-06 2021-03-15T22:24:18.930 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.930 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.930 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:24:18.931 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056722+0000", 2021-03-15T22:24:18.931 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98635435299999996 2021-03-15T22:24:18.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.931 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.932 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043076+0000", 2021-03-15T22:24:18.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100571 2021-03-15T22:24:18.932 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.932 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.933 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:18.933 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043176+0000", 2021-03-15T22:24:18.933 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0051726330000000003 2021-03-15T22:24:18.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.933 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.934 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.934 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048349+0000", 2021-03-15T22:24:18.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010839 2021-03-15T22:24:18.934 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048457+0000", 2021-03-15T22:24:18.935 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0001582 2021-03-15T22:24:18.935 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.936 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.936 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:24:18.936 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048616+0000", 2021-03-15T22:24:18.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00090888500000000001 2021-03-15T22:24:18.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.937 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.937 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:24:18.937 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049525+0000", 2021-03-15T22:24:18.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00045732299999999999 2021-03-15T22:24:18.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.938 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.938 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.938 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049982+0000", 2021-03-15T22:24:18.938 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016349299999999999 2021-03-15T22:24:18.939 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.939 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.939 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050145+0000", 2021-03-15T22:24:18.939 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9777e-05 2021-03-15T22:24:18.940 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.940 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:18.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050175+0000", 2021-03-15T22:24:18.941 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1978999999999996e-05 2021-03-15T22:24:18.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.941 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.941 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050257+0000", 2021-03-15T22:24:18.942 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99360381200000003 2021-03-15T22:24:18.942 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.942 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.942 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.943 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1593 7.0 7:00c03349:::smithi07694878-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e206)", 2021-03-15T22:24:18.943 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.056680+0000", 2021-03-15T22:24:18.943 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 137.63218175899999, 2021-03-15T22:24:18.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99375757600000003, 2021-03-15T22:24:18.944 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.944 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:24:18.944 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.944 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.945 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.945 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1593 2021-03-15T22:24:18.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.945 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.946 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056680+0000", 2021-03-15T22:24:18.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056680+0000", 2021-03-15T22:24:18.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9000000000000005e-07 2021-03-15T22:24:18.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.948 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.948 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056681+0000", 2021-03-15T22:24:18.948 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.243e-06 2021-03-15T22:24:18.949 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.949 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.949 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.949 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056682+0000", 2021-03-15T22:24:18.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5610000000000004e-06 2021-03-15T22:24:18.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.950 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.950 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056688+0000", 2021-03-15T22:24:18.950 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4100000000000001e-06 2021-03-15T22:24:18.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.951 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.951 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.951 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056693+0000", 2021-03-15T22:24:18.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.562e-05 2021-03-15T22:24:18.952 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.952 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.952 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056738+0000", 2021-03-15T22:24:18.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6179999999999999e-06 2021-03-15T22:24:18.953 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.953 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:24:18.954 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056742+0000", 2021-03-15T22:24:18.954 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98647171499999997 2021-03-15T22:24:18.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.954 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.955 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043214+0000", 2021-03-15T22:24:18.955 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9062999999999998e-05 2021-03-15T22:24:18.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.955 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.955 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:18.956 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043303+0000", 2021-03-15T22:24:18.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055719380000000002 2021-03-15T22:24:18.956 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.956 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.956 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.957 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048875+0000", 2021-03-15T22:24:18.957 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6421000000000001e-05 2021-03-15T22:24:18.957 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.957 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.958 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:18.958 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048931+0000", 2021-03-15T22:24:18.958 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013764700000000001 2021-03-15T22:24:18.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.959 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050308+0000", 2021-03-15T22:24:18.959 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6387000000000001e-05 2021-03-15T22:24:18.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050364+0000", 2021-03-15T22:24:18.960 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3139999999999994e-05 2021-03-15T22:24:18.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.961 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050437+0000", 2021-03-15T22:24:18.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99375757600000003 2021-03-15T22:24:18.961 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.962 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.962 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.962 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.963 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:2216 7.0 7:00c03349:::smithi07694878-40:head [rollback 9c] snapc a4=[a2,a0,9e,9c,93,8d,88,7d] ondisk+write+known_if_redirected e263)", 2021-03-15T22:24:18.963 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:58.051090+0000", 2021-03-15T22:24:18.963 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 80.637771478000005, 2021-03-15T22:24:18.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017630251999999999, 2021-03-15T22:24:18.963 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.964 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:18.964 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.964 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.964 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.964 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2216 2021-03-15T22:24:18.965 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.965 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.966 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051090+0000", 2021-03-15T22:24:18.966 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.967 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051090+0000", 2021-03-15T22:24:18.967 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.358e-06 2021-03-15T22:24:18.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.967 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.968 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.968 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051091+0000", 2021-03-15T22:24:18.968 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.615e-06 2021-03-15T22:24:18.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.968 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.969 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.969 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051093+0000", 2021-03-15T22:24:18.969 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9290000000000001e-06 2021-03-15T22:24:18.969 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.970 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.970 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.970 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051100+0000", 2021-03-15T22:24:18.970 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5589999999999999e-06 2021-03-15T22:24:18.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.971 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.971 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051104+0000", 2021-03-15T22:24:18.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7732999999999995e-05 2021-03-15T22:24:18.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.972 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051192+0000", 2021-03-15T22:24:18.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3170000000000008e-06 2021-03-15T22:24:18.973 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.973 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:24:18.973 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051201+0000", 2021-03-15T22:24:18.974 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003037203 2021-03-15T22:24:18.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.974 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.975 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.054239+0000", 2021-03-15T22:24:18.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1857e-05 2021-03-15T22:24:18.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.975 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.975 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.976 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.054311+0000", 2021-03-15T22:24:18.976 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0030452280000000001 2021-03-15T22:24:18.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.976 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.977 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-15T22:24:18.977 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.057356+0000", 2021-03-15T22:24:18.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0096410349999999992 2021-03-15T22:24:18.977 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.978 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.978 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.066997+0000", 2021-03-15T22:24:18.978 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5237999999999999e-05 2021-03-15T22:24:18.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:18.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.067092+0000", 2021-03-15T22:24:18.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020652999999999999 2021-03-15T22:24:18.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:24:18.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.067299+0000", 2021-03-15T22:24:18.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00094855099999999995 2021-03-15T22:24:18.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:24:18.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.068247+0000", 2021-03-15T22:24:18.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0309999999999998e-05 2021-03-15T22:24:18.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.983 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.068297+0000", 2021-03-15T22:24:18.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00026308599999999999 2021-03-15T22:24:18.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:18.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.068561+0000", 2021-03-15T22:24:18.984 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2469999999999998e-05 2021-03-15T22:24:18.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.985 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:18.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.068603+0000", 2021-03-15T22:24:18.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011723299999999999 2021-03-15T22:24:18.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.986 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:18.986 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.068720+0000", 2021-03-15T22:24:18.986 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017630251999999999 2021-03-15T22:24:18.986 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.987 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:18.987 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:18.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.987 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:2217 7.0 7:00c03349:::smithi07694878-40:head [append 0~0] snapc a4=[a2,a0,9e,9c,93,8d,88,7d] ondisk+write+known_if_redirected e263)", 2021-03-15T22:24:18.988 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:58.051110+0000", 2021-03-15T22:24:18.988 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 80.637751132000005, 2021-03-15T22:24:18.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017835157000000001, 2021-03-15T22:24:18.988 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:18.989 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:18.989 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:18.989 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:18.989 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:18.989 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2217 2021-03-15T22:24:18.990 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.990 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:18.990 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.990 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:18.991 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051110+0000", 2021-03-15T22:24:18.991 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:18.991 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.992 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:18.992 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051110+0000", 2021-03-15T22:24:18.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1519999999999999e-06 2021-03-15T22:24:18.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.992 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.993 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:18.993 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051111+0000", 2021-03-15T22:24:18.993 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3629999999999999e-06 2021-03-15T22:24:18.993 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.994 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:18.994 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051113+0000", 2021-03-15T22:24:18.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4359999999999999e-06 2021-03-15T22:24:18.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.995 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:18.995 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051119+0000", 2021-03-15T22:24:18.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1230000000000002e-06 2021-03-15T22:24:18.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:18.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051123+0000", 2021-03-15T22:24:18.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010102699999999999 2021-03-15T22:24:18.997 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.997 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.997 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051224+0000", 2021-03-15T22:24:18.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3990000000000004e-06 2021-03-15T22:24:18.998 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.998 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.998 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:24:18.998 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.051233+0000", 2021-03-15T22:24:18.999 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061811720000000004 2021-03-15T22:24:18.999 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:18.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:18.999 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:18.999 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.057414+0000", 2021-03-15T22:24:19.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3874000000000005e-05 2021-03-15T22:24:19.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-15T22:24:19.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.057488+0000", 2021-03-15T22:24:19.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010290888 2021-03-15T22:24:19.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.001 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:19.002 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.067779+0000", 2021-03-15T22:24:19.002 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7762000000000005e-05 2021-03-15T22:24:19.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:19.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.067856+0000", 2021-03-15T22:24:19.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015011700000000001 2021-03-15T22:24:19.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:24:19.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.068007+0000", 2021-03-15T22:24:19.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00033622399999999998 2021-03-15T22:24:19.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:24:19.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.068343+0000", 2021-03-15T22:24:19.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00042389400000000002 2021-03-15T22:24:19.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.006 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:19.006 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.068767+0000", 2021-03-15T22:24:19.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9388999999999998e-05 2021-03-15T22:24:19.007 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.007 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.007 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:19.008 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.068806+0000", 2021-03-15T22:24:19.008 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7673000000000003e-05 2021-03-15T22:24:19.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.008 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.009 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:19.009 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.068844+0000", 2021-03-15T22:24:19.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010166399999999999 2021-03-15T22:24:19.009 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.009 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.010 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:19.010 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:58.068945+0000", 2021-03-15T22:24:19.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017835157000000001 2021-03-15T22:24:19.010 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:19.011 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:19.011 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:19.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.011 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.012 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:2291 7.0 7:0732c451:::smithi07694878-32:head [read 0~2453855,checksum] snapc 0=[] ondisk+read+known_if_redirected e264)", 2021-03-15T22:24:19.012 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:59.685774+0000", 2021-03-15T22:24:19.012 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 79.003086945000007, 2021-03-15T22:24:19.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011850273999999999, 2021-03-15T22:24:19.012 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:19.013 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:24:19.013 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:19.013 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:19.013 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:19.014 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2291 2021-03-15T22:24:19.014 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.014 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:19.014 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:19.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:59.685774+0000", 2021-03-15T22:24:19.015 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:19.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:19.016 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:59.685774+0000", 2021-03-15T22:24:19.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4559999999999999e-06 2021-03-15T22:24:19.016 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:19.017 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:59.685776+0000", 2021-03-15T22:24:19.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.78e-07 2021-03-15T22:24:19.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.018 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:19.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:59.685777+0000", 2021-03-15T22:24:19.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0099999999999997e-06 2021-03-15T22:24:19.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:19.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:59.685781+0000", 2021-03-15T22:24:19.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2910000000000002e-06 2021-03-15T22:24:19.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:19.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:59.685783+0000", 2021-03-15T22:24:19.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011393856000000001 2021-03-15T22:24:19.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.021 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:19.021 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:59.697177+0000", 2021-03-15T22:24:19.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010570500000000001 2021-03-15T22:24:19.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:19.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:59.697283+0000", 2021-03-15T22:24:19.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000342078 2021-03-15T22:24:19.023 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:19.024 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:59.697625+0000", 2021-03-15T22:24:19.024 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011850273999999999 2021-03-15T22:24:19.024 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:19.024 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:19.024 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:19.025 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.025 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3000 7.0 7:071f7a04:::smithi07694878-37:head [copy-from ver 389] snapc e1=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e326)", 2021-03-15T22:24:19.025 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:01.213659+0000", 2021-03-15T22:24:19.026 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 17.475202009, 2021-03-15T22:24:19.026 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013925348000000001, 2021-03-15T22:24:19.026 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:19.026 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:19.026 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:19.027 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:19.027 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:19.027 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3000 2021-03-15T22:24:19.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.028 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:19.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.028 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:19.028 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213659+0000", 2021-03-15T22:24:19.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:19.029 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.029 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.029 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:19.029 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213659+0000", 2021-03-15T22:24:19.030 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9e-06 2021-03-15T22:24:19.030 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.030 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.030 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:19.031 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213666+0000", 2021-03-15T22:24:19.031 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9229999999999999e-06 2021-03-15T22:24:19.031 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.031 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.031 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:19.032 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213672+0000", 2021-03-15T22:24:19.032 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1191e-05 2021-03-15T22:24:19.032 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.033 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:19.033 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213703+0000", 2021-03-15T22:24:19.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1500000000000004e-06 2021-03-15T22:24:19.033 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.033 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.034 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:19.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213710+0000", 2021-03-15T22:24:19.034 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7355999999999996e-05 2021-03-15T22:24:19.034 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:19.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213787+0000", 2021-03-15T22:24:19.035 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4505999999999995e-05 2021-03-15T22:24:19.036 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:19.036 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213851+0000", 2021-03-15T22:24:19.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003598689 2021-03-15T22:24:19.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.037 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:24:19.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.217450+0000", 2021-03-15T22:24:19.038 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085505860000000006 2021-03-15T22:24:19.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.038 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:24:19.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.226001+0000", 2021-03-15T22:24:19.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001424373 2021-03-15T22:24:19.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:19.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.227425+0000", 2021-03-15T22:24:19.040 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8076000000000003e-05 2021-03-15T22:24:19.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.040 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:19.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.227493+0000", 2021-03-15T22:24:19.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4657000000000003e-05 2021-03-15T22:24:19.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:24:19.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.227538+0000", 2021-03-15T22:24:19.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6940999999999997e-05 2021-03-15T22:24:19.042 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.042 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:19.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.227585+0000", 2021-03-15T22:24:19.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013925348000000001 2021-03-15T22:24:19.043 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:19.043 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:19.044 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:19.044 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.044 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.044 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3001 7.0 7:071f7a04:::smithi07694878-37:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e326)", 2021-03-15T22:24:19.044 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:01.213719+0000", 2021-03-15T22:24:19.045 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 17.475142658999999, 2021-03-15T22:24:19.045 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014017162999999999, 2021-03-15T22:24:19.045 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:19.045 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:24:19.046 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:24:19.046 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:24:19.046 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:19.046 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3001 2021-03-15T22:24:19.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.047 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:19.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:19.047 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213719+0000", 2021-03-15T22:24:19.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:19.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:19.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213719+0000", 2021-03-15T22:24:19.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1299999999999999e-07 2021-03-15T22:24:19.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.049 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:19.049 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213720+0000", 2021-03-15T22:24:19.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.077e-06 2021-03-15T22:24:19.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.050 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:19.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213721+0000", 2021-03-15T22:24:19.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1902000000000001e-05 2021-03-15T22:24:19.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.051 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:19.051 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213743+0000", 2021-03-15T22:24:19.052 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.903e-06 2021-03-15T22:24:19.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.052 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:19.053 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213745+0000", 2021-03-15T22:24:19.053 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000222103 2021-03-15T22:24:19.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.053 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:19.054 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.213968+0000", 2021-03-15T22:24:19.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9202000000000002e-05 2021-03-15T22:24:19.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:19.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.214007+0000", 2021-03-15T22:24:19.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0069294220000000002 2021-03-15T22:24:19.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.055 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:19.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.220936+0000", 2021-03-15T22:24:19.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000113044 2021-03-15T22:24:19.056 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.057 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:19.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.221049+0000", 2021-03-15T22:24:19.057 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0065667249999999998 2021-03-15T22:24:19.057 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.058 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:19.058 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.227616+0000", 2021-03-15T22:24:19.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6419000000000003e-05 2021-03-15T22:24:19.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:19.059 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.227672+0000", 2021-03-15T22:24:19.059 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3553000000000006e-05 2021-03-15T22:24:19.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:19.060 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:01.227736+0000", 2021-03-15T22:24:19.060 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014017162999999999 2021-03-15T22:24:19.060 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:19.061 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:19.061 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:19.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.061 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(osd.0.8:42 7.0 7:0732c451:::smithi07694878-32:head [assert-version v640,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e330)", 2021-03-15T22:24:19.062 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:05.087621+0000", 2021-03-15T22:24:19.062 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 13.601240564999999, 2021-03-15T22:24:19.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.10411197899999999, 2021-03-15T22:24:19.062 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:24:19.062 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:24:19.063 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:24:19.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.063 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:24:19.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087621+0000", 2021-03-15T22:24:19.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:24:19.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.064 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:24:19.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087621+0000", 2021-03-15T22:24:19.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9579999999999999e-06 2021-03-15T22:24:19.065 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:24:19.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087624+0000", 2021-03-15T22:24:19.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0649999999999999e-06 2021-03-15T22:24:19.066 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:24:19.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087627+0000", 2021-03-15T22:24:19.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.509e-05 2021-03-15T22:24:19.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.067 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.067 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:24:19.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087642+0000", 2021-03-15T22:24:19.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0669999999999992e-06 2021-03-15T22:24:19.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:24:19.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087650+0000", 2021-03-15T22:24:19.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4026000000000001e-05 2021-03-15T22:24:19.069 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:19.070 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087704+0000", 2021-03-15T22:24:19.070 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6101999999999999e-05 2021-03-15T22:24:19.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.071 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:24:19.071 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087720+0000", 2021-03-15T22:24:19.071 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.103629809 2021-03-15T22:24:19.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.071 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.072 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:24:19.072 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.191350+0000", 2021-03-15T22:24:19.072 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014534500000000001 2021-03-15T22:24:19.072 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.073 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:24:19.073 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.191495+0000", 2021-03-15T22:24:19.073 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000237517 2021-03-15T22:24:19.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:24:19.073 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:24:19.074 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:24:19.074 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.191733+0000", 2021-03-15T22:24:19.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.10411197899999999 2021-03-15T22:24:19.074 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:19.075 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:19.075 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:19.075 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:24:19.075 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:24:19.075 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:24:19.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:19.077 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:19.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:19.112 INFO:tasks.rados.rados.0.smithi076.stdout:1207: done (4 left) 2021-03-15T22:24:19.112 INFO:tasks.rados.rados.0.smithi076.stdout:1208: done (3 left) 2021-03-15T22:24:19.112 INFO:tasks.rados.rados.0.smithi076.stdout:1209: done (2 left) 2021-03-15T22:24:19.112 INFO:tasks.rados.rados.0.smithi076.stdout:1210: done (1 left) 2021-03-15T22:24:19.113 INFO:tasks.rados.rados.0.smithi076.stdout:1211: done (0 left) 2021-03-15T22:24:19.113 INFO:tasks.rados.rados.0.smithi076.stdout:1212: write oid 10 current snap is 125 2021-03-15T22:24:19.113 INFO:tasks.rados.rados.0.smithi076.stdout:1212: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2021-03-15T22:24:19.115 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:24:19.118 INFO:tasks.rados.rados.0.smithi076.stdout:1212: writing smithi07694878-10 from 662308 to 1221338 tid 1 2021-03-15T22:24:19.124 INFO:tasks.rados.rados.0.smithi076.stdout:1212: writing smithi07694878-10 from 1632005 to 2235816 tid 2 2021-03-15T22:24:19.132 INFO:tasks.rados.rados.0.smithi076.stdout:1212: writing smithi07694878-10 from 2860044 to 3462308 tid 3 2021-03-15T22:24:19.132 INFO:tasks.rados.rados.0.smithi076.stdout:1213: read oid 34 snap -1 2021-03-15T22:24:19.132 INFO:tasks.rados.rados.0.smithi076.stdout:1213: expect deleted 2021-03-15T22:24:19.132 INFO:tasks.rados.rados.0.smithi076.stdout:1214: copy_from oid 30 from oid 18 current snap is 125 2021-03-15T22:24:19.133 INFO:tasks.rados.rados.0.smithi076.stdout:1212: finishing write tid 1 to smithi07694878-10 2021-03-15T22:24:19.133 INFO:tasks.rados.rados.0.smithi076.stdout:1212: finishing write tid 2 to smithi07694878-10 2021-03-15T22:24:19.133 INFO:tasks.rados.rados.0.smithi076.stdout:1215: copy_from oid 2 from oid 17 current snap is 125 2021-03-15T22:24:19.133 INFO:tasks.rados.rados.0.smithi076.stdout:1216: copy_from oid 33 from oid 38 current snap is 125 2021-03-15T22:24:19.134 INFO:tasks.rados.rados.0.smithi076.stdout:1217: snap_create 2021-03-15T22:24:19.140 INFO:tasks.rados.rados.0.smithi076.stdout:1212: finishing write tid 3 to smithi07694878-10 2021-03-15T22:24:19.140 INFO:tasks.rados.rados.0.smithi076.stdout:1212: finishing write tid 4 to smithi07694878-10 2021-03-15T22:24:19.141 INFO:tasks.rados.rados.0.smithi076.stdout:1212: finishing write tid 5 to smithi07694878-10 2021-03-15T22:24:19.141 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 229 (ObjNum 444 snap 125 seq_num 444) dirty exists 2021-03-15T22:24:19.142 INFO:tasks.rados.rados.0.smithi076.stdout:1212: left oid 10 (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:24:19.146 INFO:tasks.rados.rados.0.smithi076.stdout:1216: finishing copy_from to smithi07694878-33 2021-03-15T22:24:19.147 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 433 (ObjNum 431 snap 119 seq_num 431) dirty exists 2021-03-15T22:24:19.147 INFO:tasks.rados.rados.0.smithi076.stdout:1216: finishing copy_from racing read to smithi07694878-33 2021-03-15T22:24:19.158 INFO:tasks.rados.rados.0.smithi076.stdout:1214: finishing copy_from to smithi07694878-30 2021-03-15T22:24:19.158 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 282 (ObjNum 443 snap 124 seq_num 443) dirty exists 2021-03-15T22:24:19.158 INFO:tasks.rados.rados.0.smithi076.stdout:1215: finishing copy_from to smithi07694878-2 2021-03-15T22:24:19.159 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 104 (ObjNum 422 snap 116 seq_num 422) dirty exists 2021-03-15T22:24:19.159 INFO:tasks.rados.rados.0.smithi076.stdout:1214: finishing copy_from racing read to smithi07694878-30 2021-03-15T22:24:19.159 INFO:tasks.rados.rados.0.smithi076.stdout:1215: finishing copy_from racing read to smithi07694878-2 2021-03-15T22:24:19.179 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:19.200 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:19.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:19.202 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:19.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:19.326 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:19.328 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:19.346 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-15T22:24:19.363 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[56514]: cluster 2021-03-15T22:24:18.105137 2021-03-15T22:24:19.363 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2021-03-15T22:24:19.363 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[56514]: audit 2021-03-15T22:24:18.384672+0000 mon.c (mon.1) 30 : audit [INF] from='client.? 172.21.15.76:0/516193411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:19.363 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[56514]: audit 2021-03-15T22:24:18.384989+0000 mon.a (mon.0) 1092 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:19.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: cluster 2021-03-15T22:24:19.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:24:19.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: :24:18.105137+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2021-03-15T22:24:19.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:24:19.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: 03-15T22:24:18. 2021-03-15T22:24:19.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: 384672+0000 mon.c (mon.1 2021-03-15T22:24:19.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: ) 30 : audit [INF] from='client.? 172.21.15.76:0/516193411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:19.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: 2021-03-15T22:24:19.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: audit 2021 2021-03-15T22:24:19.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: -03-15T22:24:18.384989 2021-03-15T22:24:19.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2021-03-15T22:24:19.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:19 smithi076 conmon[62156]: 1092 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:19.408 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: cluster 2021-03-15T22:24:19.408 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: 2021-03-15T22:24: 2021-03-15T22:24:19.408 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: 18.105137+0000 mon.a (mon.0) 1091 : cluster [DBG] osdmap e343: 8 total, 8 up, 8 in 2021-03-15T22:24:19.409 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: audit 2021 2021-03-15T22:24:19.409 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: -03-15T22:24: 2021-03-15T22:24:19.409 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: 18.384672+0000 mon.c (mon 2021-03-15T22:24:19.409 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: .1) 30 : audit [INF] 2021-03-15T22:24:19.410 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/516193411' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:19.410 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: audit 2021 2021-03-15T22:24:19.410 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: -03-15T22:24 2021-03-15T22:24:19.410 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: :18.384989+0000 mon.a (mon. 2021-03-15T22:24:19.411 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[56739]: 0) 1092 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:19.429 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:19.452 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:19.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:19.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:19.554 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:19.576 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:19.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:19.578 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:19.661 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:19.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:19.705 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:19.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:19.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:24:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:24:19.806 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:19.828 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:19.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:19.830 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:19.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:19.954 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:19.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:19.955 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:20.056 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:20.075 INFO:tasks.rados.rados.0.smithi076.stdout:1212: done (5 left) 2021-03-15T22:24:20.076 INFO:tasks.rados.rados.0.smithi076.stdout:1213: done (4 left) 2021-03-15T22:24:20.076 INFO:tasks.rados.rados.0.smithi076.stdout:1214: done (3 left) 2021-03-15T22:24:20.076 INFO:tasks.rados.rados.0.smithi076.stdout:1215: done (2 left) 2021-03-15T22:24:20.076 INFO:tasks.rados.rados.0.smithi076.stdout:1216: done (1 left) 2021-03-15T22:24:20.077 INFO:tasks.rados.rados.0.smithi076.stdout:1217: done (0 left) 2021-03-15T22:24:20.077 INFO:tasks.rados.rados.0.smithi076.stdout:1218: snap_remove snap 124 2021-03-15T22:24:20.079 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:20.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:20.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:20.157 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:20 smithi125 conmon[56739]: cluster 2021-03-15T22:24:19.036334+0000 mgr.y (mgr.14144) 565 : cluster [DBG] pgmap v642: 138 pgs: 138 active+clean; 139 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 8 op/s 2021-03-15T22:24:20.157 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:20 smithi125 conmon[56739]: cluster 2021-03-15T22:24:19.100415+0000 mon.a (mon.0) 1093 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:20.158 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:20 smithi125 conmon[56739]: audit 2021-03-15T22:24:19.108370+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:20.158 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:20 smithi125 conmon[56739]: cluster 2021-03-15T22:24:19.108438+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2021-03-15T22:24:20.158 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:20 smithi125 conmon[56739]: audit 2021-03-15T22:24:19.637353+0000 mgr.y (mgr.14144) 566 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:20.158 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:20 smithi125 conmon[56739]: cluster 2021-03-15T22:24:20.072523+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2021-03-15T22:24:20.181 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:20.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:20.205 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:20.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:20.306 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:20.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:20.329 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:20.330 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:20.431 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:20.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:20.455 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:20.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:20.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[62156]: cluster 2021-03-15T22:24:19.036334+0000 mgr.y (mgr.14144) 2021-03-15T22:24:20.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[62156]: 565 : cluster [DBG] pgmap v642: 138 pgs: 138 active+clean; 139 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 8 op/s 2021-03-15T22:24:20.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[62156]: cluster 2021-03-15T22:24:19.100415+0000 mon.a (mon.0) 1093 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:20.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[62156]: audit 2021-03-15T22:24:19.108370+0000 mon.a (mon.0) 1094 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:20.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[62156]: cluster 2021-03-15T22:24:19.108438+0000 mon.a (mon.0) 1095 : cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2021-03-15T22:24:20.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[62156]: audit 2021-03-15T22:24:19.637353+0000 mgr.y (mgr.14144) 566 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:20.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[62156]: cluster 2021-03-15T22:24:20.072523+0000 mon.a (mon.0) 1096 : cluster [DBG] osdmap e345: 8 total, 8 up, 8 in 2021-03-15T22:24:20.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: cluster 2021-03-15T22:24:20.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: 2021-03-15T22:24:20.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: 2021 2021-03-15T22:24:20.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: - 2021-03-15T22:24:20.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: 03 2021-03-15T22:24:20.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: - 2021-03-15T22:24:20.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: 15 2021-03-15T22:24:20.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: T 2021-03-15T22:24:20.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: 22 2021-03-15T22:24:20.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: :24:19.036334+0000 2021-03-15T22:24:20.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: mgr.y (mgr.14144) 565 : cluster [DBG] pgmap v642: 138 pgs: 138 active+clean; 139 MiB data, 534 MiB used, 715 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 8 op/s 2021-03-15T22:24:20.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: cluster 2021-03-15T22:24:20.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: 2021-03-15 2021-03-15T22:24:20.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: T22:24:19.100415+0000 mon.a (mon.0) 1093 2021-03-15T22:24:20.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:20.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: audit 2021-03-15T22:24:20.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: 2021-03-15T 2021-03-15T22:24:20.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: 22:24:19.108370+0000 mon.a 2021-03-15T22:24:20.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: (mon.0) 1094 : audit [INF] 2021-03-15T22:24:20.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:20.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: cluster 2021-03-15T22:24:20.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: 2021-03-15T22:24:19. 2021-03-15T22:24:20.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: 108438+0000 mon.a (mon.0) 1095 : 2021-03-15T22:24:20.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: cluster [DBG] osdmap e344: 8 total, 8 up, 8 in 2021-03-15T22:24:20.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:24:20.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: 03-15T22:24:19.637353+0000 2021-03-15T22:24:20.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: mgr.y (mgr.14144) 566 : audit 2021-03-15T22:24:20.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:20.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:24:20.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: -15T22:24:20.072523+0000 2021-03-15T22:24:20.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: mon.a (mon.0) 1096 : cluster [DBG] 2021-03-15T22:24:20.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:20 smithi076 conmon[56514]: osdmap e345: 8 total, 8 up, 8 in 2021-03-15T22:24:20.557 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:20.581 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:20.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:20.683 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:20.706 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:20.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:20.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:20.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:20.835 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:20.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:20.836 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:20.937 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:20.938 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:24:20.954 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:24:20.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:20.961 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:20.962 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:20.973 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:24:20.992 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:24:21.010 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:24:21.027 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:24:21.045 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:24:21.063 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:21.064 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:24:21.065 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:24:21.065 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:24:21.065 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:24:21.065 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:24:21.066 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:24:21.066 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:24:21.066 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:24:21.066 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:24:21.093 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:21.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:21.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:21.096 INFO:tasks.rados.rados.0.smithi076.stdout:1218: done (0 left) 2021-03-15T22:24:21.096 INFO:tasks.rados.rados.0.smithi076.stdout:1219: read oid 34 snap -1 2021-03-15T22:24:21.096 INFO:tasks.rados.rados.0.smithi076.stdout:1219: expect deleted 2021-03-15T22:24:21.097 INFO:tasks.rados.rados.0.smithi076.stdout:1220: rollback oid 28 current snap is 126 2021-03-15T22:24:21.097 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 125 2021-03-15T22:24:21.097 INFO:tasks.rados.rados.0.smithi076.stdout:1221: read oid 23 snap -1 2021-03-15T22:24:21.097 INFO:tasks.rados.rados.0.smithi076.stdout:1221: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:24:21.098 INFO:tasks.rados.rados.0.smithi076.stdout:1222: write oid 30 current snap is 126 2021-03-15T22:24:21.098 INFO:tasks.rados.rados.0.smithi076.stdout:1222: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2021-03-15T22:24:21.102 INFO:tasks.rados.rados.0.smithi076.stdout:1222: writing smithi07694878-30 from 679115 to 1313394 tid 1 2021-03-15T22:24:21.109 INFO:tasks.rados.rados.0.smithi076.stdout:1222: writing smithi07694878-30 from 2090487 to 2554309 tid 2 2021-03-15T22:24:21.115 INFO:tasks.rados.rados.0.smithi076.stdout:1222: writing smithi07694878-30 from 3003820 to 3479115 tid 3 2021-03-15T22:24:21.115 INFO:tasks.rados.rados.0.smithi076.stdout:1223: snap_create 2021-03-15T22:24:21.115 INFO:tasks.rados.rados.0.smithi076.stdout:1220: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:24:21.115 INFO:tasks.rados.rados.0.smithi076.stdout:1220: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:24:21.116 INFO:tasks.rados.rados.0.smithi076.stdout:1220: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:24:21.116 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 719 (ObjNum 421 snap 116 seq_num 421) dirty exists 2021-03-15T22:24:21.116 INFO:tasks.rados.rados.0.smithi076.stdout:1221: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:24:21.117 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:21 smithi076 conmon[56514]: audit 2021-03-15T22:24:20.918968+0000 mon.a (mon.0) 1097 : audit [DBG] from='client.? 172.21.15.76:0/801251130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:21.117 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:21 smithi076 conmon[56514]: cluster 2021-03-15T22:24:21.081107+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2021-03-15T22:24:21.118 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:21 smithi076 conmon[62156]: audit 2021-03-15T22:24:20.918968+0000 mon.a (mon.0) 1097 : audit [DBG] from='client.? 172.21.15.76:0/801251130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:21.118 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:21 smithi076 conmon[62156]: cluster 2021-03-15T22:24:21.081107+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2021-03-15T22:24:21.185 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:24:21.186 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:24:21.196 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:21.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:21.220 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:21.221 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:21.262 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:21 smithi125 conmon[56739]: audit 2021-03-15T22:24:20.918968+0000 mon.a (mon.0) 1097 : audit [DBG] from='client.? 172.21.15.76:0/801251130' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:21.262 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:21 smithi125 conmon[56739]: cluster 2021-03-15T22:24:21.081107+0000 mon.a (mon.0) 1098 : cluster [DBG] osdmap e346: 8 total, 8 up, 8 in 2021-03-15T22:24:21.322 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:21.325 INFO:tasks.rados.rados.0.smithi076.stdout:1222: finishing write tid 1 to smithi07694878-30 2021-03-15T22:24:21.326 INFO:tasks.rados.rados.0.smithi076.stdout:1222: finishing write tid 2 to smithi07694878-30 2021-03-15T22:24:21.326 INFO:tasks.rados.rados.0.smithi076.stdout:1222: finishing write tid 3 to smithi07694878-30 2021-03-15T22:24:21.326 INFO:tasks.rados.rados.0.smithi076.stdout:1222: finishing write tid 4 to smithi07694878-30 2021-03-15T22:24:21.326 INFO:tasks.rados.rados.0.smithi076.stdout:1222: finishing write tid 5 to smithi07694878-30 2021-03-15T22:24:21.327 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 684 (ObjNum 445 snap 126 seq_num 445) dirty exists 2021-03-15T22:24:21.327 INFO:tasks.rados.rados.0.smithi076.stdout:1222: left oid 30 (ObjNum 445 snap 126 seq_num 445) 2021-03-15T22:24:21.344 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:21.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:21.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:21.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:21.470 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:21.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:21.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:21.573 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:21.595 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:21.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:21.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:21.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:21.721 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:21.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:21.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:21.824 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:21.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:21.848 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:21.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:21.880 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-15T22:24:21.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:21.972 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:21.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:21.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:22.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:22.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:22.098 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:22.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:22.118 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:22 smithi125 conmon[56739]: cluster 2021-03-15T22:24:21. 2021-03-15T22:24:22.118 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:22 smithi125 conmon[56739]: 036844+0000 mgr.y (mgr.14144) 567 : cluster [DBG] pgmap v645: 138 pgs: 138 active+clean; 147 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 9.4 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2021-03-15T22:24:22.119 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:22 smithi125 conmon[56739]: audit 2021-03-15T22:24:21.259010+0000 mgr.y (mgr.14144) 568 : audit [DBG] from='client.15315 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:22.119 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:22 smithi125 conmon[56739]: audit 2021-03-15T22:24:21.259803+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]: dispatch 2021-03-15T22:24:22.134 INFO:tasks.rados.rados.0.smithi076.stdout:1219: done (4 left) 2021-03-15T22:24:22.134 INFO:tasks.rados.rados.0.smithi076.stdout:1220: done (3 left) 2021-03-15T22:24:22.134 INFO:tasks.rados.rados.0.smithi076.stdout:1221: done (2 left) 2021-03-15T22:24:22.135 INFO:tasks.rados.rados.0.smithi076.stdout:1222: done (1 left) 2021-03-15T22:24:22.135 INFO:tasks.rados.rados.0.smithi076.stdout:1223: done (0 left) 2021-03-15T22:24:22.135 INFO:tasks.rados.rados.0.smithi076.stdout:1224: write oid 14 current snap is 127 2021-03-15T22:24:22.135 INFO:tasks.rados.rados.0.smithi076.stdout:1224: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2021-03-15T22:24:22.136 INFO:teuthology.orchestra.run.smithi076.stdout:moved 7 / 414 (1.69082%) 2021-03-15T22:24:22.136 INFO:teuthology.orchestra.run.smithi076.stdout:avg 51.75 2021-03-15T22:24:22.136 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 13.809 -> 12.0286 (expected baseline 6.72913) 2021-03-15T22:24:22.136 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 19 -> 22 pgs (0.36715 -> 0.425121 * mean) 2021-03-15T22:24:22.137 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.4 with 64 -> 62 pgs (1.23671 -> 1.19807 * mean) 2021-03-15T22:24:22.137 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:24:22.137 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:24:22.137 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 0.05 2021-03-15T22:24:22.137 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:24:22.138 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0008 2021-03-15T22:24:22.138 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0078 2021-03-15T22:24:22.138 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 weight 0.3898 -> 0.4398 2021-03-15T22:24:22.138 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5 weight 0.7593 -> 0.8093 2021-03-15T22:24:22.141 INFO:tasks.rados.rados.0.smithi076.stdout:1224: writing smithi07694878-14 from 695922 to 1405450 tid 1 2021-03-15T22:24:22.148 INFO:tasks.rados.rados.0.smithi076.stdout:1224: writing smithi07694878-14 from 2032616 to 2640096 tid 2 2021-03-15T22:24:22.153 INFO:tasks.rados.rados.0.smithi076.stdout:1224: writing smithi07694878-14 from 3198537 to 3495922 tid 3 2021-03-15T22:24:22.153 INFO:tasks.rados.rados.0.smithi076.stdout:1225: copy_from oid 6 from oid 24 current snap is 127 2021-03-15T22:24:22.154 INFO:tasks.rados.rados.0.smithi076.stdout:1224: finishing write tid 1 to smithi07694878-14 2021-03-15T22:24:22.154 INFO:tasks.rados.rados.0.smithi076.stdout:1224: finishing write tid 2 to smithi07694878-14 2021-03-15T22:24:22.154 INFO:tasks.rados.rados.0.smithi076.stdout:1226: read oid 12 snap -1 2021-03-15T22:24:22.154 INFO:tasks.rados.rados.0.smithi076.stdout:1226: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:24:22.155 INFO:tasks.rados.rados.0.smithi076.stdout:1227: read oid 39 snap -1 2021-03-15T22:24:22.156 INFO:tasks.rados.rados.0.smithi076.stdout:1227: expect deleted 2021-03-15T22:24:22.156 INFO:tasks.rados.rados.0.smithi076.stdout:1228: write oid 20 current snap is 127 2021-03-15T22:24:22.156 INFO:tasks.rados.rados.0.smithi076.stdout:1228: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2021-03-15T22:24:22.161 INFO:tasks.rados.rados.0.smithi076.stdout:1228: writing smithi07694878-20 from 712729 to 1497506 tid 1 2021-03-15T22:24:22.163 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:22.167 INFO:tasks.rados.rados.0.smithi076.stdout:1228: writing smithi07694878-20 from 2091098 to 2558589 tid 2 2021-03-15T22:24:22.174 INFO:tasks.rados.rados.0.smithi076.stdout:1228: writing smithi07694878-20 from 3342313 to 3512729 tid 3 2021-03-15T22:24:22.174 INFO:tasks.rados.rados.0.smithi076.stdout:1229: snap_remove snap 123 2021-03-15T22:24:22.199 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:22.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:22.223 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:22.224 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:22.325 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:22.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:22.349 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:22.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:22.451 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:22.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:22.476 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:22.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:22.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[56514]: cluster 2021-03-15T22:24:21.036844+0000 mgr.y (mgr. 2021-03-15T22:24:22.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[56514]: 14144) 567 : cluster [DBG] pgmap v645: 138 pgs: 138 active+clean; 147 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 9.4 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2021-03-15T22:24:22.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[56514]: audit 2021-03-15T22:24:21.259010+0000 mgr.y (mgr.14144) 568 : audit [DBG] from='client.15315 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:22.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[56514]: audit 2021-03-15T22:24:21.259803+0000 mon.a (mon.0) 1099 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]: dispatch 2021-03-15T22:24:22.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[62156]: cluster 2021-03-15T22:24:21.036844+0000 2021-03-15T22:24:22.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[62156]: mgr.y (mgr.14144) 567 : cluster [DBG] pgmap v645: 138 pgs: 138 active+clean; 147 MiB data, 557 MiB used, 715 GiB / 715 GiB avail; 9.4 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2021-03-15T22:24:22.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[62156]: audit 2021-03-15T22:24:22.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[62156]: 2021-03-15T22:24:21.259010+0000 mgr.y (mgr. 2021-03-15T22:24:22.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[62156]: 14144) 568 : audit [DBG] from='client.15315 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:24:22.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[62156]: audit 2021-03-15T22:24:22.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:24:22.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[62156]: 24:21.259803+0000 mon.a (mon.0) 1099 2021-03-15T22:24:22.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:22 smithi076 conmon[62156]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]: dispatch 2021-03-15T22:24:22.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:22.599 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:24:22.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:22.601 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:22.602 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:22.703 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:22.726 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:22.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:22.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:22.829 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:22.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:22.852 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:22.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:22.929 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:22.954 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:22.977 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:22.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:22.979 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:23.080 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:23.102 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:23.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:23.104 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:23.134 INFO:tasks.rados.rados.0.smithi076.stdout:1229: done (5 left) 2021-03-15T22:24:23.134 INFO:tasks.rados.rados.0.smithi076.stdout:1230: delete oid 18 current snap is 127 2021-03-15T22:24:23.134 INFO:tasks.rados.rados.0.smithi076.stdout:1224: finishing write tid 3 to smithi07694878-14 2021-03-15T22:24:23.135 INFO:tasks.rados.rados.0.smithi076.stdout:1224: finishing write tid 4 to smithi07694878-14 2021-03-15T22:24:23.135 INFO:tasks.rados.rados.0.smithi076.stdout:1224: finishing write tid 5 to smithi07694878-14 2021-03-15T22:24:23.135 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 649 (ObjNum 446 snap 127 seq_num 446) dirty exists 2021-03-15T22:24:23.135 INFO:tasks.rados.rados.0.smithi076.stdout:1224: left oid 14 (ObjNum 446 snap 127 seq_num 446) 2021-03-15T22:24:23.136 INFO:tasks.rados.rados.0.smithi076.stdout:1228: finishing write tid 1 to smithi07694878-20 2021-03-15T22:24:23.136 INFO:tasks.rados.rados.0.smithi076.stdout:1228: finishing write tid 2 to smithi07694878-20 2021-03-15T22:24:23.136 INFO:tasks.rados.rados.0.smithi076.stdout:1228: finishing write tid 3 to smithi07694878-20 2021-03-15T22:24:23.136 INFO:tasks.rados.rados.0.smithi076.stdout:1228: finishing write tid 4 to smithi07694878-20 2021-03-15T22:24:23.137 INFO:tasks.rados.rados.0.smithi076.stdout:1228: finishing write tid 5 to smithi07694878-20 2021-03-15T22:24:23.137 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 437 (ObjNum 447 snap 127 seq_num 447) dirty exists 2021-03-15T22:24:23.137 INFO:tasks.rados.rados.0.smithi076.stdout:1228: left oid 20 (ObjNum 447 snap 127 seq_num 447) 2021-03-15T22:24:23.144 INFO:tasks.rados.rados.0.smithi076.stdout:1224: done (5 left) 2021-03-15T22:24:23.144 INFO:tasks.rados.rados.0.smithi076.stdout:1227: done (4 left) 2021-03-15T22:24:23.145 INFO:tasks.rados.rados.0.smithi076.stdout:1228: done (3 left) 2021-03-15T22:24:23.145 INFO:tasks.rados.rados.0.smithi076.stdout:1230: done (2 left) 2021-03-15T22:24:23.145 INFO:tasks.rados.rados.0.smithi076.stdout:1231: rollback oid 8 current snap is 127 2021-03-15T22:24:23.145 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 112 2021-03-15T22:24:23.146 INFO:tasks.rados.rados.0.smithi076.stdout:1232: snap_create 2021-03-15T22:24:23.147 INFO:tasks.rados.rados.0.smithi076.stdout:1231: finishing rollback tid 0 to smithi07694878-8 2021-03-15T22:24:23.147 INFO:tasks.rados.rados.0.smithi076.stdout:1231: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:24:23.151 INFO:tasks.rados.rados.0.smithi076.stdout:1231: finishing rollback tid 2 to smithi07694878-8 2021-03-15T22:24:23.151 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 308 (ObjNum 399 snap 108 seq_num 399) dirty exists 2021-03-15T22:24:23.161 INFO:tasks.rados.rados.0.smithi076.stdout:1225: finishing copy_from to smithi07694878-6 2021-03-15T22:24:23.161 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 80 (ObjNum 429 snap 119 seq_num 429) dirty exists 2021-03-15T22:24:23.161 INFO:tasks.rados.rados.0.smithi076.stdout:1225: finishing copy_from racing read to smithi07694878-6 2021-03-15T22:24:23.206 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:23.230 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:23.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:23.232 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:23.333 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:23.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:23.357 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:23.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:23.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:23 smithi076 conmon[56514]: audit 2021-03-15T22:24:22.126559+0000 mon.a ( 2021-03-15T22:24:23.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:23 smithi076 conmon[56514]: mon.0) 1100 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]': finished 2021-03-15T22:24:23.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:23 smithi076 conmon[56514]: cluster 2021-03-15T22:24:22.126639+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2021-03-15T22:24:23.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:23 smithi076 conmon[62156]: audit 2021-03-15T22:24:22.126559+0000 mon.a (mon.0) 1100 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]': finished 2021-03-15T22:24:23.385 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:23 smithi076 conmon[62156]: cluster 2021-03-15T22:24:22.126639+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2021-03-15T22:24:23.390 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:23 smithi125 conmon[56739]: audit 2021-03-15T22:24:22.126559+0000 mon.a (mon.0) 1100 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'3':'28824','5':'53040'}"}]': finished 2021-03-15T22:24:23.390 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:23 smithi125 conmon[56739]: cluster 2021-03-15T22:24:22.126639+0000 mon.a (mon.0) 1101 : cluster [DBG] osdmap e347: 8 total, 8 up, 8 in 2021-03-15T22:24:23.458 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:23.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:23.481 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:23.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:23.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:23.608 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:23.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:23.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:23.632 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:24:23.632 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:24:23.710 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:23.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:23.733 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:23.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:23.832 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:24:23.835 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:23.897 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:23.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:23.899 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:24.000 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:24.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:24.025 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:24.025 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:24.116 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:24.126 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:24.129 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:24 smithi076 conmon[56514]: cluster 2021-03-15T22:24:23.037250+0000 mgr.y (mgr.14144) 569 : cluster [DBG] pgmap v648: 138 pgs: 2 peering, 136 active+clean; 160 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 5.1 MiB/s wr, 10 op/s 2021-03-15T22:24:24.130 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:24 smithi076 conmon[56514]: cluster 2021-03-15T22:24:23.128609+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2021-03-15T22:24:24.130 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:24 smithi076 conmon[62156]: cluster 2021-03-15T22:24:23.037250+0000 mgr.y (mgr.14144) 569 : cluster [DBG] pgmap v648: 138 pgs: 2 peering, 136 active+clean; 160 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 5.1 MiB/s wr, 10 op/s 2021-03-15T22:24:24.130 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:24 smithi076 conmon[62156]: cluster 2021-03-15T22:24:23.128609+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2021-03-15T22:24:24.136 INFO:tasks.rados.rados.0.smithi076.stdout:1225: done (3 left) 2021-03-15T22:24:24.137 INFO:tasks.rados.rados.0.smithi076.stdout:1231: done (2 left) 2021-03-15T22:24:24.137 INFO:tasks.rados.rados.0.smithi076.stdout:1232: done (1 left) 2021-03-15T22:24:24.137 INFO:tasks.rados.rados.0.smithi076.stdout:1233: read oid 28 snap 126 2021-03-15T22:24:24.138 INFO:tasks.rados.rados.0.smithi076.stdout:1233: expect (ObjNum 421 snap 116 seq_num 421) 2021-03-15T22:24:24.138 INFO:tasks.rados.rados.0.smithi076.stdout:1234: write (excl) oid 20 current snap is 128 2021-03-15T22:24:24.138 INFO:tasks.rados.rados.0.smithi076.stdout:1234: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2021-03-15T22:24:24.144 INFO:tasks.rados.rados.0.smithi076.stdout:1234: writing smithi07694878-20 from 746343 to 1397971 tid 1 2021-03-15T22:24:24.152 INFO:tasks.rados.rados.0.smithi076.stdout:1234: writing smithi07694878-20 from 1924415 to 2679222 tid 2 2021-03-15T22:24:24.153 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:24.155 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:24.157 INFO:tasks.rados.rados.0.smithi076.stdout:1234: writing smithi07694878-20 from 3397159 to 3546343 tid 3 2021-03-15T22:24:24.157 INFO:tasks.rados.rados.0.smithi076.stdout:1235: read oid 40 snap 117 2021-03-15T22:24:24.157 INFO:tasks.rados.rados.0.smithi076.stdout:1235: expect (ObjNum 413 snap 112 seq_num 413) 2021-03-15T22:24:24.158 INFO:tasks.rados.rados.0.smithi076.stdout:1236: write (excl) oid 49 current snap is 128 2021-03-15T22:24:24.158 INFO:tasks.rados.rados.0.smithi076.stdout:1236: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2021-03-15T22:24:24.165 INFO:tasks.rados.rados.0.smithi076.stdout:1236: writing smithi07694878-49 from 763150 to 1490027 tid 1 2021-03-15T22:24:24.172 INFO:tasks.rados.rados.0.smithi076.stdout:1236: writing smithi07694878-49 from 1982897 to 2597715 tid 2 2021-03-15T22:24:24.178 INFO:tasks.rados.rados.0.smithi076.stdout:1236: writing smithi07694878-49 from 3024582 to 3499576 tid 3 2021-03-15T22:24:24.178 INFO:tasks.rados.rados.0.smithi076.stdout:1237: read oid 24 snap -1 2021-03-15T22:24:24.178 INFO:tasks.rados.rados.0.smithi076.stdout:1237: expect (ObjNum 429 snap 119 seq_num 429) 2021-03-15T22:24:24.179 INFO:tasks.rados.rados.0.smithi076.stdout:1233: expect (ObjNum 421 snap 116 seq_num 421) 2021-03-15T22:24:24.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:24.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-15T22:24:24.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:24.281 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:24.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:24.295 INFO:tasks.rados.rados.0.smithi076.stdout:1234: finishing write tid 1 to smithi07694878-20 2021-03-15T22:24:24.295 INFO:tasks.rados.rados.0.smithi076.stdout:1238: read oid 38 snap -1 2021-03-15T22:24:24.295 INFO:tasks.rados.rados.0.smithi076.stdout:1238: expect (ObjNum 431 snap 119 seq_num 431) 2021-03-15T22:24:24.296 INFO:tasks.rados.rados.0.smithi076.stdout:1233: done (6 left) 2021-03-15T22:24:24.296 INFO:tasks.rados.rados.0.smithi076.stdout:1239: snap_create 2021-03-15T22:24:24.296 INFO:tasks.rados.rados.0.smithi076.stdout:1234: finishing write tid 2 to smithi07694878-20 2021-03-15T22:24:24.296 INFO:tasks.rados.rados.0.smithi076.stdout:1226: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:24:24.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:24.403 INFO:tasks.rados.rados.0.smithi076.stdout:1234: finishing write tid 3 to smithi07694878-20 2021-03-15T22:24:24.404 INFO:tasks.rados.rados.0.smithi076.stdout:1234: finishing write tid 4 to smithi07694878-20 2021-03-15T22:24:24.404 INFO:tasks.rados.rados.0.smithi076.stdout:1234: finishing write tid 5 to smithi07694878-20 2021-03-15T22:24:24.404 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 441 (ObjNum 449 snap 128 seq_num 449) dirty exists 2021-03-15T22:24:24.404 INFO:tasks.rados.rados.0.smithi076.stdout:1234: left oid 20 (ObjNum 449 snap 128 seq_num 449) 2021-03-15T22:24:24.405 INFO:tasks.rados.rados.0.smithi076.stdout:1235: expect (ObjNum 413 snap 112 seq_num 413) 2021-03-15T22:24:24.409 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:24.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:24.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:24.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:24.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:24 smithi125 conmon[56739]: cluster 2021-03-15T22:24:23.037250+0000 mgr.y (mgr.14144) 569 : cluster [DBG] pgmap v648: 138 pgs: 2 peering, 136 active+clean; 160 MiB data, 592 MiB used, 715 GiB / 715 GiB avail; 13 MiB/s rd, 5.1 MiB/s wr, 10 op/s 2021-03-15T22:24:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:24 smithi125 conmon[56739]: cluster 2021-03-15T22:24:23.128609+0000 mon.a (mon.0) 1102 : cluster [DBG] osdmap e348: 8 total, 8 up, 8 in 2021-03-15T22:24:24.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:24.538 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:24.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:24.566 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:24.624 INFO:tasks.rados.rados.0.smithi076.stdout:1236: finishing write tid 1 to smithi07694878-49 2021-03-15T22:24:24.625 INFO:tasks.rados.rados.0.smithi076.stdout:1236: finishing write tid 2 to smithi07694878-49 2021-03-15T22:24:24.625 INFO:tasks.rados.rados.0.smithi076.stdout:1237: expect (ObjNum 429 snap 119 seq_num 429) 2021-03-15T22:24:24.639 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:24.661 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:24.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:24.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:24.761 INFO:tasks.rados.rados.0.smithi076.stdout:1236: finishing write tid 3 to smithi07694878-49 2021-03-15T22:24:24.761 INFO:tasks.rados.rados.0.smithi076.stdout:1236: finishing write tid 4 to smithi07694878-49 2021-03-15T22:24:24.761 INFO:tasks.rados.rados.0.smithi076.stdout:1236: finishing write tid 5 to smithi07694878-49 2021-03-15T22:24:24.762 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 688 (ObjNum 450 snap 128 seq_num 450) dirty exists 2021-03-15T22:24:24.762 INFO:tasks.rados.rados.0.smithi076.stdout:1236: left oid 49 (ObjNum 450 snap 128 seq_num 450) 2021-03-15T22:24:24.762 INFO:tasks.rados.rados.0.smithi076.stdout:1238: expect (ObjNum 431 snap 119 seq_num 431) 2021-03-15T22:24:24.765 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:24.787 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:24.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:24.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:24.889 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:24.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:24.914 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:24.914 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:25.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:25.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:25.039 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:25.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:25.081 INFO:tasks.rados.rados.0.smithi076.stdout:1226: done (6 left) 2021-03-15T22:24:25.081 INFO:tasks.rados.rados.0.smithi076.stdout:1234: done (5 left) 2021-03-15T22:24:25.081 INFO:tasks.rados.rados.0.smithi076.stdout:1235: done (4 left) 2021-03-15T22:24:25.082 INFO:tasks.rados.rados.0.smithi076.stdout:1236: done (3 left) 2021-03-15T22:24:25.082 INFO:tasks.rados.rados.0.smithi076.stdout:1237: done (2 left) 2021-03-15T22:24:25.082 INFO:tasks.rados.rados.0.smithi076.stdout:1238: done (1 left) 2021-03-15T22:24:25.082 INFO:tasks.rados.rados.0.smithi076.stdout:1239: done (0 left) 2021-03-15T22:24:25.083 INFO:tasks.rados.rados.0.smithi076.stdout:1240: read oid 2 snap -1 2021-03-15T22:24:25.083 INFO:tasks.rados.rados.0.smithi076.stdout:1240: expect (ObjNum 422 snap 116 seq_num 422) 2021-03-15T22:24:25.083 INFO:tasks.rados.rados.0.smithi076.stdout:1241: rollback oid 33 current snap is 129 2021-03-15T22:24:25.083 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 33 to 115 2021-03-15T22:24:25.084 INFO:tasks.rados.rados.0.smithi076.stdout:1242: read oid 10 snap -1 2021-03-15T22:24:25.084 INFO:tasks.rados.rados.0.smithi076.stdout:1242: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:24:25.084 INFO:tasks.rados.rados.0.smithi076.stdout:1243: snap_create 2021-03-15T22:24:25.085 INFO:tasks.rados.rados.0.smithi076.stdout:1241: finishing rollback tid 0 to smithi07694878-33 2021-03-15T22:24:25.085 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:24:25.086 INFO:tasks.rados.rados.0.smithi076.stdout:1241: finishing rollback tid 1 to smithi07694878-33 2021-03-15T22:24:25.087 INFO:tasks.rados.rados.0.smithi076.stdout:1241: finishing rollback tid 2 to smithi07694878-33 2021-03-15T22:24:25.088 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 444 (ObjNum 374 snap 102 seq_num 374) dirty exists 2021-03-15T22:24:25.095 INFO:tasks.rados.rados.0.smithi076.stdout:1240: expect (ObjNum 422 snap 116 seq_num 422) 2021-03-15T22:24:25.137 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[56514]: cluster 2021-03-15T22:24:24.133514+0000 mon.a (mon. 2021-03-15T22:24:25.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[62156]: cluster 2021-03-15T22:24:24.133514 2021-03-15T22:24:25.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1103 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2021-03-15T22:24:25.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[62156]: audit 2021-03-15T22:24:24.735861+0000 mon.a (mon.0) 1104 : audit [INF] from='client.? 172.21.15.76:0/790297711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:24:25.139 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[62156]: cluster 2021-03-15T22:24:25.068623+0000 mon.a (mon.0) 1105 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:25.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[62156]: audit 2021-03-15T22:24:25.078205+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? 172.21.15.76:0/790297711' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:24:25.140 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[62156]: cluster 2021-03-15T22:24:25.078263+0000 mon.a (mon.0) 1107 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2021-03-15T22:24:25.140 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:25.164 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:25.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:25.228 INFO:tasks.rados.rados.0.smithi076.stdout:1242: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:24:25.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:25.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:25.289 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:25.289 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:25.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:25.412 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:25.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:25.515 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:25.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[56514]: 0) 1103 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2021-03-15T22:24:25.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[56514]: audit 2021-03-15T22:24:24.735861+0000 mon.a (mon.0) 1104 : audit [INF] from='client.? 172.21.15.76:0/790297711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:24:25.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[56514]: cluster 2021-03-15T22:24:25.068623+0000 mon.a (mon.0) 1105 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:25.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[56514]: audit 2021-03-15T22:24:25.078205+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? 172.21.15.76:0/790297711' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:24:25.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:25 smithi076 conmon[56514]: cluster 2021-03-15T22:24:25.078263+0000 mon.a (mon.0) 1107 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2021-03-15T22:24:25.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:25 smithi125 conmon[56739]: cluster 2021-03-15T22:24:24.133514+0000 mon.a (mon.0) 1103 : cluster [DBG] osdmap e349: 8 total, 8 up, 8 in 2021-03-15T22:24:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:25 smithi125 conmon[56739]: audit 2021-03-15T22:24:24.735861+0000 mon.a (mon.0) 1104 : audit [INF] from='client.? 172.21.15.76:0/790297711' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:24:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:25 smithi125 conmon[56739]: cluster 2021-03-15T22:24:25.068623+0000 mon.a (mon.0) 1105 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:25 smithi125 conmon[56739]: audit 2021-03-15T22:24:25.078205+0000 mon.a (mon.0) 1106 : audit [INF] from='client.? 172.21.15.76:0/790297711' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:24:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:25 smithi125 conmon[56739]: cluster 2021-03-15T22:24:25.078263+0000 mon.a (mon.0) 1107 : cluster [DBG] osdmap e350: 8 total, 8 up, 8 in 2021-03-15T22:24:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:25.539 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:25.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:25.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:25.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:25.665 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:25.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:25.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:25.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:25.790 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:25.790 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:25.892 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:25.916 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:25.916 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:26.017 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:26.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:26.041 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:26.041 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:26.047 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:24:26.047 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:24:26.077 INFO:tasks.rados.rados.0.smithi076.stdout:1240: done (3 left) 2021-03-15T22:24:26.077 INFO:tasks.rados.rados.0.smithi076.stdout:1241: done (2 left) 2021-03-15T22:24:26.077 INFO:tasks.rados.rados.0.smithi076.stdout:1242: done (1 left) 2021-03-15T22:24:26.078 INFO:tasks.rados.rados.0.smithi076.stdout:1243: done (0 left) 2021-03-15T22:24:26.078 INFO:tasks.rados.rados.0.smithi076.stdout:1244: write (excl) oid 16 current snap is 130 2021-03-15T22:24:26.078 INFO:tasks.rados.rados.0.smithi076.stdout:1244: seq_num 451 ranges {779957=402126,1925026=758476,3335652=244305} 2021-03-15T22:24:26.083 INFO:tasks.rados.rados.0.smithi076.stdout:1244: writing smithi07694878-16 from 779957 to 1182083 tid 1 2021-03-15T22:24:26.092 INFO:tasks.rados.rados.0.smithi076.stdout:1244: writing smithi07694878-16 from 1925026 to 2683502 tid 2 2021-03-15T22:24:26.097 INFO:tasks.rados.rados.0.smithi076.stdout:1244: writing smithi07694878-16 from 3335652 to 3579957 tid 3 2021-03-15T22:24:26.097 INFO:tasks.rados.rados.0.smithi076.stdout:1245: rollback oid 5 current snap is 130 2021-03-15T22:24:26.098 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 112 2021-03-15T22:24:26.098 INFO:tasks.rados.rados.0.smithi076.stdout:1244: finishing write tid 1 to smithi07694878-16 2021-03-15T22:24:26.098 INFO:tasks.rados.rados.0.smithi076.stdout:1244: finishing write tid 2 to smithi07694878-16 2021-03-15T22:24:26.098 INFO:tasks.rados.rados.0.smithi076.stdout:1246: write (excl) oid 25 current snap is 130 2021-03-15T22:24:26.098 INFO:tasks.rados.rados.0.smithi076.stdout:1246: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2021-03-15T22:24:26.104 INFO:tasks.rados.rados.0.smithi076.stdout:1246: writing smithi07694878-25 from 796764 to 1274139 tid 1 2021-03-15T22:24:26.111 INFO:tasks.rados.rados.0.smithi076.stdout:1246: writing smithi07694878-25 from 1983508 to 2601995 tid 2 2021-03-15T22:24:26.117 INFO:tasks.rados.rados.0.smithi076.stdout:1246: writing smithi07694878-25 from 3363075 to 3596764 tid 3 2021-03-15T22:24:26.118 INFO:tasks.rados.rados.0.smithi076.stdout:1247: snap_create 2021-03-15T22:24:26.118 INFO:tasks.rados.rados.0.smithi076.stdout:1245: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:24:26.118 INFO:tasks.rados.rados.0.smithi076.stdout:1245: finishing rollback tid 2 to smithi07694878-5 2021-03-15T22:24:26.118 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 591 (ObjNum 371 snap 102 seq_num 371) dirty exists 2021-03-15T22:24:26.119 INFO:tasks.rados.rados.0.smithi076.stdout:1244: finishing write tid 3 to smithi07694878-16 2021-03-15T22:24:26.119 INFO:tasks.rados.rados.0.smithi076.stdout:1244: finishing write tid 4 to smithi07694878-16 2021-03-15T22:24:26.119 INFO:tasks.rados.rados.0.smithi076.stdout:1244: finishing write tid 5 to smithi07694878-16 2021-03-15T22:24:26.119 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 695 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:24:26.119 INFO:tasks.rados.rados.0.smithi076.stdout:1244: left oid 16 (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:24:26.120 INFO:tasks.rados.rados.0.smithi076.stdout:1246: finishing write tid 1 to smithi07694878-25 2021-03-15T22:24:26.121 INFO:tasks.rados.rados.0.smithi076.stdout:1246: finishing write tid 2 to smithi07694878-25 2021-03-15T22:24:26.121 INFO:tasks.rados.rados.0.smithi076.stdout:1246: finishing write tid 3 to smithi07694878-25 2021-03-15T22:24:26.121 INFO:tasks.rados.rados.0.smithi076.stdout:1246: finishing write tid 4 to smithi07694878-25 2021-03-15T22:24:26.121 INFO:tasks.rados.rados.0.smithi076.stdout:1246: finishing write tid 5 to smithi07694878-25 2021-03-15T22:24:26.122 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 244 (ObjNum 452 snap 130 seq_num 452) dirty exists 2021-03-15T22:24:26.122 INFO:tasks.rados.rados.0.smithi076.stdout:1246: left oid 25 (ObjNum 452 snap 130 seq_num 452) 2021-03-15T22:24:26.138 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[56514]: cluster 2021-03-15T22:24:25.037754+0000 mgr.y (mgr.14144 2021-03-15T22:24:26.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:26.165 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:26.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:26.167 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:26.268 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:26.290 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:26.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:26.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:26.393 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:26.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:26.416 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:26.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:26.425 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[56514]: ) 570 : cluster [DBG] pgmap v651: 138 pgs: 2 peering, 136 active+clean; 160 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s wr, 1 op/s; 747 KiB/s, 0 objects/s recovering 2021-03-15T22:24:26.425 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[56514]: cluster 2021-03-15T22:24:25.372531+0000 osd.6 (osd.6) 51 : cluster [DBG] 6.c scrub ok 2021-03-15T22:24:26.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[56514]: cluster 2021-03-15T22:24:25.646503+0000 osd.2 (osd.2) 60 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:26.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[56514]: cluster 2021-03-15T22:24:25.647000+0000 osd.2 (osd.2) 61 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:26.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[56514]: cluster 2021-03-15T22:24:25.769845+0000 osd.5 (osd.5) 2021-03-15T22:24:26.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[56514]: 63 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:24:26.427 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[56514]: cluster 2021-03-15T22:24:26.074708+0000 mon.a (mon.0) 1108 : cluster 2021-03-15T22:24:26.427 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[56514]: [DBG] osdmap e351: 8 total, 8 up, 8 in 2021-03-15T22:24:26.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: cluster 2021-03-15T22:24:25. 2021-03-15T22:24:26.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: 037754+0000 mgr.y (mgr.14144) 570 : cluster [DBG] pgmap v651: 138 pgs: 2 peering, 136 active+clean; 160 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s wr, 1 op/s; 747 KiB/s, 0 objects/s recovering 2021-03-15T22:24:26.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: cluster 2021-03-15T22:24:25. 2021-03-15T22:24:26.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: 372531+0000 osd.6 (osd.6) 51 : cluster [DBG] 6.c scrub ok 2021-03-15T22:24:26.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:24:26.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: -03-15T22:24:25. 2021-03-15T22:24:26.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: 646503+0000 osd.2 (osd.2) 2021-03-15T22:24:26.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: 60 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:26.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:24:26.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: -03-15T22:24:25 2021-03-15T22:24:26.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: .647000+0000 osd.2 (osd. 2021-03-15T22:24:26.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: 2) 61 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:26.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: 2021-03-15T22:24:26.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:24:26.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: -15T22:24:25. 2021-03-15T22:24:26.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: 769845+0000 osd.5 (osd.5) 2021-03-15T22:24:26.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: 63 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:24:26.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:24:26.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: -03-15T22:24:26 2021-03-15T22:24:26.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: .074708+0000 mon.a (mon. 2021-03-15T22:24:26.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: 0) 1108 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2021-03-15T22:24:26.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:26 smithi076 conmon[62156]: 2021-03-15T22:24:26.502 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:24:26.517 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:26.518 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:24:26.536 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:24:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:26 smithi125 conmon[56739]: cluster 2021-03-15T22:24:25.037754+0000 mgr.y (mgr.14144 2021-03-15T22:24:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:26 smithi125 conmon[56739]: ) 570 : cluster [DBG] pgmap v651: 138 pgs: 2 peering, 136 active+clean; 160 MiB data, 608 MiB used, 715 GiB / 715 GiB avail; 3.1 MiB/s wr, 1 op/s; 747 KiB/s, 0 objects/s recovering 2021-03-15T22:24:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:26 smithi125 conmon[56739]: cluster 2021-03-15T22:24:25.372531+0000 osd.6 (osd.6) 51 : cluster [DBG] 6.c scrub ok 2021-03-15T22:24:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:26 smithi125 conmon[56739]: cluster 2021-03-15T22:24:25.646503+0000 osd.2 (osd.2) 60 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:26 smithi125 conmon[56739]: cluster 2021-03-15T22:24:25.647000+0000 osd.2 (osd.2) 61 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:26 smithi125 conmon[56739]: cluster 2021-03-15T22:24:25.769845+0000 osd.5 (osd.5) 63 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:24:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:26 smithi125 conmon[56739]: cluster 2021-03-15T22:24:26.074708+0000 mon.a (mon.0) 1108 : cluster [DBG] osdmap e351: 8 total, 8 up, 8 in 2021-03-15T22:24:26.539 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:26.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:26.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:26.553 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:24:26.570 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:24:26.588 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:24:26.605 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:24:26.623 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:24:26.624 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:24:26.624 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:24:26.624 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:24:26.624 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:24:26.625 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:24:26.625 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:24:26.625 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:24:26.625 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:24:26.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:26.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:26.665 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:26.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:26.686 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-15T22:24:26.767 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:26.793 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:26.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:26.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:26.895 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:26.920 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:26.921 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:27.006 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:27.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:27.044 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:27.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:27.046 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:27.082 INFO:tasks.rados.rados.0.smithi076.stdout:1244: done (3 left) 2021-03-15T22:24:27.082 INFO:tasks.rados.rados.0.smithi076.stdout:1245: done (2 left) 2021-03-15T22:24:27.083 INFO:tasks.rados.rados.0.smithi076.stdout:1246: done (1 left) 2021-03-15T22:24:27.083 INFO:tasks.rados.rados.0.smithi076.stdout:1247: done (0 left) 2021-03-15T22:24:27.083 INFO:tasks.rados.rados.0.smithi076.stdout:1248: snap_create 2021-03-15T22:24:27.147 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:27.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:27.172 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:27.172 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:27.273 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:27.296 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:27.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:27.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:27.399 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:27.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:27 smithi076 conmon[56514]: cluster 2021-03-15T22:24:25.931054+0000 osd.1 (osd.1) 71 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:24:27.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:27 smithi076 conmon[56514]: cluster 2021-03-15T22:24:26.024795+0000 osd.7 (osd.7) 69 : cluster [DBG] 3.d scrub ok 2021-03-15T22:24:27.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:27 smithi076 conmon[56514]: cluster 2021-03-15T22:24:26.478794+0000 osd.0 (osd.0) 72 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:24:27.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:27 smithi076 conmon[56514]: audit 2021-03-15T22:24:26.483752+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.76:0/1149061019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:27.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:27 smithi076 conmon[56514]: cluster 2021-03-15T22:24:27.080171+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2021-03-15T22:24:27.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:27 smithi076 conmon[62156]: cluster 2021-03-15T22:24:25.931054+0000 osd.1 (osd.1) 71 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:24:27.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:27 smithi076 conmon[62156]: cluster 2021-03-15T22:24:26.024795+0000 osd.7 (osd.7) 69 : cluster [DBG] 3.d scrub ok 2021-03-15T22:24:27.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:27 smithi076 conmon[62156]: cluster 2021-03-15T22:24:26.478794+0000 osd.0 (osd.0) 72 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:24:27.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:27 smithi076 conmon[62156]: audit 2021-03-15T22:24:26.483752+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.76:0/1149061019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:27.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:27 smithi076 conmon[62156]: cluster 2021-03-15T22:24:27.080171+0000 mon.a (mon.0) 1109 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2021-03-15T22:24:27.411 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:27 smithi125 conmon[56739]: cluster 2021-03-15T22:24:25.931054+0000 osd.1 (osd 2021-03-15T22:24:27.411 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:27 smithi125 conmon[56739]: .1) 71 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:24:27.411 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:27 smithi125 conmon[56739]: cluster 2021-03-15T22:24:26.024795+0000 osd.7 (osd.7) 69 : cluster [DBG] 3.d scrub ok 2021-03-15T22:24:27.412 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:27 smithi125 conmon[56739]: cluster 2021-03-15T22:24:26.478794+0000 osd.0 (osd.0) 72 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:24:27.412 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:27 smithi125 conmon[56739]: audit 2021-03-15T22:24:26.483752+0000 mon.c (mon.1) 31 : audit [DBG] from='client.? 172.21.15.76:0/1149061019' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:27.412 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:27 smithi125 conmon[56739]: cluster 2021-03-15T22:24:27.080171+0000 mon.a (mon.0 2021-03-15T22:24:27.412 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:27 smithi125 conmon[56739]: ) 1109 : cluster [DBG] osdmap e352: 8 total, 8 up, 8 in 2021-03-15T22:24:27.424 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:27.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:27.425 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:27.526 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:27.549 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:27.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:27.550 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:27.651 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:27.674 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:27.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:27.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:27.777 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:27.799 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 4, 5, 6, 7, 3] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:24:27.799 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:24:27.800 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 1, 2, 4, 5, 6, 7, 3] 2021-03-15T22:24:27.800 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd out 0 2021-03-15T22:24:27.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:27.802 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:27.802 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:27.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:27.925 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:27.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:27.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:28.028 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:28.052 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:28.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:28.079 INFO:tasks.rados.rados.0.smithi076.stdout:1248: done (0 left) 2021-03-15T22:24:28.080 INFO:tasks.rados.rados.0.smithi076.stdout:1249: copy_from oid 40 from oid 35 current snap is 132 2021-03-15T22:24:28.080 INFO:tasks.rados.rados.0.smithi076.stdout:1250: rollback oid 13 current snap is 132 2021-03-15T22:24:28.080 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 13 to 125 2021-03-15T22:24:28.080 INFO:tasks.rados.rados.0.smithi076.stdout:1251: copy_from oid 14 from oid 8 current snap is 132 2021-03-15T22:24:28.081 INFO:tasks.rados.rados.0.smithi076.stdout:1252: snap_remove snap 115 2021-03-15T22:24:28.089 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:28.146 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[56514]: cluster 2021-03-15T22:24:26.961973+0000 osd.1 (osd.1) 72 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:24:28.147 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[56514]: cluster 2021-03-15T22:24:27.038197+0000 mgr.y (mgr.14144) 571 : cluster [DBG] pgmap v654: 138 pgs: 3 peering, 135 active+clean; 176 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 2.7 MiB/s wr, 21 op/s; 6.9 MiB/s, 23 objects/s recovering 2021-03-15T22:24:28.147 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[56514]: cluster 2021-03-15T22:24:27.166675+0000 osd.3 (osd.3) 62 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:24:28.147 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[56514]: cluster 2021-03-15T22:24:27.869565+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.a scrub ok 2021-03-15T22:24:28.147 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[56514]: cluster 2021-03-15T22:24:28.076876+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2021-03-15T22:24:28.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:28.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:24:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:28.323 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:28.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:28.418 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:28 smithi125 conmon[56739]: cluster 2021-03-15T22:24:26.961973+0000 osd.1 (osd.1) 72 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:24:28.418 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:28 smithi125 conmon[56739]: cluster 2021-03-15T22:24:27.038197+0000 mgr.y (mgr.14144) 571 : cluster [DBG] pgmap v654: 138 pgs: 3 peering, 135 active+clean; 176 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 2.7 MiB/s wr, 21 op/s; 6.9 MiB/s, 23 objects/s recovering 2021-03-15T22:24:28.419 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:28 smithi125 conmon[56739]: cluster 2021-03-15T22:24:27.166675+0000 osd.3 (osd.3) 62 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:24:28.419 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:28 smithi125 conmon[56739]: cluster 2021-03-15T22:24:27.869565+0000 osd.2 (osd.2) 62 : cluster [DBG] 5.a scrub ok 2021-03-15T22:24:28.420 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:28 smithi125 conmon[56739]: cluster 2021-03-15T22:24:28.076876+0000 mon.a (mon.0) 1110 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2021-03-15T22:24:28.426 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:28.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:28.460 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:28.460 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:28.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: cluster 2021-03-15T22:24:26.961973+0000 2021-03-15T22:24:28.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: osd.1 (osd.1) 72 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:24:28.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:24:28.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: 22:24:27.038197+0000 mgr.y (mgr.14144) 571 : cluster [DBG] pgmap v654: 138 pgs: 3 peering, 135 active+clean; 176 MiB data, 624 MiB used, 715 GiB / 715 GiB avail; 16 MiB/s rd, 2.7 MiB/s wr, 21 op/s; 6.9 MiB/s, 23 objects/s recovering 2021-03-15T22:24:28.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:24:28.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: 22:24:27.166675+0000 osd.3 (osd.3) 2021-03-15T22:24:28.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: 62 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:24:28.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: cluster 2021-03-15T22:24 2021-03-15T22:24:28.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: :27.869565+0000 osd.2 (osd.2) 62 : 2021-03-15T22:24:28.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: cluster [DBG] 5.a scrub ok 2021-03-15T22:24:28.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: cluster 2021-03-15T22:24:28.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: 2021-03-15T22:24:28 2021-03-15T22:24:28.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: .076876+0000 mon.a (mon.0 2021-03-15T22:24:28.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:28 smithi076 conmon[62156]: ) 1110 : cluster [DBG] osdmap e353: 8 total, 8 up, 8 in 2021-03-15T22:24:28.563 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:28.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:28.592 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:28.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:28.624 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:24:28.625 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:24:28.625 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:24:28.626 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:24:28.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:28.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:28.720 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:28.725 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:28.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:28.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:28.856 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:28.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:28.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:28.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:28.983 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:28.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:29.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:29.088 INFO:tasks.rados.rados.0.smithi076.stdout:1252: done (3 left) 2021-03-15T22:24:29.089 INFO:tasks.rados.rados.0.smithi076.stdout:1253: copy_from oid 27 from oid 45 current snap is 132 2021-03-15T22:24:29.089 INFO:tasks.rados.rados.0.smithi076.stdout:1249: finishing copy_from to smithi07694878-40 2021-03-15T22:24:29.089 INFO:tasks.rados.rados.0.smithi076.stdout:1249: got expected ENOENT (src dne) 2021-03-15T22:24:29.089 INFO:tasks.rados.rados.0.smithi076.stdout:1249: finishing copy_from racing read to smithi07694878-40 2021-03-15T22:24:29.090 INFO:tasks.rados.rados.0.smithi076.stdout:1250: finishing rollback tid 1 to smithi07694878-13 2021-03-15T22:24:29.090 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 418 (ObjNum 261810640 snap 21896 seq_num 0) dirty dne 2021-03-15T22:24:29.091 INFO:tasks.rados.rados.0.smithi076.stdout:1251: finishing copy_from to smithi07694878-14 2021-03-15T22:24:29.091 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 308 (ObjNum 399 snap 108 seq_num 399) dirty exists 2021-03-15T22:24:29.092 INFO:tasks.rados.rados.0.smithi076.stdout:1251: finishing copy_from racing read to smithi07694878-14 2021-03-15T22:24:29.092 INFO:tasks.rados.rados.0.smithi076.stdout:1249: done (3 left) 2021-03-15T22:24:29.092 INFO:tasks.rados.rados.0.smithi076.stdout:1250: done (2 left) 2021-03-15T22:24:29.093 INFO:tasks.rados.rados.0.smithi076.stdout:1251: done (1 left) 2021-03-15T22:24:29.093 INFO:tasks.rados.rados.0.smithi076.stdout:1254: copy_from oid 18 from oid 47 current snap is 132 2021-03-15T22:24:29.093 INFO:tasks.rados.rados.0.smithi076.stdout:1255: read oid 50 snap -1 2021-03-15T22:24:29.093 INFO:tasks.rados.rados.0.smithi076.stdout:1255: expect deleted 2021-03-15T22:24:29.094 INFO:tasks.rados.rados.0.smithi076.stdout:1256: read oid 48 snap 117 2021-03-15T22:24:29.094 INFO:tasks.rados.rados.0.smithi076.stdout:1256: expect (ObjNum 420 snap 116 seq_num 420) 2021-03-15T22:24:29.094 INFO:tasks.rados.rados.0.smithi076.stdout:1257: read oid 25 snap 119 2021-03-15T22:24:29.094 INFO:tasks.rados.rados.0.smithi076.stdout:1257: expect deleted 2021-03-15T22:24:29.095 INFO:tasks.rados.rados.0.smithi076.stdout:1258: snap_remove snap 122 2021-03-15T22:24:29.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:29.112 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:29.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:29.215 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:29.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-15T22:24:29.238 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:29.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:29.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:29.250 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:29.341 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:29.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:29.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:29.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[56514]: cluster 2021-03-15T22:24:28.184604+0000 osd.3 (osd.3) 63 : cluster [DBG] 2021-03-15T22:24:29.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[56514]: 7.a scrub ok 2021-03-15T22:24:29.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[56514]: cluster 2021-03-15T22:24:28.327673+0000 osd.6 (osd.6) 52 : cluster [DBG] 3.a scrub ok 2021-03-15T22:24:29.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[56514]: cluster 2021-03-15T22:24:28.507555+0000 osd.0 (osd.0) 73 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:24:29.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[56514]: cluster 2021-03-15T22:24:28.700652+0000 osd.5 (osd.5) 64 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:24:29.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[56514]: cluster 2021-03-15T22:24:29.081273+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2021-03-15T22:24:29.410 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: cluster 2021-03-15T22:24:28.184604+0000 osd.3 2021-03-15T22:24:29.410 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: (osd.3) 63 : cluster [DBG] 7.a scrub ok 2021-03-15T22:24:29.410 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: cluster 2021-03-15T22:24:28. 2021-03-15T22:24:29.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: 327673+0000 osd.6 (osd.6) 52 : cluster [DBG] 3.a scrub ok 2021-03-15T22:24:29.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: cluster 2021-03-15T22:24:29.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: 2021-03-15T22:24:28.507555+0000 osd.0 ( 2021-03-15T22:24:29.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: osd.0) 73 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:24:29.412 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: cluster 2021-03-15T22:24:28 2021-03-15T22:24:29.412 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: .700652+0000 osd.5 (osd.5) 64 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:24:29.412 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: cluster 2021-03-15T22:24:29.412 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: 2021-03-15T22:24 2021-03-15T22:24:29.413 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[62156]: :29.081273+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2021-03-15T22:24:29.424 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:24:29.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:29.490 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:29.492 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:29.509 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:29 smithi125 conmon[56739]: cluster 2021-03-15T22:24:28.184604+0000 osd.3 (osd.3) 2021-03-15T22:24:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:29 smithi125 conmon[56739]: 63 : cluster [DBG] 7.a scrub ok 2021-03-15T22:24:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:29 smithi125 conmon[56739]: cluster 2021-03-15T22:24:28.327673+0000 osd.6 (osd.6) 52 : cluster [DBG] 3.a scrub ok 2021-03-15T22:24:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:29 smithi125 conmon[56739]: cluster 2021-03-15T22:24:28.507555+0000 osd.0 (osd.0) 73 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:24:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:29 smithi125 conmon[56739]: cluster 2021-03-15T22:24:28.700652+0000 osd.5 (osd.5) 64 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:24:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:29 smithi125 conmon[56739]: cluster 2021-03-15T22:24:29.081273+0000 mon.a (mon.0) 1111 : cluster [DBG] osdmap e354: 8 total, 8 up, 8 in 2021-03-15T22:24:29.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:29.615 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:29.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:29.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:29.718 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:29.740 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:29.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:29.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:29.843 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:29.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:29.868 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:29.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:29.911 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:29.920 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:24:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:24:29.970 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:29.992 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:29.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:29.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:30.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:29 smithi076 conmon[56514]: debug 2021-03-15T22:24:29.842+0000 7f0537c88700 -1 mon.a@0(leader).osd e354 definitely_dead 0 2021-03-15T22:24:30.089 INFO:tasks.rados.rados.0.smithi076.stdout:1258: done (5 left) 2021-03-15T22:24:30.090 INFO:tasks.rados.rados.0.smithi076.stdout:1259: write (excl) oid 14 current snap is 132 2021-03-15T22:24:30.090 INFO:tasks.rados.rados.0.smithi076.stdout:1259: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2021-03-15T22:24:30.090 INFO:tasks.rados.rados.0.smithi076.stdout:1259: writing smithi07694878-14 from 413571 to 966195 tid 1 2021-03-15T22:24:30.095 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:30.100 INFO:tasks.rados.rados.0.smithi076.stdout:1259: writing smithi07694878-14 from 1641990 to 2404135 tid 2 2021-03-15T22:24:30.106 INFO:tasks.rados.rados.0.smithi076.stdout:1259: writing smithi07694878-14 from 2990498 to 3415661 tid 3 2021-03-15T22:24:30.115 INFO:tasks.rados.rados.0.smithi076.stdout:1253: finishing copy_from to smithi07694878-27 2021-03-15T22:24:30.116 INFO:tasks.rados.rados.0.smithi076.stdout:1253: got expected ENOENT (src dne) 2021-03-15T22:24:30.116 INFO:tasks.rados.rados.0.smithi076.stdout:1256: expect (ObjNum 420 snap 116 seq_num 420) 2021-03-15T22:24:30.118 INFO:teuthology.orchestra.run.smithi076.stderr:marked out osd.0. 2021-03-15T22:24:30.121 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:30.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:30.122 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:30.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:30.246 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:30.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:30.296 INFO:tasks.rados.rados.0.smithi076.stdout:1253: finishing copy_from racing read to smithi07694878-27 2021-03-15T22:24:30.296 INFO:tasks.rados.rados.0.smithi076.stdout:1254: finishing copy_from to smithi07694878-18 2021-03-15T22:24:30.297 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 164 (ObjNum 324 snap 87 seq_num 324) dirty exists 2021-03-15T22:24:30.297 INFO:tasks.rados.rados.0.smithi076.stdout:1254: finishing copy_from racing read to smithi07694878-18 2021-03-15T22:24:30.297 INFO:tasks.rados.rados.0.smithi076.stdout:1253: done (5 left) 2021-03-15T22:24:30.297 INFO:tasks.rados.rados.0.smithi076.stdout:1254: done (4 left) 2021-03-15T22:24:30.298 INFO:tasks.rados.rados.0.smithi076.stdout:1255: done (3 left) 2021-03-15T22:24:30.298 INFO:tasks.rados.rados.0.smithi076.stdout:1256: done (2 left) 2021-03-15T22:24:30.298 INFO:tasks.rados.rados.0.smithi076.stdout:1257: done (1 left) 2021-03-15T22:24:30.298 INFO:tasks.rados.rados.0.smithi076.stdout:1260: delete oid 4 current snap is 132 2021-03-15T22:24:30.299 INFO:tasks.rados.rados.0.smithi076.stdout:1259: finishing write tid 1 to smithi07694878-14 2021-03-15T22:24:30.299 INFO:tasks.rados.rados.0.smithi076.stdout:1259: finishing write tid 2 to smithi07694878-14 2021-03-15T22:24:30.299 INFO:tasks.rados.rados.0.smithi076.stdout:1259: finishing write tid 3 to smithi07694878-14 2021-03-15T22:24:30.299 INFO:tasks.rados.rados.0.smithi076.stdout:1259: finishing write tid 4 to smithi07694878-14 2021-03-15T22:24:30.300 INFO:tasks.rados.rados.0.smithi076.stdout:1259: finishing write tid 5 to smithi07694878-14 2021-03-15T22:24:30.300 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 653 (ObjNum 453 snap 132 seq_num 453) dirty exists 2021-03-15T22:24:30.300 INFO:tasks.rados.rados.0.smithi076.stdout:1259: left oid 14 (ObjNum 453 snap 132 seq_num 453) 2021-03-15T22:24:30.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:30.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:30.370 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:30.371 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:30.472 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:30.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:30.496 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:30.496 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:30.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: cluster 2021-03-15T22:24:29.038588+0000 2021-03-15T22:24:30.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: mgr.y (mgr.14144) 572 : cluster [DBG] pgmap v657: 138 pgs: 1 peering, 137 active+clean; 176 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 35 op/s; 6.1 MiB/s, 23 objects/s recovering 2021-03-15T22:24:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: cluster 2021-03-15T22:24:29.506401+0000 osd.0 (osd.0) 74 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:24:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: audit 2021-03-15T22:24:29.648312+0000 mgr.y (mgr.14144) 573 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: audit 2021-03-15T22:24:29.842570+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.76:0/1002683111' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2021-03-15T22:24:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: audit 2021-03-15T22:24:29 2021-03-15T22:24:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: .842885+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2021-03-15T22:24:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:24:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: T22:24:29.842967+0000 mon.a (mon.0) 1113 : cluster [INF] Client client.admin marked osd.0 out, while it was still marked up 2021-03-15T22:24:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:24:30.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: 03-15T22:24:29.903436+0000 osd.2 (osd.2 2021-03-15T22:24:30.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: ) 63 : cluster [DBG] 5.d scrub ok 2021-03-15T22:24:30.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: audit 2021-03-15T22:24:30.075393+0000 2021-03-15T22:24:30.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: mon.a (mon.0) 1114 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2021-03-15T22:24:30.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: cluster 2021-03-15T22:24 2021-03-15T22:24:30.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: :30.075458+0000 mon.a (mon.0) 1115 : cluster [DBG] 2021-03-15T22:24:30.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[56514]: osdmap e355: 8 total, 8 up, 7 in 2021-03-15T22:24:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: cluster 2021-03-15T22:24:29.038588 2021-03-15T22:24:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 572 : cluster [DBG] pgmap v657: 138 pgs: 1 peering, 137 active+clean; 176 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 35 op/s; 6.1 MiB/s, 23 objects/s recovering 2021-03-15T22:24:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:24:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: 24:29.506401+0000 osd.0 (osd.0) 74 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:24:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: audit 2021-03-15T22:24 2021-03-15T22:24:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: :29.648312+0000 mgr.y (mgr.14144) 573 : audit 2021-03-15T22:24:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: audit 2021-03-15T22:24:29. 2021-03-15T22:24:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: 842570+0000 mon.c (mon.1) 32 : audit [INF] 2021-03-15T22:24:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/1002683111' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2021-03-15T22:24:30.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: audit 2021-03-15T22:24:30.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: 2021-03-15T22:24:29.842885+0000 mon.a (mon. 2021-03-15T22:24:30.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: 0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2021-03-15T22:24:30.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:24:30.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: 15T22:24:29.842967+0000 mon.a 2021-03-15T22:24:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: (mon.0) 1113 : cluster [INF] Client client.admin marked osd.0 out, while it was still marked up 2021-03-15T22:24:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: 2021-03-15T22:24:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:24:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: 22:24:29.903436+0000 osd.2 (osd.2) 63 : cluster [DBG] 5.d scrub ok 2021-03-15T22:24:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: audit 2021-03-15T22:24 2021-03-15T22:24:30.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: :30.075393+0000 mon.a (mon.0) 1114 : audit [INF] 2021-03-15T22:24:30.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2021-03-15T22:24:30.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: cluster 2021-03-15T22:24:30.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: 2021-03-15T22:24:30.075458+0000 mon.a ( 2021-03-15T22:24:30.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:30 smithi076 conmon[62156]: mon.0) 1115 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2021-03-15T22:24:30.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:30 smithi125 conmon[56739]: cluster 2021-03-15T22:24:29.038588+0000 mgr.y (mgr.14144) 572 : cluster [DBG] pgmap v657: 138 pgs: 1 peering, 137 active+clean; 176 MiB data, 656 MiB used, 715 GiB / 715 GiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 35 op/s; 6.1 MiB/s, 23 objects/s recovering 2021-03-15T22:24:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:30 smithi125 conmon[56739]: cluster 2021-03-15T22:24:29.506401+0000 osd.0 (osd.0) 74 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:24:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:30 smithi125 conmon[56739]: audit 2021-03-15T22:24:29.648312+0000 mgr.y (mgr.14144) 573 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:30 smithi125 conmon[56739]: audit 2021-03-15T22:24:29.842570+0000 mon.c (mon.1) 32 : audit [INF] from='client.? 172.21.15.76:0/1002683111' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2021-03-15T22:24:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:30 smithi125 conmon[56739]: audit 2021-03-15T22:24:29.842885+0000 mon.a (mon.0) 1112 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2021-03-15T22:24:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:30 smithi125 conmon[56739]: cluster 2021-03-15T22:24:29.842967+0000 mon.a (mon.0) 1113 : cluster [INF] Client client.admin marked osd.0 out, while it was still marked up 2021-03-15T22:24:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:30 smithi125 conmon[56739]: cluster 2021-03-15T22:24:29.903436+0000 osd.2 (osd.2) 63 : cluster [DBG] 5.d scrub ok 2021-03-15T22:24:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:30 smithi125 conmon[56739]: audit 2021-03-15T22:24:30.075393+0000 mon.a (mon.0) 2021-03-15T22:24:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:30 smithi125 conmon[56739]: 1114 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2021-03-15T22:24:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:30 smithi125 conmon[56739]: cluster 2021-03-15T22:24:30.075458+0000 mon.a (mon.0) 1115 : cluster [DBG] osdmap e355: 8 total, 8 up, 7 in 2021-03-15T22:24:30.597 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:30.620 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:30.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:30.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:30.722 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:30.745 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:30.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:30.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:30.848 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:30.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:30.871 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:30.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:30.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:30.996 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:30.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:30.998 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:31.099 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:31.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:31.132 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:31.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:31.149 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:24:31.149 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:24:31.151 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:24:31.151 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:24:31.152 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:24:31.234 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:31.257 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:31.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:31.259 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:31.333 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:31 smithi076 conmon[56514]: cluster 2021-03-15T22:24:30.068545+0000 osd.7 ( 2021-03-15T22:24:31.334 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:31 smithi076 conmon[56514]: osd.7) 70 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:24:31.334 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:31 smithi076 conmon[56514]: cluster 2021-03-15T22:24:31.079380+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2021-03-15T22:24:31.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:31 smithi076 conmon[62156]: cluster 2021-03-15T22:24:30 2021-03-15T22:24:31.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:31 smithi076 conmon[62156]: .068545+0000 osd.7 (osd.7) 70 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:24:31.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:31 smithi076 conmon[62156]: cluster 2021-03-15T22:24: 2021-03-15T22:24:31.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:31 smithi076 conmon[62156]: 31.079380+0000 mon.a (mon.0) 1116 2021-03-15T22:24:31.336 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:31 smithi076 conmon[62156]: : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2021-03-15T22:24:31.360 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:31.382 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:31.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:31.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:31.485 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:31.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:31.512 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:31.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:31.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:31 smithi125 conmon[56739]: cluster 2021-03-15T22:24:30.068545 2021-03-15T22:24:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:31 smithi125 conmon[56739]: +0000 osd.7 (osd.7) 70 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:24:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:31 smithi125 conmon[56739]: cluster 2021-03-15T22:24:31.079380+0000 mon.a (mon.0) 1116 : cluster [DBG] osdmap e356: 8 total, 8 up, 7 in 2021-03-15T22:24:31.614 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:31.642 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:31.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:31.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:31.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:31.767 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:31.767 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:31.868 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:31.870 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-15T22:24:31.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:31.933 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:31.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:32.034 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:32.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:32.058 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:32.058 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:32.105 INFO:tasks.rados.rados.0.smithi076.stdout:1259: done (1 left) 2021-03-15T22:24:32.106 INFO:tasks.rados.rados.0.smithi076.stdout:1260: done (0 left) 2021-03-15T22:24:32.106 INFO:tasks.rados.rados.0.smithi076.stdout:1261: copy_from oid 29 from oid 34 current snap is 132 2021-03-15T22:24:32.106 INFO:tasks.rados.rados.0.smithi076.stdout:1262: snap_create 2021-03-15T22:24:32.108 INFO:tasks.rados.rados.0.smithi076.stdout:1261: finishing copy_from to smithi07694878-29 2021-03-15T22:24:32.108 INFO:tasks.rados.rados.0.smithi076.stdout:1261: got expected ENOENT (src dne) 2021-03-15T22:24:32.108 INFO:tasks.rados.rados.0.smithi076.stdout:1261: finishing copy_from racing read to smithi07694878-29 2021-03-15T22:24:32.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:32.186 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:32.188 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:32.222 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:24:32.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:32 smithi125 conmon[56739]: cluster 2021-03-15T22:24:30.865602+0000 2021-03-15T22:24:32.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:32 smithi125 conmon[56739]: osd.4 (osd.4) 78 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:24:32.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:32 smithi125 conmon[56739]: cluster 2021-03-15T22:24:30.950129+0000 osd.1 (osd.1) 73 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:24:32.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:32 smithi125 conmon[56739]: cluster 2021-03-15T22:24:31.039283+0000 mgr.y (mgr.14144) 574 : cluster [DBG] pgmap v660: 138 pgs: 8 peering, 130 active+clean; 177 MiB data, 557 MiB used, 625 GiB / 626 GiB avail; 9.6 MiB/s rd, 326 KiB/s wr, 23 op/s; 144 B/s, 2 objects/s recovering 2021-03-15T22:24:32.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:32 smithi125 conmon[56739]: cluster 2021-03-15T22:24:31.228267+0000 mon.a (mon.0) 1117 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-15T22:24:32.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:32 smithi125 conmon[56739]: cluster 2021-03-15T22:24:32.078150+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2021-03-15T22:24:32.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:32.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:32.319 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:32.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:32.420 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:32.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:32.452 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:32.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:32.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[56514]: cluster 2021-03-15T22:24:30.865602+0000 osd.4 (osd.4) 78 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:24:32.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[56514]: cluster 2021-03-15T22:24:30.950129+0000 osd.1 (osd.1) 73 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:24:32.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[56514]: cluster 2021-03-15T22:24:31.039283+0000 mgr.y (mgr.14144) 574 : cluster [DBG] pgmap v660: 138 pgs: 8 peering, 130 active+clean; 177 MiB data, 557 MiB used, 625 GiB / 626 GiB avail; 9.6 MiB/s rd, 326 KiB/s wr, 23 op/s; 144 B/s, 2 objects/s recovering 2021-03-15T22:24:32.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[56514]: cluster 2021-03-15T22:24:31.228267+0000 mon.a (mon.0) 1117 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-15T22:24:32.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[56514]: cluster 2021-03-15T22:24:32.078150+0000 mon.a (mon.0) 1118 : cluster [DBG] osdmap e357: 8 total, 8 up, 7 in 2021-03-15T22:24:32.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: cluster 2021-03-15T22:24:30.865602+0000 osd.4 (osd 2021-03-15T22:24:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: .4) 78 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:24:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: cluster 2021-03-15T22:24:30.950129+0000 osd.1 (osd 2021-03-15T22:24:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: .1) 73 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:24:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:24:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: T22:24:31.039283+0000 mgr.y (mgr.14144) 574 : cluster [DBG] pgmap v660: 138 pgs: 8 peering, 130 active+clean; 177 MiB data, 557 MiB used, 625 GiB / 626 GiB avail; 9.6 MiB/s rd, 326 KiB/s wr, 23 op/s; 144 B/s, 2 objects/s recovering 2021-03-15T22:24:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: cluster 2021-03-15T22:24: 2021-03-15T22:24:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: 31.228267+0000 mon.a (mon.0) 1117 : cluster [WRN] 2021-03-15T22:24:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: Health check failed: Reduced data availability: 1 pg inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-15T22:24:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: cluster 2021-03-15T22:24:32.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: 2021-03-15T22:24:32.078150+0000 mon.a (mon.0) 1118 : cluster 2021-03-15T22:24:32.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:32 smithi076 conmon[62156]: [DBG] osdmap e357: 8 total, 8 up, 7 in 2021-03-15T22:24:32.554 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:32.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:32.577 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:32.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:32.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:32.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:32.707 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:32.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:32.809 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:32.831 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:32.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:32.832 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:32.836 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:24:32.853 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:24:32.870 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:24:32.887 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:24:32.905 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:24:32.922 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:24:32.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:32.939 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:24:32.955 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:32.956 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:24:32.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:32.957 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:32.958 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:24:32.958 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:24:32.958 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:24:32.958 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:24:32.959 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:24:32.959 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:24:32.959 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:24:32.959 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:24:33.058 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:33.081 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:33.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:33.083 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:33.089 INFO:tasks.rados.rados.0.smithi076.stdout:1261: done (1 left) 2021-03-15T22:24:33.090 INFO:tasks.rados.rados.0.smithi076.stdout:1262: done (0 left) 2021-03-15T22:24:33.090 INFO:tasks.rados.rados.0.smithi076.stdout:1263: write (excl) oid 32 current snap is 133 2021-03-15T22:24:33.090 INFO:tasks.rados.rados.0.smithi076.stdout:1263: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2021-03-15T22:24:33.091 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:24:33.092 INFO:tasks.rados.rados.0.smithi076.stdout:1263: writing smithi07694878-32 from 447185 to 1150307 tid 1 2021-03-15T22:24:33.100 INFO:tasks.rados.rados.0.smithi076.stdout:1263: writing smithi07694878-32 from 1642601 to 2408415 tid 2 2021-03-15T22:24:33.105 INFO:tasks.rados.rados.0.smithi076.stdout:1263: writing smithi07694878-32 from 2928991 to 3376698 tid 3 2021-03-15T22:24:33.106 INFO:tasks.rados.rados.0.smithi076.stdout:1264: read oid 34 snap -1 2021-03-15T22:24:33.106 INFO:tasks.rados.rados.0.smithi076.stdout:1264: expect deleted 2021-03-15T22:24:33.106 INFO:tasks.rados.rados.0.smithi076.stdout:1265: write (excl) oid 25 current snap is 133 2021-03-15T22:24:33.107 INFO:tasks.rados.rados.0.smithi076.stdout:1263: finishing write tid 1 to smithi07694878-32 2021-03-15T22:24:33.107 INFO:tasks.rados.rados.0.smithi076.stdout:1265: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2021-03-15T22:24:33.112 INFO:tasks.rados.rados.0.smithi076.stdout:1265: writing smithi07694878-25 from 463992 to 1242363 tid 1 2021-03-15T22:24:33.118 INFO:tasks.rados.rados.0.smithi076.stdout:1265: writing smithi07694878-25 from 1701083 to 2326908 tid 2 2021-03-15T22:24:33.126 INFO:tasks.rados.rados.0.smithi076.stdout:1265: writing smithi07694878-25 from 2956414 to 3615393 tid 3 2021-03-15T22:24:33.126 INFO:tasks.rados.rados.0.smithi076.stdout:1266: read oid 15 snap -1 2021-03-15T22:24:33.127 INFO:tasks.rados.rados.0.smithi076.stdout:1266: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:24:33.127 INFO:tasks.rados.rados.0.smithi076.stdout:1267: read oid 26 snap -1 2021-03-15T22:24:33.127 INFO:tasks.rados.rados.0.smithi076.stdout:1267: expect (ObjNum 322 snap 87 seq_num 322) 2021-03-15T22:24:33.127 INFO:tasks.rados.rados.0.smithi076.stdout:1268: delete oid 44 current snap is 133 2021-03-15T22:24:33.128 INFO:tasks.rados.rados.0.smithi076.stdout:1263: finishing write tid 2 to smithi07694878-32 2021-03-15T22:24:33.128 INFO:tasks.rados.rados.0.smithi076.stdout:1263: finishing write tid 3 to smithi07694878-32 2021-03-15T22:24:33.128 INFO:tasks.rados.rados.0.smithi076.stdout:1263: finishing write tid 4 to smithi07694878-32 2021-03-15T22:24:33.128 INFO:tasks.rados.rados.0.smithi076.stdout:1263: finishing write tid 5 to smithi07694878-32 2021-03-15T22:24:33.128 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 744 (ObjNum 455 snap 133 seq_num 455) dirty exists 2021-03-15T22:24:33.129 INFO:tasks.rados.rados.0.smithi076.stdout:1263: left oid 32 (ObjNum 455 snap 133 seq_num 455) 2021-03-15T22:24:33.129 INFO:tasks.rados.rados.0.smithi076.stdout:1263: done (5 left) 2021-03-15T22:24:33.130 INFO:tasks.rados.rados.0.smithi076.stdout:1268: done (4 left) 2021-03-15T22:24:33.130 INFO:tasks.rados.rados.0.smithi076.stdout:1269: copy_from oid 10 from oid 37 current snap is 133 2021-03-15T22:24:33.130 INFO:tasks.rados.rados.0.smithi076.stdout:1270: write (excl) oid 49 current snap is 133 2021-03-15T22:24:33.130 INFO:tasks.rados.rados.0.smithi076.stdout:1270: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2021-03-15T22:24:33.133 INFO:tasks.rados.rados.0.smithi076.stdout:1270: writing smithi07694878-49 from 497606 to 1142828 tid 1 2021-03-15T22:24:33.141 INFO:tasks.rados.rados.0.smithi076.stdout:1270: writing smithi07694878-49 from 1934400 to 2563894 tid 2 2021-03-15T22:24:33.148 INFO:tasks.rados.rados.0.smithi076.stdout:1270: writing smithi07694878-49 from 3243966 to 3697606 tid 3 2021-03-15T22:24:33.148 INFO:tasks.rados.rados.0.smithi076.stdout:1271: read oid 32 snap 119 2021-03-15T22:24:33.148 INFO:tasks.rados.rados.0.smithi076.stdout:1271: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:24:33.148 INFO:tasks.rados.rados.0.smithi076.stdout:1272: copy_from oid 31 from oid 2 current snap is 133 2021-03-15T22:24:33.149 INFO:tasks.rados.rados.0.smithi076.stdout:1267: expect (ObjNum 322 snap 87 seq_num 322) 2021-03-15T22:24:33.184 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:33.205 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:33.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:33.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:33.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[62156]: cluster 2021-03-15T22:24:32.078360+0000 osd.4 ( 2021-03-15T22:24:33.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[62156]: osd.4) 79 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 152'4 2021-03-15T22:24:33.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[62156]: audit 2021-03-15T22:24:32.634296+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.76:0/2969297376' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:33.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[62156]: audit 2021-03-15T22:24:32.634611+0000 mon.a (mon.0) 1119 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:33.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[62156]: audit 2021-03-15T22:24:32.815609+0000 mon.a (mon.0) 1120 : audit [DBG] from='client.? 172.21.15.76:0/843547367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:33.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[62156]: cluster 2021-03-15T22:24:33.075423+0000 mon.a (mon.0) 1121 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:24:33.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[62156]: audit 2021-03-15T22:24:33.081311+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:33.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[62156]: cluster 2021-03-15T22:24:33.081423+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2021-03-15T22:24:33.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[56514]: cluster 2021-03-15T22:24:32.078360+0000 osd.4 (osd. 2021-03-15T22:24:33.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[56514]: 4) 79 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 152'4 2021-03-15T22:24:33.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[56514]: audit 2021-03-15T22:24:32.634296+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.76:0/2969297376' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:33.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[56514]: audit 2021-03-15T22:24:32.634611+0000 mon.a (mon.0) 1119 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:33.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[56514]: audit 2021-03-15T22:24:32.815609+0000 mon.a (mon.0) 1120 : audit [DBG] from='client.? 172.21.15.76:0/843547367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:33.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[56514]: cluster 2021-03-15T22:24:33.075423+0000 mon.a (mon.0) 1121 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:24:33.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[56514]: audit 2021-03-15T22:24:33.081311+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:33.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:33 smithi076 conmon[56514]: cluster 2021-03-15T22:24:33.081423+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2021-03-15T22:24:33.305 INFO:tasks.rados.rados.0.smithi076.stdout:1271: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:24:33.308 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:33.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:33.332 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:33.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:33.431 INFO:tasks.rados.rados.0.smithi076.stdout:1267: done (7 left) 2021-03-15T22:24:33.432 INFO:tasks.rados.rados.0.smithi076.stdout:1271: done (6 left) 2021-03-15T22:24:33.432 INFO:tasks.rados.rados.0.smithi076.stdout:1273: copy_from oid 32 from oid 12 current snap is 133 2021-03-15T22:24:33.432 INFO:tasks.rados.rados.0.smithi076.stdout:1274: snap_remove snap 130 2021-03-15T22:24:33.433 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:33.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:33.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:33.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:33 smithi125 conmon[56739]: cluster 2021-03-15T22:24:32.078360+0000 osd.4 (osd.4) 79 : cluster [DBG] 5.e starting backfill to osd.2 from (0'0,0'0] MAX to 152'4 2021-03-15T22:24:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:33 smithi125 conmon[56739]: audit 2021-03-15T22:24:32.634296+0000 mon.c (mon.1) 33 : audit [INF] from='client.? 172.21.15.76:0/2969297376' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:33 smithi125 conmon[56739]: audit 2021-03-15T22:24:32.634611+0000 mon.a (mon.0) 1119 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:33 smithi125 conmon[56739]: audit 2021-03-15T22:24:32.815609+0000 mon.a (mon.0) 1120 : audit [DBG] from='client.? 172.21.15.76:0/843547367' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:33 smithi125 conmon[56739]: cluster 2021-03-15T22:24:33.075423+0000 mon.a (mon.0) 1121 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:24:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:33 smithi125 conmon[56739]: audit 2021-03-15T22:24:33.081311+0000 mon.a (mon.0) 1122 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:33 smithi125 conmon[56739]: cluster 2021-03-15T22:24:33.081423+0000 mon.a (mon.0) 1123 : cluster [DBG] osdmap e358: 8 total, 8 up, 7 in 2021-03-15T22:24:33.556 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:33.579 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:33.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:33.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:33.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:33.703 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:33.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:33.805 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:33.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:33.832 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:33.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:33.863 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:24:33.863 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:24:33.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:33.957 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:33.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:33.959 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:34.060 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:34.084 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:34.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:34.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:34.089 INFO:tasks.rados.rados.0.smithi076.stdout:1274: done (7 left) 2021-03-15T22:24:34.089 INFO:tasks.rados.rados.0.smithi076.stdout:1275: write (excl) oid 40 current snap is 133 2021-03-15T22:24:34.089 INFO:tasks.rados.rados.0.smithi076.stdout:1275: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2021-03-15T22:24:34.096 INFO:tasks.rados.rados.0.smithi076.stdout:1275: writing smithi07694878-40 from 514413 to 1234884 tid 1 2021-03-15T22:24:34.105 INFO:tasks.rados.rados.0.smithi076.stdout:1275: writing smithi07694878-40 from 1876529 to 2649681 tid 2 2021-03-15T22:24:34.112 INFO:tasks.rados.rados.0.smithi076.stdout:1275: writing smithi07694878-40 from 3438683 to 3714413 tid 3 2021-03-15T22:24:34.112 INFO:tasks.rados.rados.0.smithi076.stdout:1276: write oid 24 current snap is 133 2021-03-15T22:24:34.112 INFO:tasks.rados.rados.0.smithi076.stdout:1269: finishing copy_from to smithi07694878-10 2021-03-15T22:24:34.112 INFO:tasks.rados.rados.0.smithi076.stdout:1269: got expected ENOENT (src dne) 2021-03-15T22:24:34.113 INFO:tasks.rados.rados.0.smithi076.stdout:1269: finishing copy_from racing read to smithi07694878-10 2021-03-15T22:24:34.113 INFO:tasks.rados.rados.0.smithi076.stdout:1276: seq_num 460 ranges {531220=795720,1935011=516810,3066106=420420} 2021-03-15T22:24:34.119 INFO:tasks.rados.rados.0.smithi076.stdout:1276: writing smithi07694878-24 from 531220 to 1326940 tid 1 2021-03-15T22:24:34.126 INFO:tasks.rados.rados.0.smithi076.stdout:1276: writing smithi07694878-24 from 1935011 to 2451821 tid 2 2021-03-15T22:24:34.132 INFO:tasks.rados.rados.0.smithi076.stdout:1276: writing smithi07694878-24 from 3066106 to 3486526 tid 3 2021-03-15T22:24:34.132 INFO:tasks.rados.rados.0.smithi076.stdout:1264: done (8 left) 2021-03-15T22:24:34.132 INFO:tasks.rados.rados.0.smithi076.stdout:1269: done (7 left) 2021-03-15T22:24:34.132 INFO:tasks.rados.rados.0.smithi076.stdout:1277: snap_create 2021-03-15T22:24:34.133 INFO:tasks.rados.rados.0.smithi076.stdout:1265: finishing write tid 1 to smithi07694878-25 2021-03-15T22:24:34.133 INFO:tasks.rados.rados.0.smithi076.stdout:1266: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:24:34.186 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:34.212 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:34.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:34.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:34.266 INFO:tasks.rados.rados.0.smithi076.stdout:1265: finishing write tid 2 to smithi07694878-25 2021-03-15T22:24:34.266 INFO:tasks.rados.rados.0.smithi076.stdout:1265: finishing write tid 3 to smithi07694878-25 2021-03-15T22:24:34.266 INFO:tasks.rados.rados.0.smithi076.stdout:1265: finishing write tid 4 to smithi07694878-25 2021-03-15T22:24:34.267 INFO:tasks.rados.rados.0.smithi076.stdout:1265: finishing write tid 5 to smithi07694878-25 2021-03-15T22:24:34.267 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 255 (ObjNum 456 snap 133 seq_num 456) dirty exists 2021-03-15T22:24:34.267 INFO:tasks.rados.rados.0.smithi076.stdout:1265: left oid 25 (ObjNum 456 snap 133 seq_num 456) 2021-03-15T22:24:34.267 INFO:tasks.rados.rados.0.smithi076.stdout:1272: finishing copy_from to smithi07694878-31 2021-03-15T22:24:34.267 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 104 (ObjNum 422 snap 116 seq_num 422) dirty exists 2021-03-15T22:24:34.268 INFO:tasks.rados.rados.0.smithi076.stdout:1270: finishing write tid 1 to smithi07694878-49 2021-03-15T22:24:34.268 INFO:tasks.rados.rados.0.smithi076.stdout:1272: finishing copy_from racing read to smithi07694878-31 2021-03-15T22:24:34.268 INFO:tasks.rados.rados.0.smithi076.stdout:1270: finishing write tid 2 to smithi07694878-49 2021-03-15T22:24:34.268 INFO:tasks.rados.rados.0.smithi076.stdout:1273: finishing copy_from to smithi07694878-32 2021-03-15T22:24:34.269 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 645 (ObjNum 430 snap 119 seq_num 430) dirty exists 2021-03-15T22:24:34.269 INFO:tasks.rados.rados.0.smithi076.stdout:1270: finishing write tid 3 to smithi07694878-49 2021-03-15T22:24:34.269 INFO:tasks.rados.rados.0.smithi076.stdout:1273: finishing copy_from racing read to smithi07694878-32 2021-03-15T22:24:34.269 INFO:tasks.rados.rados.0.smithi076.stdout:1270: finishing write tid 4 to smithi07694878-49 2021-03-15T22:24:34.269 INFO:tasks.rados.rados.0.smithi076.stdout:1275: finishing write tid 1 to smithi07694878-40 2021-03-15T22:24:34.270 INFO:tasks.rados.rados.0.smithi076.stdout:1270: finishing write tid 5 to smithi07694878-49 2021-03-15T22:24:34.270 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 714 (ObjNum 458 snap 133 seq_num 458) dirty exists 2021-03-15T22:24:34.270 INFO:tasks.rados.rados.0.smithi076.stdout:1270: left oid 49 (ObjNum 458 snap 133 seq_num 458) 2021-03-15T22:24:34.271 INFO:tasks.rados.rados.0.smithi076.stdout:1276: finishing write tid 1 to smithi07694878-24 2021-03-15T22:24:34.271 INFO:tasks.rados.rados.0.smithi076.stdout:1275: finishing write tid 2 to smithi07694878-40 2021-03-15T22:24:34.271 INFO:tasks.rados.rados.0.smithi076.stdout:1275: finishing write tid 3 to smithi07694878-40 2021-03-15T22:24:34.271 INFO:tasks.rados.rados.0.smithi076.stdout:1275: finishing write tid 4 to smithi07694878-40 2021-03-15T22:24:34.272 INFO:tasks.rados.rados.0.smithi076.stdout:1275: finishing write tid 5 to smithi07694878-40 2021-03-15T22:24:34.272 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 752 (ObjNum 459 snap 133 seq_num 459) dirty exists 2021-03-15T22:24:34.272 INFO:tasks.rados.rados.0.smithi076.stdout:1275: left oid 40 (ObjNum 459 snap 133 seq_num 459) 2021-03-15T22:24:34.272 INFO:tasks.rados.rados.0.smithi076.stdout:1276: finishing write tid 2 to smithi07694878-24 2021-03-15T22:24:34.272 INFO:tasks.rados.rados.0.smithi076.stdout:1276: finishing write tid 3 to smithi07694878-24 2021-03-15T22:24:34.273 INFO:tasks.rados.rados.0.smithi076.stdout:1276: finishing write tid 4 to smithi07694878-24 2021-03-15T22:24:34.273 INFO:tasks.rados.rados.0.smithi076.stdout:1276: finishing write tid 5 to smithi07694878-24 2021-03-15T22:24:34.273 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 91 (ObjNum 460 snap 133 seq_num 460) dirty exists 2021-03-15T22:24:34.273 INFO:tasks.rados.rados.0.smithi076.stdout:1276: left oid 24 (ObjNum 460 snap 133 seq_num 460) 2021-03-15T22:24:34.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:34.337 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:34.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:34.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:34.439 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:34.461 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:34.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:34.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:34.517 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:24:34 smithi076 conmon[75764]: debug 2021-03-15T22:24:34.065+0000 7f9fba7ae700 -1 osd.3 pg_epoch: 358 pg[3.b( empty local-lis/les=355/356 n=0 ec=56/56 lis/c=355/355 les/c/f=356/356/0 sis=355) [3,4,1] r=0 lpr=355 crt=0'0 mlcod 0'0 active+clean] scrubber pg(3.b) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 4 (0x5654efab2c60) 2021-03-15T22:24:34.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:34 smithi076 conmon[56514]: cluster 2021-03-15T22:24:33.039801+0000 mgr.y (mgr.14144) 575 : cluster [DBG] pgmap v663: 138 pgs: 1 active+undersized+remapped, 2 remapped+peering, 8 peering, 127 active+clean; 186 MiB data, 576 MiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 2.2 MiB/s wr, 8 op/s; 1.1 MiB/s, 14 objects/s recovering 2021-03-15T22:24:34.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:34 smithi076 conmon[56514]: cluster 2021-03-15T22:24:33.456993+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:34.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:34 smithi076 conmon[56514]: cluster 2021-03-15T22:24:33.457286+0000 osd.6 (osd.6) 54 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:34.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:34 smithi076 conmon[56514]: cluster 2021-03-15T22:24:34.083320+0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2021-03-15T22:24:34.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:34 smithi076 conmon[62156]: cluster 2021-03-15T22:24:33.039801+0000 mgr.y (mgr.14144) 575 : cluster [DBG] pgmap v663: 138 pgs: 1 active+undersized+remapped, 2 remapped+peering, 8 peering, 127 active+clean; 186 MiB data, 576 MiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 2.2 MiB/s wr, 8 op/s; 1.1 MiB/s, 14 objects/s recovering 2021-03-15T22:24:34.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:34 smithi076 conmon[62156]: cluster 2021-03-15T22:24:33.456993+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:34.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:34 smithi076 conmon[62156]: cluster 2021-03-15T22:24:33.457286+0000 osd.6 (osd.6) 54 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:34.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:34 smithi076 conmon[62156]: cluster 2021-03-15T22:24:34.083320+0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2021-03-15T22:24:34.521 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:34 smithi125 conmon[56739]: cluster 2021-03-15T22:24:33.039801+0000 mgr.y (mgr.14144) 575 : cluster [DBG] pgmap v663: 138 pgs: 1 active+undersized+remapped, 2 remapped+peering, 8 peering, 127 active+clean; 186 MiB data, 576 MiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 2.2 MiB/s wr, 8 op/s; 1.1 MiB/s, 14 objects/s recovering 2021-03-15T22:24:34.521 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:34 smithi125 conmon[56739]: cluster 2021-03-15T22:24:33.456993+0000 osd.6 (osd.6) 53 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:34.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:34 smithi125 conmon[56739]: cluster 2021-03-15T22:24:33.457286+0000 osd.6 (osd.6) 54 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:34.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:34 smithi125 conmon[56739]: cluster 2021-03-15T22:24:34.083320+0000 mon.a (mon.0) 1124 : cluster [DBG] osdmap e359: 8 total, 8 up, 7 in 2021-03-15T22:24:34.544 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:24:34.564 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:34.586 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:34.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:34.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:34.689 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:34.710 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:34.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:34.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:34.813 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:34.818 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:34.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:34.838 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:34.838 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:34.939 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:34.962 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:34.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:34.963 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:35.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:35.083 INFO:tasks.rados.rados.0.smithi076.stdout:1265: done (7 left) 2021-03-15T22:24:35.084 INFO:tasks.rados.rados.0.smithi076.stdout:1266: done (6 left) 2021-03-15T22:24:35.084 INFO:tasks.rados.rados.0.smithi076.stdout:1270: done (5 left) 2021-03-15T22:24:35.084 INFO:tasks.rados.rados.0.smithi076.stdout:1272: done (4 left) 2021-03-15T22:24:35.084 INFO:tasks.rados.rados.0.smithi076.stdout:1273: done (3 left) 2021-03-15T22:24:35.085 INFO:tasks.rados.rados.0.smithi076.stdout:1275: done (2 left) 2021-03-15T22:24:35.086 INFO:tasks.rados.rados.0.smithi076.stdout:1276: done (1 left) 2021-03-15T22:24:35.088 INFO:tasks.rados.rados.0.smithi076.stdout:1277: done (0 left) 2021-03-15T22:24:35.088 INFO:tasks.rados.rados.0.smithi076.stdout:1278: write (excl) oid 28 current snap is 134 2021-03-15T22:24:35.088 INFO:tasks.rados.rados.0.smithi076.stdout:1278: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2021-03-15T22:24:35.089 INFO:tasks.rados.rados.0.smithi076.stdout:1278: writing smithi07694878-28 from 548027 to 1018996 tid 1 2021-03-15T22:24:35.093 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:35.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:35.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:35.096 INFO:tasks.rados.rados.0.smithi076.stdout:1278: writing smithi07694878-28 from 1593493 to 2370314 tid 2 2021-03-15T22:24:35.105 INFO:tasks.rados.rados.0.smithi076.stdout:1278: writing smithi07694878-28 from 3093529 to 3725221 tid 3 2021-03-15T22:24:35.106 INFO:tasks.rados.rados.0.smithi076.stdout:1279: rollback oid 20 current snap is 134 2021-03-15T22:24:35.106 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 20 to 132 2021-03-15T22:24:35.106 INFO:tasks.rados.rados.0.smithi076.stdout:1278: finishing write tid 1 to smithi07694878-28 2021-03-15T22:24:35.107 INFO:tasks.rados.rados.0.smithi076.stdout:1278: finishing write tid 2 to smithi07694878-28 2021-03-15T22:24:35.107 INFO:tasks.rados.rados.0.smithi076.stdout:1280: write (excl) oid 1 current snap is 134 2021-03-15T22:24:35.107 INFO:tasks.rados.rados.0.smithi076.stdout:1280: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2021-03-15T22:24:35.110 INFO:tasks.rados.rados.0.smithi076.stdout:1280: writing smithi07694878-1 from 564834 to 1111052 tid 1 2021-03-15T22:24:35.116 INFO:tasks.rados.rados.0.smithi076.stdout:1280: writing smithi07694878-1 from 1651975 to 2172454 tid 2 2021-03-15T22:24:35.123 INFO:tasks.rados.rados.0.smithi076.stdout:1280: writing smithi07694878-1 from 2720952 to 3280269 tid 3 2021-03-15T22:24:35.124 INFO:tasks.rados.rados.0.smithi076.stdout:1281: rollback oid 30 current snap is 134 2021-03-15T22:24:35.124 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 30 to 117 2021-03-15T22:24:35.125 INFO:tasks.rados.rados.0.smithi076.stdout:1279: finishing rollback tid 0 to smithi07694878-20 2021-03-15T22:24:35.125 INFO:tasks.rados.rados.0.smithi076.stdout:1279: finishing rollback tid 1 to smithi07694878-20 2021-03-15T22:24:35.126 INFO:tasks.rados.rados.0.smithi076.stdout:1278: finishing write tid 3 to smithi07694878-28 2021-03-15T22:24:35.126 INFO:tasks.rados.rados.0.smithi076.stdout:1278: finishing write tid 4 to smithi07694878-28 2021-03-15T22:24:35.127 INFO:tasks.rados.rados.0.smithi076.stdout:1279: finishing rollback tid 2 to smithi07694878-20 2021-03-15T22:24:35.127 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 447 (ObjNum 449 snap 128 seq_num 449) dirty exists 2021-03-15T22:24:35.127 INFO:tasks.rados.rados.0.smithi076.stdout:1278: finishing write tid 5 to smithi07694878-28 2021-03-15T22:24:35.128 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 757 (ObjNum 461 snap 134 seq_num 461) dirty exists 2021-03-15T22:24:35.128 INFO:tasks.rados.rados.0.smithi076.stdout:1278: left oid 28 (ObjNum 461 snap 134 seq_num 461) 2021-03-15T22:24:35.129 INFO:tasks.rados.rados.0.smithi076.stdout:1278: done (3 left) 2021-03-15T22:24:35.129 INFO:tasks.rados.rados.0.smithi076.stdout:1279: done (2 left) 2021-03-15T22:24:35.130 INFO:tasks.rados.rados.0.smithi076.stdout:1282: delete oid 19 current snap is 134 2021-03-15T22:24:35.131 INFO:tasks.rados.rados.0.smithi076.stdout:1282: done (2 left) 2021-03-15T22:24:35.132 INFO:tasks.rados.rados.0.smithi076.stdout:1283: snap_create 2021-03-15T22:24:35.132 INFO:tasks.rados.rados.0.smithi076.stdout:1281: finishing rollback tid 0 to smithi07694878-30 2021-03-15T22:24:35.132 INFO:tasks.rados.rados.0.smithi076.stdout:1281: finishing rollback tid 1 to smithi07694878-30 2021-03-15T22:24:35.133 INFO:tasks.rados.rados.0.smithi076.stdout:1281: finishing rollback tid 2 to smithi07694878-30 2021-03-15T22:24:35.133 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 718 (ObjNum 412 snap 112 seq_num 412) dirty exists 2021-03-15T22:24:35.196 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:35.218 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:35.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:35.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:35.261 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:35 smithi076 conmon[56514]: cluster 2021-03-15T22:24:34.454309+0000 osd.6 2021-03-15T22:24:35.261 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:35 smithi076 conmon[56514]: (osd.6) 55 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:24:35.261 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:35 smithi076 conmon[56514]: cluster 2021-03-15T22:24:35.079255+0000 mon.a (mon.0) 1125 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2021-03-15T22:24:35.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:35 smithi076 conmon[62156]: cluster 2021-03-15T22:24:34.454309+0000 osd.6 (osd.6 2021-03-15T22:24:35.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:35 smithi076 conmon[62156]: ) 55 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:24:35.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:35 smithi076 conmon[62156]: cluster 2021-03-15T22:24:35.079255+0000 mon.a (mon.0) 1125 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2021-03-15T22:24:35.320 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:35.343 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:35.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:35.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:35.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:35.469 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:35.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:35.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:35.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:35 smithi125 conmon[56739]: cluster 2021-03-15T22:24:34.454309+0000 osd.6 (osd.6) 55 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:24:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:35 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:24:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:35 smithi125 conmon[56739]: 15T22:24:35.079255+0000 mon.a (mon.0) 1125 : cluster [DBG] osdmap e360: 8 total, 8 up, 7 in 2021-03-15T22:24:35.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:35.594 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:35.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:35.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:35.697 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:35.720 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:35.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:35.722 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:35.823 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:35.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:35.847 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:35.847 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:35.949 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:35.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:35.972 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:35.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:36.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:36.089 INFO:tasks.rados.rados.0.smithi076.stdout:1281: done (2 left) 2021-03-15T22:24:36.089 INFO:tasks.rados.rados.0.smithi076.stdout:1283: done (1 left) 2021-03-15T22:24:36.090 INFO:tasks.rados.rados.0.smithi076.stdout:1284: rollback oid 30 current snap is 135 2021-03-15T22:24:36.090 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 30 to 133 2021-03-15T22:24:36.090 INFO:tasks.rados.rados.0.smithi076.stdout:1285: rollback oid 20 current snap is 135 2021-03-15T22:24:36.090 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 20 to 133 2021-03-15T22:24:36.091 INFO:tasks.rados.rados.0.smithi076.stdout:1286: write (excl) oid 24 current snap is 135 2021-03-15T22:24:36.091 INFO:tasks.rados.rados.0.smithi076.stdout:1286: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2021-03-15T22:24:36.095 INFO:tasks.rados.rados.0.smithi076.stdout:1286: writing smithi07694878-24 from 598448 to 1011517 tid 1 2021-03-15T22:24:36.102 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:36.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:36.104 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:36.105 INFO:tasks.rados.rados.0.smithi076.stdout:1286: writing smithi07694878-24 from 1768939 to 2293087 tid 2 2021-03-15T22:24:36.109 INFO:tasks.rados.rados.0.smithi076.stdout:1286: writing smithi07694878-24 from 2775798 to 3357659 tid 3 2021-03-15T22:24:36.112 INFO:tasks.rados.rados.0.smithi076.stdout:1286: writing smithi07694878-24 from 3781488 to 3798448 tid 4 2021-03-15T22:24:36.112 INFO:tasks.rados.rados.0.smithi076.stdout:1287: write oid 27 current snap is 135 2021-03-15T22:24:36.112 INFO:tasks.rados.rados.0.smithi076.stdout:1284: finishing rollback tid 0 to smithi07694878-30 2021-03-15T22:24:36.112 INFO:tasks.rados.rados.0.smithi076.stdout:1287: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2021-03-15T22:24:36.117 INFO:tasks.rados.rados.0.smithi076.stdout:1287: writing smithi07694878-27 from 615255 to 1103573 tid 1 2021-03-15T22:24:36.126 INFO:tasks.rados.rados.0.smithi076.stdout:1287: writing smithi07694878-27 from 1827421 to 2611580 tid 2 2021-03-15T22:24:36.134 INFO:tasks.rados.rados.0.smithi076.stdout:1287: writing smithi07694878-27 from 3203221 to 3815255 tid 3 2021-03-15T22:24:36.135 INFO:tasks.rados.rados.0.smithi076.stdout:1288: read oid 44 snap -1 2021-03-15T22:24:36.135 INFO:tasks.rados.rados.0.smithi076.stdout:1288: expect deleted 2021-03-15T22:24:36.135 INFO:tasks.rados.rados.0.smithi076.stdout:1289: rollback oid 25 current snap is 135 2021-03-15T22:24:36.135 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 129 2021-03-15T22:24:36.136 INFO:tasks.rados.rados.0.smithi076.stdout:1284: finishing rollback tid 1 to smithi07694878-30 2021-03-15T22:24:36.136 INFO:tasks.rados.rados.0.smithi076.stdout:1284: finishing rollback tid 2 to smithi07694878-30 2021-03-15T22:24:36.136 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 722 (ObjNum 445 snap 126 seq_num 445) dirty exists 2021-03-15T22:24:36.136 INFO:tasks.rados.rados.0.smithi076.stdout:1280: finishing write tid 1 to smithi07694878-1 2021-03-15T22:24:36.136 INFO:tasks.rados.rados.0.smithi076.stdout:1285: finishing rollback tid 0 to smithi07694878-20 2021-03-15T22:24:36.137 INFO:tasks.rados.rados.0.smithi076.stdout:1285: finishing rollback tid 1 to smithi07694878-20 2021-03-15T22:24:36.137 INFO:tasks.rados.rados.0.smithi076.stdout:1285: finishing rollback tid 2 to smithi07694878-20 2021-03-15T22:24:36.137 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 450 (ObjNum 449 snap 128 seq_num 449) dirty exists 2021-03-15T22:24:36.137 INFO:tasks.rados.rados.0.smithi076.stdout:1280: finishing write tid 2 to smithi07694878-1 2021-03-15T22:24:36.138 INFO:tasks.rados.rados.0.smithi076.stdout:1280: finishing write tid 3 to smithi07694878-1 2021-03-15T22:24:36.138 INFO:tasks.rados.rados.0.smithi076.stdout:1280: finishing write tid 4 to smithi07694878-1 2021-03-15T22:24:36.138 INFO:tasks.rados.rados.0.smithi076.stdout:1280: finishing write tid 5 to smithi07694878-1 2021-03-15T22:24:36.138 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 173 (ObjNum 462 snap 134 seq_num 462) dirty exists 2021-03-15T22:24:36.139 INFO:tasks.rados.rados.0.smithi076.stdout:1280: left oid 1 (ObjNum 462 snap 134 seq_num 462) 2021-03-15T22:24:36.139 INFO:tasks.rados.rados.0.smithi076.stdout:1286: finishing write tid 1 to smithi07694878-24 2021-03-15T22:24:36.139 INFO:tasks.rados.rados.0.smithi076.stdout:1286: finishing write tid 2 to smithi07694878-24 2021-03-15T22:24:36.139 INFO:tasks.rados.rados.0.smithi076.stdout:1286: finishing write tid 3 to smithi07694878-24 2021-03-15T22:24:36.139 INFO:tasks.rados.rados.0.smithi076.stdout:1280: done (6 left) 2021-03-15T22:24:36.140 INFO:tasks.rados.rados.0.smithi076.stdout:1284: done (5 left) 2021-03-15T22:24:36.140 INFO:tasks.rados.rados.0.smithi076.stdout:1285: done (4 left) 2021-03-15T22:24:36.140 INFO:tasks.rados.rados.0.smithi076.stdout:1290: snap_create 2021-03-15T22:24:36.140 INFO:tasks.rados.rados.0.smithi076.stdout:1286: finishing write tid 4 to smithi07694878-24 2021-03-15T22:24:36.141 INFO:tasks.rados.rados.0.smithi076.stdout:1286: finishing write tid 5 to smithi07694878-24 2021-03-15T22:24:36.141 INFO:tasks.rados.rados.0.smithi076.stdout:1286: finishing write tid 6 to smithi07694878-24 2021-03-15T22:24:36.141 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 97 (ObjNum 464 snap 135 seq_num 464) dirty exists 2021-03-15T22:24:36.141 INFO:tasks.rados.rados.0.smithi076.stdout:1286: left oid 24 (ObjNum 464 snap 135 seq_num 464) 2021-03-15T22:24:36.141 INFO:tasks.rados.rados.0.smithi076.stdout:1287: finishing write tid 1 to smithi07694878-27 2021-03-15T22:24:36.142 INFO:tasks.rados.rados.0.smithi076.stdout:1287: finishing write tid 2 to smithi07694878-27 2021-03-15T22:24:36.143 INFO:tasks.rados.rados.0.smithi076.stdout:1289: finishing rollback tid 0 to smithi07694878-25 2021-03-15T22:24:36.143 INFO:tasks.rados.rados.0.smithi076.stdout:1287: finishing write tid 3 to smithi07694878-27 2021-03-15T22:24:36.143 INFO:tasks.rados.rados.0.smithi076.stdout:1289: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:24:36.143 INFO:tasks.rados.rados.0.smithi076.stdout:1287: finishing write tid 4 to smithi07694878-27 2021-03-15T22:24:36.144 INFO:tasks.rados.rados.0.smithi076.stdout:1289: finishing rollback tid 2 to smithi07694878-25 2021-03-15T22:24:36.144 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 263 (ObjNum 436 snap 122 seq_num 436) dirty exists 2021-03-15T22:24:36.144 INFO:tasks.rados.rados.0.smithi076.stdout:1287: finishing write tid 5 to smithi07694878-27 2021-03-15T22:24:36.144 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 255 (ObjNum 465 snap 135 seq_num 465) dirty exists 2021-03-15T22:24:36.145 INFO:tasks.rados.rados.0.smithi076.stdout:1287: left oid 27 (ObjNum 465 snap 135 seq_num 465) 2021-03-15T22:24:36.206 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:36.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:24:36.228 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:36.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:36.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:36.231 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:24:36.332 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:36.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:36.358 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:36.359 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:36.364 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:24:36.365 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:24:36.365 INFO:tasks.thrashosds.thrasher:inject_pause on 1 2021-03-15T22:24:36.365 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-03-15T22:24:36.366 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-03-15T22:24:36.366 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 config set filestore_inject_stall 3 2021-03-15T22:24:36.388 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:24:36.389 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:24:36.460 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:36.464 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:36 smithi076 conmon[56514]: cluster 2021-03-15T22:24:34.991014+0000 osd.4 (osd 2021-03-15T22:24:36.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:36 smithi076 conmon[56514]: .4) 80 : cluster [DBG] 3.9 scrub ok 2021-03-15T22:24:36.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:36 smithi076 conmon[56514]: cluster 2021-03-15T22:24:35.040567+0000 mgr.y (mgr.14144) 576 : cluster [DBG] pgmap v666: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 7 peering, 127 active+clean; 191 MiB data, 615 MiB used, 625 GiB / 626 GiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 11 op/s; 1/1212 objects degraded (0.083%); 10 MiB/s, 52 objects/s recovering 2021-03-15T22:24:36.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:36 smithi076 conmon[56514]: cluster 2021-03-15T22:24:35.253299+0000 mon.a (mon.0) 1126 : cluster [WRN] Health check failed: Degraded data redundancy: 1/1212 objects degraded (0.083%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:24:36.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:36 smithi076 conmon[56514]: cluster 2021-03-15T22:24:35.715395+0000 osd.5 (osd.5) 65 : cluster [DBG] 5.f scrub ok 2021-03-15T22:24:36.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:36 smithi076 conmon[56514]: cluster 2021-03-15T22:24:36.086755+0000 mon.a (mon.0) 1127 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2021-03-15T22:24:36.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:36 smithi076 conmon[62156]: cluster 2021-03-15T22:24:34.991014+0000 osd.4 (osd.4) 80 : cluster [DBG] 3.9 scrub ok 2021-03-15T22:24:36.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:36 smithi076 conmon[62156]: cluster 2021-03-15T22:24:35.040567+0000 mgr.y (mgr.14144) 576 : cluster [DBG] pgmap v666: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 7 peering, 127 active+clean; 191 MiB data, 615 MiB used, 625 GiB / 626 GiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 11 op/s; 1/1212 objects degraded (0.083%); 10 MiB/s, 52 objects/s recovering 2021-03-15T22:24:36.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:36 smithi076 conmon[62156]: cluster 2021-03-15T22:24:35.253299+0000 mon.a (mon.0) 1126 : cluster [WRN] Health check failed: Degraded data redundancy: 1/1212 objects degraded (0.083%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:24:36.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:36 smithi076 conmon[62156]: cluster 2021-03-15T22:24:35.715395+0000 osd.5 (osd.5) 65 : cluster [DBG] 5.f scrub ok 2021-03-15T22:24:36.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:36 smithi076 conmon[62156]: cluster 2021-03-15T22:24:36.086755+0000 mon.a (mon.0) 1127 : cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2021-03-15T22:24:36.482 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:36.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:36.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:36.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: cluster 2021-03-15T22:24:34.991014+0000 osd.4 (osd 2021-03-15T22:24:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: .4) 80 : cluster [DBG] 3.9 scrub ok 2021-03-15T22:24:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: cluster 2021-03-15T22:24:35.040567+0000 2021-03-15T22:24:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: mgr.y (mgr.14144) 576 : cluster [DBG] pgmap v666: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 remapped+peering, 7 peering, 127 active+clean; 191 MiB data, 615 MiB used, 625 GiB / 626 GiB avail; 3.2 MiB/s rd, 3.8 MiB/s wr, 11 op/s; 1/1212 objects degraded (0.083%); 10 MiB/s, 52 objects/s recovering 2021-03-15T22:24:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:24:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: 22:24:35.253299+0000 mon.a (mon. 2021-03-15T22:24:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: 0) 1126 : cluster [WRN] Health check failed: Degraded data redundancy: 1/1212 objects degraded (0.083%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:24:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:24:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: 03-15T22:24:35.715395+0000 2021-03-15T22:24:36.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: osd.5 (osd.5) 65 : cluster [DBG] 5.f scrub ok 2021-03-15T22:24:36.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: 2021-03-15T22:24:36.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: cluster 2021-03-15T22:24: 2021-03-15T22:24:36.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: 36.086755+0000 mon.a (mon.0) 1127 : 2021-03-15T22:24:36.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:36 smithi125 conmon[56739]: cluster [DBG] osdmap e361: 8 total, 8 up, 7 in 2021-03-15T22:24:36.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:36.607 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:36.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:36.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:36.710 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:36.733 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:36.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:36.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:36.837 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:36.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:36.860 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:36.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:36.962 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:36.985 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:36.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:37.087 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:37.090 INFO:tasks.rados.rados.0.smithi076.stdout:1286: done (4 left) 2021-03-15T22:24:37.090 INFO:tasks.rados.rados.0.smithi076.stdout:1287: done (3 left) 2021-03-15T22:24:37.090 INFO:tasks.rados.rados.0.smithi076.stdout:1288: done (2 left) 2021-03-15T22:24:37.091 INFO:tasks.rados.rados.0.smithi076.stdout:1289: done (1 left) 2021-03-15T22:24:37.091 INFO:tasks.rados.rados.0.smithi076.stdout:1290: done (0 left) 2021-03-15T22:24:37.091 INFO:tasks.rados.rados.0.smithi076.stdout:1291: write oid 22 current snap is 136 2021-03-15T22:24:37.091 INFO:tasks.rados.rados.0.smithi076.stdout:1291: seq_num 466 ranges {632062=563567,1885903=527817,2830644=720758} 2021-03-15T22:24:37.096 INFO:tasks.rados.rados.0.smithi076.stdout:1291: writing smithi07694878-22 from 632062 to 1195629 tid 1 2021-03-15T22:24:37.103 INFO:tasks.rados.rados.0.smithi076.stdout:1291: writing smithi07694878-22 from 1885903 to 2413720 tid 2 2021-03-15T22:24:37.110 INFO:tasks.rados.rados.0.smithi076.stdout:1291: writing smithi07694878-22 from 2830644 to 3551402 tid 3 2021-03-15T22:24:37.110 INFO:tasks.rados.rados.0.smithi076.stdout:1292: read oid 1 snap -1 2021-03-15T22:24:37.110 INFO:tasks.rados.rados.0.smithi076.stdout:1292: expect (ObjNum 462 snap 134 seq_num 462) 2021-03-15T22:24:37.111 INFO:tasks.rados.rados.0.smithi076.stdout:1293: snap_create 2021-03-15T22:24:37.111 INFO:tasks.rados.rados.0.smithi076.stdout:1291: finishing write tid 1 to smithi07694878-22 2021-03-15T22:24:37.111 INFO:tasks.rados.rados.0.smithi076.stdout:1291: finishing write tid 2 to smithi07694878-22 2021-03-15T22:24:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:37.112 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:37.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:37.114 INFO:tasks.rados.rados.0.smithi076.stdout:1291: finishing write tid 3 to smithi07694878-22 2021-03-15T22:24:37.114 INFO:tasks.rados.rados.0.smithi076.stdout:1291: finishing write tid 4 to smithi07694878-22 2021-03-15T22:24:37.114 INFO:tasks.rados.rados.0.smithi076.stdout:1291: finishing write tid 5 to smithi07694878-22 2021-03-15T22:24:37.115 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 727 (ObjNum 466 snap 136 seq_num 466) dirty exists 2021-03-15T22:24:37.115 INFO:tasks.rados.rados.0.smithi076.stdout:1291: left oid 22 (ObjNum 466 snap 136 seq_num 466) 2021-03-15T22:24:37.122 INFO:tasks.rados.rados.0.smithi076.stdout:1292: expect (ObjNum 462 snap 134 seq_num 462) 2021-03-15T22:24:37.179 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:24:37.215 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:37.239 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:37.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:37.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:37.314 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:37.341 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:37.365 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:37.365 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:37.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:37.367 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:37.437 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:37.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:37.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:37.495 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:37.495 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:37.498 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:37 smithi076 conmon[56514]: cluster 2021-03-15T22:24:36.098498+0000 osd.7 (osd.7) 71 : cluster [DBG] 5.14 deep-scrub ok 2021-03-15T22:24:37.499 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:37 smithi076 conmon[56514]: cluster 2021-03-15T22:24:36.674889+0000 osd.5 (osd.5) 66 : cluster [DBG] 6.19 deep-scrub ok 2021-03-15T22:24:37.499 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:37 smithi076 conmon[56514]: cluster 2021-03-15T22:24:37.087557+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2021-03-15T22:24:37.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:37 smithi076 conmon[62156]: cluster 2021-03-15T22:24:36.098498 2021-03-15T22:24:37.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:37 smithi076 conmon[62156]: +0000 osd.7 (osd.7) 71 : cluster [DBG] 5.14 deep-scrub ok 2021-03-15T22:24:37.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:37 smithi076 conmon[62156]: cluster 2021-03-15T22:24:36.674889+0000 osd.5 (osd.5) 66 : cluster [DBG] 6.19 deep-scrub ok 2021-03-15T22:24:37.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:37 smithi076 conmon[62156]: cluster 2021-03-15T22:24:37.087557+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2021-03-15T22:24:37.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:37 smithi125 conmon[56739]: cluster 2021-03-15T22:24:36.098498+0000 osd.7 (osd. 2021-03-15T22:24:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:37 smithi125 conmon[56739]: 7) 71 : cluster [DBG] 5.14 deep-scrub ok 2021-03-15T22:24:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:37 smithi125 conmon[56739]: cluster 2021-03-15T22:24:36.674889+0000 osd.5 (osd.5) 66 : cluster [DBG] 6.19 deep-scrub ok 2021-03-15T22:24:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:37 smithi125 conmon[56739]: cluster 2021-03-15T22:24:37.087557+0000 mon.a (mon.0) 1128 : cluster [DBG] osdmap e362: 8 total, 8 up, 7 in 2021-03-15T22:24:37.597 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:37.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:37.622 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:37.623 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:37.724 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:37.747 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:37.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:37.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:37.873 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:37.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:37.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:37.900 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:37.976 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:38.001 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:38.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:38.093 INFO:tasks.rados.rados.0.smithi076.stdout:1291: done (2 left) 2021-03-15T22:24:38.093 INFO:tasks.rados.rados.0.smithi076.stdout:1292: done (1 left) 2021-03-15T22:24:38.094 INFO:tasks.rados.rados.0.smithi076.stdout:1293: done (0 left) 2021-03-15T22:24:38.094 INFO:tasks.rados.rados.0.smithi076.stdout:1294: delete oid 2 current snap is 137 2021-03-15T22:24:38.099 INFO:tasks.rados.rados.0.smithi076.stdout:1294: done (0 left) 2021-03-15T22:24:38.099 INFO:tasks.rados.rados.0.smithi076.stdout:1295: copy_from oid 27 from oid 19 current snap is 137 2021-03-15T22:24:38.099 INFO:tasks.rados.rados.0.smithi076.stdout:1296: write (excl) oid 44 current snap is 137 2021-03-15T22:24:38.099 INFO:tasks.rados.rados.0.smithi076.stdout:1296: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2021-03-15T22:24:38.104 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:38.107 INFO:tasks.rados.rados.0.smithi076.stdout:1296: writing smithi07694878-44 from 665676 to 1379741 tid 1 2021-03-15T22:24:38.113 INFO:tasks.rados.rados.0.smithi076.stdout:1296: writing smithi07694878-44 from 1886514 to 2418000 tid 2 2021-03-15T22:24:38.124 INFO:tasks.rados.rados.0.smithi076.stdout:1296: writing smithi07694878-44 from 3169137 to 3865676 tid 3 2021-03-15T22:24:38.124 INFO:tasks.rados.rados.0.smithi076.stdout:1297: read oid 37 snap -1 2021-03-15T22:24:38.125 INFO:tasks.rados.rados.0.smithi076.stdout:1297: expect deleted 2021-03-15T22:24:38.126 INFO:tasks.rados.rados.0.smithi076.stdout:1298: read oid 42 snap 134 2021-03-15T22:24:38.126 INFO:tasks.rados.rados.0.smithi076.stdout:1298: expect (ObjNum 442 snap 124 seq_num 442) 2021-03-15T22:24:38.126 INFO:tasks.rados.rados.0.smithi076.stdout:1295: finishing copy_from to smithi07694878-27 2021-03-15T22:24:38.127 INFO:tasks.rados.rados.0.smithi076.stdout:1295: got expected ENOENT (src dne) 2021-03-15T22:24:38.127 INFO:tasks.rados.rados.0.smithi076.stdout:1295: finishing copy_from racing read to smithi07694878-27 2021-03-15T22:24:38.127 INFO:tasks.rados.rados.0.smithi076.stdout:1296: finishing write tid 1 to smithi07694878-44 2021-03-15T22:24:38.127 INFO:tasks.rados.rados.0.smithi076.stdout:1296: finishing write tid 2 to smithi07694878-44 2021-03-15T22:24:38.127 INFO:tasks.rados.rados.0.smithi076.stdout:1295: done (3 left) 2021-03-15T22:24:38.128 INFO:tasks.rados.rados.0.smithi076.stdout:1299: copy_from oid 48 from oid 40 current snap is 137 2021-03-15T22:24:38.128 INFO:tasks.rados.rados.0.smithi076.stdout:1300: read oid 24 snap 136 2021-03-15T22:24:38.128 INFO:tasks.rados.rados.0.smithi076.stdout:1300: expect (ObjNum 464 snap 135 seq_num 464) 2021-03-15T22:24:38.129 INFO:tasks.rados.rados.0.smithi076.stdout:1301: snap_remove snap 129 2021-03-15T22:24:38.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:38.131 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:38.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:38.234 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:38.258 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:38.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:38.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:38.386 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:38.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:38.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:38.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[56514]: cluster 2021-03-15T22:24:37.041075 2021-03-15T22:24:38.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 577 : cluster [DBG] pgmap v669: 138 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 135 active+clean; 193 MiB data, 638 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.4 MiB/s wr, 14 op/s; 1/1224 objects degraded (0.082%); 1/1224 objects misplaced (0.082%); 15 MiB/s, 55 objects/s recovering 2021-03-15T22:24:38.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[56514]: cluster 2021-03-15T22:24:37.122398+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.b scrub ok 2021-03-15T22:24:38.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[56514]: cluster 2021-03-15T22:24:37.260529+0000 mon.a (mon.0) 1129 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:24:38.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[56514]: cluster 2021-03-15T22:24:37.260579+0000 mon.a (mon.0) 1130 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2021-03-15T22:24:38.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[56514]: cluster 2021-03-15T22:24:37.475215+0000 osd.6 (osd.6) 56 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:24:38.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[56514]: cluster 2021-03-15T22:24:38.089044+0000 mon.a (mon.0) 1131 : cluster [DBG] osdmap e363: 8 total, 8 up, 7 in 2021-03-15T22:24:38.408 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: cluster 2021-03-15T22:24:37.041075+0000 2021-03-15T22:24:38.408 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: mgr.y (mgr.14144) 577 : cluster [DBG] pgmap v669: 138 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 135 active+clean; 193 MiB data, 638 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.4 MiB/s wr, 14 op/s; 1/1224 objects degraded (0.082%); 1/1224 objects misplaced (0.082%); 15 MiB/s, 55 objects/s recovering 2021-03-15T22:24:38.408 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:24:38.409 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: T22:24:37.122398+0000 osd.3 (osd.3) 64 : cluster [DBG] 2021-03-15T22:24:38.409 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: 6.b scrub ok 2021-03-15T22:24:38.409 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:24:38.409 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: 15T22:24:37.260529 2021-03-15T22:24:38.410 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1129 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:24:38.410 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:24:38.410 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: T22:24:37.260579+0000 mon.a (mon.0 2021-03-15T22:24:38.410 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: ) 1130 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2021-03-15T22:24:38.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:24:38.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: 15T22:24:37.475215+0000 2021-03-15T22:24:38.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: osd.6 (osd.6) 56 : cluster [DBG] 2021-03-15T22:24:38.411 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: 3.10 scrub ok 2021-03-15T22:24:38.412 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:24:38.412 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: 15T22:24:38.089044 2021-03-15T22:24:38.412 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1131 : cluster 2021-03-15T22:24:38.412 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:38 smithi076 conmon[62156]: [DBG] osdmap e363: 8 total, 8 up, 7 in 2021-03-15T22:24:38.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:38.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:38.517 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:38.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:38.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: cluster 2021-03-15T22:24:37.041075+0000 mgr.y (mgr.14144) 577 : cluster [DBG] 2021-03-15T22:24:38.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: pgmap v669: 138 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 135 active+clean; 193 MiB data, 638 MiB used, 625 GiB / 626 GiB avail; 6.9 MiB/s rd, 2.4 MiB/s wr, 14 op/s; 1/1224 objects degraded (0.082%); 1/1224 objects misplaced (0.082%); 15 MiB/s, 55 objects/s recovering 2021-03-15T22:24:38.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: cluster 2021-03-15T22:24:37.122398+0000 osd.3 (osd.3) 64 : cluster [DBG] 6.b scrub ok 2021-03-15T22:24:38.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:24:38.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: 22:24:37.260529+0000 mon.a (mon.0) 1129 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:24:38.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: cluster 2021-03-15T22:24 2021-03-15T22:24:38.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: :37.260579+0000 mon.a (mon.0) 1130 : 2021-03-15T22:24:38.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 2 pgs peering) 2021-03-15T22:24:38.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:24:38.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: -03-15T22:24:37.475215+0000 osd.6 2021-03-15T22:24:38.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: (osd.6) 56 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:24:38.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:24:38.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: -15T22:24:38.089044+0000 mon.a (mon.0) 1131 : cluster 2021-03-15T22:24:38.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:38 smithi125 conmon[56739]: [DBG] osdmap e363: 8 total, 8 up, 7 in 2021-03-15T22:24:38.619 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:38.641 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:38.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:38.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:38.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:38.769 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:38.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:38.771 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:38.872 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:38.894 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:38.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:38.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:39.020 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:39.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:39.021 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:39.099 INFO:tasks.rados.rados.0.smithi076.stdout:1301: done (5 left) 2021-03-15T22:24:39.099 INFO:tasks.rados.rados.0.smithi076.stdout:1302: rollback oid 29 current snap is 137 2021-03-15T22:24:39.099 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 29 to 134 2021-03-15T22:24:39.100 INFO:tasks.rados.rados.0.smithi076.stdout:1303: write (excl) oid 46 current snap is 137 2021-03-15T22:24:39.100 INFO:tasks.rados.rados.0.smithi076.stdout:1303: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2021-03-15T22:24:39.109 INFO:tasks.rados.rados.0.smithi076.stdout:1303: writing smithi07694878-46 from 682483 to 1471797 tid 1 2021-03-15T22:24:39.115 INFO:tasks.rados.rados.0.smithi076.stdout:1303: writing smithi07694878-46 from 1944996 to 2736493 tid 2 2021-03-15T22:24:39.121 INFO:tasks.rados.rados.0.smithi076.stdout:1303: writing smithi07694878-46 from 3196560 to 3751134 tid 3 2021-03-15T22:24:39.122 INFO:tasks.rados.rados.0.smithi076.stdout:1304: snap_remove snap 128 2021-03-15T22:24:39.122 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:39.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:39.146 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:39.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:39.248 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:39.272 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:39.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:39.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:39.375 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:39.398 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:39.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:39.399 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:39.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:39.523 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:39.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:39.525 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:39.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[56739]: cluster 2021-03-15T22:24:37.902162+0000 osd.4 (osd. 2021-03-15T22:24:39.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[56739]: 4) 81 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:24:39.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[56739]: cluster 2021-03-15T22:24:37.977356+0000 osd.1 (osd.1) 74 : cluster 2021-03-15T22:24:39.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[56739]: [DBG] 4.6 scrub ok 2021-03-15T22:24:39.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:24:39.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[56739]: -03-15T22:24:38.433162+0000 osd.6 (osd.6 2021-03-15T22:24:39.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[56739]: ) 57 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:24:39.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[56739]: cluster 2021-03-15T22:24:39.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[56739]: 2021-03-15T22:24:39.095843+0000 2021-03-15T22:24:39.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[56739]: mon.a (mon.0) 1132 : cluster 2021-03-15T22:24:39.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[56739]: [DBG] osdmap e364: 8 total, 8 up, 7 in 2021-03-15T22:24:39.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[56514]: cluster 2021-03-15T22:24:37.902162+0000 osd.4 (osd. 2021-03-15T22:24:39.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[56514]: 4) 81 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:24:39.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[56514]: cluster 2021-03-15T22:24:37.977356+0000 osd.1 (osd.1) 74 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:24:39.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:24:39.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[56514]: T22:24:38.433162+0000 osd.6 (osd.6) 57 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:24:39.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:24:39.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[56514]: 22:24:39.095843+0000 mon.a (mon. 2021-03-15T22:24:39.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[56514]: 0) 1132 : cluster [DBG] osdmap e364: 8 total, 8 up, 7 in 2021-03-15T22:24:39.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: cluster 2021-03-15T22:24:37.902162+0000 osd.4 (osd.4) 81 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:24:39.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: cluster 2021-03-15T22:24:37.977356+0000 2021-03-15T22:24:39.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: osd.1 (osd.1) 74 : cluster 2021-03-15T22:24:39.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: [DBG] 4.6 scrub ok 2021-03-15T22:24:39.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: cluster 2021-03-15T22:24 2021-03-15T22:24:39.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: :38.433162+0000 osd.6 (osd. 2021-03-15T22:24:39.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: 6) 57 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:24:39.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: 2021-03-15T22:24:39.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:24:39.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: -03-15T22:24: 2021-03-15T22:24:39.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: 39.095843+0000 mon.a (mon 2021-03-15T22:24:39.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: .0) 1132 : cluster [DBG] 2021-03-15T22:24:39.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:39 smithi076 conmon[62156]: osdmap e364: 8 total, 8 up, 7 in 2021-03-15T22:24:39.627 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:39.650 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:39.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:39.651 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:39.752 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:39.778 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:39.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:39.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:39.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:39.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:39.906 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:39.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:40.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:40.031 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:40.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:40.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:40.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:24:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:24:40.084 INFO:tasks.rados.rados.0.smithi076.stdout:1304: done (7 left) 2021-03-15T22:24:40.084 INFO:tasks.rados.rados.0.smithi076.stdout:1305: write oid 25 current snap is 137 2021-03-15T22:24:40.084 INFO:tasks.rados.rados.0.smithi076.stdout:1305: seq_num 470 ranges {699290=464563,1603478=535155,2823983=765846} 2021-03-15T22:24:40.090 INFO:tasks.rados.rados.0.smithi076.stdout:1305: writing smithi07694878-25 from 699290 to 1163853 tid 1 2021-03-15T22:24:40.096 INFO:tasks.rados.rados.0.smithi076.stdout:1305: writing smithi07694878-25 from 1603478 to 2138633 tid 2 2021-03-15T22:24:40.107 INFO:tasks.rados.rados.0.smithi076.stdout:1305: writing smithi07694878-25 from 2823983 to 3589829 tid 3 2021-03-15T22:24:40.107 INFO:tasks.rados.rados.0.smithi076.stdout:1306: copy_from oid 9 from oid 16 current snap is 137 2021-03-15T22:24:40.107 INFO:tasks.rados.rados.0.smithi076.stdout:1307: read oid 26 snap -1 2021-03-15T22:24:40.107 INFO:tasks.rados.rados.0.smithi076.stdout:1307: expect (ObjNum 322 snap 87 seq_num 322) 2021-03-15T22:24:40.108 INFO:tasks.rados.rados.0.smithi076.stdout:1308: copy_from oid 28 from oid 23 current snap is 137 2021-03-15T22:24:40.109 INFO:tasks.rados.rados.0.smithi076.stdout:1300: expect (ObjNum 464 snap 135 seq_num 464) 2021-03-15T22:24:40.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:40.154 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:24:40.156 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:40.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:40.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:40.177 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:24:40.194 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:24:40.211 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:24:40.230 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:24:40.248 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:24:40.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:40.265 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:24:40.281 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:40.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:40.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:40.283 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:24:40.284 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:24:40.285 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:24:40.285 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:24:40.285 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:24:40.285 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:24:40.286 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:24:40.286 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:24:40.286 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:24:40.384 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:40.392 INFO:tasks.rados.rados.0.smithi076.stdout:1296: finishing write tid 3 to smithi07694878-44 2021-03-15T22:24:40.393 INFO:tasks.rados.rados.0.smithi076.stdout:1296: finishing write tid 4 to smithi07694878-44 2021-03-15T22:24:40.393 INFO:tasks.rados.rados.0.smithi076.stdout:1296: finishing write tid 5 to smithi07694878-44 2021-03-15T22:24:40.393 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 259 (ObjNum 468 snap 137 seq_num 468) dirty exists 2021-03-15T22:24:40.393 INFO:tasks.rados.rados.0.smithi076.stdout:1296: left oid 44 (ObjNum 468 snap 137 seq_num 468) 2021-03-15T22:24:40.394 INFO:tasks.rados.rados.0.smithi076.stdout:1296: done (10 left) 2021-03-15T22:24:40.394 INFO:tasks.rados.rados.0.smithi076.stdout:1297: done (9 left) 2021-03-15T22:24:40.394 INFO:tasks.rados.rados.0.smithi076.stdout:1300: done (8 left) 2021-03-15T22:24:40.394 INFO:tasks.rados.rados.0.smithi076.stdout:1309: snap_create 2021-03-15T22:24:40.395 INFO:tasks.rados.rados.0.smithi076.stdout:1298: expect (ObjNum 442 snap 124 seq_num 442) 2021-03-15T22:24:40.406 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:40.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:40.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:40.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:40.531 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:40.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:40.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:40.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: cluster 2021-03-15T22:24:39.041476+0000 mgr.y (mgr.14144) 578 : cluster [DBG] pgmap v672: 138 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 136 active+clean; 201 MiB data, 671 MiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 2.6 MiB/s wr, 13 op/s; 1/1233 objects degraded (0.081%); 1/1233 objects misplaced (0.081%); 5.9 MiB/s, 17 objects/s recovering 2021-03-15T22:24:40.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: cluster 2021-03-15T22:24:39.597633+0000 osd.3 (osd.3) 65 : 2021-03-15T22:24:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: cluster 2021-03-15T22:24:39.598005+0000 osd.3 (osd.3) 66 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: audit 2021-03-15T22:24:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: 2021-03-15T22:24:39.658522+0000 mgr.y (mgr.14144) 579 : 2021-03-15T22:24:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:24:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: :24:39.853261+0000 osd.2 (osd. 2021-03-15T22:24:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: 2) 64 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:24:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:24:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: -03-15T22:24:40.079105+0000 2021-03-15T22:24:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: mon.a (mon.0) 1133 : cluster [DBG] 2021-03-15T22:24:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: osdmap e365: 8 total, 8 up, 7 in 2021-03-15T22:24:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:24:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: 24:40.118712+0000 mon.a (mon.0 2021-03-15T22:24:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:40 smithi125 conmon[56739]: ) 1134 : audit [DBG] from='client.? 172.21.15.76:0/2378145405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:40.543 INFO:tasks.rados.rados.0.smithi076.stdout:1299: finishing copy_from to smithi07694878-48 2021-03-15T22:24:40.543 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 752 (ObjNum 459 snap 133 seq_num 459) dirty exists 2021-03-15T22:24:40.543 INFO:tasks.rados.rados.0.smithi076.stdout:1299: finishing copy_from racing read to smithi07694878-48 2021-03-15T22:24:40.544 INFO:tasks.rados.rados.0.smithi076.stdout:1302: finishing rollback tid 0 to smithi07694878-29 2021-03-15T22:24:40.544 INFO:tasks.rados.rados.0.smithi076.stdout:1302: finishing rollback tid 1 to smithi07694878-29 2021-03-15T22:24:40.544 INFO:tasks.rados.rados.0.smithi076.stdout:1302: finishing rollback tid 2 to smithi07694878-29 2021-03-15T22:24:40.544 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 765 (ObjNum 397 snap 108 seq_num 397) dirty exists 2021-03-15T22:24:40.544 INFO:tasks.rados.rados.0.smithi076.stdout:1303: finishing write tid 1 to smithi07694878-46 2021-03-15T22:24:40.545 INFO:tasks.rados.rados.0.smithi076.stdout:1303: finishing write tid 2 to smithi07694878-46 2021-03-15T22:24:40.545 INFO:tasks.rados.rados.0.smithi076.stdout:1303: finishing write tid 3 to smithi07694878-46 2021-03-15T22:24:40.545 INFO:tasks.rados.rados.0.smithi076.stdout:1303: finishing write tid 4 to smithi07694878-46 2021-03-15T22:24:40.545 INFO:tasks.rados.rados.0.smithi076.stdout:1303: finishing write tid 5 to smithi07694878-46 2021-03-15T22:24:40.546 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 203 (ObjNum 469 snap 137 seq_num 469) dirty exists 2021-03-15T22:24:40.546 INFO:tasks.rados.rados.0.smithi076.stdout:1303: left oid 46 (ObjNum 469 snap 137 seq_num 469) 2021-03-15T22:24:40.546 INFO:tasks.rados.rados.0.smithi076.stdout:1305: finishing write tid 1 to smithi07694878-25 2021-03-15T22:24:40.546 INFO:tasks.rados.rados.0.smithi076.stdout:1305: finishing write tid 2 to smithi07694878-25 2021-03-15T22:24:40.546 INFO:tasks.rados.rados.0.smithi076.stdout:1307: expect (ObjNum 322 snap 87 seq_num 322) 2021-03-15T22:24:40.619 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[56514]: cluster 2021-03-15T22:24:39.041476+0000 mgr.y (mgr.14144) 578 : cluster [DBG] pgmap v672: 138 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 136 active+clean; 201 MiB data, 671 MiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 2.6 MiB/s wr, 13 op/s; 1/1233 objects degraded (0.081%); 1/1233 objects misplaced (0.081%); 5.9 MiB/s, 17 objects/s recovering 2021-03-15T22:24:40.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[56514]: cluster 2021-03-15T22:24:39.597633+0000 osd.3 (osd.3) 65 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:40.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[56514]: cluster 2021-03-15T22:24:39.598005+0000 osd.3 (osd.3) 66 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:40.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[56514]: audit 2021-03-15T22:24:39.658522+0000 mgr.y (mgr.14144) 579 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:40.621 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[56514]: cluster 2021-03-15T22:24:39.853261+0000 osd.2 (osd.2) 64 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:24:40.621 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[56514]: cluster 2021-03-15T22:24:40.079105+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2021-03-15T22:24:40.621 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[56514]: audit 2021-03-15T22:24:40.118712+0000 mon.a (mon.0) 1134 : audit [DBG] from='client.? 172.21.15.76:0/2378145405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:40.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[62156]: cluster 2021-03-15T22:24:39.041476+0000 mgr.y (mgr.14144) 578 : cluster [DBG] pgmap v672: 138 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 136 active+clean; 201 MiB data, 671 MiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 2.6 MiB/s wr, 13 op/s; 1/1233 objects degraded (0.081%); 1/1233 objects misplaced (0.081%); 5.9 MiB/s, 17 objects/s recovering 2021-03-15T22:24:40.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[62156]: cluster 2021-03-15T22:24:39.597633+0000 osd.3 (osd.3) 65 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:40.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[62156]: cluster 2021-03-15T22:24:39.598005+0000 osd.3 (osd.3) 66 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:40.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[62156]: audit 2021-03-15T22:24:39.658522+0000 mgr.y (mgr.14144) 579 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:40.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[62156]: cluster 2021-03-15T22:24:39.853261+0000 osd.2 (osd.2) 64 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:24:40.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[62156]: cluster 2021-03-15T22:24:40.079105+0000 mon.a (mon.0) 1133 : cluster [DBG] osdmap e365: 8 total, 8 up, 7 in 2021-03-15T22:24:40.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[62156]: audit 2021-03-15T22:24:40.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:40 smithi076 conmon[62156]: 2021-03-15T22:24:40.118712+0000 mon.a (mon.0) 1134 : audit [DBG] from='client.? 172.21.15.76:0/2378145405' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:40.635 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:40.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:40.658 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:40.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:40.704 INFO:tasks.rados.rados.0.smithi076.stdout:1305: finishing write tid 3 to smithi07694878-25 2021-03-15T22:24:40.704 INFO:tasks.rados.rados.0.smithi076.stdout:1305: finishing write tid 4 to smithi07694878-25 2021-03-15T22:24:40.705 INFO:tasks.rados.rados.0.smithi076.stdout:1305: finishing write tid 5 to smithi07694878-25 2021-03-15T22:24:40.705 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 278 (ObjNum 470 snap 137 seq_num 470) dirty exists 2021-03-15T22:24:40.705 INFO:tasks.rados.rados.0.smithi076.stdout:1305: left oid 25 (ObjNum 470 snap 137 seq_num 470) 2021-03-15T22:24:40.706 INFO:tasks.rados.rados.0.smithi076.stdout:1306: finishing copy_from to smithi07694878-9 2021-03-15T22:24:40.706 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 695 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:24:40.706 INFO:tasks.rados.rados.0.smithi076.stdout:1306: finishing copy_from racing read to smithi07694878-9 2021-03-15T22:24:40.707 INFO:tasks.rados.rados.0.smithi076.stdout:1308: finishing copy_from to smithi07694878-28 2021-03-15T22:24:40.707 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 278 (ObjNum 424 snap 117 seq_num 424) dirty exists 2021-03-15T22:24:40.707 INFO:tasks.rados.rados.0.smithi076.stdout:1308: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:24:40.761 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:40.784 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:40.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:40.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:40.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:40.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:40.911 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:40.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:41.012 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:41.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:41.037 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:41.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:41.139 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:41.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:41.163 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:41.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:41.264 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:41.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:41.288 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:41.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:41.296 INFO:tasks.rados.rados.0.smithi076.stdout:1298: done (8 left) 2021-03-15T22:24:41.296 INFO:tasks.rados.rados.0.smithi076.stdout:1299: done (7 left) 2021-03-15T22:24:41.296 INFO:tasks.rados.rados.0.smithi076.stdout:1302: done (6 left) 2021-03-15T22:24:41.296 INFO:tasks.rados.rados.0.smithi076.stdout:1303: done (5 left) 2021-03-15T22:24:41.297 INFO:tasks.rados.rados.0.smithi076.stdout:1305: done (4 left) 2021-03-15T22:24:41.297 INFO:tasks.rados.rados.0.smithi076.stdout:1306: done (3 left) 2021-03-15T22:24:41.297 INFO:tasks.rados.rados.0.smithi076.stdout:1307: done (2 left) 2021-03-15T22:24:41.297 INFO:tasks.rados.rados.0.smithi076.stdout:1308: done (1 left) 2021-03-15T22:24:41.298 INFO:tasks.rados.rados.0.smithi076.stdout:1309: done (0 left) 2021-03-15T22:24:41.298 INFO:tasks.rados.rados.0.smithi076.stdout:1310: snap_remove snap 135 2021-03-15T22:24:41.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:41.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:41.413 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:41.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:41.515 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:41.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:41 smithi076 conmon[56514]: cluster 2021-03-15T22:24:39.929062+0000 osd.1 (osd.1) 75 : cluster 2021-03-15T22:24:41.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:41 smithi076 conmon[56514]: [DBG] 3.0 scrub ok 2021-03-15T22:24:41.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:41 smithi076 conmon[56514]: cluster 2021-03-15T22:24:40.007311+0000 osd.3 (osd.3) 67 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:24:41.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:41 smithi076 conmon[56514]: cluster 2021-03-15T22:24:40.468062+0000 osd.6 (osd.6) 58 : cluster [DBG] 4.a scrub ok 2021-03-15T22:24:41.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:41 smithi076 conmon[56514]: cluster 2021-03-15T22:24:40.640396+0000 osd.5 (osd.5) 67 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:24:41.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:41 smithi076 conmon[62156]: cluster 2021-03-15T22:24:39.929062+0000 2021-03-15T22:24:41.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:41 smithi076 conmon[62156]: osd.1 (osd.1) 75 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:24:41.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:41 smithi076 conmon[62156]: cluster 2021-03-15T22:24:40.007311+0000 osd.3 (osd.3) 67 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:24:41.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:41 smithi076 conmon[62156]: cluster 2021-03-15T22:24:40.468062+0000 osd.6 (osd.6) 58 : cluster [DBG] 4.a scrub ok 2021-03-15T22:24:41.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:41 smithi076 conmon[62156]: cluster 2021-03-15T22:24 2021-03-15T22:24:41.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:41 smithi076 conmon[62156]: :40.640396+0000 osd.5 (osd.5) 67 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:24:41.538 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:41.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:41.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:41.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:41.664 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:41.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:41.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:41.767 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:41.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:41 smithi125 conmon[56739]: cluster 2021-03-15T22:24:39.929062+0000 osd.1 (osd.1) 75 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:24:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:41 smithi125 conmon[56739]: cluster 2021-03-15T22:24:40.007311+0000 osd.3 (osd.3) 67 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:24:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:41 smithi125 conmon[56739]: cluster 2021-03-15T22:24:40.468062+0000 osd.6 (osd.6) 58 : cluster [DBG] 4.a scrub ok 2021-03-15T22:24:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:41 smithi125 conmon[56739]: cluster 2021-03-15T22:24:40.640396+0000 osd.5 (osd.5) 67 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:24:41.791 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:41.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:41.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:41.893 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:41.917 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:41.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:41.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:42.019 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:42.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:42.044 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:42.044 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:42.145 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:42.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:42.171 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:42.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:42.255 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:24:42.255 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:24:42.255 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:24:42.256 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:24:42.256 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:24:42.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:42.301 INFO:tasks.rados.rados.0.smithi076.stdout:1310: done (0 left) 2021-03-15T22:24:42.301 INFO:tasks.rados.rados.0.smithi076.stdout:1311: snap_remove snap 125 2021-03-15T22:24:42.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:42.303 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:42.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:42.304 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:24:42.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:42.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:42.429 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:42.429 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:42.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:42.553 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:42.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:42.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:42.611 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:24:42.612 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "" 2021-03-15T22:24:42.612 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:24:42.619 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[56514]: cluster 2021-03-15T22:24:40.910562+0000 osd.4 (osd.4) 82 : cluster [DBG] 6.e scrub ok 2021-03-15T22:24:42.619 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[56514]: cluster 2021-03-15T22:24:41.042062+0000 mgr.y (mgr.14144) 580 : cluster [DBG] pgmap v675: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 230 MiB data, 705 MiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 6.0 MiB/s wr, 27 op/s; 1/1263 objects misplaced (0.079%); 755 KiB/s, 0 objects/s recovering 2021-03-15T22:24:42.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[56514]: cluster 2021-03-15T22:24:41.279609+0000 mon.a (mon.0) 1135 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1233 objects degraded (0.081%), 1 pg degraded) 2021-03-15T22:24:42.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[56514]: cluster 2021-03-15T22:24:41.289253+0000 mon.a (mon.0) 1136 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2021-03-15T22:24:42.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[56514]: cluster 2021-03-15T22:24:41.483846+0000 osd.6 (osd.6) 59 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:24:42.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[56514]: cluster 2021-03-15T22:24:41.659446+0000 osd.5 (osd.5) 68 : cluster [DBG] 6.10 deep-scrub ok 2021-03-15T22:24:42.621 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[56514]: audit 2021-03-15T22:24:42.039459+0000 mon.a (mon.0) 1137 : audit [INF] from='client.? 172.21.15.76:0/2299526317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:24:42.621 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: cluster 2021-03-15T22:24:40.910562+0000 osd.4 2021-03-15T22:24:42.621 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: (osd.4) 82 : cluster [DBG] 6.e scrub ok 2021-03-15T22:24:42.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: cluster 2021-03-15T22:24:41.042062+0000 2021-03-15T22:24:42.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: mgr.y (mgr.14144) 580 : cluster [DBG] pgmap v675: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 230 MiB data, 705 MiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 6.0 MiB/s wr, 27 op/s; 1/1263 objects misplaced (0.079%); 755 KiB/s, 0 objects/s recovering 2021-03-15T22:24:42.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: cluster 2021-03-15T22:24:41 2021-03-15T22:24:42.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: .279609+0000 mon.a (mon.0) 1135 : cluster 2021-03-15T22:24:42.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1233 objects degraded (0.081%), 1 pg degraded) 2021-03-15T22:24:42.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: cluster 2021-03-15T22:24 2021-03-15T22:24:42.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: :41.289253+0000 mon.a (mon.0) 2021-03-15T22:24:42.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: 1136 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2021-03-15T22:24:42.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:24:42.624 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: 15T22:24:41.483846+0000 osd.6 2021-03-15T22:24:42.624 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: (osd.6) 59 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:24:42.624 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:24:42.624 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: 03-15T22:24:41.659446+0000 2021-03-15T22:24:42.625 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: osd.5 (osd.5) 68 : cluster [DBG] 6.10 deep-scrub ok 2021-03-15T22:24:42.625 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: 2021-03-15T22:24:42.625 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: audit 2021-03-15T22:24:42.625 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: 2021-03-15T22:24:42.039459+0000 mon.a ( 2021-03-15T22:24:42.626 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:42 smithi076 conmon[62156]: mon.0) 1137 : audit [INF] from='client.? 172.21.15.76:0/2299526317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:24:42.656 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:42.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:42.680 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:42.680 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:42.782 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:42.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: cluster 2021-03-15T22:24:40.910562+0000 osd.4 2021-03-15T22:24:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: (osd.4) 82 : cluster [DBG] 6.e scrub ok 2021-03-15T22:24:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: cluster 2021-03-15T22:24:41.042062+0000 mgr.y 2021-03-15T22:24:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: (mgr.14144) 580 : cluster [DBG] pgmap v675: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 230 MiB data, 705 MiB used, 625 GiB / 626 GiB avail; 7.9 MiB/s rd, 6.0 MiB/s wr, 27 op/s; 1/1263 objects misplaced (0.079%); 755 KiB/s, 0 objects/s recovering 2021-03-15T22:24:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: cluster 2021-03-15T22:24: 2021-03-15T22:24:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: 41.279609+0000 mon.a (mon.0) 1135 : cluster 2021-03-15T22:24:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/1233 objects degraded (0.081%), 1 pg degraded) 2021-03-15T22:24:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:24:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: T22:24:41.289253+0000 mon.a (mon 2021-03-15T22:24:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: .0) 1136 : cluster [DBG] osdmap e366: 8 total, 8 up, 7 in 2021-03-15T22:24:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: cluster 2021-03-15T22:24:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: 2021-03-15T22:24:41.483846+0000 2021-03-15T22:24:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: osd.6 (osd.6) 59 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:24:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: 2021-03-15T22:24:42.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: cluster 2021-03-15T22:24: 2021-03-15T22:24:42.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: 41.659446+0000 osd.5 (osd.5) 68 : 2021-03-15T22:24:42.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: cluster [DBG] 6.10 deep-scrub ok 2021-03-15T22:24:42.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: audit 2021 2021-03-15T22:24:42.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: -03-15T22:24:42.039459+0000 2021-03-15T22:24:42.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: mon.a (mon.0) 1137 : audit [INF] 2021-03-15T22:24:42.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:42 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2299526317' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:24:42.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:42.809 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:42.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:42.911 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:42.934 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:42.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:42.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:43.037 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:43.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:43.062 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:43.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:43.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:43.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:43.189 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:43.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:43.292 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:43.313 INFO:tasks.rados.rados.0.smithi076.stdout:1311: done (0 left) 2021-03-15T22:24:43.314 INFO:tasks.rados.rados.0.smithi076.stdout:1312: write oid 1 current snap is 138 2021-03-15T22:24:43.314 INFO:tasks.rados.rados.0.smithi076.stdout:1312: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2021-03-15T22:24:43.319 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:43.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:43.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:43.322 INFO:tasks.rados.rados.0.smithi076.stdout:1312: writing smithi07694878-1 from 716097 to 1255909 tid 1 2021-03-15T22:24:43.330 INFO:tasks.rados.rados.0.smithi076.stdout:1312: writing smithi07694878-1 from 1945607 to 2740773 tid 2 2021-03-15T22:24:43.335 INFO:tasks.rados.rados.0.smithi076.stdout:1312: writing smithi07694878-1 from 3535053 to 3916097 tid 3 2021-03-15T22:24:43.336 INFO:tasks.rados.rados.0.smithi076.stdout:1313: snap_create 2021-03-15T22:24:43.336 INFO:tasks.rados.rados.0.smithi076.stdout:1312: finishing write tid 1 to smithi07694878-1 2021-03-15T22:24:43.336 INFO:tasks.rados.rados.0.smithi076.stdout:1312: finishing write tid 2 to smithi07694878-1 2021-03-15T22:24:43.338 INFO:tasks.rados.rados.0.smithi076.stdout:1312: finishing write tid 3 to smithi07694878-1 2021-03-15T22:24:43.339 INFO:tasks.rados.rados.0.smithi076.stdout:1312: finishing write tid 4 to smithi07694878-1 2021-03-15T22:24:43.339 INFO:tasks.rados.rados.0.smithi076.stdout:1312: finishing write tid 5 to smithi07694878-1 2021-03-15T22:24:43.340 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 181 (ObjNum 471 snap 138 seq_num 471) dirty exists 2021-03-15T22:24:43.340 INFO:tasks.rados.rados.0.smithi076.stdout:1312: left oid 1 (ObjNum 471 snap 138 seq_num 471) 2021-03-15T22:24:43.423 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:43.449 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:43.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:43.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:43.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[56514]: cluster 2021-03-15T22:24:42.006141+0000 osd.3 ( 2021-03-15T22:24:43.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[56514]: osd.3) 68 : cluster [DBG] 7.2 deep-scrub ok 2021-03-15T22:24:43.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[56514]: cluster 2021-03-15T22:24:42.287129+0000 mon.a (mon.0) 1138 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:43.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[56514]: audit 2021-03-15T22:24:42.293395+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? 172.21.15.76:0/2299526317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:24:43.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[56514]: cluster 2021-03-15T22:24:42.293450+0000 mon.a (mon.0) 1140 : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2021-03-15T22:24:43.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[56514]: cluster 2021-03-15T22:24:42.525969+0000 osd.6 (osd.6) 60 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:24:43.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: cluster 2021-03-15T22:24:43.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: 2021-03-15T22:24:42 2021-03-15T22:24:43.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: .006141+0000 osd.3 (osd.3) 68 : cluster [DBG] 7.2 deep-scrub ok 2021-03-15T22:24:43.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: cluster 2021-03-15T22:24: 2021-03-15T22:24:43.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: 42.287129+0000 mon.a (mon.0) 1138 : cluster 2021-03-15T22:24:43.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:43.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: audit 2021-03-15T22:24: 2021-03-15T22:24:43.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: 42.293395+0000 mon.a (mon.0) 1139 : audit [INF] from='client.? 172.21.15.76:0/2299526317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:24:43.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:24:43.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: 15T22:24:42.293450+0000 mon.a (mon.0) 1140 2021-03-15T22:24:43.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: : cluster [DBG] osdmap e367: 8 total, 8 up, 7 in 2021-03-15T22:24:43.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: cluster 2021-03-15T22:24:43.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: 2021-03-15T22:24:42.525969+0000 osd.6 (osd.6 2021-03-15T22:24:43.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:43 smithi076 conmon[62156]: ) 60 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:24:43.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:43.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:43.581 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:43.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:43.683 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:43.709 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:43.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:43.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:43.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: cluster 2021-03-15T22:24:42.006141+0000 osd.3 (osd.3) 2021-03-15T22:24:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: 68 : cluster [DBG] 7.2 deep-scrub ok 2021-03-15T22:24:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: cluster 2021-03-15T22:24:42.287129+0000 mon.a (mon.0) 1138 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:24:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: 22:24:42.293395+0000 mon.a (mon. 2021-03-15T22:24:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: 0) 1139 : audit [INF] from='client.? 172.21.15.76:0/2299526317' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:24:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:24:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: -03-15T22:24:42.293450 2021-03-15T22:24:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1140 : cluster 2021-03-15T22:24:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: [DBG] osdmap e367: 8 total, 8 up, 7 in 2021-03-15T22:24:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: cluster 2021-03-15T22:24:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: 2021-03-15T22:24:42.525969+0000 2021-03-15T22:24:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: osd.6 (osd.6) 60 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:24:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:43 smithi125 conmon[56739]: 2021-03-15T22:24:43.812 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:43.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:43.836 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:43.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:43.842 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:24:43.938 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:43.964 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:43.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:44.012 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:24:44.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:44.094 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:44.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:44.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:44.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:44.222 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:44.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:44.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:44.315 INFO:tasks.rados.rados.0.smithi076.stdout:1312: done (1 left) 2021-03-15T22:24:44.316 INFO:tasks.rados.rados.0.smithi076.stdout:1313: done (0 left) 2021-03-15T22:24:44.316 INFO:tasks.rados.rados.0.smithi076.stdout:1314: read oid 7 snap -1 2021-03-15T22:24:44.316 INFO:tasks.rados.rados.0.smithi076.stdout:1314: expect (ObjNum 417 snap 112 seq_num 417) 2021-03-15T22:24:44.317 INFO:tasks.rados.rados.0.smithi076.stdout:1315: read oid 14 snap -1 2021-03-15T22:24:44.317 INFO:tasks.rados.rados.0.smithi076.stdout:1315: expect (ObjNum 453 snap 132 seq_num 453) 2021-03-15T22:24:44.317 INFO:tasks.rados.rados.0.smithi076.stdout:1316: read oid 3 snap -1 2021-03-15T22:24:44.317 INFO:tasks.rados.rados.0.smithi076.stdout:1316: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:24:44.317 INFO:tasks.rados.rados.0.smithi076.stdout:1317: rollback oid 8 current snap is 139 2021-03-15T22:24:44.318 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 132 2021-03-15T22:24:44.318 INFO:tasks.rados.rados.0.smithi076.stdout:1318: snap_create 2021-03-15T22:24:44.319 INFO:tasks.rados.rados.0.smithi076.stdout:1317: finishing rollback tid 0 to smithi07694878-8 2021-03-15T22:24:44.320 INFO:tasks.rados.rados.0.smithi076.stdout:1317: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:24:44.320 INFO:tasks.rados.rados.0.smithi076.stdout:1317: finishing rollback tid 2 to smithi07694878-8 2021-03-15T22:24:44.321 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 755 (ObjNum 399 snap 108 seq_num 399) dirty exists 2021-03-15T22:24:44.322 INFO:tasks.rados.rados.0.smithi076.stdout:1314: expect (ObjNum 417 snap 112 seq_num 417) 2021-03-15T22:24:44.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:44.349 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:44.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:44.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:44.354 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:44.412 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:44.452 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:44.467 INFO:tasks.rados.rados.0.smithi076.stdout:1315: expect (ObjNum 453 snap 132 seq_num 453) 2021-03-15T22:24:44.474 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:44.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:44.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:44.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:44 smithi076 conmon[56514]: cluster 2021-03-15T22:24:43.042545+0000 mgr.y (mgr.14144) 581 : cluster [DBG] pgmap v678: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 233 MiB data, 707 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 24 op/s; 1/1269 objects misplaced (0.079%); 755 KiB/s, 0 objects/s recovering 2021-03-15T22:24:44.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:44 smithi076 conmon[56514]: cluster 2021-03-15T22:24:43.305653+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2021-03-15T22:24:44.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:44 smithi076 conmon[62156]: cluster 2021-03-15T22:24:43.042545+0000 mgr.y (mgr.14144) 581 : cluster [DBG] pgmap v678: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 233 MiB data, 707 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 24 op/s; 1/1269 objects misplaced (0.079%); 755 KiB/s, 0 objects/s recovering 2021-03-15T22:24:44.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:44 smithi076 conmon[62156]: cluster 2021-03-15T22:24:43.305653+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2021-03-15T22:24:44.578 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:44.605 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:44.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:44.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:44.640 INFO:tasks.rados.rados.0.smithi076.stdout:1316: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:24:44.707 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:44.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:44.731 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:44.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:44.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:44 smithi125 conmon[56739]: cluster 2021-03-15T22:24:43.042545+0000 mgr.y (mgr.14144) 581 2021-03-15T22:24:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:44 smithi125 conmon[56739]: : cluster [DBG] pgmap v678: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 233 MiB data, 707 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 4.7 MiB/s wr, 24 op/s; 1/1269 objects misplaced (0.079%); 755 KiB/s, 0 objects/s recovering 2021-03-15T22:24:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:44 smithi125 conmon[56739]: cluster 2021-03-15T22:24:43.305653+0000 mon.a (mon.0) 1141 : cluster [DBG] osdmap e368: 8 total, 8 up, 7 in 2021-03-15T22:24:44.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:44.856 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:44.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:44.858 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:44.959 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:44.981 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:44.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:44.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:45.083 INFO:tasks.rados.rados.0.smithi076.stdout:1314: done (4 left) 2021-03-15T22:24:45.084 INFO:tasks.rados.rados.0.smithi076.stdout:1315: done (3 left) 2021-03-15T22:24:45.084 INFO:tasks.rados.rados.0.smithi076.stdout:1316: done (2 left) 2021-03-15T22:24:45.084 INFO:tasks.rados.rados.0.smithi076.stdout:1317: done (1 left) 2021-03-15T22:24:45.084 INFO:tasks.rados.rados.0.smithi076.stdout:1318: done (0 left) 2021-03-15T22:24:45.085 INFO:tasks.rados.rados.0.smithi076.stdout:1319: write (excl) oid 47 current snap is 140 2021-03-15T22:24:45.085 INFO:tasks.rados.rados.0.smithi076.stdout:1319: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2021-03-15T22:24:45.085 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:45.091 INFO:tasks.rados.rados.0.smithi076.stdout:1319: writing smithi07694878-47 from 732904 to 1464318 tid 1 2021-03-15T22:24:45.099 INFO:tasks.rados.rados.0.smithi076.stdout:1319: writing smithi07694878-47 from 2120442 to 2659266 tid 2 2021-03-15T22:24:45.106 INFO:tasks.rados.rados.0.smithi076.stdout:1319: writing smithi07694878-47 from 3278829 to 3783572 tid 3 2021-03-15T22:24:45.106 INFO:tasks.rados.rados.0.smithi076.stdout:1320: delete oid 38 current snap is 140 2021-03-15T22:24:45.106 INFO:tasks.rados.rados.0.smithi076.stdout:1319: finishing write tid 1 to smithi07694878-47 2021-03-15T22:24:45.106 INFO:tasks.rados.rados.0.smithi076.stdout:1319: finishing write tid 2 to smithi07694878-47 2021-03-15T22:24:45.109 INFO:tasks.rados.rados.0.smithi076.stdout:1320: done (1 left) 2021-03-15T22:24:45.109 INFO:tasks.rados.rados.0.smithi076.stdout:1321: rollback oid 43 current snap is 140 2021-03-15T22:24:45.109 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 43 to 119 2021-03-15T22:24:45.109 INFO:tasks.rados.rados.0.smithi076.stdout:1322: write (excl) oid 49 current snap is 140 2021-03-15T22:24:45.109 INFO:tasks.rados.rados.0.smithi076.stdout:1322: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2021-03-15T22:24:45.112 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:45.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:45.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:45.116 INFO:tasks.rados.rados.0.smithi076.stdout:1322: writing smithi07694878-49 from 766518 to 1248430 tid 1 2021-03-15T22:24:45.122 INFO:tasks.rados.rados.0.smithi076.stdout:1322: writing smithi07694878-49 from 1837406 to 2379899 tid 2 2021-03-15T22:24:45.128 INFO:tasks.rados.rados.0.smithi076.stdout:1322: writing smithi07694878-49 from 2933675 to 3460962 tid 3 2021-03-15T22:24:45.131 INFO:tasks.rados.rados.0.smithi076.stdout:1322: writing smithi07694878-49 from 3876584 to 3966518 tid 4 2021-03-15T22:24:45.132 INFO:tasks.rados.rados.0.smithi076.stdout:1323: read oid 6 snap 134 2021-03-15T22:24:45.132 INFO:tasks.rados.rados.0.smithi076.stdout:1323: expect (ObjNum 429 snap 119 seq_num 429) 2021-03-15T22:24:45.132 INFO:tasks.rados.rados.0.smithi076.stdout:1324: write (excl) oid 25 current snap is 140 2021-03-15T22:24:45.132 INFO:tasks.rados.rados.0.smithi076.stdout:1319: finishing write tid 3 to smithi07694878-47 2021-03-15T22:24:45.132 INFO:tasks.rados.rados.0.smithi076.stdout:1324: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2021-03-15T22:24:45.139 INFO:tasks.rados.rados.0.smithi076.stdout:1324: writing smithi07694878-25 from 783325 to 1340486 tid 1 2021-03-15T22:24:45.145 INFO:tasks.rados.rados.0.smithi076.stdout:1324: writing smithi07694878-25 from 1779535 to 2465686 tid 2 2021-03-15T22:24:45.157 INFO:tasks.rados.rados.0.smithi076.stdout:1324: writing smithi07694878-25 from 3244745 to 3699657 tid 3 2021-03-15T22:24:45.157 INFO:tasks.rados.rados.0.smithi076.stdout:1325: snap_create 2021-03-15T22:24:45.158 INFO:tasks.rados.rados.0.smithi076.stdout:1319: finishing write tid 4 to smithi07694878-47 2021-03-15T22:24:45.158 INFO:tasks.rados.rados.0.smithi076.stdout:1319: finishing write tid 5 to smithi07694878-47 2021-03-15T22:24:45.158 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 270 (ObjNum 472 snap 140 seq_num 472) dirty exists 2021-03-15T22:24:45.159 INFO:tasks.rados.rados.0.smithi076.stdout:1319: left oid 47 (ObjNum 472 snap 140 seq_num 472) 2021-03-15T22:24:45.159 INFO:tasks.rados.rados.0.smithi076.stdout:1321: finishing rollback tid 0 to smithi07694878-43 2021-03-15T22:24:45.159 INFO:tasks.rados.rados.0.smithi076.stdout:1321: finishing rollback tid 1 to smithi07694878-43 2021-03-15T22:24:45.159 INFO:tasks.rados.rados.0.smithi076.stdout:1321: finishing rollback tid 2 to smithi07694878-43 2021-03-15T22:24:45.160 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 286 (ObjNum 392 snap 106 seq_num 392) dirty exists 2021-03-15T22:24:45.160 INFO:tasks.rados.rados.0.smithi076.stdout:1322: finishing write tid 1 to smithi07694878-49 2021-03-15T22:24:45.160 INFO:tasks.rados.rados.0.smithi076.stdout:1322: finishing write tid 2 to smithi07694878-49 2021-03-15T22:24:45.160 INFO:tasks.rados.rados.0.smithi076.stdout:1322: finishing write tid 3 to smithi07694878-49 2021-03-15T22:24:45.160 INFO:tasks.rados.rados.0.smithi076.stdout:1322: finishing write tid 4 to smithi07694878-49 2021-03-15T22:24:45.161 INFO:tasks.rados.rados.0.smithi076.stdout:1322: finishing write tid 5 to smithi07694878-49 2021-03-15T22:24:45.161 INFO:tasks.rados.rados.0.smithi076.stdout:1322: finishing write tid 6 to smithi07694878-49 2021-03-15T22:24:45.161 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 758 (ObjNum 474 snap 140 seq_num 474) dirty exists 2021-03-15T22:24:45.162 INFO:tasks.rados.rados.0.smithi076.stdout:1322: left oid 49 (ObjNum 474 snap 140 seq_num 474) 2021-03-15T22:24:45.162 INFO:tasks.rados.rados.0.smithi076.stdout:1323: expect (ObjNum 429 snap 119 seq_num 429) 2021-03-15T22:24:45.215 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:45.242 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:45.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:45.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:45.286 INFO:tasks.rados.rados.0.smithi076.stdout:1324: finishing write tid 1 to smithi07694878-25 2021-03-15T22:24:45.287 INFO:tasks.rados.rados.0.smithi076.stdout:1324: finishing write tid 2 to smithi07694878-25 2021-03-15T22:24:45.287 INFO:tasks.rados.rados.0.smithi076.stdout:1324: finishing write tid 3 to smithi07694878-25 2021-03-15T22:24:45.287 INFO:tasks.rados.rados.0.smithi076.stdout:1324: finishing write tid 4 to smithi07694878-25 2021-03-15T22:24:45.287 INFO:tasks.rados.rados.0.smithi076.stdout:1324: finishing write tid 5 to smithi07694878-25 2021-03-15T22:24:45.288 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 291 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-03-15T22:24:45.288 INFO:tasks.rados.rados.0.smithi076.stdout:1324: left oid 25 (ObjNum 475 snap 140 seq_num 475) 2021-03-15T22:24:45.344 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:45.366 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:45.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:45.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:45.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:45.496 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:45.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:45.497 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:45.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:45 smithi076 conmon[56514]: cluster 2021-03-15T22:24:44.309530+0000 mon.a (mon 2021-03-15T22:24:45.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:45 smithi076 conmon[56514]: .0) 1142 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2021-03-15T22:24:45.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:45 smithi076 conmon[56514]: cluster 2021-03-15T22:24:45.080605+0000 mon.a (mon.0) 1143 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2021-03-15T22:24:45.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:45 smithi076 conmon[62156]: cluster 2021-03-15T22:24:44.309530+0000 mon.a ( 2021-03-15T22:24:45.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:45 smithi076 conmon[62156]: mon.0) 1142 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2021-03-15T22:24:45.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:45 smithi076 conmon[62156]: cluster 2021-03-15T22:24:45.080605+0000 mon.a (mon.0) 1143 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2021-03-15T22:24:45.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:45.625 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:45.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:45.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:45.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:45.754 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:45.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:45.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:45.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:45 smithi125 conmon[56739]: cluster 2021-03-15T22:24:44.309530+0000 mon.a (mon. 2021-03-15T22:24:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:45 smithi125 conmon[56739]: 0) 1142 : cluster [DBG] osdmap e369: 8 total, 8 up, 7 in 2021-03-15T22:24:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:45 smithi125 conmon[56739]: cluster 2021-03-15T22:24:45.080605+0000 mon.a (mon.0) 1143 : cluster [DBG] osdmap e370: 8 total, 8 up, 7 in 2021-03-15T22:24:45.858 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:45.883 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:45.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:45.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:45.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:46.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:46.010 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:46.010 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:46.095 INFO:tasks.rados.rados.0.smithi076.stdout:1319: done (5 left) 2021-03-15T22:24:46.096 INFO:tasks.rados.rados.0.smithi076.stdout:1321: done (4 left) 2021-03-15T22:24:46.096 INFO:tasks.rados.rados.0.smithi076.stdout:1322: done (3 left) 2021-03-15T22:24:46.097 INFO:tasks.rados.rados.0.smithi076.stdout:1323: done (2 left) 2021-03-15T22:24:46.097 INFO:tasks.rados.rados.0.smithi076.stdout:1324: done (1 left) 2021-03-15T22:24:46.098 INFO:tasks.rados.rados.0.smithi076.stdout:1325: done (0 left) 2021-03-15T22:24:46.098 INFO:tasks.rados.rados.0.smithi076.stdout:1326: rollback oid 10 current snap is 141 2021-03-15T22:24:46.098 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 10 to 140 2021-03-15T22:24:46.099 INFO:tasks.rados.rados.0.smithi076.stdout:1327: snap_create 2021-03-15T22:24:46.102 INFO:tasks.rados.rados.0.smithi076.stdout:1326: finishing rollback tid 0 to smithi07694878-10 2021-03-15T22:24:46.103 INFO:tasks.rados.rados.0.smithi076.stdout:1326: finishing rollback tid 1 to smithi07694878-10 2021-03-15T22:24:46.103 INFO:tasks.rados.rados.0.smithi076.stdout:1326: finishing rollback tid 2 to smithi07694878-10 2021-03-15T22:24:46.103 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 295 (ObjNum 444 snap 125 seq_num 444) dirty exists 2021-03-15T22:24:46.113 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:46.135 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:46.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:46.137 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:46.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:46.265 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:46.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:46.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:24:46.368 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:46.393 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:46.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:46.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:46.496 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:46.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:46.522 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:46.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:46.571 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:46.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:46 smithi076 conmon[56514]: cluster 2021-03-15T22:24:45.042975+0000 mgr.y (mgr.14144) 582 : cluster [DBG] pgmap v681: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 236 MiB data, 720 MiB used, 625 GiB / 626 GiB avail; 5.0 MiB/s rd, 2.0 MiB/s wr, 7 op/s; 1/1272 objects misplaced (0.079%) 2021-03-15T22:24:46.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:46 smithi076 conmon[56514]: cluster 2021-03-15T22:24:46.093531+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2021-03-15T22:24:46.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:46 smithi076 conmon[62156]: cluster 2021-03-15T22:24:45.042975+0000 mgr.y ( 2021-03-15T22:24:46.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:46 smithi076 conmon[62156]: mgr.14144) 582 : cluster [DBG] pgmap v681: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 236 MiB data, 720 MiB used, 625 GiB / 626 GiB avail; 5.0 MiB/s rd, 2.0 MiB/s wr, 7 op/s; 1/1272 objects misplaced (0.079%) 2021-03-15T22:24:46.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:46 smithi076 conmon[62156]: cluster 2021-03-15T22:24:46.093531+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2021-03-15T22:24:46.623 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:46.629 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:24:46.629 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:24:46.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:24:46.630 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:24:46.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.631 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1191 7.6 7:61e0cc05:::smithi07694878-16:head [copy-from ver 101] snapc 4e=[4e,4d,3f] ondisk+write+known_if_redirected e175)", 2021-03-15T22:24:46.631 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:24:46.631 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 196.55745638900001, 2021-03-15T22:24:46.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.97834980800000004, 2021-03-15T22:24:46.632 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.632 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:46.632 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.632 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.633 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.633 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1191 2021-03-15T22:24:46.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.633 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:24:46.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:24:46.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0440000000000003e-06 2021-03-15T22:24:46.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067745+0000", 2021-03-15T22:24:46.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4340000000000001e-06 2021-03-15T22:24:46.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.637 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.637 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067749+0000", 2021-03-15T22:24:46.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9241000000000001e-05 2021-03-15T22:24:46.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.638 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.638 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067768+0000", 2021-03-15T22:24:46.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3010000000000007e-06 2021-03-15T22:24:46.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.639 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.639 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067776+0000", 2021-03-15T22:24:46.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5014e-05 2021-03-15T22:24:46.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067821+0000", 2021-03-15T22:24:46.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9779999999999994e-06 2021-03-15T22:24:46.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.641 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:24:46.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067830+0000", 2021-03-15T22:24:46.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.95797833899999996 2021-03-15T22:24:46.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.025809+0000", 2021-03-15T22:24:46.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138412 2021-03-15T22:24:46.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.025947+0000", 2021-03-15T22:24:46.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006355744 2021-03-15T22:24:46.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,7", 2021-03-15T22:24:46.645 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.032303+0000", 2021-03-15T22:24:46.645 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043109439999999997 2021-03-15T22:24:46.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:46.646 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.036614+0000", 2021-03-15T22:24:46.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092878839999999997 2021-03-15T22:24:46.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.045902+0000", 2021-03-15T22:24:46.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8556999999999999e-05 2021-03-15T22:24:46.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.045990+0000", 2021-03-15T22:24:46.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6665999999999997e-05 2021-03-15T22:24:46.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:46.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046037+0000", 2021-03-15T22:24:46.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.325e-05 2021-03-15T22:24:46.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046090+0000", 2021-03-15T22:24:46.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.97834980800000004 2021-03-15T22:24:46.652 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.652 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.652 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.653 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1192 7.6 7:61e0cc05:::smithi07694878-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e175)", 2021-03-15T22:24:46.653 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:24:46.653 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 196.55740707000001, 2021-03-15T22:24:46.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.978448604, 2021-03-15T22:24:46.654 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.654 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.654 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.654 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.654 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.655 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1192 2021-03-15T22:24:46.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.655 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:24:46.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:24:46.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.401e-06 2021-03-15T22:24:46.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067791+0000", 2021-03-15T22:24:46.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.083e-06 2021-03-15T22:24:46.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.659 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067794+0000", 2021-03-15T22:24:46.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4919999999999996e-06 2021-03-15T22:24:46.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067802+0000", 2021-03-15T22:24:46.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.383e-06 2021-03-15T22:24:46.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067807+0000", 2021-03-15T22:24:46.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8674000000000002e-05 2021-03-15T22:24:46.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067856+0000", 2021-03-15T22:24:46.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5890000000000003e-06 2021-03-15T22:24:46.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:24:46.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067864+0000", 2021-03-15T22:24:46.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.95935619400000005 2021-03-15T22:24:46.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.027221+0000", 2021-03-15T22:24:46.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9426999999999998e-05 2021-03-15T22:24:46.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:46.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.027290+0000", 2021-03-15T22:24:46.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085250369999999992 2021-03-15T22:24:46.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.035815+0000", 2021-03-15T22:24:46.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9997999999999997e-05 2021-03-15T22:24:46.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:46.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.035875+0000", 2021-03-15T22:24:46.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010246089 2021-03-15T22:24:46.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046121+0000", 2021-03-15T22:24:46.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3338999999999998e-05 2021-03-15T22:24:46.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046184+0000", 2021-03-15T22:24:46.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3897999999999998e-05 2021-03-15T22:24:46.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046238+0000", 2021-03-15T22:24:46.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.978448604 2021-03-15T22:24:46.672 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.673 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.673 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.675 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1556 7.6 7:6a64397b:::smithi07694878-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e204)", 2021-03-15T22:24:46.676 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.030294+0000", 2021-03-15T22:24:46.676 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 167.59490260000001, 2021-03-15T22:24:46.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024729370000000001, 2021-03-15T22:24:46.676 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.676 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.677 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.677 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.677 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.677 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1556 2021-03-15T22:24:46.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.678 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030294+0000", 2021-03-15T22:24:46.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030294+0000", 2021-03-15T22:24:46.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.128e-06 2021-03-15T22:24:46.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030295+0000", 2021-03-15T22:24:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3790000000000001e-06 2021-03-15T22:24:46.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030297+0000", 2021-03-15T22:24:46.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4710000000000004e-06 2021-03-15T22:24:46.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030303+0000", 2021-03-15T22:24:46.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0389999999999998e-06 2021-03-15T22:24:46.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030307+0000", 2021-03-15T22:24:46.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00070512700000000005 2021-03-15T22:24:46.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.031012+0000", 2021-03-15T22:24:46.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9470000000000001e-05 2021-03-15T22:24:46.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:46.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.031042+0000", 2021-03-15T22:24:46.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015725996999999999 2021-03-15T22:24:46.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046768+0000", 2021-03-15T22:24:46.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010418500000000001 2021-03-15T22:24:46.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:46.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046872+0000", 2021-03-15T22:24:46.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079290709999999993 2021-03-15T22:24:46.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.054801+0000", 2021-03-15T22:24:46.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5829999999999996e-05 2021-03-15T22:24:46.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.054887+0000", 2021-03-15T22:24:46.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000136673 2021-03-15T22:24:46.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.055024+0000", 2021-03-15T22:24:46.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024729370000000001 2021-03-15T22:24:46.693 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.693 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.694 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.694 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1563 7.f 7:fee9f235:::smithi07694878-11:head [copy-from ver 242] snapc 6a=[6a,69,68] ondisk+write+known_if_redirected e204)", 2021-03-15T22:24:46.694 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.059465+0000", 2021-03-15T22:24:46.695 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 167.565731556, 2021-03-15T22:24:46.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026787723999999999, 2021-03-15T22:24:46.695 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.696 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:46.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.697 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1563 2021-03-15T22:24:46.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.697 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059465+0000", 2021-03-15T22:24:46.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059465+0000", 2021-03-15T22:24:46.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.382e-06 2021-03-15T22:24:46.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059469+0000", 2021-03-15T22:24:46.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.294e-06 2021-03-15T22:24:46.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059471+0000", 2021-03-15T22:24:46.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4338999999999999e-05 2021-03-15T22:24:46.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059485+0000", 2021-03-15T22:24:46.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3259999999999998e-06 2021-03-15T22:24:46.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059493+0000", 2021-03-15T22:24:46.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9969999999999995e-05 2021-03-15T22:24:46.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059573+0000", 2021-03-15T22:24:46.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017286299999999999 2021-03-15T22:24:46.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059745+0000", 2021-03-15T22:24:46.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089917209999999994 2021-03-15T22:24:46.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:24:46.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.068737+0000", 2021-03-15T22:24:46.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0046724000000000002 2021-03-15T22:24:46.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:46.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.073410+0000", 2021-03-15T22:24:46.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099697450000000003 2021-03-15T22:24:46.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.083379+0000", 2021-03-15T22:24:46.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027667730000000001 2021-03-15T22:24:46.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086146+0000", 2021-03-15T22:24:46.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6458000000000001e-05 2021-03-15T22:24:46.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:46.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086192+0000", 2021-03-15T22:24:46.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0452999999999999e-05 2021-03-15T22:24:46.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086253+0000", 2021-03-15T22:24:46.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026787723999999999 2021-03-15T22:24:46.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.713 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.713 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.714 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1564 7.f 7:fee9f235:::smithi07694878-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e204)", 2021-03-15T22:24:46.714 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.059507+0000", 2021-03-15T22:24:46.715 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 167.56569024500001, 2021-03-15T22:24:46.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026908793, 2021-03-15T22:24:46.715 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.715 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.715 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.716 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.716 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.716 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1564 2021-03-15T22:24:46.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.717 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059507+0000", 2021-03-15T22:24:46.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059507+0000", 2021-03-15T22:24:46.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.274e-06 2021-03-15T22:24:46.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059508+0000", 2021-03-15T22:24:46.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6589999999999999e-06 2021-03-15T22:24:46.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059509+0000", 2021-03-15T22:24:46.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0969999999999995e-06 2021-03-15T22:24:46.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059519+0000", 2021-03-15T22:24:46.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5739999999999997e-06 2021-03-15T22:24:46.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059523+0000", 2021-03-15T22:24:46.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00046872599999999998 2021-03-15T22:24:46.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059992+0000", 2021-03-15T22:24:46.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7046999999999999e-05 2021-03-15T22:24:46.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:46.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.060039+0000", 2021-03-15T22:24:46.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012485036 2021-03-15T22:24:46.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.072524+0000", 2021-03-15T22:24:46.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6324999999999999e-05 2021-03-15T22:24:46.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:46.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.072590+0000", 2021-03-15T22:24:46.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013691045000000001 2021-03-15T22:24:46.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086281+0000", 2021-03-15T22:24:46.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6152e-05 2021-03-15T22:24:46.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086347+0000", 2021-03-15T22:24:46.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7858e-05 2021-03-15T22:24:46.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086415+0000", 2021-03-15T22:24:46.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026908793 2021-03-15T22:24:46.731 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.731 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.731 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.732 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:548 7.6 e204/175 ondisk, result = 0)", 2021-03-15T22:24:46.732 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:24:46.732 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 166.54367415999999, 2021-03-15T22:24:46.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045793304999999999, 2021-03-15T22:24:46.733 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.733 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.733 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:24:46.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:24:46.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6390000000000001e-06 2021-03-15T22:24:46.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081524+0000", 2021-03-15T22:24:46.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.863e-06 2021-03-15T22:24:46.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081526+0000", 2021-03-15T22:24:46.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1234e-05 2021-03-15T22:24:46.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081537+0000", 2021-03-15T22:24:46.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5210000000000002e-06 2021-03-15T22:24:46.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081543+0000", 2021-03-15T22:24:46.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045750591 2021-03-15T22:24:46.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127293+0000", 2021-03-15T22:24:46.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6920000000000008e-06 2021-03-15T22:24:46.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127302+0000", 2021-03-15T22:24:46.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3764999999999999e-05 2021-03-15T22:24:46.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127316+0000", 2021-03-15T22:24:46.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045793304999999999 2021-03-15T22:24:46.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.743 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.744 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:548 7.6 e205/175 ondisk, result = 0)", 2021-03-15T22:24:46.744 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:24:46.744 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 166.53935067800001, 2021-03-15T22:24:46.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041608198999999998, 2021-03-15T22:24:46.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.745 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.745 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:24:46.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:24:46.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4779999999999998e-06 2021-03-15T22:24:46.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085849+0000", 2021-03-15T22:24:46.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.046e-06 2021-03-15T22:24:46.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085851+0000", 2021-03-15T22:24:46.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4150000000000005e-06 2021-03-15T22:24:46.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085860+0000", 2021-03-15T22:24:46.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0900000000000001e-06 2021-03-15T22:24:46.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085866+0000", 2021-03-15T22:24:46.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041511365000000001 2021-03-15T22:24:46.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127378+0000", 2021-03-15T22:24:46.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6030000000000001e-06 2021-03-15T22:24:46.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127384+0000", 2021-03-15T22:24:46.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0202000000000003e-05 2021-03-15T22:24:46.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127454+0000", 2021-03-15T22:24:46.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041608198999999998 2021-03-15T22:24:46.755 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.755 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.755 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.756 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:61 7.5 e205/91 7:a9545d62:::smithi07694878-1:68 v 205'72, mlcod=202'69)", 2021-03-15T22:24:46.756 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:24:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 166.534326932, 2021-03-15T22:24:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034962251999999999, 2021-03-15T22:24:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:46.757 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:24:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:24:46.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.798e-06 2021-03-15T22:24:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090877+0000", 2021-03-15T22:24:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3779999999999999e-06 2021-03-15T22:24:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090879+0000", 2021-03-15T22:24:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1902999999999999e-05 2021-03-15T22:24:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090891+0000", 2021-03-15T22:24:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7780000000000004e-06 2021-03-15T22:24:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090897+0000", 2021-03-15T22:24:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0942000000000002e-05 2021-03-15T22:24:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090938+0000", 2021-03-15T22:24:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6379e-05 2021-03-15T22:24:46.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090954+0000", 2021-03-15T22:24:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034836759000000002 2021-03-15T22:24:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:46.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125791+0000", 2021-03-15T22:24:46.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1315e-05 2021-03-15T22:24:46.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125832+0000", 2021-03-15T22:24:46.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034962251999999999 2021-03-15T22:24:46.769 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.769 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.769 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.770 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.6.0:214 7.d e205/91 7:bd183e1b:::smithi07694878-46:68 v 205'30, mlcod=202'27)", 2021-03-15T22:24:46.770 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:24:46.771 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 166.53388505300001, 2021-03-15T22:24:46.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034834256000000001, 2021-03-15T22:24:46.771 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.773 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:46.773 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:24:46.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:24:46.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3590000000000002e-06 2021-03-15T22:24:46.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091314+0000", 2021-03-15T22:24:46.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.57e-06 2021-03-15T22:24:46.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091316+0000", 2021-03-15T22:24:46.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3749999999999999e-06 2021-03-15T22:24:46.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091322+0000", 2021-03-15T22:24:46.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2140000000000004e-06 2021-03-15T22:24:46.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091327+0000", 2021-03-15T22:24:46.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017519700000000001 2021-03-15T22:24:46.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091502+0000", 2021-03-15T22:24:46.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2955e-05 2021-03-15T22:24:46.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091525+0000", 2021-03-15T22:24:46.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034582099999999998 2021-03-15T22:24:46.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:46.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.126108+0000", 2021-03-15T22:24:46.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8485999999999997e-05 2021-03-15T22:24:46.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.126146+0000", 2021-03-15T22:24:46.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034834256000000001 2021-03-15T22:24:46.784 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.785 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.785 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.785 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1686 7.f 7:fee9f235:::smithi07694878-11:head [copy-from ver 242] snapc 7a=[7a,79,78,76,75] ondisk+write+known_if_redirected e220)", 2021-03-15T22:24:46.786 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:15.039251+0000", 2021-03-15T22:24:46.786 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 151.58594624400001, 2021-03-15T22:24:46.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026727377, 2021-03-15T22:24:46.786 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.787 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:46.787 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.787 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.787 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.787 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1686 2021-03-15T22:24:46.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.788 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039251+0000", 2021-03-15T22:24:46.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039251+0000", 2021-03-15T22:24:46.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0810000000000001e-06 2021-03-15T22:24:46.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039256+0000", 2021-03-15T22:24:46.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6880000000000001e-06 2021-03-15T22:24:46.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039259+0000", 2021-03-15T22:24:46.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7503000000000002e-05 2021-03-15T22:24:46.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039277+0000", 2021-03-15T22:24:46.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3869999999999997e-06 2021-03-15T22:24:46.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039284+0000", 2021-03-15T22:24:46.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8112999999999998e-05 2021-03-15T22:24:46.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039332+0000", 2021-03-15T22:24:46.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7928000000000002e-05 2021-03-15T22:24:46.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039430+0000", 2021-03-15T22:24:46.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067539030000000003 2021-03-15T22:24:46.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:24:46.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.046184+0000", 2021-03-15T22:24:46.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056836600000000001 2021-03-15T22:24:46.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:46.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.051868+0000", 2021-03-15T22:24:46.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008344677 2021-03-15T22:24:46.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.060212+0000", 2021-03-15T22:24:46.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056656900000000001 2021-03-15T22:24:46.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.065878+0000", 2021-03-15T22:24:46.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9669999999999997e-05 2021-03-15T22:24:46.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:46.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.065928+0000", 2021-03-15T22:24:46.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0077000000000002e-05 2021-03-15T22:24:46.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.065978+0000", 2021-03-15T22:24:46.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026727377 2021-03-15T22:24:46.805 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.805 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.806 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.806 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1687 7.f 7:fee9f235:::smithi07694878-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e220)", 2021-03-15T22:24:46.806 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:15.039297+0000", 2021-03-15T22:24:46.807 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 151.585900267, 2021-03-15T22:24:46.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026894929000000001, 2021-03-15T22:24:46.807 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.807 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.807 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.808 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.808 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.808 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1687 2021-03-15T22:24:46.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.809 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039297+0000", 2021-03-15T22:24:46.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039297+0000", 2021-03-15T22:24:46.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5549999999999999e-06 2021-03-15T22:24:46.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039298+0000", 2021-03-15T22:24:46.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2021-03-15T22:24:46.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039300+0000", 2021-03-15T22:24:46.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8300000000000001e-06 2021-03-15T22:24:46.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039306+0000", 2021-03-15T22:24:46.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0069999999999999e-06 2021-03-15T22:24:46.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039310+0000", 2021-03-15T22:24:46.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025812099999999999 2021-03-15T22:24:46.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039568+0000", 2021-03-15T22:24:46.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8504999999999997e-05 2021-03-15T22:24:46.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:46.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039616+0000", 2021-03-15T22:24:46.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010537019999999999 2021-03-15T22:24:46.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.050153+0000", 2021-03-15T22:24:46.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011407 2021-03-15T22:24:46.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:46.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.050267+0000", 2021-03-15T22:24:46.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015755377000000001 2021-03-15T22:24:46.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066023+0000", 2021-03-15T22:24:46.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2999999999999998e-05 2021-03-15T22:24:46.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066106+0000", 2021-03-15T22:24:46.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5745e-05 2021-03-15T22:24:46.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066191+0000", 2021-03-15T22:24:46.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026894929000000001 2021-03-15T22:24:46.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.824 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.824 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.825 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2103 7.f 7:fee9f235:::smithi07694878-11:head [copy-from ver 398] snapc 9a=[9a,97,93,92,8d,88,80,7d] ondisk+write+known_if_redirected e253)", 2021-03-15T22:24:46.826 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:24:46.826 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 118.413602071, 2021-03-15T22:24:46.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029938436999999998, 2021-03-15T22:24:46.826 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.827 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:46.827 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.827 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.827 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.827 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2103 2021-03-15T22:24:46.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.828 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:24:46.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:24:46.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1350000000000002e-06 2021-03-15T22:24:46.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211599+0000", 2021-03-15T22:24:46.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1489999999999998e-06 2021-03-15T22:24:46.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211602+0000", 2021-03-15T22:24:46.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8119000000000001e-05 2021-03-15T22:24:46.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211620+0000", 2021-03-15T22:24:46.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9529999999999999e-06 2021-03-15T22:24:46.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211628+0000", 2021-03-15T22:24:46.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015272800000000001 2021-03-15T22:24:46.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211781+0000", 2021-03-15T22:24:46.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0270000000000001e-06 2021-03-15T22:24:46.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:24:46.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211790+0000", 2021-03-15T22:24:46.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035614209999999999 2021-03-15T22:24:46.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215351+0000", 2021-03-15T22:24:46.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8361999999999994e-05 2021-03-15T22:24:46.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215420+0000", 2021-03-15T22:24:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013426624999999999 2021-03-15T22:24:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:24:46.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.228846+0000", 2021-03-15T22:24:46.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041102990000000004 2021-03-15T22:24:46.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:46.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232957+0000", 2021-03-15T22:24:46.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013870029999999999 2021-03-15T22:24:46.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.234344+0000", 2021-03-15T22:24:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007045791 2021-03-15T22:24:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241389+0000", 2021-03-15T22:24:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0001999999999998e-05 2021-03-15T22:24:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241459+0000", 2021-03-15T22:24:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3823000000000001e-05 2021-03-15T22:24:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241533+0000", 2021-03-15T22:24:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029938436999999998 2021-03-15T22:24:46.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2104 7.f 7:fee9f235:::smithi07694878-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e253)", 2021-03-15T22:24:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:24:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 118.413460335, 2021-03-15T22:24:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029977025000000001, 2021-03-15T22:24:46.847 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.848 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2104 2021-03-15T22:24:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:24:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:24:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.491e-06 2021-03-15T22:24:46.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211738+0000", 2021-03-15T22:24:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.813e-06 2021-03-15T22:24:46.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211740+0000", 2021-03-15T22:24:46.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2709999999999992e-06 2021-03-15T22:24:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211748+0000", 2021-03-15T22:24:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8559999999999998e-06 2021-03-15T22:24:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211753+0000", 2021-03-15T22:24:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014630000000000001 2021-03-15T22:24:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211899+0000", 2021-03-15T22:24:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7549999999999997e-06 2021-03-15T22:24:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:24:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211906+0000", 2021-03-15T22:24:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036139900000000001 2021-03-15T22:24:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215520+0000", 2021-03-15T22:24:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7696999999999999e-05 2021-03-15T22:24:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215548+0000", 2021-03-15T22:24:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016795027000000001 2021-03-15T22:24:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232343+0000", 2021-03-15T22:24:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1192000000000005e-05 2021-03-15T22:24:46.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232414+0000", 2021-03-15T22:24:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091605279999999994 2021-03-15T22:24:46.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241574+0000", 2021-03-15T22:24:46.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5976000000000007e-05 2021-03-15T22:24:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241640+0000", 2021-03-15T22:24:46.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3128999999999997e-05 2021-03-15T22:24:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241713+0000", 2021-03-15T22:24:46.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029977025000000001 2021-03-15T22:24:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2221 7.f 7:ffef77ba:::smithi07694878-13:head [sparse-read 0~2702901,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:24:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:24:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 108.57420872500001, 2021-03-15T22:24:46.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0065843969999999, 2021-03-15T22:24:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.868 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2221 2021-03-15T22:24:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:24:46.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:24:46.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0599999999999999e-06 2021-03-15T22:24:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050991+0000", 2021-03-15T22:24:46.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0379999999999998e-06 2021-03-15T22:24:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050993+0000", 2021-03-15T22:24:46.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6500000000000002e-06 2021-03-15T22:24:46.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051002+0000", 2021-03-15T22:24:46.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3650000000000003e-06 2021-03-15T22:24:46.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051007+0000", 2021-03-15T22:24:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015642 2021-03-15T22:24:46.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051164+0000", 2021-03-15T22:24:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7530000000000007e-06 2021-03-15T22:24:46.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:24:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051172+0000", 2021-03-15T22:24:46.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.004477362 2021-03-15T22:24:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.055650+0000", 2021-03-15T22:24:46.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134535 2021-03-15T22:24:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.055784+0000", 2021-03-15T22:24:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001788214 2021-03-15T22:24:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.057572+0000", 2021-03-15T22:24:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0065843969999999 2021-03-15T22:24:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2222 7.f 7:ffef77ba:::smithi07694878-13:head [read 0~2823609,checksum] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:24:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:24:46.883 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 108.574184102, 2021-03-15T22:24:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0081352180000001, 2021-03-15T22:24:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.884 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2222 2021-03-15T22:24:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.885 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:24:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:24:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6190000000000001e-06 2021-03-15T22:24:46.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051014+0000", 2021-03-15T22:24:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.217e-06 2021-03-15T22:24:46.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051016+0000", 2021-03-15T22:24:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8270000000000003e-06 2021-03-15T22:24:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051021+0000", 2021-03-15T22:24:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4529999999999998e-06 2021-03-15T22:24:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051025+0000", 2021-03-15T22:24:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000168445 2021-03-15T22:24:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051193+0000", 2021-03-15T22:24:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5860000000000001e-06 2021-03-15T22:24:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:24:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051200+0000", 2021-03-15T22:24:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0070519229999999 2021-03-15T22:24:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.058252+0000", 2021-03-15T22:24:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1403000000000006e-05 2021-03-15T22:24:46.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.058333+0000", 2021-03-15T22:24:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00081474499999999997 2021-03-15T22:24:46.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.059148+0000", 2021-03-15T22:24:46.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0081352180000001 2021-03-15T22:24:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2223 7.f 7:ffef77ba:::smithi07694878-13:head [sparse-read 0~2702901] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:24:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:24:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 108.574168179, 2021-03-15T22:24:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.009479572, 2021-03-15T22:24:46.899 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2223 2021-03-15T22:24:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:24:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:24:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7300000000000005e-07 2021-03-15T22:24:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:24:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.02e-06 2021-03-15T22:24:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051030+0000", 2021-03-15T22:24:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9280000000000001e-06 2021-03-15T22:24:46.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051035+0000", 2021-03-15T22:24:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.129e-06 2021-03-15T22:24:46.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051039+0000", 2021-03-15T22:24:46.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018268000000000001 2021-03-15T22:24:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051221+0000", 2021-03-15T22:24:46.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4270000000000002e-06 2021-03-15T22:24:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:24:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051229+0000", 2021-03-15T22:24:46.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.007961946 2021-03-15T22:24:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.059191+0000", 2021-03-15T22:24:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010123370000000001 2021-03-15T22:24:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.060203+0000", 2021-03-15T22:24:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030523200000000001 2021-03-15T22:24:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.060508+0000", 2021-03-15T22:24:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.009479572 2021-03-15T22:24:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.913 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3021 7.6 7:6de8b070:::smithi07694878-15:head [copy-from ver 640] snapc e3=[e1,df,de,dd,db] ondisk+write+known_if_redirected e330)", 2021-03-15T22:24:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:24:46.914 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.542578341999999, 2021-03-15T22:24:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.124784161, 2021-03-15T22:24:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:46.915 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3021 2021-03-15T22:24:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.916 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:24:46.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:24:46.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7749999999999999e-06 2021-03-15T22:24:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082622+0000", 2021-03-15T22:24:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.072e-06 2021-03-15T22:24:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082625+0000", 2021-03-15T22:24:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2641e-05 2021-03-15T22:24:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082638+0000", 2021-03-15T22:24:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9170000000000006e-06 2021-03-15T22:24:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082646+0000", 2021-03-15T22:24:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1200999999999997e-05 2021-03-15T22:24:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082727+0000", 2021-03-15T22:24:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7440000000000007e-06 2021-03-15T22:24:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:24:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082735+0000", 2021-03-15T22:24:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022422739999999998 2021-03-15T22:24:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.084977+0000", 2021-03-15T22:24:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010620000000000001 2021-03-15T22:24:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085083+0000", 2021-03-15T22:24:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10953049500000001 2021-03-15T22:24:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,7", 2021-03-15T22:24:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.194614+0000", 2021-03-15T22:24:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043572510000000004 2021-03-15T22:24:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198971+0000", 2021-03-15T22:24:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002724242 2021-03-15T22:24:46.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.201695+0000", 2021-03-15T22:24:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054890260000000001 2021-03-15T22:24:46.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207184+0000", 2021-03-15T22:24:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4066000000000006e-05 2021-03-15T22:24:46.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207248+0000", 2021-03-15T22:24:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015425700000000001 2021-03-15T22:24:46.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207403+0000", 2021-03-15T22:24:46.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.124784161 2021-03-15T22:24:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3022 7.6 7:6de8b070:::smithi07694878-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e330)", 2021-03-15T22:24:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:24:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 41.542537975000002, 2021-03-15T22:24:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12494472, 2021-03-15T22:24:46.935 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.936 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3022 2021-03-15T22:24:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:24:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:24:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2669999999999999e-06 2021-03-15T22:24:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082660+0000", 2021-03-15T22:24:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7409999999999999e-06 2021-03-15T22:24:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082662+0000", 2021-03-15T22:24:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2030000000000003e-06 2021-03-15T22:24:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082669+0000", 2021-03-15T22:24:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1479999999999999e-06 2021-03-15T22:24:46.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082673+0000", 2021-03-15T22:24:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037115800000000001 2021-03-15T22:24:46.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.083044+0000", 2021-03-15T22:24:46.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1779999999999996e-06 2021-03-15T22:24:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:24:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.083049+0000", 2021-03-15T22:24:46.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021797460000000002 2021-03-15T22:24:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085228+0000", 2021-03-15T22:24:46.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5878999999999999e-05 2021-03-15T22:24:46.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:46.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085274+0000", 2021-03-15T22:24:46.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.112992462 2021-03-15T22:24:46.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198267+0000", 2021-03-15T22:24:46.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6129000000000002e-05 2021-03-15T22:24:46.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:46.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198343+0000", 2021-03-15T22:24:46.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090944800000000003 2021-03-15T22:24:46.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207437+0000", 2021-03-15T22:24:46.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1060000000000001e-05 2021-03-15T22:24:46.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207508+0000", 2021-03-15T22:24:46.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5268999999999999e-05 2021-03-15T22:24:46.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207604+0000", 2021-03-15T22:24:46.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12494472 2021-03-15T22:24:46.953 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.953 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.954 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.954 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3170 7.6 7:6283974b:::smithi07694878-30:head [copy-from ver 282] snapc f4=[f4,f3,f2,e9,e7,e1,de] ondisk+write+known_if_redirected e344)", 2021-03-15T22:24:46.954 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:19.130807+0000", 2021-03-15T22:24:46.955 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.49438967, 2021-03-15T22:24:46.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026060351999999998, 2021-03-15T22:24:46.955 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.955 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:24:46.955 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.956 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.956 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.956 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3170 2021-03-15T22:24:46.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.957 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130807+0000", 2021-03-15T22:24:46.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130807+0000", 2021-03-15T22:24:46.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4670000000000001e-06 2021-03-15T22:24:46.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130809+0000", 2021-03-15T22:24:46.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9299999999999996e-07 2021-03-15T22:24:46.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130809+0000", 2021-03-15T22:24:46.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5999999999999997e-06 2021-03-15T22:24:46.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130815+0000", 2021-03-15T22:24:46.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5019999999999999e-06 2021-03-15T22:24:46.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130818+0000", 2021-03-15T22:24:46.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1529999999999997e-05 2021-03-15T22:24:46.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130859+0000", 2021-03-15T22:24:46.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0765000000000003e-05 2021-03-15T22:24:46.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130940+0000", 2021-03-15T22:24:46.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011008983999999999 2021-03-15T22:24:46.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:24:46.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.141949+0000", 2021-03-15T22:24:46.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075953360000000003 2021-03-15T22:24:46.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.149544+0000", 2021-03-15T22:24:46.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072008669999999997 2021-03-15T22:24:46.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:24:46.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156745+0000", 2021-03-15T22:24:46.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4639999999999995e-06 2021-03-15T22:24:46.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:24:46.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156755+0000", 2021-03-15T22:24:46.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9437999999999997e-05 2021-03-15T22:24:46.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:24:46.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156794+0000", 2021-03-15T22:24:46.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3505999999999997e-05 2021-03-15T22:24:46.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156867+0000", 2021-03-15T22:24:46.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026060351999999998 2021-03-15T22:24:46.973 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.973 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.973 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.974 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3171 7.6 7:6283974b:::smithi07694878-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e344)", 2021-03-15T22:24:46.974 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:19.130826+0000", 2021-03-15T22:24:46.974 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 27.494370768, 2021-03-15T22:24:46.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026195561999999999, 2021-03-15T22:24:46.975 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:24:46.975 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:24:46.975 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:24:46.976 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:24:46.976 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:24:46.976 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3171 2021-03-15T22:24:46.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.976 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:24:46.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:24:46.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130826+0000", 2021-03-15T22:24:46.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:24:46.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:24:46.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130826+0000", 2021-03-15T22:24:46.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6599999999999994e-07 2021-03-15T22:24:46.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:24:46.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130827+0000", 2021-03-15T22:24:46.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4200000000000004e-07 2021-03-15T22:24:46.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:24:46.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130828+0000", 2021-03-15T22:24:46.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1210000000000003e-06 2021-03-15T22:24:46.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:24:46.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130834+0000", 2021-03-15T22:24:46.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.676e-06 2021-03-15T22:24:46.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:24:46.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130838+0000", 2021-03-15T22:24:46.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023820699999999999 2021-03-15T22:24:46.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.131076+0000", 2021-03-15T22:24:46.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4833999999999998e-05 2021-03-15T22:24:46.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:24:46.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.131121+0000", 2021-03-15T22:24:46.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017834289999999999 2021-03-15T22:24:46.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.148955+0000", 2021-03-15T22:24:46.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0327000000000004e-05 2021-03-15T22:24:46.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:24:46.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.149045+0000", 2021-03-15T22:24:46.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078568969999999998 2021-03-15T22:24:46.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:24:46.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156902+0000", 2021-03-15T22:24:46.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6823999999999997e-05 2021-03-15T22:24:46.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:24:46.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156959+0000", 2021-03-15T22:24:46.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2477999999999994e-05 2021-03-15T22:24:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:24:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:24:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:24:46.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.157022+0000", 2021-03-15T22:24:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026195561999999999 2021-03-15T22:24:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.992 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:24:46.993 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:24:46.993 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:24:46.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:46.995 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:46.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:46.996 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:24:46.997 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:24:46.997 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:24:46.997 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:24:46.997 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:24:46.998 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:24:46.998 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:24:46.998 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:24:46.999 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:24:46.999 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:24:46.999 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:24:46.999 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:24:46.999 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:24:47.000 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:24:47.000 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:24:47.000 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:24:47.001 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:46 smithi125 conmon[56739]: cluster 2021-03-15T22:24:45.042975+0000 mgr.y (mgr.14144) 582 : 2021-03-15T22:24:47.001 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:46 smithi125 conmon[56739]: cluster [DBG] pgmap v681: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 236 MiB data, 720 MiB used, 625 GiB / 626 GiB avail; 5.0 MiB/s rd, 2.0 MiB/s wr, 7 op/s; 1/1272 objects misplaced (0.079%) 2021-03-15T22:24:47.001 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:46 smithi125 conmon[56739]: cluster 2021-03-15T22:24:46.093531+0000 mon.a (mon.0) 1144 : cluster [DBG] osdmap e371: 8 total, 8 up, 7 in 2021-03-15T22:24:47.001 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:46 smithi125 conmon[56739]: 2021-03-15T22:24:47.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:47.101 INFO:tasks.rados.rados.0.smithi076.stdout:1326: done (1 left) 2021-03-15T22:24:47.101 INFO:tasks.rados.rados.0.smithi076.stdout:1327: done (0 left) 2021-03-15T22:24:47.101 INFO:tasks.rados.rados.0.smithi076.stdout:1328: rollback oid 20 current snap is 142 2021-03-15T22:24:47.102 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 20 to 134 2021-03-15T22:24:47.102 INFO:tasks.rados.rados.0.smithi076.stdout:1329: rollback oid 29 current snap is 142 2021-03-15T22:24:47.102 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 29 to 131 2021-03-15T22:24:47.103 INFO:tasks.rados.rados.0.smithi076.stdout:1330: write (excl) oid 37 current snap is 142 2021-03-15T22:24:47.103 INFO:tasks.rados.rados.0.smithi076.stdout:1330: seq_num 476 ranges {400132=632410,1438017=546162} 2021-03-15T22:24:47.106 INFO:tasks.rados.rados.0.smithi076.stdout:1330: writing smithi07694878-37 from 400132 to 1032542 tid 1 2021-03-15T22:24:47.113 INFO:tasks.rados.rados.0.smithi076.stdout:1330: writing smithi07694878-37 from 1438017 to 1984179 tid 2 2021-03-15T22:24:47.113 INFO:tasks.rados.rados.0.smithi076.stdout:1331: write oid 8 current snap is 142 2021-03-15T22:24:47.113 INFO:tasks.rados.rados.0.smithi076.stdout:1328: finishing rollback tid 0 to smithi07694878-20 2021-03-15T22:24:47.114 INFO:tasks.rados.rados.0.smithi076.stdout:1331: seq_num 477 ranges {416939=707659,1896499=120440} 2021-03-15T22:24:47.118 INFO:tasks.rados.rados.0.smithi076.stdout:1331: writing smithi07694878-8 from 416939 to 1124598 tid 1 2021-03-15T22:24:47.123 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:47.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:47.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:47.127 INFO:tasks.rados.rados.0.smithi076.stdout:1331: writing smithi07694878-8 from 1896499 to 2016939 tid 2 2021-03-15T22:24:47.128 INFO:tasks.rados.rados.0.smithi076.stdout:1332: read oid 28 snap -1 2021-03-15T22:24:47.128 INFO:tasks.rados.rados.0.smithi076.stdout:1332: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:24:47.129 INFO:tasks.rados.rados.0.smithi076.stdout:1333: read oid 50 snap 133 2021-03-15T22:24:47.129 INFO:tasks.rados.rados.0.smithi076.stdout:1333: expect deleted 2021-03-15T22:24:47.129 INFO:tasks.rados.rados.0.smithi076.stdout:1334: delete oid 34 current snap is 142 2021-03-15T22:24:47.129 INFO:tasks.rados.rados.0.smithi076.stdout:1328: finishing rollback tid 1 to smithi07694878-20 2021-03-15T22:24:47.130 INFO:tasks.rados.rados.0.smithi076.stdout:1328: finishing rollback tid 2 to smithi07694878-20 2021-03-15T22:24:47.130 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 453 (ObjNum 449 snap 128 seq_num 449) dirty exists 2021-03-15T22:24:47.130 INFO:tasks.rados.rados.0.smithi076.stdout:1329: finishing rollback tid 0 to smithi07694878-29 2021-03-15T22:24:47.130 INFO:tasks.rados.rados.0.smithi076.stdout:1329: finishing rollback tid 1 to smithi07694878-29 2021-03-15T22:24:47.131 INFO:tasks.rados.rados.0.smithi076.stdout:1329: finishing rollback tid 2 to smithi07694878-29 2021-03-15T22:24:47.131 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 793 (ObjNum 397 snap 108 seq_num 397) dirty exists 2021-03-15T22:24:47.132 INFO:tasks.rados.rados.0.smithi076.stdout:1330: finishing write tid 1 to smithi07694878-37 2021-03-15T22:24:47.133 INFO:tasks.rados.rados.0.smithi076.stdout:1330: finishing write tid 2 to smithi07694878-37 2021-03-15T22:24:47.134 INFO:tasks.rados.rados.0.smithi076.stdout:1330: finishing write tid 3 to smithi07694878-37 2021-03-15T22:24:47.134 INFO:tasks.rados.rados.0.smithi076.stdout:1330: finishing write tid 4 to smithi07694878-37 2021-03-15T22:24:47.134 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 796 (ObjNum 476 snap 142 seq_num 476) dirty exists 2021-03-15T22:24:47.134 INFO:tasks.rados.rados.0.smithi076.stdout:1330: left oid 37 (ObjNum 476 snap 142 seq_num 476) 2021-03-15T22:24:47.135 INFO:tasks.rados.rados.0.smithi076.stdout:1331: finishing write tid 1 to smithi07694878-8 2021-03-15T22:24:47.135 INFO:tasks.rados.rados.0.smithi076.stdout:1331: finishing write tid 2 to smithi07694878-8 2021-03-15T22:24:47.135 INFO:tasks.rados.rados.0.smithi076.stdout:1328: done (6 left) 2021-03-15T22:24:47.135 INFO:tasks.rados.rados.0.smithi076.stdout:1329: done (5 left) 2021-03-15T22:24:47.136 INFO:tasks.rados.rados.0.smithi076.stdout:1330: done (4 left) 2021-03-15T22:24:47.136 INFO:tasks.rados.rados.0.smithi076.stdout:1333: done (3 left) 2021-03-15T22:24:47.137 INFO:tasks.rados.rados.0.smithi076.stdout:1334: done (2 left) 2021-03-15T22:24:47.137 INFO:tasks.rados.rados.0.smithi076.stdout:1335: read oid 6 snap -1 2021-03-15T22:24:47.137 INFO:tasks.rados.rados.0.smithi076.stdout:1335: expect (ObjNum 429 snap 119 seq_num 429) 2021-03-15T22:24:47.137 INFO:tasks.rados.rados.0.smithi076.stdout:1336: delete oid 50 current snap is 142 2021-03-15T22:24:47.137 INFO:tasks.rados.rados.0.smithi076.stdout:1331: finishing write tid 3 to smithi07694878-8 2021-03-15T22:24:47.138 INFO:tasks.rados.rados.0.smithi076.stdout:1331: finishing write tid 4 to smithi07694878-8 2021-03-15T22:24:47.138 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 758 (ObjNum 477 snap 142 seq_num 477) dirty exists 2021-03-15T22:24:47.138 INFO:tasks.rados.rados.0.smithi076.stdout:1331: left oid 8 (ObjNum 477 snap 142 seq_num 477) 2021-03-15T22:24:47.139 INFO:tasks.rados.rados.0.smithi076.stdout:1335: expect (ObjNum 429 snap 119 seq_num 429) 2021-03-15T22:24:47.228 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:47.252 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:47.252 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:47.266 INFO:tasks.rados.rados.0.smithi076.stdout:1332: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:24:47.353 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:47.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:47.377 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:47.378 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:47.467 INFO:tasks.rados.rados.0.smithi076.stdout:1331: done (3 left) 2021-03-15T22:24:47.468 INFO:tasks.rados.rados.0.smithi076.stdout:1332: done (2 left) 2021-03-15T22:24:47.468 INFO:tasks.rados.rados.0.smithi076.stdout:1335: done (1 left) 2021-03-15T22:24:47.468 INFO:tasks.rados.rados.0.smithi076.stdout:1336: done (0 left) 2021-03-15T22:24:47.468 INFO:tasks.rados.rados.0.smithi076.stdout:1337: delete oid 16 current snap is 142 2021-03-15T22:24:47.470 INFO:tasks.rados.rados.0.smithi076.stdout:1337: done (0 left) 2021-03-15T22:24:47.470 INFO:tasks.rados.rados.0.smithi076.stdout:1338: write oid 21 current snap is 142 2021-03-15T22:24:47.471 INFO:tasks.rados.rados.0.smithi076.stdout:1338: seq_num 481 ranges {484167=725008,1730427=353740} 2021-03-15T22:24:47.476 INFO:tasks.rados.rados.0.smithi076.stdout:1338: writing smithi07694878-21 from 484167 to 1209175 tid 1 2021-03-15T22:24:47.479 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:47.482 INFO:tasks.rados.rados.0.smithi076.stdout:1338: writing smithi07694878-21 from 1730427 to 2084167 tid 2 2021-03-15T22:24:47.482 INFO:tasks.rados.rados.0.smithi076.stdout:1339: snap_remove snap 131 2021-03-15T22:24:47.501 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:47.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:47.503 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:47.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56514]: audit 2021-03-15T22:24:46.570785+0000 mon.a (mon.0) 1145 : audit [DBG] 2021-03-15T22:24:47.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:24:47.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56514]: audit 2021-03-15T22:24:46.825121+0000 mon.a (mon.0) 1146 : audit [DBG] from='client.? 172.21.15.76:0/2673542729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:47.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56514]: audit 2021-03-15T22:24:46.871405+0000 mon.a (mon.0) 1147 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:24:47.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:24:47.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56514]: :24:47.093159+0000 mon.a (mon.0) 1148 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2021-03-15T22:24:47.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56514]: audit 2021-03-15T22:24:47.155100+0000 mon.a (mon.0) 1149 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:24:47.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: audit 2021-03-15T22:24:46.570785+0000 mon.a (mon.0) 1145 : 2021-03-15T22:24:47.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:24:47.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: audit 2021-03-15T22:24:46.825121+0000 mon.a (mon.0) 1146 : audit 2021-03-15T22:24:47.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/2673542729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:47.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:24:47.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: :24:46.871405+0000 mon.a (mon.0) 1147 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:24:47.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:24:47.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: 03-15T22:24:47.093159+0000 mon.a 2021-03-15T22:24:47.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: (mon.0) 1148 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2021-03-15T22:24:47.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:24:47.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: -15T22:24:47.155100+0000 mon.a ( 2021-03-15T22:24:47.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[62156]: mon.0) 1149 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:24:47.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: debug 2021-03-15T22:24:47.154+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:24:47.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:24:47.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:47.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:24:47.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:47.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:24:47.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:47.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:47.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:24:47.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: debug 2021-03-15T22:24:47.154+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:24:47.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:24:47.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:47.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:24:47.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:47.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:24:47.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:47.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:47.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:24:47.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: debug 2021-03-15T22:24:47.155+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:24:47.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:24:47.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:47.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:24:47.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:47.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:24:47.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:47.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:47.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:24:47.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: debug 2021-03-15T22:24:47.155+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:24:47.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:24:47.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:47.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:24:47.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:47.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:24:47.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:47.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:47.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:24:47.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: debug 2021-03-15T22:24:47.155+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:24:47.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:24:47.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:47.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:24:47.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:47.533 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:24:47.533 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:47.533 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:47.533 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:24:47.534 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: debug 2021-03-15T22:24:47.156+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:24:47.534 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:24:47.534 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:47.534 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:24:47.535 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:47.535 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:24:47.535 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:47.535 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:47.536 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:24:47.536 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: debug 2021-03-15T22:24:47.156+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:24:47.536 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:24:47.536 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:47.537 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:24:47.537 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:47.537 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:24:47.537 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:47.538 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:47.538 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:24:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:24:47.603 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:47.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:47.628 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:47.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:47.729 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:47.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:47.752 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:47.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:47.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:47 smithi125 conmon[56739]: audit 2021-03-15T22:24:46.570785+0000 mon.a (mon. 2021-03-15T22:24:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:47 smithi125 conmon[56739]: 0) 1145 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:24:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:47 smithi125 conmon[56739]: audit 2021-03-15T22:24:46.825121+0000 mon.a (mon.0) 1146 : audit [DBG] from='client.? 172.21.15.76:0/2673542729' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:47 smithi125 conmon[56739]: audit 2021-03-15T22:24:46.871405+0000 mon.a (mon.0) 1147 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:24:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:47 smithi125 conmon[56739]: cluster 2021-03-15T22:24:47.093159+0000 mon.a (mon.0) 1148 : cluster [DBG] osdmap e372: 8 total, 8 up, 7 in 2021-03-15T22:24:47.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:47 smithi125 conmon[56739]: audit 2021-03-15T22:24:47.155100+0000 mon.a (mon.0) 1149 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:24:47.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:47.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:47.878 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:47.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:47.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:48.003 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:48.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:48.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:48.105 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:48.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:48.129 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:48.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:48.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:24:48.231 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:48.254 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:48.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:48.255 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:48.333 INFO:tasks.rados.rados.0.smithi076.stdout:1339: done (1 left) 2021-03-15T22:24:48.334 INFO:tasks.rados.rados.0.smithi076.stdout:1340: rollback oid 22 current snap is 142 2021-03-15T22:24:48.334 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 22 to 140 2021-03-15T22:24:48.334 INFO:tasks.rados.rados.0.smithi076.stdout:1341: read oid 18 snap -1 2021-03-15T22:24:48.335 INFO:tasks.rados.rados.0.smithi076.stdout:1341: expect (ObjNum 324 snap 87 seq_num 324) 2021-03-15T22:24:48.335 INFO:tasks.rados.rados.0.smithi076.stdout:1342: read oid 32 snap -1 2021-03-15T22:24:48.335 INFO:tasks.rados.rados.0.smithi076.stdout:1342: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:24:48.335 INFO:tasks.rados.rados.0.smithi076.stdout:1343: copy_from oid 49 from oid 14 current snap is 142 2021-03-15T22:24:48.336 INFO:tasks.rados.rados.0.smithi076.stdout:1344: copy_from oid 50 from oid 17 current snap is 142 2021-03-15T22:24:48.336 INFO:tasks.rados.rados.0.smithi076.stdout:1345: write (excl) oid 27 current snap is 142 2021-03-15T22:24:48.336 INFO:tasks.rados.rados.0.smithi076.stdout:1345: seq_num 482 ranges {500974=400257,1388909=557169} 2021-03-15T22:24:48.337 INFO:tasks.rados.rados.0.smithi076.stdout:1345: writing smithi07694878-27 from 500974 to 901231 tid 1 2021-03-15T22:24:48.342 INFO:tasks.rados.rados.0.smithi076.stdout:1345: writing smithi07694878-27 from 1388909 to 1946078 tid 2 2021-03-15T22:24:48.342 INFO:tasks.rados.rados.0.smithi076.stdout:1346: read oid 29 snap -1 2021-03-15T22:24:48.343 INFO:tasks.rados.rados.0.smithi076.stdout:1346: expect (ObjNum 397 snap 108 seq_num 397) 2021-03-15T22:24:48.343 INFO:tasks.rados.rados.0.smithi076.stdout:1347: snap_remove snap 126 2021-03-15T22:24:48.343 INFO:tasks.rados.rados.0.smithi076.stdout:1338: finishing write tid 1 to smithi07694878-21 2021-03-15T22:24:48.343 INFO:tasks.rados.rados.0.smithi076.stdout:1338: finishing write tid 2 to smithi07694878-21 2021-03-15T22:24:48.344 INFO:tasks.rados.rados.0.smithi076.stdout:1338: finishing write tid 3 to smithi07694878-21 2021-03-15T22:24:48.344 INFO:tasks.rados.rados.0.smithi076.stdout:1338: finishing write tid 4 to smithi07694878-21 2021-03-15T22:24:48.344 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 800 (ObjNum 481 snap 142 seq_num 481) dirty exists 2021-03-15T22:24:48.344 INFO:tasks.rados.rados.0.smithi076.stdout:1338: left oid 21 (ObjNum 481 snap 142 seq_num 481) 2021-03-15T22:24:48.345 INFO:tasks.rados.rados.0.smithi076.stdout:1340: finishing rollback tid 0 to smithi07694878-22 2021-03-15T22:24:48.345 INFO:tasks.rados.rados.0.smithi076.stdout:1341: expect (ObjNum 324 snap 87 seq_num 324) 2021-03-15T22:24:48.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:48.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:48.382 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:48.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:48.484 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:48.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:48.508 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:48.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:48.543 INFO:tasks.rados.rados.0.smithi076.stdout:1345: finishing write tid 1 to smithi07694878-27 2021-03-15T22:24:48.543 INFO:tasks.rados.rados.0.smithi076.stdout:1340: finishing rollback tid 1 to smithi07694878-22 2021-03-15T22:24:48.543 INFO:tasks.rados.rados.0.smithi076.stdout:1340: finishing rollback tid 2 to smithi07694878-22 2021-03-15T22:24:48.544 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 768 (ObjNum 466 snap 136 seq_num 466) dirty exists 2021-03-15T22:24:48.544 INFO:tasks.rados.rados.0.smithi076.stdout:1342: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:24:48.610 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:48.632 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:48.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:48.634 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:48.644 INFO:tasks.rados.rados.0.smithi076.stdout:1345: finishing write tid 2 to smithi07694878-27 2021-03-15T22:24:48.644 INFO:tasks.rados.rados.0.smithi076.stdout:1345: finishing write tid 3 to smithi07694878-27 2021-03-15T22:24:48.645 INFO:tasks.rados.rados.0.smithi076.stdout:1345: finishing write tid 4 to smithi07694878-27 2021-03-15T22:24:48.645 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 278 (ObjNum 482 snap 142 seq_num 482) dirty exists 2021-03-15T22:24:48.645 INFO:tasks.rados.rados.0.smithi076.stdout:1345: left oid 27 (ObjNum 482 snap 142 seq_num 482) 2021-03-15T22:24:48.735 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:48.757 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:48.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:48.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:48.843 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:48.860 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:48.883 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:48.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:48.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:48.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:49.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:49.009 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:49.009 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:49.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:49.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:49.136 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:49.137 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:49.214 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: cluster 2021-03-15T22:24: 2021-03-15T22:24:49.214 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: 47.043455+0000 mgr.y (mgr.14144) 583 : cluster [DBG] pgmap v684: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 238 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 14 op/s; 1/1281 objects misplaced (0.078%) 2021-03-15T22:24:49.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:24:47.151705+0000 mgr.y (mgr.14144) 584 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:24:49.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:24:47.156015+0000 mgr.y (mgr.14144) 585 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:24:49.215 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:24:49.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:24:49.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.216 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:24:49.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:24:49.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:24:47.156390+0000 mgr.y (mgr.14144) 586 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:24:49.218 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:24:49.218 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.218 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:24:49.218 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:24:49.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:24:49.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:24:47.156738+0000 mgr.y (mgr.14144) 587 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:24:49.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:24:49.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.220 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:24:49.221 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: cluster 2021-03-15T22:24 2021-03-15T22:24:49.221 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: :47.043455+0000 mgr.y (mgr.14144) 583 : cluster [DBG] pgmap v684: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 238 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 14 op/s; 1/1281 objects misplaced (0.078%) 2021-03-15T22:24:49.221 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:24:47. 2021-03-15T22:24:49.222 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: 151705+0000 mgr.y (mgr.14144) 584 : cephadm [INF] 2021-03-15T22:24:49.222 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:24:49.222 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:24:49.222 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: 2021-03-15T22:24:47.156015+0000 2021-03-15T22:24:49.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: mgr.y (mgr.14144) 585 : 2021-03-15T22:24:49.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:24:49.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:24:49.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.224 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:24:49.224 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.224 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:24:49.224 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:24:49.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:24:47.156390+0000 mgr.y (mgr.14144) 586 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:24:49.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:24:49.226 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.226 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:24:49.226 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.226 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:24:49.227 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.227 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.227 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:24:49.227 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:24:47.156738+0000 mgr.y (mgr.14144) 587 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:24:49.228 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:24:49.228 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.229 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:24:49.229 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.229 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.229 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:24:49.230 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:24:47.157178+0000 mgr.y (mgr.14144) 588 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:24:49.230 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:24:49.230 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.230 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:24:49.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:24:49.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.231 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.232 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:24:49.232 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:24:47.157682+0000 mgr.y (mgr.14144) 589 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:24:49.232 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:24:49.232 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:24:49.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:24:49.233 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:24:49.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:24:47.158126+0000 mgr.y (mgr.14144) 590 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:24:49.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:24:49.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:24:49.235 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:24:49.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.236 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.237 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.237 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:24:49.238 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.238 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:24:49.238 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.238 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.239 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:24:49.239 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:24:47.157178+0000 mgr.y (mgr.14144) 588 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:24:49.239 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:24:49.239 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.240 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:24:49.240 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.240 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:24:49.240 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.241 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.241 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:24:49.241 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:24:47.157682+0000 mgr.y (mgr.14144) 589 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:24:49.241 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:24:49.242 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.242 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:24:49.242 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.242 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:24:49.243 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.243 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.243 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:24:49.243 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:24:47.158126+0000 mgr.y (mgr.14144) 590 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:24:49.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:24:49.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:24:49.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.244 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:24:49.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:24:49.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:24:47.158524+0000 mgr.y (mgr.14144) 591 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:24:49.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:24:49.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:24:49.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:24:49.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:24:49.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[56514]: cluster 2021-03-15T22:24:48.323339+0000 mon.a (mon.0) 1150 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2021-03-15T22:24:49.248 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.248 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.248 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:24:49.249 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:24:47.158524+0000 mgr.y (mgr.14144) 591 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:24:49.249 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:24:49.249 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.249 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:24:49.249 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.250 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:24:49.252 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.254 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:24:49.254 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: cluster 2021-03-15T22:24:48.323339+0000 2021-03-15T22:24:49.254 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: mon.a (mon.0) 1150 : cluster 2021-03-15T22:24:49.254 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:48 smithi076 conmon[62156]: [DBG] osdmap e373: 8 total, 8 up, 7 in 2021-03-15T22:24:49.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:49.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:49.278 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:49.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:49.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: cluster 2021-03-15T22:24:47.043455+0000 mgr.y (mgr.14144) 583 : 2021-03-15T22:24:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: cluster [DBG] pgmap v684: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 238 MiB data, 723 MiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 1.7 MiB/s wr, 14 op/s; 1/1281 objects misplaced (0.078%) 2021-03-15T22:24:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:24:47.151705+0000 mgr.y (mgr.14144) 584 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:24:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:24: 2021-03-15T22:24:49.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: 47.156015+0000 mgr.y (mgr.14144) 585 : cephadm 2021-03-15T22:24:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:24:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:24:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:24:49.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:24:49.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:24:49.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:24:47.156390+0000 mgr.y (mgr.14144) 586 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:24:49.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:24:49.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.290 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:24:49.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:24:49.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.291 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:24:49.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:24:47. 2021-03-15T22:24:49.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: 156738+0000 mgr.y (mgr.14144) 587 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:24:49.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:24:49.292 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:24:49.293 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:24:49.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.294 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:24:49.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:24:47.157178+0000 mgr.y (mgr.14144) 588 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:24:49.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:24:49.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:24:49.295 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:24:49.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.296 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:24:49.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:24:47.157682+0000 mgr.y (mgr.14144) 589 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:24:49.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:24:49.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:24:49.297 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:24:49.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:24:49.298 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:24:47.158126+0000 mgr.y (mgr.14144) 590 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:24:49.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:24:49.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:24:49.299 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.300 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:24:49.300 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.300 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.300 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:24:49.301 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:24:47.158524+0000 mgr.y (mgr.14144) 591 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:24:49.301 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:24:49.301 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:24:49.301 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:24:49.302 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:24:49.302 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:24:49.302 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:24:49.302 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:24:49.302 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:24:49.303 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:48 smithi125 conmon[56739]: cluster 2021-03-15T22:24:48.323339+0000 mon.a (mon.0) 1150 : cluster [DBG] osdmap e373: 8 total, 8 up, 7 in 2021-03-15T22:24:49.326 INFO:tasks.rados.rados.0.smithi076.stdout:1338: done (8 left) 2021-03-15T22:24:49.327 INFO:tasks.rados.rados.0.smithi076.stdout:1340: done (7 left) 2021-03-15T22:24:49.327 INFO:tasks.rados.rados.0.smithi076.stdout:1341: done (6 left) 2021-03-15T22:24:49.327 INFO:tasks.rados.rados.0.smithi076.stdout:1342: done (5 left) 2021-03-15T22:24:49.327 INFO:tasks.rados.rados.0.smithi076.stdout:1345: done (4 left) 2021-03-15T22:24:49.328 INFO:tasks.rados.rados.0.smithi076.stdout:1347: done (3 left) 2021-03-15T22:24:49.328 INFO:tasks.rados.rados.0.smithi076.stdout:1348: copy_from oid 30 from oid 25 current snap is 142 2021-03-15T22:24:49.328 INFO:tasks.rados.rados.0.smithi076.stdout:1346: expect (ObjNum 397 snap 108 seq_num 397) 2021-03-15T22:24:49.328 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:24:49.329 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 1 1 2021-03-15T22:24:49.379 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:49.402 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:49.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:49.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:49.453 INFO:tasks.rados.rados.0.smithi076.stdout:1343: finishing copy_from to smithi07694878-49 2021-03-15T22:24:49.453 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 653 (ObjNum 453 snap 132 seq_num 453) dirty exists 2021-03-15T22:24:49.454 INFO:tasks.rados.rados.0.smithi076.stdout:1343: finishing copy_from racing read to smithi07694878-49 2021-03-15T22:24:49.454 INFO:tasks.rados.rados.0.smithi076.stdout:1344: finishing copy_from to smithi07694878-50 2021-03-15T22:24:49.454 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 104 (ObjNum 422 snap 116 seq_num 422) dirty exists 2021-03-15T22:24:49.455 INFO:tasks.rados.rados.0.smithi076.stdout:1344: finishing copy_from racing read to smithi07694878-50 2021-03-15T22:24:49.455 INFO:tasks.rados.rados.0.smithi076.stdout:1343: done (3 left) 2021-03-15T22:24:49.455 INFO:tasks.rados.rados.0.smithi076.stdout:1344: done (2 left) 2021-03-15T22:24:49.456 INFO:tasks.rados.rados.0.smithi076.stdout:1346: done (1 left) 2021-03-15T22:24:49.456 INFO:tasks.rados.rados.0.smithi076.stdout:1349: read oid 19 snap -1 2021-03-15T22:24:49.456 INFO:tasks.rados.rados.0.smithi076.stdout:1349: expect deleted 2021-03-15T22:24:49.457 INFO:tasks.rados.rados.0.smithi076.stdout:1350: copy_from oid 17 from oid 41 current snap is 142 2021-03-15T22:24:49.457 INFO:tasks.rados.rados.0.smithi076.stdout:1348: finishing copy_from to smithi07694878-30 2021-03-15T22:24:49.457 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 291 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-03-15T22:24:49.457 INFO:tasks.rados.rados.0.smithi076.stdout:1348: finishing copy_from racing read to smithi07694878-30 2021-03-15T22:24:49.458 INFO:tasks.rados.rados.0.smithi076.stdout:1348: done (2 left) 2021-03-15T22:24:49.458 INFO:tasks.rados.rados.0.smithi076.stdout:1351: snap_create 2021-03-15T22:24:49.466 INFO:tasks.rados.rados.0.smithi076.stdout:1350: finishing copy_from to smithi07694878-17 2021-03-15T22:24:49.467 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 304 (ObjNum 438 snap 122 seq_num 438) dirty exists 2021-03-15T22:24:49.467 INFO:tasks.rados.rados.0.smithi076.stdout:1350: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:24:49.504 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:49.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:49.527 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:49.528 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:49.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:49.652 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:49.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:49.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:49.754 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:49.778 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:49.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:49.779 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:49.881 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:49.903 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:49.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:49.905 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:50.006 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:50.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:50.029 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:50.029 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:50.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:24:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:24:50.087 INFO:tasks.rados.rados.0.smithi076.stdout:1349: done (2 left) 2021-03-15T22:24:50.088 INFO:tasks.rados.rados.0.smithi076.stdout:1350: done (1 left) 2021-03-15T22:24:50.088 INFO:tasks.rados.rados.0.smithi076.stdout:1351: done (0 left) 2021-03-15T22:24:50.088 INFO:tasks.rados.rados.0.smithi076.stdout:1352: delete oid 13 current snap is 143 2021-03-15T22:24:50.094 INFO:tasks.rados.rados.0.smithi076.stdout:1352: done (0 left) 2021-03-15T22:24:50.095 INFO:tasks.rados.rados.0.smithi076.stdout:1353: snap_remove snap 136 2021-03-15T22:24:50.131 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:50.154 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:50.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:50.156 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:50.180 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:50.257 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:50.280 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:50.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:50.281 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:50.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:50 smithi076 conmon[56514]: cluster 2021-03-15T22:24:49.043830+0000 mgr.y (mgr.14144) 592 : cluster [DBG] pgmap v687: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 241 MiB data, 732 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 967 KiB/s wr, 17 op/s; 1/1290 objects misplaced (0.078%) 2021-03-15T22:24:50.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:50 smithi076 conmon[56514]: cluster 2021-03-15T22:24:49.324016+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2021-03-15T22:24:50.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:50 smithi076 conmon[56514]: audit 2021-03-15T22:24:49.669479+0000 mgr.y (mgr.14144) 593 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:50.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:50 smithi076 conmon[56514]: cluster 2021-03-15T22:24:50.083841+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2021-03-15T22:24:50.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:50 smithi076 conmon[56514]: audit 2021-03-15T22:24:50.118368+0000 mon.a (mon.0) 1153 : audit [INF] from='client.? 172.21.15.76:0/2671630730' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:50.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:50 smithi076 conmon[62156]: cluster 2021-03-15T22:24:49.043830+0000 mgr.y (mgr.14144) 592 : cluster [DBG] pgmap v687: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 241 MiB data, 732 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 967 KiB/s wr, 17 op/s; 1/1290 objects misplaced (0.078%) 2021-03-15T22:24:50.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:50 smithi076 conmon[62156]: cluster 2021-03-15T22:24:49.324016+0000 mon.a (mon.0) 1151 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2021-03-15T22:24:50.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:50 smithi076 conmon[62156]: audit 2021-03-15T22:24:49.669479+0000 mgr.y (mgr.14144) 593 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:50.379 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:50 smithi076 conmon[62156]: cluster 2021-03-15T22:24:50.083841+0000 mon.a (mon.0) 1152 : cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2021-03-15T22:24:50.379 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:50 smithi076 conmon[62156]: audit 2021-03-15T22:24:50.118368+0000 mon.a (mon.0) 1153 : audit [INF] from='client.? 172.21.15.76:0/2671630730' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:50.382 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:50.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:50.406 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:50.406 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:50.507 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:50.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:50.530 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:50.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:50.545 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:24:50.556 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:24:50.556 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:24:50.632 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:50.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:50.660 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:50.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:50.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:50.785 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:50.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:50.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: cluster 2021-03-15T22:24:49.043830+0000 mgr.y (mgr 2021-03-15T22:24:50.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: .14144) 592 : cluster [DBG] pgmap v687: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 241 MiB data, 732 MiB used, 625 GiB / 626 GiB avail; 18 MiB/s rd, 967 KiB/s wr, 17 op/s; 1/1290 objects misplaced (0.078%) 2021-03-15T22:24:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: cluster 2021-03-15T22:24:49.324016+0000 2021-03-15T22:24:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: mon.a (mon.0) 1151 : cluster [DBG] osdmap e374: 8 total, 8 up, 7 in 2021-03-15T22:24:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:24:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: -15T22:24:49.669479 2021-03-15T22:24:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 593 : audit 2021-03-15T22:24:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:24:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:24:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: -15T22:24:50. 2021-03-15T22:24:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: 083841+0000 mon.a (mon.0) 1152 : 2021-03-15T22:24:50.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: cluster [DBG] osdmap e375: 8 total, 8 up, 7 in 2021-03-15T22:24:50.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:24:50.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: 03-15T22:24:50.118368 2021-03-15T22:24:50.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1153 2021-03-15T22:24:50.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:50 smithi125 conmon[56739]: : audit [INF] from='client.? 172.21.15.76:0/2671630730' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:24:50.886 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:50.908 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:50.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:50.910 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:51.011 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:51.033 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:51.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:51.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:51.098 INFO:tasks.rados.rados.0.smithi076.stdout:1353: done (0 left) 2021-03-15T22:24:51.098 INFO:tasks.rados.rados.0.smithi076.stdout:1354: delete oid 43 current snap is 143 2021-03-15T22:24:51.100 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:24:51.102 INFO:tasks.rados.rados.0.smithi076.stdout:1354: done (0 left) 2021-03-15T22:24:51.102 INFO:tasks.rados.rados.0.smithi076.stdout:1355: read oid 5 snap -1 2021-03-15T22:24:51.103 INFO:tasks.rados.rados.0.smithi076.stdout:1355: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:24:51.103 INFO:tasks.rados.rados.0.smithi076.stdout:1356: snap_remove snap 134 2021-03-15T22:24:51.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:51.159 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:51.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:51.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:51.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:51.291 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:51.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:51.292 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:51.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:51.417 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:51.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:51.519 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:51.541 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:51.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:51.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:51.643 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:51.666 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:51.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:51.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:51.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:51 smithi076 conmon[56514]: cluster 2021-03-15T22:24:51.086939+0000 mon.a (mon.0) 1154 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:51.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:51 smithi076 conmon[56514]: audit 2021-03-15T22:24:51.090698+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? 172.21.15.76:0/2671630730' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:51.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:51 smithi076 conmon[56514]: cluster 2021-03-15T22:24:51.090819+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2021-03-15T22:24:51.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:51 smithi076 conmon[62156]: cluster 2021-03-15T22:24:51.086939+0000 mon.a 2021-03-15T22:24:51.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:51 smithi076 conmon[62156]: (mon.0) 1154 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:51.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:51 smithi076 conmon[62156]: audit 2021-03-15T22:24:51 2021-03-15T22:24:51.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:51 smithi076 conmon[62156]: .090698+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? 172.21.15.76:0/2671630730' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:51.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:51 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:24:51.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:51 smithi076 conmon[62156]: 15T22:24:51.090819+0000 mon.a (mon. 2021-03-15T22:24:51.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:51 smithi076 conmon[62156]: 0) 1156 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2021-03-15T22:24:51.770 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:51.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:51 smithi125 conmon[56739]: cluster 2021-03-15T22:24:51.086939+0000 mon.a (mon.0) 1154 2021-03-15T22:24:51.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:51 smithi125 conmon[56739]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:24:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:51 smithi125 conmon[56739]: audit 2021-03-15T22:24:51.090698+0000 mon.a (mon.0) 1155 : audit [INF] from='client.? 172.21.15.76:0/2671630730' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:24:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:51 smithi125 conmon[56739]: cluster 2021-03-15T22:24:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:51 smithi125 conmon[56739]: 2021-03-15T22:24:51.090819+0000 mon.a (mon.0) 1156 : cluster [DBG] osdmap e376: 8 total, 8 up, 7 in 2021-03-15T22:24:51.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:51.795 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:51.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:51.801 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:24:51.896 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:51.920 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:51.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:51.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:52.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:52.045 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:52.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:52.046 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:52.094 INFO:tasks.rados.rados.0.smithi076.stdout:1356: done (1 left) 2021-03-15T22:24:52.094 INFO:tasks.rados.rados.0.smithi076.stdout:1357: read oid 42 snap -1 2021-03-15T22:24:52.094 INFO:tasks.rados.rados.0.smithi076.stdout:1357: expect (ObjNum 442 snap 124 seq_num 442) 2021-03-15T22:24:52.095 INFO:tasks.rados.rados.0.smithi076.stdout:1358: write (excl) oid 6 current snap is 143 2021-03-15T22:24:52.095 INFO:tasks.rados.rados.0.smithi076.stdout:1358: seq_num 485 ranges {551395=626004,1848002=303393} 2021-03-15T22:24:52.102 INFO:tasks.rados.rados.0.smithi076.stdout:1358: writing smithi07694878-6 from 551395 to 1177399 tid 1 2021-03-15T22:24:52.109 INFO:tasks.rados.rados.0.smithi076.stdout:1358: writing smithi07694878-6 from 1848002 to 2151395 tid 2 2021-03-15T22:24:52.109 INFO:tasks.rados.rados.0.smithi076.stdout:1359: snap_create 2021-03-15T22:24:52.110 INFO:tasks.rados.rados.0.smithi076.stdout:1355: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:24:52.147 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:52.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:52.171 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:52.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:52.233 INFO:tasks.rados.rados.0.smithi076.stdout:1358: finishing write tid 1 to smithi07694878-6 2021-03-15T22:24:52.233 INFO:tasks.rados.rados.0.smithi076.stdout:1358: finishing write tid 2 to smithi07694878-6 2021-03-15T22:24:52.234 INFO:tasks.rados.rados.0.smithi076.stdout:1358: finishing write tid 3 to smithi07694878-6 2021-03-15T22:24:52.234 INFO:tasks.rados.rados.0.smithi076.stdout:1358: finishing write tid 4 to smithi07694878-6 2021-03-15T22:24:52.234 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 277 (ObjNum 485 snap 143 seq_num 485) dirty exists 2021-03-15T22:24:52.234 INFO:tasks.rados.rados.0.smithi076.stdout:1358: left oid 6 (ObjNum 485 snap 143 seq_num 485) 2021-03-15T22:24:52.235 INFO:tasks.rados.rados.0.smithi076.stdout:1357: expect (ObjNum 442 snap 124 seq_num 442) 2021-03-15T22:24:52.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:52.294 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:52.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:52.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:52.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:52.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:52.420 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:52.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:52.465 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:52.520 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:52.522 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:52.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:52.550 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:52.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:52.651 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:52.674 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:52.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:52.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:52.697 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:52 smithi076 conmon[56514]: cluster 2021-03-15T22:24:51.044228+0000 2021-03-15T22:24:52.697 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:52 smithi076 conmon[56514]: mgr.y (mgr.14144) 594 : cluster [DBG] pgmap v690: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 249 MiB data, 761 MiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 15 op/s; 1/1314 objects misplaced (0.076%) 2021-03-15T22:24:52.697 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:52 smithi076 conmon[56514]: cluster 2021-03-15T22:24:52.091974+0000 mon.a (mon.0) 1157 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2021-03-15T22:24:52.698 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:52 smithi076 conmon[62156]: cluster 2021-03-15T22:24:51 2021-03-15T22:24:52.698 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:52 smithi076 conmon[62156]: .044228+0000 mgr.y (mgr.14144) 594 : cluster [DBG] pgmap v690: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 249 MiB data, 761 MiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 15 op/s; 1/1314 objects misplaced (0.076%) 2021-03-15T22:24:52.698 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:52 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:24:52.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:52 smithi076 conmon[62156]: -15T22:24:52. 2021-03-15T22:24:52.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:52 smithi076 conmon[62156]: 091974+0000 mon.a (mon.0) 2021-03-15T22:24:52.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:52 smithi076 conmon[62156]: 1157 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2021-03-15T22:24:52.777 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:52.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:52 smithi125 conmon[56739]: cluster 2021-03-15T22:24:51.044228+0000 mgr.y 2021-03-15T22:24:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:52 smithi125 conmon[56739]: (mgr.14144) 594 : cluster [DBG] pgmap v690: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 249 MiB data, 761 MiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 2.6 MiB/s wr, 15 op/s; 1/1314 objects misplaced (0.076%) 2021-03-15T22:24:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:52 smithi125 conmon[56739]: cluster 2021-03-15T22:24:52.091974+0000 2021-03-15T22:24:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:52 smithi125 conmon[56739]: mon.a (mon.0) 1157 : cluster [DBG] osdmap e377: 8 total, 8 up, 7 in 2021-03-15T22:24:52.798 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:52.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:52.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:52.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:52.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:52.924 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:52.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:53.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:53.047 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:53.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:53.049 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:53.099 INFO:tasks.rados.rados.0.smithi076.stdout:1355: done (3 left) 2021-03-15T22:24:53.099 INFO:tasks.rados.rados.0.smithi076.stdout:1357: done (2 left) 2021-03-15T22:24:53.099 INFO:tasks.rados.rados.0.smithi076.stdout:1358: done (1 left) 2021-03-15T22:24:53.100 INFO:tasks.rados.rados.0.smithi076.stdout:1359: done (0 left) 2021-03-15T22:24:53.100 INFO:tasks.rados.rados.0.smithi076.stdout:1360: read oid 9 snap -1 2021-03-15T22:24:53.100 INFO:tasks.rados.rados.0.smithi076.stdout:1360: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:24:53.100 INFO:tasks.rados.rados.0.smithi076.stdout:1361: write oid 6 current snap is 144 2021-03-15T22:24:53.101 INFO:tasks.rados.rados.0.smithi076.stdout:1361: seq_num 486 ranges {568202=701253,1906484=261718} 2021-03-15T22:24:53.106 INFO:tasks.rados.rados.0.smithi076.stdout:1361: writing smithi07694878-6 from 568202 to 1269455 tid 1 2021-03-15T22:24:53.112 INFO:tasks.rados.rados.0.smithi076.stdout:1361: writing smithi07694878-6 from 1906484 to 2168202 tid 2 2021-03-15T22:24:53.113 INFO:tasks.rados.rados.0.smithi076.stdout:1362: rollback oid 40 current snap is 144 2021-03-15T22:24:53.113 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 40 to 112 2021-03-15T22:24:53.113 INFO:tasks.rados.rados.0.smithi076.stdout:1363: write (excl) oid 30 current snap is 144 2021-03-15T22:24:53.113 INFO:tasks.rados.rados.0.smithi076.stdout:1363: seq_num 487 ranges {585009=492855,1564966=620043} 2021-03-15T22:24:53.117 INFO:tasks.rados.rados.0.smithi076.stdout:1363: writing smithi07694878-30 from 585009 to 1077864 tid 1 2021-03-15T22:24:53.124 INFO:tasks.rados.rados.0.smithi076.stdout:1363: writing smithi07694878-30 from 1564966 to 2185009 tid 2 2021-03-15T22:24:53.124 INFO:tasks.rados.rados.0.smithi076.stdout:1364: snap_create 2021-03-15T22:24:53.124 INFO:tasks.rados.rados.0.smithi076.stdout:1361: finishing write tid 1 to smithi07694878-6 2021-03-15T22:24:53.124 INFO:tasks.rados.rados.0.smithi076.stdout:1360: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:24:53.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:53.176 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:53.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:53.177 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:53.279 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:53.290 INFO:tasks.rados.rados.0.smithi076.stdout:1361: finishing write tid 2 to smithi07694878-6 2021-03-15T22:24:53.291 INFO:tasks.rados.rados.0.smithi076.stdout:1361: finishing write tid 3 to smithi07694878-6 2021-03-15T22:24:53.291 INFO:tasks.rados.rados.0.smithi076.stdout:1361: finishing write tid 4 to smithi07694878-6 2021-03-15T22:24:53.291 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 280 (ObjNum 486 snap 144 seq_num 486) dirty exists 2021-03-15T22:24:53.292 INFO:tasks.rados.rados.0.smithi076.stdout:1361: left oid 6 (ObjNum 486 snap 144 seq_num 486) 2021-03-15T22:24:53.292 INFO:tasks.rados.rados.0.smithi076.stdout:1362: finishing rollback tid 0 to smithi07694878-40 2021-03-15T22:24:53.293 INFO:tasks.rados.rados.0.smithi076.stdout:1362: finishing rollback tid 1 to smithi07694878-40 2021-03-15T22:24:53.293 INFO:tasks.rados.rados.0.smithi076.stdout:1362: finishing rollback tid 2 to smithi07694878-40 2021-03-15T22:24:53.293 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 836 (ObjNum 413 snap 112 seq_num 413) dirty exists 2021-03-15T22:24:53.293 INFO:tasks.rados.rados.0.smithi076.stdout:1363: finishing write tid 1 to smithi07694878-30 2021-03-15T22:24:53.293 INFO:tasks.rados.rados.0.smithi076.stdout:1363: finishing write tid 2 to smithi07694878-30 2021-03-15T22:24:53.294 INFO:tasks.rados.rados.0.smithi076.stdout:1363: finishing write tid 3 to smithi07694878-30 2021-03-15T22:24:53.294 INFO:tasks.rados.rados.0.smithi076.stdout:1363: finishing write tid 4 to smithi07694878-30 2021-03-15T22:24:53.294 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 802 (ObjNum 487 snap 144 seq_num 487) dirty exists 2021-03-15T22:24:53.294 INFO:tasks.rados.rados.0.smithi076.stdout:1363: left oid 30 (ObjNum 487 snap 144 seq_num 487) 2021-03-15T22:24:53.301 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:53.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:53.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:53.404 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:53.426 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:53.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:53.427 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:53.528 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:53.551 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:53.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:53.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:53.606 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:53 smithi076 conmon[56514]: cluster 2021-03-15T22:24:52.256453+0000 osd.7 (osd.7) 72 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:53.606 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:53 smithi076 conmon[56514]: cluster 2021-03-15T22:24:52.256873+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:53.606 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:53 smithi076 conmon[56514]: cluster 2021-03-15T22:24:53.096165+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2021-03-15T22:24:53.607 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:53 smithi076 conmon[62156]: cluster 2021-03-15T22:24:52.256453+0000 osd.7 (osd.7) 72 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:53.607 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:53 smithi076 conmon[62156]: cluster 2021-03-15T22:24:52.256873+0000 osd.7 (osd.7) 73 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:53.607 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:53 smithi076 conmon[62156]: cluster 2021-03-15T22:24:53.096165+0000 mon.a (mon.0) 1158 : cluster [DBG] osdmap e378: 8 total, 8 up, 7 in 2021-03-15T22:24:53.653 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:53.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:53.677 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:53.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:53.779 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:53.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:53 smithi125 conmon[56739]: cluster 2021-03-15T22:24:52.256453+0000 osd.7 (osd 2021-03-15T22:24:53.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:53 smithi125 conmon[56739]: .7) 72 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:53.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:53 smithi125 conmon[56739]: cluster 2021-03-15T22:24:52.256873+0000 osd.7 (osd.7) 73 : cluster [DBG] 2021-03-15T22:24:53.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:53 smithi125 conmon[56739]: purged_snaps scrub ok 2021-03-15T22:24:53.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:53 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:24:53.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:53 smithi125 conmon[56739]: 24:53.096165+0000 mon.a (mon.0) 1158 : cluster [DBG] 2021-03-15T22:24:53.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:53 smithi125 conmon[56739]: osdmap e378: 8 total, 8 up, 7 in 2021-03-15T22:24:53.802 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:53.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:53.906 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:53.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:53.929 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:53.929 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:54.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:54.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:54.054 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:54.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:54.101 INFO:tasks.rados.rados.0.smithi076.stdout:1360: done (4 left) 2021-03-15T22:24:54.101 INFO:tasks.rados.rados.0.smithi076.stdout:1361: done (3 left) 2021-03-15T22:24:54.102 INFO:tasks.rados.rados.0.smithi076.stdout:1362: done (2 left) 2021-03-15T22:24:54.102 INFO:tasks.rados.rados.0.smithi076.stdout:1363: done (1 left) 2021-03-15T22:24:54.102 INFO:tasks.rados.rados.0.smithi076.stdout:1364: done (0 left) 2021-03-15T22:24:54.103 INFO:tasks.rados.rados.0.smithi076.stdout:1365: delete oid 34 current snap is 145 2021-03-15T22:24:54.103 INFO:tasks.rados.rados.0.smithi076.stdout:1365: done (0 left) 2021-03-15T22:24:54.104 INFO:tasks.rados.rados.0.smithi076.stdout:1366: rollback oid 43 current snap is 145 2021-03-15T22:24:54.104 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 43 to 127 2021-03-15T22:24:54.104 INFO:tasks.rados.rados.0.smithi076.stdout:1367: delete oid 1 current snap is 145 2021-03-15T22:24:54.106 INFO:tasks.rados.rados.0.smithi076.stdout:1366: finishing rollback tid 1 to smithi07694878-43 2021-03-15T22:24:54.106 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.1 to 1 (10000) 2021-03-15T22:24:54.107 INFO:tasks.rados.rados.0.smithi076.stdout:1366: finishing rollback tid 2 to smithi07694878-43 2021-03-15T22:24:54.107 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 289 (ObjNum 392 snap 106 seq_num 392) dirty exists 2021-03-15T22:24:54.107 INFO:tasks.rados.rados.0.smithi076.stdout:1366: done (1 left) 2021-03-15T22:24:54.108 INFO:tasks.rados.rados.0.smithi076.stdout:1367: done (0 left) 2021-03-15T22:24:54.108 INFO:tasks.rados.rados.0.smithi076.stdout:1368: write (excl) oid 30 current snap is 145 2021-03-15T22:24:54.108 INFO:tasks.rados.rados.0.smithi076.stdout:1368: seq_num 490 ranges {635430=718602,2140412=95018} 2021-03-15T22:24:54.114 INFO:tasks.rados.rados.0.smithi076.stdout:1368: writing smithi07694878-30 from 635430 to 1354032 tid 1 2021-03-15T22:24:54.119 INFO:tasks.rados.rados.0.smithi076.stdout:1368: writing smithi07694878-30 from 2140412 to 2235430 tid 2 2021-03-15T22:24:54.119 INFO:tasks.rados.rados.0.smithi076.stdout:1369: copy_from oid 3 from oid 13 current snap is 145 2021-03-15T22:24:54.120 INFO:tasks.rados.rados.0.smithi076.stdout:1368: finishing write tid 1 to smithi07694878-30 2021-03-15T22:24:54.120 INFO:tasks.rados.rados.0.smithi076.stdout:1370: write oid 49 current snap is 145 2021-03-15T22:24:54.121 INFO:tasks.rados.rados.0.smithi076.stdout:1370: seq_num 491 ranges {652237=793851,2198894=53343} 2021-03-15T22:24:54.127 INFO:tasks.rados.rados.0.smithi076.stdout:1370: writing smithi07694878-49 from 652237 to 1446088 tid 1 2021-03-15T22:24:54.132 INFO:tasks.rados.rados.0.smithi076.stdout:1370: writing smithi07694878-49 from 2198894 to 2252237 tid 2 2021-03-15T22:24:54.132 INFO:tasks.rados.rados.0.smithi076.stdout:1369: finishing copy_from to smithi07694878-3 2021-03-15T22:24:54.133 INFO:tasks.rados.rados.0.smithi076.stdout:1369: got expected ENOENT (src dne) 2021-03-15T22:24:54.133 INFO:tasks.rados.rados.0.smithi076.stdout:1369: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:24:54.133 INFO:tasks.rados.rados.0.smithi076.stdout:1368: finishing write tid 2 to smithi07694878-30 2021-03-15T22:24:54.133 INFO:tasks.rados.rados.0.smithi076.stdout:1368: finishing write tid 3 to smithi07694878-30 2021-03-15T22:24:54.134 INFO:tasks.rados.rados.0.smithi076.stdout:1368: finishing write tid 4 to smithi07694878-30 2021-03-15T22:24:54.134 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 806 (ObjNum 490 snap 145 seq_num 490) dirty exists 2021-03-15T22:24:54.134 INFO:tasks.rados.rados.0.smithi076.stdout:1368: left oid 30 (ObjNum 490 snap 145 seq_num 490) 2021-03-15T22:24:54.134 INFO:tasks.rados.rados.0.smithi076.stdout:1370: finishing write tid 1 to smithi07694878-49 2021-03-15T22:24:54.135 INFO:tasks.rados.rados.0.smithi076.stdout:1368: done (2 left) 2021-03-15T22:24:54.135 INFO:tasks.rados.rados.0.smithi076.stdout:1369: done (1 left) 2021-03-15T22:24:54.135 INFO:tasks.rados.rados.0.smithi076.stdout:1371: write oid 35 current snap is 145 2021-03-15T22:24:54.136 INFO:tasks.rados.rados.0.smithi076.stdout:1371: seq_num 492 ranges {669044=469100,1741023=459161} 2021-03-15T22:24:54.138 INFO:tasks.rados.rados.0.smithi076.stdout:1371: writing smithi07694878-35 from 669044 to 1138144 tid 1 2021-03-15T22:24:54.145 INFO:tasks.rados.rados.0.smithi076.stdout:1371: writing smithi07694878-35 from 1741023 to 2200184 tid 2 2021-03-15T22:24:54.145 INFO:tasks.rados.rados.0.smithi076.stdout:1372: snap_create 2021-03-15T22:24:54.145 INFO:tasks.rados.rados.0.smithi076.stdout:1370: finishing write tid 2 to smithi07694878-49 2021-03-15T22:24:54.146 INFO:tasks.rados.rados.0.smithi076.stdout:1370: finishing write tid 3 to smithi07694878-49 2021-03-15T22:24:54.146 INFO:tasks.rados.rados.0.smithi076.stdout:1370: finishing write tid 4 to smithi07694878-49 2021-03-15T22:24:54.146 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 810 (ObjNum 491 snap 145 seq_num 491) dirty exists 2021-03-15T22:24:54.146 INFO:tasks.rados.rados.0.smithi076.stdout:1370: left oid 49 (ObjNum 491 snap 145 seq_num 491) 2021-03-15T22:24:54.147 INFO:tasks.rados.rados.0.smithi076.stdout:1371: finishing write tid 1 to smithi07694878-35 2021-03-15T22:24:54.148 INFO:tasks.rados.rados.0.smithi076.stdout:1371: finishing write tid 2 to smithi07694878-35 2021-03-15T22:24:54.148 INFO:tasks.rados.rados.0.smithi076.stdout:1371: finishing write tid 3 to smithi07694878-35 2021-03-15T22:24:54.148 INFO:tasks.rados.rados.0.smithi076.stdout:1371: finishing write tid 4 to smithi07694878-35 2021-03-15T22:24:54.148 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 484 (ObjNum 492 snap 145 seq_num 492) dirty exists 2021-03-15T22:24:54.149 INFO:tasks.rados.rados.0.smithi076.stdout:1371: left oid 35 (ObjNum 492 snap 145 seq_num 492) 2021-03-15T22:24:54.156 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:54.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:54.180 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:54.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:54.282 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:54.305 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:54.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:54.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:54.407 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:54.430 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:54.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:54.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:54.487 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:24:54.512 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[56514]: cluster 2021-03-15T22:24:53.003976+0000 2021-03-15T22:24:54.512 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[56514]: osd.0 (osd.0) 75 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:54.512 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[56514]: cluster 2021-03-15T22:24:53.004385+0000 osd.0 (osd.0) 76 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:54.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[56514]: cluster 2021-03-15T22:24:53.044661+0000 mgr.y (mgr.14144) 595 : cluster [DBG] pgmap v693: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 252 MiB data, 779 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 3.6 MiB/s wr, 23 op/s; 1/1320 objects misplaced (0.076%) 2021-03-15T22:24:54.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[56514]: audit 2021-03-15T22:24:53.897949+0000 mon.a (mon.0) 1159 : audit [INF] from='client.? 172.21.15.76:0/598046119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:24:54.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[56514]: audit 2021-03-15T22:24:54.094548+0000 mon.a (mon.0) 1160 : audit [INF] from='client.? 172.21.15.76:0/598046119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-15T22:24:54.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[56514]: cluster 2021-03-15T22:24:54.094607+0000 mon.a (mon.0) 1161 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2021-03-15T22:24:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: cluster 2021-03-15T22:24:53.003976+0000 osd.0 (osd 2021-03-15T22:24:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: .0) 75 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: cluster 2021-03-15T22:24:53.004385+0000 osd.0 (osd 2021-03-15T22:24:54.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: .0) 76 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:54.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: cluster 2021-03-15T22:24:54.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: 2021-03-15T22:24:53.044661+0000 2021-03-15T22:24:54.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: mgr.y (mgr.14144) 595 : cluster [DBG] pgmap v693: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 252 MiB data, 779 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 3.6 MiB/s wr, 23 op/s; 1/1320 objects misplaced (0.076%) 2021-03-15T22:24:54.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: 2021-03-15T22:24:54.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: audit 2021-03-15T22:24:54.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: 2021-03-15T22:24:53.897949+0000 mon.a 2021-03-15T22:24:54.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: (mon.0) 1159 : audit [INF] from='client.? 172.21.15.76:0/598046119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:24:54.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: audit 2021-03-15T22:24:54.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: 2021-03-15T22:24:54. 2021-03-15T22:24:54.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: 094548+0000 mon.a (mon.0) 1160 : audit 2021-03-15T22:24:54.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/598046119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-15T22:24:54.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:24:54.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: :24:54.094607+0000 mon.a (mon. 2021-03-15T22:24:54.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:54 smithi076 conmon[62156]: 0) 1161 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2021-03-15T22:24:54.533 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:54.554 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:54.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:54.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:54.657 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:54.679 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:54.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:54.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:54.783 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:54.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: cluster 2021-03-15T22:24:53.003976+0000 osd.0 (osd.0 2021-03-15T22:24:54.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: ) 75 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:24:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: cluster 2021-03-15T22:24:53.004385+0000 osd.0 (osd 2021-03-15T22:24:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: .0) 76 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:24:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:24:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: T22:24:53.044661+0000 mgr.y 2021-03-15T22:24:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: (mgr.14144) 595 : cluster [DBG] pgmap v693: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 252 MiB data, 779 MiB used, 625 GiB / 626 GiB avail; 17 MiB/s rd, 3.6 MiB/s wr, 23 op/s; 1/1320 objects misplaced (0.076%) 2021-03-15T22:24:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: 2021-03-15T22:24:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: audit 2021-03-15T22:24:54.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: 2021-03-15T22:24:53. 2021-03-15T22:24:54.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: 897949+0000 mon.a (mon.0) 1159 : audit 2021-03-15T22:24:54.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/598046119' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:24:54.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:24:54.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: -15T22:24:54.094548 2021-03-15T22:24:54.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1160 : audit [INF] 2021-03-15T22:24:54.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/598046119' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-15T22:24:54.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:24:54.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: 15T22:24:54.094607+0000 2021-03-15T22:24:54.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: mon.a (mon.0) 1161 : cluster [DBG] osdmap e379: 8 total, 8 up, 7 in 2021-03-15T22:24:54.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:54 smithi125 conmon[56739]: 2021-03-15T22:24:54.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:54.811 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:54.811 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:54.913 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:54.939 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:54.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:54.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:54.957 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:24:54.973 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:24:54.990 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:24:55.008 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:24:55.027 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:24:55.042 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:55.046 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:24:55.063 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:24:55.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:55.067 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:55.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:55.081 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:24:55.083 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:24:55.083 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:24:55.084 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:24:55.084 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:24:55.084 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:24:55.084 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:24:55.085 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:24:55.085 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:24:55.097 INFO:tasks.rados.rados.0.smithi076.stdout:1370: done (2 left) 2021-03-15T22:24:55.097 INFO:tasks.rados.rados.0.smithi076.stdout:1371: done (1 left) 2021-03-15T22:24:55.097 INFO:tasks.rados.rados.0.smithi076.stdout:1372: done (0 left) 2021-03-15T22:24:55.097 INFO:tasks.rados.rados.0.smithi076.stdout:1373: rollback oid 27 current snap is 146 2021-03-15T22:24:55.098 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 119 2021-03-15T22:24:55.098 INFO:tasks.rados.rados.0.smithi076.stdout:1374: read oid 44 snap -1 2021-03-15T22:24:55.098 INFO:tasks.rados.rados.0.smithi076.stdout:1374: expect (ObjNum 468 snap 137 seq_num 468) 2021-03-15T22:24:55.098 INFO:tasks.rados.rados.0.smithi076.stdout:1375: write (excl) oid 49 current snap is 146 2021-03-15T22:24:55.099 INFO:tasks.rados.rados.0.smithi076.stdout:1375: seq_num 493 ranges {685851=544349,1799505=486346} 2021-03-15T22:24:55.099 INFO:tasks.rados.rados.0.smithi076.stdout:1375: writing smithi07694878-49 from 685851 to 1230200 tid 1 2021-03-15T22:24:55.103 INFO:tasks.rados.rados.0.smithi076.stdout:1375: writing smithi07694878-49 from 1799505 to 2285851 tid 2 2021-03-15T22:24:55.104 INFO:tasks.rados.rados.0.smithi076.stdout:1376: read oid 10 snap 132 2021-03-15T22:24:55.104 INFO:tasks.rados.rados.0.smithi076.stdout:1376: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:24:55.104 INFO:tasks.rados.rados.0.smithi076.stdout:1377: write oid 11 current snap is 146 2021-03-15T22:24:55.104 INFO:tasks.rados.rados.0.smithi076.stdout:1377: seq_num 494 ranges {702658=619598,1857987=444671} 2021-03-15T22:24:55.111 INFO:tasks.rados.rados.0.smithi076.stdout:1377: writing smithi07694878-11 from 702658 to 1322256 tid 1 2021-03-15T22:24:55.117 INFO:tasks.rados.rados.0.smithi076.stdout:1377: writing smithi07694878-11 from 1857987 to 2302658 tid 2 2021-03-15T22:24:55.119 INFO:tasks.rados.rados.0.smithi076.stdout:1378: copy_from oid 26 from oid 4 current snap is 146 2021-03-15T22:24:55.119 INFO:tasks.rados.rados.0.smithi076.stdout:1373: finishing rollback tid 0 to smithi07694878-27 2021-03-15T22:24:55.119 INFO:tasks.rados.rados.0.smithi076.stdout:1373: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:24:55.119 INFO:tasks.rados.rados.0.smithi076.stdout:1373: finishing rollback tid 2 to smithi07694878-27 2021-03-15T22:24:55.120 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 294 (ObjNum 416 snap 112 seq_num 416) dirty exists 2021-03-15T22:24:55.120 INFO:tasks.rados.rados.0.smithi076.stdout:1374: expect (ObjNum 468 snap 137 seq_num 468) 2021-03-15T22:24:55.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:55.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:55.192 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:55.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:55.251 INFO:tasks.rados.rados.0.smithi076.stdout:1375: finishing write tid 1 to smithi07694878-49 2021-03-15T22:24:55.252 INFO:tasks.rados.rados.0.smithi076.stdout:1376: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:24:55.294 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:55.315 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:55.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:55.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:55.418 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:55.440 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:55.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:55.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:55.465 INFO:tasks.rados.rados.0.smithi076.stdout:1377: finishing write tid 1 to smithi07694878-11 2021-03-15T22:24:55.466 INFO:tasks.rados.rados.0.smithi076.stdout:1375: finishing write tid 2 to smithi07694878-49 2021-03-15T22:24:55.466 INFO:tasks.rados.rados.0.smithi076.stdout:1373: done (5 left) 2021-03-15T22:24:55.466 INFO:tasks.rados.rados.0.smithi076.stdout:1374: done (4 left) 2021-03-15T22:24:55.467 INFO:tasks.rados.rados.0.smithi076.stdout:1376: done (3 left) 2021-03-15T22:24:55.467 INFO:tasks.rados.rados.0.smithi076.stdout:1379: copy_from oid 43 from oid 48 current snap is 146 2021-03-15T22:24:55.467 INFO:tasks.rados.rados.0.smithi076.stdout:1375: finishing write tid 3 to smithi07694878-49 2021-03-15T22:24:55.468 INFO:tasks.rados.rados.0.smithi076.stdout:1375: finishing write tid 4 to smithi07694878-49 2021-03-15T22:24:55.468 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 814 (ObjNum 493 snap 146 seq_num 493) dirty exists 2021-03-15T22:24:55.468 INFO:tasks.rados.rados.0.smithi076.stdout:1375: left oid 49 (ObjNum 493 snap 146 seq_num 493) 2021-03-15T22:24:55.468 INFO:tasks.rados.rados.0.smithi076.stdout:1378: finishing copy_from to smithi07694878-26 2021-03-15T22:24:55.469 INFO:tasks.rados.rados.0.smithi076.stdout:1378: got expected ENOENT (src dne) 2021-03-15T22:24:55.469 INFO:tasks.rados.rados.0.smithi076.stdout:1378: finishing copy_from racing read to smithi07694878-26 2021-03-15T22:24:55.469 INFO:tasks.rados.rados.0.smithi076.stdout:1377: finishing write tid 2 to smithi07694878-11 2021-03-15T22:24:55.469 INFO:tasks.rados.rados.0.smithi076.stdout:1377: finishing write tid 3 to smithi07694878-11 2021-03-15T22:24:55.470 INFO:tasks.rados.rados.0.smithi076.stdout:1377: finishing write tid 4 to smithi07694878-11 2021-03-15T22:24:55.470 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 421 (ObjNum 494 snap 146 seq_num 494) dirty exists 2021-03-15T22:24:55.470 INFO:tasks.rados.rados.0.smithi076.stdout:1377: left oid 11 (ObjNum 494 snap 146 seq_num 494) 2021-03-15T22:24:55.470 INFO:tasks.rados.rados.0.smithi076.stdout:1375: done (3 left) 2021-03-15T22:24:55.471 INFO:tasks.rados.rados.0.smithi076.stdout:1377: done (2 left) 2021-03-15T22:24:55.471 INFO:tasks.rados.rados.0.smithi076.stdout:1378: done (1 left) 2021-03-15T22:24:55.471 INFO:tasks.rados.rados.0.smithi076.stdout:1380: read oid 40 snap 144 2021-03-15T22:24:55.471 INFO:tasks.rados.rados.0.smithi076.stdout:1380: expect (ObjNum 413 snap 112 seq_num 413) 2021-03-15T22:24:55.472 INFO:tasks.rados.rados.0.smithi076.stdout:1381: read oid 19 snap -1 2021-03-15T22:24:55.472 INFO:tasks.rados.rados.0.smithi076.stdout:1381: expect deleted 2021-03-15T22:24:55.472 INFO:tasks.rados.rados.0.smithi076.stdout:1382: rollback oid 26 current snap is 146 2021-03-15T22:24:55.472 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 140 2021-03-15T22:24:55.473 INFO:tasks.rados.rados.0.smithi076.stdout:1383: copy_from oid 35 from oid 18 current snap is 146 2021-03-15T22:24:55.473 INFO:tasks.rados.rados.0.smithi076.stdout:1384: read oid 14 snap -1 2021-03-15T22:24:55.473 INFO:tasks.rados.rados.0.smithi076.stdout:1384: expect (ObjNum 453 snap 132 seq_num 453) 2021-03-15T22:24:55.473 INFO:tasks.rados.rados.0.smithi076.stdout:1385: snap_create 2021-03-15T22:24:55.474 INFO:tasks.rados.rados.0.smithi076.stdout:1382: finishing rollback tid 0 to smithi07694878-26 2021-03-15T22:24:55.474 INFO:tasks.rados.rados.0.smithi076.stdout:1382: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:24:55.475 INFO:tasks.rados.rados.0.smithi076.stdout:1384: expect (ObjNum 453 snap 132 seq_num 453) 2021-03-15T22:24:55.543 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:55.565 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:55.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:55.567 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:55.644 INFO:tasks.rados.rados.0.smithi076.stdout:1382: finishing rollback tid 2 to smithi07694878-26 2021-03-15T22:24:55.644 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 656 (ObjNum 322 snap 87 seq_num 322) dirty exists 2021-03-15T22:24:55.644 INFO:tasks.rados.rados.0.smithi076.stdout:1383: finishing copy_from to smithi07694878-35 2021-03-15T22:24:55.645 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 164 (ObjNum 324 snap 87 seq_num 324) dirty exists 2021-03-15T22:24:55.645 INFO:tasks.rados.rados.0.smithi076.stdout:1380: expect (ObjNum 413 snap 112 seq_num 413) 2021-03-15T22:24:55.668 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:55.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:55.691 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:55.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:55.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:55 smithi076 conmon[56514]: audit 2021-03-15T22:24:54.935288+0000 mon.a (mon.0) 1162 : audit [DBG] from='client.? 172.21.15.76:0/3365476339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:55.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:55 smithi076 conmon[56514]: cluster 2021-03-15T22:24:55.083956+0000 mon.a (mon.0) 1163 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2021-03-15T22:24:55.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:55 smithi076 conmon[62156]: audit 2021-03-15T22:24:55.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:55 smithi076 conmon[62156]: 2021-03-15T22:24: 2021-03-15T22:24:55.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:55 smithi076 conmon[62156]: 54.935288+0000 mon.a (mon.0) 1162 : audit [DBG] from='client.? 172.21.15.76:0/3365476339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:55.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:55 smithi076 conmon[62156]: cluster 2021-03-15T22:24 2021-03-15T22:24:55.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:55 smithi076 conmon[62156]: :55.083956+0000 mon.a (mon.0) 1163 : cluster 2021-03-15T22:24:55.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:55 smithi076 conmon[62156]: [DBG] osdmap e380: 8 total, 8 up, 7 in 2021-03-15T22:24:55.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:55 smithi125 conmon[56739]: audit 2021-03-15T22:24:54.935288+0000 mon.a (mon.0) 2021-03-15T22:24:55.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:55 smithi125 conmon[56739]: 1162 : audit [DBG] from='client.? 172.21.15.76:0/3365476339' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:24:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:55 smithi125 conmon[56739]: cluster 2021-03-15T22:24:55.083956+0000 mon.a (mon.0) 1163 : cluster [DBG] osdmap e380: 8 total, 8 up, 7 in 2021-03-15T22:24:55.792 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:55.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:55.815 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:55.815 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:55.860 INFO:tasks.rados.rados.0.smithi076.stdout:1383: finishing copy_from racing read to smithi07694878-35 2021-03-15T22:24:55.860 INFO:tasks.rados.rados.0.smithi076.stdout:1379: finishing copy_from to smithi07694878-43 2021-03-15T22:24:55.860 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 752 (ObjNum 459 snap 133 seq_num 459) dirty exists 2021-03-15T22:24:55.861 INFO:tasks.rados.rados.0.smithi076.stdout:1379: finishing copy_from racing read to smithi07694878-43 2021-03-15T22:24:55.916 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:55.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:55.939 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:55.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:55.990 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 2 1 2021-03-15T22:24:56.041 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:56.064 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:56.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:56.066 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:56.169 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:56.192 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:56.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:56.194 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:56.278 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:56.295 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:56.312 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:56.324 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:56.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:56.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:56.328 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:24:56.328 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:24:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:24:56.329 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:24:56.329 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:24:56.357 INFO:tasks.rados.rados.0.smithi076.stdout:1379: done (6 left) 2021-03-15T22:24:56.358 INFO:tasks.rados.rados.0.smithi076.stdout:1380: done (5 left) 2021-03-15T22:24:56.358 INFO:tasks.rados.rados.0.smithi076.stdout:1381: done (4 left) 2021-03-15T22:24:56.358 INFO:tasks.rados.rados.0.smithi076.stdout:1382: done (3 left) 2021-03-15T22:24:56.359 INFO:tasks.rados.rados.0.smithi076.stdout:1383: done (2 left) 2021-03-15T22:24:56.359 INFO:tasks.rados.rados.0.smithi076.stdout:1384: done (1 left) 2021-03-15T22:24:56.359 INFO:tasks.rados.rados.0.smithi076.stdout:1385: done (0 left) 2021-03-15T22:24:56.359 INFO:tasks.rados.rados.0.smithi076.stdout:1386: read oid 40 snap -1 2021-03-15T22:24:56.360 INFO:tasks.rados.rados.0.smithi076.stdout:1386: expect (ObjNum 413 snap 112 seq_num 413) 2021-03-15T22:24:56.360 INFO:tasks.rados.rados.0.smithi076.stdout:1387: copy_from oid 1 from oid 49 current snap is 147 2021-03-15T22:24:56.360 INFO:tasks.rados.rados.0.smithi076.stdout:1388: read oid 37 snap -1 2021-03-15T22:24:56.360 INFO:tasks.rados.rados.0.smithi076.stdout:1388: expect (ObjNum 476 snap 142 seq_num 476) 2021-03-15T22:24:56.360 INFO:tasks.rados.rados.0.smithi076.stdout:1389: rollback oid 39 current snap is 147 2021-03-15T22:24:56.361 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 39 to 133 2021-03-15T22:24:56.361 INFO:tasks.rados.rados.0.smithi076.stdout:1390: snap_create 2021-03-15T22:24:56.362 INFO:tasks.rados.rados.0.smithi076.stdout:1389: finishing rollback tid 1 to smithi07694878-39 2021-03-15T22:24:56.362 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 0 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:24:56.373 INFO:tasks.rados.rados.0.smithi076.stdout:1386: expect (ObjNum 413 snap 112 seq_num 413) 2021-03-15T22:24:56.427 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:24:56.450 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:56.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:56.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:24:56.553 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:56.577 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:56.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:56.578 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:56.594 INFO:tasks.rados.rados.0.smithi076.stdout:1387: finishing copy_from to smithi07694878-1 2021-03-15T22:24:56.595 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 814 (ObjNum 493 snap 146 seq_num 493) dirty exists 2021-03-15T22:24:56.595 INFO:tasks.rados.rados.0.smithi076.stdout:1388: expect (ObjNum 476 snap 142 seq_num 476) 2021-03-15T22:24:56.656 INFO:tasks.rados.rados.0.smithi076.stdout:1387: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:24:56.679 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:56.702 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:56.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:56.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:56 smithi076 conmon[56514]: cluster 2021-03-15T22:24:55.045170 2021-03-15T22:24:56.713 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:56 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 596 : cluster [DBG] pgmap v696: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 256 MiB data, 788 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 3.2 MiB/s wr, 22 op/s; 1/1332 objects misplaced (0.075%) 2021-03-15T22:24:56.713 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:56 smithi076 conmon[62156]: cluster 2021-03-15T22:24:55.045170+0000 mgr.y (mgr.14144) 596 : cluster [DBG] pgmap v696: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 256 MiB data, 788 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 3.2 MiB/s wr, 22 op/s; 1/1332 objects misplaced (0.075%) 2021-03-15T22:24:56.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:56 smithi125 conmon[56739]: cluster 2021-03-15T22:24:55.045170+0000 mgr.y (mgr.14144) 596 : cluster [DBG] pgmap v696: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 256 MiB data, 788 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 3.2 MiB/s wr, 22 op/s; 1/1332 objects misplaced (0.075%) 2021-03-15T22:24:56.805 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:56.829 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:56.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:56.830 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:56.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:56.954 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:56.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:56.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:57.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:57.080 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:57.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:57.081 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:57.182 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:57.206 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:57.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:57.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:57.309 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:57.334 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:57.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:57.335 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:57.368 INFO:tasks.rados.rados.0.smithi076.stdout:1386: done (4 left) 2021-03-15T22:24:57.369 INFO:tasks.rados.rados.0.smithi076.stdout:1387: done (3 left) 2021-03-15T22:24:57.369 INFO:tasks.rados.rados.0.smithi076.stdout:1388: done (2 left) 2021-03-15T22:24:57.369 INFO:tasks.rados.rados.0.smithi076.stdout:1389: done (1 left) 2021-03-15T22:24:57.369 INFO:tasks.rados.rados.0.smithi076.stdout:1390: done (0 left) 2021-03-15T22:24:57.370 INFO:tasks.rados.rados.0.smithi076.stdout:1391: read oid 24 snap 141 2021-03-15T22:24:57.370 INFO:tasks.rados.rados.0.smithi076.stdout:1391: expect (ObjNum 464 snap 135 seq_num 464) 2021-03-15T22:24:57.370 INFO:tasks.rados.rados.0.smithi076.stdout:1392: snap_create 2021-03-15T22:24:57.375 INFO:tasks.rados.rados.0.smithi076.stdout:1391: expect (ObjNum 464 snap 135 seq_num 464) 2021-03-15T22:24:57.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:24:57.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:57.463 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:57.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:57.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:57.567 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:57.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:57 smithi076 conmon[56514]: cluster 2021-03-15T22:24:56.348780+0000 mon.a (mon.0) 1164 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2021-03-15T22:24:57.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:57 smithi076 conmon[62156]: cluster 2021-03-15T22:24:57.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:57 smithi076 conmon[62156]: 2021-03-15T22:24: 2021-03-15T22:24:57.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:57 smithi076 conmon[62156]: 56.348780+0000 mon.a (mon.0) 1164 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2021-03-15T22:24:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:57.592 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:57.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:57.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:57.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:57.722 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:57.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:57.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:57 smithi125 conmon[56739]: cluster 2021-03-15T22:24:56.348780+0000 mon.a (mon 2021-03-15T22:24:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:57 smithi125 conmon[56739]: .0) 1164 : cluster [DBG] osdmap e381: 8 total, 8 up, 7 in 2021-03-15T22:24:57.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:57.851 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:57.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:57.853 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:57.955 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:57.980 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:57.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:57.981 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:58.082 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:58.108 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:58.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:58.109 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:58.210 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:58.237 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:58.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:58.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:24:58.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:58.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:58.371 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:58.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:58.373 INFO:tasks.rados.rados.0.smithi076.stdout:1391: done (1 left) 2021-03-15T22:24:58.373 INFO:tasks.rados.rados.0.smithi076.stdout:1392: done (0 left) 2021-03-15T22:24:58.373 INFO:tasks.rados.rados.0.smithi076.stdout:1393: write (excl) oid 19 current snap is 149 2021-03-15T22:24:58.373 INFO:tasks.rados.rados.0.smithi076.stdout:1393: seq_num 495 ranges {719465=411200,1632822=686643} 2021-03-15T22:24:58.375 INFO:tasks.rados.rados.0.smithi076.stdout:1393: writing smithi07694878-19 from 719465 to 1130665 tid 1 2021-03-15T22:24:58.382 INFO:tasks.rados.rados.0.smithi076.stdout:1393: writing smithi07694878-19 from 1632822 to 2319465 tid 2 2021-03-15T22:24:58.382 INFO:tasks.rados.rados.0.smithi076.stdout:1394: snap_remove snap 148 2021-03-15T22:24:58.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:58.498 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:58.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:58.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:58 smithi076 conmon[56514]: cluster 2021-03-15T22:24:57.045731+0000 2021-03-15T22:24:58.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:58 smithi076 conmon[56514]: mgr.y (mgr.14144) 597 : cluster [DBG] pgmap v699: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 262 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 18 op/s; 1/1341 objects misplaced (0.075%); 273 B/s, 2 objects/s recovering 2021-03-15T22:24:58.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:58 smithi076 conmon[56514]: cluster 2021-03-15T22:24:57.365819+0000 mon.a (mon.0) 1165 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2021-03-15T22:24:58.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:58 smithi076 conmon[62156]: cluster 2021-03-15T22:24:57.045731 2021-03-15T22:24:58.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:58 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 597 : cluster [DBG] pgmap v699: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 262 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 18 op/s; 1/1341 objects misplaced (0.075%); 273 B/s, 2 objects/s recovering 2021-03-15T22:24:58.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:58 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:24:58.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:58 smithi076 conmon[62156]: 24:57.365819+0000 mon.a (mon.0) 1165 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2021-03-15T22:24:58.600 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:58.623 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:58.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:58.624 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:58.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:24:58.751 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:58.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:58.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:24:58.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:58 smithi125 conmon[56739]: cluster 2021-03-15T22:24:57.045731+0000 mgr.y (mgr.14144) 597 : cluster [DBG] 2021-03-15T22:24:58.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:58 smithi125 conmon[56739]: pgmap v699: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 262 MiB data, 797 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 3.1 MiB/s wr, 18 op/s; 1/1341 objects misplaced (0.075%); 273 B/s, 2 objects/s recovering 2021-03-15T22:24:58.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:58 smithi125 conmon[56739]: cluster 2021-03-15T22:24:57.365819+0000 mon.a (mon.0) 1165 : cluster [DBG] osdmap e382: 8 total, 8 up, 7 in 2021-03-15T22:24:58.854 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:24:58.880 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:58.881 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:58.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:58.882 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:24:58.961 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:24:58.983 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:59.008 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:59.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:59.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:59.110 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:24:59.218 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:59.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:59.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:24:59.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:59.349 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:59.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:59.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:59.390 INFO:tasks.rados.rados.0.smithi076.stdout:1394: done (1 left) 2021-03-15T22:24:59.390 INFO:tasks.rados.rados.0.smithi076.stdout:1395: write oid 1 current snap is 149 2021-03-15T22:24:59.390 INFO:tasks.rados.rados.0.smithi076.stdout:1393: finishing write tid 1 to smithi07694878-19 2021-03-15T22:24:59.391 INFO:tasks.rados.rados.0.smithi076.stdout:1395: seq_num 496 ranges {736272=486449,1974951=361321} 2021-03-15T22:24:59.396 INFO:tasks.rados.rados.0.smithi076.stdout:1395: writing smithi07694878-1 from 736272 to 1222721 tid 1 2021-03-15T22:24:59.403 INFO:tasks.rados.rados.0.smithi076.stdout:1395: writing smithi07694878-1 from 1974951 to 2336272 tid 2 2021-03-15T22:24:59.404 INFO:tasks.rados.rados.0.smithi076.stdout:1393: finishing write tid 2 to smithi07694878-19 2021-03-15T22:24:59.404 INFO:tasks.rados.rados.0.smithi076.stdout:1393: finishing write tid 3 to smithi07694878-19 2021-03-15T22:24:59.404 INFO:tasks.rados.rados.0.smithi076.stdout:1393: finishing write tid 4 to smithi07694878-19 2021-03-15T22:24:59.404 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 489 (ObjNum 495 snap 149 seq_num 495) dirty exists 2021-03-15T22:24:59.405 INFO:tasks.rados.rados.0.smithi076.stdout:1393: left oid 19 (ObjNum 495 snap 149 seq_num 495) 2021-03-15T22:24:59.405 INFO:tasks.rados.rados.0.smithi076.stdout:1393: done (1 left) 2021-03-15T22:24:59.405 INFO:tasks.rados.rados.0.smithi076.stdout:1396: snap_create 2021-03-15T22:24:59.406 INFO:tasks.rados.rados.0.smithi076.stdout:1395: finishing write tid 1 to smithi07694878-1 2021-03-15T22:24:59.407 INFO:tasks.rados.rados.0.smithi076.stdout:1395: finishing write tid 2 to smithi07694878-1 2021-03-15T22:24:59.407 INFO:tasks.rados.rados.0.smithi076.stdout:1395: finishing write tid 3 to smithi07694878-1 2021-03-15T22:24:59.407 INFO:tasks.rados.rados.0.smithi076.stdout:1395: finishing write tid 4 to smithi07694878-1 2021-03-15T22:24:59.407 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 817 (ObjNum 496 snap 149 seq_num 496) dirty exists 2021-03-15T22:24:59.408 INFO:tasks.rados.rados.0.smithi076.stdout:1395: left oid 1 (ObjNum 496 snap 149 seq_num 496) 2021-03-15T22:24:59.451 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:24:59.479 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:59.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:59.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:24:59.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:24:59 smithi076 conmon[56514]: cluster 2021-03-15T22:24:58.366779+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2021-03-15T22:24:59.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:59 smithi076 conmon[62156]: cluster 2021-03-15T22:24:59.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:59 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:24:59.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:24:59 smithi076 conmon[62156]: 22:24:58.366779+0000 mon.a (mon.0) 1166 : cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2021-03-15T22:24:59.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:24:59.612 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:59.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:59.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:24:59.682 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:59 smithi125 conmon[56739]: cluster 2021-03-15T22:24:58.366779+0000 mon.a (mon.0) 1166 : 2021-03-15T22:24:59.682 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:24:59 smithi125 conmon[56739]: cluster [DBG] osdmap e383: 8 total, 8 up, 7 in 2021-03-15T22:24:59.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:24:59.742 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:24:59.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:59.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:24:59.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:24:59.872 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:24:59.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:24:59.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:24:59.975 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:00.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:00.007 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:00.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:00.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:24:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:25:00.090 INFO:tasks.rados.rados.0.smithi076.stdout:1395: done (1 left) 2021-03-15T22:25:00.090 INFO:tasks.rados.rados.0.smithi076.stdout:1396: done (0 left) 2021-03-15T22:25:00.091 INFO:tasks.rados.rados.0.smithi076.stdout:1397: delete oid 26 current snap is 150 2021-03-15T22:25:00.095 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:25:00.099 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.2 to 1 (10000) 2021-03-15T22:25:00.101 INFO:tasks.rados.rados.0.smithi076.stdout:1397: done (0 left) 2021-03-15T22:25:00.101 INFO:tasks.rados.rados.0.smithi076.stdout:1398: delete oid 5 current snap is 150 2021-03-15T22:25:00.103 INFO:tasks.rados.rados.0.smithi076.stdout:1398: done (0 left) 2021-03-15T22:25:00.103 INFO:tasks.rados.rados.0.smithi076.stdout:1399: rollback oid 35 current snap is 150 2021-03-15T22:25:00.104 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 35 to 143 2021-03-15T22:25:00.104 INFO:tasks.rados.rados.0.smithi076.stdout:1400: write (excl) oid 21 current snap is 150 2021-03-15T22:25:00.105 INFO:tasks.rados.rados.0.smithi076.stdout:1400: seq_num 499 ranges {786693=712196,2150397=236296} 2021-03-15T22:25:00.109 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:00.112 INFO:tasks.rados.rados.0.smithi076.stdout:1400: writing smithi07694878-21 from 786693 to 1498889 tid 1 2021-03-15T22:25:00.118 INFO:tasks.rados.rados.0.smithi076.stdout:1400: writing smithi07694878-21 from 2150397 to 2386693 tid 2 2021-03-15T22:25:00.119 INFO:tasks.rados.rados.0.smithi076.stdout:1401: copy_from oid 28 from oid 9 current snap is 150 2021-03-15T22:25:00.119 INFO:tasks.rados.rados.0.smithi076.stdout:1399: finishing rollback tid 0 to smithi07694878-35 2021-03-15T22:25:00.119 INFO:tasks.rados.rados.0.smithi076.stdout:1399: finishing rollback tid 1 to smithi07694878-35 2021-03-15T22:25:00.119 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 492 (ObjNum 288 snap 0 seq_num 145) dirty dne 2021-03-15T22:25:00.120 INFO:tasks.rados.rados.0.smithi076.stdout:1400: finishing write tid 1 to smithi07694878-21 2021-03-15T22:25:00.120 INFO:tasks.rados.rados.0.smithi076.stdout:1399: done (2 left) 2021-03-15T22:25:00.120 INFO:tasks.rados.rados.0.smithi076.stdout:1402: write oid 26 current snap is 150 2021-03-15T22:25:00.120 INFO:tasks.rados.rados.0.smithi076.stdout:1402: seq_num 500 ranges {403500=787445,1692526=473837} 2021-03-15T22:25:00.126 INFO:tasks.rados.rados.0.smithi076.stdout:1402: writing smithi07694878-26 from 403500 to 1190945 tid 1 2021-03-15T22:25:00.132 INFO:tasks.rados.rados.0.smithi076.stdout:1402: writing smithi07694878-26 from 1692526 to 2166363 tid 2 2021-03-15T22:25:00.132 INFO:tasks.rados.rados.0.smithi076.stdout:1403: delete oid 12 current snap is 150 2021-03-15T22:25:00.132 INFO:tasks.rados.rados.0.smithi076.stdout:1400: finishing write tid 2 to smithi07694878-21 2021-03-15T22:25:00.132 INFO:tasks.rados.rados.0.smithi076.stdout:1400: finishing write tid 3 to smithi07694878-21 2021-03-15T22:25:00.133 INFO:tasks.rados.rados.0.smithi076.stdout:1400: finishing write tid 4 to smithi07694878-21 2021-03-15T22:25:00.133 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 840 (ObjNum 499 snap 150 seq_num 499) dirty exists 2021-03-15T22:25:00.133 INFO:tasks.rados.rados.0.smithi076.stdout:1400: left oid 21 (ObjNum 499 snap 150 seq_num 499) 2021-03-15T22:25:00.133 INFO:tasks.rados.rados.0.smithi076.stdout:1402: finishing write tid 1 to smithi07694878-26 2021-03-15T22:25:00.136 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:00.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:00.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:00.141 INFO:tasks.rados.rados.0.smithi076.stdout:1401: finishing copy_from to smithi07694878-28 2021-03-15T22:25:00.141 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 695 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:25:00.141 INFO:tasks.rados.rados.0.smithi076.stdout:1401: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:25:00.172 INFO:tasks.rados.rados.0.smithi076.stdout:1400: done (3 left) 2021-03-15T22:25:00.172 INFO:tasks.rados.rados.0.smithi076.stdout:1401: done (2 left) 2021-03-15T22:25:00.172 INFO:tasks.rados.rados.0.smithi076.stdout:1403: done (1 left) 2021-03-15T22:25:00.173 INFO:tasks.rados.rados.0.smithi076.stdout:1404: read oid 14 snap -1 2021-03-15T22:25:00.173 INFO:tasks.rados.rados.0.smithi076.stdout:1404: expect (ObjNum 453 snap 132 seq_num 453) 2021-03-15T22:25:00.173 INFO:tasks.rados.rados.0.smithi076.stdout:1405: write (excl) oid 20 current snap is 150 2021-03-15T22:25:00.173 INFO:tasks.rados.rados.0.smithi076.stdout:1405: seq_num 502 ranges {437114=654296,1525843=477506} 2021-03-15T22:25:00.177 INFO:tasks.rados.rados.0.smithi076.stdout:1405: writing smithi07694878-20 from 437114 to 1091410 tid 1 2021-03-15T22:25:00.184 INFO:tasks.rados.rados.0.smithi076.stdout:1405: writing smithi07694878-20 from 1525843 to 2003349 tid 2 2021-03-15T22:25:00.184 INFO:tasks.rados.rados.0.smithi076.stdout:1406: read oid 34 snap -1 2021-03-15T22:25:00.184 INFO:tasks.rados.rados.0.smithi076.stdout:1406: expect deleted 2021-03-15T22:25:00.184 INFO:tasks.rados.rados.0.smithi076.stdout:1407: write (excl) oid 50 current snap is 150 2021-03-15T22:25:00.185 INFO:tasks.rados.rados.0.smithi076.stdout:1402: finishing write tid 2 to smithi07694878-26 2021-03-15T22:25:00.185 INFO:tasks.rados.rados.0.smithi076.stdout:1407: seq_num 503 ranges {453921=729545,1584325=621164} 2021-03-15T22:25:00.190 INFO:tasks.rados.rados.0.smithi076.stdout:1407: writing smithi07694878-50 from 453921 to 1183466 tid 1 2021-03-15T22:25:00.197 INFO:tasks.rados.rados.0.smithi076.stdout:1407: writing smithi07694878-50 from 1584325 to 2205489 tid 2 2021-03-15T22:25:00.197 INFO:tasks.rados.rados.0.smithi076.stdout:1408: rollback oid 15 current snap is 150 2021-03-15T22:25:00.198 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 15 to 117 2021-03-15T22:25:00.198 INFO:tasks.rados.rados.0.smithi076.stdout:1402: finishing write tid 3 to smithi07694878-26 2021-03-15T22:25:00.198 INFO:tasks.rados.rados.0.smithi076.stdout:1402: finishing write tid 4 to smithi07694878-26 2021-03-15T22:25:00.198 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 660 (ObjNum 500 snap 150 seq_num 500) dirty exists 2021-03-15T22:25:00.199 INFO:tasks.rados.rados.0.smithi076.stdout:1402: left oid 26 (ObjNum 500 snap 150 seq_num 500) 2021-03-15T22:25:00.199 INFO:tasks.rados.rados.0.smithi076.stdout:1402: done (5 left) 2021-03-15T22:25:00.199 INFO:tasks.rados.rados.0.smithi076.stdout:1409: write (excl) oid 43 current snap is 150 2021-03-15T22:25:00.200 INFO:tasks.rados.rados.0.smithi076.stdout:1404: expect (ObjNum 453 snap 132 seq_num 453) 2021-03-15T22:25:00.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:00.261 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:00.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:00.364 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:00.371 INFO:tasks.rados.rados.0.smithi076.stdout:1407: finishing write tid 1 to smithi07694878-50 2021-03-15T22:25:00.371 INFO:tasks.rados.rados.0.smithi076.stdout:1405: finishing write tid 1 to smithi07694878-20 2021-03-15T22:25:00.372 INFO:tasks.rados.rados.0.smithi076.stdout:1408: finishing rollback tid 0 to smithi07694878-15 2021-03-15T22:25:00.372 INFO:tasks.rados.rados.0.smithi076.stdout:1408: finishing rollback tid 1 to smithi07694878-15 2021-03-15T22:25:00.372 INFO:tasks.rados.rados.0.smithi076.stdout:1409: seq_num 504 ranges {470728=404794,1526454=481175} 2021-03-15T22:25:00.376 INFO:tasks.rados.rados.0.smithi076.stdout:1409: writing smithi07694878-43 from 470728 to 875522 tid 1 2021-03-15T22:25:00.383 INFO:tasks.rados.rados.0.smithi076.stdout:1409: writing smithi07694878-43 from 1526454 to 2007629 tid 2 2021-03-15T22:25:00.383 INFO:tasks.rados.rados.0.smithi076.stdout:1404: done (5 left) 2021-03-15T22:25:00.383 INFO:tasks.rados.rados.0.smithi076.stdout:1406: done (4 left) 2021-03-15T22:25:00.384 INFO:tasks.rados.rados.0.smithi076.stdout:1410: read oid 39 snap -1 2021-03-15T22:25:00.384 INFO:tasks.rados.rados.0.smithi076.stdout:1410: expect deleted 2021-03-15T22:25:00.384 INFO:tasks.rados.rados.0.smithi076.stdout:1411: snap_create 2021-03-15T22:25:00.385 INFO:tasks.rados.rados.0.smithi076.stdout:1408: finishing rollback tid 2 to smithi07694878-15 2021-03-15T22:25:00.385 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 820 (ObjNum 371 snap 102 seq_num 371) dirty exists 2021-03-15T22:25:00.385 INFO:tasks.rados.rados.0.smithi076.stdout:1405: finishing write tid 2 to smithi07694878-20 2021-03-15T22:25:00.385 INFO:tasks.rados.rados.0.smithi076.stdout:1405: finishing write tid 3 to smithi07694878-20 2021-03-15T22:25:00.386 INFO:tasks.rados.rados.0.smithi076.stdout:1405: finishing write tid 4 to smithi07694878-20 2021-03-15T22:25:00.386 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 456 (ObjNum 502 snap 150 seq_num 502) dirty exists 2021-03-15T22:25:00.386 INFO:tasks.rados.rados.0.smithi076.stdout:1405: left oid 20 (ObjNum 502 snap 150 seq_num 502) 2021-03-15T22:25:00.387 INFO:tasks.rados.rados.0.smithi076.stdout:1407: finishing write tid 2 to smithi07694878-50 2021-03-15T22:25:00.387 INFO:tasks.rados.rados.0.smithi076.stdout:1407: finishing write tid 3 to smithi07694878-50 2021-03-15T22:25:00.387 INFO:tasks.rados.rados.0.smithi076.stdout:1407: finishing write tid 4 to smithi07694878-50 2021-03-15T22:25:00.387 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 663 (ObjNum 503 snap 150 seq_num 503) dirty exists 2021-03-15T22:25:00.388 INFO:tasks.rados.rados.0.smithi076.stdout:1407: left oid 50 (ObjNum 503 snap 150 seq_num 503) 2021-03-15T22:25:00.388 INFO:tasks.rados.rados.0.smithi076.stdout:1409: finishing write tid 1 to smithi07694878-43 2021-03-15T22:25:00.389 INFO:tasks.rados.rados.0.smithi076.stdout:1409: finishing write tid 2 to smithi07694878-43 2021-03-15T22:25:00.389 INFO:tasks.rados.rados.0.smithi076.stdout:1409: finishing write tid 3 to smithi07694878-43 2021-03-15T22:25:00.390 INFO:tasks.rados.rados.0.smithi076.stdout:1409: finishing write tid 4 to smithi07694878-43 2021-03-15T22:25:00.390 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 755 (ObjNum 504 snap 150 seq_num 504) dirty exists 2021-03-15T22:25:00.390 INFO:tasks.rados.rados.0.smithi076.stdout:1409: left oid 43 (ObjNum 504 snap 150 seq_num 504) 2021-03-15T22:25:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:00.392 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:00.392 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:00.494 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:00.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:00.518 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:00.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:00.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:00.646 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:00.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:00.748 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:00.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: cluster 2021-03-15T22:24:59.046329 2021-03-15T22:25:00.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 598 : cluster [DBG] pgmap v702: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 269 MiB data, 811 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 17 op/s; 1/1347 objects misplaced (0.074%); 11 B/s, 0 objects/s recovering 2021-03-15T22:25:00.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:25:00.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: 03-15T22:24:59.383439+0000 mon.a (mon.0) 1167 : cluster [DBG] 2021-03-15T22:25:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: osdmap e384: 8 total, 8 up, 7 in 2021-03-15T22:25:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: audit 2021-03-15T22:24:59 2021-03-15T22:25:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: .448284+0000 mon.a (mon.0) 1168 : audit 2021-03-15T22:25:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/3490284777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:25:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: cluster 2021-03-15T22:25:00.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:25:00.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: 24:59.606618+0000 osd.4 ( 2021-03-15T22:25:00.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: osd.4) 83 : cluster 2021-03-15T22:25:00.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: [DBG] purged_snaps scrub starts 2021-03-15T22:25:00.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: cluster 2021-03-15T22:24:59.606989+0000 osd.4 (osd.4) 84 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:00.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: audit 2021-03-15T22:24:59.680575+0000 mgr.y (mgr.14144) 599 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:00.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: audit 2021-03-15T22:24:59.712337+0000 mon.a (mon.0) 1169 : audit [INF] from='client.? 172.21.15.76:0/1548700794' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-15T22:25:00.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: cluster 2021-03-15T22:25:00.082228+0000 mon.a (mon.0) 1170 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:00.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: audit 2021-03-15T22:25:00.088153+0000 mon.a (mon.0) 1171 : audit [INF] from='client.? 172.21.15.76:0/3490284777' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:25:00.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: audit 2021-03-15T22:25:00.088262+0000 mon.a (mon.0) 1172 : audit [INF] from='client.? 172.21.15.76:0/1548700794' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-15T22:25:00.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[62156]: cluster 2021-03-15T22:25:00.088329+0000 mon.a (mon.0) 1173 : cluster [DBG] osdmap e385: 8 total, 8 up, 7 in 2021-03-15T22:25:00.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: cluster 2021-03-15T22:25:00.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: 2021-03-15T22:24:59.046329+0000 mgr.y (mgr.14144) 598 : cluster [DBG] pgmap v702: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 269 MiB data, 811 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 17 op/s; 1/1347 objects misplaced (0.074%); 11 B/s, 0 objects/s recovering 2021-03-15T22:25:00.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: cluster 2021-03-15T22:24:59.383439+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2021-03-15T22:25:00.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: audit 2021-03-15T22:24:59.448284+0000 mon.a (mon.0) 1168 : audit [INF] from='client.? 172.21.15.76:0/3490284777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:25:00.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: cluster 2021-03-15T22:24:59.606618+0000 osd.4 (osd.4) 83 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:00.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: cluster 2021-03-15T22:24:59.606989+0000 osd.4 (osd.4) 84 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:00.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: audit 2021-03-15T22: 2021-03-15T22:25:00.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: 24:59.680575+0000 mgr.y (mgr.14144) 2021-03-15T22:25:00.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: 599 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:00.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: audit 2021-03-15T22: 2021-03-15T22:25:00.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: 24:59.712337+0000 mon.a (mon.0) 2021-03-15T22:25:00.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: 1169 : audit [INF] from='client.? 172.21.15.76:0/1548700794' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-15T22:25:00.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:25:00.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: 25:00.082228+0000 2021-03-15T22:25:00.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: mon.a (mon.0) 1170 2021-03-15T22:25:00.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:00.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: audit 2021-03-15T22:25:00.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: 2021-03-15T22:25:00.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: 2021- 2021-03-15T22:25:00.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: 03-15T22:25 2021-03-15T22:25:00.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: :00.088153+0000 mon.a ( 2021-03-15T22:25:00.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: mon.0) 1171 : audit [INF] from='client.? 172.21.15.76:0/3490284777' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:25:00.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: 2021-03-15T22:25:00.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: audit 2021-03-15T22:25:00.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: 2021-03-15T22:25:00.088262+0000 mon.a (mon.0 2021-03-15T22:25:00.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: ) 1172 : 2021-03-15T22:25:00.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: audit [INF] from='client.? 172.21.15.76:0/1548700794' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-15T22:25:00.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:25:00.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: -03-15T22:25:00 2021-03-15T22:25:00.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: .088329 2021-03-15T22:25:00.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1173 : cluster [DBG] 2021-03-15T22:25:00.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:00 smithi076 conmon[56514]: osdmap e385: 8 total, 8 up, 7 in 2021-03-15T22:25:00.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:00.782 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:00.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: cluster 2021-03-15T22:24:59.046329+0000 mgr.y (mgr.14144) 598 : cluster [DBG] 2021-03-15T22:25:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: pgmap v702: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 269 MiB data, 811 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.4 MiB/s wr, 17 op/s; 1/1347 objects misplaced (0.074%); 11 B/s, 0 objects/s recovering 2021-03-15T22:25:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: cluster 2021-03-15T22:24:59.383439+0000 mon.a (mon.0) 1167 : cluster [DBG] osdmap e384: 8 total, 8 up, 7 in 2021-03-15T22:25:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: audit 2021-03-15T22:24:59.448284+0000 2021-03-15T22:25:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: mon.a (mon.0) 1168 : audit [INF] from='client.? 172.21.15.76:0/3490284777' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:25:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:25:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: -15T22:24:59.606618+0000 osd.4 2021-03-15T22:25:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: (osd.4) 83 : cluster 2021-03-15T22:25:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: [DBG] purged_snaps scrub starts 2021-03-15T22:25:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:25:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: -15T22:24:59.606989+0000 osd.4 (osd 2021-03-15T22:25:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: .4) 84 : cluster [DBG] 2021-03-15T22:25:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: purged_snaps scrub ok 2021-03-15T22:25:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:25:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: -15T22:24:59.680575+0000 mgr.y 2021-03-15T22:25:00.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: (mgr.14144) 599 : audit 2021-03-15T22:25:00.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:00.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:25:00.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: 03-15T22:24:59.712337+0000 mon.a 2021-03-15T22:25:00.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: (mon.0) 1169 : audit 2021-03-15T22:25:00.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/1548700794' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-15T22:25:00.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:00.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: 03-15T22:25:00.082228+0000 2021-03-15T22:25:00.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: mon.a (mon.0) 1170 : cluster 2021-03-15T22:25:00.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:00.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:25:00.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: 03-15T22:25:00.088153+0000 2021-03-15T22:25:00.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: mon.a (mon.0) 1171 : audit 2021-03-15T22:25:00.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/3490284777' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:25:00.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:25:00.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: 03-15T22:25:00.088262+0000 2021-03-15T22:25:00.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: mon.a (mon.0) 1172 : audit 2021-03-15T22:25:00.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/1548700794' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-15T22:25:00.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:00.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: 03-15T22:25:00.088329+0000 mon.a (mon.0) 1173 : cluster 2021-03-15T22:25:00.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:00 smithi125 conmon[56739]: [DBG] osdmap e385: 8 total, 8 up, 7 in 2021-03-15T22:25:00.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:00.905 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:00.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:00.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:01.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:01.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:01.035 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:01.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:01.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:01.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:01.163 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:01.163 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:01.265 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:01.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:01.288 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:01.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:01.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:01.394 INFO:tasks.rados.rados.0.smithi076.stdout:1405: done (5 left) 2021-03-15T22:25:01.394 INFO:tasks.rados.rados.0.smithi076.stdout:1407: done (4 left) 2021-03-15T22:25:01.394 INFO:tasks.rados.rados.0.smithi076.stdout:1408: done (3 left) 2021-03-15T22:25:01.395 INFO:tasks.rados.rados.0.smithi076.stdout:1409: done (2 left) 2021-03-15T22:25:01.395 INFO:tasks.rados.rados.0.smithi076.stdout:1410: done (1 left) 2021-03-15T22:25:01.395 INFO:tasks.rados.rados.0.smithi076.stdout:1411: done (0 left) 2021-03-15T22:25:01.395 INFO:tasks.rados.rados.0.smithi076.stdout:1412: read oid 6 snap 146 2021-03-15T22:25:01.395 INFO:tasks.rados.rados.0.smithi076.stdout:1412: expect (ObjNum 486 snap 144 seq_num 486) 2021-03-15T22:25:01.396 INFO:tasks.rados.rados.0.smithi076.stdout:1413: snap_remove snap 142 2021-03-15T22:25:01.418 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:01.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:01.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:01.521 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:01.546 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:01.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:01.592 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 4 1 2021-03-15T22:25:01.650 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:01.676 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:01.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:01.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:01.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[56514]: cluster 2021-03-15T22:25:00.614284+0000 osd.2 2021-03-15T22:25:01.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[56514]: (osd.2) 65 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:25:01.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[56514]: cluster 2021-03-15T22:25:00.689585+0000 osd.5 (osd.5) 69 : cluster [DBG] 6.a scrub ok 2021-03-15T22:25:01.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[56514]: audit 2021-03-15T22:25:01.089388+0000 mon.a (mon.0) 1174 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:25:01.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[56514]: audit 2021-03-15T22:25:01.091588+0000 mon.a (mon.0) 1175 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:25:01.688 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[62156]: cluster 2021-03-15T22:25:00.614284+0000 2021-03-15T22:25:01.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[62156]: osd.2 (osd.2) 65 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:25:01.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[62156]: cluster 2021-03-15T22:25:00.689585+0000 osd.5 (osd.5 2021-03-15T22:25:01.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[62156]: ) 69 : cluster [DBG] 6.a scrub ok 2021-03-15T22:25:01.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[62156]: audit 2021-03-15T22:25:01.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[62156]: 2021-03-15T22:25:01.089388+0000 mon.a ( 2021-03-15T22:25:01.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[62156]: mon.0) 1174 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:25:01.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[62156]: audit 2021-03-15T22:25:01.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[62156]: 2021-03-15T22:25:01. 2021-03-15T22:25:01.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[62156]: 091588+0000 mon.a (mon.0) 1175 : audit 2021-03-15T22:25:01.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:01 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:25:01.779 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:01.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:01 smithi125 conmon[56739]: cluster 2021-03-15T22:25:00.614284+0000 osd.2 (osd.2) 65 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:25:01.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:01 smithi125 conmon[56739]: cluster 2021-03-15T22:25:00.689585+0000 osd.5 (osd.5) 69 : cluster [DBG] 6.a scrub ok 2021-03-15T22:25:01.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:01 smithi125 conmon[56739]: audit 2021-03-15T22:25:01.089388+0000 mon.a 2021-03-15T22:25:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:01 smithi125 conmon[56739]: (mon.0) 1174 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:25:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:01 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:25:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:01 smithi125 conmon[56739]: 03-15T22:25:01.091588+0000 2021-03-15T22:25:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:01 smithi125 conmon[56739]: mon.a (mon.0) 1175 : audit 2021-03-15T22:25:01.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:01 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:25:01.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:01.806 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:01.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:01.908 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:01.930 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:01.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:01.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:02.033 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:02.060 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:02.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:02.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:02.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:02.186 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:02.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:02.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:02.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:02.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:02.311 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:02.312 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:02.399 INFO:tasks.rados.rados.0.smithi076.stdout:1413: done (1 left) 2021-03-15T22:25:02.399 INFO:tasks.rados.rados.0.smithi076.stdout:1414: write (excl) oid 2 current snap is 151 2021-03-15T22:25:02.400 INFO:tasks.rados.rados.0.smithi076.stdout:1412: expect (ObjNum 486 snap 144 seq_num 486) 2021-03-15T22:25:02.413 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:02.435 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:02.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:02.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:02.496 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:02.535 INFO:tasks.rados.rados.0.smithi076.stdout:1414: seq_num 505 ranges {487535=480043,1584936=624833} 2021-03-15T22:25:02.538 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:02.541 INFO:tasks.rados.rados.0.smithi076.stdout:1414: writing smithi07694878-2 from 487535 to 967578 tid 1 2021-03-15T22:25:02.548 INFO:tasks.rados.rados.0.smithi076.stdout:1414: writing smithi07694878-2 from 1584936 to 2209769 tid 2 2021-03-15T22:25:02.548 INFO:tasks.rados.rados.0.smithi076.stdout:1412: done (1 left) 2021-03-15T22:25:02.548 INFO:tasks.rados.rados.0.smithi076.stdout:1415: read oid 4 snap -1 2021-03-15T22:25:02.549 INFO:tasks.rados.rados.0.smithi076.stdout:1415: expect deleted 2021-03-15T22:25:02.549 INFO:tasks.rados.rados.0.smithi076.stdout:1416: write (excl) oid 17 current snap is 151 2021-03-15T22:25:02.549 INFO:tasks.rados.rados.0.smithi076.stdout:1414: finishing write tid 1 to smithi07694878-2 2021-03-15T22:25:02.549 INFO:tasks.rados.rados.0.smithi076.stdout:1416: seq_num 506 ranges {504342=555292,1643418=484844} 2021-03-15T22:25:02.553 INFO:tasks.rados.rados.0.smithi076.stdout:1416: writing smithi07694878-17 from 504342 to 1059634 tid 1 2021-03-15T22:25:02.559 INFO:tasks.rados.rados.0.smithi076.stdout:1416: writing smithi07694878-17 from 1643418 to 2128262 tid 2 2021-03-15T22:25:02.560 INFO:tasks.rados.rados.0.smithi076.stdout:1417: read oid 46 snap -1 2021-03-15T22:25:02.560 INFO:tasks.rados.rados.0.smithi076.stdout:1417: expect (ObjNum 469 snap 137 seq_num 469) 2021-03-15T22:25:02.560 INFO:tasks.rados.rados.0.smithi076.stdout:1418: snap_remove snap 149 2021-03-15T22:25:02.562 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:02.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:02.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:02.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:02.679 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:02 smithi076 conmon[56514]: cluster 2021-03-15T22:25:01.046779+0000 mgr.y (mgr.14144) 600 : cluster [DBG] pgmap v705: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 277 MiB data, 835 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 16 op/s; 1/1356 objects misplaced (0.074%) 2021-03-15T22:25:02.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:02 smithi076 conmon[56514]: cluster 2021-03-15T22:25:01.148098+0000 osd.7 (osd.7) 74 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:25:02.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:02 smithi076 conmon[56514]: cluster 2021-03-15T22:25:01.389388+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2021-03-15T22:25:02.680 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:02 smithi076 conmon[62156]: cluster 2021-03-15T22:25:01.046779+0000 mgr.y (mgr.14144) 600 : cluster [DBG] pgmap v705: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 277 MiB data, 835 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 16 op/s; 1/1356 objects misplaced (0.074%) 2021-03-15T22:25:02.680 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:02 smithi076 conmon[62156]: cluster 2021-03-15T22:25:01.148098+0000 osd.7 (osd.7) 74 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:25:02.681 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:02 smithi076 conmon[62156]: cluster 2021-03-15T22:25:01.389388+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2021-03-15T22:25:02.689 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:02.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:02.690 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:02 smithi125 conmon[56739]: cluster 2021-03-15T22:25:01.046779+0000 mgr.y (mgr.14144) 600 : cluster [DBG] pgmap v705: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 277 MiB data, 835 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.9 MiB/s wr, 16 op/s; 1/1356 objects misplaced (0.074%) 2021-03-15T22:25:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:02 smithi125 conmon[56739]: 2021-03-15T22:25:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:02 smithi125 conmon[56739]: cluster 2021-03-15T22:25:01.148098+0000 osd.7 (osd.7) 74 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:25:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:02 smithi125 conmon[56739]: cluster 2021-03-15T22:25:01.389388+0000 mon.a (mon.0) 1176 : cluster [DBG] osdmap e386: 8 total, 8 up, 7 in 2021-03-15T22:25:02.791 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:02.810 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:25:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:25:02.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:25:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:25:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDScrubReserve(6.1b REQUEST e194)", 2021-03-15T22:25:02.811 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:25:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 193.022388345, 2021-03-15T22:25:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.061386018000000001, 2021-03-15T22:25:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:02.812 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:25:02.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:25:02.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6899999999999999e-06 2021-03-15T22:25:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:02.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783093+0000", 2021-03-15T22:25:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5640000000000001e-06 2021-03-15T22:25:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:02.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783097+0000", 2021-03-15T22:25:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7742e-05 2021-03-15T22:25:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783125+0000", 2021-03-15T22:25:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3793e-05 2021-03-15T22:25:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783138+0000", 2021-03-15T22:25:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8529999999999993e-05 2021-03-15T22:25:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783237+0000", 2021-03-15T22:25:02.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.241e-05 2021-03-15T22:25:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:25:02.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783249+0000", 2021-03-15T22:25:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013943790000000001 2021-03-15T22:25:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.784644+0000", 2021-03-15T22:25:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2398000000000001e-05 2021-03-15T22:25:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.784656+0000", 2021-03-15T22:25:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.059821512 2021-03-15T22:25:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.844478+0000", 2021-03-15T22:25:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.061386018000000001 2021-03-15T22:25:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.825 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.6.0:214 7.d e205/91 7:bd183e1b:::smithi07694878-46:68 v 205'30, mlcod=202'27)", 2021-03-15T22:25:02.826 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:25:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 182.71636532900001, 2021-03-15T22:25:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036819649000000003, 2021-03-15T22:25:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:02.827 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:25:02.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:25:02.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2879999999999999e-06 2021-03-15T22:25:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:02.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089117+0000", 2021-03-15T22:25:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4639e-05 2021-03-15T22:25:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:02.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089132+0000", 2021-03-15T22:25:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0759e-05 2021-03-15T22:25:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:02.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089142+0000", 2021-03-15T22:25:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8793000000000001e-05 2021-03-15T22:25:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089161+0000", 2021-03-15T22:25:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3105999999999998e-05 2021-03-15T22:25:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089194+0000", 2021-03-15T22:25:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3468e-05 2021-03-15T22:25:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089208+0000", 2021-03-15T22:25:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036675487 2021-03-15T22:25:02.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125883+0000", 2021-03-15T22:25:02.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1109000000000001e-05 2021-03-15T22:25:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125934+0000", 2021-03-15T22:25:02.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036819649000000003 2021-03-15T22:25:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.841 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.0.0:708 7.4 e266/91 7:26337d05:::smithi07694878-20:95 v 266'191, mlcod=261'185)", 2021-03-15T22:25:02.841 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.063309+0000", 2021-03-15T22:25:02.841 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 121.742170673, 2021-03-15T22:25:02.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016260241000000002, 2021-03-15T22:25:02.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:02.842 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:02.842 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:02.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:02.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063309+0000", 2021-03-15T22:25:02.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:02.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:02.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063309+0000", 2021-03-15T22:25:02.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3489999999999999e-06 2021-03-15T22:25:02.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:02.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063315+0000", 2021-03-15T22:25:02.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.779e-06 2021-03-15T22:25:02.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:02.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063316+0000", 2021-03-15T22:25:02.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7019999999999998e-06 2021-03-15T22:25:02.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:02.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063322+0000", 2021-03-15T22:25:02.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0029999999999996e-06 2021-03-15T22:25:02.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:02.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063326+0000", 2021-03-15T22:25:02.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028956799999999998 2021-03-15T22:25:02.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063616+0000", 2021-03-15T22:25:02.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6943e-05 2021-03-15T22:25:02.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:02.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.063633+0000", 2021-03-15T22:25:02.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0158838 2021-03-15T22:25:02.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:02.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.079517+0000", 2021-03-15T22:25:02.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3096999999999997e-05 2021-03-15T22:25:02.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:02.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.079570+0000", 2021-03-15T22:25:02.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016260241000000002 2021-03-15T22:25:02.853 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.853 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:02.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.854 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.0.0:709 7.9 e266/91 7:91d654eb:::smithi07694878-5:97 v 266'89, mlcod=259'85)", 2021-03-15T22:25:02.855 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.064203+0000", 2021-03-15T22:25:02.855 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 121.741276626, 2021-03-15T22:25:02.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017427448000000002, 2021-03-15T22:25:02.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:02.855 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:02.856 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:02.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:02.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064203+0000", 2021-03-15T22:25:02.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:02.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:02.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064203+0000", 2021-03-15T22:25:02.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.922e-06 2021-03-15T22:25:02.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:02.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064210+0000", 2021-03-15T22:25:02.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.678e-06 2021-03-15T22:25:02.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:02.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064213+0000", 2021-03-15T22:25:02.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0438000000000001e-05 2021-03-15T22:25:02.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:02.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064223+0000", 2021-03-15T22:25:02.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5829999999999997e-06 2021-03-15T22:25:02.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:02.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064229+0000", 2021-03-15T22:25:02.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00054566599999999999 2021-03-15T22:25:02.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064775+0000", 2021-03-15T22:25:02.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2643e-05 2021-03-15T22:25:02.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:02.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.064797+0000", 2021-03-15T22:25:02.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016787641999999998 2021-03-15T22:25:02.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:02.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.081585+0000", 2021-03-15T22:25:02.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5875999999999999e-05 2021-03-15T22:25:02.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:02.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.081631+0000", 2021-03-15T22:25:02.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017427448000000002 2021-03-15T22:25:02.866 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.867 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:02.867 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.867 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.1.0:233 7.b e266/91 ondisk, result = 0)", 2021-03-15T22:25:02.868 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:01.065176+0000", 2021-03-15T22:25:02.868 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 121.74030416700001, 2021-03-15T22:25:02.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014869920999999999, 2021-03-15T22:25:02.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:02.868 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:02.869 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:02.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:02.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065176+0000", 2021-03-15T22:25:02.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:02.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:02.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065176+0000", 2021-03-15T22:25:02.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5919999999999999e-06 2021-03-15T22:25:02.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:02.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065178+0000", 2021-03-15T22:25:02.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.519e-06 2021-03-15T22:25:02.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:02.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065184+0000", 2021-03-15T22:25:02.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5899999999999997e-06 2021-03-15T22:25:02.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:02.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065194+0000", 2021-03-15T22:25:02.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6010000000000002e-06 2021-03-15T22:25:02.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:02.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.065199+0000", 2021-03-15T22:25:02.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014821381999999999 2021-03-15T22:25:02.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080021+0000", 2021-03-15T22:25:02.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0757000000000001e-05 2021-03-15T22:25:02.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:02.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080031+0000", 2021-03-15T22:25:02.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.448e-05 2021-03-15T22:25:02.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:02.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:01.080046+0000", 2021-03-15T22:25:02.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014869920999999999 2021-03-15T22:25:02.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.878 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:02.879 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.879 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2768 7.b 7:d10ba14d:::smithi07694878-43:head [write 1689059~741123] snapc d2=[d2,d1,cf] ondisk+write+known_if_redirected e310)", 2021-03-15T22:25:02.880 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:25:02.880 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.700389974999993, 2021-03-15T22:25:02.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078094859000000003, 2021-03-15T22:25:02.880 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:02.880 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:02.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:02.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:02.881 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:02.881 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2768 2021-03-15T22:25:02.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.882 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:02.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:02.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:25:02.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:02.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:02.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:25:02.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033110399999999998 2021-03-15T22:25:02.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:02.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105421+0000", 2021-03-15T22:25:02.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4809999999999998e-06 2021-03-15T22:25:02.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:02.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105425+0000", 2021-03-15T22:25:02.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7363999999999998e-05 2021-03-15T22:25:02.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:02.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105442+0000", 2021-03-15T22:25:02.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9480000000000002e-06 2021-03-15T22:25:02.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:02.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105449+0000", 2021-03-15T22:25:02.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6847000000000001e-05 2021-03-15T22:25:02.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105496+0000", 2021-03-15T22:25:02.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8231000000000003e-05 2021-03-15T22:25:02.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:02.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105584+0000", 2021-03-15T22:25:02.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000198369 2021-03-15T22:25:02.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:25:02.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105782+0000", 2021-03-15T22:25:02.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003007292 2021-03-15T22:25:02.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:02.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108790+0000", 2021-03-15T22:25:02.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000200027 2021-03-15T22:25:02.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:02.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108990+0000", 2021-03-15T22:25:02.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074107929000000003 2021-03-15T22:25:02.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:02.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183098+0000", 2021-03-15T22:25:02.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4002000000000002e-05 2021-03-15T22:25:02.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:02.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183152+0000", 2021-03-15T22:25:02.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3265000000000001e-05 2021-03-15T22:25:02.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:02.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183185+0000", 2021-03-15T22:25:02.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078094859000000003 2021-03-15T22:25:02.897 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.897 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:02.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.898 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2769 7.b 7:d10ba14d:::smithi07694878-43:head [setxattr _header (58) in=7b,truncate 2588344] snapc d2=[d2,d1,cf] ondisk+write+known_if_redirected e310)", 2021-03-15T22:25:02.899 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:25:02.899 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.700018947000004, 2021-03-15T22:25:02.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077795487999999996, 2021-03-15T22:25:02.899 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:02.899 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:02.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:02.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:02.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:02.900 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2769 2021-03-15T22:25:02.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.901 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:02.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:02.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:25:02.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:02.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:02.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:25:02.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2639999999999998e-06 2021-03-15T22:25:02.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:02.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105463+0000", 2021-03-15T22:25:02.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.426e-06 2021-03-15T22:25:02.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:02.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105465+0000", 2021-03-15T22:25:02.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9349999999999999e-06 2021-03-15T22:25:02.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:02.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105471+0000", 2021-03-15T22:25:02.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2420000000000002e-06 2021-03-15T22:25:02.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:02.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105475+0000", 2021-03-15T22:25:02.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001413924 2021-03-15T22:25:02.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.106889+0000", 2021-03-15T22:25:02.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5030000000000003e-05 2021-03-15T22:25:02.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:02.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.106984+0000", 2021-03-15T22:25:02.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014550399999999999 2021-03-15T22:25:02.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:25:02.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107129+0000", 2021-03-15T22:25:02.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001729529 2021-03-15T22:25:02.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:02.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108859+0000", 2021-03-15T22:25:02.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023022799999999999 2021-03-15T22:25:02.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:02.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.109089+0000", 2021-03-15T22:25:02.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074107424000000005 2021-03-15T22:25:02.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:02.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183197+0000", 2021-03-15T22:25:02.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7657e-05 2021-03-15T22:25:02.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:02.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183224+0000", 2021-03-15T22:25:02.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2325e-05 2021-03-15T22:25:02.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:02.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183257+0000", 2021-03-15T22:25:02.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077795487999999996 2021-03-15T22:25:02.916 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.916 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:02.916 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.917 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2770 7.b 7:d10ba14d:::smithi07694878-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e310)", 2021-03-15T22:25:02.917 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:25:02.917 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 77.699988739000005, 2021-03-15T22:25:02.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077923611000000004, 2021-03-15T22:25:02.917 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:02.918 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:02.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:02.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:02.918 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:02.919 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2770 2021-03-15T22:25:02.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.919 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:02.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:02.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:25:02.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:02.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:02.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:25:02.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3200000000000001e-06 2021-03-15T22:25:02.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:02.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105493+0000", 2021-03-15T22:25:02.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3960000000000001e-06 2021-03-15T22:25:02.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:02.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105494+0000", 2021-03-15T22:25:02.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6490000000000002e-06 2021-03-15T22:25:02.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:02.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105501+0000", 2021-03-15T22:25:02.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9989999999999999e-06 2021-03-15T22:25:02.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:02.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105506+0000", 2021-03-15T22:25:02.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019212770000000001 2021-03-15T22:25:02.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107427+0000", 2021-03-15T22:25:02.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8102000000000005e-05 2021-03-15T22:25:02.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:02.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107515+0000", 2021-03-15T22:25:02.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075777318999999996 2021-03-15T22:25:02.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183292+0000", 2021-03-15T22:25:02.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9417999999999999e-05 2021-03-15T22:25:02.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183352+0000", 2021-03-15T22:25:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3131000000000003e-05 2021-03-15T22:25:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:02.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183415+0000", 2021-03-15T22:25:02.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077923611000000004 2021-03-15T22:25:02.932 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.932 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:02.933 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.933 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3010 7.b 7:d980be1f:::smithi07694878-23:head [write 726168~704991,stat] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:25:02.933 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:25:02.934 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.718967161000002, 2021-03-15T22:25:02.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0570805230000002, 2021-03-15T22:25:02.934 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:02.934 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:02.934 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:02.935 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:02.935 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:02.935 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3010 2021-03-15T22:25:02.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.936 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:02.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:02.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:25:02.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:02.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:02.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:25:02.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034716800000000002 2021-03-15T22:25:02.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:02.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086860+0000", 2021-03-15T22:25:02.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3299999999999999e-06 2021-03-15T22:25:02.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:02.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086863+0000", 2021-03-15T22:25:02.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1489e-05 2021-03-15T22:25:02.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:02.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086875+0000", 2021-03-15T22:25:02.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.605e-06 2021-03-15T22:25:02.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:02.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086882+0000", 2021-03-15T22:25:02.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5296999999999997e-05 2021-03-15T22:25:02.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086928+0000", 2021-03-15T22:25:02.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2879999999999998e-06 2021-03-15T22:25:02.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:25:02.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086937+0000", 2021-03-15T22:25:02.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99891912599999999 2021-03-15T22:25:02.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085856+0000", 2021-03-15T22:25:02.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1169999999999993e-06 2021-03-15T22:25:02.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:25:02.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085865+0000", 2021-03-15T22:25:02.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0140395639999999 2021-03-15T22:25:02.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.099905+0000", 2021-03-15T22:25:02.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022436600000000001 2021-03-15T22:25:02.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:02.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.100129+0000", 2021-03-15T22:25:02.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00043622700000000002 2021-03-15T22:25:02.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:25:02.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.100565+0000", 2021-03-15T22:25:02.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024326999999999999 2021-03-15T22:25:02.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:02.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102998+0000", 2021-03-15T22:25:02.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002982012 2021-03-15T22:25:02.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:02.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.105980+0000", 2021-03-15T22:25:02.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037236714999999997 2021-03-15T22:25:02.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:02.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143217+0000", 2021-03-15T22:25:02.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147832 2021-03-15T22:25:02.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:02.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143365+0000", 2021-03-15T22:25:02.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022868699999999999 2021-03-15T22:25:02.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:02.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143593+0000", 2021-03-15T22:25:02.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0570805230000002 2021-03-15T22:25:02.956 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.956 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:02.956 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.957 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3011 7.b 7:d980be1f:::smithi07694878-23:head [write 1895071~683474] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:25:02.957 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:25:02.957 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.712686914999999, 2021-03-15T22:25:02.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.09731113, 2021-03-15T22:25:02.957 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:02.958 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:02.958 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:02.958 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:02.958 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:02.959 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3011 2021-03-15T22:25:02.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.959 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:02.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:02.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:25:02.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:02.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:02.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:25:02.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027883500000000001 2021-03-15T22:25:02.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:02.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093072+0000", 2021-03-15T22:25:02.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.809e-06 2021-03-15T22:25:02.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:02.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093077+0000", 2021-03-15T22:25:02.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9491e-05 2021-03-15T22:25:02.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:02.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093096+0000", 2021-03-15T22:25:02.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.126e-06 2021-03-15T22:25:02.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:02.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093103+0000", 2021-03-15T22:25:02.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4002000000000001e-05 2021-03-15T22:25:02.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093167+0000", 2021-03-15T22:25:02.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0259999999999996e-06 2021-03-15T22:25:02.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:25:02.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093176+0000", 2021-03-15T22:25:02.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99270957599999998 2021-03-15T22:25:02.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085886+0000", 2021-03-15T22:25:02.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9440000000000002e-06 2021-03-15T22:25:02.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:25:02.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085892+0000", 2021-03-15T22:25:02.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.016173532 2021-03-15T22:25:02.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102065+0000", 2021-03-15T22:25:02.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000141037 2021-03-15T22:25:02.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:02.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102207+0000", 2021-03-15T22:25:02.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041447062 2021-03-15T22:25:02.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143654+0000", 2021-03-15T22:25:02.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014246199999999999 2021-03-15T22:25:02.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:02.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143796+0000", 2021-03-15T22:25:02.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000272444 2021-03-15T22:25:02.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:25:02.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.144068+0000", 2021-03-15T22:25:02.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018956929999999999 2021-03-15T22:25:02.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:02.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145964+0000", 2021-03-15T22:25:02.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041997686999999999 2021-03-15T22:25:02.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:02.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.187962+0000", 2021-03-15T22:25:02.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019156150000000001 2021-03-15T22:25:02.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:02.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.189877+0000", 2021-03-15T22:25:02.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8467999999999995e-05 2021-03-15T22:25:02.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:02.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.189956+0000", 2021-03-15T22:25:02.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014832100000000001 2021-03-15T22:25:02.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:02.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190104+0000", 2021-03-15T22:25:02.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.09731113 2021-03-15T22:25:02.980 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.980 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:02.980 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:02.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.981 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3012 7.b 7:d980be1f:::smithi07694878-23:head [setxattr _header (58) in=7b,truncate 3126168] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:25:02.981 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:25:02.981 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.712364082999997, 2021-03-15T22:25:02.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0972930829999998, 2021-03-15T22:25:02.982 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:02.982 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:02.982 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:02.983 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:02.983 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:02.983 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3012 2021-03-15T22:25:02.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.983 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:02.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:02.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:25:02.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:02.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:02.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:25:02.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.61e-06 2021-03-15T22:25:02.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:02.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093118+0000", 2021-03-15T22:25:02.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.234e-06 2021-03-15T22:25:02.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:02.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093119+0000", 2021-03-15T22:25:02.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7290000000000002e-06 2021-03-15T22:25:02.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:02.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093123+0000", 2021-03-15T22:25:02.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1649999999999999e-06 2021-03-15T22:25:02.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:02.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093127+0000", 2021-03-15T22:25:02.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6592e-05 2021-03-15T22:25:02.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093213+0000", 2021-03-15T22:25:02.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2364000000000001e-05 2021-03-15T22:25:02.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:25:02.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093226+0000", 2021-03-15T22:25:02.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99268559899999997 2021-03-15T22:25:02.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085911+0000", 2021-03-15T22:25:02.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1440000000000002e-06 2021-03-15T22:25:02.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:25:02.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085916+0000", 2021-03-15T22:25:02.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0164949109999999 2021-03-15T22:25:02.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102411+0000", 2021-03-15T22:25:02.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000116049 2021-03-15T22:25:02.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:02.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102527+0000", 2021-03-15T22:25:02.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042432780000000003 2021-03-15T22:25:02.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:02.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.144960+0000", 2021-03-15T22:25:02.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7323000000000003e-05 2021-03-15T22:25:02.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:02.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:02.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:02.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145057+0000", 2021-03-15T22:25:02.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018096599999999999 2021-03-15T22:25:03.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:25:03.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145238+0000", 2021-03-15T22:25:03.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077369500000000003 2021-03-15T22:25:03.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:03.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.146012+0000", 2021-03-15T22:25:03.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041960274999999998 2021-03-15T22:25:03.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:03.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.187972+0000", 2021-03-15T22:25:03.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021867610000000002 2021-03-15T22:25:03.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:03.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190159+0000", 2021-03-15T22:25:03.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9751000000000004e-05 2021-03-15T22:25:03.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:03.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190229+0000", 2021-03-15T22:25:03.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000180135 2021-03-15T22:25:03.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:03.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190409+0000", 2021-03-15T22:25:03.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0972930829999998 2021-03-15T22:25:03.006 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.006 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:03.007 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.007 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3013 7.b 7:d980be1f:::smithi07694878-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e328)", 2021-03-15T22:25:03.008 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:25:03.008 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 59.712330795, 2021-03-15T22:25:03.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.097631684, 2021-03-15T22:25:03.008 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:03.008 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:03.009 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:03.009 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:03.009 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:03.009 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3013 2021-03-15T22:25:03.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.010 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:03.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:03.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:25:03.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:03.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:03.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:25:03.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9600000000000008e-07 2021-03-15T22:25:03.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:03.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093150+0000", 2021-03-15T22:25:03.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.514e-06 2021-03-15T22:25:03.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:03.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093152+0000", 2021-03-15T22:25:03.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2985e-05 2021-03-15T22:25:03.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:03.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093165+0000", 2021-03-15T22:25:03.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4000000000000002e-06 2021-03-15T22:25:03.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:03.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093169+0000", 2021-03-15T22:25:03.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2490999999999993e-05 2021-03-15T22:25:03.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.016 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093252+0000", 2021-03-15T22:25:03.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7490000000000001e-06 2021-03-15T22:25:03.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:25:03.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093258+0000", 2021-03-15T22:25:03.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99267630200000001 2021-03-15T22:25:03.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085935+0000", 2021-03-15T22:25:03.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0580000000000003e-06 2021-03-15T22:25:03.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:25:03.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085941+0000", 2021-03-15T22:25:03.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.01663009 2021-03-15T22:25:03.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.021 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102571+0000", 2021-03-15T22:25:03.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012747 2021-03-15T22:25:03.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:03.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102698+0000", 2021-03-15T22:25:03.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042823333999999998 2021-03-15T22:25:03.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145522+0000", 2021-03-15T22:25:03.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7650000000000004e-05 2021-03-15T22:25:03.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:03.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145599+0000", 2021-03-15T22:25:03.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044843263000000001 2021-03-15T22:25:03.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190443+0000", 2021-03-15T22:25:03.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000117391 2021-03-15T22:25:03.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:03.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190560+0000", 2021-03-15T22:25:03.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000220991 2021-03-15T22:25:03.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:03.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190781+0000", 2021-03-15T22:25:03.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.097631684 2021-03-15T22:25:03.028 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.029 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:03.029 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.029 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.6.0:404 7.7 e330/172 7:e8360a43:::smithi07694878-10:e0 v 330'210, mlcod=329'206)", 2021-03-15T22:25:03.030 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.080876+0000", 2021-03-15T22:25:03.030 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 57.724603989999999, 2021-03-15T22:25:03.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015677255000000001, 2021-03-15T22:25:03.030 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:03.030 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:03.031 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:03.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:03.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.080876+0000", 2021-03-15T22:25:03.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:03.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:03.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.080876+0000", 2021-03-15T22:25:03.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6350000000000002e-06 2021-03-15T22:25:03.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:03.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.080879+0000", 2021-03-15T22:25:03.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.525e-06 2021-03-15T22:25:03.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:03.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.080881+0000", 2021-03-15T22:25:03.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0659999999999998e-06 2021-03-15T22:25:03.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:03.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.080888+0000", 2021-03-15T22:25:03.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7839999999999998e-06 2021-03-15T22:25:03.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:03.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.080895+0000", 2021-03-15T22:25:03.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.01224166 2021-03-15T22:25:03.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.093137+0000", 2021-03-15T22:25:03.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2869000000000001e-05 2021-03-15T22:25:03.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:03.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.093160+0000", 2021-03-15T22:25:03.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0033635420000000002 2021-03-15T22:25:03.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:03.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.096523+0000", 2021-03-15T22:25:03.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0173999999999999e-05 2021-03-15T22:25:03.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:03.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.096553+0000", 2021-03-15T22:25:03.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015677255000000001 2021-03-15T22:25:03.043 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.044 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:03.044 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.044 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:3172 7.7 e344/172 7:ed4a01d2:::smithi07694878-2:head v 344'231, mlcod=344'226)", 2021-03-15T22:25:03.045 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:19.135128+0000", 2021-03-15T22:25:03.045 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 43.670352207000001, 2021-03-15T22:25:03.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.021460856, 2021-03-15T22:25:03.045 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:03.046 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:03.046 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:03.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:03.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.135128+0000", 2021-03-15T22:25:03.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:03.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:03.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.135128+0000", 2021-03-15T22:25:03.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054741980000000004 2021-03-15T22:25:03.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:03.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.140602+0000", 2021-03-15T22:25:03.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6640000000000001e-06 2021-03-15T22:25:03.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:03.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.140609+0000", 2021-03-15T22:25:03.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2505999999999997e-05 2021-03-15T22:25:03.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:03.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.140641+0000", 2021-03-15T22:25:03.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1763000000000001e-05 2021-03-15T22:25:03.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:03.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.140653+0000", 2021-03-15T22:25:03.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9007999999999999e-05 2021-03-15T22:25:03.052 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.140742+0000", 2021-03-15T22:25:03.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4705000000000001e-05 2021-03-15T22:25:03.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:03.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.140777+0000", 2021-03-15T22:25:03.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015769135 2021-03-15T22:25:03.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:03.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156546+0000", 2021-03-15T22:25:03.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2877000000000003e-05 2021-03-15T22:25:03.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:03.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156589+0000", 2021-03-15T22:25:03.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.021460856 2021-03-15T22:25:03.056 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.057 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:03.057 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.057 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3272 7.b 7:d4368cae:::smithi07694878-18:head [copy-from ver 164] snapc fd=[fd,fc,fb,fa,f9,f7,f5,f2,e9,e7,de] ondisk+write+known_if_redirected e354)", 2021-03-15T22:25:03.058 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:29.086546+0000", 2021-03-15T22:25:03.058 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.718933888999999, 2021-03-15T22:25:03.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018540847999999999, 2021-03-15T22:25:03.058 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:03.059 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:03.059 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:03.059 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:03.059 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:03.059 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3272 2021-03-15T22:25:03.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.060 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:03.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:03.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086546+0000", 2021-03-15T22:25:03.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:03.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:03.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086546+0000", 2021-03-15T22:25:03.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5191000000000002e-05 2021-03-15T22:25:03.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:03.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086591+0000", 2021-03-15T22:25:03.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3380000000000001e-06 2021-03-15T22:25:03.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:03.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086595+0000", 2021-03-15T22:25:03.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5608e-05 2021-03-15T22:25:03.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.064 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:03.065 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086650+0000", 2021-03-15T22:25:03.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1369999999999996e-06 2021-03-15T22:25:03.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:03.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086657+0000", 2021-03-15T22:25:03.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1160000000000002e-05 2021-03-15T22:25:03.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086739+0000", 2021-03-15T22:25:03.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019102400000000001 2021-03-15T22:25:03.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:03.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086930+0000", 2021-03-15T22:25:03.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096316830000000003 2021-03-15T22:25:03.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:25:03.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.096561+0000", 2021-03-15T22:25:03.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066537460000000003 2021-03-15T22:25:03.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:03.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.103215+0000", 2021-03-15T22:25:03.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0016567590000000001 2021-03-15T22:25:03.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:03.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.104872+0000", 2021-03-15T22:25:03.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118806 2021-03-15T22:25:03.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:03.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.104991+0000", 2021-03-15T22:25:03.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2608999999999998e-05 2021-03-15T22:25:03.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:03.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.105033+0000", 2021-03-15T22:25:03.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3786999999999998e-05 2021-03-15T22:25:03.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:03.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.105087+0000", 2021-03-15T22:25:03.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018540847999999999 2021-03-15T22:25:03.075 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.075 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:03.075 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.076 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3273 7.b 7:d4368cae:::smithi07694878-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e354)", 2021-03-15T22:25:03.076 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:29.086672+0000", 2021-03-15T22:25:03.076 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 33.718808504999998, 2021-03-15T22:25:03.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018569492, 2021-03-15T22:25:03.076 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:03.077 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:03.077 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:03.077 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:03.077 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:03.078 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3273 2021-03-15T22:25:03.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.078 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:03.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:03.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086672+0000", 2021-03-15T22:25:03.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:03.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:03.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086672+0000", 2021-03-15T22:25:03.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.725e-06 2021-03-15T22:25:03.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:03.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086673+0000", 2021-03-15T22:25:03.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.979e-06 2021-03-15T22:25:03.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.082 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:03.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086675+0000", 2021-03-15T22:25:03.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7249999999999997e-06 2021-03-15T22:25:03.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:03.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086683+0000", 2021-03-15T22:25:03.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6192e-05 2021-03-15T22:25:03.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:03.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.086699+0000", 2021-03-15T22:25:03.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00052117299999999999 2021-03-15T22:25:03.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.087220+0000", 2021-03-15T22:25:03.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2669999999999997e-05 2021-03-15T22:25:03.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:25:03.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.087253+0000", 2021-03-15T22:25:03.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013649529000000001 2021-03-15T22:25:03.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.100903+0000", 2021-03-15T22:25:03.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4626e-05 2021-03-15T22:25:03.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:03.088 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.100977+0000", 2021-03-15T22:25:03.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041431460000000003 2021-03-15T22:25:03.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.105120+0000", 2021-03-15T22:25:03.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4685999999999998e-05 2021-03-15T22:25:03.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:03.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.105175+0000", 2021-03-15T22:25:03.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6041e-05 2021-03-15T22:25:03.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:03.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:29.105241+0000", 2021-03-15T22:25:03.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018569492 2021-03-15T22:25:03.092 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.092 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:03.092 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.093 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3609 7.b 7:d10ba14d:::smithi07694878-43:head [copy-from ver 752] snapc 116=[116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e380)", 2021-03-15T22:25:03.093 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:55.463956+0000", 2021-03-15T22:25:03.093 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.3415240319999997, 2021-03-15T22:25:03.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037021326, 2021-03-15T22:25:03.094 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:03.094 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:03.094 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:03.094 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:03.095 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:03.095 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3609 2021-03-15T22:25:03.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.095 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:03.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:03.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463956+0000", 2021-03-15T22:25:03.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:03.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:03.097 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463956+0000", 2021-03-15T22:25:03.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.636e-06 2021-03-15T22:25:03.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.098 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:03.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463960+0000", 2021-03-15T22:25:03.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7860000000000001e-06 2021-03-15T22:25:03.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:03.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463962+0000", 2021-03-15T22:25:03.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2736e-05 2021-03-15T22:25:03.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:03.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463975+0000", 2021-03-15T22:25:03.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.852e-06 2021-03-15T22:25:03.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:03.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463981+0000", 2021-03-15T22:25:03.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0056000000000002e-05 2021-03-15T22:25:03.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.464041+0000", 2021-03-15T22:25:03.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102267 2021-03-15T22:25:03.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:03.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.464143+0000", 2021-03-15T22:25:03.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027411048 2021-03-15T22:25:03.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.104 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:25:03.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.491554+0000", 2021-03-15T22:25:03.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072668170000000001 2021-03-15T22:25:03.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:03.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.498821+0000", 2021-03-15T22:25:03.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002009434 2021-03-15T22:25:03.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:03.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.500831+0000", 2021-03-15T22:25:03.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9524000000000003e-05 2021-03-15T22:25:03.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:03.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.500870+0000", 2021-03-15T22:25:03.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9422000000000002e-05 2021-03-15T22:25:03.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:03.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.500920+0000", 2021-03-15T22:25:03.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7747999999999997e-05 2021-03-15T22:25:03.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:03.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.500977+0000", 2021-03-15T22:25:03.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037021326 2021-03-15T22:25:03.110 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.110 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:03.110 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.111 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3610 7.b 7:d10ba14d:::smithi07694878-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e380)", 2021-03-15T22:25:03.111 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:55.463991+0000", 2021-03-15T22:25:03.111 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 7.3414889409999997, 2021-03-15T22:25:03.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037113361999999997, 2021-03-15T22:25:03.112 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:03.112 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:03.112 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:03.112 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:03.113 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:03.113 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3610 2021-03-15T22:25:03.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.113 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:03.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:03.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463991+0000", 2021-03-15T22:25:03.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:03.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:03.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463991+0000", 2021-03-15T22:25:03.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3600000000000002e-07 2021-03-15T22:25:03.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:03.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463992+0000", 2021-03-15T22:25:03.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.034e-06 2021-03-15T22:25:03.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:03.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463993+0000", 2021-03-15T22:25:03.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1729999999999997e-06 2021-03-15T22:25:03.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:03.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463997+0000", 2021-03-15T22:25:03.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7920000000000001e-06 2021-03-15T22:25:03.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:03.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.464000+0000", 2021-03-15T22:25:03.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030387099999999999 2021-03-15T22:25:03.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.464304+0000", 2021-03-15T22:25:03.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4651999999999998e-05 2021-03-15T22:25:03.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:25:03.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.464358+0000", 2021-03-15T22:25:03.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.032976501999999998 2021-03-15T22:25:03.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.122 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.122 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.497335+0000", 2021-03-15T22:25:03.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0748000000000006e-05 2021-03-15T22:25:03.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:03.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.497426+0000", 2021-03-15T22:25:03.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035780479999999999 2021-03-15T22:25:03.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.501004+0000", 2021-03-15T22:25:03.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3093999999999998e-05 2021-03-15T22:25:03.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:03.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.501047+0000", 2021-03-15T22:25:03.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7612000000000003e-05 2021-03-15T22:25:03.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:03.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.501104+0000", 2021-03-15T22:25:03.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037113361999999997 2021-03-15T22:25:03.127 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.127 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:03.127 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.128 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:3658 7.0 e385/337 7:0152dc37:::smithi07694878-28:head v 385'842, mlcod=385'839)", 2021-03-15T22:25:03.128 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.122763+0000", 2021-03-15T22:25:03.128 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.6827171829999998, 2021-03-15T22:25:03.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017052660000000001, 2021-03-15T22:25:03.129 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:03.129 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:03.129 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:03.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:03.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.122763+0000", 2021-03-15T22:25:03.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:03.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:03.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.122763+0000", 2021-03-15T22:25:03.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0068164769999999996 2021-03-15T22:25:03.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:03.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.129579+0000", 2021-03-15T22:25:03.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1809999999999998e-06 2021-03-15T22:25:03.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:03.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.129586+0000", 2021-03-15T22:25:03.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8479e-05 2021-03-15T22:25:03.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:03.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.129604+0000", 2021-03-15T22:25:03.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2940000000000002e-06 2021-03-15T22:25:03.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:03.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.129610+0000", 2021-03-15T22:25:03.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4914999999999999e-05 2021-03-15T22:25:03.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.129655+0000", 2021-03-15T22:25:03.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3839999999999999e-05 2021-03-15T22:25:03.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.129679+0000", 2021-03-15T22:25:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010090909 2021-03-15T22:25:03.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:03.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.139770+0000", 2021-03-15T22:25:03.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5565000000000002e-05 2021-03-15T22:25:03.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:03.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.139816+0000", 2021-03-15T22:25:03.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017052660000000001 2021-03-15T22:25:03.139 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.140 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:03.140 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.140 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:3668 7.4 e385/358 7:26337d05:::smithi07694878-20:head v 385'309, mlcod=377'306)", 2021-03-15T22:25:03.141 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:25:03.141 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 2.6270923590000002, 2021-03-15T22:25:03.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054245017999999999, 2021-03-15T22:25:03.141 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:03.141 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:03.142 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:03.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:03.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:25:03.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:03.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.143 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:03.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:25:03.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00068176800000000002 2021-03-15T22:25:03.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:03.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179069+0000", 2021-03-15T22:25:03.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1430000000000001e-06 2021-03-15T22:25:03.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:03.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179077+0000", 2021-03-15T22:25:03.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9515000000000001e-05 2021-03-15T22:25:03.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:03.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179106+0000", 2021-03-15T22:25:03.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0193e-05 2021-03-15T22:25:03.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179116+0000", 2021-03-15T22:25:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8248000000000003e-05 2021-03-15T22:25:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:03.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179165+0000", 2021-03-15T22:25:03.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8050000000000001e-05 2021-03-15T22:25:03.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:03.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179193+0000", 2021-03-15T22:25:03.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053391015999999999 2021-03-15T22:25:03.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:03.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.232584+0000", 2021-03-15T22:25:03.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9085000000000002e-05 2021-03-15T22:25:03.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:03.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:03.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:03.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.232633+0000", 2021-03-15T22:25:03.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054245017999999999 2021-03-15T22:25:03.152 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.152 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:03.152 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.153 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:03.153 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:03.153 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:25:03.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:03.155 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:03.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:03.173 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:25:03.190 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:25:03.207 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:25:03.256 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:03.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:03.285 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:03.286 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:03.326 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:25:03.327 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:25:03.327 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:25:03.327 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:25:03.328 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:25:03.328 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:25:03.328 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:25:03.328 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:25:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:25:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:25:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:25:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:25:03.330 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:25:03.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:03.397 INFO:tasks.rados.rados.0.smithi076.stdout:1418: done (4 left) 2021-03-15T22:25:03.398 INFO:tasks.rados.rados.0.smithi076.stdout:1419: copy_from oid 16 from oid 24 current snap is 151 2021-03-15T22:25:03.398 INFO:tasks.rados.rados.0.smithi076.stdout:1414: finishing write tid 2 to smithi07694878-2 2021-03-15T22:25:03.398 INFO:tasks.rados.rados.0.smithi076.stdout:1414: finishing write tid 3 to smithi07694878-2 2021-03-15T22:25:03.398 INFO:tasks.rados.rados.0.smithi076.stdout:1414: finishing write tid 4 to smithi07694878-2 2021-03-15T22:25:03.399 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 322 (ObjNum 505 snap 151 seq_num 505) dirty exists 2021-03-15T22:25:03.399 INFO:tasks.rados.rados.0.smithi076.stdout:1414: left oid 2 (ObjNum 505 snap 151 seq_num 505) 2021-03-15T22:25:03.399 INFO:tasks.rados.rados.0.smithi076.stdout:1416: finishing write tid 1 to smithi07694878-17 2021-03-15T22:25:03.399 INFO:tasks.rados.rados.0.smithi076.stdout:1416: finishing write tid 2 to smithi07694878-17 2021-03-15T22:25:03.399 INFO:tasks.rados.rados.0.smithi076.stdout:1416: finishing write tid 3 to smithi07694878-17 2021-03-15T22:25:03.400 INFO:tasks.rados.rados.0.smithi076.stdout:1416: finishing write tid 4 to smithi07694878-17 2021-03-15T22:25:03.400 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 307 (ObjNum 506 snap 151 seq_num 506) dirty exists 2021-03-15T22:25:03.400 INFO:tasks.rados.rados.0.smithi076.stdout:1416: left oid 17 (ObjNum 506 snap 151 seq_num 506) 2021-03-15T22:25:03.400 INFO:tasks.rados.rados.0.smithi076.stdout:1417: expect (ObjNum 469 snap 137 seq_num 469) 2021-03-15T22:25:03.421 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:03.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:03.423 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:03.524 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:03.547 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:03.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:03.549 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:03.581 INFO:tasks.rados.rados.0.smithi076.stdout:1419: finishing copy_from to smithi07694878-16 2021-03-15T22:25:03.581 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 97 (ObjNum 464 snap 135 seq_num 464) dirty exists 2021-03-15T22:25:03.581 INFO:tasks.rados.rados.0.smithi076.stdout:1419: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:25:03.582 INFO:tasks.rados.rados.0.smithi076.stdout:1414: done (4 left) 2021-03-15T22:25:03.582 INFO:tasks.rados.rados.0.smithi076.stdout:1415: done (3 left) 2021-03-15T22:25:03.582 INFO:tasks.rados.rados.0.smithi076.stdout:1416: done (2 left) 2021-03-15T22:25:03.582 INFO:tasks.rados.rados.0.smithi076.stdout:1417: done (1 left) 2021-03-15T22:25:03.583 INFO:tasks.rados.rados.0.smithi076.stdout:1419: done (0 left) 2021-03-15T22:25:03.583 INFO:tasks.rados.rados.0.smithi076.stdout:1420: delete oid 26 current snap is 151 2021-03-15T22:25:03.584 INFO:tasks.rados.rados.0.smithi076.stdout:1420: done (0 left) 2021-03-15T22:25:03.585 INFO:tasks.rados.rados.0.smithi076.stdout:1421: rollback oid 35 current snap is 151 2021-03-15T22:25:03.585 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 35 to 141 2021-03-15T22:25:03.585 INFO:tasks.rados.rados.0.smithi076.stdout:1422: snap_remove snap 119 2021-03-15T22:25:03.651 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:03.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:03.676 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:03.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:03.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:25:03.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[56514]: cluster 2021-03-15T22:25:02.391102 2021-03-15T22:25:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2021-03-15T22:25:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[56514]: cluster 2021-03-15T22:25:02.452231+0000 osd.6 (osd.6) 61 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:25:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[56514]: cluster 2021-03-15T22:25:02.520580+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:25:03.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[56514]: audit 2021-03-15T22:25:03.153805+0000 mon.a (mon.0) 1178 : audit [DBG] from='client.? 172.21.15.76:0/151822881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:03.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[56514]: cluster 2021-03-15T22:25:03.393118+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2021-03-15T22:25:03.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[62156]: cluster 2021-03-15T22:25:03.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[62156]: 2021-03-15T22:25:02.391102+0000 mon.a (mon.0) 1177 : cluster [DBG] osdmap e387: 8 total, 8 up, 7 in 2021-03-15T22:25:03.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[62156]: cluster 2021-03-15T22:25:02.452231+0000 osd.6 (osd.6) 61 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:25:03.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[62156]: cluster 2021-03-15T22:25:02.520580+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:25:03.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[62156]: audit 2021-03-15T22:25:03.153805+0000 mon.a (mon.0) 1178 : audit [DBG] from='client.? 172.21.15.76:0/151822881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:03.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:03 smithi076 conmon[62156]: cluster 2021-03-15T22:25:03.393118+0000 mon.a (mon.0) 1179 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2021-03-15T22:25:03.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:03.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:03 smithi125 conmon[56739]: cluster 2021-03-15T22:25:02.391102+0000 mon.a (mon.0) 1177 : cluster 2021-03-15T22:25:03.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:03 smithi125 conmon[56739]: [DBG] osdmap e387: 8 total, 8 up, 7 in 2021-03-15T22:25:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:03 smithi125 conmon[56739]: cluster 2021-03-15T22:25:02.452231+0000 osd.6 (osd.6) 61 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:25:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:03 smithi125 conmon[56739]: cluster 2021-03-15T22:25:02. 2021-03-15T22:25:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:03 smithi125 conmon[56739]: 520580+0000 osd.2 (osd.2) 66 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:25:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:03 smithi125 conmon[56739]: audit 2021-03-15T22:25 2021-03-15T22:25:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:03 smithi125 conmon[56739]: :03.153805+0000 mon.a (mon.0) 1178 : audit [DBG] from='client.? 172.21.15.76:0/151822881' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:03.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:03 smithi125 conmon[56739]: cluster 2021-03-15T22:25:03.393118+0000 mon.a (mon. 2021-03-15T22:25:03.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:03 smithi125 conmon[56739]: 0) 1179 : cluster [DBG] osdmap e388: 8 total, 8 up, 7 in 2021-03-15T22:25:03.801 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:03.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:03.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:03.903 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:03.928 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:03.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:03.930 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:04.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:04.056 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:04.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:04.057 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:04.158 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:04.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:04.182 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:04.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:04.284 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:04.293 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-15T22:25:04.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:04.308 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:04.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:04.415 INFO:tasks.rados.rados.0.smithi076.stdout:1422: done (1 left) 2021-03-15T22:25:04.416 INFO:tasks.rados.rados.0.smithi076.stdout:1423: read oid 28 snap -1 2021-03-15T22:25:04.416 INFO:tasks.rados.rados.0.smithi076.stdout:1423: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:25:04.417 INFO:tasks.rados.rados.0.smithi076.stdout:1424: delete oid 8 current snap is 151 2021-03-15T22:25:04.417 INFO:tasks.rados.rados.0.smithi076.stdout:1421: finishing rollback tid 1 to smithi07694878-35 2021-03-15T22:25:04.418 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 492 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:25:04.419 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:04.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: cluster 2021-03-15T22:25:04.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: 2021-03-15T22:25:04.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: 2021 2021-03-15T22:25:04.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: - 2021-03-15T22:25:04.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: 03 2021-03-15T22:25:04.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: - 2021-03-15T22:25:04.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: 15 2021-03-15T22:25:04.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: T22:25:02.791105+0000 osd.3 2021-03-15T22:25:04.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: (osd.3) 69 : cluster [DBG] 3.b scrub ok 2021-03-15T22:25:04.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: cluster 2021-03-15T22:25:04.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:25:04.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: 15T22:25:02.868394+0000 2021-03-15T22:25:04.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: osd.1 (osd.1) 76 : cluster 2021-03-15T22:25:04.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: [DBG] 3.6 scrub ok 2021-03-15T22:25:04.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:25:04.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: -03-15T22:25:02.989306+0000 2021-03-15T22:25:04.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: osd.4 (osd.4) 85 : cluster [DBG] 2021-03-15T22:25:04.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: 5.1 scrub ok 2021-03-15T22:25:04.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:25:04.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: 15T22:25:03.047272+0000 2021-03-15T22:25:04.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: mgr.y (mgr.14144) 601 : cluster [DBG] 2021-03-15T22:25:04.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[62156]: pgmap v708: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 289 MiB data, 861 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 24 op/s; 1/1380 objects misplaced (0.072%) 2021-03-15T22:25:04.432 INFO:tasks.rados.rados.0.smithi076.stdout:1421: done (2 left) 2021-03-15T22:25:04.433 INFO:tasks.rados.rados.0.smithi076.stdout:1424: done (1 left) 2021-03-15T22:25:04.433 INFO:tasks.rados.rados.0.smithi076.stdout:1425: rollback oid 45 current snap is 151 2021-03-15T22:25:04.433 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 45 to 141 2021-03-15T22:25:04.433 INFO:tasks.rados.rados.0.smithi076.stdout:1426: copy_from oid 16 from oid 27 current snap is 151 2021-03-15T22:25:04.434 INFO:tasks.rados.rados.0.smithi076.stdout:1427: snap_remove snap 112 2021-03-15T22:25:04.434 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:04.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:04.453 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:04.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:04.554 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:04.580 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:04.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:04.582 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:04.683 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:04.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:04.706 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:04.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:04.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[56514]: cluster 2021-03-15T22:25:04.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[56514]: 2021-03-15T22:25:02.791105+0000 osd.3 (osd.3) 69 : cluster [DBG] 3.b scrub ok 2021-03-15T22:25:04.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[56514]: cluster 2021-03-15T22:25:02.868394+0000 osd.1 (osd.1) 76 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:25:04.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[56514]: cluster 2021-03-15T22:25:02.989306+0000 osd.4 (osd.4) 85 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:25:04.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:04 smithi076 conmon[56514]: cluster 2021-03-15T22:25:03.047272+0000 mgr.y (mgr.14144) 601 : cluster [DBG] pgmap v708: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 289 MiB data, 861 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 24 op/s; 1/1380 objects misplaced (0.072%) 2021-03-15T22:25:04.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: cluster 2021-03-15T22:25:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: 2021-03-15 2021-03-15T22:25:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: T22:25:02.791105+0000 osd.3 (osd.3) 69 : cluster [DBG] 3.b scrub ok 2021-03-15T22:25:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: cluster 2021-03-15T22:25:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: 2021-03-15T 2021-03-15T22:25:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: 22:25:02.868394+0000 osd.1 (osd 2021-03-15T22:25:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: .1) 76 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:25:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: cluster 2021-03-15T22:25:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: 2021-03-15T22:25:04.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:25:04.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: 25:02.989306+0000 osd.4 (osd.4) 85 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:25:04.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: cluster 2021-03-15T22:25:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: 2021-03-15T22:25:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: 2021 2021-03-15T22:25:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: - 2021-03-15T22:25:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: 03 2021-03-15T22:25:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: - 2021-03-15T22:25:04.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: 15 2021-03-15T22:25:04.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: T22:25:03.047272 2021-03-15T22:25:04.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:04 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 601 : cluster [DBG] pgmap v708: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 289 MiB data, 861 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.5 MiB/s wr, 24 op/s; 1/1380 objects misplaced (0.072%) 2021-03-15T22:25:04.808 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:04.833 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:04.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:04.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:04.936 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:04.961 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:04.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:04.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:05.063 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:05.069 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:05.090 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:05.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:05.091 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:05.192 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:05.217 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:05.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:05.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:05.319 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:05.343 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:05.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:05.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:05.426 INFO:tasks.rados.rados.0.smithi076.stdout:1427: done (3 left) 2021-03-15T22:25:05.427 INFO:tasks.rados.rados.0.smithi076.stdout:1428: read oid 22 snap -1 2021-03-15T22:25:05.427 INFO:tasks.rados.rados.0.smithi076.stdout:1428: expect (ObjNum 466 snap 136 seq_num 466) 2021-03-15T22:25:05.427 INFO:tasks.rados.rados.0.smithi076.stdout:1429: snap_create 2021-03-15T22:25:05.431 INFO:tasks.rados.rados.0.smithi076.stdout:1425: finishing rollback tid 1 to smithi07694878-45 2021-03-15T22:25:05.431 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 270 (ObjNum 261982224 snap 21896 seq_num 261671536) dirty dne 2021-03-15T22:25:05.432 INFO:tasks.rados.rados.0.smithi076.stdout:1423: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:25:05.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:05.469 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:05.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:05.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:05.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:05.580 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[56514]: cluster 2021-03-15T22:25:03.853073+0000 2021-03-15T22:25:05.581 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[56514]: osd.1 (osd.1) 77 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:25:05.581 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[56514]: cluster 2021-03-15T22:25:04.028776+0000 osd.4 (osd.4) 86 : cluster [DBG] 4.d scrub ok 2021-03-15T22:25:05.581 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[56514]: cluster 2021-03-15T22:25:04.412799+0000 mon.a (mon.0) 1180 : cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2021-03-15T22:25:05.582 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[75764]: debug 2021-03-15T22:25:05.422+0000 7f9fbefb7700 -1 osd.3 pg_epoch: 390 pg[7.2( v 389'219 (0'0,389'219] local-lis/les=264/265 n=6 ec=91/91 lis/c=264/264 les/c/f=265/265/0 sis=264) [3,1,2] r=0 lpr=264 crt=389'218 lcod 389'218 mlcod 389'218 active+clean+snaptrim trimq=[de~1] ps=[e9~1,111~1,119~1,11b~1]] removing snap head 2021-03-15T22:25:05.582 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: cluster 2021-03-15T22:25:05.582 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: 2021-03-15T22:25:05.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: 2021-03-15T22:25:03. 2021-03-15T22:25:05.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: 853073+0000 osd.1 (osd.1) 77 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:25:05.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: cluster 2021-03-15T22:25:05.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: 2021-03-15T22:25: 2021-03-15T22:25:05.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: 04.028776+0000 osd.4 (osd.4) 86 2021-03-15T22:25:05.584 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: : cluster [DBG] 4.d scrub ok 2021-03-15T22:25:05.584 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: cluster 2021-03-15T22:25:05.584 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: 2021-03-15T22:25:04. 2021-03-15T22:25:05.584 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: 412799+0000 mon.a (mon.0) 1180 : cluster 2021-03-15T22:25:05.585 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:05 smithi076 conmon[62156]: [DBG] osdmap e389: 8 total, 8 up, 7 in 2021-03-15T22:25:05.595 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:05.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:05.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:05.601 INFO:tasks.rados.rados.0.smithi076.stdout:1426: finishing copy_from to smithi07694878-16 2021-03-15T22:25:05.601 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 294 (ObjNum 416 snap 112 seq_num 416) dirty exists 2021-03-15T22:25:05.601 INFO:tasks.rados.rados.0.smithi076.stdout:1426: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:25:05.602 INFO:tasks.rados.rados.0.smithi076.stdout:1428: expect (ObjNum 466 snap 136 seq_num 466) 2021-03-15T22:25:05.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:05.722 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:05.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:05.724 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:05.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:05 smithi125 conmon[56739]: cluster 2021-03-15T22:25:03.853073+0000 osd.1 (osd 2021-03-15T22:25:05.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:05 smithi125 conmon[56739]: .1) 77 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:25:05.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:05 smithi125 conmon[56739]: cluster 2021-03-15T22:25:04.028776+0000 osd.4 (osd 2021-03-15T22:25:05.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:05 smithi125 conmon[56739]: .4) 86 : cluster [DBG] 4.d scrub ok 2021-03-15T22:25:05.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:05 smithi125 conmon[56739]: cluster 2021-03-15T22:25: 2021-03-15T22:25:05.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:05 smithi125 conmon[56739]: 04.412799+0000 mon.a (mon.0) 1180 : 2021-03-15T22:25:05.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:05 smithi125 conmon[56739]: cluster [DBG] osdmap e389: 8 total, 8 up, 7 in 2021-03-15T22:25:05.825 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:05.848 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:05.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:05.850 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:05.951 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:05.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:05.977 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:05.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:06.078 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:06.103 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:06.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:06.105 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:06.206 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:06.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:06.235 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:06.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:06.328 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:25:06.337 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:06.362 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:06.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:06.364 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:06.430 INFO:tasks.rados.rados.0.smithi076.stdout:1423: done (4 left) 2021-03-15T22:25:06.431 INFO:tasks.rados.rados.0.smithi076.stdout:1425: done (3 left) 2021-03-15T22:25:06.431 INFO:tasks.rados.rados.0.smithi076.stdout:1426: done (2 left) 2021-03-15T22:25:06.431 INFO:tasks.rados.rados.0.smithi076.stdout:1428: done (1 left) 2021-03-15T22:25:06.431 INFO:tasks.rados.rados.0.smithi076.stdout:1429: done (0 left) 2021-03-15T22:25:06.432 INFO:tasks.rados.rados.0.smithi076.stdout:1430: write oid 41 current snap is 152 2021-03-15T22:25:06.432 INFO:tasks.rados.rados.0.smithi076.stdout:1430: seq_num 509 ranges {554763=781039,2102511=452252} 2021-03-15T22:25:06.433 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.4 to 1 (10000) 2021-03-15T22:25:06.437 INFO:tasks.rados.rados.0.smithi076.stdout:1430: writing smithi07694878-41 from 554763 to 1335802 tid 1 2021-03-15T22:25:06.444 INFO:tasks.rados.rados.0.smithi076.stdout:1430: writing smithi07694878-41 from 2102511 to 2554763 tid 2 2021-03-15T22:25:06.445 INFO:tasks.rados.rados.0.smithi076.stdout:1431: snap_create 2021-03-15T22:25:06.445 INFO:tasks.rados.rados.0.smithi076.stdout:1430: finishing write tid 1 to smithi07694878-41 2021-03-15T22:25:06.447 INFO:tasks.rados.rados.0.smithi076.stdout:1430: finishing write tid 2 to smithi07694878-41 2021-03-15T22:25:06.447 INFO:tasks.rados.rados.0.smithi076.stdout:1430: finishing write tid 3 to smithi07694878-41 2021-03-15T22:25:06.448 INFO:tasks.rados.rados.0.smithi076.stdout:1430: finishing write tid 4 to smithi07694878-41 2021-03-15T22:25:06.448 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 762 (ObjNum 509 snap 152 seq_num 509) dirty exists 2021-03-15T22:25:06.449 INFO:tasks.rados.rados.0.smithi076.stdout:1430: left oid 41 (ObjNum 509 snap 152 seq_num 509) 2021-03-15T22:25:06.464 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:06.488 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:06.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:06.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:06.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[56514]: cluster 2021-03-15T22:25:05.028077+0000 osd.4 (osd.4) 87 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:25:06.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[56514]: cluster 2021-03-15T22:25:05.047815+0000 mgr.y (mgr.14144) 602 : cluster [DBG] pgmap v711: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 293 MiB data, 877 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 3.6 MiB/s wr, 18 op/s; 1/1389 objects misplaced (0.072%) 2021-03-15T22:25:06.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[56514]: cluster 2021-03-15T22:25:05.424511+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2021-03-15T22:25:06.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[56514]: cluster 2021-03-15T22:25:05.575543+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.b scrub ok 2021-03-15T22:25:06.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[56514]: cluster 2021-03-15T22:25:05.682019+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:25:06.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[56514]: audit 2021-03-15T22:25:05 2021-03-15T22:25:06.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[56514]: .978924+0000 mon.c (mon.1) 34 : audit [INF] from='client.? 172.21.15.76:0/227983790' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:25:06.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[56514]: audit 2021-03-15T22:25:05.979198+0000 mon.a (mon.0) 1182 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:25:06.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[62156]: cluster 2021-03-15T22:25:05.028077+0000 osd.4 (osd.4) 87 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:25:06.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[62156]: cluster 2021-03-15T22:25:05.047815+0000 mgr.y (mgr.14144) 602 : cluster [DBG] pgmap v711: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 293 MiB data, 877 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 3.6 MiB/s wr, 18 op/s; 1/1389 objects misplaced (0.072%) 2021-03-15T22:25:06.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[62156]: cluster 2021-03-15T22:25:05.424511+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2021-03-15T22:25:06.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[62156]: cluster 2021-03-15T22:25:05.575543+0000 osd.2 (osd.2) 67 : cluster [DBG] 5.b scrub ok 2021-03-15T22:25:06.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[62156]: cluster 2021-03-15T22:25: 2021-03-15T22:25:06.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[62156]: 05.682019+0000 osd.5 (osd.5) 70 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:25:06.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[62156]: audit 2021-03-15T22:25:05.978924+0000 mon.c (mon.1) 34 : audit [INF] from='client.? 172.21.15.76:0/227983790' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:25:06.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:06 smithi076 conmon[62156]: audit 2021-03-15T22:25:05.979198+0000 mon.a (mon.0) 1182 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:25:06.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:06.614 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:06.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:06.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:06.717 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:06.740 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:06.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:06.742 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:06.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: cluster 2021-03-15T22:25:05.028077+0000 osd.4 (osd.4) 87 : 2021-03-15T22:25:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: cluster [DBG] 3.1 scrub ok 2021-03-15T22:25:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: cluster 2021-03-15T22:25:05.047815+0000 mgr.y (mgr.14144) 602 : cluster [DBG] pgmap v711: 138 pgs: 1 active+remapped+backfill_toofull, 137 active+clean; 293 MiB data, 877 MiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 3.6 MiB/s wr, 18 op/s; 1/1389 objects misplaced (0.072%) 2021-03-15T22:25:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: cluster 2021-03-15T22:25:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: 2021-03-15T22:25:05.424511+0000 mon.a (mon.0) 1181 : cluster [DBG] osdmap e390: 8 total, 8 up, 7 in 2021-03-15T22:25:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:25:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: -03-15T22:25:05.575543+0000 2021-03-15T22:25:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: osd.2 (osd.2) 67 : cluster [DBG] 2021-03-15T22:25:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: 5.b scrub ok 2021-03-15T22:25:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: cluster 2021-03-15T22:25 2021-03-15T22:25:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: :05.682019+0000 osd.5 (osd.5) 70 2021-03-15T22:25:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: : cluster [DBG] 3.1c scrub ok 2021-03-15T22:25:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:25:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: 15T22:25:05.978924+0000 mon.c (mon.1) 2021-03-15T22:25:06.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: 34 : audit [INF] from='client.? 172.21.15.76:0/227983790' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:25:06.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: audit 2021-03-15T22:25:06.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: 2021-03-15T22:25:05.979198 2021-03-15T22:25:06.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1182 : audit [INF] 2021-03-15T22:25:06.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:06 smithi125 conmon[56739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:25:06.843 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:06.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:06.868 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:06.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:06.970 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:06.993 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:06.994 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:07.095 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:07.121 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:07.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:07.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:07.224 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:07.248 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:07.250 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:07.351 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:07.375 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:07.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:07.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:07.449 INFO:tasks.rados.rados.0.smithi076.stdout:1430: done (1 left) 2021-03-15T22:25:07.450 INFO:tasks.rados.rados.0.smithi076.stdout:1431: done (0 left) 2021-03-15T22:25:07.450 INFO:tasks.rados.rados.0.smithi076.stdout:1432: read oid 23 snap -1 2021-03-15T22:25:07.450 INFO:tasks.rados.rados.0.smithi076.stdout:1432: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:25:07.451 INFO:tasks.rados.rados.0.smithi076.stdout:1433: read oid 8 snap -1 2021-03-15T22:25:07.451 INFO:tasks.rados.rados.0.smithi076.stdout:1433: expect deleted 2021-03-15T22:25:07.451 INFO:tasks.rados.rados.0.smithi076.stdout:1434: snap_create 2021-03-15T22:25:07.453 INFO:tasks.rados.rados.0.smithi076.stdout:1432: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:25:07.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:07.503 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:07.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:07.504 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:07.547 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 5 1 2021-03-15T22:25:07.605 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:07.630 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:07.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:07.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:07.730 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[56514]: cluster 2021-03-15T22:25:05.857002+0000 osd.1 (osd.1) 78 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:25:07.731 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[56514]: cluster 2021-03-15T22:25:06.177748+0000 osd.7 (osd.7) 75 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:25:07.731 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[56514]: audit 2021-03-15T22:25:06.426396+0000 mon.a (mon.0) 1183 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-15T22:25:07.731 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[56514]: cluster 2021-03-15T22:25:06.426464+0000 mon.a (mon.0) 1184 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2021-03-15T22:25:07.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: cluster 2021-03-15T22:25:07.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: 2021-03-15T22:25:07.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: 2021 2021-03-15T22:25:07.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: - 2021-03-15T22:25:07.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: 03 2021-03-15T22:25:07.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: - 2021-03-15T22:25:07.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: 15 2021-03-15T22:25:07.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: T 2021-03-15T22:25:07.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: 22 2021-03-15T22:25:07.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: :25:05.857002+0000 osd.1 ( 2021-03-15T22:25:07.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: osd.1) 78 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:25:07.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: cluster 2021-03-15T22:25:07.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: 2021-03-15T22:25:06 2021-03-15T22:25:07.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: .177748+0000 osd.7 (osd.7) 2021-03-15T22:25:07.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: 75 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:25:07.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: audit 2021 2021-03-15T22:25:07.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: -03-15T22:25:06.426396 2021-03-15T22:25:07.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1183 : audit 2021-03-15T22:25:07.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-15T22:25:07.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:25:07.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: 22:25:06.426464+0000 mon.a ( 2021-03-15T22:25:07.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: mon.0) 1184 : cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2021-03-15T22:25:07.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:07 smithi076 conmon[62156]: 2021-03-15T22:25:07.737 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:07.761 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:07.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:07.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:07.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:07 smithi125 conmon[56739]: cluster 2021-03-15T22:25:05.857002+0000 osd.1 (osd 2021-03-15T22:25:07.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:07 smithi125 conmon[56739]: .1) 78 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:25:07.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:07 smithi125 conmon[56739]: cluster 2021-03-15T22:25:06.177748+0000 osd.7 (osd.7) 75 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:25:07.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:07 smithi125 conmon[56739]: audit 2021-03-15T22:25:06.426396+0000 mon.a (mon.0) 1183 : audit 2021-03-15T22:25:07.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:07 smithi125 conmon[56739]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-15T22:25:07.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:07 smithi125 conmon[56739]: cluster 2021-03-15T22:25:06 2021-03-15T22:25:07.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:07 smithi125 conmon[56739]: .426464+0000 mon.a (mon.0) 1184 : 2021-03-15T22:25:07.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:07 smithi125 conmon[56739]: cluster [DBG] osdmap e391: 8 total, 8 up, 7 in 2021-03-15T22:25:07.805 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:07.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:07.890 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:07.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:07.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:07.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:07.994 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:08.019 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:08.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:08.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:08.087 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:25:08.087 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:25:08.122 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:08.148 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:08.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:08.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:08.252 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:08.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:08.275 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:08.276 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:08.377 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:08.399 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:08.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:08.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:08.460 INFO:tasks.rados.rados.0.smithi076.stdout:1432: done (2 left) 2021-03-15T22:25:08.460 INFO:tasks.rados.rados.0.smithi076.stdout:1433: done (1 left) 2021-03-15T22:25:08.461 INFO:tasks.rados.rados.0.smithi076.stdout:1434: done (0 left) 2021-03-15T22:25:08.461 INFO:tasks.rados.rados.0.smithi076.stdout:1435: snap_remove snap 127 2021-03-15T22:25:08.462 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:25:08.503 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:08.526 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:08.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:08.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:08.628 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:08.650 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:08.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:08.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:08.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[56514]: cluster 2021-03-15T22:25:06.854925+0000 osd.1 (osd.1) 79 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:25:08.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[56514]: cluster 2021-03-15T22:25:07.027090+0000 osd.4 (osd.4) 88 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:25:08.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[56514]: cluster 2021-03-15T22:25:07.048499+0000 mgr.y (mgr.14144) 603 : cluster [DBG] pgmap v714: 138 pgs: 138 active+clean; 288 MiB data, 873 MiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:25:08.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[56514]: cluster 2021-03-15T22:25:07.163121+0000 osd.7 (osd.7) 76 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:25:08.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[56514]: cluster 2021-03-15T22:25:07.414184+0000 osd.6 (osd.6) 62 : cluster [DBG] 2021-03-15T22:25:08.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[56514]: 7.c scrub ok 2021-03-15T22:25:08.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[56514]: cluster 2021-03-15T22:25:07.434036+0000 mon.a (mon.0) 1185 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:25:08.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[56514]: cluster 2021-03-15T22:25:07.446795+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2021-03-15T22:25:08.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[56514]: audit 2021-03-15T22:25:08.060204+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? 172.21.15.76:0/3754718588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:08.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[62156]: cluster 2021-03-15T22:25:06.854925+0000 osd.1 (osd.1) 79 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:25:08.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[62156]: cluster 2021-03-15T22:25:07.027090+0000 osd.4 (osd.4) 88 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:25:08.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[62156]: cluster 2021-03-15T22:25:07.048499+0000 mgr.y (mgr.14144) 603 : cluster [DBG] pgmap v714: 138 pgs: 138 active+clean; 288 MiB data, 873 MiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:25:08.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[62156]: cluster 2021-03-15T22:25:07.163121+0000 osd.7 (osd.7) 76 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:25:08.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[62156]: cluster 2021-03-15T22:25:07.414184+0000 osd.6 (osd.6) 62 : cluster [DBG] 7.c scrub ok 2021-03-15T22:25:08.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[62156]: cluster 2021-03-15T22:25:07.434036+0000 mon.a (mon.0) 1185 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:25:08.721 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[62156]: cluster 2021-03-15T22:25:07.446795+0000 mon.a (mon.0) 1186 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2021-03-15T22:25:08.721 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:08 smithi076 conmon[62156]: audit 2021-03-15T22:25:08.060204+0000 mon.a (mon.0) 1187 : audit [INF] from='client.? 172.21.15.76:0/3754718588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:08.752 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:08.778 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:08.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:08.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:08.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: cluster 2021-03-15T22:25:06.854925+0000 osd.1 (osd.1) 79 : cluster [DBG] 2021-03-15T22:25:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: 7.9 scrub ok 2021-03-15T22:25:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: cluster 2021-03-15T22:25:07.027090+0000 osd.4 (osd.4) 88 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:25:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: cluster 2021-03-15T22:25: 2021-03-15T22:25:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: 07.048499+0000 mgr.y (mgr.14144 2021-03-15T22:25:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: ) 603 : cluster [DBG] pgmap v714: 138 pgs: 138 active+clean; 288 MiB data, 873 MiB used, 625 GiB / 626 GiB avail; 1.6 MiB/s rd, 1.6 MiB/s wr, 8 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:25:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:25:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: -15T22:25:07.163121+0000 2021-03-15T22:25:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: osd.7 (osd.7) 76 : cluster 2021-03-15T22:25:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: [DBG] 4.15 scrub ok 2021-03-15T22:25:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:25:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: -03-15T22:25:07 2021-03-15T22:25:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: .414184+0000 osd.6 (osd. 2021-03-15T22:25:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: 6) 62 : cluster [DBG] 2021-03-15T22:25:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: 7.c scrub ok 2021-03-15T22:25:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:25:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: -03-15T22:25:07 2021-03-15T22:25:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: .434036+0000 mon.a (mon. 2021-03-15T22:25:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: 0) 1185 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:25:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: 2021-03-15T22:25:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: 03-15T22:25:07. 2021-03-15T22:25:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: 446795+0000 mon.a (mon.0 2021-03-15T22:25:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: ) 1186 : cluster [DBG] osdmap e392: 8 total, 8 up, 7 in 2021-03-15T22:25:08.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: 2021-03-15T22:25:08.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: audit 2021 2021-03-15T22:25:08.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: -03-15T22:25:08 2021-03-15T22:25:08.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: .060204+0000 mon.a (mon. 2021-03-15T22:25:08.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: 0) 1187 : audit [INF] from='client.? 172.21.15.76:0/3754718588' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:08.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:08 smithi125 conmon[56739]: 2021-03-15T22:25:08.882 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:08.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:08.905 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:08.906 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:09.007 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:09.029 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:09.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:09.031 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:09.132 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:09.155 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:09.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:09.157 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:09.258 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:09.284 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:09.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:09.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:09.387 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:09.412 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:09.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:09.414 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:09.460 INFO:tasks.rados.rados.0.smithi076.stdout:1435: done (0 left) 2021-03-15T22:25:09.460 INFO:tasks.rados.rados.0.smithi076.stdout:1436: read oid 26 snap -1 2021-03-15T22:25:09.461 INFO:tasks.rados.rados.0.smithi076.stdout:1436: expect deleted 2021-03-15T22:25:09.461 INFO:tasks.rados.rados.0.smithi076.stdout:1437: read oid 13 snap -1 2021-03-15T22:25:09.461 INFO:tasks.rados.rados.0.smithi076.stdout:1437: expect deleted 2021-03-15T22:25:09.462 INFO:tasks.rados.rados.0.smithi076.stdout:1438: copy_from oid 17 from oid 46 current snap is 154 2021-03-15T22:25:09.462 INFO:tasks.rados.rados.0.smithi076.stdout:1439: delete oid 14 current snap is 154 2021-03-15T22:25:09.468 INFO:tasks.rados.rados.0.smithi076.stdout:1436: done (3 left) 2021-03-15T22:25:09.469 INFO:tasks.rados.rados.0.smithi076.stdout:1439: done (2 left) 2021-03-15T22:25:09.469 INFO:tasks.rados.rados.0.smithi076.stdout:1440: copy_from oid 48 from oid 11 current snap is 154 2021-03-15T22:25:09.469 INFO:tasks.rados.rados.0.smithi076.stdout:1441: read oid 38 snap -1 2021-03-15T22:25:09.470 INFO:tasks.rados.rados.0.smithi076.stdout:1441: expect deleted 2021-03-15T22:25:09.470 INFO:tasks.rados.rados.0.smithi076.stdout:1442: delete oid 16 current snap is 154 2021-03-15T22:25:09.475 INFO:tasks.rados.rados.0.smithi076.stdout:1437: done (4 left) 2021-03-15T22:25:09.475 INFO:tasks.rados.rados.0.smithi076.stdout:1441: done (3 left) 2021-03-15T22:25:09.476 INFO:tasks.rados.rados.0.smithi076.stdout:1442: done (2 left) 2021-03-15T22:25:09.476 INFO:tasks.rados.rados.0.smithi076.stdout:1443: rollback oid 3 current snap is 154 2021-03-15T22:25:09.476 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 152 2021-03-15T22:25:09.477 INFO:tasks.rados.rados.0.smithi076.stdout:1444: rollback oid 6 current snap is 154 2021-03-15T22:25:09.477 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 146 2021-03-15T22:25:09.478 INFO:tasks.rados.rados.0.smithi076.stdout:1445: snap_remove snap 137 2021-03-15T22:25:09.515 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:09.539 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:09.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:09.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:09.555 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-15T22:25:09.642 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:09.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:09.666 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:09.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:09.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[56514]: cluster 2021-03-15T22:25:07.992811+0000 osd.4 (osd.4) 89 : cluster 2021-03-15T22:25:09.702 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[56514]: [DBG] 6.17 scrub ok 2021-03-15T22:25:09.702 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[56514]: cluster 2021-03-15T22:25:08.192159+0000 osd.7 (osd.7) 77 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:25:09.702 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[56514]: cluster 2021-03-15T22:25:08.443568+0000 mon.a (mon.0) 1188 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:25:09.703 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[56514]: cluster 2021-03-15T22:25:08.443594+0000 mon.a (mon.0) 1189 : cluster [INF] Cluster is now healthy 2021-03-15T22:25:09.703 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[56514]: audit 2021-03-15T22:25:08.457040+0000 mon.a (mon.0) 1190 : audit [INF] from='client.? 172.21.15.76:0/3754718588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:09.703 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[56514]: cluster 2021-03-15T22:25:08.457095+0000 mon.a (mon. 2021-03-15T22:25:09.703 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[56514]: 0) 1191 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2021-03-15T22:25:09.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: cluster 2021-03-15T22:25:07. 2021-03-15T22:25:09.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: 992811+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:25:09.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: cluster 2021-03-15T22:25:09.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: 2021-03-15T22:25:08.192159+0000 osd.7 (osd.7) 77 : cluster 2021-03-15T22:25:09.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: [DBG] 4.1e scrub ok 2021-03-15T22:25:09.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:25:09.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: 03-15T22:25:08.443568+0000 mon.a 2021-03-15T22:25:09.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: (mon.0) 1188 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:25:09.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: 2021-03-15T22:25:09.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: cluster 2021-03-15T22:25: 2021-03-15T22:25:09.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: 08.443594+0000 mon.a (mon.0) 1189 2021-03-15T22:25:09.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: : cluster [INF] Cluster is now healthy 2021-03-15T22:25:09.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: audit 2021-03-15T22:25:09.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: 2021-03-15T22:25:08.457040+0000 2021-03-15T22:25:09.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: mon.a (mon.0) 1190 : audit [INF] 2021-03-15T22:25:09.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3754718588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:09.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:25:09.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: 22:25:08.457095+0000 mon.a (mon. 2021-03-15T22:25:09.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:09 smithi076 conmon[62156]: 0) 1191 : cluster [DBG] osdmap e393: 8 total, 8 up, 7 in 2021-03-15T22:25:09.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:09.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:25:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: cluster 2021-03-15T22:25:07.992811+0000 osd.4 (osd.4) 89 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:25:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: cluster 2021-03-15T22:25:08.192159+0000 osd.7 (osd.7) 77 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:25:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:09.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: 03-15T22:25:08.443568+0000 mon.a (mon 2021-03-15T22:25:09.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: .0) 1188 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:25:09.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: 2021-03-15T22:25:09.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: cluster 2021-03-15T22:25:09.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: 2021-03-15T22:25 2021-03-15T22:25:09.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: :08.443594+0000 mon.a (mon 2021-03-15T22:25:09.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: .0) 1189 : cluster [INF] 2021-03-15T22:25:09.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: Cluster is now healthy 2021-03-15T22:25:09.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: audit 2021 2021-03-15T22:25:09.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: -03-15T22:25:08. 2021-03-15T22:25:09.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: 457040+0000 mon.a (mon.0) 2021-03-15T22:25:09.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: 1190 : audit [INF] from='client.? 172.21.15.76:0/3754718588' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:09.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:09.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: 03-15T22:25:08.457095 2021-03-15T22:25:09.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2021-03-15T22:25:09.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: 1191 : cluster [DBG] 2021-03-15T22:25:09.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:09 smithi125 conmon[56739]: osdmap e393: 8 total, 8 up, 7 in 2021-03-15T22:25:09.794 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:09.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:09.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:09.897 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:09.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:09.921 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:09.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:10.023 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:10.046 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:10.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:10.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:10.095 INFO:tasks.rados.rados.0.smithi076.stdout:1445: done (4 left) 2021-03-15T22:25:10.095 INFO:tasks.rados.rados.0.smithi076.stdout:1446: snap_create 2021-03-15T22:25:10.096 INFO:tasks.rados.rados.0.smithi076.stdout:1444: finishing rollback tid 0 to smithi07694878-6 2021-03-15T22:25:10.096 INFO:tasks.rados.rados.0.smithi076.stdout:1444: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:25:10.096 INFO:tasks.rados.rados.0.smithi076.stdout:1444: finishing rollback tid 2 to smithi07694878-6 2021-03-15T22:25:10.096 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 283 (ObjNum 486 snap 144 seq_num 486) dirty exists 2021-03-15T22:25:10.097 INFO:tasks.rados.rados.0.smithi076.stdout:1443: finishing rollback tid 0 to smithi07694878-3 2021-03-15T22:25:10.097 INFO:tasks.rados.rados.0.smithi076.stdout:1443: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:25:10.097 INFO:tasks.rados.rados.0.smithi076.stdout:1443: finishing rollback tid 2 to smithi07694878-3 2021-03-15T22:25:10.097 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 668 (ObjNum 430 snap 119 seq_num 430) dirty exists 2021-03-15T22:25:10.097 INFO:tasks.rados.rados.0.smithi076.stdout:1438: finishing copy_from to smithi07694878-17 2021-03-15T22:25:10.098 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 203 (ObjNum 469 snap 137 seq_num 469) dirty exists 2021-03-15T22:25:10.098 INFO:tasks.rados.rados.0.smithi076.stdout:1440: finishing copy_from to smithi07694878-48 2021-03-15T22:25:10.098 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 421 (ObjNum 494 snap 146 seq_num 494) dirty exists 2021-03-15T22:25:10.099 INFO:tasks.rados.rados.0.smithi076.stdout:1438: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:25:10.099 INFO:tasks.rados.rados.0.smithi076.stdout:1440: finishing copy_from racing read to smithi07694878-48 2021-03-15T22:25:10.149 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:10.172 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:10.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:10.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:10.275 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:10.297 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:10.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:10.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:10.400 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:10.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:10.424 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:10.425 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:10.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:10.551 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:10.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:10.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:10.654 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:10.679 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:10.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:10.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:10.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[56514]: cluster 2021-03-15T22:25:09.048919+0000 mgr.y (mgr.14144) 604 : cluster [DBG] pgmap v717: 138 pgs: 138 active+clean; 281 MiB data, 866 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 524 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:25:10.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[56514]: cluster 2021-03-15T22:25:09.177516+0000 osd.7 (osd.7) 78 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:25:10.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[56514]: cluster 2021-03-15T22:25:09.456944+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2021-03-15T22:25:10.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[56514]: audit 2021-03-15T22:25:09.687481+0000 mgr.y (mgr.14144) 605 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:10.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[56514]: cluster 2021-03-15T22:25:10.090661+0000 mon.a (mon.0) 1193 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2021-03-15T22:25:10.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: cluster 2021-03-15T22:25:09.048919+0000 mgr.y ( 2021-03-15T22:25:10.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: mgr.14144) 604 : cluster [DBG] pgmap v717: 138 pgs: 138 active+clean; 281 MiB data, 866 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 524 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:25:10.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:25:10.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: T22:25:09.177516+0000 osd.7 (osd.7) 78 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:25:10.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: cluster 2021-03-15T22:25: 2021-03-15T22:25:10.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: 09.456944+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2021-03-15T22:25:10.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:25:10.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: 15T22:25:09.687481+0000 mgr.y (mgr.14144) 2021-03-15T22:25:10.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: 605 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:10.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:25:10.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: 15T22:25:10.090661+0000 mon.a 2021-03-15T22:25:10.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: (mon.0) 1193 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2021-03-15T22:25:10.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:10 smithi076 conmon[62156]: 2021-03-15T22:25:10.781 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:10.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:10 smithi125 conmon[56739]: cluster 2021-03-15T22:25:09.048919+0000 mgr.y (mgr.14144) 604 : cluster [DBG] pgmap v717: 138 pgs: 138 active+clean; 281 MiB data, 866 MiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 524 KiB/s wr, 9 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:25:10.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:10 smithi125 conmon[56739]: cluster 2021-03-15T22:25:09.177516+0000 osd.7 (osd.7) 78 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:25:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:10 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:25:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:10 smithi125 conmon[56739]: 25:09.456944+0000 mon.a (mon.0) 1192 : cluster [DBG] osdmap e394: 8 total, 8 up, 7 in 2021-03-15T22:25:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:10 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:25:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:10 smithi125 conmon[56739]: -15T22:25:09.687481+0000 mgr.y 2021-03-15T22:25:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:10 smithi125 conmon[56739]: (mgr.14144) 605 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:10 smithi125 conmon[56739]: 2021-03-15T22:25:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:10 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:25:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:10 smithi125 conmon[56739]: 22:25:10.090661+0000 mon.a ( 2021-03-15T22:25:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:10 smithi125 conmon[56739]: mon.0) 1193 : cluster [DBG] osdmap e395: 8 total, 8 up, 7 in 2021-03-15T22:25:10.805 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:10.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:10.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:10.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:10.933 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:10.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:10.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:11.036 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:11.061 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:11.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:11.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:11.107 INFO:tasks.rados.rados.0.smithi076.stdout:1438: done (4 left) 2021-03-15T22:25:11.107 INFO:tasks.rados.rados.0.smithi076.stdout:1440: done (3 left) 2021-03-15T22:25:11.107 INFO:tasks.rados.rados.0.smithi076.stdout:1443: done (2 left) 2021-03-15T22:25:11.108 INFO:tasks.rados.rados.0.smithi076.stdout:1444: done (1 left) 2021-03-15T22:25:11.108 INFO:tasks.rados.rados.0.smithi076.stdout:1446: done (0 left) 2021-03-15T22:25:11.108 INFO:tasks.rados.rados.0.smithi076.stdout:1447: snap_remove snap 150 2021-03-15T22:25:11.164 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:11.186 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:11.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:11.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:11.194 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:11.288 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:11.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:11.319 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:11.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:11.421 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:11.444 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:11.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:11.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:11.516 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:25:11.532 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:25:11.546 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:11.549 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:25:11.566 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:25:11.568 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:11.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:11.585 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:25:11.605 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:25:11.624 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:25:11.643 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:25:11.644 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:25:11.644 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:25:11.644 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:25:11.644 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:25:11.645 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:25:11.645 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:25:11.645 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:25:11.645 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:25:11.671 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:11.697 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:11.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:11.699 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:11.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[56514]: cluster 2021-03-15T22:25:09.803356+0000 2021-03-15T22:25:11.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[56514]: osd.1 (osd.1) 80 : cluster [DBG] 7.b scrub ok 2021-03-15T22:25:11.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[56514]: cluster 2021-03-15T22:25:10.130416+0000 osd.7 (osd.7) 79 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:25:11.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[56514]: cluster 2021-03-15T22:25:10.616923+0000 osd.2 (osd.2) 68 : cluster [DBG] 7.e scrub ok 2021-03-15T22:25:11.741 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[56514]: audit 2021-03-15T22:25:11.056768+0000 mon.a (mon.0) 1194 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:25:11.742 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[56514]: cluster 2021-03-15T22:25:11.104847+0000 mon.a (mon.0) 1195 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2021-03-15T22:25:11.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: cluster 2021-03-15T22:25:09.803356+0000 osd.1 (osd. 2021-03-15T22:25:11.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: 1) 80 : cluster [DBG] 7.b scrub ok 2021-03-15T22:25:11.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: cluster 2021-03-15T22:25:10.130416+0000 osd.7 (osd.7) 79 : cluster 2021-03-15T22:25:11.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: [DBG] 3.11 scrub ok 2021-03-15T22:25:11.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:25:11.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: 15T22:25:10.616923+0000 osd.2 (osd.2) 68 2021-03-15T22:25:11.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: : cluster [DBG] 7.e scrub ok 2021-03-15T22:25:11.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:25:11.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: T22:25:11.056768+0000 mon.a ( 2021-03-15T22:25:11.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: mon.0) 1194 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:25:11.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:25:11.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: -15T22:25:11.104847+0000 2021-03-15T22:25:11.745 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: mon.a (mon.0) 1195 : cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2021-03-15T22:25:11.745 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:11 smithi076 conmon[62156]: 2021-03-15T22:25:11.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:25:11.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:11 smithi125 conmon[56739]: cluster 2021-03-15T22:25:09.803356+0000 osd.1 (osd.1) 80 : cluster [DBG] 7.b scrub ok 2021-03-15T22:25:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:11 smithi125 conmon[56739]: cluster 2021-03-15T22:25:10.130416+0000 osd.7 (osd.7) 79 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:25:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:11 smithi125 conmon[56739]: cluster 2021-03-15T22:25:10.616923+0000 osd.2 (osd.2) 68 : cluster [DBG] 7.e scrub ok 2021-03-15T22:25:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:11 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:25:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:11 smithi125 conmon[56739]: 15T22:25:11.056768+0000 mon.a (mon.0) 1194 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:25:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:11 smithi125 conmon[56739]: cluster 2021-03-15T22:25:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:11 smithi125 conmon[56739]: 2021-03-15T22:25:11.104847 2021-03-15T22:25:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:11 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1195 : 2021-03-15T22:25:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:11 smithi125 conmon[56739]: cluster [DBG] osdmap e396: 8 total, 8 up, 7 in 2021-03-15T22:25:11.800 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:11.825 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:11.827 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:11.928 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:11.953 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:11.955 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:12.056 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:12.082 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:12.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:12.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:12.107 INFO:tasks.rados.rados.0.smithi076.stdout:1447: done (0 left) 2021-03-15T22:25:12.107 INFO:tasks.rados.rados.0.smithi076.stdout:1448: delete oid 35 current snap is 155 2021-03-15T22:25:12.113 INFO:tasks.rados.rados.0.smithi076.stdout:1448: done (0 left) 2021-03-15T22:25:12.113 INFO:tasks.rados.rados.0.smithi076.stdout:1449: rollback oid 30 current snap is 155 2021-03-15T22:25:12.113 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 30 to 143 2021-03-15T22:25:12.113 INFO:tasks.rados.rados.0.smithi076.stdout:1450: write oid 50 current snap is 155 2021-03-15T22:25:12.113 INFO:tasks.rados.rados.0.smithi076.stdout:1450: seq_num 513 ranges {621991=798388,1936439=639509} 2021-03-15T22:25:12.120 INFO:tasks.rados.rados.0.smithi076.stdout:1450: writing smithi07694878-50 from 621991 to 1420379 tid 1 2021-03-15T22:25:12.127 INFO:tasks.rados.rados.0.smithi076.stdout:1450: writing smithi07694878-50 from 1936439 to 2575948 tid 2 2021-03-15T22:25:12.128 INFO:tasks.rados.rados.0.smithi076.stdout:1449: finishing rollback tid 0 to smithi07694878-30 2021-03-15T22:25:12.128 INFO:tasks.rados.rados.0.smithi076.stdout:1449: finishing rollback tid 1 to smithi07694878-30 2021-03-15T22:25:12.128 INFO:tasks.rados.rados.0.smithi076.stdout:1449: finishing rollback tid 2 to smithi07694878-30 2021-03-15T22:25:12.128 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 882 (ObjNum 475 snap 140 seq_num 475) dirty exists 2021-03-15T22:25:12.129 INFO:tasks.rados.rados.0.smithi076.stdout:1450: finishing write tid 1 to smithi07694878-50 2021-03-15T22:25:12.129 INFO:tasks.rados.rados.0.smithi076.stdout:1449: done (1 left) 2021-03-15T22:25:12.129 INFO:tasks.rados.rados.0.smithi076.stdout:1451: write oid 24 current snap is 155 2021-03-15T22:25:12.129 INFO:tasks.rados.rados.0.smithi076.stdout:1451: seq_num 514 ranges {638798=473637,1594921=783167} 2021-03-15T22:25:12.133 INFO:tasks.rados.rados.0.smithi076.stdout:1451: writing smithi07694878-24 from 638798 to 1112435 tid 1 2021-03-15T22:25:12.141 INFO:tasks.rados.rados.0.smithi076.stdout:1451: writing smithi07694878-24 from 1594921 to 2378088 tid 2 2021-03-15T22:25:12.141 INFO:tasks.rados.rados.0.smithi076.stdout:1452: snap_create 2021-03-15T22:25:12.141 INFO:tasks.rados.rados.0.smithi076.stdout:1450: finishing write tid 2 to smithi07694878-50 2021-03-15T22:25:12.141 INFO:tasks.rados.rados.0.smithi076.stdout:1450: finishing write tid 3 to smithi07694878-50 2021-03-15T22:25:12.142 INFO:tasks.rados.rados.0.smithi076.stdout:1450: finishing write tid 4 to smithi07694878-50 2021-03-15T22:25:12.142 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 671 (ObjNum 513 snap 155 seq_num 513) dirty exists 2021-03-15T22:25:12.142 INFO:tasks.rados.rados.0.smithi076.stdout:1450: left oid 50 (ObjNum 513 snap 155 seq_num 513) 2021-03-15T22:25:12.142 INFO:tasks.rados.rados.0.smithi076.stdout:1451: finishing write tid 1 to smithi07694878-24 2021-03-15T22:25:12.144 INFO:tasks.rados.rados.0.smithi076.stdout:1451: finishing write tid 2 to smithi07694878-24 2021-03-15T22:25:12.144 INFO:tasks.rados.rados.0.smithi076.stdout:1451: finishing write tid 3 to smithi07694878-24 2021-03-15T22:25:12.144 INFO:tasks.rados.rados.0.smithi076.stdout:1451: finishing write tid 4 to smithi07694878-24 2021-03-15T22:25:12.144 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 119 (ObjNum 514 snap 155 seq_num 514) dirty exists 2021-03-15T22:25:12.145 INFO:tasks.rados.rados.0.smithi076.stdout:1451: left oid 24 (ObjNum 514 snap 155 seq_num 514) 2021-03-15T22:25:12.185 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:12.211 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:12.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:12.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:12.314 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:12.339 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:12.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:12.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:12.441 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:12.463 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:12.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:12.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:12.566 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:12.588 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:12.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:12.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:12.691 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:12.717 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:12.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:12.719 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:12.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[56514]: cluster 2021-03-15T22:25:10.802591+0000 osd.1 (osd.1) 81 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:12.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[56514]: cluster 2021-03-15T22:25:10.803200+0000 osd.1 (osd.1) 82 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:12.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[56514]: cluster 2021-03-15T22:25:11.028417+0000 osd.4 (osd.4) 90 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:25:12.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[56514]: cluster 2021-03-15T22:25:11.049378+0000 mgr.y (mgr.14144) 606 : cluster [DBG] pgmap v720: 138 pgs: 138 active+clean; 279 MiB data, 854 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 10 op/s 2021-03-15T22:25:12.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[56514]: cluster 2021-03-15T22:25:11.117399+0000 osd.7 (osd.7) 80 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:25:12.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[56514]: audit 2021-03-15T22:25:11.496808+0000 mon.a (mon.0) 1196 : audit [DBG] from='client.? 172.21.15.76:0/1429717229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:12.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[56514]: cluster 2021-03-15T22:25:11.614968+0000 osd.5 (osd.5) 71 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:25:12.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[56514]: cluster 2021-03-15T22:25:12.102304+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2021-03-15T22:25:12.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[62156]: cluster 2021-03-15T22:25:10.802591+0000 osd.1 (osd.1) 81 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:12.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[62156]: cluster 2021-03-15T22:25:10.803200+0000 osd.1 (osd.1) 82 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:12.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[62156]: cluster 2021-03-15T22:25:11.028417+0000 osd.4 (osd.4) 90 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:25:12.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[62156]: cluster 2021-03-15T22:25:11.049378+0000 mgr.y (mgr.14144) 606 : cluster [DBG] pgmap v720: 138 pgs: 138 active+clean; 279 MiB data, 854 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 10 op/s 2021-03-15T22:25:12.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[62156]: cluster 2021-03-15T22:25:11.117399+0000 osd.7 2021-03-15T22:25:12.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[62156]: (osd.7) 80 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:25:12.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[62156]: audit 2021-03-15T22:25:11.496808+0000 mon.a (mon.0) 1196 : audit [DBG] from='client.? 172.21.15.76:0/1429717229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:12.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[62156]: cluster 2021-03-15T22:25:11.614968+0000 osd.5 (osd.5) 71 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:25:12.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:12 smithi076 conmon[62156]: cluster 2021-03-15T22:25:12.102304+0000 mon.a (mon.0) 1197 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2021-03-15T22:25:12.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: cluster 2021-03-15T22:25:10.802591+0000 osd.1 (osd.1) 81 : 2021-03-15T22:25:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: cluster 2021-03-15T22:25:10.803200+0000 osd.1 (osd.1) 82 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: cluster 2021-03-15T22:25:11 2021-03-15T22:25:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: .028417+0000 osd.4 (osd.4) 90 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:25:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: cluster 2021-03-15T22:25:11 2021-03-15T22:25:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: .049378+0000 mgr.y (mgr.14144) 606 : cluster 2021-03-15T22:25:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: [DBG] pgmap v720: 138 pgs: 138 active+clean; 279 MiB data, 854 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 10 op/s 2021-03-15T22:25:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:25:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: :25:11.117399+0000 osd.7 (osd.7 2021-03-15T22:25:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: ) 80 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:25:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: audit 2021 2021-03-15T22:25:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: -03-15T22:25:11.496808+0000 2021-03-15T22:25:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: mon.a (mon.0) 1196 : audit [DBG] from='client.? 172.21.15.76:0/1429717229' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: 2021-03-15T22:25:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: cluster 2021-03-15T22:25: 2021-03-15T22:25:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: 11.614968+0000 osd.5 (osd.5) 71 : 2021-03-15T22:25:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: cluster [DBG] 2.2 scrub ok 2021-03-15T22:25:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:25:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: T22:25:12.102304+0000 mon.a ( 2021-03-15T22:25:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:12 smithi125 conmon[56739]: mon.0) 1197 : cluster [DBG] osdmap e397: 8 total, 8 up, 7 in 2021-03-15T22:25:12.820 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:12.846 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:12.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:12.849 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:12.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:12.975 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:12.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:12.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:13.050 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:13.077 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:13.103 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:13.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:13.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:13.111 INFO:tasks.rados.rados.0.smithi076.stdout:1450: done (2 left) 2021-03-15T22:25:13.111 INFO:tasks.rados.rados.0.smithi076.stdout:1451: done (1 left) 2021-03-15T22:25:13.112 INFO:tasks.rados.rados.0.smithi076.stdout:1452: done (0 left) 2021-03-15T22:25:13.112 INFO:tasks.rados.rados.0.smithi076.stdout:1453: snap_create 2021-03-15T22:25:13.117 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.5 to 1 (10000) 2021-03-15T22:25:13.205 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:13.231 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:13.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:13.233 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:13.334 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:13.359 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:13.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:13.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:13.462 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:13.488 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:13.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:13.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:13.590 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:13.614 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:13.615 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:13.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:13.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:13.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[56514]: cluster 2021-03-15T22:25:11.794428+0000 osd.1 (osd.1) 83 : 2021-03-15T22:25:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[56514]: cluster [DBG] 5.12 deep-scrub ok 2021-03-15T22:25:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[56514]: cluster 2021-03-15T22:25:12.147409+0000 osd.7 (osd.7) 81 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:25:13.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[56514]: cluster 2021-03-15T22:25:12.353186+0000 osd.6 (osd.6) 63 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:25:13.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[56514]: audit 2021-03-15T22:25:12.564759+0000 mon.a (mon.0) 1198 : audit [INF] from='client.? 172.21.15.76:0/1027735886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:25:13.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[56514]: audit 2021-03-15T22:25:13.106655+0000 mon.a (mon.0) 1199 : audit [INF] from='client.? 172.21.15.76:0/1027735886' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-15T22:25:13.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[56514]: cluster 2021-03-15T22:25:13.106737+0000 mon.a (mon.0) 1200 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2021-03-15T22:25:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: cluster 2021-03-15T22:25:11.794428+0000 osd.1 (osd 2021-03-15T22:25:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: .1) 83 : cluster [DBG] 5.12 deep-scrub ok 2021-03-15T22:25:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: cluster 2021-03-15T22:25:12.147409+0000 osd.7 (osd.7) 2021-03-15T22:25:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: 81 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:25:13.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:25:13.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: -15T22:25:12.353186 2021-03-15T22:25:13.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: +0000 osd.6 (osd.6) 2021-03-15T22:25:13.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: 63 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:25:13.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:25:13.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: 03-15T22:25:12 2021-03-15T22:25:13.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: .564759+0000 mon.a (mon. 2021-03-15T22:25:13.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: 0) 1198 : audit [INF] from='client.? 172.21.15.76:0/1027735886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:25:13.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: 2021-03-15T22:25:13.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: audit 2021 2021-03-15T22:25:13.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: -03-15T22:25:13.106655 2021-03-15T22:25:13.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2021-03-15T22:25:13.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: 1199 : audit [INF] from='client.? 172.21.15.76:0/1027735886' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-15T22:25:13.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: cluster 2021-03-15T22:25:13.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: 2021-03-15T22:25 2021-03-15T22:25:13.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: :13.106737+0000 mon.a (mon 2021-03-15T22:25:13.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: .0) 1200 : cluster [DBG] 2021-03-15T22:25:13.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:13 smithi076 conmon[62156]: osdmap e398: 8 total, 8 up, 7 in 2021-03-15T22:25:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: cluster 2021-03-15T22:25:11.794428+0000 osd.1 (osd.1) 83 : cluster [DBG] 5.12 deep-scrub ok 2021-03-15T22:25:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: cluster 2021-03-15T22:25:12.147409+0000 osd.7 (osd.7) 81 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:25:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: 03-15T22:25:12.353186+0000 osd.6 (osd. 2021-03-15T22:25:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: 6) 63 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:25:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:25:13.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: 03-15T22:25:12.564759+0000 2021-03-15T22:25:13.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: mon.a (mon.0) 1198 : audit 2021-03-15T22:25:13.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/1027735886' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:25:13.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: audit 2021 2021-03-15T22:25:13.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: -03-15T22:25:13. 2021-03-15T22:25:13.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: 106655+0000 mon.a (mon.0) 2021-03-15T22:25:13.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: 1199 : audit [INF] from='client.? 172.21.15.76:0/1027735886' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-15T22:25:13.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:25:13.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: -03-15T22:25:13 2021-03-15T22:25:13.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: .106737+0000 mon.a (mon. 2021-03-15T22:25:13.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: 0) 1200 : cluster [DBG] osdmap e398: 8 total, 8 up, 7 in 2021-03-15T22:25:13.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:13 smithi125 conmon[56739]: 2021-03-15T22:25:13.840 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:13.867 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:13.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:13.868 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:13.931 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 6 1 2021-03-15T22:25:13.969 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:13.996 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:13.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:13.998 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:14.099 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:14.112 INFO:tasks.rados.rados.0.smithi076.stdout:1453: done (0 left) 2021-03-15T22:25:14.113 INFO:tasks.rados.rados.0.smithi076.stdout:1454: delete oid 40 current snap is 157 2021-03-15T22:25:14.119 INFO:tasks.rados.rados.0.smithi076.stdout:1454: done (0 left) 2021-03-15T22:25:14.119 INFO:tasks.rados.rados.0.smithi076.stdout:1455: snap_remove snap 138 2021-03-15T22:25:14.130 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:14.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:14.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:14.232 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:14.259 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:14.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:14.261 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:14.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:14.388 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:14.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:14.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:14.441 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:25:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:25:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:25:14.441 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:25:14.442 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:25:14.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:14.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:14.517 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:14.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:14.618 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:14.642 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:14.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:14.744 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:14.765 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:14.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:14.767 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:14.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:14 smithi076 conmon[62156]: cluster 2021-03-15T22:25:12.911135+0000 osd.3 (osd.3) 70 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:25:14.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:14 smithi076 conmon[62156]: cluster 2021-03-15T22:25:13.049912+0000 mgr.y (mgr.14144) 607 : cluster [DBG] pgmap v723: 138 pgs: 138 active+clean; 276 MiB data, 856 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 7 op/s 2021-03-15T22:25:14.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:14 smithi076 conmon[62156]: cluster 2021-03-15T22:25:13.181218+0000 osd.7 (osd.7) 82 : cluster [DBG] 4.17 deep-scrub ok 2021-03-15T22:25:14.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:14 smithi076 conmon[62156]: cluster 2021-03-15T22:25:14.106809+0000 mon.a (mon.0) 1201 : cluster [DBG] osdmap e399: 8 total, 8 up, 7 in 2021-03-15T22:25:14.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:14 smithi076 conmon[56514]: cluster 2021-03-15T22:25:12.911135+0000 osd.3 (osd.3) 70 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:25:14.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:14 smithi076 conmon[56514]: cluster 2021-03-15T22:25:13.049912+0000 mgr.y (mgr.14144) 607 : cluster [DBG] pgmap v723: 138 pgs: 138 active+clean; 276 MiB data, 856 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 7 op/s 2021-03-15T22:25:14.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:14 smithi076 conmon[56514]: cluster 2021-03-15T22:25:13.181218+0000 osd.7 (osd.7) 82 : cluster [DBG] 4.17 deep-scrub ok 2021-03-15T22:25:14.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:14 smithi076 conmon[56514]: cluster 2021-03-15T22:25:14.106809+0000 mon.a (mon.0) 1201 : cluster [DBG] osdmap e399: 8 total, 8 up, 7 in 2021-03-15T22:25:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:14 smithi125 conmon[56739]: cluster 2021-03-15T22:25:12.911135+0000 osd.3 (osd.3) 70 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:25:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:14 smithi125 conmon[56739]: 2021-03-15T22:25:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:14 smithi125 conmon[56739]: cluster 2021-03-15T22:25:13.049912+0000 mgr.y (mgr.14144) 607 : cluster [DBG] pgmap v723: 138 pgs: 138 active+clean; 276 MiB data, 856 MiB used, 625 GiB / 626 GiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 7 op/s 2021-03-15T22:25:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:14 smithi125 conmon[56739]: cluster 2021-03-15T22:25:13.181218+0000 2021-03-15T22:25:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:14 smithi125 conmon[56739]: osd.7 (osd.7) 82 : cluster [DBG] 4.17 deep-scrub ok 2021-03-15T22:25:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:14 smithi125 conmon[56739]: cluster 2021-03-15T22:25:14.106809+0000 mon.a (mon.0) 1201 : cluster [DBG] 2021-03-15T22:25:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:14 smithi125 conmon[56739]: osdmap e399: 8 total, 8 up, 7 in 2021-03-15T22:25:14.868 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:14.894 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:14.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:14.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:15.017 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:25:15.020 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:15.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:15.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:15.101 INFO:tasks.rados.rados.0.smithi076.stdout:1455: done (0 left) 2021-03-15T22:25:15.102 INFO:tasks.rados.rados.0.smithi076.stdout:1456: snap_create 2021-03-15T22:25:15.123 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:15.147 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:15.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:15.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:15.186 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:15.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:15.273 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:15.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:15.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:15.376 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:15.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:15.400 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:15.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:15.502 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:15.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-15T22:25:15.524 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:15.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:15.526 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:15.592 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:15.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:15.651 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:15.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:15.653 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:15.754 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:15.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:15 smithi076 conmon[56514]: cluster 2021-03-15T22:25:14.014770+0000 osd.4 2021-03-15T22:25:15.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:15 smithi076 conmon[56514]: (osd.4) 91 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:25:15.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:15 smithi076 conmon[56514]: cluster 2021-03-15T22:25:15.094013+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2021-03-15T22:25:15.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:15 smithi076 conmon[62156]: cluster 2021-03-15T22:25:14.014770 2021-03-15T22:25:15.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:15 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 91 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:25:15.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:15 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:25:15.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:15 smithi076 conmon[62156]: -03-15T22:25:15.094013+0000 mon.a (mon.0) 2021-03-15T22:25:15.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:15 smithi076 conmon[62156]: 1202 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2021-03-15T22:25:15.780 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:15.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:15.782 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:15.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:15 smithi125 conmon[56739]: cluster 2021-03-15T22:25:14.014770+0000 osd.4 (osd.4) 91 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:25:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:15 smithi125 conmon[56739]: cluster 2021-03-15T22:25:15. 2021-03-15T22:25:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:15 smithi125 conmon[56739]: 094013+0000 mon.a (mon.0) 1202 : cluster [DBG] osdmap e400: 8 total, 8 up, 7 in 2021-03-15T22:25:15.883 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:15.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:15.907 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:15.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:16.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:16.021 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:16.034 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:16.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:16.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:16.111 INFO:tasks.rados.rados.0.smithi076.stdout:1456: done (0 left) 2021-03-15T22:25:16.112 INFO:tasks.rados.rados.0.smithi076.stdout:1457: rollback oid 39 current snap is 158 2021-03-15T22:25:16.112 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 39 to 141 2021-03-15T22:25:16.112 INFO:tasks.rados.rados.0.smithi076.stdout:1458: copy_from oid 28 from oid 41 current snap is 158 2021-03-15T22:25:16.112 INFO:tasks.rados.rados.0.smithi076.stdout:1459: write oid 31 current snap is 158 2021-03-15T22:25:16.113 INFO:tasks.rados.rados.0.smithi076.stdout:1459: seq_num 516 ranges {672412=624135,1995532=676880} 2021-03-15T22:25:16.117 INFO:tasks.rados.rados.0.smithi076.stdout:1459: writing smithi07694878-31 from 672412 to 1296547 tid 1 2021-03-15T22:25:16.126 INFO:tasks.rados.rados.0.smithi076.stdout:1459: writing smithi07694878-31 from 1995532 to 2672412 tid 2 2021-03-15T22:25:16.127 INFO:tasks.rados.rados.0.smithi076.stdout:1460: read oid 38 snap -1 2021-03-15T22:25:16.127 INFO:tasks.rados.rados.0.smithi076.stdout:1460: expect deleted 2021-03-15T22:25:16.127 INFO:tasks.rados.rados.0.smithi076.stdout:1461: write oid 29 current snap is 158 2021-03-15T22:25:16.127 INFO:tasks.rados.rados.0.smithi076.stdout:1457: finishing rollback tid 1 to smithi07694878-39 2021-03-15T22:25:16.128 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 0 (ObjNum 261801328 snap 21896 seq_num 261990032) dirty dne 2021-03-15T22:25:16.128 INFO:tasks.rados.rados.0.smithi076.stdout:1461: seq_num 517 ranges {689219=415737,1770367=646847} 2021-03-15T22:25:16.132 INFO:tasks.rados.rados.0.smithi076.stdout:1461: writing smithi07694878-29 from 689219 to 1104956 tid 1 2021-03-15T22:25:16.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:16.140 INFO:tasks.rados.rados.0.smithi076.stdout:1461: writing smithi07694878-29 from 1770367 to 2417214 tid 2 2021-03-15T22:25:16.141 INFO:tasks.rados.rados.0.smithi076.stdout:1457: done (4 left) 2021-03-15T22:25:16.141 INFO:tasks.rados.rados.0.smithi076.stdout:1462: copy_from oid 34 from oid 19 current snap is 158 2021-03-15T22:25:16.141 INFO:tasks.rados.rados.0.smithi076.stdout:1459: finishing write tid 1 to smithi07694878-31 2021-03-15T22:25:16.141 INFO:tasks.rados.rados.0.smithi076.stdout:1458: finishing copy_from to smithi07694878-28 2021-03-15T22:25:16.142 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 762 (ObjNum 509 snap 152 seq_num 509) dirty exists 2021-03-15T22:25:16.142 INFO:tasks.rados.rados.0.smithi076.stdout:1458: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:25:16.145 INFO:tasks.rados.rados.0.smithi076.stdout:1459: finishing write tid 2 to smithi07694878-31 2021-03-15T22:25:16.146 INFO:tasks.rados.rados.0.smithi076.stdout:1459: finishing write tid 3 to smithi07694878-31 2021-03-15T22:25:16.146 INFO:tasks.rados.rados.0.smithi076.stdout:1459: finishing write tid 4 to smithi07694878-31 2021-03-15T22:25:16.146 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 820 (ObjNum 516 snap 158 seq_num 516) dirty exists 2021-03-15T22:25:16.147 INFO:tasks.rados.rados.0.smithi076.stdout:1459: left oid 31 (ObjNum 516 snap 158 seq_num 516) 2021-03-15T22:25:16.147 INFO:tasks.rados.rados.0.smithi076.stdout:1461: finishing write tid 1 to smithi07694878-29 2021-03-15T22:25:16.147 INFO:tasks.rados.rados.0.smithi076.stdout:1458: done (4 left) 2021-03-15T22:25:16.147 INFO:tasks.rados.rados.0.smithi076.stdout:1459: done (3 left) 2021-03-15T22:25:16.147 INFO:tasks.rados.rados.0.smithi076.stdout:1460: done (2 left) 2021-03-15T22:25:16.148 INFO:tasks.rados.rados.0.smithi076.stdout:1463: write oid 50 current snap is 158 2021-03-15T22:25:16.148 INFO:tasks.rados.rados.0.smithi076.stdout:1463: seq_num 518 ranges {706026=490986,1828849=790505} 2021-03-15T22:25:16.148 INFO:tasks.rados.rados.0.smithi076.stdout:1463: writing smithi07694878-50 from 706026 to 1197012 tid 1 2021-03-15T22:25:16.155 INFO:tasks.rados.rados.0.smithi076.stdout:1463: writing smithi07694878-50 from 1828849 to 2619354 tid 2 2021-03-15T22:25:16.155 INFO:tasks.rados.rados.0.smithi076.stdout:1464: snap_remove snap 156 2021-03-15T22:25:16.162 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:16.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:16.163 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:16.264 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:16.287 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:16.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:16.288 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:16.389 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:16.412 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:16.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:16.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:16.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:16.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:16 smithi076 conmon[56514]: cluster 2021-03-15T22:25:15.050383+0000 mgr.y (mgr.14144) 608 : cluster [DBG] pgmap v726: 138 pgs: 1 remapped+peering, 137 active+clean; 279 MiB data, 867 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 MiB/s wr, 5 op/s 2021-03-15T22:25:16.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:16 smithi076 conmon[56514]: cluster 2021-03-15T22:25:15.333743+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:25:16.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:16 smithi076 conmon[56514]: cluster 2021-03-15T22:25:16.106569+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2021-03-15T22:25:16.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:16 smithi076 conmon[62156]: cluster 2021-03-15T22:25:15.050383+0000 2021-03-15T22:25:16.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:16 smithi076 conmon[62156]: mgr.y (mgr.14144) 608 : cluster [DBG] pgmap v726: 138 pgs: 1 remapped+peering, 137 active+clean; 279 MiB data, 867 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 MiB/s wr, 5 op/s 2021-03-15T22:25:16.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:16 smithi076 conmon[62156]: cluster 2021-03-15T22:25:15.333743+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:25:16.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:16 smithi076 conmon[62156]: cluster 2021-03-15T22:25:16.106569+0000 mon.a (mon.0) 1203 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2021-03-15T22:25:16.536 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:16.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:16.538 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:16.639 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:16.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:16.663 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:16.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:16.765 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:16.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:16 smithi125 conmon[56739]: cluster 2021-03-15T22:25:15.050383+0000 mgr.y (mgr.14144) 608 : cluster [DBG] pgmap v726: 138 pgs: 1 remapped+peering, 137 active+clean; 279 MiB data, 867 MiB used, 625 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.2 MiB/s wr, 5 op/s 2021-03-15T22:25:16.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:16 smithi125 conmon[56739]: cluster 2021-03-15T22:25:15.333743+0000 osd.6 (osd.6) 64 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:25:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:16 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:25:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:16 smithi125 conmon[56739]: T22:25:16.106569+0000 mon.a ( 2021-03-15T22:25:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:16 smithi125 conmon[56739]: mon.0) 1203 : cluster [DBG] osdmap e401: 8 total, 8 up, 7 in 2021-03-15T22:25:16.788 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:16.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:16.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:16.891 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:16.915 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:16.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:16.916 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:17.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:17.041 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:17.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:17.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:17.108 INFO:tasks.rados.rados.0.smithi076.stdout:1464: done (3 left) 2021-03-15T22:25:17.108 INFO:tasks.rados.rados.0.smithi076.stdout:1465: write oid 27 current snap is 158 2021-03-15T22:25:17.109 INFO:tasks.rados.rados.0.smithi076.stdout:1461: finishing write tid 2 to smithi07694878-29 2021-03-15T22:25:17.109 INFO:tasks.rados.rados.0.smithi076.stdout:1465: seq_num 519 ranges {722833=566235,1887331=650516} 2021-03-15T22:25:17.113 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:25:17.116 INFO:tasks.rados.rados.0.smithi076.stdout:1465: writing smithi07694878-27 from 722833 to 1289068 tid 1 2021-03-15T22:25:17.123 INFO:tasks.rados.rados.0.smithi076.stdout:1465: writing smithi07694878-27 from 1887331 to 2537847 tid 2 2021-03-15T22:25:17.123 INFO:tasks.rados.rados.0.smithi076.stdout:1466: delete oid 25 current snap is 158 2021-03-15T22:25:17.124 INFO:tasks.rados.rados.0.smithi076.stdout:1461: finishing write tid 3 to smithi07694878-29 2021-03-15T22:25:17.124 INFO:tasks.rados.rados.0.smithi076.stdout:1461: finishing write tid 4 to smithi07694878-29 2021-03-15T22:25:17.124 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 910 (ObjNum 517 snap 158 seq_num 517) dirty exists 2021-03-15T22:25:17.124 INFO:tasks.rados.rados.0.smithi076.stdout:1461: left oid 29 (ObjNum 517 snap 158 seq_num 517) 2021-03-15T22:25:17.125 INFO:tasks.rados.rados.0.smithi076.stdout:1462: finishing copy_from to smithi07694878-34 2021-03-15T22:25:17.125 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 489 (ObjNum 495 snap 149 seq_num 495) dirty exists 2021-03-15T22:25:17.125 INFO:tasks.rados.rados.0.smithi076.stdout:1463: finishing write tid 1 to smithi07694878-50 2021-03-15T22:25:17.125 INFO:tasks.rados.rados.0.smithi076.stdout:1462: finishing copy_from racing read to smithi07694878-34 2021-03-15T22:25:17.125 INFO:tasks.rados.rados.0.smithi076.stdout:1463: finishing write tid 2 to smithi07694878-50 2021-03-15T22:25:17.126 INFO:tasks.rados.rados.0.smithi076.stdout:1463: finishing write tid 3 to smithi07694878-50 2021-03-15T22:25:17.126 INFO:tasks.rados.rados.0.smithi076.stdout:1463: finishing write tid 4 to smithi07694878-50 2021-03-15T22:25:17.126 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 674 (ObjNum 518 snap 158 seq_num 518) dirty exists 2021-03-15T22:25:17.126 INFO:tasks.rados.rados.0.smithi076.stdout:1463: left oid 50 (ObjNum 518 snap 158 seq_num 518) 2021-03-15T22:25:17.127 INFO:tasks.rados.rados.0.smithi076.stdout:1465: finishing write tid 1 to smithi07694878-27 2021-03-15T22:25:17.127 INFO:tasks.rados.rados.0.smithi076.stdout:1461: done (4 left) 2021-03-15T22:25:17.127 INFO:tasks.rados.rados.0.smithi076.stdout:1462: done (3 left) 2021-03-15T22:25:17.128 INFO:tasks.rados.rados.0.smithi076.stdout:1463: done (2 left) 2021-03-15T22:25:17.128 INFO:tasks.rados.rados.0.smithi076.stdout:1466: done (1 left) 2021-03-15T22:25:17.128 INFO:tasks.rados.rados.0.smithi076.stdout:1467: write oid 1 current snap is 158 2021-03-15T22:25:17.128 INFO:tasks.rados.rados.0.smithi076.stdout:1467: seq_num 521 ranges {756447=716733,1887942=537832} 2021-03-15T22:25:17.134 INFO:tasks.rados.rados.0.smithi076.stdout:1467: writing smithi07694878-1 from 756447 to 1473180 tid 1 2021-03-15T22:25:17.140 INFO:tasks.rados.rados.0.smithi076.stdout:1467: writing smithi07694878-1 from 1887942 to 2425774 tid 2 2021-03-15T22:25:17.140 INFO:tasks.rados.rados.0.smithi076.stdout:1468: read oid 30 snap 145 2021-03-15T22:25:17.141 INFO:tasks.rados.rados.0.smithi076.stdout:1468: expect (ObjNum 490 snap 145 seq_num 490) 2021-03-15T22:25:17.141 INFO:tasks.rados.rados.0.smithi076.stdout:1469: delete oid 40 current snap is 158 2021-03-15T22:25:17.141 INFO:tasks.rados.rados.0.smithi076.stdout:1465: finishing write tid 2 to smithi07694878-27 2021-03-15T22:25:17.141 INFO:tasks.rados.rados.0.smithi076.stdout:1465: finishing write tid 3 to smithi07694878-27 2021-03-15T22:25:17.141 INFO:tasks.rados.rados.0.smithi076.stdout:1465: finishing write tid 4 to smithi07694878-27 2021-03-15T22:25:17.142 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 322 (ObjNum 519 snap 158 seq_num 519) dirty exists 2021-03-15T22:25:17.142 INFO:tasks.rados.rados.0.smithi076.stdout:1465: left oid 27 (ObjNum 519 snap 158 seq_num 519) 2021-03-15T22:25:17.142 INFO:tasks.rados.rados.0.smithi076.stdout:1467: finishing write tid 1 to smithi07694878-1 2021-03-15T22:25:17.143 INFO:tasks.rados.rados.0.smithi076.stdout:1465: done (3 left) 2021-03-15T22:25:17.143 INFO:tasks.rados.rados.0.smithi076.stdout:1469: done (2 left) 2021-03-15T22:25:17.143 INFO:tasks.rados.rados.0.smithi076.stdout:1470: write oid 39 current snap is 158 2021-03-15T22:25:17.144 INFO:tasks.rados.rados.0.smithi076.stdout:1470: seq_num 523 ranges {790061=467231,2004906=541501} 2021-03-15T22:25:17.144 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:17.149 INFO:tasks.rados.rados.0.smithi076.stdout:1470: writing smithi07694878-39 from 790061 to 1257292 tid 1 2021-03-15T22:25:17.157 INFO:tasks.rados.rados.0.smithi076.stdout:1470: writing smithi07694878-39 from 2004906 to 2546407 tid 2 2021-03-15T22:25:17.157 INFO:tasks.rados.rados.0.smithi076.stdout:1471: read oid 15 snap -1 2021-03-15T22:25:17.157 INFO:tasks.rados.rados.0.smithi076.stdout:1471: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:25:17.158 INFO:tasks.rados.rados.0.smithi076.stdout:1472: write oid 43 current snap is 158 2021-03-15T22:25:17.158 INFO:tasks.rados.rados.0.smithi076.stdout:1467: finishing write tid 2 to smithi07694878-1 2021-03-15T22:25:17.158 INFO:tasks.rados.rados.0.smithi076.stdout:1472: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2021-03-15T22:25:17.162 INFO:tasks.rados.rados.0.smithi076.stdout:1472: writing smithi07694878-43 from 406868 to 949348 tid 1 2021-03-15T22:25:17.167 INFO:tasks.rados.rados.0.smithi076.stdout:1472: writing smithi07694878-43 from 1547035 to 1948547 tid 2 2021-03-15T22:25:17.168 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:17.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:17.169 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:17.173 INFO:tasks.rados.rados.0.smithi076.stdout:1472: writing smithi07694878-43 from 2690354 to 2806868 tid 3 2021-03-15T22:25:17.173 INFO:tasks.rados.rados.0.smithi076.stdout:1473: delete oid 42 current snap is 158 2021-03-15T22:25:17.173 INFO:tasks.rados.rados.0.smithi076.stdout:1467: finishing write tid 3 to smithi07694878-1 2021-03-15T22:25:17.173 INFO:tasks.rados.rados.0.smithi076.stdout:1467: finishing write tid 4 to smithi07694878-1 2021-03-15T22:25:17.174 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 823 (ObjNum 521 snap 158 seq_num 521) dirty exists 2021-03-15T22:25:17.174 INFO:tasks.rados.rados.0.smithi076.stdout:1467: left oid 1 (ObjNum 521 snap 158 seq_num 521) 2021-03-15T22:25:17.174 INFO:tasks.rados.rados.0.smithi076.stdout:1468: expect (ObjNum 490 snap 145 seq_num 490) 2021-03-15T22:25:17.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:17.289 INFO:tasks.rados.rados.0.smithi076.stdout:1470: finishing write tid 1 to smithi07694878-39 2021-03-15T22:25:17.289 INFO:tasks.rados.rados.0.smithi076.stdout:1467: done (5 left) 2021-03-15T22:25:17.289 INFO:tasks.rados.rados.0.smithi076.stdout:1468: done (4 left) 2021-03-15T22:25:17.290 INFO:tasks.rados.rados.0.smithi076.stdout:1473: done (3 left) 2021-03-15T22:25:17.290 INFO:tasks.rados.rados.0.smithi076.stdout:1474: snap_create 2021-03-15T22:25:17.290 INFO:tasks.rados.rados.0.smithi076.stdout:1470: finishing write tid 2 to smithi07694878-39 2021-03-15T22:25:17.290 INFO:tasks.rados.rados.0.smithi076.stdout:1470: finishing write tid 3 to smithi07694878-39 2021-03-15T22:25:17.291 INFO:tasks.rados.rados.0.smithi076.stdout:1470: finishing write tid 4 to smithi07694878-39 2021-03-15T22:25:17.291 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 516 (ObjNum 523 snap 158 seq_num 523) dirty exists 2021-03-15T22:25:17.291 INFO:tasks.rados.rados.0.smithi076.stdout:1470: left oid 39 (ObjNum 523 snap 158 seq_num 523) 2021-03-15T22:25:17.291 INFO:tasks.rados.rados.0.smithi076.stdout:1471: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:25:17.292 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:17.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:17.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:17.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:17.418 INFO:tasks.rados.rados.0.smithi076.stdout:1472: finishing write tid 1 to smithi07694878-43 2021-03-15T22:25:17.418 INFO:tasks.rados.rados.0.smithi076.stdout:1472: finishing write tid 2 to smithi07694878-43 2021-03-15T22:25:17.418 INFO:tasks.rados.rados.0.smithi076.stdout:1472: finishing write tid 3 to smithi07694878-43 2021-03-15T22:25:17.419 INFO:tasks.rados.rados.0.smithi076.stdout:1472: finishing write tid 4 to smithi07694878-43 2021-03-15T22:25:17.419 INFO:tasks.rados.rados.0.smithi076.stdout:1472: finishing write tid 5 to smithi07694878-43 2021-03-15T22:25:17.419 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 759 (ObjNum 524 snap 158 seq_num 524) dirty exists 2021-03-15T22:25:17.419 INFO:tasks.rados.rados.0.smithi076.stdout:1472: left oid 43 (ObjNum 524 snap 158 seq_num 524) 2021-03-15T22:25:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:17.421 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:17.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:17.522 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:17.546 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:17.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:17.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:17.648 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:17.673 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:17.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:17.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:17.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: cluster 2021-03-15T22:25:15.810369+0000 osd.1 ( 2021-03-15T22:25:17.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: osd.1) 84 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:25:17.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: cluster 2021-03-15T22:25:15.835666+0000 osd.3 (osd.3) 71 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:25:17.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: cluster 2021-03-15T22:25:16.170931+0000 osd.5 (osd.5) 72 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:17.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: cluster 2021-03-15T22:25:16.171892+0000 osd.5 (osd.5) 73 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:17.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: cluster 2021-03-15T22:25:16.234562+0000 osd.7 (osd.7) 83 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:25:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: audit 2021-03-15T22:25:16.513571+0000 mon.c (mon.1) 35 : audit [INF] from='client.? 172.21.15.76:0/2313577899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: audit 2021-03-15T22:25:16.513817+0000 mon.a (mon.0) 1204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: cluster 2021-03-15T22:25:16.597994+0000 osd.5 (osd.5) 74 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:25:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:25:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: 25:17.101388+0000 mon.a (mon 2021-03-15T22:25:17.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: .0) 1205 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:17.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: 2021-03-15T22:25:17.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:25:17.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: T22:25:17.105712+0000 2021-03-15T22:25:17.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: mon.a (mon.0) 1206 : 2021-03-15T22:25:17.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:25:17.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: cluster 2021-03-15T22:25:17.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: 2021-03-15T22:25: 2021-03-15T22:25:17.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: 17.105760+0000 mon.a (mon 2021-03-15T22:25:17.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: .0) 1207 : cluster [DBG] 2021-03-15T22:25:17.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[62156]: osdmap e402: 8 total, 8 up, 7 in 2021-03-15T22:25:17.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: cluster 2021-03-15T22:25:15.810369+0000 osd.1 (osd.1 2021-03-15T22:25:17.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: ) 84 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:25:17.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: cluster 2021-03-15T22:25:15.835666+0000 osd.3 (osd.3) 71 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:25:17.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: cluster 2021-03-15T22:25:16.170931+0000 osd.5 (osd.5) 72 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:17.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: cluster 2021-03-15T22:25:16.171892+0000 osd.5 (osd.5) 73 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:17.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: cluster 2021-03-15T22:25:16.234562+0000 osd.7 (osd.7) 83 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:25:17.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: audit 2021-03-15T22:25:16.513571+0000 mon.c (mon.1) 35 : audit [INF] from='client.? 172.21.15.76:0/2313577899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:17.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: audit 2021-03-15T22:25:16.513817+0000 mon.a (mon.0) 1204 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:17.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: cluster 2021-03-15T22:25:16.597994+0000 osd.5 (osd.5) 74 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:25:17.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: cluster 2021-03-15T22:25:17.101388+0000 mon.a (mon.0) 1205 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:17.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: audit 2021-03-15T22:25:17.105712+0000 mon.a (mon.0) 1206 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:25:17.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:17 smithi076 conmon[56514]: cluster 2021-03-15T22:25:17.105760+0000 mon.a (mon.0) 1207 : cluster [DBG] osdmap e402: 8 total, 8 up, 7 in 2021-03-15T22:25:17.776 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:17.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: cluster 2021-03-15T22:25:15.810369+0000 2021-03-15T22:25:17.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: osd.1 (osd.1) 84 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:25:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: cluster 2021-03-15T22:25:15.835666+0000 osd.3 (osd.3) 71 : cluster [DBG] 2021-03-15T22:25:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 4.10 scrub ok 2021-03-15T22:25:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:25:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 15T22:25:16.170931+0000 osd.5 (osd.5) 72 2021-03-15T22:25:17.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:17.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:25:17.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 15T22:25:16.171892+0000 osd.5 ( 2021-03-15T22:25:17.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: osd.5) 73 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:17.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:17.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 03-15T22:25:16.234562+0000 2021-03-15T22:25:17.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: osd.7 (osd.7) 83 : cluster [DBG] 2021-03-15T22:25:17.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 6.3 scrub ok 2021-03-15T22:25:17.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:25:17.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: :25:16.513571+0000 mon.c (mon. 2021-03-15T22:25:17.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 1) 35 : audit [INF] from='client.? 172.21.15.76:0/2313577899' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:17.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:25:17.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 03-15T22:25:16.513817+0000 2021-03-15T22:25:17.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: mon.a (mon.0) 1204 : audit [INF] 2021-03-15T22:25:17.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:17.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: cluster 2021-03-15T22:25 2021-03-15T22:25:17.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: :16.597994+0000 osd.5 (osd.5) 2021-03-15T22:25:17.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 74 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:25:17.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:17.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 03-15T22:25:17.101388+0000 2021-03-15T22:25:17.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: mon.a (mon.0) 1205 : cluster [WRN] 2021-03-15T22:25:17.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:17.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:25:17.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: :25:17.105712+0000 mon.a (mon. 2021-03-15T22:25:17.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 0) 1206 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:25:17.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: cluster 2021-03-15T22:25:17.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 2021-03-15T22:25:17. 2021-03-15T22:25:17.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: 105760+0000 mon.a (mon.0) 1207 : cluster 2021-03-15T22:25:17.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:17 smithi125 conmon[56739]: [DBG] osdmap e402: 8 total, 8 up, 7 in 2021-03-15T22:25:17.800 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:17.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:17.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:17.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:17.925 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:17.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:17.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:18.028 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:18.053 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:18.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:18.112 INFO:tasks.rados.rados.0.smithi076.stdout:1470: done (3 left) 2021-03-15T22:25:18.112 INFO:tasks.rados.rados.0.smithi076.stdout:1471: done (2 left) 2021-03-15T22:25:18.113 INFO:tasks.rados.rados.0.smithi076.stdout:1472: done (1 left) 2021-03-15T22:25:18.113 INFO:tasks.rados.rados.0.smithi076.stdout:1474: done (0 left) 2021-03-15T22:25:18.113 INFO:tasks.rados.rados.0.smithi076.stdout:1475: read oid 18 snap 140 2021-03-15T22:25:18.113 INFO:tasks.rados.rados.0.smithi076.stdout:1475: expect (ObjNum 324 snap 87 seq_num 324) 2021-03-15T22:25:18.113 INFO:tasks.rados.rados.0.smithi076.stdout:1476: write oid 7 current snap is 159 2021-03-15T22:25:18.114 INFO:tasks.rados.rados.0.smithi076.stdout:1476: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2021-03-15T22:25:18.117 INFO:tasks.rados.rados.0.smithi076.stdout:1476: writing smithi07694878-7 from 440482 to 849813 tid 1 2021-03-15T22:25:18.122 INFO:tasks.rados.rados.0.smithi076.stdout:1476: writing smithi07694878-7 from 1380352 to 1785533 tid 2 2021-03-15T22:25:18.128 INFO:tasks.rados.rados.0.smithi076.stdout:1476: writing smithi07694878-7 from 2461553 to 2840482 tid 3 2021-03-15T22:25:18.128 INFO:tasks.rados.rados.0.smithi076.stdout:1477: copy_from oid 6 from oid 26 current snap is 159 2021-03-15T22:25:18.129 INFO:tasks.rados.rados.0.smithi076.stdout:1475: expect (ObjNum 324 snap 87 seq_num 324) 2021-03-15T22:25:18.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:18.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:18.182 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:18.182 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:18.284 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:18.305 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:18.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:18.337 INFO:tasks.rados.rados.0.smithi076.stdout:1476: finishing write tid 1 to smithi07694878-7 2021-03-15T22:25:18.337 INFO:tasks.rados.rados.0.smithi076.stdout:1476: finishing write tid 2 to smithi07694878-7 2021-03-15T22:25:18.338 INFO:tasks.rados.rados.0.smithi076.stdout:1475: done (2 left) 2021-03-15T22:25:18.338 INFO:tasks.rados.rados.0.smithi076.stdout:1478: read oid 13 snap -1 2021-03-15T22:25:18.338 INFO:tasks.rados.rados.0.smithi076.stdout:1478: expect deleted 2021-03-15T22:25:18.338 INFO:tasks.rados.rados.0.smithi076.stdout:1479: read oid 38 snap -1 2021-03-15T22:25:18.339 INFO:tasks.rados.rados.0.smithi076.stdout:1479: expect deleted 2021-03-15T22:25:18.339 INFO:tasks.rados.rados.0.smithi076.stdout:1480: rollback oid 43 current snap is 159 2021-03-15T22:25:18.339 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 43 to 145 2021-03-15T22:25:18.339 INFO:tasks.rados.rados.0.smithi076.stdout:1476: finishing write tid 3 to smithi07694878-7 2021-03-15T22:25:18.339 INFO:tasks.rados.rados.0.smithi076.stdout:1476: finishing write tid 4 to smithi07694878-7 2021-03-15T22:25:18.340 INFO:tasks.rados.rados.0.smithi076.stdout:1476: finishing write tid 5 to smithi07694878-7 2021-03-15T22:25:18.340 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 521 (ObjNum 526 snap 159 seq_num 526) dirty exists 2021-03-15T22:25:18.340 INFO:tasks.rados.rados.0.smithi076.stdout:1476: left oid 7 (ObjNum 526 snap 159 seq_num 526) 2021-03-15T22:25:18.340 INFO:tasks.rados.rados.0.smithi076.stdout:1477: finishing copy_from to smithi07694878-6 2021-03-15T22:25:18.340 INFO:tasks.rados.rados.0.smithi076.stdout:1477: got expected ENOENT (src dne) 2021-03-15T22:25:18.341 INFO:tasks.rados.rados.0.smithi076.stdout:1477: finishing copy_from racing read to smithi07694878-6 2021-03-15T22:25:18.341 INFO:tasks.rados.rados.0.smithi076.stdout:1476: done (4 left) 2021-03-15T22:25:18.341 INFO:tasks.rados.rados.0.smithi076.stdout:1477: done (3 left) 2021-03-15T22:25:18.341 INFO:tasks.rados.rados.0.smithi076.stdout:1481: write oid 45 current snap is 159 2021-03-15T22:25:18.341 INFO:tasks.rados.rados.0.smithi076.stdout:1481: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2021-03-15T22:25:18.342 INFO:tasks.rados.rados.0.smithi076.stdout:1481: writing smithi07694878-45 from 457289 to 941869 tid 1 2021-03-15T22:25:18.348 INFO:tasks.rados.rados.0.smithi076.stdout:1481: writing smithi07694878-45 from 1438834 to 1987673 tid 2 2021-03-15T22:25:18.353 INFO:tasks.rados.rados.0.smithi076.stdout:1481: writing smithi07694878-45 from 2772623 to 2857289 tid 3 2021-03-15T22:25:18.353 INFO:tasks.rados.rados.0.smithi076.stdout:1482: copy_from oid 42 from oid 9 current snap is 159 2021-03-15T22:25:18.354 INFO:tasks.rados.rados.0.smithi076.stdout:1480: finishing rollback tid 0 to smithi07694878-43 2021-03-15T22:25:18.354 INFO:tasks.rados.rados.0.smithi076.stdout:1480: finishing rollback tid 1 to smithi07694878-43 2021-03-15T22:25:18.354 INFO:tasks.rados.rados.0.smithi076.stdout:1480: finishing rollback tid 2 to smithi07694878-43 2021-03-15T22:25:18.354 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 762 (ObjNum 392 snap 106 seq_num 392) dirty exists 2021-03-15T22:25:18.355 INFO:tasks.rados.rados.0.smithi076.stdout:1481: finishing write tid 1 to smithi07694878-45 2021-03-15T22:25:18.355 INFO:tasks.rados.rados.0.smithi076.stdout:1481: finishing write tid 2 to smithi07694878-45 2021-03-15T22:25:18.355 INFO:tasks.rados.rados.0.smithi076.stdout:1478: done (4 left) 2021-03-15T22:25:18.358 INFO:tasks.rados.rados.0.smithi076.stdout:1479: done (3 left) 2021-03-15T22:25:18.358 INFO:tasks.rados.rados.0.smithi076.stdout:1480: done (2 left) 2021-03-15T22:25:18.359 INFO:tasks.rados.rados.0.smithi076.stdout:1483: delete oid 34 current snap is 159 2021-03-15T22:25:18.360 INFO:tasks.rados.rados.0.smithi076.stdout:1481: finishing write tid 3 to smithi07694878-45 2021-03-15T22:25:18.360 INFO:tasks.rados.rados.0.smithi076.stdout:1481: finishing write tid 4 to smithi07694878-45 2021-03-15T22:25:18.360 INFO:tasks.rados.rados.0.smithi076.stdout:1483: done (2 left) 2021-03-15T22:25:18.360 INFO:tasks.rados.rados.0.smithi076.stdout:1484: snap_create 2021-03-15T22:25:18.360 INFO:tasks.rados.rados.0.smithi076.stdout:1481: finishing write tid 5 to smithi07694878-45 2021-03-15T22:25:18.361 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 287 (ObjNum 527 snap 159 seq_num 527) dirty exists 2021-03-15T22:25:18.361 INFO:tasks.rados.rados.0.smithi076.stdout:1481: left oid 45 (ObjNum 527 snap 159 seq_num 527) 2021-03-15T22:25:18.375 INFO:tasks.rados.rados.0.smithi076.stdout:1482: finishing copy_from to smithi07694878-42 2021-03-15T22:25:18.376 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 695 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:25:18.376 INFO:tasks.rados.rados.0.smithi076.stdout:1482: finishing copy_from racing read to smithi07694878-42 2021-03-15T22:25:18.408 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:18.431 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:18.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:18.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:18.534 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:18.555 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:18.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:18.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:18.659 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:18.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:18.686 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:18.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:18.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:18 smithi076 conmon[56514]: cluster 2021-03-15T22:25:16.834405+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:25:18.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:18 smithi076 conmon[56514]: cluster 2021-03-15T22:25:17.050929+0000 mgr.y (mgr.14144) 609 : cluster [DBG] pgmap v729: 138 pgs: 1 remapped+peering, 137 active+clean; 281 MiB data, 871 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.5 MiB/s wr, 6 op/s 2021-03-15T22:25:18.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:18 smithi076 conmon[56514]: cluster 2021-03-15T22:25:18.109866+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2021-03-15T22:25:18.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:18 smithi076 conmon[62156]: cluster 2021-03-15T22:25:16.834405+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:25:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:18 smithi076 conmon[62156]: cluster 2021-03-15T22:25:17.050929+0000 mgr.y (mgr.14144) 609 : cluster [DBG] pgmap v729: 138 pgs: 1 remapped+peering, 137 active+clean; 281 MiB data, 871 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.5 MiB/s wr, 6 op/s 2021-03-15T22:25:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:18 smithi076 conmon[62156]: cluster 2021-03-15T22:25:18.109866+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2021-03-15T22:25:18.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:18 smithi125 conmon[56739]: cluster 2021-03-15T22:25:16.834405+0000 osd.3 (osd.3) 72 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:25:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:18 smithi125 conmon[56739]: cluster 2021-03-15T22:25:17.050929+0000 mgr.y (mgr.14144) 609 : cluster [DBG] pgmap v729: 138 pgs: 1 remapped+peering, 137 active+clean; 281 MiB data, 871 MiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.5 MiB/s wr, 6 op/s 2021-03-15T22:25:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:18 smithi125 conmon[56739]: cluster 2021-03-15T22:25:18.109866+0000 mon.a (mon.0) 1208 : cluster [DBG] osdmap e403: 8 total, 8 up, 7 in 2021-03-15T22:25:18.787 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:18.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:18.811 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:18.811 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:18.912 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:18.936 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:18.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:18.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:19.039 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:19.067 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:19.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:19.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:19.113 INFO:tasks.rados.rados.0.smithi076.stdout:1481: done (2 left) 2021-03-15T22:25:19.114 INFO:tasks.rados.rados.0.smithi076.stdout:1482: done (1 left) 2021-03-15T22:25:19.114 INFO:tasks.rados.rados.0.smithi076.stdout:1484: done (0 left) 2021-03-15T22:25:19.114 INFO:tasks.rados.rados.0.smithi076.stdout:1485: rollback oid 30 current snap is 160 2021-03-15T22:25:19.114 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 30 to 132 2021-03-15T22:25:19.115 INFO:tasks.rados.rados.0.smithi076.stdout:1486: snap_create 2021-03-15T22:25:19.119 INFO:tasks.rados.rados.0.smithi076.stdout:1485: finishing rollback tid 0 to smithi07694878-30 2021-03-15T22:25:19.120 INFO:tasks.rados.rados.0.smithi076.stdout:1485: finishing rollback tid 1 to smithi07694878-30 2021-03-15T22:25:19.120 INFO:tasks.rados.rados.0.smithi076.stdout:1485: finishing rollback tid 2 to smithi07694878-30 2021-03-15T22:25:19.121 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 898 (ObjNum 445 snap 126 seq_num 445) dirty exists 2021-03-15T22:25:19.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:19.196 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:19.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:19.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:19.299 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:19.322 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:19.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:19.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:19.424 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:25:19.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:19.441 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:25:19.452 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:19.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:19.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:19.460 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:25:19.477 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:25:19.497 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:25:19.516 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:25:19.534 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:25:19.551 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:25:19.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:25:19.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:25:19.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:25:19.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:25:19.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:25:19.554 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:25:19.554 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:25:19.554 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:25:19.555 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:19.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:19.580 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:19.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:19.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:19.689 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:25:19.690 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:25:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:19.705 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:19.705 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:19.807 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:19.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:19.834 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:19.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:19.937 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:19.961 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:19.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:19.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:20.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:25:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:25:20.063 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:20.085 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:20.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:20.087 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:20.099 INFO:tasks.rados.rados.0.smithi076.stdout:1485: done (1 left) 2021-03-15T22:25:20.099 INFO:tasks.rados.rados.0.smithi076.stdout:1486: done (0 left) 2021-03-15T22:25:20.099 INFO:tasks.rados.rados.0.smithi076.stdout:1487: read oid 34 snap -1 2021-03-15T22:25:20.100 INFO:tasks.rados.rados.0.smithi076.stdout:1487: expect deleted 2021-03-15T22:25:20.101 INFO:tasks.rados.rados.0.smithi076.stdout:1488: write (excl) oid 12 current snap is 161 2021-03-15T22:25:20.101 INFO:tasks.rados.rados.0.smithi076.stdout:1488: seq_num 529 ranges {490903=635078,1839445=552508} 2021-03-15T22:25:20.103 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.6 to 1 (10000) 2021-03-15T22:25:20.104 INFO:tasks.rados.rados.0.smithi076.stdout:1488: writing smithi07694878-12 from 490903 to 1125981 tid 1 2021-03-15T22:25:20.112 INFO:tasks.rados.rados.0.smithi076.stdout:1488: writing smithi07694878-12 from 1839445 to 2391953 tid 2 2021-03-15T22:25:20.113 INFO:tasks.rados.rados.0.smithi076.stdout:1488: finishing write tid 1 to smithi07694878-12 2021-03-15T22:25:20.113 INFO:tasks.rados.rados.0.smithi076.stdout:1487: done (1 left) 2021-03-15T22:25:20.113 INFO:tasks.rados.rados.0.smithi076.stdout:1489: delete oid 5 current snap is 161 2021-03-15T22:25:20.114 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: cluster 2021-03-15T22:25:19.051289+0000 mgr.y 2021-03-15T22:25:20.114 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: (mgr.14144) 610 : cluster [DBG] pgmap v732: 138 pgs: 1 remapped+peering, 137 active+clean; 286 MiB data, 890 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.3 MiB/s wr, 7 op/s; 3.4 MiB/s, 11 objects/s recovering 2021-03-15T22:25:20.115 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: cluster 2021-03-15T22:25:20.115 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: 2021-03-15T22:25:19.108425+0000 mon.a (mon.0) 1209 : cluster 2021-03-15T22:25:20.115 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: [DBG] osdmap e404: 8 total, 8 up, 7 in 2021-03-15T22:25:20.115 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: audit 2021-03-15T22:25:19.172385 2021-03-15T22:25:20.115 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1210 : audit [INF] from='client.? 172.21.15.76:0/1991061576' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-15T22:25:20.116 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: 2021-03-15T22:25:20.116 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:25:20.116 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: 03-15T22:25:19.404670+0000 mon.a (mon.0) 1211 2021-03-15T22:25:20.117 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: : audit [DBG] from='client.? 172.21.15.76:0/416826839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:20.117 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: audit 2021-03-15T22:25: 2021-03-15T22:25:20.117 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: 19.698284+0000 mgr.y (mgr.14144) 611 : 2021-03-15T22:25:20.117 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:20.117 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: audit 2021-03-15T22:25 2021-03-15T22:25:20.118 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: :20.096533+0000 mon.a (mon.0) 1212 2021-03-15T22:25:20.118 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/1991061576' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-15T22:25:20.118 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: cluster 2021-03-15T22:25 2021-03-15T22:25:20.118 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: :20.096604+0000 mon.a (mon.0) 1213 : 2021-03-15T22:25:20.119 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[62156]: cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2021-03-15T22:25:20.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[56514]: cluster 2021-03-15T22:25:19.051289+0000 mgr.y (mgr.14144) 610 : cluster [DBG] pgmap v732: 138 pgs: 1 remapped+peering, 137 active+clean; 286 MiB data, 890 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.3 MiB/s wr, 7 op/s; 3.4 MiB/s, 11 objects/s recovering 2021-03-15T22:25:20.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[56514]: cluster 2021-03-15T22:25:19.108425+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e404: 8 total, 8 up, 7 in 2021-03-15T22:25:20.119 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[56514]: audit 2021-03-15T22:25:19.172385+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? 172.21.15.76:0/1991061576' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-15T22:25:20.120 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[56514]: audit 2021-03-15T22:25:19.404670+0000 mon.a (mon.0) 1211 : audit [DBG] from='client.? 172.21.15.76:0/416826839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:20.120 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[56514]: audit 2021-03-15T22:25:19.698284+0000 mgr.y (mgr.14144) 611 : audit [DBG] 2021-03-15T22:25:20.120 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[56514]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:20.120 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[56514]: audit 2021-03-15T22:25:20.096533+0000 mon.a (mon.0) 1212 : audit [INF] from='client.? 172.21.15.76:0/1991061576' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-15T22:25:20.121 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:20 smithi076 conmon[56514]: cluster 2021-03-15T22:25:20.096604+0000 mon.a (mon.0) 1213 : cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2021-03-15T22:25:20.122 INFO:tasks.rados.rados.0.smithi076.stdout:1488: finishing write tid 2 to smithi07694878-12 2021-03-15T22:25:20.122 INFO:tasks.rados.rados.0.smithi076.stdout:1488: finishing write tid 3 to smithi07694878-12 2021-03-15T22:25:20.122 INFO:tasks.rados.rados.0.smithi076.stdout:1488: finishing write tid 4 to smithi07694878-12 2021-03-15T22:25:20.122 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 901 (ObjNum 529 snap 161 seq_num 529) dirty exists 2021-03-15T22:25:20.123 INFO:tasks.rados.rados.0.smithi076.stdout:1488: left oid 12 (ObjNum 529 snap 161 seq_num 529) 2021-03-15T22:25:20.123 INFO:tasks.rados.rados.0.smithi076.stdout:1488: done (1 left) 2021-03-15T22:25:20.123 INFO:tasks.rados.rados.0.smithi076.stdout:1489: done (0 left) 2021-03-15T22:25:20.123 INFO:tasks.rados.rados.0.smithi076.stdout:1490: read oid 19 snap 147 2021-03-15T22:25:20.124 INFO:tasks.rados.rados.0.smithi076.stdout:1490: expect deleted 2021-03-15T22:25:20.124 INFO:tasks.rados.rados.0.smithi076.stdout:1491: delete oid 17 current snap is 161 2021-03-15T22:25:20.124 INFO:tasks.rados.rados.0.smithi076.stdout:1490: done (1 left) 2021-03-15T22:25:20.124 INFO:tasks.rados.rados.0.smithi076.stdout:1491: done (0 left) 2021-03-15T22:25:20.124 INFO:tasks.rados.rados.0.smithi076.stdout:1492: snap_create 2021-03-15T22:25:20.189 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:20.212 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:20.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:20.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:20.315 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:20.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:20.339 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:20.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:20.441 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:20.463 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:20.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:20.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:20.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: cluster 2021-03-15T22:25:19.051289+0000 mgr.y (mgr.14144) 610 2021-03-15T22:25:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: : cluster [DBG] pgmap v732: 138 pgs: 1 remapped+peering, 137 active+clean; 286 MiB data, 890 MiB used, 625 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.3 MiB/s wr, 7 op/s; 3.4 MiB/s, 11 objects/s recovering 2021-03-15T22:25:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: cluster 2021-03-15T22:25:19.108425+0000 mon.a (mon.0) 1209 : cluster [DBG] osdmap e404: 8 total, 8 up, 7 in 2021-03-15T22:25:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: audit 2021-03-15T22:25: 2021-03-15T22:25:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: 19.172385+0000 mon.a (mon.0) 1210 : audit [INF] from='client.? 172.21.15.76:0/1991061576' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]: dispatch 2021-03-15T22:25:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: audit 2021-03-15T22:25:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: 2021-03-15T22:25:19.404670+0000 mon.a (mon.0) 1211 : 2021-03-15T22:25:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: audit [DBG] from='client.? 172.21.15.76:0/416826839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:25:20.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: T22:25:19.698284+0000 mgr.y (mgr. 2021-03-15T22:25:20.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: 14144) 611 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:20.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: audit 2021 2021-03-15T22:25:20.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: -03-15T22:25:20.096533+0000 mon.a 2021-03-15T22:25:20.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: (mon.0) 1212 : audit [INF] from='client.? 172.21.15.76:0/1991061576' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 6, "weight": 1.0}]': finished 2021-03-15T22:25:20.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:25:20.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: 15T22:25:20.096604+0000 mon.a (mon.0) 1213 2021-03-15T22:25:20.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:20 smithi125 conmon[56739]: : cluster [DBG] osdmap e405: 8 total, 8 up, 7 in 2021-03-15T22:25:20.566 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:20.588 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:20.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:20.691 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:20.717 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:20.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:20.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:20.841 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:20.843 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:20.945 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:20.967 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:20.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:20.969 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:21.041 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:25:21.070 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:21.109 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:21.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:21.111 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:21.119 INFO:tasks.rados.rados.0.smithi076.stdout:1492: done (0 left) 2021-03-15T22:25:21.119 INFO:tasks.rados.rados.0.smithi076.stdout:1493: snap_create 2021-03-15T22:25:21.186 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 7 1 2021-03-15T22:25:21.212 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:21.218 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-15T22:25:21.238 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:21.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:21.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:21.340 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:21.358 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:21.366 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:21.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:21.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:21.469 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:21.493 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:21.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:21.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:21.596 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:21.624 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:21.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:21.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:21.727 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:21.748 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:21.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:21.750 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:21.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:21.859 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:21.859 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:21.877 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:21.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:21.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:21.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:22.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:22.004 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:22.005 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:22.106 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:22.124 INFO:tasks.rados.rados.0.smithi076.stdout:1493: done (0 left) 2021-03-15T22:25:22.124 INFO:tasks.rados.rados.0.smithi076.stdout:1494: read oid 31 snap -1 2021-03-15T22:25:22.124 INFO:tasks.rados.rados.0.smithi076.stdout:1494: expect (ObjNum 516 snap 158 seq_num 516) 2021-03-15T22:25:22.124 INFO:tasks.rados.rados.0.smithi076.stdout:1495: copy_from oid 2 from oid 26 current snap is 163 2021-03-15T22:25:22.125 INFO:tasks.rados.rados.0.smithi076.stdout:1496: read oid 3 snap -1 2021-03-15T22:25:22.125 INFO:tasks.rados.rados.0.smithi076.stdout:1496: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:25:22.125 INFO:tasks.rados.rados.0.smithi076.stdout:1497: read oid 46 snap -1 2021-03-15T22:25:22.125 INFO:tasks.rados.rados.0.smithi076.stdout:1497: expect (ObjNum 469 snap 137 seq_num 469) 2021-03-15T22:25:22.126 INFO:tasks.rados.rados.0.smithi076.stdout:1498: rollback oid 29 current snap is 163 2021-03-15T22:25:22.126 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 29 to 140 2021-03-15T22:25:22.126 INFO:tasks.rados.rados.0.smithi076.stdout:1499: snap_remove snap 158 2021-03-15T22:25:22.136 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:22.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:22.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:22.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:22.264 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:22.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:22.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:22.367 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:22.392 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:22.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:22.394 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:22.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:22 smithi076 conmon[56514]: cluster 2021-03-15T22:25:21.051698+0000 mgr.y 2021-03-15T22:25:22.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:22 smithi076 conmon[56514]: (mgr.14144) 612 : cluster [DBG] pgmap v735: 138 pgs: 138 active+clean; 293 MiB data, 895 MiB used, 625 GiB / 626 GiB avail; 5.9 MiB/s rd, 2.0 MiB/s wr, 14 op/s; 4.6 MiB/s, 20 objects/s recovering 2021-03-15T22:25:22.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:22 smithi076 conmon[56514]: cluster 2021-03-15T22:25:21.115535+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2021-03-15T22:25:22.435 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:22 smithi076 conmon[62156]: cluster 2021-03-15T22:25:21.051698+0000 mgr.y (mgr.14144) 2021-03-15T22:25:22.435 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:22 smithi076 conmon[62156]: 612 : cluster [DBG] pgmap v735: 138 pgs: 138 active+clean; 293 MiB data, 895 MiB used, 625 GiB / 626 GiB avail; 5.9 MiB/s rd, 2.0 MiB/s wr, 14 op/s; 4.6 MiB/s, 20 objects/s recovering 2021-03-15T22:25:22.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:22 smithi076 conmon[62156]: cluster 2021-03-15T22:25:21.115535+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2021-03-15T22:25:22.495 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:22.518 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:22.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:22.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:22 smithi125 conmon[56739]: cluster 2021-03-15T22:25:21.051698+0000 mgr.y (mgr.14144) 612 : cluster [DBG] pgmap v735: 138 pgs: 138 active+clean; 293 MiB data, 895 MiB used, 625 GiB / 626 GiB avail; 5.9 MiB/s rd, 2.0 MiB/s wr, 14 op/s; 4.6 MiB/s, 20 objects/s recovering 2021-03-15T22:25:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:22 smithi125 conmon[56739]: 2021-03-15T22:25:22.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:22 smithi125 conmon[56739]: cluster 2021-03-15T22:25:21.115535+0000 mon.a (mon.0) 1214 : cluster [DBG] osdmap e406: 8 total, 8 up, 7 in 2021-03-15T22:25:22.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:22.646 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:22.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:22.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:22.670 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:25:22.749 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:22.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:22.775 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:22.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:22.877 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:22.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:22.903 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:22.904 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:23.005 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:23.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:23.030 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:23.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:23.131 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:23.139 INFO:tasks.rados.rados.0.smithi076.stdout:1499: done (5 left) 2021-03-15T22:25:23.139 INFO:tasks.rados.rados.0.smithi076.stdout:1500: snap_create 2021-03-15T22:25:23.139 INFO:tasks.rados.rados.0.smithi076.stdout:1496: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:25:23.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:23.162 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:23.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:23.234 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:23 smithi076 conmon[56514]: cluster 2021-03-15T22:25:22.120973+0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2021-03-15T22:25:23.235 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:23 smithi076 conmon[62156]: cluster 2021-03-15T22:25:22.120973 2021-03-15T22:25:23.235 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:23 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1215 : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2021-03-15T22:25:23.250 INFO:tasks.rados.rados.0.smithi076.stdout:1495: finishing copy_from to smithi07694878-2 2021-03-15T22:25:23.251 INFO:tasks.rados.rados.0.smithi076.stdout:1495: got expected ENOENT (src dne) 2021-03-15T22:25:23.251 INFO:tasks.rados.rados.0.smithi076.stdout:1495: finishing copy_from racing read to smithi07694878-2 2021-03-15T22:25:23.251 INFO:tasks.rados.rados.0.smithi076.stdout:1498: finishing rollback tid 0 to smithi07694878-29 2021-03-15T22:25:23.251 INFO:tasks.rados.rados.0.smithi076.stdout:1498: finishing rollback tid 1 to smithi07694878-29 2021-03-15T22:25:23.252 INFO:tasks.rados.rados.0.smithi076.stdout:1498: finishing rollback tid 2 to smithi07694878-29 2021-03-15T22:25:23.252 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 924 (ObjNum 397 snap 108 seq_num 397) dirty exists 2021-03-15T22:25:23.252 INFO:tasks.rados.rados.0.smithi076.stdout:1497: expect (ObjNum 469 snap 137 seq_num 469) 2021-03-15T22:25:23.264 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:23.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:23.288 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:23.288 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:23.390 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:23.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:23.412 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:23.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:23.431 INFO:tasks.rados.rados.0.smithi076.stdout:1494: expect (ObjNum 516 snap 158 seq_num 516) 2021-03-15T22:25:23.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:23.536 INFO:journalctl@ceph.osd.4.smithi125.stdout:Mar 15 22:25:23 smithi125 conmon[60581]: debug 2021-03-15T22:25:23.126+0000 7fe6a739a700 -1 osd.4 pg_epoch: 408 pg[7.4( v 403'334 (0'0,403'334] local-lis/les=358/359 n=9 ec=91/91 lis/c=358/358 les/c/f=359/359/0 sis=358) [4,1,6] r=0 lpr=358 crt=403'334 lcod 401'332 mlcod 401'332 active+clean+snaptrim trimq=[12d~1] ps=[10b~1,129~1]] removing snap head 2021-03-15T22:25:23.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:23 smithi125 conmon[56739]: cluster 2021-03-15T22:25:22.120973+0000 mon.a (mon.0) 1215 2021-03-15T22:25:23.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:23 smithi125 conmon[56739]: : cluster [DBG] osdmap e407: 8 total, 8 up, 7 in 2021-03-15T22:25:23.540 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:23.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:23.643 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:23.669 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:23.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:23.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:23.772 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:23.794 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:23.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:23.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:23.898 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:23.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:23.921 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:23.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:24.023 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:24.051 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:24.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:24.133 INFO:tasks.rados.rados.0.smithi076.stdout:1494: done (5 left) 2021-03-15T22:25:24.133 INFO:tasks.rados.rados.0.smithi076.stdout:1495: done (4 left) 2021-03-15T22:25:24.133 INFO:tasks.rados.rados.0.smithi076.stdout:1496: done (3 left) 2021-03-15T22:25:24.134 INFO:tasks.rados.rados.0.smithi076.stdout:1497: done (2 left) 2021-03-15T22:25:24.134 INFO:tasks.rados.rados.0.smithi076.stdout:1498: done (1 left) 2021-03-15T22:25:24.134 INFO:tasks.rados.rados.0.smithi076.stdout:1500: done (0 left) 2021-03-15T22:25:24.134 INFO:tasks.rados.rados.0.smithi076.stdout:1501: copy_from oid 21 from oid 46 current snap is 164 2021-03-15T22:25:24.134 INFO:tasks.rados.rados.0.smithi076.stdout:1502: snap_create 2021-03-15T22:25:24.135 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:25:24.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:24.157 INFO:tasks.rados.rados.0.smithi076.stdout:1501: finishing copy_from to smithi07694878-21 2021-03-15T22:25:24.158 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 203 (ObjNum 469 snap 137 seq_num 469) dirty exists 2021-03-15T22:25:24.159 INFO:tasks.rados.rados.0.smithi076.stdout:1501: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:25:24.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:24.179 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:24.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:24.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:24 smithi076 conmon[56514]: cluster 2021-03-15T22:25:23 2021-03-15T22:25:24.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:24 smithi076 conmon[56514]: .052106+0000 mgr.y (mgr.14144) 613 : cluster [DBG] pgmap v738: 138 pgs: 138 active+clean; 307 MiB data, 912 MiB used, 625 GiB / 626 GiB avail; 9.2 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 1.2 MiB/s, 8 objects/s recovering 2021-03-15T22:25:24.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:24 smithi076 conmon[56514]: cluster 2021-03-15T22:25:23.130839+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2021-03-15T22:25:24.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:24 smithi076 conmon[56514]: audit 2021-03-15T22:25:23.334770+0000 mon.c (mon.1) 36 : audit [INF] from='client.? 172.21.15.76:0/2936386726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:24.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:24 smithi076 conmon[56514]: audit 2021-03-15T22:25:23.335020+0000 mon.a (mon.0) 1217 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:24.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:24 smithi076 conmon[62156]: cluster 2021-03-15T22:25:23.052106+0000 mgr.y (mgr. 2021-03-15T22:25:24.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:24 smithi076 conmon[62156]: 14144) 613 : cluster [DBG] pgmap v738: 138 pgs: 138 active+clean; 307 MiB data, 912 MiB used, 625 GiB / 626 GiB avail; 9.2 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 1.2 MiB/s, 8 objects/s recovering 2021-03-15T22:25:24.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:24 smithi076 conmon[62156]: cluster 2021-03-15T22:25:23.130839+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2021-03-15T22:25:24.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:24 smithi076 conmon[62156]: audit 2021-03-15T22:25:23.334770+0000 mon.c (mon.1) 36 : audit [INF] from='client.? 172.21.15.76:0/2936386726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:24.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:24 smithi076 conmon[62156]: audit 2021-03-15T22:25:23.335020+0000 mon.a (mon.0) 1217 : 2021-03-15T22:25:24.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:24 smithi076 conmon[62156]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:24.280 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:24.304 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:24.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:24.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:24.407 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:24.430 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:24.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:24.432 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:24.533 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:24.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:24 smithi125 conmon[56739]: cluster 2021-03-15T22:25:23.052106+0000 mgr.y (mgr.14144) 2021-03-15T22:25:24.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:24 smithi125 conmon[56739]: 613 : cluster [DBG] pgmap v738: 138 pgs: 138 active+clean; 307 MiB data, 912 MiB used, 625 GiB / 626 GiB avail; 9.2 MiB/s rd, 2.5 MiB/s wr, 19 op/s; 1.2 MiB/s, 8 objects/s recovering 2021-03-15T22:25:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:24 smithi125 conmon[56739]: cluster 2021-03-15T22:25:23.130839+0000 mon.a (mon.0) 1216 : cluster [DBG] osdmap e408: 8 total, 8 up, 7 in 2021-03-15T22:25:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:24 smithi125 conmon[56739]: audit 2021-03-15T22:25: 2021-03-15T22:25:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:24 smithi125 conmon[56739]: 23.334770+0000 mon.c (mon.1) 36 : audit [INF] from='client.? 172.21.15.76:0/2936386726' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:24 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:25:24.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:24 smithi125 conmon[56739]: -15T22:25:23.335020+0000 mon.a (mon.0) 1217 : audit 2021-03-15T22:25:24.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:24 smithi125 conmon[56739]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:24.558 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:24.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:24.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:24.661 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:24.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:24.684 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:24.684 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:24.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:24.811 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:24.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:24.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:24.914 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:24.940 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:24.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:24.942 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:25.043 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:25.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:25.070 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:25.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:25.100 INFO:tasks.rados.rados.0.smithi076.stdout:1501: done (1 left) 2021-03-15T22:25:25.101 INFO:tasks.rados.rados.0.smithi076.stdout:1502: done (0 left) 2021-03-15T22:25:25.101 INFO:tasks.rados.rados.0.smithi076.stdout:1503: read oid 24 snap -1 2021-03-15T22:25:25.101 INFO:tasks.rados.rados.0.smithi076.stdout:1503: expect (ObjNum 514 snap 155 seq_num 514) 2021-03-15T22:25:25.102 INFO:tasks.rados.rados.0.smithi076.stdout:1504: read oid 50 snap -1 2021-03-15T22:25:25.102 INFO:tasks.rados.rados.0.smithi076.stdout:1504: expect (ObjNum 518 snap 158 seq_num 518) 2021-03-15T22:25:25.103 INFO:tasks.rados.rados.0.smithi076.stdout:1505: delete oid 13 current snap is 165 2021-03-15T22:25:25.106 INFO:tasks.rados.rados.0.smithi076.stdout:1503: expect (ObjNum 514 snap 155 seq_num 514) 2021-03-15T22:25:25.172 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:25.200 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:25.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:25.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:25.286 INFO:tasks.rados.rados.0.smithi076.stdout:1504: expect (ObjNum 518 snap 158 seq_num 518) 2021-03-15T22:25:25.299 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:25.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:25.329 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:25.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:25.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:25.350 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:25:25.350 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:25:25.421 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[56514]: cluster 2021-03-15T22:25:24.124475+0000 mon.a (mon.0) 1218 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:25.421 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[56514]: audit 2021-03-15T22:25:24.128805+0000 mon.a (mon.0) 1219 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:25.422 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[56514]: cluster 2021-03-15T22:25:24.128876+0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e409: 8 total, 8 up, 7 in 2021-03-15T22:25:25.422 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[56514]: cluster 2021-03-15T22:25:25.095882+0000 mon.a (mon.0) 1221 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2021-03-15T22:25:25.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: cluster 2021-03-15T22:25:25.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: 2021-03-15T22:25:25.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: 2021-03-15T22:25:24.124475 2021-03-15T22:25:25.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1218 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:25.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: audit 2021-03-15T22:25:24 2021-03-15T22:25:25.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: .128805+0000 mon.a (mon.0) 1219 : audit 2021-03-15T22:25:25.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:25.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: cluster 2021-03-15T22:25:24 2021-03-15T22:25:25.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: .128876+0000 mon.a (mon.0) 1220 : cluster [DBG] 2021-03-15T22:25:25.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: osdmap e409: 8 total, 8 up, 7 in 2021-03-15T22:25:25.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:25:25.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: -03-15T22:25:25.095882+0000 mon.a (mon.0 2021-03-15T22:25:25.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:25 smithi076 conmon[62156]: ) 1221 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2021-03-15T22:25:25.434 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:25.459 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:25.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:25.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:25.487 INFO:tasks.rados.rados.0.smithi076.stdout:1503: done (2 left) 2021-03-15T22:25:25.487 INFO:tasks.rados.rados.0.smithi076.stdout:1504: done (1 left) 2021-03-15T22:25:25.487 INFO:tasks.rados.rados.0.smithi076.stdout:1505: done (0 left) 2021-03-15T22:25:25.487 INFO:tasks.rados.rados.0.smithi076.stdout:1506: snap_create 2021-03-15T22:25:25.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:25 smithi125 conmon[56739]: cluster 2021-03-15T22:25:24.124475+0000 mon.a (mon.0) 1218 : cluster [WRN] 2021-03-15T22:25:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:25 smithi125 conmon[56739]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:25.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:25 smithi125 conmon[56739]: audit 2021-03-15T22:25:24.128805+0000 mon.a (mon.0) 1219 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:25.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:25 smithi125 conmon[56739]: cluster 2021-03-15T22:25:24.128876 2021-03-15T22:25:25.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:25 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1220 : cluster [DBG] osdmap e409: 8 total, 8 up, 7 in 2021-03-15T22:25:25.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:25 smithi125 conmon[56739]: cluster 2021-03-15T22:25:25.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:25 smithi125 conmon[56739]: 2021-03-15T22:25:25.095882+0000 mon.a (mon. 2021-03-15T22:25:25.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:25 smithi125 conmon[56739]: 0) 1221 : cluster [DBG] osdmap e410: 8 total, 8 up, 7 in 2021-03-15T22:25:25.562 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:25.587 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:25.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:25.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:25.690 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:25.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:25.717 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:25.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:25.820 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:25.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:25.844 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:25.844 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:25.946 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:25.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:25.969 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:25.969 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:26.070 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:26.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:26.094 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:26.094 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:26.158 INFO:tasks.rados.rados.0.smithi076.stdout:1506: done (0 left) 2021-03-15T22:25:26.159 INFO:tasks.rados.rados.0.smithi076.stdout:1507: read oid 22 snap -1 2021-03-15T22:25:26.159 INFO:tasks.rados.rados.0.smithi076.stdout:1507: expect (ObjNum 466 snap 136 seq_num 466) 2021-03-15T22:25:26.159 INFO:tasks.rados.rados.0.smithi076.stdout:1508: read oid 23 snap -1 2021-03-15T22:25:26.160 INFO:tasks.rados.rados.0.smithi076.stdout:1508: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:25:26.160 INFO:tasks.rados.rados.0.smithi076.stdout:1509: read oid 30 snap 159 2021-03-15T22:25:26.160 INFO:tasks.rados.rados.0.smithi076.stdout:1509: expect (ObjNum 475 snap 140 seq_num 475) 2021-03-15T22:25:26.160 INFO:tasks.rados.rados.0.smithi076.stdout:1510: write (excl) oid 18 current snap is 166 2021-03-15T22:25:26.161 INFO:tasks.rados.rados.0.smithi076.stdout:1510: seq_num 533 ranges {558131=652427,1673373=559846,2820808=137323} 2021-03-15T22:25:26.161 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.7 to 1 (10000) 2021-03-15T22:25:26.164 INFO:tasks.rados.rados.0.smithi076.stdout:1510: writing smithi07694878-18 from 558131 to 1210558 tid 1 2021-03-15T22:25:26.170 INFO:tasks.rados.rados.0.smithi076.stdout:1510: writing smithi07694878-18 from 1673373 to 2233219 tid 2 2021-03-15T22:25:26.174 INFO:tasks.rados.rados.0.smithi076.stdout:1510: writing smithi07694878-18 from 2820808 to 2958131 tid 3 2021-03-15T22:25:26.175 INFO:tasks.rados.rados.0.smithi076.stdout:1511: delete oid 39 current snap is 166 2021-03-15T22:25:26.175 INFO:tasks.rados.rados.0.smithi076.stdout:1508: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:25:26.196 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:26.217 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:26.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:26.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:26.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:26.345 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:26.347 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:26.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:26 smithi076 conmon[56514]: cluster 2021-03-15T22:25:25.052708+0000 mgr.y (mgr.14144) 614 : cluster [DBG] pgmap v741: 138 pgs: 138 active+clean; 309 MiB data, 918 MiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2021-03-15T22:25:26.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:26 smithi076 conmon[56514]: cluster 2021-03-15T22:25:25.790843+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:26.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:26 smithi076 conmon[56514]: cluster 2021-03-15T22:25:25.793081+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:26.377 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:26 smithi076 conmon[56514]: audit 2021-03-15T22:25:25.980142+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? 172.21.15.76:0/3615049485' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:25:26.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:26 smithi076 conmon[62156]: cluster 2021-03-15T22:25:25.052708+0000 mgr.y (mgr.14144) 614 : cluster [DBG] pgmap v741: 138 pgs: 138 active+clean; 309 MiB data, 918 MiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2021-03-15T22:25:26.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:26 smithi076 conmon[62156]: cluster 2021-03-15T22:25:25.790843+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:26.378 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:26 smithi076 conmon[62156]: cluster 2021-03-15T22:25:25.793081+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:26.379 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:26 smithi076 conmon[62156]: audit 2021-03-15T22:25:25.980142+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? 172.21.15.76:0/3615049485' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:25:26.382 INFO:tasks.rados.rados.0.smithi076.stdout:1510: finishing write tid 1 to smithi07694878-18 2021-03-15T22:25:26.382 INFO:tasks.rados.rados.0.smithi076.stdout:1510: finishing write tid 2 to smithi07694878-18 2021-03-15T22:25:26.382 INFO:tasks.rados.rados.0.smithi076.stdout:1507: expect (ObjNum 466 snap 136 seq_num 466) 2021-03-15T22:25:26.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-15T22:25:26.448 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:26.474 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:26.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:26 smithi125 conmon[56739]: cluster 2021-03-15T22:25:25.052708+0000 mgr.y (mgr.14144) 614 : cluster [DBG] pgmap v741: 138 pgs: 138 active+clean; 309 MiB data, 918 MiB used, 625 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.9 MiB/s wr, 13 op/s 2021-03-15T22:25:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:26 smithi125 conmon[56739]: cluster 2021-03-15T22:25:25.790843+0000 osd.2 (osd.2) 69 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:26 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:26.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:26 smithi125 conmon[56739]: 03-15T22:25:25.793081+0000 osd.2 (osd.2) 70 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:26.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:26 smithi125 conmon[56739]: audit 2021-03-15T22:25: 2021-03-15T22:25:26.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:26 smithi125 conmon[56739]: 25.980142+0000 mon.a (mon.0) 1222 : audit [INF] from='client.? 172.21.15.76:0/3615049485' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:25:26.576 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:26.600 INFO:tasks.rados.rados.0.smithi076.stdout:1510: finishing write tid 3 to smithi07694878-18 2021-03-15T22:25:26.600 INFO:tasks.rados.rados.0.smithi076.stdout:1510: finishing write tid 4 to smithi07694878-18 2021-03-15T22:25:26.600 INFO:tasks.rados.rados.0.smithi076.stdout:1507: done (4 left) 2021-03-15T22:25:26.600 INFO:tasks.rados.rados.0.smithi076.stdout:1508: done (3 left) 2021-03-15T22:25:26.601 INFO:tasks.rados.rados.0.smithi076.stdout:1511: done (2 left) 2021-03-15T22:25:26.601 INFO:tasks.rados.rados.0.smithi076.stdout:1512: snap_remove snap 144 2021-03-15T22:25:26.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:26.603 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:26.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:26.704 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:26.731 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:26.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:26.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:26.833 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:26.856 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:26.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:26.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:26.959 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:26.985 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:26.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:26.987 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:27.088 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:27.116 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:27.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:27.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:27.168 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:25:27.169 INFO:tasks.rados.rados.0.smithi076.stdout:1512: done (2 left) 2021-03-15T22:25:27.170 INFO:tasks.rados.rados.0.smithi076.stdout:1513: rollback oid 28 current snap is 166 2021-03-15T22:25:27.170 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 165 2021-03-15T22:25:27.170 INFO:tasks.rados.rados.0.smithi076.stdout:1514: snap_remove snap 145 2021-03-15T22:25:27.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 3 1 2021-03-15T22:25:27.220 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:27.243 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:27.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:27.244 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:27.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:27 smithi076 conmon[62156]: audit 2021-03-15T22:25:26.156371+0000 mon.a (mon.0) 1223 : audit [INF] from='client.? 172.21.15.76:0/3615049485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:25:27.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:27 smithi076 conmon[62156]: cluster 2021-03-15T22:25:26.156461+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2021-03-15T22:25:27.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:27 smithi076 conmon[56514]: audit 2021-03-15T22:25:27.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:27 smithi076 conmon[56514]: 2021-03-15T22:25 2021-03-15T22:25:27.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:27 smithi076 conmon[56514]: :26.156371+0000 mon.a (mon.0) 1223 : audit [INF] from='client.? 172.21.15.76:0/3615049485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:25:27.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:27 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:25:27.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:27 smithi076 conmon[56514]: :25:26.156461+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2021-03-15T22:25:27.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:27.367 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:27.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:27.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:27.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:27.496 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:27.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:27.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:27.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:27 smithi125 conmon[56739]: audit 2021-03-15T22:25:26.156371+0000 mon.a (mon.0) 2021-03-15T22:25:27.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:27 smithi125 conmon[56739]: 1223 : audit [INF] from='client.? 172.21.15.76:0/3615049485' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:25:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:27 smithi125 conmon[56739]: cluster 2021-03-15T22:25:26.156461+0000 mon.a (mon.0) 1224 : cluster [DBG] osdmap e411: 8 total, 8 up, 7 in 2021-03-15T22:25:27.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:27.624 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:27.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:27.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:27.680 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:27.707 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:27.726 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:27.757 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:27.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:27.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:27.759 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:27.861 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:27.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:27.886 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:27.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:27.988 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:28.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:28.012 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:28.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:28.114 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:28.136 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:28.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:28.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:28.167 INFO:tasks.rados.rados.0.smithi076.stdout:1514: done (3 left) 2021-03-15T22:25:28.168 INFO:tasks.rados.rados.0.smithi076.stdout:1515: rollback oid 7 current snap is 166 2021-03-15T22:25:28.168 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 7 to 143 2021-03-15T22:25:28.168 INFO:tasks.rados.rados.0.smithi076.stdout:1510: finishing write tid 5 to smithi07694878-18 2021-03-15T22:25:28.169 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 766 (ObjNum 533 snap 166 seq_num 533) dirty exists 2021-03-15T22:25:28.169 INFO:tasks.rados.rados.0.smithi076.stdout:1510: left oid 18 (ObjNum 533 snap 166 seq_num 533) 2021-03-15T22:25:28.169 INFO:tasks.rados.rados.0.smithi076.stdout:1509: expect (ObjNum 475 snap 140 seq_num 475) 2021-03-15T22:25:28.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:28.262 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:28.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:28.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:28 smithi076 conmon[56514]: cluster 2021-03-15T22:25:27.053166+0000 mgr.y (mgr. 2021-03-15T22:25:28.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:28 smithi076 conmon[56514]: 14144) 615 : cluster [DBG] pgmap v744: 138 pgs: 138 active+clean; 307 MiB data, 913 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 290 KiB/s wr, 11 op/s 2021-03-15T22:25:28.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:28 smithi076 conmon[56514]: cluster 2021-03-15T22:25:27.164597+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2021-03-15T22:25:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:28 smithi076 conmon[62156]: cluster 2021-03-15T22:25:27. 2021-03-15T22:25:28.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:28 smithi076 conmon[62156]: 053166+0000 mgr.y (mgr.14144) 615 : cluster [DBG] pgmap v744: 138 pgs: 138 active+clean; 307 MiB data, 913 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 290 KiB/s wr, 11 op/s 2021-03-15T22:25:28.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:28 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:25:28.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:28 smithi076 conmon[62156]: 22:25:27.164597+0000 mon.a (mon. 2021-03-15T22:25:28.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:28 smithi076 conmon[62156]: 0) 1225 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2021-03-15T22:25:28.366 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:28.391 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:28.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:28.418 INFO:tasks.rados.rados.0.smithi076.stdout:1513: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:25:28.418 INFO:tasks.rados.rados.0.smithi076.stdout:1513: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:25:28.418 INFO:tasks.rados.rados.0.smithi076.stdout:1513: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:25:28.418 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 936 (ObjNum 509 snap 152 seq_num 509) dirty exists 2021-03-15T22:25:28.419 INFO:tasks.rados.rados.0.smithi076.stdout:1509: done (3 left) 2021-03-15T22:25:28.419 INFO:tasks.rados.rados.0.smithi076.stdout:1510: done (2 left) 2021-03-15T22:25:28.419 INFO:tasks.rados.rados.0.smithi076.stdout:1513: done (1 left) 2021-03-15T22:25:28.419 INFO:tasks.rados.rados.0.smithi076.stdout:1516: snap_create 2021-03-15T22:25:28.420 INFO:tasks.rados.rados.0.smithi076.stdout:1515: finishing rollback tid 0 to smithi07694878-7 2021-03-15T22:25:28.420 INFO:tasks.rados.rados.0.smithi076.stdout:1515: finishing rollback tid 1 to smithi07694878-7 2021-03-15T22:25:28.420 INFO:tasks.rados.rados.0.smithi076.stdout:1515: finishing rollback tid 2 to smithi07694878-7 2021-03-15T22:25:28.420 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 531 (ObjNum 417 snap 112 seq_num 417) dirty exists 2021-03-15T22:25:28.494 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:28.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:28.517 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:28.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:28 smithi125 conmon[56739]: cluster 2021-03-15T22:25:27.053166+0000 mgr.y (mgr.14144) 615 2021-03-15T22:25:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:28 smithi125 conmon[56739]: : cluster [DBG] pgmap v744: 138 pgs: 138 active+clean; 307 MiB data, 913 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 290 KiB/s wr, 11 op/s 2021-03-15T22:25:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:28 smithi125 conmon[56739]: cluster 2021-03-15T22:25:27.164597+0000 mon.a (mon.0) 1225 : cluster [DBG] osdmap e412: 8 total, 8 up, 7 in 2021-03-15T22:25:28.619 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:28.641 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:28.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:28.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:28.770 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:28.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:28.772 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:28.825 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:25:28.830 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:25:28.852 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:25:28.869 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:25:28.874 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:28.888 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:25:28.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:28.900 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:28.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:28.907 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:25:28.925 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:25:28.948 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:25:28.948 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:25:28.949 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:25:28.949 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:25:28.949 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:25:28.950 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:25:28.950 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:25:28.950 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:25:28.951 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:25:29.002 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:29.025 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:29.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:29.026 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:29.128 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:29.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:29.155 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:29.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:29.183 INFO:tasks.rados.rados.0.smithi076.stdout:1515: done (1 left) 2021-03-15T22:25:29.184 INFO:tasks.rados.rados.0.smithi076.stdout:1516: done (0 left) 2021-03-15T22:25:29.184 INFO:tasks.rados.rados.0.smithi076.stdout:1517: snap_create 2021-03-15T22:25:29.257 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:29.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:29.284 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:29.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:29.385 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:29.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:29.409 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:29.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:29.511 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:29.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:29 smithi076 conmon[56514]: cluster 2021-03-15T22:25:28.164428+0000 mon.a (mon.0) 1226 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2021-03-15T22:25:29.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:29 smithi076 conmon[56514]: audit 2021-03-15T22:25:28.791218+0000 mon.a (mon.0) 1227 : audit [DBG] from='client.? 172.21.15.76:0/2865239190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:29.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:29 smithi076 conmon[62156]: cluster 2021-03-15T22:25:28.164428+0000 2021-03-15T22:25:29.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:29 smithi076 conmon[62156]: mon.a (mon.0) 1226 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2021-03-15T22:25:29.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:29 smithi076 conmon[62156]: audit 2021-03-15T22:25:28.791218+0000 mon.a (mon.0) 1227 : audit [DBG] from='client.? 172.21.15.76:0/2865239190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:29.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:29.535 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:29.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:29 smithi125 conmon[56739]: cluster 2021-03-15T22:25:28.164428+0000 mon.a (mon.0) 1226 : cluster [DBG] osdmap e413: 8 total, 8 up, 7 in 2021-03-15T22:25:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:29 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:25:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:29 smithi125 conmon[56739]: 25:28.791218+0000 mon.a (mon.0) 1227 : audit [DBG] from='client.? 172.21.15.76:0/2865239190' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:29.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:29.663 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:29.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:29.665 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:29.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:29.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:29.793 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:29.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:29.895 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:29.919 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:29.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:29.920 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:30.021 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:30.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:25:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:25:30.044 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:30.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:30.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:30.102 INFO:tasks.rados.rados.0.smithi076.stdout:1517: done (0 left) 2021-03-15T22:25:30.239 INFO:tasks.rados.rados.0.smithi076.stdout:1518: delete oid 5 current snap is 168 2021-03-15T22:25:30.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:30.240 INFO:tasks.rados.rados.0.smithi076.stdout:1518: done (0 left) 2021-03-15T22:25:30.241 INFO:tasks.rados.rados.0.smithi076.stdout:1519: snap_create 2021-03-15T22:25:30.272 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:30.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:30.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:30.375 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:30.413 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:30.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:30.415 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:30.516 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:30.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[56514]: cluster 2021-03-15T22:25:29.053534+0000 2021-03-15T22:25:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[56514]: mgr.y (mgr.14144) 616 : cluster [DBG] pgmap v747: 138 pgs: 138 active+clean; 311 MiB data, 919 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-03-15T22:25:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[56514]: cluster 2021-03-15T22:25:29. 2021-03-15T22:25:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[56514]: 181138+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2021-03-15T22:25:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[56514]: audit 2021-03-15T 2021-03-15T22:25:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[56514]: 22:25:29.709513+0000 mgr.y (mgr 2021-03-15T22:25:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[56514]: .14144) 617 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[56514]: cluster 2021-03-15T22:25:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[56514]: 2021-03-15T22:25:30.098079 2021-03-15T22:25:30.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1229 : cluster [DBG] 2021-03-15T22:25:30.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[56514]: osdmap e415: 8 total, 8 up, 7 in 2021-03-15T22:25:30.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: cluster 2021-03-15T22:25:29.053534+0000 2021-03-15T22:25:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: mgr.y (mgr.14144) 616 : cluster [DBG] pgmap v747: 138 pgs: 138 active+clean; 311 MiB data, 919 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-03-15T22:25:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:25:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: 15T22:25:29.181138+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2021-03-15T22:25:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: 2021-03-15T22:25:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: audit 2021-03-15T22:25 2021-03-15T22:25:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: :29.709513+0000 mgr.y (mgr.14144) 2021-03-15T22:25:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: 617 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:25:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: -03-15T22:25:30.098079+0000 2021-03-15T22:25:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: mon.a (mon.0) 1229 : cluster 2021-03-15T22:25:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:30 smithi076 conmon[62156]: [DBG] osdmap e415: 8 total, 8 up, 7 in 2021-03-15T22:25:30.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:30 smithi125 conmon[56739]: cluster 2021-03-15T22:25:29.053534+0000 mgr.y (mgr.14144) 616 : cluster 2021-03-15T22:25:30.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:30 smithi125 conmon[56739]: [DBG] pgmap v747: 138 pgs: 138 active+clean; 311 MiB data, 919 MiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.2 MiB/s wr, 11 op/s 2021-03-15T22:25:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:30 smithi125 conmon[56739]: cluster 2021-03-15T22:25:29.181138+0000 mon.a (mon.0) 1228 : cluster [DBG] osdmap e414: 8 total, 8 up, 7 in 2021-03-15T22:25:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:30 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:25:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:30 smithi125 conmon[56739]: 03-15T22:25:29.709513+0000 mgr.y (mgr.14144) 617 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:30 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:30 smithi125 conmon[56739]: 03-15T22:25:30.098079+0000 2021-03-15T22:25:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:30 smithi125 conmon[56739]: mon.a (mon.0) 1229 : cluster [DBG] osdmap e415: 8 total, 8 up, 7 in 2021-03-15T22:25:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:30 smithi125 conmon[56739]: 2021-03-15T22:25:30.553 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:30.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:30.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:30.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:30.692 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:30.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:30.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:30.795 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:30.827 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:30.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:30.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:30.929 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:30.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:30.956 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:30.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:31.059 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:31.083 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:31.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:31.109 INFO:tasks.rados.rados.0.smithi076.stdout:1519: done (0 left) 2021-03-15T22:25:31.110 INFO:tasks.rados.rados.0.smithi076.stdout:1520: read oid 18 snap -1 2021-03-15T22:25:31.110 INFO:tasks.rados.rados.0.smithi076.stdout:1520: expect (ObjNum 533 snap 166 seq_num 533) 2021-03-15T22:25:31.110 INFO:tasks.rados.rados.0.smithi076.stdout:1521: snap_create 2021-03-15T22:25:31.114 INFO:tasks.rados.rados.0.smithi076.stdout:1520: expect (ObjNum 533 snap 166 seq_num 533) 2021-03-15T22:25:31.186 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:31.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:31.211 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:31.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:31.312 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:31.335 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:31.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:31.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:31.437 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:31.458 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:25:31.458 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:25:31.458 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:25:31.458 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:25:31.459 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:25:31.463 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:31.465 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:31.566 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:31.589 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:31.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:31.591 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:31.692 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:31.718 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:31.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:31.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:31.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:31.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:31.847 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:31.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:31.949 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:31.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:31.977 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:31.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:32.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:32.105 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:32.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:32.107 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:32.120 INFO:tasks.rados.rados.0.smithi076.stdout:1520: done (1 left) 2021-03-15T22:25:32.120 INFO:tasks.rados.rados.0.smithi076.stdout:1521: done (0 left) 2021-03-15T22:25:32.121 INFO:tasks.rados.rados.0.smithi076.stdout:1522: snap_remove snap 157 2021-03-15T22:25:32.129 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.3 to 1 (10000) 2021-03-15T22:25:32.130 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:25:32.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:32.232 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:32.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:32.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:32.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:32.359 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:32.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:32.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:32.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:32 smithi076 conmon[56514]: cluster 2021-03-15T22:25:31.053915+0000 mgr.y (mgr.14144) 618 : cluster [DBG] pgmap v750: 138 pgs: 138 active+clean; 310 MiB data, 918 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2021-03-15T22:25:32.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:32 smithi076 conmon[56514]: cluster 2021-03-15T22:25:31.105856+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2021-03-15T22:25:32.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:32 smithi076 conmon[56514]: audit 2021-03-15T22:25:31.272749+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? 172.21.15.76:0/3323949668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-15T22:25:32.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:32 smithi076 conmon[56514]: audit 2021-03-15T22:25:31.810805+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? 172.21.15.76:0/2018235431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:25:32.417 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:32 smithi076 conmon[62156]: cluster 2021-03-15T22:25:31.053915+0000 mgr.y (mgr.14144) 618 : cluster [DBG] pgmap v750: 138 pgs: 138 active+clean; 310 MiB data, 918 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2021-03-15T22:25:32.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:32 smithi076 conmon[62156]: cluster 2021-03-15T22:25:31.105856+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2021-03-15T22:25:32.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:32 smithi076 conmon[62156]: audit 2021-03-15T22:25:31.272749+0000 mon.a (mon.0) 1231 : audit [INF] from='client.? 172.21.15.76:0/3323949668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-15T22:25:32.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:32 smithi076 conmon[62156]: audit 2021-03-15T22:25:31.810805+0000 mon.a (mon.0) 1232 : audit [INF] from='client.? 172.21.15.76:0/2018235431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:25:32.443 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-15T22:25:32.461 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:32.510 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:32.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:32 smithi125 conmon[56739]: cluster 2021-03-15T22:25:31.053915+0000 mgr.y (mgr.14144) 618 : cluster 2021-03-15T22:25:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:32 smithi125 conmon[56739]: [DBG] pgmap v750: 138 pgs: 138 active+clean; 310 MiB data, 918 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2021-03-15T22:25:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:32 smithi125 conmon[56739]: cluster 2021-03-15T22:25:31.105856+0000 mon.a (mon.0) 1230 : cluster [DBG] osdmap e416: 8 total, 8 up, 7 in 2021-03-15T22:25:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:32 smithi125 conmon[56739]: audit 2021-03-15T22:25:31.272749+0000 2021-03-15T22:25:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:32 smithi125 conmon[56739]: mon.a (mon.0) 1231 : audit [INF] from='client.? 172.21.15.76:0/3323949668' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-15T22:25:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:32 smithi125 conmon[56739]: 2021-03-15T22:25:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:32 smithi125 conmon[56739]: audit 2021-03-15T22:25: 2021-03-15T22:25:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:32 smithi125 conmon[56739]: 31.810805+0000 mon.a (mon.0) 1232 : 2021-03-15T22:25:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:32 smithi125 conmon[56739]: audit [INF] from='client.? 172.21.15.76:0/2018235431' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:25:32.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:32.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:25:32.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:32.640 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:32.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:32.741 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:32.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:32.764 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:32.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:32.866 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:32.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:32.892 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:32.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:32.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:33.020 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:33.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:33.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:33.120 INFO:tasks.rados.rados.0.smithi076.stdout:1522: done (0 left) 2021-03-15T22:25:33.120 INFO:tasks.rados.rados.0.smithi076.stdout:1523: read oid 20 snap -1 2021-03-15T22:25:33.121 INFO:tasks.rados.rados.0.smithi076.stdout:1523: expect (ObjNum 502 snap 150 seq_num 502) 2021-03-15T22:25:33.121 INFO:tasks.rados.rados.0.smithi076.stdout:1524: write (excl) oid 37 current snap is 170 2021-03-15T22:25:33.121 INFO:tasks.rados.rados.0.smithi076.stdout:1524: seq_num 536 ranges {608552=478174,1848819=707173} 2021-03-15T22:25:33.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:33.127 INFO:tasks.rados.rados.0.smithi076.stdout:1524: writing smithi07694878-37 from 608552 to 1086726 tid 1 2021-03-15T22:25:33.137 INFO:tasks.rados.rados.0.smithi076.stdout:1524: writing smithi07694878-37 from 1848819 to 2555992 tid 2 2021-03-15T22:25:33.137 INFO:tasks.rados.rados.0.smithi076.stdout:1525: rollback oid 1 current snap is 170 2021-03-15T22:25:33.137 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 1 to 153 2021-03-15T22:25:33.138 INFO:tasks.rados.rados.0.smithi076.stdout:1526: delete oid 19 current snap is 170 2021-03-15T22:25:33.142 INFO:tasks.rados.rados.0.smithi076.stdout:1523: expect (ObjNum 502 snap 150 seq_num 502) 2021-03-15T22:25:33.156 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:33.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:33.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:33.259 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:33.294 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:33.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:33.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:33.309 INFO:tasks.rados.rados.0.smithi076.stdout:1523: done (3 left) 2021-03-15T22:25:33.310 INFO:tasks.rados.rados.0.smithi076.stdout:1526: done (2 left) 2021-03-15T22:25:33.310 INFO:tasks.rados.rados.0.smithi076.stdout:1527: snap_remove snap 167 2021-03-15T22:25:33.337 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:33.363 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: cluster 2021-03-15T22:25:32.104170+0000 mon.a (mon.0) 2021-03-15T22:25:33.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: 1233 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:33.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: audit 2021-03-15T22:25:32.113448+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? 172.21.15.76:0/3323949668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-15T22:25:33.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: 2021-03-15T22:25:33.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:25:33.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: T22:25:32.113523+0000 mon.a (mon.0) 1235 : audit 2021-03-15T22:25:33.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/2018235431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:25:33.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:25:33.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: 15T22:25:32.113574+0000 mon.a 2021-03-15T22:25:33.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: (mon.0) 1236 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2021-03-15T22:25:33.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: cluster 2021-03-15T22:25:33.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: 2021-03-15T22:25:32 2021-03-15T22:25:33.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: .490686+0000 osd.6 (osd.6) 65 : 2021-03-15T22:25:33.367 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: cluster [DBG] 7.d scrub ok 2021-03-15T22:25:33.367 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: cluster 2021-03-15T22:25:33.367 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: 2021-03-15T22:25: 2021-03-15T22:25:33.367 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: 32.799877+0000 osd.5 (osd.5) 75 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:25:33.368 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[62156]: 2021-03-15T22:25:33.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[56514]: cluster 2021-03-15T22:25:32.104170+0000 mon.a (mon.0) 1233 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:33.371 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[56514]: audit 2021-03-15T22:25:32.113448+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? 172.21.15.76:0/3323949668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-15T22:25:33.372 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[56514]: audit 2021-03-15T22:25:32.113523+0000 mon.a (mon.0) 1235 : audit [INF] from='client.? 172.21.15.76:0/2018235431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:25:33.372 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[56514]: cluster 2021-03-15T22:25:32.113574+0000 mon.a (mon.0) 1236 : cluster [DBG] osdmap e417: 8 total, 8 up, 7 in 2021-03-15T22:25:33.372 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[56514]: cluster 2021-03-15T22:25:32.490686+0000 osd.6 (osd.6) 65 : cluster [DBG] 7.d scrub ok 2021-03-15T22:25:33.372 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:33 smithi076 conmon[56514]: cluster 2021-03-15T22:25:32.799877+0000 osd.5 (osd.5) 75 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:25:33.373 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2021-03-15T22:25:33.374 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:25:33.389 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:33.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:33.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:33.465 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:33.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:33.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: cluster 2021-03-15T22:25:32.104170+0000 mon.a (mon.0) 1233 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: audit 2021-03-15T22:25:32.113448+0000 mon.a (mon.0) 1234 : audit [INF] from='client.? 172.21.15.76:0/3323949668' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-15T22:25:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:25:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: :25:32.113523+0000 mon.a (mon. 2021-03-15T22:25:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: 0) 1235 : audit [INF] from='client.? 172.21.15.76:0/2018235431' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:25:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:25:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: 15T22:25:32.113574+0000 2021-03-15T22:25:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: mon.a (mon.0) 1236 : cluster [DBG] 2021-03-15T22:25:33.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: osdmap e417: 8 total, 8 up, 7 in 2021-03-15T22:25:33.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:33.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: 03-15T22:25:32. 2021-03-15T22:25:33.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: 490686+0000 osd.6 (osd.6) 65 : cluster 2021-03-15T22:25:33.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: [DBG] 7.d scrub ok 2021-03-15T22:25:33.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:25:33.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: -03-15T22:25: 2021-03-15T22:25:33.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: 32.799877+0000 osd.5 (osd.5) 75 2021-03-15T22:25:33.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:33 smithi125 conmon[56739]: : cluster [DBG] 4.8 scrub ok 2021-03-15T22:25:33.566 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:33.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:33.616 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:33.617 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:33.718 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:33.746 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:33.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:33.748 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:33.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:33.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:33.872 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:33.873 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:33.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:33.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:33.998 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:33.998 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:34.099 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:34.123 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:34.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:34.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:34.130 INFO:tasks.rados.rados.0.smithi076.stdout:1527: done (2 left) 2021-03-15T22:25:34.131 INFO:tasks.rados.rados.0.smithi076.stdout:1528: snap_create 2021-03-15T22:25:34.131 INFO:tasks.rados.rados.0.smithi076.stdout:1525: finishing rollback tid 0 to smithi07694878-1 2021-03-15T22:25:34.131 INFO:tasks.rados.rados.0.smithi076.stdout:1525: finishing rollback tid 1 to smithi07694878-1 2021-03-15T22:25:34.132 INFO:tasks.rados.rados.0.smithi076.stdout:1525: finishing rollback tid 2 to smithi07694878-1 2021-03-15T22:25:34.132 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 826 (ObjNum 496 snap 149 seq_num 496) dirty exists 2021-03-15T22:25:34.132 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:34.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:34.237 INFO:tasks.rados.rados.0.smithi076.stdout:1524: finishing write tid 1 to smithi07694878-37 2021-03-15T22:25:34.240 INFO:tasks.rados.rados.0.smithi076.stdout:1524: finishing write tid 2 to smithi07694878-37 2021-03-15T22:25:34.241 INFO:tasks.rados.rados.0.smithi076.stdout:1524: finishing write tid 3 to smithi07694878-37 2021-03-15T22:25:34.242 INFO:tasks.rados.rados.0.smithi076.stdout:1524: finishing write tid 4 to smithi07694878-37 2021-03-15T22:25:34.242 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 956 (ObjNum 536 snap 170 seq_num 536) dirty exists 2021-03-15T22:25:34.242 INFO:tasks.rados.rados.0.smithi076.stdout:1524: left oid 37 (ObjNum 536 snap 170 seq_num 536) 2021-03-15T22:25:34.249 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:34.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:34.250 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:34 smithi076 conmon[56514]: cluster 2021-03-15T22:25:33.054645+0000 mgr.y (mgr.14144) 619 : cluster [DBG] pgmap v753: 138 pgs: 8 peering, 130 active+clean; 307 MiB data, 916 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 0 B/s wr, 10 op/s 2021-03-15T22:25:34.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:34 smithi076 conmon[56514]: cluster 2021-03-15T22:25:33.116121+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2021-03-15T22:25:34.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:34 smithi076 conmon[56514]: cluster 2021-03-15T22:25:33.525448+0000 osd.6 (osd.6) 66 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:34.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:34 smithi076 conmon[56514]: cluster 2021-03-15T22:25:33.525877+0000 osd.6 (osd.6) 67 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:34.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:34 smithi076 conmon[62156]: cluster 2021-03-15T22:25:33.054645+0000 mgr.y (mgr.14144) 619 : cluster [DBG] pgmap v753: 138 pgs: 8 peering, 130 active+clean; 307 MiB data, 916 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 0 B/s wr, 10 op/s 2021-03-15T22:25:34.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:34 smithi076 conmon[62156]: cluster 2021-03-15T22:25:33.116121+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2021-03-15T22:25:34.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:34 smithi076 conmon[62156]: cluster 2021-03-15T22:25:33.525448+0000 osd.6 (osd.6) 66 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:34.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:34 smithi076 conmon[62156]: cluster 2021-03-15T22:25:33.525877+0000 osd.6 (osd.6) 67 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:34.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:34.374 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:34.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:34.375 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:34.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:34.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:34.499 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:34.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:34.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:34 smithi125 conmon[56739]: cluster 2021-03-15T22:25:33.054645+0000 mgr.y (mgr.14144) 619 : 2021-03-15T22:25:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:34 smithi125 conmon[56739]: cluster [DBG] pgmap v753: 138 pgs: 8 peering, 130 active+clean; 307 MiB data, 916 MiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 0 B/s wr, 10 op/s 2021-03-15T22:25:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:34 smithi125 conmon[56739]: cluster 2021-03-15T22:25:33.116121+0000 mon.a (mon.0) 1237 : cluster [DBG] osdmap e418: 8 total, 8 up, 7 in 2021-03-15T22:25:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:34 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:25:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:34 smithi125 conmon[56739]: :25:33.525448+0000 osd.6 (osd. 2021-03-15T22:25:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:34 smithi125 conmon[56739]: 6) 66 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:34 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:25:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:34 smithi125 conmon[56739]: -03-15T22:25: 2021-03-15T22:25:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:34 smithi125 conmon[56739]: 33.525877+0000 osd.6 (osd.6) 2021-03-15T22:25:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:34 smithi125 conmon[56739]: 67 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:34.600 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:34.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:34.623 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:34.623 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:34.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:34.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:34.748 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:34.749 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:34.850 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:34.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:34.873 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:34.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:34.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:34.998 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:34.998 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:35.099 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:35.103 INFO:tasks.rados.rados.0.smithi076.stdout:1524: done (2 left) 2021-03-15T22:25:35.103 INFO:tasks.rados.rados.0.smithi076.stdout:1525: done (1 left) 2021-03-15T22:25:35.104 INFO:tasks.rados.rados.0.smithi076.stdout:1528: done (0 left) 2021-03-15T22:25:35.104 INFO:tasks.rados.rados.0.smithi076.stdout:1529: write (excl) oid 6 current snap is 171 2021-03-15T22:25:35.104 INFO:tasks.rados.rados.0.smithi076.stdout:1529: seq_num 538 ranges {642166=628672,1849430=710842} 2021-03-15T22:25:35.112 INFO:tasks.rados.rados.0.smithi076.stdout:1529: writing smithi07694878-6 from 642166 to 1270838 tid 1 2021-03-15T22:25:35.120 INFO:tasks.rados.rados.0.smithi076.stdout:1529: writing smithi07694878-6 from 1849430 to 2560272 tid 2 2021-03-15T22:25:35.120 INFO:tasks.rados.rados.0.smithi076.stdout:1530: write (excl) oid 2 current snap is 171 2021-03-15T22:25:35.121 INFO:tasks.rados.rados.0.smithi076.stdout:1529: finishing write tid 1 to smithi07694878-6 2021-03-15T22:25:35.121 INFO:tasks.rados.rados.0.smithi076.stdout:1530: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2021-03-15T22:25:35.128 INFO:tasks.rados.rados.0.smithi076.stdout:1530: writing smithi07694878-2 from 658973 to 1362894 tid 1 2021-03-15T22:25:35.130 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:35.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:35.131 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:35.134 INFO:tasks.rados.rados.0.smithi076.stdout:1530: writing smithi07694878-2 from 1907912 to 2478765 tid 2 2021-03-15T22:25:35.138 INFO:tasks.rados.rados.0.smithi076.stdout:1530: writing smithi07694878-2 from 2985346 to 3058973 tid 3 2021-03-15T22:25:35.138 INFO:tasks.rados.rados.0.smithi076.stdout:1531: delete oid 22 current snap is 171 2021-03-15T22:25:35.138 INFO:tasks.rados.rados.0.smithi076.stdout:1529: finishing write tid 2 to smithi07694878-6 2021-03-15T22:25:35.139 INFO:tasks.rados.rados.0.smithi076.stdout:1529: finishing write tid 3 to smithi07694878-6 2021-03-15T22:25:35.139 INFO:tasks.rados.rados.0.smithi076.stdout:1529: finishing write tid 4 to smithi07694878-6 2021-03-15T22:25:35.139 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 290 (ObjNum 538 snap 171 seq_num 538) dirty exists 2021-03-15T22:25:35.140 INFO:tasks.rados.rados.0.smithi076.stdout:1529: left oid 6 (ObjNum 538 snap 171 seq_num 538) 2021-03-15T22:25:35.142 INFO:tasks.rados.rados.0.smithi076.stdout:1529: done (2 left) 2021-03-15T22:25:35.142 INFO:tasks.rados.rados.0.smithi076.stdout:1531: done (1 left) 2021-03-15T22:25:35.142 INFO:tasks.rados.rados.0.smithi076.stdout:1532: read oid 29 snap -1 2021-03-15T22:25:35.142 INFO:tasks.rados.rados.0.smithi076.stdout:1532: expect (ObjNum 397 snap 108 seq_num 397) 2021-03-15T22:25:35.143 INFO:tasks.rados.rados.0.smithi076.stdout:1533: write oid 50 current snap is 171 2021-03-15T22:25:35.143 INFO:tasks.rados.rados.0.smithi076.stdout:1533: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2021-03-15T22:25:35.149 INFO:tasks.rados.rados.0.smithi076.stdout:1533: writing smithi07694878-50 from 692587 to 1263359 tid 1 2021-03-15T22:25:35.156 INFO:tasks.rados.rados.0.smithi076.stdout:1533: writing smithi07694878-50 from 2024876 to 2599398 tid 2 2021-03-15T22:25:35.159 INFO:tasks.rados.rados.0.smithi076.stdout:1533: writing smithi07694878-50 from 3040192 to 3092587 tid 3 2021-03-15T22:25:35.160 INFO:tasks.rados.rados.0.smithi076.stdout:1534: rollback oid 33 current snap is 171 2021-03-15T22:25:35.160 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 33 to 147 2021-03-15T22:25:35.160 INFO:tasks.rados.rados.0.smithi076.stdout:1532: expect (ObjNum 397 snap 108 seq_num 397) 2021-03-15T22:25:35.233 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:35.254 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:35.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:35.256 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:35.273 INFO:tasks.rados.rados.0.smithi076.stdout:1533: finishing write tid 1 to smithi07694878-50 2021-03-15T22:25:35.274 INFO:tasks.rados.rados.0.smithi076.stdout:1533: finishing write tid 2 to smithi07694878-50 2021-03-15T22:25:35.274 INFO:tasks.rados.rados.0.smithi076.stdout:1532: done (3 left) 2021-03-15T22:25:35.274 INFO:tasks.rados.rados.0.smithi076.stdout:1535: read oid 24 snap -1 2021-03-15T22:25:35.274 INFO:tasks.rados.rados.0.smithi076.stdout:1535: expect (ObjNum 514 snap 155 seq_num 514) 2021-03-15T22:25:35.275 INFO:tasks.rados.rados.0.smithi076.stdout:1536: write oid 25 current snap is 171 2021-03-15T22:25:35.275 INFO:tasks.rados.rados.0.smithi076.stdout:1533: finishing write tid 3 to smithi07694878-50 2021-03-15T22:25:35.275 INFO:tasks.rados.rados.0.smithi076.stdout:1533: finishing write tid 4 to smithi07694878-50 2021-03-15T22:25:35.275 INFO:tasks.rados.rados.0.smithi076.stdout:1536: seq_num 542 ranges {709394=646021,2083358=718180} 2021-03-15T22:25:35.279 INFO:tasks.rados.rados.0.smithi076.stdout:1536: writing smithi07694878-25 from 709394 to 1355415 tid 1 2021-03-15T22:25:35.288 INFO:tasks.rados.rados.0.smithi076.stdout:1536: writing smithi07694878-25 from 2083358 to 2801538 tid 2 2021-03-15T22:25:35.289 INFO:tasks.rados.rados.0.smithi076.stdout:1537: read oid 42 snap -1 2021-03-15T22:25:35.289 INFO:tasks.rados.rados.0.smithi076.stdout:1537: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:25:35.289 INFO:tasks.rados.rados.0.smithi076.stdout:1538: read oid 40 snap -1 2021-03-15T22:25:35.290 INFO:tasks.rados.rados.0.smithi076.stdout:1538: expect deleted 2021-03-15T22:25:35.290 INFO:tasks.rados.rados.0.smithi076.stdout:1539: rollback oid 35 current snap is 171 2021-03-15T22:25:35.290 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 35 to 164 2021-03-15T22:25:35.290 INFO:tasks.rados.rados.0.smithi076.stdout:1540: snap_create 2021-03-15T22:25:35.291 INFO:tasks.rados.rados.0.smithi076.stdout:1534: finishing rollback tid 0 to smithi07694878-33 2021-03-15T22:25:35.291 INFO:tasks.rados.rados.0.smithi076.stdout:1534: finishing rollback tid 1 to smithi07694878-33 2021-03-15T22:25:35.291 INFO:tasks.rados.rados.0.smithi076.stdout:1533: finishing write tid 5 to smithi07694878-50 2021-03-15T22:25:35.291 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 678 (ObjNum 541 snap 171 seq_num 541) dirty exists 2021-03-15T22:25:35.292 INFO:tasks.rados.rados.0.smithi076.stdout:1533: left oid 50 (ObjNum 541 snap 171 seq_num 541) 2021-03-15T22:25:35.292 INFO:tasks.rados.rados.0.smithi076.stdout:1534: finishing rollback tid 2 to smithi07694878-33 2021-03-15T22:25:35.292 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 493 (ObjNum 374 snap 102 seq_num 374) dirty exists 2021-03-15T22:25:35.292 INFO:tasks.rados.rados.0.smithi076.stdout:1535: expect (ObjNum 514 snap 155 seq_num 514) 2021-03-15T22:25:35.310 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:25:35.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:35.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:35 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:25:35.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:35 smithi076 conmon[56514]: 22:25:34.124273+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2021-03-15T22:25:35.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:35 smithi076 conmon[56514]: cluster 2021-03-15T22:25:35.099624+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2021-03-15T22:25:35.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:35.385 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:35.385 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:35.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:35 smithi076 conmon[62156]: cluster 2021-03-15T22:25:34 2021-03-15T22:25:35.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:35 smithi076 conmon[62156]: .124273+0000 mon.a (mon.0) 1238 : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2021-03-15T22:25:35.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:35 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:25:35.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:35 smithi076 conmon[62156]: :25:35.099624+0000 mon.a (mon.0 2021-03-15T22:25:35.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:35 smithi076 conmon[62156]: ) 1239 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2021-03-15T22:25:35.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:35.488 INFO:tasks.rados.rados.0.smithi076.stdout:1539: finishing rollback tid 1 to smithi07694878-35 2021-03-15T22:25:35.488 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 492 (ObjNum 264829504 snap 21896 seq_num 264828288) dirty dne 2021-03-15T22:25:35.488 INFO:tasks.rados.rados.0.smithi076.stdout:1537: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:25:35.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:35.520 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:35.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:35.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:35 smithi125 conmon[56739]: cluster 2021-03-15T22:25:34.124273+0000 mon.a (mon.0) 1238 2021-03-15T22:25:35.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:35 smithi125 conmon[56739]: : cluster [DBG] osdmap e419: 8 total, 8 up, 7 in 2021-03-15T22:25:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:35 smithi125 conmon[56739]: cluster 2021-03-15T22:25:35.099624+0000 mon.a (mon.0) 1239 : cluster [DBG] osdmap e420: 8 total, 8 up, 7 in 2021-03-15T22:25:35.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:35.644 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:35.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:35.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:35.747 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:35.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:35.769 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:35.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:35.871 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:35.895 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:35.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:35.896 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:35.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:36.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:36.019 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:36.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:36.036 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:25:36.036 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:25:36.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:25:36.037 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:25:36.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.037 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:836 7.3 7:c458c800:::smithi07694878-3:head [copy-from ver 127] snapc 34=[34,32,30] ondisk+write+known_if_redirected e149)", 2021-03-15T22:25:36.037 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018906+0000", 2021-03-15T22:25:36.038 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 272.01167842000001, 2021-03-15T22:25:36.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024704783000000001, 2021-03-15T22:25:36.038 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.038 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:36.039 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.039 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.039 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.039 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 836 2021-03-15T22:25:36.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.040 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018906+0000", 2021-03-15T22:25:36.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018906+0000", 2021-03-15T22:25:36.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.209e-06 2021-03-15T22:25:36.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018908+0000", 2021-03-15T22:25:36.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9100000000000003e-07 2021-03-15T22:25:36.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018908+0000", 2021-03-15T22:25:36.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2780000000000002e-06 2021-03-15T22:25:36.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018912+0000", 2021-03-15T22:25:36.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9469999999999999e-06 2021-03-15T22:25:36.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018915+0000", 2021-03-15T22:25:36.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0328000000000003e-05 2021-03-15T22:25:36.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018965+0000", 2021-03-15T22:25:36.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7919999999999993e-06 2021-03-15T22:25:36.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:25:36.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018973+0000", 2021-03-15T22:25:36.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025313699999999998 2021-03-15T22:25:36.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021504+0000", 2021-03-15T22:25:36.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4449000000000006e-05 2021-03-15T22:25:36.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021568+0000", 2021-03-15T22:25:36.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071715939999999999 2021-03-15T22:25:36.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:25:36.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.028740+0000", 2021-03-15T22:25:36.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012489007 2021-03-15T22:25:36.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.041229+0000", 2021-03-15T22:25:36.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001956981 2021-03-15T22:25:36.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:36.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043186+0000", 2021-03-15T22:25:36.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000330864 2021-03-15T22:25:36.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043517+0000", 2021-03-15T22:25:36.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8581000000000001e-05 2021-03-15T22:25:36.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:36.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043565+0000", 2021-03-15T22:25:36.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5692000000000002e-05 2021-03-15T22:25:36.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043611+0000", 2021-03-15T22:25:36.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024704783000000001 2021-03-15T22:25:36.061 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.061 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.061 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.062 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:837 7.3 7:c458c800:::smithi07694878-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e149)", 2021-03-15T22:25:36.062 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:04.018929+0000", 2021-03-15T22:25:36.063 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 272.01165586799999, 2021-03-15T22:25:36.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024813472, 2021-03-15T22:25:36.063 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.063 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:36.063 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.064 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.064 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.064 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 837 2021-03-15T22:25:36.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.065 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.065 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018929+0000", 2021-03-15T22:25:36.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018929+0000", 2021-03-15T22:25:36.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0210000000000001e-06 2021-03-15T22:25:36.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018930+0000", 2021-03-15T22:25:36.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1789999999999997e-06 2021-03-15T22:25:36.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018939+0000", 2021-03-15T22:25:36.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7000000000000003e-06 2021-03-15T22:25:36.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018949+0000", 2021-03-15T22:25:36.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0043e-05 2021-03-15T22:25:36.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018959+0000", 2021-03-15T22:25:36.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2956000000000001e-05 2021-03-15T22:25:36.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018992+0000", 2021-03-15T22:25:36.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6950000000000002e-06 2021-03-15T22:25:36.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:25:36.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.018996+0000", 2021-03-15T22:25:36.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026765769999999999 2021-03-15T22:25:36.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021672+0000", 2021-03-15T22:25:36.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1886999999999999e-05 2021-03-15T22:25:36.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:25:36.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.021734+0000", 2021-03-15T22:25:36.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015444232 2021-03-15T22:25:36.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.037178+0000", 2021-03-15T22:25:36.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8273999999999998e-05 2021-03-15T22:25:36.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:36.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.037236+0000", 2021-03-15T22:25:36.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0064013719999999998 2021-03-15T22:25:36.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.082 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043638+0000", 2021-03-15T22:25:36.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4591e-05 2021-03-15T22:25:36.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043682+0000", 2021-03-15T22:25:36.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9945000000000003e-05 2021-03-15T22:25:36.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:04.043742+0000", 2021-03-15T22:25:36.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024813472 2021-03-15T22:25:36.085 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.085 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.085 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.086 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1378 7.a 7:53dee572:::smithi07694878-35:head [copy-from ver 3] snapc 5b=[5a,56] ondisk+write+known_if_redirected e190)", 2021-03-15T22:25:36.086 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:45.023845+0000", 2021-03-15T22:25:36.086 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 231.00674017599999, 2021-03-15T22:25:36.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023497436, 2021-03-15T22:25:36.087 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.088 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:36.088 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.088 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.088 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.089 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1378 2021-03-15T22:25:36.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.089 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023845+0000", 2021-03-15T22:25:36.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023845+0000", 2021-03-15T22:25:36.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5330000000000002e-06 2021-03-15T22:25:36.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023850+0000", 2021-03-15T22:25:36.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6090000000000003e-06 2021-03-15T22:25:36.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023855+0000", 2021-03-15T22:25:36.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9403e-05 2021-03-15T22:25:36.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023874+0000", 2021-03-15T22:25:36.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6260000000000003e-06 2021-03-15T22:25:36.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.097 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023882+0000", 2021-03-15T22:25:36.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3671999999999997e-05 2021-03-15T22:25:36.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023935+0000", 2021-03-15T22:25:36.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0026e-05 2021-03-15T22:25:36.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:25:36.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023945+0000", 2021-03-15T22:25:36.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0086676960000000008 2021-03-15T22:25:36.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.032613+0000", 2021-03-15T22:25:36.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2980999999999996e-05 2021-03-15T22:25:36.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.032706+0000", 2021-03-15T22:25:36.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0080630400000000005 2021-03-15T22:25:36.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:25:36.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.040769+0000", 2021-03-15T22:25:36.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035132739999999998 2021-03-15T22:25:36.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:36.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.044282+0000", 2021-03-15T22:25:36.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002886647 2021-03-15T22:25:36.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047169+0000", 2021-03-15T22:25:36.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9790999999999998e-05 2021-03-15T22:25:36.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047219+0000", 2021-03-15T22:25:36.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9148999999999999e-05 2021-03-15T22:25:36.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:36.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047278+0000", 2021-03-15T22:25:36.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3989e-05 2021-03-15T22:25:36.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047342+0000", 2021-03-15T22:25:36.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023497436 2021-03-15T22:25:36.110 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.111 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.111 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.111 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1379 7.a 7:53dee572:::smithi07694878-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e190)", 2021-03-15T22:25:36.112 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:45.023893+0000", 2021-03-15T22:25:36.112 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 231.006692246, 2021-03-15T22:25:36.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023671298, 2021-03-15T22:25:36.112 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.112 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:36.113 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.113 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.113 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.113 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1379 2021-03-15T22:25:36.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.114 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023893+0000", 2021-03-15T22:25:36.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023893+0000", 2021-03-15T22:25:36.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4050000000000001e-06 2021-03-15T22:25:36.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023894+0000", 2021-03-15T22:25:36.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.657e-06 2021-03-15T22:25:36.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023896+0000", 2021-03-15T22:25:36.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6059999999999999e-06 2021-03-15T22:25:36.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023902+0000", 2021-03-15T22:25:36.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4329999999999997e-06 2021-03-15T22:25:36.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023907+0000", 2021-03-15T22:25:36.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6927000000000004e-05 2021-03-15T22:25:36.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023984+0000", 2021-03-15T22:25:36.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.563e-06 2021-03-15T22:25:36.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.122 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:25:36.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.023987+0000", 2021-03-15T22:25:36.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0094145919999999994 2021-03-15T22:25:36.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.033402+0000", 2021-03-15T22:25:36.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8862999999999997e-05 2021-03-15T22:25:36.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:25:36.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.033441+0000", 2021-03-15T22:25:36.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010337974999999999 2021-03-15T22:25:36.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.043779+0000", 2021-03-15T22:25:36.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3967000000000003e-05 2021-03-15T22:25:36.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:36.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.043873+0000", 2021-03-15T22:25:36.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035103690000000002 2021-03-15T22:25:36.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047383+0000", 2021-03-15T22:25:36.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9901000000000005e-05 2021-03-15T22:25:36.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047473+0000", 2021-03-15T22:25:36.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1039999999999996e-05 2021-03-15T22:25:36.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:45.047564+0000", 2021-03-15T22:25:36.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023671298 2021-03-15T22:25:36.131 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.131 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.131 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.132 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2260 7.2 7:42f18a95:::smithi07694878-45:8d [sparse-read 0~2486556,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e264)", 2021-03-15T22:25:36.132 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.527310+0000", 2021-03-15T22:25:36.132 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 156.50327513400001, 2021-03-15T22:25:36.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53810241999999997, 2021-03-15T22:25:36.133 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.133 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:36.133 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.133 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.133 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.134 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2260 2021-03-15T22:25:36.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.134 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527310+0000", 2021-03-15T22:25:36.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527310+0000", 2021-03-15T22:25:36.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4089999999999996e-06 2021-03-15T22:25:36.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527314+0000", 2021-03-15T22:25:36.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2021-03-15T22:25:36.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527317+0000", 2021-03-15T22:25:36.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7672999999999998e-05 2021-03-15T22:25:36.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527335+0000", 2021-03-15T22:25:36.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6059999999999999e-06 2021-03-15T22:25:36.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527342+0000", 2021-03-15T22:25:36.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2244999999999996e-05 2021-03-15T22:25:36.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527424+0000", 2021-03-15T22:25:36.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.431e-06 2021-03-15T22:25:36.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:25:36.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527433+0000", 2021-03-15T22:25:36.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53064125699999998 2021-03-15T22:25:36.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.143 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.058075+0000", 2021-03-15T22:25:36.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015664499999999999 2021-03-15T22:25:36.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.058231+0000", 2021-03-15T22:25:36.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071808219999999999 2021-03-15T22:25:36.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.065412+0000", 2021-03-15T22:25:36.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53810241999999997 2021-03-15T22:25:36.146 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.146 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.146 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.147 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2261 7.2 7:42f18a95:::smithi07694878-45:8d [sparse-read 0~2486556] snapc 0=[] ondisk+read+known_if_redirected e264)", 2021-03-15T22:25:36.147 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.527444+0000", 2021-03-15T22:25:36.147 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 156.503140903, 2021-03-15T22:25:36.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53896513499999998, 2021-03-15T22:25:36.148 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.148 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:36.148 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.148 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.148 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.149 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2261 2021-03-15T22:25:36.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.149 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527444+0000", 2021-03-15T22:25:36.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527444+0000", 2021-03-15T22:25:36.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.581e-06 2021-03-15T22:25:36.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527445+0000", 2021-03-15T22:25:36.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0140000000000001e-06 2021-03-15T22:25:36.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527447+0000", 2021-03-15T22:25:36.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.644e-06 2021-03-15T22:25:36.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527455+0000", 2021-03-15T22:25:36.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.42e-06 2021-03-15T22:25:36.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527460+0000", 2021-03-15T22:25:36.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6919000000000003e-05 2021-03-15T22:25:36.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527506+0000", 2021-03-15T22:25:36.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6880000000000005e-06 2021-03-15T22:25:36.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:25:36.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527515+0000", 2021-03-15T22:25:36.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53819720400000004 2021-03-15T22:25:36.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.065712+0000", 2021-03-15T22:25:36.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118725 2021-03-15T22:25:36.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.065831+0000", 2021-03-15T22:25:36.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057793999999999999 2021-03-15T22:25:36.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.161 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.161 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066409+0000", 2021-03-15T22:25:36.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53896513499999998 2021-03-15T22:25:36.161 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.161 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.162 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.162 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2262 7.2 7:42f18a95:::smithi07694878-45:8d [sparse-read 0~2486556] snapc 0=[] ondisk+read+known_if_redirected e264)", 2021-03-15T22:25:36.163 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.527470+0000", 2021-03-15T22:25:36.163 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 156.50311477, 2021-03-15T22:25:36.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53929365200000001, 2021-03-15T22:25:36.163 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.163 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:36.164 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.164 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.164 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.164 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2262 2021-03-15T22:25:36.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.165 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527470+0000", 2021-03-15T22:25:36.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527470+0000", 2021-03-15T22:25:36.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1170000000000001e-06 2021-03-15T22:25:36.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527471+0000", 2021-03-15T22:25:36.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.468e-06 2021-03-15T22:25:36.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527473+0000", 2021-03-15T22:25:36.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1550000000000004e-06 2021-03-15T22:25:36.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527479+0000", 2021-03-15T22:25:36.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1910000000000002e-06 2021-03-15T22:25:36.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527483+0000", 2021-03-15T22:25:36.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104373 2021-03-15T22:25:36.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527587+0000", 2021-03-15T22:25:36.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3989999999999993e-06 2021-03-15T22:25:36.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:25:36.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527597+0000", 2021-03-15T22:25:36.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53885792499999996 2021-03-15T22:25:36.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066455+0000", 2021-03-15T22:25:36.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000101804 2021-03-15T22:25:36.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066556+0000", 2021-03-15T22:25:36.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00020722000000000001 2021-03-15T22:25:36.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066764+0000", 2021-03-15T22:25:36.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53929365200000001 2021-03-15T22:25:36.176 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.176 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.177 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.177 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2827 7.a 7:58400cda:::smithi07694878-39:head [copy-from ver 215] snapc d7=[] ondisk+write+known_if_redirected e317)", 2021-03-15T22:25:36.178 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:52.252765+0000", 2021-03-15T22:25:36.178 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 103.77782013300001, 2021-03-15T22:25:36.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023253293000000001, 2021-03-15T22:25:36.178 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.178 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:36.179 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.179 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.179 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.179 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2827 2021-03-15T22:25:36.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.180 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252765+0000", 2021-03-15T22:25:36.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252765+0000", 2021-03-15T22:25:36.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5710000000000003e-06 2021-03-15T22:25:36.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252771+0000", 2021-03-15T22:25:36.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4500000000000003e-06 2021-03-15T22:25:36.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252777+0000", 2021-03-15T22:25:36.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1381000000000001e-05 2021-03-15T22:25:36.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252798+0000", 2021-03-15T22:25:36.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.747e-06 2021-03-15T22:25:36.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252808+0000", 2021-03-15T22:25:36.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1598000000000002e-05 2021-03-15T22:25:36.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252869+0000", 2021-03-15T22:25:36.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.02e-06 2021-03-15T22:25:36.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:25:36.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252875+0000", 2021-03-15T22:25:36.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0074058630000000004 2021-03-15T22:25:36.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.189 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.260281+0000", 2021-03-15T22:25:36.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0491e-05 2021-03-15T22:25:36.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.260352+0000", 2021-03-15T22:25:36.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0051586779999999999 2021-03-15T22:25:36.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:25:36.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.265510+0000", 2021-03-15T22:25:36.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088347449999999998 2021-03-15T22:25:36.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.274345+0000", 2021-03-15T22:25:36.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015050790000000001 2021-03-15T22:25:36.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.275850+0000", 2021-03-15T22:25:36.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8770000000000002e-05 2021-03-15T22:25:36.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:36.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.275869+0000", 2021-03-15T22:25:36.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0575999999999996e-05 2021-03-15T22:25:36.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:36.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.275950+0000", 2021-03-15T22:25:36.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8324000000000005e-05 2021-03-15T22:25:36.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.276018+0000", 2021-03-15T22:25:36.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023253293000000001 2021-03-15T22:25:36.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.197 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.198 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.198 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2828 7.a 7:58400cda:::smithi07694878-39:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e317)", 2021-03-15T22:25:36.199 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:52.252824+0000", 2021-03-15T22:25:36.199 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 103.77776108099999, 2021-03-15T22:25:36.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023410568, 2021-03-15T22:25:36.199 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.199 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:36.200 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.200 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.200 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.200 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2828 2021-03-15T22:25:36.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.201 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.201 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252824+0000", 2021-03-15T22:25:36.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252824+0000", 2021-03-15T22:25:36.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.55e-06 2021-03-15T22:25:36.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252825+0000", 2021-03-15T22:25:36.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.812e-06 2021-03-15T22:25:36.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252827+0000", 2021-03-15T22:25:36.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2676e-05 2021-03-15T22:25:36.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252840+0000", 2021-03-15T22:25:36.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9070000000000002e-06 2021-03-15T22:25:36.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252850+0000", 2021-03-15T22:25:36.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.97e-05 2021-03-15T22:25:36.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252919+0000", 2021-03-15T22:25:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8239999999999996e-06 2021-03-15T22:25:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:25:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.252925+0000", 2021-03-15T22:25:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075364489999999998 2021-03-15T22:25:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.260462+0000", 2021-03-15T22:25:36.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6307999999999998e-05 2021-03-15T22:25:36.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:25:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.260508+0000", 2021-03-15T22:25:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089375010000000005 2021-03-15T22:25:36.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.269445+0000", 2021-03-15T22:25:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6283999999999998e-05 2021-03-15T22:25:36.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.269502+0000", 2021-03-15T22:25:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006549722 2021-03-15T22:25:36.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.276051+0000", 2021-03-15T22:25:36.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0444000000000004e-05 2021-03-15T22:25:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.276132+0000", 2021-03-15T22:25:36.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102391 2021-03-15T22:25:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:52.276234+0000", 2021-03-15T22:25:36.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023410568 2021-03-15T22:25:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3023 7.2 7:42f18a95:::smithi07694878-45:head [stat,delete] snapc e3=[e1,df,de,dd,db] ondisk+write+known_if_redirected e330)", 2021-03-15T22:25:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.084125+0000", 2021-03-15T22:25:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 90.946459727000004, 2021-03-15T22:25:36.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.021223840000000001, 2021-03-15T22:25:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.219 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.220 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.220 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3023 2021-03-15T22:25:36.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.220 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.084125+0000", 2021-03-15T22:25:36.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.084125+0000", 2021-03-15T22:25:36.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8870000000000001e-06 2021-03-15T22:25:36.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.084128+0000", 2021-03-15T22:25:36.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0690000000000002e-06 2021-03-15T22:25:36.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.084131+0000", 2021-03-15T22:25:36.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1883e-05 2021-03-15T22:25:36.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.084143+0000", 2021-03-15T22:25:36.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0049999999999998e-06 2021-03-15T22:25:36.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.084150+0000", 2021-03-15T22:25:36.226 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010193590000000001 2021-03-15T22:25:36.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085169+0000", 2021-03-15T22:25:36.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6044000000000005e-05 2021-03-15T22:25:36.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085255+0000", 2021-03-15T22:25:36.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024674000000000001 2021-03-15T22:25:36.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:25:36.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085502+0000", 2021-03-15T22:25:36.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0070993180000000003 2021-03-15T22:25:36.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.230 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:36.230 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.092601+0000", 2021-03-15T22:25:36.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052819649999999996 2021-03-15T22:25:36.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.231 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.097883+0000", 2021-03-15T22:25:36.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073439940000000004 2021-03-15T22:25:36.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.105227+0000", 2021-03-15T22:25:36.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4447000000000001e-05 2021-03-15T22:25:36.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:36.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.105292+0000", 2021-03-15T22:25:36.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7129000000000001e-05 2021-03-15T22:25:36.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.105349+0000", 2021-03-15T22:25:36.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.021223840000000001 2021-03-15T22:25:36.235 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.235 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.236 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.236 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3267 7.3 7:c62fb1e0:::smithi07694878-14:head [copy-from ver 308] snapc fd=[fd,fc,fb,fa,f9,f7,f5,f2,e9,e7,e1,de] ondisk+write+known_if_redirected e353)", 2021-03-15T22:25:36.236 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:28.078573+0000", 2021-03-15T22:25:36.237 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.952011823000007, 2021-03-15T22:25:36.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019687232999999998, 2021-03-15T22:25:36.237 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.237 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:36.238 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.238 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.238 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.238 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3267 2021-03-15T22:25:36.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.239 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078573+0000", 2021-03-15T22:25:36.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078573+0000", 2021-03-15T22:25:36.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3170000000000001e-06 2021-03-15T22:25:36.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078578+0000", 2021-03-15T22:25:36.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5073e-05 2021-03-15T22:25:36.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078593+0000", 2021-03-15T22:25:36.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2135000000000001e-05 2021-03-15T22:25:36.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078616+0000", 2021-03-15T22:25:36.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7079999999999992e-06 2021-03-15T22:25:36.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078625+0000", 2021-03-15T22:25:36.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4250000000000002e-05 2021-03-15T22:25:36.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078699+0000", 2021-03-15T22:25:36.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4239999999999993e-06 2021-03-15T22:25:36.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:25:36.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078708+0000", 2021-03-15T22:25:36.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022018580000000001 2021-03-15T22:25:36.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.080910+0000", 2021-03-15T22:25:36.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015089500000000001 2021-03-15T22:25:36.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.081061+0000", 2021-03-15T22:25:36.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0094544680000000006 2021-03-15T22:25:36.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:25:36.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.090515+0000", 2021-03-15T22:25:36.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061450640000000004 2021-03-15T22:25:36.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.096660+0000", 2021-03-15T22:25:36.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013721600000000001 2021-03-15T22:25:36.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:36.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.098032+0000", 2021-03-15T22:25:36.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6743999999999994e-05 2021-03-15T22:25:36.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.098129+0000", 2021-03-15T22:25:36.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2440999999999994e-05 2021-03-15T22:25:36.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:36.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.098192+0000", 2021-03-15T22:25:36.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8695999999999994e-05 2021-03-15T22:25:36.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.098260+0000", 2021-03-15T22:25:36.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019687232999999998 2021-03-15T22:25:36.256 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.256 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.256 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.257 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3268 7.3 7:c62fb1e0:::smithi07694878-14:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e353)", 2021-03-15T22:25:36.257 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:28.078649+0000", 2021-03-15T22:25:36.257 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.951935684999995, 2021-03-15T22:25:36.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019771463, 2021-03-15T22:25:36.258 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.258 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:36.258 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.259 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.259 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.259 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3268 2021-03-15T22:25:36.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.259 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078649+0000", 2021-03-15T22:25:36.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078649+0000", 2021-03-15T22:25:36.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.624e-06 2021-03-15T22:25:36.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078651+0000", 2021-03-15T22:25:36.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9910000000000001e-06 2021-03-15T22:25:36.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078653+0000", 2021-03-15T22:25:36.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0990000000000005e-06 2021-03-15T22:25:36.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078662+0000", 2021-03-15T22:25:36.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8960000000000002e-06 2021-03-15T22:25:36.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078667+0000", 2021-03-15T22:25:36.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100833 2021-03-15T22:25:36.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078768+0000", 2021-03-15T22:25:36.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0079999999999996e-06 2021-03-15T22:25:36.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:25:36.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.078775+0000", 2021-03-15T22:25:36.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024338329999999998 2021-03-15T22:25:36.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.081208+0000", 2021-03-15T22:25:36.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5333000000000001e-05 2021-03-15T22:25:36.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:25:36.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.081254+0000", 2021-03-15T22:25:36.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013289871 2021-03-15T22:25:36.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.094544+0000", 2021-03-15T22:25:36.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8207999999999995e-05 2021-03-15T22:25:36.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:36.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.094612+0000", 2021-03-15T22:25:36.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003680446 2021-03-15T22:25:36.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.098292+0000", 2021-03-15T22:25:36.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4121000000000005e-05 2021-03-15T22:25:36.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.098356+0000", 2021-03-15T22:25:36.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4200000000000002e-05 2021-03-15T22:25:36.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:28.098421+0000", 2021-03-15T22:25:36.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019771463 2021-03-15T22:25:36.275 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.275 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.276 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.276 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3285 7.3 7:c62fb1e0:::smithi07694878-14:head [write 1641990~762145] snapc fd=[fd,fc,fb,fa,f9,f7,f5,e9,e7,de] ondisk+write+known_if_redirected e355)", 2021-03-15T22:25:36.276 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:30.099391+0000", 2021-03-15T22:25:36.277 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 65.931193941999993, 2021-03-15T22:25:36.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019588174, 2021-03-15T22:25:36.277 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.277 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:36.277 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.278 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.278 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.278 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3285 2021-03-15T22:25:36.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.279 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.099391+0000", 2021-03-15T22:25:36.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.099391+0000", 2021-03-15T22:25:36.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000409873 2021-03-15T22:25:36.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.099801+0000", 2021-03-15T22:25:36.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1820000000000002e-06 2021-03-15T22:25:36.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.099804+0000", 2021-03-15T22:25:36.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5642999999999999e-05 2021-03-15T22:25:36.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.099820+0000", 2021-03-15T22:25:36.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.827e-06 2021-03-15T22:25:36.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.099826+0000", 2021-03-15T22:25:36.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.799e-05 2021-03-15T22:25:36.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.099874+0000", 2021-03-15T22:25:36.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010590300000000001 2021-03-15T22:25:36.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.099980+0000", 2021-03-15T22:25:36.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017600800000000001 2021-03-15T22:25:36.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:25:36.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.100156+0000", 2021-03-15T22:25:36.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0042527229999999999 2021-03-15T22:25:36.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:36.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.104409+0000", 2021-03-15T22:25:36.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00458041 2021-03-15T22:25:36.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.108989+0000", 2021-03-15T22:25:36.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0098721639999999992 2021-03-15T22:25:36.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.118862+0000", 2021-03-15T22:25:36.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6649000000000005e-05 2021-03-15T22:25:36.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:36.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.118928+0000", 2021-03-15T22:25:36.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0801999999999999e-05 2021-03-15T22:25:36.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:30.118979+0000", 2021-03-15T22:25:36.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019588174 2021-03-15T22:25:36.293 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.294 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.294 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.295 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3661 7.3 7:c7825e8f:::smithi07694878-26:head [write 1692526~473837] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:25:36.295 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.130939+0000", 2021-03-15T22:25:36.295 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.899645360000001, 2021-03-15T22:25:36.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039320842000000002, 2021-03-15T22:25:36.295 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.296 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:36.296 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.296 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.296 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.296 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3661 2021-03-15T22:25:36.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.297 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.130939+0000", 2021-03-15T22:25:36.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.130939+0000", 2021-03-15T22:25:36.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024949 2021-03-15T22:25:36.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131189+0000", 2021-03-15T22:25:36.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0850000000000002e-06 2021-03-15T22:25:36.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.301 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131195+0000", 2021-03-15T22:25:36.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1999000000000001e-05 2021-03-15T22:25:36.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131217+0000", 2021-03-15T22:25:36.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2747e-05 2021-03-15T22:25:36.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131230+0000", 2021-03-15T22:25:36.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3211000000000001e-05 2021-03-15T22:25:36.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131303+0000", 2021-03-15T22:25:36.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134301 2021-03-15T22:25:36.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131437+0000", 2021-03-15T22:25:36.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019751799999999999 2021-03-15T22:25:36.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.306 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:25:36.306 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131635+0000", 2021-03-15T22:25:36.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019844060000000002 2021-03-15T22:25:36.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:36.307 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.133619+0000", 2021-03-15T22:25:36.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061900419999999998 2021-03-15T22:25:36.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.139809+0000", 2021-03-15T22:25:36.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.030327191999999999 2021-03-15T22:25:36.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170136+0000", 2021-03-15T22:25:36.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6150999999999997e-05 2021-03-15T22:25:36.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.310 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:36.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170203+0000", 2021-03-15T22:25:36.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.77e-05 2021-03-15T22:25:36.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170260+0000", 2021-03-15T22:25:36.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039320842000000002 2021-03-15T22:25:36.312 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.312 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.312 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.313 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3662 7.3 7:c7825e8f:::smithi07694878-26:head [setxattr _header (58) in=7b,truncate 2403500] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:25:36.313 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.131248+0000", 2021-03-15T22:25:36.314 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.899337099, 2021-03-15T22:25:36.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039224930999999998, 2021-03-15T22:25:36.314 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.314 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:36.314 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.315 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.315 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.315 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3662 2021-03-15T22:25:36.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.315 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131248+0000", 2021-03-15T22:25:36.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131248+0000", 2021-03-15T22:25:36.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9e-06 2021-03-15T22:25:36.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131250+0000", 2021-03-15T22:25:36.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3710000000000001e-06 2021-03-15T22:25:36.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131251+0000", 2021-03-15T22:25:36.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1639999999999998e-06 2021-03-15T22:25:36.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131257+0000", 2021-03-15T22:25:36.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.403e-06 2021-03-15T22:25:36.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131262+0000", 2021-03-15T22:25:36.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014190439999999999 2021-03-15T22:25:36.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.132681+0000", 2021-03-15T22:25:36.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013826999999999999 2021-03-15T22:25:36.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.132819+0000", 2021-03-15T22:25:36.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023518200000000001 2021-03-15T22:25:36.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:25:36.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.133054+0000", 2021-03-15T22:25:36.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044182889999999997 2021-03-15T22:25:36.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:36.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.137472+0000", 2021-03-15T22:25:36.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025551509999999999 2021-03-15T22:25:36.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.140027+0000", 2021-03-15T22:25:36.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.030306403999999999 2021-03-15T22:25:36.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170334+0000", 2021-03-15T22:25:36.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1277000000000002e-05 2021-03-15T22:25:36.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:36.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170405+0000", 2021-03-15T22:25:36.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7476000000000002e-05 2021-03-15T22:25:36.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170473+0000", 2021-03-15T22:25:36.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039224930999999998 2021-03-15T22:25:36.330 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.331 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.331 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.332 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3663 7.3 7:c7825e8f:::smithi07694878-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e385)", 2021-03-15T22:25:36.332 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.131268+0000", 2021-03-15T22:25:36.332 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.899317240000002, 2021-03-15T22:25:36.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039463573000000002, 2021-03-15T22:25:36.333 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.333 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:36.333 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.333 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.333 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.334 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3663 2021-03-15T22:25:36.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.334 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131268+0000", 2021-03-15T22:25:36.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131268+0000", 2021-03-15T22:25:36.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2619999999999999e-06 2021-03-15T22:25:36.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131269+0000", 2021-03-15T22:25:36.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4309999999999999e-06 2021-03-15T22:25:36.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131270+0000", 2021-03-15T22:25:36.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4150000000000004e-06 2021-03-15T22:25:36.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131277+0000", 2021-03-15T22:25:36.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0239999999999999e-06 2021-03-15T22:25:36.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131281+0000", 2021-03-15T22:25:36.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002198247 2021-03-15T22:25:36.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.133479+0000", 2021-03-15T22:25:36.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125449 2021-03-15T22:25:36.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:36.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.133604+0000", 2021-03-15T22:25:36.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036921066000000002 2021-03-15T22:25:36.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170525+0000", 2021-03-15T22:25:36.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1867999999999995e-05 2021-03-15T22:25:36.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170617+0000", 2021-03-15T22:25:36.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000113811 2021-03-15T22:25:36.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170731+0000", 2021-03-15T22:25:36.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039463573000000002 2021-03-15T22:25:36.346 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.347 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.347 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.347 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3675 7.3 7:c00fa989:::smithi07694878-50:head [write 453921~729545,stat] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:25:36.348 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.189699+0000", 2021-03-15T22:25:36.348 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.840885307000001, 2021-03-15T22:25:36.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043512464000000001, 2021-03-15T22:25:36.348 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.349 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:36.349 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.349 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.349 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.350 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3675 2021-03-15T22:25:36.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.350 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189699+0000", 2021-03-15T22:25:36.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189699+0000", 2021-03-15T22:25:36.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015413499999999999 2021-03-15T22:25:36.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189854+0000", 2021-03-15T22:25:36.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1420000000000001e-06 2021-03-15T22:25:36.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189859+0000", 2021-03-15T22:25:36.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3388999999999999e-05 2021-03-15T22:25:36.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189882+0000", 2021-03-15T22:25:36.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8409999999999995e-06 2021-03-15T22:25:36.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189892+0000", 2021-03-15T22:25:36.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2676999999999996e-05 2021-03-15T22:25:36.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189955+0000", 2021-03-15T22:25:36.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119745 2021-03-15T22:25:36.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.190074+0000", 2021-03-15T22:25:36.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026131199999999998 2021-03-15T22:25:36.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:25:36.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.190336+0000", 2021-03-15T22:25:36.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034304180000000002 2021-03-15T22:25:36.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.193766+0000", 2021-03-15T22:25:36.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039036350999999997 2021-03-15T22:25:36.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:36.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.232803+0000", 2021-03-15T22:25:36.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030217699999999999 2021-03-15T22:25:36.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233105+0000", 2021-03-15T22:25:36.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9181999999999999e-05 2021-03-15T22:25:36.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:36.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233144+0000", 2021-03-15T22:25:36.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8095000000000004e-05 2021-03-15T22:25:36.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233212+0000", 2021-03-15T22:25:36.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043512464000000001 2021-03-15T22:25:36.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.365 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.365 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.366 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3676 7.3 7:c00fa989:::smithi07694878-50:head [write 1584325~621164] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:25:36.366 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.195497+0000", 2021-03-15T22:25:36.366 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.835087518999998, 2021-03-15T22:25:36.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042481251999999997, 2021-03-15T22:25:36.367 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.367 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:36.367 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.367 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.368 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.368 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3676 2021-03-15T22:25:36.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.368 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195497+0000", 2021-03-15T22:25:36.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195497+0000", 2021-03-15T22:25:36.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013062600000000001 2021-03-15T22:25:36.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195628+0000", 2021-03-15T22:25:36.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9829999999999998e-06 2021-03-15T22:25:36.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195632+0000", 2021-03-15T22:25:36.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6456e-05 2021-03-15T22:25:36.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195648+0000", 2021-03-15T22:25:36.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7849999999999997e-06 2021-03-15T22:25:36.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195654+0000", 2021-03-15T22:25:36.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6258999999999999e-05 2021-03-15T22:25:36.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195700+0000", 2021-03-15T22:25:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3122999999999998e-05 2021-03-15T22:25:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195774+0000", 2021-03-15T22:25:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037474535000000003 2021-03-15T22:25:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233248+0000", 2021-03-15T22:25:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3729e-05 2021-03-15T22:25:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233302+0000", 2021-03-15T22:25:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000120444 2021-03-15T22:25:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:25:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233422+0000", 2021-03-15T22:25:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020581520000000002 2021-03-15T22:25:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.235480+0000", 2021-03-15T22:25:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1943000000000001e-05 2021-03-15T22:25:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.235492+0000", 2021-03-15T22:25:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023698780000000002 2021-03-15T22:25:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.237862+0000", 2021-03-15T22:25:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0804999999999998e-05 2021-03-15T22:25:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.237923+0000", 2021-03-15T22:25:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5534000000000001e-05 2021-03-15T22:25:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.237979+0000", 2021-03-15T22:25:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042481251999999997 2021-03-15T22:25:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3677 7.3 7:c00fa989:::smithi07694878-50:head [setxattr _header (58) in=7b,truncate 2453921] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:25:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.195666+0000", 2021-03-15T22:25:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.834919184, 2021-03-15T22:25:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042508698999999997, 2021-03-15T22:25:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3677 2021-03-15T22:25:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195666+0000", 2021-03-15T22:25:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195666+0000", 2021-03-15T22:25:36.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.385e-06 2021-03-15T22:25:36.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195668+0000", 2021-03-15T22:25:36.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4890000000000001e-06 2021-03-15T22:25:36.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195669+0000", 2021-03-15T22:25:36.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9700000000000001e-06 2021-03-15T22:25:36.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195673+0000", 2021-03-15T22:25:36.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6120000000000001e-06 2021-03-15T22:25:36.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195676+0000", 2021-03-15T22:25:36.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012614700000000001 2021-03-15T22:25:36.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195802+0000", 2021-03-15T22:25:36.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6866000000000001e-05 2021-03-15T22:25:36.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:36.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195859+0000", 2021-03-15T22:25:36.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038488834999999999 2021-03-15T22:25:36.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.234348+0000", 2021-03-15T22:25:36.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6688999999999999e-05 2021-03-15T22:25:36.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.234405+0000", 2021-03-15T22:25:36.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000111799 2021-03-15T22:25:36.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:25:36.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.234516+0000", 2021-03-15T22:25:36.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00099373400000000002 2021-03-15T22:25:36.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:25:36.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.235510+0000", 2021-03-15T22:25:36.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3916000000000003e-05 2021-03-15T22:25:36.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.235574+0000", 2021-03-15T22:25:36.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024673519999999999 2021-03-15T22:25:36.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:36.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238041+0000", 2021-03-15T22:25:36.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7310000000000004e-05 2021-03-15T22:25:36.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:25:36.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238109+0000", 2021-03-15T22:25:36.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5594999999999998e-05 2021-03-15T22:25:36.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238174+0000", 2021-03-15T22:25:36.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042508698999999997 2021-03-15T22:25:36.406 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.406 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.406 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.407 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3678 7.3 7:c00fa989:::smithi07694878-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e385)", 2021-03-15T22:25:36.407 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.195704+0000", 2021-03-15T22:25:36.407 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 35.834881058999997, 2021-03-15T22:25:36.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042827408999999997, 2021-03-15T22:25:36.408 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:25:36.408 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:25:36.408 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:25:36.408 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:25:36.409 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:36.409 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3678 2021-03-15T22:25:36.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.409 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:25:36.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:25:36.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195704+0000", 2021-03-15T22:25:36.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:25:36.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:25:36.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195704+0000", 2021-03-15T22:25:36.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.184e-06 2021-03-15T22:25:36.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:25:36.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195705+0000", 2021-03-15T22:25:36.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4649999999999999e-06 2021-03-15T22:25:36.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:25:36.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195706+0000", 2021-03-15T22:25:36.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7109999999999998e-06 2021-03-15T22:25:36.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:25:36.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195713+0000", 2021-03-15T22:25:36.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2100000000000002e-06 2021-03-15T22:25:36.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:25:36.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195716+0000", 2021-03-15T22:25:36.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016275600000000001 2021-03-15T22:25:36.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195879+0000", 2021-03-15T22:25:36.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9199000000000002e-05 2021-03-15T22:25:36.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:36.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195918+0000", 2021-03-15T22:25:36.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038899652999999999 2021-03-15T22:25:36.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.234818+0000", 2021-03-15T22:25:36.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1717999999999993e-05 2021-03-15T22:25:36.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:36.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.234890+0000", 2021-03-15T22:25:36.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003332939 2021-03-15T22:25:36.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:25:36.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238223+0000", 2021-03-15T22:25:36.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2285000000000002e-05 2021-03-15T22:25:36.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:25:36.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238305+0000", 2021-03-15T22:25:36.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022628899999999999 2021-03-15T22:25:36.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:25:36.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:25:36.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:25:36.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238531+0000", 2021-03-15T22:25:36.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042827408999999997 2021-03-15T22:25:36.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.423 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:25:36.424 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:25:36.424 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:25:36.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:36.426 INFO:tasks.rados.rados.0.smithi076.stdout:1533: done (8 left) 2021-03-15T22:25:36.427 INFO:tasks.rados.rados.0.smithi076.stdout:1534: done (7 left) 2021-03-15T22:25:36.427 INFO:tasks.rados.rados.0.smithi076.stdout:1535: done (6 left) 2021-03-15T22:25:36.427 INFO:tasks.rados.rados.0.smithi076.stdout:1537: done (5 left) 2021-03-15T22:25:36.427 INFO:tasks.rados.rados.0.smithi076.stdout:1538: done (4 left) 2021-03-15T22:25:36.428 INFO:tasks.rados.rados.0.smithi076.stdout:1539: done (3 left) 2021-03-15T22:25:36.428 INFO:tasks.rados.rados.0.smithi076.stdout:1540: done (2 left) 2021-03-15T22:25:36.428 INFO:tasks.rados.rados.0.smithi076.stdout:1541: copy_from oid 3 from oid 24 current snap is 172 2021-03-15T22:25:36.428 INFO:tasks.rados.rados.0.smithi076.stdout:1542: write (excl) oid 29 current snap is 172 2021-03-15T22:25:36.428 INFO:tasks.rados.rados.0.smithi076.stdout:1542: seq_num 543 ranges {726201=721270,2141840=578191} 2021-03-15T22:25:36.429 INFO:tasks.rados.rados.0.smithi076.stdout:1542: writing smithi07694878-29 from 726201 to 1447471 tid 1 2021-03-15T22:25:36.429 INFO:tasks.rados.rados.0.smithi076.stdout:1542: writing smithi07694878-29 from 2141840 to 2720031 tid 2 2021-03-15T22:25:36.429 INFO:tasks.rados.rados.0.smithi076.stdout:1543: read oid 35 snap -1 2021-03-15T22:25:36.429 INFO:tasks.rados.rados.0.smithi076.stdout:1543: expect deleted 2021-03-15T22:25:36.430 INFO:tasks.rados.rados.0.smithi076.stdout:1544: snap_remove snap 166 2021-03-15T22:25:36.430 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:36 smithi076 conmon[56514]: cluster 2021-03-15T22:25:35.055141+0000 mgr.y (mgr.14144) 620 : cluster [DBG] pgmap v756: 138 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 8 peering, 128 active+clean; 308 MiB data, 917 MiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 0 B/s wr, 4 op/s; 95/1425 objects degraded (6.667%); 4/1425 objects misplaced (0.281%); 593 KiB/s, 8 objects/s recovering 2021-03-15T22:25:36.430 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:36 smithi076 conmon[56514]: cluster 2021-03-15T22:25:35.122327+0000 mon.a (mon.0) 1240 : cluster [WRN] Health check failed: Degraded data redundancy: 95/1425 objects degraded (6.667%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:25:36.431 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:36 smithi076 conmon[56514]: cluster 2021-03-15T22:25:36.109531+0000 mon.a (mon.0) 1241 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2021-03-15T22:25:36.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:36 smithi076 conmon[62156]: cluster 2021-03-15T22:25:35.055141+0000 2021-03-15T22:25:36.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:36 smithi076 conmon[62156]: mgr.y (mgr.14144) 620 : cluster [DBG] pgmap v756: 138 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 8 peering, 128 active+clean; 308 MiB data, 917 MiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 0 B/s wr, 4 op/s; 95/1425 objects degraded (6.667%); 4/1425 objects misplaced (0.281%); 593 KiB/s, 8 objects/s recovering 2021-03-15T22:25:36.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:36 smithi076 conmon[62156]: cluster 2021-03-15T22:25:35.122327+0000 mon.a 2021-03-15T22:25:36.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:36 smithi076 conmon[62156]: (mon.0) 1240 : cluster [WRN] Health check failed: Degraded data redundancy: 95/1425 objects degraded (6.667%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:25:36.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:36 smithi076 conmon[62156]: cluster 2021-03-15T22:25:36.109531 2021-03-15T22:25:36.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:36 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1241 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2021-03-15T22:25:36.445 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:36.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:36.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:36.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:36 smithi125 conmon[56739]: cluster 2021-03-15T22:25:35.055141 2021-03-15T22:25:36.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:36 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 620 : cluster [DBG] pgmap v756: 138 pgs: 1 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 8 peering, 128 active+clean; 308 MiB data, 917 MiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 0 B/s wr, 4 op/s; 95/1425 objects degraded (6.667%); 4/1425 objects misplaced (0.281%); 593 KiB/s, 8 objects/s recovering 2021-03-15T22:25:36.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:36 smithi125 conmon[56739]: cluster 2021-03-15T22:25:35 2021-03-15T22:25:36.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:36 smithi125 conmon[56739]: .122327+0000 mon.a (mon.0) 1240 : cluster [WRN] Health check failed: Degraded data redundancy: 95/1425 objects degraded (6.667%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:25:36.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:36 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:25:36.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:36 smithi125 conmon[56739]: :25:36.109531+0000 mon.a (mon. 2021-03-15T22:25:36.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:36 smithi125 conmon[56739]: 0) 1241 : cluster [DBG] osdmap e421: 8 total, 8 up, 7 in 2021-03-15T22:25:36.548 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:36.569 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:36.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:36.570 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:36.671 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:36.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:36.696 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:36.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:36.798 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:36.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:36.824 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:36.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:36.926 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:36.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:36.949 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:36.949 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:37.051 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:37.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:37.073 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:37.073 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:37.117 INFO:tasks.rados.rados.0.smithi076.stdout:1544: done (5 left) 2021-03-15T22:25:37.118 INFO:tasks.rados.rados.0.smithi076.stdout:1545: write (excl) oid 17 current snap is 172 2021-03-15T22:25:37.118 INFO:tasks.rados.rados.0.smithi076.stdout:1545: seq_num 544 ranges {743008=796519,2200322=721849} 2021-03-15T22:25:37.131 INFO:tasks.rados.rados.0.smithi076.stdout:1545: writing smithi07694878-17 from 743008 to 1539527 tid 1 2021-03-15T22:25:37.140 INFO:tasks.rados.rados.0.smithi076.stdout:1545: writing smithi07694878-17 from 2200322 to 2922171 tid 2 2021-03-15T22:25:37.140 INFO:tasks.rados.rados.0.smithi076.stdout:1546: snap_create 2021-03-15T22:25:37.140 INFO:tasks.rados.rados.0.smithi076.stdout:1530: finishing write tid 1 to smithi07694878-2 2021-03-15T22:25:37.141 INFO:tasks.rados.rados.0.smithi076.stdout:1536: finishing write tid 1 to smithi07694878-25 2021-03-15T22:25:37.141 INFO:tasks.rados.rados.0.smithi076.stdout:1536: finishing write tid 2 to smithi07694878-25 2021-03-15T22:25:37.141 INFO:tasks.rados.rados.0.smithi076.stdout:1536: finishing write tid 3 to smithi07694878-25 2021-03-15T22:25:37.141 INFO:tasks.rados.rados.0.smithi076.stdout:1542: finishing write tid 1 to smithi07694878-29 2021-03-15T22:25:37.141 INFO:tasks.rados.rados.0.smithi076.stdout:1536: finishing write tid 4 to smithi07694878-25 2021-03-15T22:25:37.142 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 355 (ObjNum 542 snap 171 seq_num 542) dirty exists 2021-03-15T22:25:37.142 INFO:tasks.rados.rados.0.smithi076.stdout:1536: left oid 25 (ObjNum 542 snap 171 seq_num 542) 2021-03-15T22:25:37.142 INFO:tasks.rados.rados.0.smithi076.stdout:1530: finishing write tid 2 to smithi07694878-2 2021-03-15T22:25:37.142 INFO:tasks.rados.rados.0.smithi076.stdout:1530: finishing write tid 3 to smithi07694878-2 2021-03-15T22:25:37.143 INFO:tasks.rados.rados.0.smithi076.stdout:1530: finishing write tid 4 to smithi07694878-2 2021-03-15T22:25:37.143 INFO:tasks.rados.rados.0.smithi076.stdout:1530: finishing write tid 5 to smithi07694878-2 2021-03-15T22:25:37.143 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 358 (ObjNum 539 snap 171 seq_num 539) dirty exists 2021-03-15T22:25:37.144 INFO:tasks.rados.rados.0.smithi076.stdout:1530: left oid 2 (ObjNum 539 snap 171 seq_num 539) 2021-03-15T22:25:37.144 INFO:tasks.rados.rados.0.smithi076.stdout:1542: finishing write tid 2 to smithi07694878-29 2021-03-15T22:25:37.144 INFO:tasks.rados.rados.0.smithi076.stdout:1542: finishing write tid 3 to smithi07694878-29 2021-03-15T22:25:37.144 INFO:tasks.rados.rados.0.smithi076.stdout:1542: finishing write tid 4 to smithi07694878-29 2021-03-15T22:25:37.145 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 960 (ObjNum 543 snap 172 seq_num 543) dirty exists 2021-03-15T22:25:37.145 INFO:tasks.rados.rados.0.smithi076.stdout:1542: left oid 29 (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:25:37.145 INFO:tasks.rados.rados.0.smithi076.stdout:1541: finishing copy_from to smithi07694878-3 2021-03-15T22:25:37.146 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 119 (ObjNum 514 snap 155 seq_num 514) dirty exists 2021-03-15T22:25:37.146 INFO:tasks.rados.rados.0.smithi076.stdout:1541: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:25:37.146 INFO:tasks.rados.rados.0.smithi076.stdout:1545: finishing write tid 1 to smithi07694878-17 2021-03-15T22:25:37.148 INFO:tasks.rados.rados.0.smithi076.stdout:1545: finishing write tid 2 to smithi07694878-17 2021-03-15T22:25:37.148 INFO:tasks.rados.rados.0.smithi076.stdout:1545: finishing write tid 3 to smithi07694878-17 2021-03-15T22:25:37.149 INFO:tasks.rados.rados.0.smithi076.stdout:1545: finishing write tid 4 to smithi07694878-17 2021-03-15T22:25:37.149 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 311 (ObjNum 544 snap 172 seq_num 544) dirty exists 2021-03-15T22:25:37.149 INFO:tasks.rados.rados.0.smithi076.stdout:1545: left oid 17 (ObjNum 544 snap 172 seq_num 544) 2021-03-15T22:25:37.174 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:37.196 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:37.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:37.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:37.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:37.320 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:37.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:37.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:37.336 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:37.423 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:37.445 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:37.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:37.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:37.501 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-15T22:25:37.506 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:25:37.523 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:25:37.541 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:25:37.548 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:37.557 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:25:37.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:37.573 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:37.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:37.576 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:25:37.597 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:25:37.614 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:25:37.634 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:25:37.634 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:25:37.635 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:25:37.635 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:25:37.635 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:25:37.635 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:25:37.636 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:25:37.636 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:25:37.636 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:25:37.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:37.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:37.702 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:37.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:37.787 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:25:37.803 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:37.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:37.826 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:37.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:37.928 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:37.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:37.952 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:37.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:38.053 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:38.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:38.076 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:38.077 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:38.138 INFO:tasks.rados.rados.0.smithi076.stdout:1530: done (6 left) 2021-03-15T22:25:38.138 INFO:tasks.rados.rados.0.smithi076.stdout:1536: done (5 left) 2021-03-15T22:25:38.138 INFO:tasks.rados.rados.0.smithi076.stdout:1541: done (4 left) 2021-03-15T22:25:38.138 INFO:tasks.rados.rados.0.smithi076.stdout:1542: done (3 left) 2021-03-15T22:25:38.139 INFO:tasks.rados.rados.0.smithi076.stdout:1543: done (2 left) 2021-03-15T22:25:38.139 INFO:tasks.rados.rados.0.smithi076.stdout:1545: done (1 left) 2021-03-15T22:25:38.139 INFO:tasks.rados.rados.0.smithi076.stdout:1546: done (0 left) 2021-03-15T22:25:38.139 INFO:tasks.rados.rados.0.smithi076.stdout:1547: snap_create 2021-03-15T22:25:38.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:38.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:38.202 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:38.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:38.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:38 smithi076 conmon[56514]: cluster 2021-03-15T22:25:37.055817+0000 2021-03-15T22:25:38.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:38 smithi076 conmon[56514]: mgr.y (mgr.14144) 621 : cluster [DBG] pgmap v759: 138 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 11 peering, 124 active+clean; 311 MiB data, 1011 MiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 620 KiB/s wr, 7 op/s; 95/1437 objects degraded (6.611%); 7/1437 objects misplaced (0.487%); 11 MiB/s, 36 objects/s recovering 2021-03-15T22:25:38.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:38 smithi076 conmon[56514]: cluster 2021-03-15T22:25:37.112655+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2021-03-15T22:25:38.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:38 smithi076 conmon[56514]: cluster 2021-03-15T22:25:37.458286+0000 osd.6 (osd.6) 68 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:25:38.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:38 smithi076 conmon[56514]: audit 2021-03-15T22:25:37.486627+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? 172.21.15.76:0/2570334942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:38.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:38 smithi076 conmon[62156]: cluster 2021-03-15T22:25:37.055817+0000 mgr.y (mgr 2021-03-15T22:25:38.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:38 smithi076 conmon[62156]: .14144) 621 : cluster [DBG] pgmap v759: 138 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 11 peering, 124 active+clean; 311 MiB data, 1011 MiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 620 KiB/s wr, 7 op/s; 95/1437 objects degraded (6.611%); 7/1437 objects misplaced (0.487%); 11 MiB/s, 36 objects/s recovering 2021-03-15T22:25:38.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:38 smithi076 conmon[62156]: cluster 2021-03-15T22:25:37.112655+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2021-03-15T22:25:38.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:38 smithi076 conmon[62156]: cluster 2021-03-15T22:25:37.458286+0000 osd.6 (osd.6) 68 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:25:38.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:38 smithi076 conmon[62156]: audit 2021-03-15T22:25:37.486627+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? 172.21.15.76:0/2570334942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:38.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:38 smithi125 conmon[56739]: cluster 2021-03-15T22:25:37.055817 2021-03-15T22:25:38.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:38 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 621 : cluster [DBG] pgmap v759: 138 pgs: 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 11 peering, 124 active+clean; 311 MiB data, 1011 MiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 620 KiB/s wr, 7 op/s; 95/1437 objects degraded (6.611%); 7/1437 objects misplaced (0.487%); 11 MiB/s, 36 objects/s recovering 2021-03-15T22:25:38.277 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:38 smithi125 conmon[56739]: cluster 2021-03-15T22:25:37.112655+0000 mon.a (mon.0) 1242 : cluster [DBG] osdmap e422: 8 total, 8 up, 7 in 2021-03-15T22:25:38.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:38 smithi125 conmon[56739]: cluster 2021-03-15T22:25:37.458286+0000 osd.6 (osd.6) 68 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:25:38.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:38 smithi125 conmon[56739]: audit 2021-03-15T22:25:38.278 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:38 smithi125 conmon[56739]: 2021-03-15T22:25:37.486627+0000 mon.c (mon.1) 37 : audit [DBG] from='client.? 172.21.15.76:0/2570334942' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:38.304 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:38.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:38.327 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:38.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:38.428 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:38.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:38.452 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:38.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:38.554 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:38.576 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:38.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:38.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:38.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:38.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:38.702 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:38.703 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:38.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:38.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:38.828 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:38.828 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:38.930 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:38.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:38.953 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:38.954 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:39.055 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:39.077 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:39.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:39.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:39.141 INFO:tasks.rados.rados.0.smithi076.stdout:1547: done (0 left) 2021-03-15T22:25:39.142 INFO:tasks.rados.rados.0.smithi076.stdout:1548: copy_from oid 7 from oid 29 current snap is 174 2021-03-15T22:25:39.142 INFO:tasks.rados.rados.0.smithi076.stdout:1549: copy_from oid 34 from oid 9 current snap is 174 2021-03-15T22:25:39.142 INFO:tasks.rados.rados.0.smithi076.stdout:1550: delete oid 20 current snap is 174 2021-03-15T22:25:39.147 INFO:tasks.rados.rados.0.smithi076.stdout:1550: done (2 left) 2021-03-15T22:25:39.147 INFO:tasks.rados.rados.0.smithi076.stdout:1551: delete oid 2 current snap is 174 2021-03-15T22:25:39.149 INFO:tasks.rados.rados.0.smithi076.stdout:1551: done (2 left) 2021-03-15T22:25:39.149 INFO:tasks.rados.rados.0.smithi076.stdout:1552: write (excl) oid 23 current snap is 174 2021-03-15T22:25:39.149 INFO:tasks.rados.rados.0.smithi076.stdout:1552: seq_num 547 ranges {793429=622266,1859415=469176,2972024=221405} 2021-03-15T22:25:39.157 INFO:tasks.rados.rados.0.smithi076.stdout:1552: writing smithi07694878-23 from 793429 to 1415695 tid 1 2021-03-15T22:25:39.163 INFO:tasks.rados.rados.0.smithi076.stdout:1552: writing smithi07694878-23 from 1859415 to 2328591 tid 2 2021-03-15T22:25:39.168 INFO:tasks.rados.rados.0.smithi076.stdout:1552: writing smithi07694878-23 from 2972024 to 3193429 tid 3 2021-03-15T22:25:39.169 INFO:tasks.rados.rados.0.smithi076.stdout:1553: snap_create 2021-03-15T22:25:39.169 INFO:tasks.rados.rados.0.smithi076.stdout:1549: finishing copy_from to smithi07694878-34 2021-03-15T22:25:39.169 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 695 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:25:39.170 INFO:tasks.rados.rados.0.smithi076.stdout:1549: finishing copy_from racing read to smithi07694878-34 2021-03-15T22:25:39.170 INFO:tasks.rados.rados.0.smithi076.stdout:1552: finishing write tid 1 to smithi07694878-23 2021-03-15T22:25:39.170 INFO:tasks.rados.rados.0.smithi076.stdout:1548: finishing copy_from to smithi07694878-7 2021-03-15T22:25:39.170 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 960 (ObjNum 543 snap 172 seq_num 543) dirty exists 2021-03-15T22:25:39.171 INFO:tasks.rados.rados.0.smithi076.stdout:1548: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:25:39.172 INFO:tasks.rados.rados.0.smithi076.stdout:1552: finishing write tid 2 to smithi07694878-23 2021-03-15T22:25:39.172 INFO:tasks.rados.rados.0.smithi076.stdout:1552: finishing write tid 3 to smithi07694878-23 2021-03-15T22:25:39.173 INFO:tasks.rados.rados.0.smithi076.stdout:1552: finishing write tid 4 to smithi07694878-23 2021-03-15T22:25:39.173 INFO:tasks.rados.rados.0.smithi076.stdout:1552: finishing write tid 5 to smithi07694878-23 2021-03-15T22:25:39.173 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 770 (ObjNum 547 snap 174 seq_num 547) dirty exists 2021-03-15T22:25:39.173 INFO:tasks.rados.rados.0.smithi076.stdout:1552: left oid 23 (ObjNum 547 snap 174 seq_num 547) 2021-03-15T22:25:39.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:39.203 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:39.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:39.305 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:39.329 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:39.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:39.386 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:25:39.389 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:25:39.406 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:39 smithi125 conmon[56739]: cluster 2021-03-15T22:25:39.407 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:39 smithi125 conmon[56739]: 2021-03-15T22 2021-03-15T22:25:39.407 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:39 smithi125 conmon[56739]: :25:37.707803+0000 osd.2 (osd.2) 71 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:25:39.407 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:39 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:25:39.407 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:39 smithi125 conmon[56739]: T22:25:38.136317+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2021-03-15T22:25:39.429 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:39 smithi076 conmon[56514]: cluster 2021-03-15T22:25:37.707803+0000 osd.2 (osd.2) 71 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:25:39.429 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:39 smithi076 conmon[56514]: cluster 2021-03-15T22:25:38.136317+0000 mon.a (mon.0) 1243 : cluster [DBG] osdmap e423: 8 total, 8 up, 7 in 2021-03-15T22:25:39.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:39 smithi076 conmon[62156]: cluster 2021-03-15T22:25:37.707803+0000 osd.2 (osd. 2021-03-15T22:25:39.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:39 smithi076 conmon[62156]: 2) 71 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:25:39.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:39 smithi076 conmon[62156]: cluster 2021-03-15T22:25:38.136317+0000 mon.a (mon.0) 1243 : cluster 2021-03-15T22:25:39.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:39 smithi076 conmon[62156]: [DBG] osdmap e423: 8 total, 8 up, 7 in 2021-03-15T22:25:39.431 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:39.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:39.458 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:39.460 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:39.562 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:39.585 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:39.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:39.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:39.611 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:25:39.611 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:25:39.688 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:39.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:39.713 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:39.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:39.815 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:39.837 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:39.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:39.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:39.940 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:39.962 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:39.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:39.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:40.039 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:25:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:25:40.066 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:40.087 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:40.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:40.103 INFO:tasks.rados.rados.0.smithi076.stdout:1548: done (3 left) 2021-03-15T22:25:40.103 INFO:tasks.rados.rados.0.smithi076.stdout:1549: done (2 left) 2021-03-15T22:25:40.104 INFO:tasks.rados.rados.0.smithi076.stdout:1552: done (1 left) 2021-03-15T22:25:40.104 INFO:tasks.rados.rados.0.smithi076.stdout:1553: done (0 left) 2021-03-15T22:25:40.104 INFO:tasks.rados.rados.0.smithi076.stdout:1554: copy_from oid 46 from oid 1 current snap is 175 2021-03-15T22:25:40.104 INFO:tasks.rados.rados.0.smithi076.stdout:1555: delete oid 8 current snap is 175 2021-03-15T22:25:40.107 INFO:tasks.rados.rados.0.smithi076.stdout:1555: done (1 left) 2021-03-15T22:25:40.108 INFO:tasks.rados.rados.0.smithi076.stdout:1556: read oid 45 snap -1 2021-03-15T22:25:40.108 INFO:tasks.rados.rados.0.smithi076.stdout:1556: expect (ObjNum 527 snap 159 seq_num 527) 2021-03-15T22:25:40.108 INFO:tasks.rados.rados.0.smithi076.stdout:1557: write (excl) oid 17 current snap is 175 2021-03-15T22:25:40.108 INFO:tasks.rados.rados.0.smithi076.stdout:1557: seq_num 549 ranges {427043=489117,1576379=472845,2626870=600173} 2021-03-15T22:25:40.112 INFO:tasks.rados.rados.0.smithi076.stdout:1557: writing smithi07694878-17 from 427043 to 916160 tid 1 2021-03-15T22:25:40.120 INFO:tasks.rados.rados.0.smithi076.stdout:1557: writing smithi07694878-17 from 1576379 to 2049224 tid 2 2021-03-15T22:25:40.127 INFO:tasks.rados.rados.0.smithi076.stdout:1557: writing smithi07694878-17 from 2626870 to 3227043 tid 3 2021-03-15T22:25:40.128 INFO:tasks.rados.rados.0.smithi076.stdout:1558: copy_from oid 14 from oid 43 current snap is 175 2021-03-15T22:25:40.128 INFO:tasks.rados.rados.0.smithi076.stdout:1556: expect (ObjNum 527 snap 159 seq_num 527) 2021-03-15T22:25:40.166 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":763,"stamp":"2021-03-15T22:25:39.135139+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":329286803,"num_objects":481,"num_object_clones":117,"num_object_copies":1443,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":99,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":481,"num_whiteouts":13,"num_read":4522,"num_read_kb":3031453,"num_write":2747,"num_write_kb":820473,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1506,"num_bytes_recovered":399513229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6570,"ondisk_log_size":6570,"up":414,"acting":410,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":437,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1030424,"kb_used_data":851872,"kb_used_omap":0,"kb_used_meta":178496,"kb_avail":655185640,"statfs":{"total":671965249536,"available":670910095360,"internally_reserved":0,"allocated":872316928,"data_stored":1196373776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":182779904},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6696801,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":29,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":34,"num_read_kb":39404,"num_write":20,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":156,"num_bytes_recovered":44489131,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.038028"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"381","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101037+0000","last_change":"2021-03-15T22:23:33.729004+0000","last_active":"2021-03-15T22:25:35.101037+0000","last_peered":"2021-03-15T22:25:35.101037+0000","last_clean":"2021-03-15T22:25:35.101037+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:25:35.101037+0000","last_undegraded":"2021-03-15T22:25:35.101037+0000","last_fullsized":"2021-03-15T22:25:35.101037+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:33.728929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:33.728929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"88","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115394+0000","last_change":"2021-03-15T22:24:31.865035+0000","last_active":"2021-03-15T22:25:36.115394+0000","last_peered":"2021-03-15T22:25:36.115394+0000","last_clean":"2021-03-15T22:25:36.115394+0000","last_became_active":"2021-03-15T22:24:31.103578+0000","last_became_peered":"2021-03-15T22:24:31.103578+0000","last_unstale":"2021-03-15T22:25:36.115394+0000","last_undegraded":"2021-03-15T22:25:36.115394+0000","last_fullsized":"2021-03-15T22:25:36.115394+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:23:59.500680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:59.500680+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"413","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.100963+0000","last_change":"2021-03-15T22:23:26.743905+0000","last_active":"2021-03-15T22:25:35.100963+0000","last_peered":"2021-03-15T22:25:35.100963+0000","last_clean":"2021-03-15T22:25:35.100963+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:25:35.100963+0000","last_undegraded":"2021-03-15T22:25:35.100963+0000","last_fullsized":"2021-03-15T22:25:35.100963+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:26.743811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.743811+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"13","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.100299+0000","last_change":"2021-03-15T22:25:33.149020+0000","last_active":"2021-03-15T22:25:35.100299+0000","last_peered":"2021-03-15T22:25:35.100299+0000","last_clean":"2021-03-15T22:25:35.100299+0000","last_became_active":"2021-03-15T22:25:33.148710+0000","last_became_peered":"2021-03-15T22:25:33.148710+0000","last_unstale":"2021-03-15T22:25:35.100299+0000","last_undegraded":"2021-03-15T22:25:35.100299+0000","last_fullsized":"2021-03-15T22:25:35.100299+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:06.177754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.177754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"13","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.105609+0000","last_change":"2021-03-15T22:25:33.147017+0000","last_active":"2021-03-15T22:25:35.105609+0000","last_peered":"2021-03-15T22:25:35.105609+0000","last_clean":"2021-03-15T22:25:35.105609+0000","last_became_active":"2021-03-15T22:25:33.146697+0000","last_became_peered":"2021-03-15T22:25:33.146697+0000","last_unstale":"2021-03-15T22:25:35.105609+0000","last_undegraded":"2021-03-15T22:25:35.105609+0000","last_fullsized":"2021-03-15T22:25:35.105609+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:15.333746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.333746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"39","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101522+0000","last_change":"2021-03-15T22:25:33.175542+0000","last_active":"2021-03-15T22:25:35.101522+0000","last_peered":"2021-03-15T22:25:35.101522+0000","last_clean":"2021-03-15T22:25:35.101522+0000","last_became_active":"2021-03-15T22:25:33.137906+0000","last_became_peered":"2021-03-15T22:25:33.137906+0000","last_unstale":"2021-03-15T22:25:35.101522+0000","last_undegraded":"2021-03-15T22:25:35.101522+0000","last_fullsized":"2021-03-15T22:25:35.101522+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:23:42.144821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.144821+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":21,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"77","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.114898+0000","last_change":"2021-03-15T22:24:31.109745+0000","last_active":"2021-03-15T22:25:36.114898+0000","last_peered":"2021-03-15T22:25:36.114898+0000","last_clean":"2021-03-15T22:25:36.114898+0000","last_became_active":"2021-03-15T22:24:31.108652+0000","last_became_peered":"2021-03-15T22:24:31.108652+0000","last_unstale":"2021-03-15T22:25:36.114898+0000","last_undegraded":"2021-03-15T22:25:36.114898+0000","last_fullsized":"2021-03-15T22:25:36.114898+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:35.677439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:35.677439+0000","last_clean_scrub_stamp":"2021-03-15T22:23:35.677439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"387","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116452+0000","last_change":"2021-03-15T22:25:16.598073+0000","last_active":"2021-03-15T22:25:36.116452+0000","last_peered":"2021-03-15T22:25:36.116452+0000","last_clean":"2021-03-15T22:25:36.116452+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:25:36.116452+0000","last_undegraded":"2021-03-15T22:25:36.116452+0000","last_fullsized":"2021-03-15T22:25:36.116452+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.597998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.597998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"490","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.102218+0000","last_change":"2021-03-15T22:24:31.776304+0000","last_active":"2021-03-15T22:25:35.102218+0000","last_peered":"2021-03-15T22:25:35.102218+0000","last_clean":"2021-03-15T22:25:35.102218+0000","last_became_active":"2021-03-15T22:24:31.083373+0000","last_became_peered":"2021-03-15T22:24:31.083373+0000","last_unstale":"2021-03-15T22:25:35.102218+0000","last_undegraded":"2021-03-15T22:25:35.102218+0000","last_fullsized":"2021-03-15T22:25:35.102218+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:24:10.979322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:10.979322+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"96","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115119+0000","last_change":"2021-03-15T22:25:33.143128+0000","last_active":"2021-03-15T22:25:36.115119+0000","last_peered":"2021-03-15T22:25:36.115119+0000","last_clean":"2021-03-15T22:25:36.115119+0000","last_became_active":"2021-03-15T22:25:33.142493+0000","last_became_peered":"2021-03-15T22:25:33.142493+0000","last_unstale":"2021-03-15T22:25:36.115119+0000","last_undegraded":"2021-03-15T22:25:36.115119+0000","last_fullsized":"2021-03-15T22:25:36.115119+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.674895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"387","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115517+0000","last_change":"2021-03-15T22:25:05.682093+0000","last_active":"2021-03-15T22:25:36.115517+0000","last_peered":"2021-03-15T22:25:36.115517+0000","last_clean":"2021-03-15T22:25:36.115517+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:25:36.115517+0000","last_undegraded":"2021-03-15T22:25:36.115517+0000","last_fullsized":"2021-03-15T22:25:36.115517+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.682031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.682031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"121'9","reported_seq":"456","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.103144+0000","last_change":"2021-03-15T22:24:37.902248+0000","last_active":"2021-03-15T22:25:35.103144+0000","last_peered":"2021-03-15T22:25:35.103144+0000","last_clean":"2021-03-15T22:25:35.103144+0000","last_became_active":"2021-03-15T22:22:56.055083+0000","last_became_peered":"2021-03-15T22:22:56.055083+0000","last_unstale":"2021-03-15T22:25:35.103144+0000","last_undegraded":"2021-03-15T22:25:35.103144+0000","last_fullsized":"2021-03-15T22:25:35.103144+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'9","last_scrub_stamp":"2021-03-15T22:24:37.902166+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.902166+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"383","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101606+0000","last_change":"2021-03-15T22:24:30.865676+0000","last_active":"2021-03-15T22:25:35.101606+0000","last_peered":"2021-03-15T22:25:35.101606+0000","last_clean":"2021-03-15T22:25:35.101606+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:25:35.101606+0000","last_undegraded":"2021-03-15T22:25:35.101606+0000","last_fullsized":"2021-03-15T22:25:35.101606+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.865609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.865609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"131","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.103059+0000","last_change":"2021-03-15T22:24:55.103109+0000","last_active":"2021-03-15T22:25:35.103059+0000","last_peered":"2021-03-15T22:25:35.103059+0000","last_clean":"2021-03-15T22:25:35.103059+0000","last_became_active":"2021-03-15T22:24:55.090536+0000","last_became_peered":"2021-03-15T22:24:55.090536+0000","last_unstale":"2021-03-15T22:25:35.103059+0000","last_undegraded":"2021-03-15T22:25:35.103059+0000","last_fullsized":"2021-03-15T22:25:35.103059+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:24:28.507563+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.507563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"483","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.740773+0000","last_change":"2021-03-15T22:25:33.740318+0000","last_active":"2021-03-15T22:25:33.740773+0000","last_peered":"2021-03-15T22:25:33.740773+0000","last_clean":"2021-03-15T22:25:33.740773+0000","last_became_active":"2021-03-15T22:25:33.134238+0000","last_became_peered":"2021-03-15T22:25:33.134238+0000","last_unstale":"2021-03-15T22:25:33.740773+0000","last_undegraded":"2021-03-15T22:25:33.740773+0000","last_fullsized":"2021-03-15T22:25:33.740773+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:03.482307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:03.482307+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"211","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.106496+0000","last_change":"2021-03-15T22:25:14.117883+0000","last_active":"2021-03-15T22:25:35.106496+0000","last_peered":"2021-03-15T22:25:35.106496+0000","last_clean":"2021-03-15T22:25:35.106496+0000","last_became_active":"2021-03-15T22:25:14.117558+0000","last_became_peered":"2021-03-15T22:25:14.117558+0000","last_unstale":"2021-03-15T22:25:35.106496+0000","last_undegraded":"2021-03-15T22:25:35.106496+0000","last_fullsized":"2021-03-15T22:25:35.106496+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:00.480944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:00.480944+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"392","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.102293+0000","last_change":"2021-03-15T22:25:11.028502+0000","last_active":"2021-03-15T22:25:35.102293+0000","last_peered":"2021-03-15T22:25:35.102293+0000","last_clean":"2021-03-15T22:25:35.102293+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:25:35.102293+0000","last_undegraded":"2021-03-15T22:25:35.102293+0000","last_fullsized":"2021-03-15T22:25:35.102293+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.028422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.028422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"33","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.106339+0000","last_change":"2021-03-15T22:25:33.169024+0000","last_active":"2021-03-15T22:25:35.106339+0000","last_peered":"2021-03-15T22:25:35.106339+0000","last_clean":"2021-03-15T22:25:35.106339+0000","last_became_active":"2021-03-15T22:25:33.134781+0000","last_became_peered":"2021-03-15T22:25:33.134781+0000","last_unstale":"2021-03-15T22:25:35.106339+0000","last_undegraded":"2021-03-15T22:25:35.106339+0000","last_fullsized":"2021-03-15T22:25:35.106339+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:23:32.049047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:32.049047+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":36,"num_read_kb":19,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"374","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.137000+0000","last_change":"2021-03-15T22:23:52.272270+0000","last_active":"2021-03-15T22:25:33.137000+0000","last_peered":"2021-03-15T22:25:33.137000+0000","last_clean":"2021-03-15T22:25:33.137000+0000","last_became_active":"2021-03-15T22:23:52.271925+0000","last_became_peered":"2021-03-15T22:23:52.271925+0000","last_unstale":"2021-03-15T22:25:33.137000+0000","last_undegraded":"2021-03-15T22:25:33.137000+0000","last_fullsized":"2021-03-15T22:25:33.137000+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:34.782225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:34.782225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"408","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.108848+0000","last_change":"2021-03-15T22:23:40.199174+0000","last_active":"2021-03-15T22:25:36.108848+0000","last_peered":"2021-03-15T22:25:36.108848+0000","last_clean":"2021-03-15T22:25:36.108848+0000","last_became_active":"2021-03-15T22:22:56.062833+0000","last_became_peered":"2021-03-15T22:22:56.062833+0000","last_unstale":"2021-03-15T22:25:36.108848+0000","last_undegraded":"2021-03-15T22:25:36.108848+0000","last_fullsized":"2021-03-15T22:25:36.108848+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:40.199095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:40.199095+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"201","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.753097+0000","last_change":"2021-03-15T22:25:33.752847+0000","last_active":"2021-03-15T22:25:33.753097+0000","last_peered":"2021-03-15T22:25:33.753097+0000","last_clean":"2021-03-15T22:25:33.753097+0000","last_became_active":"2021-03-15T22:25:33.140078+0000","last_became_peered":"2021-03-15T22:25:33.140078+0000","last_unstale":"2021-03-15T22:25:33.753097+0000","last_undegraded":"2021-03-15T22:25:33.753097+0000","last_fullsized":"2021-03-15T22:25:33.753097+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:25:08.192165+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:25:08.192165+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"384","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112652+0000","last_change":"2021-03-15T22:23:31.003298+0000","last_active":"2021-03-15T22:25:36.112652+0000","last_peered":"2021-03-15T22:25:36.112652+0000","last_clean":"2021-03-15T22:25:36.112652+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:25:36.112652+0000","last_undegraded":"2021-03-15T22:25:36.112652+0000","last_fullsized":"2021-03-15T22:25:36.112652+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:31.003234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:31.003234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"13","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.099507+0000","last_change":"2021-03-15T22:25:33.133625+0000","last_active":"2021-03-15T22:25:35.099507+0000","last_peered":"2021-03-15T22:25:35.099507+0000","last_clean":"2021-03-15T22:25:35.099507+0000","last_became_active":"2021-03-15T22:25:33.133418+0000","last_became_peered":"2021-03-15T22:25:33.133418+0000","last_unstale":"2021-03-15T22:25:35.099507+0000","last_undegraded":"2021-03-15T22:25:35.099507+0000","last_fullsized":"2021-03-15T22:25:35.099507+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:29.506406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.506406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"416","reported_epoch":"419","state":"active+clean","last_fresh":"2021-03-15T22:25:34.128429+0000","last_change":"2021-03-15T22:24:41.483906+0000","last_active":"2021-03-15T22:25:34.128429+0000","last_peered":"2021-03-15T22:25:34.128429+0000","last_clean":"2021-03-15T22:25:34.128429+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:25:34.128429+0000","last_undegraded":"2021-03-15T22:25:34.128429+0000","last_fullsized":"2021-03-15T22:25:34.128429+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:24:41.483849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:41.483849+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"379","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.137386+0000","last_change":"2021-03-15T22:25:09.177600+0000","last_active":"2021-03-15T22:25:33.137386+0000","last_peered":"2021-03-15T22:25:33.137386+0000","last_clean":"2021-03-15T22:25:33.137386+0000","last_became_active":"2021-03-15T22:23:52.260757+0000","last_became_peered":"2021-03-15T22:23:52.260757+0000","last_unstale":"2021-03-15T22:25:33.137386+0000","last_undegraded":"2021-03-15T22:25:33.137386+0000","last_fullsized":"2021-03-15T22:25:33.137386+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:09.177521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.177521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"390","reported_epoch":"419","state":"active+clean","last_fresh":"2021-03-15T22:25:34.127793+0000","last_change":"2021-03-15T22:25:02.452304+0000","last_active":"2021-03-15T22:25:34.127793+0000","last_peered":"2021-03-15T22:25:34.127793+0000","last_clean":"2021-03-15T22:25:34.127793+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:25:34.127793+0000","last_undegraded":"2021-03-15T22:25:34.127793+0000","last_fullsized":"2021-03-15T22:25:34.127793+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.452236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.452236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"219","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115474+0000","last_change":"2021-03-15T22:25:33.147111+0000","last_active":"2021-03-15T22:25:36.115474+0000","last_peered":"2021-03-15T22:25:36.115474+0000","last_clean":"2021-03-15T22:25:36.115474+0000","last_became_active":"2021-03-15T22:25:33.146680+0000","last_became_peered":"2021-03-15T22:25:33.146680+0000","last_unstale":"2021-03-15T22:25:36.115474+0000","last_undegraded":"2021-03-15T22:25:36.115474+0000","last_fullsized":"2021-03-15T22:25:36.115474+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:57.444598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:23:57.444598+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"110","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.099982+0000","last_change":"2021-03-15T22:25:15.835780+0000","last_active":"2021-03-15T22:25:35.099982+0000","last_peered":"2021-03-15T22:25:35.099982+0000","last_clean":"2021-03-15T22:25:35.099982+0000","last_became_active":"2021-03-15T22:24:23.142746+0000","last_became_peered":"2021-03-15T22:24:23.142746+0000","last_unstale":"2021-03-15T22:25:35.099982+0000","last_undegraded":"2021-03-15T22:25:35.099982+0000","last_fullsized":"2021-03-15T22:25:35.099982+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:25:15.835675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.835675+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":11,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"13","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.103787+0000","last_change":"2021-03-15T22:25:33.148415+0000","last_active":"2021-03-15T22:25:35.103787+0000","last_peered":"2021-03-15T22:25:35.103787+0000","last_clean":"2021-03-15T22:25:35.103787+0000","last_became_active":"2021-03-15T22:25:33.148083+0000","last_became_peered":"2021-03-15T22:25:33.148083+0000","last_unstale":"2021-03-15T22:25:35.103787+0000","last_undegraded":"2021-03-15T22:25:35.103787+0000","last_fullsized":"2021-03-15T22:25:35.103787+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:42.525975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:40.670674+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.525975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"371","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.134324+0000","last_change":"2021-03-15T22:23:52.263156+0000","last_active":"2021-03-15T22:25:33.134324+0000","last_peered":"2021-03-15T22:25:33.134324+0000","last_clean":"2021-03-15T22:25:33.134324+0000","last_became_active":"2021-03-15T22:23:52.262741+0000","last_became_peered":"2021-03-15T22:23:52.262741+0000","last_unstale":"2021-03-15T22:25:33.134324+0000","last_undegraded":"2021-03-15T22:25:33.134324+0000","last_fullsized":"2021-03-15T22:25:33.134324+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:41.710658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.710658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"382","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116037+0000","last_change":"2021-03-15T22:23:30.005865+0000","last_active":"2021-03-15T22:25:36.116037+0000","last_peered":"2021-03-15T22:25:36.116037+0000","last_clean":"2021-03-15T22:25:36.116037+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:25:36.116037+0000","last_undegraded":"2021-03-15T22:25:36.116037+0000","last_fullsized":"2021-03-15T22:25:36.116037+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:30.005791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:30.005791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"90'22","reported_seq":"198","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101787+0000","last_change":"2021-03-15T22:25:16.834493+0000","last_active":"2021-03-15T22:25:35.101787+0000","last_peered":"2021-03-15T22:25:35.101787+0000","last_clean":"2021-03-15T22:25:35.101787+0000","last_became_active":"2021-03-15T22:22:56.056175+0000","last_became_peered":"2021-03-15T22:22:56.056175+0000","last_unstale":"2021-03-15T22:25:35.101787+0000","last_undegraded":"2021-03-15T22:25:35.101787+0000","last_fullsized":"2021-03-15T22:25:35.101787+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:25:16.834409+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.834409+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"81","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116578+0000","last_change":"2021-03-15T22:24:41.659524+0000","last_active":"2021-03-15T22:25:36.116578+0000","last_peered":"2021-03-15T22:25:36.116578+0000","last_clean":"2021-03-15T22:25:36.116578+0000","last_became_active":"2021-03-15T22:24:31.105380+0000","last_became_peered":"2021-03-15T22:24:31.105380+0000","last_unstale":"2021-03-15T22:25:36.116578+0000","last_undegraded":"2021-03-15T22:25:36.116578+0000","last_fullsized":"2021-03-15T22:25:36.116578+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_clean_scrub_stamp":"2021-03-15T22:24:41.659450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"198","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101218+0000","last_change":"2021-03-15T22:25:14.115163+0000","last_active":"2021-03-15T22:25:35.101218+0000","last_peered":"2021-03-15T22:25:35.101218+0000","last_clean":"2021-03-15T22:25:35.101218+0000","last_became_active":"2021-03-15T22:25:14.114508+0000","last_became_peered":"2021-03-15T22:25:14.114508+0000","last_unstale":"2021-03-15T22:25:35.101218+0000","last_undegraded":"2021-03-15T22:25:35.101218+0000","last_fullsized":"2021-03-15T22:25:35.101218+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.007316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:04.613000+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.007316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"393","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.135727+0000","last_change":"2021-03-15T22:23:52.266674+0000","last_active":"2021-03-15T22:25:33.135727+0000","last_peered":"2021-03-15T22:25:33.135727+0000","last_clean":"2021-03-15T22:25:33.135727+0000","last_became_active":"2021-03-15T22:23:52.266011+0000","last_became_peered":"2021-03-15T22:23:52.266011+0000","last_unstale":"2021-03-15T22:25:33.135727+0000","last_undegraded":"2021-03-15T22:25:33.135727+0000","last_fullsized":"2021-03-15T22:25:33.135727+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.449472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"73","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112927+0000","last_change":"2021-03-15T22:24:56.824808+0000","last_active":"2021-03-15T22:25:36.112927+0000","last_peered":"2021-03-15T22:25:36.112927+0000","last_clean":"2021-03-15T22:25:36.112927+0000","last_became_active":"2021-03-15T22:24:55.097354+0000","last_became_peered":"2021-03-15T22:24:55.097354+0000","last_unstale":"2021-03-15T22:25:36.112927+0000","last_undegraded":"2021-03-15T22:25:36.112927+0000","last_fullsized":"2021-03-15T22:25:36.112927+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:24:38.433166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:38.433166+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"13","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101439+0000","last_change":"2021-03-15T22:25:33.139212+0000","last_active":"2021-03-15T22:25:35.101439+0000","last_peered":"2021-03-15T22:25:35.101439+0000","last_clean":"2021-03-15T22:25:35.101439+0000","last_became_active":"2021-03-15T22:25:33.138883+0000","last_became_peered":"2021-03-15T22:25:33.138883+0000","last_unstale":"2021-03-15T22:25:35.101439+0000","last_undegraded":"2021-03-15T22:25:35.101439+0000","last_fullsized":"2021-03-15T22:25:35.101439+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:33.717517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:33.717517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"404","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112753+0000","last_change":"2021-03-15T22:25:33.129883+0000","last_active":"2021-03-15T22:25:36.112753+0000","last_peered":"2021-03-15T22:25:36.112753+0000","last_clean":"2021-03-15T22:25:36.112753+0000","last_became_active":"2021-03-15T22:25:33.129553+0000","last_became_peered":"2021-03-15T22:25:33.129553+0000","last_unstale":"2021-03-15T22:25:36.112753+0000","last_undegraded":"2021-03-15T22:25:36.112753+0000","last_fullsized":"2021-03-15T22:25:36.112753+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.794432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"106","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.120232+0000","last_change":"2021-03-15T22:24:12.017160+0000","last_active":"2021-03-15T22:25:33.120232+0000","last_peered":"2021-03-15T22:25:33.120232+0000","last_clean":"2021-03-15T22:25:33.120232+0000","last_became_active":"2021-03-15T22:24:03.086278+0000","last_became_peered":"2021-03-15T22:24:03.086278+0000","last_unstale":"2021-03-15T22:25:33.120232+0000","last_undegraded":"2021-03-15T22:25:33.120232+0000","last_fullsized":"2021-03-15T22:25:33.120232+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:12.017097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:12.017097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"412","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.102972+0000","last_change":"2021-03-15T22:23:29.709766+0000","last_active":"2021-03-15T22:25:35.102972+0000","last_peered":"2021-03-15T22:25:35.102972+0000","last_clean":"2021-03-15T22:25:35.102972+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:25:35.102972+0000","last_undegraded":"2021-03-15T22:25:35.102972+0000","last_fullsized":"2021-03-15T22:25:35.102972+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:29.709674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.709674+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"212","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.146807+0000","last_change":"2021-03-15T22:25:33.144478+0000","last_active":"2021-03-15T22:25:33.146807+0000","last_peered":"2021-03-15T22:25:33.146807+0000","last_clean":"2021-03-15T22:25:33.146807+0000","last_became_active":"2021-03-15T22:25:33.143832+0000","last_became_peered":"2021-03-15T22:25:33.143832+0000","last_unstale":"2021-03-15T22:25:33.146807+0000","last_undegraded":"2021-03-15T22:25:33.146807+0000","last_fullsized":"2021-03-15T22:25:33.146807+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.147414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.147414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"356'3","reported_seq":"485","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:37.085753+0000","last_change":"2021-03-15T22:24:31.129210+0000","last_active":"2021-03-15T22:25:37.085753+0000","last_peered":"2021-03-15T22:25:37.085753+0000","last_clean":"2021-03-15T22:25:37.085753+0000","last_became_active":"2021-03-15T22:24:31.093479+0000","last_became_peered":"2021-03-15T22:24:31.093479+0000","last_unstale":"2021-03-15T22:25:37.085753+0000","last_undegraded":"2021-03-15T22:25:37.085753+0000","last_fullsized":"2021-03-15T22:25:37.085753+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:24:28.700659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.700659+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"380","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.135364+0000","last_change":"2021-03-15T22:25:01.148183+0000","last_active":"2021-03-15T22:25:33.135364+0000","last_peered":"2021-03-15T22:25:33.135364+0000","last_clean":"2021-03-15T22:25:33.135364+0000","last_became_active":"2021-03-15T22:23:52.261234+0000","last_became_peered":"2021-03-15T22:23:52.261234+0000","last_unstale":"2021-03-15T22:25:33.135364+0000","last_undegraded":"2021-03-15T22:25:33.135364+0000","last_fullsized":"2021-03-15T22:25:33.135364+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:01.148105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:01.148105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"197","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.105253+0000","last_change":"2021-03-15T22:24:27.166757+0000","last_active":"2021-03-15T22:25:35.105253+0000","last_peered":"2021-03-15T22:25:35.105253+0000","last_clean":"2021-03-15T22:25:35.105253+0000","last_became_active":"2021-03-15T22:22:56.064081+0000","last_became_peered":"2021-03-15T22:22:56.064081+0000","last_unstale":"2021-03-15T22:25:35.105253+0000","last_undegraded":"2021-03-15T22:25:35.105253+0000","last_fullsized":"2021-03-15T22:25:35.105253+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:27.166678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.166678+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":13,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"381","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101889+0000","last_change":"2021-03-15T22:25:07.992897+0000","last_active":"2021-03-15T22:25:35.101889+0000","last_peered":"2021-03-15T22:25:35.101889+0000","last_clean":"2021-03-15T22:25:35.101889+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:25:35.101889+0000","last_undegraded":"2021-03-15T22:25:35.101889+0000","last_fullsized":"2021-03-15T22:25:35.101889+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:07.992815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.992815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"361'6","reported_seq":"17","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.105060+0000","last_change":"2021-03-15T22:25:33.172230+0000","last_active":"2021-03-15T22:25:35.105060+0000","last_peered":"2021-03-15T22:25:35.105060+0000","last_clean":"2021-03-15T22:25:35.105060+0000","last_became_active":"2021-03-15T22:25:33.147484+0000","last_became_peered":"2021-03-15T22:25:33.147484+0000","last_unstale":"2021-03-15T22:25:35.105060+0000","last_undegraded":"2021-03-15T22:25:35.105060+0000","last_fullsized":"2021-03-15T22:25:35.105060+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'6","last_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.098511+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"218","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.145615+0000","last_change":"2021-03-15T22:25:33.141774+0000","last_active":"2021-03-15T22:25:33.145615+0000","last_peered":"2021-03-15T22:25:33.145615+0000","last_clean":"2021-03-15T22:25:33.145615+0000","last_became_active":"2021-03-15T22:25:33.141466+0000","last_became_peered":"2021-03-15T22:25:33.141466+0000","last_unstale":"2021-03-15T22:25:33.145615+0000","last_undegraded":"2021-03-15T22:25:33.145615+0000","last_fullsized":"2021-03-15T22:25:33.145615+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:23:39.695671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"51","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116249+0000","last_change":"2021-03-15T22:25:14.129885+0000","last_active":"2021-03-15T22:25:36.116249+0000","last_peered":"2021-03-15T22:25:36.116249+0000","last_clean":"2021-03-15T22:25:36.116249+0000","last_became_active":"2021-03-15T22:25:14.114060+0000","last_became_peered":"2021-03-15T22:25:14.114060+0000","last_unstale":"2021-03-15T22:25:36.116249+0000","last_undegraded":"2021-03-15T22:25:36.116249+0000","last_fullsized":"2021-03-15T22:25:36.116249+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:25:07.163128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.163128+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"384","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.121671+0000","last_change":"2021-03-15T22:25:00.614351+0000","last_active":"2021-03-15T22:25:33.121671+0000","last_peered":"2021-03-15T22:25:33.121671+0000","last_clean":"2021-03-15T22:25:33.121671+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:25:33.121671+0000","last_undegraded":"2021-03-15T22:25:33.121671+0000","last_fullsized":"2021-03-15T22:25:33.121671+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.614288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.614288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"382","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.135766+0000","last_change":"2021-03-15T22:25:10.130486+0000","last_active":"2021-03-15T22:25:33.135766+0000","last_peered":"2021-03-15T22:25:33.135766+0000","last_clean":"2021-03-15T22:25:33.135766+0000","last_became_active":"2021-03-15T22:23:52.263426+0000","last_became_peered":"2021-03-15T22:23:52.263426+0000","last_unstale":"2021-03-15T22:25:33.135766+0000","last_undegraded":"2021-03-15T22:25:33.135766+0000","last_fullsized":"2021-03-15T22:25:33.135766+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:10.130422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.130422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"37","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.105527+0000","last_change":"2021-03-15T22:25:35.064301+0000","last_active":"2021-03-15T22:25:35.105527+0000","last_peered":"2021-03-15T22:25:35.105527+0000","last_clean":"2021-03-15T22:25:35.105527+0000","last_became_active":"2021-03-15T22:25:33.151159+0000","last_became_peered":"2021-03-15T22:25:33.151159+0000","last_unstale":"2021-03-15T22:25:35.105527+0000","last_undegraded":"2021-03-15T22:25:35.105527+0000","last_fullsized":"2021-03-15T22:25:35.105527+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:26.478799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.478799+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3618,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"374","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.135630+0000","last_change":"2021-03-15T22:23:52.266876+0000","last_active":"2021-03-15T22:25:33.135630+0000","last_peered":"2021-03-15T22:25:33.135630+0000","last_clean":"2021-03-15T22:25:33.135630+0000","last_became_active":"2021-03-15T22:23:52.266133+0000","last_became_peered":"2021-03-15T22:23:52.266133+0000","last_unstale":"2021-03-15T22:25:33.135630+0000","last_undegraded":"2021-03-15T22:25:33.135630+0000","last_fullsized":"2021-03-15T22:25:33.135630+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:28.808873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:28.808873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"430","reported_epoch":"419","state":"active+clean","last_fresh":"2021-03-15T22:25:34.127146+0000","last_change":"2021-03-15T22:25:33.150477+0000","last_active":"2021-03-15T22:25:34.127146+0000","last_peered":"2021-03-15T22:25:34.127146+0000","last_clean":"2021-03-15T22:25:34.127146+0000","last_became_active":"2021-03-15T22:25:33.150147+0000","last_became_peered":"2021-03-15T22:25:33.150147+0000","last_unstale":"2021-03-15T22:25:34.127146+0000","last_undegraded":"2021-03-15T22:25:34.127146+0000","last_fullsized":"2021-03-15T22:25:34.127146+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:37.475218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.475218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"398'11","reported_seq":"23","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.106060+0000","last_change":"2021-03-15T22:25:33.864526+0000","last_active":"2021-03-15T22:25:35.106060+0000","last_peered":"2021-03-15T22:25:35.106060+0000","last_clean":"2021-03-15T22:25:35.106060+0000","last_became_active":"2021-03-15T22:25:33.152373+0000","last_became_peered":"2021-03-15T22:25:33.152373+0000","last_unstale":"2021-03-15T22:25:35.106060+0000","last_undegraded":"2021-03-15T22:25:35.106060+0000","last_fullsized":"2021-03-15T22:25:35.106060+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'11","last_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:25:13.181225+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"418'312","reported_seq":"1497","reported_epoch":"422","state":"active+clean","last_fresh":"2021-03-15T22:25:37.112878+0000","last_change":"2021-03-15T22:25:37.113594+0000","last_active":"2021-03-15T22:25:37.112878+0000","last_peered":"2021-03-15T22:25:37.112878+0000","last_clean":"2021-03-15T22:25:37.112878+0000","last_became_active":"2021-03-15T22:25:17.113174+0000","last_became_peered":"2021-03-15T22:25:17.113174+0000","last_unstale":"2021-03-15T22:25:37.112878+0000","last_undegraded":"2021-03-15T22:25:37.112878+0000","last_fullsized":"2021-03-15T22:25:37.112878+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'282","last_scrub_stamp":"2021-03-15T22:25:09.803362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.803362+0000","log_size":312,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20454716,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":223,"num_read_kb":245574,"num_write":136,"num_write_kb":43781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":21669077,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.a","version":"0'0","reported_seq":"390","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116249+0000","last_change":"2021-03-15T22:25:00.689659+0000","last_active":"2021-03-15T22:25:36.116249+0000","last_peered":"2021-03-15T22:25:36.116249+0000","last_clean":"2021-03-15T22:25:36.116249+0000","last_became_active":"2021-03-15T22:24:31.103470+0000","last_became_peered":"2021-03-15T22:24:31.103470+0000","last_unstale":"2021-03-15T22:25:36.116249+0000","last_undegraded":"2021-03-15T22:25:36.116249+0000","last_fullsized":"2021-03-15T22:25:36.116249+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.689591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.689591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"398","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.137438+0000","last_change":"2021-03-15T22:24:31.115206+0000","last_active":"2021-03-15T22:25:33.137438+0000","last_peered":"2021-03-15T22:25:33.137438+0000","last_clean":"2021-03-15T22:25:33.137438+0000","last_became_active":"2021-03-15T22:24:31.097688+0000","last_became_peered":"2021-03-15T22:24:31.097688+0000","last_unstale":"2021-03-15T22:25:33.137438+0000","last_undegraded":"2021-03-15T22:25:33.137438+0000","last_fullsized":"2021-03-15T22:25:33.137438+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:23:03.888766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:03.888766+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"109","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115088+0000","last_change":"2021-03-15T22:25:32.799965+0000","last_active":"2021-03-15T22:25:36.115088+0000","last_peered":"2021-03-15T22:25:36.115088+0000","last_clean":"2021-03-15T22:25:36.115088+0000","last_became_active":"2021-03-15T22:24:23.137531+0000","last_became_peered":"2021-03-15T22:24:23.137531+0000","last_unstale":"2021-03-15T22:25:36.115088+0000","last_undegraded":"2021-03-15T22:25:36.115088+0000","last_fullsized":"2021-03-15T22:25:36.115088+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:25:32.799882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.799882+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"317'7","reported_seq":"487","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.135993+0000","last_change":"2021-03-15T22:24:30.068613+0000","last_active":"2021-03-15T22:25:33.135993+0000","last_peered":"2021-03-15T22:25:33.135993+0000","last_clean":"2021-03-15T22:25:33.135993+0000","last_became_active":"2021-03-15T22:23:52.262878+0000","last_became_peered":"2021-03-15T22:23:52.262878+0000","last_unstale":"2021-03-15T22:25:33.135993+0000","last_undegraded":"2021-03-15T22:25:33.135993+0000","last_fullsized":"2021-03-15T22:25:33.135993+0000","mapping_epoch":316,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:24:30.068550+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.068550+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"420'542","reported_seq":"1636","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.288903+0000","last_change":"2021-03-15T22:25:34.209844+0000","last_active":"2021-03-15T22:25:35.288903+0000","last_peered":"2021-03-15T22:25:35.288903+0000","last_clean":"2021-03-15T22:25:35.288903+0000","last_became_active":"2021-03-15T22:23:04.067641+0000","last_became_peered":"2021-03-15T22:23:04.067641+0000","last_unstale":"2021-03-15T22:25:35.288903+0000","last_undegraded":"2021-03-15T22:25:35.288903+0000","last_fullsized":"2021-03-15T22:25:35.288903+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'455","last_scrub_stamp":"2021-03-15T22:24:28.184612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.184612+0000","log_size":542,"ondisk_log_size":542,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20852784,"num_objects":14,"num_object_clones":9,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":4,"num_read":308,"num_read_kb":316000,"num_write":201,"num_write_kb":72052,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":32120558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"12b","length":"1"},{"start":"139","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"180","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.104077+0000","last_change":"2021-03-15T22:24:37.122501+0000","last_active":"2021-03-15T22:25:35.104077+0000","last_peered":"2021-03-15T22:25:35.104077+0000","last_clean":"2021-03-15T22:25:35.104077+0000","last_became_active":"2021-03-15T22:22:56.057174+0000","last_became_peered":"2021-03-15T22:22:56.057174+0000","last_unstale":"2021-03-15T22:25:35.104077+0000","last_undegraded":"2021-03-15T22:25:35.104077+0000","last_fullsized":"2021-03-15T22:25:35.104077+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:37.122407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.122407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"380","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.137720+0000","last_change":"2021-03-15T22:23:52.265794+0000","last_active":"2021-03-15T22:25:33.137720+0000","last_peered":"2021-03-15T22:25:33.137720+0000","last_clean":"2021-03-15T22:25:33.137720+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:25:33.137720+0000","last_undegraded":"2021-03-15T22:25:33.137720+0000","last_fullsized":"2021-03-15T22:25:33.137720+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:38.810397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.810397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"468","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101433+0000","last_change":"2021-03-15T22:25:33.862633+0000","last_active":"2021-03-15T22:25:35.101433+0000","last_peered":"2021-03-15T22:25:35.101433+0000","last_clean":"2021-03-15T22:25:35.101433+0000","last_became_active":"2021-03-15T22:25:33.138262+0000","last_became_peered":"2021-03-15T22:25:33.138262+0000","last_unstale":"2021-03-15T22:25:35.101433+0000","last_undegraded":"2021-03-15T22:25:35.101433+0000","last_fullsized":"2021-03-15T22:25:35.101433+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:07.027095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.027095+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"413","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.120048+0000","last_change":"2021-03-15T22:24:39.853332+0000","last_active":"2021-03-15T22:25:33.120048+0000","last_peered":"2021-03-15T22:25:33.120048+0000","last_clean":"2021-03-15T22:25:33.120048+0000","last_became_active":"2021-03-15T22:24:31.109058+0000","last_became_peered":"2021-03-15T22:24:31.109058+0000","last_unstale":"2021-03-15T22:25:33.120048+0000","last_undegraded":"2021-03-15T22:25:33.120048+0000","last_fullsized":"2021-03-15T22:25:33.120048+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.853267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.853267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"415'140","reported_seq":"1177","reported_epoch":"422","state":"active+clean","last_fresh":"2021-03-15T22:25:37.112823+0000","last_change":"2021-03-15T22:25:37.113314+0000","last_active":"2021-03-15T22:25:37.112823+0000","last_peered":"2021-03-15T22:25:37.112823+0000","last_clean":"2021-03-15T22:25:37.112823+0000","last_became_active":"2021-03-15T22:24:34.087074+0000","last_became_peered":"2021-03-15T22:24:34.087074+0000","last_unstale":"2021-03-15T22:25:37.112823+0000","last_undegraded":"2021-03-15T22:25:37.112823+0000","last_fullsized":"2021-03-15T22:25:37.112823+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'130","last_scrub_stamp":"2021-03-15T22:25:06.854931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.854931+0000","log_size":140,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2235397,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":42,"num_read_kb":30032,"num_write":45,"num_write_kb":15718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4470794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.8","version":"0'0","reported_seq":"392","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.133769+0000","last_change":"2021-03-15T22:23:52.267464+0000","last_active":"2021-03-15T22:25:33.133769+0000","last_peered":"2021-03-15T22:25:33.133769+0000","last_clean":"2021-03-15T22:25:33.133769+0000","last_became_active":"2021-03-15T22:23:52.267164+0000","last_became_peered":"2021-03-15T22:23:52.267164+0000","last_unstale":"2021-03-15T22:25:33.133769+0000","last_undegraded":"2021-03-15T22:25:33.133769+0000","last_fullsized":"2021-03-15T22:25:33.133769+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:29.845442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.845442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"431","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.120368+0000","last_change":"2021-03-15T22:25:14.114184+0000","last_active":"2021-03-15T22:25:33.120368+0000","last_peered":"2021-03-15T22:25:33.120368+0000","last_clean":"2021-03-15T22:25:33.120368+0000","last_became_active":"2021-03-15T22:25:14.113857+0000","last_became_peered":"2021-03-15T22:25:14.113857+0000","last_unstale":"2021-03-15T22:25:33.120368+0000","last_undegraded":"2021-03-15T22:25:33.120368+0000","last_fullsized":"2021-03-15T22:25:33.120368+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.575552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.575552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"400","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.135880+0000","last_change":"2021-03-15T22:25:14.118841+0000","last_active":"2021-03-15T22:25:33.135880+0000","last_peered":"2021-03-15T22:25:33.135880+0000","last_clean":"2021-03-15T22:25:33.135880+0000","last_became_active":"2021-03-15T22:25:14.118469+0000","last_became_peered":"2021-03-15T22:25:14.118469+0000","last_unstale":"2021-03-15T22:25:33.135880+0000","last_undegraded":"2021-03-15T22:25:33.135880+0000","last_fullsized":"2021-03-15T22:25:33.135880+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.024799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.024799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"461","reported_epoch":"419","state":"active+clean","last_fresh":"2021-03-15T22:25:34.127579+0000","last_change":"2021-03-15T22:24:40.468134+0000","last_active":"2021-03-15T22:25:34.127579+0000","last_peered":"2021-03-15T22:25:34.127579+0000","last_clean":"2021-03-15T22:25:34.127579+0000","last_became_active":"2021-03-15T22:24:03.085862+0000","last_became_peered":"2021-03-15T22:24:03.085862+0000","last_unstale":"2021-03-15T22:25:34.127579+0000","last_undegraded":"2021-03-15T22:25:34.127579+0000","last_fullsized":"2021-03-15T22:25:34.127579+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:24:40.468066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.468066+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"418'328","reported_seq":"1354","reported_epoch":"419","state":"active+clean","last_fresh":"2021-03-15T22:25:34.126082+0000","last_change":"2021-03-15T22:25:34.126286+0000","last_active":"2021-03-15T22:25:34.126082+0000","last_peered":"2021-03-15T22:25:34.126082+0000","last_clean":"2021-03-15T22:25:34.126082+0000","last_became_active":"2021-03-15T22:20:05.032282+0000","last_became_peered":"2021-03-15T22:20:05.032282+0000","last_unstale":"2021-03-15T22:25:34.126082+0000","last_undegraded":"2021-03-15T22:25:34.126082+0000","last_fullsized":"2021-03-15T22:25:34.126082+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"270'176","last_scrub_stamp":"2021-03-15T22:23:05.541386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:05.541386+0000","log_size":328,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21341533,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":196,"num_read_kb":234047,"num_write":111,"num_write_kb":37367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"115","length":"2"},{"start":"12b","length":"1"},{"start":"139","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"80","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.135476+0000","last_change":"2021-03-15T22:25:11.117468+0000","last_active":"2021-03-15T22:25:33.135476+0000","last_peered":"2021-03-15T22:25:33.135476+0000","last_clean":"2021-03-15T22:25:33.135476+0000","last_became_active":"2021-03-15T22:24:31.093577+0000","last_became_peered":"2021-03-15T22:24:31.093577+0000","last_unstale":"2021-03-15T22:25:33.135476+0000","last_undegraded":"2021-03-15T22:25:33.135476+0000","last_fullsized":"2021-03-15T22:25:33.135476+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.117403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.117403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"431","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.135512+0000","last_change":"2021-03-15T22:25:33.134762+0000","last_active":"2021-03-15T22:25:33.135512+0000","last_peered":"2021-03-15T22:25:33.135512+0000","last_clean":"2021-03-15T22:25:33.135512+0000","last_became_active":"2021-03-15T22:25:33.134358+0000","last_became_peered":"2021-03-15T22:25:33.134358+0000","last_unstale":"2021-03-15T22:25:33.135512+0000","last_undegraded":"2021-03-15T22:25:33.135512+0000","last_fullsized":"2021-03-15T22:25:33.135512+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:27.869570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.869570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"400","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115646+0000","last_change":"2021-03-15T22:25:33.145899+0000","last_active":"2021-03-15T22:25:36.115646+0000","last_peered":"2021-03-15T22:25:36.115646+0000","last_clean":"2021-03-15T22:25:36.115646+0000","last_became_active":"2021-03-15T22:25:33.145548+0000","last_became_peered":"2021-03-15T22:25:33.145548+0000","last_unstale":"2021-03-15T22:25:36.115646+0000","last_undegraded":"2021-03-15T22:25:36.115646+0000","last_fullsized":"2021-03-15T22:25:36.115646+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:08.105283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:08.105283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"181","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.912948+0000","last_change":"2021-03-15T22:25:33.912570+0000","last_active":"2021-03-15T22:25:33.912948+0000","last_peered":"2021-03-15T22:25:33.912948+0000","last_clean":"2021-03-15T22:25:33.912948+0000","last_became_active":"2021-03-15T22:25:33.135102+0000","last_became_peered":"2021-03-15T22:25:33.135102+0000","last_unstale":"2021-03-15T22:25:33.912948+0000","last_undegraded":"2021-03-15T22:25:33.912948+0000","last_fullsized":"2021-03-15T22:25:33.912948+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:01.499099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.499099+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"413'350","reported_seq":"1517","reported_epoch":"420","state":"peering","last_fresh":"2021-03-15T22:25:35.104524+0000","last_change":"2021-03-15T22:25:35.104692+0000","last_active":"2021-03-15T22:25:35.104524+0000","last_peered":"2021-03-15T22:25:34.443404+0000","last_clean":"2021-03-15T22:25:31.532647+0000","last_became_active":"2021-03-15T22:25:34.223662+0000","last_became_peered":"2021-03-15T22:25:34.223662+0000","last_unstale":"2021-03-15T22:25:35.104524+0000","last_undegraded":"2021-03-15T22:25:35.104524+0000","last_fullsized":"2021-03-15T22:25:35.104524+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'340","last_scrub_stamp":"2021-03-15T22:25:12.353191+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.353191+0000","log_size":350,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23199176,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":131,"num_read_kb":85559,"num_write":158,"num_write_kb":51328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":35579701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,7],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"115","length":"2"}]},{"pgid":"6.6","version":"0'0","reported_seq":"13","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.105567+0000","last_change":"2021-03-15T22:25:33.152550+0000","last_active":"2021-03-15T22:25:35.105567+0000","last_peered":"2021-03-15T22:25:35.105567+0000","last_clean":"2021-03-15T22:25:35.105567+0000","last_became_active":"2021-03-15T22:25:33.152232+0000","last_became_peered":"2021-03-15T22:25:33.152232+0000","last_unstale":"2021-03-15T22:25:35.105567+0000","last_undegraded":"2021-03-15T22:25:35.105567+0000","last_fullsized":"2021-03-15T22:25:35.105567+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:34.763323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:34.763323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"389","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116102+0000","last_change":"2021-03-15T22:25:11.615097+0000","last_active":"2021-03-15T22:25:36.116102+0000","last_peered":"2021-03-15T22:25:36.116102+0000","last_clean":"2021-03-15T22:25:36.116102+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:25:36.116102+0000","last_undegraded":"2021-03-15T22:25:36.116102+0000","last_fullsized":"2021-03-15T22:25:36.116102+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.614975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.614975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"110","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.122294+0000","last_change":"2021-03-15T22:24:31.106764+0000","last_active":"2021-03-15T22:25:33.122294+0000","last_peered":"2021-03-15T22:25:33.122294+0000","last_clean":"2021-03-15T22:25:33.122294+0000","last_became_active":"2021-03-15T22:24:31.106344+0000","last_became_peered":"2021-03-15T22:24:31.106344+0000","last_unstale":"2021-03-15T22:25:33.122294+0000","last_undegraded":"2021-03-15T22:25:33.122294+0000","last_fullsized":"2021-03-15T22:25:33.122294+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:00.938809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:00.938809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"527","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112164+0000","last_change":"2021-03-15T22:25:33.174619+0000","last_active":"2021-03-15T22:25:36.112164+0000","last_peered":"2021-03-15T22:25:36.112164+0000","last_clean":"2021-03-15T22:25:36.112164+0000","last_became_active":"2021-03-15T22:25:33.126039+0000","last_became_peered":"2021-03-15T22:25:33.126039+0000","last_unstale":"2021-03-15T22:25:36.112164+0000","last_undegraded":"2021-03-15T22:25:36.112164+0000","last_fullsized":"2021-03-15T22:25:36.112164+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:25:15.810377+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.810377+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"77","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112488+0000","last_change":"2021-03-15T22:24:31.102016+0000","last_active":"2021-03-15T22:25:36.112488+0000","last_peered":"2021-03-15T22:25:36.112488+0000","last_clean":"2021-03-15T22:25:36.112488+0000","last_became_active":"2021-03-15T22:24:31.101668+0000","last_became_peered":"2021-03-15T22:24:31.101668+0000","last_unstale":"2021-03-15T22:25:36.112488+0000","last_undegraded":"2021-03-15T22:25:36.112488+0000","last_fullsized":"2021-03-15T22:25:36.112488+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:08.898279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:23:08.898279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"420'344","reported_seq":"1404","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.161466+0000","last_change":"2021-03-15T22:25:34.124711+0000","last_active":"2021-03-15T22:25:35.161466+0000","last_peered":"2021-03-15T22:25:35.161466+0000","last_clean":"2021-03-15T22:25:35.161466+0000","last_became_active":"2021-03-15T22:24:34.087484+0000","last_became_peered":"2021-03-15T22:24:34.087484+0000","last_unstale":"2021-03-15T22:25:35.161466+0000","last_undegraded":"2021-03-15T22:25:35.161466+0000","last_fullsized":"2021-03-15T22:25:35.161466+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'329","last_scrub_stamp":"2021-03-15T22:25:14.014776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:14.014776+0000","log_size":344,"ondisk_log_size":344,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10429575,"num_objects":8,"num_object_clones":6,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":123,"num_read_kb":112611,"num_write":121,"num_write_kb":36588,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":10768640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"1"},{"start":"139","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"392","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.136829+0000","last_change":"2021-03-15T22:23:52.267559+0000","last_active":"2021-03-15T22:25:33.136829+0000","last_peered":"2021-03-15T22:25:33.136829+0000","last_clean":"2021-03-15T22:25:33.136829+0000","last_became_active":"2021-03-15T22:23:52.266556+0000","last_became_peered":"2021-03-15T22:23:52.266556+0000","last_unstale":"2021-03-15T22:25:33.136829+0000","last_undegraded":"2021-03-15T22:25:33.136829+0000","last_fullsized":"2021-03-15T22:25:33.136829+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:30.831598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:30.831598+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"390","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.114999+0000","last_change":"2021-03-15T22:24:39.929128+0000","last_active":"2021-03-15T22:25:36.114999+0000","last_peered":"2021-03-15T22:25:36.114999+0000","last_clean":"2021-03-15T22:25:36.114999+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:25:36.114999+0000","last_undegraded":"2021-03-15T22:25:36.114999+0000","last_fullsized":"2021-03-15T22:25:36.114999+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.929066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.929066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"417","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.130922+0000","last_change":"2021-03-15T22:25:33.130332+0000","last_active":"2021-03-15T22:25:33.130922+0000","last_peered":"2021-03-15T22:25:33.130922+0000","last_clean":"2021-03-15T22:25:33.130922+0000","last_became_active":"2021-03-15T22:25:33.129875+0000","last_became_peered":"2021-03-15T22:25:33.129875+0000","last_unstale":"2021-03-15T22:25:33.130922+0000","last_undegraded":"2021-03-15T22:25:33.130922+0000","last_fullsized":"2021-03-15T22:25:33.130922+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:29.266174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.266174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"149'22","reported_seq":"72","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.111773+0000","last_change":"2021-03-15T22:25:03.853145+0000","last_active":"2021-03-15T22:25:36.111773+0000","last_peered":"2021-03-15T22:25:36.111773+0000","last_clean":"2021-03-15T22:25:36.111773+0000","last_became_active":"2021-03-15T22:24:55.093124+0000","last_became_peered":"2021-03-15T22:24:55.093124+0000","last_unstale":"2021-03-15T22:25:36.111773+0000","last_undegraded":"2021-03-15T22:25:36.111773+0000","last_fullsized":"2021-03-15T22:25:36.111773+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'22","last_scrub_stamp":"2021-03-15T22:25:03.853079+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:25:03.853079+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"404","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.123408+0000","last_change":"2021-03-15T22:25:02.520674+0000","last_active":"2021-03-15T22:25:33.123408+0000","last_peered":"2021-03-15T22:25:33.123408+0000","last_clean":"2021-03-15T22:25:33.123408+0000","last_became_active":"2021-03-15T22:21:38.147364+0000","last_became_peered":"2021-03-15T22:21:38.147364+0000","last_unstale":"2021-03-15T22:25:33.123408+0000","last_undegraded":"2021-03-15T22:25:33.123408+0000","last_fullsized":"2021-03-15T22:25:33.123408+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.520588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.520588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"413'919","reported_seq":"2463","reported_epoch":"419","state":"activating+undersized+degraded+remapped","last_fresh":"2021-03-15T22:25:34.133292+0000","last_change":"2021-03-15T22:25:34.134149+0000","last_active":"2021-03-15T22:25:32.115440+0000","last_peered":"2021-03-15T22:25:31.264183+0000","last_clean":"2021-03-15T22:25:31.264183+0000","last_became_active":"2021-03-15T22:24:34.087382+0000","last_became_peered":"2021-03-15T22:24:34.087382+0000","last_unstale":"2021-03-15T22:25:34.133292+0000","last_undegraded":"2021-03-15T22:25:34.133292+0000","last_fullsized":"2021-03-15T22:25:34.127460+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'350","last_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:22:40.995789+0000","log_size":919,"ondisk_log_size":919,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":62106226,"num_objects":25,"num_object_clones":18,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":95,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":352,"num_read_kb":305564,"num_write":348,"num_write_kb":154675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":255,"num_bytes_recovered":87004437,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,5],"avail_no_missing":["7","2","5"],"object_location_counts":[{"shards":"5,7","objects":25}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"2"}]},{"pgid":"6.7","version":"0'0","reported_seq":"394","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115748+0000","last_change":"2021-03-15T22:23:38.968009+0000","last_active":"2021-03-15T22:25:36.115748+0000","last_peered":"2021-03-15T22:25:36.115748+0000","last_clean":"2021-03-15T22:25:36.115748+0000","last_became_active":"2021-03-15T22:22:56.062745+0000","last_became_peered":"2021-03-15T22:22:56.062745+0000","last_unstale":"2021-03-15T22:25:36.115748+0000","last_undegraded":"2021-03-15T22:25:36.115748+0000","last_fullsized":"2021-03-15T22:25:36.115748+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:38.967934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:38.967934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"394","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.133958+0000","last_change":"2021-03-15T22:24:31.111721+0000","last_active":"2021-03-15T22:25:33.133958+0000","last_peered":"2021-03-15T22:25:33.133958+0000","last_clean":"2021-03-15T22:25:33.133958+0000","last_became_active":"2021-03-15T22:24:31.111104+0000","last_became_peered":"2021-03-15T22:24:31.111104+0000","last_unstale":"2021-03-15T22:25:33.133958+0000","last_undegraded":"2021-03-15T22:25:33.133958+0000","last_fullsized":"2021-03-15T22:25:33.133958+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.872030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.872030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"13","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101025+0000","last_change":"2021-03-15T22:25:33.147575+0000","last_active":"2021-03-15T22:25:35.101025+0000","last_peered":"2021-03-15T22:25:35.101025+0000","last_clean":"2021-03-15T22:25:35.101025+0000","last_became_active":"2021-03-15T22:25:33.147285+0000","last_became_peered":"2021-03-15T22:25:33.147285+0000","last_unstale":"2021-03-15T22:25:35.101025+0000","last_undegraded":"2021-03-15T22:25:35.101025+0000","last_fullsized":"2021-03-15T22:25:35.101025+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:27.994280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:27.994280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"409","reported_epoch":"419","state":"active+clean","last_fresh":"2021-03-15T22:25:34.128274+0000","last_change":"2021-03-15T22:23:57.460290+0000","last_active":"2021-03-15T22:25:34.128274+0000","last_peered":"2021-03-15T22:25:34.128274+0000","last_clean":"2021-03-15T22:25:34.128274+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:25:34.128274+0000","last_undegraded":"2021-03-15T22:25:34.128274+0000","last_fullsized":"2021-03-15T22:25:34.128274+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:57.460218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:57.460218+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"378","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.137096+0000","last_change":"2021-03-15T22:23:52.271864+0000","last_active":"2021-03-15T22:25:33.137096+0000","last_peered":"2021-03-15T22:25:33.137096+0000","last_clean":"2021-03-15T22:25:33.137096+0000","last_became_active":"2021-03-15T22:23:52.271442+0000","last_became_peered":"2021-03-15T22:23:52.271442+0000","last_unstale":"2021-03-15T22:25:33.137096+0000","last_undegraded":"2021-03-15T22:25:33.137096+0000","last_fullsized":"2021-03-15T22:25:33.137096+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:26.748485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.748485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"419'238","reported_seq":"1311","reported_epoch":"419","state":"active+clean","last_fresh":"2021-03-15T22:25:34.137137+0000","last_change":"2021-03-15T22:25:34.137350+0000","last_active":"2021-03-15T22:25:34.137137+0000","last_peered":"2021-03-15T22:25:34.137137+0000","last_clean":"2021-03-15T22:25:34.137137+0000","last_became_active":"2021-03-15T22:24:36.092680+0000","last_became_peered":"2021-03-15T22:24:36.092680+0000","last_unstale":"2021-03-15T22:25:34.137137+0000","last_undegraded":"2021-03-15T22:25:34.137137+0000","last_fullsized":"2021-03-15T22:25:34.137137+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'138","last_scrub_stamp":"2021-03-15T22:23:41.141218+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.141218+0000","log_size":238,"ondisk_log_size":238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20445891,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":101,"num_read_kb":93801,"num_write":95,"num_write_kb":37009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8487583,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"115","length":"2"},{"start":"12b","length":"1"},{"start":"139","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"402","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.111926+0000","last_change":"2021-03-15T22:25:33.129904+0000","last_active":"2021-03-15T22:25:36.111926+0000","last_peered":"2021-03-15T22:25:36.111926+0000","last_clean":"2021-03-15T22:25:36.111926+0000","last_became_active":"2021-03-15T22:25:33.129602+0000","last_became_peered":"2021-03-15T22:25:33.129602+0000","last_unstale":"2021-03-15T22:25:36.111926+0000","last_undegraded":"2021-03-15T22:25:36.111926+0000","last_fullsized":"2021-03-15T22:25:36.111926+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.931058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.931058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"219","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101209+0000","last_change":"2021-03-15T22:25:33.143239+0000","last_active":"2021-03-15T22:25:35.101209+0000","last_peered":"2021-03-15T22:25:35.101209+0000","last_clean":"2021-03-15T22:25:35.101209+0000","last_became_active":"2021-03-15T22:25:33.142133+0000","last_became_peered":"2021-03-15T22:25:33.142133+0000","last_unstale":"2021-03-15T22:25:35.101209+0000","last_undegraded":"2021-03-15T22:25:35.101209+0000","last_fullsized":"2021-03-15T22:25:35.101209+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.028082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.028082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"846","reported_epoch":"419","state":"active+clean","last_fresh":"2021-03-15T22:25:34.126468+0000","last_change":"2021-03-15T22:24:31.248181+0000","last_active":"2021-03-15T22:25:34.126468+0000","last_peered":"2021-03-15T22:25:34.126468+0000","last_clean":"2021-03-15T22:25:34.126468+0000","last_became_active":"2021-03-15T22:24:31.109631+0000","last_became_peered":"2021-03-15T22:24:31.109631+0000","last_unstale":"2021-03-15T22:25:34.126468+0000","last_undegraded":"2021-03-15T22:25:34.126468+0000","last_fullsized":"2021-03-15T22:25:34.126468+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:23:27.790040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:23:27.790040+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":446,"num_read_kb":441,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"143","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.111574+0000","last_change":"2021-03-15T22:24:37.977457+0000","last_active":"2021-03-15T22:25:36.111574+0000","last_peered":"2021-03-15T22:25:36.111574+0000","last_clean":"2021-03-15T22:25:36.111574+0000","last_became_active":"2021-03-15T22:24:31.100536+0000","last_became_peered":"2021-03-15T22:24:31.100536+0000","last_unstale":"2021-03-15T22:25:36.111574+0000","last_undegraded":"2021-03-15T22:25:36.111574+0000","last_fullsized":"2021-03-15T22:25:36.111574+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:37.977363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.977363+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"33","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115758+0000","last_change":"2021-03-15T22:25:14.115071+0000","last_active":"2021-03-15T22:25:36.115758+0000","last_peered":"2021-03-15T22:25:36.115758+0000","last_clean":"2021-03-15T22:25:36.115758+0000","last_became_active":"2021-03-15T22:25:14.114729+0000","last_became_peered":"2021-03-15T22:25:14.114729+0000","last_unstale":"2021-03-15T22:25:36.115758+0000","last_undegraded":"2021-03-15T22:25:36.115758+0000","last_fullsized":"2021-03-15T22:25:36.115758+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.857011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.857011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"421'960","reported_seq":"2465","reported_epoch":"422","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-03-15T22:25:37.970124+0000","last_change":"2021-03-15T22:25:35.390581+0000","last_active":"2021-03-15T22:25:37.970124+0000","last_peered":"2021-03-15T22:25:37.970124+0000","last_clean":"2021-03-15T22:25:31.734041+0000","last_became_active":"2021-03-15T22:25:34.232739+0000","last_became_peered":"2021-03-15T22:25:34.232739+0000","last_unstale":"2021-03-15T22:25:37.970124+0000","last_undegraded":"2021-03-15T22:25:34.132435+0000","last_fullsized":"2021-03-15T22:25:34.126282+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'777","last_scrub_stamp":"2021-03-15T22:24:40.640400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.640400+0000","log_size":960,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":57399209,"num_objects":30,"num_object_clones":23,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":1,"num_read":566,"num_read_kb":611849,"num_write":355,"num_write_kb":136357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":71334852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1],"avail_no_missing":["5","1","2"],"object_location_counts":[{"shards":"1,2,5","objects":26},{"shards":"1,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"384","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.113085+0000","last_change":"2021-03-15T22:24:30.950198+0000","last_active":"2021-03-15T22:25:36.113085+0000","last_peered":"2021-03-15T22:25:36.113085+0000","last_clean":"2021-03-15T22:25:36.113085+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:25:36.113085+0000","last_undegraded":"2021-03-15T22:25:36.113085+0000","last_fullsized":"2021-03-15T22:25:36.113085+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.950135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.950135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"390","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112111+0000","last_change":"2021-03-15T22:24:26.962034+0000","last_active":"2021-03-15T22:25:36.112111+0000","last_peered":"2021-03-15T22:25:36.112111+0000","last_clean":"2021-03-15T22:25:36.112111+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:25:36.112111+0000","last_undegraded":"2021-03-15T22:25:36.112111+0000","last_fullsized":"2021-03-15T22:25:36.112111+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.961978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.961978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"166","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116394+0000","last_change":"2021-03-15T22:24:31.147838+0000","last_active":"2021-03-15T22:25:36.116394+0000","last_peered":"2021-03-15T22:25:36.116394+0000","last_clean":"2021-03-15T22:25:36.116394+0000","last_became_active":"2021-03-15T22:24:31.098665+0000","last_became_peered":"2021-03-15T22:24:31.098665+0000","last_unstale":"2021-03-15T22:25:36.116394+0000","last_undegraded":"2021-03-15T22:25:36.116394+0000","last_fullsized":"2021-03-15T22:25:36.116394+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:23:36.678184+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:23:36.678184+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"394","reported_epoch":"419","state":"active+clean","last_fresh":"2021-03-15T22:25:34.127667+0000","last_change":"2021-03-15T22:24:34.454377+0000","last_active":"2021-03-15T22:25:34.127667+0000","last_peered":"2021-03-15T22:25:34.127667+0000","last_clean":"2021-03-15T22:25:34.127667+0000","last_became_active":"2021-03-15T22:24:31.107629+0000","last_became_peered":"2021-03-15T22:24:31.107629+0000","last_unstale":"2021-03-15T22:25:34.127667+0000","last_undegraded":"2021-03-15T22:25:34.127667+0000","last_fullsized":"2021-03-15T22:25:34.127667+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.454313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.454313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"413'144","reported_seq":"1205","reported_epoch":"421","state":"peering","last_fresh":"2021-03-15T22:25:36.108729+0000","last_change":"2021-03-15T22:25:36.108921+0000","last_active":"2021-03-15T22:25:36.108729+0000","last_peered":"2021-03-15T22:25:35.333310+0000","last_clean":"2021-03-15T22:25:31.105707+0000","last_became_active":"2021-03-15T22:25:34.214131+0000","last_became_peered":"2021-03-15T22:25:34.214131+0000","last_unstale":"2021-03-15T22:25:36.108729+0000","last_undegraded":"2021-03-15T22:25:36.108729+0000","last_fullsized":"2021-03-15T22:25:36.108729+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":173,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'104","last_scrub_stamp":"2021-03-15T22:24:02.949625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:02.949625+0000","log_size":144,"ondisk_log_size":144,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11376639,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":77,"num_read_kb":100353,"num_write":49,"num_write_kb":21048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":14412391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,3,6],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"419","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.099907+0000","last_change":"2021-03-15T22:25:14.112119+0000","last_active":"2021-03-15T22:25:35.099907+0000","last_peered":"2021-03-15T22:25:35.099907+0000","last_clean":"2021-03-15T22:25:35.099907+0000","last_became_active":"2021-03-15T22:25:14.111797+0000","last_became_peered":"2021-03-15T22:25:14.111797+0000","last_unstale":"2021-03-15T22:25:35.099907+0000","last_undegraded":"2021-03-15T22:25:35.099907+0000","last_fullsized":"2021-03-15T22:25:35.099907+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.480279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"400","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116610+0000","last_change":"2021-03-15T22:25:33.141491+0000","last_active":"2021-03-15T22:25:36.116610+0000","last_peered":"2021-03-15T22:25:36.116610+0000","last_clean":"2021-03-15T22:25:36.116610+0000","last_became_active":"2021-03-15T22:25:33.141147+0000","last_became_peered":"2021-03-15T22:25:33.141147+0000","last_unstale":"2021-03-15T22:25:36.116610+0000","last_undegraded":"2021-03-15T22:25:36.116610+0000","last_fullsized":"2021-03-15T22:25:36.116610+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.769850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.769850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"513","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112842+0000","last_change":"2021-03-15T22:25:15.063557+0000","last_active":"2021-03-15T22:25:36.112842+0000","last_peered":"2021-03-15T22:25:36.112842+0000","last_clean":"2021-03-15T22:25:36.112842+0000","last_became_active":"2021-03-15T22:25:14.114107+0000","last_became_peered":"2021-03-15T22:25:14.114107+0000","last_unstale":"2021-03-15T22:25:36.112842+0000","last_undegraded":"2021-03-15T22:25:36.112842+0000","last_fullsized":"2021-03-15T22:25:36.112842+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:23:29.014596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.014596+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"356'3","reported_seq":"101","reported_epoch":"419","state":"active+clean","last_fresh":"2021-03-15T22:25:34.127934+0000","last_change":"2021-03-15T22:24:31.111939+0000","last_active":"2021-03-15T22:25:34.127934+0000","last_peered":"2021-03-15T22:25:34.127934+0000","last_clean":"2021-03-15T22:25:34.127934+0000","last_became_active":"2021-03-15T22:24:31.105948+0000","last_became_peered":"2021-03-15T22:24:31.105948+0000","last_unstale":"2021-03-15T22:25:34.127934+0000","last_undegraded":"2021-03-15T22:25:34.127934+0000","last_fullsized":"2021-03-15T22:25:34.127934+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:24:02.507323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:02.507323+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"420'247","reported_seq":"1268","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.122269+0000","last_change":"2021-03-15T22:25:34.207911+0000","last_active":"2021-03-15T22:25:35.122269+0000","last_peered":"2021-03-15T22:25:35.122269+0000","last_clean":"2021-03-15T22:25:35.122269+0000","last_became_active":"2021-03-15T22:23:00.057259+0000","last_became_peered":"2021-03-15T22:23:00.057259+0000","last_unstale":"2021-03-15T22:25:35.122269+0000","last_undegraded":"2021-03-15T22:25:35.122269+0000","last_fullsized":"2021-03-15T22:25:35.122269+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'202","last_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.006150+0000","log_size":247,"ondisk_log_size":247,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13478841,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":114,"num_read_kb":109752,"num_write":89,"num_write_kb":33243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":6765552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"12b","length":"1"},{"start":"139","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"379","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.136817+0000","last_change":"2021-03-15T22:25:16.234631+0000","last_active":"2021-03-15T22:25:33.136817+0000","last_peered":"2021-03-15T22:25:33.136817+0000","last_clean":"2021-03-15T22:25:33.136817+0000","last_became_active":"2021-03-15T22:23:52.260549+0000","last_became_peered":"2021-03-15T22:23:52.260549+0000","last_unstale":"2021-03-15T22:25:33.136817+0000","last_undegraded":"2021-03-15T22:25:33.136817+0000","last_fullsized":"2021-03-15T22:25:33.136817+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.234565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.234565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"84","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.110621+0000","last_change":"2021-03-15T22:25:02.868460+0000","last_active":"2021-03-15T22:25:36.110621+0000","last_peered":"2021-03-15T22:25:36.110621+0000","last_clean":"2021-03-15T22:25:36.110621+0000","last_became_active":"2021-03-15T22:24:31.099610+0000","last_became_peered":"2021-03-15T22:24:31.099610+0000","last_unstale":"2021-03-15T22:25:36.110621+0000","last_undegraded":"2021-03-15T22:25:36.110621+0000","last_fullsized":"2021-03-15T22:25:36.110621+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.868399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.868399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"418","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101345+0000","last_change":"2021-03-15T22:23:07.762430+0000","last_active":"2021-03-15T22:25:35.101345+0000","last_peered":"2021-03-15T22:25:35.101345+0000","last_clean":"2021-03-15T22:25:35.101345+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:25:35.101345+0000","last_undegraded":"2021-03-15T22:25:35.101345+0000","last_fullsized":"2021-03-15T22:25:35.101345+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:23:07.762338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:07.762338+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"356'5","reported_seq":"17","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.099740+0000","last_change":"2021-03-15T22:25:33.171536+0000","last_active":"2021-03-15T22:25:35.099740+0000","last_peered":"2021-03-15T22:25:35.099740+0000","last_clean":"2021-03-15T22:25:35.099740+0000","last_became_active":"2021-03-15T22:25:33.130973+0000","last_became_peered":"2021-03-15T22:25:33.130973+0000","last_unstale":"2021-03-15T22:25:35.099740+0000","last_undegraded":"2021-03-15T22:25:35.099740+0000","last_fullsized":"2021-03-15T22:25:35.099740+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'4","last_scrub_stamp":"2021-03-15T22:23:35.100079+0000","last_deep_scrub":"300'4","last_deep_scrub_stamp":"2021-03-15T22:23:35.100079+0000","last_clean_scrub_stamp":"2021-03-15T22:23:35.100079+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"420'428","reported_seq":"1551","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.161114+0000","last_change":"2021-03-15T22:25:34.208140+0000","last_active":"2021-03-15T22:25:35.161114+0000","last_peered":"2021-03-15T22:25:35.161114+0000","last_clean":"2021-03-15T22:25:35.161114+0000","last_became_active":"2021-03-15T22:23:01.072357+0000","last_became_peered":"2021-03-15T22:23:01.072357+0000","last_unstale":"2021-03-15T22:25:35.161114+0000","last_undegraded":"2021-03-15T22:25:35.161114+0000","last_fullsized":"2021-03-15T22:25:35.161114+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'302","last_scrub_stamp":"2021-03-15T22:23:59.467761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:59.467761+0000","log_size":428,"ondisk_log_size":428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21631749,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":238,"num_read_kb":259097,"num_write":143,"num_write_kb":53508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31233387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"12b","length":"1"},{"start":"139","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"399","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.102083+0000","last_change":"2021-03-15T22:25:33.136943+0000","last_active":"2021-03-15T22:25:35.102083+0000","last_peered":"2021-03-15T22:25:35.102083+0000","last_clean":"2021-03-15T22:25:35.102083+0000","last_became_active":"2021-03-15T22:25:33.136621+0000","last_became_peered":"2021-03-15T22:25:33.136621+0000","last_unstale":"2021-03-15T22:25:35.102083+0000","last_undegraded":"2021-03-15T22:25:35.102083+0000","last_fullsized":"2021-03-15T22:25:35.102083+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:32.751660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:32.751660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"189","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.106247+0000","last_change":"2021-03-15T22:25:12.911203+0000","last_active":"2021-03-15T22:25:35.106247+0000","last_peered":"2021-03-15T22:25:35.106247+0000","last_clean":"2021-03-15T22:25:35.106247+0000","last_became_active":"2021-03-15T22:24:31.091206+0000","last_became_peered":"2021-03-15T22:24:31.091206+0000","last_unstale":"2021-03-15T22:25:35.106247+0000","last_undegraded":"2021-03-15T22:25:35.106247+0000","last_fullsized":"2021-03-15T22:25:35.106247+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.911140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.911140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"39","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.099593+0000","last_change":"2021-03-15T22:25:33.758535+0000","last_active":"2021-03-15T22:25:35.099593+0000","last_peered":"2021-03-15T22:25:35.099593+0000","last_clean":"2021-03-15T22:25:35.099593+0000","last_became_active":"2021-03-15T22:25:33.149362+0000","last_became_peered":"2021-03-15T22:25:33.149362+0000","last_unstale":"2021-03-15T22:25:35.099593+0000","last_undegraded":"2021-03-15T22:25:35.099593+0000","last_fullsized":"2021-03-15T22:25:35.099593+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:24:01.896988+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.896988+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"412","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101090+0000","last_change":"2021-03-15T22:25:33.143430+0000","last_active":"2021-03-15T22:25:35.101090+0000","last_peered":"2021-03-15T22:25:35.101090+0000","last_clean":"2021-03-15T22:25:35.101090+0000","last_became_active":"2021-03-15T22:25:33.142712+0000","last_became_peered":"2021-03-15T22:25:33.142712+0000","last_unstale":"2021-03-15T22:25:35.101090+0000","last_undegraded":"2021-03-15T22:25:35.101090+0000","last_fullsized":"2021-03-15T22:25:35.101090+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.989313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.989313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"413'408","reported_seq":"1527","reported_epoch":"420","state":"peering","last_fresh":"2021-03-15T22:25:35.104219+0000","last_change":"2021-03-15T22:25:35.104412+0000","last_active":"2021-03-15T22:25:35.104219+0000","last_peered":"2021-03-15T22:25:34.331488+0000","last_clean":"2021-03-15T22:25:31.532432+0000","last_became_active":"2021-03-15T22:25:34.218896+0000","last_became_peered":"2021-03-15T22:25:34.218896+0000","last_unstale":"2021-03-15T22:25:35.104219+0000","last_undegraded":"2021-03-15T22:25:35.104219+0000","last_fullsized":"2021-03-15T22:25:35.104219+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":176,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'370","last_scrub_stamp":"2021-03-15T22:25:07.414191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.414191+0000","log_size":408,"ondisk_log_size":408,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21706719,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":180,"num_read_kb":168082,"num_write":128,"num_write_kb":48783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25937939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3,7],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"115","length":"2"}]},{"pgid":"6.d","version":"0'0","reported_seq":"385","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116380+0000","last_change":"2021-03-15T22:24:31.094621+0000","last_active":"2021-03-15T22:25:36.116380+0000","last_peered":"2021-03-15T22:25:36.116380+0000","last_clean":"2021-03-15T22:25:36.116380+0000","last_became_active":"2021-03-15T22:24:31.094299+0000","last_became_peered":"2021-03-15T22:24:31.094299+0000","last_unstale":"2021-03-15T22:25:36.116380+0000","last_undegraded":"2021-03-15T22:25:36.116380+0000","last_fullsized":"2021-03-15T22:25:36.116380+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.936738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.936738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"391'7","reported_seq":"525","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101703+0000","last_change":"2021-03-15T22:25:03.403843+0000","last_active":"2021-03-15T22:25:35.101703+0000","last_peered":"2021-03-15T22:25:35.101703+0000","last_clean":"2021-03-15T22:25:35.101703+0000","last_became_active":"2021-03-15T22:25:03.403350+0000","last_became_peered":"2021-03-15T22:25:03.403350+0000","last_unstale":"2021-03-15T22:25:35.101703+0000","last_undegraded":"2021-03-15T22:25:35.101703+0000","last_fullsized":"2021-03-15T22:25:35.101703+0000","mapping_epoch":387,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'4","last_scrub_stamp":"2021-03-15T22:23:35.793433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:23:35.793433+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"13","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.099660+0000","last_change":"2021-03-15T22:25:33.148929+0000","last_active":"2021-03-15T22:25:35.099660+0000","last_peered":"2021-03-15T22:25:35.099660+0000","last_clean":"2021-03-15T22:25:35.099660+0000","last_became_active":"2021-03-15T22:25:33.148614+0000","last_became_peered":"2021-03-15T22:25:33.148614+0000","last_unstale":"2021-03-15T22:25:35.099660+0000","last_undegraded":"2021-03-15T22:25:35.099660+0000","last_fullsized":"2021-03-15T22:25:35.099660+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:04.268977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:04.268977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"444","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.120143+0000","last_change":"2021-03-15T22:23:58.133442+0000","last_active":"2021-03-15T22:25:33.120143+0000","last_peered":"2021-03-15T22:25:33.120143+0000","last_clean":"2021-03-15T22:25:33.120143+0000","last_became_active":"2021-03-15T22:22:56.057343+0000","last_became_peered":"2021-03-15T22:22:56.057343+0000","last_unstale":"2021-03-15T22:25:33.120143+0000","last_undegraded":"2021-03-15T22:25:33.120143+0000","last_fullsized":"2021-03-15T22:25:33.120143+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:23:58.133378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.133378+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"394'77","reported_seq":"1117","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.104107+0000","last_change":"2021-03-15T22:25:34.128857+0000","last_active":"2021-03-15T22:25:35.104107+0000","last_peered":"2021-03-15T22:25:35.104107+0000","last_clean":"2021-03-15T22:25:35.104107+0000","last_became_active":"2021-03-15T22:24:31.104356+0000","last_became_peered":"2021-03-15T22:24:31.104356+0000","last_unstale":"2021-03-15T22:25:35.104107+0000","last_undegraded":"2021-03-15T22:25:35.104107+0000","last_fullsized":"2021-03-15T22:25:35.104107+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'77","last_scrub_stamp":"2021-03-15T22:25:32.490694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.490694+0000","log_size":77,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5415987,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":79,"num_read_kb":80817,"num_write":31,"num_write_kb":11490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1551588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12b","length":"1"},{"start":"139","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"13","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.104923+0000","last_change":"2021-03-15T22:25:33.148157+0000","last_active":"2021-03-15T22:25:35.104923+0000","last_peered":"2021-03-15T22:25:35.104923+0000","last_clean":"2021-03-15T22:25:35.104923+0000","last_became_active":"2021-03-15T22:25:33.147717+0000","last_became_peered":"2021-03-15T22:25:33.147717+0000","last_unstale":"2021-03-15T22:25:35.104923+0000","last_undegraded":"2021-03-15T22:25:35.104923+0000","last_fullsized":"2021-03-15T22:25:35.104923+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.372535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.372535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"147'5","reported_seq":"458","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:37.085461+0000","last_change":"2021-03-15T22:24:35.715464+0000","last_active":"2021-03-15T22:25:37.085461+0000","last_peered":"2021-03-15T22:25:37.085461+0000","last_clean":"2021-03-15T22:25:37.085461+0000","last_became_active":"2021-03-15T22:21:02.017066+0000","last_became_peered":"2021-03-15T22:21:02.017066+0000","last_unstale":"2021-03-15T22:25:37.085461+0000","last_undegraded":"2021-03-15T22:25:37.085461+0000","last_fullsized":"2021-03-15T22:25:37.085461+0000","mapping_epoch":60,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'5","last_scrub_stamp":"2021-03-15T22:24:35.715399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:35.715399+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"389","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101165+0000","last_change":"2021-03-15T22:24:34.991087+0000","last_active":"2021-03-15T22:25:35.101165+0000","last_peered":"2021-03-15T22:25:35.101165+0000","last_clean":"2021-03-15T22:25:35.101165+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:25:35.101165+0000","last_undegraded":"2021-03-15T22:25:35.101165+0000","last_fullsized":"2021-03-15T22:25:35.101165+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.991018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.991018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"470","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101522+0000","last_change":"2021-03-15T22:24:55.128840+0000","last_active":"2021-03-15T22:25:35.101522+0000","last_peered":"2021-03-15T22:25:35.101522+0000","last_clean":"2021-03-15T22:25:35.101522+0000","last_became_active":"2021-03-15T22:24:55.090544+0000","last_became_peered":"2021-03-15T22:24:55.090544+0000","last_unstale":"2021-03-15T22:25:35.101522+0000","last_undegraded":"2021-03-15T22:25:35.101522+0000","last_fullsized":"2021-03-15T22:25:35.101522+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:38.685706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:38.685706+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"413'125","reported_seq":"1244","reported_epoch":"419","state":"active+recovering+undersized+remapped","last_fresh":"2021-03-15T22:25:34.973570+0000","last_change":"2021-03-15T22:25:34.444713+0000","last_active":"2021-03-15T22:25:34.973570+0000","last_peered":"2021-03-15T22:25:34.973570+0000","last_clean":"2021-03-15T22:25:31.106804+0000","last_became_active":"2021-03-15T22:25:34.230028+0000","last_became_peered":"2021-03-15T22:25:34.230028+0000","last_unstale":"2021-03-15T22:25:34.973570+0000","last_undegraded":"2021-03-15T22:25:34.973570+0000","last_fullsized":"2021-03-15T22:25:34.128183+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":357,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'115","last_scrub_stamp":"2021-03-15T22:25:10.616929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.616929+0000","log_size":125,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":9539709,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":100,"num_read_kb":121122,"num_write":53,"num_write_kb":14217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":11612512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5],"avail_no_missing":["2","1","5"],"object_location_counts":[{"shards":"1,2,5","objects":4},{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"115","length":"2"}]},{"pgid":"6.f","version":"0'0","reported_seq":"397","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.135456+0000","last_change":"2021-03-15T22:25:33.134612+0000","last_active":"2021-03-15T22:25:33.135456+0000","last_peered":"2021-03-15T22:25:33.135456+0000","last_clean":"2021-03-15T22:25:33.135456+0000","last_became_active":"2021-03-15T22:25:33.134275+0000","last_became_peered":"2021-03-15T22:25:33.134275+0000","last_unstale":"2021-03-15T22:25:33.135456+0000","last_undegraded":"2021-03-15T22:25:33.135456+0000","last_fullsized":"2021-03-15T22:25:33.135456+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:23:02.474236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"413","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112249+0000","last_change":"2021-03-15T22:24:31.107075+0000","last_active":"2021-03-15T22:25:36.112249+0000","last_peered":"2021-03-15T22:25:36.112249+0000","last_clean":"2021-03-15T22:25:36.112249+0000","last_became_active":"2021-03-15T22:24:31.106634+0000","last_became_peered":"2021-03-15T22:24:31.106634+0000","last_unstale":"2021-03-15T22:25:36.112249+0000","last_undegraded":"2021-03-15T22:25:36.112249+0000","last_fullsized":"2021-03-15T22:25:36.112249+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:42.177422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.177422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"392","reported_epoch":"419","state":"active+clean","last_fresh":"2021-03-15T22:25:34.127464+0000","last_change":"2021-03-15T22:24:28.327738+0000","last_active":"2021-03-15T22:25:34.127464+0000","last_peered":"2021-03-15T22:25:34.127464+0000","last_clean":"2021-03-15T22:25:34.127464+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:25:34.127464+0000","last_undegraded":"2021-03-15T22:25:34.127464+0000","last_fullsized":"2021-03-15T22:25:34.127464+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:28.327677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.327677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"115'33","reported_seq":"435","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101788+0000","last_change":"2021-03-15T22:25:04.028853+0000","last_active":"2021-03-15T22:25:35.101788+0000","last_peered":"2021-03-15T22:25:35.101788+0000","last_clean":"2021-03-15T22:25:35.101788+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:25:35.101788+0000","last_undegraded":"2021-03-15T22:25:35.101788+0000","last_fullsized":"2021-03-15T22:25:35.101788+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'33","last_scrub_stamp":"2021-03-15T22:25:04.028780+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:25:04.028780+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"410'378","reported_seq":"1691","reported_epoch":"421","state":"active+recovering+undersized+remapped","last_fresh":"2021-03-15T22:25:36.987363+0000","last_change":"2021-03-15T22:25:34.226291+0000","last_active":"2021-03-15T22:25:36.987363+0000","last_peered":"2021-03-15T22:25:36.987363+0000","last_clean":"2021-03-15T22:25:05.439431+0000","last_became_active":"2021-03-15T22:25:34.223404+0000","last_became_peered":"2021-03-15T22:25:34.223404+0000","last_unstale":"2021-03-15T22:25:36.987363+0000","last_undegraded":"2021-03-15T22:25:36.987363+0000","last_fullsized":"2021-03-15T22:25:34.124777+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'343","last_scrub_stamp":"2021-03-15T22:23:26.791043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.791043+0000","log_size":378,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"stat_sum":{"num_bytes":7647416,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":183,"num_read_kb":156184,"num_write":134,"num_write_kb":53144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":36525000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[4,7],"avail_no_missing":["4","5","7"],"object_location_counts":[{"shards":"4,5,7","objects":3},{"shards":"4,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"381","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.103286+0000","last_change":"2021-03-15T22:24:40.910630+0000","last_active":"2021-03-15T22:25:35.103286+0000","last_peered":"2021-03-15T22:25:35.103286+0000","last_clean":"2021-03-15T22:25:35.103286+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:25:35.103286+0000","last_undegraded":"2021-03-15T22:25:35.103286+0000","last_fullsized":"2021-03-15T22:25:35.103286+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.910565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.910565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"155'4","reported_seq":"498","reported_epoch":"418","state":"active+clean","last_fresh":"2021-03-15T22:25:33.120280+0000","last_change":"2021-03-15T22:24:29.903505+0000","last_active":"2021-03-15T22:25:33.120280+0000","last_peered":"2021-03-15T22:25:33.120280+0000","last_clean":"2021-03-15T22:25:33.120280+0000","last_became_active":"2021-03-15T22:21:10.013838+0000","last_became_peered":"2021-03-15T22:21:10.013838+0000","last_unstale":"2021-03-15T22:25:33.120280+0000","last_undegraded":"2021-03-15T22:25:33.120280+0000","last_fullsized":"2021-03-15T22:25:33.120280+0000","mapping_epoch":60,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'4","last_scrub_stamp":"2021-03-15T22:24:29.903442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.903442+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"99","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.100439+0000","last_change":"2021-03-15T22:25:02.791176+0000","last_active":"2021-03-15T22:25:35.100439+0000","last_peered":"2021-03-15T22:25:35.100439+0000","last_clean":"2021-03-15T22:25:35.100439+0000","last_became_active":"2021-03-15T22:24:31.097434+0000","last_became_peered":"2021-03-15T22:24:31.097434+0000","last_unstale":"2021-03-15T22:25:35.100439+0000","last_undegraded":"2021-03-15T22:25:35.100439+0000","last_fullsized":"2021-03-15T22:25:35.100439+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.791108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:06.573181+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.791108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"462","reported_epoch":"420","state":"active+clean","last_fresh":"2021-03-15T22:25:35.101463+0000","last_change":"2021-03-15T22:25:34.027206+0000","last_active":"2021-03-15T22:25:35.101463+0000","last_peered":"2021-03-15T22:25:35.101463+0000","last_clean":"2021-03-15T22:25:35.101463+0000","last_became_active":"2021-03-15T22:22:56.053800+0000","last_became_peered":"2021-03-15T22:22:56.053800+0000","last_unstale":"2021-03-15T22:25:35.101463+0000","last_undegraded":"2021-03-15T22:25:35.101463+0000","last_fullsized":"2021-03-15T22:25:35.101463+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:23:05.791845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:05.791845+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":329261567,"num_objects":165,"num_object_clones":117,"num_object_copies":495,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":99,"num_objects_misplaced":33,"num_objects_unfound":0,"num_objects_dirty":165,"num_whiteouts":13,"num_read":3013,"num_read_kb":3030444,"num_write":2197,"num_write_kb":820308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1023,"num_bytes_recovered":399474011,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":849752064,"data_stored":1185565307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5940,"ondisk_log_size":5940,"up":48,"acting":44,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":446,"num_read_kb":441,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1061,"num_read_kb":566,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":37148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2232320,"data_stored":80092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":30,"ondisk_log_size":30,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299488,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":152960,"kb_used_data":118840,"kb_used_omap":0,"kb_used_meta":34112,"kb_avail":93592192,"statfs":{"total":95995035648,"available":95838404608,"internally_reserved":0,"allocated":121692160,"data_stored":162563426,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34930688},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.0189999999999999,"5min":2.73,"15min":2.73},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65000000000000002,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.627}]},{"osd":1,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.504,"15min":0.48599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":0.97299999999999998,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.436,"5min":0.48499999999999999,"15min":0.48999999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72699999999999998,"5min":1.877,"15min":1.877},"last":0.57199999999999995}]},{"osd":2,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.47199999999999998,"15min":0.47899999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73499999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.498,"5min":0.51900000000000002,"15min":0.5},"min":{"1min":0.311,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":0.95099999999999996,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.496}]},{"osd":3,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.54200000000000004},"min":{"1min":0.315,"5min":0.31,"15min":0.31},"max":{"1min":0.69499999999999995,"5min":0.95999999999999996,"15min":2.0339999999999998},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98899999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.59199999999999997}]},{"osd":4,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.46600000000000003,"15min":0.436},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.75700000000000001,"5min":1.835,"15min":1.835},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47699999999999998,"15min":0.44600000000000001},"min":{"1min":0.29099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66200000000000003,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.40300000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52200000000000002,"15min":0.45800000000000002},"min":{"1min":0.31900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.034,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":0.373},{"interface":"front","average":{"1min":0.495,"5min":0.53100000000000003,"15min":0.504},"min":{"1min":0.33900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.78300000000000003,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.51800000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54600000000000004,"15min":0.51000000000000001},"min":{"1min":0.34499999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80700000000000005,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.53700000000000003,"15min":0.50900000000000001},"min":{"1min":0.35299999999999998,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.77700000000000002,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.52800000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528420,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":102700,"kb_used_data":75172,"kb_used_omap":0,"kb_used_meta":27520,"kb_avail":93642452,"statfs":{"total":95995035648,"available":95889870848,"internally_reserved":0,"allocated":76976128,"data_stored":99775419,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28180480},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.71399999999999997,"5min":0.995,"15min":0.995},"last":0.27100000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.47999999999999998,"15min":0.48899999999999999},"min":{"1min":0.29299999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63100000000000001,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.40400000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.67800000000000005,"5min":1.05,"15min":1.05},"last":0.308},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65300000000000002,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.46500000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.69499999999999995,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.5},{"interface":"front","average":{"1min":0.5,"5min":0.495,"15min":0.5},"min":{"1min":0.33100000000000002,"5min":0.32200000000000001,"15min":0.31900000000000001},"max":{"1min":1.1419999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.377}]},{"osd":3,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51300000000000001,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76500000000000001,"5min":1.026,"15min":1.026},"last":0.44},{"interface":"front","average":{"1min":0.504,"5min":0.51600000000000001,"15min":0.52500000000000002},"min":{"1min":0.38700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69999999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.33600000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.44900000000000001,"15min":0.41299999999999998},"min":{"1min":0.31,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.45500000000000002},"min":{"1min":0.315,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82799999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.35999999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.499,"15min":0.45600000000000002},"min":{"1min":0.31,"5min":0.309,"15min":0.309},"max":{"1min":0.71599999999999997,"5min":1.135,"15min":1.135},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53000000000000003,"15min":0.48399999999999999},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.84999999999999998,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.38900000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52300000000000002,"15min":0.50600000000000001},"min":{"1min":0.36799999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.79600000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.76100000000000001,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.48799999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147276,"kb_used_data":122692,"kb_used_omap":0,"kb_used_meta":24576,"kb_avail":93597876,"statfs":{"total":95995035648,"available":95844225024,"internally_reserved":0,"allocated":125636608,"data_stored":179150765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25165824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.34499999999999997,"15min":0.52700000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.48899999999999999,"5min":1.371,"15min":54.427999999999997},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.54400000000000004},"min":{"1min":0.20999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.48199999999999998,"5min":1.23,"15min":54.356999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.84699999999999998,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.314},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.39100000000000001},"min":{"1min":0.25600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":2.0760000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:24:57 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38500000000000001,"15min":0.41899999999999998},"min":{"1min":0.26100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.796},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40799999999999997,"15min":0.46400000000000002},"min":{"1min":0.29299999999999998,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.623,"5min":1.135,"15min":1.7150000000000001},"last":0.376}]},{"osd":4,"last update":"Mon Mar 15 22:25:10 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.95199999999999996,"5min":3.3439999999999999,"15min":3.3439999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.621,"15min":0.626},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.93999999999999995,"5min":4.6139999999999999,"15min":4.6139999999999999},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.627},"min":{"1min":0.42199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.93400000000000005,"5min":1.524,"15min":2.153},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58499999999999996,"15min":0.95899999999999996},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88900000000000001,"5min":1.1950000000000001,"15min":15.472},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.60799999999999998,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.99199999999999999,"5min":1.276,"15min":1.75},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58199999999999996,"15min":0.627},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.3580000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.63500000000000001,"15min":0.871},"min":{"1min":0.377,"5min":0.316,"15min":0.316},"max":{"1min":0.96299999999999997,"5min":2.1219999999999999,"15min":8.8100000000000005},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.623},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84799999999999998,"5min":1.0329999999999999,"15min":1.325},"last":0.55400000000000005}]}]},{"osd":0,"up_from":0,"seq":34359738488,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411442,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":211068,"kb_used_data":185652,"kb_used_omap":0,"kb_used_meta":25408,"kb_avail":93534084,"statfs":{"total":95995035648,"available":95778902016,"internally_reserved":0,"allocated":190107648,"data_stored":269885565,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26017792},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:24:41 2021","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.33500000000000002,"15min":0.35099999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.39900000000000002,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.376,"15min":0.379},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.65900000000000003,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.309}]},{"osd":1,"last update":"Mon Mar 15 22:24:41 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.42499999999999999,"15min":0.40000000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.38700000000000001,"5min":5.4489999999999998,"15min":5.4489999999999998},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.499,"15min":0.46500000000000002},"min":{"1min":0.23400000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.435,"5min":11.659000000000001,"15min":11.659000000000001},"last":0.34699999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:24:53 2021","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.38400000000000001,"15min":0.41799999999999998},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.55600000000000005,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.443,"15min":0.48299999999999998},"min":{"1min":0.27400000000000002,"5min":0.26400000000000001,"15min":0.24099999999999999},"max":{"1min":2.8759999999999999,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.38100000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:25:09 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56999999999999995,"15min":0.57499999999999996},"min":{"1min":0.40999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73899999999999999,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54500000000000004,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.60199999999999998,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.48799999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:24:35 2021","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.57399999999999995,"15min":0.57799999999999996},"min":{"1min":0.36499999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75700000000000001,"5min":1.954,"15min":1.954},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.59099999999999997,"15min":0.56999999999999995},"min":{"1min":0.29199999999999998,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.73099999999999998,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.40000000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:24:41 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59099999999999997,"15min":0.57099999999999995},"min":{"1min":0.35999999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.748,"5min":1.875,"15min":1.875},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58299999999999996,"15min":0.57099999999999995},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75700000000000001,"5min":1.609,"15min":1.609},"last":0.47899999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:25:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.66900000000000004,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.40600000000000003,"15min":0.38700000000000001},"max":{"1min":0.64700000000000002,"5min":1.923,"15min":1.923},"last":0.52500000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149806,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":175512,"kb_used_data":155280,"kb_used_omap":0,"kb_used_meta":20224,"kb_avail":93569640,"statfs":{"total":95995035648,"available":95815311360,"internally_reserved":0,"allocated":159006720,"data_stored":205925363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20709376},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,2],"upper_bound":512},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.35199999999999998,"15min":0.35799999999999998},"min":{"1min":0.22700000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.35699999999999998,"5min":2.625,"15min":2.625},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.36499999999999999,"15min":0.379},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.189},"max":{"1min":0.438,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.315}]},{"osd":1,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.44900000000000001,"15min":0.434},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.50700000000000001,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.373},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.46100000000000002,"15min":0.437},"min":{"1min":0.28699999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.45300000000000001,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.45000000000000001,"15min":0.435},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.48199999999999998,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.251},{"interface":"front","average":{"1min":0.316,"5min":0.42299999999999999,"15min":0.40400000000000003},"min":{"1min":0.26600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46300000000000002,"5min":7.0860000000000003,"15min":7.0860000000000003},"last":0.27700000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:25:10 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55500000000000005,"15min":0.52400000000000002},"min":{"1min":0.32600000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.66800000000000004,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.31},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.318,"15min":0.30399999999999999},"max":{"1min":0.69299999999999995,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.53800000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.54500000000000004},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.62,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":0.40500000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:24:41 2021","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.73499999999999999,"5min":2.6499999999999999,"15min":2.6499999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.57199999999999995,"15min":0.58299999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.36499999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.58399999999999996,"15min":0.57699999999999996},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67400000000000004,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.377},"max":{"1min":0.72699999999999998,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.56499999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986283,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":85844,"kb_used_data":58124,"kb_used_omap":0,"kb_used_meta":27712,"kb_avail":93659308,"statfs":{"total":95995035648,"available":95907131392,"internally_reserved":0,"allocated":59518976,"data_stored":82272805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28377088},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.19500000000000001},"max":{"1min":0.73099999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48999999999999999},"min":{"1min":0.34999999999999998,"5min":0.29099999999999998,"15min":0.251},"max":{"1min":0.66600000000000004,"5min":0.72199999999999998,"15min":0.74199999999999999},"last":0.41799999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.47599999999999998,"15min":0.5},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23599999999999999},"max":{"1min":0.68100000000000005,"5min":1.631,"15min":1.631},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26800000000000002},"max":{"1min":0.73999999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.51100000000000001},"min":{"1min":0.40300000000000002,"5min":0.32300000000000001,"15min":0.313},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.76100000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":1.1910000000000001},"last":0.439}]},{"osd":3,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52600000000000002,"15min":0.55000000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68899999999999995,"5min":0.75,"15min":0.98799999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51500000000000001,"15min":0.55800000000000005},"min":{"1min":0.40200000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68899999999999995,"5min":0.69199999999999995,"15min":0.99399999999999999},"last":0.49299999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:25:28 2021","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.41799999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.249},{"interface":"front","average":{"1min":0.502,"5min":0.49299999999999999,"15min":0.441},"min":{"1min":0.36099999999999999,"5min":0.29699999999999999,"15min":0.28799999999999998},"max":{"1min":0.64400000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.26000000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:24:45 2021","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.46700000000000003,"15min":0.432},"min":{"1min":0.29999999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.57499999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55100000000000005,"15min":0.51100000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34300000000000003},"max":{"1min":0.73899999999999999,"5min":0.78400000000000003,"15min":0.84799999999999998},"last":0.309}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.48899999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.70899999999999996,"5min":0.73799999999999999,"15min":0.80600000000000005},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55400000000000005,"15min":0.52500000000000002},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.75,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.35299999999999998}]}]},{"osd":5,"up_from":38,"seq":163208757352,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":155064,"kb_used_data":136112,"kb_used_omap":0,"kb_used_meta":18944,"kb_avail":93590088,"statfs":{"total":95995035648,"available":95836250112,"internally_reserved":0,"allocated":139378688,"data_stored":196800433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19398656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48799999999999999,"15min":0.48299999999999998},"min":{"1min":0.29499999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63500000000000001,"5min":2.855,"15min":2.855},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.498},"min":{"1min":0.34699999999999998,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.75700000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.374}]},{"osd":1,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.33100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.72999999999999998,"5min":2.665,"15min":2.665},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.72099999999999997,"5min":0.81999999999999995,"15min":0.83999999999999997},"last":0.315}]},{"osd":2,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.39000000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61799999999999999,"5min":2.827,"15min":2.827},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.51400000000000001},"min":{"1min":0.29899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.0740000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.55100000000000005,"15min":0.51800000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54800000000000004,"15min":0.53300000000000003},"min":{"1min":0.39400000000000002,"5min":0.34100000000000003,"15min":0.33200000000000002},"max":{"1min":0.65500000000000003,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46999999999999997,"15min":0.40899999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24099999999999999},"max":{"1min":0.70899999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.245},{"interface":"front","average":{"1min":0.496,"5min":0.505,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.25},"max":{"1min":0.748,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.27800000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.50700000000000001,"15min":0.46899999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28499999999999998},"max":{"1min":0.751,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57799999999999996,"15min":0.53300000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.32800000000000001},"max":{"1min":0.86199999999999999,"5min":1.327,"15min":1.327},"last":0.40400000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.34699999999999998,"15min":0.30499999999999999},"max":{"1min":0.872,"5min":1.3,"15min":1.3},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57699999999999996,"15min":0.53500000000000003},"min":{"1min":0.38,"5min":0.307,"15min":0.307},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.39700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":9520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":14458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":8356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":6840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":8774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5443584,"data_stored":5395029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121090048,"data_stored":176357670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":185782272,"data_stored":267123878,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":157282304,"data_stored":206358409,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":55132160,"data_stored":79501470,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":135057408,"data_stored":194014922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":72609792,"data_stored":97003943,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":117354496,"data_stored":159809986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:25:40.176 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:25:40.191 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:40.213 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:40.214 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:40.237 INFO:tasks.rados.rados.0.smithi076.stdout:1557: finishing write tid 1 to smithi07694878-17 2021-03-15T22:25:40.237 INFO:tasks.rados.rados.0.smithi076.stdout:1557: finishing write tid 2 to smithi07694878-17 2021-03-15T22:25:40.237 INFO:tasks.rados.rados.0.smithi076.stdout:1554: finishing copy_from to smithi07694878-46 2021-03-15T22:25:40.238 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 826 (ObjNum 496 snap 149 seq_num 496) dirty exists 2021-03-15T22:25:40.238 INFO:tasks.rados.rados.0.smithi076.stdout:1554: finishing copy_from racing read to smithi07694878-46 2021-03-15T22:25:40.238 INFO:tasks.rados.rados.0.smithi076.stdout:1557: finishing write tid 3 to smithi07694878-17 2021-03-15T22:25:40.238 INFO:tasks.rados.rados.0.smithi076.stdout:1557: finishing write tid 4 to smithi07694878-17 2021-03-15T22:25:40.238 INFO:tasks.rados.rados.0.smithi076.stdout:1557: finishing write tid 5 to smithi07694878-17 2021-03-15T22:25:40.239 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 315 (ObjNum 549 snap 175 seq_num 549) dirty exists 2021-03-15T22:25:40.239 INFO:tasks.rados.rados.0.smithi076.stdout:1557: left oid 17 (ObjNum 549 snap 175 seq_num 549) 2021-03-15T22:25:40.239 INFO:tasks.rados.rados.0.smithi076.stdout:1554: done (3 left) 2021-03-15T22:25:40.240 INFO:tasks.rados.rados.0.smithi076.stdout:1556: done (2 left) 2021-03-15T22:25:40.240 INFO:tasks.rados.rados.0.smithi076.stdout:1557: done (1 left) 2021-03-15T22:25:40.240 INFO:tasks.rados.rados.0.smithi076.stdout:1559: copy_from oid 26 from oid 35 current snap is 175 2021-03-15T22:25:40.240 INFO:tasks.rados.rados.0.smithi076.stdout:1560: read oid 28 snap -1 2021-03-15T22:25:40.240 INFO:tasks.rados.rados.0.smithi076.stdout:1560: expect (ObjNum 509 snap 152 seq_num 509) 2021-03-15T22:25:40.241 INFO:tasks.rados.rados.0.smithi076.stdout:1561: snap_remove snap 164 2021-03-15T22:25:40.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-15T22:25:40.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:40.339 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:40.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:40.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:40.443 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:40.465 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:40.467 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:40.506 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:40.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: cluster 2021-03-15T22:25:40.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 2021-03-15T22:25:40.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 2021 2021-03-15T22:25:40.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: - 2021-03-15T22:25:40.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 03 2021-03-15T22:25:40.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: - 2021-03-15T22:25:40.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 15 2021-03-15T22:25:40.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: T 2021-03-15T22:25:40.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 22 2021-03-15T22:25:40.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: : 2021-03-15T22:25:40.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 25 2021-03-15T22:25:40.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: : 2021-03-15T22:25:40.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 39 2021-03-15T22:25:40.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: . 2021-03-15T22:25:40.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 056220 2021-03-15T22:25:40.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: +0000 2021-03-15T22:25:40.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 2021-03-15T22:25:40.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: mgr.y 2021-03-15T22:25:40.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: ( 2021-03-15T22:25:40.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: mgr 2021-03-15T22:25:40.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: . 2021-03-15T22:25:40.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 14144 2021-03-15T22:25:40.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: ) 2021-03-15T22:25:40.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 622 2021-03-15T22:25:40.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: : 2021-03-15T22:25:40.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: cluster 2021-03-15T22:25:40.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 2021-03-15T22:25:40.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: [DBG] 2021-03-15T22:25:40.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 2021-03-15T22:25:40.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: pgmap v762: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 3 peering, 131 active+clean; 314 MiB data, 1006 MiB used, 625 GiB / 626 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 13 op/s; 99/1443 objects degraded (6.861%); 33/1443 objects misplaced (2.287%); 11 MiB/s, 38 objects/s recovering 2021-03-15T22:25:40.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 2021-03-15T22:25:40.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: cluster 2021-03-15T22:25:40.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 2021-03-15T22:25:40.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 2021 2021-03-15T22:25:40.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: - 2021-03-15T22:25:40.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 03 2021-03-15T22:25:40.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: - 2021-03-15T22:25:40.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 15 2021-03-15T22:25:40.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: T 2021-03-15T22:25:40.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 22 2021-03-15T22:25:40.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: : 2021-03-15T22:25:40.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 25 2021-03-15T22:25:40.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: : 2021-03-15T22:25:40.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 39 2021-03-15T22:25:40.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: . 2021-03-15T22:25:40.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 138020 2021-03-15T22:25:40.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: +0000 2021-03-15T22:25:40.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 2021-03-15T22:25:40.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: mon.a (mon.0) 1244 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2021-03-15T22:25:40.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 2021-03-15T22:25:40.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: audit 2021-03-15T 2021-03-15T22:25:40.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 22:25:39.370265+0000 mgr.y (mgr.14144) 623 : audit [DBG] 2021-03-15T22:25:40.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:25:40.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: audit 2021-03-15T22:25:39 2021-03-15T22:25:40.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: .720259+0000 2021-03-15T22:25:40.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: mgr.y (mgr.14144) 624 : audit [DBG] 2021-03-15T22:25:40.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:40.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: cluster 2021-03-15T22:25 2021-03-15T22:25:40.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: :40.101087+0000 mon.a (mon.0) 2021-03-15T22:25:40.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[56514]: 1245 : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2021-03-15T22:25:40.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: cluster 2021-03-15T22:25:40.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021-03-15T22:25:40.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021 2021-03-15T22:25:40.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: - 2021-03-15T22:25:40.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 03 2021-03-15T22:25:40.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: - 2021-03-15T22:25:40.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 15 2021-03-15T22:25:40.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: T 2021-03-15T22:25:40.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 22 2021-03-15T22:25:40.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: : 2021-03-15T22:25:40.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 25 2021-03-15T22:25:40.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: : 2021-03-15T22:25:40.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 39 2021-03-15T22:25:40.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: . 2021-03-15T22:25:40.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 056220 2021-03-15T22:25:40.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: +0000 2021-03-15T22:25:40.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021-03-15T22:25:40.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: mgr.y 2021-03-15T22:25:40.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: ( 2021-03-15T22:25:40.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: mgr 2021-03-15T22:25:40.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: . 2021-03-15T22:25:40.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 14144 2021-03-15T22:25:40.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: ) 2021-03-15T22:25:40.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 622 2021-03-15T22:25:40.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: : 2021-03-15T22:25:40.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: cluster 2021-03-15T22:25:40.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021-03-15T22:25:40.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: [DBG] 2021-03-15T22:25:40.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021-03-15T22:25:40.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: pgmap v762: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 3 peering, 131 active+clean; 314 MiB data, 1006 MiB used, 625 GiB / 626 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 13 op/s; 99/1443 objects degraded (6.861%); 33/1443 objects misplaced (2.287%); 11 MiB/s, 38 objects/s recovering 2021-03-15T22:25:40.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021-03-15T22:25:40.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: cluster 2021-03-15T22:25:40.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021-03-15T22:25:40.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021 2021-03-15T22:25:40.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: - 2021-03-15T22:25:40.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 03 2021-03-15T22:25:40.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: - 2021-03-15T22:25:40.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 15 2021-03-15T22:25:40.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: T 2021-03-15T22:25:40.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 22 2021-03-15T22:25:40.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: : 2021-03-15T22:25:40.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 25 2021-03-15T22:25:40.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: : 2021-03-15T22:25:40.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 39 2021-03-15T22:25:40.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: . 2021-03-15T22:25:40.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 138020 2021-03-15T22:25:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: +0000 2021-03-15T22:25:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021-03-15T22:25:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: mon.a 2021-03-15T22:25:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: ( 2021-03-15T22:25:40.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: mon 2021-03-15T22:25:40.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: .0) 1244 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2021-03-15T22:25:40.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021-03-15T22:25:40.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: audit 2021 2021-03-15T22:25:40.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: -03-15T22:25:39.370265+0000 mgr.y ( 2021-03-15T22:25:40.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: mgr.14144) 623 : audit [DBG] from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:25:40.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: audit 2021-03-15T22:25:40.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021-03-15T22:25: 2021-03-15T22:25:40.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 39.720259+0000 mgr.y (mgr.14144) 2021-03-15T22:25:40.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 624 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:40.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: cluster 2021-03-15T22:25:40.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: 2021-03-15T22:25:40 2021-03-15T22:25:40.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: .101087+0000 mon.a (mon.0) 1245 2021-03-15T22:25:40.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:40 smithi076 conmon[62156]: : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2021-03-15T22:25:40.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:40 smithi125 conmon[56739]: cluster 2021-03-15T22:25:39.056220+0000 mgr.y (mgr.14144) 622 : cluster [DBG] pgmap v762: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+recovering+undersized+remapped, 1 activating+undersized+degraded+remapped, 3 peering, 131 active+clean; 314 MiB data, 1006 MiB used, 625 GiB / 626 GiB avail; 9.6 MiB/s rd, 1.2 MiB/s wr, 13 op/s; 99/1443 objects degraded (6.861%); 33/1443 objects misplaced (2.287%); 11 MiB/s, 38 objects/s recovering 2021-03-15T22:25:40.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:40 smithi125 conmon[56739]: 2021-03-15T22:25:40.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:40 smithi125 conmon[56739]: cluster 2021-03-15T22:25:39.138020+0000 mon.a (mon.0) 1244 : cluster [DBG] osdmap e424: 8 total, 8 up, 7 in 2021-03-15T22:25:40.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:40 smithi125 conmon[56739]: audit 2021-03-15T22:25:39.370265+0000 mgr.y (mgr.14144) 623 : audit [DBG] from='client.15483 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:25:40.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:40 smithi125 conmon[56739]: audit 2021-03-15T22:25:39.720259+0000 mgr.y (mgr.14144) 624 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:40.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:40 smithi125 conmon[56739]: cluster 2021-03-15T22:25:40.101087+0000 mon.a (mon.0) 1245 : cluster [DBG] osdmap e425: 8 total, 8 up, 7 in 2021-03-15T22:25:40.556 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:25:40.568 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:40.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:40.593 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:40.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:40.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:40.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:40.719 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:40.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:40.821 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:40.844 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:40.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:40.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:40.970 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:40.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:41.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:41.099 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:41.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:41.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:41.142 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:25:41.163 INFO:tasks.rados.rados.0.smithi076.stdout:1561: done (3 left) 2021-03-15T22:25:41.163 INFO:tasks.rados.rados.0.smithi076.stdout:1562: copy_from oid 32 from oid 21 current snap is 175 2021-03-15T22:25:41.163 INFO:tasks.rados.rados.0.smithi076.stdout:1559: finishing copy_from to smithi07694878-26 2021-03-15T22:25:41.164 INFO:tasks.rados.rados.0.smithi076.stdout:1559: got expected ENOENT (src dne) 2021-03-15T22:25:41.164 INFO:tasks.rados.rados.0.smithi076.stdout:1559: finishing copy_from racing read to smithi07694878-26 2021-03-15T22:25:41.164 INFO:tasks.rados.rados.0.smithi076.stdout:1558: finishing copy_from to smithi07694878-14 2021-03-15T22:25:41.164 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 762 (ObjNum 392 snap 106 seq_num 392) dirty exists 2021-03-15T22:25:41.164 INFO:tasks.rados.rados.0.smithi076.stdout:1558: finishing copy_from racing read to smithi07694878-14 2021-03-15T22:25:41.165 INFO:tasks.rados.rados.0.smithi076.stdout:1560: expect (ObjNum 509 snap 152 seq_num 509) 2021-03-15T22:25:41.165 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:25:41.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:41.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:41.225 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:41.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:41.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:41 smithi076 conmon[56514]: cluster 2021-03-15T22:25:40.360390+0000 osd.3 (osd.3) 73 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:41.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:41 smithi076 conmon[56514]: cluster 2021-03-15T22:25:40.361288+0000 osd.3 (osd.3) 74 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:41.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:41 smithi076 conmon[56514]: audit 2021-03-15T22:25:40.527538+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? 172.21.15.76:0/3758341754' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:41.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:41 smithi076 conmon[62156]: cluster 2021-03-15T22:25:40.360390+0000 2021-03-15T22:25:41.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:41 smithi076 conmon[62156]: osd.3 (osd.3) 73 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:41.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:41 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:25:41.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:41 smithi076 conmon[62156]: 15T22:25:40.361288+0000 osd.3 (osd.3) 74 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:41.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:41 smithi076 conmon[62156]: 2021-03-15T22:25:41.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:41 smithi076 conmon[62156]: audit 2021-03-15T22:25:40 2021-03-15T22:25:41.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:41 smithi076 conmon[62156]: .527538+0000 mon.a (mon.0) 1246 : 2021-03-15T22:25:41.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:41 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/3758341754' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:41.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:41 smithi125 conmon[56739]: cluster 2021-03-15T22:25:40.360390+0000 osd.3 (osd.3) 73 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:41.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:41 smithi125 conmon[56739]: cluster 2021-03-15T22:25:40.361288+0000 osd.3 (osd.3) 74 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:41.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:41 smithi125 conmon[56739]: audit 2021-03-15T22:25:40.527538+0000 mon.a (mon.0) 1246 : audit [INF] from='client.? 172.21.15.76:0/3758341754' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:41.309 INFO:tasks.rados.rados.0.smithi076.stdout:1562: finishing copy_from to smithi07694878-32 2021-03-15T22:25:41.309 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 203 (ObjNum 469 snap 137 seq_num 469) dirty exists 2021-03-15T22:25:41.309 INFO:tasks.rados.rados.0.smithi076.stdout:1562: finishing copy_from racing read to smithi07694878-32 2021-03-15T22:25:41.309 INFO:tasks.rados.rados.0.smithi076.stdout:1558: done (3 left) 2021-03-15T22:25:41.310 INFO:tasks.rados.rados.0.smithi076.stdout:1559: done (2 left) 2021-03-15T22:25:41.310 INFO:tasks.rados.rados.0.smithi076.stdout:1560: done (1 left) 2021-03-15T22:25:41.310 INFO:tasks.rados.rados.0.smithi076.stdout:1562: done (0 left) 2021-03-15T22:25:41.311 INFO:tasks.rados.rados.0.smithi076.stdout:1563: read oid 23 snap -1 2021-03-15T22:25:41.311 INFO:tasks.rados.rados.0.smithi076.stdout:1563: expect (ObjNum 547 snap 174 seq_num 547) 2021-03-15T22:25:41.311 INFO:tasks.rados.rados.0.smithi076.stdout:1564: delete oid 27 current snap is 175 2021-03-15T22:25:41.314 INFO:tasks.rados.rados.0.smithi076.stdout:1564: done (1 left) 2021-03-15T22:25:41.314 INFO:tasks.rados.rados.0.smithi076.stdout:1565: snap_create 2021-03-15T22:25:41.315 INFO:tasks.rados.rados.0.smithi076.stdout:1563: expect (ObjNum 547 snap 174 seq_num 547) 2021-03-15T22:25:41.327 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:41.348 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:41.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:41.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:41.451 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:41.472 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:41.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:41.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:41.575 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:41.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:41.600 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:41.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:41.702 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:41.725 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:41.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:41.728 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:41.829 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:41.854 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:41.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:41.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:41.957 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:41.979 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:41.980 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:42.081 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:42.105 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:42.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:42.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:42.116 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:25:42.117 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:25:42.158 INFO:tasks.rados.rados.0.smithi076.stdout:1563: done (1 left) 2021-03-15T22:25:42.159 INFO:tasks.rados.rados.0.smithi076.stdout:1565: done (0 left) 2021-03-15T22:25:42.159 INFO:tasks.rados.rados.0.smithi076.stdout:1566: read oid 7 snap 175 2021-03-15T22:25:42.159 INFO:tasks.rados.rados.0.smithi076.stdout:1566: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:25:42.159 INFO:tasks.rados.rados.0.smithi076.stdout:1567: rollback oid 1 current snap is 176 2021-03-15T22:25:42.160 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 1 to 146 2021-03-15T22:25:42.160 INFO:tasks.rados.rados.0.smithi076.stdout:1568: read oid 23 snap -1 2021-03-15T22:25:42.160 INFO:tasks.rados.rados.0.smithi076.stdout:1568: expect (ObjNum 547 snap 174 seq_num 547) 2021-03-15T22:25:42.160 INFO:tasks.rados.rados.0.smithi076.stdout:1569: copy_from oid 45 from oid 43 current snap is 176 2021-03-15T22:25:42.160 INFO:tasks.rados.rados.0.smithi076.stdout:1570: read oid 42 snap -1 2021-03-15T22:25:42.161 INFO:tasks.rados.rados.0.smithi076.stdout:1570: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:25:42.161 INFO:tasks.rados.rados.0.smithi076.stdout:1571: delete oid 3 current snap is 176 2021-03-15T22:25:42.162 INFO:tasks.rados.rados.0.smithi076.stdout:1571: done (5 left) 2021-03-15T22:25:42.162 INFO:tasks.rados.rados.0.smithi076.stdout:1572: snap_remove snap 173 2021-03-15T22:25:42.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: cluster 2021-03-15T22:25: 2021-03-15T22:25:42.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: 41.056623+0000 mgr.y (mgr.14144) 625 : cluster [DBG] pgmap v765: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 135 active+clean; 321 MiB data, 935 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 15 op/s; 4/1455 objects degraded (0.275%); 29/1455 objects misplaced (1.993%); 14 MiB/s, 35 objects/s recovering 2021-03-15T22:25:42.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: cluster 2021-03-15T22:25 2021-03-15T22:25:42.166 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: :41.143626+0000 mon.a (mon.0) 1247 : cluster 2021-03-15T22:25:42.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: [WRN] Health check update: Degraded data redundancy: 4/1455 objects degraded (0.275%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:25:42.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:25:42.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: 15T22:25:41.144226+0000 mon.a (mon 2021-03-15T22:25:42.167 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: .0) 1248 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:25:42.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:25:42.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: 15T22:25:41.154406+0000 mon.a (mon 2021-03-15T22:25:42.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: .0) 1249 : audit [INF] from='client.? 172.21.15.76:0/3758341754' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:42.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:25:42.168 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: -15T22:25:41.154455+0000 mon.a ( 2021-03-15T22:25:42.169 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[62156]: mon.0) 1250 : cluster [DBG] osdmap e426: 8 total, 8 up, 7 in 2021-03-15T22:25:42.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: cluster 2021-03-15T22:25:42.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: 2021-03-15T22:25:42.169 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: 2021 2021-03-15T22:25:42.170 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: - 2021-03-15T22:25:42.170 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: 03 2021-03-15T22:25:42.170 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: -15T22:25:41.056623+0000 2021-03-15T22:25:42.170 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: mgr.y (mgr.14144) 625 : cluster [DBG] pgmap v765: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 135 active+clean; 321 MiB data, 935 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 15 op/s; 4/1455 objects degraded (0.275%); 29/1455 objects misplaced (1.993%); 14 MiB/s, 35 objects/s recovering 2021-03-15T22:25:42.171 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:25:42.171 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: -15T22:25:41.143626+0000 mon.a ( 2021-03-15T22:25:42.171 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: mon.0) 1247 : cluster [WRN] Health check update: Degraded data redundancy: 4/1455 objects degraded (0.275%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:25:42.171 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:25:42.171 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: -03-15T22:25:41.144226+0000 2021-03-15T22:25:42.172 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: mon.a (mon.0) 1248 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:25:42.172 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:25:42.172 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: 15T22:25:41.154406+0000 mon.a (mon 2021-03-15T22:25:42.172 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: .0) 1249 : audit [INF] from='client.? 172.21.15.76:0/3758341754' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:42.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:25:42.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: 25:41.154455+0000 mon.a (mon.0) 1250 : cluster [DBG] 2021-03-15T22:25:42.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:42 smithi076 conmon[56514]: osdmap e426: 8 total, 8 up, 7 in 2021-03-15T22:25:42.181 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:42.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:42.230 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:42.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:42.232 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:42.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:42 smithi125 conmon[56739]: cluster 2021-03-15T22:25:41.056623+0000 mgr.y (mgr.14144 2021-03-15T22:25:42.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:42 smithi125 conmon[56739]: ) 625 : cluster [DBG] pgmap v765: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 1 peering, 135 active+clean; 321 MiB data, 935 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 15 op/s; 4/1455 objects degraded (0.275%); 29/1455 objects misplaced (1.993%); 14 MiB/s, 35 objects/s recovering 2021-03-15T22:25:42.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:42 smithi125 conmon[56739]: cluster 2021-03-15T22:25:41.143626+0000 mon.a (mon.0) 1247 : cluster [WRN] Health check update: Degraded data redundancy: 4/1455 objects degraded (0.275%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:25:42.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:42 smithi125 conmon[56739]: cluster 2021-03-15T22:25:41.144226+0000 mon.a (mon.0) 1248 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:25:42.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:42 smithi125 conmon[56739]: audit 2021-03-15T22:25:41.154406+0000 mon.a (mon.0) 1249 : audit [INF] from='client.? 172.21.15.76:0/3758341754' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:42.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:42 smithi125 conmon[56739]: cluster 2021-03-15T22:25:41.154455+0000 mon.a (mon.0) 1250 : cluster [DBG] osdmap e426: 8 total, 8 up, 7 in 2021-03-15T22:25:42.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:42.356 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:42.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:42.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:42.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:42.480 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:42.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:42.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:42.583 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:42.607 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:42.609 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:42.710 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:42.732 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:42.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:42.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:42.809 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-15T22:25:42.822 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:25:42.825 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:25:42.834 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:42.873 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:42.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:42.976 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:42.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:43.000 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:43.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:43.091 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:25:43.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:43.123 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:43.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:43.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:43.173 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:43 smithi076 conmon[56514]: cluster 2021-03-15T22:25:42.155389+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2021-03-15T22:25:43.174 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:43 smithi076 conmon[56514]: audit 2021-03-15T22:25:42.806827+0000 mgr.y (mgr.14144) 626 : audit [DBG] from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:25:43.176 INFO:tasks.rados.rados.0.smithi076.stdout:1572: done (5 left) 2021-03-15T22:25:43.176 INFO:tasks.rados.rados.0.smithi076.stdout:1573: write oid 39 current snap is 176 2021-03-15T22:25:43.176 INFO:tasks.rados.rados.0.smithi076.stdout:1573: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2021-03-15T22:25:43.182 INFO:tasks.rados.rados.0.smithi076.stdout:1573: writing smithi07694878-39 from 477464 to 1192328 tid 1 2021-03-15T22:25:43.189 INFO:tasks.rados.rados.0.smithi076.stdout:1573: writing smithi07694878-39 from 1751825 to 2488350 tid 2 2021-03-15T22:25:43.194 INFO:tasks.rados.rados.0.smithi076.stdout:1573: writing smithi07694878-39 from 3225492 to 3277464 tid 3 2021-03-15T22:25:43.194 INFO:tasks.rados.rados.0.smithi076.stdout:1574: read oid 30 snap -1 2021-03-15T22:25:43.194 INFO:tasks.rados.rados.0.smithi076.stdout:1574: expect (ObjNum 445 snap 126 seq_num 445) 2021-03-15T22:25:43.194 INFO:tasks.rados.rados.0.smithi076.stdout:1575: copy_from oid 8 from oid 11 current snap is 176 2021-03-15T22:25:43.195 INFO:tasks.rados.rados.0.smithi076.stdout:1566: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:25:43.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:43.249 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:43.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:43.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:43.351 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:43.356 INFO:tasks.rados.rados.0.smithi076.stdout:1567: finishing rollback tid 0 to smithi07694878-1 2021-03-15T22:25:43.357 INFO:tasks.rados.rados.0.smithi076.stdout:1567: finishing rollback tid 1 to smithi07694878-1 2021-03-15T22:25:43.357 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 828 (ObjNum 1 snap 21896 seq_num 265139456) dirty dne 2021-03-15T22:25:43.357 INFO:tasks.rados.rados.0.smithi076.stdout:1568: expect (ObjNum 547 snap 174 seq_num 547) 2021-03-15T22:25:43.373 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:43.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:43.479 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:43.494 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:43 smithi125 conmon[56739]: cluster 2021-03-15T22:25:42.155389+0000 mon.a ( 2021-03-15T22:25:43.494 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:43 smithi125 conmon[56739]: mon.0) 1251 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2021-03-15T22:25:43.494 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:43 smithi125 conmon[56739]: audit 2021-03-15T22:25:42.806827+0000 mgr.y (mgr.14144) 626 : audit [DBG] from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:25:43.501 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:43.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:43.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:43.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:43 smithi076 conmon[62156]: cluster 2021-03-15T22:25:43.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:43 smithi076 conmon[62156]: 2021-03-15T22:25 2021-03-15T22:25:43.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:43 smithi076 conmon[62156]: :42.155389+0000 mon.a (mon.0) 1251 : cluster [DBG] osdmap e427: 8 total, 8 up, 7 in 2021-03-15T22:25:43.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:43 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:25:43.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:43 smithi076 conmon[62156]: 22:25:42.806827+0000 mgr.y (mgr.14144) 626 : audit [DBG] from='client.15495 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:25:43.597 INFO:tasks.rados.rados.0.smithi076.stdout:1570: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:25:43.604 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:43.626 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:43.627 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:43.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:43.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:43.753 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:43.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:43.757 INFO:tasks.rados.rados.0.smithi076.stdout:1569: finishing copy_from to smithi07694878-45 2021-03-15T22:25:43.757 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 762 (ObjNum 392 snap 106 seq_num 392) dirty exists 2021-03-15T22:25:43.757 INFO:tasks.rados.rados.0.smithi076.stdout:1569: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:25:43.758 INFO:tasks.rados.rados.0.smithi076.stdout:1566: done (7 left) 2021-03-15T22:25:43.758 INFO:tasks.rados.rados.0.smithi076.stdout:1567: done (6 left) 2021-03-15T22:25:43.758 INFO:tasks.rados.rados.0.smithi076.stdout:1568: done (5 left) 2021-03-15T22:25:43.758 INFO:tasks.rados.rados.0.smithi076.stdout:1569: done (4 left) 2021-03-15T22:25:43.758 INFO:tasks.rados.rados.0.smithi076.stdout:1570: done (3 left) 2021-03-15T22:25:43.759 INFO:tasks.rados.rados.0.smithi076.stdout:1576: read oid 40 snap -1 2021-03-15T22:25:43.759 INFO:tasks.rados.rados.0.smithi076.stdout:1576: expect deleted 2021-03-15T22:25:43.759 INFO:tasks.rados.rados.0.smithi076.stdout:1577: read oid 10 snap -1 2021-03-15T22:25:43.759 INFO:tasks.rados.rados.0.smithi076.stdout:1577: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:25:43.760 INFO:tasks.rados.rados.0.smithi076.stdout:1578: copy_from oid 28 from oid 17 current snap is 176 2021-03-15T22:25:43.760 INFO:tasks.rados.rados.0.smithi076.stdout:1574: expect (ObjNum 445 snap 126 seq_num 445) 2021-03-15T22:25:43.854 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:43.875 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:43.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:43.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:43.934 INFO:tasks.rados.rados.0.smithi076.stdout:1573: finishing write tid 1 to smithi07694878-39 2021-03-15T22:25:43.934 INFO:tasks.rados.rados.0.smithi076.stdout:1573: finishing write tid 2 to smithi07694878-39 2021-03-15T22:25:43.934 INFO:tasks.rados.rados.0.smithi076.stdout:1573: finishing write tid 3 to smithi07694878-39 2021-03-15T22:25:43.935 INFO:tasks.rados.rados.0.smithi076.stdout:1573: finishing write tid 4 to smithi07694878-39 2021-03-15T22:25:43.935 INFO:tasks.rados.rados.0.smithi076.stdout:1573: finishing write tid 5 to smithi07694878-39 2021-03-15T22:25:43.935 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 964 (ObjNum 552 snap 176 seq_num 552) dirty exists 2021-03-15T22:25:43.935 INFO:tasks.rados.rados.0.smithi076.stdout:1573: left oid 39 (ObjNum 552 snap 176 seq_num 552) 2021-03-15T22:25:43.936 INFO:tasks.rados.rados.0.smithi076.stdout:1575: finishing copy_from to smithi07694878-8 2021-03-15T22:25:43.936 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 421 (ObjNum 494 snap 146 seq_num 494) dirty exists 2021-03-15T22:25:43.936 INFO:tasks.rados.rados.0.smithi076.stdout:1575: finishing copy_from racing read to smithi07694878-8 2021-03-15T22:25:43.936 INFO:tasks.rados.rados.0.smithi076.stdout:1573: done (5 left) 2021-03-15T22:25:43.937 INFO:tasks.rados.rados.0.smithi076.stdout:1574: done (4 left) 2021-03-15T22:25:43.937 INFO:tasks.rados.rados.0.smithi076.stdout:1575: done (3 left) 2021-03-15T22:25:43.937 INFO:tasks.rados.rados.0.smithi076.stdout:1579: delete oid 25 current snap is 176 2021-03-15T22:25:43.937 INFO:tasks.rados.rados.0.smithi076.stdout:1577: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:25:43.978 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:44.000 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:44.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:44.002 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:44.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:44.125 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:44.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:44.127 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:44.142 INFO:tasks.rados.rados.0.smithi076.stdout:1578: finishing copy_from to smithi07694878-28 2021-03-15T22:25:44.142 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 315 (ObjNum 549 snap 175 seq_num 549) dirty exists 2021-03-15T22:25:44.142 INFO:tasks.rados.rados.0.smithi076.stdout:1578: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:25:44.142 INFO:tasks.rados.rados.0.smithi076.stdout:1576: done (3 left) 2021-03-15T22:25:44.143 INFO:tasks.rados.rados.0.smithi076.stdout:1577: done (2 left) 2021-03-15T22:25:44.143 INFO:tasks.rados.rados.0.smithi076.stdout:1578: done (1 left) 2021-03-15T22:25:44.143 INFO:tasks.rados.rados.0.smithi076.stdout:1579: done (0 left) 2021-03-15T22:25:44.143 INFO:tasks.rados.rados.0.smithi076.stdout:1580: write oid 48 current snap is 176 2021-03-15T22:25:44.143 INFO:tasks.rados.rados.0.smithi076.stdout:1580: seq_num 554 ranges {511078=465362,1752436=740194,3163985=147093} 2021-03-15T22:25:44.147 INFO:tasks.rados.rados.0.smithi076.stdout:1580: writing smithi07694878-48 from 511078 to 976440 tid 1 2021-03-15T22:25:44.156 INFO:tasks.rados.rados.0.smithi076.stdout:1580: writing smithi07694878-48 from 1752436 to 2492630 tid 2 2021-03-15T22:25:44.161 INFO:tasks.rados.rados.0.smithi076.stdout:1580: writing smithi07694878-48 from 3163985 to 3311078 tid 3 2021-03-15T22:25:44.161 INFO:tasks.rados.rados.0.smithi076.stdout:1581: write (excl) oid 49 current snap is 176 2021-03-15T22:25:44.162 INFO:tasks.rados.rados.0.smithi076.stdout:1580: finishing write tid 1 to smithi07694878-48 2021-03-15T22:25:44.162 INFO:tasks.rados.rados.0.smithi076.stdout:1581: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2021-03-15T22:25:44.168 INFO:tasks.rados.rados.0.smithi076.stdout:1581: writing smithi07694878-49 from 527885 to 1184849 tid 1 2021-03-15T22:25:44.175 INFO:tasks.rados.rados.0.smithi076.stdout:1581: writing smithi07694878-49 from 1927271 to 2411123 tid 2 2021-03-15T22:25:44.181 INFO:tasks.rados.rados.0.smithi076.stdout:1581: writing smithi07694878-49 from 3191408 to 3327885 tid 3 2021-03-15T22:25:44.182 INFO:tasks.rados.rados.0.smithi076.stdout:1580: finishing write tid 2 to smithi07694878-48 2021-03-15T22:25:44.182 INFO:tasks.rados.rados.0.smithi076.stdout:1580: finishing write tid 3 to smithi07694878-48 2021-03-15T22:25:44.182 INFO:tasks.rados.rados.0.smithi076.stdout:1580: finishing write tid 4 to smithi07694878-48 2021-03-15T22:25:44.182 INFO:tasks.rados.rados.0.smithi076.stdout:1580: finishing write tid 5 to smithi07694878-48 2021-03-15T22:25:44.183 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 766 (ObjNum 554 snap 176 seq_num 554) dirty exists 2021-03-15T22:25:44.183 INFO:tasks.rados.rados.0.smithi076.stdout:1580: left oid 48 (ObjNum 554 snap 176 seq_num 554) 2021-03-15T22:25:44.183 INFO:tasks.rados.rados.0.smithi076.stdout:1581: finishing write tid 1 to smithi07694878-49 2021-03-15T22:25:44.183 INFO:tasks.rados.rados.0.smithi076.stdout:1581: finishing write tid 2 to smithi07694878-49 2021-03-15T22:25:44.184 INFO:tasks.rados.rados.0.smithi076.stdout:1580: done (1 left) 2021-03-15T22:25:44.184 INFO:tasks.rados.rados.0.smithi076.stdout:1582: read oid 43 snap 132 2021-03-15T22:25:44.184 INFO:tasks.rados.rados.0.smithi076.stdout:1582: expect (ObjNum 392 snap 106 seq_num 392) 2021-03-15T22:25:44.185 INFO:tasks.rados.rados.0.smithi076.stdout:1583: read oid 38 snap 169 2021-03-15T22:25:44.185 INFO:tasks.rados.rados.0.smithi076.stdout:1583: expect deleted 2021-03-15T22:25:44.185 INFO:tasks.rados.rados.0.smithi076.stdout:1583: done (2 left) 2021-03-15T22:25:44.185 INFO:tasks.rados.rados.0.smithi076.stdout:1584: delete oid 42 current snap is 176 2021-03-15T22:25:44.186 INFO:tasks.rados.rados.0.smithi076.stdout:1582: expect (ObjNum 392 snap 106 seq_num 392) 2021-03-15T22:25:44.197 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":767,"stamp":"2021-03-15T22:25:42.153082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":353217124,"num_objects":490,"num_object_clones":125,"num_object_copies":1470,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":490,"num_whiteouts":13,"num_read":4559,"num_read_kb":3059188,"num_write":2773,"num_write_kb":840071,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1609,"num_bytes_recovered":460505716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6639,"ondisk_log_size":6639,"up":414,"acting":413,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":424,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":993260,"kb_used_data":829876,"kb_used_omap":0,"kb_used_meta":163328,"kb_avail":655222804,"statfs":{"total":671965249536,"available":670948151296,"internally_reserved":0,"allocated":849793024,"data_stored":1150046774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":167247872},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,5],"upper_bound":1024},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23930321,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-95,"num_objects_misplaced":-7,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":37,"num_read_kb":27735,"num_write":26,"num_write_kb":19598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":60992487,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.019102"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"388","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:41.036097+0000","last_change":"2021-03-15T22:25:41.036097+0000","last_active":"2021-03-15T22:25:41.036097+0000","last_peered":"2021-03-15T22:25:41.036097+0000","last_clean":"2021-03-15T22:25:41.036097+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:25:41.036097+0000","last_undegraded":"2021-03-15T22:25:41.036097+0000","last_fullsized":"2021-03-15T22:25:41.036097+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:33.728929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:33.728929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"88","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115394+0000","last_change":"2021-03-15T22:24:31.865035+0000","last_active":"2021-03-15T22:25:36.115394+0000","last_peered":"2021-03-15T22:25:36.115394+0000","last_clean":"2021-03-15T22:25:36.115394+0000","last_became_active":"2021-03-15T22:24:31.103578+0000","last_became_peered":"2021-03-15T22:24:31.103578+0000","last_unstale":"2021-03-15T22:25:36.115394+0000","last_undegraded":"2021-03-15T22:25:36.115394+0000","last_fullsized":"2021-03-15T22:25:36.115394+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:23:59.500680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:59.500680+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"420","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.113805+0000","last_change":"2021-03-15T22:25:40.007165+0000","last_active":"2021-03-15T22:25:40.113805+0000","last_peered":"2021-03-15T22:25:40.113805+0000","last_clean":"2021-03-15T22:25:40.113805+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:25:40.113805+0000","last_undegraded":"2021-03-15T22:25:40.113805+0000","last_fullsized":"2021-03-15T22:25:40.113805+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:26.743811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.743811+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"18","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.100487+0000","last_change":"2021-03-15T22:25:33.149020+0000","last_active":"2021-03-15T22:25:40.100487+0000","last_peered":"2021-03-15T22:25:40.100487+0000","last_clean":"2021-03-15T22:25:40.100487+0000","last_became_active":"2021-03-15T22:25:33.148710+0000","last_became_peered":"2021-03-15T22:25:33.148710+0000","last_unstale":"2021-03-15T22:25:40.100487+0000","last_undegraded":"2021-03-15T22:25:40.100487+0000","last_fullsized":"2021-03-15T22:25:40.100487+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:06.177754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.177754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"18","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.103138+0000","last_change":"2021-03-15T22:25:33.147017+0000","last_active":"2021-03-15T22:25:40.103138+0000","last_peered":"2021-03-15T22:25:40.103138+0000","last_clean":"2021-03-15T22:25:40.103138+0000","last_became_active":"2021-03-15T22:25:33.146697+0000","last_became_peered":"2021-03-15T22:25:33.146697+0000","last_unstale":"2021-03-15T22:25:40.103138+0000","last_undegraded":"2021-03-15T22:25:40.103138+0000","last_fullsized":"2021-03-15T22:25:40.103138+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:15.333746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.333746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"44","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.100869+0000","last_change":"2021-03-15T22:25:33.175542+0000","last_active":"2021-03-15T22:25:40.100869+0000","last_peered":"2021-03-15T22:25:40.100869+0000","last_clean":"2021-03-15T22:25:40.100869+0000","last_became_active":"2021-03-15T22:25:33.137906+0000","last_became_peered":"2021-03-15T22:25:33.137906+0000","last_unstale":"2021-03-15T22:25:40.100869+0000","last_undegraded":"2021-03-15T22:25:40.100869+0000","last_fullsized":"2021-03-15T22:25:40.100869+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:23:42.144821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.144821+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":21,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"77","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.114898+0000","last_change":"2021-03-15T22:24:31.109745+0000","last_active":"2021-03-15T22:25:36.114898+0000","last_peered":"2021-03-15T22:25:36.114898+0000","last_clean":"2021-03-15T22:25:36.114898+0000","last_became_active":"2021-03-15T22:24:31.108652+0000","last_became_peered":"2021-03-15T22:24:31.108652+0000","last_unstale":"2021-03-15T22:25:36.114898+0000","last_undegraded":"2021-03-15T22:25:36.114898+0000","last_fullsized":"2021-03-15T22:25:36.114898+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:35.677439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:35.677439+0000","last_clean_scrub_stamp":"2021-03-15T22:23:35.677439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"387","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116452+0000","last_change":"2021-03-15T22:25:16.598073+0000","last_active":"2021-03-15T22:25:36.116452+0000","last_peered":"2021-03-15T22:25:36.116452+0000","last_clean":"2021-03-15T22:25:36.116452+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:25:36.116452+0000","last_undegraded":"2021-03-15T22:25:36.116452+0000","last_fullsized":"2021-03-15T22:25:36.116452+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.597998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.597998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"495","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114292+0000","last_change":"2021-03-15T22:24:31.776304+0000","last_active":"2021-03-15T22:25:40.114292+0000","last_peered":"2021-03-15T22:25:40.114292+0000","last_clean":"2021-03-15T22:25:40.114292+0000","last_became_active":"2021-03-15T22:24:31.083373+0000","last_became_peered":"2021-03-15T22:24:31.083373+0000","last_unstale":"2021-03-15T22:25:40.114292+0000","last_undegraded":"2021-03-15T22:25:40.114292+0000","last_fullsized":"2021-03-15T22:25:40.114292+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:24:10.979322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:10.979322+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"96","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115119+0000","last_change":"2021-03-15T22:25:33.143128+0000","last_active":"2021-03-15T22:25:36.115119+0000","last_peered":"2021-03-15T22:25:36.115119+0000","last_clean":"2021-03-15T22:25:36.115119+0000","last_became_active":"2021-03-15T22:25:33.142493+0000","last_became_peered":"2021-03-15T22:25:33.142493+0000","last_unstale":"2021-03-15T22:25:36.115119+0000","last_undegraded":"2021-03-15T22:25:36.115119+0000","last_fullsized":"2021-03-15T22:25:36.115119+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.674895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"387","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115517+0000","last_change":"2021-03-15T22:25:05.682093+0000","last_active":"2021-03-15T22:25:36.115517+0000","last_peered":"2021-03-15T22:25:36.115517+0000","last_clean":"2021-03-15T22:25:36.115517+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:25:36.115517+0000","last_undegraded":"2021-03-15T22:25:36.115517+0000","last_fullsized":"2021-03-15T22:25:36.115517+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.682031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.682031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"121'9","reported_seq":"461","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114239+0000","last_change":"2021-03-15T22:24:37.902248+0000","last_active":"2021-03-15T22:25:40.114239+0000","last_peered":"2021-03-15T22:25:40.114239+0000","last_clean":"2021-03-15T22:25:40.114239+0000","last_became_active":"2021-03-15T22:22:56.055083+0000","last_became_peered":"2021-03-15T22:22:56.055083+0000","last_unstale":"2021-03-15T22:25:40.114239+0000","last_undegraded":"2021-03-15T22:25:40.114239+0000","last_fullsized":"2021-03-15T22:25:40.114239+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'9","last_scrub_stamp":"2021-03-15T22:24:37.902166+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.902166+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"388","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114732+0000","last_change":"2021-03-15T22:24:30.865676+0000","last_active":"2021-03-15T22:25:40.114732+0000","last_peered":"2021-03-15T22:25:40.114732+0000","last_clean":"2021-03-15T22:25:40.114732+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:25:40.114732+0000","last_undegraded":"2021-03-15T22:25:40.114732+0000","last_fullsized":"2021-03-15T22:25:40.114732+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.865609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.865609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"136","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114188+0000","last_change":"2021-03-15T22:24:55.103109+0000","last_active":"2021-03-15T22:25:40.114188+0000","last_peered":"2021-03-15T22:25:40.114188+0000","last_clean":"2021-03-15T22:25:40.114188+0000","last_became_active":"2021-03-15T22:24:55.090536+0000","last_became_peered":"2021-03-15T22:24:55.090536+0000","last_unstale":"2021-03-15T22:25:40.114188+0000","last_undegraded":"2021-03-15T22:25:40.114188+0000","last_fullsized":"2021-03-15T22:25:40.114188+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:24:28.507563+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.507563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"488","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164754+0000","last_change":"2021-03-15T22:25:33.740318+0000","last_active":"2021-03-15T22:25:38.164754+0000","last_peered":"2021-03-15T22:25:38.164754+0000","last_clean":"2021-03-15T22:25:38.164754+0000","last_became_active":"2021-03-15T22:25:33.134238+0000","last_became_peered":"2021-03-15T22:25:33.134238+0000","last_unstale":"2021-03-15T22:25:38.164754+0000","last_undegraded":"2021-03-15T22:25:38.164754+0000","last_fullsized":"2021-03-15T22:25:38.164754+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:03.482307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:03.482307+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"216","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.103834+0000","last_change":"2021-03-15T22:25:14.117883+0000","last_active":"2021-03-15T22:25:40.103834+0000","last_peered":"2021-03-15T22:25:40.103834+0000","last_clean":"2021-03-15T22:25:40.103834+0000","last_became_active":"2021-03-15T22:25:14.117558+0000","last_became_peered":"2021-03-15T22:25:14.117558+0000","last_unstale":"2021-03-15T22:25:40.103834+0000","last_undegraded":"2021-03-15T22:25:40.103834+0000","last_fullsized":"2021-03-15T22:25:40.103834+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:00.480944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:00.480944+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"397","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114375+0000","last_change":"2021-03-15T22:25:11.028502+0000","last_active":"2021-03-15T22:25:40.114375+0000","last_peered":"2021-03-15T22:25:40.114375+0000","last_clean":"2021-03-15T22:25:40.114375+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:25:40.114375+0000","last_undegraded":"2021-03-15T22:25:40.114375+0000","last_fullsized":"2021-03-15T22:25:40.114375+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.028422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.028422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"38","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.102634+0000","last_change":"2021-03-15T22:25:33.169024+0000","last_active":"2021-03-15T22:25:40.102634+0000","last_peered":"2021-03-15T22:25:40.102634+0000","last_clean":"2021-03-15T22:25:40.102634+0000","last_became_active":"2021-03-15T22:25:33.134781+0000","last_became_peered":"2021-03-15T22:25:33.134781+0000","last_unstale":"2021-03-15T22:25:40.102634+0000","last_undegraded":"2021-03-15T22:25:40.102634+0000","last_fullsized":"2021-03-15T22:25:40.102634+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:23:32.049047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:32.049047+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":36,"num_read_kb":19,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"379","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140464+0000","last_change":"2021-03-15T22:23:52.272270+0000","last_active":"2021-03-15T22:25:38.140464+0000","last_peered":"2021-03-15T22:25:38.140464+0000","last_clean":"2021-03-15T22:25:38.140464+0000","last_became_active":"2021-03-15T22:23:52.271925+0000","last_became_peered":"2021-03-15T22:23:52.271925+0000","last_unstale":"2021-03-15T22:25:38.140464+0000","last_undegraded":"2021-03-15T22:25:38.140464+0000","last_fullsized":"2021-03-15T22:25:38.140464+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:34.782225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:34.782225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"408","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.108848+0000","last_change":"2021-03-15T22:23:40.199174+0000","last_active":"2021-03-15T22:25:36.108848+0000","last_peered":"2021-03-15T22:25:36.108848+0000","last_clean":"2021-03-15T22:25:36.108848+0000","last_became_active":"2021-03-15T22:22:56.062833+0000","last_became_peered":"2021-03-15T22:22:56.062833+0000","last_unstale":"2021-03-15T22:25:36.108848+0000","last_undegraded":"2021-03-15T22:25:36.108848+0000","last_fullsized":"2021-03-15T22:25:36.108848+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:40.199095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:40.199095+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"206","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.141350+0000","last_change":"2021-03-15T22:25:33.752847+0000","last_active":"2021-03-15T22:25:38.141350+0000","last_peered":"2021-03-15T22:25:38.141350+0000","last_clean":"2021-03-15T22:25:38.141350+0000","last_became_active":"2021-03-15T22:25:33.140078+0000","last_became_peered":"2021-03-15T22:25:33.140078+0000","last_unstale":"2021-03-15T22:25:38.141350+0000","last_undegraded":"2021-03-15T22:25:38.141350+0000","last_fullsized":"2021-03-15T22:25:38.141350+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:25:08.192165+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:25:08.192165+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"384","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112652+0000","last_change":"2021-03-15T22:23:31.003298+0000","last_active":"2021-03-15T22:25:36.112652+0000","last_peered":"2021-03-15T22:25:36.112652+0000","last_clean":"2021-03-15T22:25:36.112652+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:25:36.112652+0000","last_undegraded":"2021-03-15T22:25:36.112652+0000","last_fullsized":"2021-03-15T22:25:36.112652+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:31.003234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:31.003234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"18","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.100307+0000","last_change":"2021-03-15T22:25:33.133625+0000","last_active":"2021-03-15T22:25:40.100307+0000","last_peered":"2021-03-15T22:25:40.100307+0000","last_clean":"2021-03-15T22:25:40.100307+0000","last_became_active":"2021-03-15T22:25:33.133418+0000","last_became_peered":"2021-03-15T22:25:33.133418+0000","last_unstale":"2021-03-15T22:25:40.100307+0000","last_undegraded":"2021-03-15T22:25:40.100307+0000","last_fullsized":"2021-03-15T22:25:40.100307+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:29.506406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.506406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"421","reported_epoch":"424","state":"active+clean","last_fresh":"2021-03-15T22:25:39.143989+0000","last_change":"2021-03-15T22:24:41.483906+0000","last_active":"2021-03-15T22:25:39.143989+0000","last_peered":"2021-03-15T22:25:39.143989+0000","last_clean":"2021-03-15T22:25:39.143989+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:25:39.143989+0000","last_undegraded":"2021-03-15T22:25:39.143989+0000","last_fullsized":"2021-03-15T22:25:39.143989+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:24:41.483849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:41.483849+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"384","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.139928+0000","last_change":"2021-03-15T22:25:09.177600+0000","last_active":"2021-03-15T22:25:38.139928+0000","last_peered":"2021-03-15T22:25:38.139928+0000","last_clean":"2021-03-15T22:25:38.139928+0000","last_became_active":"2021-03-15T22:23:52.260757+0000","last_became_peered":"2021-03-15T22:23:52.260757+0000","last_unstale":"2021-03-15T22:25:38.139928+0000","last_undegraded":"2021-03-15T22:25:38.139928+0000","last_fullsized":"2021-03-15T22:25:38.139928+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:09.177521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.177521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"395","reported_epoch":"424","state":"active+clean","last_fresh":"2021-03-15T22:25:39.142189+0000","last_change":"2021-03-15T22:25:02.452304+0000","last_active":"2021-03-15T22:25:39.142189+0000","last_peered":"2021-03-15T22:25:39.142189+0000","last_clean":"2021-03-15T22:25:39.142189+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:25:39.142189+0000","last_undegraded":"2021-03-15T22:25:39.142189+0000","last_fullsized":"2021-03-15T22:25:39.142189+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.452236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.452236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"219","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115474+0000","last_change":"2021-03-15T22:25:33.147111+0000","last_active":"2021-03-15T22:25:36.115474+0000","last_peered":"2021-03-15T22:25:36.115474+0000","last_clean":"2021-03-15T22:25:36.115474+0000","last_became_active":"2021-03-15T22:25:33.146680+0000","last_became_peered":"2021-03-15T22:25:33.146680+0000","last_unstale":"2021-03-15T22:25:36.115474+0000","last_undegraded":"2021-03-15T22:25:36.115474+0000","last_fullsized":"2021-03-15T22:25:36.115474+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:57.444598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:23:57.444598+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"115","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.100854+0000","last_change":"2021-03-15T22:25:15.835780+0000","last_active":"2021-03-15T22:25:40.100854+0000","last_peered":"2021-03-15T22:25:40.100854+0000","last_clean":"2021-03-15T22:25:40.100854+0000","last_became_active":"2021-03-15T22:24:23.142746+0000","last_became_peered":"2021-03-15T22:24:23.142746+0000","last_unstale":"2021-03-15T22:25:40.100854+0000","last_undegraded":"2021-03-15T22:25:40.100854+0000","last_fullsized":"2021-03-15T22:25:40.100854+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:25:15.835675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.835675+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":11,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"18","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.102512+0000","last_change":"2021-03-15T22:25:33.148415+0000","last_active":"2021-03-15T22:25:40.102512+0000","last_peered":"2021-03-15T22:25:40.102512+0000","last_clean":"2021-03-15T22:25:40.102512+0000","last_became_active":"2021-03-15T22:25:33.148083+0000","last_became_peered":"2021-03-15T22:25:33.148083+0000","last_unstale":"2021-03-15T22:25:40.102512+0000","last_undegraded":"2021-03-15T22:25:40.102512+0000","last_fullsized":"2021-03-15T22:25:40.102512+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:42.525975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:40.670674+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.525975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"376","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140237+0000","last_change":"2021-03-15T22:23:52.263156+0000","last_active":"2021-03-15T22:25:38.140237+0000","last_peered":"2021-03-15T22:25:38.140237+0000","last_clean":"2021-03-15T22:25:38.140237+0000","last_became_active":"2021-03-15T22:23:52.262741+0000","last_became_peered":"2021-03-15T22:23:52.262741+0000","last_unstale":"2021-03-15T22:25:38.140237+0000","last_undegraded":"2021-03-15T22:25:38.140237+0000","last_fullsized":"2021-03-15T22:25:38.140237+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:41.710658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.710658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"382","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116037+0000","last_change":"2021-03-15T22:23:30.005865+0000","last_active":"2021-03-15T22:25:36.116037+0000","last_peered":"2021-03-15T22:25:36.116037+0000","last_clean":"2021-03-15T22:25:36.116037+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:25:36.116037+0000","last_undegraded":"2021-03-15T22:25:36.116037+0000","last_fullsized":"2021-03-15T22:25:36.116037+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:30.005791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:30.005791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"90'22","reported_seq":"203","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.101114+0000","last_change":"2021-03-15T22:25:16.834493+0000","last_active":"2021-03-15T22:25:40.101114+0000","last_peered":"2021-03-15T22:25:40.101114+0000","last_clean":"2021-03-15T22:25:40.101114+0000","last_became_active":"2021-03-15T22:22:56.056175+0000","last_became_peered":"2021-03-15T22:22:56.056175+0000","last_unstale":"2021-03-15T22:25:40.101114+0000","last_undegraded":"2021-03-15T22:25:40.101114+0000","last_fullsized":"2021-03-15T22:25:40.101114+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:25:16.834409+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.834409+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"81","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116578+0000","last_change":"2021-03-15T22:24:41.659524+0000","last_active":"2021-03-15T22:25:36.116578+0000","last_peered":"2021-03-15T22:25:36.116578+0000","last_clean":"2021-03-15T22:25:36.116578+0000","last_became_active":"2021-03-15T22:24:31.105380+0000","last_became_peered":"2021-03-15T22:24:31.105380+0000","last_unstale":"2021-03-15T22:25:36.116578+0000","last_undegraded":"2021-03-15T22:25:36.116578+0000","last_fullsized":"2021-03-15T22:25:36.116578+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_clean_scrub_stamp":"2021-03-15T22:24:41.659450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"203","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.102423+0000","last_change":"2021-03-15T22:25:14.115163+0000","last_active":"2021-03-15T22:25:40.102423+0000","last_peered":"2021-03-15T22:25:40.102423+0000","last_clean":"2021-03-15T22:25:40.102423+0000","last_became_active":"2021-03-15T22:25:14.114508+0000","last_became_peered":"2021-03-15T22:25:14.114508+0000","last_unstale":"2021-03-15T22:25:40.102423+0000","last_undegraded":"2021-03-15T22:25:40.102423+0000","last_fullsized":"2021-03-15T22:25:40.102423+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.007316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:04.613000+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.007316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"398","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140894+0000","last_change":"2021-03-15T22:23:52.266674+0000","last_active":"2021-03-15T22:25:38.140894+0000","last_peered":"2021-03-15T22:25:38.140894+0000","last_clean":"2021-03-15T22:25:38.140894+0000","last_became_active":"2021-03-15T22:23:52.266011+0000","last_became_peered":"2021-03-15T22:23:52.266011+0000","last_unstale":"2021-03-15T22:25:38.140894+0000","last_undegraded":"2021-03-15T22:25:38.140894+0000","last_fullsized":"2021-03-15T22:25:38.140894+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.449472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"73","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112927+0000","last_change":"2021-03-15T22:24:56.824808+0000","last_active":"2021-03-15T22:25:36.112927+0000","last_peered":"2021-03-15T22:25:36.112927+0000","last_clean":"2021-03-15T22:25:36.112927+0000","last_became_active":"2021-03-15T22:24:55.097354+0000","last_became_peered":"2021-03-15T22:24:55.097354+0000","last_unstale":"2021-03-15T22:25:36.112927+0000","last_undegraded":"2021-03-15T22:25:36.112927+0000","last_fullsized":"2021-03-15T22:25:36.112927+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:24:38.433166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:38.433166+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"18","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.100776+0000","last_change":"2021-03-15T22:25:33.139212+0000","last_active":"2021-03-15T22:25:40.100776+0000","last_peered":"2021-03-15T22:25:40.100776+0000","last_clean":"2021-03-15T22:25:40.100776+0000","last_became_active":"2021-03-15T22:25:33.138883+0000","last_became_peered":"2021-03-15T22:25:33.138883+0000","last_unstale":"2021-03-15T22:25:40.100776+0000","last_undegraded":"2021-03-15T22:25:40.100776+0000","last_fullsized":"2021-03-15T22:25:40.100776+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:33.717517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:33.717517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"404","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112753+0000","last_change":"2021-03-15T22:25:33.129883+0000","last_active":"2021-03-15T22:25:36.112753+0000","last_peered":"2021-03-15T22:25:36.112753+0000","last_clean":"2021-03-15T22:25:36.112753+0000","last_became_active":"2021-03-15T22:25:33.129553+0000","last_became_peered":"2021-03-15T22:25:33.129553+0000","last_unstale":"2021-03-15T22:25:36.112753+0000","last_undegraded":"2021-03-15T22:25:36.112753+0000","last_fullsized":"2021-03-15T22:25:36.112753+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.794432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"111","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164615+0000","last_change":"2021-03-15T22:24:12.017160+0000","last_active":"2021-03-15T22:25:38.164615+0000","last_peered":"2021-03-15T22:25:38.164615+0000","last_clean":"2021-03-15T22:25:38.164615+0000","last_became_active":"2021-03-15T22:24:03.086278+0000","last_became_peered":"2021-03-15T22:24:03.086278+0000","last_unstale":"2021-03-15T22:25:38.164615+0000","last_undegraded":"2021-03-15T22:25:38.164615+0000","last_fullsized":"2021-03-15T22:25:38.164615+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:12.017097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:12.017097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"417","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114147+0000","last_change":"2021-03-15T22:23:29.709766+0000","last_active":"2021-03-15T22:25:40.114147+0000","last_peered":"2021-03-15T22:25:40.114147+0000","last_clean":"2021-03-15T22:25:40.114147+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:25:40.114147+0000","last_undegraded":"2021-03-15T22:25:40.114147+0000","last_fullsized":"2021-03-15T22:25:40.114147+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:29.709674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.709674+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"217","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.141254+0000","last_change":"2021-03-15T22:25:33.144478+0000","last_active":"2021-03-15T22:25:38.141254+0000","last_peered":"2021-03-15T22:25:38.141254+0000","last_clean":"2021-03-15T22:25:38.141254+0000","last_became_active":"2021-03-15T22:25:33.143832+0000","last_became_peered":"2021-03-15T22:25:33.143832+0000","last_unstale":"2021-03-15T22:25:38.141254+0000","last_undegraded":"2021-03-15T22:25:38.141254+0000","last_fullsized":"2021-03-15T22:25:38.141254+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.147414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.147414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"356'3","reported_seq":"485","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:37.085753+0000","last_change":"2021-03-15T22:24:31.129210+0000","last_active":"2021-03-15T22:25:37.085753+0000","last_peered":"2021-03-15T22:25:37.085753+0000","last_clean":"2021-03-15T22:25:37.085753+0000","last_became_active":"2021-03-15T22:24:31.093479+0000","last_became_peered":"2021-03-15T22:24:31.093479+0000","last_unstale":"2021-03-15T22:25:37.085753+0000","last_undegraded":"2021-03-15T22:25:37.085753+0000","last_fullsized":"2021-03-15T22:25:37.085753+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:24:28.700659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.700659+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"385","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140825+0000","last_change":"2021-03-15T22:25:01.148183+0000","last_active":"2021-03-15T22:25:38.140825+0000","last_peered":"2021-03-15T22:25:38.140825+0000","last_clean":"2021-03-15T22:25:38.140825+0000","last_became_active":"2021-03-15T22:23:52.261234+0000","last_became_peered":"2021-03-15T22:23:52.261234+0000","last_unstale":"2021-03-15T22:25:38.140825+0000","last_undegraded":"2021-03-15T22:25:38.140825+0000","last_fullsized":"2021-03-15T22:25:38.140825+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:01.148105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:01.148105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"202","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.102218+0000","last_change":"2021-03-15T22:24:27.166757+0000","last_active":"2021-03-15T22:25:40.102218+0000","last_peered":"2021-03-15T22:25:40.102218+0000","last_clean":"2021-03-15T22:25:40.102218+0000","last_became_active":"2021-03-15T22:22:56.064081+0000","last_became_peered":"2021-03-15T22:22:56.064081+0000","last_unstale":"2021-03-15T22:25:40.102218+0000","last_undegraded":"2021-03-15T22:25:40.102218+0000","last_fullsized":"2021-03-15T22:25:40.102218+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:27.166678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.166678+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":13,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"386","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114889+0000","last_change":"2021-03-15T22:25:07.992897+0000","last_active":"2021-03-15T22:25:40.114889+0000","last_peered":"2021-03-15T22:25:40.114889+0000","last_clean":"2021-03-15T22:25:40.114889+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:25:40.114889+0000","last_undegraded":"2021-03-15T22:25:40.114889+0000","last_fullsized":"2021-03-15T22:25:40.114889+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:07.992815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.992815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"421'7","reported_seq":"23","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.102072+0000","last_change":"2021-03-15T22:25:33.172230+0000","last_active":"2021-03-15T22:25:40.102072+0000","last_peered":"2021-03-15T22:25:40.102072+0000","last_clean":"2021-03-15T22:25:40.102072+0000","last_became_active":"2021-03-15T22:25:33.147484+0000","last_became_peered":"2021-03-15T22:25:33.147484+0000","last_unstale":"2021-03-15T22:25:40.102072+0000","last_undegraded":"2021-03-15T22:25:40.102072+0000","last_fullsized":"2021-03-15T22:25:40.102072+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'6","last_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.098511+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"223","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140013+0000","last_change":"2021-03-15T22:25:33.141774+0000","last_active":"2021-03-15T22:25:38.140013+0000","last_peered":"2021-03-15T22:25:38.140013+0000","last_clean":"2021-03-15T22:25:38.140013+0000","last_became_active":"2021-03-15T22:25:33.141466+0000","last_became_peered":"2021-03-15T22:25:33.141466+0000","last_unstale":"2021-03-15T22:25:38.140013+0000","last_undegraded":"2021-03-15T22:25:38.140013+0000","last_fullsized":"2021-03-15T22:25:38.140013+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:23:39.695671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"51","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116249+0000","last_change":"2021-03-15T22:25:14.129885+0000","last_active":"2021-03-15T22:25:36.116249+0000","last_peered":"2021-03-15T22:25:36.116249+0000","last_clean":"2021-03-15T22:25:36.116249+0000","last_became_active":"2021-03-15T22:25:14.114060+0000","last_became_peered":"2021-03-15T22:25:14.114060+0000","last_unstale":"2021-03-15T22:25:36.116249+0000","last_undegraded":"2021-03-15T22:25:36.116249+0000","last_fullsized":"2021-03-15T22:25:36.116249+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:25:07.163128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.163128+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"389","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164933+0000","last_change":"2021-03-15T22:25:00.614351+0000","last_active":"2021-03-15T22:25:38.164933+0000","last_peered":"2021-03-15T22:25:38.164933+0000","last_clean":"2021-03-15T22:25:38.164933+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:25:38.164933+0000","last_undegraded":"2021-03-15T22:25:38.164933+0000","last_fullsized":"2021-03-15T22:25:38.164933+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.614288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.614288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"387","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140499+0000","last_change":"2021-03-15T22:25:10.130486+0000","last_active":"2021-03-15T22:25:38.140499+0000","last_peered":"2021-03-15T22:25:38.140499+0000","last_clean":"2021-03-15T22:25:38.140499+0000","last_became_active":"2021-03-15T22:23:52.263426+0000","last_became_peered":"2021-03-15T22:23:52.263426+0000","last_unstale":"2021-03-15T22:25:38.140499+0000","last_undegraded":"2021-03-15T22:25:38.140499+0000","last_fullsized":"2021-03-15T22:25:38.140499+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:10.130422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.130422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"42","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.102977+0000","last_change":"2021-03-15T22:25:35.064301+0000","last_active":"2021-03-15T22:25:40.102977+0000","last_peered":"2021-03-15T22:25:40.102977+0000","last_clean":"2021-03-15T22:25:40.102977+0000","last_became_active":"2021-03-15T22:25:33.151159+0000","last_became_peered":"2021-03-15T22:25:33.151159+0000","last_unstale":"2021-03-15T22:25:40.102977+0000","last_undegraded":"2021-03-15T22:25:40.102977+0000","last_fullsized":"2021-03-15T22:25:40.102977+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:26.478799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.478799+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3618,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"379","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140818+0000","last_change":"2021-03-15T22:23:52.266876+0000","last_active":"2021-03-15T22:25:38.140818+0000","last_peered":"2021-03-15T22:25:38.140818+0000","last_clean":"2021-03-15T22:25:38.140818+0000","last_became_active":"2021-03-15T22:23:52.266133+0000","last_became_peered":"2021-03-15T22:23:52.266133+0000","last_unstale":"2021-03-15T22:25:38.140818+0000","last_undegraded":"2021-03-15T22:25:38.140818+0000","last_fullsized":"2021-03-15T22:25:38.140818+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:28.808873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:28.808873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"435","reported_epoch":"424","state":"active+clean","last_fresh":"2021-03-15T22:25:39.141749+0000","last_change":"2021-03-15T22:25:33.150477+0000","last_active":"2021-03-15T22:25:39.141749+0000","last_peered":"2021-03-15T22:25:39.141749+0000","last_clean":"2021-03-15T22:25:39.141749+0000","last_became_active":"2021-03-15T22:25:33.150147+0000","last_became_peered":"2021-03-15T22:25:33.150147+0000","last_unstale":"2021-03-15T22:25:39.141749+0000","last_undegraded":"2021-03-15T22:25:39.141749+0000","last_fullsized":"2021-03-15T22:25:39.141749+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:37.475218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.475218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"398'11","reported_seq":"28","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.103418+0000","last_change":"2021-03-15T22:25:33.864526+0000","last_active":"2021-03-15T22:25:40.103418+0000","last_peered":"2021-03-15T22:25:40.103418+0000","last_clean":"2021-03-15T22:25:40.103418+0000","last_became_active":"2021-03-15T22:25:33.152373+0000","last_became_peered":"2021-03-15T22:25:33.152373+0000","last_unstale":"2021-03-15T22:25:40.103418+0000","last_undegraded":"2021-03-15T22:25:40.103418+0000","last_fullsized":"2021-03-15T22:25:40.103418+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'11","last_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:25:13.181225+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"418'312","reported_seq":"1497","reported_epoch":"422","state":"active+clean","last_fresh":"2021-03-15T22:25:37.112878+0000","last_change":"2021-03-15T22:25:37.113594+0000","last_active":"2021-03-15T22:25:37.112878+0000","last_peered":"2021-03-15T22:25:37.112878+0000","last_clean":"2021-03-15T22:25:37.112878+0000","last_became_active":"2021-03-15T22:25:17.113174+0000","last_became_peered":"2021-03-15T22:25:17.113174+0000","last_unstale":"2021-03-15T22:25:37.112878+0000","last_undegraded":"2021-03-15T22:25:37.112878+0000","last_fullsized":"2021-03-15T22:25:37.112878+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'282","last_scrub_stamp":"2021-03-15T22:25:09.803362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.803362+0000","log_size":312,"ondisk_log_size":312,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20454716,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":223,"num_read_kb":245574,"num_write":136,"num_write_kb":43781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":21669077,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.a","version":"0'0","reported_seq":"390","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116249+0000","last_change":"2021-03-15T22:25:00.689659+0000","last_active":"2021-03-15T22:25:36.116249+0000","last_peered":"2021-03-15T22:25:36.116249+0000","last_clean":"2021-03-15T22:25:36.116249+0000","last_became_active":"2021-03-15T22:24:31.103470+0000","last_became_peered":"2021-03-15T22:24:31.103470+0000","last_unstale":"2021-03-15T22:25:36.116249+0000","last_undegraded":"2021-03-15T22:25:36.116249+0000","last_fullsized":"2021-03-15T22:25:36.116249+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.689591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.689591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"403","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140309+0000","last_change":"2021-03-15T22:24:31.115206+0000","last_active":"2021-03-15T22:25:38.140309+0000","last_peered":"2021-03-15T22:25:38.140309+0000","last_clean":"2021-03-15T22:25:38.140309+0000","last_became_active":"2021-03-15T22:24:31.097688+0000","last_became_peered":"2021-03-15T22:24:31.097688+0000","last_unstale":"2021-03-15T22:25:38.140309+0000","last_undegraded":"2021-03-15T22:25:38.140309+0000","last_fullsized":"2021-03-15T22:25:38.140309+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:23:03.888766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:03.888766+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"109","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115088+0000","last_change":"2021-03-15T22:25:32.799965+0000","last_active":"2021-03-15T22:25:36.115088+0000","last_peered":"2021-03-15T22:25:36.115088+0000","last_clean":"2021-03-15T22:25:36.115088+0000","last_became_active":"2021-03-15T22:24:23.137531+0000","last_became_peered":"2021-03-15T22:24:23.137531+0000","last_unstale":"2021-03-15T22:25:36.115088+0000","last_undegraded":"2021-03-15T22:25:36.115088+0000","last_fullsized":"2021-03-15T22:25:36.115088+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:25:32.799882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.799882+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"317'7","reported_seq":"493","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140619+0000","last_change":"2021-03-15T22:24:30.068613+0000","last_active":"2021-03-15T22:25:38.140619+0000","last_peered":"2021-03-15T22:25:38.140619+0000","last_clean":"2021-03-15T22:25:38.140619+0000","last_became_active":"2021-03-15T22:23:52.262878+0000","last_became_peered":"2021-03-15T22:23:52.262878+0000","last_unstale":"2021-03-15T22:25:38.140619+0000","last_undegraded":"2021-03-15T22:25:38.140619+0000","last_fullsized":"2021-03-15T22:25:38.140619+0000","mapping_epoch":316,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:24:30.068550+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.068550+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"426'552","reported_seq":"1658","reported_epoch":"426","state":"active+clean","last_fresh":"2021-03-15T22:25:41.161382+0000","last_change":"2021-03-15T22:25:41.161598+0000","last_active":"2021-03-15T22:25:41.161382+0000","last_peered":"2021-03-15T22:25:41.161382+0000","last_clean":"2021-03-15T22:25:41.161382+0000","last_became_active":"2021-03-15T22:23:04.067641+0000","last_became_peered":"2021-03-15T22:23:04.067641+0000","last_unstale":"2021-03-15T22:25:41.161382+0000","last_undegraded":"2021-03-15T22:25:41.161382+0000","last_fullsized":"2021-03-15T22:25:41.161382+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'455","last_scrub_stamp":"2021-03-15T22:24:28.184612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.184612+0000","log_size":552,"ondisk_log_size":552,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23978985,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":4,"num_read":309,"num_read_kb":316000,"num_write":202,"num_write_kb":75105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":32120558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.b","version":"0'0","reported_seq":"185","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.102699+0000","last_change":"2021-03-15T22:24:37.122501+0000","last_active":"2021-03-15T22:25:40.102699+0000","last_peered":"2021-03-15T22:25:40.102699+0000","last_clean":"2021-03-15T22:25:40.102699+0000","last_became_active":"2021-03-15T22:22:56.057174+0000","last_became_peered":"2021-03-15T22:22:56.057174+0000","last_unstale":"2021-03-15T22:25:40.102699+0000","last_undegraded":"2021-03-15T22:25:40.102699+0000","last_fullsized":"2021-03-15T22:25:40.102699+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:37.122407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.122407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"385","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.141328+0000","last_change":"2021-03-15T22:23:52.265794+0000","last_active":"2021-03-15T22:25:38.141328+0000","last_peered":"2021-03-15T22:25:38.141328+0000","last_clean":"2021-03-15T22:25:38.141328+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:25:38.141328+0000","last_undegraded":"2021-03-15T22:25:38.141328+0000","last_fullsized":"2021-03-15T22:25:38.141328+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:38.810397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.810397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"473","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.115406+0000","last_change":"2021-03-15T22:25:33.862633+0000","last_active":"2021-03-15T22:25:40.115406+0000","last_peered":"2021-03-15T22:25:40.115406+0000","last_clean":"2021-03-15T22:25:40.115406+0000","last_became_active":"2021-03-15T22:25:33.138262+0000","last_became_peered":"2021-03-15T22:25:33.138262+0000","last_unstale":"2021-03-15T22:25:40.115406+0000","last_undegraded":"2021-03-15T22:25:40.115406+0000","last_fullsized":"2021-03-15T22:25:40.115406+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:07.027095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.027095+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"418","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164384+0000","last_change":"2021-03-15T22:24:39.853332+0000","last_active":"2021-03-15T22:25:38.164384+0000","last_peered":"2021-03-15T22:25:38.164384+0000","last_clean":"2021-03-15T22:25:38.164384+0000","last_became_active":"2021-03-15T22:24:31.109058+0000","last_became_peered":"2021-03-15T22:24:31.109058+0000","last_unstale":"2021-03-15T22:25:38.164384+0000","last_undegraded":"2021-03-15T22:25:38.164384+0000","last_fullsized":"2021-03-15T22:25:38.164384+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.853267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.853267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"415'140","reported_seq":"1177","reported_epoch":"422","state":"active+clean","last_fresh":"2021-03-15T22:25:37.112823+0000","last_change":"2021-03-15T22:25:37.113314+0000","last_active":"2021-03-15T22:25:37.112823+0000","last_peered":"2021-03-15T22:25:37.112823+0000","last_clean":"2021-03-15T22:25:37.112823+0000","last_became_active":"2021-03-15T22:24:34.087074+0000","last_became_peered":"2021-03-15T22:24:34.087074+0000","last_unstale":"2021-03-15T22:25:37.112823+0000","last_undegraded":"2021-03-15T22:25:37.112823+0000","last_fullsized":"2021-03-15T22:25:37.112823+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'130","last_scrub_stamp":"2021-03-15T22:25:06.854931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.854931+0000","log_size":140,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2235397,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":42,"num_read_kb":30032,"num_write":45,"num_write_kb":15718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4470794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"12b","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.8","version":"0'0","reported_seq":"397","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.139797+0000","last_change":"2021-03-15T22:23:52.267464+0000","last_active":"2021-03-15T22:25:38.139797+0000","last_peered":"2021-03-15T22:25:38.139797+0000","last_clean":"2021-03-15T22:25:38.139797+0000","last_became_active":"2021-03-15T22:23:52.267164+0000","last_became_peered":"2021-03-15T22:23:52.267164+0000","last_unstale":"2021-03-15T22:25:38.139797+0000","last_undegraded":"2021-03-15T22:25:38.139797+0000","last_fullsized":"2021-03-15T22:25:38.139797+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:29.845442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.845442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"436","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164557+0000","last_change":"2021-03-15T22:25:14.114184+0000","last_active":"2021-03-15T22:25:38.164557+0000","last_peered":"2021-03-15T22:25:38.164557+0000","last_clean":"2021-03-15T22:25:38.164557+0000","last_became_active":"2021-03-15T22:25:14.113857+0000","last_became_peered":"2021-03-15T22:25:14.113857+0000","last_unstale":"2021-03-15T22:25:38.164557+0000","last_undegraded":"2021-03-15T22:25:38.164557+0000","last_fullsized":"2021-03-15T22:25:38.164557+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.575552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.575552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"405","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.141092+0000","last_change":"2021-03-15T22:25:14.118841+0000","last_active":"2021-03-15T22:25:38.141092+0000","last_peered":"2021-03-15T22:25:38.141092+0000","last_clean":"2021-03-15T22:25:38.141092+0000","last_became_active":"2021-03-15T22:25:14.118469+0000","last_became_peered":"2021-03-15T22:25:14.118469+0000","last_unstale":"2021-03-15T22:25:38.141092+0000","last_undegraded":"2021-03-15T22:25:38.141092+0000","last_fullsized":"2021-03-15T22:25:38.141092+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.024799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.024799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"466","reported_epoch":"424","state":"active+clean","last_fresh":"2021-03-15T22:25:39.142235+0000","last_change":"2021-03-15T22:24:40.468134+0000","last_active":"2021-03-15T22:25:39.142235+0000","last_peered":"2021-03-15T22:25:39.142235+0000","last_clean":"2021-03-15T22:25:39.142235+0000","last_became_active":"2021-03-15T22:24:03.085862+0000","last_became_peered":"2021-03-15T22:24:03.085862+0000","last_unstale":"2021-03-15T22:25:39.142235+0000","last_undegraded":"2021-03-15T22:25:39.142235+0000","last_fullsized":"2021-03-15T22:25:39.142235+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:24:40.468066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.468066+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"418'328","reported_seq":"1369","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164856+0000","last_change":"2021-03-15T22:25:37.707924+0000","last_active":"2021-03-15T22:25:38.164856+0000","last_peered":"2021-03-15T22:25:38.164856+0000","last_clean":"2021-03-15T22:25:38.164856+0000","last_became_active":"2021-03-15T22:20:05.032282+0000","last_became_peered":"2021-03-15T22:20:05.032282+0000","last_unstale":"2021-03-15T22:25:38.164856+0000","last_undegraded":"2021-03-15T22:25:38.164856+0000","last_fullsized":"2021-03-15T22:25:38.164856+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'328","last_scrub_stamp":"2021-03-15T22:25:37.707812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.707812+0000","log_size":328,"ondisk_log_size":328,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21341533,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":196,"num_read_kb":234047,"num_write":111,"num_write_kb":37367,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12b","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.9","version":"0'0","reported_seq":"85","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140421+0000","last_change":"2021-03-15T22:25:11.117468+0000","last_active":"2021-03-15T22:25:38.140421+0000","last_peered":"2021-03-15T22:25:38.140421+0000","last_clean":"2021-03-15T22:25:38.140421+0000","last_became_active":"2021-03-15T22:24:31.093577+0000","last_became_peered":"2021-03-15T22:24:31.093577+0000","last_unstale":"2021-03-15T22:25:38.140421+0000","last_undegraded":"2021-03-15T22:25:38.140421+0000","last_fullsized":"2021-03-15T22:25:38.140421+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.117403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.117403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"436","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164658+0000","last_change":"2021-03-15T22:25:33.134762+0000","last_active":"2021-03-15T22:25:38.164658+0000","last_peered":"2021-03-15T22:25:38.164658+0000","last_clean":"2021-03-15T22:25:38.164658+0000","last_became_active":"2021-03-15T22:25:33.134358+0000","last_became_peered":"2021-03-15T22:25:33.134358+0000","last_unstale":"2021-03-15T22:25:38.164658+0000","last_undegraded":"2021-03-15T22:25:38.164658+0000","last_fullsized":"2021-03-15T22:25:38.164658+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:27.869570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.869570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"400","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115646+0000","last_change":"2021-03-15T22:25:33.145899+0000","last_active":"2021-03-15T22:25:36.115646+0000","last_peered":"2021-03-15T22:25:36.115646+0000","last_clean":"2021-03-15T22:25:36.115646+0000","last_became_active":"2021-03-15T22:25:33.145548+0000","last_became_peered":"2021-03-15T22:25:33.145548+0000","last_unstale":"2021-03-15T22:25:36.115646+0000","last_undegraded":"2021-03-15T22:25:36.115646+0000","last_fullsized":"2021-03-15T22:25:36.115646+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:08.105283+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:08.105283+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"186","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164678+0000","last_change":"2021-03-15T22:25:33.912570+0000","last_active":"2021-03-15T22:25:38.164678+0000","last_peered":"2021-03-15T22:25:38.164678+0000","last_clean":"2021-03-15T22:25:38.164678+0000","last_became_active":"2021-03-15T22:25:33.135102+0000","last_became_peered":"2021-03-15T22:25:33.135102+0000","last_unstale":"2021-03-15T22:25:38.164678+0000","last_undegraded":"2021-03-15T22:25:38.164678+0000","last_fullsized":"2021-03-15T22:25:38.164678+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:01.499099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.499099+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"424'364","reported_seq":"1557","reported_epoch":"424","state":"active+clean","last_fresh":"2021-03-15T22:25:39.148388+0000","last_change":"2021-03-15T22:25:37.117331+0000","last_active":"2021-03-15T22:25:39.148388+0000","last_peered":"2021-03-15T22:25:39.148388+0000","last_clean":"2021-03-15T22:25:39.148388+0000","last_became_active":"2021-03-15T22:25:36.118229+0000","last_became_peered":"2021-03-15T22:25:36.118229+0000","last_unstale":"2021-03-15T22:25:39.148388+0000","last_undegraded":"2021-03-15T22:25:39.148388+0000","last_fullsized":"2021-03-15T22:25:39.148388+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'340","last_scrub_stamp":"2021-03-15T22:25:12.353191+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.353191+0000","log_size":364,"ondisk_log_size":364,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28154782,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":134,"num_read_kb":85561,"num_write":168,"num_write_kb":53979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":35579701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12b","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.6","version":"0'0","reported_seq":"18","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.103062+0000","last_change":"2021-03-15T22:25:33.152550+0000","last_active":"2021-03-15T22:25:40.103062+0000","last_peered":"2021-03-15T22:25:40.103062+0000","last_clean":"2021-03-15T22:25:40.103062+0000","last_became_active":"2021-03-15T22:25:33.152232+0000","last_became_peered":"2021-03-15T22:25:33.152232+0000","last_unstale":"2021-03-15T22:25:40.103062+0000","last_undegraded":"2021-03-15T22:25:40.103062+0000","last_fullsized":"2021-03-15T22:25:40.103062+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:34.763323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:34.763323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"389","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116102+0000","last_change":"2021-03-15T22:25:11.615097+0000","last_active":"2021-03-15T22:25:36.116102+0000","last_peered":"2021-03-15T22:25:36.116102+0000","last_clean":"2021-03-15T22:25:36.116102+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:25:36.116102+0000","last_undegraded":"2021-03-15T22:25:36.116102+0000","last_fullsized":"2021-03-15T22:25:36.116102+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.614975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.614975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"115","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164536+0000","last_change":"2021-03-15T22:24:31.106764+0000","last_active":"2021-03-15T22:25:38.164536+0000","last_peered":"2021-03-15T22:25:38.164536+0000","last_clean":"2021-03-15T22:25:38.164536+0000","last_became_active":"2021-03-15T22:24:31.106344+0000","last_became_peered":"2021-03-15T22:24:31.106344+0000","last_unstale":"2021-03-15T22:25:38.164536+0000","last_undegraded":"2021-03-15T22:25:38.164536+0000","last_fullsized":"2021-03-15T22:25:38.164536+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:00.938809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:00.938809+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"527","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112164+0000","last_change":"2021-03-15T22:25:33.174619+0000","last_active":"2021-03-15T22:25:36.112164+0000","last_peered":"2021-03-15T22:25:36.112164+0000","last_clean":"2021-03-15T22:25:36.112164+0000","last_became_active":"2021-03-15T22:25:33.126039+0000","last_became_peered":"2021-03-15T22:25:33.126039+0000","last_unstale":"2021-03-15T22:25:36.112164+0000","last_undegraded":"2021-03-15T22:25:36.112164+0000","last_fullsized":"2021-03-15T22:25:36.112164+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:25:15.810377+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.810377+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"77","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112488+0000","last_change":"2021-03-15T22:24:31.102016+0000","last_active":"2021-03-15T22:25:36.112488+0000","last_peered":"2021-03-15T22:25:36.112488+0000","last_clean":"2021-03-15T22:25:36.112488+0000","last_became_active":"2021-03-15T22:24:31.101668+0000","last_became_peered":"2021-03-15T22:24:31.101668+0000","last_unstale":"2021-03-15T22:25:36.112488+0000","last_undegraded":"2021-03-15T22:25:36.112488+0000","last_fullsized":"2021-03-15T22:25:36.112488+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:08.898279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:23:08.898279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"426'353","reported_seq":"1425","reported_epoch":"426","state":"active+clean","last_fresh":"2021-03-15T22:25:41.161377+0000","last_change":"2021-03-15T22:25:41.161569+0000","last_active":"2021-03-15T22:25:41.161377+0000","last_peered":"2021-03-15T22:25:41.161377+0000","last_clean":"2021-03-15T22:25:41.161377+0000","last_became_active":"2021-03-15T22:24:34.087484+0000","last_became_peered":"2021-03-15T22:24:34.087484+0000","last_unstale":"2021-03-15T22:25:41.161377+0000","last_undegraded":"2021-03-15T22:25:41.161377+0000","last_fullsized":"2021-03-15T22:25:41.161377+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'329","last_scrub_stamp":"2021-03-15T22:25:14.014776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:14.014776+0000","log_size":353,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14009532,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":124,"num_read_kb":112611,"num_write":123,"num_write_kb":40085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":10768640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.5","version":"0'0","reported_seq":"397","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.141178+0000","last_change":"2021-03-15T22:23:52.267559+0000","last_active":"2021-03-15T22:25:38.141178+0000","last_peered":"2021-03-15T22:25:38.141178+0000","last_clean":"2021-03-15T22:25:38.141178+0000","last_became_active":"2021-03-15T22:23:52.266556+0000","last_became_peered":"2021-03-15T22:23:52.266556+0000","last_unstale":"2021-03-15T22:25:38.141178+0000","last_undegraded":"2021-03-15T22:25:38.141178+0000","last_fullsized":"2021-03-15T22:25:38.141178+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:30.831598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:30.831598+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"390","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.114999+0000","last_change":"2021-03-15T22:24:39.929128+0000","last_active":"2021-03-15T22:25:36.114999+0000","last_peered":"2021-03-15T22:25:36.114999+0000","last_clean":"2021-03-15T22:25:36.114999+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:25:36.114999+0000","last_undegraded":"2021-03-15T22:25:36.114999+0000","last_fullsized":"2021-03-15T22:25:36.114999+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.929066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.929066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"422","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164421+0000","last_change":"2021-03-15T22:25:33.130332+0000","last_active":"2021-03-15T22:25:38.164421+0000","last_peered":"2021-03-15T22:25:38.164421+0000","last_clean":"2021-03-15T22:25:38.164421+0000","last_became_active":"2021-03-15T22:25:33.129875+0000","last_became_peered":"2021-03-15T22:25:33.129875+0000","last_unstale":"2021-03-15T22:25:38.164421+0000","last_undegraded":"2021-03-15T22:25:38.164421+0000","last_fullsized":"2021-03-15T22:25:38.164421+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:29.266174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.266174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"149'22","reported_seq":"72","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.111773+0000","last_change":"2021-03-15T22:25:03.853145+0000","last_active":"2021-03-15T22:25:36.111773+0000","last_peered":"2021-03-15T22:25:36.111773+0000","last_clean":"2021-03-15T22:25:36.111773+0000","last_became_active":"2021-03-15T22:24:55.093124+0000","last_became_peered":"2021-03-15T22:24:55.093124+0000","last_unstale":"2021-03-15T22:25:36.111773+0000","last_undegraded":"2021-03-15T22:25:36.111773+0000","last_fullsized":"2021-03-15T22:25:36.111773+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'22","last_scrub_stamp":"2021-03-15T22:25:03.853079+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:25:03.853079+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"409","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164245+0000","last_change":"2021-03-15T22:25:02.520674+0000","last_active":"2021-03-15T22:25:38.164245+0000","last_peered":"2021-03-15T22:25:38.164245+0000","last_clean":"2021-03-15T22:25:38.164245+0000","last_became_active":"2021-03-15T22:21:38.147364+0000","last_became_peered":"2021-03-15T22:21:38.147364+0000","last_unstale":"2021-03-15T22:25:38.164245+0000","last_undegraded":"2021-03-15T22:25:38.164245+0000","last_fullsized":"2021-03-15T22:25:38.164245+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.520588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.520588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"422'925","reported_seq":"2694","reported_epoch":"424","state":"active+clean","last_fresh":"2021-03-15T22:25:39.144521+0000","last_change":"2021-03-15T22:25:37.125253+0000","last_active":"2021-03-15T22:25:39.144521+0000","last_peered":"2021-03-15T22:25:39.144521+0000","last_clean":"2021-03-15T22:25:39.144521+0000","last_became_active":"2021-03-15T22:25:37.118396+0000","last_became_peered":"2021-03-15T22:25:37.118396+0000","last_unstale":"2021-03-15T22:25:39.144521+0000","last_undegraded":"2021-03-15T22:25:39.144521+0000","last_fullsized":"2021-03-15T22:25:39.144521+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'350","last_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:22:40.995789+0000","log_size":925,"ondisk_log_size":925,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":62106226,"num_objects":26,"num_object_clones":19,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":2,"num_read":353,"num_read_kb":305564,"num_write":349,"num_write_kb":154675,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":140467323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.7","version":"0'0","reported_seq":"394","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115748+0000","last_change":"2021-03-15T22:23:38.968009+0000","last_active":"2021-03-15T22:25:36.115748+0000","last_peered":"2021-03-15T22:25:36.115748+0000","last_clean":"2021-03-15T22:25:36.115748+0000","last_became_active":"2021-03-15T22:22:56.062745+0000","last_became_peered":"2021-03-15T22:22:56.062745+0000","last_unstale":"2021-03-15T22:25:36.115748+0000","last_undegraded":"2021-03-15T22:25:36.115748+0000","last_fullsized":"2021-03-15T22:25:36.115748+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:38.967934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:38.967934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"399","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.139973+0000","last_change":"2021-03-15T22:24:31.111721+0000","last_active":"2021-03-15T22:25:38.139973+0000","last_peered":"2021-03-15T22:25:38.139973+0000","last_clean":"2021-03-15T22:25:38.139973+0000","last_became_active":"2021-03-15T22:24:31.111104+0000","last_became_peered":"2021-03-15T22:24:31.111104+0000","last_unstale":"2021-03-15T22:25:38.139973+0000","last_undegraded":"2021-03-15T22:25:38.139973+0000","last_fullsized":"2021-03-15T22:25:38.139973+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.872030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.872030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"18","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.101419+0000","last_change":"2021-03-15T22:25:33.147575+0000","last_active":"2021-03-15T22:25:40.101419+0000","last_peered":"2021-03-15T22:25:40.101419+0000","last_clean":"2021-03-15T22:25:40.101419+0000","last_became_active":"2021-03-15T22:25:33.147285+0000","last_became_peered":"2021-03-15T22:25:33.147285+0000","last_unstale":"2021-03-15T22:25:40.101419+0000","last_undegraded":"2021-03-15T22:25:40.101419+0000","last_fullsized":"2021-03-15T22:25:40.101419+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:27.994280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:27.994280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"419","reported_epoch":"424","state":"active+clean","last_fresh":"2021-03-15T22:25:39.143632+0000","last_change":"2021-03-15T22:25:37.458353+0000","last_active":"2021-03-15T22:25:39.143632+0000","last_peered":"2021-03-15T22:25:39.143632+0000","last_clean":"2021-03-15T22:25:39.143632+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:25:39.143632+0000","last_undegraded":"2021-03-15T22:25:39.143632+0000","last_fullsized":"2021-03-15T22:25:39.143632+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:37.458291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.458291+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"383","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140542+0000","last_change":"2021-03-15T22:23:52.271864+0000","last_active":"2021-03-15T22:25:38.140542+0000","last_peered":"2021-03-15T22:25:38.140542+0000","last_clean":"2021-03-15T22:25:38.140542+0000","last_became_active":"2021-03-15T22:23:52.271442+0000","last_became_peered":"2021-03-15T22:23:52.271442+0000","last_unstale":"2021-03-15T22:25:38.140542+0000","last_undegraded":"2021-03-15T22:25:38.140542+0000","last_fullsized":"2021-03-15T22:25:38.140542+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:26.748485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.748485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"422'240","reported_seq":"1323","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140725+0000","last_change":"2021-03-15T22:25:37.120219+0000","last_active":"2021-03-15T22:25:38.140725+0000","last_peered":"2021-03-15T22:25:38.140725+0000","last_clean":"2021-03-15T22:25:38.140725+0000","last_became_active":"2021-03-15T22:24:36.092680+0000","last_became_peered":"2021-03-15T22:24:36.092680+0000","last_unstale":"2021-03-15T22:25:38.140725+0000","last_undegraded":"2021-03-15T22:25:38.140725+0000","last_fullsized":"2021-03-15T22:25:38.140725+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'138","last_scrub_stamp":"2021-03-15T22:23:41.141218+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.141218+0000","log_size":240,"ondisk_log_size":240,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20445891,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":101,"num_read_kb":93801,"num_write":95,"num_write_kb":37009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8487583,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"12b","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.4","version":"0'0","reported_seq":"402","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.111926+0000","last_change":"2021-03-15T22:25:33.129904+0000","last_active":"2021-03-15T22:25:36.111926+0000","last_peered":"2021-03-15T22:25:36.111926+0000","last_clean":"2021-03-15T22:25:36.111926+0000","last_became_active":"2021-03-15T22:25:33.129602+0000","last_became_peered":"2021-03-15T22:25:33.129602+0000","last_unstale":"2021-03-15T22:25:36.111926+0000","last_undegraded":"2021-03-15T22:25:36.111926+0000","last_fullsized":"2021-03-15T22:25:36.111926+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.931058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.931058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"224","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.115299+0000","last_change":"2021-03-15T22:25:33.143239+0000","last_active":"2021-03-15T22:25:40.115299+0000","last_peered":"2021-03-15T22:25:40.115299+0000","last_clean":"2021-03-15T22:25:40.115299+0000","last_became_active":"2021-03-15T22:25:33.142133+0000","last_became_peered":"2021-03-15T22:25:33.142133+0000","last_unstale":"2021-03-15T22:25:40.115299+0000","last_undegraded":"2021-03-15T22:25:40.115299+0000","last_fullsized":"2021-03-15T22:25:40.115299+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.028082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.028082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"856","reported_epoch":"424","state":"active+clean","last_fresh":"2021-03-15T22:25:39.141996+0000","last_change":"2021-03-15T22:24:31.248181+0000","last_active":"2021-03-15T22:25:39.141996+0000","last_peered":"2021-03-15T22:25:39.141996+0000","last_clean":"2021-03-15T22:25:39.141996+0000","last_became_active":"2021-03-15T22:24:31.109631+0000","last_became_peered":"2021-03-15T22:24:31.109631+0000","last_unstale":"2021-03-15T22:25:39.141996+0000","last_undegraded":"2021-03-15T22:25:39.141996+0000","last_fullsized":"2021-03-15T22:25:39.141996+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:23:27.790040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:23:27.790040+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":451,"num_read_kb":446,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"143","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.111574+0000","last_change":"2021-03-15T22:24:37.977457+0000","last_active":"2021-03-15T22:25:36.111574+0000","last_peered":"2021-03-15T22:25:36.111574+0000","last_clean":"2021-03-15T22:25:36.111574+0000","last_became_active":"2021-03-15T22:24:31.100536+0000","last_became_peered":"2021-03-15T22:24:31.100536+0000","last_unstale":"2021-03-15T22:25:36.111574+0000","last_undegraded":"2021-03-15T22:25:36.111574+0000","last_fullsized":"2021-03-15T22:25:36.111574+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:37.977363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.977363+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"33","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.115758+0000","last_change":"2021-03-15T22:25:14.115071+0000","last_active":"2021-03-15T22:25:36.115758+0000","last_peered":"2021-03-15T22:25:36.115758+0000","last_clean":"2021-03-15T22:25:36.115758+0000","last_became_active":"2021-03-15T22:25:14.114729+0000","last_became_peered":"2021-03-15T22:25:14.114729+0000","last_unstale":"2021-03-15T22:25:36.115758+0000","last_undegraded":"2021-03-15T22:25:36.115758+0000","last_fullsized":"2021-03-15T22:25:36.115758+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.857011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.857011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"421'960","reported_seq":"2465","reported_epoch":"422","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-03-15T22:25:37.970124+0000","last_change":"2021-03-15T22:25:35.390581+0000","last_active":"2021-03-15T22:25:37.970124+0000","last_peered":"2021-03-15T22:25:37.970124+0000","last_clean":"2021-03-15T22:25:31.734041+0000","last_became_active":"2021-03-15T22:25:34.232739+0000","last_became_peered":"2021-03-15T22:25:34.232739+0000","last_unstale":"2021-03-15T22:25:37.970124+0000","last_undegraded":"2021-03-15T22:25:34.132435+0000","last_fullsized":"2021-03-15T22:25:34.126282+0000","mapping_epoch":418,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":338,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'777","last_scrub_stamp":"2021-03-15T22:24:40.640400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.640400+0000","log_size":960,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":57399209,"num_objects":30,"num_object_clones":23,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":1,"num_read":566,"num_read_kb":611849,"num_write":355,"num_write_kb":136357,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":206,"num_bytes_recovered":71334852,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1],"avail_no_missing":["5","1","2"],"object_location_counts":[{"shards":"1,2,5","objects":26},{"shards":"1,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"384","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.113085+0000","last_change":"2021-03-15T22:24:30.950198+0000","last_active":"2021-03-15T22:25:36.113085+0000","last_peered":"2021-03-15T22:25:36.113085+0000","last_clean":"2021-03-15T22:25:36.113085+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:25:36.113085+0000","last_undegraded":"2021-03-15T22:25:36.113085+0000","last_fullsized":"2021-03-15T22:25:36.113085+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.950135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.950135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"390","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112111+0000","last_change":"2021-03-15T22:24:26.962034+0000","last_active":"2021-03-15T22:25:36.112111+0000","last_peered":"2021-03-15T22:25:36.112111+0000","last_clean":"2021-03-15T22:25:36.112111+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:25:36.112111+0000","last_undegraded":"2021-03-15T22:25:36.112111+0000","last_fullsized":"2021-03-15T22:25:36.112111+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.961978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.961978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"166","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116394+0000","last_change":"2021-03-15T22:24:31.147838+0000","last_active":"2021-03-15T22:25:36.116394+0000","last_peered":"2021-03-15T22:25:36.116394+0000","last_clean":"2021-03-15T22:25:36.116394+0000","last_became_active":"2021-03-15T22:24:31.098665+0000","last_became_peered":"2021-03-15T22:24:31.098665+0000","last_unstale":"2021-03-15T22:25:36.116394+0000","last_undegraded":"2021-03-15T22:25:36.116394+0000","last_fullsized":"2021-03-15T22:25:36.116394+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:23:36.678184+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:23:36.678184+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"399","reported_epoch":"424","state":"active+clean","last_fresh":"2021-03-15T22:25:39.142311+0000","last_change":"2021-03-15T22:24:34.454377+0000","last_active":"2021-03-15T22:25:39.142311+0000","last_peered":"2021-03-15T22:25:39.142311+0000","last_clean":"2021-03-15T22:25:39.142311+0000","last_became_active":"2021-03-15T22:24:31.107629+0000","last_became_peered":"2021-03-15T22:24:31.107629+0000","last_unstale":"2021-03-15T22:25:39.142311+0000","last_undegraded":"2021-03-15T22:25:39.142311+0000","last_fullsized":"2021-03-15T22:25:39.142311+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.454313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.454313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"425'154","reported_seq":"1245","reported_epoch":"426","state":"active+clean","last_fresh":"2021-03-15T22:25:41.154398+0000","last_change":"2021-03-15T22:25:41.155378+0000","last_active":"2021-03-15T22:25:41.154398+0000","last_peered":"2021-03-15T22:25:41.154398+0000","last_clean":"2021-03-15T22:25:41.154398+0000","last_became_active":"2021-03-15T22:25:37.112715+0000","last_became_peered":"2021-03-15T22:25:37.112715+0000","last_unstale":"2021-03-15T22:25:41.154398+0000","last_undegraded":"2021-03-15T22:25:41.154398+0000","last_fullsized":"2021-03-15T22:25:41.154398+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'104","last_scrub_stamp":"2021-03-15T22:24:02.949625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:02.949625+0000","log_size":154,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16081782,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":81,"num_read_kb":100355,"num_write":58,"num_write_kb":24058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":14412391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.0","version":"0'0","reported_seq":"424","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.100768+0000","last_change":"2021-03-15T22:25:14.112119+0000","last_active":"2021-03-15T22:25:40.100768+0000","last_peered":"2021-03-15T22:25:40.100768+0000","last_clean":"2021-03-15T22:25:40.100768+0000","last_became_active":"2021-03-15T22:25:14.111797+0000","last_became_peered":"2021-03-15T22:25:14.111797+0000","last_unstale":"2021-03-15T22:25:40.100768+0000","last_undegraded":"2021-03-15T22:25:40.100768+0000","last_fullsized":"2021-03-15T22:25:40.100768+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.480279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"400","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116610+0000","last_change":"2021-03-15T22:25:33.141491+0000","last_active":"2021-03-15T22:25:36.116610+0000","last_peered":"2021-03-15T22:25:36.116610+0000","last_clean":"2021-03-15T22:25:36.116610+0000","last_became_active":"2021-03-15T22:25:33.141147+0000","last_became_peered":"2021-03-15T22:25:33.141147+0000","last_unstale":"2021-03-15T22:25:36.116610+0000","last_undegraded":"2021-03-15T22:25:36.116610+0000","last_fullsized":"2021-03-15T22:25:36.116610+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.769850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.769850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"513","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112842+0000","last_change":"2021-03-15T22:25:15.063557+0000","last_active":"2021-03-15T22:25:36.112842+0000","last_peered":"2021-03-15T22:25:36.112842+0000","last_clean":"2021-03-15T22:25:36.112842+0000","last_became_active":"2021-03-15T22:25:14.114107+0000","last_became_peered":"2021-03-15T22:25:14.114107+0000","last_unstale":"2021-03-15T22:25:36.112842+0000","last_undegraded":"2021-03-15T22:25:36.112842+0000","last_fullsized":"2021-03-15T22:25:36.112842+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:23:29.014596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.014596+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"356'3","reported_seq":"107","reported_epoch":"424","state":"active+clean","last_fresh":"2021-03-15T22:25:39.142320+0000","last_change":"2021-03-15T22:24:31.111939+0000","last_active":"2021-03-15T22:25:39.142320+0000","last_peered":"2021-03-15T22:25:39.142320+0000","last_clean":"2021-03-15T22:25:39.142320+0000","last_became_active":"2021-03-15T22:24:31.105948+0000","last_became_peered":"2021-03-15T22:24:31.105948+0000","last_unstale":"2021-03-15T22:25:39.142320+0000","last_undegraded":"2021-03-15T22:25:39.142320+0000","last_fullsized":"2021-03-15T22:25:39.142320+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:24:02.507323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:02.507323+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"426'251","reported_seq":"1288","reported_epoch":"426","state":"active+clean","last_fresh":"2021-03-15T22:25:41.164406+0000","last_change":"2021-03-15T22:25:41.164567+0000","last_active":"2021-03-15T22:25:41.164406+0000","last_peered":"2021-03-15T22:25:41.164406+0000","last_clean":"2021-03-15T22:25:41.164406+0000","last_became_active":"2021-03-15T22:23:00.057259+0000","last_became_peered":"2021-03-15T22:23:00.057259+0000","last_unstale":"2021-03-15T22:25:41.164406+0000","last_undegraded":"2021-03-15T22:25:41.164406+0000","last_fullsized":"2021-03-15T22:25:41.164406+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'202","last_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.006150+0000","log_size":251,"ondisk_log_size":251,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13478841,"num_objects":7,"num_object_clones":5,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":123,"num_read_kb":122012,"num_write":89,"num_write_kb":33243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":6765552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.3","version":"0'0","reported_seq":"384","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.140967+0000","last_change":"2021-03-15T22:25:16.234631+0000","last_active":"2021-03-15T22:25:38.140967+0000","last_peered":"2021-03-15T22:25:38.140967+0000","last_clean":"2021-03-15T22:25:38.140967+0000","last_became_active":"2021-03-15T22:23:52.260549+0000","last_became_peered":"2021-03-15T22:23:52.260549+0000","last_unstale":"2021-03-15T22:25:38.140967+0000","last_undegraded":"2021-03-15T22:25:38.140967+0000","last_fullsized":"2021-03-15T22:25:38.140967+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.234565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.234565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"84","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.110621+0000","last_change":"2021-03-15T22:25:02.868460+0000","last_active":"2021-03-15T22:25:36.110621+0000","last_peered":"2021-03-15T22:25:36.110621+0000","last_clean":"2021-03-15T22:25:36.110621+0000","last_became_active":"2021-03-15T22:24:31.099610+0000","last_became_peered":"2021-03-15T22:24:31.099610+0000","last_unstale":"2021-03-15T22:25:36.110621+0000","last_undegraded":"2021-03-15T22:25:36.110621+0000","last_fullsized":"2021-03-15T22:25:36.110621+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.868399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.868399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"425","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.115380+0000","last_change":"2021-03-15T22:25:39.041168+0000","last_active":"2021-03-15T22:25:40.115380+0000","last_peered":"2021-03-15T22:25:40.115380+0000","last_clean":"2021-03-15T22:25:40.115380+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:25:40.115380+0000","last_undegraded":"2021-03-15T22:25:40.115380+0000","last_fullsized":"2021-03-15T22:25:40.115380+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:23:07.762338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:07.762338+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"421'6","reported_seq":"23","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.100560+0000","last_change":"2021-03-15T22:25:33.171536+0000","last_active":"2021-03-15T22:25:40.100560+0000","last_peered":"2021-03-15T22:25:40.100560+0000","last_clean":"2021-03-15T22:25:40.100560+0000","last_became_active":"2021-03-15T22:25:33.130973+0000","last_became_peered":"2021-03-15T22:25:33.130973+0000","last_unstale":"2021-03-15T22:25:40.100560+0000","last_undegraded":"2021-03-15T22:25:40.100560+0000","last_fullsized":"2021-03-15T22:25:40.100560+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'4","last_scrub_stamp":"2021-03-15T22:23:35.100079+0000","last_deep_scrub":"300'4","last_deep_scrub_stamp":"2021-03-15T22:23:35.100079+0000","last_clean_scrub_stamp":"2021-03-15T22:23:35.100079+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"426'437","reported_seq":"1573","reported_epoch":"426","state":"active+clean","last_fresh":"2021-03-15T22:25:41.159660+0000","last_change":"2021-03-15T22:25:41.159846+0000","last_active":"2021-03-15T22:25:41.159660+0000","last_peered":"2021-03-15T22:25:41.159660+0000","last_clean":"2021-03-15T22:25:41.159660+0000","last_became_active":"2021-03-15T22:23:01.072357+0000","last_became_peered":"2021-03-15T22:23:01.072357+0000","last_unstale":"2021-03-15T22:25:41.159660+0000","last_undegraded":"2021-03-15T22:25:41.159660+0000","last_fullsized":"2021-03-15T22:25:41.159660+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'302","last_scrub_stamp":"2021-03-15T22:23:59.467761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:59.467761+0000","log_size":437,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26858891,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":240,"num_read_kb":259097,"num_write":145,"num_write_kb":58613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31233387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.2","version":"0'0","reported_seq":"404","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.113989+0000","last_change":"2021-03-15T22:25:33.136943+0000","last_active":"2021-03-15T22:25:40.113989+0000","last_peered":"2021-03-15T22:25:40.113989+0000","last_clean":"2021-03-15T22:25:40.113989+0000","last_became_active":"2021-03-15T22:25:33.136621+0000","last_became_peered":"2021-03-15T22:25:33.136621+0000","last_unstale":"2021-03-15T22:25:40.113989+0000","last_undegraded":"2021-03-15T22:25:40.113989+0000","last_fullsized":"2021-03-15T22:25:40.113989+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:32.751660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:32.751660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"194","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.103623+0000","last_change":"2021-03-15T22:25:12.911203+0000","last_active":"2021-03-15T22:25:40.103623+0000","last_peered":"2021-03-15T22:25:40.103623+0000","last_clean":"2021-03-15T22:25:40.103623+0000","last_became_active":"2021-03-15T22:24:31.091206+0000","last_became_peered":"2021-03-15T22:24:31.091206+0000","last_unstale":"2021-03-15T22:25:40.103623+0000","last_undegraded":"2021-03-15T22:25:40.103623+0000","last_fullsized":"2021-03-15T22:25:40.103623+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.911140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.911140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"44","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.100399+0000","last_change":"2021-03-15T22:25:33.758535+0000","last_active":"2021-03-15T22:25:40.100399+0000","last_peered":"2021-03-15T22:25:40.100399+0000","last_clean":"2021-03-15T22:25:40.100399+0000","last_became_active":"2021-03-15T22:25:33.149362+0000","last_became_peered":"2021-03-15T22:25:33.149362+0000","last_unstale":"2021-03-15T22:25:40.100399+0000","last_undegraded":"2021-03-15T22:25:40.100399+0000","last_fullsized":"2021-03-15T22:25:40.100399+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:24:01.896988+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.896988+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"417","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.115104+0000","last_change":"2021-03-15T22:25:33.143430+0000","last_active":"2021-03-15T22:25:40.115104+0000","last_peered":"2021-03-15T22:25:40.115104+0000","last_clean":"2021-03-15T22:25:40.115104+0000","last_became_active":"2021-03-15T22:25:33.142712+0000","last_became_peered":"2021-03-15T22:25:33.142712+0000","last_unstale":"2021-03-15T22:25:40.115104+0000","last_undegraded":"2021-03-15T22:25:40.115104+0000","last_fullsized":"2021-03-15T22:25:40.115104+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.989313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.989313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"425'409","reported_seq":"1553","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.104375+0000","last_change":"2021-03-15T22:25:37.111677+0000","last_active":"2021-03-15T22:25:40.104375+0000","last_peered":"2021-03-15T22:25:40.104375+0000","last_clean":"2021-03-15T22:25:40.104375+0000","last_became_active":"2021-03-15T22:25:36.118668+0000","last_became_peered":"2021-03-15T22:25:36.118668+0000","last_unstale":"2021-03-15T22:25:40.104375+0000","last_undegraded":"2021-03-15T22:25:40.104375+0000","last_fullsized":"2021-03-15T22:25:40.104375+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'370","last_scrub_stamp":"2021-03-15T22:25:07.414191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.414191+0000","log_size":409,"ondisk_log_size":409,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21706719,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":180,"num_read_kb":168082,"num_write":128,"num_write_kb":48783,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25937939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12b","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.d","version":"0'0","reported_seq":"385","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.116380+0000","last_change":"2021-03-15T22:24:31.094621+0000","last_active":"2021-03-15T22:25:36.116380+0000","last_peered":"2021-03-15T22:25:36.116380+0000","last_clean":"2021-03-15T22:25:36.116380+0000","last_became_active":"2021-03-15T22:24:31.094299+0000","last_became_peered":"2021-03-15T22:24:31.094299+0000","last_unstale":"2021-03-15T22:25:36.116380+0000","last_undegraded":"2021-03-15T22:25:36.116380+0000","last_fullsized":"2021-03-15T22:25:36.116380+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.936738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.936738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"391'7","reported_seq":"531","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114817+0000","last_change":"2021-03-15T22:25:03.403843+0000","last_active":"2021-03-15T22:25:40.114817+0000","last_peered":"2021-03-15T22:25:40.114817+0000","last_clean":"2021-03-15T22:25:40.114817+0000","last_became_active":"2021-03-15T22:25:03.403350+0000","last_became_peered":"2021-03-15T22:25:03.403350+0000","last_unstale":"2021-03-15T22:25:40.114817+0000","last_undegraded":"2021-03-15T22:25:40.114817+0000","last_fullsized":"2021-03-15T22:25:40.114817+0000","mapping_epoch":387,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"152'4","last_scrub_stamp":"2021-03-15T22:23:35.793433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:23:35.793433+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"18","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.100476+0000","last_change":"2021-03-15T22:25:33.148929+0000","last_active":"2021-03-15T22:25:40.100476+0000","last_peered":"2021-03-15T22:25:40.100476+0000","last_clean":"2021-03-15T22:25:40.100476+0000","last_became_active":"2021-03-15T22:25:33.148614+0000","last_became_peered":"2021-03-15T22:25:33.148614+0000","last_unstale":"2021-03-15T22:25:40.100476+0000","last_undegraded":"2021-03-15T22:25:40.100476+0000","last_fullsized":"2021-03-15T22:25:40.100476+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:04.268977+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:04.268977+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"449","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.165070+0000","last_change":"2021-03-15T22:23:58.133442+0000","last_active":"2021-03-15T22:25:38.165070+0000","last_peered":"2021-03-15T22:25:38.165070+0000","last_clean":"2021-03-15T22:25:38.165070+0000","last_became_active":"2021-03-15T22:22:56.057343+0000","last_became_peered":"2021-03-15T22:22:56.057343+0000","last_unstale":"2021-03-15T22:25:38.165070+0000","last_undegraded":"2021-03-15T22:25:38.165070+0000","last_fullsized":"2021-03-15T22:25:38.165070+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:23:58.133378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.133378+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"425'79","reported_seq":"1128","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.127981+0000","last_change":"2021-03-15T22:25:37.111384+0000","last_active":"2021-03-15T22:25:40.127981+0000","last_peered":"2021-03-15T22:25:40.127981+0000","last_clean":"2021-03-15T22:25:40.127981+0000","last_became_active":"2021-03-15T22:24:31.104356+0000","last_became_peered":"2021-03-15T22:24:31.104356+0000","last_unstale":"2021-03-15T22:25:40.127981+0000","last_undegraded":"2021-03-15T22:25:40.127981+0000","last_fullsized":"2021-03-15T22:25:40.127981+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'77","last_scrub_stamp":"2021-03-15T22:25:32.490694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.490694+0000","log_size":79,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7752259,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":80,"num_read_kb":80817,"num_write":32,"num_write_kb":13772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1551588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"12b","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.c","version":"0'0","reported_seq":"18","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.101962+0000","last_change":"2021-03-15T22:25:33.148157+0000","last_active":"2021-03-15T22:25:40.101962+0000","last_peered":"2021-03-15T22:25:40.101962+0000","last_clean":"2021-03-15T22:25:40.101962+0000","last_became_active":"2021-03-15T22:25:33.147717+0000","last_became_peered":"2021-03-15T22:25:33.147717+0000","last_unstale":"2021-03-15T22:25:40.101962+0000","last_undegraded":"2021-03-15T22:25:40.101962+0000","last_fullsized":"2021-03-15T22:25:40.101962+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.372535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.372535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"147'5","reported_seq":"458","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:37.085461+0000","last_change":"2021-03-15T22:24:35.715464+0000","last_active":"2021-03-15T22:25:37.085461+0000","last_peered":"2021-03-15T22:25:37.085461+0000","last_clean":"2021-03-15T22:25:37.085461+0000","last_became_active":"2021-03-15T22:21:02.017066+0000","last_became_peered":"2021-03-15T22:21:02.017066+0000","last_unstale":"2021-03-15T22:25:37.085461+0000","last_undegraded":"2021-03-15T22:25:37.085461+0000","last_fullsized":"2021-03-15T22:25:37.085461+0000","mapping_epoch":60,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'5","last_scrub_stamp":"2021-03-15T22:24:35.715399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:35.715399+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"394","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.115225+0000","last_change":"2021-03-15T22:24:34.991087+0000","last_active":"2021-03-15T22:25:40.115225+0000","last_peered":"2021-03-15T22:25:40.115225+0000","last_clean":"2021-03-15T22:25:40.115225+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:25:40.115225+0000","last_undegraded":"2021-03-15T22:25:40.115225+0000","last_fullsized":"2021-03-15T22:25:40.115225+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.991018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.991018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"475","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114642+0000","last_change":"2021-03-15T22:24:55.128840+0000","last_active":"2021-03-15T22:25:40.114642+0000","last_peered":"2021-03-15T22:25:40.114642+0000","last_clean":"2021-03-15T22:25:40.114642+0000","last_became_active":"2021-03-15T22:24:55.090544+0000","last_became_peered":"2021-03-15T22:24:55.090544+0000","last_unstale":"2021-03-15T22:25:40.114642+0000","last_undegraded":"2021-03-15T22:25:40.114642+0000","last_fullsized":"2021-03-15T22:25:40.114642+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:38.685706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:38.685706+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"413'125","reported_seq":"1284","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.165075+0000","last_change":"2021-03-15T22:25:37.121408+0000","last_active":"2021-03-15T22:25:38.165075+0000","last_peered":"2021-03-15T22:25:38.165075+0000","last_clean":"2021-03-15T22:25:38.165075+0000","last_became_active":"2021-03-15T22:25:37.119078+0000","last_became_peered":"2021-03-15T22:25:37.119078+0000","last_unstale":"2021-03-15T22:25:38.165075+0000","last_undegraded":"2021-03-15T22:25:38.165075+0000","last_fullsized":"2021-03-15T22:25:38.165075+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'115","last_scrub_stamp":"2021-03-15T22:25:10.616929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.616929+0000","log_size":125,"ondisk_log_size":125,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9539709,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":110,"num_read_kb":136588,"num_write":53,"num_write_kb":14217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":16774328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"12b","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.f","version":"0'0","reported_seq":"404","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.702159+0000","last_change":"2021-03-15T22:25:38.702159+0000","last_active":"2021-03-15T22:25:38.702159+0000","last_peered":"2021-03-15T22:25:38.702159+0000","last_clean":"2021-03-15T22:25:38.702159+0000","last_became_active":"2021-03-15T22:25:33.134275+0000","last_became_peered":"2021-03-15T22:25:33.134275+0000","last_unstale":"2021-03-15T22:25:38.702159+0000","last_undegraded":"2021-03-15T22:25:38.702159+0000","last_fullsized":"2021-03-15T22:25:38.702159+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:23:02.474236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"413","reported_epoch":"421","state":"active+clean","last_fresh":"2021-03-15T22:25:36.112249+0000","last_change":"2021-03-15T22:24:31.107075+0000","last_active":"2021-03-15T22:25:36.112249+0000","last_peered":"2021-03-15T22:25:36.112249+0000","last_clean":"2021-03-15T22:25:36.112249+0000","last_became_active":"2021-03-15T22:24:31.106634+0000","last_became_peered":"2021-03-15T22:24:31.106634+0000","last_unstale":"2021-03-15T22:25:36.112249+0000","last_undegraded":"2021-03-15T22:25:36.112249+0000","last_fullsized":"2021-03-15T22:25:36.112249+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:42.177422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.177422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"397","reported_epoch":"424","state":"active+clean","last_fresh":"2021-03-15T22:25:39.142101+0000","last_change":"2021-03-15T22:24:28.327738+0000","last_active":"2021-03-15T22:25:39.142101+0000","last_peered":"2021-03-15T22:25:39.142101+0000","last_clean":"2021-03-15T22:25:39.142101+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:25:39.142101+0000","last_undegraded":"2021-03-15T22:25:39.142101+0000","last_fullsized":"2021-03-15T22:25:39.142101+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:28.327677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.327677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"115'33","reported_seq":"440","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114313+0000","last_change":"2021-03-15T22:25:04.028853+0000","last_active":"2021-03-15T22:25:40.114313+0000","last_peered":"2021-03-15T22:25:40.114313+0000","last_clean":"2021-03-15T22:25:40.114313+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:25:40.114313+0000","last_undegraded":"2021-03-15T22:25:40.114313+0000","last_fullsized":"2021-03-15T22:25:40.114313+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'33","last_scrub_stamp":"2021-03-15T22:25:04.028780+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:25:04.028780+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"410'378","reported_seq":"1645","reported_epoch":"426","state":"active+clean","last_fresh":"2021-03-15T22:25:41.161700+0000","last_change":"2021-03-15T22:25:41.161917+0000","last_active":"2021-03-15T22:25:41.161700+0000","last_peered":"2021-03-15T22:25:41.161700+0000","last_clean":"2021-03-15T22:25:41.161700+0000","last_became_active":"2021-03-15T22:25:39.143407+0000","last_became_peered":"2021-03-15T22:25:39.143407+0000","last_unstale":"2021-03-15T22:25:41.161700+0000","last_undegraded":"2021-03-15T22:25:41.161700+0000","last_fullsized":"2021-03-15T22:25:41.161700+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'343","last_scrub_stamp":"2021-03-15T22:23:26.791043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.791043+0000","log_size":378,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7647416,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":183,"num_read_kb":156184,"num_write":134,"num_write_kb":53144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":38892785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"}]},{"pgid":"6.e","version":"0'0","reported_seq":"386","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114917+0000","last_change":"2021-03-15T22:24:40.910630+0000","last_active":"2021-03-15T22:25:40.114917+0000","last_peered":"2021-03-15T22:25:40.114917+0000","last_clean":"2021-03-15T22:25:40.114917+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:25:40.114917+0000","last_undegraded":"2021-03-15T22:25:40.114917+0000","last_fullsized":"2021-03-15T22:25:40.114917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.910565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.910565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"155'4","reported_seq":"504","reported_epoch":"423","state":"active+clean","last_fresh":"2021-03-15T22:25:38.164815+0000","last_change":"2021-03-15T22:24:29.903505+0000","last_active":"2021-03-15T22:25:38.164815+0000","last_peered":"2021-03-15T22:25:38.164815+0000","last_clean":"2021-03-15T22:25:38.164815+0000","last_became_active":"2021-03-15T22:21:10.013838+0000","last_became_peered":"2021-03-15T22:21:10.013838+0000","last_unstale":"2021-03-15T22:25:38.164815+0000","last_undegraded":"2021-03-15T22:25:38.164815+0000","last_fullsized":"2021-03-15T22:25:38.164815+0000","mapping_epoch":60,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'4","last_scrub_stamp":"2021-03-15T22:24:29.903442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.903442+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"104","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.102345+0000","last_change":"2021-03-15T22:25:02.791176+0000","last_active":"2021-03-15T22:25:40.102345+0000","last_peered":"2021-03-15T22:25:40.102345+0000","last_clean":"2021-03-15T22:25:40.102345+0000","last_became_active":"2021-03-15T22:24:31.097434+0000","last_became_peered":"2021-03-15T22:24:31.097434+0000","last_unstale":"2021-03-15T22:25:40.102345+0000","last_undegraded":"2021-03-15T22:25:40.102345+0000","last_fullsized":"2021-03-15T22:25:40.102345+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.791108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:06.573181+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.791108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"467","reported_epoch":"425","state":"active+clean","last_fresh":"2021-03-15T22:25:40.114072+0000","last_change":"2021-03-15T22:25:34.027206+0000","last_active":"2021-03-15T22:25:40.114072+0000","last_peered":"2021-03-15T22:25:40.114072+0000","last_clean":"2021-03-15T22:25:40.114072+0000","last_became_active":"2021-03-15T22:22:56.053800+0000","last_became_peered":"2021-03-15T22:22:56.053800+0000","last_unstale":"2021-03-15T22:25:40.114072+0000","last_undegraded":"2021-03-15T22:25:40.114072+0000","last_fullsized":"2021-03-15T22:25:40.114072+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:23:05.791845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:05.791845+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":353191888,"num_objects":174,"num_object_clones":125,"num_object_copies":522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":26,"num_objects_unfound":0,"num_objects_dirty":174,"num_whiteouts":13,"num_read":3045,"num_read_kb":3058174,"num_write":2223,"num_write_kb":839906,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1126,"num_bytes_recovered":460466498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":828796928,"data_stored":1140417107,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6007,"ondisk_log_size":6007,"up":48,"acting":47,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":451,"num_read_kb":446,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1061,"num_read_kb":566,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":37148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2232320,"data_stored":80092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299489,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":119412,"kb_used_data":101484,"kb_used_omap":0,"kb_used_meta":17920,"kb_avail":93625740,"statfs":{"total":95995035648,"available":95872757760,"internally_reserved":0,"allocated":103919616,"data_stored":139989977,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18350080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.0189999999999999,"5min":2.73,"15min":2.73},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65000000000000002,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.53700000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.504,"15min":0.48599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":0.97299999999999998,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.48499999999999999,"15min":0.48999999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72699999999999998,"5min":1.877,"15min":1.877},"last":0.47999999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.47199999999999998,"15min":0.47899999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73499999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.51900000000000002,"15min":0.5},"min":{"1min":0.311,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":0.95099999999999996,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.59199999999999997}]},{"osd":3,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.54200000000000004},"min":{"1min":0.315,"5min":0.31,"15min":0.31},"max":{"1min":0.69499999999999995,"5min":0.95999999999999996,"15min":2.0339999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98899999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.52300000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.46600000000000003,"15min":0.436},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.75700000000000001,"5min":1.835,"15min":1.835},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47699999999999998,"15min":0.44600000000000001},"min":{"1min":0.29099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66200000000000003,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.35399999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52200000000000002,"15min":0.45800000000000002},"min":{"1min":0.31900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.034,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.53100000000000003,"15min":0.504},"min":{"1min":0.33900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.78300000000000003,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.37}]},{"osd":6,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54600000000000004,"15min":0.51000000000000001},"min":{"1min":0.34499999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80700000000000005,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.53700000000000003,"15min":0.50900000000000001},"min":{"1min":0.35299999999999998,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.77700000000000002,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.40400000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528421,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":115724,"kb_used_data":88004,"kb_used_omap":0,"kb_used_meta":27712,"kb_avail":93629428,"statfs":{"total":95995035648,"available":95876534272,"internally_reserved":0,"allocated":90116096,"data_stored":115404586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28377088},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48899999999999999,"15min":0.49099999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.71399999999999997,"5min":0.995,"15min":0.995},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.47999999999999998,"15min":0.48899999999999999},"min":{"1min":0.29299999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.63100000000000001,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.41099999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.505,"15min":0.51100000000000001},"min":{"1min":0.28100000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.67800000000000005,"5min":1.05,"15min":1.05},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.495,"15min":0.48499999999999999},"min":{"1min":0.32600000000000001,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.65300000000000002,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.33300000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.502},"min":{"1min":0.32300000000000001,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.69499999999999995,"5min":0.80800000000000005,"15min":0.80800000000000005},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.5,"5min":0.495,"15min":0.5},"min":{"1min":0.33100000000000002,"5min":0.32200000000000001,"15min":0.31900000000000001},"max":{"1min":1.1419999999999999,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.45300000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51300000000000001,"15min":0.52500000000000002},"min":{"1min":0.32000000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.76500000000000001,"5min":1.026,"15min":1.026},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.51600000000000001,"15min":0.52500000000000002},"min":{"1min":0.38700000000000001,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.69999999999999996,"5min":0.77800000000000002,"15min":0.77800000000000002},"last":0.497}]},{"osd":4,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.44900000000000001,"15min":0.41299999999999998},"min":{"1min":0.31,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.70099999999999996,"5min":0.70099999999999996,"15min":0.70099999999999996},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.50800000000000001,"15min":0.45500000000000002},"min":{"1min":0.315,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.82799999999999996,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.74299999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.499,"15min":0.45600000000000002},"min":{"1min":0.31,"5min":0.309,"15min":0.309},"max":{"1min":0.71599999999999997,"5min":1.135,"15min":1.135},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.53000000000000003,"15min":0.48399999999999999},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.84999999999999998,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52300000000000002,"15min":0.50600000000000001},"min":{"1min":0.36799999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.79600000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.76100000000000001,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.72499999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574966,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147276,"kb_used_data":122692,"kb_used_omap":0,"kb_used_meta":24576,"kb_avail":93597876,"statfs":{"total":95995035648,"available":95844225024,"internally_reserved":0,"allocated":125636608,"data_stored":179150765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25165824},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.34499999999999997,"15min":0.52700000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.48899999999999999,"5min":1.371,"15min":54.427999999999997},"last":0.23499999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.54400000000000004},"min":{"1min":0.20999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.48199999999999998,"5min":1.23,"15min":54.356999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.41599999999999998,"15min":0.41699999999999998},"min":{"1min":0.23899999999999999,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.84699999999999998,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.314},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.33400000000000002,"15min":0.39100000000000001},"min":{"1min":0.25600000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":2.0760000000000001},"last":0.49099999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:24:57 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38500000000000001,"15min":0.41899999999999998},"min":{"1min":0.26100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.796},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40799999999999997,"15min":0.46400000000000002},"min":{"1min":0.29299999999999998,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.623,"5min":1.135,"15min":1.7150000000000001},"last":0.376}]},{"osd":4,"last update":"Mon Mar 15 22:25:10 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.95199999999999996,"5min":3.3439999999999999,"15min":3.3439999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.621,"15min":0.626},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.93999999999999995,"5min":4.6139999999999999,"15min":4.6139999999999999},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.627},"min":{"1min":0.42199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.93400000000000005,"5min":1.524,"15min":2.153},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58499999999999996,"15min":0.95899999999999996},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88900000000000001,"5min":1.1950000000000001,"15min":15.472},"last":0.64100000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.61299999999999999,"5min":0.60799999999999998,"15min":0.66200000000000003},"min":{"1min":0.41099999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.99199999999999999,"5min":1.276,"15min":1.75},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58199999999999996,"15min":0.627},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.98299999999999998,"5min":0.98299999999999998,"15min":1.3580000000000001},"last":0.57999999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.63500000000000001,"15min":0.871},"min":{"1min":0.377,"5min":0.316,"15min":0.316},"max":{"1min":0.96299999999999997,"5min":2.1219999999999999,"15min":8.8100000000000005},"last":0.67700000000000005},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.623},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84799999999999998,"5min":1.0329999999999999,"15min":1.325},"last":0.55400000000000005}]}]},{"osd":0,"up_from":0,"seq":34359738489,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411443,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158540,"kb_used_data":132868,"kb_used_omap":0,"kb_used_meta":25664,"kb_avail":93586612,"statfs":{"total":95995035648,"available":95832690688,"internally_reserved":0,"allocated":136056832,"data_stored":191876735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26279936},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,1],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:24:41 2021","interfaces":[{"interface":"back","average":{"1min":0.26900000000000002,"5min":0.33500000000000002,"15min":0.35099999999999998},"min":{"1min":0.18099999999999999,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.39900000000000002,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.28799999999999998},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.376,"15min":0.379},"min":{"1min":0.23599999999999999,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":0.65900000000000003,"5min":1.8939999999999999,"15min":1.8939999999999999},"last":0.33200000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:24:41 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.42499999999999999,"15min":0.40000000000000002},"min":{"1min":0.22,"5min":0.22,"15min":0.22},"max":{"1min":0.38700000000000001,"5min":5.4489999999999998,"15min":5.4489999999999998},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.499,"15min":0.46500000000000002},"min":{"1min":0.23400000000000001,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":0.435,"5min":11.659000000000001,"15min":11.659000000000001},"last":0.372}]},{"osd":3,"last update":"Mon Mar 15 22:24:53 2021","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.38400000000000001,"15min":0.41799999999999998},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.55600000000000005,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.315},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.443,"15min":0.48299999999999998},"min":{"1min":0.27400000000000002,"5min":0.26400000000000001,"15min":0.24099999999999999},"max":{"1min":2.8759999999999999,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.40400000000000003}]},{"osd":4,"last update":"Mon Mar 15 22:25:09 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56999999999999995,"15min":0.57499999999999996},"min":{"1min":0.40999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73899999999999999,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.432},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54500000000000004,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.60199999999999998,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.45000000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:25:36 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.38400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75600000000000001,"5min":1.954,"15min":1.954},"last":0.502},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.745,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.46600000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:24:41 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59099999999999997,"15min":0.57099999999999995},"min":{"1min":0.35999999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.748,"5min":1.875,"15min":1.875},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.58299999999999996,"15min":0.57099999999999995},"min":{"1min":0.373,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.75700000000000001,"5min":1.609,"15min":1.609},"last":0.51600000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:25:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.66900000000000004,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.40600000000000003,"15min":0.38700000000000001},"max":{"1min":0.64700000000000002,"5min":1.923,"15min":1.923},"last":0.59799999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149807,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":199516,"kb_used_data":178900,"kb_used_omap":0,"kb_used_meta":20608,"kb_avail":93545636,"statfs":{"total":95995035648,"available":95790731264,"internally_reserved":0,"allocated":183193600,"data_stored":231572211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21102592},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.35199999999999998,"15min":0.35799999999999998},"min":{"1min":0.22700000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.35699999999999998,"5min":2.625,"15min":2.625},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.36499999999999999,"15min":0.379},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.189},"max":{"1min":0.438,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.23499999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.44900000000000001,"15min":0.434},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.50700000000000001,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.29399999999999998},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.46100000000000002,"15min":0.437},"min":{"1min":0.28699999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.45300000000000001,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.32900000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.45000000000000001,"15min":0.435},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.48199999999999998,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.316,"5min":0.42299999999999999,"15min":0.40400000000000003},"min":{"1min":0.26600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46300000000000002,"5min":7.0860000000000003,"15min":7.0860000000000003},"last":0.35099999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:25:10 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55500000000000005,"15min":0.52400000000000002},"min":{"1min":0.32600000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.66800000000000004,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.318,"15min":0.30399999999999999},"max":{"1min":0.69299999999999995,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.34399999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.54500000000000004},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.62,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":0.52800000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:24:41 2021","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.59999999999999998,"15min":0.59799999999999998},"min":{"1min":0.42699999999999999,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.73499999999999999,"5min":2.6499999999999999,"15min":2.6499999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.57199999999999995,"15min":0.58299999999999996},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":1.268,"5min":1.268,"15min":1.268},"last":0.44}]},{"osd":7,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.58399999999999996,"15min":0.57699999999999996},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67400000000000004,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.377},"max":{"1min":0.72699999999999998,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.41399999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986284,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":97728,"kb_used_data":69816,"kb_used_omap":0,"kb_used_meta":27904,"kb_avail":93647424,"statfs":{"total":95995035648,"available":95894962176,"internally_reserved":0,"allocated":71491584,"data_stored":95252067,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28573696},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.19500000000000001},"max":{"1min":0.73099999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48999999999999999},"min":{"1min":0.34999999999999998,"5min":0.29099999999999998,"15min":0.251},"max":{"1min":0.66600000000000004,"5min":0.72199999999999998,"15min":0.74199999999999999},"last":0.59699999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.47599999999999998,"15min":0.5},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23599999999999999},"max":{"1min":0.68100000000000005,"5min":1.631,"15min":1.631},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26800000000000002},"max":{"1min":0.73999999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.58799999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.51100000000000001},"min":{"1min":0.40300000000000002,"5min":0.32300000000000001,"15min":0.313},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.76100000000000001},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":1.1910000000000001},"last":0.43099999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52600000000000002,"15min":0.55000000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68899999999999995,"5min":0.75,"15min":0.98799999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51500000000000001,"15min":0.55800000000000005},"min":{"1min":0.40200000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68899999999999995,"5min":0.69199999999999995,"15min":0.99399999999999999},"last":0.57599999999999996}]},{"osd":5,"last update":"Mon Mar 15 22:25:28 2021","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.41799999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.248},{"interface":"front","average":{"1min":0.502,"5min":0.49299999999999999,"15min":0.441},"min":{"1min":0.36099999999999999,"5min":0.29699999999999999,"15min":0.28799999999999998},"max":{"1min":0.64400000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.307}]},{"osd":6,"last update":"Mon Mar 15 22:24:45 2021","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.46700000000000003,"15min":0.432},"min":{"1min":0.29999999999999999,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.57499999999999996,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.55100000000000005,"15min":0.51100000000000001},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.34300000000000003},"max":{"1min":0.73899999999999999,"5min":0.78400000000000003,"15min":0.84799999999999998},"last":0.32100000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.48899999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.70899999999999996,"5min":0.73799999999999999,"15min":0.80600000000000005},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55400000000000005,"15min":0.52500000000000002},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.75,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.33500000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757352,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":155064,"kb_used_data":136112,"kb_used_omap":0,"kb_used_meta":18944,"kb_avail":93590088,"statfs":{"total":95995035648,"available":95836250112,"internally_reserved":0,"allocated":139378688,"data_stored":196800433,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19398656},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,4],"upper_bound":1024},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48799999999999999,"15min":0.48299999999999998},"min":{"1min":0.29499999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63500000000000001,"5min":2.855,"15min":2.855},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.498},"min":{"1min":0.34699999999999998,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.75700000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.374}]},{"osd":1,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.33100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.72999999999999998,"5min":2.665,"15min":2.665},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.72099999999999997,"5min":0.81999999999999995,"15min":0.83999999999999997},"last":0.315}]},{"osd":2,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.39000000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61799999999999999,"5min":2.827,"15min":2.827},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.51400000000000001},"min":{"1min":0.29899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.0740000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.55100000000000005,"15min":0.51800000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54800000000000004,"15min":0.53300000000000003},"min":{"1min":0.39400000000000002,"5min":0.34100000000000003,"15min":0.33200000000000002},"max":{"1min":0.65500000000000003,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.38500000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46999999999999997,"15min":0.40899999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24099999999999999},"max":{"1min":0.70899999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.245},{"interface":"front","average":{"1min":0.496,"5min":0.505,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.25},"max":{"1min":0.748,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.27800000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:24:44 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.50700000000000001,"15min":0.46899999999999997},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.28499999999999998},"max":{"1min":0.751,"5min":0.76500000000000001,"15min":0.76500000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.57799999999999996,"15min":0.53300000000000003},"min":{"1min":0.378,"5min":0.378,"15min":0.32800000000000001},"max":{"1min":0.86199999999999999,"5min":1.327,"15min":1.327},"last":0.40400000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.34699999999999998,"15min":0.30499999999999999},"max":{"1min":0.872,"5min":1.3,"15min":1.3},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57699999999999996,"15min":0.53500000000000003},"min":{"1min":0.38,"5min":0.307,"15min":0.307},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.39700000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":9520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":14458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":8356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":6840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":8774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5443584,"data_stored":5395029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121090048,"data_stored":176357670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":131670016,"data_stored":189073081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":183283712,"data_stored":233351811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":67043328,"data_stored":92438987,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":135057408,"data_stored":194014922,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":85688320,"data_stored":112591365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":99520512,"data_stored":137194242,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:25:44.206 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph status --format=json 2021-03-15T22:25:44.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:25:44.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:44.252 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:44.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:44.253 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:44.269 INFO:tasks.rados.rados.0.smithi076.stdout:1581: finishing write tid 3 to smithi07694878-49 2021-03-15T22:25:44.270 INFO:tasks.rados.rados.0.smithi076.stdout:1581: finishing write tid 4 to smithi07694878-49 2021-03-15T22:25:44.270 INFO:tasks.rados.rados.0.smithi076.stdout:1581: finishing write tid 5 to smithi07694878-49 2021-03-15T22:25:44.270 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 932 (ObjNum 555 snap 176 seq_num 555) dirty exists 2021-03-15T22:25:44.270 INFO:tasks.rados.rados.0.smithi076.stdout:1581: left oid 49 (ObjNum 555 snap 176 seq_num 555) 2021-03-15T22:25:44.354 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:44.369 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:44 smithi125 conmon[56739]: cluster 2021-03-15T22:25:43.057293+0000 mgr.y (mgr.14144) 627 : cluster [DBG] pgmap v768: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 137 active+clean; 342 MiB data, 987 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 27 op/s; 6/1476 objects degraded (0.407%); 7/1476 objects misplaced (0.474%); 22 MiB/s, 40 objects/s recovering 2021-03-15T22:25:44.370 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:44 smithi125 conmon[56739]: cluster 2021-03-15T22:25:43.172258+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e428: 8 total, 8 up, 7 in 2021-03-15T22:25:44.377 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:44.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:44.378 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:44.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:44 smithi076 conmon[62156]: cluster 2021-03-15T22:25:43.057293 2021-03-15T22:25:44.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:44 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 627 : cluster [DBG] pgmap v768: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 137 active+clean; 342 MiB data, 987 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 27 op/s; 6/1476 objects degraded (0.407%); 7/1476 objects misplaced (0.474%); 22 MiB/s, 40 objects/s recovering 2021-03-15T22:25:44.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:44 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:25:44.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:44 smithi076 conmon[62156]: -03-15T22:25:43.172258+0000 mon.a (mon.0) 1252 : cluster [DBG] 2021-03-15T22:25:44.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:44 smithi076 conmon[62156]: osdmap e428: 8 total, 8 up, 7 in 2021-03-15T22:25:44.389 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:44 smithi076 conmon[56514]: cluster 2021-03-15T22:25:43.057293+0000 mgr.y 2021-03-15T22:25:44.389 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:44 smithi076 conmon[56514]: (mgr.14144) 627 : cluster [DBG] pgmap v768: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 137 active+clean; 342 MiB data, 987 MiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 6.0 MiB/s wr, 27 op/s; 6/1476 objects degraded (0.407%); 7/1476 objects misplaced (0.474%); 22 MiB/s, 40 objects/s recovering 2021-03-15T22:25:44.390 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:44 smithi076 conmon[56514]: cluster 2021-03-15T22:25:43.172258+0000 mon.a (mon.0) 1252 : cluster [DBG] osdmap e428: 8 total, 8 up, 7 in 2021-03-15T22:25:44.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:44.504 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:44.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:44.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:44.561 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:25:44.561 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:25:44.608 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:44.609 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:44.614 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:25:44.633 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:44.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:44.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:44.636 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:25:44.655 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:25:44.676 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:25:44.694 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:25:44.712 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:25:44.730 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:25:44.736 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:44.747 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:25:44.749 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:25:44.749 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:25:44.749 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:25:44.749 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:25:44.750 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:25:44.750 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:25:44.750 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:25:44.750 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:25:44.757 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:44.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:44.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:44.860 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:44.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:44.886 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:44.886 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:44.988 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:45.010 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:45.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:45.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:45.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:45.137 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:45.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:45.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:45.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:25:45.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:45.241 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: cluster 2021-03-15T22:25:45.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 2021-03-15T22:25:45.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 2021 2021-03-15T22:25:45.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: - 2021-03-15T22:25:45.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 03 2021-03-15T22:25:45.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: - 2021-03-15T22:25:45.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 15 2021-03-15T22:25:45.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: T 2021-03-15T22:25:45.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 22 2021-03-15T22:25:45.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: : 2021-03-15T22:25:45.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 25 2021-03-15T22:25:45.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: : 2021-03-15T22:25:45.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 44 2021-03-15T22:25:45.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: . 2021-03-15T22:25:45.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 175367 2021-03-15T22:25:45.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: +0000 2021-03-15T22:25:45.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 2021-03-15T22:25:45.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: mon.a 2021-03-15T22:25:45.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: ( 2021-03-15T22:25:45.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: mon 2021-03-15T22:25:45.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: . 2021-03-15T22:25:45.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 0 2021-03-15T22:25:45.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: ) 2021-03-15T22:25:45.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 1253 2021-03-15T22:25:45.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: : 2021-03-15T22:25:45.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: cluster 2021-03-15T22:25:45.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 2021-03-15T22:25:45.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: [DBG] 2021-03-15T22:25:45.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 2021-03-15T22:25:45.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: osdmap e429: 8 total, 8 up, 7 in 2021-03-15T22:25:45.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 2021-03-15T22:25:45.249 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: cluster 2021-03-15T22:25:45.250 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 2021-03-15T22:25:45.250 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 2021 2021-03-15T22:25:45.250 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: - 2021-03-15T22:25:45.250 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 03 2021-03-15T22:25:45.251 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: - 2021-03-15T22:25:45.251 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 15 2021-03-15T22:25:45.251 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: T 2021-03-15T22:25:45.251 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 22 2021-03-15T22:25:45.252 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: : 2021-03-15T22:25:45.252 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 25 2021-03-15T22:25:45.252 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: : 2021-03-15T22:25:45.252 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 44 2021-03-15T22:25:45.252 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: . 2021-03-15T22:25:45.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 175367 2021-03-15T22:25:45.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: +0000 2021-03-15T22:25:45.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 2021-03-15T22:25:45.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: mon.a 2021-03-15T22:25:45.253 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: ( 2021-03-15T22:25:45.254 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: mon 2021-03-15T22:25:45.254 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: . 2021-03-15T22:25:45.254 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 0 2021-03-15T22:25:45.254 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: ) 2021-03-15T22:25:45.255 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 1253 2021-03-15T22:25:45.255 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: : 2021-03-15T22:25:45.255 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: cluster 2021-03-15T22:25:45.255 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 2021-03-15T22:25:45.255 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: [DBG] 2021-03-15T22:25:45.256 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 2021-03-15T22:25:45.256 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: osdmap e429: 8 total, 8 up, 7 in 2021-03-15T22:25:45.256 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 2021-03-15T22:25:45.256 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: audit 2021-03-15T22:25:45.257 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 2021-03-15T22:25:45.257 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 2021 2021-03-15T22:25:45.257 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: - 2021-03-15T22:25:45.257 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 03 2021-03-15T22:25:45.257 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: - 2021-03-15T22:25:45.258 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 15 2021-03-15T22:25:45.258 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: T 2021-03-15T22:25:45.258 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 22 2021-03-15T22:25:45.258 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: : 2021-03-15T22:25:45.258 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 25 2021-03-15T22:25:45.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: : 2021-03-15T22:25:45.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 44 2021-03-15T22:25:45.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: . 2021-03-15T22:25:45.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 594880 2021-03-15T22:25:45.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: +0000 2021-03-15T22:25:45.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: 2021-03-15T22:25:45.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: mon.a (mon.0) 1254 : audit 2021-03-15T22:25:45.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/1716392036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:45.261 INFO:tasks.rados.rados.0.smithi076.stdout:1581: done (2 left) 2021-03-15T22:25:45.261 INFO:tasks.rados.rados.0.smithi076.stdout:1582: done (1 left) 2021-03-15T22:25:45.261 INFO:tasks.rados.rados.0.smithi076.stdout:1584: done (0 left) 2021-03-15T22:25:45.261 INFO:tasks.rados.rados.0.smithi076.stdout:1585: snap_remove snap 141 2021-03-15T22:25:45.271 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:45.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:45.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:45.373 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:45.395 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:45.397 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:45.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: audit 2021-03-15T22:25:45.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 2021-03-15T22:25:45.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 2021 2021-03-15T22:25:45.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: - 2021-03-15T22:25:45.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 03 2021-03-15T22:25:45.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: - 2021-03-15T22:25:45.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 15 2021-03-15T22:25:45.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: T 2021-03-15T22:25:45.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 22 2021-03-15T22:25:45.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: : 2021-03-15T22:25:45.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: 25 2021-03-15T22:25:45.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: :44.594880+0000 mon.a (mon.0 2021-03-15T22:25:45.493 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:45 smithi076 conmon[56514]: ) 1254 : audit [DBG] from='client.? 172.21.15.76:0/1716392036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:45.493 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:25:45.498 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:45.521 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:45.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:45.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:45.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:45 smithi125 conmon[56739]: cluster 2021-03-15T22:25:44.175367+0000 2021-03-15T22:25:45.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:45 smithi125 conmon[56739]: mon.a (mon.0) 1253 : cluster [DBG] osdmap e429: 8 total, 8 up, 7 in 2021-03-15T22:25:45.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:45 smithi125 conmon[56739]: audit 2021-03-15T22:25:44.594880+0000 mon.a (mon.0) 1254 : audit [DBG] from='client.? 172.21.15.76:0/1716392036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:45.624 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:45.647 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:45.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:45.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:45.750 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:45.774 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:45.775 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:45.876 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:45.898 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:45.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:45.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:46.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:46.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:46.028 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:46.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:46.130 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:46.153 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:46.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:46.245 INFO:tasks.rados.rados.0.smithi076.stdout:1585: done (0 left) 2021-03-15T22:25:46.245 INFO:tasks.rados.rados.0.smithi076.stdout:1586: write (excl) oid 33 current snap is 176 2021-03-15T22:25:46.246 INFO:tasks.rados.rados.0.smithi076.stdout:1586: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2021-03-15T22:25:46.248 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: cluster 2021-03-15T22:25:46.249 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: 2021-03-15T22:25:46.249 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: 2021-03-15T22:25:44.619078+0000 2021-03-15T22:25:46.249 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: osd.1 (osd.1) 85 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:25:46.249 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: cluster 2021-03-15T22:25:46.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: 2021-03-15T22:25:44. 2021-03-15T22:25:46.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: 867623+0000 osd.3 (osd.3) 75 2021-03-15T22:25:46.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: : cluster [DBG] 3.8 scrub ok 2021-03-15T22:25:46.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:25:46.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: 15T22:25:45.058055+0000 2021-03-15T22:25:46.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: mgr.y (mgr.14144) 628 : cluster 2021-03-15T22:25:46.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: [DBG] pgmap v771: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 137 active+clean; 344 MiB data, 996 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 5.1 MiB/s wr, 25 op/s; 6/1485 objects degraded (0.404%); 7/1485 objects misplaced (0.471%); 7.6 MiB/s, 15 objects/s recovering 2021-03-15T22:25:46.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:25:46.252 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: 03-15T22:25:45. 2021-03-15T22:25:46.252 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: 238539+0000 mon.a (mon.0) 1255 : 2021-03-15T22:25:46.252 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2021-03-15T22:25:46.253 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:25:46.253 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: 15T22:25:45.692540+0000 2021-03-15T22:25:46.254 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: osd.2 (osd.2) 72 : cluster [DBG] 2021-03-15T22:25:46.254 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:46 smithi125 conmon[56739]: 3.3 scrub ok 2021-03-15T22:25:46.254 INFO:tasks.rados.rados.0.smithi076.stdout:1586: writing smithi07694878-33 from 561499 to 968961 tid 1 2021-03-15T22:25:46.256 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:46.258 INFO:tasks.rados.rados.0.smithi076.stdout:1586: writing smithi07694878-33 from 1527882 to 2015403 tid 2 2021-03-15T22:25:46.266 INFO:tasks.rados.rados.0.smithi076.stdout:1586: writing smithi07694878-33 from 2729901 to 3287118 tid 3 2021-03-15T22:25:46.266 INFO:tasks.rados.rados.0.smithi076.stdout:1587: write (excl) oid 6 current snap is 176 2021-03-15T22:25:46.267 INFO:tasks.rados.rados.0.smithi076.stdout:1586: finishing write tid 1 to smithi07694878-33 2021-03-15T22:25:46.267 INFO:tasks.rados.rados.0.smithi076.stdout:1587: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2021-03-15T22:25:46.272 INFO:tasks.rados.rados.0.smithi076.stdout:1587: writing smithi07694878-6 from 578306 to 1061017 tid 1 2021-03-15T22:25:46.278 INFO:tasks.rados.rados.0.smithi076.stdout:1587: writing smithi07694878-6 from 1586364 to 2217543 tid 2 2021-03-15T22:25:46.282 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:46.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:46.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:46.285 INFO:tasks.rados.rados.0.smithi076.stdout:1587: writing smithi07694878-6 from 2757324 to 3242166 tid 3 2021-03-15T22:25:46.285 INFO:tasks.rados.rados.0.smithi076.stdout:1588: rollback oid 23 current snap is 176 2021-03-15T22:25:46.285 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 23 to 153 2021-03-15T22:25:46.286 INFO:tasks.rados.rados.0.smithi076.stdout:1586: finishing write tid 2 to smithi07694878-33 2021-03-15T22:25:46.286 INFO:tasks.rados.rados.0.smithi076.stdout:1586: finishing write tid 3 to smithi07694878-33 2021-03-15T22:25:46.286 INFO:tasks.rados.rados.0.smithi076.stdout:1586: finishing write tid 4 to smithi07694878-33 2021-03-15T22:25:46.286 INFO:tasks.rados.rados.0.smithi076.stdout:1586: finishing write tid 5 to smithi07694878-33 2021-03-15T22:25:46.287 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 699 (ObjNum 557 snap 176 seq_num 557) dirty exists 2021-03-15T22:25:46.287 INFO:tasks.rados.rados.0.smithi076.stdout:1586: left oid 33 (ObjNum 557 snap 176 seq_num 557) 2021-03-15T22:25:46.287 INFO:tasks.rados.rados.0.smithi076.stdout:1587: finishing write tid 1 to smithi07694878-6 2021-03-15T22:25:46.287 INFO:tasks.rados.rados.0.smithi076.stdout:1587: finishing write tid 2 to smithi07694878-6 2021-03-15T22:25:46.288 INFO:tasks.rados.rados.0.smithi076.stdout:1586: done (2 left) 2021-03-15T22:25:46.288 INFO:tasks.rados.rados.0.smithi076.stdout:1589: copy_from oid 20 from oid 48 current snap is 176 2021-03-15T22:25:46.288 INFO:tasks.rados.rados.0.smithi076.stdout:1590: delete oid 44 current snap is 176 2021-03-15T22:25:46.289 INFO:tasks.rados.rados.0.smithi076.stdout:1590: done (3 left) 2021-03-15T22:25:46.289 INFO:tasks.rados.rados.0.smithi076.stdout:1591: write oid 42 current snap is 176 2021-03-15T22:25:46.291 INFO:tasks.rados.rados.0.smithi076.stdout:1591: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2021-03-15T22:25:46.294 INFO:tasks.rados.rados.0.smithi076.stdout:1591: writing smithi07694878-42 from 611920 to 1245129 tid 1 2021-03-15T22:25:46.300 INFO:tasks.rados.rados.0.smithi076.stdout:1591: writing smithi07694878-42 from 1703328 to 2338176 tid 2 2021-03-15T22:25:46.307 INFO:tasks.rados.rados.0.smithi076.stdout:1591: writing smithi07694878-42 from 2812170 to 3319556 tid 3 2021-03-15T22:25:46.307 INFO:tasks.rados.rados.0.smithi076.stdout:1592: snap_create 2021-03-15T22:25:46.308 INFO:tasks.rados.rados.0.smithi076.stdout:1588: finishing rollback tid 0 to smithi07694878-23 2021-03-15T22:25:46.308 INFO:tasks.rados.rados.0.smithi076.stdout:1587: finishing write tid 3 to smithi07694878-6 2021-03-15T22:25:46.308 INFO:tasks.rados.rados.0.smithi076.stdout:1588: finishing rollback tid 1 to smithi07694878-23 2021-03-15T22:25:46.309 INFO:tasks.rados.rados.0.smithi076.stdout:1587: finishing write tid 4 to smithi07694878-6 2021-03-15T22:25:46.309 INFO:tasks.rados.rados.0.smithi076.stdout:1588: finishing rollback tid 2 to smithi07694878-23 2021-03-15T22:25:46.310 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 773 (ObjNum 424 snap 117 seq_num 424) dirty exists 2021-03-15T22:25:46.310 INFO:tasks.rados.rados.0.smithi076.stdout:1587: finishing write tid 5 to smithi07694878-6 2021-03-15T22:25:46.311 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 766 (ObjNum 558 snap 176 seq_num 558) dirty exists 2021-03-15T22:25:46.311 INFO:tasks.rados.rados.0.smithi076.stdout:1587: left oid 6 (ObjNum 558 snap 176 seq_num 558) 2021-03-15T22:25:46.312 INFO:tasks.rados.rados.0.smithi076.stdout:1591: finishing write tid 1 to smithi07694878-42 2021-03-15T22:25:46.312 INFO:tasks.rados.rados.0.smithi076.stdout:1589: finishing copy_from to smithi07694878-20 2021-03-15T22:25:46.312 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 766 (ObjNum 554 snap 176 seq_num 554) dirty exists 2021-03-15T22:25:46.313 INFO:tasks.rados.rados.0.smithi076.stdout:1589: finishing copy_from racing read to smithi07694878-20 2021-03-15T22:25:46.313 INFO:tasks.rados.rados.0.smithi076.stdout:1591: finishing write tid 2 to smithi07694878-42 2021-03-15T22:25:46.315 INFO:tasks.rados.rados.0.smithi076.stdout:1591: finishing write tid 3 to smithi07694878-42 2021-03-15T22:25:46.315 INFO:tasks.rados.rados.0.smithi076.stdout:1591: finishing write tid 4 to smithi07694878-42 2021-03-15T22:25:46.315 INFO:tasks.rados.rados.0.smithi076.stdout:1591: finishing write tid 5 to smithi07694878-42 2021-03-15T22:25:46.316 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 998 (ObjNum 560 snap 176 seq_num 560) dirty exists 2021-03-15T22:25:46.316 INFO:tasks.rados.rados.0.smithi076.stdout:1591: left oid 42 (ObjNum 560 snap 176 seq_num 560) 2021-03-15T22:25:46.385 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:46.407 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:46.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:46.511 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:46.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[56514]: cluster 2021-03-15T22:25:44.619078+0000 osd.1 ( 2021-03-15T22:25:46.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[56514]: osd.1) 85 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:25:46.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[56514]: cluster 2021-03-15T22:25:44.867623+0000 osd.3 (osd.3) 75 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:25:46.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[56514]: cluster 2021-03-15T22:25:45.058055+0000 mgr.y (mgr.14144) 628 : cluster [DBG] pgmap v771: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 137 active+clean; 344 MiB data, 996 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 5.1 MiB/s wr, 25 op/s; 6/1485 objects degraded (0.404%); 7/1485 objects misplaced (0.471%); 7.6 MiB/s, 15 objects/s recovering 2021-03-15T22:25:46.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[56514]: cluster 2021-03-15T22:25:45.238539+0000 mon.a (mon.0) 1255 : cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2021-03-15T22:25:46.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[56514]: cluster 2021-03-15T22:25:45.692540+0000 osd.2 (osd.2) 72 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:25:46.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: cluster 2021-03-15T22:25:46.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:25:46.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: 25:44.619078+0000 osd.1 (osd.1) 85 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:25:46.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: cluster 2021-03-15T22:25:46.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: 2021-03-15T22:25:44. 2021-03-15T22:25:46.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: 867623+0000 osd.3 (osd.3) 75 : cluster 2021-03-15T22:25:46.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: [DBG] 3.8 scrub ok 2021-03-15T22:25:46.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:25:46.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: 03-15T22:25:45.058055 2021-03-15T22:25:46.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 628 : cluster 2021-03-15T22:25:46.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: [DBG] pgmap v771: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 137 active+clean; 344 MiB data, 996 MiB used, 625 GiB / 626 GiB avail; 23 MiB/s rd, 5.1 MiB/s wr, 25 op/s; 6/1485 objects degraded (0.404%); 7/1485 objects misplaced (0.471%); 7.6 MiB/s, 15 objects/s recovering 2021-03-15T22:25:46.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:25:46.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: 22:25:45.238539+0000 mon.a ( 2021-03-15T22:25:46.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: mon.0) 1255 : cluster [DBG] osdmap e430: 8 total, 8 up, 7 in 2021-03-15T22:25:46.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: 2021-03-15T22:25:46.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: cluster 2021-03-15T22:25:46.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: 2021-03-15T22:25:45.692540+0000 osd.2 2021-03-15T22:25:46.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: (osd.2) 72 : cluster [DBG] 2021-03-15T22:25:46.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:46 smithi076 conmon[62156]: 3.3 scrub ok 2021-03-15T22:25:46.533 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:46.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:46.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:46.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:46.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:46.660 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:46.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:46.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:46.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:46.787 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:46.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:46.889 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:46.911 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:46.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:46.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:47.009 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:25:47.009 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:25:47.009 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:25:47.010 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:25:47.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:47.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:47.038 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:47.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:47.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:47.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:47.164 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:47.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:47.263 INFO:tasks.rados.rados.0.smithi076.stdout:1587: done (4 left) 2021-03-15T22:25:47.263 INFO:tasks.rados.rados.0.smithi076.stdout:1588: done (3 left) 2021-03-15T22:25:47.263 INFO:tasks.rados.rados.0.smithi076.stdout:1589: done (2 left) 2021-03-15T22:25:47.264 INFO:tasks.rados.rados.0.smithi076.stdout:1591: done (1 left) 2021-03-15T22:25:47.264 INFO:tasks.rados.rados.0.smithi076.stdout:1592: done (0 left) 2021-03-15T22:25:47.264 INFO:tasks.rados.rados.0.smithi076.stdout:1593: snap_remove snap 169 2021-03-15T22:25:47.266 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:47.288 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:47.291 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:47.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: cluster 2021-03-15T22:25:47.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:25:47.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: 25:46.103500+0000 osd.4 (osd.4) 92 : cluster [DBG] 5.e scrub ok 2021-03-15T22:25:47.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:25:47.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: 15T22:25:46.239754+0000 mon.a (mon.0) 1256 : cluster 2021-03-15T22:25:47.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: [DBG] osdmap e431: 8 total, 8 up, 7 in 2021-03-15T22:25:47.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: cluster 2021-03-15T22:25:47.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: 2021-03-15T22:25:46.725485+0000 osd.2 ( 2021-03-15T22:25:47.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: osd.2) 73 : cluster [DBG] 4.f scrub ok 2021-03-15T22:25:47.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: audit 2021-03-15T22:25:47.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: 2021-03-15T22:25:47.159560+0000 mon.a 2021-03-15T22:25:47.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56514]: (mon.0) 1257 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:25:47.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: cluster 2021-03-15T22:25:47.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: 2021-03-15T22:25:47.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: 2021 2021-03-15T22:25:47.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: - 2021-03-15T22:25:47.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: 03-15T22:25: 2021-03-15T22:25:47.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: 46.103500+0000 osd.4 (osd.4) 92 : cluster [DBG] 5.e scrub ok 2021-03-15T22:25:47.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: cluster 2021-03-15T22:25:47.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: 2021-03 2021-03-15T22:25:47.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: -15T22:25:46.239754+0000 mon.a (mon.0 2021-03-15T22:25:47.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: ) 1256 : cluster [DBG] osdmap e431: 8 total, 8 up, 7 in 2021-03-15T22:25:47.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:25:47.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: 03-15T22:25:46.725485+0000 osd.2 (osd. 2021-03-15T22:25:47.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: 2) 73 : cluster [DBG] 4.f scrub ok 2021-03-15T22:25:47.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: audit 2021-03-15T22:25:47.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: 2021-03-15T22:25:47.159560+0000 mon.a (mon. 2021-03-15T22:25:47.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[62156]: 0) 1257 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:25:47.392 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:47.418 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:47.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:47.419 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:47.520 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:47.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:47 smithi125 conmon[56739]: cluster 2021-03-15T22:25:46.103500+0000 osd.4 (osd.4) 92 : cluster [DBG] 5.e scrub ok 2021-03-15T22:25:47.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:47 smithi125 conmon[56739]: cluster 2021-03-15T22:25:46.239754+0000 mon.a (mon.0) 1256 : cluster [DBG] osdmap e431: 8 total, 8 up, 7 in 2021-03-15T22:25:47.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:47 smithi125 conmon[56739]: cluster 2021-03-15T22:25:46.725485+0000 osd.2 (osd.2) 73 : cluster [DBG] 4.f scrub ok 2021-03-15T22:25:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:47 smithi125 conmon[56739]: audit 2021-03-15T22:25:47.159560+0000 mon.a (mon.0) 1257 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:25:47.542 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:47.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:47.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:47.605 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:25:47.605 INFO:teuthology.orchestra.run.smithi076.stdout:{"fsid":"8a572802-85db-11eb-9091-001a4aab830c","health":{"status":"HEALTH_WARN","checks":{"PG_DEGRADED":{"severity":"HEALTH_WARN","summary":{"message":"Degraded data redundancy: 6/1503 objects degraded (0.399%), 1 pg degraded","count":1},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":627,"monmap":{"epoch":3,"min_mon_release_name":"quincy","num_mons":3},"osdmap":{"epoch":432,"num_osds":8,"num_up_osds":8,"osd_up_since":1615846669,"num_in_osds":7,"osd_in_since":1615847070,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":137},{"state_name":"active+recovering+undersized+degraded+remapped","count":1}],"num_pgs":138,"num_pools":7,"num_objects":501,"data_bytes":377679116,"bytes_used":1120923648,"bytes_avail":670844325888,"bytes_total":671965249536,"degraded_objects":6,"degraded_total":1503,"degraded_ratio":0.0039920159680638719,"misplaced_objects":7,"misplaced_total":1503,"misplaced_ratio":0.0046573519627411842,"read_bytes_sec":10807073,"write_bytes_sec":3847513,"read_op_per_sec":10,"write_op_per_sec":8},"fsmap":{"epoch":1,"by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","restful"],"services":{"dashboard":"https://ceph-8a572802-85db-11eb-9091-001a4aab830c-mgr.y:8443/"}},"servicemap":{"epoch":14,"modified":"2021-03-15T22:18:50.956625+0000","services":{"rgw":{"daemons":{"summary":"","foo.a":{"start_epoch":11,"start_stamp":"2021-03-15T22:18:02.091371+0000","gid":14427,"addr":"172.21.15.76:0/2912955800","metadata":{"arch":"x86_64","ceph_release":"quincy","ceph_version":"ceph version 17.0.0-2063-gb658018a (b658018acf46e32d2ec283d302076352e17476f5) quincy (dev)","ceph_version_short":"17.0.0-2063-gb658018a","container_hostname":"smithi076","container_image":"quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Linux 8","distro_version":"8","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi076","kernel_description":"#1 SMP Mon Mar 1 17:16:16 UTC 2021","kernel_version":"4.18.0-240.15.1.el8_3.x86_64","mem_swap_kb":"0","mem_total_kb":"32612052","num_handles":"1","os":"Linux","pid":"2","zone_id":"38e1d1be-4dea-4a9a-b9e9-57f557eecf75","zone_name":"default","zonegroup_id":"9cdd70dc-2472-46c3-b39c-c4226f951217","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2021-03-15T22:25:47.645 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:47.671 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:47.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:47.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:47.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-15T22:25:47.774 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:47.800 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:47.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:47.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:47.902 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:47.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:47.927 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:47.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:48.012 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:25:48.017 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: debug 2021-03-15T22:25:47.692+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:25:48.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:25:48.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:25:48.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:25:48.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:25:48.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: debug 2021-03-15T22:25:47.693+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:25:48.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:25:48.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:25:48.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:25:48.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:25:48.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: debug 2021-03-15T22:25:47.693+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:25:48.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:25:48.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:25:48.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:25:48.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:25:48.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: debug 2021-03-15T22:25:47.694+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:25:48.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:25:48.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.028 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:25:48.028 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:25:48.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.030 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:25:48.030 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: debug 2021-03-15T22:25:47.694+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:25:48.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:25:48.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:25:48.032 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.032 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:25:48.033 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.033 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.034 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:25:48.034 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: debug 2021-03-15T22:25:47.695+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:25:48.035 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:25:48.035 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.035 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:25:48.035 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.037 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:25:48.038 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.039 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.039 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:25:48.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: debug 2021-03-15T22:25:47.695+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:25:48.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:25:48.040 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:25:48.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:25:48.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.041 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.042 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:25:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:25:48.042 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:48.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:48.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:48.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:48.191 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:48.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:48.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:48.274 INFO:tasks.rados.rados.0.smithi076.stdout:1593: done (0 left) 2021-03-15T22:25:48.274 INFO:tasks.rados.rados.0.smithi076.stdout:1594: snap_remove snap 168 2021-03-15T22:25:48.281 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:25:48.294 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:48.317 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:48.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:48.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:48.420 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:48.442 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:48.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:48.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:48.450 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cluster 2021-03-15T22:25:46.813600+0000 osd.3 (osd.3) 76 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:25:48.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:25:48.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: 22:25:47.058617+0000 mgr.y (mgr.14144) 629 : cluster [DBG] pgmap v774: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 137 active+clean; 360 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 19 op/s; 6/1503 objects degraded (0.399%); 7/1503 objects misplaced (0.466%) 2021-03-15T22:25:48.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cluster 2021-03-15T22:25:47.117248+0000 osd.4 (osd.4) 93 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:25:48.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cluster 2021-03-15T22:25:47.244456+0000 mon.a (mon.0) 1258 : cluster [WRN] Health check update: Degraded data redundancy: 6/1503 objects degraded (0.399%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:25:48.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cluster 2021-03-15T22:25:47.259875+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2021-03-15T22:25:48.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: audit 2021-03-15T22:25:47.412409+0000 mon.a (mon.0) 1260 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:25:48.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: audit 2021-03-15T22:25:47.603072+0000 mon.a (mon.0) 1261 : audit [DBG] from='client.? 172.21.15.76:0/2491792331' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2021-03-15T22:25:48.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:25:47.689479+0000 mgr.y (mgr.14144) 630 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:25:48.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: audit 2021-03-15T22:25:47.692722+0000 mon.a (mon.0) 1262 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:25:48.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:25:47.693647+0000 mgr.y (mgr.14144) 631 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:25:48.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:25:48.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:25:48.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:25:48.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:25:48.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:25:47.694063+0000 mgr.y (mgr.14144) 632 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:25:48.456 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:25:48.456 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.456 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:25:48.456 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:25:48.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:25:48.457 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:25:47.694545+0000 mgr.y (mgr.14144) 633 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:25:48.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:25:48.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:25:48.458 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.459 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:25:48.459 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.459 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.459 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:25:48.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:25:47.694966+0000 mgr.y (mgr.14144) 634 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:25:48.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:25:48.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:25:48.460 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:25:48.461 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:25:48.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:25:47.695352+0000 mgr.y (mgr.14144) 635 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:25:48.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:25:48.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:25:48.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:25:48.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:25:48.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:25:47.695728+0000 mgr.y (mgr.14144) 636 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:25:48.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:25:48.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:25:48.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:25:48.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.466 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.466 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:25:48.466 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:25:47.696096+0000 mgr.y (mgr.14144) 637 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:25:48.466 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:25:48.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:25:48.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.467 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:25:48.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:25:48.468 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:25:48.469 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: 15T22:25:47.753809+0000 osd.5 (osd.5) 76 : cluster [DBG] 3.c scrub ok 2021-03-15T22:25:48.469 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: audit 2021-03-15T22:25:47.860422+0000 mon.c (mon.1) 38 : audit [INF] from='client.? 172.21.15.76:0/609453207' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:48.469 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:48 smithi125 conmon[56739]: audit 2021-03-15T22:25:47.860726+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cluster 2021-03-15T22:25:46.813600+0000 2021-03-15T22:25:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: osd.3 (osd.3) 76 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:25:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cluster 2021-03-15T22:25:47.058617+0000 mgr.y (mgr.14144) 629 : cluster [DBG] pgmap v774: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 137 active+clean; 360 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 19 op/s; 6/1503 objects degraded (0.399%); 7/1503 objects misplaced (0.466%) 2021-03-15T22:25:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cluster 2021-03-15T22:25:47.117248+0000 2021-03-15T22:25:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: osd.4 (osd.4) 93 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:25:48.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:25:48.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: -15T22:25:47.244456+0000 mon.a (mon.0) 1258 : 2021-03-15T22:25:48.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cluster [WRN] Health check update: Degraded data redundancy: 6/1503 objects degraded (0.399%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:25:48.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cluster 2021-03-15T22:25:47.259875+0000 2021-03-15T22:25:48.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: mon.a (mon.0) 1259 : cluster [DBG] 2021-03-15T22:25:48.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: osdmap e432: 8 total, 8 up, 7 in 2021-03-15T22:25:48.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: audit 2021-03-15T22:25: 2021-03-15T22:25:48.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: 47.412409+0000 mon.a (mon.0) 1260 : audit [INF] 2021-03-15T22:25:48.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:25:48.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: audit 2021-03-15T22:25:48.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: 2021-03-15T22:25:47.603072+0000 mon.a (mon.0) 1261 : audit 2021-03-15T22:25:48.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: [DBG] from='client.? 172.21.15.76:0/2491792331' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2021-03-15T22:25:48.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:25:47.689479+0000 mgr.y 2021-03-15T22:25:48.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: (mgr.14144) 630 : cephadm [INF] 2021-03-15T22:25:48.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:25:48.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: audit 2021-03-15T22:25:47.692722+0000 2021-03-15T22:25:48.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: mon.a (mon.0) 1262 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:25:48.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cephadm 2021-03 2021-03-15T22:25:48.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: -15T22:25:47.693647+0000 mgr.y (mgr.14144) 631 2021-03-15T22:25:48.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:25:48.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:25:48.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:25:48.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:25:48.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:25:48.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cluster 2021-03-15T22:25:46 2021-03-15T22:25:48.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: .813600+0000 osd.3 (osd.3) 76 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:25:48.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cluster 2021-03-15T22:25:48.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: 2021-03-15T22:25:47.058617+0000 2021-03-15T22:25:48.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: mgr.y (mgr.14144) 629 : cluster [DBG] pgmap v774: 138 pgs: 1 active+recovering+undersized+degraded+remapped, 137 active+clean; 360 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 3.7 MiB/s wr, 19 op/s; 6/1503 objects degraded (0.399%); 7/1503 objects misplaced (0.466%) 2021-03-15T22:25:48.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:25:48.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: 15T22:25:47.117248+0000 osd.4 (osd.4) 2021-03-15T22:25:48.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: 93 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:25:48.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:25:48.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: -03-15T22:25:47.244456+0000 mon.a 2021-03-15T22:25:48.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: (mon.0) 1258 : cluster [WRN] Health check update: Degraded data redundancy: 6/1503 objects degraded (0.399%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:25:48.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cluster 2021-03-15T22:25:47.259875+0000 mon.a (mon.0) 1259 : cluster [DBG] osdmap e432: 8 total, 8 up, 7 in 2021-03-15T22:25:48.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: audit 2021-03-15T22:25:47.412409+0000 mon.a (mon.0) 1260 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:25:48.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: audit 2021-03-15T22:25:48.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: 2021-03-15T22:25:47.603072+0000 mon.a (mon.0) 1261 : audit [DBG] from='client.? 172.21.15.76:0/2491792331' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2021-03-15T22:25:48.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cephadm 2021-03-15T22: 2021-03-15T22:25:48.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: 25:47.689479+0000 mgr.y (mgr.14144) 630 : cephadm [INF] 2021-03-15T22:25:48.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:25:48.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:25:48.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: -15T22:25:47.692722+0000 mon.a (mon.0) 1262 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:25:48.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cephadm 2021-03-15T22: 2021-03-15T22:25:48.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: 25:47.693647+0000 mgr.y (mgr.14144) 631 : cephadm [ERR] 2021-03-15T22:25:48.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:25:48.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:25:48.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:25:48.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:25:48.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:25:48.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cephadm 2021-03 2021-03-15T22:25:48.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:25:47.694063+0000 mgr.y (mgr.14144) 632 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:25:48.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:25:48.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:25:48.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:25:48.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:25:48.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cephadm 2021-03-15T 2021-03-15T22:25:48.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: 22:25:47.694545+0000 mgr.y (mgr.14144) 633 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:25:48.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:25:48.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:25:48.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:25:48.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:25:48.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:25:47.694966+0000 mgr.y (mgr.14144) 634 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:25:48.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:25:48.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:25:48.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:25:48.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:25:48.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:25: 2021-03-15T22:25:48.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: 47.695352+0000 mgr.y (mgr.14144) 635 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:25:48.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: -15T22:25:47.694063+0000 mgr.y (mgr.14144) 632 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:25:48.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:25:48.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:25:48.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:25:48.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:25:48.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:25:47.694545+0000 mgr.y (mgr.14144) 633 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:25:48.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:25:48.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:25:48.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:25:48.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:25:48.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cephadm 2021-03-15 2021-03-15T22:25:48.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: T22:25:47.694966+0000 mgr.y (mgr.14144) 634 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:25:48.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:25:48.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:25:48.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:25:48.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:25:48.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:25:47.695352+0000 mgr.y (mgr.14144) 635 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:25:48.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:25:48.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:25:48.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:25:48.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:25:48.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:25:47.695728+0000 mgr.y (mgr.14144) 636 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:25:48.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:25:48.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:25:48.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:25:48.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:25:48.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:25:47.696096+0000 mgr.y (mgr.14144) 637 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:25:48.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:25:48.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:25:48.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:25:48.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:25:48.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: cluster 2021-03-15T22:25:47.753809+0000 osd.5 (osd.5) 76 : cluster [DBG] 3.c scrub ok 2021-03-15T22:25:48.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: audit 2021-03-15T22:25:47.860422+0000 mon.c (mon.1) 38 : audit [INF] from='client.? 172.21.15.76:0/609453207' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:48.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[62156]: audit 2021-03-15T22:25:47.860726+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:48.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:25:48.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:25:48.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:25:48.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:25:48.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:25:47.695728+0000 mgr.y (mgr.14144) 636 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:25:48.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:25:48.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:25:48.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:25:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:25:48.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:25:47.696096+0000 mgr.y (mgr.14144) 637 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:25:48.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:25:48.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:25:48.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:25:48.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:25:48.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:25:48.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:25:48.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:25:48.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:25:48.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: cluster 2021-03-15T22:25:47.753809+0000 osd.5 (osd.5) 76 : cluster [DBG] 3.c scrub ok 2021-03-15T22:25:48.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: audit 2021-03-15T22:25:47.860422+0000 mon.c (mon.1) 38 : audit [INF] from='client.? 172.21.15.76:0/609453207' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:48.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:48 smithi076 conmon[56514]: audit 2021-03-15T22:25:47.860726+0000 mon.a (mon.0) 1263 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:25:48.561 INFO:tasks.cephadm.ceph_manager.ceph:no progress seen, keeping timeout for now 2021-03-15T22:25:48.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:25:48.562 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:48.584 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:48.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:48.586 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:48.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:48.710 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:48.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:48.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:48.767 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:25:48.813 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:48.835 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:48.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:48.837 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:48.938 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:48.961 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:48.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:48.962 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:49.063 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:49.087 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:49.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:49.089 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:49.089 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:49.154 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:49.190 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:49.211 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:49.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:49.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:49.278 INFO:tasks.rados.rados.0.smithi076.stdout:1594: done (0 left) 2021-03-15T22:25:49.278 INFO:tasks.rados.rados.0.smithi076.stdout:1595: write oid 1 current snap is 177 2021-03-15T22:25:49.278 INFO:tasks.rados.rados.0.smithi076.stdout:1595: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2021-03-15T22:25:49.286 INFO:tasks.rados.rados.0.smithi076.stdout:1595: writing smithi07694878-1 from 628727 to 1337185 tid 1 2021-03-15T22:25:49.292 INFO:tasks.rados.rados.0.smithi076.stdout:1595: writing smithi07694878-1 from 2045457 to 2540316 tid 2 2021-03-15T22:25:49.298 INFO:tasks.rados.rados.0.smithi076.stdout:1595: writing smithi07694878-1 from 3123240 to 3428727 tid 3 2021-03-15T22:25:49.298 INFO:tasks.rados.rados.0.smithi076.stdout:1596: rollback oid 10 current snap is 177 2021-03-15T22:25:49.299 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 10 to 140 2021-03-15T22:25:49.299 INFO:tasks.rados.rados.0.smithi076.stdout:1595: finishing write tid 1 to smithi07694878-1 2021-03-15T22:25:49.299 INFO:tasks.rados.rados.0.smithi076.stdout:1595: finishing write tid 2 to smithi07694878-1 2021-03-15T22:25:49.299 INFO:tasks.rados.rados.0.smithi076.stdout:1597: read oid 46 snap -1 2021-03-15T22:25:49.300 INFO:tasks.rados.rados.0.smithi076.stdout:1597: expect (ObjNum 496 snap 149 seq_num 496) 2021-03-15T22:25:49.300 INFO:tasks.rados.rados.0.smithi076.stdout:1598: write oid 19 current snap is 177 2021-03-15T22:25:49.300 INFO:tasks.rados.rados.0.smithi076.stdout:1598: seq_num 562 ranges {645534=783707,2103939=638517,3150663=294871} 2021-03-15T22:25:49.305 INFO:tasks.rados.rados.0.smithi076.stdout:1598: writing smithi07694878-19 from 645534 to 1429241 tid 1 2021-03-15T22:25:49.313 INFO:tasks.rados.rados.0.smithi076.stdout:1598: writing smithi07694878-19 from 2103939 to 2742456 tid 2 2021-03-15T22:25:49.313 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:49.314 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[56514]: cluster 2021-03-15T22:25:48.008908+0000 osd.7 (osd.7) 84 : cluster [DBG] 2021-03-15T22:25:49.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[56514]: 3.f scrub ok 2021-03-15T22:25:49.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[56514]: cluster 2021-03-15T22:25:48.254110+0000 mon.a (mon.0) 1264 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:49.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[56514]: audit 2021-03-15T22:25:48.265498+0000 mon.a (mon.0) 1265 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:25:49.315 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[56514]: cluster 2021-03-15T22:25:48.265575+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2021-03-15T22:25:49.316 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: cluster 2021-03-15T22:25:48. 2021-03-15T22:25:49.316 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: 008908+0000 osd.7 (osd.7) 84 : cluster [DBG] 3.f scrub ok 2021-03-15T22:25:49.316 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: cluster 2021-03-15T22:25:49.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: 2021-03-15T22:25:48.254110+0000 mon.a (mon.0) 1264 : cluster [WRN] 2021-03-15T22:25:49.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:49.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: audit 2021 2021-03-15T22:25:49.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: -03-15T22:25:48.265498+0000 mon.a (mon.0) 1265 2021-03-15T22:25:49.317 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:25:49.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: cluster 2021-03-15T22:25:49.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: 2021-03-15T22:25:48.265575 2021-03-15T22:25:49.318 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2021-03-15T22:25:49.319 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:49 smithi076 conmon[62156]: 2021-03-15T22:25:49.320 INFO:tasks.rados.rados.0.smithi076.stdout:1598: writing smithi07694878-19 from 3150663 to 3445534 tid 3 2021-03-15T22:25:49.320 INFO:tasks.rados.rados.0.smithi076.stdout:1599: read oid 32 snap -1 2021-03-15T22:25:49.321 INFO:tasks.rados.rados.0.smithi076.stdout:1599: expect (ObjNum 469 snap 137 seq_num 469) 2021-03-15T22:25:49.321 INFO:tasks.rados.rados.0.smithi076.stdout:1600: write oid 5 current snap is 177 2021-03-15T22:25:49.321 INFO:tasks.rados.rados.0.smithi076.stdout:1600: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2021-03-15T22:25:49.324 INFO:tasks.rados.rados.0.smithi076.stdout:1600: writing smithi07694878-5 from 662341 to 1237650 tid 1 2021-03-15T22:25:49.330 INFO:tasks.rados.rados.0.smithi076.stdout:1600: writing smithi07694878-5 from 1878774 to 2377302 tid 2 2021-03-15T22:25:49.336 INFO:tasks.rados.rados.0.smithi076.stdout:1600: writing smithi07694878-5 from 2894439 to 3351994 tid 3 2021-03-15T22:25:49.336 INFO:tasks.rados.rados.0.smithi076.stdout:1601: read oid 23 snap -1 2021-03-15T22:25:49.337 INFO:tasks.rados.rados.0.smithi076.stdout:1601: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:25:49.337 INFO:tasks.rados.rados.0.smithi076.stdout:1602: write oid 47 current snap is 177 2021-03-15T22:25:49.337 INFO:tasks.rados.rados.0.smithi076.stdout:1595: finishing write tid 3 to smithi07694878-1 2021-03-15T22:25:49.337 INFO:tasks.rados.rados.0.smithi076.stdout:1602: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2021-03-15T22:25:49.343 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:49.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:49.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:49.345 INFO:tasks.rados.rados.0.smithi076.stdout:1602: writing smithi07694878-47 from 679148 to 1329706 tid 1 2021-03-15T22:25:49.352 INFO:tasks.rados.rados.0.smithi076.stdout:1602: writing smithi07694878-47 from 1937256 to 2579442 tid 2 2021-03-15T22:25:49.357 INFO:tasks.rados.rados.0.smithi076.stdout:1602: writing smithi07694878-47 from 3321862 to 3479148 tid 3 2021-03-15T22:25:49.357 INFO:tasks.rados.rados.0.smithi076.stdout:1603: snap_create 2021-03-15T22:25:49.357 INFO:tasks.rados.rados.0.smithi076.stdout:1595: finishing write tid 4 to smithi07694878-1 2021-03-15T22:25:49.358 INFO:tasks.rados.rados.0.smithi076.stdout:1595: finishing write tid 5 to smithi07694878-1 2021-03-15T22:25:49.358 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 832 (ObjNum 561 snap 177 seq_num 561) dirty exists 2021-03-15T22:25:49.358 INFO:tasks.rados.rados.0.smithi076.stdout:1595: left oid 1 (ObjNum 561 snap 177 seq_num 561) 2021-03-15T22:25:49.358 INFO:tasks.rados.rados.0.smithi076.stdout:1596: finishing rollback tid 0 to smithi07694878-10 2021-03-15T22:25:49.359 INFO:tasks.rados.rados.0.smithi076.stdout:1596: finishing rollback tid 1 to smithi07694878-10 2021-03-15T22:25:49.359 INFO:tasks.rados.rados.0.smithi076.stdout:1597: expect (ObjNum 496 snap 149 seq_num 496) 2021-03-15T22:25:49.417 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:49 smithi125 conmon[56739]: cluster 2021-03-15T22:25:48.008908+0000 osd.7 (osd.7) 84 : cluster [DBG] 3.f scrub ok 2021-03-15T22:25:49.417 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:49 smithi125 conmon[56739]: cluster 2021-03-15T22:25:48.254110+0000 mon.a (mon.0) 1264 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:49.418 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:49 smithi125 conmon[56739]: audit 2021-03-15T22:25:48.265498+0000 mon.a (mon.0) 1265 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:25:49.418 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:49 smithi125 conmon[56739]: cluster 2021-03-15T22:25:48.265575+0000 mon.a (mon.0) 1266 : cluster [DBG] osdmap e433: 8 total, 8 up, 7 in 2021-03-15T22:25:49.446 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:49.469 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:49.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:49.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:49.536 INFO:tasks.rados.rados.0.smithi076.stdout:1596: finishing rollback tid 2 to smithi07694878-10 2021-03-15T22:25:49.537 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 380 (ObjNum 444 snap 125 seq_num 444) dirty exists 2021-03-15T22:25:49.537 INFO:tasks.rados.rados.0.smithi076.stdout:1598: finishing write tid 1 to smithi07694878-19 2021-03-15T22:25:49.537 INFO:tasks.rados.rados.0.smithi076.stdout:1598: finishing write tid 2 to smithi07694878-19 2021-03-15T22:25:49.537 INFO:tasks.rados.rados.0.smithi076.stdout:1598: finishing write tid 3 to smithi07694878-19 2021-03-15T22:25:49.537 INFO:tasks.rados.rados.0.smithi076.stdout:1598: finishing write tid 4 to smithi07694878-19 2021-03-15T22:25:49.538 INFO:tasks.rados.rados.0.smithi076.stdout:1598: finishing write tid 5 to smithi07694878-19 2021-03-15T22:25:49.538 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 968 (ObjNum 562 snap 177 seq_num 562) dirty exists 2021-03-15T22:25:49.538 INFO:tasks.rados.rados.0.smithi076.stdout:1598: left oid 19 (ObjNum 562 snap 177 seq_num 562) 2021-03-15T22:25:49.538 INFO:tasks.rados.rados.0.smithi076.stdout:1600: finishing write tid 1 to smithi07694878-5 2021-03-15T22:25:49.539 INFO:tasks.rados.rados.0.smithi076.stdout:1599: expect (ObjNum 469 snap 137 seq_num 469) 2021-03-15T22:25:49.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:49.587 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:25:49.588 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:25:49.588 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:25:49.588 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:25:49.589 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:25:49.595 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:49.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:49.596 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:49.697 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:49.721 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:49.722 INFO:tasks.rados.rados.0.smithi076.stdout:1600: finishing write tid 2 to smithi07694878-5 2021-03-15T22:25:49.722 INFO:tasks.rados.rados.0.smithi076.stdout:1601: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:25:49.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:49.723 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:49.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:49.847 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:49.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:49.848 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:49.922 INFO:tasks.rados.rados.0.smithi076.stdout:1600: finishing write tid 3 to smithi07694878-5 2021-03-15T22:25:49.923 INFO:tasks.rados.rados.0.smithi076.stdout:1600: finishing write tid 4 to smithi07694878-5 2021-03-15T22:25:49.923 INFO:tasks.rados.rados.0.smithi076.stdout:1600: finishing write tid 5 to smithi07694878-5 2021-03-15T22:25:49.923 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 596 (ObjNum 563 snap 177 seq_num 563) dirty exists 2021-03-15T22:25:49.923 INFO:tasks.rados.rados.0.smithi076.stdout:1600: left oid 5 (ObjNum 563 snap 177 seq_num 563) 2021-03-15T22:25:49.924 INFO:tasks.rados.rados.0.smithi076.stdout:1602: finishing write tid 1 to smithi07694878-47 2021-03-15T22:25:49.924 INFO:tasks.rados.rados.0.smithi076.stdout:1602: finishing write tid 2 to smithi07694878-47 2021-03-15T22:25:49.924 INFO:tasks.rados.rados.0.smithi076.stdout:1602: finishing write tid 3 to smithi07694878-47 2021-03-15T22:25:49.925 INFO:tasks.rados.rados.0.smithi076.stdout:1602: finishing write tid 4 to smithi07694878-47 2021-03-15T22:25:49.925 INFO:tasks.rados.rados.0.smithi076.stdout:1602: finishing write tid 5 to smithi07694878-47 2021-03-15T22:25:49.925 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 349 (ObjNum 564 snap 177 seq_num 564) dirty exists 2021-03-15T22:25:49.926 INFO:tasks.rados.rados.0.smithi076.stdout:1602: left oid 47 (ObjNum 564 snap 177 seq_num 564) 2021-03-15T22:25:49.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:49.972 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:49.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:49.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:50.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:25:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:25:50.075 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:50.098 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:50.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:50.099 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:50.111 INFO:tasks.rados.rados.0.smithi076.stdout:1595: done (8 left) 2021-03-15T22:25:50.112 INFO:tasks.rados.rados.0.smithi076.stdout:1596: done (7 left) 2021-03-15T22:25:50.112 INFO:tasks.rados.rados.0.smithi076.stdout:1597: done (6 left) 2021-03-15T22:25:50.112 INFO:tasks.rados.rados.0.smithi076.stdout:1598: done (5 left) 2021-03-15T22:25:50.112 INFO:tasks.rados.rados.0.smithi076.stdout:1599: done (4 left) 2021-03-15T22:25:50.113 INFO:tasks.rados.rados.0.smithi076.stdout:1600: done (3 left) 2021-03-15T22:25:50.113 INFO:tasks.rados.rados.0.smithi076.stdout:1601: done (2 left) 2021-03-15T22:25:50.113 INFO:tasks.rados.rados.0.smithi076.stdout:1602: done (1 left) 2021-03-15T22:25:50.113 INFO:tasks.rados.rados.0.smithi076.stdout:1603: done (0 left) 2021-03-15T22:25:50.114 INFO:tasks.rados.rados.0.smithi076.stdout:1604: write (excl) oid 19 current snap is 178 2021-03-15T22:25:50.114 INFO:tasks.rados.rados.0.smithi076.stdout:1604: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2021-03-15T22:25:50.119 INFO:tasks.rados.rados.0.smithi076.stdout:1604: writing smithi07694878-19 from 695955 to 1421762 tid 1 2021-03-15T22:25:50.124 INFO:tasks.rados.rados.0.smithi076.stdout:1604: writing smithi07694878-19 from 1879385 to 2381582 tid 2 2021-03-15T22:25:50.130 INFO:tasks.rados.rados.0.smithi076.stdout:1604: writing smithi07694878-19 from 2949285 to 3429384 tid 3 2021-03-15T22:25:50.130 INFO:tasks.rados.rados.0.smithi076.stdout:1605: snap_remove snap 147 2021-03-15T22:25:50.200 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:50.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:50.224 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:50.225 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:50.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-15T22:25:50.327 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:50.352 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:50.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:50.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:50.425 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:50 smithi076 conmon[56514]: cluster 2021-03-15T22:25:49.058956+0000 mgr.y (mgr.14144) 638 : cluster [DBG] pgmap v777: 138 pgs: 138 active+clean; 368 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 4.6 MiB/s wr, 17 op/s; 5.8 MiB/s, 3 objects/s recovering 2021-03-15T22:25:50.425 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:50 smithi076 conmon[56514]: cluster 2021-03-15T22:25:49.263333+0000 mon.a (mon.0) 1267 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/1503 objects degraded (0.399%), 1 pg degraded) 2021-03-15T22:25:50.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:50 smithi076 conmon[56514]: cluster 2021-03-15T22:25:49.273064+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2021-03-15T22:25:50.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:50 smithi076 conmon[56514]: audit 2021-03-15T22:25:49.730422+0000 mgr.y (mgr.14144) 639 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:50.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:50 smithi076 conmon[56514]: cluster 2021-03-15T22:25:50.104817+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2021-03-15T22:25:50.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:50 smithi076 conmon[62156]: cluster 2021-03-15T22:25:49.058956+0000 mgr.y (mgr.14144) 638 : cluster [DBG] pgmap v777: 138 pgs: 138 active+clean; 368 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 4.6 MiB/s wr, 17 op/s; 5.8 MiB/s, 3 objects/s recovering 2021-03-15T22:25:50.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:50 smithi076 conmon[62156]: cluster 2021-03-15T22:25:49.263333+0000 mon.a (mon.0) 1267 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/1503 objects degraded (0.399%), 1 pg degraded) 2021-03-15T22:25:50.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:50 smithi076 conmon[62156]: cluster 2021-03-15T22:25:49.273064+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2021-03-15T22:25:50.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:50 smithi076 conmon[62156]: audit 2021-03-15T22:25:49.730422+0000 mgr.y (mgr.14144) 639 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:50.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:50 smithi076 conmon[62156]: cluster 2021-03-15T22:25:50.104817+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2021-03-15T22:25:50.454 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:50.478 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:50.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:50 smithi125 conmon[56739]: cluster 2021-03-15T22:25:49.058956+0000 mgr.y (mgr.14144) 638 : cluster [DBG] pgmap v777: 138 pgs: 138 active+clean; 368 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.0 MiB/s rd, 4.6 MiB/s wr, 17 op/s; 5.8 MiB/s, 3 objects/s recovering 2021-03-15T22:25:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:50 smithi125 conmon[56739]: cluster 2021-03-15T22:25:49.263333+0000 mon.a (mon.0) 1267 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/1503 objects degraded (0.399%), 1 pg degraded) 2021-03-15T22:25:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:50 smithi125 conmon[56739]: cluster 2021-03-15T22:25:49.273064+0000 mon.a (mon.0) 1268 : cluster [DBG] osdmap e434: 8 total, 8 up, 7 in 2021-03-15T22:25:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:50 smithi125 conmon[56739]: audit 2021-03-15T22:25:49.730422+0000 mgr.y (mgr.14144) 639 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:25:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:50 smithi125 conmon[56739]: cluster 2021-03-15T22:25:50.104817+0000 mon.a (mon.0) 1269 : cluster [DBG] osdmap e435: 8 total, 8 up, 7 in 2021-03-15T22:25:50.579 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:25:50.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:50.605 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:50.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:50.707 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:50.730 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:50.732 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:50.833 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:50.855 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:50.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:50.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:50.958 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:50.981 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:50.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:50.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:51.071 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:25:51.084 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:51.107 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:51.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:51.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:51.123 INFO:tasks.rados.rados.0.smithi076.stdout:1605: done (1 left) 2021-03-15T22:25:51.124 INFO:tasks.rados.rados.0.smithi076.stdout:1606: rollback oid 8 current snap is 178 2021-03-15T22:25:51.124 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 151 2021-03-15T22:25:51.124 INFO:tasks.rados.rados.0.smithi076.stdout:1607: write (excl) oid 18 current snap is 178 2021-03-15T22:25:51.125 INFO:tasks.rados.rados.0.smithi076.stdout:1607: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2021-03-15T22:25:51.127 INFO:tasks.rados.rados.0.smithi076.stdout:1607: writing smithi07694878-18 from 712762 to 1113818 tid 1 2021-03-15T22:25:51.134 INFO:tasks.rados.rados.0.smithi076.stdout:1607: writing smithi07694878-18 from 1537867 to 2183722 tid 2 2021-03-15T22:25:51.143 INFO:tasks.rados.rados.0.smithi076.stdout:1607: writing smithi07694878-18 from 2860355 to 3512762 tid 3 2021-03-15T22:25:51.143 INFO:tasks.rados.rados.0.smithi076.stdout:1608: delete oid 45 current snap is 178 2021-03-15T22:25:51.143 INFO:tasks.rados.rados.0.smithi076.stdout:1604: finishing write tid 1 to smithi07694878-19 2021-03-15T22:25:51.143 INFO:tasks.rados.rados.0.smithi076.stdout:1604: finishing write tid 2 to smithi07694878-19 2021-03-15T22:25:51.143 INFO:tasks.rados.rados.0.smithi076.stdout:1604: finishing write tid 3 to smithi07694878-19 2021-03-15T22:25:51.144 INFO:tasks.rados.rados.0.smithi076.stdout:1604: finishing write tid 4 to smithi07694878-19 2021-03-15T22:25:51.144 INFO:tasks.rados.rados.0.smithi076.stdout:1604: finishing write tid 5 to smithi07694878-19 2021-03-15T22:25:51.144 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 972 (ObjNum 565 snap 178 seq_num 565) dirty exists 2021-03-15T22:25:51.144 INFO:tasks.rados.rados.0.smithi076.stdout:1604: left oid 19 (ObjNum 565 snap 178 seq_num 565) 2021-03-15T22:25:51.145 INFO:tasks.rados.rados.0.smithi076.stdout:1606: finishing rollback tid 0 to smithi07694878-8 2021-03-15T22:25:51.145 INFO:tasks.rados.rados.0.smithi076.stdout:1606: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:25:51.145 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 768 (ObjNum 1 snap 32564 seq_num 265160304) dirty dne 2021-03-15T22:25:51.145 INFO:tasks.rados.rados.0.smithi076.stdout:1607: finishing write tid 1 to smithi07694878-18 2021-03-15T22:25:51.146 INFO:tasks.rados.rados.0.smithi076.stdout:1607: finishing write tid 2 to smithi07694878-18 2021-03-15T22:25:51.147 INFO:tasks.rados.rados.0.smithi076.stdout:1604: done (3 left) 2021-03-15T22:25:51.147 INFO:tasks.rados.rados.0.smithi076.stdout:1606: done (2 left) 2021-03-15T22:25:51.147 INFO:tasks.rados.rados.0.smithi076.stdout:1608: done (1 left) 2021-03-15T22:25:51.148 INFO:tasks.rados.rados.0.smithi076.stdout:1609: copy_from oid 28 from oid 44 current snap is 178 2021-03-15T22:25:51.148 INFO:tasks.rados.rados.0.smithi076.stdout:1610: delete oid 48 current snap is 178 2021-03-15T22:25:51.148 INFO:tasks.rados.rados.0.smithi076.stdout:1609: finishing copy_from to smithi07694878-28 2021-03-15T22:25:51.148 INFO:tasks.rados.rados.0.smithi076.stdout:1609: got expected ENOENT (src dne) 2021-03-15T22:25:51.149 INFO:tasks.rados.rados.0.smithi076.stdout:1609: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:25:51.149 INFO:tasks.rados.rados.0.smithi076.stdout:1607: finishing write tid 3 to smithi07694878-18 2021-03-15T22:25:51.149 INFO:tasks.rados.rados.0.smithi076.stdout:1607: finishing write tid 4 to smithi07694878-18 2021-03-15T22:25:51.149 INFO:tasks.rados.rados.0.smithi076.stdout:1609: done (2 left) 2021-03-15T22:25:51.149 INFO:tasks.rados.rados.0.smithi076.stdout:1610: done (1 left) 2021-03-15T22:25:51.150 INFO:tasks.rados.rados.0.smithi076.stdout:1611: delete oid 14 current snap is 178 2021-03-15T22:25:51.150 INFO:tasks.rados.rados.0.smithi076.stdout:1607: finishing write tid 5 to smithi07694878-18 2021-03-15T22:25:51.150 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 777 (ObjNum 566 snap 178 seq_num 566) dirty exists 2021-03-15T22:25:51.150 INFO:tasks.rados.rados.0.smithi076.stdout:1607: left oid 18 (ObjNum 566 snap 178 seq_num 566) 2021-03-15T22:25:51.152 INFO:tasks.rados.rados.0.smithi076.stdout:1607: done (1 left) 2021-03-15T22:25:51.153 INFO:tasks.rados.rados.0.smithi076.stdout:1611: done (0 left) 2021-03-15T22:25:51.153 INFO:tasks.rados.rados.0.smithi076.stdout:1612: read oid 18 snap -1 2021-03-15T22:25:51.153 INFO:tasks.rados.rados.0.smithi076.stdout:1612: expect (ObjNum 566 snap 178 seq_num 566) 2021-03-15T22:25:51.153 INFO:tasks.rados.rados.0.smithi076.stdout:1613: read oid 45 snap -1 2021-03-15T22:25:51.153 INFO:tasks.rados.rados.0.smithi076.stdout:1613: expect deleted 2021-03-15T22:25:51.154 INFO:tasks.rados.rados.0.smithi076.stdout:1614: write (excl) oid 33 current snap is 178 2021-03-15T22:25:51.154 INFO:tasks.rados.rados.0.smithi076.stdout:1614: seq_num 570 ranges {779990=702052,2055442=653193,3253694=326296} 2021-03-15T22:25:51.156 INFO:tasks.rados.rados.0.smithi076.stdout:1614: writing smithi07694878-33 from 779990 to 1482042 tid 1 2021-03-15T22:25:51.162 INFO:tasks.rados.rados.0.smithi076.stdout:1614: writing smithi07694878-33 from 2055442 to 2708635 tid 2 2021-03-15T22:25:51.167 INFO:tasks.rados.rados.0.smithi076.stdout:1614: writing smithi07694878-33 from 3253694 to 3579990 tid 3 2021-03-15T22:25:51.168 INFO:tasks.rados.rados.0.smithi076.stdout:1615: snap_create 2021-03-15T22:25:51.168 INFO:tasks.rados.rados.0.smithi076.stdout:1612: expect (ObjNum 566 snap 178 seq_num 566) 2021-03-15T22:25:51.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:51.232 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:51.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:51.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:51.334 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:51.356 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:51.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:51.358 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:51.427 INFO:tasks.rados.rados.0.smithi076.stdout:1614: finishing write tid 1 to smithi07694878-33 2021-03-15T22:25:51.428 INFO:tasks.rados.rados.0.smithi076.stdout:1614: finishing write tid 2 to smithi07694878-33 2021-03-15T22:25:51.428 INFO:tasks.rados.rados.0.smithi076.stdout:1614: finishing write tid 3 to smithi07694878-33 2021-03-15T22:25:51.428 INFO:tasks.rados.rados.0.smithi076.stdout:1614: finishing write tid 4 to smithi07694878-33 2021-03-15T22:25:51.428 INFO:tasks.rados.rados.0.smithi076.stdout:1614: finishing write tid 5 to smithi07694878-33 2021-03-15T22:25:51.429 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 770 (ObjNum 570 snap 178 seq_num 570) dirty exists 2021-03-15T22:25:51.429 INFO:tasks.rados.rados.0.smithi076.stdout:1614: left oid 33 (ObjNum 570 snap 178 seq_num 570) 2021-03-15T22:25:51.460 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:51.474 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:51.486 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:51.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:51.488 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:51.589 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:51.613 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:51.665 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:25:51.667 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:25:51.714 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:51.737 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:51.738 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:51.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:51.862 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:51.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:51.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:51.920 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:25:51.937 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:25:51.957 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:25:51.964 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:51.977 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:25:51.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:51.989 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:51.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:51.996 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:25:52.014 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:25:52.031 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:25:52.049 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:25:52.050 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:25:52.050 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:25:52.050 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:25:52.051 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:25:52.051 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:25:52.051 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:25:52.051 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:25:52.051 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:25:52.091 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:52.115 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:52.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:52.117 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:52.122 INFO:tasks.rados.rados.0.smithi076.stdout:1612: done (3 left) 2021-03-15T22:25:52.123 INFO:tasks.rados.rados.0.smithi076.stdout:1613: done (2 left) 2021-03-15T22:25:52.123 INFO:tasks.rados.rados.0.smithi076.stdout:1614: done (1 left) 2021-03-15T22:25:52.123 INFO:tasks.rados.rados.0.smithi076.stdout:1615: done (0 left) 2021-03-15T22:25:52.123 INFO:tasks.rados.rados.0.smithi076.stdout:1616: snap_remove snap 175 2021-03-15T22:25:52.142 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:25:52.142 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:25:52.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:52 smithi076 conmon[56514]: cluster 2021-03-15T22:25:51.059617+0000 mgr.y (mgr.14144) 640 : cluster [DBG] pgmap v780: 138 pgs: 138 active+clean; 369 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 1.5 MiB/s wr, 10 op/s; 5.8 MiB/s, 3 objects/s recovering 2021-03-15T22:25:52.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:52 smithi076 conmon[56514]: cluster 2021-03-15T22:25:51.115410+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2021-03-15T22:25:52.197 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:52 smithi076 conmon[56514]: audit 2021-03-15T22:25:51.647919+0000 mgr.y (mgr.14144) 641 : audit [DBG] from='client.15519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:25:52.198 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:52 smithi076 conmon[56514]: audit 2021-03-15T22:25:51.898763+0000 mon.a (mon.0) 1271 : audit [DBG] from='client.? 172.21.15.76:0/3576105343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:52.198 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:52 smithi076 conmon[62156]: cluster 2021-03-15T22:25:51.059617+0000 mgr.y (mgr.14144) 640 : cluster [DBG] pgmap v780: 138 pgs: 138 active+clean; 369 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 1.5 MiB/s wr, 10 op/s; 5.8 MiB/s, 3 objects/s recovering 2021-03-15T22:25:52.198 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:52 smithi076 conmon[62156]: cluster 2021-03-15T22:25:51.115410+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2021-03-15T22:25:52.199 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:52 smithi076 conmon[62156]: audit 2021-03-15T22:25:51.647919+0000 mgr.y (mgr.14144) 641 : audit [DBG] from='client.15519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:25:52.199 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:52 smithi076 conmon[62156]: audit 2021-03-15T22:25:51.898763+0000 mon.a (mon.0) 1271 : audit [DBG] from='client.? 172.21.15.76:0/3576105343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:52.217 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:52.240 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:52.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:52.242 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:52.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:52 smithi125 conmon[56739]: cluster 2021-03-15T22:25:51.059617+0000 mgr.y (mgr.14144) 640 : cluster [DBG] pgmap v780: 138 pgs: 138 active+clean; 369 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.1 MiB/s rd, 1.5 MiB/s wr, 10 op/s; 5.8 MiB/s, 3 objects/s recovering 2021-03-15T22:25:52.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:52 smithi125 conmon[56739]: cluster 2021-03-15T22:25:51.115410+0000 mon.a (mon.0) 1270 : cluster [DBG] osdmap e436: 8 total, 8 up, 7 in 2021-03-15T22:25:52.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:52 smithi125 conmon[56739]: audit 2021-03-15T22:25:51.647919+0000 mgr.y (mgr.14144) 641 : audit [DBG] from='client.15519 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:25:52.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:52 smithi125 conmon[56739]: audit 2021-03-15T22:25:51.898763+0000 mon.a (mon.0) 1271 : audit [DBG] from='client.? 172.21.15.76:0/3576105343' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:52.343 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:52.366 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:52.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:52.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:52.469 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:52.490 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:52.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:52.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:52.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:52.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:52.616 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:52.617 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:52.718 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:52.741 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:52.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:52.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:52.792 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-15T22:25:52.844 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:52.866 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:52.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:52.867 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:52.969 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:52.992 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:52.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:52.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:53.064 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:25:53.095 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:53.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:53.119 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:53.119 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:53.128 INFO:tasks.rados.rados.0.smithi076.stdout:1616: done (0 left) 2021-03-15T22:25:53.128 INFO:tasks.rados.rados.0.smithi076.stdout:1617: read oid 28 snap -1 2021-03-15T22:25:53.129 INFO:tasks.rados.rados.0.smithi076.stdout:1617: expect (ObjNum 549 snap 175 seq_num 549) 2021-03-15T22:25:53.129 INFO:tasks.rados.rados.0.smithi076.stdout:1618: write (excl) oid 13 current snap is 179 2021-03-15T22:25:53.129 INFO:tasks.rados.rados.0.smithi076.stdout:1618: seq_num 571 ranges {796797=493654,1830277=796851,3397470=199327} 2021-03-15T22:25:53.135 INFO:tasks.rados.rados.0.smithi076.stdout:1618: writing smithi07694878-13 from 796797 to 1290451 tid 1 2021-03-15T22:25:53.143 INFO:tasks.rados.rados.0.smithi076.stdout:1618: writing smithi07694878-13 from 1830277 to 2627128 tid 2 2021-03-15T22:25:53.149 INFO:tasks.rados.rados.0.smithi076.stdout:1618: writing smithi07694878-13 from 3397470 to 3596797 tid 3 2021-03-15T22:25:53.149 INFO:tasks.rados.rados.0.smithi076.stdout:1619: read oid 7 snap -1 2021-03-15T22:25:53.149 INFO:tasks.rados.rados.0.smithi076.stdout:1619: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:25:53.150 INFO:tasks.rados.rados.0.smithi076.stdout:1620: read oid 31 snap -1 2021-03-15T22:25:53.150 INFO:tasks.rados.rados.0.smithi076.stdout:1620: expect (ObjNum 516 snap 158 seq_num 516) 2021-03-15T22:25:53.150 INFO:tasks.rados.rados.0.smithi076.stdout:1621: write (excl) oid 43 current snap is 179 2021-03-15T22:25:53.150 INFO:tasks.rados.rados.0.smithi076.stdout:1618: finishing write tid 1 to smithi07694878-13 2021-03-15T22:25:53.151 INFO:tasks.rados.rados.0.smithi076.stdout:1617: expect (ObjNum 549 snap 175 seq_num 549) 2021-03-15T22:25:53.220 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:53.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:53.243 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:53.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:53.279 INFO:tasks.rados.rados.0.smithi076.stdout:1618: finishing write tid 2 to smithi07694878-13 2021-03-15T22:25:53.279 INFO:tasks.rados.rados.0.smithi076.stdout:1619: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:25:53.345 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:53.366 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:53.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:53.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:53.443 INFO:tasks.rados.rados.0.smithi076.stdout:1618: finishing write tid 3 to smithi07694878-13 2021-03-15T22:25:53.444 INFO:tasks.rados.rados.0.smithi076.stdout:1618: finishing write tid 4 to smithi07694878-13 2021-03-15T22:25:53.444 INFO:tasks.rados.rados.0.smithi076.stdout:1618: finishing write tid 5 to smithi07694878-13 2021-03-15T22:25:53.444 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 425 (ObjNum 571 snap 179 seq_num 571) dirty exists 2021-03-15T22:25:53.445 INFO:tasks.rados.rados.0.smithi076.stdout:1618: left oid 13 (ObjNum 571 snap 179 seq_num 571) 2021-03-15T22:25:53.445 INFO:tasks.rados.rados.0.smithi076.stdout:1620: expect (ObjNum 516 snap 158 seq_num 516) 2021-03-15T22:25:53.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:53 smithi125 conmon[56739]: cluster 2021-03-15T22:25:52.119721+0000 mon.a 2021-03-15T22:25:53.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:53 smithi125 conmon[56739]: (mon.0) 1272 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2021-03-15T22:25:53.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:53 smithi125 conmon[56739]: cluster 2021-03-15T22:25:52.276959+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:53.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:53 smithi125 conmon[56739]: cluster 2021-03-15T22:25:52.277739+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:53.469 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:53.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[56514]: cluster 2021-03-15T22:25:52.119721+0000 mon.a ( 2021-03-15T22:25:53.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[56514]: mon.0) 1272 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2021-03-15T22:25:53.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[56514]: cluster 2021-03-15T22:25:52.276959+0000 osd.7 (osd.7) 85 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:53.493 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[56514]: cluster 2021-03-15T22:25:52.277739+0000 osd.7 (osd.7) 86 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:53.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[62156]: cluster 2021-03-15T22:25:52.119721 2021-03-15T22:25:53.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1272 : cluster [DBG] osdmap e437: 8 total, 8 up, 7 in 2021-03-15T22:25:53.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[62156]: cluster 2021-03-15T22:25:53.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[62156]: 2021-03-15T22:25:52.276959+0000 2021-03-15T22:25:53.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[62156]: osd.7 (osd.7) 85 : 2021-03-15T22:25:53.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[62156]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:53.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[62156]: cluster 2021-03-15T22:25:53.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[62156]: 2021-03-15T22:25 2021-03-15T22:25:53.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[62156]: :52.277739+0000 osd.7 ( 2021-03-15T22:25:53.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[62156]: osd.7) 86 : cluster 2021-03-15T22:25:53.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:53 smithi076 conmon[62156]: [DBG] purged_snaps scrub ok 2021-03-15T22:25:53.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:53.500 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:53.500 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:53.581 INFO:tasks.rados.rados.0.smithi076.stdout:1621: seq_num 572 ranges {413604=568903,1488759=656862,2624893=591709} 2021-03-15T22:25:53.587 INFO:tasks.rados.rados.0.smithi076.stdout:1621: writing smithi07694878-43 from 413604 to 982507 tid 1 2021-03-15T22:25:53.593 INFO:tasks.rados.rados.0.smithi076.stdout:1621: writing smithi07694878-43 from 1488759 to 2145621 tid 2 2021-03-15T22:25:53.600 INFO:tasks.rados.rados.0.smithi076.stdout:1621: writing smithi07694878-43 from 2624893 to 3216602 tid 3 2021-03-15T22:25:53.600 INFO:tasks.rados.rados.0.smithi076.stdout:1617: done (4 left) 2021-03-15T22:25:53.601 INFO:tasks.rados.rados.0.smithi076.stdout:1618: done (3 left) 2021-03-15T22:25:53.601 INFO:tasks.rados.rados.0.smithi076.stdout:1619: done (2 left) 2021-03-15T22:25:53.602 INFO:tasks.rados.rados.0.smithi076.stdout:1620: done (1 left) 2021-03-15T22:25:53.602 INFO:tasks.rados.rados.0.smithi076.stdout:1622: snap_remove snap 151 2021-03-15T22:25:53.603 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:53.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:53.626 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:53.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:53.709 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":781,"stamp":"2021-03-15T22:25:51.113258+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":387013695,"num_objects":507,"num_object_clones":142,"num_object_copies":1521,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":507,"num_whiteouts":15,"num_read":4668,"num_read_kb":3181747,"num_write":2829,"num_write_kb":866487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1680,"num_bytes_recovered":514354880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6846,"ondisk_log_size":6846,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1084004,"kb_used_data":909740,"kb_used_omap":0,"kb_used_meta":174208,"kb_avail":655132060,"statfs":{"total":671965249536,"available":670855229440,"internally_reserved":0,"allocated":931573760,"data_stored":1255882357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":178388992},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":9334579,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-6,"num_objects_misplaced":-7,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":25,"num_read_kb":16881,"num_write":17,"num_write_kb":6305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":24407333,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.857126"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"393","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.235917+0000","last_change":"2021-03-15T22:25:41.036097+0000","last_active":"2021-03-15T22:25:45.235917+0000","last_peered":"2021-03-15T22:25:45.235917+0000","last_clean":"2021-03-15T22:25:45.235917+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:25:45.235917+0000","last_undegraded":"2021-03-15T22:25:45.235917+0000","last_fullsized":"2021-03-15T22:25:45.235917+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:33.728929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:33.728929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"98","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.245786+0000","last_change":"2021-03-15T22:24:31.865035+0000","last_active":"2021-03-15T22:25:46.245786+0000","last_peered":"2021-03-15T22:25:46.245786+0000","last_clean":"2021-03-15T22:25:46.245786+0000","last_became_active":"2021-03-15T22:24:31.103578+0000","last_became_peered":"2021-03-15T22:24:31.103578+0000","last_unstale":"2021-03-15T22:25:46.245786+0000","last_undegraded":"2021-03-15T22:25:46.245786+0000","last_fullsized":"2021-03-15T22:25:46.245786+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:23:59.500680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:59.500680+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"425","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.235756+0000","last_change":"2021-03-15T22:25:40.007165+0000","last_active":"2021-03-15T22:25:45.235756+0000","last_peered":"2021-03-15T22:25:45.235756+0000","last_clean":"2021-03-15T22:25:45.235756+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:25:45.235756+0000","last_undegraded":"2021-03-15T22:25:45.235756+0000","last_fullsized":"2021-03-15T22:25:45.235756+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:26.743811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.743811+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"23","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239280+0000","last_change":"2021-03-15T22:25:33.149020+0000","last_active":"2021-03-15T22:25:45.239280+0000","last_peered":"2021-03-15T22:25:45.239280+0000","last_clean":"2021-03-15T22:25:45.239280+0000","last_became_active":"2021-03-15T22:25:33.148710+0000","last_became_peered":"2021-03-15T22:25:33.148710+0000","last_unstale":"2021-03-15T22:25:45.239280+0000","last_undegraded":"2021-03-15T22:25:45.239280+0000","last_fullsized":"2021-03-15T22:25:45.239280+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:06.177754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.177754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"23","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.240291+0000","last_change":"2021-03-15T22:25:33.147017+0000","last_active":"2021-03-15T22:25:45.240291+0000","last_peered":"2021-03-15T22:25:45.240291+0000","last_clean":"2021-03-15T22:25:45.240291+0000","last_became_active":"2021-03-15T22:25:33.146697+0000","last_became_peered":"2021-03-15T22:25:33.146697+0000","last_unstale":"2021-03-15T22:25:45.240291+0000","last_undegraded":"2021-03-15T22:25:45.240291+0000","last_fullsized":"2021-03-15T22:25:45.240291+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:15.333746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.333746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"49","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.238936+0000","last_change":"2021-03-15T22:25:33.175542+0000","last_active":"2021-03-15T22:25:45.238936+0000","last_peered":"2021-03-15T22:25:45.238936+0000","last_clean":"2021-03-15T22:25:45.238936+0000","last_became_active":"2021-03-15T22:25:33.137906+0000","last_became_peered":"2021-03-15T22:25:33.137906+0000","last_unstale":"2021-03-15T22:25:45.238936+0000","last_undegraded":"2021-03-15T22:25:45.238936+0000","last_fullsized":"2021-03-15T22:25:45.238936+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:23:42.144821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.144821+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":21,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"87","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.241391+0000","last_change":"2021-03-15T22:24:31.109745+0000","last_active":"2021-03-15T22:25:46.241391+0000","last_peered":"2021-03-15T22:25:46.241391+0000","last_clean":"2021-03-15T22:25:46.241391+0000","last_became_active":"2021-03-15T22:24:31.108652+0000","last_became_peered":"2021-03-15T22:24:31.108652+0000","last_unstale":"2021-03-15T22:25:46.241391+0000","last_undegraded":"2021-03-15T22:25:46.241391+0000","last_fullsized":"2021-03-15T22:25:46.241391+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:35.677439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:35.677439+0000","last_clean_scrub_stamp":"2021-03-15T22:23:35.677439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"397","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.245035+0000","last_change":"2021-03-15T22:25:16.598073+0000","last_active":"2021-03-15T22:25:46.245035+0000","last_peered":"2021-03-15T22:25:46.245035+0000","last_clean":"2021-03-15T22:25:46.245035+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:25:46.245035+0000","last_undegraded":"2021-03-15T22:25:46.245035+0000","last_fullsized":"2021-03-15T22:25:46.245035+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.597998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.597998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"500","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.236048+0000","last_change":"2021-03-15T22:24:31.776304+0000","last_active":"2021-03-15T22:25:45.236048+0000","last_peered":"2021-03-15T22:25:45.236048+0000","last_clean":"2021-03-15T22:25:45.236048+0000","last_became_active":"2021-03-15T22:24:31.083373+0000","last_became_peered":"2021-03-15T22:24:31.083373+0000","last_unstale":"2021-03-15T22:25:45.236048+0000","last_undegraded":"2021-03-15T22:25:45.236048+0000","last_fullsized":"2021-03-15T22:25:45.236048+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:24:10.979322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:10.979322+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"106","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.244017+0000","last_change":"2021-03-15T22:25:33.143128+0000","last_active":"2021-03-15T22:25:46.244017+0000","last_peered":"2021-03-15T22:25:46.244017+0000","last_clean":"2021-03-15T22:25:46.244017+0000","last_became_active":"2021-03-15T22:25:33.142493+0000","last_became_peered":"2021-03-15T22:25:33.142493+0000","last_unstale":"2021-03-15T22:25:46.244017+0000","last_undegraded":"2021-03-15T22:25:46.244017+0000","last_fullsized":"2021-03-15T22:25:46.244017+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.674895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"397","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.244397+0000","last_change":"2021-03-15T22:25:05.682093+0000","last_active":"2021-03-15T22:25:46.244397+0000","last_peered":"2021-03-15T22:25:46.244397+0000","last_clean":"2021-03-15T22:25:46.244397+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:25:46.244397+0000","last_undegraded":"2021-03-15T22:25:46.244397+0000","last_fullsized":"2021-03-15T22:25:46.244397+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.682031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.682031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"121'9","reported_seq":"466","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.236516+0000","last_change":"2021-03-15T22:24:37.902248+0000","last_active":"2021-03-15T22:25:45.236516+0000","last_peered":"2021-03-15T22:25:45.236516+0000","last_clean":"2021-03-15T22:25:45.236516+0000","last_became_active":"2021-03-15T22:22:56.055083+0000","last_became_peered":"2021-03-15T22:22:56.055083+0000","last_unstale":"2021-03-15T22:25:45.236516+0000","last_undegraded":"2021-03-15T22:25:45.236516+0000","last_fullsized":"2021-03-15T22:25:45.236516+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'9","last_scrub_stamp":"2021-03-15T22:24:37.902166+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.902166+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"393","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.238153+0000","last_change":"2021-03-15T22:24:30.865676+0000","last_active":"2021-03-15T22:25:45.238153+0000","last_peered":"2021-03-15T22:25:45.238153+0000","last_clean":"2021-03-15T22:25:45.238153+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:25:45.238153+0000","last_undegraded":"2021-03-15T22:25:45.238153+0000","last_fullsized":"2021-03-15T22:25:45.238153+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.865609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.865609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"141","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.236454+0000","last_change":"2021-03-15T22:24:55.103109+0000","last_active":"2021-03-15T22:25:45.236454+0000","last_peered":"2021-03-15T22:25:45.236454+0000","last_clean":"2021-03-15T22:25:45.236454+0000","last_became_active":"2021-03-15T22:24:55.090536+0000","last_became_peered":"2021-03-15T22:24:55.090536+0000","last_unstale":"2021-03-15T22:25:45.236454+0000","last_undegraded":"2021-03-15T22:25:45.236454+0000","last_fullsized":"2021-03-15T22:25:45.236454+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:24:28.507563+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.507563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"500","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267599+0000","last_change":"2021-03-15T22:25:41.703337+0000","last_active":"2021-03-15T22:25:48.267599+0000","last_peered":"2021-03-15T22:25:48.267599+0000","last_clean":"2021-03-15T22:25:48.267599+0000","last_became_active":"2021-03-15T22:25:33.134238+0000","last_became_peered":"2021-03-15T22:25:33.134238+0000","last_unstale":"2021-03-15T22:25:48.267599+0000","last_undegraded":"2021-03-15T22:25:48.267599+0000","last_fullsized":"2021-03-15T22:25:48.267599+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:03.482307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:03.482307+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"221","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.240380+0000","last_change":"2021-03-15T22:25:14.117883+0000","last_active":"2021-03-15T22:25:45.240380+0000","last_peered":"2021-03-15T22:25:45.240380+0000","last_clean":"2021-03-15T22:25:45.240380+0000","last_became_active":"2021-03-15T22:25:14.117558+0000","last_became_peered":"2021-03-15T22:25:14.117558+0000","last_unstale":"2021-03-15T22:25:45.240380+0000","last_undegraded":"2021-03-15T22:25:45.240380+0000","last_fullsized":"2021-03-15T22:25:45.240380+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:00.480944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:00.480944+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"402","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.236090+0000","last_change":"2021-03-15T22:25:11.028502+0000","last_active":"2021-03-15T22:25:45.236090+0000","last_peered":"2021-03-15T22:25:45.236090+0000","last_clean":"2021-03-15T22:25:45.236090+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:25:45.236090+0000","last_undegraded":"2021-03-15T22:25:45.236090+0000","last_fullsized":"2021-03-15T22:25:45.236090+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.028422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.028422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"43","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239991+0000","last_change":"2021-03-15T22:25:33.169024+0000","last_active":"2021-03-15T22:25:45.239991+0000","last_peered":"2021-03-15T22:25:45.239991+0000","last_clean":"2021-03-15T22:25:45.239991+0000","last_became_active":"2021-03-15T22:25:33.134781+0000","last_became_peered":"2021-03-15T22:25:33.134781+0000","last_unstale":"2021-03-15T22:25:45.239991+0000","last_undegraded":"2021-03-15T22:25:45.239991+0000","last_fullsized":"2021-03-15T22:25:45.239991+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:23:32.049047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:32.049047+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":36,"num_read_kb":19,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"388","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366470+0000","last_change":"2021-03-15T22:23:52.272270+0000","last_active":"2021-03-15T22:25:47.366470+0000","last_peered":"2021-03-15T22:25:47.366470+0000","last_clean":"2021-03-15T22:25:47.366470+0000","last_became_active":"2021-03-15T22:23:52.271925+0000","last_became_peered":"2021-03-15T22:23:52.271925+0000","last_unstale":"2021-03-15T22:25:47.366470+0000","last_undegraded":"2021-03-15T22:25:47.366470+0000","last_fullsized":"2021-03-15T22:25:47.366470+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:34.782225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:34.782225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"418","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.243269+0000","last_change":"2021-03-15T22:23:40.199174+0000","last_active":"2021-03-15T22:25:46.243269+0000","last_peered":"2021-03-15T22:25:46.243269+0000","last_clean":"2021-03-15T22:25:46.243269+0000","last_became_active":"2021-03-15T22:22:56.062833+0000","last_became_peered":"2021-03-15T22:22:56.062833+0000","last_unstale":"2021-03-15T22:25:46.243269+0000","last_undegraded":"2021-03-15T22:25:46.243269+0000","last_fullsized":"2021-03-15T22:25:46.243269+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:40.199095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:40.199095+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"215","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366761+0000","last_change":"2021-03-15T22:25:33.752847+0000","last_active":"2021-03-15T22:25:47.366761+0000","last_peered":"2021-03-15T22:25:47.366761+0000","last_clean":"2021-03-15T22:25:47.366761+0000","last_became_active":"2021-03-15T22:25:33.140078+0000","last_became_peered":"2021-03-15T22:25:33.140078+0000","last_unstale":"2021-03-15T22:25:47.366761+0000","last_undegraded":"2021-03-15T22:25:47.366761+0000","last_fullsized":"2021-03-15T22:25:47.366761+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:25:08.192165+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:25:08.192165+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"394","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.251030+0000","last_change":"2021-03-15T22:23:31.003298+0000","last_active":"2021-03-15T22:25:46.251030+0000","last_peered":"2021-03-15T22:25:46.251030+0000","last_clean":"2021-03-15T22:25:46.251030+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:25:46.251030+0000","last_undegraded":"2021-03-15T22:25:46.251030+0000","last_fullsized":"2021-03-15T22:25:46.251030+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:31.003234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:31.003234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"23","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.238796+0000","last_change":"2021-03-15T22:25:33.133625+0000","last_active":"2021-03-15T22:25:45.238796+0000","last_peered":"2021-03-15T22:25:45.238796+0000","last_clean":"2021-03-15T22:25:45.238796+0000","last_became_active":"2021-03-15T22:25:33.133418+0000","last_became_peered":"2021-03-15T22:25:33.133418+0000","last_unstale":"2021-03-15T22:25:45.238796+0000","last_undegraded":"2021-03-15T22:25:45.238796+0000","last_fullsized":"2021-03-15T22:25:45.238796+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:29.506406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.506406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"431","reported_epoch":"434","state":"active+clean","last_fresh":"2021-03-15T22:25:49.280681+0000","last_change":"2021-03-15T22:24:41.483906+0000","last_active":"2021-03-15T22:25:49.280681+0000","last_peered":"2021-03-15T22:25:49.280681+0000","last_clean":"2021-03-15T22:25:49.280681+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:25:49.280681+0000","last_undegraded":"2021-03-15T22:25:49.280681+0000","last_fullsized":"2021-03-15T22:25:49.280681+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:24:41.483849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:41.483849+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"393","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.365977+0000","last_change":"2021-03-15T22:25:09.177600+0000","last_active":"2021-03-15T22:25:47.365977+0000","last_peered":"2021-03-15T22:25:47.365977+0000","last_clean":"2021-03-15T22:25:47.365977+0000","last_became_active":"2021-03-15T22:23:52.260757+0000","last_became_peered":"2021-03-15T22:23:52.260757+0000","last_unstale":"2021-03-15T22:25:47.365977+0000","last_undegraded":"2021-03-15T22:25:47.365977+0000","last_fullsized":"2021-03-15T22:25:47.365977+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:09.177521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.177521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"405","reported_epoch":"434","state":"active+clean","last_fresh":"2021-03-15T22:25:49.279309+0000","last_change":"2021-03-15T22:25:02.452304+0000","last_active":"2021-03-15T22:25:49.279309+0000","last_peered":"2021-03-15T22:25:49.279309+0000","last_clean":"2021-03-15T22:25:49.279309+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:25:49.279309+0000","last_undegraded":"2021-03-15T22:25:49.279309+0000","last_fullsized":"2021-03-15T22:25:49.279309+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.452236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.452236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"229","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.245854+0000","last_change":"2021-03-15T22:25:33.147111+0000","last_active":"2021-03-15T22:25:46.245854+0000","last_peered":"2021-03-15T22:25:46.245854+0000","last_clean":"2021-03-15T22:25:46.245854+0000","last_became_active":"2021-03-15T22:25:33.146680+0000","last_became_peered":"2021-03-15T22:25:33.146680+0000","last_unstale":"2021-03-15T22:25:46.245854+0000","last_undegraded":"2021-03-15T22:25:46.245854+0000","last_fullsized":"2021-03-15T22:25:46.245854+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:57.444598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:23:57.444598+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"120","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239772+0000","last_change":"2021-03-15T22:25:15.835780+0000","last_active":"2021-03-15T22:25:45.239772+0000","last_peered":"2021-03-15T22:25:45.239772+0000","last_clean":"2021-03-15T22:25:45.239772+0000","last_became_active":"2021-03-15T22:24:23.142746+0000","last_became_peered":"2021-03-15T22:24:23.142746+0000","last_unstale":"2021-03-15T22:25:45.239772+0000","last_undegraded":"2021-03-15T22:25:45.239772+0000","last_fullsized":"2021-03-15T22:25:45.239772+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:25:15.835675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.835675+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":11,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"23","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239644+0000","last_change":"2021-03-15T22:25:33.148415+0000","last_active":"2021-03-15T22:25:45.239644+0000","last_peered":"2021-03-15T22:25:45.239644+0000","last_clean":"2021-03-15T22:25:45.239644+0000","last_became_active":"2021-03-15T22:25:33.148083+0000","last_became_peered":"2021-03-15T22:25:33.148083+0000","last_unstale":"2021-03-15T22:25:45.239644+0000","last_undegraded":"2021-03-15T22:25:45.239644+0000","last_fullsized":"2021-03-15T22:25:45.239644+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:42.525975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:40.670674+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.525975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"385","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366178+0000","last_change":"2021-03-15T22:23:52.263156+0000","last_active":"2021-03-15T22:25:47.366178+0000","last_peered":"2021-03-15T22:25:47.366178+0000","last_clean":"2021-03-15T22:25:47.366178+0000","last_became_active":"2021-03-15T22:23:52.262741+0000","last_became_peered":"2021-03-15T22:23:52.262741+0000","last_unstale":"2021-03-15T22:25:47.366178+0000","last_undegraded":"2021-03-15T22:25:47.366178+0000","last_fullsized":"2021-03-15T22:25:47.366178+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:41.710658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.710658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"392","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.246694+0000","last_change":"2021-03-15T22:23:30.005865+0000","last_active":"2021-03-15T22:25:46.246694+0000","last_peered":"2021-03-15T22:25:46.246694+0000","last_clean":"2021-03-15T22:25:46.246694+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:25:46.246694+0000","last_undegraded":"2021-03-15T22:25:46.246694+0000","last_fullsized":"2021-03-15T22:25:46.246694+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:30.005791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:30.005791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"90'22","reported_seq":"208","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239055+0000","last_change":"2021-03-15T22:25:16.834493+0000","last_active":"2021-03-15T22:25:45.239055+0000","last_peered":"2021-03-15T22:25:45.239055+0000","last_clean":"2021-03-15T22:25:45.239055+0000","last_became_active":"2021-03-15T22:22:56.056175+0000","last_became_peered":"2021-03-15T22:22:56.056175+0000","last_unstale":"2021-03-15T22:25:45.239055+0000","last_undegraded":"2021-03-15T22:25:45.239055+0000","last_fullsized":"2021-03-15T22:25:45.239055+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:25:16.834409+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.834409+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"91","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.243998+0000","last_change":"2021-03-15T22:24:41.659524+0000","last_active":"2021-03-15T22:25:46.243998+0000","last_peered":"2021-03-15T22:25:46.243998+0000","last_clean":"2021-03-15T22:25:46.243998+0000","last_became_active":"2021-03-15T22:24:31.105380+0000","last_became_peered":"2021-03-15T22:24:31.105380+0000","last_unstale":"2021-03-15T22:25:46.243998+0000","last_undegraded":"2021-03-15T22:25:46.243998+0000","last_fullsized":"2021-03-15T22:25:46.243998+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_clean_scrub_stamp":"2021-03-15T22:24:41.659450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"208","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239561+0000","last_change":"2021-03-15T22:25:14.115163+0000","last_active":"2021-03-15T22:25:45.239561+0000","last_peered":"2021-03-15T22:25:45.239561+0000","last_clean":"2021-03-15T22:25:45.239561+0000","last_became_active":"2021-03-15T22:25:14.114508+0000","last_became_peered":"2021-03-15T22:25:14.114508+0000","last_unstale":"2021-03-15T22:25:45.239561+0000","last_undegraded":"2021-03-15T22:25:45.239561+0000","last_fullsized":"2021-03-15T22:25:45.239561+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.007316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:04.613000+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.007316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"407","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366587+0000","last_change":"2021-03-15T22:23:52.266674+0000","last_active":"2021-03-15T22:25:47.366587+0000","last_peered":"2021-03-15T22:25:47.366587+0000","last_clean":"2021-03-15T22:25:47.366587+0000","last_became_active":"2021-03-15T22:23:52.266011+0000","last_became_peered":"2021-03-15T22:23:52.266011+0000","last_unstale":"2021-03-15T22:25:47.366587+0000","last_undegraded":"2021-03-15T22:25:47.366587+0000","last_fullsized":"2021-03-15T22:25:47.366587+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.449472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"83","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.243974+0000","last_change":"2021-03-15T22:24:56.824808+0000","last_active":"2021-03-15T22:25:46.243974+0000","last_peered":"2021-03-15T22:25:46.243974+0000","last_clean":"2021-03-15T22:25:46.243974+0000","last_became_active":"2021-03-15T22:24:55.097354+0000","last_became_peered":"2021-03-15T22:24:55.097354+0000","last_unstale":"2021-03-15T22:25:46.243974+0000","last_undegraded":"2021-03-15T22:25:46.243974+0000","last_fullsized":"2021-03-15T22:25:46.243974+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:24:38.433166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:38.433166+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"23","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.238879+0000","last_change":"2021-03-15T22:25:33.139212+0000","last_active":"2021-03-15T22:25:45.238879+0000","last_peered":"2021-03-15T22:25:45.238879+0000","last_clean":"2021-03-15T22:25:45.238879+0000","last_became_active":"2021-03-15T22:25:33.138883+0000","last_became_peered":"2021-03-15T22:25:33.138883+0000","last_unstale":"2021-03-15T22:25:45.238879+0000","last_undegraded":"2021-03-15T22:25:45.238879+0000","last_fullsized":"2021-03-15T22:25:45.238879+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:33.717517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:33.717517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"414","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.243803+0000","last_change":"2021-03-15T22:25:33.129883+0000","last_active":"2021-03-15T22:25:46.243803+0000","last_peered":"2021-03-15T22:25:46.243803+0000","last_clean":"2021-03-15T22:25:46.243803+0000","last_became_active":"2021-03-15T22:25:33.129553+0000","last_became_peered":"2021-03-15T22:25:33.129553+0000","last_unstale":"2021-03-15T22:25:46.243803+0000","last_undegraded":"2021-03-15T22:25:46.243803+0000","last_fullsized":"2021-03-15T22:25:46.243803+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.794432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"121","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267307+0000","last_change":"2021-03-15T22:24:12.017160+0000","last_active":"2021-03-15T22:25:48.267307+0000","last_peered":"2021-03-15T22:25:48.267307+0000","last_clean":"2021-03-15T22:25:48.267307+0000","last_became_active":"2021-03-15T22:24:03.086278+0000","last_became_peered":"2021-03-15T22:24:03.086278+0000","last_unstale":"2021-03-15T22:25:48.267307+0000","last_undegraded":"2021-03-15T22:25:48.267307+0000","last_fullsized":"2021-03-15T22:25:48.267307+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:12.017097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:12.017097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"424","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.236402+0000","last_change":"2021-03-15T22:25:43.037390+0000","last_active":"2021-03-15T22:25:45.236402+0000","last_peered":"2021-03-15T22:25:45.236402+0000","last_clean":"2021-03-15T22:25:45.236402+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:25:45.236402+0000","last_undegraded":"2021-03-15T22:25:45.236402+0000","last_fullsized":"2021-03-15T22:25:45.236402+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:29.709674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.709674+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"226","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366662+0000","last_change":"2021-03-15T22:25:33.144478+0000","last_active":"2021-03-15T22:25:47.366662+0000","last_peered":"2021-03-15T22:25:47.366662+0000","last_clean":"2021-03-15T22:25:47.366662+0000","last_became_active":"2021-03-15T22:25:33.143832+0000","last_became_peered":"2021-03-15T22:25:33.143832+0000","last_unstale":"2021-03-15T22:25:47.366662+0000","last_undegraded":"2021-03-15T22:25:47.366662+0000","last_fullsized":"2021-03-15T22:25:47.366662+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.147414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.147414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"356'3","reported_seq":"497","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:47.087966+0000","last_change":"2021-03-15T22:24:31.129210+0000","last_active":"2021-03-15T22:25:47.087966+0000","last_peered":"2021-03-15T22:25:47.087966+0000","last_clean":"2021-03-15T22:25:47.087966+0000","last_became_active":"2021-03-15T22:24:31.093479+0000","last_became_peered":"2021-03-15T22:24:31.093479+0000","last_unstale":"2021-03-15T22:25:47.087966+0000","last_undegraded":"2021-03-15T22:25:47.087966+0000","last_fullsized":"2021-03-15T22:25:47.087966+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:24:28.700659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.700659+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"394","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366207+0000","last_change":"2021-03-15T22:25:01.148183+0000","last_active":"2021-03-15T22:25:47.366207+0000","last_peered":"2021-03-15T22:25:47.366207+0000","last_clean":"2021-03-15T22:25:47.366207+0000","last_became_active":"2021-03-15T22:23:52.261234+0000","last_became_peered":"2021-03-15T22:23:52.261234+0000","last_unstale":"2021-03-15T22:25:47.366207+0000","last_undegraded":"2021-03-15T22:25:47.366207+0000","last_fullsized":"2021-03-15T22:25:47.366207+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:01.148105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:01.148105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"207","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239711+0000","last_change":"2021-03-15T22:24:27.166757+0000","last_active":"2021-03-15T22:25:45.239711+0000","last_peered":"2021-03-15T22:25:45.239711+0000","last_clean":"2021-03-15T22:25:45.239711+0000","last_became_active":"2021-03-15T22:22:56.064081+0000","last_became_peered":"2021-03-15T22:22:56.064081+0000","last_unstale":"2021-03-15T22:25:45.239711+0000","last_undegraded":"2021-03-15T22:25:45.239711+0000","last_fullsized":"2021-03-15T22:25:45.239711+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:27.166678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.166678+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":13,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"391","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.238192+0000","last_change":"2021-03-15T22:25:07.992897+0000","last_active":"2021-03-15T22:25:45.238192+0000","last_peered":"2021-03-15T22:25:45.238192+0000","last_clean":"2021-03-15T22:25:45.238192+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:25:45.238192+0000","last_undegraded":"2021-03-15T22:25:45.238192+0000","last_fullsized":"2021-03-15T22:25:45.238192+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:07.992815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.992815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"421'7","reported_seq":"29","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239558+0000","last_change":"2021-03-15T22:25:33.172230+0000","last_active":"2021-03-15T22:25:45.239558+0000","last_peered":"2021-03-15T22:25:45.239558+0000","last_clean":"2021-03-15T22:25:45.239558+0000","last_became_active":"2021-03-15T22:25:33.147484+0000","last_became_peered":"2021-03-15T22:25:33.147484+0000","last_unstale":"2021-03-15T22:25:45.239558+0000","last_undegraded":"2021-03-15T22:25:45.239558+0000","last_fullsized":"2021-03-15T22:25:45.239558+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'6","last_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.098511+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"232","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366066+0000","last_change":"2021-03-15T22:25:33.141774+0000","last_active":"2021-03-15T22:25:47.366066+0000","last_peered":"2021-03-15T22:25:47.366066+0000","last_clean":"2021-03-15T22:25:47.366066+0000","last_became_active":"2021-03-15T22:25:33.141466+0000","last_became_peered":"2021-03-15T22:25:33.141466+0000","last_unstale":"2021-03-15T22:25:47.366066+0000","last_undegraded":"2021-03-15T22:25:47.366066+0000","last_fullsized":"2021-03-15T22:25:47.366066+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:23:39.695671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"61","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.244835+0000","last_change":"2021-03-15T22:25:14.129885+0000","last_active":"2021-03-15T22:25:46.244835+0000","last_peered":"2021-03-15T22:25:46.244835+0000","last_clean":"2021-03-15T22:25:46.244835+0000","last_became_active":"2021-03-15T22:25:14.114060+0000","last_became_peered":"2021-03-15T22:25:14.114060+0000","last_unstale":"2021-03-15T22:25:46.244835+0000","last_undegraded":"2021-03-15T22:25:46.244835+0000","last_fullsized":"2021-03-15T22:25:46.244835+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:25:07.163128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.163128+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"399","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267857+0000","last_change":"2021-03-15T22:25:00.614351+0000","last_active":"2021-03-15T22:25:48.267857+0000","last_peered":"2021-03-15T22:25:48.267857+0000","last_clean":"2021-03-15T22:25:48.267857+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:25:48.267857+0000","last_undegraded":"2021-03-15T22:25:48.267857+0000","last_fullsized":"2021-03-15T22:25:48.267857+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.614288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.614288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"396","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366097+0000","last_change":"2021-03-15T22:25:10.130486+0000","last_active":"2021-03-15T22:25:47.366097+0000","last_peered":"2021-03-15T22:25:47.366097+0000","last_clean":"2021-03-15T22:25:47.366097+0000","last_became_active":"2021-03-15T22:23:52.263426+0000","last_became_peered":"2021-03-15T22:23:52.263426+0000","last_unstale":"2021-03-15T22:25:47.366097+0000","last_undegraded":"2021-03-15T22:25:47.366097+0000","last_fullsized":"2021-03-15T22:25:47.366097+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:10.130422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.130422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"47","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.240078+0000","last_change":"2021-03-15T22:25:35.064301+0000","last_active":"2021-03-15T22:25:45.240078+0000","last_peered":"2021-03-15T22:25:45.240078+0000","last_clean":"2021-03-15T22:25:45.240078+0000","last_became_active":"2021-03-15T22:25:33.151159+0000","last_became_peered":"2021-03-15T22:25:33.151159+0000","last_unstale":"2021-03-15T22:25:45.240078+0000","last_undegraded":"2021-03-15T22:25:45.240078+0000","last_fullsized":"2021-03-15T22:25:45.240078+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:26.478799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.478799+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3618,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"388","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366496+0000","last_change":"2021-03-15T22:23:52.266876+0000","last_active":"2021-03-15T22:25:47.366496+0000","last_peered":"2021-03-15T22:25:47.366496+0000","last_clean":"2021-03-15T22:25:47.366496+0000","last_became_active":"2021-03-15T22:23:52.266133+0000","last_became_peered":"2021-03-15T22:23:52.266133+0000","last_unstale":"2021-03-15T22:25:47.366496+0000","last_undegraded":"2021-03-15T22:25:47.366496+0000","last_fullsized":"2021-03-15T22:25:47.366496+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:28.808873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:28.808873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"445","reported_epoch":"434","state":"active+clean","last_fresh":"2021-03-15T22:25:49.278878+0000","last_change":"2021-03-15T22:25:33.150477+0000","last_active":"2021-03-15T22:25:49.278878+0000","last_peered":"2021-03-15T22:25:49.278878+0000","last_clean":"2021-03-15T22:25:49.278878+0000","last_became_active":"2021-03-15T22:25:33.150147+0000","last_became_peered":"2021-03-15T22:25:33.150147+0000","last_unstale":"2021-03-15T22:25:49.278878+0000","last_undegraded":"2021-03-15T22:25:49.278878+0000","last_fullsized":"2021-03-15T22:25:49.278878+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:37.475218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.475218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"398'11","reported_seq":"33","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239919+0000","last_change":"2021-03-15T22:25:33.864526+0000","last_active":"2021-03-15T22:25:45.239919+0000","last_peered":"2021-03-15T22:25:45.239919+0000","last_clean":"2021-03-15T22:25:45.239919+0000","last_became_active":"2021-03-15T22:25:33.152373+0000","last_became_peered":"2021-03-15T22:25:33.152373+0000","last_unstale":"2021-03-15T22:25:45.239919+0000","last_undegraded":"2021-03-15T22:25:45.239919+0000","last_fullsized":"2021-03-15T22:25:45.239919+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'11","last_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:25:13.181225+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"431'333","reported_seq":"1548","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.286593+0000","last_change":"2021-03-15T22:25:46.257665+0000","last_active":"2021-03-15T22:25:46.286593+0000","last_peered":"2021-03-15T22:25:46.286593+0000","last_clean":"2021-03-15T22:25:46.286593+0000","last_became_active":"2021-03-15T22:25:17.113174+0000","last_became_peered":"2021-03-15T22:25:17.113174+0000","last_unstale":"2021-03-15T22:25:46.286593+0000","last_undegraded":"2021-03-15T22:25:46.286593+0000","last_fullsized":"2021-03-15T22:25:46.286593+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'282","last_scrub_stamp":"2021-03-15T22:25:09.803362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.803362+0000","log_size":333,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23013149,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":252,"num_read_kb":286795,"num_write":143,"num_write_kb":45065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":21669077,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"},{"start":"142","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"400","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.244651+0000","last_change":"2021-03-15T22:25:00.689659+0000","last_active":"2021-03-15T22:25:46.244651+0000","last_peered":"2021-03-15T22:25:46.244651+0000","last_clean":"2021-03-15T22:25:46.244651+0000","last_became_active":"2021-03-15T22:24:31.103470+0000","last_became_peered":"2021-03-15T22:24:31.103470+0000","last_unstale":"2021-03-15T22:25:46.244651+0000","last_undegraded":"2021-03-15T22:25:46.244651+0000","last_fullsized":"2021-03-15T22:25:46.244651+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.689591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.689591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"417","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:48.009013+0000","last_change":"2021-03-15T22:25:48.008983+0000","last_active":"2021-03-15T22:25:48.009013+0000","last_peered":"2021-03-15T22:25:48.009013+0000","last_clean":"2021-03-15T22:25:48.009013+0000","last_became_active":"2021-03-15T22:24:31.097688+0000","last_became_peered":"2021-03-15T22:24:31.097688+0000","last_unstale":"2021-03-15T22:25:48.009013+0000","last_undegraded":"2021-03-15T22:25:48.009013+0000","last_fullsized":"2021-03-15T22:25:48.009013+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:25:48.008911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:48.008911+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"119","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.244097+0000","last_change":"2021-03-15T22:25:32.799965+0000","last_active":"2021-03-15T22:25:46.244097+0000","last_peered":"2021-03-15T22:25:46.244097+0000","last_clean":"2021-03-15T22:25:46.244097+0000","last_became_active":"2021-03-15T22:24:23.137531+0000","last_became_peered":"2021-03-15T22:24:23.137531+0000","last_unstale":"2021-03-15T22:25:46.244097+0000","last_undegraded":"2021-03-15T22:25:46.244097+0000","last_fullsized":"2021-03-15T22:25:46.244097+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:25:32.799882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.799882+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"317'7","reported_seq":"504","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366180+0000","last_change":"2021-03-15T22:24:30.068613+0000","last_active":"2021-03-15T22:25:47.366180+0000","last_peered":"2021-03-15T22:25:47.366180+0000","last_clean":"2021-03-15T22:25:47.366180+0000","last_became_active":"2021-03-15T22:23:52.262878+0000","last_became_peered":"2021-03-15T22:23:52.262878+0000","last_unstale":"2021-03-15T22:25:47.366180+0000","last_undegraded":"2021-03-15T22:25:47.366180+0000","last_fullsized":"2021-03-15T22:25:47.366180+0000","mapping_epoch":316,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:24:30.068550+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.068550+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"431'560","reported_seq":"1683","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.253352+0000","last_change":"2021-03-15T22:25:46.253538+0000","last_active":"2021-03-15T22:25:46.253352+0000","last_peered":"2021-03-15T22:25:46.253352+0000","last_clean":"2021-03-15T22:25:46.253352+0000","last_became_active":"2021-03-15T22:23:04.067641+0000","last_became_peered":"2021-03-15T22:23:04.067641+0000","last_unstale":"2021-03-15T22:25:46.253352+0000","last_undegraded":"2021-03-15T22:25:46.253352+0000","last_fullsized":"2021-03-15T22:25:46.253352+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'455","last_scrub_stamp":"2021-03-15T22:24:28.184612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.184612+0000","log_size":560,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27256449,"num_objects":15,"num_object_clones":10,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":3,"num_read":317,"num_read_kb":325164,"num_write":207,"num_write_kb":76575,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":32120558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"},{"start":"142","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"190","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239802+0000","last_change":"2021-03-15T22:24:37.122501+0000","last_active":"2021-03-15T22:25:45.239802+0000","last_peered":"2021-03-15T22:25:45.239802+0000","last_clean":"2021-03-15T22:25:45.239802+0000","last_became_active":"2021-03-15T22:22:56.057174+0000","last_became_peered":"2021-03-15T22:22:56.057174+0000","last_unstale":"2021-03-15T22:25:45.239802+0000","last_undegraded":"2021-03-15T22:25:45.239802+0000","last_fullsized":"2021-03-15T22:25:45.239802+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:37.122407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.122407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"396","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366749+0000","last_change":"2021-03-15T22:25:39.991710+0000","last_active":"2021-03-15T22:25:47.366749+0000","last_peered":"2021-03-15T22:25:47.366749+0000","last_clean":"2021-03-15T22:25:47.366749+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:25:47.366749+0000","last_undegraded":"2021-03-15T22:25:47.366749+0000","last_fullsized":"2021-03-15T22:25:47.366749+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:38.810397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.810397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"478","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.236265+0000","last_change":"2021-03-15T22:25:33.862633+0000","last_active":"2021-03-15T22:25:45.236265+0000","last_peered":"2021-03-15T22:25:45.236265+0000","last_clean":"2021-03-15T22:25:45.236265+0000","last_became_active":"2021-03-15T22:25:33.138262+0000","last_became_peered":"2021-03-15T22:25:33.138262+0000","last_unstale":"2021-03-15T22:25:45.236265+0000","last_undegraded":"2021-03-15T22:25:45.236265+0000","last_fullsized":"2021-03-15T22:25:45.236265+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:07.027095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.027095+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"428","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267050+0000","last_change":"2021-03-15T22:24:39.853332+0000","last_active":"2021-03-15T22:25:48.267050+0000","last_peered":"2021-03-15T22:25:48.267050+0000","last_clean":"2021-03-15T22:25:48.267050+0000","last_became_active":"2021-03-15T22:24:31.109058+0000","last_became_peered":"2021-03-15T22:24:31.109058+0000","last_unstale":"2021-03-15T22:25:48.267050+0000","last_undegraded":"2021-03-15T22:25:48.267050+0000","last_fullsized":"2021-03-15T22:25:48.267050+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.853267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.853267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"431'142","reported_seq":"1201","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.259935+0000","last_change":"2021-03-15T22:25:46.259935+0000","last_active":"2021-03-15T22:25:46.259935+0000","last_peered":"2021-03-15T22:25:46.259935+0000","last_clean":"2021-03-15T22:25:46.259935+0000","last_became_active":"2021-03-15T22:24:34.087074+0000","last_became_peered":"2021-03-15T22:24:34.087074+0000","last_unstale":"2021-03-15T22:25:46.259935+0000","last_undegraded":"2021-03-15T22:25:46.259935+0000","last_fullsized":"2021-03-15T22:25:46.259935+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'130","last_scrub_stamp":"2021-03-15T22:25:06.854931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.854931+0000","log_size":142,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":2235397,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":42,"num_read_kb":30032,"num_write":45,"num_write_kb":15718,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4470794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"},{"start":"142","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"406","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.365935+0000","last_change":"2021-03-15T22:23:52.267464+0000","last_active":"2021-03-15T22:25:47.365935+0000","last_peered":"2021-03-15T22:25:47.365935+0000","last_clean":"2021-03-15T22:25:47.365935+0000","last_became_active":"2021-03-15T22:23:52.267164+0000","last_became_peered":"2021-03-15T22:23:52.267164+0000","last_unstale":"2021-03-15T22:25:47.365935+0000","last_undegraded":"2021-03-15T22:25:47.365935+0000","last_fullsized":"2021-03-15T22:25:47.365935+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:29.845442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.845442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"446","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267256+0000","last_change":"2021-03-15T22:25:14.114184+0000","last_active":"2021-03-15T22:25:48.267256+0000","last_peered":"2021-03-15T22:25:48.267256+0000","last_clean":"2021-03-15T22:25:48.267256+0000","last_became_active":"2021-03-15T22:25:14.113857+0000","last_became_peered":"2021-03-15T22:25:14.113857+0000","last_unstale":"2021-03-15T22:25:48.267256+0000","last_undegraded":"2021-03-15T22:25:48.267256+0000","last_fullsized":"2021-03-15T22:25:48.267256+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.575552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.575552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"414","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366703+0000","last_change":"2021-03-15T22:25:14.118841+0000","last_active":"2021-03-15T22:25:47.366703+0000","last_peered":"2021-03-15T22:25:47.366703+0000","last_clean":"2021-03-15T22:25:47.366703+0000","last_became_active":"2021-03-15T22:25:14.118469+0000","last_became_peered":"2021-03-15T22:25:14.118469+0000","last_unstale":"2021-03-15T22:25:47.366703+0000","last_undegraded":"2021-03-15T22:25:47.366703+0000","last_fullsized":"2021-03-15T22:25:47.366703+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.024799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.024799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"476","reported_epoch":"434","state":"active+clean","last_fresh":"2021-03-15T22:25:49.279476+0000","last_change":"2021-03-15T22:24:40.468134+0000","last_active":"2021-03-15T22:25:49.279476+0000","last_peered":"2021-03-15T22:25:49.279476+0000","last_clean":"2021-03-15T22:25:49.279476+0000","last_became_active":"2021-03-15T22:24:03.085862+0000","last_became_peered":"2021-03-15T22:24:03.085862+0000","last_unstale":"2021-03-15T22:25:49.279476+0000","last_undegraded":"2021-03-15T22:25:49.279476+0000","last_fullsized":"2021-03-15T22:25:49.279476+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:24:40.468066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.468066+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"434'349","reported_seq":"1416","reported_epoch":"434","state":"active+clean","last_fresh":"2021-03-15T22:25:49.358204+0000","last_change":"2021-03-15T22:25:49.283036+0000","last_active":"2021-03-15T22:25:49.358204+0000","last_peered":"2021-03-15T22:25:49.358204+0000","last_clean":"2021-03-15T22:25:49.358204+0000","last_became_active":"2021-03-15T22:20:05.032282+0000","last_became_peered":"2021-03-15T22:20:05.032282+0000","last_unstale":"2021-03-15T22:25:49.358204+0000","last_undegraded":"2021-03-15T22:25:49.358204+0000","last_fullsized":"2021-03-15T22:25:49.358204+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'328","last_scrub_stamp":"2021-03-15T22:25:37.707812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.707812+0000","log_size":349,"ondisk_log_size":349,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22791563,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":198,"num_read_kb":234048,"num_write":118,"num_write_kb":38785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"4"},{"start":"142","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"94","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.365935+0000","last_change":"2021-03-15T22:25:11.117468+0000","last_active":"2021-03-15T22:25:47.365935+0000","last_peered":"2021-03-15T22:25:47.365935+0000","last_clean":"2021-03-15T22:25:47.365935+0000","last_became_active":"2021-03-15T22:24:31.093577+0000","last_became_peered":"2021-03-15T22:24:31.093577+0000","last_unstale":"2021-03-15T22:25:47.365935+0000","last_undegraded":"2021-03-15T22:25:47.365935+0000","last_fullsized":"2021-03-15T22:25:47.365935+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.117403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.117403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"446","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267549+0000","last_change":"2021-03-15T22:25:33.134762+0000","last_active":"2021-03-15T22:25:48.267549+0000","last_peered":"2021-03-15T22:25:48.267549+0000","last_clean":"2021-03-15T22:25:48.267549+0000","last_became_active":"2021-03-15T22:25:33.134358+0000","last_became_peered":"2021-03-15T22:25:33.134358+0000","last_unstale":"2021-03-15T22:25:48.267549+0000","last_undegraded":"2021-03-15T22:25:48.267549+0000","last_fullsized":"2021-03-15T22:25:48.267549+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:27.869570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.869570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"415","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.753887+0000","last_change":"2021-03-15T22:25:47.753887+0000","last_active":"2021-03-15T22:25:47.753887+0000","last_peered":"2021-03-15T22:25:47.753887+0000","last_clean":"2021-03-15T22:25:47.753887+0000","last_became_active":"2021-03-15T22:25:33.145548+0000","last_became_peered":"2021-03-15T22:25:33.145548+0000","last_unstale":"2021-03-15T22:25:47.753887+0000","last_undegraded":"2021-03-15T22:25:47.753887+0000","last_fullsized":"2021-03-15T22:25:47.753887+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:47.753814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:47.753814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"196","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267457+0000","last_change":"2021-03-15T22:25:33.912570+0000","last_active":"2021-03-15T22:25:48.267457+0000","last_peered":"2021-03-15T22:25:48.267457+0000","last_clean":"2021-03-15T22:25:48.267457+0000","last_became_active":"2021-03-15T22:25:33.135102+0000","last_became_peered":"2021-03-15T22:25:33.135102+0000","last_unstale":"2021-03-15T22:25:48.267457+0000","last_undegraded":"2021-03-15T22:25:48.267457+0000","last_fullsized":"2021-03-15T22:25:48.267457+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:01.499099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.499099+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"434'380","reported_seq":"1605","reported_epoch":"435","state":"active+clean","last_fresh":"2021-03-15T22:25:50.459806+0000","last_change":"2021-03-15T22:25:49.286461+0000","last_active":"2021-03-15T22:25:50.459806+0000","last_peered":"2021-03-15T22:25:50.459806+0000","last_clean":"2021-03-15T22:25:50.459806+0000","last_became_active":"2021-03-15T22:25:36.118229+0000","last_became_peered":"2021-03-15T22:25:36.118229+0000","last_unstale":"2021-03-15T22:25:50.459806+0000","last_undegraded":"2021-03-15T22:25:50.459806+0000","last_fullsized":"2021-03-15T22:25:50.459806+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'340","last_scrub_stamp":"2021-03-15T22:25:12.353191+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.353191+0000","log_size":380,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28154782,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":144,"num_read_kb":101625,"num_write":171,"num_write_kb":53979,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":35579701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10e","length":"1"},{"start":"13a","length":"2"}]},{"pgid":"6.6","version":"0'0","reported_seq":"23","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.240195+0000","last_change":"2021-03-15T22:25:33.152550+0000","last_active":"2021-03-15T22:25:45.240195+0000","last_peered":"2021-03-15T22:25:45.240195+0000","last_clean":"2021-03-15T22:25:45.240195+0000","last_became_active":"2021-03-15T22:25:33.152232+0000","last_became_peered":"2021-03-15T22:25:33.152232+0000","last_unstale":"2021-03-15T22:25:45.240195+0000","last_undegraded":"2021-03-15T22:25:45.240195+0000","last_fullsized":"2021-03-15T22:25:45.240195+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:34.763323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:34.763323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"399","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.244478+0000","last_change":"2021-03-15T22:25:11.615097+0000","last_active":"2021-03-15T22:25:46.244478+0000","last_peered":"2021-03-15T22:25:46.244478+0000","last_clean":"2021-03-15T22:25:46.244478+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:25:46.244478+0000","last_undegraded":"2021-03-15T22:25:46.244478+0000","last_fullsized":"2021-03-15T22:25:46.244478+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.614975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.614975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"130","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267171+0000","last_change":"2021-03-15T22:25:45.692608+0000","last_active":"2021-03-15T22:25:48.267171+0000","last_peered":"2021-03-15T22:25:48.267171+0000","last_clean":"2021-03-15T22:25:48.267171+0000","last_became_active":"2021-03-15T22:24:31.106344+0000","last_became_peered":"2021-03-15T22:24:31.106344+0000","last_unstale":"2021-03-15T22:25:48.267171+0000","last_undegraded":"2021-03-15T22:25:48.267171+0000","last_fullsized":"2021-03-15T22:25:48.267171+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:45.692545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:45.692545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"537","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.249791+0000","last_change":"2021-03-15T22:25:33.174619+0000","last_active":"2021-03-15T22:25:46.249791+0000","last_peered":"2021-03-15T22:25:46.249791+0000","last_clean":"2021-03-15T22:25:46.249791+0000","last_became_active":"2021-03-15T22:25:33.126039+0000","last_became_peered":"2021-03-15T22:25:33.126039+0000","last_unstale":"2021-03-15T22:25:46.249791+0000","last_undegraded":"2021-03-15T22:25:46.249791+0000","last_fullsized":"2021-03-15T22:25:46.249791+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:25:15.810377+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.810377+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"92","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.250935+0000","last_change":"2021-03-15T22:25:44.619146+0000","last_active":"2021-03-15T22:25:46.250935+0000","last_peered":"2021-03-15T22:25:46.250935+0000","last_clean":"2021-03-15T22:25:46.250935+0000","last_became_active":"2021-03-15T22:24:31.101668+0000","last_became_peered":"2021-03-15T22:24:31.101668+0000","last_unstale":"2021-03-15T22:25:46.250935+0000","last_undegraded":"2021-03-15T22:25:46.250935+0000","last_fullsized":"2021-03-15T22:25:46.250935+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:44.619082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:44.619082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"431'365","reported_seq":"1450","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.298135+0000","last_change":"2021-03-15T22:25:46.243247+0000","last_active":"2021-03-15T22:25:46.298135+0000","last_peered":"2021-03-15T22:25:46.298135+0000","last_clean":"2021-03-15T22:25:46.298135+0000","last_became_active":"2021-03-15T22:24:34.087484+0000","last_became_peered":"2021-03-15T22:24:34.087484+0000","last_unstale":"2021-03-15T22:25:46.298135+0000","last_undegraded":"2021-03-15T22:25:46.298135+0000","last_fullsized":"2021-03-15T22:25:46.298135+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'329","last_scrub_stamp":"2021-03-15T22:25:14.014776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:14.014776+0000","log_size":365,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19291274,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":127,"num_read_kb":112612,"num_write":129,"num_write_kb":44739,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":10768640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"},{"start":"142","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"406","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366795+0000","last_change":"2021-03-15T22:23:52.267559+0000","last_active":"2021-03-15T22:25:47.366795+0000","last_peered":"2021-03-15T22:25:47.366795+0000","last_clean":"2021-03-15T22:25:47.366795+0000","last_became_active":"2021-03-15T22:23:52.266556+0000","last_became_peered":"2021-03-15T22:23:52.266556+0000","last_unstale":"2021-03-15T22:25:47.366795+0000","last_undegraded":"2021-03-15T22:25:47.366795+0000","last_fullsized":"2021-03-15T22:25:47.366795+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:30.831598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:30.831598+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"400","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.241629+0000","last_change":"2021-03-15T22:24:39.929128+0000","last_active":"2021-03-15T22:25:46.241629+0000","last_peered":"2021-03-15T22:25:46.241629+0000","last_clean":"2021-03-15T22:25:46.241629+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:25:46.241629+0000","last_undegraded":"2021-03-15T22:25:46.241629+0000","last_fullsized":"2021-03-15T22:25:46.241629+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.929066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.929066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"434","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267108+0000","last_change":"2021-03-15T22:25:42.660209+0000","last_active":"2021-03-15T22:25:48.267108+0000","last_peered":"2021-03-15T22:25:48.267108+0000","last_clean":"2021-03-15T22:25:48.267108+0000","last_became_active":"2021-03-15T22:25:33.129875+0000","last_became_peered":"2021-03-15T22:25:33.129875+0000","last_unstale":"2021-03-15T22:25:48.267108+0000","last_undegraded":"2021-03-15T22:25:48.267108+0000","last_fullsized":"2021-03-15T22:25:48.267108+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:29.266174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.266174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"149'22","reported_seq":"82","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.250294+0000","last_change":"2021-03-15T22:25:03.853145+0000","last_active":"2021-03-15T22:25:46.250294+0000","last_peered":"2021-03-15T22:25:46.250294+0000","last_clean":"2021-03-15T22:25:46.250294+0000","last_became_active":"2021-03-15T22:24:55.093124+0000","last_became_peered":"2021-03-15T22:24:55.093124+0000","last_unstale":"2021-03-15T22:25:46.250294+0000","last_undegraded":"2021-03-15T22:25:46.250294+0000","last_fullsized":"2021-03-15T22:25:46.250294+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'22","last_scrub_stamp":"2021-03-15T22:25:03.853079+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:25:03.853079+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"419","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.268134+0000","last_change":"2021-03-15T22:25:02.520674+0000","last_active":"2021-03-15T22:25:48.268134+0000","last_peered":"2021-03-15T22:25:48.268134+0000","last_clean":"2021-03-15T22:25:48.268134+0000","last_became_active":"2021-03-15T22:21:38.147364+0000","last_became_peered":"2021-03-15T22:21:38.147364+0000","last_unstale":"2021-03-15T22:25:48.268134+0000","last_undegraded":"2021-03-15T22:25:48.268134+0000","last_fullsized":"2021-03-15T22:25:48.268134+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.520588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.520588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"433'948","reported_seq":"2743","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.276883+0000","last_change":"2021-03-15T22:25:48.277065+0000","last_active":"2021-03-15T22:25:48.276883+0000","last_peered":"2021-03-15T22:25:48.276883+0000","last_clean":"2021-03-15T22:25:48.276883+0000","last_became_active":"2021-03-15T22:25:37.118396+0000","last_became_peered":"2021-03-15T22:25:37.118396+0000","last_unstale":"2021-03-15T22:25:48.276883+0000","last_undegraded":"2021-03-15T22:25:48.276883+0000","last_fullsized":"2021-03-15T22:25:48.276883+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'350","last_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:22:40.995789+0000","log_size":948,"ondisk_log_size":948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":64163804,"num_objects":27,"num_object_clones":20,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":2,"num_read":364,"num_read_kb":322559,"num_write":354,"num_write_kb":155924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":140467323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"},{"start":"13b","length":"1"},{"start":"142","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"404","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.245716+0000","last_change":"2021-03-15T22:23:38.968009+0000","last_active":"2021-03-15T22:25:46.245716+0000","last_peered":"2021-03-15T22:25:46.245716+0000","last_clean":"2021-03-15T22:25:46.245716+0000","last_became_active":"2021-03-15T22:22:56.062745+0000","last_became_peered":"2021-03-15T22:22:56.062745+0000","last_unstale":"2021-03-15T22:25:46.245716+0000","last_undegraded":"2021-03-15T22:25:46.245716+0000","last_fullsized":"2021-03-15T22:25:46.245716+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:38.967934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:38.967934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"408","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366032+0000","last_change":"2021-03-15T22:24:31.111721+0000","last_active":"2021-03-15T22:25:47.366032+0000","last_peered":"2021-03-15T22:25:47.366032+0000","last_clean":"2021-03-15T22:25:47.366032+0000","last_became_active":"2021-03-15T22:24:31.111104+0000","last_became_peered":"2021-03-15T22:24:31.111104+0000","last_unstale":"2021-03-15T22:25:47.366032+0000","last_undegraded":"2021-03-15T22:25:47.366032+0000","last_fullsized":"2021-03-15T22:25:47.366032+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.872030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.872030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"23","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239305+0000","last_change":"2021-03-15T22:25:33.147575+0000","last_active":"2021-03-15T22:25:45.239305+0000","last_peered":"2021-03-15T22:25:45.239305+0000","last_clean":"2021-03-15T22:25:45.239305+0000","last_became_active":"2021-03-15T22:25:33.147285+0000","last_became_peered":"2021-03-15T22:25:33.147285+0000","last_unstale":"2021-03-15T22:25:45.239305+0000","last_undegraded":"2021-03-15T22:25:45.239305+0000","last_fullsized":"2021-03-15T22:25:45.239305+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:27.994280+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:27.994280+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"429","reported_epoch":"434","state":"active+clean","last_fresh":"2021-03-15T22:25:49.279628+0000","last_change":"2021-03-15T22:25:37.458353+0000","last_active":"2021-03-15T22:25:49.279628+0000","last_peered":"2021-03-15T22:25:49.279628+0000","last_clean":"2021-03-15T22:25:49.279628+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:25:49.279628+0000","last_undegraded":"2021-03-15T22:25:49.279628+0000","last_fullsized":"2021-03-15T22:25:49.279628+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:37.458291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.458291+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"392","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366527+0000","last_change":"2021-03-15T22:23:52.271864+0000","last_active":"2021-03-15T22:25:47.366527+0000","last_peered":"2021-03-15T22:25:47.366527+0000","last_clean":"2021-03-15T22:25:47.366527+0000","last_became_active":"2021-03-15T22:23:52.271442+0000","last_became_peered":"2021-03-15T22:23:52.271442+0000","last_unstale":"2021-03-15T22:25:47.366527+0000","last_undegraded":"2021-03-15T22:25:47.366527+0000","last_fullsized":"2021-03-15T22:25:47.366527+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:26.748485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.748485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"433'253","reported_seq":"1360","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.275248+0000","last_change":"2021-03-15T22:25:48.275679+0000","last_active":"2021-03-15T22:25:48.275248+0000","last_peered":"2021-03-15T22:25:48.275248+0000","last_clean":"2021-03-15T22:25:48.275248+0000","last_became_active":"2021-03-15T22:24:36.092680+0000","last_became_peered":"2021-03-15T22:24:36.092680+0000","last_unstale":"2021-03-15T22:25:48.275248+0000","last_undegraded":"2021-03-15T22:25:48.275248+0000","last_fullsized":"2021-03-15T22:25:48.275248+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'138","last_scrub_stamp":"2021-03-15T22:23:41.141218+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.141218+0000","log_size":253,"ondisk_log_size":253,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20445891,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":101,"num_read_kb":93801,"num_write":97,"num_write_kb":37009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8487583,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"},{"start":"13b","length":"1"},{"start":"142","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"412","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.249342+0000","last_change":"2021-03-15T22:25:33.129904+0000","last_active":"2021-03-15T22:25:46.249342+0000","last_peered":"2021-03-15T22:25:46.249342+0000","last_clean":"2021-03-15T22:25:46.249342+0000","last_became_active":"2021-03-15T22:25:33.129602+0000","last_became_peered":"2021-03-15T22:25:33.129602+0000","last_unstale":"2021-03-15T22:25:46.249342+0000","last_undegraded":"2021-03-15T22:25:46.249342+0000","last_fullsized":"2021-03-15T22:25:46.249342+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.931058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.931058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"229","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.236098+0000","last_change":"2021-03-15T22:25:33.143239+0000","last_active":"2021-03-15T22:25:45.236098+0000","last_peered":"2021-03-15T22:25:45.236098+0000","last_clean":"2021-03-15T22:25:45.236098+0000","last_became_active":"2021-03-15T22:25:33.142133+0000","last_became_peered":"2021-03-15T22:25:33.142133+0000","last_unstale":"2021-03-15T22:25:45.236098+0000","last_undegraded":"2021-03-15T22:25:45.236098+0000","last_fullsized":"2021-03-15T22:25:45.236098+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.028082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.028082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"875","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.271478+0000","last_change":"2021-03-15T22:24:31.248181+0000","last_active":"2021-03-15T22:25:48.271478+0000","last_peered":"2021-03-15T22:25:48.271478+0000","last_clean":"2021-03-15T22:25:48.271478+0000","last_became_active":"2021-03-15T22:24:31.109631+0000","last_became_peered":"2021-03-15T22:24:31.109631+0000","last_unstale":"2021-03-15T22:25:48.271478+0000","last_undegraded":"2021-03-15T22:25:48.271478+0000","last_fullsized":"2021-03-15T22:25:48.271478+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:23:27.790040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:23:27.790040+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":461,"num_read_kb":456,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"153","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.250145+0000","last_change":"2021-03-15T22:24:37.977457+0000","last_active":"2021-03-15T22:25:46.250145+0000","last_peered":"2021-03-15T22:25:46.250145+0000","last_clean":"2021-03-15T22:25:46.250145+0000","last_became_active":"2021-03-15T22:24:31.100536+0000","last_became_peered":"2021-03-15T22:24:31.100536+0000","last_unstale":"2021-03-15T22:25:46.250145+0000","last_undegraded":"2021-03-15T22:25:46.250145+0000","last_fullsized":"2021-03-15T22:25:46.250145+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:37.977363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.977363+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"43","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.246123+0000","last_change":"2021-03-15T22:25:14.115071+0000","last_active":"2021-03-15T22:25:46.246123+0000","last_peered":"2021-03-15T22:25:46.246123+0000","last_clean":"2021-03-15T22:25:46.246123+0000","last_became_active":"2021-03-15T22:25:14.114729+0000","last_became_peered":"2021-03-15T22:25:14.114729+0000","last_unstale":"2021-03-15T22:25:46.246123+0000","last_undegraded":"2021-03-15T22:25:46.246123+0000","last_fullsized":"2021-03-15T22:25:46.246123+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.857011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.857011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"431'998","reported_seq":"2692","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.308133+0000","last_change":"2021-03-15T22:25:46.263407+0000","last_active":"2021-03-15T22:25:46.308133+0000","last_peered":"2021-03-15T22:25:46.308133+0000","last_clean":"2021-03-15T22:25:46.308133+0000","last_became_active":"2021-03-15T22:25:45.240370+0000","last_became_peered":"2021-03-15T22:25:45.240370+0000","last_unstale":"2021-03-15T22:25:46.308133+0000","last_undegraded":"2021-03-15T22:25:46.308133+0000","last_fullsized":"2021-03-15T22:25:46.308133+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'777","last_scrub_stamp":"2021-03-15T22:24:40.640400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.640400+0000","log_size":998,"ondisk_log_size":998,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67920655,"num_objects":33,"num_object_clones":26,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":1,"num_read":588,"num_read_kb":641818,"num_write":363,"num_write_kb":145036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":125184016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"},{"start":"142","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"394","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.243353+0000","last_change":"2021-03-15T22:24:30.950198+0000","last_active":"2021-03-15T22:25:46.243353+0000","last_peered":"2021-03-15T22:25:46.243353+0000","last_clean":"2021-03-15T22:25:46.243353+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:25:46.243353+0000","last_undegraded":"2021-03-15T22:25:46.243353+0000","last_fullsized":"2021-03-15T22:25:46.243353+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.950135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.950135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"400","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.249700+0000","last_change":"2021-03-15T22:24:26.962034+0000","last_active":"2021-03-15T22:25:46.249700+0000","last_peered":"2021-03-15T22:25:46.249700+0000","last_clean":"2021-03-15T22:25:46.249700+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:25:46.249700+0000","last_undegraded":"2021-03-15T22:25:46.249700+0000","last_fullsized":"2021-03-15T22:25:46.249700+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.961978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.961978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"176","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.244306+0000","last_change":"2021-03-15T22:24:31.147838+0000","last_active":"2021-03-15T22:25:46.244306+0000","last_peered":"2021-03-15T22:25:46.244306+0000","last_clean":"2021-03-15T22:25:46.244306+0000","last_became_active":"2021-03-15T22:24:31.098665+0000","last_became_peered":"2021-03-15T22:24:31.098665+0000","last_unstale":"2021-03-15T22:25:46.244306+0000","last_undegraded":"2021-03-15T22:25:46.244306+0000","last_fullsized":"2021-03-15T22:25:46.244306+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:23:36.678184+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:23:36.678184+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"409","reported_epoch":"434","state":"active+clean","last_fresh":"2021-03-15T22:25:49.279557+0000","last_change":"2021-03-15T22:24:34.454377+0000","last_active":"2021-03-15T22:25:49.279557+0000","last_peered":"2021-03-15T22:25:49.279557+0000","last_clean":"2021-03-15T22:25:49.279557+0000","last_became_active":"2021-03-15T22:24:31.107629+0000","last_became_peered":"2021-03-15T22:24:31.107629+0000","last_unstale":"2021-03-15T22:25:49.279557+0000","last_undegraded":"2021-03-15T22:25:49.279557+0000","last_fullsized":"2021-03-15T22:25:49.279557+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.454313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.454313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"431'158","reported_seq":"1263","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.240745+0000","last_change":"2021-03-15T22:25:46.240976+0000","last_active":"2021-03-15T22:25:46.240745+0000","last_peered":"2021-03-15T22:25:46.240745+0000","last_clean":"2021-03-15T22:25:46.240745+0000","last_became_active":"2021-03-15T22:25:37.112715+0000","last_became_peered":"2021-03-15T22:25:37.112715+0000","last_unstale":"2021-03-15T22:25:46.240745+0000","last_undegraded":"2021-03-15T22:25:46.240745+0000","last_fullsized":"2021-03-15T22:25:46.240745+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'104","last_scrub_stamp":"2021-03-15T22:24:02.949625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:02.949625+0000","log_size":158,"ondisk_log_size":158,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16081782,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":81,"num_read_kb":100355,"num_write":58,"num_write_kb":24058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":14412391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"},{"start":"142","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"429","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.240019+0000","last_change":"2021-03-15T22:25:14.112119+0000","last_active":"2021-03-15T22:25:45.240019+0000","last_peered":"2021-03-15T22:25:45.240019+0000","last_clean":"2021-03-15T22:25:45.240019+0000","last_became_active":"2021-03-15T22:25:14.111797+0000","last_became_peered":"2021-03-15T22:25:14.111797+0000","last_unstale":"2021-03-15T22:25:45.240019+0000","last_undegraded":"2021-03-15T22:25:45.240019+0000","last_fullsized":"2021-03-15T22:25:45.240019+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.480279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"410","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.245185+0000","last_change":"2021-03-15T22:25:33.141491+0000","last_active":"2021-03-15T22:25:46.245185+0000","last_peered":"2021-03-15T22:25:46.245185+0000","last_clean":"2021-03-15T22:25:46.245185+0000","last_became_active":"2021-03-15T22:25:33.141147+0000","last_became_peered":"2021-03-15T22:25:33.141147+0000","last_unstale":"2021-03-15T22:25:46.245185+0000","last_undegraded":"2021-03-15T22:25:46.245185+0000","last_fullsized":"2021-03-15T22:25:46.245185+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.769850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.769850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"523","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.243888+0000","last_change":"2021-03-15T22:25:15.063557+0000","last_active":"2021-03-15T22:25:46.243888+0000","last_peered":"2021-03-15T22:25:46.243888+0000","last_clean":"2021-03-15T22:25:46.243888+0000","last_became_active":"2021-03-15T22:25:14.114107+0000","last_became_peered":"2021-03-15T22:25:14.114107+0000","last_unstale":"2021-03-15T22:25:46.243888+0000","last_undegraded":"2021-03-15T22:25:46.243888+0000","last_fullsized":"2021-03-15T22:25:46.243888+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:23:29.014596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.014596+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"356'3","reported_seq":"121","reported_epoch":"434","state":"active+clean","last_fresh":"2021-03-15T22:25:49.282293+0000","last_change":"2021-03-15T22:25:43.596578+0000","last_active":"2021-03-15T22:25:49.282293+0000","last_peered":"2021-03-15T22:25:49.282293+0000","last_clean":"2021-03-15T22:25:49.282293+0000","last_became_active":"2021-03-15T22:24:31.105948+0000","last_became_peered":"2021-03-15T22:24:31.105948+0000","last_unstale":"2021-03-15T22:25:49.282293+0000","last_undegraded":"2021-03-15T22:25:49.282293+0000","last_fullsized":"2021-03-15T22:25:49.282293+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:24:02.507323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:02.507323+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"431'262","reported_seq":"1312","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.286355+0000","last_change":"2021-03-15T22:25:46.256141+0000","last_active":"2021-03-15T22:25:46.286355+0000","last_peered":"2021-03-15T22:25:46.286355+0000","last_clean":"2021-03-15T22:25:46.286355+0000","last_became_active":"2021-03-15T22:23:00.057259+0000","last_became_peered":"2021-03-15T22:23:00.057259+0000","last_unstale":"2021-03-15T22:25:46.286355+0000","last_undegraded":"2021-03-15T22:25:46.286355+0000","last_fullsized":"2021-03-15T22:25:46.286355+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'202","last_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.006150+0000","log_size":262,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17802057,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":126,"num_read_kb":122013,"num_write":95,"num_write_kb":37334,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":6765552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"},{"start":"142","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"393","reported_epoch":"432","state":"active+clean","last_fresh":"2021-03-15T22:25:47.366346+0000","last_change":"2021-03-15T22:25:16.234631+0000","last_active":"2021-03-15T22:25:47.366346+0000","last_peered":"2021-03-15T22:25:47.366346+0000","last_clean":"2021-03-15T22:25:47.366346+0000","last_became_active":"2021-03-15T22:23:52.260549+0000","last_became_peered":"2021-03-15T22:23:52.260549+0000","last_unstale":"2021-03-15T22:25:47.366346+0000","last_undegraded":"2021-03-15T22:25:47.366346+0000","last_fullsized":"2021-03-15T22:25:47.366346+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.234565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.234565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"94","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.249955+0000","last_change":"2021-03-15T22:25:02.868460+0000","last_active":"2021-03-15T22:25:46.249955+0000","last_peered":"2021-03-15T22:25:46.249955+0000","last_clean":"2021-03-15T22:25:46.249955+0000","last_became_active":"2021-03-15T22:24:31.099610+0000","last_became_peered":"2021-03-15T22:24:31.099610+0000","last_unstale":"2021-03-15T22:25:46.249955+0000","last_undegraded":"2021-03-15T22:25:46.249955+0000","last_fullsized":"2021-03-15T22:25:46.249955+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.868399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.868399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"430","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.236180+0000","last_change":"2021-03-15T22:25:39.041168+0000","last_active":"2021-03-15T22:25:45.236180+0000","last_peered":"2021-03-15T22:25:45.236180+0000","last_clean":"2021-03-15T22:25:45.236180+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:25:45.236180+0000","last_undegraded":"2021-03-15T22:25:45.236180+0000","last_fullsized":"2021-03-15T22:25:45.236180+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:23:07.762338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:07.762338+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"421'6","reported_seq":"29","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239007+0000","last_change":"2021-03-15T22:25:33.171536+0000","last_active":"2021-03-15T22:25:45.239007+0000","last_peered":"2021-03-15T22:25:45.239007+0000","last_clean":"2021-03-15T22:25:45.239007+0000","last_became_active":"2021-03-15T22:25:33.130973+0000","last_became_peered":"2021-03-15T22:25:33.130973+0000","last_unstale":"2021-03-15T22:25:45.239007+0000","last_undegraded":"2021-03-15T22:25:45.239007+0000","last_fullsized":"2021-03-15T22:25:45.239007+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'4","last_scrub_stamp":"2021-03-15T22:23:35.100079+0000","last_deep_scrub":"300'4","last_deep_scrub_stamp":"2021-03-15T22:23:35.100079+0000","last_clean_scrub_stamp":"2021-03-15T22:23:35.100079+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"431'447","reported_seq":"1594","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.253772+0000","last_change":"2021-03-15T22:25:46.255552+0000","last_active":"2021-03-15T22:25:46.253772+0000","last_peered":"2021-03-15T22:25:46.253772+0000","last_clean":"2021-03-15T22:25:46.253772+0000","last_became_active":"2021-03-15T22:23:01.072357+0000","last_became_peered":"2021-03-15T22:23:01.072357+0000","last_unstale":"2021-03-15T22:25:46.253772+0000","last_undegraded":"2021-03-15T22:25:46.253772+0000","last_fullsized":"2021-03-15T22:25:46.253772+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'302","last_scrub_stamp":"2021-03-15T22:23:59.467761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:59.467761+0000","log_size":447,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26858891,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":241,"num_read_kb":259097,"num_write":146,"num_write_kb":58613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31233387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"},{"start":"142","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"409","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.235965+0000","last_change":"2021-03-15T22:25:33.136943+0000","last_active":"2021-03-15T22:25:45.235965+0000","last_peered":"2021-03-15T22:25:45.235965+0000","last_clean":"2021-03-15T22:25:45.235965+0000","last_became_active":"2021-03-15T22:25:33.136621+0000","last_became_peered":"2021-03-15T22:25:33.136621+0000","last_unstale":"2021-03-15T22:25:45.235965+0000","last_undegraded":"2021-03-15T22:25:45.235965+0000","last_fullsized":"2021-03-15T22:25:45.235965+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:32.751660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:32.751660+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"199","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.240270+0000","last_change":"2021-03-15T22:25:12.911203+0000","last_active":"2021-03-15T22:25:45.240270+0000","last_peered":"2021-03-15T22:25:45.240270+0000","last_clean":"2021-03-15T22:25:45.240270+0000","last_became_active":"2021-03-15T22:24:31.091206+0000","last_became_peered":"2021-03-15T22:24:31.091206+0000","last_unstale":"2021-03-15T22:25:45.240270+0000","last_undegraded":"2021-03-15T22:25:45.240270+0000","last_fullsized":"2021-03-15T22:25:45.240270+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.911140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.911140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"49","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.238875+0000","last_change":"2021-03-15T22:25:33.758535+0000","last_active":"2021-03-15T22:25:45.238875+0000","last_peered":"2021-03-15T22:25:45.238875+0000","last_clean":"2021-03-15T22:25:45.238875+0000","last_became_active":"2021-03-15T22:25:33.149362+0000","last_became_peered":"2021-03-15T22:25:33.149362+0000","last_unstale":"2021-03-15T22:25:45.238875+0000","last_undegraded":"2021-03-15T22:25:45.238875+0000","last_fullsized":"2021-03-15T22:25:45.238875+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:24:01.896988+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.896988+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"422","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.235926+0000","last_change":"2021-03-15T22:25:33.143430+0000","last_active":"2021-03-15T22:25:45.235926+0000","last_peered":"2021-03-15T22:25:45.235926+0000","last_clean":"2021-03-15T22:25:45.235926+0000","last_became_active":"2021-03-15T22:25:33.142712+0000","last_became_peered":"2021-03-15T22:25:33.142712+0000","last_unstale":"2021-03-15T22:25:45.235926+0000","last_undegraded":"2021-03-15T22:25:45.235926+0000","last_fullsized":"2021-03-15T22:25:45.235926+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.989313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.989313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"434'425","reported_seq":"1601","reported_epoch":"435","state":"active+clean","last_fresh":"2021-03-15T22:25:50.459410+0000","last_change":"2021-03-15T22:25:49.286726+0000","last_active":"2021-03-15T22:25:50.459410+0000","last_peered":"2021-03-15T22:25:50.459410+0000","last_clean":"2021-03-15T22:25:50.459410+0000","last_became_active":"2021-03-15T22:25:36.118668+0000","last_became_peered":"2021-03-15T22:25:36.118668+0000","last_unstale":"2021-03-15T22:25:50.459410+0000","last_undegraded":"2021-03-15T22:25:50.459410+0000","last_fullsized":"2021-03-15T22:25:50.459410+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'370","last_scrub_stamp":"2021-03-15T22:25:07.414191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.414191+0000","log_size":425,"ondisk_log_size":425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26033381,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":182,"num_read_kb":168083,"num_write":134,"num_write_kb":52354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25937939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10e","length":"1"},{"start":"13a","length":"2"}]},{"pgid":"6.d","version":"0'0","reported_seq":"395","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.244964+0000","last_change":"2021-03-15T22:24:31.094621+0000","last_active":"2021-03-15T22:25:46.244964+0000","last_peered":"2021-03-15T22:25:46.244964+0000","last_clean":"2021-03-15T22:25:46.244964+0000","last_became_active":"2021-03-15T22:24:31.094299+0000","last_became_peered":"2021-03-15T22:24:31.094299+0000","last_unstale":"2021-03-15T22:25:46.244964+0000","last_undegraded":"2021-03-15T22:25:46.244964+0000","last_fullsized":"2021-03-15T22:25:46.244964+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.936738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.936738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"391'7","reported_seq":"542","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:46.103595+0000","last_change":"2021-03-15T22:25:46.103567+0000","last_active":"2021-03-15T22:25:46.103595+0000","last_peered":"2021-03-15T22:25:46.103595+0000","last_clean":"2021-03-15T22:25:46.103595+0000","last_became_active":"2021-03-15T22:25:03.403350+0000","last_became_peered":"2021-03-15T22:25:03.403350+0000","last_unstale":"2021-03-15T22:25:46.103595+0000","last_undegraded":"2021-03-15T22:25:46.103595+0000","last_fullsized":"2021-03-15T22:25:46.103595+0000","mapping_epoch":387,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'7","last_scrub_stamp":"2021-03-15T22:25:46.103504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.103504+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"28","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.238954+0000","last_change":"2021-03-15T22:25:44.867726+0000","last_active":"2021-03-15T22:25:45.238954+0000","last_peered":"2021-03-15T22:25:45.238954+0000","last_clean":"2021-03-15T22:25:45.238954+0000","last_became_active":"2021-03-15T22:25:33.148614+0000","last_became_peered":"2021-03-15T22:25:33.148614+0000","last_unstale":"2021-03-15T22:25:45.238954+0000","last_undegraded":"2021-03-15T22:25:45.238954+0000","last_fullsized":"2021-03-15T22:25:45.238954+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:44.867633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:44.867633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"464","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267544+0000","last_change":"2021-03-15T22:25:46.725552+0000","last_active":"2021-03-15T22:25:48.267544+0000","last_peered":"2021-03-15T22:25:48.267544+0000","last_clean":"2021-03-15T22:25:48.267544+0000","last_became_active":"2021-03-15T22:22:56.057343+0000","last_became_peered":"2021-03-15T22:22:56.057343+0000","last_unstale":"2021-03-15T22:25:48.267544+0000","last_undegraded":"2021-03-15T22:25:48.267544+0000","last_fullsized":"2021-03-15T22:25:48.267544+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:25:46.725489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.725489+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"434'89","reported_seq":"1172","reported_epoch":"435","state":"active+clean","last_fresh":"2021-03-15T22:25:50.459332+0000","last_change":"2021-03-15T22:25:49.284471+0000","last_active":"2021-03-15T22:25:50.459332+0000","last_peered":"2021-03-15T22:25:50.459332+0000","last_clean":"2021-03-15T22:25:50.459332+0000","last_became_active":"2021-03-15T22:24:31.104356+0000","last_became_peered":"2021-03-15T22:24:31.104356+0000","last_unstale":"2021-03-15T22:25:50.459332+0000","last_undegraded":"2021-03-15T22:25:50.459332+0000","last_fullsized":"2021-03-15T22:25:50.459332+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'77","last_scrub_stamp":"2021-03-15T22:25:32.490694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.490694+0000","log_size":89,"ondisk_log_size":89,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7752259,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":89949,"num_write":32,"num_write_kb":13772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1551588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"10e","length":"1"},{"start":"13a","length":"2"}]},{"pgid":"6.c","version":"0'0","reported_seq":"23","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239407+0000","last_change":"2021-03-15T22:25:33.148157+0000","last_active":"2021-03-15T22:25:45.239407+0000","last_peered":"2021-03-15T22:25:45.239407+0000","last_clean":"2021-03-15T22:25:45.239407+0000","last_became_active":"2021-03-15T22:25:33.147717+0000","last_became_peered":"2021-03-15T22:25:33.147717+0000","last_unstale":"2021-03-15T22:25:45.239407+0000","last_undegraded":"2021-03-15T22:25:45.239407+0000","last_fullsized":"2021-03-15T22:25:45.239407+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.372535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.372535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"147'5","reported_seq":"470","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:47.087826+0000","last_change":"2021-03-15T22:24:35.715464+0000","last_active":"2021-03-15T22:25:47.087826+0000","last_peered":"2021-03-15T22:25:47.087826+0000","last_clean":"2021-03-15T22:25:47.087826+0000","last_became_active":"2021-03-15T22:21:02.017066+0000","last_became_peered":"2021-03-15T22:21:02.017066+0000","last_unstale":"2021-03-15T22:25:47.087826+0000","last_undegraded":"2021-03-15T22:25:47.087826+0000","last_fullsized":"2021-03-15T22:25:47.087826+0000","mapping_epoch":60,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'5","last_scrub_stamp":"2021-03-15T22:24:35.715399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:35.715399+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"399","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.236014+0000","last_change":"2021-03-15T22:24:34.991087+0000","last_active":"2021-03-15T22:25:45.236014+0000","last_peered":"2021-03-15T22:25:45.236014+0000","last_clean":"2021-03-15T22:25:45.236014+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:25:45.236014+0000","last_undegraded":"2021-03-15T22:25:45.236014+0000","last_fullsized":"2021-03-15T22:25:45.236014+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.991018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.991018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"480","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.238155+0000","last_change":"2021-03-15T22:24:55.128840+0000","last_active":"2021-03-15T22:25:45.238155+0000","last_peered":"2021-03-15T22:25:45.238155+0000","last_clean":"2021-03-15T22:25:45.238155+0000","last_became_active":"2021-03-15T22:24:55.090544+0000","last_became_peered":"2021-03-15T22:24:55.090544+0000","last_unstale":"2021-03-15T22:25:45.238155+0000","last_undegraded":"2021-03-15T22:25:45.238155+0000","last_fullsized":"2021-03-15T22:25:45.238155+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:38.685706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:38.685706+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"431'127","reported_seq":"1316","reported_epoch":"434","state":"active+clean","last_fresh":"2021-03-15T22:25:49.277380+0000","last_change":"2021-03-15T22:25:49.277741+0000","last_active":"2021-03-15T22:25:49.277380+0000","last_peered":"2021-03-15T22:25:49.277380+0000","last_clean":"2021-03-15T22:25:49.277380+0000","last_became_active":"2021-03-15T22:25:37.119078+0000","last_became_peered":"2021-03-15T22:25:37.119078+0000","last_unstale":"2021-03-15T22:25:49.277380+0000","last_undegraded":"2021-03-15T22:25:49.277380+0000","last_fullsized":"2021-03-15T22:25:49.277380+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'115","last_scrub_stamp":"2021-03-15T22:25:10.616929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.616929+0000","log_size":127,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9539709,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":110,"num_read_kb":136588,"num_write":53,"num_write_kb":14217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":16774328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"4"},{"start":"142","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"414","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267794+0000","last_change":"2021-03-15T22:25:38.702159+0000","last_active":"2021-03-15T22:25:48.267794+0000","last_peered":"2021-03-15T22:25:48.267794+0000","last_clean":"2021-03-15T22:25:48.267794+0000","last_became_active":"2021-03-15T22:25:33.134275+0000","last_became_peered":"2021-03-15T22:25:33.134275+0000","last_unstale":"2021-03-15T22:25:48.267794+0000","last_undegraded":"2021-03-15T22:25:48.267794+0000","last_fullsized":"2021-03-15T22:25:48.267794+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:23:02.474236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"423","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.249929+0000","last_change":"2021-03-15T22:24:31.107075+0000","last_active":"2021-03-15T22:25:46.249929+0000","last_peered":"2021-03-15T22:25:46.249929+0000","last_clean":"2021-03-15T22:25:46.249929+0000","last_became_active":"2021-03-15T22:24:31.106634+0000","last_became_peered":"2021-03-15T22:24:31.106634+0000","last_unstale":"2021-03-15T22:25:46.249929+0000","last_undegraded":"2021-03-15T22:25:46.249929+0000","last_fullsized":"2021-03-15T22:25:46.249929+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:42.177422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.177422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"407","reported_epoch":"434","state":"active+clean","last_fresh":"2021-03-15T22:25:49.279351+0000","last_change":"2021-03-15T22:24:28.327738+0000","last_active":"2021-03-15T22:25:49.279351+0000","last_peered":"2021-03-15T22:25:49.279351+0000","last_clean":"2021-03-15T22:25:49.279351+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:25:49.279351+0000","last_undegraded":"2021-03-15T22:25:49.279351+0000","last_fullsized":"2021-03-15T22:25:49.279351+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:28.327677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.327677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"115'33","reported_seq":"445","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.237766+0000","last_change":"2021-03-15T22:25:04.028853+0000","last_active":"2021-03-15T22:25:45.237766+0000","last_peered":"2021-03-15T22:25:45.237766+0000","last_clean":"2021-03-15T22:25:45.237766+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:25:45.237766+0000","last_undegraded":"2021-03-15T22:25:45.237766+0000","last_fullsized":"2021-03-15T22:25:45.237766+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'33","last_scrub_stamp":"2021-03-15T22:25:04.028780+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:25:04.028780+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"410'378","reported_seq":"1659","reported_epoch":"431","state":"active+clean","last_fresh":"2021-03-15T22:25:46.249183+0000","last_change":"2021-03-15T22:25:46.249829+0000","last_active":"2021-03-15T22:25:46.249183+0000","last_peered":"2021-03-15T22:25:46.249183+0000","last_clean":"2021-03-15T22:25:46.249183+0000","last_became_active":"2021-03-15T22:25:39.143407+0000","last_became_peered":"2021-03-15T22:25:39.143407+0000","last_unstale":"2021-03-15T22:25:46.249183+0000","last_undegraded":"2021-03-15T22:25:46.249183+0000","last_fullsized":"2021-03-15T22:25:46.249183+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'343","last_scrub_stamp":"2021-03-15T22:23:26.791043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.791043+0000","log_size":378,"ondisk_log_size":378,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7647416,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":183,"num_read_kb":156184,"num_write":134,"num_write_kb":53144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":38892785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"10e","length":"1"},{"start":"12b","length":"1"},{"start":"134","length":"1"},{"start":"138","length":"2"},{"start":"142","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"391","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.238323+0000","last_change":"2021-03-15T22:24:40.910630+0000","last_active":"2021-03-15T22:25:45.238323+0000","last_peered":"2021-03-15T22:25:45.238323+0000","last_clean":"2021-03-15T22:25:45.238323+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:25:45.238323+0000","last_undegraded":"2021-03-15T22:25:45.238323+0000","last_fullsized":"2021-03-15T22:25:45.238323+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.910565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.910565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"155'4","reported_seq":"516","reported_epoch":"433","state":"active+clean","last_fresh":"2021-03-15T22:25:48.267428+0000","last_change":"2021-03-15T22:24:29.903505+0000","last_active":"2021-03-15T22:25:48.267428+0000","last_peered":"2021-03-15T22:25:48.267428+0000","last_clean":"2021-03-15T22:25:48.267428+0000","last_became_active":"2021-03-15T22:21:10.013838+0000","last_became_peered":"2021-03-15T22:21:10.013838+0000","last_unstale":"2021-03-15T22:25:48.267428+0000","last_undegraded":"2021-03-15T22:25:48.267428+0000","last_fullsized":"2021-03-15T22:25:48.267428+0000","mapping_epoch":60,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'4","last_scrub_stamp":"2021-03-15T22:24:29.903442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.903442+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"109","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.239408+0000","last_change":"2021-03-15T22:25:02.791176+0000","last_active":"2021-03-15T22:25:45.239408+0000","last_peered":"2021-03-15T22:25:45.239408+0000","last_clean":"2021-03-15T22:25:45.239408+0000","last_became_active":"2021-03-15T22:24:31.097434+0000","last_became_peered":"2021-03-15T22:24:31.097434+0000","last_unstale":"2021-03-15T22:25:45.239408+0000","last_undegraded":"2021-03-15T22:25:45.239408+0000","last_fullsized":"2021-03-15T22:25:45.239408+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.791108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:06.573181+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.791108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"472","reported_epoch":"430","state":"active+clean","last_fresh":"2021-03-15T22:25:45.236836+0000","last_change":"2021-03-15T22:25:34.027206+0000","last_active":"2021-03-15T22:25:45.236836+0000","last_peered":"2021-03-15T22:25:45.236836+0000","last_clean":"2021-03-15T22:25:45.236836+0000","last_became_active":"2021-03-15T22:22:56.053800+0000","last_became_peered":"2021-03-15T22:22:56.053800+0000","last_unstale":"2021-03-15T22:25:45.236836+0000","last_undegraded":"2021-03-15T22:25:45.236836+0000","last_fullsized":"2021-03-15T22:25:45.236836+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:23:05.791845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:05.791845+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":386988459,"num_objects":191,"num_object_clones":142,"num_object_copies":573,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":191,"num_whiteouts":15,"num_read":3144,"num_read_kb":3180723,"num_write":2279,"num_write_kb":866322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1197,"num_bytes_recovered":514315662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":913403904,"data_stored":1251424103,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6214,"ondisk_log_size":6214,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":461,"num_read_kb":456,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1061,"num_read_kb":566,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":37148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2232320,"data_stored":80092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299491,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124636,"kb_used_data":106452,"kb_used_omap":0,"kb_used_meta":18176,"kb_avail":93620516,"statfs":{"total":95995035648,"available":95867408384,"internally_reserved":0,"allocated":109006848,"data_stored":147562690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18612224},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.0189999999999999,"5min":2.73,"15min":2.73},"last":0.377},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65000000000000002,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.504,"15min":0.48599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":0.97299999999999998,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.436,"5min":0.48499999999999999,"15min":0.48999999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72699999999999998,"5min":1.877,"15min":1.877},"last":0.48499999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.47199999999999998,"15min":0.47899999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73499999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.496},{"interface":"front","average":{"1min":0.498,"5min":0.51900000000000002,"15min":0.5},"min":{"1min":0.311,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":0.95099999999999996,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.50600000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.54200000000000004},"min":{"1min":0.315,"5min":0.31,"15min":0.31},"max":{"1min":0.69499999999999995,"5min":0.95999999999999996,"15min":2.0339999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98899999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.47799999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.46600000000000003,"15min":0.436},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.75700000000000001,"5min":1.835,"15min":1.835},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47699999999999998,"15min":0.44600000000000001},"min":{"1min":0.29099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66200000000000003,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.46200000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52200000000000002,"15min":0.45800000000000002},"min":{"1min":0.31900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.034,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.53100000000000003,"15min":0.504},"min":{"1min":0.33900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.78300000000000003,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.32200000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54600000000000004,"15min":0.51000000000000001},"min":{"1min":0.34499999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80700000000000005,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.53700000000000003,"15min":0.50900000000000001},"min":{"1min":0.35299999999999998,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.77700000000000002,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.53900000000000003}]}]},{"osd":6,"up_from":45,"seq":193273528423,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124480,"kb_used_data":96376,"kb_used_omap":0,"kb_used_meta":28096,"kb_avail":93620672,"statfs":{"total":95995035648,"available":95867568128,"internally_reserved":0,"allocated":98689024,"data_stored":130577076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28770304},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47499999999999998,"15min":0.48799999999999999},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.68100000000000005,"5min":0.995,"15min":0.995},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.46800000000000003,"15min":0.48599999999999999},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.372}]},{"osd":1,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.49099999999999999,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.254,"15min":0.254},"max":{"1min":0.69399999999999995,"5min":1.05,"15min":1.05},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.33300000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68000000000000005,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.32600000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.317,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65200000000000002,"5min":0.77600000000000002,"15min":0.80800000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48899999999999999,"15min":0.498},"min":{"1min":0.35399999999999998,"5min":0.32800000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.35199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.90400000000000003,"5min":1.026,"15min":1.026},"last":0.499},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51100000000000001,"15min":0.52400000000000002},"min":{"1min":0.33600000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.47699999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.44800000000000001,"15min":0.41799999999999998},"min":{"1min":0.28000000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51500000000000001,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.501,"15min":0.46200000000000002},"min":{"1min":0.32300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.89200000000000002,"5min":1.135,"15min":1.135},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53200000000000003,"15min":0.48999999999999999},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.436}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52300000000000002,"15min":0.50600000000000001},"min":{"1min":0.36799999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.79600000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.76100000000000001,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.46700000000000003}]}]},{"osd":1,"up_from":13,"seq":55834574968,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172376,"kb_used_data":147344,"kb_used_omap":0,"kb_used_meta":25024,"kb_avail":93572776,"statfs":{"total":95995035648,"available":95818522624,"internally_reserved":0,"allocated":150880256,"data_stored":209556296,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25624576},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.34499999999999997,"15min":0.52700000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.48899999999999999,"5min":1.371,"15min":54.427999999999997},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.54400000000000004},"min":{"1min":0.20999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.48199999999999998,"5min":1.23,"15min":54.356999999999999},"last":0.26000000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.54300000000000004,"5min":1.1220000000000001,"15min":2.4100000000000001},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.38600000000000001},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.502,"5min":0.69799999999999995,"15min":2.0760000000000001},"last":0.33900000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:24:57 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38500000000000001,"15min":0.41899999999999998},"min":{"1min":0.26100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.796},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40799999999999997,"15min":0.46400000000000002},"min":{"1min":0.29299999999999998,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.623,"5min":1.135,"15min":1.7150000000000001},"last":0.39400000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:25:10 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.95199999999999996,"5min":3.3439999999999999,"15min":3.3439999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.621,"15min":0.626},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.93999999999999995,"5min":4.6139999999999999,"15min":4.6139999999999999},"last":0.47199999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.627},"min":{"1min":0.42199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.93400000000000005,"5min":1.524,"15min":2.153},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58499999999999996,"15min":0.95899999999999996},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88900000000000001,"5min":1.1950000000000001,"15min":15.472},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61099999999999999,"15min":0.65600000000000003},"min":{"1min":0.47599999999999998,"5min":0.36899999999999999,"15min":0.33200000000000002},"max":{"1min":0.91800000000000004,"5min":1.276,"15min":1.75},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.3580000000000001},"last":0.61399999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.63500000000000001,"15min":0.871},"min":{"1min":0.377,"5min":0.316,"15min":0.316},"max":{"1min":0.96299999999999997,"5min":2.1219999999999999,"15min":8.8100000000000005},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.623},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84799999999999998,"5min":1.0329999999999999,"15min":1.325},"last":0.63700000000000001}]}]},{"osd":0,"up_from":0,"seq":34359738491,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411445,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":127580,"kb_used_data":101396,"kb_used_omap":0,"kb_used_meta":26176,"kb_avail":93617572,"statfs":{"total":95995035648,"available":95864393728,"internally_reserved":0,"allocated":103829504,"data_stored":141531546,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26804224},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.246,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.45700000000000002,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.246},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.437,"15min":0.39500000000000002},"min":{"1min":0.246,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":5.5250000000000004,"5min":5.5250000000000004,"15min":5.5250000000000004},"last":0.27800000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.42599999999999999,"15min":0.39600000000000002},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.60999999999999999,"5min":5.4489999999999998,"15min":5.4489999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51800000000000002,"15min":0.46600000000000003},"min":{"1min":0.27400000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.3740000000000001,"5min":11.659000000000001,"15min":11.659000000000001},"last":0.35799999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:24:53 2021","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.38400000000000001,"15min":0.41799999999999998},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.55600000000000005,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.443,"15min":0.48299999999999998},"min":{"1min":0.27400000000000002,"5min":0.26400000000000001,"15min":0.24099999999999999},"max":{"1min":2.8759999999999999,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.33800000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:25:09 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56999999999999995,"15min":0.57499999999999996},"min":{"1min":0.40999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73899999999999999,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54500000000000004,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.60199999999999998,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.48399999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:25:36 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.38400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75600000000000001,"5min":1.954,"15min":1.954},"last":0.433},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.745,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.53800000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":1.875,"15min":1.875},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.435,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.3520000000000001,"5min":1.609,"15min":1.609},"last":0.45400000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:25:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.66900000000000004,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.40600000000000003,"15min":0.38700000000000001},"max":{"1min":0.64700000000000002,"5min":1.923,"15min":1.923},"last":0.45900000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149808,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":266408,"kb_used_data":241056,"kb_used_omap":0,"kb_used_meta":25344,"kb_avail":93478744,"statfs":{"total":95995035648,"available":95722233856,"internally_reserved":0,"allocated":246841344,"data_stored":318842365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25952256},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.35199999999999998,"15min":0.35799999999999998},"min":{"1min":0.22700000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.35699999999999998,"5min":2.625,"15min":2.625},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.36499999999999999,"15min":0.379},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.189},"max":{"1min":0.438,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.26600000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.44900000000000001,"15min":0.434},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.50700000000000001,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.372},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.46100000000000002,"15min":0.437},"min":{"1min":0.28699999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.45300000000000001,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.38900000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.45000000000000001,"15min":0.435},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.48199999999999998,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.316,"5min":0.42299999999999999,"15min":0.40400000000000003},"min":{"1min":0.26600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46300000000000002,"5min":7.0860000000000003,"15min":7.0860000000000003},"last":0.29699999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:25:10 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55500000000000005,"15min":0.52400000000000002},"min":{"1min":0.32600000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.66800000000000004,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.318,"15min":0.30399999999999999},"max":{"1min":0.69299999999999995,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":1.0409999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.54500000000000004},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.62,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":0.77700000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:25:43 2021","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68500000000000005,"5min":2.6499999999999999,"15min":2.6499999999999999},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70999999999999996,"5min":1.268,"15min":1.268},"last":0.69499999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.58399999999999996,"15min":0.57699999999999996},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67400000000000004,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.67000000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.377},"max":{"1min":0.72699999999999998,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":1.0129999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986285,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101948,"kb_used_data":69876,"kb_used_omap":0,"kb_used_meta":32064,"kb_avail":93643204,"statfs":{"total":95995035648,"available":95890640896,"internally_reserved":0,"allocated":71553024,"data_stored":95293131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32833536},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.19500000000000001},"max":{"1min":0.73099999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48999999999999999},"min":{"1min":0.34999999999999998,"5min":0.29099999999999998,"15min":0.251},"max":{"1min":0.66600000000000004,"5min":0.72199999999999998,"15min":0.74199999999999999},"last":0.46500000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.47599999999999998,"15min":0.5},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23599999999999999},"max":{"1min":0.68100000000000005,"5min":1.631,"15min":1.631},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26800000000000002},"max":{"1min":0.73999999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.51100000000000001},"min":{"1min":0.40300000000000002,"5min":0.32300000000000001,"15min":0.313},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.76100000000000001},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":1.1910000000000001},"last":0.501}]},{"osd":3,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52600000000000002,"15min":0.55000000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68899999999999995,"5min":0.75,"15min":0.98799999999999999},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51500000000000001,"15min":0.55800000000000005},"min":{"1min":0.40200000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68899999999999995,"5min":0.69199999999999995,"15min":0.99399999999999999},"last":0.54400000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:25:28 2021","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.41799999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.49299999999999999,"15min":0.441},"min":{"1min":0.36099999999999999,"5min":0.29699999999999999,"15min":0.28799999999999998},"max":{"1min":0.64400000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.35099999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46800000000000003,"15min":0.437},"min":{"1min":0.29099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65400000000000003,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53900000000000003,"15min":0.51400000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.71499999999999997,"5min":0.78400000000000003,"15min":0.84799999999999998},"last":0.41599999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.48899999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.70899999999999996,"5min":0.73799999999999999,"15min":0.80600000000000005},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55400000000000005,"15min":0.52500000000000002},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.75,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.39500000000000002}]}]},{"osd":5,"up_from":38,"seq":163208757354,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":166576,"kb_used_data":147240,"kb_used_omap":0,"kb_used_meta":19328,"kb_avail":93578576,"statfs":{"total":95995035648,"available":95824461824,"internally_reserved":0,"allocated":150773760,"data_stored":212519253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19791872},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48799999999999999,"15min":0.48299999999999998},"min":{"1min":0.29499999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63500000000000001,"5min":2.855,"15min":2.855},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.498},"min":{"1min":0.34699999999999998,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.75700000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.48399999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.33100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.72999999999999998,"5min":2.665,"15min":2.665},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.72099999999999997,"5min":0.81999999999999995,"15min":0.83999999999999997},"last":0.502}]},{"osd":2,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.39000000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61799999999999999,"5min":2.827,"15min":2.827},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.51400000000000001},"min":{"1min":0.29899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.52300000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.55100000000000005,"15min":0.51800000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54800000000000004,"15min":0.53300000000000003},"min":{"1min":0.39400000000000002,"5min":0.34100000000000003,"15min":0.33200000000000002},"max":{"1min":0.65500000000000003,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.49199999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46999999999999997,"15min":0.40899999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24099999999999999},"max":{"1min":0.70899999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.505,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.25},"max":{"1min":0.748,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.34200000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:25:45 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49399999999999999,"15min":0.47199999999999998},"min":{"1min":0.32300000000000001,"5min":0.29199999999999998,"15min":0.28499999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.435},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56599999999999995,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.32800000000000001},"max":{"1min":0.78600000000000003,"5min":1.327,"15min":1.327},"last":0.42399999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.34699999999999998,"15min":0.30499999999999999},"max":{"1min":0.872,"5min":1.3,"15min":1.3},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57699999999999996,"15min":0.53500000000000003},"min":{"1min":0.38,"5min":0.307,"15min":0.307},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.46400000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":9520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":14458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":8356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":6840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":8774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5443584,"data_stored":5395029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":146210816,"data_stored":206680531,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":99319808,"data_stored":138645093,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":245657600,"data_stored":320806143,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":71819264,"data_stored":97881079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":146329600,"data_stored":209651072,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":94138368,"data_stored":127680933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":104484864,"data_stored":144684223,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:25:53.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:53.753 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:53.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:53.855 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:53.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:53.879 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:53.879 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:53.981 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:54.004 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:54.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:54.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:54.107 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:54.129 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:54.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:54.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:54.132 INFO:tasks.rados.rados.0.smithi076.stdout:1622: done (1 left) 2021-03-15T22:25:54.133 INFO:tasks.rados.rados.0.smithi076.stdout:1623: copy_from oid 31 from oid 34 current snap is 179 2021-03-15T22:25:54.133 INFO:tasks.rados.rados.0.smithi076.stdout:1621: finishing write tid 1 to smithi07694878-43 2021-03-15T22:25:54.133 INFO:tasks.rados.rados.0.smithi076.stdout:1621: finishing write tid 2 to smithi07694878-43 2021-03-15T22:25:54.133 INFO:tasks.rados.rados.0.smithi076.stdout:1621: finishing write tid 3 to smithi07694878-43 2021-03-15T22:25:54.134 INFO:tasks.rados.rados.0.smithi076.stdout:1621: finishing write tid 4 to smithi07694878-43 2021-03-15T22:25:54.134 INFO:tasks.rados.rados.0.smithi076.stdout:1621: finishing write tid 5 to smithi07694878-43 2021-03-15T22:25:54.134 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 781 (ObjNum 572 snap 179 seq_num 572) dirty exists 2021-03-15T22:25:54.134 INFO:tasks.rados.rados.0.smithi076.stdout:1621: left oid 43 (ObjNum 572 snap 179 seq_num 572) 2021-03-15T22:25:54.134 INFO:tasks.rados.rados.0.smithi076.stdout:1621: done (1 left) 2021-03-15T22:25:54.135 INFO:tasks.rados.rados.0.smithi076.stdout:1624: snap_remove snap 165 2021-03-15T22:25:54.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:54.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:54.255 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:54.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:54.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:54.380 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:54.381 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:54.482 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:54.491 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:54 smithi125 conmon[56739]: cluster 2021-03-15T22:25:53.060106+0000 mgr.y (mgr.14144) 642 : cluster 2021-03-15T22:25:54.491 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:54 smithi125 conmon[56739]: [DBG] pgmap v783: 138 pgs: 138 active+clean; 381 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2021-03-15T22:25:54.491 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:54 smithi125 conmon[56739]: cluster 2021-03-15T22:25:53.125051+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2021-03-15T22:25:54.491 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:54 smithi125 conmon[56739]: cluster 2021-03-15T22:25:53.782367+0000 osd.0 (osd.0) 77 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:54.492 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:54 smithi125 conmon[56739]: cluster 2021-03-15T22:25:53.782642+0000 osd.0 (osd.0) 78 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:54.505 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:54.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:54.507 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:54.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:54 smithi076 conmon[56514]: cluster 2021-03-15T22:25:53.060106+0000 mgr.y (mgr.14144) 642 : cluster [DBG] pgmap v783: 138 pgs: 138 active+clean; 381 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2021-03-15T22:25:54.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:54 smithi076 conmon[56514]: cluster 2021-03-15T22:25:53.125051+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2021-03-15T22:25:54.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:54 smithi076 conmon[56514]: cluster 2021-03-15T22:25:53.782367+0000 osd.0 (osd.0) 77 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:54.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:54 smithi076 conmon[56514]: cluster 2021-03-15T22:25:53.782642+0000 osd.0 (osd.0) 78 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:54.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:54 smithi076 conmon[62156]: cluster 2021-03-15T22:25:53.060106+0000 mgr.y (mgr.14144) 642 : cluster [DBG] pgmap v783: 138 pgs: 138 active+clean; 381 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 2.3 MiB/s wr, 20 op/s 2021-03-15T22:25:54.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:54 smithi076 conmon[62156]: cluster 2021-03-15T22:25:53.125051+0000 mon.a (mon.0) 1273 : cluster [DBG] osdmap e438: 8 total, 8 up, 7 in 2021-03-15T22:25:54.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:54 smithi076 conmon[62156]: cluster 2021-03-15T22:25:53.782367+0000 osd.0 (osd.0) 77 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:25:54.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:54 smithi076 conmon[62156]: cluster 2021-03-15T22:25:53.782642+0000 osd.0 (osd.0) 78 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:25:54.608 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:54.630 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:54.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:54.631 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:54.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:54.755 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:54.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:54.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:54.773 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:25:54.774 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:25:54.858 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:54.880 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:54.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:54.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:54.983 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:55.005 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:55.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:55.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:55.107 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:55.109 INFO:tasks.rados.rados.0.smithi076.stdout:1624: done (1 left) 2021-03-15T22:25:55.109 INFO:tasks.rados.rados.0.smithi076.stdout:1625: write oid 25 current snap is 179 2021-03-15T22:25:55.109 INFO:tasks.rados.rados.0.smithi076.stdout:1625: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2021-03-15T22:25:55.111 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:25:55.115 INFO:tasks.rados.rados.0.smithi076.stdout:1625: writing smithi07694878-25 from 430411 to 1074563 tid 1 2021-03-15T22:25:55.122 INFO:tasks.rados.rados.0.smithi076.stdout:1625: writing smithi07694878-25 from 1830888 to 2231408 tid 2 2021-03-15T22:25:55.129 INFO:tasks.rados.rados.0.smithi076.stdout:1625: writing smithi07694878-25 from 2935963 to 3338944 tid 3 2021-03-15T22:25:55.130 INFO:tasks.rados.rados.0.smithi076.stdout:1626: write (excl) oid 12 current snap is 179 2021-03-15T22:25:55.130 INFO:tasks.rados.rados.0.smithi076.stdout:1623: finishing copy_from to smithi07694878-31 2021-03-15T22:25:55.130 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 695 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:25:55.130 INFO:tasks.rados.rados.0.smithi076.stdout:1626: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2021-03-15T22:25:55.135 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:55.136 INFO:tasks.rados.rados.0.smithi076.stdout:1626: writing smithi07694878-12 from 447218 to 1166619 tid 1 2021-03-15T22:25:55.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:55.137 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:55.143 INFO:tasks.rados.rados.0.smithi076.stdout:1626: writing smithi07694878-12 from 1889370 to 2549901 tid 2 2021-03-15T22:25:55.149 INFO:tasks.rados.rados.0.smithi076.stdout:1626: writing smithi07694878-12 from 2963386 to 3577639 tid 3 2021-03-15T22:25:55.149 INFO:tasks.rados.rados.0.smithi076.stdout:1627: snap_create 2021-03-15T22:25:55.150 INFO:tasks.rados.rados.0.smithi076.stdout:1623: finishing copy_from racing read to smithi07694878-31 2021-03-15T22:25:55.150 INFO:tasks.rados.rados.0.smithi076.stdout:1625: finishing write tid 1 to smithi07694878-25 2021-03-15T22:25:55.150 INFO:tasks.rados.rados.0.smithi076.stdout:1625: finishing write tid 2 to smithi07694878-25 2021-03-15T22:25:55.150 INFO:tasks.rados.rados.0.smithi076.stdout:1625: finishing write tid 3 to smithi07694878-25 2021-03-15T22:25:55.151 INFO:tasks.rados.rados.0.smithi076.stdout:1625: finishing write tid 4 to smithi07694878-25 2021-03-15T22:25:55.151 INFO:tasks.rados.rados.0.smithi076.stdout:1625: finishing write tid 5 to smithi07694878-25 2021-03-15T22:25:55.151 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 402 (ObjNum 573 snap 179 seq_num 573) dirty exists 2021-03-15T22:25:55.151 INFO:tasks.rados.rados.0.smithi076.stdout:1625: left oid 25 (ObjNum 573 snap 179 seq_num 573) 2021-03-15T22:25:55.152 INFO:tasks.rados.rados.0.smithi076.stdout:1626: finishing write tid 1 to smithi07694878-12 2021-03-15T22:25:55.152 INFO:tasks.rados.rados.0.smithi076.stdout:1626: finishing write tid 2 to smithi07694878-12 2021-03-15T22:25:55.153 INFO:tasks.rados.rados.0.smithi076.stdout:1626: finishing write tid 3 to smithi07694878-12 2021-03-15T22:25:55.153 INFO:tasks.rados.rados.0.smithi076.stdout:1626: finishing write tid 4 to smithi07694878-12 2021-03-15T22:25:55.154 INFO:tasks.rados.rados.0.smithi076.stdout:1626: finishing write tid 5 to smithi07694878-12 2021-03-15T22:25:55.154 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 981 (ObjNum 574 snap 179 seq_num 574) dirty exists 2021-03-15T22:25:55.154 INFO:tasks.rados.rados.0.smithi076.stdout:1626: left oid 12 (ObjNum 574 snap 179 seq_num 574) 2021-03-15T22:25:55.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:55.260 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:55.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:55.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:55.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:55.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-15T22:25:55.385 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:55.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:55.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:55.406 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:55 smithi125 conmon[56739]: cluster 2021-03-15T22:25:54.129745+0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2021-03-15T22:25:55.406 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:55 smithi125 conmon[56739]: audit 2021-03-15T22:25:54.248573+0000 mon.a (mon.0) 1275 : audit [INF] from='client.? 172.21.15.76:0/1554990184' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:55.406 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:55 smithi125 conmon[56739]: cluster 2021-03-15T22:25:55.102248+0000 mon.a (mon.0) 1276 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:55.407 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:55 smithi125 conmon[56739]: audit 2021-03-15T22:25:55.106408+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? 172.21.15.76:0/1554990184' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:55.407 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:55 smithi125 conmon[56739]: cluster 2021-03-15T22:25:55.106489+0000 mon.a (mon.0) 1278 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2021-03-15T22:25:55.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:55.512 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:55.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:55.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:55.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[56514]: cluster 2021-03-15T22:25:54.129745+0000 mon.a ( 2021-03-15T22:25:55.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[56514]: mon.0) 1274 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2021-03-15T22:25:55.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[56514]: audit 2021-03-15T22:25:54.248573+0000 mon.a (mon.0) 1275 : audit [INF] from='client.? 172.21.15.76:0/1554990184' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[56514]: cluster 2021-03-15T22:25:55.102248+0000 mon.a (mon.0) 1276 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[56514]: 2021-03-15T22:25:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[56514]: audit 2021-03-15T22:25:55.106408+0000 mon.a (mon. 2021-03-15T22:25:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[56514]: 0) 1277 : audit [INF] from='client.? 172.21.15.76:0/1554990184' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:25:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[56514]: 22:25:55.106489+0000 mon.a (mon. 2021-03-15T22:25:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[56514]: 0) 1278 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2021-03-15T22:25:55.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[62156]: cluster 2021-03-15T22:25:54.129745 2021-03-15T22:25:55.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1274 : cluster [DBG] osdmap e439: 8 total, 8 up, 7 in 2021-03-15T22:25:55.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[62156]: audit 2021-03-15T22:25:54.248573+0000 mon.a (mon.0) 1275 : audit 2021-03-15T22:25:55.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/1554990184' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:25:55.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[62156]: cluster 2021-03-15T22:25:55.102248+0000 mon.a (mon.0) 2021-03-15T22:25:55.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[62156]: 1276 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:25:55.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:25:55.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[62156]: -15T22:25:55.106408+0000 mon.a (mon.0) 1277 : audit [INF] from='client.? 172.21.15.76:0/1554990184' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:25:55.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[62156]: cluster 2021-03-15T22:25:55.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:55 smithi076 conmon[62156]: 2021-03-15T22:25:55.106489+0000 mon.a (mon.0) 1278 : cluster [DBG] osdmap e440: 8 total, 8 up, 7 in 2021-03-15T22:25:55.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:55.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:55.640 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:55.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:55.658 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:25:55.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:25:55.741 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:55.765 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:55.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:55.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:55.868 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:55.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:55.892 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:55.892 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:55.962 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:55.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:56.017 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:56.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:56.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:56.142 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:56.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:56.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:56.147 INFO:tasks.rados.rados.0.smithi076.stdout:1623: done (3 left) 2021-03-15T22:25:56.147 INFO:tasks.rados.rados.0.smithi076.stdout:1625: done (2 left) 2021-03-15T22:25:56.148 INFO:tasks.rados.rados.0.smithi076.stdout:1626: done (1 left) 2021-03-15T22:25:56.148 INFO:tasks.rados.rados.0.smithi076.stdout:1627: done (0 left) 2021-03-15T22:25:56.148 INFO:tasks.rados.rados.0.smithi076.stdout:1628: write (excl) oid 6 current snap is 180 2021-03-15T22:25:56.148 INFO:tasks.rados.rados.0.smithi076.stdout:1628: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2021-03-15T22:25:56.154 INFO:tasks.rados.rados.0.smithi076.stdout:1628: writing smithi07694878-6 from 464025 to 1258675 tid 1 2021-03-15T22:25:56.161 INFO:tasks.rados.rados.0.smithi076.stdout:1628: writing smithi07694878-6 from 1947852 to 2352041 tid 2 2021-03-15T22:25:56.168 INFO:tasks.rados.rados.0.smithi076.stdout:1628: writing smithi07694878-6 from 2990809 to 3416334 tid 3 2021-03-15T22:25:56.169 INFO:tasks.rados.rados.0.smithi076.stdout:1629: write oid 31 current snap is 180 2021-03-15T22:25:56.169 INFO:tasks.rados.rados.0.smithi076.stdout:1628: finishing write tid 1 to smithi07694878-6 2021-03-15T22:25:56.169 INFO:tasks.rados.rados.0.smithi076.stdout:1629: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2021-03-15T22:25:56.172 INFO:tasks.rados.rados.0.smithi076.stdout:1629: writing smithi07694878-31 from 480832 to 950731 tid 1 2021-03-15T22:25:56.180 INFO:tasks.rados.rados.0.smithi076.stdout:1629: writing smithi07694878-31 from 1606334 to 2270534 tid 2 2021-03-15T22:25:56.189 INFO:tasks.rados.rados.0.smithi076.stdout:1629: writing smithi07694878-31 from 3018232 to 3680832 tid 3 2021-03-15T22:25:56.189 INFO:tasks.rados.rados.0.smithi076.stdout:1630: delete oid 42 current snap is 180 2021-03-15T22:25:56.189 INFO:tasks.rados.rados.0.smithi076.stdout:1628: finishing write tid 2 to smithi07694878-6 2021-03-15T22:25:56.190 INFO:tasks.rados.rados.0.smithi076.stdout:1628: finishing write tid 3 to smithi07694878-6 2021-03-15T22:25:56.190 INFO:tasks.rados.rados.0.smithi076.stdout:1628: finishing write tid 4 to smithi07694878-6 2021-03-15T22:25:56.190 INFO:tasks.rados.rados.0.smithi076.stdout:1628: finishing write tid 5 to smithi07694878-6 2021-03-15T22:25:56.191 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 771 (ObjNum 575 snap 180 seq_num 575) dirty exists 2021-03-15T22:25:56.191 INFO:tasks.rados.rados.0.smithi076.stdout:1628: left oid 6 (ObjNum 575 snap 180 seq_num 575) 2021-03-15T22:25:56.191 INFO:tasks.rados.rados.0.smithi076.stdout:1629: finishing write tid 1 to smithi07694878-31 2021-03-15T22:25:56.192 INFO:tasks.rados.rados.0.smithi076.stdout:1629: finishing write tid 2 to smithi07694878-31 2021-03-15T22:25:56.193 INFO:tasks.rados.rados.0.smithi076.stdout:1629: finishing write tid 3 to smithi07694878-31 2021-03-15T22:25:56.193 INFO:tasks.rados.rados.0.smithi076.stdout:1629: finishing write tid 4 to smithi07694878-31 2021-03-15T22:25:56.193 INFO:tasks.rados.rados.0.smithi076.stdout:1628: done (2 left) 2021-03-15T22:25:56.194 INFO:tasks.rados.rados.0.smithi076.stdout:1630: done (1 left) 2021-03-15T22:25:56.194 INFO:tasks.rados.rados.0.smithi076.stdout:1631: rollback oid 4 current snap is 180 2021-03-15T22:25:56.194 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 4 to 178 2021-03-15T22:25:56.194 INFO:tasks.rados.rados.0.smithi076.stdout:1632: write (excl) oid 23 current snap is 180 2021-03-15T22:25:56.195 INFO:tasks.rados.rados.0.smithi076.stdout:1632: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2021-03-15T22:25:56.199 INFO:tasks.rados.rados.0.smithi076.stdout:1632: writing smithi07694878-23 from 514446 to 1251196 tid 1 2021-03-15T22:25:56.205 INFO:tasks.rados.rados.0.smithi076.stdout:1632: writing smithi07694878-23 from 1723298 to 2391167 tid 2 2021-03-15T22:25:56.212 INFO:tasks.rados.rados.0.smithi076.stdout:1632: writing smithi07694878-23 from 3073078 to 3714446 tid 3 2021-03-15T22:25:56.213 INFO:tasks.rados.rados.0.smithi076.stdout:1633: write oid 8 current snap is 180 2021-03-15T22:25:56.213 INFO:tasks.rados.rados.0.smithi076.stdout:1633: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2021-03-15T22:25:56.217 INFO:tasks.rados.rados.0.smithi076.stdout:1633: writing smithi07694878-8 from 531253 to 943252 tid 1 2021-03-15T22:25:56.222 INFO:tasks.rados.rados.0.smithi076.stdout:1633: writing smithi07694878-8 from 1381780 to 1793307 tid 2 2021-03-15T22:25:56.228 INFO:tasks.rados.rados.0.smithi076.stdout:1633: writing smithi07694878-8 from 2300501 to 2887467 tid 3 2021-03-15T22:25:56.233 INFO:tasks.rados.rados.0.smithi076.stdout:1633: writing smithi07694878-8 from 3376974 to 3731253 tid 4 2021-03-15T22:25:56.233 INFO:tasks.rados.rados.0.smithi076.stdout:1634: snap_remove snap 140 2021-03-15T22:25:56.244 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:56.267 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:56.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:56.269 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:56.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:56.392 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:56.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:56.393 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:56.404 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:56 smithi076 conmon[56514]: cluster 2021-03-15T22:25:55.060556+0000 2021-03-15T22:25:56.405 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:56 smithi076 conmon[56514]: mgr.y (mgr.14144) 643 : cluster [DBG] pgmap v786: 138 pgs: 138 active+clean; 386 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-03-15T22:25:56.405 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:56 smithi076 conmon[62156]: cluster 2021-03-15T22:25:55.060556+0000 mgr.y 2021-03-15T22:25:56.405 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:56 smithi076 conmon[62156]: (mgr.14144) 643 : cluster [DBG] pgmap v786: 138 pgs: 138 active+clean; 386 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-03-15T22:25:56.494 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:56.517 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:56.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:56.518 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:56 smithi125 conmon[56739]: cluster 2021-03-15T22:25:55.060556+0000 2021-03-15T22:25:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:56 smithi125 conmon[56739]: mgr.y (mgr.14144) 643 : cluster [DBG] pgmap v786: 138 pgs: 138 active+clean; 386 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 14 MiB/s rd, 3.2 MiB/s wr, 20 op/s 2021-03-15T22:25:56.619 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:56.641 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:56.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:56.724 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:25:56.743 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:56.789 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:56.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:56.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:56.892 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:56.915 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:56.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:56.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:57.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:57.041 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:57.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:57.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:57.121 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:57.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:25:57.169 INFO:tasks.rados.rados.0.smithi076.stdout:1634: done (4 left) 2021-03-15T22:25:57.169 INFO:tasks.rados.rados.0.smithi076.stdout:1635: read oid 41 snap -1 2021-03-15T22:25:57.170 INFO:tasks.rados.rados.0.smithi076.stdout:1635: expect (ObjNum 509 snap 152 seq_num 509) 2021-03-15T22:25:57.170 INFO:tasks.rados.rados.0.smithi076.stdout:1636: delete oid 30 current snap is 180 2021-03-15T22:25:57.170 INFO:tasks.rados.rados.0.smithi076.stdout:1629: finishing write tid 5 to smithi07694878-31 2021-03-15T22:25:57.170 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 836 (ObjNum 576 snap 180 seq_num 576) dirty exists 2021-03-15T22:25:57.171 INFO:tasks.rados.rados.0.smithi076.stdout:1629: left oid 31 (ObjNum 576 snap 180 seq_num 576) 2021-03-15T22:25:57.171 INFO:tasks.rados.rados.0.smithi076.stdout:1631: finishing rollback tid 1 to smithi07694878-4 2021-03-15T22:25:57.171 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 0 (ObjNum 265078112 snap 21896 seq_num 0) dirty dne 2021-03-15T22:25:57.171 INFO:tasks.rados.rados.0.smithi076.stdout:1632: finishing write tid 1 to smithi07694878-23 2021-03-15T22:25:57.172 INFO:tasks.rados.rados.0.smithi076.stdout:1632: finishing write tid 2 to smithi07694878-23 2021-03-15T22:25:57.172 INFO:tasks.rados.rados.0.smithi076.stdout:1632: finishing write tid 3 to smithi07694878-23 2021-03-15T22:25:57.172 INFO:tasks.rados.rados.0.smithi076.stdout:1632: finishing write tid 4 to smithi07694878-23 2021-03-15T22:25:57.172 INFO:tasks.rados.rados.0.smithi076.stdout:1632: finishing write tid 5 to smithi07694878-23 2021-03-15T22:25:57.172 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 785 (ObjNum 578 snap 180 seq_num 578) dirty exists 2021-03-15T22:25:57.173 INFO:tasks.rados.rados.0.smithi076.stdout:1632: left oid 23 (ObjNum 578 snap 180 seq_num 578) 2021-03-15T22:25:57.173 INFO:tasks.rados.rados.0.smithi076.stdout:1633: finishing write tid 1 to smithi07694878-8 2021-03-15T22:25:57.173 INFO:tasks.rados.rados.0.smithi076.stdout:1633: finishing write tid 2 to smithi07694878-8 2021-03-15T22:25:57.173 INFO:tasks.rados.rados.0.smithi076.stdout:1633: finishing write tid 3 to smithi07694878-8 2021-03-15T22:25:57.173 INFO:tasks.rados.rados.0.smithi076.stdout:1633: finishing write tid 4 to smithi07694878-8 2021-03-15T22:25:57.174 INFO:tasks.rados.rados.0.smithi076.stdout:1633: finishing write tid 5 to smithi07694878-8 2021-03-15T22:25:57.174 INFO:tasks.rados.rados.0.smithi076.stdout:1633: finishing write tid 6 to smithi07694878-8 2021-03-15T22:25:57.175 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 774 (ObjNum 579 snap 180 seq_num 579) dirty exists 2021-03-15T22:25:57.175 INFO:tasks.rados.rados.0.smithi076.stdout:1633: left oid 8 (ObjNum 579 snap 180 seq_num 579) 2021-03-15T22:25:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:57.177 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:57.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:25:57.179 INFO:tasks.rados.rados.0.smithi076.stdout:1629: done (5 left) 2021-03-15T22:25:57.179 INFO:tasks.rados.rados.0.smithi076.stdout:1631: done (4 left) 2021-03-15T22:25:57.180 INFO:tasks.rados.rados.0.smithi076.stdout:1632: done (3 left) 2021-03-15T22:25:57.180 INFO:tasks.rados.rados.0.smithi076.stdout:1633: done (2 left) 2021-03-15T22:25:57.180 INFO:tasks.rados.rados.0.smithi076.stdout:1636: done (1 left) 2021-03-15T22:25:57.180 INFO:tasks.rados.rados.0.smithi076.stdout:1637: rollback oid 7 current snap is 180 2021-03-15T22:25:57.181 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 7 to 152 2021-03-15T22:25:57.181 INFO:tasks.rados.rados.0.smithi076.stdout:1638: write oid 31 current snap is 180 2021-03-15T22:25:57.181 INFO:tasks.rados.rados.0.smithi076.stdout:1638: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2021-03-15T22:25:57.184 INFO:tasks.rados.rados.0.smithi076.stdout:1638: writing smithi07694878-31 from 564867 to 1127364 tid 1 2021-03-15T22:25:57.192 INFO:tasks.rados.rados.0.smithi076.stdout:1638: writing smithi07694878-31 from 1898744 to 2313940 tid 2 2021-03-15T22:25:57.200 INFO:tasks.rados.rados.0.smithi076.stdout:1638: writing smithi07694878-31 from 2755347 to 3481210 tid 3 2021-03-15T22:25:57.200 INFO:tasks.rados.rados.0.smithi076.stdout:1639: read oid 28 snap -1 2021-03-15T22:25:57.201 INFO:tasks.rados.rados.0.smithi076.stdout:1639: expect (ObjNum 549 snap 175 seq_num 549) 2021-03-15T22:25:57.201 INFO:tasks.rados.rados.0.smithi076.stdout:1635: expect (ObjNum 509 snap 152 seq_num 509) 2021-03-15T22:25:57.224 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:25:57.225 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:25:57.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:57.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:57 smithi125 conmon[56739]: cluster 2021-03-15T22:25:56.144995+0000 mon.a (mon.0) 1279 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2021-03-15T22:25:57.302 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:57.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:57.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:57.347 INFO:tasks.rados.rados.0.smithi076.stdout:1637: finishing rollback tid 0 to smithi07694878-7 2021-03-15T22:25:57.348 INFO:tasks.rados.rados.0.smithi076.stdout:1637: finishing rollback tid 1 to smithi07694878-7 2021-03-15T22:25:57.348 INFO:tasks.rados.rados.0.smithi076.stdout:1637: finishing rollback tid 2 to smithi07694878-7 2021-03-15T22:25:57.348 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 975 (ObjNum 417 snap 112 seq_num 417) dirty exists 2021-03-15T22:25:57.349 INFO:tasks.rados.rados.0.smithi076.stdout:1638: finishing write tid 1 to smithi07694878-31 2021-03-15T22:25:57.349 INFO:tasks.rados.rados.0.smithi076.stdout:1638: finishing write tid 2 to smithi07694878-31 2021-03-15T22:25:57.349 INFO:tasks.rados.rados.0.smithi076.stdout:1635: done (3 left) 2021-03-15T22:25:57.349 INFO:tasks.rados.rados.0.smithi076.stdout:1637: done (2 left) 2021-03-15T22:25:57.349 INFO:tasks.rados.rados.0.smithi076.stdout:1640: read oid 47 snap -1 2021-03-15T22:25:57.350 INFO:tasks.rados.rados.0.smithi076.stdout:1640: expect (ObjNum 564 snap 177 seq_num 564) 2021-03-15T22:25:57.350 INFO:tasks.rados.rados.0.smithi076.stdout:1638: finishing write tid 3 to smithi07694878-31 2021-03-15T22:25:57.350 INFO:tasks.rados.rados.0.smithi076.stdout:1638: finishing write tid 4 to smithi07694878-31 2021-03-15T22:25:57.350 INFO:tasks.rados.rados.0.smithi076.stdout:1638: finishing write tid 5 to smithi07694878-31 2021-03-15T22:25:57.351 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 840 (ObjNum 581 snap 180 seq_num 581) dirty exists 2021-03-15T22:25:57.351 INFO:tasks.rados.rados.0.smithi076.stdout:1638: left oid 31 (ObjNum 581 snap 180 seq_num 581) 2021-03-15T22:25:57.351 INFO:tasks.rados.rados.0.smithi076.stdout:1641: read oid 37 snap 153 2021-03-15T22:25:57.351 INFO:tasks.rados.rados.0.smithi076.stdout:1641: expect (ObjNum 476 snap 142 seq_num 476) 2021-03-15T22:25:57.351 INFO:tasks.rados.rados.0.smithi076.stdout:1638: done (3 left) 2021-03-15T22:25:57.352 INFO:tasks.rados.rados.0.smithi076.stdout:1642: delete oid 24 current snap is 180 2021-03-15T22:25:57.353 INFO:tasks.rados.rados.0.smithi076.stdout:1642: done (3 left) 2021-03-15T22:25:57.353 INFO:tasks.rados.rados.0.smithi076.stdout:1643: delete oid 50 current snap is 180 2021-03-15T22:25:57.353 INFO:tasks.rados.rados.0.smithi076.stdout:1643: done (3 left) 2021-03-15T22:25:57.353 INFO:tasks.rados.rados.0.smithi076.stdout:1644: read oid 19 snap 174 2021-03-15T22:25:57.354 INFO:tasks.rados.rados.0.smithi076.stdout:1644: expect deleted 2021-03-15T22:25:57.354 INFO:tasks.rados.rados.0.smithi076.stdout:1645: copy_from oid 41 from oid 34 current snap is 180 2021-03-15T22:25:57.354 INFO:tasks.rados.rados.0.smithi076.stdout:1646: snap_create 2021-03-15T22:25:57.354 INFO:tasks.rados.rados.0.smithi076.stdout:1640: expect (ObjNum 564 snap 177 seq_num 564) 2021-03-15T22:25:57.355 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:57 smithi076 conmon[56514]: cluster 2021-03-15T22:25:56.144995+0000 2021-03-15T22:25:57.355 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:57 smithi076 conmon[56514]: mon.a (mon.0) 1279 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2021-03-15T22:25:57.356 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:57 smithi076 conmon[62156]: cluster 2021-03-15T22:25:56.144995+0000 mon.a (mon 2021-03-15T22:25:57.356 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:57 smithi076 conmon[62156]: .0) 1279 : cluster [DBG] osdmap e441: 8 total, 8 up, 7 in 2021-03-15T22:25:57.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:57.430 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:57.432 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:57.533 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:57.556 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:57.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:57.647 INFO:tasks.rados.rados.0.smithi076.stdout:1639: expect (ObjNum 549 snap 175 seq_num 549) 2021-03-15T22:25:57.659 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:57.681 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:57.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:57.753 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:25:57.771 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:25:57.784 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:57.788 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:25:57.789 INFO:tasks.rados.rados.0.smithi076.stdout:1641: expect (ObjNum 476 snap 142 seq_num 476) 2021-03-15T22:25:57.804 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:25:57.806 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:25:57.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:57.808 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:57.808 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:57.827 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:25:57.846 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:25:57.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-15T22:25:57.857 INFO:tasks.rados.rados.0.smithi076.stdout:1645: finishing copy_from to smithi07694878-41 2021-03-15T22:25:57.857 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 695 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:25:57.858 INFO:tasks.rados.rados.0.smithi076.stdout:1645: finishing copy_from racing read to smithi07694878-41 2021-03-15T22:25:57.869 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:25:57.887 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:25:57.888 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:25:57.889 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:25:57.889 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:25:57.889 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:25:57.889 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:25:57.889 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:25:57.890 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:25:57.890 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:25:57.910 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:57.932 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:57.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:57.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:58.034 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:25:58.058 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:58.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:58.059 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:25:58.133 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:25:58.160 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:25:58.165 INFO:tasks.rados.rados.0.smithi076.stdout:1639: done (5 left) 2021-03-15T22:25:58.165 INFO:tasks.rados.rados.0.smithi076.stdout:1640: done (4 left) 2021-03-15T22:25:58.166 INFO:tasks.rados.rados.0.smithi076.stdout:1641: done (3 left) 2021-03-15T22:25:58.166 INFO:tasks.rados.rados.0.smithi076.stdout:1644: done (2 left) 2021-03-15T22:25:58.166 INFO:tasks.rados.rados.0.smithi076.stdout:1645: done (1 left) 2021-03-15T22:25:58.167 INFO:tasks.rados.rados.0.smithi076.stdout:1646: done (0 left) 2021-03-15T22:25:58.167 INFO:tasks.rados.rados.0.smithi076.stdout:1647: delete oid 24 current snap is 181 2021-03-15T22:25:58.183 INFO:tasks.rados.rados.0.smithi076.stdout:1647: done (0 left) 2021-03-15T22:25:58.183 INFO:tasks.rados.rados.0.smithi076.stdout:1648: snap_remove snap 152 2021-03-15T22:25:58.185 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:58.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:58.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:25:58.287 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:58.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:58.311 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:58.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:58.413 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:58.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:58.436 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:58.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:58.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:58 smithi076 conmon[56514]: cluster 2021-03-15T22:25:57.061303+0000 mgr.y (mgr.14144) 644 : cluster [DBG] pgmap v789: 138 pgs: 138 active+clean; 395 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-03-15T22:25:58.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:58 smithi076 conmon[56514]: cluster 2021-03-15T22:25:57.157507+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2021-03-15T22:25:58.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:58 smithi076 conmon[56514]: audit 2021-03-15T22:25:57.724629+0000 mon.a (mon.0) 1281 : audit [DBG] from='client.? 172.21.15.76:0/3705984195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:58.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:58 smithi076 conmon[62156]: cluster 2021-03-15T22:25:57.061303+0000 mgr.y ( 2021-03-15T22:25:58.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:58 smithi076 conmon[62156]: mgr.14144) 644 : cluster [DBG] pgmap v789: 138 pgs: 138 active+clean; 395 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-03-15T22:25:58.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:58 smithi076 conmon[62156]: cluster 2021-03-15T22:25:57.157507 2021-03-15T22:25:58.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:58 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2021-03-15T22:25:58.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:58 smithi076 conmon[62156]: audit 2021-03-15T22:25:58.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:58 smithi076 conmon[62156]: 2021-03-15T22:25:57.724629 2021-03-15T22:25:58.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:58 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1281 : 2021-03-15T22:25:58.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:58 smithi076 conmon[62156]: audit [DBG] from='client.? 172.21.15.76:0/3705984195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:58 smithi125 conmon[56739]: cluster 2021-03-15T22:25:57.061303+0000 mgr.y (mgr.14144) 644 2021-03-15T22:25:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:58 smithi125 conmon[56739]: : cluster [DBG] pgmap v789: 138 pgs: 138 active+clean; 395 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.5 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-03-15T22:25:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:58 smithi125 conmon[56739]: cluster 2021-03-15T22:25:57.157507+0000 mon.a (mon.0) 1280 : cluster [DBG] osdmap e442: 8 total, 8 up, 7 in 2021-03-15T22:25:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:58 smithi125 conmon[56739]: audit 2021-03-15T22:25:57.724629+0000 mon.a (mon.0) 1281 : audit [DBG] from='client.? 172.21.15.76:0/3705984195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:25:58.538 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:58.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:58.561 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:58.561 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:58.663 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:58.686 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:58.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:58.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:58.788 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:25:58.810 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:58.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:58.812 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:25:58.913 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:58.934 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:58.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:58.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:59.037 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:59.061 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:59.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:59.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:59.091 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:25:59.163 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:25:59.168 INFO:tasks.rados.rados.0.smithi076.stdout:1648: done (0 left) 2021-03-15T22:25:59.168 INFO:tasks.rados.rados.0.smithi076.stdout:1649: read oid 44 snap -1 2021-03-15T22:25:59.169 INFO:tasks.rados.rados.0.smithi076.stdout:1649: expect deleted 2021-03-15T22:25:59.169 INFO:tasks.rados.rados.0.smithi076.stdout:1650: read oid 8 snap 155 2021-03-15T22:25:59.169 INFO:tasks.rados.rados.0.smithi076.stdout:1650: expect deleted 2021-03-15T22:25:59.169 INFO:tasks.rados.rados.0.smithi076.stdout:1651: rollback oid 30 current snap is 181 2021-03-15T22:25:59.170 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 30 to 154 2021-03-15T22:25:59.170 INFO:tasks.rados.rados.0.smithi076.stdout:1652: snap_remove snap 177 2021-03-15T22:25:59.193 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:59.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:59.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:25:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:59 smithi076 conmon[56514]: cluster 2021-03-15T22:25:58.162307+0000 mon.a 2021-03-15T22:25:59.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:25:59 smithi076 conmon[56514]: (mon.0) 1282 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2021-03-15T22:25:59.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:59 smithi076 conmon[62156]: cluster 2021-03-15T22:25:58. 2021-03-15T22:25:59.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:25:59 smithi076 conmon[62156]: 162307+0000 mon.a (mon.0) 1282 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2021-03-15T22:25:59.296 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:25:59.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:59.321 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:59.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:25:59.423 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:25:59.426 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:59 smithi125 conmon[56739]: cluster 2021-03-15T22:25:58.162307+0000 2021-03-15T22:25:59.427 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:25:59 smithi125 conmon[56739]: mon.a (mon.0) 1282 : cluster [DBG] osdmap e443: 8 total, 8 up, 7 in 2021-03-15T22:25:59.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:59.449 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:25:59.450 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:25:59.551 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:25:59.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:25:59.576 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:25:59.577 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:25:59.621 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:25:59.621 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:25:59.622 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:25:59.622 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:25:59.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.623 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:350 7.7 7:ecc0cb20:::smithi07694878-25:head [copy-from ver 34] snapc a=[a,9,8] ondisk+write+known_if_redirected e106)", 2021-03-15T22:25:59.623 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:21.030292+0000", 2021-03-15T22:25:59.624 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 338.58458180500003, 2021-03-15T22:25:59.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018576321999999999, 2021-03-15T22:25:59.625 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.625 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.626 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.626 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.626 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.627 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 350 2021-03-15T22:25:59.627 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.628 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.628 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.629 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.629 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030292+0000", 2021-03-15T22:25:59.630 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.631 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.631 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030292+0000", 2021-03-15T22:25:59.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9689999999999999e-06 2021-03-15T22:25:59.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.632 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030298+0000", 2021-03-15T22:25:59.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.481e-06 2021-03-15T22:25:59.633 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.634 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030303+0000", 2021-03-15T22:25:59.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2648999999999999e-05 2021-03-15T22:25:59.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030325+0000", 2021-03-15T22:25:59.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1140000000000006e-06 2021-03-15T22:25:59.635 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.636 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030334+0000", 2021-03-15T22:25:59.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7649000000000003e-05 2021-03-15T22:25:59.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.637 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.637 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030391+0000", 2021-03-15T22:25:59.637 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7892999999999994e-05 2021-03-15T22:25:59.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.638 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.638 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030469+0000", 2021-03-15T22:25:59.638 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052504530000000004 2021-03-15T22:25:59.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.639 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.639 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-15T22:25:59.639 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.035720+0000", 2021-03-15T22:25:59.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054168109999999997 2021-03-15T22:25:59.640 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.640 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.041136+0000", 2021-03-15T22:25:59.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054259169999999997 2021-03-15T22:25:59.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.641 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.641 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.046562+0000", 2021-03-15T22:25:59.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002223937 2021-03-15T22:25:59.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.642 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.642 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.642 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.048786+0000", 2021-03-15T22:25:59.643 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1217999999999997e-05 2021-03-15T22:25:59.643 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.643 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.643 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.643 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.048827+0000", 2021-03-15T22:25:59.644 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1230999999999999e-05 2021-03-15T22:25:59.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.644 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.644 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.048869+0000", 2021-03-15T22:25:59.645 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018576321999999999 2021-03-15T22:25:59.645 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.645 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.645 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.646 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.646 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:351 7.7 7:ecc0cb20:::smithi07694878-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e106)", 2021-03-15T22:25:59.646 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:20:21.030346+0000", 2021-03-15T22:25:59.646 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 338.58452853300003, 2021-03-15T22:25:59.647 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018695765999999999, 2021-03-15T22:25:59.647 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.647 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:25:59.647 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.648 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.648 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.648 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 351 2021-03-15T22:25:59.648 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.648 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.649 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.649 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.649 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030346+0000", 2021-03-15T22:25:59.649 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.649 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.650 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.650 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030346+0000", 2021-03-15T22:25:59.650 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.062e-06 2021-03-15T22:25:59.651 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.651 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.651 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.651 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030347+0000", 2021-03-15T22:25:59.651 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.113e-06 2021-03-15T22:25:59.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.652 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030348+0000", 2021-03-15T22:25:59.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7149999999999997e-06 2021-03-15T22:25:59.653 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.653 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.653 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.653 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030353+0000", 2021-03-15T22:25:59.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0359999999999998e-06 2021-03-15T22:25:59.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.654 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.654 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030356+0000", 2021-03-15T22:25:59.655 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00026027 2021-03-15T22:25:59.655 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.655 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.656 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030616+0000", 2021-03-15T22:25:59.656 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9258000000000001e-05 2021-03-15T22:25:59.656 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.656 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:25:59.657 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.030675+0000", 2021-03-15T22:25:59.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0098426569999999994 2021-03-15T22:25:59.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.657 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.040518+0000", 2021-03-15T22:25:59.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6144000000000006e-05 2021-03-15T22:25:59.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.658 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:59.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.040614+0000", 2021-03-15T22:25:59.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0082962580000000008 2021-03-15T22:25:59.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.659 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.660 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.660 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.048910+0000", 2021-03-15T22:25:59.660 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3939e-05 2021-03-15T22:25:59.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.661 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.661 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.048964+0000", 2021-03-15T22:25:59.661 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7313999999999999e-05 2021-03-15T22:25:59.661 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.662 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.662 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.662 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:20:21.049041+0000", 2021-03-15T22:25:59.662 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018695765999999999 2021-03-15T22:25:59.662 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.663 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.663 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.663 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.663 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1215 7.7 7:ed4a01d2:::smithi07694878-2:head [copy-from ver 101] snapc 50=[3f] ondisk+write+known_if_redirected e179)", 2021-03-15T22:25:59.664 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:34.031426+0000", 2021-03-15T22:25:59.664 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 265.58344847900003, 2021-03-15T22:25:59.664 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028066849000000001, 2021-03-15T22:25:59.664 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.665 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.665 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.665 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.665 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.665 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1215 2021-03-15T22:25:59.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.666 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.666 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.666 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031426+0000", 2021-03-15T22:25:59.667 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.667 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.667 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.667 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.668 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031426+0000", 2021-03-15T22:25:59.668 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5599999999999998e-06 2021-03-15T22:25:59.668 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.668 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.668 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.669 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031429+0000", 2021-03-15T22:25:59.669 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0299999999999998e-06 2021-03-15T22:25:59.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.669 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.670 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031432+0000", 2021-03-15T22:25:59.670 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4944e-05 2021-03-15T22:25:59.670 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.670 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.671 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.671 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031447+0000", 2021-03-15T22:25:59.671 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9879999999999998e-06 2021-03-15T22:25:59.671 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.671 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.672 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.672 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031454+0000", 2021-03-15T22:25:59.672 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7342000000000005e-05 2021-03-15T22:25:59.672 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.673 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031542+0000", 2021-03-15T22:25:59.673 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1929999999999999e-05 2021-03-15T22:25:59.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.674 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.674 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:25:59.674 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031553+0000", 2021-03-15T22:25:59.674 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0019329010000000001 2021-03-15T22:25:59.674 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.675 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.675 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.675 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.033486+0000", 2021-03-15T22:25:59.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100469 2021-03-15T22:25:59.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.676 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.676 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.676 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.033587+0000", 2021-03-15T22:25:59.676 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012160482 2021-03-15T22:25:59.676 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.677 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.677 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,1", 2021-03-15T22:25:59.677 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.045747+0000", 2021-03-15T22:25:59.677 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005094922 2021-03-15T22:25:59.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.678 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.678 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.050842+0000", 2021-03-15T22:25:59.678 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0083807459999999997 2021-03-15T22:25:59.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.679 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.679 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059223+0000", 2021-03-15T22:25:59.679 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013855599999999999 2021-03-15T22:25:59.680 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.680 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.680 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.680 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059362+0000", 2021-03-15T22:25:59.681 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4486999999999994e-05 2021-03-15T22:25:59.681 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.681 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.681 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059426+0000", 2021-03-15T22:25:59.682 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6492e-05 2021-03-15T22:25:59.682 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.682 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.682 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.682 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059493+0000", 2021-03-15T22:25:59.683 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028066849000000001 2021-03-15T22:25:59.683 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.683 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.683 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.684 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.684 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1216 7.7 7:ed4a01d2:::smithi07694878-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e179)", 2021-03-15T22:25:59.684 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:34.031465+0000", 2021-03-15T22:25:59.684 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 265.58340962, 2021-03-15T22:25:59.685 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028173786999999999, 2021-03-15T22:25:59.685 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.685 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:25:59.685 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.686 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.686 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.686 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1216 2021-03-15T22:25:59.686 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.686 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.687 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.687 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.687 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031465+0000", 2021-03-15T22:25:59.687 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.688 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.688 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.688 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031465+0000", 2021-03-15T22:25:59.688 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0999999999999997e-07 2021-03-15T22:25:59.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.689 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.689 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.689 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031465+0000", 2021-03-15T22:25:59.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1480000000000001e-06 2021-03-15T22:25:59.690 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.690 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031467+0000", 2021-03-15T22:25:59.691 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2569e-05 2021-03-15T22:25:59.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.691 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031489+0000", 2021-03-15T22:25:59.692 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9050000000000001e-06 2021-03-15T22:25:59.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.692 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.692 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.693 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031492+0000", 2021-03-15T22:25:59.693 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000117608 2021-03-15T22:25:59.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.693 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.694 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031610+0000", 2021-03-15T22:25:59.694 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1499999999999998e-06 2021-03-15T22:25:59.694 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.694 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:25:59.695 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031615+0000", 2021-03-15T22:25:59.695 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002194278 2021-03-15T22:25:59.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.695 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.696 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.696 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.033809+0000", 2021-03-15T22:25:59.696 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3332999999999998e-05 2021-03-15T22:25:59.696 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.696 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.697 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:25:59.697 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.033872+0000", 2021-03-15T22:25:59.697 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016247014000000001 2021-03-15T22:25:59.697 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.698 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.698 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.050119+0000", 2021-03-15T22:25:59.698 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011831400000000001 2021-03-15T22:25:59.698 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.699 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.699 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:59.699 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.050238+0000", 2021-03-15T22:25:59.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0092772289999999997 2021-03-15T22:25:59.699 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059515+0000", 2021-03-15T22:25:59.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2630000000000003e-05 2021-03-15T22:25:59.701 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.701 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.701 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.701 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059568+0000", 2021-03-15T22:25:59.701 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0798999999999998e-05 2021-03-15T22:25:59.702 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.702 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.702 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.702 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059638+0000", 2021-03-15T22:25:59.702 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028173786999999999 2021-03-15T22:25:59.703 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.703 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.703 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.704 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.704 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1425 7.7 7:ecc0cb20:::smithi07694878-25:head [copy-from ver 3] snapc 61=[] ondisk+write+known_if_redirected e196)", 2021-03-15T22:25:59.704 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:51.033384+0000", 2021-03-15T22:25:59.704 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 248.58149028299999, 2021-03-15T22:25:59.704 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021309669, 2021-03-15T22:25:59.705 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.705 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.705 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.705 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.705 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.706 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1425 2021-03-15T22:25:59.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.706 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.706 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.707 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.707 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033384+0000", 2021-03-15T22:25:59.707 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.707 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.708 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.708 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033384+0000", 2021-03-15T22:25:59.708 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4e-06 2021-03-15T22:25:59.708 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.708 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.709 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.709 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033389+0000", 2021-03-15T22:25:59.709 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.121e-06 2021-03-15T22:25:59.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.710 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.710 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.710 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033393+0000", 2021-03-15T22:25:59.710 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5709000000000002e-05 2021-03-15T22:25:59.710 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.711 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.711 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.711 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033409+0000", 2021-03-15T22:25:59.711 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0939999999999997e-06 2021-03-15T22:25:59.711 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.712 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.712 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.712 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033418+0000", 2021-03-15T22:25:59.712 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010001999999999999 2021-03-15T22:25:59.713 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.713 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.713 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.713 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033518+0000", 2021-03-15T22:25:59.713 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0380000000000005e-06 2021-03-15T22:25:59.714 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.714 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.714 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:25:59.714 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033527+0000", 2021-03-15T22:25:59.714 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022977140000000002 2021-03-15T22:25:59.715 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.715 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.715 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.715 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.035825+0000", 2021-03-15T22:25:59.716 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5749999999999998e-05 2021-03-15T22:25:59.716 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.716 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.716 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.716 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.035901+0000", 2021-03-15T22:25:59.717 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076098540000000001 2021-03-15T22:25:59.717 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.717 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.717 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,1", 2021-03-15T22:25:59.717 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.043511+0000", 2021-03-15T22:25:59.718 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040407949999999998 2021-03-15T22:25:59.718 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.718 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.718 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.718 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.047551+0000", 2021-03-15T22:25:59.719 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0026008889999999999 2021-03-15T22:25:59.719 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.719 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.719 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.720 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.050152+0000", 2021-03-15T22:25:59.720 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044466410000000003 2021-03-15T22:25:59.720 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.720 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.720 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.721 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.054599+0000", 2021-03-15T22:25:59.721 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1737999999999999e-05 2021-03-15T22:25:59.721 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.721 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.721 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.722 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.054641+0000", 2021-03-15T22:25:59.722 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2905999999999998e-05 2021-03-15T22:25:59.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.722 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.723 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.723 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.054694+0000", 2021-03-15T22:25:59.723 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021309669 2021-03-15T22:25:59.723 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.723 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.724 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.724 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.724 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1426 7.7 7:ecc0cb20:::smithi07694878-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e196)", 2021-03-15T22:25:59.724 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:51.033436+0000", 2021-03-15T22:25:59.725 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 248.58143865900001, 2021-03-15T22:25:59.725 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021470156000000001, 2021-03-15T22:25:59.725 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.725 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:25:59.726 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.726 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.726 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.726 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1426 2021-03-15T22:25:59.726 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.727 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.727 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.727 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.727 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033436+0000", 2021-03-15T22:25:59.727 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.728 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.728 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.728 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.728 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033436+0000", 2021-03-15T22:25:59.728 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6360000000000001e-06 2021-03-15T22:25:59.729 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.729 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.729 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.729 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033437+0000", 2021-03-15T22:25:59.730 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.086e-06 2021-03-15T22:25:59.730 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.730 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.730 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.730 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033439+0000", 2021-03-15T22:25:59.731 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5630000000000003e-06 2021-03-15T22:25:59.731 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.731 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.731 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.731 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033447+0000", 2021-03-15T22:25:59.732 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1909999999999997e-06 2021-03-15T22:25:59.732 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.732 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.732 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.733 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033453+0000", 2021-03-15T22:25:59.733 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013471199999999999 2021-03-15T22:25:59.733 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.733 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.734 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.734 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033588+0000", 2021-03-15T22:25:59.734 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2900000000000002e-06 2021-03-15T22:25:59.734 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.734 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.735 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:25:59.735 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033593+0000", 2021-03-15T22:25:59.735 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024725210000000001 2021-03-15T22:25:59.735 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.736 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.736 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.736 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.036066+0000", 2021-03-15T22:25:59.736 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5259999999999998e-05 2021-03-15T22:25:59.736 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.737 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.737 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:25:59.737 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.036101+0000", 2021-03-15T22:25:59.737 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010876768 2021-03-15T22:25:59.737 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.738 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.738 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.738 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.046978+0000", 2021-03-15T22:25:59.738 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010354 2021-03-15T22:25:59.739 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.739 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.739 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:59.739 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.047081+0000", 2021-03-15T22:25:59.739 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076443860000000004 2021-03-15T22:25:59.740 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.740 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.740 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.740 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.054725+0000", 2021-03-15T22:25:59.740 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3589000000000004e-05 2021-03-15T22:25:59.741 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.741 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.741 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.741 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.054789+0000", 2021-03-15T22:25:59.741 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011661400000000001 2021-03-15T22:25:59.742 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.742 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.742 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.742 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.054906+0000", 2021-03-15T22:25:59.743 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021470156000000001 2021-03-15T22:25:59.743 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.743 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.743 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.743 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.744 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.744 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:2783 7.c 7:37eb566e:::smithi07694878-48:head [copy-from ver 263] snapc d2=[d1,cf] ondisk+write+known_if_redirected e311)", 2021-03-15T22:25:59.744 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:46.216071+0000", 2021-03-15T22:25:59.744 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 133.39880294400001, 2021-03-15T22:25:59.744 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018362507, 2021-03-15T22:25:59.745 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.745 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.745 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.745 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.746 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.746 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2783 2021-03-15T22:25:59.746 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.746 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.746 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.747 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.747 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216071+0000", 2021-03-15T22:25:59.747 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.747 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.747 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.748 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.748 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216071+0000", 2021-03-15T22:25:59.748 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.267e-06 2021-03-15T22:25:59.748 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.749 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.749 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.749 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216079+0000", 2021-03-15T22:25:59.749 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8949999999999997e-06 2021-03-15T22:25:59.749 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.750 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.750 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.750 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216083+0000", 2021-03-15T22:25:59.750 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7069000000000001e-05 2021-03-15T22:25:59.750 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.751 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.751 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.751 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216110+0000", 2021-03-15T22:25:59.751 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7170000000000003e-06 2021-03-15T22:25:59.752 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.752 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.752 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.752 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216120+0000", 2021-03-15T22:25:59.752 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6308999999999999e-05 2021-03-15T22:25:59.753 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.753 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.753 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.753 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216176+0000", 2021-03-15T22:25:59.753 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000125017 2021-03-15T22:25:59.754 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.754 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.754 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.754 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216302+0000", 2021-03-15T22:25:59.755 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005856193 2021-03-15T22:25:59.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.755 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:25:59.755 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.222158+0000", 2021-03-15T22:25:59.756 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011135378 2021-03-15T22:25:59.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.756 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.756 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.233293+0000", 2021-03-15T22:25:59.757 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001008875 2021-03-15T22:25:59.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.757 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.757 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.234302+0000", 2021-03-15T22:25:59.758 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3900999999999999e-05 2021-03-15T22:25:59.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.758 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.759 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.234336+0000", 2021-03-15T22:25:59.759 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5503000000000001e-05 2021-03-15T22:25:59.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.759 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.760 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.234381+0000", 2021-03-15T22:25:59.760 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2383000000000003e-05 2021-03-15T22:25:59.760 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.760 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.760 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.761 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.234434+0000", 2021-03-15T22:25:59.761 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018362507 2021-03-15T22:25:59.761 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.761 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.762 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.762 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.762 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.762 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:2784 7.c 7:37eb566e:::smithi07694878-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e311)", 2021-03-15T22:25:59.762 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:46.216139+0000", 2021-03-15T22:25:59.763 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 133.39873468900001, 2021-03-15T22:25:59.763 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018443800999999999, 2021-03-15T22:25:59.763 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.763 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:25:59.763 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.764 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.764 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.764 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2784 2021-03-15T22:25:59.764 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.765 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.765 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.765 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.765 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216139+0000", 2021-03-15T22:25:59.765 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.766 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.766 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.766 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.766 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216139+0000", 2021-03-15T22:25:59.766 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9319999999999999e-06 2021-03-15T22:25:59.767 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.767 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.767 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.767 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216141+0000", 2021-03-15T22:25:59.767 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4499999999999998e-06 2021-03-15T22:25:59.768 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.768 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.768 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.768 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216144+0000", 2021-03-15T22:25:59.769 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2740000000000007e-06 2021-03-15T22:25:59.769 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.769 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.769 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.769 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216152+0000", 2021-03-15T22:25:59.770 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.356e-06 2021-03-15T22:25:59.770 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.770 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.770 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.770 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216158+0000", 2021-03-15T22:25:59.771 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000384034 2021-03-15T22:25:59.771 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.771 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.771 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.771 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216542+0000", 2021-03-15T22:25:59.772 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9052999999999999e-05 2021-03-15T22:25:59.772 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.772 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.772 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:25:59.773 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.216601+0000", 2021-03-15T22:25:59.773 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011626421 2021-03-15T22:25:59.773 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.773 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.773 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.774 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.228227+0000", 2021-03-15T22:25:59.774 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9771000000000007e-05 2021-03-15T22:25:59.774 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.774 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.774 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:59.775 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.228297+0000", 2021-03-15T22:25:59.775 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0061653589999999996 2021-03-15T22:25:59.775 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.775 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.776 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.776 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.234462+0000", 2021-03-15T22:25:59.776 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1054000000000003e-05 2021-03-15T22:25:59.776 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.776 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.777 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.777 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.234513+0000", 2021-03-15T22:25:59.777 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0097000000000003e-05 2021-03-15T22:25:59.777 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.777 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.778 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.778 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:46.234583+0000", 2021-03-15T22:25:59.778 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018443800999999999 2021-03-15T22:25:59.778 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.779 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.779 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.779 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.779 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.779 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:3010 7.b e330/329 7:d980be1f:::smithi07694878-23:head v 330'208, mlcod=0'0)", 2021-03-15T22:25:59.780 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:05.105001+0000", 2021-03-15T22:25:59.780 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 114.509873274, 2021-03-15T22:25:59.780 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037948101999999997, 2021-03-15T22:25:59.780 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.780 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.781 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.781 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.781 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.781 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105001+0000", 2021-03-15T22:25:59.781 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.782 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.782 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105001+0000", 2021-03-15T22:25:59.783 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076767199999999995 2021-03-15T22:25:59.783 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.783 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.783 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.783 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105769+0000", 2021-03-15T22:25:59.784 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1410000000000001e-06 2021-03-15T22:25:59.784 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.784 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.784 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.784 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105772+0000", 2021-03-15T22:25:59.785 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2334000000000001e-05 2021-03-15T22:25:59.785 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.785 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.785 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.786 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105784+0000", 2021-03-15T22:25:59.786 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4559999999999999e-06 2021-03-15T22:25:59.786 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.786 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.786 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.787 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105790+0000", 2021-03-15T22:25:59.787 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.66e-05 2021-03-15T22:25:59.787 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.787 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.788 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.788 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105846+0000", 2021-03-15T22:25:59.788 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0828000000000001e-05 2021-03-15T22:25:59.788 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.788 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.789 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.789 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105867+0000", 2021-03-15T22:25:59.789 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036960346999999998 2021-03-15T22:25:59.789 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.789 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.790 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.790 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.142827+0000", 2021-03-15T22:25:59.790 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121724 2021-03-15T22:25:59.790 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.791 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.791 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.791 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.142949+0000", 2021-03-15T22:25:59.791 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037948101999999997 2021-03-15T22:25:59.791 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.792 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.792 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.792 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.792 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.792 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3172 7.7 7:ed4a01d2:::smithi07694878-2:head [copy-from ver 104] snapc f4=[f4,f3,f2,e9,e7,e1,de] ondisk+write+known_if_redirected e344)", 2021-03-15T22:25:59.793 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:19.131288+0000", 2021-03-15T22:25:59.793 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 100.483585936, 2021-03-15T22:25:59.793 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025760814999999999, 2021-03-15T22:25:59.793 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.794 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.794 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.794 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.794 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.794 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3172 2021-03-15T22:25:59.795 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.795 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.795 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.795 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.795 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131288+0000", 2021-03-15T22:25:59.796 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.796 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.796 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.796 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.797 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131288+0000", 2021-03-15T22:25:59.797 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0330000000000001e-06 2021-03-15T22:25:59.797 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.797 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.797 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.798 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131289+0000", 2021-03-15T22:25:59.798 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3699999999999999e-07 2021-03-15T22:25:59.798 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.798 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.798 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.799 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131290+0000", 2021-03-15T22:25:59.799 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2509999999999999e-06 2021-03-15T22:25:59.799 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.799 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.799 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.800 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131293+0000", 2021-03-15T22:25:59.800 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4480000000000001e-06 2021-03-15T22:25:59.800 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.800 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.801 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131296+0000", 2021-03-15T22:25:59.801 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012789800000000001 2021-03-15T22:25:59.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.801 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.802 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.802 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.132575+0000", 2021-03-15T22:25:59.802 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4247999999999999e-05 2021-03-15T22:25:59.802 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.802 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.803 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.803 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.132639+0000", 2021-03-15T22:25:59.803 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002374929 2021-03-15T22:25:59.803 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.804 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.804 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,1", 2021-03-15T22:25:59.804 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.135014+0000", 2021-03-15T22:25:59.804 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050646509999999999 2021-03-15T22:25:59.804 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.805 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.805 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.805 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.140079+0000", 2021-03-15T22:25:59.805 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016804399000000001 2021-03-15T22:25:59.805 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.806 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.806 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.806 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.156883+0000", 2021-03-15T22:25:59.806 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9956999999999997e-05 2021-03-15T22:25:59.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.807 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.807 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.807 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.156923+0000", 2021-03-15T22:25:59.807 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7237999999999997e-05 2021-03-15T22:25:59.808 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.808 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.808 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.808 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.156970+0000", 2021-03-15T22:25:59.808 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8844000000000006e-05 2021-03-15T22:25:59.809 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.809 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.809 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.809 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.157049+0000", 2021-03-15T22:25:59.809 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025760814999999999 2021-03-15T22:25:59.810 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.810 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.810 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.810 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.811 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3173 7.7 7:ed4a01d2:::smithi07694878-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e344)", 2021-03-15T22:25:59.811 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:19.131298+0000", 2021-03-15T22:25:59.811 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 100.48357621300001, 2021-03-15T22:25:59.811 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025917880000000001, 2021-03-15T22:25:59.812 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.812 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:25:59.812 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.812 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.812 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.813 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3173 2021-03-15T22:25:59.813 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.813 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.813 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.813 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.814 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131298+0000", 2021-03-15T22:25:59.814 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.815 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.815 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131298+0000", 2021-03-15T22:25:59.815 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2499999999999995e-07 2021-03-15T22:25:59.815 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.816 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.816 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131298+0000", 2021-03-15T22:25:59.816 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4899999999999995e-07 2021-03-15T22:25:59.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.817 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.817 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131299+0000", 2021-03-15T22:25:59.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9730000000000002e-06 2021-03-15T22:25:59.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.817 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131302+0000", 2021-03-15T22:25:59.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.21e-06 2021-03-15T22:25:59.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.819 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131304+0000", 2021-03-15T22:25:59.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001449365 2021-03-15T22:25:59.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.820 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.132754+0000", 2021-03-15T22:25:59.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0812000000000002e-05 2021-03-15T22:25:59.820 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.821 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:25:59.821 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.132785+0000", 2021-03-15T22:25:59.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0063974619999999996 2021-03-15T22:25:59.822 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.822 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.822 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.822 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.139182+0000", 2021-03-15T22:25:59.822 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5412000000000002e-05 2021-03-15T22:25:59.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.823 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:59.823 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.139257+0000", 2021-03-15T22:25:59.824 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017824311999999998 2021-03-15T22:25:59.824 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.824 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.824 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.824 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.157082+0000", 2021-03-15T22:25:59.825 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6941999999999997e-05 2021-03-15T22:25:59.825 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.825 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.825 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.157149+0000", 2021-03-15T22:25:59.826 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7218000000000006e-05 2021-03-15T22:25:59.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.826 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.826 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.827 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.157216+0000", 2021-03-15T22:25:59.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025917880000000001 2021-03-15T22:25:59.827 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.827 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.827 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.828 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.828 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.828 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3950 7.7 7:ed4a01d2:::smithi07694878-2:head [write 658973~703921,stat] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:25:59.828 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:25:59.828 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 24.487633025000001, 2021-03-15T22:25:59.829 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99727343300000004, 2021-03-15T22:25:59.829 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.829 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.829 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.829 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.830 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.830 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3950 2021-03-15T22:25:59.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.830 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.831 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.831 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.831 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:25:59.831 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.831 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.832 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:25:59.832 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076287800000000004 2021-03-15T22:25:59.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.833 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128004+0000", 2021-03-15T22:25:59.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8489999999999999e-06 2021-03-15T22:25:59.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.834 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.834 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.834 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128008+0000", 2021-03-15T22:25:59.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3244e-05 2021-03-15T22:25:59.835 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.835 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.835 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.835 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128031+0000", 2021-03-15T22:25:59.836 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2050000000000002e-06 2021-03-15T22:25:59.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.836 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.836 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128039+0000", 2021-03-15T22:25:59.837 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9277999999999997e-05 2021-03-15T22:25:59.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.837 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128099+0000", 2021-03-15T22:25:59.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2401999999999999e-05 2021-03-15T22:25:59.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.838 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:25:59.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128111+0000", 2021-03-15T22:25:59.839 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99084192299999996 2021-03-15T22:25:59.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.840 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.118953+0000", 2021-03-15T22:25:59.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000162363 2021-03-15T22:25:59.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.840 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.840 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.841 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.119115+0000", 2021-03-15T22:25:59.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023901900000000001 2021-03-15T22:25:59.841 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.842 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:25:59.842 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.119354+0000", 2021-03-15T22:25:59.842 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020445160000000001 2021-03-15T22:25:59.842 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.842 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.843 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.843 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121399+0000", 2021-03-15T22:25:59.843 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021134299999999999 2021-03-15T22:25:59.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.844 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.844 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123512+0000", 2021-03-15T22:25:59.844 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000816028 2021-03-15T22:25:59.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.844 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.845 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.845 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124328+0000", 2021-03-15T22:25:59.845 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2724000000000005e-05 2021-03-15T22:25:59.845 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.846 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.846 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124391+0000", 2021-03-15T22:25:59.846 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012357399999999999 2021-03-15T22:25:59.846 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124515+0000", 2021-03-15T22:25:59.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99727343300000004 2021-03-15T22:25:59.847 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.848 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.848 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.848 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3951 7.7 7:ed4a01d2:::smithi07694878-2:head [write 1907912~570853] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:25:59.849 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:25:59.849 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 24.480890889000001, 2021-03-15T22:25:59.849 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99540528800000005, 2021-03-15T22:25:59.849 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.850 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.850 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.850 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.850 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.850 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3951 2021-03-15T22:25:59.851 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.851 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.851 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.851 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.851 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:25:59.852 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.852 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.852 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.852 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.853 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:25:59.853 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00050036199999999996 2021-03-15T22:25:59.853 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.853 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.853 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.854 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134484+0000", 2021-03-15T22:25:59.854 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9979999999999999e-06 2021-03-15T22:25:59.854 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.854 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.854 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.855 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134487+0000", 2021-03-15T22:25:59.855 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3523000000000001e-05 2021-03-15T22:25:59.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.855 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.856 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.856 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134500+0000", 2021-03-15T22:25:59.856 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5979999999999996e-06 2021-03-15T22:25:59.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.856 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.857 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134506+0000", 2021-03-15T22:25:59.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0407000000000001e-05 2021-03-15T22:25:59.857 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.857 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.858 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.858 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134546+0000", 2021-03-15T22:25:59.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5650000000000002e-06 2021-03-15T22:25:59.858 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.859 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.859 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:25:59.859 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134555+0000", 2021-03-15T22:25:59.859 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98647964300000002 2021-03-15T22:25:59.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.860 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.860 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.860 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121034+0000", 2021-03-15T22:25:59.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104298 2021-03-15T22:25:59.860 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.861 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:59.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121139+0000", 2021-03-15T22:25:59.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0034252140000000002 2021-03-15T22:25:59.861 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.862 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124564+0000", 2021-03-15T22:25:59.862 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3158000000000005e-05 2021-03-15T22:25:59.863 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.863 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.863 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.863 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124657+0000", 2021-03-15T22:25:59.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000172645 2021-03-15T22:25:59.864 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.864 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.864 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:25:59.864 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124830+0000", 2021-03-15T22:25:59.864 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001768096 2021-03-15T22:25:59.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.865 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.865 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.865 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126598+0000", 2021-03-15T22:25:59.865 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020473919999999999 2021-03-15T22:25:59.866 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.866 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.866 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128645+0000", 2021-03-15T22:25:59.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00067562299999999998 2021-03-15T22:25:59.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.867 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.867 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.867 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129321+0000", 2021-03-15T22:25:59.868 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3652999999999998e-05 2021-03-15T22:25:59.868 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.868 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.868 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.868 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129354+0000", 2021-03-15T22:25:59.869 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4112999999999997e-05 2021-03-15T22:25:59.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.869 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.869 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.869 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129389+0000", 2021-03-15T22:25:59.870 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99540528800000005 2021-03-15T22:25:59.870 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.870 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.870 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.871 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3952 7.7 7:ed4a01d2:::smithi07694878-2:head [write 2985346~73627] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:25:59.871 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:25:59.871 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 24.477696389999998, 2021-03-15T22:25:59.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99236444199999996, 2021-03-15T22:25:59.872 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.872 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.872 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.872 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.873 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.873 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3952 2021-03-15T22:25:59.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.873 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.874 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.874 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.874 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:25:59.874 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.874 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.875 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.875 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.875 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:25:59.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5050000000000006e-05 2021-03-15T22:25:59.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.876 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.876 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.876 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137273+0000", 2021-03-15T22:25:59.876 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.883e-06 2021-03-15T22:25:59.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.877 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.877 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137275+0000", 2021-03-15T22:25:59.877 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0764000000000001e-05 2021-03-15T22:25:59.878 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.878 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.878 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.878 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137285+0000", 2021-03-15T22:25:59.878 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2190000000000001e-06 2021-03-15T22:25:59.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.879 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.879 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.879 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137292+0000", 2021-03-15T22:25:59.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5432999999999997e-05 2021-03-15T22:25:59.880 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.880 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.880 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137337+0000", 2021-03-15T22:25:59.881 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.738e-06 2021-03-15T22:25:59.881 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.881 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:25:59.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137346+0000", 2021-03-15T22:25:59.882 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98385600600000001 2021-03-15T22:25:59.882 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.882 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121202+0000", 2021-03-15T22:25:59.883 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9746e-05 2021-03-15T22:25:59.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.883 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.883 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:59.883 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121262+0000", 2021-03-15T22:25:59.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041684740000000001 2021-03-15T22:25:59.884 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.884 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.884 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.885 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125430+0000", 2021-03-15T22:25:59.885 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0578000000000001e-05 2021-03-15T22:25:59.885 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.885 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.885 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.886 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125501+0000", 2021-03-15T22:25:59.886 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8120000000000002e-05 2021-03-15T22:25:59.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.886 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.886 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:25:59.887 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125599+0000", 2021-03-15T22:25:59.887 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010039140000000001 2021-03-15T22:25:59.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.887 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.888 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126603+0000", 2021-03-15T22:25:59.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020828969999999998 2021-03-15T22:25:59.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.888 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.889 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.889 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128686+0000", 2021-03-15T22:25:59.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00075311200000000001 2021-03-15T22:25:59.889 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.889 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.890 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.890 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129439+0000", 2021-03-15T22:25:59.890 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7739999999999999e-05 2021-03-15T22:25:59.890 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.891 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.891 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.891 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129496+0000", 2021-03-15T22:25:59.891 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5768e-05 2021-03-15T22:25:59.891 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.892 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.892 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129542+0000", 2021-03-15T22:25:59.892 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99236444199999996 2021-03-15T22:25:59.892 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.893 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.893 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.893 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.893 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.893 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3953 7.7 7:ed4a01d2:::smithi07694878-2:head [setxattr _header (57) in=7b,truncate 3058973] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:25:59.894 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:25:59.894 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 24.477571535999999, 2021-03-15T22:25:59.894 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99574138599999995, 2021-03-15T22:25:59.894 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.895 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.895 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.895 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.895 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.895 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3953 2021-03-15T22:25:59.896 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.896 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:25:59.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.897 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.897 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.898 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:25:59.898 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9230000000000001e-06 2021-03-15T22:25:59.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.898 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.898 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.899 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137305+0000", 2021-03-15T22:25:59.899 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.252e-06 2021-03-15T22:25:59.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.899 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.899 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.900 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137306+0000", 2021-03-15T22:25:59.900 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9039999999999997e-06 2021-03-15T22:25:59.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.900 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.901 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.901 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137312+0000", 2021-03-15T22:25:59.901 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.856e-06 2021-03-15T22:25:59.901 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.901 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.902 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.902 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137316+0000", 2021-03-15T22:25:59.902 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6783000000000002e-05 2021-03-15T22:25:59.902 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.902 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.903 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.903 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137372+0000", 2021-03-15T22:25:59.903 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9349999999999999e-06 2021-03-15T22:25:59.903 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.903 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.904 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:25:59.904 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137378+0000", 2021-03-15T22:25:59.904 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98391964700000001 2021-03-15T22:25:59.904 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.905 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.905 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.905 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121298+0000", 2021-03-15T22:25:59.905 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9109000000000004e-05 2021-03-15T22:25:59.905 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.906 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.906 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:59.906 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121387+0000", 2021-03-15T22:25:59.906 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045047059999999998 2021-03-15T22:25:59.906 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.907 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.907 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.907 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125892+0000", 2021-03-15T22:25:59.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6956999999999998e-05 2021-03-15T22:25:59.907 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.908 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.908 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.908 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125939+0000", 2021-03-15T22:25:59.908 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012742400000000001 2021-03-15T22:25:59.909 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:25:59.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126066+0000", 2021-03-15T22:25:59.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000545302 2021-03-15T22:25:59.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.910 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.910 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.910 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126611+0000", 2021-03-15T22:25:59.911 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021128950000000001 2021-03-15T22:25:59.911 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.911 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.911 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.911 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128724+0000", 2021-03-15T22:25:59.912 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042103729999999999 2021-03-15T22:25:59.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.912 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.912 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.912 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.132935+0000", 2021-03-15T22:25:59.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5052000000000001e-05 2021-03-15T22:25:59.913 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.913 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.914 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.132990+0000", 2021-03-15T22:25:59.914 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4268000000000003e-05 2021-03-15T22:25:59.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.914 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.914 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133044+0000", 2021-03-15T22:25:59.915 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99574138599999995 2021-03-15T22:25:59.915 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.915 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.915 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.916 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3954 7.7 7:ed4a01d2:::smithi07694878-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e420)", 2021-03-15T22:25:59.916 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:25:59.917 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 24.477542367000002, 2021-03-15T22:25:59.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99588332599999996, 2021-03-15T22:25:59.917 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.917 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:25:59.917 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.918 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.918 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.918 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3954 2021-03-15T22:25:59.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.918 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.919 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.919 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.919 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:25:59.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.920 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.920 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:25:59.920 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.015e-06 2021-03-15T22:25:59.921 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.921 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.921 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.921 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137333+0000", 2021-03-15T22:25:59.921 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5290000000000001e-06 2021-03-15T22:25:59.922 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.922 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.922 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137334+0000", 2021-03-15T22:25:59.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.489e-06 2021-03-15T22:25:59.923 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.923 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.923 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137341+0000", 2021-03-15T22:25:59.924 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1989999999999997e-06 2021-03-15T22:25:59.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.924 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.924 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137346+0000", 2021-03-15T22:25:59.925 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0890999999999998e-05 2021-03-15T22:25:59.925 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.925 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.925 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.925 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137397+0000", 2021-03-15T22:25:59.926 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3159999999999997e-06 2021-03-15T22:25:59.926 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.926 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.926 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:25:59.926 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137405+0000", 2021-03-15T22:25:59.927 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98402323800000002 2021-03-15T22:25:59.927 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121428+0000", 2021-03-15T22:25:59.928 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2376999999999998e-05 2021-03-15T22:25:59.928 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.928 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.928 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:59.929 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121471+0000", 2021-03-15T22:25:59.929 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050435389999999997 2021-03-15T22:25:59.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.929 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.930 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126514+0000", 2021-03-15T22:25:59.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3162000000000003e-05 2021-03-15T22:25:59.930 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.930 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.930 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:25:59.931 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126578+0000", 2021-03-15T22:25:59.931 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0065105759999999997 2021-03-15T22:25:59.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.932 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.932 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133088+0000", 2021-03-15T22:25:59.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4956000000000006e-05 2021-03-15T22:25:59.932 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.932 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.933 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.933 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133153+0000", 2021-03-15T22:25:59.933 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2038999999999993e-05 2021-03-15T22:25:59.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.933 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.934 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.934 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133215+0000", 2021-03-15T22:25:59.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99588332599999996 2021-03-15T22:25:59.934 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.935 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.935 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.935 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.935 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3970 7.7 7:ecc0cb20:::smithi07694878-25:head [write 709394~646021] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:25:59.936 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:25:59.936 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 24.335605504, 2021-03-15T22:25:59.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84719129900000001, 2021-03-15T22:25:59.936 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.936 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.937 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.937 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.937 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.937 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3970 2021-03-15T22:25:59.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.938 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.938 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.938 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.938 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:25:59.939 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.939 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.939 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.939 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:25:59.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000550652 2021-03-15T22:25:59.940 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.940 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279819+0000", 2021-03-15T22:25:59.941 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8980000000000004e-06 2021-03-15T22:25:59.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.941 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.941 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279824+0000", 2021-03-15T22:25:59.942 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1767999999999999e-05 2021-03-15T22:25:59.942 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.942 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.942 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.943 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279846+0000", 2021-03-15T22:25:59.943 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2509999999999994e-06 2021-03-15T22:25:59.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.943 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.944 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279854+0000", 2021-03-15T22:25:59.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7908000000000001e-05 2021-03-15T22:25:59.944 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.944 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.944 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279922+0000", 2021-03-15T22:25:59.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2315e-05 2021-03-15T22:25:59.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.945 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:25:59.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279934+0000", 2021-03-15T22:25:59.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84156851799999999 2021-03-15T22:25:59.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121503+0000", 2021-03-15T22:25:59.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012479099999999999 2021-03-15T22:25:59.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.948 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.948 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121628+0000", 2021-03-15T22:25:59.948 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000208667 2021-03-15T22:25:59.948 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.949 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:25:59.949 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121836+0000", 2021-03-15T22:25:59.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023170439999999999 2021-03-15T22:25:59.949 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.949 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.950 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124153+0000", 2021-03-15T22:25:59.950 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012293359999999999 2021-03-15T22:25:59.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.951 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.951 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.951 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125383+0000", 2021-03-15T22:25:59.951 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00099435699999999997 2021-03-15T22:25:59.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.952 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.952 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126377+0000", 2021-03-15T22:25:59.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3995999999999997e-05 2021-03-15T22:25:59.953 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.953 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.953 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126421+0000", 2021-03-15T22:25:59.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8797999999999997e-05 2021-03-15T22:25:59.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.954 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.954 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126460+0000", 2021-03-15T22:25:59.954 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84719129900000001 2021-03-15T22:25:59.955 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.955 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.955 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.955 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.956 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3971 7.7 7:ecc0cb20:::smithi07694878-25:head [write 2083358~718180] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:25:59.956 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:25:59.956 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 24.327639874999999, 2021-03-15T22:25:59.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.840294447, 2021-03-15T22:25:59.957 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.957 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.957 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.957 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.957 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.958 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3971 2021-03-15T22:25:59.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.958 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.958 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:25:59.959 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.959 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:25:59.960 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00065142800000000001 2021-03-15T22:25:59.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.961 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287886+0000", 2021-03-15T22:25:59.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.418e-06 2021-03-15T22:25:59.961 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.962 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.962 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287888+0000", 2021-03-15T22:25:59.962 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2062000000000001e-05 2021-03-15T22:25:59.962 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287900+0000", 2021-03-15T22:25:59.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7250000000000002e-06 2021-03-15T22:25:59.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.964 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287906+0000", 2021-03-15T22:25:59.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9849000000000003e-05 2021-03-15T22:25:59.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287956+0000", 2021-03-15T22:25:59.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6460000000000005e-06 2021-03-15T22:25:59.965 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:25:59.966 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287965+0000", 2021-03-15T22:25:59.966 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83445325400000003 2021-03-15T22:25:59.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.967 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.967 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.967 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122419+0000", 2021-03-15T22:25:59.967 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7459e-05 2021-03-15T22:25:59.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.968 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.968 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.968 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122476+0000", 2021-03-15T22:25:59.968 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012662400000000001 2021-03-15T22:25:59.969 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.969 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.969 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:25:59.969 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122603+0000", 2021-03-15T22:25:59.969 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037072580000000002 2021-03-15T22:25:59.970 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.970 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.970 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.970 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126310+0000", 2021-03-15T22:25:59.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000275853 2021-03-15T22:25:59.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.971 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.971 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126586+0000", 2021-03-15T22:25:59.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00087237800000000004 2021-03-15T22:25:59.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.972 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127458+0000", 2021-03-15T22:25:59.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5484000000000001e-05 2021-03-15T22:25:59.973 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.973 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.974 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127494+0000", 2021-03-15T22:25:59.974 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5009000000000003e-05 2021-03-15T22:25:59.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.974 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.975 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127529+0000", 2021-03-15T22:25:59.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.840294447 2021-03-15T22:25:59.975 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.975 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.975 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.976 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.976 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3972 7.7 7:ecc0cb20:::smithi07694878-25:head [setxattr _header (58) in=7b,truncate 3109394] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:25:59.976 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:25:59.976 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 24.326957917000001, 2021-03-15T22:25:59.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84111847799999995, 2021-03-15T22:25:59.977 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.977 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:25:59.977 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.978 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.978 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.978 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3972 2021-03-15T22:25:59.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.978 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:25:59.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:25:59.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:25:59.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:25:59.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9489999999999999e-06 2021-03-15T22:25:59.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:25:59.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287918+0000", 2021-03-15T22:25:59.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.359e-06 2021-03-15T22:25:59.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:25:59.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287920+0000", 2021-03-15T22:25:59.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8050000000000003e-06 2021-03-15T22:25:59.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.983 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:25:59.983 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287925+0000", 2021-03-15T22:25:59.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5959999999999997e-06 2021-03-15T22:25:59.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:25:59.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287930+0000", 2021-03-15T22:25:59.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2366999999999993e-05 2021-03-15T22:25:59.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.985 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288002+0000", 2021-03-15T22:25:59.986 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2240000000000001e-06 2021-03-15T22:25:59.986 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.986 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.986 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:25:59.986 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288012+0000", 2021-03-15T22:25:59.987 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83555615299999997 2021-03-15T22:25:59.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.987 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:25:59.987 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123568+0000", 2021-03-15T22:25:59.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4648999999999995e-05 2021-03-15T22:25:59.988 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.988 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.988 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:25:59.989 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123662+0000", 2021-03-15T22:25:59.989 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020287300000000001 2021-03-15T22:25:59.989 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.989 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.989 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:25:59.990 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123865+0000", 2021-03-15T22:25:59.990 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002471843 2021-03-15T22:25:59.990 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.990 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.990 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.991 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126337+0000", 2021-03-15T22:25:59.991 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025485200000000002 2021-03-15T22:25:59.991 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.992 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:25:59.992 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126592+0000", 2021-03-15T22:25:59.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002302077 2021-03-15T22:25:59.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.992 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.993 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:25:59.993 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128894+0000", 2021-03-15T22:25:59.993 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6276999999999998e-05 2021-03-15T22:25:59.993 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.993 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:25:59.994 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128930+0000", 2021-03-15T22:25:59.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104454 2021-03-15T22:25:59.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.994 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:25:59.995 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129035+0000", 2021-03-15T22:25:59.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84111847799999995 2021-03-15T22:25:59.995 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.996 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:25:59.996 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:25:59.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.996 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3973 7.7 7:ecc0cb20:::smithi07694878-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e420)", 2021-03-15T22:25:59.997 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:25:59.997 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 24.326926647000001, 2021-03-15T22:25:59.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.841325871, 2021-03-15T22:25:59.997 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:25:59.997 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:25:59.998 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:25:59.998 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:25:59.998 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:25:59.998 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3973 2021-03-15T22:25:59.998 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:25:59.999 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:25:59.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:25:59.999 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:25:59.999 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:26:00.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:26:00.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:26:00.000 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:26:00.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0400000000000005e-07 2021-03-15T22:26:00.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.001 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:26:00.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:26:00.002 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3090000000000001e-06 2021-03-15T22:26:00.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.002 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:26:00.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287950+0000", 2021-03-15T22:26:00.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1809999999999998e-06 2021-03-15T22:26:00.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:26:00.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287956+0000", 2021-03-15T22:26:00.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9060000000000004e-06 2021-03-15T22:26:00.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.004 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:26:00.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287960+0000", 2021-03-15T22:26:00.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8666999999999997e-05 2021-03-15T22:26:00.005 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:26:00.006 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288039+0000", 2021-03-15T22:26:00.006 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7850000000000003e-06 2021-03-15T22:26:00.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.007 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:26:00.007 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288045+0000", 2021-03-15T22:26:00.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83615280199999997 2021-03-15T22:26:00.007 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.007 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.008 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:26:00.008 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124198+0000", 2021-03-15T22:26:00.008 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2801999999999998e-05 2021-03-15T22:26:00.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.008 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.009 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:26:00.009 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124281+0000", 2021-03-15T22:26:00.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047980210000000004 2021-03-15T22:26:00.009 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.010 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.010 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:26:00.010 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129079+0000", 2021-03-15T22:26:00.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2120000000000007e-05 2021-03-15T22:26:00.010 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.011 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.011 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:26:00.011 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129161+0000", 2021-03-15T22:26:00.011 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011237400000000001 2021-03-15T22:26:00.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:26:00.012 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:26:00.012 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:26:00.012 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129273+0000", 2021-03-15T22:26:00.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.841325871 2021-03-15T22:26:00.012 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:26:00.013 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:26:00.013 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:26:00.013 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:26:00.013 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:26:00.014 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:26:00.014 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:00.019 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:26:00.022 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:26:00.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:25:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:26:00.082 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:00.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:00.084 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:00.123 INFO:tasks.rados.rados.0.smithi076.stdout:1652: done (3 left) 2021-03-15T22:26:00.123 INFO:tasks.rados.rados.0.smithi076.stdout:1653: write (excl) oid 13 current snap is 181 2021-03-15T22:26:00.124 INFO:tasks.rados.rados.0.smithi076.stdout:1653: seq_num 585 ranges {632095=463493,1616319=422534,2748686=487304} 2021-03-15T22:26:00.127 INFO:tasks.rados.rados.0.smithi076.stdout:1653: writing smithi07694878-13 from 632095 to 1095588 tid 1 2021-03-15T22:26:00.133 INFO:tasks.rados.rados.0.smithi076.stdout:1653: writing smithi07694878-13 from 1616319 to 2038853 tid 2 2021-03-15T22:26:00.140 INFO:tasks.rados.rados.0.smithi076.stdout:1653: writing smithi07694878-13 from 2748686 to 3235990 tid 3 2021-03-15T22:26:00.140 INFO:tasks.rados.rados.0.smithi076.stdout:1654: delete oid 3 current snap is 181 2021-03-15T22:26:00.141 INFO:tasks.rados.rados.0.smithi076.stdout:1651: finishing rollback tid 1 to smithi07694878-30 2021-03-15T22:26:00.141 INFO:tasks.rados.rados.0.smithi076.stdout:1651: finishing rollback tid 2 to smithi07694878-30 2021-03-15T22:26:00.141 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 1001 (ObjNum 490 snap 145 seq_num 490) dirty exists 2021-03-15T22:26:00.141 INFO:tasks.rados.rados.0.smithi076.stdout:1653: finishing write tid 1 to smithi07694878-13 2021-03-15T22:26:00.141 INFO:tasks.rados.rados.0.smithi076.stdout:1653: finishing write tid 2 to smithi07694878-13 2021-03-15T22:26:00.143 INFO:tasks.rados.rados.0.smithi076.stdout:1649: done (4 left) 2021-03-15T22:26:00.143 INFO:tasks.rados.rados.0.smithi076.stdout:1650: done (3 left) 2021-03-15T22:26:00.144 INFO:tasks.rados.rados.0.smithi076.stdout:1651: done (2 left) 2021-03-15T22:26:00.144 INFO:tasks.rados.rados.0.smithi076.stdout:1654: done (1 left) 2021-03-15T22:26:00.144 INFO:tasks.rados.rados.0.smithi076.stdout:1655: rollback oid 3 current snap is 181 2021-03-15T22:26:00.145 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 143 2021-03-15T22:26:00.145 INFO:tasks.rados.rados.0.smithi076.stdout:1656: write oid 18 current snap is 181 2021-03-15T22:26:00.145 INFO:tasks.rados.rados.0.smithi076.stdout:1656: seq_num 587 ranges {665709=730344,2133283=426203,3203532=509848} 2021-03-15T22:26:00.150 INFO:tasks.rados.rados.0.smithi076.stdout:1656: writing smithi07694878-18 from 665709 to 1396053 tid 1 2021-03-15T22:26:00.157 INFO:tasks.rados.rados.0.smithi076.stdout:1656: writing smithi07694878-18 from 2133283 to 2559486 tid 2 2021-03-15T22:26:00.164 INFO:tasks.rados.rados.0.smithi076.stdout:1656: writing smithi07694878-18 from 3203532 to 3713380 tid 3 2021-03-15T22:26:00.165 INFO:tasks.rados.rados.0.smithi076.stdout:1657: rollback oid 2 current snap is 181 2021-03-15T22:26:00.165 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 2 to 155 2021-03-15T22:26:00.166 INFO:tasks.rados.rados.0.smithi076.stdout:1653: finishing write tid 3 to smithi07694878-13 2021-03-15T22:26:00.166 INFO:tasks.rados.rados.0.smithi076.stdout:1653: finishing write tid 4 to smithi07694878-13 2021-03-15T22:26:00.166 INFO:tasks.rados.rados.0.smithi076.stdout:1653: finishing write tid 5 to smithi07694878-13 2021-03-15T22:26:00.167 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 429 (ObjNum 585 snap 181 seq_num 585) dirty exists 2021-03-15T22:26:00.167 INFO:tasks.rados.rados.0.smithi076.stdout:1653: left oid 13 (ObjNum 585 snap 181 seq_num 585) 2021-03-15T22:26:00.167 INFO:tasks.rados.rados.0.smithi076.stdout:1655: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:26:00.168 INFO:tasks.rados.rados.0.smithi076.stdout:1655: finishing rollback tid 2 to smithi07694878-3 2021-03-15T22:26:00.168 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 767 (ObjNum 430 snap 119 seq_num 430) dirty exists 2021-03-15T22:26:00.169 INFO:tasks.rados.rados.0.smithi076.stdout:1656: finishing write tid 1 to smithi07694878-18 2021-03-15T22:26:00.169 INFO:tasks.rados.rados.0.smithi076.stdout:1653: done (3 left) 2021-03-15T22:26:00.170 INFO:tasks.rados.rados.0.smithi076.stdout:1655: done (2 left) 2021-03-15T22:26:00.170 INFO:tasks.rados.rados.0.smithi076.stdout:1658: rollback oid 26 current snap is 181 2021-03-15T22:26:00.170 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 161 2021-03-15T22:26:00.171 INFO:tasks.rados.rados.0.smithi076.stdout:1656: finishing write tid 2 to smithi07694878-18 2021-03-15T22:26:00.171 INFO:tasks.rados.rados.0.smithi076.stdout:1659: delete oid 46 current snap is 181 2021-03-15T22:26:00.185 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:00.195 INFO:tasks.rados.rados.0.smithi076.stdout:1658: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:26:00.195 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 664 (ObjNum 261778096 snap 21896 seq_num 2521158368) dirty dne 2021-03-15T22:26:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:00.210 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:00.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:00.212 INFO:tasks.rados.rados.0.smithi076.stdout:1658: done (3 left) 2021-03-15T22:26:00.213 INFO:tasks.rados.rados.0.smithi076.stdout:1659: done (2 left) 2021-03-15T22:26:00.213 INFO:tasks.rados.rados.0.smithi076.stdout:1660: snap_remove snap 117 2021-03-15T22:26:00.262 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-15T22:26:00.312 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:00.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:00.337 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:00.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:00.438 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:00.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:00.464 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:00.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:00.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[56514]: cluster 2021-03-15T22:25:59.061632+0000 mgr.y (mgr.14144) 645 : cluster [DBG] pgmap v792: 138 pgs: 138 active+clean; 399 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 18 op/s 2021-03-15T22:26:00.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[56514]: cluster 2021-03-15T22:25:59.166450+0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2021-03-15T22:26:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[56514]: audit 2021-03-15T22:25:59.741438+0000 mgr.y (mgr.14144) 646 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[56514]: audit 2021-03-15T22:25:59.823643+0000 mgr.y (mgr.14144) 647 : audit [DBG] from='client.15543 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:26:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[56514]: cluster 2021-03-15T22:26:00.116357+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2021-03-15T22:26:00.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[62156]: cluster 2021-03-15T22:25:59.061632+0000 mgr.y ( 2021-03-15T22:26:00.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[62156]: mgr.14144) 645 : cluster [DBG] pgmap v792: 138 pgs: 138 active+clean; 399 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 18 op/s 2021-03-15T22:26:00.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[62156]: cluster 2021-03-15T22:25:59.166450+0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2021-03-15T22:26:00.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[62156]: audit 2021-03-15T22:25:59.741438+0000 mgr.y (mgr.14144) 646 : audit [DBG] 2021-03-15T22:26:00.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[62156]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:00.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[62156]: audit 2021-03-15T22:25:59.823643+0000 mgr.y (mgr.14144) 647 : audit [DBG] 2021-03-15T22:26:00.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[62156]: from='client.15543 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:26:00.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[62156]: cluster 2021-03-15T22:26:00. 2021-03-15T22:26:00.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:00 smithi076 conmon[62156]: 116357+0000 mon.a (mon.0) 1284 : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2021-03-15T22:26:00.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:00 smithi125 conmon[56739]: cluster 2021-03-15T22:25:59. 2021-03-15T22:26:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:00 smithi125 conmon[56739]: 061632+0000 mgr.y (mgr.14144) 645 : cluster [DBG] pgmap v792: 138 pgs: 138 active+clean; 399 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 18 op/s 2021-03-15T22:26:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:00 smithi125 conmon[56739]: cluster 2021-03-15T22:25:59.166450+0000 mon.a (mon.0) 1283 : cluster [DBG] osdmap e444: 8 total, 8 up, 7 in 2021-03-15T22:26:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:00 smithi125 conmon[56739]: audit 2021-03-15T22:25:59.741438+0000 mgr.y (mgr.14144) 646 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:00 smithi125 conmon[56739]: audit 2021-03-15T22:25:59.823643+0000 mgr.y (mgr.14144) 647 : audit [DBG] from='client.15543 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:26:00.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:00 smithi125 conmon[56739]: cluster 2021-03-15T22:26:00.116357+0000 mon.a (mon.0) 1284 2021-03-15T22:26:00.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:00 smithi125 conmon[56739]: : cluster [DBG] osdmap e445: 8 total, 8 up, 7 in 2021-03-15T22:26:00.538 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:26:00.566 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:00.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:00.591 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:00.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:00.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:00.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:00.718 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:00.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:00.820 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:00.845 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:00.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:00.847 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:00.933 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":793,"stamp":"2021-03-15T22:25:59.163934+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":418668563,"num_objects":517,"num_object_clones":152,"num_object_copies":1551,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":517,"num_whiteouts":15,"num_read":4762,"num_read_kb":3288741,"num_write":2891,"num_write_kb":887843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1680,"num_bytes_recovered":514354880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7212,"ondisk_log_size":7212,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1154948,"kb_used_data":976032,"kb_used_omap":0,"kb_used_meta":177664,"kb_avail":655061116,"statfs":{"total":671965249536,"available":670782582784,"internally_reserved":0,"allocated":999456768,"data_stored":1344859827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":181927936},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13396275,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":44,"num_read_kb":49568,"num_write":29,"num_write_kb":8237,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.059949"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"403","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.118318+0000","last_change":"2021-03-15T22:25:41.036097+0000","last_active":"2021-03-15T22:25:55.118318+0000","last_peered":"2021-03-15T22:25:55.118318+0000","last_clean":"2021-03-15T22:25:55.118318+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:25:55.118318+0000","last_undegraded":"2021-03-15T22:25:55.118318+0000","last_fullsized":"2021-03-15T22:25:55.118318+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:33.728929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:33.728929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"108","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.189264+0000","last_change":"2021-03-15T22:24:31.865035+0000","last_active":"2021-03-15T22:25:56.189264+0000","last_peered":"2021-03-15T22:25:56.189264+0000","last_clean":"2021-03-15T22:25:56.189264+0000","last_became_active":"2021-03-15T22:24:31.103578+0000","last_became_peered":"2021-03-15T22:24:31.103578+0000","last_unstale":"2021-03-15T22:25:56.189264+0000","last_undegraded":"2021-03-15T22:25:56.189264+0000","last_fullsized":"2021-03-15T22:25:56.189264+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:23:59.500680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:59.500680+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"435","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.118340+0000","last_change":"2021-03-15T22:25:40.007165+0000","last_active":"2021-03-15T22:25:55.118340+0000","last_peered":"2021-03-15T22:25:55.118340+0000","last_clean":"2021-03-15T22:25:55.118340+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:25:55.118340+0000","last_undegraded":"2021-03-15T22:25:55.118340+0000","last_fullsized":"2021-03-15T22:25:55.118340+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:26.743811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.743811+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"33","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.111835+0000","last_change":"2021-03-15T22:25:33.149020+0000","last_active":"2021-03-15T22:25:55.111835+0000","last_peered":"2021-03-15T22:25:55.111835+0000","last_clean":"2021-03-15T22:25:55.111835+0000","last_became_active":"2021-03-15T22:25:33.148710+0000","last_became_peered":"2021-03-15T22:25:33.148710+0000","last_unstale":"2021-03-15T22:25:55.111835+0000","last_undegraded":"2021-03-15T22:25:55.111835+0000","last_fullsized":"2021-03-15T22:25:55.111835+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:06.177754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.177754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"33","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.112285+0000","last_change":"2021-03-15T22:25:33.147017+0000","last_active":"2021-03-15T22:25:55.112285+0000","last_peered":"2021-03-15T22:25:55.112285+0000","last_clean":"2021-03-15T22:25:55.112285+0000","last_became_active":"2021-03-15T22:25:33.146697+0000","last_became_peered":"2021-03-15T22:25:33.146697+0000","last_unstale":"2021-03-15T22:25:55.112285+0000","last_undegraded":"2021-03-15T22:25:55.112285+0000","last_fullsized":"2021-03-15T22:25:55.112285+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:15.333746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.333746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"59","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.111995+0000","last_change":"2021-03-15T22:25:33.175542+0000","last_active":"2021-03-15T22:25:55.111995+0000","last_peered":"2021-03-15T22:25:55.111995+0000","last_clean":"2021-03-15T22:25:55.111995+0000","last_became_active":"2021-03-15T22:25:33.137906+0000","last_became_peered":"2021-03-15T22:25:33.137906+0000","last_unstale":"2021-03-15T22:25:55.111995+0000","last_undegraded":"2021-03-15T22:25:55.111995+0000","last_fullsized":"2021-03-15T22:25:55.111995+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:23:42.144821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.144821+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":21,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"97","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.143673+0000","last_change":"2021-03-15T22:24:31.109745+0000","last_active":"2021-03-15T22:25:56.143673+0000","last_peered":"2021-03-15T22:25:56.143673+0000","last_clean":"2021-03-15T22:25:56.143673+0000","last_became_active":"2021-03-15T22:24:31.108652+0000","last_became_peered":"2021-03-15T22:24:31.108652+0000","last_unstale":"2021-03-15T22:25:56.143673+0000","last_undegraded":"2021-03-15T22:25:56.143673+0000","last_fullsized":"2021-03-15T22:25:56.143673+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:35.677439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:35.677439+0000","last_clean_scrub_stamp":"2021-03-15T22:23:35.677439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"407","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.189247+0000","last_change":"2021-03-15T22:25:16.598073+0000","last_active":"2021-03-15T22:25:56.189247+0000","last_peered":"2021-03-15T22:25:56.189247+0000","last_clean":"2021-03-15T22:25:56.189247+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:25:56.189247+0000","last_undegraded":"2021-03-15T22:25:56.189247+0000","last_fullsized":"2021-03-15T22:25:56.189247+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.597998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.597998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"510","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.118526+0000","last_change":"2021-03-15T22:24:31.776304+0000","last_active":"2021-03-15T22:25:55.118526+0000","last_peered":"2021-03-15T22:25:55.118526+0000","last_clean":"2021-03-15T22:25:55.118526+0000","last_became_active":"2021-03-15T22:24:31.083373+0000","last_became_peered":"2021-03-15T22:24:31.083373+0000","last_unstale":"2021-03-15T22:25:55.118526+0000","last_undegraded":"2021-03-15T22:25:55.118526+0000","last_fullsized":"2021-03-15T22:25:55.118526+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:24:10.979322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:10.979322+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"116","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.188402+0000","last_change":"2021-03-15T22:25:33.143128+0000","last_active":"2021-03-15T22:25:56.188402+0000","last_peered":"2021-03-15T22:25:56.188402+0000","last_clean":"2021-03-15T22:25:56.188402+0000","last_became_active":"2021-03-15T22:25:33.142493+0000","last_became_peered":"2021-03-15T22:25:33.142493+0000","last_unstale":"2021-03-15T22:25:56.188402+0000","last_undegraded":"2021-03-15T22:25:56.188402+0000","last_fullsized":"2021-03-15T22:25:56.188402+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.674895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"407","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.188633+0000","last_change":"2021-03-15T22:25:05.682093+0000","last_active":"2021-03-15T22:25:56.188633+0000","last_peered":"2021-03-15T22:25:56.188633+0000","last_clean":"2021-03-15T22:25:56.188633+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:25:56.188633+0000","last_undegraded":"2021-03-15T22:25:56.188633+0000","last_fullsized":"2021-03-15T22:25:56.188633+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.682031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.682031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"121'9","reported_seq":"476","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.118529+0000","last_change":"2021-03-15T22:24:37.902248+0000","last_active":"2021-03-15T22:25:55.118529+0000","last_peered":"2021-03-15T22:25:55.118529+0000","last_clean":"2021-03-15T22:25:55.118529+0000","last_became_active":"2021-03-15T22:22:56.055083+0000","last_became_peered":"2021-03-15T22:22:56.055083+0000","last_unstale":"2021-03-15T22:25:55.118529+0000","last_undegraded":"2021-03-15T22:25:55.118529+0000","last_fullsized":"2021-03-15T22:25:55.118529+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'9","last_scrub_stamp":"2021-03-15T22:24:37.902166+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.902166+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"403","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.119344+0000","last_change":"2021-03-15T22:24:30.865676+0000","last_active":"2021-03-15T22:25:55.119344+0000","last_peered":"2021-03-15T22:25:55.119344+0000","last_clean":"2021-03-15T22:25:55.119344+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:25:55.119344+0000","last_undegraded":"2021-03-15T22:25:55.119344+0000","last_fullsized":"2021-03-15T22:25:55.119344+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.865609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.865609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"151","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.118507+0000","last_change":"2021-03-15T22:24:55.103109+0000","last_active":"2021-03-15T22:25:55.118507+0000","last_peered":"2021-03-15T22:25:55.118507+0000","last_clean":"2021-03-15T22:25:55.118507+0000","last_became_active":"2021-03-15T22:24:55.090536+0000","last_became_peered":"2021-03-15T22:24:55.090536+0000","last_unstale":"2021-03-15T22:25:55.118507+0000","last_undegraded":"2021-03-15T22:25:55.118507+0000","last_fullsized":"2021-03-15T22:25:55.118507+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:24:28.507563+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.507563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"505","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.123785+0000","last_change":"2021-03-15T22:25:41.703337+0000","last_active":"2021-03-15T22:25:53.123785+0000","last_peered":"2021-03-15T22:25:53.123785+0000","last_clean":"2021-03-15T22:25:53.123785+0000","last_became_active":"2021-03-15T22:25:33.134238+0000","last_became_peered":"2021-03-15T22:25:33.134238+0000","last_unstale":"2021-03-15T22:25:53.123785+0000","last_undegraded":"2021-03-15T22:25:53.123785+0000","last_fullsized":"2021-03-15T22:25:53.123785+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:03.482307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:03.482307+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"231","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.113971+0000","last_change":"2021-03-15T22:25:14.117883+0000","last_active":"2021-03-15T22:25:55.113971+0000","last_peered":"2021-03-15T22:25:55.113971+0000","last_clean":"2021-03-15T22:25:55.113971+0000","last_became_active":"2021-03-15T22:25:14.117558+0000","last_became_peered":"2021-03-15T22:25:14.117558+0000","last_unstale":"2021-03-15T22:25:55.113971+0000","last_undegraded":"2021-03-15T22:25:55.113971+0000","last_fullsized":"2021-03-15T22:25:55.113971+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:00.480944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:00.480944+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"412","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.118610+0000","last_change":"2021-03-15T22:25:11.028502+0000","last_active":"2021-03-15T22:25:55.118610+0000","last_peered":"2021-03-15T22:25:55.118610+0000","last_clean":"2021-03-15T22:25:55.118610+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:25:55.118610+0000","last_undegraded":"2021-03-15T22:25:55.118610+0000","last_fullsized":"2021-03-15T22:25:55.118610+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.028422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.028422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"53","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.112289+0000","last_change":"2021-03-15T22:25:33.169024+0000","last_active":"2021-03-15T22:25:55.112289+0000","last_peered":"2021-03-15T22:25:55.112289+0000","last_clean":"2021-03-15T22:25:55.112289+0000","last_became_active":"2021-03-15T22:25:33.134781+0000","last_became_peered":"2021-03-15T22:25:33.134781+0000","last_unstale":"2021-03-15T22:25:55.112289+0000","last_undegraded":"2021-03-15T22:25:55.112289+0000","last_fullsized":"2021-03-15T22:25:55.112289+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:23:32.049047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:32.049047+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":36,"num_read_kb":19,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"394","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.127854+0000","last_change":"2021-03-15T22:23:52.272270+0000","last_active":"2021-03-15T22:25:53.127854+0000","last_peered":"2021-03-15T22:25:53.127854+0000","last_clean":"2021-03-15T22:25:53.127854+0000","last_became_active":"2021-03-15T22:23:52.271925+0000","last_became_peered":"2021-03-15T22:23:52.271925+0000","last_unstale":"2021-03-15T22:25:53.127854+0000","last_undegraded":"2021-03-15T22:25:53.127854+0000","last_fullsized":"2021-03-15T22:25:53.127854+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:34.782225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:34.782225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"428","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.143751+0000","last_change":"2021-03-15T22:23:40.199174+0000","last_active":"2021-03-15T22:25:56.143751+0000","last_peered":"2021-03-15T22:25:56.143751+0000","last_clean":"2021-03-15T22:25:56.143751+0000","last_became_active":"2021-03-15T22:22:56.062833+0000","last_became_peered":"2021-03-15T22:22:56.062833+0000","last_unstale":"2021-03-15T22:25:56.143751+0000","last_undegraded":"2021-03-15T22:25:56.143751+0000","last_fullsized":"2021-03-15T22:25:56.143751+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:40.199095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:40.199095+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"221","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.128306+0000","last_change":"2021-03-15T22:25:33.752847+0000","last_active":"2021-03-15T22:25:53.128306+0000","last_peered":"2021-03-15T22:25:53.128306+0000","last_clean":"2021-03-15T22:25:53.128306+0000","last_became_active":"2021-03-15T22:25:33.140078+0000","last_became_peered":"2021-03-15T22:25:33.140078+0000","last_unstale":"2021-03-15T22:25:53.128306+0000","last_undegraded":"2021-03-15T22:25:53.128306+0000","last_fullsized":"2021-03-15T22:25:53.128306+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:25:08.192165+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:25:08.192165+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"404","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.144616+0000","last_change":"2021-03-15T22:23:31.003298+0000","last_active":"2021-03-15T22:25:56.144616+0000","last_peered":"2021-03-15T22:25:56.144616+0000","last_clean":"2021-03-15T22:25:56.144616+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:25:56.144616+0000","last_undegraded":"2021-03-15T22:25:56.144616+0000","last_fullsized":"2021-03-15T22:25:56.144616+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:31.003234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:31.003234+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"33","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.114933+0000","last_change":"2021-03-15T22:25:33.133625+0000","last_active":"2021-03-15T22:25:55.114933+0000","last_peered":"2021-03-15T22:25:55.114933+0000","last_clean":"2021-03-15T22:25:55.114933+0000","last_became_active":"2021-03-15T22:25:33.133418+0000","last_became_peered":"2021-03-15T22:25:33.133418+0000","last_unstale":"2021-03-15T22:25:55.114933+0000","last_undegraded":"2021-03-15T22:25:55.114933+0000","last_fullsized":"2021-03-15T22:25:55.114933+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:29.506406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.506406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"436","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.136851+0000","last_change":"2021-03-15T22:24:41.483906+0000","last_active":"2021-03-15T22:25:54.136851+0000","last_peered":"2021-03-15T22:25:54.136851+0000","last_clean":"2021-03-15T22:25:54.136851+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:25:54.136851+0000","last_undegraded":"2021-03-15T22:25:54.136851+0000","last_fullsized":"2021-03-15T22:25:54.136851+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:24:41.483849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:41.483849+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"399","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.126426+0000","last_change":"2021-03-15T22:25:09.177600+0000","last_active":"2021-03-15T22:25:53.126426+0000","last_peered":"2021-03-15T22:25:53.126426+0000","last_clean":"2021-03-15T22:25:53.126426+0000","last_became_active":"2021-03-15T22:23:52.260757+0000","last_became_peered":"2021-03-15T22:23:52.260757+0000","last_unstale":"2021-03-15T22:25:53.126426+0000","last_undegraded":"2021-03-15T22:25:53.126426+0000","last_fullsized":"2021-03-15T22:25:53.126426+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:09.177521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.177521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"410","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.135781+0000","last_change":"2021-03-15T22:25:02.452304+0000","last_active":"2021-03-15T22:25:54.135781+0000","last_peered":"2021-03-15T22:25:54.135781+0000","last_clean":"2021-03-15T22:25:54.135781+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:25:54.135781+0000","last_undegraded":"2021-03-15T22:25:54.135781+0000","last_fullsized":"2021-03-15T22:25:54.135781+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.452236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.452236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"239","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.189348+0000","last_change":"2021-03-15T22:25:33.147111+0000","last_active":"2021-03-15T22:25:56.189348+0000","last_peered":"2021-03-15T22:25:56.189348+0000","last_clean":"2021-03-15T22:25:56.189348+0000","last_became_active":"2021-03-15T22:25:33.146680+0000","last_became_peered":"2021-03-15T22:25:33.146680+0000","last_unstale":"2021-03-15T22:25:56.189348+0000","last_undegraded":"2021-03-15T22:25:56.189348+0000","last_fullsized":"2021-03-15T22:25:56.189348+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:57.444598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:23:57.444598+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"130","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.115857+0000","last_change":"2021-03-15T22:25:15.835780+0000","last_active":"2021-03-15T22:25:55.115857+0000","last_peered":"2021-03-15T22:25:55.115857+0000","last_clean":"2021-03-15T22:25:55.115857+0000","last_became_active":"2021-03-15T22:24:23.142746+0000","last_became_peered":"2021-03-15T22:24:23.142746+0000","last_unstale":"2021-03-15T22:25:55.115857+0000","last_undegraded":"2021-03-15T22:25:55.115857+0000","last_fullsized":"2021-03-15T22:25:55.115857+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:25:15.835675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.835675+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":11,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"33","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.112264+0000","last_change":"2021-03-15T22:25:33.148415+0000","last_active":"2021-03-15T22:25:55.112264+0000","last_peered":"2021-03-15T22:25:55.112264+0000","last_clean":"2021-03-15T22:25:55.112264+0000","last_became_active":"2021-03-15T22:25:33.148083+0000","last_became_peered":"2021-03-15T22:25:33.148083+0000","last_unstale":"2021-03-15T22:25:55.112264+0000","last_undegraded":"2021-03-15T22:25:55.112264+0000","last_fullsized":"2021-03-15T22:25:55.112264+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:42.525975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:40.670674+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.525975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"391","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.127217+0000","last_change":"2021-03-15T22:23:52.263156+0000","last_active":"2021-03-15T22:25:53.127217+0000","last_peered":"2021-03-15T22:25:53.127217+0000","last_clean":"2021-03-15T22:25:53.127217+0000","last_became_active":"2021-03-15T22:23:52.262741+0000","last_became_peered":"2021-03-15T22:23:52.262741+0000","last_unstale":"2021-03-15T22:25:53.127217+0000","last_undegraded":"2021-03-15T22:25:53.127217+0000","last_fullsized":"2021-03-15T22:25:53.127217+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:41.710658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.710658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"402","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.189578+0000","last_change":"2021-03-15T22:23:30.005865+0000","last_active":"2021-03-15T22:25:56.189578+0000","last_peered":"2021-03-15T22:25:56.189578+0000","last_clean":"2021-03-15T22:25:56.189578+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:25:56.189578+0000","last_undegraded":"2021-03-15T22:25:56.189578+0000","last_fullsized":"2021-03-15T22:25:56.189578+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:30.005791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:30.005791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"90'22","reported_seq":"218","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.112257+0000","last_change":"2021-03-15T22:25:16.834493+0000","last_active":"2021-03-15T22:25:55.112257+0000","last_peered":"2021-03-15T22:25:55.112257+0000","last_clean":"2021-03-15T22:25:55.112257+0000","last_became_active":"2021-03-15T22:22:56.056175+0000","last_became_peered":"2021-03-15T22:22:56.056175+0000","last_unstale":"2021-03-15T22:25:55.112257+0000","last_undegraded":"2021-03-15T22:25:55.112257+0000","last_fullsized":"2021-03-15T22:25:55.112257+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:25:16.834409+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.834409+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"101","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.188371+0000","last_change":"2021-03-15T22:24:41.659524+0000","last_active":"2021-03-15T22:25:56.188371+0000","last_peered":"2021-03-15T22:25:56.188371+0000","last_clean":"2021-03-15T22:25:56.188371+0000","last_became_active":"2021-03-15T22:24:31.105380+0000","last_became_peered":"2021-03-15T22:24:31.105380+0000","last_unstale":"2021-03-15T22:25:56.188371+0000","last_undegraded":"2021-03-15T22:25:56.188371+0000","last_fullsized":"2021-03-15T22:25:56.188371+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_clean_scrub_stamp":"2021-03-15T22:24:41.659450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"218","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.112155+0000","last_change":"2021-03-15T22:25:14.115163+0000","last_active":"2021-03-15T22:25:55.112155+0000","last_peered":"2021-03-15T22:25:55.112155+0000","last_clean":"2021-03-15T22:25:55.112155+0000","last_became_active":"2021-03-15T22:25:14.114508+0000","last_became_peered":"2021-03-15T22:25:14.114508+0000","last_unstale":"2021-03-15T22:25:55.112155+0000","last_undegraded":"2021-03-15T22:25:55.112155+0000","last_fullsized":"2021-03-15T22:25:55.112155+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.007316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:04.613000+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.007316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"413","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.128565+0000","last_change":"2021-03-15T22:23:52.266674+0000","last_active":"2021-03-15T22:25:53.128565+0000","last_peered":"2021-03-15T22:25:53.128565+0000","last_clean":"2021-03-15T22:25:53.128565+0000","last_became_active":"2021-03-15T22:23:52.266011+0000","last_became_peered":"2021-03-15T22:23:52.266011+0000","last_unstale":"2021-03-15T22:25:53.128565+0000","last_undegraded":"2021-03-15T22:25:53.128565+0000","last_fullsized":"2021-03-15T22:25:53.128565+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.449472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"93","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.144129+0000","last_change":"2021-03-15T22:24:56.824808+0000","last_active":"2021-03-15T22:25:56.144129+0000","last_peered":"2021-03-15T22:25:56.144129+0000","last_clean":"2021-03-15T22:25:56.144129+0000","last_became_active":"2021-03-15T22:24:55.097354+0000","last_became_peered":"2021-03-15T22:24:55.097354+0000","last_unstale":"2021-03-15T22:25:56.144129+0000","last_undegraded":"2021-03-15T22:25:56.144129+0000","last_fullsized":"2021-03-15T22:25:56.144129+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:24:38.433166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:38.433166+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"33","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.111905+0000","last_change":"2021-03-15T22:25:33.139212+0000","last_active":"2021-03-15T22:25:55.111905+0000","last_peered":"2021-03-15T22:25:55.111905+0000","last_clean":"2021-03-15T22:25:55.111905+0000","last_became_active":"2021-03-15T22:25:33.138883+0000","last_became_peered":"2021-03-15T22:25:33.138883+0000","last_unstale":"2021-03-15T22:25:55.111905+0000","last_undegraded":"2021-03-15T22:25:55.111905+0000","last_fullsized":"2021-03-15T22:25:55.111905+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:33.717517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:33.717517+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"424","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.144040+0000","last_change":"2021-03-15T22:25:33.129883+0000","last_active":"2021-03-15T22:25:56.144040+0000","last_peered":"2021-03-15T22:25:56.144040+0000","last_clean":"2021-03-15T22:25:56.144040+0000","last_became_active":"2021-03-15T22:25:33.129553+0000","last_became_peered":"2021-03-15T22:25:33.129553+0000","last_unstale":"2021-03-15T22:25:56.144040+0000","last_undegraded":"2021-03-15T22:25:56.144040+0000","last_fullsized":"2021-03-15T22:25:56.144040+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.794432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"126","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.123631+0000","last_change":"2021-03-15T22:24:12.017160+0000","last_active":"2021-03-15T22:25:53.123631+0000","last_peered":"2021-03-15T22:25:53.123631+0000","last_clean":"2021-03-15T22:25:53.123631+0000","last_became_active":"2021-03-15T22:24:03.086278+0000","last_became_peered":"2021-03-15T22:24:03.086278+0000","last_unstale":"2021-03-15T22:25:53.123631+0000","last_undegraded":"2021-03-15T22:25:53.123631+0000","last_fullsized":"2021-03-15T22:25:53.123631+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:12.017097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:12.017097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"434","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.118461+0000","last_change":"2021-03-15T22:25:43.037390+0000","last_active":"2021-03-15T22:25:55.118461+0000","last_peered":"2021-03-15T22:25:55.118461+0000","last_clean":"2021-03-15T22:25:55.118461+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:25:55.118461+0000","last_undegraded":"2021-03-15T22:25:55.118461+0000","last_fullsized":"2021-03-15T22:25:55.118461+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:29.709674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.709674+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"232","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.128198+0000","last_change":"2021-03-15T22:25:33.144478+0000","last_active":"2021-03-15T22:25:53.128198+0000","last_peered":"2021-03-15T22:25:53.128198+0000","last_clean":"2021-03-15T22:25:53.128198+0000","last_became_active":"2021-03-15T22:25:33.143832+0000","last_became_peered":"2021-03-15T22:25:33.143832+0000","last_unstale":"2021-03-15T22:25:53.128198+0000","last_undegraded":"2021-03-15T22:25:53.128198+0000","last_fullsized":"2021-03-15T22:25:53.128198+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.147414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.147414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"356'3","reported_seq":"509","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:57.088412+0000","last_change":"2021-03-15T22:24:31.129210+0000","last_active":"2021-03-15T22:25:57.088412+0000","last_peered":"2021-03-15T22:25:57.088412+0000","last_clean":"2021-03-15T22:25:57.088412+0000","last_became_active":"2021-03-15T22:24:31.093479+0000","last_became_peered":"2021-03-15T22:24:31.093479+0000","last_unstale":"2021-03-15T22:25:57.088412+0000","last_undegraded":"2021-03-15T22:25:57.088412+0000","last_fullsized":"2021-03-15T22:25:57.088412+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:24:28.700659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.700659+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"400","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.127003+0000","last_change":"2021-03-15T22:25:01.148183+0000","last_active":"2021-03-15T22:25:53.127003+0000","last_peered":"2021-03-15T22:25:53.127003+0000","last_clean":"2021-03-15T22:25:53.127003+0000","last_became_active":"2021-03-15T22:23:52.261234+0000","last_became_peered":"2021-03-15T22:23:52.261234+0000","last_unstale":"2021-03-15T22:25:53.127003+0000","last_undegraded":"2021-03-15T22:25:53.127003+0000","last_fullsized":"2021-03-15T22:25:53.127003+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:01.148105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:01.148105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"217","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.113903+0000","last_change":"2021-03-15T22:24:27.166757+0000","last_active":"2021-03-15T22:25:55.113903+0000","last_peered":"2021-03-15T22:25:55.113903+0000","last_clean":"2021-03-15T22:25:55.113903+0000","last_became_active":"2021-03-15T22:22:56.064081+0000","last_became_peered":"2021-03-15T22:22:56.064081+0000","last_unstale":"2021-03-15T22:25:55.113903+0000","last_undegraded":"2021-03-15T22:25:55.113903+0000","last_fullsized":"2021-03-15T22:25:55.113903+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:27.166678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.166678+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":13,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"401","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.119734+0000","last_change":"2021-03-15T22:25:07.992897+0000","last_active":"2021-03-15T22:25:55.119734+0000","last_peered":"2021-03-15T22:25:55.119734+0000","last_clean":"2021-03-15T22:25:55.119734+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:25:55.119734+0000","last_undegraded":"2021-03-15T22:25:55.119734+0000","last_fullsized":"2021-03-15T22:25:55.119734+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:07.992815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.992815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"421'7","reported_seq":"41","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.113780+0000","last_change":"2021-03-15T22:25:33.172230+0000","last_active":"2021-03-15T22:25:55.113780+0000","last_peered":"2021-03-15T22:25:55.113780+0000","last_clean":"2021-03-15T22:25:55.113780+0000","last_became_active":"2021-03-15T22:25:33.147484+0000","last_became_peered":"2021-03-15T22:25:33.147484+0000","last_unstale":"2021-03-15T22:25:55.113780+0000","last_undegraded":"2021-03-15T22:25:55.113780+0000","last_fullsized":"2021-03-15T22:25:55.113780+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'6","last_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.098511+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"238","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.126639+0000","last_change":"2021-03-15T22:25:33.141774+0000","last_active":"2021-03-15T22:25:53.126639+0000","last_peered":"2021-03-15T22:25:53.126639+0000","last_clean":"2021-03-15T22:25:53.126639+0000","last_became_active":"2021-03-15T22:25:33.141466+0000","last_became_peered":"2021-03-15T22:25:33.141466+0000","last_unstale":"2021-03-15T22:25:53.126639+0000","last_undegraded":"2021-03-15T22:25:53.126639+0000","last_fullsized":"2021-03-15T22:25:53.126639+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:23:39.695671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"71","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.189195+0000","last_change":"2021-03-15T22:25:14.129885+0000","last_active":"2021-03-15T22:25:56.189195+0000","last_peered":"2021-03-15T22:25:56.189195+0000","last_clean":"2021-03-15T22:25:56.189195+0000","last_became_active":"2021-03-15T22:25:14.114060+0000","last_became_peered":"2021-03-15T22:25:14.114060+0000","last_unstale":"2021-03-15T22:25:56.189195+0000","last_undegraded":"2021-03-15T22:25:56.189195+0000","last_fullsized":"2021-03-15T22:25:56.189195+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:25:07.163128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.163128+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"404","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.123923+0000","last_change":"2021-03-15T22:25:00.614351+0000","last_active":"2021-03-15T22:25:53.123923+0000","last_peered":"2021-03-15T22:25:53.123923+0000","last_clean":"2021-03-15T22:25:53.123923+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:25:53.123923+0000","last_undegraded":"2021-03-15T22:25:53.123923+0000","last_fullsized":"2021-03-15T22:25:53.123923+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.614288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.614288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"402","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.126470+0000","last_change":"2021-03-15T22:25:10.130486+0000","last_active":"2021-03-15T22:25:53.126470+0000","last_peered":"2021-03-15T22:25:53.126470+0000","last_clean":"2021-03-15T22:25:53.126470+0000","last_became_active":"2021-03-15T22:23:52.263426+0000","last_became_peered":"2021-03-15T22:23:52.263426+0000","last_unstale":"2021-03-15T22:25:53.126470+0000","last_undegraded":"2021-03-15T22:25:53.126470+0000","last_fullsized":"2021-03-15T22:25:53.126470+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:10.130422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.130422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"57","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.112114+0000","last_change":"2021-03-15T22:25:35.064301+0000","last_active":"2021-03-15T22:25:55.112114+0000","last_peered":"2021-03-15T22:25:55.112114+0000","last_clean":"2021-03-15T22:25:55.112114+0000","last_became_active":"2021-03-15T22:25:33.151159+0000","last_became_peered":"2021-03-15T22:25:33.151159+0000","last_unstale":"2021-03-15T22:25:55.112114+0000","last_undegraded":"2021-03-15T22:25:55.112114+0000","last_fullsized":"2021-03-15T22:25:55.112114+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:26.478799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.478799+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3618,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"394","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.128522+0000","last_change":"2021-03-15T22:23:52.266876+0000","last_active":"2021-03-15T22:25:53.128522+0000","last_peered":"2021-03-15T22:25:53.128522+0000","last_clean":"2021-03-15T22:25:53.128522+0000","last_became_active":"2021-03-15T22:23:52.266133+0000","last_became_peered":"2021-03-15T22:23:52.266133+0000","last_unstale":"2021-03-15T22:25:53.128522+0000","last_undegraded":"2021-03-15T22:25:53.128522+0000","last_fullsized":"2021-03-15T22:25:53.128522+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:28.808873+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:28.808873+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"450","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.135094+0000","last_change":"2021-03-15T22:25:33.150477+0000","last_active":"2021-03-15T22:25:54.135094+0000","last_peered":"2021-03-15T22:25:54.135094+0000","last_clean":"2021-03-15T22:25:54.135094+0000","last_became_active":"2021-03-15T22:25:33.150147+0000","last_became_peered":"2021-03-15T22:25:33.150147+0000","last_unstale":"2021-03-15T22:25:54.135094+0000","last_undegraded":"2021-03-15T22:25:54.135094+0000","last_fullsized":"2021-03-15T22:25:54.135094+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:37.475218+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.475218+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"398'11","reported_seq":"43","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.113743+0000","last_change":"2021-03-15T22:25:33.864526+0000","last_active":"2021-03-15T22:25:55.113743+0000","last_peered":"2021-03-15T22:25:55.113743+0000","last_clean":"2021-03-15T22:25:55.113743+0000","last_became_active":"2021-03-15T22:25:33.152373+0000","last_became_peered":"2021-03-15T22:25:33.152373+0000","last_unstale":"2021-03-15T22:25:55.113743+0000","last_undegraded":"2021-03-15T22:25:55.113743+0000","last_fullsized":"2021-03-15T22:25:55.113743+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'11","last_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:25:13.181225+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"442'380","reported_seq":"1636","reported_epoch":"442","state":"active+clean","last_fresh":"2021-03-15T22:25:57.173463+0000","last_change":"2021-03-15T22:25:57.173710+0000","last_active":"2021-03-15T22:25:57.173463+0000","last_peered":"2021-03-15T22:25:57.173463+0000","last_clean":"2021-03-15T22:25:57.173463+0000","last_became_active":"2021-03-15T22:25:17.113174+0000","last_became_peered":"2021-03-15T22:25:17.113174+0000","last_unstale":"2021-03-15T22:25:57.173463+0000","last_undegraded":"2021-03-15T22:25:57.173463+0000","last_fullsized":"2021-03-15T22:25:57.173463+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'282","last_scrub_stamp":"2021-03-15T22:25:09.803362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.803362+0000","log_size":380,"ondisk_log_size":380,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29009479,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":275,"num_read_kb":314805,"num_write":158,"num_write_kb":50502,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":21669077,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10d","length":"1"},{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"410","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.188849+0000","last_change":"2021-03-15T22:25:00.689659+0000","last_active":"2021-03-15T22:25:56.188849+0000","last_peered":"2021-03-15T22:25:56.188849+0000","last_clean":"2021-03-15T22:25:56.188849+0000","last_became_active":"2021-03-15T22:24:31.103470+0000","last_became_peered":"2021-03-15T22:24:31.103470+0000","last_unstale":"2021-03-15T22:25:56.188849+0000","last_undegraded":"2021-03-15T22:25:56.188849+0000","last_fullsized":"2021-03-15T22:25:56.188849+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.689591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.689591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"423","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.127442+0000","last_change":"2021-03-15T22:25:48.008983+0000","last_active":"2021-03-15T22:25:53.127442+0000","last_peered":"2021-03-15T22:25:53.127442+0000","last_clean":"2021-03-15T22:25:53.127442+0000","last_became_active":"2021-03-15T22:24:31.097688+0000","last_became_peered":"2021-03-15T22:24:31.097688+0000","last_unstale":"2021-03-15T22:25:53.127442+0000","last_undegraded":"2021-03-15T22:25:53.127442+0000","last_fullsized":"2021-03-15T22:25:53.127442+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:25:48.008911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:48.008911+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"129","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.188637+0000","last_change":"2021-03-15T22:25:32.799965+0000","last_active":"2021-03-15T22:25:56.188637+0000","last_peered":"2021-03-15T22:25:56.188637+0000","last_clean":"2021-03-15T22:25:56.188637+0000","last_became_active":"2021-03-15T22:24:23.137531+0000","last_became_peered":"2021-03-15T22:24:23.137531+0000","last_unstale":"2021-03-15T22:25:56.188637+0000","last_undegraded":"2021-03-15T22:25:56.188637+0000","last_fullsized":"2021-03-15T22:25:56.188637+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:25:32.799882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.799882+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"317'7","reported_seq":"511","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.126577+0000","last_change":"2021-03-15T22:24:30.068613+0000","last_active":"2021-03-15T22:25:53.126577+0000","last_peered":"2021-03-15T22:25:53.126577+0000","last_clean":"2021-03-15T22:25:53.126577+0000","last_became_active":"2021-03-15T22:23:52.262878+0000","last_became_peered":"2021-03-15T22:23:52.262878+0000","last_unstale":"2021-03-15T22:25:53.126577+0000","last_undegraded":"2021-03-15T22:25:53.126577+0000","last_fullsized":"2021-03-15T22:25:53.126577+0000","mapping_epoch":316,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:24:30.068550+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.068550+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"440'591","reported_seq":"1749","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.128019+0000","last_change":"2021-03-15T22:25:55.128019+0000","last_active":"2021-03-15T22:25:55.128019+0000","last_peered":"2021-03-15T22:25:55.128019+0000","last_clean":"2021-03-15T22:25:55.128019+0000","last_became_active":"2021-03-15T22:23:04.067641+0000","last_became_peered":"2021-03-15T22:23:04.067641+0000","last_unstale":"2021-03-15T22:25:55.128019+0000","last_undegraded":"2021-03-15T22:25:55.128019+0000","last_fullsized":"2021-03-15T22:25:55.128019+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'455","last_scrub_stamp":"2021-03-15T22:24:28.184612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.184612+0000","log_size":591,"ondisk_log_size":591,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32460507,"num_objects":16,"num_object_clones":11,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":328,"num_read_kb":337382,"num_write":217,"num_write_kb":79922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":32120558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"200","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.113758+0000","last_change":"2021-03-15T22:24:37.122501+0000","last_active":"2021-03-15T22:25:55.113758+0000","last_peered":"2021-03-15T22:25:55.113758+0000","last_clean":"2021-03-15T22:25:55.113758+0000","last_became_active":"2021-03-15T22:22:56.057174+0000","last_became_peered":"2021-03-15T22:22:56.057174+0000","last_unstale":"2021-03-15T22:25:55.113758+0000","last_undegraded":"2021-03-15T22:25:55.113758+0000","last_fullsized":"2021-03-15T22:25:55.113758+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:37.122407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.122407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"402","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.128268+0000","last_change":"2021-03-15T22:25:39.991710+0000","last_active":"2021-03-15T22:25:53.128268+0000","last_peered":"2021-03-15T22:25:53.128268+0000","last_clean":"2021-03-15T22:25:53.128268+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:25:53.128268+0000","last_undegraded":"2021-03-15T22:25:53.128268+0000","last_fullsized":"2021-03-15T22:25:53.128268+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:22:38.810397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:22:38.810397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"488","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.120031+0000","last_change":"2021-03-15T22:25:33.862633+0000","last_active":"2021-03-15T22:25:55.120031+0000","last_peered":"2021-03-15T22:25:55.120031+0000","last_clean":"2021-03-15T22:25:55.120031+0000","last_became_active":"2021-03-15T22:25:33.138262+0000","last_became_peered":"2021-03-15T22:25:33.138262+0000","last_unstale":"2021-03-15T22:25:55.120031+0000","last_undegraded":"2021-03-15T22:25:55.120031+0000","last_fullsized":"2021-03-15T22:25:55.120031+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:07.027095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.027095+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"433","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.123672+0000","last_change":"2021-03-15T22:24:39.853332+0000","last_active":"2021-03-15T22:25:53.123672+0000","last_peered":"2021-03-15T22:25:53.123672+0000","last_clean":"2021-03-15T22:25:53.123672+0000","last_became_active":"2021-03-15T22:24:31.109058+0000","last_became_peered":"2021-03-15T22:24:31.109058+0000","last_unstale":"2021-03-15T22:25:53.123672+0000","last_undegraded":"2021-03-15T22:25:53.123672+0000","last_fullsized":"2021-03-15T22:25:53.123672+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.853267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.853267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"442'150","reported_seq":"1250","reported_epoch":"442","state":"active+clean","last_fresh":"2021-03-15T22:25:57.172992+0000","last_change":"2021-03-15T22:25:57.173210+0000","last_active":"2021-03-15T22:25:57.172992+0000","last_peered":"2021-03-15T22:25:57.172992+0000","last_clean":"2021-03-15T22:25:57.172992+0000","last_became_active":"2021-03-15T22:24:34.087074+0000","last_became_peered":"2021-03-15T22:24:34.087074+0000","last_unstale":"2021-03-15T22:25:57.172992+0000","last_undegraded":"2021-03-15T22:25:57.172992+0000","last_fullsized":"2021-03-15T22:25:57.172992+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'130","last_scrub_stamp":"2021-03-15T22:25:06.854931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.854931+0000","log_size":150,"ondisk_log_size":150,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5697738,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":30033,"num_write":50,"num_write_kb":17214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4470794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"10d","length":"1"},{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"412","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.126864+0000","last_change":"2021-03-15T22:23:52.267464+0000","last_active":"2021-03-15T22:25:53.126864+0000","last_peered":"2021-03-15T22:25:53.126864+0000","last_clean":"2021-03-15T22:25:53.126864+0000","last_became_active":"2021-03-15T22:23:52.267164+0000","last_became_peered":"2021-03-15T22:23:52.267164+0000","last_unstale":"2021-03-15T22:25:53.126864+0000","last_undegraded":"2021-03-15T22:25:53.126864+0000","last_fullsized":"2021-03-15T22:25:53.126864+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:29.845442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.845442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"451","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.124751+0000","last_change":"2021-03-15T22:25:14.114184+0000","last_active":"2021-03-15T22:25:53.124751+0000","last_peered":"2021-03-15T22:25:53.124751+0000","last_clean":"2021-03-15T22:25:53.124751+0000","last_became_active":"2021-03-15T22:25:14.113857+0000","last_became_peered":"2021-03-15T22:25:14.113857+0000","last_unstale":"2021-03-15T22:25:53.124751+0000","last_undegraded":"2021-03-15T22:25:53.124751+0000","last_fullsized":"2021-03-15T22:25:53.124751+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.575552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.575552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"420","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.128758+0000","last_change":"2021-03-15T22:25:14.118841+0000","last_active":"2021-03-15T22:25:53.128758+0000","last_peered":"2021-03-15T22:25:53.128758+0000","last_clean":"2021-03-15T22:25:53.128758+0000","last_became_active":"2021-03-15T22:25:14.118469+0000","last_became_peered":"2021-03-15T22:25:14.118469+0000","last_unstale":"2021-03-15T22:25:53.128758+0000","last_undegraded":"2021-03-15T22:25:53.128758+0000","last_fullsized":"2021-03-15T22:25:53.128758+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.024799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.024799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"481","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.136063+0000","last_change":"2021-03-15T22:24:40.468134+0000","last_active":"2021-03-15T22:25:54.136063+0000","last_peered":"2021-03-15T22:25:54.136063+0000","last_clean":"2021-03-15T22:25:54.136063+0000","last_became_active":"2021-03-15T22:24:03.085862+0000","last_became_peered":"2021-03-15T22:24:03.085862+0000","last_unstale":"2021-03-15T22:25:54.136063+0000","last_undegraded":"2021-03-15T22:25:54.136063+0000","last_fullsized":"2021-03-15T22:25:54.136063+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:24:40.468066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.468066+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"439'365","reported_seq":"1447","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.141062+0000","last_change":"2021-03-15T22:25:54.141290+0000","last_active":"2021-03-15T22:25:54.141062+0000","last_peered":"2021-03-15T22:25:54.141062+0000","last_clean":"2021-03-15T22:25:54.141062+0000","last_became_active":"2021-03-15T22:20:05.032282+0000","last_became_peered":"2021-03-15T22:20:05.032282+0000","last_unstale":"2021-03-15T22:25:54.141062+0000","last_undegraded":"2021-03-15T22:25:54.141062+0000","last_fullsized":"2021-03-15T22:25:54.141062+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'328","last_scrub_stamp":"2021-03-15T22:25:37.707812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.707812+0000","log_size":365,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22791563,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":198,"num_read_kb":234048,"num_write":118,"num_write_kb":38785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10e","length":"1"},{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"13a","length":"2"},{"start":"145","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"100","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.126427+0000","last_change":"2021-03-15T22:25:11.117468+0000","last_active":"2021-03-15T22:25:53.126427+0000","last_peered":"2021-03-15T22:25:53.126427+0000","last_clean":"2021-03-15T22:25:53.126427+0000","last_became_active":"2021-03-15T22:24:31.093577+0000","last_became_peered":"2021-03-15T22:24:31.093577+0000","last_unstale":"2021-03-15T22:25:53.126427+0000","last_undegraded":"2021-03-15T22:25:53.126427+0000","last_fullsized":"2021-03-15T22:25:53.126427+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.117403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.117403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"451","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.125173+0000","last_change":"2021-03-15T22:25:33.134762+0000","last_active":"2021-03-15T22:25:53.125173+0000","last_peered":"2021-03-15T22:25:53.125173+0000","last_clean":"2021-03-15T22:25:53.125173+0000","last_became_active":"2021-03-15T22:25:33.134358+0000","last_became_peered":"2021-03-15T22:25:33.134358+0000","last_unstale":"2021-03-15T22:25:53.125173+0000","last_undegraded":"2021-03-15T22:25:53.125173+0000","last_fullsized":"2021-03-15T22:25:53.125173+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:27.869570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.869570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"425","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.188770+0000","last_change":"2021-03-15T22:25:47.753887+0000","last_active":"2021-03-15T22:25:56.188770+0000","last_peered":"2021-03-15T22:25:56.188770+0000","last_clean":"2021-03-15T22:25:56.188770+0000","last_became_active":"2021-03-15T22:25:33.145548+0000","last_became_peered":"2021-03-15T22:25:33.145548+0000","last_unstale":"2021-03-15T22:25:56.188770+0000","last_undegraded":"2021-03-15T22:25:56.188770+0000","last_fullsized":"2021-03-15T22:25:56.188770+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:47.753814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:47.753814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"201","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.124971+0000","last_change":"2021-03-15T22:25:33.912570+0000","last_active":"2021-03-15T22:25:53.124971+0000","last_peered":"2021-03-15T22:25:53.124971+0000","last_clean":"2021-03-15T22:25:53.124971+0000","last_became_active":"2021-03-15T22:25:33.135102+0000","last_became_peered":"2021-03-15T22:25:33.135102+0000","last_unstale":"2021-03-15T22:25:53.124971+0000","last_undegraded":"2021-03-15T22:25:53.124971+0000","last_fullsized":"2021-03-15T22:25:53.124971+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:01.499099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.499099+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"440'402","reported_seq":"1646","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.133055+0000","last_change":"2021-03-15T22:25:55.127162+0000","last_active":"2021-03-15T22:25:55.133055+0000","last_peered":"2021-03-15T22:25:55.133055+0000","last_clean":"2021-03-15T22:25:55.133055+0000","last_became_active":"2021-03-15T22:25:36.118229+0000","last_became_peered":"2021-03-15T22:25:36.118229+0000","last_unstale":"2021-03-15T22:25:55.133055+0000","last_undegraded":"2021-03-15T22:25:55.133055+0000","last_fullsized":"2021-03-15T22:25:55.133055+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'340","last_scrub_stamp":"2021-03-15T22:25:12.353191+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.353191+0000","log_size":402,"ondisk_log_size":402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31785193,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":145,"num_read_kb":101626,"num_write":176,"num_write_kb":55395,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":35579701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"33","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.112203+0000","last_change":"2021-03-15T22:25:33.152550+0000","last_active":"2021-03-15T22:25:55.112203+0000","last_peered":"2021-03-15T22:25:55.112203+0000","last_clean":"2021-03-15T22:25:55.112203+0000","last_became_active":"2021-03-15T22:25:33.152232+0000","last_became_peered":"2021-03-15T22:25:33.152232+0000","last_unstale":"2021-03-15T22:25:55.112203+0000","last_undegraded":"2021-03-15T22:25:55.112203+0000","last_fullsized":"2021-03-15T22:25:55.112203+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:34.763323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:34.763323+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"409","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.188591+0000","last_change":"2021-03-15T22:25:11.615097+0000","last_active":"2021-03-15T22:25:56.188591+0000","last_peered":"2021-03-15T22:25:56.188591+0000","last_clean":"2021-03-15T22:25:56.188591+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:25:56.188591+0000","last_undegraded":"2021-03-15T22:25:56.188591+0000","last_fullsized":"2021-03-15T22:25:56.188591+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.614975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.614975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"135","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.124629+0000","last_change":"2021-03-15T22:25:45.692608+0000","last_active":"2021-03-15T22:25:53.124629+0000","last_peered":"2021-03-15T22:25:53.124629+0000","last_clean":"2021-03-15T22:25:53.124629+0000","last_became_active":"2021-03-15T22:24:31.106344+0000","last_became_peered":"2021-03-15T22:24:31.106344+0000","last_unstale":"2021-03-15T22:25:53.124629+0000","last_undegraded":"2021-03-15T22:25:53.124629+0000","last_fullsized":"2021-03-15T22:25:53.124629+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:45.692545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:45.692545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"547","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.145019+0000","last_change":"2021-03-15T22:25:33.174619+0000","last_active":"2021-03-15T22:25:56.145019+0000","last_peered":"2021-03-15T22:25:56.145019+0000","last_clean":"2021-03-15T22:25:56.145019+0000","last_became_active":"2021-03-15T22:25:33.126039+0000","last_became_peered":"2021-03-15T22:25:33.126039+0000","last_unstale":"2021-03-15T22:25:56.145019+0000","last_undegraded":"2021-03-15T22:25:56.145019+0000","last_fullsized":"2021-03-15T22:25:56.145019+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:25:15.810377+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.810377+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"102","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.144517+0000","last_change":"2021-03-15T22:25:44.619146+0000","last_active":"2021-03-15T22:25:56.144517+0000","last_peered":"2021-03-15T22:25:56.144517+0000","last_clean":"2021-03-15T22:25:56.144517+0000","last_became_active":"2021-03-15T22:24:31.101668+0000","last_became_peered":"2021-03-15T22:24:31.101668+0000","last_unstale":"2021-03-15T22:25:56.144517+0000","last_undegraded":"2021-03-15T22:25:56.144517+0000","last_fullsized":"2021-03-15T22:25:56.144517+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:44.619082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:44.619082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"440'392","reported_seq":"1509","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.124305+0000","last_change":"2021-03-15T22:25:55.124305+0000","last_active":"2021-03-15T22:25:55.124305+0000","last_peered":"2021-03-15T22:25:55.124305+0000","last_clean":"2021-03-15T22:25:55.124305+0000","last_became_active":"2021-03-15T22:24:34.087484+0000","last_became_peered":"2021-03-15T22:24:34.087484+0000","last_unstale":"2021-03-15T22:25:55.124305+0000","last_undegraded":"2021-03-15T22:25:55.124305+0000","last_fullsized":"2021-03-15T22:25:55.124305+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'329","last_scrub_stamp":"2021-03-15T22:25:14.014776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:14.014776+0000","log_size":392,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20972815,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":129,"num_read_kb":112613,"num_write":134,"num_write_kb":46382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":10768640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"412","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.128920+0000","last_change":"2021-03-15T22:23:52.267559+0000","last_active":"2021-03-15T22:25:53.128920+0000","last_peered":"2021-03-15T22:25:53.128920+0000","last_clean":"2021-03-15T22:25:53.128920+0000","last_became_active":"2021-03-15T22:23:52.266556+0000","last_became_peered":"2021-03-15T22:23:52.266556+0000","last_unstale":"2021-03-15T22:25:53.128920+0000","last_undegraded":"2021-03-15T22:25:53.128920+0000","last_fullsized":"2021-03-15T22:25:53.128920+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:30.831598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:30.831598+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"410","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.143893+0000","last_change":"2021-03-15T22:24:39.929128+0000","last_active":"2021-03-15T22:25:56.143893+0000","last_peered":"2021-03-15T22:25:56.143893+0000","last_clean":"2021-03-15T22:25:56.143893+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:25:56.143893+0000","last_undegraded":"2021-03-15T22:25:56.143893+0000","last_fullsized":"2021-03-15T22:25:56.143893+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.929066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.929066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"439","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.124108+0000","last_change":"2021-03-15T22:25:42.660209+0000","last_active":"2021-03-15T22:25:53.124108+0000","last_peered":"2021-03-15T22:25:53.124108+0000","last_clean":"2021-03-15T22:25:53.124108+0000","last_became_active":"2021-03-15T22:25:33.129875+0000","last_became_peered":"2021-03-15T22:25:33.129875+0000","last_unstale":"2021-03-15T22:25:53.124108+0000","last_undegraded":"2021-03-15T22:25:53.124108+0000","last_fullsized":"2021-03-15T22:25:53.124108+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:29.266174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.266174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"149'22","reported_seq":"92","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.145184+0000","last_change":"2021-03-15T22:25:03.853145+0000","last_active":"2021-03-15T22:25:56.145184+0000","last_peered":"2021-03-15T22:25:56.145184+0000","last_clean":"2021-03-15T22:25:56.145184+0000","last_became_active":"2021-03-15T22:24:55.093124+0000","last_became_peered":"2021-03-15T22:24:55.093124+0000","last_unstale":"2021-03-15T22:25:56.145184+0000","last_undegraded":"2021-03-15T22:25:56.145184+0000","last_fullsized":"2021-03-15T22:25:56.145184+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'22","last_scrub_stamp":"2021-03-15T22:25:03.853079+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:25:03.853079+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"424","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.125191+0000","last_change":"2021-03-15T22:25:02.520674+0000","last_active":"2021-03-15T22:25:53.125191+0000","last_peered":"2021-03-15T22:25:53.125191+0000","last_clean":"2021-03-15T22:25:53.125191+0000","last_became_active":"2021-03-15T22:21:38.147364+0000","last_became_peered":"2021-03-15T22:21:38.147364+0000","last_unstale":"2021-03-15T22:25:53.125191+0000","last_undegraded":"2021-03-15T22:25:53.125191+0000","last_fullsized":"2021-03-15T22:25:53.125191+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.520588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.520588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"439'972","reported_seq":"2785","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.148771+0000","last_change":"2021-03-15T22:25:54.149035+0000","last_active":"2021-03-15T22:25:54.148771+0000","last_peered":"2021-03-15T22:25:54.148771+0000","last_clean":"2021-03-15T22:25:54.148771+0000","last_became_active":"2021-03-15T22:25:37.118396+0000","last_became_peered":"2021-03-15T22:25:37.118396+0000","last_unstale":"2021-03-15T22:25:54.148771+0000","last_undegraded":"2021-03-15T22:25:54.148771+0000","last_fullsized":"2021-03-15T22:25:54.148771+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"245'350","last_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:22:40.995789+0000","log_size":972,"ondisk_log_size":972,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":64163804,"num_objects":27,"num_object_clones":20,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":2,"num_read":364,"num_read_kb":322559,"num_write":354,"num_write_kb":155924,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":140467323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10e","length":"1"},{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"13a","length":"2"},{"start":"145","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"414","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.189263+0000","last_change":"2021-03-15T22:23:38.968009+0000","last_active":"2021-03-15T22:25:56.189263+0000","last_peered":"2021-03-15T22:25:56.189263+0000","last_clean":"2021-03-15T22:25:56.189263+0000","last_became_active":"2021-03-15T22:22:56.062745+0000","last_became_peered":"2021-03-15T22:22:56.062745+0000","last_unstale":"2021-03-15T22:25:56.189263+0000","last_undegraded":"2021-03-15T22:25:56.189263+0000","last_fullsized":"2021-03-15T22:25:56.189263+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:38.967934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:38.967934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"414","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.127050+0000","last_change":"2021-03-15T22:24:31.111721+0000","last_active":"2021-03-15T22:25:53.127050+0000","last_peered":"2021-03-15T22:25:53.127050+0000","last_clean":"2021-03-15T22:25:53.127050+0000","last_became_active":"2021-03-15T22:24:31.111104+0000","last_became_peered":"2021-03-15T22:24:31.111104+0000","last_unstale":"2021-03-15T22:25:53.127050+0000","last_undegraded":"2021-03-15T22:25:53.127050+0000","last_fullsized":"2021-03-15T22:25:53.127050+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.872030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.872030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"38","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.114729+0000","last_change":"2021-03-15T22:25:46.813664+0000","last_active":"2021-03-15T22:25:55.114729+0000","last_peered":"2021-03-15T22:25:55.114729+0000","last_clean":"2021-03-15T22:25:55.114729+0000","last_became_active":"2021-03-15T22:25:33.147285+0000","last_became_peered":"2021-03-15T22:25:33.147285+0000","last_unstale":"2021-03-15T22:25:55.114729+0000","last_undegraded":"2021-03-15T22:25:55.114729+0000","last_fullsized":"2021-03-15T22:25:55.114729+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:46.813605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.813605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"434","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.135417+0000","last_change":"2021-03-15T22:25:37.458353+0000","last_active":"2021-03-15T22:25:54.135417+0000","last_peered":"2021-03-15T22:25:54.135417+0000","last_clean":"2021-03-15T22:25:54.135417+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:25:54.135417+0000","last_undegraded":"2021-03-15T22:25:54.135417+0000","last_fullsized":"2021-03-15T22:25:54.135417+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:37.458291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.458291+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"398","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.127896+0000","last_change":"2021-03-15T22:23:52.271864+0000","last_active":"2021-03-15T22:25:53.127896+0000","last_peered":"2021-03-15T22:25:53.127896+0000","last_clean":"2021-03-15T22:25:53.127896+0000","last_became_active":"2021-03-15T22:23:52.271442+0000","last_became_peered":"2021-03-15T22:23:52.271442+0000","last_unstale":"2021-03-15T22:25:53.127896+0000","last_undegraded":"2021-03-15T22:25:53.127896+0000","last_fullsized":"2021-03-15T22:25:53.127896+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:26.748485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.748485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"439'271","reported_seq":"1403","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.161045+0000","last_change":"2021-03-15T22:25:54.142414+0000","last_active":"2021-03-15T22:25:54.161045+0000","last_peered":"2021-03-15T22:25:54.161045+0000","last_clean":"2021-03-15T22:25:54.161045+0000","last_became_active":"2021-03-15T22:24:36.092680+0000","last_became_peered":"2021-03-15T22:24:36.092680+0000","last_unstale":"2021-03-15T22:25:54.161045+0000","last_undegraded":"2021-03-15T22:25:54.161045+0000","last_fullsized":"2021-03-15T22:25:54.161045+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'138","last_scrub_stamp":"2021-03-15T22:23:41.141218+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.141218+0000","log_size":271,"ondisk_log_size":271,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26657226,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":111,"num_read_kb":104246,"num_write":103,"num_write_kb":41981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8487583,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"10e","length":"1"},{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"13a","length":"2"},{"start":"145","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"422","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.144763+0000","last_change":"2021-03-15T22:25:33.129904+0000","last_active":"2021-03-15T22:25:56.144763+0000","last_peered":"2021-03-15T22:25:56.144763+0000","last_clean":"2021-03-15T22:25:56.144763+0000","last_became_active":"2021-03-15T22:25:33.129602+0000","last_became_peered":"2021-03-15T22:25:33.129602+0000","last_unstale":"2021-03-15T22:25:56.144763+0000","last_undegraded":"2021-03-15T22:25:56.144763+0000","last_fullsized":"2021-03-15T22:25:56.144763+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.931058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.931058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"239","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.119808+0000","last_change":"2021-03-15T22:25:33.143239+0000","last_active":"2021-03-15T22:25:55.119808+0000","last_peered":"2021-03-15T22:25:55.119808+0000","last_clean":"2021-03-15T22:25:55.119808+0000","last_became_active":"2021-03-15T22:25:33.142133+0000","last_became_peered":"2021-03-15T22:25:33.142133+0000","last_unstale":"2021-03-15T22:25:55.119808+0000","last_undegraded":"2021-03-15T22:25:55.119808+0000","last_fullsized":"2021-03-15T22:25:55.119808+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.028082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.028082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"886","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.136780+0000","last_change":"2021-03-15T22:24:31.248181+0000","last_active":"2021-03-15T22:25:54.136780+0000","last_peered":"2021-03-15T22:25:54.136780+0000","last_clean":"2021-03-15T22:25:54.136780+0000","last_became_active":"2021-03-15T22:24:31.109631+0000","last_became_peered":"2021-03-15T22:24:31.109631+0000","last_unstale":"2021-03-15T22:25:54.136780+0000","last_undegraded":"2021-03-15T22:25:54.136780+0000","last_fullsized":"2021-03-15T22:25:54.136780+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:23:27.790040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:23:27.790040+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":466,"num_read_kb":461,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"163","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.145160+0000","last_change":"2021-03-15T22:24:37.977457+0000","last_active":"2021-03-15T22:25:56.145160+0000","last_peered":"2021-03-15T22:25:56.145160+0000","last_clean":"2021-03-15T22:25:56.145160+0000","last_became_active":"2021-03-15T22:24:31.100536+0000","last_became_peered":"2021-03-15T22:24:31.100536+0000","last_unstale":"2021-03-15T22:25:56.145160+0000","last_undegraded":"2021-03-15T22:25:56.145160+0000","last_fullsized":"2021-03-15T22:25:56.145160+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:37.977363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.977363+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"53","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.189460+0000","last_change":"2021-03-15T22:25:14.115071+0000","last_active":"2021-03-15T22:25:56.189460+0000","last_peered":"2021-03-15T22:25:56.189460+0000","last_clean":"2021-03-15T22:25:56.189460+0000","last_became_active":"2021-03-15T22:25:14.114729+0000","last_became_peered":"2021-03-15T22:25:14.114729+0000","last_unstale":"2021-03-15T22:25:56.189460+0000","last_undegraded":"2021-03-15T22:25:56.189460+0000","last_fullsized":"2021-03-15T22:25:56.189460+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.857011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.857011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"442'1074","reported_seq":"2804","reported_epoch":"442","state":"active+clean","last_fresh":"2021-03-15T22:25:57.349289+0000","last_change":"2021-03-15T22:25:57.184274+0000","last_active":"2021-03-15T22:25:57.349289+0000","last_peered":"2021-03-15T22:25:57.349289+0000","last_clean":"2021-03-15T22:25:57.349289+0000","last_became_active":"2021-03-15T22:25:45.240370+0000","last_became_peered":"2021-03-15T22:25:45.240370+0000","last_unstale":"2021-03-15T22:25:57.349289+0000","last_undegraded":"2021-03-15T22:25:57.349289+0000","last_fullsized":"2021-03-15T22:25:57.349289+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'777","last_scrub_stamp":"2021-03-15T22:24:40.640400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.640400+0000","log_size":1074,"ondisk_log_size":1074,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67920655,"num_objects":34,"num_object_clones":27,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":2,"num_read":625,"num_read_kb":698129,"num_write":364,"num_write_kb":145036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":125184016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"10d","length":"1"},{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"404","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.143842+0000","last_change":"2021-03-15T22:24:30.950198+0000","last_active":"2021-03-15T22:25:56.143842+0000","last_peered":"2021-03-15T22:25:56.143842+0000","last_clean":"2021-03-15T22:25:56.143842+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:25:56.143842+0000","last_undegraded":"2021-03-15T22:25:56.143842+0000","last_fullsized":"2021-03-15T22:25:56.143842+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.950135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.950135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"410","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.145210+0000","last_change":"2021-03-15T22:24:26.962034+0000","last_active":"2021-03-15T22:25:56.145210+0000","last_peered":"2021-03-15T22:25:56.145210+0000","last_clean":"2021-03-15T22:25:56.145210+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:25:56.145210+0000","last_undegraded":"2021-03-15T22:25:56.145210+0000","last_fullsized":"2021-03-15T22:25:56.145210+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.961978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.961978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"186","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.188667+0000","last_change":"2021-03-15T22:24:31.147838+0000","last_active":"2021-03-15T22:25:56.188667+0000","last_peered":"2021-03-15T22:25:56.188667+0000","last_clean":"2021-03-15T22:25:56.188667+0000","last_became_active":"2021-03-15T22:24:31.098665+0000","last_became_peered":"2021-03-15T22:24:31.098665+0000","last_unstale":"2021-03-15T22:25:56.188667+0000","last_undegraded":"2021-03-15T22:25:56.188667+0000","last_fullsized":"2021-03-15T22:25:56.188667+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:23:36.678184+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:23:36.678184+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"414","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.136141+0000","last_change":"2021-03-15T22:24:34.454377+0000","last_active":"2021-03-15T22:25:54.136141+0000","last_peered":"2021-03-15T22:25:54.136141+0000","last_clean":"2021-03-15T22:25:54.136141+0000","last_became_active":"2021-03-15T22:24:31.107629+0000","last_became_peered":"2021-03-15T22:24:31.107629+0000","last_unstale":"2021-03-15T22:25:54.136141+0000","last_undegraded":"2021-03-15T22:25:54.136141+0000","last_fullsized":"2021-03-15T22:25:54.136141+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.454313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.454313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"439'162","reported_seq":"1303","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.122087+0000","last_change":"2021-03-15T22:25:55.122087+0000","last_active":"2021-03-15T22:25:55.122087+0000","last_peered":"2021-03-15T22:25:55.122087+0000","last_clean":"2021-03-15T22:25:55.122087+0000","last_became_active":"2021-03-15T22:25:37.112715+0000","last_became_peered":"2021-03-15T22:25:37.112715+0000","last_unstale":"2021-03-15T22:25:55.122087+0000","last_undegraded":"2021-03-15T22:25:55.122087+0000","last_fullsized":"2021-03-15T22:25:55.122087+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'104","last_scrub_stamp":"2021-03-15T22:24:02.949625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:02.949625+0000","log_size":162,"ondisk_log_size":162,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16081782,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":81,"num_read_kb":100355,"num_write":58,"num_write_kb":24058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":14412391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"439","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.115789+0000","last_change":"2021-03-15T22:25:14.112119+0000","last_active":"2021-03-15T22:25:55.115789+0000","last_peered":"2021-03-15T22:25:55.115789+0000","last_clean":"2021-03-15T22:25:55.115789+0000","last_became_active":"2021-03-15T22:25:14.111797+0000","last_became_peered":"2021-03-15T22:25:14.111797+0000","last_unstale":"2021-03-15T22:25:55.115789+0000","last_undegraded":"2021-03-15T22:25:55.115789+0000","last_fullsized":"2021-03-15T22:25:55.115789+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.480279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"420","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.189298+0000","last_change":"2021-03-15T22:25:33.141491+0000","last_active":"2021-03-15T22:25:56.189298+0000","last_peered":"2021-03-15T22:25:56.189298+0000","last_clean":"2021-03-15T22:25:56.189298+0000","last_became_active":"2021-03-15T22:25:33.141147+0000","last_became_peered":"2021-03-15T22:25:33.141147+0000","last_unstale":"2021-03-15T22:25:56.189298+0000","last_undegraded":"2021-03-15T22:25:56.189298+0000","last_fullsized":"2021-03-15T22:25:56.189298+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.769850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.769850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"533","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.144085+0000","last_change":"2021-03-15T22:25:15.063557+0000","last_active":"2021-03-15T22:25:56.144085+0000","last_peered":"2021-03-15T22:25:56.144085+0000","last_clean":"2021-03-15T22:25:56.144085+0000","last_became_active":"2021-03-15T22:25:14.114107+0000","last_became_peered":"2021-03-15T22:25:14.114107+0000","last_unstale":"2021-03-15T22:25:56.144085+0000","last_undegraded":"2021-03-15T22:25:56.144085+0000","last_fullsized":"2021-03-15T22:25:56.144085+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:23:29.014596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.014596+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"356'3","reported_seq":"127","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.136515+0000","last_change":"2021-03-15T22:25:43.596578+0000","last_active":"2021-03-15T22:25:54.136515+0000","last_peered":"2021-03-15T22:25:54.136515+0000","last_clean":"2021-03-15T22:25:54.136515+0000","last_became_active":"2021-03-15T22:24:31.105948+0000","last_became_peered":"2021-03-15T22:24:31.105948+0000","last_unstale":"2021-03-15T22:25:54.136515+0000","last_undegraded":"2021-03-15T22:25:54.136515+0000","last_fullsized":"2021-03-15T22:25:54.136515+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:24:02.507323+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:02.507323+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"441'289","reported_seq":"1370","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.168517+0000","last_change":"2021-03-15T22:25:55.127677+0000","last_active":"2021-03-15T22:25:56.168517+0000","last_peered":"2021-03-15T22:25:56.168517+0000","last_clean":"2021-03-15T22:25:56.168517+0000","last_became_active":"2021-03-15T22:23:00.057259+0000","last_became_peered":"2021-03-15T22:23:00.057259+0000","last_unstale":"2021-03-15T22:25:56.168517+0000","last_undegraded":"2021-03-15T22:25:56.168517+0000","last_fullsized":"2021-03-15T22:25:56.168517+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'202","last_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.006150+0000","log_size":289,"ondisk_log_size":289,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19674112,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":128,"num_read_kb":122014,"num_write":101,"num_write_kb":38922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":6765552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"399","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.127268+0000","last_change":"2021-03-15T22:25:16.234631+0000","last_active":"2021-03-15T22:25:53.127268+0000","last_peered":"2021-03-15T22:25:53.127268+0000","last_clean":"2021-03-15T22:25:53.127268+0000","last_became_active":"2021-03-15T22:23:52.260549+0000","last_became_peered":"2021-03-15T22:23:52.260549+0000","last_unstale":"2021-03-15T22:25:53.127268+0000","last_undegraded":"2021-03-15T22:25:53.127268+0000","last_fullsized":"2021-03-15T22:25:53.127268+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.234565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.234565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"104","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.144891+0000","last_change":"2021-03-15T22:25:02.868460+0000","last_active":"2021-03-15T22:25:56.144891+0000","last_peered":"2021-03-15T22:25:56.144891+0000","last_clean":"2021-03-15T22:25:56.144891+0000","last_became_active":"2021-03-15T22:24:31.099610+0000","last_became_peered":"2021-03-15T22:24:31.099610+0000","last_unstale":"2021-03-15T22:25:56.144891+0000","last_undegraded":"2021-03-15T22:25:56.144891+0000","last_fullsized":"2021-03-15T22:25:56.144891+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.868399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.868399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"440","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.119939+0000","last_change":"2021-03-15T22:25:39.041168+0000","last_active":"2021-03-15T22:25:55.119939+0000","last_peered":"2021-03-15T22:25:55.119939+0000","last_clean":"2021-03-15T22:25:55.119939+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:25:55.119939+0000","last_undegraded":"2021-03-15T22:25:55.119939+0000","last_fullsized":"2021-03-15T22:25:55.119939+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:23:07.762338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:07.762338+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"421'6","reported_seq":"41","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.115175+0000","last_change":"2021-03-15T22:25:33.171536+0000","last_active":"2021-03-15T22:25:55.115175+0000","last_peered":"2021-03-15T22:25:55.115175+0000","last_clean":"2021-03-15T22:25:55.115175+0000","last_became_active":"2021-03-15T22:25:33.130973+0000","last_became_peered":"2021-03-15T22:25:33.130973+0000","last_unstale":"2021-03-15T22:25:55.115175+0000","last_undegraded":"2021-03-15T22:25:55.115175+0000","last_fullsized":"2021-03-15T22:25:55.115175+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'4","last_scrub_stamp":"2021-03-15T22:23:35.100079+0000","last_deep_scrub":"300'4","last_deep_scrub_stamp":"2021-03-15T22:23:35.100079+0000","last_clean_scrub_stamp":"2021-03-15T22:23:35.100079+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"440'479","reported_seq":"1654","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.122147+0000","last_change":"2021-03-15T22:25:55.122147+0000","last_active":"2021-03-15T22:25:55.122147+0000","last_peered":"2021-03-15T22:25:55.122147+0000","last_clean":"2021-03-15T22:25:55.122147+0000","last_became_active":"2021-03-15T22:23:01.072357+0000","last_became_peered":"2021-03-15T22:23:01.072357+0000","last_unstale":"2021-03-15T22:25:55.122147+0000","last_undegraded":"2021-03-15T22:25:55.122147+0000","last_fullsized":"2021-03-15T22:25:55.122147+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'302","last_scrub_stamp":"2021-03-15T22:23:59.467761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:59.467761+0000","log_size":479,"ondisk_log_size":479,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26858891,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":3,"num_read":241,"num_read_kb":259097,"num_write":147,"num_write_kb":58613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31233387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"424","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.118812+0000","last_change":"2021-03-15T22:25:47.117313+0000","last_active":"2021-03-15T22:25:55.118812+0000","last_peered":"2021-03-15T22:25:55.118812+0000","last_clean":"2021-03-15T22:25:55.118812+0000","last_became_active":"2021-03-15T22:25:33.136621+0000","last_became_peered":"2021-03-15T22:25:33.136621+0000","last_unstale":"2021-03-15T22:25:55.118812+0000","last_undegraded":"2021-03-15T22:25:55.118812+0000","last_fullsized":"2021-03-15T22:25:55.118812+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:47.117252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:47.117252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"209","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.113891+0000","last_change":"2021-03-15T22:25:12.911203+0000","last_active":"2021-03-15T22:25:55.113891+0000","last_peered":"2021-03-15T22:25:55.113891+0000","last_clean":"2021-03-15T22:25:55.113891+0000","last_became_active":"2021-03-15T22:24:31.091206+0000","last_became_peered":"2021-03-15T22:24:31.091206+0000","last_unstale":"2021-03-15T22:25:55.113891+0000","last_undegraded":"2021-03-15T22:25:55.113891+0000","last_fullsized":"2021-03-15T22:25:55.113891+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.911140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.911140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"59","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.115018+0000","last_change":"2021-03-15T22:25:33.758535+0000","last_active":"2021-03-15T22:25:55.115018+0000","last_peered":"2021-03-15T22:25:55.115018+0000","last_clean":"2021-03-15T22:25:55.115018+0000","last_became_active":"2021-03-15T22:25:33.149362+0000","last_became_peered":"2021-03-15T22:25:33.149362+0000","last_unstale":"2021-03-15T22:25:55.115018+0000","last_undegraded":"2021-03-15T22:25:55.115018+0000","last_fullsized":"2021-03-15T22:25:55.115018+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:24:01.896988+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.896988+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"432","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.119609+0000","last_change":"2021-03-15T22:25:33.143430+0000","last_active":"2021-03-15T22:25:55.119609+0000","last_peered":"2021-03-15T22:25:55.119609+0000","last_clean":"2021-03-15T22:25:55.119609+0000","last_became_active":"2021-03-15T22:25:33.142712+0000","last_became_peered":"2021-03-15T22:25:33.142712+0000","last_unstale":"2021-03-15T22:25:55.119609+0000","last_undegraded":"2021-03-15T22:25:55.119609+0000","last_fullsized":"2021-03-15T22:25:55.119609+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.989313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.989313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"440'444","reported_seq":"1638","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.119401+0000","last_change":"2021-03-15T22:25:55.119631+0000","last_active":"2021-03-15T22:25:55.119401+0000","last_peered":"2021-03-15T22:25:55.119401+0000","last_clean":"2021-03-15T22:25:55.119401+0000","last_became_active":"2021-03-15T22:25:36.118668+0000","last_became_peered":"2021-03-15T22:25:36.118668+0000","last_unstale":"2021-03-15T22:25:55.119401+0000","last_undegraded":"2021-03-15T22:25:55.119401+0000","last_fullsized":"2021-03-15T22:25:55.119401+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'370","last_scrub_stamp":"2021-03-15T22:25:07.414191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.414191+0000","log_size":444,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26033381,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":182,"num_read_kb":168083,"num_write":137,"num_write_kb":52354,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25937939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"405","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.189166+0000","last_change":"2021-03-15T22:24:31.094621+0000","last_active":"2021-03-15T22:25:56.189166+0000","last_peered":"2021-03-15T22:25:56.189166+0000","last_clean":"2021-03-15T22:25:56.189166+0000","last_became_active":"2021-03-15T22:24:31.094299+0000","last_became_peered":"2021-03-15T22:24:31.094299+0000","last_unstale":"2021-03-15T22:25:56.189166+0000","last_undegraded":"2021-03-15T22:25:56.189166+0000","last_fullsized":"2021-03-15T22:25:56.189166+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.936738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.936738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"391'7","reported_seq":"554","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.119431+0000","last_change":"2021-03-15T22:25:46.103567+0000","last_active":"2021-03-15T22:25:55.119431+0000","last_peered":"2021-03-15T22:25:55.119431+0000","last_clean":"2021-03-15T22:25:55.119431+0000","last_became_active":"2021-03-15T22:25:03.403350+0000","last_became_peered":"2021-03-15T22:25:03.403350+0000","last_unstale":"2021-03-15T22:25:55.119431+0000","last_undegraded":"2021-03-15T22:25:55.119431+0000","last_fullsized":"2021-03-15T22:25:55.119431+0000","mapping_epoch":387,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'7","last_scrub_stamp":"2021-03-15T22:25:46.103504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.103504+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"38","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.115092+0000","last_change":"2021-03-15T22:25:44.867726+0000","last_active":"2021-03-15T22:25:55.115092+0000","last_peered":"2021-03-15T22:25:55.115092+0000","last_clean":"2021-03-15T22:25:55.115092+0000","last_became_active":"2021-03-15T22:25:33.148614+0000","last_became_peered":"2021-03-15T22:25:33.148614+0000","last_unstale":"2021-03-15T22:25:55.115092+0000","last_undegraded":"2021-03-15T22:25:55.115092+0000","last_fullsized":"2021-03-15T22:25:55.115092+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:44.867633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:44.867633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"469","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.124010+0000","last_change":"2021-03-15T22:25:46.725552+0000","last_active":"2021-03-15T22:25:53.124010+0000","last_peered":"2021-03-15T22:25:53.124010+0000","last_clean":"2021-03-15T22:25:53.124010+0000","last_became_active":"2021-03-15T22:22:56.057343+0000","last_became_peered":"2021-03-15T22:22:56.057343+0000","last_unstale":"2021-03-15T22:25:53.124010+0000","last_undegraded":"2021-03-15T22:25:53.124010+0000","last_fullsized":"2021-03-15T22:25:53.124010+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:25:46.725489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.725489+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"440'95","reported_seq":"1200","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.119774+0000","last_change":"2021-03-15T22:25:55.120004+0000","last_active":"2021-03-15T22:25:55.119774+0000","last_peered":"2021-03-15T22:25:55.119774+0000","last_clean":"2021-03-15T22:25:55.119774+0000","last_became_active":"2021-03-15T22:24:31.104356+0000","last_became_peered":"2021-03-15T22:24:31.104356+0000","last_unstale":"2021-03-15T22:25:55.119774+0000","last_undegraded":"2021-03-15T22:25:55.119774+0000","last_fullsized":"2021-03-15T22:25:55.119774+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'77","last_scrub_stamp":"2021-03-15T22:25:32.490694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.490694+0000","log_size":95,"ondisk_log_size":95,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7752259,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":89949,"num_write":32,"num_write_kb":13772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1551588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"33","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.113627+0000","last_change":"2021-03-15T22:25:33.148157+0000","last_active":"2021-03-15T22:25:55.113627+0000","last_peered":"2021-03-15T22:25:55.113627+0000","last_clean":"2021-03-15T22:25:55.113627+0000","last_became_active":"2021-03-15T22:25:33.147717+0000","last_became_peered":"2021-03-15T22:25:33.147717+0000","last_unstale":"2021-03-15T22:25:55.113627+0000","last_undegraded":"2021-03-15T22:25:55.113627+0000","last_fullsized":"2021-03-15T22:25:55.113627+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.372535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.372535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"147'5","reported_seq":"482","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:57.088390+0000","last_change":"2021-03-15T22:24:35.715464+0000","last_active":"2021-03-15T22:25:57.088390+0000","last_peered":"2021-03-15T22:25:57.088390+0000","last_clean":"2021-03-15T22:25:57.088390+0000","last_became_active":"2021-03-15T22:21:02.017066+0000","last_became_peered":"2021-03-15T22:21:02.017066+0000","last_unstale":"2021-03-15T22:25:57.088390+0000","last_undegraded":"2021-03-15T22:25:57.088390+0000","last_fullsized":"2021-03-15T22:25:57.088390+0000","mapping_epoch":60,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'5","last_scrub_stamp":"2021-03-15T22:24:35.715399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:35.715399+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"409","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.119735+0000","last_change":"2021-03-15T22:24:34.991087+0000","last_active":"2021-03-15T22:25:55.119735+0000","last_peered":"2021-03-15T22:25:55.119735+0000","last_clean":"2021-03-15T22:25:55.119735+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:25:55.119735+0000","last_undegraded":"2021-03-15T22:25:55.119735+0000","last_fullsized":"2021-03-15T22:25:55.119735+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.991018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.991018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"490","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.119228+0000","last_change":"2021-03-15T22:24:55.128840+0000","last_active":"2021-03-15T22:25:55.119228+0000","last_peered":"2021-03-15T22:25:55.119228+0000","last_clean":"2021-03-15T22:25:55.119228+0000","last_became_active":"2021-03-15T22:24:55.090544+0000","last_became_peered":"2021-03-15T22:24:55.090544+0000","last_unstale":"2021-03-15T22:25:55.119228+0000","last_undegraded":"2021-03-15T22:25:55.119228+0000","last_fullsized":"2021-03-15T22:25:55.119228+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:38.685706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:38.685706+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"439'131","reported_seq":"1338","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.133174+0000","last_change":"2021-03-15T22:25:54.133436+0000","last_active":"2021-03-15T22:25:54.133174+0000","last_peered":"2021-03-15T22:25:54.133174+0000","last_clean":"2021-03-15T22:25:54.133174+0000","last_became_active":"2021-03-15T22:25:37.119078+0000","last_became_peered":"2021-03-15T22:25:37.119078+0000","last_unstale":"2021-03-15T22:25:54.133174+0000","last_undegraded":"2021-03-15T22:25:54.133174+0000","last_fullsized":"2021-03-15T22:25:54.133174+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'115","last_scrub_stamp":"2021-03-15T22:25:10.616929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.616929+0000","log_size":131,"ondisk_log_size":131,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9539709,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":110,"num_read_kb":136588,"num_write":53,"num_write_kb":14217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":16774328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"10e","length":"1"},{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"13a","length":"2"},{"start":"145","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"419","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.124240+0000","last_change":"2021-03-15T22:25:38.702159+0000","last_active":"2021-03-15T22:25:53.124240+0000","last_peered":"2021-03-15T22:25:53.124240+0000","last_clean":"2021-03-15T22:25:53.124240+0000","last_became_active":"2021-03-15T22:25:33.134275+0000","last_became_peered":"2021-03-15T22:25:33.134275+0000","last_unstale":"2021-03-15T22:25:53.124240+0000","last_undegraded":"2021-03-15T22:25:53.124240+0000","last_fullsized":"2021-03-15T22:25:53.124240+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:23:02.474236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"433","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.145161+0000","last_change":"2021-03-15T22:24:31.107075+0000","last_active":"2021-03-15T22:25:56.145161+0000","last_peered":"2021-03-15T22:25:56.145161+0000","last_clean":"2021-03-15T22:25:56.145161+0000","last_became_active":"2021-03-15T22:24:31.106634+0000","last_became_peered":"2021-03-15T22:24:31.106634+0000","last_unstale":"2021-03-15T22:25:56.145161+0000","last_undegraded":"2021-03-15T22:25:56.145161+0000","last_fullsized":"2021-03-15T22:25:56.145161+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:42.177422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.177422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"412","reported_epoch":"439","state":"active+clean","last_fresh":"2021-03-15T22:25:54.135924+0000","last_change":"2021-03-15T22:24:28.327738+0000","last_active":"2021-03-15T22:25:54.135924+0000","last_peered":"2021-03-15T22:25:54.135924+0000","last_clean":"2021-03-15T22:25:54.135924+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:25:54.135924+0000","last_undegraded":"2021-03-15T22:25:54.135924+0000","last_fullsized":"2021-03-15T22:25:54.135924+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:28.327677+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.327677+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"115'33","reported_seq":"455","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.118769+0000","last_change":"2021-03-15T22:25:04.028853+0000","last_active":"2021-03-15T22:25:55.118769+0000","last_peered":"2021-03-15T22:25:55.118769+0000","last_clean":"2021-03-15T22:25:55.118769+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:25:55.118769+0000","last_undegraded":"2021-03-15T22:25:55.118769+0000","last_fullsized":"2021-03-15T22:25:55.118769+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'33","last_scrub_stamp":"2021-03-15T22:25:04.028780+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:25:04.028780+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"441'383","reported_seq":"1701","reported_epoch":"441","state":"active+clean","last_fresh":"2021-03-15T22:25:56.156996+0000","last_change":"2021-03-15T22:25:55.114504+0000","last_active":"2021-03-15T22:25:56.156996+0000","last_peered":"2021-03-15T22:25:56.156996+0000","last_clean":"2021-03-15T22:25:56.156996+0000","last_became_active":"2021-03-15T22:25:39.143407+0000","last_became_peered":"2021-03-15T22:25:39.143407+0000","last_unstale":"2021-03-15T22:25:56.156996+0000","last_undegraded":"2021-03-15T22:25:56.156996+0000","last_fullsized":"2021-03-15T22:25:56.156996+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'343","last_scrub_stamp":"2021-03-15T22:23:26.791043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.791043+0000","log_size":383,"ondisk_log_size":383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11244213,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":185,"num_read_kb":156185,"num_write":139,"num_write_kb":54601,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":38892785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"118","length":"1"},{"start":"121","length":"1"},{"start":"135","length":"1"},{"start":"13a","length":"1"},{"start":"145","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"401","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.119512+0000","last_change":"2021-03-15T22:24:40.910630+0000","last_active":"2021-03-15T22:25:55.119512+0000","last_peered":"2021-03-15T22:25:55.119512+0000","last_clean":"2021-03-15T22:25:55.119512+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:25:55.119512+0000","last_undegraded":"2021-03-15T22:25:55.119512+0000","last_fullsized":"2021-03-15T22:25:55.119512+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.910565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.910565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"155'4","reported_seq":"522","reported_epoch":"438","state":"active+clean","last_fresh":"2021-03-15T22:25:53.124343+0000","last_change":"2021-03-15T22:24:29.903505+0000","last_active":"2021-03-15T22:25:53.124343+0000","last_peered":"2021-03-15T22:25:53.124343+0000","last_clean":"2021-03-15T22:25:53.124343+0000","last_became_active":"2021-03-15T22:21:10.013838+0000","last_became_peered":"2021-03-15T22:21:10.013838+0000","last_unstale":"2021-03-15T22:25:53.124343+0000","last_undegraded":"2021-03-15T22:25:53.124343+0000","last_fullsized":"2021-03-15T22:25:53.124343+0000","mapping_epoch":60,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'4","last_scrub_stamp":"2021-03-15T22:24:29.903442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.903442+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"119","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.112001+0000","last_change":"2021-03-15T22:25:02.791176+0000","last_active":"2021-03-15T22:25:55.112001+0000","last_peered":"2021-03-15T22:25:55.112001+0000","last_clean":"2021-03-15T22:25:55.112001+0000","last_became_active":"2021-03-15T22:24:31.097434+0000","last_became_peered":"2021-03-15T22:24:31.097434+0000","last_unstale":"2021-03-15T22:25:55.112001+0000","last_undegraded":"2021-03-15T22:25:55.112001+0000","last_fullsized":"2021-03-15T22:25:55.112001+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.791108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:06.573181+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.791108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"482","reported_epoch":"440","state":"active+clean","last_fresh":"2021-03-15T22:25:55.120831+0000","last_change":"2021-03-15T22:25:34.027206+0000","last_active":"2021-03-15T22:25:55.120831+0000","last_peered":"2021-03-15T22:25:55.120831+0000","last_clean":"2021-03-15T22:25:55.120831+0000","last_became_active":"2021-03-15T22:22:56.053800+0000","last_became_peered":"2021-03-15T22:22:56.053800+0000","last_unstale":"2021-03-15T22:25:55.120831+0000","last_undegraded":"2021-03-15T22:25:55.120831+0000","last_fullsized":"2021-03-15T22:25:55.120831+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:23:05.791845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:05.791845+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":418643327,"num_objects":201,"num_object_clones":152,"num_object_copies":603,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":201,"num_whiteouts":15,"num_read":3233,"num_read_kb":3287712,"num_write":2341,"num_write_kb":887678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1197,"num_bytes_recovered":514315662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":979439616,"data_stored":1338242286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6580,"ondisk_log_size":6580,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":466,"num_read_kb":461,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1061,"num_read_kb":566,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":37148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2232320,"data_stored":80092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299492,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":130076,"kb_used_data":111636,"kb_used_omap":0,"kb_used_meta":18432,"kb_avail":93615076,"statfs":{"total":95995035648,"available":95861837824,"internally_reserved":0,"allocated":114315264,"data_stored":153188867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":18874368},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.215,"5min":0.215,"15min":0.215},"max":{"1min":1.0189999999999999,"5min":2.73,"15min":2.73},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.42299999999999999,"5min":0.498,"15min":0.499},"min":{"1min":0.252,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.65000000000000002,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.41999999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.504,"15min":0.48599999999999999},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":0.97299999999999998,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.432},{"interface":"front","average":{"1min":0.436,"5min":0.48499999999999999,"15min":0.48999999999999999},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.72699999999999998,"5min":1.877,"15min":1.877},"last":0.44800000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.47199999999999998,"15min":0.47899999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.73499999999999999,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.377},{"interface":"front","average":{"1min":0.498,"5min":0.51900000000000002,"15min":0.5},"min":{"1min":0.311,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":0.95099999999999996,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.39300000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48399999999999999,"15min":0.54200000000000004},"min":{"1min":0.315,"5min":0.31,"15min":0.31},"max":{"1min":0.69499999999999995,"5min":0.95999999999999996,"15min":2.0339999999999998},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.53400000000000003,"15min":0.55400000000000005},"min":{"1min":0.34799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.98899999999999999,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.36799999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.40500000000000003,"5min":0.46600000000000003,"15min":0.436},"min":{"1min":0.247,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.75700000000000001,"5min":1.835,"15min":1.835},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.47699999999999998,"15min":0.44600000000000001},"min":{"1min":0.29099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.66200000000000003,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.40300000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.52200000000000002,"15min":0.45800000000000002},"min":{"1min":0.31900000000000001,"5min":0.253,"15min":0.253},"max":{"1min":1.034,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.495,"5min":0.53100000000000003,"15min":0.504},"min":{"1min":0.33900000000000002,"5min":0.255,"15min":0.255},"max":{"1min":0.78300000000000003,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.47099999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.54600000000000004,"15min":0.51000000000000001},"min":{"1min":0.34499999999999997,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.80700000000000005,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.53700000000000003,"15min":0.50900000000000001},"min":{"1min":0.35299999999999998,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.77700000000000002,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.439}]}]},{"osd":6,"up_from":45,"seq":193273528424,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":131400,"kb_used_data":102976,"kb_used_omap":0,"kb_used_meta":28416,"kb_avail":93613752,"statfs":{"total":95995035648,"available":95860482048,"internally_reserved":0,"allocated":105447424,"data_stored":140917380,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29097984},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47499999999999998,"15min":0.48799999999999999},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.68100000000000005,"5min":0.995,"15min":0.995},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.46800000000000003,"15min":0.48599999999999999},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.41199999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.49099999999999999,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.254,"15min":0.254},"max":{"1min":0.69399999999999995,"5min":1.05,"15min":1.05},"last":0.34000000000000002},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.33300000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68000000000000005,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.375}]},{"osd":2,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.317,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65200000000000002,"5min":0.77600000000000002,"15min":0.80800000000000005},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48899999999999999,"15min":0.498},"min":{"1min":0.35399999999999998,"5min":0.32800000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.35199999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.35199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.90400000000000003,"5min":1.026,"15min":1.026},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51100000000000001,"15min":0.52400000000000002},"min":{"1min":0.33600000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.33300000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.44800000000000001,"15min":0.41799999999999998},"min":{"1min":0.28000000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51500000000000001,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.38800000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.501,"15min":0.46200000000000002},"min":{"1min":0.32300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.89200000000000002,"5min":1.135,"15min":1.135},"last":0.308},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53200000000000003,"15min":0.48999999999999999},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.438}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.52300000000000002,"15min":0.50600000000000001},"min":{"1min":0.36799999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.79600000000000004,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.56299999999999994,"15min":0.54200000000000004},"min":{"1min":0.36699999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.76100000000000001,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.44800000000000001}]}]},{"osd":1,"up_from":13,"seq":55834574970,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":192632,"kb_used_data":165572,"kb_used_omap":0,"kb_used_meta":25856,"kb_avail":93552520,"statfs":{"total":95995035648,"available":95797780480,"internally_reserved":0,"allocated":169545728,"data_stored":235146619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26476544},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.34499999999999997,"15min":0.52700000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.48899999999999999,"5min":1.371,"15min":54.427999999999997},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.54400000000000004},"min":{"1min":0.20999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.48199999999999998,"5min":1.23,"15min":54.356999999999999},"last":0.435}]},{"osd":2,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.54300000000000004,"5min":1.1220000000000001,"15min":2.4100000000000001},"last":0.5},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.38600000000000001},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.502,"5min":0.69799999999999995,"15min":2.0760000000000001},"last":0.376}]},{"osd":3,"last update":"Mon Mar 15 22:24:57 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.38500000000000001,"15min":0.41899999999999998},"min":{"1min":0.26100000000000001,"5min":0.222,"15min":0.222},"max":{"1min":1.4910000000000001,"5min":1.4910000000000001,"15min":1.796},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40799999999999997,"15min":0.46400000000000002},"min":{"1min":0.29299999999999998,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.623,"5min":1.135,"15min":1.7150000000000001},"last":0.53400000000000003}]},{"osd":4,"last update":"Mon Mar 15 22:25:10 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57399999999999995,"15min":0.57699999999999996},"min":{"1min":0.29999999999999999,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.95199999999999996,"5min":3.3439999999999999,"15min":3.3439999999999999},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.621,"15min":0.626},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.93999999999999995,"5min":4.6139999999999999,"15min":4.6139999999999999},"last":0.48499999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.627},"min":{"1min":0.42199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.93400000000000005,"5min":1.524,"15min":2.153},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58499999999999996,"15min":0.95899999999999996},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88900000000000001,"5min":1.1950000000000001,"15min":15.472},"last":0.70899999999999996}]},{"osd":6,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61099999999999999,"15min":0.65600000000000003},"min":{"1min":0.47599999999999998,"5min":0.36899999999999999,"15min":0.33200000000000002},"max":{"1min":0.91800000000000004,"5min":1.276,"15min":1.75},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.3580000000000001},"last":0.47299999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.60199999999999998,"5min":0.63500000000000001,"15min":0.871},"min":{"1min":0.377,"5min":0.316,"15min":0.316},"max":{"1min":0.96299999999999997,"5min":2.1219999999999999,"15min":8.8100000000000005},"last":0.72499999999999998},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.623},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.84799999999999998,"5min":1.0329999999999999,"15min":1.325},"last":0.496}]}]},{"osd":0,"up_from":0,"seq":34359738492,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411446,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132240,"kb_used_data":105864,"kb_used_omap":0,"kb_used_meta":26368,"kb_avail":93612912,"statfs":{"total":95995035648,"available":95859621888,"internally_reserved":0,"allocated":108404736,"data_stored":145010541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27000832},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.246,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.45700000000000002,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.437,"15min":0.39500000000000002},"min":{"1min":0.246,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":5.5250000000000004,"5min":5.5250000000000004,"15min":5.5250000000000004},"last":0.30599999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.42599999999999999,"15min":0.39600000000000002},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.60999999999999999,"5min":5.4489999999999998,"15min":5.4489999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51800000000000002,"15min":0.46600000000000003},"min":{"1min":0.27400000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.3740000000000001,"5min":11.659000000000001,"15min":11.659000000000001},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:24:53 2021","interfaces":[{"interface":"back","average":{"1min":0.35199999999999998,"5min":0.38400000000000001,"15min":0.41799999999999998},"min":{"1min":0.26900000000000002,"5min":0.22,"15min":0.22},"max":{"1min":0.55600000000000005,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.443,"15min":0.48299999999999998},"min":{"1min":0.27400000000000002,"5min":0.26400000000000001,"15min":0.24099999999999999},"max":{"1min":2.8759999999999999,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.441}]},{"osd":4,"last update":"Mon Mar 15 22:25:09 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56999999999999995,"15min":0.57499999999999996},"min":{"1min":0.40999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73899999999999999,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54500000000000004,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.60199999999999998,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.499}]},{"osd":5,"last update":"Mon Mar 15 22:25:36 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.38400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75600000000000001,"5min":1.954,"15min":1.954},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.745,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.432}]},{"osd":6,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":1.875,"15min":1.875},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.435,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.3520000000000001,"5min":1.609,"15min":1.609},"last":0.51100000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:25:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.66900000000000004,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.40600000000000003,"15min":0.38700000000000001},"max":{"1min":0.64700000000000002,"5min":1.923,"15min":1.923},"last":0.57199999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149810,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":278840,"kb_used_data":252720,"kb_used_omap":0,"kb_used_meta":26112,"kb_avail":93466312,"statfs":{"total":95995035648,"available":95709503488,"internally_reserved":0,"allocated":258785280,"data_stored":335181140,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26738688},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.29099999999999998,"5min":0.35199999999999998,"15min":0.35799999999999998},"min":{"1min":0.22700000000000001,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.35699999999999998,"5min":2.625,"15min":2.625},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.36499999999999999,"15min":0.379},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.189},"max":{"1min":0.438,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.36299999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.34499999999999997,"5min":0.44900000000000001,"15min":0.434},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.50700000000000001,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.34499999999999997,"5min":0.46100000000000002,"15min":0.437},"min":{"1min":0.28699999999999998,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.45300000000000001,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.436}]},{"osd":2,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.45000000000000001,"15min":0.435},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.48199999999999998,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.34399999999999997},{"interface":"front","average":{"1min":0.316,"5min":0.42299999999999999,"15min":0.40400000000000003},"min":{"1min":0.26600000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.46300000000000002,"5min":7.0860000000000003,"15min":7.0860000000000003},"last":0.32400000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:25:10 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55500000000000005,"15min":0.52400000000000002},"min":{"1min":0.32600000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.66800000000000004,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.318,"15min":0.30399999999999999},"max":{"1min":0.69299999999999995,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.41699999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.54500000000000004},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.62,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":0.53700000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:25:43 2021","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68500000000000005,"5min":2.6499999999999999,"15min":2.6499999999999999},"last":0.504},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70999999999999996,"5min":1.268,"15min":1.268},"last":0.45100000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:25:02 2021","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.58399999999999996,"15min":0.57699999999999996},"min":{"1min":0.377,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.67400000000000004,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.57699999999999996,"15min":0.57199999999999995},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.377},"max":{"1min":0.72699999999999998,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.46000000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986287,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113248,"kb_used_data":80728,"kb_used_omap":0,"kb_used_meta":32512,"kb_avail":93631904,"statfs":{"total":95995035648,"available":95879069696,"internally_reserved":0,"allocated":82665472,"data_stored":110178899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33292288},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.19500000000000001},"max":{"1min":0.73099999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48999999999999999},"min":{"1min":0.34999999999999998,"5min":0.29099999999999998,"15min":0.251},"max":{"1min":0.66600000000000004,"5min":0.72199999999999998,"15min":0.74199999999999999},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.47599999999999998,"15min":0.5},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23599999999999999},"max":{"1min":0.68100000000000005,"5min":1.631,"15min":1.631},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26800000000000002},"max":{"1min":0.73999999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.47199999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.51100000000000001},"min":{"1min":0.40300000000000002,"5min":0.32300000000000001,"15min":0.313},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.76100000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":1.1910000000000001},"last":0.51600000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52600000000000002,"15min":0.55000000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68899999999999995,"5min":0.75,"15min":0.98799999999999999},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51500000000000001,"15min":0.55800000000000005},"min":{"1min":0.40200000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68899999999999995,"5min":0.69199999999999995,"15min":0.99399999999999999},"last":0.46300000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:25:28 2021","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.41799999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.49299999999999999,"15min":0.441},"min":{"1min":0.36099999999999999,"5min":0.29699999999999999,"15min":0.28799999999999998},"max":{"1min":0.64400000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.442}]},{"osd":6,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46800000000000003,"15min":0.437},"min":{"1min":0.29099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65400000000000003,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53900000000000003,"15min":0.51400000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.71499999999999997,"5min":0.78400000000000003,"15min":0.84799999999999998},"last":0.503}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.51800000000000002,"15min":0.48899999999999999},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.70899999999999996,"5min":0.73799999999999999,"15min":0.80600000000000005},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.52000000000000002,"5min":0.55400000000000005,"15min":0.52500000000000002},"min":{"1min":0.33000000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.75,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.57699999999999996}]}]},{"osd":5,"up_from":38,"seq":163208757356,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":176512,"kb_used_data":156536,"kb_used_omap":0,"kb_used_meta":19968,"kb_avail":93568640,"statfs":{"total":95995035648,"available":95814287360,"internally_reserved":0,"allocated":160292864,"data_stored":225236381,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20447232},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48799999999999999,"15min":0.48299999999999998},"min":{"1min":0.29499999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63500000000000001,"5min":2.855,"15min":2.855},"last":0.55700000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.498},"min":{"1min":0.34699999999999998,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.75700000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.33100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.72999999999999998,"5min":2.665,"15min":2.665},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.72099999999999997,"5min":0.81999999999999995,"15min":0.83999999999999997},"last":0.54600000000000004}]},{"osd":2,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.39000000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61799999999999999,"5min":2.827,"15min":2.827},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.51400000000000001},"min":{"1min":0.29899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.502}]},{"osd":3,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.55100000000000005,"15min":0.51800000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54800000000000004,"15min":0.53300000000000003},"min":{"1min":0.39400000000000002,"5min":0.34100000000000003,"15min":0.33200000000000002},"max":{"1min":0.65500000000000003,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.53700000000000003}]},{"osd":4,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46999999999999997,"15min":0.40899999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24099999999999999},"max":{"1min":0.70899999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.496,"5min":0.505,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.25},"max":{"1min":0.748,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.38700000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:25:45 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49399999999999999,"15min":0.47199999999999998},"min":{"1min":0.32300000000000001,"5min":0.29199999999999998,"15min":0.28499999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56599999999999995,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.32800000000000001},"max":{"1min":0.78600000000000003,"5min":1.327,"15min":1.327},"last":0.441}]},{"osd":7,"last update":"Mon Mar 15 22:24:59 2021","interfaces":[{"interface":"back","average":{"1min":0.54100000000000004,"5min":0.56899999999999995,"15min":0.52100000000000002},"min":{"1min":0.36899999999999999,"5min":0.34699999999999998,"15min":0.30499999999999999},"max":{"1min":0.872,"5min":1.3,"15min":1.3},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57699999999999996,"15min":0.53500000000000003},"min":{"1min":0.38,"5min":0.307,"15min":0.307},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.45200000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":9520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":14458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":8356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":6840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":8774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5443584,"data_stored":5395029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":164753408,"data_stored":232187078,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":103833600,"data_stored":142082226,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":257499136,"data_stored":337224654,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":78032896,"data_stored":107240995,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155725824,"data_stored":222284424,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":100835328,"data_stored":137979335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":113315840,"data_stored":153848545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:26:00.946 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2021-03-15T22:26:00.946 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:26:00.946 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 0.999251 2021-03-15T22:26:00.947 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd primary-affinity 1 0.9992505298965398 2021-03-15T22:26:00.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:00.972 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:00.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:00.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:01.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:01.098 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:01.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:01.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:26:01.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:01.226 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:01.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:01.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:01.275 INFO:tasks.rados.rados.0.smithi076.stdout:1660: done (2 left) 2021-03-15T22:26:01.276 INFO:tasks.rados.rados.0.smithi076.stdout:1661: rollback oid 16 current snap is 181 2021-03-15T22:26:01.276 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 133 2021-03-15T22:26:01.276 INFO:tasks.rados.rados.0.smithi076.stdout:1662: read oid 26 snap -1 2021-03-15T22:26:01.276 INFO:tasks.rados.rados.0.smithi076.stdout:1662: expect deleted 2021-03-15T22:26:01.277 INFO:tasks.rados.rados.0.smithi076.stdout:1663: write (excl) oid 21 current snap is 181 2021-03-15T22:26:01.277 INFO:tasks.rados.rados.0.smithi076.stdout:1663: seq_num 589 ranges {699323=480842,1850247=429872,2858378=532392} 2021-03-15T22:26:01.282 INFO:tasks.rados.rados.0.smithi076.stdout:1663: writing smithi07694878-21 from 699323 to 1180165 tid 1 2021-03-15T22:26:01.289 INFO:tasks.rados.rados.0.smithi076.stdout:1663: writing smithi07694878-21 from 1850247 to 2280119 tid 2 2021-03-15T22:26:01.362 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:01.365 INFO:tasks.rados.rados.0.smithi076.stdout:1663: writing smithi07694878-21 from 2858378 to 3390770 tid 3 2021-03-15T22:26:01.365 INFO:tasks.rados.rados.0.smithi076.stdout:1656: finishing write tid 3 to smithi07694878-18 2021-03-15T22:26:01.365 INFO:tasks.rados.rados.0.smithi076.stdout:1656: finishing write tid 4 to smithi07694878-18 2021-03-15T22:26:01.366 INFO:tasks.rados.rados.0.smithi076.stdout:1656: finishing write tid 5 to smithi07694878-18 2021-03-15T22:26:01.366 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 789 (ObjNum 587 snap 181 seq_num 587) dirty exists 2021-03-15T22:26:01.366 INFO:tasks.rados.rados.0.smithi076.stdout:1656: left oid 18 (ObjNum 587 snap 181 seq_num 587) 2021-03-15T22:26:01.367 INFO:tasks.rados.rados.0.smithi076.stdout:1657: finishing rollback tid 1 to smithi07694878-2 2021-03-15T22:26:01.367 INFO:tasks.rados.rados.0.smithi076.stdout:1657: finishing rollback tid 2 to smithi07694878-2 2021-03-15T22:26:01.367 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 414 (ObjNum 505 snap 151 seq_num 505) dirty exists 2021-03-15T22:26:01.368 INFO:tasks.rados.rados.0.smithi076.stdout:1661: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:26:01.368 INFO:tasks.rados.rados.0.smithi076.stdout:1661: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:26:01.368 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1015 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:26:01.368 INFO:tasks.rados.rados.0.smithi076.stdout:1663: finishing write tid 1 to smithi07694878-21 2021-03-15T22:26:01.369 INFO:tasks.rados.rados.0.smithi076.stdout:1656: done (4 left) 2021-03-15T22:26:01.369 INFO:tasks.rados.rados.0.smithi076.stdout:1657: done (3 left) 2021-03-15T22:26:01.369 INFO:tasks.rados.rados.0.smithi076.stdout:1661: done (2 left) 2021-03-15T22:26:01.370 INFO:tasks.rados.rados.0.smithi076.stdout:1664: write (excl) oid 18 current snap is 181 2021-03-15T22:26:01.370 INFO:tasks.rados.rados.0.smithi076.stdout:1664: seq_num 590 ranges {716130=556091,1792376=573530,2769448=460017,3821956=94174} 2021-03-15T22:26:01.370 INFO:tasks.rados.rados.0.smithi076.stdout:1664: writing smithi07694878-18 from 716130 to 1272221 tid 1 2021-03-15T22:26:01.371 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:01.375 INFO:tasks.rados.rados.0.smithi076.stdout:1664: writing smithi07694878-18 from 1792376 to 2365906 tid 2 2021-03-15T22:26:01.375 INFO:tasks.rados.rados.0.smithi076.stdout:1664: writing smithi07694878-18 from 2769448 to 3229465 tid 3 2021-03-15T22:26:01.375 INFO:tasks.rados.rados.0.smithi076.stdout:1664: writing smithi07694878-18 from 3821956 to 3916130 tid 4 2021-03-15T22:26:01.376 INFO:tasks.rados.rados.0.smithi076.stdout:1665: write (excl) oid 7 current snap is 181 2021-03-15T22:26:01.376 INFO:tasks.rados.rados.0.smithi076.stdout:1665: seq_num 591 ranges {732937=631340,1850858=433541,2796871=671289,3888195=44742} 2021-03-15T22:26:01.376 INFO:tasks.rados.rados.0.smithi076.stdout:1665: writing smithi07694878-7 from 732937 to 1364277 tid 1 2021-03-15T22:26:01.377 INFO:tasks.rados.rados.0.smithi076.stdout:1665: writing smithi07694878-7 from 1850858 to 2284399 tid 2 2021-03-15T22:26:01.377 INFO:tasks.rados.rados.0.smithi076.stdout:1665: writing smithi07694878-7 from 2796871 to 3468160 tid 3 2021-03-15T22:26:01.377 INFO:tasks.rados.rados.0.smithi076.stdout:1665: writing smithi07694878-7 from 3888195 to 3932937 tid 4 2021-03-15T22:26:01.378 INFO:tasks.rados.rados.0.smithi076.stdout:1663: finishing write tid 2 to smithi07694878-21 2021-03-15T22:26:01.378 INFO:tasks.rados.rados.0.smithi076.stdout:1663: finishing write tid 3 to smithi07694878-21 2021-03-15T22:26:01.378 INFO:tasks.rados.rados.0.smithi076.stdout:1663: finishing write tid 4 to smithi07694878-21 2021-03-15T22:26:01.379 INFO:tasks.rados.rados.0.smithi076.stdout:1663: finishing write tid 5 to smithi07694878-21 2021-03-15T22:26:01.379 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 1107 (ObjNum 589 snap 181 seq_num 589) dirty exists 2021-03-15T22:26:01.379 INFO:tasks.rados.rados.0.smithi076.stdout:1663: left oid 21 (ObjNum 589 snap 181 seq_num 589) 2021-03-15T22:26:01.379 INFO:tasks.rados.rados.0.smithi076.stdout:1664: finishing write tid 1 to smithi07694878-18 2021-03-15T22:26:01.380 INFO:tasks.rados.rados.0.smithi076.stdout:1664: finishing write tid 2 to smithi07694878-18 2021-03-15T22:26:01.380 INFO:tasks.rados.rados.0.smithi076.stdout:1664: finishing write tid 3 to smithi07694878-18 2021-03-15T22:26:01.381 INFO:tasks.rados.rados.0.smithi076.stdout:1664: finishing write tid 4 to smithi07694878-18 2021-03-15T22:26:01.381 INFO:tasks.rados.rados.0.smithi076.stdout:1664: finishing write tid 5 to smithi07694878-18 2021-03-15T22:26:01.381 INFO:tasks.rados.rados.0.smithi076.stdout:1664: finishing write tid 6 to smithi07694878-18 2021-03-15T22:26:01.382 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 794 (ObjNum 590 snap 181 seq_num 590) dirty exists 2021-03-15T22:26:01.384 INFO:tasks.rados.rados.0.smithi076.stdout:1664: left oid 18 (ObjNum 590 snap 181 seq_num 590) 2021-03-15T22:26:01.384 INFO:tasks.rados.rados.0.smithi076.stdout:1665: finishing write tid 1 to smithi07694878-7 2021-03-15T22:26:01.384 INFO:tasks.rados.rados.0.smithi076.stdout:1665: finishing write tid 2 to smithi07694878-7 2021-03-15T22:26:01.385 INFO:tasks.rados.rados.0.smithi076.stdout:1662: done (3 left) 2021-03-15T22:26:01.385 INFO:tasks.rados.rados.0.smithi076.stdout:1663: done (2 left) 2021-03-15T22:26:01.386 INFO:tasks.rados.rados.0.smithi076.stdout:1664: done (1 left) 2021-03-15T22:26:01.386 INFO:tasks.rados.rados.0.smithi076.stdout:1666: rollback oid 21 current snap is 181 2021-03-15T22:26:01.386 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 21 to 163 2021-03-15T22:26:01.387 INFO:tasks.rados.rados.0.smithi076.stdout:1667: read oid 29 snap -1 2021-03-15T22:26:01.387 INFO:tasks.rados.rados.0.smithi076.stdout:1667: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:26:01.387 INFO:tasks.rados.rados.0.smithi076.stdout:1668: read oid 27 snap -1 2021-03-15T22:26:01.388 INFO:tasks.rados.rados.0.smithi076.stdout:1668: expect deleted 2021-03-15T22:26:01.388 INFO:tasks.rados.rados.0.smithi076.stdout:1669: snap_remove snap 143 2021-03-15T22:26:01.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:01.426 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:01.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:01.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[56514]: cluster 2021-03-15T22:26:00.261785+0000 osd.4 (osd.4) 94 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:01.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[56514]: cluster 2021-03-15T22:26:00.262438+0000 osd.4 (osd.4) 95 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:01.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[56514]: audit 2021-03-15T22:26:01.091751+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:26:01.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[56514]: audit 2021-03-15T22:26:01.094183+0000 mon.a (mon.0) 1286 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:26:01.488 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[62156]: cluster 2021-03-15T22:26:00.261785+0000 2021-03-15T22:26:01.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[62156]: osd.4 (osd.4) 94 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:01.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[62156]: cluster 2021-03-15T22:26:00. 2021-03-15T22:26:01.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[62156]: 262438+0000 osd.4 (osd.4) 95 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:01.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:26:01.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[62156]: T22:26:01.091751+0000 mon.a (mon. 2021-03-15T22:26:01.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[62156]: 0) 1285 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:26:01.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:26:01.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[62156]: -15T22:26:01.094183+0000 mon.a ( 2021-03-15T22:26:01.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:01 smithi076 conmon[62156]: mon.0) 1286 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:26:01.521 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:01.528 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:01.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:01 smithi125 conmon[56739]: cluster 2021-03-15T22:26:00.261785+0000 2021-03-15T22:26:01.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:01 smithi125 conmon[56739]: osd.4 (osd.4) 94 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:01.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:01 smithi125 conmon[56739]: cluster 2021-03-15T22:26:00.262438+0000 osd.4 (osd.4) 95 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:01.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:01 smithi125 conmon[56739]: audit 2021-03-15T22:26:01.091751+0000 mon.a (mon.0) 1285 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:26:01.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:01 smithi125 conmon[56739]: audit 2021-03-15T22:26:01.094183+0000 mon.a (mon.0) 1286 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:26:01.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:01.583 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:01.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:01.685 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:01.731 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:01.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:01.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:01.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:01.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:01.868 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:01.868 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:01.969 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:01.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:01.999 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:02.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:02.076 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:26:02.077 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:26:02.077 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:26:02.077 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:26:02.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:02.129 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:02.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:02.130 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:02.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:02.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:02.255 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:02.255 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:02.278 INFO:tasks.rados.rados.0.smithi076.stdout:1669: done (4 left) 2021-03-15T22:26:02.279 INFO:tasks.rados.rados.0.smithi076.stdout:1670: read oid 1 snap -1 2021-03-15T22:26:02.279 INFO:tasks.rados.rados.0.smithi076.stdout:1670: expect (ObjNum 561 snap 177 seq_num 561) 2021-03-15T22:26:02.279 INFO:tasks.rados.rados.0.smithi076.stdout:1671: snap_remove snap 132 2021-03-15T22:26:02.287 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:26:02.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:02.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:02.379 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:02.379 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:02.481 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:02.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:02.503 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:02.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:02.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[56514]: cluster 2021-03-15T22:26:01.061972+0000 mgr.y 2021-03-15T22:26:02.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[56514]: (mgr.14144) 648 : cluster [DBG] pgmap v795: 138 pgs: 138 active+clean; 413 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 24 op/s 2021-03-15T22:26:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[56514]: cluster 2021-03-15T22:26:01.269773+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2021-03-15T22:26:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[56514]: audit 2021-03-15T22:26:01.556533+0000 mon.a (mon.0) 1288 : audit [INF] from='client.? 172.21.15.76:0/2695799717' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:26:02.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[62156]: cluster 2021-03-15T22:26:01 2021-03-15T22:26:02.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[62156]: .061972+0000 mgr.y (mgr.14144) 648 : cluster [DBG] pgmap v795: 138 pgs: 138 active+clean; 413 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 24 op/s 2021-03-15T22:26:02.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:26:02.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[62156]: 22:26:01.269773+0000 mon.a (mon.0) 1287 : cluster 2021-03-15T22:26:02.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[62156]: [DBG] osdmap e446: 8 total, 8 up, 7 in 2021-03-15T22:26:02.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:26:02.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[62156]: :26:01.556533+0000 mon.a (mon.0) 2021-03-15T22:26:02.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:02 smithi076 conmon[62156]: 1288 : audit [INF] from='client.? 172.21.15.76:0/2695799717' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:26:02.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:02 smithi125 conmon[56739]: cluster 2021-03-15T22:26:01.061972+0000 mgr.y (mgr.14144) 648 : cluster [DBG] pgmap v795: 138 pgs: 138 active+clean; 413 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 24 op/s 2021-03-15T22:26:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:02 smithi125 conmon[56739]: cluster 2021-03-15T22:26:01.269773+0000 mon.a (mon.0) 1287 : cluster [DBG] osdmap e446: 8 total, 8 up, 7 in 2021-03-15T22:26:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:02 smithi125 conmon[56739]: audit 2021-03-15T22:26:01.556533+0000 mon.a (mon.0) 1288 : audit [INF] from='client.? 172.21.15.76:0/2695799717' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:26:02.605 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:02.627 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:02.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:02.629 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:02.730 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:02.752 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:02.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:02.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:02.779 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-15T22:26:02.855 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:02.884 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:02.885 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:02.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:03.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:03.010 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:03.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:03.096 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:26:03.112 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:03.134 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:03.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:03.136 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:03.237 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:03.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:03.263 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:03.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:03.282 INFO:tasks.rados.rados.0.smithi076.stdout:1671: done (5 left) 2021-03-15T22:26:03.283 INFO:tasks.rados.rados.0.smithi076.stdout:1672: read oid 39 snap 170 2021-03-15T22:26:03.283 INFO:tasks.rados.rados.0.smithi076.stdout:1672: expect deleted 2021-03-15T22:26:03.283 INFO:tasks.rados.rados.0.smithi076.stdout:1673: rollback oid 17 current snap is 181 2021-03-15T22:26:03.283 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 17 to 153 2021-03-15T22:26:03.284 INFO:tasks.rados.rados.0.smithi076.stdout:1674: write oid 41 current snap is 181 2021-03-15T22:26:03.284 INFO:tasks.rados.rados.0.smithi076.stdout:1674: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2021-03-15T22:26:03.291 INFO:tasks.rados.rados.0.smithi076.stdout:1674: writing smithi07694878-41 from 749744 to 1456333 tid 1 2021-03-15T22:26:03.297 INFO:tasks.rados.rados.0.smithi076.stdout:1674: writing smithi07694878-41 from 1909340 to 2486539 tid 2 2021-03-15T22:26:03.304 INFO:tasks.rados.rados.0.smithi076.stdout:1674: writing smithi07694878-41 from 3224294 to 3706855 tid 3 2021-03-15T22:26:03.305 INFO:tasks.rados.rados.0.smithi076.stdout:1675: snap_remove snap 159 2021-03-15T22:26:03.365 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:03.386 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:03.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:03.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:03.489 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:03.511 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:03.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:03.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:03.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:03 smithi125 conmon[56739]: cluster 2021-03-15T22:26:02.267875+0000 mon.a ( 2021-03-15T22:26:03.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:03 smithi125 conmon[56739]: mon.0) 1289 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:03.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:03 smithi125 conmon[56739]: audit 2021-03-15T22:26:02.276023+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? 172.21.15.76:0/2695799717' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:26:03.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:03 smithi125 conmon[56739]: cluster 2021-03-15T22:26:02.276079+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2021-03-15T22:26:03.525 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:03 smithi125 conmon[56739]: cluster 2021-03-15T22:26:02.839934+0000 osd.5 (osd.5) 77 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:26:03.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:03.636 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:03.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:03.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:03.670 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[56514]: cluster 2021-03-15T22:26:02 2021-03-15T22:26:03.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[56514]: .267875+0000 mon.a (mon.0) 1289 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:03.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[56514]: audit 2021-03-15T22:26:02.276023+0000 mon.a (mon.0) 1290 : audit [INF] from='client.? 172.21.15.76:0/2695799717' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:26:03.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[56514]: cluster 2021-03-15T22:26:02.276079+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2021-03-15T22:26:03.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[56514]: cluster 2021-03-15T22:26:02.839934+0000 osd.5 (osd.5) 77 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:26:03.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[62156]: cluster 2021-03-15T22:26: 2021-03-15T22:26:03.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[62156]: 02.267875+0000 mon.a (mon.0) 1289 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:03.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:26:03.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[62156]: 22:26:02.276023+0000 mon.a (mon.0 2021-03-15T22:26:03.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[62156]: ) 1290 : audit [INF] from='client.? 172.21.15.76:0/2695799717' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:26:03.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[62156]: cluster 2021-03-15T22:26:02. 2021-03-15T22:26:03.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[62156]: 276079+0000 mon.a (mon.0) 1291 : cluster [DBG] osdmap e447: 8 total, 8 up, 7 in 2021-03-15T22:26:03.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[62156]: cluster 2021-03-15T22:26: 2021-03-15T22:26:03.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[62156]: 02.839934+0000 osd.5 (osd.5) 77 : 2021-03-15T22:26:03.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:03 smithi076 conmon[62156]: cluster [DBG] 3.16 scrub ok 2021-03-15T22:26:03.738 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:03.761 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:03.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:03.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:03.866 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:03.888 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:03.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:03.889 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:03.991 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:04.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:04.014 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:04.014 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:04.115 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:04.139 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:04.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:04.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:04.241 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:04.253 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:26:04.256 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:26:04.265 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:04.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:04.266 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:04.281 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:26:04.300 INFO:tasks.rados.rados.0.smithi076.stdout:1675: done (8 left) 2021-03-15T22:26:04.301 INFO:tasks.rados.rados.0.smithi076.stdout:1676: snap_create 2021-03-15T22:26:04.301 INFO:tasks.rados.rados.0.smithi076.stdout:1665: finishing write tid 3 to smithi07694878-7 2021-03-15T22:26:04.301 INFO:tasks.rados.rados.0.smithi076.stdout:1665: finishing write tid 4 to smithi07694878-7 2021-03-15T22:26:04.302 INFO:tasks.rados.rados.0.smithi076.stdout:1665: finishing write tid 5 to smithi07694878-7 2021-03-15T22:26:04.302 INFO:tasks.rados.rados.0.smithi076.stdout:1665: finishing write tid 6 to smithi07694878-7 2021-03-15T22:26:04.302 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 980 (ObjNum 591 snap 181 seq_num 591) dirty exists 2021-03-15T22:26:04.302 INFO:tasks.rados.rados.0.smithi076.stdout:1665: left oid 7 (ObjNum 591 snap 181 seq_num 591) 2021-03-15T22:26:04.302 INFO:tasks.rados.rados.0.smithi076.stdout:1666: finishing rollback tid 0 to smithi07694878-21 2021-03-15T22:26:04.303 INFO:tasks.rados.rados.0.smithi076.stdout:1666: finishing rollback tid 1 to smithi07694878-21 2021-03-15T22:26:04.303 INFO:tasks.rados.rados.0.smithi076.stdout:1666: finishing rollback tid 2 to smithi07694878-21 2021-03-15T22:26:04.303 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 1112 (ObjNum 499 snap 150 seq_num 499) dirty exists 2021-03-15T22:26:04.303 INFO:tasks.rados.rados.0.smithi076.stdout:1667: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:26:04.316 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:26:04.350 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:26:04.367 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:04.368 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:26:04.375 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: cluster 2021-03-15T22:26:02.720787+0000 osd.1 (osd.1) 2021-03-15T22:26:04.376 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: 86 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:26:04.376 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: cluster 2021-03-15T22:26:02.781470+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:26:04.376 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: cluster 2021-03-15T22:26:03.062559+0000 mgr.y (mgr.14144) 649 : cluster [DBG] pgmap v798: 138 pgs: 138 active+clean; 404 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.5 MiB/s rd, 4.5 MiB/s wr, 26 op/s 2021-03-15T22:26:04.376 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: cluster 2021-03-15T22:26:03.105441+0000 osd.7 (osd.7) 87 : 2021-03-15T22:26:04.377 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: cluster [DBG] 3.e scrub ok 2021-03-15T22:26:04.377 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: cluster 2021-03-15T22:26:03. 2021-03-15T22:26:04.377 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: 280140+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2021-03-15T22:26:04.377 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: cluster 2021-03-15T22:26:04.377 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: 2021-03-15T22:26 2021-03-15T22:26:04.378 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: :03.754848+0000 osd.2 (osd.2 2021-03-15T22:26:04.378 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: ) 74 : cluster [DBG] 6.f scrub ok 2021-03-15T22:26:04.378 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:26:04.378 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: -03-15T22:26:03. 2021-03-15T22:26:04.379 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: 807783+0000 osd.5 (osd.5) 78 : 2021-03-15T22:26:04.379 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: cluster [DBG] 6.7 scrub ok 2021-03-15T22:26:04.379 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:26:04.379 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: -15T22:26:04.219008+0000 2021-03-15T22:26:04.379 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: mon.a (mon.0) 1293 : audit [DBG] 2021-03-15T22:26:04.380 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:04 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1162313840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:04.384 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:26:04.390 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:04.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:04.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:26:04.404 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:26:04.404 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:26:04.404 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:26:04.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:26:04.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:26:04.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:26:04.406 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:26:04.406 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:26:04.464 INFO:tasks.rados.rados.0.smithi076.stdout:1670: expect (ObjNum 561 snap 177 seq_num 561) 2021-03-15T22:26:04.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[56514]: cluster 2021-03-15T22:26:02.720787+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:26:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[56514]: cluster 2021-03-15T22:26:02.781470+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:26:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[56514]: cluster 2021-03-15T22:26:03.062559+0000 mgr.y (mgr.14144) 649 : cluster [DBG] pgmap v798: 138 pgs: 138 active+clean; 404 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.5 MiB/s rd, 4.5 MiB/s wr, 26 op/s 2021-03-15T22:26:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[56514]: cluster 2021-03-15T22:26:03.105441+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.e scrub ok 2021-03-15T22:26:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[56514]: cluster 2021-03-15T22:26:03.280140+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2021-03-15T22:26:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[56514]: cluster 2021-03-15T22:26:03.754848+0000 osd.2 (osd.2) 74 : cluster [DBG] 6.f scrub ok 2021-03-15T22:26:04.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[56514]: cluster 2021-03-15T22:26:03.807783+0000 osd.5 (osd.5) 78 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:26:04.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[56514]: audit 2021-03-15T22:26:04.219008+0000 mon.a (mon.0) 1293 : audit [DBG] from='client.? 172.21.15.76:0/1162313840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:04.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[62156]: cluster 2021-03-15T22:26:02.720787+0000 osd.1 (osd.1) 86 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:26:04.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[62156]: cluster 2021-03-15T22:26:02.781470+0000 osd.6 (osd.6) 69 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:26:04.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[62156]: cluster 2021-03-15T22:26:03.062559+0000 mgr.y (mgr.14144) 649 : cluster [DBG] pgmap v798: 138 pgs: 138 active+clean; 404 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.5 MiB/s rd, 4.5 MiB/s wr, 26 op/s 2021-03-15T22:26:04.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[62156]: cluster 2021-03-15T22:26:03.105441+0000 osd.7 (osd.7) 87 : cluster [DBG] 3.e scrub ok 2021-03-15T22:26:04.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[62156]: cluster 2021-03-15T22:26:03.280140+0000 mon.a (mon.0) 1292 : cluster [DBG] osdmap e448: 8 total, 8 up, 7 in 2021-03-15T22:26:04.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[62156]: cluster 2021-03-15T22:26:04.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[62156]: 2021-03-15T22:26:03.754848+0000 osd.2 (osd.2) 74 : cluster [DBG] 6.f scrub ok 2021-03-15T22:26:04.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[62156]: cluster 2021-03-15T22:26:03.807783+0000 osd.5 (osd.5) 78 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:26:04.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:04 smithi076 conmon[62156]: audit 2021-03-15T22:26:04.219008+0000 mon.a (mon.0) 1293 : audit [DBG] from='client.? 172.21.15.76:0/1162313840' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:04.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:04.523 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:04.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:04.559 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:26:04.559 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:26:04.581 INFO:tasks.rados.rados.0.smithi076.stdout:1673: finishing rollback tid 0 to smithi07694878-17 2021-03-15T22:26:04.581 INFO:tasks.rados.rados.0.smithi076.stdout:1673: finishing rollback tid 1 to smithi07694878-17 2021-03-15T22:26:04.581 INFO:tasks.rados.rados.0.smithi076.stdout:1673: finishing rollback tid 2 to smithi07694878-17 2021-03-15T22:26:04.582 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 318 (ObjNum 506 snap 151 seq_num 506) dirty exists 2021-03-15T22:26:04.582 INFO:tasks.rados.rados.0.smithi076.stdout:1674: finishing write tid 1 to smithi07694878-41 2021-03-15T22:26:04.582 INFO:tasks.rados.rados.0.smithi076.stdout:1674: finishing write tid 2 to smithi07694878-41 2021-03-15T22:26:04.582 INFO:tasks.rados.rados.0.smithi076.stdout:1674: finishing write tid 3 to smithi07694878-41 2021-03-15T22:26:04.583 INFO:tasks.rados.rados.0.smithi076.stdout:1674: finishing write tid 4 to smithi07694878-41 2021-03-15T22:26:04.583 INFO:tasks.rados.rados.0.smithi076.stdout:1674: finishing write tid 5 to smithi07694878-41 2021-03-15T22:26:04.584 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 778 (ObjNum 592 snap 181 seq_num 592) dirty exists 2021-03-15T22:26:04.584 INFO:tasks.rados.rados.0.smithi076.stdout:1674: left oid 41 (ObjNum 592 snap 181 seq_num 592) 2021-03-15T22:26:04.625 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:04.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:04.649 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:04.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:04.751 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:04.773 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:04.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:04.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:04.876 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:04.897 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:04.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:04.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:05.000 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:05.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:05.025 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:05.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:05.115 INFO:tasks.rados.rados.0.smithi076.stdout:1665: done (8 left) 2021-03-15T22:26:05.115 INFO:tasks.rados.rados.0.smithi076.stdout:1666: done (7 left) 2021-03-15T22:26:05.116 INFO:tasks.rados.rados.0.smithi076.stdout:1667: done (6 left) 2021-03-15T22:26:05.116 INFO:tasks.rados.rados.0.smithi076.stdout:1668: done (5 left) 2021-03-15T22:26:05.116 INFO:tasks.rados.rados.0.smithi076.stdout:1670: done (4 left) 2021-03-15T22:26:05.116 INFO:tasks.rados.rados.0.smithi076.stdout:1672: done (3 left) 2021-03-15T22:26:05.117 INFO:tasks.rados.rados.0.smithi076.stdout:1673: done (2 left) 2021-03-15T22:26:05.117 INFO:tasks.rados.rados.0.smithi076.stdout:1674: done (1 left) 2021-03-15T22:26:05.117 INFO:tasks.rados.rados.0.smithi076.stdout:1676: done (0 left) 2021-03-15T22:26:05.118 INFO:tasks.rados.rados.0.smithi076.stdout:1677: rollback oid 38 current snap is 182 2021-03-15T22:26:05.118 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 38 to 174 2021-03-15T22:26:05.118 INFO:tasks.rados.rados.0.smithi076.stdout:1678: snap_create 2021-03-15T22:26:05.121 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.1 primary-affinity to 0.999251 (8654862) 2021-03-15T22:26:05.122 INFO:tasks.rados.rados.0.smithi076.stdout:1677: finishing rollback tid 1 to smithi07694878-38 2021-03-15T22:26:05.122 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 469 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:26:05.126 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:05.149 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:05.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:05.150 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:05.229 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-15T22:26:05.251 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:05.274 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:05.275 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:05.376 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:05.400 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:05.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:05.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:05.505 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:26:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[56514]: cluster 2021-03-15T22:26:03.831366+0000 osd.6 (osd. 2021-03-15T22:26:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[56514]: 6) 70 : cluster [DBG] 3.a scrub ok 2021-03-15T22:26:05.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[56514]: cluster 2021-03-15T22:26:04.286130+0000 mon.a (mon.0) 1294 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2021-03-15T22:26:05.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[56514]: audit 2021-03-15T22:26:04.479157+0000 mon.c (mon.1) 39 : audit [INF] from='client.? 172.21.15.76:0/1990657205' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9992505298965398}]: dispatch 2021-03-15T22:26:05.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[56514]: audit 2021-03-15T22:26:04.479576+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9992505298965398}]: dispatch 2021-03-15T22:26:05.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[56514]: cluster 2021-03-15T22:26:04.729160+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:26:05.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[56514]: audit 2021-03-15T22:26:05.110411+0000 mon.a (mon.0) 1296 : audit 2021-03-15T22:26:05.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[56514]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9992505298965398}]': finished 2021-03-15T22:26:05.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[56514]: cluster 2021-03-15T22:26:05.110716+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2021-03-15T22:26:05.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:26:05.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: 26:03.831366+0000 osd.6 (osd.6) 70 : cluster [DBG] 3.a scrub ok 2021-03-15T22:26:05.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:26:05.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: 15T22:26:04.286130+0000 mon.a (mon.0) 2021-03-15T22:26:05.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: 1294 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2021-03-15T22:26:05.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:26:05.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: 15T22:26:04.479157+0000 mon.c (mon.1) 39 : audit 2021-03-15T22:26:05.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/1990657205' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9992505298965398}]: dispatch 2021-03-15T22:26:05.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: audit 2021-03-15T22:26:05.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: 2021-03-15T22:26:04.479576+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9992505298965398}]: dispatch 2021-03-15T22:26:05.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: cluster 2021-03-15T22:26:04. 2021-03-15T22:26:05.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: 729160+0000 osd.3 (osd.3) 77 : cluster [DBG] 2021-03-15T22:26:05.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: 6.6 scrub ok 2021-03-15T22:26:05.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: audit 2021-03-15T22:26: 2021-03-15T22:26:05.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: 05.110411+0000 mon.a (mon.0) 1296 : 2021-03-15T22:26:05.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9992505298965398}]': finished 2021-03-15T22:26:05.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: cluster 2021-03-15T22:26:05.110716+0000 mon.a ( 2021-03-15T22:26:05.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:05 smithi076 conmon[62156]: mon.0) 1297 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2021-03-15T22:26:05.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:05.526 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:05.526 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:05.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:05 smithi125 conmon[56739]: cluster 2021-03-15T22:26:03.831366+0000 osd.6 (osd.6) 70 : cluster [DBG] 3.a scrub ok 2021-03-15T22:26:05.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:05 smithi125 conmon[56739]: cluster 2021-03-15T22:26:04.286130+0000 mon.a (mon.0) 1294 : cluster [DBG] osdmap e449: 8 total, 8 up, 7 in 2021-03-15T22:26:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:05 smithi125 conmon[56739]: audit 2021-03-15T22:26:04.479157+0000 mon.c (mon.1) 39 : audit [INF] from='client.? 172.21.15.76:0/1990657205' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9992505298965398}]: dispatch 2021-03-15T22:26:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:05 smithi125 conmon[56739]: audit 2021-03-15T22:26:04.479576+0000 mon.a (mon.0) 1295 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9992505298965398}]: dispatch 2021-03-15T22:26:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:05 smithi125 conmon[56739]: cluster 2021-03-15T22:26:04.729160+0000 osd.3 (osd.3) 77 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:26:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:05 smithi125 conmon[56739]: audit 2021-03-15T22:26:05.110411+0000 mon.a (mon.0) 1296 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 0.9992505298965398}]': finished 2021-03-15T22:26:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:05 smithi125 conmon[56739]: cluster 2021-03-15T22:26:05.110716+0000 mon.a (mon.0) 1297 : cluster [DBG] osdmap e450: 8 total, 8 up, 7 in 2021-03-15T22:26:05.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:05.650 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:05.651 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:05.752 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:05.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:05.776 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:05.777 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:05.855 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:26:05.877 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:05.921 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:05.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:05.924 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:06.024 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:06.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:06.048 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:06.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:06.125 INFO:tasks.rados.rados.0.smithi076.stdout:1677: done (1 left) 2021-03-15T22:26:06.125 INFO:tasks.rados.rados.0.smithi076.stdout:1678: done (0 left) 2021-03-15T22:26:06.125 INFO:tasks.rados.rados.0.smithi076.stdout:1679: rollback oid 12 current snap is 183 2021-03-15T22:26:06.125 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 12 to 154 2021-03-15T22:26:06.126 INFO:tasks.rados.rados.0.smithi076.stdout:1680: read oid 4 snap -1 2021-03-15T22:26:06.126 INFO:tasks.rados.rados.0.smithi076.stdout:1680: expect deleted 2021-03-15T22:26:06.126 INFO:tasks.rados.rados.0.smithi076.stdout:1681: rollback oid 46 current snap is 183 2021-03-15T22:26:06.126 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 46 to 179 2021-03-15T22:26:06.127 INFO:tasks.rados.rados.0.smithi076.stdout:1682: copy_from oid 7 from oid 18 current snap is 183 2021-03-15T22:26:06.127 INFO:tasks.rados.rados.0.smithi076.stdout:1683: write oid 14 current snap is 183 2021-03-15T22:26:06.127 INFO:tasks.rados.rados.0.smithi076.stdout:1683: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2021-03-15T22:26:06.132 INFO:tasks.rados.rados.0.smithi076.stdout:1683: writing smithi07694878-14 from 766551 to 1264742 tid 1 2021-03-15T22:26:06.139 INFO:tasks.rados.rados.0.smithi076.stdout:1683: writing smithi07694878-14 from 1684175 to 2405032 tid 2 2021-03-15T22:26:06.146 INFO:tasks.rados.rados.0.smithi076.stdout:1683: writing smithi07694878-14 from 2851717 to 3545550 tid 3 2021-03-15T22:26:06.146 INFO:tasks.rados.rados.0.smithi076.stdout:1684: write oid 1 current snap is 183 2021-03-15T22:26:06.147 INFO:tasks.rados.rados.0.smithi076.stdout:1684: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2021-03-15T22:26:06.147 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:06.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:06.153 INFO:tasks.rados.rados.0.smithi076.stdout:1684: writing smithi07694878-1 from 783358 to 1356798 tid 1 2021-03-15T22:26:06.161 INFO:tasks.rados.rados.0.smithi076.stdout:1684: writing smithi07694878-1 from 2026304 to 2607172 tid 2 2021-03-15T22:26:06.169 INFO:tasks.rados.rados.0.smithi076.stdout:1684: writing smithi07694878-1 from 3279140 to 3900598 tid 3 2021-03-15T22:26:06.169 INFO:tasks.rados.rados.0.smithi076.stdout:1685: snap_remove snap 161 2021-03-15T22:26:06.173 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:06.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:06.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:06.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:06.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:06.301 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:06.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:06.403 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:06.425 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:06.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:06.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:06.528 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:06.551 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:06.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:06.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:06.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[56514]: cluster 2021-03-15T22:26:05.046609+0000 osd.7 2021-03-15T22:26:06.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[56514]: (osd.7) 88 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:26:06.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[56514]: cluster 2021-03-15T22:26:05.063018+0000 mgr.y (mgr.14144) 650 : cluster [DBG] pgmap v801: 138 pgs: 138 active+clean; 382 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2021-03-15T22:26:06.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[56514]: cluster 2021-03-15T22:26:05.733844+0000 osd.5 (osd.5) 79 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:26:06.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[56514]: cluster 2021-03-15T22:26:06.120132+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2021-03-15T22:26:06.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[62156]: cluster 2021-03-15T22:26:05.046609+0000 osd.7 (osd 2021-03-15T22:26:06.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[62156]: .7) 88 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:26:06.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[62156]: cluster 2021-03-15T22:26:05.063018+0000 mgr.y (mgr.14144) 650 2021-03-15T22:26:06.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[62156]: : cluster [DBG] pgmap v801: 138 pgs: 138 active+clean; 382 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2021-03-15T22:26:06.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[62156]: cluster 2021-03-15T22:26:05.733844 2021-03-15T22:26:06.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 79 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:26:06.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[62156]: cluster 2021-03-15T22:26:06.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:06 smithi076 conmon[62156]: 2021-03-15T22:26:06.120132+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2021-03-15T22:26:06.653 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:06.676 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:06.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:06.678 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:06.697 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:06 smithi125 conmon[56739]: cluster 2021-03-15T22:26:05.046609+0000 osd.7 (osd.7) 88 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:26:06.697 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:06 smithi125 conmon[56739]: cluster 2021-03-15T22:26:05.063018+0000 mgr.y (mgr.14144) 650 : cluster [DBG] pgmap v801: 138 pgs: 138 active+clean; 382 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 6.4 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2021-03-15T22:26:06.698 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:06 smithi125 conmon[56739]: cluster 2021-03-15T22:26:05.733844+0000 osd.5 (osd.5) 79 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:26:06.698 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:06 smithi125 conmon[56739]: cluster 2021-03-15T22:26:06.120132+0000 mon.a (mon.0) 1298 : cluster [DBG] osdmap e451: 8 total, 8 up, 7 in 2021-03-15T22:26:06.780 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:06.805 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:06.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:06.907 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:06.919 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:26:06.920 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:26:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:06.932 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:06.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:07.034 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:07.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:07.060 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:07.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:07.121 INFO:tasks.rados.rados.0.smithi076.stdout:1685: done (6 left) 2021-03-15T22:26:07.121 INFO:tasks.rados.rados.0.smithi076.stdout:1686: delete oid 19 current snap is 183 2021-03-15T22:26:07.122 INFO:tasks.rados.rados.0.smithi076.stdout:1679: finishing rollback tid 0 to smithi07694878-12 2021-03-15T22:26:07.122 INFO:tasks.rados.rados.0.smithi076.stdout:1681: finishing rollback tid 1 to smithi07694878-46 2021-03-15T22:26:07.122 INFO:tasks.rados.rados.0.smithi076.stdout:1681: finishing rollback tid 2 to smithi07694878-46 2021-03-15T22:26:07.122 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 829 (ObjNum 496 snap 149 seq_num 496) dirty exists 2021-03-15T22:26:07.122 INFO:tasks.rados.rados.0.smithi076.stdout:1679: finishing rollback tid 1 to smithi07694878-12 2021-03-15T22:26:07.123 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 1050 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:26:07.123 INFO:tasks.rados.rados.0.smithi076.stdout:1683: finishing write tid 1 to smithi07694878-14 2021-03-15T22:26:07.123 INFO:tasks.rados.rados.0.smithi076.stdout:1682: finishing copy_from to smithi07694878-7 2021-03-15T22:26:07.123 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 794 (ObjNum 590 snap 181 seq_num 590) dirty exists 2021-03-15T22:26:07.124 INFO:tasks.rados.rados.0.smithi076.stdout:1682: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:26:07.124 INFO:tasks.rados.rados.0.smithi076.stdout:1683: finishing write tid 2 to smithi07694878-14 2021-03-15T22:26:07.124 INFO:tasks.rados.rados.0.smithi076.stdout:1683: finishing write tid 3 to smithi07694878-14 2021-03-15T22:26:07.124 INFO:tasks.rados.rados.0.smithi076.stdout:1683: finishing write tid 4 to smithi07694878-14 2021-03-15T22:26:07.124 INFO:tasks.rados.rados.0.smithi076.stdout:1683: finishing write tid 5 to smithi07694878-14 2021-03-15T22:26:07.125 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 771 (ObjNum 593 snap 183 seq_num 593) dirty exists 2021-03-15T22:26:07.125 INFO:tasks.rados.rados.0.smithi076.stdout:1683: left oid 14 (ObjNum 593 snap 183 seq_num 593) 2021-03-15T22:26:07.125 INFO:tasks.rados.rados.0.smithi076.stdout:1684: finishing write tid 1 to smithi07694878-1 2021-03-15T22:26:07.125 INFO:tasks.rados.rados.0.smithi076.stdout:1684: finishing write tid 2 to smithi07694878-1 2021-03-15T22:26:07.126 INFO:tasks.rados.rados.0.smithi076.stdout:1684: finishing write tid 3 to smithi07694878-1 2021-03-15T22:26:07.126 INFO:tasks.rados.rados.0.smithi076.stdout:1684: finishing write tid 4 to smithi07694878-1 2021-03-15T22:26:07.126 INFO:tasks.rados.rados.0.smithi076.stdout:1684: finishing write tid 5 to smithi07694878-1 2021-03-15T22:26:07.126 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 844 (ObjNum 594 snap 183 seq_num 594) dirty exists 2021-03-15T22:26:07.126 INFO:tasks.rados.rados.0.smithi076.stdout:1684: left oid 1 (ObjNum 594 snap 183 seq_num 594) 2021-03-15T22:26:07.127 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:26:07.129 INFO:tasks.rados.rados.0.smithi076.stdout:1679: done (6 left) 2021-03-15T22:26:07.130 INFO:tasks.rados.rados.0.smithi076.stdout:1680: done (5 left) 2021-03-15T22:26:07.130 INFO:tasks.rados.rados.0.smithi076.stdout:1681: done (4 left) 2021-03-15T22:26:07.130 INFO:tasks.rados.rados.0.smithi076.stdout:1682: done (3 left) 2021-03-15T22:26:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:1683: done (2 left) 2021-03-15T22:26:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:1684: done (1 left) 2021-03-15T22:26:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:1686: done (0 left) 2021-03-15T22:26:07.131 INFO:tasks.rados.rados.0.smithi076.stdout:1687: write oid 2 current snap is 183 2021-03-15T22:26:07.132 INFO:tasks.rados.rados.0.smithi076.stdout:1687: seq_num 596 ranges {416972=723938,1743268=273704} 2021-03-15T22:26:07.134 INFO:tasks.rados.rados.0.smithi076.stdout:1687: writing smithi07694878-2 from 416972 to 1140910 tid 1 2021-03-15T22:26:07.140 INFO:tasks.rados.rados.0.smithi076.stdout:1687: writing smithi07694878-2 from 1743268 to 2016972 tid 2 2021-03-15T22:26:07.140 INFO:tasks.rados.rados.0.smithi076.stdout:1688: delete oid 50 current snap is 183 2021-03-15T22:26:07.140 INFO:tasks.rados.rados.0.smithi076.stdout:1687: finishing write tid 1 to smithi07694878-2 2021-03-15T22:26:07.141 INFO:tasks.rados.rados.0.smithi076.stdout:1688: done (1 left) 2021-03-15T22:26:07.141 INFO:tasks.rados.rados.0.smithi076.stdout:1689: snap_remove snap 146 2021-03-15T22:26:07.161 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:07.183 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:07.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:07.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:07.286 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:07.309 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:07.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:07.310 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:07.411 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:07.434 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:07.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:07.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:07.466 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:07.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:07 smithi076 conmon[56514]: cluster 2021-03-15T22:26:06 2021-03-15T22:26:07.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:07 smithi076 conmon[56514]: .763982+0000 osd.5 (osd.5) 80 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:26:07.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:07 smithi076 conmon[56514]: cluster 2021-03-15T22:26:07.119527+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2021-03-15T22:26:07.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:07 smithi076 conmon[62156]: cluster 2021-03-15T22:26:06. 2021-03-15T22:26:07.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:07 smithi076 conmon[62156]: 763982+0000 osd.5 (osd.5) 80 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:26:07.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:07 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:26:07.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:07 smithi076 conmon[62156]: 26:07.119527+0000 mon.a (mon.0) 1299 2021-03-15T22:26:07.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:07 smithi076 conmon[62156]: : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2021-03-15T22:26:07.538 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:07.560 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:07.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:07.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:07.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:07 smithi125 conmon[56739]: cluster 2021-03-15T22:26:06.763982+0000 osd.5 (osd.5) 80 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:26:07.564 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:07 smithi125 conmon[56739]: cluster 2021-03-15T22:26:07.119527+0000 mon.a (mon.0) 1299 : cluster [DBG] osdmap e452: 8 total, 8 up, 7 in 2021-03-15T22:26:07.588 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:26:07.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:07.686 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:07.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:07.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:07.789 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:07.811 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:07.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:07.812 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:07.914 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:07.936 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:07.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:07.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:08.039 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:08.063 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:08.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:08.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:08.097 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:08.126 INFO:tasks.rados.rados.0.smithi076.stdout:1689: done (1 left) 2021-03-15T22:26:08.127 INFO:tasks.rados.rados.0.smithi076.stdout:1690: read oid 16 snap -1 2021-03-15T22:26:08.127 INFO:tasks.rados.rados.0.smithi076.stdout:1690: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:26:08.131 INFO:tasks.rados.rados.0.smithi076.stdout:1691: write oid 50 current snap is 183 2021-03-15T22:26:08.131 INFO:tasks.rados.rados.0.smithi076.stdout:1691: seq_num 598 ranges {450586=474436,1343879=588206} 2021-03-15T22:26:08.132 INFO:tasks.rados.rados.0.smithi076.stdout:1691: writing smithi07694878-50 from 450586 to 925022 tid 1 2021-03-15T22:26:08.139 INFO:tasks.rados.rados.0.smithi076.stdout:1691: writing smithi07694878-50 from 1343879 to 1932085 tid 2 2021-03-15T22:26:08.142 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:26:08.142 INFO:tasks.rados.rados.0.smithi076.stdout:1687: finishing write tid 2 to smithi07694878-2 2021-03-15T22:26:08.142 INFO:tasks.rados.rados.0.smithi076.stdout:1687: finishing write tid 3 to smithi07694878-2 2021-03-15T22:26:08.143 INFO:tasks.rados.rados.0.smithi076.stdout:1687: finishing write tid 4 to smithi07694878-2 2021-03-15T22:26:08.143 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 440 (ObjNum 596 snap 183 seq_num 596) dirty exists 2021-03-15T22:26:08.143 INFO:tasks.rados.rados.0.smithi076.stdout:1687: left oid 2 (ObjNum 596 snap 183 seq_num 596) 2021-03-15T22:26:08.143 INFO:tasks.rados.rados.0.smithi076.stdout:1690: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:26:08.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:08.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:08.189 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:08.189 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:08.292 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:08.305 INFO:tasks.rados.rados.0.smithi076.stdout:1687: done (2 left) 2021-03-15T22:26:08.305 INFO:tasks.rados.rados.0.smithi076.stdout:1690: done (1 left) 2021-03-15T22:26:08.306 INFO:tasks.rados.rados.0.smithi076.stdout:1692: snap_remove snap 178 2021-03-15T22:26:08.314 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:08.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:08.418 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:08.440 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:08.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:08.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:08.517 INFO:journalctl@ceph.osd.3.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[75764]: debug 2021-03-15T22:26:08.138+0000 7f9fbefb7700 -1 osd.3 pg_epoch: 453 pg[7.a( v 453'644 (0'0,453'644] local-lis/les=268/269 n=17 ec=91/91 lis/c=268/268 les/c/f=269/269/0 sis=268) [3,2,5] r=0 lpr=268 luod=452'642 crt=453'644 lcod 452'640 mlcod 452'640 active+clean+snaptrim trimq=[117~1] ps=[e7~1,100~1,114~1,12e~1,130~1,14b~1]] removing snap head 2021-03-15T22:26:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: cluster 2021-03-15T22:26:06.698711+0000 osd.1 ( 2021-03-15T22:26:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: osd.1) 87 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:26:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: cluster 2021-03-15T22:26:06.865253+0000 osd.6 (osd.6) 71 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:26:08.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: cluster 2021-03-15T22:26:07.054494+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:26:08.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: cluster 2021-03-15T22:26:07.063608+0000 mgr.y (mgr.14144) 651 : cluster [DBG] pgmap v804: 138 pgs: 138 active+clean; 383 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-03-15T22:26:08.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: cluster 2021-03-15T22:26: 2021-03-15T22:26:08.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: 07.787519+0000 osd.5 (osd.5) 81 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:26:08.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: audit 2021-03-15T 2021-03-15T22:26:08.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: 22:26:07.954883+0000 mon.a (mon.0) 1300 : audit [INF] 2021-03-15T22:26:08.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/4047227469' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:26:08.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:26:08.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: :26:08.117083+0000 mon.a (mon.0 2021-03-15T22:26:08.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: ) 1301 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:26:08.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:26:08.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: -03-15T22:26:08.117121+0000 2021-03-15T22:26:08.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: mon.a (mon.0) 1302 : cluster [INF] Cluster is now healthy 2021-03-15T22:26:08.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: 2021-03-15T22:26:08.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: audit 2021-03-15T22:26: 2021-03-15T22:26:08.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: 08.121859+0000 mon.a (mon.0) 1303 2021-03-15T22:26:08.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: : audit [INF] from='client.? 172.21.15.76:0/4047227469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:26:08.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: cluster 2021-03-15T22:26:08.121913 2021-03-15T22:26:08.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1304 : cluster [DBG] 2021-03-15T22:26:08.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[56514]: osdmap e453: 8 total, 8 up, 7 in 2021-03-15T22:26:08.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: cluster 2021-03-15T22:26: 2021-03-15T22:26:08.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: 06.698711+0000 osd.1 (osd.1) 87 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:26:08.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: cluster 2021-03-15T22:26:06.865253+0000 2021-03-15T22:26:08.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: osd.6 (osd.6) 71 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:26:08.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: 2021-03-15T22:26:08.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:26:08.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: -15T22:26:07. 2021-03-15T22:26:08.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: 054494+0000 osd.7 (osd.7 2021-03-15T22:26:08.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: ) 89 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:26:08.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:26:08.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: 15T22:26:07.063608+0000 2021-03-15T22:26:08.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: mgr.y (mgr.14144) 651 : cluster 2021-03-15T22:26:08.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: [DBG] pgmap v804: 138 pgs: 138 active+clean; 383 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-03-15T22:26:08.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:26:08.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: 03-15T22:26:07. 2021-03-15T22:26:08.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: 787519+0000 osd.5 (osd.5) 2021-03-15T22:26:08.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: 81 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:26:08.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:26:08.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: 03-15T22:26:07.954883+0000 2021-03-15T22:26:08.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: mon.a (mon.0) 1300 : audit [INF] 2021-03-15T22:26:08.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/4047227469' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:26:08.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: cluster 2021-03-15T22:26:08.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: 2021-03-15T22:26:08.117083+0000 2021-03-15T22:26:08.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: mon.a (mon.0) 1301 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:26:08.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:26:08.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: 03-15T22:26:08.117121+0000 mon.a 2021-03-15T22:26:08.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: (mon.0) 1302 : cluster [INF] Cluster is now healthy 2021-03-15T22:26:08.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:26:08.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: -15T22:26:08.121859+0000 mon.a ( 2021-03-15T22:26:08.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: mon.0) 1303 : audit [INF] from='client.? 172.21.15.76:0/4047227469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:26:08.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:26:08.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: 03-15T22:26:08.121913+0000 2021-03-15T22:26:08.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:08 smithi076 conmon[62156]: mon.a (mon.0) 1304 : cluster [DBG] osdmap e453: 8 total, 8 up, 7 in 2021-03-15T22:26:08.544 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:08.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:08.568 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:08.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:08.669 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:08.692 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:08.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:08.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:08.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: cluster 2021-03-15T22:26:06.698711+0000 osd.1 2021-03-15T22:26:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: (osd.1) 87 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:26:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: cluster 2021-03-15T22:26:06.865253+0000 osd.6 (osd.6) 71 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:26:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: cluster 2021-03-15T22:26:07.054494+0000 osd.7 (osd.7) 89 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:26:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: cluster 2021-03-15T22:26:07.063608+0000 mgr.y (mgr.14144) 651 : cluster [DBG] 2021-03-15T22:26:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: pgmap v804: 138 pgs: 138 active+clean; 383 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 1.8 MiB/s wr, 7 op/s 2021-03-15T22:26:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: cluster 2021-03-15T22:26:07.787519+0000 2021-03-15T22:26:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: osd.5 (osd.5) 81 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:26:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:26:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: 15T22:26:07.954883+0000 mon.a ( 2021-03-15T22:26:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: mon.0) 1300 : audit [INF] from='client.? 172.21.15.76:0/4047227469' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:26:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: cluster 2021-03-15T22:26 2021-03-15T22:26:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: :08.117083+0000 mon.a (mon.0) 2021-03-15T22:26:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: 1301 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:26:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: -15T22:26:08.117121+0000 2021-03-15T22:26:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: mon.a (mon.0) 1302 : cluster [INF] 2021-03-15T22:26:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: Cluster is now healthy 2021-03-15T22:26:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: audit 2021-03-15T22:26: 2021-03-15T22:26:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: 08.121859+0000 mon.a (mon.0) 1303 2021-03-15T22:26:08.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: : audit [INF] from='client.? 172.21.15.76:0/4047227469' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:26:08.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:26:08.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: :26:08.121913+0000 mon.a (mon. 2021-03-15T22:26:08.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:08 smithi125 conmon[56739]: 0) 1304 : cluster [DBG] osdmap e453: 8 total, 8 up, 7 in 2021-03-15T22:26:08.795 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:08.819 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:08.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:08.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:08.922 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:08.947 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:08.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:08.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:09.050 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:09.075 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:09.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:09.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:09.178 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:09.202 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:09.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:09.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:09.304 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:09.308 INFO:tasks.rados.rados.0.smithi076.stdout:1692: done (1 left) 2021-03-15T22:26:09.308 INFO:tasks.rados.rados.0.smithi076.stdout:1693: snap_create 2021-03-15T22:26:09.315 INFO:tasks.rados.rados.0.smithi076.stdout:1691: finishing write tid 1 to smithi07694878-50 2021-03-15T22:26:09.315 INFO:tasks.rados.rados.0.smithi076.stdout:1691: finishing write tid 2 to smithi07694878-50 2021-03-15T22:26:09.316 INFO:tasks.rados.rados.0.smithi076.stdout:1691: finishing write tid 3 to smithi07694878-50 2021-03-15T22:26:09.316 INFO:tasks.rados.rados.0.smithi076.stdout:1691: finishing write tid 4 to smithi07694878-50 2021-03-15T22:26:09.316 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 774 (ObjNum 598 snap 183 seq_num 598) dirty exists 2021-03-15T22:26:09.316 INFO:tasks.rados.rados.0.smithi076.stdout:1691: left oid 50 (ObjNum 598 snap 183 seq_num 598) 2021-03-15T22:26:09.337 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:09.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:09.339 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:09.440 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:09.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:09.465 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:09.466 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:09.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:09 smithi076 conmon[56514]: cluster 2021-03-15T22:26:07.656371+0000 osd.1 (osd.1) 2021-03-15T22:26:09.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:09 smithi076 conmon[56514]: 88 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:26:09.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:09 smithi076 conmon[56514]: cluster 2021-03-15T22:26:07.977638+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub ok 2021-03-15T22:26:09.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:09 smithi076 conmon[56514]: cluster 2021-03-15T22:26:08.042583+0000 osd.7 (osd 2021-03-15T22:26:09.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:09 smithi076 conmon[56514]: .7) 90 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:26:09.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:09 smithi076 conmon[62156]: cluster 2021-03-15T22:26:07.656371+0000 osd.1 ( 2021-03-15T22:26:09.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:09 smithi076 conmon[62156]: osd.1) 88 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:26:09.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:09 smithi076 conmon[62156]: cluster 2021-03-15T22:26:07.977638+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub ok 2021-03-15T22:26:09.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:09 smithi076 conmon[62156]: cluster 2021-03-15T22:26:08.042583+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:26:09.567 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:09.594 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:09.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:09.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:09.696 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:09.722 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:09.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:09.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:09.754 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:09 smithi125 conmon[56739]: cluster 2021-03-15T22:26:07.656371+0000 osd.1 (osd.1) 88 : 2021-03-15T22:26:09.754 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:09 smithi125 conmon[56739]: cluster [DBG] 3.19 scrub ok 2021-03-15T22:26:09.754 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:09 smithi125 conmon[56739]: cluster 2021-03-15T22:26:07.977638+0000 osd.4 (osd.4) 96 : cluster [DBG] 4.c scrub ok 2021-03-15T22:26:09.755 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:09 smithi125 conmon[56739]: cluster 2021-03-15T22:26:08.042583+0000 osd.7 (osd.7) 90 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:26:09.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:09.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:09.850 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:09.850 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:09.872 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:26:09.889 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:26:09.908 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:26:09.929 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:26:09.949 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:26:09.951 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:09.969 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:26:09.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:09.977 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:09.977 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:09.986 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:26:10.004 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:26:10.006 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:26:10.006 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:26:10.006 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:26:10.007 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:26:10.007 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:26:10.007 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:26:10.007 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:26:10.008 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:26:10.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:26:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:26:10.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:10.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:10.107 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:10.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:10.115 INFO:tasks.rados.rados.0.smithi076.stdout:1691: done (1 left) 2021-03-15T22:26:10.115 INFO:tasks.rados.rados.0.smithi076.stdout:1693: done (0 left) 2021-03-15T22:26:10.116 INFO:tasks.rados.rados.0.smithi076.stdout:1694: delete oid 49 current snap is 184 2021-03-15T22:26:10.125 INFO:tasks.rados.rados.0.smithi076.stdout:1694: done (0 left) 2021-03-15T22:26:10.126 INFO:tasks.rados.rados.0.smithi076.stdout:1695: write (excl) oid 30 current snap is 184 2021-03-15T22:26:10.126 INFO:tasks.rados.rados.0.smithi076.stdout:1695: seq_num 600 ranges {484200=624934,1860843=223357} 2021-03-15T22:26:10.131 INFO:tasks.rados.rados.0.smithi076.stdout:1695: writing smithi07694878-30 from 484200 to 1109134 tid 1 2021-03-15T22:26:10.137 INFO:tasks.rados.rados.0.smithi076.stdout:1695: writing smithi07694878-30 from 1860843 to 2084200 tid 2 2021-03-15T22:26:10.138 INFO:tasks.rados.rados.0.smithi076.stdout:1696: rollback oid 23 current snap is 184 2021-03-15T22:26:10.138 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 23 to 155 2021-03-15T22:26:10.139 INFO:tasks.rados.rados.0.smithi076.stdout:1697: snap_create 2021-03-15T22:26:10.209 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:10.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:10.232 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:10.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:10.251 INFO:tasks.rados.rados.0.smithi076.stdout:1696: finishing rollback tid 0 to smithi07694878-23 2021-03-15T22:26:10.252 INFO:tasks.rados.rados.0.smithi076.stdout:1696: finishing rollback tid 1 to smithi07694878-23 2021-03-15T22:26:10.253 INFO:tasks.rados.rados.0.smithi076.stdout:1696: finishing rollback tid 2 to smithi07694878-23 2021-03-15T22:26:10.253 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 797 (ObjNum 424 snap 117 seq_num 424) dirty exists 2021-03-15T22:26:10.254 INFO:tasks.rados.rados.0.smithi076.stdout:1695: finishing write tid 1 to smithi07694878-30 2021-03-15T22:26:10.257 INFO:tasks.rados.rados.0.smithi076.stdout:1695: finishing write tid 2 to smithi07694878-30 2021-03-15T22:26:10.257 INFO:tasks.rados.rados.0.smithi076.stdout:1695: finishing write tid 3 to smithi07694878-30 2021-03-15T22:26:10.257 INFO:tasks.rados.rados.0.smithi076.stdout:1695: finishing write tid 4 to smithi07694878-30 2021-03-15T22:26:10.257 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 1078 (ObjNum 600 snap 184 seq_num 600) dirty exists 2021-03-15T22:26:10.258 INFO:tasks.rados.rados.0.smithi076.stdout:1695: left oid 30 (ObjNum 600 snap 184 seq_num 600) 2021-03-15T22:26:10.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:10.358 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:10.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:10.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:10.461 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:10.485 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:10.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:10.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[62156]: cluster 2021-03-15T22:26:09.008582+0000 osd.4 (osd.4) 97 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:26:10.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[62156]: cluster 2021-03-15T22:26:09.051789+0000 osd.7 (osd.7) 91 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:26:10.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[62156]: cluster 2021-03-15T22:26:09.063957+0000 mgr.y (mgr.14144) 652 : cluster [DBG] pgmap v807: 138 pgs: 138 active+clean; 380 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 1.8 MiB/s wr, 3 op/s 2021-03-15T22:26:10.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[62156]: cluster 2021-03-15T22:26:09.304931+0000 mon.a (mon.0) 1305 : cluster [DBG] osdmap e454: 8 total, 8 up, 7 in 2021-03-15T22:26:10.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[62156]: cluster 2021-03-15T22:26:09.660541+0000 osd.2 (osd.2) 75 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:26:10.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[62156]: audit 2021-03-15T22:26:09.752587+0000 mgr.y (mgr. 2021-03-15T22:26:10.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[62156]: 14144) 653 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:10.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[62156]: audit 2021-03-15T22:26:09.853232+0000 mon.a (mon.0) 1306 : audit [DBG] from='client.? 172.21.15.76:0/1186884515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:10.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[62156]: cluster 2021-03-15T22:26:10.113233+0000 mon.a (mon.0) 1307 : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2021-03-15T22:26:10.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: cluster 2021-03-15T22:26:09.008582+0000 osd.4 (osd.4) 97 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:26:10.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: cluster 2021-03-15T22:26:10.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:26:10.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 26:09.051789+0000 osd.7 (osd.7) 91 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:26:10.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:26:10.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: -03-15T22:26:09 2021-03-15T22:26:10.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: .063957+0000 mgr.y (mgr. 2021-03-15T22:26:10.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 14144) 652 : cluster [DBG] 2021-03-15T22:26:10.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: pgmap v807: 138 pgs: 138 active+clean; 380 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 1.8 MiB/s wr, 3 op/s 2021-03-15T22:26:10.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: cluster 2021-03-15T22:26:10.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:26:10.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 26:09.304931+0000 mon.a 2021-03-15T22:26:10.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: (mon.0) 1305 : cluster 2021-03-15T22:26:10.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: [DBG] osdmap e454: 8 total, 8 up, 7 in 2021-03-15T22:26:10.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:26:10.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: -03-15T22:26: 2021-03-15T22:26:10.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 09.660541+0000 osd.2 (osd 2021-03-15T22:26:10.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: .2) 75 : cluster [DBG] 2021-03-15T22:26:10.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 4.1c scrub ok 2021-03-15T22:26:10.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: audit 2021-03-15T22:26:10.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:26:10.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 26:09.752587+0000 mgr.y ( 2021-03-15T22:26:10.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: mgr.14144) 653 : audit [DBG] 2021-03-15T22:26:10.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:10.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: audit 2021 2021-03-15T22:26:10.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: -03-15T22:26:09. 2021-03-15T22:26:10.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 853232+0000 mon.a (mon.0) 1306 2021-03-15T22:26:10.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: : audit [DBG] from='client.? 172.21.15.76:0/1186884515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:10.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:26:10.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 03-15T22:26:10. 2021-03-15T22:26:10.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 113233+0000 mon.a (mon.0) 2021-03-15T22:26:10.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:10 smithi076 conmon[56514]: 1307 : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2021-03-15T22:26:10.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:10.609 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:10.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:10.612 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:10.713 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:10.717 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:26:10.717 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:26:10.718 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2021-03-15T22:26:10.718 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2021-03-15T22:26:10.719 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:26:10.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:10.737 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:10.737 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:10.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: cluster 2021-03-15T22:26:09.008582+0000 osd.4 (osd.4) 97 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:26:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: cluster 2021-03-15T22:26:09.051789+0000 osd.7 (osd.7) 91 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:26:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: cluster 2021-03-15T22:26:09.063957+0000 mgr.y (mgr.14144) 652 : cluster [DBG] 2021-03-15T22:26:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: pgmap v807: 138 pgs: 138 active+clean; 380 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 511 B/s rd, 1.8 MiB/s wr, 3 op/s 2021-03-15T22:26:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:26:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: 26:09.304931+0000 mon.a (mon 2021-03-15T22:26:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: .0) 1305 : cluster [DBG] osdmap e454: 8 total, 8 up, 7 in 2021-03-15T22:26:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: 2021-03-15T22:26:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: cluster 2021-03-15T22:26:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: 2021-03-15T22:26 2021-03-15T22:26:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: :09.660541+0000 osd.2 (osd 2021-03-15T22:26:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: .2) 75 : cluster [DBG] 2021-03-15T22:26:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: 4.1c scrub ok 2021-03-15T22:26:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: audit 2021-03-15T22:26:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: 2021-03-15T22:26: 2021-03-15T22:26:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: 09.752587+0000 mgr.y (mgr. 2021-03-15T22:26:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: 14144) 653 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: 2021-03-15T22:26:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: audit 2021-03-15T22:26:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:26:10.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: 26:09.853232+0000 mon.a 2021-03-15T22:26:10.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: (mon.0) 1306 : audit 2021-03-15T22:26:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/1186884515' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: -15T22:26:10.113233 2021-03-15T22:26:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1307 2021-03-15T22:26:10.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:10 smithi125 conmon[56739]: : cluster [DBG] osdmap e455: 8 total, 8 up, 7 in 2021-03-15T22:26:10.838 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:10.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:10.866 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:10.866 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:10.968 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:10.993 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:10.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:10.995 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:11.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:11.122 INFO:tasks.rados.rados.0.smithi076.stdout:1695: done (2 left) 2021-03-15T22:26:11.123 INFO:tasks.rados.rados.0.smithi076.stdout:1696: done (1 left) 2021-03-15T22:26:11.123 INFO:tasks.rados.rados.0.smithi076.stdout:1697: done (0 left) 2021-03-15T22:26:11.123 INFO:tasks.rados.rados.0.smithi076.stdout:1698: rollback oid 44 current snap is 185 2021-03-15T22:26:11.123 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 44 to 179 2021-03-15T22:26:11.124 INFO:tasks.rados.rados.0.smithi076.stdout:1699: snap_create 2021-03-15T22:26:11.124 INFO:tasks.rados.rados.0.smithi076.stdout:1698: finishing rollback tid 1 to smithi07694878-44 2021-03-15T22:26:11.125 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 336 (ObjNum 261706784 snap 21896 seq_num 0) dirty dne 2021-03-15T22:26:11.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:11.126 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:11.126 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:11.180 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:11.228 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:11.251 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:11.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:11.253 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:11.284 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:26:11.354 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:11.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:11 smithi076 conmon[62156]: cluster 2021-03-15T22:26:09. 2021-03-15T22:26:11.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:11 smithi076 conmon[62156]: 832274+0000 osd.6 (osd.6) 72 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:26:11.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:11 smithi076 conmon[62156]: cluster 2021-03-15T22:26:10.680616+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.11 deep-scrub ok 2021-03-15T22:26:11.375 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:11 smithi076 conmon[62156]: cluster 2021-03-15T22:26:11.118629+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2021-03-15T22:26:11.375 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:11 smithi076 conmon[56514]: cluster 2021-03-15T22:26:09.832274+0000 osd.6 2021-03-15T22:26:11.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:11 smithi076 conmon[56514]: (osd.6) 72 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:26:11.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:11 smithi076 conmon[56514]: cluster 2021-03-15T22:26:10.680616+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.11 deep-scrub ok 2021-03-15T22:26:11.376 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:11 smithi076 conmon[56514]: cluster 2021-03-15T22:26:11.118629+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2021-03-15T22:26:11.380 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:11.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:11.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:11.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:11.509 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:11.602 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:26:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:26:11.602 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:26:11.603 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:26:11.609 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:11.633 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:11.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:11.635 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:11.736 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:11.761 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:11.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:11.763 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:11.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:11 smithi125 conmon[56739]: cluster 2021-03-15T22:26:09.832274+0000 osd.6 (osd.6) 72 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:26:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:11 smithi125 conmon[56739]: 2021-03-15T22:26:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:11 smithi125 conmon[56739]: cluster 2021-03-15T22:26:10.680616+0000 osd.3 (osd.3) 78 : cluster [DBG] 6.11 deep-scrub ok 2021-03-15T22:26:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:11 smithi125 conmon[56739]: cluster 2021-03-15T22:26:11.118629+0000 mon.a (mon.0) 1308 : cluster [DBG] osdmap e456: 8 total, 8 up, 7 in 2021-03-15T22:26:11.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:11.889 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:11.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:11.891 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:11.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:12.018 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:12.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:12.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:12.120 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:12.124 INFO:tasks.rados.rados.0.smithi076.stdout:1698: done (1 left) 2021-03-15T22:26:12.125 INFO:tasks.rados.rados.0.smithi076.stdout:1699: done (0 left) 2021-03-15T22:26:12.125 INFO:tasks.rados.rados.0.smithi076.stdout:1700: rollback oid 49 current snap is 186 2021-03-15T22:26:12.125 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 49 to 180 2021-03-15T22:26:12.126 INFO:tasks.rados.rados.0.smithi076.stdout:1701: read oid 21 snap 184 2021-03-15T22:26:12.126 INFO:tasks.rados.rados.0.smithi076.stdout:1701: expect (ObjNum 499 snap 150 seq_num 499) 2021-03-15T22:26:12.126 INFO:tasks.rados.rados.0.smithi076.stdout:1702: delete oid 6 current snap is 186 2021-03-15T22:26:12.127 INFO:tasks.rados.rados.0.smithi076.stdout:1702: done (2 left) 2021-03-15T22:26:12.128 INFO:tasks.rados.rados.0.smithi076.stdout:1703: copy_from oid 16 from oid 3 current snap is 186 2021-03-15T22:26:12.128 INFO:tasks.rados.rados.0.smithi076.stdout:1704: read oid 24 snap -1 2021-03-15T22:26:12.128 INFO:tasks.rados.rados.0.smithi076.stdout:1704: expect deleted 2021-03-15T22:26:12.128 INFO:tasks.rados.rados.0.smithi076.stdout:1705: copy_from oid 7 from oid 37 current snap is 186 2021-03-15T22:26:12.129 INFO:tasks.rados.rados.0.smithi076.stdout:1706: copy_from oid 39 from oid 9 current snap is 186 2021-03-15T22:26:12.129 INFO:tasks.rados.rados.0.smithi076.stdout:1707: snap_create 2021-03-15T22:26:12.131 INFO:tasks.rados.rados.0.smithi076.stdout:1700: finishing rollback tid 1 to smithi07694878-49 2021-03-15T22:26:12.132 INFO:tasks.rados.rados.0.smithi076.stdout:1700: finishing rollback tid 2 to smithi07694878-49 2021-03-15T22:26:12.132 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 1080 (ObjNum 555 snap 176 seq_num 555) dirty exists 2021-03-15T22:26:12.136 INFO:tasks.rados.rados.0.smithi076.stdout:1701: expect (ObjNum 499 snap 150 seq_num 499) 2021-03-15T22:26:12.150 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:12.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:12.152 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:12.253 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:12.265 INFO:tasks.rados.rados.0.smithi076.stdout:1706: finishing copy_from to smithi07694878-39 2021-03-15T22:26:12.265 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 695 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:26:12.266 INFO:tasks.rados.rados.0.smithi076.stdout:1706: finishing copy_from racing read to smithi07694878-39 2021-03-15T22:26:12.266 INFO:tasks.rados.rados.0.smithi076.stdout:1703: finishing copy_from to smithi07694878-16 2021-03-15T22:26:12.266 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 767 (ObjNum 430 snap 119 seq_num 430) dirty exists 2021-03-15T22:26:12.266 INFO:tasks.rados.rados.0.smithi076.stdout:1705: finishing copy_from to smithi07694878-7 2021-03-15T22:26:12.266 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 956 (ObjNum 536 snap 170 seq_num 536) dirty exists 2021-03-15T22:26:12.267 INFO:tasks.rados.rados.0.smithi076.stdout:1705: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:26:12.267 INFO:tasks.rados.rados.0.smithi076.stdout:1703: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:26:12.277 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:12.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:12.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:12.361 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:12 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:26:12.362 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:12 smithi076 conmon[56514]: :26:10.840836+0000 osd.6 (osd.6) 73 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:26:12.362 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:12 smithi076 conmon[56514]: cluster 2021-03-15T22:26:11.025543+0000 osd.4 (osd.4) 98 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:26:12.362 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:12 smithi076 conmon[56514]: cluster 2021-03-15T22:26:11.064516+0000 mgr.y (mgr.14144) 654 : cluster [DBG] pgmap v810: 138 pgs: 138 active+clean; 384 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.1 MiB/s rd, 677 KiB/s wr, 6 op/s 2021-03-15T22:26:12.362 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:12 smithi076 conmon[56514]: cluster 2021-03-15T22:26:12.121051+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2021-03-15T22:26:12.363 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:12 smithi076 conmon[62156]: cluster 2021-03-15T22:26:10.840836 2021-03-15T22:26:12.363 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:12 smithi076 conmon[62156]: +0000 osd.6 (osd.6) 73 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:26:12.363 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:12 smithi076 conmon[62156]: cluster 2021-03-15T22:26:11.025543+0000 osd.4 (osd.4) 98 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:26:12.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:12 smithi076 conmon[62156]: cluster 2021-03-15T22:26:11.064516+0000 mgr.y (mgr.14144) 654 : cluster [DBG] pgmap v810: 138 pgs: 138 active+clean; 384 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.1 MiB/s rd, 677 KiB/s wr, 6 op/s 2021-03-15T22:26:12.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:12 smithi076 conmon[62156]: cluster 2021-03-15T22:26:12.121051+0000 mon.a (mon.0) 1309 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2021-03-15T22:26:12.380 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:12.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:12.405 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:12.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:12.507 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:12.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:12.530 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:12.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:12.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:12.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:12.657 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:12.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:12.759 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:12.773 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:12.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:12.787 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:12 smithi125 conmon[56739]: cluster 2021-03-15T22:26:10.840836+0000 osd.6 (osd.6) 73 : cluster 2021-03-15T22:26:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:12 smithi125 conmon[56739]: [DBG] 3.10 scrub ok 2021-03-15T22:26:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:12 smithi125 conmon[56739]: cluster 2021-03-15T22:26:11.025543+0000 osd.4 (osd.4) 98 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:26:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:12 smithi125 conmon[56739]: cluster 2021-03-15T22:26:11.064516+0000 mgr.y (mgr.14144) 654 : cluster [DBG] 2021-03-15T22:26:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:12 smithi125 conmon[56739]: pgmap v810: 138 pgs: 138 active+clean; 384 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.1 MiB/s rd, 677 KiB/s wr, 6 op/s 2021-03-15T22:26:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:12 smithi125 conmon[56739]: cluster 2021-03-15T22:26:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:12 smithi125 conmon[56739]: 2021-03-15T22:26: 2021-03-15T22:26:12.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:12 smithi125 conmon[56739]: 12.121051+0000 mon.a (mon. 2021-03-15T22:26:12.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:12 smithi125 conmon[56739]: 0) 1309 : cluster [DBG] osdmap e457: 8 total, 8 up, 7 in 2021-03-15T22:26:12.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:12 smithi125 conmon[56739]: 2021-03-15T22:26:12.801 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:26:12.889 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:12.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:12.913 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:12.913 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:13.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:13.042 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:13.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:13.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:13.125 INFO:tasks.rados.rados.0.smithi076.stdout:1700: done (6 left) 2021-03-15T22:26:13.126 INFO:tasks.rados.rados.0.smithi076.stdout:1701: done (5 left) 2021-03-15T22:26:13.126 INFO:tasks.rados.rados.0.smithi076.stdout:1703: done (4 left) 2021-03-15T22:26:13.127 INFO:tasks.rados.rados.0.smithi076.stdout:1704: done (3 left) 2021-03-15T22:26:13.127 INFO:tasks.rados.rados.0.smithi076.stdout:1705: done (2 left) 2021-03-15T22:26:13.127 INFO:tasks.rados.rados.0.smithi076.stdout:1706: done (1 left) 2021-03-15T22:26:13.128 INFO:tasks.rados.rados.0.smithi076.stdout:1707: done (0 left) 2021-03-15T22:26:13.128 INFO:tasks.rados.rados.0.smithi076.stdout:1708: rollback oid 26 current snap is 187 2021-03-15T22:26:13.129 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 180 2021-03-15T22:26:13.129 INFO:tasks.rados.rados.0.smithi076.stdout:1709: delete oid 2 current snap is 187 2021-03-15T22:26:13.131 INFO:tasks.rados.rados.0.smithi076.stdout:1708: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:26:13.131 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 664 (ObjNum 1902431504 snap 32564 seq_num 262001856) dirty dne 2021-03-15T22:26:13.132 INFO:tasks.rados.rados.0.smithi076.stdout:1708: done (1 left) 2021-03-15T22:26:13.133 INFO:tasks.rados.rados.0.smithi076.stdout:1709: done (0 left) 2021-03-15T22:26:13.133 INFO:tasks.rados.rados.0.smithi076.stdout:1710: snap_create 2021-03-15T22:26:13.146 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:13.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:13.172 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:13.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:13.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:26:13.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:13.301 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:13.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:13.404 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:13.432 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:13.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:13.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:13.444 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:13.536 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:13.562 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:13.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:13.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:13.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:13 smithi076 conmon[62156]: cluster 2021-03-15T22:26:11 2021-03-15T22:26:13.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:13 smithi076 conmon[62156]: .749754+0000 osd.1 (osd.1) 89 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:13.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:13 smithi076 conmon[62156]: cluster 2021-03-15T22:26:11.750684+0000 osd.1 (osd.1) 90 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:13.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:13 smithi076 conmon[62156]: cluster 2021-03-15T22:26:11.821560+0000 osd.6 (osd.6) 74 : cluster [DBG] 4.a scrub ok 2021-03-15T22:26:13.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:13 smithi076 conmon[62156]: cluster 2021-03-15T22:26:12.013713+0000 osd.7 (osd.7) 92 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:26:13.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:13 smithi076 conmon[62156]: cluster 2021-03-15T22:26:13.121758+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e458: 8 total, 8 up, 7 in 2021-03-15T22:26:13.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:13 smithi076 conmon[56514]: cluster 2021-03-15T22:26:11.749754+0000 osd.1 (osd.1) 89 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:13.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:13 smithi076 conmon[56514]: cluster 2021-03-15T22:26:11.750684+0000 osd.1 (osd.1) 90 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:13.573 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:13 smithi076 conmon[56514]: cluster 2021-03-15T22:26:11.821560+0000 osd.6 (osd.6) 74 : cluster [DBG] 4.a scrub ok 2021-03-15T22:26:13.573 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:13 smithi076 conmon[56514]: cluster 2021-03-15T22:26:12.013713+0000 osd.7 (osd.7) 92 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:26:13.573 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:13 smithi076 conmon[56514]: cluster 2021-03-15T22:26:13.121758+0000 mon.a (mon.0) 1310 : cluster [DBG] osdmap e458: 8 total, 8 up, 7 in 2021-03-15T22:26:13.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:13.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:13.688 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:13.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:13.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: cluster 2021-03-15T22:26:11.749754+0000 osd.1 (osd.1) 89 : 2021-03-15T22:26:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: cluster 2021-03-15T22:26:11.750684+0000 osd.1 (osd.1) 90 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: cluster 2021-03-15T22:26: 2021-03-15T22:26:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: 11.821560+0000 osd.6 (osd.6 2021-03-15T22:26:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: ) 74 : cluster [DBG] 4.a scrub ok 2021-03-15T22:26:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:26:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: 22:26:12.013713+0000 osd.7 (osd. 2021-03-15T22:26:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: 7) 92 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:26:13.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: 2021-03-15T22:26:13.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: cluster 2021-03-15T22:26:13.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:26:13.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: 26:13.121758+0000 mon.a ( 2021-03-15T22:26:13.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: mon.0) 1310 : cluster 2021-03-15T22:26:13.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:13 smithi125 conmon[56739]: [DBG] osdmap e458: 8 total, 8 up, 7 in 2021-03-15T22:26:13.790 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:13.816 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:13.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:13.818 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:13.919 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:13.934 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:26:13.936 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:26:13.942 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:13.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:13.943 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:14.044 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:14.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:14.068 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:14.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:14.126 INFO:tasks.rados.rados.0.smithi076.stdout:1710: done (0 left) 2021-03-15T22:26:14.126 INFO:tasks.rados.rados.0.smithi076.stdout:1711: write (excl) oid 36 current snap is 188 2021-03-15T22:26:14.127 INFO:tasks.rados.rados.0.smithi076.stdout:1711: seq_num 603 ranges {534621=567034,1636289=498332} 2021-03-15T22:26:14.132 INFO:tasks.rados.rados.0.smithi076.stdout:1711: writing smithi07694878-36 from 534621 to 1101655 tid 1 2021-03-15T22:26:14.138 INFO:tasks.rados.rados.0.smithi076.stdout:1711: writing smithi07694878-36 from 1636289 to 2134621 tid 2 2021-03-15T22:26:14.139 INFO:tasks.rados.rados.0.smithi076.stdout:1712: rollback oid 5 current snap is 188 2021-03-15T22:26:14.139 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 162 2021-03-15T22:26:14.139 INFO:tasks.rados.rados.0.smithi076.stdout:1713: write oid 16 current snap is 188 2021-03-15T22:26:14.139 INFO:tasks.rados.rados.0.smithi076.stdout:1713: seq_num 604 ranges {551428=642283,1694771=456657} 2021-03-15T22:26:14.145 INFO:tasks.rados.rados.0.smithi076.stdout:1713: writing smithi07694878-16 from 551428 to 1193711 tid 1 2021-03-15T22:26:14.151 INFO:tasks.rados.rados.0.smithi076.stdout:1713: writing smithi07694878-16 from 1694771 to 2151428 tid 2 2021-03-15T22:26:14.151 INFO:tasks.rados.rados.0.smithi076.stdout:1714: snap_remove snap 176 2021-03-15T22:26:14.170 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:14.192 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:14.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:14.193 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:14.294 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:14.316 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:14.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:14.318 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:14.419 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:14.445 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:14.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:14.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:14.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[56514]: cluster 2021-03-15T22:26:12.608078+0000 2021-03-15T22:26:14.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[56514]: osd.1 (osd.1) 91 : cluster [DBG] 6.18 deep-scrub ok 2021-03-15T22:26:14.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[56514]: cluster 2021-03-15T22:26:13.044462+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:26:14.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[56514]: cluster 2021-03-15T22:26:13.065021+0000 mgr.y (mgr.14144) 655 : cluster [DBG] pgmap v813: 138 pgs: 138 active+clean; 378 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.1 MiB/s rd, 937 KiB/s wr, 10 op/s 2021-03-15T22:26:14.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[56514]: audit 2021-03-15T22:26:13.917604+0000 mgr.y (mgr.14144) 656 : audit [DBG] from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:26:14.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[56514]: cluster 2021-03-15T22:26:14.123577+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e459: 8 total, 8 up, 7 in 2021-03-15T22:26:14.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[62156]: cluster 2021-03-15T22:26:12.608078+0000 2021-03-15T22:26:14.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[62156]: osd.1 (osd.1) 91 : cluster [DBG] 6.18 deep-scrub ok 2021-03-15T22:26:14.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[62156]: cluster 2021-03-15T22:26:13.044462+0000 osd.7 (osd.7) 93 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:26:14.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[62156]: cluster 2021-03-15T22:26:13.065021+0000 mgr.y (mgr.14144) 655 : cluster [DBG] pgmap v813: 138 pgs: 138 active+clean; 378 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.1 MiB/s rd, 937 KiB/s wr, 10 op/s 2021-03-15T22:26:14.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[62156]: audit 2021-03-15T22:26:13.917604+0000 mgr.y (mgr.14144) 656 : audit [DBG] from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:26:14.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:14 smithi076 conmon[62156]: cluster 2021-03-15T22:26:14.123577+0000 mon.a (mon.0) 1311 : cluster [DBG] osdmap e459: 8 total, 8 up, 7 in 2021-03-15T22:26:14.548 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:14.573 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:14.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:14.574 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:14.675 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:14.701 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:14.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:14.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:14 smithi125 conmon[56739]: cluster 2021-03-15T22:26:12.608078+0000 osd.1 (osd.1) 91 : cluster [DBG] 6.18 deep-scrub ok 2021-03-15T22:26:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:14 smithi125 conmon[56739]: cluster 2021-03-15T22:26:13.044462 2021-03-15T22:26:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:14 smithi125 conmon[56739]: +0000 osd.7 (osd.7) 93 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:26:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:14 smithi125 conmon[56739]: cluster 2021-03-15T22:26:13.065021+0000 mgr.y (mgr.14144) 655 : cluster [DBG] pgmap v813: 138 pgs: 138 active+clean; 378 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 3.1 MiB/s rd, 937 KiB/s wr, 10 op/s 2021-03-15T22:26:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:14 smithi125 conmon[56739]: audit 2021-03-15T22:26:13.917604 2021-03-15T22:26:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:14 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 656 : audit [DBG] from='client.15579 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:26:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:14 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:26:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:14 smithi125 conmon[56739]: T22:26:14.123577+0000 2021-03-15T22:26:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:14 smithi125 conmon[56739]: mon.a (mon.0) 1311 : cluster [DBG] 2021-03-15T22:26:14.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:14 smithi125 conmon[56739]: osdmap e459: 8 total, 8 up, 7 in 2021-03-15T22:26:14.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:14.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:14.832 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:14.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:14.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:14.959 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:14.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:14.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:15.041 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:15.062 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:15.064 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":814,"stamp":"2021-03-15T22:26:13.119233+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":396378991,"num_objects":492,"num_object_clones":128,"num_object_copies":1476,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":492,"num_whiteouts":12,"num_read":4847,"num_read_kb":3351264,"num_write":2984,"num_write_kb":917374,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1680,"num_bytes_recovered":514354880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7947,"ondisk_log_size":7947,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1074080,"kb_used_data":898088,"kb_used_omap":0,"kb_used_meta":175936,"kb_avail":655141984,"statfs":{"total":671965249536,"available":670865391616,"internally_reserved":0,"allocated":919642112,"data_stored":1240767762,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":180158464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-3604543,"num_objects":-6,"num_object_clones":-5,"num_object_copies":-18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-6,"num_whiteouts":-2,"num_read":10,"num_read_kb":8222,"num_write":12,"num_write_kb":2014,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.816584"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"418","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.112593+0000","last_change":"2021-03-15T22:25:41.036097+0000","last_active":"2021-03-15T22:26:10.112593+0000","last_peered":"2021-03-15T22:26:10.112593+0000","last_clean":"2021-03-15T22:26:10.112593+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:26:10.112593+0000","last_undegraded":"2021-03-15T22:26:10.112593+0000","last_fullsized":"2021-03-15T22:26:10.112593+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:33.728929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:33.728929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"128","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.361396+0000","last_change":"2021-03-15T22:26:07.787587+0000","last_active":"2021-03-15T22:26:11.361396+0000","last_peered":"2021-03-15T22:26:11.361396+0000","last_clean":"2021-03-15T22:26:11.361396+0000","last_became_active":"2021-03-15T22:24:31.103578+0000","last_became_peered":"2021-03-15T22:24:31.103578+0000","last_unstale":"2021-03-15T22:26:11.361396+0000","last_undegraded":"2021-03-15T22:26:11.361396+0000","last_fullsized":"2021-03-15T22:26:11.361396+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:26:07.787524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.787524+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"455","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:11.025658+0000","last_change":"2021-03-15T22:26:11.025622+0000","last_active":"2021-03-15T22:26:11.025658+0000","last_peered":"2021-03-15T22:26:11.025658+0000","last_clean":"2021-03-15T22:26:11.025658+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:26:11.025658+0000","last_undegraded":"2021-03-15T22:26:11.025658+0000","last_fullsized":"2021-03-15T22:26:11.025658+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:26:11.025548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:11.025548+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"47","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.307817+0000","last_change":"2021-03-15T22:25:33.149020+0000","last_active":"2021-03-15T22:26:09.307817+0000","last_peered":"2021-03-15T22:26:09.307817+0000","last_clean":"2021-03-15T22:26:09.307817+0000","last_became_active":"2021-03-15T22:25:33.148710+0000","last_became_peered":"2021-03-15T22:25:33.148710+0000","last_unstale":"2021-03-15T22:26:09.307817+0000","last_undegraded":"2021-03-15T22:26:09.307817+0000","last_fullsized":"2021-03-15T22:26:09.307817+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:06.177754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.177754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"47","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.309167+0000","last_change":"2021-03-15T22:25:33.147017+0000","last_active":"2021-03-15T22:26:09.309167+0000","last_peered":"2021-03-15T22:26:09.309167+0000","last_clean":"2021-03-15T22:26:09.309167+0000","last_became_active":"2021-03-15T22:25:33.146697+0000","last_became_peered":"2021-03-15T22:25:33.146697+0000","last_unstale":"2021-03-15T22:26:09.309167+0000","last_undegraded":"2021-03-15T22:26:09.309167+0000","last_fullsized":"2021-03-15T22:26:09.309167+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:15.333746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.333746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"73","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.307382+0000","last_change":"2021-03-15T22:25:33.175542+0000","last_active":"2021-03-15T22:26:09.307382+0000","last_peered":"2021-03-15T22:26:09.307382+0000","last_clean":"2021-03-15T22:26:09.307382+0000","last_became_active":"2021-03-15T22:25:33.137906+0000","last_became_peered":"2021-03-15T22:25:33.137906+0000","last_unstale":"2021-03-15T22:26:09.307382+0000","last_undegraded":"2021-03-15T22:26:09.307382+0000","last_fullsized":"2021-03-15T22:26:09.307382+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:23:42.144821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.144821+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":21,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"117","reported_epoch":"457","state":"active+clean","last_fresh":"2021-03-15T22:26:12.608148+0000","last_change":"2021-03-15T22:26:12.608148+0000","last_active":"2021-03-15T22:26:12.608148+0000","last_peered":"2021-03-15T22:26:12.608148+0000","last_clean":"2021-03-15T22:26:12.608148+0000","last_became_active":"2021-03-15T22:24:31.108652+0000","last_became_peered":"2021-03-15T22:24:31.108652+0000","last_unstale":"2021-03-15T22:26:12.608148+0000","last_undegraded":"2021-03-15T22:26:12.608148+0000","last_fullsized":"2021-03-15T22:26:12.608148+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:26:12.608082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"422","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.361110+0000","last_change":"2021-03-15T22:25:16.598073+0000","last_active":"2021-03-15T22:26:11.361110+0000","last_peered":"2021-03-15T22:26:11.361110+0000","last_clean":"2021-03-15T22:26:11.361110+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:26:11.361110+0000","last_undegraded":"2021-03-15T22:26:11.361110+0000","last_fullsized":"2021-03-15T22:26:11.361110+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.597998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.597998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"525","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.112843+0000","last_change":"2021-03-15T22:24:31.776304+0000","last_active":"2021-03-15T22:26:10.112843+0000","last_peered":"2021-03-15T22:26:10.112843+0000","last_clean":"2021-03-15T22:26:10.112843+0000","last_became_active":"2021-03-15T22:24:31.083373+0000","last_became_peered":"2021-03-15T22:24:31.083373+0000","last_unstale":"2021-03-15T22:26:10.112843+0000","last_undegraded":"2021-03-15T22:26:10.112843+0000","last_fullsized":"2021-03-15T22:26:10.112843+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:24:10.979322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:10.979322+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"131","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.360400+0000","last_change":"2021-03-15T22:25:33.143128+0000","last_active":"2021-03-15T22:26:11.360400+0000","last_peered":"2021-03-15T22:26:11.360400+0000","last_clean":"2021-03-15T22:26:11.360400+0000","last_became_active":"2021-03-15T22:25:33.142493+0000","last_became_peered":"2021-03-15T22:25:33.142493+0000","last_unstale":"2021-03-15T22:26:11.360400+0000","last_undegraded":"2021-03-15T22:26:11.360400+0000","last_fullsized":"2021-03-15T22:26:11.360400+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.674895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"422","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.360878+0000","last_change":"2021-03-15T22:25:05.682093+0000","last_active":"2021-03-15T22:26:11.360878+0000","last_peered":"2021-03-15T22:26:11.360878+0000","last_clean":"2021-03-15T22:26:11.360878+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:26:11.360878+0000","last_undegraded":"2021-03-15T22:26:11.360878+0000","last_fullsized":"2021-03-15T22:26:11.360878+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.682031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.682031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"121'9","reported_seq":"491","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113814+0000","last_change":"2021-03-15T22:24:37.902248+0000","last_active":"2021-03-15T22:26:10.113814+0000","last_peered":"2021-03-15T22:26:10.113814+0000","last_clean":"2021-03-15T22:26:10.113814+0000","last_became_active":"2021-03-15T22:22:56.055083+0000","last_became_peered":"2021-03-15T22:22:56.055083+0000","last_unstale":"2021-03-15T22:26:10.113814+0000","last_undegraded":"2021-03-15T22:26:10.113814+0000","last_fullsized":"2021-03-15T22:26:10.113814+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'9","last_scrub_stamp":"2021-03-15T22:24:37.902166+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.902166+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"418","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113263+0000","last_change":"2021-03-15T22:24:30.865676+0000","last_active":"2021-03-15T22:26:10.113263+0000","last_peered":"2021-03-15T22:26:10.113263+0000","last_clean":"2021-03-15T22:26:10.113263+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:26:10.113263+0000","last_undegraded":"2021-03-15T22:26:10.113263+0000","last_fullsized":"2021-03-15T22:26:10.113263+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.865609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.865609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"166","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113824+0000","last_change":"2021-03-15T22:24:55.103109+0000","last_active":"2021-03-15T22:26:10.113824+0000","last_peered":"2021-03-15T22:26:10.113824+0000","last_clean":"2021-03-15T22:26:10.113824+0000","last_became_active":"2021-03-15T22:24:55.090536+0000","last_became_peered":"2021-03-15T22:24:55.090536+0000","last_unstale":"2021-03-15T22:26:10.113824+0000","last_undegraded":"2021-03-15T22:26:10.113824+0000","last_fullsized":"2021-03-15T22:26:10.113824+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:24:28.507563+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.507563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"525","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.660649+0000","last_change":"2021-03-15T22:26:09.660649+0000","last_active":"2021-03-15T22:26:09.660649+0000","last_peered":"2021-03-15T22:26:09.660649+0000","last_clean":"2021-03-15T22:26:09.660649+0000","last_became_active":"2021-03-15T22:25:33.134238+0000","last_became_peered":"2021-03-15T22:25:33.134238+0000","last_unstale":"2021-03-15T22:26:09.660649+0000","last_undegraded":"2021-03-15T22:26:09.660649+0000","last_fullsized":"2021-03-15T22:26:09.660649+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:26:09.660549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.660549+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"245","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.309235+0000","last_change":"2021-03-15T22:25:14.117883+0000","last_active":"2021-03-15T22:26:09.309235+0000","last_peered":"2021-03-15T22:26:09.309235+0000","last_clean":"2021-03-15T22:26:09.309235+0000","last_became_active":"2021-03-15T22:25:14.117558+0000","last_became_peered":"2021-03-15T22:25:14.117558+0000","last_unstale":"2021-03-15T22:26:09.309235+0000","last_undegraded":"2021-03-15T22:26:09.309235+0000","last_fullsized":"2021-03-15T22:26:09.309235+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:00.480944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:00.480944+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"427","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.112959+0000","last_change":"2021-03-15T22:25:11.028502+0000","last_active":"2021-03-15T22:26:10.112959+0000","last_peered":"2021-03-15T22:26:10.112959+0000","last_clean":"2021-03-15T22:26:10.112959+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:26:10.112959+0000","last_undegraded":"2021-03-15T22:26:10.112959+0000","last_fullsized":"2021-03-15T22:26:10.112959+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.028422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.028422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"67","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.308166+0000","last_change":"2021-03-15T22:25:33.169024+0000","last_active":"2021-03-15T22:26:09.308166+0000","last_peered":"2021-03-15T22:26:09.308166+0000","last_clean":"2021-03-15T22:26:09.308166+0000","last_became_active":"2021-03-15T22:25:33.134781+0000","last_became_peered":"2021-03-15T22:25:33.134781+0000","last_unstale":"2021-03-15T22:26:09.308166+0000","last_undegraded":"2021-03-15T22:26:09.308166+0000","last_fullsized":"2021-03-15T22:26:09.308166+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:23:32.049047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:32.049047+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":36,"num_read_kb":19,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"408","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.127590+0000","last_change":"2021-03-15T22:23:52.272270+0000","last_active":"2021-03-15T22:26:07.127590+0000","last_peered":"2021-03-15T22:26:07.127590+0000","last_clean":"2021-03-15T22:26:07.127590+0000","last_became_active":"2021-03-15T22:23:52.271925+0000","last_became_peered":"2021-03-15T22:23:52.271925+0000","last_unstale":"2021-03-15T22:26:07.127590+0000","last_undegraded":"2021-03-15T22:26:07.127590+0000","last_fullsized":"2021-03-15T22:26:07.127590+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:34.782225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:34.782225+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"448","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.119318+0000","last_change":"2021-03-15T22:26:07.656468+0000","last_active":"2021-03-15T22:26:11.119318+0000","last_peered":"2021-03-15T22:26:11.119318+0000","last_clean":"2021-03-15T22:26:11.119318+0000","last_became_active":"2021-03-15T22:22:56.062833+0000","last_became_peered":"2021-03-15T22:22:56.062833+0000","last_unstale":"2021-03-15T22:26:11.119318+0000","last_undegraded":"2021-03-15T22:26:11.119318+0000","last_fullsized":"2021-03-15T22:26:11.119318+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:07.656378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.656378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"235","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.127815+0000","last_change":"2021-03-15T22:25:33.752847+0000","last_active":"2021-03-15T22:26:07.127815+0000","last_peered":"2021-03-15T22:26:07.127815+0000","last_clean":"2021-03-15T22:26:07.127815+0000","last_became_active":"2021-03-15T22:25:33.140078+0000","last_became_peered":"2021-03-15T22:25:33.140078+0000","last_unstale":"2021-03-15T22:26:07.127815+0000","last_undegraded":"2021-03-15T22:26:07.127815+0000","last_fullsized":"2021-03-15T22:26:07.127815+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:25:08.192165+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:25:08.192165+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"424","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.120500+0000","last_change":"2021-03-15T22:26:02.720851+0000","last_active":"2021-03-15T22:26:11.120500+0000","last_peered":"2021-03-15T22:26:11.120500+0000","last_clean":"2021-03-15T22:26:11.120500+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:26:11.120500+0000","last_undegraded":"2021-03-15T22:26:11.120500+0000","last_fullsized":"2021-03-15T22:26:11.120500+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:02.720791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:02.720791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"47","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.307890+0000","last_change":"2021-03-15T22:25:33.133625+0000","last_active":"2021-03-15T22:26:09.307890+0000","last_peered":"2021-03-15T22:26:09.307890+0000","last_clean":"2021-03-15T22:26:09.307890+0000","last_became_active":"2021-03-15T22:25:33.133418+0000","last_became_peered":"2021-03-15T22:25:33.133418+0000","last_unstale":"2021-03-15T22:26:09.307890+0000","last_undegraded":"2021-03-15T22:26:09.307890+0000","last_fullsized":"2021-03-15T22:26:09.307890+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:29.506406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.506406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"456","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.832413+0000","last_change":"2021-03-15T22:26:09.832358+0000","last_active":"2021-03-15T22:26:09.832413+0000","last_peered":"2021-03-15T22:26:09.832413+0000","last_clean":"2021-03-15T22:26:09.832413+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:26:09.832413+0000","last_undegraded":"2021-03-15T22:26:09.832413+0000","last_fullsized":"2021-03-15T22:26:09.832413+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:26:09.832278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.832278+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"413","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.126643+0000","last_change":"2021-03-15T22:25:09.177600+0000","last_active":"2021-03-15T22:26:07.126643+0000","last_peered":"2021-03-15T22:26:07.126643+0000","last_clean":"2021-03-15T22:26:07.126643+0000","last_became_active":"2021-03-15T22:23:52.260757+0000","last_became_peered":"2021-03-15T22:23:52.260757+0000","last_unstale":"2021-03-15T22:26:07.126643+0000","last_undegraded":"2021-03-15T22:26:07.126643+0000","last_fullsized":"2021-03-15T22:26:07.126643+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:09.177521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.177521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"425","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.314520+0000","last_change":"2021-03-15T22:25:02.452304+0000","last_active":"2021-03-15T22:26:09.314520+0000","last_peered":"2021-03-15T22:26:09.314520+0000","last_clean":"2021-03-15T22:26:09.314520+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:26:09.314520+0000","last_undegraded":"2021-03-15T22:26:09.314520+0000","last_fullsized":"2021-03-15T22:26:09.314520+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.452236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.452236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"259","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.361450+0000","last_change":"2021-03-15T22:26:06.764056+0000","last_active":"2021-03-15T22:26:11.361450+0000","last_peered":"2021-03-15T22:26:11.361450+0000","last_clean":"2021-03-15T22:26:11.361450+0000","last_became_active":"2021-03-15T22:25:33.146680+0000","last_became_peered":"2021-03-15T22:25:33.146680+0000","last_unstale":"2021-03-15T22:26:11.361450+0000","last_undegraded":"2021-03-15T22:26:11.361450+0000","last_fullsized":"2021-03-15T22:26:11.361450+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:06.763986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:26:06.763986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"144","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.308437+0000","last_change":"2021-03-15T22:25:15.835780+0000","last_active":"2021-03-15T22:26:09.308437+0000","last_peered":"2021-03-15T22:26:09.308437+0000","last_clean":"2021-03-15T22:26:09.308437+0000","last_became_active":"2021-03-15T22:24:23.142746+0000","last_became_peered":"2021-03-15T22:24:23.142746+0000","last_unstale":"2021-03-15T22:26:09.308437+0000","last_undegraded":"2021-03-15T22:26:09.308437+0000","last_fullsized":"2021-03-15T22:26:09.308437+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:25:15.835675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.835675+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":11,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"47","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.308009+0000","last_change":"2021-03-15T22:25:33.148415+0000","last_active":"2021-03-15T22:26:09.308009+0000","last_peered":"2021-03-15T22:26:09.308009+0000","last_clean":"2021-03-15T22:26:09.308009+0000","last_became_active":"2021-03-15T22:25:33.148083+0000","last_became_peered":"2021-03-15T22:25:33.148083+0000","last_unstale":"2021-03-15T22:26:09.308009+0000","last_undegraded":"2021-03-15T22:26:09.308009+0000","last_fullsized":"2021-03-15T22:26:09.308009+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:42.525975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:40.670674+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.525975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"405","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.126619+0000","last_change":"2021-03-15T22:23:52.263156+0000","last_active":"2021-03-15T22:26:07.126619+0000","last_peered":"2021-03-15T22:26:07.126619+0000","last_clean":"2021-03-15T22:26:07.126619+0000","last_became_active":"2021-03-15T22:23:52.262741+0000","last_became_peered":"2021-03-15T22:23:52.262741+0000","last_unstale":"2021-03-15T22:26:07.126619+0000","last_undegraded":"2021-03-15T22:26:07.126619+0000","last_fullsized":"2021-03-15T22:26:07.126619+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:41.710658+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.710658+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"422","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.361305+0000","last_change":"2021-03-15T22:26:02.840036+0000","last_active":"2021-03-15T22:26:11.361305+0000","last_peered":"2021-03-15T22:26:11.361305+0000","last_clean":"2021-03-15T22:26:11.361305+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:26:11.361305+0000","last_undegraded":"2021-03-15T22:26:11.361305+0000","last_fullsized":"2021-03-15T22:26:11.361305+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:02.839942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:02.839942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"90'22","reported_seq":"232","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.307515+0000","last_change":"2021-03-15T22:25:16.834493+0000","last_active":"2021-03-15T22:26:09.307515+0000","last_peered":"2021-03-15T22:26:09.307515+0000","last_clean":"2021-03-15T22:26:09.307515+0000","last_became_active":"2021-03-15T22:22:56.056175+0000","last_became_peered":"2021-03-15T22:22:56.056175+0000","last_unstale":"2021-03-15T22:26:09.307515+0000","last_undegraded":"2021-03-15T22:26:09.307515+0000","last_fullsized":"2021-03-15T22:26:09.307515+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:25:16.834409+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.834409+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"116","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.360374+0000","last_change":"2021-03-15T22:24:41.659524+0000","last_active":"2021-03-15T22:26:11.360374+0000","last_peered":"2021-03-15T22:26:11.360374+0000","last_clean":"2021-03-15T22:26:11.360374+0000","last_became_active":"2021-03-15T22:24:31.105380+0000","last_became_peered":"2021-03-15T22:24:31.105380+0000","last_unstale":"2021-03-15T22:26:11.360374+0000","last_undegraded":"2021-03-15T22:26:11.360374+0000","last_fullsized":"2021-03-15T22:26:11.360374+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_clean_scrub_stamp":"2021-03-15T22:24:41.659450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"232","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.307965+0000","last_change":"2021-03-15T22:25:14.115163+0000","last_active":"2021-03-15T22:26:09.307965+0000","last_peered":"2021-03-15T22:26:09.307965+0000","last_clean":"2021-03-15T22:26:09.307965+0000","last_became_active":"2021-03-15T22:25:14.114508+0000","last_became_peered":"2021-03-15T22:25:14.114508+0000","last_unstale":"2021-03-15T22:26:09.307965+0000","last_undegraded":"2021-03-15T22:26:09.307965+0000","last_fullsized":"2021-03-15T22:26:09.307965+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.007316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:04.613000+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.007316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"427","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.127308+0000","last_change":"2021-03-15T22:23:52.266674+0000","last_active":"2021-03-15T22:26:07.127308+0000","last_peered":"2021-03-15T22:26:07.127308+0000","last_clean":"2021-03-15T22:26:07.127308+0000","last_became_active":"2021-03-15T22:23:52.266011+0000","last_became_peered":"2021-03-15T22:23:52.266011+0000","last_unstale":"2021-03-15T22:26:07.127308+0000","last_undegraded":"2021-03-15T22:26:07.127308+0000","last_fullsized":"2021-03-15T22:26:07.127308+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.449472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"108","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.119879+0000","last_change":"2021-03-15T22:24:56.824808+0000","last_active":"2021-03-15T22:26:11.119879+0000","last_peered":"2021-03-15T22:26:11.119879+0000","last_clean":"2021-03-15T22:26:11.119879+0000","last_became_active":"2021-03-15T22:24:55.097354+0000","last_became_peered":"2021-03-15T22:24:55.097354+0000","last_unstale":"2021-03-15T22:26:11.119879+0000","last_undegraded":"2021-03-15T22:26:11.119879+0000","last_fullsized":"2021-03-15T22:26:11.119879+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:24:38.433166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:38.433166+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"52","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.680684+0000","last_change":"2021-03-15T22:26:10.680684+0000","last_active":"2021-03-15T22:26:10.680684+0000","last_peered":"2021-03-15T22:26:10.680684+0000","last_clean":"2021-03-15T22:26:10.680684+0000","last_became_active":"2021-03-15T22:25:33.138883+0000","last_became_peered":"2021-03-15T22:25:33.138883+0000","last_unstale":"2021-03-15T22:26:10.680684+0000","last_undegraded":"2021-03-15T22:26:10.680684+0000","last_fullsized":"2021-03-15T22:26:10.680684+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:26:10.680621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"439","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.119787+0000","last_change":"2021-03-15T22:25:33.129883+0000","last_active":"2021-03-15T22:26:11.119787+0000","last_peered":"2021-03-15T22:26:11.119787+0000","last_clean":"2021-03-15T22:26:11.119787+0000","last_became_active":"2021-03-15T22:25:33.129553+0000","last_became_peered":"2021-03-15T22:25:33.129553+0000","last_unstale":"2021-03-15T22:26:11.119787+0000","last_undegraded":"2021-03-15T22:26:11.119787+0000","last_fullsized":"2021-03-15T22:26:11.119787+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.794432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"141","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.120548+0000","last_change":"2021-03-15T22:24:12.017160+0000","last_active":"2021-03-15T22:26:08.120548+0000","last_peered":"2021-03-15T22:26:08.120548+0000","last_clean":"2021-03-15T22:26:08.120548+0000","last_became_active":"2021-03-15T22:24:03.086278+0000","last_became_peered":"2021-03-15T22:24:03.086278+0000","last_unstale":"2021-03-15T22:26:08.120548+0000","last_undegraded":"2021-03-15T22:26:08.120548+0000","last_fullsized":"2021-03-15T22:26:08.120548+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:12.017097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:12.017097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"449","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113710+0000","last_change":"2021-03-15T22:25:43.037390+0000","last_active":"2021-03-15T22:26:10.113710+0000","last_peered":"2021-03-15T22:26:10.113710+0000","last_clean":"2021-03-15T22:26:10.113710+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:26:10.113710+0000","last_undegraded":"2021-03-15T22:26:10.113710+0000","last_fullsized":"2021-03-15T22:26:10.113710+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:29.709674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.709674+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"246","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.127086+0000","last_change":"2021-03-15T22:25:33.144478+0000","last_active":"2021-03-15T22:26:07.127086+0000","last_peered":"2021-03-15T22:26:07.127086+0000","last_clean":"2021-03-15T22:26:07.127086+0000","last_became_active":"2021-03-15T22:25:33.143832+0000","last_became_peered":"2021-03-15T22:25:33.143832+0000","last_unstale":"2021-03-15T22:26:07.127086+0000","last_undegraded":"2021-03-15T22:26:07.127086+0000","last_fullsized":"2021-03-15T22:26:07.127086+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.147414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.147414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"356'3","reported_seq":"527","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:12.091616+0000","last_change":"2021-03-15T22:24:31.129210+0000","last_active":"2021-03-15T22:26:12.091616+0000","last_peered":"2021-03-15T22:26:12.091616+0000","last_clean":"2021-03-15T22:26:12.091616+0000","last_became_active":"2021-03-15T22:24:31.093479+0000","last_became_peered":"2021-03-15T22:24:31.093479+0000","last_unstale":"2021-03-15T22:26:12.091616+0000","last_undegraded":"2021-03-15T22:26:12.091616+0000","last_fullsized":"2021-03-15T22:26:12.091616+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:24:28.700659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.700659+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"414","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.126904+0000","last_change":"2021-03-15T22:25:01.148183+0000","last_active":"2021-03-15T22:26:07.126904+0000","last_peered":"2021-03-15T22:26:07.126904+0000","last_clean":"2021-03-15T22:26:07.126904+0000","last_became_active":"2021-03-15T22:23:52.261234+0000","last_became_peered":"2021-03-15T22:23:52.261234+0000","last_unstale":"2021-03-15T22:26:07.126904+0000","last_undegraded":"2021-03-15T22:26:07.126904+0000","last_fullsized":"2021-03-15T22:26:07.126904+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:01.148105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:01.148105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"231","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.307773+0000","last_change":"2021-03-15T22:24:27.166757+0000","last_active":"2021-03-15T22:26:09.307773+0000","last_peered":"2021-03-15T22:26:09.307773+0000","last_clean":"2021-03-15T22:26:09.307773+0000","last_became_active":"2021-03-15T22:22:56.064081+0000","last_became_peered":"2021-03-15T22:22:56.064081+0000","last_unstale":"2021-03-15T22:26:09.307773+0000","last_undegraded":"2021-03-15T22:26:09.307773+0000","last_fullsized":"2021-03-15T22:26:09.307773+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:27.166678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.166678+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":13,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"416","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113944+0000","last_change":"2021-03-15T22:25:07.992897+0000","last_active":"2021-03-15T22:26:10.113944+0000","last_peered":"2021-03-15T22:26:10.113944+0000","last_clean":"2021-03-15T22:26:10.113944+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:26:10.113944+0000","last_undegraded":"2021-03-15T22:26:10.113944+0000","last_fullsized":"2021-03-15T22:26:10.113944+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:07.992815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.992815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"421'7","reported_seq":"58","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.307626+0000","last_change":"2021-03-15T22:25:33.172230+0000","last_active":"2021-03-15T22:26:09.307626+0000","last_peered":"2021-03-15T22:26:09.307626+0000","last_clean":"2021-03-15T22:26:09.307626+0000","last_became_active":"2021-03-15T22:25:33.147484+0000","last_became_peered":"2021-03-15T22:25:33.147484+0000","last_unstale":"2021-03-15T22:26:09.307626+0000","last_undegraded":"2021-03-15T22:26:09.307626+0000","last_fullsized":"2021-03-15T22:26:09.307626+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'6","last_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.098511+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"252","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.126728+0000","last_change":"2021-03-15T22:25:33.141774+0000","last_active":"2021-03-15T22:26:07.126728+0000","last_peered":"2021-03-15T22:26:07.126728+0000","last_clean":"2021-03-15T22:26:07.126728+0000","last_became_active":"2021-03-15T22:25:33.141466+0000","last_became_peered":"2021-03-15T22:25:33.141466+0000","last_unstale":"2021-03-15T22:26:07.126728+0000","last_undegraded":"2021-03-15T22:26:07.126728+0000","last_fullsized":"2021-03-15T22:26:07.126728+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:23:39.695671+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"86","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.360946+0000","last_change":"2021-03-15T22:25:14.129885+0000","last_active":"2021-03-15T22:26:11.360946+0000","last_peered":"2021-03-15T22:26:11.360946+0000","last_clean":"2021-03-15T22:26:11.360946+0000","last_became_active":"2021-03-15T22:25:14.114060+0000","last_became_peered":"2021-03-15T22:25:14.114060+0000","last_unstale":"2021-03-15T22:26:11.360946+0000","last_undegraded":"2021-03-15T22:26:11.360946+0000","last_fullsized":"2021-03-15T22:26:11.360946+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:25:07.163128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.163128+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"419","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.120963+0000","last_change":"2021-03-15T22:25:00.614351+0000","last_active":"2021-03-15T22:26:08.120963+0000","last_peered":"2021-03-15T22:26:08.120963+0000","last_clean":"2021-03-15T22:26:08.120963+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:26:08.120963+0000","last_undegraded":"2021-03-15T22:26:08.120963+0000","last_fullsized":"2021-03-15T22:26:08.120963+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.614288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.614288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"416","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.126695+0000","last_change":"2021-03-15T22:25:10.130486+0000","last_active":"2021-03-15T22:26:07.126695+0000","last_peered":"2021-03-15T22:26:07.126695+0000","last_clean":"2021-03-15T22:26:07.126695+0000","last_became_active":"2021-03-15T22:23:52.263426+0000","last_became_peered":"2021-03-15T22:23:52.263426+0000","last_unstale":"2021-03-15T22:26:07.126695+0000","last_undegraded":"2021-03-15T22:26:07.126695+0000","last_fullsized":"2021-03-15T22:26:07.126695+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:10.130422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.130422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"71","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.308981+0000","last_change":"2021-03-15T22:25:35.064301+0000","last_active":"2021-03-15T22:26:09.308981+0000","last_peered":"2021-03-15T22:26:09.308981+0000","last_clean":"2021-03-15T22:26:09.308981+0000","last_became_active":"2021-03-15T22:25:33.151159+0000","last_became_peered":"2021-03-15T22:25:33.151159+0000","last_unstale":"2021-03-15T22:26:09.308981+0000","last_undegraded":"2021-03-15T22:26:09.308981+0000","last_fullsized":"2021-03-15T22:26:09.308981+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:26.478799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.478799+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3618,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"413","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.127258+0000","last_change":"2021-03-15T22:26:07.054564+0000","last_active":"2021-03-15T22:26:07.127258+0000","last_peered":"2021-03-15T22:26:07.127258+0000","last_clean":"2021-03-15T22:26:07.127258+0000","last_became_active":"2021-03-15T22:23:52.266133+0000","last_became_peered":"2021-03-15T22:23:52.266133+0000","last_unstale":"2021-03-15T22:26:07.127258+0000","last_undegraded":"2021-03-15T22:26:07.127258+0000","last_fullsized":"2021-03-15T22:26:07.127258+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:07.054500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.054500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"470","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.840930+0000","last_change":"2021-03-15T22:26:10.840930+0000","last_active":"2021-03-15T22:26:10.840930+0000","last_peered":"2021-03-15T22:26:10.840930+0000","last_clean":"2021-03-15T22:26:10.840930+0000","last_became_active":"2021-03-15T22:25:33.150147+0000","last_became_peered":"2021-03-15T22:25:33.150147+0000","last_unstale":"2021-03-15T22:26:10.840930+0000","last_undegraded":"2021-03-15T22:26:10.840930+0000","last_fullsized":"2021-03-15T22:26:10.840930+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:10.840841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:10.840841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"398'11","reported_seq":"57","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.308774+0000","last_change":"2021-03-15T22:25:33.864526+0000","last_active":"2021-03-15T22:26:09.308774+0000","last_peered":"2021-03-15T22:26:09.308774+0000","last_clean":"2021-03-15T22:26:09.308774+0000","last_became_active":"2021-03-15T22:25:33.152373+0000","last_became_peered":"2021-03-15T22:25:33.152373+0000","last_unstale":"2021-03-15T22:26:09.308774+0000","last_undegraded":"2021-03-15T22:26:09.308774+0000","last_fullsized":"2021-03-15T22:26:09.308774+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'11","last_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:25:13.181225+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"455'446","reported_seq":"1748","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.120534+0000","last_change":"2021-03-15T22:26:09.318313+0000","last_active":"2021-03-15T22:26:11.120534+0000","last_peered":"2021-03-15T22:26:11.120534+0000","last_clean":"2021-03-15T22:26:11.120534+0000","last_became_active":"2021-03-15T22:25:17.113174+0000","last_became_peered":"2021-03-15T22:25:17.113174+0000","last_unstale":"2021-03-15T22:26:11.120534+0000","last_undegraded":"2021-03-15T22:26:11.120534+0000","last_fullsized":"2021-03-15T22:26:11.120534+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'282","last_scrub_stamp":"2021-03-15T22:25:09.803362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.803362+0000","log_size":446,"ondisk_log_size":446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28490809,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":278,"num_read_kb":314807,"num_write":171,"num_write_kb":53778,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":21669077,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"425","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.360792+0000","last_change":"2021-03-15T22:25:00.689659+0000","last_active":"2021-03-15T22:26:11.360792+0000","last_peered":"2021-03-15T22:26:11.360792+0000","last_clean":"2021-03-15T22:26:11.360792+0000","last_became_active":"2021-03-15T22:24:31.103470+0000","last_became_peered":"2021-03-15T22:24:31.103470+0000","last_unstale":"2021-03-15T22:26:11.360792+0000","last_undegraded":"2021-03-15T22:26:11.360792+0000","last_fullsized":"2021-03-15T22:26:11.360792+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.689591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.689591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"437","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.127164+0000","last_change":"2021-03-15T22:25:48.008983+0000","last_active":"2021-03-15T22:26:07.127164+0000","last_peered":"2021-03-15T22:26:07.127164+0000","last_clean":"2021-03-15T22:26:07.127164+0000","last_became_active":"2021-03-15T22:24:31.097688+0000","last_became_peered":"2021-03-15T22:24:31.097688+0000","last_unstale":"2021-03-15T22:26:07.127164+0000","last_undegraded":"2021-03-15T22:26:07.127164+0000","last_fullsized":"2021-03-15T22:26:07.127164+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:25:48.008911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:48.008911+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"144","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.360508+0000","last_change":"2021-03-15T22:25:32.799965+0000","last_active":"2021-03-15T22:26:11.360508+0000","last_peered":"2021-03-15T22:26:11.360508+0000","last_clean":"2021-03-15T22:26:11.360508+0000","last_became_active":"2021-03-15T22:24:23.137531+0000","last_became_peered":"2021-03-15T22:24:23.137531+0000","last_unstale":"2021-03-15T22:26:11.360508+0000","last_undegraded":"2021-03-15T22:26:11.360508+0000","last_fullsized":"2021-03-15T22:26:11.360508+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:25:32.799882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.799882+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"317'7","reported_seq":"528","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.126790+0000","last_change":"2021-03-15T22:24:30.068613+0000","last_active":"2021-03-15T22:26:07.126790+0000","last_peered":"2021-03-15T22:26:07.126790+0000","last_clean":"2021-03-15T22:26:07.126790+0000","last_became_active":"2021-03-15T22:23:52.262878+0000","last_became_peered":"2021-03-15T22:23:52.262878+0000","last_unstale":"2021-03-15T22:26:07.126790+0000","last_undegraded":"2021-03-15T22:26:07.126790+0000","last_fullsized":"2021-03-15T22:26:07.126790+0000","mapping_epoch":316,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:24:30.068550+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.068550+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"454'650","reported_seq":"1855","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.123354+0000","last_change":"2021-03-15T22:26:09.318922+0000","last_active":"2021-03-15T22:26:10.123354+0000","last_peered":"2021-03-15T22:26:10.123354+0000","last_clean":"2021-03-15T22:26:10.123354+0000","last_became_active":"2021-03-15T22:23:04.067641+0000","last_became_peered":"2021-03-15T22:23:04.067641+0000","last_unstale":"2021-03-15T22:26:10.123354+0000","last_undegraded":"2021-03-15T22:26:10.123354+0000","last_fullsized":"2021-03-15T22:26:10.123354+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'455","last_scrub_stamp":"2021-03-15T22:24:28.184612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.184612+0000","log_size":650,"ondisk_log_size":650,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35446306,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":331,"num_read_kb":337383,"num_write":227,"num_write_kb":85488,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":32120558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"214","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.308196+0000","last_change":"2021-03-15T22:24:37.122501+0000","last_active":"2021-03-15T22:26:09.308196+0000","last_peered":"2021-03-15T22:26:09.308196+0000","last_clean":"2021-03-15T22:26:09.308196+0000","last_became_active":"2021-03-15T22:22:56.057174+0000","last_became_peered":"2021-03-15T22:22:56.057174+0000","last_unstale":"2021-03-15T22:26:09.308196+0000","last_undegraded":"2021-03-15T22:26:09.308196+0000","last_fullsized":"2021-03-15T22:26:09.308196+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:37.122407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.122407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"421","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.127196+0000","last_change":"2021-03-15T22:26:03.105518+0000","last_active":"2021-03-15T22:26:07.127196+0000","last_peered":"2021-03-15T22:26:07.127196+0000","last_clean":"2021-03-15T22:26:07.127196+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:26:07.127196+0000","last_undegraded":"2021-03-15T22:26:07.127196+0000","last_fullsized":"2021-03-15T22:26:07.127196+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.105446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.105446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"503","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113497+0000","last_change":"2021-03-15T22:25:33.862633+0000","last_active":"2021-03-15T22:26:10.113497+0000","last_peered":"2021-03-15T22:26:10.113497+0000","last_clean":"2021-03-15T22:26:10.113497+0000","last_became_active":"2021-03-15T22:25:33.138262+0000","last_became_peered":"2021-03-15T22:25:33.138262+0000","last_unstale":"2021-03-15T22:26:10.113497+0000","last_undegraded":"2021-03-15T22:26:10.113497+0000","last_fullsized":"2021-03-15T22:26:10.113497+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:07.027095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.027095+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"448","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.121065+0000","last_change":"2021-03-15T22:24:39.853332+0000","last_active":"2021-03-15T22:26:08.121065+0000","last_peered":"2021-03-15T22:26:08.121065+0000","last_clean":"2021-03-15T22:26:08.121065+0000","last_became_active":"2021-03-15T22:24:31.109058+0000","last_became_peered":"2021-03-15T22:24:31.109058+0000","last_unstale":"2021-03-15T22:26:08.121065+0000","last_undegraded":"2021-03-15T22:26:08.121065+0000","last_fullsized":"2021-03-15T22:26:08.121065+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.853267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.853267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"453'156","reported_seq":"1310","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.119519+0000","last_change":"2021-03-15T22:26:09.307286+0000","last_active":"2021-03-15T22:26:11.119519+0000","last_peered":"2021-03-15T22:26:11.119519+0000","last_clean":"2021-03-15T22:26:11.119519+0000","last_became_active":"2021-03-15T22:24:34.087074+0000","last_became_peered":"2021-03-15T22:24:34.087074+0000","last_unstale":"2021-03-15T22:26:11.119519+0000","last_undegraded":"2021-03-15T22:26:11.119519+0000","last_fullsized":"2021-03-15T22:26:11.119519+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'130","last_scrub_stamp":"2021-03-15T22:25:06.854931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.854931+0000","log_size":156,"ondisk_log_size":156,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5697738,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":43,"num_read_kb":30033,"num_write":50,"num_write_kb":17214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4470794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"426","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.126380+0000","last_change":"2021-03-15T22:23:52.267464+0000","last_active":"2021-03-15T22:26:07.126380+0000","last_peered":"2021-03-15T22:26:07.126380+0000","last_clean":"2021-03-15T22:26:07.126380+0000","last_became_active":"2021-03-15T22:23:52.267164+0000","last_became_peered":"2021-03-15T22:23:52.267164+0000","last_unstale":"2021-03-15T22:26:07.126380+0000","last_undegraded":"2021-03-15T22:26:07.126380+0000","last_fullsized":"2021-03-15T22:26:07.126380+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:29.845442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.845442+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"466","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.121458+0000","last_change":"2021-03-15T22:25:14.114184+0000","last_active":"2021-03-15T22:26:08.121458+0000","last_peered":"2021-03-15T22:26:08.121458+0000","last_clean":"2021-03-15T22:26:08.121458+0000","last_became_active":"2021-03-15T22:25:14.113857+0000","last_became_peered":"2021-03-15T22:25:14.113857+0000","last_unstale":"2021-03-15T22:26:08.121458+0000","last_undegraded":"2021-03-15T22:26:08.121458+0000","last_fullsized":"2021-03-15T22:26:08.121458+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.575552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.575552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"434","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.127410+0000","last_change":"2021-03-15T22:25:14.118841+0000","last_active":"2021-03-15T22:26:07.127410+0000","last_peered":"2021-03-15T22:26:07.127410+0000","last_clean":"2021-03-15T22:26:07.127410+0000","last_became_active":"2021-03-15T22:25:14.118469+0000","last_became_peered":"2021-03-15T22:25:14.118469+0000","last_unstale":"2021-03-15T22:26:07.127410+0000","last_undegraded":"2021-03-15T22:26:07.127410+0000","last_fullsized":"2021-03-15T22:26:07.127410+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.024799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.024799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"496","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.313898+0000","last_change":"2021-03-15T22:24:40.468134+0000","last_active":"2021-03-15T22:26:09.313898+0000","last_peered":"2021-03-15T22:26:09.313898+0000","last_clean":"2021-03-15T22:26:09.313898+0000","last_became_active":"2021-03-15T22:24:03.085862+0000","last_became_peered":"2021-03-15T22:24:03.085862+0000","last_unstale":"2021-03-15T22:26:09.313898+0000","last_undegraded":"2021-03-15T22:26:09.313898+0000","last_fullsized":"2021-03-15T22:26:09.313898+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:24:40.468066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.468066+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"453'417","reported_seq":"1555","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.304753+0000","last_change":"2021-03-15T22:26:09.307039+0000","last_active":"2021-03-15T22:26:09.304753+0000","last_peered":"2021-03-15T22:26:09.304753+0000","last_clean":"2021-03-15T22:26:09.304753+0000","last_became_active":"2021-03-15T22:20:05.032282+0000","last_became_peered":"2021-03-15T22:20:05.032282+0000","last_unstale":"2021-03-15T22:26:09.304753+0000","last_undegraded":"2021-03-15T22:26:09.304753+0000","last_fullsized":"2021-03-15T22:26:09.304753+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'328","last_scrub_stamp":"2021-03-15T22:25:37.707812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.707812+0000","log_size":417,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19195372,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":206,"num_read_kb":247372,"num_write":118,"num_write_kb":38785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"100","length":"1"},{"start":"114","length":"1"},{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14b","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"114","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.126596+0000","last_change":"2021-03-15T22:25:11.117468+0000","last_active":"2021-03-15T22:26:07.126596+0000","last_peered":"2021-03-15T22:26:07.126596+0000","last_clean":"2021-03-15T22:26:07.126596+0000","last_became_active":"2021-03-15T22:24:31.093577+0000","last_became_peered":"2021-03-15T22:24:31.093577+0000","last_unstale":"2021-03-15T22:26:07.126596+0000","last_undegraded":"2021-03-15T22:26:07.126596+0000","last_fullsized":"2021-03-15T22:26:07.126596+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.117403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.117403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"466","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.121239+0000","last_change":"2021-03-15T22:25:33.134762+0000","last_active":"2021-03-15T22:26:08.121239+0000","last_peered":"2021-03-15T22:26:08.121239+0000","last_clean":"2021-03-15T22:26:08.121239+0000","last_became_active":"2021-03-15T22:25:33.134358+0000","last_became_peered":"2021-03-15T22:25:33.134358+0000","last_unstale":"2021-03-15T22:26:08.121239+0000","last_undegraded":"2021-03-15T22:26:08.121239+0000","last_fullsized":"2021-03-15T22:26:08.121239+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:27.869570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.869570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"440","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.360995+0000","last_change":"2021-03-15T22:25:47.753887+0000","last_active":"2021-03-15T22:26:11.360995+0000","last_peered":"2021-03-15T22:26:11.360995+0000","last_clean":"2021-03-15T22:26:11.360995+0000","last_became_active":"2021-03-15T22:25:33.145548+0000","last_became_peered":"2021-03-15T22:25:33.145548+0000","last_unstale":"2021-03-15T22:26:11.360995+0000","last_undegraded":"2021-03-15T22:26:11.360995+0000","last_fullsized":"2021-03-15T22:26:11.360995+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:47.753814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:47.753814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"216","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.121640+0000","last_change":"2021-03-15T22:25:33.912570+0000","last_active":"2021-03-15T22:26:08.121640+0000","last_peered":"2021-03-15T22:26:08.121640+0000","last_clean":"2021-03-15T22:26:08.121640+0000","last_became_active":"2021-03-15T22:25:33.135102+0000","last_became_peered":"2021-03-15T22:25:33.135102+0000","last_unstale":"2021-03-15T22:26:08.121640+0000","last_undegraded":"2021-03-15T22:26:08.121640+0000","last_fullsized":"2021-03-15T22:26:08.121640+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:01.499099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.499099+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"453'444","reported_seq":"1738","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.212962+0000","last_change":"2021-03-15T22:26:09.317097+0000","last_active":"2021-03-15T22:26:10.212962+0000","last_peered":"2021-03-15T22:26:10.212962+0000","last_clean":"2021-03-15T22:26:10.212962+0000","last_became_active":"2021-03-15T22:25:36.118229+0000","last_became_peered":"2021-03-15T22:25:36.118229+0000","last_unstale":"2021-03-15T22:26:10.212962+0000","last_undegraded":"2021-03-15T22:26:10.212962+0000","last_fullsized":"2021-03-15T22:26:10.212962+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'340","last_scrub_stamp":"2021-03-15T22:25:12.353191+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.353191+0000","log_size":444,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29443813,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":146,"num_read_kb":101627,"num_write":181,"num_write_kb":56370,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":35579701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"52","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.309079+0000","last_change":"2021-03-15T22:26:04.729228+0000","last_active":"2021-03-15T22:26:09.309079+0000","last_peered":"2021-03-15T22:26:09.309079+0000","last_clean":"2021-03-15T22:26:09.309079+0000","last_became_active":"2021-03-15T22:25:33.152232+0000","last_became_peered":"2021-03-15T22:25:33.152232+0000","last_unstale":"2021-03-15T22:26:09.309079+0000","last_undegraded":"2021-03-15T22:26:09.309079+0000","last_fullsized":"2021-03-15T22:26:09.309079+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:04.729163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:04.729163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"424","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.360611+0000","last_change":"2021-03-15T22:25:11.615097+0000","last_active":"2021-03-15T22:26:11.360611+0000","last_peered":"2021-03-15T22:26:11.360611+0000","last_clean":"2021-03-15T22:26:11.360611+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:26:11.360611+0000","last_undegraded":"2021-03-15T22:26:11.360611+0000","last_fullsized":"2021-03-15T22:26:11.360611+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.614975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.614975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"150","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.121384+0000","last_change":"2021-03-15T22:25:45.692608+0000","last_active":"2021-03-15T22:26:08.121384+0000","last_peered":"2021-03-15T22:26:08.121384+0000","last_clean":"2021-03-15T22:26:08.121384+0000","last_became_active":"2021-03-15T22:24:31.106344+0000","last_became_peered":"2021-03-15T22:24:31.106344+0000","last_unstale":"2021-03-15T22:26:08.121384+0000","last_undegraded":"2021-03-15T22:26:08.121384+0000","last_fullsized":"2021-03-15T22:26:08.121384+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:45.692545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:45.692545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"562","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.120054+0000","last_change":"2021-03-15T22:25:33.174619+0000","last_active":"2021-03-15T22:26:11.120054+0000","last_peered":"2021-03-15T22:26:11.120054+0000","last_clean":"2021-03-15T22:26:11.120054+0000","last_became_active":"2021-03-15T22:25:33.126039+0000","last_became_peered":"2021-03-15T22:25:33.126039+0000","last_unstale":"2021-03-15T22:26:11.120054+0000","last_undegraded":"2021-03-15T22:26:11.120054+0000","last_fullsized":"2021-03-15T22:26:11.120054+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:25:15.810377+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.810377+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"117","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.120410+0000","last_change":"2021-03-15T22:25:44.619146+0000","last_active":"2021-03-15T22:26:11.120410+0000","last_peered":"2021-03-15T22:26:11.120410+0000","last_clean":"2021-03-15T22:26:11.120410+0000","last_became_active":"2021-03-15T22:24:31.101668+0000","last_became_peered":"2021-03-15T22:24:31.101668+0000","last_unstale":"2021-03-15T22:26:11.120410+0000","last_undegraded":"2021-03-15T22:26:11.120410+0000","last_fullsized":"2021-03-15T22:26:11.120410+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:44.619082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:44.619082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"453'426","reported_seq":"1594","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113262+0000","last_change":"2021-03-15T22:26:09.313057+0000","last_active":"2021-03-15T22:26:10.113262+0000","last_peered":"2021-03-15T22:26:10.113262+0000","last_clean":"2021-03-15T22:26:10.113262+0000","last_became_active":"2021-03-15T22:24:34.087484+0000","last_became_peered":"2021-03-15T22:24:34.087484+0000","last_unstale":"2021-03-15T22:26:10.113262+0000","last_undegraded":"2021-03-15T22:26:10.113262+0000","last_fullsized":"2021-03-15T22:26:10.113262+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'329","last_scrub_stamp":"2021-03-15T22:25:14.014776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:14.014776+0000","log_size":426,"ondisk_log_size":426,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17507203,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":129,"num_read_kb":112613,"num_write":134,"num_write_kb":46382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":10768640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"431","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:08.042677+0000","last_change":"2021-03-15T22:26:08.042648+0000","last_active":"2021-03-15T22:26:08.042677+0000","last_peered":"2021-03-15T22:26:08.042677+0000","last_clean":"2021-03-15T22:26:08.042677+0000","last_became_active":"2021-03-15T22:23:52.266556+0000","last_became_peered":"2021-03-15T22:23:52.266556+0000","last_unstale":"2021-03-15T22:26:08.042677+0000","last_undegraded":"2021-03-15T22:26:08.042677+0000","last_fullsized":"2021-03-15T22:26:08.042677+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:08.042588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:08.042588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"425","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.119696+0000","last_change":"2021-03-15T22:24:39.929128+0000","last_active":"2021-03-15T22:26:11.119696+0000","last_peered":"2021-03-15T22:26:11.119696+0000","last_clean":"2021-03-15T22:26:11.119696+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:26:11.119696+0000","last_undegraded":"2021-03-15T22:26:11.119696+0000","last_fullsized":"2021-03-15T22:26:11.119696+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.929066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.929066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"454","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.120412+0000","last_change":"2021-03-15T22:25:42.660209+0000","last_active":"2021-03-15T22:26:08.120412+0000","last_peered":"2021-03-15T22:26:08.120412+0000","last_clean":"2021-03-15T22:26:08.120412+0000","last_became_active":"2021-03-15T22:25:33.129875+0000","last_became_peered":"2021-03-15T22:25:33.129875+0000","last_unstale":"2021-03-15T22:26:08.120412+0000","last_undegraded":"2021-03-15T22:26:08.120412+0000","last_fullsized":"2021-03-15T22:26:08.120412+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:29.266174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.266174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"149'22","reported_seq":"107","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.120652+0000","last_change":"2021-03-15T22:25:03.853145+0000","last_active":"2021-03-15T22:26:11.120652+0000","last_peered":"2021-03-15T22:26:11.120652+0000","last_clean":"2021-03-15T22:26:11.120652+0000","last_became_active":"2021-03-15T22:24:55.093124+0000","last_became_peered":"2021-03-15T22:24:55.093124+0000","last_unstale":"2021-03-15T22:26:11.120652+0000","last_undegraded":"2021-03-15T22:26:11.120652+0000","last_fullsized":"2021-03-15T22:26:11.120652+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'22","last_scrub_stamp":"2021-03-15T22:25:03.853079+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:25:03.853079+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"439","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.122345+0000","last_change":"2021-03-15T22:25:02.520674+0000","last_active":"2021-03-15T22:26:08.122345+0000","last_peered":"2021-03-15T22:26:08.122345+0000","last_clean":"2021-03-15T22:26:08.122345+0000","last_became_active":"2021-03-15T22:21:38.147364+0000","last_became_peered":"2021-03-15T22:21:38.147364+0000","last_unstale":"2021-03-15T22:26:08.122345+0000","last_undegraded":"2021-03-15T22:26:08.122345+0000","last_fullsized":"2021-03-15T22:26:08.122345+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.520588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.520588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"453'1068","reported_seq":"2925","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.149129+0000","last_change":"2021-03-15T22:26:08.149323+0000","last_active":"2021-03-15T22:26:08.149129+0000","last_peered":"2021-03-15T22:26:08.149129+0000","last_clean":"2021-03-15T22:26:08.149129+0000","last_became_active":"2021-03-15T22:25:37.118396+0000","last_became_peered":"2021-03-15T22:25:37.118396+0000","last_unstale":"2021-03-15T22:26:08.149129+0000","last_undegraded":"2021-03-15T22:26:08.149129+0000","last_fullsized":"2021-03-15T22:26:08.149129+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'1047","last_scrub_stamp":"2021-03-15T22:26:05.046615+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:26:05.046615+0000","log_size":1068,"ondisk_log_size":1068,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55845644,"num_objects":23,"num_object_clones":16,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":2,"num_read":373,"num_read_kb":327229,"num_write":364,"num_write_kb":157873,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":140467323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"100","length":"1"},{"start":"114","length":"1"},{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14b","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"434","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.361341+0000","last_change":"2021-03-15T22:26:03.807852+0000","last_active":"2021-03-15T22:26:11.361341+0000","last_peered":"2021-03-15T22:26:11.361341+0000","last_clean":"2021-03-15T22:26:11.361341+0000","last_became_active":"2021-03-15T22:22:56.062745+0000","last_became_peered":"2021-03-15T22:22:56.062745+0000","last_unstale":"2021-03-15T22:26:11.361341+0000","last_undegraded":"2021-03-15T22:26:11.361341+0000","last_fullsized":"2021-03-15T22:26:11.361341+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.807788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.807788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"428","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.126456+0000","last_change":"2021-03-15T22:24:31.111721+0000","last_active":"2021-03-15T22:26:07.126456+0000","last_peered":"2021-03-15T22:26:07.126456+0000","last_clean":"2021-03-15T22:26:07.126456+0000","last_became_active":"2021-03-15T22:24:31.111104+0000","last_became_peered":"2021-03-15T22:24:31.111104+0000","last_unstale":"2021-03-15T22:26:07.126456+0000","last_undegraded":"2021-03-15T22:26:07.126456+0000","last_fullsized":"2021-03-15T22:26:07.126456+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.872030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.872030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"52","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.308819+0000","last_change":"2021-03-15T22:25:46.813664+0000","last_active":"2021-03-15T22:26:09.308819+0000","last_peered":"2021-03-15T22:26:09.308819+0000","last_clean":"2021-03-15T22:26:09.308819+0000","last_became_active":"2021-03-15T22:25:33.147285+0000","last_became_peered":"2021-03-15T22:25:33.147285+0000","last_unstale":"2021-03-15T22:26:09.308819+0000","last_undegraded":"2021-03-15T22:26:09.308819+0000","last_fullsized":"2021-03-15T22:26:09.308819+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:46.813605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.813605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"449","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.315706+0000","last_change":"2021-03-15T22:25:37.458353+0000","last_active":"2021-03-15T22:26:09.315706+0000","last_peered":"2021-03-15T22:26:09.315706+0000","last_clean":"2021-03-15T22:26:09.315706+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:26:09.315706+0000","last_undegraded":"2021-03-15T22:26:09.315706+0000","last_fullsized":"2021-03-15T22:26:09.315706+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:37.458291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.458291+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"412","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.127684+0000","last_change":"2021-03-15T22:23:52.271864+0000","last_active":"2021-03-15T22:26:07.127684+0000","last_peered":"2021-03-15T22:26:07.127684+0000","last_clean":"2021-03-15T22:26:07.127684+0000","last_became_active":"2021-03-15T22:23:52.271442+0000","last_became_peered":"2021-03-15T22:23:52.271442+0000","last_unstale":"2021-03-15T22:26:07.127684+0000","last_undegraded":"2021-03-15T22:26:07.127684+0000","last_fullsized":"2021-03-15T22:26:07.127684+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:26.748485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.748485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"453'319","reported_seq":"1505","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.133245+0000","last_change":"2021-03-15T22:26:08.133424+0000","last_active":"2021-03-15T22:26:08.133245+0000","last_peered":"2021-03-15T22:26:08.133245+0000","last_clean":"2021-03-15T22:26:08.133245+0000","last_became_active":"2021-03-15T22:24:36.092680+0000","last_became_peered":"2021-03-15T22:24:36.092680+0000","last_unstale":"2021-03-15T22:26:08.133245+0000","last_undegraded":"2021-03-15T22:26:08.133245+0000","last_fullsized":"2021-03-15T22:26:08.133245+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'138","last_scrub_stamp":"2021-03-15T22:23:41.141218+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.141218+0000","log_size":319,"ondisk_log_size":319,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24627107,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":123,"num_read_kb":119123,"num_write":118,"num_write_kb":47137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8487583,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"e7","length":"1"},{"start":"100","length":"1"},{"start":"114","length":"1"},{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14b","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"437","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.119984+0000","last_change":"2021-03-15T22:25:33.129904+0000","last_active":"2021-03-15T22:26:11.119984+0000","last_peered":"2021-03-15T22:26:11.119984+0000","last_clean":"2021-03-15T22:26:11.119984+0000","last_became_active":"2021-03-15T22:25:33.129602+0000","last_became_peered":"2021-03-15T22:25:33.129602+0000","last_unstale":"2021-03-15T22:26:11.119984+0000","last_undegraded":"2021-03-15T22:26:11.119984+0000","last_fullsized":"2021-03-15T22:26:11.119984+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.931058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.931058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"254","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113172+0000","last_change":"2021-03-15T22:25:33.143239+0000","last_active":"2021-03-15T22:26:10.113172+0000","last_peered":"2021-03-15T22:26:10.113172+0000","last_clean":"2021-03-15T22:26:10.113172+0000","last_became_active":"2021-03-15T22:25:33.142133+0000","last_became_peered":"2021-03-15T22:25:33.142133+0000","last_unstale":"2021-03-15T22:26:10.113172+0000","last_undegraded":"2021-03-15T22:26:10.113172+0000","last_fullsized":"2021-03-15T22:26:10.113172+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.028082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.028082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"919","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:09.051870+0000","last_change":"2021-03-15T22:26:09.051870+0000","last_active":"2021-03-15T22:26:09.051870+0000","last_peered":"2021-03-15T22:26:09.051870+0000","last_clean":"2021-03-15T22:26:09.051870+0000","last_became_active":"2021-03-15T22:24:31.109631+0000","last_became_peered":"2021-03-15T22:24:31.109631+0000","last_unstale":"2021-03-15T22:26:09.051870+0000","last_undegraded":"2021-03-15T22:26:09.051870+0000","last_fullsized":"2021-03-15T22:26:09.051870+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:26:09.051794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.051794+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":480,"num_read_kb":475,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"178","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.120551+0000","last_change":"2021-03-15T22:24:37.977457+0000","last_active":"2021-03-15T22:26:11.120551+0000","last_peered":"2021-03-15T22:26:11.120551+0000","last_clean":"2021-03-15T22:26:11.120551+0000","last_became_active":"2021-03-15T22:24:31.100536+0000","last_became_peered":"2021-03-15T22:24:31.100536+0000","last_unstale":"2021-03-15T22:26:11.120551+0000","last_undegraded":"2021-03-15T22:26:11.120551+0000","last_fullsized":"2021-03-15T22:26:11.120551+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:37.977363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.977363+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"68","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.361542+0000","last_change":"2021-03-15T22:25:14.115071+0000","last_active":"2021-03-15T22:26:11.361542+0000","last_peered":"2021-03-15T22:26:11.361542+0000","last_clean":"2021-03-15T22:26:11.361542+0000","last_became_active":"2021-03-15T22:25:14.114729+0000","last_became_peered":"2021-03-15T22:25:14.114729+0000","last_unstale":"2021-03-15T22:26:11.361542+0000","last_undegraded":"2021-03-15T22:26:11.361542+0000","last_fullsized":"2021-03-15T22:26:11.361542+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.857011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.857011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"454'1180","reported_seq":"2950","reported_epoch":"457","state":"active+clean","last_fresh":"2021-03-15T22:26:12.128578+0000","last_change":"2021-03-15T22:26:09.315011+0000","last_active":"2021-03-15T22:26:12.128578+0000","last_peered":"2021-03-15T22:26:12.128578+0000","last_clean":"2021-03-15T22:26:12.128578+0000","last_became_active":"2021-03-15T22:25:45.240370+0000","last_became_peered":"2021-03-15T22:25:45.240370+0000","last_unstale":"2021-03-15T22:26:12.128578+0000","last_undegraded":"2021-03-15T22:26:12.128578+0000","last_fullsized":"2021-03-15T22:26:12.128578+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'777","last_scrub_stamp":"2021-03-15T22:24:40.640400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.640400+0000","log_size":1180,"ondisk_log_size":1180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59811886,"num_objects":27,"num_object_clones":20,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":2,"num_read":643,"num_read_kb":717774,"num_write":371,"num_write_kb":146446,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":125184016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"419","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.119405+0000","last_change":"2021-03-15T22:24:30.950198+0000","last_active":"2021-03-15T22:26:11.119405+0000","last_peered":"2021-03-15T22:26:11.119405+0000","last_clean":"2021-03-15T22:26:11.119405+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:26:11.119405+0000","last_undegraded":"2021-03-15T22:26:11.119405+0000","last_fullsized":"2021-03-15T22:26:11.119405+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.950135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.950135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"425","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.120064+0000","last_change":"2021-03-15T22:24:26.962034+0000","last_active":"2021-03-15T22:26:11.120064+0000","last_peered":"2021-03-15T22:26:11.120064+0000","last_clean":"2021-03-15T22:26:11.120064+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:26:11.120064+0000","last_undegraded":"2021-03-15T22:26:11.120064+0000","last_fullsized":"2021-03-15T22:26:11.120064+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.961978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.961978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"206","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.360737+0000","last_change":"2021-03-15T22:26:05.733910+0000","last_active":"2021-03-15T22:26:11.360737+0000","last_peered":"2021-03-15T22:26:11.360737+0000","last_clean":"2021-03-15T22:26:11.360737+0000","last_became_active":"2021-03-15T22:24:31.098665+0000","last_became_peered":"2021-03-15T22:24:31.098665+0000","last_unstale":"2021-03-15T22:26:11.360737+0000","last_undegraded":"2021-03-15T22:26:11.360737+0000","last_fullsized":"2021-03-15T22:26:11.360737+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:26:05.733849+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:26:05.733849+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"434","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.313968+0000","last_change":"2021-03-15T22:26:06.865328+0000","last_active":"2021-03-15T22:26:09.313968+0000","last_peered":"2021-03-15T22:26:09.313968+0000","last_clean":"2021-03-15T22:26:09.313968+0000","last_became_active":"2021-03-15T22:24:31.107629+0000","last_became_peered":"2021-03-15T22:24:31.107629+0000","last_unstale":"2021-03-15T22:26:09.313968+0000","last_undegraded":"2021-03-15T22:26:09.313968+0000","last_fullsized":"2021-03-15T22:26:09.313968+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:06.865257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:26:06.865257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"454'182","reported_seq":"1380","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.112990+0000","last_change":"2021-03-15T22:26:09.315968+0000","last_active":"2021-03-15T22:26:10.112990+0000","last_peered":"2021-03-15T22:26:10.112990+0000","last_clean":"2021-03-15T22:26:10.112990+0000","last_became_active":"2021-03-15T22:25:37.112715+0000","last_became_peered":"2021-03-15T22:25:37.112715+0000","last_unstale":"2021-03-15T22:26:10.112990+0000","last_undegraded":"2021-03-15T22:26:10.112990+0000","last_fullsized":"2021-03-15T22:26:10.112990+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'104","last_scrub_stamp":"2021-03-15T22:24:02.949625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:02.949625+0000","log_size":182,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16688864,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":81,"num_read_kb":100355,"num_write":60,"num_write_kb":24058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":14412391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"453","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.308349+0000","last_change":"2021-03-15T22:25:14.112119+0000","last_active":"2021-03-15T22:26:09.308349+0000","last_peered":"2021-03-15T22:26:09.308349+0000","last_clean":"2021-03-15T22:26:09.308349+0000","last_became_active":"2021-03-15T22:25:14.111797+0000","last_became_peered":"2021-03-15T22:25:14.111797+0000","last_unstale":"2021-03-15T22:26:09.308349+0000","last_undegraded":"2021-03-15T22:26:09.308349+0000","last_fullsized":"2021-03-15T22:26:09.308349+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.480279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"435","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.361263+0000","last_change":"2021-03-15T22:25:33.141491+0000","last_active":"2021-03-15T22:26:11.361263+0000","last_peered":"2021-03-15T22:26:11.361263+0000","last_clean":"2021-03-15T22:26:11.361263+0000","last_became_active":"2021-03-15T22:25:33.141147+0000","last_became_peered":"2021-03-15T22:25:33.141147+0000","last_unstale":"2021-03-15T22:26:11.361263+0000","last_undegraded":"2021-03-15T22:26:11.361263+0000","last_fullsized":"2021-03-15T22:26:11.361263+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.769850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.769850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"553","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.119870+0000","last_change":"2021-03-15T22:26:06.698784+0000","last_active":"2021-03-15T22:26:11.119870+0000","last_peered":"2021-03-15T22:26:11.119870+0000","last_clean":"2021-03-15T22:26:11.119870+0000","last_became_active":"2021-03-15T22:25:14.114107+0000","last_became_peered":"2021-03-15T22:25:14.114107+0000","last_unstale":"2021-03-15T22:26:11.119870+0000","last_undegraded":"2021-03-15T22:26:11.119870+0000","last_fullsized":"2021-03-15T22:26:11.119870+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:26:06.698716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:06.698716+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"356'3","reported_seq":"150","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.313972+0000","last_change":"2021-03-15T22:26:02.781579+0000","last_active":"2021-03-15T22:26:09.313972+0000","last_peered":"2021-03-15T22:26:09.313972+0000","last_clean":"2021-03-15T22:26:09.313972+0000","last_became_active":"2021-03-15T22:24:31.105948+0000","last_became_peered":"2021-03-15T22:24:31.105948+0000","last_unstale":"2021-03-15T22:26:09.313972+0000","last_undegraded":"2021-03-15T22:26:09.313972+0000","last_fullsized":"2021-03-15T22:26:09.313972+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'3","last_scrub_stamp":"2021-03-15T22:26:02.781478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:26:02.781478+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"454'315","reported_seq":"1450","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.123360+0000","last_change":"2021-03-15T22:26:09.314876+0000","last_active":"2021-03-15T22:26:10.123360+0000","last_peered":"2021-03-15T22:26:10.123360+0000","last_clean":"2021-03-15T22:26:10.123360+0000","last_became_active":"2021-03-15T22:23:00.057259+0000","last_became_peered":"2021-03-15T22:23:00.057259+0000","last_unstale":"2021-03-15T22:26:10.123360+0000","last_undegraded":"2021-03-15T22:26:10.123360+0000","last_fullsized":"2021-03-15T22:26:10.123360+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'202","last_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.006150+0000","log_size":315,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15122431,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":128,"num_read_kb":122014,"num_write":101,"num_write_kb":38922,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":6765552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"413","reported_epoch":"452","state":"active+clean","last_fresh":"2021-03-15T22:26:07.127150+0000","last_change":"2021-03-15T22:25:16.234631+0000","last_active":"2021-03-15T22:26:07.127150+0000","last_peered":"2021-03-15T22:26:07.127150+0000","last_clean":"2021-03-15T22:26:07.127150+0000","last_became_active":"2021-03-15T22:23:52.260549+0000","last_became_peered":"2021-03-15T22:23:52.260549+0000","last_unstale":"2021-03-15T22:26:07.127150+0000","last_undegraded":"2021-03-15T22:26:07.127150+0000","last_fullsized":"2021-03-15T22:26:07.127150+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.234565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.234565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"119","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.120412+0000","last_change":"2021-03-15T22:25:02.868460+0000","last_active":"2021-03-15T22:26:11.120412+0000","last_peered":"2021-03-15T22:26:11.120412+0000","last_clean":"2021-03-15T22:26:11.120412+0000","last_became_active":"2021-03-15T22:24:31.099610+0000","last_became_peered":"2021-03-15T22:24:31.099610+0000","last_unstale":"2021-03-15T22:26:11.120412+0000","last_undegraded":"2021-03-15T22:26:11.120412+0000","last_fullsized":"2021-03-15T22:26:11.120412+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.868399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.868399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"460","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113374+0000","last_change":"2021-03-15T22:26:09.008669+0000","last_active":"2021-03-15T22:26:10.113374+0000","last_peered":"2021-03-15T22:26:10.113374+0000","last_clean":"2021-03-15T22:26:10.113374+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:26:10.113374+0000","last_undegraded":"2021-03-15T22:26:10.113374+0000","last_fullsized":"2021-03-15T22:26:10.113374+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:26:09.008589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.008589+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"421'6","reported_seq":"58","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.308150+0000","last_change":"2021-03-15T22:25:33.171536+0000","last_active":"2021-03-15T22:26:09.308150+0000","last_peered":"2021-03-15T22:26:09.308150+0000","last_clean":"2021-03-15T22:26:09.308150+0000","last_became_active":"2021-03-15T22:25:33.130973+0000","last_became_peered":"2021-03-15T22:25:33.130973+0000","last_unstale":"2021-03-15T22:26:09.308150+0000","last_undegraded":"2021-03-15T22:26:09.308150+0000","last_fullsized":"2021-03-15T22:26:09.308150+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"300'4","last_scrub_stamp":"2021-03-15T22:23:35.100079+0000","last_deep_scrub":"300'4","last_deep_scrub_stamp":"2021-03-15T22:23:35.100079+0000","last_clean_scrub_stamp":"2021-03-15T22:23:35.100079+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"454'545","reported_seq":"1767","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.205342+0000","last_change":"2021-03-15T22:26:09.311810+0000","last_active":"2021-03-15T22:26:10.205342+0000","last_peered":"2021-03-15T22:26:10.205342+0000","last_clean":"2021-03-15T22:26:10.205342+0000","last_became_active":"2021-03-15T22:23:01.072357+0000","last_became_peered":"2021-03-15T22:23:01.072357+0000","last_unstale":"2021-03-15T22:26:10.205342+0000","last_undegraded":"2021-03-15T22:26:10.205342+0000","last_fullsized":"2021-03-15T22:26:10.205342+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'302","last_scrub_stamp":"2021-03-15T22:23:59.467761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:59.467761+0000","log_size":545,"ondisk_log_size":545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31558966,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":244,"num_read_kb":259099,"num_write":158,"num_write_kb":61521,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31233387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"439","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.112956+0000","last_change":"2021-03-15T22:25:47.117313+0000","last_active":"2021-03-15T22:26:10.112956+0000","last_peered":"2021-03-15T22:26:10.112956+0000","last_clean":"2021-03-15T22:26:10.112956+0000","last_became_active":"2021-03-15T22:25:33.136621+0000","last_became_peered":"2021-03-15T22:25:33.136621+0000","last_unstale":"2021-03-15T22:26:10.112956+0000","last_undegraded":"2021-03-15T22:26:10.112956+0000","last_fullsized":"2021-03-15T22:26:10.112956+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:47.117252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:47.117252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"223","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.308961+0000","last_change":"2021-03-15T22:25:12.911203+0000","last_active":"2021-03-15T22:26:09.308961+0000","last_peered":"2021-03-15T22:26:09.308961+0000","last_clean":"2021-03-15T22:26:09.308961+0000","last_became_active":"2021-03-15T22:24:31.091206+0000","last_became_peered":"2021-03-15T22:24:31.091206+0000","last_unstale":"2021-03-15T22:26:09.308961+0000","last_undegraded":"2021-03-15T22:26:09.308961+0000","last_fullsized":"2021-03-15T22:26:09.308961+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.911140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.911140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"73","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.307983+0000","last_change":"2021-03-15T22:25:33.758535+0000","last_active":"2021-03-15T22:26:09.307983+0000","last_peered":"2021-03-15T22:26:09.307983+0000","last_clean":"2021-03-15T22:26:09.307983+0000","last_became_active":"2021-03-15T22:25:33.149362+0000","last_became_peered":"2021-03-15T22:25:33.149362+0000","last_unstale":"2021-03-15T22:26:09.307983+0000","last_undegraded":"2021-03-15T22:26:09.307983+0000","last_fullsized":"2021-03-15T22:26:09.307983+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:24:01.896988+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.896988+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"447","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.112831+0000","last_change":"2021-03-15T22:25:33.143430+0000","last_active":"2021-03-15T22:26:10.112831+0000","last_peered":"2021-03-15T22:26:10.112831+0000","last_clean":"2021-03-15T22:26:10.112831+0000","last_became_active":"2021-03-15T22:25:33.142712+0000","last_became_peered":"2021-03-15T22:25:33.142712+0000","last_unstale":"2021-03-15T22:26:10.112831+0000","last_undegraded":"2021-03-15T22:26:10.112831+0000","last_fullsized":"2021-03-15T22:26:10.112831+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.989313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.989313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"454'506","reported_seq":"1753","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.212369+0000","last_change":"2021-03-15T22:26:09.320595+0000","last_active":"2021-03-15T22:26:10.212369+0000","last_peered":"2021-03-15T22:26:10.212369+0000","last_clean":"2021-03-15T22:26:10.212369+0000","last_became_active":"2021-03-15T22:25:36.118668+0000","last_became_peered":"2021-03-15T22:25:36.118668+0000","last_unstale":"2021-03-15T22:26:10.212369+0000","last_undegraded":"2021-03-15T22:26:10.212369+0000","last_fullsized":"2021-03-15T22:26:10.212369+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'370","last_scrub_stamp":"2021-03-15T22:25:07.414191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.414191+0000","log_size":506,"ondisk_log_size":506,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30242362,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":193,"num_read_kb":178069,"num_write":149,"num_write_kb":59303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25937939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"420","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.361076+0000","last_change":"2021-03-15T22:24:31.094621+0000","last_active":"2021-03-15T22:26:11.361076+0000","last_peered":"2021-03-15T22:26:11.361076+0000","last_clean":"2021-03-15T22:26:11.361076+0000","last_became_active":"2021-03-15T22:24:31.094299+0000","last_became_peered":"2021-03-15T22:24:31.094299+0000","last_unstale":"2021-03-15T22:26:11.361076+0000","last_undegraded":"2021-03-15T22:26:11.361076+0000","last_fullsized":"2021-03-15T22:26:11.361076+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.936738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.936738+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"391'7","reported_seq":"572","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113416+0000","last_change":"2021-03-15T22:25:46.103567+0000","last_active":"2021-03-15T22:26:10.113416+0000","last_peered":"2021-03-15T22:26:10.113416+0000","last_clean":"2021-03-15T22:26:10.113416+0000","last_became_active":"2021-03-15T22:25:03.403350+0000","last_became_peered":"2021-03-15T22:25:03.403350+0000","last_unstale":"2021-03-15T22:26:10.113416+0000","last_undegraded":"2021-03-15T22:26:10.113416+0000","last_fullsized":"2021-03-15T22:26:10.113416+0000","mapping_epoch":387,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'7","last_scrub_stamp":"2021-03-15T22:25:46.103504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.103504+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"52","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.308065+0000","last_change":"2021-03-15T22:25:44.867726+0000","last_active":"2021-03-15T22:26:09.308065+0000","last_peered":"2021-03-15T22:26:09.308065+0000","last_clean":"2021-03-15T22:26:09.308065+0000","last_became_active":"2021-03-15T22:25:33.148614+0000","last_became_peered":"2021-03-15T22:25:33.148614+0000","last_unstale":"2021-03-15T22:26:09.308065+0000","last_undegraded":"2021-03-15T22:26:09.308065+0000","last_fullsized":"2021-03-15T22:26:09.308065+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:44.867633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:44.867633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"484","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.121107+0000","last_change":"2021-03-15T22:25:46.725552+0000","last_active":"2021-03-15T22:26:08.121107+0000","last_peered":"2021-03-15T22:26:08.121107+0000","last_clean":"2021-03-15T22:26:08.121107+0000","last_became_active":"2021-03-15T22:22:56.057343+0000","last_became_peered":"2021-03-15T22:22:56.057343+0000","last_unstale":"2021-03-15T22:26:08.121107+0000","last_undegraded":"2021-03-15T22:26:08.121107+0000","last_fullsized":"2021-03-15T22:26:08.121107+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:25:46.725489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.725489+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"454'117","reported_seq":"1279","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.212596+0000","last_change":"2021-03-15T22:26:09.319432+0000","last_active":"2021-03-15T22:26:10.212596+0000","last_peered":"2021-03-15T22:26:10.212596+0000","last_clean":"2021-03-15T22:26:10.212596+0000","last_became_active":"2021-03-15T22:24:31.104356+0000","last_became_peered":"2021-03-15T22:24:31.104356+0000","last_unstale":"2021-03-15T22:26:10.212596+0000","last_undegraded":"2021-03-15T22:26:10.212596+0000","last_fullsized":"2021-03-15T22:26:10.212596+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'77","last_scrub_stamp":"2021-03-15T22:25:32.490694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.490694+0000","log_size":117,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10088531,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":89,"num_read_kb":89949,"num_write":34,"num_write_kb":13772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1551588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"47","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.307551+0000","last_change":"2021-03-15T22:25:33.148157+0000","last_active":"2021-03-15T22:26:09.307551+0000","last_peered":"2021-03-15T22:26:09.307551+0000","last_clean":"2021-03-15T22:26:09.307551+0000","last_became_active":"2021-03-15T22:25:33.147717+0000","last_became_peered":"2021-03-15T22:25:33.147717+0000","last_unstale":"2021-03-15T22:26:09.307551+0000","last_undegraded":"2021-03-15T22:26:09.307551+0000","last_fullsized":"2021-03-15T22:26:09.307551+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.372535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.372535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"147'5","reported_seq":"500","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:12.091644+0000","last_change":"2021-03-15T22:24:35.715464+0000","last_active":"2021-03-15T22:26:12.091644+0000","last_peered":"2021-03-15T22:26:12.091644+0000","last_clean":"2021-03-15T22:26:12.091644+0000","last_became_active":"2021-03-15T22:21:02.017066+0000","last_became_peered":"2021-03-15T22:21:02.017066+0000","last_unstale":"2021-03-15T22:26:12.091644+0000","last_undegraded":"2021-03-15T22:26:12.091644+0000","last_fullsized":"2021-03-15T22:26:12.091644+0000","mapping_epoch":60,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'5","last_scrub_stamp":"2021-03-15T22:24:35.715399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:35.715399+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"424","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113074+0000","last_change":"2021-03-15T22:24:34.991087+0000","last_active":"2021-03-15T22:26:10.113074+0000","last_peered":"2021-03-15T22:26:10.113074+0000","last_clean":"2021-03-15T22:26:10.113074+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:26:10.113074+0000","last_undegraded":"2021-03-15T22:26:10.113074+0000","last_fullsized":"2021-03-15T22:26:10.113074+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.991018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.991018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"505","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113145+0000","last_change":"2021-03-15T22:24:55.128840+0000","last_active":"2021-03-15T22:26:10.113145+0000","last_peered":"2021-03-15T22:26:10.113145+0000","last_clean":"2021-03-15T22:26:10.113145+0000","last_became_active":"2021-03-15T22:24:55.090544+0000","last_became_peered":"2021-03-15T22:24:55.090544+0000","last_unstale":"2021-03-15T22:26:10.113145+0000","last_undegraded":"2021-03-15T22:26:10.113145+0000","last_fullsized":"2021-03-15T22:26:10.113145+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:38.685706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:38.685706+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"454'154","reported_seq":"1422","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.312427+0000","last_change":"2021-03-15T22:26:09.312824+0000","last_active":"2021-03-15T22:26:09.312427+0000","last_peered":"2021-03-15T22:26:09.312427+0000","last_clean":"2021-03-15T22:26:09.312427+0000","last_became_active":"2021-03-15T22:25:37.119078+0000","last_became_peered":"2021-03-15T22:25:37.119078+0000","last_unstale":"2021-03-15T22:26:09.312427+0000","last_undegraded":"2021-03-15T22:26:09.312427+0000","last_fullsized":"2021-03-15T22:26:09.312427+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'115","last_scrub_stamp":"2021-03-15T22:25:10.616929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.616929+0000","log_size":154,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6574330,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":110,"num_read_kb":136588,"num_write":54,"num_write_kb":14217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":16774328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"e7","length":"1"},{"start":"100","length":"1"},{"start":"114","length":"1"},{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14b","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"439","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.121300+0000","last_change":"2021-03-15T22:26:03.754920+0000","last_active":"2021-03-15T22:26:08.121300+0000","last_peered":"2021-03-15T22:26:08.121300+0000","last_clean":"2021-03-15T22:26:08.121300+0000","last_became_active":"2021-03-15T22:25:33.134275+0000","last_became_peered":"2021-03-15T22:25:33.134275+0000","last_unstale":"2021-03-15T22:26:08.121300+0000","last_undegraded":"2021-03-15T22:26:08.121300+0000","last_fullsized":"2021-03-15T22:26:08.121300+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.754853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.754853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"448","reported_epoch":"456","state":"active+clean","last_fresh":"2021-03-15T22:26:11.120142+0000","last_change":"2021-03-15T22:24:31.107075+0000","last_active":"2021-03-15T22:26:11.120142+0000","last_peered":"2021-03-15T22:26:11.120142+0000","last_clean":"2021-03-15T22:26:11.120142+0000","last_became_active":"2021-03-15T22:24:31.106634+0000","last_became_peered":"2021-03-15T22:24:31.106634+0000","last_unstale":"2021-03-15T22:26:11.120142+0000","last_undegraded":"2021-03-15T22:26:11.120142+0000","last_fullsized":"2021-03-15T22:26:11.120142+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:42.177422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.177422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"432","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.313776+0000","last_change":"2021-03-15T22:26:03.831424+0000","last_active":"2021-03-15T22:26:09.313776+0000","last_peered":"2021-03-15T22:26:09.313776+0000","last_clean":"2021-03-15T22:26:09.313776+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:26:09.313776+0000","last_undegraded":"2021-03-15T22:26:09.313776+0000","last_fullsized":"2021-03-15T22:26:09.313776+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.831371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.831371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"115'33","reported_seq":"470","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113061+0000","last_change":"2021-03-15T22:25:04.028853+0000","last_active":"2021-03-15T22:26:10.113061+0000","last_peered":"2021-03-15T22:26:10.113061+0000","last_clean":"2021-03-15T22:26:10.113061+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:26:10.113061+0000","last_undegraded":"2021-03-15T22:26:10.113061+0000","last_fullsized":"2021-03-15T22:26:10.113061+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'33","last_scrub_stamp":"2021-03-15T22:25:04.028780+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:25:04.028780+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"446'390","reported_seq":"1765","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.204864+0000","last_change":"2021-03-15T22:26:09.312877+0000","last_active":"2021-03-15T22:26:10.204864+0000","last_peered":"2021-03-15T22:26:10.204864+0000","last_clean":"2021-03-15T22:26:10.204864+0000","last_became_active":"2021-03-15T22:25:39.143407+0000","last_became_peered":"2021-03-15T22:25:39.143407+0000","last_unstale":"2021-03-15T22:26:10.204864+0000","last_undegraded":"2021-03-15T22:26:10.204864+0000","last_fullsized":"2021-03-15T22:26:10.204864+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'343","last_scrub_stamp":"2021-03-15T22:23:26.791043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.791043+0000","log_size":390,"ondisk_log_size":390,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10012393,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":187,"num_read_kb":156186,"num_write":144,"num_write_kb":55943,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":38892785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"117","length":"1"},{"start":"12e","length":"1"},{"start":"130","length":"1"},{"start":"14d","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"416","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.113433+0000","last_change":"2021-03-15T22:24:40.910630+0000","last_active":"2021-03-15T22:26:10.113433+0000","last_peered":"2021-03-15T22:26:10.113433+0000","last_clean":"2021-03-15T22:26:10.113433+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:26:10.113433+0000","last_undegraded":"2021-03-15T22:26:10.113433+0000","last_fullsized":"2021-03-15T22:26:10.113433+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.910565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.910565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"155'4","reported_seq":"540","reported_epoch":"453","state":"active+clean","last_fresh":"2021-03-15T22:26:08.120720+0000","last_change":"2021-03-15T22:24:29.903505+0000","last_active":"2021-03-15T22:26:08.120720+0000","last_peered":"2021-03-15T22:26:08.120720+0000","last_clean":"2021-03-15T22:26:08.120720+0000","last_became_active":"2021-03-15T22:21:10.013838+0000","last_became_peered":"2021-03-15T22:21:10.013838+0000","last_unstale":"2021-03-15T22:26:08.120720+0000","last_undegraded":"2021-03-15T22:26:08.120720+0000","last_fullsized":"2021-03-15T22:26:08.120720+0000","mapping_epoch":60,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'4","last_scrub_stamp":"2021-03-15T22:24:29.903442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.903442+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"133","reported_epoch":"454","state":"active+clean","last_fresh":"2021-03-15T22:26:09.307892+0000","last_change":"2021-03-15T22:25:02.791176+0000","last_active":"2021-03-15T22:26:09.307892+0000","last_peered":"2021-03-15T22:26:09.307892+0000","last_clean":"2021-03-15T22:26:09.307892+0000","last_became_active":"2021-03-15T22:24:31.097434+0000","last_became_peered":"2021-03-15T22:24:31.097434+0000","last_unstale":"2021-03-15T22:26:09.307892+0000","last_undegraded":"2021-03-15T22:26:09.307892+0000","last_fullsized":"2021-03-15T22:26:09.307892+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.791108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:06.573181+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.791108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"502","reported_epoch":"455","state":"active+clean","last_fresh":"2021-03-15T22:26:10.114080+0000","last_change":"2021-03-15T22:26:07.977706+0000","last_active":"2021-03-15T22:26:10.114080+0000","last_peered":"2021-03-15T22:26:10.114080+0000","last_clean":"2021-03-15T22:26:10.114080+0000","last_became_active":"2021-03-15T22:22:56.053800+0000","last_became_peered":"2021-03-15T22:22:56.053800+0000","last_unstale":"2021-03-15T22:26:10.114080+0000","last_undegraded":"2021-03-15T22:26:10.114080+0000","last_fullsized":"2021-03-15T22:26:10.114080+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:26:07.977641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.977641+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":396353755,"num_objects":176,"num_object_clones":128,"num_object_copies":528,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":176,"num_whiteouts":12,"num_read":3304,"num_read_kb":3350221,"num_write":2434,"num_write_kb":917209,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1197,"num_bytes_recovered":514315662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":891334656,"data_stored":1224704581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7315,"ondisk_log_size":7315,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":480,"num_read_kb":475,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1061,"num_read_kb":566,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":37148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2232320,"data_stored":80092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299495,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":125164,"kb_used_data":105828,"kb_used_omap":0,"kb_used_meta":19328,"kb_avail":93619988,"statfs":{"total":95995035648,"available":95866867712,"internally_reserved":0,"allocated":108367872,"data_stored":145545786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19791872},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.49299999999999999},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.873,"5min":2.73,"15min":2.73},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.495,"15min":0.497},"min":{"1min":0.253,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70899999999999996,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.377}]},{"osd":1,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51400000000000001,"15min":0.48899999999999999},"min":{"1min":0.29299999999999998,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":0.85199999999999998,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.27400000000000002},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.497,"15min":0.49399999999999999},"min":{"1min":0.29099999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.5149999999999999,"5min":1.877,"15min":1.877},"last":0.23400000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.80700000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.32400000000000001,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":0.73099999999999998,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.41199999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48099999999999998,"15min":0.53600000000000003},"min":{"1min":0.33000000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.83299999999999996,"5min":0.95999999999999996,"15min":2.0339999999999998},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53200000000000003,"15min":0.54900000000000004},"min":{"1min":0.36799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69299999999999995,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.36199999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.45100000000000001,"15min":0.436},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.63900000000000001,"5min":1.835,"15min":1.835},"last":0.433},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.44800000000000001},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69699999999999995,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.44600000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50900000000000001,"15min":0.46000000000000002},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.72299999999999998,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.31,"5min":0.255,"15min":0.255},"max":{"1min":0.71199999999999997,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.46999999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53400000000000003,"15min":0.51200000000000001},"min":{"1min":0.32600000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76000000000000001,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.496},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.51100000000000001},"min":{"1min":0.36099999999999999,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.68000000000000005,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.39200000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528427,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":132692,"kb_used_data":99404,"kb_used_omap":0,"kb_used_meta":33280,"kb_avail":93612460,"statfs":{"total":95995035648,"available":95859159040,"internally_reserved":0,"allocated":101789696,"data_stored":137896810,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34078720},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47499999999999998,"15min":0.48799999999999999},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.68100000000000005,"5min":0.995,"15min":0.995},"last":0.61199999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.46800000000000003,"15min":0.48599999999999999},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.53000000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.49099999999999999,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.254,"15min":0.254},"max":{"1min":0.69399999999999995,"5min":1.05,"15min":1.05},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.33300000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68000000000000005,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.59299999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.317,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65200000000000002,"5min":0.77600000000000002,"15min":0.80800000000000005},"last":0.5},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48899999999999999,"15min":0.498},"min":{"1min":0.35399999999999998,"5min":0.32800000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.56899999999999995}]},{"osd":3,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.35199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.90400000000000003,"5min":1.026,"15min":1.026},"last":0.65600000000000003},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51100000000000001,"15min":0.52400000000000002},"min":{"1min":0.33600000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.54000000000000004}]},{"osd":4,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.44800000000000001,"15min":0.41799999999999998},"min":{"1min":0.28000000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51500000000000001,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.55000000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.501,"15min":0.46200000000000002},"min":{"1min":0.32300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.89200000000000002,"5min":1.135,"15min":1.135},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53200000000000003,"15min":0.48999999999999999},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.51700000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:26:01 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.78300000000000003,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.55900000000000005},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.54500000000000004},"min":{"1min":0.42999999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.92600000000000005,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.60199999999999998}]}]},{"osd":1,"up_from":13,"seq":55834574973,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172748,"kb_used_data":145988,"kb_used_omap":0,"kb_used_meta":26752,"kb_avail":93572404,"statfs":{"total":95995035648,"available":95818141696,"internally_reserved":0,"allocated":149491712,"data_stored":207032799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27394048},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.34499999999999997,"15min":0.52700000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.48899999999999999,"5min":1.371,"15min":54.427999999999997},"last":0.28100000000000003},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.54400000000000004},"min":{"1min":0.20999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.48199999999999998,"5min":1.23,"15min":54.356999999999999},"last":0.255}]},{"osd":2,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.54300000000000004,"5min":1.1220000000000001,"15min":2.4100000000000001},"last":0.31},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.38600000000000001},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.502,"5min":0.69799999999999995,"15min":2.0760000000000001},"last":0.215}]},{"osd":3,"last update":"Mon Mar 15 22:26:01 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38500000000000001,"15min":0.41499999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.222},"max":{"1min":0.53400000000000003,"5min":1.4910000000000001,"15min":1.796},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.437,"5min":0.41799999999999998,"15min":0.46000000000000002},"min":{"1min":0.32200000000000001,"5min":0.23000000000000001,"15min":0.20999999999999999},"max":{"1min":0.56799999999999995,"5min":1.135,"15min":1.7150000000000001},"last":0.29399999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:26:11 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.309,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":0.69899999999999995,"5min":3.3439999999999999,"15min":3.3439999999999999},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.37,"5min":0.253,"15min":0.253},"max":{"1min":0.81200000000000006,"5min":4.6139999999999999,"15min":4.6139999999999999},"last":0.37}]},{"osd":5,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.627},"min":{"1min":0.42199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.93400000000000005,"5min":1.524,"15min":2.153},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58499999999999996,"15min":0.95899999999999996},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88900000000000001,"5min":1.1950000000000001,"15min":15.472},"last":0.46999999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61099999999999999,"15min":0.65600000000000003},"min":{"1min":0.47599999999999998,"5min":0.36899999999999999,"15min":0.33200000000000002},"max":{"1min":0.91800000000000004,"5min":1.276,"15min":1.75},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.3580000000000001},"last":0.41699999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:26:05 2021","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.63500000000000001,"15min":0.84299999999999997},"min":{"1min":0.44800000000000001,"5min":0.377,"15min":0.316},"max":{"1min":0.78800000000000003,"5min":2.1219999999999999,"15min":8.8100000000000005},"last":0.501},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58599999999999997,"15min":0.61799999999999999},"min":{"1min":0.38400000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82599999999999996,"5min":1.0329999999999999,"15min":1.325},"last":0.433}]}]},{"osd":0,"up_from":0,"seq":34359738495,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411449,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":126304,"kb_used_data":99032,"kb_used_omap":0,"kb_used_meta":27264,"kb_avail":93618848,"statfs":{"total":95995035648,"available":95865700352,"internally_reserved":0,"allocated":101408768,"data_stored":137214009,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27918336},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.246,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.45700000000000002,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.437,"15min":0.39500000000000002},"min":{"1min":0.246,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":5.5250000000000004,"5min":5.5250000000000004,"15min":5.5250000000000004},"last":0.34300000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.42599999999999999,"15min":0.39600000000000002},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.60999999999999999,"5min":5.4489999999999998,"15min":5.4489999999999998},"last":0.245},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51800000000000002,"15min":0.46600000000000003},"min":{"1min":0.27400000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.3740000000000001,"5min":11.659000000000001,"15min":11.659000000000001},"last":0.30599999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:25:58 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.38400000000000001,"15min":0.41399999999999998},"min":{"1min":0.23699999999999999,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":1.2010000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.47699999999999998},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.24099999999999999},"max":{"1min":1.28,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:25:09 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56999999999999995,"15min":0.57499999999999996},"min":{"1min":0.40999999999999998,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.73899999999999999,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.54500000000000004,"15min":0.55600000000000005},"min":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.60199999999999998,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.41499999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:25:36 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.38400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75600000000000001,"5min":1.954,"15min":1.954},"last":0.505},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.745,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":1.875,"15min":1.875},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.435,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.3520000000000001,"5min":1.609,"15min":1.609},"last":0.46700000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:25:09 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.57799999999999996,"15min":0.55700000000000005},"min":{"1min":0.40999999999999998,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.66900000000000004,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.432},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.42799999999999999,"5min":0.40600000000000003,"15min":0.38700000000000001},"max":{"1min":0.64700000000000002,"5min":1.923,"15min":1.923},"last":0.44800000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149813,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":253052,"kb_used_data":225524,"kb_used_omap":0,"kb_used_meta":27520,"kb_avail":93492100,"statfs":{"total":95995035648,"available":95735910400,"internally_reserved":0,"allocated":230936576,"data_stored":303417357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28180480},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35199999999999998,"15min":0.35499999999999998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.505,"5min":2.625,"15min":2.625},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.36599999999999999,"15min":0.374},"min":{"1min":0.23499999999999999,"5min":0.20000000000000001,"15min":0.189},"max":{"1min":0.48199999999999998,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.30599999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.44600000000000001,"15min":0.42799999999999999},"min":{"1min":0.23400000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.51400000000000001,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.377},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.45900000000000002,"15min":0.42999999999999999},"min":{"1min":0.24199999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47299999999999998,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51800000000000002,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.42199999999999999,"15min":0.39900000000000002},"min":{"1min":0.27000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54100000000000004,"5min":7.0860000000000003,"15min":7.0860000000000003},"last":0.34699999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:25:10 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.55500000000000005,"15min":0.52400000000000002},"min":{"1min":0.32600000000000001,"5min":0.31,"15min":0.31},"max":{"1min":0.66800000000000004,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54600000000000004,"15min":0.53000000000000003},"min":{"1min":0.38400000000000001,"5min":0.318,"15min":0.30399999999999999},"max":{"1min":0.69299999999999995,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.51900000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.54500000000000004},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.62,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":0.57499999999999996}]},{"osd":6,"last update":"Mon Mar 15 22:25:43 2021","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68500000000000005,"5min":2.6499999999999999,"15min":2.6499999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70999999999999996,"5min":1.268,"15min":1.268},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.38300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69599999999999995,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57799999999999996,"15min":0.57299999999999995},"min":{"1min":0.41399999999999998,"5min":0.39500000000000002,"15min":0.377},"max":{"1min":1.0129999999999999,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.59899999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986290,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":92552,"kb_used_data":75776,"kb_used_omap":0,"kb_used_meta":16768,"kb_avail":93652600,"statfs":{"total":95995035648,"available":95900262400,"internally_reserved":0,"allocated":77594624,"data_stored":102667117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17170432},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.442,"15min":0.49199999999999999},"min":{"1min":0.308,"5min":0.24199999999999999,"15min":0.19500000000000001},"max":{"1min":0.73099999999999998,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.502},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48499999999999999,"15min":0.48999999999999999},"min":{"1min":0.34999999999999998,"5min":0.29099999999999998,"15min":0.251},"max":{"1min":0.66600000000000004,"5min":0.72199999999999998,"15min":0.74199999999999999},"last":0.39900000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.47599999999999998,"15min":0.5},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.23599999999999999},"max":{"1min":0.68100000000000005,"5min":1.631,"15min":1.631},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.5,"15min":0.51800000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.26800000000000002},"max":{"1min":0.73999999999999999,"5min":2.2989999999999999,"15min":2.2989999999999999},"last":0.48299999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.48999999999999999,"15min":0.51100000000000001},"min":{"1min":0.40300000000000002,"5min":0.32300000000000001,"15min":0.313},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.76100000000000001},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.48599999999999999,"15min":0.504},"min":{"1min":0.29399999999999998,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":1.1910000000000001},"last":0.30599999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:25:17 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.52600000000000002,"15min":0.55000000000000004},"min":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"max":{"1min":0.68899999999999995,"5min":0.75,"15min":0.98799999999999999},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.51500000000000001,"15min":0.55800000000000005},"min":{"1min":0.40200000000000002,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.68899999999999995,"5min":0.69199999999999995,"15min":0.99399999999999999},"last":0.53900000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:25:28 2021","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.41799999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.49299999999999999,"15min":0.441},"min":{"1min":0.36099999999999999,"5min":0.29699999999999999,"15min":0.28799999999999998},"max":{"1min":0.64400000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.51600000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46800000000000003,"15min":0.437},"min":{"1min":0.29099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65400000000000003,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53900000000000003,"15min":0.51400000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.71499999999999997,"5min":0.78400000000000003,"15min":0.84799999999999998},"last":0.56200000000000006}]},{"osd":7,"last update":"Mon Mar 15 22:26:00 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.503,"15min":0.48999999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67000000000000004,"5min":0.73799999999999999,"15min":0.80600000000000005},"last":0.626},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.33500000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.75600000000000001,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.55300000000000005}]}]},{"osd":5,"up_from":38,"seq":163208757359,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":171568,"kb_used_data":146536,"kb_used_omap":0,"kb_used_meta":25024,"kb_avail":93573584,"statfs":{"total":95995035648,"available":95819350016,"internally_reserved":0,"allocated":150052864,"data_stored":206993884,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":25624576},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48799999999999999,"15min":0.48299999999999998},"min":{"1min":0.29499999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63500000000000001,"5min":2.855,"15min":2.855},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.498},"min":{"1min":0.34699999999999998,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.75700000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.378}]},{"osd":1,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.33100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.72999999999999998,"5min":2.665,"15min":2.665},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.72099999999999997,"5min":0.81999999999999995,"15min":0.83999999999999997},"last":0.439}]},{"osd":2,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.39000000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61799999999999999,"5min":2.827,"15min":2.827},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.51400000000000001},"min":{"1min":0.29899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":1.137}]},{"osd":3,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.55100000000000005,"15min":0.51800000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54800000000000004,"15min":0.53300000000000003},"min":{"1min":0.39400000000000002,"5min":0.34100000000000003,"15min":0.33200000000000002},"max":{"1min":0.65500000000000003,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.47599999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46999999999999997,"15min":0.40899999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24099999999999999},"max":{"1min":0.70899999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.496,"5min":0.505,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.25},"max":{"1min":0.748,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.65600000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:25:45 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49399999999999999,"15min":0.47199999999999998},"min":{"1min":0.32300000000000001,"5min":0.29199999999999998,"15min":0.28499999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.502},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56599999999999995,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.32800000000000001},"max":{"1min":0.78600000000000003,"5min":1.327,"15min":1.327},"last":1.1739999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:26:02 2021","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.55500000000000005,"15min":0.52200000000000002},"min":{"1min":0.35399999999999998,"5min":0.34699999999999998,"15min":0.30499999999999999},"max":{"1min":0.73099999999999998,"5min":1.3,"15min":1.3},"last":1.198},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.56000000000000005,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.307,"15min":0.307},"max":{"1min":0.77100000000000002,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.746}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":9520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":14458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":8356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":6840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":8774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5443584,"data_stored":5395029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":144515072,"data_stored":203947352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":96653312,"data_stored":134159726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":226050048,"data_stored":300350858,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":72777728,"data_stored":99603253,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":145301504,"data_stored":203916021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":96993280,"data_stored":134832805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":103600128,"data_stored":142499537,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:26:15.072 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2021-03-15T22:26:15.099 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:15.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:15.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:15.121 INFO:tasks.rados.rados.0.smithi076.stdout:1714: done (3 left) 2021-03-15T22:26:15.121 INFO:tasks.rados.rados.0.smithi076.stdout:1715: snap_remove snap 163 2021-03-15T22:26:15.201 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:15.226 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:15.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:15.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:15.329 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:15.354 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:15.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:15.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:15.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:15.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:15.481 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:15.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:15.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:15 smithi076 conmon[56514]: cluster 2021-03-15T22:26:14.026100 2021-03-15T22:26:15.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:15 smithi076 conmon[56514]: +0000 osd.7 (osd.7) 94 : cluster [DBG] 5.4 scrub ok 2021-03-15T22:26:15.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:15 smithi076 conmon[56514]: cluster 2021-03-15T22:26:14.869971+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.d scrub ok 2021-03-15T22:26:15.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:15 smithi076 conmon[56514]: cluster 2021-03-15T22:26:15.116562+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2021-03-15T22:26:15.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:15 smithi076 conmon[62156]: cluster 2021-03-15T22:26:14.026100+0000 osd.7 (osd.7) 2021-03-15T22:26:15.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:15 smithi076 conmon[62156]: 94 : cluster [DBG] 5.4 scrub ok 2021-03-15T22:26:15.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:15 smithi076 conmon[62156]: cluster 2021-03-15T22:26:14.869971+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.d scrub ok 2021-03-15T22:26:15.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:15 smithi076 conmon[62156]: cluster 2021-03-15T22:26:15.116562+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2021-03-15T22:26:15.583 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:15.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:15.606 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:15.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:15.708 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:15.730 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:15.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:15.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:15 smithi125 conmon[56739]: cluster 2021-03-15T22:26:14.026100+0000 osd.7 (osd.7) 94 : cluster 2021-03-15T22:26:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:15 smithi125 conmon[56739]: [DBG] 5.4 scrub ok 2021-03-15T22:26:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:15 smithi125 conmon[56739]: cluster 2021-03-15T22:26:14.869971+0000 osd.5 (osd.5) 82 : cluster [DBG] 6.d scrub ok 2021-03-15T22:26:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:15 smithi125 conmon[56739]: cluster 2021-03-15T22:26:15 2021-03-15T22:26:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:15 smithi125 conmon[56739]: .116562+0000 mon.a (mon.0) 1312 : cluster [DBG] osdmap e460: 8 total, 8 up, 7 in 2021-03-15T22:26:15.832 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:15.854 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:15.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:15.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:15.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:15.986 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:15.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:16.088 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:16.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:16.112 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:16.112 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:16.134 INFO:tasks.rados.rados.0.smithi076.stdout:1715: done (3 left) 2021-03-15T22:26:16.135 INFO:tasks.rados.rados.0.smithi076.stdout:1716: write (excl) oid 2 current snap is 188 2021-03-15T22:26:16.135 INFO:tasks.rados.rados.0.smithi076.stdout:1716: seq_num 605 ranges {568235=717532,1753253=414982} 2021-03-15T22:26:16.142 INFO:tasks.rados.rados.0.smithi076.stdout:1716: writing smithi07694878-2 from 568235 to 1285767 tid 1 2021-03-15T22:26:16.148 INFO:tasks.rados.rados.0.smithi076.stdout:1716: writing smithi07694878-2 from 1753253 to 2168235 tid 2 2021-03-15T22:26:16.148 INFO:tasks.rados.rados.0.smithi076.stdout:1717: read oid 4 snap -1 2021-03-15T22:26:16.148 INFO:tasks.rados.rados.0.smithi076.stdout:1717: expect deleted 2021-03-15T22:26:16.149 INFO:tasks.rados.rados.0.smithi076.stdout:1718: snap_create 2021-03-15T22:26:16.149 INFO:tasks.rados.rados.0.smithi076.stdout:1711: finishing write tid 1 to smithi07694878-36 2021-03-15T22:26:16.149 INFO:tasks.rados.rados.0.smithi076.stdout:1712: finishing rollback tid 0 to smithi07694878-5 2021-03-15T22:26:16.149 INFO:tasks.rados.rados.0.smithi076.stdout:1712: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:26:16.149 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 598 (ObjNum 264828800 snap 21896 seq_num 265077232) dirty dne 2021-03-15T22:26:16.150 INFO:tasks.rados.rados.0.smithi076.stdout:1711: finishing write tid 2 to smithi07694878-36 2021-03-15T22:26:16.150 INFO:tasks.rados.rados.0.smithi076.stdout:1711: finishing write tid 3 to smithi07694878-36 2021-03-15T22:26:16.150 INFO:tasks.rados.rados.0.smithi076.stdout:1711: finishing write tid 4 to smithi07694878-36 2021-03-15T22:26:16.151 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 432 (ObjNum 603 snap 188 seq_num 603) dirty exists 2021-03-15T22:26:16.151 INFO:tasks.rados.rados.0.smithi076.stdout:1711: left oid 36 (ObjNum 603 snap 188 seq_num 603) 2021-03-15T22:26:16.151 INFO:tasks.rados.rados.0.smithi076.stdout:1713: finishing write tid 1 to smithi07694878-16 2021-03-15T22:26:16.151 INFO:tasks.rados.rados.0.smithi076.stdout:1713: finishing write tid 2 to smithi07694878-16 2021-03-15T22:26:16.151 INFO:tasks.rados.rados.0.smithi076.stdout:1713: finishing write tid 3 to smithi07694878-16 2021-03-15T22:26:16.152 INFO:tasks.rados.rados.0.smithi076.stdout:1713: finishing write tid 4 to smithi07694878-16 2021-03-15T22:26:16.152 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1086 (ObjNum 604 snap 188 seq_num 604) dirty exists 2021-03-15T22:26:16.152 INFO:tasks.rados.rados.0.smithi076.stdout:1713: left oid 16 (ObjNum 604 snap 188 seq_num 604) 2021-03-15T22:26:16.153 INFO:tasks.rados.rados.0.smithi076.stdout:1716: finishing write tid 1 to smithi07694878-2 2021-03-15T22:26:16.153 INFO:tasks.rados.rados.0.smithi076.stdout:1716: finishing write tid 2 to smithi07694878-2 2021-03-15T22:26:16.154 INFO:tasks.rados.rados.0.smithi076.stdout:1716: finishing write tid 3 to smithi07694878-2 2021-03-15T22:26:16.154 INFO:tasks.rados.rados.0.smithi076.stdout:1716: finishing write tid 4 to smithi07694878-2 2021-03-15T22:26:16.154 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 453 (ObjNum 605 snap 188 seq_num 605) dirty exists 2021-03-15T22:26:16.154 INFO:tasks.rados.rados.0.smithi076.stdout:1716: left oid 2 (ObjNum 605 snap 188 seq_num 605) 2021-03-15T22:26:16.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:16.237 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:16.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:16.239 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:16.340 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:16.362 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:16.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:16.364 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:16.465 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:16.491 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:16.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:16.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[56514]: cluster 2021-03-15T22:26:15.065484+0000 mgr.y (mgr.14144) 657 : cluster [DBG] pgmap v816: 138 pgs: 138 active+clean; 386 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2021-03-15T22:26:16.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[56514]: cluster 2021-03-15T22:26:15.579668+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:26:16.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[56514]: cluster 2021-03-15T22:26:15.749331+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.0 deep-scrub ok 2021-03-15T22:26:16.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[56514]: cluster 2021-03-15T22:26:16.127471+0000 mon.a (mon.0) 1313 : cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2021-03-15T22:26:16.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[62156]: cluster 2021-03-15T22:26:15.065484+0000 mgr.y 2021-03-15T22:26:16.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[62156]: (mgr.14144) 657 : cluster [DBG] pgmap v816: 138 pgs: 138 active+clean; 386 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2021-03-15T22:26:16.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[62156]: cluster 2021-03-15T22:26:15.579668+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:26:16.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[62156]: 2021-03-15T22:26:16.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[62156]: cluster 2021-03-15T22:26 2021-03-15T22:26:16.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[62156]: :15.749331+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.0 deep-scrub ok 2021-03-15T22:26:16.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[62156]: 2021-03-15T22:26:16.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[62156]: cluster 2021-03-15T22:26:16 2021-03-15T22:26:16.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[62156]: .127471+0000 mon.a (mon.0) 1313 : 2021-03-15T22:26:16.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:16 smithi076 conmon[62156]: cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2021-03-15T22:26:16.594 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:16.619 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:16.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:16.723 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:16.746 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:16.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:16.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:16.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:16 smithi125 conmon[56739]: cluster 2021-03-15T22:26:15.065484+0000 mgr.y (mgr.14144) 657 : cluster [DBG] pgmap v816: 138 pgs: 138 active+clean; 386 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.0 MiB/s rd, 1.5 MiB/s wr, 8 op/s 2021-03-15T22:26:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:16 smithi125 conmon[56739]: cluster 2021-03-15T22:26:15.579668+0000 osd.2 (osd.2) 76 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:26:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:16 smithi125 conmon[56739]: cluster 2021-03-15T22:26:15.749331+0000 osd.3 (osd.3) 79 : cluster [DBG] 5.0 deep-scrub ok 2021-03-15T22:26:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:16 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:26:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:16 smithi125 conmon[56739]: T22:26:16.127471+0000 mon.a (mon. 2021-03-15T22:26:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:16 smithi125 conmon[56739]: 0) 1313 : cluster [DBG] osdmap e461: 8 total, 8 up, 7 in 2021-03-15T22:26:16.848 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:16.874 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:16.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:16.976 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:17.000 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:17.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:17.102 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:17.130 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:17.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:17.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:17.135 INFO:tasks.rados.rados.0.smithi076.stdout:1711: done (5 left) 2021-03-15T22:26:17.135 INFO:tasks.rados.rados.0.smithi076.stdout:1712: done (4 left) 2021-03-15T22:26:17.135 INFO:tasks.rados.rados.0.smithi076.stdout:1713: done (3 left) 2021-03-15T22:26:17.136 INFO:tasks.rados.rados.0.smithi076.stdout:1716: done (2 left) 2021-03-15T22:26:17.136 INFO:tasks.rados.rados.0.smithi076.stdout:1717: done (1 left) 2021-03-15T22:26:17.136 INFO:tasks.rados.rados.0.smithi076.stdout:1718: done (0 left) 2021-03-15T22:26:17.136 INFO:tasks.rados.rados.0.smithi076.stdout:1719: rollback oid 32 current snap is 189 2021-03-15T22:26:17.137 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 32 to 139 2021-03-15T22:26:17.137 INFO:tasks.rados.rados.0.smithi076.stdout:1720: delete oid 36 current snap is 189 2021-03-15T22:26:17.139 INFO:tasks.rados.rados.0.smithi076.stdout:1720: done (1 left) 2021-03-15T22:26:17.139 INFO:tasks.rados.rados.0.smithi076.stdout:1721: read oid 50 snap 181 2021-03-15T22:26:17.140 INFO:tasks.rados.rados.0.smithi076.stdout:1721: expect deleted 2021-03-15T22:26:17.141 INFO:tasks.rados.rados.0.smithi076.stdout:1722: write (excl) oid 45 current snap is 189 2021-03-15T22:26:17.141 INFO:tasks.rados.rados.0.smithi076.stdout:1722: seq_num 607 ranges {601849=468030,1753864=447985} 2021-03-15T22:26:17.143 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:26:17.145 INFO:tasks.rados.rados.0.smithi076.stdout:1722: writing smithi07694878-45 from 601849 to 1069879 tid 1 2021-03-15T22:26:17.152 INFO:tasks.rados.rados.0.smithi076.stdout:1722: writing smithi07694878-45 from 1753864 to 2201849 tid 2 2021-03-15T22:26:17.153 INFO:tasks.rados.rados.0.smithi076.stdout:1723: delete oid 6 current snap is 189 2021-03-15T22:26:17.153 INFO:tasks.rados.rados.0.smithi076.stdout:1719: finishing rollback tid 0 to smithi07694878-32 2021-03-15T22:26:17.153 INFO:tasks.rados.rados.0.smithi076.stdout:1719: finishing rollback tid 1 to smithi07694878-32 2021-03-15T22:26:17.153 INFO:tasks.rados.rados.0.smithi076.stdout:1719: finishing rollback tid 2 to smithi07694878-32 2021-03-15T22:26:17.154 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 1200 (ObjNum 430 snap 119 seq_num 430) dirty exists 2021-03-15T22:26:17.154 INFO:tasks.rados.rados.0.smithi076.stdout:1722: finishing write tid 1 to smithi07694878-45 2021-03-15T22:26:17.156 INFO:tasks.rados.rados.0.smithi076.stdout:1722: finishing write tid 2 to smithi07694878-45 2021-03-15T22:26:17.156 INFO:tasks.rados.rados.0.smithi076.stdout:1722: finishing write tid 3 to smithi07694878-45 2021-03-15T22:26:17.156 INFO:tasks.rados.rados.0.smithi076.stdout:1719: done (3 left) 2021-03-15T22:26:17.157 INFO:tasks.rados.rados.0.smithi076.stdout:1721: done (2 left) 2021-03-15T22:26:17.157 INFO:tasks.rados.rados.0.smithi076.stdout:1723: done (1 left) 2021-03-15T22:26:17.157 INFO:tasks.rados.rados.0.smithi076.stdout:1724: write (excl) oid 34 current snap is 189 2021-03-15T22:26:17.157 INFO:tasks.rados.rados.0.smithi076.stdout:1724: seq_num 609 ranges {635463=734881,1987181=248282} 2021-03-15T22:26:17.164 INFO:tasks.rados.rados.0.smithi076.stdout:1724: writing smithi07694878-34 from 635463 to 1370344 tid 1 2021-03-15T22:26:17.169 INFO:tasks.rados.rados.0.smithi076.stdout:1724: writing smithi07694878-34 from 1987181 to 2235463 tid 2 2021-03-15T22:26:17.169 INFO:tasks.rados.rados.0.smithi076.stdout:1722: finishing write tid 4 to smithi07694878-45 2021-03-15T22:26:17.169 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 775 (ObjNum 607 snap 189 seq_num 607) dirty exists 2021-03-15T22:26:17.170 INFO:tasks.rados.rados.0.smithi076.stdout:1722: left oid 45 (ObjNum 607 snap 189 seq_num 607) 2021-03-15T22:26:17.170 INFO:tasks.rados.rados.0.smithi076.stdout:1724: finishing write tid 1 to smithi07694878-34 2021-03-15T22:26:17.170 INFO:tasks.rados.rados.0.smithi076.stdout:1722: done (1 left) 2021-03-15T22:26:17.170 INFO:tasks.rados.rados.0.smithi076.stdout:1725: copy_from oid 38 from oid 17 current snap is 189 2021-03-15T22:26:17.171 INFO:tasks.rados.rados.0.smithi076.stdout:1726: write oid 14 current snap is 189 2021-03-15T22:26:17.171 INFO:tasks.rados.rados.0.smithi076.stdout:1726: seq_num 610 ranges {652270=410130,1529310=493867} 2021-03-15T22:26:17.174 INFO:tasks.rados.rados.0.smithi076.stdout:1726: writing smithi07694878-14 from 652270 to 1062400 tid 1 2021-03-15T22:26:17.180 INFO:tasks.rados.rados.0.smithi076.stdout:1726: writing smithi07694878-14 from 1529310 to 2023177 tid 2 2021-03-15T22:26:17.180 INFO:tasks.rados.rados.0.smithi076.stdout:1727: delete oid 1 current snap is 189 2021-03-15T22:26:17.181 INFO:tasks.rados.rados.0.smithi076.stdout:1724: finishing write tid 2 to smithi07694878-34 2021-03-15T22:26:17.181 INFO:tasks.rados.rados.0.smithi076.stdout:1724: finishing write tid 3 to smithi07694878-34 2021-03-15T22:26:17.181 INFO:tasks.rados.rados.0.smithi076.stdout:1724: finishing write tid 4 to smithi07694878-34 2021-03-15T22:26:17.181 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 773 (ObjNum 609 snap 189 seq_num 609) dirty exists 2021-03-15T22:26:17.182 INFO:tasks.rados.rados.0.smithi076.stdout:1724: left oid 34 (ObjNum 609 snap 189 seq_num 609) 2021-03-15T22:26:17.182 INFO:tasks.rados.rados.0.smithi076.stdout:1726: finishing write tid 1 to smithi07694878-14 2021-03-15T22:26:17.183 INFO:tasks.rados.rados.0.smithi076.stdout:1725: finishing copy_from to smithi07694878-38 2021-03-15T22:26:17.183 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 318 (ObjNum 506 snap 151 seq_num 506) dirty exists 2021-03-15T22:26:17.183 INFO:tasks.rados.rados.0.smithi076.stdout:1725: finishing copy_from racing read to smithi07694878-38 2021-03-15T22:26:17.184 INFO:tasks.rados.rados.0.smithi076.stdout:1724: done (3 left) 2021-03-15T22:26:17.184 INFO:tasks.rados.rados.0.smithi076.stdout:1725: done (2 left) 2021-03-15T22:26:17.184 INFO:tasks.rados.rados.0.smithi076.stdout:1727: done (1 left) 2021-03-15T22:26:17.185 INFO:tasks.rados.rados.0.smithi076.stdout:1728: write oid 21 current snap is 189 2021-03-15T22:26:17.185 INFO:tasks.rados.rados.0.smithi076.stdout:1728: seq_num 612 ranges {685884=560628,2046274=239610} 2021-03-15T22:26:17.190 INFO:tasks.rados.rados.0.smithi076.stdout:1728: writing smithi07694878-21 from 685884 to 1246512 tid 1 2021-03-15T22:26:17.196 INFO:tasks.rados.rados.0.smithi076.stdout:1728: writing smithi07694878-21 from 2046274 to 2285884 tid 2 2021-03-15T22:26:17.197 INFO:tasks.rados.rados.0.smithi076.stdout:1729: read oid 43 snap 187 2021-03-15T22:26:17.197 INFO:tasks.rados.rados.0.smithi076.stdout:1729: expect (ObjNum 572 snap 179 seq_num 572) 2021-03-15T22:26:17.197 INFO:tasks.rados.rados.0.smithi076.stdout:1730: snap_remove snap 180 2021-03-15T22:26:17.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:17.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:17.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:17.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:17.384 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:17.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:17.394 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:26:17.394 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:26:17.395 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:26:17.395 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:26:17.395 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:26:17.487 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:17.513 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:17.515 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:17.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[56514]: cluster 2021-03-15T22:26:15.994789+0000 osd.7 (osd.7) 95 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:26:17.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[56514]: audit 2021-03-15T22:26:16.391424+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? 172.21.15.76:0/3935176767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:26:17.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[56514]: cluster 2021-03-15T22:26:16.598306+0000 osd.5 (osd.5) 83 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:17.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[56514]: cluster 2021-03-15T22:26:16.599302+0000 osd.5 (osd.5) 84 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:17.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[56514]: cluster 2021-03-15T22:26:17.123857+0000 mon.a (mon.0) 1315 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:17.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[56514]: audit 2021-03-15T22:26:17.133490+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? 172.21.15.76:0/3935176767' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:26:17.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[56514]: cluster 2021-03-15T22:26:17.133566+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2021-03-15T22:26:17.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[62156]: cluster 2021-03-15T22:26:15.994789+0000 2021-03-15T22:26:17.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[62156]: osd.7 (osd.7) 95 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:26:17.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[62156]: audit 2021-03-15T22:26:16.391424+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? 172.21.15.76:0/3935176767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:26:17.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[62156]: cluster 2021-03-15T22:26:16.598306+0000 osd.5 (osd.5) 83 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:17.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[62156]: cluster 2021-03-15T22:26:16.599302+0000 osd.5 (osd.5) 84 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:17.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[62156]: cluster 2021-03-15T22:26:17.123857+0000 mon.a (mon.0) 1315 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:17.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[62156]: audit 2021-03-15T22:26:17.133490+0000 mon.a (mon.0) 1316 : audit [INF] from='client.? 172.21.15.76:0/3935176767' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:26:17.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:17 smithi076 conmon[62156]: cluster 2021-03-15T22:26:17.133566+0000 mon.a (mon.0) 1317 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2021-03-15T22:26:17.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:17.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:17.641 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:17.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:17.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:17.768 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:17.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:17.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:17.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: cluster 2021-03-15T22:26:15.994789+0000 osd.7 (osd.7) 95 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:26:17.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: audit 2021-03-15T22:26:16.391424+0000 mon.a (mon.0) 1314 : audit [INF] from='client.? 172.21.15.76:0/3935176767' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:26:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: cluster 2021-03-15T22:26:16.598306+0000 osd.5 (osd 2021-03-15T22:26:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: .5) 83 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:26:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: 26:16.599302+0000 osd.5 (osd.5) 84 2021-03-15T22:26:17.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:17.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:17.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: -15T22:26:17.123857+0000 2021-03-15T22:26:17.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: mon.a (mon.0) 1315 : cluster [WRN] 2021-03-15T22:26:17.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:26:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: 26:17.133490+0000 mon.a (mon.0) 2021-03-15T22:26:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: 1316 : audit [INF] from='client.? 172.21.15.76:0/3935176767' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:26:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:26:17.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: T22:26:17.133566+0000 mon.a (mon 2021-03-15T22:26:17.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:17 smithi125 conmon[56739]: .0) 1317 : cluster [DBG] osdmap e462: 8 total, 8 up, 7 in 2021-03-15T22:26:17.848 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:26:17.848 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:26:17.848 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:26:17.849 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:26:17.849 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:26:17.849 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:26:17.849 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:26:17.849 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:26:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:26:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:26:17.850 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:26:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:26:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:26:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:26:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:26:17.851 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:26:17.871 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:17.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:17.897 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:17.898 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:17.999 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:18.024 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:18.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:18.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:18.127 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:18.133 INFO:tasks.rados.rados.0.smithi076.stdout:1730: done (3 left) 2021-03-15T22:26:18.133 INFO:tasks.rados.rados.0.smithi076.stdout:1731: snap_create 2021-03-15T22:26:18.134 INFO:tasks.rados.rados.0.smithi076.stdout:1726: finishing write tid 2 to smithi07694878-14 2021-03-15T22:26:18.134 INFO:tasks.rados.rados.0.smithi076.stdout:1726: finishing write tid 3 to smithi07694878-14 2021-03-15T22:26:18.134 INFO:tasks.rados.rados.0.smithi076.stdout:1726: finishing write tid 4 to smithi07694878-14 2021-03-15T22:26:18.134 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 777 (ObjNum 610 snap 189 seq_num 610) dirty exists 2021-03-15T22:26:18.135 INFO:tasks.rados.rados.0.smithi076.stdout:1726: left oid 14 (ObjNum 610 snap 189 seq_num 610) 2021-03-15T22:26:18.135 INFO:tasks.rados.rados.0.smithi076.stdout:1728: finishing write tid 1 to smithi07694878-21 2021-03-15T22:26:18.135 INFO:tasks.rados.rados.0.smithi076.stdout:1728: finishing write tid 2 to smithi07694878-21 2021-03-15T22:26:18.135 INFO:tasks.rados.rados.0.smithi076.stdout:1728: finishing write tid 3 to smithi07694878-21 2021-03-15T22:26:18.135 INFO:tasks.rados.rados.0.smithi076.stdout:1728: finishing write tid 4 to smithi07694878-21 2021-03-15T22:26:18.136 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 1204 (ObjNum 612 snap 189 seq_num 612) dirty exists 2021-03-15T22:26:18.136 INFO:tasks.rados.rados.0.smithi076.stdout:1728: left oid 21 (ObjNum 612 snap 189 seq_num 612) 2021-03-15T22:26:18.136 INFO:tasks.rados.rados.0.smithi076.stdout:1729: expect (ObjNum 572 snap 179 seq_num 572) 2021-03-15T22:26:18.156 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:18.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:18.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:18.258 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:18.281 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:18.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:18.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:18.385 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:18.408 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:18.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:18.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:18.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:18.524 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:26:18.538 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:18.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:18.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:18.587 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[56514]: cluster 2021-03-15T22:26: 2021-03-15T22:26:18.587 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[56514]: 16.580428+0000 osd.1 (osd.1) 92 : cluster [DBG] 5.c scrub ok 2021-03-15T22:26:18.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[56514]: cluster 2021-03-15T22:26:17.012645+0000 osd.7 (osd.7) 96 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:26:18.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[56514]: cluster 2021-03-15T22:26:17.066000+0000 mgr.y (mgr.14144) 658 : cluster [DBG] pgmap v819: 138 pgs: 138 active+clean; 384 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 3.1 MiB/s wr, 8 op/s 2021-03-15T22:26:18.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[56514]: audit 2021-03-15T22:26:17.679126+0000 mon.c (mon.1) 40 : audit [DBG] from='client.? 172.21.15.76:0/2224574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:18.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[56514]: cluster 2021-03-15T22:26:18.130699+0000 mon.a (mon.0) 1318 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2021-03-15T22:26:18.589 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[62156]: cluster 2021-03-15T22:26: 2021-03-15T22:26:18.589 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[62156]: 16.580428+0000 osd.1 (osd.1) 92 : cluster [DBG] 5.c scrub ok 2021-03-15T22:26:18.589 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[62156]: cluster 2021-03-15T22:26:17.012645+0000 osd.7 (osd.7) 96 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:26:18.589 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[62156]: cluster 2021-03-15T22:26:17.066000+0000 mgr.y (mgr.14144) 658 : cluster [DBG] pgmap v819: 138 pgs: 138 active+clean; 384 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 3.1 MiB/s wr, 8 op/s 2021-03-15T22:26:18.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[62156]: audit 2021-03-15T22:26:17.679126+0000 mon.c (mon.1) 40 : audit [DBG] from='client.? 172.21.15.76:0/2224574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:18.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:18 smithi076 conmon[62156]: cluster 2021-03-15T22:26:18.130699+0000 mon.a (mon.0) 1318 : cluster [DBG] osdmap e463: 8 total, 8 up, 7 in 2021-03-15T22:26:18.641 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:18.666 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:18.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:18.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:18.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:18.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:18 smithi125 conmon[56739]: cluster 2021-03-15T22:26:16.580428+0000 osd.1 (osd.1) 92 : cluster [DBG] 5.c scrub ok 2021-03-15T22:26:18.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:18 smithi125 conmon[56739]: cluster 2021-03-15T22:26:17.012645+0000 osd.7 (osd.7) 96 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:26:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:18 smithi125 conmon[56739]: cluster 2021-03-15T22:26:17.066000+0000 mgr.y (mgr.14144) 658 : cluster 2021-03-15T22:26:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:18 smithi125 conmon[56739]: [DBG] pgmap v819: 138 pgs: 138 active+clean; 384 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 3.1 MiB/s wr, 8 op/s 2021-03-15T22:26:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:18 smithi125 conmon[56739]: audit 2021-03-15T22:26:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:18 smithi125 conmon[56739]: 2021-03-15T22:26:17.679126+0000 mon.c (mon.1 2021-03-15T22:26:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:18 smithi125 conmon[56739]: ) 40 : audit [DBG] from='client.? 172.21.15.76:0/2224574' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:18 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:26:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:18 smithi125 conmon[56739]: -03-15T22:26:18.130699 2021-03-15T22:26:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:18 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1318 : cluster [DBG] 2021-03-15T22:26:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:18 smithi125 conmon[56739]: osdmap e463: 8 total, 8 up, 7 in 2021-03-15T22:26:18.793 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:18.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:18.795 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:18.896 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:18.918 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:18.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:18.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:19.020 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:19.042 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:19.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:19.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:19.112 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:19.135 INFO:tasks.rados.rados.0.smithi076.stdout:1726: done (3 left) 2021-03-15T22:26:19.135 INFO:tasks.rados.rados.0.smithi076.stdout:1728: done (2 left) 2021-03-15T22:26:19.135 INFO:tasks.rados.rados.0.smithi076.stdout:1729: done (1 left) 2021-03-15T22:26:19.135 INFO:tasks.rados.rados.0.smithi076.stdout:1731: done (0 left) 2021-03-15T22:26:19.136 INFO:tasks.rados.rados.0.smithi076.stdout:1732: snap_remove snap 155 2021-03-15T22:26:19.145 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:19.169 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:19.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:19.272 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:19.298 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:19.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:19.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:19.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:19.426 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:19.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:19.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:19.529 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:19.552 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:19.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:19.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:19.654 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:19.681 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:19.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:19.784 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:19.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:19.807 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:19.808 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:19.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:19.931 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:19.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:19.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:20.033 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:20.037 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:26:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:26:20.055 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:20.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:20.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:20.075 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:26:20.075 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:26:20.076 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump -f json-pretty 2021-03-15T22:26:20.124 INFO:tasks.rados.rados.0.smithi076.stdout:1732: done (0 left) 2021-03-15T22:26:20.124 INFO:tasks.rados.rados.0.smithi076.stdout:1733: read oid 1 snap -1 2021-03-15T22:26:20.125 INFO:tasks.rados.rados.0.smithi076.stdout:1733: expect deleted 2021-03-15T22:26:20.125 INFO:tasks.rados.rados.0.smithi076.stdout:1734: rollback oid 14 current snap is 190 2021-03-15T22:26:20.125 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 14 to 183 2021-03-15T22:26:20.125 INFO:tasks.rados.rados.0.smithi076.stdout:1735: rollback oid 40 current snap is 190 2021-03-15T22:26:20.126 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 40 to 181 2021-03-15T22:26:20.126 INFO:tasks.rados.rados.0.smithi076.stdout:1736: read oid 23 snap -1 2021-03-15T22:26:20.126 INFO:tasks.rados.rados.0.smithi076.stdout:1736: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:26:20.127 INFO:tasks.rados.rados.0.smithi076.stdout:1737: rollback oid 46 current snap is 190 2021-03-15T22:26:20.127 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 46 to 189 2021-03-15T22:26:20.127 INFO:tasks.rados.rados.0.smithi076.stdout:1738: copy_from oid 45 from oid 34 current snap is 190 2021-03-15T22:26:20.127 INFO:tasks.rados.rados.0.smithi076.stdout:1739: write (excl) oid 43 current snap is 190 2021-03-15T22:26:20.128 INFO:tasks.rados.rados.0.smithi076.stdout:1739: seq_num 613 ranges {702691=635877,2104756=197935} 2021-03-15T22:26:20.153 INFO:tasks.rados.rados.0.smithi076.stdout:1739: writing smithi07694878-43 from 702691 to 1338568 tid 1 2021-03-15T22:26:20.158 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:20.160 INFO:tasks.rados.rados.0.smithi076.stdout:1739: writing smithi07694878-43 from 2104756 to 2302691 tid 2 2021-03-15T22:26:20.160 INFO:tasks.rados.rados.0.smithi076.stdout:1740: read oid 24 snap 171 2021-03-15T22:26:20.160 INFO:tasks.rados.rados.0.smithi076.stdout:1740: expect (ObjNum 514 snap 155 seq_num 514) 2021-03-15T22:26:20.160 INFO:tasks.rados.rados.0.smithi076.stdout:1741: snap_remove snap 179 2021-03-15T22:26:20.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:20.193 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:20.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:20.295 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:20.318 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:20.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:20.319 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:20.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: cluster 2021-03-15T22:26:20.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021 2021-03-15T22:26:20.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: - 2021-03-15T22:26:20.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 03 2021-03-15T22:26:20.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: - 2021-03-15T22:26:20.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 15 2021-03-15T22:26:20.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: T 2021-03-15T22:26:20.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 22 2021-03-15T22:26:20.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 26 2021-03-15T22:26:20.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 19 2021-03-15T22:26:20.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: . 2021-03-15T22:26:20.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 066541 2021-03-15T22:26:20.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: +0000 2021-03-15T22:26:20.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: mgr.y 2021-03-15T22:26:20.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: ( 2021-03-15T22:26:20.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: mgr 2021-03-15T22:26:20.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: . 2021-03-15T22:26:20.390 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 14144 2021-03-15T22:26:20.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: ) 2021-03-15T22:26:20.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 659 2021-03-15T22:26:20.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: cluster 2021-03-15T22:26:20.391 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: [DBG] 2021-03-15T22:26:20.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: pgmap v822: 138 pgs: 138 active+clean; 388 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 7 op/s 2021-03-15T22:26:20.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: cluster 2021-03-15T22:26:20.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021 2021-03-15T22:26:20.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: - 2021-03-15T22:26:20.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 03 2021-03-15T22:26:20.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: - 2021-03-15T22:26:20.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 15 2021-03-15T22:26:20.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: T 2021-03-15T22:26:20.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 22 2021-03-15T22:26:20.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 26 2021-03-15T22:26:20.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 19 2021-03-15T22:26:20.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: . 2021-03-15T22:26:20.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 131160 2021-03-15T22:26:20.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: +0000 2021-03-15T22:26:20.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: mon.a 2021-03-15T22:26:20.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: ( 2021-03-15T22:26:20.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: mon 2021-03-15T22:26:20.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: . 2021-03-15T22:26:20.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 0 2021-03-15T22:26:20.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: ) 2021-03-15T22:26:20.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 1319 2021-03-15T22:26:20.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: cluster 2021-03-15T22:26:20.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: [DBG] 2021-03-15T22:26:20.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: osdmap e464: 8 total, 8 up, 7 in 2021-03-15T22:26:20.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: audit 2021-03-15T22:26:20.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021 2021-03-15T22:26:20.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: - 2021-03-15T22:26:20.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 03 2021-03-15T22:26:20.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: - 2021-03-15T22:26:20.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 15 2021-03-15T22:26:20.401 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: T 2021-03-15T22:26:20.401 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 22 2021-03-15T22:26:20.401 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.401 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 26 2021-03-15T22:26:20.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 19 2021-03-15T22:26:20.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: . 2021-03-15T22:26:20.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 763410 2021-03-15T22:26:20.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: +0000 2021-03-15T22:26:20.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: mgr.y 2021-03-15T22:26:20.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: ( 2021-03-15T22:26:20.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: mgr 2021-03-15T22:26:20.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: . 2021-03-15T22:26:20.404 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 14144 2021-03-15T22:26:20.404 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: ) 2021-03-15T22:26:20.404 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 660 2021-03-15T22:26:20.404 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.405 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: audit 2021-03-15T22:26:20.405 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: cluster 2021-03-15T22:26:20.405 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 2021-03-15T22:26:20.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 2021 2021-03-15T22:26:20.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: - 2021-03-15T22:26:20.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 03 2021-03-15T22:26:20.406 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: - 2021-03-15T22:26:20.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 15 2021-03-15T22:26:20.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: T 2021-03-15T22:26:20.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 22 2021-03-15T22:26:20.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: : 2021-03-15T22:26:20.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 26 2021-03-15T22:26:20.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: : 2021-03-15T22:26:20.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 19 2021-03-15T22:26:20.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: . 2021-03-15T22:26:20.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 066541 2021-03-15T22:26:20.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: +0000 2021-03-15T22:26:20.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 2021-03-15T22:26:20.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: mgr.y 2021-03-15T22:26:20.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: ( 2021-03-15T22:26:20.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: mgr 2021-03-15T22:26:20.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: . 2021-03-15T22:26:20.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 14144 2021-03-15T22:26:20.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: ) 2021-03-15T22:26:20.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 659 2021-03-15T22:26:20.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: : 2021-03-15T22:26:20.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: cluster 2021-03-15T22:26:20.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 2021-03-15T22:26:20.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: [DBG] 2021-03-15T22:26:20.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 2021-03-15T22:26:20.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: pgmap v822: 138 pgs: 138 active+clean; 388 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 7 op/s 2021-03-15T22:26:20.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 2021-03-15T22:26:20.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: cluster 2021-03-15T22:26:20.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 2021-03-15T22:26:20.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 2021 2021-03-15T22:26:20.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: - 2021-03-15T22:26:20.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 03 2021-03-15T22:26:20.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: - 2021-03-15T22:26:20.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 15 2021-03-15T22:26:20.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: T 2021-03-15T22:26:20.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 22 2021-03-15T22:26:20.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: : 2021-03-15T22:26:20.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 26 2021-03-15T22:26:20.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: : 2021-03-15T22:26:20.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 19 2021-03-15T22:26:20.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: . 2021-03-15T22:26:20.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 131160 2021-03-15T22:26:20.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: +0000 2021-03-15T22:26:20.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 2021-03-15T22:26:20.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: mon.a 2021-03-15T22:26:20.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: ( 2021-03-15T22:26:20.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: mon 2021-03-15T22:26:20.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: . 2021-03-15T22:26:20.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 0 2021-03-15T22:26:20.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: ) 2021-03-15T22:26:20.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 1319 2021-03-15T22:26:20.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: : 2021-03-15T22:26:20.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: cluster 2021-03-15T22:26:20.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 2021-03-15T22:26:20.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: [DBG] 2021-03-15T22:26:20.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: osdmap e464: 8 total, 8 up, 7 in 2021-03-15T22:26:20.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: audit 2021 2021-03-15T22:26:20.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: -03-15T22:26:19.763410+0000 mgr.y ( 2021-03-15T22:26:20.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: mgr.14144) 660 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:20.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: cluster 2021-03-15T22:26:20.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: 2021-03-15T22:26:20.118529+0000 2021-03-15T22:26:20.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: mon.a (mon.0) 1320 : cluster [DBG] 2021-03-15T22:26:20.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[56514]: osdmap e465: 8 total, 8 up, 7 in 2021-03-15T22:26:20.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: [DBG] 2021-03-15T22:26:20.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:20.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: cluster 2021-03-15T22:26:20.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021 2021-03-15T22:26:20.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: - 2021-03-15T22:26:20.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 03 2021-03-15T22:26:20.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: - 2021-03-15T22:26:20.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 15 2021-03-15T22:26:20.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: T 2021-03-15T22:26:20.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 22 2021-03-15T22:26:20.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 26 2021-03-15T22:26:20.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 20 2021-03-15T22:26:20.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: . 2021-03-15T22:26:20.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 118529 2021-03-15T22:26:20.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: +0000 2021-03-15T22:26:20.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: mon.a 2021-03-15T22:26:20.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: ( 2021-03-15T22:26:20.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: mon 2021-03-15T22:26:20.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: . 2021-03-15T22:26:20.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 0 2021-03-15T22:26:20.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: ) 2021-03-15T22:26:20.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 1320 2021-03-15T22:26:20.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: : 2021-03-15T22:26:20.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: cluster 2021-03-15T22:26:20.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: [DBG] 2021-03-15T22:26:20.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: osdmap e465: 8 total, 8 up, 7 in 2021-03-15T22:26:20.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:20 smithi076 conmon[62156]: 2021-03-15T22:26:20.428 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:20.452 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:20.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:20.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:20.495 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:26:20.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:20 smithi125 conmon[56739]: cluster 2021-03-15T22:26:19.066541+0000 mgr.y (mgr.14144 2021-03-15T22:26:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:20 smithi125 conmon[56739]: ) 659 : cluster [DBG] pgmap v822: 138 pgs: 138 active+clean; 388 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 7 op/s 2021-03-15T22:26:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:20 smithi125 conmon[56739]: cluster 2021-03-15T22:26:19.131160+0000 mon.a (mon.0) 1319 : cluster [DBG] osdmap e464: 8 total, 8 up, 7 in 2021-03-15T22:26:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:20 smithi125 conmon[56739]: audit 2021-03-15T22:26:19 2021-03-15T22:26:20.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:20 smithi125 conmon[56739]: .763410+0000 mgr.y (mgr.14144 2021-03-15T22:26:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:20 smithi125 conmon[56739]: ) 660 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:20 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:26:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:20 smithi125 conmon[56739]: 03-15T22:26:20.118529+0000 2021-03-15T22:26:20.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:20 smithi125 conmon[56739]: mon.a (mon.0) 1320 : cluster [DBG] osdmap e465: 8 total, 8 up, 7 in 2021-03-15T22:26:20.555 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:20.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:20.581 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:20.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:20.600 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:20.683 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:20.700 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:26:20.700 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:26:20.707 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:20.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:20.709 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:20.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:20.837 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:20.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:20.839 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:20.940 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:20.965 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:20.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:20.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:21.068 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:21.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:21.092 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:21.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:21.128 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:26:21.146 INFO:tasks.rados.rados.0.smithi076.stdout:1741: done (8 left) 2021-03-15T22:26:21.146 INFO:tasks.rados.rados.0.smithi076.stdout:1742: snap_remove snap 174 2021-03-15T22:26:21.193 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:21.215 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:21.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:21.318 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:21.346 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:21.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:21.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:21.449 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:21.475 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:21.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:21.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:21.577 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:21.604 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:21.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:21.605 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:21.706 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:21.732 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:21.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:21.734 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:21.835 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:21.861 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:21.863 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:21.964 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:21.987 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:21.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:21.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:22.052 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:26:22.089 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:22.116 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:22.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:22.118 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:22.153 INFO:tasks.rados.rados.0.smithi076.stdout:1742: done (8 left) 2021-03-15T22:26:22.154 INFO:tasks.rados.rados.0.smithi076.stdout:1743: rollback oid 26 current snap is 190 2021-03-15T22:26:22.154 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 185 2021-03-15T22:26:22.154 INFO:tasks.rados.rados.0.smithi076.stdout:1736: expect (ObjNum 424 snap 117 seq_num 424) 2021-03-15T22:26:22.214 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:22.219 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:22.243 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:22.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:22.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:22.345 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:22.369 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:22.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:22.376 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:22.377 INFO:tasks.rados.rados.0.smithi076.stdout:1737: finishing rollback tid 0 to smithi07694878-46 2021-03-15T22:26:22.377 INFO:tasks.rados.rados.0.smithi076.stdout:1737: finishing rollback tid 1 to smithi07694878-46 2021-03-15T22:26:22.377 INFO:tasks.rados.rados.0.smithi076.stdout:1737: finishing rollback tid 2 to smithi07694878-46 2021-03-15T22:26:22.377 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 832 (ObjNum 496 snap 149 seq_num 496) dirty exists 2021-03-15T22:26:22.378 INFO:tasks.rados.rados.0.smithi076.stdout:1735: finishing rollback tid 1 to smithi07694878-40 2021-03-15T22:26:22.378 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 896 (ObjNum 261732928 snap 21896 seq_num 0) dirty dne 2021-03-15T22:26:22.378 INFO:tasks.rados.rados.0.smithi076.stdout:1734: finishing rollback tid 0 to smithi07694878-14 2021-03-15T22:26:22.378 INFO:tasks.rados.rados.0.smithi076.stdout:1734: finishing rollback tid 1 to smithi07694878-14 2021-03-15T22:26:22.379 INFO:tasks.rados.rados.0.smithi076.stdout:1734: finishing rollback tid 2 to smithi07694878-14 2021-03-15T22:26:22.379 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 780 (ObjNum 593 snap 183 seq_num 593) dirty exists 2021-03-15T22:26:22.379 INFO:tasks.rados.rados.0.smithi076.stdout:1733: done (8 left) 2021-03-15T22:26:22.379 INFO:tasks.rados.rados.0.smithi076.stdout:1734: done (7 left) 2021-03-15T22:26:22.379 INFO:tasks.rados.rados.0.smithi076.stdout:1735: done (6 left) 2021-03-15T22:26:22.380 INFO:tasks.rados.rados.0.smithi076.stdout:1736: done (5 left) 2021-03-15T22:26:22.380 INFO:tasks.rados.rados.0.smithi076.stdout:1737: done (4 left) 2021-03-15T22:26:22.380 INFO:tasks.rados.rados.0.smithi076.stdout:1744: snap_remove snap 154 2021-03-15T22:26:22.384 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[56514]: cluster 2021-03-15T22:26:22.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[56514]: 2021-03-15T22:26:21.066973 2021-03-15T22:26:22.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 661 : cluster [DBG] pgmap v825: 138 pgs: 138 active+clean; 388 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 472 KiB/s wr, 7 op/s 2021-03-15T22:26:22.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:26:22.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[56514]: 26:21.143175+0000 mon.a (mon.0) 1321 : cluster 2021-03-15T22:26:22.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[56514]: [DBG] osdmap e466: 8 total, 8 up, 7 in 2021-03-15T22:26:22.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[62156]: cluster 2021-03-15T22:26:22.386 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[62156]: 2021-03-15T22:26 2021-03-15T22:26:22.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[62156]: :21.066973+0000 mgr.y (mgr.14144) 661 : cluster [DBG] pgmap v825: 138 pgs: 138 active+clean; 388 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 472 KiB/s wr, 7 op/s 2021-03-15T22:26:22.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:26:22.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[62156]: 15T22:26:21.143175+0000 mon.a (mon.0 2021-03-15T22:26:22.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:22 smithi076 conmon[62156]: ) 1321 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2021-03-15T22:26:22.438 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:22.440 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:22.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:22.500 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:22.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:22.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:22 smithi125 conmon[56739]: cluster 2021-03-15T22:26:21.066973+0000 mgr.y (mgr.14144 2021-03-15T22:26:22.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:22 smithi125 conmon[56739]: ) 661 : cluster [DBG] pgmap v825: 138 pgs: 138 active+clean; 388 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 2.8 MiB/s rd, 472 KiB/s wr, 7 op/s 2021-03-15T22:26:22.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:22 smithi125 conmon[56739]: cluster 2021-03-15T22:26:21.143175+0000 mon.a (mon.0) 1321 : cluster [DBG] osdmap e466: 8 total, 8 up, 7 in 2021-03-15T22:26:22.606 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:22.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:22.631 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:22.632 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:22.733 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:22.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:22.763 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:22.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:22.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:22.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:22.894 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:22.895 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:22.996 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:23.022 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:23.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:23.024 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:23.125 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:23.150 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:23.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:23.152 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:23.177 INFO:tasks.rados.rados.0.smithi076.stdout:1744: done (4 left) 2021-03-15T22:26:23.178 INFO:tasks.rados.rados.0.smithi076.stdout:1745: write (excl) oid 7 current snap is 190 2021-03-15T22:26:23.178 INFO:tasks.rados.rados.0.smithi076.stdout:1745: seq_num 614 ranges {719498=711126,2046885=272613} 2021-03-15T22:26:23.183 INFO:tasks.rados.rados.0.smithi076.stdout:1745: writing smithi07694878-7 from 719498 to 1430624 tid 1 2021-03-15T22:26:23.188 INFO:tasks.rados.rados.0.smithi076.stdout:1745: writing smithi07694878-7 from 2046885 to 2319498 tid 2 2021-03-15T22:26:23.189 INFO:tasks.rados.rados.0.smithi076.stdout:1746: snap_create 2021-03-15T22:26:23.189 INFO:tasks.rados.rados.0.smithi076.stdout:1739: finishing write tid 1 to smithi07694878-43 2021-03-15T22:26:23.189 INFO:tasks.rados.rados.0.smithi076.stdout:1738: finishing copy_from to smithi07694878-45 2021-03-15T22:26:23.189 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 773 (ObjNum 609 snap 189 seq_num 609) dirty exists 2021-03-15T22:26:23.190 INFO:tasks.rados.rados.0.smithi076.stdout:1739: finishing write tid 2 to smithi07694878-43 2021-03-15T22:26:23.190 INFO:tasks.rados.rados.0.smithi076.stdout:1738: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:26:23.190 INFO:tasks.rados.rados.0.smithi076.stdout:1739: finishing write tid 3 to smithi07694878-43 2021-03-15T22:26:23.190 INFO:tasks.rados.rados.0.smithi076.stdout:1740: expect (ObjNum 514 snap 155 seq_num 514) 2021-03-15T22:26:23.254 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:23.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:23 smithi076 conmon[56514]: cluster 2021-03-15T22:26:22.147704+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2021-03-15T22:26:23.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:23 smithi076 conmon[62156]: cluster 2021-03-15T22:26:22.147704+0000 mon.a (mon.0) 1322 : cluster [DBG] osdmap e467: 8 total, 8 up, 7 in 2021-03-15T22:26:23.277 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:23.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:23.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:23.365 INFO:tasks.rados.rados.0.smithi076.stdout:1739: finishing write tid 4 to smithi07694878-43 2021-03-15T22:26:23.365 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 800 (ObjNum 613 snap 190 seq_num 613) dirty exists 2021-03-15T22:26:23.366 INFO:tasks.rados.rados.0.smithi076.stdout:1739: left oid 43 (ObjNum 613 snap 190 seq_num 613) 2021-03-15T22:26:23.366 INFO:tasks.rados.rados.0.smithi076.stdout:1743: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:26:23.366 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 664 (ObjNum 265096496 snap 21896 seq_num 265164864) dirty dne 2021-03-15T22:26:23.366 INFO:tasks.rados.rados.0.smithi076.stdout:1745: finishing write tid 1 to smithi07694878-7 2021-03-15T22:26:23.367 INFO:tasks.rados.rados.0.smithi076.stdout:1745: finishing write tid 2 to smithi07694878-7 2021-03-15T22:26:23.367 INFO:tasks.rados.rados.0.smithi076.stdout:1745: finishing write tid 3 to smithi07694878-7 2021-03-15T22:26:23.367 INFO:tasks.rados.rados.0.smithi076.stdout:1745: finishing write tid 4 to smithi07694878-7 2021-03-15T22:26:23.367 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 984 (ObjNum 614 snap 190 seq_num 614) dirty exists 2021-03-15T22:26:23.368 INFO:tasks.rados.rados.0.smithi076.stdout:1745: left oid 7 (ObjNum 614 snap 190 seq_num 614) 2021-03-15T22:26:23.380 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:23.408 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:23.409 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:23.510 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:23.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:23.535 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:23.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:23.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:23 smithi125 conmon[56739]: cluster 2021-03-15T22:26:22.147704+0000 mon.a (mon.0) 1322 : cluster 2021-03-15T22:26:23.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:23 smithi125 conmon[56739]: [DBG] osdmap e467: 8 total, 8 up, 7 in 2021-03-15T22:26:23.636 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:23.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:23.659 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:23.660 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:23.761 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:23.783 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:23.784 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:23.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:23.907 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:23.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:23.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:24.010 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:24.032 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:24.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:24.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:24.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:24.157 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:24.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:24.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:24.178 INFO:tasks.rados.rados.0.smithi076.stdout:1738: done (5 left) 2021-03-15T22:26:24.179 INFO:tasks.rados.rados.0.smithi076.stdout:1739: done (4 left) 2021-03-15T22:26:24.179 INFO:tasks.rados.rados.0.smithi076.stdout:1740: done (3 left) 2021-03-15T22:26:24.179 INFO:tasks.rados.rados.0.smithi076.stdout:1743: done (2 left) 2021-03-15T22:26:24.179 INFO:tasks.rados.rados.0.smithi076.stdout:1745: done (1 left) 2021-03-15T22:26:24.180 INFO:tasks.rados.rados.0.smithi076.stdout:1746: done (0 left) 2021-03-15T22:26:24.180 INFO:tasks.rados.rados.0.smithi076.stdout:1747: write oid 19 current snap is 191 2021-03-15T22:26:24.180 INFO:tasks.rados.rados.0.smithi076.stdout:1747: seq_num 615 ranges {736305=786375,2105367=230938} 2021-03-15T22:26:24.186 INFO:tasks.rados.rados.0.smithi076.stdout:1747: writing smithi07694878-19 from 736305 to 1522680 tid 1 2021-03-15T22:26:24.190 INFO:tasks.rados.rados.0.smithi076.stdout:1747: writing smithi07694878-19 from 2105367 to 2336305 tid 2 2021-03-15T22:26:24.191 INFO:tasks.rados.rados.0.smithi076.stdout:1748: write oid 14 current snap is 191 2021-03-15T22:26:24.191 INFO:tasks.rados.rados.0.smithi076.stdout:1747: finishing write tid 1 to smithi07694878-19 2021-03-15T22:26:24.191 INFO:tasks.rados.rados.0.smithi076.stdout:1748: seq_num 616 ranges {753112=577977,1880202=472910} 2021-03-15T22:26:24.198 INFO:tasks.rados.rados.0.smithi076.stdout:1748: writing smithi07694878-14 from 753112 to 1331089 tid 1 2021-03-15T22:26:24.204 INFO:tasks.rados.rados.0.smithi076.stdout:1748: writing smithi07694878-14 from 1880202 to 2353112 tid 2 2021-03-15T22:26:24.205 INFO:tasks.rados.rados.0.smithi076.stdout:1749: write oid 26 current snap is 191 2021-03-15T22:26:24.205 INFO:tasks.rados.rados.0.smithi076.stdout:1747: finishing write tid 2 to smithi07694878-19 2021-03-15T22:26:24.205 INFO:tasks.rados.rados.0.smithi076.stdout:1749: seq_num 617 ranges {769919=653226,1938684=431235} 2021-03-15T22:26:24.213 INFO:tasks.rados.rados.0.smithi076.stdout:1749: writing smithi07694878-26 from 769919 to 1423145 tid 1 2021-03-15T22:26:24.220 INFO:tasks.rados.rados.0.smithi076.stdout:1749: writing smithi07694878-26 from 1938684 to 2369919 tid 2 2021-03-15T22:26:24.220 INFO:tasks.rados.rados.0.smithi076.stdout:1750: copy_from oid 17 from oid 46 current snap is 191 2021-03-15T22:26:24.220 INFO:tasks.rados.rados.0.smithi076.stdout:1747: finishing write tid 3 to smithi07694878-19 2021-03-15T22:26:24.220 INFO:tasks.rados.rados.0.smithi076.stdout:1747: finishing write tid 4 to smithi07694878-19 2021-03-15T22:26:24.221 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 987 (ObjNum 615 snap 191 seq_num 615) dirty exists 2021-03-15T22:26:24.221 INFO:tasks.rados.rados.0.smithi076.stdout:1747: left oid 19 (ObjNum 615 snap 191 seq_num 615) 2021-03-15T22:26:24.221 INFO:tasks.rados.rados.0.smithi076.stdout:1748: finishing write tid 1 to smithi07694878-14 2021-03-15T22:26:24.221 INFO:tasks.rados.rados.0.smithi076.stdout:1748: finishing write tid 2 to smithi07694878-14 2021-03-15T22:26:24.222 INFO:tasks.rados.rados.0.smithi076.stdout:1748: finishing write tid 3 to smithi07694878-14 2021-03-15T22:26:24.222 INFO:tasks.rados.rados.0.smithi076.stdout:1748: finishing write tid 4 to smithi07694878-14 2021-03-15T22:26:24.222 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 783 (ObjNum 616 snap 191 seq_num 616) dirty exists 2021-03-15T22:26:24.222 INFO:tasks.rados.rados.0.smithi076.stdout:1748: left oid 14 (ObjNum 616 snap 191 seq_num 616) 2021-03-15T22:26:24.223 INFO:tasks.rados.rados.0.smithi076.stdout:1749: finishing write tid 1 to smithi07694878-26 2021-03-15T22:26:24.223 INFO:tasks.rados.rados.0.smithi076.stdout:1747: done (3 left) 2021-03-15T22:26:24.223 INFO:tasks.rados.rados.0.smithi076.stdout:1748: done (2 left) 2021-03-15T22:26:24.223 INFO:tasks.rados.rados.0.smithi076.stdout:1751: read oid 19 snap -1 2021-03-15T22:26:24.224 INFO:tasks.rados.rados.0.smithi076.stdout:1751: expect (ObjNum 615 snap 191 seq_num 615) 2021-03-15T22:26:24.224 INFO:tasks.rados.rados.0.smithi076.stdout:1752: read oid 15 snap -1 2021-03-15T22:26:24.224 INFO:tasks.rados.rados.0.smithi076.stdout:1752: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:26:24.224 INFO:tasks.rados.rados.0.smithi076.stdout:1753: snap_create 2021-03-15T22:26:24.225 INFO:tasks.rados.rados.0.smithi076.stdout:1751: expect (ObjNum 615 snap 191 seq_num 615) 2021-03-15T22:26:24.260 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:24.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:24.285 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:24.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:24.304 INFO:tasks.rados.rados.0.smithi076.stdout:1749: finishing write tid 2 to smithi07694878-26 2021-03-15T22:26:24.304 INFO:tasks.rados.rados.0.smithi076.stdout:1749: finishing write tid 3 to smithi07694878-26 2021-03-15T22:26:24.305 INFO:tasks.rados.rados.0.smithi076.stdout:1749: finishing write tid 4 to smithi07694878-26 2021-03-15T22:26:24.305 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 786 (ObjNum 617 snap 191 seq_num 617) dirty exists 2021-03-15T22:26:24.305 INFO:tasks.rados.rados.0.smithi076.stdout:1749: left oid 26 (ObjNum 617 snap 191 seq_num 617) 2021-03-15T22:26:24.305 INFO:tasks.rados.rados.0.smithi076.stdout:1750: finishing copy_from to smithi07694878-17 2021-03-15T22:26:24.306 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 832 (ObjNum 496 snap 149 seq_num 496) dirty exists 2021-03-15T22:26:24.306 INFO:tasks.rados.rados.0.smithi076.stdout:1750: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:26:24.306 INFO:tasks.rados.rados.0.smithi076.stdout:1752: expect (ObjNum 371 snap 102 seq_num 371) 2021-03-15T22:26:24.391 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:24.417 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:24.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:24.419 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:24.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:24 smithi076 conmon[56514]: cluster 2021-03-15T22:26:23.067629+0000 mgr.y (mgr.14144) 662 : cluster [DBG] 2021-03-15T22:26:24.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:24 smithi076 conmon[56514]: pgmap v828: 138 pgs: 138 active+clean; 387 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-03-15T22:26:24.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:24 smithi076 conmon[56514]: cluster 2021-03-15T22:26:23.170686+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2021-03-15T22:26:24.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:24 smithi076 conmon[56514]: audit 2021-03-15T22:26:23.963775+0000 mon.a (mon.0) 1324 : audit [DBG] from='client.? 172.21.15.76:0/142236951' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:26:24.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:24 smithi076 conmon[62156]: cluster 2021-03-15T22:26:23. 2021-03-15T22:26:24.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:24 smithi076 conmon[62156]: 067629+0000 mgr.y (mgr.14144) 662 : cluster [DBG] pgmap v828: 138 pgs: 138 active+clean; 387 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-03-15T22:26:24.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:24 smithi076 conmon[62156]: cluster 2021-03-15T22:26:23.170686+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2021-03-15T22:26:24.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:24 smithi076 conmon[62156]: audit 2021-03-15T22:26:24.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:24 smithi076 conmon[62156]: 2021-03-15T22:26:23.963775+0000 2021-03-15T22:26:24.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:24 smithi076 conmon[62156]: mon.a (mon.0) 1324 2021-03-15T22:26:24.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:24 smithi076 conmon[62156]: : audit [DBG] from='client.? 172.21.15.76:0/142236951' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:26:24.520 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:24.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:24 smithi125 conmon[56739]: cluster 2021-03-15T22:26:23.067629+0000 mgr.y (mgr.14144) 662 : cluster [DBG] pgmap v828: 138 pgs: 138 active+clean; 387 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 1.5 MiB/s rd, 2.3 MiB/s wr, 12 op/s 2021-03-15T22:26:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:24 smithi125 conmon[56739]: cluster 2021-03-15T22:26:23.170686+0000 mon.a (mon.0) 1323 : cluster [DBG] osdmap e468: 8 total, 8 up, 7 in 2021-03-15T22:26:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:24 smithi125 conmon[56739]: audit 2021-03-15T22:26:23.963775+0000 mon.a (mon.0) 1324 : audit 2021-03-15T22:26:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:24 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/142236951' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:26:24.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:24.544 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:24.544 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:24.585 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:26:24.585 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:26:24.586 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 468, 2021-03-15T22:26:24.586 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:26:24.586 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2021-03-15T22:13:20.370795+0000", 2021-03-15T22:26:24.586 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2021-03-15T22:26:23.158722+0000", 2021-03-15T22:26:24.587 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2021-03-15T22:17:49.162889+0000", 2021-03-15T22:26:24.587 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2021-03-15T22:24:30.070026+0000", 2021-03-15T22:26:24.587 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-03-15T22:26:24.587 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5801984, 2021-03-15T22:26:24.588 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2021-03-15T22:26:24.588 INFO:teuthology.orchestra.run.smithi076.stdout: "noscrub", 2021-03-15T22:26:24.588 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2021-03-15T22:26:24.588 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2021-03-15T22:26:24.588 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2021-03-15T22:26:24.589 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2021-03-15T22:26:24.589 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:26:24.589 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 18, 2021-03-15T22:26:24.589 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2021-03-15T22:26:24.590 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-03-15T22:26:24.590 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2021-03-15T22:26:24.590 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2021-03-15T22:26:24.590 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 7, 2021-03-15T22:26:24.590 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2021-03-15T22:26:24.591 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2021-03-15T22:26:24.591 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2021-03-15T22:26:24.591 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2021-03-15T22:26:24.591 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2021-03-15T22:26:24.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.592 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2021-03-15T22:26:24.592 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "device_health_metrics", 2021-03-15T22:26:24.592 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:16:34.817191+0000", 2021-03-15T22:26:24.592 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:26:24.593 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:26:24.593 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:26:24.593 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:26:24.593 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:26:24.593 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:26:24.594 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:26:24.594 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:26:24.594 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:26:24.594 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:26:24.595 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:26:24.595 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:26:24.595 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2021-03-15T22:26:24.595 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2021-03-15T22:26:24.595 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2021-03-15T22:26:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2021-03-15T22:26:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2021-03-15T22:26:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:26:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:26:24.597 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:26:24.597 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:26:24.597 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:26:24.597 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:26:24.597 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:26:24.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.598 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "23", 2021-03-15T22:26:24.602 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:26:24.603 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:26:24.603 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:26:24.603 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:26:24.603 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:26:24.604 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:26:24.604 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:26:24.604 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:26:24.604 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:26:24.605 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:26:24.605 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:26:24.605 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:26:24.605 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:26:24.605 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:26:24.606 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:26:24.606 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:26:24.606 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:26:24.606 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:26:24.607 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:26:24.607 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:26:24.607 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:26:24.607 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:26:24.607 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:26:24.608 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:26:24.608 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:26:24.608 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:26:24.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.608 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:26:24.609 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:26:24.609 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:26:24.609 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:26:24.609 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:26:24.610 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:26:24.610 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:26:24.610 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:26:24.610 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:26:24.610 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:26:24.611 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:26:24.611 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:26:24.611 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2021-03-15T22:26:24.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.612 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:26:24.612 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_devicehealth": {} 2021-03-15T22:26:24.612 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.613 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2021-03-15T22:26:24.613 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "iscsi", 2021-03-15T22:26:24.613 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:49.906400+0000", 2021-03-15T22:26:24.613 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:26:24.614 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:26:24.614 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:26:24.614 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:26:24.614 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:26:24.614 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:26:24.615 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:26:24.615 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:26:24.615 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:26:24.615 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:26:24.616 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:26:24.616 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:26:24.616 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 3, 2021-03-15T22:26:24.616 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 3, 2021-03-15T22:26:24.616 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 3, 2021-03-15T22:26:24.617 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 3, 2021-03-15T22:26:24.617 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 3, 2021-03-15T22:26:24.617 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:26:24.617 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:26:24.617 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:26:24.618 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:26:24.618 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:26:24.619 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:26:24.620 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:26:24.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.620 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "56", 2021-03-15T22:26:24.621 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:26:24.621 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:26:24.621 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:26:24.621 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:26:24.621 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:26:24.622 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:26:24.622 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:26:24.622 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:26:24.622 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:26:24.623 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:26:24.623 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:26:24.623 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:26:24.624 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:26:24.624 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:26:24.624 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:26:24.625 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:26:24.625 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:26:24.625 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:26:24.625 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:26:24.625 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:26:24.626 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:26:24.626 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:26:24.626 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:26:24.626 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:26:24.627 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:26:24.627 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:26:24.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.627 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:26:24.627 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:26:24.628 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:26:24.628 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:26:24.628 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:26:24.628 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:26:24.629 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:26:24.629 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:26:24.629 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:26:24.629 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:26:24.629 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:26:24.630 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:26:24.631 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:26:24.631 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2021-03-15T22:26:24.631 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.632 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2021-03-15T22:26:24.632 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".rgw.root", 2021-03-15T22:26:24.632 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:52.257970+0000", 2021-03-15T22:26:24.632 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:26:24.633 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:26:24.633 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:26:24.633 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:26:24.633 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:26:24.634 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:26:24.634 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:26:24.634 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:26:24.634 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:26:24.635 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:26:24.635 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:26:24.635 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:26:24.636 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:26:24.636 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:26:24.636 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:26:24.638 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:26:24.638 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:26:24.638 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:26:24.639 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:26:24.639 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:26:24.639 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:26:24.640 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:26:24.640 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:26:24.640 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:26:24.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.641 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "58", 2021-03-15T22:26:24.641 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:26:24.641 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:26:24.641 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:26:24.641 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:26:24.642 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:26:24.642 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:26:24.642 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:26:24.642 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:26:24.642 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:26:24.643 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:26:24.643 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:26:24.643 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:26:24.643 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:26:24.644 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:26:24.644 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:26:24.644 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:26:24.644 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:26:24.644 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:26:24.645 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:26:24.645 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:26:24.645 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:26:24.645 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:26:24.645 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:26:24.646 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:26:24.646 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:26:24.646 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:26:24.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.647 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:26:24.647 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:26:24.647 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:26:24.647 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:26:24.647 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:26:24.648 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:26:24.648 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:26:24.648 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:26:24.648 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:26:24.649 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:26:24.649 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:26:24.649 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:26:24.649 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:26:24.649 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:26:24.650 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.650 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2021-03-15T22:26:24.651 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.log", 2021-03-15T22:26:24.651 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:54.562498+0000", 2021-03-15T22:26:24.651 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:26:24.651 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:26:24.651 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:26:24.652 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:26:24.652 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:26:24.652 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:26:24.652 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:26:24.652 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:26:24.653 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:26:24.653 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:26:24.653 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:26:24.653 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:26:24.654 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:26:24.654 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:26:24.654 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:26:24.654 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:26:24.654 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:26:24.655 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:26:24.655 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:26:24.655 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:26:24.655 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:26:24.655 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:26:24.656 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:26:24.656 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:26:24.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.656 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "60", 2021-03-15T22:26:24.657 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:26:24.657 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:26:24.657 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:26:24.657 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:26:24.657 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:26:24.658 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:26:24.658 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:26:24.658 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:26:24.658 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:26:24.659 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:26:24.659 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:26:24.659 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:26:24.659 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:26:24.659 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:26:24.660 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:26:24.660 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:26:24.660 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:26:24.660 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:26:24.660 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:26:24.661 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:26:24.661 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:26:24.661 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:26:24.661 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:26:24.662 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:26:24.662 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:26:24.662 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:26:24.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.662 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:26:24.663 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:26:24.663 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:26:24.663 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:26:24.663 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:26:24.663 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:26:24.664 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:26:24.664 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:26:24.664 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:26:24.665 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:26:24.665 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:26:24.665 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:26:24.665 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:26:24.665 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:26:24.666 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.666 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 5, 2021-03-15T22:26:24.666 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.control", 2021-03-15T22:26:24.667 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:56.243806+0000", 2021-03-15T22:26:24.667 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:26:24.667 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:26:24.667 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:26:24.668 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:26:24.668 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:26:24.668 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:26:24.668 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:26:24.668 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:26:24.669 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:26:24.669 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:26:24.669 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:26:24.669 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:26:24.669 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 22, 2021-03-15T22:26:24.670 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 22, 2021-03-15T22:26:24.670 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 22, 2021-03-15T22:26:24.670 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 22, 2021-03-15T22:26:24.670 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 22, 2021-03-15T22:26:24.671 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:26:24.671 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "5.16", 2021-03-15T22:26:24.671 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 182, 2021-03-15T22:26:24.671 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 181, 2021-03-15T22:26:24.671 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 181, 2021-03-15T22:26:24.672 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:26:24.672 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:26:24.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.672 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "182", 2021-03-15T22:26:24.672 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:26:24.673 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "182", 2021-03-15T22:26:24.673 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "180", 2021-03-15T22:26:24.673 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:26:24.673 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:26:24.674 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:26:24.674 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:26:24.674 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:26:24.674 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:26:24.674 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:26:24.675 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:26:24.675 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:26:24.675 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:26:24.675 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:26:24.676 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:26:24.676 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:26:24.676 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:26:24.676 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:26:24.676 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:26:24.677 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:26:24.677 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:26:24.677 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:26:24.677 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:26:24.677 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:26:24.678 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:26:24.678 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:26:24.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.679 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:26:24.679 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:26:24.679 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:26:24.679 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:26:24.680 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:26:24.680 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:26:24.680 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:26:24.680 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:26:24.680 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:26:24.681 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:26:24.681 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:26:24.681 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:26:24.681 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:26:24.681 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:26:24.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.682 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 6, 2021-03-15T22:26:24.683 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.meta", 2021-03-15T22:26:24.683 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:58.730534+0000", 2021-03-15T22:26:24.683 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:26:24.683 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:26:24.683 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:26:24.684 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:26:24.684 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:26:24.684 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:26:24.684 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:26:24.685 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:26:24.685 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:26:24.685 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:26:24.685 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:26:24.685 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:26:24.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:26:24.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:26:24.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:26:24.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:26:24.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:26:24.687 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:26:24.687 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:26:24.687 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:26:24.687 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:26:24.688 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:26:24.688 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:26:24.688 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:26:24.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.688 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "109", 2021-03-15T22:26:24.689 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:26:24.689 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:26:24.689 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:26:24.689 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:26:24.689 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:26:24.690 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:26:24.690 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:26:24.690 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:26:24.690 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:26:24.690 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:26:24.691 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:26:24.691 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:26:24.691 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:26:24.691 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:26:24.692 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:26:24.692 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:26:24.692 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:26:24.692 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:26:24.693 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:26:24.693 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:26:24.693 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:26:24.693 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:26:24.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:26:24.694 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:26:24.694 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:26:24.694 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:26:24.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.695 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:26:24.695 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:26:24.695 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:26:24.695 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:26:24.696 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:26:24.696 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:26:24.696 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:26:24.696 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:26:24.696 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:26:24.697 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:26:24.697 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:26:24.697 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:26:24.697 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_bias": 4, 2021-03-15T22:26:24.698 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 8 2021-03-15T22:26:24.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.698 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:26:24.698 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:26:24.698 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.699 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:26:24.699 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2021-03-15T22:26:24.700 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:20:03.377294+0000", 2021-03-15T22:26:24.700 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2021-03-15T22:26:24.700 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-03-15T22:26:24.700 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:26:24.700 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:26:24.701 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:26:24.701 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:26:24.701 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:26:24.701 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:26:24.701 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:26:24.702 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:26:24.702 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:26:24.702 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:26:24.702 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2021-03-15T22:26:24.702 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2021-03-15T22:26:24.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2021-03-15T22:26:24.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2021-03-15T22:26:24.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2021-03-15T22:26:24.703 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:26:24.704 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:26:24.704 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:26:24.704 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:26:24.704 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:26:24.704 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:26:24.705 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:26:24.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.705 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "468", 2021-03-15T22:26:24.705 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:26:24.705 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:26:24.706 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:26:24.706 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:26:24.706 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:26:24.706 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 364, 2021-03-15T22:26:24.707 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 468, 2021-03-15T22:26:24.707 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:26:24.707 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:26:24.707 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:26:24.707 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:26:24.708 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:26:24.708 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:26:24.708 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:26:24.708 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:26:24.708 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:26:24.709 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:26:24.709 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:26:24.709 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:26:24.709 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:26:24.709 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:26:24.710 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:26:24.710 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:26:24.710 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:26:24.710 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:26:24.711 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:26:24.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.711 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:26:24.711 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:26:24.711 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:26:24.712 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:26:24.712 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:26:24.712 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:26:24.712 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:26:24.712 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:26:24.713 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:26:24.713 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:26:24.713 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:26:24.713 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:26:24.714 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:26:24.714 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2021-03-15T22:26:24.714 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.714 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.715 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:26:24.715 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:26:24.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.715 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:26:24.715 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd", 2021-03-15T22:26:24.716 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:26:24.716 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 0, 2021-03-15T22:26:24.716 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2021-03-15T22:26:24.716 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:26:24.716 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:26:24.717 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:26:24.717 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 8, 2021-03-15T22:26:24.717 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 350, 2021-03-15T22:26:24.717 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:26:24.717 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:26:24.718 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:26:24.718 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.718 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.719 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2021-03-15T22:26:24.719 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:26:24.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.719 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.720 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2021-03-15T22:26:24.720 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:26:24.720 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.720 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.721 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:26:24.721 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.721 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.721 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2021-03-15T22:26:24.722 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:26:24.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.722 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.723 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2021-03-15T22:26:24.723 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:26:24.723 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.723 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.724 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:26:24.724 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.724 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.724 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2021-03-15T22:26:24.725 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:26:24.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.725 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.725 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2021-03-15T22:26:24.726 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:26:24.726 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.726 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.727 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:26:24.727 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.727 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.727 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2021-03-15T22:26:24.728 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:26:24.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.728 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.728 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2021-03-15T22:26:24.729 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:26:24.729 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.729 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.729 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/188228176", 2021-03-15T22:26:24.730 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/188228176", 2021-03-15T22:26:24.730 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/188228176", 2021-03-15T22:26:24.730 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/188228176", 2021-03-15T22:26:24.730 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:26:24.731 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:26:24.731 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:26:24.731 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.732 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:26:24.732 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526", 2021-03-15T22:26:24.732 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:26:24.732 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:26:24.732 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:26:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.999237060546875, 2021-03-15T22:26:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:26:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:26:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 13, 2021-03-15T22:26:24.733 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 417, 2021-03-15T22:26:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:26:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:26:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:26:24.734 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2021-03-15T22:26:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:26:24.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2021-03-15T22:26:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:26:24.736 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:26:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2021-03-15T22:26:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:26:24.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2021-03-15T22:26:24.739 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:26:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:26:24.740 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2021-03-15T22:26:24.741 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:26:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2021-03-15T22:26:24.742 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:26:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:26:24.743 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.744 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.744 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2021-03-15T22:26:24.744 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:26:24.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.745 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2021-03-15T22:26:24.745 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:26:24.746 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.746 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.746 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/3924955423", 2021-03-15T22:26:24.746 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/3924955423", 2021-03-15T22:26:24.747 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/3924955423", 2021-03-15T22:26:24.747 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/3924955423", 2021-03-15T22:26:24.747 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:26:24.747 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:26:24.747 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:26:24.748 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.748 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:26:24.748 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05", 2021-03-15T22:26:24.749 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:26:24.749 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:26:24.749 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:26:24.749 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:26:24.749 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:26:24.750 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:26:24.750 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 18, 2021-03-15T22:26:24.750 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 421, 2021-03-15T22:26:24.750 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:26:24.751 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:26:24.751 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:26:24.751 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.751 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.752 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2021-03-15T22:26:24.752 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:26:24.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.752 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.753 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2021-03-15T22:26:24.753 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:26:24.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.753 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.754 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:26:24.754 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.754 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.755 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2021-03-15T22:26:24.755 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:26:24.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.755 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.756 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2021-03-15T22:26:24.756 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:26:24.756 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.756 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.757 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:26:24.757 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.757 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.758 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2021-03-15T22:26:24.758 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:26:24.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.758 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.759 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2021-03-15T22:26:24.759 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:26:24.759 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.759 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.760 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:26:24.760 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.760 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.761 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2021-03-15T22:26:24.761 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:26:24.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.761 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2021-03-15T22:26:24.762 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:26:24.762 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.762 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.763 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/1401263050", 2021-03-15T22:26:24.763 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/1401263050", 2021-03-15T22:26:24.763 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/1401263050", 2021-03-15T22:26:24.763 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/1401263050", 2021-03-15T22:26:24.763 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:26:24.764 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:26:24.764 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:26:24.764 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.765 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:26:24.765 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56", 2021-03-15T22:26:24.765 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:26:24.765 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:26:24.766 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:26:24.766 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:26:24.766 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:26:24.766 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:26:24.766 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 26, 2021-03-15T22:26:24.767 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 423, 2021-03-15T22:26:24.767 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:26:24.767 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:26:24.767 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:26:24.767 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.768 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.768 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2021-03-15T22:26:24.768 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:26:24.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.769 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.769 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2021-03-15T22:26:24.769 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:26:24.770 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.770 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.770 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:26:24.770 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.771 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.771 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2021-03-15T22:26:24.771 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:26:24.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.772 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.772 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2021-03-15T22:26:24.772 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:26:24.773 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.773 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.773 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:26:24.773 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.774 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.774 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2021-03-15T22:26:24.774 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:26:24.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.775 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.775 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2021-03-15T22:26:24.775 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:26:24.776 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.776 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.776 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:26:24.776 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.777 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.777 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2021-03-15T22:26:24.777 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:26:24.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.778 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.778 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2021-03-15T22:26:24.778 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:26:24.778 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.779 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.779 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/3960968636", 2021-03-15T22:26:24.779 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/3960968636", 2021-03-15T22:26:24.779 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/3960968636", 2021-03-15T22:26:24.780 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/3960968636", 2021-03-15T22:26:24.780 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:26:24.780 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:26:24.780 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:26:24.781 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.781 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:26:24.781 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237", 2021-03-15T22:26:24.782 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:26:24.782 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:26:24.782 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:26:24.782 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:26:24.782 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:26:24.783 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:26:24.783 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 31, 2021-03-15T22:26:24.783 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 422, 2021-03-15T22:26:24.783 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:26:24.783 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:26:24.784 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:26:24.784 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.784 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.785 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6800", 2021-03-15T22:26:24.785 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:26:24.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.785 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.786 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6801", 2021-03-15T22:26:24.786 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:26:24.786 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.786 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.787 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:26:24.787 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.787 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.787 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6802", 2021-03-15T22:26:24.788 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:26:24.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.788 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.788 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6803", 2021-03-15T22:26:24.789 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:26:24.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.789 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.790 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:26:24.790 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.790 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6806", 2021-03-15T22:26:24.791 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:26:24.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.791 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.791 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6807", 2021-03-15T22:26:24.792 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:26:24.792 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.792 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.793 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:26:24.793 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.794 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.794 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6804", 2021-03-15T22:26:24.794 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:26:24.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.795 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.795 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6805", 2021-03-15T22:26:24.795 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:26:24.795 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.795 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.796 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6801/37386122", 2021-03-15T22:26:24.796 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6803/37386122", 2021-03-15T22:26:24.796 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6807/37386122", 2021-03-15T22:26:24.797 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6805/37386122", 2021-03-15T22:26:24.797 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:26:24.797 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:26:24.797 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:26:24.797 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.798 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:26:24.798 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2", 2021-03-15T22:26:24.798 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:26:24.799 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:26:24.799 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:26:24.799 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:26:24.799 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:26:24.799 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:26:24.800 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 38, 2021-03-15T22:26:24.800 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 429, 2021-03-15T22:26:24.800 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:26:24.800 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:26:24.801 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:26:24.801 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.801 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.801 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6808", 2021-03-15T22:26:24.802 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:26:24.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.802 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.802 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6809", 2021-03-15T22:26:24.803 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:26:24.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.803 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.803 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:26:24.804 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.804 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.804 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6810", 2021-03-15T22:26:24.805 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:26:24.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.805 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.805 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6811", 2021-03-15T22:26:24.806 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:26:24.806 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.806 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.806 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:26:24.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.807 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6814", 2021-03-15T22:26:24.807 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:26:24.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.808 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6815", 2021-03-15T22:26:24.809 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:26:24.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.809 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.809 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:26:24.810 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.810 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.810 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6812", 2021-03-15T22:26:24.810 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:26:24.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.811 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.811 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6813", 2021-03-15T22:26:24.811 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:26:24.812 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.812 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.812 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6809/3610838738", 2021-03-15T22:26:24.813 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6811/3610838738", 2021-03-15T22:26:24.813 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6815/3610838738", 2021-03-15T22:26:24.813 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6813/3610838738", 2021-03-15T22:26:24.813 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:26:24.813 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:26:24.814 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:26:24.814 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.814 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:26:24.815 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe", 2021-03-15T22:26:24.815 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:26:24.815 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:26:24.815 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:26:24.815 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:26:24.816 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:26:24.816 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:26:24.816 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 45, 2021-03-15T22:26:24.816 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 420, 2021-03-15T22:26:24.817 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:26:24.817 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:26:24.817 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:26:24.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.818 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.818 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6816", 2021-03-15T22:26:24.818 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:26:24.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.819 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.819 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6817", 2021-03-15T22:26:24.819 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:26:24.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.819 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.820 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:26:24.820 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.821 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.821 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6818", 2021-03-15T22:26:24.821 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:26:24.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.822 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.822 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6819", 2021-03-15T22:26:24.822 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:26:24.822 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.822 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.823 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:26:24.823 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.824 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6822", 2021-03-15T22:26:24.824 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:26:24.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.825 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.825 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6823", 2021-03-15T22:26:24.825 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:26:24.825 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.825 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.826 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:26:24.826 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.826 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.827 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6820", 2021-03-15T22:26:24.827 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:26:24.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.827 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.828 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6821", 2021-03-15T22:26:24.828 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:26:24.828 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.828 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.829 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6817/171378275", 2021-03-15T22:26:24.829 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6819/171378275", 2021-03-15T22:26:24.829 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6823/171378275", 2021-03-15T22:26:24.829 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6821/171378275", 2021-03-15T22:26:24.830 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:26:24.830 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:26:24.830 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:26:24.830 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.831 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:26:24.831 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614", 2021-03-15T22:26:24.831 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:26:24.832 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:26:24.832 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:26:24.832 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:26:24.832 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:26:24.832 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:26:24.833 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 52, 2021-03-15T22:26:24.833 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 421, 2021-03-15T22:26:24.833 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:26:24.833 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:26:24.833 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:26:24.834 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.834 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.834 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6824", 2021-03-15T22:26:24.835 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:26:24.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.835 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6825", 2021-03-15T22:26:24.836 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:26:24.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.836 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.836 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:26:24.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6826", 2021-03-15T22:26:24.838 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:26:24.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.838 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.838 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6827", 2021-03-15T22:26:24.839 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:26:24.839 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.839 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.839 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:26:24.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.840 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6830", 2021-03-15T22:26:24.841 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:26:24.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.841 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6831", 2021-03-15T22:26:24.842 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:26:24.842 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.842 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.842 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:26:24.843 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:26:24.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.843 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:26:24.843 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6828", 2021-03-15T22:26:24.843 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:26:24.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.844 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:26:24.844 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6829", 2021-03-15T22:26:24.845 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:26:24.845 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.845 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.845 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6825/652003948", 2021-03-15T22:26:24.846 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6827/652003948", 2021-03-15T22:26:24.846 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6831/652003948", 2021-03-15T22:26:24.846 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6829/652003948", 2021-03-15T22:26:24.846 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:26:24.846 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:26:24.847 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:26:24.847 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.847 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:26:24.847 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2021-03-15T22:26:24.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.848 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:26:24.848 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:26:24.848 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:26:24.848 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:26:24.849 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:26:24.849 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 65536, 2021-03-15T22:26:24.849 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:25:53.782645+0000", 2021-03-15T22:26:24.849 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:26:24.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.850 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:26:24.850 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:26:24.850 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:26:24.851 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:26:24.851 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:26:24.851 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:26:24.851 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:26:11.750694+0000", 2021-03-15T22:26:24.851 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:26:24.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.852 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:26:24.852 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:26:24.852 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:26:24.853 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:26:24.853 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:26:24.853 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:26:24.853 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:25:25.793083+0000", 2021-03-15T22:26:24.854 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:26:24.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.854 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:26:24.854 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:26:24.855 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:26:24.855 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:26:24.855 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:26:24.855 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:26:24.855 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:25:40.361290+0000", 2021-03-15T22:26:24.856 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:26:24.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.856 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:26:24.856 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:26:24.857 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:26:24.857 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:26:24.857 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:26:24.857 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:26:24.858 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:26:00.262440+0000", 2021-03-15T22:26:24.858 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:26:24.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.858 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:26:24.859 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:26:24.859 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:26:24.859 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:26:24.859 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:26:24.859 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:26:24.860 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:26:16.599305+0000", 2021-03-15T22:26:24.860 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:26:24.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.861 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:26:24.861 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:26:24.861 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:26:24.861 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:26:24.861 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:26:24.862 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:26:24.862 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:25:33.525879+0000", 2021-03-15T22:26:24.862 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:26:24.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.863 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:26:24.863 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:26:24.863 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:26:24.863 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:26:24.863 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:26:24.864 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:26:24.864 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:25:52.277742+0000", 2021-03-15T22:26:24.864 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:26:24.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.865 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:26:24.865 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [], 2021-03-15T22:26:24.865 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2021-03-15T22:26:24.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.865 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.3", 2021-03-15T22:26:24.866 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:26:24.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.866 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:26:24.866 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:26:24.866 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.867 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.867 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.14", 2021-03-15T22:26:24.867 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:26:24.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.868 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:26:24.868 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:26:24.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.869 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.869 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.a", 2021-03-15T22:26:24.869 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:26:24.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.870 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:26:24.870 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:26:24.870 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.870 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.871 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.b", 2021-03-15T22:26:24.871 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:26:24.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.872 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:26:24.872 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:26:24.872 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.872 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.873 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2021-03-15T22:26:24.873 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:26:24.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.874 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:26:24.874 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:26:24.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.874 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.875 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.17", 2021-03-15T22:26:24.875 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:26:24.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.875 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:26:24.876 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 5 2021-03-15T22:26:24.876 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.876 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.877 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.1b", 2021-03-15T22:26:24.877 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:26:24.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.877 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:26:24.878 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 5 2021-03-15T22:26:24.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.878 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.878 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.878 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:26:24.879 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2021-03-15T22:26:24.879 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2021-03-15T22:26:24.879 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": { 2021-03-15T22:26:24.879 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:26:24.879 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4087324200": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:26:24.880 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4141143304": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:26:24.880 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:26:24.880 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2779879991": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:26:24.880 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:26:24.881 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:26:24.881 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2483102204": "2021-03-16T22:13:41.603652+0000" 2021-03-15T22:26:24.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.881 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2021-03-15T22:26:24.881 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2021-03-15T22:26:24.882 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2021-03-15T22:26:24.882 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2021-03-15T22:26:24.882 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2021-03-15T22:26:24.882 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2021-03-15T22:26:24.882 INFO:teuthology.orchestra.run.smithi076.stdout: "ruleset-failure-domain": "osd", 2021-03-15T22:26:24.883 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2021-03-15T22:26:24.883 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.883 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2021-03-15T22:26:24.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.884 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:26:24.884 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:26:24.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.884 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 294, 2021-03-15T22:26:24.885 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:26:24.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.885 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 296, 2021-03-15T22:26:24.885 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:26:24.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.886 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 307, 2021-03-15T22:26:24.886 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:26:24.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.887 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 323, 2021-03-15T22:26:24.887 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:26:24.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.888 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 328, 2021-03-15T22:26:24.888 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:26:24.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.889 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 337, 2021-03-15T22:26:24.889 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:26:24.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:26:24.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.889 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 339, 2021-03-15T22:26:24.890 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:26:24.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.890 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.890 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.890 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:26:24.891 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [ 2021-03-15T22:26:24.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.891 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:26:24.891 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:26:24.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:26:24.892 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 294, 2021-03-15T22:26:24.892 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:26:24.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.892 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:26:24.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.893 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:26:24.893 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2021-03-15T22:26:24.893 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2021-03-15T22:26:24.893 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2021-03-15T22:26:24.894 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2021-03-15T22:26:24.894 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2021-03-15T22:26:24.894 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2021-03-15T22:26:24.894 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2021-03-15T22:26:24.894 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2021-03-15T22:26:24.895 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2021-03-15T22:26:24.895 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:26:24.895 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:26:24.895 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:24.917 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:24.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:24.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:25.020 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:25.046 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:25.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:25.048 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:25.123 INFO:tasks.rados.rados.0.smithi076.stdout:1749: done (4 left) 2021-03-15T22:26:25.123 INFO:tasks.rados.rados.0.smithi076.stdout:1750: done (3 left) 2021-03-15T22:26:25.124 INFO:tasks.rados.rados.0.smithi076.stdout:1751: done (2 left) 2021-03-15T22:26:25.124 INFO:tasks.rados.rados.0.smithi076.stdout:1752: done (1 left) 2021-03-15T22:26:25.124 INFO:tasks.rados.rados.0.smithi076.stdout:1753: done (0 left) 2021-03-15T22:26:25.124 INFO:tasks.rados.rados.0.smithi076.stdout:1754: snap_remove snap 184 2021-03-15T22:26:25.149 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:25.175 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:25.175 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:25.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:25.302 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:25.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:25.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:25.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:25.428 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:25.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:25.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:25.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:25 smithi076 conmon[56514]: cluster 2021-03-15T22:26:24.175856+0000 mon.a (mon.0) 1325 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2021-03-15T22:26:25.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:25 smithi076 conmon[56514]: cluster 2021-03-15T22:26:25.118925+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2021-03-15T22:26:25.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:25 smithi076 conmon[62156]: cluster 2021-03-15T22:26:24.175856+0000 mon.a 2021-03-15T22:26:25.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:25 smithi076 conmon[62156]: (mon.0) 1325 : cluster [DBG] osdmap e469: 8 total, 8 up, 7 in 2021-03-15T22:26:25.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:25 smithi076 conmon[62156]: cluster 2021-03-15T22:26:25.118925+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2021-03-15T22:26:25.531 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:25.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:25 smithi125 conmon[56739]: cluster 2021-03-15T22:26:24.175856+0000 mon.a (mon.0) 1325 : cluster [DBG] 2021-03-15T22:26:25.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:25 smithi125 conmon[56739]: osdmap e469: 8 total, 8 up, 7 in 2021-03-15T22:26:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:25 smithi125 conmon[56739]: cluster 2021-03-15T22:26:25.118925+0000 mon.a (mon.0) 1326 : cluster [DBG] osdmap e470: 8 total, 8 up, 7 in 2021-03-15T22:26:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:25.558 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:25.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:25.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:25.685 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:25.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:25.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:25.770 INFO:tasks.thrashosds.thrasher:j is {'epoch': 468, 'fsid': '8a572802-85db-11eb-9091-001a4aab830c', 'created': '2021-03-15T22:13:20.370795+0000', 'modified': '2021-03-15T22:26:23.158722+0000', 'last_up_change': '2021-03-15T22:17:49.162889+0000', 'last_in_change': '2021-03-15T22:24:30.070026+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-15T22:16:34.817191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-15T22:17:49.906400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-15T22:17:52.257970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-15T22:17:54.562498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-15T22:17:56.243806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 182, 'last_epoch_started': 181, 'last_epoch_clean': 181, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '182', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '182', 'last_force_op_resend_preluminous': '180', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-15T22:17:58.730534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-03-15T22:20:03.377294+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '468', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 364, 'snap_epoch': 468, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 350, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 188228176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 188228176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 188228176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 188228176}]}, 'public_addr': '172.21.15.76:6803/188228176', 'cluster_addr': '172.21.15.76:6805/188228176', 'heartbeat_back_addr': '172.21.15.76:6809/188228176', 'heartbeat_front_addr': '172.21.15.76:6807/188228176', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '97bd3ffc-15c8-4a6f-b37a-217813def526', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.999237060546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 417, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 3924955423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 3924955423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 3924955423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 3924955423}]}, 'public_addr': '172.21.15.76:6811/3924955423', 'cluster_addr': '172.21.15.76:6813/3924955423', 'heartbeat_back_addr': '172.21.15.76:6817/3924955423', 'heartbeat_front_addr': '172.21.15.76:6815/3924955423', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a7f5f115-e69e-47d9-b6c3-ff386ddfaa05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 421, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 1401263050}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 1401263050}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 1401263050}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 1401263050}]}, 'public_addr': '172.21.15.76:6819/1401263050', 'cluster_addr': '172.21.15.76:6821/1401263050', 'heartbeat_back_addr': '172.21.15.76:6825/1401263050', 'heartbeat_front_addr': '172.21.15.76:6823/1401263050', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a0bb7f88-0114-47cb-a82c-d8fb002aac56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 423, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 3960968636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 3960968636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 3960968636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 3960968636}]}, 'public_addr': '172.21.15.76:6827/3960968636', 'cluster_addr': '172.21.15.76:6829/3960968636', 'heartbeat_back_addr': '172.21.15.76:6833/3960968636', 'heartbeat_front_addr': '172.21.15.76:6831/3960968636', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b650611c-6101-47e8-9ca9-94e02cf8b237', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 422, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6800', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6801', 'nonce': 37386122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6802', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6803', 'nonce': 37386122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6806', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6807', 'nonce': 37386122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6804', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6805', 'nonce': 37386122}]}, 'public_addr': '172.21.15.125:6801/37386122', 'cluster_addr': '172.21.15.125:6803/37386122', 'heartbeat_back_addr': '172.21.15.125:6807/37386122', 'heartbeat_front_addr': '172.21.15.125:6805/37386122', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7d6ceff2-7093-4ab9-8b96-ade448093bf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 429, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6808', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6809', 'nonce': 3610838738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6810', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6811', 'nonce': 3610838738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6814', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6815', 'nonce': 3610838738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6812', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6813', 'nonce': 3610838738}]}, 'public_addr': '172.21.15.125:6809/3610838738', 'cluster_addr': '172.21.15.125:6811/3610838738', 'heartbeat_back_addr': '172.21.15.125:6815/3610838738', 'heartbeat_front_addr': '172.21.15.125:6813/3610838738', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98b72022-a9a7-4ee2-89a1-cac037de7afe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 420, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6816', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6817', 'nonce': 171378275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6818', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6819', 'nonce': 171378275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6822', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6823', 'nonce': 171378275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6820', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6821', 'nonce': 171378275}]}, 'public_addr': '172.21.15.125:6817/171378275', 'cluster_addr': '172.21.15.125:6819/171378275', 'heartbeat_back_addr': '172.21.15.125:6823/171378275', 'heartbeat_front_addr': '172.21.15.125:6821/171378275', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f99ea650-b2dc-4d87-a44c-56a897edd614', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 421, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6824', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6825', 'nonce': 652003948}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6826', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6827', 'nonce': 652003948}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6830', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6831', 'nonce': 652003948}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6828', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6829', 'nonce': 652003948}]}, 'public_addr': '172.21.15.125:6825/652003948', 'cluster_addr': '172.21.15.125:6827/652003948', 'heartbeat_back_addr': '172.21.15.125:6831/652003948', 'heartbeat_front_addr': '172.21.15.125:6829/652003948', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 65536, 'last_purged_snaps_scrub': '2021-03-15T22:25:53.782645+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:26:11.750694+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:25:25.793083+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:25:40.361290+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:26:00.262440+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:26:16.599305+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:25:33.525879+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:25:52.277742+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.14', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.76:6800/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4087324200': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4141143304': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:6801/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/2779879991': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6801/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6800/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:0/2483102204': '2021-03-16T22:13:41.603652+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 294, 'length': 1}, {'begin': 296, 'length': 1}, {'begin': 307, 'length': 1}, {'begin': 323, 'length': 1}, {'begin': 328, 'length': 1}, {'begin': 337, 'length': 1}, {'begin': 339, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 294, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-03-15T22:26:25.771 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:26:25.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:25.813 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:25.815 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:25.916 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:25.942 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:25.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:25.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:26.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:26.068 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:26.070 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:26.128 INFO:tasks.rados.rados.0.smithi076.stdout:1754: done (0 left) 2021-03-15T22:26:26.129 INFO:tasks.rados.rados.0.smithi076.stdout:1755: read oid 20 snap -1 2021-03-15T22:26:26.129 INFO:tasks.rados.rados.0.smithi076.stdout:1755: expect (ObjNum 554 snap 176 seq_num 554) 2021-03-15T22:26:26.129 INFO:tasks.rados.rados.0.smithi076.stdout:1756: write oid 17 current snap is 192 2021-03-15T22:26:26.130 INFO:tasks.rados.rados.0.smithi076.stdout:1756: seq_num 618 ranges {786726=728475,2280813=105913} 2021-03-15T22:26:26.137 INFO:tasks.rados.rados.0.smithi076.stdout:1756: writing smithi07694878-17 from 786726 to 1515201 tid 1 2021-03-15T22:26:26.143 INFO:tasks.rados.rados.0.smithi076.stdout:1756: writing smithi07694878-17 from 2280813 to 2386726 tid 2 2021-03-15T22:26:26.143 INFO:tasks.rados.rados.0.smithi076.stdout:1757: read oid 44 snap -1 2021-03-15T22:26:26.143 INFO:tasks.rados.rados.0.smithi076.stdout:1757: expect deleted 2021-03-15T22:26:26.144 INFO:tasks.rados.rados.0.smithi076.stdout:1758: snap_create 2021-03-15T22:26:26.146 INFO:tasks.rados.rados.0.smithi076.stdout:1755: expect (ObjNum 554 snap 176 seq_num 554) 2021-03-15T22:26:26.156 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:26:26.172 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:26.176 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:26:26.196 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:26:26.197 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:26.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:26.198 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:26.215 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:26:26.236 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:26:26.255 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:26:26.275 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:26:26.293 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:26:26.294 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:26:26.295 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:26:26.295 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:26:26.295 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:26:26.295 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:26:26.296 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:26:26.296 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:26:26.296 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:26:26.299 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:26.302 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:26:26.303 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:26:26.303 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:26:26.303 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:26:26.303 INFO:tasks.rados.rados.0.smithi076.stdout:1756: finishing write tid 1 to smithi07694878-17 2021-03-15T22:26:26.304 INFO:tasks.rados.rados.0.smithi076.stdout:1756: finishing write tid 2 to smithi07694878-17 2021-03-15T22:26:26.304 INFO:tasks.rados.rados.0.smithi076.stdout:1756: finishing write tid 3 to smithi07694878-17 2021-03-15T22:26:26.304 INFO:tasks.rados.rados.0.smithi076.stdout:1756: finishing write tid 4 to smithi07694878-17 2021-03-15T22:26:26.304 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 835 (ObjNum 618 snap 192 seq_num 618) dirty exists 2021-03-15T22:26:26.304 INFO:tasks.rados.rados.0.smithi076.stdout:1756: left oid 17 (ObjNum 618 snap 192 seq_num 618) 2021-03-15T22:26:26.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:26.324 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:26.324 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:26.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:26.451 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:26.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:26.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:26.461 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:26.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[56514]: cluster 2021-03-15T22:26:25.068246+0000 mgr.y (mgr.14144) 663 : cluster [DBG] pgmap v831: 138 pgs: 138 active+clean; 382 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2021-03-15T22:26:26.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[56514]: cluster 2021-03-15T22:26:26.126388+0000 mon.a (mon.0) 1327 : cluster [DBG] osdmap e471: 8 total, 8 up, 7 in 2021-03-15T22:26:26.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[56514]: audit 2021-03-15T22:26:26.130169+0000 mon.c (mon.1) 41 : audit [DBG] from='client.? 172.21.15.76:0/3253427776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:26.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[62156]: cluster 2021-03-15T22:26:25.068246 2021-03-15T22:26:26.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 663 : cluster [DBG] pgmap v831: 138 pgs: 138 active+clean; 382 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2021-03-15T22:26:26.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[62156]: cluster 2021-03-15T22:26:26.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[62156]: 2021-03-15T22:26:26.126388+0000 mon.a 2021-03-15T22:26:26.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[62156]: (mon.0) 1327 : cluster 2021-03-15T22:26:26.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[62156]: [DBG] osdmap e471: 8 total, 8 up, 7 in 2021-03-15T22:26:26.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[62156]: audit 2021-03-15T22:26:26.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[62156]: 2021-03-15T22:26 2021-03-15T22:26:26.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[62156]: :26.130169+0000 mon.c (mon 2021-03-15T22:26:26.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:26 smithi076 conmon[62156]: .1) 41 : audit [DBG] from='client.? 172.21.15.76:0/3253427776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:26 smithi125 conmon[56739]: cluster 2021-03-15T22:26:25.068246+0000 mgr.y (mgr.14144) 663 2021-03-15T22:26:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:26 smithi125 conmon[56739]: : cluster [DBG] pgmap v831: 138 pgs: 138 active+clean; 382 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 4.7 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2021-03-15T22:26:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:26 smithi125 conmon[56739]: cluster 2021-03-15T22:26:26.126388+0000 mon.a (mon.0) 1327 : cluster [DBG] 2021-03-15T22:26:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:26 smithi125 conmon[56739]: osdmap e471: 8 total, 8 up, 7 in 2021-03-15T22:26:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:26 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:26:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:26 smithi125 conmon[56739]: 22:26:26.130169+0000 mon.c (mon.1) 41 : audit 2021-03-15T22:26:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:26 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/3253427776' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:26.554 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:26.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:26.577 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:26.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:26.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:26.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:26.702 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:26.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:26.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:26.830 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:26.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:26.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:26.932 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:26.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:26.959 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:26.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:27.061 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:27.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:27.084 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:27.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:27.134 INFO:tasks.rados.rados.0.smithi076.stdout:1755: done (3 left) 2021-03-15T22:26:27.134 INFO:tasks.rados.rados.0.smithi076.stdout:1756: done (2 left) 2021-03-15T22:26:27.134 INFO:tasks.rados.rados.0.smithi076.stdout:1757: done (1 left) 2021-03-15T22:26:27.135 INFO:tasks.rados.rados.0.smithi076.stdout:1758: done (0 left) 2021-03-15T22:26:27.135 INFO:tasks.rados.rados.0.smithi076.stdout:1759: read oid 30 snap -1 2021-03-15T22:26:27.135 INFO:tasks.rados.rados.0.smithi076.stdout:1759: expect (ObjNum 600 snap 184 seq_num 600) 2021-03-15T22:26:27.135 INFO:tasks.rados.rados.0.smithi076.stdout:1760: rollback oid 12 current snap is 193 2021-03-15T22:26:27.135 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 12 to 188 2021-03-15T22:26:27.136 INFO:tasks.rados.rados.0.smithi076.stdout:1761: read oid 45 snap -1 2021-03-15T22:26:27.136 INFO:tasks.rados.rados.0.smithi076.stdout:1761: expect (ObjNum 609 snap 189 seq_num 609) 2021-03-15T22:26:27.136 INFO:tasks.rados.rados.0.smithi076.stdout:1762: delete oid 41 current snap is 193 2021-03-15T22:26:27.137 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:26:27.138 INFO:tasks.rados.rados.0.smithi076.stdout:1761: expect (ObjNum 609 snap 189 seq_num 609) 2021-03-15T22:26:27.186 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:27.212 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:27.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:27.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:27.245 INFO:tasks.rados.rados.0.smithi076.stdout:1761: done (3 left) 2021-03-15T22:26:27.245 INFO:tasks.rados.rados.0.smithi076.stdout:1762: done (2 left) 2021-03-15T22:26:27.246 INFO:tasks.rados.rados.0.smithi076.stdout:1763: write (excl) oid 28 current snap is 193 2021-03-15T22:26:27.246 INFO:tasks.rados.rados.0.smithi076.stdout:1759: expect (ObjNum 600 snap 184 seq_num 600) 2021-03-15T22:26:27.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:27.339 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:27.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:27.341 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:27.363 INFO:tasks.rados.rados.0.smithi076.stdout:1760: finishing rollback tid 1 to smithi07694878-12 2021-03-15T22:26:27.363 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 1050 (ObjNum 1953066355 snap 925919592 seq_num 942946614) dirty dne 2021-03-15T22:26:27.364 INFO:tasks.rados.rados.0.smithi076.stdout:1763: seq_num 620 ranges {420340=478973,1597777=512212} 2021-03-15T22:26:27.367 INFO:tasks.rados.rados.0.smithi076.stdout:1763: writing smithi07694878-28 from 420340 to 899313 tid 1 2021-03-15T22:26:27.374 INFO:tasks.rados.rados.0.smithi076.stdout:1763: writing smithi07694878-28 from 1597777 to 2109989 tid 2 2021-03-15T22:26:27.375 INFO:tasks.rados.rados.0.smithi076.stdout:1759: done (2 left) 2021-03-15T22:26:27.375 INFO:tasks.rados.rados.0.smithi076.stdout:1760: done (1 left) 2021-03-15T22:26:27.375 INFO:tasks.rados.rados.0.smithi076.stdout:1764: snap_remove snap 181 2021-03-15T22:26:27.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:27.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:27.467 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:27.468 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:27.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[56514]: cluster 2021-03-15T22:26 2021-03-15T22:26:27.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[56514]: :26.403081+0000 osd.2 (osd.2) 77 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:27.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[56514]: cluster 2021-03-15T22:26:26.403921+0000 osd.2 (osd.2) 78 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:27.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[56514]: audit 2021-03-15T22:26:26.646319+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? 172.21.15.76:0/3608105408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:26:27.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[56514]: cluster 2021-03-15T22:26:27.123996+0000 mon.a (mon.0) 1329 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:27.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[56514]: audit 2021-03-15T22:26:27.130493+0000 mon.a (mon.0) 1330 : audit [INF] from='client.? 172.21.15.76:0/3608105408' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:26:27.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:26:27.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[56514]: 15T22:26:27.130550+0000 mon.a (mon.0) 1331 : cluster [DBG] osdmap e472: 8 total, 8 up, 7 in 2021-03-15T22:26:27.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: cluster 2021-03-15T22:26:26 2021-03-15T22:26:27.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: .403081+0000 osd.2 (osd.2) 77 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:27.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: cluster 2021-03-15T22:26 2021-03-15T22:26:27.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: :26.403921+0000 osd.2 (osd.2) 2021-03-15T22:26:27.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: 78 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:27.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:26:27.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: 22:26:26.646319+0000 2021-03-15T22:26:27.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: mon.a (mon.0) 1328 : 2021-03-15T22:26:27.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/3608105408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:26:27.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:26:27.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: -03-15T22:26 2021-03-15T22:26:27.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: :27.123996+0000 mon.a ( 2021-03-15T22:26:27.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: mon.0) 1329 : cluster 2021-03-15T22:26:27.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:27.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:26:27.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: 03-15T22:26: 2021-03-15T22:26:27.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: 27.130493+0000 mon.a ( 2021-03-15T22:26:27.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: mon.0) 1330 : audit 2021-03-15T22:26:27.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/3608105408' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:26:27.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:26:27.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: 03-15T22:26: 2021-03-15T22:26:27.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: 27.130550+0000 mon.a (mon 2021-03-15T22:26:27.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: .0) 1331 : cluster [DBG] 2021-03-15T22:26:27.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:27 smithi076 conmon[62156]: osdmap e472: 8 total, 8 up, 7 in 2021-03-15T22:26:27.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: cluster 2021-03-15T22:26:26.403081+0000 osd.2 (osd.2) 77 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: cluster 2021-03-15T22:26:26 2021-03-15T22:26:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: .403921+0000 osd.2 (osd.2) 78 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: audit 2021-03-15T22:26:26.646319+0000 mon.a (mon.0) 1328 : audit [INF] from='client.? 172.21.15.76:0/3608105408' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:26:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: cluster 2021-03-15T22:26:27.123996+0000 mon.a (mon. 2021-03-15T22:26:27.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: 0) 1329 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:27.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:26:27.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: T22:26:27.130493+0000 mon.a ( 2021-03-15T22:26:27.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: mon.0) 1330 : audit [INF] from='client.? 172.21.15.76:0/3608105408' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:26:27.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:27.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: -15T22:26:27.130550+0000 2021-03-15T22:26:27.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: mon.a (mon.0) 1331 : cluster [DBG] 2021-03-15T22:26:27.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:27 smithi125 conmon[56739]: osdmap e472: 8 total, 8 up, 7 in 2021-03-15T22:26:27.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:27.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:27.595 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:27.596 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:27.697 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:27.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:27.720 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:27.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:27.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:27.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:27.845 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:27.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:27.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:27.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:27.975 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:27.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:28.077 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:28.102 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:28.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:28.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:28.129 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:26:28.205 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:28.228 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:28.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:28.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:28.267 INFO:tasks.rados.rados.0.smithi076.stdout:1764: done (1 left) 2021-03-15T22:26:28.267 INFO:tasks.rados.rados.0.smithi076.stdout:1765: read oid 30 snap 162 2021-03-15T22:26:28.267 INFO:tasks.rados.rados.0.smithi076.stdout:1765: expect (ObjNum 445 snap 126 seq_num 445) 2021-03-15T22:26:28.268 INFO:tasks.rados.rados.0.smithi076.stdout:1766: read oid 3 snap -1 2021-03-15T22:26:28.268 INFO:tasks.rados.rados.0.smithi076.stdout:1766: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:26:28.268 INFO:tasks.rados.rados.0.smithi076.stdout:1767: snap_remove snap 187 2021-03-15T22:26:28.331 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:28.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:28.357 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:28.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:28.380 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[56514]: cluster 2021-03-15T22:26:28.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[56514]: 2021-03-15T22:26:28.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[56514]: 2021 2021-03-15T22:26:28.381 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[56514]: - 2021-03-15T22:26:28.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[56514]: 03 2021-03-15T22:26:28.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[56514]: - 2021-03-15T22:26:28.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[56514]: 15 2021-03-15T22:26:28.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[56514]: T 2021-03-15T22:26:28.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[56514]: 22:26:27.068720+0000 mgr.y (mgr 2021-03-15T22:26:28.383 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[56514]: .14144) 664 : cluster [DBG] pgmap v834: 138 pgs: 138 active+clean; 386 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2021-03-15T22:26:28.383 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[62156]: cluster 2021-03-15T22:26:27.068720+0000 mgr.y (mgr 2021-03-15T22:26:28.384 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:28 smithi076 conmon[62156]: .14144) 664 : cluster [DBG] pgmap v834: 138 pgs: 138 active+clean; 386 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2021-03-15T22:26:28.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:28.485 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:28.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:28.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:28.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: cluster 2021-03-15T22:26:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: 2021-03-15T22:26:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: 2021 2021-03-15T22:26:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: - 2021-03-15T22:26:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: 03 2021-03-15T22:26:28.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: - 2021-03-15T22:26:28.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: 15 2021-03-15T22:26:28.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: T 2021-03-15T22:26:28.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: 22 2021-03-15T22:26:28.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: : 2021-03-15T22:26:28.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: 26 2021-03-15T22:26:28.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: : 2021-03-15T22:26:28.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: 27.068720+0000 mgr.y (mgr.14144) 664 2021-03-15T22:26:28.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:28 smithi125 conmon[56739]: : cluster [DBG] pgmap v834: 138 pgs: 138 active+clean; 386 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 16 op/s 2021-03-15T22:26:28.588 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:28.615 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:28.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:28.617 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:28.718 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:28.744 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:28.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:28.745 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:28.753 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:28.846 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:28.872 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:28.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:28.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:28.975 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:28.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:29.000 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:29.000 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:29.101 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:29.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:29.128 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:29.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:29.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:29.257 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:29.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:29.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:29.263 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:26:29.286 INFO:tasks.rados.rados.0.smithi076.stdout:1767: done (3 left) 2021-03-15T22:26:29.286 INFO:tasks.rados.rados.0.smithi076.stdout:1768: rollback oid 6 current snap is 193 2021-03-15T22:26:29.286 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 162 2021-03-15T22:26:29.287 INFO:tasks.rados.rados.0.smithi076.stdout:1769: read oid 37 snap -1 2021-03-15T22:26:29.287 INFO:tasks.rados.rados.0.smithi076.stdout:1769: expect (ObjNum 536 snap 170 seq_num 536) 2021-03-15T22:26:29.287 INFO:tasks.rados.rados.0.smithi076.stdout:1770: rollback oid 11 current snap is 193 2021-03-15T22:26:29.287 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 11 to 139 2021-03-15T22:26:29.288 INFO:tasks.rados.rados.0.smithi076.stdout:1771: read oid 31 snap -1 2021-03-15T22:26:29.288 INFO:tasks.rados.rados.0.smithi076.stdout:1771: expect (ObjNum 581 snap 180 seq_num 581) 2021-03-15T22:26:29.288 INFO:tasks.rados.rados.0.smithi076.stdout:1772: read oid 32 snap 185 2021-03-15T22:26:29.289 INFO:tasks.rados.rados.0.smithi076.stdout:1772: expect (ObjNum 469 snap 137 seq_num 469) 2021-03-15T22:26:29.289 INFO:tasks.rados.rados.0.smithi076.stdout:1773: read oid 18 snap 171 2021-03-15T22:26:29.289 INFO:tasks.rados.rados.0.smithi076.stdout:1773: expect (ObjNum 533 snap 166 seq_num 533) 2021-03-15T22:26:29.289 INFO:tasks.rados.rados.0.smithi076.stdout:1774: copy_from oid 49 from oid 29 current snap is 193 2021-03-15T22:26:29.290 INFO:tasks.rados.rados.0.smithi076.stdout:1775: write oid 19 current snap is 193 2021-03-15T22:26:29.290 INFO:tasks.rados.rados.0.smithi076.stdout:1775: seq_num 621 ranges {437147=554222,1656259=655870} 2021-03-15T22:26:29.293 INFO:tasks.rados.rados.0.smithi076.stdout:1775: writing smithi07694878-19 from 437147 to 991369 tid 1 2021-03-15T22:26:29.301 INFO:tasks.rados.rados.0.smithi076.stdout:1775: writing smithi07694878-19 from 1656259 to 2312129 tid 2 2021-03-15T22:26:29.302 INFO:tasks.rados.rados.0.smithi076.stdout:1776: delete oid 42 current snap is 193 2021-03-15T22:26:29.302 INFO:tasks.rados.rados.0.smithi076.stdout:1763: finishing write tid 1 to smithi07694878-28 2021-03-15T22:26:29.302 INFO:tasks.rados.rados.0.smithi076.stdout:1763: finishing write tid 2 to smithi07694878-28 2021-03-15T22:26:29.303 INFO:tasks.rados.rados.0.smithi076.stdout:1763: finishing write tid 3 to smithi07694878-28 2021-03-15T22:26:29.303 INFO:tasks.rados.rados.0.smithi076.stdout:1763: finishing write tid 4 to smithi07694878-28 2021-03-15T22:26:29.303 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 1259 (ObjNum 620 snap 193 seq_num 620) dirty exists 2021-03-15T22:26:29.303 INFO:tasks.rados.rados.0.smithi076.stdout:1763: left oid 28 (ObjNum 620 snap 193 seq_num 620) 2021-03-15T22:26:29.303 INFO:tasks.rados.rados.0.smithi076.stdout:1766: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:26:29.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:29.384 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:29.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:29.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:29.423 INFO:tasks.rados.rados.0.smithi076.stdout:1765: expect (ObjNum 445 snap 126 seq_num 445) 2021-03-15T22:26:29.486 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:29.510 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:29.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:29.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:29 smithi125 conmon[56739]: cluster 2021-03-15T22:26:28.259950+0000 mon.a ( 2021-03-15T22:26:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:29 smithi125 conmon[56739]: mon.0) 1332 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2021-03-15T22:26:29.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:29.620 INFO:tasks.rados.rados.0.smithi076.stdout:1773: expect (ObjNum 533 snap 166 seq_num 533) 2021-03-15T22:26:29.635 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:29.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:29.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:29 smithi076 conmon[56514]: cluster 2021-03-15T22:26:28.259950+0000 2021-03-15T22:26:29.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:29 smithi076 conmon[56514]: mon.a (mon.0) 1332 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2021-03-15T22:26:29.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:29 smithi076 conmon[62156]: cluster 2021-03-15T22:26:29.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:29 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:26:29.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:29 smithi076 conmon[62156]: 15T22:26:28.259950+0000 mon.a (mon.0) 1332 : cluster [DBG] osdmap e473: 8 total, 8 up, 7 in 2021-03-15T22:26:29.738 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:29.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:29.763 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:29.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:29.830 INFO:tasks.rados.rados.0.smithi076.stdout:1768: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:26:29.830 INFO:tasks.rados.rados.0.smithi076.stdout:1770: finishing rollback tid 0 to smithi07694878-11 2021-03-15T22:26:29.830 INFO:tasks.rados.rados.0.smithi076.stdout:1770: finishing rollback tid 1 to smithi07694878-11 2021-03-15T22:26:29.831 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 435 (ObjNum 261808064 snap 21896 seq_num 264859424) dirty dne 2021-03-15T22:26:29.831 INFO:tasks.rados.rados.0.smithi076.stdout:1768: finishing rollback tid 2 to smithi07694878-6 2021-03-15T22:26:29.831 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 777 (ObjNum 486 snap 144 seq_num 486) dirty exists 2021-03-15T22:26:29.831 INFO:tasks.rados.rados.0.smithi076.stdout:1763: done (11 left) 2021-03-15T22:26:29.832 INFO:tasks.rados.rados.0.smithi076.stdout:1765: done (10 left) 2021-03-15T22:26:29.832 INFO:tasks.rados.rados.0.smithi076.stdout:1766: done (9 left) 2021-03-15T22:26:29.832 INFO:tasks.rados.rados.0.smithi076.stdout:1768: done (8 left) 2021-03-15T22:26:29.832 INFO:tasks.rados.rados.0.smithi076.stdout:1770: done (7 left) 2021-03-15T22:26:29.833 INFO:tasks.rados.rados.0.smithi076.stdout:1773: done (6 left) 2021-03-15T22:26:29.833 INFO:tasks.rados.rados.0.smithi076.stdout:1776: done (5 left) 2021-03-15T22:26:29.833 INFO:tasks.rados.rados.0.smithi076.stdout:1777: write oid 14 current snap is 193 2021-03-15T22:26:29.834 INFO:tasks.rados.rados.0.smithi076.stdout:1775: finishing write tid 1 to smithi07694878-19 2021-03-15T22:26:29.834 INFO:tasks.rados.rados.0.smithi076.stdout:1777: seq_num 623 ranges {470761=704720,1656870=659539} 2021-03-15T22:26:29.836 INFO:tasks.rados.rados.0.smithi076.stdout:1777: writing smithi07694878-14 from 470761 to 1175481 tid 1 2021-03-15T22:26:29.843 INFO:tasks.rados.rados.0.smithi076.stdout:1777: writing smithi07694878-14 from 1656870 to 2316409 tid 2 2021-03-15T22:26:29.843 INFO:tasks.rados.rados.0.smithi076.stdout:1778: snap_create 2021-03-15T22:26:29.843 INFO:tasks.rados.rados.0.smithi076.stdout:1775: finishing write tid 2 to smithi07694878-19 2021-03-15T22:26:29.843 INFO:tasks.rados.rados.0.smithi076.stdout:1775: finishing write tid 3 to smithi07694878-19 2021-03-15T22:26:29.843 INFO:tasks.rados.rados.0.smithi076.stdout:1775: finishing write tid 4 to smithi07694878-19 2021-03-15T22:26:29.844 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 990 (ObjNum 621 snap 193 seq_num 621) dirty exists 2021-03-15T22:26:29.844 INFO:tasks.rados.rados.0.smithi076.stdout:1775: left oid 19 (ObjNum 621 snap 193 seq_num 621) 2021-03-15T22:26:29.844 INFO:tasks.rados.rados.0.smithi076.stdout:1769: expect (ObjNum 536 snap 170 seq_num 536) 2021-03-15T22:26:29.866 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:29.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:29.893 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:29.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:29.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:29.988 INFO:tasks.rados.rados.0.smithi076.stdout:1771: expect (ObjNum 581 snap 180 seq_num 581) 2021-03-15T22:26:29.996 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:30.018 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:30.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:30.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:30.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:26:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:26:30.102 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:26:30.105 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:26:30.121 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:30.147 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:30.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:30.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:30.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:30.254 INFO:tasks.rados.rados.0.smithi076.stdout:1774: finishing copy_from to smithi07694878-49 2021-03-15T22:26:30.255 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 960 (ObjNum 543 snap 172 seq_num 543) dirty exists 2021-03-15T22:26:30.255 INFO:tasks.rados.rados.0.smithi076.stdout:1774: finishing copy_from racing read to smithi07694878-49 2021-03-15T22:26:30.255 INFO:tasks.rados.rados.0.smithi076.stdout:1769: done (6 left) 2021-03-15T22:26:30.255 INFO:tasks.rados.rados.0.smithi076.stdout:1771: done (5 left) 2021-03-15T22:26:30.255 INFO:tasks.rados.rados.0.smithi076.stdout:1774: done (4 left) 2021-03-15T22:26:30.256 INFO:tasks.rados.rados.0.smithi076.stdout:1775: done (3 left) 2021-03-15T22:26:30.256 INFO:tasks.rados.rados.0.smithi076.stdout:1778: done (2 left) 2021-03-15T22:26:30.256 INFO:tasks.rados.rados.0.smithi076.stdout:1779: snap_create 2021-03-15T22:26:30.256 INFO:tasks.rados.rados.0.smithi076.stdout:1772: expect (ObjNum 469 snap 137 seq_num 469) 2021-03-15T22:26:30.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:30.277 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:30.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:30.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:30 smithi076 conmon[56514]: cluster 2021-03-15T22:26:29.069179+0000 mgr.y (mgr.14144) 665 : cluster [DBG] pgmap v837: 138 pgs: 138 active+clean; 387 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2021-03-15T22:26:30.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:30 smithi076 conmon[56514]: cluster 2021-03-15T22:26:29.280046+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2021-03-15T22:26:30.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:30 smithi076 conmon[56514]: audit 2021-03-15T22:26:29.774283+0000 mgr.y (mgr.14144) 666 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:30.327 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:30 smithi076 conmon[56514]: cluster 2021-03-15T22:26:30.124003+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2021-03-15T22:26:30.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:30 smithi076 conmon[62156]: cluster 2021-03-15T22:26:29. 2021-03-15T22:26:30.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:30 smithi076 conmon[62156]: 069179+0000 mgr.y (mgr.14144) 665 : cluster [DBG] pgmap v837: 138 pgs: 138 active+clean; 387 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2021-03-15T22:26:30.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:30 smithi076 conmon[62156]: cluster 2021-03-15T22:26:29.280046+0000 mon.a (mon.0) 1333 : cluster [DBG] osdmap e474: 8 total, 8 up, 7 in 2021-03-15T22:26:30.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:30 smithi076 conmon[62156]: audit 2021-03-15T22:26:29.774283+0000 mgr.y (mgr.14144) 666 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:30.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:30 smithi076 conmon[62156]: cluster 2021-03-15T22:26:30.124003+0000 mon.a (mon.0) 1334 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2021-03-15T22:26:30.378 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:30.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:30.402 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:30.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:30.435 INFO:tasks.rados.rados.0.smithi076.stdout:1777: finishing write tid 1 to smithi07694878-14 2021-03-15T22:26:30.439 INFO:tasks.rados.rados.0.smithi076.stdout:1777: finishing write tid 2 to smithi07694878-14 2021-03-15T22:26:30.439 INFO:tasks.rados.rados.0.smithi076.stdout:1777: finishing write tid 3 to smithi07694878-14 2021-03-15T22:26:30.439 INFO:tasks.rados.rados.0.smithi076.stdout:1777: finishing write tid 4 to smithi07694878-14 2021-03-15T22:26:30.440 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 789 (ObjNum 623 snap 193 seq_num 623) dirty exists 2021-03-15T22:26:30.440 INFO:tasks.rados.rados.0.smithi076.stdout:1777: left oid 14 (ObjNum 623 snap 193 seq_num 623) 2021-03-15T22:26:30.504 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:30.530 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:30.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:30.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:30 smithi125 conmon[56739]: cluster 2021-03-15T22:26:29.069179+0000 mgr.y (mgr.14144) 665 2021-03-15T22:26:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:30 smithi125 conmon[56739]: : cluster [DBG] pgmap v837: 138 pgs: 138 active+clean; 387 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.0 MiB/s wr, 13 op/s 2021-03-15T22:26:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:30 smithi125 conmon[56739]: cluster 2021-03-15T22:26:29.280046+0000 mon.a (mon.0) 1333 : cluster [DBG] 2021-03-15T22:26:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:30 smithi125 conmon[56739]: osdmap e474: 8 total, 8 up, 7 in 2021-03-15T22:26:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:30 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:26:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:30 smithi125 conmon[56739]: 03-15T22:26:29.774283+0000 mgr.y ( 2021-03-15T22:26:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:30 smithi125 conmon[56739]: mgr.14144) 666 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:30 smithi125 conmon[56739]: cluster 2021-03-15T22:26:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:30 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:26:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:30 smithi125 conmon[56739]: 26:30.124003+0000 mon.a (mon.0) 2021-03-15T22:26:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:30 smithi125 conmon[56739]: 1334 : cluster [DBG] osdmap e475: 8 total, 8 up, 7 in 2021-03-15T22:26:30.543 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:26:30.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:30.658 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:30.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:30.660 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:30.761 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:30.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:30.787 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:30.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:30.889 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:30.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:30.912 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:30.912 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:31.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:31.040 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:31.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:31.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:31.134 INFO:tasks.rados.rados.0.smithi076.stdout:1772: done (2 left) 2021-03-15T22:26:31.134 INFO:tasks.rados.rados.0.smithi076.stdout:1777: done (1 left) 2021-03-15T22:26:31.134 INFO:tasks.rados.rados.0.smithi076.stdout:1779: done (0 left) 2021-03-15T22:26:31.135 INFO:tasks.rados.rados.0.smithi076.stdout:1780: snap_create 2021-03-15T22:26:31.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:31.170 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:31.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:31.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:31.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:31.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:31.299 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:31.300 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:31.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:31.427 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:31.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:31.429 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:31.530 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:31.556 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:31.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:31.558 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:31.609 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:31 smithi076 conmon[56514]: audit 2021-03-15T22:26:30.085922+0000 mgr.y (mgr.14144) 667 : audit [DBG] from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:26:31.609 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:31 smithi076 conmon[56514]: cluster 2021-03-15T22:26:31.131836+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2021-03-15T22:26:31.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:31 smithi076 conmon[62156]: audit 2021-03-15T22:26:31.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:31 smithi076 conmon[62156]: 2021- 2021-03-15T22:26:31.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:31 smithi076 conmon[62156]: 03-15T22:26:30.085922+0000 mgr.y (mgr.14144) 667 : audit [DBG] from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:26:31.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:31 smithi076 conmon[62156]: 2021-03-15T22:26:31.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:31 smithi076 conmon[62156]: cluster 2021-03-15T22:26:31.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:31 smithi076 conmon[62156]: 2021-03-15T22:26:31.131836+0000 2021-03-15T22:26:31.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:31 smithi076 conmon[62156]: mon.a (mon.0) 1335 : cluster [DBG] 2021-03-15T22:26:31.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:31 smithi076 conmon[62156]: osdmap e476: 8 total, 8 up, 7 in 2021-03-15T22:26:31.659 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:31.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:31.682 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:31.682 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:31.784 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:31.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:31 smithi125 conmon[56739]: audit 2021-03-15T22:26:30.085922+0000 mgr.y (mgr.14144) 667 : audit [DBG] from='client.15615 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:26:31.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:31 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:26:31.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:31 smithi125 conmon[56739]: 26:31.131836+0000 mon.a (mon.0) 1335 : cluster [DBG] osdmap e476: 8 total, 8 up, 7 in 2021-03-15T22:26:31.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:31.807 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:31.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:31.909 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:31.936 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:31.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:31.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:32.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:32.064 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:32.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:32.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:32.113 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":838,"stamp":"2021-03-15T22:26:29.277774+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":405547673,"num_objects":498,"num_object_clones":134,"num_object_copies":1494,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":498,"num_whiteouts":12,"num_read":4964,"num_read_kb":3438032,"num_write":3064,"num_write_kb":948092,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1680,"num_bytes_recovered":514354880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8489,"ondisk_log_size":8489,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1059636,"kb_used_data":895228,"kb_used_omap":0,"kb_used_meta":164352,"kb_avail":655156428,"statfs":{"total":671965249536,"available":670880182272,"internally_reserved":0,"allocated":916713472,"data_stored":1223030908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":168296448},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5440222,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":-2,"num_read":51,"num_read_kb":55845,"num_write":25,"num_write_kb":8110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.161381"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"435","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963315+0000","last_change":"2021-03-15T22:26:14.026132+0000","last_active":"2021-03-15T22:26:25.963315+0000","last_peered":"2021-03-15T22:26:25.963315+0000","last_clean":"2021-03-15T22:26:25.963315+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:26:25.963315+0000","last_undegraded":"2021-03-15T22:26:25.963315+0000","last_fullsized":"2021-03-15T22:26:25.963315+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:33.728929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:33.728929+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"143","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.133683+0000","last_change":"2021-03-15T22:26:07.787587+0000","last_active":"2021-03-15T22:26:26.133683+0000","last_peered":"2021-03-15T22:26:26.133683+0000","last_clean":"2021-03-15T22:26:26.133683+0000","last_became_active":"2021-03-15T22:24:31.103578+0000","last_became_peered":"2021-03-15T22:24:31.103578+0000","last_unstale":"2021-03-15T22:26:26.133683+0000","last_undegraded":"2021-03-15T22:26:26.133683+0000","last_fullsized":"2021-03-15T22:26:26.133683+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:26:07.787524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.787524+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"470","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.962870+0000","last_change":"2021-03-15T22:26:11.025622+0000","last_active":"2021-03-15T22:26:25.962870+0000","last_peered":"2021-03-15T22:26:25.962870+0000","last_clean":"2021-03-15T22:26:25.962870+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:26:25.962870+0000","last_undegraded":"2021-03-15T22:26:25.962870+0000","last_fullsized":"2021-03-15T22:26:25.962870+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:26:11.025548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:11.025548+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"63","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727412+0000","last_change":"2021-03-15T22:25:33.149020+0000","last_active":"2021-03-15T22:26:25.727412+0000","last_peered":"2021-03-15T22:26:25.727412+0000","last_clean":"2021-03-15T22:26:25.727412+0000","last_became_active":"2021-03-15T22:25:33.148710+0000","last_became_peered":"2021-03-15T22:25:33.148710+0000","last_unstale":"2021-03-15T22:26:25.727412+0000","last_undegraded":"2021-03-15T22:26:25.727412+0000","last_fullsized":"2021-03-15T22:26:25.727412+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:06.177754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.177754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"63","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727462+0000","last_change":"2021-03-15T22:25:33.147017+0000","last_active":"2021-03-15T22:26:25.727462+0000","last_peered":"2021-03-15T22:26:25.727462+0000","last_clean":"2021-03-15T22:26:25.727462+0000","last_became_active":"2021-03-15T22:25:33.146697+0000","last_became_peered":"2021-03-15T22:25:33.146697+0000","last_unstale":"2021-03-15T22:26:25.727462+0000","last_undegraded":"2021-03-15T22:26:25.727462+0000","last_fullsized":"2021-03-15T22:26:25.727462+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:15.333746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.333746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"89","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.726823+0000","last_change":"2021-03-15T22:25:33.175542+0000","last_active":"2021-03-15T22:26:25.726823+0000","last_peered":"2021-03-15T22:26:25.726823+0000","last_clean":"2021-03-15T22:26:25.726823+0000","last_became_active":"2021-03-15T22:25:33.137906+0000","last_became_peered":"2021-03-15T22:25:33.137906+0000","last_unstale":"2021-03-15T22:26:25.726823+0000","last_undegraded":"2021-03-15T22:26:25.726823+0000","last_fullsized":"2021-03-15T22:26:25.726823+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:23:42.144821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.144821+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":21,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"132","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.125257+0000","last_change":"2021-03-15T22:26:12.608148+0000","last_active":"2021-03-15T22:26:26.125257+0000","last_peered":"2021-03-15T22:26:26.125257+0000","last_clean":"2021-03-15T22:26:26.125257+0000","last_became_active":"2021-03-15T22:24:31.108652+0000","last_became_peered":"2021-03-15T22:24:31.108652+0000","last_unstale":"2021-03-15T22:26:26.125257+0000","last_undegraded":"2021-03-15T22:26:26.125257+0000","last_fullsized":"2021-03-15T22:26:26.125257+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:26:12.608082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"437","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.132910+0000","last_change":"2021-03-15T22:25:16.598073+0000","last_active":"2021-03-15T22:26:26.132910+0000","last_peered":"2021-03-15T22:26:26.132910+0000","last_clean":"2021-03-15T22:26:26.132910+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:26:26.132910+0000","last_undegraded":"2021-03-15T22:26:26.132910+0000","last_fullsized":"2021-03-15T22:26:26.132910+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.597998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.597998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"540","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963558+0000","last_change":"2021-03-15T22:24:31.776304+0000","last_active":"2021-03-15T22:26:25.963558+0000","last_peered":"2021-03-15T22:26:25.963558+0000","last_clean":"2021-03-15T22:26:25.963558+0000","last_became_active":"2021-03-15T22:24:31.083373+0000","last_became_peered":"2021-03-15T22:24:31.083373+0000","last_unstale":"2021-03-15T22:26:25.963558+0000","last_undegraded":"2021-03-15T22:26:25.963558+0000","last_fullsized":"2021-03-15T22:26:25.963558+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:24:10.979322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:10.979322+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"146","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.132149+0000","last_change":"2021-03-15T22:25:33.143128+0000","last_active":"2021-03-15T22:26:26.132149+0000","last_peered":"2021-03-15T22:26:26.132149+0000","last_clean":"2021-03-15T22:26:26.132149+0000","last_became_active":"2021-03-15T22:25:33.142493+0000","last_became_peered":"2021-03-15T22:25:33.142493+0000","last_unstale":"2021-03-15T22:26:26.132149+0000","last_undegraded":"2021-03-15T22:26:26.132149+0000","last_fullsized":"2021-03-15T22:26:26.132149+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.674895+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"437","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.132649+0000","last_change":"2021-03-15T22:25:05.682093+0000","last_active":"2021-03-15T22:26:26.132649+0000","last_peered":"2021-03-15T22:26:26.132649+0000","last_clean":"2021-03-15T22:26:26.132649+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:26:26.132649+0000","last_undegraded":"2021-03-15T22:26:26.132649+0000","last_fullsized":"2021-03-15T22:26:26.132649+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.682031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.682031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"121'9","reported_seq":"506","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963351+0000","last_change":"2021-03-15T22:24:37.902248+0000","last_active":"2021-03-15T22:26:25.963351+0000","last_peered":"2021-03-15T22:26:25.963351+0000","last_clean":"2021-03-15T22:26:25.963351+0000","last_became_active":"2021-03-15T22:22:56.055083+0000","last_became_peered":"2021-03-15T22:22:56.055083+0000","last_unstale":"2021-03-15T22:26:25.963351+0000","last_undegraded":"2021-03-15T22:26:25.963351+0000","last_fullsized":"2021-03-15T22:26:25.963351+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'9","last_scrub_stamp":"2021-03-15T22:24:37.902166+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.902166+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"433","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963495+0000","last_change":"2021-03-15T22:24:30.865676+0000","last_active":"2021-03-15T22:26:25.963495+0000","last_peered":"2021-03-15T22:26:25.963495+0000","last_clean":"2021-03-15T22:26:25.963495+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:26:25.963495+0000","last_undegraded":"2021-03-15T22:26:25.963495+0000","last_fullsized":"2021-03-15T22:26:25.963495+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.865609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.865609+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"181","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963254+0000","last_change":"2021-03-15T22:24:55.103109+0000","last_active":"2021-03-15T22:26:25.963254+0000","last_peered":"2021-03-15T22:26:25.963254+0000","last_clean":"2021-03-15T22:26:25.963254+0000","last_became_active":"2021-03-15T22:24:55.090536+0000","last_became_peered":"2021-03-15T22:24:55.090536+0000","last_unstale":"2021-03-15T22:26:25.963254+0000","last_undegraded":"2021-03-15T22:26:25.963254+0000","last_fullsized":"2021-03-15T22:26:25.963254+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:24:28.507563+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.507563+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"540","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.169974+0000","last_change":"2021-03-15T22:26:09.660649+0000","last_active":"2021-03-15T22:26:23.169974+0000","last_peered":"2021-03-15T22:26:23.169974+0000","last_clean":"2021-03-15T22:26:23.169974+0000","last_became_active":"2021-03-15T22:25:33.134238+0000","last_became_peered":"2021-03-15T22:25:33.134238+0000","last_unstale":"2021-03-15T22:26:23.169974+0000","last_undegraded":"2021-03-15T22:26:23.169974+0000","last_fullsized":"2021-03-15T22:26:23.169974+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:26:09.660549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.660549+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"261","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727767+0000","last_change":"2021-03-15T22:25:14.117883+0000","last_active":"2021-03-15T22:26:25.727767+0000","last_peered":"2021-03-15T22:26:25.727767+0000","last_clean":"2021-03-15T22:26:25.727767+0000","last_became_active":"2021-03-15T22:25:14.117558+0000","last_became_peered":"2021-03-15T22:25:14.117558+0000","last_unstale":"2021-03-15T22:26:25.727767+0000","last_undegraded":"2021-03-15T22:26:25.727767+0000","last_fullsized":"2021-03-15T22:26:25.727767+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:00.480944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:00.480944+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"442","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963642+0000","last_change":"2021-03-15T22:25:11.028502+0000","last_active":"2021-03-15T22:26:25.963642+0000","last_peered":"2021-03-15T22:26:25.963642+0000","last_clean":"2021-03-15T22:26:25.963642+0000","last_became_active":"2021-03-15T22:17:54.283826+0000","last_became_peered":"2021-03-15T22:17:54.283826+0000","last_unstale":"2021-03-15T22:26:25.963642+0000","last_undegraded":"2021-03-15T22:26:25.963642+0000","last_fullsized":"2021-03-15T22:26:25.963642+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.028422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.028422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"83","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727492+0000","last_change":"2021-03-15T22:25:33.169024+0000","last_active":"2021-03-15T22:26:25.727492+0000","last_peered":"2021-03-15T22:26:25.727492+0000","last_clean":"2021-03-15T22:26:25.727492+0000","last_became_active":"2021-03-15T22:25:33.134781+0000","last_became_peered":"2021-03-15T22:25:33.134781+0000","last_unstale":"2021-03-15T22:26:25.727492+0000","last_undegraded":"2021-03-15T22:26:25.727492+0000","last_fullsized":"2021-03-15T22:26:25.727492+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:23:32.049047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:32.049047+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":36,"num_read_kb":19,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"433","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.135002+0000","last_change":"2021-03-15T22:26:15.994881+0000","last_active":"2021-03-15T22:26:27.135002+0000","last_peered":"2021-03-15T22:26:27.135002+0000","last_clean":"2021-03-15T22:26:27.135002+0000","last_became_active":"2021-03-15T22:23:52.271925+0000","last_became_peered":"2021-03-15T22:23:52.271925+0000","last_unstale":"2021-03-15T22:26:27.135002+0000","last_undegraded":"2021-03-15T22:26:27.135002+0000","last_fullsized":"2021-03-15T22:26:27.135002+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:15.994794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:15.994794+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"463","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.125296+0000","last_change":"2021-03-15T22:26:07.656468+0000","last_active":"2021-03-15T22:26:26.125296+0000","last_peered":"2021-03-15T22:26:26.125296+0000","last_clean":"2021-03-15T22:26:26.125296+0000","last_became_active":"2021-03-15T22:22:56.062833+0000","last_became_peered":"2021-03-15T22:22:56.062833+0000","last_unstale":"2021-03-15T22:26:26.125296+0000","last_undegraded":"2021-03-15T22:26:26.125296+0000","last_fullsized":"2021-03-15T22:26:26.125296+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:07.656378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.656378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"255","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.135543+0000","last_change":"2021-03-15T22:25:33.752847+0000","last_active":"2021-03-15T22:26:27.135543+0000","last_peered":"2021-03-15T22:26:27.135543+0000","last_clean":"2021-03-15T22:26:27.135543+0000","last_became_active":"2021-03-15T22:25:33.140078+0000","last_became_peered":"2021-03-15T22:25:33.140078+0000","last_unstale":"2021-03-15T22:26:27.135543+0000","last_undegraded":"2021-03-15T22:26:27.135543+0000","last_fullsized":"2021-03-15T22:26:27.135543+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:25:08.192165+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:25:08.192165+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"439","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.127807+0000","last_change":"2021-03-15T22:26:02.720851+0000","last_active":"2021-03-15T22:26:26.127807+0000","last_peered":"2021-03-15T22:26:26.127807+0000","last_clean":"2021-03-15T22:26:26.127807+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:26:26.127807+0000","last_undegraded":"2021-03-15T22:26:26.127807+0000","last_fullsized":"2021-03-15T22:26:26.127807+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:02.720791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:02.720791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"63","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.726605+0000","last_change":"2021-03-15T22:25:33.133625+0000","last_active":"2021-03-15T22:26:25.726605+0000","last_peered":"2021-03-15T22:26:25.726605+0000","last_clean":"2021-03-15T22:26:25.726605+0000","last_became_active":"2021-03-15T22:25:33.133418+0000","last_became_peered":"2021-03-15T22:25:33.133418+0000","last_unstale":"2021-03-15T22:26:25.726605+0000","last_undegraded":"2021-03-15T22:26:25.726605+0000","last_fullsized":"2021-03-15T22:26:25.726605+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:29.506406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.506406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"470","reported_epoch":"469","state":"active+clean","last_fresh":"2021-03-15T22:26:24.221386+0000","last_change":"2021-03-15T22:26:09.832358+0000","last_active":"2021-03-15T22:26:24.221386+0000","last_peered":"2021-03-15T22:26:24.221386+0000","last_clean":"2021-03-15T22:26:24.221386+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:26:24.221386+0000","last_undegraded":"2021-03-15T22:26:24.221386+0000","last_fullsized":"2021-03-15T22:26:24.221386+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:26:09.832278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.832278+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"433","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.134507+0000","last_change":"2021-03-15T22:25:09.177600+0000","last_active":"2021-03-15T22:26:27.134507+0000","last_peered":"2021-03-15T22:26:27.134507+0000","last_clean":"2021-03-15T22:26:27.134507+0000","last_became_active":"2021-03-15T22:23:52.260757+0000","last_became_peered":"2021-03-15T22:23:52.260757+0000","last_unstale":"2021-03-15T22:26:27.134507+0000","last_undegraded":"2021-03-15T22:26:27.134507+0000","last_fullsized":"2021-03-15T22:26:27.134507+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:09.177521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.177521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"439","reported_epoch":"469","state":"active+clean","last_fresh":"2021-03-15T22:26:24.220878+0000","last_change":"2021-03-15T22:25:02.452304+0000","last_active":"2021-03-15T22:26:24.220878+0000","last_peered":"2021-03-15T22:26:24.220878+0000","last_clean":"2021-03-15T22:26:24.220878+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:26:24.220878+0000","last_undegraded":"2021-03-15T22:26:24.220878+0000","last_fullsized":"2021-03-15T22:26:24.220878+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.452236+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.452236+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"274","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.133802+0000","last_change":"2021-03-15T22:26:06.764056+0000","last_active":"2021-03-15T22:26:26.133802+0000","last_peered":"2021-03-15T22:26:26.133802+0000","last_clean":"2021-03-15T22:26:26.133802+0000","last_became_active":"2021-03-15T22:25:33.146680+0000","last_became_peered":"2021-03-15T22:25:33.146680+0000","last_unstale":"2021-03-15T22:26:26.133802+0000","last_undegraded":"2021-03-15T22:26:26.133802+0000","last_fullsized":"2021-03-15T22:26:26.133802+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:06.763986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:26:06.763986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"160","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.726899+0000","last_change":"2021-03-15T22:25:15.835780+0000","last_active":"2021-03-15T22:26:25.726899+0000","last_peered":"2021-03-15T22:26:25.726899+0000","last_clean":"2021-03-15T22:26:25.726899+0000","last_became_active":"2021-03-15T22:24:23.142746+0000","last_became_peered":"2021-03-15T22:24:23.142746+0000","last_unstale":"2021-03-15T22:26:25.726899+0000","last_undegraded":"2021-03-15T22:26:25.726899+0000","last_fullsized":"2021-03-15T22:26:25.726899+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:25:15.835675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.835675+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":11,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"63","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727568+0000","last_change":"2021-03-15T22:25:33.148415+0000","last_active":"2021-03-15T22:26:25.727568+0000","last_peered":"2021-03-15T22:26:25.727568+0000","last_clean":"2021-03-15T22:26:25.727568+0000","last_became_active":"2021-03-15T22:25:33.148083+0000","last_became_peered":"2021-03-15T22:25:33.148083+0000","last_unstale":"2021-03-15T22:26:25.727568+0000","last_undegraded":"2021-03-15T22:26:25.727568+0000","last_fullsized":"2021-03-15T22:26:25.727568+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:42.525975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:40.670674+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.525975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"430","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.134882+0000","last_change":"2021-03-15T22:26:12.013804+0000","last_active":"2021-03-15T22:26:27.134882+0000","last_peered":"2021-03-15T22:26:27.134882+0000","last_clean":"2021-03-15T22:26:27.134882+0000","last_became_active":"2021-03-15T22:23:52.262741+0000","last_became_peered":"2021-03-15T22:23:52.262741+0000","last_unstale":"2021-03-15T22:26:27.134882+0000","last_undegraded":"2021-03-15T22:26:27.134882+0000","last_fullsized":"2021-03-15T22:26:27.134882+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:12.013718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:26:12.013718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"437","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.134075+0000","last_change":"2021-03-15T22:26:02.840036+0000","last_active":"2021-03-15T22:26:26.134075+0000","last_peered":"2021-03-15T22:26:26.134075+0000","last_clean":"2021-03-15T22:26:26.134075+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:26:26.134075+0000","last_undegraded":"2021-03-15T22:26:26.134075+0000","last_fullsized":"2021-03-15T22:26:26.134075+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:02.839942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:02.839942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"90'22","reported_seq":"248","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.726975+0000","last_change":"2021-03-15T22:25:16.834493+0000","last_active":"2021-03-15T22:26:25.726975+0000","last_peered":"2021-03-15T22:26:25.726975+0000","last_clean":"2021-03-15T22:26:25.726975+0000","last_became_active":"2021-03-15T22:22:56.056175+0000","last_became_peered":"2021-03-15T22:22:56.056175+0000","last_unstale":"2021-03-15T22:26:25.726975+0000","last_undegraded":"2021-03-15T22:26:25.726975+0000","last_fullsized":"2021-03-15T22:26:25.726975+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:25:16.834409+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.834409+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"131","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.133332+0000","last_change":"2021-03-15T22:24:41.659524+0000","last_active":"2021-03-15T22:26:26.133332+0000","last_peered":"2021-03-15T22:26:26.133332+0000","last_clean":"2021-03-15T22:26:26.133332+0000","last_became_active":"2021-03-15T22:24:31.105380+0000","last_became_peered":"2021-03-15T22:24:31.105380+0000","last_unstale":"2021-03-15T22:26:26.133332+0000","last_undegraded":"2021-03-15T22:26:26.133332+0000","last_fullsized":"2021-03-15T22:26:26.133332+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:41.659450+0000","last_clean_scrub_stamp":"2021-03-15T22:24:41.659450+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"248","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727557+0000","last_change":"2021-03-15T22:25:14.115163+0000","last_active":"2021-03-15T22:26:25.727557+0000","last_peered":"2021-03-15T22:26:25.727557+0000","last_clean":"2021-03-15T22:26:25.727557+0000","last_became_active":"2021-03-15T22:25:14.114508+0000","last_became_peered":"2021-03-15T22:25:14.114508+0000","last_unstale":"2021-03-15T22:26:25.727557+0000","last_undegraded":"2021-03-15T22:26:25.727557+0000","last_fullsized":"2021-03-15T22:26:25.727557+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.007316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:04.613000+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.007316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"447","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.135203+0000","last_change":"2021-03-15T22:23:52.266674+0000","last_active":"2021-03-15T22:26:27.135203+0000","last_peered":"2021-03-15T22:26:27.135203+0000","last_clean":"2021-03-15T22:26:27.135203+0000","last_became_active":"2021-03-15T22:23:52.266011+0000","last_became_peered":"2021-03-15T22:23:52.266011+0000","last_unstale":"2021-03-15T22:26:27.135203+0000","last_undegraded":"2021-03-15T22:26:27.135203+0000","last_fullsized":"2021-03-15T22:26:27.135203+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.449472+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"123","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.127331+0000","last_change":"2021-03-15T22:24:56.824808+0000","last_active":"2021-03-15T22:26:26.127331+0000","last_peered":"2021-03-15T22:26:26.127331+0000","last_clean":"2021-03-15T22:26:26.127331+0000","last_became_active":"2021-03-15T22:24:55.097354+0000","last_became_peered":"2021-03-15T22:24:55.097354+0000","last_unstale":"2021-03-15T22:26:26.127331+0000","last_undegraded":"2021-03-15T22:26:26.127331+0000","last_fullsized":"2021-03-15T22:26:26.127331+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:24:38.433166+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:38.433166+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"68","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.726738+0000","last_change":"2021-03-15T22:26:10.680684+0000","last_active":"2021-03-15T22:26:25.726738+0000","last_peered":"2021-03-15T22:26:25.726738+0000","last_clean":"2021-03-15T22:26:25.726738+0000","last_became_active":"2021-03-15T22:25:33.138883+0000","last_became_peered":"2021-03-15T22:25:33.138883+0000","last_unstale":"2021-03-15T22:26:25.726738+0000","last_undegraded":"2021-03-15T22:26:25.726738+0000","last_fullsized":"2021-03-15T22:26:25.726738+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:26:10.680621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"454","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.127160+0000","last_change":"2021-03-15T22:25:33.129883+0000","last_active":"2021-03-15T22:26:26.127160+0000","last_peered":"2021-03-15T22:26:26.127160+0000","last_clean":"2021-03-15T22:26:26.127160+0000","last_became_active":"2021-03-15T22:25:33.129553+0000","last_became_peered":"2021-03-15T22:25:33.129553+0000","last_unstale":"2021-03-15T22:26:26.127160+0000","last_undegraded":"2021-03-15T22:26:26.127160+0000","last_fullsized":"2021-03-15T22:26:26.127160+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.794432+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"156","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.169830+0000","last_change":"2021-03-15T22:24:12.017160+0000","last_active":"2021-03-15T22:26:23.169830+0000","last_peered":"2021-03-15T22:26:23.169830+0000","last_clean":"2021-03-15T22:26:23.169830+0000","last_became_active":"2021-03-15T22:24:03.086278+0000","last_became_peered":"2021-03-15T22:24:03.086278+0000","last_unstale":"2021-03-15T22:26:23.169830+0000","last_undegraded":"2021-03-15T22:26:23.169830+0000","last_fullsized":"2021-03-15T22:26:23.169830+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:12.017097+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:12.017097+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"464","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963148+0000","last_change":"2021-03-15T22:25:43.037390+0000","last_active":"2021-03-15T22:26:25.963148+0000","last_peered":"2021-03-15T22:26:25.963148+0000","last_clean":"2021-03-15T22:26:25.963148+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:26:25.963148+0000","last_undegraded":"2021-03-15T22:26:25.963148+0000","last_fullsized":"2021-03-15T22:26:25.963148+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:29.709674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:29.709674+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"266","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.135281+0000","last_change":"2021-03-15T22:25:33.144478+0000","last_active":"2021-03-15T22:26:27.135281+0000","last_peered":"2021-03-15T22:26:27.135281+0000","last_clean":"2021-03-15T22:26:27.135281+0000","last_became_active":"2021-03-15T22:25:33.143832+0000","last_became_peered":"2021-03-15T22:25:33.143832+0000","last_unstale":"2021-03-15T22:26:27.135281+0000","last_undegraded":"2021-03-15T22:26:27.135281+0000","last_fullsized":"2021-03-15T22:26:27.135281+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.147414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.147414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"356'3","reported_seq":"545","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:27.094694+0000","last_change":"2021-03-15T22:24:31.129210+0000","last_active":"2021-03-15T22:26:27.094694+0000","last_peered":"2021-03-15T22:26:27.094694+0000","last_clean":"2021-03-15T22:26:27.094694+0000","last_became_active":"2021-03-15T22:24:31.093479+0000","last_became_peered":"2021-03-15T22:24:31.093479+0000","last_unstale":"2021-03-15T22:26:27.094694+0000","last_undegraded":"2021-03-15T22:26:27.094694+0000","last_fullsized":"2021-03-15T22:26:27.094694+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"61'2","last_scrub_stamp":"2021-03-15T22:24:28.700659+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.700659+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"434","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.134532+0000","last_change":"2021-03-15T22:25:01.148183+0000","last_active":"2021-03-15T22:26:27.134532+0000","last_peered":"2021-03-15T22:26:27.134532+0000","last_clean":"2021-03-15T22:26:27.134532+0000","last_became_active":"2021-03-15T22:23:52.261234+0000","last_became_peered":"2021-03-15T22:23:52.261234+0000","last_unstale":"2021-03-15T22:26:27.134532+0000","last_undegraded":"2021-03-15T22:26:27.134532+0000","last_fullsized":"2021-03-15T22:26:27.134532+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:01.148105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:01.148105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"247","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727338+0000","last_change":"2021-03-15T22:24:27.166757+0000","last_active":"2021-03-15T22:26:25.727338+0000","last_peered":"2021-03-15T22:26:25.727338+0000","last_clean":"2021-03-15T22:26:25.727338+0000","last_became_active":"2021-03-15T22:22:56.064081+0000","last_became_peered":"2021-03-15T22:22:56.064081+0000","last_unstale":"2021-03-15T22:26:25.727338+0000","last_undegraded":"2021-03-15T22:26:25.727338+0000","last_fullsized":"2021-03-15T22:26:25.727338+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:27.166678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.166678+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":13,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"431","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.965390+0000","last_change":"2021-03-15T22:25:07.992897+0000","last_active":"2021-03-15T22:26:25.965390+0000","last_peered":"2021-03-15T22:26:25.965390+0000","last_clean":"2021-03-15T22:26:25.965390+0000","last_became_active":"2021-03-15T22:18:00.014992+0000","last_became_peered":"2021-03-15T22:18:00.014992+0000","last_unstale":"2021-03-15T22:26:25.965390+0000","last_undegraded":"2021-03-15T22:26:25.965390+0000","last_fullsized":"2021-03-15T22:26:25.965390+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:07.992815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.992815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"421'7","reported_seq":"77","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727248+0000","last_change":"2021-03-15T22:25:33.172230+0000","last_active":"2021-03-15T22:26:25.727248+0000","last_peered":"2021-03-15T22:26:25.727248+0000","last_clean":"2021-03-15T22:26:25.727248+0000","last_became_active":"2021-03-15T22:25:33.147484+0000","last_became_peered":"2021-03-15T22:25:33.147484+0000","last_unstale":"2021-03-15T22:26:25.727248+0000","last_undegraded":"2021-03-15T22:26:25.727248+0000","last_fullsized":"2021-03-15T22:26:25.727248+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'6","last_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.098511+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"277","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.134613+0000","last_change":"2021-03-15T22:26:17.012763+0000","last_active":"2021-03-15T22:26:27.134613+0000","last_peered":"2021-03-15T22:26:27.134613+0000","last_clean":"2021-03-15T22:26:27.134613+0000","last_became_active":"2021-03-15T22:25:33.141466+0000","last_became_peered":"2021-03-15T22:25:33.141466+0000","last_unstale":"2021-03-15T22:26:27.134613+0000","last_undegraded":"2021-03-15T22:26:27.134613+0000","last_fullsized":"2021-03-15T22:26:27.134613+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:17.012652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:26:17.012652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"65'13","reported_seq":"101","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.132587+0000","last_change":"2021-03-15T22:25:14.129885+0000","last_active":"2021-03-15T22:26:26.132587+0000","last_peered":"2021-03-15T22:26:26.132587+0000","last_clean":"2021-03-15T22:26:26.132587+0000","last_became_active":"2021-03-15T22:25:14.114060+0000","last_became_peered":"2021-03-15T22:25:14.114060+0000","last_unstale":"2021-03-15T22:26:26.132587+0000","last_undegraded":"2021-03-15T22:26:26.132587+0000","last_fullsized":"2021-03-15T22:26:26.132587+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:25:07.163128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.163128+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"434","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.170220+0000","last_change":"2021-03-15T22:25:00.614351+0000","last_active":"2021-03-15T22:26:23.170220+0000","last_peered":"2021-03-15T22:26:23.170220+0000","last_clean":"2021-03-15T22:26:23.170220+0000","last_became_active":"2021-03-15T22:18:00.016954+0000","last_became_peered":"2021-03-15T22:18:00.016954+0000","last_unstale":"2021-03-15T22:26:23.170220+0000","last_undegraded":"2021-03-15T22:26:23.170220+0000","last_fullsized":"2021-03-15T22:26:23.170220+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.614288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.614288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"436","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.134553+0000","last_change":"2021-03-15T22:25:10.130486+0000","last_active":"2021-03-15T22:26:27.134553+0000","last_peered":"2021-03-15T22:26:27.134553+0000","last_clean":"2021-03-15T22:26:27.134553+0000","last_became_active":"2021-03-15T22:23:52.263426+0000","last_became_peered":"2021-03-15T22:23:52.263426+0000","last_unstale":"2021-03-15T22:26:27.134553+0000","last_undegraded":"2021-03-15T22:26:27.134553+0000","last_fullsized":"2021-03-15T22:26:27.134553+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:10.130422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.130422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"87","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727385+0000","last_change":"2021-03-15T22:25:35.064301+0000","last_active":"2021-03-15T22:26:25.727385+0000","last_peered":"2021-03-15T22:26:25.727385+0000","last_clean":"2021-03-15T22:26:25.727385+0000","last_became_active":"2021-03-15T22:25:33.151159+0000","last_became_peered":"2021-03-15T22:25:33.151159+0000","last_unstale":"2021-03-15T22:26:25.727385+0000","last_undegraded":"2021-03-15T22:26:25.727385+0000","last_fullsized":"2021-03-15T22:26:25.727385+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:26.478799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.478799+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3618,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"433","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.135121+0000","last_change":"2021-03-15T22:26:07.054564+0000","last_active":"2021-03-15T22:26:27.135121+0000","last_peered":"2021-03-15T22:26:27.135121+0000","last_clean":"2021-03-15T22:26:27.135121+0000","last_became_active":"2021-03-15T22:23:52.266133+0000","last_became_peered":"2021-03-15T22:23:52.266133+0000","last_unstale":"2021-03-15T22:26:27.135121+0000","last_undegraded":"2021-03-15T22:26:27.135121+0000","last_fullsized":"2021-03-15T22:26:27.135121+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:07.054500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.054500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"484","reported_epoch":"469","state":"active+clean","last_fresh":"2021-03-15T22:26:24.220464+0000","last_change":"2021-03-15T22:26:10.840930+0000","last_active":"2021-03-15T22:26:24.220464+0000","last_peered":"2021-03-15T22:26:24.220464+0000","last_clean":"2021-03-15T22:26:24.220464+0000","last_became_active":"2021-03-15T22:25:33.150147+0000","last_became_peered":"2021-03-15T22:25:33.150147+0000","last_unstale":"2021-03-15T22:26:24.220464+0000","last_undegraded":"2021-03-15T22:26:24.220464+0000","last_fullsized":"2021-03-15T22:26:24.220464+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:10.840841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:10.840841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"398'11","reported_seq":"73","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727459+0000","last_change":"2021-03-15T22:25:33.864526+0000","last_active":"2021-03-15T22:26:25.727459+0000","last_peered":"2021-03-15T22:26:25.727459+0000","last_clean":"2021-03-15T22:26:25.727459+0000","last_became_active":"2021-03-15T22:25:33.152373+0000","last_became_peered":"2021-03-15T22:25:33.152373+0000","last_unstale":"2021-03-15T22:26:25.727459+0000","last_undegraded":"2021-03-15T22:26:25.727459+0000","last_fullsized":"2021-03-15T22:26:25.727459+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'11","last_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:25:13.181225+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"471'492","reported_seq":"1843","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.132495+0000","last_change":"2021-03-15T22:26:26.132495+0000","last_active":"2021-03-15T22:26:26.132495+0000","last_peered":"2021-03-15T22:26:26.132495+0000","last_clean":"2021-03-15T22:26:26.132495+0000","last_became_active":"2021-03-15T22:25:17.113174+0000","last_became_peered":"2021-03-15T22:25:17.113174+0000","last_unstale":"2021-03-15T22:26:26.132495+0000","last_undegraded":"2021-03-15T22:26:26.132495+0000","last_fullsized":"2021-03-15T22:26:26.132495+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'282","last_scrub_stamp":"2021-03-15T22:25:09.803362+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.803362+0000","log_size":492,"ondisk_log_size":492,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25586932,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":295,"num_read_kb":332633,"num_write":175,"num_write_kb":54593,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":21669077,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"440","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.133397+0000","last_change":"2021-03-15T22:25:00.689659+0000","last_active":"2021-03-15T22:26:26.133397+0000","last_peered":"2021-03-15T22:26:26.133397+0000","last_clean":"2021-03-15T22:26:26.133397+0000","last_became_active":"2021-03-15T22:24:31.103470+0000","last_became_peered":"2021-03-15T22:24:31.103470+0000","last_unstale":"2021-03-15T22:26:26.133397+0000","last_undegraded":"2021-03-15T22:26:26.133397+0000","last_fullsized":"2021-03-15T22:26:26.133397+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.689591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.689591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,2],"acting":[5,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"457","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.135485+0000","last_change":"2021-03-15T22:25:48.008983+0000","last_active":"2021-03-15T22:26:27.135485+0000","last_peered":"2021-03-15T22:26:27.135485+0000","last_clean":"2021-03-15T22:26:27.135485+0000","last_became_active":"2021-03-15T22:24:31.097688+0000","last_became_peered":"2021-03-15T22:24:31.097688+0000","last_unstale":"2021-03-15T22:26:27.135485+0000","last_undegraded":"2021-03-15T22:26:27.135485+0000","last_fullsized":"2021-03-15T22:26:27.135485+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:25:48.008911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:48.008911+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"159","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.133606+0000","last_change":"2021-03-15T22:25:32.799965+0000","last_active":"2021-03-15T22:26:26.133606+0000","last_peered":"2021-03-15T22:26:26.133606+0000","last_clean":"2021-03-15T22:26:26.133606+0000","last_became_active":"2021-03-15T22:24:23.137531+0000","last_became_peered":"2021-03-15T22:24:23.137531+0000","last_unstale":"2021-03-15T22:26:26.133606+0000","last_undegraded":"2021-03-15T22:26:26.133606+0000","last_fullsized":"2021-03-15T22:26:26.133606+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:25:32.799882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.799882+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"317'7","reported_seq":"552","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.134799+0000","last_change":"2021-03-15T22:24:30.068613+0000","last_active":"2021-03-15T22:26:27.134799+0000","last_peered":"2021-03-15T22:26:27.134799+0000","last_clean":"2021-03-15T22:26:27.134799+0000","last_became_active":"2021-03-15T22:23:52.262878+0000","last_became_peered":"2021-03-15T22:23:52.262878+0000","last_unstale":"2021-03-15T22:26:27.134799+0000","last_undegraded":"2021-03-15T22:26:27.134799+0000","last_fullsized":"2021-03-15T22:26:27.134799+0000","mapping_epoch":316,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:24:30.068550+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.068550+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"471'698","reported_seq":"1952","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.144731+0000","last_change":"2021-03-15T22:26:26.144903+0000","last_active":"2021-03-15T22:26:26.144731+0000","last_peered":"2021-03-15T22:26:26.144731+0000","last_clean":"2021-03-15T22:26:26.144731+0000","last_became_active":"2021-03-15T22:23:04.067641+0000","last_became_peered":"2021-03-15T22:23:04.067641+0000","last_unstale":"2021-03-15T22:26:26.144731+0000","last_undegraded":"2021-03-15T22:26:26.144731+0000","last_fullsized":"2021-03-15T22:26:26.144731+0000","mapping_epoch":268,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'455","last_scrub_stamp":"2021-03-15T22:24:28.184612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.184612+0000","log_size":698,"ondisk_log_size":698,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43456471,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":347,"num_read_kb":351081,"num_write":238,"num_write_kb":96326,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":32120558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"230","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727679+0000","last_change":"2021-03-15T22:24:37.122501+0000","last_active":"2021-03-15T22:26:25.727679+0000","last_peered":"2021-03-15T22:26:25.727679+0000","last_clean":"2021-03-15T22:26:25.727679+0000","last_became_active":"2021-03-15T22:22:56.057174+0000","last_became_peered":"2021-03-15T22:22:56.057174+0000","last_unstale":"2021-03-15T22:26:25.727679+0000","last_undegraded":"2021-03-15T22:26:25.727679+0000","last_fullsized":"2021-03-15T22:26:25.727679+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:37.122407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.122407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"441","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.135430+0000","last_change":"2021-03-15T22:26:03.105518+0000","last_active":"2021-03-15T22:26:27.135430+0000","last_peered":"2021-03-15T22:26:27.135430+0000","last_clean":"2021-03-15T22:26:27.135430+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:26:27.135430+0000","last_undegraded":"2021-03-15T22:26:27.135430+0000","last_fullsized":"2021-03-15T22:26:27.135430+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.105446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.105446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"518","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963799+0000","last_change":"2021-03-15T22:25:33.862633+0000","last_active":"2021-03-15T22:26:25.963799+0000","last_peered":"2021-03-15T22:26:25.963799+0000","last_clean":"2021-03-15T22:26:25.963799+0000","last_became_active":"2021-03-15T22:25:33.138262+0000","last_became_peered":"2021-03-15T22:25:33.138262+0000","last_unstale":"2021-03-15T22:26:25.963799+0000","last_undegraded":"2021-03-15T22:26:25.963799+0000","last_fullsized":"2021-03-15T22:26:25.963799+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:07.027095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.027095+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"463","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.169993+0000","last_change":"2021-03-15T22:24:39.853332+0000","last_active":"2021-03-15T22:26:23.169993+0000","last_peered":"2021-03-15T22:26:23.169993+0000","last_clean":"2021-03-15T22:26:23.169993+0000","last_became_active":"2021-03-15T22:24:31.109058+0000","last_became_peered":"2021-03-15T22:24:31.109058+0000","last_unstale":"2021-03-15T22:26:23.169993+0000","last_undegraded":"2021-03-15T22:26:23.169993+0000","last_fullsized":"2021-03-15T22:26:23.169993+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.853267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.853267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"471'165","reported_seq":"1368","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.131598+0000","last_change":"2021-03-15T22:26:26.131598+0000","last_active":"2021-03-15T22:26:26.131598+0000","last_peered":"2021-03-15T22:26:26.131598+0000","last_clean":"2021-03-15T22:26:26.131598+0000","last_became_active":"2021-03-15T22:24:34.087074+0000","last_became_peered":"2021-03-15T22:24:34.087074+0000","last_unstale":"2021-03-15T22:26:26.131598+0000","last_undegraded":"2021-03-15T22:26:26.131598+0000","last_fullsized":"2021-03-15T22:26:26.131598+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"390'130","last_scrub_stamp":"2021-03-15T22:25:06.854931+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.854931+0000","log_size":165,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5697738,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":43,"num_read_kb":30033,"num_write":52,"num_write_kb":17214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4470794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"451","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.134346+0000","last_change":"2021-03-15T22:26:13.044569+0000","last_active":"2021-03-15T22:26:27.134346+0000","last_peered":"2021-03-15T22:26:27.134346+0000","last_clean":"2021-03-15T22:26:27.134346+0000","last_became_active":"2021-03-15T22:23:52.267164+0000","last_became_peered":"2021-03-15T22:23:52.267164+0000","last_unstale":"2021-03-15T22:26:27.134346+0000","last_undegraded":"2021-03-15T22:26:27.134346+0000","last_fullsized":"2021-03-15T22:26:27.134346+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:13.044467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:13.044467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"481","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.170748+0000","last_change":"2021-03-15T22:25:14.114184+0000","last_active":"2021-03-15T22:26:23.170748+0000","last_peered":"2021-03-15T22:26:23.170748+0000","last_clean":"2021-03-15T22:26:23.170748+0000","last_became_active":"2021-03-15T22:25:14.113857+0000","last_became_peered":"2021-03-15T22:25:14.113857+0000","last_unstale":"2021-03-15T22:26:23.170748+0000","last_undegraded":"2021-03-15T22:26:23.170748+0000","last_fullsized":"2021-03-15T22:26:23.170748+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.575552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.575552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"454","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.135298+0000","last_change":"2021-03-15T22:25:14.118841+0000","last_active":"2021-03-15T22:26:27.135298+0000","last_peered":"2021-03-15T22:26:27.135298+0000","last_clean":"2021-03-15T22:26:27.135298+0000","last_became_active":"2021-03-15T22:25:14.118469+0000","last_became_peered":"2021-03-15T22:25:14.118469+0000","last_unstale":"2021-03-15T22:26:27.135298+0000","last_undegraded":"2021-03-15T22:26:27.135298+0000","last_fullsized":"2021-03-15T22:26:27.135298+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.024799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.024799+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"515","reported_epoch":"469","state":"active+clean","last_fresh":"2021-03-15T22:26:24.220860+0000","last_change":"2021-03-15T22:26:11.821654+0000","last_active":"2021-03-15T22:26:24.220860+0000","last_peered":"2021-03-15T22:26:24.220860+0000","last_clean":"2021-03-15T22:26:24.220860+0000","last_became_active":"2021-03-15T22:24:03.085862+0000","last_became_peered":"2021-03-15T22:24:03.085862+0000","last_unstale":"2021-03-15T22:26:24.220860+0000","last_undegraded":"2021-03-15T22:26:24.220860+0000","last_fullsized":"2021-03-15T22:26:24.220860+0000","mapping_epoch":327,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:26:11.821564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:11.821564+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"468'444","reported_seq":"1624","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.180223+0000","last_change":"2021-03-15T22:26:23.180223+0000","last_active":"2021-03-15T22:26:23.180223+0000","last_peered":"2021-03-15T22:26:23.180223+0000","last_clean":"2021-03-15T22:26:23.180223+0000","last_became_active":"2021-03-15T22:20:05.032282+0000","last_became_peered":"2021-03-15T22:20:05.032282+0000","last_unstale":"2021-03-15T22:26:23.180223+0000","last_undegraded":"2021-03-15T22:26:23.180223+0000","last_fullsized":"2021-03-15T22:26:23.180223+0000","mapping_epoch":91,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'328","last_scrub_stamp":"2021-03-15T22:25:37.707812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.707812+0000","log_size":444,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19195372,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":206,"num_read_kb":247372,"num_write":118,"num_write_kb":38785,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"133","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"151","length":"1"},{"start":"153","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"134","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.134378+0000","last_change":"2021-03-15T22:25:11.117468+0000","last_active":"2021-03-15T22:26:27.134378+0000","last_peered":"2021-03-15T22:26:27.134378+0000","last_clean":"2021-03-15T22:26:27.134378+0000","last_became_active":"2021-03-15T22:24:31.093577+0000","last_became_peered":"2021-03-15T22:24:31.093577+0000","last_unstale":"2021-03-15T22:26:27.134378+0000","last_undegraded":"2021-03-15T22:26:27.134378+0000","last_fullsized":"2021-03-15T22:26:27.134378+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.117403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.117403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"481","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.170826+0000","last_change":"2021-03-15T22:25:33.134762+0000","last_active":"2021-03-15T22:26:23.170826+0000","last_peered":"2021-03-15T22:26:23.170826+0000","last_clean":"2021-03-15T22:26:23.170826+0000","last_became_active":"2021-03-15T22:25:33.134358+0000","last_became_peered":"2021-03-15T22:25:33.134358+0000","last_unstale":"2021-03-15T22:26:23.170826+0000","last_undegraded":"2021-03-15T22:26:23.170826+0000","last_fullsized":"2021-03-15T22:26:23.170826+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:27.869570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.869570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"455","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.132808+0000","last_change":"2021-03-15T22:25:47.753887+0000","last_active":"2021-03-15T22:26:26.132808+0000","last_peered":"2021-03-15T22:26:26.132808+0000","last_clean":"2021-03-15T22:26:26.132808+0000","last_became_active":"2021-03-15T22:25:33.145548+0000","last_became_peered":"2021-03-15T22:25:33.145548+0000","last_unstale":"2021-03-15T22:26:26.132808+0000","last_undegraded":"2021-03-15T22:26:26.132808+0000","last_fullsized":"2021-03-15T22:26:26.132808+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:47.753814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:47.753814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"231","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.170938+0000","last_change":"2021-03-15T22:25:33.912570+0000","last_active":"2021-03-15T22:26:23.170938+0000","last_peered":"2021-03-15T22:26:23.170938+0000","last_clean":"2021-03-15T22:26:23.170938+0000","last_became_active":"2021-03-15T22:25:33.135102+0000","last_became_peered":"2021-03-15T22:25:33.135102+0000","last_unstale":"2021-03-15T22:26:23.170938+0000","last_undegraded":"2021-03-15T22:26:23.170938+0000","last_fullsized":"2021-03-15T22:26:23.170938+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:01.499099+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.499099+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":16,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"468'471","reported_seq":"1808","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.119463+0000","last_change":"2021-03-15T22:26:23.182457+0000","last_active":"2021-03-15T22:26:25.119463+0000","last_peered":"2021-03-15T22:26:25.119463+0000","last_clean":"2021-03-15T22:26:25.119463+0000","last_became_active":"2021-03-15T22:25:36.118229+0000","last_became_peered":"2021-03-15T22:25:36.118229+0000","last_unstale":"2021-03-15T22:26:25.119463+0000","last_undegraded":"2021-03-15T22:26:25.119463+0000","last_fullsized":"2021-03-15T22:26:25.119463+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'340","last_scrub_stamp":"2021-03-15T22:25:12.353191+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.353191+0000","log_size":471,"ondisk_log_size":471,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31612048,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":148,"num_read_kb":101628,"num_write":186,"num_write_kb":57477,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":35579701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"68","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727444+0000","last_change":"2021-03-15T22:26:04.729228+0000","last_active":"2021-03-15T22:26:25.727444+0000","last_peered":"2021-03-15T22:26:25.727444+0000","last_clean":"2021-03-15T22:26:25.727444+0000","last_became_active":"2021-03-15T22:25:33.152232+0000","last_became_peered":"2021-03-15T22:25:33.152232+0000","last_unstale":"2021-03-15T22:26:25.727444+0000","last_undegraded":"2021-03-15T22:26:25.727444+0000","last_fullsized":"2021-03-15T22:26:25.727444+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:04.729163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:04.729163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"439","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.133135+0000","last_change":"2021-03-15T22:25:11.615097+0000","last_active":"2021-03-15T22:26:26.133135+0000","last_peered":"2021-03-15T22:26:26.133135+0000","last_clean":"2021-03-15T22:26:26.133135+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:26:26.133135+0000","last_undegraded":"2021-03-15T22:26:26.133135+0000","last_fullsized":"2021-03-15T22:26:26.133135+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.614975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.614975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"165","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.170667+0000","last_change":"2021-03-15T22:25:45.692608+0000","last_active":"2021-03-15T22:26:23.170667+0000","last_peered":"2021-03-15T22:26:23.170667+0000","last_clean":"2021-03-15T22:26:23.170667+0000","last_became_active":"2021-03-15T22:24:31.106344+0000","last_became_peered":"2021-03-15T22:24:31.106344+0000","last_unstale":"2021-03-15T22:26:23.170667+0000","last_undegraded":"2021-03-15T22:26:23.170667+0000","last_fullsized":"2021-03-15T22:26:23.170667+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:45.692545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:45.692545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,1],"acting":[2,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"577","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.126316+0000","last_change":"2021-03-15T22:25:33.174619+0000","last_active":"2021-03-15T22:26:26.126316+0000","last_peered":"2021-03-15T22:26:26.126316+0000","last_clean":"2021-03-15T22:26:26.126316+0000","last_became_active":"2021-03-15T22:25:33.126039+0000","last_became_peered":"2021-03-15T22:25:33.126039+0000","last_unstale":"2021-03-15T22:26:26.126316+0000","last_undegraded":"2021-03-15T22:26:26.126316+0000","last_fullsized":"2021-03-15T22:26:26.126316+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:25:15.810377+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.810377+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"132","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.127731+0000","last_change":"2021-03-15T22:25:44.619146+0000","last_active":"2021-03-15T22:26:26.127731+0000","last_peered":"2021-03-15T22:26:26.127731+0000","last_clean":"2021-03-15T22:26:26.127731+0000","last_became_active":"2021-03-15T22:24:31.101668+0000","last_became_peered":"2021-03-15T22:24:31.101668+0000","last_unstale":"2021-03-15T22:26:26.127731+0000","last_undegraded":"2021-03-15T22:26:26.127731+0000","last_fullsized":"2021-03-15T22:26:26.127731+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:44.619082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:44.619082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"471'454","reported_seq":"1672","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.134884+0000","last_change":"2021-03-15T22:26:26.135332+0000","last_active":"2021-03-15T22:26:26.134884+0000","last_peered":"2021-03-15T22:26:26.134884+0000","last_clean":"2021-03-15T22:26:26.134884+0000","last_became_active":"2021-03-15T22:24:34.087484+0000","last_became_peered":"2021-03-15T22:24:34.087484+0000","last_unstale":"2021-03-15T22:26:26.134884+0000","last_undegraded":"2021-03-15T22:26:26.134884+0000","last_fullsized":"2021-03-15T22:26:26.134884+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'329","last_scrub_stamp":"2021-03-15T22:25:14.014776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:14.014776+0000","log_size":454,"ondisk_log_size":454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17770773,"num_objects":9,"num_object_clones":6,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":140,"num_read_kb":128788,"num_write":138,"num_write_kb":47343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":10768640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"451","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.135454+0000","last_change":"2021-03-15T22:26:08.042648+0000","last_active":"2021-03-15T22:26:27.135454+0000","last_peered":"2021-03-15T22:26:27.135454+0000","last_clean":"2021-03-15T22:26:27.135454+0000","last_became_active":"2021-03-15T22:23:52.266556+0000","last_became_peered":"2021-03-15T22:23:52.266556+0000","last_unstale":"2021-03-15T22:26:27.135454+0000","last_undegraded":"2021-03-15T22:26:27.135454+0000","last_fullsized":"2021-03-15T22:26:27.135454+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:08.042588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:08.042588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"440","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.125396+0000","last_change":"2021-03-15T22:24:39.929128+0000","last_active":"2021-03-15T22:26:26.125396+0000","last_peered":"2021-03-15T22:26:26.125396+0000","last_clean":"2021-03-15T22:26:26.125396+0000","last_became_active":"2021-03-15T22:17:54.284371+0000","last_became_peered":"2021-03-15T22:17:54.284371+0000","last_unstale":"2021-03-15T22:26:26.125396+0000","last_undegraded":"2021-03-15T22:26:26.125396+0000","last_fullsized":"2021-03-15T22:26:26.125396+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.929066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.929066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"474","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.169723+0000","last_change":"2021-03-15T22:26:15.579743+0000","last_active":"2021-03-15T22:26:23.169723+0000","last_peered":"2021-03-15T22:26:23.169723+0000","last_clean":"2021-03-15T22:26:23.169723+0000","last_became_active":"2021-03-15T22:25:33.129875+0000","last_became_peered":"2021-03-15T22:25:33.129875+0000","last_unstale":"2021-03-15T22:26:23.169723+0000","last_undegraded":"2021-03-15T22:26:23.169723+0000","last_fullsized":"2021-03-15T22:26:23.169723+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:15.579672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:26:15.579672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,1],"acting":[2,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"149'22","reported_seq":"122","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.126609+0000","last_change":"2021-03-15T22:25:03.853145+0000","last_active":"2021-03-15T22:26:26.126609+0000","last_peered":"2021-03-15T22:26:26.126609+0000","last_clean":"2021-03-15T22:26:26.126609+0000","last_became_active":"2021-03-15T22:24:55.093124+0000","last_became_peered":"2021-03-15T22:24:55.093124+0000","last_unstale":"2021-03-15T22:26:26.126609+0000","last_undegraded":"2021-03-15T22:26:26.126609+0000","last_fullsized":"2021-03-15T22:26:26.126609+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'22","last_scrub_stamp":"2021-03-15T22:25:03.853079+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:25:03.853079+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"454","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.171658+0000","last_change":"2021-03-15T22:25:02.520674+0000","last_active":"2021-03-15T22:26:23.171658+0000","last_peered":"2021-03-15T22:26:23.171658+0000","last_clean":"2021-03-15T22:26:23.171658+0000","last_became_active":"2021-03-15T22:21:38.147364+0000","last_became_peered":"2021-03-15T22:21:38.147364+0000","last_unstale":"2021-03-15T22:26:23.171658+0000","last_undegraded":"2021-03-15T22:26:23.171658+0000","last_fullsized":"2021-03-15T22:26:23.171658+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.520588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.520588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"473'1139","reported_seq":"3058","reported_epoch":"473","state":"active+clean","last_fresh":"2021-03-15T22:26:28.279568+0000","last_change":"2021-03-15T22:26:28.279758+0000","last_active":"2021-03-15T22:26:28.279568+0000","last_peered":"2021-03-15T22:26:28.279568+0000","last_clean":"2021-03-15T22:26:28.279568+0000","last_became_active":"2021-03-15T22:25:37.118396+0000","last_became_peered":"2021-03-15T22:25:37.118396+0000","last_unstale":"2021-03-15T22:26:28.279568+0000","last_undegraded":"2021-03-15T22:26:28.279568+0000","last_fullsized":"2021-03-15T22:26:28.279568+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'1047","last_scrub_stamp":"2021-03-15T22:26:05.046615+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:26:05.046615+0000","log_size":1139,"ondisk_log_size":1139,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":61160388,"num_objects":26,"num_object_clones":19,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":2,"num_read":401,"num_read_kb":357567,"num_write":375,"num_write_kb":162929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":140467323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"},{"start":"15a","length":"1"},{"start":"160","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"449","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.133666+0000","last_change":"2021-03-15T22:26:03.807852+0000","last_active":"2021-03-15T22:26:26.133666+0000","last_peered":"2021-03-15T22:26:26.133666+0000","last_clean":"2021-03-15T22:26:26.133666+0000","last_became_active":"2021-03-15T22:22:56.062745+0000","last_became_peered":"2021-03-15T22:22:56.062745+0000","last_unstale":"2021-03-15T22:26:26.133666+0000","last_undegraded":"2021-03-15T22:26:26.133666+0000","last_fullsized":"2021-03-15T22:26:26.133666+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.807788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.807788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"448","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.134568+0000","last_change":"2021-03-15T22:24:31.111721+0000","last_active":"2021-03-15T22:26:27.134568+0000","last_peered":"2021-03-15T22:26:27.134568+0000","last_clean":"2021-03-15T22:26:27.134568+0000","last_became_active":"2021-03-15T22:24:31.111104+0000","last_became_peered":"2021-03-15T22:24:31.111104+0000","last_unstale":"2021-03-15T22:26:27.134568+0000","last_undegraded":"2021-03-15T22:26:27.134568+0000","last_fullsized":"2021-03-15T22:26:27.134568+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.872030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.872030+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"68","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727066+0000","last_change":"2021-03-15T22:25:46.813664+0000","last_active":"2021-03-15T22:26:25.727066+0000","last_peered":"2021-03-15T22:26:25.727066+0000","last_clean":"2021-03-15T22:26:25.727066+0000","last_became_active":"2021-03-15T22:25:33.147285+0000","last_became_peered":"2021-03-15T22:25:33.147285+0000","last_unstale":"2021-03-15T22:26:25.727066+0000","last_undegraded":"2021-03-15T22:26:25.727066+0000","last_fullsized":"2021-03-15T22:26:25.727066+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:46.813605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.813605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"463","reported_epoch":"469","state":"active+clean","last_fresh":"2021-03-15T22:26:24.221201+0000","last_change":"2021-03-15T22:25:37.458353+0000","last_active":"2021-03-15T22:26:24.221201+0000","last_peered":"2021-03-15T22:26:24.221201+0000","last_clean":"2021-03-15T22:26:24.221201+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:26:24.221201+0000","last_undegraded":"2021-03-15T22:26:24.221201+0000","last_fullsized":"2021-03-15T22:26:24.221201+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:37.458291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.458291+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"437","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.135058+0000","last_change":"2021-03-15T22:26:14.026200+0000","last_active":"2021-03-15T22:26:27.135058+0000","last_peered":"2021-03-15T22:26:27.135058+0000","last_clean":"2021-03-15T22:26:27.135058+0000","last_became_active":"2021-03-15T22:23:52.271442+0000","last_became_peered":"2021-03-15T22:23:52.271442+0000","last_unstale":"2021-03-15T22:26:27.135058+0000","last_undegraded":"2021-03-15T22:26:27.135058+0000","last_fullsized":"2021-03-15T22:26:27.135058+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:14.026107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:14.026107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"473'353","reported_seq":"1598","reported_epoch":"473","state":"active+clean","last_fresh":"2021-03-15T22:26:28.271125+0000","last_change":"2021-03-15T22:26:28.271398+0000","last_active":"2021-03-15T22:26:28.271125+0000","last_peered":"2021-03-15T22:26:28.271125+0000","last_clean":"2021-03-15T22:26:28.271125+0000","last_became_active":"2021-03-15T22:24:36.092680+0000","last_became_peered":"2021-03-15T22:24:36.092680+0000","last_unstale":"2021-03-15T22:26:28.271125+0000","last_undegraded":"2021-03-15T22:26:28.271125+0000","last_fullsized":"2021-03-15T22:26:28.271125+0000","mapping_epoch":360,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'138","last_scrub_stamp":"2021-03-15T22:23:41.141218+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.141218+0000","log_size":353,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22448359,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":124,"num_read_kb":119123,"num_write":119,"num_write_kb":47137,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8487583,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"},{"start":"15a","length":"1"},{"start":"160","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"452","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.126048+0000","last_change":"2021-03-15T22:25:33.129904+0000","last_active":"2021-03-15T22:26:26.126048+0000","last_peered":"2021-03-15T22:26:26.126048+0000","last_clean":"2021-03-15T22:26:26.126048+0000","last_became_active":"2021-03-15T22:25:33.129602+0000","last_became_peered":"2021-03-15T22:25:33.129602+0000","last_unstale":"2021-03-15T22:26:26.126048+0000","last_undegraded":"2021-03-15T22:26:26.126048+0000","last_fullsized":"2021-03-15T22:26:26.126048+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.931058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.931058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"269","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963541+0000","last_change":"2021-03-15T22:25:33.143239+0000","last_active":"2021-03-15T22:26:25.963541+0000","last_peered":"2021-03-15T22:26:25.963541+0000","last_clean":"2021-03-15T22:26:25.963541+0000","last_became_active":"2021-03-15T22:25:33.142133+0000","last_became_peered":"2021-03-15T22:25:33.142133+0000","last_unstale":"2021-03-15T22:26:25.963541+0000","last_undegraded":"2021-03-15T22:26:25.963541+0000","last_fullsized":"2021-03-15T22:26:25.963541+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.028082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.028082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"959","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:28.262475+0000","last_change":"2021-03-15T22:26:09.051870+0000","last_active":"2021-03-15T22:26:28.262475+0000","last_peered":"2021-03-15T22:26:28.262475+0000","last_clean":"2021-03-15T22:26:28.262475+0000","last_became_active":"2021-03-15T22:24:31.109631+0000","last_became_peered":"2021-03-15T22:24:31.109631+0000","last_unstale":"2021-03-15T22:26:28.262475+0000","last_undegraded":"2021-03-15T22:26:28.262475+0000","last_fullsized":"2021-03-15T22:26:28.262475+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:26:09.051794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.051794+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":500,"num_read_kb":495,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"193","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.125840+0000","last_change":"2021-03-15T22:24:37.977457+0000","last_active":"2021-03-15T22:26:26.125840+0000","last_peered":"2021-03-15T22:26:26.125840+0000","last_clean":"2021-03-15T22:26:26.125840+0000","last_became_active":"2021-03-15T22:24:31.100536+0000","last_became_peered":"2021-03-15T22:24:31.100536+0000","last_unstale":"2021-03-15T22:26:26.125840+0000","last_undegraded":"2021-03-15T22:26:26.125840+0000","last_fullsized":"2021-03-15T22:26:26.125840+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:37.977363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.977363+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"83","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.133901+0000","last_change":"2021-03-15T22:25:14.115071+0000","last_active":"2021-03-15T22:26:26.133901+0000","last_peered":"2021-03-15T22:26:26.133901+0000","last_clean":"2021-03-15T22:26:26.133901+0000","last_became_active":"2021-03-15T22:25:14.114729+0000","last_became_peered":"2021-03-15T22:25:14.114729+0000","last_unstale":"2021-03-15T22:26:26.133901+0000","last_undegraded":"2021-03-15T22:26:26.133901+0000","last_fullsized":"2021-03-15T22:26:26.133901+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.857011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.857011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"472'1259","reported_seq":"3063","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.377336+0000","last_change":"2021-03-15T22:26:26.140373+0000","last_active":"2021-03-15T22:26:27.377336+0000","last_peered":"2021-03-15T22:26:27.377336+0000","last_clean":"2021-03-15T22:26:27.377336+0000","last_became_active":"2021-03-15T22:25:45.240370+0000","last_became_peered":"2021-03-15T22:25:45.240370+0000","last_unstale":"2021-03-15T22:26:27.377336+0000","last_undegraded":"2021-03-15T22:26:27.377336+0000","last_fullsized":"2021-03-15T22:26:27.377336+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"365'777","last_scrub_stamp":"2021-03-15T22:24:40.640400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.640400+0000","log_size":1259,"ondisk_log_size":1259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":61418399,"num_objects":29,"num_object_clones":22,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":2,"num_read":646,"num_read_kb":717776,"num_write":381,"num_write_kb":148197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":125184016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"434","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.125375+0000","last_change":"2021-03-15T22:24:30.950198+0000","last_active":"2021-03-15T22:26:26.125375+0000","last_peered":"2021-03-15T22:26:26.125375+0000","last_clean":"2021-03-15T22:26:26.125375+0000","last_became_active":"2021-03-15T22:18:00.287089+0000","last_became_peered":"2021-03-15T22:18:00.287089+0000","last_unstale":"2021-03-15T22:26:26.125375+0000","last_undegraded":"2021-03-15T22:26:26.125375+0000","last_fullsized":"2021-03-15T22:26:26.125375+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.950135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.950135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"440","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.126234+0000","last_change":"2021-03-15T22:24:26.962034+0000","last_active":"2021-03-15T22:26:26.126234+0000","last_peered":"2021-03-15T22:26:26.126234+0000","last_clean":"2021-03-15T22:26:26.126234+0000","last_became_active":"2021-03-15T22:17:54.283806+0000","last_became_peered":"2021-03-15T22:17:54.283806+0000","last_unstale":"2021-03-15T22:26:26.126234+0000","last_undegraded":"2021-03-15T22:26:26.126234+0000","last_fullsized":"2021-03-15T22:26:26.126234+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.961978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.961978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"221","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.132455+0000","last_change":"2021-03-15T22:26:05.733910+0000","last_active":"2021-03-15T22:26:26.132455+0000","last_peered":"2021-03-15T22:26:26.132455+0000","last_clean":"2021-03-15T22:26:26.132455+0000","last_became_active":"2021-03-15T22:24:31.098665+0000","last_became_peered":"2021-03-15T22:24:31.098665+0000","last_unstale":"2021-03-15T22:26:26.132455+0000","last_undegraded":"2021-03-15T22:26:26.132455+0000","last_fullsized":"2021-03-15T22:26:26.132455+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:26:05.733849+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:26:05.733849+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"448","reported_epoch":"469","state":"active+clean","last_fresh":"2021-03-15T22:26:24.220863+0000","last_change":"2021-03-15T22:26:06.865328+0000","last_active":"2021-03-15T22:26:24.220863+0000","last_peered":"2021-03-15T22:26:24.220863+0000","last_clean":"2021-03-15T22:26:24.220863+0000","last_became_active":"2021-03-15T22:24:31.107629+0000","last_became_peered":"2021-03-15T22:24:31.107629+0000","last_unstale":"2021-03-15T22:26:24.220863+0000","last_undegraded":"2021-03-15T22:26:24.220863+0000","last_fullsized":"2021-03-15T22:26:24.220863+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:06.865257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:26:06.865257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,2],"acting":[6,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"471'202","reported_seq":"1451","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.144364+0000","last_change":"2021-03-15T22:26:26.135642+0000","last_active":"2021-03-15T22:26:26.144364+0000","last_peered":"2021-03-15T22:26:26.144364+0000","last_clean":"2021-03-15T22:26:26.144364+0000","last_became_active":"2021-03-15T22:25:37.112715+0000","last_became_peered":"2021-03-15T22:25:37.112715+0000","last_unstale":"2021-03-15T22:26:26.144364+0000","last_undegraded":"2021-03-15T22:26:26.144364+0000","last_fullsized":"2021-03-15T22:26:26.144364+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"325'104","last_scrub_stamp":"2021-03-15T22:24:02.949625+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:02.949625+0000","log_size":202,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18297389,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":83,"num_read_kb":100356,"num_write":65,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":14412391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"469","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.726973+0000","last_change":"2021-03-15T22:25:14.112119+0000","last_active":"2021-03-15T22:26:25.726973+0000","last_peered":"2021-03-15T22:26:25.726973+0000","last_clean":"2021-03-15T22:26:25.726973+0000","last_became_active":"2021-03-15T22:25:14.111797+0000","last_became_peered":"2021-03-15T22:25:14.111797+0000","last_unstale":"2021-03-15T22:26:25.726973+0000","last_undegraded":"2021-03-15T22:26:25.726973+0000","last_fullsized":"2021-03-15T22:26:25.726973+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.480279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"450","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.133169+0000","last_change":"2021-03-15T22:25:33.141491+0000","last_active":"2021-03-15T22:26:26.133169+0000","last_peered":"2021-03-15T22:26:26.133169+0000","last_clean":"2021-03-15T22:26:26.133169+0000","last_became_active":"2021-03-15T22:25:33.141147+0000","last_became_peered":"2021-03-15T22:25:33.141147+0000","last_unstale":"2021-03-15T22:26:26.133169+0000","last_undegraded":"2021-03-15T22:26:26.133169+0000","last_fullsized":"2021-03-15T22:26:26.133169+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.769850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.769850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"568","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.127245+0000","last_change":"2021-03-15T22:26:06.698784+0000","last_active":"2021-03-15T22:26:26.127245+0000","last_peered":"2021-03-15T22:26:26.127245+0000","last_clean":"2021-03-15T22:26:26.127245+0000","last_became_active":"2021-03-15T22:25:14.114107+0000","last_became_peered":"2021-03-15T22:25:14.114107+0000","last_unstale":"2021-03-15T22:26:26.127245+0000","last_undegraded":"2021-03-15T22:26:26.127245+0000","last_fullsized":"2021-03-15T22:26:26.127245+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:26:06.698716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:06.698716+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"356'3","reported_seq":"167","reported_epoch":"469","state":"active+clean","last_fresh":"2021-03-15T22:26:24.220970+0000","last_change":"2021-03-15T22:26:02.781579+0000","last_active":"2021-03-15T22:26:24.220970+0000","last_peered":"2021-03-15T22:26:24.220970+0000","last_clean":"2021-03-15T22:26:24.220970+0000","last_became_active":"2021-03-15T22:24:31.105948+0000","last_became_peered":"2021-03-15T22:24:31.105948+0000","last_unstale":"2021-03-15T22:26:24.220970+0000","last_undegraded":"2021-03-15T22:26:24.220970+0000","last_fullsized":"2021-03-15T22:26:24.220970+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'3","last_scrub_stamp":"2021-03-15T22:26:02.781478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:26:02.781478+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"471'345","reported_seq":"1528","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.143370+0000","last_change":"2021-03-15T22:26:26.143582+0000","last_active":"2021-03-15T22:26:26.143370+0000","last_peered":"2021-03-15T22:26:26.143370+0000","last_clean":"2021-03-15T22:26:26.143370+0000","last_became_active":"2021-03-15T22:23:00.057259+0000","last_became_peered":"2021-03-15T22:23:00.057259+0000","last_unstale":"2021-03-15T22:26:26.143370+0000","last_undegraded":"2021-03-15T22:26:26.143370+0000","last_fullsized":"2021-03-15T22:26:26.143370+0000","mapping_epoch":264,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'202","last_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.006150+0000","log_size":345,"ondisk_log_size":345,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15831500,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":132,"num_read_kb":122015,"num_write":107,"num_write_kb":42002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":6765552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"433","reported_epoch":"472","state":"active+clean","last_fresh":"2021-03-15T22:26:27.135399+0000","last_change":"2021-03-15T22:25:16.234631+0000","last_active":"2021-03-15T22:26:27.135399+0000","last_peered":"2021-03-15T22:26:27.135399+0000","last_clean":"2021-03-15T22:26:27.135399+0000","last_became_active":"2021-03-15T22:23:52.260549+0000","last_became_peered":"2021-03-15T22:23:52.260549+0000","last_unstale":"2021-03-15T22:26:27.135399+0000","last_undegraded":"2021-03-15T22:26:27.135399+0000","last_fullsized":"2021-03-15T22:26:27.135399+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.234565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.234565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"134","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.125642+0000","last_change":"2021-03-15T22:25:02.868460+0000","last_active":"2021-03-15T22:26:26.125642+0000","last_peered":"2021-03-15T22:26:26.125642+0000","last_clean":"2021-03-15T22:26:26.125642+0000","last_became_active":"2021-03-15T22:24:31.099610+0000","last_became_peered":"2021-03-15T22:24:31.099610+0000","last_unstale":"2021-03-15T22:26:26.125642+0000","last_undegraded":"2021-03-15T22:26:26.125642+0000","last_fullsized":"2021-03-15T22:26:26.125642+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.868399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.868399+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"475","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963697+0000","last_change":"2021-03-15T22:26:09.008669+0000","last_active":"2021-03-15T22:26:25.963697+0000","last_peered":"2021-03-15T22:26:25.963697+0000","last_clean":"2021-03-15T22:26:25.963697+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:26:25.963697+0000","last_undegraded":"2021-03-15T22:26:25.963697+0000","last_fullsized":"2021-03-15T22:26:25.963697+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:26:09.008589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.008589+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"421'6","reported_seq":"82","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.726746+0000","last_change":"2021-03-15T22:26:15.749409+0000","last_active":"2021-03-15T22:26:25.726746+0000","last_peered":"2021-03-15T22:26:25.726746+0000","last_clean":"2021-03-15T22:26:25.726746+0000","last_became_active":"2021-03-15T22:25:33.130973+0000","last_became_peered":"2021-03-15T22:25:33.130973+0000","last_unstale":"2021-03-15T22:26:25.726746+0000","last_undegraded":"2021-03-15T22:26:25.726746+0000","last_fullsized":"2021-03-15T22:26:25.726746+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'6","last_scrub_stamp":"2021-03-15T22:26:15.749336+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:26:15.749336+0000","last_clean_scrub_stamp":"2021-03-15T22:26:15.749336+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"471'586","reported_seq":"1855","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.145242+0000","last_change":"2021-03-15T22:26:26.145523+0000","last_active":"2021-03-15T22:26:26.145242+0000","last_peered":"2021-03-15T22:26:26.145242+0000","last_clean":"2021-03-15T22:26:26.145242+0000","last_became_active":"2021-03-15T22:23:01.072357+0000","last_became_peered":"2021-03-15T22:23:01.072357+0000","last_unstale":"2021-03-15T22:26:26.145242+0000","last_undegraded":"2021-03-15T22:26:26.145242+0000","last_fullsized":"2021-03-15T22:26:26.145242+0000","mapping_epoch":265,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'302","last_scrub_stamp":"2021-03-15T22:23:59.467761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:59.467761+0000","log_size":586,"ondisk_log_size":586,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34739802,"num_objects":15,"num_object_clones":11,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":247,"num_read_kb":259102,"num_write":172,"num_write_kb":64492,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31233387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"454","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963419+0000","last_change":"2021-03-15T22:25:47.117313+0000","last_active":"2021-03-15T22:26:25.963419+0000","last_peered":"2021-03-15T22:26:25.963419+0000","last_clean":"2021-03-15T22:26:25.963419+0000","last_became_active":"2021-03-15T22:25:33.136621+0000","last_became_peered":"2021-03-15T22:25:33.136621+0000","last_unstale":"2021-03-15T22:26:25.963419+0000","last_undegraded":"2021-03-15T22:26:25.963419+0000","last_fullsized":"2021-03-15T22:26:25.963419+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:47.117252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:47.117252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"239","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727652+0000","last_change":"2021-03-15T22:25:12.911203+0000","last_active":"2021-03-15T22:26:25.727652+0000","last_peered":"2021-03-15T22:26:25.727652+0000","last_clean":"2021-03-15T22:26:25.727652+0000","last_became_active":"2021-03-15T22:24:31.091206+0000","last_became_peered":"2021-03-15T22:24:31.091206+0000","last_unstale":"2021-03-15T22:26:25.727652+0000","last_undegraded":"2021-03-15T22:26:25.727652+0000","last_fullsized":"2021-03-15T22:26:25.727652+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.911140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.911140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"89","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.726641+0000","last_change":"2021-03-15T22:25:33.758535+0000","last_active":"2021-03-15T22:26:25.726641+0000","last_peered":"2021-03-15T22:26:25.726641+0000","last_clean":"2021-03-15T22:26:25.726641+0000","last_became_active":"2021-03-15T22:25:33.149362+0000","last_became_peered":"2021-03-15T22:25:33.149362+0000","last_unstale":"2021-03-15T22:26:25.726641+0000","last_undegraded":"2021-03-15T22:26:25.726641+0000","last_fullsized":"2021-03-15T22:26:25.726641+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:24:01.896988+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:19:04.746224+0000","last_clean_scrub_stamp":"2021-03-15T22:24:01.896988+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"462","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963293+0000","last_change":"2021-03-15T22:25:33.143430+0000","last_active":"2021-03-15T22:26:25.963293+0000","last_peered":"2021-03-15T22:26:25.963293+0000","last_clean":"2021-03-15T22:26:25.963293+0000","last_became_active":"2021-03-15T22:25:33.142712+0000","last_became_peered":"2021-03-15T22:25:33.142712+0000","last_unstale":"2021-03-15T22:26:25.963293+0000","last_undegraded":"2021-03-15T22:26:25.963293+0000","last_fullsized":"2021-03-15T22:26:25.963293+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.989313+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.989313+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"468'532","reported_seq":"1819","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.118932+0000","last_change":"2021-03-15T22:26:23.183584+0000","last_active":"2021-03-15T22:26:25.118932+0000","last_peered":"2021-03-15T22:26:25.118932+0000","last_clean":"2021-03-15T22:26:25.118932+0000","last_became_active":"2021-03-15T22:25:36.118668+0000","last_became_peered":"2021-03-15T22:25:36.118668+0000","last_unstale":"2021-03-15T22:26:25.118932+0000","last_undegraded":"2021-03-15T22:26:25.118932+0000","last_fullsized":"2021-03-15T22:26:25.118932+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'370","last_scrub_stamp":"2021-03-15T22:25:07.414191+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.414191+0000","log_size":532,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24276249,"num_objects":10,"num_object_clones":7,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":193,"num_read_kb":178069,"num_write":149,"num_write_kb":59303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25937939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"440","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.132810+0000","last_change":"2021-03-15T22:26:14.870073+0000","last_active":"2021-03-15T22:26:26.132810+0000","last_peered":"2021-03-15T22:26:26.132810+0000","last_clean":"2021-03-15T22:26:26.132810+0000","last_became_active":"2021-03-15T22:24:31.094299+0000","last_became_peered":"2021-03-15T22:24:31.094299+0000","last_unstale":"2021-03-15T22:26:26.132810+0000","last_undegraded":"2021-03-15T22:26:26.132810+0000","last_fullsized":"2021-03-15T22:26:26.132810+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:14.869980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:14.869980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"391'7","reported_seq":"590","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963578+0000","last_change":"2021-03-15T22:25:46.103567+0000","last_active":"2021-03-15T22:26:25.963578+0000","last_peered":"2021-03-15T22:26:25.963578+0000","last_clean":"2021-03-15T22:26:25.963578+0000","last_became_active":"2021-03-15T22:25:03.403350+0000","last_became_peered":"2021-03-15T22:25:03.403350+0000","last_unstale":"2021-03-15T22:26:25.963578+0000","last_undegraded":"2021-03-15T22:26:25.963578+0000","last_fullsized":"2021-03-15T22:26:25.963578+0000","mapping_epoch":387,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'7","last_scrub_stamp":"2021-03-15T22:25:46.103504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.103504+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"68","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.726703+0000","last_change":"2021-03-15T22:25:44.867726+0000","last_active":"2021-03-15T22:26:25.726703+0000","last_peered":"2021-03-15T22:26:25.726703+0000","last_clean":"2021-03-15T22:26:25.726703+0000","last_became_active":"2021-03-15T22:25:33.148614+0000","last_became_peered":"2021-03-15T22:25:33.148614+0000","last_unstale":"2021-03-15T22:26:25.726703+0000","last_undegraded":"2021-03-15T22:26:25.726703+0000","last_fullsized":"2021-03-15T22:26:25.726703+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:44.867633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:44.867633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"499","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.170369+0000","last_change":"2021-03-15T22:25:46.725552+0000","last_active":"2021-03-15T22:26:23.170369+0000","last_peered":"2021-03-15T22:26:23.170369+0000","last_clean":"2021-03-15T22:26:23.170369+0000","last_became_active":"2021-03-15T22:22:56.057343+0000","last_became_peered":"2021-03-15T22:22:56.057343+0000","last_unstale":"2021-03-15T22:26:23.170369+0000","last_undegraded":"2021-03-15T22:26:23.170369+0000","last_fullsized":"2021-03-15T22:26:23.170369+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:25:46.725489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.725489+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"468'135","reported_seq":"1342","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.118998+0000","last_change":"2021-03-15T22:26:23.179736+0000","last_active":"2021-03-15T22:26:25.118998+0000","last_peered":"2021-03-15T22:26:25.118998+0000","last_clean":"2021-03-15T22:26:25.118998+0000","last_became_active":"2021-03-15T22:24:31.104356+0000","last_became_peered":"2021-03-15T22:24:31.104356+0000","last_unstale":"2021-03-15T22:26:25.118998+0000","last_undegraded":"2021-03-15T22:26:25.118998+0000","last_fullsized":"2021-03-15T22:26:25.118998+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'77","last_scrub_stamp":"2021-03-15T22:25:32.490694+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.490694+0000","log_size":135,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":7752259,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":89,"num_read_kb":89949,"num_write":36,"num_write_kb":13772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1551588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"63","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727178+0000","last_change":"2021-03-15T22:25:33.148157+0000","last_active":"2021-03-15T22:26:25.727178+0000","last_peered":"2021-03-15T22:26:25.727178+0000","last_clean":"2021-03-15T22:26:25.727178+0000","last_became_active":"2021-03-15T22:25:33.147717+0000","last_became_peered":"2021-03-15T22:25:33.147717+0000","last_unstale":"2021-03-15T22:26:25.727178+0000","last_undegraded":"2021-03-15T22:26:25.727178+0000","last_fullsized":"2021-03-15T22:26:25.727178+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:25.372535+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:24:25.372535+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"147'5","reported_seq":"518","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:27.094675+0000","last_change":"2021-03-15T22:24:35.715464+0000","last_active":"2021-03-15T22:26:27.094675+0000","last_peered":"2021-03-15T22:26:27.094675+0000","last_clean":"2021-03-15T22:26:27.094675+0000","last_became_active":"2021-03-15T22:21:02.017066+0000","last_became_peered":"2021-03-15T22:21:02.017066+0000","last_unstale":"2021-03-15T22:26:27.094675+0000","last_undegraded":"2021-03-15T22:26:27.094675+0000","last_fullsized":"2021-03-15T22:26:27.094675+0000","mapping_epoch":60,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'5","last_scrub_stamp":"2021-03-15T22:24:35.715399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:35.715399+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"439","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963456+0000","last_change":"2021-03-15T22:24:34.991087+0000","last_active":"2021-03-15T22:26:25.963456+0000","last_peered":"2021-03-15T22:26:25.963456+0000","last_clean":"2021-03-15T22:26:25.963456+0000","last_became_active":"2021-03-15T22:17:54.285645+0000","last_became_peered":"2021-03-15T22:17:54.285645+0000","last_unstale":"2021-03-15T22:26:25.963456+0000","last_undegraded":"2021-03-15T22:26:25.963456+0000","last_fullsized":"2021-03-15T22:26:25.963456+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.991018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.991018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"520","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963406+0000","last_change":"2021-03-15T22:24:55.128840+0000","last_active":"2021-03-15T22:26:25.963406+0000","last_peered":"2021-03-15T22:26:25.963406+0000","last_clean":"2021-03-15T22:26:25.963406+0000","last_became_active":"2021-03-15T22:24:55.090544+0000","last_became_peered":"2021-03-15T22:24:55.090544+0000","last_unstale":"2021-03-15T22:26:25.963406+0000","last_undegraded":"2021-03-15T22:26:25.963406+0000","last_fullsized":"2021-03-15T22:26:25.963406+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:23:38.685706+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:38.685706+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"468'166","reported_seq":"1483","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.182809+0000","last_change":"2021-03-15T22:26:23.182809+0000","last_active":"2021-03-15T22:26:23.182809+0000","last_peered":"2021-03-15T22:26:23.182809+0000","last_clean":"2021-03-15T22:26:23.182809+0000","last_became_active":"2021-03-15T22:25:37.119078+0000","last_became_peered":"2021-03-15T22:25:37.119078+0000","last_unstale":"2021-03-15T22:26:23.182809+0000","last_undegraded":"2021-03-15T22:26:23.182809+0000","last_fullsized":"2021-03-15T22:26:23.182809+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'115","last_scrub_stamp":"2021-03-15T22:25:10.616929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.616929+0000","log_size":166,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":6574330,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":118,"num_read_kb":145290,"num_write":54,"num_write_kb":14217,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":16774328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,3],"acting":[2,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"133","length":"1"},{"start":"143","length":"1"},{"start":"148","length":"1"},{"start":"151","length":"1"},{"start":"153","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"454","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.170569+0000","last_change":"2021-03-15T22:26:03.754920+0000","last_active":"2021-03-15T22:26:23.170569+0000","last_peered":"2021-03-15T22:26:23.170569+0000","last_clean":"2021-03-15T22:26:23.170569+0000","last_became_active":"2021-03-15T22:25:33.134275+0000","last_became_peered":"2021-03-15T22:25:33.134275+0000","last_unstale":"2021-03-15T22:26:23.170569+0000","last_undegraded":"2021-03-15T22:26:23.170569+0000","last_fullsized":"2021-03-15T22:26:23.170569+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.754853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.754853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"468","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.126463+0000","last_change":"2021-03-15T22:26:16.580494+0000","last_active":"2021-03-15T22:26:26.126463+0000","last_peered":"2021-03-15T22:26:26.126463+0000","last_clean":"2021-03-15T22:26:26.126463+0000","last_became_active":"2021-03-15T22:24:31.106634+0000","last_became_peered":"2021-03-15T22:24:31.106634+0000","last_unstale":"2021-03-15T22:26:26.126463+0000","last_undegraded":"2021-03-15T22:26:26.126463+0000","last_fullsized":"2021-03-15T22:26:26.126463+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:16.580433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:26:16.580433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"446","reported_epoch":"469","state":"active+clean","last_fresh":"2021-03-15T22:26:24.220709+0000","last_change":"2021-03-15T22:26:03.831424+0000","last_active":"2021-03-15T22:26:24.220709+0000","last_peered":"2021-03-15T22:26:24.220709+0000","last_clean":"2021-03-15T22:26:24.220709+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:26:24.220709+0000","last_undegraded":"2021-03-15T22:26:24.220709+0000","last_fullsized":"2021-03-15T22:26:24.220709+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.831371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.831371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"115'33","reported_seq":"485","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963325+0000","last_change":"2021-03-15T22:25:04.028853+0000","last_active":"2021-03-15T22:26:25.963325+0000","last_peered":"2021-03-15T22:26:25.963325+0000","last_clean":"2021-03-15T22:26:25.963325+0000","last_became_active":"2021-03-15T22:17:55.992221+0000","last_became_peered":"2021-03-15T22:17:55.992221+0000","last_unstale":"2021-03-15T22:26:25.963325+0000","last_undegraded":"2021-03-15T22:26:25.963325+0000","last_fullsized":"2021-03-15T22:26:25.963325+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'33","last_scrub_stamp":"2021-03-15T22:25:04.028780+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:25:04.028780+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"471'416","reported_seq":"1839","reported_epoch":"471","state":"active+clean","last_fresh":"2021-03-15T22:26:26.140664+0000","last_change":"2021-03-15T22:26:26.140847+0000","last_active":"2021-03-15T22:26:26.140664+0000","last_peered":"2021-03-15T22:26:26.140664+0000","last_clean":"2021-03-15T22:26:26.140664+0000","last_became_active":"2021-03-15T22:25:39.143407+0000","last_became_peered":"2021-03-15T22:25:39.143407+0000","last_unstale":"2021-03-15T22:26:26.140664+0000","last_undegraded":"2021-03-15T22:26:26.140664+0000","last_fullsized":"2021-03-15T22:26:26.140664+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'343","last_scrub_stamp":"2021-03-15T22:23:26.791043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:26.791043+0000","log_size":416,"ondisk_log_size":416,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9704428,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":189,"num_read_kb":156187,"num_write":149,"num_write_kb":56984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":38892785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"126","length":"1"},{"start":"128","length":"1"},{"start":"143","length":"1"},{"start":"151","length":"1"},{"start":"160","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"431","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.963667+0000","last_change":"2021-03-15T22:24:40.910630+0000","last_active":"2021-03-15T22:26:25.963667+0000","last_peered":"2021-03-15T22:26:25.963667+0000","last_clean":"2021-03-15T22:26:25.963667+0000","last_became_active":"2021-03-15T22:18:00.019365+0000","last_became_peered":"2021-03-15T22:18:00.019365+0000","last_unstale":"2021-03-15T22:26:25.963667+0000","last_undegraded":"2021-03-15T22:26:25.963667+0000","last_fullsized":"2021-03-15T22:26:25.963667+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.910565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.910565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"155'4","reported_seq":"558","reported_epoch":"468","state":"active+clean","last_fresh":"2021-03-15T22:26:23.171825+0000","last_change":"2021-03-15T22:24:29.903505+0000","last_active":"2021-03-15T22:26:23.171825+0000","last_peered":"2021-03-15T22:26:23.171825+0000","last_clean":"2021-03-15T22:26:23.171825+0000","last_became_active":"2021-03-15T22:21:10.013838+0000","last_became_peered":"2021-03-15T22:21:10.013838+0000","last_unstale":"2021-03-15T22:26:23.171825+0000","last_undegraded":"2021-03-15T22:26:23.171825+0000","last_fullsized":"2021-03-15T22:26:23.171825+0000","mapping_epoch":60,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'4","last_scrub_stamp":"2021-03-15T22:24:29.903442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.903442+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"149","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.727175+0000","last_change":"2021-03-15T22:25:02.791176+0000","last_active":"2021-03-15T22:26:25.727175+0000","last_peered":"2021-03-15T22:26:25.727175+0000","last_clean":"2021-03-15T22:26:25.727175+0000","last_became_active":"2021-03-15T22:24:31.097434+0000","last_became_peered":"2021-03-15T22:24:31.097434+0000","last_unstale":"2021-03-15T22:26:25.727175+0000","last_undegraded":"2021-03-15T22:26:25.727175+0000","last_fullsized":"2021-03-15T22:26:25.727175+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.791108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:06.573181+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.791108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"517","reported_epoch":"470","state":"active+clean","last_fresh":"2021-03-15T22:26:25.964136+0000","last_change":"2021-03-15T22:26:07.977706+0000","last_active":"2021-03-15T22:26:25.964136+0000","last_peered":"2021-03-15T22:26:25.964136+0000","last_clean":"2021-03-15T22:26:25.964136+0000","last_became_active":"2021-03-15T22:22:56.053800+0000","last_became_peered":"2021-03-15T22:22:56.053800+0000","last_unstale":"2021-03-15T22:26:25.964136+0000","last_undegraded":"2021-03-15T22:26:25.964136+0000","last_fullsized":"2021-03-15T22:26:25.964136+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:26:07.977641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.977641+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":405522437,"num_objects":182,"num_object_clones":134,"num_object_copies":546,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":182,"num_whiteouts":12,"num_read":3401,"num_read_kb":3436969,"num_write":2514,"num_write_kb":947927,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1197,"num_bytes_recovered":514315662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":888745984,"data_stored":1210830035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7857,"ondisk_log_size":7857,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":500,"num_read_kb":495,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1061,"num_read_kb":566,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":468,"num_bytes_recovered":37148,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2232320,"data_stored":80092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":590,"ondisk_log_size":590,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":32,"ondisk_log_size":32,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299499,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":121320,"kb_used_data":101280,"kb_used_omap":0,"kb_used_meta":20032,"kb_avail":93623832,"statfs":{"total":95995035648,"available":95870803968,"internally_reserved":0,"allocated":103710720,"data_stored":140282519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":20512768},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48799999999999999,"15min":0.49299999999999999},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.873,"5min":2.73,"15min":2.73},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.495,"15min":0.497},"min":{"1min":0.253,"5min":0.24399999999999999,"15min":0.24399999999999999},"max":{"1min":0.70899999999999996,"5min":3.8799999999999999,"15min":3.8799999999999999},"last":0.69399999999999995}]},{"osd":1,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.51400000000000001,"15min":0.48899999999999999},"min":{"1min":0.29299999999999998,"5min":0.28100000000000003,"15min":0.22600000000000001},"max":{"1min":0.85199999999999998,"5min":2.6859999999999999,"15min":2.6859999999999999},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.497,"15min":0.49399999999999999},"min":{"1min":0.29099999999999998,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":1.5149999999999999,"5min":1.877,"15min":1.877},"last":0.52900000000000003}]},{"osd":2,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.80700000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.628},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.32400000000000001,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":0.73099999999999998,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.51500000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.48099999999999998,"15min":0.53600000000000003},"min":{"1min":0.33000000000000002,"5min":0.31,"15min":0.31},"max":{"1min":0.83299999999999996,"5min":0.95999999999999996,"15min":2.0339999999999998},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.53200000000000003,"15min":0.54900000000000004},"min":{"1min":0.36799999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69299999999999995,"5min":2.7949999999999999,"15min":2.7949999999999999},"last":0.60099999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.45100000000000001,"15min":0.436},"min":{"1min":0.28899999999999998,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":0.63900000000000001,"5min":1.835,"15min":1.835},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46800000000000003,"15min":0.44800000000000001},"min":{"1min":0.317,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.69699999999999995,"5min":2.5339999999999998,"15min":2.5339999999999998},"last":0.42799999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.50900000000000001,"15min":0.46000000000000002},"min":{"1min":0.26400000000000001,"5min":0.253,"15min":0.253},"max":{"1min":0.72299999999999998,"5min":2.5110000000000001,"15min":2.5110000000000001},"last":0.379},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.51100000000000001,"15min":0.503},"min":{"1min":0.31,"5min":0.255,"15min":0.255},"max":{"1min":0.71199999999999997,"5min":1.9870000000000001,"15min":1.9870000000000001},"last":0.47699999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.53400000000000003,"15min":0.51200000000000001},"min":{"1min":0.32600000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76000000000000001,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.52600000000000002,"15min":0.51100000000000001},"min":{"1min":0.36099999999999999,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":0.68000000000000005,"5min":1.2250000000000001,"15min":1.2250000000000001},"last":0.504}]}]},{"osd":6,"up_from":45,"seq":193273528430,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112604,"kb_used_data":95124,"kb_used_omap":0,"kb_used_meta":17472,"kb_avail":93632548,"statfs":{"total":95995035648,"available":95879729152,"internally_reserved":0,"allocated":97406976,"data_stored":124241016,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17891328},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.47499999999999998,"15min":0.48799999999999999},"min":{"1min":0.27100000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.68100000000000005,"5min":0.995,"15min":0.995},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.46800000000000003,"15min":0.48599999999999999},"min":{"1min":0.28999999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.60099999999999998,"5min":1.1899999999999999,"15min":1.1899999999999999},"last":0.50700000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.49099999999999999,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.254,"15min":0.254},"max":{"1min":0.69399999999999995,"5min":1.05,"15min":1.05},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.48399999999999999,"15min":0.48299999999999998},"min":{"1min":0.33300000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.68000000000000005,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.626}]},{"osd":2,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.317,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65200000000000002,"5min":0.77600000000000002,"15min":0.80800000000000005},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48899999999999999,"15min":0.498},"min":{"1min":0.35399999999999998,"5min":0.32800000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.61099999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.51000000000000001,"15min":0.52400000000000002},"min":{"1min":0.35199999999999998,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.90400000000000003,"5min":1.026,"15min":1.026},"last":0.72299999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51100000000000001,"15min":0.52400000000000002},"min":{"1min":0.33600000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.83399999999999996,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.55900000000000005}]},{"osd":4,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.44800000000000001,"15min":0.41799999999999998},"min":{"1min":0.28000000000000003,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.51500000000000001,"15min":0.46300000000000002},"min":{"1min":0.29499999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.95899999999999996,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.48199999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.501,"15min":0.46200000000000002},"min":{"1min":0.32300000000000001,"5min":0.309,"15min":0.309},"max":{"1min":0.89200000000000002,"5min":1.135,"15min":1.135},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.53200000000000003,"15min":0.48999999999999999},"min":{"1min":0.33200000000000002,"5min":0.311,"15min":0.311},"max":{"1min":0.99299999999999999,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.40000000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:26:01 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.38400000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.78300000000000003,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.56000000000000005,"15min":0.54500000000000004},"min":{"1min":0.42999999999999999,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.92600000000000005,"5min":1.1659999999999999,"15min":1.1659999999999999},"last":0.68300000000000005}]}]},{"osd":1,"up_from":13,"seq":55834574976,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":171148,"kb_used_data":143620,"kb_used_omap":0,"kb_used_meta":27520,"kb_avail":93574004,"statfs":{"total":95995035648,"available":95819780096,"internally_reserved":0,"allocated":147066880,"data_stored":201769551,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28180480},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.313,"5min":0.34499999999999997,"15min":0.52700000000000002},"min":{"1min":0.191,"5min":0.191,"15min":0.191},"max":{"1min":0.48899999999999999,"5min":1.371,"15min":54.427999999999997},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.54400000000000004},"min":{"1min":0.20999999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.48199999999999998,"5min":1.23,"15min":54.356999999999999},"last":0.34999999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.54300000000000004,"5min":1.1220000000000001,"15min":2.4100000000000001},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.38600000000000001},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.502,"5min":0.69799999999999995,"15min":2.0760000000000001},"last":0.374}]},{"osd":3,"last update":"Mon Mar 15 22:26:01 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.38500000000000001,"15min":0.41499999999999998},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.222},"max":{"1min":0.53400000000000003,"5min":1.4910000000000001,"15min":1.796},"last":0.436},{"interface":"front","average":{"1min":0.437,"5min":0.41799999999999998,"15min":0.46000000000000002},"min":{"1min":0.32200000000000001,"5min":0.23000000000000001,"15min":0.20999999999999999},"max":{"1min":0.56799999999999995,"5min":1.135,"15min":1.7150000000000001},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:26:11 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.56299999999999994,"15min":0.56999999999999995},"min":{"1min":0.309,"5min":0.29999999999999999,"15min":0.27700000000000002},"max":{"1min":0.69899999999999995,"5min":3.3439999999999999,"15min":3.3439999999999999},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.61099999999999999,"15min":0.61799999999999999},"min":{"1min":0.37,"5min":0.253,"15min":0.253},"max":{"1min":0.81200000000000006,"5min":4.6139999999999999,"15min":4.6139999999999999},"last":0.80600000000000005}]},{"osd":5,"last update":"Mon Mar 15 22:25:27 2021","interfaces":[{"interface":"back","average":{"1min":0.58299999999999996,"5min":0.57599999999999996,"15min":0.627},"min":{"1min":0.42199999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.93400000000000005,"5min":1.524,"15min":2.153},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.60199999999999998,"5min":0.58499999999999996,"15min":0.95899999999999996},"min":{"1min":0.41399999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.88900000000000001,"5min":1.1950000000000001,"15min":15.472},"last":0.77200000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.59999999999999998,"5min":0.61099999999999999,"15min":0.65600000000000003},"min":{"1min":0.47599999999999998,"5min":0.36899999999999999,"15min":0.33200000000000002},"max":{"1min":0.91800000000000004,"5min":1.276,"15min":1.75},"last":0.83799999999999997},{"interface":"front","average":{"1min":0.59999999999999998,"5min":0.57899999999999996,"15min":0.623},"min":{"1min":0.40500000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.80000000000000004,"5min":0.98299999999999998,"15min":1.3580000000000001},"last":0.82699999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:26:05 2021","interfaces":[{"interface":"back","average":{"1min":0.61099999999999999,"5min":0.63500000000000001,"15min":0.84299999999999997},"min":{"1min":0.44800000000000001,"5min":0.377,"15min":0.316},"max":{"1min":0.78800000000000003,"5min":2.1219999999999999,"15min":8.8100000000000005},"last":0.747},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.58599999999999997,"15min":0.61799999999999999},"min":{"1min":0.38400000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.82599999999999996,"5min":1.0329999999999999,"15min":1.325},"last":0.69999999999999996}]}]},{"osd":0,"up_from":0,"seq":34359738498,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411452,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":131676,"kb_used_data":103828,"kb_used_omap":0,"kb_used_meta":27840,"kb_avail":93613476,"statfs":{"total":95995035648,"available":95860199424,"internally_reserved":0,"allocated":106319872,"data_stored":138893817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":28508160},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.246,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.45700000000000002,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.437,"15min":0.39500000000000002},"min":{"1min":0.246,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":5.5250000000000004,"5min":5.5250000000000004,"15min":5.5250000000000004},"last":0.24199999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.42599999999999999,"15min":0.39600000000000002},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.60999999999999999,"5min":5.4489999999999998,"15min":5.4489999999999998},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51800000000000002,"15min":0.46600000000000003},"min":{"1min":0.27400000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.3740000000000001,"5min":11.659000000000001,"15min":11.659000000000001},"last":0.29499999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:25:58 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.38400000000000001,"15min":0.41399999999999998},"min":{"1min":0.23699999999999999,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":1.2010000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.26300000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.47699999999999998},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.24099999999999999},"max":{"1min":1.28,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.318}]},{"osd":4,"last update":"Mon Mar 15 22:26:12 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.57299999999999995},"min":{"1min":0.38,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.3100000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.39100000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78400000000000003,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.73799999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:25:36 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.57599999999999996,"15min":0.57499999999999996},"min":{"1min":0.38400000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.75600000000000001,"5min":1.954,"15min":1.954},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.56999999999999995},"min":{"1min":0.40000000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.745,"5min":2.2919999999999998,"15min":2.2919999999999998},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":1.875,"15min":1.875},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.435,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.3520000000000001,"5min":1.609,"15min":1.609},"last":0.67100000000000004}]},{"osd":7,"last update":"Mon Mar 15 22:26:12 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56999999999999995,"15min":0.55800000000000005},"min":{"1min":0.38900000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.3320000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.502},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57799999999999996,"15min":0.58099999999999996},"min":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.38700000000000001},"max":{"1min":0.68400000000000005,"5min":1.923,"15min":1.923},"last":0.70899999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149816,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":258376,"kb_used_data":229824,"kb_used_omap":0,"kb_used_meta":28544,"kb_avail":93486776,"statfs":{"total":95995035648,"available":95730458624,"internally_reserved":0,"allocated":235339776,"data_stored":308826511,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":29229056},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.33300000000000002,"5min":0.35199999999999998,"15min":0.35499999999999998},"min":{"1min":0.17699999999999999,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.505,"5min":2.625,"15min":2.625},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.36599999999999999,"15min":0.374},"min":{"1min":0.23499999999999999,"5min":0.20000000000000001,"15min":0.189},"max":{"1min":0.48199999999999998,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.51400000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.44600000000000001,"15min":0.42799999999999999},"min":{"1min":0.23400000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.51400000000000001,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.35099999999999998,"5min":0.45900000000000002,"15min":0.42999999999999999},"min":{"1min":0.24199999999999999,"5min":0.22500000000000001,"15min":0.22500000000000001},"max":{"1min":0.47299999999999998,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.41599999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51800000000000002,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.378},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.42199999999999999,"15min":0.39900000000000002},"min":{"1min":0.27000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54100000000000004,"5min":7.0860000000000003,"15min":7.0860000000000003},"last":0.65900000000000003}]},{"osd":4,"last update":"Mon Mar 15 22:26:13 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55600000000000005,"15min":0.52600000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.34399999999999997,"5min":0.318,"15min":0.30399999999999999},"max":{"1min":1.0409999999999999,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.49299999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.57399999999999995,"15min":0.54500000000000004},"min":{"1min":0.39500000000000002,"5min":0.36199999999999999,"15min":0.33400000000000002},"max":{"1min":0.70399999999999996,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.57099999999999995,"15min":0.56499999999999995},"min":{"1min":0.36799999999999999,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.62,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":0.67900000000000005}]},{"osd":6,"last update":"Mon Mar 15 22:25:43 2021","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.58899999999999997,"15min":0.59499999999999997},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.68500000000000005,"5min":2.6499999999999999,"15min":2.6499999999999999},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.56999999999999995,"15min":0.58099999999999996},"min":{"1min":0.36499999999999999,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.70999999999999996,"5min":1.268,"15min":1.268},"last":0.60799999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.53900000000000003,"5min":0.58099999999999996,"15min":0.57499999999999996},"min":{"1min":0.38300000000000001,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.69599999999999995,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.57499999999999996},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.57799999999999996,"15min":0.57299999999999995},"min":{"1min":0.41399999999999998,"5min":0.39500000000000002,"15min":0.377},"max":{"1min":1.0129999999999999,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.55000000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986293,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94648,"kb_used_data":77360,"kb_used_omap":0,"kb_used_meta":17280,"kb_avail":93650504,"statfs":{"total":95995035648,"available":95898116096,"internally_reserved":0,"allocated":79216640,"data_stored":103681311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":17694720},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:26:18 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.44800000000000001,"15min":0.48699999999999999},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.19500000000000001},"max":{"1min":0.70599999999999996,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47899999999999998,"15min":0.48799999999999999},"min":{"1min":0.32100000000000001,"5min":0.29099999999999998,"15min":0.251},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.74199999999999999},"last":0.69599999999999995}]},{"osd":1,"last update":"Mon Mar 15 22:26:18 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46000000000000002,"15min":0.495},"min":{"1min":0.26300000000000001,"5min":0.25600000000000001,"15min":0.23599999999999999},"max":{"1min":0.56100000000000005,"5min":0.69499999999999995,"15min":1.631},"last":0.432},{"interface":"front","average":{"1min":0.505,"5min":0.48799999999999999,"15min":0.51600000000000001},"min":{"1min":0.30599999999999999,"5min":0.27300000000000002,"15min":0.26800000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":2.2989999999999999},"last":0.55900000000000005}]},{"osd":2,"last update":"Mon Mar 15 22:26:18 2021","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48399999999999999,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.313},"max":{"1min":0.61299999999999999,"5min":0.70399999999999996,"15min":0.76100000000000001},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48299999999999998,"15min":0.502},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68799999999999994,"5min":0.71699999999999997,"15min":1.1910000000000001},"last":0.65500000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:26:18 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51900000000000002,"15min":0.54700000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.98799999999999999},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51500000000000001,"15min":0.55300000000000005},"min":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.99399999999999999},"last":0.53300000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:25:28 2021","interfaces":[{"interface":"back","average":{"1min":0.47899999999999998,"5min":0.45400000000000001,"15min":0.41799999999999998},"min":{"1min":0.27800000000000002,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.745,"5min":0.745,"15min":0.745},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.49299999999999999,"15min":0.441},"min":{"1min":0.36099999999999999,"5min":0.29699999999999999,"15min":0.28799999999999998},"max":{"1min":0.64400000000000002,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.71599999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46800000000000003,"15min":0.437},"min":{"1min":0.29099999999999998,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.65400000000000003,"5min":0.76800000000000002,"15min":0.76800000000000002},"last":0.58599999999999997},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53900000000000003,"15min":0.51400000000000001},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.71499999999999997,"5min":0.78400000000000003,"15min":0.84799999999999998},"last":0.624}]},{"osd":7,"last update":"Mon Mar 15 22:26:00 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.503,"15min":0.48999999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.67000000000000004,"5min":0.73799999999999999,"15min":0.80600000000000005},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54300000000000004,"15min":0.52600000000000002},"min":{"1min":0.33500000000000002,"5min":0.317,"15min":0.317},"max":{"1min":0.75600000000000001,"5min":0.81200000000000006,"15min":0.81200000000000006},"last":0.73399999999999999}]}]},{"osd":5,"up_from":38,"seq":163208757362,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":169864,"kb_used_data":144192,"kb_used_omap":0,"kb_used_meta":25664,"kb_avail":93575288,"statfs":{"total":95995035648,"available":95821094912,"internally_reserved":0,"allocated":147652608,"data_stored":205336183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26279936},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48799999999999999,"15min":0.48299999999999998},"min":{"1min":0.29499999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.63500000000000001,"5min":2.855,"15min":2.855},"last":0.66100000000000003},{"interface":"front","average":{"1min":0.5,"5min":0.49099999999999999,"15min":0.498},"min":{"1min":0.34699999999999998,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.75700000000000001,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.55900000000000005}]},{"osd":1,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.52600000000000002,"15min":0.51500000000000001},"min":{"1min":0.33100000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.72999999999999998,"5min":2.665,"15min":2.665},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.51200000000000001,"15min":0.51000000000000001},"min":{"1min":0.39400000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.72099999999999997,"5min":0.81999999999999995,"15min":0.83999999999999997},"last":0.59399999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.51700000000000002,"15min":0.50900000000000001},"min":{"1min":0.39000000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.61799999999999999,"5min":2.827,"15min":2.827},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.52700000000000002,"15min":0.51400000000000001},"min":{"1min":0.29899999999999999,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.66900000000000004,"5min":1.3600000000000001,"15min":1.3600000000000001},"last":0.71299999999999997}]},{"osd":3,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.55100000000000005,"15min":0.51800000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.308},"max":{"1min":0.68899999999999995,"5min":2.7869999999999999,"15min":2.7869999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.54800000000000004,"15min":0.53300000000000003},"min":{"1min":0.39400000000000002,"5min":0.34100000000000003,"15min":0.33200000000000002},"max":{"1min":0.65500000000000003,"5min":1.4630000000000001,"15min":1.4630000000000001},"last":0.67500000000000004}]},{"osd":4,"last update":"Mon Mar 15 22:25:29 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46999999999999997,"15min":0.40899999999999997},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.24099999999999999},"max":{"1min":0.70899999999999996,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.378},{"interface":"front","average":{"1min":0.496,"5min":0.505,"15min":0.44500000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.25},"max":{"1min":0.748,"5min":1.5109999999999999,"15min":1.5109999999999999},"last":0.41599999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:25:45 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.49399999999999999,"15min":0.47199999999999998},"min":{"1min":0.32300000000000001,"5min":0.29199999999999998,"15min":0.28499999999999998},"max":{"1min":0.80700000000000005,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.56599999999999995,"15min":0.53600000000000003},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.32800000000000001},"max":{"1min":0.78600000000000003,"5min":1.327,"15min":1.327},"last":0.495}]},{"osd":7,"last update":"Mon Mar 15 22:26:02 2021","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.55500000000000005,"15min":0.52200000000000002},"min":{"1min":0.35399999999999998,"5min":0.34699999999999998,"15min":0.30499999999999999},"max":{"1min":0.73099999999999998,"5min":1.3,"15min":1.3},"last":0.439},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.56000000000000005,"15min":0.53600000000000003},"min":{"1min":0.377,"5min":0.307,"15min":0.307},"max":{"1min":0.77100000000000002,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.45900000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":9520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":14458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":8356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":6840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":8774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5443584,"data_stored":5395029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":141905920,"data_stored":198558678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":101380096,"data_stored":135714164,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":231120896,"data_stored":308029818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":75055104,"data_stored":102878835,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":142716928,"data_stored":202132894,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":92426240,"data_stored":121051673,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98697216,"data_stored":137068944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:26:32.125 INFO:tasks.thrashosds.thrasher:Setting 3.3 to [3, 0, 6] 2021-03-15T22:26:32.126 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.3', '3', '0', '6'] 2021-03-15T22:26:32.126 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pg-upmap 3.3 3 0 6 2021-03-15T22:26:32.130 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:32.140 INFO:tasks.rados.rados.0.smithi076.stdout:1780: done (0 left) 2021-03-15T22:26:32.141 INFO:tasks.rados.rados.0.smithi076.stdout:1781: write (excl) oid 4 current snap is 196 2021-03-15T22:26:32.141 INFO:tasks.rados.rados.0.smithi076.stdout:1781: seq_num 624 ranges {487568=496322,1431705=519550} 2021-03-15T22:26:32.147 INFO:tasks.rados.rados.0.smithi076.stdout:1781: writing smithi07694878-4 from 487568 to 983890 tid 1 2021-03-15T22:26:32.154 INFO:tasks.rados.rados.0.smithi076.stdout:1781: writing smithi07694878-4 from 1431705 to 1951255 tid 2 2021-03-15T22:26:32.155 INFO:tasks.rados.rados.0.smithi076.stdout:1781: finishing write tid 1 to smithi07694878-4 2021-03-15T22:26:32.155 INFO:tasks.rados.rados.0.smithi076.stdout:1782: delete oid 25 current snap is 196 2021-03-15T22:26:32.159 INFO:tasks.rados.rados.0.smithi076.stdout:1781: finishing write tid 2 to smithi07694878-4 2021-03-15T22:26:32.159 INFO:tasks.rados.rados.0.smithi076.stdout:1781: finishing write tid 3 to smithi07694878-4 2021-03-15T22:26:32.159 INFO:tasks.rados.rados.0.smithi076.stdout:1781: finishing write tid 4 to smithi07694878-4 2021-03-15T22:26:32.159 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 438 (ObjNum 624 snap 196 seq_num 624) dirty exists 2021-03-15T22:26:32.160 INFO:tasks.rados.rados.0.smithi076.stdout:1781: left oid 4 (ObjNum 624 snap 196 seq_num 624) 2021-03-15T22:26:32.160 INFO:tasks.rados.rados.0.smithi076.stdout:1781: done (1 left) 2021-03-15T22:26:32.161 INFO:tasks.rados.rados.0.smithi076.stdout:1782: done (0 left) 2021-03-15T22:26:32.161 INFO:tasks.rados.rados.0.smithi076.stdout:1783: read oid 49 snap -1 2021-03-15T22:26:32.161 INFO:tasks.rados.rados.0.smithi076.stdout:1783: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:26:32.161 INFO:tasks.rados.rados.0.smithi076.stdout:1784: snap_create 2021-03-15T22:26:32.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:32.171 INFO:tasks.rados.rados.0.smithi076.stdout:1783: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:26:32.190 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:32.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:32.293 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:32.346 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:26:32.346 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:26:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:26:32.347 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:26:32.347 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:26:32.386 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:32.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:32.388 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:32.494 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:32.526 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:32.528 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:32.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:32.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:32.658 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:32.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:32.759 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:32.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:32 smithi076 conmon[56514]: cluster 2021-03-15T22:26:31.069787+0000 2021-03-15T22:26:32.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:32 smithi076 conmon[56514]: mgr.y (mgr.14144) 668 : cluster [DBG] pgmap v840: 138 pgs: 138 active+clean; 387 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 242 KiB/s wr, 7 op/s 2021-03-15T22:26:32.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:32 smithi076 conmon[56514]: cluster 2021-03-15T22:26:32.136222+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2021-03-15T22:26:32.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:32 smithi076 conmon[62156]: cluster 2021-03-15T22:26:31.069787 2021-03-15T22:26:32.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:32 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 668 : cluster [DBG] pgmap v840: 138 pgs: 138 active+clean; 387 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 242 KiB/s wr, 7 op/s 2021-03-15T22:26:32.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:32 smithi076 conmon[62156]: cluster 2021-03-15T22:26 2021-03-15T22:26:32.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:32 smithi076 conmon[62156]: :32.136222+0000 mon.a (mon.0) 1336 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2021-03-15T22:26:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:32.786 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:32.786 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:32.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:32 smithi125 conmon[56739]: cluster 2021-03-15T22:26:31.069787+0000 mgr.y (mgr.14144) 668 : cluster [DBG] pgmap v840: 138 pgs: 138 active+clean; 387 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 6.3 MiB/s rd, 242 KiB/s wr, 7 op/s 2021-03-15T22:26:32.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:32 smithi125 conmon[56739]: cluster 2021-03-15T22:26:32.136222+0000 mon.a 2021-03-15T22:26:32.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:32 smithi125 conmon[56739]: (mon.0) 1336 : cluster [DBG] osdmap e477: 8 total, 8 up, 7 in 2021-03-15T22:26:32.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:32.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:32.912 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:32.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:33.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:33.041 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:33.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:33.052 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:26:33.052 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:26:33.053 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:26:33.053 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:26:33.053 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:26:33.053 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:26:33.054 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:26:33.054 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:26:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:26:33.054 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:26:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:26:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:26:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:26:33.055 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:26:33.056 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:26:33.057 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:26:33.118 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:26:33.141 INFO:tasks.rados.rados.0.smithi076.stdout:1783: done (1 left) 2021-03-15T22:26:33.142 INFO:tasks.rados.rados.0.smithi076.stdout:1784: done (0 left) 2021-03-15T22:26:33.142 INFO:tasks.rados.rados.0.smithi076.stdout:1785: rollback oid 38 current snap is 197 2021-03-15T22:26:33.142 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 38 to 192 2021-03-15T22:26:33.142 INFO:tasks.rados.rados.0.smithi076.stdout:1786: delete oid 36 current snap is 197 2021-03-15T22:26:33.144 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:33.145 INFO:tasks.rados.rados.0.smithi076.stdout:1786: done (1 left) 2021-03-15T22:26:33.145 INFO:tasks.rados.rados.0.smithi076.stdout:1787: delete oid 27 current snap is 197 2021-03-15T22:26:33.146 INFO:tasks.rados.rados.0.smithi076.stdout:1785: finishing rollback tid 0 to smithi07694878-38 2021-03-15T22:26:33.147 INFO:tasks.rados.rados.0.smithi076.stdout:1785: finishing rollback tid 1 to smithi07694878-38 2021-03-15T22:26:33.147 INFO:tasks.rados.rados.0.smithi076.stdout:1785: finishing rollback tid 2 to smithi07694878-38 2021-03-15T22:26:33.148 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 993 (ObjNum 506 snap 151 seq_num 506) dirty exists 2021-03-15T22:26:33.148 INFO:tasks.rados.rados.0.smithi076.stdout:1785: done (1 left) 2021-03-15T22:26:33.148 INFO:tasks.rados.rados.0.smithi076.stdout:1787: done (0 left) 2021-03-15T22:26:33.148 INFO:tasks.rados.rados.0.smithi076.stdout:1788: rollback oid 19 current snap is 197 2021-03-15T22:26:33.149 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 19 to 160 2021-03-15T22:26:33.149 INFO:tasks.rados.rados.0.smithi076.stdout:1789: read oid 9 snap -1 2021-03-15T22:26:33.149 INFO:tasks.rados.rados.0.smithi076.stdout:1789: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:26:33.149 INFO:tasks.rados.rados.0.smithi076.stdout:1790: snap_create 2021-03-15T22:26:33.151 INFO:tasks.rados.rados.0.smithi076.stdout:1788: finishing rollback tid 0 to smithi07694878-19 2021-03-15T22:26:33.151 INFO:tasks.rados.rados.0.smithi076.stdout:1788: finishing rollback tid 1 to smithi07694878-19 2021-03-15T22:26:33.151 INFO:tasks.rados.rados.0.smithi076.stdout:1788: finishing rollback tid 2 to smithi07694878-19 2021-03-15T22:26:33.151 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 996 (ObjNum 495 snap 149 seq_num 495) dirty exists 2021-03-15T22:26:33.160 INFO:tasks.rados.rados.0.smithi076.stdout:1789: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:26:33.171 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:33.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:33.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:33.274 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:33.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:33.300 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:33.301 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:33.402 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:33.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:33.429 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:33.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:33.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:33 smithi076 conmon[56514]: audit 2021-03-15T22:26:33.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:33 smithi076 conmon[56514]: 2021-03-15 2021-03-15T22:26:33.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:33 smithi076 conmon[56514]: T22:26:32.901987+0000 mon.a (mon.0) 1337 : audit [DBG] from='client.? 172.21.15.76:0/2708311559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:33.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:33 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:26:33.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:33 smithi076 conmon[56514]: -15T22:26:33.136778+0000 mon.a ( 2021-03-15T22:26:33.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:33 smithi076 conmon[56514]: mon.0) 1338 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2021-03-15T22:26:33.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:33 smithi076 conmon[62156]: audit 2021-03-15T22:26:32.901987+0000 mon.a (mon.0) 1337 : audit [DBG] from='client.? 172.21.15.76:0/2708311559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:33.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:33 smithi076 conmon[62156]: cluster 2021-03-15T22:26:33.136778+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2021-03-15T22:26:33.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:33.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:33.555 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:33.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:33.656 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:33.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:33.679 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:33.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:33.780 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:33.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:33 smithi125 conmon[56739]: audit 2021-03-15T22:26:32.901987+0000 mon.a (mon.0) 1337 : audit [DBG] from='client.? 172.21.15.76:0/2708311559' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:33.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:33 smithi125 conmon[56739]: cluster 2021-03-15T22:26:33.136778+0000 mon.a (mon.0) 1338 : cluster [DBG] osdmap e478: 8 total, 8 up, 7 in 2021-03-15T22:26:33.802 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:33.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:33.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:33.905 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:33.927 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:33.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:33.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:33.931 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:33.981 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:34.030 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:34.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:34.053 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:34.053 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:34.138 INFO:tasks.rados.rados.0.smithi076.stdout:1788: done (2 left) 2021-03-15T22:26:34.138 INFO:tasks.rados.rados.0.smithi076.stdout:1789: done (1 left) 2021-03-15T22:26:34.139 INFO:tasks.rados.rados.0.smithi076.stdout:1790: done (0 left) 2021-03-15T22:26:34.139 INFO:tasks.rados.rados.0.smithi076.stdout:1791: delete oid 43 current snap is 198 2021-03-15T22:26:34.142 INFO:tasks.rados.rados.0.smithi076.stdout:1791: done (0 left) 2021-03-15T22:26:34.142 INFO:tasks.rados.rados.0.smithi076.stdout:1792: read oid 5 snap 170 2021-03-15T22:26:34.142 INFO:tasks.rados.rados.0.smithi076.stdout:1792: expect deleted 2021-03-15T22:26:34.142 INFO:tasks.rados.rados.0.smithi076.stdout:1793: copy_from oid 33 from oid 31 current snap is 198 2021-03-15T22:26:34.143 INFO:tasks.rados.rados.0.smithi076.stdout:1794: rollback oid 36 current snap is 198 2021-03-15T22:26:34.143 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 36 to 186 2021-03-15T22:26:34.143 INFO:tasks.rados.rados.0.smithi076.stdout:1795: write oid 35 current snap is 198 2021-03-15T22:26:34.143 INFO:tasks.rados.rados.0.smithi076.stdout:1795: seq_num 629 ranges {571603=472567,1607762=670546} 2021-03-15T22:26:34.147 INFO:tasks.rados.rados.0.smithi076.stdout:1795: writing smithi07694878-35 from 571603 to 1044170 tid 1 2021-03-15T22:26:34.155 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:34.156 INFO:tasks.rados.rados.0.smithi076.stdout:1795: writing smithi07694878-35 from 1607762 to 2278308 tid 2 2021-03-15T22:26:34.156 INFO:tasks.rados.rados.0.smithi076.stdout:1796: snap_create 2021-03-15T22:26:34.157 INFO:tasks.rados.rados.0.smithi076.stdout:1794: finishing rollback tid 1 to smithi07694878-36 2021-03-15T22:26:34.157 INFO:tasks.rados.rados.0.smithi076.stdout:1794: finishing rollback tid 2 to smithi07694878-36 2021-03-15T22:26:34.157 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 440 (ObjNum 387 snap 104 seq_num 387) dirty exists 2021-03-15T22:26:34.157 INFO:tasks.rados.rados.0.smithi076.stdout:1795: finishing write tid 1 to smithi07694878-35 2021-03-15T22:26:34.163 INFO:tasks.rados.rados.0.smithi076.stdout:1793: finishing copy_from to smithi07694878-33 2021-03-15T22:26:34.164 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 840 (ObjNum 581 snap 180 seq_num 581) dirty exists 2021-03-15T22:26:34.164 INFO:tasks.rados.rados.0.smithi076.stdout:1795: finishing write tid 2 to smithi07694878-35 2021-03-15T22:26:34.164 INFO:tasks.rados.rados.0.smithi076.stdout:1795: finishing write tid 3 to smithi07694878-35 2021-03-15T22:26:34.164 INFO:tasks.rados.rados.0.smithi076.stdout:1793: finishing copy_from racing read to smithi07694878-33 2021-03-15T22:26:34.165 INFO:tasks.rados.rados.0.smithi076.stdout:1795: finishing write tid 4 to smithi07694878-35 2021-03-15T22:26:34.165 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 999 (ObjNum 629 snap 198 seq_num 629) dirty exists 2021-03-15T22:26:34.165 INFO:tasks.rados.rados.0.smithi076.stdout:1795: left oid 35 (ObjNum 629 snap 198 seq_num 629) 2021-03-15T22:26:34.179 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:34.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:34.181 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:34.282 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:34.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:34.306 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:34.407 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:34.432 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:34.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:34.433 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:34.534 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:34.558 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:34.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:34.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:34.684 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:34.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:34.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:34.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:34 smithi076 conmon[56514]: cluster 2021-03-15T22:26:33.070311+0000 2021-03-15T22:26:34.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:34 smithi076 conmon[56514]: mgr.y (mgr.14144) 669 : cluster [DBG] pgmap v843: 138 pgs: 138 active+clean; 391 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 629 KiB/s wr, 19 op/s 2021-03-15T22:26:34.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:34 smithi076 conmon[56514]: cluster 2021-03-15T22:26:34.135850+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2021-03-15T22:26:34.688 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:34 smithi076 conmon[62156]: cluster 2021-03-15T22:26:33.070311+0000 mgr.y (mgr.14144) 2021-03-15T22:26:34.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:34 smithi076 conmon[62156]: 669 : cluster [DBG] pgmap v843: 138 pgs: 138 active+clean; 391 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 629 KiB/s wr, 19 op/s 2021-03-15T22:26:34.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:34 smithi076 conmon[62156]: cluster 2021-03-15T22:26:34.135850+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2021-03-15T22:26:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:34 smithi125 conmon[56739]: cluster 2021-03-15T22:26:33.070311+0000 mgr.y (mgr.14144) 2021-03-15T22:26:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:34 smithi125 conmon[56739]: 669 : cluster [DBG] pgmap v843: 138 pgs: 138 active+clean; 391 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 22 MiB/s rd, 629 KiB/s wr, 19 op/s 2021-03-15T22:26:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:34 smithi125 conmon[56739]: cluster 2021-03-15T22:26:34.135850+0000 mon.a (mon.0) 1339 : cluster [DBG] osdmap e479: 8 total, 8 up, 7 in 2021-03-15T22:26:34.788 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:34.810 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:34.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:34.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:34.913 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:34.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:34.936 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:34.936 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:35.038 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:35.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:35.062 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:35.063 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:35.135 INFO:tasks.rados.rados.0.smithi076.stdout:1792: done (4 left) 2021-03-15T22:26:35.136 INFO:tasks.rados.rados.0.smithi076.stdout:1793: done (3 left) 2021-03-15T22:26:35.136 INFO:tasks.rados.rados.0.smithi076.stdout:1794: done (2 left) 2021-03-15T22:26:35.136 INFO:tasks.rados.rados.0.smithi076.stdout:1795: done (1 left) 2021-03-15T22:26:35.137 INFO:tasks.rados.rados.0.smithi076.stdout:1796: done (0 left) 2021-03-15T22:26:35.137 INFO:tasks.rados.rados.0.smithi076.stdout:1797: read oid 17 snap 182 2021-03-15T22:26:35.137 INFO:tasks.rados.rados.0.smithi076.stdout:1797: expect (ObjNum 506 snap 151 seq_num 506) 2021-03-15T22:26:35.137 INFO:tasks.rados.rados.0.smithi076.stdout:1798: rollback oid 39 current snap is 199 2021-03-15T22:26:35.137 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 39 to 162 2021-03-15T22:26:35.138 INFO:tasks.rados.rados.0.smithi076.stdout:1799: delete oid 20 current snap is 199 2021-03-15T22:26:35.145 INFO:tasks.rados.rados.0.smithi076.stdout:1798: finishing rollback tid 0 to smithi07694878-39 2021-03-15T22:26:35.146 INFO:tasks.rados.rados.0.smithi076.stdout:1798: finishing rollback tid 1 to smithi07694878-39 2021-03-15T22:26:35.146 INFO:tasks.rados.rados.0.smithi076.stdout:1798: finishing rollback tid 2 to smithi07694878-39 2021-03-15T22:26:35.146 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 1002 (ObjNum 523 snap 158 seq_num 523) dirty exists 2021-03-15T22:26:35.146 INFO:tasks.rados.rados.0.smithi076.stdout:1797: expect (ObjNum 506 snap 151 seq_num 506) 2021-03-15T22:26:35.164 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:35.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:35.190 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:35.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:35.292 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:35.297 INFO:tasks.rados.rados.0.smithi076.stdout:1797: done (2 left) 2021-03-15T22:26:35.297 INFO:tasks.rados.rados.0.smithi076.stdout:1798: done (1 left) 2021-03-15T22:26:35.298 INFO:tasks.rados.rados.0.smithi076.stdout:1799: done (0 left) 2021-03-15T22:26:35.298 INFO:tasks.rados.rados.0.smithi076.stdout:1800: delete oid 1 current snap is 199 2021-03-15T22:26:35.300 INFO:tasks.rados.rados.0.smithi076.stdout:1800: done (0 left) 2021-03-15T22:26:35.300 INFO:tasks.rados.rados.0.smithi076.stdout:1801: rollback oid 8 current snap is 199 2021-03-15T22:26:35.300 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 195 2021-03-15T22:26:35.301 INFO:tasks.rados.rados.0.smithi076.stdout:1802: snap_remove snap 194 2021-03-15T22:26:35.315 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:35.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:35.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:35.418 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:35.440 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:35.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:35.442 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:35.543 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:35.565 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:35.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:35.566 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:35.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:35 smithi076 conmon[56514]: cluster 2021-03-15T22:26:34.568057+0000 osd.6 (osd.6) 75 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:35.577 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:35 smithi076 conmon[56514]: cluster 2021-03-15T22:26:34.568610+0000 osd.6 (osd.6) 76 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:35.577 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:35 smithi076 conmon[56514]: cluster 2021-03-15T22:26:35.125363+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2021-03-15T22:26:35.577 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:35 smithi076 conmon[62156]: cluster 2021-03-15T22:26:34.568057+0000 osd.6 (osd 2021-03-15T22:26:35.578 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:35 smithi076 conmon[62156]: .6) 75 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:35.578 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:35 smithi076 conmon[62156]: cluster 2021-03-15T22:26:34.568610+0000 osd.6 (osd.6 2021-03-15T22:26:35.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:35 smithi076 conmon[62156]: ) 76 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:35.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:35 smithi076 conmon[62156]: cluster 2021-03-15T22:26:35 2021-03-15T22:26:35.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:35 smithi076 conmon[62156]: .125363+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2021-03-15T22:26:35.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:35.693 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:35.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:35.694 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:35.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:35 smithi125 conmon[56739]: cluster 2021-03-15T22:26:34.568057+0000 osd.6 (osd.6) 75 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:35.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:35 smithi125 conmon[56739]: cluster 2021-03-15T22:26:34.568610+0000 osd.6 (osd.6) 76 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:35.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:35 smithi125 conmon[56739]: cluster 2021-03-15T22:26:35 2021-03-15T22:26:35.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:35 smithi125 conmon[56739]: .125363+0000 mon.a (mon.0) 1340 : cluster [DBG] osdmap e480: 8 total, 8 up, 7 in 2021-03-15T22:26:35.799 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:35.823 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:35.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:35.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:35.897 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:26:35.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:35.962 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:35.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:35.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:36.065 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:36.091 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:36.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:36.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:36.143 INFO:tasks.rados.rados.0.smithi076.stdout:1802: done (1 left) 2021-03-15T22:26:36.143 INFO:tasks.rados.rados.0.smithi076.stdout:1803: snap_create 2021-03-15T22:26:36.144 INFO:tasks.rados.rados.0.smithi076.stdout:1801: finishing rollback tid 0 to smithi07694878-8 2021-03-15T22:26:36.144 INFO:tasks.rados.rados.0.smithi076.stdout:1801: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:26:36.144 INFO:tasks.rados.rados.0.smithi076.stdout:1801: finishing rollback tid 2 to smithi07694878-8 2021-03-15T22:26:36.144 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 782 (ObjNum 579 snap 180 seq_num 579) dirty exists 2021-03-15T22:26:36.194 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:36.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:36.223 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:36.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:36.324 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:36.346 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:36.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:36.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:36.449 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:36.471 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:36.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:36.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:36.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:36.601 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:36.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:36.602 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:36.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[56514]: cluster 2021-03-15T22:26:35.070953+0000 mgr.y (mgr.14144 2021-03-15T22:26:36.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[56514]: ) 670 : cluster [DBG] pgmap v846: 138 pgs: 138 active+clean; 394 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 1.4 MiB/s wr, 21 op/s 2021-03-15T22:26:36.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[56514]: cluster 2021-03-15T22:26:36.136374+0000 mon.a (mon.0) 1341 : cluster [DBG] osdmap e481: 8 total, 8 up, 7 in 2021-03-15T22:26:36.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[56514]: audit 2021-03-15T22:26:36.210586+0000 mon.a (mon.0) 1342 : audit [INF] from='client.? 172.21.15.76:0/3460594737' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:26:36.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[62156]: cluster 2021-03-15T22:26:35.070953+0000 mgr.y ( 2021-03-15T22:26:36.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[62156]: mgr.14144) 670 : cluster [DBG] pgmap v846: 138 pgs: 138 active+clean; 394 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 1.4 MiB/s wr, 21 op/s 2021-03-15T22:26:36.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[62156]: cluster 2021-03-15T22:26:36.136374+0000 2021-03-15T22:26:36.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[62156]: mon.a (mon.0) 1341 : cluster [DBG] osdmap e481: 8 total, 8 up, 7 in 2021-03-15T22:26:36.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:26:36.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[62156]: 15T22:26:36.210586 2021-03-15T22:26:36.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1342 : audit 2021-03-15T22:26:36.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:36 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/3460594737' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:26:36.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:36.707 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:36.732 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:36.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:36.734 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:36.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:36 smithi125 conmon[56739]: cluster 2021-03-15T22:26:35.070953+0000 mgr.y (mgr. 2021-03-15T22:26:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:36 smithi125 conmon[56739]: 14144) 670 : cluster [DBG] pgmap v846: 138 pgs: 138 active+clean; 394 MiB data, 1.0 GiB used, 625 GiB / 626 GiB avail; 27 MiB/s rd, 1.4 MiB/s wr, 21 op/s 2021-03-15T22:26:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:36 smithi125 conmon[56739]: cluster 2021-03-15T22:26:36.136374+0000 mon.a (mon.0) 1341 : cluster [DBG] 2021-03-15T22:26:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:36 smithi125 conmon[56739]: osdmap e481: 8 total, 8 up, 7 in 2021-03-15T22:26:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:36 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:26:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:36 smithi125 conmon[56739]: :26:36.210586+0000 mon.a (mon.0) 1342 : audit 2021-03-15T22:26:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:36 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/3460594737' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:26:36.835 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:36.861 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:36.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:36.862 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:36.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:36.989 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:36.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:36.990 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:37.091 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:37.117 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:37.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:37.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:37.144 INFO:tasks.rados.rados.0.smithi076.stdout:1801: done (1 left) 2021-03-15T22:26:37.144 INFO:tasks.rados.rados.0.smithi076.stdout:1803: done (0 left) 2021-03-15T22:26:37.144 INFO:tasks.rados.rados.0.smithi076.stdout:1804: rollback oid 23 current snap is 200 2021-03-15T22:26:37.144 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 23 to 182 2021-03-15T22:26:37.145 INFO:tasks.rados.rados.0.smithi076.stdout:1805: write (excl) oid 3 current snap is 200 2021-03-15T22:26:37.145 INFO:tasks.rados.rados.0.smithi076.stdout:1805: seq_num 632 ranges {622024=414667,1783208=417873} 2021-03-15T22:26:37.147 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:26:37.149 INFO:tasks.rados.rados.0.smithi076.stdout:1805: writing smithi07694878-3 from 622024 to 1036691 tid 1 2021-03-15T22:26:37.157 INFO:tasks.rados.rados.0.smithi076.stdout:1805: writing smithi07694878-3 from 1783208 to 2201081 tid 2 2021-03-15T22:26:37.158 INFO:tasks.rados.rados.0.smithi076.stdout:1806: rollback oid 45 current snap is 200 2021-03-15T22:26:37.158 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 45 to 190 2021-03-15T22:26:37.158 INFO:tasks.rados.rados.0.smithi076.stdout:1804: finishing rollback tid 0 to smithi07694878-23 2021-03-15T22:26:37.158 INFO:tasks.rados.rados.0.smithi076.stdout:1804: finishing rollback tid 1 to smithi07694878-23 2021-03-15T22:26:37.158 INFO:tasks.rados.rados.0.smithi076.stdout:1804: finishing rollback tid 2 to smithi07694878-23 2021-03-15T22:26:37.159 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 804 (ObjNum 578 snap 180 seq_num 578) dirty exists 2021-03-15T22:26:37.159 INFO:tasks.rados.rados.0.smithi076.stdout:1804: done (2 left) 2021-03-15T22:26:37.159 INFO:tasks.rados.rados.0.smithi076.stdout:1807: read oid 35 snap -1 2021-03-15T22:26:37.159 INFO:tasks.rados.rados.0.smithi076.stdout:1807: expect (ObjNum 629 snap 198 seq_num 629) 2021-03-15T22:26:37.160 INFO:tasks.rados.rados.0.smithi076.stdout:1808: rollback oid 22 current snap is 200 2021-03-15T22:26:37.160 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 22 to 191 2021-03-15T22:26:37.160 INFO:tasks.rados.rados.0.smithi076.stdout:1809: copy_from oid 19 from oid 42 current snap is 200 2021-03-15T22:26:37.160 INFO:tasks.rados.rados.0.smithi076.stdout:1810: delete oid 33 current snap is 200 2021-03-15T22:26:37.161 INFO:tasks.rados.rados.0.smithi076.stdout:1805: finishing write tid 1 to smithi07694878-3 2021-03-15T22:26:37.161 INFO:tasks.rados.rados.0.smithi076.stdout:1805: finishing write tid 2 to smithi07694878-3 2021-03-15T22:26:37.162 INFO:tasks.rados.rados.0.smithi076.stdout:1810: done (5 left) 2021-03-15T22:26:37.162 INFO:tasks.rados.rados.0.smithi076.stdout:1811: write (excl) oid 13 current snap is 200 2021-03-15T22:26:37.162 INFO:tasks.rados.rados.0.smithi076.stdout:1811: seq_num 634 ranges {655638=565165,1900172=421542} 2021-03-15T22:26:37.168 INFO:tasks.rados.rados.0.smithi076.stdout:1811: writing smithi07694878-13 from 655638 to 1220803 tid 1 2021-03-15T22:26:37.175 INFO:tasks.rados.rados.0.smithi076.stdout:1811: writing smithi07694878-13 from 1900172 to 2321714 tid 2 2021-03-15T22:26:37.175 INFO:tasks.rados.rados.0.smithi076.stdout:1812: read oid 7 snap -1 2021-03-15T22:26:37.175 INFO:tasks.rados.rados.0.smithi076.stdout:1812: expect (ObjNum 614 snap 190 seq_num 614) 2021-03-15T22:26:37.176 INFO:tasks.rados.rados.0.smithi076.stdout:1813: read oid 40 snap -1 2021-03-15T22:26:37.176 INFO:tasks.rados.rados.0.smithi076.stdout:1813: expect deleted 2021-03-15T22:26:37.176 INFO:tasks.rados.rados.0.smithi076.stdout:1814: delete oid 31 current snap is 200 2021-03-15T22:26:37.176 INFO:tasks.rados.rados.0.smithi076.stdout:1808: finishing rollback tid 1 to smithi07694878-22 2021-03-15T22:26:37.177 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 921 (ObjNum 261778096 snap 21896 seq_num 1902431440) dirty dne 2021-03-15T22:26:37.177 INFO:tasks.rados.rados.0.smithi076.stdout:1805: finishing write tid 3 to smithi07694878-3 2021-03-15T22:26:37.177 INFO:tasks.rados.rados.0.smithi076.stdout:1805: finishing write tid 4 to smithi07694878-3 2021-03-15T22:26:37.177 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 792 (ObjNum 632 snap 200 seq_num 632) dirty exists 2021-03-15T22:26:37.178 INFO:tasks.rados.rados.0.smithi076.stdout:1805: left oid 3 (ObjNum 632 snap 200 seq_num 632) 2021-03-15T22:26:37.178 INFO:tasks.rados.rados.0.smithi076.stdout:1806: finishing rollback tid 0 to smithi07694878-45 2021-03-15T22:26:37.178 INFO:tasks.rados.rados.0.smithi076.stdout:1807: expect (ObjNum 629 snap 198 seq_num 629) 2021-03-15T22:26:37.220 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:37.247 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:37.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:37.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:37.269 INFO:tasks.rados.rados.0.smithi076.stdout:1806: finishing rollback tid 1 to smithi07694878-45 2021-03-15T22:26:37.270 INFO:tasks.rados.rados.0.smithi076.stdout:1806: finishing rollback tid 2 to smithi07694878-45 2021-03-15T22:26:37.270 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 780 (ObjNum 609 snap 189 seq_num 609) dirty exists 2021-03-15T22:26:37.270 INFO:tasks.rados.rados.0.smithi076.stdout:1809: finishing copy_from to smithi07694878-19 2021-03-15T22:26:37.270 INFO:tasks.rados.rados.0.smithi076.stdout:1809: got expected ENOENT (src dne) 2021-03-15T22:26:37.271 INFO:tasks.rados.rados.0.smithi076.stdout:1809: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:26:37.271 INFO:tasks.rados.rados.0.smithi076.stdout:1811: finishing write tid 1 to smithi07694878-13 2021-03-15T22:26:37.271 INFO:tasks.rados.rados.0.smithi076.stdout:1805: done (8 left) 2021-03-15T22:26:37.271 INFO:tasks.rados.rados.0.smithi076.stdout:1806: done (7 left) 2021-03-15T22:26:37.272 INFO:tasks.rados.rados.0.smithi076.stdout:1807: done (6 left) 2021-03-15T22:26:37.272 INFO:tasks.rados.rados.0.smithi076.stdout:1808: done (5 left) 2021-03-15T22:26:37.272 INFO:tasks.rados.rados.0.smithi076.stdout:1809: done (4 left) 2021-03-15T22:26:37.272 INFO:tasks.rados.rados.0.smithi076.stdout:1814: done (3 left) 2021-03-15T22:26:37.272 INFO:tasks.rados.rados.0.smithi076.stdout:1815: snap_remove snap 186 2021-03-15T22:26:37.349 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:37.353 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:26:37.353 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:26:37.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:37.374 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:37.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:37.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:37.503 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:37.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:37.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:37.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:37 smithi076 conmon[56514]: cluster 2021-03-15T22:26:37.131000+0000 mon.a (mon.0) 1343 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:37.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:37 smithi076 conmon[56514]: audit 2021-03-15T22:26:37.137866+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.76:0/3460594737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:26:37.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:37 smithi076 conmon[56514]: cluster 2021-03-15T22:26:37.137929+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2021-03-15T22:26:37.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:37 smithi076 conmon[56514]: audit 2021-03-15T22:26:37.209381+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.76:0/4248692772' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.3", "id": [3, 0, 6]}]: dispatch 2021-03-15T22:26:37.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:37 smithi076 conmon[62156]: cluster 2021-03-15T22:26:37. 2021-03-15T22:26:37.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:37 smithi076 conmon[62156]: 131000+0000 mon.a (mon.0) 1343 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:37.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:37 smithi076 conmon[62156]: audit 2021-03-15T22:26:37.137866+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.76:0/3460594737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:26:37.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:37 smithi076 conmon[62156]: cluster 2021-03-15T22:26:37.137929+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2021-03-15T22:26:37.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:37 smithi076 conmon[62156]: audit 2021-03-15T22:26:37.209381+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.76:0/4248692772' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.3", "id": [3, 0, 6]}]: dispatch 2021-03-15T22:26:37.606 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:37.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:37.633 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:37.634 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:37.736 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:37.760 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:37.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:37.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:37.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:37 smithi125 conmon[56739]: cluster 2021-03-15T22:26:37.131000+0000 mon.a (mon.0) 1343 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:37 smithi125 conmon[56739]: audit 2021-03-15T22:26:37.137866+0000 mon.a (mon.0) 1344 : audit [INF] from='client.? 172.21.15.76:0/3460594737' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:26:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:37 smithi125 conmon[56739]: cluster 2021-03-15T22:26:37.137929+0000 mon.a (mon.0) 1345 : cluster [DBG] osdmap e482: 8 total, 8 up, 7 in 2021-03-15T22:26:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:37 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:26:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:37 smithi125 conmon[56739]: 15T22:26:37.209381+0000 mon.a (mon.0) 1346 : audit [INF] from='client.? 172.21.15.76:0/4248692772' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.3", "id": [3, 0, 6]}]: dispatch 2021-03-15T22:26:37.862 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:37.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:37.886 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:37.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:37.987 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:38.013 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:38.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:38.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:38.116 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:38.140 INFO:tasks.rados.rados.0.smithi076.stdout:1815: done (3 left) 2021-03-15T22:26:38.140 INFO:tasks.rados.rados.0.smithi076.stdout:1816: read oid 32 snap 196 2021-03-15T22:26:38.141 INFO:tasks.rados.rados.0.smithi076.stdout:1816: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:26:38.141 INFO:tasks.rados.rados.0.smithi076.stdout:1817: snap_remove snap 198 2021-03-15T22:26:38.145 INFO:teuthology.orchestra.run.smithi076.stderr:set 3.3 pg_upmap mapping to [3,0,6] 2021-03-15T22:26:38.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:38.150 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:38.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:38.252 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:38.277 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:38.279 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:38.380 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:38.406 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:38.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:38.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:38.510 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:38.535 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:38.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:38.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:38.660 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:38.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:38.763 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:38.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:38 smithi076 conmon[56514]: cluster 2021-03-15T22:26:37.071533+0000 mgr.y (mgr.14144) 671 : cluster 2021-03-15T22:26:38.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:38 smithi076 conmon[56514]: [DBG] pgmap v849: 138 pgs: 138 active+clean; 410 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 16 op/s 2021-03-15T22:26:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:38 smithi076 conmon[56514]: audit 2021-03-15T22:26:38.137935+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? 172.21.15.76:0/4248692772' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.3", "id": [3, 0, 6]}]': finished 2021-03-15T22:26:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:38 smithi076 conmon[56514]: cluster 2021-03-15T22:26:38.138064+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2021-03-15T22:26:38.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:38 smithi076 conmon[62156]: cluster 2021-03-15T22:26:37.071533+0000 mgr.y 2021-03-15T22:26:38.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:38 smithi076 conmon[62156]: (mgr.14144) 671 : cluster [DBG] pgmap v849: 138 pgs: 138 active+clean; 410 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 16 op/s 2021-03-15T22:26:38.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:38 smithi076 conmon[62156]: audit 2021-03-15T22:26:38.137935+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? 172.21.15.76:0/4248692772' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.3", "id": [3, 0, 6]}]': finished 2021-03-15T22:26:38.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:38 smithi076 conmon[62156]: cluster 2021-03-15T22:26:38.138064+0000 mon.a (mon.0) 1348 : cluster [DBG] osdmap e483: 8 total, 8 up, 7 in 2021-03-15T22:26:38.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:38 smithi125 conmon[56739]: cluster 2021-03-15T22:26:37.071533+0000 mgr.y (mgr.14144) 671 : cluster [DBG] pgmap v849: 138 pgs: 138 active+clean; 410 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 16 op/s 2021-03-15T22:26:38.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:38 smithi125 conmon[56739]: audit 2021-03-15T22:26:38.137935+0000 mon.a (mon.0) 1347 : audit [INF] from='client.? 172.21.15.76:0/4248692772' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.3", "id": [3, 0, 6]}]': finished 2021-03-15T22:26:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:38 smithi125 conmon[56739]: cluster 2021-03-15T22:26:38.138064 2021-03-15T22:26:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:38 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1348 : cluster [DBG] 2021-03-15T22:26:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:38 smithi125 conmon[56739]: osdmap e483: 8 total, 8 up, 7 in 2021-03-15T22:26:38.790 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:38.791 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:38.892 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:38.918 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:38.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:38.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:38.969 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-15T22:26:39.021 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:39.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:39.048 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:39.048 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:39.140 INFO:tasks.rados.rados.0.smithi076.stdout:1817: done (4 left) 2021-03-15T22:26:39.141 INFO:tasks.rados.rados.0.smithi076.stdout:1818: rollback oid 3 current snap is 200 2021-03-15T22:26:39.141 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 195 2021-03-15T22:26:39.141 INFO:tasks.rados.rados.0.smithi076.stdout:1811: finishing write tid 2 to smithi07694878-13 2021-03-15T22:26:39.141 INFO:tasks.rados.rados.0.smithi076.stdout:1811: finishing write tid 3 to smithi07694878-13 2021-03-15T22:26:39.141 INFO:tasks.rados.rados.0.smithi076.stdout:1812: expect (ObjNum 614 snap 190 seq_num 614) 2021-03-15T22:26:39.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:39.175 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:39.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:39.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:39.274 INFO:tasks.rados.rados.0.smithi076.stdout:1811: finishing write tid 4 to smithi07694878-13 2021-03-15T22:26:39.274 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 443 (ObjNum 634 snap 200 seq_num 634) dirty exists 2021-03-15T22:26:39.274 INFO:tasks.rados.rados.0.smithi076.stdout:1811: left oid 13 (ObjNum 634 snap 200 seq_num 634) 2021-03-15T22:26:39.275 INFO:tasks.rados.rados.0.smithi076.stdout:1811: done (4 left) 2021-03-15T22:26:39.275 INFO:tasks.rados.rados.0.smithi076.stdout:1812: done (3 left) 2021-03-15T22:26:39.275 INFO:tasks.rados.rados.0.smithi076.stdout:1813: done (2 left) 2021-03-15T22:26:39.275 INFO:tasks.rados.rados.0.smithi076.stdout:1819: write oid 20 current snap is 200 2021-03-15T22:26:39.276 INFO:tasks.rados.rados.0.smithi076.stdout:1816: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:26:39.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:39.303 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:39.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:39.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:39.378 INFO:tasks.rados.rados.0.smithi076.stdout:1818: finishing rollback tid 0 to smithi07694878-3 2021-03-15T22:26:39.378 INFO:tasks.rados.rados.0.smithi076.stdout:1818: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:26:39.379 INFO:tasks.rados.rados.0.smithi076.stdout:1818: finishing rollback tid 2 to smithi07694878-3 2021-03-15T22:26:39.379 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 795 (ObjNum 430 snap 119 seq_num 430) dirty exists 2021-03-15T22:26:39.379 INFO:tasks.rados.rados.0.smithi076.stdout:1819: seq_num 636 ranges {689252=715663,1900783=425211} 2021-03-15T22:26:39.386 INFO:tasks.rados.rados.0.smithi076.stdout:1819: writing smithi07694878-20 from 689252 to 1404915 tid 1 2021-03-15T22:26:39.392 INFO:tasks.rados.rados.0.smithi076.stdout:1819: writing smithi07694878-20 from 1900783 to 2325994 tid 2 2021-03-15T22:26:39.392 INFO:tasks.rados.rados.0.smithi076.stdout:1816: done (2 left) 2021-03-15T22:26:39.392 INFO:tasks.rados.rados.0.smithi076.stdout:1818: done (1 left) 2021-03-15T22:26:39.392 INFO:tasks.rados.rados.0.smithi076.stdout:1820: delete oid 23 current snap is 200 2021-03-15T22:26:39.393 INFO:tasks.rados.rados.0.smithi076.stdout:1819: finishing write tid 1 to smithi07694878-20 2021-03-15T22:26:39.393 INFO:tasks.rados.rados.0.smithi076.stdout:1820: done (1 left) 2021-03-15T22:26:39.393 INFO:tasks.rados.rados.0.smithi076.stdout:1821: snap_remove snap 171 2021-03-15T22:26:39.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:39.430 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:39.431 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:39.532 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:39.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:39.558 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:39.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:39.659 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:39.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:39.685 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:39.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:39.733 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:39 smithi076 conmon[56514]: cluster 2021-03-15T22:26:38.736206+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:26:39.733 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:39 smithi076 conmon[56514]: cluster 2021-03-15T22:26:39.138150+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2021-03-15T22:26:39.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:39 smithi076 conmon[62156]: cluster 2021-03-15T22:26:38.736206+0000 osd.5 (osd.5) 85 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:26:39.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:39 smithi076 conmon[62156]: cluster 2021-03-15T22:26:39.138150+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2021-03-15T22:26:39.741 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:39.782 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:39 smithi125 conmon[56739]: cluster 2021-03-15T22:26:38.736206+0000 osd.5 (osd.5) 85 : cluster 2021-03-15T22:26:39.783 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:39 smithi125 conmon[56739]: [DBG] 5.15 scrub ok 2021-03-15T22:26:39.783 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:39 smithi125 conmon[56739]: cluster 2021-03-15T22:26:39.138150+0000 mon.a (mon.0) 1349 : cluster [DBG] osdmap e484: 8 total, 8 up, 7 in 2021-03-15T22:26:39.787 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:39.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:39.814 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:39.814 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:39.916 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:39.942 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:39.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:39.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:40.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:26:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:26:40.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:40.070 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:40.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:40.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:40.129 INFO:tasks.rados.rados.0.smithi076.stdout:1821: done (1 left) 2021-03-15T22:26:40.130 INFO:tasks.rados.rados.0.smithi076.stdout:1822: write (excl) oid 24 current snap is 200 2021-03-15T22:26:40.130 INFO:tasks.rados.rados.0.smithi076.stdout:1819: finishing write tid 2 to smithi07694878-20 2021-03-15T22:26:40.130 INFO:tasks.rados.rados.0.smithi076.stdout:1819: finishing write tid 3 to smithi07694878-20 2021-03-15T22:26:40.131 INFO:tasks.rados.rados.0.smithi076.stdout:1819: finishing write tid 4 to smithi07694878-20 2021-03-15T22:26:40.131 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 845 (ObjNum 636 snap 200 seq_num 636) dirty exists 2021-03-15T22:26:40.131 INFO:tasks.rados.rados.0.smithi076.stdout:1819: left oid 20 (ObjNum 636 snap 200 seq_num 636) 2021-03-15T22:26:40.132 INFO:tasks.rados.rados.0.smithi076.stdout:1822: seq_num 638 ranges {722866=466161,1617747=428880,2587634=135232} 2021-03-15T22:26:40.136 INFO:tasks.rados.rados.0.smithi076.stdout:1822: writing smithi07694878-24 from 722866 to 1189027 tid 1 2021-03-15T22:26:40.142 INFO:tasks.rados.rados.0.smithi076.stdout:1822: writing smithi07694878-24 from 1617747 to 2046627 tid 2 2021-03-15T22:26:40.147 INFO:tasks.rados.rados.0.smithi076.stdout:1822: writing smithi07694878-24 from 2587634 to 2722866 tid 3 2021-03-15T22:26:40.147 INFO:tasks.rados.rados.0.smithi076.stdout:1819: done (1 left) 2021-03-15T22:26:40.148 INFO:tasks.rados.rados.0.smithi076.stdout:1823: copy_from oid 29 from oid 43 current snap is 200 2021-03-15T22:26:40.148 INFO:tasks.rados.rados.0.smithi076.stdout:1824: write oid 3 current snap is 200 2021-03-15T22:26:40.148 INFO:tasks.rados.rados.0.smithi076.stdout:1824: seq_num 639 ranges {739673=657763,2076229=663444} 2021-03-15T22:26:40.155 INFO:tasks.rados.rados.0.smithi076.stdout:1824: writing smithi07694878-3 from 739673 to 1397436 tid 1 2021-03-15T22:26:40.164 INFO:tasks.rados.rados.0.smithi076.stdout:1824: writing smithi07694878-3 from 2076229 to 2739673 tid 2 2021-03-15T22:26:40.164 INFO:tasks.rados.rados.0.smithi076.stdout:1825: write oid 2 current snap is 200 2021-03-15T22:26:40.164 INFO:tasks.rados.rados.0.smithi076.stdout:1823: finishing copy_from to smithi07694878-29 2021-03-15T22:26:40.164 INFO:tasks.rados.rados.0.smithi076.stdout:1823: got expected ENOENT (src dne) 2021-03-15T22:26:40.165 INFO:tasks.rados.rados.0.smithi076.stdout:1825: seq_num 640 ranges {756480=733012,2134711=432549} 2021-03-15T22:26:40.172 INFO:tasks.rados.rados.0.smithi076.stdout:1825: writing smithi07694878-2 from 756480 to 1489492 tid 1 2021-03-15T22:26:40.172 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:40.179 INFO:tasks.rados.rados.0.smithi076.stdout:1825: writing smithi07694878-2 from 2134711 to 2567260 tid 2 2021-03-15T22:26:40.179 INFO:tasks.rados.rados.0.smithi076.stdout:1826: copy_from oid 45 from oid 32 current snap is 200 2021-03-15T22:26:40.179 INFO:tasks.rados.rados.0.smithi076.stdout:1823: finishing copy_from racing read to smithi07694878-29 2021-03-15T22:26:40.180 INFO:tasks.rados.rados.0.smithi076.stdout:1822: finishing write tid 1 to smithi07694878-24 2021-03-15T22:26:40.180 INFO:tasks.rados.rados.0.smithi076.stdout:1822: finishing write tid 2 to smithi07694878-24 2021-03-15T22:26:40.180 INFO:tasks.rados.rados.0.smithi076.stdout:1822: finishing write tid 3 to smithi07694878-24 2021-03-15T22:26:40.180 INFO:tasks.rados.rados.0.smithi076.stdout:1822: finishing write tid 4 to smithi07694878-24 2021-03-15T22:26:40.181 INFO:tasks.rados.rados.0.smithi076.stdout:1822: finishing write tid 5 to smithi07694878-24 2021-03-15T22:26:40.181 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 172 (ObjNum 638 snap 200 seq_num 638) dirty exists 2021-03-15T22:26:40.181 INFO:tasks.rados.rados.0.smithi076.stdout:1822: left oid 24 (ObjNum 638 snap 200 seq_num 638) 2021-03-15T22:26:40.181 INFO:tasks.rados.rados.0.smithi076.stdout:1822: done (4 left) 2021-03-15T22:26:40.181 INFO:tasks.rados.rados.0.smithi076.stdout:1823: done (3 left) 2021-03-15T22:26:40.182 INFO:tasks.rados.rados.0.smithi076.stdout:1827: write oid 48 current snap is 200 2021-03-15T22:26:40.182 INFO:tasks.rados.rados.0.smithi076.stdout:1824: finishing write tid 1 to smithi07694878-3 2021-03-15T22:26:40.182 INFO:tasks.rados.rados.0.smithi076.stdout:1827: seq_num 641 ranges {773287=408261,1793193=576207} 2021-03-15T22:26:40.185 INFO:tasks.rados.rados.0.smithi076.stdout:1827: writing smithi07694878-48 from 773287 to 1181548 tid 1 2021-03-15T22:26:40.192 INFO:tasks.rados.rados.0.smithi076.stdout:1827: writing smithi07694878-48 from 1793193 to 2369400 tid 2 2021-03-15T22:26:40.193 INFO:tasks.rados.rados.0.smithi076.stdout:1828: delete oid 20 current snap is 200 2021-03-15T22:26:40.193 INFO:tasks.rados.rados.0.smithi076.stdout:1824: finishing write tid 2 to smithi07694878-3 2021-03-15T22:26:40.194 INFO:tasks.rados.rados.0.smithi076.stdout:1824: finishing write tid 3 to smithi07694878-3 2021-03-15T22:26:40.194 INFO:tasks.rados.rados.0.smithi076.stdout:1824: finishing write tid 4 to smithi07694878-3 2021-03-15T22:26:40.194 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 798 (ObjNum 639 snap 200 seq_num 639) dirty exists 2021-03-15T22:26:40.194 INFO:tasks.rados.rados.0.smithi076.stdout:1824: left oid 3 (ObjNum 639 snap 200 seq_num 639) 2021-03-15T22:26:40.197 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:40.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:40.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:40.223 INFO:tasks.rados.rados.0.smithi076.stdout:1825: finishing write tid 1 to smithi07694878-2 2021-03-15T22:26:40.224 INFO:tasks.rados.rados.0.smithi076.stdout:1824: done (4 left) 2021-03-15T22:26:40.224 INFO:tasks.rados.rados.0.smithi076.stdout:1828: done (3 left) 2021-03-15T22:26:40.225 INFO:tasks.rados.rados.0.smithi076.stdout:1829: delete oid 12 current snap is 200 2021-03-15T22:26:40.264 INFO:tasks.rados.rados.0.smithi076.stdout:1825: finishing write tid 2 to smithi07694878-2 2021-03-15T22:26:40.264 INFO:tasks.rados.rados.0.smithi076.stdout:1825: finishing write tid 3 to smithi07694878-2 2021-03-15T22:26:40.264 INFO:tasks.rados.rados.0.smithi076.stdout:1825: finishing write tid 4 to smithi07694878-2 2021-03-15T22:26:40.265 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 493 (ObjNum 640 snap 200 seq_num 640) dirty exists 2021-03-15T22:26:40.265 INFO:tasks.rados.rados.0.smithi076.stdout:1825: left oid 2 (ObjNum 640 snap 200 seq_num 640) 2021-03-15T22:26:40.269 INFO:tasks.rados.rados.0.smithi076.stdout:1827: finishing write tid 1 to smithi07694878-48 2021-03-15T22:26:40.269 INFO:tasks.rados.rados.0.smithi076.stdout:1827: finishing write tid 2 to smithi07694878-48 2021-03-15T22:26:40.269 INFO:tasks.rados.rados.0.smithi076.stdout:1827: finishing write tid 3 to smithi07694878-48 2021-03-15T22:26:40.269 INFO:tasks.rados.rados.0.smithi076.stdout:1825: done (3 left) 2021-03-15T22:26:40.270 INFO:tasks.rados.rados.0.smithi076.stdout:1829: done (2 left) 2021-03-15T22:26:40.270 INFO:tasks.rados.rados.0.smithi076.stdout:1830: rollback oid 9 current snap is 200 2021-03-15T22:26:40.270 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 9 to 193 2021-03-15T22:26:40.270 INFO:tasks.rados.rados.0.smithi076.stdout:1831: read oid 2 snap -1 2021-03-15T22:26:40.271 INFO:tasks.rados.rados.0.smithi076.stdout:1831: expect (ObjNum 640 snap 200 seq_num 640) 2021-03-15T22:26:40.271 INFO:tasks.rados.rados.0.smithi076.stdout:1832: snap_create 2021-03-15T22:26:40.271 INFO:tasks.rados.rados.0.smithi076.stdout:1827: finishing write tid 4 to smithi07694878-48 2021-03-15T22:26:40.271 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 785 (ObjNum 641 snap 200 seq_num 641) dirty exists 2021-03-15T22:26:40.272 INFO:tasks.rados.rados.0.smithi076.stdout:1827: left oid 48 (ObjNum 641 snap 200 seq_num 641) 2021-03-15T22:26:40.272 INFO:tasks.rados.rados.0.smithi076.stdout:1830: finishing rollback tid 0 to smithi07694878-9 2021-03-15T22:26:40.273 INFO:tasks.rados.rados.0.smithi076.stdout:1830: finishing rollback tid 1 to smithi07694878-9 2021-03-15T22:26:40.273 INFO:tasks.rados.rados.0.smithi076.stdout:1830: finishing rollback tid 2 to smithi07694878-9 2021-03-15T22:26:40.273 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 1159 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:26:40.274 INFO:tasks.rados.rados.0.smithi076.stdout:1826: finishing copy_from to smithi07694878-45 2021-03-15T22:26:40.274 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 1200 (ObjNum 430 snap 119 seq_num 430) dirty exists 2021-03-15T22:26:40.274 INFO:tasks.rados.rados.0.smithi076.stdout:1826: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:26:40.276 INFO:tasks.rados.rados.0.smithi076.stdout:1831: expect (ObjNum 640 snap 200 seq_num 640) 2021-03-15T22:26:40.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:40.322 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:40.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:40.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:40.406 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:26:40.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:40.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:40 smithi076 conmon[56514]: cluster 2021-03-15T22:26:39.071940+0000 mgr.y ( 2021-03-15T22:26:40.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:40 smithi076 conmon[56514]: mgr.14144) 672 : cluster [DBG] pgmap v852: 138 pgs: 138 active+clean; 412 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-03-15T22:26:40.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:40 smithi076 conmon[56514]: cluster 2021-03-15T22:26:39.716906+0000 osd.5 (osd.5) 86 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:26:40.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:40 smithi076 conmon[56514]: audit 2021-03-15T22:26:39.780466+0000 mgr.y (mgr.14144) 673 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:40.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:40 smithi076 conmon[56514]: cluster 2021-03-15T22:26:40.125056+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2021-03-15T22:26:40.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:40 smithi076 conmon[62156]: cluster 2021-03-15T22:26:39.071940 2021-03-15T22:26:40.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:40 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 672 : cluster [DBG] pgmap v852: 138 pgs: 138 active+clean; 412 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-03-15T22:26:40.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:40 smithi076 conmon[62156]: cluster 2021-03-15T22:26:39.716906+0000 osd.5 (osd.5) 86 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:26:40.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:40 smithi076 conmon[62156]: audit 2021-03-15T22:26:39.780466+0000 mgr.y (mgr.14144) 673 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:40.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:40 smithi076 conmon[62156]: cluster 2021-03-15T22:26:40.125056+0000 mon.a (mon.0) 1350 : cluster [DBG] osdmap e485: 8 total, 8 up, 7 in 2021-03-15T22:26:40.448 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:40.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:40.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:40.457 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:26:40.476 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:26:40.493 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:26:40.511 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:26:40.529 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:26:40.546 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:26:40.550 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:40.564 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:26:40.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:40.573 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:40.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:40.582 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:26:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:26:40.583 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:26:40.584 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:26:40.584 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:26:40.584 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:26:40.584 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:26:40.585 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:26:40.585 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:26:40.675 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:40.693 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:40.698 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:40.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:40.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:40.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:40 smithi125 conmon[56739]: cluster 2021-03-15T22:26:39.071940+0000 mgr.y (mgr.14144) 672 2021-03-15T22:26:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:40 smithi125 conmon[56739]: : cluster [DBG] pgmap v852: 138 pgs: 138 active+clean; 412 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 1.8 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-03-15T22:26:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:40 smithi125 conmon[56739]: cluster 2021-03-15T22:26:39.716906+0000 osd.5 (osd.5) 86 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:26:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:40 smithi125 conmon[56739]: audit 2021-03-15T22:26:39.780466+0000 mgr.y (mgr.14144) 673 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:40 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:26:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:40 smithi125 conmon[56739]: T22:26:40.125056+0000 mon.a (mon.0) 1350 : cluster 2021-03-15T22:26:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:40 smithi125 conmon[56739]: [DBG] osdmap e485: 8 total, 8 up, 7 in 2021-03-15T22:26:40.801 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:40.823 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:40.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:40.825 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:40.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:40.949 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:40.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:40.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:41.052 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:41.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:41.075 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:41.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:41.138 INFO:tasks.rados.rados.0.smithi076.stdout:1826: done (4 left) 2021-03-15T22:26:41.138 INFO:tasks.rados.rados.0.smithi076.stdout:1827: done (3 left) 2021-03-15T22:26:41.139 INFO:tasks.rados.rados.0.smithi076.stdout:1830: done (2 left) 2021-03-15T22:26:41.139 INFO:tasks.rados.rados.0.smithi076.stdout:1831: done (1 left) 2021-03-15T22:26:41.139 INFO:tasks.rados.rados.0.smithi076.stdout:1832: done (0 left) 2021-03-15T22:26:41.140 INFO:tasks.rados.rados.0.smithi076.stdout:1833: read oid 49 snap 193 2021-03-15T22:26:41.140 INFO:tasks.rados.rados.0.smithi076.stdout:1833: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:26:41.140 INFO:tasks.rados.rados.0.smithi076.stdout:1834: delete oid 6 current snap is 201 2021-03-15T22:26:41.141 INFO:tasks.rados.rados.0.smithi076.stdout:1834: done (1 left) 2021-03-15T22:26:41.141 INFO:tasks.rados.rados.0.smithi076.stdout:1835: delete oid 10 current snap is 201 2021-03-15T22:26:41.146 INFO:tasks.rados.rados.0.smithi076.stdout:1835: done (1 left) 2021-03-15T22:26:41.147 INFO:tasks.rados.rados.0.smithi076.stdout:1836: rollback oid 4 current snap is 201 2021-03-15T22:26:41.147 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 4 to 196 2021-03-15T22:26:41.147 INFO:tasks.rados.rados.0.smithi076.stdout:1837: write oid 25 current snap is 201 2021-03-15T22:26:41.147 INFO:tasks.rados.rados.0.smithi076.stdout:1837: seq_num 646 ranges {457322=784506,1969250=443556} 2021-03-15T22:26:41.153 INFO:tasks.rados.rados.0.smithi076.stdout:1837: writing smithi07694878-25 from 457322 to 1241828 tid 1 2021-03-15T22:26:41.160 INFO:tasks.rados.rados.0.smithi076.stdout:1837: writing smithi07694878-25 from 1969250 to 2412806 tid 2 2021-03-15T22:26:41.160 INFO:tasks.rados.rados.0.smithi076.stdout:1838: read oid 36 snap -1 2021-03-15T22:26:41.161 INFO:tasks.rados.rados.0.smithi076.stdout:1838: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:26:41.161 INFO:tasks.rados.rados.0.smithi076.stdout:1839: write oid 30 current snap is 201 2021-03-15T22:26:41.161 INFO:tasks.rados.rados.0.smithi076.stdout:1839: seq_num 647 ranges {474129=576108,1627732=587214,2718088=156041} 2021-03-15T22:26:41.166 INFO:tasks.rados.rados.0.smithi076.stdout:1839: writing smithi07694878-30 from 474129 to 1050237 tid 1 2021-03-15T22:26:41.172 INFO:tasks.rados.rados.0.smithi076.stdout:1839: writing smithi07694878-30 from 1627732 to 2214946 tid 2 2021-03-15T22:26:41.176 INFO:tasks.rados.rados.0.smithi076.stdout:1839: writing smithi07694878-30 from 2718088 to 2874129 tid 3 2021-03-15T22:26:41.176 INFO:tasks.rados.rados.0.smithi076.stdout:1840: delete oid 27 current snap is 201 2021-03-15T22:26:41.176 INFO:tasks.rados.rados.0.smithi076.stdout:1836: finishing rollback tid 0 to smithi07694878-4 2021-03-15T22:26:41.176 INFO:tasks.rados.rados.0.smithi076.stdout:1836: finishing rollback tid 1 to smithi07694878-4 2021-03-15T22:26:41.177 INFO:tasks.rados.rados.0.smithi076.stdout:1836: finishing rollback tid 2 to smithi07694878-4 2021-03-15T22:26:41.177 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 449 (ObjNum 624 snap 196 seq_num 624) dirty exists 2021-03-15T22:26:41.177 INFO:tasks.rados.rados.0.smithi076.stdout:1833: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:26:41.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:41.200 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:41.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:41.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:41.303 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:41.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:41.327 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:41.327 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:41.342 INFO:tasks.rados.rados.0.smithi076.stdout:1837: finishing write tid 1 to smithi07694878-25 2021-03-15T22:26:41.342 INFO:tasks.rados.rados.0.smithi076.stdout:1833: done (5 left) 2021-03-15T22:26:41.343 INFO:tasks.rados.rados.0.smithi076.stdout:1836: done (4 left) 2021-03-15T22:26:41.343 INFO:tasks.rados.rados.0.smithi076.stdout:1840: done (3 left) 2021-03-15T22:26:41.343 INFO:tasks.rados.rados.0.smithi076.stdout:1841: write oid 15 current snap is 201 2021-03-15T22:26:41.343 INFO:tasks.rados.rados.0.smithi076.stdout:1837: finishing write tid 2 to smithi07694878-25 2021-03-15T22:26:41.343 INFO:tasks.rados.rados.0.smithi076.stdout:1841: seq_num 649 ranges {507743=726606,1744696=590883,2772934=134809} 2021-03-15T22:26:41.348 INFO:tasks.rados.rados.0.smithi076.stdout:1841: writing smithi07694878-15 from 507743 to 1234349 tid 1 2021-03-15T22:26:41.354 INFO:tasks.rados.rados.0.smithi076.stdout:1841: writing smithi07694878-15 from 1744696 to 2335579 tid 2 2021-03-15T22:26:41.358 INFO:tasks.rados.rados.0.smithi076.stdout:1841: writing smithi07694878-15 from 2772934 to 2907743 tid 3 2021-03-15T22:26:41.358 INFO:tasks.rados.rados.0.smithi076.stdout:1842: delete oid 47 current snap is 201 2021-03-15T22:26:41.358 INFO:tasks.rados.rados.0.smithi076.stdout:1837: finishing write tid 3 to smithi07694878-25 2021-03-15T22:26:41.358 INFO:tasks.rados.rados.0.smithi076.stdout:1837: finishing write tid 4 to smithi07694878-25 2021-03-15T22:26:41.359 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 498 (ObjNum 646 snap 201 seq_num 646) dirty exists 2021-03-15T22:26:41.359 INFO:tasks.rados.rados.0.smithi076.stdout:1837: left oid 25 (ObjNum 646 snap 201 seq_num 646) 2021-03-15T22:26:41.359 INFO:tasks.rados.rados.0.smithi076.stdout:1838: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:26:41.428 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:41.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:41.452 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:41.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:41.514 INFO:tasks.rados.rados.0.smithi076.stdout:1839: finishing write tid 1 to smithi07694878-30 2021-03-15T22:26:41.515 INFO:tasks.rados.rados.0.smithi076.stdout:1839: finishing write tid 2 to smithi07694878-30 2021-03-15T22:26:41.515 INFO:tasks.rados.rados.0.smithi076.stdout:1839: finishing write tid 3 to smithi07694878-30 2021-03-15T22:26:41.515 INFO:tasks.rados.rados.0.smithi076.stdout:1839: finishing write tid 4 to smithi07694878-30 2021-03-15T22:26:41.516 INFO:tasks.rados.rados.0.smithi076.stdout:1839: finishing write tid 5 to smithi07694878-30 2021-03-15T22:26:41.516 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 1164 (ObjNum 647 snap 201 seq_num 647) dirty exists 2021-03-15T22:26:41.516 INFO:tasks.rados.rados.0.smithi076.stdout:1839: left oid 30 (ObjNum 647 snap 201 seq_num 647) 2021-03-15T22:26:41.516 INFO:tasks.rados.rados.0.smithi076.stdout:1837: done (4 left) 2021-03-15T22:26:41.516 INFO:tasks.rados.rados.0.smithi076.stdout:1838: done (3 left) 2021-03-15T22:26:41.517 INFO:tasks.rados.rados.0.smithi076.stdout:1839: done (2 left) 2021-03-15T22:26:41.517 INFO:tasks.rados.rados.0.smithi076.stdout:1842: done (1 left) 2021-03-15T22:26:41.517 INFO:tasks.rados.rados.0.smithi076.stdout:1843: snap_create 2021-03-15T22:26:41.517 INFO:tasks.rados.rados.0.smithi076.stdout:1841: finishing write tid 1 to smithi07694878-15 2021-03-15T22:26:41.518 INFO:tasks.rados.rados.0.smithi076.stdout:1841: finishing write tid 2 to smithi07694878-15 2021-03-15T22:26:41.518 INFO:tasks.rados.rados.0.smithi076.stdout:1841: finishing write tid 3 to smithi07694878-15 2021-03-15T22:26:41.518 INFO:tasks.rados.rados.0.smithi076.stdout:1841: finishing write tid 4 to smithi07694878-15 2021-03-15T22:26:41.518 INFO:tasks.rados.rados.0.smithi076.stdout:1841: finishing write tid 5 to smithi07694878-15 2021-03-15T22:26:41.518 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 1169 (ObjNum 649 snap 201 seq_num 649) dirty exists 2021-03-15T22:26:41.519 INFO:tasks.rados.rados.0.smithi076.stdout:1841: left oid 15 (ObjNum 649 snap 201 seq_num 649) 2021-03-15T22:26:41.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[56514]: cluster 2021-03-15T22:26:39.963062+0000 osd.4 (osd.4) 99 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:26:41.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[56514]: cluster 2021-03-15T22:26:40.387915+0000 osd.3 (osd.3) 80 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:41.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[56514]: cluster 2021-03-15T22:26:40.388914+0000 osd.3 (osd.3) 81 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:41.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[56514]: cluster 2021-03-15T22:26:40.425541+0000 osd.2 (osd.2) 79 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:26:41.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[56514]: audit 2021-03-15T22:26:41.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[56514]: 2021-03-15T22:26:40.432096+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.76:0/3331506352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:41.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[56514]: cluster 2021-03-15T22:26:41.134779+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2021-03-15T22:26:41.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[62156]: cluster 2021-03-15T22:26:39.963062+0000 osd.4 (osd.4) 99 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:26:41.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[62156]: cluster 2021-03-15T22:26:40.387915+0000 osd.3 (osd.3) 80 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:41.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[62156]: cluster 2021-03-15T22:26:40.388914+0000 osd.3 (osd.3) 81 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:41.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[62156]: cluster 2021-03-15T22:26:40.425541+0000 osd.2 (osd.2) 79 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:26:41.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[62156]: audit 2021-03-15T22:26:40.432096+0000 mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.76:0/3331506352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:41.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:41 smithi076 conmon[62156]: cluster 2021-03-15T22:26:41.134779+0000 mon.a (mon.0) 1351 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2021-03-15T22:26:41.553 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:41.577 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:41.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:41.578 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:41.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:41.703 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:41.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:41.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:41.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: cluster 2021-03-15T22:26:39.963062+0000 osd.4 (osd.4) 99 : 2021-03-15T22:26:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: cluster [DBG] 6.1a scrub ok 2021-03-15T22:26:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: cluster 2021-03-15T22:26:40.387915+0000 osd.3 (osd.3) 80 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: cluster 2021-03-15T22:26:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: 2021-03-15T22:26:40.388914+0000 osd.3 (osd.3) 81 : cluster [DBG] 2021-03-15T22:26:41.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: purged_snaps scrub ok 2021-03-15T22:26:41.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: cluster 2021-03-15T22:26 2021-03-15T22:26:41.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: :40.425541+0000 osd.2 (osd.2) 2021-03-15T22:26:41.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: 79 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:26:41.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:26:41.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: -15T22:26:40.432096+0000 2021-03-15T22:26:41.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: mon.c (mon.1) 42 : audit [DBG] from='client.? 172.21.15.76:0/3331506352' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:41.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: 2021-03-15T22:26:41.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: cluster 2021-03-15T22:26:41.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: 2021-03-15T 2021-03-15T22:26:41.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: 22:26:41.134779+0000 mon.a (mon 2021-03-15T22:26:41.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:41 smithi125 conmon[56739]: .0) 1351 : cluster [DBG] osdmap e486: 8 total, 8 up, 7 in 2021-03-15T22:26:41.806 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:41.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:41.832 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:41.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:41.933 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:41.960 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:41.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:41.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:42.063 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:42.089 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:42.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:42.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:42.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:42.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:42.217 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:42.217 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:42.319 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:42.334 INFO:tasks.rados.rados.0.smithi076.stdout:1841: done (1 left) 2021-03-15T22:26:42.335 INFO:tasks.rados.rados.0.smithi076.stdout:1843: done (0 left) 2021-03-15T22:26:42.335 INFO:tasks.rados.rados.0.smithi076.stdout:1844: write (excl) oid 5 current snap is 202 2021-03-15T22:26:42.335 INFO:tasks.rados.rados.0.smithi076.stdout:1844: seq_num 651 ranges {541357=477104,1745307=594552} 2021-03-15T22:26:42.340 INFO:tasks.rados.rados.0.smithi076.stdout:1844: writing smithi07694878-5 from 541357 to 1018461 tid 1 2021-03-15T22:26:42.347 INFO:tasks.rados.rados.0.smithi076.stdout:1844: writing smithi07694878-5 from 1745307 to 2339859 tid 2 2021-03-15T22:26:42.347 INFO:tasks.rados.rados.0.smithi076.stdout:1845: read oid 43 snap -1 2021-03-15T22:26:42.348 INFO:tasks.rados.rados.0.smithi076.stdout:1845: expect deleted 2021-03-15T22:26:42.348 INFO:tasks.rados.rados.0.smithi076.stdout:1846: copy_from oid 21 from oid 24 current snap is 202 2021-03-15T22:26:42.348 INFO:tasks.rados.rados.0.smithi076.stdout:1844: finishing write tid 1 to smithi07694878-5 2021-03-15T22:26:42.348 INFO:tasks.rados.rados.0.smithi076.stdout:1847: snap_create 2021-03-15T22:26:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:42.350 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:42.350 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:42.352 INFO:tasks.rados.rados.0.smithi076.stdout:1844: finishing write tid 2 to smithi07694878-5 2021-03-15T22:26:42.352 INFO:tasks.rados.rados.0.smithi076.stdout:1844: finishing write tid 3 to smithi07694878-5 2021-03-15T22:26:42.352 INFO:tasks.rados.rados.0.smithi076.stdout:1844: finishing write tid 4 to smithi07694878-5 2021-03-15T22:26:42.353 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 601 (ObjNum 651 snap 202 seq_num 651) dirty exists 2021-03-15T22:26:42.353 INFO:tasks.rados.rados.0.smithi076.stdout:1844: left oid 5 (ObjNum 651 snap 202 seq_num 651) 2021-03-15T22:26:42.368 INFO:tasks.rados.rados.0.smithi076.stdout:1846: finishing copy_from to smithi07694878-21 2021-03-15T22:26:42.369 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 172 (ObjNum 638 snap 200 seq_num 638) dirty exists 2021-03-15T22:26:42.369 INFO:tasks.rados.rados.0.smithi076.stdout:1846: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:26:42.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:42.478 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:42.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:42.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:42.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:42.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:42.605 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:42.606 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:42.629 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:42 smithi076 conmon[56514]: cluster 2021-03-15T22:26:41.072301+0000 mgr.y (mgr 2021-03-15T22:26:42.629 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:42 smithi076 conmon[56514]: .14144) 674 : cluster [DBG] pgmap v855: 138 pgs: 138 active+clean; 414 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 525 KiB/s wr, 6 op/s 2021-03-15T22:26:42.630 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:42 smithi076 conmon[56514]: cluster 2021-03-15T22:26:41.428993+0000 osd.1 (osd.1) 93 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:26:42.630 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:42 smithi076 conmon[56514]: cluster 2021-03-15T22:26:41.736462+0000 osd.5 (osd.5) 87 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:26:42.630 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:42 smithi076 conmon[62156]: cluster 2021-03-15T22:26:41.072301 2021-03-15T22:26:42.631 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:42 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 674 : cluster [DBG] pgmap v855: 138 pgs: 138 active+clean; 414 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 525 KiB/s wr, 6 op/s 2021-03-15T22:26:42.631 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:42 smithi076 conmon[62156]: cluster 2021-03-15T22:26: 2021-03-15T22:26:42.631 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:42 smithi076 conmon[62156]: 41.428993+0000 osd.1 (osd.1) 93 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:26:42.631 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:42 smithi076 conmon[62156]: cluster 2021-03-15T22:26:41.736462+0000 osd.5 (osd 2021-03-15T22:26:42.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:42 smithi076 conmon[62156]: .5) 87 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:26:42.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:42.733 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:42.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:42.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:42 smithi125 conmon[56739]: cluster 2021-03-15T22:26:41.072301+0000 mgr.y (mgr.14144) 674 : cluster [DBG] 2021-03-15T22:26:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:42 smithi125 conmon[56739]: pgmap v855: 138 pgs: 138 active+clean; 414 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 2.1 MiB/s rd, 525 KiB/s wr, 6 op/s 2021-03-15T22:26:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:42 smithi125 conmon[56739]: cluster 2021-03-15T22:26:41.428993+0000 osd.1 (osd.1) 93 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:26:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:42 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:26:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:42 smithi125 conmon[56739]: :26:41.736462+0000 osd.5 (osd.5) 87 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:26:42.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:42.837 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:26:42.838 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:26:42.838 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:26:42.838 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:26:42.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:42.860 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:42.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:42.962 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:42.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:42.988 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:42.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:43.090 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:43.113 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:43.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:43.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:43.216 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:43.243 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:43.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:43.244 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:43.341 INFO:tasks.rados.rados.0.smithi076.stdout:1844: done (3 left) 2021-03-15T22:26:43.341 INFO:tasks.rados.rados.0.smithi076.stdout:1845: done (2 left) 2021-03-15T22:26:43.342 INFO:tasks.rados.rados.0.smithi076.stdout:1846: done (1 left) 2021-03-15T22:26:43.342 INFO:tasks.rados.rados.0.smithi076.stdout:1847: done (0 left) 2021-03-15T22:26:43.342 INFO:tasks.rados.rados.0.smithi076.stdout:1848: snap_create 2021-03-15T22:26:43.346 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:43.372 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:43.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:43.373 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:43.475 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:43.497 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:43.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:43.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:43.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:43.626 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:43.628 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:43.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:43.755 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:43.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:43.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[56514]: cluster 2021-03-15T22:26:41.924272+0000 osd.4 (osd.4) 100 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:26:43.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[56514]: cluster 2021-03-15T22:26:42.075262+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.c scrub ok 2021-03-15T22:26:43.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[56514]: cluster 2021-03-15T22:26:42.331541+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2021-03-15T22:26:43.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[56514]: cluster 2021-03-15T22:26:42.456922+0000 osd.1 (osd.1) 94 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:26:43.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[56514]: cluster 2021-03-15T22:26:42.479450+0000 osd.3 (osd.3) 82 : cluster [DBG] 7.f scrub ok 2021-03-15T22:26:43.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[62156]: cluster 2021-03-15T22:26:41.924272 2021-03-15T22:26:43.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 100 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:26:43.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[62156]: cluster 2021-03-15T22:26:42.075262+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.c scrub ok 2021-03-15T22:26:43.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[62156]: cluster 2021-03-15T22:26:42.331541+0000 mon.a (mon.0) 1352 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2021-03-15T22:26:43.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:26:43.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[62156]: T22:26:42.456922+0000 osd.1 (osd.1) 94 : cluster 2021-03-15T22:26:43.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[62156]: [DBG] 4.12 scrub ok 2021-03-15T22:26:43.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:26:43.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[62156]: 03-15T22:26:42.479450+0000 osd.3 (osd.3) 82 : cluster [DBG] 2021-03-15T22:26:43.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:43 smithi076 conmon[62156]: 7.f scrub ok 2021-03-15T22:26:43.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: cluster 2021-03-15T22:26:41.924272+0000 osd.4 (osd.4) 100 2021-03-15T22:26:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: : cluster [DBG] 4.13 scrub ok 2021-03-15T22:26:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: cluster 2021-03-15T22:26:42.075262+0000 osd.6 (osd.6) 77 : cluster [DBG] 7.c scrub ok 2021-03-15T22:26:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: cluster 2021-03-15T22:26:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: 2021-03-15T22:26 2021-03-15T22:26:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: :42.331541+0000 mon.a (mon.0) 2021-03-15T22:26:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: 1352 : cluster [DBG] osdmap e487: 8 total, 8 up, 7 in 2021-03-15T22:26:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:26:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: -03-15T22:26:42 2021-03-15T22:26:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: .456922+0000 osd.1 (osd.1) 94 2021-03-15T22:26:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: : cluster [DBG] 4.12 scrub ok 2021-03-15T22:26:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: cluster 2021-03-15T22:26:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: 2021-03-15T22:26 2021-03-15T22:26:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: :42.479450+0000 osd.3 (osd.3) 2021-03-15T22:26:43.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:43 smithi125 conmon[56739]: 82 : cluster [DBG] 7.f scrub ok 2021-03-15T22:26:43.794 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:26:43.839 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-15T22:26:43.858 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:43.901 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:43.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:43.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:44.004 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:26:44.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:44.029 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:44.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:26:44.090 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:44.132 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:44.153 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:26:44.154 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:26:44.154 INFO:tasks.thrashosds.thrasher:Killing osd 2, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2021-03-15T22:26:44.154 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2021-03-15T22:26:44.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.2 2021-03-15T22:26:44.158 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:44.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:44.159 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:44.216 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:44.260 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:44.285 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:44.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:44.287 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:44.346 INFO:tasks.rados.rados.0.smithi076.stdout:1848: done (0 left) 2021-03-15T22:26:44.347 INFO:tasks.rados.rados.0.smithi076.stdout:1849: snap_remove snap 197 2021-03-15T22:26:44.349 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:26:44.388 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:44.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:44.423 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:44.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:44.511 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:44 smithi076 systemd[1]: Stopping Ceph osd.2 for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:26:44.511 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[72291]: debug 2021-03-15T22:26:44.381+0000 7fb9539ed700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-03-15T22:26:44.511 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[72291]: debug 2021-03-15T22:26:44.381+0000 7fb9539ed700 -1 osd.2 489 *** Got signal Terminated *** 2021-03-15T22:26:44.511 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[72291]: debug 2021-03-15T22:26:44.381+0000 7fb9539ed700 -1 osd.2 489 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-03-15T22:26:44.512 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[56514]: cluster 2021-03-15T22:26:43.072829+0000 mgr.y (mgr. 2021-03-15T22:26:44.512 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[56514]: 14144) 675 : cluster [DBG] pgmap v858: 138 pgs: 138 active+clean; 423 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 26 op/s 2021-03-15T22:26:44.512 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[56514]: cluster 2021-03-15T22:26:43.337790+0000 mon.a (mon.0) 1353 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2021-03-15T22:26:44.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[56514]: audit 2021-03-15T22:26:43.368891+0000 mon.a (mon.0) 1354 : audit [INF] from='client.? 172.21.15.76:0/2711545224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:26:44.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[56514]: cluster 2021-03-15T22:26:43.510083+0000 osd.2 (osd.2) 80 : cluster [DBG] 4.b scrub ok 2021-03-15T22:26:44.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[56514]: cluster 2021-03-15T22:26:43.725201+0000 osd.5 (osd.5) 88 : cluster [DBG] 5.f scrub ok 2021-03-15T22:26:44.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[62156]: cluster 2021-03-15T22:26:43. 2021-03-15T22:26:44.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[62156]: 072829+0000 mgr.y (mgr.14144) 675 : cluster [DBG] pgmap v858: 138 pgs: 138 active+clean; 423 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 26 op/s 2021-03-15T22:26:44.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[62156]: cluster 2021-03-15T22:26:43.337790+0000 mon.a (mon.0) 1353 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2021-03-15T22:26:44.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[62156]: audit 2021-03-15T22:26:43.368891+0000 mon.a (mon.0) 1354 : audit [INF] from='client.? 172.21.15.76:0/2711545224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:26:44.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[62156]: cluster 2021-03-15T22:26:43.510083+0000 osd.2 (osd.2) 80 : cluster [DBG] 4.b scrub ok 2021-03-15T22:26:44.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:44 smithi076 conmon[62156]: cluster 2021-03-15T22:26:43.725201+0000 osd.5 (osd.5) 88 : cluster [DBG] 5.f scrub ok 2021-03-15T22:26:44.524 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:44.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:44.553 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:44.553 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:44.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:44.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:44.683 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:44.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:44.786 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:44 smithi125 conmon[56739]: cluster 2021-03-15T22:26:43.072829+0000 mgr.y (mgr.14144) 675 : cluster [DBG] pgmap v858: 138 pgs: 138 active+clean; 423 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.2 MiB/s wr, 26 op/s 2021-03-15T22:26:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:44 smithi125 conmon[56739]: cluster 2021-03-15T22:26: 2021-03-15T22:26:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:44 smithi125 conmon[56739]: 43.337790+0000 mon.a (mon.0) 1353 : cluster [DBG] osdmap e488: 8 total, 8 up, 7 in 2021-03-15T22:26:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:44 smithi125 conmon[56739]: audit 2021-03-15T22:26:43.368891+0000 mon.a (mon.0) 1354 : audit [INF] from='client.? 172.21.15.76:0/2711545224' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:26:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:44 smithi125 conmon[56739]: cluster 2021-03-15T22:26:43.510083+0000 osd.2 (osd. 2021-03-15T22:26:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:44 smithi125 conmon[56739]: 2) 80 : cluster [DBG] 4.b scrub ok 2021-03-15T22:26:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:44 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:26:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:44 smithi125 conmon[56739]: 26:43.725201+0000 osd.5 (osd.5) 2021-03-15T22:26:44.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:44 smithi125 conmon[56739]: 88 : cluster [DBG] 5.f scrub ok 2021-03-15T22:26:44.816 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:44.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:44.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:44.919 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:44.943 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:44.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:44.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:45.017 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:44 smithi076 podman[176661]: 2021-03-15 22:26:44.581829347 +0000 UTC m=+0.383393427 container stop e744467cc6bb7d757a920f37a3b7d7aaa08cca1161401a5fd29e6df34eac88dd (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, maintainer=Dimitri Savineau , io.buildah.version=1.18.0, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0) 2021-03-15T22:26:45.017 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:44 smithi076 podman[176661]: 2021-03-15 22:26:44.748895303 +0000 UTC m=+0.550459500 container died e744467cc6bb7d757a920f37a3b7d7aaa08cca1161401a5fd29e6df34eac88dd (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2) 2021-03-15T22:26:45.046 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:45.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:45.070 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:45.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:45.133 INFO:tasks.rados.rados.0.smithi076.stdout:1849: done (0 left) 2021-03-15T22:26:45.133 INFO:tasks.rados.rados.0.smithi076.stdout:1850: snap_remove snap 193 2021-03-15T22:26:45.171 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:45.197 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:45.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:45.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:45.299 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:45.322 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:45.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:45.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:45.424 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:45.447 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:45.449 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:45.550 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:45.575 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:45.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:45.576 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:45.649 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:45 smithi076 podman[176661]: e744467cc6bb7d757a920f37a3b7d7aaa08cca1161401a5fd29e6df34eac88dd 2021-03-15T22:26:45.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:44.107200+0000 osd.6 (osd.6) 78 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:26:45.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:44.334676+0000 mon.a (mon.0) 1355 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:26:45.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:44.334720+0000 mon.a (mon.0) 1356 : cluster [INF] Cluster is now healthy 2021-03-15T22:26:45.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: audit 2021-03-15T22:26:44.344160+0000 mon.a (mon.0) 1357 : audit [INF] from='client.? 172.21.15.76:0/2711545224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:26:45.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:44.344215+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2021-03-15T22:26:45.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:44.407041+0000 mon.a (mon.0) 1359 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:44.407077+0000 2021-03-15T22:26:45.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: mon.a (mon.0) 1360 : cluster [INF] osd.2 failed (root=default,host=smithi076) (connection refused reported by osd.0) 2021-03-15T22:26:45.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:44.407181+0000 mon.a (mon.0) 1361 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:26:45.652 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: :26:44.407411+0000 mon.a (mon.0) 1362 : cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.652 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26:45.652 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:26:45.652 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 22:26:44.407507+0000 mon.a ( 2021-03-15T22:26:45.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: mon.0) 1363 : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:26:45.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 26:44.408100+0000 mon.a (mon.0 2021-03-15T22:26:45.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: ) 1364 : cluster [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26 2021-03-15T22:26:45.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: :44.408267+0000 mon.a (mon.0 2021-03-15T22:26:45.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: ) 1365 : cluster [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:26:45.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 26:44.408552+0000 mon.a (mon.0 2021-03-15T22:26:45.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: ) 1366 : cluster [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26 2021-03-15T22:26:45.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: :44.408715+0000 mon.a ( 2021-03-15T22:26:45.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: mon.0) 1367 : cluster [DBG] 2021-03-15T22:26:45.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:26:45.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: -03-15T22:26: 2021-03-15T22:26:45.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 44.417604+0000 osd.3 (osd.3) 83 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:26:45.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:26:45.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: -03-15T22:26:44.606005+0000 mon.a 2021-03-15T22:26:45.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: (mon.0) 1368 : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:26:45.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 22:26:44.606219+0000 mon.a (mon.0) 1369 : cluster 2021-03-15T22:26:45.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: [DBG] osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26:44.606400+0000 mon.a (mon.0 2021-03-15T22:26:45.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: ) 1370 : cluster [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26 2021-03-15T22:26:45.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26:44.107200+0000 osd.6 (osd.6 2021-03-15T22:26:45.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: ) 78 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:26:45.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26:44. 2021-03-15T22:26:45.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 334676+0000 mon.a (mon.0) 1355 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:26:45.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26:44.334720+0000 mon.a ( 2021-03-15T22:26:45.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: mon.0) 1356 : cluster [INF] Cluster is now healthy 2021-03-15T22:26:45.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: audit 2021-03-15T22:26:44.344160+0000 2021-03-15T22:26:45.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: mon.a (mon.0) 1357 : audit [INF] from='client.? 172.21.15.76:0/2711545224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:26:45.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26:44 2021-03-15T22:26:45.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: .344215+0000 mon.a (mon.0) 1358 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2021-03-15T22:26:45.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:26:45.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: :26:44.407041+0000 mon.a (mon.0) 1359 : cluster 2021-03-15T22:26:45.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:26:45.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 03-15T22:26:44.407077+0000 mon.a (mon.0) 1360 : cluster [INF] osd.2 failed (root=default,host=smithi076) (connection refused reported by osd.0) 2021-03-15T22:26:45.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 2021-03-15T22:26:45.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26:44.407181+0000 mon.a (mon. 2021-03-15T22:26:45.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 0) 1361 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:26:45.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 15T22:26:44.407411+0000 mon.a (mon.0) 1362 : cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26: 2021-03-15T22:26:45.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 44.407507+0000 mon.a (mon.0) 1363 : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:26:45.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: :26:44.408100+0000 mon.a (mon.0) 1364 : cluster 2021-03-15T22:26:45.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:26:45.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: -15T22:26:44.408267+0000 mon.a (mon.0) 1365 : cluster [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:26:45.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: :26:44.408552+0000 mon.a (mon.0) 1366 : cluster [DBG] 2021-03-15T22:26:45.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:26:45.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: -15T22:26:44.408715+0000 mon.a (mon.0) 1367 : cluster [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:26:45.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: :26:44.417604+0000 osd.3 (osd.3) 83 : cluster [DBG] 2021-03-15T22:26:45.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 4.1d scrub ok 2021-03-15T22:26:45.668 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:26:45.668 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: -15T22:26:44.606005+0000 mon.a (mon.0) 1368 : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.668 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 2021-03-15T22:26:45.668 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:26:45.668 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 22:26:44.606219+0000 mon.a (mon.0) 1369 : cluster [DBG] osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26 2021-03-15T22:26:45.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: :44.606400+0000 mon.a (mon.0) 1370 : cluster [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 2021-03-15T22:26:45.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:26:45.670 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: :44.606581+0000 mon.a (mon.0) 1371 : cluster [DBG] 2021-03-15T22:26:45.670 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:26:45.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: -03-15T22:26:44.606770+0000 mon.a (mon.0 2021-03-15T22:26:45.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: ) 1372 : cluster [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:26:45.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: T22:26:44.606905+0000 mon.a (mon.0) 2021-03-15T22:26:45.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 1373 : cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26 2021-03-15T22:26:45.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: :44.607054+0000 mon.a (mon.0) 1374 : cluster [DBG] 2021-03-15T22:26:45.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26:44.607180+0000 mon.a (mon.0) 1375 : 2021-03-15T22:26:45.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26:44.607952+0000 mon.a (mon 2021-03-15T22:26:45.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: .0) 1376 : cluster [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:26:45.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 26:44.608039+0000 mon.a (mon.0) 1377 : cluster 2021-03-15T22:26:45.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26:44.608127+0000 mon.a (mon.0) 2021-03-15T22:26:45.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 1378 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:26:45.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: T22:26:44.608257+0000 mon.a (mon.0) 2021-03-15T22:26:45.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 1379 : cluster [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:26:45.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 26:44.608425+0000 mon.a (mon.0) 1380 : cluster 2021-03-15T22:26:45.677 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.677 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.677 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26:44.608588+0000 mon.a (mon.0 2021-03-15T22:26:45.677 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: ) 1381 : cluster [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.677 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26 2021-03-15T22:26:45.678 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: :44.759737+0000 osd.5 (osd.5) 89 : cluster [DBG] 2021-03-15T22:26:45.678 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 7.0 scrub ok 2021-03-15T22:26:45.678 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:26:45.678 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: -15T22:26:45.006940+0000 mon.a (mon.0) 1382 : cluster 2021-03-15T22:26:45.679 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.679 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.679 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26:45.007085+0000 mon.a ( 2021-03-15T22:26:45.679 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: mon.0) 1383 : cluster [DBG] osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:26:45.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: -03-15T22:26:45.007206+0000 2021-03-15T22:26:45.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: mon.a (mon.0) 1384 : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:26:45.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: T22:26:45.007275+0000 mon.a (mon.0) 2021-03-15T22:26:45.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 1385 : cluster [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26: 2021-03-15T22:26:45.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 45.007436+0000 mon.a (mon.0) 1386 : cluster [DBG] osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26:45.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:26:45.682 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: -03-15T22:26:45.007912+0000 mon.a (mon.0 2021-03-15T22:26:45.682 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: ) 1387 : cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.682 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.008085 2021-03-15T22:26:45.682 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1388 : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:26:45.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 03-15T22:26:45.008245+0000 mon.a ( 2021-03-15T22:26:45.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: mon.0) 1389 : cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:26:45.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 22:26:45.008420+0000 mon.a (mon. 2021-03-15T22:26:45.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 0) 1390 : cluster [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:26:45.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: -15T22:26:45.008959+0000 mon.a (mon.0) 2021-03-15T22:26:45.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 1391 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26 2021-03-15T22:26:45.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: :45.009087+0000 mon.a (mon.0) 1392 : cluster [DBG] 2021-03-15T22:26:45.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26:45.009242+0000 mon.a (mon.0) 1393 2021-03-15T22:26:45.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: : cluster [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26:45.009366+0000 mon.a (mon.0) 1394 : 2021-03-15T22:26:45.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 2021-03-15T22:26:45.009459+0000 mon.a ( 2021-03-15T22:26:45.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: mon.0) 1395 : cluster [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:26:45.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: -03-15T22:26:45.086422+0000 2021-03-15T22:26:45.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: mon.a (mon.0) 1396 : cluster [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:26:45.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: T22:26:45.122277+0000 mon.a (mon.0) 2021-03-15T22:26:45.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 1397 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-03-15T22:26:45.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: cluster 2021-03-15T22:26: 2021-03-15T22:26:45.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[56514]: 45.126712+0000 mon.a (mon.0) 1398 : cluster [DBG] osdmap e490: 8 total, 7 up, 7 in 2021-03-15T22:26:45.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 22:26:44.606581+0000 mon.a (mon.0) 1371 : cluster [DBG] osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26 2021-03-15T22:26:45.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: :44.606770+0000 mon.a (mon.0) 1372 : cluster [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 2021-03-15T22:26:45.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:26:45.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 22:26:44.606905+0000 mon.a (mon.0) 1373 : cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26 2021-03-15T22:26:45.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: :44.607054+0000 mon.a (mon.0) 1374 : cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.692 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 2021-03-15T22:26:45.692 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:26:45.692 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 22:26:44.607180+0000 mon.a (mon.0) 1375 : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.692 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26 2021-03-15T22:26:45.693 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: :44.607952+0000 mon.a (mon.0) 1376 : cluster [DBG] 2021-03-15T22:26:45.693 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.693 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:26:45.693 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 15T22:26:44.608039+0000 mon.a (mon.0) 1377 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.693 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:26:45.694 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: T22:26:44.608127+0000 mon.a (mon.0) 1378 : 2021-03-15T22:26:45.694 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.694 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26:45.694 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 2021-03-15T22:26:44.608257+0000 mon.a (mon.0) 1379 : cluster [DBG] 2021-03-15T22:26:45.694 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.695 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26:44.608425+0000 2021-03-15T22:26:45.695 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: mon.a (mon.0) 1380 : cluster [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.695 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26:45.695 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 2021-03-15T22:26:44.608588+0000 mon.a (mon.0) 1381 : cluster 2021-03-15T22:26:45.696 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.696 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:26:45.696 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 03-15T22:26:44.759737+0000 osd.5 (osd.5) 89 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:26:45.696 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 2021-03-15T22:26:45.696 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:26:45.697 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 22:26:45.006940+0000 mon.a (mon.0) 1382 : cluster [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.697 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26 2021-03-15T22:26:45.697 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: :45.007085+0000 mon.a (mon.0) 1383 : cluster [DBG] 2021-03-15T22:26:45.697 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.698 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:26:45.698 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 15T22:26:45.007206+0000 mon.a (mon.0) 1384 : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.698 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:26:45.698 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 26:45.007275+0000 mon.a (mon.0) 1385 : cluster [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.698 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 2021-03-15T22:26:45.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:26:45.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 22:26:45.007436+0000 mon.a (mon.0) 1386 : cluster [DBG] osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:26:45.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 26:45.007912+0000 mon.a (mon.0) 1387 : cluster [DBG] 2021-03-15T22:26:45.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:26:45.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: -15T22:26:45.008085+0000 mon.a (mon.0) 1388 : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:26:45.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 22:26:45.008245+0000 mon.a (mon.0) 1389 : cluster 2021-03-15T22:26:45.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:26:45.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 03-15T22:26:45.008420+0000 mon.a (mon.0) 1390 : cluster [DBG] 2021-03-15T22:26:45.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:26:45.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: T22:26:45.008959+0000 mon.a (mon.0) 1391 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:26:45.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 26:45.009087+0000 mon.a (mon.0 2021-03-15T22:26:45.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: ) 1392 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:26:45.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: -15T22:26:45.009242+0000 2021-03-15T22:26:45.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: mon.a (mon.0) 1393 : cluster [DBG] 2021-03-15T22:26:45.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26:45.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 2021-03-15T22:26:45.009366+0000 2021-03-15T22:26:45.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: mon.a (mon.0) 1394 : cluster [DBG] 2021-03-15T22:26:45.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021-03-15T22:26:45.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:26:45.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 26:45.009459+0000 mon.a 2021-03-15T22:26:45.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: (mon.0) 1395 : cluster 2021-03-15T22:26:45.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:26:45.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: -03-15T22:26:45.086422 2021-03-15T22:26:45.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1396 : cluster 2021-03-15T22:26:45.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:26:45.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: -03-15T22:26:45.122277 2021-03-15T22:26:45.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1397 2021-03-15T22:26:45.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-03-15T22:26:45.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:26:45.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: 03-15T22:26:45.126712 2021-03-15T22:26:45.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1398 : cluster [DBG] 2021-03-15T22:26:45.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:45 smithi076 conmon[62156]: osdmap e490: 8 total, 7 up, 7 in 2021-03-15T22:26:45.708 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:45.730 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:45.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:45.733 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.107200+0000 osd.6 (osd.6) 78 : cluster [DBG] 2021-03-15T22:26:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 5.11 scrub ok 2021-03-15T22:26:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.334676+0000 mon.a (mon.0) 1355 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:26:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.334720+0000 2021-03-15T22:26:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: mon.a (mon.0) 1356 : cluster [INF] Cluster is now healthy 2021-03-15T22:26:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: audit 2021-03-15T22:26:44.344160 2021-03-15T22:26:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1357 : audit [INF] from='client.? 172.21.15.76:0/2711545224' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:26:45.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.344215+0000 mon.a (mon 2021-03-15T22:26:45.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: .0) 1358 : cluster [DBG] osdmap e489: 8 total, 8 up, 7 in 2021-03-15T22:26:45.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26: 2021-03-15T22:26:45.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 44.407041+0000 mon.a (mon.0) 1359 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:45.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: -15T22:26:44.407077+0000 mon.a (mon.0) 1360 : cluster 2021-03-15T22:26:45.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: [INF] osd.2 failed (root=default,host=smithi076) (connection refused reported by osd.0) 2021-03-15T22:26:45.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:45.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: -15T22:26:44.407181+0000 mon.a (mon.0) 1361 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:45.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: -15T22:26:44.407411+0000 mon.a (mon.0) 1362 : 2021-03-15T22:26:45.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.407507+0000 2021-03-15T22:26:45.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: mon.a (mon.0) 1363 : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:26:45.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 22:26:44.408100+0000 mon.a (mon.0) 1364 : cluster [DBG] 2021-03-15T22:26:45.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.408267+0000 mon.a ( 2021-03-15T22:26:45.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: mon.0) 1365 : cluster [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:26:45.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 26:44.408552+0000 mon.a (mon.0) 1366 : cluster [DBG] 2021-03-15T22:26:45.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 2021-03-15T22:26:44.408715+0000 mon.a (mon.0) 1367 : cluster 2021-03-15T22:26:45.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.417604+0000 osd.3 ( 2021-03-15T22:26:45.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: osd.3) 83 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:26:45.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.606005+0000 mon.a (mon.0) 1368 2021-03-15T22:26:45.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.606219+0000 mon.a (mon 2021-03-15T22:26:45.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: .0) 1369 : cluster [DBG] osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44 2021-03-15T22:26:45.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: .606400+0000 mon.a (mon.0) 1370 : cluster [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:26:45.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 03-15T22:26:44.606581+0000 mon.a (mon.0) 1371 2021-03-15T22:26:45.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: : cluster [DBG] osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.606770 2021-03-15T22:26:45.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1372 : cluster [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:45.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: -15T22:26:44.606905+0000 mon.a (mon.0) 1373 : 2021-03-15T22:26:45.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.607054 2021-03-15T22:26:45.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1374 : cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:26:45.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 26:44.607180+0000 mon.a (mon.0) 1375 : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 2021-03-15T22:26:45.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:26:45.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: -03-15T22:26:44.607952+0000 mon.a (mon.0) 1376 : cluster [DBG] 2021-03-15T22:26:45.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.608039+0000 mon.a ( 2021-03-15T22:26:45.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: mon.0) 1377 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.608127+0000 2021-03-15T22:26:45.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: mon.a (mon.0) 1378 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:26:45.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: T22:26:44.608257+0000 mon.a (mon.0) 1379 : cluster 2021-03-15T22:26:45.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:44.608425+0000 2021-03-15T22:26:45.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: mon.a (mon.0) 1380 : cluster [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:26:45.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 22:26:44.608588+0000 mon.a (mon.0) 1381 : cluster [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:26:45.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: T22:26:44.759737+0000 osd.5 (osd.5) 89 : cluster 2021-03-15T22:26:45.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: [DBG] 7.0 scrub ok 2021-03-15T22:26:45.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45.006940+0000 mon.a 2021-03-15T22:26:45.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: (mon.0) 1382 : cluster [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45.007085 2021-03-15T22:26:45.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1383 : cluster [DBG] osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:45.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: -15T22:26:45.007206+0000 mon.a (mon.0) 1384 2021-03-15T22:26:45.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45. 2021-03-15T22:26:45.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 007275+0000 mon.a (mon.0) 1385 : cluster [DBG] osd.2 reported immediately failed by osd.6 2021-03-15T22:26:45.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:26:45.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 15T22:26:45.007436+0000 mon.a (mon.0) 1386 : cluster 2021-03-15T22:26:45.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: [DBG] osd.2 reported immediately failed by osd.7 2021-03-15T22:26:45.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45.007912 2021-03-15T22:26:45.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1387 : cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:45.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: -15T22:26:45.008085+0000 mon.a (mon.0) 1388 2021-03-15T22:26:45.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: : cluster [DBG] osd.2 reported immediately failed by osd.5 2021-03-15T22:26:45.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45. 2021-03-15T22:26:45.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 008245+0000 mon.a (mon.0) 1389 : cluster [DBG] osd.2 reported immediately failed by osd.4 2021-03-15T22:26:45.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:26:45.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 03-15T22:26:45.008420+0000 mon.a (mon.0) 1390 2021-03-15T22:26:45.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: : cluster [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45. 2021-03-15T22:26:45.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 008959+0000 mon.a (mon.0) 1391 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45.009087+0000 2021-03-15T22:26:45.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: mon.a (mon.0) 1392 : cluster [DBG] osd.2 reported immediately failed by osd.0 2021-03-15T22:26:45.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:45.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: -15T22:26:45.009242+0000 mon.a (mon.0) 1393 2021-03-15T22:26:45.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: : cluster [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:26:45.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: -03-15T22:26:45.009366+0000 mon.a (mon.0) 1394 : cluster [DBG] osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:26:45.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: :26:45.009459+0000 mon.a (mon.0) 1395 : cluster [DBG] 2021-03-15T22:26:45.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: osd.2 reported immediately failed by osd.1 2021-03-15T22:26:45.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 2021-03-15T22:26:45.086422+0000 mon.a (mon.0) 1396 : cluster 2021-03-15T22:26:45.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: [DBG] osd.2 reported immediately failed by osd.3 2021-03-15T22:26:45.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:26:45.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 03-15T22:26:45.122277+0000 mon.a (mon.0) 1397 : cluster [WRN] Health check failed: 1 osds down (OSD_DOWN) 2021-03-15T22:26:45.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 2021-03-15T22:26:45.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45.812 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: 2021-03-15T22:26:45.126712+0000 mon.a (mon.0) 1398 : cluster 2021-03-15T22:26:45.812 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:45 smithi125 conmon[56739]: [DBG] osdmap e490: 8 total, 7 up, 7 in 2021-03-15T22:26:45.833 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:45.856 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:45.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:45.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:45.959 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:45.985 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:45.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:45.986 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:46.087 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:46.113 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:46.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:46.114 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:46.143 INFO:tasks.rados.rados.0.smithi076.stdout:1850: done (0 left) 2021-03-15T22:26:46.144 INFO:tasks.rados.rados.0.smithi076.stdout:1851: read oid 18 snap -1 2021-03-15T22:26:46.144 INFO:tasks.rados.rados.0.smithi076.stdout:1851: expect (ObjNum 590 snap 181 seq_num 590) 2021-03-15T22:26:46.144 INFO:tasks.rados.rados.0.smithi076.stdout:1852: write oid 40 current snap is 204 2021-03-15T22:26:46.144 INFO:tasks.rados.rados.0.smithi076.stdout:1852: seq_num 652 ranges {558164=552353,1803789=454563,2855203=102961} 2021-03-15T22:26:46.150 INFO:tasks.rados.rados.0.smithi076.stdout:1852: writing smithi07694878-40 from 558164 to 1110517 tid 1 2021-03-15T22:26:46.157 INFO:tasks.rados.rados.0.smithi076.stdout:1852: writing smithi07694878-40 from 1803789 to 2258352 tid 2 2021-03-15T22:26:46.162 INFO:tasks.rados.rados.0.smithi076.stdout:1852: writing smithi07694878-40 from 2855203 to 2958164 tid 3 2021-03-15T22:26:46.162 INFO:tasks.rados.rados.0.smithi076.stdout:1853: write oid 38 current snap is 204 2021-03-15T22:26:46.162 INFO:tasks.rados.rados.0.smithi076.stdout:1853: seq_num 653 ranges {574971=627602,1862271=598221} 2021-03-15T22:26:46.168 INFO:tasks.rados.rados.0.smithi076.stdout:1853: writing smithi07694878-38 from 574971 to 1202573 tid 1 2021-03-15T22:26:46.175 INFO:tasks.rados.rados.0.smithi076.stdout:1853: writing smithi07694878-38 from 1862271 to 2460492 tid 2 2021-03-15T22:26:46.176 INFO:tasks.rados.rados.0.smithi076.stdout:1854: snap_create 2021-03-15T22:26:46.176 INFO:tasks.rados.rados.0.smithi076.stdout:1851: expect (ObjNum 590 snap 181 seq_num 590) 2021-03-15T22:26:46.215 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:46.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:46.241 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:46.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:46.343 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:46.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:46.369 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:46.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:46.463 INFO:tasks.rados.rados.0.smithi076.stdout:1852: finishing write tid 1 to smithi07694878-40 2021-03-15T22:26:46.464 INFO:tasks.rados.rados.0.smithi076.stdout:1852: finishing write tid 2 to smithi07694878-40 2021-03-15T22:26:46.464 INFO:tasks.rados.rados.0.smithi076.stdout:1852: finishing write tid 3 to smithi07694878-40 2021-03-15T22:26:46.464 INFO:tasks.rados.rados.0.smithi076.stdout:1852: finishing write tid 4 to smithi07694878-40 2021-03-15T22:26:46.464 INFO:tasks.rados.rados.0.smithi076.stdout:1852: finishing write tid 5 to smithi07694878-40 2021-03-15T22:26:46.465 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1298 (ObjNum 652 snap 204 seq_num 652) dirty exists 2021-03-15T22:26:46.465 INFO:tasks.rados.rados.0.smithi076.stdout:1852: left oid 40 (ObjNum 652 snap 204 seq_num 652) 2021-03-15T22:26:46.465 INFO:tasks.rados.rados.0.smithi076.stdout:1853: finishing write tid 1 to smithi07694878-38 2021-03-15T22:26:46.465 INFO:tasks.rados.rados.0.smithi076.stdout:1853: finishing write tid 2 to smithi07694878-38 2021-03-15T22:26:46.466 INFO:tasks.rados.rados.0.smithi076.stdout:1853: finishing write tid 3 to smithi07694878-38 2021-03-15T22:26:46.466 INFO:tasks.rados.rados.0.smithi076.stdout:1853: finishing write tid 4 to smithi07694878-38 2021-03-15T22:26:46.466 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 1005 (ObjNum 653 snap 204 seq_num 653) dirty exists 2021-03-15T22:26:46.466 INFO:tasks.rados.rados.0.smithi076.stdout:1853: left oid 38 (ObjNum 653 snap 204 seq_num 653) 2021-03-15T22:26:46.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:46.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.073400+0000 mgr.y (mgr.14144) 676 : cluster 2021-03-15T22:26:46.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[56514]: [DBG] pgmap v861: 138 pgs: 138 active+clean; 427 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2021-03-15T22:26:46.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.153410+0000 osd.7 (osd.7) 97 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:26:46.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.157111+0000 osd.6 (osd.6) 79 : cluster [DBG] 7.d scrub ok 2021-03-15T22:26:46.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.403765+0000 osd.3 (osd.3) 84 : cluster [DBG] 6.1f deep-scrub ok 2021-03-15T22:26:46.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[56514]: cluster 2021-03-15T22:26:45.500905+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:26:46.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[56514]: cluster 2021-03-15T22:26:46.140119+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e491: 8 total, 7 up, 7 in 2021-03-15T22:26:46.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[62156]: cluster 2021-03-15T22:26:45.073400+0000 mgr.y 2021-03-15T22:26:46.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[62156]: (mgr.14144) 676 : cluster [DBG] pgmap v861: 138 pgs: 138 active+clean; 427 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2021-03-15T22:26:46.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[62156]: cluster 2021-03-15T22:26:45.153410+0000 osd.7 (osd.7) 97 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:26:46.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[62156]: cluster 2021-03-15T22:26:45.157111+0000 osd.6 (osd.6) 79 : cluster [DBG] 7.d scrub ok 2021-03-15T22:26:46.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[62156]: cluster 2021-03-15T22:26:45.403765+0000 osd.3 (osd.3) 84 : cluster [DBG] 6.1f deep-scrub ok 2021-03-15T22:26:46.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[62156]: cluster 2021-03-15T22:26:45.500905+0000 osd.1 (osd.1) 95 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:26:46.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:46 smithi076 conmon[62156]: cluster 2021-03-15T22:26:46.140119+0000 mon.a (mon.0) 1399 : cluster [DBG] osdmap e491: 8 total, 7 up, 7 in 2021-03-15T22:26:46.496 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:46.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:46.497 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:46.598 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:46.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:46.623 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:46.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:46.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:46.751 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:46.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:46.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45.073400+0000 mgr.y (mgr.14144) 676 : cluster 2021-03-15T22:26:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: [DBG] pgmap v861: 138 pgs: 138 active+clean; 427 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 3.4 MiB/s wr, 27 op/s 2021-03-15T22:26:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45.153410+0000 osd.7 (osd.7) 97 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:26:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: cluster 2021-03-15T22:26:45. 2021-03-15T22:26:46.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: 157111+0000 osd.6 (osd.6) 79 : cluster 2021-03-15T22:26:46.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: [DBG] 7.d scrub ok 2021-03-15T22:26:46.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:26:46.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: :26:45.403765+0000 osd.3 (osd.3 2021-03-15T22:26:46.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: ) 84 : cluster [DBG] 6.1f deep-scrub ok 2021-03-15T22:26:46.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:26:46.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: -03-15T22:26:45.500905+0000 2021-03-15T22:26:46.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: osd.1 (osd.1) 95 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:26:46.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: 2021-03-15T22:26:46.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: cluster 2021-03-15T22:26:46.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: 2021-03-15T22:26:46.140119+0000 2021-03-15T22:26:46.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: mon.a (mon.0) 1399 : cluster [DBG] osdmap e491: 8 total, 7 up, 7 in 2021-03-15T22:26:46.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:46 smithi125 conmon[56739]: 2021-03-15T22:26:46.853 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:46.877 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:46.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:46.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:46.979 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:47.001 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:47.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:47.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:47.017 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:46 smithi076 podman[177106]: 2021-03-15 22:26:46.746636811 +0000 UTC m=+0.310175473 container create e0860ae9f3f73128b310d422405a177bdae4b56a37deb548ad88588f39bdb380 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20201204, ceph=True, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.license=GPLv2, io.buildah.version=1.18.0) 2021-03-15T22:26:47.105 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:47.127 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:47.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:47.129 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:47.142 INFO:tasks.rados.rados.0.smithi076.stdout:1851: done (3 left) 2021-03-15T22:26:47.142 INFO:tasks.rados.rados.0.smithi076.stdout:1852: done (2 left) 2021-03-15T22:26:47.143 INFO:tasks.rados.rados.0.smithi076.stdout:1853: done (1 left) 2021-03-15T22:26:47.143 INFO:tasks.rados.rados.0.smithi076.stdout:1854: done (0 left) 2021-03-15T22:26:47.143 INFO:tasks.rados.rados.0.smithi076.stdout:1855: copy_from oid 46 from oid 43 current snap is 205 2021-03-15T22:26:47.143 INFO:tasks.rados.rados.0.smithi076.stdout:1856: write (excl) oid 28 current snap is 205 2021-03-15T22:26:47.144 INFO:tasks.rados.rados.0.smithi076.stdout:1856: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2021-03-15T22:26:47.147 INFO:tasks.rados.rados.0.smithi076.stdout:1856: writing smithi07694878-28 from 591778 to 1010982 tid 1 2021-03-15T22:26:47.155 INFO:tasks.rados.rados.0.smithi076.stdout:1856: writing smithi07694878-28 from 1637106 to 2378985 tid 2 2021-03-15T22:26:47.159 INFO:tasks.rados.rados.0.smithi076.stdout:1856: writing smithi07694878-28 from 2910049 to 2991778 tid 3 2021-03-15T22:26:47.159 INFO:tasks.rados.rados.0.smithi076.stdout:1857: snap_create 2021-03-15T22:26:47.159 INFO:tasks.rados.rados.0.smithi076.stdout:1855: finishing copy_from to smithi07694878-46 2021-03-15T22:26:47.159 INFO:tasks.rados.rados.0.smithi076.stdout:1855: got expected ENOENT (src dne) 2021-03-15T22:26:47.160 INFO:tasks.rados.rados.0.smithi076.stdout:1855: finishing copy_from racing read to smithi07694878-46 2021-03-15T22:26:47.160 INFO:tasks.rados.rados.0.smithi076.stdout:1856: finishing write tid 1 to smithi07694878-28 2021-03-15T22:26:47.163 INFO:tasks.rados.rados.0.smithi076.stdout:1856: finishing write tid 2 to smithi07694878-28 2021-03-15T22:26:47.164 INFO:tasks.rados.rados.0.smithi076.stdout:1856: finishing write tid 3 to smithi07694878-28 2021-03-15T22:26:47.164 INFO:tasks.rados.rados.0.smithi076.stdout:1856: finishing write tid 4 to smithi07694878-28 2021-03-15T22:26:47.165 INFO:tasks.rados.rados.0.smithi076.stdout:1856: finishing write tid 5 to smithi07694878-28 2021-03-15T22:26:47.165 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 1303 (ObjNum 654 snap 205 seq_num 654) dirty exists 2021-03-15T22:26:47.165 INFO:tasks.rados.rados.0.smithi076.stdout:1856: left oid 28 (ObjNum 654 snap 205 seq_num 654) 2021-03-15T22:26:47.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:47.253 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:47.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:47.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:47.323 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:47 smithi076 podman[177106]: 2021-03-15 22:26:47.06389905 +0000 UTC m=+0.627437787 container init e0860ae9f3f73128b310d422405a177bdae4b56a37deb548ad88588f39bdb380 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, io.buildah.version=1.18.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git) 2021-03-15T22:26:47.356 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:47.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:47.380 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:47.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:47.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:47.504 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:47.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:47.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[62156]: cluster 2021-03-15T22:26:46.453315+0000 2021-03-15T22:26:47.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[62156]: osd.3 (osd.3) 85 : cluster [DBG] 4.0 deep-scrub ok 2021-03-15T22:26:47.599 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[62156]: cluster 2021-03-15T22:26:46.756515+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.10 deep-scrub ok 2021-03-15T22:26:47.599 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[62156]: cluster 2021-03-15T22:26:47.138932+0000 mon.a (mon.0) 1400 : cluster [DBG] osdmap e492: 8 total, 7 up, 7 in 2021-03-15T22:26:47.599 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56514]: cluster 2021-03-15T22:26:46.453315+0000 osd.3 (osd.3 2021-03-15T22:26:47.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56514]: ) 85 : cluster [DBG] 4.0 deep-scrub ok 2021-03-15T22:26:47.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56514]: cluster 2021-03-15T22:26:46.756515+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.10 deep-scrub ok 2021-03-15T22:26:47.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56514]: cluster 2021-03-15T22:26:47.138932+0000 mon.a (mon.0) 1400 : cluster [DBG] osdmap e492: 8 total, 7 up, 7 in 2021-03-15T22:26:47.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:47.628 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:47.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:47.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:47.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:47.776 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:47.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:47.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:47.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:47 smithi125 conmon[56739]: cluster 2021-03-15T22:26:46.453315+0000 osd.3 (osd.3) 85 : 2021-03-15T22:26:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:47 smithi125 conmon[56739]: cluster [DBG] 4.0 deep-scrub ok 2021-03-15T22:26:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:47 smithi125 conmon[56739]: cluster 2021-03-15T22:26:46.756515+0000 osd.5 (osd.5) 90 : cluster [DBG] 6.10 deep-scrub ok 2021-03-15T22:26:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:47 smithi125 conmon[56739]: cluster 2021-03-15T22:26:47.138932+0000 mon.a (mon.0 2021-03-15T22:26:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:47 smithi125 conmon[56739]: ) 1400 : cluster [DBG] osdmap e492: 8 total, 7 up, 7 in 2021-03-15T22:26:47.879 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:47.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:47.909 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:47.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:47.999 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:47 smithi076 podman[177106]: 2021-03-15 22:26:47.629800948 +0000 UTC m=+1.193339613 container start e0860ae9f3f73128b310d422405a177bdae4b56a37deb548ad88588f39bdb380 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2-deactivate, io.buildah.version=1.18.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, org.label-schema.build-date=20201204, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , GIT_BRANCH=HEAD, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) 2021-03-15T22:26:48.000 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:47 smithi076 podman[177106]: 2021-03-15 22:26:47.629917709 +0000 UTC m=+1.193456389 container attach e0860ae9f3f73128b310d422405a177bdae4b56a37deb548ad88588f39bdb380 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2-deactivate, io.buildah.version=1.18.0, org.label-schema.schema-version=1.0, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , org.label-schema.build-date=20201204, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, GIT_BRANCH=HEAD) 2021-03-15T22:26:48.000 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:26:47 smithi076 podman[177106]: 2021-03-15 22:26:47.708963778 +0000 UTC m=+1.272502458 container died e0860ae9f3f73128b310d422405a177bdae4b56a37deb548ad88588f39bdb380 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2-deactivate) 2021-03-15T22:26:48.011 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:48.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:48.036 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:48.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:48.070 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:26:48.071 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:26:48.071 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:26:48.071 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:26:48.072 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:26:48.072 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:26:48.072 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:26:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:26:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:26:48.073 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:26:48.074 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:26:48.074 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:26:48.074 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:26:48.075 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:26:48.075 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:26:48.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:48.148 INFO:tasks.rados.rados.0.smithi076.stdout:1855: done (2 left) 2021-03-15T22:26:48.148 INFO:tasks.rados.rados.0.smithi076.stdout:1856: done (1 left) 2021-03-15T22:26:48.149 INFO:tasks.rados.rados.0.smithi076.stdout:1857: done (0 left) 2021-03-15T22:26:48.149 INFO:tasks.rados.rados.0.smithi076.stdout:1858: write oid 18 current snap is 206 2021-03-15T22:26:48.149 INFO:tasks.rados.rados.0.smithi076.stdout:1858: seq_num 655 ranges {608585=494453,1579235=601890,2821119=187466} 2021-03-15T22:26:48.154 INFO:tasks.rados.rados.0.smithi076.stdout:1858: writing smithi07694878-18 from 608585 to 1103038 tid 1 2021-03-15T22:26:48.160 INFO:tasks.rados.rados.0.smithi076.stdout:1858: writing smithi07694878-18 from 1579235 to 2181125 tid 2 2021-03-15T22:26:48.165 INFO:tasks.rados.rados.0.smithi076.stdout:1858: writing smithi07694878-18 from 2821119 to 3008585 tid 3 2021-03-15T22:26:48.165 INFO:tasks.rados.rados.0.smithi076.stdout:1859: delete oid 10 current snap is 206 2021-03-15T22:26:48.166 INFO:tasks.rados.rados.0.smithi076.stdout:1858: finishing write tid 1 to smithi07694878-18 2021-03-15T22:26:48.166 INFO:tasks.rados.rados.0.smithi076.stdout:1858: finishing write tid 2 to smithi07694878-18 2021-03-15T22:26:48.167 INFO:tasks.rados.rados.0.smithi076.stdout:1859: done (1 left) 2021-03-15T22:26:48.167 INFO:tasks.rados.rados.0.smithi076.stdout:1860: delete oid 7 current snap is 206 2021-03-15T22:26:48.168 INFO:tasks.rados.rados.0.smithi076.stdout:1858: finishing write tid 3 to smithi07694878-18 2021-03-15T22:26:48.169 INFO:tasks.rados.rados.0.smithi076.stdout:1858: finishing write tid 4 to smithi07694878-18 2021-03-15T22:26:48.170 INFO:tasks.rados.rados.0.smithi076.stdout:1858: finishing write tid 5 to smithi07694878-18 2021-03-15T22:26:48.170 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 809 (ObjNum 655 snap 206 seq_num 655) dirty exists 2021-03-15T22:26:48.171 INFO:tasks.rados.rados.0.smithi076.stdout:1858: left oid 18 (ObjNum 655 snap 206 seq_num 655) 2021-03-15T22:26:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:48.173 INFO:tasks.rados.rados.0.smithi076.stdout:1858: done (1 left) 2021-03-15T22:26:48.174 INFO:tasks.rados.rados.0.smithi076.stdout:1860: done (0 left) 2021-03-15T22:26:48.174 INFO:tasks.rados.rados.0.smithi076.stdout:1861: read oid 33 snap 172 2021-03-15T22:26:48.174 INFO:tasks.rados.rados.0.smithi076.stdout:1861: expect (ObjNum 374 snap 102 seq_num 374) 2021-03-15T22:26:48.175 INFO:tasks.rados.rados.0.smithi076.stdout:1862: write (excl) oid 50 current snap is 206 2021-03-15T22:26:48.176 INFO:tasks.rados.rados.0.smithi076.stdout:1862: seq_num 658 ranges {659006=720200,2154681=749217} 2021-03-15T22:26:48.176 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:48.177 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:48.178 INFO:tasks.rados.rados.0.smithi076.stdout:1862: writing smithi07694878-50 from 659006 to 1379206 tid 1 2021-03-15T22:26:48.187 INFO:tasks.rados.rados.0.smithi076.stdout:1862: writing smithi07694878-50 from 2154681 to 2903898 tid 2 2021-03-15T22:26:48.187 INFO:tasks.rados.rados.0.smithi076.stdout:1863: delete oid 20 current snap is 206 2021-03-15T22:26:48.187 INFO:tasks.rados.rados.0.smithi076.stdout:1861: expect (ObjNum 374 snap 102 seq_num 374) 2021-03-15T22:26:48.268 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: debug 2021-03-15T22:26:47.996+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:26:48.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:26:48.269 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.270 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:26:48.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:26:48.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.271 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:26:48.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: debug 2021-03-15T22:26:47.997+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:26:48.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:26:48.272 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.273 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:26:48.273 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.273 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:26:48.274 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.274 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.274 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:26:48.275 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: debug 2021-03-15T22:26:47.997+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:26:48.275 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:26:48.276 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.276 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:26:48.276 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.277 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:26:48.277 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.278 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.278 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:26:48.278 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: debug 2021-03-15T22:26:47.998+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:26:48.278 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:26:48.279 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.279 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:26:48.280 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.280 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:26:48.280 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.280 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.280 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:26:48.281 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: debug 2021-03-15T22:26:47.998+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:26:48.281 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:26:48.282 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.282 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:26:48.283 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.283 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:26:48.283 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.283 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.284 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:47 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:26:48.284 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: debug 2021-03-15T22:26:47.999+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:26:48.285 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:26:48.285 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.285 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:26:48.285 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.286 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:26:48.286 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.287 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.287 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:26:48.290 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: debug 2021-03-15T22:26:47.999+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:26:48.290 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:26:48.291 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.291 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:26:48.291 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.291 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:26:48.292 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.292 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.292 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:26:48.292 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:48.294 INFO:tasks.rados.rados.0.smithi076.stdout:1862: finishing write tid 1 to smithi07694878-50 2021-03-15T22:26:48.295 INFO:tasks.rados.rados.0.smithi076.stdout:1861: done (2 left) 2021-03-15T22:26:48.295 INFO:tasks.rados.rados.0.smithi076.stdout:1863: done (1 left) 2021-03-15T22:26:48.295 INFO:tasks.rados.rados.0.smithi076.stdout:1864: copy_from oid 40 from oid 11 current snap is 206 2021-03-15T22:26:48.295 INFO:tasks.rados.rados.0.smithi076.stdout:1862: finishing write tid 2 to smithi07694878-50 2021-03-15T22:26:48.296 INFO:tasks.rados.rados.0.smithi076.stdout:1862: finishing write tid 3 to smithi07694878-50 2021-03-15T22:26:48.296 INFO:tasks.rados.rados.0.smithi076.stdout:1862: finishing write tid 4 to smithi07694878-50 2021-03-15T22:26:48.296 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 801 (ObjNum 658 snap 206 seq_num 658) dirty exists 2021-03-15T22:26:48.296 INFO:tasks.rados.rados.0.smithi076.stdout:1862: left oid 50 (ObjNum 658 snap 206 seq_num 658) 2021-03-15T22:26:48.297 INFO:tasks.rados.rados.0.smithi076.stdout:1862: done (1 left) 2021-03-15T22:26:48.297 INFO:tasks.rados.rados.0.smithi076.stdout:1865: read oid 9 snap -1 2021-03-15T22:26:48.297 INFO:tasks.rados.rados.0.smithi076.stdout:1865: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:26:48.297 INFO:tasks.rados.rados.0.smithi076.stdout:1866: copy_from oid 44 from oid 15 current snap is 206 2021-03-15T22:26:48.297 INFO:tasks.rados.rados.0.smithi076.stdout:1867: copy_from oid 12 from oid 26 current snap is 206 2021-03-15T22:26:48.298 INFO:tasks.rados.rados.0.smithi076.stdout:1868: snap_remove snap 170 2021-03-15T22:26:48.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:48.324 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:48.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:48.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:48.454 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:48.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:48.456 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:48.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:26:48.558 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:48.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:48.581 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:48.581 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:48.682 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:48.705 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:48.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:48.707 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:48.711 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cluster 2021-03-15T22:26:46.860005+0000 osd.4 2021-03-15T22:26:48.712 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: (osd.4) 101 : cluster [DBG] 4.e scrub ok 2021-03-15T22:26:48.712 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cluster 2021-03-15T22:26:47.074128+0000 mgr.y (mgr.14144) 677 : cluster [DBG] 2021-03-15T22:26:48.712 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: pgmap v864: 138 pgs: 11 active+undersized+degraded, 14 active+undersized, 2 peering, 8 stale+active+clean, 103 active+clean; 431 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 113/1581 objects degraded (7.147%) 2021-03-15T22:26:48.713 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cluster 2021-03-15T22:26 2021-03-15T22:26:48.713 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: :47.346571+0000 mon.a (mon.0) 1401 : cluster [WRN] Health check failed: Degraded data redundancy: 113/1581 objects degraded (7.147%), 11 pgs degraded (PG_DEGRADED) 2021-03-15T22:26:48.713 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:26:48.713 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: -15T22:26:47.697248+0000 mon.a 2021-03-15T22:26:48.714 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: (mon.0) 1402 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:26:48.714 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: 2021-03-15T22:26:48.714 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:26:48.714 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: -15T22:26:47.815551+0000 mon.a (mon.0 2021-03-15T22:26:48.715 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: ) 1403 : audit [DBG] from='client.? 172.21.15.76:0/817930157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:48.715 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cephadm 2021-03-15T22 2021-03-15T22:26:48.715 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: :26:47.992907+0000 mgr.y (mgr. 2021-03-15T22:26:48.715 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: 14144) 678 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:26:48.716 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:26:48.716 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: 22:26:47.996517+0000 mon.a (mon 2021-03-15T22:26:48.716 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: .0) 1404 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:26:48.716 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cephadm 2021-03- 2021-03-15T22:26:48.717 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: 15T22:26:47.997486+0000 mgr.y 2021-03-15T22:26:48.717 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: (mgr.14144) 679 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:26:48.717 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:26:48.717 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.718 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:26:48.718 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.718 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:26:48.718 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:26:48.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: 2021-03-15T22:26:48.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:26:47.998077+0000 mgr.y (mgr.14144) 680 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:26:48.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:26:48.720 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cluster 2021-03-15T22:26:46.860005+0000 osd.4 (osd 2021-03-15T22:26:48.720 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: .4) 101 : cluster [DBG] 4.e scrub ok 2021-03-15T22:26:48.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cluster 2021-03-15T22:26:47.074128+0000 mgr.y (mgr.14144) 677 : cluster [DBG] pgmap v864: 138 pgs: 11 active+undersized+degraded, 14 active+undersized, 2 peering, 8 stale+active+clean, 103 active+clean; 431 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 113/1581 objects degraded (7.147%) 2021-03-15T22:26:48.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cluster 2021-03-15T22:26:47.346571+0000 mon.a (mon.0) 1401 : cluster [WRN] Health check failed: Degraded data redundancy: 113/1581 objects degraded (7.147%), 11 pgs degraded (PG_DEGRADED) 2021-03-15T22:26:48.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: audit 2021-03-15T22:26:47.697248+0000 mon.a (mon.0) 1402 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:26:48.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: audit 2021-03-15T22:26:47.815551+0000 mon.a (mon.0) 1403 : audit [DBG] from='client.? 172.21.15.76:0/817930157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:48.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:26:47.992907+0000 mgr.y (mgr.14144) 678 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:26:48.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: audit 2021-03-15T22:26:47.996517+0000 mon.a (mon.0) 1404 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:26:48.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:26:47.997486+0000 mgr.y (mgr.14144) 679 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:26:48.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:26:48.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:26:48.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:26:48.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:26:48.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:26:47.998077+0000 mgr.y (mgr.14144) 680 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:26:48.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:26:48.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.725 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:26:48.725 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.725 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:26:48.725 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.725 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:26:48.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:26:48.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:26:48.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:26:47.998598+0000 mgr.y (mgr.14144) 681 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:26:48.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:26:48.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:26:48.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:26:48.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:26:48.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:26:47.999093 2021-03-15T22:26:48.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 682 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:26:48.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:26:48.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:26:48.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:26:48.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:26:48.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:26:47.999612+0000 mgr.y (mgr.14144) 683 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:26:48.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:26:48.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:26:48.733 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:26:48.733 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:26:47.998598+0000 mgr.y (mgr.14144) 681 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:26:48.733 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:26:48.734 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.734 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:26:48.734 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.734 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:26:48.735 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.735 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.735 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:26:48.735 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:26:47.999093+0000 mgr.y (mgr.14144) 682 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:26:48.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:26:48.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:26:48.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:26:48.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:26:48.737 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:26:47.999612+0000 mgr.y (mgr.14144) 683 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:26:48.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:26:48.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:26:48.738 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:26:48.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:26:48.740 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.740 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:26:48.740 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:26:48.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:26:48.000108+0000 mgr.y (mgr. 2021-03-15T22:26:48.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: 14144) 684 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:26:48.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:26:48.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:26:48.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:26:48.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:26:48.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cephadm 2021-03-15T22:26:48.000449+0000 mgr.y (mgr.14144) 685 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:26:48.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:26:48.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:26:48.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:26:48.745 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.745 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.745 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:26:48.745 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[62156]: cluster 2021-03-15T22:26:48.144985+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e493: 8 total, 7 up, 7 in 2021-03-15T22:26:48.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:26:48.000108+0000 mgr.y (mgr.14144) 684 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:26:48.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:26:48.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:26:48.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:26:48.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:26:48.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cephadm 2021-03-15T22:26:48.000449+0000 mgr.y (mgr.14144) 685 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:26:48.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:26:48.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:26:48.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.749 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:26:48.749 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.749 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.749 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:26:48.750 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:48 smithi076 conmon[56514]: cluster 2021-03-15T22:26:48.144985+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e493: 8 total, 7 up, 7 in 2021-03-15T22:26:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cluster 2021-03-15T22:26:46.860005+0000 osd.4 ( 2021-03-15T22:26:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: osd.4) 101 : cluster [DBG] 4.e scrub ok 2021-03-15T22:26:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cluster 2021-03-15T22:26:47.074128+0000 mgr.y (mgr.14144) 677 : cluster [DBG] pgmap v864: 138 pgs: 11 active+undersized+degraded, 14 active+undersized, 2 peering, 8 stale+active+clean, 103 active+clean; 431 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.7 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 113/1581 objects degraded (7.147%) 2021-03-15T22:26:48.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cluster 2021-03-15T22:26:47.346571+0000 mon.a (mon.0) 1401 : cluster [WRN] Health check failed: Degraded data redundancy: 113/1581 objects degraded (7.147%), 11 pgs degraded (PG_DEGRADED) 2021-03-15T22:26:48.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: audit 2021-03-15T22:26:47.697248+0000 mon.a (mon.0) 1402 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:26:48.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: audit 2021 2021-03-15T22:26:48.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: -03-15T22:26:47.815551+0000 mon.a (mon.0) 1403 : audit [DBG] from='client.? 172.21.15.76:0/817930157' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:48.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:26:48.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: 2021-03-15T22:26:47.992907+0000 mgr.y 2021-03-15T22:26:48.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: (mgr.14144) 678 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:26:48.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: audit 2021-03-15T22:26:47.996517+0000 mon.a ( 2021-03-15T22:26:48.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: mon.0) 1404 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:26:48.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cephadm 2021 2021-03-15T22:26:48.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: -03-15T22:26:47.997486 2021-03-15T22:26:48.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 679 : cephadm [ERR] 2021-03-15T22:26:48.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:26:48.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:26:48.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:26:48.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:26:48.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:26:48.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:26:47.998077+0000 mgr.y (mgr.14144) 680 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:26:48.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:26:48.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:26:48.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:26:48.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:26:48.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cephadm 2021 2021-03-15T22:26:48.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: -03-15T22:26:47.998598+0000 mgr.y (mgr.14144) 681 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:26:48.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:26:48.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:26:48.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:26:48.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:26:48.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:26:47 2021-03-15T22:26:48.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: .999093+0000 mgr.y (mgr.14144) 682 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:26:48.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:26:48.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:26:48.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:26:48.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:26:48.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:26:47.999612+0000 mgr.y (mgr.14144) 683 2021-03-15T22:26:48.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:26:48.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:26:48.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:26:48.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:26:48.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:26:48.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:26:48.000108+0000 mgr.y (mgr.14144) 684 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:26:48.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:26:48.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:26:48.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:26:48.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:26:48.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cephadm 2021-03-15T22:26:48.000449+0000 mgr.y (mgr.14144) 685 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:26:48.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:26:48.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:26:48.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:26:48.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:26:48.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:26:48.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:26:48.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:26:48.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:26:48.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:48 smithi125 conmon[56739]: cluster 2021-03-15T22:26:48.144985+0000 mon.a (mon.0) 1405 : cluster [DBG] osdmap e493: 8 total, 7 up, 7 in 2021-03-15T22:26:48.807 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:48.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:48.832 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:48.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:48.934 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:48.956 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:48.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:48.958 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:49.012 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:26:49.013 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:26:49.059 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:49.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:49.084 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:49.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:49.146 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:49.147 INFO:teuthology.orchestra.run.smithi076.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2021-03-15T22:26:49.152 INFO:tasks.rados.rados.0.smithi076.stdout:1868: done (4 left) 2021-03-15T22:26:49.152 INFO:tasks.rados.rados.0.smithi076.stdout:1869: snap_create 2021-03-15T22:26:49.153 INFO:tasks.rados.rados.0.smithi076.stdout:1864: finishing copy_from to smithi07694878-40 2021-03-15T22:26:49.153 INFO:tasks.rados.rados.0.smithi076.stdout:1864: got expected ENOENT (src dne) 2021-03-15T22:26:49.153 INFO:tasks.rados.rados.0.smithi076.stdout:1864: finishing copy_from racing read to smithi07694878-40 2021-03-15T22:26:49.153 INFO:tasks.rados.rados.0.smithi076.stdout:1865: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:26:49.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.2.service' 2021-03-15T22:26:49.186 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:49.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:49.216 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:49.216 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:49.245 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-03-15T22:26:49.245 INFO:tasks.cephadm.osd.2:Stopped osd.2 2021-03-15T22:26:49.246 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.2 2021-03-15T22:26:49.246 INFO:tasks.thrashosds.thrasher:remote for osd 2 is ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T22:26:49.246 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -n osd.2 --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2021-03-15T22:26:49.317 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:49.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:49.340 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:49.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:49.357 INFO:tasks.rados.rados.0.smithi076.stdout:1866: finishing copy_from to smithi07694878-44 2021-03-15T22:26:49.357 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1169 (ObjNum 649 snap 201 seq_num 649) dirty exists 2021-03-15T22:26:49.358 INFO:tasks.rados.rados.0.smithi076.stdout:1866: finishing copy_from racing read to smithi07694878-44 2021-03-15T22:26:49.358 INFO:tasks.rados.rados.0.smithi076.stdout:1867: finishing copy_from to smithi07694878-12 2021-03-15T22:26:49.358 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 786 (ObjNum 617 snap 191 seq_num 617) dirty exists 2021-03-15T22:26:49.358 INFO:tasks.rados.rados.0.smithi076.stdout:1867: finishing copy_from racing read to smithi07694878-12 2021-03-15T22:26:49.441 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:49.466 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:49.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:49.473 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/osd.2/config 2021-03-15T22:26:49.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:49 smithi076 conmon[56514]: cluster 2021-03-15T22:26:48.205838+0000 osd.6 (osd.6) 80 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:26:49.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:49 smithi076 conmon[56514]: cluster 2021-03-15T22:26:48.509180+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.c scrub ok 2021-03-15T22:26:49.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:49 smithi076 conmon[56514]: cluster 2021-03-15T22:26:49.144373+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e494: 8 total, 7 up, 7 in 2021-03-15T22:26:49.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:49 smithi076 conmon[62156]: cluster 2021-03-15T22:26:48.205838+0000 osd.6 (osd.6) 80 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:26:49.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:49 smithi076 conmon[62156]: cluster 2021-03-15T22:26:48.509180+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.c scrub ok 2021-03-15T22:26:49.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:49 smithi076 conmon[62156]: cluster 2021-03-15T22:26:49.144373+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e494: 8 total, 7 up, 7 in 2021-03-15T22:26:49.568 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:49.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:49.593 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:49.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:49.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:49.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:49.719 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:49.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:49.785 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:49 smithi125 conmon[56739]: cluster 2021-03-15T22:26:48.205838+0000 osd.6 (osd.6) 80 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:26:49.785 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:49 smithi125 conmon[56739]: cluster 2021-03-15T22:26 2021-03-15T22:26:49.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:49 smithi125 conmon[56739]: :48.509180+0000 osd.3 (osd.3) 86 : cluster [DBG] 6.c scrub ok 2021-03-15T22:26:49.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:49 smithi125 conmon[56739]: cluster 2021-03-15T22:26:49.144373+0000 mon.a (mon.0) 1406 : cluster [DBG] osdmap e494: 8 total, 7 up, 7 in 2021-03-15T22:26:49.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:49 smithi125 conmon[56739]: 2021-03-15T22:26:49.821 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:49.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:49.844 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:49.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:49.925 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-03-15T22:26:49.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-15T22:26:49.945 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:49.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:49.970 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:49.970 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:50.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:26:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:26:50.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:50.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:50.096 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:50.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:50.135 INFO:tasks.rados.rados.0.smithi076.stdout:1864: done (4 left) 2021-03-15T22:26:50.137 INFO:tasks.rados.rados.0.smithi076.stdout:1865: done (3 left) 2021-03-15T22:26:50.137 INFO:tasks.rados.rados.0.smithi076.stdout:1866: done (2 left) 2021-03-15T22:26:50.137 INFO:tasks.rados.rados.0.smithi076.stdout:1867: done (1 left) 2021-03-15T22:26:50.137 INFO:tasks.rados.rados.0.smithi076.stdout:1869: done (0 left) 2021-03-15T22:26:50.137 INFO:tasks.rados.rados.0.smithi076.stdout:1870: read oid 38 snap 183 2021-03-15T22:26:50.138 INFO:tasks.rados.rados.0.smithi076.stdout:1870: expect deleted 2021-03-15T22:26:50.138 INFO:tasks.rados.rados.0.smithi076.stdout:1871: snap_remove snap 188 2021-03-15T22:26:50.198 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:50.231 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:50.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:50.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:50.334 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:50.356 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:50.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:50.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:50.367 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:50.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:50.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[56514]: cluster 2021-03-15T22:26:49.074519+0000 mgr.y (mgr.14144) 686 : cluster [DBG] pgmap v867: 138 pgs: 17 active+undersized+degraded, 24 active+undersized, 2 peering, 3 stale+active+clean, 92 active+clean; 438 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 1.4 MiB/s wr, 9 op/s; 158/1584 objects degraded (9.975%) 2021-03-15T22:26:50.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[56514]: cluster 2021-03-15T22:26:49.219216+0000 osd.6 (osd.6) 81 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:26:50.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[56514]: cluster 2021-03-15T22:26:49.448163+0000 osd.1 (osd.1) 96 : cluster [DBG] 7.b scrub ok 2021-03-15T22:26:50.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[56514]: audit 2021-03-15T22:26:49.783681+0000 mgr.y (mgr.14144) 687 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:50.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[56514]: cluster 2021-03-15T22:26:50.131309+0000 mon.a (mon.0) 1407 : cluster [DBG] osdmap e495: 8 total, 7 up, 7 in 2021-03-15T22:26:50.482 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: cluster 2021-03-15T22:26: 2021-03-15T22:26:50.483 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: 49.074519+0000 mgr.y (mgr.14144) 686 : cluster [DBG] pgmap v867: 138 pgs: 17 active+undersized+degraded, 24 active+undersized, 2 peering, 3 stale+active+clean, 92 active+clean; 438 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 1.4 MiB/s wr, 9 op/s; 158/1584 objects degraded (9.975%) 2021-03-15T22:26:50.483 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: cluster 2021-03-15T22:26:49.219216+0000 osd.6 (osd.6) 81 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:26:50.483 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: cluster 2021-03-15T22:26:50.483 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: 2021-03-15T22:26:49.448163+0000 osd.1 (osd.1) 96 : cluster [DBG] 2021-03-15T22:26:50.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: 7.b scrub ok 2021-03-15T22:26:50.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:26:50.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: 03-15T22:26:49.783681+0000 2021-03-15T22:26:50.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: mgr.y (mgr.14144) 687 : audit [DBG] 2021-03-15T22:26:50.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:50.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:26:50.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: 03-15T22:26:50.131309+0000 2021-03-15T22:26:50.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: mon.a (mon.0) 1407 : cluster [DBG] 2021-03-15T22:26:50.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:50 smithi076 conmon[62156]: osdmap e495: 8 total, 7 up, 7 in 2021-03-15T22:26:50.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:50.487 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:50.487 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:50.588 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:50.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:50.612 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:50.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:50.714 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:50.736 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:50.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:50.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:50.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:50 smithi125 conmon[56739]: cluster 2021-03-15T22:26:49.074519+0000 mgr.y (mgr.14144) 686 2021-03-15T22:26:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:50 smithi125 conmon[56739]: : cluster [DBG] pgmap v867: 138 pgs: 17 active+undersized+degraded, 24 active+undersized, 2 peering, 3 stale+active+clean, 92 active+clean; 438 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 5.6 MiB/s rd, 1.4 MiB/s wr, 9 op/s; 158/1584 objects degraded (9.975%) 2021-03-15T22:26:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:50 smithi125 conmon[56739]: cluster 2021-03-15T22:26:49.219216+0000 osd.6 (osd.6) 81 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:26:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:50 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:26:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:50 smithi125 conmon[56739]: 26:49.448163+0000 osd.1 (osd.1) 96 : cluster [DBG] 7.b scrub ok 2021-03-15T22:26:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:50 smithi125 conmon[56739]: audit 2021 2021-03-15T22:26:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:50 smithi125 conmon[56739]: -03-15T22:26:49.783681+0000 mgr.y (mgr.14144) 2021-03-15T22:26:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:50 smithi125 conmon[56739]: 687 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:26:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:50 smithi125 conmon[56739]: cluster 2021-03-15T22:26:50 2021-03-15T22:26:50.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:50 smithi125 conmon[56739]: .131309+0000 mon.a (mon.0) 1407 : cluster [DBG] 2021-03-15T22:26:50.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:50 smithi125 conmon[56739]: osdmap e495: 8 total, 7 up, 7 in 2021-03-15T22:26:50.839 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:50.860 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:50.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:50.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:50.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:50.986 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:50.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:50.988 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:51.014 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:26:51.088 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:51.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:51.112 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:51.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:51.142 INFO:tasks.rados.rados.0.smithi076.stdout:1871: done (1 left) 2021-03-15T22:26:51.142 INFO:tasks.rados.rados.0.smithi076.stdout:1872: delete oid 14 current snap is 207 2021-03-15T22:26:51.162 INFO:tasks.rados.rados.0.smithi076.stdout:1870: done (1 left) 2021-03-15T22:26:51.163 INFO:tasks.rados.rados.0.smithi076.stdout:1872: done (0 left) 2021-03-15T22:26:51.163 INFO:tasks.rados.rados.0.smithi076.stdout:1873: rollback oid 46 current snap is 207 2021-03-15T22:26:51.163 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 46 to 139 2021-03-15T22:26:51.163 INFO:tasks.rados.rados.0.smithi076.stdout:1874: snap_remove snap 192 2021-03-15T22:26:51.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:51.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:51.240 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:51.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:51.341 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:51.363 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:51.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:51.365 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:51.466 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:51.488 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:51.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:51.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:51.547 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:51.590 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:51.611 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:51.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:51.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:51 smithi076 conmon[56514]: cluster 2021-03-15T22:26:50.440725+0000 2021-03-15T22:26:51.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:51 smithi076 conmon[56514]: osd.1 (osd.1) 97 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:26:51.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:51 smithi076 conmon[56514]: cluster 2021-03-15T22:26:51.139838+0000 mon.a (mon.0) 1408 : cluster [DBG] osdmap e496: 8 total, 7 up, 7 in 2021-03-15T22:26:51.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:51 smithi076 conmon[62156]: cluster 2021-03-15T22:26:50.440725+0000 2021-03-15T22:26:51.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:51 smithi076 conmon[62156]: osd.1 (osd.1) 97 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:26:51.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:51 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:26:51.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:51 smithi076 conmon[62156]: -15T22:26:51.139838+0000 mon.a (mon.0) 1408 : 2021-03-15T22:26:51.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:51 smithi076 conmon[62156]: cluster [DBG] osdmap e496: 8 total, 7 up, 7 in 2021-03-15T22:26:51.714 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:51.740 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:51.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:51.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:51.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:51 smithi125 conmon[56739]: cluster 2021-03-15T22:26:50.440725+0000 osd.1 2021-03-15T22:26:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:51 smithi125 conmon[56739]: (osd.1) 97 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:26:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:51 smithi125 conmon[56739]: cluster 2021-03-15T22:26:51.139838+0000 mon.a 2021-03-15T22:26:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:51 smithi125 conmon[56739]: (mon.0) 1408 : cluster [DBG] osdmap e496: 8 total, 7 up, 7 in 2021-03-15T22:26:51.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:51.865 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:51.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:51.866 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:51.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:51.990 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:51.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:51.992 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:52.093 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:52.116 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:52.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:52.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:52.147 INFO:tasks.rados.rados.0.smithi076.stdout:1874: done (1 left) 2021-03-15T22:26:52.147 INFO:tasks.rados.rados.0.smithi076.stdout:1875: read oid 6 snap -1 2021-03-15T22:26:52.148 INFO:tasks.rados.rados.0.smithi076.stdout:1875: expect deleted 2021-03-15T22:26:52.148 INFO:tasks.rados.rados.0.smithi076.stdout:1876: read oid 5 snap -1 2021-03-15T22:26:52.148 INFO:tasks.rados.rados.0.smithi076.stdout:1876: expect (ObjNum 651 snap 202 seq_num 651) 2021-03-15T22:26:52.148 INFO:tasks.rados.rados.0.smithi076.stdout:1877: snap_create 2021-03-15T22:26:52.149 INFO:tasks.rados.rados.0.smithi076.stdout:1873: finishing rollback tid 0 to smithi07694878-46 2021-03-15T22:26:52.149 INFO:tasks.rados.rados.0.smithi076.stdout:1873: finishing rollback tid 1 to smithi07694878-46 2021-03-15T22:26:52.149 INFO:tasks.rados.rados.0.smithi076.stdout:1873: finishing rollback tid 2 to smithi07694878-46 2021-03-15T22:26:52.149 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 835 (ObjNum 469 snap 137 seq_num 469) dirty exists 2021-03-15T22:26:52.152 INFO:tasks.rados.rados.0.smithi076.stdout:1876: expect (ObjNum 651 snap 202 seq_num 651) 2021-03-15T22:26:52.218 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:52.250 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:52.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:52.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:52.353 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:52.375 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:52.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:52.377 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:52.478 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:52.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:52 smithi076 conmon[56514]: cluster 2021-03-15T22:26:51.075023+0000 mgr.y (mgr.14144) 688 : cluster [DBG] pgmap v870: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 443 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 9.9 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 194/1587 objects degraded (12.224%) 2021-03-15T22:26:52.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:52 smithi076 conmon[56514]: cluster 2021-03-15T22:26:52.141237+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e497: 8 total, 7 up, 7 in 2021-03-15T22:26:52.483 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:52 smithi076 conmon[62156]: cluster 2021-03-15T22:26:51.075023+0000 mgr.y (mgr.14144) 688 : cluster [DBG] pgmap v870: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 443 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 9.9 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 194/1587 objects degraded (12.224%) 2021-03-15T22:26:52.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:52 smithi076 conmon[62156]: cluster 2021-03-15T22:26:52.141237+0000 mon.a (mon.0) 1409 : cluster [DBG] osdmap e497: 8 total, 7 up, 7 in 2021-03-15T22:26:52.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:52.503 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:52.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:52.605 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:52.616 INFO:teuthology.orchestra.run.smithi076.stdout:7.e 2021-03-15T22:26:52.616 INFO:teuthology.orchestra.run.smithi076.stdout:7.8 2021-03-15T22:26:52.617 INFO:teuthology.orchestra.run.smithi076.stdout:7.3 2021-03-15T22:26:52.617 INFO:teuthology.orchestra.run.smithi076.stdout:6.e 2021-03-15T22:26:52.617 INFO:teuthology.orchestra.run.smithi076.stdout:6.9 2021-03-15T22:26:52.617 INFO:teuthology.orchestra.run.smithi076.stdout:6.8 2021-03-15T22:26:52.618 INFO:teuthology.orchestra.run.smithi076.stdout:6.3 2021-03-15T22:26:52.618 INFO:teuthology.orchestra.run.smithi076.stdout:6.1 2021-03-15T22:26:52.618 INFO:teuthology.orchestra.run.smithi076.stdout:6.18 2021-03-15T22:26:52.618 INFO:teuthology.orchestra.run.smithi076.stdout:6.17 2021-03-15T22:26:52.619 INFO:teuthology.orchestra.run.smithi076.stdout:7.a 2021-03-15T22:26:52.619 INFO:teuthology.orchestra.run.smithi076.stdout:4.1c 2021-03-15T22:26:52.619 INFO:teuthology.orchestra.run.smithi076.stdout:7.2 2021-03-15T22:26:52.619 INFO:teuthology.orchestra.run.smithi076.stdout:5.a 2021-03-15T22:26:52.619 INFO:teuthology.orchestra.run.smithi076.stdout:4.1a 2021-03-15T22:26:52.620 INFO:teuthology.orchestra.run.smithi076.stdout:4.12 2021-03-15T22:26:52.620 INFO:teuthology.orchestra.run.smithi076.stdout:6.f 2021-03-15T22:26:52.620 INFO:teuthology.orchestra.run.smithi076.stdout:6.a 2021-03-15T22:26:52.620 INFO:teuthology.orchestra.run.smithi076.stdout:6.2 2021-03-15T22:26:52.620 INFO:teuthology.orchestra.run.smithi076.stdout:5.2 2021-03-15T22:26:52.621 INFO:teuthology.orchestra.run.smithi076.stdout:4.16 2021-03-15T22:26:52.621 INFO:teuthology.orchestra.run.smithi076.stdout:7.5 2021-03-15T22:26:52.621 INFO:teuthology.orchestra.run.smithi076.stdout:3.1e 2021-03-15T22:26:52.621 INFO:teuthology.orchestra.run.smithi076.stdout:3.18 2021-03-15T22:26:52.622 INFO:teuthology.orchestra.run.smithi076.stdout:3.4 2021-03-15T22:26:52.622 INFO:teuthology.orchestra.run.smithi076.stdout:3.1a 2021-03-15T22:26:52.622 INFO:teuthology.orchestra.run.smithi076.stdout:3.1f 2021-03-15T22:26:52.622 INFO:teuthology.orchestra.run.smithi076.stdout:3.0 2021-03-15T22:26:52.623 INFO:teuthology.orchestra.run.smithi076.stdout:3.13 2021-03-15T22:26:52.623 INFO:teuthology.orchestra.run.smithi076.stdout:6.11 2021-03-15T22:26:52.623 INFO:teuthology.orchestra.run.smithi076.stdout:2.1 2021-03-15T22:26:52.623 INFO:teuthology.orchestra.run.smithi076.stdout:5.14 2021-03-15T22:26:52.623 INFO:teuthology.orchestra.run.smithi076.stdout:5.b 2021-03-15T22:26:52.624 INFO:teuthology.orchestra.run.smithi076.stdout:6.1c 2021-03-15T22:26:52.624 INFO:teuthology.orchestra.run.smithi076.stdout:4.4 2021-03-15T22:26:52.624 INFO:teuthology.orchestra.run.smithi076.stdout:3.14 2021-03-15T22:26:52.624 INFO:teuthology.orchestra.run.smithi076.stdout:5.d 2021-03-15T22:26:52.625 INFO:teuthology.orchestra.run.smithi076.stdout:4.6 2021-03-15T22:26:52.625 INFO:teuthology.orchestra.run.smithi076.stdout:4.a 2021-03-15T22:26:52.625 INFO:teuthology.orchestra.run.smithi076.stdout:5.e 2021-03-15T22:26:52.625 INFO:teuthology.orchestra.run.smithi076.stdout:3.9 2021-03-15T22:26:52.625 INFO:teuthology.orchestra.run.smithi076.stdout:4.b 2021-03-15T22:26:52.626 INFO:teuthology.orchestra.run.smithi076.stdout:5.4 2021-03-15T22:26:52.626 INFO:teuthology.orchestra.run.smithi076.stdout:6.14 2021-03-15T22:26:52.626 INFO:teuthology.orchestra.run.smithi076.stdout:4.d 2021-03-15T22:26:52.626 INFO:teuthology.orchestra.run.smithi076.stdout:5.6 2021-03-15T22:26:52.626 INFO:teuthology.orchestra.run.smithi076.stdout:6.d 2021-03-15T22:26:52.627 INFO:teuthology.orchestra.run.smithi076.stdout:6.12 2021-03-15T22:26:52.627 INFO:teuthology.orchestra.run.smithi076.stdout:3.5 2021-03-15T22:26:52.627 INFO:teuthology.orchestra.run.smithi076.stdout:6.13 2021-03-15T22:26:52.627 INFO:teuthology.orchestra.run.smithi076.stdout:4.f 2021-03-15T22:26:52.628 INFO:teuthology.orchestra.run.smithi076.stdout:4.19 2021-03-15T22:26:52.628 INFO:teuthology.orchestra.run.smithi076.stdout:5.8 2021-03-15T22:26:52.628 INFO:teuthology.orchestra.run.smithi076.stdout:6.0 2021-03-15T22:26:52.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:52.630 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:52.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:52.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:52.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:52.754 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:52.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:52.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:52 smithi125 conmon[56739]: cluster 2021-03-15T22:26: 2021-03-15T22:26:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:52 smithi125 conmon[56739]: 51.075023+0000 mgr.y (mgr.14144) 688 : cluster [DBG] pgmap v870: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 443 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 9.9 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 194/1587 objects degraded (12.224%) 2021-03-15T22:26:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:52 smithi125 conmon[56739]: cluster 2021-03-15T22:26 2021-03-15T22:26:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:52 smithi125 conmon[56739]: :52.141237+0000 mon.a (mon. 2021-03-15T22:26:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:52 smithi125 conmon[56739]: 0) 1409 : cluster [DBG] 2021-03-15T22:26:52.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:52 smithi125 conmon[56739]: osdmap e497: 8 total, 7 up, 7 in 2021-03-15T22:26:52.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:52.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:52.880 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:52.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:52.981 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:53.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:53.005 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:53.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:53.107 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:53.133 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:53.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:53.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:53.146 INFO:tasks.rados.rados.0.smithi076.stdout:1873: done (3 left) 2021-03-15T22:26:53.146 INFO:tasks.rados.rados.0.smithi076.stdout:1875: done (2 left) 2021-03-15T22:26:53.146 INFO:tasks.rados.rados.0.smithi076.stdout:1876: done (1 left) 2021-03-15T22:26:53.147 INFO:tasks.rados.rados.0.smithi076.stdout:1877: done (0 left) 2021-03-15T22:26:53.147 INFO:tasks.rados.rados.0.smithi076.stdout:1878: delete oid 7 current snap is 208 2021-03-15T22:26:53.151 INFO:tasks.rados.rados.0.smithi076.stdout:1878: done (0 left) 2021-03-15T22:26:53.151 INFO:tasks.rados.rados.0.smithi076.stdout:1879: write oid 31 current snap is 208 2021-03-15T22:26:53.152 INFO:tasks.rados.rados.0.smithi076.stdout:1879: seq_num 662 ranges {726234=737549,1988609=756555} 2021-03-15T22:26:53.153 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:26:53.160 INFO:tasks.rados.rados.0.smithi076.stdout:1879: writing smithi07694878-31 from 726234 to 1463783 tid 1 2021-03-15T22:26:53.169 INFO:tasks.rados.rados.0.smithi076.stdout:1879: writing smithi07694878-31 from 1988609 to 2745164 tid 2 2021-03-15T22:26:53.170 INFO:tasks.rados.rados.0.smithi076.stdout:1880: write oid 7 current snap is 208 2021-03-15T22:26:53.170 INFO:tasks.rados.rados.0.smithi076.stdout:1880: seq_num 663 ranges {743041=412798,1930738=616566,3040503=102538} 2021-03-15T22:26:53.176 INFO:tasks.rados.rados.0.smithi076.stdout:1880: writing smithi07694878-7 from 743041 to 1155839 tid 1 2021-03-15T22:26:53.185 INFO:tasks.rados.rados.0.smithi076.stdout:1880: writing smithi07694878-7 from 1930738 to 2547304 tid 2 2021-03-15T22:26:53.189 INFO:tasks.rados.rados.0.smithi076.stdout:1880: writing smithi07694878-7 from 3040503 to 3143041 tid 3 2021-03-15T22:26:53.189 INFO:tasks.rados.rados.0.smithi076.stdout:1881: snap_create 2021-03-15T22:26:53.190 INFO:tasks.rados.rados.0.smithi076.stdout:1879: finishing write tid 1 to smithi07694878-31 2021-03-15T22:26:53.190 INFO:tasks.rados.rados.0.smithi076.stdout:1879: finishing write tid 2 to smithi07694878-31 2021-03-15T22:26:53.190 INFO:tasks.rados.rados.0.smithi076.stdout:1879: finishing write tid 3 to smithi07694878-31 2021-03-15T22:26:53.191 INFO:tasks.rados.rados.0.smithi076.stdout:1879: finishing write tid 4 to smithi07694878-31 2021-03-15T22:26:53.191 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 849 (ObjNum 662 snap 208 seq_num 662) dirty exists 2021-03-15T22:26:53.191 INFO:tasks.rados.rados.0.smithi076.stdout:1879: left oid 31 (ObjNum 662 snap 208 seq_num 662) 2021-03-15T22:26:53.191 INFO:tasks.rados.rados.0.smithi076.stdout:1880: finishing write tid 1 to smithi07694878-7 2021-03-15T22:26:53.192 INFO:tasks.rados.rados.0.smithi076.stdout:1880: finishing write tid 2 to smithi07694878-7 2021-03-15T22:26:53.192 INFO:tasks.rados.rados.0.smithi076.stdout:1880: finishing write tid 3 to smithi07694878-7 2021-03-15T22:26:53.193 INFO:tasks.rados.rados.0.smithi076.stdout:1880: finishing write tid 4 to smithi07694878-7 2021-03-15T22:26:53.193 INFO:tasks.rados.rados.0.smithi076.stdout:1880: finishing write tid 5 to smithi07694878-7 2021-03-15T22:26:53.193 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 1010 (ObjNum 663 snap 208 seq_num 663) dirty exists 2021-03-15T22:26:53.194 INFO:tasks.rados.rados.0.smithi076.stdout:1880: left oid 7 (ObjNum 663 snap 208 seq_num 663) 2021-03-15T22:26:53.236 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:53.266 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:53.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:53.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:53.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:53.390 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:53.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:53.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:53.493 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:53.515 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:53.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:53.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:53.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[56514]: cluster 2021-03-15T22:26:51.966924+0000 osd.4 (osd.4 2021-03-15T22:26:53.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[56514]: ) 102 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:26:53.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[56514]: audit 2021-03-15T22:26:52.460605+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? 172.21.15.76:0/3930100367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:26:53.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[56514]: cluster 2021-03-15T22:26:52.558901+0000 osd.3 (osd.3) 87 : cluster [DBG] 5.13 deep-scrub ok 2021-03-15T22:26:53.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[56514]: cluster 2021-03-15T22:26:52.609853+0000 osd.5 (osd.5) 91 : cluster [DBG] 4.15 deep-scrub ok 2021-03-15T22:26:53.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[56514]: cluster 2021-03-15T22:26:53.138534+0000 mon.a (mon.0) 1411 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:53.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[56514]: audit 2021-03-15T22:26:53.142771+0000 mon.a (mon.0) 1412 : audit [INF] from='client.? 172.21.15.76:0/3930100367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:26:53.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[56514]: cluster 2021-03-15T22:26:53.142822+0000 mon.a (mon.0) 1413 : cluster [DBG] osdmap e498: 8 total, 7 up, 7 in 2021-03-15T22:26:53.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: cluster 2021-03-15T22:26: 2021-03-15T22:26:53.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: 51.966924+0000 osd.4 (osd.4) 102 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:26:53.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: audit 2021-03-15T22:26: 2021-03-15T22:26:53.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: 52.460605+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? 172.21.15.76:0/3930100367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:26:53.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: 2021-03-15T22:26:53.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:26:53.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: -03-15T22:26:52.558901+0000 osd.3 (osd.3) 2021-03-15T22:26:53.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: 87 : cluster [DBG] 5.13 deep-scrub ok 2021-03-15T22:26:53.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:26:53.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: 26:52.609853+0000 osd.5 (osd.5) 91 : cluster 2021-03-15T22:26:53.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: [DBG] 4.15 deep-scrub ok 2021-03-15T22:26:53.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: cluster 2021-03-15T22:26:53.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: 2021-03-15T22:26:53.138534+0000 mon.a (mon 2021-03-15T22:26:53.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: .0) 1411 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:53.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:26:53.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: 22:26:53.142771+0000 mon.a (mon.0) 1412 2021-03-15T22:26:53.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/3930100367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:26:53.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: cluster 2021-03-15T22:26: 2021-03-15T22:26:53.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: 53.142822+0000 mon.a (mon.0) 1413 : cluster [DBG] 2021-03-15T22:26:53.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:53 smithi076 conmon[62156]: osdmap e498: 8 total, 7 up, 7 in 2021-03-15T22:26:53.617 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:53.640 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:53.641 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:53.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:53.764 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:53.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:53.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:53.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: cluster 2021-03-15T22:26:51.966924+0000 osd.4 (osd.4) 102 : cluster [DBG] 2021-03-15T22:26:53.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: 7.1 scrub ok 2021-03-15T22:26:53.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: audit 2021-03-15T22:26:52.460605+0000 mon.a (mon.0) 1410 : audit [INF] from='client.? 172.21.15.76:0/3930100367' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:26:53.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:26:53.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: -03-15T22:26:52.558901+0000 osd.3 (osd.3) 87 : cluster [DBG] 5.13 deep-scrub ok 2021-03-15T22:26:53.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:26:53.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: -15T22:26:52.609853+0000 osd.5 ( 2021-03-15T22:26:53.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: osd.5) 91 : cluster [DBG] 4.15 deep-scrub ok 2021-03-15T22:26:53.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:26:53.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: 03-15T22:26:53.138534+0000 2021-03-15T22:26:53.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: mon.a (mon.0) 1411 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:26:53.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:26:53.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: T22:26:53.142771+0000 mon.a (mon.0 2021-03-15T22:26:53.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: ) 1412 : audit [INF] from='client.? 172.21.15.76:0/3930100367' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:26:53.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: cluster 2021-03-15T22:26:53.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: 2021-03-15T22:26:53.142822 2021-03-15T22:26:53.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1413 : cluster [DBG] 2021-03-15T22:26:53.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:53 smithi125 conmon[56739]: osdmap e498: 8 total, 7 up, 7 in 2021-03-15T22:26:53.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:53.892 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:53.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:53.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:53.898 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -n osd.2 --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 3.1f --file /var/log/ceph/exp.3.1f.2 2021-03-15T22:26:53.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:54.017 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:54.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:54.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:54.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:54.141 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:54.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:54.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:54.146 INFO:tasks.rados.rados.0.smithi076.stdout:1879: done (2 left) 2021-03-15T22:26:54.146 INFO:tasks.rados.rados.0.smithi076.stdout:1880: done (1 left) 2021-03-15T22:26:54.147 INFO:tasks.rados.rados.0.smithi076.stdout:1881: done (0 left) 2021-03-15T22:26:54.147 INFO:tasks.rados.rados.0.smithi076.stdout:1882: read oid 10 snap 200 2021-03-15T22:26:54.147 INFO:tasks.rados.rados.0.smithi076.stdout:1882: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:26:54.147 INFO:tasks.rados.rados.0.smithi076.stdout:1883: copy_from oid 11 from oid 1 current snap is 209 2021-03-15T22:26:54.148 INFO:tasks.rados.rados.0.smithi076.stdout:1884: read oid 48 snap -1 2021-03-15T22:26:54.148 INFO:tasks.rados.rados.0.smithi076.stdout:1884: expect (ObjNum 641 snap 200 seq_num 641) 2021-03-15T22:26:54.148 INFO:tasks.rados.rados.0.smithi076.stdout:1885: snap_remove snap 206 2021-03-15T22:26:54.168 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/osd.2/config 2021-03-15T22:26:54.244 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:54.266 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:54.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:54.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:54.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:54.391 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:54.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:54.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:54.494 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:54.505 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[56514]: cluster 2021-03-15T22:26: 2021-03-15T22:26:54.505 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[56514]: 53.075555+0000 mgr.y (mgr.14144) 689 : cluster [DBG] pgmap v873: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 444 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.7 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1593 objects degraded (12.367%) 2021-03-15T22:26:54.505 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[56514]: cluster 2021-03-15T22:26:53.086655+0000 osd.7 (osd.7) 98 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:54.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[56514]: cluster 2021-03-15T22:26:53.087555+0000 osd.7 (osd.7) 99 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:54.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[56514]: cluster 2021-03-15T22:26:53.373966+0000 mon.a (mon.0) 1414 : cluster [WRN] Health check update: Degraded data redundancy: 197/1593 objects degraded (12.367%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:26:54.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[56514]: cluster 2021-03-15T22:26:54.144006+0000 mon.a (mon.0) 1415 : cluster [DBG] osdmap e499: 8 total, 7 up, 7 in 2021-03-15T22:26:54.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: cluster 2021-03-15T22:26:54.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 2021-03-15T22:26:54.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 2021 2021-03-15T22:26:54.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: - 2021-03-15T22:26:54.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 03 2021-03-15T22:26:54.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: - 2021-03-15T22:26:54.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 15 2021-03-15T22:26:54.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: T 2021-03-15T22:26:54.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 22 2021-03-15T22:26:54.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: : 2021-03-15T22:26:54.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 26 2021-03-15T22:26:54.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: : 2021-03-15T22:26:54.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 53 2021-03-15T22:26:54.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: . 2021-03-15T22:26:54.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 075555 2021-03-15T22:26:54.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: +0000 2021-03-15T22:26:54.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 2021-03-15T22:26:54.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: mgr.y 2021-03-15T22:26:54.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: ( 2021-03-15T22:26:54.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: mgr.14144) 689 : cluster 2021-03-15T22:26:54.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: [DBG] pgmap v873: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 444 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.7 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1593 objects degraded (12.367%) 2021-03-15T22:26:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: cluster 2021-03-15T22:26:53. 2021-03-15T22:26:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 086655+0000 osd.7 (osd.7) 98 : cluster 2021-03-15T22:26:54.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: [DBG] purged_snaps scrub starts 2021-03-15T22:26:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:26:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 22:26:53.087555+0000 osd.7 (osd. 2021-03-15T22:26:54.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 7) 99 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:26:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: -03-15T22:26:53.373966+0000 2021-03-15T22:26:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: mon.a (mon.0) 1414 : cluster [WRN] Health check update: Degraded data redundancy: 197/1593 objects degraded (12.367%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:26:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 2021-03-15T22:26:54.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: cluster 2021-03-15T22:26:54.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: 2021-03-15T22:26:54.144006+0000 mon.a (mon 2021-03-15T22:26:54.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:54 smithi076 conmon[62156]: .0) 1415 : cluster [DBG] osdmap e499: 8 total, 7 up, 7 in 2021-03-15T22:26:54.517 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:54.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:54.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:54.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:54.643 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:54.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:54.644 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:54.745 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:54.767 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:54.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:54.769 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:54.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: cluster 2021-03-15T22:26:53.075555+0000 2021-03-15T22:26:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: mgr.y (mgr.14144) 689 : cluster [DBG] pgmap v873: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 444 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.7 MiB/s rd, 1.9 MiB/s wr, 13 op/s; 197/1593 objects degraded (12.367%) 2021-03-15T22:26:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:26:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: 26:53.086655+0000 osd.7 (osd. 2021-03-15T22:26:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: 7) 98 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:26:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: 15T22:26:53.087555+0000 osd.7 (osd.7 2021-03-15T22:26:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: ) 99 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:26:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: -03-15T22:26:53.373966+0000 2021-03-15T22:26:54.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: mon.a (mon.0) 1414 : cluster [WRN] 2021-03-15T22:26:54.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: Health check update: Degraded data redundancy: 197/1593 objects degraded (12.367%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:26:54.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:26:54.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: 22:26:54.144006+0000 mon.a (mon.0 2021-03-15T22:26:54.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:54 smithi125 conmon[56739]: ) 1415 : cluster [DBG] osdmap e499: 8 total, 7 up, 7 in 2021-03-15T22:26:54.870 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:54.893 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:54.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:54.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:54.996 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:55.019 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:55.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:55.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:55.138 INFO:tasks.rados.rados.0.smithi076.stdout:1885: done (3 left) 2021-03-15T22:26:55.140 INFO:tasks.rados.rados.0.smithi076.stdout:1886: read oid 26 snap -1 2021-03-15T22:26:55.140 INFO:tasks.rados.rados.0.smithi076.stdout:1886: expect (ObjNum 617 snap 191 seq_num 617) 2021-03-15T22:26:55.140 INFO:tasks.rados.rados.0.smithi076.stdout:1887: read oid 29 snap -1 2021-03-15T22:26:55.141 INFO:tasks.rados.rados.0.smithi076.stdout:1887: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:26:55.141 INFO:tasks.rados.rados.0.smithi076.stdout:1888: write oid 39 current snap is 209 2021-03-15T22:26:55.141 INFO:tasks.rados.rados.0.smithi076.stdout:1888: seq_num 664 ranges {759848=488047,1989220=760224} 2021-03-15T22:26:55.143 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:55.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:55.145 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:55.146 INFO:tasks.rados.rados.0.smithi076.stdout:1888: writing smithi07694878-39 from 759848 to 1247895 tid 1 2021-03-15T22:26:55.155 INFO:tasks.rados.rados.0.smithi076.stdout:1888: writing smithi07694878-39 from 1989220 to 2749444 tid 2 2021-03-15T22:26:55.156 INFO:tasks.rados.rados.0.smithi076.stdout:1889: snap_remove snap 208 2021-03-15T22:26:55.246 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:55.267 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:55.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:55.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:26:55.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:55.401 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:55.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:55.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:55.504 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:55.527 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:55.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:55.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:55.651 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:55.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:55.754 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:55.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:55 smithi076 conmon[56514]: cluster 2021-03-15T22:26:54.725246+0000 osd.0 (osd.0) 2021-03-15T22:26:55.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:55 smithi076 conmon[56514]: 79 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:55.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:55 smithi076 conmon[56514]: cluster 2021-03-15T22:26:54.725653+0000 osd.0 (osd.0) 80 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:55.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:55 smithi076 conmon[56514]: cluster 2021-03-15T22:26:55.135792+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e500: 8 total, 7 up, 7 in 2021-03-15T22:26:55.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:55 smithi076 conmon[62156]: cluster 2021-03-15T22:26:54.725246+0000 osd.0 (osd. 2021-03-15T22:26:55.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:55 smithi076 conmon[62156]: 0) 79 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:55.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:55 smithi076 conmon[62156]: cluster 2021-03-15T22:26:54.725653+0000 osd.0 (osd.0) 80 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:55.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:55 smithi076 conmon[62156]: cluster 2021-03-15T22:26:55.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:55 smithi076 conmon[62156]: 2021-03-15T22:26:55.135792+0000 mon.a (mon.0) 1416 : cluster [DBG] osdmap e500: 8 total, 7 up, 7 in 2021-03-15T22:26:55.777 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:55.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:55.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:55.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:55 smithi125 conmon[56739]: cluster 2021-03-15T22:26:54.725246+0000 osd.0 (osd.0) 79 2021-03-15T22:26:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:55 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:26:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:55 smithi125 conmon[56739]: cluster 2021-03-15T22:26:54.725653+0000 osd.0 (osd.0) 80 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:26:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:55 smithi125 conmon[56739]: cluster 2021-03-15T22:26:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:55 smithi125 conmon[56739]: 2021-03-15T22:26:55.135792+0000 mon.a (mon. 2021-03-15T22:26:55.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:55 smithi125 conmon[56739]: 0) 1416 : cluster [DBG] osdmap e500: 8 total, 7 up, 7 in 2021-03-15T22:26:55.880 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:55.903 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:55.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:55.905 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:55.956 INFO:teuthology.orchestra.run.smithi076.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2021-03-15T22:26:56.006 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:56.029 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:56.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:56.031 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:56.132 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:56.152 INFO:tasks.rados.rados.0.smithi076.stdout:1889: done (6 left) 2021-03-15T22:26:56.152 INFO:tasks.rados.rados.0.smithi076.stdout:1890: snap_remove snap 182 2021-03-15T22:26:56.156 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:56.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:56.158 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:56.258 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:56.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:56.283 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:56.284 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:56.385 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:56.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:56 smithi076 conmon[62156]: cluster 2021-03-15T22:26:55.076232 2021-03-15T22:26:56.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:56 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 690 : cluster [DBG] pgmap v876: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 447 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 1.0 MiB/s wr, 10 op/s; 197/1593 objects degraded (12.367%) 2021-03-15T22:26:56.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:56 smithi076 conmon[62156]: cluster 2021-03-15T22:26:56.147564+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e501: 8 total, 7 up, 7 in 2021-03-15T22:26:56.406 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:56.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:56.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:56.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:56.530 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:26:56.531 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:56.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:56.533 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:56.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:56.655 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:56.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:56.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:56.758 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:56.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:56 smithi076 conmon[56514]: cluster 2021-03-15T22:26:55.076232+0000 mgr.y (mgr.14144) 690 : cluster [DBG] pgmap v876: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 447 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 1.0 MiB/s wr, 10 op/s; 197/1593 objects degraded (12.367%) 2021-03-15T22:26:56.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:56 smithi076 conmon[56514]: cluster 2021-03-15T22:26:56.147564+0000 mon.a (mon.0) 1417 : cluster [DBG] osdmap e501: 8 total, 7 up, 7 in 2021-03-15T22:26:56.780 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:56.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:56.781 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:56.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:56 smithi125 conmon[56739]: cluster 2021-03-15T22:26:55. 2021-03-15T22:26:56.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:56 smithi125 conmon[56739]: 076232+0000 mgr.y (mgr.14144) 690 : cluster [DBG] pgmap v876: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 447 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 4.4 MiB/s rd, 1.0 MiB/s wr, 10 op/s; 197/1593 objects degraded (12.367%) 2021-03-15T22:26:56.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:56 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:26:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:56 smithi125 conmon[56739]: 26:56.147564+0000 mon.a 2021-03-15T22:26:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:56 smithi125 conmon[56739]: (mon.0) 1417 : cluster 2021-03-15T22:26:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:56 smithi125 conmon[56739]: [DBG] osdmap e501: 8 total, 7 up, 7 in 2021-03-15T22:26:56.882 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:56.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:56.907 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:56.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:57.009 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:57.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:57.032 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:57.033 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:57.134 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:57.152 INFO:tasks.rados.rados.0.smithi076.stdout:1890: done (6 left) 2021-03-15T22:26:57.153 INFO:tasks.rados.rados.0.smithi076.stdout:1891: snap_create 2021-03-15T22:26:57.156 INFO:tasks.rados.rados.0.smithi076.stdout:1883: finishing copy_from to smithi07694878-11 2021-03-15T22:26:57.157 INFO:tasks.rados.rados.0.smithi076.stdout:1883: got expected ENOENT (src dne) 2021-03-15T22:26:57.157 INFO:tasks.rados.rados.0.smithi076.stdout:1883: finishing copy_from racing read to smithi07694878-11 2021-03-15T22:26:57.157 INFO:tasks.rados.rados.0.smithi076.stdout:1882: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:26:57.163 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:57.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:57.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:57.236 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:26:57.266 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:57.267 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:26:57.289 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:57.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:57.291 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:57.294 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-03-15T22:26:57.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-15T22:26:57.297 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:26:57.311 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:26:57.328 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:26:57.348 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:26:57.367 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:26:57.385 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:26:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:26:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:26:57.386 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:26:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:26:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:26:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:26:57.387 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:26:57.388 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:26:57.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:57.406 INFO:tasks.rados.rados.0.smithi076.stdout:1884: expect (ObjNum 641 snap 200 seq_num 641) 2021-03-15T22:26:57.414 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:57.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:57.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:57.457 INFO:teuthology.orchestra.run.smithi076.stderr:Exporting 3.1f info 3.1f( v 57'1 (0'0,57'1] local-lis/les=355/356 n=1 ec=56/56 lis/c=355/355 les/c/f=356/356/0 sis=355) 2021-03-15T22:26:57.460 INFO:teuthology.orchestra.run.smithi076.stderr:Read #3:fcb6c186:::zone_info.38e1d1be-4dea-4a9a-b9e9-57f557eecf75:head# 2021-03-15T22:26:57.460 INFO:teuthology.orchestra.run.smithi076.stderr:Export successful 2021-03-15T22:26:57.504 INFO:tasks.rados.rados.0.smithi076.stdout:1886: expect (ObjNum 617 snap 191 seq_num 617) 2021-03-15T22:26:57.516 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:57.539 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:57.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:57.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:57.571 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:26:57.584 INFO:tasks.rados.rados.0.smithi076.stdout:1887: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:26:57.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:57.665 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:57.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:57.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:57.747 INFO:tasks.rados.rados.0.smithi076.stdout:1888: finishing write tid 1 to smithi07694878-39 2021-03-15T22:26:57.748 INFO:tasks.rados.rados.0.smithi076.stdout:1888: finishing write tid 2 to smithi07694878-39 2021-03-15T22:26:57.748 INFO:tasks.rados.rados.0.smithi076.stdout:1888: finishing write tid 3 to smithi07694878-39 2021-03-15T22:26:57.748 INFO:tasks.rados.rados.0.smithi076.stdout:1888: finishing write tid 4 to smithi07694878-39 2021-03-15T22:26:57.748 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 1013 (ObjNum 664 snap 209 seq_num 664) dirty exists 2021-03-15T22:26:57.749 INFO:tasks.rados.rados.0.smithi076.stdout:1888: left oid 39 (ObjNum 664 snap 209 seq_num 664) 2021-03-15T22:26:57.767 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:57.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:57.791 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:57.792 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:57.893 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:57.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:57.916 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:57.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:58.017 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:58.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:58.042 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:58.042 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:58.144 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:58.153 INFO:tasks.rados.rados.0.smithi076.stdout:1882: done (6 left) 2021-03-15T22:26:58.154 INFO:tasks.rados.rados.0.smithi076.stdout:1883: done (5 left) 2021-03-15T22:26:58.154 INFO:tasks.rados.rados.0.smithi076.stdout:1884: done (4 left) 2021-03-15T22:26:58.154 INFO:tasks.rados.rados.0.smithi076.stdout:1886: done (3 left) 2021-03-15T22:26:58.154 INFO:tasks.rados.rados.0.smithi076.stdout:1887: done (2 left) 2021-03-15T22:26:58.155 INFO:tasks.rados.rados.0.smithi076.stdout:1888: done (1 left) 2021-03-15T22:26:58.155 INFO:tasks.rados.rados.0.smithi076.stdout:1891: done (0 left) 2021-03-15T22:26:58.155 INFO:tasks.rados.rados.0.smithi076.stdout:1892: snap_create 2021-03-15T22:26:58.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:58.170 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:58.170 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:58.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:58 smithi076 conmon[56514]: cluster 2021-03-15T22:26:57.077022+0000 mgr.y (mgr.14144) 691 : cluster [DBG] pgmap v879: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 456 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 947 KiB/s wr, 12 op/s; 198/1599 objects degraded (12.383%) 2021-03-15T22:26:58.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:58 smithi076 conmon[56514]: cluster 2021-03-15T22:26:57.150445+0000 mon.a (mon.0) 1418 : cluster [DBG] osdmap e502: 8 total, 7 up, 7 in 2021-03-15T22:26:58.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:58 smithi076 conmon[56514]: audit 2021-03-15T22:26:57.202063+0000 mon.c (mon.1) 43 : audit [DBG] from='client.? 172.21.15.76:0/3190288941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:58.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:58 smithi076 conmon[62156]: cluster 2021-03-15T22:26:57.077022+0000 mgr.y (mgr.14144) 691 : cluster [DBG] pgmap v879: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 456 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 947 KiB/s wr, 12 op/s; 198/1599 objects degraded (12.383%) 2021-03-15T22:26:58.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:58 smithi076 conmon[62156]: cluster 2021-03-15T22:26:57.150445+0000 mon.a (mon.0) 1418 : cluster [DBG] osdmap e502: 8 total, 7 up, 7 in 2021-03-15T22:26:58.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:58 smithi076 conmon[62156]: audit 2021-03-15T22:26:57.202063+0000 mon.c (mon.1) 43 : audit [DBG] from='client.? 172.21.15.76:0/3190288941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:58.271 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:58.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:58.294 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:58.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:58.396 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:26:58.418 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:58.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:58.420 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:26:58.521 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:58 smithi125 conmon[56739]: cluster 2021-03-15T22:26:57.077022+0000 mgr.y (mgr.14144) 691 : cluster [DBG] pgmap v879: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 456 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 947 KiB/s wr, 12 op/s; 198/1599 objects degraded (12.383%) 2021-03-15T22:26:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:58 smithi125 conmon[56739]: cluster 2021-03-15T22:26:57.150445+0000 mon.a (mon.0) 1418 : cluster [DBG] osdmap e502: 8 total, 7 up, 7 in 2021-03-15T22:26:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:58 smithi125 conmon[56739]: audit 2021-03-15T22:26:57.202063+0000 mon.c (mon.1) 43 : audit [DBG] from='client.? 172.21.15.76:0/3190288941' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:26:58.543 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:58.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:58.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:58.646 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:58.668 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:58.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:58.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:58.771 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:58.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:58.794 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:58.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:58.873 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -n osd.2 --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 3.1f 2021-03-15T22:26:58.896 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:26:58.938 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:58.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:58.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:26:59.041 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:59.065 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:59.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:59.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:59.088 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/osd.2/config 2021-03-15T22:26:59.142 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:26:59.143 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:26:59.162 INFO:tasks.rados.rados.0.smithi076.stdout:1892: done (0 left) 2021-03-15T22:26:59.162 INFO:tasks.rados.rados.0.smithi076.stdout:1893: write (excl) oid 42 current snap is 211 2021-03-15T22:26:59.163 INFO:tasks.rados.rados.0.smithi076.stdout:1893: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2021-03-15T22:26:59.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:26:59.170 INFO:tasks.rados.rados.0.smithi076.stdout:1893: writing smithi07694878-42 from 776655 to 1339951 tid 1 2021-03-15T22:26:59.178 INFO:tasks.rados.rados.0.smithi076.stdout:1893: writing smithi07694878-42 from 2047702 to 2667937 tid 2 2021-03-15T22:26:59.181 INFO:tasks.rados.rados.0.smithi076.stdout:1893: writing smithi07694878-42 from 3095349 to 3176655 tid 3 2021-03-15T22:26:59.181 INFO:tasks.rados.rados.0.smithi076.stdout:1894: read oid 50 snap 204 2021-03-15T22:26:59.181 INFO:tasks.rados.rados.0.smithi076.stdout:1894: expect (ObjNum 598 snap 183 seq_num 598) 2021-03-15T22:26:59.182 INFO:tasks.rados.rados.0.smithi076.stdout:1895: snap_remove snap 200 2021-03-15T22:26:59.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:59.194 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:59.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:26:59.284 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:26:59.284 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:26:59.285 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:26:59.285 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:26:59.296 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:59.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:59.322 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:59.323 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:59.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:59.449 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:59.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:59.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:59 smithi076 conmon[56514]: cluster 2021-03-15T22:26:58.151470+0000 mon.a (mon 2021-03-15T22:26:59.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:26:59 smithi076 conmon[56514]: .0) 1419 : cluster [DBG] osdmap e503: 8 total, 7 up, 7 in 2021-03-15T22:26:59.488 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:59 smithi076 conmon[62156]: cluster 2021-03-15T22:26:58.151470+0000 2021-03-15T22:26:59.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:26:59 smithi076 conmon[62156]: mon.a (mon.0) 1419 : cluster [DBG] osdmap e503: 8 total, 7 up, 7 in 2021-03-15T22:26:59.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:59 smithi125 conmon[56739]: cluster 2021-03-15T22:26:58.151470+0000 mon.a ( 2021-03-15T22:26:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:26:59 smithi125 conmon[56739]: mon.0) 1419 : cluster [DBG] osdmap e503: 8 total, 7 up, 7 in 2021-03-15T22:26:59.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:26:59.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:59.575 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:59.576 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:26:59.677 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:26:59.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:59.701 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:26:59.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:26:59.803 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:26:59.824 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:59.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:59.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:26:59.927 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-15T22:26:59.928 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:26:59.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:26:59.951 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:26:59.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:00.052 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:00.076 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:00.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:00.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:00.136 INFO:tasks.rados.rados.0.smithi076.stdout:1895: done (2 left) 2021-03-15T22:27:00.136 INFO:tasks.rados.rados.0.smithi076.stdout:1896: snap_remove snap 190 2021-03-15T22:27:00.175 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:00 smithi125 conmon[56739]: cluster 2021-03-15T22:26:59.077449+0000 2021-03-15T22:27:00.175 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:00 smithi125 conmon[56739]: mgr.y (mgr.14144) 692 : cluster [DBG] pgmap v882: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 456 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 582 KiB/s wr, 11 op/s; 198/1599 objects degraded (12.383%) 2021-03-15T22:27:00.176 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:00 smithi125 conmon[56739]: cluster 2021-03-15T22:26:59.156373+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e504: 8 total, 7 up, 7 in 2021-03-15T22:27:00.176 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:00 smithi125 conmon[56739]: audit 2021-03-15T22:26:59.794524+0000 mgr.y (mgr.14144) 693 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:00.176 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:00 smithi125 conmon[56739]: cluster 2021-03-15T22:27:00.127611+0000 mon.a (mon.0) 1421 : cluster [WRN] Health check update: Degraded data redundancy: 198/1599 objects degraded (12.383%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:00.176 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:00 smithi125 conmon[56739]: cluster 2021-03-15T22:27:00.133195+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e505: 8 total, 7 up, 7 in 2021-03-15T22:27:00.177 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:26:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:27:00.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:00.210 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:00.210 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:00.258 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:00.311 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:00.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:00.335 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:00.335 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:00.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:00.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:00.462 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:00.463 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:00.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[62156]: cluster 2021-03-15T22:26:59.077449+0000 mgr.y (mgr.14144) 692 : cluster [DBG] pgmap v882: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 456 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 582 KiB/s wr, 11 op/s; 198/1599 objects degraded (12.383%) 2021-03-15T22:27:00.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[62156]: cluster 2021-03-15T22:26:59.156373+0000 mon.a (mon.0) 1420 : cluster [DBG] osdmap e504: 8 total, 7 up, 7 in 2021-03-15T22:27:00.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[62156]: audit 2021-03-15T22:26:59.794524+0000 mgr.y (mgr.14144) 693 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:00.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[62156]: cluster 2021-03-15T22:27:00.127611+0000 mon.a (mon.0) 1421 : cluster [WRN] Health check update: Degraded data redundancy: 198/1599 objects degraded (12.383%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:00.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[62156]: cluster 2021-03-15T22:27:00.133195+0000 mon.a (mon.0) 1422 : cluster [DBG] osdmap e505: 8 total, 7 up, 7 in 2021-03-15T22:27:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: cluster 2021-03-15T22:27:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: 2021-03-15T22:27:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: 2021 2021-03-15T22:27:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: - 2021-03-15T22:27:00.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: 03 2021-03-15T22:27:00.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: - 2021-03-15T22:27:00.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: 15 2021-03-15T22:27:00.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: T 2021-03-15T22:27:00.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: 22 2021-03-15T22:27:00.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: : 2021-03-15T22:27:00.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: 26 2021-03-15T22:27:00.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: : 2021-03-15T22:27:00.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: 59.077449 2021-03-15T22:27:00.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 692 : cluster [DBG] pgmap v882: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 456 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 582 KiB/s wr, 11 op/s; 198/1599 objects degraded (12.383%) 2021-03-15T22:27:00.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:27:00.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: T22:26:59.156373+0000 mon.a (mon.0) 1420 : 2021-03-15T22:27:00.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: cluster [DBG] osdmap e504: 8 total, 7 up, 7 in 2021-03-15T22:27:00.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: audit 2021-03-15 2021-03-15T22:27:00.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: T22:26:59.794524+0000 mgr.y (mgr. 2021-03-15T22:27:00.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: 14144) 693 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:00.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:27:00.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: -03-15T22:27:00.127611+0000 mon.a (mon.0) 1421 2021-03-15T22:27:00.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: : cluster [WRN] Health check update: Degraded data redundancy: 198/1599 objects degraded (12.383%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:00.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: cluster 2021-03-15T22:27 2021-03-15T22:27:00.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: :00.133195+0000 mon.a (mon.0) 2021-03-15T22:27:00.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:00 smithi076 conmon[56514]: 1422 : cluster [DBG] osdmap e505: 8 total, 7 up, 7 in 2021-03-15T22:27:00.564 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:00.587 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:00.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:00.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:00.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:00.713 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:00.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:00.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:00.816 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:00.840 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:00.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:00.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:00.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:00.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:00.966 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:00.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:01.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:01.089 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:01.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:01.144 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:27:01.153 INFO:tasks.rados.rados.0.smithi076.stdout:1896: done (2 left) 2021-03-15T22:27:01.154 INFO:tasks.rados.rados.0.smithi076.stdout:1897: write oid 40 current snap is 211 2021-03-15T22:27:01.154 INFO:tasks.rados.rados.0.smithi076.stdout:1897: seq_num 666 ranges {793462=638545,2106184=763893} 2021-03-15T22:27:01.157 INFO:tasks.rados.rados.0.smithi076.stdout:1897: writing smithi07694878-40 from 793462 to 1432007 tid 1 2021-03-15T22:27:01.158 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:27:01.167 INFO:tasks.rados.rados.0.smithi076.stdout:1897: writing smithi07694878-40 from 2106184 to 2870077 tid 2 2021-03-15T22:27:01.167 INFO:tasks.rados.rados.0.smithi076.stdout:1898: write oid 26 current snap is 211 2021-03-15T22:27:01.167 INFO:tasks.rados.rados.0.smithi076.stdout:1898: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2021-03-15T22:27:01.174 INFO:tasks.rados.rados.0.smithi076.stdout:1898: writing smithi07694878-26 from 410269 to 1124063 tid 1 2021-03-15T22:27:01.180 INFO:tasks.rados.rados.0.smithi076.stdout:1898: writing smithi07694878-26 from 1648313 to 2155864 tid 2 2021-03-15T22:27:01.187 INFO:tasks.rados.rados.0.smithi076.stdout:1898: writing smithi07694878-26 from 2917489 to 3210269 tid 3 2021-03-15T22:27:01.188 INFO:tasks.rados.rados.0.smithi076.stdout:1899: copy_from oid 8 from oid 15 current snap is 211 2021-03-15T22:27:01.188 INFO:tasks.rados.rados.0.smithi076.stdout:1900: snap_create 2021-03-15T22:27:01.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:01.194 INFO:tasks.rados.rados.0.smithi076.stdout:1893: finishing write tid 1 to smithi07694878-42 2021-03-15T22:27:01.195 INFO:tasks.rados.rados.0.smithi076.stdout:1894: expect (ObjNum 598 snap 183 seq_num 598) 2021-03-15T22:27:01.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:01.228 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:01.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:01.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: audit 2021-03-15T22:27:00.204900+0000 mon.a ( 2021-03-15T22:27:01.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: mon.0) 1423 : audit [INF] from='client.? 172.21.15.76:0/578267005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:01.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: audit 2021-03-15T22:27:01.093819+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:27:01.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: audit 2021-03-15T22:27:01.096744+0000 mon.a (mon.0) 1425 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:27:01.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:27:01.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: 15T22:27:01.136786+0000 mon.a (mon.0) 1426 : cluster [WRN] 2021-03-15T22:27:01.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:01.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:27:01.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: 15T22:27:01.143549+0000 mon.a (mon.0) 2021-03-15T22:27:01.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: 1427 : audit [INF] from='client.? 172.21.15.76:0/578267005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:01.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: cluster 2021-03-15T22:27:01.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: 2021-03-15T22:27:01.143605+0000 2021-03-15T22:27:01.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: mon.a (mon.0) 1428 : cluster [DBG] 2021-03-15T22:27:01.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[56514]: osdmap e506: 8 total, 7 up, 7 in 2021-03-15T22:27:01.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: audit 2021-03-15T22:27:01.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 2021-03-15T22:27:01.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 2021 2021-03-15T22:27:01.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: - 2021-03-15T22:27:01.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 03 2021-03-15T22:27:01.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: - 2021-03-15T22:27:01.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 15 2021-03-15T22:27:01.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: T 2021-03-15T22:27:01.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 22 2021-03-15T22:27:01.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: : 2021-03-15T22:27:01.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 27 2021-03-15T22:27:01.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: : 2021-03-15T22:27:01.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 00 2021-03-15T22:27:01.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: . 2021-03-15T22:27:01.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 204900 2021-03-15T22:27:01.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: +0000 2021-03-15T22:27:01.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 2021-03-15T22:27:01.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: mon.a 2021-03-15T22:27:01.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: ( 2021-03-15T22:27:01.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: mon 2021-03-15T22:27:01.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: . 2021-03-15T22:27:01.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 0 2021-03-15T22:27:01.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: ) 2021-03-15T22:27:01.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 1423 2021-03-15T22:27:01.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: : 2021-03-15T22:27:01.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: audit 2021-03-15T22:27:01.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 2021-03-15T22:27:01.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: [INF] 2021-03-15T22:27:01.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 2021-03-15T22:27:01.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/578267005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:01.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 2021-03-15T22:27:01.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: audit 2021-03-15T22:27:01.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 2021-03-15T22:27: 2021-03-15T22:27:01.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 01.093819+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:27:01.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: audit 2021-03-15T22:27:01.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 2021-03-15T22:27:01.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:27:01.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: :27:01.096744+0000 mon.a (mon.0) 1425 : audit 2021-03-15T22:27:01.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:27:01.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:27:01.280 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 22:27:01.136786+0000 mon.a (mon. 2021-03-15T22:27:01.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 0) 1426 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:01.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:27:01.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 03-15T22:27:01.143549 2021-03-15T22:27:01.281 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1427 : audit 2021-03-15T22:27:01.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/578267005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:01.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: cluster 2021-03-15T22:27: 2021-03-15T22:27:01.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: 01.143605+0000 mon.a (mon.0) 1428 2021-03-15T22:27:01.282 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:01 smithi076 conmon[62156]: : cluster [DBG] osdmap e506: 8 total, 7 up, 7 in 2021-03-15T22:27:01.288 INFO:tasks.rados.rados.0.smithi076.stdout:1893: finishing write tid 2 to smithi07694878-42 2021-03-15T22:27:01.288 INFO:tasks.rados.rados.0.smithi076.stdout:1893: finishing write tid 3 to smithi07694878-42 2021-03-15T22:27:01.289 INFO:tasks.rados.rados.0.smithi076.stdout:1893: finishing write tid 4 to smithi07694878-42 2021-03-15T22:27:01.289 INFO:tasks.rados.rados.0.smithi076.stdout:1893: finishing write tid 5 to smithi07694878-42 2021-03-15T22:27:01.289 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 1333 (ObjNum 665 snap 211 seq_num 665) dirty exists 2021-03-15T22:27:01.290 INFO:tasks.rados.rados.0.smithi076.stdout:1893: left oid 42 (ObjNum 665 snap 211 seq_num 665) 2021-03-15T22:27:01.290 INFO:tasks.rados.rados.0.smithi076.stdout:1897: finishing write tid 1 to smithi07694878-40 2021-03-15T22:27:01.290 INFO:tasks.rados.rados.0.smithi076.stdout:1898: finishing write tid 1 to smithi07694878-26 2021-03-15T22:27:01.290 INFO:tasks.rados.rados.0.smithi076.stdout:1898: finishing write tid 2 to smithi07694878-26 2021-03-15T22:27:01.291 INFO:tasks.rados.rados.0.smithi076.stdout:1897: finishing write tid 2 to smithi07694878-40 2021-03-15T22:27:01.291 INFO:tasks.rados.rados.0.smithi076.stdout:1897: finishing write tid 3 to smithi07694878-40 2021-03-15T22:27:01.291 INFO:tasks.rados.rados.0.smithi076.stdout:1897: finishing write tid 4 to smithi07694878-40 2021-03-15T22:27:01.291 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1345 (ObjNum 666 snap 211 seq_num 666) dirty exists 2021-03-15T22:27:01.292 INFO:tasks.rados.rados.0.smithi076.stdout:1897: left oid 40 (ObjNum 666 snap 211 seq_num 666) 2021-03-15T22:27:01.292 INFO:tasks.rados.rados.0.smithi076.stdout:1898: finishing write tid 3 to smithi07694878-26 2021-03-15T22:27:01.292 INFO:tasks.rados.rados.0.smithi076.stdout:1898: finishing write tid 4 to smithi07694878-26 2021-03-15T22:27:01.292 INFO:tasks.rados.rados.0.smithi076.stdout:1898: finishing write tid 5 to smithi07694878-26 2021-03-15T22:27:01.292 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 806 (ObjNum 667 snap 211 seq_num 667) dirty exists 2021-03-15T22:27:01.293 INFO:tasks.rados.rados.0.smithi076.stdout:1898: left oid 26 (ObjNum 667 snap 211 seq_num 667) 2021-03-15T22:27:01.293 INFO:tasks.rados.rados.0.smithi076.stdout:1899: finishing copy_from to smithi07694878-8 2021-03-15T22:27:01.293 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1169 (ObjNum 649 snap 201 seq_num 649) dirty exists 2021-03-15T22:27:01.293 INFO:tasks.rados.rados.0.smithi076.stdout:1899: finishing copy_from racing read to smithi07694878-8 2021-03-15T22:27:01.330 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:01.355 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:01.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:01.412 INFO:teuthology.orchestra.run.smithi076.stdout: marking collection for removal 2021-03-15T22:27:01.412 INFO:teuthology.orchestra.run.smithi076.stdout:setting '_remove' omap key 2021-03-15T22:27:01.412 INFO:teuthology.orchestra.run.smithi076.stdout:finish_remove_pgs 3.1f_head removing 3.1f 2021-03-15T22:27:01.412 INFO:teuthology.orchestra.run.smithi076.stdout:Remove successful 2021-03-15T22:27:01.438 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:01 smithi125 conmon[56739]: audit 2021-03-15T22:27:00.204900+0000 mon.a (mon.0) 1423 : audit [INF] from='client.? 172.21.15.76:0/578267005' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:01.438 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:01 smithi125 conmon[56739]: audit 2021-03-15T22:27:01.093819+0000 mon.a (mon.0) 1424 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:27:01.439 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:01 smithi125 conmon[56739]: audit 2021-03-15T22:27:01.096744+0000 mon.a (mon.0) 1425 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:27:01.439 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:01 smithi125 conmon[56739]: cluster 2021-03-15T22:27:01.136786+0000 mon.a (mon.0) 1426 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:01.439 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:01 smithi125 conmon[56739]: audit 2021-03-15T22:27:01.143549+0000 mon.a (mon.0) 1427 : audit [INF] from='client.? 172.21.15.76:0/578267005' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:01.439 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:01 smithi125 conmon[56739]: cluster 2021-03-15T22:27:01.143605+0000 mon.a (mon.0) 1428 : cluster [DBG] osdmap e506: 8 total, 7 up, 7 in 2021-03-15T22:27:01.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:01.480 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:01.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:01.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:01.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:01.606 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:01.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:01.677 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:27:01.677 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:27:01.677 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:27:01.678 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:27:01.678 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:27:01.709 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:01.732 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:01.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:01.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:01.835 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:01.857 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:01.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:01.859 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:01.937 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:01.960 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:01.981 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:01.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:01.983 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:02.084 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:02.108 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:02.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:02.110 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:02.197 INFO:tasks.rados.rados.0.smithi076.stdout:1893: done (5 left) 2021-03-15T22:27:02.197 INFO:tasks.rados.rados.0.smithi076.stdout:1894: done (4 left) 2021-03-15T22:27:02.197 INFO:tasks.rados.rados.0.smithi076.stdout:1897: done (3 left) 2021-03-15T22:27:02.198 INFO:tasks.rados.rados.0.smithi076.stdout:1898: done (2 left) 2021-03-15T22:27:02.198 INFO:tasks.rados.rados.0.smithi076.stdout:1899: done (1 left) 2021-03-15T22:27:02.198 INFO:tasks.rados.rados.0.smithi076.stdout:1900: done (0 left) 2021-03-15T22:27:02.198 INFO:tasks.rados.rados.0.smithi076.stdout:1901: write (excl) oid 5 current snap is 212 2021-03-15T22:27:02.198 INFO:tasks.rados.rados.0.smithi076.stdout:1901: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2021-03-15T22:27:02.203 INFO:tasks.rados.rados.0.smithi076.stdout:1901: writing smithi07694878-5 from 427076 to 1216119 tid 1 2021-03-15T22:27:02.210 INFO:tasks.rados.rados.0.smithi076.stdout:1901: writing smithi07694878-5 from 1706795 to 2474357 tid 2 2021-03-15T22:27:02.211 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:02.215 INFO:tasks.rados.rados.0.smithi076.stdout:1901: writing smithi07694878-5 from 2944912 to 3227076 tid 3 2021-03-15T22:27:02.215 INFO:tasks.rados.rados.0.smithi076.stdout:1902: rollback oid 10 current snap is 212 2021-03-15T22:27:02.215 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 10 to 189 2021-03-15T22:27:02.215 INFO:tasks.rados.rados.0.smithi076.stdout:1901: finishing write tid 1 to smithi07694878-5 2021-03-15T22:27:02.216 INFO:tasks.rados.rados.0.smithi076.stdout:1903: write (excl) oid 47 current snap is 212 2021-03-15T22:27:02.216 INFO:tasks.rados.rados.0.smithi076.stdout:1903: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2021-03-15T22:27:02.219 INFO:tasks.rados.rados.0.smithi076.stdout:1903: writing smithi07694878-47 from 443883 to 908175 tid 1 2021-03-15T22:27:02.226 INFO:tasks.rados.rados.0.smithi076.stdout:1903: writing smithi07694878-47 from 1365277 to 1876497 tid 2 2021-03-15T22:27:02.234 INFO:tasks.rados.rados.0.smithi076.stdout:1903: writing smithi07694878-47 from 2572335 to 3243883 tid 3 2021-03-15T22:27:02.235 INFO:tasks.rados.rados.0.smithi076.stdout:1904: write oid 44 current snap is 212 2021-03-15T22:27:02.235 INFO:tasks.rados.rados.0.smithi076.stdout:1901: finishing write tid 2 to smithi07694878-5 2021-03-15T22:27:02.235 INFO:tasks.rados.rados.0.smithi076.stdout:1904: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2021-03-15T22:27:02.241 INFO:tasks.rados.rados.0.smithi076.stdout:1904: writing smithi07694878-44 from 460690 to 1116584 tid 1 2021-03-15T22:27:02.244 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:02.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:02.249 INFO:tasks.rados.rados.0.smithi076.stdout:1904: writing smithi07694878-44 from 1540112 to 2311343 tid 2 2021-03-15T22:27:02.256 INFO:tasks.rados.rados.0.smithi076.stdout:1904: writing smithi07694878-44 from 2716111 to 3255889 tid 3 2021-03-15T22:27:02.256 INFO:tasks.rados.rados.0.smithi076.stdout:1905: read oid 3 snap -1 2021-03-15T22:27:02.257 INFO:tasks.rados.rados.0.smithi076.stdout:1905: expect (ObjNum 639 snap 200 seq_num 639) 2021-03-15T22:27:02.257 INFO:tasks.rados.rados.0.smithi076.stdout:1906: read oid 34 snap -1 2021-03-15T22:27:02.257 INFO:tasks.rados.rados.0.smithi076.stdout:1906: expect (ObjNum 609 snap 189 seq_num 609) 2021-03-15T22:27:02.257 INFO:tasks.rados.rados.0.smithi076.stdout:1907: snap_create 2021-03-15T22:27:02.257 INFO:tasks.rados.rados.0.smithi076.stdout:1902: finishing rollback tid 1 to smithi07694878-10 2021-03-15T22:27:02.258 INFO:tasks.rados.rados.0.smithi076.stdout:1902: finishing rollback tid 2 to smithi07694878-10 2021-03-15T22:27:02.258 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 541 (ObjNum 444 snap 125 seq_num 444) dirty exists 2021-03-15T22:27:02.258 INFO:tasks.rados.rados.0.smithi076.stdout:1903: finishing write tid 1 to smithi07694878-47 2021-03-15T22:27:02.258 INFO:tasks.rados.rados.0.smithi076.stdout:1901: finishing write tid 3 to smithi07694878-5 2021-03-15T22:27:02.259 INFO:tasks.rados.rados.0.smithi076.stdout:1901: finishing write tid 4 to smithi07694878-5 2021-03-15T22:27:02.259 INFO:tasks.rados.rados.0.smithi076.stdout:1901: finishing write tid 5 to smithi07694878-5 2021-03-15T22:27:02.259 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 605 (ObjNum 668 snap 212 seq_num 668) dirty exists 2021-03-15T22:27:02.259 INFO:tasks.rados.rados.0.smithi076.stdout:1901: left oid 5 (ObjNum 668 snap 212 seq_num 668) 2021-03-15T22:27:02.259 INFO:tasks.rados.rados.0.smithi076.stdout:1903: finishing write tid 2 to smithi07694878-47 2021-03-15T22:27:02.260 INFO:tasks.rados.rados.0.smithi076.stdout:1903: finishing write tid 3 to smithi07694878-47 2021-03-15T22:27:02.260 INFO:tasks.rados.rados.0.smithi076.stdout:1903: finishing write tid 4 to smithi07694878-47 2021-03-15T22:27:02.260 INFO:tasks.rados.rados.0.smithi076.stdout:1903: finishing write tid 5 to smithi07694878-47 2021-03-15T22:27:02.260 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1173 (ObjNum 669 snap 212 seq_num 669) dirty exists 2021-03-15T22:27:02.261 INFO:tasks.rados.rados.0.smithi076.stdout:1903: left oid 47 (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:27:02.261 INFO:tasks.rados.rados.0.smithi076.stdout:1904: finishing write tid 1 to smithi07694878-44 2021-03-15T22:27:02.261 INFO:tasks.rados.rados.0.smithi076.stdout:1904: finishing write tid 2 to smithi07694878-44 2021-03-15T22:27:02.262 INFO:tasks.rados.rados.0.smithi076.stdout:1904: finishing write tid 3 to smithi07694878-44 2021-03-15T22:27:02.262 INFO:tasks.rados.rados.0.smithi076.stdout:1904: finishing write tid 4 to smithi07694878-44 2021-03-15T22:27:02.262 INFO:tasks.rados.rados.0.smithi076.stdout:1905: expect (ObjNum 639 snap 200 seq_num 639) 2021-03-15T22:27:02.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-15T22:27:02.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:02.369 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:02.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:02.379 INFO:tasks.rados.rados.0.smithi076.stdout:1904: finishing write tid 5 to smithi07694878-44 2021-03-15T22:27:02.380 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1177 (ObjNum 670 snap 212 seq_num 670) dirty exists 2021-03-15T22:27:02.380 INFO:tasks.rados.rados.0.smithi076.stdout:1904: left oid 44 (ObjNum 670 snap 212 seq_num 670) 2021-03-15T22:27:02.380 INFO:tasks.rados.rados.0.smithi076.stdout:1906: expect (ObjNum 609 snap 189 seq_num 609) 2021-03-15T22:27:02.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:02 smithi076 conmon[62156]: cluster 2021-03-15T22:27:00.658064+0000 osd.4 (osd.4) 103 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:02.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:02 smithi076 conmon[62156]: cluster 2021-03-15T22:27:00.658710+0000 osd.4 (osd.4) 104 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:02.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:02 smithi076 conmon[62156]: cluster 2021-03-15T22:27:01.077894+0000 mgr.y (mgr.14144) 694 : cluster [DBG] pgmap v885: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 452 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 3 op/s; 198/1593 objects degraded (12.429%) 2021-03-15T22:27:02.438 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:02 smithi076 conmon[56514]: cluster 2021-03-15T22:27:00.658064+0000 osd.4 (osd.4) 103 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:02.439 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:02 smithi076 conmon[56514]: cluster 2021-03-15T22:27:00.658710+0000 osd.4 (osd.4) 104 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:02.439 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:02 smithi076 conmon[56514]: cluster 2021-03-15T22:27:01.077894+0000 mgr.y (mgr.14144) 694 : cluster [DBG] pgmap v885: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 452 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 3 op/s; 198/1593 objects degraded (12.429%) 2021-03-15T22:27:02.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:02.482 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -n osd.2 --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.1f.2 2021-03-15T22:27:02.493 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:02.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:02.494 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:02.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:02 smithi125 conmon[56739]: cluster 2021-03-15T22:27:00.658064+0000 osd.4 ( 2021-03-15T22:27:02.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:02 smithi125 conmon[56739]: osd.4) 103 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:02 smithi125 conmon[56739]: cluster 2021-03-15T22:27:00.658710+0000 osd.4 (osd.4) 104 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:02 smithi125 conmon[56739]: cluster 2021-03-15T22:27:01.077894+0000 mgr.y (mgr.14144) 694 : cluster [DBG] pgmap v885: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 452 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 3.6 MiB/s rd, 3 op/s; 198/1593 objects degraded (12.429%) 2021-03-15T22:27:02.595 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:02.596 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:02.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:02.619 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:02.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:02.721 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:02.742 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/osd.2/config 2021-03-15T22:27:02.743 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:02.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:02.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:02.846 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:02.868 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:02.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:02.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:02.971 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:02.994 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:02.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:02.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:03.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:03.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:03.122 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:03.123 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:03.201 INFO:tasks.rados.rados.0.smithi076.stdout:1901: done (6 left) 2021-03-15T22:27:03.201 INFO:tasks.rados.rados.0.smithi076.stdout:1902: done (5 left) 2021-03-15T22:27:03.201 INFO:tasks.rados.rados.0.smithi076.stdout:1903: done (4 left) 2021-03-15T22:27:03.201 INFO:tasks.rados.rados.0.smithi076.stdout:1904: done (3 left) 2021-03-15T22:27:03.202 INFO:tasks.rados.rados.0.smithi076.stdout:1905: done (2 left) 2021-03-15T22:27:03.202 INFO:tasks.rados.rados.0.smithi076.stdout:1906: done (1 left) 2021-03-15T22:27:03.202 INFO:tasks.rados.rados.0.smithi076.stdout:1907: done (0 left) 2021-03-15T22:27:03.202 INFO:tasks.rados.rados.0.smithi076.stdout:1908: write (excl) oid 12 current snap is 213 2021-03-15T22:27:03.203 INFO:tasks.rados.rados.0.smithi076.stdout:1908: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2021-03-15T22:27:03.209 INFO:tasks.rados.rados.0.smithi076.stdout:1908: writing smithi07694878-12 from 477497 to 1208640 tid 1 2021-03-15T22:27:03.217 INFO:tasks.rados.rados.0.smithi076.stdout:1908: writing smithi07694878-12 from 1998594 to 2513483 tid 2 2021-03-15T22:27:03.222 INFO:tasks.rados.rados.0.smithi076.stdout:1908: writing smithi07694878-12 from 3143534 to 3277497 tid 3 2021-03-15T22:27:03.222 INFO:tasks.rados.rados.0.smithi076.stdout:1909: delete oid 30 current snap is 213 2021-03-15T22:27:03.222 INFO:tasks.rados.rados.0.smithi076.stdout:1908: finishing write tid 1 to smithi07694878-12 2021-03-15T22:27:03.224 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:03.226 INFO:tasks.rados.rados.0.smithi076.stdout:1908: finishing write tid 2 to smithi07694878-12 2021-03-15T22:27:03.229 INFO:tasks.rados.rados.0.smithi076.stdout:1908: finishing write tid 3 to smithi07694878-12 2021-03-15T22:27:03.229 INFO:tasks.rados.rados.0.smithi076.stdout:1908: finishing write tid 4 to smithi07694878-12 2021-03-15T22:27:03.230 INFO:tasks.rados.rados.0.smithi076.stdout:1908: finishing write tid 5 to smithi07694878-12 2021-03-15T22:27:03.230 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 1239 (ObjNum 671 snap 213 seq_num 671) dirty exists 2021-03-15T22:27:03.231 INFO:tasks.rados.rados.0.smithi076.stdout:1908: left oid 12 (ObjNum 671 snap 213 seq_num 671) 2021-03-15T22:27:03.231 INFO:tasks.rados.rados.0.smithi076.stdout:1908: done (1 left) 2021-03-15T22:27:03.231 INFO:tasks.rados.rados.0.smithi076.stdout:1909: done (0 left) 2021-03-15T22:27:03.232 INFO:tasks.rados.rados.0.smithi076.stdout:1910: delete oid 29 current snap is 213 2021-03-15T22:27:03.235 INFO:tasks.rados.rados.0.smithi076.stdout:1910: done (0 left) 2021-03-15T22:27:03.235 INFO:tasks.rados.rados.0.smithi076.stdout:1911: read oid 20 snap -1 2021-03-15T22:27:03.235 INFO:tasks.rados.rados.0.smithi076.stdout:1911: expect deleted 2021-03-15T22:27:03.235 INFO:tasks.rados.rados.0.smithi076.stdout:1912: snap_create 2021-03-15T22:27:03.258 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:03.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:03.305 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:03.407 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:03.428 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:03.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:03.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:03.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:03 smithi076 conmon[56514]: cluster 2021-03-15T22:27:02.192102+0000 mon.a ( 2021-03-15T22:27:03.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:03 smithi076 conmon[56514]: mon.0) 1429 : cluster [DBG] osdmap e507: 8 total, 7 up, 7 in 2021-03-15T22:27:03.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:03 smithi076 conmon[62156]: cluster 2021-03-15T22:27:02.192102+0000 2021-03-15T22:27:03.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:03 smithi076 conmon[62156]: mon.a (mon.0) 1429 : cluster [DBG] osdmap e507: 8 total, 7 up, 7 in 2021-03-15T22:27:03.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:03.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:03 smithi125 conmon[56739]: cluster 2021-03-15T22:27:02.192102+0000 mon.a (mon.0) 1429 : cluster [DBG] osdmap e507: 8 total, 7 up, 7 in 2021-03-15T22:27:03.554 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:03.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:03.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:03.656 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:03.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:03.680 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:03.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:03.781 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:03.805 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:03.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:03.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:03.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:27:03.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:03.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:03.933 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:03.934 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:04.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:04.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:04.059 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:04.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:04.161 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:04.175 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:27:04.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:04.188 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:04.188 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:04.192 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:27:04.206 INFO:tasks.rados.rados.0.smithi076.stdout:1911: done (1 left) 2021-03-15T22:27:04.206 INFO:tasks.rados.rados.0.smithi076.stdout:1912: done (0 left) 2021-03-15T22:27:04.206 INFO:tasks.rados.rados.0.smithi076.stdout:1913: rollback oid 7 current snap is 214 2021-03-15T22:27:04.207 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 7 to 191 2021-03-15T22:27:04.207 INFO:tasks.rados.rados.0.smithi076.stdout:1914: read oid 15 snap -1 2021-03-15T22:27:04.207 INFO:tasks.rados.rados.0.smithi076.stdout:1914: expect (ObjNum 649 snap 201 seq_num 649) 2021-03-15T22:27:04.207 INFO:tasks.rados.rados.0.smithi076.stdout:1915: snap_remove snap 189 2021-03-15T22:27:04.216 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:27:04.251 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:27:04.265 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:27:04.266 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:27:04.270 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:27:04.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:04 smithi125 conmon[56739]: cluster 2021-03-15T22:27:03.078357+0000 mgr.y (mgr.14144) 695 : cluster [DBG] pgmap v888: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 460 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 782 KiB/s rd, 1.4 MiB/s wr, 9 op/s; 199/1602 objects degraded (12.422%) 2021-03-15T22:27:04.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:04 smithi125 conmon[56739]: 2021-03-15T22:27:04.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:04 smithi125 conmon[56739]: cluster 2021-03-15T22:27:03.198997+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e508: 8 total, 7 up, 7 in 2021-03-15T22:27:04.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:04 smithi125 conmon[56739]: audit 2021-03-15T22:27:04.152132+0000 mon.a (mon.0) 1431 : audit [DBG] from='client.? 172.21.15.76:0/104037030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:04.289 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:04.290 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:27:04.308 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:27:04.308 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:27:04.308 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:27:04.309 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:27:04.309 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:27:04.309 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:27:04.309 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:27:04.309 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:27:04.310 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:27:04.313 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:04.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:04.315 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:04.368 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:04 smithi076 conmon[56514]: cluster 2021-03-15T22:27:03.078357+0000 mgr.y (mgr.14144) 695 : cluster [DBG] pgmap v888: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 460 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 782 KiB/s rd, 1.4 MiB/s wr, 9 op/s; 199/1602 objects degraded (12.422%) 2021-03-15T22:27:04.369 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:04 smithi076 conmon[56514]: cluster 2021-03-15T22:27:03.198997+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e508: 8 total, 7 up, 7 in 2021-03-15T22:27:04.369 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:04 smithi076 conmon[56514]: audit 2021-03-15T22:27:04.152132+0000 mon.a (mon.0) 1431 : audit [DBG] from='client.? 172.21.15.76:0/104037030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:04.369 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:04 smithi076 conmon[62156]: cluster 2021-03-15T22:27:03.078357+0000 mgr.y ( 2021-03-15T22:27:04.370 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:04 smithi076 conmon[62156]: mgr.14144) 695 : cluster [DBG] pgmap v888: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 460 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 782 KiB/s rd, 1.4 MiB/s wr, 9 op/s; 199/1602 objects degraded (12.422%) 2021-03-15T22:27:04.370 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:04 smithi076 conmon[62156]: cluster 2021-03-15T22:27:03.198997+0000 mon.a (mon.0) 1430 : cluster [DBG] osdmap e508: 8 total, 7 up, 7 in 2021-03-15T22:27:04.370 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:04 smithi076 conmon[62156]: audit 2021-03-15T22:27:04.152132+0000 mon.a (mon.0) 1431 : audit [DBG] from='client.? 172.21.15.76:0/104037030' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:04.416 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:04.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:04.439 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:04.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:04.541 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:04.563 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:04.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:04.565 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:04.667 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:04.690 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:04.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:04.691 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:04.792 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:04.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:04.816 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:04.817 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:04.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:04.926 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:27:04.940 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:04.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:04.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:05.043 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:05.066 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:05.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:05.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:05.168 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:05.175 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:05.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:05.193 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:05.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:05.201 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:05.213 INFO:tasks.rados.rados.0.smithi076.stdout:1915: done (2 left) 2021-03-15T22:27:05.213 INFO:tasks.rados.rados.0.smithi076.stdout:1916: rollback oid 42 current snap is 214 2021-03-15T22:27:05.213 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 42 to 204 2021-03-15T22:27:05.225 INFO:tasks.rados.rados.0.smithi076.stdout:1913: finishing rollback tid 0 to smithi07694878-7 2021-03-15T22:27:05.225 INFO:tasks.rados.rados.0.smithi076.stdout:1913: finishing rollback tid 1 to smithi07694878-7 2021-03-15T22:27:05.225 INFO:tasks.rados.rados.0.smithi076.stdout:1913: finishing rollback tid 2 to smithi07694878-7 2021-03-15T22:27:05.225 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 1016 (ObjNum 614 snap 190 seq_num 614) dirty exists 2021-03-15T22:27:05.226 INFO:tasks.rados.rados.0.smithi076.stdout:1914: expect (ObjNum 649 snap 201 seq_num 649) 2021-03-15T22:27:05.294 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:05.319 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:05.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:05.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:05.392 INFO:tasks.rados.rados.0.smithi076.stdout:1916: finishing rollback tid 0 to smithi07694878-42 2021-03-15T22:27:05.393 INFO:tasks.rados.rados.0.smithi076.stdout:1916: finishing rollback tid 1 to smithi07694878-42 2021-03-15T22:27:05.393 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 1350 (ObjNum 264900288 snap 21896 seq_num 0) dirty dne 2021-03-15T22:27:05.393 INFO:tasks.rados.rados.0.smithi076.stdout:1913: done (2 left) 2021-03-15T22:27:05.393 INFO:tasks.rados.rados.0.smithi076.stdout:1914: done (1 left) 2021-03-15T22:27:05.394 INFO:tasks.rados.rados.0.smithi076.stdout:1916: done (0 left) 2021-03-15T22:27:05.394 INFO:tasks.rados.rados.0.smithi076.stdout:1917: snap_remove snap 133 2021-03-15T22:27:05.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:05 smithi076 conmon[56514]: cluster 2021-03-15T22:27:04.203652+0000 mon.a (mon. 2021-03-15T22:27:05.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:05 smithi076 conmon[56514]: 0) 1432 : cluster [DBG] osdmap e509: 8 total, 7 up, 7 in 2021-03-15T22:27:05.403 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:05 smithi076 conmon[56514]: cluster 2021-03-15T22:27:05.130229+0000 mon.a (mon.0) 1433 : cluster [WRN] Health check update: Degraded data redundancy: 199/1602 objects degraded (12.422%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:05.404 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:05 smithi076 conmon[62156]: cluster 2021-03-15T22:27:04.203652+0000 2021-03-15T22:27:05.404 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:05 smithi076 conmon[62156]: mon.a (mon.0) 1432 : cluster [DBG] osdmap e509: 8 total, 7 up, 7 in 2021-03-15T22:27:05.404 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:05 smithi076 conmon[62156]: cluster 2021-03-15T22:27: 2021-03-15T22:27:05.405 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:05 smithi076 conmon[62156]: 05.130229+0000 mon.a (mon.0) 1433 : cluster [WRN] Health check update: Degraded data redundancy: 199/1602 objects degraded (12.422%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:05.405 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:27:05.406 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:27:05.422 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:05.444 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:05.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:05.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:05.462 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:05 smithi125 conmon[56739]: cluster 2021-03-15T22:27:04.203652+0000 mon.a (mon.0) 1432 : cluster [DBG] osdmap e509: 8 total, 7 up, 7 in 2021-03-15T22:27:05.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:05 smithi125 conmon[56739]: cluster 2021-03-15T22:27:05.130229+0000 mon.a (mon.0) 1433 : cluster [WRN] Health check update: Degraded data redundancy: 199/1602 objects degraded (12.422%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:05.546 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:05.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:05.570 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:05.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:05.671 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:05.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:05.695 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:05.695 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:05.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:05.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:05.820 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:05.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:05.922 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:05.944 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:05.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:05.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:06.046 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:06.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:06.072 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:06.073 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:06.114 INFO:teuthology.orchestra.run.smithi076.stdout:get_pg_num_history pg_num_history pg_num_history(e489 pg_nums {1={21=1},2={53=3},3={56=32},4={58=32},5={60=32,146=31,150=30,154=29,158=28,162=27,166=26,170=25,174=24,178=23,182=22},6={62=32},7={91=16}} deleted_pools ) 2021-03-15T22:27:06.115 INFO:teuthology.orchestra.run.smithi076.stdout:Importing pgid 3.1f 2021-03-15T22:27:06.115 INFO:teuthology.orchestra.run.smithi076.stdout:Write #3:fcb6c186:::zone_info.38e1d1be-4dea-4a9a-b9e9-57f557eecf75:head# 2021-03-15T22:27:06.115 INFO:teuthology.orchestra.run.smithi076.stdout:snapset 0=[]:{} 2021-03-15T22:27:06.116 INFO:teuthology.orchestra.run.smithi076.stdout:write_pg epoch 452 info 3.1f( v 57'1 (0'0,57'1] local-lis/les=355/356 n=1 ec=56/56 lis/c=355/355 les/c/f=356/356/0 sis=355) 2021-03-15T22:27:06.116 INFO:teuthology.orchestra.run.smithi076.stdout:Import successful 2021-03-15T22:27:06.174 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:06.196 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:06.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:06.197 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:06.219 INFO:tasks.rados.rados.0.smithi076.stdout:1917: done (0 left) 2021-03-15T22:27:06.219 INFO:tasks.rados.rados.0.smithi076.stdout:1918: read oid 6 snap 172 2021-03-15T22:27:06.220 INFO:tasks.rados.rados.0.smithi076.stdout:1918: expect (ObjNum 538 snap 171 seq_num 538) 2021-03-15T22:27:06.220 INFO:tasks.rados.rados.0.smithi076.stdout:1919: write (excl) oid 22 current snap is 214 2021-03-15T22:27:06.220 INFO:tasks.rados.rados.0.smithi076.stdout:1919: seq_num 674 ranges {527918=556890,1657687=778569,3225803=102115} 2021-03-15T22:27:06.226 INFO:tasks.rados.rados.0.smithi076.stdout:1919: writing smithi07694878-22 from 527918 to 1084808 tid 1 2021-03-15T22:27:06.234 INFO:tasks.rados.rados.0.smithi076.stdout:1919: writing smithi07694878-22 from 1657687 to 2436256 tid 2 2021-03-15T22:27:06.240 INFO:tasks.rados.rados.0.smithi076.stdout:1919: writing smithi07694878-22 from 3225803 to 3327918 tid 3 2021-03-15T22:27:06.240 INFO:tasks.rados.rados.0.smithi076.stdout:1920: read oid 4 snap -1 2021-03-15T22:27:06.240 INFO:tasks.rados.rados.0.smithi076.stdout:1920: expect (ObjNum 624 snap 196 seq_num 624) 2021-03-15T22:27:06.240 INFO:tasks.rados.rados.0.smithi076.stdout:1921: write oid 21 current snap is 214 2021-03-15T22:27:06.241 INFO:tasks.rados.rados.0.smithi076.stdout:1921: seq_num 675 ranges {544725=632139,1716169=522227,2736873=512491} 2021-03-15T22:27:06.246 INFO:tasks.rados.rados.0.smithi076.stdout:1921: writing smithi07694878-21 from 544725 to 1176864 tid 1 2021-03-15T22:27:06.253 INFO:tasks.rados.rados.0.smithi076.stdout:1921: writing smithi07694878-21 from 1716169 to 2238396 tid 2 2021-03-15T22:27:06.258 INFO:tasks.rados.rados.0.smithi076.stdout:1921: writing smithi07694878-21 from 2736873 to 3249364 tid 3 2021-03-15T22:27:06.258 INFO:tasks.rados.rados.0.smithi076.stdout:1922: read oid 41 snap 172 2021-03-15T22:27:06.259 INFO:tasks.rados.rados.0.smithi076.stdout:1922: expect (ObjNum 509 snap 152 seq_num 509) 2021-03-15T22:27:06.259 INFO:tasks.rados.rados.0.smithi076.stdout:1923: snap_remove snap 201 2021-03-15T22:27:06.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:06.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:06.322 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:06.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:06.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:06.447 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:06.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:06.449 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:06.470 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:06 smithi125 conmon[56739]: cluster 2021-03-15T22:27:05.078928+0000 mgr.y ( 2021-03-15T22:27:06.471 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:06 smithi125 conmon[56739]: mgr.14144) 696 : cluster [DBG] pgmap v891: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 467 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 200/1611 objects degraded (12.415%) 2021-03-15T22:27:06.471 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:06 smithi125 conmon[56739]: cluster 2021-03-15T22:27:05.210420+0000 mon.a (mon.0) 1434 : cluster [DBG] osdmap e510: 8 total, 7 up, 7 in 2021-03-15T22:27:06.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:06 smithi076 conmon[56514]: cluster 2021-03-15T22:27:05.078928+0000 mgr.y (mgr.14144) 696 : cluster [DBG] pgmap v891: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 467 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 200/1611 objects degraded (12.415%) 2021-03-15T22:27:06.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:06 smithi076 conmon[56514]: cluster 2021-03-15T22:27:05.210420+0000 mon.a (mon.0) 1434 : cluster [DBG] osdmap e510: 8 total, 7 up, 7 in 2021-03-15T22:27:06.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:06 smithi076 conmon[62156]: cluster 2021-03-15T22:27:05.078928+0000 mgr.y (mgr.14144) 696 : cluster [DBG] pgmap v891: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 467 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 4.9 MiB/s rd, 2.6 MiB/s wr, 16 op/s; 200/1611 objects degraded (12.415%) 2021-03-15T22:27:06.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:06 smithi076 conmon[62156]: cluster 2021-03-15T22:27:05.210420+0000 mon.a (mon.0) 1434 : cluster [DBG] osdmap e510: 8 total, 7 up, 7 in 2021-03-15T22:27:06.551 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:06.574 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:06.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:06.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:06.676 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:06.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:06.701 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:06.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:06.743 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:27:06.743 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:27:06.803 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:06.827 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:06.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:06.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:06.930 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:06.953 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:06.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:07.055 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:07.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:07.078 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:07.079 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:07.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:07.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:07.204 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:07.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:07.234 INFO:tasks.rados.rados.0.smithi076.stdout:1923: done (5 left) 2021-03-15T22:27:07.234 INFO:tasks.rados.rados.0.smithi076.stdout:1924: write oid 45 current snap is 214 2021-03-15T22:27:07.234 INFO:tasks.rados.rados.0.smithi076.stdout:1918: expect (ObjNum 538 snap 171 seq_num 538) 2021-03-15T22:27:07.306 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:07.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:07.330 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:07.330 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:07.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-15T22:27:07.407 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:27:07.432 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:07.435 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c/exp.3.1f.2 2021-03-15T22:27:07.436 INFO:tasks.rados.rados.0.smithi076.stdout:1919: finishing write tid 1 to smithi07694878-22 2021-03-15T22:27:07.436 INFO:tasks.rados.rados.0.smithi076.stdout:1920: expect (ObjNum 624 snap 196 seq_num 624) 2021-03-15T22:27:07.466 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:07.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:07.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:07.496 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:07 smithi076 conmon[56514]: cluster 2021-03-15T22:27:06.216703+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e511: 8 total, 7 up, 7 in 2021-03-15T22:27:07.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:07 smithi076 conmon[62156]: cluster 2021-03-15T22:27:06.216703+0000 mon.a (mon.0) 1435 : cluster [DBG] osdmap e511: 8 total, 7 up, 7 in 2021-03-15T22:27:07.520 INFO:tasks.rados.rados.0.smithi076.stdout:1919: finishing write tid 2 to smithi07694878-22 2021-03-15T22:27:07.521 INFO:tasks.rados.rados.0.smithi076.stdout:1919: finishing write tid 3 to smithi07694878-22 2021-03-15T22:27:07.521 INFO:tasks.rados.rados.0.smithi076.stdout:1919: finishing write tid 4 to smithi07694878-22 2021-03-15T22:27:07.522 INFO:tasks.rados.rados.0.smithi076.stdout:1919: finishing write tid 5 to smithi07694878-22 2021-03-15T22:27:07.522 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 1261 (ObjNum 674 snap 214 seq_num 674) dirty exists 2021-03-15T22:27:07.523 INFO:tasks.rados.rados.0.smithi076.stdout:1919: left oid 22 (ObjNum 674 snap 214 seq_num 674) 2021-03-15T22:27:07.523 INFO:tasks.rados.rados.0.smithi076.stdout:1921: finishing write tid 1 to smithi07694878-21 2021-03-15T22:27:07.524 INFO:tasks.rados.rados.0.smithi076.stdout:1921: finishing write tid 2 to smithi07694878-21 2021-03-15T22:27:07.524 INFO:tasks.rados.rados.0.smithi076.stdout:1921: finishing write tid 3 to smithi07694878-21 2021-03-15T22:27:07.524 INFO:tasks.rados.rados.0.smithi076.stdout:1921: finishing write tid 4 to smithi07694878-21 2021-03-15T22:27:07.525 INFO:tasks.rados.rados.0.smithi076.stdout:1921: finishing write tid 5 to smithi07694878-21 2021-03-15T22:27:07.525 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 1373 (ObjNum 675 snap 214 seq_num 675) dirty exists 2021-03-15T22:27:07.525 INFO:tasks.rados.rados.0.smithi076.stdout:1921: left oid 21 (ObjNum 675 snap 214 seq_num 675) 2021-03-15T22:27:07.526 INFO:tasks.rados.rados.0.smithi076.stdout:1922: expect (ObjNum 509 snap 152 seq_num 509) 2021-03-15T22:27:07.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:07 smithi125 conmon[56739]: cluster 2021-03-15T22:27:06.216703+0000 mon.a (mon. 2021-03-15T22:27:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:07 smithi125 conmon[56739]: 0) 1435 : cluster [DBG] osdmap e511: 8 total, 7 up, 7 in 2021-03-15T22:27:07.569 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:07.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:07.593 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:07.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:07.663 INFO:tasks.rados.rados.0.smithi076.stdout:1924: seq_num 676 ranges {561532=707388,2058298=782238} 2021-03-15T22:27:07.670 INFO:tasks.rados.rados.0.smithi076.stdout:1924: writing smithi07694878-45 from 561532 to 1268920 tid 1 2021-03-15T22:27:07.679 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:07.680 INFO:tasks.rados.rados.0.smithi076.stdout:1924: writing smithi07694878-45 from 2058298 to 2840536 tid 2 2021-03-15T22:27:07.680 INFO:tasks.rados.rados.0.smithi076.stdout:1924: finishing write tid 1 to smithi07694878-45 2021-03-15T22:27:07.681 INFO:tasks.rados.rados.0.smithi076.stdout:1918: done (5 left) 2021-03-15T22:27:07.681 INFO:tasks.rados.rados.0.smithi076.stdout:1919: done (4 left) 2021-03-15T22:27:07.681 INFO:tasks.rados.rados.0.smithi076.stdout:1920: done (3 left) 2021-03-15T22:27:07.681 INFO:tasks.rados.rados.0.smithi076.stdout:1921: done (2 left) 2021-03-15T22:27:07.682 INFO:tasks.rados.rados.0.smithi076.stdout:1922: done (1 left) 2021-03-15T22:27:07.682 INFO:tasks.rados.rados.0.smithi076.stdout:1925: read oid 25 snap -1 2021-03-15T22:27:07.682 INFO:tasks.rados.rados.0.smithi076.stdout:1925: expect (ObjNum 646 snap 201 seq_num 646) 2021-03-15T22:27:07.682 INFO:tasks.rados.rados.0.smithi076.stdout:1926: read oid 13 snap -1 2021-03-15T22:27:07.682 INFO:tasks.rados.rados.0.smithi076.stdout:1926: expect (ObjNum 634 snap 200 seq_num 634) 2021-03-15T22:27:07.683 INFO:tasks.rados.rados.0.smithi076.stdout:1927: read oid 7 snap 160 2021-03-15T22:27:07.683 INFO:tasks.rados.rados.0.smithi076.stdout:1927: expect (ObjNum 526 snap 159 seq_num 526) 2021-03-15T22:27:07.683 INFO:tasks.rados.rados.0.smithi076.stdout:1928: delete oid 29 current snap is 214 2021-03-15T22:27:07.684 INFO:tasks.rados.rados.0.smithi076.stdout:1928: done (4 left) 2021-03-15T22:27:07.684 INFO:tasks.rados.rados.0.smithi076.stdout:1929: rollback oid 12 current snap is 214 2021-03-15T22:27:07.684 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 12 to 196 2021-03-15T22:27:07.685 INFO:tasks.rados.rados.0.smithi076.stdout:1930: copy_from oid 6 from oid 33 current snap is 214 2021-03-15T22:27:07.685 INFO:tasks.rados.rados.0.smithi076.stdout:1931: snap_create 2021-03-15T22:27:07.688 INFO:tasks.rados.rados.0.smithi076.stdout:1926: expect (ObjNum 634 snap 200 seq_num 634) 2021-03-15T22:27:07.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:07.715 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:07.718 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:07.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:07.719 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:07.820 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:07.843 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:07.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:07.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:07.867 INFO:tasks.rados.rados.0.smithi076.stdout:1929: finishing rollback tid 0 to smithi07694878-12 2021-03-15T22:27:07.868 INFO:tasks.rados.rados.0.smithi076.stdout:1929: finishing rollback tid 1 to smithi07694878-12 2021-03-15T22:27:07.868 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 1270 (ObjNum 265088320 snap 21896 seq_num 0) dirty dne 2021-03-15T22:27:07.868 INFO:tasks.rados.rados.0.smithi076.stdout:1925: expect (ObjNum 646 snap 201 seq_num 646) 2021-03-15T22:27:07.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:07.963 INFO:tasks.rados.rados.0.smithi076.stdout:1927: expect (ObjNum 526 snap 159 seq_num 526) 2021-03-15T22:27:07.969 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:07.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:07.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:08.072 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:08.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:08.095 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:08.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:08.114 INFO:tasks.rados.rados.0.smithi076.stdout:1924: finishing write tid 2 to smithi07694878-45 2021-03-15T22:27:08.114 INFO:tasks.rados.rados.0.smithi076.stdout:1924: finishing write tid 3 to smithi07694878-45 2021-03-15T22:27:08.114 INFO:tasks.rados.rados.0.smithi076.stdout:1924: finishing write tid 4 to smithi07694878-45 2021-03-15T22:27:08.115 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 1204 (ObjNum 676 snap 214 seq_num 676) dirty exists 2021-03-15T22:27:08.115 INFO:tasks.rados.rados.0.smithi076.stdout:1924: left oid 45 (ObjNum 676 snap 214 seq_num 676) 2021-03-15T22:27:08.115 INFO:tasks.rados.rados.0.smithi076.stdout:1930: finishing copy_from to smithi07694878-6 2021-03-15T22:27:08.115 INFO:tasks.rados.rados.0.smithi076.stdout:1930: got expected ENOENT (src dne) 2021-03-15T22:27:08.116 INFO:tasks.rados.rados.0.smithi076.stdout:1930: finishing copy_from racing read to smithi07694878-6 2021-03-15T22:27:08.196 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:08.221 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:08.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:08.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:08.231 INFO:tasks.rados.rados.0.smithi076.stdout:1924: done (6 left) 2021-03-15T22:27:08.231 INFO:tasks.rados.rados.0.smithi076.stdout:1925: done (5 left) 2021-03-15T22:27:08.231 INFO:tasks.rados.rados.0.smithi076.stdout:1926: done (4 left) 2021-03-15T22:27:08.232 INFO:tasks.rados.rados.0.smithi076.stdout:1927: done (3 left) 2021-03-15T22:27:08.232 INFO:tasks.rados.rados.0.smithi076.stdout:1929: done (2 left) 2021-03-15T22:27:08.232 INFO:tasks.rados.rados.0.smithi076.stdout:1930: done (1 left) 2021-03-15T22:27:08.233 INFO:tasks.rados.rados.0.smithi076.stdout:1931: done (0 left) 2021-03-15T22:27:08.233 INFO:tasks.rados.rados.0.smithi076.stdout:1932: snap_create 2021-03-15T22:27:08.234 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:27:08.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:08.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:08.357 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:08.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:08.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:08.461 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:08 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:27:08.461 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:08 smithi076 conmon[56514]: 22:27:07.079544+0000 mgr.y (mgr.14144) 697 : cluster [DBG] pgmap v894: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 476 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 1.9 MiB/s wr, 18 op/s; 201/1614 objects degraded (12.454%) 2021-03-15T22:27:08.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:08 smithi076 conmon[56514]: cluster 2021-03-15T22:27:07.226056+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e512: 8 total, 7 up, 7 in 2021-03-15T22:27:08.462 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:08 smithi076 conmon[56514]: audit 2021-03-15T22:27:07.804883+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? 172.21.15.76:0/3246315765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:27:08.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:08 smithi076 conmon[62156]: cluster 2021-03-15T22:27:07. 2021-03-15T22:27:08.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:08 smithi076 conmon[62156]: 079544+0000 mgr.y (mgr.14144) 697 : cluster [DBG] pgmap v894: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 476 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 1.9 MiB/s wr, 18 op/s; 201/1614 objects degraded (12.454%) 2021-03-15T22:27:08.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:08 smithi076 conmon[62156]: cluster 2021-03-15T22:27:07.226056+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e512: 8 total, 7 up, 7 in 2021-03-15T22:27:08.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:08 smithi076 conmon[62156]: audit 2021-03-15T22:27:07.804883+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? 172.21.15.76:0/3246315765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:27:08.480 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:08.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:08.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:08 smithi125 conmon[56739]: cluster 2021-03-15T22:27:07.079544+0000 mgr.y (mgr.14144) 697 : cluster [DBG] pgmap v894: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 476 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 1.9 MiB/s wr, 18 op/s; 201/1614 objects degraded (12.454%) 2021-03-15T22:27:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:08 smithi125 conmon[56739]: cluster 2021-03-15T22:27:07.226056+0000 mon.a (mon.0) 1436 : cluster [DBG] osdmap e512: 8 total, 7 up, 7 in 2021-03-15T22:27:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:08 smithi125 conmon[56739]: audit 2021-03-15T22:27:07.804883+0000 mon.a (mon.0) 1437 : audit [INF] from='client.? 172.21.15.76:0/3246315765' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:27:08.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:08.606 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:08.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:08.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:08.709 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:08.732 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:08.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:08.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:08.836 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:08.860 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:08.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:08.861 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:08.963 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:08.987 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:08.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:09.089 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:09.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:09.112 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:09.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:09.214 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:09.238 INFO:tasks.rados.rados.0.smithi076.stdout:1932: done (0 left) 2021-03-15T22:27:09.238 INFO:tasks.rados.rados.0.smithi076.stdout:1933: delete oid 42 current snap is 216 2021-03-15T22:27:09.241 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:09.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:09.243 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:09.245 INFO:tasks.rados.rados.0.smithi076.stdout:1933: done (0 left) 2021-03-15T22:27:09.246 INFO:tasks.rados.rados.0.smithi076.stdout:1934: write (excl) oid 46 current snap is 216 2021-03-15T22:27:09.246 INFO:tasks.rados.rados.0.smithi076.stdout:1934: seq_num 679 ranges {611953=649488,1950097=529565,3246565=165388} 2021-03-15T22:27:09.247 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:27:09.247 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:27:09.252 INFO:tasks.rados.rados.0.smithi076.stdout:1934: writing smithi07694878-46 from 611953 to 1261441 tid 1 2021-03-15T22:27:09.259 INFO:tasks.rados.rados.0.smithi076.stdout:1934: writing smithi07694878-46 from 1950097 to 2479662 tid 2 2021-03-15T22:27:09.266 INFO:tasks.rados.rados.0.smithi076.stdout:1934: writing smithi07694878-46 from 3246565 to 3411953 tid 3 2021-03-15T22:27:09.266 INFO:tasks.rados.rados.0.smithi076.stdout:1935: read oid 21 snap 214 2021-03-15T22:27:09.266 INFO:tasks.rados.rados.0.smithi076.stdout:1935: expect (ObjNum 675 snap 214 seq_num 675) 2021-03-15T22:27:09.266 INFO:tasks.rados.rados.0.smithi076.stdout:1936: rollback oid 30 current snap is 216 2021-03-15T22:27:09.267 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 30 to 205 2021-03-15T22:27:09.267 INFO:tasks.rados.rados.0.smithi076.stdout:1934: finishing write tid 1 to smithi07694878-46 2021-03-15T22:27:09.267 INFO:tasks.rados.rados.0.smithi076.stdout:1934: finishing write tid 2 to smithi07694878-46 2021-03-15T22:27:09.267 INFO:tasks.rados.rados.0.smithi076.stdout:1937: read oid 15 snap -1 2021-03-15T22:27:09.268 INFO:tasks.rados.rados.0.smithi076.stdout:1937: expect (ObjNum 649 snap 201 seq_num 649) 2021-03-15T22:27:09.268 INFO:tasks.rados.rados.0.smithi076.stdout:1938: snap_create 2021-03-15T22:27:09.272 INFO:tasks.rados.rados.0.smithi076.stdout:1934: finishing write tid 3 to smithi07694878-46 2021-03-15T22:27:09.272 INFO:tasks.rados.rados.0.smithi076.stdout:1934: finishing write tid 4 to smithi07694878-46 2021-03-15T22:27:09.272 INFO:tasks.rados.rados.0.smithi076.stdout:1934: finishing write tid 5 to smithi07694878-46 2021-03-15T22:27:09.273 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 839 (ObjNum 679 snap 216 seq_num 679) dirty exists 2021-03-15T22:27:09.273 INFO:tasks.rados.rados.0.smithi076.stdout:1934: left oid 46 (ObjNum 679 snap 216 seq_num 679) 2021-03-15T22:27:09.275 INFO:tasks.rados.rados.0.smithi076.stdout:1937: expect (ObjNum 649 snap 201 seq_num 649) 2021-03-15T22:27:09.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:09 smithi125 conmon[56739]: cluster 2021-03-15T22:27:08.221159+0000 mon.a (mon.0) 1438 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:09.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:09 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:27:09.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:09 smithi125 conmon[56739]: :27:08.228485+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? 172.21.15.76:0/3246315765' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:27:09.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:09 smithi125 conmon[56739]: cluster 2021-03-15T22:27:08.228543+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e513: 8 total, 7 up, 7 in 2021-03-15T22:27:09.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:09 smithi125 conmon[56739]: cluster 2021-03-15T22:27:08.372866+0000 osd.1 (osd.1) 98 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:27:09.289 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:09 smithi125 conmon[56739]: cluster 2021-03-15T22:27:08.415624+0000 osd.7 (osd.7) 100 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:27:09.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:09.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:09.367 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:09.367 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:09.441 INFO:tasks.rados.rados.0.smithi076.stdout:1936: finishing rollback tid 1 to smithi07694878-30 2021-03-15T22:27:09.441 INFO:tasks.rados.rados.0.smithi076.stdout:1936: finishing rollback tid 2 to smithi07694878-30 2021-03-15T22:27:09.441 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 1272 (ObjNum 647 snap 201 seq_num 647) dirty exists 2021-03-15T22:27:09.441 INFO:tasks.rados.rados.0.smithi076.stdout:1935: expect (ObjNum 675 snap 214 seq_num 675) 2021-03-15T22:27:09.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:09.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[56514]: cluster 2021-03-15T22:27:08.221159+0000 mon.a (mon.0) 1438 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:09.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[56514]: audit 2021-03-15T22:27:08.228485+0000 mon.a (mon.0) 1439 : audit [INF] from='client.? 172.21.15.76:0/3246315765' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:27:09.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[56514]: cluster 2021-03-15T22:27:08.228543+0000 mon.a (mon.0) 1440 : cluster [DBG] osdmap e513: 8 total, 7 up, 7 in 2021-03-15T22:27:09.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[56514]: cluster 2021-03-15T22:27:08.372866+0000 osd.1 (osd.1) 98 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:27:09.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[56514]: cluster 2021-03-15T22:27:08.415624+0000 osd.7 (osd.7) 100 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:27:09.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: cluster 2021-03-15T22:27:08.221159+0000 mon.a (mon. 2021-03-15T22:27:09.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: 0) 1438 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:09.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:27:09.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: T22:27:08.228485+0000 mon.a (mon.0) 1439 : audit [INF] 2021-03-15T22:27:09.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3246315765' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:27:09.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:27:09.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: :27:08.228543+0000 mon.a (mon. 2021-03-15T22:27:09.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: 0) 1440 : cluster [DBG] osdmap e513: 8 total, 7 up, 7 in 2021-03-15T22:27:09.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: cluster 2021-03-15T22:27:09.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: 2021-03-15T22:27:08. 2021-03-15T22:27:09.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: 372866+0000 osd.1 (osd.1) 98 : cluster 2021-03-15T22:27:09.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: [DBG] 3.6 scrub ok 2021-03-15T22:27:09.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:27:09.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: -03-15T22:27:08.415624+0000 osd.7 (osd.7) 2021-03-15T22:27:09.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:09 smithi076 conmon[62156]: 100 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:27:09.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:09.498 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:09.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:09.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:09.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:09.623 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:09.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:09.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:09.748 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:09.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:09.774 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:27:09.791 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:27:09.808 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:27:09.825 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:27:09.842 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:27:09.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:09.859 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:27:09.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:09.874 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:09.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:09.876 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:27:09.879 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:27:09.879 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:27:09.879 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:27:09.880 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:27:09.881 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:27:09.881 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:27:09.882 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:27:09.882 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:27:09.951 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-15T22:27:09.976 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:10.017 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:10.019 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:10.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:10.140 INFO:tasks.rados.rados.0.smithi076.stdout:1934: done (4 left) 2021-03-15T22:27:10.140 INFO:tasks.rados.rados.0.smithi076.stdout:1935: done (3 left) 2021-03-15T22:27:10.140 INFO:tasks.rados.rados.0.smithi076.stdout:1936: done (2 left) 2021-03-15T22:27:10.140 INFO:tasks.rados.rados.0.smithi076.stdout:1937: done (1 left) 2021-03-15T22:27:10.141 INFO:tasks.rados.rados.0.smithi076.stdout:1938: done (0 left) 2021-03-15T22:27:10.141 INFO:tasks.rados.rados.0.smithi076.stdout:1939: snap_create 2021-03-15T22:27:10.144 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:10.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:10.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:10.237 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:10.247 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:10.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:10.271 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:10.272 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:10.281 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: cluster 2021-03-15T22:27:08.823972+0000 osd.4 (osd. 2021-03-15T22:27:10.281 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: 4) 105 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:27:10.281 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: cluster 2021-03-15T22:27:09.079975+0000 mgr.y (mgr.14144) 698 : cluster [DBG] pgmap v897: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 474 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 201/1617 objects degraded (12.430%) 2021-03-15T22:27:10.281 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: cluster 2021-03-15T22:27:09.233914+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e514: 8 total, 7 up, 7 in 2021-03-15T22:27:10.282 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: cluster 2021-03-15T22:27:09.383400+0000 osd.1 (osd.1) 99 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:27:10.282 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: audit 2021-03-15T22:27:09.755042+0000 mon.a (mon.0) 1442 : audit [DBG] from='client.? 172.21.15.76:0/981047385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:10.282 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: audit 2021-03-15T22:27:09.805566+0000 mgr.y (mgr.14144) 699 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:10.282 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: cluster 2021-03-15T22:27:10.131767+0000 mon.a (mon.0) 1443 : cluster 2021-03-15T22:27:10.283 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: [WRN] Health check update: Degraded data redundancy: 201/1617 objects degraded (12.430%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:10.283 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:27:10.283 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: 03-15T22:27:10.137370+0000 2021-03-15T22:27:10.283 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: mon.a (mon.0) 1444 : cluster [DBG] 2021-03-15T22:27:10.284 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:10 smithi125 conmon[56739]: osdmap e515: 8 total, 7 up, 7 in 2021-03-15T22:27:10.284 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:27:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:27:10.372 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:10.396 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:10.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:10.498 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:10.520 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:10.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:10.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[56514]: cluster 2021-03-15T22:27: 2021-03-15T22:27:10.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[56514]: 08.823972+0000 osd.4 (osd.4) 105 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:27:10.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[56514]: cluster 2021-03-15T22:27:09.079975+0000 mgr.y (mgr.14144) 698 : cluster [DBG] pgmap v897: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 474 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 201/1617 objects degraded (12.430%) 2021-03-15T22:27:10.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[56514]: cluster 2021-03-15T22:27:09.233914+0000 mon.a (mon.0) 1441 : cluster [DBG] osdmap e514: 8 total, 7 up, 7 in 2021-03-15T22:27:10.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[56514]: cluster 2021-03-15T22:27:09.383400+0000 osd.1 (osd.1) 99 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:27:10.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[56514]: audit 2021-03-15T22:27:09.755042+0000 mon.a (mon.0) 1442 : audit [DBG] from='client.? 172.21.15.76:0/981047385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:10.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[56514]: audit 2021-03-15T22:27:09.805566+0000 mgr.y (mgr.14144) 699 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:10.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[56514]: cluster 2021-03-15T22:27:10.131767+0000 mon.a (mon.0) 1443 : cluster [WRN] Health check update: Degraded data redundancy: 201/1617 objects degraded (12.430%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:10.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[56514]: cluster 2021-03-15T22:27:10.137370+0000 mon.a (mon.0) 1444 : cluster [DBG] osdmap e515: 8 total, 7 up, 7 in 2021-03-15T22:27:10.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: cluster 2021-03-15T22:27:08.823972 2021-03-15T22:27:10.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 105 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:27:10.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:27:10.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: T22:27:09.079975+0000 mgr.y (mgr.14144) 698 : cluster [DBG] 2021-03-15T22:27:10.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: pgmap v897: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 474 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 201/1617 objects degraded (12.430%) 2021-03-15T22:27:10.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: cluster 2021-03-15T22:27: 2021-03-15T22:27:10.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: 09.233914+0000 mon.a (mon.0) 1441 2021-03-15T22:27:10.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: : cluster [DBG] osdmap e514: 8 total, 7 up, 7 in 2021-03-15T22:27:10.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: cluster 2021-03-15T22:27:10.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:27:10.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: 27:09.383400+0000 osd.1 (osd.1 2021-03-15T22:27:10.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: ) 99 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:27:10.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: audit 2021 2021-03-15T22:27:10.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: -03-15T22:27:09. 2021-03-15T22:27:10.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: 755042+0000 mon.a (mon.0) 1442 : audit 2021-03-15T22:27:10.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/981047385' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:10.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: audit 2021-03-15T22:27:10.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: 2021-03-15T22:27:09. 2021-03-15T22:27:10.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: 805566+0000 mgr.y (mgr.14144) 699 : audit 2021-03-15T22:27:10.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:10.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:27:10.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: 22:27:10.131767+0000 mon.a (mon. 2021-03-15T22:27:10.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: 0) 1443 : cluster [WRN] Health check update: Degraded data redundancy: 201/1617 objects degraded (12.430%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:10.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:27:10.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: -03-15T22:27:10.137370 2021-03-15T22:27:10.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1444 : cluster 2021-03-15T22:27:10.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:10 smithi076 conmon[62156]: [DBG] osdmap e515: 8 total, 7 up, 7 in 2021-03-15T22:27:10.551 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:27:10.552 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:27:10.623 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:10.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:10.647 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:10.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:10.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:10.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:10.774 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:10.774 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:10.876 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:10.897 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:10.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:10.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:11.000 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:11.022 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:11.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:11.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:11.125 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:11.147 INFO:tasks.rados.rados.0.smithi076.stdout:1939: done (0 left) 2021-03-15T22:27:11.148 INFO:tasks.rados.rados.0.smithi076.stdout:1940: snap_remove snap 205 2021-03-15T22:27:11.149 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:11.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:11.151 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:11.252 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:11.279 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:11.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:11.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:11.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:11.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:11.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:11.497 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:27:11.508 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:11.530 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:11.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:11.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:11.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:11 smithi125 conmon[56739]: cluster 2021-03-15T22:27:09.810509 2021-03-15T22:27:11.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:11 smithi125 conmon[56739]: +0000 osd.4 (osd.4) 106 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:27:11.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:11 smithi125 conmon[56739]: cluster 2021-03-15T22:27:10.399383+0000 osd.1 (osd.1) 100 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:27:11.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:11 smithi125 conmon[56739]: cluster 2021-03-15T22:27:11.145176+0000 mon.a (mon.0) 1445 : cluster [DBG] osdmap e516: 8 total, 7 up, 7 in 2021-03-15T22:27:11.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:11.655 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:11.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:11.657 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:11.758 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:11.761 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:27:11.761 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:27:11.761 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:27:11.762 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:27:11.762 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.762 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1589 7.0 7:00c03349:::smithi07694878-40:head [write 766452~565707,stat] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:27:11.762 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.044128+0000", 2021-03-15T22:27:11.762 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 310.71285368500003, 2021-03-15T22:27:11.763 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.002067477, 2021-03-15T22:27:11.763 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.763 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:11.763 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:11.764 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:11.764 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:11.764 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1589 2021-03-15T22:27:11.764 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.764 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.765 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.765 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.765 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044128+0000", 2021-03-15T22:27:11.765 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.765 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.766 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.766 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.766 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044128+0000", 2021-03-15T22:27:11.766 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00075323800000000004 2021-03-15T22:27:11.767 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.767 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.767 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.767 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044882+0000", 2021-03-15T22:27:11.767 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3730000000000001e-06 2021-03-15T22:27:11.768 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.768 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.768 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.768 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044885+0000", 2021-03-15T22:27:11.768 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3543999999999999e-05 2021-03-15T22:27:11.769 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.769 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.769 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.769 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044899+0000", 2021-03-15T22:27:11.770 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2700000000000001e-06 2021-03-15T22:27:11.770 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.770 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.770 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.770 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044905+0000", 2021-03-15T22:27:11.771 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1384000000000001e-05 2021-03-15T22:27:11.771 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.771 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.771 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.771 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044956+0000", 2021-03-15T22:27:11.772 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1709999999999996e-06 2021-03-15T22:27:11.772 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.772 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.772 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:11.772 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.044963+0000", 2021-03-15T22:27:11.773 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99623219900000004 2021-03-15T22:27:11.773 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.773 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.773 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.774 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.041196+0000", 2021-03-15T22:27:11.774 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000118097 2021-03-15T22:27:11.774 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.774 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.774 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.775 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.041314+0000", 2021-03-15T22:27:11.775 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018693400000000001 2021-03-15T22:27:11.775 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.775 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.775 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:27:11.776 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.041501+0000", 2021-03-15T22:27:11.776 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0029280589999999998 2021-03-15T22:27:11.776 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.776 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.776 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.777 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.044429+0000", 2021-03-15T22:27:11.777 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000135282 2021-03-15T22:27:11.777 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.777 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.778 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.778 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.044564+0000", 2021-03-15T22:27:11.778 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001471113 2021-03-15T22:27:11.778 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.778 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.779 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:11.779 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046035+0000", 2021-03-15T22:27:11.779 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9391999999999997e-05 2021-03-15T22:27:11.779 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.779 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.780 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:11.780 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046085+0000", 2021-03-15T22:27:11.780 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111421 2021-03-15T22:27:11.780 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.780 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.781 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.781 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046196+0000", 2021-03-15T22:27:11.781 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.002067477 2021-03-15T22:27:11.782 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.782 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.782 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.783 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1590 7.0 7:00c03349:::smithi07694878-40:head [write 1860221~469412] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:27:11.783 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.049091+0000", 2021-03-15T22:27:11.783 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 310.70789120900002, 2021-03-15T22:27:11.783 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0004167690000001, 2021-03-15T22:27:11.784 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.784 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:11.784 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:11.784 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:11.784 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:11.785 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1590 2021-03-15T22:27:11.785 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.785 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.785 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.785 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.786 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049091+0000", 2021-03-15T22:27:11.786 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.786 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.786 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.786 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.787 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049091+0000", 2021-03-15T22:27:11.787 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036754099999999998 2021-03-15T22:27:11.787 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.787 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.788 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.788 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049458+0000", 2021-03-15T22:27:11.788 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2369999999999999e-06 2021-03-15T22:27:11.788 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.788 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.789 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.789 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049461+0000", 2021-03-15T22:27:11.789 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3468e-05 2021-03-15T22:27:11.789 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.789 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.790 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.790 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049474+0000", 2021-03-15T22:27:11.790 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7660000000000001e-06 2021-03-15T22:27:11.790 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.790 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.791 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.791 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049481+0000", 2021-03-15T22:27:11.791 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0846000000000006e-05 2021-03-15T22:27:11.791 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.792 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.792 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.792 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049552+0000", 2021-03-15T22:27:11.792 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4910000000000005e-06 2021-03-15T22:27:11.792 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.793 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.793 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:11.793 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.049560+0000", 2021-03-15T22:27:11.793 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99323641500000004 2021-03-15T22:27:11.793 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.794 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.794 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.794 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.042797+0000", 2021-03-15T22:27:11.794 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4145000000000002e-05 2021-03-15T22:27:11.794 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.795 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.795 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:11.795 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.042881+0000", 2021-03-15T22:27:11.795 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033443700000000002 2021-03-15T22:27:11.796 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.796 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.796 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.796 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046225+0000", 2021-03-15T22:27:11.796 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1945000000000004e-05 2021-03-15T22:27:11.797 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.797 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.797 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.797 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046287+0000", 2021-03-15T22:27:11.797 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013465800000000001 2021-03-15T22:27:11.798 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.798 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.798 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:27:11.798 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.046422+0000", 2021-03-15T22:27:11.798 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0025218760000000002 2021-03-15T22:27:11.799 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.799 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.799 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:11.799 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048944+0000", 2021-03-15T22:27:11.800 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00036462099999999998 2021-03-15T22:27:11.800 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.800 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.800 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.800 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049308+0000", 2021-03-15T22:27:11.801 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013490400000000001 2021-03-15T22:27:11.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.801 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.801 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049443+0000", 2021-03-15T22:27:11.802 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4076999999999999e-05 2021-03-15T22:27:11.802 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.802 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.802 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:11.803 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049477+0000", 2021-03-15T22:27:11.803 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0409e-05 2021-03-15T22:27:11.803 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.803 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.803 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.804 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049508+0000", 2021-03-15T22:27:11.804 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0004167690000001 2021-03-15T22:27:11.804 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.804 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.804 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.805 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.805 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.805 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1591 7.0 7:00c03349:::smithi07694878-40:head [write 2981238~594195] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:27:11.805 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.056067+0000", 2021-03-15T22:27:11.805 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 310.700915501, 2021-03-15T22:27:11.806 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99404281400000005, 2021-03-15T22:27:11.806 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.806 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:11.806 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:11.807 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:11.807 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:11.807 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1591 2021-03-15T22:27:11.807 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.807 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.808 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.808 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.808 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056067+0000", 2021-03-15T22:27:11.808 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.808 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.809 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.809 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.809 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056067+0000", 2021-03-15T22:27:11.809 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00055381400000000002 2021-03-15T22:27:11.809 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.810 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.810 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056620+0000", 2021-03-15T22:27:11.810 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5569999999999998e-06 2021-03-15T22:27:11.811 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.811 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.811 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.811 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056623+0000", 2021-03-15T22:27:11.811 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3344e-05 2021-03-15T22:27:11.812 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.812 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.812 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056636+0000", 2021-03-15T22:27:11.812 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2089999999999996e-06 2021-03-15T22:27:11.813 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.813 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.813 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.813 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056642+0000", 2021-03-15T22:27:11.813 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4657000000000003e-05 2021-03-15T22:27:11.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.814 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.814 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056686+0000", 2021-03-15T22:27:11.815 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0719999999999996e-06 2021-03-15T22:27:11.815 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.815 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:11.815 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056692+0000", 2021-03-15T22:27:11.816 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98623406700000005 2021-03-15T22:27:11.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.816 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.816 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.042926+0000", 2021-03-15T22:27:11.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101845 2021-03-15T22:27:11.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.817 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.817 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:11.817 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043028+0000", 2021-03-15T22:27:11.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042733099999999998 2021-03-15T22:27:11.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.818 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.047302+0000", 2021-03-15T22:27:11.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8188000000000004e-05 2021-03-15T22:27:11.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.047380+0000", 2021-03-15T22:27:11.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013972 2021-03-15T22:27:11.820 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.821 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:27:11.821 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.047519+0000", 2021-03-15T22:27:11.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020016600000000002 2021-03-15T22:27:11.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.822 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.822 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:11.822 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049521+0000", 2021-03-15T22:27:11.822 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034165299999999998 2021-03-15T22:27:11.822 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.823 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.823 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049863+0000", 2021-03-15T22:27:11.823 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015579400000000001 2021-03-15T22:27:11.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.824 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.824 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.824 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050019+0000", 2021-03-15T22:27:11.824 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1232e-05 2021-03-15T22:27:11.824 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.825 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:11.825 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050070+0000", 2021-03-15T22:27:11.825 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9691999999999999e-05 2021-03-15T22:27:11.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.826 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.826 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.826 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050109+0000", 2021-03-15T22:27:11.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99404281400000005 2021-03-15T22:27:11.827 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.827 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.827 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.828 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.828 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.828 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1592 7.0 7:00c03349:::smithi07694878-40:head [setxattr _header (57) in=7b,truncate 3966452] snapc 6c=[6c,6a,69] ondisk+write+known_if_redirected e206)", 2021-03-15T22:27:11.828 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.056653+0000", 2021-03-15T22:27:11.828 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 310.70032881700001, 2021-03-15T22:27:11.829 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99360381200000003, 2021-03-15T22:27:11.829 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.829 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:11.829 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:11.829 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:11.830 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:11.830 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1592 2021-03-15T22:27:11.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.830 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.830 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.831 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.831 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056653+0000", 2021-03-15T22:27:11.831 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.831 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.832 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056653+0000", 2021-03-15T22:27:11.832 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1789999999999998e-06 2021-03-15T22:27:11.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.833 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056655+0000", 2021-03-15T22:27:11.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.689e-06 2021-03-15T22:27:11.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.834 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.834 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.834 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056657+0000", 2021-03-15T22:27:11.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5310000000000001e-06 2021-03-15T22:27:11.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.835 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.835 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.835 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056663+0000", 2021-03-15T22:27:11.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3579999999999996e-06 2021-03-15T22:27:11.835 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.836 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.836 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056667+0000", 2021-03-15T22:27:11.836 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8161000000000002e-05 2021-03-15T22:27:11.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.837 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056715+0000", 2021-03-15T22:27:11.837 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2899999999999999e-06 2021-03-15T22:27:11.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.838 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:11.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056722+0000", 2021-03-15T22:27:11.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98635435299999996 2021-03-15T22:27:11.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043076+0000", 2021-03-15T22:27:11.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100571 2021-03-15T22:27:11.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.840 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.840 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:11.841 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043176+0000", 2021-03-15T22:27:11.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0051726330000000003 2021-03-15T22:27:11.841 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.842 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.842 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048349+0000", 2021-03-15T22:27:11.842 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010839 2021-03-15T22:27:11.842 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.843 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.843 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048457+0000", 2021-03-15T22:27:11.843 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0001582 2021-03-15T22:27:11.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.844 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.844 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:27:11.844 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048616+0000", 2021-03-15T22:27:11.844 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00090888500000000001 2021-03-15T22:27:11.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.845 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.845 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:11.845 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049525+0000", 2021-03-15T22:27:11.845 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00045732299999999999 2021-03-15T22:27:11.845 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.846 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.846 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.049982+0000", 2021-03-15T22:27:11.846 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016349299999999999 2021-03-15T22:27:11.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050145+0000", 2021-03-15T22:27:11.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9777e-05 2021-03-15T22:27:11.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:11.848 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050175+0000", 2021-03-15T22:27:11.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1978999999999996e-05 2021-03-15T22:27:11.849 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.849 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.849 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050257+0000", 2021-03-15T22:27:11.850 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99360381200000003 2021-03-15T22:27:11.850 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.850 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.850 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.850 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.851 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.851 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1593 7.0 7:00c03349:::smithi07694878-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e206)", 2021-03-15T22:27:11.851 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:22:01.056680+0000", 2021-03-15T22:27:11.851 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 310.700302459, 2021-03-15T22:27:11.851 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99375757600000003, 2021-03-15T22:27:11.852 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.852 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:11.852 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:11.852 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:11.852 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:11.853 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1593 2021-03-15T22:27:11.853 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.853 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.853 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.854 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.854 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056680+0000", 2021-03-15T22:27:11.854 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.854 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.854 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.855 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.855 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056680+0000", 2021-03-15T22:27:11.855 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9000000000000005e-07 2021-03-15T22:27:11.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.855 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.856 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.856 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056681+0000", 2021-03-15T22:27:11.856 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.243e-06 2021-03-15T22:27:11.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.856 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.857 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056682+0000", 2021-03-15T22:27:11.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5610000000000004e-06 2021-03-15T22:27:11.857 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.857 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.858 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.858 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056688+0000", 2021-03-15T22:27:11.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4100000000000001e-06 2021-03-15T22:27:11.858 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.859 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.859 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.859 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056693+0000", 2021-03-15T22:27:11.859 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.562e-05 2021-03-15T22:27:11.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.860 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.860 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.860 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056738+0000", 2021-03-15T22:27:11.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6179999999999999e-06 2021-03-15T22:27:11.860 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.861 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:11.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:01.056742+0000", 2021-03-15T22:27:11.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98647171499999997 2021-03-15T22:27:11.861 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.862 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043214+0000", 2021-03-15T22:27:11.862 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9062999999999998e-05 2021-03-15T22:27:11.862 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.863 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.863 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:11.863 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.043303+0000", 2021-03-15T22:27:11.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055719380000000002 2021-03-15T22:27:11.864 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.864 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.864 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.864 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048875+0000", 2021-03-15T22:27:11.864 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6421000000000001e-05 2021-03-15T22:27:11.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.865 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.865 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:11.865 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.048931+0000", 2021-03-15T22:27:11.865 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0013764700000000001 2021-03-15T22:27:11.866 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.866 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.866 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050308+0000", 2021-03-15T22:27:11.866 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6387000000000001e-05 2021-03-15T22:27:11.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.867 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.867 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.867 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050364+0000", 2021-03-15T22:27:11.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3139999999999994e-05 2021-03-15T22:27:11.868 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.868 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.868 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.868 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:22:02.050437+0000", 2021-03-15T22:27:11.869 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99375757600000003 2021-03-15T22:27:11.869 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.869 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.869 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.870 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.870 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(osd.0.8:42 7.0 7:0732c451:::smithi07694878-32:head [assert-version v640,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e330)", 2021-03-15T22:27:11.870 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:05.087621+0000", 2021-03-15T22:27:11.870 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 186.66936126499999, 2021-03-15T22:27:11.870 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.10411197899999999, 2021-03-15T22:27:11.871 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.871 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:11.871 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.871 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.872 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087621+0000", 2021-03-15T22:27:11.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.872 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.872 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.873 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087621+0000", 2021-03-15T22:27:11.873 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9579999999999999e-06 2021-03-15T22:27:11.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.874 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.874 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087624+0000", 2021-03-15T22:27:11.874 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0649999999999999e-06 2021-03-15T22:27:11.874 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.874 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.875 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.875 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087627+0000", 2021-03-15T22:27:11.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.509e-05 2021-03-15T22:27:11.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.875 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.876 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.876 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087642+0000", 2021-03-15T22:27:11.876 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0669999999999992e-06 2021-03-15T22:27:11.876 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.876 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.877 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.877 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087650+0000", 2021-03-15T22:27:11.877 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4026000000000001e-05 2021-03-15T22:27:11.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.878 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.878 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087704+0000", 2021-03-15T22:27:11.878 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6101999999999999e-05 2021-03-15T22:27:11.878 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.879 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.879 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:11.879 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.087720+0000", 2021-03-15T22:27:11.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.103629809 2021-03-15T22:27:11.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.880 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.880 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.191350+0000", 2021-03-15T22:27:11.880 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014534500000000001 2021-03-15T22:27:11.880 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.881 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.191495+0000", 2021-03-15T22:27:11.881 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000237517 2021-03-15T22:27:11.881 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.882 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.191733+0000", 2021-03-15T22:27:11.882 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.10411197899999999 2021-03-15T22:27:11.883 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.883 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.883 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.883 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.884 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3658 7.0 7:0152dc37:::smithi07694878-28:head [copy-from ver 695] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:27:11.884 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.117923+0000", 2021-03-15T22:27:11.884 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 131.639058769, 2021-03-15T22:27:11.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022161012000000001, 2021-03-15T22:27:11.884 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.885 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:11.885 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:11.885 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:11.885 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:11.885 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3658 2021-03-15T22:27:11.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.886 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.886 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.886 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.887 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117923+0000", 2021-03-15T22:27:11.887 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.887 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.887 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.888 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117923+0000", 2021-03-15T22:27:11.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3010000000000001e-06 2021-03-15T22:27:11.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.888 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.889 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117926+0000", 2021-03-15T22:27:11.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1540000000000001e-06 2021-03-15T22:27:11.889 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.889 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.889 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.890 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117927+0000", 2021-03-15T22:27:11.890 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7890000000000002e-06 2021-03-15T22:27:11.890 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.890 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.890 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.891 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117932+0000", 2021-03-15T22:27:11.891 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.16e-06 2021-03-15T22:27:11.891 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.891 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.892 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.892 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117936+0000", 2021-03-15T22:27:11.892 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0016787620000000001 2021-03-15T22:27:11.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.893 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.893 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.119615+0000", 2021-03-15T22:27:11.893 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3735e-05 2021-03-15T22:27:11.893 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.893 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.894 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.894 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.119668+0000", 2021-03-15T22:27:11.894 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0030274709999999999 2021-03-15T22:27:11.894 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.895 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.895 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,2", 2021-03-15T22:27:11.895 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.122696+0000", 2021-03-15T22:27:11.895 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0065775290000000004 2021-03-15T22:27:11.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:11.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.129273+0000", 2021-03-15T22:27:11.896 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010432643 2021-03-15T22:27:11.896 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.897 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.897 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.139706+0000", 2021-03-15T22:27:11.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027323400000000002 2021-03-15T22:27:11.897 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.898 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.898 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.898 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.139979+0000", 2021-03-15T22:27:11.898 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0559000000000002e-05 2021-03-15T22:27:11.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.899 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.899 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:11.899 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.140030+0000", 2021-03-15T22:27:11.899 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4675000000000001e-05 2021-03-15T22:27:11.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.900 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.900 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.900 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.140084+0000", 2021-03-15T22:27:11.900 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022161012000000001 2021-03-15T22:27:11.901 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.901 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.901 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.901 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.901 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.902 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3659 7.0 7:0152dc37:::smithi07694878-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e385)", 2021-03-15T22:27:11.902 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.117940+0000", 2021-03-15T22:27:11.902 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 131.63904214499999, 2021-03-15T22:27:11.902 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022319004999999999, 2021-03-15T22:27:11.902 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.903 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:11.903 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:11.903 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:11.903 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:11.904 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3659 2021-03-15T22:27:11.904 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.904 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.904 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.904 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.905 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117940+0000", 2021-03-15T22:27:11.905 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.905 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.905 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.905 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.906 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117940+0000", 2021-03-15T22:27:11.906 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1790000000000001e-06 2021-03-15T22:27:11.906 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.906 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.906 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.907 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117941+0000", 2021-03-15T22:27:11.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9600000000000008e-07 2021-03-15T22:27:11.907 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.907 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.907 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.908 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117942+0000", 2021-03-15T22:27:11.908 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4900000000000002e-06 2021-03-15T22:27:11.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.908 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117947+0000", 2021-03-15T22:27:11.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3050000000000001e-06 2021-03-15T22:27:11.909 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.910 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.910 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.117950+0000", 2021-03-15T22:27:11.910 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0018296300000000001 2021-03-15T22:27:11.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.910 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.911 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.911 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.119780+0000", 2021-03-15T22:27:11.911 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9079000000000001e-05 2021-03-15T22:27:11.911 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.911 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.912 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:11.912 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.119809+0000", 2021-03-15T22:27:11.912 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0083358630000000006 2021-03-15T22:27:11.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.912 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.913 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.913 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.128145+0000", 2021-03-15T22:27:11.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121271 2021-03-15T22:27:11.913 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.914 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:11.914 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.128266+0000", 2021-03-15T22:27:11.914 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011857065999999999 2021-03-15T22:27:11.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.915 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.140123+0000", 2021-03-15T22:27:11.915 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9793000000000001e-05 2021-03-15T22:27:11.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.916 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.916 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.140193+0000", 2021-03-15T22:27:11.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6333000000000003e-05 2021-03-15T22:27:11.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.917 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.917 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.140259+0000", 2021-03-15T22:27:11.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022319004999999999 2021-03-15T22:27:11.917 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.918 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.918 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.919 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:3664 7.6 e385/358 7:608078b0:::smithi07694878-12:head v 385'816, mlcod=380'813)", 2021-03-15T22:27:11.919 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.131555+0000", 2021-03-15T22:27:11.919 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 131.625426804, 2021-03-15T22:27:11.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037191206999999997, 2021-03-15T22:27:11.919 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.920 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:11.920 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.920 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.920 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131555+0000", 2021-03-15T22:27:11.921 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.921 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.921 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.921 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.921 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131555+0000", 2021-03-15T22:27:11.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4609999999999998e-06 2021-03-15T22:27:11.922 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.922 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.923 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131558+0000", 2021-03-15T22:27:11.923 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0429999999999999e-06 2021-03-15T22:27:11.923 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.923 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.924 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131561+0000", 2021-03-15T22:27:11.924 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8675999999999999e-05 2021-03-15T22:27:11.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.924 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.925 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131580+0000", 2021-03-15T22:27:11.925 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4439999999999996e-06 2021-03-15T22:27:11.925 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.925 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.925 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.926 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131586+0000", 2021-03-15T22:27:11.926 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0652000000000001e-05 2021-03-15T22:27:11.926 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.926 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.927 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131657+0000", 2021-03-15T22:27:11.927 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0899999999999999e-05 2021-03-15T22:27:11.927 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.928 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131688+0000", 2021-03-15T22:27:11.928 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036987938999999997 2021-03-15T22:27:11.928 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.928 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.929 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:11.929 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.168675+0000", 2021-03-15T22:27:11.929 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1092000000000003e-05 2021-03-15T22:27:11.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.930 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.930 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.168747+0000", 2021-03-15T22:27:11.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037191206999999997 2021-03-15T22:27:11.930 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.930 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.931 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.931 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3853 7.0 7:08414d91:::smithi07694878-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e403)", 2021-03-15T22:27:11.932 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:18.352603+0000", 2021-03-15T22:27:11.932 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 113.404379271, 2021-03-15T22:27:11.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022097461999999998, 2021-03-15T22:27:11.932 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.932 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:11.933 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:11.933 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:11.933 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:11.933 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3853 2021-03-15T22:27:11.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.934 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.934 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.934 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.934 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.352603+0000", 2021-03-15T22:27:11.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.935 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.352603+0000", 2021-03-15T22:27:11.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1710000000000001e-06 2021-03-15T22:27:11.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.936 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.936 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.936 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.352604+0000", 2021-03-15T22:27:11.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5409999999999999e-06 2021-03-15T22:27:11.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.937 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.937 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.937 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.352606+0000", 2021-03-15T22:27:11.938 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4582e-05 2021-03-15T22:27:11.938 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.938 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.938 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.938 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.352620+0000", 2021-03-15T22:27:11.939 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6409999999999999e-06 2021-03-15T22:27:11.939 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.939 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.352626+0000", 2021-03-15T22:27:11.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000268316 2021-03-15T22:27:11.940 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.941 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.941 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.352894+0000", 2021-03-15T22:27:11.941 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7543000000000003e-05 2021-03-15T22:27:11.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.942 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.942 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:11.942 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.352932+0000", 2021-03-15T22:27:11.942 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012328168 2021-03-15T22:27:11.942 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.943 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.943 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.365260+0000", 2021-03-15T22:27:11.943 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011484699999999999 2021-03-15T22:27:11.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.944 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.944 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:11.944 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.365375+0000", 2021-03-15T22:27:11.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0091798299999999999 2021-03-15T22:27:11.944 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.945 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.374555+0000", 2021-03-15T22:27:11.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2432000000000001e-05 2021-03-15T22:27:11.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.946 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.374617+0000", 2021-03-15T22:27:11.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3391000000000002e-05 2021-03-15T22:27:11.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:18.374700+0000", 2021-03-15T22:27:11.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022097461999999998 2021-03-15T22:27:11.948 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.948 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.948 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.948 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.949 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3936 7.0 7:071f7a04:::smithi07694878-37:head [write 608552~478174,stat] snapc 13b=[13b,13a,139,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e418)", 2021-03-15T22:27:11.949 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:33.127436+0000", 2021-03-15T22:27:11.949 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 98.629546297000005, 2021-03-15T22:27:11.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1094841310000001, 2021-03-15T22:27:11.949 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.950 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:11.950 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:11.950 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:11.950 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:11.951 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3936 2021-03-15T22:27:11.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.951 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.951 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.951 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.127436+0000", 2021-03-15T22:27:11.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.952 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.952 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.952 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.953 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.127436+0000", 2021-03-15T22:27:11.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00063044800000000003 2021-03-15T22:27:11.953 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.953 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.954 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.128066+0000", 2021-03-15T22:27:11.954 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.836e-06 2021-03-15T22:27:11.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.955 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.955 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.128069+0000", 2021-03-15T22:27:11.955 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0226e-05 2021-03-15T22:27:11.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.955 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.956 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.956 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.128079+0000", 2021-03-15T22:27:11.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9500000000000007e-06 2021-03-15T22:27:11.956 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.956 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.957 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.957 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.128088+0000", 2021-03-15T22:27:11.957 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005206904 2021-03-15T22:27:11.957 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.957 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.958 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.958 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.133295+0000", 2021-03-15T22:27:11.958 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1031e-05 2021-03-15T22:27:11.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.959 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:11.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.133306+0000", 2021-03-15T22:27:11.959 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1006149810000001 2021-03-15T22:27:11.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.233921+0000", 2021-03-15T22:27:11.960 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015497899999999999 2021-03-15T22:27:11.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.961 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.234076+0000", 2021-03-15T22:27:11.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000180657 2021-03-15T22:27:11.961 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.962 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:27:11.962 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.234257+0000", 2021-03-15T22:27:11.962 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0016471260000000001 2021-03-15T22:27:11.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.963 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.235904+0000", 2021-03-15T22:27:11.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1849999999999998e-06 2021-03-15T22:27:11.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.964 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:11.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.235911+0000", 2021-03-15T22:27:11.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00064259500000000004 2021-03-15T22:27:11.965 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.236554+0000", 2021-03-15T22:27:11.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7145000000000006e-05 2021-03-15T22:27:11.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:11.966 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.236651+0000", 2021-03-15T22:27:11.966 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00026906800000000002 2021-03-15T22:27:11.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.967 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.967 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.967 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.236920+0000", 2021-03-15T22:27:11.968 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1094841310000001 2021-03-15T22:27:11.968 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.968 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.968 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.969 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.969 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3937 7.0 7:071f7a04:::smithi07694878-37:head [write 1848819~707173] snapc 13b=[13b,13a,139,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e418)", 2021-03-15T22:27:11.969 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:33.136354+0000", 2021-03-15T22:27:11.969 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 98.620627870999996, 2021-03-15T22:27:11.969 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1037518989999999, 2021-03-15T22:27:11.970 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.970 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:11.970 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:11.970 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:11.970 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:11.971 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3937 2021-03-15T22:27:11.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.971 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.971 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.972 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.136354+0000", 2021-03-15T22:27:11.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.973 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.136354+0000", 2021-03-15T22:27:11.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00098768599999999999 2021-03-15T22:27:11.973 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.973 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.974 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.974 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137342+0000", 2021-03-15T22:27:11.974 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5430000000000004e-06 2021-03-15T22:27:11.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.975 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.975 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137346+0000", 2021-03-15T22:27:11.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.154e-05 2021-03-15T22:27:11.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.975 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.976 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.976 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137368+0000", 2021-03-15T22:27:11.976 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9119999999999993e-06 2021-03-15T22:27:11.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.977 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.977 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137378+0000", 2021-03-15T22:27:11.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010867 2021-03-15T22:27:11.977 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.978 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.978 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.978 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137487+0000", 2021-03-15T22:27:11.978 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2079000000000001e-05 2021-03-15T22:27:11.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:11.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137499+0000", 2021-03-15T22:27:11.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.097889586 2021-03-15T22:27:11.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.235388+0000", 2021-03-15T22:27:11.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000103433 2021-03-15T22:27:11.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:11.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.235492+0000", 2021-03-15T22:27:11.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014891609999999999 2021-03-15T22:27:11.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:11.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.236981+0000", 2021-03-15T22:27:11.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012385900000000001 2021-03-15T22:27:11.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.983 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:11.983 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.237105+0000", 2021-03-15T22:27:11.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025257799999999999 2021-03-15T22:27:11.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:27:11.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.237357+0000", 2021-03-15T22:27:11.984 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00213904 2021-03-15T22:27:11.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.985 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:11.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.239496+0000", 2021-03-15T22:27:11.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0784000000000002e-05 2021-03-15T22:27:11.986 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.986 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.986 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.986 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.239527+0000", 2021-03-15T22:27:11.986 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00040293099999999998 2021-03-15T22:27:11.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.987 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:11.987 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.239930+0000", 2021-03-15T22:27:11.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2738000000000003e-05 2021-03-15T22:27:11.988 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.988 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.988 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:11.988 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.239973+0000", 2021-03-15T22:27:11.989 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013335899999999999 2021-03-15T22:27:11.989 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.989 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.989 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:11.989 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.240106+0000", 2021-03-15T22:27:11.990 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1037518989999999 2021-03-15T22:27:11.990 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.990 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:11.990 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:11.990 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.991 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3938 7.0 7:071f7a04:::smithi07694878-37:head [setxattr _header (58) in=7b,truncate 3008552] snapc 13b=[13b,13a,139,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e418)", 2021-03-15T22:27:11.991 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:33.137398+0000", 2021-03-15T22:27:11.991 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 98.619584601, 2021-03-15T22:27:11.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1030187840000001, 2021-03-15T22:27:11.992 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:11.992 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:11.992 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:11.992 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:11.993 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:11.993 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3938 2021-03-15T22:27:11.993 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.993 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:11.993 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:11.994 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137398+0000", 2021-03-15T22:27:11.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:11.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.994 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:11.995 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137398+0000", 2021-03-15T22:27:11.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.994e-06 2021-03-15T22:27:11.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:11.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137401+0000", 2021-03-15T22:27:11.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2460000000000002e-06 2021-03-15T22:27:11.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.997 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:11.997 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137403+0000", 2021-03-15T22:27:11.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6240000000000008e-06 2021-03-15T22:27:11.997 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.998 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.998 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:11.998 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137411+0000", 2021-03-15T22:27:11.998 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.429e-06 2021-03-15T22:27:11.998 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:11.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:11.999 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:11.999 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137417+0000", 2021-03-15T22:27:11.999 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0001054 2021-03-15T22:27:12.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.000 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137522+0000", 2021-03-15T22:27:12.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9880000000000001e-06 2021-03-15T22:27:12.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.001 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:12.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137528+0000", 2021-03-15T22:27:12.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.097996132 2021-03-15T22:27:12.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.002 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.002 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.235524+0000", 2021-03-15T22:27:12.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4463999999999997e-05 2021-03-15T22:27:12.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:12.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.235599+0000", 2021-03-15T22:27:12.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002991648 2021-03-15T22:27:12.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.004 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.238590+0000", 2021-03-15T22:27:12.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000112976 2021-03-15T22:27:12.005 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:12.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.238703+0000", 2021-03-15T22:27:12.006 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016261699999999999 2021-03-15T22:27:12.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.006 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:27:12.006 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.238866+0000", 2021-03-15T22:27:12.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00127433 2021-03-15T22:27:12.007 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.007 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.007 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:12.008 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.240140+0000", 2021-03-15T22:27:12.008 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.028e-05 2021-03-15T22:27:12.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.008 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.008 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:12.009 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.240151+0000", 2021-03-15T22:27:12.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012934700000000001 2021-03-15T22:27:12.009 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.009 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.009 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:12.010 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.240280+0000", 2021-03-15T22:27:12.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0354e-05 2021-03-15T22:27:12.010 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.010 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.010 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:12.011 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.240320+0000", 2021-03-15T22:27:12.011 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5954999999999999e-05 2021-03-15T22:27:12.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.011 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.011 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:12.012 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.240416+0000", 2021-03-15T22:27:12.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1030187840000001 2021-03-15T22:27:12.012 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.012 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:12.013 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.013 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.013 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3939 7.0 7:071f7a04:::smithi07694878-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e418)", 2021-03-15T22:27:12.013 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:33.137423+0000", 2021-03-15T22:27:12.014 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 98.619559288999994, 2021-03-15T22:27:12.014 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.103277442, 2021-03-15T22:27:12.014 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:12.014 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:12.014 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:12.015 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:12.015 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:12.015 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3939 2021-03-15T22:27:12.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.015 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:12.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:12.016 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137423+0000", 2021-03-15T22:27:12.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:12.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:12.017 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137423+0000", 2021-03-15T22:27:12.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.192e-06 2021-03-15T22:27:12.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.018 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.018 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:12.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137424+0000", 2021-03-15T22:27:12.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4339999999999999e-06 2021-03-15T22:27:12.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:12.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137425+0000", 2021-03-15T22:27:12.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3740000000000003e-06 2021-03-15T22:27:12.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:12.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137432+0000", 2021-03-15T22:27:12.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.318e-06 2021-03-15T22:27:12.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.021 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:12.021 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137436+0000", 2021-03-15T22:27:12.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111286 2021-03-15T22:27:12.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137547+0000", 2021-03-15T22:27:12.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8420000000000001e-06 2021-03-15T22:27:12.023 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:12.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:33.137553+0000", 2021-03-15T22:27:12.024 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0980723640000001 2021-03-15T22:27:12.024 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.024 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.024 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.235626+0000", 2021-03-15T22:27:12.025 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6082999999999997e-05 2021-03-15T22:27:12.025 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.025 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:12.025 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.235692+0000", 2021-03-15T22:27:12.026 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0036620089999999999 2021-03-15T22:27:12.026 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.026 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.027 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.239354+0000", 2021-03-15T22:27:12.027 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012807899999999999 2021-03-15T22:27:12.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.027 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:12.028 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.239482+0000", 2021-03-15T22:27:12.028 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00096410799999999998 2021-03-15T22:27:12.028 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.028 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.029 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.240446+0000", 2021-03-15T22:27:12.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8826000000000002e-05 2021-03-15T22:27:12.029 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.029 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.029 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:12.030 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.240505+0000", 2021-03-15T22:27:12.030 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000195527 2021-03-15T22:27:12.030 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.030 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.030 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:12.031 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:34.240700+0000", 2021-03-15T22:27:12.031 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.103277442 2021-03-15T22:27:12.031 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.031 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:12.032 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.032 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.032 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:4284 7.b e445/401 7:d4368cae:::smithi07694878-18:head v 445'396, mlcod=445'394)", 2021-03-15T22:27:12.032 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.163535+0000", 2021-03-15T22:27:12.033 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 71.593447566999998, 2021-03-15T22:27:12.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043389199000000003, 2021-03-15T22:27:12.033 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:12.033 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:12.033 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:12.034 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.034 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:12.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163535+0000", 2021-03-15T22:27:12.034 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:12.034 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:12.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163535+0000", 2021-03-15T22:27:12.035 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00045097399999999998 2021-03-15T22:27:12.035 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:12.036 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163986+0000", 2021-03-15T22:27:12.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6230000000000002e-06 2021-03-15T22:27:12.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.037 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:12.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163991+0000", 2021-03-15T22:27:12.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0372000000000001e-05 2021-03-15T22:27:12.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.038 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:12.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164012+0000", 2021-03-15T22:27:12.038 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5627000000000001e-05 2021-03-15T22:27:12.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:12.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164027+0000", 2021-03-15T22:27:12.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9771000000000002e-05 2021-03-15T22:27:12.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.040 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.040 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164077+0000", 2021-03-15T22:27:12.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5298000000000001e-05 2021-03-15T22:27:12.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164102+0000", 2021-03-15T22:27:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042761651999999997 2021-03-15T22:27:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.043 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:12.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.206864+0000", 2021-03-15T22:27:12.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9882e-05 2021-03-15T22:27:12.043 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.043 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.044 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:12.044 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.206924+0000", 2021-03-15T22:27:12.044 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.043389199000000003 2021-03-15T22:27:12.044 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.044 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:12.045 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.045 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:4285 7.b e445/401 7:d4368cae:::smithi07694878-18:head v 445'397, mlcod=445'394)", 2021-03-15T22:27:12.046 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.164690+0000", 2021-03-15T22:27:12.046 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 71.592292178999998, 2021-03-15T22:27:12.046 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.04226456, 2021-03-15T22:27:12.046 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:12.046 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:12.047 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:12.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:12.047 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164690+0000", 2021-03-15T22:27:12.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:12.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:12.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164690+0000", 2021-03-15T22:27:12.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8653000000000002e-05 2021-03-15T22:27:12.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.049 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:12.049 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164719+0000", 2021-03-15T22:27:12.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1849999999999997e-06 2021-03-15T22:27:12.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.050 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:12.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164723+0000", 2021-03-15T22:27:12.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2520000000000001e-05 2021-03-15T22:27:12.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.051 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:12.051 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164745+0000", 2021-03-15T22:27:12.052 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0770000000000002e-06 2021-03-15T22:27:12.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.052 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:12.052 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164752+0000", 2021-03-15T22:27:12.053 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5573e-05 2021-03-15T22:27:12.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.053 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.053 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164788+0000", 2021-03-15T22:27:12.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3039000000000001e-05 2021-03-15T22:27:12.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:12.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164811+0000", 2021-03-15T22:27:12.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042120235999999998 2021-03-15T22:27:12.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.055 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:12.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.206931+0000", 2021-03-15T22:27:12.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3277000000000001e-05 2021-03-15T22:27:12.056 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:12.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.206955+0000", 2021-03-15T22:27:12.057 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.04226456 2021-03-15T22:27:12.057 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.057 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:12.057 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.058 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4843 7.0 7:00c03349:::smithi07694878-40:head [write 793462~638545] snapc 190=[190,18f,18a,185,184,181,180,17f,17a,176,175,16e,168,161,15f,141,131,12f,123,10c,102] ondisk+write+known_if_redirected e506)", 2021-03-15T22:27:12.058 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:27:01.156422+0000", 2021-03-15T22:27:12.059 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 10.60055972, 2021-03-15T22:27:12.059 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023187557000000001, 2021-03-15T22:27:12.059 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:12.059 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:12.059 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:12.060 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:12.060 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:12.060 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4843 2021-03-15T22:27:12.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.060 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:12.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.061 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:12.061 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.156422+0000", 2021-03-15T22:27:12.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:12.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.062 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.062 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:12.062 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.156422+0000", 2021-03-15T22:27:12.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00094844799999999996 2021-03-15T22:27:12.062 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.063 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:12.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.157371+0000", 2021-03-15T22:27:12.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9620000000000002e-06 2021-03-15T22:27:12.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.064 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:12.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.157374+0000", 2021-03-15T22:27:12.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3579e-05 2021-03-15T22:27:12.065 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:12.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.157387+0000", 2021-03-15T22:27:12.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9099999999999999e-06 2021-03-15T22:27:12.066 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:12.066 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.157394+0000", 2021-03-15T22:27:12.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2385000000000001e-05 2021-03-15T22:27:12.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.067 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.067 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.157447+0000", 2021-03-15T22:27:12.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000107272 2021-03-15T22:27:12.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:12.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.157554+0000", 2021-03-15T22:27:12.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024869699999999998 2021-03-15T22:27:12.069 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:27:12.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.157803+0000", 2021-03-15T22:27:12.070 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027078979999999998 2021-03-15T22:27:12.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.070 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:12.070 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.160511+0000", 2021-03-15T22:27:12.071 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003570068 2021-03-15T22:27:12.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.071 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.071 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:12.072 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.164081+0000", 2021-03-15T22:27:12.072 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015398113 2021-03-15T22:27:12.072 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.072 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:12.073 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.179479+0000", 2021-03-15T22:27:12.073 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7973999999999997e-05 2021-03-15T22:27:12.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.073 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.073 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:12.074 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.179547+0000", 2021-03-15T22:27:12.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3250999999999995e-05 2021-03-15T22:27:12.074 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.074 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:12.075 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.179610+0000", 2021-03-15T22:27:12.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023187557000000001 2021-03-15T22:27:12.075 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.075 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:12.076 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.076 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.076 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.076 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4844 7.0 7:00c03349:::smithi07694878-40:head [write 2106184~763893] snapc 190=[190,18f,18a,185,184,181,180,17f,17a,176,175,16e,168,161,15f,141,131,12f,123,10c,102] ondisk+write+known_if_redirected e506)", 2021-03-15T22:27:12.076 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:27:01.165530+0000", 2021-03-15T22:27:12.077 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 10.591452432000001, 2021-03-15T22:27:12.077 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023848316000000001, 2021-03-15T22:27:12.077 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:12.077 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:12.077 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:12.078 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:12.078 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:12.078 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4844 2021-03-15T22:27:12.078 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.078 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:12.079 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.079 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:12.079 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.165530+0000", 2021-03-15T22:27:12.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:12.079 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.080 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.080 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:12.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.165530+0000", 2021-03-15T22:27:12.080 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00065060599999999997 2021-03-15T22:27:12.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:12.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166180+0000", 2021-03-15T22:27:12.081 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1889999999999999e-06 2021-03-15T22:27:12.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:12.082 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166183+0000", 2021-03-15T22:27:12.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6084000000000001e-05 2021-03-15T22:27:12.083 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.083 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:12.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166200+0000", 2021-03-15T22:27:12.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0840000000000003e-06 2021-03-15T22:27:12.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.084 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:12.084 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166207+0000", 2021-03-15T22:27:12.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7478000000000005e-05 2021-03-15T22:27:12.085 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.085 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.085 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166294+0000", 2021-03-15T22:27:12.086 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000118904 2021-03-15T22:27:12.086 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.086 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.086 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:12.086 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166413+0000", 2021-03-15T22:27:12.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019472599999999999 2021-03-15T22:27:12.087 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.087 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.087 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:27:12.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166608+0000", 2021-03-15T22:27:12.088 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002848279 2021-03-15T22:27:12.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.088 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.088 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:12.088 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.169456+0000", 2021-03-15T22:27:12.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00033634300000000002 2021-03-15T22:27:12.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.089 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.089 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:12.090 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.169792+0000", 2021-03-15T22:27:12.090 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019447684 2021-03-15T22:27:12.090 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.090 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.090 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:12.091 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.189240+0000", 2021-03-15T22:27:12.091 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9422e-05 2021-03-15T22:27:12.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.091 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:12.092 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.189309+0000", 2021-03-15T22:27:12.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8516999999999995e-05 2021-03-15T22:27:12.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.092 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:12.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.189378+0000", 2021-03-15T22:27:12.093 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023848316000000001 2021-03-15T22:27:12.093 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.093 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:12.094 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.094 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4845 7.0 7:00c03349:::smithi07694878-40:head [setxattr _header (58) in=7b,truncate 3193462] snapc 190=[190,18f,18a,185,184,181,180,17f,17a,176,175,16e,168,161,15f,141,131,12f,123,10c,102] ondisk+write+known_if_redirected e506)", 2021-03-15T22:27:12.094 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:27:01.166221+0000", 2021-03-15T22:27:12.095 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 10.590761597, 2021-03-15T22:27:12.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023322527999999999, 2021-03-15T22:27:12.095 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:12.095 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:12.095 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:12.096 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:12.096 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:12.096 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4845 2021-03-15T22:27:12.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.096 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:12.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.097 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:12.097 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166221+0000", 2021-03-15T22:27:12.097 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:12.097 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.098 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:12.098 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166221+0000", 2021-03-15T22:27:12.098 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9769999999999999e-06 2021-03-15T22:27:12.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.099 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.099 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:12.099 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166223+0000", 2021-03-15T22:27:12.099 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3540000000000001e-06 2021-03-15T22:27:12.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.100 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:12.100 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166224+0000", 2021-03-15T22:27:12.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2509999999999994e-06 2021-03-15T22:27:12.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.101 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:12.101 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166232+0000", 2021-03-15T22:27:12.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2679999999999996e-06 2021-03-15T22:27:12.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.102 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.102 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:12.102 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166236+0000", 2021-03-15T22:27:12.103 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0018603999999999999 2021-03-15T22:27:12.103 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.168097+0000", 2021-03-15T22:27:12.104 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000113454 2021-03-15T22:27:12.104 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.104 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.104 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:12.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.168210+0000", 2021-03-15T22:27:12.105 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00052352500000000005 2021-03-15T22:27:12.105 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.105 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.105 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:27:12.105 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.168734+0000", 2021-03-15T22:27:12.106 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00073268499999999996 2021-03-15T22:27:12.106 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.106 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.106 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:12.106 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.169466+0000", 2021-03-15T22:27:12.107 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001762252 2021-03-15T22:27:12.107 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.107 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.107 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:12.107 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.171229+0000", 2021-03-15T22:27:12.108 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018193285999999999 2021-03-15T22:27:12.108 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.108 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.108 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:12.109 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.189422+0000", 2021-03-15T22:27:12.109 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5550999999999997e-05 2021-03-15T22:27:12.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.109 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:12.110 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.189478+0000", 2021-03-15T22:27:12.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5524999999999994e-05 2021-03-15T22:27:12.110 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.110 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:12.111 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.189543+0000", 2021-03-15T22:27:12.111 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023322527999999999 2021-03-15T22:27:12.111 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.111 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:12.111 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.112 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4846 7.0 7:00c03349:::smithi07694878-40:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e506)", 2021-03-15T22:27:12.112 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:27:01.166241+0000", 2021-03-15T22:27:12.112 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 10.590741123999999, 2021-03-15T22:27:12.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023532212, 2021-03-15T22:27:12.113 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:12.113 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:12.113 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:12.114 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:12.114 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:12.114 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4846 2021-03-15T22:27:12.114 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.114 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:12.115 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.115 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:12.115 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166241+0000", 2021-03-15T22:27:12.115 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:12.115 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.116 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.116 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:12.116 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166241+0000", 2021-03-15T22:27:12.116 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0400000000000005e-07 2021-03-15T22:27:12.116 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.117 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.117 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:12.117 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166242+0000", 2021-03-15T22:27:12.117 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0610000000000001e-06 2021-03-15T22:27:12.117 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.118 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.118 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:12.118 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166243+0000", 2021-03-15T22:27:12.118 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1079999999999993e-06 2021-03-15T22:27:12.118 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.119 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.119 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:12.119 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166251+0000", 2021-03-15T22:27:12.119 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9899999999999999e-06 2021-03-15T22:27:12.120 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.120 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.120 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:12.120 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.166255+0000", 2021-03-15T22:27:12.120 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0032343089999999999 2021-03-15T22:27:12.121 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.121 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.121 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.121 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.169489+0000", 2021-03-15T22:27:12.121 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5768999999999998e-05 2021-03-15T22:27:12.122 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.122 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.122 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:12.122 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.169575+0000", 2021-03-15T22:27:12.122 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019995878000000002 2021-03-15T22:27:12.123 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.123 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.123 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:12.123 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.189571+0000", 2021-03-15T22:27:12.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4575999999999997e-05 2021-03-15T22:27:12.124 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.124 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:12.124 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.189666+0000", 2021-03-15T22:27:12.125 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000107617 2021-03-15T22:27:12.125 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:12.125 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:12.125 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:12.125 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:01.189773+0000", 2021-03-15T22:27:12.126 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.023532212 2021-03-15T22:27:12.126 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.126 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:12.126 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.126 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:12.127 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:12.127 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:27:12.128 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[62156]: cluster 2021-03-15T22:27:09.810509+0000 osd.4 (osd. 2021-03-15T22:27:12.129 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[62156]: 4) 106 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:27:12.129 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[62156]: cluster 2021-03-15T22:27:10.399383+0000 osd.1 (osd.1 2021-03-15T22:27:12.129 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[62156]: ) 100 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:27:12.129 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[62156]: cluster 2021-03-15T22:27:12.129 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[62156]: 2021-03-15T22:27:11.145176+0000 mon.a (mon. 2021-03-15T22:27:12.130 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[62156]: 0) 1445 : cluster [DBG] osdmap e516: 8 total, 7 up, 7 in 2021-03-15T22:27:12.130 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[56514]: cluster 2021-03-15T22:27:12.130 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[56514]: 2021-03-15T22:27 2021-03-15T22:27:12.131 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[56514]: :09.810509+0000 osd.4 (osd.4) 106 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:27:12.131 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[56514]: cluster 2021-03-15T22:27:12.131 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[56514]: 2021-03-15T22:27:10. 2021-03-15T22:27:12.131 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[56514]: 399383+0000 osd.1 (osd.1) 100 : cluster [DBG] 2021-03-15T22:27:12.131 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[56514]: 5.5 scrub ok 2021-03-15T22:27:12.132 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[56514]: cluster 2021-03-15T22:27 2021-03-15T22:27:12.132 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[56514]: :11.145176+0000 mon.a (mon.0) 1445 : cluster 2021-03-15T22:27:12.132 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:11 smithi076 conmon[56514]: [DBG] osdmap e516: 8 total, 7 up, 7 in 2021-03-15T22:27:12.133 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:12.134 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:12.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:12.153 INFO:tasks.rados.rados.0.smithi076.stdout:1940: done (0 left) 2021-03-15T22:27:12.153 INFO:tasks.rados.rados.0.smithi076.stdout:1941: read oid 19 snap -1 2021-03-15T22:27:12.154 INFO:tasks.rados.rados.0.smithi076.stdout:1941: expect (ObjNum 495 snap 149 seq_num 495) 2021-03-15T22:27:12.154 INFO:tasks.rados.rados.0.smithi076.stdout:1942: copy_from oid 42 from oid 28 current snap is 218 2021-03-15T22:27:12.155 INFO:tasks.rados.rados.0.smithi076.stdout:1943: write oid 30 current snap is 218 2021-03-15T22:27:12.155 INFO:tasks.rados.rados.0.smithi076.stdout:1943: seq_num 680 ranges {628760=724737,1892226=673223,3157635=271125} 2021-03-15T22:27:12.161 INFO:tasks.rados.rados.0.smithi076.stdout:1943: writing smithi07694878-30 from 628760 to 1353497 tid 1 2021-03-15T22:27:12.169 INFO:tasks.rados.rados.0.smithi076.stdout:1943: writing smithi07694878-30 from 1892226 to 2565449 tid 2 2021-03-15T22:27:12.174 INFO:tasks.rados.rados.0.smithi076.stdout:1943: writing smithi07694878-30 from 3157635 to 3428760 tid 3 2021-03-15T22:27:12.175 INFO:tasks.rados.rados.0.smithi076.stdout:1944: write (excl) oid 40 current snap is 218 2021-03-15T22:27:12.175 INFO:tasks.rados.rados.0.smithi076.stdout:1944: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2021-03-15T22:27:12.183 INFO:tasks.rados.rados.0.smithi076.stdout:1944: writing smithi07694878-40 from 645567 to 1445553 tid 1 2021-03-15T22:27:12.190 INFO:tasks.rados.rados.0.smithi076.stdout:1944: writing smithi07694878-40 from 1950708 to 2483942 tid 2 2021-03-15T22:27:12.196 INFO:tasks.rados.rados.0.smithi076.stdout:1944: writing smithi07694878-40 from 3185058 to 3445567 tid 3 2021-03-15T22:27:12.196 INFO:tasks.rados.rados.0.smithi076.stdout:1945: copy_from oid 7 from oid 18 current snap is 218 2021-03-15T22:27:12.196 INFO:tasks.rados.rados.0.smithi076.stdout:1941: expect (ObjNum 495 snap 149 seq_num 495) 2021-03-15T22:27:12.235 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:12.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:12.261 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:12.262 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:12.287 INFO:tasks.rados.rados.0.smithi076.stdout:1941: done (4 left) 2021-03-15T22:27:12.288 INFO:tasks.rados.rados.0.smithi076.stdout:1946: write (excl) oid 32 current snap is 218 2021-03-15T22:27:12.288 INFO:tasks.rados.rados.0.smithi076.stdout:1942: finishing copy_from to smithi07694878-42 2021-03-15T22:27:12.288 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 1303 (ObjNum 654 snap 205 seq_num 654) dirty exists 2021-03-15T22:27:12.288 INFO:tasks.rados.rados.0.smithi076.stdout:1943: finishing write tid 1 to smithi07694878-30 2021-03-15T22:27:12.289 INFO:tasks.rados.rados.0.smithi076.stdout:1943: finishing write tid 2 to smithi07694878-30 2021-03-15T22:27:12.289 INFO:tasks.rados.rados.0.smithi076.stdout:1942: finishing copy_from racing read to smithi07694878-42 2021-03-15T22:27:12.289 INFO:tasks.rados.rados.0.smithi076.stdout:1943: finishing write tid 3 to smithi07694878-30 2021-03-15T22:27:12.289 INFO:tasks.rados.rados.0.smithi076.stdout:1943: finishing write tid 4 to smithi07694878-30 2021-03-15T22:27:12.289 INFO:tasks.rados.rados.0.smithi076.stdout:1943: finishing write tid 5 to smithi07694878-30 2021-03-15T22:27:12.290 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 1279 (ObjNum 680 snap 218 seq_num 680) dirty exists 2021-03-15T22:27:12.290 INFO:tasks.rados.rados.0.smithi076.stdout:1943: left oid 30 (ObjNum 680 snap 218 seq_num 680) 2021-03-15T22:27:12.290 INFO:tasks.rados.rados.0.smithi076.stdout:1944: finishing write tid 1 to smithi07694878-40 2021-03-15T22:27:12.290 INFO:tasks.rados.rados.0.smithi076.stdout:1944: finishing write tid 2 to smithi07694878-40 2021-03-15T22:27:12.291 INFO:tasks.rados.rados.0.smithi076.stdout:1944: finishing write tid 3 to smithi07694878-40 2021-03-15T22:27:12.291 INFO:tasks.rados.rados.0.smithi076.stdout:1944: finishing write tid 4 to smithi07694878-40 2021-03-15T22:27:12.291 INFO:tasks.rados.rados.0.smithi076.stdout:1946: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2021-03-15T22:27:12.294 INFO:tasks.rados.rados.0.smithi076.stdout:1946: writing smithi07694878-32 from 662374 to 1137609 tid 1 2021-03-15T22:27:12.300 INFO:tasks.rados.rados.0.smithi076.stdout:1946: writing smithi07694878-32 from 1609190 to 2286082 tid 2 2021-03-15T22:27:12.308 INFO:tasks.rados.rados.0.smithi076.stdout:1946: writing smithi07694878-32 from 2812481 to 3436582 tid 3 2021-03-15T22:27:12.308 INFO:tasks.rados.rados.0.smithi076.stdout:1942: done (4 left) 2021-03-15T22:27:12.308 INFO:tasks.rados.rados.0.smithi076.stdout:1943: done (3 left) 2021-03-15T22:27:12.309 INFO:tasks.rados.rados.0.smithi076.stdout:1947: delete oid 3 current snap is 218 2021-03-15T22:27:12.309 INFO:tasks.rados.rados.0.smithi076.stdout:1944: finishing write tid 5 to smithi07694878-40 2021-03-15T22:27:12.309 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1393 (ObjNum 681 snap 218 seq_num 681) dirty exists 2021-03-15T22:27:12.309 INFO:tasks.rados.rados.0.smithi076.stdout:1944: left oid 40 (ObjNum 681 snap 218 seq_num 681) 2021-03-15T22:27:12.309 INFO:tasks.rados.rados.0.smithi076.stdout:1945: finishing copy_from to smithi07694878-7 2021-03-15T22:27:12.310 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 809 (ObjNum 655 snap 206 seq_num 655) dirty exists 2021-03-15T22:27:12.310 INFO:tasks.rados.rados.0.smithi076.stdout:1945: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:27:12.310 INFO:tasks.rados.rados.0.smithi076.stdout:1946: finishing write tid 1 to smithi07694878-32 2021-03-15T22:27:12.310 INFO:tasks.rados.rados.0.smithi076.stdout:1946: finishing write tid 2 to smithi07694878-32 2021-03-15T22:27:12.311 INFO:tasks.rados.rados.0.smithi076.stdout:1944: done (3 left) 2021-03-15T22:27:12.311 INFO:tasks.rados.rados.0.smithi076.stdout:1945: done (2 left) 2021-03-15T22:27:12.311 INFO:tasks.rados.rados.0.smithi076.stdout:1947: done (1 left) 2021-03-15T22:27:12.311 INFO:tasks.rados.rados.0.smithi076.stdout:1948: snap_remove snap 195 2021-03-15T22:27:12.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:12.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:12.388 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:12.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:12.462 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2021-03-15T22:27:12.462 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:27:12.463 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:27:12.466 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-15T22:27:12.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:12.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:12 smithi076 conmon[56514]: cluster 2021-03-15T22:27:10.770747+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:27:12.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:12 smithi076 conmon[56514]: cluster 2021-03-15T22:27:11.080519+0000 mgr.y (mgr.14144) 700 : cluster [DBG] pgmap v900: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 464 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 201/1602 objects degraded (12.547%) 2021-03-15T22:27:12.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:12 smithi076 conmon[56514]: cluster 2021-03-15T22:27:11.357390+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:27:12.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:12 smithi076 conmon[56514]: cluster 2021-03-15T22:27:12.149403+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e517: 8 total, 7 up, 7 in 2021-03-15T22:27:12.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:12 smithi076 conmon[62156]: cluster 2021-03-15T22:27:10.770747 2021-03-15T22:27:12.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:12 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 107 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:27:12.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:12 smithi076 conmon[62156]: cluster 2021-03-15T22:27:11.080519+0000 mgr.y (mgr.14144) 700 : cluster [DBG] pgmap v900: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 464 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 201/1602 objects degraded (12.547%) 2021-03-15T22:27:12.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:12 smithi076 conmon[62156]: cluster 2021-03-15T22:27:11.357390+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:27:12.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:12 smithi076 conmon[62156]: cluster 2021-03-15T22:27:12.149403+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e517: 8 total, 7 up, 7 in 2021-03-15T22:27:12.531 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:12.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:12.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:12.554 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:27:12.635 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:12.657 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:12.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:12.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:12.741 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:12.761 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:12.783 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:12.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:12.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:12.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:12 smithi125 conmon[56739]: cluster 2021-03-15T22:27:10.770747+0000 osd.4 (osd.4) 107 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:27:12.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:12 smithi125 conmon[56739]: cluster 2021-03-15T22:27:11.080519+0000 mgr.y (mgr.14144) 700 : cluster [DBG] pgmap v900: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 464 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 201/1602 objects degraded (12.547%) 2021-03-15T22:27:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:12 smithi125 conmon[56739]: cluster 2021-03-15T22:27:11.357390+0000 osd.7 (osd.7) 101 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:27:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:12 smithi125 conmon[56739]: cluster 2021-03-15T22:27:12.149403+0000 mon.a (mon.0) 1446 : cluster [DBG] osdmap e517: 8 total, 7 up, 7 in 2021-03-15T22:27:12.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:12.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:12.909 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:12.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:12.999 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:13.011 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:13.035 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:13.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:13.053 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:13.137 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:13.159 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:13.160 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:13.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:13.284 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:13.286 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:13.298 INFO:tasks.rados.rados.0.smithi076.stdout:1948: done (1 left) 2021-03-15T22:27:13.298 INFO:tasks.rados.rados.0.smithi076.stdout:1949: snap_create 2021-03-15T22:27:13.298 INFO:tasks.rados.rados.0.smithi076.stdout:1946: finishing write tid 3 to smithi07694878-32 2021-03-15T22:27:13.299 INFO:tasks.rados.rados.0.smithi076.stdout:1946: finishing write tid 4 to smithi07694878-32 2021-03-15T22:27:13.299 INFO:tasks.rados.rados.0.smithi076.stdout:1946: finishing write tid 5 to smithi07694878-32 2021-03-15T22:27:13.299 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 1398 (ObjNum 682 snap 218 seq_num 682) dirty exists 2021-03-15T22:27:13.299 INFO:tasks.rados.rados.0.smithi076.stdout:1946: left oid 32 (ObjNum 682 snap 218 seq_num 682) 2021-03-15T22:27:13.387 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:13.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:13.410 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:13.410 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:13.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:13.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:13 smithi076 conmon[56514]: cluster 2021-03-15T22:27:11.773011 2021-03-15T22:27:13.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:13 smithi076 conmon[56514]: +0000 osd.4 (osd.4) 108 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:27:13.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:13 smithi076 conmon[56514]: cluster 2021-03-15T22:27:12.404490+0000 osd.7 (osd.7) 102 : cluster [DBG] 3.d scrub ok 2021-03-15T22:27:13.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:13 smithi076 conmon[56514]: cluster 2021-03-15T22:27:12.679699+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.b scrub ok 2021-03-15T22:27:13.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:13 smithi076 conmon[62156]: cluster 2021-03-15T22:27:11 2021-03-15T22:27:13.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:13 smithi076 conmon[62156]: .773011+0000 osd.4 (osd.4) 108 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:27:13.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:13 smithi076 conmon[62156]: cluster 2021-03-15T22:27:12.404490+0000 osd.7 (osd.7) 102 : cluster [DBG] 3.d scrub ok 2021-03-15T22:27:13.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:13 smithi076 conmon[62156]: cluster 2021-03-15T22:27:12.679699+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.b scrub ok 2021-03-15T22:27:13.533 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:13.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:13.535 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:13.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:13 smithi125 conmon[56739]: cluster 2021-03-15T22:27:11.773011 2021-03-15T22:27:13.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:13 smithi125 conmon[56739]: +0000 osd.4 (osd.4) 108 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:27:13.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:13 smithi125 conmon[56739]: cluster 2021-03-15T22:27:12.404490+0000 osd.7 (osd.7) 102 : cluster [DBG] 3.d scrub ok 2021-03-15T22:27:13.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:13 smithi125 conmon[56739]: cluster 2021-03-15T22:27:12.679699+0000 osd.3 (osd.3) 88 : cluster [DBG] 6.b scrub ok 2021-03-15T22:27:13.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:13.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:13.658 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:13.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:13.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:13.786 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:13.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:13.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:13.911 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:13.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:14.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:14.037 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:14.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:14.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:14.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:14.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:14.163 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:14.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:14.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:14.288 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:14.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:14.289 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:14.302 INFO:tasks.rados.rados.0.smithi076.stdout:1946: done (1 left) 2021-03-15T22:27:14.302 INFO:tasks.rados.rados.0.smithi076.stdout:1949: done (0 left) 2021-03-15T22:27:14.303 INFO:tasks.rados.rados.0.smithi076.stdout:1950: delete oid 12 current snap is 219 2021-03-15T22:27:14.310 INFO:tasks.rados.rados.0.smithi076.stdout:1950: done (0 left) 2021-03-15T22:27:14.310 INFO:tasks.rados.rados.0.smithi076.stdout:1951: read oid 5 snap -1 2021-03-15T22:27:14.310 INFO:tasks.rados.rados.0.smithi076.stdout:1951: expect (ObjNum 668 snap 212 seq_num 668) 2021-03-15T22:27:14.310 INFO:tasks.rados.rados.0.smithi076.stdout:1952: delete oid 27 current snap is 219 2021-03-15T22:27:14.311 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:27:14.311 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:27:14.312 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:27:14.312 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:27:14.312 INFO:tasks.rados.rados.0.smithi076.stdout:1952: done (1 left) 2021-03-15T22:27:14.313 INFO:tasks.rados.rados.0.smithi076.stdout:1953: copy_from oid 31 from oid 14 current snap is 219 2021-03-15T22:27:14.317 INFO:tasks.rados.rados.0.smithi076.stdout:1954: write oid 25 current snap is 219 2021-03-15T22:27:14.317 INFO:tasks.rados.rados.0.smithi076.stdout:1954: seq_num 686 ranges {729602=492584,1843118=684230,3322173=207429} 2021-03-15T22:27:14.319 INFO:tasks.rados.rados.0.smithi076.stdout:1954: writing smithi07694878-25 from 729602 to 1222186 tid 1 2021-03-15T22:27:14.326 INFO:tasks.rados.rados.0.smithi076.stdout:1954: writing smithi07694878-25 from 1843118 to 2527348 tid 2 2021-03-15T22:27:14.332 INFO:tasks.rados.rados.0.smithi076.stdout:1954: writing smithi07694878-25 from 3322173 to 3529602 tid 3 2021-03-15T22:27:14.333 INFO:tasks.rados.rados.0.smithi076.stdout:1955: write (excl) oid 13 current snap is 219 2021-03-15T22:27:14.333 INFO:tasks.rados.rados.0.smithi076.stdout:1955: seq_num 687 ranges {746409=567833,1901600=544241,3065949=480460} 2021-03-15T22:27:14.336 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:27:14.339 INFO:tasks.rados.rados.0.smithi076.stdout:1955: writing smithi07694878-13 from 746409 to 1314242 tid 1 2021-03-15T22:27:14.345 INFO:tasks.rados.rados.0.smithi076.stdout:1955: writing smithi07694878-13 from 1901600 to 2445841 tid 2 2021-03-15T22:27:14.356 INFO:tasks.rados.rados.0.smithi076.stdout:1955: writing smithi07694878-13 from 3065949 to 3546409 tid 3 2021-03-15T22:27:14.357 INFO:tasks.rados.rados.0.smithi076.stdout:1956: write (excl) oid 17 current snap is 219 2021-03-15T22:27:14.357 INFO:tasks.rados.rados.0.smithi076.stdout:1956: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2021-03-15T22:27:14.359 INFO:tasks.rados.rados.0.smithi076.stdout:1956: writing smithi07694878-17 from 763216 to 1406298 tid 1 2021-03-15T22:27:14.362 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:14 smithi125 conmon[56739]: cluster 2021-03-15T22:27:12.689069+0000 osd.1 (osd 2021-03-15T22:27:14.363 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:14 smithi125 conmon[56739]: .1) 101 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:14.363 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:14 smithi125 conmon[56739]: cluster 2021-03-15T22:27:12.690015+0000 osd.1 (osd.1) 102 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:14.363 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:14 smithi125 conmon[56739]: cluster 2021-03-15T22:27:12.808288+0000 osd.4 (osd.4) 109 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:27:14.364 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:14 smithi125 conmon[56739]: cluster 2021-03-15T22:27:13.081074+0000 mgr.y (mgr.14144) 701 : cluster [DBG] pgmap v903: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 471 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 21 op/s; 202/1611 objects degraded (12.539%) 2021-03-15T22:27:14.364 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:14 smithi125 conmon[56739]: cluster 2021-03-15T22:27:13.294088+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e518: 8 total, 7 up, 7 in 2021-03-15T22:27:14.364 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:14 smithi125 conmon[56739]: audit 2021-03-15T22:27:14.012451+0000 mon.a (mon.0) 1448 : audit [INF] from='client.? 172.21.15.76:0/3713738496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:14.366 INFO:tasks.rados.rados.0.smithi076.stdout:1956: writing smithi07694878-17 from 1843729 to 2531628 tid 2 2021-03-15T22:27:14.372 INFO:tasks.rados.rados.0.smithi076.stdout:1956: writing smithi07694878-17 from 3260666 to 3563216 tid 3 2021-03-15T22:27:14.372 INFO:tasks.rados.rados.0.smithi076.stdout:1957: snap_create 2021-03-15T22:27:14.372 INFO:tasks.rados.rados.0.smithi076.stdout:1951: expect (ObjNum 668 snap 212 seq_num 668) 2021-03-15T22:27:14.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:14.414 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:14.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:14.416 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:14.517 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:14.523 INFO:tasks.rados.rados.0.smithi076.stdout:1953: finishing copy_from to smithi07694878-31 2021-03-15T22:27:14.524 INFO:tasks.rados.rados.0.smithi076.stdout:1953: got expected ENOENT (src dne) 2021-03-15T22:27:14.525 INFO:tasks.rados.rados.0.smithi076.stdout:1953: finishing copy_from racing read to smithi07694878-31 2021-03-15T22:27:14.525 INFO:tasks.rados.rados.0.smithi076.stdout:1954: finishing write tid 1 to smithi07694878-25 2021-03-15T22:27:14.525 INFO:tasks.rados.rados.0.smithi076.stdout:1954: finishing write tid 2 to smithi07694878-25 2021-03-15T22:27:14.525 INFO:tasks.rados.rados.0.smithi076.stdout:1954: finishing write tid 3 to smithi07694878-25 2021-03-15T22:27:14.526 INFO:tasks.rados.rados.0.smithi076.stdout:1954: finishing write tid 4 to smithi07694878-25 2021-03-15T22:27:14.526 INFO:tasks.rados.rados.0.smithi076.stdout:1954: finishing write tid 5 to smithi07694878-25 2021-03-15T22:27:14.526 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 564 (ObjNum 686 snap 219 seq_num 686) dirty exists 2021-03-15T22:27:14.526 INFO:tasks.rados.rados.0.smithi076.stdout:1954: left oid 25 (ObjNum 686 snap 219 seq_num 686) 2021-03-15T22:27:14.527 INFO:tasks.rados.rados.0.smithi076.stdout:1955: finishing write tid 1 to smithi07694878-13 2021-03-15T22:27:14.527 INFO:tasks.rados.rados.0.smithi076.stdout:1955: finishing write tid 2 to smithi07694878-13 2021-03-15T22:27:14.527 INFO:tasks.rados.rados.0.smithi076.stdout:1955: finishing write tid 3 to smithi07694878-13 2021-03-15T22:27:14.527 INFO:tasks.rados.rados.0.smithi076.stdout:1955: finishing write tid 4 to smithi07694878-13 2021-03-15T22:27:14.527 INFO:tasks.rados.rados.0.smithi076.stdout:1955: finishing write tid 5 to smithi07694878-13 2021-03-15T22:27:14.528 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 494 (ObjNum 687 snap 219 seq_num 687) dirty exists 2021-03-15T22:27:14.528 INFO:tasks.rados.rados.0.smithi076.stdout:1955: left oid 13 (ObjNum 687 snap 219 seq_num 687) 2021-03-15T22:27:14.528 INFO:tasks.rados.rados.0.smithi076.stdout:1956: finishing write tid 1 to smithi07694878-17 2021-03-15T22:27:14.528 INFO:tasks.rados.rados.0.smithi076.stdout:1956: finishing write tid 2 to smithi07694878-17 2021-03-15T22:27:14.529 INFO:tasks.rados.rados.0.smithi076.stdout:1956: finishing write tid 3 to smithi07694878-17 2021-03-15T22:27:14.529 INFO:tasks.rados.rados.0.smithi076.stdout:1956: finishing write tid 4 to smithi07694878-17 2021-03-15T22:27:14.529 INFO:tasks.rados.rados.0.smithi076.stdout:1956: finishing write tid 5 to smithi07694878-17 2021-03-15T22:27:14.530 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 839 (ObjNum 688 snap 219 seq_num 688) dirty exists 2021-03-15T22:27:14.530 INFO:tasks.rados.rados.0.smithi076.stdout:1956: left oid 17 (ObjNum 688 snap 219 seq_num 688) 2021-03-15T22:27:14.539 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:14.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:14.540 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:14.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:14.663 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:14.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:14.665 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:14.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[62156]: cluster 2021-03-15T22:27:12.689069+0000 osd.1 (osd.1 2021-03-15T22:27:14.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[62156]: ) 101 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:14.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[62156]: cluster 2021-03-15T22:27:12.690015+0000 osd.1 (osd.1) 102 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:14.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[62156]: cluster 2021-03-15T22:27:12.808288+0000 osd.4 (osd.4) 109 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:27:14.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[62156]: cluster 2021-03-15T22:27:13.081074+0000 mgr.y 2021-03-15T22:27:14.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[62156]: (mgr.14144) 701 : cluster [DBG] pgmap v903: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 471 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 21 op/s; 202/1611 objects degraded (12.539%) 2021-03-15T22:27:14.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[62156]: cluster 2021-03-15T22:27:13 2021-03-15T22:27:14.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[62156]: .294088+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e518: 8 total, 7 up, 7 in 2021-03-15T22:27:14.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:27:14.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[62156]: 27:14.012451+0000 mon.a (mon.0) 1448 : audit [INF] 2021-03-15T22:27:14.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3713738496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:14.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[56514]: cluster 2021-03-15T22:27:12.689069+0000 osd.1 (osd.1) 101 : 2021-03-15T22:27:14.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[56514]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:14.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[56514]: cluster 2021-03-15T22:27:12.690015+0000 osd.1 (osd.1) 102 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:14.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[56514]: cluster 2021-03-15T22:27:12.808288+0000 osd.4 (osd.4) 109 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:27:14.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[56514]: cluster 2021-03-15T22:27:13.081074+0000 mgr.y (mgr.14144) 701 : cluster [DBG] pgmap v903: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 471 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 15 MiB/s rd, 2.5 MiB/s wr, 21 op/s; 202/1611 objects degraded (12.539%) 2021-03-15T22:27:14.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[56514]: cluster 2021-03-15T22:27:13.294088+0000 mon.a (mon.0) 1447 : cluster [DBG] osdmap e518: 8 total, 7 up, 7 in 2021-03-15T22:27:14.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:14 smithi076 conmon[56514]: audit 2021-03-15T22:27:14.012451+0000 mon.a (mon.0) 1448 : audit [INF] from='client.? 172.21.15.76:0/3713738496' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:14.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:14.788 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:14.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:14.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:14.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:14.915 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:14.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:14.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:14.958 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-15T22:27:15.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:15.040 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:15.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:15.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:15.142 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:15.165 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:15.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:15.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:15.234 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:15.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:15.291 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:15.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:15.293 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:15.306 INFO:tasks.rados.rados.0.smithi076.stdout:1951: done (5 left) 2021-03-15T22:27:15.307 INFO:tasks.rados.rados.0.smithi076.stdout:1953: done (4 left) 2021-03-15T22:27:15.307 INFO:tasks.rados.rados.0.smithi076.stdout:1954: done (3 left) 2021-03-15T22:27:15.307 INFO:tasks.rados.rados.0.smithi076.stdout:1955: done (2 left) 2021-03-15T22:27:15.308 INFO:tasks.rados.rados.0.smithi076.stdout:1956: done (1 left) 2021-03-15T22:27:15.308 INFO:tasks.rados.rados.0.smithi076.stdout:1957: done (0 left) 2021-03-15T22:27:15.309 INFO:tasks.rados.rados.0.smithi076.stdout:1958: read oid 21 snap 196 2021-03-15T22:27:15.309 INFO:tasks.rados.rados.0.smithi076.stdout:1958: expect (ObjNum 612 snap 189 seq_num 612) 2021-03-15T22:27:15.309 INFO:tasks.rados.rados.0.smithi076.stdout:1959: read oid 45 snap -1 2021-03-15T22:27:15.310 INFO:tasks.rados.rados.0.smithi076.stdout:1959: expect (ObjNum 676 snap 214 seq_num 676) 2021-03-15T22:27:15.310 INFO:tasks.rados.rados.0.smithi076.stdout:1960: rollback oid 16 current snap is 220 2021-03-15T22:27:15.311 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 199 2021-03-15T22:27:15.311 INFO:tasks.rados.rados.0.smithi076.stdout:1961: read oid 5 snap -1 2021-03-15T22:27:15.312 INFO:tasks.rados.rados.0.smithi076.stdout:1961: expect (ObjNum 668 snap 212 seq_num 668) 2021-03-15T22:27:15.312 INFO:tasks.rados.rados.0.smithi076.stdout:1962: write oid 41 current snap is 220 2021-03-15T22:27:15.312 INFO:tasks.rados.rados.0.smithi076.stdout:1962: seq_num 689 ranges {780023=718331,1902211=547910,3004442=575581} 2021-03-15T22:27:15.314 INFO:tasks.rados.rados.0.smithi076.stdout:1962: writing smithi07694878-41 from 780023 to 1498354 tid 1 2021-03-15T22:27:15.320 INFO:tasks.rados.rados.0.smithi076.stdout:1962: writing smithi07694878-41 from 1902211 to 2450121 tid 2 2021-03-15T22:27:15.328 INFO:tasks.rados.rados.0.smithi076.stdout:1962: writing smithi07694878-41 from 3004442 to 3580023 tid 3 2021-03-15T22:27:15.328 INFO:tasks.rados.rados.0.smithi076.stdout:1963: read oid 19 snap -1 2021-03-15T22:27:15.328 INFO:tasks.rados.rados.0.smithi076.stdout:1963: expect (ObjNum 495 snap 149 seq_num 495) 2021-03-15T22:27:15.328 INFO:tasks.rados.rados.0.smithi076.stdout:1964: snap_create 2021-03-15T22:27:15.329 INFO:tasks.rados.rados.0.smithi076.stdout:1961: expect (ObjNum 668 snap 212 seq_num 668) 2021-03-15T22:27:15.394 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:15.416 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:15.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:15.418 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:15.488 INFO:tasks.rados.rados.0.smithi076.stdout:1959: expect (ObjNum 676 snap 214 seq_num 676) 2021-03-15T22:27:15.519 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:15.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:15.542 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:15.543 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:15.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:27:15.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[56514]: 27:14.291292+0000 mon.a (mon.0) 1449 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:27:15.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[56514]: audit 2021-03-15T22:27:14.298186+0000 mon.a (mon.0) 1450 : audit [INF] from='client.? 172.21.15.76:0/3713738496' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:15.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[56514]: cluster 2021-03-15T22:27:14.298243+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e519: 8 total, 7 up, 7 in 2021-03-15T22:27:15.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[56514]: cluster 2021-03-15T22:27:14.352623+0000 osd.7 (osd.7) 103 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:27:15.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[56514]: cluster 2021-03-15T22:27:15.134050+0000 mon.a (mon.0) 1452 : cluster [WRN] Health check update: Degraded data redundancy: 202/1611 objects degraded (12.539%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:15.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: cluster 2021-03-15T22:27:14.291292+0000 2021-03-15T22:27:15.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: mon.a (mon.0) 1449 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:27:15.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: audit 2021 2021-03-15T22:27:15.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: -03-15T22:27:14.298186+0000 mon.a (mon.0) 1450 : 2021-03-15T22:27:15.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/3713738496' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:15.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:27:15.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: 15T22:27:14.298243+0000 mon.a ( 2021-03-15T22:27:15.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: mon.0) 1451 : cluster [DBG] osdmap e519: 8 total, 7 up, 7 in 2021-03-15T22:27:15.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: cluster 2021-03-15T22:27:15.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: 2021-03-15T22:27:14.352623+0000 2021-03-15T22:27:15.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: osd.7 (osd.7) 103 : cluster [DBG] 2021-03-15T22:27:15.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: 4.1e scrub ok 2021-03-15T22:27:15.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:27:15.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: :27:15.134050+0000 mon.a (mon.0 2021-03-15T22:27:15.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:15 smithi076 conmon[62156]: ) 1452 : cluster [WRN] Health check update: Degraded data redundancy: 202/1611 objects degraded (12.539%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:15.641 INFO:tasks.rados.rados.0.smithi076.stdout:1958: expect (ObjNum 612 snap 189 seq_num 612) 2021-03-15T22:27:15.644 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:15.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:15 smithi125 conmon[56739]: cluster 2021-03-15T22:27:14.291292+0000 mon.a (mon.0) 1449 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:27:15.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:15 smithi125 conmon[56739]: audit 2021-03-15T22:27:14.298186+0000 mon.a (mon.0) 1450 : audit [INF] from='client.? 172.21.15.76:0/3713738496' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:15.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:15 smithi125 conmon[56739]: cluster 2021-03-15T22:27:14.298243+0000 mon.a (mon.0) 1451 : cluster [DBG] osdmap e519: 8 total, 7 up, 7 in 2021-03-15T22:27:15.646 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:15 smithi125 conmon[56739]: cluster 2021-03-15T22:27:14.352623+0000 osd.7 (osd.7) 103 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:27:15.646 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:15 smithi125 conmon[56739]: cluster 2021-03-15T22:27:15.134050+0000 mon.a (mon.0) 1452 : cluster [WRN] Health check update: Degraded data redundancy: 202/1611 objects degraded (12.539%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:15.668 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:15.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:15.670 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:15.771 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:15.792 INFO:tasks.rados.rados.0.smithi076.stdout:1960: finishing rollback tid 0 to smithi07694878-16 2021-03-15T22:27:15.792 INFO:tasks.rados.rados.0.smithi076.stdout:1960: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:27:15.792 INFO:tasks.rados.rados.0.smithi076.stdout:1960: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:27:15.792 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1290 (ObjNum 604 snap 188 seq_num 604) dirty exists 2021-03-15T22:27:15.793 INFO:tasks.rados.rados.0.smithi076.stdout:1962: finishing write tid 1 to smithi07694878-41 2021-03-15T22:27:15.793 INFO:tasks.rados.rados.0.smithi076.stdout:1963: expect (ObjNum 495 snap 149 seq_num 495) 2021-03-15T22:27:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:15.795 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:15.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:15.868 INFO:tasks.rados.rados.0.smithi076.stdout:1962: finishing write tid 2 to smithi07694878-41 2021-03-15T22:27:15.868 INFO:tasks.rados.rados.0.smithi076.stdout:1962: finishing write tid 3 to smithi07694878-41 2021-03-15T22:27:15.869 INFO:tasks.rados.rados.0.smithi076.stdout:1962: finishing write tid 4 to smithi07694878-41 2021-03-15T22:27:15.869 INFO:tasks.rados.rados.0.smithi076.stdout:1962: finishing write tid 5 to smithi07694878-41 2021-03-15T22:27:15.869 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 1173 (ObjNum 689 snap 220 seq_num 689) dirty exists 2021-03-15T22:27:15.869 INFO:tasks.rados.rados.0.smithi076.stdout:1962: left oid 41 (ObjNum 689 snap 220 seq_num 689) 2021-03-15T22:27:15.896 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:15.918 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:15.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:15.920 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:16.021 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:16.045 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:16.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:16.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:16.148 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:16.171 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:16.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:16.172 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:16.203 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:27:16.210 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:27:16.213 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:27:16.228 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:27:16.246 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:27:16.264 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:27:16.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:16.282 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:27:16.296 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:16.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:16.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:16.302 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:27:16.313 INFO:tasks.rados.rados.0.smithi076.stdout:1958: done (6 left) 2021-03-15T22:27:16.314 INFO:tasks.rados.rados.0.smithi076.stdout:1959: done (5 left) 2021-03-15T22:27:16.314 INFO:tasks.rados.rados.0.smithi076.stdout:1960: done (4 left) 2021-03-15T22:27:16.314 INFO:tasks.rados.rados.0.smithi076.stdout:1961: done (3 left) 2021-03-15T22:27:16.314 INFO:tasks.rados.rados.0.smithi076.stdout:1962: done (2 left) 2021-03-15T22:27:16.315 INFO:tasks.rados.rados.0.smithi076.stdout:1963: done (1 left) 2021-03-15T22:27:16.315 INFO:tasks.rados.rados.0.smithi076.stdout:1964: done (0 left) 2021-03-15T22:27:16.315 INFO:tasks.rados.rados.0.smithi076.stdout:1965: rollback oid 31 current snap is 221 2021-03-15T22:27:16.315 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 31 to 202 2021-03-15T22:27:16.316 INFO:tasks.rados.rados.0.smithi076.stdout:1966: snap_create 2021-03-15T22:27:16.318 INFO:tasks.rados.rados.0.smithi076.stdout:1965: finishing rollback tid 0 to smithi07694878-31 2021-03-15T22:27:16.319 INFO:tasks.rados.rados.0.smithi076.stdout:1965: finishing rollback tid 1 to smithi07694878-31 2021-03-15T22:27:16.319 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 851 (ObjNum 261808336 snap 21896 seq_num 261675040) dirty dne 2021-03-15T22:27:16.327 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:27:16.328 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:27:16.328 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:27:16.328 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:27:16.329 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:27:16.329 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:27:16.329 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:27:16.330 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:27:16.330 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:27:16.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[56514]: cluster 2021-03-15T22:27:15.081724+0000 mgr.y (mgr.14144) 702 : cluster [DBG] pgmap v906: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 476 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 202/1614 objects degraded (12.515%) 2021-03-15T22:27:16.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[56514]: cluster 2021-03-15T22:27:15.301990+0000 mon.a (mon.0) 1453 : cluster [DBG] osdmap e520: 8 total, 7 up, 7 in 2021-03-15T22:27:16.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[56514]: cluster 2021-03-15T22:27:15.539069+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:27:16.397 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[56514]: audit 2021-03-15T22:27:16.176104+0000 mon.a (mon.0) 1454 : audit [DBG] from='client.? 172.21.15.76:0/2956527180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:16.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: cluster 2021-03-15T22:27:15.081724 2021-03-15T22:27:16.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 702 : cluster [DBG] pgmap v906: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 476 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 202/1614 objects degraded (12.515%) 2021-03-15T22:27:16.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: cluster 2021-03-15T22:27:15 2021-03-15T22:27:16.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: .301990+0000 mon.a (mon.0) 1453 : cluster 2021-03-15T22:27:16.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: [DBG] osdmap e520: 8 total, 7 up, 7 in 2021-03-15T22:27:16.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: 2021-03-15T22:27:16.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:27:16.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: 22:27:15.539069+0000 osd.5 (osd. 2021-03-15T22:27:16.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: 5) 92 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:27:16.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:27:16.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: T22:27:16.176104+0000 mon.a (mon.0) 2021-03-15T22:27:16.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:16 smithi076 conmon[62156]: 1454 : audit [DBG] from='client.? 172.21.15.76:0/2956527180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:16.401 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:16.427 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:16.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:16.428 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:16.529 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:16.551 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:16.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:16.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:16.602 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:16 smithi125 conmon[56739]: cluster 2021-03-15T22:27:15.081724+0000 mgr.y (mgr 2021-03-15T22:27:16.602 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:16 smithi125 conmon[56739]: .14144) 702 : cluster [DBG] pgmap v906: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 476 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.3 MiB/s wr, 16 op/s; 202/1614 objects degraded (12.515%) 2021-03-15T22:27:16.602 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:16 smithi125 conmon[56739]: cluster 2021-03-15T22:27:15.301990+0000 mon.a (mon.0) 1453 : cluster [DBG] osdmap e520: 8 total, 7 up, 7 in 2021-03-15T22:27:16.603 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:16 smithi125 conmon[56739]: cluster 2021-03-15T22:27:15.539069+0000 osd.5 (osd.5) 92 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:27:16.603 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:16 smithi125 conmon[56739]: audit 2021-03-15T22:27:16.176104+0000 mon.a (mon.0) 1454 : audit [DBG] from='client.? 172.21.15.76:0/2956527180' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:16.654 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:16.676 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:16.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:16.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:16.779 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:16.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:16.802 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:16.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:16.834 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:27:16.835 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:27:16.904 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:16.926 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:16.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:16.928 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:17.029 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:17.051 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:17.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:17.052 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:17.153 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:17.175 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:17.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:17.177 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:17.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:17.302 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:17.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:17.303 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:17.322 INFO:tasks.rados.rados.0.smithi076.stdout:1965: done (1 left) 2021-03-15T22:27:17.323 INFO:tasks.rados.rados.0.smithi076.stdout:1966: done (0 left) 2021-03-15T22:27:17.323 INFO:tasks.rados.rados.0.smithi076.stdout:1967: write oid 29 current snap is 222 2021-03-15T22:27:17.323 INFO:tasks.rados.rados.0.smithi076.stdout:1967: seq_num 690 ranges {796830=793580,2360693=691568} 2021-03-15T22:27:17.332 INFO:tasks.rados.rados.0.smithi076.stdout:1967: writing smithi07694878-29 from 796830 to 1590410 tid 1 2021-03-15T22:27:17.342 INFO:tasks.rados.rados.0.smithi076.stdout:1967: writing smithi07694878-29 from 2360693 to 3052261 tid 2 2021-03-15T22:27:17.342 INFO:tasks.rados.rados.0.smithi076.stdout:1968: write oid 32 current snap is 222 2021-03-15T22:27:17.342 INFO:tasks.rados.rados.0.smithi076.stdout:1967: finishing write tid 1 to smithi07694878-29 2021-03-15T22:27:17.343 INFO:tasks.rados.rados.0.smithi076.stdout:1968: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2021-03-15T22:27:17.347 INFO:tasks.rados.rados.0.smithi076.stdout:1968: writing smithi07694878-32 from 413637 to 882466 tid 1 2021-03-15T22:27:17.354 INFO:tasks.rados.rados.0.smithi076.stdout:1968: writing smithi07694878-32 from 1619175 to 2054401 tid 2 2021-03-15T22:27:17.363 INFO:tasks.rados.rados.0.smithi076.stdout:1968: writing smithi07694878-32 from 2542935 to 3301190 tid 3 2021-03-15T22:27:17.363 INFO:tasks.rados.rados.0.smithi076.stdout:1969: write (excl) oid 8 current snap is 222 2021-03-15T22:27:17.363 INFO:tasks.rados.rados.0.smithi076.stdout:1967: finishing write tid 2 to smithi07694878-29 2021-03-15T22:27:17.364 INFO:tasks.rados.rados.0.smithi076.stdout:1969: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2021-03-15T22:27:17.369 INFO:tasks.rados.rados.0.smithi076.stdout:1969: writing smithi07694878-8 from 430444 to 1090875 tid 1 2021-03-15T22:27:17.377 INFO:tasks.rados.rados.0.smithi076.stdout:1969: writing smithi07694878-8 from 1677657 to 2372894 tid 2 2021-03-15T22:27:17.385 INFO:tasks.rados.rados.0.smithi076.stdout:1969: writing smithi07694878-8 from 2970358 to 3539885 tid 3 2021-03-15T22:27:17.385 INFO:tasks.rados.rados.0.smithi076.stdout:1970: copy_from oid 10 from oid 42 current snap is 222 2021-03-15T22:27:17.385 INFO:tasks.rados.rados.0.smithi076.stdout:1971: rollback oid 27 current snap is 222 2021-03-15T22:27:17.386 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 214 2021-03-15T22:27:17.386 INFO:tasks.rados.rados.0.smithi076.stdout:1972: copy_from oid 33 from oid 2 current snap is 222 2021-03-15T22:27:17.386 INFO:tasks.rados.rados.0.smithi076.stdout:1973: rollback oid 24 current snap is 222 2021-03-15T22:27:17.386 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 221 2021-03-15T22:27:17.387 INFO:tasks.rados.rados.0.smithi076.stdout:1974: read oid 16 snap -1 2021-03-15T22:27:17.387 INFO:tasks.rados.rados.0.smithi076.stdout:1974: expect (ObjNum 604 snap 188 seq_num 604) 2021-03-15T22:27:17.388 INFO:tasks.rados.rados.0.smithi076.stdout:1975: write oid 38 current snap is 222 2021-03-15T22:27:17.388 INFO:tasks.rados.rados.0.smithi076.stdout:1975: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2021-03-15T22:27:17.392 INFO:tasks.rados.rados.0.smithi076.stdout:1975: writing smithi07694878-38 from 447251 to 1182931 tid 1 2021-03-15T22:27:17.397 INFO:tasks.rados.rados.0.smithi076.stdout:1975: writing smithi07694878-38 from 1736139 to 2175034 tid 2 2021-03-15T22:27:17.404 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:17.405 INFO:tasks.rados.rados.0.smithi076.stdout:1975: writing smithi07694878-38 from 2597781 to 3378580 tid 3 2021-03-15T22:27:17.406 INFO:tasks.rados.rados.0.smithi076.stdout:1976: write (excl) oid 13 current snap is 222 2021-03-15T22:27:17.406 INFO:tasks.rados.rados.0.smithi076.stdout:1967: finishing write tid 3 to smithi07694878-29 2021-03-15T22:27:17.406 INFO:tasks.rados.rados.0.smithi076.stdout:1976: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2021-03-15T22:27:17.410 INFO:tasks.rados.rados.0.smithi076.stdout:1976: writing smithi07694878-13 from 464058 to 874987 tid 1 2021-03-15T22:27:17.417 INFO:tasks.rados.rados.0.smithi076.stdout:1976: writing smithi07694878-13 from 1394621 to 2093527 tid 2 2021-03-15T22:27:17.418 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":907,"stamp":"2021-03-15T22:27:15.300286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":499209324,"num_objects":538,"num_object_clones":172,"num_object_copies":1614,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":202,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":538,"num_whiteouts":12,"num_read":5380,"num_read_kb":3873249,"num_write":3286,"num_write_kb":1019902,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1632,"num_bytes_recovered":514352340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9395,"ondisk_log_size":9395,"up":360,"acting":360,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":415,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1221044,"kb_used_data":1029884,"kb_used_omap":0,"kb_used_meta":191104,"kb_avail":654995020,"statfs":{"total":671965249536,"available":670714900480,"internally_reserved":0,"allocated":1054601216,"data_stored":1450924379,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":195690496},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":12838395,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":-2,"num_read":45,"num_read_kb":45480,"num_write":21,"num_write_kb":9303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.157453"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"485","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.136446+0000","last_change":"2021-03-15T22:26:39.963141+0000","last_active":"2021-03-15T22:27:10.136446+0000","last_peered":"2021-03-15T22:27:10.136446+0000","last_clean":"2021-03-15T22:27:10.136446+0000","last_became_active":"2021-03-15T22:18:00.020155+0000","last_became_peered":"2021-03-15T22:18:00.020155+0000","last_unstale":"2021-03-15T22:27:10.136446+0000","last_undegraded":"2021-03-15T22:27:10.136446+0000","last_fullsized":"2021-03-15T22:27:10.136446+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:39.963067+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:39.963067+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"193","reported_epoch":"516","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:11.250374+0000","last_change":"2021-03-15T22:26:46.155373+0000","last_active":"2021-03-15T22:27:11.250374+0000","last_peered":"2021-03-15T22:27:11.250374+0000","last_clean":"2021-03-15T22:26:44.404362+0000","last_became_active":"2021-03-15T22:26:46.155373+0000","last_became_peered":"2021-03-15T22:26:46.155373+0000","last_unstale":"2021-03-15T22:27:11.250374+0000","last_undegraded":"2021-03-15T22:26:46.151422+0000","last_fullsized":"2021-03-15T22:26:46.151306+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:26:07.787524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.787524+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[{"shards":"5,6","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"65'15","reported_seq":"515","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.136311+0000","last_change":"2021-03-15T22:26:11.025622+0000","last_active":"2021-03-15T22:27:10.136311+0000","last_peered":"2021-03-15T22:27:10.136311+0000","last_clean":"2021-03-15T22:27:10.136311+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:27:10.136311+0000","last_undegraded":"2021-03-15T22:27:10.136311+0000","last_fullsized":"2021-03-15T22:27:10.136311+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:26:11.025548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:11.025548+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":35,"num_read_kb":17,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"108","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.656442+0000","last_change":"2021-03-15T22:25:33.149020+0000","last_active":"2021-03-15T22:27:10.656442+0000","last_peered":"2021-03-15T22:27:10.656442+0000","last_clean":"2021-03-15T22:27:10.656442+0000","last_became_active":"2021-03-15T22:25:33.148710+0000","last_became_peered":"2021-03-15T22:25:33.148710+0000","last_unstale":"2021-03-15T22:27:10.656442+0000","last_undegraded":"2021-03-15T22:27:10.656442+0000","last_fullsized":"2021-03-15T22:27:10.656442+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:06.177754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:06.177754+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"114","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.657064+0000","last_change":"2021-03-15T22:26:46.154923+0000","last_active":"2021-03-15T22:27:10.657064+0000","last_peered":"2021-03-15T22:27:10.657064+0000","last_clean":"2021-03-15T22:26:44.343907+0000","last_became_active":"2021-03-15T22:26:46.154923+0000","last_became_peered":"2021-03-15T22:26:46.154923+0000","last_unstale":"2021-03-15T22:27:10.657064+0000","last_undegraded":"2021-03-15T22:27:10.657064+0000","last_fullsized":"2021-03-15T22:26:46.144953+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:15.333746+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.333746+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"65'20","reported_seq":"140","reported_epoch":"515","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:10.656178+0000","last_change":"2021-03-15T22:26:46.153984+0000","last_active":"2021-03-15T22:27:10.656178+0000","last_peered":"2021-03-15T22:27:10.656178+0000","last_clean":"2021-03-15T22:26:44.343419+0000","last_became_active":"2021-03-15T22:26:46.153984+0000","last_became_peered":"2021-03-15T22:26:46.153984+0000","last_unstale":"2021-03-15T22:27:10.656178+0000","last_undegraded":"2021-03-15T22:26:46.143935+0000","last_fullsized":"2021-03-15T22:26:46.143826+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:23:42.144821+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:23:42.144821+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":40,"num_read_kb":21,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1780,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"183","reported_epoch":"516","state":"active+undersized","last_fresh":"2021-03-15T22:27:11.144187+0000","last_change":"2021-03-15T22:26:46.146468+0000","last_active":"2021-03-15T22:27:11.144187+0000","last_peered":"2021-03-15T22:27:11.144187+0000","last_clean":"2021-03-15T22:26:44.402940+0000","last_became_active":"2021-03-15T22:26:46.146468+0000","last_became_peered":"2021-03-15T22:26:46.146468+0000","last_unstale":"2021-03-15T22:27:11.144187+0000","last_undegraded":"2021-03-15T22:27:11.144187+0000","last_fullsized":"2021-03-15T22:26:46.139115+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:26:12.608082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"481","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.251804+0000","last_change":"2021-03-15T22:25:16.598073+0000","last_active":"2021-03-15T22:27:11.251804+0000","last_peered":"2021-03-15T22:27:11.251804+0000","last_clean":"2021-03-15T22:27:11.251804+0000","last_became_active":"2021-03-15T22:17:53.980388+0000","last_became_peered":"2021-03-15T22:17:53.980388+0000","last_unstale":"2021-03-15T22:27:11.251804+0000","last_undegraded":"2021-03-15T22:27:11.251804+0000","last_fullsized":"2021-03-15T22:27:11.251804+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.597998+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.597998+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"65'13","reported_seq":"591","reported_epoch":"515","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:10.136564+0000","last_change":"2021-03-15T22:26:46.149031+0000","last_active":"2021-03-15T22:27:10.136564+0000","last_peered":"2021-03-15T22:27:10.136564+0000","last_clean":"2021-03-15T22:26:44.341489+0000","last_became_active":"2021-03-15T22:26:46.149031+0000","last_became_peered":"2021-03-15T22:26:46.149031+0000","last_unstale":"2021-03-15T22:27:10.136564+0000","last_undegraded":"2021-03-15T22:26:46.138268+0000","last_fullsized":"2021-03-15T22:26:46.138152+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:24:10.979322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:10.979322+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"195","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.247477+0000","last_change":"2021-03-15T22:26:41.736543+0000","last_active":"2021-03-15T22:27:11.247477+0000","last_peered":"2021-03-15T22:27:11.247477+0000","last_clean":"2021-03-15T22:27:11.247477+0000","last_became_active":"2021-03-15T22:25:33.142493+0000","last_became_peered":"2021-03-15T22:25:33.142493+0000","last_unstale":"2021-03-15T22:27:11.247477+0000","last_undegraded":"2021-03-15T22:27:11.247477+0000","last_fullsized":"2021-03-15T22:27:11.247477+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:41.736469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:26:41.736469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"481","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.247951+0000","last_change":"2021-03-15T22:25:05.682093+0000","last_active":"2021-03-15T22:27:11.247951+0000","last_peered":"2021-03-15T22:27:11.247951+0000","last_clean":"2021-03-15T22:27:11.247951+0000","last_became_active":"2021-03-15T22:17:53.976526+0000","last_became_peered":"2021-03-15T22:17:53.976526+0000","last_unstale":"2021-03-15T22:27:11.247951+0000","last_undegraded":"2021-03-15T22:27:11.247951+0000","last_fullsized":"2021-03-15T22:27:11.247951+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.682031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.682031+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"121'9","reported_seq":"556","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.770862+0000","last_change":"2021-03-15T22:27:10.770829+0000","last_active":"2021-03-15T22:27:10.770862+0000","last_peered":"2021-03-15T22:27:10.770862+0000","last_clean":"2021-03-15T22:27:10.770862+0000","last_became_active":"2021-03-15T22:22:56.055083+0000","last_became_peered":"2021-03-15T22:22:56.055083+0000","last_unstale":"2021-03-15T22:27:10.770862+0000","last_undegraded":"2021-03-15T22:27:10.770862+0000","last_fullsized":"2021-03-15T22:27:10.770862+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"121'9","last_scrub_stamp":"2021-03-15T22:27:10.770751+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:27:10.770751+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":8,"num_write":7,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"483","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.137517+0000","last_change":"2021-03-15T22:27:09.810583+0000","last_active":"2021-03-15T22:27:10.137517+0000","last_peered":"2021-03-15T22:27:10.137517+0000","last_clean":"2021-03-15T22:27:10.137517+0000","last_became_active":"2021-03-15T22:18:00.286837+0000","last_became_peered":"2021-03-15T22:18:00.286837+0000","last_unstale":"2021-03-15T22:27:10.137517+0000","last_undegraded":"2021-03-15T22:27:10.137517+0000","last_fullsized":"2021-03-15T22:27:10.137517+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:09.810514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:27:09.810514+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"231","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.136845+0000","last_change":"2021-03-15T22:27:08.824047+0000","last_active":"2021-03-15T22:27:10.136845+0000","last_peered":"2021-03-15T22:27:10.136845+0000","last_clean":"2021-03-15T22:27:10.136845+0000","last_became_active":"2021-03-15T22:24:55.090536+0000","last_became_peered":"2021-03-15T22:24:55.090536+0000","last_unstale":"2021-03-15T22:27:10.136845+0000","last_undegraded":"2021-03-15T22:27:10.136845+0000","last_fullsized":"2021-03-15T22:27:10.136845+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:27:08.823978+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:27:08.823978+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"65'15","reported_seq":"73","reported_epoch":"516","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:11.145881+0000","last_change":"2021-03-15T22:26:46.149202+0000","last_active":"2021-03-15T22:27:11.145881+0000","last_peered":"2021-03-15T22:27:11.145881+0000","last_clean":"2021-03-15T22:18:02.295877+0000","last_became_active":"2021-03-15T22:26:46.149202+0000","last_became_peered":"2021-03-15T22:26:46.149202+0000","last_unstale":"2021-03-15T22:27:11.145881+0000","last_undegraded":"2021-03-15T22:26:46.140869+0000","last_fullsized":"2021-03-15T22:26:46.140757+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:26:09.660549+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.660549+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"311","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.657225+0000","last_change":"2021-03-15T22:26:45.403832+0000","last_active":"2021-03-15T22:27:10.657225+0000","last_peered":"2021-03-15T22:27:10.657225+0000","last_clean":"2021-03-15T22:27:10.657225+0000","last_became_active":"2021-03-15T22:25:14.117558+0000","last_became_peered":"2021-03-15T22:25:14.117558+0000","last_unstale":"2021-03-15T22:27:10.657225+0000","last_undegraded":"2021-03-15T22:27:10.657225+0000","last_fullsized":"2021-03-15T22:27:10.657225+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:45.403769+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:45.403769+0000","last_clean_scrub_stamp":"2021-03-15T22:26:45.403769+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"493","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.136585+0000","last_change":"2021-03-15T22:26:46.144450+0000","last_active":"2021-03-15T22:27:10.136585+0000","last_peered":"2021-03-15T22:27:10.136585+0000","last_clean":"2021-03-15T22:26:44.341675+0000","last_became_active":"2021-03-15T22:26:46.144450+0000","last_became_peered":"2021-03-15T22:26:46.144450+0000","last_unstale":"2021-03-15T22:27:10.136585+0000","last_undegraded":"2021-03-15T22:27:10.136585+0000","last_fullsized":"2021-03-15T22:26:46.140114+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.028422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.028422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"65'16","reported_seq":"133","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.656868+0000","last_change":"2021-03-15T22:26:44.417763+0000","last_active":"2021-03-15T22:27:10.656868+0000","last_peered":"2021-03-15T22:27:10.656868+0000","last_clean":"2021-03-15T22:27:10.656868+0000","last_became_active":"2021-03-15T22:25:33.134781+0000","last_became_peered":"2021-03-15T22:25:33.134781+0000","last_unstale":"2021-03-15T22:27:10.656868+0000","last_undegraded":"2021-03-15T22:27:10.656868+0000","last_fullsized":"2021-03-15T22:27:10.656868+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:26:44.417608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:44.417608+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":36,"num_read_kb":19,"num_write":18,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"484","reported_epoch":"517","state":"active+undersized","last_fresh":"2021-03-15T22:27:12.155060+0000","last_change":"2021-03-15T22:26:46.153151+0000","last_active":"2021-03-15T22:27:12.155060+0000","last_peered":"2021-03-15T22:27:12.155060+0000","last_clean":"2021-03-15T22:26:44.404322+0000","last_became_active":"2021-03-15T22:26:46.153151+0000","last_became_peered":"2021-03-15T22:26:46.153151+0000","last_unstale":"2021-03-15T22:27:12.155060+0000","last_undegraded":"2021-03-15T22:27:12.155060+0000","last_fullsized":"2021-03-15T22:26:46.141634+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:15.994794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:15.994794+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"508","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.144076+0000","last_change":"2021-03-15T22:26:07.656468+0000","last_active":"2021-03-15T22:27:11.144076+0000","last_peered":"2021-03-15T22:27:11.144076+0000","last_clean":"2021-03-15T22:27:11.144076+0000","last_became_active":"2021-03-15T22:22:56.062833+0000","last_became_peered":"2021-03-15T22:22:56.062833+0000","last_unstale":"2021-03-15T22:27:11.144076+0000","last_undegraded":"2021-03-15T22:27:11.144076+0000","last_fullsized":"2021-03-15T22:27:11.144076+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:07.656378+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.656378+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"87'27","reported_seq":"300","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.153906+0000","last_change":"2021-03-15T22:25:33.752847+0000","last_active":"2021-03-15T22:27:12.153906+0000","last_peered":"2021-03-15T22:27:12.153906+0000","last_clean":"2021-03-15T22:27:12.153906+0000","last_became_active":"2021-03-15T22:25:33.140078+0000","last_became_peered":"2021-03-15T22:25:33.140078+0000","last_unstale":"2021-03-15T22:27:12.153906+0000","last_undegraded":"2021-03-15T22:27:12.153906+0000","last_fullsized":"2021-03-15T22:27:12.153906+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:25:08.192165+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:25:08.192165+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":1868,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"484","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.146019+0000","last_change":"2021-03-15T22:26:02.720851+0000","last_active":"2021-03-15T22:27:11.146019+0000","last_peered":"2021-03-15T22:27:11.146019+0000","last_clean":"2021-03-15T22:27:11.146019+0000","last_became_active":"2021-03-15T22:18:00.014099+0000","last_became_peered":"2021-03-15T22:18:00.014099+0000","last_unstale":"2021-03-15T22:27:11.146019+0000","last_undegraded":"2021-03-15T22:27:11.146019+0000","last_fullsized":"2021-03-15T22:27:11.146019+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:02.720791+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:02.720791+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"114","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.655485+0000","last_change":"2021-03-15T22:26:46.144765+0000","last_active":"2021-03-15T22:27:10.655485+0000","last_peered":"2021-03-15T22:27:10.655485+0000","last_clean":"2021-03-15T22:26:44.342285+0000","last_became_active":"2021-03-15T22:26:46.144765+0000","last_became_peered":"2021-03-15T22:26:46.144765+0000","last_unstale":"2021-03-15T22:27:10.655485+0000","last_undegraded":"2021-03-15T22:27:10.655485+0000","last_fullsized":"2021-03-15T22:26:46.141954+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:29.506406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.506406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"65'17","reported_seq":"514","reported_epoch":"514","state":"active+clean","last_fresh":"2021-03-15T22:27:09.256931+0000","last_change":"2021-03-15T22:26:09.832358+0000","last_active":"2021-03-15T22:27:09.256931+0000","last_peered":"2021-03-15T22:27:09.256931+0000","last_clean":"2021-03-15T22:27:09.256931+0000","last_became_active":"2021-03-15T22:17:55.995406+0000","last_became_peered":"2021-03-15T22:17:55.995406+0000","last_unstale":"2021-03-15T22:27:09.256931+0000","last_undegraded":"2021-03-15T22:27:09.256931+0000","last_fullsized":"2021-03-15T22:27:09.256931+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:26:09.832278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.832278+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":19,"num_write":18,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"484","reported_epoch":"517","state":"active+undersized","last_fresh":"2021-03-15T22:27:12.152525+0000","last_change":"2021-03-15T22:26:46.146944+0000","last_active":"2021-03-15T22:27:12.152525+0000","last_peered":"2021-03-15T22:27:12.152525+0000","last_clean":"2021-03-15T22:26:44.403038+0000","last_became_active":"2021-03-15T22:26:46.146944+0000","last_became_peered":"2021-03-15T22:26:46.146944+0000","last_unstale":"2021-03-15T22:27:12.152525+0000","last_undegraded":"2021-03-15T22:27:12.152525+0000","last_fullsized":"2021-03-15T22:26:46.139288+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:09.177521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.177521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"488","reported_epoch":"514","state":"active+clean","last_fresh":"2021-03-15T22:27:09.256766+0000","last_change":"2021-03-15T22:26:44.107287+0000","last_active":"2021-03-15T22:27:09.256766+0000","last_peered":"2021-03-15T22:27:09.256766+0000","last_clean":"2021-03-15T22:27:09.256766+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:27:09.256766+0000","last_undegraded":"2021-03-15T22:27:09.256766+0000","last_fullsized":"2021-03-15T22:27:09.256766+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:44.107208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:26:44.107208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"318","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.250893+0000","last_change":"2021-03-15T22:26:06.764056+0000","last_active":"2021-03-15T22:27:11.250893+0000","last_peered":"2021-03-15T22:27:11.250893+0000","last_clean":"2021-03-15T22:27:11.250893+0000","last_became_active":"2021-03-15T22:25:33.146680+0000","last_became_peered":"2021-03-15T22:25:33.146680+0000","last_unstale":"2021-03-15T22:27:11.250893+0000","last_undegraded":"2021-03-15T22:27:11.250893+0000","last_fullsized":"2021-03-15T22:27:11.250893+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:06.763986+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:26:06.763986+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.10","version":"65'12","reported_seq":"205","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.655965+0000","last_change":"2021-03-15T22:25:15.835780+0000","last_active":"2021-03-15T22:27:10.655965+0000","last_peered":"2021-03-15T22:27:10.655965+0000","last_clean":"2021-03-15T22:27:10.655965+0000","last_became_active":"2021-03-15T22:24:23.142746+0000","last_became_peered":"2021-03-15T22:24:23.142746+0000","last_unstale":"2021-03-15T22:27:10.655965+0000","last_undegraded":"2021-03-15T22:27:10.655965+0000","last_fullsized":"2021-03-15T22:27:10.655965+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":348,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:25:15.835675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.835675+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":11,"num_write":12,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":3756,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"114","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.656621+0000","last_change":"2021-03-15T22:26:46.151876+0000","last_active":"2021-03-15T22:27:10.656621+0000","last_peered":"2021-03-15T22:27:10.656621+0000","last_clean":"2021-03-15T22:26:44.343296+0000","last_became_active":"2021-03-15T22:26:46.151876+0000","last_became_peered":"2021-03-15T22:26:46.151876+0000","last_unstale":"2021-03-15T22:27:10.656621+0000","last_undegraded":"2021-03-15T22:27:10.656621+0000","last_fullsized":"2021-03-15T22:26:46.146600+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:42.525975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:40.670674+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.525975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":["3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"475","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.153096+0000","last_change":"2021-03-15T22:26:12.013804+0000","last_active":"2021-03-15T22:27:12.153096+0000","last_peered":"2021-03-15T22:27:12.153096+0000","last_clean":"2021-03-15T22:27:12.153096+0000","last_became_active":"2021-03-15T22:23:52.262741+0000","last_became_peered":"2021-03-15T22:23:52.262741+0000","last_unstale":"2021-03-15T22:27:12.153096+0000","last_undegraded":"2021-03-15T22:27:12.153096+0000","last_fullsized":"2021-03-15T22:27:12.153096+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:12.013718+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:26:12.013718+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"481","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.251401+0000","last_change":"2021-03-15T22:26:02.840036+0000","last_active":"2021-03-15T22:27:11.251401+0000","last_peered":"2021-03-15T22:27:11.251401+0000","last_clean":"2021-03-15T22:27:11.251401+0000","last_became_active":"2021-03-15T22:17:53.975049+0000","last_became_peered":"2021-03-15T22:17:53.975049+0000","last_unstale":"2021-03-15T22:27:11.251401+0000","last_undegraded":"2021-03-15T22:27:11.251401+0000","last_fullsized":"2021-03-15T22:27:11.251401+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:02.839942+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:02.839942+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"90'22","reported_seq":"293","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.656315+0000","last_change":"2021-03-15T22:25:16.834493+0000","last_active":"2021-03-15T22:27:10.656315+0000","last_peered":"2021-03-15T22:27:10.656315+0000","last_clean":"2021-03-15T22:27:10.656315+0000","last_became_active":"2021-03-15T22:22:56.056175+0000","last_became_peered":"2021-03-15T22:22:56.056175+0000","last_unstale":"2021-03-15T22:27:10.656315+0000","last_undegraded":"2021-03-15T22:27:10.656315+0000","last_fullsized":"2021-03-15T22:27:10.656315+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:25:16.834409+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.834409+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":16,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"180","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.247614+0000","last_change":"2021-03-15T22:26:46.756621+0000","last_active":"2021-03-15T22:27:11.247614+0000","last_peered":"2021-03-15T22:27:11.247614+0000","last_clean":"2021-03-15T22:27:11.247614+0000","last_became_active":"2021-03-15T22:24:31.105380+0000","last_became_peered":"2021-03-15T22:24:31.105380+0000","last_unstale":"2021-03-15T22:27:11.247614+0000","last_undegraded":"2021-03-15T22:27:11.247614+0000","last_fullsized":"2021-03-15T22:27:11.247614+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:26:46.756523+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"298","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.656582+0000","last_change":"2021-03-15T22:26:52.558959+0000","last_active":"2021-03-15T22:27:10.656582+0000","last_peered":"2021-03-15T22:27:10.656582+0000","last_clean":"2021-03-15T22:27:10.656582+0000","last_became_active":"2021-03-15T22:25:14.114508+0000","last_became_peered":"2021-03-15T22:25:14.114508+0000","last_unstale":"2021-03-15T22:27:10.656582+0000","last_undegraded":"2021-03-15T22:27:10.656582+0000","last_fullsized":"2021-03-15T22:27:10.656582+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:26:52.558905+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,5],"acting":[3,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"497","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.155072+0000","last_change":"2021-03-15T22:26:45.153513+0000","last_active":"2021-03-15T22:27:12.155072+0000","last_peered":"2021-03-15T22:27:12.155072+0000","last_clean":"2021-03-15T22:27:12.155072+0000","last_became_active":"2021-03-15T22:23:52.266011+0000","last_became_peered":"2021-03-15T22:23:52.266011+0000","last_unstale":"2021-03-15T22:27:12.155072+0000","last_undegraded":"2021-03-15T22:27:12.155072+0000","last_fullsized":"2021-03-15T22:27:12.155072+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:45.153417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:26:45.153417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"65'15","reported_seq":"179","reported_epoch":"516","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:11.145545+0000","last_change":"2021-03-15T22:26:46.149349+0000","last_active":"2021-03-15T22:27:11.145545+0000","last_peered":"2021-03-15T22:27:11.145545+0000","last_clean":"2021-03-15T22:26:44.404259+0000","last_became_active":"2021-03-15T22:26:46.149349+0000","last_became_peered":"2021-03-15T22:26:46.149349+0000","last_unstale":"2021-03-15T22:27:11.145545+0000","last_undegraded":"2021-03-15T22:26:46.140554+0000","last_fullsized":"2021-03-15T22:26:46.140451+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:26:42.456927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:42.456927+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"119","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.656143+0000","last_change":"2021-03-15T22:26:46.156542+0000","last_active":"2021-03-15T22:27:10.656143+0000","last_peered":"2021-03-15T22:27:10.656143+0000","last_clean":"2021-03-15T22:26:44.343330+0000","last_became_active":"2021-03-15T22:26:46.156542+0000","last_became_peered":"2021-03-15T22:26:46.156542+0000","last_unstale":"2021-03-15T22:27:10.656143+0000","last_undegraded":"2021-03-15T22:27:10.656143+0000","last_fullsized":"2021-03-15T22:26:46.144431+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:26:10.680621+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"504","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.145346+0000","last_change":"2021-03-15T22:26:50.440802+0000","last_active":"2021-03-15T22:27:11.145346+0000","last_peered":"2021-03-15T22:27:11.145346+0000","last_clean":"2021-03-15T22:27:11.145346+0000","last_became_active":"2021-03-15T22:25:33.129553+0000","last_became_peered":"2021-03-15T22:25:33.129553+0000","last_unstale":"2021-03-15T22:27:11.145346+0000","last_undegraded":"2021-03-15T22:27:11.145346+0000","last_fullsized":"2021-03-15T22:27:11.145346+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:50.440728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_clean_scrub_stamp":"2021-03-15T22:26:50.440728+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"37","reported_epoch":"517","state":"active+undersized","last_fresh":"2021-03-15T22:27:12.154851+0000","last_change":"2021-03-15T22:26:46.156620+0000","last_active":"2021-03-15T22:27:12.154851+0000","last_peered":"2021-03-15T22:27:12.154851+0000","last_clean":"2021-03-15T22:24:01.075356+0000","last_became_active":"2021-03-15T22:26:46.156620+0000","last_became_peered":"2021-03-15T22:26:46.156620+0000","last_unstale":"2021-03-15T22:27:12.154851+0000","last_undegraded":"2021-03-15T22:27:12.154851+0000","last_fullsized":"2021-03-15T22:26:46.141354+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:40.425545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:40.425545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"65'15","reported_seq":"514","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.136837+0000","last_change":"2021-03-15T22:26:41.924354+0000","last_active":"2021-03-15T22:27:10.136837+0000","last_peered":"2021-03-15T22:27:10.136837+0000","last_clean":"2021-03-15T22:27:10.136837+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:27:10.136837+0000","last_undegraded":"2021-03-15T22:27:10.136837+0000","last_fullsized":"2021-03-15T22:27:10.136837+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:26:41.924276+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:41.924276+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"311","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.153657+0000","last_change":"2021-03-15T22:25:33.144478+0000","last_active":"2021-03-15T22:27:12.153657+0000","last_peered":"2021-03-15T22:27:12.153657+0000","last_clean":"2021-03-15T22:27:12.153657+0000","last_became_active":"2021-03-15T22:25:33.143832+0000","last_became_peered":"2021-03-15T22:25:33.143832+0000","last_unstale":"2021-03-15T22:27:12.153657+0000","last_undegraded":"2021-03-15T22:27:12.153657+0000","last_fullsized":"2021-03-15T22:27:12.153657+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.147414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.147414+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"356'3","reported_seq":"603","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:12.101276+0000","last_change":"2021-03-15T22:26:38.736317+0000","last_active":"2021-03-15T22:27:12.101276+0000","last_peered":"2021-03-15T22:27:12.101276+0000","last_clean":"2021-03-15T22:27:12.101276+0000","last_became_active":"2021-03-15T22:24:31.093479+0000","last_became_peered":"2021-03-15T22:24:31.093479+0000","last_unstale":"2021-03-15T22:27:12.101276+0000","last_undegraded":"2021-03-15T22:27:12.101276+0000","last_fullsized":"2021-03-15T22:27:12.101276+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'3","last_scrub_stamp":"2021-03-15T22:26:38.736211+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:26:38.736211+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"485","reported_epoch":"517","state":"active+undersized","last_fresh":"2021-03-15T22:27:12.152948+0000","last_change":"2021-03-15T22:26:46.155875+0000","last_active":"2021-03-15T22:27:12.152948+0000","last_peered":"2021-03-15T22:27:12.152948+0000","last_clean":"2021-03-15T22:26:44.403391+0000","last_became_active":"2021-03-15T22:26:46.155875+0000","last_became_peered":"2021-03-15T22:26:46.155875+0000","last_unstale":"2021-03-15T22:27:12.152948+0000","last_undegraded":"2021-03-15T22:27:12.152948+0000","last_fullsized":"2021-03-15T22:26:46.139933+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:01.148105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:01.148105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"65'14","reported_seq":"292","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.656796+0000","last_change":"2021-03-15T22:24:27.166757+0000","last_active":"2021-03-15T22:27:10.656796+0000","last_peered":"2021-03-15T22:27:10.656796+0000","last_clean":"2021-03-15T22:27:10.656796+0000","last_became_active":"2021-03-15T22:22:56.064081+0000","last_became_peered":"2021-03-15T22:22:56.064081+0000","last_unstale":"2021-03-15T22:27:10.656796+0000","last_undegraded":"2021-03-15T22:27:10.656796+0000","last_fullsized":"2021-03-15T22:27:10.656796+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:24:27.166678+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.166678+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":26,"num_read_kb":13,"num_write":14,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"482","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.137831+0000","last_change":"2021-03-15T22:26:46.155002+0000","last_active":"2021-03-15T22:27:10.137831+0000","last_peered":"2021-03-15T22:27:10.137831+0000","last_clean":"2021-03-15T22:26:44.342180+0000","last_became_active":"2021-03-15T22:26:46.155002+0000","last_became_peered":"2021-03-15T22:26:46.155002+0000","last_unstale":"2021-03-15T22:27:10.137831+0000","last_undegraded":"2021-03-15T22:27:10.137831+0000","last_fullsized":"2021-03-15T22:26:46.138914+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:07.992815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.992815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"491'8","reported_seq":"137","reported_epoch":"515","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:10.656636+0000","last_change":"2021-03-15T22:26:46.154890+0000","last_active":"2021-03-15T22:27:10.656636+0000","last_peered":"2021-03-15T22:27:10.656636+0000","last_clean":"2021-03-15T22:26:44.343327+0000","last_became_active":"2021-03-15T22:26:46.154890+0000","last_became_peered":"2021-03-15T22:26:46.154890+0000","last_unstale":"2021-03-15T22:27:10.656636+0000","last_undegraded":"2021-03-15T22:26:46.143073+0000","last_fullsized":"2021-03-15T22:26:46.142951+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"361'6","last_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:24:36.098511+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"322","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.152649+0000","last_change":"2021-03-15T22:26:17.012763+0000","last_active":"2021-03-15T22:27:12.152649+0000","last_peered":"2021-03-15T22:27:12.152649+0000","last_clean":"2021-03-15T22:27:12.152649+0000","last_became_active":"2021-03-15T22:25:33.141466+0000","last_became_peered":"2021-03-15T22:25:33.141466+0000","last_unstale":"2021-03-15T22:27:12.152649+0000","last_undegraded":"2021-03-15T22:27:12.152649+0000","last_fullsized":"2021-03-15T22:27:12.152649+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:17.012652+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:26:17.012652+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,1],"acting":[7,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.15","version":"497'21","reported_seq":"158","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.251493+0000","last_change":"2021-03-15T22:26:52.609920+0000","last_active":"2021-03-15T22:27:11.251493+0000","last_peered":"2021-03-15T22:27:11.251493+0000","last_clean":"2021-03-15T22:27:11.251493+0000","last_became_active":"2021-03-15T22:25:14.114060+0000","last_became_peered":"2021-03-15T22:25:14.114060+0000","last_unstale":"2021-03-15T22:27:11.251493+0000","last_undegraded":"2021-03-15T22:27:11.251493+0000","last_fullsized":"2021-03-15T22:27:11.251493+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"497'21","last_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:26:52.609858+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":14,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"34","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.137257+0000","last_change":"2021-03-15T22:26:46.147347+0000","last_active":"2021-03-15T22:27:10.137257+0000","last_peered":"2021-03-15T22:27:10.137257+0000","last_clean":"2021-03-15T22:17:58.999017+0000","last_became_active":"2021-03-15T22:26:46.147347+0000","last_became_peered":"2021-03-15T22:26:46.147347+0000","last_unstale":"2021-03-15T22:27:10.137257+0000","last_undegraded":"2021-03-15T22:27:10.137257+0000","last_fullsized":"2021-03-15T22:26:46.138696+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.614288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.614288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"481","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.155885+0000","last_change":"2021-03-15T22:25:10.130486+0000","last_active":"2021-03-15T22:27:12.155885+0000","last_peered":"2021-03-15T22:27:12.155885+0000","last_clean":"2021-03-15T22:27:12.155885+0000","last_became_active":"2021-03-15T22:23:52.263426+0000","last_became_peered":"2021-03-15T22:23:52.263426+0000","last_unstale":"2021-03-15T22:27:12.155885+0000","last_undegraded":"2021-03-15T22:27:12.155885+0000","last_fullsized":"2021-03-15T22:27:12.155885+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:10.130422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.130422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"65'16","reported_seq":"138","reported_epoch":"515","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:10.657042+0000","last_change":"2021-03-15T22:26:46.155290+0000","last_active":"2021-03-15T22:27:10.657042+0000","last_peered":"2021-03-15T22:27:10.657042+0000","last_clean":"2021-03-15T22:26:44.343740+0000","last_became_active":"2021-03-15T22:26:46.155290+0000","last_became_peered":"2021-03-15T22:26:46.155290+0000","last_unstale":"2021-03-15T22:27:10.657042+0000","last_undegraded":"2021-03-15T22:26:46.144722+0000","last_fullsized":"2021-03-15T22:26:46.144605+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":419,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:26.478799+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.478799+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":38,"num_read_kb":18,"num_write":17,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3618,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":["3","7"],"object_location_counts":[{"shards":"3,7","objects":11}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"478","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.154991+0000","last_change":"2021-03-15T22:26:07.054564+0000","last_active":"2021-03-15T22:27:12.154991+0000","last_peered":"2021-03-15T22:27:12.154991+0000","last_clean":"2021-03-15T22:27:12.154991+0000","last_became_active":"2021-03-15T22:23:52.266133+0000","last_became_peered":"2021-03-15T22:23:52.266133+0000","last_unstale":"2021-03-15T22:27:12.154991+0000","last_undegraded":"2021-03-15T22:27:12.154991+0000","last_fullsized":"2021-03-15T22:27:12.154991+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:07.054500+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.054500+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"528","reported_epoch":"514","state":"active+clean","last_fresh":"2021-03-15T22:27:09.253720+0000","last_change":"2021-03-15T22:26:10.840930+0000","last_active":"2021-03-15T22:27:09.253720+0000","last_peered":"2021-03-15T22:27:09.253720+0000","last_clean":"2021-03-15T22:27:09.253720+0000","last_became_active":"2021-03-15T22:25:33.150147+0000","last_became_peered":"2021-03-15T22:25:33.150147+0000","last_unstale":"2021-03-15T22:27:09.253720+0000","last_undegraded":"2021-03-15T22:27:09.253720+0000","last_fullsized":"2021-03-15T22:27:09.253720+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:10.840841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:10.840841+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"398'11","reported_seq":"118","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.656844+0000","last_change":"2021-03-15T22:25:33.864526+0000","last_active":"2021-03-15T22:27:10.656844+0000","last_peered":"2021-03-15T22:27:10.656844+0000","last_clean":"2021-03-15T22:27:10.656844+0000","last_became_active":"2021-03-15T22:25:33.152373+0000","last_became_peered":"2021-03-15T22:25:33.152373+0000","last_unstale":"2021-03-15T22:27:10.656844+0000","last_undegraded":"2021-03-15T22:27:10.656844+0000","last_fullsized":"2021-03-15T22:27:10.656844+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"398'11","last_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:25:13.181225+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":11,"num_read_kb":6,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":474,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"517'580","reported_seq":"2059","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.158334+0000","last_change":"2021-03-15T22:27:12.158334+0000","last_active":"2021-03-15T22:27:12.158334+0000","last_peered":"2021-03-15T22:27:12.158334+0000","last_clean":"2021-03-15T22:27:12.158334+0000","last_became_active":"2021-03-15T22:25:17.113174+0000","last_became_peered":"2021-03-15T22:25:17.113174+0000","last_unstale":"2021-03-15T22:27:12.158334+0000","last_undegraded":"2021-03-15T22:27:12.158334+0000","last_fullsized":"2021-03-15T22:27:12.158334+0000","mapping_epoch":401,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":402,"parent":"0.0","parent_split_bits":0,"last_scrub":"494'538","last_scrub_stamp":"2021-03-15T22:26:49.448172+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:26:49.448172+0000","log_size":580,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26870741,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":316,"num_read_kb":370037,"num_write":184,"num_write_kb":55848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":21669077,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"102","length":"1"},{"start":"13e","length":"1"},{"start":"15b","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"2"},{"start":"185","length":"1"},{"start":"187","length":"1"},{"start":"18b","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"490","reported_epoch":"516","state":"active+undersized","last_fresh":"2021-03-15T22:27:11.248529+0000","last_change":"2021-03-15T22:26:46.154521+0000","last_active":"2021-03-15T22:27:11.248529+0000","last_peered":"2021-03-15T22:27:11.248529+0000","last_clean":"2021-03-15T22:26:44.405122+0000","last_became_active":"2021-03-15T22:26:46.154521+0000","last_became_peered":"2021-03-15T22:26:46.154521+0000","last_unstale":"2021-03-15T22:27:11.248529+0000","last_undegraded":"2021-03-15T22:27:11.248529+0000","last_fullsized":"2021-03-15T22:26:46.143824+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.689591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.689591+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":["5","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"502","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.154443+0000","last_change":"2021-03-15T22:25:48.008983+0000","last_active":"2021-03-15T22:27:12.154443+0000","last_peered":"2021-03-15T22:27:12.154443+0000","last_clean":"2021-03-15T22:27:12.154443+0000","last_became_active":"2021-03-15T22:24:31.097688+0000","last_became_peered":"2021-03-15T22:24:31.097688+0000","last_unstale":"2021-03-15T22:27:12.154443+0000","last_undegraded":"2021-03-15T22:27:12.154443+0000","last_fullsized":"2021-03-15T22:27:12.154443+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:25:48.008911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:48.008911+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"65'18","reported_seq":"203","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.247690+0000","last_change":"2021-03-15T22:25:32.799965+0000","last_active":"2021-03-15T22:27:11.247690+0000","last_peered":"2021-03-15T22:27:11.247690+0000","last_clean":"2021-03-15T22:27:11.247690+0000","last_became_active":"2021-03-15T22:24:23.137531+0000","last_became_peered":"2021-03-15T22:24:23.137531+0000","last_unstale":"2021-03-15T22:27:11.247690+0000","last_undegraded":"2021-03-15T22:27:11.247690+0000","last_fullsized":"2021-03-15T22:27:11.247690+0000","mapping_epoch":347,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":349,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'18","last_scrub_stamp":"2021-03-15T22:25:32.799882+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:32.799882+0000","log_size":18,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":578,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"317'7","reported_seq":"611","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.156096+0000","last_change":"2021-03-15T22:27:11.357502+0000","last_active":"2021-03-15T22:27:12.156096+0000","last_peered":"2021-03-15T22:27:12.156096+0000","last_clean":"2021-03-15T22:27:12.156096+0000","last_became_active":"2021-03-15T22:23:52.262878+0000","last_became_peered":"2021-03-15T22:23:52.262878+0000","last_unstale":"2021-03-15T22:27:12.156096+0000","last_undegraded":"2021-03-15T22:27:12.156096+0000","last_fullsized":"2021-03-15T22:27:12.156096+0000","mapping_epoch":316,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:27:11.357399+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:27:11.357399+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"509'758","reported_seq":"2095","reported_epoch":"515","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:10.656597+0000","last_change":"2021-03-15T22:26:46.156957+0000","last_active":"2021-03-15T22:27:10.656597+0000","last_peered":"2021-03-15T22:27:10.656597+0000","last_clean":"2021-03-15T22:26:44.344066+0000","last_became_active":"2021-03-15T22:26:46.156957+0000","last_became_peered":"2021-03-15T22:26:46.156957+0000","last_unstale":"2021-03-15T22:27:10.656597+0000","last_undegraded":"2021-03-15T22:26:46.144128+0000","last_fullsized":"2021-03-15T22:26:46.143999+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":269,"parent":"0.0","parent_split_bits":0,"last_scrub":"341'455","last_scrub_stamp":"2021-03-15T22:24:28.184612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:24:28.184612+0000","log_size":758,"ondisk_log_size":758,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":61225583,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":25,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":379,"num_read_kb":381415,"num_write":268,"num_write_kb":102152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":32120558,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[{"shards":"3,5","objects":25}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"275","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.656694+0000","last_change":"2021-03-15T22:24:37.122501+0000","last_active":"2021-03-15T22:27:10.656694+0000","last_peered":"2021-03-15T22:27:10.656694+0000","last_clean":"2021-03-15T22:27:10.656694+0000","last_became_active":"2021-03-15T22:22:56.057174+0000","last_became_peered":"2021-03-15T22:22:56.057174+0000","last_unstale":"2021-03-15T22:27:10.656694+0000","last_undegraded":"2021-03-15T22:27:10.656694+0000","last_fullsized":"2021-03-15T22:27:10.656694+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:37.122407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.122407+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"486","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.153809+0000","last_change":"2021-03-15T22:26:03.105518+0000","last_active":"2021-03-15T22:27:12.153809+0000","last_peered":"2021-03-15T22:27:12.153809+0000","last_clean":"2021-03-15T22:27:12.153809+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:27:12.153809+0000","last_undegraded":"2021-03-15T22:27:12.153809+0000","last_fullsized":"2021-03-15T22:27:12.153809+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.105446+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.105446+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"65'15","reported_seq":"563","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.137201+0000","last_change":"2021-03-15T22:25:33.862633+0000","last_active":"2021-03-15T22:27:10.137201+0000","last_peered":"2021-03-15T22:27:10.137201+0000","last_clean":"2021-03-15T22:27:10.137201+0000","last_became_active":"2021-03-15T22:25:33.138262+0000","last_became_peered":"2021-03-15T22:25:33.138262+0000","last_unstale":"2021-03-15T22:27:10.137201+0000","last_undegraded":"2021-03-15T22:27:10.137201+0000","last_fullsized":"2021-03-15T22:27:10.137201+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:25:07.027095+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:07.027095+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":19,"num_write":17,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"371","reported_epoch":"516","state":"active+undersized","last_fresh":"2021-03-15T22:27:11.144178+0000","last_change":"2021-03-15T22:26:46.152622+0000","last_active":"2021-03-15T22:27:11.144178+0000","last_peered":"2021-03-15T22:27:11.144178+0000","last_clean":"2021-03-15T22:24:29.084343+0000","last_became_active":"2021-03-15T22:26:46.152622+0000","last_became_peered":"2021-03-15T22:26:46.152622+0000","last_unstale":"2021-03-15T22:27:11.144178+0000","last_undegraded":"2021-03-15T22:27:11.144178+0000","last_fullsized":"2021-03-15T22:26:46.139165+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.853267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.853267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.9","version":"517'185","reported_seq":"1527","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.156168+0000","last_change":"2021-03-15T22:27:12.156449+0000","last_active":"2021-03-15T22:27:12.156168+0000","last_peered":"2021-03-15T22:27:12.156168+0000","last_clean":"2021-03-15T22:27:12.156168+0000","last_became_active":"2021-03-15T22:24:34.087074+0000","last_became_peered":"2021-03-15T22:24:34.087074+0000","last_unstale":"2021-03-15T22:27:12.156168+0000","last_undegraded":"2021-03-15T22:27:12.156168+0000","last_fullsized":"2021-03-15T22:27:12.156168+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'183","last_scrub_stamp":"2021-03-15T22:27:09.383406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:27:09.383406+0000","log_size":185,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10481419,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":55,"num_read_kb":37879,"num_write":61,"num_write_kb":20058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":4470794,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"102","length":"1"},{"start":"13e","length":"1"},{"start":"15b","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"2"},{"start":"185","length":"1"},{"start":"187","length":"1"},{"start":"18b","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"502","reported_epoch":"517","state":"active+undersized","last_fresh":"2021-03-15T22:27:12.152499+0000","last_change":"2021-03-15T22:26:46.150057+0000","last_active":"2021-03-15T22:27:12.152499+0000","last_peered":"2021-03-15T22:27:12.152499+0000","last_clean":"2021-03-15T22:26:44.403038+0000","last_became_active":"2021-03-15T22:26:46.150057+0000","last_became_peered":"2021-03-15T22:26:46.150057+0000","last_unstale":"2021-03-15T22:27:12.152499+0000","last_undegraded":"2021-03-15T22:27:12.152499+0000","last_fullsized":"2021-03-15T22:26:46.139870+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:13.044467+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:13.044467+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"436","reported_epoch":"516","state":"active+undersized","last_fresh":"2021-03-15T22:27:11.248822+0000","last_change":"2021-03-15T22:26:46.156177+0000","last_active":"2021-03-15T22:27:11.248822+0000","last_peered":"2021-03-15T22:27:11.248822+0000","last_clean":"2021-03-15T22:25:12.103298+0000","last_became_active":"2021-03-15T22:26:46.156177+0000","last_became_peered":"2021-03-15T22:26:46.156177+0000","last_unstale":"2021-03-15T22:27:11.248822+0000","last_undegraded":"2021-03-15T22:27:11.248822+0000","last_fullsized":"2021-03-15T22:26:46.144880+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.575552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.575552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"504","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.404626+0000","last_change":"2021-03-15T22:27:12.404583+0000","last_active":"2021-03-15T22:27:12.404626+0000","last_peered":"2021-03-15T22:27:12.404626+0000","last_clean":"2021-03-15T22:27:12.404626+0000","last_became_active":"2021-03-15T22:25:14.118469+0000","last_became_peered":"2021-03-15T22:25:14.118469+0000","last_unstale":"2021-03-15T22:27:12.404626+0000","last_undegraded":"2021-03-15T22:27:12.404626+0000","last_fullsized":"2021-03-15T22:27:12.404626+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:12.404498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:27:12.404498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"65'23","reported_seq":"565","reported_epoch":"514","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:09.254388+0000","last_change":"2021-03-15T22:26:46.158466+0000","last_active":"2021-03-15T22:27:09.254388+0000","last_peered":"2021-03-15T22:27:09.254388+0000","last_clean":"2021-03-15T22:26:44.404347+0000","last_became_active":"2021-03-15T22:26:46.158466+0000","last_became_peered":"2021-03-15T22:26:46.158466+0000","last_unstale":"2021-03-15T22:27:09.254388+0000","last_undegraded":"2021-03-15T22:26:46.143093+0000","last_fullsized":"2021-03-15T22:26:46.142970+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:26:11.821564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:11.821564+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":44,"num_read_kb":25,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":13}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"507'464","reported_seq":"1728","reported_epoch":"517","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:12.152237+0000","last_change":"2021-03-15T22:26:46.147101+0000","last_active":"2021-03-15T22:27:12.152237+0000","last_peered":"2021-03-15T22:27:12.152237+0000","last_clean":"2021-03-15T22:26:41.176252+0000","last_became_active":"2021-03-15T22:26:46.147101+0000","last_became_peered":"2021-03-15T22:26:46.147101+0000","last_unstale":"2021-03-15T22:27:12.152237+0000","last_undegraded":"2021-03-15T22:26:46.139401+0000","last_fullsized":"2021-03-15T22:26:46.139252+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'328","last_scrub_stamp":"2021-03-15T22:25:37.707812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.707812+0000","log_size":464,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"stat_sum":{"num_bytes":27318702,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":210,"num_read_kb":247374,"num_write":130,"num_write_kb":45158,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":12}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"185","reported_epoch":"517","state":"active+undersized","last_fresh":"2021-03-15T22:27:12.155706+0000","last_change":"2021-03-15T22:26:46.154912+0000","last_active":"2021-03-15T22:27:12.155706+0000","last_peered":"2021-03-15T22:27:12.155706+0000","last_clean":"2021-03-15T22:26:44.403426+0000","last_became_active":"2021-03-15T22:26:46.154912+0000","last_became_peered":"2021-03-15T22:26:46.154912+0000","last_unstale":"2021-03-15T22:27:12.155706+0000","last_undegraded":"2021-03-15T22:27:12.155706+0000","last_fullsized":"2021-03-15T22:26:46.139887+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.117403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.117403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":["7","3"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"455","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.137379+0000","last_change":"2021-03-15T22:26:46.149637+0000","last_active":"2021-03-15T22:27:10.137379+0000","last_peered":"2021-03-15T22:27:10.137379+0000","last_clean":"2021-03-15T22:25:31.105447+0000","last_became_active":"2021-03-15T22:26:46.149637+0000","last_became_peered":"2021-03-15T22:26:46.149637+0000","last_unstale":"2021-03-15T22:27:10.137379+0000","last_undegraded":"2021-03-15T22:27:10.137379+0000","last_fullsized":"2021-03-15T22:26:46.138635+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:27.869570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:27.869570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"499","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.248079+0000","last_change":"2021-03-15T22:25:47.753887+0000","last_active":"2021-03-15T22:27:11.248079+0000","last_peered":"2021-03-15T22:27:11.248079+0000","last_clean":"2021-03-15T22:27:11.248079+0000","last_became_active":"2021-03-15T22:25:33.145548+0000","last_became_peered":"2021-03-15T22:25:33.145548+0000","last_unstale":"2021-03-15T22:27:11.248079+0000","last_undegraded":"2021-03-15T22:27:11.248079+0000","last_fullsized":"2021-03-15T22:27:11.248079+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:47.753814+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:47.753814+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"65'14","reported_seq":"73","reported_epoch":"515","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:10.136704+0000","last_change":"2021-03-15T22:26:46.151549+0000","last_active":"2021-03-15T22:27:10.136704+0000","last_peered":"2021-03-15T22:27:10.136704+0000","last_clean":"2021-03-15T22:18:02.328613+0000","last_became_active":"2021-03-15T22:26:46.151549+0000","last_became_peered":"2021-03-15T22:26:46.151549+0000","last_unstale":"2021-03-15T22:27:10.136704+0000","last_undegraded":"2021-03-15T22:26:46.139335+0000","last_fullsized":"2021-03-15T22:26:46.139265+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'14","last_scrub_stamp":"2021-03-15T22:26:43.510088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:43.510088+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":15,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[{"shards":"3,4","objects":9}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.7","version":"511'553","reported_seq":"2021","reported_epoch":"514","state":"active+clean","last_fresh":"2021-03-15T22:27:09.257173+0000","last_change":"2021-03-15T22:27:07.229042+0000","last_active":"2021-03-15T22:27:09.257173+0000","last_peered":"2021-03-15T22:27:09.257173+0000","last_clean":"2021-03-15T22:27:09.257173+0000","last_became_active":"2021-03-15T22:25:36.118229+0000","last_became_peered":"2021-03-15T22:25:36.118229+0000","last_unstale":"2021-03-15T22:27:09.257173+0000","last_undegraded":"2021-03-15T22:27:09.257173+0000","last_fullsized":"2021-03-15T22:27:09.257173+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"493'509","last_scrub_stamp":"2021-03-15T22:26:48.205847+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:26:48.205847+0000","log_size":553,"ondisk_log_size":553,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32077027,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":177,"num_read_kb":134608,"num_write":197,"num_write_kb":59817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":35579701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,1],"acting":[6,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"102","length":"1"},{"start":"13e","length":"1"},{"start":"15b","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"2"},{"start":"187","length":"1"},{"start":"18b","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"113","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.657011+0000","last_change":"2021-03-15T22:26:04.729228+0000","last_active":"2021-03-15T22:27:10.657011+0000","last_peered":"2021-03-15T22:27:10.657011+0000","last_clean":"2021-03-15T22:27:10.657011+0000","last_became_active":"2021-03-15T22:25:33.152232+0000","last_became_peered":"2021-03-15T22:25:33.152232+0000","last_unstale":"2021-03-15T22:27:10.657011+0000","last_undegraded":"2021-03-15T22:27:10.657011+0000","last_fullsized":"2021-03-15T22:27:10.657011+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:04.729163+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:04.729163+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"483","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.247724+0000","last_change":"2021-03-15T22:25:11.615097+0000","last_active":"2021-03-15T22:27:11.247724+0000","last_peered":"2021-03-15T22:27:11.247724+0000","last_clean":"2021-03-15T22:27:11.247724+0000","last_became_active":"2021-03-15T22:17:50.950737+0000","last_became_peered":"2021-03-15T22:17:50.950737+0000","last_unstale":"2021-03-15T22:27:11.247724+0000","last_undegraded":"2021-03-15T22:27:11.247724+0000","last_fullsized":"2021-03-15T22:27:11.247724+0000","mapping_epoch":53,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":54,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.614975+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.614975+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"42","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.136355+0000","last_change":"2021-03-15T22:26:39.144869+0000","last_active":"2021-03-15T22:27:10.136355+0000","last_peered":"2021-03-15T22:27:10.136355+0000","last_clean":"2021-03-15T22:27:10.136355+0000","last_became_active":"2021-03-15T22:26:39.144536+0000","last_became_peered":"2021-03-15T22:26:39.144536+0000","last_unstale":"2021-03-15T22:27:10.136355+0000","last_undegraded":"2021-03-15T22:27:10.136355+0000","last_fullsized":"2021-03-15T22:27:10.136355+0000","mapping_epoch":483,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":484,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:45.692545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:45.692545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"84'38","reported_seq":"628","reported_epoch":"516","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:11.146119+0000","last_change":"2021-03-15T22:26:46.148535+0000","last_active":"2021-03-15T22:27:11.146119+0000","last_peered":"2021-03-15T22:27:11.146119+0000","last_clean":"2021-03-15T22:26:44.404801+0000","last_became_active":"2021-03-15T22:26:46.148535+0000","last_became_peered":"2021-03-15T22:26:46.148535+0000","last_unstale":"2021-03-15T22:27:11.146119+0000","last_undegraded":"2021-03-15T22:26:46.141260+0000","last_fullsized":"2021-03-15T22:26:46.141143+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'38","last_scrub_stamp":"2021-03-15T22:25:15.810377+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:25:15.810377+0000","log_size":38,"ondisk_log_size":38,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":54,"num_read_kb":31,"num_write":28,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1908,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":["1","3"],"object_location_counts":[{"shards":"1,3","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"182","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.145901+0000","last_change":"2021-03-15T22:27:10.399449+0000","last_active":"2021-03-15T22:27:11.145901+0000","last_peered":"2021-03-15T22:27:11.145901+0000","last_clean":"2021-03-15T22:27:11.145901+0000","last_became_active":"2021-03-15T22:24:31.101668+0000","last_became_peered":"2021-03-15T22:24:31.101668+0000","last_unstale":"2021-03-15T22:27:11.145901+0000","last_undegraded":"2021-03-15T22:27:11.145901+0000","last_fullsized":"2021-03-15T22:27:11.145901+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:10.399386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:27:10.399386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.4","version":"511'527","reported_seq":"1863","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.136792+0000","last_change":"2021-03-15T22:27:07.690286+0000","last_active":"2021-03-15T22:27:10.136792+0000","last_peered":"2021-03-15T22:27:10.136792+0000","last_clean":"2021-03-15T22:27:10.136792+0000","last_became_active":"2021-03-15T22:24:34.087484+0000","last_became_peered":"2021-03-15T22:24:34.087484+0000","last_unstale":"2021-03-15T22:27:10.136792+0000","last_undegraded":"2021-03-15T22:27:10.136792+0000","last_fullsized":"2021-03-15T22:27:10.136792+0000","mapping_epoch":358,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":359,"parent":"0.0","parent_split_bits":0,"last_scrub":"395'329","last_scrub_stamp":"2021-03-15T22:25:14.014776+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:14.014776+0000","log_size":527,"ondisk_log_size":527,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21535640,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":162,"num_read_kb":150087,"num_write":146,"num_write_kb":52135,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":10768640,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"102","length":"1"},{"start":"13e","length":"1"},{"start":"15b","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"2"},{"start":"187","length":"1"},{"start":"18b","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"496","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.155507+0000","last_change":"2021-03-15T22:26:08.042648+0000","last_active":"2021-03-15T22:27:12.155507+0000","last_peered":"2021-03-15T22:27:12.155507+0000","last_clean":"2021-03-15T22:27:12.155507+0000","last_became_active":"2021-03-15T22:23:52.266556+0000","last_became_peered":"2021-03-15T22:23:52.266556+0000","last_unstale":"2021-03-15T22:27:12.155507+0000","last_undegraded":"2021-03-15T22:27:12.155507+0000","last_fullsized":"2021-03-15T22:27:12.155507+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:08.042588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:08.042588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"491","reported_epoch":"516","state":"active+undersized","last_fresh":"2021-03-15T22:27:11.144348+0000","last_change":"2021-03-15T22:26:46.145950+0000","last_active":"2021-03-15T22:27:11.144348+0000","last_peered":"2021-03-15T22:27:11.144348+0000","last_clean":"2021-03-15T22:26:44.403138+0000","last_became_active":"2021-03-15T22:26:46.145950+0000","last_became_peered":"2021-03-15T22:26:46.145950+0000","last_unstale":"2021-03-15T22:27:11.144348+0000","last_undegraded":"2021-03-15T22:27:11.144348+0000","last_fullsized":"2021-03-15T22:26:46.139516+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.929066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.929066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"441","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.655500+0000","last_change":"2021-03-15T22:26:46.145222+0000","last_active":"2021-03-15T22:27:10.655500+0000","last_peered":"2021-03-15T22:27:10.655500+0000","last_clean":"2021-03-15T22:25:31.105080+0000","last_became_active":"2021-03-15T22:26:46.145222+0000","last_became_peered":"2021-03-15T22:26:46.145222+0000","last_unstale":"2021-03-15T22:27:10.655500+0000","last_undegraded":"2021-03-15T22:27:10.655500+0000","last_fullsized":"2021-03-15T22:26:46.142576+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:15.579672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:26:15.579672+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"149'22","reported_seq":"172","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.145479+0000","last_change":"2021-03-15T22:26:45.500975+0000","last_active":"2021-03-15T22:27:11.145479+0000","last_peered":"2021-03-15T22:27:11.145479+0000","last_clean":"2021-03-15T22:27:11.145479+0000","last_became_active":"2021-03-15T22:24:55.093124+0000","last_became_peered":"2021-03-15T22:24:55.093124+0000","last_unstale":"2021-03-15T22:27:11.145479+0000","last_undegraded":"2021-03-15T22:27:11.145479+0000","last_fullsized":"2021-03-15T22:27:11.145479+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"149'22","last_scrub_stamp":"2021-03-15T22:26:45.500910+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:26:45.500910+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":21,"num_write":18,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":1288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"184","reported_epoch":"516","state":"active+undersized","last_fresh":"2021-03-15T22:27:11.253080+0000","last_change":"2021-03-15T22:26:46.156962+0000","last_active":"2021-03-15T22:27:11.253080+0000","last_peered":"2021-03-15T22:27:11.253080+0000","last_clean":"2021-03-15T22:21:36.205338+0000","last_became_active":"2021-03-15T22:26:46.156962+0000","last_became_peered":"2021-03-15T22:26:46.156962+0000","last_unstale":"2021-03-15T22:27:11.253080+0000","last_undegraded":"2021-03-15T22:27:11.253080+0000","last_fullsized":"2021-03-15T22:26:46.153068+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":183,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.520588+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.520588+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":["5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"518'1285","reported_seq":"3331","reported_epoch":"518","state":"active+clean","last_fresh":"2021-03-15T22:27:13.317541+0000","last_change":"2021-03-15T22:27:13.317807+0000","last_active":"2021-03-15T22:27:13.317541+0000","last_peered":"2021-03-15T22:27:13.317541+0000","last_clean":"2021-03-15T22:27:13.317541+0000","last_became_active":"2021-03-15T22:25:37.118396+0000","last_became_peered":"2021-03-15T22:25:37.118396+0000","last_unstale":"2021-03-15T22:27:13.317541+0000","last_undegraded":"2021-03-15T22:27:13.317541+0000","last_fullsized":"2021-03-15T22:27:13.317541+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"449'1047","last_scrub_stamp":"2021-03-15T22:26:05.046615+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:26:05.046615+0000","log_size":1285,"ondisk_log_size":1285,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71087646,"num_objects":30,"num_object_clones":23,"num_object_copies":90,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":30,"num_whiteouts":1,"num_read":464,"num_read_kb":450198,"num_write":408,"num_write_kb":175392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":351,"num_bytes_recovered":140467323,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"102","length":"1"},{"start":"13e","length":"1"},{"start":"15b","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"2"},{"start":"185","length":"1"},{"start":"187","length":"1"},{"start":"18b","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"493","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.250302+0000","last_change":"2021-03-15T22:26:03.807852+0000","last_active":"2021-03-15T22:27:11.250302+0000","last_peered":"2021-03-15T22:27:11.250302+0000","last_clean":"2021-03-15T22:27:11.250302+0000","last_became_active":"2021-03-15T22:22:56.062745+0000","last_became_peered":"2021-03-15T22:22:56.062745+0000","last_unstale":"2021-03-15T22:27:11.250302+0000","last_undegraded":"2021-03-15T22:27:11.250302+0000","last_fullsized":"2021-03-15T22:27:11.250302+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.807788+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.807788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"498","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.152833+0000","last_change":"2021-03-15T22:27:08.415705+0000","last_active":"2021-03-15T22:27:12.152833+0000","last_peered":"2021-03-15T22:27:12.152833+0000","last_clean":"2021-03-15T22:27:12.152833+0000","last_became_active":"2021-03-15T22:24:31.111104+0000","last_became_peered":"2021-03-15T22:24:31.111104+0000","last_unstale":"2021-03-15T22:27:12.152833+0000","last_undegraded":"2021-03-15T22:27:12.152833+0000","last_fullsized":"2021-03-15T22:27:12.152833+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:08.415630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:27:08.415630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"113","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.656415+0000","last_change":"2021-03-15T22:25:46.813664+0000","last_active":"2021-03-15T22:27:10.656415+0000","last_peered":"2021-03-15T22:27:10.656415+0000","last_clean":"2021-03-15T22:27:10.656415+0000","last_became_active":"2021-03-15T22:25:33.147285+0000","last_became_peered":"2021-03-15T22:25:33.147285+0000","last_unstale":"2021-03-15T22:27:10.656415+0000","last_undegraded":"2021-03-15T22:27:10.656415+0000","last_fullsized":"2021-03-15T22:27:10.656415+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:46.813605+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.813605+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"65'15","reported_seq":"512","reported_epoch":"514","state":"active+clean","last_fresh":"2021-03-15T22:27:09.254693+0000","last_change":"2021-03-15T22:26:49.219325+0000","last_active":"2021-03-15T22:27:09.254693+0000","last_peered":"2021-03-15T22:27:09.254693+0000","last_clean":"2021-03-15T22:27:09.254693+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:27:09.254693+0000","last_undegraded":"2021-03-15T22:27:09.254693+0000","last_fullsized":"2021-03-15T22:27:09.254693+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:26:49.219225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:49.219225+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":27,"num_read_kb":15,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"488","reported_epoch":"517","state":"active+undersized","last_fresh":"2021-03-15T22:27:12.155203+0000","last_change":"2021-03-15T22:26:46.153760+0000","last_active":"2021-03-15T22:27:12.155203+0000","last_peered":"2021-03-15T22:27:12.155203+0000","last_clean":"2021-03-15T22:26:44.404421+0000","last_became_active":"2021-03-15T22:26:46.153760+0000","last_became_peered":"2021-03-15T22:26:46.153760+0000","last_unstale":"2021-03-15T22:27:12.155203+0000","last_undegraded":"2021-03-15T22:27:12.155203+0000","last_fullsized":"2021-03-15T22:26:46.141981+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:14.026107+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:07.981223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:14.026107+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"498'371","reported_seq":"1693","reported_epoch":"517","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:12.154811+0000","last_change":"2021-03-15T22:26:46.149047+0000","last_active":"2021-03-15T22:27:12.154811+0000","last_peered":"2021-03-15T22:27:12.154811+0000","last_clean":"2021-03-15T22:26:44.404174+0000","last_became_active":"2021-03-15T22:26:46.149047+0000","last_became_peered":"2021-03-15T22:26:46.149047+0000","last_unstale":"2021-03-15T22:27:12.154811+0000","last_undegraded":"2021-03-15T22:26:46.143588+0000","last_fullsized":"2021-03-15T22:26:46.143444+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":361,"parent":"0.0","parent_split_bits":0,"last_scrub":"306'138","last_scrub_stamp":"2021-03-15T22:23:41.141218+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:23:41.141218+0000","log_size":371,"ondisk_log_size":371,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"stat_sum":{"num_bytes":25574593,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":133,"num_read_kb":133672,"num_write":124,"num_write_kb":48597,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":8487583,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":["7","1"],"object_location_counts":[{"shards":"1,7","objects":11}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"502","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.145733+0000","last_change":"2021-03-15T22:26:41.429066+0000","last_active":"2021-03-15T22:27:11.145733+0000","last_peered":"2021-03-15T22:27:11.145733+0000","last_clean":"2021-03-15T22:27:11.145733+0000","last_became_active":"2021-03-15T22:25:33.129602+0000","last_became_peered":"2021-03-15T22:25:33.129602+0000","last_unstale":"2021-03-15T22:27:11.145733+0000","last_undegraded":"2021-03-15T22:27:11.145733+0000","last_fullsized":"2021-03-15T22:27:11.145733+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:41.428997+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:41.428997+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"314","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.136960+0000","last_change":"2021-03-15T22:25:33.143239+0000","last_active":"2021-03-15T22:27:10.136960+0000","last_peered":"2021-03-15T22:27:10.136960+0000","last_clean":"2021-03-15T22:27:10.136960+0000","last_became_active":"2021-03-15T22:25:33.142133+0000","last_became_peered":"2021-03-15T22:25:33.142133+0000","last_unstale":"2021-03-15T22:27:10.136960+0000","last_undegraded":"2021-03-15T22:27:10.136960+0000","last_fullsized":"2021-03-15T22:27:10.136960+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.028082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.028082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,5],"acting":[4,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"1049","reported_epoch":"518","state":"active+clean","last_fresh":"2021-03-15T22:27:13.294422+0000","last_change":"2021-03-15T22:26:09.051870+0000","last_active":"2021-03-15T22:27:13.294422+0000","last_peered":"2021-03-15T22:27:13.294422+0000","last_clean":"2021-03-15T22:27:13.294422+0000","last_became_active":"2021-03-15T22:24:31.109631+0000","last_became_peered":"2021-03-15T22:24:31.109631+0000","last_unstale":"2021-03-15T22:27:13.294422+0000","last_undegraded":"2021-03-15T22:27:13.294422+0000","last_fullsized":"2021-03-15T22:27:13.294422+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:26:09.051794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.051794+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":545,"num_read_kb":540,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,3],"acting":[7,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"65'16","reported_seq":"244","reported_epoch":"516","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:11.145829+0000","last_change":"2021-03-15T22:26:46.146697+0000","last_active":"2021-03-15T22:27:11.145829+0000","last_peered":"2021-03-15T22:27:11.145829+0000","last_clean":"2021-03-15T22:26:44.403437+0000","last_became_active":"2021-03-15T22:26:46.146697+0000","last_became_peered":"2021-03-15T22:26:46.146697+0000","last_unstale":"2021-03-15T22:27:11.145829+0000","last_undegraded":"2021-03-15T22:26:46.140628+0000","last_fullsized":"2021-03-15T22:26:46.140511+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:24:37.977363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:24:37.977363+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":29,"num_read_kb":15,"num_write":16,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":630,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":["1","7"],"object_location_counts":[{"shards":"1,7","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"127","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.250742+0000","last_change":"2021-03-15T22:25:14.115071+0000","last_active":"2021-03-15T22:27:11.250742+0000","last_peered":"2021-03-15T22:27:11.250742+0000","last_clean":"2021-03-15T22:27:11.250742+0000","last_became_active":"2021-03-15T22:25:14.114729+0000","last_became_peered":"2021-03-15T22:25:14.114729+0000","last_unstale":"2021-03-15T22:27:11.250742+0000","last_undegraded":"2021-03-15T22:27:11.250742+0000","last_fullsized":"2021-03-15T22:27:11.250742+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:05.857011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:25:05.857011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"517'1398","reported_seq":"3339","reported_epoch":"517","state":"active+clean","last_fresh":"2021-03-15T22:27:12.310966+0000","last_change":"2021-03-15T22:27:12.180441+0000","last_active":"2021-03-15T22:27:12.310966+0000","last_peered":"2021-03-15T22:27:12.310966+0000","last_clean":"2021-03-15T22:27:12.310966+0000","last_became_active":"2021-03-15T22:25:45.240370+0000","last_became_peered":"2021-03-15T22:25:45.240370+0000","last_unstale":"2021-03-15T22:27:12.310966+0000","last_undegraded":"2021-03-15T22:27:12.310966+0000","last_fullsized":"2021-03-15T22:27:12.310966+0000","mapping_epoch":429,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":430,"parent":"0.0","parent_split_bits":0,"last_scrub":"487'1290","last_scrub_stamp":"2021-03-15T22:26:44.759743+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:26:44.759743+0000","log_size":1398,"ondisk_log_size":1398,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78080217,"num_objects":34,"num_object_clones":27,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":1,"num_read":703,"num_read_kb":786498,"num_write":420,"num_write_kb":163610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":277,"num_bytes_recovered":125184016,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"102","length":"1"},{"start":"13e","length":"1"},{"start":"15b","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"2"},{"start":"185","length":"1"},{"start":"187","length":"1"},{"start":"18b","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"485","reported_epoch":"516","state":"active+undersized","last_fresh":"2021-03-15T22:27:11.144156+0000","last_change":"2021-03-15T22:26:46.145792+0000","last_active":"2021-03-15T22:27:11.144156+0000","last_peered":"2021-03-15T22:27:11.144156+0000","last_clean":"2021-03-15T22:26:44.402909+0000","last_became_active":"2021-03-15T22:26:46.145792+0000","last_became_peered":"2021-03-15T22:26:46.145792+0000","last_unstale":"2021-03-15T22:27:11.144156+0000","last_undegraded":"2021-03-15T22:27:11.144156+0000","last_fullsized":"2021-03-15T22:26:46.139336+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.950135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.950135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"491","reported_epoch":"516","state":"active+undersized","last_fresh":"2021-03-15T22:27:11.146022+0000","last_change":"2021-03-15T22:26:46.153638+0000","last_active":"2021-03-15T22:27:11.146022+0000","last_peered":"2021-03-15T22:27:11.146022+0000","last_clean":"2021-03-15T22:26:44.404764+0000","last_became_active":"2021-03-15T22:26:46.153638+0000","last_became_peered":"2021-03-15T22:26:46.153638+0000","last_unstale":"2021-03-15T22:27:11.146022+0000","last_undegraded":"2021-03-15T22:27:11.146022+0000","last_fullsized":"2021-03-15T22:26:46.141071+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:26.961978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:26.961978+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"78'31","reported_seq":"265","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.249458+0000","last_change":"2021-03-15T22:26:05.733910+0000","last_active":"2021-03-15T22:27:11.249458+0000","last_peered":"2021-03-15T22:27:11.249458+0000","last_clean":"2021-03-15T22:27:11.249458+0000","last_became_active":"2021-03-15T22:24:31.098665+0000","last_became_peered":"2021-03-15T22:24:31.098665+0000","last_unstale":"2021-03-15T22:27:11.249458+0000","last_undegraded":"2021-03-15T22:27:11.249458+0000","last_fullsized":"2021-03-15T22:27:11.249458+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"78'31","last_scrub_stamp":"2021-03-15T22:26:05.733849+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:26:05.733849+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":45,"num_read_kb":27,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":642,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"498","reported_epoch":"514","state":"active+undersized","last_fresh":"2021-03-15T22:27:09.254466+0000","last_change":"2021-03-15T22:26:46.154344+0000","last_active":"2021-03-15T22:27:09.254466+0000","last_peered":"2021-03-15T22:27:09.254466+0000","last_clean":"2021-03-15T22:26:44.404426+0000","last_became_active":"2021-03-15T22:26:46.154344+0000","last_became_peered":"2021-03-15T22:26:46.154344+0000","last_unstale":"2021-03-15T22:27:09.254466+0000","last_undegraded":"2021-03-15T22:27:09.254466+0000","last_fullsized":"2021-03-15T22:26:46.143315+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:06.865257+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:26:06.865257+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":["6","5"],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"511'218","reported_seq":"1601","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.136451+0000","last_change":"2021-03-15T22:27:07.690266+0000","last_active":"2021-03-15T22:27:10.136451+0000","last_peered":"2021-03-15T22:27:10.136451+0000","last_clean":"2021-03-15T22:27:10.136451+0000","last_became_active":"2021-03-15T22:25:37.112715+0000","last_became_peered":"2021-03-15T22:25:37.112715+0000","last_unstale":"2021-03-15T22:27:10.136451+0000","last_undegraded":"2021-03-15T22:27:10.136451+0000","last_fullsized":"2021-03-15T22:27:10.136451+0000","mapping_epoch":421,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"496'210","last_scrub_stamp":"2021-03-15T22:26:51.966934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:26:51.966934+0000","log_size":218,"ondisk_log_size":218,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":18297389,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":90,"num_read_kb":107698,"num_write":65,"num_write_kb":27156,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":14412391,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"102","length":"1"},{"start":"13e","length":"1"},{"start":"15b","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"2"},{"start":"187","length":"1"},{"start":"18b","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"520","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.655918+0000","last_change":"2021-03-15T22:26:46.159027+0000","last_active":"2021-03-15T22:27:10.655918+0000","last_peered":"2021-03-15T22:27:10.655918+0000","last_clean":"2021-03-15T22:26:44.342756+0000","last_became_active":"2021-03-15T22:26:46.159027+0000","last_became_peered":"2021-03-15T22:26:46.159027+0000","last_unstale":"2021-03-15T22:27:10.655918+0000","last_undegraded":"2021-03-15T22:27:10.655918+0000","last_fullsized":"2021-03-15T22:26:46.144155+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:23:58.480279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:23:58.480279+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":["3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"505","reported_epoch":"516","state":"active+undersized","last_fresh":"2021-03-15T22:27:11.251968+0000","last_change":"2021-03-15T22:26:46.153759+0000","last_active":"2021-03-15T22:27:11.251968+0000","last_peered":"2021-03-15T22:27:11.251968+0000","last_clean":"2021-03-15T22:26:44.404948+0000","last_became_active":"2021-03-15T22:26:46.153759+0000","last_became_peered":"2021-03-15T22:26:46.153759+0000","last_unstale":"2021-03-15T22:27:11.251968+0000","last_undegraded":"2021-03-15T22:27:11.251968+0000","last_fullsized":"2021-03-15T22:26:46.145238+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:39.716910+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:26:39.716910+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"65'16","reported_seq":"613","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.145437+0000","last_change":"2021-03-15T22:26:06.698784+0000","last_active":"2021-03-15T22:27:11.145437+0000","last_peered":"2021-03-15T22:27:11.145437+0000","last_clean":"2021-03-15T22:27:11.145437+0000","last_became_active":"2021-03-15T22:25:14.114107+0000","last_became_peered":"2021-03-15T22:25:14.114107+0000","last_unstale":"2021-03-15T22:27:11.145437+0000","last_undegraded":"2021-03-15T22:27:11.145437+0000","last_fullsized":"2021-03-15T22:27:11.145437+0000","mapping_epoch":398,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":399,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'16","last_scrub_stamp":"2021-03-15T22:26:06.698716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:06.698716+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":33,"num_read_kb":16,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"356'3","reported_seq":"220","reported_epoch":"514","state":"active+clean","last_fresh":"2021-03-15T22:27:09.254121+0000","last_change":"2021-03-15T22:26:02.781579+0000","last_active":"2021-03-15T22:27:09.254121+0000","last_peered":"2021-03-15T22:27:09.254121+0000","last_clean":"2021-03-15T22:27:09.254121+0000","last_became_active":"2021-03-15T22:24:31.105948+0000","last_became_peered":"2021-03-15T22:24:31.105948+0000","last_unstale":"2021-03-15T22:27:09.254121+0000","last_undegraded":"2021-03-15T22:27:09.254121+0000","last_fullsized":"2021-03-15T22:27:09.254121+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"356'3","last_scrub_stamp":"2021-03-15T22:26:02.781478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:26:02.781478+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,3],"acting":[6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.2","version":"512'366","reported_seq":"1632","reported_epoch":"515","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:10.655585+0000","last_change":"2021-03-15T22:26:46.144773+0000","last_active":"2021-03-15T22:27:10.655585+0000","last_peered":"2021-03-15T22:27:10.655585+0000","last_clean":"2021-03-15T22:26:44.344101+0000","last_became_active":"2021-03-15T22:26:46.144773+0000","last_became_peered":"2021-03-15T22:26:46.144773+0000","last_unstale":"2021-03-15T22:27:10.655585+0000","last_undegraded":"2021-03-15T22:26:46.142146+0000","last_fullsized":"2021-03-15T22:26:46.142003+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":265,"parent":"0.0","parent_split_bits":0,"last_scrub":"366'202","last_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:24:42.006150+0000","log_size":366,"ondisk_log_size":366,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":22850860,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":151,"num_read_kb":141704,"num_write":116,"num_write_kb":46609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":6765552,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":["3","1"],"object_location_counts":[{"shards":"1,3","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"484","reported_epoch":"517","state":"active+undersized","last_fresh":"2021-03-15T22:27:12.153287+0000","last_change":"2021-03-15T22:26:46.158212+0000","last_active":"2021-03-15T22:27:12.153287+0000","last_peered":"2021-03-15T22:27:12.153287+0000","last_clean":"2021-03-15T22:26:44.403963+0000","last_became_active":"2021-03-15T22:26:46.158212+0000","last_became_peered":"2021-03-15T22:26:46.158212+0000","last_unstale":"2021-03-15T22:27:12.153287+0000","last_undegraded":"2021-03-15T22:27:12.153287+0000","last_fullsized":"2021-03-15T22:26:46.146288+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.234565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.234565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"184","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.145614+0000","last_change":"2021-03-15T22:27:08.372942+0000","last_active":"2021-03-15T22:27:11.145614+0000","last_peered":"2021-03-15T22:27:11.145614+0000","last_clean":"2021-03-15T22:27:11.145614+0000","last_became_active":"2021-03-15T22:24:31.099610+0000","last_became_peered":"2021-03-15T22:24:31.099610+0000","last_unstale":"2021-03-15T22:27:11.145614+0000","last_undegraded":"2021-03-15T22:27:11.145614+0000","last_fullsized":"2021-03-15T22:27:11.145614+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:08.372872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:08.372872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"65'20","reported_seq":"520","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.137111+0000","last_change":"2021-03-15T22:26:09.008669+0000","last_active":"2021-03-15T22:27:10.137111+0000","last_peered":"2021-03-15T22:27:10.137111+0000","last_clean":"2021-03-15T22:27:10.137111+0000","last_became_active":"2021-03-15T22:17:55.986569+0000","last_became_peered":"2021-03-15T22:17:55.986569+0000","last_unstale":"2021-03-15T22:27:10.137111+0000","last_undegraded":"2021-03-15T22:27:10.137111+0000","last_fullsized":"2021-03-15T22:27:10.137111+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'20","last_scrub_stamp":"2021-03-15T22:26:09.008589+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:09.008589+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":39,"num_read_kb":20,"num_write":20,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"421'6","reported_seq":"136","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.655816+0000","last_change":"2021-03-15T22:26:15.749409+0000","last_active":"2021-03-15T22:27:10.655816+0000","last_peered":"2021-03-15T22:27:10.655816+0000","last_clean":"2021-03-15T22:27:10.655816+0000","last_became_active":"2021-03-15T22:25:33.130973+0000","last_became_peered":"2021-03-15T22:25:33.130973+0000","last_unstale":"2021-03-15T22:27:10.655816+0000","last_undegraded":"2021-03-15T22:27:10.655816+0000","last_fullsized":"2021-03-15T22:27:10.655816+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'6","last_scrub_stamp":"2021-03-15T22:26:15.749336+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:26:15.749336+0000","last_clean_scrub_stamp":"2021-03-15T22:26:15.749336+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"506'623","reported_seq":"1982","reported_epoch":"515","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:10.655599+0000","last_change":"2021-03-15T22:26:46.151657+0000","last_active":"2021-03-15T22:27:10.655599+0000","last_peered":"2021-03-15T22:27:10.655599+0000","last_clean":"2021-03-15T22:26:44.343409+0000","last_became_active":"2021-03-15T22:26:46.151657+0000","last_became_peered":"2021-03-15T22:26:46.151657+0000","last_unstale":"2021-03-15T22:27:10.655599+0000","last_undegraded":"2021-03-15T22:26:46.143102+0000","last_fullsized":"2021-03-15T22:26:46.142948+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":266,"parent":"0.0","parent_split_bits":0,"last_scrub":"318'302","last_scrub_stamp":"2021-03-15T22:23:59.467761+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:23:59.467761+0000","log_size":623,"ondisk_log_size":623,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"stat_sum":{"num_bytes":41743902,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":287,"num_read_kb":295869,"num_write":196,"num_write_kb":70847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31233387,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":20}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"505","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.136468+0000","last_change":"2021-03-15T22:26:46.150237+0000","last_active":"2021-03-15T22:27:10.136468+0000","last_peered":"2021-03-15T22:27:10.136468+0000","last_clean":"2021-03-15T22:26:44.341370+0000","last_became_active":"2021-03-15T22:26:46.150237+0000","last_became_peered":"2021-03-15T22:26:46.150237+0000","last_unstale":"2021-03-15T22:27:10.136468+0000","last_undegraded":"2021-03-15T22:27:10.136468+0000","last_fullsized":"2021-03-15T22:26:46.137731+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:47.117252+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:47.117252+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3],"avail_no_missing":["4","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"284","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.656999+0000","last_change":"2021-03-15T22:25:12.911203+0000","last_active":"2021-03-15T22:27:10.656999+0000","last_peered":"2021-03-15T22:27:10.656999+0000","last_clean":"2021-03-15T22:27:10.656999+0000","last_became_active":"2021-03-15T22:24:31.091206+0000","last_became_peered":"2021-03-15T22:24:31.091206+0000","last_unstale":"2021-03-15T22:27:10.656999+0000","last_undegraded":"2021-03-15T22:27:10.656999+0000","last_fullsized":"2021-03-15T22:27:10.656999+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:12.911140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:25:12.911140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"82'34","reported_seq":"139","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.655720+0000","last_change":"2021-03-15T22:26:46.453382+0000","last_active":"2021-03-15T22:27:10.655720+0000","last_peered":"2021-03-15T22:27:10.655720+0000","last_clean":"2021-03-15T22:27:10.655720+0000","last_became_active":"2021-03-15T22:25:33.149362+0000","last_became_peered":"2021-03-15T22:25:33.149362+0000","last_unstale":"2021-03-15T22:27:10.655720+0000","last_undegraded":"2021-03-15T22:27:10.655720+0000","last_fullsized":"2021-03-15T22:27:10.655720+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"82'34","last_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:26:46.453319+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":46,"num_read_kb":26,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":846,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"512","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.773080+0000","last_change":"2021-03-15T22:27:11.773080+0000","last_active":"2021-03-15T22:27:11.773080+0000","last_peered":"2021-03-15T22:27:11.773080+0000","last_clean":"2021-03-15T22:27:11.773080+0000","last_became_active":"2021-03-15T22:25:33.142712+0000","last_became_peered":"2021-03-15T22:25:33.142712+0000","last_unstale":"2021-03-15T22:27:11.773080+0000","last_undegraded":"2021-03-15T22:27:11.773080+0000","last_fullsized":"2021-03-15T22:27:11.773080+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:11.773015+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:27:11.773015+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"512'595","reported_seq":"2012","reported_epoch":"514","state":"active+clean","last_fresh":"2021-03-15T22:27:09.257323+0000","last_change":"2021-03-15T22:27:07.238532+0000","last_active":"2021-03-15T22:27:09.257323+0000","last_peered":"2021-03-15T22:27:09.257323+0000","last_clean":"2021-03-15T22:27:09.257323+0000","last_became_active":"2021-03-15T22:25:36.118668+0000","last_became_peered":"2021-03-15T22:25:36.118668+0000","last_unstale":"2021-03-15T22:27:09.257323+0000","last_undegraded":"2021-03-15T22:27:09.257323+0000","last_fullsized":"2021-03-15T22:27:09.257323+0000","mapping_epoch":420,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":421,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'557","last_scrub_stamp":"2021-03-15T22:26:42.075267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:26:42.075267+0000","log_size":595,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26679848,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":215,"num_read_kb":204846,"num_write":157,"num_write_kb":63105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25937939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"102","length":"1"},{"start":"13e","length":"1"},{"start":"15b","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"2"},{"start":"187","length":"1"},{"start":"18b","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"490","reported_epoch":"516","state":"active+undersized","last_fresh":"2021-03-15T22:27:11.251681+0000","last_change":"2021-03-15T22:26:46.152596+0000","last_active":"2021-03-15T22:27:11.251681+0000","last_peered":"2021-03-15T22:27:11.251681+0000","last_clean":"2021-03-15T22:26:44.404186+0000","last_became_active":"2021-03-15T22:26:46.152596+0000","last_became_peered":"2021-03-15T22:26:46.152596+0000","last_unstale":"2021-03-15T22:27:11.251681+0000","last_undegraded":"2021-03-15T22:27:11.251681+0000","last_fullsized":"2021-03-15T22:26:46.142421+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:14.869980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:26:14.869980+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"491'8","reported_seq":"650","reported_epoch":"515","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:10.137570+0000","last_change":"2021-03-15T22:26:46.157394+0000","last_active":"2021-03-15T22:27:10.137570+0000","last_peered":"2021-03-15T22:27:10.137570+0000","last_clean":"2021-03-15T22:26:44.342144+0000","last_became_active":"2021-03-15T22:26:46.157394+0000","last_became_peered":"2021-03-15T22:26:46.157394+0000","last_unstale":"2021-03-15T22:27:10.137570+0000","last_undegraded":"2021-03-15T22:26:46.138720+0000","last_fullsized":"2021-03-15T22:26:46.138407+0000","mapping_epoch":490,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":388,"parent":"0.0","parent_split_bits":0,"last_scrub":"391'7","last_scrub_stamp":"2021-03-15T22:25:46.103504+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.103504+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":["4","5"],"object_location_counts":[{"shards":"4,5","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"113","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.655753+0000","last_change":"2021-03-15T22:25:44.867726+0000","last_active":"2021-03-15T22:27:10.655753+0000","last_peered":"2021-03-15T22:27:10.655753+0000","last_clean":"2021-03-15T22:27:10.655753+0000","last_became_active":"2021-03-15T22:25:33.148614+0000","last_became_peered":"2021-03-15T22:25:33.148614+0000","last_unstale":"2021-03-15T22:27:10.655753+0000","last_undegraded":"2021-03-15T22:27:10.655753+0000","last_fullsized":"2021-03-15T22:27:10.655753+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:44.867633+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:44.867633+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"65'17","reported_seq":"286","reported_epoch":"515","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:10.655684+0000","last_change":"2021-03-15T22:26:46.152570+0000","last_active":"2021-03-15T22:27:10.655684+0000","last_peered":"2021-03-15T22:27:10.655684+0000","last_clean":"2021-03-15T22:22:54.044964+0000","last_became_active":"2021-03-15T22:26:46.152570+0000","last_became_peered":"2021-03-15T22:26:46.152570+0000","last_unstale":"2021-03-15T22:27:10.655684+0000","last_undegraded":"2021-03-15T22:26:46.144265+0000","last_fullsized":"2021-03-15T22:26:46.144161+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'17","last_scrub_stamp":"2021-03-15T22:25:46.725489+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:25:46.725489+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":19,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[{"shards":"3,4","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"514'170","reported_seq":"1511","reported_epoch":"514","state":"active+clean","last_fresh":"2021-03-15T22:27:09.270337+0000","last_change":"2021-03-15T22:27:07.230708+0000","last_active":"2021-03-15T22:27:09.270337+0000","last_peered":"2021-03-15T22:27:09.270337+0000","last_clean":"2021-03-15T22:27:09.270337+0000","last_became_active":"2021-03-15T22:24:31.104356+0000","last_became_peered":"2021-03-15T22:24:31.104356+0000","last_unstale":"2021-03-15T22:27:09.270337+0000","last_undegraded":"2021-03-15T22:27:09.270337+0000","last_fullsized":"2021-03-15T22:27:09.270337+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"485'143","last_scrub_stamp":"2021-03-15T22:26:45.157116+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:26:45.157116+0000","log_size":170,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11445679,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":92,"num_read_kb":89950,"num_write":43,"num_write_kb":15087,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1551588,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,3],"acting":[6,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"102","length":"1"},{"start":"13e","length":"1"},{"start":"15b","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"2"},{"start":"187","length":"1"},{"start":"18b","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"113","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.656580+0000","last_change":"2021-03-15T22:26:48.509246+0000","last_active":"2021-03-15T22:27:10.656580+0000","last_peered":"2021-03-15T22:27:10.656580+0000","last_clean":"2021-03-15T22:27:10.656580+0000","last_became_active":"2021-03-15T22:25:33.147717+0000","last_became_peered":"2021-03-15T22:25:33.147717+0000","last_unstale":"2021-03-15T22:27:10.656580+0000","last_undegraded":"2021-03-15T22:27:10.656580+0000","last_fullsized":"2021-03-15T22:27:10.656580+0000","mapping_epoch":417,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:48.509183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:26:48.509183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"147'5","reported_seq":"576","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:12.101256+0000","last_change":"2021-03-15T22:26:43.725292+0000","last_active":"2021-03-15T22:27:12.101256+0000","last_peered":"2021-03-15T22:27:12.101256+0000","last_clean":"2021-03-15T22:27:12.101256+0000","last_became_active":"2021-03-15T22:21:02.017066+0000","last_became_peered":"2021-03-15T22:21:02.017066+0000","last_unstale":"2021-03-15T22:27:12.101256+0000","last_undegraded":"2021-03-15T22:27:12.101256+0000","last_fullsized":"2021-03-15T22:27:12.101256+0000","mapping_epoch":60,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":147,"parent":"0.0","parent_split_bits":0,"last_scrub":"147'5","last_scrub_stamp":"2021-03-15T22:26:43.725206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:26:43.725206+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"490","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.136825+0000","last_change":"2021-03-15T22:26:46.147246+0000","last_active":"2021-03-15T22:27:10.136825+0000","last_peered":"2021-03-15T22:27:10.136825+0000","last_clean":"2021-03-15T22:26:44.341791+0000","last_became_active":"2021-03-15T22:26:46.147246+0000","last_became_peered":"2021-03-15T22:26:46.147246+0000","last_unstale":"2021-03-15T22:27:10.136825+0000","last_undegraded":"2021-03-15T22:27:10.136825+0000","last_fullsized":"2021-03-15T22:26:46.138599+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.991018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.991018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"65'15","reported_seq":"570","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.137511+0000","last_change":"2021-03-15T22:26:46.860101+0000","last_active":"2021-03-15T22:27:10.137511+0000","last_peered":"2021-03-15T22:27:10.137511+0000","last_clean":"2021-03-15T22:27:10.137511+0000","last_became_active":"2021-03-15T22:24:55.090544+0000","last_became_peered":"2021-03-15T22:24:55.090544+0000","last_unstale":"2021-03-15T22:27:10.137511+0000","last_undegraded":"2021-03-15T22:27:10.137511+0000","last_fullsized":"2021-03-15T22:27:10.137511+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:26:46.860014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:46.860014+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":17,"num_write":16,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"485'172","reported_seq":"1568","reported_epoch":"516","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:11.251202+0000","last_change":"2021-03-15T22:26:46.156901+0000","last_active":"2021-03-15T22:27:11.251202+0000","last_peered":"2021-03-15T22:27:11.251202+0000","last_clean":"2021-03-15T22:26:40.154486+0000","last_became_active":"2021-03-15T22:26:46.156901+0000","last_became_peered":"2021-03-15T22:26:46.156901+0000","last_unstale":"2021-03-15T22:27:11.251202+0000","last_undegraded":"2021-03-15T22:26:46.151937+0000","last_fullsized":"2021-03-15T22:26:46.151801+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":422,"parent":"0.0","parent_split_bits":0,"last_scrub":"394'115","last_scrub_stamp":"2021-03-15T22:25:10.616929+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:10.616929+0000","log_size":172,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"stat_sum":{"num_bytes":9297196,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":119,"num_read_kb":145290,"num_write":59,"num_write_kb":15225,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":16774328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":["5","3"],"object_location_counts":[{"shards":"3,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"421","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.655873+0000","last_change":"2021-03-15T22:26:46.151840+0000","last_active":"2021-03-15T22:27:10.655873+0000","last_peered":"2021-03-15T22:27:10.655873+0000","last_clean":"2021-03-15T22:25:31.105421+0000","last_became_active":"2021-03-15T22:26:46.151840+0000","last_became_peered":"2021-03-15T22:26:46.151840+0000","last_unstale":"2021-03-15T22:27:10.655873+0000","last_undegraded":"2021-03-15T22:27:10.655873+0000","last_fullsized":"2021-03-15T22:26:46.144481+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":418,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.754853+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.754853+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":["3","4"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"513","reported_epoch":"516","state":"active+clean","last_fresh":"2021-03-15T22:27:11.146339+0000","last_change":"2021-03-15T22:26:16.580494+0000","last_active":"2021-03-15T22:27:11.146339+0000","last_peered":"2021-03-15T22:27:11.146339+0000","last_clean":"2021-03-15T22:27:11.146339+0000","last_became_active":"2021-03-15T22:24:31.106634+0000","last_became_peered":"2021-03-15T22:24:31.106634+0000","last_unstale":"2021-03-15T22:27:11.146339+0000","last_undegraded":"2021-03-15T22:27:11.146339+0000","last_fullsized":"2021-03-15T22:27:11.146339+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:16.580433+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:26:16.580433+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,6],"acting":[1,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"490","reported_epoch":"514","state":"active+clean","last_fresh":"2021-03-15T22:27:09.254309+0000","last_change":"2021-03-15T22:26:03.831424+0000","last_active":"2021-03-15T22:27:09.254309+0000","last_peered":"2021-03-15T22:27:09.254309+0000","last_clean":"2021-03-15T22:27:09.254309+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:27:09.254309+0000","last_undegraded":"2021-03-15T22:27:09.254309+0000","last_fullsized":"2021-03-15T22:27:09.254309+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:03.831371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:03.831371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"115'33","reported_seq":"536","reported_epoch":"515","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:10.137125+0000","last_change":"2021-03-15T22:26:46.147116+0000","last_active":"2021-03-15T22:27:10.137125+0000","last_peered":"2021-03-15T22:27:10.137125+0000","last_clean":"2021-03-15T22:26:44.341791+0000","last_became_active":"2021-03-15T22:26:46.147116+0000","last_became_peered":"2021-03-15T22:26:46.147116+0000","last_unstale":"2021-03-15T22:27:10.137125+0000","last_undegraded":"2021-03-15T22:26:46.138362+0000","last_fullsized":"2021-03-15T22:26:46.138236+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'33","last_scrub_stamp":"2021-03-15T22:25:04.028780+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:25:04.028780+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":41,"num_read_kb":23,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":12}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"511'487","reported_seq":"2040","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.657131+0000","last_change":"2021-03-15T22:27:07.232891+0000","last_active":"2021-03-15T22:27:10.657131+0000","last_peered":"2021-03-15T22:27:10.657131+0000","last_clean":"2021-03-15T22:27:10.657131+0000","last_became_active":"2021-03-15T22:25:39.143407+0000","last_became_peered":"2021-03-15T22:25:39.143407+0000","last_unstale":"2021-03-15T22:27:10.657131+0000","last_undegraded":"2021-03-15T22:27:10.657131+0000","last_fullsized":"2021-03-15T22:27:10.657131+0000","mapping_epoch":423,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":424,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'449","last_scrub_stamp":"2021-03-15T22:26:42.479453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:26:42.479453+0000","log_size":487,"ondisk_log_size":487,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14617646,"num_objects":8,"num_object_clones":4,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":221,"num_read_kb":195018,"num_write":162,"num_write_kb":58941,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":155,"num_bytes_recovered":38892785,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,4],"acting":[3,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"102","length":"1"},{"start":"13e","length":"1"},{"start":"15b","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"1"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"2"},{"start":"187","length":"1"},{"start":"18b","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"482","reported_epoch":"515","state":"active+undersized","last_fresh":"2021-03-15T22:27:10.137590+0000","last_change":"2021-03-15T22:26:46.144801+0000","last_active":"2021-03-15T22:27:10.137590+0000","last_peered":"2021-03-15T22:27:10.137590+0000","last_clean":"2021-03-15T22:26:44.342145+0000","last_became_active":"2021-03-15T22:26:46.144801+0000","last_became_peered":"2021-03-15T22:26:46.144801+0000","last_unstale":"2021-03-15T22:27:10.137590+0000","last_undegraded":"2021-03-15T22:27:10.137590+0000","last_fullsized":"2021-03-15T22:26:46.138535+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.910565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.910565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"491'5","reported_seq":"51","reported_epoch":"517","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:27:12.155227+0000","last_change":"2021-03-15T22:26:46.153832+0000","last_active":"2021-03-15T22:27:12.155227+0000","last_peered":"2021-03-15T22:27:12.155227+0000","last_clean":"2021-03-15T22:17:58.023417+0000","last_became_active":"2021-03-15T22:26:46.153832+0000","last_became_peered":"2021-03-15T22:26:46.153832+0000","last_unstale":"2021-03-15T22:27:12.155227+0000","last_undegraded":"2021-03-15T22:26:46.148369+0000","last_fullsized":"2021-03-15T22:26:46.148248+0000","mapping_epoch":490,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"155'4","last_scrub_stamp":"2021-03-15T22:24:29.903442+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:24:29.903442+0000","log_size":3,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":["7","5"],"object_location_counts":[{"shards":"5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"194","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.656498+0000","last_change":"2021-03-15T22:25:02.791176+0000","last_active":"2021-03-15T22:27:10.656498+0000","last_peered":"2021-03-15T22:27:10.656498+0000","last_clean":"2021-03-15T22:27:10.656498+0000","last_became_active":"2021-03-15T22:24:31.097434+0000","last_became_peered":"2021-03-15T22:24:31.097434+0000","last_unstale":"2021-03-15T22:27:10.656498+0000","last_undegraded":"2021-03-15T22:27:10.656498+0000","last_fullsized":"2021-03-15T22:27:10.656498+0000","mapping_epoch":355,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":356,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:02.791108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:06.573181+0000","last_clean_scrub_stamp":"2021-03-15T22:25:02.791108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,1],"acting":[3,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"65'13","reported_seq":"562","reported_epoch":"515","state":"active+clean","last_fresh":"2021-03-15T22:27:10.137311+0000","last_change":"2021-03-15T22:26:07.977706+0000","last_active":"2021-03-15T22:27:10.137311+0000","last_peered":"2021-03-15T22:27:10.137311+0000","last_clean":"2021-03-15T22:27:10.137311+0000","last_became_active":"2021-03-15T22:22:56.053800+0000","last_became_peered":"2021-03-15T22:22:56.053800+0000","last_unstale":"2021-03-15T22:27:10.137311+0000","last_undegraded":"2021-03-15T22:27:10.137311+0000","last_fullsized":"2021-03-15T22:27:10.137311+0000","mapping_epoch":260,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":261,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'13","last_scrub_stamp":"2021-03-15T22:26:07.977641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:07.977641+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":13,"num_write":13,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":499184088,"num_objects":222,"num_object_clones":172,"num_object_copies":666,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":84,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":222,"num_whiteouts":12,"num_read":3774,"num_read_kb":3872143,"num_write":2736,"num_write_kb":1019737,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1197,"num_bytes_recovered":514315662,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1021452288,"data_stored":1431550063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8752,"ondisk_log_size":8752,"up":42,"acting":42,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":545,"num_read_kb":540,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1556,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":8,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1681,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":5292,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":86,"acting":86,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":114,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1059,"num_read_kb":564,"num_write":542,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":420,"num_bytes_recovered":34608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2232320,"data_stored":80092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":598,"ondisk_log_size":598,"up":85,"acting":85,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":35,"ondisk_log_size":35,"up":57,"acting":57,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":79,"acting":79,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299508,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146924,"kb_used_data":121380,"kb_used_omap":0,"kb_used_meta":25536,"kb_avail":93598228,"statfs":{"total":95995035648,"available":95844585472,"internally_reserved":0,"allocated":124293120,"data_stored":169974357,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26148864},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:27:06 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.47799999999999998,"15min":0.49399999999999999},"min":{"1min":0.317,"5min":0.215,"15min":0.215},"max":{"1min":1.4750000000000001,"5min":1.4750000000000001,"15min":2.73},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.48699999999999999,"15min":0.503},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":2.0499999999999998,"5min":2.0499999999999998,"15min":3.8799999999999999},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:27:06 2021","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.499,"15min":0.49099999999999999},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.22600000000000001},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":2.6859999999999999},"last":0.372},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.50600000000000001,"15min":0.5},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":2.0779999999999998,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.35599999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.80700000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.502},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.32400000000000001,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":0.73099999999999998,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.48799999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:27:06 2021","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.49199999999999999,"15min":0.53300000000000003},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.30299999999999999},"max":{"1min":1.702,"5min":1.702,"15min":2.0339999999999998},"last":0.32900000000000001},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.52700000000000002,"15min":0.55100000000000005},"min":{"1min":0.35099999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.496,"5min":1.496,"15min":2.7949999999999999},"last":0.27300000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:27:06 2021","interfaces":[{"interface":"back","average":{"1min":0.57699999999999996,"5min":0.46100000000000002,"15min":0.44700000000000001},"min":{"1min":0.313,"5min":0.23799999999999999,"15min":0.23799999999999999},"max":{"1min":1.927,"5min":1.927,"15min":1.927},"last":0.307},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.47299999999999998,"15min":0.45900000000000002},"min":{"1min":0.33300000000000002,"5min":0.29099999999999998,"15min":0.28799999999999998},"max":{"1min":1.6779999999999999,"5min":1.6779999999999999,"15min":2.5339999999999998},"last":0.28299999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:27:06 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.51300000000000001,"15min":0.47299999999999998},"min":{"1min":0.379,"5min":0.253,"15min":0.253},"max":{"1min":1.407,"5min":1.407,"15min":2.5110000000000001},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.34999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.47499999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:27:06 2021","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.54400000000000004,"15min":0.52300000000000002},"min":{"1min":0.38300000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":2.1179999999999999,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.502},{"interface":"front","average":{"1min":0.60799999999999998,"5min":0.53900000000000003,"15min":0.51900000000000002},"min":{"1min":0.36599999999999999,"5min":0.32600000000000001,"15min":0.29299999999999998},"max":{"1min":1.8560000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":0.36599999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528439,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":118072,"kb_used_data":99184,"kb_used_omap":0,"kb_used_meta":18880,"kb_avail":93627080,"statfs":{"total":95995035648,"available":95874129920,"internally_reserved":0,"allocated":101564416,"data_stored":134706665,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19333120},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:26:47 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47299999999999998,"15min":0.48699999999999999},"min":{"1min":0.34000000000000002,"5min":0.249,"15min":0.249},"max":{"1min":0.70099999999999996,"5min":0.71399999999999997,"15min":0.995},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46700000000000003,"15min":0.48399999999999999},"min":{"1min":0.32800000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59499999999999997,"5min":0.71099999999999997,"15min":1.1899999999999999},"last":0.49199999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:26:47 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.49199999999999999,"15min":0.50800000000000001},"min":{"1min":0.34000000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.628,"5min":0.69399999999999995,"15min":1.05},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.48999999999999999,"15min":0.48299999999999998},"min":{"1min":0.32600000000000001,"5min":0.29799999999999999,"15min":0.29299999999999998},"max":{"1min":0.626,"5min":1.8280000000000001,"15min":1.8280000000000001},"last":0.44}]},{"osd":2,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.317,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65200000000000002,"5min":0.77600000000000002,"15min":0.80800000000000005},"last":0.625},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48899999999999999,"15min":0.498},"min":{"1min":0.35399999999999998,"5min":0.32800000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.42899999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:26:47 2021","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.51500000000000001,"15min":0.52300000000000002},"min":{"1min":0.33500000000000002,"5min":0.32000000000000001,"15min":0.29899999999999999},"max":{"1min":0.72299999999999998,"5min":0.90400000000000003,"15min":1.026},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51300000000000001,"15min":0.52300000000000002},"min":{"1min":0.33300000000000002,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.66600000000000004,"5min":0.83399999999999996,"15min":0.83399999999999996},"last":0.66700000000000004}]},{"osd":4,"last update":"Mon Mar 15 22:26:47 2021","interfaces":[{"interface":"back","average":{"1min":0.41299999999999998,"5min":0.45300000000000001,"15min":0.41899999999999998},"min":{"1min":0.27500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.55300000000000005,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.49299999999999999,"5min":0.51500000000000001,"15min":0.46800000000000003},"min":{"1min":0.32500000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.76500000000000001,"5min":0.95899999999999996,"15min":0.95899999999999996},"last":0.36699999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:26:47 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.498,"15min":0.46500000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66700000000000004,"5min":0.89200000000000002,"15min":1.135},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53700000000000003,"15min":0.495},"min":{"1min":0.371,"5min":0.33200000000000002,"15min":0.311},"max":{"1min":0.65100000000000002,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.39700000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:27:01 2021","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52900000000000003,"15min":0.51000000000000001},"min":{"1min":0.40500000000000003,"5min":0.36699999999999999,"15min":0.29599999999999999},"max":{"1min":0.73799999999999999,"5min":0.82499999999999996,"15min":0.82499999999999996},"last":0.309},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.56000000000000005,"15min":0.54600000000000004},"min":{"1min":0.32900000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.70599999999999996,"5min":0.92600000000000005,"15min":1.1659999999999999},"last":0.42599999999999999}]}]},{"osd":1,"up_from":13,"seq":55834574985,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":202644,"kb_used_data":169228,"kb_used_omap":0,"kb_used_meta":33408,"kb_avail":93542508,"statfs":{"total":95995035648,"available":95787528192,"internally_reserved":0,"allocated":173289472,"data_stored":236311822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34209792},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:26:32 2021","interfaces":[{"interface":"back","average":{"1min":0.309,"5min":0.33800000000000002,"15min":0.52500000000000002},"min":{"1min":0.19900000000000001,"5min":0.191,"15min":0.191},"max":{"1min":0.46999999999999997,"5min":1.371,"15min":54.427999999999997},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.375,"5min":0.36499999999999999,"15min":0.54500000000000004},"min":{"1min":0.23000000000000001,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.60699999999999998,"5min":1.23,"15min":54.356999999999999},"last":0.51100000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.54300000000000004,"5min":1.1220000000000001,"15min":2.4100000000000001},"last":0.316},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.38600000000000001},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.502,"5min":0.69799999999999995,"15min":2.0760000000000001},"last":0.25900000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:27:03 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.36799999999999999,"15min":0.40899999999999997},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.222},"max":{"1min":0.54600000000000004,"5min":1.4910000000000001,"15min":1.796},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.37,"5min":0.40500000000000003,"15min":0.45100000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.20999999999999999},"max":{"1min":0.53900000000000003,"5min":1.1339999999999999,"15min":1.7150000000000001},"last":0.46600000000000003}]},{"osd":4,"last update":"Mon Mar 15 22:27:12 2021","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.53000000000000003,"15min":0.56399999999999995},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.27700000000000002},"max":{"1min":0.69099999999999995,"5min":0.95199999999999996,"15min":3.3439999999999999},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.55300000000000005,"15min":0.61099999999999999},"min":{"1min":0.33700000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.80600000000000005,"5min":1.4590000000000001,"15min":4.6139999999999999},"last":0.52400000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:26:32 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.56499999999999995,"15min":0.61499999999999999},"min":{"1min":0.39800000000000002,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.68400000000000005,"5min":1.524,"15min":2.153},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55500000000000005,"5min":0.57999999999999996,"15min":0.91000000000000003},"min":{"1min":0.39500000000000002,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77200000000000002,"5min":1.1950000000000001,"15min":15.472},"last":0.69299999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:26:46 2021","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.60599999999999998,"15min":0.64600000000000002},"min":{"1min":0.41799999999999998,"5min":0.36899999999999999,"15min":0.33200000000000002},"max":{"1min":0.83799999999999997,"5min":1.276,"15min":1.75},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58099999999999996,"15min":0.61599999999999999},"min":{"1min":0.40500000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.82699999999999996,"5min":0.98299999999999998,"15min":1.3580000000000001},"last":0.55500000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:27:06 2021","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.629,"15min":0.81299999999999994},"min":{"1min":0.39500000000000002,"5min":0.377,"15min":0.316},"max":{"1min":0.85599999999999998,"5min":2.1219999999999999,"15min":8.8100000000000005},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.55800000000000005,"5min":0.58599999999999997,"15min":0.61199999999999999},"min":{"1min":0.38700000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83499999999999996,"5min":1.0329999999999999,"15min":1.325},"last":0.60099999999999998}]}]},{"osd":0,"up_from":0,"seq":34359738507,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":2,"up_from":18,"seq":77309411455,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140056,"kb_used_data":107664,"kb_used_omap":0,"kb_used_meta":32384,"kb_avail":93605096,"statfs":{"total":95995035648,"available":95851618304,"internally_reserved":0,"allocated":110247936,"data_stored":155725916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33161216},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.246,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.45700000000000002,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.437,"15min":0.39500000000000002},"min":{"1min":0.246,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":5.5250000000000004,"5min":5.5250000000000004,"15min":5.5250000000000004},"last":0.30599999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.42599999999999999,"15min":0.39600000000000002},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.60999999999999999,"5min":5.4489999999999998,"15min":5.4489999999999998},"last":0.373},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51800000000000002,"15min":0.46600000000000003},"min":{"1min":0.27400000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.3740000000000001,"5min":11.659000000000001,"15min":11.659000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:25:58 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.38400000000000001,"15min":0.41399999999999998},"min":{"1min":0.23699999999999999,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":1.2010000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.47699999999999998},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.24099999999999999},"max":{"1min":1.28,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:26:12 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.57299999999999995},"min":{"1min":0.38,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.3100000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.39100000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78400000000000003,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.755}]},{"osd":5,"last update":"Mon Mar 15 22:26:37 2021","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54800000000000004,"15min":0.57199999999999995},"min":{"1min":0.372,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.954},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.54400000000000004,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66000000000000003,"5min":0.85799999999999998,"15min":2.2919999999999998},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":1.875,"15min":1.875},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.435,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.3520000000000001,"5min":1.609,"15min":1.609},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:26:12 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56999999999999995,"15min":0.55800000000000005},"min":{"1min":0.38900000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.3320000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57799999999999996,"15min":0.58099999999999996},"min":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.38700000000000001},"max":{"1min":0.68400000000000005,"5min":1.923,"15min":1.923},"last":0.625}]}]},{"osd":3,"up_from":26,"seq":111669149825,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":292208,"kb_used_data":257192,"kb_used_omap":0,"kb_used_meta":35008,"kb_avail":93452944,"statfs":{"total":95995035648,"available":95695814656,"internally_reserved":0,"allocated":263364608,"data_stored":368855519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35848192},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:27:05 2021","interfaces":[{"interface":"back","average":{"1min":0.31900000000000001,"5min":0.34300000000000003,"15min":0.35099999999999998},"min":{"1min":0.22,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.58599999999999997,"5min":2.625,"15min":2.625},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.34300000000000003,"5min":0.36799999999999999,"15min":0.37},"min":{"1min":0.23300000000000001,"5min":0.20000000000000001,"15min":0.189},"max":{"1min":0.68100000000000005,"5min":2.6339999999999999,"15min":2.6339999999999999},"last":0.224}]},{"osd":1,"last update":"Mon Mar 15 22:27:05 2021","interfaces":[{"interface":"back","average":{"1min":0.38,"5min":0.44,"15min":0.42299999999999999},"min":{"1min":0.28100000000000003,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.626,"5min":6.0199999999999996,"15min":6.0199999999999996},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.36899999999999999,"5min":0.44900000000000001,"15min":0.42499999999999999},"min":{"1min":0.25,"5min":0.24199999999999999,"15min":0.22500000000000001},"max":{"1min":0.76800000000000002,"5min":7.6130000000000004,"15min":7.6130000000000004},"last":0.29199999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51800000000000002,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.42199999999999999,"15min":0.39900000000000002},"min":{"1min":0.27000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54100000000000004,"5min":7.0860000000000003,"15min":7.0860000000000003},"last":0.36599999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:26:13 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.55600000000000005,"15min":0.52600000000000002},"min":{"1min":0.31,"5min":0.31,"15min":0.31},"max":{"1min":0.72699999999999998,"5min":2.5920000000000001,"15min":2.5920000000000001},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.54900000000000004,"5min":0.54500000000000004,"15min":0.53300000000000003},"min":{"1min":0.34399999999999997,"5min":0.318,"15min":0.30399999999999999},"max":{"1min":1.0409999999999999,"5min":2.2890000000000001,"15min":2.2890000000000001},"last":0.39200000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:26:31 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.56699999999999995,"15min":0.54300000000000004},"min":{"1min":0.38400000000000001,"5min":0.36199999999999999,"15min":0.33400000000000002},"max":{"1min":0.65400000000000003,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.57499999999999996,"15min":0.56599999999999995},"min":{"1min":0.39900000000000002,"5min":0.36399999999999999,"15min":0.35899999999999999},"max":{"1min":0.77700000000000002,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":0.497}]},{"osd":6,"last update":"Mon Mar 15 22:26:45 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.59099999999999997,"15min":0.59299999999999997},"min":{"1min":0.42499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":0.75800000000000001,"5min":2.6499999999999999,"15min":2.6499999999999999},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.56999999999999995,"15min":0.57799999999999996},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.70599999999999996,"5min":1.268,"15min":1.268},"last":0.35599999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:27:05 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.57399999999999995,"15min":0.57299999999999995},"min":{"1min":0.35899999999999999,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.72799999999999998,"5min":2.7269999999999999,"15min":2.7269999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.58499999999999996,"15min":0.57399999999999995},"min":{"1min":0.44500000000000001,"5min":0.39500000000000002,"15min":0.377},"max":{"1min":0.753,"5min":2.0960000000000001,"15min":2.0960000000000001},"last":0.36699999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986302,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116716,"kb_used_data":98148,"kb_used_omap":0,"kb_used_meta":18560,"kb_avail":93628436,"statfs":{"total":95995035648,"available":95875518464,"internally_reserved":0,"allocated":100503552,"data_stored":131399764,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":19005440},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:26:18 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.44800000000000001,"15min":0.48699999999999999},"min":{"1min":0.29999999999999999,"5min":0.24199999999999999,"15min":0.19500000000000001},"max":{"1min":0.70599999999999996,"5min":0.74399999999999999,"15min":0.74399999999999999},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.47899999999999998,"15min":0.48799999999999999},"min":{"1min":0.32100000000000001,"5min":0.29099999999999998,"15min":0.251},"max":{"1min":0.66800000000000004,"5min":0.66800000000000004,"15min":0.74199999999999999},"last":0.34799999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:26:18 2021","interfaces":[{"interface":"back","average":{"1min":0.45100000000000001,"5min":0.46000000000000002,"15min":0.495},"min":{"1min":0.26300000000000001,"5min":0.25600000000000001,"15min":0.23599999999999999},"max":{"1min":0.56100000000000005,"5min":0.69499999999999995,"15min":1.631},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.505,"5min":0.48799999999999999,"15min":0.51600000000000001},"min":{"1min":0.30599999999999999,"5min":0.27300000000000002,"15min":0.26800000000000002},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":2.2989999999999999},"last":0.42499999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:26:18 2021","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48399999999999999,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.313},"max":{"1min":0.61299999999999999,"5min":0.70399999999999996,"15min":0.76100000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48299999999999998,"15min":0.502},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68799999999999994,"5min":0.71699999999999997,"15min":1.1910000000000001},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:26:18 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.51900000000000002,"15min":0.54700000000000004},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.98799999999999999},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.51500000000000001,"15min":0.55300000000000005},"min":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75,"5min":0.75,"15min":0.99399999999999999},"last":0.46000000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:26:31 2021","interfaces":[{"interface":"back","average":{"1min":0.432,"5min":0.45200000000000001,"15min":0.42099999999999999},"min":{"1min":0.248,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.58499999999999996,"5min":0.745,"15min":0.745},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.49299999999999999,"15min":0.44500000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.72399999999999998,"5min":0.79200000000000004,"15min":0.79200000000000004},"last":0.27100000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:26:47 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.47499999999999998,"15min":0.442},"min":{"1min":0.32300000000000001,"5min":0.26500000000000001,"15min":0.26500000000000001},"max":{"1min":0.68300000000000005,"5min":0.72599999999999998,"15min":0.76800000000000002},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54600000000000004,"15min":0.51900000000000002},"min":{"1min":0.42599999999999999,"5min":0.309,"15min":0.309},"max":{"1min":0.88700000000000001,"5min":0.88700000000000001,"15min":0.88700000000000001},"last":0.56699999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:27:04 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.503,"15min":0.49299999999999999},"min":{"1min":0.32600000000000001,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.745,"5min":0.745,"15min":0.80600000000000005},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.432,"5min":0.317,"15min":0.317},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.435}]}]},{"osd":5,"up_from":38,"seq":163208757371,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":204424,"kb_used_data":177088,"kb_used_omap":0,"kb_used_meta":27328,"kb_avail":93540728,"statfs":{"total":95995035648,"available":95785705472,"internally_reserved":0,"allocated":181338112,"data_stored":253950336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27983872},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:26:31 2021","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.505,"15min":0.48999999999999999},"min":{"1min":0.28999999999999998,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":2.9279999999999999,"5min":2.9279999999999999,"15min":2.9279999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.41399999999999998,"5min":0.47899999999999998,"15min":0.49199999999999999},"min":{"1min":0.32000000000000001,"5min":0.26500000000000001,"15min":0.26100000000000001},"max":{"1min":0.55900000000000005,"5min":0.85099999999999998,"15min":0.85099999999999998},"last":0.438}]},{"osd":1,"last update":"Mon Mar 15 22:26:31 2021","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.502,"15min":0.50800000000000001},"min":{"1min":0.27000000000000002,"5min":0.253,"15min":0.253},"max":{"1min":0.54400000000000004,"5min":2.665,"15min":2.665},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.50600000000000001,"15min":0.50600000000000001},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.27400000000000002},"max":{"1min":0.65000000000000002,"5min":0.81999999999999995,"15min":0.83999999999999997},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Mar 15 22:26:31 2021","interfaces":[{"interface":"back","average":{"1min":0.60899999999999999,"5min":0.54100000000000004,"15min":0.51600000000000001},"min":{"1min":0.35499999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":3.2639999999999998,"5min":3.2639999999999998,"15min":3.2639999999999998},"last":0.66000000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51700000000000002,"15min":0.51400000000000001},"min":{"1min":0.32200000000000001,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":1.137,"5min":1.137,"15min":1.3600000000000001},"last":0.51200000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:26:31 2021","interfaces":[{"interface":"back","average":{"1min":0.624,"5min":0.57099999999999995,"15min":0.52600000000000002},"min":{"1min":0.32600000000000001,"5min":0.312,"15min":0.308},"max":{"1min":3.2450000000000001,"5min":3.2450000000000001,"15min":3.2450000000000001},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.52300000000000002,"15min":0.52900000000000003},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68600000000000005,"5min":0.81399999999999995,"15min":1.4630000000000001},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:26:31 2021","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.45900000000000002,"15min":0.41099999999999998},"min":{"1min":0.245,"5min":0.245,"15min":0.24099999999999999},"max":{"1min":0.63800000000000001,"5min":0.73099999999999998,"15min":0.73099999999999998},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.59299999999999997,"5min":0.52300000000000002,"15min":0.45800000000000002},"min":{"1min":0.27800000000000002,"5min":0.27400000000000002,"15min":0.25},"max":{"1min":3.0630000000000002,"5min":3.0630000000000002,"15min":3.0630000000000002},"last":0.32500000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:26:50 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.496,"15min":0.47399999999999998},"min":{"1min":0.31900000000000001,"5min":0.29199999999999998,"15min":0.28499999999999998},"max":{"1min":0.75600000000000001,"5min":0.80700000000000005,"15min":0.80700000000000005},"last":0.311},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.56399999999999995,"15min":0.54000000000000004},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.32800000000000001},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.327},"last":0.38400000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:27:02 2021","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.55500000000000005,"15min":0.52700000000000002},"min":{"1min":0.376,"5min":0.35399999999999998,"15min":0.30499999999999999},"max":{"1min":1.198,"5min":1.198,"15min":1.3},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.38200000000000001,"5min":0.36599999999999999,"15min":0.307},"max":{"1min":1.6559999999999999,"5min":1.6559999999999999,"15min":1.6559999999999999},"last":0.36599999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":9520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":405504,"data_stored":14458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":8356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":6840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":245760,"data_stored":8774,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":5443584,"data_stored":5395029,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":167575552,"data_stored":232721725,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":105123840,"data_stored":152421018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":257753088,"data_stored":365293173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94937088,"data_stored":127823135,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":175849472,"data_stored":250367823,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":96043008,"data_stored":131146813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":118726656,"data_stored":166381347,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:27:17.427 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg force-backfill 4.1c 4.12 5.14 4.a 7.8 4.b 7.5 4.6 7.2 7.3 5.e 4.f 7.e 2021-03-15T22:27:17.429 INFO:tasks.rados.rados.0.smithi076.stdout:1976: writing smithi07694878-13 from 2625204 to 3217275 tid 3 2021-03-15T22:27:17.430 INFO:tasks.rados.rados.0.smithi076.stdout:1976: writing smithi07694878-13 from 3656107 to 3664058 tid 4 2021-03-15T22:27:17.430 INFO:tasks.rados.rados.0.smithi076.stdout:1977: delete oid 1 current snap is 222 2021-03-15T22:27:17.430 INFO:tasks.rados.rados.0.smithi076.stdout:1967: finishing write tid 4 to smithi07694878-29 2021-03-15T22:27:17.430 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 1409 (ObjNum 690 snap 222 seq_num 690) dirty exists 2021-03-15T22:27:17.431 INFO:tasks.rados.rados.0.smithi076.stdout:1967: left oid 29 (ObjNum 690 snap 222 seq_num 690) 2021-03-15T22:27:17.431 INFO:tasks.rados.rados.0.smithi076.stdout:1968: finishing write tid 1 to smithi07694878-32 2021-03-15T22:27:17.431 INFO:tasks.rados.rados.0.smithi076.stdout:1968: finishing write tid 2 to smithi07694878-32 2021-03-15T22:27:17.431 INFO:tasks.rados.rados.0.smithi076.stdout:1968: finishing write tid 3 to smithi07694878-32 2021-03-15T22:27:17.431 INFO:tasks.rados.rados.0.smithi076.stdout:1968: finishing write tid 4 to smithi07694878-32 2021-03-15T22:27:17.432 INFO:tasks.rados.rados.0.smithi076.stdout:1968: finishing write tid 5 to smithi07694878-32 2021-03-15T22:27:17.432 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 1414 (ObjNum 691 snap 222 seq_num 691) dirty exists 2021-03-15T22:27:17.432 INFO:tasks.rados.rados.0.smithi076.stdout:1968: left oid 32 (ObjNum 691 snap 222 seq_num 691) 2021-03-15T22:27:17.432 INFO:tasks.rados.rados.0.smithi076.stdout:1969: finishing write tid 1 to smithi07694878-8 2021-03-15T22:27:17.432 INFO:tasks.rados.rados.0.smithi076.stdout:1971: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:27:17.433 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 330 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:27:17.433 INFO:tasks.rados.rados.0.smithi076.stdout:1973: finishing rollback tid 0 to smithi07694878-24 2021-03-15T22:27:17.433 INFO:tasks.rados.rados.0.smithi076.stdout:1969: finishing write tid 2 to smithi07694878-8 2021-03-15T22:27:17.433 INFO:tasks.rados.rados.0.smithi076.stdout:1973: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:27:17.434 INFO:tasks.rados.rados.0.smithi076.stdout:1973: finishing rollback tid 2 to smithi07694878-24 2021-03-15T22:27:17.434 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 176 (ObjNum 638 snap 200 seq_num 638) dirty exists 2021-03-15T22:27:17.434 INFO:tasks.rados.rados.0.smithi076.stdout:1969: finishing write tid 3 to smithi07694878-8 2021-03-15T22:27:17.434 INFO:tasks.rados.rados.0.smithi076.stdout:1969: finishing write tid 4 to smithi07694878-8 2021-03-15T22:27:17.435 INFO:tasks.rados.rados.0.smithi076.stdout:1969: finishing write tid 5 to smithi07694878-8 2021-03-15T22:27:17.435 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1177 (ObjNum 692 snap 222 seq_num 692) dirty exists 2021-03-15T22:27:17.435 INFO:tasks.rados.rados.0.smithi076.stdout:1969: left oid 8 (ObjNum 692 snap 222 seq_num 692) 2021-03-15T22:27:17.435 INFO:tasks.rados.rados.0.smithi076.stdout:1974: expect (ObjNum 604 snap 188 seq_num 604) 2021-03-15T22:27:17.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:17.437 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:17.437 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:17.441 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:27:17.442 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:27:17.470 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:27:17.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-15T22:27:17.535 INFO:tasks.rados.rados.0.smithi076.stdout:1975: finishing write tid 1 to smithi07694878-38 2021-03-15T22:27:17.535 INFO:tasks.rados.rados.0.smithi076.stdout:1972: finishing copy_from to smithi07694878-33 2021-03-15T22:27:17.535 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 493 (ObjNum 640 snap 200 seq_num 640) dirty exists 2021-03-15T22:27:17.536 INFO:tasks.rados.rados.0.smithi076.stdout:1972: finishing copy_from racing read to smithi07694878-33 2021-03-15T22:27:17.536 INFO:tasks.rados.rados.0.smithi076.stdout:1975: finishing write tid 2 to smithi07694878-38 2021-03-15T22:27:17.536 INFO:tasks.rados.rados.0.smithi076.stdout:1975: finishing write tid 3 to smithi07694878-38 2021-03-15T22:27:17.537 INFO:tasks.rados.rados.0.smithi076.stdout:1975: finishing write tid 4 to smithi07694878-38 2021-03-15T22:27:17.537 INFO:tasks.rados.rados.0.smithi076.stdout:1970: finishing copy_from to smithi07694878-10 2021-03-15T22:27:17.537 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 1303 (ObjNum 654 snap 205 seq_num 654) dirty exists 2021-03-15T22:27:17.537 INFO:tasks.rados.rados.0.smithi076.stdout:1970: finishing copy_from racing read to smithi07694878-10 2021-03-15T22:27:17.538 INFO:tasks.rados.rados.0.smithi076.stdout:1976: finishing write tid 1 to smithi07694878-13 2021-03-15T22:27:17.538 INFO:tasks.rados.rados.0.smithi076.stdout:1975: finishing write tid 5 to smithi07694878-38 2021-03-15T22:27:17.538 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 1020 (ObjNum 693 snap 222 seq_num 693) dirty exists 2021-03-15T22:27:17.539 INFO:tasks.rados.rados.0.smithi076.stdout:1975: left oid 38 (ObjNum 693 snap 222 seq_num 693) 2021-03-15T22:27:17.539 INFO:tasks.rados.rados.0.smithi076.stdout:1976: finishing write tid 2 to smithi07694878-13 2021-03-15T22:27:17.539 INFO:tasks.rados.rados.0.smithi076.stdout:1967: done (10 left) 2021-03-15T22:27:17.539 INFO:tasks.rados.rados.0.smithi076.stdout:1968: done (9 left) 2021-03-15T22:27:17.540 INFO:tasks.rados.rados.0.smithi076.stdout:1969: done (8 left) 2021-03-15T22:27:17.540 INFO:tasks.rados.rados.0.smithi076.stdout:1970: done (7 left) 2021-03-15T22:27:17.540 INFO:tasks.rados.rados.0.smithi076.stdout:1971: done (6 left) 2021-03-15T22:27:17.541 INFO:tasks.rados.rados.0.smithi076.stdout:1972: done (5 left) 2021-03-15T22:27:17.541 INFO:tasks.rados.rados.0.smithi076.stdout:1973: done (4 left) 2021-03-15T22:27:17.541 INFO:tasks.rados.rados.0.smithi076.stdout:1974: done (3 left) 2021-03-15T22:27:17.541 INFO:tasks.rados.rados.0.smithi076.stdout:1975: done (2 left) 2021-03-15T22:27:17.542 INFO:tasks.rados.rados.0.smithi076.stdout:1977: done (1 left) 2021-03-15T22:27:17.542 INFO:tasks.rados.rados.0.smithi076.stdout:1978: rollback oid 23 current snap is 222 2021-03-15T22:27:17.542 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 23 to 214 2021-03-15T22:27:17.542 INFO:tasks.rados.rados.0.smithi076.stdout:1976: finishing write tid 3 to smithi07694878-13 2021-03-15T22:27:17.543 INFO:tasks.rados.rados.0.smithi076.stdout:1976: finishing write tid 4 to smithi07694878-13 2021-03-15T22:27:17.543 INFO:tasks.rados.rados.0.smithi076.stdout:1976: finishing write tid 5 to smithi07694878-13 2021-03-15T22:27:17.543 INFO:tasks.rados.rados.0.smithi076.stdout:1976: finishing write tid 6 to smithi07694878-13 2021-03-15T22:27:17.543 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 500 (ObjNum 694 snap 222 seq_num 694) dirty exists 2021-03-15T22:27:17.544 INFO:tasks.rados.rados.0.smithi076.stdout:1976: left oid 13 (ObjNum 694 snap 222 seq_num 694) 2021-03-15T22:27:17.544 INFO:tasks.rados.rados.0.smithi076.stdout:1976: done (1 left) 2021-03-15T22:27:17.544 INFO:tasks.rados.rados.0.smithi076.stdout:1979: write oid 10 current snap is 222 2021-03-15T22:27:17.544 INFO:tasks.rados.rados.0.smithi076.stdout:1979: seq_num 696 ranges {497672=561427,1795232=702575,2963697=730968} 2021-03-15T22:27:17.545 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:17.546 INFO:tasks.rados.rados.0.smithi076.stdout:1979: writing smithi07694878-10 from 497672 to 1059099 tid 1 2021-03-15T22:27:17.547 INFO:tasks.rados.rados.0.smithi076.stdout:1979: writing smithi07694878-10 from 1795232 to 2497807 tid 2 2021-03-15T22:27:17.555 INFO:tasks.rados.rados.0.smithi076.stdout:1979: writing smithi07694878-10 from 2963697 to 3694665 tid 3 2021-03-15T22:27:17.555 INFO:tasks.rados.rados.0.smithi076.stdout:1980: rollback oid 34 current snap is 222 2021-03-15T22:27:17.555 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 34 to 220 2021-03-15T22:27:17.556 INFO:tasks.rados.rados.0.smithi076.stdout:1978: finishing rollback tid 1 to smithi07694878-23 2021-03-15T22:27:17.556 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 805 (ObjNum 264853584 snap 21896 seq_num 261914944) dirty dne 2021-03-15T22:27:17.556 INFO:tasks.rados.rados.0.smithi076.stdout:1978: done (2 left) 2021-03-15T22:27:17.557 INFO:tasks.rados.rados.0.smithi076.stdout:1981: snap_create 2021-03-15T22:27:17.557 INFO:tasks.rados.rados.0.smithi076.stdout:1979: finishing write tid 1 to smithi07694878-10 2021-03-15T22:27:17.557 INFO:tasks.rados.rados.0.smithi076.stdout:1979: finishing write tid 2 to smithi07694878-10 2021-03-15T22:27:17.559 INFO:tasks.rados.rados.0.smithi076.stdout:1980: finishing rollback tid 0 to smithi07694878-34 2021-03-15T22:27:17.559 INFO:tasks.rados.rados.0.smithi076.stdout:1980: finishing rollback tid 1 to smithi07694878-34 2021-03-15T22:27:17.559 INFO:tasks.rados.rados.0.smithi076.stdout:1980: finishing rollback tid 2 to smithi07694878-34 2021-03-15T22:27:17.560 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 849 (ObjNum 609 snap 189 seq_num 609) dirty exists 2021-03-15T22:27:17.561 INFO:tasks.rados.rados.0.smithi076.stdout:1979: finishing write tid 3 to smithi07694878-10 2021-03-15T22:27:17.561 INFO:tasks.rados.rados.0.smithi076.stdout:1979: finishing write tid 4 to smithi07694878-10 2021-03-15T22:27:17.562 INFO:tasks.rados.rados.0.smithi076.stdout:1979: finishing write tid 5 to smithi07694878-10 2021-03-15T22:27:17.562 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 1307 (ObjNum 696 snap 222 seq_num 696) dirty exists 2021-03-15T22:27:17.562 INFO:tasks.rados.rados.0.smithi076.stdout:1979: left oid 10 (ObjNum 696 snap 222 seq_num 696) 2021-03-15T22:27:17.570 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:17.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:17.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:17.673 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:17.696 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:17.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:17.698 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:17.734 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:17.766 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:17.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:17 smithi076 conmon[56514]: cluster 2021-03-15T22:27:15.885670+0000 osd.4 (osd 2021-03-15T22:27:17.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:17 smithi076 conmon[56514]: .4) 110 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:27:17.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:17 smithi076 conmon[56514]: audit 2021-03-15T22:27:16.194194+0000 mgr.y (mgr.14144) 703 : audit [DBG] from='client.15702 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:27:17.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:17 smithi076 conmon[56514]: cluster 2021-03-15T22:27:16.309341+0000 mon.a (mon.0) 1455 : cluster [DBG] osdmap e521: 8 total, 7 up, 7 in 2021-03-15T22:27:17.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:17 smithi076 conmon[62156]: cluster 2021-03-15T22:27:15.885670+0000 osd.4 (osd.4) 110 : 2021-03-15T22:27:17.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:17 smithi076 conmon[62156]: cluster [DBG] 4.9 scrub ok 2021-03-15T22:27:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:17 smithi076 conmon[62156]: audit 2021-03-15T22:27:16.194194+0000 mgr.y (mgr.14144) 703 : audit [DBG] from='client.15702 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:27:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:17 smithi076 conmon[62156]: cluster 2021-03-15T22:27: 2021-03-15T22:27:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:17 smithi076 conmon[62156]: 16.309341+0000 mon.a (mon.0) 1455 : cluster [DBG] osdmap e521: 8 total, 7 up, 7 in 2021-03-15T22:27:17.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:17 smithi076 conmon[62156]: 2021-03-15T22:27:17.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:17 smithi125 conmon[56739]: cluster 2021-03-15T22:27:15.885670+0000 osd.4 (osd.4) 110 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:27:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:17 smithi125 conmon[56739]: audit 2021-03-15T22:27:16.194194+0000 mgr.y (mgr.14144) 703 : audit [DBG] from='client.15702 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:27:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:17 smithi125 conmon[56739]: cluster 2021-03-15T22:27:16.309341+0000 mon.a (mon.0) 1455 : cluster [DBG] osdmap e521: 8 total, 7 up, 7 in 2021-03-15T22:27:17.788 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:17.800 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:17.824 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:17.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:17.826 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:17.927 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:17.950 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:17.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:17.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:18.053 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:18.074 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:18.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:18.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:18.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:18.199 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:18.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:18.201 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:18.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:18.324 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:18.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:18.326 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:18.327 INFO:tasks.rados.rados.0.smithi076.stdout:1979: done (2 left) 2021-03-15T22:27:18.327 INFO:tasks.rados.rados.0.smithi076.stdout:1980: done (1 left) 2021-03-15T22:27:18.328 INFO:tasks.rados.rados.0.smithi076.stdout:1981: done (0 left) 2021-03-15T22:27:18.328 INFO:tasks.rados.rados.0.smithi076.stdout:1982: read oid 26 snap -1 2021-03-15T22:27:18.328 INFO:tasks.rados.rados.0.smithi076.stdout:1982: expect (ObjNum 667 snap 211 seq_num 667) 2021-03-15T22:27:18.328 INFO:tasks.rados.rados.0.smithi076.stdout:1983: read oid 15 snap 207 2021-03-15T22:27:18.329 INFO:tasks.rados.rados.0.smithi076.stdout:1983: expect (ObjNum 649 snap 201 seq_num 649) 2021-03-15T22:27:18.329 INFO:tasks.rados.rados.0.smithi076.stdout:1984: snap_create 2021-03-15T22:27:18.333 INFO:tasks.rados.rados.0.smithi076.stdout:1982: expect (ObjNum 667 snap 211 seq_num 667) 2021-03-15T22:27:18.427 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:18.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:18.454 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:18.455 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:18.468 INFO:tasks.rados.rados.0.smithi076.stdout:1983: expect (ObjNum 649 snap 201 seq_num 649) 2021-03-15T22:27:18.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:18 smithi076 conmon[56514]: cluster 2021-03-15T22:27:17.082312+0000 mgr.y (mgr.14144) 704 : cluster [DBG] pgmap v909: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 488 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 204/1629 objects degraded (12.523%) 2021-03-15T22:27:18.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:18 smithi076 conmon[56514]: cluster 2021-03-15T22:27:17.315685+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e522: 8 total, 7 up, 7 in 2021-03-15T22:27:18.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:18 smithi076 conmon[56514]: cluster 2021-03-15T22:27:17.334454+0000 osd.5 (osd.5) 93 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:18.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:18 smithi076 conmon[56514]: cluster 2021-03-15T22:27:17.335401+0000 osd.5 (osd.5) 94 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:18.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:18 smithi076 conmon[56514]: cluster 2021-03-15T22:27:17.522625+0000 osd.5 (osd.5) 95 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:27:18.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:18 smithi076 conmon[62156]: cluster 2021-03-15T22:27:17.082312+0000 mgr.y (mgr.14144) 704 : cluster [DBG] pgmap v909: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 488 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 204/1629 objects degraded (12.523%) 2021-03-15T22:27:18.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:18 smithi076 conmon[62156]: cluster 2021-03-15T22:27:17.315685+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e522: 8 total, 7 up, 7 in 2021-03-15T22:27:18.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:18 smithi076 conmon[62156]: cluster 2021-03-15T22:27:17.334454+0000 osd.5 (osd.5) 93 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:18.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:18 smithi076 conmon[62156]: cluster 2021-03-15T22:27:17.335401+0000 osd.5 (osd.5) 94 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:18.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:18 smithi076 conmon[62156]: cluster 2021-03-15T22:27:17.522625+0000 osd.5 (osd.5) 95 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:27:18.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:18 smithi125 conmon[56739]: cluster 2021-03-15T22:27:17.082312+0000 mgr.y (mgr.14144) 704 : cluster [DBG] pgmap v909: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 488 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 204/1629 objects degraded (12.523%) 2021-03-15T22:27:18.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:18 smithi125 conmon[56739]: cluster 2021-03-15T22:27:17.315685+0000 mon.a (mon.0) 1456 : cluster [DBG] osdmap e522: 8 total, 7 up, 7 in 2021-03-15T22:27:18.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:18 smithi125 conmon[56739]: cluster 2021-03-15T22:27:17.334454+0000 osd.5 (osd.5) 93 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:18.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:18 smithi125 conmon[56739]: cluster 2021-03-15T22:27:17.335401+0000 osd.5 (osd.5) 94 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:18.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:18 smithi125 conmon[56739]: cluster 2021-03-15T22:27:17.522625+0000 osd.5 (osd.5) 95 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:27:18.557 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:18.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:18.580 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:18.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:18.682 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:18.712 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:18.713 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:18.814 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:18.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:18.858 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:18.858 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:18.959 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:18.987 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:18.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:18.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:19.090 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:19.121 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:19.122 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:19.223 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:19.247 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:19.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:19.248 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:19.290 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:27:19.291 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:27:19.329 INFO:tasks.rados.rados.0.smithi076.stdout:1982: done (2 left) 2021-03-15T22:27:19.329 INFO:tasks.rados.rados.0.smithi076.stdout:1983: done (1 left) 2021-03-15T22:27:19.330 INFO:tasks.rados.rados.0.smithi076.stdout:1984: done (0 left) 2021-03-15T22:27:19.330 INFO:tasks.rados.rados.0.smithi076.stdout:1985: snap_remove snap 215 2021-03-15T22:27:19.350 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:19.377 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:19.377 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:19.443 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:27:19.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:19.500 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:19.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:19.502 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:19.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:19 smithi125 conmon[56739]: cluster 2021-03-15T22:27:17.909056+0000 osd.4 (osd.4) 111 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:27:19.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:19 smithi125 conmon[56739]: cluster 2021-03-15T22:27:18.325457+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e523: 8 total, 7 up, 7 in 2021-03-15T22:27:19.602 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:19.624 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:19.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:19.626 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:19.727 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:19.748 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:19.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:19.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:19.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:19 smithi076 conmon[56514]: cluster 2021-03-15T22:27:17.909056+0000 2021-03-15T22:27:19.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:19 smithi076 conmon[56514]: osd.4 (osd.4) 111 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:27:19.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:19 smithi076 conmon[56514]: cluster 2021-03-15T22:27:18.325457+0000 mon.a (mon.0) 1457 : cluster [DBG] osdmap e523: 8 total, 7 up, 7 in 2021-03-15T22:27:19.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:19 smithi076 conmon[62156]: cluster 2021-03-15T22:27:17.909056+0000 osd.4 (osd. 2021-03-15T22:27:19.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:19 smithi076 conmon[62156]: 4) 111 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:27:19.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:19 smithi076 conmon[62156]: cluster 2021-03-15T22:27:18.325457+0000 mon.a (mon 2021-03-15T22:27:19.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:19 smithi076 conmon[62156]: .0) 1457 : cluster [DBG] osdmap e523: 8 total, 7 up, 7 in 2021-03-15T22:27:19.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:19.872 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:19.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:19.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:19.891 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-15T22:27:19.976 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:19.999 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:20.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:20.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:20.079 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:20.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:20.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:20.125 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:20.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:20.144 INFO:tasks.rados.rados.0.smithi076.stdout:1985: done (0 left) 2021-03-15T22:27:20.144 INFO:tasks.rados.rados.0.smithi076.stdout:1986: rollback oid 47 current snap is 224 2021-03-15T22:27:20.144 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 47 to 222 2021-03-15T22:27:20.145 INFO:tasks.rados.rados.0.smithi076.stdout:1987: write (excl) oid 30 current snap is 224 2021-03-15T22:27:20.145 INFO:tasks.rados.rados.0.smithi076.stdout:1987: seq_num 697 ranges {514479=636676,1853714=446233,2991120=542240} 2021-03-15T22:27:20.159 INFO:tasks.rados.rados.0.smithi076.stdout:1987: writing smithi07694878-30 from 514479 to 1151155 tid 1 2021-03-15T22:27:20.159 INFO:tasks.rados.rados.0.smithi076.stdout:1987: writing smithi07694878-30 from 1853714 to 2299947 tid 2 2021-03-15T22:27:20.168 INFO:tasks.rados.rados.0.smithi076.stdout:1987: writing smithi07694878-30 from 2991120 to 3533360 tid 3 2021-03-15T22:27:20.168 INFO:tasks.rados.rados.0.smithi076.stdout:1986: finishing rollback tid 0 to smithi07694878-47 2021-03-15T22:27:20.168 INFO:tasks.rados.rados.0.smithi076.stdout:1986: finishing rollback tid 1 to smithi07694878-47 2021-03-15T22:27:20.168 INFO:tasks.rados.rados.0.smithi076.stdout:1986: finishing rollback tid 2 to smithi07694878-47 2021-03-15T22:27:20.169 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1180 (ObjNum 669 snap 212 seq_num 669) dirty exists 2021-03-15T22:27:20.169 INFO:tasks.rados.rados.0.smithi076.stdout:1986: done (1 left) 2021-03-15T22:27:20.169 INFO:tasks.rados.rados.0.smithi076.stdout:1988: delete oid 25 current snap is 224 2021-03-15T22:27:20.182 INFO:tasks.rados.rados.0.smithi076.stdout:1988: done (1 left) 2021-03-15T22:27:20.183 INFO:tasks.rados.rados.0.smithi076.stdout:1989: rollback oid 44 current snap is 224 2021-03-15T22:27:20.183 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 44 to 191 2021-03-15T22:27:20.183 INFO:tasks.rados.rados.0.smithi076.stdout:1990: snap_create 2021-03-15T22:27:20.185 INFO:tasks.rados.rados.0.smithi076.stdout:1987: finishing write tid 1 to smithi07694878-30 2021-03-15T22:27:20.191 INFO:tasks.rados.rados.0.smithi076.stdout:1987: finishing write tid 2 to smithi07694878-30 2021-03-15T22:27:20.191 INFO:tasks.rados.rados.0.smithi076.stdout:1989: finishing rollback tid 0 to smithi07694878-44 2021-03-15T22:27:20.192 INFO:tasks.rados.rados.0.smithi076.stdout:1989: finishing rollback tid 1 to smithi07694878-44 2021-03-15T22:27:20.193 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1182 (ObjNum 265101392 snap 21896 seq_num 265162352) dirty dne 2021-03-15T22:27:20.200 INFO:tasks.rados.rados.0.smithi076.stdout:1987: finishing write tid 3 to smithi07694878-30 2021-03-15T22:27:20.200 INFO:tasks.rados.rados.0.smithi076.stdout:1987: finishing write tid 4 to smithi07694878-30 2021-03-15T22:27:20.201 INFO:tasks.rados.rados.0.smithi076.stdout:1987: finishing write tid 5 to smithi07694878-30 2021-03-15T22:27:20.201 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 1297 (ObjNum 697 snap 224 seq_num 697) dirty exists 2021-03-15T22:27:20.201 INFO:tasks.rados.rados.0.smithi076.stdout:1987: left oid 30 (ObjNum 697 snap 224 seq_num 697) 2021-03-15T22:27:20.227 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:20.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:20.265 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:20.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:20.267 INFO:teuthology.orchestra.run.smithi076.stderr:instructing pg(s) [4.6,4.12,4.1c] on osd.1 to force-backfill; instructing pg(s) [4.f,5.14,7.2,7.3] on osd.3 to force-backfill; instructing pg(s) [4.b,5.e] on osd.4 to force-backfill; instructing pg(s) [7.e] on osd.5 to force-backfill; instructing pg(s) [4.a] on osd.6 to force-backfill; instructing pg(s) [7.5,7.8] on osd.7 to force-backfill; 2021-03-15T22:27:20.267 INFO:teuthology.orchestra.run.smithi076.stderr: 2021-03-15T22:27:20.268 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:20.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:27:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:27:20.334 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[56514]: cluster 2021-03-15T22:27:18.885977+0000 osd.3 (osd.3) 89 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:27:20.334 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[62156]: cluster 2021-03-15T22:27:18. 2021-03-15T22:27:20.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[62156]: 885977+0000 osd.3 (osd.3) 89 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:27:20.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[62156]: cluster 2021-03-15T22:27:19.082804+0000 mgr.y (mgr.14144) 705 : cluster [DBG] pgmap v912: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 493 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 23 op/s; 205/1635 objects degraded (12.538%) 2021-03-15T22:27:20.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[62156]: cluster 2021-03-15T22:27:19.324060+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e524: 8 total, 7 up, 7 in 2021-03-15T22:27:20.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[62156]: audit 2021-03-15T22:27:19.815889+0000 mgr.y (mgr.14144) 706 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:20.336 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[62156]: cluster 2021-03-15T22:27:20.135664+0000 mon.a (mon.0) 1459 : cluster [WRN] Health check update: Degraded data redundancy: 205/1635 objects degraded (12.538%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:20.336 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[62156]: cluster 2021-03-15T22:27:20.140633+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e525: 8 total, 7 up, 7 in 2021-03-15T22:27:20.367 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:20.389 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:20.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:20.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:20.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:20.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:20.519 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:20.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:20.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:20.633 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[56514]: cluster 2021-03-15T22:27:19.082804+0000 mgr.y (mgr.14144) 705 : cluster [DBG] pgmap v912: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 493 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 23 op/s; 205/1635 objects degraded (12.538%) 2021-03-15T22:27:20.633 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[56514]: cluster 2021-03-15T22:27:19.324060+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e524: 8 total, 7 up, 7 in 2021-03-15T22:27:20.634 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[56514]: audit 2021-03-15T22:27:19.815889+0000 mgr.y (mgr.14144) 706 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:20.634 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[56514]: cluster 2021-03-15T22:27:20.135664+0000 mon.a (mon.0) 1459 : cluster [WRN] Health check update: Degraded data redundancy: 205/1635 objects degraded (12.538%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:20.634 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:20 smithi076 conmon[56514]: cluster 2021-03-15T22:27:20.140633+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e525: 8 total, 7 up, 7 in 2021-03-15T22:27:20.643 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:20.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:20.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:20.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:20 smithi125 conmon[56739]: cluster 2021-03-15T22:27:18.885977+0000 osd.3 (osd.3) 89 2021-03-15T22:27:20.654 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:20 smithi125 conmon[56739]: : cluster [DBG] 4.14 scrub ok 2021-03-15T22:27:20.654 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:20 smithi125 conmon[56739]: cluster 2021-03-15T22:27:19.082804+0000 mgr.y (mgr.14144) 705 : cluster [DBG] pgmap v912: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 493 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 16 MiB/s rd, 3.0 MiB/s wr, 23 op/s; 205/1635 objects degraded (12.538%) 2021-03-15T22:27:20.654 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:20 smithi125 conmon[56739]: cluster 2021-03-15T22:27:19.324060+0000 mon.a (mon.0) 1458 : cluster [DBG] osdmap e524: 8 total, 7 up, 7 in 2021-03-15T22:27:20.654 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:20 smithi125 conmon[56739]: audit 2021-03-15T22:27:19.815889+0000 mgr.y (mgr.14144) 706 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:20.655 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:20 smithi125 conmon[56739]: cluster 2021-03-15T22:27:20.135664+0000 mon.a (mon.0) 1459 : cluster [WRN] Health check update: Degraded data redundancy: 205/1635 objects degraded (12.538%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:20.655 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:20 smithi125 conmon[56739]: cluster 2021-03-15T22:27:20.140633+0000 mon.a (mon.0) 1460 : cluster [DBG] osdmap e525: 8 total, 7 up, 7 in 2021-03-15T22:27:20.746 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:20.769 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:20.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:20.770 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:20.871 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:20.894 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:20.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:20.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:20.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:21.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:21.021 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:21.021 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:21.122 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:21.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:21.147 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:21.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:21.150 INFO:tasks.rados.rados.0.smithi076.stdout:1987: done (2 left) 2021-03-15T22:27:21.151 INFO:tasks.rados.rados.0.smithi076.stdout:1989: done (1 left) 2021-03-15T22:27:21.151 INFO:tasks.rados.rados.0.smithi076.stdout:1990: done (0 left) 2021-03-15T22:27:21.151 INFO:tasks.rados.rados.0.smithi076.stdout:1991: copy_from oid 27 from oid 4 current snap is 225 2021-03-15T22:27:21.152 INFO:tasks.rados.rados.0.smithi076.stdout:1992: write oid 16 current snap is 225 2021-03-15T22:27:21.152 INFO:tasks.rados.rados.0.smithi076.stdout:1992: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2021-03-15T22:27:21.153 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:27:21.158 INFO:tasks.rados.rados.0.smithi076.stdout:1992: writing smithi07694878-16 from 548093 to 1335267 tid 1 2021-03-15T22:27:21.165 INFO:tasks.rados.rados.0.smithi076.stdout:1992: writing smithi07694878-16 from 1970678 to 2420580 tid 2 2021-03-15T22:27:21.173 INFO:tasks.rados.rados.0.smithi076.stdout:1992: writing smithi07694878-16 from 3045966 to 3727103 tid 3 2021-03-15T22:27:21.173 INFO:tasks.rados.rados.0.smithi076.stdout:1992: finishing write tid 1 to smithi07694878-16 2021-03-15T22:27:21.174 INFO:tasks.rados.rados.0.smithi076.stdout:1991: finishing copy_from to smithi07694878-27 2021-03-15T22:27:21.174 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 449 (ObjNum 624 snap 196 seq_num 624) dirty exists 2021-03-15T22:27:21.174 INFO:tasks.rados.rados.0.smithi076.stdout:1992: finishing write tid 2 to smithi07694878-16 2021-03-15T22:27:21.174 INFO:tasks.rados.rados.0.smithi076.stdout:1991: finishing copy_from racing read to smithi07694878-27 2021-03-15T22:27:21.175 INFO:tasks.rados.rados.0.smithi076.stdout:1991: done (1 left) 2021-03-15T22:27:21.175 INFO:tasks.rados.rados.0.smithi076.stdout:1993: read oid 15 snap -1 2021-03-15T22:27:21.175 INFO:tasks.rados.rados.0.smithi076.stdout:1993: expect (ObjNum 649 snap 201 seq_num 649) 2021-03-15T22:27:21.175 INFO:tasks.rados.rados.0.smithi076.stdout:1994: read oid 23 snap -1 2021-03-15T22:27:21.176 INFO:tasks.rados.rados.0.smithi076.stdout:1994: expect deleted 2021-03-15T22:27:21.176 INFO:tasks.rados.rados.0.smithi076.stdout:1995: read oid 14 snap 185 2021-03-15T22:27:21.176 INFO:tasks.rados.rados.0.smithi076.stdout:1995: expect (ObjNum 593 snap 183 seq_num 593) 2021-03-15T22:27:21.176 INFO:tasks.rados.rados.0.smithi076.stdout:1996: read oid 46 snap -1 2021-03-15T22:27:21.177 INFO:tasks.rados.rados.0.smithi076.stdout:1996: expect (ObjNum 679 snap 216 seq_num 679) 2021-03-15T22:27:21.177 INFO:tasks.rados.rados.0.smithi076.stdout:1997: snap_create 2021-03-15T22:27:21.181 INFO:tasks.rados.rados.0.smithi076.stdout:1995: expect (ObjNum 593 snap 183 seq_num 593) 2021-03-15T22:27:21.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:21.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:21.271 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:21.272 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:21.318 INFO:tasks.rados.rados.0.smithi076.stdout:1996: expect (ObjNum 679 snap 216 seq_num 679) 2021-03-15T22:27:21.373 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:21.396 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:21.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:21.398 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:21.499 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:21.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[56514]: cluster 2021-03-15T22:27:19.901198+0000 osd.3 (osd. 2021-03-15T22:27:21.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[56514]: 3) 90 : cluster [DBG] 3.b deep-scrub ok 2021-03-15T22:27:21.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[56514]: audit 2021-03-15T22:27:20.259440+0000 mgr.y (mgr.14144) 707 : audit [DBG] from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["4.1c", "4.12", "5.14", "4.a", "7.8", "4.b", "7.5", "4.6", "7.2", "7.3", "5.e", "4.f", "7.e"], "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:27:21.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[56514]: audit 2021-03-15T22:27:20.584305+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? 172.21.15.76:0/537387067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:27:21.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[56514]: cluster 2021-03-15T22:27:21.142238+0000 mon.a (mon.0) 1462 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:21.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[56514]: audit 2021-03-15T22:27:21.148342+0000 mon.a (mon.0) 1463 : audit [INF] from='client.? 172.21.15.76:0/537387067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:27:21.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[56514]: cluster 2021-03-15T22:27:21.148407+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e526: 8 total, 7 up, 7 in 2021-03-15T22:27:21.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: cluster 2021-03-15T22:27:19.901198 2021-03-15T22:27:21.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: +0000 osd.3 (osd.3) 90 : cluster [DBG] 3.b deep-scrub ok 2021-03-15T22:27:21.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:27:21.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: T22:27:20.259440+0000 mgr.y (mgr.14144) 707 : audit [DBG] 2021-03-15T22:27:21.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["4.1c", "4.12", "5.14", "4.a", "7.8", "4.b", "7.5", "4.6", "7.2", "7.3", "5.e", "4.f", "7.e"], "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:27:21.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: audit 2021-03-15T22:27 2021-03-15T22:27:21.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: :20.584305+0000 mon.a (mon.0) 1461 2021-03-15T22:27:21.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/537387067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:27:21.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:27:21.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: 22:27:21.142238+0000 mon.a (mon. 2021-03-15T22:27:21.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: 0) 1462 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:21.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: audit 2021 2021-03-15T22:27:21.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: -03-15T22:27:21.148342 2021-03-15T22:27:21.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1463 : audit [INF] 2021-03-15T22:27:21.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/537387067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:27:21.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:27:21.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: 27:21.148407+0000 mon.a (mon.0) 2021-03-15T22:27:21.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:21 smithi076 conmon[62156]: 1464 : cluster [DBG] osdmap e526: 8 total, 7 up, 7 in 2021-03-15T22:27:21.524 INFO:tasks.rados.rados.0.smithi076.stdout:1993: expect (ObjNum 649 snap 201 seq_num 649) 2021-03-15T22:27:21.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:21.525 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:21.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:21.603 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:21 smithi125 conmon[56739]: cluster 2021-03-15T22:27:19.901198+0000 osd.3 (osd.3 2021-03-15T22:27:21.604 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:21 smithi125 conmon[56739]: ) 90 : cluster [DBG] 3.b deep-scrub ok 2021-03-15T22:27:21.605 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:21 smithi125 conmon[56739]: audit 2021-03-15T22:27:20.259440+0000 mgr.y (mgr.14144) 707 : audit [DBG] from='client.15714 -' entity='client.admin' cmd=[{"prefix": "pg force-backfill", "pgid": ["4.1c", "4.12", "5.14", "4.a", "7.8", "4.b", "7.5", "4.6", "7.2", "7.3", "5.e", "4.f", "7.e"], "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:27:21.605 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:21 smithi125 conmon[56739]: audit 2021-03-15T22:27:20.584305+0000 mon.a (mon.0) 1461 : audit [INF] from='client.? 172.21.15.76:0/537387067' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:27:21.605 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:21 smithi125 conmon[56739]: cluster 2021-03-15T22:27:21.142238+0000 mon.a (mon.0) 1462 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:21.606 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:21 smithi125 conmon[56739]: audit 2021-03-15T22:27:21.148342+0000 mon.a (mon.0) 1463 : audit [INF] from='client.? 172.21.15.76:0/537387067' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:27:21.606 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:21 smithi125 conmon[56739]: cluster 2021-03-15T22:27:21.148407+0000 mon.a (mon.0) 1464 : cluster [DBG] osdmap e526: 8 total, 7 up, 7 in 2021-03-15T22:27:21.626 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:21.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:21.653 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:21.653 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:21.681 INFO:tasks.rados.rados.0.smithi076.stdout:1992: finishing write tid 3 to smithi07694878-16 2021-03-15T22:27:21.682 INFO:tasks.rados.rados.0.smithi076.stdout:1992: finishing write tid 4 to smithi07694878-16 2021-03-15T22:27:21.682 INFO:tasks.rados.rados.0.smithi076.stdout:1992: finishing write tid 5 to smithi07694878-16 2021-03-15T22:27:21.683 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1302 (ObjNum 699 snap 225 seq_num 699) dirty exists 2021-03-15T22:27:21.683 INFO:tasks.rados.rados.0.smithi076.stdout:1992: left oid 16 (ObjNum 699 snap 225 seq_num 699) 2021-03-15T22:27:21.754 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:21.776 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:21.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:21.778 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:21.801 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:27:21.802 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:27:21.879 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:21.900 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:21.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:21.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:22.003 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:22.025 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:22.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:22.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:22.128 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:22.151 INFO:tasks.rados.rados.0.smithi076.stdout:1992: done (5 left) 2021-03-15T22:27:22.151 INFO:tasks.rados.rados.0.smithi076.stdout:1993: done (4 left) 2021-03-15T22:27:22.152 INFO:tasks.rados.rados.0.smithi076.stdout:1994: done (3 left) 2021-03-15T22:27:22.152 INFO:tasks.rados.rados.0.smithi076.stdout:1995: done (2 left) 2021-03-15T22:27:22.152 INFO:tasks.rados.rados.0.smithi076.stdout:1996: done (1 left) 2021-03-15T22:27:22.152 INFO:tasks.rados.rados.0.smithi076.stdout:1997: done (0 left) 2021-03-15T22:27:22.153 INFO:tasks.rados.rados.0.smithi076.stdout:1998: delete oid 8 current snap is 226 2021-03-15T22:27:22.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:22.154 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:22.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:22.158 INFO:tasks.rados.rados.0.smithi076.stdout:1998: done (0 left) 2021-03-15T22:27:22.158 INFO:tasks.rados.rados.0.smithi076.stdout:1999: rollback oid 41 current snap is 226 2021-03-15T22:27:22.159 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 41 to 223 2021-03-15T22:27:22.159 INFO:tasks.rados.rados.0.smithi076.stdout:2000: delete oid 46 current snap is 226 2021-03-15T22:27:22.163 INFO:tasks.rados.rados.0.smithi076.stdout:2000: done (1 left) 2021-03-15T22:27:22.164 INFO:tasks.rados.rados.0.smithi076.stdout:2001: read oid 6 snap -1 2021-03-15T22:27:22.164 INFO:tasks.rados.rados.0.smithi076.stdout:2001: expect deleted 2021-03-15T22:27:22.164 INFO:tasks.rados.rados.0.smithi076.stdout:2002: write oid 11 current snap is 226 2021-03-15T22:27:22.164 INFO:tasks.rados.rados.0.smithi076.stdout:2002: seq_num 702 ranges {598514=729274,1746124=597229,3128235=514953} 2021-03-15T22:27:22.171 INFO:tasks.rados.rados.0.smithi076.stdout:2002: writing smithi07694878-11 from 598514 to 1327788 tid 1 2021-03-15T22:27:22.178 INFO:tasks.rados.rados.0.smithi076.stdout:2002: writing smithi07694878-11 from 1746124 to 2343353 tid 2 2021-03-15T22:27:22.186 INFO:tasks.rados.rados.0.smithi076.stdout:2002: writing smithi07694878-11 from 3128235 to 3643188 tid 3 2021-03-15T22:27:22.187 INFO:tasks.rados.rados.0.smithi076.stdout:2003: snap_remove snap 160 2021-03-15T22:27:22.256 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:22.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:22.279 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:22.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:22.381 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:22.403 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:22.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:22.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:22.448 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-15T22:27:22.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:22 smithi076 conmon[56514]: cluster 2021-03-15T22:27:21. 2021-03-15T22:27:22.455 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:22 smithi076 conmon[56514]: 083241+0000 mgr.y (mgr.14144) 708 : cluster [DBG] pgmap v915: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 499 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 20 op/s; 206/1650 objects degraded (12.485%) 2021-03-15T22:27:22.455 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:22 smithi076 conmon[56514]: cluster 2021-03-15T22:27:22.147536+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e527: 8 total, 7 up, 7 in 2021-03-15T22:27:22.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:22 smithi076 conmon[62156]: cluster 2021-03-15T22:27:21.083241+0000 mgr.y 2021-03-15T22:27:22.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:22 smithi076 conmon[62156]: (mgr.14144) 708 : cluster [DBG] pgmap v915: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 499 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 20 op/s; 206/1650 objects degraded (12.485%) 2021-03-15T22:27:22.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:22 smithi076 conmon[62156]: cluster 2021-03-15T22:27:22.147536+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e527: 8 total, 7 up, 7 in 2021-03-15T22:27:22.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:22.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:22.529 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:22.529 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:22.631 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:22.653 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:22.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:22.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:22.659 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:27:22.676 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:27:22.692 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:27:22.709 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:27:22.724 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:22.727 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:27:22.745 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:27:22.756 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:22.762 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:27:22.762 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:27:22.762 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:27:22.763 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:27:22.763 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:27:22.763 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:27:22.763 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:27:22.763 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:27:22.764 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:27:22.780 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:22.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:22.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:22.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:22 smithi125 conmon[56739]: cluster 2021-03-15T22:27:21.083241+0000 mgr.y (mgr.14144) 708 : cluster [DBG] pgmap v915: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 499 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 20 op/s; 206/1650 objects degraded (12.485%) 2021-03-15T22:27:22.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:22 smithi125 conmon[56739]: cluster 2021-03-15T22:27:22.147536+0000 mon.a (mon.0) 1465 : cluster [DBG] osdmap e527: 8 total, 7 up, 7 in 2021-03-15T22:27:22.883 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:22.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:22.906 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:22.907 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:23.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:23.030 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:23.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:23.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:23.132 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:23.157 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:23.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:23.158 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:23.159 INFO:tasks.rados.rados.0.smithi076.stdout:2003: done (3 left) 2021-03-15T22:27:23.160 INFO:tasks.rados.rados.0.smithi076.stdout:2004: snap_create 2021-03-15T22:27:23.160 INFO:tasks.rados.rados.0.smithi076.stdout:1999: finishing rollback tid 0 to smithi07694878-41 2021-03-15T22:27:23.160 INFO:tasks.rados.rados.0.smithi076.stdout:1999: finishing rollback tid 1 to smithi07694878-41 2021-03-15T22:27:23.160 INFO:tasks.rados.rados.0.smithi076.stdout:1999: finishing rollback tid 2 to smithi07694878-41 2021-03-15T22:27:23.161 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 1181 (ObjNum 689 snap 220 seq_num 689) dirty exists 2021-03-15T22:27:23.161 INFO:tasks.rados.rados.0.smithi076.stdout:2002: finishing write tid 1 to smithi07694878-11 2021-03-15T22:27:23.161 INFO:tasks.rados.rados.0.smithi076.stdout:2002: finishing write tid 2 to smithi07694878-11 2021-03-15T22:27:23.161 INFO:tasks.rados.rados.0.smithi076.stdout:2002: finishing write tid 3 to smithi07694878-11 2021-03-15T22:27:23.161 INFO:tasks.rados.rados.0.smithi076.stdout:2002: finishing write tid 4 to smithi07694878-11 2021-03-15T22:27:23.162 INFO:tasks.rados.rados.0.smithi076.stdout:2002: finishing write tid 5 to smithi07694878-11 2021-03-15T22:27:23.162 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 506 (ObjNum 702 snap 226 seq_num 702) dirty exists 2021-03-15T22:27:23.162 INFO:tasks.rados.rados.0.smithi076.stdout:2002: left oid 11 (ObjNum 702 snap 226 seq_num 702) 2021-03-15T22:27:23.260 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:23.282 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:23.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:23.284 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:23.385 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:23.406 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:23.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:23.408 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:23.509 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:23.531 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:23.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:23.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:23.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:23 smithi125 conmon[56739]: audit 2021-03-15T22:27:22.640684+0000 mon.a (mon. 2021-03-15T22:27:23.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:23 smithi125 conmon[56739]: 0) 1466 : audit [DBG] from='client.? 172.21.15.76:0/1212468998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:23.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:23 smithi125 conmon[56739]: cluster 2021-03-15T22:27:23.153888+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e528: 8 total, 7 up, 7 in 2021-03-15T22:27:23.597 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:23 smithi076 conmon[56514]: audit 2021-03-15T22:27:22.640684+0000 mon.a (mon.0) 1466 : audit [DBG] from='client.? 172.21.15.76:0/1212468998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:23.598 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:23 smithi076 conmon[56514]: cluster 2021-03-15T22:27:23.153888+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e528: 8 total, 7 up, 7 in 2021-03-15T22:27:23.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:23 smithi076 conmon[62156]: audit 2021-03-15T22:27:22.640684+0000 mon.a (mon.0) 1466 : audit [DBG] from='client.? 172.21.15.76:0/1212468998' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:23.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:23 smithi076 conmon[62156]: cluster 2021-03-15T22:27:23.153888+0000 mon.a (mon.0) 1467 : cluster [DBG] osdmap e528: 8 total, 7 up, 7 in 2021-03-15T22:27:23.599 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:27:23.600 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:27:23.635 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:23.657 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:23.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:23.659 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:23.761 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:23.783 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:23.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:23.785 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:23.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:23.909 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:23.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:23.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:24.011 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:24.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:24.035 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:24.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:24.137 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:24.154 INFO:tasks.rados.rados.0.smithi076.stdout:1999: done (3 left) 2021-03-15T22:27:24.154 INFO:tasks.rados.rados.0.smithi076.stdout:2001: done (2 left) 2021-03-15T22:27:24.154 INFO:tasks.rados.rados.0.smithi076.stdout:2002: done (1 left) 2021-03-15T22:27:24.154 INFO:tasks.rados.rados.0.smithi076.stdout:2004: done (0 left) 2021-03-15T22:27:24.155 INFO:tasks.rados.rados.0.smithi076.stdout:2005: read oid 39 snap 219 2021-03-15T22:27:24.155 INFO:tasks.rados.rados.0.smithi076.stdout:2005: expect (ObjNum 664 snap 209 seq_num 664) 2021-03-15T22:27:24.155 INFO:tasks.rados.rados.0.smithi076.stdout:2006: read oid 44 snap 219 2021-03-15T22:27:24.155 INFO:tasks.rados.rados.0.smithi076.stdout:2006: expect (ObjNum 670 snap 212 seq_num 670) 2021-03-15T22:27:24.156 INFO:tasks.rados.rados.0.smithi076.stdout:2007: rollback oid 6 current snap is 227 2021-03-15T22:27:24.156 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 211 2021-03-15T22:27:24.156 INFO:tasks.rados.rados.0.smithi076.stdout:2008: delete oid 28 current snap is 227 2021-03-15T22:27:24.159 INFO:tasks.rados.rados.0.smithi076.stdout:2007: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:27:24.159 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 1201 (ObjNum 255155200 snap 21896 seq_num 4000000) dirty dne 2021-03-15T22:27:24.159 INFO:tasks.rados.rados.0.smithi076.stdout:2005: expect (ObjNum 664 snap 209 seq_num 664) 2021-03-15T22:27:24.164 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:24.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:24.203 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:27:24.220 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:27:24.220 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:27:24.220 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:27:24.220 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:27:24.221 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:27:24.268 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:24.292 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:24.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:24.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:24.313 INFO:tasks.rados.rados.0.smithi076.stdout:2006: expect (ObjNum 670 snap 212 seq_num 670) 2021-03-15T22:27:24.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:24.418 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:24.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:24.504 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:24.512 INFO:tasks.rados.rados.0.smithi076.stdout:2005: done (3 left) 2021-03-15T22:27:24.512 INFO:tasks.rados.rados.0.smithi076.stdout:2006: done (2 left) 2021-03-15T22:27:24.513 INFO:tasks.rados.rados.0.smithi076.stdout:2007: done (1 left) 2021-03-15T22:27:24.513 INFO:tasks.rados.rados.0.smithi076.stdout:2008: done (0 left) 2021-03-15T22:27:24.513 INFO:tasks.rados.rados.0.smithi076.stdout:2009: rollback oid 9 current snap is 227 2021-03-15T22:27:24.513 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 9 to 223 2021-03-15T22:27:24.514 INFO:tasks.rados.rados.0.smithi076.stdout:2010: read oid 11 snap -1 2021-03-15T22:27:24.514 INFO:tasks.rados.rados.0.smithi076.stdout:2010: expect (ObjNum 702 snap 226 seq_num 702) 2021-03-15T22:27:24.514 INFO:tasks.rados.rados.0.smithi076.stdout:2011: copy_from oid 32 from oid 21 current snap is 227 2021-03-15T22:27:24.514 INFO:tasks.rados.rados.0.smithi076.stdout:2012: rollback oid 5 current snap is 227 2021-03-15T22:27:24.515 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 217 2021-03-15T22:27:24.515 INFO:tasks.rados.rados.0.smithi076.stdout:2013: copy_from oid 1 from oid 26 current snap is 227 2021-03-15T22:27:24.515 INFO:tasks.rados.rados.0.smithi076.stdout:2014: copy_from oid 40 from oid 36 current snap is 227 2021-03-15T22:27:24.515 INFO:tasks.rados.rados.0.smithi076.stdout:2015: snap_remove snap 209 2021-03-15T22:27:24.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:24 smithi076 conmon[56514]: cluster 2021-03-15T22:27:23.083859+0000 mgr.y (mgr.14144) 2021-03-15T22:27:24.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:24 smithi076 conmon[56514]: 709 : cluster [DBG] pgmap v918: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 505 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 19 op/s; 207/1659 objects degraded (12.477%) 2021-03-15T22:27:24.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:24 smithi076 conmon[56514]: cluster 2021-03-15T22:27:24.151480+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e529: 8 total, 7 up, 7 in 2021-03-15T22:27:24.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:24 smithi076 conmon[62156]: cluster 2021-03-15T22:27 2021-03-15T22:27:24.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:24 smithi076 conmon[62156]: :23.083859+0000 mgr.y (mgr.14144) 709 : cluster [DBG] pgmap v918: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 505 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 19 op/s; 207/1659 objects degraded (12.477%) 2021-03-15T22:27:24.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:24 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:27:24.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:24 smithi076 conmon[62156]: 03-15T22:27:24.151480 2021-03-15T22:27:24.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:24 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2021-03-15T22:27:24.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:24 smithi076 conmon[62156]: 1468 : cluster [DBG] osdmap e529: 8 total, 7 up, 7 in 2021-03-15T22:27:24.520 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:24.543 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:24.545 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:24.646 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:24.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:24.669 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:24.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:24.771 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:24.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:24 smithi125 conmon[56739]: cluster 2021-03-15T22:27:23.083859+0000 mgr.y (mgr.14144) 709 : cluster [DBG] pgmap v918: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 505 MiB data, 1.2 GiB used, 625 GiB / 626 GiB avail; 12 MiB/s rd, 3.2 MiB/s wr, 19 op/s; 207/1659 objects degraded (12.477%) 2021-03-15T22:27:24.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:24 smithi125 conmon[56739]: cluster 2021-03-15T22:27:24.151480+0000 mon.a (mon.0) 1468 : cluster [DBG] osdmap e529: 8 total, 7 up, 7 in 2021-03-15T22:27:24.793 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:24.795 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:24.896 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:24.919 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:24.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:24.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:24.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-15T22:27:25.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:25.046 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:25.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:25.149 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:25.173 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:25.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:25.175 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:25.215 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:25.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:25.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:25.300 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:25.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:25.362 INFO:tasks.rados.rados.0.smithi076.stdout:2015: done (6 left) 2021-03-15T22:27:25.362 INFO:tasks.rados.rados.0.smithi076.stdout:2016: delete oid 39 current snap is 227 2021-03-15T22:27:25.363 INFO:tasks.rados.rados.0.smithi076.stdout:2009: finishing rollback tid 0 to smithi07694878-9 2021-03-15T22:27:25.363 INFO:tasks.rados.rados.0.smithi076.stdout:2012: finishing rollback tid 0 to smithi07694878-5 2021-03-15T22:27:25.363 INFO:tasks.rados.rados.0.smithi076.stdout:2012: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:27:25.364 INFO:tasks.rados.rados.0.smithi076.stdout:2012: finishing rollback tid 2 to smithi07694878-5 2021-03-15T22:27:25.364 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 608 (ObjNum 668 snap 212 seq_num 668) dirty exists 2021-03-15T22:27:25.364 INFO:tasks.rados.rados.0.smithi076.stdout:2009: finishing rollback tid 1 to smithi07694878-9 2021-03-15T22:27:25.364 INFO:tasks.rados.rados.0.smithi076.stdout:2009: finishing rollback tid 2 to smithi07694878-9 2021-03-15T22:27:25.365 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 1316 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:27:25.365 INFO:tasks.rados.rados.0.smithi076.stdout:2010: expect (ObjNum 702 snap 226 seq_num 702) 2021-03-15T22:27:25.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:25.436 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:25.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:25.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:25.496 INFO:tasks.rados.rados.0.smithi076.stdout:2013: finishing copy_from to smithi07694878-1 2021-03-15T22:27:25.496 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 806 (ObjNum 667 snap 211 seq_num 667) dirty exists 2021-03-15T22:27:25.497 INFO:tasks.rados.rados.0.smithi076.stdout:2013: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:27:25.497 INFO:tasks.rados.rados.0.smithi076.stdout:2011: finishing copy_from to smithi07694878-32 2021-03-15T22:27:25.497 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 1373 (ObjNum 675 snap 214 seq_num 675) dirty exists 2021-03-15T22:27:25.497 INFO:tasks.rados.rados.0.smithi076.stdout:2011: finishing copy_from racing read to smithi07694878-32 2021-03-15T22:27:25.498 INFO:tasks.rados.rados.0.smithi076.stdout:2014: finishing copy_from to smithi07694878-40 2021-03-15T22:27:25.498 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 440 (ObjNum 387 snap 104 seq_num 387) dirty exists 2021-03-15T22:27:25.498 INFO:tasks.rados.rados.0.smithi076.stdout:2014: finishing copy_from racing read to smithi07694878-40 2021-03-15T22:27:25.498 INFO:tasks.rados.rados.0.smithi076.stdout:2009: done (6 left) 2021-03-15T22:27:25.498 INFO:tasks.rados.rados.0.smithi076.stdout:2010: done (5 left) 2021-03-15T22:27:25.499 INFO:tasks.rados.rados.0.smithi076.stdout:2011: done (4 left) 2021-03-15T22:27:25.499 INFO:tasks.rados.rados.0.smithi076.stdout:2012: done (3 left) 2021-03-15T22:27:25.499 INFO:tasks.rados.rados.0.smithi076.stdout:2013: done (2 left) 2021-03-15T22:27:25.499 INFO:tasks.rados.rados.0.smithi076.stdout:2014: done (1 left) 2021-03-15T22:27:25.500 INFO:tasks.rados.rados.0.smithi076.stdout:2016: done (0 left) 2021-03-15T22:27:25.500 INFO:tasks.rados.rados.0.smithi076.stdout:2017: snap_remove snap 214 2021-03-15T22:27:25.539 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:25.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:25.562 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:25.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:25.601 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:27:25.618 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:25 smithi125 conmon[56739]: cluster 2021-03-15T22:27:25.137757+0000 mon.a 2021-03-15T22:27:25.618 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:25 smithi125 conmon[56739]: (mon.0) 1469 : cluster [WRN] Health check update: Degraded data redundancy: 207/1659 objects degraded (12.477%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:25.624 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:25 smithi076 conmon[56514]: cluster 2021-03-15T22:27:25.137757+0000 mon.a (mon.0) 1469 : cluster 2021-03-15T22:27:25.625 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:25 smithi076 conmon[56514]: [WRN] Health check update: Degraded data redundancy: 207/1659 objects degraded (12.477%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:25.625 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:25 smithi076 conmon[62156]: cluster 2021-03-15T22:27:25.137757+0000 2021-03-15T22:27:25.625 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:25 smithi076 conmon[62156]: mon.a (mon.0) 1469 : cluster [WRN] Health check update: Degraded data redundancy: 207/1659 objects degraded (12.477%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:25.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:25.686 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:25.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:25.688 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:25.789 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:25.813 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:25.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:25.815 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:25.916 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:25.941 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:25.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:26.042 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:26.066 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:26.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:26.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:26.114 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:26.168 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:26.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:26.194 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:26.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:26.296 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:26.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:26.320 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:26.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:26.371 INFO:tasks.rados.rados.0.smithi076.stdout:2017: done (0 left) 2021-03-15T22:27:26.371 INFO:tasks.rados.rados.0.smithi076.stdout:2018: delete oid 19 current snap is 227 2021-03-15T22:27:26.377 INFO:tasks.rados.rados.0.smithi076.stdout:2018: done (0 left) 2021-03-15T22:27:26.378 INFO:tasks.rados.rados.0.smithi076.stdout:2019: write oid 20 current snap is 227 2021-03-15T22:27:26.378 INFO:tasks.rados.rados.0.smithi076.stdout:2019: seq_num 706 ranges {665742=630270,1863699=604567,3121574=676394} 2021-03-15T22:27:26.383 INFO:tasks.rados.rados.0.smithi076.stdout:2019: writing smithi07694878-20 from 665742 to 1296012 tid 1 2021-03-15T22:27:26.389 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:27:26.389 INFO:tasks.rados.rados.0.smithi076.stdout:2019: writing smithi07694878-20 from 1863699 to 2468266 tid 2 2021-03-15T22:27:26.397 INFO:tasks.rados.rados.0.smithi076.stdout:2019: writing smithi07694878-20 from 3121574 to 3797968 tid 3 2021-03-15T22:27:26.398 INFO:tasks.rados.rados.0.smithi076.stdout:2020: read oid 42 snap -1 2021-03-15T22:27:26.398 INFO:tasks.rados.rados.0.smithi076.stdout:2020: expect (ObjNum 654 snap 205 seq_num 654) 2021-03-15T22:27:26.398 INFO:tasks.rados.rados.0.smithi076.stdout:2021: rollback oid 4 current snap is 227 2021-03-15T22:27:26.398 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 4 to 224 2021-03-15T22:27:26.399 INFO:tasks.rados.rados.0.smithi076.stdout:2019: finishing write tid 1 to smithi07694878-20 2021-03-15T22:27:26.399 INFO:tasks.rados.rados.0.smithi076.stdout:2019: finishing write tid 2 to smithi07694878-20 2021-03-15T22:27:26.399 INFO:tasks.rados.rados.0.smithi076.stdout:2022: rollback oid 10 current snap is 227 2021-03-15T22:27:26.399 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 10 to 217 2021-03-15T22:27:26.400 INFO:tasks.rados.rados.0.smithi076.stdout:2023: read oid 49 snap -1 2021-03-15T22:27:26.401 INFO:tasks.rados.rados.0.smithi076.stdout:2023: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:27:26.401 INFO:tasks.rados.rados.0.smithi076.stdout:2024: copy_from oid 30 from oid 9 current snap is 227 2021-03-15T22:27:26.401 INFO:tasks.rados.rados.0.smithi076.stdout:2025: delete oid 19 current snap is 227 2021-03-15T22:27:26.402 INFO:tasks.rados.rados.0.smithi076.stdout:2025: done (6 left) 2021-03-15T22:27:26.402 INFO:tasks.rados.rados.0.smithi076.stdout:2026: write oid 8 current snap is 227 2021-03-15T22:27:26.402 INFO:tasks.rados.rados.0.smithi076.stdout:2026: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2021-03-15T22:27:26.406 INFO:tasks.rados.rados.0.smithi076.stdout:2026: writing smithi07694878-8 from 699356 to 1196477 tid 1 2021-03-15T22:27:26.415 INFO:tasks.rados.rados.0.smithi076.stdout:2026: writing smithi07694878-8 from 1980663 to 2588899 tid 2 2021-03-15T22:27:26.422 INFO:tasks.rados.rados.0.smithi076.stdout:2026: writing smithi07694878-8 from 3176420 to 3591711 tid 3 2021-03-15T22:27:26.422 INFO:tasks.rados.rados.0.smithi076.stdout:2027: copy_from oid 29 from oid 11 current snap is 227 2021-03-15T22:27:26.422 INFO:tasks.rados.rados.0.smithi076.stdout:2028: write oid 31 current snap is 227 2021-03-15T22:27:26.422 INFO:tasks.rados.rados.0.smithi076.stdout:2028: seq_num 709 ranges {716163=572370,2039145=468247,3203843=626563} 2021-03-15T22:27:26.423 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:26.429 INFO:tasks.rados.rados.0.smithi076.stdout:2028: writing smithi07694878-31 from 716163 to 1288533 tid 1 2021-03-15T22:27:26.437 INFO:tasks.rados.rados.0.smithi076.stdout:2028: writing smithi07694878-31 from 2039145 to 2507392 tid 2 2021-03-15T22:27:26.444 INFO:tasks.rados.rados.0.smithi076.stdout:2028: writing smithi07694878-31 from 3203843 to 3830406 tid 3 2021-03-15T22:27:26.445 INFO:tasks.rados.rados.0.smithi076.stdout:2022: finishing rollback tid 0 to smithi07694878-10 2021-03-15T22:27:26.445 INFO:tasks.rados.rados.0.smithi076.stdout:2022: finishing rollback tid 1 to smithi07694878-10 2021-03-15T22:27:26.445 INFO:tasks.rados.rados.0.smithi076.stdout:2022: finishing rollback tid 2 to smithi07694878-10 2021-03-15T22:27:26.446 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 1311 (ObjNum 444 snap 125 seq_num 444) dirty exists 2021-03-15T22:27:26.446 INFO:tasks.rados.rados.0.smithi076.stdout:2021: finishing rollback tid 0 to smithi07694878-4 2021-03-15T22:27:26.446 INFO:tasks.rados.rados.0.smithi076.stdout:2021: finishing rollback tid 1 to smithi07694878-4 2021-03-15T22:27:26.446 INFO:tasks.rados.rados.0.smithi076.stdout:2021: finishing rollback tid 2 to smithi07694878-4 2021-03-15T22:27:26.447 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 518 (ObjNum 624 snap 196 seq_num 624) dirty exists 2021-03-15T22:27:26.447 INFO:tasks.rados.rados.0.smithi076.stdout:2019: finishing write tid 3 to smithi07694878-20 2021-03-15T22:27:26.447 INFO:tasks.rados.rados.0.smithi076.stdout:2019: finishing write tid 4 to smithi07694878-20 2021-03-15T22:27:26.447 INFO:tasks.rados.rados.0.smithi076.stdout:2019: finishing write tid 5 to smithi07694878-20 2021-03-15T22:27:26.447 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 853 (ObjNum 706 snap 227 seq_num 706) dirty exists 2021-03-15T22:27:26.448 INFO:tasks.rados.rados.0.smithi076.stdout:2019: left oid 20 (ObjNum 706 snap 227 seq_num 706) 2021-03-15T22:27:26.448 INFO:tasks.rados.rados.0.smithi076.stdout:2020: expect (ObjNum 654 snap 205 seq_num 654) 2021-03-15T22:27:26.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:26.460 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:26.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:26.500 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:26 smithi125 conmon[56739]: cluster 2021-03-15T22:27:25.084617+0000 mgr.y (mgr.14144) 2021-03-15T22:27:26.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:26 smithi125 conmon[56739]: 710 : cluster [DBG] pgmap v921: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 512 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.0 MiB/s wr, 18 op/s; 209/1671 objects degraded (12.507%) 2021-03-15T22:27:26.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:26 smithi125 conmon[56739]: cluster 2021-03-15T22:27:25.354388+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e530: 8 total, 7 up, 7 in 2021-03-15T22:27:26.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:26 smithi125 conmon[56739]: audit 2021-03-15T22:27:26.314487+0000 mon.c (mon.1) 44 : audit [INF] from='client.? 172.21.15.76:0/272546282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:26.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:26 smithi125 conmon[56739]: audit 2021-03-15T22:27:26.314783+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:26.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:26 smithi076 conmon[56514]: cluster 2021-03-15T22:27:25.084617+0000 mgr.y (mgr.14144) 710 : cluster [DBG] pgmap v921: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 512 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.0 MiB/s wr, 18 op/s; 209/1671 objects degraded (12.507%) 2021-03-15T22:27:26.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:26 smithi076 conmon[56514]: cluster 2021-03-15T22:27:25.354388+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e530: 8 total, 7 up, 7 in 2021-03-15T22:27:26.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:26 smithi076 conmon[56514]: audit 2021-03-15T22:27:26.314487+0000 mon.c (mon.1) 44 : audit [INF] from='client.? 172.21.15.76:0/272546282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:26.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:26 smithi076 conmon[56514]: audit 2021-03-15T22:27:26.314783+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:26.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:26 smithi076 conmon[62156]: cluster 2021-03-15T22:27:25.084617+0000 mgr.y (mgr.14144) 710 : cluster [DBG] pgmap v921: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 512 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 3.0 MiB/s wr, 18 op/s; 209/1671 objects degraded (12.507%) 2021-03-15T22:27:26.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:26 smithi076 conmon[62156]: cluster 2021-03-15T22:27:25.354388+0000 mon.a (mon.0) 1470 : cluster [DBG] osdmap e530: 8 total, 7 up, 7 in 2021-03-15T22:27:26.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:26 smithi076 conmon[62156]: audit 2021-03-15T22:27:26.314487+0000 mon.c (mon.1) 44 : audit [INF] from='client.? 172.21.15.76:0/272546282' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:26.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:26 smithi076 conmon[62156]: audit 2021-03-15T22:27:26.314783+0000 mon.a (mon.0) 1471 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:26.562 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:26.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:26.585 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:26.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:26.641 INFO:tasks.rados.rados.0.smithi076.stdout:2023: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:27:26.686 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:26.709 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:26.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:26.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:26.733 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:27:26.733 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:27:26.734 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:27:26.734 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:27:26.734 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.734 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:3 3.13 3:cbbf6c60:::converted:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected e57)", 2021-03-15T22:27:26.735 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:17:53.975558+0000", 2021-03-15T22:27:26.735 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 572.75238304000004, 2021-03-15T22:27:26.735 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.31057361700000002, 2021-03-15T22:27:26.735 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.735 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-15T22:27:26.736 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.736 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:27:26.736 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:27:26.736 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3 2021-03-15T22:27:26.737 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.737 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.738 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.738 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.738 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975558+0000", 2021-03-15T22:27:26.739 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.739 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.739 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.739 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.739 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975558+0000", 2021-03-15T22:27:26.740 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3149999999999999e-06 2021-03-15T22:27:26.740 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.740 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.740 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.741 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975560+0000", 2021-03-15T22:27:26.741 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7670000000000001e-06 2021-03-15T22:27:26.741 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.741 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.742 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.742 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975564+0000", 2021-03-15T22:27:26.743 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0937e-05 2021-03-15T22:27:26.743 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.743 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.744 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.744 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975575+0000", 2021-03-15T22:27:26.744 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1800000000000002e-06 2021-03-15T22:27:26.745 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.745 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.745 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.746 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975584+0000", 2021-03-15T22:27:26.746 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4805000000000001e-05 2021-03-15T22:27:26.747 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.747 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.747 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.748 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975639+0000", 2021-03-15T22:27:26.748 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2069999999999995e-06 2021-03-15T22:27:26.748 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.749 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.750 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:26.751 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:53.975647+0000", 2021-03-15T22:27:26.751 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.31038343600000001 2021-03-15T22:27:26.751 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.752 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.753 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.753 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:54.286030+0000", 2021-03-15T22:27:26.753 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010097 2021-03-15T22:27:26.754 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.754 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.754 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.754 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:17:54.286131+0000", 2021-03-15T22:27:26.755 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.31057361700000002 2021-03-15T22:27:26.755 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.755 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.755 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.756 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:1359 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94751391777792 gen 3] snapc 0=[] ondisk+write+known_if_redirected e316)", 2021-03-15T22:27:26.756 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:52.068429+0000", 2021-03-15T22:27:26.757 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 214.65951188400001, 2021-03-15T22:27:26.757 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.20407250399999999, 2021-03-15T22:27:26.757 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.757 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:26.758 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.758 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:27:26.758 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:27:26.759 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1359 2021-03-15T22:27:26.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.759 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.760 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.760 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.760 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068429+0000", 2021-03-15T22:27:26.760 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.760 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.761 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.761 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.761 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068429+0000", 2021-03-15T22:27:26.761 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.123e-06 2021-03-15T22:27:26.762 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.762 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.762 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.762 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068432+0000", 2021-03-15T22:27:26.762 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.066e-06 2021-03-15T22:27:26.763 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.763 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.763 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.764 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068435+0000", 2021-03-15T22:27:26.764 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5502000000000002e-05 2021-03-15T22:27:26.764 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.764 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.765 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.765 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068451+0000", 2021-03-15T22:27:26.765 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.348e-06 2021-03-15T22:27:26.765 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.765 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.766 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.766 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068457+0000", 2021-03-15T22:27:26.766 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0260999999999998e-05 2021-03-15T22:27:26.766 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.767 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.767 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.767 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068517+0000", 2021-03-15T22:27:26.767 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1440999999999999e-05 2021-03-15T22:27:26.768 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.768 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.768 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:26.768 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068529+0000", 2021-03-15T22:27:26.768 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19618954299999999 2021-03-15T22:27:26.769 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.769 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.769 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.769 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.264718+0000", 2021-03-15T22:27:26.770 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000154158 2021-03-15T22:27:26.770 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.770 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.770 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.771 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.264872+0000", 2021-03-15T22:27:26.771 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076290619999999998 2021-03-15T22:27:26.772 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.772 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.772 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.772 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.272501+0000", 2021-03-15T22:27:26.772 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.20407250399999999 2021-03-15T22:27:26.773 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.773 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.773 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.774 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.774 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.774 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:1360 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94751391778944 gen 4] snapc 0=[] ondisk+write+known_if_redirected e316)", 2021-03-15T22:27:26.774 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:52.068469+0000", 2021-03-15T22:27:26.774 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 214.659471268, 2021-03-15T22:27:26.775 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19939890299999999, 2021-03-15T22:27:26.775 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.775 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:26.775 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.776 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:27:26.776 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:27:26.776 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1360 2021-03-15T22:27:26.776 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.776 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.777 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.777 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.777 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068469+0000", 2021-03-15T22:27:26.777 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.778 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.778 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.778 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.778 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068469+0000", 2021-03-15T22:27:26.779 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5300000000000003e-07 2021-03-15T22:27:26.779 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.780 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.780 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.780 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068470+0000", 2021-03-15T22:27:26.780 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1710000000000001e-06 2021-03-15T22:27:26.780 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.781 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.781 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.781 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068472+0000", 2021-03-15T22:27:26.781 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3089999999999997e-06 2021-03-15T22:27:26.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.782 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.782 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068476+0000", 2021-03-15T22:27:26.782 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.489e-06 2021-03-15T22:27:26.783 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.783 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.783 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.783 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068479+0000", 2021-03-15T22:27:26.783 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1756999999999998e-05 2021-03-15T22:27:26.784 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.784 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.784 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.784 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068541+0000", 2021-03-15T22:27:26.785 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.311e-05 2021-03-15T22:27:26.785 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.785 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.785 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:26.785 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068554+0000", 2021-03-15T22:27:26.786 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19462378499999999 2021-03-15T22:27:26.786 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.786 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.786 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.787 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.263178+0000", 2021-03-15T22:27:26.787 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000179648 2021-03-15T22:27:26.787 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.787 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.787 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.788 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.263358+0000", 2021-03-15T22:27:26.788 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045107810000000002 2021-03-15T22:27:26.788 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.788 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.789 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.789 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.267868+0000", 2021-03-15T22:27:26.790 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19939890299999999 2021-03-15T22:27:26.790 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.790 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.791 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.791 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.791 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.792 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34361.0:352 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e316)", 2021-03-15T22:27:26.792 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:52.097658+0000", 2021-03-15T22:27:26.793 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 214.630282601, 2021-03-15T22:27:26.793 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.17297211700000001, 2021-03-15T22:27:26.794 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.794 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:26.794 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.795 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34361", 2021-03-15T22:27:26.795 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.125:0/1442040869", 2021-03-15T22:27:26.796 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 352 2021-03-15T22:27:26.796 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.796 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.797 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.797 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.798 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097658+0000", 2021-03-15T22:27:26.798 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.798 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.799 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.799 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.800 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097658+0000", 2021-03-15T22:27:26.800 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2650000000000001e-06 2021-03-15T22:27:26.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.801 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.802 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097661+0000", 2021-03-15T22:27:26.802 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0199999999999999e-06 2021-03-15T22:27:26.803 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.804 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.804 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.804 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097664+0000", 2021-03-15T22:27:26.804 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6565000000000001e-05 2021-03-15T22:27:26.805 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.805 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.805 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.805 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097681+0000", 2021-03-15T22:27:26.806 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9939999999999997e-06 2021-03-15T22:27:26.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.807 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.807 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.807 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097687+0000", 2021-03-15T22:27:26.807 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2488000000000003e-05 2021-03-15T22:27:26.808 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.808 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.808 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.808 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097739+0000", 2021-03-15T22:27:26.808 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0113e-05 2021-03-15T22:27:26.809 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.809 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.809 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:26.809 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097750+0000", 2021-03-15T22:27:26.810 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.172684848 2021-03-15T22:27:26.810 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.810 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.810 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.270434+0000", 2021-03-15T22:27:26.811 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9377999999999998e-05 2021-03-15T22:27:26.811 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.811 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.811 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.811 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.270514+0000", 2021-03-15T22:27:26.812 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000116446 2021-03-15T22:27:26.812 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.812 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.813 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.270630+0000", 2021-03-15T22:27:26.813 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.17297211700000001 2021-03-15T22:27:26.813 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.813 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.813 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.814 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3289 7.f 7:fc7cd984:::smithi07694878-4:head [delete] snapc fd=[fd,fc,fb,fa,f9,f7,f5,e9,e7,de] ondisk+write+peerstat_old+known_if_redirected e355)", 2021-03-15T22:27:26.814 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:30.295366+0000", 2021-03-15T22:27:26.815 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 176.43257442800001, 2021-03-15T22:27:26.815 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8092550220000001, 2021-03-15T22:27:26.815 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.816 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:26.816 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.816 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:26.816 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:26.817 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3289 2021-03-15T22:27:26.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.817 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.817 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295366+0000", 2021-03-15T22:27:26.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.819 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295366+0000", 2021-03-15T22:27:26.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8169999999999999e-06 2021-03-15T22:27:26.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.820 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295374+0000", 2021-03-15T22:27:26.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6029999999999999e-06 2021-03-15T22:27:26.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.821 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.821 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295382+0000", 2021-03-15T22:27:26.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1372000000000003e-05 2021-03-15T22:27:26.822 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.822 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.822 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.822 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295423+0000", 2021-03-15T22:27:26.823 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1537000000000001e-05 2021-03-15T22:27:26.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.823 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.823 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295445+0000", 2021-03-15T22:27:26.824 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012213400000000001 2021-03-15T22:27:26.824 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.824 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.824 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.824 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295567+0000", 2021-03-15T22:27:26.825 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1673000000000001e-05 2021-03-15T22:27:26.825 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.825 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:26.826 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295588+0000", 2021-03-15T22:27:26.826 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79131791699999998 2021-03-15T22:27:26.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.826 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.826 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.827 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.086906+0000", 2021-03-15T22:27:26.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0113e-05 2021-03-15T22:27:26.827 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.827 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.827 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:26.828 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.086916+0000", 2021-03-15T22:27:26.828 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0123092789999999 2021-03-15T22:27:26.828 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.828 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.829 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.829 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.099226+0000", 2021-03-15T22:27:26.829 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016286 2021-03-15T22:27:26.829 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.829 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.830 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for missing object", 2021-03-15T22:27:26.830 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.099389+0000", 2021-03-15T22:27:26.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001614868 2021-03-15T22:27:26.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.830 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.831 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.831 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.101003+0000", 2021-03-15T22:27:26.831 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023411800000000001 2021-03-15T22:27:26.831 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.832 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.101238+0000", 2021-03-15T22:27:26.832 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033837310000000001 2021-03-15T22:27:26.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.833 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.104621+0000", 2021-03-15T22:27:26.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8092550220000001 2021-03-15T22:27:26.833 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.834 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.834 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.834 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.835 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34361.0:390 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e354)", 2021-03-15T22:27:26.835 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:30.317842+0000", 2021-03-15T22:27:26.835 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 176.410098584, 2021-03-15T22:27:26.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79261869600000001, 2021-03-15T22:27:26.835 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.836 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:26.836 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.836 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34361", 2021-03-15T22:27:26.836 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.125:0/1442040869", 2021-03-15T22:27:26.836 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 390 2021-03-15T22:27:26.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.837 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317842+0000", 2021-03-15T22:27:26.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317842+0000", 2021-03-15T22:27:26.839 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2830000000000004e-06 2021-03-15T22:27:26.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.840 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.840 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317849+0000", 2021-03-15T22:27:26.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2929999999999997e-06 2021-03-15T22:27:26.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.840 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.841 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.841 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317856+0000", 2021-03-15T22:27:26.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2073000000000003e-05 2021-03-15T22:27:26.841 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.842 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.842 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.843 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317888+0000", 2021-03-15T22:27:26.843 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7453e-05 2021-03-15T22:27:26.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.843 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.844 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317905+0000", 2021-03-15T22:27:26.844 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015652 2021-03-15T22:27:26.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.844 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.844 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.845 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.318062+0000", 2021-03-15T22:27:26.845 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8180000000000004e-06 2021-03-15T22:27:26.845 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.845 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.846 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:27:26.846 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.318070+0000", 2021-03-15T22:27:26.846 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79211001599999997 2021-03-15T22:27:26.846 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.110180+0000", 2021-03-15T22:27:26.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000129782 2021-03-15T22:27:26.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.848 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.110309+0000", 2021-03-15T22:27:26.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015145800000000001 2021-03-15T22:27:26.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.849 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.849 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.110461+0000", 2021-03-15T22:27:26.849 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79261869600000001 2021-03-15T22:27:26.849 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.850 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.850 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.850 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.850 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.851 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3664 7.6 7:608078b0:::smithi07694878-12:head [delete] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+peerstat_old+known_if_redirected e385)", 2021-03-15T22:27:26.851 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.131030+0000", 2021-03-15T22:27:26.851 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 146.596910998, 2021-03-15T22:27:26.851 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039032994000000001, 2021-03-15T22:27:26.851 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.852 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:26.852 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.852 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:26.852 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:26.852 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3664 2021-03-15T22:27:26.853 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.853 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.853 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.853 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.853 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131030+0000", 2021-03-15T22:27:26.854 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.854 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.854 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.854 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.855 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131030+0000", 2021-03-15T22:27:26.855 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9670000000000003e-06 2021-03-15T22:27:26.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.855 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.855 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.856 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131034+0000", 2021-03-15T22:27:26.856 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5580000000000002e-06 2021-03-15T22:27:26.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.856 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.857 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131038+0000", 2021-03-15T22:27:26.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0582e-05 2021-03-15T22:27:26.857 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.857 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.858 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.858 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131059+0000", 2021-03-15T22:27:26.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1455e-05 2021-03-15T22:27:26.858 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.858 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.859 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.859 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131070+0000", 2021-03-15T22:27:26.859 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3596999999999994e-05 2021-03-15T22:27:26.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.860 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.860 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.860 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131134+0000", 2021-03-15T22:27:26.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8918999999999994e-05 2021-03-15T22:27:26.860 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.861 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131213+0000", 2021-03-15T22:27:26.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000169662 2021-03-15T22:27:26.861 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:27:26.862 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131382+0000", 2021-03-15T22:27:26.862 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001594575 2021-03-15T22:27:26.863 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.863 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.863 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.863 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.132977+0000", 2021-03-15T22:27:26.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035954851000000003 2021-03-15T22:27:26.864 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.864 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.864 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.864 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.168932+0000", 2021-03-15T22:27:26.864 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010296630000000001 2021-03-15T22:27:26.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.865 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.865 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:26.865 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.169962+0000", 2021-03-15T22:27:26.865 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2741000000000003e-05 2021-03-15T22:27:26.866 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.866 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:26.866 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.170004+0000", 2021-03-15T22:27:26.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8424000000000002e-05 2021-03-15T22:27:26.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.867 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.867 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.867 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.170063+0000", 2021-03-15T22:27:26.868 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039032994000000001 2021-03-15T22:27:26.868 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.868 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.868 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.868 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.869 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.869 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3679 7.6 7:6de8b070:::smithi07694878-15:head [append 0~0] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:27:26.869 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.195576+0000", 2021-03-15T22:27:26.869 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 146.532364648, 2021-03-15T22:27:26.870 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038774930999999999, 2021-03-15T22:27:26.870 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.870 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:26.870 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.870 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:26.871 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:26.871 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3679 2021-03-15T22:27:26.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.871 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.872 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.872 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.872 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195576+0000", 2021-03-15T22:27:26.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.873 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.873 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195576+0000", 2021-03-15T22:27:26.874 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1049999999999999e-06 2021-03-15T22:27:26.874 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.874 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.874 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.875 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195579+0000", 2021-03-15T22:27:26.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4559999999999999e-06 2021-03-15T22:27:26.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.875 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.875 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.876 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195582+0000", 2021-03-15T22:27:26.876 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2951e-05 2021-03-15T22:27:26.876 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.876 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.876 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.877 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195595+0000", 2021-03-15T22:27:26.877 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3730000000000001e-06 2021-03-15T22:27:26.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.878 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.878 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195600+0000", 2021-03-15T22:27:26.878 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6076999999999996e-05 2021-03-15T22:27:26.878 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.878 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.879 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.879 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195686+0000", 2021-03-15T22:27:26.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2466999999999996e-05 2021-03-15T22:27:26.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.880 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.880 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195759+0000", 2021-03-15T22:27:26.880 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016388000000000001 2021-03-15T22:27:26.880 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.881 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:27:26.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195922+0000", 2021-03-15T22:27:26.881 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0018566839999999999 2021-03-15T22:27:26.881 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:26.882 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197779+0000", 2021-03-15T22:27:26.882 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1146000000000001e-05 2021-03-15T22:27:26.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.883 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.883 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.883 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197830+0000", 2021-03-15T22:27:26.883 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036392299000000003 2021-03-15T22:27:26.884 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.884 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.884 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.884 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234223+0000", 2021-03-15T22:27:26.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6313e-05 2021-03-15T22:27:26.885 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.885 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.885 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:26.885 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234289+0000", 2021-03-15T22:27:26.886 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2180000000000004e-05 2021-03-15T22:27:26.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.886 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.886 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.886 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234351+0000", 2021-03-15T22:27:26.887 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038774930999999999 2021-03-15T22:27:26.887 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.887 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.887 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.888 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.888 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3680 7.6 7:6de8b070:::smithi07694878-15:head [rollback e7] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:27:26.888 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.195608+0000", 2021-03-15T22:27:26.888 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 146.532332698, 2021-03-15T22:27:26.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038942472999999998, 2021-03-15T22:27:26.889 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.889 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:26.889 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.889 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:26.890 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:26.890 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3680 2021-03-15T22:27:26.890 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.890 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.890 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.891 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.891 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195608+0000", 2021-03-15T22:27:26.891 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.891 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.892 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.892 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195608+0000", 2021-03-15T22:27:26.892 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0800000000000004e-07 2021-03-15T22:27:26.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.893 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.893 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.893 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195609+0000", 2021-03-15T22:27:26.893 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.015e-06 2021-03-15T22:27:26.893 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.894 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.894 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.894 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195610+0000", 2021-03-15T22:27:26.894 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4369999999999999e-06 2021-03-15T22:27:26.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.895 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.895 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.895 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195613+0000", 2021-03-15T22:27:26.895 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4880000000000001e-06 2021-03-15T22:27:26.896 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195616+0000", 2021-03-15T22:27:26.896 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001023368 2021-03-15T22:27:26.897 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.897 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.897 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.196639+0000", 2021-03-15T22:27:26.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5290000000000001e-05 2021-03-15T22:27:26.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.898 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.898 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.898 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.196694+0000", 2021-03-15T22:27:26.899 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014822899999999999 2021-03-15T22:27:26.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.899 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.899 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:27:26.899 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.196843+0000", 2021-03-15T22:27:26.900 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010197610000000001 2021-03-15T22:27:26.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.900 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.900 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.900 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197862+0000", 2021-03-15T22:27:26.901 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0350000000000001e-06 2021-03-15T22:27:26.901 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.901 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.901 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:26.902 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197867+0000", 2021-03-15T22:27:26.902 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036554689000000001 2021-03-15T22:27:26.902 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.902 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.902 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.903 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234422+0000", 2021-03-15T22:27:26.903 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2453000000000006e-05 2021-03-15T22:27:26.903 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.903 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.903 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:26.904 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234495+0000", 2021-03-15T22:27:26.904 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5999999999999999e-05 2021-03-15T22:27:26.904 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.904 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.905 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.905 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234551+0000", 2021-03-15T22:27:26.905 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038942472999999998 2021-03-15T22:27:26.905 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.905 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.906 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.906 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.906 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.906 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3681 7.6 7:6de8b070:::smithi07694878-15:head [append 0~0] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:27:26.906 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.195641+0000", 2021-03-15T22:27:26.907 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 146.53229968700001, 2021-03-15T22:27:26.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039044739000000002, 2021-03-15T22:27:26.907 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.907 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:26.908 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.908 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:26.908 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:26.908 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3681 2021-03-15T22:27:26.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.909 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195641+0000", 2021-03-15T22:27:26.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.910 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.910 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.910 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195641+0000", 2021-03-15T22:27:26.911 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6000000000000002e-07 2021-03-15T22:27:26.911 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.911 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.911 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.911 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195642+0000", 2021-03-15T22:27:26.912 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.23e-07 2021-03-15T22:27:26.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.912 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.912 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.912 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195643+0000", 2021-03-15T22:27:26.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0450000000000001e-06 2021-03-15T22:27:26.913 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.913 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.913 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195646+0000", 2021-03-15T22:27:26.914 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0839999999999999e-06 2021-03-15T22:27:26.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.914 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.914 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195648+0000", 2021-03-15T22:27:26.915 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015890559999999999 2021-03-15T22:27:26.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.915 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.916 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197237+0000", 2021-03-15T22:27:26.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4195000000000005e-05 2021-03-15T22:27:26.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.917 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197301+0000", 2021-03-15T22:27:26.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017653100000000001 2021-03-15T22:27:26.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.917 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.918 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:27:26.918 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197478+0000", 2021-03-15T22:27:26.918 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000394583 2021-03-15T22:27:26.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.919 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:26.919 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197872+0000", 2021-03-15T22:27:26.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024062000000000001 2021-03-15T22:27:26.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.920 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.920 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.198113+0000", 2021-03-15T22:27:26.920 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036489966999999998 2021-03-15T22:27:26.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.921 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.921 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.921 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234603+0000", 2021-03-15T22:27:26.921 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0292e-05 2021-03-15T22:27:26.921 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.922 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:26.922 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234643+0000", 2021-03-15T22:27:26.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2783e-05 2021-03-15T22:27:26.923 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.923 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.923 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234686+0000", 2021-03-15T22:27:26.923 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039044739000000002 2021-03-15T22:27:26.924 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.924 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.924 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.925 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4380 7.6 7:6283974b:::smithi07694878-30:head [write 484200~624934,stat] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:27:26.925 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:27:26.925 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 76.597090066999996, 2021-03-15T22:27:26.925 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.122158787, 2021-03-15T22:27:26.926 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.926 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:26.926 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.926 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:26.926 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:26.927 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4380 2021-03-15T22:27:26.927 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.927 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:27:26.928 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.928 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.928 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.928 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.929 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:27:26.929 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00065488299999999998 2021-03-15T22:27:26.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.930 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.930 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131506+0000", 2021-03-15T22:27:26.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.037e-06 2021-03-15T22:27:26.930 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.930 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.931 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.931 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131512+0000", 2021-03-15T22:27:26.931 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9637000000000001e-05 2021-03-15T22:27:26.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.932 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.932 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131541+0000", 2021-03-15T22:27:26.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9325000000000001e-05 2021-03-15T22:27:26.932 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.933 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.933 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.933 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131571+0000", 2021-03-15T22:27:26.933 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6747000000000002e-05 2021-03-15T22:27:26.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.934 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.934 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.934 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131647+0000", 2021-03-15T22:27:26.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000176789 2021-03-15T22:27:26.934 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131824+0000", 2021-03-15T22:27:26.935 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00042344700000000003 2021-03-15T22:27:26.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.936 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.936 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:27:26.936 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.132248+0000", 2021-03-15T22:27:26.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0026234660000000001 2021-03-15T22:27:26.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.937 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.937 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.937 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.134871+0000", 2021-03-15T22:27:26.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031408100000000002 2021-03-15T22:27:26.938 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.938 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.938 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:26.938 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.135185+0000", 2021-03-15T22:27:26.938 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.117673658 2021-03-15T22:27:26.939 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.939 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.939 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.252859+0000", 2021-03-15T22:27:26.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6511999999999997e-05 2021-03-15T22:27:26.940 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.940 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:26.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.252915+0000", 2021-03-15T22:27:26.941 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4204999999999997e-05 2021-03-15T22:27:26.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.941 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.941 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253009+0000", 2021-03-15T22:27:26.942 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.122158787 2021-03-15T22:27:26.942 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.942 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.942 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.943 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4381 7.6 7:6283974b:::smithi07694878-30:head [write 1860843~223357] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:27:26.943 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:27:26.943 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 76.591819119999997, 2021-03-15T22:27:26.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119715842, 2021-03-15T22:27:26.944 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.944 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:26.944 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.944 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:26.945 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:26.945 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4381 2021-03-15T22:27:26.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.945 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.946 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:27:26.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:27:26.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019479299999999999 2021-03-15T22:27:26.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.948 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.948 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136316+0000", 2021-03-15T22:27:26.948 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7419999999999999e-06 2021-03-15T22:27:26.949 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.949 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.949 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.949 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136320+0000", 2021-03-15T22:27:26.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5999999999999999e-05 2021-03-15T22:27:26.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.950 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.951 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136336+0000", 2021-03-15T22:27:26.951 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4386e-05 2021-03-15T22:27:26.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.955 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.955 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.955 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136361+0000", 2021-03-15T22:27:26.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4256999999999999e-05 2021-03-15T22:27:26.956 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.956 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.956 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.956 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136415+0000", 2021-03-15T22:27:26.957 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000158257 2021-03-15T22:27:26.957 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.957 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.957 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:26.958 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136573+0000", 2021-03-15T22:27:26.958 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.116481746 2021-03-15T22:27:26.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.958 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253055+0000", 2021-03-15T22:27:26.959 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1245e-05 2021-03-15T22:27:26.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253126+0000", 2021-03-15T22:27:26.960 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000138727 2021-03-15T22:27:26.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.961 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:27:26.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253265+0000", 2021-03-15T22:27:26.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0017125980000000001 2021-03-15T22:27:26.961 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.962 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:26.962 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254977+0000", 2021-03-15T22:27:26.962 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000112898 2021-03-15T22:27:26.962 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255090+0000", 2021-03-15T22:27:26.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00066560399999999998 2021-03-15T22:27:26.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.964 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255756+0000", 2021-03-15T22:27:26.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8887999999999998e-05 2021-03-15T22:27:26.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255795+0000", 2021-03-15T22:27:26.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2701000000000003e-05 2021-03-15T22:27:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255837+0000", 2021-03-15T22:27:26.966 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119715842 2021-03-15T22:27:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4382 7.6 7:6283974b:::smithi07694878-30:head [setxattr _header (58) in=7b,truncate 2084200] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:27:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:27:26.968 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 76.591570039999993, 2021-03-15T22:27:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119633089, 2021-03-15T22:27:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.969 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4382 2021-03-15T22:27:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.970 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:27:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:27:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4609999999999998e-06 2021-03-15T22:27:26.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.973 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.973 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136373+0000", 2021-03-15T22:27:26.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7880000000000001e-06 2021-03-15T22:27:26.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.974 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.974 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136375+0000", 2021-03-15T22:27:26.974 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2340000000000002e-06 2021-03-15T22:27:26.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.975 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.975 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.975 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136380+0000", 2021-03-15T22:27:26.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3160000000000001e-06 2021-03-15T22:27:26.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.976 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.976 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.977 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136384+0000", 2021-03-15T22:27:26.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024020700000000001 2021-03-15T22:27:26.977 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.977 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.978 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136624+0000", 2021-03-15T22:27:26.978 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7928000000000003e-05 2021-03-15T22:27:26.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.978 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:26.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136702+0000", 2021-03-15T22:27:26.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.11744721399999999 2021-03-15T22:27:26.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254149+0000", 2021-03-15T22:27:26.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5769000000000001e-05 2021-03-15T22:27:26.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:26.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254205+0000", 2021-03-15T22:27:26.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000127489 2021-03-15T22:27:26.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:27:26.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254332+0000", 2021-03-15T22:27:26.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00068339200000000001 2021-03-15T22:27:26.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.983 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:26.983 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255016+0000", 2021-03-15T22:27:26.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000233143 2021-03-15T22:27:26.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255249+0000", 2021-03-15T22:27:26.984 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00063040000000000004 2021-03-15T22:27:26.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.985 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:26.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255879+0000", 2021-03-15T22:27:26.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9571000000000003e-05 2021-03-15T22:27:26.986 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.986 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.986 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:26.986 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255939+0000", 2021-03-15T22:27:26.987 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5177000000000003e-05 2021-03-15T22:27:26.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.987 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:26.987 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256004+0000", 2021-03-15T22:27:26.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119633089 2021-03-15T22:27:26.988 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.988 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:26.988 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:26.988 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.989 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.989 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4383 7.6 7:6283974b:::smithi07694878-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e455)", 2021-03-15T22:27:26.989 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:27:26.989 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 76.591552308999994, 2021-03-15T22:27:26.990 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119776914, 2021-03-15T22:27:26.990 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:26.990 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:26.990 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:26.990 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:26.991 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:26.991 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4383 2021-03-15T22:27:26.991 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.991 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:26.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.992 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:26.992 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:27:26.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:26.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.993 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.993 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:26.993 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:27:26.993 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9899999999999999e-07 2021-03-15T22:27:26.993 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.994 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:26.994 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136389+0000", 2021-03-15T22:27:26.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1799999999999999e-06 2021-03-15T22:27:26.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.995 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:26.995 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136390+0000", 2021-03-15T22:27:26.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0630000000000002e-06 2021-03-15T22:27:26.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:26.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136396+0000", 2021-03-15T22:27:26.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6900000000000001e-06 2021-03-15T22:27:26.997 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.997 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:26.997 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136399+0000", 2021-03-15T22:27:26.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034116099999999998 2021-03-15T22:27:26.998 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.998 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.998 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:26.998 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136740+0000", 2021-03-15T22:27:26.998 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000123552 2021-03-15T22:27:26.999 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:26.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:26.999 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:26.999 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136864+0000", 2021-03-15T22:27:27.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.118007994 2021-03-15T22:27:27.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.000 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254872+0000", 2021-03-15T22:27:27.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0631999999999996e-05 2021-03-15T22:27:27.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.001 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:27.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254963+0000", 2021-03-15T22:27:27.002 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010704379999999999 2021-03-15T22:27:27.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.002 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256033+0000", 2021-03-15T22:27:27.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2941e-05 2021-03-15T22:27:27.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:27.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256086+0000", 2021-03-15T22:27:27.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9463999999999997e-05 2021-03-15T22:27:27.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.004 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:27.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256165+0000", 2021-03-15T22:27:27.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119776914 2021-03-15T22:27:27.005 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.005 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:27.005 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.006 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4396 7.6 7:61e0cc05:::smithi07694878-16:head [copy-from ver 767] snapc 161=[161,160,15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e457)", 2021-03-15T22:27:27.006 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:12.125558+0000", 2021-03-15T22:27:27.007 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 74.602382813999995, 2021-03-15T22:27:27.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035732501, 2021-03-15T22:27:27.007 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:27.007 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:27.007 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:27.008 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:27.008 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:27.008 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4396 2021-03-15T22:27:27.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.009 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:27.009 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.009 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:27.009 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125558+0000", 2021-03-15T22:27:27.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:27.010 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.010 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.010 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:27.010 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125558+0000", 2021-03-15T22:27:27.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2009999999999996e-06 2021-03-15T22:27:27.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.011 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.011 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:27.011 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125564+0000", 2021-03-15T22:27:27.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8043e-05 2021-03-15T22:27:27.012 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.012 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.012 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:27.012 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125582+0000", 2021-03-15T22:27:27.013 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4352e-05 2021-03-15T22:27:27.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.013 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.013 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:27.013 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125607+0000", 2021-03-15T22:27:27.014 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4769999999999992e-06 2021-03-15T22:27:27.014 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.014 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:27.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125616+0000", 2021-03-15T22:27:27.015 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047446260000000001 2021-03-15T22:27:27.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.015 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.016 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.130361+0000", 2021-03-15T22:27:27.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0346999999999996e-05 2021-03-15T22:27:27.016 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:27.017 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.130431+0000", 2021-03-15T22:27:27.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012214641 2021-03-15T22:27:27.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.018 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:27:27.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.142646+0000", 2021-03-15T22:27:27.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013352012 2021-03-15T22:27:27.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:27.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.155998+0000", 2021-03-15T22:27:27.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000108766 2021-03-15T22:27:27.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:27.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.156106+0000", 2021-03-15T22:27:27.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050357570000000001 2021-03-15T22:27:27.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.021 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:27.021 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161142+0000", 2021-03-15T22:27:27.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1378000000000007e-05 2021-03-15T22:27:27.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:27.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161214+0000", 2021-03-15T22:27:27.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6901000000000001e-05 2021-03-15T22:27:27.023 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:27.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161290+0000", 2021-03-15T22:27:27.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035732501 2021-03-15T22:27:27.024 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.024 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:27.024 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.024 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.025 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4397 7.6 7:61e0cc05:::smithi07694878-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e457)", 2021-03-15T22:27:27.025 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:12.125625+0000", 2021-03-15T22:27:27.025 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 74.602315691000001, 2021-03-15T22:27:27.025 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035880710000000003, 2021-03-15T22:27:27.026 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:27.026 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:27.026 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:27.026 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:27.026 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:27.027 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4397 2021-03-15T22:27:27.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.027 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:27.027 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:27.028 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125625+0000", 2021-03-15T22:27:27.028 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:27.028 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.029 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:27.029 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125625+0000", 2021-03-15T22:27:27.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9230000000000001e-06 2021-03-15T22:27:27.029 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.029 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.030 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:27.030 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125627+0000", 2021-03-15T22:27:27.030 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.198e-06 2021-03-15T22:27:27.030 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.031 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.031 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:27.031 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125629+0000", 2021-03-15T22:27:27.031 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1629999999999996e-06 2021-03-15T22:27:27.031 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.032 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:27.032 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125637+0000", 2021-03-15T22:27:27.032 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5560000000000001e-06 2021-03-15T22:27:27.033 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.033 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.033 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:27.033 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125642+0000", 2021-03-15T22:27:27.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053224839999999997 2021-03-15T22:27:27.034 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.034 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.034 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.130964+0000", 2021-03-15T22:27:27.034 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3690999999999996e-05 2021-03-15T22:27:27.035 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:27.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.131038+0000", 2021-03-15T22:27:27.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017722974999999998 2021-03-15T22:27:27.036 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.036 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.148761+0000", 2021-03-15T22:27:27.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5197000000000004e-05 2021-03-15T22:27:27.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.037 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:27.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.148856+0000", 2021-03-15T22:27:27.038 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012464467999999999 2021-03-15T22:27:27.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.038 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161321+0000", 2021-03-15T22:27:27.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3552000000000003e-05 2021-03-15T22:27:27.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:27.040 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161394+0000", 2021-03-15T22:27:27.040 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011150300000000001 2021-03-15T22:27:27.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.040 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:27.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161506+0000", 2021-03-15T22:27:27.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035880710000000003 2021-03-15T22:27:27.041 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.041 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:27.042 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.042 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.042 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4562 7.6 7:6a64397b:::smithi07694878-49:head [copy-from ver 960] snapc 170=[170,16e,16d,168,166,162,161,15f,15b,141,13f,13e,131,12f,123,10c,102] ondisk+write+known_if_redirected e474)", 2021-03-15T22:27:27.043 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:29.288710+0000", 2021-03-15T22:27:27.043 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 57.439230852000001, 2021-03-15T22:27:27.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030539015999999999, 2021-03-15T22:27:27.043 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:27.043 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:27.044 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:27.044 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:27.044 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:27.044 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4562 2021-03-15T22:27:27.044 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.045 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:27.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.045 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:27.045 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288710+0000", 2021-03-15T22:27:27.046 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:27.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.046 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.046 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:27.046 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288710+0000", 2021-03-15T22:27:27.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2800000000000002e-06 2021-03-15T22:27:27.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:27.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288712+0000", 2021-03-15T22:27:27.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.742e-06 2021-03-15T22:27:27.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:27.049 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288714+0000", 2021-03-15T22:27:27.049 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3130000000000001e-06 2021-03-15T22:27:27.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.049 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:27.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288721+0000", 2021-03-15T22:27:27.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6489999999999999e-06 2021-03-15T22:27:27.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.050 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:27.051 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288726+0000", 2021-03-15T22:27:27.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00084560199999999999 2021-03-15T22:27:27.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.052 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.052 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.289571+0000", 2021-03-15T22:27:27.052 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104343 2021-03-15T22:27:27.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.053 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:27.053 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.289676+0000", 2021-03-15T22:27:27.053 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014401884 2021-03-15T22:27:27.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:27:27.054 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.304078+0000", 2021-03-15T22:27:27.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010704943 2021-03-15T22:27:27.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.055 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:27.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.314783+0000", 2021-03-15T22:27:27.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0736e-05 2021-03-15T22:27:27.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:27.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.314793+0000", 2021-03-15T22:27:27.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042097990000000002 2021-03-15T22:27:27.057 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.057 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:27.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319003+0000", 2021-03-15T22:27:27.057 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016160299999999999 2021-03-15T22:27:27.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.058 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:27.058 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319165+0000", 2021-03-15T22:27:27.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4122000000000005e-05 2021-03-15T22:27:27.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:27.059 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319249+0000", 2021-03-15T22:27:27.060 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030539015999999999 2021-03-15T22:27:27.060 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.060 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:27.060 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.061 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4563 7.6 7:6a64397b:::smithi07694878-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e474)", 2021-03-15T22:27:27.061 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:29.288736+0000", 2021-03-15T22:27:27.061 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 57.439204607000001, 2021-03-15T22:27:27.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030703508000000001, 2021-03-15T22:27:27.062 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:27.062 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:27.062 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:27.062 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:27.063 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:27.063 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4563 2021-03-15T22:27:27.063 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.063 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:27.064 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:27.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288736+0000", 2021-03-15T22:27:27.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:27.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:27.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288736+0000", 2021-03-15T22:27:27.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1170000000000001e-06 2021-03-15T22:27:27.065 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:27.066 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288737+0000", 2021-03-15T22:27:27.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.66e-06 2021-03-15T22:27:27.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.067 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.067 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:27.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288739+0000", 2021-03-15T22:27:27.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.02e-06 2021-03-15T22:27:27.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:27.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288748+0000", 2021-03-15T22:27:27.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8380000000000001e-06 2021-03-15T22:27:27.069 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:27.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288753+0000", 2021-03-15T22:27:27.070 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012277779999999999 2021-03-15T22:27:27.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.070 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.070 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.289981+0000", 2021-03-15T22:27:27.071 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4601000000000002e-05 2021-03-15T22:27:27.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.071 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.071 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:27.072 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.290025+0000", 2021-03-15T22:27:27.072 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019574004999999998 2021-03-15T22:27:27.072 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.072 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.073 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.309599+0000", 2021-03-15T22:27:27.073 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8437000000000006e-05 2021-03-15T22:27:27.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.073 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.074 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:27.074 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.309698+0000", 2021-03-15T22:27:27.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0095754180000000005 2021-03-15T22:27:27.074 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.075 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.075 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319273+0000", 2021-03-15T22:27:27.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3226999999999994e-05 2021-03-15T22:27:27.075 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.076 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.076 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:27.076 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319356+0000", 2021-03-15T22:27:27.076 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3406999999999996e-05 2021-03-15T22:27:27.076 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.077 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.077 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:27.077 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319440+0000", 2021-03-15T22:27:27.077 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030703508000000001 2021-03-15T22:27:27.077 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.078 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:27.078 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.078 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.078 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.078 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5175 7.6 7:6283974b:::smithi07694878-30:head [copy-from ver 1316] snapc 1a9=[1a9,1a7,1a6,1a4,1a3,1a2,1a1,1a0,19f,19c,19b,195,194,193,190,18a,184,181,180,17a,176,16e,161,15f,141,131,123,10c] ondisk+write+known_if_redirected e531)", 2021-03-15T22:27:27.079 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:27:26.396817+0000", 2021-03-15T22:27:27.079 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.33112387399999998, 2021-03-15T22:27:27.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027266911000000001, 2021-03-15T22:27:27.079 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:27.080 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:27.080 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:27.080 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:27.080 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:27.080 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5175 2021-03-15T22:27:27.081 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.081 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:27.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:27.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396817+0000", 2021-03-15T22:27:27.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:27.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:27.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396817+0000", 2021-03-15T22:27:27.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2899999999999998e-06 2021-03-15T22:27:27.083 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.083 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:27.084 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396820+0000", 2021-03-15T22:27:27.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5000000000000002e-06 2021-03-15T22:27:27.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.084 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:27.085 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396823+0000", 2021-03-15T22:27:27.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8169999999999994e-06 2021-03-15T22:27:27.085 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.086 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:27.086 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396832+0000", 2021-03-15T22:27:27.086 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1040000000000003e-06 2021-03-15T22:27:27.086 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.086 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.087 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:27.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396839+0000", 2021-03-15T22:27:27.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0036395339999999998 2021-03-15T22:27:27.087 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.088 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.088 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.088 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.400478+0000", 2021-03-15T22:27:27.088 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1018000000000002e-05 2021-03-15T22:27:27.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.089 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.089 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:27.089 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.400569+0000", 2021-03-15T22:27:27.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.007817404 2021-03-15T22:27:27.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.090 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.090 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:27:27.090 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.408387+0000", 2021-03-15T22:27:27.090 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014672318 2021-03-15T22:27:27.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.091 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:27.091 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.423059+0000", 2021-03-15T22:27:27.091 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00087863399999999999 2021-03-15T22:27:27.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.092 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:27.092 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.423937+0000", 2021-03-15T22:27:27.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3392000000000001e-05 2021-03-15T22:27:27.093 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.093 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.093 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:27:27.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.423971+0000", 2021-03-15T22:27:27.093 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3430999999999998e-05 2021-03-15T22:27:27.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:27:27.094 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.424014+0000", 2021-03-15T22:27:27.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9468999999999995e-05 2021-03-15T22:27:27.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.095 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:27.095 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.424084+0000", 2021-03-15T22:27:27.096 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027266911000000001 2021-03-15T22:27:27.096 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.096 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:27.096 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.097 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5176 7.6 7:6283974b:::smithi07694878-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e531)", 2021-03-15T22:27:27.097 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:27:26.396845+0000", 2021-03-15T22:27:27.097 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.331095788, 2021-03-15T22:27:27.097 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027609142, 2021-03-15T22:27:27.098 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:27:27.098 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:27:27.098 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:27:27.098 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:27:27.099 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:27.099 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5176 2021-03-15T22:27:27.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.099 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:27:27.099 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.100 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:27:27.100 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396845+0000", 2021-03-15T22:27:27.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:27:27.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.101 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:27:27.101 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396845+0000", 2021-03-15T22:27:27.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.232e-06 2021-03-15T22:27:27.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.102 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:27:27.102 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396846+0000", 2021-03-15T22:27:27.102 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.353e-06 2021-03-15T22:27:27.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:27:27.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396848+0000", 2021-03-15T22:27:27.103 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6959999999999999e-06 2021-03-15T22:27:27.103 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.104 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.104 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:27:27.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396854+0000", 2021-03-15T22:27:27.104 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2690000000000001e-06 2021-03-15T22:27:27.104 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.105 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.105 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:27:27.105 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396859+0000", 2021-03-15T22:27:27.105 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003890082 2021-03-15T22:27:27.106 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.106 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.106 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.106 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.400749+0000", 2021-03-15T22:27:27.106 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2172000000000001e-05 2021-03-15T22:27:27.107 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.107 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.107 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:27.107 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.400801+0000", 2021-03-15T22:27:27.107 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015282324999999999 2021-03-15T22:27:27.108 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.108 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.108 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.108 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.416083+0000", 2021-03-15T22:27:27.108 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010308 2021-03-15T22:27:27.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.109 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:27.109 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.416186+0000", 2021-03-15T22:27:27.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0079379370000000008 2021-03-15T22:27:27.110 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.110 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:27:27.110 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.424124+0000", 2021-03-15T22:27:27.111 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6258e-05 2021-03-15T22:27:27.111 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.111 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.111 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:27:27.111 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.424210+0000", 2021-03-15T22:27:27.112 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024373799999999999 2021-03-15T22:27:27.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:27:27.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:27:27.112 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:27:27.113 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.424454+0000", 2021-03-15T22:27:27.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027609142 2021-03-15T22:27:27.113 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.113 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:27.113 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.114 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:27:27.114 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:27:27.114 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:27:27.114 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:27.115 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2021-03-15T22:27:27.115 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:27:27.116 INFO:tasks.thrashosds.thrasher:inject_pause on 7 2021-03-15T22:27:27.116 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-03-15T22:27:27.116 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-03-15T22:27:27.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 config set filestore_inject_stall 3 2021-03-15T22:27:27.118 INFO:tasks.rados.rados.0.smithi076.stdout:2024: finishing copy_from to smithi07694878-30 2021-03-15T22:27:27.118 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 1316 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:27:27.118 INFO:tasks.rados.rados.0.smithi076.stdout:2024: finishing copy_from racing read to smithi07694878-30 2021-03-15T22:27:27.119 INFO:tasks.rados.rados.0.smithi076.stdout:2026: finishing write tid 1 to smithi07694878-8 2021-03-15T22:27:27.119 INFO:tasks.rados.rados.0.smithi076.stdout:2026: finishing write tid 2 to smithi07694878-8 2021-03-15T22:27:27.119 INFO:tasks.rados.rados.0.smithi076.stdout:2026: finishing write tid 3 to smithi07694878-8 2021-03-15T22:27:27.119 INFO:tasks.rados.rados.0.smithi076.stdout:2026: finishing write tid 4 to smithi07694878-8 2021-03-15T22:27:27.120 INFO:tasks.rados.rados.0.smithi076.stdout:2026: finishing write tid 5 to smithi07694878-8 2021-03-15T22:27:27.120 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1185 (ObjNum 708 snap 227 seq_num 708) dirty exists 2021-03-15T22:27:27.120 INFO:tasks.rados.rados.0.smithi076.stdout:2026: left oid 8 (ObjNum 708 snap 227 seq_num 708) 2021-03-15T22:27:27.120 INFO:tasks.rados.rados.0.smithi076.stdout:2028: finishing write tid 1 to smithi07694878-31 2021-03-15T22:27:27.120 INFO:tasks.rados.rados.0.smithi076.stdout:2028: finishing write tid 2 to smithi07694878-31 2021-03-15T22:27:27.121 INFO:tasks.rados.rados.0.smithi076.stdout:2019: done (8 left) 2021-03-15T22:27:27.121 INFO:tasks.rados.rados.0.smithi076.stdout:2020: done (7 left) 2021-03-15T22:27:27.121 INFO:tasks.rados.rados.0.smithi076.stdout:2021: done (6 left) 2021-03-15T22:27:27.121 INFO:tasks.rados.rados.0.smithi076.stdout:2022: done (5 left) 2021-03-15T22:27:27.122 INFO:tasks.rados.rados.0.smithi076.stdout:2023: done (4 left) 2021-03-15T22:27:27.122 INFO:tasks.rados.rados.0.smithi076.stdout:2024: done (3 left) 2021-03-15T22:27:27.122 INFO:tasks.rados.rados.0.smithi076.stdout:2026: done (2 left) 2021-03-15T22:27:27.122 INFO:tasks.rados.rados.0.smithi076.stdout:2029: snap_create 2021-03-15T22:27:27.122 INFO:tasks.rados.rados.0.smithi076.stdout:2027: finishing copy_from to smithi07694878-29 2021-03-15T22:27:27.123 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 506 (ObjNum 702 snap 226 seq_num 702) dirty exists 2021-03-15T22:27:27.123 INFO:tasks.rados.rados.0.smithi076.stdout:2027: finishing copy_from racing read to smithi07694878-29 2021-03-15T22:27:27.123 INFO:tasks.rados.rados.0.smithi076.stdout:2028: finishing write tid 3 to smithi07694878-31 2021-03-15T22:27:27.123 INFO:tasks.rados.rados.0.smithi076.stdout:2028: finishing write tid 4 to smithi07694878-31 2021-03-15T22:27:27.123 INFO:tasks.rados.rados.0.smithi076.stdout:2028: finishing write tid 5 to smithi07694878-31 2021-03-15T22:27:27.124 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 855 (ObjNum 709 snap 227 seq_num 709) dirty exists 2021-03-15T22:27:27.124 INFO:tasks.rados.rados.0.smithi076.stdout:2028: left oid 31 (ObjNum 709 snap 227 seq_num 709) 2021-03-15T22:27:27.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:27.148 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:27.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:27.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:27.271 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:27.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:27.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:27.368 INFO:tasks.rados.rados.0.smithi076.stdout:2027: done (2 left) 2021-03-15T22:27:27.368 INFO:tasks.rados.rados.0.smithi076.stdout:2028: done (1 left) 2021-03-15T22:27:27.369 INFO:tasks.rados.rados.0.smithi076.stdout:2029: done (0 left) 2021-03-15T22:27:27.369 INFO:tasks.rados.rados.0.smithi076.stdout:2030: rollback oid 17 current snap is 228 2021-03-15T22:27:27.369 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 17 to 212 2021-03-15T22:27:27.369 INFO:tasks.rados.rados.0.smithi076.stdout:2031: rollback oid 32 current snap is 228 2021-03-15T22:27:27.370 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 32 to 219 2021-03-15T22:27:27.370 INFO:tasks.rados.rados.0.smithi076.stdout:2032: snap_remove snap 204 2021-03-15T22:27:27.373 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:27.409 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:27.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:27.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:27.441 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:27 smithi125 conmon[56739]: cluster 2021-03-15T22:27:26.352936+0000 mon.a (mon.0) 1472 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:27.441 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:27 smithi125 conmon[56739]: audit 2021-03-15T22:27:26.368034+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:27.442 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:27 smithi125 conmon[56739]: cluster 2021-03-15T22:27:26.368107+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e531: 8 total, 7 up, 7 in 2021-03-15T22:27:27.444 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:27:27.463 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:27:27.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:27.533 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:27.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:27.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:27.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:27.659 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:27.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:27.661 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:27.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:27 smithi076 conmon[56514]: cluster 2021-03-15T22:27:26.352936+0000 mon.a (mon.0 2021-03-15T22:27:27.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:27 smithi076 conmon[56514]: ) 1472 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:27.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:27 smithi076 conmon[56514]: audit 2021-03-15T22:27:26.368034+0000 mon.a (mon.0) 1473 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:27.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:27 smithi076 conmon[56514]: cluster 2021-03-15T22:27:26.368107+0000 mon.a (mon.0) 1474 : cluster [DBG] osdmap e531: 8 total, 7 up, 7 in 2021-03-15T22:27:27.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:27 smithi076 conmon[62156]: cluster 2021-03-15T22:27:26.352936+0000 mon.a (mon.0 2021-03-15T22:27:27.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:27 smithi076 conmon[62156]: ) 1472 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:27.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:27 smithi076 conmon[62156]: audit 2021-03-15T22:27:26.368034+0000 2021-03-15T22:27:27.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:27 smithi076 conmon[62156]: mon.a (mon.0) 1473 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:27.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:27 smithi076 conmon[62156]: cluster 2021-03-15T22:27:27.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:27 smithi076 conmon[62156]: 2021-03-15T22:27:26.368107+0000 mon.a (mon.0) 1474 2021-03-15T22:27:27.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:27 smithi076 conmon[62156]: : cluster [DBG] osdmap e531: 8 total, 7 up, 7 in 2021-03-15T22:27:27.762 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:27.779 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:27.790 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:27.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:27.791 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:27.893 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:27.916 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:27.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:27.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:28.019 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:28.042 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:28.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:28.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:28.145 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:28.166 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:27:28.167 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:28.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:28.168 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:28.186 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:27:28.207 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:27:28.225 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:27:28.243 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:27:28.259 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:27:28.269 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:28.277 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:27:28.278 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:27:28.278 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:27:28.278 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:27:28.278 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:27:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:27:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:27:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:27:28.279 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:27:28.290 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:28.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:28.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:28.371 INFO:tasks.rados.rados.0.smithi076.stdout:2032: done (2 left) 2021-03-15T22:27:28.371 INFO:tasks.rados.rados.0.smithi076.stdout:2033: delete oid 26 current snap is 228 2021-03-15T22:27:28.371 INFO:tasks.rados.rados.0.smithi076.stdout:2030: finishing rollback tid 0 to smithi07694878-17 2021-03-15T22:27:28.372 INFO:tasks.rados.rados.0.smithi076.stdout:2030: finishing rollback tid 1 to smithi07694878-17 2021-03-15T22:27:28.372 INFO:tasks.rados.rados.0.smithi076.stdout:2030: finishing rollback tid 2 to smithi07694878-17 2021-03-15T22:27:28.372 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 842 (ObjNum 618 snap 192 seq_num 618) dirty exists 2021-03-15T22:27:28.372 INFO:tasks.rados.rados.0.smithi076.stdout:2031: finishing rollback tid 0 to smithi07694878-32 2021-03-15T22:27:28.373 INFO:tasks.rados.rados.0.smithi076.stdout:2031: finishing rollback tid 1 to smithi07694878-32 2021-03-15T22:27:28.373 INFO:tasks.rados.rados.0.smithi076.stdout:2031: finishing rollback tid 2 to smithi07694878-32 2021-03-15T22:27:28.373 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 1458 (ObjNum 682 snap 218 seq_num 682) dirty exists 2021-03-15T22:27:28.374 INFO:tasks.rados.rados.0.smithi076.stdout:2030: done (2 left) 2021-03-15T22:27:28.374 INFO:tasks.rados.rados.0.smithi076.stdout:2031: done (1 left) 2021-03-15T22:27:28.374 INFO:tasks.rados.rados.0.smithi076.stdout:2033: done (0 left) 2021-03-15T22:27:28.375 INFO:tasks.rados.rados.0.smithi076.stdout:2034: snap_create 2021-03-15T22:27:28.393 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:28.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:28.428 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:28.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:28.530 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:28.553 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:28.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:28.555 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:28.656 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:28.678 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:28.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:28.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:28.732 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:28 smithi125 conmon[56739]: cluster 2021-03-15T22:27:27.085207+0000 mgr.y ( 2021-03-15T22:27:28.733 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:28 smithi125 conmon[56739]: mgr.14144) 711 : cluster [DBG] pgmap v924: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 516 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 21 op/s; 211/1686 objects degraded (12.515%) 2021-03-15T22:27:28.733 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:28 smithi125 conmon[56739]: cluster 2021-03-15T22:27:27.365583+0000 mon.a (mon.0) 1475 : cluster [DBG] osdmap e532: 8 total, 7 up, 7 in 2021-03-15T22:27:28.733 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:28 smithi125 conmon[56739]: audit 2021-03-15T22:27:28.145793+0000 mon.a (mon.0) 1476 : audit [DBG] from='client.? 172.21.15.76:0/4200885894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:28.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[62156]: cluster 2021-03-15T22:27:27.085207+0000 mgr.y (mgr.14144 2021-03-15T22:27:28.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[62156]: ) 711 : cluster [DBG] pgmap v924: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 516 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 21 op/s; 211/1686 objects degraded (12.515%) 2021-03-15T22:27:28.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[62156]: cluster 2021-03-15T22:27:27.365583+0000 mon.a (mon.0 2021-03-15T22:27:28.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[62156]: ) 1475 : cluster [DBG] osdmap e532: 8 total, 7 up, 7 in 2021-03-15T22:27:28.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:27:28.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[62156]: -15T22:27:28.145793+0000 mon.a (mon.0) 1476 : audit [DBG] from='client.? 172.21.15.76:0/4200885894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:28.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[56514]: cluster 2021-03-15T22:27:28.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[56514]: 2021-03-15T22:27:28.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[56514]: 2021-03-15T22:27:27. 2021-03-15T22:27:28.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[56514]: 085207+0000 mgr.y (mgr.14144) 711 : cluster [DBG] pgmap v924: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 516 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 13 MiB/s rd, 2.3 MiB/s wr, 21 op/s; 211/1686 objects degraded (12.515%) 2021-03-15T22:27:28.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:27:28.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[56514]: 27:27.365583+0000 mon.a (mon.0) 1475 : cluster [DBG] 2021-03-15T22:27:28.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[56514]: osdmap e532: 8 total, 7 up, 7 in 2021-03-15T22:27:28.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[56514]: audit 2021-03-15T22:27:28 2021-03-15T22:27:28.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[56514]: .145793+0000 mon.a (mon.0) 1476 : audit 2021-03-15T22:27:28.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:28 smithi076 conmon[56514]: [DBG] from='client.? 172.21.15.76:0/4200885894' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:28.782 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:28.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:28.806 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:28.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:28.907 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:28.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:28.930 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:28.930 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:28.999 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:27:28.999 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-03-15T22:27:28.999 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:27:29.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:29.054 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:29.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:29.056 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:29.157 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:29.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:29.180 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:29.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:29.282 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:29.304 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:29.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:29.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:29.382 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:29 smithi125 conmon[56739]: cluster 2021-03-15T22:27:28.367404+0000 mon.a (mon.0) 1477 : cluster [DBG] osdmap e533: 8 total, 7 up, 7 in 2021-03-15T22:27:29.384 INFO:tasks.rados.rados.0.smithi076.stdout:2034: done (0 left) 2021-03-15T22:27:29.384 INFO:tasks.rados.rados.0.smithi076.stdout:2035: read oid 41 snap 220 2021-03-15T22:27:29.385 INFO:tasks.rados.rados.0.smithi076.stdout:2035: expect (ObjNum 689 snap 220 seq_num 689) 2021-03-15T22:27:29.385 INFO:tasks.rados.rados.0.smithi076.stdout:2036: snap_remove snap 210 2021-03-15T22:27:29.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:29.428 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:29.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:29.429 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:29.435 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:27:29.436 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:27:29.531 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:29.552 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:29.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:29.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:29.569 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:27:29.654 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:29.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:29 smithi076 conmon[56514]: cluster 2021-03-15T22:27:28.367404+0000 mon.a ( 2021-03-15T22:27:29.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:29 smithi076 conmon[56514]: mon.0) 1477 : cluster [DBG] osdmap e533: 8 total, 7 up, 7 in 2021-03-15T22:27:29.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:29 smithi076 conmon[62156]: cluster 2021-03-15T22:27:28.367404+0000 2021-03-15T22:27:29.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:29 smithi076 conmon[62156]: mon.a (mon.0) 1477 : cluster [DBG] osdmap e533: 8 total, 7 up, 7 in 2021-03-15T22:27:29.717 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:29.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:29.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:29.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:29.844 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:29.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:29.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:29.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:29.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:29.971 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:29.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:30.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:27:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:27:30.073 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:30.080 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:30.096 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:30.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:30.097 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:30.153 INFO:tasks.rados.rados.0.smithi076.stdout:2036: done (1 left) 2021-03-15T22:27:30.153 INFO:tasks.rados.rados.0.smithi076.stdout:2037: read oid 47 snap -1 2021-03-15T22:27:30.153 INFO:tasks.rados.rados.0.smithi076.stdout:2037: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:27:30.157 INFO:tasks.rados.rados.0.smithi076.stdout:2038: write oid 38 current snap is 229 2021-03-15T22:27:30.157 INFO:tasks.rados.rados.0.smithi076.stdout:2038: seq_num 711 ranges {749777=722868,2156109=471916,3258689=649107} 2021-03-15T22:27:30.162 INFO:tasks.rados.rados.0.smithi076.stdout:2038: writing smithi07694878-38 from 749777 to 1472645 tid 1 2021-03-15T22:27:30.170 INFO:tasks.rados.rados.0.smithi076.stdout:2038: writing smithi07694878-38 from 2156109 to 2628025 tid 2 2021-03-15T22:27:30.178 INFO:tasks.rados.rados.0.smithi076.stdout:2038: writing smithi07694878-38 from 3258689 to 3907796 tid 3 2021-03-15T22:27:30.179 INFO:tasks.rados.rados.0.smithi076.stdout:2039: snap_create 2021-03-15T22:27:30.179 INFO:tasks.rados.rados.0.smithi076.stdout:2035: expect (ObjNum 689 snap 220 seq_num 689) 2021-03-15T22:27:30.199 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:30.207 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:27:30.207 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:27:30.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:30.246 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:30.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:30.321 INFO:tasks.rados.rados.0.smithi076.stdout:2037: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:27:30.348 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:30.370 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:30.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:30.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:30.429 INFO:tasks.rados.rados.0.smithi076.stdout:2038: finishing write tid 1 to smithi07694878-38 2021-03-15T22:27:30.429 INFO:tasks.rados.rados.0.smithi076.stdout:2038: finishing write tid 2 to smithi07694878-38 2021-03-15T22:27:30.430 INFO:tasks.rados.rados.0.smithi076.stdout:2038: finishing write tid 3 to smithi07694878-38 2021-03-15T22:27:30.430 INFO:tasks.rados.rados.0.smithi076.stdout:2038: finishing write tid 4 to smithi07694878-38 2021-03-15T22:27:30.430 INFO:tasks.rados.rados.0.smithi076.stdout:2038: finishing write tid 5 to smithi07694878-38 2021-03-15T22:27:30.430 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 1026 (ObjNum 711 snap 229 seq_num 711) dirty exists 2021-03-15T22:27:30.430 INFO:tasks.rados.rados.0.smithi076.stdout:2038: left oid 38 (ObjNum 711 snap 229 seq_num 711) 2021-03-15T22:27:30.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:30.495 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:30.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:30.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:30.598 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[56514]: cluster 2021-03-15T22:27:29.085877+0000 mgr.y (mgr.14144) 712 : cluster [DBG] pgmap v927: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 528 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 3.2 MiB/s wr, 16 op/s; 211/1704 objects degraded (12.383%) 2021-03-15T22:27:30.598 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[56514]: cluster 2021-03-15T22:27:29.380406+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e534: 8 total, 7 up, 7 in 2021-03-15T22:27:30.599 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[56514]: audit 2021-03-15T22:27:29.826865+0000 mgr.y (mgr.14144) 713 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:30.599 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[56514]: cluster 2021-03-15T22:27:30.139123+0000 mon.a (mon.0) 1479 : cluster [WRN] Health check update: Degraded data redundancy: 211/1704 objects degraded (12.383%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:30.599 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[56514]: cluster 2021-03-15T22:27:30.149319+0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e535: 8 total, 7 up, 7 in 2021-03-15T22:27:30.599 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: cluster 2021-03-15T22:27:29 2021-03-15T22:27:30.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: .085877+0000 mgr.y (mgr.14144) 712 : cluster [DBG] pgmap v927: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 528 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 3.2 MiB/s wr, 16 op/s; 211/1704 objects degraded (12.383%) 2021-03-15T22:27:30.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: cluster 2021-03-15T22:27:29.380406+0000 mon.a ( 2021-03-15T22:27:30.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: mon.0) 1478 : cluster [DBG] osdmap e534: 8 total, 7 up, 7 in 2021-03-15T22:27:30.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:27:30.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: 03-15T22:27:29.826865+0000 mgr.y ( 2021-03-15T22:27:30.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: mgr.14144) 713 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:30.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:27:30.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: T22:27:30.139123+0000 mon.a (mon.0) 1479 : cluster 2021-03-15T22:27:30.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: [WRN] Health check update: Degraded data redundancy: 211/1704 objects degraded (12.383%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:30.602 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: cluster 2021-03-15T22:27:30.149319 2021-03-15T22:27:30.602 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:30 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1480 : cluster [DBG] osdmap e535: 8 total, 7 up, 7 in 2021-03-15T22:27:30.602 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:30.624 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:30.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:30.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:30.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:30.748 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:30.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:30.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:30.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: cluster 2021-03-15T22:27:29.085877+0000 mgr.y (mgr.14144) 712 : cluster 2021-03-15T22:27:30.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: [DBG] pgmap v927: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 528 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 8.3 MiB/s rd, 3.2 MiB/s wr, 16 op/s; 211/1704 objects degraded (12.383%) 2021-03-15T22:27:30.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: cluster 2021-03-15T22:27:29.380406+0000 mon.a (mon.0) 1478 : cluster [DBG] osdmap e534: 8 total, 7 up, 7 in 2021-03-15T22:27:30.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: audit 2021-03-15T22:27:29.826865+0000 mgr.y (mgr.14144) 713 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:30.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: 2021-03-15T22:27:30.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: cluster 2021-03-15T22:27:30.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: 2021-03-15T22:27: 2021-03-15T22:27:30.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: 30.139123+0000 mon.a (mon 2021-03-15T22:27:30.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: .0) 1479 : cluster [WRN] 2021-03-15T22:27:30.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: Health check update: Degraded data redundancy: 211/1704 objects degraded (12.383%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:30.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: cluster 2021-03-15T22:27:30.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: 2021-03-15T 2021-03-15T22:27:30.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: 22:27:30.149319+0000 2021-03-15T22:27:30.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: mon.a (mon.0) 1480 2021-03-15T22:27:30.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:30 smithi125 conmon[56739]: : cluster [DBG] osdmap e535: 8 total, 7 up, 7 in 2021-03-15T22:27:30.852 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:30.875 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:30.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:30.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:30.977 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:30.999 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:31.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:31.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:31.101 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:31.123 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:31.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:31.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:31.165 INFO:tasks.rados.rados.0.smithi076.stdout:2035: done (3 left) 2021-03-15T22:27:31.166 INFO:tasks.rados.rados.0.smithi076.stdout:2037: done (2 left) 2021-03-15T22:27:31.166 INFO:tasks.rados.rados.0.smithi076.stdout:2038: done (1 left) 2021-03-15T22:27:31.166 INFO:tasks.rados.rados.0.smithi076.stdout:2039: done (0 left) 2021-03-15T22:27:31.167 INFO:tasks.rados.rados.0.smithi076.stdout:2040: rollback oid 8 current snap is 230 2021-03-15T22:27:31.167 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 213 2021-03-15T22:27:31.167 INFO:tasks.rados.rados.0.smithi076.stdout:2041: snap_create 2021-03-15T22:27:31.170 INFO:tasks.rados.rados.0.smithi076.stdout:2040: finishing rollback tid 0 to smithi07694878-8 2021-03-15T22:27:31.170 INFO:tasks.rados.rados.0.smithi076.stdout:2040: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:27:31.170 INFO:tasks.rados.rados.0.smithi076.stdout:2040: finishing rollback tid 2 to smithi07694878-8 2021-03-15T22:27:31.170 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1188 (ObjNum 649 snap 201 seq_num 649) dirty exists 2021-03-15T22:27:31.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:31.248 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:31.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:31.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:31.351 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:31.372 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:31.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:31.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:31.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:27:31.475 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:31.502 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:31.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:31.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:31.604 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:31.627 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:31.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:31.629 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:31.693 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:27:31.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:31.750 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:31.752 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:31.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:31.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:31.855 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:31.878 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:31.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:31.880 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:31.981 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:32.005 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:32.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:32.007 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:32.015 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:32.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:32.133 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:32.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:32.135 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:32.168 INFO:tasks.rados.rados.0.smithi076.stdout:2040: done (1 left) 2021-03-15T22:27:32.168 INFO:tasks.rados.rados.0.smithi076.stdout:2041: done (0 left) 2021-03-15T22:27:32.169 INFO:tasks.rados.rados.0.smithi076.stdout:2042: write oid 27 current snap is 231 2021-03-15T22:27:32.169 INFO:tasks.rados.rados.0.smithi076.stdout:2042: seq_num 712 ranges {766584=798117,2098238=615574,3169759=460379} 2021-03-15T22:27:32.178 INFO:tasks.rados.rados.0.smithi076.stdout:2042: writing smithi07694878-27 from 766584 to 1564701 tid 1 2021-03-15T22:27:32.185 INFO:tasks.rados.rados.0.smithi076.stdout:2042: writing smithi07694878-27 from 2098238 to 2713812 tid 2 2021-03-15T22:27:32.191 INFO:tasks.rados.rados.0.smithi076.stdout:2042: writing smithi07694878-27 from 3169759 to 3630138 tid 3 2021-03-15T22:27:32.192 INFO:tasks.rados.rados.0.smithi076.stdout:2043: read oid 15 snap -1 2021-03-15T22:27:32.192 INFO:tasks.rados.rados.0.smithi076.stdout:2043: expect (ObjNum 649 snap 201 seq_num 649) 2021-03-15T22:27:32.192 INFO:tasks.rados.rados.0.smithi076.stdout:2044: write oid 44 current snap is 231 2021-03-15T22:27:32.192 INFO:tasks.rados.rados.0.smithi076.stdout:2044: seq_num 713 ranges {783391=473366,1756720=475585,2797182=788004} 2021-03-15T22:27:32.201 INFO:tasks.rados.rados.0.smithi076.stdout:2044: writing smithi07694878-44 from 783391 to 1256757 tid 1 2021-03-15T22:27:32.208 INFO:tasks.rados.rados.0.smithi076.stdout:2044: writing smithi07694878-44 from 1756720 to 2232305 tid 2 2021-03-15T22:27:32.219 INFO:tasks.rados.rados.0.smithi076.stdout:2044: writing smithi07694878-44 from 2797182 to 3585186 tid 3 2021-03-15T22:27:32.219 INFO:tasks.rados.rados.0.smithi076.stdout:2045: rollback oid 6 current snap is 231 2021-03-15T22:27:32.219 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 229 2021-03-15T22:27:32.219 INFO:tasks.rados.rados.0.smithi076.stdout:2042: finishing write tid 1 to smithi07694878-27 2021-03-15T22:27:32.220 INFO:tasks.rados.rados.0.smithi076.stdout:2042: finishing write tid 2 to smithi07694878-27 2021-03-15T22:27:32.220 INFO:tasks.rados.rados.0.smithi076.stdout:2042: finishing write tid 3 to smithi07694878-27 2021-03-15T22:27:32.221 INFO:tasks.rados.rados.0.smithi076.stdout:2042: finishing write tid 4 to smithi07694878-27 2021-03-15T22:27:32.221 INFO:tasks.rados.rados.0.smithi076.stdout:2043: expect (ObjNum 649 snap 201 seq_num 649) 2021-03-15T22:27:32.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:32.261 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:32.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:32.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:32.333 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:32 smithi076 conmon[56514]: cluster 2021-03-15T22:27:31.086335+0000 mgr.y (mgr.14144) 714 : cluster [DBG] pgmap v930: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 545 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.7 MiB/s rd, 4.7 MiB/s wr, 15 op/s; 211/1713 objects degraded (12.318%) 2021-03-15T22:27:32.334 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:32 smithi076 conmon[56514]: cluster 2021-03-15T22:27:31.162980+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e536: 8 total, 7 up, 7 in 2021-03-15T22:27:32.334 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:32 smithi076 conmon[62156]: cluster 2021-03-15T22:27:31.086335+0000 mgr.y (mgr.14144) 714 : cluster [DBG] pgmap v930: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 545 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.7 MiB/s rd, 4.7 MiB/s wr, 15 op/s; 211/1713 objects degraded (12.318%) 2021-03-15T22:27:32.335 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:32 smithi076 conmon[62156]: cluster 2021-03-15T22:27:31.162980+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e536: 8 total, 7 up, 7 in 2021-03-15T22:27:32.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:32.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:32.387 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:32.387 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:32.393 INFO:tasks.rados.rados.0.smithi076.stdout:2042: finishing write tid 5 to smithi07694878-27 2021-03-15T22:27:32.393 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1186 (ObjNum 712 snap 231 seq_num 712) dirty exists 2021-03-15T22:27:32.394 INFO:tasks.rados.rados.0.smithi076.stdout:2042: left oid 27 (ObjNum 712 snap 231 seq_num 712) 2021-03-15T22:27:32.394 INFO:tasks.rados.rados.0.smithi076.stdout:2044: finishing write tid 1 to smithi07694878-44 2021-03-15T22:27:32.394 INFO:tasks.rados.rados.0.smithi076.stdout:2044: finishing write tid 2 to smithi07694878-44 2021-03-15T22:27:32.395 INFO:tasks.rados.rados.0.smithi076.stdout:2045: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:27:32.395 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 1201 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:27:32.395 INFO:tasks.rados.rados.0.smithi076.stdout:2044: finishing write tid 3 to smithi07694878-44 2021-03-15T22:27:32.396 INFO:tasks.rados.rados.0.smithi076.stdout:2042: done (3 left) 2021-03-15T22:27:32.396 INFO:tasks.rados.rados.0.smithi076.stdout:2043: done (2 left) 2021-03-15T22:27:32.396 INFO:tasks.rados.rados.0.smithi076.stdout:2045: done (1 left) 2021-03-15T22:27:32.396 INFO:tasks.rados.rados.0.smithi076.stdout:2046: read oid 33 snap -1 2021-03-15T22:27:32.396 INFO:tasks.rados.rados.0.smithi076.stdout:2046: expect (ObjNum 640 snap 200 seq_num 640) 2021-03-15T22:27:32.397 INFO:tasks.rados.rados.0.smithi076.stdout:2047: read oid 24 snap 227 2021-03-15T22:27:32.397 INFO:tasks.rados.rados.0.smithi076.stdout:2047: expect (ObjNum 638 snap 200 seq_num 638) 2021-03-15T22:27:32.397 INFO:tasks.rados.rados.0.smithi076.stdout:2048: snap_remove snap 225 2021-03-15T22:27:32.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:32.510 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:32.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:32.511 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:32 smithi125 conmon[56739]: cluster 2021-03-15T22:27:31.086335+0000 mgr.y (mgr.14144) 714 : cluster [DBG] pgmap v930: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 545 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.7 MiB/s rd, 4.7 MiB/s wr, 15 op/s; 211/1713 objects degraded (12.318%) 2021-03-15T22:27:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:32 smithi125 conmon[56739]: cluster 2021-03-15T22:27:31.162980+0000 mon.a (mon.0) 1481 : cluster [DBG] osdmap e536: 8 total, 7 up, 7 in 2021-03-15T22:27:32.613 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:32.636 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:32.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:32.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:32.738 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:32.761 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:32.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:32.863 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:32.884 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:32.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:32.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:32.987 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:27:33.011 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:33.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:33.012 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:27:33.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:33.138 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:33.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:33.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:33.185 INFO:tasks.rados.rados.0.smithi076.stdout:2048: done (3 left) 2021-03-15T22:27:33.185 INFO:tasks.rados.rados.0.smithi076.stdout:2049: write (excl) oid 15 current snap is 231 2021-03-15T22:27:33.185 INFO:tasks.rados.rados.0.smithi076.stdout:2044: finishing write tid 4 to smithi07694878-44 2021-03-15T22:27:33.186 INFO:tasks.rados.rados.0.smithi076.stdout:2049: seq_num 714 ranges {400198=548615,1415202=584996} 2021-03-15T22:27:33.191 INFO:tasks.rados.rados.0.smithi076.stdout:2049: writing smithi07694878-15 from 400198 to 948813 tid 1 2021-03-15T22:27:33.197 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:27:33.198 INFO:tasks.rados.rados.0.smithi076.stdout:2049: writing smithi07694878-15 from 1415202 to 2000198 tid 2 2021-03-15T22:27:33.198 INFO:tasks.rados.rados.0.smithi076.stdout:2050: write (excl) oid 26 current snap is 231 2021-03-15T22:27:33.199 INFO:tasks.rados.rados.0.smithi076.stdout:2044: finishing write tid 5 to smithi07694878-44 2021-03-15T22:27:33.199 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1190 (ObjNum 713 snap 231 seq_num 713) dirty exists 2021-03-15T22:27:33.199 INFO:tasks.rados.rados.0.smithi076.stdout:2044: left oid 44 (ObjNum 713 snap 231 seq_num 713) 2021-03-15T22:27:33.199 INFO:tasks.rados.rados.0.smithi076.stdout:2050: seq_num 715 ranges {417005=740217,1590037=426968} 2021-03-15T22:27:33.205 INFO:tasks.rados.rados.0.smithi076.stdout:2050: writing smithi07694878-26 from 417005 to 1157222 tid 1 2021-03-15T22:27:33.211 INFO:tasks.rados.rados.0.smithi076.stdout:2050: writing smithi07694878-26 from 1590037 to 2017005 tid 2 2021-03-15T22:27:33.212 INFO:tasks.rados.rados.0.smithi076.stdout:2044: done (4 left) 2021-03-15T22:27:33.212 INFO:tasks.rados.rados.0.smithi076.stdout:2051: read oid 22 snap -1 2021-03-15T22:27:33.212 INFO:tasks.rados.rados.0.smithi076.stdout:2051: expect (ObjNum 674 snap 214 seq_num 674) 2021-03-15T22:27:33.213 INFO:tasks.rados.rados.0.smithi076.stdout:2052: copy_from oid 7 from oid 11 current snap is 231 2021-03-15T22:27:33.213 INFO:tasks.rados.rados.0.smithi076.stdout:2053: copy_from oid 27 from oid 14 current snap is 231 2021-03-15T22:27:33.216 INFO:tasks.rados.rados.0.smithi076.stdout:2047: expect (ObjNum 638 snap 200 seq_num 638) 2021-03-15T22:27:33.242 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:33.271 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:33.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:33.272 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:33.326 INFO:tasks.rados.rados.0.smithi076.stdout:2046: expect (ObjNum 640 snap 200 seq_num 640) 2021-03-15T22:27:33.374 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:33.396 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:33.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:33.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:33.450 INFO:tasks.rados.rados.0.smithi076.stdout:2049: finishing write tid 1 to smithi07694878-15 2021-03-15T22:27:33.450 INFO:tasks.rados.rados.0.smithi076.stdout:2049: finishing write tid 2 to smithi07694878-15 2021-03-15T22:27:33.450 INFO:tasks.rados.rados.0.smithi076.stdout:2049: finishing write tid 3 to smithi07694878-15 2021-03-15T22:27:33.451 INFO:tasks.rados.rados.0.smithi076.stdout:2049: finishing write tid 4 to smithi07694878-15 2021-03-15T22:27:33.451 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 1352 (ObjNum 714 snap 231 seq_num 714) dirty exists 2021-03-15T22:27:33.451 INFO:tasks.rados.rados.0.smithi076.stdout:2049: left oid 15 (ObjNum 714 snap 231 seq_num 714) 2021-03-15T22:27:33.451 INFO:tasks.rados.rados.0.smithi076.stdout:2050: finishing write tid 1 to smithi07694878-26 2021-03-15T22:27:33.451 INFO:tasks.rados.rados.0.smithi076.stdout:2050: finishing write tid 2 to smithi07694878-26 2021-03-15T22:27:33.452 INFO:tasks.rados.rados.0.smithi076.stdout:2050: finishing write tid 3 to smithi07694878-26 2021-03-15T22:27:33.452 INFO:tasks.rados.rados.0.smithi076.stdout:2053: finishing copy_from to smithi07694878-27 2021-03-15T22:27:33.452 INFO:tasks.rados.rados.0.smithi076.stdout:2053: got expected ENOENT (src dne) 2021-03-15T22:27:33.452 INFO:tasks.rados.rados.0.smithi076.stdout:2053: finishing copy_from racing read to smithi07694878-27 2021-03-15T22:27:33.453 INFO:tasks.rados.rados.0.smithi076.stdout:2050: finishing write tid 4 to smithi07694878-26 2021-03-15T22:27:33.453 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 811 (ObjNum 715 snap 231 seq_num 715) dirty exists 2021-03-15T22:27:33.453 INFO:tasks.rados.rados.0.smithi076.stdout:2050: left oid 26 (ObjNum 715 snap 231 seq_num 715) 2021-03-15T22:27:33.453 INFO:tasks.rados.rados.0.smithi076.stdout:2046: done (6 left) 2021-03-15T22:27:33.453 INFO:tasks.rados.rados.0.smithi076.stdout:2047: done (5 left) 2021-03-15T22:27:33.454 INFO:tasks.rados.rados.0.smithi076.stdout:2049: done (4 left) 2021-03-15T22:27:33.454 INFO:tasks.rados.rados.0.smithi076.stdout:2050: done (3 left) 2021-03-15T22:27:33.454 INFO:tasks.rados.rados.0.smithi076.stdout:2053: done (2 left) 2021-03-15T22:27:33.454 INFO:tasks.rados.rados.0.smithi076.stdout:2054: read oid 36 snap -1 2021-03-15T22:27:33.455 INFO:tasks.rados.rados.0.smithi076.stdout:2054: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:27:33.455 INFO:tasks.rados.rados.0.smithi076.stdout:2055: read oid 48 snap -1 2021-03-15T22:27:33.455 INFO:tasks.rados.rados.0.smithi076.stdout:2055: expect (ObjNum 641 snap 200 seq_num 641) 2021-03-15T22:27:33.455 INFO:tasks.rados.rados.0.smithi076.stdout:2056: write (excl) oid 13 current snap is 231 2021-03-15T22:27:33.455 INFO:tasks.rados.rados.0.smithi076.stdout:2051: expect (ObjNum 674 snap 214 seq_num 674) 2021-03-15T22:27:33.498 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:33.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[56514]: cluster 2021-03-15T22:27:32.164313+0000 mon.a 2021-03-15T22:27:33.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[56514]: (mon.0) 1482 : cluster [DBG] osdmap e537: 8 total, 7 up, 7 in 2021-03-15T22:27:33.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[56514]: audit 2021-03-15T22:27:32.206362+0000 mon.c (mon.1) 45 : audit [INF] from='client.? 172.21.15.76:0/16228789' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:27:33.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[56514]: audit 2021-03-15T22:27:32.206740+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:27:33.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[62156]: cluster 2021-03-15T22:27:32 2021-03-15T22:27:33.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[62156]: .164313+0000 mon.a (mon.0) 1482 : cluster [DBG] osdmap e537: 8 total, 7 up, 7 in 2021-03-15T22:27:33.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:27:33.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[62156]: 15T22:27:32.206362 2021-03-15T22:27:33.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[62156]: +0000 mon.c (mon.1) 45 2021-03-15T22:27:33.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/16228789' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:27:33.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:27:33.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[62156]: 15T22:27:32.206740+0000 2021-03-15T22:27:33.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[62156]: mon.a (mon.0) 1483 : 2021-03-15T22:27:33.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:33 smithi076 conmon[62156]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:27:33.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:33.523 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:33.523 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:33.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:33 smithi125 conmon[56739]: cluster 2021-03-15T22:27:32.164313+0000 mon.a (mon 2021-03-15T22:27:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:33 smithi125 conmon[56739]: .0) 1482 : cluster [DBG] osdmap e537: 8 total, 7 up, 7 in 2021-03-15T22:27:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:33 smithi125 conmon[56739]: audit 2021-03-15T22:27:32.206362+0000 mon.c (mon.1) 45 : audit [INF] from='client.? 172.21.15.76:0/16228789' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:27:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:33 smithi125 conmon[56739]: audit 2021-03-15T22:27:32.206740+0000 mon.a (mon.0) 1483 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:27:33.557 INFO:tasks.rados.rados.0.smithi076.stdout:2052: finishing copy_from to smithi07694878-7 2021-03-15T22:27:33.557 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 506 (ObjNum 702 snap 226 seq_num 702) dirty exists 2021-03-15T22:27:33.557 INFO:tasks.rados.rados.0.smithi076.stdout:2052: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:27:33.558 INFO:tasks.rados.rados.0.smithi076.stdout:2054: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:27:33.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:33.646 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:33.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:33.648 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:33.709 INFO:tasks.rados.rados.0.smithi076.stdout:2055: expect (ObjNum 641 snap 200 seq_num 641) 2021-03-15T22:27:33.749 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:33.771 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:33.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:33.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:33.801 INFO:tasks.rados.rados.0.smithi076.stdout:2056: seq_num 716 ranges {433812=415466,1648519=385293} 2021-03-15T22:27:33.805 INFO:tasks.rados.rados.0.smithi076.stdout:2056: writing smithi07694878-13 from 433812 to 849278 tid 1 2021-03-15T22:27:33.812 INFO:tasks.rados.rados.0.smithi076.stdout:2056: writing smithi07694878-13 from 1648519 to 2033812 tid 2 2021-03-15T22:27:33.812 INFO:tasks.rados.rados.0.smithi076.stdout:2051: done (4 left) 2021-03-15T22:27:33.813 INFO:tasks.rados.rados.0.smithi076.stdout:2052: done (3 left) 2021-03-15T22:27:33.813 INFO:tasks.rados.rados.0.smithi076.stdout:2054: done (2 left) 2021-03-15T22:27:33.813 INFO:tasks.rados.rados.0.smithi076.stdout:2055: done (1 left) 2021-03-15T22:27:33.813 INFO:tasks.rados.rados.0.smithi076.stdout:2057: snap_create 2021-03-15T22:27:33.813 INFO:tasks.rados.rados.0.smithi076.stdout:2056: finishing write tid 1 to smithi07694878-13 2021-03-15T22:27:33.814 INFO:tasks.rados.rados.0.smithi076.stdout:2056: finishing write tid 2 to smithi07694878-13 2021-03-15T22:27:33.815 INFO:tasks.rados.rados.0.smithi076.stdout:2056: finishing write tid 3 to smithi07694878-13 2021-03-15T22:27:33.815 INFO:tasks.rados.rados.0.smithi076.stdout:2056: finishing write tid 4 to smithi07694878-13 2021-03-15T22:27:33.815 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 532 (ObjNum 716 snap 231 seq_num 716) dirty exists 2021-03-15T22:27:33.815 INFO:tasks.rados.rados.0.smithi076.stdout:2056: left oid 13 (ObjNum 716 snap 231 seq_num 716) 2021-03-15T22:27:33.873 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:33.896 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:33.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:33.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:33.999 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:34.022 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:34.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:34.125 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:34.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:34.148 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:34.148 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:34.189 INFO:tasks.rados.rados.0.smithi076.stdout:2056: done (1 left) 2021-03-15T22:27:34.189 INFO:tasks.rados.rados.0.smithi076.stdout:2057: done (0 left) 2021-03-15T22:27:34.190 INFO:tasks.rados.rados.0.smithi076.stdout:2058: copy_from oid 24 from oid 5 current snap is 232 2021-03-15T22:27:34.190 INFO:tasks.rados.rados.0.smithi076.stdout:2059: snap_create 2021-03-15T22:27:34.209 INFO:tasks.rados.rados.0.smithi076.stdout:2058: finishing copy_from to smithi07694878-24 2021-03-15T22:27:34.210 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 608 (ObjNum 668 snap 212 seq_num 668) dirty exists 2021-03-15T22:27:34.210 INFO:tasks.rados.rados.0.smithi076.stdout:2058: finishing copy_from racing read to smithi07694878-24 2021-03-15T22:27:34.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:34.319 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:34.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:34.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:34.422 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:34.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[56514]: cluster 2021-03-15T22:27:33.086867+0000 mgr.y (mgr.14144 2021-03-15T22:27:34.433 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[56514]: ) 715 : cluster [DBG] pgmap v933: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 548 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.4 MiB/s rd, 2.8 MiB/s wr, 14 op/s; 213/1722 objects degraded (12.369%) 2021-03-15T22:27:34.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[56514]: cluster 2021-03-15T22:27:33.173745+0000 mon.a (mon.0) 1484 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:34.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[56514]: audit 2021-03-15T22:27:33.182215+0000 mon.a (mon.0) 1485 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:27:34.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[56514]: cluster 2021-03-15T22:27:33.182286+0000 mon.a (mon.0) 1486 : cluster [DBG] osdmap e538: 8 total, 7 up, 7 in 2021-03-15T22:27:34.434 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[56514]: cluster 2021-03-15T22:27:33.386693+0000 osd.7 (osd.7) 104 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:27:34.435 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[56514]: cluster 2021-03-15T22:27:33.481541+0000 osd.1 (osd.1) 103 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:27:34.435 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: cluster 2021-03-15T22:27:33 2021-03-15T22:27:34.435 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: .086867+0000 mgr.y (mgr.14144) 715 : cluster [DBG] pgmap v933: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 548 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.4 MiB/s rd, 2.8 MiB/s wr, 14 op/s; 213/1722 objects degraded (12.369%) 2021-03-15T22:27:34.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: cluster 2021-03-15T22:27:33.173745+0000 mon.a (mon.0) 1484 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:34.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: audit 2021-03-15T22:27:34.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: 2021-03-15T22:27 2021-03-15T22:27:34.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: :33.182215+0000 mon.a (mon.0) 1485 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:27:34.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: cluster 2021-03-15T22:27:34.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: 2021-03-15T22:27:33.182286+0000 2021-03-15T22:27:34.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: mon.a (mon.0) 1486 : cluster [DBG] 2021-03-15T22:27:34.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: osdmap e538: 8 total, 7 up, 7 in 2021-03-15T22:27:34.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:27:34.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: 22:27:33.386693+0000 osd.7 (osd 2021-03-15T22:27:34.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: .7) 104 : cluster [DBG] 2021-03-15T22:27:34.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: 3.11 scrub ok 2021-03-15T22:27:34.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:27:34.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: :27:33.481541+0000 osd.1 (osd.1) 103 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:27:34.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:34 smithi076 conmon[62156]: 2021-03-15T22:27:34.446 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:34.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:34.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:34.533 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:27:34.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: cluster 2021-03-15T22:27:33.086867+0000 mgr.y (mgr.14144) 715 : cluster 2021-03-15T22:27:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: [DBG] pgmap v933: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 548 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 9.4 MiB/s rd, 2.8 MiB/s wr, 14 op/s; 213/1722 objects degraded (12.369%) 2021-03-15T22:27:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: cluster 2021-03-15T22:27:33.173745+0000 mon.a (mon.0) 1484 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: audit 2021-03-15T22:27:33.182215+0000 2021-03-15T22:27:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: mon.a (mon.0) 1485 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:27:34.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:27:34.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: 15T22:27:33.182286+0000 mon.a (mon.0) 1486 2021-03-15T22:27:34.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: : cluster [DBG] osdmap e538: 8 total, 7 up, 7 in 2021-03-15T22:27:34.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:27:34.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: T22:27:33.386693+0000 osd.7 (osd 2021-03-15T22:27:34.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: .7) 104 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:27:34.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: cluster 2021-03-15T22:27:34.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: 2021-03-15T22:27:33 2021-03-15T22:27:34.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: .481541+0000 osd.1 (osd.1) 103 : 2021-03-15T22:27:34.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[56739]: cluster [DBG] 4.2 scrub ok 2021-03-15T22:27:34.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:34.552 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:27:34.569 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:27:34.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:34.576 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:34.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:34.586 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:27:34.603 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:27:34.620 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:27:34.638 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:27:34.639 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:27:34.639 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:27:34.639 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:27:34.640 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:27:34.640 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:27:34.640 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:27:34.640 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:27:34.641 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:27:34.657 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [2] live_osds: [0, 1, 3, 4, 5, 6, 7] 2021-03-15T22:27:34.658 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:27:34.658 INFO:tasks.thrashosds.thrasher:Killing osd 5, live_osds are [0, 1, 3, 4, 5, 6, 7] 2021-03-15T22:27:34.658 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2021-03-15T22:27:34.659 DEBUG:teuthology.orchestra.run.smithi125:> sudo systemctl stop ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.5 2021-03-15T22:27:34.678 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:34.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:34.702 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:34.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:34.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:34.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:34.828 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:34.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:34.930 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:34.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:34.954 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:34.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:34.997 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:27:34.997 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:27:34.997 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:27:34.998 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:27:34.998 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:27:35.036 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:27:34 smithi125 systemd[1]: Stopping Ceph osd.5 for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:27:35.037 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[63905]: debug 2021-03-15T22:27:34.799+0000 7fce1fca0700 -1 received signal: Terminated from /dev/init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug (PID: 1) UID: 0 2021-03-15T22:27:35.037 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[63905]: debug 2021-03-15T22:27:34.799+0000 7fce1fca0700 -1 osd.5 539 *** Got signal Terminated *** 2021-03-15T22:27:35.037 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:27:34 smithi125 conmon[63905]: debug 2021-03-15T22:27:34.799+0000 7fce1fca0700 -1 osd.5 539 *** Immediate shutdown (osd_fast_shutdown=true) *** 2021-03-15T22:27:35.056 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:35.078 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:35.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:35.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:35.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:35.207 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:35.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:35.209 INFO:tasks.rados.rados.0.smithi076.stdout:2058: done (1 left) 2021-03-15T22:27:35.209 INFO:tasks.rados.rados.0.smithi076.stdout:2059: done (0 left) 2021-03-15T22:27:35.210 INFO:tasks.rados.rados.0.smithi076.stdout:2060: write oid 38 current snap is 233 2021-03-15T22:27:35.210 INFO:tasks.rados.rados.0.smithi076.stdout:2060: seq_num 717 ranges {450619=490715,1590648=459971} 2021-03-15T22:27:35.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:35.213 INFO:tasks.rados.rados.0.smithi076.stdout:2060: writing smithi07694878-38 from 450619 to 941334 tid 1 2021-03-15T22:27:35.221 INFO:tasks.rados.rados.0.smithi076.stdout:2060: writing smithi07694878-38 from 1590648 to 2050619 tid 2 2021-03-15T22:27:35.221 INFO:tasks.rados.rados.0.smithi076.stdout:2061: snap_remove snap 219 2021-03-15T22:27:35.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:35.334 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:35.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:35.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:35.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:35.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22:27:34.103336+0000 osd.3 (osd 2021-03-15T22:27:35.454 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: .3) 91 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:27:35.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22:27:34.187509+0000 mon.a (mon.0) 1487 : cluster [DBG] 2021-03-15T22:27:35.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: osdmap e539: 8 total, 7 up, 7 in 2021-03-15T22:27:35.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:27:35.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 15T22:27:34.513854+0000 mon.a (mon.0) 1488 : 2021-03-15T22:27:35.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: audit [DBG] from='client.? 172.21.15.76:0/183012678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:35.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:27:35.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 22:27:34.828073+0000 mon.a (mon. 2021-03-15T22:27:35.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 0) 1489 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:27:35.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 03-15T22:27:34.828108+0000 2021-03-15T22:27:35.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: mon.a (mon.0) 1490 : cluster [INF] osd.5 failed (root=default,host=smithi125) (connection refused reported by osd.7) 2021-03-15T22:27:35.457 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 2021-03-15T22:27:35.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22:27 2021-03-15T22:27:35.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: :34.828307+0000 mon.a (mon.0) 2021-03-15T22:27:35.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 1491 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.458 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:27:35.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 03-15T22:27:34.828385+0000 2021-03-15T22:27:35.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: mon.a (mon.0) 1492 : cluster [DBG] 2021-03-15T22:27:35.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:27:35.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 27:34.828489+0000 mon.a (mon.0) 2021-03-15T22:27:35.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 1493 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:27:35.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: -03-15T22:27:34.828611+0000 2021-03-15T22:27:35.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: mon.a (mon.0) 1494 : cluster [DBG] 2021-03-15T22:27:35.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:27:35.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 27:34.828695+0000 mon.a (mon.0) 2021-03-15T22:27:35.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 1495 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.462 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:27:35.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 03-15T22:27:34.828798+0000 mon.a 2021-03-15T22:27:35.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: (mon.0) 1496 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 2021-03-15T22:27:35.463 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22:27:35.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 2021-03-15T22:27:34.828919+0000 mon.a 2021-03-15T22:27:35.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: (mon.0) 1497 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.464 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:27:35.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: -03-15T22:27:35.028319+0000 2021-03-15T22:27:35.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: mon.a (mon.0) 1498 : cluster [DBG] 2021-03-15T22:27:35.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: osd.5 reported immediately failed by osd.3 2021-03-15T22:27:35.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22:27:35 2021-03-15T22:27:35.465 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: .028543+0000 mon.a (mon.0) 1499 : 2021-03-15T22:27:35.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:27:35.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 27:35.028732+0000 mon.a (mon.0 2021-03-15T22:27:35.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: ) 1500 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:27:35.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: -15T22:27:35.028909+0000 mon.a 2021-03-15T22:27:35.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: (mon.0) 1501 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-15T22:27:35.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22:27:35.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15T22:27:35.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021 2021-03-15T22:27:35.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: - 2021-03-15T22:27:35.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 03 2021-03-15T22:27:35.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: -15T22:27:34 2021-03-15T22:27:35.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: .103336+0000 osd.3 (osd.3) 91 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:27:35.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22:27:35.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15T22:27:35.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15 2021-03-15T22:27:35.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: T22:27:34.187509+0000 mon.a (mon.0) 1487 2021-03-15T22:27:35.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: : cluster [DBG] osdmap e539: 8 total, 7 up, 7 in 2021-03-15T22:27:35.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: audit 2021-03-15T22:27:35.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15T22 2021-03-15T22:27:35.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: :27:34.513854+0000 mon.a (mon.0) 1488 : audit [DBG] from='client.? 172.21.15.76:0/183012678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:35.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:27:35.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: :27:34.828073+0000 mon.a (mon.0) 1489 2021-03-15T22:27:35.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:27:35.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: -03-15T22:27:34.828108+0000 mon.a ( 2021-03-15T22:27:35.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.0) 1490 : cluster [INF] osd.5 failed (root=default,host=smithi125) (connection refused reported by osd.7) 2021-03-15T22:27:35.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22:27: 2021-03-15T22:27:35.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 34.828307+0000 mon.a (mon.0) 1491 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22:27:34.828385+0000 2021-03-15T22:27:35.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.a (mon.0) 1492 : cluster [DBG] 2021-03-15T22:27:35.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22:27:35.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15T 2021-03-15T22:27:35.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 22:27:34.828489+0000 mon.a ( 2021-03-15T22:27:35.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.0) 1493 : cluster [DBG] 2021-03-15T22:27:35.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:27:35.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 03-15T22:27:34.828611+0000 mon.a ( 2021-03-15T22:27:35.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.0) 1494 : cluster [DBG] 2021-03-15T22:27:35.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:27:35.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 15T22:27:34.828695+0000 mon.a (mon. 2021-03-15T22:27:35.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 0) 1495 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:27:35.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: -15T22:27:34.828798+0000 mon.a ( 2021-03-15T22:27:35.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.0) 1496 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.479 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:27:35.479 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 03-15T22:27:34.828919+0000 2021-03-15T22:27:35.480 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.a (mon.0) 1497 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:27:35.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: -03-15T22:27: 2021-03-15T22:27:35.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 35.028319+0000 mon.a (mon 2021-03-15T22:27:35.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: .0) 1498 : cluster [DBG] 2021-03-15T22:27:35.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: osd.5 reported immediately failed by osd.3 2021-03-15T22:27:35.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:27:35.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: -15T22:27:35 2021-03-15T22:27:35.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: .028543+0000 mon.a (mon. 2021-03-15T22:27:35.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 0) 1499 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15T22:27:35.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:27:35.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: T22:27:35.028732+0000 2021-03-15T22:27:35.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 2021-03-15T22:27:35.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22:27:35.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 2021-03-15T22:27:35.029042+0000 mon.a ( 2021-03-15T22:27:35.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: mon.0) 1502 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:27:35.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 03-15T22:27:35.029176+0000 2021-03-15T22:27:35.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: mon.a (mon.0) 1503 : cluster [DBG] 2021-03-15T22:27:35.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22:27:35 2021-03-15T22:27:35.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: .029281+0000 mon.a (mon.0) 1504 : 2021-03-15T22:27:35.488 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-15T22:27:35.488 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:27:35.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: :27:35.029382+0000 mon.a (mon.0 2021-03-15T22:27:35.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: ) 1505 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:27:35.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 15T22:27:35.029476+0000 mon.a 2021-03-15T22:27:35.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: (mon.0) 1506 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 2021-03-15T22:27:35.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22:27:35.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 2021-03-15T22:27:35.029583+0000 mon.a (mon 2021-03-15T22:27:35.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: .0) 1507 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:27:35.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: -15T22:27:35.029679+0000 mon.a (mon 2021-03-15T22:27:35.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: .0) 1508 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-15T22:27:35.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:27:35.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: -15T22:27:35.029814+0000 2021-03-15T22:27:35.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: mon.a (mon.0) 1509 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 2021-03-15T22:27:35.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22:27:35.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 2021-03-15T22:27:35.096586+0000 2021-03-15T22:27:35.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: mon.a (mon.0) 1510 : cluster [DBG] 2021-03-15T22:27:35.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: osd.5 reported immediately failed by osd.3 2021-03-15T22:27:35.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:27:35.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: :27:35.096827+0000 mon.a (mon.0 2021-03-15T22:27:35.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: ) 1511 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:27:35.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 03-15T22:27:35.110569+0000 2021-03-15T22:27:35.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: mon.a (mon.0) 1512 : cluster [DBG] 2021-03-15T22:27:35.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: cluster 2021-03-15T22:27:35.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:27:35.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: :27:35.141403+0000 mon.a (mon.0) 1513 : cluster [WRN] 2021-03-15T22:27:35.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[62156]: Health check update: Degraded data redundancy: 213/1722 objects degraded (12.369%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:35.505 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.a (mon.0) 1500 : 2021-03-15T22:27:35.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22:27:35.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15T22:27 2021-03-15T22:27:35.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: :35.028909+0000 mon.a ( 2021-03-15T22:27:35.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.0) 1501 : cluster 2021-03-15T22:27:35.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: [DBG] osd.5 reported immediately failed by osd.4 2021-03-15T22:27:35.512 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22:27:35.512 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:27:35.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 27:35.029042+0000 mon.a 2021-03-15T22:27:35.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: (mon.0) 1502 : cluster 2021-03-15T22:27:35.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22:27:35.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15T22:27 2021-03-15T22:27:35.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: :35.029176+0000 mon.a ( 2021-03-15T22:27:35.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.0) 1503 : cluster [DBG] 2021-03-15T22:27:35.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22:27:35.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15T22:27 2021-03-15T22:27:35.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: :35.029281+0000 mon.a ( 2021-03-15T22:27:35.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.0) 1504 : cluster 2021-03-15T22:27:35.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: [DBG] osd.5 reported immediately failed by osd.3 2021-03-15T22:27:35.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:27:35.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 22:27:35.029382+0000 mon.a (mon. 2021-03-15T22:27:35.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 0) 1505 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:27:35.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: -15T22:27:35.029476+0000 mon.a 2021-03-15T22:27:35.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: (mon.0) 1506 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:27:35.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: -03-15T22:27:35.029583+0000 2021-03-15T22:27:35.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.a (mon.0) 1507 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:27:35.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: -03-15T22:27: 2021-03-15T22:27:35.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 35.029679+0000 mon.a (mon 2021-03-15T22:27:35.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: .0) 1508 : cluster [DBG] 2021-03-15T22:27:35.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: osd.5 reported immediately failed by osd.4 2021-03-15T22:27:35.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:27:35.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: -15T22:27:35.029814+0000 2021-03-15T22:27:35.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.a (mon.0) 1509 : 2021-03-15T22:27:35.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:27:35.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 03-15T22:27:35 2021-03-15T22:27:35.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: .096586+0000 mon.a (mon. 2021-03-15T22:27:35.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 0) 1510 : cluster [DBG] 2021-03-15T22:27:35.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: osd.5 reported immediately failed by osd.3 2021-03-15T22:27:35.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22:27:35.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15T22:27 2021-03-15T22:27:35.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: :35.096827+0000 mon.a 2021-03-15T22:27:35.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: (mon.0) 1511 : cluster 2021-03-15T22:27:35.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021-03-15T22:27:35.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:27:35.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 27:35.110569+0000 mon.a ( 2021-03-15T22:27:35.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: mon.0) 1512 : cluster 2021-03-15T22:27:35.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:27:35.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: -03-15T22:27: 2021-03-15T22:27:35.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: 35.141403+0000 mon.a (mon 2021-03-15T22:27:35.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:35 smithi076 conmon[56514]: .0) 1513 : cluster [WRN] Health check update: Degraded data redundancy: 213/1722 objects degraded (12.369%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:35.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:35.535 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:35.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:35.536 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:27:35 smithi125 podman[125265]: 14aac52a89ef77eff8ba91dda4cd14f892ca25051125cebdcf99365e8e7285a6 2021-03-15T22:27:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:34.103336+0000 2021-03-15T22:27:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: osd.3 (osd.3) 91 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:27:35.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:34.187509+0000 2021-03-15T22:27:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: mon.a (mon.0) 1487 : cluster [DBG] osdmap e539: 8 total, 7 up, 7 in 2021-03-15T22:27:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:27:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: :27:34.513854+0000 mon.a (mon. 2021-03-15T22:27:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: 0) 1488 : audit [DBG] from='client.? 172.21.15.76:0/183012678' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:35.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:27:35.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: -15T22:27:34.828073+0000 2021-03-15T22:27:35.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: mon.a (mon.0) 1489 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: 2021-03-15T22:27:35.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:27:35.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: 27:34.828108+0000 mon.a (mon.0) 1490 : cluster 2021-03-15T22:27:35.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: [INF] osd.5 failed (root=default,host=smithi125) (connection refused reported by osd.7) 2021-03-15T22:27:35.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:27:35.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: 22:27:34.828307+0000 mon.a ( 2021-03-15T22:27:35.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: mon.0) 1491 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: 2021-03-15T22:27:35.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27 2021-03-15T22:27:35.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: :34.828385+0000 mon.a (mon.0) 1492 : 2021-03-15T22:27:35.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:27:35.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: :27:34.828489+0000 mon.a (mon. 2021-03-15T22:27:35.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: 0) 1493 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: 2021-03-15T22:27:34. 2021-03-15T22:27:35.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: 828611+0000 mon.a (mon.0) 1494 : cluster 2021-03-15T22:27:35.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:27:35.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: -15T22:27:34.828695+0000 2021-03-15T22:27:35.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: mon.a (mon.0) 1495 : cluster [DBG] 2021-03-15T22:27:35.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:27:35.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: T22:27:34.828798+0000 mon.a ( 2021-03-15T22:27:35.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: mon.0) 1496 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27 2021-03-15T22:27:35.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: :34.828919+0000 mon.a (mon.0) 1497 : 2021-03-15T22:27:35.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:27:35.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: T22:27:35.028319+0000 mon.a (mon 2021-03-15T22:27:35.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: .0) 1498 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-15T22:27:35.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:27:35.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: -03-15T22:27:35.028543+0000 2021-03-15T22:27:35.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: mon.a (mon.0) 1499 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: 2021-03-15T22:27:35.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.028732+0000 mon.a (mon.0) 1500 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:27:35.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: T22:27:35.028909+0000 mon.a (mon.0) 1501 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-15T22:27:35.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: 2021-03-15T22:27:35.029042+0000 mon.a 2021-03-15T22:27:35.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: (mon.0) 1502 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:27:35.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: 27:35.029176+0000 mon.a (mon.0) 1503 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.029281+0000 mon.a (mon.0) 1504 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-15T22:27:35.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.029382+0000 mon.a (mon.0) 1505 : cluster [DBG] osd.5 reported immediately failed by osd.7 2021-03-15T22:27:35.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.029476+0000 mon.a (mon.0) 1506 : cluster [DBG] osd.5 reported immediately failed by osd.0 2021-03-15T22:27:35.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.029583+0000 mon.a (mon.0) 1507 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.029679+0000 mon.a (mon.0) 1508 : cluster [DBG] osd.5 reported immediately failed by osd.4 2021-03-15T22:27:35.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.029814+0000 mon.a (mon.0) 1509 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.096586+0000 mon.a (mon.0) 1510 : cluster [DBG] osd.5 reported immediately failed by osd.3 2021-03-15T22:27:35.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.096827+0000 mon.a (mon.0) 1511 : cluster [DBG] osd.5 reported immediately failed by osd.1 2021-03-15T22:27:35.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.110569+0000 mon.a (mon.0) 1512 : cluster [DBG] osd.5 reported immediately failed by osd.6 2021-03-15T22:27:35.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:35 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.141403+0000 mon.a (mon.0) 1513 : cluster [WRN] Health check update: Degraded data redundancy: 213/1722 objects degraded (12.369%), 21 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:35.636 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:35.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:35.661 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:35.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:35.764 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:35.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:35.790 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:35.790 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:27:35.793 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:27:35.892 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:35.916 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:35.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:36.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:36.037 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:27:35 smithi125 podman[125504]: 2021-03-15 22:27:35.705961931 +0000 UTC m=+0.209060956 container create 18e88a61e17bff37ab49329fc5abfa057b8af06a62575f0ef8c5e0bd6677ce0a (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, ceph=True, org.label-schema.build-date=20201204, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) 2021-03-15T22:27:36.037 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:27:35 smithi125 podman[125504]: 2021-03-15 22:27:35.83089474 +0000 UTC m=+0.333993668 container init 18e88a61e17bff37ab49329fc5abfa057b8af06a62575f0ef8c5e0bd6677ce0a (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5-deactivate, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, ceph=True, maintainer=Dimitri Savineau , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.18.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2021-03-15T22:27:36.038 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:27:35 smithi125 podman[125504]: 2021-03-15 22:27:35.880829432 +0000 UTC m=+0.383928354 container start 18e88a61e17bff37ab49329fc5abfa057b8af06a62575f0ef8c5e0bd6677ce0a (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5-deactivate, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, io.buildah.version=1.18.0) 2021-03-15T22:27:36.038 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:27:35 smithi125 podman[125504]: 2021-03-15 22:27:35.880924883 +0000 UTC m=+0.384023819 container attach 18e88a61e17bff37ab49329fc5abfa057b8af06a62575f0ef8c5e0bd6677ce0a (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5-deactivate, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_BRANCH=HEAD, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20201204, org.label-schema.license=GPLv2, io.buildah.version=1.18.0) 2021-03-15T22:27:36.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:36.041 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:36.042 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:36.047 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:27:36.142 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:36.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:36.166 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:36.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:36.202 INFO:tasks.rados.rados.0.smithi076.stdout:2061: done (1 left) 2021-03-15T22:27:36.202 INFO:tasks.rados.rados.0.smithi076.stdout:2062: delete oid 44 current snap is 233 2021-03-15T22:27:36.206 INFO:tasks.rados.rados.0.smithi076.stdout:2062: done (1 left) 2021-03-15T22:27:36.207 INFO:tasks.rados.rados.0.smithi076.stdout:2063: delete oid 43 current snap is 233 2021-03-15T22:27:36.215 INFO:tasks.rados.rados.0.smithi076.stdout:2063: done (1 left) 2021-03-15T22:27:36.216 INFO:tasks.rados.rados.0.smithi076.stdout:2064: delete oid 7 current snap is 233 2021-03-15T22:27:36.268 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:36.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:36.293 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:36.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:36.348 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:36.390 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:27:36.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:36.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: cluster 2021-03-15T22:27:36.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: 2021-03-15T22:27:36.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: 2021-03-15T22:27: 2021-03-15T22:27:36.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: 35.087413+0000 mgr.y (mgr.14144) 716 : cluster [DBG] pgmap v936: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 556 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 16 op/s; 214/1728 objects degraded (12.384%) 2021-03-15T22:27:36.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:27:36.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: 27:35.184356+0000 mon.a (mon.0) 1514 : 2021-03-15T22:27:36.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2021-03-15T22:27:36.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:27:36.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: :27:35.199722+0000 mon.a 2021-03-15T22:27:36.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: (mon.0) 1515 : cluster 2021-03-15T22:27:36.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: [DBG] osdmap e540: 8 total, 6 up, 7 in 2021-03-15T22:27:36.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: cluster 2021-03-15T22:27:36.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:27:36.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: 27:35.395953+0000 osd.6 2021-03-15T22:27:36.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: (osd.6) 82 : cluster 2021-03-15T22:27:36.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: [DBG] 3.a scrub ok 2021-03-15T22:27:36.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: cluster 2021-03-15T22:27:36.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:27:36.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: 27:35.403027+0000 osd.6 2021-03-15T22:27:36.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: (osd.6) 83 : 2021-03-15T22:27:36.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:36.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: cluster 2021-03-15T22:27 2021-03-15T22:27:36.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: :35.403877+0000 osd.6 (osd.6) 84 : 2021-03-15T22:27:36.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[56514]: cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:36.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: cluster 2021-03-15T22:27:36.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021-03-15T22:27:36.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021 2021-03-15T22:27:36.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: - 2021-03-15T22:27:36.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 03 2021-03-15T22:27:36.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: - 2021-03-15T22:27:36.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 15 2021-03-15T22:27:36.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: T 2021-03-15T22:27:36.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 22 2021-03-15T22:27:36.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: : 2021-03-15T22:27:36.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 27 2021-03-15T22:27:36.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: : 2021-03-15T22:27:36.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 35 2021-03-15T22:27:36.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: . 2021-03-15T22:27:36.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 087413 2021-03-15T22:27:36.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: +0000 2021-03-15T22:27:36.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021-03-15T22:27:36.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: mgr.y 2021-03-15T22:27:36.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: ( 2021-03-15T22:27:36.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: mgr 2021-03-15T22:27:36.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: . 2021-03-15T22:27:36.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 14144 2021-03-15T22:27:36.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: ) 2021-03-15T22:27:36.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 716 2021-03-15T22:27:36.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: : 2021-03-15T22:27:36.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: cluster 2021-03-15T22:27:36.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021-03-15T22:27:36.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: [DBG] 2021-03-15T22:27:36.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021-03-15T22:27:36.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: pgmap v936: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 556 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 16 op/s; 214/1728 objects degraded (12.384%) 2021-03-15T22:27:36.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021-03-15T22:27:36.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: cluster 2021-03-15T22:27:36.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021-03-15T22:27:36.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021 2021-03-15T22:27:36.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: - 2021-03-15T22:27:36.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 03 2021-03-15T22:27:36.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: - 2021-03-15T22:27:36.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 15 2021-03-15T22:27:36.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: T 2021-03-15T22:27:36.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 22 2021-03-15T22:27:36.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: : 2021-03-15T22:27:36.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 27 2021-03-15T22:27:36.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: : 2021-03-15T22:27:36.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 35 2021-03-15T22:27:36.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: . 2021-03-15T22:27:36.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 184356 2021-03-15T22:27:36.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: +0000 2021-03-15T22:27:36.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021-03-15T22:27:36.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: mon.a 2021-03-15T22:27:36.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: ( 2021-03-15T22:27:36.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: mon 2021-03-15T22:27:36.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: . 2021-03-15T22:27:36.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 0 2021-03-15T22:27:36.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: ) 2021-03-15T22:27:36.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 1514 2021-03-15T22:27:36.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: : 2021-03-15T22:27:36.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: cluster 2021-03-15T22:27:36.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021-03-15T22:27:36.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: [WRN] 2021-03-15T22:27:36.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021-03-15T22:27:36.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: Health check update: 2 osds down (OSD_DOWN) 2021-03-15T22:27:36.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021-03-15T22:27:36.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: cluster 2021-03-15T22:27:35.199722+0000 mon.a 2021-03-15T22:27:36.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: (mon.0) 1515 : cluster [DBG] osdmap e540: 8 total, 6 up, 7 in 2021-03-15T22:27:36.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:27:36.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: -03-15T22: 2021-03-15T22:27:36.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 27:35.395953+0000 osd.6 (osd.6 2021-03-15T22:27:36.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: ) 82 : cluster [DBG] 3.a scrub ok 2021-03-15T22:27:36.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 2021-03-15T22:27:36.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:27:36.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 03-15T22:27:35.403027+0000 osd.6 (osd.6) 83 : cluster 2021-03-15T22:27:36.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: [DBG] purged_snaps scrub starts 2021-03-15T22:27:36.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: cluster 2021-03-15T22:27: 2021-03-15T22:27:36.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: 35.403877+0000 osd.6 (osd.6) 84 : 2021-03-15T22:27:36.435 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:36 smithi076 conmon[62156]: cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:36.439 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:36.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:36.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:36.536 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:27:36 smithi125 podman[125504]: 2021-03-15 22:27:36.167213499 +0000 UTC m=+0.670312447 container died 18e88a61e17bff37ab49329fc5abfa057b8af06a62575f0ef8c5e0bd6677ce0a (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5-deactivate) 2021-03-15T22:27:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:36 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.087413+0000 mgr.y (mgr.14144) 716 : cluster [DBG] pgmap v936: 138 pgs: 21 active+undersized+degraded, 33 active+undersized, 84 active+clean; 556 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 16 op/s; 214/1728 objects degraded (12.384%) 2021-03-15T22:27:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:36 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.184356+0000 mon.a (mon.0) 1514 : cluster [WRN] Health check update: 2 osds down (OSD_DOWN) 2021-03-15T22:27:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:36 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.199722+0000 mon.a (mon.0) 1515 : cluster [DBG] osdmap e540: 8 total, 6 up, 7 in 2021-03-15T22:27:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:36 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.395953+0000 osd.6 (osd.6) 82 : cluster [DBG] 3.a scrub ok 2021-03-15T22:27:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:36 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.403027+0000 osd.6 (osd.6) 83 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:36 smithi125 conmon[56739]: cluster 2021-03-15T22:27:35.403877+0000 osd.6 (osd.6) 84 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:36.542 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:36.564 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:36.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:36.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:36.667 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:36.692 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:36.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:36.793 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:36.816 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:36.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:36.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:36.841 DEBUG:teuthology.orchestra.run.smithi125:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.5.service' 2021-03-15T22:27:36.861 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:36.905 DEBUG:teuthology.orchestra.run:got remote process result: None 2021-03-15T22:27:36.906 INFO:tasks.cephadm.osd.5:Stopped osd.5 2021-03-15T22:27:36.906 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.5 2021-03-15T22:27:36.907 INFO:tasks.thrashosds.thrasher:remote for osd 5 is ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:27:36.907 INFO:tasks.thrashosds.thrasher:remote for exp osd 2 is ubuntu@smithi076.front.sepia.ceph.com 2021-03-15T22:27:36.907 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -n osd.2 --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op list-pgs 2021-03-15T22:27:36.920 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:36.972 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:36.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:36.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:37.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:37.098 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:37.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:37.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:37.128 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/osd.2/config 2021-03-15T22:27:37.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:37.211 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:37 smithi076 conmon[62156]: cluster 2021-03-15T22:27:36.196594+0000 mon.a (mon.0) 1516 : cluster [DBG] osdmap e541: 8 total, 6 up, 7 in 2021-03-15T22:27:37.211 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:37 smithi076 conmon[62156]: cluster 2021-03-15T22:27:36.335523+0000 osd.7 (osd.7) 105 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:27:37.212 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:37 smithi076 conmon[62156]: cluster 2021-03-15T22:27:37.202882+0000 mon.a (mon.0) 1517 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:27:37.227 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:37.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:37.330 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:37.353 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:37.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:37.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:37.456 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:37.478 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:37.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:37.480 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:37.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:37 smithi076 conmon[56514]: cluster 2021-03-15T22:27:36.196594+0000 mon.a (mon.0) 1516 : cluster [DBG] osdmap e541: 8 total, 6 up, 7 in 2021-03-15T22:27:37.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:37 smithi076 conmon[56514]: cluster 2021-03-15T22:27:36.335523+0000 osd.7 (osd.7) 105 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:27:37.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:37 smithi076 conmon[56514]: cluster 2021-03-15T22:27:37.202882+0000 mon.a (mon.0) 1517 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:27:37.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:37 smithi125 conmon[56739]: cluster 2021-03-15T22:27:36.196594+0000 mon.a (mon.0) 1516 : cluster [DBG] osdmap e541: 8 total, 6 up, 7 in 2021-03-15T22:27:37.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:37 smithi125 conmon[56739]: cluster 2021-03-15T22:27: 2021-03-15T22:27:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:37 smithi125 conmon[56739]: 36.335523+0000 osd.7 (osd.7) 105 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:27:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:37 smithi125 conmon[56739]: cluster 2021-03-15T22:27:37.202882+0000 mon.a (mon.0) 1517 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:27:37.582 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:37.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:37.607 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:37.607 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:37.708 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:37.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:37.735 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:37.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:37.795 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:27:37.837 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:37.860 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:37.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:37.861 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:37.962 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:37.988 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:37.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:37.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:38.091 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:38.114 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:38.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:38.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:38.216 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:38.238 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:38.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:38.239 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:38.340 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:38.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:38.369 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:38.369 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:38.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:38.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[56514]: cluster 2021-03-15T22:27:36.883118+0000 osd.4 (osd.4) 112 2021-03-15T22:27:38.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[56514]: : cluster [DBG] 3.3 scrub ok 2021-03-15T22:27:38.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[56514]: cluster 2021-03-15T22:27:37.088010+0000 mgr.y (mgr.14144) 717 : cluster [DBG] pgmap v939: 138 pgs: 6 peering, 3 stale+active+undersized, 10 stale+active+clean, 4 undersized+degraded+peered, 20 active+undersized+degraded, 4 undersized+peered, 37 active+undersized, 54 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 3.3 MiB/s wr, 23 op/s; 286/1740 objects degraded (16.437%) 2021-03-15T22:27:38.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[56514]: cluster 2021-03-15T22:27:37.107460+0000 osd.3 (osd.3) 92 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:27:38.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[56514]: cluster 2021-03-15T22:27:37.430045+0000 osd.1 (osd.1) 104 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:27:38.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[62156]: cluster 2021-03-15T22:27:36.883118+0000 osd.4 ( 2021-03-15T22:27:38.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[62156]: osd.4) 112 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:27:38.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[62156]: cluster 2021-03-15T22:27:37.088010+0000 mgr.y (mgr. 2021-03-15T22:27:38.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[62156]: 14144) 717 : cluster [DBG] pgmap v939: 138 pgs: 6 peering, 3 stale+active+undersized, 10 stale+active+clean, 4 undersized+degraded+peered, 20 active+undersized+degraded, 4 undersized+peered, 37 active+undersized, 54 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 3.3 MiB/s wr, 23 op/s; 286/1740 objects degraded (16.437%) 2021-03-15T22:27:38.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[62156]: cluster 2021-03-15T22:27:37.107460+0000 2021-03-15T22:27:38.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[62156]: osd.3 (osd.3) 92 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:27:38.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:27:38.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:38 smithi076 conmon[62156]: 27:37.430045+0000 osd.1 (osd.1) 104 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:27:38.495 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:38.497 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:38.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:38 smithi125 conmon[56739]: cluster 2021-03-15T22:27:36.883118+0000 osd.4 (osd.4) 112 : cluster 2021-03-15T22:27:38.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:38 smithi125 conmon[56739]: [DBG] 3.3 scrub ok 2021-03-15T22:27:38.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:38 smithi125 conmon[56739]: cluster 2021-03-15T22:27:37.088010+0000 mgr.y (mgr.14144) 717 : cluster [DBG] pgmap v939: 138 pgs: 6 peering, 3 stale+active+undersized, 10 stale+active+clean, 4 undersized+degraded+peered, 20 active+undersized+degraded, 4 undersized+peered, 37 active+undersized, 54 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 19 MiB/s rd, 3.3 MiB/s wr, 23 op/s; 286/1740 objects degraded (16.437%) 2021-03-15T22:27:38.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:38 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:27:38.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:38 smithi125 conmon[56739]: -03-15T22:27:37.107460+0000 osd.3 (osd.3) 92 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:27:38.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:38 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:27:38.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:38 smithi125 conmon[56739]: T22:27:37.430045+0000 osd.1 (osd.1) 104 : 2021-03-15T22:27:38.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:38 smithi125 conmon[56739]: cluster [DBG] 3.19 scrub ok 2021-03-15T22:27:38.598 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:38.623 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:38.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:38.625 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:38.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:38.752 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:38.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:38.753 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:38.767 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:27:38.767 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:27:38.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:27:38.768 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:27:38.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.768 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1191 7.6 7:61e0cc05:::smithi07694878-16:head [copy-from ver 101] snapc 4e=[4e,4d,3f] ondisk+write+known_if_redirected e175)", 2021-03-15T22:27:38.769 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:27:38.769 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 368.54515345300001, 2021-03-15T22:27:38.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.97834980800000004, 2021-03-15T22:27:38.769 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.769 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:38.770 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:38.770 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:38.770 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:38.770 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1191 2021-03-15T22:27:38.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.771 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:27:38.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:27:38.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0440000000000003e-06 2021-03-15T22:27:38.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067745+0000", 2021-03-15T22:27:38.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4340000000000001e-06 2021-03-15T22:27:38.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067749+0000", 2021-03-15T22:27:38.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9241000000000001e-05 2021-03-15T22:27:38.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067768+0000", 2021-03-15T22:27:38.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3010000000000007e-06 2021-03-15T22:27:38.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067776+0000", 2021-03-15T22:27:38.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5014e-05 2021-03-15T22:27:38.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067821+0000", 2021-03-15T22:27:38.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9779999999999994e-06 2021-03-15T22:27:38.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:38.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067830+0000", 2021-03-15T22:27:38.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.95797833899999996 2021-03-15T22:27:38.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.025809+0000", 2021-03-15T22:27:38.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138412 2021-03-15T22:27:38.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.025947+0000", 2021-03-15T22:27:38.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006355744 2021-03-15T22:27:38.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,7", 2021-03-15T22:27:38.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.032303+0000", 2021-03-15T22:27:38.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043109439999999997 2021-03-15T22:27:38.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:38.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.036614+0000", 2021-03-15T22:27:38.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092878839999999997 2021-03-15T22:27:38.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:38.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.045902+0000", 2021-03-15T22:27:38.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8556999999999999e-05 2021-03-15T22:27:38.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:38.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.045990+0000", 2021-03-15T22:27:38.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6665999999999997e-05 2021-03-15T22:27:38.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:38.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046037+0000", 2021-03-15T22:27:38.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.325e-05 2021-03-15T22:27:38.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:38.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046090+0000", 2021-03-15T22:27:38.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.97834980800000004 2021-03-15T22:27:38.793 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.793 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:38.794 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.794 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1192 7.6 7:61e0cc05:::smithi07694878-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e175)", 2021-03-15T22:27:38.795 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:27:38.795 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 368.54510413399998, 2021-03-15T22:27:38.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.978448604, 2021-03-15T22:27:38.795 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.796 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:38.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:38.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:38.796 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:38.796 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1192 2021-03-15T22:27:38.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.797 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:27:38.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:27:38.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.401e-06 2021-03-15T22:27:38.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067791+0000", 2021-03-15T22:27:38.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.083e-06 2021-03-15T22:27:38.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067794+0000", 2021-03-15T22:27:38.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4919999999999996e-06 2021-03-15T22:27:38.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067802+0000", 2021-03-15T22:27:38.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.383e-06 2021-03-15T22:27:38.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067807+0000", 2021-03-15T22:27:38.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8674000000000002e-05 2021-03-15T22:27:38.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067856+0000", 2021-03-15T22:27:38.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5890000000000003e-06 2021-03-15T22:27:38.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:38.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067864+0000", 2021-03-15T22:27:38.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.95935619400000005 2021-03-15T22:27:38.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.027221+0000", 2021-03-15T22:27:38.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9426999999999998e-05 2021-03-15T22:27:38.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:38.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.027290+0000", 2021-03-15T22:27:38.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085250369999999992 2021-03-15T22:27:38.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.035815+0000", 2021-03-15T22:27:38.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9997999999999997e-05 2021-03-15T22:27:38.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:38.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.035875+0000", 2021-03-15T22:27:38.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010246089 2021-03-15T22:27:38.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046121+0000", 2021-03-15T22:27:38.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3338999999999998e-05 2021-03-15T22:27:38.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046184+0000", 2021-03-15T22:27:38.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3897999999999998e-05 2021-03-15T22:27:38.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:38.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046238+0000", 2021-03-15T22:27:38.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.978448604 2021-03-15T22:27:38.813 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.814 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:38.814 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.815 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1556 7.6 7:6a64397b:::smithi07694878-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e204)", 2021-03-15T22:27:38.815 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.030294+0000", 2021-03-15T22:27:38.815 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 339.58259966399999, 2021-03-15T22:27:38.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024729370000000001, 2021-03-15T22:27:38.815 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.816 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:38.816 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:38.816 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:38.816 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:38.816 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1556 2021-03-15T22:27:38.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.817 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030294+0000", 2021-03-15T22:27:38.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030294+0000", 2021-03-15T22:27:38.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.128e-06 2021-03-15T22:27:38.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030295+0000", 2021-03-15T22:27:38.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3790000000000001e-06 2021-03-15T22:27:38.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030297+0000", 2021-03-15T22:27:38.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4710000000000004e-06 2021-03-15T22:27:38.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030303+0000", 2021-03-15T22:27:38.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0389999999999998e-06 2021-03-15T22:27:38.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.030307+0000", 2021-03-15T22:27:38.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00070512700000000005 2021-03-15T22:27:38.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.031012+0000", 2021-03-15T22:27:38.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9470000000000001e-05 2021-03-15T22:27:38.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:38.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.031042+0000", 2021-03-15T22:27:38.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015725996999999999 2021-03-15T22:27:38.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046768+0000", 2021-03-15T22:27:38.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010418500000000001 2021-03-15T22:27:38.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:38.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.046872+0000", 2021-03-15T22:27:38.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0079290709999999993 2021-03-15T22:27:38.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.054801+0000", 2021-03-15T22:27:38.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5829999999999996e-05 2021-03-15T22:27:38.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.054887+0000", 2021-03-15T22:27:38.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000136673 2021-03-15T22:27:38.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:38.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.055024+0000", 2021-03-15T22:27:38.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024729370000000001 2021-03-15T22:27:38.831 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.831 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:38.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.832 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1563 7.f 7:fee9f235:::smithi07694878-11:head [copy-from ver 242] snapc 6a=[6a,69,68] ondisk+write+known_if_redirected e204)", 2021-03-15T22:27:38.832 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.059465+0000", 2021-03-15T22:27:38.833 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 339.55342861999998, 2021-03-15T22:27:38.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026787723999999999, 2021-03-15T22:27:38.833 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.833 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:38.833 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:38.834 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:38.834 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:38.834 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1563 2021-03-15T22:27:38.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.835 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059465+0000", 2021-03-15T22:27:38.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059465+0000", 2021-03-15T22:27:38.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.382e-06 2021-03-15T22:27:38.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059469+0000", 2021-03-15T22:27:38.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.294e-06 2021-03-15T22:27:38.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059471+0000", 2021-03-15T22:27:38.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4338999999999999e-05 2021-03-15T22:27:38.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059485+0000", 2021-03-15T22:27:38.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3259999999999998e-06 2021-03-15T22:27:38.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059493+0000", 2021-03-15T22:27:38.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9969999999999995e-05 2021-03-15T22:27:38.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059573+0000", 2021-03-15T22:27:38.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017286299999999999 2021-03-15T22:27:38.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059745+0000", 2021-03-15T22:27:38.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089917209999999994 2021-03-15T22:27:38.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:27:38.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.068737+0000", 2021-03-15T22:27:38.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0046724000000000002 2021-03-15T22:27:38.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:38.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.073410+0000", 2021-03-15T22:27:38.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099697450000000003 2021-03-15T22:27:38.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:38.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.083379+0000", 2021-03-15T22:27:38.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027667730000000001 2021-03-15T22:27:38.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:38.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086146+0000", 2021-03-15T22:27:38.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6458000000000001e-05 2021-03-15T22:27:38.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:38.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086192+0000", 2021-03-15T22:27:38.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0452999999999999e-05 2021-03-15T22:27:38.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:38.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086253+0000", 2021-03-15T22:27:38.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026787723999999999 2021-03-15T22:27:38.850 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.850 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:38.850 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.851 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1564 7.f 7:fee9f235:::smithi07694878-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e204)", 2021-03-15T22:27:38.851 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.059507+0000", 2021-03-15T22:27:38.851 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 339.55338730900002, 2021-03-15T22:27:38.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026908793, 2021-03-15T22:27:38.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.852 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:38.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:38.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:38.852 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:38.853 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1564 2021-03-15T22:27:38.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.853 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059507+0000", 2021-03-15T22:27:38.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059507+0000", 2021-03-15T22:27:38.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.274e-06 2021-03-15T22:27:38.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059508+0000", 2021-03-15T22:27:38.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6589999999999999e-06 2021-03-15T22:27:38.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059509+0000", 2021-03-15T22:27:38.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0969999999999995e-06 2021-03-15T22:27:38.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059519+0000", 2021-03-15T22:27:38.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5739999999999997e-06 2021-03-15T22:27:38.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059523+0000", 2021-03-15T22:27:38.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00046872599999999998 2021-03-15T22:27:38.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059992+0000", 2021-03-15T22:27:38.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7046999999999999e-05 2021-03-15T22:27:38.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:38.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.060039+0000", 2021-03-15T22:27:38.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012485036 2021-03-15T22:27:38.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.072524+0000", 2021-03-15T22:27:38.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6324999999999999e-05 2021-03-15T22:27:38.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:38.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.072590+0000", 2021-03-15T22:27:38.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013691045000000001 2021-03-15T22:27:38.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086281+0000", 2021-03-15T22:27:38.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6152e-05 2021-03-15T22:27:38.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086347+0000", 2021-03-15T22:27:38.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7858e-05 2021-03-15T22:27:38.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:38.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086415+0000", 2021-03-15T22:27:38.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026908793 2021-03-15T22:27:38.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.871 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:38.872 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.873 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:548 7.6 e204/175 ondisk, result = 0)", 2021-03-15T22:27:38.873 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:27:38.873 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 338.531371224, 2021-03-15T22:27:38.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045793304999999999, 2021-03-15T22:27:38.874 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.874 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:38.874 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:27:38.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:27:38.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6390000000000001e-06 2021-03-15T22:27:38.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081524+0000", 2021-03-15T22:27:38.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.863e-06 2021-03-15T22:27:38.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081526+0000", 2021-03-15T22:27:38.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1234e-05 2021-03-15T22:27:38.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081537+0000", 2021-03-15T22:27:38.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5210000000000002e-06 2021-03-15T22:27:38.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081543+0000", 2021-03-15T22:27:38.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045750591 2021-03-15T22:27:38.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127293+0000", 2021-03-15T22:27:38.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6920000000000008e-06 2021-03-15T22:27:38.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127302+0000", 2021-03-15T22:27:38.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3764999999999999e-05 2021-03-15T22:27:38.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:38.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127316+0000", 2021-03-15T22:27:38.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045793304999999999 2021-03-15T22:27:38.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.885 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:38.885 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.886 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:548 7.6 e205/175 ondisk, result = 0)", 2021-03-15T22:27:38.886 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:27:38.887 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 338.52704774199998, 2021-03-15T22:27:38.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041608198999999998, 2021-03-15T22:27:38.887 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.887 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:38.888 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:27:38.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:27:38.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4779999999999998e-06 2021-03-15T22:27:38.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085849+0000", 2021-03-15T22:27:38.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.046e-06 2021-03-15T22:27:38.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085851+0000", 2021-03-15T22:27:38.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4150000000000005e-06 2021-03-15T22:27:38.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085860+0000", 2021-03-15T22:27:38.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0900000000000001e-06 2021-03-15T22:27:38.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085866+0000", 2021-03-15T22:27:38.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041511365000000001 2021-03-15T22:27:38.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127378+0000", 2021-03-15T22:27:38.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6030000000000001e-06 2021-03-15T22:27:38.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127384+0000", 2021-03-15T22:27:38.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0202000000000003e-05 2021-03-15T22:27:38.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:38.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127454+0000", 2021-03-15T22:27:38.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041608198999999998 2021-03-15T22:27:38.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.903 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:38.903 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.905 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:61 7.5 e205/91 7:a9545d62:::smithi07694878-1:68 v 205'72, mlcod=202'69)", 2021-03-15T22:27:38.905 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:27:38.906 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 338.52202399599997, 2021-03-15T22:27:38.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034962251999999999, 2021-03-15T22:27:38.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.907 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:38.907 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:27:38.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:27:38.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.798e-06 2021-03-15T22:27:38.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090877+0000", 2021-03-15T22:27:38.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3779999999999999e-06 2021-03-15T22:27:38.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090879+0000", 2021-03-15T22:27:38.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1902999999999999e-05 2021-03-15T22:27:38.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090891+0000", 2021-03-15T22:27:38.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7780000000000004e-06 2021-03-15T22:27:38.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090897+0000", 2021-03-15T22:27:38.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0942000000000002e-05 2021-03-15T22:27:38.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090938+0000", 2021-03-15T22:27:38.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6379e-05 2021-03-15T22:27:38.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090954+0000", 2021-03-15T22:27:38.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034836759000000002 2021-03-15T22:27:38.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:38.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125791+0000", 2021-03-15T22:27:38.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1315e-05 2021-03-15T22:27:38.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:38.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125832+0000", 2021-03-15T22:27:38.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034962251999999999 2021-03-15T22:27:38.921 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.921 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:38.921 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.922 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.6.0:214 7.d e205/91 7:bd183e1b:::smithi07694878-46:68 v 205'30, mlcod=202'27)", 2021-03-15T22:27:38.922 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:27:38.923 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 338.52158211699998, 2021-03-15T22:27:38.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034834256000000001, 2021-03-15T22:27:38.923 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.923 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:38.924 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:27:38.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:27:38.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3590000000000002e-06 2021-03-15T22:27:38.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091314+0000", 2021-03-15T22:27:38.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.57e-06 2021-03-15T22:27:38.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091316+0000", 2021-03-15T22:27:38.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3749999999999999e-06 2021-03-15T22:27:38.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091322+0000", 2021-03-15T22:27:38.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2140000000000004e-06 2021-03-15T22:27:38.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091327+0000", 2021-03-15T22:27:38.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017519700000000001 2021-03-15T22:27:38.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091502+0000", 2021-03-15T22:27:38.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2955e-05 2021-03-15T22:27:38.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091525+0000", 2021-03-15T22:27:38.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034582099999999998 2021-03-15T22:27:38.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:38.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.126108+0000", 2021-03-15T22:27:38.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8485999999999997e-05 2021-03-15T22:27:38.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:38.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.126146+0000", 2021-03-15T22:27:38.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034834256000000001 2021-03-15T22:27:38.936 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.936 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:38.937 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.937 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1686 7.f 7:fee9f235:::smithi07694878-11:head [copy-from ver 242] snapc 7a=[7a,79,78,76,75] ondisk+write+known_if_redirected e220)", 2021-03-15T22:27:38.938 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:15.039251+0000", 2021-03-15T22:27:38.938 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 323.57364330799999, 2021-03-15T22:27:38.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026727377, 2021-03-15T22:27:38.938 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.939 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:38.939 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:38.939 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:38.939 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:38.940 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1686 2021-03-15T22:27:38.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.940 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039251+0000", 2021-03-15T22:27:38.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039251+0000", 2021-03-15T22:27:38.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0810000000000001e-06 2021-03-15T22:27:38.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039256+0000", 2021-03-15T22:27:38.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6880000000000001e-06 2021-03-15T22:27:38.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039259+0000", 2021-03-15T22:27:38.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7503000000000002e-05 2021-03-15T22:27:38.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039277+0000", 2021-03-15T22:27:38.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3869999999999997e-06 2021-03-15T22:27:38.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039284+0000", 2021-03-15T22:27:38.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8112999999999998e-05 2021-03-15T22:27:38.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039332+0000", 2021-03-15T22:27:38.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7928000000000002e-05 2021-03-15T22:27:38.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039430+0000", 2021-03-15T22:27:38.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067539030000000003 2021-03-15T22:27:38.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:27:38.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.046184+0000", 2021-03-15T22:27:38.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056836600000000001 2021-03-15T22:27:38.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:38.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.051868+0000", 2021-03-15T22:27:38.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008344677 2021-03-15T22:27:38.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:38.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.060212+0000", 2021-03-15T22:27:38.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056656900000000001 2021-03-15T22:27:38.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:38.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.065878+0000", 2021-03-15T22:27:38.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9669999999999997e-05 2021-03-15T22:27:38.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:38.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.065928+0000", 2021-03-15T22:27:38.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0077000000000002e-05 2021-03-15T22:27:38.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:38.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.065978+0000", 2021-03-15T22:27:38.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026727377 2021-03-15T22:27:38.957 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.957 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:38.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.958 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1687 7.f 7:fee9f235:::smithi07694878-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e220)", 2021-03-15T22:27:38.959 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:15.039297+0000", 2021-03-15T22:27:38.959 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 323.57359733099997, 2021-03-15T22:27:38.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026894929000000001, 2021-03-15T22:27:38.959 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.959 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:38.960 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:38.960 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:38.960 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:38.961 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1687 2021-03-15T22:27:38.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.961 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039297+0000", 2021-03-15T22:27:38.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039297+0000", 2021-03-15T22:27:38.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5549999999999999e-06 2021-03-15T22:27:38.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039298+0000", 2021-03-15T22:27:38.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2021-03-15T22:27:38.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039300+0000", 2021-03-15T22:27:38.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8300000000000001e-06 2021-03-15T22:27:38.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039306+0000", 2021-03-15T22:27:38.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0069999999999999e-06 2021-03-15T22:27:38.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039310+0000", 2021-03-15T22:27:38.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025812099999999999 2021-03-15T22:27:38.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039568+0000", 2021-03-15T22:27:38.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8504999999999997e-05 2021-03-15T22:27:38.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:38.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039616+0000", 2021-03-15T22:27:38.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010537019999999999 2021-03-15T22:27:38.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.050153+0000", 2021-03-15T22:27:38.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011407 2021-03-15T22:27:38.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:38.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.050267+0000", 2021-03-15T22:27:38.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015755377000000001 2021-03-15T22:27:38.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066023+0000", 2021-03-15T22:27:38.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2999999999999998e-05 2021-03-15T22:27:38.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066106+0000", 2021-03-15T22:27:38.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5745e-05 2021-03-15T22:27:38.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:38.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066191+0000", 2021-03-15T22:27:38.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026894929000000001 2021-03-15T22:27:38.979 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.979 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:38.979 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:38.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.980 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2103 7.f 7:fee9f235:::smithi07694878-11:head [copy-from ver 398] snapc 9a=[9a,97,93,92,8d,88,80,7d] ondisk+write+known_if_redirected e253)", 2021-03-15T22:27:38.980 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:27:38.980 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 290.40129913499999, 2021-03-15T22:27:38.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029938436999999998, 2021-03-15T22:27:38.981 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:38.981 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:38.981 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:38.982 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:38.982 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:38.982 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2103 2021-03-15T22:27:38.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.982 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:38.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:38.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:27:38.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:38.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:38.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:27:38.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1350000000000002e-06 2021-03-15T22:27:38.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:38.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211599+0000", 2021-03-15T22:27:38.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1489999999999998e-06 2021-03-15T22:27:38.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:38.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211602+0000", 2021-03-15T22:27:38.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8119000000000001e-05 2021-03-15T22:27:38.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:38.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211620+0000", 2021-03-15T22:27:38.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9529999999999999e-06 2021-03-15T22:27:38.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:38.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211628+0000", 2021-03-15T22:27:38.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015272800000000001 2021-03-15T22:27:38.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211781+0000", 2021-03-15T22:27:38.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0270000000000001e-06 2021-03-15T22:27:38.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:27:38.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211790+0000", 2021-03-15T22:27:38.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035614209999999999 2021-03-15T22:27:38.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:38.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215351+0000", 2021-03-15T22:27:38.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8361999999999994e-05 2021-03-15T22:27:38.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:38.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215420+0000", 2021-03-15T22:27:38.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013426624999999999 2021-03-15T22:27:38.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:27:38.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.228846+0000", 2021-03-15T22:27:38.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041102990000000004 2021-03-15T22:27:38.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:38.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232957+0000", 2021-03-15T22:27:38.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013870029999999999 2021-03-15T22:27:38.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:38.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.234344+0000", 2021-03-15T22:27:38.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007045791 2021-03-15T22:27:38.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:38.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241389+0000", 2021-03-15T22:27:38.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0001999999999998e-05 2021-03-15T22:27:38.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:38.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241459+0000", 2021-03-15T22:27:38.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3823000000000001e-05 2021-03-15T22:27:38.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:38.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:38.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:39.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241533+0000", 2021-03-15T22:27:39.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029938436999999998 2021-03-15T22:27:39.000 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.000 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:39.001 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.001 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2104 7.f 7:fee9f235:::smithi07694878-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e253)", 2021-03-15T22:27:39.001 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:27:39.002 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 290.401157399, 2021-03-15T22:27:39.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029977025000000001, 2021-03-15T22:27:39.002 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:39.002 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:39.002 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:39.003 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:39.003 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:39.003 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2104 2021-03-15T22:27:39.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.004 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:39.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:39.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:27:39.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:39.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:39.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:27:39.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.491e-06 2021-03-15T22:27:39.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:39.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211738+0000", 2021-03-15T22:27:39.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.813e-06 2021-03-15T22:27:39.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:39.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211740+0000", 2021-03-15T22:27:39.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2709999999999992e-06 2021-03-15T22:27:39.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:39.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211748+0000", 2021-03-15T22:27:39.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8559999999999998e-06 2021-03-15T22:27:39.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:39.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211753+0000", 2021-03-15T22:27:39.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014630000000000001 2021-03-15T22:27:39.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211899+0000", 2021-03-15T22:27:39.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7549999999999997e-06 2021-03-15T22:27:39.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:27:39.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211906+0000", 2021-03-15T22:27:39.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036139900000000001 2021-03-15T22:27:39.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215520+0000", 2021-03-15T22:27:39.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7696999999999999e-05 2021-03-15T22:27:39.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:39.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215548+0000", 2021-03-15T22:27:39.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016795027000000001 2021-03-15T22:27:39.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232343+0000", 2021-03-15T22:27:39.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1192000000000005e-05 2021-03-15T22:27:39.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.016 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:39.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232414+0000", 2021-03-15T22:27:39.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091605279999999994 2021-03-15T22:27:39.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241574+0000", 2021-03-15T22:27:39.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5976000000000007e-05 2021-03-15T22:27:39.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:39.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241640+0000", 2021-03-15T22:27:39.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3128999999999997e-05 2021-03-15T22:27:39.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:39.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241713+0000", 2021-03-15T22:27:39.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029977025000000001 2021-03-15T22:27:39.020 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.020 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:39.021 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.021 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2221 7.f 7:ffef77ba:::smithi07694878-13:head [sparse-read 0~2702901,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:27:39.021 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:27:39.022 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 280.56190578899998, 2021-03-15T22:27:39.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0065843969999999, 2021-03-15T22:27:39.022 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:39.022 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:39.023 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:39.023 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:39.023 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:39.023 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2221 2021-03-15T22:27:39.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.024 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:39.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:39.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:27:39.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:39.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:39.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:27:39.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0599999999999999e-06 2021-03-15T22:27:39.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:39.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050991+0000", 2021-03-15T22:27:39.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0379999999999998e-06 2021-03-15T22:27:39.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:39.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050993+0000", 2021-03-15T22:27:39.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6500000000000002e-06 2021-03-15T22:27:39.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:39.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051002+0000", 2021-03-15T22:27:39.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3650000000000003e-06 2021-03-15T22:27:39.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:39.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051007+0000", 2021-03-15T22:27:39.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015642 2021-03-15T22:27:39.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051164+0000", 2021-03-15T22:27:39.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7530000000000007e-06 2021-03-15T22:27:39.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:39.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051172+0000", 2021-03-15T22:27:39.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.004477362 2021-03-15T22:27:39.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.055650+0000", 2021-03-15T22:27:39.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134535 2021-03-15T22:27:39.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:39.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.055784+0000", 2021-03-15T22:27:39.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001788214 2021-03-15T22:27:39.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:39.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.057572+0000", 2021-03-15T22:27:39.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0065843969999999 2021-03-15T22:27:39.036 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.036 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:39.036 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.037 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2222 7.f 7:ffef77ba:::smithi07694878-13:head [read 0~2823609,checksum] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:27:39.037 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:27:39.037 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 280.56188116599998, 2021-03-15T22:27:39.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0081352180000001, 2021-03-15T22:27:39.038 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:39.038 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:39.038 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:39.038 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:39.038 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:39.039 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2222 2021-03-15T22:27:39.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.039 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:39.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:39.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:27:39.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:39.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:39.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:27:39.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6190000000000001e-06 2021-03-15T22:27:39.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:39.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051014+0000", 2021-03-15T22:27:39.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.217e-06 2021-03-15T22:27:39.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:39.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051016+0000", 2021-03-15T22:27:39.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8270000000000003e-06 2021-03-15T22:27:39.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:39.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051021+0000", 2021-03-15T22:27:39.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4529999999999998e-06 2021-03-15T22:27:39.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:39.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051025+0000", 2021-03-15T22:27:39.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000168445 2021-03-15T22:27:39.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051193+0000", 2021-03-15T22:27:39.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5860000000000001e-06 2021-03-15T22:27:39.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:39.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051200+0000", 2021-03-15T22:27:39.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0070519229999999 2021-03-15T22:27:39.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.058252+0000", 2021-03-15T22:27:39.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1403000000000006e-05 2021-03-15T22:27:39.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.053 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:39.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.058333+0000", 2021-03-15T22:27:39.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00081474499999999997 2021-03-15T22:27:39.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:39.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.059148+0000", 2021-03-15T22:27:39.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0081352180000001 2021-03-15T22:27:39.055 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.055 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2223 7.f 7:ffef77ba:::smithi07694878-13:head [sparse-read 0~2702901] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:27:39.056 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:27:39.057 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 280.561865243, 2021-03-15T22:27:39.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.009479572, 2021-03-15T22:27:39.057 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:39.057 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:39.058 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:39.058 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:39.058 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:39.058 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2223 2021-03-15T22:27:39.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.059 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:39.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:39.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:27:39.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:39.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:39.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:27:39.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7300000000000005e-07 2021-03-15T22:27:39.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:39.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:27:39.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.02e-06 2021-03-15T22:27:39.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:39.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051030+0000", 2021-03-15T22:27:39.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9280000000000001e-06 2021-03-15T22:27:39.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:39.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051035+0000", 2021-03-15T22:27:39.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.129e-06 2021-03-15T22:27:39.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:39.065 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051039+0000", 2021-03-15T22:27:39.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018268000000000001 2021-03-15T22:27:39.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051221+0000", 2021-03-15T22:27:39.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4270000000000002e-06 2021-03-15T22:27:39.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:39.067 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051229+0000", 2021-03-15T22:27:39.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.007961946 2021-03-15T22:27:39.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.068 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.059191+0000", 2021-03-15T22:27:39.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010123370000000001 2021-03-15T22:27:39.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:39.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.060203+0000", 2021-03-15T22:27:39.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030523200000000001 2021-03-15T22:27:39.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:39.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.060508+0000", 2021-03-15T22:27:39.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.009479572 2021-03-15T22:27:39.071 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.071 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:39.071 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.072 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3021 7.6 7:6de8b070:::smithi07694878-15:head [copy-from ver 640] snapc e3=[e1,df,de,dd,db] ondisk+write+known_if_redirected e330)", 2021-03-15T22:27:39.072 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:27:39.072 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 213.53027540599999, 2021-03-15T22:27:39.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.124784161, 2021-03-15T22:27:39.073 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:39.073 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:39.073 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:39.073 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:39.074 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:39.074 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3021 2021-03-15T22:27:39.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.074 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:39.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:39.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:27:39.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:39.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.076 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:39.076 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:27:39.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7749999999999999e-06 2021-03-15T22:27:39.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:39.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082622+0000", 2021-03-15T22:27:39.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.072e-06 2021-03-15T22:27:39.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:39.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082625+0000", 2021-03-15T22:27:39.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2641e-05 2021-03-15T22:27:39.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:39.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082638+0000", 2021-03-15T22:27:39.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9170000000000006e-06 2021-03-15T22:27:39.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:39.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082646+0000", 2021-03-15T22:27:39.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1200999999999997e-05 2021-03-15T22:27:39.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.082 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082727+0000", 2021-03-15T22:27:39.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7440000000000007e-06 2021-03-15T22:27:39.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:27:39.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082735+0000", 2021-03-15T22:27:39.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022422739999999998 2021-03-15T22:27:39.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.084977+0000", 2021-03-15T22:27:39.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010620000000000001 2021-03-15T22:27:39.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:39.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085083+0000", 2021-03-15T22:27:39.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10953049500000001 2021-03-15T22:27:39.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,7", 2021-03-15T22:27:39.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.194614+0000", 2021-03-15T22:27:39.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043572510000000004 2021-03-15T22:27:39.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:39.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198971+0000", 2021-03-15T22:27:39.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002724242 2021-03-15T22:27:39.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:39.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.201695+0000", 2021-03-15T22:27:39.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054890260000000001 2021-03-15T22:27:39.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:39.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207184+0000", 2021-03-15T22:27:39.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4066000000000006e-05 2021-03-15T22:27:39.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:39.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207248+0000", 2021-03-15T22:27:39.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015425700000000001 2021-03-15T22:27:39.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:39.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207403+0000", 2021-03-15T22:27:39.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.124784161 2021-03-15T22:27:39.092 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.092 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:39.093 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.093 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3022 7.6 7:6de8b070:::smithi07694878-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e330)", 2021-03-15T22:27:39.094 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:27:39.094 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 213.53023503899999, 2021-03-15T22:27:39.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12494472, 2021-03-15T22:27:39.094 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:39.095 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:39.095 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:39.095 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:39.095 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:39.095 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3022 2021-03-15T22:27:39.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.096 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:39.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:39.097 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:27:39.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:39.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:27:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2669999999999999e-06 2021-03-15T22:27:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082660+0000", 2021-03-15T22:27:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7409999999999999e-06 2021-03-15T22:27:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:39.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082662+0000", 2021-03-15T22:27:39.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2030000000000003e-06 2021-03-15T22:27:39.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082669+0000", 2021-03-15T22:27:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1479999999999999e-06 2021-03-15T22:27:39.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:39.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082673+0000", 2021-03-15T22:27:39.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037115800000000001 2021-03-15T22:27:39.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.083044+0000", 2021-03-15T22:27:39.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1779999999999996e-06 2021-03-15T22:27:39.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.104 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:27:39.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.083049+0000", 2021-03-15T22:27:39.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021797460000000002 2021-03-15T22:27:39.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085228+0000", 2021-03-15T22:27:39.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5878999999999999e-05 2021-03-15T22:27:39.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:39.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085274+0000", 2021-03-15T22:27:39.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.112992462 2021-03-15T22:27:39.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198267+0000", 2021-03-15T22:27:39.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6129000000000002e-05 2021-03-15T22:27:39.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:39.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198343+0000", 2021-03-15T22:27:39.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090944800000000003 2021-03-15T22:27:39.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207437+0000", 2021-03-15T22:27:39.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1060000000000001e-05 2021-03-15T22:27:39.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.110 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:39.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207508+0000", 2021-03-15T22:27:39.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5268999999999999e-05 2021-03-15T22:27:39.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207604+0000", 2021-03-15T22:27:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12494472 2021-03-15T22:27:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.112 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3170 7.6 7:6283974b:::smithi07694878-30:head [copy-from ver 282] snapc f4=[f4,f3,f2,e9,e7,e1,de] ondisk+write+known_if_redirected e344)", 2021-03-15T22:27:39.113 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:19.130807+0000", 2021-03-15T22:27:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 199.48208673400001, 2021-03-15T22:27:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026060351999999998, 2021-03-15T22:27:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:39.114 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3170 2021-03-15T22:27:39.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.116 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:39.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:39.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130807+0000", 2021-03-15T22:27:39.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:39.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:39.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130807+0000", 2021-03-15T22:27:39.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4670000000000001e-06 2021-03-15T22:27:39.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:39.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130809+0000", 2021-03-15T22:27:39.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9299999999999996e-07 2021-03-15T22:27:39.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:39.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130809+0000", 2021-03-15T22:27:39.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5999999999999997e-06 2021-03-15T22:27:39.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:39.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130815+0000", 2021-03-15T22:27:39.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5019999999999999e-06 2021-03-15T22:27:39.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:39.122 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130818+0000", 2021-03-15T22:27:39.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1529999999999997e-05 2021-03-15T22:27:39.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130859+0000", 2021-03-15T22:27:39.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0765000000000003e-05 2021-03-15T22:27:39.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:39.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130940+0000", 2021-03-15T22:27:39.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011008983999999999 2021-03-15T22:27:39.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:27:39.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.141949+0000", 2021-03-15T22:27:39.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075953360000000003 2021-03-15T22:27:39.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:39.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.149544+0000", 2021-03-15T22:27:39.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072008669999999997 2021-03-15T22:27:39.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:39.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156745+0000", 2021-03-15T22:27:39.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4639999999999995e-06 2021-03-15T22:27:39.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:39.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156755+0000", 2021-03-15T22:27:39.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9437999999999997e-05 2021-03-15T22:27:39.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:39.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156794+0000", 2021-03-15T22:27:39.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3505999999999997e-05 2021-03-15T22:27:39.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:39.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156867+0000", 2021-03-15T22:27:39.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026060351999999998 2021-03-15T22:27:39.131 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.131 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:39.131 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.132 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3171 7.6 7:6283974b:::smithi07694878-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e344)", 2021-03-15T22:27:39.132 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:19.130826+0000", 2021-03-15T22:27:39.132 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 199.48206783200001, 2021-03-15T22:27:39.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026195561999999999, 2021-03-15T22:27:39.133 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:39.133 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:39.133 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:39.134 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:39.134 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:39.134 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3171 2021-03-15T22:27:39.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.134 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:39.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:39.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130826+0000", 2021-03-15T22:27:39.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:39.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:39.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130826+0000", 2021-03-15T22:27:39.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6599999999999994e-07 2021-03-15T22:27:39.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:39.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130827+0000", 2021-03-15T22:27:39.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4200000000000004e-07 2021-03-15T22:27:39.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:39.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130828+0000", 2021-03-15T22:27:39.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1210000000000003e-06 2021-03-15T22:27:39.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:39.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130834+0000", 2021-03-15T22:27:39.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.676e-06 2021-03-15T22:27:39.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:39.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130838+0000", 2021-03-15T22:27:39.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023820699999999999 2021-03-15T22:27:39.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.131076+0000", 2021-03-15T22:27:39.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4833999999999998e-05 2021-03-15T22:27:39.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.143 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:39.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.131121+0000", 2021-03-15T22:27:39.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017834289999999999 2021-03-15T22:27:39.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.148955+0000", 2021-03-15T22:27:39.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0327000000000004e-05 2021-03-15T22:27:39.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:39.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.149045+0000", 2021-03-15T22:27:39.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078568969999999998 2021-03-15T22:27:39.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:39.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156902+0000", 2021-03-15T22:27:39.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6823999999999997e-05 2021-03-15T22:27:39.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:39.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156959+0000", 2021-03-15T22:27:39.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2477999999999994e-05 2021-03-15T22:27:39.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:39.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:39.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:39.148 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.157022+0000", 2021-03-15T22:27:39.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026195561999999999 2021-03-15T22:27:39.149 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.149 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:39.149 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.149 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:39.150 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:39.150 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:27:39.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:39.173 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:39.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:39.174 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:39.276 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:39.302 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:39.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:39.304 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:39.366 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:39 smithi076 conmon[56514]: cluster 2021-03-15T22:27:37.906997+0000 osd.4 (osd.4) 113 : cluster [DBG] 4.c scrub ok 2021-03-15T22:27:39.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:39 smithi076 conmon[62156]: cluster 2021-03-15T22:27:37.906997+0000 osd.4 (osd.4) 113 : cluster [DBG] 4.c scrub ok 2021-03-15T22:27:39.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:39.430 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:39.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:39.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:39.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:39.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:39 smithi125 conmon[56739]: cluster 2021-03-15T22:27:37.906997+0000 osd.4 (osd.4) 113 : cluster [DBG] 4.c scrub ok 2021-03-15T22:27:39.559 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:39.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:39.561 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:39.662 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:39.686 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:39.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:39.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:39.789 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:39.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:39.813 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:39.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:39.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:27:39.914 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:39.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:39.942 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:39.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:39.949 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:40.043 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:40.069 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:40.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:40.071 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:40.172 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:40.198 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:40.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:40.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:40.286 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:27:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:27:40.301 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:40.325 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:40.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:40.327 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:40.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[56514]: cluster 2021-03-15T22:27:38.890272+0000 osd.4 (osd. 2021-03-15T22:27:40.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[56514]: 4) 114 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:27:40.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[56514]: cluster 2021-03-15T22:27:39.088650+0000 mgr.y (mgr.14144) 718 : cluster [DBG] pgmap v940: 138 pgs: 6 peering, 2 stale+active+undersized, 4 stale+active+clean, 4 undersized+degraded+peered, 25 active+undersized+degraded, 7 undersized+peered, 42 active+undersized, 48 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 359/1740 objects degraded (20.632%) 2021-03-15T22:27:40.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[56514]: cluster 2021-03-15T22:27:39.160456+0000 osd.3 (osd.3) 93 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:27:40.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[56514]: audit 2021-03-15T22:27:39.837761+0000 mgr.y (mgr.14144) 719 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:40.352 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[56514]: cluster 2021-03-15T22:27:40.142915+0000 mon.a (mon.0) 1518 : cluster [WRN] Health check update: Degraded data redundancy: 359/1740 objects degraded (20.632%), 29 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:40.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[62156]: cluster 2021-03-15T22:27:38.890272+0000 osd.4 (osd.4) 114 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:27:40.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[62156]: cluster 2021-03-15T22:27:39.088650+0000 mgr.y (mgr.14144) 718 : cluster [DBG] pgmap v940: 138 pgs: 6 peering, 2 stale+active+undersized, 4 stale+active+clean, 4 undersized+degraded+peered, 25 active+undersized+degraded, 7 undersized+peered, 42 active+undersized, 48 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 359/1740 objects degraded (20.632%) 2021-03-15T22:27:40.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[62156]: cluster 2021-03-15T22:27:39.160456+0000 osd.3 (osd.3) 93 : cluster 2021-03-15T22:27:40.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[62156]: [DBG] 6.1b scrub ok 2021-03-15T22:27:40.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[62156]: audit 2021-03-15T22:27:39.837761+0000 mgr.y (mgr.14144) 719 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:40.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:40 smithi076 conmon[62156]: cluster 2021-03-15T22:27:40.142915+0000 mon.a (mon.0) 1518 : cluster [WRN] Health check update: Degraded data redundancy: 359/1740 objects degraded (20.632%), 29 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:40.405 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:40.428 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:40.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:40.460 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:40.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:40.562 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:40.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:40.587 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:40.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:40.688 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:40.715 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:40.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:40.716 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: cluster 2021-03-15T22:27:38.890272+0000 osd.4 (osd 2021-03-15T22:27:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: .4) 114 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:27:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: cluster 2021-03-15T22:27:39.088650+0000 mgr.y (mgr.14144) 718 : cluster 2021-03-15T22:27:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: [DBG] pgmap v940: 138 pgs: 6 peering, 2 stale+active+undersized, 4 stale+active+clean, 4 undersized+degraded+peered, 25 active+undersized+degraded, 7 undersized+peered, 42 active+undersized, 48 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 359/1740 objects degraded (20.632%) 2021-03-15T22:27:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: cluster 2021-03-15T22:27:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: 2021-03-15T22:27:39.160456+0000 osd.3 (osd.3) 2021-03-15T22:27:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: 93 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:27:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: audit 2021-03-15T22:27:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: 2021-03-15T 2021-03-15T22:27:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: 22:27:39.837761+0000 mgr.y (mgr 2021-03-15T22:27:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: .14144) 719 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:27:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: 03-15T22:27:40.142915+0000 mon.a (mon.0) 2021-03-15T22:27:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:40 smithi125 conmon[56739]: 1518 : cluster [WRN] Health check update: Degraded data redundancy: 359/1740 objects degraded (20.632%), 29 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:40.818 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:40.846 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:40.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:40.847 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:40.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:40.974 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:40.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:40.976 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:41.077 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:41.102 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:41.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:41.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:41.127 INFO:teuthology.orchestra.run.smithi076.stdout:7.e 2021-03-15T22:27:41.128 INFO:teuthology.orchestra.run.smithi076.stdout:7.8 2021-03-15T22:27:41.128 INFO:teuthology.orchestra.run.smithi076.stdout:7.3 2021-03-15T22:27:41.128 INFO:teuthology.orchestra.run.smithi076.stdout:6.e 2021-03-15T22:27:41.128 INFO:teuthology.orchestra.run.smithi076.stdout:6.9 2021-03-15T22:27:41.128 INFO:teuthology.orchestra.run.smithi076.stdout:6.8 2021-03-15T22:27:41.129 INFO:teuthology.orchestra.run.smithi076.stdout:6.3 2021-03-15T22:27:41.129 INFO:teuthology.orchestra.run.smithi076.stdout:6.1 2021-03-15T22:27:41.129 INFO:teuthology.orchestra.run.smithi076.stdout:6.18 2021-03-15T22:27:41.129 INFO:teuthology.orchestra.run.smithi076.stdout:6.17 2021-03-15T22:27:41.130 INFO:teuthology.orchestra.run.smithi076.stdout:7.a 2021-03-15T22:27:41.130 INFO:teuthology.orchestra.run.smithi076.stdout:4.1c 2021-03-15T22:27:41.130 INFO:teuthology.orchestra.run.smithi076.stdout:7.2 2021-03-15T22:27:41.130 INFO:teuthology.orchestra.run.smithi076.stdout:5.a 2021-03-15T22:27:41.131 INFO:teuthology.orchestra.run.smithi076.stdout:4.1a 2021-03-15T22:27:41.131 INFO:teuthology.orchestra.run.smithi076.stdout:4.12 2021-03-15T22:27:41.131 INFO:teuthology.orchestra.run.smithi076.stdout:6.f 2021-03-15T22:27:41.131 INFO:teuthology.orchestra.run.smithi076.stdout:6.a 2021-03-15T22:27:41.131 INFO:teuthology.orchestra.run.smithi076.stdout:6.2 2021-03-15T22:27:41.132 INFO:teuthology.orchestra.run.smithi076.stdout:5.2 2021-03-15T22:27:41.132 INFO:teuthology.orchestra.run.smithi076.stdout:4.16 2021-03-15T22:27:41.132 INFO:teuthology.orchestra.run.smithi076.stdout:7.5 2021-03-15T22:27:41.132 INFO:teuthology.orchestra.run.smithi076.stdout:3.1e 2021-03-15T22:27:41.133 INFO:teuthology.orchestra.run.smithi076.stdout:3.18 2021-03-15T22:27:41.133 INFO:teuthology.orchestra.run.smithi076.stdout:3.4 2021-03-15T22:27:41.133 INFO:teuthology.orchestra.run.smithi076.stdout:3.1a 2021-03-15T22:27:41.133 INFO:teuthology.orchestra.run.smithi076.stdout:3.1f 2021-03-15T22:27:41.134 INFO:teuthology.orchestra.run.smithi076.stdout:3.0 2021-03-15T22:27:41.134 INFO:teuthology.orchestra.run.smithi076.stdout:3.13 2021-03-15T22:27:41.134 INFO:teuthology.orchestra.run.smithi076.stdout:6.11 2021-03-15T22:27:41.134 INFO:teuthology.orchestra.run.smithi076.stdout:2.1 2021-03-15T22:27:41.135 INFO:teuthology.orchestra.run.smithi076.stdout:5.14 2021-03-15T22:27:41.135 INFO:teuthology.orchestra.run.smithi076.stdout:5.b 2021-03-15T22:27:41.135 INFO:teuthology.orchestra.run.smithi076.stdout:6.1c 2021-03-15T22:27:41.136 INFO:teuthology.orchestra.run.smithi076.stdout:4.4 2021-03-15T22:27:41.136 INFO:teuthology.orchestra.run.smithi076.stdout:3.14 2021-03-15T22:27:41.136 INFO:teuthology.orchestra.run.smithi076.stdout:5.d 2021-03-15T22:27:41.136 INFO:teuthology.orchestra.run.smithi076.stdout:4.6 2021-03-15T22:27:41.136 INFO:teuthology.orchestra.run.smithi076.stdout:4.a 2021-03-15T22:27:41.137 INFO:teuthology.orchestra.run.smithi076.stdout:5.e 2021-03-15T22:27:41.137 INFO:teuthology.orchestra.run.smithi076.stdout:3.9 2021-03-15T22:27:41.137 INFO:teuthology.orchestra.run.smithi076.stdout:4.b 2021-03-15T22:27:41.138 INFO:teuthology.orchestra.run.smithi076.stdout:5.4 2021-03-15T22:27:41.138 INFO:teuthology.orchestra.run.smithi076.stdout:6.14 2021-03-15T22:27:41.138 INFO:teuthology.orchestra.run.smithi076.stdout:4.d 2021-03-15T22:27:41.138 INFO:teuthology.orchestra.run.smithi076.stdout:5.6 2021-03-15T22:27:41.138 INFO:teuthology.orchestra.run.smithi076.stdout:6.d 2021-03-15T22:27:41.139 INFO:teuthology.orchestra.run.smithi076.stdout:6.12 2021-03-15T22:27:41.139 INFO:teuthology.orchestra.run.smithi076.stdout:3.5 2021-03-15T22:27:41.139 INFO:teuthology.orchestra.run.smithi076.stdout:6.13 2021-03-15T22:27:41.139 INFO:teuthology.orchestra.run.smithi076.stdout:4.f 2021-03-15T22:27:41.139 INFO:teuthology.orchestra.run.smithi076.stdout:4.19 2021-03-15T22:27:41.140 INFO:teuthology.orchestra.run.smithi076.stdout:5.8 2021-03-15T22:27:41.140 INFO:teuthology.orchestra.run.smithi076.stdout:6.0 2021-03-15T22:27:41.205 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:41.227 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:41.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:41.308 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:27:41.329 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:41.353 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:41.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:41.355 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:41.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:41.484 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:41.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:41.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:41.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:41.616 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:41.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:41.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:41.685 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: cluster 2021-03-15T22:27:40.278704+0000 osd.7 2021-03-15T22:27:41.685 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: (osd.7) 106 : cluster [DBG] 3.f scrub ok 2021-03-15T22:27:41.686 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: cluster 2021-03-15T22:27:40.473726+0000 osd.1 (osd.1) 105 : 2021-03-15T22:27:41.686 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: cluster [DBG] 5.c scrub ok 2021-03-15T22:27:41.686 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:27:41.686 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: T22:27:40.585889+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? 172.21.15.76:0/1407183818' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:41.687 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: 2021-03-15T22:27:41.687 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: cluster 2021-03-15T22:27:41.687 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: 2021-03-15T22:27:40.589045+0000 osd.3 (osd.3) 2021-03-15T22:27:41.687 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: 94 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:41.687 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:27:41.688 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: 27:40.590277+0000 osd.3 (osd.3) 95 : cluster 2021-03-15T22:27:41.688 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[62156]: [DBG] purged_snaps scrub ok 2021-03-15T22:27:41.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[56514]: cluster 2021-03-15T22:27:40.278704+0000 osd.7 (osd.7) 106 : cluster [DBG] 3.f scrub ok 2021-03-15T22:27:41.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[56514]: cluster 2021-03-15T22:27:40.473726+0000 osd.1 (osd.1) 105 : cluster [DBG] 5.c scrub ok 2021-03-15T22:27:41.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[56514]: audit 2021-03-15T22:27:40.585889+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? 172.21.15.76:0/1407183818' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:41.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[56514]: cluster 2021-03-15T22:27:40.589045+0000 osd.3 (osd.3) 94 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:41.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:41 smithi076 conmon[56514]: cluster 2021-03-15T22:27:40.590277+0000 osd.3 (osd.3) 95 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:41.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:41.742 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:41.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:41.743 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:41.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:41 smithi125 conmon[56739]: cluster 2021-03-15T22:27:40.278704+0000 osd.7 (osd.7) 106 2021-03-15T22:27:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:41 smithi125 conmon[56739]: : cluster [DBG] 3.f scrub ok 2021-03-15T22:27:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:41 smithi125 conmon[56739]: cluster 2021-03-15T22:27:40.473726+0000 osd.1 (osd.1) 105 : cluster [DBG] 5.c scrub ok 2021-03-15T22:27:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:41 smithi125 conmon[56739]: audit 2021-03-15T22:27:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:41 smithi125 conmon[56739]: 2021-03-15T22:27:40.585889+0000 mon.a (mon.0) 1519 : audit [INF] from='client.? 172.21.15.76:0/1407183818' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:41.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:41 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:27:41.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:41 smithi125 conmon[56739]: T22:27:40.589045+0000 osd.3 (osd.3) 94 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:41.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:41 smithi125 conmon[56739]: cluster 2021-03-15T22:27:40.590277+0000 osd.3 ( 2021-03-15T22:27:41.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:41 smithi125 conmon[56739]: osd.3) 95 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:41.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:41.866 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:41.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:41.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:41.969 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:41.991 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:41.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:41.993 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:42.094 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:42.116 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:42.117 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:42.219 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:42.241 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:42.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:42.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:42.344 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:42.366 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:42.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:42.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:42.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:42.491 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:42.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:42.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:42.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:42 smithi076 conmon[56514]: cluster 2021-03-15T22:27:41.089286+0000 mgr.y (mgr.14144) 720 : cluster [DBG] pgmap v941: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.9 MiB/s wr, 14 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:42.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:42 smithi076 conmon[56514]: cluster 2021-03-15T22:27:41.254462+0000 osd.7 (osd.7) 107 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:27:42.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:42 smithi076 conmon[56514]: cluster 2021-03-15T22:27:41.288543+0000 mon.a (mon.0) 1520 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:27:42.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:42 smithi076 conmon[56514]: audit 2021-03-15T22:27:41.301748+0000 mon.a (mon.0) 1521 : audit [INF] from='client.? 172.21.15.76:0/1407183818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:42.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:42 smithi076 conmon[56514]: cluster 2021-03-15T22:27:41.301817+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e542: 8 total, 6 up, 7 in 2021-03-15T22:27:42.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:42 smithi076 conmon[62156]: cluster 2021-03-15T22:27:41.089286+0000 mgr.y (mgr.14144) 720 : cluster [DBG] pgmap v941: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.9 MiB/s wr, 14 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:42.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:42 smithi076 conmon[62156]: cluster 2021-03-15T22:27:41.254462+0000 osd.7 (osd.7) 107 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:27:42.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:42 smithi076 conmon[62156]: cluster 2021-03-15T22:27:41.288543+0000 mon.a (mon.0) 1520 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:27:42.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:42 smithi076 conmon[62156]: audit 2021-03-15T22:27:41.301748+0000 mon.a (mon.0) 1521 : audit [INF] from='client.? 172.21.15.76:0/1407183818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:42.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:42 smithi076 conmon[62156]: cluster 2021-03-15T22:27:41.301817+0000 mon.a (mon.0) 1522 : cluster [DBG] osdmap e542: 8 total, 6 up, 7 in 2021-03-15T22:27:42.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:42.616 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:42.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:42.617 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:42.719 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:42.741 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:42.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:42.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: cluster 2021-03-15T22:27:41.089286+0000 mgr.y ( 2021-03-15T22:27:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: mgr.14144) 720 : cluster [DBG] pgmap v941: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.9 MiB/s wr, 14 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: cluster 2021-03-15T22:27:41.254462+0000 osd.7 2021-03-15T22:27:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: (osd.7) 107 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:27:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:27:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: 27:41.288543+0000 mon.a (mon.0) 2021-03-15T22:27:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: 1520 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:27:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: audit 2021 2021-03-15T22:27:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: -03-15T22:27:41.301748+0000 2021-03-15T22:27:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: mon.a (mon.0) 1521 : audit [INF] 2021-03-15T22:27:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1407183818' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:27:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:27:42.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: :27:41.301817+0000 mon.a (mon. 2021-03-15T22:27:42.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:42 smithi125 conmon[56739]: 0) 1522 : cluster [DBG] osdmap e542: 8 total, 6 up, 7 in 2021-03-15T22:27:42.844 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:42.868 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:42.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:42.870 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:42.971 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:42.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:42.995 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:42.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:43.097 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:43.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:43.123 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:43.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:43.225 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:43.247 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:43.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:43.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:43.349 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:43.371 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:43.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:43.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:43.434 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -n osd.2 --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --op export --pgid 4.1a --file /var/log/ceph/exp.4.1a.2 2021-03-15T22:27:43.474 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:43.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:43.502 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:43.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:43.604 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:43.629 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:43.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:43.663 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/osd.2/config 2021-03-15T22:27:43.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:43.745 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:27:43.756 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:43.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:43.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:43.767 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:27:43.786 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:27:43.804 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:27:43.824 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:27:43.843 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:27:43.843 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:27:43.844 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:27:43.844 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:27:43.844 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:27:43.844 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:27:43.845 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:27:43.845 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:27:43.845 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:27:43.859 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:43.883 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:43.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:43.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:43.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:44.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:44.011 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:44.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:44.036 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:27:44.036 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:27:44.112 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:44.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:44.136 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:44.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:44.238 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:44.262 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:44.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:44.263 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:44.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:44.388 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:44.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:44.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:44.490 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:44.513 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:44.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:44.615 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:44.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:44.640 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:44.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:44.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:44.764 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:44.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:44.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:44.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:44 smithi076 conmon[62156]: cluster 2021-03-15T22:27:43.090201+0000 mgr.y (mgr.14144) 721 : cluster [DBG] pgmap v943: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 4.3 MiB/s rd, 1.1 MiB/s wr, 6 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:44.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:44 smithi076 conmon[62156]: cluster 2021-03-15T22:27:43.184421+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.8 deep-scrub ok 2021-03-15T22:27:44.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:44 smithi076 conmon[62156]: audit 2021-03-15T22:27:43.726600+0000 mon.a (mon.0) 1523 : audit [DBG] from='client.? 172.21.15.76:0/1179654793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:44 smithi076 conmon[56514]: cluster 2021-03-15T22:27:43.090201+0000 mgr.y (mgr.14144) 721 : cluster [DBG] pgmap v943: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 4.3 MiB/s rd, 1.1 MiB/s wr, 6 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:44 smithi076 conmon[56514]: cluster 2021-03-15T22:27:43.184421+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.8 deep-scrub ok 2021-03-15T22:27:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:44 smithi076 conmon[56514]: audit 2021-03-15T22:27:43.726600+0000 mon.a (mon.0) 1523 : audit [DBG] from='client.? 172.21.15.76:0/1179654793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:44.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:44 smithi125 conmon[56739]: cluster 2021-03-15T22:27:43.090201+0000 mgr.y (mgr.14144) 721 2021-03-15T22:27:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:44 smithi125 conmon[56739]: : cluster [DBG] pgmap v943: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 4.3 MiB/s rd, 1.1 MiB/s wr, 6 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:44 smithi125 conmon[56739]: cluster 2021-03-15T22:27:43.184421+0000 osd.3 (osd.3) 96 : cluster [DBG] 3.8 deep-scrub ok 2021-03-15T22:27:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:44 smithi125 conmon[56739]: audit 2021-03-15T22:27:43. 2021-03-15T22:27:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:44 smithi125 conmon[56739]: 726600+0000 mon.a (mon.0) 1523 : audit [DBG] 2021-03-15T22:27:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:44 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1179654793' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:44.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:44.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:44.891 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:44.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:44.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:45.017 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:45.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:45.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:45.119 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:45.133 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:27:45.143 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:45.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:45.145 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:27:45.198 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:27:45.229 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:27:45.246 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:45.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:45.276 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:45.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:45.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[56514]: cluster 2021-03-15T22:27:44.156493 2021-03-15T22:27:45.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[56514]: +0000 osd.3 (osd.3) 97 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:27:45.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[56514]: cluster 2021-03-15T22:27:44.227915+0000 osd.7 (osd.7) 108 : cluster [DBG] 3.e scrub ok 2021-03-15T22:27:45.336 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[56514]: cluster 2021-03-15T22:27:45.144385+0000 mon.a (mon.0) 1524 : cluster [WRN] Health check update: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:27:45.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[56514]: cluster 2021-03-15T22:27:45.144436+0000 mon.a (mon.0) 1525 : cluster [WRN] Health check update: Degraded data redundancy: 439/1743 objects degraded (25.186%), 36 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:45.337 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[56514]: cluster 2021-03-15T22:27:45.150808+0000 mon.a (mon.0) 1526 : cluster [DBG] osdmap e543: 8 total, 6 up, 7 in 2021-03-15T22:27:45.337 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: cluster 2021-03-15T22:27:45.338 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:27:45.338 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 15T22:27:44.156493+0000 osd.3 (osd.3) 97 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:27:45.338 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: cluster 2021-03-15T22:27:45.338 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.338 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021 2021-03-15T22:27:45.339 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: - 2021-03-15T22:27:45.339 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 03 2021-03-15T22:27:45.339 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: - 2021-03-15T22:27:45.339 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 15 2021-03-15T22:27:45.339 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: T 2021-03-15T22:27:45.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 22 2021-03-15T22:27:45.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 27 2021-03-15T22:27:45.340 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.341 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 44 2021-03-15T22:27:45.341 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: . 2021-03-15T22:27:45.341 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 227915 2021-03-15T22:27:45.341 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: +0000 2021-03-15T22:27:45.341 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.342 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: osd.7 2021-03-15T22:27:45.342 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: ( 2021-03-15T22:27:45.342 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: osd 2021-03-15T22:27:45.342 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: . 2021-03-15T22:27:45.342 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 7 2021-03-15T22:27:45.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: ) 2021-03-15T22:27:45.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 108 2021-03-15T22:27:45.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: cluster 2021-03-15T22:27:45.344 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.344 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: [DBG] 2021-03-15T22:27:45.344 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.344 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 3.e scrub ok 2021-03-15T22:27:45.345 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.345 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: cluster 2021-03-15T22:27:45.345 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.345 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021 2021-03-15T22:27:45.345 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: - 2021-03-15T22:27:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 03 2021-03-15T22:27:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: - 2021-03-15T22:27:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 15 2021-03-15T22:27:45.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: T 2021-03-15T22:27:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 22 2021-03-15T22:27:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 27 2021-03-15T22:27:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.347 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 45 2021-03-15T22:27:45.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: . 2021-03-15T22:27:45.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 144385 2021-03-15T22:27:45.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: +0000 2021-03-15T22:27:45.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: mon.a 2021-03-15T22:27:45.349 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: ( 2021-03-15T22:27:45.349 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: mon 2021-03-15T22:27:45.349 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: . 2021-03-15T22:27:45.349 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 0 2021-03-15T22:27:45.349 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: ) 2021-03-15T22:27:45.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 1524 2021-03-15T22:27:45.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: cluster 2021-03-15T22:27:45.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: [WRN] 2021-03-15T22:27:45.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: Health check update: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:27:45.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: cluster 2021-03-15T22:27:45.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021 2021-03-15T22:27:45.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: - 2021-03-15T22:27:45.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 03 2021-03-15T22:27:45.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: - 2021-03-15T22:27:45.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 15 2021-03-15T22:27:45.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: T 2021-03-15T22:27:45.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 22 2021-03-15T22:27:45.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.354 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 27 2021-03-15T22:27:45.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 45 2021-03-15T22:27:45.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: . 2021-03-15T22:27:45.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 144436 2021-03-15T22:27:45.355 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: +0000 2021-03-15T22:27:45.356 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.356 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: mon.a 2021-03-15T22:27:45.357 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: ( 2021-03-15T22:27:45.357 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: mon 2021-03-15T22:27:45.357 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: . 2021-03-15T22:27:45.357 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 0 2021-03-15T22:27:45.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: ) 2021-03-15T22:27:45.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 1525 2021-03-15T22:27:45.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: cluster 2021-03-15T22:27:45.358 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: [WRN] 2021-03-15T22:27:45.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.359 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: Health check update: Degraded data redundancy: 439/1743 objects degraded (25.186%), 36 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:45.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: cluster 2021-03-15T22:27:45.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021 2021-03-15T22:27:45.360 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: - 2021-03-15T22:27:45.361 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 03 2021-03-15T22:27:45.361 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: - 2021-03-15T22:27:45.361 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 15 2021-03-15T22:27:45.361 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: T 2021-03-15T22:27:45.361 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 22 2021-03-15T22:27:45.362 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.362 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 27 2021-03-15T22:27:45.362 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.363 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 45 2021-03-15T22:27:45.363 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: . 2021-03-15T22:27:45.363 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 150808 2021-03-15T22:27:45.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: +0000 2021-03-15T22:27:45.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: mon.a 2021-03-15T22:27:45.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: ( 2021-03-15T22:27:45.364 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: mon 2021-03-15T22:27:45.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: . 2021-03-15T22:27:45.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 0 2021-03-15T22:27:45.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: ) 2021-03-15T22:27:45.365 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 1526 2021-03-15T22:27:45.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: : 2021-03-15T22:27:45.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: cluster 2021-03-15T22:27:45.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.366 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: [DBG] 2021-03-15T22:27:45.367 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.367 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: osdmap e543: 8 total, 6 up, 7 in 2021-03-15T22:27:45.368 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:45 smithi076 conmon[62156]: 2021-03-15T22:27:45.378 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:45.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:45.415 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:45.415 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:45.517 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:45.538 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:45.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:45.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:45.620 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:45.641 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:45.668 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:45.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:45.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:45.771 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:45.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:45 smithi125 conmon[56739]: cluster 2021-03-15T22:27:44.156493 2021-03-15T22:27:45.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:45 smithi125 conmon[56739]: +0000 osd.3 (osd.3) 97 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:27:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:45 smithi125 conmon[56739]: cluster 2021-03-15T22:27:44.227915+0000 osd.7 (osd.7) 108 : cluster [DBG] 3.e scrub ok 2021-03-15T22:27:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:45 smithi125 conmon[56739]: cluster 2021-03-15T22:27:45.144385+0000 mon.a (mon.0) 1524 : cluster [WRN] Health check update: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:27:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:45 smithi125 conmon[56739]: cluster 2021-03-15T22:27:45.144436+0000 mon.a (mon.0) 1525 : cluster [WRN] Health check update: Degraded data redundancy: 439/1743 objects degraded (25.186%), 36 pgs degraded (PG_DEGRADED) 2021-03-15T22:27:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:45 smithi125 conmon[56739]: cluster 2021-03-15T22:27:45.150808+0000 mon.a (mon.0) 1526 : cluster 2021-03-15T22:27:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:45 smithi125 conmon[56739]: [DBG] osdmap e543: 8 total, 6 up, 7 in 2021-03-15T22:27:45.794 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:45.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:45.796 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:45.897 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:45.919 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:45.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:45.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:46.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:46.044 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:46.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:46.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:46.147 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:46.169 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:46.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:46.171 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:46.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:46.294 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:46.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:46.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:46.352 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:46.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:46.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:46.423 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:46.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:46.465 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:46 smithi076 conmon[56514]: cluster 2021-03-15T22:27:45.090844+0000 mgr.y (mgr.14144) 722 : cluster [DBG] pgmap v944: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.6 MiB/s rd, 988 KiB/s wr, 4 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:46.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:46 smithi076 conmon[62156]: cluster 2021-03-15T22:27:45.090844+0000 mgr.y (mgr.14144) 2021-03-15T22:27:46.466 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:46 smithi076 conmon[62156]: 722 : cluster [DBG] pgmap v944: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.6 MiB/s rd, 988 KiB/s wr, 4 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:46.525 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:46.550 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:46.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:46.552 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:46.653 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:46.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:46.677 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:46.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:46.779 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:46.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:46 smithi125 conmon[56739]: cluster 2021-03-15T22:27:45.090844+0000 mgr.y (mgr.14144) 722 : cluster [DBG] pgmap v944: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.6 MiB/s rd, 988 KiB/s wr, 4 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:46.805 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:46.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:46.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:46.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:46.933 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:46.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:46.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:47.036 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:47.059 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:47.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:47.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:47.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:47.188 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:47.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:47.190 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:47.200 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:27:47.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:47.316 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:47.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:47.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:47.419 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:47.443 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:47.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:47.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:47.545 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:47.571 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:47.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:47.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:47.674 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:47.698 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:47.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:47.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:47.800 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:47.823 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:47.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:47.824 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:47.913 INFO:teuthology.orchestra.run.smithi076.stderr:Exporting 4.1a info 4.1a( v 65'13 (0'0,65'13] local-lis/les=355/356 n=8 ec=58/58 lis/c=355/355 les/c/f=356/356/0 sis=355) 2021-03-15T22:27:47.915 INFO:teuthology.orchestra.run.smithi076.stderr:Read #4:58680d93:::data_log.19:head# 2021-03-15T22:27:47.916 INFO:teuthology.orchestra.run.smithi076.stderr:Read #4:5909e671:gc::gc.27:head# 2021-03-15T22:27:47.916 INFO:teuthology.orchestra.run.smithi076.stderr:Read #4:5930a485:::data_log.15:head# 2021-03-15T22:27:47.916 INFO:teuthology.orchestra.run.smithi076.stderr:Read #4:59893ed7:::data_log.46:head# 2021-03-15T22:27:47.916 INFO:teuthology.orchestra.run.smithi076.stderr:Read #4:5a206276:::obj_delete_at_hint.0000000077:head# 2021-03-15T22:27:47.917 INFO:teuthology.orchestra.run.smithi076.stderr:Read #4:5ae4284b:::obj_delete_at_hint.0000000032:head# 2021-03-15T22:27:47.917 INFO:teuthology.orchestra.run.smithi076.stderr:Read #4:5c7e8a6d:reshard::reshard.0000000015:head# 2021-03-15T22:27:47.917 INFO:teuthology.orchestra.run.smithi076.stderr:Read #4:5c9fd2c7:::data_log.77:head# 2021-03-15T22:27:47.917 INFO:teuthology.orchestra.run.smithi076.stderr:Export successful 2021-03-15T22:27:47.926 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:47.951 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:47.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:47.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:48.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:48.076 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:48.077 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:48.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:48.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:48.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:48.204 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:48.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:48.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:48.307 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:48.329 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: debug 2021-03-15T22:27:48.284+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:27:48.329 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:27:48.330 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:48.330 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:27:48.330 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:48.330 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:27:48.331 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:48.331 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:48.331 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:27:48.331 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: debug 2021-03-15T22:27:48.285+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:27:48.332 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:27:48.332 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:48.332 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:27:48.332 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:48.332 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:27:48.333 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:48.333 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:48.333 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:27:48.333 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: debug 2021-03-15T22:27:48.285+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:27:48.334 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:27:48.334 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:48.334 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:27:48.334 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:48.334 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:27:48.335 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:48.335 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:48.335 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:27:48.335 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: debug 2021-03-15T22:27:48.285+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:27:48.335 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:27:48.336 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:48.336 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:27:48.336 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:48.336 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:27:48.337 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:48.337 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:48.337 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:27:48.337 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: debug 2021-03-15T22:27:48.286+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:27:48.337 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:27:48.338 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:48.338 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:27:48.339 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:48.339 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:27:48.339 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:48.339 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:48.339 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:27:48.340 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: debug 2021-03-15T22:27:48.286+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:27:48.340 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:27:48.340 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:48.340 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:27:48.340 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:48.341 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:27:48.341 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:48.341 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:48.341 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:27:48.342 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: debug 2021-03-15T22:27:48.286+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:27:48.342 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:27:48.342 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:48.342 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:27:48.343 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:48.343 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:27:48.343 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:48.343 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:48.344 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:27:48.344 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56514]: cluster 2021-03-15T22:27:47.091673+0000 mgr.y (mgr.14144) 723 : cluster [DBG] pgmap v946: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:48.344 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56514]: cluster 2021-03-15T22:27:47.109632+0000 osd.3 (osd.3) 98 : cluster [DBG] 5.0 deep-scrub ok 2021-03-15T22:27:48.345 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56514]: audit 2021-03-15T22:27:48.001435+0000 mon.a (mon.0) 1527 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:27:48.345 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[56514]: audit 2021-03-15T22:27:48.284476+0000 mon.a (mon.0) 1528 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:27:48.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:48.346 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:48.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:48.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:48.470 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:48.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:48.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:48.545 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:27:48.545 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:27:48.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:48.597 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:48.598 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:48.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[62156]: cluster 2021-03-15T22:27:47.091673+0000 mgr.y (mgr.14144) 723 : cluster [DBG] pgmap v946: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:48.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[62156]: cluster 2021-03-15T22:27:47.109632+0000 osd.3 (osd.3) 98 : cluster [DBG] 5.0 deep-scrub ok 2021-03-15T22:27:48.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:27:48.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[62156]: -15T22:27:48.001435+0000 mon.a (mon.0) 2021-03-15T22:27:48.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[62156]: 1527 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:27:48.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:27:48.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[62156]: -15T22:27:48.284476+0000 mon.a (mon.0) 2021-03-15T22:27:48.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:48 smithi076 conmon[62156]: 1528 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:27:48.699 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:48.722 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:48.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:48.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:48.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:48 smithi125 conmon[56739]: cluster 2021-03-15T22:27:47.091673+0000 mgr.y (mgr.14144) 723 : cluster [DBG] pgmap v946: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 1 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:48 smithi125 conmon[56739]: cluster 2021-03-15T22:27:47.109632+0000 osd.3 (osd.3) 98 : cluster [DBG] 5.0 deep-scrub ok 2021-03-15T22:27:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:48 smithi125 conmon[56739]: audit 2021-03-15T22:27:48.001435+0000 mon.a (mon.0) 1527 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:27:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:48 smithi125 conmon[56739]: audit 2021-03-15T22:27:48.284476+0000 mon.a (mon.0) 1528 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:27:48.825 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:48.847 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:48.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:48.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:48.973 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:48.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:48.974 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:49.075 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:49.098 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:49.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:49.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:49.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:49.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:49.232 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:49.233 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:49.334 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:49.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:49.359 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:49.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:49.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:49.484 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:49.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:49.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:49.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:49.610 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:49.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:49.612 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:49.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:27:48.280899+0000 mgr.y (mgr.14144) 724 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:27:49.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:27:48.285387+0000 mgr.y (mgr.14144) 725 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:27:49.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:27:49.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:27:49.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:27:49.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:27:49.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:27:48.285763+0000 mgr.y (mgr.14144) 726 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:27:49.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:27:49.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:27:49.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:27:49.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:27:49.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:27:48.286157+0000 mgr.y (mgr.14144) 727 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:27:49.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:27:49.679 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.679 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:27:49.679 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.679 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:27:49.680 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.680 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:27:48.280899+0000 mgr.y (mgr.14144) 724 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:27:49.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:27:48.285387+0000 mgr.y (mgr.14144) 725 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:27:49.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:27:49.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:27:49.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.682 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:27:49.682 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.682 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:27:49.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:27:48.285763+0000 mgr.y (mgr.14144) 726 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:27:49.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:27:49.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:27:49.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:27:49.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:27:49.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:27:48.286157+0000 mgr.y (mgr.14144) 727 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:27:49.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:27:49.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:27:49.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:27:49.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.687 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:27:49.688 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:27:48.286499+0000 mgr.y (mgr.14144) 728 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:27:49.688 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:27:49.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:27:49.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:27:49.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:27:49.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:27:48.286877+0000 mgr.y (mgr.14144) 729 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:27:49.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:27:49.692 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.692 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:27:49.692 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.693 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:27:49.693 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.694 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.695 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:27:49.698 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:27:48.287225+0000 mgr.y (mgr.14144) 730 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:27:49.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:27:49.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:27:49.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:27:49.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.711 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.712 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:27:49.712 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:27:48.287560+0000 mgr.y (mgr.14144) 731 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:27:49.712 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:27:49.712 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.713 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:27:49.713 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.713 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:27:49.713 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.714 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.714 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:27:49.715 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:27:49.715 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:27:48.286499+0000 mgr.y (mgr.14144) 728 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:27:49.715 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:27:49.715 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.716 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:27:49.716 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.716 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:27:49.716 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:27:49.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:27:48.286877+0000 mgr.y (mgr.14144) 729 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:27:49.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:27:49.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:27:49.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:27:49.720 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.720 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.720 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:27:49.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:27:48.287225+0000 mgr.y (mgr.14144) 730 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:27:49.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:27:49.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:27:49.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:27:49.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:27:49.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:27:48.287560+0000 mgr.y (mgr.14144) 731 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:27:49.724 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:27:49.725 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.725 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:27:49.725 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.726 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:27:49.726 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.726 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.727 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:27:49.727 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:49.755 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:49.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:49.756 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:49.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:27:48.280899+0000 mgr.y (mgr.14144) 724 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:27:49.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:27:48. 2021-03-15T22:27:49.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: 285387+0000 mgr.y (mgr.14144) 725 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:27:49.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:27:49.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:27:49.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:27:49.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:27:49.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:27:48.285763+0000 mgr.y (mgr.14144) 726 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:27:49.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:27:49.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:27:49.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:27:49.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:27:49.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: cephadm 2021-03-15T 2021-03-15T22:27:49.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: 22:27:48.286157+0000 mgr.y (mgr.14144) 727 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:27:49.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:27:49.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:27:49.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:27:49.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:27:49.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:27:48.286499+0000 mgr.y (mgr.14144) 728 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:27:49.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:27:49.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:27:49.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:27:49.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:27:49.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:27:48.286877+0000 mgr.y (mgr.14144) 2021-03-15T22:27:49.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: 729 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:27:49.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:27:49.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:27:49.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:27:49.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:27:49.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:27:48.287225+0000 mgr.y (mgr.14144) 730 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:27:49.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:27:49.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:27:49.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:27:49.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:27:49.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:27:48.287560+0000 mgr.y (mgr.14144) 731 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:27:49.812 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:27:49.813 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:27:49.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:27:49.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:27:49.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:27:49.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:27:49.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:27:49.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:27:49.846 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -n osd.2 --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-2 --journal-path /var/lib/ceph/osd/ceph-2/journal --force --op remove --pgid 4.1a 2021-03-15T22:27:49.858 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:49.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:49.891 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:49.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:49.948 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:27:49.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:50.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:50.028 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:50.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:50.086 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/osd.2/config 2021-03-15T22:27:50.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:50.164 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:50.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:50.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:50.268 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:50.286 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:27:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:27:50.301 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:50.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:50.302 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:50.404 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:50.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:50.433 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:50.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:50.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[56514]: cluster 2021-03-15T22:27:49.092361 2021-03-15T22:27:50.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 732 : cluster [DBG] pgmap v947: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:50.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[56514]: cluster 2021-03-15T22:27:49.384100+0000 osd.6 (osd.6) 85 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:27:50.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[56514]: audit 2021-03-15T22:27:49.848640+0000 mgr.y (mgr.14144) 733 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:50.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[56514]: cluster 2021-03-15T22:27:50.146557+0000 mon.a (mon.0) 1529 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:27:50.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[56514]: cluster 2021-03-15T22:27:50.146595+0000 mon.a (mon.0) 1530 : cluster [WRN] Health check update: Degraded data redundancy: 439/1743 objects degraded (25.186%), 36 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2021-03-15T22:27:50.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: cluster 2021-03-15T22:27:49. 2021-03-15T22:27:50.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: 092361+0000 mgr.y (mgr.14144) 732 : cluster [DBG] pgmap v947: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:50.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:27:50.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: 27:49.384100+0000 osd.6 (osd.6) 85 2021-03-15T22:27:50.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: : cluster [DBG] 7.7 scrub ok 2021-03-15T22:27:50.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:27:50.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: -15T22:27:49.848640+0000 mgr.y (mgr.14144) 733 : audit 2021-03-15T22:27:50.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:50.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:27:50.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: -03-15T22:27:50.146557+0000 mon.a (mon.0) 1529 2021-03-15T22:27:50.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:27:50.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:27:50.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: T22:27:50.146595+0000 mon.a (mon 2021-03-15T22:27:50.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:50 smithi076 conmon[62156]: .0) 1530 : cluster [WRN] Health check update: Degraded data redundancy: 439/1743 objects degraded (25.186%), 36 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2021-03-15T22:27:50.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:50.557 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:50.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:50.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:50.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:50.686 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:50.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:50.688 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:50.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: cluster 2021-03-15T22:27:49.092361+0000 mgr.y (mgr.14144) 732 2021-03-15T22:27:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: : cluster [DBG] pgmap v947: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: cluster 2021-03-15T22:27:49.384100+0000 osd.6 (osd.6) 85 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:27:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: audit 2021-03-15T22:27:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: 2021-03-15T22:27:49.848640+0000 mgr.y (mgr.14144) 733 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:27:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:27:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: 03-15T22:27:50.146557+0000 2021-03-15T22:27:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: mon.a (mon.0) 1529 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:27:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: 2021-03-15T22:27:50.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: cluster 2021-03-15T22:27 2021-03-15T22:27:50.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: :50.146595+0000 mon.a (mon.0) 2021-03-15T22:27:50.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:50 smithi125 conmon[56739]: 1530 : cluster [WRN] Health check update: Degraded data redundancy: 439/1743 objects degraded (25.186%), 36 pgs degraded, 35 pgs undersized (PG_DEGRADED) 2021-03-15T22:27:50.790 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:50.812 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:50.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:50.814 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:50.826 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:50.915 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:50.941 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:50.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:50.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:51.044 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:51.070 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:51.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:51.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:51.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:51.199 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:51.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:51.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:51.303 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:51.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:51.330 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:51.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:51.359 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:27:51.424 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:27:51.432 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:51.442 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:27:51.463 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:51.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:51.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:51.475 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:27:51.501 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:27:51.533 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:27:51.555 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:27:51.556 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:27:51.556 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:27:51.556 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:27:51.557 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:27:51.557 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:27:51.557 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:27:51.557 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:27:51.558 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:27:51.566 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:51.587 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:51.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:51.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:51.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:51 smithi076 conmon[56514]: audit 2021-03-15T22:27:50.570247+0000 mon.a (mon.0) 1531 : audit [INF] from='client.? 172.21.15.76:0/1861090655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:27:51.621 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:51 smithi076 conmon[62156]: audit 2021-03-15T22:27:50.570247+0000 mon.a (mon.0) 1531 : audit [INF] from='client.? 172.21.15.76:0/1861090655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:27:51.691 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:51.716 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:51.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:51.717 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:51.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:51 smithi125 conmon[56739]: audit 2021-03-15T22:27:50.570247+0000 mon.a (mon.0) 1531 : audit 2021-03-15T22:27:51.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:51 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/1861090655' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:27:51.818 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:51.842 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:51.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:51.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:51.965 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:51.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:51.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:52.068 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:52.090 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:52.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:52.092 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:52.193 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:52.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:52.217 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:52.217 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:52.318 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:52.341 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:52.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:52.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:52.444 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:52.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:52.469 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:52.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:52.540 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:27:52.541 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:27:52.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:52.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:52.594 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:52.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:52.696 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:52.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:52.720 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:52.720 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:52.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[56514]: cluster 2021-03-15T22:27:51.093281+0000 mgr.y (mgr.14144) 734 : cluster [DBG] pgmap v948: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:52.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[56514]: cluster 2021-03-15T22:27:51.335247+0000 mon.a (mon.0) 1532 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:52.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[56514]: audit 2021-03-15T22:27:51.346733+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? 172.21.15.76:0/1861090655' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:27:52.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[56514]: cluster 2021-03-15T22:27:51.346789+0000 mon.a (mon.0) 1534 : cluster [DBG] osdmap e544: 8 total, 6 up, 7 in 2021-03-15T22:27:52.740 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[56514]: audit 2021-03-15T22:27:51.404175+0000 mon.a (mon.0) 1535 : audit [DBG] from='client.? 172.21.15.76:0/1786314437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:52.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: cluster 2021-03-15T22:27:51.093281+0000 2021-03-15T22:27:52.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: mgr.y (mgr.14144) 734 : cluster [DBG] pgmap v948: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:52.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: cluster 2021-03-15T22:27:51.335247 2021-03-15T22:27:52.741 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1532 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:52.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:27:52.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: 03-15T22:27:51 2021-03-15T22:27:52.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: .346733+0000 mon.a (mon.0 2021-03-15T22:27:52.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: ) 1533 : audit [INF] from='client.? 172.21.15.76:0/1861090655' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:27:52.742 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: cluster 2021-03-15T22:27:52.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: 2021-03-15T22:27 2021-03-15T22:27:52.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: :51.346789+0000 mon.a (mon.0) 2021-03-15T22:27:52.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: 1534 : cluster [DBG] osdmap e544: 8 total, 6 up, 7 in 2021-03-15T22:27:52.743 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: audit 2021 2021-03-15T22:27:52.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: -03-15T22:27:51 2021-03-15T22:27:52.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: .404175+0000 mon.a (mon.0) 1535 : 2021-03-15T22:27:52.744 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:52 smithi076 conmon[62156]: audit [DBG] from='client.? 172.21.15.76:0/1786314437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:52.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: cluster 2021-03-15T22:27:51.093281+0000 mgr.y (mgr.14144) 734 2021-03-15T22:27:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: : cluster [DBG] pgmap v948: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.0 KiB/s rd, 1 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: cluster 2021-03-15T22:27:51.335247+0000 mon.a (mon.0) 1532 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:27:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:27:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: 15T22:27:51.346733+0000 mon.a (mon.0) 1533 : audit [INF] from='client.? 172.21.15.76:0/1861090655' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:27:52.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:27:52.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: -15T22:27:51.346789+0000 2021-03-15T22:27:52.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: mon.a (mon.0) 1534 : cluster [DBG] 2021-03-15T22:27:52.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: osdmap e544: 8 total, 6 up, 7 in 2021-03-15T22:27:52.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: audit 2021-03-15T22:27 2021-03-15T22:27:52.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: :51.404175+0000 mon.a (mon.0) 2021-03-15T22:27:52.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:52 smithi125 conmon[56739]: 1535 : audit [DBG] from='client.? 172.21.15.76:0/1786314437' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:52.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:52.844 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:52.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:52.846 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:52.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:52.972 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:52.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:52.973 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:53.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:53.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:53.099 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:53.099 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:53.201 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:53.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:53.225 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:53.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:53.327 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:53.354 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:53.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:53.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:53.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:53.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:53.483 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:53.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:53.585 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:53.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:53.609 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:53.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:53.711 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:53.733 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:53.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:53.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:53.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:53.858 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:53.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:53.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:53.961 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:53.983 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:53.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:53.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:54.086 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:54.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:54.110 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:54.111 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:54.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:54.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:54.236 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:54.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:54.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:54.367 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:54.369 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:54.468 INFO:teuthology.orchestra.run.smithi076.stdout: marking collection for removal 2021-03-15T22:27:54.468 INFO:teuthology.orchestra.run.smithi076.stdout:setting '_remove' omap key 2021-03-15T22:27:54.468 INFO:teuthology.orchestra.run.smithi076.stdout:finish_remove_pgs 4.1a_head removing 4.1a 2021-03-15T22:27:54.469 INFO:teuthology.orchestra.run.smithi076.stdout:Remove successful 2021-03-15T22:27:54.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:54.491 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:54.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:54.501 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:27:54.501 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:27:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:27:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:27:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.502 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDScrubReserve(6.1b REQUEST e194)", 2021-03-15T22:27:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:27:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 364.71435212300003, 2021-03-15T22:27:54.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.061386018000000001, 2021-03-15T22:27:54.504 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.504 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:54.504 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:27:54.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:27:54.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6899999999999999e-06 2021-03-15T22:27:54.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783093+0000", 2021-03-15T22:27:54.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5640000000000001e-06 2021-03-15T22:27:54.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783097+0000", 2021-03-15T22:27:54.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7742e-05 2021-03-15T22:27:54.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783125+0000", 2021-03-15T22:27:54.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3793e-05 2021-03-15T22:27:54.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783138+0000", 2021-03-15T22:27:54.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8529999999999993e-05 2021-03-15T22:27:54.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783237+0000", 2021-03-15T22:27:54.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.241e-05 2021-03-15T22:27:54.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:27:54.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783249+0000", 2021-03-15T22:27:54.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013943790000000001 2021-03-15T22:27:54.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.514 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.784644+0000", 2021-03-15T22:27:54.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2398000000000001e-05 2021-03-15T22:27:54.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.784656+0000", 2021-03-15T22:27:54.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.059821512 2021-03-15T22:27:54.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.844478+0000", 2021-03-15T22:27:54.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.061386018000000001 2021-03-15T22:27:54.517 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.517 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.518 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.518 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.6.0:214 7.d e205/91 7:bd183e1b:::smithi07694878-46:68 v 205'30, mlcod=202'27)", 2021-03-15T22:27:54.519 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:27:54.519 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 354.40832910699999, 2021-03-15T22:27:54.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036819649000000003, 2021-03-15T22:27:54.519 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.519 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.520 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:27:54.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089115+0000", 2021-03-15T22:27:54.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2879999999999999e-06 2021-03-15T22:27:54.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089117+0000", 2021-03-15T22:27:54.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4639e-05 2021-03-15T22:27:54.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089132+0000", 2021-03-15T22:27:54.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0759e-05 2021-03-15T22:27:54.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089142+0000", 2021-03-15T22:27:54.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8793000000000001e-05 2021-03-15T22:27:54.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089161+0000", 2021-03-15T22:27:54.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3105999999999998e-05 2021-03-15T22:27:54.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089194+0000", 2021-03-15T22:27:54.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3468e-05 2021-03-15T22:27:54.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.089208+0000", 2021-03-15T22:27:54.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036675487 2021-03-15T22:27:54.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.529 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.529 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125883+0000", 2021-03-15T22:27:54.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1109000000000001e-05 2021-03-15T22:27:54.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125934+0000", 2021-03-15T22:27:54.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036819649000000003 2021-03-15T22:27:54.531 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.531 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.531 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.532 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2768 7.b 7:d10ba14d:::smithi07694878-43:head [write 1689059~741123] snapc d2=[d2,d1,cf] ondisk+write+known_if_redirected e310)", 2021-03-15T22:27:54.532 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:27:54.532 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 249.39235375300001, 2021-03-15T22:27:54.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078094859000000003, 2021-03-15T22:27:54.533 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.533 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.533 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.534 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.534 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.534 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2768 2021-03-15T22:27:54.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.534 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.535 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:27:54.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:27:54.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033110399999999998 2021-03-15T22:27:54.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105421+0000", 2021-03-15T22:27:54.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4809999999999998e-06 2021-03-15T22:27:54.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105425+0000", 2021-03-15T22:27:54.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7363999999999998e-05 2021-03-15T22:27:54.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105442+0000", 2021-03-15T22:27:54.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9480000000000002e-06 2021-03-15T22:27:54.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105449+0000", 2021-03-15T22:27:54.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6847000000000001e-05 2021-03-15T22:27:54.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105496+0000", 2021-03-15T22:27:54.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8231000000000003e-05 2021-03-15T22:27:54.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105584+0000", 2021-03-15T22:27:54.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000198369 2021-03-15T22:27:54.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:27:54.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105782+0000", 2021-03-15T22:27:54.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003007292 2021-03-15T22:27:54.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108790+0000", 2021-03-15T22:27:54.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000200027 2021-03-15T22:27:54.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108990+0000", 2021-03-15T22:27:54.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074107929000000003 2021-03-15T22:27:54.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:54.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183098+0000", 2021-03-15T22:27:54.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4002000000000002e-05 2021-03-15T22:27:54.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183152+0000", 2021-03-15T22:27:54.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3265000000000001e-05 2021-03-15T22:27:54.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183185+0000", 2021-03-15T22:27:54.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078094859000000003 2021-03-15T22:27:54.551 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.551 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.551 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.552 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2769 7.b 7:d10ba14d:::smithi07694878-43:head [setxattr _header (58) in=7b,truncate 2588344] snapc d2=[d2,d1,cf] ondisk+write+known_if_redirected e310)", 2021-03-15T22:27:54.552 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:27:54.553 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 249.39198272499999, 2021-03-15T22:27:54.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077795487999999996, 2021-03-15T22:27:54.553 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.553 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.553 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.554 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.554 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.554 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2769 2021-03-15T22:27:54.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.555 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:27:54.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:27:54.557 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2639999999999998e-06 2021-03-15T22:27:54.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105463+0000", 2021-03-15T22:27:54.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.426e-06 2021-03-15T22:27:54.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105465+0000", 2021-03-15T22:27:54.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9349999999999999e-06 2021-03-15T22:27:54.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105471+0000", 2021-03-15T22:27:54.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2420000000000002e-06 2021-03-15T22:27:54.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105475+0000", 2021-03-15T22:27:54.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001413924 2021-03-15T22:27:54.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.106889+0000", 2021-03-15T22:27:54.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5030000000000003e-05 2021-03-15T22:27:54.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.563 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.563 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.106984+0000", 2021-03-15T22:27:54.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014550399999999999 2021-03-15T22:27:54.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.564 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:27:54.564 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107129+0000", 2021-03-15T22:27:54.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001729529 2021-03-15T22:27:54.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.565 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108859+0000", 2021-03-15T22:27:54.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023022799999999999 2021-03-15T22:27:54.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.566 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.109089+0000", 2021-03-15T22:27:54.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074107424000000005 2021-03-15T22:27:54.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:54.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183197+0000", 2021-03-15T22:27:54.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7657e-05 2021-03-15T22:27:54.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183224+0000", 2021-03-15T22:27:54.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2325e-05 2021-03-15T22:27:54.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.569 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.570 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183257+0000", 2021-03-15T22:27:54.570 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077795487999999996 2021-03-15T22:27:54.570 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.570 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.571 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.571 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2770 7.b 7:d10ba14d:::smithi07694878-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e310)", 2021-03-15T22:27:54.571 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:27:54.572 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 249.39195251699999, 2021-03-15T22:27:54.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077923611000000004, 2021-03-15T22:27:54.572 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.572 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:54.572 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.573 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.573 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.573 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2770 2021-03-15T22:27:54.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.574 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:27:54.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:27:54.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3200000000000001e-06 2021-03-15T22:27:54.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.576 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105493+0000", 2021-03-15T22:27:54.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3960000000000001e-06 2021-03-15T22:27:54.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.577 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105494+0000", 2021-03-15T22:27:54.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6490000000000002e-06 2021-03-15T22:27:54.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105501+0000", 2021-03-15T22:27:54.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9989999999999999e-06 2021-03-15T22:27:54.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.581 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.581 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105506+0000", 2021-03-15T22:27:54.581 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019212770000000001 2021-03-15T22:27:54.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.582 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.582 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.582 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107427+0000", 2021-03-15T22:27:54.582 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8102000000000005e-05 2021-03-15T22:27:54.583 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.583 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.583 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:54.583 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107515+0000", 2021-03-15T22:27:54.584 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075777318999999996 2021-03-15T22:27:54.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.585 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.585 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183292+0000", 2021-03-15T22:27:54.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9417999999999999e-05 2021-03-15T22:27:54.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.586 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183352+0000", 2021-03-15T22:27:54.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3131000000000003e-05 2021-03-15T22:27:54.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183415+0000", 2021-03-15T22:27:54.588 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077923611000000004 2021-03-15T22:27:54.588 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.589 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.589 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.589 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3010 7.b 7:d980be1f:::smithi07694878-23:head [write 726168~704991,stat] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:27:54.590 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:27:54.590 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 231.410930939, 2021-03-15T22:27:54.590 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0570805230000002, 2021-03-15T22:27:54.590 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.591 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.591 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3010 2021-03-15T22:27:54.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.592 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.592 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:27:54.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:27:54.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034716800000000002 2021-03-15T22:27:54.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086860+0000", 2021-03-15T22:27:54.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3299999999999999e-06 2021-03-15T22:27:54.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086863+0000", 2021-03-15T22:27:54.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1489e-05 2021-03-15T22:27:54.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086875+0000", 2021-03-15T22:27:54.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.605e-06 2021-03-15T22:27:54.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086882+0000", 2021-03-15T22:27:54.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5296999999999997e-05 2021-03-15T22:27:54.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.599 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.599 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086928+0000", 2021-03-15T22:27:54.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2879999999999998e-06 2021-03-15T22:27:54.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:54.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086937+0000", 2021-03-15T22:27:54.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99891912599999999 2021-03-15T22:27:54.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085856+0000", 2021-03-15T22:27:54.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1169999999999993e-06 2021-03-15T22:27:54.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:54.602 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085865+0000", 2021-03-15T22:27:54.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0140395639999999 2021-03-15T22:27:54.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.099905+0000", 2021-03-15T22:27:54.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022436600000000001 2021-03-15T22:27:54.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.100129+0000", 2021-03-15T22:27:54.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00043622700000000002 2021-03-15T22:27:54.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:27:54.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.100565+0000", 2021-03-15T22:27:54.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024326999999999999 2021-03-15T22:27:54.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:54.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102998+0000", 2021-03-15T22:27:54.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002982012 2021-03-15T22:27:54.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.608 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.105980+0000", 2021-03-15T22:27:54.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037236714999999997 2021-03-15T22:27:54.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143217+0000", 2021-03-15T22:27:54.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147832 2021-03-15T22:27:54.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143365+0000", 2021-03-15T22:27:54.610 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022868699999999999 2021-03-15T22:27:54.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.611 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143593+0000", 2021-03-15T22:27:54.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0570805230000002 2021-03-15T22:27:54.611 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.611 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.611 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.612 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3011 7.b 7:d980be1f:::smithi07694878-23:head [write 1895071~683474] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:27:54.612 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:27:54.612 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 231.40465069300001, 2021-03-15T22:27:54.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.09731113, 2021-03-15T22:27:54.613 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.613 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.613 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.614 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.614 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.614 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3011 2021-03-15T22:27:54.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.614 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.615 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.615 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:27:54.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.616 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.616 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:27:54.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027883500000000001 2021-03-15T22:27:54.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093072+0000", 2021-03-15T22:27:54.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.809e-06 2021-03-15T22:27:54.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093077+0000", 2021-03-15T22:27:54.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9491e-05 2021-03-15T22:27:54.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.619 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.619 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093096+0000", 2021-03-15T22:27:54.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.126e-06 2021-03-15T22:27:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093103+0000", 2021-03-15T22:27:54.620 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4002000000000001e-05 2021-03-15T22:27:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.621 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093167+0000", 2021-03-15T22:27:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0259999999999996e-06 2021-03-15T22:27:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:54.622 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093176+0000", 2021-03-15T22:27:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99270957599999998 2021-03-15T22:27:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085886+0000", 2021-03-15T22:27:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9440000000000002e-06 2021-03-15T22:27:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085892+0000", 2021-03-15T22:27:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.016173532 2021-03-15T22:27:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102065+0000", 2021-03-15T22:27:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000141037 2021-03-15T22:27:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102207+0000", 2021-03-15T22:27:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041447062 2021-03-15T22:27:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143654+0000", 2021-03-15T22:27:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014246199999999999 2021-03-15T22:27:54.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143796+0000", 2021-03-15T22:27:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000272444 2021-03-15T22:27:54.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:27:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.144068+0000", 2021-03-15T22:27:54.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018956929999999999 2021-03-15T22:27:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145964+0000", 2021-03-15T22:27:54.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041997686999999999 2021-03-15T22:27:54.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.632 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:54.633 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.187962+0000", 2021-03-15T22:27:54.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019156150000000001 2021-03-15T22:27:54.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.633 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.189877+0000", 2021-03-15T22:27:54.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8467999999999995e-05 2021-03-15T22:27:54.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.189956+0000", 2021-03-15T22:27:54.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014832100000000001 2021-03-15T22:27:54.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190104+0000", 2021-03-15T22:27:54.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.09731113 2021-03-15T22:27:54.636 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.636 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.637 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.637 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3012 7.b 7:d980be1f:::smithi07694878-23:head [setxattr _header (58) in=7b,truncate 3126168] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:27:54.637 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:27:54.638 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 231.40432786100001, 2021-03-15T22:27:54.638 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0972930829999998, 2021-03-15T22:27:54.638 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.639 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.639 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.639 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.639 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.639 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3012 2021-03-15T22:27:54.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.640 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:27:54.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.641 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.641 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:27:54.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.61e-06 2021-03-15T22:27:54.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093118+0000", 2021-03-15T22:27:54.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.234e-06 2021-03-15T22:27:54.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.643 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093119+0000", 2021-03-15T22:27:54.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7290000000000002e-06 2021-03-15T22:27:54.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.645 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093123+0000", 2021-03-15T22:27:54.645 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1649999999999999e-06 2021-03-15T22:27:54.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.646 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093127+0000", 2021-03-15T22:27:54.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6592e-05 2021-03-15T22:27:54.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093213+0000", 2021-03-15T22:27:54.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2364000000000001e-05 2021-03-15T22:27:54.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:54.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093226+0000", 2021-03-15T22:27:54.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99268559899999997 2021-03-15T22:27:54.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085911+0000", 2021-03-15T22:27:54.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1440000000000002e-06 2021-03-15T22:27:54.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:54.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085916+0000", 2021-03-15T22:27:54.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0164949109999999 2021-03-15T22:27:54.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102411+0000", 2021-03-15T22:27:54.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000116049 2021-03-15T22:27:54.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:54.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102527+0000", 2021-03-15T22:27:54.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042432780000000003 2021-03-15T22:27:54.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.144960+0000", 2021-03-15T22:27:54.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7323000000000003e-05 2021-03-15T22:27:54.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145057+0000", 2021-03-15T22:27:54.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018096599999999999 2021-03-15T22:27:54.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:27:54.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145238+0000", 2021-03-15T22:27:54.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077369500000000003 2021-03-15T22:27:54.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.146012+0000", 2021-03-15T22:27:54.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041960274999999998 2021-03-15T22:27:54.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:54.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.187972+0000", 2021-03-15T22:27:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021867610000000002 2021-03-15T22:27:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190159+0000", 2021-03-15T22:27:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9751000000000004e-05 2021-03-15T22:27:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190229+0000", 2021-03-15T22:27:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000180135 2021-03-15T22:27:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190409+0000", 2021-03-15T22:27:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0972930829999998 2021-03-15T22:27:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.661 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3013 7.b 7:d980be1f:::smithi07694878-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e328)", 2021-03-15T22:27:54.662 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:27:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 231.40429457299999, 2021-03-15T22:27:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.097631684, 2021-03-15T22:27:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:54.663 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3013 2021-03-15T22:27:54.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:27:54.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:27:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9600000000000008e-07 2021-03-15T22:27:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093150+0000", 2021-03-15T22:27:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.514e-06 2021-03-15T22:27:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093152+0000", 2021-03-15T22:27:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2985e-05 2021-03-15T22:27:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093165+0000", 2021-03-15T22:27:54.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4000000000000002e-06 2021-03-15T22:27:54.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093169+0000", 2021-03-15T22:27:54.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2490999999999993e-05 2021-03-15T22:27:54.671 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093252+0000", 2021-03-15T22:27:54.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7490000000000001e-06 2021-03-15T22:27:54.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:54.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093258+0000", 2021-03-15T22:27:54.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99267630200000001 2021-03-15T22:27:54.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085935+0000", 2021-03-15T22:27:54.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0580000000000003e-06 2021-03-15T22:27:54.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:27:54.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085941+0000", 2021-03-15T22:27:54.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.01663009 2021-03-15T22:27:54.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102571+0000", 2021-03-15T22:27:54.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012747 2021-03-15T22:27:54.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:54.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102698+0000", 2021-03-15T22:27:54.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042823333999999998 2021-03-15T22:27:54.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145522+0000", 2021-03-15T22:27:54.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7650000000000004e-05 2021-03-15T22:27:54.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:54.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145599+0000", 2021-03-15T22:27:54.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044843263000000001 2021-03-15T22:27:54.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190443+0000", 2021-03-15T22:27:54.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000117391 2021-03-15T22:27:54.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190560+0000", 2021-03-15T22:27:54.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000220991 2021-03-15T22:27:54.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190781+0000", 2021-03-15T22:27:54.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.097631684 2021-03-15T22:27:54.683 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.683 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.684 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.684 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3609 7.b 7:d10ba14d:::smithi07694878-43:head [copy-from ver 752] snapc 116=[116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e380)", 2021-03-15T22:27:54.684 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:55.463956+0000", 2021-03-15T22:27:54.685 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 179.03348781, 2021-03-15T22:27:54.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037021326, 2021-03-15T22:27:54.685 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.685 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.685 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.686 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.686 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.686 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3609 2021-03-15T22:27:54.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.686 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463956+0000", 2021-03-15T22:27:54.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463956+0000", 2021-03-15T22:27:54.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.636e-06 2021-03-15T22:27:54.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463960+0000", 2021-03-15T22:27:54.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7860000000000001e-06 2021-03-15T22:27:54.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463962+0000", 2021-03-15T22:27:54.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2736e-05 2021-03-15T22:27:54.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463975+0000", 2021-03-15T22:27:54.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.852e-06 2021-03-15T22:27:54.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463981+0000", 2021-03-15T22:27:54.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0056000000000002e-05 2021-03-15T22:27:54.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.464041+0000", 2021-03-15T22:27:54.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102267 2021-03-15T22:27:54.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.464143+0000", 2021-03-15T22:27:54.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.027411048 2021-03-15T22:27:54.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:27:54.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.491554+0000", 2021-03-15T22:27:54.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072668170000000001 2021-03-15T22:27:54.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.498821+0000", 2021-03-15T22:27:54.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002009434 2021-03-15T22:27:54.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.500831+0000", 2021-03-15T22:27:54.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9524000000000003e-05 2021-03-15T22:27:54.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:54.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.500870+0000", 2021-03-15T22:27:54.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9422000000000002e-05 2021-03-15T22:27:54.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.500920+0000", 2021-03-15T22:27:54.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7747999999999997e-05 2021-03-15T22:27:54.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.500977+0000", 2021-03-15T22:27:54.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037021326 2021-03-15T22:27:54.702 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.702 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.702 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.703 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3610 7.b 7:d10ba14d:::smithi07694878-43:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e380)", 2021-03-15T22:27:54.703 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:55.463991+0000", 2021-03-15T22:27:54.703 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 179.033452719, 2021-03-15T22:27:54.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037113361999999997, 2021-03-15T22:27:54.703 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.704 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:54.704 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.704 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.704 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.704 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3610 2021-03-15T22:27:54.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.705 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463991+0000", 2021-03-15T22:27:54.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463991+0000", 2021-03-15T22:27:54.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3600000000000002e-07 2021-03-15T22:27:54.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463992+0000", 2021-03-15T22:27:54.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.034e-06 2021-03-15T22:27:54.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463993+0000", 2021-03-15T22:27:54.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1729999999999997e-06 2021-03-15T22:27:54.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.463997+0000", 2021-03-15T22:27:54.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7920000000000001e-06 2021-03-15T22:27:54.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.464000+0000", 2021-03-15T22:27:54.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030387099999999999 2021-03-15T22:27:54.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.464304+0000", 2021-03-15T22:27:54.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4651999999999998e-05 2021-03-15T22:27:54.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:27:54.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.464358+0000", 2021-03-15T22:27:54.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.032976501999999998 2021-03-15T22:27:54.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.497335+0000", 2021-03-15T22:27:54.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0748000000000006e-05 2021-03-15T22:27:54.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:54.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.497426+0000", 2021-03-15T22:27:54.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035780479999999999 2021-03-15T22:27:54.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.501004+0000", 2021-03-15T22:27:54.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3093999999999998e-05 2021-03-15T22:27:54.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.501047+0000", 2021-03-15T22:27:54.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7612000000000003e-05 2021-03-15T22:27:54.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:55.501104+0000", 2021-03-15T22:27:54.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037113361999999997 2021-03-15T22:27:54.719 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.719 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.720 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.720 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:3668 7.4 e385/358 7:26337d05:::smithi07694878-20:head v 385'309, mlcod=377'306)", 2021-03-15T22:27:54.720 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:27:54.721 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 174.31905613699999, 2021-03-15T22:27:54.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054245017999999999, 2021-03-15T22:27:54.721 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.721 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.721 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:27:54.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:27:54.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00068176800000000002 2021-03-15T22:27:54.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179069+0000", 2021-03-15T22:27:54.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1430000000000001e-06 2021-03-15T22:27:54.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179077+0000", 2021-03-15T22:27:54.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9515000000000001e-05 2021-03-15T22:27:54.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179106+0000", 2021-03-15T22:27:54.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0193e-05 2021-03-15T22:27:54.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179116+0000", 2021-03-15T22:27:54.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8248000000000003e-05 2021-03-15T22:27:54.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179165+0000", 2021-03-15T22:27:54.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8050000000000001e-05 2021-03-15T22:27:54.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179193+0000", 2021-03-15T22:27:54.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053391015999999999 2021-03-15T22:27:54.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.232584+0000", 2021-03-15T22:27:54.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9085000000000002e-05 2021-03-15T22:27:54.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.232633+0000", 2021-03-15T22:27:54.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054245017999999999 2021-03-15T22:27:54.732 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.732 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.733 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.733 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4284 7.b 7:d4368cae:::smithi07694878-18:head [write 3203532~509848] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:27:54.733 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:27:54.734 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 114.334665603, 2021-03-15T22:27:54.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068311582999999995, 2021-03-15T22:27:54.734 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.734 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.734 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.735 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.735 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.735 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4284 2021-03-15T22:27:54.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.736 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:27:54.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:27:54.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015451700000000001 2021-03-15T22:27:54.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162933+0000", 2021-03-15T22:27:54.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.25e-06 2021-03-15T22:27:54.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162937+0000", 2021-03-15T22:27:54.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2824e-05 2021-03-15T22:27:54.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162960+0000", 2021-03-15T22:27:54.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.072e-06 2021-03-15T22:27:54.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162968+0000", 2021-03-15T22:27:54.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7661999999999999e-05 2021-03-15T22:27:54.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163016+0000", 2021-03-15T22:27:54.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108743 2021-03-15T22:27:54.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163124+0000", 2021-03-15T22:27:54.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021732500000000001 2021-03-15T22:27:54.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:27:54.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163342+0000", 2021-03-15T22:27:54.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043853354999999997 2021-03-15T22:27:54.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.207195+0000", 2021-03-15T22:27:54.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055661469999999996 2021-03-15T22:27:54.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.212761+0000", 2021-03-15T22:27:54.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018240684 2021-03-15T22:27:54.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:54.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231002+0000", 2021-03-15T22:27:54.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6122000000000003e-05 2021-03-15T22:27:54.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231048+0000", 2021-03-15T22:27:54.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1882000000000003e-05 2021-03-15T22:27:54.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231090+0000", 2021-03-15T22:27:54.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068311582999999995 2021-03-15T22:27:54.751 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.751 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.751 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.752 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4285 7.b 7:d4368cae:::smithi07694878-18:head [setxattr _header (58) in=7b,truncate 3865709] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:27:54.752 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:27:54.752 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 114.334461131, 2021-03-15T22:27:54.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068193614999999999, 2021-03-15T22:27:54.753 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.753 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.753 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.753 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.753 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.754 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4285 2021-03-15T22:27:54.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.754 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:27:54.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:27:54.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.633e-06 2021-03-15T22:27:54.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162985+0000", 2021-03-15T22:27:54.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6789999999999999e-06 2021-03-15T22:27:54.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162987+0000", 2021-03-15T22:27:54.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9500000000000001e-06 2021-03-15T22:27:54.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162995+0000", 2021-03-15T22:27:54.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.459e-06 2021-03-15T22:27:54.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162999+0000", 2021-03-15T22:27:54.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012207240000000001 2021-03-15T22:27:54.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164220+0000", 2021-03-15T22:27:54.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0790000000000004e-05 2021-03-15T22:27:54.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164301+0000", 2021-03-15T22:27:54.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014765399999999999 2021-03-15T22:27:54.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:27:54.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164449+0000", 2021-03-15T22:27:54.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042818110999999999 2021-03-15T22:27:54.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.207267+0000", 2021-03-15T22:27:54.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055270550000000003 2021-03-15T22:27:54.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.212794+0000", 2021-03-15T22:27:54.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018305267 2021-03-15T22:27:54.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:54.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231099+0000", 2021-03-15T22:27:54.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4928000000000002e-05 2021-03-15T22:27:54.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231134+0000", 2021-03-15T22:27:54.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2364999999999998e-05 2021-03-15T22:27:54.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231176+0000", 2021-03-15T22:27:54.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068193614999999999 2021-03-15T22:27:54.769 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.769 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.769 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.770 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4286 7.b 7:d4368cae:::smithi07694878-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e445)", 2021-03-15T22:27:54.770 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:27:54.770 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 114.334439483, 2021-03-15T22:27:54.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068364477000000007, 2021-03-15T22:27:54.771 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.771 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:27:54.771 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.771 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.772 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.772 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4286 2021-03-15T22:27:54.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.772 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:27:54.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:27:54.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6500000000000008e-07 2021-03-15T22:27:54.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163005+0000", 2021-03-15T22:27:54.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.218e-06 2021-03-15T22:27:54.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163007+0000", 2021-03-15T22:27:54.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3890000000000002e-06 2021-03-15T22:27:54.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163013+0000", 2021-03-15T22:27:54.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.053e-06 2021-03-15T22:27:54.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163017+0000", 2021-03-15T22:27:54.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017994630000000001 2021-03-15T22:27:54.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164816+0000", 2021-03-15T22:27:54.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0958000000000007e-05 2021-03-15T22:27:54.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:27:54.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164897+0000", 2021-03-15T22:27:54.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066313677000000001 2021-03-15T22:27:54.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231211+0000", 2021-03-15T22:27:54.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9739000000000004e-05 2021-03-15T22:27:54.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231281+0000", 2021-03-15T22:27:54.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8015000000000003e-05 2021-03-15T22:27:54.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231369+0000", 2021-03-15T22:27:54.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068364477000000007 2021-03-15T22:27:54.784 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.784 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.784 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.785 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4287 7.7 e445/420 7:ed4a01d2:::smithi07694878-2:head v 445'413, mlcod=444'410)", 2021-03-15T22:27:54.785 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:27:54.785 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 114.333421187, 2021-03-15T22:27:54.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066973671999999998, 2021-03-15T22:27:54.785 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.786 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.786 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:27:54.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:27:54.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.258e-06 2021-03-15T22:27:54.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164026+0000", 2021-03-15T22:27:54.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7980000000000001e-06 2021-03-15T22:27:54.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164029+0000", 2021-03-15T22:27:54.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0787e-05 2021-03-15T22:27:54.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164039+0000", 2021-03-15T22:27:54.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2829999999999998e-06 2021-03-15T22:27:54.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164046+0000", 2021-03-15T22:27:54.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5735000000000001e-05 2021-03-15T22:27:54.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164101+0000", 2021-03-15T22:27:54.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9206000000000001e-05 2021-03-15T22:27:54.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164131+0000", 2021-03-15T22:27:54.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066830296999999997 2021-03-15T22:27:54.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.230961+0000", 2021-03-15T22:27:54.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5308000000000001e-05 2021-03-15T22:27:54.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.230996+0000", 2021-03-15T22:27:54.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066973671999999998 2021-03-15T22:27:54.797 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.797 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.798 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4288 7.7 e445/420 7:ed4a01d2:::smithi07694878-2:head v 445'414, mlcod=444'410)", 2021-03-15T22:27:54.799 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.164500+0000", 2021-03-15T22:27:54.799 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 114.332943693, 2021-03-15T22:27:54.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066520592000000003, 2021-03-15T22:27:54.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.799 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.800 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164500+0000", 2021-03-15T22:27:54.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164500+0000", 2021-03-15T22:27:54.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9610000000000001e-06 2021-03-15T22:27:54.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164503+0000", 2021-03-15T22:27:54.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.407e-06 2021-03-15T22:27:54.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164505+0000", 2021-03-15T22:27:54.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8310000000000001e-06 2021-03-15T22:27:54.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164513+0000", 2021-03-15T22:27:54.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9269999999999998e-06 2021-03-15T22:27:54.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164517+0000", 2021-03-15T22:27:54.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6690999999999998e-05 2021-03-15T22:27:54.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164554+0000", 2021-03-15T22:27:54.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6207000000000002e-05 2021-03-15T22:27:54.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164580+0000", 2021-03-15T22:27:54.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066419645999999999 2021-03-15T22:27:54.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231000+0000", 2021-03-15T22:27:54.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0922000000000001e-05 2021-03-15T22:27:54.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231021+0000", 2021-03-15T22:27:54.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066520592000000003 2021-03-15T22:27:54.810 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.811 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.811 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.811 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4384 7.b 7:d980be1f:::smithi07694878-23:head [append 0~0] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:27:54.812 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:10.205239+0000", 2021-03-15T22:27:54.812 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 104.29220482700001, 2021-03-15T22:27:54.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044851078000000003, 2021-03-15T22:27:54.812 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.812 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.813 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.813 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.813 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.813 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4384 2021-03-15T22:27:54.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.814 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205239+0000", 2021-03-15T22:27:54.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205239+0000", 2021-03-15T22:27:54.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6599999999999999e-06 2021-03-15T22:27:54.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205242+0000", 2021-03-15T22:27:54.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4120000000000001e-06 2021-03-15T22:27:54.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205245+0000", 2021-03-15T22:27:54.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2353e-05 2021-03-15T22:27:54.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205257+0000", 2021-03-15T22:27:54.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7439999999999996e-06 2021-03-15T22:27:54.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205266+0000", 2021-03-15T22:27:54.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00035602600000000001 2021-03-15T22:27:54.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205622+0000", 2021-03-15T22:27:54.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000115955 2021-03-15T22:27:54.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205738+0000", 2021-03-15T22:27:54.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024261500000000001 2021-03-15T22:27:54.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:27:54.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205981+0000", 2021-03-15T22:27:54.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001692928 2021-03-15T22:27:54.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207674+0000", 2021-03-15T22:27:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9629e-05 2021-03-15T22:27:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207703+0000", 2021-03-15T22:27:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042243636000000001 2021-03-15T22:27:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.249947+0000", 2021-03-15T22:27:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8107000000000003e-05 2021-03-15T22:27:54.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250025+0000", 2021-03-15T22:27:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5012999999999996e-05 2021-03-15T22:27:54.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250090+0000", 2021-03-15T22:27:54.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044851078000000003 2021-03-15T22:27:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4385 7.b 7:d980be1f:::smithi07694878-23:head [rollback 128] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:27:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:10.205276+0000", 2021-03-15T22:27:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 104.292167851, 2021-03-15T22:27:54.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044905154000000003, 2021-03-15T22:27:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.831 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4385 2021-03-15T22:27:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205276+0000", 2021-03-15T22:27:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205276+0000", 2021-03-15T22:27:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3400000000000001e-06 2021-03-15T22:27:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205277+0000", 2021-03-15T22:27:54.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2640000000000001e-06 2021-03-15T22:27:54.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205279+0000", 2021-03-15T22:27:54.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0669999999999997e-06 2021-03-15T22:27:54.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205285+0000", 2021-03-15T22:27:54.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1370000000000001e-06 2021-03-15T22:27:54.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205290+0000", 2021-03-15T22:27:54.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013506429999999999 2021-03-15T22:27:54.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.206640+0000", 2021-03-15T22:27:54.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1700999999999998e-05 2021-03-15T22:27:54.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.206702+0000", 2021-03-15T22:27:54.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015400200000000001 2021-03-15T22:27:54.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:27:54.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.206856+0000", 2021-03-15T22:27:54.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010626349999999999 2021-03-15T22:27:54.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207919+0000", 2021-03-15T22:27:54.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0203000000000004e-05 2021-03-15T22:27:54.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207999+0000", 2021-03-15T22:27:54.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042099725999999997 2021-03-15T22:27:54.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:54.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250099+0000", 2021-03-15T22:27:54.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3871000000000001e-05 2021-03-15T22:27:54.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250133+0000", 2021-03-15T22:27:54.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8565e-05 2021-03-15T22:27:54.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250181+0000", 2021-03-15T22:27:54.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044905154000000003 2021-03-15T22:27:54.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.847 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.848 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4386 7.b 7:d980be1f:::smithi07694878-23:head [append 0~0] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:27:54.848 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:10.205294+0000", 2021-03-15T22:27:54.848 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 104.292150177, 2021-03-15T22:27:54.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044975109999999999, 2021-03-15T22:27:54.849 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:27:54.849 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:27:54.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:27:54.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:27:54.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:27:54.850 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4386 2021-03-15T22:27:54.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.850 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:27:54.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:27:54.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205294+0000", 2021-03-15T22:27:54.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:27:54.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:27:54.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205294+0000", 2021-03-15T22:27:54.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0389999999999999e-06 2021-03-15T22:27:54.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:27:54.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205295+0000", 2021-03-15T22:27:54.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.145e-06 2021-03-15T22:27:54.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:27:54.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205296+0000", 2021-03-15T22:27:54.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6250000000000004e-06 2021-03-15T22:27:54.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:27:54.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205301+0000", 2021-03-15T22:27:54.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9639999999999997e-06 2021-03-15T22:27:54.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:27:54.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205305+0000", 2021-03-15T22:27:54.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019453750000000001 2021-03-15T22:27:54.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:27:54.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207251+0000", 2021-03-15T22:27:54.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8658000000000006e-05 2021-03-15T22:27:54.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:27:54.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207319+0000", 2021-03-15T22:27:54.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011772700000000001 2021-03-15T22:27:54.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:27:54.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207437+0000", 2021-03-15T22:27:54.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062831400000000004 2021-03-15T22:27:54.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.208065+0000", 2021-03-15T22:27:54.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2487000000000003e-05 2021-03-15T22:27:54.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:27:54.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.208098+0000", 2021-03-15T22:27:54.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042089556 2021-03-15T22:27:54.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:27:54.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250188+0000", 2021-03-15T22:27:54.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4434999999999998e-05 2021-03-15T22:27:54.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:27:54.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250222+0000", 2021-03-15T22:27:54.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6785e-05 2021-03-15T22:27:54.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:27:54.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:27:54.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:27:54.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250269+0000", 2021-03-15T22:27:54.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044975109999999999 2021-03-15T22:27:54.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.866 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:27:54.866 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:27:54.866 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:27:54.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:27:54.868 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:54.870 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:54 smithi125 conmon[56739]: cluster 2021-03-15T22:27:53.094068+0000 mgr.y (mgr.14144 2021-03-15T22:27:54.870 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:54 smithi125 conmon[56739]: ) 735 : cluster [DBG] pgmap v950: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:54.870 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:54 smithi125 conmon[56739]: cluster 2021-03-15T22:27:53.369507+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:54.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:54 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:27:54.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:54 smithi125 conmon[56739]: 22:27:53.370475+0000 osd.7 (osd 2021-03-15T22:27:54.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:54 smithi125 conmon[56739]: .7) 110 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:54.872 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:54 smithi076 conmon[56514]: cluster 2021-03-15T22:27:53. 2021-03-15T22:27:54.872 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:54 smithi076 conmon[56514]: 094068+0000 mgr.y (mgr.14144) 735 : cluster [DBG] pgmap v950: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:54.872 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:54 smithi076 conmon[56514]: cluster 2021-03-15T22:27:53.369507+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:54.873 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:54 smithi076 conmon[56514]: cluster 2021-03-15T22:27:53.370475+0000 osd.7 (osd.7) 110 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:54.873 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:54 smithi076 conmon[62156]: cluster 2021-03-15T22:27:53.094068 2021-03-15T22:27:54.873 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:54 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 735 : cluster [DBG] pgmap v950: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 639 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:54.874 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:54 smithi076 conmon[62156]: cluster 2021-03-15T22:27:53.369507+0000 osd.7 (osd.7) 109 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:54.874 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:54 smithi076 conmon[62156]: cluster 2021-03-15T22:27:53.370475+0000 osd.7 (osd.7) 110 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:54.892 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:54.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:54.894 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:54.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:55.019 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:55.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:55.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:55.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:55.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:55.147 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:55.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:55.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:55.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:55.278 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:55.278 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:55.328 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-15T22:27:55.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:27:55.380 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:55.401 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:55.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:55.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:55.504 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:55.525 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:55.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:55.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:55.628 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:55.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:55.652 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:55.652 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:55.753 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:55.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:55 smithi076 conmon[56514]: cluster 2021-03-15T22:27:54.352290+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e545: 8 total, 6 up, 7 in 2021-03-15T22:27:55.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:55 smithi076 conmon[62156]: cluster 2021-03-15T22:27:54.352290+0000 mon.a (mon.0) 1536 : cluster [DBG] osdmap e545: 8 total, 6 up, 7 in 2021-03-15T22:27:55.779 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:55.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:55.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:55.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:55 smithi125 conmon[56739]: cluster 2021-03-15T22:27:54.352290+0000 mon.a (mon.0 2021-03-15T22:27:55.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:55 smithi125 conmon[56739]: ) 1536 : cluster [DBG] osdmap e545: 8 total, 6 up, 7 in 2021-03-15T22:27:55.802 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:55.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -n osd.5 --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal --op list-pgs 2021-03-15T22:27:55.837 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:55.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:55.906 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:55.908 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:55.944 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:27:56.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:56.033 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/osd.5/config 2021-03-15T22:27:56.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:56.035 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:56.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:56.136 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:56.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:56.160 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:56.161 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:56.262 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:56.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:56.288 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:56.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:56.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:56.414 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:56.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:56.516 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:56.517 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:56 smithi125 conmon[56739]: cluster 2021-03-15T22:27:55.094598+0000 mgr.y (mgr.14144) 736 : cluster [DBG] pgmap v952: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:56.517 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:56 smithi125 conmon[56739]: cluster 2021-03-15T22:27:55.678428+0000 osd.0 (osd.0) 81 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:56.518 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:56 smithi125 conmon[56739]: cluster 2021-03-15T22:27:55.678906+0000 osd.0 (osd.0) 82 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:56.548 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:56.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:56.651 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:56.675 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:56.675 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:56.725 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:56 smithi076 conmon[56514]: cluster 2021-03-15T22:27:55.094598+0000 mgr.y (mgr.14144) 736 : cluster [DBG] pgmap v952: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:56.726 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:56 smithi076 conmon[56514]: cluster 2021-03-15T22:27:55.678428+0000 osd.0 (osd.0) 81 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:56.726 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:56 smithi076 conmon[56514]: cluster 2021-03-15T22:27:55.678906+0000 osd.0 (osd.0) 82 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:56.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:56 smithi076 conmon[62156]: cluster 2021-03-15T22:27:55.094598+0000 mgr.y (mgr.14144) 736 : cluster [DBG] pgmap v952: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 1 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:56.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:56 smithi076 conmon[62156]: cluster 2021-03-15T22:27:55.678428+0000 osd.0 (osd.0) 81 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:27:56.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:56 smithi076 conmon[62156]: cluster 2021-03-15T22:27:55.678906+0000 osd.0 (osd.0) 82 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:27:56.777 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:56.799 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:56.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:56.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:56.902 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:56.926 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:56.926 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:57.028 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:57.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:57.051 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:57.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:57.152 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:57.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:57.177 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:57.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:57.279 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:57.301 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:57.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:57.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:57.404 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:57.427 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:57.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:57.428 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:57.498 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:57 smithi125 conmon[56739]: cluster 2021-03-15T22:27:56.373165+0000 mon.a (mon.0) 1537 2021-03-15T22:27:57.498 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:57 smithi125 conmon[56739]: : cluster [DBG] osdmap e546: 8 total, 6 up, 7 in 2021-03-15T22:27:57.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:57 smithi076 conmon[56514]: cluster 2021-03-15T22:27:56.373165+0000 mon.a (mon. 2021-03-15T22:27:57.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:57 smithi076 conmon[56514]: 0) 1537 : cluster [DBG] osdmap e546: 8 total, 6 up, 7 in 2021-03-15T22:27:57.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:57 smithi076 conmon[62156]: cluster 2021-03-15T22:27:56.373165+0000 mon.a 2021-03-15T22:27:57.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:57 smithi076 conmon[62156]: (mon.0) 1537 : cluster [DBG] osdmap e546: 8 total, 6 up, 7 in 2021-03-15T22:27:57.530 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:57.552 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:57.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:57.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:57.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:57.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:57.679 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:57.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:57.781 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:57.805 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:57.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:57.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:57.908 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:57.930 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:57.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:57.932 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:58.033 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:58.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:58.056 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:58.057 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:58.057 INFO:teuthology.orchestra.run.smithi125.stdout:7.e 2021-03-15T22:27:58.058 INFO:teuthology.orchestra.run.smithi125.stdout:7.b 2021-03-15T22:27:58.058 INFO:teuthology.orchestra.run.smithi125.stdout:6.d 2021-03-15T22:27:58.058 INFO:teuthology.orchestra.run.smithi125.stdout:6.c 2021-03-15T22:27:58.058 INFO:teuthology.orchestra.run.smithi125.stdout:6.4 2021-03-15T22:27:58.059 INFO:teuthology.orchestra.run.smithi125.stdout:6.1e 2021-03-15T22:27:58.059 INFO:teuthology.orchestra.run.smithi125.stdout:6.1c 2021-03-15T22:27:58.059 INFO:teuthology.orchestra.run.smithi125.stdout:6.1a 2021-03-15T22:27:58.059 INFO:teuthology.orchestra.run.smithi125.stdout:6.7 2021-03-15T22:27:58.060 INFO:teuthology.orchestra.run.smithi125.stdout:6.11 2021-03-15T22:27:58.060 INFO:teuthology.orchestra.run.smithi125.stdout:6.17 2021-03-15T22:27:58.060 INFO:teuthology.orchestra.run.smithi125.stdout:6.10 2021-03-15T22:27:58.060 INFO:teuthology.orchestra.run.smithi125.stdout:5.f 2021-03-15T22:27:58.060 INFO:teuthology.orchestra.run.smithi125.stdout:6.19 2021-03-15T22:27:58.061 INFO:teuthology.orchestra.run.smithi125.stdout:5.4 2021-03-15T22:27:58.061 INFO:teuthology.orchestra.run.smithi125.stdout:3.2 2021-03-15T22:27:58.061 INFO:teuthology.orchestra.run.smithi125.stdout:3.d 2021-03-15T22:27:58.061 INFO:teuthology.orchestra.run.smithi125.stdout:6.1f 2021-03-15T22:27:58.062 INFO:teuthology.orchestra.run.smithi125.stdout:3.4 2021-03-15T22:27:58.062 INFO:teuthology.orchestra.run.smithi125.stdout:3.1f 2021-03-15T22:27:58.062 INFO:teuthology.orchestra.run.smithi125.stdout:7.6 2021-03-15T22:27:58.062 INFO:teuthology.orchestra.run.smithi125.stdout:5.7 2021-03-15T22:27:58.063 INFO:teuthology.orchestra.run.smithi125.stdout:3.5 2021-03-15T22:27:58.063 INFO:teuthology.orchestra.run.smithi125.stdout:3.10 2021-03-15T22:27:58.063 INFO:teuthology.orchestra.run.smithi125.stdout:3.1d 2021-03-15T22:27:58.063 INFO:teuthology.orchestra.run.smithi125.stdout:3.17 2021-03-15T22:27:58.063 INFO:teuthology.orchestra.run.smithi125.stdout:2.2 2021-03-15T22:27:58.064 INFO:teuthology.orchestra.run.smithi125.stdout:3.16 2021-03-15T22:27:58.064 INFO:teuthology.orchestra.run.smithi125.stdout:7.a 2021-03-15T22:27:58.064 INFO:teuthology.orchestra.run.smithi125.stdout:4.15 2021-03-15T22:27:58.064 INFO:teuthology.orchestra.run.smithi125.stdout:3.7 2021-03-15T22:27:58.064 INFO:teuthology.orchestra.run.smithi125.stdout:3.1c 2021-03-15T22:27:58.065 INFO:teuthology.orchestra.run.smithi125.stdout:3.c 2021-03-15T22:27:58.065 INFO:teuthology.orchestra.run.smithi125.stdout:4.8 2021-03-15T22:27:58.065 INFO:teuthology.orchestra.run.smithi125.stdout:4.1f 2021-03-15T22:27:58.065 INFO:teuthology.orchestra.run.smithi125.stdout:6.1d 2021-03-15T22:27:58.066 INFO:teuthology.orchestra.run.smithi125.stdout:4.17 2021-03-15T22:27:58.066 INFO:teuthology.orchestra.run.smithi125.stdout:5.d 2021-03-15T22:27:58.066 INFO:teuthology.orchestra.run.smithi125.stdout:4.1 2021-03-15T22:27:58.066 INFO:teuthology.orchestra.run.smithi125.stdout:4.1b 2021-03-15T22:27:58.066 INFO:teuthology.orchestra.run.smithi125.stdout:6.a 2021-03-15T22:27:58.067 INFO:teuthology.orchestra.run.smithi125.stdout:5.2 2021-03-15T22:27:58.067 INFO:teuthology.orchestra.run.smithi125.stdout:4.3 2021-03-15T22:27:58.067 INFO:teuthology.orchestra.run.smithi125.stdout:7.0 2021-03-15T22:27:58.067 INFO:teuthology.orchestra.run.smithi125.stdout:4.7 2021-03-15T22:27:58.067 INFO:teuthology.orchestra.run.smithi125.stdout:3.1 2021-03-15T22:27:58.068 INFO:teuthology.orchestra.run.smithi125.stdout:4.19 2021-03-15T22:27:58.068 INFO:teuthology.orchestra.run.smithi125.stdout:5.8 2021-03-15T22:27:58.068 INFO:teuthology.orchestra.run.smithi125.stdout:5.12 2021-03-15T22:27:58.068 INFO:teuthology.orchestra.run.smithi125.stdout:6.0 2021-03-15T22:27:58.069 INFO:teuthology.orchestra.run.smithi125.stdout:5.13 2021-03-15T22:27:58.069 INFO:teuthology.orchestra.run.smithi125.stdout:5.15 2021-03-15T22:27:58.069 INFO:teuthology.orchestra.run.smithi125.stdout:5.3 2021-03-15T22:27:58.069 INFO:teuthology.orchestra.run.smithi125.stdout:4.2 2021-03-15T22:27:58.069 INFO:teuthology.orchestra.run.smithi125.stdout:5.6 2021-03-15T22:27:58.070 INFO:teuthology.orchestra.run.smithi125.stdout:5.b 2021-03-15T22:27:58.070 INFO:teuthology.orchestra.run.smithi125.stdout:5.e 2021-03-15T22:27:58.158 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:58.180 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:58.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:58.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:58.282 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:58.304 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:58.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:58.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:58.407 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:58.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:58.432 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:58.432 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:58.534 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:58.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:58.558 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:58.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:58.613 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:58 smithi076 conmon[56514]: cluster 2021-03-15T22:27:57.095299+0000 mgr.y (mgr.14144) 737 2021-03-15T22:27:58.613 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:58 smithi076 conmon[56514]: : cluster [DBG] pgmap v954: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:58.614 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:58 smithi076 conmon[62156]: cluster 2021-03-15T22:27:57.095299+0000 2021-03-15T22:27:58.614 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:58 smithi076 conmon[62156]: mgr.y (mgr.14144) 737 : cluster [DBG] pgmap v954: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:58.634 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:58 smithi125 conmon[56739]: cluster 2021-03-15T22:27:57.095299+0000 mgr.y (mgr.14144) 737 : cluster [DBG] pgmap v954: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 853 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:27:58.659 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:27:58.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:58.683 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:58.684 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:27:58.695 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:27:58.715 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:27:58.733 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:27:58.751 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:27:58.769 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:27:58.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:58.791 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:27:58.793 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:27:58.793 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:27:58.793 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:27:58.794 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:27:58.794 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:27:58.795 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:27:58.795 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:27:58.796 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:27:58.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:58.810 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:58.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:58.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:58.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:58.936 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:58.936 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:59.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:59.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:59.060 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:59.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:27:59.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:59.184 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:59.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:59.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:59.242 INFO:tasks.thrashosds.thrasher:Moving pg 4.1a from osd.2 to osd.5 2021-03-15T22:27:59.243 INFO:tasks.thrashosds.thrasher:Transfer export file from ubuntu@smithi076.front.sepia.ceph.com to ubuntu@smithi125.front.sepia.ceph.com 2021-03-15T22:27:59.243 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /var/log/ceph 2021-03-15T22:27:59.272 DEBUG:teuthology.orchestra.run.smithi125:> sudo chmod 777 /var/log/ceph 2021-03-15T22:27:59.287 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:27:59.298 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2021-03-15T22:27:59.327 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.ZMGpi2bN57 2021-03-15T22:27:59.327 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c/exp.4.1a.2 /tmp/tmp.ZMGpi2bN57 2021-03-15T22:27:59.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:59.335 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:59.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:27:59.342 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:27:59.342 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:27:59.437 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:59.460 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:59.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:59.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:59.481 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:27:59.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:59 smithi076 conmon[56514]: audit 2021-03-15T22:27:58.673990+0000 mon.a (mon.0) 1538 : audit [DBG] from='client.? 172.21.15.76:0/1410198839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:59.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:27:59 smithi076 conmon[56514]: audit 2021-03-15T22:27:59.199933+0000 mon.a (mon.0) 1539 : audit [INF] from='client.? 172.21.15.76:0/3295198140' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:59.483 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:59 smithi076 conmon[62156]: audit 2021-03-15T22:27:58.673990+0000 mon.a (mon.0) 1538 : audit [DBG] from='client.? 172.21.15.76:0/1410198839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:59.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:27:59 smithi076 conmon[62156]: audit 2021-03-15T22:27:59.199933+0000 mon.a (mon.0) 1539 : audit [INF] from='client.? 172.21.15.76:0/3295198140' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:59.485 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.ZMGpi2bN57 2021-03-15T22:27:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:59 smithi125 conmon[56739]: audit 2021-03-15T22:27:58.673990 2021-03-15T22:27:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:59 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1538 : audit [DBG] from='client.? 172.21.15.76:0/1410198839' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:27:59.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:27:59 smithi125 conmon[56739]: audit 2021-03-15T22:27:59.199933+0000 mon.a (mon.0) 1539 : audit [INF] from='client.? 172.21.15.76:0/3295198140' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:27:59.563 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:27:59.573 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.ZMGpi2bN57 is 19KB 2021-03-15T22:27:59.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:59.593 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:59.594 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:27:59.622 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.ZMGpi2bN57 2021-03-15T22:27:59.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo mv /var/log/ceph/exp.4.1a.2 /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c/exp.4.1a.2 2021-03-15T22:27:59.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell -n osd.5 --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-5 --journal-path /var/lib/ceph/osd/ceph-5/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.1a.2 2021-03-15T22:27:59.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:27:59.740 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:59.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:59.742 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:27:59.843 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:27:59.866 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:27:59.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:59.867 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:27:59.929 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/osd.5/config 2021-03-15T22:27:59.969 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:27:59.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:27:59.993 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:27:59.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:00.036 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:27:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:28:00.095 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:00.118 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:00.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:00.219 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:00.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:00.243 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:00.243 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:00.344 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:00.367 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:00.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:00.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:00.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:00.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:00.494 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:00.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:00.568 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:28:00.569 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:28:00.596 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:00.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:00.621 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:00.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:00.723 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:00.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:00.747 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:00.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:00.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[56514]: cluster 2021-03-15T22:27:59.095982+0000 mgr.y (mgr.14144) 738 : cluster 2021-03-15T22:28:00.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[56514]: [DBG] pgmap v955: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 660 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:00.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[56514]: cluster 2021-03-15T22:27:59.378115+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:00.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[56514]: audit 2021-03-15T22:27:59.382972+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? 172.21.15.76:0/3295198140' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:00.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[56514]: cluster 2021-03-15T22:27:59.383033+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e547: 8 total, 6 up, 7 in 2021-03-15T22:28:00.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[56514]: audit 2021-03-15T22:27:59.855305+0000 mgr.y (mgr.14144) 739 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: cluster 2021-03-15T22:27:59.095982+0000 mgr.y (mgr 2021-03-15T22:28:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: .14144) 738 : cluster [DBG] pgmap v955: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 660 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: cluster 2021-03-15T22:27: 2021-03-15T22:28:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: 59.378115+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:00.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: audit 2021-03-15T22:27: 2021-03-15T22:28:00.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: 59.382972+0000 mon.a (mon.0) 1541 : audit [INF] 2021-03-15T22:28:00.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3295198140' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:00.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: cluster 2021-03-15T22:28:00.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: 2021-03-15T22:27:59.383033+0000 mon.a (mon.0 2021-03-15T22:28:00.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: ) 1542 : cluster [DBG] osdmap e547: 8 total, 6 up, 7 in 2021-03-15T22:28:00.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:28:00.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: 22:27:59.855305+0000 mgr.y (mgr. 2021-03-15T22:28:00.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:00 smithi076 conmon[62156]: 14144) 739 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:00.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:00 smithi125 conmon[56739]: cluster 2021-03-15T22:27:59.095982+0000 mgr.y (mgr.14144) 738 : cluster [DBG] pgmap v955: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 660 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:00 smithi125 conmon[56739]: cluster 2021-03-15T22:27:59.378115+0000 mon.a (mon.0) 1540 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:00 smithi125 conmon[56739]: audit 2021-03-15T22:27:59.382972+0000 mon.a (mon.0) 1541 : audit [INF] from='client.? 172.21.15.76:0/3295198140' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:00 smithi125 conmon[56739]: cluster 2021-03-15T22:27:59.383033+0000 mon.a (mon.0) 1542 : cluster [DBG] osdmap e547: 8 total, 6 up, 7 in 2021-03-15T22:28:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:00 smithi125 conmon[56739]: audit 2021-03-15T22:27:59.855305+0000 mgr.y (mgr.14144) 739 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:00.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:00.871 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:00.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:00.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:00.911 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-15T22:28:00.974 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:01.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:01.001 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:01.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:01.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:01.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:01.127 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:01.128 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:01.226 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:01.228 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:01.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:01.253 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:01.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:01.355 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:01.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:01.379 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:01.380 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:01.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:01.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:01.505 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:01.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:01.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:01 smithi125 conmon[56739]: audit 2021-03-15T22:28:01.096157+0000 mon.a (mon.0) 1543 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:28:01.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:01 smithi125 conmon[56739]: audit 2021-03-15T22:28:01.099034+0000 mon.a (mon.0) 1544 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:28:01.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:01.630 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:01.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:01.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:01.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:01 smithi076 conmon[56514]: audit 2021-03-15T22:28:01.096157 2021-03-15T22:28:01.700 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:01 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1543 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:28:01.700 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:01 smithi076 conmon[56514]: audit 2021-03-15T22:28:01.099034+0000 mon.a (mon.0) 1544 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:28:01.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:01 smithi076 conmon[62156]: audit 2021-03-15T22:28:01.096157+0000 mon.a 2021-03-15T22:28:01.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:01 smithi076 conmon[62156]: (mon.0) 1543 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:28:01.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:01 smithi076 conmon[62156]: audit 2021-03-15T22:28: 2021-03-15T22:28:01.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:01 smithi076 conmon[62156]: 01.099034+0000 mon.a (mon.0) 1544 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:28:01.705 INFO:teuthology.orchestra.run.smithi125.stdout:get_pg_num_history pg_num_history pg_num_history(e539 pg_nums {1={21=1},2={53=3},3={56=32},4={58=32},5={60=32,146=31,150=30,154=29,158=28,162=27,166=26,170=25,174=24,178=23,182=22},6={62=32},7={91=16}} deleted_pools ) 2021-03-15T22:28:01.705 INFO:teuthology.orchestra.run.smithi125.stdout:Importing pgid 4.1a 2021-03-15T22:28:01.706 INFO:teuthology.orchestra.run.smithi125.stdout:Write #4:58680d93:::data_log.19:head# 2021-03-15T22:28:01.706 INFO:teuthology.orchestra.run.smithi125.stdout:snapset 0=[]:{} 2021-03-15T22:28:01.706 INFO:teuthology.orchestra.run.smithi125.stdout:Write #4:5909e671:gc::gc.27:head# 2021-03-15T22:28:01.707 INFO:teuthology.orchestra.run.smithi125.stdout:snapset 0=[]:{} 2021-03-15T22:28:01.707 INFO:teuthology.orchestra.run.smithi125.stdout:Write #4:5930a485:::data_log.15:head# 2021-03-15T22:28:01.707 INFO:teuthology.orchestra.run.smithi125.stdout:snapset 0=[]:{} 2021-03-15T22:28:01.708 INFO:teuthology.orchestra.run.smithi125.stdout:Write #4:59893ed7:::data_log.46:head# 2021-03-15T22:28:01.708 INFO:teuthology.orchestra.run.smithi125.stdout:snapset 0=[]:{} 2021-03-15T22:28:01.708 INFO:teuthology.orchestra.run.smithi125.stdout:Write #4:5a206276:::obj_delete_at_hint.0000000077:head# 2021-03-15T22:28:01.708 INFO:teuthology.orchestra.run.smithi125.stdout:snapset 0=[]:{} 2021-03-15T22:28:01.708 INFO:teuthology.orchestra.run.smithi125.stdout:Write #4:5ae4284b:::obj_delete_at_hint.0000000032:head# 2021-03-15T22:28:01.709 INFO:teuthology.orchestra.run.smithi125.stdout:snapset 0=[]:{} 2021-03-15T22:28:01.709 INFO:teuthology.orchestra.run.smithi125.stdout:Write #4:5c7e8a6d:reshard::reshard.0000000015:head# 2021-03-15T22:28:01.709 INFO:teuthology.orchestra.run.smithi125.stdout:snapset 0=[]:{} 2021-03-15T22:28:01.709 INFO:teuthology.orchestra.run.smithi125.stdout:Write #4:5c9fd2c7:::data_log.77:head# 2021-03-15T22:28:01.710 INFO:teuthology.orchestra.run.smithi125.stdout:snapset 0=[]:{} 2021-03-15T22:28:01.710 INFO:teuthology.orchestra.run.smithi125.stdout:write_pg epoch 479 info 4.1a( v 65'13 (0'0,65'13] local-lis/les=355/356 n=8 ec=58/58 lis/c=355/355 les/c/f=356/356/0 sis=355) 2021-03-15T22:28:01.711 INFO:teuthology.orchestra.run.smithi125.stdout:Import successful 2021-03-15T22:28:01.733 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:01.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:01.757 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:01.758 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:01.859 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:01.881 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:01.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:01.883 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:01.984 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:02.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:02.008 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:02.009 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:02.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:02.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:02.135 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:02.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:02.237 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:02.258 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:02.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:02.260 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:02.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:02.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:02.386 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:02.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:02.487 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:02.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:02.511 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:02.511 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:02.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:02 smithi125 conmon[56739]: cluster 2021-03-15T22:28:01.096579+0000 mgr.y (mgr.14144) 740 : cluster [DBG] pgmap v957: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 758 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:02 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:28:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:02 smithi125 conmon[56739]: -03-15T22:28:01.554215+0000 osd.4 (osd.4) 115 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:02 smithi125 conmon[56739]: cluster 2021-03-15T22:28:01.554962+0000 osd.4 (osd.4) 116 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:02.562 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:02 smithi076 conmon[62156]: cluster 2021-03-15T22:28:01.096579+0000 mgr.y (mgr.14144) 740 : cluster [DBG] pgmap v957: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 758 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:02.562 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:02 smithi076 conmon[62156]: cluster 2021-03-15T22:28:01.554215+0000 osd.4 (osd.4) 115 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:02.562 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:02 smithi076 conmon[62156]: cluster 2021-03-15T22:28:01.554962+0000 osd.4 (osd.4) 116 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:02.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:02 smithi076 conmon[56514]: cluster 2021-03-15T22:28:01.096579+0000 mgr.y (mgr.14144) 740 : cluster [DBG] pgmap v957: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 758 B/s rd, 0 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:02.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:02 smithi076 conmon[56514]: cluster 2021-03-15T22:28:01.554215+0000 osd.4 (osd.4) 115 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:02.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:02 smithi076 conmon[56514]: cluster 2021-03-15T22:28:01.554962+0000 osd.4 (osd.4) 116 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:02.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:28:02.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:02.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:02.635 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:02.635 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:02.737 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:02.755 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:28:02.755 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:28:02.758 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:02.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:02.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:02.861 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:02.884 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:02.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:02.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:02.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c/exp.4.1a.2 2021-03-15T22:28:02.951 DEBUG:teuthology.orchestra.run.smithi125:> sudo rm -f /var/log/ceph/8a572802-85db-11eb-9091-001a4aab830c/exp.4.1a.2 2021-03-15T22:28:02.962 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:28:02.987 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:03.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:03.040 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:03.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:03.142 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:03.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:03.166 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:03.167 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:03.268 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:03.290 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:03.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:03.292 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:03.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:03.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:03.417 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:03.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:03.452 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:03.477 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-15T22:28:03.490 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:03.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:03.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:03.541 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:03.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:03.643 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:03.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:03.665 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:03.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:03.767 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:03.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:03 smithi076 conmon[56514]: cluster 2021-03-15T22:28:02.453717+0000 mon.a (mon 2021-03-15T22:28:03.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:03 smithi076 conmon[56514]: .0) 1545 : cluster [DBG] osdmap e548: 8 total, 6 up, 7 in 2021-03-15T22:28:03.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:03 smithi076 conmon[62156]: cluster 2021-03-15T22:28:02.453717+0000 mon.a (mon. 2021-03-15T22:28:03.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:03 smithi076 conmon[62156]: 0) 1545 : cluster [DBG] osdmap e548: 8 total, 6 up, 7 in 2021-03-15T22:28:03.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:03 smithi125 conmon[56739]: cluster 2021-03-15T22:28:02.453717+0000 mon.a (mon.0) 1545 : cluster [DBG] osdmap e548: 8 total, 6 up, 7 in 2021-03-15T22:28:03.792 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:03.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:03.794 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:03.895 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:03.918 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:03.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:03.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:03.965 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:04.021 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:04.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:04.043 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:04.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:04.144 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:04.167 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:04.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:04.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:04.269 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:04.291 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:04.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:04.293 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:04.394 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:04.415 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:04.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:04.417 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:04.518 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:04.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:04.543 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:04.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:04.644 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:04.666 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:04.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:04.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:04.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:04 smithi076 conmon[56514]: cluster 2021-03-15T22:28:03.097280+0000 mgr.y (mgr.14144) 741 : cluster [DBG] pgmap v959: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:04.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:04 smithi076 conmon[62156]: cluster 2021-03-15T22:28:03.097280+0000 mgr.y (mgr.14144) 741 : cluster [DBG] pgmap v959: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:04.769 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:04.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:04 smithi125 conmon[56739]: cluster 2021-03-15T22:28:03.097280+0000 mgr.y (mgr.14144) 741 : cluster [DBG] pgmap v959: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 2.1 KiB/s rd, 0 B/s wr, 2 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:04.791 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:04.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:04.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:04.894 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:04.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:04.918 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:04.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:05.020 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:05.041 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:05.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:05.043 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:05.144 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:05.170 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:05.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:05.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:05.273 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:05.296 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:05.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:05.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:05.398 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:05.425 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:05.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:05.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:05.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:05.550 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:05.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:05.552 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:05.653 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:05.679 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:05.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:05.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:05.782 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:05.805 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:05.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:05.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:05.907 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:05.931 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:05.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:06.033 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:06.056 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:06.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:06.058 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:06.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:06.192 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:06.192 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:06.293 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:06.315 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:06.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:06.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:06.358 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:28:06.359 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:28:06.359 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:28:06.359 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:28:06.359 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:28:06.360 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:28:06.360 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:28:06.360 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:28:06.360 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:28:06.361 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:28:06.361 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:28:06.361 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:28:06.361 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:28:06.362 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:28:06.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:06.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:06.440 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:06.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:06.481 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:28:06.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:06 smithi076 conmon[56514]: cluster 2021-03-15T22:28:05.098048+0000 mgr.y (mgr.14144) 742 : cluster [DBG] pgmap v960: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:06.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:06 smithi076 conmon[56514]: cluster 2021-03-15T22:28:05.155812+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e549: 8 total, 6 up, 7 in 2021-03-15T22:28:06.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:06 smithi076 conmon[56514]: audit 2021-03-15T22:28:05.921895+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? 172.21.15.76:0/419176893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:28:06.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:06 smithi076 conmon[56514]: audit 2021-03-15T22:28:05.934106+0000 mon.c (mon.1) 46 : audit [DBG] from='client.? 172.21.15.76:0/677902868' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:06.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:06 smithi076 conmon[62156]: cluster 2021-03-15T22:28:05.098048+0000 mgr.y (mgr.14144) 742 : cluster [DBG] pgmap v960: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:06.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:06 smithi076 conmon[62156]: cluster 2021-03-15T22:28:05.155812+0000 mon.a (mon.0) 1546 : cluster [DBG] osdmap e549: 8 total, 6 up, 7 in 2021-03-15T22:28:06.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:06 smithi076 conmon[62156]: audit 2021-03-15T22:28:05.921895+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? 172.21.15.76:0/419176893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:28:06.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:06 smithi076 conmon[62156]: audit 2021-03-15T22:28:05.934106+0000 mon.c (mon.1) 46 : audit [DBG] from='client.? 172.21.15.76:0/677902868' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: cluster 2021-03-15T22:28:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021 2021-03-15T22:28:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: - 2021-03-15T22:28:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 03 2021-03-15T22:28:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: - 2021-03-15T22:28:06.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 15 2021-03-15T22:28:06.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: T 2021-03-15T22:28:06.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 22 2021-03-15T22:28:06.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: : 2021-03-15T22:28:06.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 28 2021-03-15T22:28:06.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: : 2021-03-15T22:28:06.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 05 2021-03-15T22:28:06.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: . 2021-03-15T22:28:06.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 098048 2021-03-15T22:28:06.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: +0000 2021-03-15T22:28:06.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:06.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: ( 2021-03-15T22:28:06.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: mgr 2021-03-15T22:28:06.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: . 2021-03-15T22:28:06.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 14144 2021-03-15T22:28:06.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: ) 2021-03-15T22:28:06.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 742 2021-03-15T22:28:06.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: : 2021-03-15T22:28:06.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: cluster 2021-03-15T22:28:06.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:06.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: pgmap v960: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 3.1 KiB/s rd, 0 B/s wr, 4 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:06.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: cluster 2021-03-15T22:28:06.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021 2021-03-15T22:28:06.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: - 2021-03-15T22:28:06.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 03 2021-03-15T22:28:06.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: - 2021-03-15T22:28:06.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 15 2021-03-15T22:28:06.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: T 2021-03-15T22:28:06.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 22 2021-03-15T22:28:06.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: : 2021-03-15T22:28:06.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 28 2021-03-15T22:28:06.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: : 2021-03-15T22:28:06.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 05 2021-03-15T22:28:06.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: . 2021-03-15T22:28:06.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 155812 2021-03-15T22:28:06.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: +0000 2021-03-15T22:28:06.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:06.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: ( 2021-03-15T22:28:06.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: mon 2021-03-15T22:28:06.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: . 2021-03-15T22:28:06.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 0 2021-03-15T22:28:06.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: ) 2021-03-15T22:28:06.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 1546 2021-03-15T22:28:06.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: : 2021-03-15T22:28:06.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: cluster 2021-03-15T22:28:06.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:06.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: osdmap e549: 8 total, 6 up, 7 in 2021-03-15T22:28:06.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: audit 2021-03-15T22:28:06.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021 2021-03-15T22:28:06.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: -03-15T22:28: 2021-03-15T22:28:06.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 05.921895+0000 mon.a (mon.0) 1547 : audit [INF] from='client.? 172.21.15.76:0/419176893' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:28:06.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: audit 2021-03-15T22:28:06.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021 2021-03-15T22:28:06.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: - 2021-03-15T22:28:06.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 03 2021-03-15T22:28:06.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: - 2021-03-15T22:28:06.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 15 2021-03-15T22:28:06.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: T 2021-03-15T22:28:06.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 22 2021-03-15T22:28:06.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: : 2021-03-15T22:28:06.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 28 2021-03-15T22:28:06.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: : 2021-03-15T22:28:06.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 05 2021-03-15T22:28:06.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: . 2021-03-15T22:28:06.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 934106 2021-03-15T22:28:06.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: +0000 2021-03-15T22:28:06.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: mon.c 2021-03-15T22:28:06.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: ( 2021-03-15T22:28:06.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: mon 2021-03-15T22:28:06.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: . 2021-03-15T22:28:06.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 1 2021-03-15T22:28:06.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: ) 2021-03-15T22:28:06.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 46 2021-03-15T22:28:06.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: : 2021-03-15T22:28:06.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: audit 2021-03-15T22:28:06.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:06.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/677902868' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:06.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:06 smithi125 conmon[56739]: 2021-03-15T22:28:06.563 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:06.586 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:06.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:06.688 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:06.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:06.710 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:06.711 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:06.812 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:06.840 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:06.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:06.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:06.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:06.968 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:06.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:07.069 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:07.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:07.092 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:07.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:07.173 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:28:07.173 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:28:07.193 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:07.219 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:07.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:07.321 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:07.343 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:07.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:07.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:07.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:07.468 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:07.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:07.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:07.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[56514]: cluster 2021-03-15T22:28:06.468491+0000 mon.a (mon. 2021-03-15T22:28:07.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[56514]: 0) 1548 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:07.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[56514]: audit 2021-03-15T22:28:06.473337+0000 mon.a (mon.0) 1549 : audit [INF] from='client.? 172.21.15.76:0/419176893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:28:07.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[56514]: cluster 2021-03-15T22:28:06.473384+0000 mon.a (mon.0) 1550 : cluster [DBG] osdmap e550: 8 total, 6 up, 7 in 2021-03-15T22:28:07.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[62156]: cluster 2021-03-15T22:28:06.468491+0000 mon.a 2021-03-15T22:28:07.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[62156]: (mon.0) 1548 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:07.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[62156]: audit 2021-03-15T22:28: 2021-03-15T22:28:07.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[62156]: 06.473337+0000 mon.a (mon.0) 1549 : audit [INF] from='client.? 172.21.15.76:0/419176893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:28:07.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:28:07.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[62156]: -03-15T22:28:06.473384+0000 2021-03-15T22:28:07.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[62156]: mon.a (mon.0) 1550 : cluster [DBG] 2021-03-15T22:28:07.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:07 smithi076 conmon[62156]: osdmap e550: 8 total, 6 up, 7 in 2021-03-15T22:28:07.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:07 smithi125 conmon[56739]: cluster 2021-03-15T22:28:06.468491+0000 mon.a (mon.0 2021-03-15T22:28:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:07 smithi125 conmon[56739]: ) 1548 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:07 smithi125 conmon[56739]: audit 2021-03-15T22:28:06.473337+0000 mon.a (mon.0) 1549 : audit 2021-03-15T22:28:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:07 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/419176893' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:28:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:07 smithi125 conmon[56739]: cluster 2021-03-15T22:28:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:07 smithi125 conmon[56739]: 2021-03-15T22:28:06.473384 2021-03-15T22:28:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:07 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1550 : cluster [DBG] 2021-03-15T22:28:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:07 smithi125 conmon[56739]: osdmap e550: 8 total, 6 up, 7 in 2021-03-15T22:28:07.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:07.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:07.596 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:07.596 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:07.698 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:07.720 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:07.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:07.721 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:07.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:07.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:07.847 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:07.847 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:07.948 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:07.971 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:07.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:07.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:07.979 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3] out_osds: [0] dead_osds: [2, 5] live_osds: [0, 1, 3, 4, 6, 7] 2021-03-15T22:28:07.979 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:28:07.980 INFO:tasks.thrashosds.thrasher:Adding osd 0 2021-03-15T22:28:07.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd in 0 2021-03-15T22:28:08.074 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:08.085 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:28:08.085 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:28:08.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:28:08.086 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:28:08.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.086 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2260 7.2 7:42f18a95:::smithi07694878-45:8d [sparse-read 0~2486556,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e264)", 2021-03-15T22:28:08.087 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.527310+0000", 2021-03-15T22:28:08.087 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 308.553632134, 2021-03-15T22:28:08.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53810241999999997, 2021-03-15T22:28:08.087 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.087 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:08.088 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.088 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.088 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.088 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2260 2021-03-15T22:28:08.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.089 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527310+0000", 2021-03-15T22:28:08.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527310+0000", 2021-03-15T22:28:08.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4089999999999996e-06 2021-03-15T22:28:08.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527314+0000", 2021-03-15T22:28:08.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.332e-06 2021-03-15T22:28:08.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527317+0000", 2021-03-15T22:28:08.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7672999999999998e-05 2021-03-15T22:28:08.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.095 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.095 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527335+0000", 2021-03-15T22:28:08.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6059999999999999e-06 2021-03-15T22:28:08.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.097 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527342+0000", 2021-03-15T22:28:08.097 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2244999999999996e-05 2021-03-15T22:28:08.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527424+0000", 2021-03-15T22:28:08.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.431e-06 2021-03-15T22:28:08.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:08.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527433+0000", 2021-03-15T22:28:08.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53064125699999998 2021-03-15T22:28:08.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.058075+0000", 2021-03-15T22:28:08.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015664499999999999 2021-03-15T22:28:08.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.058231+0000", 2021-03-15T22:28:08.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071808219999999999 2021-03-15T22:28:08.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.065412+0000", 2021-03-15T22:28:08.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53810241999999997 2021-03-15T22:28:08.103 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.103 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.104 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.104 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2261 7.2 7:42f18a95:::smithi07694878-45:8d [sparse-read 0~2486556] snapc 0=[] ondisk+read+known_if_redirected e264)", 2021-03-15T22:28:08.105 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.527444+0000", 2021-03-15T22:28:08.105 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 308.55349790299999, 2021-03-15T22:28:08.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53896513499999998, 2021-03-15T22:28:08.105 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.106 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:08.106 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.106 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.107 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.107 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2261 2021-03-15T22:28:08.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.107 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527444+0000", 2021-03-15T22:28:08.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527444+0000", 2021-03-15T22:28:08.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.581e-06 2021-03-15T22:28:08.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.110 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527445+0000", 2021-03-15T22:28:08.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0140000000000001e-06 2021-03-15T22:28:08.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527447+0000", 2021-03-15T22:28:08.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.644e-06 2021-03-15T22:28:08.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527455+0000", 2021-03-15T22:28:08.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.42e-06 2021-03-15T22:28:08.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527460+0000", 2021-03-15T22:28:08.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6919000000000003e-05 2021-03-15T22:28:08.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527506+0000", 2021-03-15T22:28:08.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6880000000000005e-06 2021-03-15T22:28:08.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:08.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527515+0000", 2021-03-15T22:28:08.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53819720400000004 2021-03-15T22:28:08.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.065712+0000", 2021-03-15T22:28:08.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118725 2021-03-15T22:28:08.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.065831+0000", 2021-03-15T22:28:08.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00057793999999999999 2021-03-15T22:28:08.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066409+0000", 2021-03-15T22:28:08.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53896513499999998 2021-03-15T22:28:08.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.121 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.122 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.122 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2262 7.2 7:42f18a95:::smithi07694878-45:8d [sparse-read 0~2486556] snapc 0=[] ondisk+read+known_if_redirected e264)", 2021-03-15T22:28:08.122 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:59.527470+0000", 2021-03-15T22:28:08.122 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 308.55347176999999, 2021-03-15T22:28:08.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53929365200000001, 2021-03-15T22:28:08.123 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.123 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:08.124 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.124 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.124 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.124 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2262 2021-03-15T22:28:08.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.125 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527470+0000", 2021-03-15T22:28:08.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527470+0000", 2021-03-15T22:28:08.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1170000000000001e-06 2021-03-15T22:28:08.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527471+0000", 2021-03-15T22:28:08.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.468e-06 2021-03-15T22:28:08.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527473+0000", 2021-03-15T22:28:08.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1550000000000004e-06 2021-03-15T22:28:08.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527479+0000", 2021-03-15T22:28:08.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1910000000000002e-06 2021-03-15T22:28:08.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527483+0000", 2021-03-15T22:28:08.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104373 2021-03-15T22:28:08.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.133 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527587+0000", 2021-03-15T22:28:08.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3989999999999993e-06 2021-03-15T22:28:08.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:08.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.527597+0000", 2021-03-15T22:28:08.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53885792499999996 2021-03-15T22:28:08.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066455+0000", 2021-03-15T22:28:08.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000101804 2021-03-15T22:28:08.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.137 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066556+0000", 2021-03-15T22:28:08.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00020722000000000001 2021-03-15T22:28:08.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:00.066764+0000", 2021-03-15T22:28:08.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.53929365200000001 2021-03-15T22:28:08.139 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.139 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.140 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.140 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3661 7.3 7:c7825e8f:::smithi07694878-26:head [write 1692526~473837] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:28:08.140 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.130939+0000", 2021-03-15T22:28:08.141 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 187.95000236000001, 2021-03-15T22:28:08.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039320842000000002, 2021-03-15T22:28:08.141 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.141 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:08.141 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.142 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.142 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.142 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3661 2021-03-15T22:28:08.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.143 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.143 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.130939+0000", 2021-03-15T22:28:08.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.130939+0000", 2021-03-15T22:28:08.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024949 2021-03-15T22:28:08.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131189+0000", 2021-03-15T22:28:08.146 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0850000000000002e-06 2021-03-15T22:28:08.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131195+0000", 2021-03-15T22:28:08.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1999000000000001e-05 2021-03-15T22:28:08.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131217+0000", 2021-03-15T22:28:08.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2747e-05 2021-03-15T22:28:08.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131230+0000", 2021-03-15T22:28:08.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3211000000000001e-05 2021-03-15T22:28:08.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131303+0000", 2021-03-15T22:28:08.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134301 2021-03-15T22:28:08.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131437+0000", 2021-03-15T22:28:08.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019751799999999999 2021-03-15T22:28:08.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:28:08.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131635+0000", 2021-03-15T22:28:08.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019844060000000002 2021-03-15T22:28:08.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:08.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.133619+0000", 2021-03-15T22:28:08.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0061900419999999998 2021-03-15T22:28:08.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.139809+0000", 2021-03-15T22:28:08.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.030327191999999999 2021-03-15T22:28:08.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170136+0000", 2021-03-15T22:28:08.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6150999999999997e-05 2021-03-15T22:28:08.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:08.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170203+0000", 2021-03-15T22:28:08.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.77e-05 2021-03-15T22:28:08.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170260+0000", 2021-03-15T22:28:08.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039320842000000002 2021-03-15T22:28:08.158 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.158 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.158 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.159 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3662 7.3 7:c7825e8f:::smithi07694878-26:head [setxattr _header (58) in=7b,truncate 2403500] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:28:08.159 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.131248+0000", 2021-03-15T22:28:08.159 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 187.949694099, 2021-03-15T22:28:08.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039224930999999998, 2021-03-15T22:28:08.159 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.160 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:08.160 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.160 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.160 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.160 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3662 2021-03-15T22:28:08.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.161 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.161 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.161 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131248+0000", 2021-03-15T22:28:08.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131248+0000", 2021-03-15T22:28:08.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9e-06 2021-03-15T22:28:08.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131250+0000", 2021-03-15T22:28:08.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3710000000000001e-06 2021-03-15T22:28:08.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.164 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131251+0000", 2021-03-15T22:28:08.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1639999999999998e-06 2021-03-15T22:28:08.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131257+0000", 2021-03-15T22:28:08.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.403e-06 2021-03-15T22:28:08.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131262+0000", 2021-03-15T22:28:08.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0014190439999999999 2021-03-15T22:28:08.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.132681+0000", 2021-03-15T22:28:08.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013826999999999999 2021-03-15T22:28:08.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.132819+0000", 2021-03-15T22:28:08.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023518200000000001 2021-03-15T22:28:08.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:28:08.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.133054+0000", 2021-03-15T22:28:08.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0044182889999999997 2021-03-15T22:28:08.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:08.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.137472+0000", 2021-03-15T22:28:08.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0025551509999999999 2021-03-15T22:28:08.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.172 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.140027+0000", 2021-03-15T22:28:08.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.030306403999999999 2021-03-15T22:28:08.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170334+0000", 2021-03-15T22:28:08.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1277000000000002e-05 2021-03-15T22:28:08.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:08.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170405+0000", 2021-03-15T22:28:08.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7476000000000002e-05 2021-03-15T22:28:08.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170473+0000", 2021-03-15T22:28:08.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039224930999999998 2021-03-15T22:28:08.176 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.176 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.176 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.177 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3663 7.3 7:c7825e8f:::smithi07694878-26:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e385)", 2021-03-15T22:28:08.177 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.131268+0000", 2021-03-15T22:28:08.177 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 187.94967424000001, 2021-03-15T22:28:08.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039463573000000002, 2021-03-15T22:28:08.178 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.178 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:08.178 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.178 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.179 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.179 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3663 2021-03-15T22:28:08.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.179 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131268+0000", 2021-03-15T22:28:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131268+0000", 2021-03-15T22:28:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2619999999999999e-06 2021-03-15T22:28:08.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131269+0000", 2021-03-15T22:28:08.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4309999999999999e-06 2021-03-15T22:28:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131270+0000", 2021-03-15T22:28:08.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4150000000000004e-06 2021-03-15T22:28:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131277+0000", 2021-03-15T22:28:08.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0239999999999999e-06 2021-03-15T22:28:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.131281+0000", 2021-03-15T22:28:08.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002198247 2021-03-15T22:28:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.186 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.133479+0000", 2021-03-15T22:28:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000125449 2021-03-15T22:28:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:08.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.133604+0000", 2021-03-15T22:28:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036921066000000002 2021-03-15T22:28:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170525+0000", 2021-03-15T22:28:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1867999999999995e-05 2021-03-15T22:28:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170617+0000", 2021-03-15T22:28:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000113811 2021-03-15T22:28:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.170731+0000", 2021-03-15T22:28:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039463573000000002 2021-03-15T22:28:08.191 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.192 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.192 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.193 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3675 7.3 7:c00fa989:::smithi07694878-50:head [write 453921~729545,stat] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:28:08.193 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.189699+0000", 2021-03-15T22:28:08.193 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 187.891242307, 2021-03-15T22:28:08.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043512464000000001, 2021-03-15T22:28:08.193 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.194 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:08.194 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.194 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.194 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.194 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3675 2021-03-15T22:28:08.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.195 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189699+0000", 2021-03-15T22:28:08.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189699+0000", 2021-03-15T22:28:08.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015413499999999999 2021-03-15T22:28:08.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189854+0000", 2021-03-15T22:28:08.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1420000000000001e-06 2021-03-15T22:28:08.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189859+0000", 2021-03-15T22:28:08.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3388999999999999e-05 2021-03-15T22:28:08.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189882+0000", 2021-03-15T22:28:08.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8409999999999995e-06 2021-03-15T22:28:08.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.201 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189892+0000", 2021-03-15T22:28:08.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2676999999999996e-05 2021-03-15T22:28:08.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.189955+0000", 2021-03-15T22:28:08.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119745 2021-03-15T22:28:08.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.190074+0000", 2021-03-15T22:28:08.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026131199999999998 2021-03-15T22:28:08.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:28:08.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.190336+0000", 2021-03-15T22:28:08.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0034304180000000002 2021-03-15T22:28:08.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.193766+0000", 2021-03-15T22:28:08.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039036350999999997 2021-03-15T22:28:08.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:08.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.232803+0000", 2021-03-15T22:28:08.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030217699999999999 2021-03-15T22:28:08.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233105+0000", 2021-03-15T22:28:08.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9181999999999999e-05 2021-03-15T22:28:08.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:08.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233144+0000", 2021-03-15T22:28:08.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8095000000000004e-05 2021-03-15T22:28:08.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233212+0000", 2021-03-15T22:28:08.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043512464000000001 2021-03-15T22:28:08.210 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.211 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.211 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.211 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3676 7.3 7:c00fa989:::smithi07694878-50:head [write 1584325~621164] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:28:08.212 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.195497+0000", 2021-03-15T22:28:08.212 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 187.885444519, 2021-03-15T22:28:08.212 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042481251999999997, 2021-03-15T22:28:08.212 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.212 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:08.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.213 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.213 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3676 2021-03-15T22:28:08.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.214 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195497+0000", 2021-03-15T22:28:08.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195497+0000", 2021-03-15T22:28:08.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013062600000000001 2021-03-15T22:28:08.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195628+0000", 2021-03-15T22:28:08.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9829999999999998e-06 2021-03-15T22:28:08.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195632+0000", 2021-03-15T22:28:08.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6456e-05 2021-03-15T22:28:08.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195648+0000", 2021-03-15T22:28:08.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7849999999999997e-06 2021-03-15T22:28:08.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195654+0000", 2021-03-15T22:28:08.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6258999999999999e-05 2021-03-15T22:28:08.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195700+0000", 2021-03-15T22:28:08.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3122999999999998e-05 2021-03-15T22:28:08.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:08.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195774+0000", 2021-03-15T22:28:08.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037474535000000003 2021-03-15T22:28:08.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233248+0000", 2021-03-15T22:28:08.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3729e-05 2021-03-15T22:28:08.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233302+0000", 2021-03-15T22:28:08.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000120444 2021-03-15T22:28:08.225 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:28:08.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.233422+0000", 2021-03-15T22:28:08.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020581520000000002 2021-03-15T22:28:08.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.226 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.226 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.235480+0000", 2021-03-15T22:28:08.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1943000000000001e-05 2021-03-15T22:28:08.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.227 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:08.227 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.235492+0000", 2021-03-15T22:28:08.228 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023698780000000002 2021-03-15T22:28:08.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.228 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.228 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.237862+0000", 2021-03-15T22:28:08.229 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0804999999999998e-05 2021-03-15T22:28:08.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:08.229 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.237923+0000", 2021-03-15T22:28:08.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5534000000000001e-05 2021-03-15T22:28:08.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.230 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.231 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.237979+0000", 2021-03-15T22:28:08.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042481251999999997 2021-03-15T22:28:08.231 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.231 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.231 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.232 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.232 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3677 7.3 7:c00fa989:::smithi07694878-50:head [setxattr _header (58) in=7b,truncate 2453921] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:28:08.232 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.195666+0000", 2021-03-15T22:28:08.232 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 187.88527618399999, 2021-03-15T22:28:08.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042508698999999997, 2021-03-15T22:28:08.233 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.233 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:08.233 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.233 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.234 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.234 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3677 2021-03-15T22:28:08.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.234 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195666+0000", 2021-03-15T22:28:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195666+0000", 2021-03-15T22:28:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.385e-06 2021-03-15T22:28:08.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195668+0000", 2021-03-15T22:28:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4890000000000001e-06 2021-03-15T22:28:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.238 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195669+0000", 2021-03-15T22:28:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9700000000000001e-06 2021-03-15T22:28:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195673+0000", 2021-03-15T22:28:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6120000000000001e-06 2021-03-15T22:28:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195676+0000", 2021-03-15T22:28:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012614700000000001 2021-03-15T22:28:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195802+0000", 2021-03-15T22:28:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6866000000000001e-05 2021-03-15T22:28:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195859+0000", 2021-03-15T22:28:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038488834999999999 2021-03-15T22:28:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.234348+0000", 2021-03-15T22:28:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6688999999999999e-05 2021-03-15T22:28:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.234405+0000", 2021-03-15T22:28:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000111799 2021-03-15T22:28:08.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:28:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.234516+0000", 2021-03-15T22:28:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00099373400000000002 2021-03-15T22:28:08.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.235510+0000", 2021-03-15T22:28:08.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3916000000000003e-05 2021-03-15T22:28:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.235574+0000", 2021-03-15T22:28:08.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024673519999999999 2021-03-15T22:28:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238041+0000", 2021-03-15T22:28:08.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7310000000000004e-05 2021-03-15T22:28:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238109+0000", 2021-03-15T22:28:08.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5594999999999998e-05 2021-03-15T22:28:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238174+0000", 2021-03-15T22:28:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042508698999999997 2021-03-15T22:28:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3678 7.3 7:c00fa989:::smithi07694878-50:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e385)", 2021-03-15T22:28:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.195704+0000", 2021-03-15T22:28:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 187.88523805899999, 2021-03-15T22:28:08.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042827408999999997, 2021-03-15T22:28:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.254 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3678 2021-03-15T22:28:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195704+0000", 2021-03-15T22:28:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195704+0000", 2021-03-15T22:28:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.184e-06 2021-03-15T22:28:08.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195705+0000", 2021-03-15T22:28:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4649999999999999e-06 2021-03-15T22:28:08.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195706+0000", 2021-03-15T22:28:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7109999999999998e-06 2021-03-15T22:28:08.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195713+0000", 2021-03-15T22:28:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2100000000000002e-06 2021-03-15T22:28:08.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195716+0000", 2021-03-15T22:28:08.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016275600000000001 2021-03-15T22:28:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195879+0000", 2021-03-15T22:28:08.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9199000000000002e-05 2021-03-15T22:28:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.195918+0000", 2021-03-15T22:28:08.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.038899652999999999 2021-03-15T22:28:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.234818+0000", 2021-03-15T22:28:08.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1717999999999993e-05 2021-03-15T22:28:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:08.265 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.234890+0000", 2021-03-15T22:28:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003332939 2021-03-15T22:28:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.266 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238223+0000", 2021-03-15T22:28:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2285000000000002e-05 2021-03-15T22:28:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.267 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238305+0000", 2021-03-15T22:28:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022628899999999999 2021-03-15T22:28:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.268 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.238531+0000", 2021-03-15T22:28:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042827408999999997 2021-03-15T22:28:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3982 7.3 7:c458c800:::smithi07694878-3:head [copy-from ver 119] snapc 13f=[13f,13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e421)", 2021-03-15T22:28:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:36.115531+0000", 2021-03-15T22:28:08.270 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 151.96541048099999, 2021-03-15T22:28:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0139079740000001, 2021-03-15T22:28:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.271 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3982 2021-03-15T22:28:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115531+0000", 2021-03-15T22:28:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115531+0000", 2021-03-15T22:28:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4710000000000001e-06 2021-03-15T22:28:08.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115536+0000", 2021-03-15T22:28:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5420000000000001e-06 2021-03-15T22:28:08.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115539+0000", 2021-03-15T22:28:08.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3487e-05 2021-03-15T22:28:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115553+0000", 2021-03-15T22:28:08.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4500000000000004e-06 2021-03-15T22:28:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115561+0000", 2021-03-15T22:28:08.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0590000000000001e-05 2021-03-15T22:28:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115622+0000", 2021-03-15T22:28:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3449999999999995e-06 2021-03-15T22:28:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:28:08.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115631+0000", 2021-03-15T22:28:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00089869400000000001 2021-03-15T22:28:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.116530+0000", 2021-03-15T22:28:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8217000000000001e-05 2021-03-15T22:28:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.116598+0000", 2021-03-15T22:28:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.00515501 2021-03-15T22:28:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:28:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.121753+0000", 2021-03-15T22:28:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073058109999999997 2021-03-15T22:28:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129059+0000", 2021-03-15T22:28:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1652e-05 2021-03-15T22:28:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129071+0000", 2021-03-15T22:28:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025268099999999999 2021-03-15T22:28:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129323+0000", 2021-03-15T22:28:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5415999999999999e-05 2021-03-15T22:28:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129379+0000", 2021-03-15T22:28:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0608e-05 2021-03-15T22:28:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129439+0000", 2021-03-15T22:28:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0139079740000001 2021-03-15T22:28:08.289 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.290 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3983 7.3 7:c458c800:::smithi07694878-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e421)", 2021-03-15T22:28:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:36.115576+0000", 2021-03-15T22:28:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 151.965366179, 2021-03-15T22:28:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0140553489999999, 2021-03-15T22:28:08.291 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.292 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3983 2021-03-15T22:28:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115576+0000", 2021-03-15T22:28:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115576+0000", 2021-03-15T22:28:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2550000000000001e-06 2021-03-15T22:28:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115577+0000", 2021-03-15T22:28:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8390000000000001e-06 2021-03-15T22:28:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115579+0000", 2021-03-15T22:28:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0740000000000004e-06 2021-03-15T22:28:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115586+0000", 2021-03-15T22:28:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3220000000000003e-06 2021-03-15T22:28:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115590+0000", 2021-03-15T22:28:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108726 2021-03-15T22:28:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115699+0000", 2021-03-15T22:28:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9349999999999997e-06 2021-03-15T22:28:08.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:28:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115706+0000", 2021-03-15T22:28:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021999540000000001 2021-03-15T22:28:08.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.117906+0000", 2021-03-15T22:28:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6322e-05 2021-03-15T22:28:08.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.117962+0000", 2021-03-15T22:28:08.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.00844165 2021-03-15T22:28:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.126404+0000", 2021-03-15T22:28:08.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5353999999999996e-05 2021-03-15T22:28:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.126489+0000", 2021-03-15T22:28:08.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029835399999999998 2021-03-15T22:28:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.306 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129473+0000", 2021-03-15T22:28:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5050999999999997e-05 2021-03-15T22:28:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.307 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129548+0000", 2021-03-15T22:28:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3326999999999997e-05 2021-03-15T22:28:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129631+0000", 2021-03-15T22:28:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0140553489999999 2021-03-15T22:28:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.309 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4290 7.d e445/355 7:bd183e1b:::smithi07694878-46:head v 445'101, mlcod=442'97)", 2021-03-15T22:28:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.163403+0000", 2021-03-15T22:28:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 127.917539193, 2021-03-15T22:28:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047940739000000003, 2021-03-15T22:28:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:08.311 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163403+0000", 2021-03-15T22:28:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163403+0000", 2021-03-15T22:28:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2109999999999997e-06 2021-03-15T22:28:08.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163408+0000", 2021-03-15T22:28:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4139999999999998e-06 2021-03-15T22:28:08.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163411+0000", 2021-03-15T22:28:08.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7283e-05 2021-03-15T22:28:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163429+0000", 2021-03-15T22:28:08.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.793e-06 2021-03-15T22:28:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163435+0000", 2021-03-15T22:28:08.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1350000000000001e-05 2021-03-15T22:28:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163487+0000", 2021-03-15T22:28:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6448000000000002e-05 2021-03-15T22:28:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163513+0000", 2021-03-15T22:28:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047789320000000003 2021-03-15T22:28:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:08.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.211302+0000", 2021-03-15T22:28:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0920000000000001e-05 2021-03-15T22:28:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.211343+0000", 2021-03-15T22:28:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047940739000000003 2021-03-15T22:28:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.322 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4287 7.7 e445/420 7:ed4a01d2:::smithi07694878-2:head v 445'413, mlcod=444'410)", 2021-03-15T22:28:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.163605+0000", 2021-03-15T22:28:08.323 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 127.917337284, 2021-03-15T22:28:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047765347999999999, 2021-03-15T22:28:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163605+0000", 2021-03-15T22:28:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163605+0000", 2021-03-15T22:28:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4999999999999999e-06 2021-03-15T22:28:08.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163608+0000", 2021-03-15T22:28:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5450000000000002e-06 2021-03-15T22:28:08.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163611+0000", 2021-03-15T22:28:08.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2568000000000001e-05 2021-03-15T22:28:08.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163623+0000", 2021-03-15T22:28:08.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5350000000000004e-06 2021-03-15T22:28:08.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163629+0000", 2021-03-15T22:28:08.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9073999999999998e-05 2021-03-15T22:28:08.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163678+0000", 2021-03-15T22:28:08.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.029e-05 2021-03-15T22:28:08.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163708+0000", 2021-03-15T22:28:08.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047628396000000003 2021-03-15T22:28:08.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:08.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.211336+0000", 2021-03-15T22:28:08.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3439999999999998e-05 2021-03-15T22:28:08.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.211370+0000", 2021-03-15T22:28:08.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047765347999999999 2021-03-15T22:28:08.335 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.335 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.335 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.336 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4288 7.7 e445/420 7:ed4a01d2:::smithi07694878-2:head v 445'414, mlcod=444'410)", 2021-03-15T22:28:08.336 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.164544+0000", 2021-03-15T22:28:08.336 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 127.916398141, 2021-03-15T22:28:08.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046850402999999999, 2021-03-15T22:28:08.337 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.337 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:08.337 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164544+0000", 2021-03-15T22:28:08.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164544+0000", 2021-03-15T22:28:08.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2189999999999998e-06 2021-03-15T22:28:08.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164548+0000", 2021-03-15T22:28:08.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.196e-06 2021-03-15T22:28:08.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164551+0000", 2021-03-15T22:28:08.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4538e-05 2021-03-15T22:28:08.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164566+0000", 2021-03-15T22:28:08.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7120000000000003e-06 2021-03-15T22:28:08.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164572+0000", 2021-03-15T22:28:08.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9184999999999996e-05 2021-03-15T22:28:08.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164642+0000", 2021-03-15T22:28:08.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4492000000000005e-05 2021-03-15T22:28:08.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164706+0000", 2021-03-15T22:28:08.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046666672999999999 2021-03-15T22:28:08.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:08.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.211373+0000", 2021-03-15T22:28:08.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1387999999999999e-05 2021-03-15T22:28:08.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.211394+0000", 2021-03-15T22:28:08.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.046850402999999999 2021-03-15T22:28:08.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.348 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.349 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4380 7.6 e455/421 7:6283974b:::smithi07694878-30:head v 455'1076, mlcod=454'1072)", 2021-03-15T22:28:08.349 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:10.132423+0000", 2021-03-15T22:28:08.349 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 117.94851865, 2021-03-15T22:28:08.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.120130079, 2021-03-15T22:28:08.350 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.350 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:08.350 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.132423+0000", 2021-03-15T22:28:08.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.132423+0000", 2021-03-15T22:28:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010757970000000001 2021-03-15T22:28:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.133499+0000", 2021-03-15T22:28:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5153e-05 2021-03-15T22:28:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.133514+0000", 2021-03-15T22:28:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.070374909999999999 2021-03-15T22:28:08.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.203889+0000", 2021-03-15T22:28:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6134000000000001e-05 2021-03-15T22:28:08.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.203915+0000", 2021-03-15T22:28:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8656000000000001e-05 2021-03-15T22:28:08.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.203984+0000", 2021-03-15T22:28:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5831999999999999e-05 2021-03-15T22:28:08.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.204010+0000", 2021-03-15T22:28:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048499043999999998 2021-03-15T22:28:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.252509+0000", 2021-03-15T22:28:08.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4552999999999998e-05 2021-03-15T22:28:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.252553+0000", 2021-03-15T22:28:08.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.120130079 2021-03-15T22:28:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5267 7.a 7.68cfe3da (undecoded) ondisk+write+known_if_redirected e540)", 2021-03-15T22:28:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.213130+0000", 2021-03-15T22:28:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.867811623999998, 2021-03-15T22:28:08.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99493337699999995, 2021-03-15T22:28:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:28:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.363 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5267 2021-03-15T22:28:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213130+0000", 2021-03-15T22:28:08.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213130+0000", 2021-03-15T22:28:08.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027025900000000002 2021-03-15T22:28:08.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213400+0000", 2021-03-15T22:28:08.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7129999999999999e-06 2021-03-15T22:28:08.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213404+0000", 2021-03-15T22:28:08.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3495e-05 2021-03-15T22:28:08.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213418+0000", 2021-03-15T22:28:08.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5329999999999997e-06 2021-03-15T22:28:08.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213425+0000", 2021-03-15T22:28:08.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5578999999999998e-05 2021-03-15T22:28:08.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213471+0000", 2021-03-15T22:28:08.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0062e-05 2021-03-15T22:28:08.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:08.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213481+0000", 2021-03-15T22:28:08.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99456081500000004 2021-03-15T22:28:08.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208042+0000", 2021-03-15T22:28:08.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1920999999999999e-05 2021-03-15T22:28:08.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208064+0000", 2021-03-15T22:28:08.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99493337699999995 2021-03-15T22:28:08.375 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.375 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.375 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.376 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5268 7.a 7.68cfe3da (undecoded) ondisk+write+known_if_redirected e540)", 2021-03-15T22:28:08.376 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.219467+0000", 2021-03-15T22:28:08.376 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.861475139, 2021-03-15T22:28:08.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98862604499999995, 2021-03-15T22:28:08.376 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.377 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:28:08.377 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.377 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.377 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.378 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5268 2021-03-15T22:28:08.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.378 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219467+0000", 2021-03-15T22:28:08.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219467+0000", 2021-03-15T22:28:08.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025054199999999997 2021-03-15T22:28:08.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219717+0000", 2021-03-15T22:28:08.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6829999999999999e-06 2021-03-15T22:28:08.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219722+0000", 2021-03-15T22:28:08.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1461e-05 2021-03-15T22:28:08.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219743+0000", 2021-03-15T22:28:08.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4905e-05 2021-03-15T22:28:08.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219758+0000", 2021-03-15T22:28:08.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6631000000000006e-05 2021-03-15T22:28:08.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219825+0000", 2021-03-15T22:28:08.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1799e-05 2021-03-15T22:28:08.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:08.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219837+0000", 2021-03-15T22:28:08.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98824981899999997 2021-03-15T22:28:08.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208087+0000", 2021-03-15T22:28:08.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2049999999999999e-06 2021-03-15T22:28:08.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208093+0000", 2021-03-15T22:28:08.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98862604499999995 2021-03-15T22:28:08.389 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.389 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.389 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.390 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5269 7.a 7.68cfe3da (undecoded) ondisk+write+known_if_redirected e540)", 2021-03-15T22:28:08.390 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.219771+0000", 2021-03-15T22:28:08.390 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.861170610999999, 2021-03-15T22:28:08.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98834424600000004, 2021-03-15T22:28:08.391 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.391 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:28:08.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.391 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.392 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.392 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5269 2021-03-15T22:28:08.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.392 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219771+0000", 2021-03-15T22:28:08.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219771+0000", 2021-03-15T22:28:08.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3630000000000001e-06 2021-03-15T22:28:08.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219774+0000", 2021-03-15T22:28:08.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2139999999999999e-06 2021-03-15T22:28:08.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219775+0000", 2021-03-15T22:28:08.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5919000000000001e-05 2021-03-15T22:28:08.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219791+0000", 2021-03-15T22:28:08.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0747999999999999e-05 2021-03-15T22:28:08.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219801+0000", 2021-03-15T22:28:08.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4376999999999996e-05 2021-03-15T22:28:08.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219876+0000", 2021-03-15T22:28:08.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.585e-06 2021-03-15T22:28:08.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:08.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219884+0000", 2021-03-15T22:28:08.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98822621899999996 2021-03-15T22:28:08.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208111+0000", 2021-03-15T22:28:08.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8210000000000001e-06 2021-03-15T22:28:08.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208115+0000", 2021-03-15T22:28:08.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98834424600000004 2021-03-15T22:28:08.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.403 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.403 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.404 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5270 7.a 7.68cfe3da (undecoded) ondisk+read+rwordered+known_if_redirected e540)", 2021-03-15T22:28:08.404 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.219812+0000", 2021-03-15T22:28:08.404 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 32.861129503000001, 2021-03-15T22:28:08.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98832836899999998, 2021-03-15T22:28:08.405 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:08.405 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:28:08.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:08.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:08.406 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:08.406 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5270 2021-03-15T22:28:08.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.406 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:08.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:08.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219812+0000", 2021-03-15T22:28:08.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:08.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:08.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219812+0000", 2021-03-15T22:28:08.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.88e-06 2021-03-15T22:28:08.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:08.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219814+0000", 2021-03-15T22:28:08.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2960000000000001e-06 2021-03-15T22:28:08.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:08.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219816+0000", 2021-03-15T22:28:08.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0669999999999998e-06 2021-03-15T22:28:08.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:08.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219826+0000", 2021-03-15T22:28:08.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.014e-06 2021-03-15T22:28:08.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:08.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219830+0000", 2021-03-15T22:28:08.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8069e-05 2021-03-15T22:28:08.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219918+0000", 2021-03-15T22:28:08.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6109999999999994e-06 2021-03-15T22:28:08.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:08.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219926+0000", 2021-03-15T22:28:08.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98820947100000001 2021-03-15T22:28:08.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:08.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208136+0000", 2021-03-15T22:28:08.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9609999999999996e-06 2021-03-15T22:28:08.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:08.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:08.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:08.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208141+0000", 2021-03-15T22:28:08.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98832836899999998 2021-03-15T22:28:08.417 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.417 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.417 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.417 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:08.418 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:08.418 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:28:08.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:08.419 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:08.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:08.521 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:08.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:08.546 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:08.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:08.648 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:08.673 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:08.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:08.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:08.717 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-15T22:28:08.742 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:08 smithi125 conmon[56739]: cluster 2021-03-15T22:28:06.941848+0000 osd.4 (osd.4) 117 : cluster 2021-03-15T22:28:08.742 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:08 smithi125 conmon[56739]: [DBG] 4.13 scrub ok 2021-03-15T22:28:08.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:08 smithi125 conmon[56739]: cluster 2021-03-15T22:28:07.098833+0000 mgr.y (mgr.14144) 743 : cluster [DBG] pgmap v963: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:08.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:08 smithi125 conmon[56739]: cluster 2021-03-15T22:28:07 2021-03-15T22:28:08.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:08 smithi125 conmon[56739]: .223364+0000 osd.6 (osd.6) 86 : cluster 2021-03-15T22:28:08.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:08 smithi125 conmon[56739]: [DBG] 7.c scrub ok 2021-03-15T22:28:08.752 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:08.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:08 smithi076 conmon[56514]: cluster 2021-03-15T22:28:06.941848+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:28:08.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:08 smithi076 conmon[56514]: cluster 2021-03-15T22:28:07.098833+0000 mgr.y (mgr.14144) 743 : cluster [DBG] pgmap v963: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:08.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:08 smithi076 conmon[56514]: cluster 2021-03-15T22:28:07.223364+0000 osd.6 (osd.6) 86 : cluster [DBG] 7.c scrub ok 2021-03-15T22:28:08.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:08 smithi076 conmon[62156]: cluster 2021-03-15T22:28:06.941848+0000 osd.4 (osd.4) 117 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:28:08.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:08 smithi076 conmon[62156]: cluster 2021-03-15T22:28:07.098833+0000 mgr.y (mgr.14144) 743 : cluster [DBG] pgmap v963: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 11 KiB/s rd, 0 B/s wr, 17 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:08.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:08 smithi076 conmon[62156]: cluster 2021-03-15T22:28:07.223364+0000 osd.6 (osd.6) 86 : cluster [DBG] 7.c scrub ok 2021-03-15T22:28:08.776 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:08.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:08.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:08.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:08.927 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:08.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:08.995 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:28:09.029 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:09.053 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:09.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:09.155 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:09.174 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:28:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:09.179 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:09.179 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:09.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:09.303 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:09.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:09.406 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:09.429 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:09.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:09.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:09 smithi076 conmon[62156]: cluster 2021-03-15T22:28 2021-03-15T22:28:09.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:09 smithi076 conmon[62156]: :07.906592+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.e scrub ok 2021-03-15T22:28:09.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:09 smithi076 conmon[62156]: cluster 2021-03-15T22:28:08.211152+0000 osd.6 (osd.6) 87 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:28:09.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:09.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:09.556 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:09.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:09.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:09.680 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:09.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:09.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:09.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:09 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:28:09.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:09 smithi076 conmon[56514]: 28:07.906592+0000 osd.4 (osd.4) 118 : cluster [DBG] 4.e scrub ok 2021-03-15T22:28:09.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:09 smithi076 conmon[56514]: cluster 2021-03-15T22:28:08.211152+0000 osd.6 (osd.6) 87 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:28:09.783 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:09.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:09 smithi125 conmon[56739]: cluster 2021-03-15T22:28:07.906592+0000 2021-03-15T22:28:09.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:09 smithi125 conmon[56739]: osd.4 (osd.4) 118 : cluster [DBG] 4.e scrub ok 2021-03-15T22:28:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:09 smithi125 conmon[56739]: cluster 2021-03-15T22:28:08.211152+0000 osd.6 (osd.6) 87 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:28:09.806 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:09.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:09.807 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:09.903 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:09.909 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:09.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:28:09.931 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:09.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:09.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:10.033 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:10.055 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:10.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:10.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:10.158 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:10.187 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:10.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:10.189 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:10.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:28:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:28:10.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:10.322 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:10.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:10.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:10.352 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:10.425 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:10.447 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:10.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:10.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:10.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[56514]: cluster 2021-03-15T22:28:09 2021-03-15T22:28:10.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[56514]: .088679+0000 osd.7 (osd.7) 111 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:28:10.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[56514]: cluster 2021-03-15T22:28:09.099480+0000 mgr.y (mgr.14144) 744 : cluster [DBG] pgmap v964: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:10.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[56514]: audit 2021-03-15T22:28:09.861644+0000 mgr.y (mgr.14144) 745 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:10.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[56514]: cluster 2021-03-15T22:28:10.158512+0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e551: 8 total, 6 up, 7 in 2021-03-15T22:28:10.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:10.573 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:10.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:10.575 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:10.604 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:28:10.604 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:28:10.605 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:28:10.605 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:28:10.674 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:10 smithi125 conmon[56739]: cluster 2021-03-15T22:28:09.088679+0000 osd.7 (osd 2021-03-15T22:28:10.675 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:10 smithi125 conmon[56739]: .7) 111 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:28:10.675 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:10 smithi125 conmon[56739]: cluster 2021-03-15T22:28:09.099480+0000 mgr.y (mgr.14144) 744 : cluster [DBG] pgmap v964: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:10.675 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:10 smithi125 conmon[56739]: audit 2021-03-15T22:28:09.861644+0000 mgr.y (mgr.14144) 745 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:10.675 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:10 smithi125 conmon[56739]: cluster 2021-03-15T22:28:10.158512 2021-03-15T22:28:10.676 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:10 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1551 : cluster [DBG] osdmap e551: 8 total, 6 up, 7 in 2021-03-15T22:28:10.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:10.702 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:10.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:10.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:10.756 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[56514]: debug 2021-03-15T22:28:10.486+0000 7f0537c88700 -1 mon.a@0(leader).osd e551 definitely_dead 0 2021-03-15T22:28:10.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: cluster 2021-03-15T22:28: 2021-03-15T22:28:10.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: 09.088679+0000 osd.7 (osd.7) 111 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:28:10.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:28:10.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: :28:09.099480+0000 mgr.y 2021-03-15T22:28:10.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: (mgr.14144) 744 : cluster 2021-03-15T22:28:10.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: [DBG] pgmap v964: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 9.8 KiB/s rd, 0 B/s wr, 15 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:10.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:28:10.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: 03-15T22:28: 2021-03-15T22:28:10.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: 09.861644+0000 mgr.y (mgr 2021-03-15T22:28:10.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: .14144) 745 : audit [DBG] 2021-03-15T22:28:10.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:10.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:28:10.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: T22:28:10.158512+0000 2021-03-15T22:28:10.760 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: mon.a (mon.0) 1551 : 2021-03-15T22:28:10.760 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:10 smithi076 conmon[62156]: cluster [DBG] osdmap e551: 8 total, 6 up, 7 in 2021-03-15T22:28:10.805 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:10.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:10.828 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:10.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:10.930 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:10.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:10.959 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:10.960 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:11.061 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:11.085 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:11.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:11.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:11.187 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:11.208 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:11.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:11.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:11.223 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-15T22:28:11.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:11.334 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:11.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:11.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:11.436 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:11.458 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:11.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:11.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:11.507 INFO:teuthology.orchestra.run.smithi076.stderr:marked in osd.0. 2021-03-15T22:28:11.513 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:28:11.560 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:11.591 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:11.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:11.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:11.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:11.715 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:11.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:11.717 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:11.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:11 smithi076 conmon[56514]: audit 2021-03-15T22:28:10.487584+0000 2021-03-15T22:28:11.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:11 smithi076 conmon[56514]: mon.a (mon.0) 1552 : audit [DBG] from='client.? 172.21.15.76:0/3943711742' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2021-03-15T22:28:11.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:11 smithi076 conmon[62156]: audit 2021-03-15T22:28:10.487584+0000 2021-03-15T22:28:11.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:11 smithi076 conmon[62156]: mon.a (mon.0) 1552 : audit [DBG] from='client.? 172.21.15.76:0/3943711742' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2021-03-15T22:28:11.769 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:11 smithi125 conmon[56739]: audit 2021-03-15T22:28:10.487584+0000 mon.a (mon.0) 1552 : audit [DBG] from='client.? 172.21.15.76:0/3943711742' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["0"]}]: dispatch 2021-03-15T22:28:11.819 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:11.841 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:11.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:11.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:11.945 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:11.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:11.968 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:11.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:12.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:12.096 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:12.096 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:12.198 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:12.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:12.223 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:12.224 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:12.325 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:12.347 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:12.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:12.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:12.450 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:12.474 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:12.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:12.476 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:12.482 INFO:tasks.thrashosds.thrasher:Added osd 0 2021-03-15T22:28:12.577 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:12.600 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:12.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:12.603 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:12.704 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:12.729 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:12.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:12.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:12.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:12 smithi076 conmon[56514]: cluster 2021-03-15T22:28:11.100112+0000 mgr.y (mgr 2021-03-15T22:28:12.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:12 smithi076 conmon[56514]: .14144) 746 : cluster [DBG] pgmap v966: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.3 KiB/s rd, 0 B/s wr, 13 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:12.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:12 smithi076 conmon[56514]: audit 2021-03-15T22:28:11.493530+0000 mon.a (mon.0) 1553 : audit [INF] from='client.? 172.21.15.76:0/3943711742' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2021-03-15T22:28:12.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:12 smithi076 conmon[56514]: cluster 2021-03-15T22:28:11.493600+0000 mon.a (mon.0) 1554 : cluster [DBG] osdmap e552: 8 total, 6 up, 8 in 2021-03-15T22:28:12.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:12 smithi076 conmon[62156]: cluster 2021-03-15T22:28:11.100112+0000 2021-03-15T22:28:12.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:12 smithi076 conmon[62156]: mgr.y (mgr.14144) 746 : cluster [DBG] pgmap v966: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.3 KiB/s rd, 0 B/s wr, 13 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:12.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:12 smithi076 conmon[62156]: audit 2021-03-15T22:28:11.493530+0000 mon.a (mon.0) 1553 : audit 2021-03-15T22:28:12.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:12 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/3943711742' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2021-03-15T22:28:12.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:12 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:28:12.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:12 smithi076 conmon[62156]: :28:11.493600+0000 mon.a (mon.0) 1554 : cluster 2021-03-15T22:28:12.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:12 smithi076 conmon[62156]: [DBG] osdmap e552: 8 total, 6 up, 8 in 2021-03-15T22:28:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: cluster 2021-03-15T22:28:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021 2021-03-15T22:28:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: - 2021-03-15T22:28:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 03 2021-03-15T22:28:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: - 2021-03-15T22:28:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 15 2021-03-15T22:28:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: T 2021-03-15T22:28:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 22 2021-03-15T22:28:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: : 2021-03-15T22:28:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 28 2021-03-15T22:28:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: : 2021-03-15T22:28:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 11 2021-03-15T22:28:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: . 2021-03-15T22:28:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 100112 2021-03-15T22:28:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: +0000 2021-03-15T22:28:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:12.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: ( 2021-03-15T22:28:12.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: mgr 2021-03-15T22:28:12.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: . 2021-03-15T22:28:12.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 14144 2021-03-15T22:28:12.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: ) 2021-03-15T22:28:12.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 746 2021-03-15T22:28:12.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: : 2021-03-15T22:28:12.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: cluster 2021-03-15T22:28:12.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:12.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: pgmap v966: 138 pgs: 6 undersized+degraded+peered, 30 active+undersized+degraded, 13 undersized+peered, 43 active+undersized, 46 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 8.3 KiB/s rd, 0 B/s wr, 13 op/s; 439/1743 objects degraded (25.186%) 2021-03-15T22:28:12.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: audit 2021-03-15T22:28:12.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021 2021-03-15T22:28:12.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: - 2021-03-15T22:28:12.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 03 2021-03-15T22:28:12.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: - 2021-03-15T22:28:12.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 15 2021-03-15T22:28:12.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: T 2021-03-15T22:28:12.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 22 2021-03-15T22:28:12.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: : 2021-03-15T22:28:12.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 28 2021-03-15T22:28:12.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: : 2021-03-15T22:28:12.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 11 2021-03-15T22:28:12.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: . 2021-03-15T22:28:12.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 493530 2021-03-15T22:28:12.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: +0000 2021-03-15T22:28:12.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:12.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: ( 2021-03-15T22:28:12.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: mon 2021-03-15T22:28:12.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: . 2021-03-15T22:28:12.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 0 2021-03-15T22:28:12.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: ) 2021-03-15T22:28:12.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 1553 2021-03-15T22:28:12.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: : 2021-03-15T22:28:12.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: audit 2021-03-15T22:28:12.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: [INF] 2021-03-15T22:28:12.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/3943711742' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["0"]}]': finished 2021-03-15T22:28:12.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: cluster 2021-03-15T22:28:12.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021 2021-03-15T22:28:12.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: - 2021-03-15T22:28:12.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 03 2021-03-15T22:28:12.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: - 2021-03-15T22:28:12.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 15 2021-03-15T22:28:12.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: T 2021-03-15T22:28:12.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 22 2021-03-15T22:28:12.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: : 2021-03-15T22:28:12.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 28 2021-03-15T22:28:12.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: : 2021-03-15T22:28:12.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 11 2021-03-15T22:28:12.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: . 2021-03-15T22:28:12.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 493600 2021-03-15T22:28:12.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: +0000 2021-03-15T22:28:12.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:12.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: ( 2021-03-15T22:28:12.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: mon 2021-03-15T22:28:12.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: . 2021-03-15T22:28:12.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 0 2021-03-15T22:28:12.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: ) 2021-03-15T22:28:12.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 1554 2021-03-15T22:28:12.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: : 2021-03-15T22:28:12.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: cluster 2021-03-15T22:28:12.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:12.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: osdmap e552: 8 total, 6 up, 8 in 2021-03-15T22:28:12.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:12 smithi125 conmon[56739]: 2021-03-15T22:28:12.832 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:12.854 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:12.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:12.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:12.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:12.979 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:12.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:12.980 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:13.021 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:28:13.037 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:28:13.054 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:28:13.071 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:28:13.082 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:13.088 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:28:13.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:13.105 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:13.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:13.108 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:28:13.109 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:28:13.110 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:28:13.110 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:28:13.110 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:28:13.111 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:28:13.111 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:28:13.111 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:28:13.111 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:28:13.122 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:28:13.122 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:28:13.207 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:13.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:13.232 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:13.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:13.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:13.356 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:13.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:13.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:13.459 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:13.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:13.483 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:13.484 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:13.516 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:28:13.585 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:13.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:13.615 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:13.615 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:13.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:13.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:13.750 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:13.750 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:13.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: cluster 2021-03-15T22:28:13.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021 2021-03-15T22:28:13.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 03 2021-03-15T22:28:13.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 15 2021-03-15T22:28:13.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: T 2021-03-15T22:28:13.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 22 2021-03-15T22:28:13.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 28 2021-03-15T22:28:13.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 12 2021-03-15T22:28:13.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 492530 2021-03-15T22:28:13.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: +0000 2021-03-15T22:28:13.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:13.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ( 2021-03-15T22:28:13.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon 2021-03-15T22:28:13.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 0 2021-03-15T22:28:13.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ) 2021-03-15T22:28:13.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 1555 2021-03-15T22:28:13.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: cluster 2021-03-15T22:28:13.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: [DBG] 2021-03-15T22:28:13.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: osdmap e553: 8 total, 6 up, 8 in 2021-03-15T22:28:13.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: audit 2021-03-15T22:28:13.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021 2021-03-15T22:28:13.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 03 2021-03-15T22:28:13.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 15 2021-03-15T22:28:13.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: T 2021-03-15T22:28:13.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 22 2021-03-15T22:28:13.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 28 2021-03-15T22:28:13.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 12 2021-03-15T22:28:13.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 963923 2021-03-15T22:28:13.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: +0000 2021-03-15T22:28:13.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:13.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ( 2021-03-15T22:28:13.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon 2021-03-15T22:28:13.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 0 2021-03-15T22:28:13.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ) 2021-03-15T22:28:13.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 1556 2021-03-15T22:28:13.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: audit 2021-03-15T22:28:13.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: [INF] 2021-03-15T22:28:13.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/1174000013' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:13.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: audit 2021-03-15T22:28:13.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021 2021-03-15T22:28:13.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 03 2021-03-15T22:28:13.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 15 2021-03-15T22:28:13.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: T 2021-03-15T22:28:13.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 22 2021-03-15T22:28:13.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 28 2021-03-15T22:28:13.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 13 2021-03-15T22:28:13.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 001499 2021-03-15T22:28:13.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: +0000 2021-03-15T22:28:13.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:13.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ( 2021-03-15T22:28:13.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon 2021-03-15T22:28:13.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 0 2021-03-15T22:28:13.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ) 2021-03-15T22:28:13.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 1557 2021-03-15T22:28:13.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: audit 2021-03-15T22:28:13.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: [DBG] 2021-03-15T22:28:13.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/1581711008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:13.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: cluster 2021-03-15T22:28:13.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021 2021-03-15T22:28:13.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 03 2021-03-15T22:28:13.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 15 2021-03-15T22:28:13.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: T 2021-03-15T22:28:13.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 22 2021-03-15T22:28:13.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 28 2021-03-15T22:28:13.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 13 2021-03-15T22:28:13.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 489084 2021-03-15T22:28:13.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: +0000 2021-03-15T22:28:13.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:13.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ( 2021-03-15T22:28:13.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon 2021-03-15T22:28:13.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 0 2021-03-15T22:28:13.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ) 2021-03-15T22:28:13.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 1558 2021-03-15T22:28:13.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: cluster 2021-03-15T22:28:13.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: [INF] 2021-03-15T22:28:13.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:28:13.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: audit 2021-03-15T22:28:13.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021 2021-03-15T22:28:13.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 03 2021-03-15T22:28:13.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 15 2021-03-15T22:28:13.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: T 2021-03-15T22:28:13.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 22 2021-03-15T22:28:13.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.806 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 28 2021-03-15T22:28:13.806 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.806 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 13 2021-03-15T22:28:13.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 493453 2021-03-15T22:28:13.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: +0000 2021-03-15T22:28:13.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:13.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ( 2021-03-15T22:28:13.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon 2021-03-15T22:28:13.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 0 2021-03-15T22:28:13.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ) 2021-03-15T22:28:13.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 1559 2021-03-15T22:28:13.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: audit 2021-03-15T22:28:13.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: [INF] 2021-03-15T22:28:13.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/1174000013' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:13.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: cluster 2021-03-15T22:28:13.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021 2021-03-15T22:28:13.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 03 2021-03-15T22:28:13.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: - 2021-03-15T22:28:13.813 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 15 2021-03-15T22:28:13.813 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: T 2021-03-15T22:28:13.814 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 22 2021-03-15T22:28:13.814 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.814 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 28 2021-03-15T22:28:13.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 13 2021-03-15T22:28:13.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 493509 2021-03-15T22:28:13.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: +0000 2021-03-15T22:28:13.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:13.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ( 2021-03-15T22:28:13.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: mon 2021-03-15T22:28:13.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: . 2021-03-15T22:28:13.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 0 2021-03-15T22:28:13.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: ) 2021-03-15T22:28:13.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 1560 2021-03-15T22:28:13.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: : 2021-03-15T22:28:13.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: cluster 2021-03-15T22:28:13.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: [DBG] 2021-03-15T22:28:13.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: osdmap e554: 8 total, 6 up, 8 in 2021-03-15T22:28:13.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[56514]: 2021-03-15T22:28:13.820 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: cluster 2021-03-15T22:28:12.492530 2021-03-15T22:28:13.820 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1555 : cluster [DBG] osdmap e553: 8 total, 6 up, 8 in 2021-03-15T22:28:13.820 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:28:13.821 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: 15T22:28:12.963923+0000 mon.a (mon.0) 1556 : audit 2021-03-15T22:28:13.821 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/1174000013' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:13.821 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: audit 2021-03-15T22:28 2021-03-15T22:28:13.821 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: :13.001499+0000 mon.a (mon.0) 2021-03-15T22:28:13.821 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: 1557 : audit [DBG] from='client.? 172.21.15.76:0/1581711008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:13.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:28:13.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: 22:28:13.489084+0000 mon.a (mon 2021-03-15T22:28:13.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: .0) 1558 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:28:13.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:28:13.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: 03-15T22:28:13.493453+0000 2021-03-15T22:28:13.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: mon.a (mon.0) 1559 : audit [INF] 2021-03-15T22:28:13.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/1174000013' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:13.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: cluster 2021-03-15T22:28:13.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: 2021-03-15T22:28:13.493509+0000 2021-03-15T22:28:13.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: mon.a (mon.0) 1560 : cluster [DBG] 2021-03-15T22:28:13.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:13 smithi076 conmon[62156]: osdmap e554: 8 total, 6 up, 8 in 2021-03-15T22:28:13.825 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: cluster 2021-03-15T22:28:13.825 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.826 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021 2021-03-15T22:28:13.826 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.826 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 03 2021-03-15T22:28:13.827 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.828 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 15 2021-03-15T22:28:13.828 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: T 2021-03-15T22:28:13.828 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 22 2021-03-15T22:28:13.829 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.829 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 28 2021-03-15T22:28:13.829 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.830 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 12 2021-03-15T22:28:13.830 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.830 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 492530 2021-03-15T22:28:13.831 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: +0000 2021-03-15T22:28:13.831 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.831 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:13.832 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ( 2021-03-15T22:28:13.832 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon 2021-03-15T22:28:13.832 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.833 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 0 2021-03-15T22:28:13.833 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ) 2021-03-15T22:28:13.834 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 1555 2021-03-15T22:28:13.834 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.835 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: cluster 2021-03-15T22:28:13.836 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.836 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:13.836 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.836 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: osdmap e553: 8 total, 6 up, 8 in 2021-03-15T22:28:13.837 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.837 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: audit 2021-03-15T22:28:13.837 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.837 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021 2021-03-15T22:28:13.837 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.838 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 03 2021-03-15T22:28:13.838 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.838 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 15 2021-03-15T22:28:13.838 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: T 2021-03-15T22:28:13.838 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 22 2021-03-15T22:28:13.839 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.839 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 28 2021-03-15T22:28:13.839 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.839 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 12 2021-03-15T22:28:13.840 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.840 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 963923 2021-03-15T22:28:13.840 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: +0000 2021-03-15T22:28:13.841 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.841 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:13.841 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ( 2021-03-15T22:28:13.841 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon 2021-03-15T22:28:13.842 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.842 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 0 2021-03-15T22:28:13.842 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ) 2021-03-15T22:28:13.842 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 1556 2021-03-15T22:28:13.843 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.844 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: audit 2021-03-15T22:28:13.844 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.844 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: [INF] 2021-03-15T22:28:13.844 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.845 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1174000013' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:13.845 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.845 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: audit 2021-03-15T22:28:13.845 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.846 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021 2021-03-15T22:28:13.846 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.846 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 03 2021-03-15T22:28:13.846 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.847 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 15 2021-03-15T22:28:13.847 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: T 2021-03-15T22:28:13.847 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 22 2021-03-15T22:28:13.847 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.847 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 28 2021-03-15T22:28:13.848 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.848 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 13 2021-03-15T22:28:13.848 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.848 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 001499 2021-03-15T22:28:13.849 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: +0000 2021-03-15T22:28:13.849 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.849 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:13.850 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ( 2021-03-15T22:28:13.850 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon 2021-03-15T22:28:13.850 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.851 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 0 2021-03-15T22:28:13.851 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ) 2021-03-15T22:28:13.851 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 1557 2021-03-15T22:28:13.851 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.852 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: audit 2021-03-15T22:28:13.852 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.852 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:13.852 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.852 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1581711008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:13.853 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.853 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: cluster 2021-03-15T22:28:13.853 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.854 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021 2021-03-15T22:28:13.854 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.855 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 03 2021-03-15T22:28:13.855 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.855 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 15 2021-03-15T22:28:13.855 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: T 2021-03-15T22:28:13.855 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 22 2021-03-15T22:28:13.856 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.856 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 28 2021-03-15T22:28:13.856 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.856 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 13 2021-03-15T22:28:13.857 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.857 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 489084 2021-03-15T22:28:13.857 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: +0000 2021-03-15T22:28:13.857 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.857 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:13.858 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ( 2021-03-15T22:28:13.858 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon 2021-03-15T22:28:13.858 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.858 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 0 2021-03-15T22:28:13.858 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ) 2021-03-15T22:28:13.859 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 1558 2021-03-15T22:28:13.859 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.859 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: cluster 2021-03-15T22:28:13.859 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.860 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: [INF] 2021-03-15T22:28:13.860 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.860 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:28:13.860 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.860 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: audit 2021-03-15T22:28:13.861 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.861 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021 2021-03-15T22:28:13.861 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.861 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 03 2021-03-15T22:28:13.862 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.862 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 15 2021-03-15T22:28:13.862 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: T 2021-03-15T22:28:13.862 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 22 2021-03-15T22:28:13.862 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.863 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 28 2021-03-15T22:28:13.863 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.863 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 13 2021-03-15T22:28:13.863 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.864 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 493453 2021-03-15T22:28:13.864 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: +0000 2021-03-15T22:28:13.865 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.865 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:13.865 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ( 2021-03-15T22:28:13.865 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon 2021-03-15T22:28:13.865 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.866 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 0 2021-03-15T22:28:13.866 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ) 2021-03-15T22:28:13.866 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 1559 2021-03-15T22:28:13.866 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.867 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: audit 2021-03-15T22:28:13.867 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.867 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: [INF] 2021-03-15T22:28:13.867 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.867 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1174000013' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:13.868 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.868 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: cluster 2021-03-15T22:28:13.868 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.868 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021 2021-03-15T22:28:13.868 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.869 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 03 2021-03-15T22:28:13.869 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: - 2021-03-15T22:28:13.869 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 15 2021-03-15T22:28:13.869 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: T 2021-03-15T22:28:13.869 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 22 2021-03-15T22:28:13.870 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.870 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 28 2021-03-15T22:28:13.870 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.870 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 13 2021-03-15T22:28:13.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 493509 2021-03-15T22:28:13.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: +0000 2021-03-15T22:28:13.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:13.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ( 2021-03-15T22:28:13.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: mon 2021-03-15T22:28:13.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: . 2021-03-15T22:28:13.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 0 2021-03-15T22:28:13.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: ) 2021-03-15T22:28:13.873 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 1560 2021-03-15T22:28:13.873 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: : 2021-03-15T22:28:13.873 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: cluster 2021-03-15T22:28:13.873 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.874 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:13.874 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.875 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: osdmap e554: 8 total, 6 up, 8 in 2021-03-15T22:28:13.875 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:13 smithi125 conmon[56739]: 2021-03-15T22:28:13.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:13.877 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-15T22:28:13.939 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:13.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:13.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:14.042 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:14.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:14.065 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:14.065 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:14.166 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:14.190 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:14.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:14.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:14.192 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:28:14.293 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:14.315 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:14.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:14.316 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:14.339 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:28:14.339 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:28:14.417 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:14.439 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:14.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:14.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:14.541 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:14.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: cluster 2021-03-15T22:28:12 2021-03-15T22:28:14.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: .904337+0000 osd.1 (osd.1) 106 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:14.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:28:14.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: :28:12.905185+0000 osd.1 (osd. 2021-03-15T22:28:14.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: 1) 107 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:14.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:28:14.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: -03-15T22:28:13.101035+0000 2021-03-15T22:28:14.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: mgr.y (mgr.14144) 747 : cluster [DBG] 2021-03-15T22:28:14.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: pgmap v969: 138 pgs: 5 remapped+peering, 12 peering, 4 undersized+degraded+peered, 27 active+undersized+degraded, 11 undersized+peered, 38 active+undersized, 41 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s; 397/1743 objects degraded (22.777%) 2021-03-15T22:28:14.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:28:14.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: 28:13.496644+0000 osd.4 (osd.4 2021-03-15T22:28:14.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: ) 119 : cluster [DBG] 5.e starting backfill to osd.0 from (0'0,0'0] MAX to 491'8 2021-03-15T22:28:14.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:28:14.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: 03-15T22:28:13.502029+0000 2021-03-15T22:28:14.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: mon.a (mon.0) 1561 : cluster [WRN] 2021-03-15T22:28:14.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: Health check update: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:14.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:28:14.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: 28:13 2021-03-15T22:28:14.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: .502069+0000 mon.a 2021-03-15T22:28:14.575 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[62156]: (mon.0) 1562 : cluster [WRN] Health check update: Degraded data redundancy: 397/1743 objects degraded (22.777%), 31 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:14.576 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:14.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:14.577 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[56514]: cluster 2021-03-15T22:28:12.904337+0000 osd.1 (osd.1) 2021-03-15T22:28:14.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[56514]: 106 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:14.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[56514]: cluster 2021-03-15T22:28:12.905185+0000 osd.1 (osd.1) 107 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:14.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[56514]: cluster 2021-03-15T22:28:13.101035+0000 mgr.y (mgr.14144) 747 : cluster [DBG] pgmap v969: 138 pgs: 5 remapped+peering, 12 peering, 4 undersized+degraded+peered, 27 active+undersized+degraded, 11 undersized+peered, 38 active+undersized, 41 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s; 397/1743 objects degraded (22.777%) 2021-03-15T22:28:14.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[56514]: cluster 2021-03-15T22:28:13.496644+0000 osd.4 (osd.4) 119 : cluster [DBG] 5.e starting backfill to osd.0 from (0'0,0'0] MAX to 491'8 2021-03-15T22:28:14.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[56514]: cluster 2021-03-15T22:28:13.502029+0000 mon.a (mon.0) 1561 : cluster [WRN] Health check update: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:14.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:28:14.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:14 smithi076 conmon[56514]: 28:13.502069+0000 mon.a (mon.0) 1562 : cluster [WRN] Health check update: Degraded data redundancy: 397/1743 objects degraded (22.777%), 31 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:14.669 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:14 smithi125 conmon[56739]: cluster 2021-03-15T22:28:12.904337+0000 2021-03-15T22:28:14.670 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:14 smithi125 conmon[56739]: osd.1 (osd.1) 106 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:14.670 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:14 smithi125 conmon[56739]: cluster 2021-03-15T22:28:12.905185+0000 osd.1 (osd.1) 107 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:14.670 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:14 smithi125 conmon[56739]: cluster 2021-03-15T22:28:13.101035+0000 mgr.y (mgr.14144) 747 : cluster [DBG] pgmap v969: 138 pgs: 5 remapped+peering, 12 peering, 4 undersized+degraded+peered, 27 active+undersized+degraded, 11 undersized+peered, 38 active+undersized, 41 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.8 KiB/s rd, 0 B/s wr, 2 op/s; 397/1743 objects degraded (22.777%) 2021-03-15T22:28:14.671 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:14 smithi125 conmon[56739]: cluster 2021-03-15T22:28:13.496644+0000 osd.4 (osd.4) 119 : cluster [DBG] 5.e starting backfill to osd.0 from (0'0,0'0] MAX to 491'8 2021-03-15T22:28:14.671 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:14 smithi125 conmon[56739]: cluster 2021-03-15T22:28:13.502029+0000 mon.a (mon.0) 1561 : cluster [WRN] Health check update: Reduced data availability: 5 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:14.671 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:14 smithi125 conmon[56739]: cluster 2021-03-15T22:28:13.502069+0000 mon.a (mon.0) 1562 : cluster [WRN] Health check update: Degraded data redundancy: 397/1743 objects degraded (22.777%), 31 pgs degraded, 31 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:14.678 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:14.702 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:14.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:14.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:14.804 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:14.829 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:14.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:14.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:14.954 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:14.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:14.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:15.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:15.082 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:15.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:15.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:15.184 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:15.219 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:15.219 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:15.320 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:15.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:15.350 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:15.350 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:15.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:15.474 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:15.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:15.476 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:15.577 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:15.600 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:15.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:15.602 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:15.681 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:15 smithi125 conmon[56739]: cluster 2021-03-15T22:28:14.520422+0000 mon.a (mon.0) 1563 : cluster [DBG] osdmap e555: 8 total, 6 up, 8 in 2021-03-15T22:28:15.681 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:15 smithi125 conmon[56739]: cluster 2021-03-15T22:28:15.162108+0000 2021-03-15T22:28:15.682 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:15 smithi125 conmon[56739]: mon.a (mon.0) 1564 : cluster [DBG] osdmap e556: 8 total, 6 up, 8 in 2021-03-15T22:28:15.704 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:15.727 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:15.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:15.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:15.829 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:15.840 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:28:15.841 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:28:15.851 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:15.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:15.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:15.954 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:15.975 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:15.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:15.976 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:16.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:15 smithi076 conmon[56514]: cluster 2021-03-15T22:28:14.520422+0000 2021-03-15T22:28:16.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:15 smithi076 conmon[56514]: mon.a (mon.0) 1563 : cluster [DBG] osdmap e555: 8 total, 6 up, 8 in 2021-03-15T22:28:16.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:15 smithi076 conmon[56514]: cluster 2021-03-15T22:28:15.162108+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e556: 8 total, 6 up, 8 in 2021-03-15T22:28:16.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:15 smithi076 conmon[62156]: cluster 2021-03-15T22:28:14.520422+0000 mon.a (mon.0) 1563 2021-03-15T22:28:16.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:15 smithi076 conmon[62156]: : cluster [DBG] osdmap e555: 8 total, 6 up, 8 in 2021-03-15T22:28:16.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:15 smithi076 conmon[62156]: cluster 2021-03-15T22:28:15.162108+0000 mon.a (mon.0) 1564 : cluster [DBG] osdmap e556: 8 total, 6 up, 8 in 2021-03-15T22:28:16.077 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:16.099 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:16.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:16.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:16.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:16.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:16.234 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:16.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:16.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:16.340 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:28:16.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:16.359 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:16.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:16.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:16.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:16.483 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:16.484 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:16.540 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-15T22:28:16.585 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:16.603 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:28:16.607 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:16.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:16.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:16.630 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:16 smithi076 conmon[62156]: cluster 2021-03-15T22:28: 2021-03-15T22:28:16.631 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:16 smithi076 conmon[62156]: 15.101629+0000 mgr.y (mgr.14144) 748 : cluster [DBG] pgmap v972: 138 pgs: 5 remapped+peering, 12 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 undersized+degraded+peered, 26 active+undersized+degraded, 11 undersized+peered, 38 active+undersized, 41 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s; 397/1743 objects degraded (22.777%); 32/1743 objects misplaced (1.836%); 120 B/s, 0 objects/s recovering 2021-03-15T22:28:16.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:16 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:28:16.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:16 smithi076 conmon[62156]: 22:28:16.165479+0000 mon.a (mon 2021-03-15T22:28:16.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:16 smithi076 conmon[62156]: .0) 1565 : cluster [DBG] osdmap e557: 8 total, 6 up, 8 in 2021-03-15T22:28:16.633 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:16 smithi076 conmon[56514]: cluster 2021-03-15T22:28:16.634 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:16 smithi076 conmon[56514]: 2021-03-15T22:28 2021-03-15T22:28:16.634 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:16 smithi076 conmon[56514]: :15.101629+0000 mgr.y (mgr.14144) 748 : cluster [DBG] pgmap v972: 138 pgs: 5 remapped+peering, 12 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 undersized+degraded+peered, 26 active+undersized+degraded, 11 undersized+peered, 38 active+undersized, 41 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s; 397/1743 objects degraded (22.777%); 32/1743 objects misplaced (1.836%); 120 B/s, 0 objects/s recovering 2021-03-15T22:28:16.635 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:16 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:28:16.635 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:16 smithi076 conmon[56514]: T22:28:16.165479+0000 mon.a (mon 2021-03-15T22:28:16.636 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:16 smithi076 conmon[56514]: .0) 1565 : cluster [DBG] osdmap e557: 8 total, 6 up, 8 in 2021-03-15T22:28:16.652 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:16.710 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:16.740 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:16.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:16.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:16 smithi125 conmon[56739]: cluster 2021-03-15T22:28:15.101629+0000 mgr.y (mgr.14144) 748 : cluster [DBG] pgmap v972: 138 pgs: 5 remapped+peering, 12 peering, 1 active+recovery_wait+undersized+degraded+remapped, 4 undersized+degraded+peered, 26 active+undersized+degraded, 11 undersized+peered, 38 active+undersized, 41 active+clean; 568 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 3.2 KiB/s rd, 0 B/s wr, 4 op/s; 397/1743 objects degraded (22.777%); 32/1743 objects misplaced (1.836%); 120 B/s, 0 objects/s recovering 2021-03-15T22:28:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:16 smithi125 conmon[56739]: cluster 2021-03-15T22:28:16.165479+0000 mon.a (mon.0) 1565 : cluster [DBG] osdmap e557: 8 total, 6 up, 8 in 2021-03-15T22:28:16.818 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:28:16.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:16.865 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:16.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:16.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:16.968 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:16.991 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:16.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:16.992 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:17.094 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:17.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:17.120 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:17.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:17.172 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:17.222 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:17.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:17.254 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:17.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:17.356 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:17.383 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:17.384 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:17.483 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3, 0] out_osds: [] dead_osds: [2, 5] live_osds: [0, 1, 3, 4, 6, 7] 2021-03-15T22:28:17.483 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:28:17.484 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump -f json-pretty 2021-03-15T22:28:17.485 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:17.509 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:17.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:17.611 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:17.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:17.634 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:17.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:17.736 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:17.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:17.759 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:17.760 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:17.861 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:17.886 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:17.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:17.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:17.985 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:17.990 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:18.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:18.014 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:18.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:18.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:18.140 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:18.142 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:18.241 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[62156]: cluster 2021-03-15T22:28:17.102298+0000 mgr.y (mgr.14144) 749 : cluster [DBG] pgmap v975: 138 pgs: 7 active+clean+remapped, 5 remapped+peering, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 1 active+recovering+undersized+remapped, 6 undersized+peered, 37 active+undersized, 2 active+recovering+degraded+remapped, 51 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 118 op/s; 374/1743 objects degraded (21.457%); 60/1743 objects misplaced (3.442%); 8.3 MiB/s, 40 objects/s recovering 2021-03-15T22:28:18.242 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[62156]: cluster 2021-03-15T22:28:17.164641+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e558: 8 total, 6 up, 8 in 2021-03-15T22:28:18.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: cluster 2021-03-15T22:28:18.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021-03-15T22:28:18.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021 2021-03-15T22:28:18.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: - 2021-03-15T22:28:18.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 03 2021-03-15T22:28:18.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: - 2021-03-15T22:28:18.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 15 2021-03-15T22:28:18.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: T 2021-03-15T22:28:18.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 22 2021-03-15T22:28:18.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: : 2021-03-15T22:28:18.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 28 2021-03-15T22:28:18.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: : 2021-03-15T22:28:18.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 17 2021-03-15T22:28:18.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: . 2021-03-15T22:28:18.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 102298 2021-03-15T22:28:18.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: +0000 2021-03-15T22:28:18.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021-03-15T22:28:18.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: mgr.y 2021-03-15T22:28:18.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: ( 2021-03-15T22:28:18.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: mgr 2021-03-15T22:28:18.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: . 2021-03-15T22:28:18.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 14144 2021-03-15T22:28:18.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: ) 2021-03-15T22:28:18.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 749 2021-03-15T22:28:18.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: : 2021-03-15T22:28:18.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: cluster 2021-03-15T22:28:18.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021-03-15T22:28:18.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: [DBG] 2021-03-15T22:28:18.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021-03-15T22:28:18.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: pgmap v975: 138 pgs: 7 active+clean+remapped, 5 remapped+peering, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 1 active+recovering+undersized+remapped, 6 undersized+peered, 37 active+undersized, 2 active+recovering+degraded+remapped, 51 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 118 op/s; 374/1743 objects degraded (21.457%); 60/1743 objects misplaced (3.442%); 8.3 MiB/s, 40 objects/s recovering 2021-03-15T22:28:18.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021-03-15T22:28:18.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: cluster 2021-03-15T22:28:18.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021-03-15T22:28:18.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021 2021-03-15T22:28:18.251 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: - 2021-03-15T22:28:18.251 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 03 2021-03-15T22:28:18.251 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: - 2021-03-15T22:28:18.251 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 15 2021-03-15T22:28:18.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: T 2021-03-15T22:28:18.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 22 2021-03-15T22:28:18.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: : 2021-03-15T22:28:18.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 28 2021-03-15T22:28:18.253 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: : 2021-03-15T22:28:18.253 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 17 2021-03-15T22:28:18.253 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: . 2021-03-15T22:28:18.253 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 164641 2021-03-15T22:28:18.254 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: +0000 2021-03-15T22:28:18.254 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021-03-15T22:28:18.254 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:18.254 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: ( 2021-03-15T22:28:18.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: mon 2021-03-15T22:28:18.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: . 2021-03-15T22:28:18.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 0 2021-03-15T22:28:18.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: ) 2021-03-15T22:28:18.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 1566 2021-03-15T22:28:18.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: : 2021-03-15T22:28:18.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: cluster 2021-03-15T22:28:18.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021-03-15T22:28:18.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: [DBG] 2021-03-15T22:28:18.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021-03-15T22:28:18.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: osdmap e558: 8 total, 6 up, 8 in 2021-03-15T22:28:18.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:18 smithi076 conmon[56514]: 2021-03-15T22:28:18.258 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:18.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:18.283 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:18.283 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:18.332 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:28:18.333 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:28:18.384 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:18.407 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:18 smithi125 conmon[56739]: cluster 2021-03-15T22:28:17.102298+0000 mgr.y ( 2021-03-15T22:28:18.407 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:18 smithi125 conmon[56739]: mgr.14144) 749 : cluster [DBG] pgmap v975: 138 pgs: 7 active+clean+remapped, 5 remapped+peering, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 1 active+recovering+undersized+remapped, 6 undersized+peered, 37 active+undersized, 2 active+recovering+degraded+remapped, 51 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 118 op/s; 374/1743 objects degraded (21.457%); 60/1743 objects misplaced (3.442%); 8.3 MiB/s, 40 objects/s recovering 2021-03-15T22:28:18.408 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:18 smithi125 conmon[56739]: cluster 2021-03-15T22:28:17.164641+0000 mon.a (mon.0) 1566 : cluster [DBG] osdmap e558: 8 total, 6 up, 8 in 2021-03-15T22:28:18.409 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:18.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:18.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:18.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:18.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:18.536 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:18.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:18.637 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:18.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:18.661 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:18.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:18.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:18.784 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:18.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:18.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:18.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:18.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:18.909 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:18.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:18.939 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-15T22:28:19.011 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:19.035 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:19.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:19.036 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:19.137 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:19.160 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:19.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:19.162 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:19.173 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:19 smithi125 conmon[56739]: audit 2021-03-15T22:28:19.070291+0000 mon.a (mon.0) 1567 : audit [DBG] from='client.? 172.21.15.76:0/3407209325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:19.263 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:19.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: audit 2021-03-15T22:28:19.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: 2021-03-15T22:28:19.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: 2021 2021-03-15T22:28:19.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: - 2021-03-15T22:28:19.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: 03 2021-03-15T22:28:19.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: - 2021-03-15T22:28:19.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: 15 2021-03-15T22:28:19.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: T 2021-03-15T22:28:19.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: 22 2021-03-15T22:28:19.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: : 2021-03-15T22:28:19.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: 28:19.070291+0000 mon.a ( 2021-03-15T22:28:19.271 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[56514]: mon.0) 1567 : audit [DBG] from='client.? 172.21.15.76:0/3407209325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:19.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[62156]: audit 2021-03-15T22:28:19.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:28:19.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:19 smithi076 conmon[62156]: 28:19.070291+0000 mon.a (mon.0) 1567 : audit [DBG] from='client.? 172.21.15.76:0/3407209325' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:19.285 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:19.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:19.331 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:28:19.388 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:19.402 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:28:19.402 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:28:19.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:28:19.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:28:19.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:28:19.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:28:19.404 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:28:19.404 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:28:19.404 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:28:19.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:28:19.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:28:19.405 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:28:19.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:28:19.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:28:19.415 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:19.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:19.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:19.518 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:19.541 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:19.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:19.543 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:19.644 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:19.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:19.667 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:19.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:19.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:19.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:19.792 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:19.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:19.894 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:19.917 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:19.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:19.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:20.019 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:20.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:20.043 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:20.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:20.145 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:20.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:20.172 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:20.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:20.190 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: cluster 2021-03-15T22:28:19.103178+0000 mgr.y (mgr. 2021-03-15T22:28:20.190 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: 14144) 750 : cluster [DBG] pgmap v977: 138 pgs: 10 active+clean+remapped, 2 remapped+peering, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 1 active+recovering+undersized+remapped, 6 undersized+peered, 37 active+undersized, 2 active+recovering+degraded+remapped, 51 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 121 op/s; 374/1743 objects degraded (21.457%); 68/1743 objects misplaced (3.901%); 7.2 MiB/s, 36 objects/s recovering 2021-03-15T22:28:20.191 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: cluster 2021-03-15T22:28:19.173962+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e559: 8 total, 6 up, 8 in 2021-03-15T22:28:20.191 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: audit 2021-03-15T22:28:19.872611+0000 mgr.y (mgr.14144) 751 2021-03-15T22:28:20.191 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:20.191 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:28:20.191 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: 22:28:20.157753+0000 mon.a (mon.0) 1569 : cluster 2021-03-15T22:28:20.192 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: [WRN] Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:20.192 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:28:20.192 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: :28:20.157786+0000 mon.a (mon.0) 1570 : cluster 2021-03-15T22:28:20.192 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: [WRN] Health check update: Degraded data redundancy: 374/1743 objects degraded (21.457%), 28 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:20.193 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:28:20.193 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: 28:20.165170+0000 mon.a (mon.0) 1571 : cluster [DBG] 2021-03-15T22:28:20.193 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:20 smithi125 conmon[56739]: osdmap e560: 8 total, 6 up, 8 in 2021-03-15T22:28:20.193 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:28:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:28:20.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:20.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:20.303 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:20.303 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:20.398 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:28:20.399 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:28:20.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:20.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:20.430 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:20.430 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:20.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[56514]: cluster 2021-03-15T22:28:19.103178+0000 mgr.y (mgr.14144) 750 2021-03-15T22:28:20.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[56514]: : cluster [DBG] pgmap v977: 138 pgs: 10 active+clean+remapped, 2 remapped+peering, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 1 active+recovering+undersized+remapped, 6 undersized+peered, 37 active+undersized, 2 active+recovering+degraded+remapped, 51 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 121 op/s; 374/1743 objects degraded (21.457%); 68/1743 objects misplaced (3.901%); 7.2 MiB/s, 36 objects/s recovering 2021-03-15T22:28:20.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[56514]: cluster 2021-03-15T22:28:19.173962+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e559: 8 total, 6 up, 8 in 2021-03-15T22:28:20.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[56514]: audit 2021-03-15T22:28:19.872611+0000 mgr.y (mgr.14144) 751 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:20.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[56514]: cluster 2021-03-15T22:28:20.157753+0000 mon.a (mon.0) 1569 : cluster [WRN] Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:20.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[56514]: cluster 2021-03-15T22:28:20.157786+0000 mon.a (mon 2021-03-15T22:28:20.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[56514]: .0) 1570 : cluster [WRN] Health check update: Degraded data redundancy: 374/1743 objects degraded (21.457%), 28 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:20.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[56514]: cluster 2021-03-15T22:28:20.165170+0000 mon.a (mon.0) 1571 2021-03-15T22:28:20.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[56514]: : cluster [DBG] osdmap e560: 8 total, 6 up, 8 in 2021-03-15T22:28:20.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[62156]: cluster 2021-03-15T22:28:19.103178+0000 mgr.y (mgr.14144) 750 : cluster [DBG] pgmap v977: 138 pgs: 10 active+clean+remapped, 2 remapped+peering, 1 peering, 1 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 1 active+recovering+undersized+remapped, 6 undersized+peered, 37 active+undersized, 2 active+recovering+degraded+remapped, 51 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 121 op/s; 374/1743 objects degraded (21.457%); 68/1743 objects misplaced (3.901%); 7.2 MiB/s, 36 objects/s recovering 2021-03-15T22:28:20.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[62156]: cluster 2021-03-15T22:28:19.173962+0000 mon.a (mon.0) 1568 : cluster [DBG] osdmap e559: 8 total, 6 up, 8 in 2021-03-15T22:28:20.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[62156]: audit 2021-03-15T22:28:19.872611+0000 mgr.y (mgr.14144) 751 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:20.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[62156]: cluster 2021-03-15T22:28:20.157753+0000 mon.a (mon.0) 1569 : cluster [WRN] Health check update: Reduced data availability: 2 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:20.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[62156]: cluster 2021-03-15T22:28:20.157786+0000 mon.a (mon.0) 1570 : cluster [WRN] Health check update: Degraded data redundancy: 374/1743 objects degraded (21.457%), 28 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:20.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:20 smithi076 conmon[62156]: cluster 2021-03-15T22:28:20.165170+0000 mon.a (mon.0) 1571 : cluster [DBG] osdmap e560: 8 total, 6 up, 8 in 2021-03-15T22:28:20.532 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:20.553 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:20.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:20.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:20.656 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:20.702 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:20.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:20.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:20.805 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:20.832 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:20.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:20.884 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:28:20.885 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:28:20.886 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:28:20.887 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:28:20.888 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:28:20.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:20.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:20.961 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:20.962 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:21.063 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:21.088 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:21.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:21.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:21.183 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:28:21.191 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:21.225 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:21.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:21.226 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:21.327 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:21.357 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:21.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:21.460 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:21.485 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:21.485 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:21.500 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: audit 2021-03-15T22:28:21.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021 2021-03-15T22:28:21.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: - 2021-03-15T22:28:21.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 03 2021-03-15T22:28:21.502 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: - 2021-03-15T22:28:21.502 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 15 2021-03-15T22:28:21.502 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: T 2021-03-15T22:28:21.503 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 22 2021-03-15T22:28:21.503 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: : 2021-03-15T22:28:21.503 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 28 2021-03-15T22:28:21.503 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: : 2021-03-15T22:28:21.503 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 20 2021-03-15T22:28:21.504 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: . 2021-03-15T22:28:21.504 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 464103 2021-03-15T22:28:21.504 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: +0000 2021-03-15T22:28:21.504 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.505 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:21.505 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: ( 2021-03-15T22:28:21.505 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: mon 2021-03-15T22:28:21.505 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: . 2021-03-15T22:28:21.506 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 0 2021-03-15T22:28:21.506 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: ) 2021-03-15T22:28:21.506 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 1572 2021-03-15T22:28:21.507 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: : 2021-03-15T22:28:21.507 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: audit 2021-03-15T22:28:21.508 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.508 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: [INF] 2021-03-15T22:28:21.509 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.509 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2312258877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:28:21.509 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.510 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: cluster 2021-03-15T22:28:21.510 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.511 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021 2021-03-15T22:28:21.511 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: - 2021-03-15T22:28:21.512 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 03 2021-03-15T22:28:21.512 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: - 2021-03-15T22:28:21.513 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 15 2021-03-15T22:28:21.513 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: T 2021-03-15T22:28:21.513 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 22 2021-03-15T22:28:21.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: : 2021-03-15T22:28:21.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 28 2021-03-15T22:28:21.515 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: : 2021-03-15T22:28:21.515 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 21 2021-03-15T22:28:21.516 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: . 2021-03-15T22:28:21.516 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 160210 2021-03-15T22:28:21.516 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: +0000 2021-03-15T22:28:21.517 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.517 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:21.518 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: ( 2021-03-15T22:28:21.518 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: mon 2021-03-15T22:28:21.518 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: . 2021-03-15T22:28:21.519 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 0 2021-03-15T22:28:21.519 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: ) 2021-03-15T22:28:21.520 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 1573 2021-03-15T22:28:21.520 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: : 2021-03-15T22:28:21.521 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: cluster 2021-03-15T22:28:21.521 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.521 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: [WRN] 2021-03-15T22:28:21.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:21.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.523 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: audit 2021-03-15T22:28:21.523 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.523 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021 2021-03-15T22:28:21.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: - 2021-03-15T22:28:21.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 03 2021-03-15T22:28:21.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: - 2021-03-15T22:28:21.525 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 15 2021-03-15T22:28:21.525 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: T 2021-03-15T22:28:21.525 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 22 2021-03-15T22:28:21.526 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: : 2021-03-15T22:28:21.526 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 28 2021-03-15T22:28:21.526 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: : 2021-03-15T22:28:21.527 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 21 2021-03-15T22:28:21.527 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: . 2021-03-15T22:28:21.527 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 164842 2021-03-15T22:28:21.527 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: +0000 2021-03-15T22:28:21.528 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.528 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:21.528 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: ( 2021-03-15T22:28:21.528 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: mon 2021-03-15T22:28:21.529 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: . 2021-03-15T22:28:21.529 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 0 2021-03-15T22:28:21.529 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: ) 2021-03-15T22:28:21.530 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 1574 2021-03-15T22:28:21.530 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: : 2021-03-15T22:28:21.530 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: audit 2021-03-15T22:28:21.531 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.531 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: [INF] 2021-03-15T22:28:21.531 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.532 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2312258877' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:28:21.532 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.533 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: cluster 2021-03-15T22:28:21.533 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021-03-15T22:28:21.533 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 2021 2021-03-15T22:28:21.534 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: - 2021-03-15T22:28:21.534 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 03 2021-03-15T22:28:21.534 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: - 2021-03-15T22:28:21.535 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 15 2021-03-15T22:28:21.535 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: T 2021-03-15T22:28:21.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:21 smithi125 conmon[56739]: 22:28:21.164903+0000 mon.a (mon.0) 1575 : cluster [DBG] osdmap e561: 8 total, 6 up, 8 in 2021-03-15T22:28:21.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: audit 2021-03-15T22:28:21.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021 2021-03-15T22:28:21.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: - 2021-03-15T22:28:21.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 03 2021-03-15T22:28:21.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: - 2021-03-15T22:28:21.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 15 2021-03-15T22:28:21.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: T 2021-03-15T22:28:21.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 22 2021-03-15T22:28:21.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 28 2021-03-15T22:28:21.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 20 2021-03-15T22:28:21.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: . 2021-03-15T22:28:21.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 464103 2021-03-15T22:28:21.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: +0000 2021-03-15T22:28:21.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:21.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: ( 2021-03-15T22:28:21.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: mon 2021-03-15T22:28:21.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: . 2021-03-15T22:28:21.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 0 2021-03-15T22:28:21.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: ) 2021-03-15T22:28:21.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 1572 2021-03-15T22:28:21.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: audit 2021-03-15T22:28:21.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: [INF] 2021-03-15T22:28:21.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/2312258877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:28:21.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: cluster 2021-03-15T22:28:21.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.552 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021 2021-03-15T22:28:21.552 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: - 2021-03-15T22:28:21.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 03 2021-03-15T22:28:21.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: - 2021-03-15T22:28:21.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 15 2021-03-15T22:28:21.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: T 2021-03-15T22:28:21.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 22 2021-03-15T22:28:21.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 28 2021-03-15T22:28:21.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 21 2021-03-15T22:28:21.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: . 2021-03-15T22:28:21.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 160210 2021-03-15T22:28:21.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: +0000 2021-03-15T22:28:21.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:21.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: ( 2021-03-15T22:28:21.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: mon 2021-03-15T22:28:21.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: . 2021-03-15T22:28:21.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 0 2021-03-15T22:28:21.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: ) 2021-03-15T22:28:21.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 1573 2021-03-15T22:28:21.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: cluster 2021-03-15T22:28:21.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: [WRN] 2021-03-15T22:28:21.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.564 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:21.564 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.565 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: audit 2021-03-15T22:28:21.565 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.565 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021 2021-03-15T22:28:21.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: - 2021-03-15T22:28:21.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 03 2021-03-15T22:28:21.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: - 2021-03-15T22:28:21.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 15 2021-03-15T22:28:21.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: T 2021-03-15T22:28:21.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 22 2021-03-15T22:28:21.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 28 2021-03-15T22:28:21.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 21 2021-03-15T22:28:21.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: . 2021-03-15T22:28:21.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 164842 2021-03-15T22:28:21.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: +0000 2021-03-15T22:28:21.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:21.573 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: ( 2021-03-15T22:28:21.573 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: mon 2021-03-15T22:28:21.574 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: . 2021-03-15T22:28:21.574 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 0 2021-03-15T22:28:21.574 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: ) 2021-03-15T22:28:21.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 1574 2021-03-15T22:28:21.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: audit 2021-03-15T22:28:21.576 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.577 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: [INF] 2021-03-15T22:28:21.577 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.577 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/2312258877' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:28:21.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: cluster 2021-03-15T22:28:21.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021 2021-03-15T22:28:21.580 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: - 2021-03-15T22:28:21.580 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 03 2021-03-15T22:28:21.580 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: - 2021-03-15T22:28:21.580 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 15 2021-03-15T22:28:21.581 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: T 2021-03-15T22:28:21.581 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 22 2021-03-15T22:28:21.581 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.581 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 28 2021-03-15T22:28:21.581 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.582 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 21 2021-03-15T22:28:21.582 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: . 2021-03-15T22:28:21.582 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 164903 2021-03-15T22:28:21.583 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: +0000 2021-03-15T22:28:21.583 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.584 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:21.584 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: ( 2021-03-15T22:28:21.584 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: mon 2021-03-15T22:28:21.584 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: . 2021-03-15T22:28:21.585 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 0 2021-03-15T22:28:21.585 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: ) 2021-03-15T22:28:21.585 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 1575 2021-03-15T22:28:21.585 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: : 2021-03-15T22:28:21.585 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: cluster 2021-03-15T22:28:21.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: [DBG] 2021-03-15T22:28:21.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: osdmap e561: 8 total, 6 up, 8 in 2021-03-15T22:28:21.586 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[56514]: 2021-03-15T22:28:21.587 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: audit 2021-03-15T22:28:21.588 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.588 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021 2021-03-15T22:28:21.588 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: - 2021-03-15T22:28:21.588 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 03 2021-03-15T22:28:21.589 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: - 2021-03-15T22:28:21.589 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 15 2021-03-15T22:28:21.589 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: T 2021-03-15T22:28:21.589 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 22 2021-03-15T22:28:21.589 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 28 2021-03-15T22:28:21.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 20 2021-03-15T22:28:21.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: . 2021-03-15T22:28:21.591 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 464103 2021-03-15T22:28:21.591 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: +0000 2021-03-15T22:28:21.591 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.591 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: mon.a 2021-03-15T22:28:21.591 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: ( 2021-03-15T22:28:21.592 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: mon 2021-03-15T22:28:21.592 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: . 2021-03-15T22:28:21.592 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 0 2021-03-15T22:28:21.592 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: ) 2021-03-15T22:28:21.592 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 1572 2021-03-15T22:28:21.593 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.593 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: audit 2021-03-15T22:28:21.593 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.593 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: [INF] 2021-03-15T22:28:21.593 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.594 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2312258877' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:28:21.594 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.594 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: cluster 2021-03-15T22:28:21.594 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.595 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021 2021-03-15T22:28:21.595 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: - 2021-03-15T22:28:21.595 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 03 2021-03-15T22:28:21.595 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: - 2021-03-15T22:28:21.595 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 15 2021-03-15T22:28:21.596 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: T 2021-03-15T22:28:21.596 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 22 2021-03-15T22:28:21.596 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.596 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 28 2021-03-15T22:28:21.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 21 2021-03-15T22:28:21.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: . 2021-03-15T22:28:21.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 160210 2021-03-15T22:28:21.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: +0000 2021-03-15T22:28:21.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: mon.a 2021-03-15T22:28:21.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: ( 2021-03-15T22:28:21.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: mon 2021-03-15T22:28:21.599 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: . 2021-03-15T22:28:21.599 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 0 2021-03-15T22:28:21.599 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: ) 2021-03-15T22:28:21.599 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 1573 2021-03-15T22:28:21.599 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: cluster 2021-03-15T22:28:21.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: [WRN] 2021-03-15T22:28:21.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:21.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: audit 2021-03-15T22:28:21.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021 2021-03-15T22:28:21.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: - 2021-03-15T22:28:21.602 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 03 2021-03-15T22:28:21.602 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: - 2021-03-15T22:28:21.602 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 15 2021-03-15T22:28:21.602 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: T 2021-03-15T22:28:21.603 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 22 2021-03-15T22:28:21.603 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.603 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 28 2021-03-15T22:28:21.603 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.603 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 21 2021-03-15T22:28:21.604 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: . 2021-03-15T22:28:21.604 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 164842 2021-03-15T22:28:21.604 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: +0000 2021-03-15T22:28:21.604 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.605 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: mon.a 2021-03-15T22:28:21.605 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: ( 2021-03-15T22:28:21.605 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: mon 2021-03-15T22:28:21.605 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: . 2021-03-15T22:28:21.605 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 0 2021-03-15T22:28:21.606 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: ) 2021-03-15T22:28:21.606 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 1574 2021-03-15T22:28:21.606 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.606 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: audit 2021-03-15T22:28:21.607 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.607 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: [INF] 2021-03-15T22:28:21.607 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.607 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2312258877' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:28:21.608 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.608 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: cluster 2021-03-15T22:28:21.608 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.608 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021 2021-03-15T22:28:21.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: - 2021-03-15T22:28:21.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 03 2021-03-15T22:28:21.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: - 2021-03-15T22:28:21.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 15 2021-03-15T22:28:21.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: T 2021-03-15T22:28:21.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 22 2021-03-15T22:28:21.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 28 2021-03-15T22:28:21.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 21 2021-03-15T22:28:21.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: . 2021-03-15T22:28:21.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 164903 2021-03-15T22:28:21.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: +0000 2021-03-15T22:28:21.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.612 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: mon.a 2021-03-15T22:28:21.612 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: ( 2021-03-15T22:28:21.612 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: mon 2021-03-15T22:28:21.612 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: . 2021-03-15T22:28:21.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 0 2021-03-15T22:28:21.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: ) 2021-03-15T22:28:21.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 1575 2021-03-15T22:28:21.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: : 2021-03-15T22:28:21.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: cluster 2021-03-15T22:28:21.614 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.614 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: [DBG] 2021-03-15T22:28:21.614 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.614 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: osdmap e561: 8 total, 6 up, 8 in 2021-03-15T22:28:21.615 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:21 smithi076 conmon[62156]: 2021-03-15T22:28:21.616 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-15T22:28:21.617 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:28:21.617 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:28:21.618 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 561, 2021-03-15T22:28:21.618 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:28:21.618 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2021-03-15T22:13:20.370795+0000", 2021-03-15T22:28:21.618 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2021-03-15T22:28:21.159996+0000", 2021-03-15T22:28:21.618 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2021-03-15T22:27:35.183905+0000", 2021-03-15T22:28:21.619 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2021-03-15T22:28:11.480156+0000", 2021-03-15T22:28:21.619 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-03-15T22:28:21.619 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5801984, 2021-03-15T22:28:21.619 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2021-03-15T22:28:21.620 INFO:teuthology.orchestra.run.smithi076.stdout: "noscrub", 2021-03-15T22:28:21.620 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2021-03-15T22:28:21.620 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2021-03-15T22:28:21.620 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2021-03-15T22:28:21.620 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2021-03-15T22:28:21.621 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:21.621 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 18, 2021-03-15T22:28:21.621 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2021-03-15T22:28:21.621 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-03-15T22:28:21.621 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2021-03-15T22:28:21.622 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2021-03-15T22:28:21.622 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 7, 2021-03-15T22:28:21.622 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2021-03-15T22:28:21.622 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2021-03-15T22:28:21.623 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2021-03-15T22:28:21.623 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2021-03-15T22:28:21.623 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2021-03-15T22:28:21.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.623 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2021-03-15T22:28:21.624 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "device_health_metrics", 2021-03-15T22:28:21.624 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:16:34.817191+0000", 2021-03-15T22:28:21.624 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:21.624 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:21.624 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:21.625 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:21.625 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:21.625 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:21.625 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:21.625 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:21.626 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:21.626 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:21.626 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:21.626 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:21.627 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2021-03-15T22:28:21.627 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2021-03-15T22:28:21.627 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2021-03-15T22:28:21.627 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2021-03-15T22:28:21.627 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2021-03-15T22:28:21.628 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:21.628 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:21.628 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:21.628 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:21.629 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:21.629 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:21.629 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:21.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.629 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "23", 2021-03-15T22:28:21.630 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:21.630 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:21.630 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:21.630 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:21.630 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:21.631 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:21.631 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:21.631 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:21.631 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:21.632 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:21.632 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:21.632 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:21.632 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:21.632 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:21.633 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:21.633 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:21.633 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:21.633 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:21.633 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:21.634 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:21.634 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:21.634 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:21.634 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:21.635 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:21.635 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:21.635 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:21.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.635 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:21.636 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:21.636 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:21.636 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:21.636 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:21.637 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:21.637 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:21.637 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:21.637 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:21.637 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:21.638 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:21.638 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:28:21.638 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2021-03-15T22:28:21.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.639 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:21.639 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_devicehealth": {} 2021-03-15T22:28:21.639 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.640 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2021-03-15T22:28:21.640 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "iscsi", 2021-03-15T22:28:21.640 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:49.906400+0000", 2021-03-15T22:28:21.640 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:21.641 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:21.641 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:21.641 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:21.641 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:21.641 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:21.642 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:21.642 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:21.642 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:21.642 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:21.643 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:21.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:21.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 3, 2021-03-15T22:28:21.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 3, 2021-03-15T22:28:21.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 3, 2021-03-15T22:28:21.644 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 3, 2021-03-15T22:28:21.644 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 3, 2021-03-15T22:28:21.644 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:21.645 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:21.645 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:21.645 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:21.645 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:21.646 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:21.646 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:21.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.646 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "56", 2021-03-15T22:28:21.646 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:21.647 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:21.647 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:21.647 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:21.647 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:21.648 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:21.648 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:21.648 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:21.648 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:21.649 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:21.649 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:21.649 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:21.649 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:21.649 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:21.650 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:21.650 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:21.650 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:21.650 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:21.651 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:21.651 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:21.651 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:21.651 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:21.651 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:21.652 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:21.652 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:21.652 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:21.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.653 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:21.653 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:21.653 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:21.653 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:21.653 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:21.654 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:21.654 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:21.654 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:21.654 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:21.655 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:21.655 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:21.655 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:28:21.655 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:21.655 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2021-03-15T22:28:21.656 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2021-03-15T22:28:21.656 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".rgw.root", 2021-03-15T22:28:21.657 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:52.257970+0000", 2021-03-15T22:28:21.657 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:21.657 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:21.657 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:21.658 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:21.658 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:21.658 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:21.658 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:21.659 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:21.659 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:21.659 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:21.659 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:21.659 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:21.660 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:28:21.660 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:28:21.660 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:28:21.660 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:28:21.660 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:28:21.661 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:21.661 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:21.661 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:21.661 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:21.662 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:21.662 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:21.662 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:21.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.663 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "58", 2021-03-15T22:28:21.663 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:21.663 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:21.663 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:21.663 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:21.664 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:21.664 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:21.664 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:21.664 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:21.664 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:21.665 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:21.665 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:21.665 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:21.665 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:21.666 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:21.666 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:21.666 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:21.666 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:21.666 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:21.667 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:21.667 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:21.667 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:21.667 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:21.668 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:21.668 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:21.668 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:21.668 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:21.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.669 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:21.669 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:21.669 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:21.669 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:21.670 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:21.670 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:21.670 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:21.670 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:21.670 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:21.671 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:21.671 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:21.671 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:28:21.671 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:21.672 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:28:21.672 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.673 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2021-03-15T22:28:21.673 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.log", 2021-03-15T22:28:21.673 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:54.562498+0000", 2021-03-15T22:28:21.673 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:21.673 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:21.674 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:21.674 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:21.674 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:21.674 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:21.675 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:21.675 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:21.675 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:21.675 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:21.675 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:21.676 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:21.676 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:28:21.676 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:28:21.676 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:28:21.676 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:28:21.677 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:28:21.677 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:21.677 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:21.678 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:21.678 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:21.678 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:21.679 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:21.679 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:21.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.680 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "60", 2021-03-15T22:28:21.680 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:21.681 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:21.681 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:21.681 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:21.681 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:21.682 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:21.682 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:21.682 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:21.682 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:21.683 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:21.683 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:21.683 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:21.683 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:21.683 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:21.684 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:21.684 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:21.684 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:21.684 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:21.684 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:21.685 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:21.685 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:21.685 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:21.685 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:21.686 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:21.686 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:21.686 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:21.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.687 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:21.687 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:21.687 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:21.687 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:21.687 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:21.688 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:21.688 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:21.688 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:21.688 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:21.689 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:21.689 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:21.689 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:28:21.689 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:21.689 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:28:21.690 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.690 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 5, 2021-03-15T22:28:21.691 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.control", 2021-03-15T22:28:21.691 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:56.243806+0000", 2021-03-15T22:28:21.691 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:21.691 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:21.692 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:21.692 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:21.692 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:21.692 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:21.692 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:21.693 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:21.693 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:21.693 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:21.693 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:21.693 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:21.694 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 22, 2021-03-15T22:28:21.694 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 22, 2021-03-15T22:28:21.694 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 22, 2021-03-15T22:28:21.695 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 22, 2021-03-15T22:28:21.695 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 22, 2021-03-15T22:28:21.695 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:21.695 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "5.16", 2021-03-15T22:28:21.696 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 182, 2021-03-15T22:28:21.696 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 181, 2021-03-15T22:28:21.696 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 181, 2021-03-15T22:28:21.696 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:21.696 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:21.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.697 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "182", 2021-03-15T22:28:21.697 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:21.698 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "182", 2021-03-15T22:28:21.698 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "180", 2021-03-15T22:28:21.698 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:21.698 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:21.698 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:21.699 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:21.699 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:21.699 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:21.699 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:21.700 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:21.700 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:21.700 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:21.701 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:21.701 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:21.701 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:21.701 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:21.702 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:21.702 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:21.702 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:21.702 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:21.702 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:21.703 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:21.703 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:21.703 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:21.703 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:21.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.704 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:21.704 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:21.704 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:21.704 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:21.705 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:21.705 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:21.705 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:21.705 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:21.705 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:21.706 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:21.706 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:21.706 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:28:21.706 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:21.707 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:28:21.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.707 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 6, 2021-03-15T22:28:21.708 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.meta", 2021-03-15T22:28:21.708 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:58.730534+0000", 2021-03-15T22:28:21.708 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:21.708 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:21.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:21.709 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:21.709 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:21.709 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:21.710 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:21.710 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:21.710 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:21.710 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:21.711 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:21.711 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:21.711 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:28:21.711 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:28:21.711 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:28:21.712 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:28:21.712 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:28:21.712 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:21.712 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:21.712 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:21.713 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:21.713 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:21.713 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:21.713 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:21.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.714 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "109", 2021-03-15T22:28:21.714 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:21.714 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:21.714 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:21.715 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:21.715 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:21.715 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:21.715 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:21.716 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:21.716 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:21.716 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:21.716 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:21.716 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:21.717 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:21.717 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:21.717 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:21.717 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:21.718 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:21.718 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:21.718 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:21.718 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:21.718 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:21.719 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:21.719 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:21.719 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:21.719 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:21.720 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:21.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.720 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:21.720 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:21.720 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:21.721 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:21.721 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:21.721 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:21.721 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:21.721 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:21.722 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:21.722 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:21.722 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:21.722 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:28:21.722 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_bias": 4, 2021-03-15T22:28:21.723 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 8 2021-03-15T22:28:21.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.723 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:21.723 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:28:21.724 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.724 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:28:21.724 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2021-03-15T22:28:21.725 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:20:03.377294+0000", 2021-03-15T22:28:21.725 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2021-03-15T22:28:21.725 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-03-15T22:28:21.725 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:21.725 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:21.726 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:21.726 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:21.726 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:21.726 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:21.727 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:21.727 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:21.727 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:21.727 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:21.727 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2021-03-15T22:28:21.728 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2021-03-15T22:28:21.728 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2021-03-15T22:28:21.728 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2021-03-15T22:28:21.728 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2021-03-15T22:28:21.728 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:21.729 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:21.729 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:21.729 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:21.729 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:21.730 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:21.730 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:21.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.730 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "541", 2021-03-15T22:28:21.731 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:21.731 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:21.731 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:21.732 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:21.732 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:21.732 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 437, 2021-03-15T22:28:21.733 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 541, 2021-03-15T22:28:21.733 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:21.733 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:21.734 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:21.734 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:21.735 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:21.735 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:21.735 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:21.735 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:21.736 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:21.736 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:21.736 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:21.737 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:21.737 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:21.737 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:21.737 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:21.738 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:21.738 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:21.738 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:21.738 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:21.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.739 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:21.739 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:21.739 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:21.740 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:21.740 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:21.740 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:21.740 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:21.740 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:21.741 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:21.741 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:21.741 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:21.741 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:28:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2021-03-15T22:28:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.742 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:28:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd", 2021-03-15T22:28:21.743 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:21.744 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 8, 2021-03-15T22:28:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 560, 2021-03-15T22:28:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:21.745 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.746 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2021-03-15T22:28:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.747 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2021-03-15T22:28:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.749 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2021-03-15T22:28:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.750 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2021-03-15T22:28:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.752 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2021-03-15T22:28:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.753 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2021-03-15T22:28:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.755 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2021-03-15T22:28:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.756 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2021-03-15T22:28:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.757 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/188228176", 2021-03-15T22:28:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/188228176", 2021-03-15T22:28:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/188228176", 2021-03-15T22:28:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/188228176", 2021-03-15T22:28:21.758 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:28:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526", 2021-03-15T22:28:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:21.760 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.999237060546875, 2021-03-15T22:28:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 13, 2021-03-15T22:28:21.761 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 556, 2021-03-15T22:28:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:21.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2021-03-15T22:28:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:21.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2021-03-15T22:28:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:21.764 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2021-03-15T22:28:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:21.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2021-03-15T22:28:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:21.767 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2021-03-15T22:28:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:21.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2021-03-15T22:28:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:21.770 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:21.771 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2021-03-15T22:28:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:21.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2021-03-15T22:28:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:21.773 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/3924955423", 2021-03-15T22:28:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/3924955423", 2021-03-15T22:28:21.774 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/3924955423", 2021-03-15T22:28:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/3924955423", 2021-03-15T22:28:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:21.775 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:28:21.776 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05", 2021-03-15T22:28:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 0, 2021-03-15T22:28:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:21.777 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 18, 2021-03-15T22:28:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 421, 2021-03-15T22:28:21.778 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 490, 2021-03-15T22:28:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.779 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2021-03-15T22:28:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2021-03-15T22:28:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.781 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.782 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2021-03-15T22:28:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.784 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.784 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2021-03-15T22:28:21.784 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:21.784 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.784 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.785 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:21.785 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2021-03-15T22:28:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2021-03-15T22:28:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:21.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2021-03-15T22:28:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:21.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2021-03-15T22:28:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:21.790 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/1401263050", 2021-03-15T22:28:21.791 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/1401263050", 2021-03-15T22:28:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/1401263050", 2021-03-15T22:28:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/1401263050", 2021-03-15T22:28:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: "exists" 2021-03-15T22:28:21.792 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:28:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56", 2021-03-15T22:28:21.793 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:21.794 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 26, 2021-03-15T22:28:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 559, 2021-03-15T22:28:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:21.795 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.796 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2021-03-15T22:28:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.797 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2021-03-15T22:28:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.799 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2021-03-15T22:28:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.800 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2021-03-15T22:28:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.801 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.802 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2021-03-15T22:28:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.803 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2021-03-15T22:28:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.804 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2021-03-15T22:28:21.805 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2021-03-15T22:28:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.807 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/3960968636", 2021-03-15T22:28:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/3960968636", 2021-03-15T22:28:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/3960968636", 2021-03-15T22:28:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/3960968636", 2021-03-15T22:28:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:21.808 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.809 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:28:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237", 2021-03-15T22:28:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:21.810 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 31, 2021-03-15T22:28:21.811 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 555, 2021-03-15T22:28:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6800", 2021-03-15T22:28:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:21.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6801", 2021-03-15T22:28:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:21.814 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6802", 2021-03-15T22:28:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:21.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6803", 2021-03-15T22:28:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:21.817 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6806", 2021-03-15T22:28:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:21.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6807", 2021-03-15T22:28:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:21.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6804", 2021-03-15T22:28:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:21.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6805", 2021-03-15T22:28:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:21.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6801/37386122", 2021-03-15T22:28:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6803/37386122", 2021-03-15T22:28:21.824 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6807/37386122", 2021-03-15T22:28:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6805/37386122", 2021-03-15T22:28:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:21.825 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:28:21.826 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2", 2021-03-15T22:28:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 0, 2021-03-15T22:28:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:21.827 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 38, 2021-03-15T22:28:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 490, 2021-03-15T22:28:21.828 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 540, 2021-03-15T22:28:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6808", 2021-03-15T22:28:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.830 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6809", 2021-03-15T22:28:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.831 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.832 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6810", 2021-03-15T22:28:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6811", 2021-03-15T22:28:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:21.834 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6814", 2021-03-15T22:28:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:21.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6815", 2021-03-15T22:28:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:21.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6812", 2021-03-15T22:28:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:21.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6813", 2021-03-15T22:28:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:21.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6809/3610838738", 2021-03-15T22:28:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6811/3610838738", 2021-03-15T22:28:21.841 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6815/3610838738", 2021-03-15T22:28:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6813/3610838738", 2021-03-15T22:28:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: "exists" 2021-03-15T22:28:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:28:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe", 2021-03-15T22:28:21.843 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:21.844 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 45, 2021-03-15T22:28:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 555, 2021-03-15T22:28:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:21.845 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.846 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6816", 2021-03-15T22:28:21.847 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:21.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.847 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.847 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6817", 2021-03-15T22:28:21.848 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:21.848 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.848 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:21.849 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.849 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.849 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6818", 2021-03-15T22:28:21.849 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:21.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.850 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.850 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6819", 2021-03-15T22:28:21.851 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:21.851 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.851 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.851 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:21.852 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.852 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6822", 2021-03-15T22:28:21.852 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:21.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.853 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.853 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6823", 2021-03-15T22:28:21.854 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:21.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.854 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.854 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:21.855 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.855 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6820", 2021-03-15T22:28:21.855 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:21.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.856 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.856 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6821", 2021-03-15T22:28:21.857 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:21.857 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.857 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.857 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6817/171378275", 2021-03-15T22:28:21.858 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6819/171378275", 2021-03-15T22:28:21.858 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6823/171378275", 2021-03-15T22:28:21.858 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6821/171378275", 2021-03-15T22:28:21.858 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:21.858 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:21.859 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:21.859 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.860 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:28:21.860 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614", 2021-03-15T22:28:21.861 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:21.861 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:21.861 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:21.861 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:21.862 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:21.862 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:21.862 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 52, 2021-03-15T22:28:21.862 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 559, 2021-03-15T22:28:21.862 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.863 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.864 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6824", 2021-03-15T22:28:21.864 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:21.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6825", 2021-03-15T22:28:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:21.865 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.866 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.866 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:21.866 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.867 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.867 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6826", 2021-03-15T22:28:21.867 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:21.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.868 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6827", 2021-03-15T22:28:21.868 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:21.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6830", 2021-03-15T22:28:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6831", 2021-03-15T22:28:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.871 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:21.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6828", 2021-03-15T22:28:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6829", 2021-03-15T22:28:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.874 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6825/652003948", 2021-03-15T22:28:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6827/652003948", 2021-03-15T22:28:21.875 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6831/652003948", 2021-03-15T22:28:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6829/652003948", 2021-03-15T22:28:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:21.876 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2021-03-15T22:28:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.877 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:28:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:21.878 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:27:55.678908+0000", 2021-03-15T22:28:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:28:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:21.880 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:28:12.905187+0000", 2021-03-15T22:28:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:28:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:21.882 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:26:26.403924+0000", 2021-03-15T22:28:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 490 2021-03-15T22:28:21.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:28:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:21.884 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:27:40.590279+0000", 2021-03-15T22:28:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:21.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:28:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:21.886 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:28:01.554965+0000", 2021-03-15T22:28:21.887 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:28:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:21.888 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:27:17.335404+0000", 2021-03-15T22:28:21.889 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 540 2021-03-15T22:28:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:28:21.890 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:21.891 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:27:35.403880+0000", 2021-03-15T22:28:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:28:21.892 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:21.893 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:27:53.370478+0000", 2021-03-15T22:28:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:21.894 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [ 2021-03-15T22:28:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.3", 2021-03-15T22:28:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:28:21.895 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:28:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:21.896 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2021-03-15T22:28:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.3", 2021-03-15T22:28:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.897 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:28:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:28:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.14", 2021-03-15T22:28:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.899 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:28:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:28:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.a", 2021-03-15T22:28:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.901 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:28:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:28:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.b", 2021-03-15T22:28:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.903 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:28:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:28:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2021-03-15T22:28:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:28:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:28:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.17", 2021-03-15T22:28:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:28:21.907 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 5 2021-03-15T22:28:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.1b", 2021-03-15T22:28:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:28:21.909 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 5 2021-03-15T22:28:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.910 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [ 2021-03-15T22:28:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.1", 2021-03-15T22:28:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.911 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:28:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2021-03-15T22:28:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.10", 2021-03-15T22:28:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.914 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.17", 2021-03-15T22:28:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:28:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:28:21.915 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.1f", 2021-03-15T22:28:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.916 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:28:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.917 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.3", 2021-03-15T22:28:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:21.918 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2021-03-15T22:28:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.6", 2021-03-15T22:28:21.919 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: 7 2021-03-15T22:28:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.7", 2021-03-15T22:28:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:21.921 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: 7 2021-03-15T22:28:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.b", 2021-03-15T22:28:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:21.923 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.924 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.3", 2021-03-15T22:28:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:21.925 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.7", 2021-03-15T22:28:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.926 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: 4 2021-03-15T22:28:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.b", 2021-03-15T22:28:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.928 INFO:teuthology.orchestra.run.smithi076.stdout: 7 2021-03-15T22:28:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.15", 2021-03-15T22:28:21.929 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.9", 2021-03-15T22:28:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.931 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.10", 2021-03-15T22:28:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.933 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:28:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.5", 2021-03-15T22:28:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.935 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2021-03-15T22:28:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.6", 2021-03-15T22:28:21.936 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:28:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: 7 2021-03-15T22:28:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.e", 2021-03-15T22:28:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:21.938 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2021-03-15T22:28:21.939 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": { 2021-03-15T22:28:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:28:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4087324200": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:28:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4141143304": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:28:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:28:21.940 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2779879991": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:28:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:28:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:28:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2483102204": "2021-03-16T22:13:41.603652+0000" 2021-03-15T22:28:21.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2021-03-15T22:28:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2021-03-15T22:28:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2021-03-15T22:28:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2021-03-15T22:28:21.942 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2021-03-15T22:28:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2021-03-15T22:28:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "ruleset-failure-domain": "osd", 2021-03-15T22:28:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2021-03-15T22:28:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2021-03-15T22:28:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:28:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:28:21.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 258, 2021-03-15T22:28:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 303, 2021-03-15T22:28:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.946 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 318, 2021-03-15T22:28:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.947 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 347, 2021-03-15T22:28:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 358, 2021-03-15T22:28:21.948 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 360, 2021-03-15T22:28:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 365, 2021-03-15T22:28:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 368, 2021-03-15T22:28:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.951 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 371, 2021-03-15T22:28:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.952 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 373, 2021-03-15T22:28:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 375, 2021-03-15T22:28:21.953 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 382, 2021-03-15T22:28:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:28:21.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 388, 2021-03-15T22:28:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:28:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 391, 2021-03-15T22:28:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 395, 2021-03-15T22:28:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.957 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 399, 2021-03-15T22:28:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:28:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 409, 2021-03-15T22:28:21.958 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:28:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 416, 2021-03-15T22:28:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 423, 2021-03-15T22:28:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.960 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [], 2021-03-15T22:28:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2021-03-15T22:28:21.961 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2021-03-15T22:28:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2021-03-15T22:28:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2021-03-15T22:28:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2021-03-15T22:28:21.962 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2021-03-15T22:28:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2021-03-15T22:28:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2021-03-15T22:28:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2021-03-15T22:28:21.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:21.963 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:28:21.964 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:21.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:21.988 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:21.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:22.090 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:22.112 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:22.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:22.113 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:22.214 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:22.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:22.237 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:22.237 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:22.239 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:28:22.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:22 smithi125 conmon[56739]: cluster 2021-03-15T22:28:21.103887+0000 mgr.y (mgr.14144) 752 : 2021-03-15T22:28:22.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:22 smithi125 conmon[56739]: cluster [DBG] pgmap v980: 138 pgs: 1 peering, 12 active+clean+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 53 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 86 op/s; 343/1743 objects degraded (19.679%); 52/1743 objects misplaced (2.983%); 11 MiB/s, 29 objects/s recovering 2021-03-15T22:28:22.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:22 smithi125 conmon[56739]: audit 2021-03-15T22:28:21.549126+0000 mon.c (mon.1) 47 : audit [DBG] from='client.? 172.21.15.76:0/2204234928' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:28:22.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:22.360 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:22.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:22.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:22.400 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:28:22.463 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:22.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:22 smithi076 conmon[56514]: cluster 2021-03-15T22:28:21.103887+0000 mgr.y (mgr.14144) 752 : cluster [DBG] pgmap v980: 138 pgs: 1 peering, 12 active+clean+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 53 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 86 op/s; 343/1743 objects degraded (19.679%); 52/1743 objects misplaced (2.983%); 11 MiB/s, 29 objects/s recovering 2021-03-15T22:28:22.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:22 smithi076 conmon[56514]: audit 2021-03-15T22:28:21.549126+0000 mon.c (mon.1) 47 : audit [DBG] from='client.? 172.21.15.76:0/2204234928' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:28:22.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:22 smithi076 conmon[62156]: cluster 2021-03-15T22:28 2021-03-15T22:28:22.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:22 smithi076 conmon[62156]: :21.103887+0000 mgr.y (mgr.14144) 752 : cluster [DBG] pgmap v980: 138 pgs: 1 peering, 12 active+clean+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 53 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 0 B/s rd, 0 B/s wr, 86 op/s; 343/1743 objects degraded (19.679%); 52/1743 objects misplaced (2.983%); 11 MiB/s, 29 objects/s recovering 2021-03-15T22:28:22.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:22 smithi076 conmon[62156]: audit 2021-03-15T22:28:21. 2021-03-15T22:28:22.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:22 smithi076 conmon[62156]: 549126+0000 mon.c (mon.1) 47 : audit [DBG] 2021-03-15T22:28:22.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:22 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2204234928' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:28:22.485 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:22.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:22.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:22.507 INFO:tasks.thrashosds.thrasher:j is {'epoch': 561, 'fsid': '8a572802-85db-11eb-9091-001a4aab830c', 'created': '2021-03-15T22:13:20.370795+0000', 'modified': '2021-03-15T22:28:21.159996+0000', 'last_up_change': '2021-03-15T22:27:35.183905+0000', 'last_in_change': '2021-03-15T22:28:11.480156+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-15T22:16:34.817191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-15T22:17:49.906400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-15T22:17:52.257970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-15T22:17:54.562498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-15T22:17:56.243806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 182, 'last_epoch_started': 181, 'last_epoch_clean': 181, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '182', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '182', 'last_force_op_resend_preluminous': '180', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-15T22:17:58.730534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-03-15T22:20:03.377294+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '541', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 437, 'snap_epoch': 541, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 560, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 188228176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 188228176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 188228176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 188228176}]}, 'public_addr': '172.21.15.76:6803/188228176', 'cluster_addr': '172.21.15.76:6805/188228176', 'heartbeat_back_addr': '172.21.15.76:6809/188228176', 'heartbeat_front_addr': '172.21.15.76:6807/188228176', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '97bd3ffc-15c8-4a6f-b37a-217813def526', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.999237060546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 556, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 3924955423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 3924955423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 3924955423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 3924955423}]}, 'public_addr': '172.21.15.76:6811/3924955423', 'cluster_addr': '172.21.15.76:6813/3924955423', 'heartbeat_back_addr': '172.21.15.76:6817/3924955423', 'heartbeat_front_addr': '172.21.15.76:6815/3924955423', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a7f5f115-e69e-47d9-b6c3-ff386ddfaa05', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 421, 'down_at': 490, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 1401263050}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 1401263050}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 1401263050}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 1401263050}]}, 'public_addr': '172.21.15.76:6819/1401263050', 'cluster_addr': '172.21.15.76:6821/1401263050', 'heartbeat_back_addr': '172.21.15.76:6825/1401263050', 'heartbeat_front_addr': '172.21.15.76:6823/1401263050', 'state': ['exists']}, {'osd': 3, 'uuid': 'a0bb7f88-0114-47cb-a82c-d8fb002aac56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 559, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 3960968636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 3960968636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 3960968636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 3960968636}]}, 'public_addr': '172.21.15.76:6827/3960968636', 'cluster_addr': '172.21.15.76:6829/3960968636', 'heartbeat_back_addr': '172.21.15.76:6833/3960968636', 'heartbeat_front_addr': '172.21.15.76:6831/3960968636', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b650611c-6101-47e8-9ca9-94e02cf8b237', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 555, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6800', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6801', 'nonce': 37386122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6802', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6803', 'nonce': 37386122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6806', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6807', 'nonce': 37386122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6804', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6805', 'nonce': 37386122}]}, 'public_addr': '172.21.15.125:6801/37386122', 'cluster_addr': '172.21.15.125:6803/37386122', 'heartbeat_back_addr': '172.21.15.125:6807/37386122', 'heartbeat_front_addr': '172.21.15.125:6805/37386122', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7d6ceff2-7093-4ab9-8b96-ade448093bf2', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 490, 'down_at': 540, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6808', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6809', 'nonce': 3610838738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6810', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6811', 'nonce': 3610838738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6814', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6815', 'nonce': 3610838738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6812', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6813', 'nonce': 3610838738}]}, 'public_addr': '172.21.15.125:6809/3610838738', 'cluster_addr': '172.21.15.125:6811/3610838738', 'heartbeat_back_addr': '172.21.15.125:6815/3610838738', 'heartbeat_front_addr': '172.21.15.125:6813/3610838738', 'state': ['exists']}, {'osd': 6, 'uuid': '98b72022-a9a7-4ee2-89a1-cac037de7afe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 555, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6816', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6817', 'nonce': 171378275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6818', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6819', 'nonce': 171378275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6822', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6823', 'nonce': 171378275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6820', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6821', 'nonce': 171378275}]}, 'public_addr': '172.21.15.125:6817/171378275', 'cluster_addr': '172.21.15.125:6819/171378275', 'heartbeat_back_addr': '172.21.15.125:6823/171378275', 'heartbeat_front_addr': '172.21.15.125:6821/171378275', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f99ea650-b2dc-4d87-a44c-56a897edd614', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 559, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6824', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6825', 'nonce': 652003948}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6826', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6827', 'nonce': 652003948}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6830', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6831', 'nonce': 652003948}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6828', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6829', 'nonce': 652003948}]}, 'public_addr': '172.21.15.125:6825/652003948', 'cluster_addr': '172.21.15.125:6827/652003948', 'heartbeat_back_addr': '172.21.15.125:6831/652003948', 'heartbeat_front_addr': '172.21.15.125:6829/652003948', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:27:55.678908+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:28:12.905187+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:26:26.403924+0000', 'dead_epoch': 490}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:27:40.590279+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:28:01.554965+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:27:17.335404+0000', 'dead_epoch': 540}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:27:35.403880+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:27:53.370478+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '3.3', 'osds': [3, 0, 6]}], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.14', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [{'pgid': '2.1', 'osds': [3, 0, 1]}, {'pgid': '3.10', 'osds': [6, 0, 3]}, {'pgid': '3.17', 'osds': [0, 3, 6]}, {'pgid': '3.1f', 'osds': [0, 6]}, {'pgid': '4.3', 'osds': [0, 7, 1]}, {'pgid': '4.6', 'osds': [0, 1, 7]}, {'pgid': '4.7', 'osds': [1, 0, 7]}, {'pgid': '4.b', 'osds': [0, 4, 3]}, {'pgid': '5.3', 'osds': [0, 6, 3]}, {'pgid': '5.7', 'osds': [1, 0, 4]}, {'pgid': '5.b', 'osds': [0, 7]}, {'pgid': '5.15', 'osds': [1, 0, 3]}, {'pgid': '6.9', 'osds': [0, 7, 3]}, {'pgid': '6.10', 'osds': [0, 1, 6]}, {'pgid': '7.5', 'osds': [7, 0, 1]}, {'pgid': '7.6', 'osds': [3, 7]}, {'pgid': '7.e', 'osds': [0, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.76:6800/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4087324200': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4141143304': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:6801/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/2779879991': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6801/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6800/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:0/2483102204': '2021-03-16T22:13:41.603652+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 258, 'length': 1}, {'begin': 303, 'length': 1}, {'begin': 318, 'length': 1}, {'begin': 347, 'length': 1}, {'begin': 358, 'length': 1}, {'begin': 360, 'length': 1}, {'begin': 365, 'length': 1}, {'begin': 368, 'length': 1}, {'begin': 371, 'length': 1}, {'begin': 373, 'length': 1}, {'begin': 375, 'length': 1}, {'begin': 382, 'length': 2}, {'begin': 388, 'length': 2}, {'begin': 391, 'length': 1}, {'begin': 395, 'length': 1}, {'begin': 399, 'length': 2}, {'begin': 409, 'length': 2}, {'begin': 416, 'length': 1}, {'begin': 423, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-03-15T22:28:22.508 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.3 2021-03-15T22:28:22.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd rm-pg-upmap 3.3 2021-03-15T22:28:22.588 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:22.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:22.611 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:22.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:22.705 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:22.712 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:22.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:22.735 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:22.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:22.837 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:22.858 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:22.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:22.860 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:22.871 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:22.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:22.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:22.984 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:22.984 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:23.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:23.107 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:23.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:23.109 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:23.210 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:23.235 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:23.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:23.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:23.337 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:23.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:23.364 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:23.465 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:23.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:23.491 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:23.491 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:23.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:23.617 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:23.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:23.618 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:23.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:23.742 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:23.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:23.743 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:23.818 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:28:23.818 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:28:23.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:28:23.819 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:28:23.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.819 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1215 7.7 7:ed4a01d2:::smithi07694878-2:head [copy-from ver 101] snapc 50=[3f] ondisk+write+known_if_redirected e179)", 2021-03-15T22:28:23.819 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:34.031426+0000", 2021-03-15T22:28:23.819 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 409.782035855, 2021-03-15T22:28:23.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028066849000000001, 2021-03-15T22:28:23.820 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:23.820 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:23.820 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:23.821 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:23.821 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:23.821 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1215 2021-03-15T22:28:23.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.821 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:23.822 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.822 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:23.822 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031426+0000", 2021-03-15T22:28:23.822 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:23.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.823 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:23.823 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031426+0000", 2021-03-15T22:28:23.823 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5599999999999998e-06 2021-03-15T22:28:23.824 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.824 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.824 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:23.824 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031429+0000", 2021-03-15T22:28:23.825 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0299999999999998e-06 2021-03-15T22:28:23.825 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.825 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:23.826 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031432+0000", 2021-03-15T22:28:23.826 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4944e-05 2021-03-15T22:28:23.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.826 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.827 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:23.827 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031447+0000", 2021-03-15T22:28:23.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9879999999999998e-06 2021-03-15T22:28:23.828 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.828 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.828 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:23.828 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031454+0000", 2021-03-15T22:28:23.828 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7342000000000005e-05 2021-03-15T22:28:23.829 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.829 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.829 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.829 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031542+0000", 2021-03-15T22:28:23.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1929999999999999e-05 2021-03-15T22:28:23.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.830 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.830 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:28:23.830 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031553+0000", 2021-03-15T22:28:23.831 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0019329010000000001 2021-03-15T22:28:23.831 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.831 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.831 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.831 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.033486+0000", 2021-03-15T22:28:23.832 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100469 2021-03-15T22:28:23.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:23.833 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.033587+0000", 2021-03-15T22:28:23.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012160482 2021-03-15T22:28:23.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.834 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,1", 2021-03-15T22:28:23.834 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.045747+0000", 2021-03-15T22:28:23.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005094922 2021-03-15T22:28:23.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.835 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.835 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:23.835 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.050842+0000", 2021-03-15T22:28:23.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0083807459999999997 2021-03-15T22:28:23.835 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.836 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:23.836 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059223+0000", 2021-03-15T22:28:23.836 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013855599999999999 2021-03-15T22:28:23.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:23.837 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059362+0000", 2021-03-15T22:28:23.837 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4486999999999994e-05 2021-03-15T22:28:23.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.838 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:23.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059426+0000", 2021-03-15T22:28:23.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6492e-05 2021-03-15T22:28:23.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:23.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059493+0000", 2021-03-15T22:28:23.839 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028066849000000001 2021-03-15T22:28:23.840 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.840 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:23.840 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.841 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1216 7.7 7:ed4a01d2:::smithi07694878-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e179)", 2021-03-15T22:28:23.841 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:34.031465+0000", 2021-03-15T22:28:23.841 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 409.78199699599998, 2021-03-15T22:28:23.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028173786999999999, 2021-03-15T22:28:23.842 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:23.842 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:23.843 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:23.843 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:23.844 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:23.844 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1216 2021-03-15T22:28:23.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.844 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:23.845 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.845 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:23.845 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031465+0000", 2021-03-15T22:28:23.845 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:23.845 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.846 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:23.846 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031465+0000", 2021-03-15T22:28:23.846 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0999999999999997e-07 2021-03-15T22:28:23.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:23.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031465+0000", 2021-03-15T22:28:23.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1480000000000001e-06 2021-03-15T22:28:23.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:23.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031467+0000", 2021-03-15T22:28:23.849 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2569e-05 2021-03-15T22:28:23.849 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.849 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.849 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:23.850 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031489+0000", 2021-03-15T22:28:23.850 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9050000000000001e-06 2021-03-15T22:28:23.850 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.851 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.851 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:23.851 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031492+0000", 2021-03-15T22:28:23.852 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000117608 2021-03-15T22:28:23.852 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.852 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.852 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.852 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031610+0000", 2021-03-15T22:28:23.853 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1499999999999998e-06 2021-03-15T22:28:23.853 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.853 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.853 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:28:23.853 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.031615+0000", 2021-03-15T22:28:23.854 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002194278 2021-03-15T22:28:23.854 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.854 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.855 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.855 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.033809+0000", 2021-03-15T22:28:23.855 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3332999999999998e-05 2021-03-15T22:28:23.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.856 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.856 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:23.856 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.033872+0000", 2021-03-15T22:28:23.856 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016247014000000001 2021-03-15T22:28:23.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.857 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.857 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.050119+0000", 2021-03-15T22:28:23.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011831400000000001 2021-03-15T22:28:23.860 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.860 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.860 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:23.860 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.050238+0000", 2021-03-15T22:28:23.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0092772289999999997 2021-03-15T22:28:23.861 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.861 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059515+0000", 2021-03-15T22:28:23.862 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2630000000000003e-05 2021-03-15T22:28:23.862 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.863 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:23.863 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059568+0000", 2021-03-15T22:28:23.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0798999999999998e-05 2021-03-15T22:28:23.863 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.863 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.864 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:23.864 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:34.059638+0000", 2021-03-15T22:28:23.864 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028173786999999999 2021-03-15T22:28:23.864 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.865 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.866 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:1426 7.7 7:ecc0cb20:::smithi07694878-25:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e196)", 2021-03-15T22:28:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:21:51.033436+0000", 2021-03-15T22:28:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 392.78002603499999, 2021-03-15T22:28:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021470156000000001, 2021-03-15T22:28:23.867 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:23.868 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1426 2021-03-15T22:28:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:23.869 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033436+0000", 2021-03-15T22:28:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.870 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033436+0000", 2021-03-15T22:28:23.871 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6360000000000001e-06 2021-03-15T22:28:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.872 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033437+0000", 2021-03-15T22:28:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.086e-06 2021-03-15T22:28:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.874 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:23.874 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033439+0000", 2021-03-15T22:28:23.874 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5630000000000003e-06 2021-03-15T22:28:23.874 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.874 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.875 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:23.875 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033447+0000", 2021-03-15T22:28:23.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1909999999999997e-06 2021-03-15T22:28:23.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.876 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.876 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:23.876 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033453+0000", 2021-03-15T22:28:23.876 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013471199999999999 2021-03-15T22:28:23.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.877 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.877 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033588+0000", 2021-03-15T22:28:23.878 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2900000000000002e-06 2021-03-15T22:28:23.878 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.878 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.878 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:28:23.879 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.033593+0000", 2021-03-15T22:28:23.880 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024725210000000001 2021-03-15T22:28:23.882 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.883 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.036066+0000", 2021-03-15T22:28:23.883 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5259999999999998e-05 2021-03-15T22:28:23.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.883 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.884 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:23.884 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.036101+0000", 2021-03-15T22:28:23.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010876768 2021-03-15T22:28:23.884 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.884 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.885 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.885 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.046978+0000", 2021-03-15T22:28:23.886 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010354 2021-03-15T22:28:23.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.886 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.886 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:23.886 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.047081+0000", 2021-03-15T22:28:23.887 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076443860000000004 2021-03-15T22:28:23.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.887 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.887 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.888 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.054725+0000", 2021-03-15T22:28:23.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3589000000000004e-05 2021-03-15T22:28:23.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.888 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:23.889 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.054789+0000", 2021-03-15T22:28:23.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011661400000000001 2021-03-15T22:28:23.889 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.889 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.889 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:23.890 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:21:51.054906+0000", 2021-03-15T22:28:23.890 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021470156000000001 2021-03-15T22:28:23.890 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.891 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:23.891 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.892 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:3010 7.b e330/329 7:d980be1f:::smithi07694878-23:head v 330'208, mlcod=0'0)", 2021-03-15T22:28:23.892 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:05.105001+0000", 2021-03-15T22:28:23.892 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 258.70846065000001, 2021-03-15T22:28:23.893 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037948101999999997, 2021-03-15T22:28:23.893 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:23.893 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:23.893 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:23.894 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.894 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:23.894 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105001+0000", 2021-03-15T22:28:23.894 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:23.894 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.895 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.895 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:23.895 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105001+0000", 2021-03-15T22:28:23.895 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076767199999999995 2021-03-15T22:28:23.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:23.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105769+0000", 2021-03-15T22:28:23.896 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1410000000000001e-06 2021-03-15T22:28:23.897 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.897 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:23.897 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105772+0000", 2021-03-15T22:28:23.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2334000000000001e-05 2021-03-15T22:28:23.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.898 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.898 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:23.898 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105784+0000", 2021-03-15T22:28:23.899 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4559999999999999e-06 2021-03-15T22:28:23.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.899 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.899 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:23.899 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105790+0000", 2021-03-15T22:28:23.900 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.66e-05 2021-03-15T22:28:23.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.900 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.900 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.900 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105846+0000", 2021-03-15T22:28:23.901 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0828000000000001e-05 2021-03-15T22:28:23.901 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.902 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.902 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:23.902 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105867+0000", 2021-03-15T22:28:23.902 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036960346999999998 2021-03-15T22:28:23.902 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.903 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.903 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:23.903 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.142827+0000", 2021-03-15T22:28:23.903 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121724 2021-03-15T22:28:23.904 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.904 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.904 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:23.904 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.142949+0000", 2021-03-15T22:28:23.904 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037948101999999997 2021-03-15T22:28:23.905 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.905 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:23.905 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.905 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.905 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.906 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3172 7.7 7:ed4a01d2:::smithi07694878-2:head [copy-from ver 104] snapc f4=[f4,f3,f2,e9,e7,e1,de] ondisk+write+known_if_redirected e344)", 2021-03-15T22:28:23.906 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:19.131288+0000", 2021-03-15T22:28:23.906 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 244.682173312, 2021-03-15T22:28:23.906 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025760814999999999, 2021-03-15T22:28:23.907 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:23.907 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:23.907 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:23.907 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:23.907 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:23.908 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3172 2021-03-15T22:28:23.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.908 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:23.908 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.908 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:23.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131288+0000", 2021-03-15T22:28:23.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:23.909 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:23.910 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131288+0000", 2021-03-15T22:28:23.910 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0330000000000001e-06 2021-03-15T22:28:23.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.910 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.911 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:23.911 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131289+0000", 2021-03-15T22:28:23.911 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3699999999999999e-07 2021-03-15T22:28:23.911 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.911 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.912 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:23.912 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131290+0000", 2021-03-15T22:28:23.912 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2509999999999999e-06 2021-03-15T22:28:23.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.912 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.913 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:23.913 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131293+0000", 2021-03-15T22:28:23.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4480000000000001e-06 2021-03-15T22:28:23.913 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.914 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:23.914 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131296+0000", 2021-03-15T22:28:23.914 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012789800000000001 2021-03-15T22:28:23.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.915 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.132575+0000", 2021-03-15T22:28:23.915 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4247999999999999e-05 2021-03-15T22:28:23.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.916 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:23.916 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.132639+0000", 2021-03-15T22:28:23.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002374929 2021-03-15T22:28:23.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.917 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,1", 2021-03-15T22:28:23.917 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.135014+0000", 2021-03-15T22:28:23.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050646509999999999 2021-03-15T22:28:23.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.918 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:23.918 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.140079+0000", 2021-03-15T22:28:23.918 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016804399000000001 2021-03-15T22:28:23.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.919 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.919 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:23.919 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.156883+0000", 2021-03-15T22:28:23.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9956999999999997e-05 2021-03-15T22:28:23.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.920 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:23.920 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.156923+0000", 2021-03-15T22:28:23.921 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7237999999999997e-05 2021-03-15T22:28:23.921 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.921 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.921 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:23.921 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.156970+0000", 2021-03-15T22:28:23.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8844000000000006e-05 2021-03-15T22:28:23.922 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.922 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:23.922 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.157049+0000", 2021-03-15T22:28:23.923 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025760814999999999 2021-03-15T22:28:23.923 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.923 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:23.923 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.924 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3173 7.7 7:ed4a01d2:::smithi07694878-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e344)", 2021-03-15T22:28:23.924 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:19.131298+0000", 2021-03-15T22:28:23.924 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 244.682163589, 2021-03-15T22:28:23.925 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025917880000000001, 2021-03-15T22:28:23.925 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:23.925 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:23.926 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:23.926 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:23.926 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:23.926 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3173 2021-03-15T22:28:23.927 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.927 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:23.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:23.927 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131298+0000", 2021-03-15T22:28:23.928 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:23.928 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.928 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.928 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:23.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131298+0000", 2021-03-15T22:28:23.929 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2499999999999995e-07 2021-03-15T22:28:23.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.929 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:23.929 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131298+0000", 2021-03-15T22:28:23.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4899999999999995e-07 2021-03-15T22:28:23.930 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.930 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.930 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:23.931 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131299+0000", 2021-03-15T22:28:23.931 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9730000000000002e-06 2021-03-15T22:28:23.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.931 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131302+0000", 2021-03-15T22:28:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.21e-06 2021-03-15T22:28:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.932 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:23.933 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.131304+0000", 2021-03-15T22:28:23.933 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001449365 2021-03-15T22:28:23.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.933 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.934 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.934 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.132754+0000", 2021-03-15T22:28:23.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0812000000000002e-05 2021-03-15T22:28:23.934 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.934 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:23.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.132785+0000", 2021-03-15T22:28:23.935 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0063974619999999996 2021-03-15T22:28:23.935 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.936 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.936 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.139182+0000", 2021-03-15T22:28:23.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5412000000000002e-05 2021-03-15T22:28:23.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.937 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.937 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:23.937 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.139257+0000", 2021-03-15T22:28:23.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017824311999999998 2021-03-15T22:28:23.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.938 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.938 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.938 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.157082+0000", 2021-03-15T22:28:23.938 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6941999999999997e-05 2021-03-15T22:28:23.938 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.939 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:23.939 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.157149+0000", 2021-03-15T22:28:23.939 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7218000000000006e-05 2021-03-15T22:28:23.939 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.940 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:23.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:19.157216+0000", 2021-03-15T22:28:23.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025917880000000001 2021-03-15T22:28:23.941 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.941 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:23.941 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.942 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3950 7.7 7:ed4a01d2:::smithi07694878-2:head [write 658973~703921,stat] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:28:23.942 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:28:23.942 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 168.68622040100001, 2021-03-15T22:28:23.942 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99727343300000004, 2021-03-15T22:28:23.942 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:23.943 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:23.943 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:23.943 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:23.944 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:23.944 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3950 2021-03-15T22:28:23.944 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.944 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:23.944 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.945 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:23.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:28:23.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:23.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.946 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:23.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:28:23.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076287800000000004 2021-03-15T22:28:23.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:23.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128004+0000", 2021-03-15T22:28:23.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8489999999999999e-06 2021-03-15T22:28:23.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.948 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:23.948 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128008+0000", 2021-03-15T22:28:23.948 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3244e-05 2021-03-15T22:28:23.948 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.949 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.949 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:23.949 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128031+0000", 2021-03-15T22:28:23.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2050000000000002e-06 2021-03-15T22:28:23.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.950 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:23.950 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128039+0000", 2021-03-15T22:28:23.950 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9277999999999997e-05 2021-03-15T22:28:23.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.951 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.951 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.951 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128099+0000", 2021-03-15T22:28:23.951 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2401999999999999e-05 2021-03-15T22:28:23.952 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.952 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.952 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:23.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128111+0000", 2021-03-15T22:28:23.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99084192299999996 2021-03-15T22:28:23.953 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.953 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.953 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.118953+0000", 2021-03-15T22:28:23.954 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000162363 2021-03-15T22:28:23.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.954 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:23.954 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.119115+0000", 2021-03-15T22:28:23.955 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023901900000000001 2021-03-15T22:28:23.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.955 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.955 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:28:23.955 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.119354+0000", 2021-03-15T22:28:23.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020445160000000001 2021-03-15T22:28:23.956 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.956 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.956 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:23.957 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121399+0000", 2021-03-15T22:28:23.957 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021134299999999999 2021-03-15T22:28:23.957 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.957 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.957 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:23.958 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123512+0000", 2021-03-15T22:28:23.958 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000816028 2021-03-15T22:28:23.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.958 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:23.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124328+0000", 2021-03-15T22:28:23.959 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2724000000000005e-05 2021-03-15T22:28:23.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.959 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:23.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124391+0000", 2021-03-15T22:28:23.960 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012357399999999999 2021-03-15T22:28:23.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:23.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124515+0000", 2021-03-15T22:28:23.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99727343300000004 2021-03-15T22:28:23.961 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.961 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:23.962 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.962 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.962 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3951 7.7 7:ed4a01d2:::smithi07694878-2:head [write 1907912~570853] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:28:23.962 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:28:23.963 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 168.679478265, 2021-03-15T22:28:23.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99540528800000005, 2021-03-15T22:28:23.963 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:23.963 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:23.963 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:23.964 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:23.964 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:23.964 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3951 2021-03-15T22:28:23.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.965 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:23.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:23.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:28:23.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:23.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:23.967 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:28:23.967 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00050036199999999996 2021-03-15T22:28:23.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.967 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.967 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:23.968 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134484+0000", 2021-03-15T22:28:23.968 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9979999999999999e-06 2021-03-15T22:28:23.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.968 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.969 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:23.969 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134487+0000", 2021-03-15T22:28:23.969 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3523000000000001e-05 2021-03-15T22:28:23.969 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.969 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.970 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:23.970 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134500+0000", 2021-03-15T22:28:23.970 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5979999999999996e-06 2021-03-15T22:28:23.970 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.971 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:23.971 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134506+0000", 2021-03-15T22:28:23.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0407000000000001e-05 2021-03-15T22:28:23.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.972 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134546+0000", 2021-03-15T22:28:23.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5650000000000002e-06 2021-03-15T22:28:23.973 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.973 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:23.973 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134555+0000", 2021-03-15T22:28:23.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98647964300000002 2021-03-15T22:28:23.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.974 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.974 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121034+0000", 2021-03-15T22:28:23.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104298 2021-03-15T22:28:23.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.975 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.975 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:23.975 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121139+0000", 2021-03-15T22:28:23.976 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0034252140000000002 2021-03-15T22:28:23.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.976 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.976 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.977 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124564+0000", 2021-03-15T22:28:23.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3158000000000005e-05 2021-03-15T22:28:23.977 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.977 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:23.978 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124657+0000", 2021-03-15T22:28:23.978 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000172645 2021-03-15T22:28:23.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.978 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:28:23.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124830+0000", 2021-03-15T22:28:23.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001768096 2021-03-15T22:28:23.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:23.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126598+0000", 2021-03-15T22:28:23.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020473919999999999 2021-03-15T22:28:23.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:23.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128645+0000", 2021-03-15T22:28:23.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00067562299999999998 2021-03-15T22:28:23.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:23.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129321+0000", 2021-03-15T22:28:23.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3652999999999998e-05 2021-03-15T22:28:23.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.983 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:23.983 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129354+0000", 2021-03-15T22:28:23.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4112999999999997e-05 2021-03-15T22:28:23.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:23.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129389+0000", 2021-03-15T22:28:23.984 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99540528800000005 2021-03-15T22:28:23.985 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.985 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:23.985 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:23.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.986 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3952 7.7 7:ed4a01d2:::smithi07694878-2:head [write 2985346~73627] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:28:23.986 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:28:23.986 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 168.67628376600001, 2021-03-15T22:28:23.986 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99236444199999996, 2021-03-15T22:28:23.987 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:23.987 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:23.987 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:23.987 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:23.987 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:23.988 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3952 2021-03-15T22:28:23.988 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.988 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:23.988 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.989 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:23.989 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:28:23.989 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:23.989 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.989 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.990 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:23.990 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:28:23.990 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5050000000000006e-05 2021-03-15T22:28:23.990 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.991 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:23.991 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137273+0000", 2021-03-15T22:28:23.991 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.883e-06 2021-03-15T22:28:23.991 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.992 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.992 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:23.992 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137275+0000", 2021-03-15T22:28:23.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0764000000000001e-05 2021-03-15T22:28:23.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.993 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.993 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:23.993 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137285+0000", 2021-03-15T22:28:23.993 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2190000000000001e-06 2021-03-15T22:28:23.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.994 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:23.994 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137292+0000", 2021-03-15T22:28:23.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5432999999999997e-05 2021-03-15T22:28:23.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.995 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.995 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137337+0000", 2021-03-15T22:28:23.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.738e-06 2021-03-15T22:28:23.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:23.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137346+0000", 2021-03-15T22:28:23.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98385600600000001 2021-03-15T22:28:23.997 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.997 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:23.998 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121202+0000", 2021-03-15T22:28:23.998 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9746e-05 2021-03-15T22:28:23.998 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.998 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.998 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:23.999 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121262+0000", 2021-03-15T22:28:23.999 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041684740000000001 2021-03-15T22:28:23.999 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:23.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:23.999 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.000 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125430+0000", 2021-03-15T22:28:24.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0578000000000001e-05 2021-03-15T22:28:24.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125501+0000", 2021-03-15T22:28:24.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8120000000000002e-05 2021-03-15T22:28:24.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.001 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.002 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:28:24.002 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125599+0000", 2021-03-15T22:28:24.002 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010039140000000001 2021-03-15T22:28:24.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:24.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126603+0000", 2021-03-15T22:28:24.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020828969999999998 2021-03-15T22:28:24.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.004 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128686+0000", 2021-03-15T22:28:24.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00075311200000000001 2021-03-15T22:28:24.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129439+0000", 2021-03-15T22:28:24.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7739999999999999e-05 2021-03-15T22:28:24.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.006 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:24.006 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129496+0000", 2021-03-15T22:28:24.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5768e-05 2021-03-15T22:28:24.007 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.007 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.007 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.007 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129542+0000", 2021-03-15T22:28:24.008 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99236444199999996 2021-03-15T22:28:24.008 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.008 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.008 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.009 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.009 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3953 7.7 7:ed4a01d2:::smithi07694878-2:head [setxattr _header (57) in=7b,truncate 3058973] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:28:24.009 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:28:24.009 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 168.67615891200001, 2021-03-15T22:28:24.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99574138599999995, 2021-03-15T22:28:24.010 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:24.010 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:24.011 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:24.011 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:24.011 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:24.011 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3953 2021-03-15T22:28:24.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.012 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:24.012 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.012 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:24.012 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:28:24.013 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:24.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.013 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.013 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:24.013 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:28:24.014 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9230000000000001e-06 2021-03-15T22:28:24.014 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.014 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:24.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137305+0000", 2021-03-15T22:28:24.015 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.252e-06 2021-03-15T22:28:24.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.015 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:24.016 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137306+0000", 2021-03-15T22:28:24.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9039999999999997e-06 2021-03-15T22:28:24.016 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:24.017 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137312+0000", 2021-03-15T22:28:24.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.856e-06 2021-03-15T22:28:24.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.018 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:24.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137316+0000", 2021-03-15T22:28:24.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6783000000000002e-05 2021-03-15T22:28:24.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.018 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137372+0000", 2021-03-15T22:28:24.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9349999999999999e-06 2021-03-15T22:28:24.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:24.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137378+0000", 2021-03-15T22:28:24.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98391964700000001 2021-03-15T22:28:24.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.021 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.021 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121298+0000", 2021-03-15T22:28:24.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9109000000000004e-05 2021-03-15T22:28:24.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:24.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121387+0000", 2021-03-15T22:28:24.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045047059999999998 2021-03-15T22:28:24.023 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125892+0000", 2021-03-15T22:28:24.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6956999999999998e-05 2021-03-15T22:28:24.024 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.024 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.024 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125939+0000", 2021-03-15T22:28:24.025 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012742400000000001 2021-03-15T22:28:24.025 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.025 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:28:24.025 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126066+0000", 2021-03-15T22:28:24.026 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000545302 2021-03-15T22:28:24.026 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.026 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:24.027 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126611+0000", 2021-03-15T22:28:24.027 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021128950000000001 2021-03-15T22:28:24.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.027 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.028 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128724+0000", 2021-03-15T22:28:24.028 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042103729999999999 2021-03-15T22:28:24.028 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.028 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.029 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.132935+0000", 2021-03-15T22:28:24.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5052000000000001e-05 2021-03-15T22:28:24.029 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.029 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.030 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:24.030 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.132990+0000", 2021-03-15T22:28:24.030 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4268000000000003e-05 2021-03-15T22:28:24.030 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.030 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.031 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.031 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133044+0000", 2021-03-15T22:28:24.031 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99574138599999995 2021-03-15T22:28:24.031 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.032 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.032 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.032 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.032 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3954 7.7 7:ed4a01d2:::smithi07694878-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e420)", 2021-03-15T22:28:24.033 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:28:24.033 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 168.67612974299999, 2021-03-15T22:28:24.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99588332599999996, 2021-03-15T22:28:24.033 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:24.033 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:24.034 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:24.034 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:24.034 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:24.034 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3954 2021-03-15T22:28:24.035 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.035 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:24.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:24.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:28:24.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:24.036 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:24.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:28:24.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.015e-06 2021-03-15T22:28:24.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.037 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:24.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137333+0000", 2021-03-15T22:28:24.038 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5290000000000001e-06 2021-03-15T22:28:24.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:24.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137334+0000", 2021-03-15T22:28:24.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.489e-06 2021-03-15T22:28:24.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:24.040 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137341+0000", 2021-03-15T22:28:24.040 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1989999999999997e-06 2021-03-15T22:28:24.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:24.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137346+0000", 2021-03-15T22:28:24.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0890999999999998e-05 2021-03-15T22:28:24.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.042 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137397+0000", 2021-03-15T22:28:24.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3159999999999997e-06 2021-03-15T22:28:24.043 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.043 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.043 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:24.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137405+0000", 2021-03-15T22:28:24.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98402323800000002 2021-03-15T22:28:24.044 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.044 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.044 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.044 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121428+0000", 2021-03-15T22:28:24.044 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2376999999999998e-05 2021-03-15T22:28:24.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.045 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:24.045 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121471+0000", 2021-03-15T22:28:24.046 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050435389999999997 2021-03-15T22:28:24.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.046 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.046 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.046 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126514+0000", 2021-03-15T22:28:24.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3162000000000003e-05 2021-03-15T22:28:24.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:24.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126578+0000", 2021-03-15T22:28:24.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0065105759999999997 2021-03-15T22:28:24.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.049 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133088+0000", 2021-03-15T22:28:24.049 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4956000000000006e-05 2021-03-15T22:28:24.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.050 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133153+0000", 2021-03-15T22:28:24.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2038999999999993e-05 2021-03-15T22:28:24.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.051 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.052 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133215+0000", 2021-03-15T22:28:24.052 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99588332599999996 2021-03-15T22:28:24.052 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.052 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.052 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.053 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3970 7.7 7:ecc0cb20:::smithi07694878-25:head [write 709394~646021] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:28:24.053 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:28:24.054 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 168.53419288000001, 2021-03-15T22:28:24.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84719129900000001, 2021-03-15T22:28:24.054 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:24.054 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:24.054 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:24.055 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:24.055 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:24.055 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3970 2021-03-15T22:28:24.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.056 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:24.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:24.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:28:24.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:24.057 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.057 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:24.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:28:24.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000550652 2021-03-15T22:28:24.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.058 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:24.058 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279819+0000", 2021-03-15T22:28:24.059 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8980000000000004e-06 2021-03-15T22:28:24.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:24.060 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279824+0000", 2021-03-15T22:28:24.060 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1767999999999999e-05 2021-03-15T22:28:24.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.060 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:24.061 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279846+0000", 2021-03-15T22:28:24.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2509999999999994e-06 2021-03-15T22:28:24.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.061 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:24.062 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279854+0000", 2021-03-15T22:28:24.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7908000000000001e-05 2021-03-15T22:28:24.062 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.062 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.063 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279922+0000", 2021-03-15T22:28:24.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2315e-05 2021-03-15T22:28:24.063 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:24.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279934+0000", 2021-03-15T22:28:24.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84156851799999999 2021-03-15T22:28:24.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121503+0000", 2021-03-15T22:28:24.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012479099999999999 2021-03-15T22:28:24.065 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.066 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121628+0000", 2021-03-15T22:28:24.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000208667 2021-03-15T22:28:24.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.067 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.067 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:28:24.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121836+0000", 2021-03-15T22:28:24.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023170439999999999 2021-03-15T22:28:24.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:24.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124153+0000", 2021-03-15T22:28:24.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012293359999999999 2021-03-15T22:28:24.069 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125383+0000", 2021-03-15T22:28:24.070 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00099435699999999997 2021-03-15T22:28:24.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.070 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.071 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126377+0000", 2021-03-15T22:28:24.071 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3995999999999997e-05 2021-03-15T22:28:24.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.071 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.071 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:24.072 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126421+0000", 2021-03-15T22:28:24.072 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8797999999999997e-05 2021-03-15T22:28:24.072 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.073 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.073 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126460+0000", 2021-03-15T22:28:24.073 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84719129900000001 2021-03-15T22:28:24.073 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.073 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.074 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.074 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.074 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3971 7.7 7:ecc0cb20:::smithi07694878-25:head [write 2083358~718180] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:28:24.075 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:28:24.075 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 168.52622725099999, 2021-03-15T22:28:24.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.840294447, 2021-03-15T22:28:24.075 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:24.075 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:24.076 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:24.076 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:24.076 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:24.076 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3971 2021-03-15T22:28:24.076 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.077 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:24.077 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.077 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:24.077 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:28:24.078 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:24.078 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.078 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.078 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:24.079 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:28:24.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00065142800000000001 2021-03-15T22:28:24.079 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.079 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.079 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:24.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287886+0000", 2021-03-15T22:28:24.080 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.418e-06 2021-03-15T22:28:24.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.080 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:24.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287888+0000", 2021-03-15T22:28:24.081 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2062000000000001e-05 2021-03-15T22:28:24.081 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:24.082 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287900+0000", 2021-03-15T22:28:24.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7250000000000002e-06 2021-03-15T22:28:24.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:24.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287906+0000", 2021-03-15T22:28:24.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9849000000000003e-05 2021-03-15T22:28:24.083 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.084 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.084 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287956+0000", 2021-03-15T22:28:24.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6460000000000005e-06 2021-03-15T22:28:24.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.085 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:24.085 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287965+0000", 2021-03-15T22:28:24.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83445325400000003 2021-03-15T22:28:24.086 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.086 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.086 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.086 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122419+0000", 2021-03-15T22:28:24.086 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7459e-05 2021-03-15T22:28:24.087 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.087 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.087 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122476+0000", 2021-03-15T22:28:24.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012662400000000001 2021-03-15T22:28:24.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.088 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.088 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:28:24.088 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122603+0000", 2021-03-15T22:28:24.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037072580000000002 2021-03-15T22:28:24.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.089 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.089 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.089 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126310+0000", 2021-03-15T22:28:24.090 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000275853 2021-03-15T22:28:24.090 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.090 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.090 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:24.091 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126586+0000", 2021-03-15T22:28:24.091 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00087237800000000004 2021-03-15T22:28:24.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.091 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.092 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127458+0000", 2021-03-15T22:28:24.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5484000000000001e-05 2021-03-15T22:28:24.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.092 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:24.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127494+0000", 2021-03-15T22:28:24.093 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5009000000000003e-05 2021-03-15T22:28:24.093 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.093 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.094 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127529+0000", 2021-03-15T22:28:24.094 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.840294447 2021-03-15T22:28:24.094 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.094 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.095 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.095 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3972 7.7 7:ecc0cb20:::smithi07694878-25:head [setxattr _header (58) in=7b,truncate 3109394] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:28:24.096 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:28:24.096 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 168.52554529299999, 2021-03-15T22:28:24.096 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84111847799999995, 2021-03-15T22:28:24.096 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:24.096 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:24.097 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:24.097 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:24.097 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:24.097 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3972 2021-03-15T22:28:24.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.098 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:24.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.098 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:24.098 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:28:24.099 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:24.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.099 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.099 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:24.099 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:28:24.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9489999999999999e-06 2021-03-15T22:28:24.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.100 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:24.101 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287918+0000", 2021-03-15T22:28:24.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.359e-06 2021-03-15T22:28:24.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.101 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:24.102 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287920+0000", 2021-03-15T22:28:24.102 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8050000000000003e-06 2021-03-15T22:28:24.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.102 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:24.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287925+0000", 2021-03-15T22:28:24.103 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5959999999999997e-06 2021-03-15T22:28:24.103 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.104 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:24.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287930+0000", 2021-03-15T22:28:24.104 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2366999999999993e-05 2021-03-15T22:28:24.104 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.105 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.105 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.105 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288002+0000", 2021-03-15T22:28:24.105 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2240000000000001e-06 2021-03-15T22:28:24.105 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.106 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.106 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:24.106 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288012+0000", 2021-03-15T22:28:24.106 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83555615299999997 2021-03-15T22:28:24.106 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.107 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.107 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.107 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123568+0000", 2021-03-15T22:28:24.107 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4648999999999995e-05 2021-03-15T22:28:24.108 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.108 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.108 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.108 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123662+0000", 2021-03-15T22:28:24.108 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020287300000000001 2021-03-15T22:28:24.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.109 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:28:24.109 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123865+0000", 2021-03-15T22:28:24.109 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002471843 2021-03-15T22:28:24.110 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.110 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.110 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126337+0000", 2021-03-15T22:28:24.111 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025485200000000002 2021-03-15T22:28:24.111 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.111 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.111 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:24.111 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126592+0000", 2021-03-15T22:28:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002302077 2021-03-15T22:28:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.112 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.113 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128894+0000", 2021-03-15T22:28:24.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6276999999999998e-05 2021-03-15T22:28:24.113 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.113 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.113 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:24.114 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128930+0000", 2021-03-15T22:28:24.114 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104454 2021-03-15T22:28:24.114 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.114 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.114 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.115 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129035+0000", 2021-03-15T22:28:24.115 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84111847799999995 2021-03-15T22:28:24.115 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.115 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.116 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.116 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.117 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.117 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3973 7.7 7:ecc0cb20:::smithi07694878-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e420)", 2021-03-15T22:28:24.117 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:28:24.118 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 168.525514023, 2021-03-15T22:28:24.118 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.841325871, 2021-03-15T22:28:24.119 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:24.119 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:24.119 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:24.120 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:24.120 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:24.120 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3973 2021-03-15T22:28:24.120 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.120 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:24.121 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.121 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:24.121 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:28:24.121 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:24.122 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.122 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.122 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:24.122 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:28:24.123 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0400000000000005e-07 2021-03-15T22:28:24.123 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.123 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.123 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:24.123 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:28:24.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3090000000000001e-06 2021-03-15T22:28:24.124 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.124 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:24.125 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287950+0000", 2021-03-15T22:28:24.125 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1809999999999998e-06 2021-03-15T22:28:24.125 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.125 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.125 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:24.126 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287956+0000", 2021-03-15T22:28:24.126 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9060000000000004e-06 2021-03-15T22:28:24.126 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.126 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.127 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:24.127 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287960+0000", 2021-03-15T22:28:24.127 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8666999999999997e-05 2021-03-15T22:28:24.127 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.127 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.128 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.128 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288039+0000", 2021-03-15T22:28:24.128 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7850000000000003e-06 2021-03-15T22:28:24.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.128 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.129 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:24.129 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288045+0000", 2021-03-15T22:28:24.129 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83615280199999997 2021-03-15T22:28:24.129 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.130 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.130 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.130 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124198+0000", 2021-03-15T22:28:24.130 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2801999999999998e-05 2021-03-15T22:28:24.130 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.131 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:24.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124281+0000", 2021-03-15T22:28:24.131 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047980210000000004 2021-03-15T22:28:24.132 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.132 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.132 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.132 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129079+0000", 2021-03-15T22:28:24.132 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2120000000000007e-05 2021-03-15T22:28:24.133 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.133 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.133 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129161+0000", 2021-03-15T22:28:24.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011237400000000001 2021-03-15T22:28:24.134 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.134 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.134 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.134 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129273+0000", 2021-03-15T22:28:24.135 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.841325871 2021-03-15T22:28:24.135 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.135 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.135 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.136 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.136 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.136 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4287 7.7 7:ed4a01d2:::smithi07694878-2:head [rollback 128] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:28:24.136 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.162901+0000", 2021-03-15T22:28:24.136 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 143.65056048700001, 2021-03-15T22:28:24.137 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068606072000000004, 2021-03-15T22:28:24.137 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:24.137 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:24.137 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:24.137 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:24.138 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:24.138 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4287 2021-03-15T22:28:24.138 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.138 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:24.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:24.139 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162901+0000", 2021-03-15T22:28:24.139 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:24.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.140 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.140 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:24.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162901+0000", 2021-03-15T22:28:24.140 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9990000000000002e-06 2021-03-15T22:28:24.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.141 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:24.141 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162905+0000", 2021-03-15T22:28:24.141 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8239999999999999e-06 2021-03-15T22:28:24.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.142 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.142 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:24.142 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162910+0000", 2021-03-15T22:28:24.143 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5261000000000001e-05 2021-03-15T22:28:24.143 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.143 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.143 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:24.143 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162935+0000", 2021-03-15T22:28:24.144 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6001000000000001e-05 2021-03-15T22:28:24.144 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.144 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.144 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:24.144 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162961+0000", 2021-03-15T22:28:24.145 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.522e-05 2021-03-15T22:28:24.145 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.145 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.145 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.146 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163056+0000", 2021-03-15T22:28:24.146 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000127291 2021-03-15T22:28:24.146 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.146 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.146 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.147 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163184+0000", 2021-03-15T22:28:24.147 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030452000000000002 2021-03-15T22:28:24.147 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.147 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.148 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:28:24.149 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163488+0000", 2021-03-15T22:28:24.149 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026637311 2021-03-15T22:28:24.149 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.150 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.150 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:24.150 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.190125+0000", 2021-03-15T22:28:24.150 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021491026999999999 2021-03-15T22:28:24.151 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.151 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.151 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211617+0000", 2021-03-15T22:28:24.151 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019763414 2021-03-15T22:28:24.152 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.152 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.152 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.153 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231380+0000", 2021-03-15T22:28:24.153 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2330000000000005e-05 2021-03-15T22:28:24.153 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.153 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.153 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:24.154 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231472+0000", 2021-03-15T22:28:24.154 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4873999999999998e-05 2021-03-15T22:28:24.154 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.154 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.155 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.155 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231507+0000", 2021-03-15T22:28:24.155 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068606072000000004 2021-03-15T22:28:24.155 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.155 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.156 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.156 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.156 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.156 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4288 7.7 7:ed4a01d2:::smithi07694878-2:head [append 0~0] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:28:24.157 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.162974+0000", 2021-03-15T22:28:24.157 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 143.650487277, 2021-03-15T22:28:24.157 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068646398999999997, 2021-03-15T22:28:24.157 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:24.157 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:24.158 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:24.158 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:24.158 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:24.158 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4288 2021-03-15T22:28:24.159 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.159 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:24.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.159 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:24.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162974+0000", 2021-03-15T22:28:24.160 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:24.160 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.160 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.160 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:24.160 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162974+0000", 2021-03-15T22:28:24.161 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6800000000000009e-07 2021-03-15T22:28:24.161 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.161 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.161 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:24.162 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162975+0000", 2021-03-15T22:28:24.162 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0729999999999999e-06 2021-03-15T22:28:24.162 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.162 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.162 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:24.163 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162976+0000", 2021-03-15T22:28:24.163 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5429999999999998e-06 2021-03-15T22:28:24.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.164 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:24.164 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162980+0000", 2021-03-15T22:28:24.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4399999999999999e-06 2021-03-15T22:28:24.164 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.164 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.165 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:24.165 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162982+0000", 2021-03-15T22:28:24.165 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011790209999999999 2021-03-15T22:28:24.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.165 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.166 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.166 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164161+0000", 2021-03-15T22:28:24.166 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104054 2021-03-15T22:28:24.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.167 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.167 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.167 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164265+0000", 2021-03-15T22:28:24.167 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019185799999999999 2021-03-15T22:28:24.167 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.168 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.168 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:28:24.168 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164457+0000", 2021-03-15T22:28:24.168 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025674285000000002 2021-03-15T22:28:24.169 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.169 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.169 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:24.169 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.190132+0000", 2021-03-15T22:28:24.169 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021520336000000001 2021-03-15T22:28:24.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.170 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.170 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.170 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211652+0000", 2021-03-15T22:28:24.171 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019892508999999999 2021-03-15T22:28:24.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.171 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.171 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.171 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231544+0000", 2021-03-15T22:28:24.172 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0618000000000003e-05 2021-03-15T22:28:24.172 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.172 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.172 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:24.173 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231585+0000", 2021-03-15T22:28:24.173 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5694000000000001e-05 2021-03-15T22:28:24.173 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.173 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.173 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.174 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231621+0000", 2021-03-15T22:28:24.174 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068646398999999997 2021-03-15T22:28:24.174 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.174 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.174 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.175 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.175 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.175 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4290 7.d 7:bd183e1b:::smithi07694878-46:head [stat,delete] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:28:24.175 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.163011+0000", 2021-03-15T22:28:24.176 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 143.65045028099999, 2021-03-15T22:28:24.176 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048696824999999999, 2021-03-15T22:28:24.176 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:24.176 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:24.177 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:24.177 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:24.177 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:24.177 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4290 2021-03-15T22:28:24.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.178 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:24.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.178 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:24.178 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163011+0000", 2021-03-15T22:28:24.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:24.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.179 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.179 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:24.179 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163011+0000", 2021-03-15T22:28:24.180 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.079e-06 2021-03-15T22:28:24.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.180 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:24.181 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163012+0000", 2021-03-15T22:28:24.181 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.043e-06 2021-03-15T22:28:24.181 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.181 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.181 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:24.182 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163013+0000", 2021-03-15T22:28:24.182 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7620000000000003e-06 2021-03-15T22:28:24.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.183 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:24.183 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163018+0000", 2021-03-15T22:28:24.183 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4980000000000003e-06 2021-03-15T22:28:24.183 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.184 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:24.184 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163025+0000", 2021-03-15T22:28:24.184 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6739999999999997e-05 2021-03-15T22:28:24.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.185 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.185 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.185 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163061+0000", 2021-03-15T22:28:24.185 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9894000000000003e-05 2021-03-15T22:28:24.185 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.186 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.186 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.186 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163161+0000", 2021-03-15T22:28:24.186 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017699600000000001 2021-03-15T22:28:24.187 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.187 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.187 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:28:24.187 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163338+0000", 2021-03-15T22:28:24.187 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001543619 2021-03-15T22:28:24.188 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.188 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.188 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:24.188 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164882+0000", 2021-03-15T22:28:24.189 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021337467999999998 2021-03-15T22:28:24.189 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.189 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.189 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.189 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.186219+0000", 2021-03-15T22:28:24.190 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025381363000000001 2021-03-15T22:28:24.190 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.190 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.190 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:24.191 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211601+0000", 2021-03-15T22:28:24.191 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9228999999999998e-05 2021-03-15T22:28:24.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.191 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.191 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:24.192 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211660+0000", 2021-03-15T22:28:24.192 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8133999999999997e-05 2021-03-15T22:28:24.192 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.192 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.192 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.193 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211708+0000", 2021-03-15T22:28:24.193 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048696824999999999 2021-03-15T22:28:24.193 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.193 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.194 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.194 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.194 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:4284 7.b e445/401 7:d4368cae:::smithi07694878-18:head v 445'396, mlcod=445'394)", 2021-03-15T22:28:24.194 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.164429+0000", 2021-03-15T22:28:24.195 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 143.64903206100001, 2021-03-15T22:28:24.195 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048217024999999997, 2021-03-15T22:28:24.195 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:24.195 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:24.196 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:24.196 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.196 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:24.196 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164429+0000", 2021-03-15T22:28:24.196 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:24.197 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.197 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:24.197 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164429+0000", 2021-03-15T22:28:24.198 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00035440200000000003 2021-03-15T22:28:24.198 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.198 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.198 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:24.198 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164784+0000", 2021-03-15T22:28:24.199 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.659e-06 2021-03-15T22:28:24.199 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.199 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.199 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:24.199 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164788+0000", 2021-03-15T22:28:24.200 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8576999999999998e-05 2021-03-15T22:28:24.200 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.200 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.200 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:24.201 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164806+0000", 2021-03-15T22:28:24.201 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0139999999999998e-06 2021-03-15T22:28:24.201 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.201 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.201 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:24.202 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164814+0000", 2021-03-15T22:28:24.202 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7796000000000001e-05 2021-03-15T22:28:24.202 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.202 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.203 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.203 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164862+0000", 2021-03-15T22:28:24.203 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2391e-05 2021-03-15T22:28:24.203 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.203 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.204 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.204 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164884+0000", 2021-03-15T22:28:24.204 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047722025000000001 2021-03-15T22:28:24.204 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.204 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.205 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:24.205 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.212606+0000", 2021-03-15T22:28:24.205 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0160999999999997e-05 2021-03-15T22:28:24.205 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.206 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.206 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.206 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.212647+0000", 2021-03-15T22:28:24.206 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048217024999999997 2021-03-15T22:28:24.206 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.207 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.207 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.207 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.208 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:4285 7.b e445/401 7:d4368cae:::smithi07694878-18:head v 445'397, mlcod=445'394)", 2021-03-15T22:28:24.208 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.165543+0000", 2021-03-15T22:28:24.208 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 143.64791812499999, 2021-03-15T22:28:24.208 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047128319000000002, 2021-03-15T22:28:24.208 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:24.209 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:24.209 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:24.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.209 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:24.209 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165543+0000", 2021-03-15T22:28:24.210 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:24.210 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.210 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.210 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:24.211 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165543+0000", 2021-03-15T22:28:24.211 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.305e-06 2021-03-15T22:28:24.211 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.211 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.211 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:24.212 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165546+0000", 2021-03-15T22:28:24.212 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9570000000000001e-06 2021-03-15T22:28:24.212 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.212 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.212 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:24.213 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165548+0000", 2021-03-15T22:28:24.213 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4930000000000003e-06 2021-03-15T22:28:24.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.214 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:24.214 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165554+0000", 2021-03-15T22:28:24.214 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5200000000000002e-06 2021-03-15T22:28:24.214 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.214 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:24.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165558+0000", 2021-03-15T22:28:24.215 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0366999999999999e-05 2021-03-15T22:28:24.215 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.216 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:24.216 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165588+0000", 2021-03-15T22:28:24.216 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3482999999999999e-05 2021-03-15T22:28:24.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.217 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.217 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:24.217 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165602+0000", 2021-03-15T22:28:24.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047050275000000003 2021-03-15T22:28:24.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.218 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.218 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:24.218 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.212652+0000", 2021-03-15T22:28:24.218 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9919e-05 2021-03-15T22:28:24.219 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:24.219 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:24.219 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:24.219 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.212672+0000", 2021-03-15T22:28:24.219 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047128319000000002 2021-03-15T22:28:24.220 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.220 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.220 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.220 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:24.221 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:24.221 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:28:24.221 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:24.222 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:24 smithi076 conmon[62156]: cluster 2021-03-15T22:28: 2021-03-15T22:28:24.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:24 smithi076 conmon[62156]: 23.104891+0000 mgr.y (mgr.14144) 753 : cluster [DBG] pgmap v982: 138 pgs: 1 peering, 12 active+clean+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 53 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s; 343/1743 objects degraded (19.679%); 52/1743 objects misplaced (2.983%); 9.0 MiB/s, 23 objects/s recovering 2021-03-15T22:28:24.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:24.248 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:24.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:24.292 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:28:24.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:24.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:24.376 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:24.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:24.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-15T22:28:24.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:24 smithi125 conmon[56739]: cluster 2021-03-15T22:28:23.104891+0000 mgr.y (mgr.14144) 753 : cluster [DBG] pgmap v982: 138 pgs: 1 peering, 12 active+clean+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 53 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s; 343/1743 objects degraded (19.679%); 52/1743 objects misplaced (2.983%); 9.0 MiB/s, 23 objects/s recovering 2021-03-15T22:28:24.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:24.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:24.503 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:24.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:24.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:24 smithi076 conmon[56514]: cluster 2021-03-15T22:28: 2021-03-15T22:28:24.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:24 smithi076 conmon[56514]: 23.104891+0000 mgr.y (mgr.14144) 753 : cluster [DBG] pgmap v982: 138 pgs: 1 peering, 12 active+clean+remapped, 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 53 active+clean; 568 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 13 KiB/s rd, 0 B/s wr, 21 op/s; 343/1743 objects degraded (19.679%); 52/1743 objects misplaced (2.983%); 9.0 MiB/s, 23 objects/s recovering 2021-03-15T22:28:24.604 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:24.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:24.628 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:24.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:24.710 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:28:24.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:24.753 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:24.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:24.754 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:24.845 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:24.850 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:28:24.855 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:24.871 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:28:24.880 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:24.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:24.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:24.891 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:28:24.912 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:28:24.932 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:28:24.949 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:28:24.949 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:28:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:28:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:28:24.950 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:28:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:28:24.951 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:28:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:28:24.952 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:28:24.983 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:25.009 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:25.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:25.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:25.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:25.134 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:25.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:25.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:25.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:25.260 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:25.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:25.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:25.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:25 smithi076 conmon[56514]: audit 2021-03-15T22:28:24.829874+0000 mon.c (mon.1) 48 : audit [DBG] from='client.? 172.21.15.76:0/2075491340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:25.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:25 smithi076 conmon[56514]: cluster 2021-03-15T22:28:25.159555+0000 mon.a (mon.0) 1576 : cluster [WRN] Health check update: Degraded data redundancy: 343/1743 objects degraded (19.679%), 25 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:25.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:25 smithi076 conmon[62156]: audit 2021-03-15T22:28:24.829874+0000 mon.c (mon.1) 48 : audit [DBG] from='client.? 172.21.15.76:0/2075491340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:25.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:25 smithi076 conmon[62156]: cluster 2021-03-15T22:28:25.159555+0000 mon.a (mon.0) 1576 : cluster [WRN] Health check update: Degraded data redundancy: 343/1743 objects degraded (19.679%), 25 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:25.363 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:25.385 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:25.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:25.387 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:25.488 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:25.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:25.511 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:25.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:25.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:25 smithi125 conmon[56739]: audit 2021-03-15T22:28:24.829874+0000 mon.c (mon.1) 48 : audit [DBG] from='client.? 172.21.15.76:0/2075491340' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:25 smithi125 conmon[56739]: cluster 2021-03-15T22:28:25.159555+0000 mon.a (mon.0) 1576 : cluster [WRN] Health check update: Degraded data redundancy: 343/1743 objects degraded (19.679%), 25 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:25.613 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:25.635 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:25.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:25.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:25.738 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:25.740 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:28:25.740 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:28:25.803 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:25.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:25.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:25.906 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:25.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:25.930 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:25.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:26.032 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:26.054 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:26.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:26.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:26.157 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:26.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:26.180 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:26.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:26.218 INFO:teuthology.orchestra.run.smithi076.stderr:clear 3.3 pg_upmap mapping 2021-03-15T22:28:26.249 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:28:26.250 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:28:26.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:26 smithi076 conmon[56514]: cluster 2021-03-15T22:28:25 2021-03-15T22:28:26.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:26 smithi076 conmon[56514]: .105521+0000 mgr.y (mgr.14144) 754 : cluster [DBG] pgmap v983: 138 pgs: 1 peering, 13 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 53 active+clean; 567 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s; 343/1740 objects degraded (19.713%); 63/1740 objects misplaced (3.621%); 8.9 MiB/s, 21 objects/s recovering 2021-03-15T22:28:26.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:26 smithi076 conmon[56514]: audit 2021-03-15T22:28:25.672203+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? 172.21.15.76:0/2610478513' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.3"}]: dispatch 2021-03-15T22:28:26.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:26 smithi076 conmon[62156]: cluster 2021-03-15T22:28:25.105521+0000 mgr.y (mgr. 2021-03-15T22:28:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:26 smithi076 conmon[62156]: 14144) 754 : cluster [DBG] pgmap v983: 138 pgs: 1 peering, 13 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 53 active+clean; 567 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s; 343/1740 objects degraded (19.713%); 63/1740 objects misplaced (3.621%); 8.9 MiB/s, 21 objects/s recovering 2021-03-15T22:28:26.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:26 smithi076 conmon[62156]: audit 2021-03-15T22:28:25.672203+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? 172.21.15.76:0/2610478513' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.3"}]: dispatch 2021-03-15T22:28:26.282 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:26 smithi125 conmon[56739]: cluster 2021-03-15T22:28:25. 2021-03-15T22:28:26.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:26 smithi125 conmon[56739]: 105521+0000 mgr.y (mgr.14144) 754 : cluster [DBG] pgmap v983: 138 pgs: 1 peering, 13 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 53 active+clean; 567 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 5.2 KiB/s rd, 0 B/s wr, 7 op/s; 343/1740 objects degraded (19.713%); 63/1740 objects misplaced (3.621%); 8.9 MiB/s, 21 objects/s recovering 2021-03-15T22:28:26.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:26 smithi125 conmon[56739]: audit 2021-03-15T22:28:25.672203+0000 mon.a (mon.0) 1577 : audit [INF] from='client.? 172.21.15.76:0/2610478513' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap", "pgid": "3.3"}]: dispatch 2021-03-15T22:28:26.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:26.343 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:26.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:26.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:26.467 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:26.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:26.469 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:26.570 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:26.591 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:26.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:26.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:26.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:26.718 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:26.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:26.819 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:26.842 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:26.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:26.843 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:26.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:26.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:26.970 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:26.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:26.973 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-15T22:28:27.072 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:27.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:27.095 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:27.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:27.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:27.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:27.228 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:27.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:27.284 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:28:27.330 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:27.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:27.353 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:27.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:27.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[56514]: audit 2021-03-15T22:28:26.205165+0000 mon.a (mon.0) 1578 : audit [INF] from='client.? 172.21.15.76:0/2610478513' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "3.3"}]': finished 2021-03-15T22:28:27.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[56514]: cluster 2021-03-15T22:28:26.205207+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e562: 8 total, 6 up, 8 in 2021-03-15T22:28:27.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: audit 2021-03-15T22:28:27.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:28:27.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: T22:28:26.205165+0000 mon.a (mon.0) 1578 : audit [INF] from='client.? 172.21.15.76:0/2610478513' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "3.3"}]': finished 2021-03-15T22:28:27.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: cluster 2021-03-15T22:28:27.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: 2021-03-15T22:28:27.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: 2021 2021-03-15T22:28:27.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: - 2021-03-15T22:28:27.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: 03 2021-03-15T22:28:27.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: - 2021-03-15T22:28:27.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: 15 2021-03-15T22:28:27.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: T22:28:26.205207+0000 2021-03-15T22:28:27.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:27 smithi076 conmon[62156]: mon.a (mon.0) 1579 : cluster [DBG] osdmap e562: 8 total, 6 up, 8 in 2021-03-15T22:28:27.455 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:27.478 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:27.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:27.480 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:27.523 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:27 smithi125 conmon[56739]: audit 2021-03-15T22:28:26.205165+0000 mon.a (mon.0) 1578 : audit [INF] from='client.? 172.21.15.76:0/2610478513' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap", "pgid": "3.3"}]': finished 2021-03-15T22:28:27.524 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:27 smithi125 conmon[56739]: cluster 2021-03-15T22:28:26.205207+0000 mon.a (mon.0) 1579 : cluster [DBG] osdmap e562: 8 total, 6 up, 8 in 2021-03-15T22:28:27.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:27.604 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:27.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:27.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:27.707 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:27.730 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:27.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:27.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:27.741 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:28:27.832 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:27.855 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:27.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:27.856 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:27.957 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:27.980 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:27.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:27.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:28.051 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:28.083 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:28.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:28.108 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:28.108 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:28.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:28.221 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: cluster 2021-03-15T22:28:28.222 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:28:28.222 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: 28:27.106471+0000 mgr.y (mgr.14144) 755 : cluster [DBG] pgmap v985: 138 pgs: 13 active+clean+remapped, 4 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 737 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%); 15 MiB/s, 27 objects/s recovering 2021-03-15T22:28:28.222 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:28:28.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: 22:28:27.211341+0000 mon.a (mon. 2021-03-15T22:28:28.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: 0) 1580 : cluster [DBG] osdmap e563: 8 total, 6 up, 8 in 2021-03-15T22:28:28.223 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: audit 2021 2021-03-15T22:28:28.224 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: -03-15T22:28:27. 2021-03-15T22:28:28.224 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: 811050+0000 mon.c (mon.1) 49 : 2021-03-15T22:28:28.224 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/150951873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:28.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:28:28.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: 03-15T22:28:27.811364 2021-03-15T22:28:28.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1581 : 2021-03-15T22:28:28.225 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[62156]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:28.226 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:28:28.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:28.250 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:28.251 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:28.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:28.375 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:28.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:28.377 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:28.478 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:28.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:28.503 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:28.503 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[56514]: cluster 2021-03-15T22:28: 2021-03-15T22:28:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[56514]: 27.106471+0000 mgr.y (mgr.14144) 755 : cluster [DBG] pgmap v985: 138 pgs: 13 active+clean+remapped, 4 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 737 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%); 15 MiB/s, 27 objects/s recovering 2021-03-15T22:28:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[56514]: cluster 2021-03-15T22:28:27.211341+0000 mon.a (mon.0) 1580 : cluster [DBG] osdmap e563: 8 total, 6 up, 8 in 2021-03-15T22:28:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[56514]: audit 2021-03-15T22:28:27.811050+0000 mon.c (mon.1) 49 : audit [INF] from='client.? 172.21.15.76:0/150951873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:28 smithi076 conmon[56514]: audit 2021-03-15T22:28:27.811364+0000 mon.a (mon.0) 1581 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:28.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: cluster 2021-03-15T22:28:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 2021-03-15T22:28:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 2021 2021-03-15T22:28:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: - 2021-03-15T22:28:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 03 2021-03-15T22:28:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: - 2021-03-15T22:28:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 15 2021-03-15T22:28:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: T 2021-03-15T22:28:28.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 22 2021-03-15T22:28:28.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: : 2021-03-15T22:28:28.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 28 2021-03-15T22:28:28.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: : 2021-03-15T22:28:28.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 27 2021-03-15T22:28:28.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: . 2021-03-15T22:28:28.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 106471 2021-03-15T22:28:28.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: +0000 2021-03-15T22:28:28.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 2021-03-15T22:28:28.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:28.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: ( 2021-03-15T22:28:28.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: mgr.14144) 755 : cluster [DBG] pgmap v985: 138 pgs: 13 active+clean+remapped, 4 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 737 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%); 15 MiB/s, 27 objects/s recovering 2021-03-15T22:28:28.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 2021-03-15T22:28:28.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:28:28.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: -15T22:28:27.211341+0000 mon.a 2021-03-15T22:28:28.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: (mon.0) 1580 : cluster [DBG] osdmap e563: 8 total, 6 up, 8 in 2021-03-15T22:28:28.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 2021-03-15T22:28:28.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:28:28.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 22:28:27.811050+0000 mon.c (mon. 2021-03-15T22:28:28.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 1) 49 : audit [INF] from='client.? 172.21.15.76:0/150951873' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:28.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:28:28.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: 15T22:28:27.811364+0000 mon.a (mon 2021-03-15T22:28:28.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:28 smithi125 conmon[56739]: .0) 1581 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:28.604 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:28.626 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:28.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:28.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:28.729 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:28.750 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:28.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:28.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:28.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:28.877 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:28.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:28.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:28.916 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:28:28.916 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:28:28.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:29.004 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:29.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:29.005 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:29.107 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:29.129 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:29.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:29.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:29.231 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:29.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:29.255 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:29.255 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:29.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:29 smithi076 conmon[56514]: cluster 2021-03-15T22:28:28.207396+0000 mon.a (mon.0) 1582 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:29.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:29 smithi076 conmon[56514]: audit 2021-03-15T22:28:28.212107+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:29.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:29 smithi076 conmon[56514]: cluster 2021-03-15T22:28:28.212195+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e564: 8 total, 6 up, 8 in 2021-03-15T22:28:29.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:29 smithi076 conmon[62156]: cluster 2021-03-15T22:28:28.207396+0000 mon.a (mon.0) 1582 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:29.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:29 smithi076 conmon[62156]: audit 2021-03-15T22:28:28.212107+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:29.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:29 smithi076 conmon[62156]: cluster 2021-03-15T22:28:28.212195+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e564: 8 total, 6 up, 8 in 2021-03-15T22:28:29.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:29 smithi125 conmon[56739]: cluster 2021-03-15T22:28:28.207396+0000 mon.a (mon.0) 1582 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:29.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:29 smithi125 conmon[56739]: audit 2021-03-15T22:28:28.212107+0000 mon.a (mon.0) 1583 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:29.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:29 smithi125 conmon[56739]: cluster 2021-03-15T22:28:28.212195+0000 mon.a (mon.0) 1584 : cluster [DBG] osdmap e564: 8 total, 6 up, 8 in 2021-03-15T22:28:29.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:29.378 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:29.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:29.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:29.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:29.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:29.504 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:29.505 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:29.579 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-15T22:28:29.605 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:29.628 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:29.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:29.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:29.730 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:29.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:29.754 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:29.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:29.852 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:28:29.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:29.879 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:29.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:29.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:29.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:29.994 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:28:30.004 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:30.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:30.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:30.010 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:28:30.027 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:28:30.045 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:28:30.062 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:28:30.080 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:28:30.080 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:28:30.081 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:28:30.081 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:28:30.082 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:28:30.082 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:28:30.083 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:28:30.083 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:28:30.083 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:28:30.106 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:30.131 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:30.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:30.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:30.235 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:30.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:30.258 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:30.258 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:30.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:28:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:28:30.359 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:30.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:30.383 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:30.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:30.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:30.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:30.509 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:30.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:30.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[56514]: cluster 2021-03-15T22:28:29. 2021-03-15T22:28:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[56514]: 107231+0000 mgr.y (mgr.14144) 756 : cluster [DBG] pgmap v988: 138 pgs: 13 active+clean+remapped, 4 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%); 13 MiB/s, 15 objects/s recovering 2021-03-15T22:28:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[56514]: audit 2021-03-15T22:28:29.882303+0000 mgr.y (mgr.14144) 757 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[56514]: audit 2021-03-15T22:28:29.975309+0000 mon.c (mon.1) 50 : audit [DBG] from='client.? 172.21.15.76:0/206128188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[56514]: cluster 2021-03-15T22:28:30.161003+0000 mon.a (mon.0) 1585 : cluster [WRN] Health check update: Degraded data redundancy: 307/1740 objects degraded (17.644%), 24 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[56514]: cluster 2021-03-15T22:28:30.169046+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e565: 8 total, 6 up, 8 in 2021-03-15T22:28:30.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[62156]: cluster 2021-03-15T22:28:29.107231+0000 mgr.y (mgr.14144) 756 : cluster [DBG] pgmap v988: 138 pgs: 13 active+clean+remapped, 4 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%); 13 MiB/s, 15 objects/s recovering 2021-03-15T22:28:30.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[62156]: audit 2021-03-15T22:28:29.882303+0000 mgr.y (mgr.14144) 757 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:30.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[62156]: audit 2021-03-15T22:28:29.975309+0000 mon.c (mon.1) 50 : audit [DBG] from='client.? 172.21.15.76:0/206128188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[62156]: cluster 2021-03-15T22:28:30.161003+0000 mon.a (mon.0) 1585 : cluster [WRN] Health check update: Degraded data redundancy: 307/1740 objects degraded (17.644%), 24 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:28:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:30 smithi076 conmon[62156]: -03-15T22:28:30.169046+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e565: 8 total, 6 up, 8 in 2021-03-15T22:28:30.611 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:30.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:30.637 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:30.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:30.739 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:30.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:30.763 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:30.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:30.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:30 smithi125 conmon[56739]: cluster 2021-03-15T22:28:29. 2021-03-15T22:28:30.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:30 smithi125 conmon[56739]: 107231+0000 mgr.y (mgr.14144) 756 : cluster [DBG] pgmap v988: 138 pgs: 13 active+clean+remapped, 4 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%); 13 MiB/s, 15 objects/s recovering 2021-03-15T22:28:30.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:30 smithi125 conmon[56739]: audit 2021-03-15T22:28:29.882303+0000 mgr.y (mgr.14144) 757 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:30.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:30 smithi125 conmon[56739]: audit 2021-03-15T22:28:29.975309+0000 mon.c (mon.1) 50 : audit [DBG] from='client.? 172.21.15.76:0/206128188' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:30.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:30 smithi125 conmon[56739]: cluster 2021-03-15T22:28:30.161003+0000 mon.a (mon.0) 1585 : cluster [WRN] Health check update: Degraded data redundancy: 307/1740 objects degraded (17.644%), 24 pgs degraded, 25 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:30.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:30 smithi125 conmon[56739]: cluster 2021-03-15T22:28:30.169046+0000 mon.a (mon.0) 1586 : cluster [DBG] osdmap e565: 8 total, 6 up, 8 in 2021-03-15T22:28:30.798 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:28:30.799 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:28:30.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:30.889 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:30.891 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:30.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:31.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:28:31.015 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:31.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:31.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:31.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:31.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:31.143 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:31.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:31.245 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:31.267 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:31.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:31.287 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:31.348 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:28:31.349 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:28:31.349 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:28:31.349 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:28:31.349 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.350 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:1359 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94751391777792 gen 3] snapc 0=[] ondisk+write+known_if_redirected e316)", 2021-03-15T22:28:31.350 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:52.068429+0000", 2021-03-15T22:28:31.350 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 279.27639527899998, 2021-03-15T22:28:31.350 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.20407250399999999, 2021-03-15T22:28:31.350 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.351 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:31.351 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.351 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:28:31.351 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:28:31.352 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1359 2021-03-15T22:28:31.352 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.352 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.352 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.352 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.353 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068429+0000", 2021-03-15T22:28:31.353 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.353 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.353 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.354 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.354 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068429+0000", 2021-03-15T22:28:31.354 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.123e-06 2021-03-15T22:28:31.354 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.354 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.355 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.355 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068432+0000", 2021-03-15T22:28:31.355 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.066e-06 2021-03-15T22:28:31.355 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.356 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.356 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.356 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068435+0000", 2021-03-15T22:28:31.356 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5502000000000002e-05 2021-03-15T22:28:31.356 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.357 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.357 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.357 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068451+0000", 2021-03-15T22:28:31.357 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.348e-06 2021-03-15T22:28:31.357 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.358 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.358 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.358 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068457+0000", 2021-03-15T22:28:31.358 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0260999999999998e-05 2021-03-15T22:28:31.359 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.359 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.359 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.359 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068517+0000", 2021-03-15T22:28:31.359 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1440999999999999e-05 2021-03-15T22:28:31.360 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.360 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.360 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:31.360 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068529+0000", 2021-03-15T22:28:31.361 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19618954299999999 2021-03-15T22:28:31.361 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.361 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.361 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.362 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.264718+0000", 2021-03-15T22:28:31.362 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000154158 2021-03-15T22:28:31.362 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.362 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.362 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.363 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.264872+0000", 2021-03-15T22:28:31.363 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076290619999999998 2021-03-15T22:28:31.363 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.363 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.364 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.364 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.272501+0000", 2021-03-15T22:28:31.364 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.20407250399999999 2021-03-15T22:28:31.364 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.364 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.365 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.365 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.365 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.365 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:1360 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94751391778944 gen 4] snapc 0=[] ondisk+write+known_if_redirected e316)", 2021-03-15T22:28:31.365 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:52.068469+0000", 2021-03-15T22:28:31.366 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 279.27635466300001, 2021-03-15T22:28:31.366 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19939890299999999, 2021-03-15T22:28:31.366 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.366 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:31.367 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.367 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:28:31.367 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:28:31.367 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1360 2021-03-15T22:28:31.367 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.368 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.368 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.368 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.368 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068469+0000", 2021-03-15T22:28:31.369 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.369 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.369 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.369 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.369 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068469+0000", 2021-03-15T22:28:31.370 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5300000000000003e-07 2021-03-15T22:28:31.370 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.370 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.370 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.370 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068470+0000", 2021-03-15T22:28:31.371 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1710000000000001e-06 2021-03-15T22:28:31.371 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.371 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.371 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.372 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068472+0000", 2021-03-15T22:28:31.372 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3089999999999997e-06 2021-03-15T22:28:31.372 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.372 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.372 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.373 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068476+0000", 2021-03-15T22:28:31.373 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.489e-06 2021-03-15T22:28:31.373 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.373 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.373 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.374 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068479+0000", 2021-03-15T22:28:31.374 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1756999999999998e-05 2021-03-15T22:28:31.374 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.374 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.374 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.375 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068541+0000", 2021-03-15T22:28:31.375 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.311e-05 2021-03-15T22:28:31.375 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.375 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.376 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:31.376 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068554+0000", 2021-03-15T22:28:31.376 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19462378499999999 2021-03-15T22:28:31.376 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.376 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.377 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.377 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.263178+0000", 2021-03-15T22:28:31.377 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000179648 2021-03-15T22:28:31.377 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.378 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.378 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.378 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.263358+0000", 2021-03-15T22:28:31.378 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045107810000000002 2021-03-15T22:28:31.378 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.379 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.379 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.379 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.267868+0000", 2021-03-15T22:28:31.379 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19939890299999999 2021-03-15T22:28:31.379 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.380 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.380 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.380 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.380 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.380 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34361.0:352 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e316)", 2021-03-15T22:28:31.381 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:52.097658+0000", 2021-03-15T22:28:31.381 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 279.24716599599998, 2021-03-15T22:28:31.381 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.17297211700000001, 2021-03-15T22:28:31.381 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.382 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:31.382 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.382 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34361", 2021-03-15T22:28:31.382 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.125:0/1442040869", 2021-03-15T22:28:31.382 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 352 2021-03-15T22:28:31.383 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.383 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.383 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.383 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.384 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097658+0000", 2021-03-15T22:28:31.384 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.384 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.384 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.384 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.385 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097658+0000", 2021-03-15T22:28:31.385 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2650000000000001e-06 2021-03-15T22:28:31.385 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.385 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.386 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.386 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097661+0000", 2021-03-15T22:28:31.386 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0199999999999999e-06 2021-03-15T22:28:31.386 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.387 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.387 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.387 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097664+0000", 2021-03-15T22:28:31.387 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6565000000000001e-05 2021-03-15T22:28:31.388 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.388 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.388 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.388 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097681+0000", 2021-03-15T22:28:31.388 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9939999999999997e-06 2021-03-15T22:28:31.389 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.389 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.389 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.389 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097687+0000", 2021-03-15T22:28:31.390 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2488000000000003e-05 2021-03-15T22:28:31.390 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.390 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.390 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.390 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097739+0000", 2021-03-15T22:28:31.391 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0113e-05 2021-03-15T22:28:31.391 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.391 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.391 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:31.392 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097750+0000", 2021-03-15T22:28:31.392 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.172684848 2021-03-15T22:28:31.392 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.392 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.393 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.393 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.270434+0000", 2021-03-15T22:28:31.393 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9377999999999998e-05 2021-03-15T22:28:31.393 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.393 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.394 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.394 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.270514+0000", 2021-03-15T22:28:31.394 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000116446 2021-03-15T22:28:31.394 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.395 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.396 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.396 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.270630+0000", 2021-03-15T22:28:31.396 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.17297211700000001 2021-03-15T22:28:31.396 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.396 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.397 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.397 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.397 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.397 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3289 7.f 7:fc7cd984:::smithi07694878-4:head [delete] snapc fd=[fd,fc,fb,fa,f9,f7,f5,e9,e7,de] ondisk+write+peerstat_old+known_if_redirected e355)", 2021-03-15T22:28:31.398 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:30.295366+0000", 2021-03-15T22:28:31.398 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 241.04945782300001, 2021-03-15T22:28:31.398 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8092550220000001, 2021-03-15T22:28:31.398 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.398 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:31.399 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.399 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.399 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.399 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3289 2021-03-15T22:28:31.400 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.400 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.400 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.400 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.400 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295366+0000", 2021-03-15T22:28:31.401 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.401 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.401 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.401 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.402 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295366+0000", 2021-03-15T22:28:31.402 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8169999999999999e-06 2021-03-15T22:28:31.402 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.402 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.403 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.403 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295374+0000", 2021-03-15T22:28:31.403 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6029999999999999e-06 2021-03-15T22:28:31.403 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.403 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.404 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.404 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295382+0000", 2021-03-15T22:28:31.404 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1372000000000003e-05 2021-03-15T22:28:31.404 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.404 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.405 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.405 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295423+0000", 2021-03-15T22:28:31.405 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1537000000000001e-05 2021-03-15T22:28:31.405 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.406 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.406 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.406 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295445+0000", 2021-03-15T22:28:31.406 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012213400000000001 2021-03-15T22:28:31.407 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.407 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.407 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.407 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295567+0000", 2021-03-15T22:28:31.407 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1673000000000001e-05 2021-03-15T22:28:31.408 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.408 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.408 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:31.408 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295588+0000", 2021-03-15T22:28:31.408 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79131791699999998 2021-03-15T22:28:31.409 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.409 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.409 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.409 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.086906+0000", 2021-03-15T22:28:31.410 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0113e-05 2021-03-15T22:28:31.410 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.410 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.410 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:31.411 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.086916+0000", 2021-03-15T22:28:31.411 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0123092789999999 2021-03-15T22:28:31.411 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.411 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.411 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.412 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.099226+0000", 2021-03-15T22:28:31.412 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016286 2021-03-15T22:28:31.412 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.412 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.413 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for missing object", 2021-03-15T22:28:31.413 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.099389+0000", 2021-03-15T22:28:31.413 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001614868 2021-03-15T22:28:31.413 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.413 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.414 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.414 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.101003+0000", 2021-03-15T22:28:31.414 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023411800000000001 2021-03-15T22:28:31.414 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.415 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.415 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.415 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.101238+0000", 2021-03-15T22:28:31.415 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033837310000000001 2021-03-15T22:28:31.415 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.416 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.416 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.416 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.104621+0000", 2021-03-15T22:28:31.416 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8092550220000001 2021-03-15T22:28:31.417 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.417 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.417 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.417 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.417 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.418 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34361.0:390 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e354)", 2021-03-15T22:28:31.418 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:30.317842+0000", 2021-03-15T22:28:31.418 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 241.026981979, 2021-03-15T22:28:31.418 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79261869600000001, 2021-03-15T22:28:31.419 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.419 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:31.419 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.419 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34361", 2021-03-15T22:28:31.420 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.125:0/1442040869", 2021-03-15T22:28:31.420 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 390 2021-03-15T22:28:31.420 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.420 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.420 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.421 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.421 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317842+0000", 2021-03-15T22:28:31.421 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.421 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.421 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.422 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.422 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317842+0000", 2021-03-15T22:28:31.422 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2830000000000004e-06 2021-03-15T22:28:31.422 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.423 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.423 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.423 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317849+0000", 2021-03-15T22:28:31.423 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2929999999999997e-06 2021-03-15T22:28:31.423 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.424 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.424 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.424 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317856+0000", 2021-03-15T22:28:31.424 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2073000000000003e-05 2021-03-15T22:28:31.425 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.425 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.425 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.425 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317888+0000", 2021-03-15T22:28:31.426 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7453e-05 2021-03-15T22:28:31.426 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.426 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.426 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.426 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317905+0000", 2021-03-15T22:28:31.427 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015652 2021-03-15T22:28:31.427 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.427 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.427 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.428 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.318062+0000", 2021-03-15T22:28:31.428 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8180000000000004e-06 2021-03-15T22:28:31.428 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.428 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.428 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:31.429 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.318070+0000", 2021-03-15T22:28:31.429 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79211001599999997 2021-03-15T22:28:31.429 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.429 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.430 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.430 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.110180+0000", 2021-03-15T22:28:31.430 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000129782 2021-03-15T22:28:31.430 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.430 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.431 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.431 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.110309+0000", 2021-03-15T22:28:31.431 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015145800000000001 2021-03-15T22:28:31.431 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.431 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.432 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.432 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.110461+0000", 2021-03-15T22:28:31.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79261869600000001 2021-03-15T22:28:31.432 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.433 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.433 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.433 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.433 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.433 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3664 7.6 7:608078b0:::smithi07694878-12:head [delete] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+peerstat_old+known_if_redirected e385)", 2021-03-15T22:28:31.434 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.131030+0000", 2021-03-15T22:28:31.434 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 211.213794393, 2021-03-15T22:28:31.434 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039032994000000001, 2021-03-15T22:28:31.434 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.435 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:31.435 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.435 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.435 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.435 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3664 2021-03-15T22:28:31.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.436 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.436 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.437 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131030+0000", 2021-03-15T22:28:31.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.437 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.437 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.437 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.438 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131030+0000", 2021-03-15T22:28:31.438 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9670000000000003e-06 2021-03-15T22:28:31.438 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.439 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131034+0000", 2021-03-15T22:28:31.439 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5580000000000002e-06 2021-03-15T22:28:31.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.440 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131038+0000", 2021-03-15T22:28:31.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0582e-05 2021-03-15T22:28:31.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.441 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.441 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131059+0000", 2021-03-15T22:28:31.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1455e-05 2021-03-15T22:28:31.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.442 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.442 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131070+0000", 2021-03-15T22:28:31.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3596999999999994e-05 2021-03-15T22:28:31.443 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.443 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131134+0000", 2021-03-15T22:28:31.444 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8918999999999994e-05 2021-03-15T22:28:31.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.444 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131213+0000", 2021-03-15T22:28:31.445 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000169662 2021-03-15T22:28:31.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.445 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:28:31.446 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131382+0000", 2021-03-15T22:28:31.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001594575 2021-03-15T22:28:31.446 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.446 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.447 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.447 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.132977+0000", 2021-03-15T22:28:31.447 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035954851000000003 2021-03-15T22:28:31.447 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.448 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.448 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.168932+0000", 2021-03-15T22:28:31.448 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010296630000000001 2021-03-15T22:28:31.448 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.449 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.449 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:31.449 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.169962+0000", 2021-03-15T22:28:31.449 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2741000000000003e-05 2021-03-15T22:28:31.449 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.450 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:31.450 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.170004+0000", 2021-03-15T22:28:31.450 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8424000000000002e-05 2021-03-15T22:28:31.451 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.451 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.170063+0000", 2021-03-15T22:28:31.452 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039032994000000001 2021-03-15T22:28:31.452 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.452 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.452 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.453 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3679 7.6 7:6de8b070:::smithi07694878-15:head [append 0~0] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:28:31.453 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.195576+0000", 2021-03-15T22:28:31.453 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 211.149248043, 2021-03-15T22:28:31.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038774930999999999, 2021-03-15T22:28:31.454 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.454 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:31.454 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.454 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.455 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.455 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3679 2021-03-15T22:28:31.455 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.455 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.455 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.456 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195576+0000", 2021-03-15T22:28:31.456 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.457 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195576+0000", 2021-03-15T22:28:31.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1049999999999999e-06 2021-03-15T22:28:31.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195579+0000", 2021-03-15T22:28:31.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4559999999999999e-06 2021-03-15T22:28:31.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.463 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.463 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195582+0000", 2021-03-15T22:28:31.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2951e-05 2021-03-15T22:28:31.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.464 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.464 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.464 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195595+0000", 2021-03-15T22:28:31.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3730000000000001e-06 2021-03-15T22:28:31.464 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.465 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.465 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195600+0000", 2021-03-15T22:28:31.465 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6076999999999996e-05 2021-03-15T22:28:31.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.466 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.466 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.466 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195686+0000", 2021-03-15T22:28:31.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2466999999999996e-05 2021-03-15T22:28:31.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.467 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.467 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195759+0000", 2021-03-15T22:28:31.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016388000000000001 2021-03-15T22:28:31.468 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.468 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.468 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:28:31.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195922+0000", 2021-03-15T22:28:31.469 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0018566839999999999 2021-03-15T22:28:31.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.469 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:31.470 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197779+0000", 2021-03-15T22:28:31.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1146000000000001e-05 2021-03-15T22:28:31.470 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.470 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.470 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197830+0000", 2021-03-15T22:28:31.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036392299000000003 2021-03-15T22:28:31.471 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234223+0000", 2021-03-15T22:28:31.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6313e-05 2021-03-15T22:28:31.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.473 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:31.473 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234289+0000", 2021-03-15T22:28:31.473 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2180000000000004e-05 2021-03-15T22:28:31.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.474 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234351+0000", 2021-03-15T22:28:31.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038774930999999999 2021-03-15T22:28:31.474 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.475 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.475 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.475 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.475 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3680 7.6 7:6de8b070:::smithi07694878-15:head [rollback e7] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:28:31.476 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.195608+0000", 2021-03-15T22:28:31.476 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 211.14921609300001, 2021-03-15T22:28:31.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038942472999999998, 2021-03-15T22:28:31.476 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.477 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:31.477 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.477 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.477 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.477 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3680 2021-03-15T22:28:31.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.478 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195608+0000", 2021-03-15T22:28:31.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.479 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.479 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195608+0000", 2021-03-15T22:28:31.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0800000000000004e-07 2021-03-15T22:28:31.480 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.481 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195609+0000", 2021-03-15T22:28:31.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.015e-06 2021-03-15T22:28:31.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.482 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195610+0000", 2021-03-15T22:28:31.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4369999999999999e-06 2021-03-15T22:28:31.482 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.483 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.483 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195613+0000", 2021-03-15T22:28:31.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4880000000000001e-06 2021-03-15T22:28:31.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.484 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.484 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.484 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195616+0000", 2021-03-15T22:28:31.484 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001023368 2021-03-15T22:28:31.485 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.485 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.196639+0000", 2021-03-15T22:28:31.485 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5290000000000001e-05 2021-03-15T22:28:31.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.486 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.486 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.196694+0000", 2021-03-15T22:28:31.487 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014822899999999999 2021-03-15T22:28:31.487 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.487 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.487 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:28:31.487 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.196843+0000", 2021-03-15T22:28:31.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010197610000000001 2021-03-15T22:28:31.488 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.488 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.488 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.489 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197862+0000", 2021-03-15T22:28:31.489 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0350000000000001e-06 2021-03-15T22:28:31.489 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.489 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.489 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:31.490 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197867+0000", 2021-03-15T22:28:31.490 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036554689000000001 2021-03-15T22:28:31.490 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.490 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.491 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.491 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234422+0000", 2021-03-15T22:28:31.491 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2453000000000006e-05 2021-03-15T22:28:31.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.491 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.492 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:31.492 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234495+0000", 2021-03-15T22:28:31.492 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5999999999999999e-05 2021-03-15T22:28:31.492 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.492 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.493 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.493 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234551+0000", 2021-03-15T22:28:31.493 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038942472999999998 2021-03-15T22:28:31.493 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.493 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.494 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.494 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.494 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3681 7.6 7:6de8b070:::smithi07694878-15:head [append 0~0] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:28:31.495 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.195641+0000", 2021-03-15T22:28:31.495 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 211.14918308200001, 2021-03-15T22:28:31.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039044739000000002, 2021-03-15T22:28:31.495 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.495 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:31.496 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.496 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.496 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.496 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3681 2021-03-15T22:28:31.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.497 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.497 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.497 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195641+0000", 2021-03-15T22:28:31.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.498 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.499 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195641+0000", 2021-03-15T22:28:31.499 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6000000000000002e-07 2021-03-15T22:28:31.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.499 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.500 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195642+0000", 2021-03-15T22:28:31.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.23e-07 2021-03-15T22:28:31.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195643+0000", 2021-03-15T22:28:31.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0450000000000001e-06 2021-03-15T22:28:31.501 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.501 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.502 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.502 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195646+0000", 2021-03-15T22:28:31.502 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0839999999999999e-06 2021-03-15T22:28:31.502 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.503 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.503 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195648+0000", 2021-03-15T22:28:31.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015890559999999999 2021-03-15T22:28:31.503 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.504 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.504 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.504 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197237+0000", 2021-03-15T22:28:31.504 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4195000000000005e-05 2021-03-15T22:28:31.505 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.505 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.505 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.505 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197301+0000", 2021-03-15T22:28:31.505 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017653100000000001 2021-03-15T22:28:31.506 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.506 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.506 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:28:31.506 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197478+0000", 2021-03-15T22:28:31.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000394583 2021-03-15T22:28:31.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.507 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.507 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:31.508 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197872+0000", 2021-03-15T22:28:31.508 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024062000000000001 2021-03-15T22:28:31.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.508 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.509 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.198113+0000", 2021-03-15T22:28:31.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036489966999999998 2021-03-15T22:28:31.509 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.509 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.509 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.510 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234603+0000", 2021-03-15T22:28:31.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0292e-05 2021-03-15T22:28:31.510 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.510 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.511 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:31.511 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234643+0000", 2021-03-15T22:28:31.511 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2783e-05 2021-03-15T22:28:31.511 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.512 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.512 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234686+0000", 2021-03-15T22:28:31.512 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039044739000000002 2021-03-15T22:28:31.512 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.513 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.513 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.513 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.513 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.513 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4380 7.6 7:6283974b:::smithi07694878-30:head [write 484200~624934,stat] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:28:31.514 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:28:31.514 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.21397346200001, 2021-03-15T22:28:31.514 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.122158787, 2021-03-15T22:28:31.515 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.515 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:31.515 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.515 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.516 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.516 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4380 2021-03-15T22:28:31.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.516 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.517 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.517 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:28:31.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.517 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.518 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.518 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:28:31.518 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00065488299999999998 2021-03-15T22:28:31.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.519 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131506+0000", 2021-03-15T22:28:31.519 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.037e-06 2021-03-15T22:28:31.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.520 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131512+0000", 2021-03-15T22:28:31.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9637000000000001e-05 2021-03-15T22:28:31.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.521 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131541+0000", 2021-03-15T22:28:31.521 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9325000000000001e-05 2021-03-15T22:28:31.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131571+0000", 2021-03-15T22:28:31.523 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6747000000000002e-05 2021-03-15T22:28:31.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131647+0000", 2021-03-15T22:28:31.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000176789 2021-03-15T22:28:31.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.524 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.525 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131824+0000", 2021-03-15T22:28:31.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00042344700000000003 2021-03-15T22:28:31.525 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.525 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:28:31.526 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.132248+0000", 2021-03-15T22:28:31.526 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0026234660000000001 2021-03-15T22:28:31.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.526 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.527 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.134871+0000", 2021-03-15T22:28:31.527 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031408100000000002 2021-03-15T22:28:31.527 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.527 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.528 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:31.528 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.135185+0000", 2021-03-15T22:28:31.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.117673658 2021-03-15T22:28:31.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.529 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.529 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.252859+0000", 2021-03-15T22:28:31.529 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6511999999999997e-05 2021-03-15T22:28:31.529 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.529 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.530 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:31.530 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.252915+0000", 2021-03-15T22:28:31.530 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4204999999999997e-05 2021-03-15T22:28:31.530 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.531 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.531 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253009+0000", 2021-03-15T22:28:31.531 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.122158787 2021-03-15T22:28:31.531 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.532 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.532 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.532 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.532 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.532 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4381 7.6 7:6283974b:::smithi07694878-30:head [write 1860843~223357] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:28:31.533 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:28:31.533 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.208702515, 2021-03-15T22:28:31.533 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119715842, 2021-03-15T22:28:31.533 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.533 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:31.534 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.534 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.534 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.534 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4381 2021-03-15T22:28:31.535 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.535 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.535 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.535 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.535 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:28:31.536 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.536 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.536 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.536 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.536 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:28:31.537 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019479299999999999 2021-03-15T22:28:31.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.537 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.537 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.537 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136316+0000", 2021-03-15T22:28:31.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7419999999999999e-06 2021-03-15T22:28:31.538 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.538 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136320+0000", 2021-03-15T22:28:31.539 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5999999999999999e-05 2021-03-15T22:28:31.539 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.539 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.539 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136336+0000", 2021-03-15T22:28:31.540 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4386e-05 2021-03-15T22:28:31.540 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.541 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.541 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136361+0000", 2021-03-15T22:28:31.541 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4256999999999999e-05 2021-03-15T22:28:31.541 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.541 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.542 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.542 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136415+0000", 2021-03-15T22:28:31.542 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000158257 2021-03-15T22:28:31.542 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.543 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:31.543 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136573+0000", 2021-03-15T22:28:31.543 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.116481746 2021-03-15T22:28:31.543 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.543 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.544 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.544 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253055+0000", 2021-03-15T22:28:31.544 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1245e-05 2021-03-15T22:28:31.544 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.545 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.545 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253126+0000", 2021-03-15T22:28:31.545 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000138727 2021-03-15T22:28:31.545 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.546 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.546 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:28:31.546 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253265+0000", 2021-03-15T22:28:31.546 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0017125980000000001 2021-03-15T22:28:31.546 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.547 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:31.547 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254977+0000", 2021-03-15T22:28:31.547 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000112898 2021-03-15T22:28:31.548 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.548 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.548 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.548 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255090+0000", 2021-03-15T22:28:31.548 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00066560399999999998 2021-03-15T22:28:31.549 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.549 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.549 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.549 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255756+0000", 2021-03-15T22:28:31.549 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8887999999999998e-05 2021-03-15T22:28:31.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.550 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:31.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255795+0000", 2021-03-15T22:28:31.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2701000000000003e-05 2021-03-15T22:28:31.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.551 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.551 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255837+0000", 2021-03-15T22:28:31.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119715842 2021-03-15T22:28:31.552 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.552 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.552 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.553 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.553 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4382 7.6 7:6283974b:::smithi07694878-30:head [setxattr _header (58) in=7b,truncate 2084200] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:28:31.553 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:28:31.553 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.208453435, 2021-03-15T22:28:31.554 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119633089, 2021-03-15T22:28:31.554 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.554 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:31.554 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.554 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.555 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.555 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4382 2021-03-15T22:28:31.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.555 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:28:31.556 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.556 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.557 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.557 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:28:31.557 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4609999999999998e-06 2021-03-15T22:28:31.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.558 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.558 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136373+0000", 2021-03-15T22:28:31.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7880000000000001e-06 2021-03-15T22:28:31.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136375+0000", 2021-03-15T22:28:31.559 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2340000000000002e-06 2021-03-15T22:28:31.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.560 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.560 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136380+0000", 2021-03-15T22:28:31.560 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3160000000000001e-06 2021-03-15T22:28:31.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.561 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.561 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136384+0000", 2021-03-15T22:28:31.561 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024020700000000001 2021-03-15T22:28:31.562 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.562 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.562 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.562 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136624+0000", 2021-03-15T22:28:31.563 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7928000000000003e-05 2021-03-15T22:28:31.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.563 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:31.564 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136702+0000", 2021-03-15T22:28:31.564 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.11744721399999999 2021-03-15T22:28:31.564 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.564 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.564 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.565 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254149+0000", 2021-03-15T22:28:31.565 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5769000000000001e-05 2021-03-15T22:28:31.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.565 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.566 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254205+0000", 2021-03-15T22:28:31.566 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000127489 2021-03-15T22:28:31.566 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.566 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.567 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:28:31.567 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254332+0000", 2021-03-15T22:28:31.567 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00068339200000000001 2021-03-15T22:28:31.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.568 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:31.568 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255016+0000", 2021-03-15T22:28:31.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000233143 2021-03-15T22:28:31.568 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.568 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.569 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.569 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255249+0000", 2021-03-15T22:28:31.569 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00063040000000000004 2021-03-15T22:28:31.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.570 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.570 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255879+0000", 2021-03-15T22:28:31.570 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9571000000000003e-05 2021-03-15T22:28:31.571 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.571 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.571 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:31.571 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255939+0000", 2021-03-15T22:28:31.571 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5177000000000003e-05 2021-03-15T22:28:31.572 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.572 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.572 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.572 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256004+0000", 2021-03-15T22:28:31.572 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119633089 2021-03-15T22:28:31.573 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.573 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.573 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.573 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.574 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.574 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4383 7.6 7:6283974b:::smithi07694878-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e455)", 2021-03-15T22:28:31.574 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:28:31.574 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 141.20843570400001, 2021-03-15T22:28:31.574 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119776914, 2021-03-15T22:28:31.575 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.575 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:31.575 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.575 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.576 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.576 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4383 2021-03-15T22:28:31.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.576 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.576 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.577 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.577 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:28:31.577 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.578 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:28:31.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9899999999999999e-07 2021-03-15T22:28:31.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.579 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.579 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136389+0000", 2021-03-15T22:28:31.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1799999999999999e-06 2021-03-15T22:28:31.579 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.580 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.580 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.580 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136390+0000", 2021-03-15T22:28:31.580 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0630000000000002e-06 2021-03-15T22:28:31.581 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.581 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.581 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.581 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136396+0000", 2021-03-15T22:28:31.581 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6900000000000001e-06 2021-03-15T22:28:31.582 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.582 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.582 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136399+0000", 2021-03-15T22:28:31.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034116099999999998 2021-03-15T22:28:31.583 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.583 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.583 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.583 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136740+0000", 2021-03-15T22:28:31.584 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000123552 2021-03-15T22:28:31.584 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.584 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.584 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:31.584 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136864+0000", 2021-03-15T22:28:31.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.118007994 2021-03-15T22:28:31.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.585 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254872+0000", 2021-03-15T22:28:31.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0631999999999996e-05 2021-03-15T22:28:31.586 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:31.587 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254963+0000", 2021-03-15T22:28:31.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010704379999999999 2021-03-15T22:28:31.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256033+0000", 2021-03-15T22:28:31.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2941e-05 2021-03-15T22:28:31.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.588 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.589 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256086+0000", 2021-03-15T22:28:31.589 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9463999999999997e-05 2021-03-15T22:28:31.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.589 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.589 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.590 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256165+0000", 2021-03-15T22:28:31.590 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119776914 2021-03-15T22:28:31.590 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.590 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.591 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.591 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.591 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.591 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4396 7.6 7:61e0cc05:::smithi07694878-16:head [copy-from ver 767] snapc 161=[161,160,15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e457)", 2021-03-15T22:28:31.591 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:12.125558+0000", 2021-03-15T22:28:31.592 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 139.21926620900001, 2021-03-15T22:28:31.592 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035732501, 2021-03-15T22:28:31.592 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.592 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:31.592 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.593 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.593 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.593 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4396 2021-03-15T22:28:31.593 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.594 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.594 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.594 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125558+0000", 2021-03-15T22:28:31.594 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125558+0000", 2021-03-15T22:28:31.595 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2009999999999996e-06 2021-03-15T22:28:31.596 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.596 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.596 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125564+0000", 2021-03-15T22:28:31.597 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8043e-05 2021-03-15T22:28:31.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.597 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.597 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.598 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125582+0000", 2021-03-15T22:28:31.598 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4352e-05 2021-03-15T22:28:31.598 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.598 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.598 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.599 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125607+0000", 2021-03-15T22:28:31.599 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4769999999999992e-06 2021-03-15T22:28:31.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.599 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.600 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.600 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125616+0000", 2021-03-15T22:28:31.600 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047446260000000001 2021-03-15T22:28:31.600 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.601 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.130361+0000", 2021-03-15T22:28:31.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0346999999999996e-05 2021-03-15T22:28:31.601 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.601 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.602 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.602 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.130431+0000", 2021-03-15T22:28:31.602 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012214641 2021-03-15T22:28:31.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.603 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:28:31.603 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.142646+0000", 2021-03-15T22:28:31.603 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013352012 2021-03-15T22:28:31.603 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.604 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:31.604 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.155998+0000", 2021-03-15T22:28:31.604 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000108766 2021-03-15T22:28:31.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.605 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.605 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.156106+0000", 2021-03-15T22:28:31.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050357570000000001 2021-03-15T22:28:31.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.606 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161142+0000", 2021-03-15T22:28:31.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1378000000000007e-05 2021-03-15T22:28:31.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.607 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.607 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:31.607 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161214+0000", 2021-03-15T22:28:31.607 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6901000000000001e-05 2021-03-15T22:28:31.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.608 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.608 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161290+0000", 2021-03-15T22:28:31.608 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035732501 2021-03-15T22:28:31.609 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.609 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.609 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.609 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.610 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.610 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4397 7.6 7:61e0cc05:::smithi07694878-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e457)", 2021-03-15T22:28:31.610 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:12.125625+0000", 2021-03-15T22:28:31.610 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 139.219199086, 2021-03-15T22:28:31.610 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035880710000000003, 2021-03-15T22:28:31.611 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.611 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:31.611 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.611 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.611 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.612 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4397 2021-03-15T22:28:31.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.612 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.612 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.612 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.613 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125625+0000", 2021-03-15T22:28:31.613 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.613 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.613 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.614 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.614 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125625+0000", 2021-03-15T22:28:31.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9230000000000001e-06 2021-03-15T22:28:31.614 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.615 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.615 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125627+0000", 2021-03-15T22:28:31.615 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.198e-06 2021-03-15T22:28:31.615 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.616 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.616 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.616 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125629+0000", 2021-03-15T22:28:31.616 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1629999999999996e-06 2021-03-15T22:28:31.616 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.617 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.617 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125637+0000", 2021-03-15T22:28:31.617 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5560000000000001e-06 2021-03-15T22:28:31.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.618 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.618 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.618 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125642+0000", 2021-03-15T22:28:31.619 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053224839999999997 2021-03-15T22:28:31.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.619 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.619 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.619 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.130964+0000", 2021-03-15T22:28:31.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3690999999999996e-05 2021-03-15T22:28:31.620 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.620 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:31.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.131038+0000", 2021-03-15T22:28:31.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017722974999999998 2021-03-15T22:28:31.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.622 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.148761+0000", 2021-03-15T22:28:31.622 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5197000000000004e-05 2021-03-15T22:28:31.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.623 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:31.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.148856+0000", 2021-03-15T22:28:31.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012464467999999999 2021-03-15T22:28:31.623 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.624 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161321+0000", 2021-03-15T22:28:31.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3552000000000003e-05 2021-03-15T22:28:31.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.625 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161394+0000", 2021-03-15T22:28:31.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011150300000000001 2021-03-15T22:28:31.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.626 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.626 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.626 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161506+0000", 2021-03-15T22:28:31.626 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035880710000000003 2021-03-15T22:28:31.627 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.627 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.627 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.627 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.628 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.628 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4562 7.6 7:6a64397b:::smithi07694878-49:head [copy-from ver 960] snapc 170=[170,16e,16d,168,166,162,161,15f,15b,141,13f,13e,131,12f,123,10c,102] ondisk+write+known_if_redirected e474)", 2021-03-15T22:28:31.628 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:29.288710+0000", 2021-03-15T22:28:31.628 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 122.056114247, 2021-03-15T22:28:31.628 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030539015999999999, 2021-03-15T22:28:31.629 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.629 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:31.629 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.629 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.629 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.630 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4562 2021-03-15T22:28:31.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.630 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.631 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.631 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288710+0000", 2021-03-15T22:28:31.631 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288710+0000", 2021-03-15T22:28:31.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2800000000000002e-06 2021-03-15T22:28:31.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288712+0000", 2021-03-15T22:28:31.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.742e-06 2021-03-15T22:28:31.633 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.634 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288714+0000", 2021-03-15T22:28:31.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3130000000000001e-06 2021-03-15T22:28:31.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288721+0000", 2021-03-15T22:28:31.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6489999999999999e-06 2021-03-15T22:28:31.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.636 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288726+0000", 2021-03-15T22:28:31.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00084560199999999999 2021-03-15T22:28:31.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.637 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.637 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.289571+0000", 2021-03-15T22:28:31.638 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104343 2021-03-15T22:28:31.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.638 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.638 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.289676+0000", 2021-03-15T22:28:31.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014401884 2021-03-15T22:28:31.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.639 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.639 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:28:31.640 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.304078+0000", 2021-03-15T22:28:31.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010704943 2021-03-15T22:28:31.640 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.641 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.641 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.314783+0000", 2021-03-15T22:28:31.641 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0736e-05 2021-03-15T22:28:31.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.642 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:31.642 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.314793+0000", 2021-03-15T22:28:31.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042097990000000002 2021-03-15T22:28:31.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.643 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.643 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.643 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319003+0000", 2021-03-15T22:28:31.643 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016160299999999999 2021-03-15T22:28:31.643 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.644 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.644 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:31.644 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319165+0000", 2021-03-15T22:28:31.644 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4122000000000005e-05 2021-03-15T22:28:31.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.645 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319249+0000", 2021-03-15T22:28:31.645 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030539015999999999 2021-03-15T22:28:31.646 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.646 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.646 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.647 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4563 7.6 7:6a64397b:::smithi07694878-49:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e474)", 2021-03-15T22:28:31.647 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:29.288736+0000", 2021-03-15T22:28:31.647 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 122.056088002, 2021-03-15T22:28:31.648 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030703508000000001, 2021-03-15T22:28:31.648 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.648 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:31.648 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.648 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.649 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.649 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4563 2021-03-15T22:28:31.649 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.649 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.650 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.650 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288736+0000", 2021-03-15T22:28:31.650 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.651 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.651 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.651 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.651 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288736+0000", 2021-03-15T22:28:31.651 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1170000000000001e-06 2021-03-15T22:28:31.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.652 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288737+0000", 2021-03-15T22:28:31.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.66e-06 2021-03-15T22:28:31.653 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.653 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.653 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.653 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288739+0000", 2021-03-15T22:28:31.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.02e-06 2021-03-15T22:28:31.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.654 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.655 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288748+0000", 2021-03-15T22:28:31.655 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8380000000000001e-06 2021-03-15T22:28:31.655 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.655 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.656 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.288753+0000", 2021-03-15T22:28:31.656 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012277779999999999 2021-03-15T22:28:31.656 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.657 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.657 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.289981+0000", 2021-03-15T22:28:31.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4601000000000002e-05 2021-03-15T22:28:31.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.658 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:31.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.290025+0000", 2021-03-15T22:28:31.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019574004999999998 2021-03-15T22:28:31.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.659 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.309599+0000", 2021-03-15T22:28:31.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8437000000000006e-05 2021-03-15T22:28:31.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.660 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:31.660 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.309698+0000", 2021-03-15T22:28:31.660 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0095754180000000005 2021-03-15T22:28:31.661 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.661 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.661 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.661 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319273+0000", 2021-03-15T22:28:31.662 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3226999999999994e-05 2021-03-15T22:28:31.662 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.662 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.662 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.663 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319356+0000", 2021-03-15T22:28:31.663 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3406999999999996e-05 2021-03-15T22:28:31.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.663 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.663 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.664 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:29.319440+0000", 2021-03-15T22:28:31.664 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.030703508000000001 2021-03-15T22:28:31.664 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.664 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.665 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.665 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5175 7.6 7:6283974b:::smithi07694878-30:head [copy-from ver 1316] snapc 1a9=[1a9,1a7,1a6,1a4,1a3,1a2,1a1,1a0,19f,19c,19b,195,194,193,190,18a,184,181,180,17a,176,16e,161,15f,141,131,123,10c] ondisk+write+known_if_redirected e531)", 2021-03-15T22:28:31.665 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:27:26.396817+0000", 2021-03-15T22:28:31.666 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 64.948007269000001, 2021-03-15T22:28:31.666 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027266911000000001, 2021-03-15T22:28:31.666 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.666 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:31.667 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.667 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.667 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.667 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5175 2021-03-15T22:28:31.668 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.668 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.668 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.668 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.668 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396817+0000", 2021-03-15T22:28:31.669 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.669 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.669 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396817+0000", 2021-03-15T22:28:31.670 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2899999999999998e-06 2021-03-15T22:28:31.670 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.670 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.670 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.670 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396820+0000", 2021-03-15T22:28:31.671 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5000000000000002e-06 2021-03-15T22:28:31.671 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.671 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.671 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.672 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396823+0000", 2021-03-15T22:28:31.672 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8169999999999994e-06 2021-03-15T22:28:31.672 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.672 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396832+0000", 2021-03-15T22:28:31.673 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1040000000000003e-06 2021-03-15T22:28:31.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.673 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.673 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.674 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396839+0000", 2021-03-15T22:28:31.674 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0036395339999999998 2021-03-15T22:28:31.674 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.674 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.675 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.675 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.400478+0000", 2021-03-15T22:28:31.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1018000000000002e-05 2021-03-15T22:28:31.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.675 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.676 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.676 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.400569+0000", 2021-03-15T22:28:31.676 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.007817404 2021-03-15T22:28:31.676 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.677 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.677 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:28:31.677 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.408387+0000", 2021-03-15T22:28:31.677 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014672318 2021-03-15T22:28:31.677 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.678 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.678 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.423059+0000", 2021-03-15T22:28:31.678 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00087863399999999999 2021-03-15T22:28:31.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.679 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:31.679 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.423937+0000", 2021-03-15T22:28:31.679 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3392000000000001e-05 2021-03-15T22:28:31.680 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.680 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.680 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:28:31.680 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.423971+0000", 2021-03-15T22:28:31.680 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3430999999999998e-05 2021-03-15T22:28:31.681 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.681 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:28:31.681 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.424014+0000", 2021-03-15T22:28:31.682 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9468999999999995e-05 2021-03-15T22:28:31.682 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.682 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.682 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.682 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.424084+0000", 2021-03-15T22:28:31.683 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027266911000000001 2021-03-15T22:28:31.683 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.683 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.683 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.684 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.684 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5176 7.6 7:6283974b:::smithi07694878-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e531)", 2021-03-15T22:28:31.684 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:27:26.396845+0000", 2021-03-15T22:28:31.684 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 64.947979183000001, 2021-03-15T22:28:31.685 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027609142, 2021-03-15T22:28:31.685 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.685 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:31.685 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.685 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:28:31.686 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:31.686 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5176 2021-03-15T22:28:31.686 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.686 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.687 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.687 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.687 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396845+0000", 2021-03-15T22:28:31.687 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.688 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.688 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.688 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396845+0000", 2021-03-15T22:28:31.688 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.232e-06 2021-03-15T22:28:31.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.689 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.689 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.689 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396846+0000", 2021-03-15T22:28:31.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.353e-06 2021-03-15T22:28:31.690 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.690 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396848+0000", 2021-03-15T22:28:31.691 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6959999999999999e-06 2021-03-15T22:28:31.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.691 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396854+0000", 2021-03-15T22:28:31.692 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2690000000000001e-06 2021-03-15T22:28:31.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.692 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.692 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.693 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396859+0000", 2021-03-15T22:28:31.693 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003890082 2021-03-15T22:28:31.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.693 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.694 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.400749+0000", 2021-03-15T22:28:31.694 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2172000000000001e-05 2021-03-15T22:28:31.694 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.695 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:31.695 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.400801+0000", 2021-03-15T22:28:31.695 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015282324999999999 2021-03-15T22:28:31.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.695 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.696 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.696 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.416083+0000", 2021-03-15T22:28:31.696 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010308 2021-03-15T22:28:31.696 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.697 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.697 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:31.697 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.416186+0000", 2021-03-15T22:28:31.697 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0079379370000000008 2021-03-15T22:28:31.697 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.698 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.698 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.424124+0000", 2021-03-15T22:28:31.698 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6258e-05 2021-03-15T22:28:31.698 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.699 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.699 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.699 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.424210+0000", 2021-03-15T22:28:31.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024373799999999999 2021-03-15T22:28:31.700 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.424454+0000", 2021-03-15T22:28:31.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027609142 2021-03-15T22:28:31.701 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.701 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.701 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.701 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.702 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.702 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34361.0:611 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e551)", 2021-03-15T22:28:31.702 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:28:12.419483+0000", 2021-03-15T22:28:31.702 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 18.925341046, 2021-03-15T22:28:31.702 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.091628773999999996, 2021-03-15T22:28:31.703 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:28:31.703 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:28:31.703 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:28:31.703 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34361", 2021-03-15T22:28:31.703 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.125:0/1442040869", 2021-03-15T22:28:31.704 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 611 2021-03-15T22:28:31.704 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.704 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:28:31.704 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.705 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:28:31.705 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419483+0000", 2021-03-15T22:28:31.705 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:28:31.705 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.706 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:28:31.706 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419483+0000", 2021-03-15T22:28:31.706 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2217000000000001e-05 2021-03-15T22:28:31.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.707 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.707 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:28:31.707 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419505+0000", 2021-03-15T22:28:31.707 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9129999999999996e-06 2021-03-15T22:28:31.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.708 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.708 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:28:31.708 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419509+0000", 2021-03-15T22:28:31.708 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5500999999999999e-05 2021-03-15T22:28:31.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.709 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.709 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:28:31.709 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419525+0000", 2021-03-15T22:28:31.709 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3603000000000001e-05 2021-03-15T22:28:31.710 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.710 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.710 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:28:31.710 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419548+0000", 2021-03-15T22:28:31.711 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8461999999999996e-05 2021-03-15T22:28:31.711 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.711 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.711 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.711 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419617+0000", 2021-03-15T22:28:31.712 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.277e-05 2021-03-15T22:28:31.712 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.712 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.712 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:28:31.713 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419630+0000", 2021-03-15T22:28:31.713 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.091151858000000002 2021-03-15T22:28:31.713 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.713 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.713 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:28:31.714 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.510781+0000", 2021-03-15T22:28:31.714 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014667799999999999 2021-03-15T22:28:31.714 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.714 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.715 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:28:31.715 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.510928+0000", 2021-03-15T22:28:31.715 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018377199999999999 2021-03-15T22:28:31.715 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:28:31.715 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:28:31.716 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:28:31.716 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.511112+0000", 2021-03-15T22:28:31.716 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.091628773999999996 2021-03-15T22:28:31.716 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.717 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.717 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.717 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:28:31.717 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:28:31.717 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:28:31.718 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:31.740 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:31.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:31.742 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:31.843 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:31.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:31.868 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:31.869 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:31.970 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:31.994 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:31.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:31.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:31.999 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:28:32.097 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:32.119 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:32.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:32.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:32.222 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:32.245 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:32.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:32.246 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:32.277 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 3, 0] out_osds: [] dead_osds: [2, 5] live_osds: [0, 1, 3, 4, 6, 7] 2021-03-15T22:28:32.278 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:28:32.278 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [1, 2, 4, 5, 6, 7, 3, 0] 2021-03-15T22:28:32.279 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd out 3 2021-03-15T22:28:32.347 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:32.369 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:32.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:32.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:32.395 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:32.472 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:32.494 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:32.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:32.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:32.597 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:32.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:32.620 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:32.621 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:32.621 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:32 smithi076 conmon[56514]: cluster 2021-03-15T22:28:31.107937+0000 mgr.y (mgr.14144) 758 : cluster [DBG] pgmap v990: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%); 13 MiB/s, 9 objects/s recovering 2021-03-15T22:28:32.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:32 smithi076 conmon[62156]: cluster 2021-03-15T22:28:31.107937+0000 mgr.y (mgr.14144) 758 : cluster [DBG] pgmap v990: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%); 13 MiB/s, 9 objects/s recovering 2021-03-15T22:28:32.722 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:32.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:32.748 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:32.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:32.773 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:32.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:32 smithi125 conmon[56739]: cluster 2021-03-15T22:28:31.107937+0000 mgr.y (mgr.14144) 758 : cluster [DBG] pgmap v990: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%); 13 MiB/s, 9 objects/s recovering 2021-03-15T22:28:32.801 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:28:32.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:32.872 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:32.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:32.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:32.975 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:32.997 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:32.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:32.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:33.099 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:33.122 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:33.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:33.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:33.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:33.247 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:33.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:33.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:33.324 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:28:33.350 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:33.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:33.379 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:33.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:33.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:33 smithi076 conmon[56514]: audit 2021-03-15T22:28:33.031865+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? 172.21.15.76:0/915264265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:28:33.448 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:33 smithi076 conmon[62156]: audit 2021-03-15T22:28:33.448 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:33 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:28:33.449 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:33 smithi076 conmon[62156]: :28:33.031865+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? 172.21.15.76:0/915264265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:28:33.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:33.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:33.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:33.512 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:33.609 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:33.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:33.631 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:33.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:33.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:33.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:33.758 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:33.758 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:33.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:33 smithi125 conmon[56739]: audit 2021-03-15T22:28:33.031865+0000 mon.a (mon.0) 1587 : audit [INF] from='client.? 172.21.15.76:0/915264265' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:28:33.860 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:33.885 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:33.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:33.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:33.988 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:34.011 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:34.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:34.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:34.114 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:34.137 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:34.138 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:34.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:34.265 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:34.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:34.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:34.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:34.390 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:34.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:34.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:34.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:34 smithi076 conmon[56514]: cluster 2021-03-15T22:28:33.108913 2021-03-15T22:28:34.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:34 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 759 : cluster [DBG] pgmap v991: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:34.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:34 smithi076 conmon[56514]: cluster 2021-03-15T22:28:33.307547+0000 mon.a (mon.0) 1588 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:34.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:34 smithi076 conmon[56514]: audit 2021-03-15T22:28:33.317399+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? 172.21.15.76:0/915264265' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:28:34.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:34 smithi076 conmon[56514]: cluster 2021-03-15T22:28:33.317453+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e566: 8 total, 6 up, 8 in 2021-03-15T22:28:34.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:34 smithi076 conmon[62156]: cluster 2021-03-15T22:28:33.108913+0000 2021-03-15T22:28:34.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:34 smithi076 conmon[62156]: mgr.y (mgr.14144) 759 : cluster [DBG] pgmap v991: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:34.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:34 smithi076 conmon[62156]: cluster 2021-03-15T22:28:33.307547+0000 mon.a (mon.0) 1588 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:34.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:34 smithi076 conmon[62156]: audit 2021-03-15T22:28:33.317399+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? 172.21.15.76:0/915264265' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:28:34.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:34 smithi076 conmon[62156]: cluster 2021-03-15T22:28:33.317453+0000 mon.a (mon.0) 1590 : cluster [DBG] osdmap e566: 8 total, 6 up, 8 in 2021-03-15T22:28:34.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:34.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:34.517 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:34.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:34.619 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:34.641 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:34.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:34.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:34.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:34.765 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:34.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:34.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:34.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:34 smithi125 conmon[56739]: cluster 2021-03-15T22:28:33.108913+0000 mgr.y (mgr.14144) 759 : 2021-03-15T22:28:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:34 smithi125 conmon[56739]: cluster [DBG] pgmap v991: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 853 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:34 smithi125 conmon[56739]: cluster 2021-03-15T22:28:33.307547+0000 mon.a (mon.0) 1588 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:34 smithi125 conmon[56739]: audit 2021 2021-03-15T22:28:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:34 smithi125 conmon[56739]: -03-15T22:28:33.317399+0000 mon.a (mon.0) 1589 : audit [INF] from='client.? 172.21.15.76:0/915264265' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:28:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:34 smithi125 conmon[56739]: cluster 2021-03-15T22:28:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:34 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:28:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:34 smithi125 conmon[56739]: 28:33.317453+0000 mon.a (mon 2021-03-15T22:28:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:34 smithi125 conmon[56739]: .0) 1590 : cluster [DBG] osdmap e566: 8 total, 6 up, 8 in 2021-03-15T22:28:34.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:34 smithi125 conmon[56739]: 2021-03-15T22:28:34.869 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:34.890 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:34.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:34.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:34.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:35.016 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:35.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:35.119 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:35.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:35.143 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:35.143 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:35.245 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:35.266 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:35.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:35.369 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:35.394 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:35.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:35.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:35.497 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:35.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:35.519 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:35.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:35.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:35 smithi076 conmon[56514]: cluster 2021-03-15T22:28:33.647668+0000 osd.1 2021-03-15T22:28:35.578 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:35 smithi076 conmon[56514]: (osd.1) 108 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:28:35.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:35 smithi076 conmon[56514]: cluster 2021-03-15T22:28:34.099133+0000 osd.3 (osd.3) 99 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:28:35.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:35 smithi076 conmon[56514]: cluster 2021-03-15T22:28:35.168456+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e567: 8 total, 6 up, 8 in 2021-03-15T22:28:35.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:35 smithi076 conmon[62156]: cluster 2021-03-15T22:28:33.647668+0000 2021-03-15T22:28:35.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:35 smithi076 conmon[62156]: osd.1 (osd.1) 108 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:28:35.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:35 smithi076 conmon[62156]: cluster 2021-03-15T22:28:34.099133+0000 osd.3 (osd.3) 99 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:28:35.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:35 smithi076 conmon[62156]: cluster 2021-03-15T22:28:35.168456+0000 mon.a (mon.0) 1591 : cluster [DBG] osdmap e567: 8 total, 6 up, 8 in 2021-03-15T22:28:35.621 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:35.643 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:35.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:35.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:35.746 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:35.768 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:35.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:35.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:35.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:35 smithi125 conmon[56739]: cluster 2021-03-15T22:28:33.647668+0000 osd.1 (osd.1) 108 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:28:35.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:35 smithi125 conmon[56739]: cluster 2021-03-15T22:28:34.099133+0000 osd.3 (osd.3) 99 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:28:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:35 smithi125 conmon[56739]: 2021-03-15T22:28:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:35 smithi125 conmon[56739]: cluster 2021-03-15T22:28:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:35 smithi125 conmon[56739]: 2021-03-15T22:28:35.168456+0000 mon.a 2021-03-15T22:28:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:35 smithi125 conmon[56739]: (mon.0) 1591 : cluster 2021-03-15T22:28:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:35 smithi125 conmon[56739]: [DBG] osdmap e567: 8 total, 6 up, 8 in 2021-03-15T22:28:35.871 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:35.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:35.894 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:35.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:35.983 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:28:35.984 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:28:35.984 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:28:35.984 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:28:35.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:36.017 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:36.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:36.018 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:36.119 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:36.141 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:36.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:36.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:36.243 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:36.265 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:36.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:36.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:36.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:36.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:36.392 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:36.392 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:36.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:36.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:36.520 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:36.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:36.622 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:36.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:36.646 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:36.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:36.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:36.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:36 smithi076 conmon[56514]: cluster 2021-03-15T22:28:35.109757+0000 mgr.y (mgr.14144) 760 : cluster [DBG] pgmap v993: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:36.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:36 smithi076 conmon[62156]: cluster 2021-03-15T22:28:35.109757+0000 mgr.y (mgr. 2021-03-15T22:28:36.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:36 smithi076 conmon[62156]: 14144) 760 : cluster [DBG] pgmap v993: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:36.770 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:36.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:36.772 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:36 smithi125 conmon[56739]: cluster 2021-03-15T22:28:35.109757+0000 mgr.y (mgr.14144 2021-03-15T22:28:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:36 smithi125 conmon[56739]: ) 760 : cluster [DBG] pgmap v993: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.4 KiB/s rd, 1 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:36.874 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:36.896 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:36.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:36.898 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:36.999 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:37.023 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:37.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:37.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:37.037 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:28:37.126 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:37.155 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:37.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:37.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:37.258 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:37.283 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:37.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:37.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:37.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:37.387 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:28:37.392 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:37.409 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:28:37.414 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:37.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:37.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:37.429 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:28:37.448 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:28:37.467 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:28:37.485 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:28:37.485 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:28:37.486 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:28:37.486 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:28:37.486 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:28:37.486 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:28:37.487 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:28:37.487 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:28:37.487 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:28:37.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[56514]: cluster 2021-03-15T22:28:35.742151+0000 osd.6 (osd.6) 2021-03-15T22:28:37.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[56514]: 88 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:37.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[56514]: cluster 2021-03-15T22:28:35.742707+0000 osd.6 (osd.6) 89 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:37.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[56514]: cluster 2021-03-15T22:28:36.101996+0000 osd.3 (osd.3) 100 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:28:37.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[56514]: cluster 2021-03-15T22:28:36.326679+0000 mon.a (mon.0) 1592 : cluster [DBG] osdmap e568: 8 total, 6 up, 8 in 2021-03-15T22:28:37.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: cluster 2021-03-15T22:28:37.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:28:37.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: :28:35.742151+0000 osd.6 (osd.6) 88 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:37.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: cluster 2021-03-15T22:28:37.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: 2021-03-15T22:28:35. 2021-03-15T22:28:37.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: 742707+0000 osd.6 (osd.6 2021-03-15T22:28:37.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: ) 89 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:37.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:28:37.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: -03-15T22:28:36.101996+0000 2021-03-15T22:28:37.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: osd.3 (osd.3) 100 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:28:37.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: 2021-03-15T22:28:37.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: cluster 2021-03-15T22:28: 2021-03-15T22:28:37.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: 36.326679+0000 mon.a (mon.0) 1592 2021-03-15T22:28:37.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[62156]: : cluster [DBG] osdmap e568: 8 total, 6 up, 8 in 2021-03-15T22:28:37.517 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:37.539 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:37.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:37.642 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:37.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:37.666 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:37.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:37.765 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:28:37.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:37 smithi076 conmon[56514]: debug 2021-03-15T22:28:37.484+0000 7f0537c88700 -1 mon.a@0(leader).osd e568 definitely_dead 0 2021-03-15T22:28:37.767 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:37 smithi125 conmon[56739]: cluster 2021-03-15T22:28:35.742151+0000 osd.6 (osd.6) 88 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:37 smithi125 conmon[56739]: cluster 2021-03-15T22:28:35.742707+0000 osd.6 (osd.6) 89 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:37 smithi125 conmon[56739]: cluster 2021-03-15T22:28:36.101996+0000 2021-03-15T22:28:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:37 smithi125 conmon[56739]: osd.3 (osd.3) 100 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:28:37.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:37 smithi125 conmon[56739]: cluster 2021-03-15T22:28:37.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:37 smithi125 conmon[56739]: 2021-03-15T22:28:36.326679+0000 mon.a 2021-03-15T22:28:37.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:37 smithi125 conmon[56739]: (mon.0) 1592 : cluster [DBG] osdmap e568: 8 total, 6 up, 8 in 2021-03-15T22:28:37.789 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:37.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:37.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:37.891 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:37.916 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:37.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:37.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:38.020 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:38.045 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:38.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:38.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:38.148 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:38.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:38.173 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:38.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:38.275 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:38.300 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:38.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:38.302 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:38.355 INFO:teuthology.orchestra.run.smithi076.stderr:marked out osd.3. 2021-03-15T22:28:38.403 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:38.446 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:28:38.447 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:28:38.454 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:38.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:38.455 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:38.556 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:38.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:38.579 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:38.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:38.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:38.704 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:38.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:38.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:38.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[56514]: cluster 2021-03-15T22:28:37.110485+0000 mgr.y (mgr.14144) 761 : cluster [DBG] pgmap v996: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:38.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[56514]: cluster 2021-03-15T22:28:37.136497+0000 osd.3 (osd.3) 101 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:28:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[56514]: cluster 2021-03-15T22:28:37.329171+0000 mon.a (mon.0) 1593 : cluster [WRN] Health check update: Reduced data availability: 8 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[56514]: cluster 2021-03-15T22:28:37.329218+0000 mon.a (mon.0) 1594 : cluster [WRN] Health check update: Degraded data redundancy: 307/1740 objects degraded (17.644%), 24 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[56514]: audit 2021-03-15T22:28:37.352371+0000 mon.a (mon.0) 1595 : audit [DBG] from='client.? 172.21.15.76:0/549672518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[56514]: audit 2021-03-15T22:28:37.485117+0000 mon.c (mon.1) 51 : audit [INF] from='client.? 172.21.15.76:0/2694862666' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2021-03-15T22:28:38.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[56514]: audit 2021-03-15T22:28:37.485465+0000 mon.a (mon.0) 1596 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2021-03-15T22:28:38.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[56514]: cluster 2021-03-15T22:28:37.485566+0000 mon.a (mon.0) 1597 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2021-03-15T22:28:38.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[62156]: cluster 2021-03-15T22:28:37.110485+0000 mgr.y (mgr.14144) 761 : cluster [DBG] pgmap v996: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:38.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[62156]: cluster 2021-03-15T22:28:37.136497+0000 osd.3 (osd.3) 101 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:28:38.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[62156]: cluster 2021-03-15T22:28:37.329171+0000 mon.a (mon.0) 1593 : cluster [WRN] Health check update: Reduced data availability: 8 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:38.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[62156]: cluster 2021-03-15T22:28:37.329218+0000 mon.a (mon.0) 1594 : cluster [WRN] Health check update: Degraded data redundancy: 307/1740 objects degraded (17.644%), 24 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:38.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[62156]: audit 2021-03-15T22:28:37.352371+0000 mon.a (mon.0) 1595 : audit [DBG] from='client.? 172.21.15.76:0/549672518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:38.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[62156]: audit 2021-03-15T22:28:37.485117+0000 mon.c (mon.1) 51 : audit [INF] from='client.? 172.21.15.76:0/2694862666' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2021-03-15T22:28:38.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[62156]: audit 2021-03-15T22:28:37.485465+0000 mon.a (mon.0) 1596 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2021-03-15T22:28:38.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:38 smithi076 conmon[62156]: cluster 2021-03-15T22:28:37.485566+0000 mon.a (mon.0) 1597 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2021-03-15T22:28:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: cluster 2021-03-15T22:28:37.110485+0000 mgr.y (mgr.14144) 761 : cluster [DBG] pgmap v996: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 852 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: 2021-03-15T22:28:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: cluster 2021-03-15T22:28:37.136497+0000 osd.3 (osd.3) 101 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:28:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: cluster 2021-03-15T22:28: 2021-03-15T22:28:38.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: 37.329171+0000 mon.a (mon.0) 1593 : cluster [WRN] Health check update: Reduced data availability: 8 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:38.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:28:38.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: 15T22:28:37.329218+0000 mon.a (mon.0) 1594 : cluster [WRN] Health check update: Degraded data redundancy: 307/1740 objects degraded (17.644%), 24 pgs degraded, 59 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:38.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: audit 2021-03-15T22:28:37.352371 2021-03-15T22:28:38.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1595 : audit [DBG] from='client.? 172.21.15.76:0/549672518' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:38.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: audit 2021-03-15T22:28: 2021-03-15T22:28:38.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: 37.485117+0000 mon.c (mon.1) 51 : audit [INF] from='client.? 172.21.15.76:0/2694862666' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2021-03-15T22:28:38.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: audit 2021-03-15T22:28: 2021-03-15T22:28:38.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: 37.485465+0000 mon.a (mon.0) 1596 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2021-03-15T22:28:38.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: cluster 2021-03-15T22:28 2021-03-15T22:28:38.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:38 smithi125 conmon[56739]: :37.485566+0000 mon.a (mon.0) 1597 : cluster [INF] Client client.admin marked osd.3 out, while it was still marked up 2021-03-15T22:28:38.807 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:38.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:38.830 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:38.830 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:38.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:38.954 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:38.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:38.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:39.056 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:39.078 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:39.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:39.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:39.181 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:39.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:39.204 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:39.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:39.306 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:39.329 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:39.330 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:39.431 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:39.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:39.468 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:39.468 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:39.570 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:39.591 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:39.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:39.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:39.642 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[56514]: cluster 2021-03-15T22:28:37.623886+0000 osd.1 (osd.1) 109 : cluster [DBG] 2021-03-15T22:28:39.642 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[56514]: 4.7 scrub ok 2021-03-15T22:28:39.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[56514]: cluster 2021-03-15T22:28:38.142750+0000 osd.3 (osd.3) 102 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:28:39.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[56514]: audit 2021-03-15T22:28:38.345726+0000 mon.a (mon.0) 1598 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2021-03-15T22:28:39.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[56514]: cluster 2021-03-15T22:28:38.346075+0000 mon.a (mon.0) 1599 : cluster [DBG] osdmap e569: 8 total, 6 up, 7 in 2021-03-15T22:28:39.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: cluster 2021-03-15T22:28:39.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: 2021-03-15T22:28:39.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: 2021 2021-03-15T22:28:39.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: - 2021-03-15T22:28:39.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: 03 2021-03-15T22:28:39.645 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: - 2021-03-15T22:28:39.645 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: 15 2021-03-15T22:28:39.645 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: T 2021-03-15T22:28:39.645 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: 22 2021-03-15T22:28:39.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: : 2021-03-15T22:28:39.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: 28 2021-03-15T22:28:39.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: : 2021-03-15T22:28:39.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: 37 2021-03-15T22:28:39.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: . 2021-03-15T22:28:39.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: 623886 2021-03-15T22:28:39.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: +0000 2021-03-15T22:28:39.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: 2021-03-15T22:28:39.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: osd.1 2021-03-15T22:28:39.648 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: ( 2021-03-15T22:28:39.648 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: osd.1) 109 : cluster 2021-03-15T22:28:39.648 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: [DBG] 4.7 scrub ok 2021-03-15T22:28:39.648 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: cluster 2021-03-15T22:28: 2021-03-15T22:28:39.648 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: 38.142750+0000 osd.3 (osd.3) 102 2021-03-15T22:28:39.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: : cluster [DBG] 2.1 scrub ok 2021-03-15T22:28:39.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:28:39.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: 03-15T22:28:38.345726+0000 2021-03-15T22:28:39.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: mon.a (mon.0) 1598 : audit [INF] 2021-03-15T22:28:39.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2021-03-15T22:28:39.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: cluster 2021-03-15T22:28:38 2021-03-15T22:28:39.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: .346075+0000 mon.a (mon.0) 1599 : 2021-03-15T22:28:39.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:39 smithi076 conmon[62156]: cluster [DBG] osdmap e569: 8 total, 6 up, 7 in 2021-03-15T22:28:39.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:39.717 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:39.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:39.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:39 smithi125 conmon[56739]: cluster 2021-03-15T22:28:37.623886+0000 2021-03-15T22:28:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:39 smithi125 conmon[56739]: osd.1 (osd.1) 109 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:28:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:39 smithi125 conmon[56739]: cluster 2021-03-15T22:28:38.142750+0000 osd.3 (osd.3 2021-03-15T22:28:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:39 smithi125 conmon[56739]: ) 102 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:28:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:39 smithi125 conmon[56739]: audit 2021 2021-03-15T22:28:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:39 smithi125 conmon[56739]: -03-15T22:28:38.345726+0000 mon.a (mon.0) 1598 2021-03-15T22:28:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:39 smithi125 conmon[56739]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2021-03-15T22:28:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:39 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:28:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:39 smithi125 conmon[56739]: :28:38.346075+0000 mon.a (mon.0 2021-03-15T22:28:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:39 smithi125 conmon[56739]: ) 1599 : cluster [DBG] osdmap e569: 8 total, 6 up, 7 in 2021-03-15T22:28:39.819 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:39.840 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:39.842 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:39.879 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:39.943 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:39.964 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:39.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:39.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:40.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:40.087 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:40.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:40.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:40.189 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:40.226 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:40.228 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:40.286 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:28:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:28:40.329 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:40.344 INFO:tasks.rados.rados.0.smithi076.stdout:2060: finishing write tid 1 to smithi07694878-38 2021-03-15T22:28:40.348 INFO:tasks.rados.rados.0.smithi076.stdout:2060: finishing write tid 2 to smithi07694878-38 2021-03-15T22:28:40.349 INFO:tasks.rados.rados.0.smithi076.stdout:2060: finishing write tid 3 to smithi07694878-38 2021-03-15T22:28:40.349 INFO:tasks.rados.rados.0.smithi076.stdout:2060: finishing write tid 4 to smithi07694878-38 2021-03-15T22:28:40.349 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 1029 (ObjNum 717 snap 233 seq_num 717) dirty exists 2021-03-15T22:28:40.350 INFO:tasks.rados.rados.0.smithi076.stdout:2060: left oid 38 (ObjNum 717 snap 233 seq_num 717) 2021-03-15T22:28:40.350 INFO:tasks.rados.rados.0.smithi076.stdout:2060: done (1 left) 2021-03-15T22:28:40.350 INFO:tasks.rados.rados.0.smithi076.stdout:2064: done (0 left) 2021-03-15T22:28:40.351 INFO:tasks.rados.rados.0.smithi076.stdout:2065: snap_create 2021-03-15T22:28:40.359 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:40.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:40.416 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:28:40.416 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:28:40.449 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:28:40.461 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:40.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:40.517 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:40.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:40.620 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:40.646 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:40.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:40.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:40.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:40.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[56514]: cluster 2021-03-15T22:28:39.111072+0000 mgr.y (mgr.14144) 762 : cluster [DBG] pgmap v998: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 884 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:40.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[56514]: cluster 2021-03-15T22:28:39.356236+0000 mon.a (mon.0) 1600 : cluster [DBG] osdmap e570: 8 total, 6 up, 7 in 2021-03-15T22:28:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[56514]: audit 2021-03-15T22:28:39.892343+0000 mgr.y (mgr.14144) 763 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[56514]: cluster 2021-03-15T22:28:40.169874+0000 mon.a (mon.0) 1601 : cluster [DBG] osdmap e571: 8 total, 6 up, 7 in 2021-03-15T22:28:40.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: cluster 2021-03-15T22:28:40.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 2021-03-15T22:28:40.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 2021 2021-03-15T22:28:40.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: - 2021-03-15T22:28:40.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 03 2021-03-15T22:28:40.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: - 2021-03-15T22:28:40.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 15 2021-03-15T22:28:40.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: T 2021-03-15T22:28:40.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 22 2021-03-15T22:28:40.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: : 2021-03-15T22:28:40.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 28 2021-03-15T22:28:40.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: : 2021-03-15T22:28:40.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 39 2021-03-15T22:28:40.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: . 2021-03-15T22:28:40.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 111072 2021-03-15T22:28:40.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: +0000 2021-03-15T22:28:40.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 2021-03-15T22:28:40.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: mgr.y 2021-03-15T22:28:40.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: ( 2021-03-15T22:28:40.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: mgr 2021-03-15T22:28:40.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: . 2021-03-15T22:28:40.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 14144 2021-03-15T22:28:40.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: ) 2021-03-15T22:28:40.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 762 2021-03-15T22:28:40.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: : 2021-03-15T22:28:40.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: cluster 2021-03-15T22:28:40.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 2021-03-15T22:28:40.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: [DBG] 2021-03-15T22:28:40.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 2021-03-15T22:28:40.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: pgmap v998: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 884 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:40.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:28:40.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: :28:39.356236+0000 mon.a (mon.0) 1600 : 2021-03-15T22:28:40.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: cluster [DBG] osdmap e570: 8 total, 6 up, 7 in 2021-03-15T22:28:40.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:28:40.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: -15T22:28:39.892343+0000 mgr.y ( 2021-03-15T22:28:40.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: mgr.14144) 763 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:40.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: cluster 2021-03-15T22:28:40.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: 2021-03-15T22:28:40.169874+0000 2021-03-15T22:28:40.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: mon.a (mon.0) 1601 2021-03-15T22:28:40.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:40 smithi076 conmon[62156]: : cluster [DBG] osdmap e571: 8 total, 6 up, 7 in 2021-03-15T22:28:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:40.780 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:40.780 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:40.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: cluster 2021-03-15T22:28:39.111072 2021-03-15T22:28:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 762 : cluster [DBG] pgmap v998: 138 pgs: 14 active+clean+remapped, 3 active+undersized+remapped, 2 undersized+degraded+peered, 22 active+undersized+degraded, 6 undersized+peered, 37 active+undersized, 54 active+clean; 567 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 884 B/s rd, 0 op/s; 307/1740 objects degraded (17.644%); 63/1740 objects misplaced (3.621%) 2021-03-15T22:28:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:28:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: 15T22:28:39.356236+0000 mon.a (mon.0) 1600 : cluster [DBG] osdmap e570: 8 total, 6 up, 7 in 2021-03-15T22:28:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: 2021-03-15T22:28:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: audit 2021-03-15T22:28:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: 2021-03-15T22:28:39.892343+0000 2021-03-15T22:28:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: mgr.y (mgr.14144) 763 : audit [DBG] 2021-03-15T22:28:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:40.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:28:40.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: :28:40.169874+0000 mon.a (mon 2021-03-15T22:28:40.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:40 smithi125 conmon[56739]: .0) 1601 : cluster [DBG] osdmap e571: 8 total, 6 up, 7 in 2021-03-15T22:28:40.881 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:40.904 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:40.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:40.906 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:41.007 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:41.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:41.031 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:41.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:41.132 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:41.155 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:41.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:41.156 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:41.189 INFO:tasks.rados.rados.0.smithi076.stdout:2065: done (0 left) 2021-03-15T22:28:41.190 INFO:tasks.rados.rados.0.smithi076.stdout:2066: rollback oid 46 current snap is 234 2021-03-15T22:28:41.190 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 46 to 203 2021-03-15T22:28:41.190 INFO:tasks.rados.rados.0.smithi076.stdout:2067: read oid 32 snap 203 2021-03-15T22:28:41.191 INFO:tasks.rados.rados.0.smithi076.stdout:2067: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:28:41.191 INFO:tasks.rados.rados.0.smithi076.stdout:2068: rollback oid 5 current snap is 234 2021-03-15T22:28:41.191 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 212 2021-03-15T22:28:41.191 INFO:tasks.rados.rados.0.smithi076.stdout:2069: rollback oid 23 current snap is 234 2021-03-15T22:28:41.191 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 23 to 226 2021-03-15T22:28:41.192 INFO:tasks.rados.rados.0.smithi076.stdout:2070: snap_create 2021-03-15T22:28:41.195 INFO:tasks.rados.rados.0.smithi076.stdout:2066: finishing rollback tid 1 to smithi07694878-46 2021-03-15T22:28:41.195 INFO:tasks.rados.rados.0.smithi076.stdout:2066: finishing rollback tid 2 to smithi07694878-46 2021-03-15T22:28:41.195 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 842 (ObjNum 496 snap 149 seq_num 496) dirty exists 2021-03-15T22:28:41.199 INFO:tasks.rados.rados.0.smithi076.stdout:2067: expect (ObjNum 430 snap 119 seq_num 430) 2021-03-15T22:28:41.257 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:41.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:41.284 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:41.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:41.307 INFO:tasks.rados.rados.0.smithi076.stdout:2069: finishing rollback tid 1 to smithi07694878-23 2021-03-15T22:28:41.308 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 805 (ObjNum 264935024 snap 21896 seq_num 261733632) dirty dne 2021-03-15T22:28:41.308 INFO:tasks.rados.rados.0.smithi076.stdout:2068: finishing rollback tid 0 to smithi07694878-5 2021-03-15T22:28:41.308 INFO:tasks.rados.rados.0.smithi076.stdout:2068: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:28:41.309 INFO:tasks.rados.rados.0.smithi076.stdout:2068: finishing rollback tid 2 to smithi07694878-5 2021-03-15T22:28:41.309 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 611 (ObjNum 668 snap 212 seq_num 668) dirty exists 2021-03-15T22:28:41.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:41.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:41.408 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:41.409 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:41.510 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:41.532 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:41.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:41.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:41.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:41.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:41.657 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:41.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:41.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:28:41.759 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:41.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:41.783 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:41.783 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:41.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:41.886 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:41.914 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:41.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:41.915 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:42.016 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:42.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:42.041 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:42.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:42.054 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:42.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:42.165 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:42.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:42.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:42.189 INFO:tasks.rados.rados.0.smithi076.stdout:2066: done (4 left) 2021-03-15T22:28:42.190 INFO:tasks.rados.rados.0.smithi076.stdout:2067: done (3 left) 2021-03-15T22:28:42.190 INFO:tasks.rados.rados.0.smithi076.stdout:2068: done (2 left) 2021-03-15T22:28:42.191 INFO:tasks.rados.rados.0.smithi076.stdout:2069: done (1 left) 2021-03-15T22:28:42.191 INFO:tasks.rados.rados.0.smithi076.stdout:2070: done (0 left) 2021-03-15T22:28:42.192 INFO:tasks.rados.rados.0.smithi076.stdout:2071: read oid 48 snap -1 2021-03-15T22:28:42.192 INFO:tasks.rados.rados.0.smithi076.stdout:2071: expect (ObjNum 641 snap 200 seq_num 641) 2021-03-15T22:28:42.193 INFO:tasks.rados.rados.0.smithi076.stdout:2072: write (excl) oid 5 current snap is 235 2021-03-15T22:28:42.194 INFO:tasks.rados.rados.0.smithi076.stdout:2072: seq_num 721 ranges {517847=791711,2108223=9624} 2021-03-15T22:28:42.197 INFO:tasks.rados.rados.0.smithi076.stdout:2072: writing smithi07694878-5 from 517847 to 1309558 tid 1 2021-03-15T22:28:42.202 INFO:tasks.rados.rados.0.smithi076.stdout:2072: writing smithi07694878-5 from 2108223 to 2117847 tid 2 2021-03-15T22:28:42.203 INFO:tasks.rados.rados.0.smithi076.stdout:2071: expect (ObjNum 641 snap 200 seq_num 641) 2021-03-15T22:28:42.268 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:42.289 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:42.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:42.291 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:42.298 INFO:tasks.rados.rados.0.smithi076.stdout:2071: done (1 left) 2021-03-15T22:28:42.299 INFO:tasks.rados.rados.0.smithi076.stdout:2073: delete oid 45 current snap is 235 2021-03-15T22:28:42.299 INFO:tasks.rados.rados.0.smithi076.stdout:2072: finishing write tid 1 to smithi07694878-5 2021-03-15T22:28:42.299 INFO:tasks.rados.rados.0.smithi076.stdout:2072: finishing write tid 2 to smithi07694878-5 2021-03-15T22:28:42.299 INFO:tasks.rados.rados.0.smithi076.stdout:2072: finishing write tid 3 to smithi07694878-5 2021-03-15T22:28:42.300 INFO:tasks.rados.rados.0.smithi076.stdout:2072: finishing write tid 4 to smithi07694878-5 2021-03-15T22:28:42.300 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 614 (ObjNum 721 snap 235 seq_num 721) dirty exists 2021-03-15T22:28:42.301 INFO:tasks.rados.rados.0.smithi076.stdout:2072: left oid 5 (ObjNum 721 snap 235 seq_num 721) 2021-03-15T22:28:42.302 INFO:tasks.rados.rados.0.smithi076.stdout:2072: done (1 left) 2021-03-15T22:28:42.302 INFO:tasks.rados.rados.0.smithi076.stdout:2073: done (0 left) 2021-03-15T22:28:42.302 INFO:tasks.rados.rados.0.smithi076.stdout:2074: read oid 28 snap 191 2021-03-15T22:28:42.303 INFO:tasks.rados.rados.0.smithi076.stdout:2074: expect (ObjNum 549 snap 175 seq_num 549) 2021-03-15T22:28:42.303 INFO:tasks.rados.rados.0.smithi076.stdout:2075: copy_from oid 30 from oid 31 current snap is 235 2021-03-15T22:28:42.303 INFO:tasks.rados.rados.0.smithi076.stdout:2076: write (excl) oid 4 current snap is 235 2021-03-15T22:28:42.303 INFO:tasks.rados.rados.0.smithi076.stdout:2076: seq_num 723 ranges {551461=658562,1941540=209921} 2021-03-15T22:28:42.307 INFO:tasks.rados.rados.0.smithi076.stdout:2076: writing smithi07694878-4 from 551461 to 1210023 tid 1 2021-03-15T22:28:42.313 INFO:tasks.rados.rados.0.smithi076.stdout:2076: writing smithi07694878-4 from 1941540 to 2151461 tid 2 2021-03-15T22:28:42.313 INFO:tasks.rados.rados.0.smithi076.stdout:2077: snap_remove snap 227 2021-03-15T22:28:42.391 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[56514]: cluster 2021-03-15T22:28:41. 2021-03-15T22:28:42.391 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[56514]: 111826+0000 mgr.y (mgr.14144) 764 : cluster [DBG] pgmap v1001: 138 pgs: 11 peering, 26 active+clean+remapped, 9 active+undersized+remapped, 2 undersized+degraded+peered, 18 active+undersized+degraded, 5 undersized+peered, 29 active+undersized, 38 active+clean; 567 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 213 B/s rd, 0 op/s; 282/1740 objects degraded (16.207%); 149/1740 objects misplaced (8.563%); 8.6 MiB/s, 21 objects/s recovering 2021-03-15T22:28:42.391 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[56514]: cluster 2021-03-15T22:28:41.186978+0000 mon.a (mon.0) 1602 : cluster [DBG] osdmap e572: 8 total, 6 up, 7 in 2021-03-15T22:28:42.392 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[56514]: cluster 2021-03-15T22:28:41.441092+0000 osd.3 (osd.3) 103 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:42.392 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[56514]: cluster 2021-03-15T22:28:41.442196+0000 osd.3 (osd.3) 104 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:42.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: cluster 2021-03-15T22:28:42.392 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 2021-03-15T22:28:42.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 2021 2021-03-15T22:28:42.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: - 2021-03-15T22:28:42.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 03 2021-03-15T22:28:42.393 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: - 2021-03-15T22:28:42.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 15 2021-03-15T22:28:42.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: T 2021-03-15T22:28:42.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 22 2021-03-15T22:28:42.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: : 2021-03-15T22:28:42.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 28 2021-03-15T22:28:42.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: : 2021-03-15T22:28:42.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 41 2021-03-15T22:28:42.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: . 2021-03-15T22:28:42.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 111826 2021-03-15T22:28:42.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: +0000 2021-03-15T22:28:42.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 2021-03-15T22:28:42.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: mgr.y 2021-03-15T22:28:42.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: ( 2021-03-15T22:28:42.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: mgr 2021-03-15T22:28:42.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: . 2021-03-15T22:28:42.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 14144 2021-03-15T22:28:42.397 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: ) 2021-03-15T22:28:42.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 764 2021-03-15T22:28:42.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: : 2021-03-15T22:28:42.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: cluster 2021-03-15T22:28:42.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 2021-03-15T22:28:42.398 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: [DBG] 2021-03-15T22:28:42.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 2021-03-15T22:28:42.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: pgmap v1001: 138 pgs: 11 peering, 26 active+clean+remapped, 9 active+undersized+remapped, 2 undersized+degraded+peered, 18 active+undersized+degraded, 5 undersized+peered, 29 active+undersized, 38 active+clean; 567 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 213 B/s rd, 0 op/s; 282/1740 objects degraded (16.207%); 149/1740 objects misplaced (8.563%); 8.6 MiB/s, 21 objects/s recovering 2021-03-15T22:28:42.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 2021-03-15T22:28:42.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:28:42.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 28:41.186978+0000 mon.a (mon.0) 1602 : cluster [DBG] osdmap e572: 8 total, 6 up, 7 in 2021-03-15T22:28:42.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: cluster 2021-03-15T22:28 2021-03-15T22:28:42.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: :41.441092+0000 osd.3 (osd.3) 2021-03-15T22:28:42.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: 103 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:42.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:28:42.401 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: -15T22:28:41.442196+0000 osd.3 2021-03-15T22:28:42.401 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:42 smithi076 conmon[62156]: (osd.3) 104 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:42.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:42.423 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:42.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:42.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:42.526 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:42.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: cluster 2021-03-15T22:28:42.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021 2021-03-15T22:28:42.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: - 2021-03-15T22:28:42.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 03 2021-03-15T22:28:42.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: - 2021-03-15T22:28:42.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 15 2021-03-15T22:28:42.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: T 2021-03-15T22:28:42.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 22 2021-03-15T22:28:42.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: : 2021-03-15T22:28:42.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 28 2021-03-15T22:28:42.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: : 2021-03-15T22:28:42.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 41 2021-03-15T22:28:42.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: . 2021-03-15T22:28:42.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 111826 2021-03-15T22:28:42.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: +0000 2021-03-15T22:28:42.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:42.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: ( 2021-03-15T22:28:42.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: mgr 2021-03-15T22:28:42.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: . 2021-03-15T22:28:42.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 14144 2021-03-15T22:28:42.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: ) 2021-03-15T22:28:42.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 764 2021-03-15T22:28:42.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: : 2021-03-15T22:28:42.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: cluster 2021-03-15T22:28:42.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:42.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: pgmap v1001: 138 pgs: 11 peering, 26 active+clean+remapped, 9 active+undersized+remapped, 2 undersized+degraded+peered, 18 active+undersized+degraded, 5 undersized+peered, 29 active+undersized, 38 active+clean; 567 MiB data, 1.1 GiB used, 625 GiB / 626 GiB avail; 213 B/s rd, 0 op/s; 282/1740 objects degraded (16.207%); 149/1740 objects misplaced (8.563%); 8.6 MiB/s, 21 objects/s recovering 2021-03-15T22:28:42.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: cluster 2021-03-15T22:28:42.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021 2021-03-15T22:28:42.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: - 2021-03-15T22:28:42.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 03 2021-03-15T22:28:42.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: - 2021-03-15T22:28:42.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 15 2021-03-15T22:28:42.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: T 2021-03-15T22:28:42.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 22 2021-03-15T22:28:42.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: : 2021-03-15T22:28:42.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 28 2021-03-15T22:28:42.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: : 2021-03-15T22:28:42.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 41 2021-03-15T22:28:42.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: . 2021-03-15T22:28:42.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 186978 2021-03-15T22:28:42.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: +0000 2021-03-15T22:28:42.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:42.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: ( 2021-03-15T22:28:42.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: mon 2021-03-15T22:28:42.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: . 2021-03-15T22:28:42.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 0 2021-03-15T22:28:42.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: ) 2021-03-15T22:28:42.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 1602 2021-03-15T22:28:42.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: : 2021-03-15T22:28:42.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: cluster 2021-03-15T22:28:42.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:42.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: osdmap e572: 8 total, 6 up, 7 in 2021-03-15T22:28:42.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: cluster 2021-03-15T22:28:42.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021 2021-03-15T22:28:42.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: - 2021-03-15T22:28:42.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 03 2021-03-15T22:28:42.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: - 2021-03-15T22:28:42.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 15 2021-03-15T22:28:42.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: T 2021-03-15T22:28:42.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 22 2021-03-15T22:28:42.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: : 2021-03-15T22:28:42.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 28 2021-03-15T22:28:42.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: : 2021-03-15T22:28:42.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 41 2021-03-15T22:28:42.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: . 2021-03-15T22:28:42.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 441092 2021-03-15T22:28:42.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: +0000 2021-03-15T22:28:42.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: osd.3 2021-03-15T22:28:42.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: ( 2021-03-15T22:28:42.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: osd 2021-03-15T22:28:42.564 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: . 2021-03-15T22:28:42.564 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 3 2021-03-15T22:28:42.564 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: ) 2021-03-15T22:28:42.565 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 103 2021-03-15T22:28:42.565 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: : 2021-03-15T22:28:42.565 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: cluster 2021-03-15T22:28:42.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:42.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: purged_snaps scrub starts 2021-03-15T22:28:42.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: cluster 2021-03-15T22:28:42.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021-03-15T22:28:42.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 2021 2021-03-15T22:28:42.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: - 2021-03-15T22:28:42.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 03 2021-03-15T22:28:42.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: - 2021-03-15T22:28:42.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 15 2021-03-15T22:28:42.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: T 2021-03-15T22:28:42.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 22 2021-03-15T22:28:42.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: : 2021-03-15T22:28:42.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 28 2021-03-15T22:28:42.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: : 2021-03-15T22:28:42.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: 41 2021-03-15T22:28:42.572 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: .442196+0000 osd.3 ( 2021-03-15T22:28:42.572 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:42 smithi125 conmon[56739]: osd.3) 104 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:42.573 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:42.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:42.675 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:42.699 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:42.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:42.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:42.802 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:42.824 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:42.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:42.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:42.926 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:42.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:42.951 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:42.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:43.053 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:43.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:43.078 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:43.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:43.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:43.196 INFO:tasks.rados.rados.0.smithi076.stdout:2077: done (3 left) 2021-03-15T22:28:43.196 INFO:tasks.rados.rados.0.smithi076.stdout:2078: rollback oid 42 current snap is 235 2021-03-15T22:28:43.196 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 42 to 162 2021-03-15T22:28:43.196 INFO:tasks.rados.rados.0.smithi076.stdout:2079: snap_remove snap 226 2021-03-15T22:28:43.211 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:43.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:43.213 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:43.314 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:43.337 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:43.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:43.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:43.439 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:43.461 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:43.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:43.463 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:43.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:43 smithi076 conmon[56514]: cluster 2021-03-15T22:28:42.185295 2021-03-15T22:28:43.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:43 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e573: 8 total, 6 up, 7 in 2021-03-15T22:28:43.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:43 smithi076 conmon[62156]: cluster 2021-03-15T22:28:42.185295 2021-03-15T22:28:43.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:43 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1603 : cluster [DBG] osdmap e573: 8 total, 6 up, 7 in 2021-03-15T22:28:43.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:43 smithi125 conmon[56739]: cluster 2021-03-15T22:28:42.185295+0000 mon.a ( 2021-03-15T22:28:43.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:43 smithi125 conmon[56739]: mon.0) 1603 : cluster [DBG] osdmap e573: 8 total, 6 up, 7 in 2021-03-15T22:28:43.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:43.588 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:43.689 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:43.711 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:43.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:43.713 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:43.814 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:43.836 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:43.838 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:43.939 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:43.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:43.962 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:43.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:44.063 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:44.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:44.087 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:44.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:44.189 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:44.196 INFO:tasks.rados.rados.0.smithi076.stdout:2079: done (4 left) 2021-03-15T22:28:44.196 INFO:tasks.rados.rados.0.smithi076.stdout:2080: snap_remove snap 230 2021-03-15T22:28:44.207 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:28:44.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:44.232 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:44.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:44.307 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[56514]: cluster 2021-03-15T22:28:43.112468+0000 mgr.y (mgr 2021-03-15T22:28:44.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[56514]: .14144) 765 : cluster [DBG] pgmap v1004: 138 pgs: 1 active+undersized+degraded+remapped, 11 peering, 37 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+undersized+remapped, 1 undersized+degraded+peered, 11 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 22 active+undersized, 35 active+clean; 568 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 232 KiB/s wr, 2 op/s; 127/1749 objects degraded (7.261%); 380/1749 objects misplaced (21.727%); 24 MiB/s, 67 objects/s recovering 2021-03-15T22:28:44.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[56514]: cluster 2021-03-15T22:28:43.191283+0000 mon.a (mon.0) 1604 : cluster [DBG] osdmap e574: 8 total, 6 up, 7 in 2021-03-15T22:28:44.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[56514]: cluster 2021-03-15T22:28:43.193064+0000 mon.a (mon.0) 1605 : cluster [WRN] Health check update: Reduced data availability: 8 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2021-03-15T22:28:44.308 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[56514]: cluster 2021-03-15T22:28:43.193092+0000 mon.a (mon.0) 1606 : cluster [WRN] Health check update: Degraded data redundancy: 127/1749 objects degraded (7.261%), 14 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:44.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[56514]: audit 2021-03-15T22:28:43.682782+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? 172.21.15.76:0/2262348751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:44.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[56514]: cluster 2021-03-15T22:28:44.183996+0000 mon.a (mon.0) 1608 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:28:44.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[56514]: audit 2021-03-15T22:28:44.188341+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? 172.21.15.76:0/2262348751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:44.309 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[56514]: cluster 2021-03-15T22:28:44.188397+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e575: 8 total, 6 up, 7 in 2021-03-15T22:28:44.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: cluster 2021-03-15T22:28:43.112468+0000 mgr.y 2021-03-15T22:28:44.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: (mgr.14144) 765 : cluster [DBG] pgmap v1004: 138 pgs: 1 active+undersized+degraded+remapped, 11 peering, 37 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+undersized+remapped, 1 undersized+degraded+peered, 11 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 22 active+undersized, 35 active+clean; 568 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 232 KiB/s wr, 2 op/s; 127/1749 objects degraded (7.261%); 380/1749 objects misplaced (21.727%); 24 MiB/s, 67 objects/s recovering 2021-03-15T22:28:44.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: cluster 2021-03-15T22:28:43.191283+0000 mon.a (mon.0 2021-03-15T22:28:44.310 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: ) 1604 : cluster [DBG] osdmap e574: 8 total, 6 up, 7 in 2021-03-15T22:28:44.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:28:44.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: T22:28:43.193064+0000 mon.a (mon.0) 1605 : 2021-03-15T22:28:44.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: cluster [WRN] Health check update: Reduced data availability: 8 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2021-03-15T22:28:44.311 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:28:44.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: 28:43.193092+0000 mon.a (mon.0 2021-03-15T22:28:44.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: ) 1606 : cluster [WRN] Health check update: Degraded data redundancy: 127/1749 objects degraded (7.261%), 14 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:44.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:28:44.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: 28:43.682782+0000 mon.a (mon.0) 2021-03-15T22:28:44.312 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: 1607 : audit [INF] from='client.? 172.21.15.76:0/2262348751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:44.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:28:44.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: 15T22:28:44.183996+0000 mon.a (mon 2021-03-15T22:28:44.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: .0) 1608 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:28:44.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: audit 2021 2021-03-15T22:28:44.313 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: -03-15T22:28:44.188341+0000 2021-03-15T22:28:44.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: mon.a (mon.0) 1609 : audit [INF] from='client.? 172.21.15.76:0/2262348751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:44.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: 2021-03-15T22:28:44.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: cluster 2021-03-15T22:28:44.314 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: 2021-03-15T22:28:44.188397+0000 mon.a 2021-03-15T22:28:44.315 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:44 smithi076 conmon[62156]: (mon.0) 1610 : cluster [DBG] osdmap e575: 8 total, 6 up, 7 in 2021-03-15T22:28:44.334 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:44.355 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:44.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:44.357 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:44.441 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:28:44.458 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:44.460 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:28:44.480 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:28:44.484 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:44.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:44.485 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:44.499 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:28:44.518 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:28:44.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: cluster 2021-03-15T22:28:44.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021 2021-03-15T22:28:44.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: - 2021-03-15T22:28:44.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 03 2021-03-15T22:28:44.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: - 2021-03-15T22:28:44.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 15 2021-03-15T22:28:44.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: T 2021-03-15T22:28:44.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 22 2021-03-15T22:28:44.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: : 2021-03-15T22:28:44.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 28 2021-03-15T22:28:44.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: : 2021-03-15T22:28:44.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 43 2021-03-15T22:28:44.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: . 2021-03-15T22:28:44.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 112468 2021-03-15T22:28:44.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: +0000 2021-03-15T22:28:44.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:44.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: ( 2021-03-15T22:28:44.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: mgr 2021-03-15T22:28:44.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: . 2021-03-15T22:28:44.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 14144 2021-03-15T22:28:44.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: ) 2021-03-15T22:28:44.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 765 2021-03-15T22:28:44.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: : 2021-03-15T22:28:44.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: cluster 2021-03-15T22:28:44.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:44.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: pgmap v1004: 138 pgs: 1 active+undersized+degraded+remapped, 11 peering, 37 active+clean+remapped, 1 active+recovering+undersized+degraded+remapped, 13 active+undersized+remapped, 1 undersized+degraded+peered, 11 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 22 active+undersized, 35 active+clean; 568 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 232 KiB/s wr, 2 op/s; 127/1749 objects degraded (7.261%); 380/1749 objects misplaced (21.727%); 24 MiB/s, 67 objects/s recovering 2021-03-15T22:28:44.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: cluster 2021-03-15T22:28:44.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021 2021-03-15T22:28:44.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: - 2021-03-15T22:28:44.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 03 2021-03-15T22:28:44.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: - 2021-03-15T22:28:44.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 15 2021-03-15T22:28:44.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: T 2021-03-15T22:28:44.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 22 2021-03-15T22:28:44.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: : 2021-03-15T22:28:44.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 28 2021-03-15T22:28:44.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: : 2021-03-15T22:28:44.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 43 2021-03-15T22:28:44.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: . 2021-03-15T22:28:44.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 191283 2021-03-15T22:28:44.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: +0000 2021-03-15T22:28:44.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:44.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: ( 2021-03-15T22:28:44.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: mon 2021-03-15T22:28:44.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: . 2021-03-15T22:28:44.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 0 2021-03-15T22:28:44.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: ) 2021-03-15T22:28:44.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 1604 2021-03-15T22:28:44.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: : 2021-03-15T22:28:44.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: cluster 2021-03-15T22:28:44.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:44.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: osdmap e574: 8 total, 6 up, 7 in 2021-03-15T22:28:44.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: cluster 2021-03-15T22:28:44.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021 2021-03-15T22:28:44.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: - 2021-03-15T22:28:44.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 03 2021-03-15T22:28:44.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: - 2021-03-15T22:28:44.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 15 2021-03-15T22:28:44.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: T 2021-03-15T22:28:44.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 22 2021-03-15T22:28:44.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: : 2021-03-15T22:28:44.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 28 2021-03-15T22:28:44.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: : 2021-03-15T22:28:44.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 43 2021-03-15T22:28:44.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: . 2021-03-15T22:28:44.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 193064 2021-03-15T22:28:44.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: +0000 2021-03-15T22:28:44.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:44.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: ( 2021-03-15T22:28:44.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: mon 2021-03-15T22:28:44.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: . 2021-03-15T22:28:44.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 0 2021-03-15T22:28:44.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: ) 2021-03-15T22:28:44.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 1605 2021-03-15T22:28:44.558 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: : 2021-03-15T22:28:44.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: cluster 2021-03-15T22:28:44.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: [WRN] 2021-03-15T22:28:44.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: Health check update: Reduced data availability: 8 pgs inactive, 3 pgs peering (PG_AVAILABILITY) 2021-03-15T22:28:44.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: cluster 2021-03-15T22:28:44.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021-03-15T22:28:44.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 2021 2021-03-15T22:28:44.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: - 2021-03-15T22:28:44.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 03 2021-03-15T22:28:44.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: - 2021-03-15T22:28:44.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 15 2021-03-15T22:28:44.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: T 2021-03-15T22:28:44.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: 22 2021-03-15T22:28:44.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: :28:43.193092+0000 mon.a (mon.0) 1606 : cluster [WRN] Health check update: Degraded data redundancy: 127/1749 objects degraded (7.261%), 14 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:44.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: audit 2021-03-15T22:28:43.682782+0000 mon.a (mon.0) 1607 : audit [INF] from='client.? 172.21.15.76:0/2262348751' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:44.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: cluster 2021-03-15T22:28:44.183996+0000 mon.a (mon.0) 1608 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:28:44.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: audit 2021-03-15T22:28:44.188341+0000 mon.a (mon.0) 1609 : audit [INF] from='client.? 172.21.15.76:0/2262348751' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:28:44.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:44 smithi125 conmon[56739]: cluster 2021-03-15T22:28:44.188397+0000 mon.a (mon.0) 1610 : cluster [DBG] osdmap e575: 8 total, 6 up, 7 in 2021-03-15T22:28:44.564 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:28:44.564 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:28:44.565 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:28:44.565 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:28:44.565 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:28:44.566 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:28:44.566 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:28:44.566 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:28:44.566 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:28:44.586 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:44.612 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:44.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:44.614 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:44.652 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 0] out_osds: [3] dead_osds: [2, 5] live_osds: [0, 1, 3, 4, 6, 7] 2021-03-15T22:28:44.652 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:28:44.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump -f json-pretty 2021-03-15T22:28:44.715 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:44.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:44.739 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:44.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:44.841 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:44.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:44.866 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:44.866 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:44.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:44.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:44.990 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:44.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:45.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:45.116 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:45.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:45.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:45.181 INFO:tasks.rados.rados.0.smithi076.stdout:2080: done (4 left) 2021-03-15T22:28:45.182 INFO:tasks.rados.rados.0.smithi076.stdout:2081: copy_from oid 32 from oid 38 current snap is 235 2021-03-15T22:28:45.182 INFO:tasks.rados.rados.0.smithi076.stdout:2082: write oid 21 current snap is 235 2021-03-15T22:28:45.182 INFO:tasks.rados.rados.0.smithi076.stdout:2082: seq_num 724 ranges {568268=733811,2000022=168246} 2021-03-15T22:28:45.189 INFO:tasks.rados.rados.0.smithi076.stdout:2082: writing smithi07694878-21 from 568268 to 1302079 tid 1 2021-03-15T22:28:45.195 INFO:tasks.rados.rados.0.smithi076.stdout:2082: writing smithi07694878-21 from 2000022 to 2168268 tid 2 2021-03-15T22:28:45.195 INFO:tasks.rados.rados.0.smithi076.stdout:2074: expect (ObjNum 549 snap 175 seq_num 549) 2021-03-15T22:28:45.219 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:45.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:45.269 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:45.270 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:45.353 INFO:tasks.rados.rados.0.smithi076.stdout:2076: finishing write tid 1 to smithi07694878-4 2021-03-15T22:28:45.353 INFO:tasks.rados.rados.0.smithi076.stdout:2076: finishing write tid 2 to smithi07694878-4 2021-03-15T22:28:45.353 INFO:tasks.rados.rados.0.smithi076.stdout:2076: finishing write tid 3 to smithi07694878-4 2021-03-15T22:28:45.354 INFO:tasks.rados.rados.0.smithi076.stdout:2076: finishing write tid 4 to smithi07694878-4 2021-03-15T22:28:45.354 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 540 (ObjNum 723 snap 235 seq_num 723) dirty exists 2021-03-15T22:28:45.354 INFO:tasks.rados.rados.0.smithi076.stdout:2076: left oid 4 (ObjNum 723 snap 235 seq_num 723) 2021-03-15T22:28:45.354 INFO:tasks.rados.rados.0.smithi076.stdout:2075: finishing copy_from to smithi07694878-30 2021-03-15T22:28:45.355 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 855 (ObjNum 709 snap 227 seq_num 709) dirty exists 2021-03-15T22:28:45.355 INFO:tasks.rados.rados.0.smithi076.stdout:2075: finishing copy_from racing read to smithi07694878-30 2021-03-15T22:28:45.355 INFO:tasks.rados.rados.0.smithi076.stdout:2078: finishing rollback tid 0 to smithi07694878-42 2021-03-15T22:28:45.355 INFO:tasks.rados.rados.0.smithi076.stdout:2078: finishing rollback tid 1 to smithi07694878-42 2021-03-15T22:28:45.355 INFO:tasks.rados.rados.0.smithi076.stdout:2078: finishing rollback tid 2 to smithi07694878-42 2021-03-15T22:28:45.356 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 1490 (ObjNum 451 snap 130 seq_num 451) dirty exists 2021-03-15T22:28:45.356 INFO:tasks.rados.rados.0.smithi076.stdout:2081: finishing copy_from to smithi07694878-32 2021-03-15T22:28:45.356 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 1029 (ObjNum 717 snap 233 seq_num 717) dirty exists 2021-03-15T22:28:45.356 INFO:tasks.rados.rados.0.smithi076.stdout:2081: finishing copy_from racing read to smithi07694878-32 2021-03-15T22:28:45.357 INFO:tasks.rados.rados.0.smithi076.stdout:2082: finishing write tid 1 to smithi07694878-21 2021-03-15T22:28:45.357 INFO:tasks.rados.rados.0.smithi076.stdout:2074: done (5 left) 2021-03-15T22:28:45.357 INFO:tasks.rados.rados.0.smithi076.stdout:2075: done (4 left) 2021-03-15T22:28:45.357 INFO:tasks.rados.rados.0.smithi076.stdout:2076: done (3 left) 2021-03-15T22:28:45.357 INFO:tasks.rados.rados.0.smithi076.stdout:2078: done (2 left) 2021-03-15T22:28:45.358 INFO:tasks.rados.rados.0.smithi076.stdout:2081: done (1 left) 2021-03-15T22:28:45.358 INFO:tasks.rados.rados.0.smithi076.stdout:2083: read oid 9 snap -1 2021-03-15T22:28:45.358 INFO:tasks.rados.rados.0.smithi076.stdout:2083: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:28:45.358 INFO:tasks.rados.rados.0.smithi076.stdout:2084: delete oid 33 current snap is 235 2021-03-15T22:28:45.359 INFO:tasks.rados.rados.0.smithi076.stdout:2082: finishing write tid 2 to smithi07694878-21 2021-03-15T22:28:45.359 INFO:tasks.rados.rados.0.smithi076.stdout:2082: finishing write tid 3 to smithi07694878-21 2021-03-15T22:28:45.359 INFO:tasks.rados.rados.0.smithi076.stdout:2082: finishing write tid 4 to smithi07694878-21 2021-03-15T22:28:45.359 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 1496 (ObjNum 724 snap 235 seq_num 724) dirty exists 2021-03-15T22:28:45.359 INFO:tasks.rados.rados.0.smithi076.stdout:2082: left oid 21 (ObjNum 724 snap 235 seq_num 724) 2021-03-15T22:28:45.360 INFO:tasks.rados.rados.0.smithi076.stdout:2082: done (2 left) 2021-03-15T22:28:45.360 INFO:tasks.rados.rados.0.smithi076.stdout:2084: done (1 left) 2021-03-15T22:28:45.361 INFO:tasks.rados.rados.0.smithi076.stdout:2085: snap_create 2021-03-15T22:28:45.361 INFO:tasks.rados.rados.0.smithi076.stdout:2083: expect (ObjNum 451 snap 130 seq_num 451) 2021-03-15T22:28:45.371 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:45.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:45.395 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:45.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:45.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: audit 2021-03-15T22:28:45.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 2021-03-15T22:28:45.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 2021 2021-03-15T22:28:45.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: - 2021-03-15T22:28:45.483 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 03 2021-03-15T22:28:45.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: - 2021-03-15T22:28:45.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 15 2021-03-15T22:28:45.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: T 2021-03-15T22:28:45.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 22 2021-03-15T22:28:45.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: : 2021-03-15T22:28:45.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 28 2021-03-15T22:28:45.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: : 2021-03-15T22:28:45.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 44 2021-03-15T22:28:45.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: . 2021-03-15T22:28:45.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 420799 2021-03-15T22:28:45.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: +0000 2021-03-15T22:28:45.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 2021-03-15T22:28:45.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: mon.a 2021-03-15T22:28:45.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: ( 2021-03-15T22:28:45.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: mon 2021-03-15T22:28:45.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: . 2021-03-15T22:28:45.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 0 2021-03-15T22:28:45.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: ) 2021-03-15T22:28:45.487 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 1611 2021-03-15T22:28:45.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: : 2021-03-15T22:28:45.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: audit 2021-03-15T22:28:45.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 2021-03-15T22:28:45.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: [DBG] 2021-03-15T22:28:45.488 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 2021-03-15T22:28:45.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/2434567820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:45.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 2021-03-15T22:28:45.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: cluster 2021-03-15T22:28:45.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 2021-03-15T22:28:45.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 2021 2021-03-15T22:28:45.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: - 2021-03-15T22:28:45.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 03 2021-03-15T22:28:45.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: - 2021-03-15T22:28:45.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 15 2021-03-15T22:28:45.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: T 2021-03-15T22:28:45.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 22 2021-03-15T22:28:45.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: : 2021-03-15T22:28:45.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 28 2021-03-15T22:28:45.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: : 2021-03-15T22:28:45.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 44 2021-03-15T22:28:45.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: . 2021-03-15T22:28:45.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 484263 2021-03-15T22:28:45.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: +0000 2021-03-15T22:28:45.493 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 2021-03-15T22:28:45.493 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: osd.6 2021-03-15T22:28:45.493 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: ( 2021-03-15T22:28:45.493 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: osd 2021-03-15T22:28:45.493 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: . 2021-03-15T22:28:45.494 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 6 2021-03-15T22:28:45.494 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: ) 2021-03-15T22:28:45.494 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 90 : cluster [DBG] 3.c scrub ok 2021-03-15T22:28:45.494 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:28:45.495 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: 03-15T22:28:44.765841+0000 osd.0 ( 2021-03-15T22:28:45.495 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: osd.0) 83 : cluster [DBG] 4.16 deep-scrub ok 2021-03-15T22:28:45.495 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[56514]: cluster 2021-03-15T22:28:45.172517+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e576: 8 total, 6 up, 7 in 2021-03-15T22:28:45.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: audit 2021-03-15T22:28:45.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021 2021-03-15T22:28:45.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: - 2021-03-15T22:28:45.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 03 2021-03-15T22:28:45.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: - 2021-03-15T22:28:45.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 15 2021-03-15T22:28:45.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: T 2021-03-15T22:28:45.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 22 2021-03-15T22:28:45.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 28 2021-03-15T22:28:45.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 44 2021-03-15T22:28:45.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: . 2021-03-15T22:28:45.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 420799 2021-03-15T22:28:45.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: +0000 2021-03-15T22:28:45.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: mon.a 2021-03-15T22:28:45.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: ( 2021-03-15T22:28:45.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: mon 2021-03-15T22:28:45.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: . 2021-03-15T22:28:45.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 0 2021-03-15T22:28:45.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: ) 2021-03-15T22:28:45.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 1611 2021-03-15T22:28:45.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: audit 2021-03-15T22:28:45.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: [DBG] 2021-03-15T22:28:45.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2434567820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:45.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: cluster 2021-03-15T22:28:45.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021 2021-03-15T22:28:45.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: - 2021-03-15T22:28:45.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 03 2021-03-15T22:28:45.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: - 2021-03-15T22:28:45.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 15 2021-03-15T22:28:45.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: T 2021-03-15T22:28:45.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 22 2021-03-15T22:28:45.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 28 2021-03-15T22:28:45.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 44 2021-03-15T22:28:45.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: . 2021-03-15T22:28:45.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 484263 2021-03-15T22:28:45.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: +0000 2021-03-15T22:28:45.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: osd.6 2021-03-15T22:28:45.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: ( 2021-03-15T22:28:45.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: osd 2021-03-15T22:28:45.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: . 2021-03-15T22:28:45.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 6 2021-03-15T22:28:45.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: ) 2021-03-15T22:28:45.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 90 2021-03-15T22:28:45.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: cluster 2021-03-15T22:28:45.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: [DBG] 2021-03-15T22:28:45.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 3.c scrub ok 2021-03-15T22:28:45.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: cluster 2021-03-15T22:28:45.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021 2021-03-15T22:28:45.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: - 2021-03-15T22:28:45.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 03 2021-03-15T22:28:45.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: - 2021-03-15T22:28:45.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 15 2021-03-15T22:28:45.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: T 2021-03-15T22:28:45.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 22 2021-03-15T22:28:45.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 28 2021-03-15T22:28:45.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 44 2021-03-15T22:28:45.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: . 2021-03-15T22:28:45.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 765841 2021-03-15T22:28:45.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: +0000 2021-03-15T22:28:45.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: osd.0 2021-03-15T22:28:45.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: ( 2021-03-15T22:28:45.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: osd 2021-03-15T22:28:45.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: . 2021-03-15T22:28:45.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 0 2021-03-15T22:28:45.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: ) 2021-03-15T22:28:45.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 83 2021-03-15T22:28:45.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: cluster 2021-03-15T22:28:45.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: [DBG] 2021-03-15T22:28:45.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 4.16 deep-scrub ok 2021-03-15T22:28:45.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: cluster 2021-03-15T22:28:45.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021 2021-03-15T22:28:45.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: - 2021-03-15T22:28:45.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 03 2021-03-15T22:28:45.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: - 2021-03-15T22:28:45.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 15 2021-03-15T22:28:45.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: T 2021-03-15T22:28:45.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 22 2021-03-15T22:28:45.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 28 2021-03-15T22:28:45.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 45 2021-03-15T22:28:45.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: . 2021-03-15T22:28:45.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 172517 2021-03-15T22:28:45.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: +0000 2021-03-15T22:28:45.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: mon.a 2021-03-15T22:28:45.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: ( 2021-03-15T22:28:45.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: mon 2021-03-15T22:28:45.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: . 2021-03-15T22:28:45.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 0 2021-03-15T22:28:45.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: ) 2021-03-15T22:28:45.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 1612 2021-03-15T22:28:45.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: : 2021-03-15T22:28:45.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: cluster 2021-03-15T22:28:45.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: [DBG] 2021-03-15T22:28:45.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: osdmap e576: 8 total, 6 up, 7 in 2021-03-15T22:28:45.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:45 smithi076 conmon[62156]: 2021-03-15T22:28:45.525 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:28:45.526 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:28:45.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:28:45.526 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:28:45.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.526 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1191 7.6 7:61e0cc05:::smithi07694878-16:head [copy-from ver 101] snapc 4e=[4e,4d,3f] ondisk+write+known_if_redirected e175)", 2021-03-15T22:28:45.527 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:28:45.527 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 434.66282120300002, 2021-03-15T22:28:45.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.97834980800000004, 2021-03-15T22:28:45.527 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.528 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:45.528 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.528 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.528 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.528 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1191 2021-03-15T22:28:45.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.529 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:28:45.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:28:45.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0440000000000003e-06 2021-03-15T22:28:45.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067745+0000", 2021-03-15T22:28:45.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4340000000000001e-06 2021-03-15T22:28:45.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067749+0000", 2021-03-15T22:28:45.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9241000000000001e-05 2021-03-15T22:28:45.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067768+0000", 2021-03-15T22:28:45.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3010000000000007e-06 2021-03-15T22:28:45.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.535 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067776+0000", 2021-03-15T22:28:45.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5014e-05 2021-03-15T22:28:45.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067821+0000", 2021-03-15T22:28:45.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9779999999999994e-06 2021-03-15T22:28:45.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:45.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067830+0000", 2021-03-15T22:28:45.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.95797833899999996 2021-03-15T22:28:45.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.025809+0000", 2021-03-15T22:28:45.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138412 2021-03-15T22:28:45.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.025947+0000", 2021-03-15T22:28:45.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006355744 2021-03-15T22:28:45.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,7", 2021-03-15T22:28:45.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.032303+0000", 2021-03-15T22:28:45.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043109439999999997 2021-03-15T22:28:45.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:45.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.036614+0000", 2021-03-15T22:28:45.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092878839999999997 2021-03-15T22:28:45.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.045902+0000", 2021-03-15T22:28:45.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8556999999999999e-05 2021-03-15T22:28:45.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.045990+0000", 2021-03-15T22:28:45.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6665999999999997e-05 2021-03-15T22:28:45.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:45.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046037+0000", 2021-03-15T22:28:45.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.325e-05 2021-03-15T22:28:45.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046090+0000", 2021-03-15T22:28:45.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.97834980800000004 2021-03-15T22:28:45.547 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.547 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.548 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.549 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1192 7.6 7:61e0cc05:::smithi07694878-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e175)", 2021-03-15T22:28:45.549 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:28:45.549 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 434.66277188399999, 2021-03-15T22:28:45.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.978448604, 2021-03-15T22:28:45.549 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.550 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:45.550 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.550 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.550 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.551 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1192 2021-03-15T22:28:45.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.551 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:28:45.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:28:45.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.401e-06 2021-03-15T22:28:45.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067791+0000", 2021-03-15T22:28:45.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.083e-06 2021-03-15T22:28:45.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067794+0000", 2021-03-15T22:28:45.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4919999999999996e-06 2021-03-15T22:28:45.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067802+0000", 2021-03-15T22:28:45.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.383e-06 2021-03-15T22:28:45.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067807+0000", 2021-03-15T22:28:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8674000000000002e-05 2021-03-15T22:28:45.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067856+0000", 2021-03-15T22:28:45.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5890000000000003e-06 2021-03-15T22:28:45.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:45.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067864+0000", 2021-03-15T22:28:45.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.95935619400000005 2021-03-15T22:28:45.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.027221+0000", 2021-03-15T22:28:45.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9426999999999998e-05 2021-03-15T22:28:45.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:45.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.027290+0000", 2021-03-15T22:28:45.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085250369999999992 2021-03-15T22:28:45.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.035815+0000", 2021-03-15T22:28:45.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9997999999999997e-05 2021-03-15T22:28:45.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.563 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:45.563 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.035875+0000", 2021-03-15T22:28:45.564 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010246089 2021-03-15T22:28:45.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.564 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.564 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046121+0000", 2021-03-15T22:28:45.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3338999999999998e-05 2021-03-15T22:28:45.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046184+0000", 2021-03-15T22:28:45.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3897999999999998e-05 2021-03-15T22:28:45.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.566 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.567 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046238+0000", 2021-03-15T22:28:45.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.978448604 2021-03-15T22:28:45.567 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.567 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.568 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.568 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1563 7.f 7:fee9f235:::smithi07694878-11:head [copy-from ver 242] snapc 6a=[6a,69,68] ondisk+write+known_if_redirected e204)", 2021-03-15T22:28:45.568 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.059465+0000", 2021-03-15T22:28:45.569 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 405.67109636999999, 2021-03-15T22:28:45.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026787723999999999, 2021-03-15T22:28:45.569 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.569 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:45.570 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.570 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.570 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.570 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1563 2021-03-15T22:28:45.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.571 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059465+0000", 2021-03-15T22:28:45.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059465+0000", 2021-03-15T22:28:45.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.382e-06 2021-03-15T22:28:45.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.576 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059469+0000", 2021-03-15T22:28:45.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.294e-06 2021-03-15T22:28:45.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.578 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059471+0000", 2021-03-15T22:28:45.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4338999999999999e-05 2021-03-15T22:28:45.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.578 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059485+0000", 2021-03-15T22:28:45.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3259999999999998e-06 2021-03-15T22:28:45.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059493+0000", 2021-03-15T22:28:45.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9969999999999995e-05 2021-03-15T22:28:45.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.581 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.581 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059573+0000", 2021-03-15T22:28:45.581 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017286299999999999 2021-03-15T22:28:45.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.582 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.582 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059745+0000", 2021-03-15T22:28:45.582 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0089917209999999994 2021-03-15T22:28:45.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.582 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.583 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:28:45.583 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.068737+0000", 2021-03-15T22:28:45.583 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0046724000000000002 2021-03-15T22:28:45.583 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.583 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.584 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:45.584 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.073410+0000", 2021-03-15T22:28:45.584 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0099697450000000003 2021-03-15T22:28:45.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.585 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.585 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.585 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.083379+0000", 2021-03-15T22:28:45.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027667730000000001 2021-03-15T22:28:45.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.586 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.586 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086146+0000", 2021-03-15T22:28:45.586 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6458000000000001e-05 2021-03-15T22:28:45.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.587 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:45.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086192+0000", 2021-03-15T22:28:45.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0452999999999999e-05 2021-03-15T22:28:45.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086253+0000", 2021-03-15T22:28:45.588 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026787723999999999 2021-03-15T22:28:45.589 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.589 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.589 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.590 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1564 7.f 7:fee9f235:::smithi07694878-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e204)", 2021-03-15T22:28:45.590 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:59.059507+0000", 2021-03-15T22:28:45.590 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 405.67105505900003, 2021-03-15T22:28:45.590 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026908793, 2021-03-15T22:28:45.591 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.591 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:45.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.592 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1564 2021-03-15T22:28:45.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.592 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059507+0000", 2021-03-15T22:28:45.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059507+0000", 2021-03-15T22:28:45.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.274e-06 2021-03-15T22:28:45.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059508+0000", 2021-03-15T22:28:45.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6589999999999999e-06 2021-03-15T22:28:45.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059509+0000", 2021-03-15T22:28:45.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0969999999999995e-06 2021-03-15T22:28:45.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059519+0000", 2021-03-15T22:28:45.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5739999999999997e-06 2021-03-15T22:28:45.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059523+0000", 2021-03-15T22:28:45.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00046872599999999998 2021-03-15T22:28:45.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.599 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.599 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.059992+0000", 2021-03-15T22:28:45.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7046999999999999e-05 2021-03-15T22:28:45.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:45.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.060039+0000", 2021-03-15T22:28:45.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012485036 2021-03-15T22:28:45.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.072524+0000", 2021-03-15T22:28:45.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6324999999999999e-05 2021-03-15T22:28:45.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:45.602 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.072590+0000", 2021-03-15T22:28:45.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013691045000000001 2021-03-15T22:28:45.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086281+0000", 2021-03-15T22:28:45.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6152e-05 2021-03-15T22:28:45.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086347+0000", 2021-03-15T22:28:45.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7858e-05 2021-03-15T22:28:45.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:59.086415+0000", 2021-03-15T22:28:45.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026908793 2021-03-15T22:28:45.606 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.606 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.606 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.607 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:548 7.6 e204/175 ondisk, result = 0)", 2021-03-15T22:28:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:28:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 404.64903897400001, 2021-03-15T22:28:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045793304999999999, 2021-03-15T22:28:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.608 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:28:45.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:28:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6390000000000001e-06 2021-03-15T22:28:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.611 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081524+0000", 2021-03-15T22:28:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.863e-06 2021-03-15T22:28:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081526+0000", 2021-03-15T22:28:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1234e-05 2021-03-15T22:28:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081537+0000", 2021-03-15T22:28:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5210000000000002e-06 2021-03-15T22:28:45.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081543+0000", 2021-03-15T22:28:45.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045750591 2021-03-15T22:28:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127293+0000", 2021-03-15T22:28:45.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6920000000000008e-06 2021-03-15T22:28:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127302+0000", 2021-03-15T22:28:45.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3764999999999999e-05 2021-03-15T22:28:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127316+0000", 2021-03-15T22:28:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045793304999999999 2021-03-15T22:28:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.620 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:548 7.6 e205/175 ondisk, result = 0)", 2021-03-15T22:28:45.620 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:28:45.620 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 404.64471549199999, 2021-03-15T22:28:45.621 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041608198999999998, 2021-03-15T22:28:45.621 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.621 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:45.621 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.622 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:28:45.622 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:28:45.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4779999999999998e-06 2021-03-15T22:28:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085849+0000", 2021-03-15T22:28:45.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.046e-06 2021-03-15T22:28:45.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085851+0000", 2021-03-15T22:28:45.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4150000000000005e-06 2021-03-15T22:28:45.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085860+0000", 2021-03-15T22:28:45.627 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0900000000000001e-06 2021-03-15T22:28:45.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085866+0000", 2021-03-15T22:28:45.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041511365000000001 2021-03-15T22:28:45.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.628 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.628 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127378+0000", 2021-03-15T22:28:45.629 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6030000000000001e-06 2021-03-15T22:28:45.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127384+0000", 2021-03-15T22:28:45.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0202000000000003e-05 2021-03-15T22:28:45.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127454+0000", 2021-03-15T22:28:45.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041608198999999998 2021-03-15T22:28:45.631 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.631 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.631 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.632 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:61 7.5 e205/91 7:a9545d62:::smithi07694878-1:68 v 205'72, mlcod=202'69)", 2021-03-15T22:28:45.632 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:28:45.633 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 404.63969174599998, 2021-03-15T22:28:45.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034962251999999999, 2021-03-15T22:28:45.633 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.633 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:45.633 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:28:45.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:28:45.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.798e-06 2021-03-15T22:28:45.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090877+0000", 2021-03-15T22:28:45.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3779999999999999e-06 2021-03-15T22:28:45.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.637 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.637 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090879+0000", 2021-03-15T22:28:45.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1902999999999999e-05 2021-03-15T22:28:45.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.638 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.638 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090891+0000", 2021-03-15T22:28:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7780000000000004e-06 2021-03-15T22:28:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.639 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090897+0000", 2021-03-15T22:28:45.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0942000000000002e-05 2021-03-15T22:28:45.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090938+0000", 2021-03-15T22:28:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6379e-05 2021-03-15T22:28:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.641 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090954+0000", 2021-03-15T22:28:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034836759000000002 2021-03-15T22:28:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125791+0000", 2021-03-15T22:28:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1315e-05 2021-03-15T22:28:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125832+0000", 2021-03-15T22:28:45.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034962251999999999 2021-03-15T22:28:45.644 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.644 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.645 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.645 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.6.0:214 7.d e205/91 7:bd183e1b:::smithi07694878-46:68 v 205'30, mlcod=202'27)", 2021-03-15T22:28:45.645 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:28:45.646 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 404.63924986699999, 2021-03-15T22:28:45.646 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034834256000000001, 2021-03-15T22:28:45.646 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.646 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:45.647 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:28:45.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:28:45.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3590000000000002e-06 2021-03-15T22:28:45.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091314+0000", 2021-03-15T22:28:45.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.57e-06 2021-03-15T22:28:45.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091316+0000", 2021-03-15T22:28:45.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3749999999999999e-06 2021-03-15T22:28:45.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.652 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091322+0000", 2021-03-15T22:28:45.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2140000000000004e-06 2021-03-15T22:28:45.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091327+0000", 2021-03-15T22:28:45.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017519700000000001 2021-03-15T22:28:45.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091502+0000", 2021-03-15T22:28:45.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2955e-05 2021-03-15T22:28:45.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091525+0000", 2021-03-15T22:28:45.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034582099999999998 2021-03-15T22:28:45.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:45.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.126108+0000", 2021-03-15T22:28:45.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8485999999999997e-05 2021-03-15T22:28:45.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.126146+0000", 2021-03-15T22:28:45.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034834256000000001 2021-03-15T22:28:45.658 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.658 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.658 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.659 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1686 7.f 7:fee9f235:::smithi07694878-11:head [copy-from ver 242] snapc 7a=[7a,79,78,76,75] ondisk+write+known_if_redirected e220)", 2021-03-15T22:28:45.659 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:15.039251+0000", 2021-03-15T22:28:45.659 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 389.691311058, 2021-03-15T22:28:45.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026727377, 2021-03-15T22:28:45.659 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.660 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:45.660 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.660 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.660 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.660 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1686 2021-03-15T22:28:45.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.661 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039251+0000", 2021-03-15T22:28:45.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039251+0000", 2021-03-15T22:28:45.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0810000000000001e-06 2021-03-15T22:28:45.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039256+0000", 2021-03-15T22:28:45.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6880000000000001e-06 2021-03-15T22:28:45.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039259+0000", 2021-03-15T22:28:45.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7503000000000002e-05 2021-03-15T22:28:45.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039277+0000", 2021-03-15T22:28:45.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3869999999999997e-06 2021-03-15T22:28:45.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039284+0000", 2021-03-15T22:28:45.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8112999999999998e-05 2021-03-15T22:28:45.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039332+0000", 2021-03-15T22:28:45.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7928000000000002e-05 2021-03-15T22:28:45.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039430+0000", 2021-03-15T22:28:45.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0067539030000000003 2021-03-15T22:28:45.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:28:45.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.046184+0000", 2021-03-15T22:28:45.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056836600000000001 2021-03-15T22:28:45.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:45.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.051868+0000", 2021-03-15T22:28:45.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.008344677 2021-03-15T22:28:45.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.672 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.672 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.060212+0000", 2021-03-15T22:28:45.672 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0056656900000000001 2021-03-15T22:28:45.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.065878+0000", 2021-03-15T22:28:45.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9669999999999997e-05 2021-03-15T22:28:45.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:45.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.065928+0000", 2021-03-15T22:28:45.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0077000000000002e-05 2021-03-15T22:28:45.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.065978+0000", 2021-03-15T22:28:45.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026727377 2021-03-15T22:28:45.676 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.676 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.677 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.677 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1687 7.f 7:fee9f235:::smithi07694878-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e220)", 2021-03-15T22:28:45.677 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:15.039297+0000", 2021-03-15T22:28:45.678 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 389.69126508099998, 2021-03-15T22:28:45.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026894929000000001, 2021-03-15T22:28:45.678 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.678 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:45.678 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.679 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.679 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.679 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1687 2021-03-15T22:28:45.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.680 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039297+0000", 2021-03-15T22:28:45.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039297+0000", 2021-03-15T22:28:45.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5549999999999999e-06 2021-03-15T22:28:45.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039298+0000", 2021-03-15T22:28:45.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.699e-06 2021-03-15T22:28:45.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039300+0000", 2021-03-15T22:28:45.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8300000000000001e-06 2021-03-15T22:28:45.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.684 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039306+0000", 2021-03-15T22:28:45.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0069999999999999e-06 2021-03-15T22:28:45.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039310+0000", 2021-03-15T22:28:45.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025812099999999999 2021-03-15T22:28:45.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039568+0000", 2021-03-15T22:28:45.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8504999999999997e-05 2021-03-15T22:28:45.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:45.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.039616+0000", 2021-03-15T22:28:45.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010537019999999999 2021-03-15T22:28:45.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.050153+0000", 2021-03-15T22:28:45.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011407 2021-03-15T22:28:45.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:45.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.050267+0000", 2021-03-15T22:28:45.690 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015755377000000001 2021-03-15T22:28:45.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066023+0000", 2021-03-15T22:28:45.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2999999999999998e-05 2021-03-15T22:28:45.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066106+0000", 2021-03-15T22:28:45.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5745e-05 2021-03-15T22:28:45.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:15.066191+0000", 2021-03-15T22:28:45.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026894929000000001 2021-03-15T22:28:45.693 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.693 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.694 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.694 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2103 7.f 7:fee9f235:::smithi07694878-11:head [copy-from ver 398] snapc 9a=[9a,97,93,92,8d,88,80,7d] ondisk+write+known_if_redirected e253)", 2021-03-15T22:28:45.695 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:28:45.695 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 356.518966885, 2021-03-15T22:28:45.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029938436999999998, 2021-03-15T22:28:45.695 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.695 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:45.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.696 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2103 2021-03-15T22:28:45.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.697 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:28:45.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:28:45.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1350000000000002e-06 2021-03-15T22:28:45.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211599+0000", 2021-03-15T22:28:45.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1489999999999998e-06 2021-03-15T22:28:45.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211602+0000", 2021-03-15T22:28:45.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8119000000000001e-05 2021-03-15T22:28:45.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211620+0000", 2021-03-15T22:28:45.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9529999999999999e-06 2021-03-15T22:28:45.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211628+0000", 2021-03-15T22:28:45.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015272800000000001 2021-03-15T22:28:45.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211781+0000", 2021-03-15T22:28:45.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0270000000000001e-06 2021-03-15T22:28:45.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:28:45.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211790+0000", 2021-03-15T22:28:45.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035614209999999999 2021-03-15T22:28:45.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215351+0000", 2021-03-15T22:28:45.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8361999999999994e-05 2021-03-15T22:28:45.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215420+0000", 2021-03-15T22:28:45.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013426624999999999 2021-03-15T22:28:45.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:28:45.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.228846+0000", 2021-03-15T22:28:45.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041102990000000004 2021-03-15T22:28:45.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:45.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232957+0000", 2021-03-15T22:28:45.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013870029999999999 2021-03-15T22:28:45.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.234344+0000", 2021-03-15T22:28:45.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007045791 2021-03-15T22:28:45.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241389+0000", 2021-03-15T22:28:45.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0001999999999998e-05 2021-03-15T22:28:45.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:45.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241459+0000", 2021-03-15T22:28:45.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3823000000000001e-05 2021-03-15T22:28:45.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241533+0000", 2021-03-15T22:28:45.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029938436999999998 2021-03-15T22:28:45.714 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.714 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.715 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.715 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2104 7.f 7:fee9f235:::smithi07694878-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e253)", 2021-03-15T22:28:45.715 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:28:45.716 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 356.51882514900001, 2021-03-15T22:28:45.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029977025000000001, 2021-03-15T22:28:45.716 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.716 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:45.716 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.717 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.717 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.717 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2104 2021-03-15T22:28:45.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.718 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:28:45.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:28:45.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.491e-06 2021-03-15T22:28:45.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211738+0000", 2021-03-15T22:28:45.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.813e-06 2021-03-15T22:28:45.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211740+0000", 2021-03-15T22:28:45.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2709999999999992e-06 2021-03-15T22:28:45.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211748+0000", 2021-03-15T22:28:45.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8559999999999998e-06 2021-03-15T22:28:45.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211753+0000", 2021-03-15T22:28:45.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014630000000000001 2021-03-15T22:28:45.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211899+0000", 2021-03-15T22:28:45.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7549999999999997e-06 2021-03-15T22:28:45.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:28:45.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211906+0000", 2021-03-15T22:28:45.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036139900000000001 2021-03-15T22:28:45.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215520+0000", 2021-03-15T22:28:45.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7696999999999999e-05 2021-03-15T22:28:45.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:45.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215548+0000", 2021-03-15T22:28:45.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016795027000000001 2021-03-15T22:28:45.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232343+0000", 2021-03-15T22:28:45.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1192000000000005e-05 2021-03-15T22:28:45.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:45.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232414+0000", 2021-03-15T22:28:45.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091605279999999994 2021-03-15T22:28:45.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241574+0000", 2021-03-15T22:28:45.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5976000000000007e-05 2021-03-15T22:28:45.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241640+0000", 2021-03-15T22:28:45.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3128999999999997e-05 2021-03-15T22:28:45.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241713+0000", 2021-03-15T22:28:45.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029977025000000001 2021-03-15T22:28:45.733 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.734 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.734 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2221 7.f 7:ffef77ba:::smithi07694878-13:head [sparse-read 0~2702901,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:28:45.735 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:28:45.735 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 346.67957353899999, 2021-03-15T22:28:45.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0065843969999999, 2021-03-15T22:28:45.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.736 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:45.736 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.736 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.736 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.736 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2221 2021-03-15T22:28:45.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.737 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:28:45.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:28:45.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0599999999999999e-06 2021-03-15T22:28:45.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050991+0000", 2021-03-15T22:28:45.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0379999999999998e-06 2021-03-15T22:28:45.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050993+0000", 2021-03-15T22:28:45.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6500000000000002e-06 2021-03-15T22:28:45.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051002+0000", 2021-03-15T22:28:45.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3650000000000003e-06 2021-03-15T22:28:45.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051007+0000", 2021-03-15T22:28:45.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015642 2021-03-15T22:28:45.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051164+0000", 2021-03-15T22:28:45.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7530000000000007e-06 2021-03-15T22:28:45.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.745 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:45.745 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051172+0000", 2021-03-15T22:28:45.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.004477362 2021-03-15T22:28:45.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.055650+0000", 2021-03-15T22:28:45.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134535 2021-03-15T22:28:45.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.055784+0000", 2021-03-15T22:28:45.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001788214 2021-03-15T22:28:45.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.057572+0000", 2021-03-15T22:28:45.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0065843969999999 2021-03-15T22:28:45.748 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.749 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.749 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.750 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2222 7.f 7:ffef77ba:::smithi07694878-13:head [read 0~2823609,checksum] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:28:45.750 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:28:45.750 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 346.67954891599999, 2021-03-15T22:28:45.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0081352180000001, 2021-03-15T22:28:45.750 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.751 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:45.751 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.751 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.751 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.751 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2222 2021-03-15T22:28:45.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.752 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:28:45.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:28:45.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6190000000000001e-06 2021-03-15T22:28:45.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051014+0000", 2021-03-15T22:28:45.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.217e-06 2021-03-15T22:28:45.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051016+0000", 2021-03-15T22:28:45.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8270000000000003e-06 2021-03-15T22:28:45.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051021+0000", 2021-03-15T22:28:45.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4529999999999998e-06 2021-03-15T22:28:45.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.758 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051025+0000", 2021-03-15T22:28:45.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000168445 2021-03-15T22:28:45.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.759 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051193+0000", 2021-03-15T22:28:45.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5860000000000001e-06 2021-03-15T22:28:45.759 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:45.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051200+0000", 2021-03-15T22:28:45.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0070519229999999 2021-03-15T22:28:45.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.058252+0000", 2021-03-15T22:28:45.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1403000000000006e-05 2021-03-15T22:28:45.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.058333+0000", 2021-03-15T22:28:45.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00081474499999999997 2021-03-15T22:28:45.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.059148+0000", 2021-03-15T22:28:45.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0081352180000001 2021-03-15T22:28:45.763 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.764 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.764 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.765 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2223 7.f 7:ffef77ba:::smithi07694878-13:head [sparse-read 0~2702901] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:28:45.765 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:28:45.765 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 346.67953299300001, 2021-03-15T22:28:45.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.009479572, 2021-03-15T22:28:45.765 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.766 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:45.766 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.766 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.766 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.766 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2223 2021-03-15T22:28:45.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.767 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:28:45.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:28:45.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7300000000000005e-07 2021-03-15T22:28:45.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:28:45.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.02e-06 2021-03-15T22:28:45.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051030+0000", 2021-03-15T22:28:45.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9280000000000001e-06 2021-03-15T22:28:45.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051035+0000", 2021-03-15T22:28:45.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.129e-06 2021-03-15T22:28:45.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051039+0000", 2021-03-15T22:28:45.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018268000000000001 2021-03-15T22:28:45.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051221+0000", 2021-03-15T22:28:45.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4270000000000002e-06 2021-03-15T22:28:45.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:28:45.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051229+0000", 2021-03-15T22:28:45.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.007961946 2021-03-15T22:28:45.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.059191+0000", 2021-03-15T22:28:45.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010123370000000001 2021-03-15T22:28:45.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.060203+0000", 2021-03-15T22:28:45.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030523200000000001 2021-03-15T22:28:45.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.060508+0000", 2021-03-15T22:28:45.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.009479572 2021-03-15T22:28:45.779 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.779 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.779 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.780 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3021 7.6 7:6de8b070:::smithi07694878-15:head [copy-from ver 640] snapc e3=[e1,df,de,dd,db] ondisk+write+known_if_redirected e330)", 2021-03-15T22:28:45.780 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:28:45.780 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 279.647943156, 2021-03-15T22:28:45.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.124784161, 2021-03-15T22:28:45.781 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.781 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:45.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.782 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3021 2021-03-15T22:28:45.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.782 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.783 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:28:45.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:28:45.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7749999999999999e-06 2021-03-15T22:28:45.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082622+0000", 2021-03-15T22:28:45.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.072e-06 2021-03-15T22:28:45.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082625+0000", 2021-03-15T22:28:45.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2641e-05 2021-03-15T22:28:45.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082638+0000", 2021-03-15T22:28:45.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9170000000000006e-06 2021-03-15T22:28:45.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082646+0000", 2021-03-15T22:28:45.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1200999999999997e-05 2021-03-15T22:28:45.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082727+0000", 2021-03-15T22:28:45.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7440000000000007e-06 2021-03-15T22:28:45.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:28:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082735+0000", 2021-03-15T22:28:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022422739999999998 2021-03-15T22:28:45.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.084977+0000", 2021-03-15T22:28:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010620000000000001 2021-03-15T22:28:45.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085083+0000", 2021-03-15T22:28:45.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10953049500000001 2021-03-15T22:28:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,7", 2021-03-15T22:28:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.194614+0000", 2021-03-15T22:28:45.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043572510000000004 2021-03-15T22:28:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198971+0000", 2021-03-15T22:28:45.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002724242 2021-03-15T22:28:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.201695+0000", 2021-03-15T22:28:45.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054890260000000001 2021-03-15T22:28:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207184+0000", 2021-03-15T22:28:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4066000000000006e-05 2021-03-15T22:28:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:45.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207248+0000", 2021-03-15T22:28:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015425700000000001 2021-03-15T22:28:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207403+0000", 2021-03-15T22:28:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.124784161 2021-03-15T22:28:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.799 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3022 7.6 7:6de8b070:::smithi07694878-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e330)", 2021-03-15T22:28:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:28:45.800 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 279.647902789, 2021-03-15T22:28:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12494472, 2021-03-15T22:28:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.801 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3022 2021-03-15T22:28:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.802 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:28:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:28:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2669999999999999e-06 2021-03-15T22:28:45.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082660+0000", 2021-03-15T22:28:45.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7409999999999999e-06 2021-03-15T22:28:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082662+0000", 2021-03-15T22:28:45.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2030000000000003e-06 2021-03-15T22:28:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082669+0000", 2021-03-15T22:28:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1479999999999999e-06 2021-03-15T22:28:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082673+0000", 2021-03-15T22:28:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037115800000000001 2021-03-15T22:28:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.083044+0000", 2021-03-15T22:28:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1779999999999996e-06 2021-03-15T22:28:45.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:28:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.083049+0000", 2021-03-15T22:28:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021797460000000002 2021-03-15T22:28:45.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085228+0000", 2021-03-15T22:28:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5878999999999999e-05 2021-03-15T22:28:45.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085274+0000", 2021-03-15T22:28:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.112992462 2021-03-15T22:28:45.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198267+0000", 2021-03-15T22:28:45.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6129000000000002e-05 2021-03-15T22:28:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198343+0000", 2021-03-15T22:28:45.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090944800000000003 2021-03-15T22:28:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207437+0000", 2021-03-15T22:28:45.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1060000000000001e-05 2021-03-15T22:28:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207508+0000", 2021-03-15T22:28:45.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5268999999999999e-05 2021-03-15T22:28:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207604+0000", 2021-03-15T22:28:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12494472 2021-03-15T22:28:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3170 7.6 7:6283974b:::smithi07694878-30:head [copy-from ver 282] snapc f4=[f4,f3,f2,e9,e7,e1,de] ondisk+write+known_if_redirected e344)", 2021-03-15T22:28:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:19.130807+0000", 2021-03-15T22:28:45.820 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 265.59975448400002, 2021-03-15T22:28:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026060351999999998, 2021-03-15T22:28:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.821 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3170 2021-03-15T22:28:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130807+0000", 2021-03-15T22:28:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130807+0000", 2021-03-15T22:28:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4670000000000001e-06 2021-03-15T22:28:45.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130809+0000", 2021-03-15T22:28:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9299999999999996e-07 2021-03-15T22:28:45.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130809+0000", 2021-03-15T22:28:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5999999999999997e-06 2021-03-15T22:28:45.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130815+0000", 2021-03-15T22:28:45.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5019999999999999e-06 2021-03-15T22:28:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130818+0000", 2021-03-15T22:28:45.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1529999999999997e-05 2021-03-15T22:28:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130859+0000", 2021-03-15T22:28:45.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0765000000000003e-05 2021-03-15T22:28:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130940+0000", 2021-03-15T22:28:45.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011008983999999999 2021-03-15T22:28:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:28:45.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.141949+0000", 2021-03-15T22:28:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075953360000000003 2021-03-15T22:28:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.149544+0000", 2021-03-15T22:28:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0072008669999999997 2021-03-15T22:28:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:28:45.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156745+0000", 2021-03-15T22:28:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4639999999999995e-06 2021-03-15T22:28:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:28:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156755+0000", 2021-03-15T22:28:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9437999999999997e-05 2021-03-15T22:28:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156794+0000", 2021-03-15T22:28:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3505999999999997e-05 2021-03-15T22:28:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156867+0000", 2021-03-15T22:28:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026060351999999998 2021-03-15T22:28:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3171 7.6 7:6283974b:::smithi07694878-30:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e344)", 2021-03-15T22:28:45.838 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:19.130826+0000", 2021-03-15T22:28:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 265.59973558199999, 2021-03-15T22:28:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026195561999999999, 2021-03-15T22:28:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:28:45.839 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:28:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:28:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:28:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3171 2021-03-15T22:28:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.840 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130826+0000", 2021-03-15T22:28:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130826+0000", 2021-03-15T22:28:45.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6599999999999994e-07 2021-03-15T22:28:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130827+0000", 2021-03-15T22:28:45.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4200000000000004e-07 2021-03-15T22:28:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130828+0000", 2021-03-15T22:28:45.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1210000000000003e-06 2021-03-15T22:28:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130834+0000", 2021-03-15T22:28:45.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.676e-06 2021-03-15T22:28:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.130838+0000", 2021-03-15T22:28:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023820699999999999 2021-03-15T22:28:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.131076+0000", 2021-03-15T22:28:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4833999999999998e-05 2021-03-15T22:28:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:28:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.131121+0000", 2021-03-15T22:28:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017834289999999999 2021-03-15T22:28:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.148955+0000", 2021-03-15T22:28:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0327000000000004e-05 2021-03-15T22:28:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:28:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.149045+0000", 2021-03-15T22:28:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0078568969999999998 2021-03-15T22:28:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156902+0000", 2021-03-15T22:28:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6823999999999997e-05 2021-03-15T22:28:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.156959+0000", 2021-03-15T22:28:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2477999999999994e-05 2021-03-15T22:28:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:19.157022+0000", 2021-03-15T22:28:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026195561999999999 2021-03-15T22:28:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.854 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:679 7.5 e560/559 7:ac84e656:::smithi07694878-31:1a1 v 560'396, mlcod=560'392)", 2021-03-15T22:28:45.855 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:28:20.276500+0000", 2021-03-15T22:28:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 24.454061762999999, 2021-03-15T22:28:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051465865, 2021-03-15T22:28:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:28:45.856 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:28:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:28:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:28:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276500+0000", 2021-03-15T22:28:45.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:28:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:28:45.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276500+0000", 2021-03-15T22:28:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5639999999999996e-06 2021-03-15T22:28:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:28:45.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276505+0000", 2021-03-15T22:28:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7400000000000002e-06 2021-03-15T22:28:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:28:45.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276509+0000", 2021-03-15T22:28:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4290999999999999e-05 2021-03-15T22:28:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:28:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276523+0000", 2021-03-15T22:28:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2339999999999997e-06 2021-03-15T22:28:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:28:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276531+0000", 2021-03-15T22:28:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0612999999999999e-05 2021-03-15T22:28:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:28:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276571+0000", 2021-03-15T22:28:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4697999999999999e-05 2021-03-15T22:28:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:28:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276606+0000", 2021-03-15T22:28:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051301986000000001 2021-03-15T22:28:45.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:28:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.327908+0000", 2021-03-15T22:28:45.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7738999999999997e-05 2021-03-15T22:28:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:28:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.327966+0000", 2021-03-15T22:28:45.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051465865 2021-03-15T22:28:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:45.868 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:45.869 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:28:45.869 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:45.870 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: audit 2021-03-15T22:28:45.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021-03-15T22:28:45.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021 2021-03-15T22:28:45.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: - 2021-03-15T22:28:45.871 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 03 2021-03-15T22:28:45.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: - 2021-03-15T22:28:45.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 15 2021-03-15T22:28:45.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: T 2021-03-15T22:28:45.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 22 2021-03-15T22:28:45.872 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: : 2021-03-15T22:28:45.873 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 28 2021-03-15T22:28:45.873 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: : 2021-03-15T22:28:45.873 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 44 2021-03-15T22:28:45.873 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: . 2021-03-15T22:28:45.874 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 420799 2021-03-15T22:28:45.874 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: +0000 2021-03-15T22:28:45.874 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021-03-15T22:28:45.874 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:45.874 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: ( 2021-03-15T22:28:45.875 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: mon 2021-03-15T22:28:45.875 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: . 2021-03-15T22:28:45.875 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 0 2021-03-15T22:28:45.875 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: ) 2021-03-15T22:28:45.875 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 1611 2021-03-15T22:28:45.876 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: : 2021-03-15T22:28:45.876 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: audit 2021-03-15T22:28:45.876 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021-03-15T22:28:45.876 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:45.876 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021-03-15T22:28:45.877 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2434567820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:45.877 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021-03-15T22:28:45.877 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: cluster 2021-03-15T22:28:45.877 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021-03-15T22:28:45.878 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021 2021-03-15T22:28:45.878 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: - 2021-03-15T22:28:45.878 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 03 2021-03-15T22:28:45.878 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: - 2021-03-15T22:28:45.878 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 15 2021-03-15T22:28:45.879 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: T 2021-03-15T22:28:45.879 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 22 2021-03-15T22:28:45.879 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: : 2021-03-15T22:28:45.879 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 28 2021-03-15T22:28:45.880 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: : 2021-03-15T22:28:45.880 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 44 2021-03-15T22:28:45.880 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: . 2021-03-15T22:28:45.880 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 484263 2021-03-15T22:28:45.880 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: +0000 2021-03-15T22:28:45.881 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021-03-15T22:28:45.881 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: osd.6 2021-03-15T22:28:45.881 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: ( 2021-03-15T22:28:45.881 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: osd 2021-03-15T22:28:45.881 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: . 2021-03-15T22:28:45.882 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 6 2021-03-15T22:28:45.882 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: ) 2021-03-15T22:28:45.882 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 90 2021-03-15T22:28:45.882 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: : 2021-03-15T22:28:45.882 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: cluster 2021-03-15T22:28:45.883 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021-03-15T22:28:45.883 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:45.883 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021-03-15T22:28:45.883 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 3.c scrub ok 2021-03-15T22:28:45.884 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021-03-15T22:28:45.884 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: cluster 2021-03-15T22:28:45.884 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021-03-15T22:28:45.884 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 2021 2021-03-15T22:28:45.884 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: - 2021-03-15T22:28:45.885 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 03 2021-03-15T22:28:45.885 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: - 2021-03-15T22:28:45.885 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 15 2021-03-15T22:28:45.885 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: T 2021-03-15T22:28:45.885 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 22 2021-03-15T22:28:45.886 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: : 2021-03-15T22:28:45.886 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 28 2021-03-15T22:28:45.886 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: : 2021-03-15T22:28:45.886 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: 44.765841+0000 osd.0 (osd.0) 83 : cluster [DBG] 4.16 deep-scrub ok 2021-03-15T22:28:45.887 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:45 smithi125 conmon[56739]: cluster 2021-03-15T22:28:45.172517+0000 mon.a (mon.0) 1612 : cluster [DBG] osdmap e576: 8 total, 6 up, 7 in 2021-03-15T22:28:45.888 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:28:45.889 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:28:45.891 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:45.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:45.893 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:45.994 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:46.018 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:46.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:46.119 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:46.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:46.143 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:46.143 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:46.188 INFO:tasks.rados.rados.0.smithi076.stdout:2083: done (1 left) 2021-03-15T22:28:46.189 INFO:tasks.rados.rados.0.smithi076.stdout:2085: done (0 left) 2021-03-15T22:28:46.189 INFO:tasks.rados.rados.0.smithi076.stdout:2086: snap_remove snap 222 2021-03-15T22:28:46.245 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:28:46.246 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:46.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:46.272 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:46.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:46.275 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:46.374 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:46.396 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:46.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:46.397 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:46.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[56514]: cluster 2021-03-15T22:28:45.043824+0000 osd.4 (osd.4) 120 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:28:46.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[56514]: cluster 2021-03-15T22:28:45.113098+0000 mgr.y (mgr.14144) 766 : cluster [DBG] pgmap v1007: 138 pgs: 1 active+undersized+degraded+remapped, 43 active+clean+remapped, 1 undersized+degraded+peered, 1 active+recovering+undersized+degraded+remapped, 14 active+undersized+remapped, 11 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 22 active+undersized, 39 active+clean; 568 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 232 KiB/s wr, 3 op/s; 127/1749 objects degraded (7.261%); 405/1749 objects misplaced (23.156%); 14 MiB/s, 47 objects/s recovering 2021-03-15T22:28:46.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[56514]: cluster 2021-03-15T22:28:45.222366+0000 osd.7 (osd.7) 112 : cluster [DBG] 7.5 scrub ok 2021-03-15T22:28:46.492 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[56514]: cluster 2021-03-15T22:28:45.454454+0000 osd.6 (osd.6) 91 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:28:46.493 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[56514]: cluster 2021-03-15T22:28:45.780430+0000 osd.0 (osd.0) 84 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:28:46.493 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[56514]: cluster 2021-03-15T22:28:46.185728+0000 mon.a (mon.0) 1613 : cluster [DBG] osdmap e577: 8 total, 6 up, 7 in 2021-03-15T22:28:46.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: cluster 2021-03-15T22:28:46.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021 2021-03-15T22:28:46.494 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: - 2021-03-15T22:28:46.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 03 2021-03-15T22:28:46.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: - 2021-03-15T22:28:46.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 15 2021-03-15T22:28:46.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: T 2021-03-15T22:28:46.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 22 2021-03-15T22:28:46.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: : 2021-03-15T22:28:46.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 28 2021-03-15T22:28:46.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: : 2021-03-15T22:28:46.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 45 2021-03-15T22:28:46.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: . 2021-03-15T22:28:46.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 043824 2021-03-15T22:28:46.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: +0000 2021-03-15T22:28:46.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: osd.4 2021-03-15T22:28:46.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: ( 2021-03-15T22:28:46.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: osd 2021-03-15T22:28:46.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: . 2021-03-15T22:28:46.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 4 2021-03-15T22:28:46.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: ) 2021-03-15T22:28:46.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 120 2021-03-15T22:28:46.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: : 2021-03-15T22:28:46.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: cluster 2021-03-15T22:28:46.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: [DBG] 2021-03-15T22:28:46.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 7.3 scrub ok 2021-03-15T22:28:46.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: cluster 2021-03-15T22:28:46.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021 2021-03-15T22:28:46.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: - 2021-03-15T22:28:46.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 03 2021-03-15T22:28:46.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: - 2021-03-15T22:28:46.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 15 2021-03-15T22:28:46.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: T 2021-03-15T22:28:46.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 22 2021-03-15T22:28:46.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: : 2021-03-15T22:28:46.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 28 2021-03-15T22:28:46.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: : 2021-03-15T22:28:46.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 45 2021-03-15T22:28:46.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: . 2021-03-15T22:28:46.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 113098 2021-03-15T22:28:46.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: +0000 2021-03-15T22:28:46.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: mgr.y 2021-03-15T22:28:46.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: ( 2021-03-15T22:28:46.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: mgr 2021-03-15T22:28:46.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: . 2021-03-15T22:28:46.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 14144 2021-03-15T22:28:46.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: ) 2021-03-15T22:28:46.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 766 2021-03-15T22:28:46.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: : 2021-03-15T22:28:46.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: cluster 2021-03-15T22:28:46.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: [DBG] 2021-03-15T22:28:46.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: pgmap v1007: 138 pgs: 1 active+undersized+degraded+remapped, 43 active+clean+remapped, 1 undersized+degraded+peered, 1 active+recovering+undersized+degraded+remapped, 14 active+undersized+remapped, 11 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 22 active+undersized, 39 active+clean; 568 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 232 KiB/s wr, 3 op/s; 127/1749 objects degraded (7.261%); 405/1749 objects misplaced (23.156%); 14 MiB/s, 47 objects/s recovering 2021-03-15T22:28:46.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: cluster 2021-03-15T22:28:46.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021 2021-03-15T22:28:46.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: - 2021-03-15T22:28:46.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 03 2021-03-15T22:28:46.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: - 2021-03-15T22:28:46.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 15 2021-03-15T22:28:46.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: T 2021-03-15T22:28:46.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 22 2021-03-15T22:28:46.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: : 2021-03-15T22:28:46.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 28 2021-03-15T22:28:46.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: : 2021-03-15T22:28:46.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 45 2021-03-15T22:28:46.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: . 2021-03-15T22:28:46.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 222366 2021-03-15T22:28:46.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: +0000 2021-03-15T22:28:46.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: osd.7 2021-03-15T22:28:46.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: ( 2021-03-15T22:28:46.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: osd 2021-03-15T22:28:46.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: . 2021-03-15T22:28:46.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 7 2021-03-15T22:28:46.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: ) 2021-03-15T22:28:46.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 112 2021-03-15T22:28:46.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: : cluster [DBG] 7.5 scrub ok 2021-03-15T22:28:46.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 2021-03-15T22:28:46.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: cluster 2021-03-15T22:28:45.454454+0000 osd.6 (osd.6) 91 : cluster [DBG] 2021-03-15T22:28:46.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 3.2 scrub ok 2021-03-15T22:28:46.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: cluster 2021-03-15T22:28 2021-03-15T22:28:46.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: :45.780430+0000 osd.0 (osd.0) 2021-03-15T22:28:46.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 84 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:28:46.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:28:46.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: 03-15T22:28:46.185728+0000 2021-03-15T22:28:46.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: mon.a (mon.0) 1613 : cluster [DBG] 2021-03-15T22:28:46.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:46 smithi076 conmon[62156]: osdmap e577: 8 total, 6 up, 7 in 2021-03-15T22:28:46.517 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:46.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: cluster 2021-03-15T22:28:45.043824+0000 2021-03-15T22:28:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: osd.4 (osd.4) 120 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:28:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: cluster 2021-03-15T22:28:45.113098+0000 mgr.y (mgr.14144) 766 : cluster 2021-03-15T22:28:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: [DBG] pgmap v1007: 138 pgs: 1 active+undersized+degraded+remapped, 43 active+clean+remapped, 1 undersized+degraded+peered, 1 active+recovering+undersized+degraded+remapped, 14 active+undersized+remapped, 11 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 22 active+undersized, 39 active+clean; 568 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.9 MiB/s rd, 232 KiB/s wr, 3 op/s; 127/1749 objects degraded (7.261%); 405/1749 objects misplaced (23.156%); 14 MiB/s, 47 objects/s recovering 2021-03-15T22:28:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: cluster 2021-03-15T22:28:46.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: 2021-03-15T22:28:45.222366+0000 osd.7 (osd.7) 112 : 2021-03-15T22:28:46.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: cluster [DBG] 7.5 scrub ok 2021-03-15T22:28:46.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:28:46.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: T22:28:45.454454+0000 osd.6 ( 2021-03-15T22:28:46.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: osd.6) 91 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:28:46.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:28:46.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: -03-15T22:28:45. 2021-03-15T22:28:46.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: 780430+0000 osd.0 (osd.0) 84 : cluster 2021-03-15T22:28:46.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: [DBG] 4.6 scrub ok 2021-03-15T22:28:46.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:28:46.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: 15T22:28:46.185728+0000 mon.a 2021-03-15T22:28:46.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: (mon.0) 1613 : cluster [DBG] osdmap e577: 8 total, 6 up, 7 in 2021-03-15T22:28:46.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:46 smithi125 conmon[56739]: 2021-03-15T22:28:46.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:46.542 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:46.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:46.643 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:46.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:46.667 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:46.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:46.751 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:46.770 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:46.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:46.796 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:46.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:46.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:46.922 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:46.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:46.924 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:47.025 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:47.047 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:47.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:47.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:47.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:47.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:47.173 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:47.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:47.195 INFO:tasks.rados.rados.0.smithi076.stdout:2086: done (0 left) 2021-03-15T22:28:47.195 INFO:tasks.rados.rados.0.smithi076.stdout:2087: snap_create 2021-03-15T22:28:47.275 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:47.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:47.298 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:47.298 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:47.400 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:47.422 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:47.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:47.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:47.483 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:28:47.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:47 smithi076 conmon[62156]: cluster 2021-03-15T22:28:45.649395 2021-03-15T22:28:47.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:47 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 110 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:28:47.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:47 smithi076 conmon[62156]: cluster 2021-03-15T22:28:47.189929+0000 mon.a (mon.0) 1614 : cluster [DBG] osdmap e578: 8 total, 6 up, 7 in 2021-03-15T22:28:47.508 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:47 smithi076 conmon[56514]: cluster 2021-03-15T22:28:45.649395 2021-03-15T22:28:47.508 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:47 smithi076 conmon[56514]: +0000 osd.1 (osd.1) 110 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:28:47.508 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:47 smithi076 conmon[56514]: cluster 2021-03-15T22:28:47.189929+0000 mon.a (mon.0) 1614 : cluster [DBG] osdmap e578: 8 total, 6 up, 7 in 2021-03-15T22:28:47.524 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:47.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: cluster 2021-03-15T22:28:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021-03-15T22:28:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021 2021-03-15T22:28:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: - 2021-03-15T22:28:47.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 03 2021-03-15T22:28:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: - 2021-03-15T22:28:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 15 2021-03-15T22:28:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: T 2021-03-15T22:28:47.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 22 2021-03-15T22:28:47.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: : 2021-03-15T22:28:47.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 28 2021-03-15T22:28:47.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: : 2021-03-15T22:28:47.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 45 2021-03-15T22:28:47.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: . 2021-03-15T22:28:47.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 649395 2021-03-15T22:28:47.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: +0000 2021-03-15T22:28:47.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021-03-15T22:28:47.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: osd.1 2021-03-15T22:28:47.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: ( 2021-03-15T22:28:47.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: osd 2021-03-15T22:28:47.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: . 2021-03-15T22:28:47.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 1 2021-03-15T22:28:47.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: ) 2021-03-15T22:28:47.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 110 2021-03-15T22:28:47.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: : 2021-03-15T22:28:47.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: cluster 2021-03-15T22:28:47.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021-03-15T22:28:47.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:47.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021-03-15T22:28:47.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 4.4 scrub ok 2021-03-15T22:28:47.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021-03-15T22:28:47.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: cluster 2021-03-15T22:28:47.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021-03-15T22:28:47.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021 2021-03-15T22:28:47.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: - 2021-03-15T22:28:47.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 03 2021-03-15T22:28:47.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: - 2021-03-15T22:28:47.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 15 2021-03-15T22:28:47.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: T 2021-03-15T22:28:47.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 22 2021-03-15T22:28:47.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: : 2021-03-15T22:28:47.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 28 2021-03-15T22:28:47.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: : 2021-03-15T22:28:47.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 47 2021-03-15T22:28:47.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: . 2021-03-15T22:28:47.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 189929 2021-03-15T22:28:47.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: +0000 2021-03-15T22:28:47.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021-03-15T22:28:47.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:47.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: ( 2021-03-15T22:28:47.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: mon 2021-03-15T22:28:47.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: . 2021-03-15T22:28:47.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 0 2021-03-15T22:28:47.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: ) 2021-03-15T22:28:47.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 1614 2021-03-15T22:28:47.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: : 2021-03-15T22:28:47.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: cluster 2021-03-15T22:28:47.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021-03-15T22:28:47.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:47.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021-03-15T22:28:47.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: osdmap e578: 8 total, 6 up, 7 in 2021-03-15T22:28:47.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:47 smithi125 conmon[56739]: 2021-03-15T22:28:47.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:47.551 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:47.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:47.653 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:47.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:47.676 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:47.677 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:47.778 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:47.799 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:47.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:47.801 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:47.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:28:47.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:47.958 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:47.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:47.959 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:48.060 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:48.082 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:48.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:48.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:48.185 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:48.194 INFO:tasks.rados.rados.0.smithi076.stdout:2087: done (0 left) 2021-03-15T22:28:48.194 INFO:tasks.rados.rados.0.smithi076.stdout:2088: write oid 7 current snap is 237 2021-03-15T22:28:48.194 INFO:tasks.rados.rados.0.smithi076.stdout:2088: seq_num 726 ranges {601882=484309,1600633=524904} 2021-03-15T22:28:48.202 INFO:tasks.rados.rados.0.smithi076.stdout:2088: writing smithi07694878-7 from 601882 to 1086191 tid 1 2021-03-15T22:28:48.212 INFO:tasks.rados.rados.0.smithi076.stdout:2088: writing smithi07694878-7 from 1600633 to 2125537 tid 2 2021-03-15T22:28:48.212 INFO:tasks.rados.rados.0.smithi076.stdout:2088: finishing write tid 1 to smithi07694878-7 2021-03-15T22:28:48.213 INFO:tasks.rados.rados.0.smithi076.stdout:2089: read oid 26 snap -1 2021-03-15T22:28:48.213 INFO:tasks.rados.rados.0.smithi076.stdout:2089: expect (ObjNum 715 snap 231 seq_num 715) 2021-03-15T22:28:48.213 INFO:tasks.rados.rados.0.smithi076.stdout:2090: snap_create 2021-03-15T22:28:48.215 INFO:tasks.rados.rados.0.smithi076.stdout:2088: finishing write tid 2 to smithi07694878-7 2021-03-15T22:28:48.215 INFO:tasks.rados.rados.0.smithi076.stdout:2088: finishing write tid 3 to smithi07694878-7 2021-03-15T22:28:48.216 INFO:tasks.rados.rados.0.smithi076.stdout:2088: finishing write tid 4 to smithi07694878-7 2021-03-15T22:28:48.216 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 1033 (ObjNum 726 snap 237 seq_num 726) dirty exists 2021-03-15T22:28:48.216 INFO:tasks.rados.rados.0.smithi076.stdout:2088: left oid 7 (ObjNum 726 snap 237 seq_num 726) 2021-03-15T22:28:48.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:48.219 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:48.220 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:48.220 INFO:tasks.rados.rados.0.smithi076.stdout:2089: expect (ObjNum 715 snap 231 seq_num 715) 2021-03-15T22:28:48.321 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:48.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:48.345 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:48.345 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:48.440 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56514]: cluster 2021-03-15T22:28:47.113752+0000 mgr.y (mgr.14144) 767 : cluster [DBG] pgmap v1010: 138 pgs: 44 active+clean+remapped, 1 undersized+degraded+peered, 1 active+recovering+undersized+degraded+remapped, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 572 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 1.3 MiB/s wr, 10 op/s; 127/1758 objects degraded (7.224%); 402/1758 objects misplaced (22.867%); 689 KiB/s, 5 objects/s recovering 2021-03-15T22:28:48.440 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56514]: cluster 2021-03-15T22:28:47.140107+0000 osd.7 (osd.7) 113 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:28:48.441 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56514]: cluster 2021-03-15T22:28:48.188764+0000 mon.a (mon.0) 1615 : cluster [DBG] osdmap e579: 8 total, 6 up, 7 in 2021-03-15T22:28:48.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: cluster 2021-03-15T22:28:48.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 2021-03-15T22:28:48.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 2021 2021-03-15T22:28:48.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: - 2021-03-15T22:28:48.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 03 2021-03-15T22:28:48.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: - 2021-03-15T22:28:48.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 15 2021-03-15T22:28:48.443 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: T 2021-03-15T22:28:48.443 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 22 2021-03-15T22:28:48.443 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: : 2021-03-15T22:28:48.443 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 28 2021-03-15T22:28:48.444 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: : 2021-03-15T22:28:48.444 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 47 2021-03-15T22:28:48.444 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: . 2021-03-15T22:28:48.444 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 113752 2021-03-15T22:28:48.444 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144 2021-03-15T22:28:48.445 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: ) 767 : cluster [DBG] pgmap v1010: 138 pgs: 44 active+clean+remapped, 1 undersized+degraded+peered, 1 active+recovering+undersized+degraded+remapped, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 572 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 1.3 MiB/s wr, 10 op/s; 127/1758 objects degraded (7.224%); 402/1758 objects misplaced (22.867%); 689 KiB/s, 5 objects/s recovering 2021-03-15T22:28:48.445 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:28:48.445 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 15T22:28:47.140107+0000 osd.7 (osd.7) 2021-03-15T22:28:48.445 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 113 : cluster [DBG] 2021-03-15T22:28:48.445 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 5.14 scrub ok 2021-03-15T22:28:48.446 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:28:48.446 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: 03-15T22:28:48.188764+0000 mon.a (mon 2021-03-15T22:28:48.446 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: .0) 1615 : cluster 2021-03-15T22:28:48.446 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[62156]: [DBG] osdmap e579: 8 total, 6 up, 7 in 2021-03-15T22:28:48.447 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:48.468 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:48.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:48.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:48.482 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:48.534 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:28:48.535 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:28:48.535 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 579, 2021-03-15T22:28:48.535 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:28:48.536 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2021-03-15T22:13:20.370795+0000", 2021-03-15T22:28:48.536 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2021-03-15T22:28:48.184760+0000", 2021-03-15T22:28:48.536 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2021-03-15T22:27:35.183905+0000", 2021-03-15T22:28:48.536 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2021-03-15T22:28:38.336225+0000", 2021-03-15T22:28:48.537 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-03-15T22:28:48.537 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5799936, 2021-03-15T22:28:48.537 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2021-03-15T22:28:48.537 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2021-03-15T22:28:48.537 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2021-03-15T22:28:48.538 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2021-03-15T22:28:48.538 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2021-03-15T22:28:48.538 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:48.538 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 18, 2021-03-15T22:28:48.539 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2021-03-15T22:28:48.539 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-03-15T22:28:48.539 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2021-03-15T22:28:48.539 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2021-03-15T22:28:48.540 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 7, 2021-03-15T22:28:48.540 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2021-03-15T22:28:48.540 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2021-03-15T22:28:48.540 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2021-03-15T22:28:48.540 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2021-03-15T22:28:48.541 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2021-03-15T22:28:48.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.541 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2021-03-15T22:28:48.541 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "device_health_metrics", 2021-03-15T22:28:48.541 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:16:34.817191+0000", 2021-03-15T22:28:48.542 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:48.542 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:48.542 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:48.542 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:48.543 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:48.543 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:48.543 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:48.543 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:48.544 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:48.544 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:48.544 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:48.544 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:48.544 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2021-03-15T22:28:48.545 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2021-03-15T22:28:48.545 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2021-03-15T22:28:48.545 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2021-03-15T22:28:48.545 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2021-03-15T22:28:48.546 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:48.546 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:48.546 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:48.546 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:48.546 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:48.547 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:48.547 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:48.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.547 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "23", 2021-03-15T22:28:48.548 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:48.548 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:48.548 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:48.548 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:48.548 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:48.549 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:48.549 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:48.549 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:48.549 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:48.550 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:48.550 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:48.550 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:48.550 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:48.551 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:48.551 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:48.551 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:48.551 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:48.551 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:48.552 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:48.552 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:48.552 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:48.552 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:48.552 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:48.553 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:48.553 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:48.553 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:48.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.554 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:48.554 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:48.554 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:48.554 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:48.554 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:48.555 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:48.555 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:48.555 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:48.555 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:48.556 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:48.556 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:48.556 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:28:48.557 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2021-03-15T22:28:48.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.557 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:48.557 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_devicehealth": {} 2021-03-15T22:28:48.557 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.558 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2021-03-15T22:28:48.558 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "iscsi", 2021-03-15T22:28:48.559 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:49.906400+0000", 2021-03-15T22:28:48.559 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:48.559 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:48.559 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:48.559 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:48.560 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:48.560 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:48.560 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:48.560 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:48.561 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:48.561 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:48.561 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:48.561 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:48.561 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 3, 2021-03-15T22:28:48.562 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 3, 2021-03-15T22:28:48.562 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 3, 2021-03-15T22:28:48.562 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 3, 2021-03-15T22:28:48.562 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 3, 2021-03-15T22:28:48.563 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:48.563 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:48.563 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:48.563 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:48.563 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:48.564 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:48.564 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:48.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.564 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "56", 2021-03-15T22:28:48.565 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:48.565 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:48.565 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:48.565 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:48.565 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:48.566 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:48.566 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:48.566 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:48.566 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:48.567 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:48.567 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:48.567 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:48.567 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:48.567 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:48.568 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:48.568 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:48.568 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:48.568 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:48.569 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:48.569 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:48.569 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:48.569 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:48.569 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:48.570 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:48.570 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:48.570 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:48.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.571 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:48.571 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:48.571 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:48.571 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:48.571 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:48.572 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:48.572 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:48.572 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:48.572 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:48.573 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:48.573 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:48.573 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:28:48.573 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:48.573 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2021-03-15T22:28:48.574 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.574 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2021-03-15T22:28:48.575 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".rgw.root", 2021-03-15T22:28:48.575 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:52.257970+0000", 2021-03-15T22:28:48.575 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:48.575 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:48.576 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:48.576 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:48.576 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:48.576 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:48.576 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:48.577 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:48.577 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:48.577 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:48.577 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:48.577 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:48.578 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:28:48.578 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:28:48.578 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:28:48.578 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:28:48.579 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:28:48.579 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:48.579 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:48.579 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:48.579 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:48.580 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:48.580 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:48.580 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:48.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.581 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "58", 2021-03-15T22:28:48.581 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:48.581 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:48.581 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:48.581 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:48.582 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:48.582 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:48.582 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:48.582 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:48.582 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:48.583 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:48.583 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:48.583 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:48.583 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:48.584 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:48.584 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:48.584 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:48.584 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:48.584 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:48.585 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:48.585 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:48.585 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:48.585 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:48.586 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:48.586 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:48.586 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:48.586 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:48.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.587 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:48.587 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:48.587 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:48.587 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:48.588 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:48.588 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:48.588 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:48.588 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:48.588 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:48.589 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:48.589 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:48.589 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:28:48.589 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:48.590 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:28:48.590 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.591 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2021-03-15T22:28:48.591 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.log", 2021-03-15T22:28:48.591 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:54.562498+0000", 2021-03-15T22:28:48.591 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:48.591 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:48.592 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:48.592 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:48.592 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:48.592 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:48.592 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:48.593 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:48.593 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:48.593 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:48.593 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:48.594 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:48.594 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:28:48.594 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:28:48.594 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:28:48.594 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:28:48.595 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:28:48.595 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:48.595 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:48.595 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:48.596 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:48.596 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:48.596 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:48.596 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:48.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.597 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "60", 2021-03-15T22:28:48.597 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:48.597 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:48.597 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:48.598 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:48.598 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:48.598 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:48.598 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:48.598 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:48.599 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:48.599 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:48.599 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:48.599 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:48.599 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:48.600 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:48.600 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:48.600 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:48.600 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:48.601 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:48.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:48.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:48.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:48.601 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:48.602 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:48.602 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:48.602 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:48.602 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:48.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.603 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:48.603 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:48.603 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:48.603 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:48.604 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:48.604 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:48.604 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:48.604 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:48.604 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:48.605 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:48.605 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:48.605 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:28:48.605 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:48.606 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:28:48.606 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.606 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 5, 2021-03-15T22:28:48.607 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.control", 2021-03-15T22:28:48.607 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:56.243806+0000", 2021-03-15T22:28:48.607 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:48.607 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:48.608 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:48.608 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:48.608 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:48.608 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:48.608 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:48.609 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:48.609 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:48.609 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:48.609 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:48.610 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:48.610 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 22, 2021-03-15T22:28:48.610 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 22, 2021-03-15T22:28:48.610 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 22, 2021-03-15T22:28:48.611 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 22, 2021-03-15T22:28:48.611 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 22, 2021-03-15T22:28:48.611 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:48.611 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "5.16", 2021-03-15T22:28:48.611 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 182, 2021-03-15T22:28:48.612 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 181, 2021-03-15T22:28:48.612 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 181, 2021-03-15T22:28:48.612 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:48.612 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:48.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.613 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "182", 2021-03-15T22:28:48.613 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:48.613 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "182", 2021-03-15T22:28:48.613 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "180", 2021-03-15T22:28:48.613 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:48.614 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:48.614 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:48.614 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:48.614 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:48.615 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:48.615 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:48.615 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:48.615 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:48.615 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:48.616 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:48.616 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:48.616 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:48.616 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:48.617 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:48.617 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:48.617 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:48.617 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:48.617 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:48.618 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:48.618 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:48.618 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:48.618 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:48.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.619 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:48.619 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:48.619 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:48.619 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:48.620 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:48.620 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:48.620 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:48.620 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:48.620 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:48.621 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:48.621 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:48.621 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:28:48.621 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:48.622 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:28:48.622 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.622 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 6, 2021-03-15T22:28:48.623 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.meta", 2021-03-15T22:28:48.623 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:58.730534+0000", 2021-03-15T22:28:48.623 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:28:48.623 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:28:48.623 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:48.624 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:48.624 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:48.624 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:48.624 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:48.625 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:48.625 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:48.625 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:48.625 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:48.625 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:48.626 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:28:48.626 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:28:48.626 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:28:48.626 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:28:48.627 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:28:48.627 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:48.627 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:48.627 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:48.628 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:48.628 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:48.628 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:48.628 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:48.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.629 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "109", 2021-03-15T22:28:48.629 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:48.629 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:48.629 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:48.630 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:48.630 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:48.630 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:28:48.630 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:28:48.631 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:48.631 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:48.631 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:48.631 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:48.631 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:48.632 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:48.632 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:48.632 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:48.632 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:48.633 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:48.633 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:48.633 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:48.633 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:48.633 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:48.634 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:48.634 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:48.634 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:48.634 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:48.635 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:48.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.635 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:48.635 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:48.635 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:48.636 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:48.636 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:48.636 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:48.636 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:48.637 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:48.637 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:48.637 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:48.637 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:48.637 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:28:48.638 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_bias": 4, 2021-03-15T22:28:48.638 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 8 2021-03-15T22:28:48.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.638 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:48.639 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:28:48.639 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.639 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:28:48.640 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2021-03-15T22:28:48.640 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:20:03.377294+0000", 2021-03-15T22:28:48.640 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2021-03-15T22:28:48.640 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-03-15T22:28:48.641 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:28:48.641 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:28:48.641 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:28:48.641 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:28:48.641 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:28:48.642 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:28:48.642 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:28:48.642 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:28:48.642 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:28:48.642 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:28:48.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2021-03-15T22:28:48.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2021-03-15T22:28:48.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2021-03-15T22:28:48.643 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2021-03-15T22:28:48.644 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2021-03-15T22:28:48.644 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:28:48.644 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:28:48.644 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:28:48.644 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:28:48.645 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:28:48.645 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:28:48.645 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:28:48.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.646 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "579", 2021-03-15T22:28:48.646 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:28:48.646 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:28:48.646 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:28:48.646 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:28:48.647 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:28:48.647 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 445, 2021-03-15T22:28:48.647 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 579, 2021-03-15T22:28:48.647 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:28:48.648 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:28:48.648 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:28:48.648 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:28:48.648 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:28:48.648 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:28:48.649 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:28:48.649 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:28:48.649 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:28:48.649 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:28:48.649 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:28:48.650 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:28:48.650 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:28:48.650 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:28:48.650 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:28:48.651 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:28:48.651 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:28:48.651 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:28:48.651 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:28:48.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.652 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:28:48.652 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:28:48.652 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:28:48.652 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:28:48.652 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:28:48.653 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:28:48.653 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:28:48.653 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:28:48.653 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:28:48.654 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:28:48.654 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:28:48.654 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:28:48.654 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:28:48.655 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2021-03-15T22:28:48.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.655 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.655 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:48.655 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.656 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:28:48.656 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd", 2021-03-15T22:28:48.656 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:48.657 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:48.657 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:48.657 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:48.657 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:48.657 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:48.658 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 8, 2021-03-15T22:28:48.658 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 572, 2021-03-15T22:28:48.658 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:48.658 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:48.658 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:48.659 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.659 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.659 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2021-03-15T22:28:48.660 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:48.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.660 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.660 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2021-03-15T22:28:48.661 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:48.661 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.661 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.662 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:48.662 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.662 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.662 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2021-03-15T22:28:48.663 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:48.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.663 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.664 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2021-03-15T22:28:48.664 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:48.664 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.664 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.665 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:48.665 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.665 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.665 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2021-03-15T22:28:48.666 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:48.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.666 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.666 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2021-03-15T22:28:48.667 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:48.667 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.667 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.668 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:48.668 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.668 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.668 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2021-03-15T22:28:48.669 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:48.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.669 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.669 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2021-03-15T22:28:48.670 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:28:48.670 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.670 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.671 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/188228176", 2021-03-15T22:28:48.671 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/188228176", 2021-03-15T22:28:48.671 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/188228176", 2021-03-15T22:28:48.671 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/188228176", 2021-03-15T22:28:48.671 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:48.672 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:48.672 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:48.672 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.673 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:28:48.673 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526", 2021-03-15T22:28:48.673 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:48.673 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:48.674 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:48.674 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.999237060546875, 2021-03-15T22:28:48.674 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:48.674 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:48.674 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 13, 2021-03-15T22:28:48.675 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 576, 2021-03-15T22:28:48.675 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:48.675 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:48.675 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:48.675 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.676 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.676 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2021-03-15T22:28:48.676 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:48.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.677 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.677 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2021-03-15T22:28:48.677 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:48.678 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.678 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.678 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:48.679 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.679 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.679 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2021-03-15T22:28:48.679 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:48.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.680 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.680 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2021-03-15T22:28:48.680 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:48.681 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.681 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.681 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:48.682 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.682 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.682 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2021-03-15T22:28:48.682 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:48.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.683 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.683 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2021-03-15T22:28:48.683 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:48.684 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.684 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.684 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:48.685 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.685 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.685 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2021-03-15T22:28:48.685 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:48.686 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.686 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.686 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2021-03-15T22:28:48.686 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:28:48.687 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.687 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.687 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/3924955423", 2021-03-15T22:28:48.688 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/3924955423", 2021-03-15T22:28:48.688 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/3924955423", 2021-03-15T22:28:48.688 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/3924955423", 2021-03-15T22:28:48.688 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:48.688 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:48.689 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:48.689 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.689 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:28:48.690 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05", 2021-03-15T22:28:48.690 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 0, 2021-03-15T22:28:48.690 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:48.690 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:48.691 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:48.691 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:48.691 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:48.691 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 18, 2021-03-15T22:28:48.691 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 421, 2021-03-15T22:28:48.692 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 490, 2021-03-15T22:28:48.692 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:48.692 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:48.692 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.693 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.693 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2021-03-15T22:28:48.693 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:48.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.694 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.694 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2021-03-15T22:28:48.694 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:48.694 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.695 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.695 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:48.695 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.696 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.696 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2021-03-15T22:28:48.696 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:48.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.697 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.697 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2021-03-15T22:28:48.697 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:48.697 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.698 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.698 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:48.698 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.699 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.699 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2021-03-15T22:28:48.699 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:48.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.700 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.700 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2021-03-15T22:28:48.700 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:48.700 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.701 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.701 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:48.701 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.702 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.702 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2021-03-15T22:28:48.702 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:48.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.703 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.703 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2021-03-15T22:28:48.703 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:28:48.703 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.704 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.704 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/1401263050", 2021-03-15T22:28:48.704 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/1401263050", 2021-03-15T22:28:48.704 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/1401263050", 2021-03-15T22:28:48.705 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/1401263050", 2021-03-15T22:28:48.705 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:48.705 INFO:teuthology.orchestra.run.smithi076.stdout: "exists" 2021-03-15T22:28:48.705 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.706 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:28:48.706 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56", 2021-03-15T22:28:48.706 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:48.707 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 0, 2021-03-15T22:28:48.707 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2021-03-15T22:28:48.707 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:48.707 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:48.708 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:48.708 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 26, 2021-03-15T22:28:48.708 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 571, 2021-03-15T22:28:48.708 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:48.708 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:48.709 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:48.709 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.709 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2021-03-15T22:28:48.710 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:48.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.710 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.711 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2021-03-15T22:28:48.711 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:48.711 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.711 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.712 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:48.712 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.712 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.712 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2021-03-15T22:28:48.713 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:48.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.713 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.714 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2021-03-15T22:28:48.714 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:48.714 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.714 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.715 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:48.715 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.715 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.715 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2021-03-15T22:28:48.716 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:48.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.716 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.717 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2021-03-15T22:28:48.717 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:48.717 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.717 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.718 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:48.718 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.718 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.718 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2021-03-15T22:28:48.719 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:48.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.719 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.720 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2021-03-15T22:28:48.720 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:28:48.720 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.720 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.721 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/3960968636", 2021-03-15T22:28:48.721 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/3960968636", 2021-03-15T22:28:48.721 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/3960968636", 2021-03-15T22:28:48.721 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/3960968636", 2021-03-15T22:28:48.721 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:48.722 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:48.722 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:48.722 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.723 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:28:48.723 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237", 2021-03-15T22:28:48.723 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:48.723 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:48.724 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:48.724 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:48.724 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:48.724 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:48.724 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 31, 2021-03-15T22:28:48.725 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 574, 2021-03-15T22:28:48.725 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:48.725 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:48.725 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:48.726 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.726 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.726 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6800", 2021-03-15T22:28:48.726 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:48.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.727 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.727 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6801", 2021-03-15T22:28:48.728 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:48.728 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.728 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.728 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:48.729 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.729 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.729 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6802", 2021-03-15T22:28:48.729 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:48.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.730 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.730 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6803", 2021-03-15T22:28:48.731 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:48.731 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.731 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.731 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:48.732 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.732 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.732 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6806", 2021-03-15T22:28:48.732 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:48.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.733 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.733 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6807", 2021-03-15T22:28:48.733 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:48.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.734 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.734 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:48.735 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.735 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6804", 2021-03-15T22:28:48.735 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:48.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.736 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.736 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6805", 2021-03-15T22:28:48.737 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:28:48.737 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.737 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.737 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6801/37386122", 2021-03-15T22:28:48.738 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6803/37386122", 2021-03-15T22:28:48.738 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6807/37386122", 2021-03-15T22:28:48.738 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6805/37386122", 2021-03-15T22:28:48.738 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:48.739 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:48.739 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:48.739 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.740 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:28:48.740 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2", 2021-03-15T22:28:48.740 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 0, 2021-03-15T22:28:48.740 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:48.740 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:48.741 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:48.741 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:48.741 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:48.741 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 38, 2021-03-15T22:28:48.741 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 490, 2021-03-15T22:28:48.742 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 540, 2021-03-15T22:28:48.742 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:48.742 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:48.742 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.743 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.743 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6808", 2021-03-15T22:28:48.743 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:48.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.744 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.744 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6809", 2021-03-15T22:28:48.744 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:48.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.745 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.745 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:48.745 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.746 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.746 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6810", 2021-03-15T22:28:48.746 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:48.746 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.747 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.747 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6811", 2021-03-15T22:28:48.747 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:48.747 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.748 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.748 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:48.748 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.749 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.749 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6814", 2021-03-15T22:28:48.749 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:48.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.750 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.750 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6815", 2021-03-15T22:28:48.750 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:48.750 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.751 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.751 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:48.751 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.752 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.752 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6812", 2021-03-15T22:28:48.752 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:48.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.753 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.753 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6813", 2021-03-15T22:28:48.753 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3610838738 2021-03-15T22:28:48.753 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.754 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.754 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6809/3610838738", 2021-03-15T22:28:48.754 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6811/3610838738", 2021-03-15T22:28:48.755 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6815/3610838738", 2021-03-15T22:28:48.755 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6813/3610838738", 2021-03-15T22:28:48.755 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:48.755 INFO:teuthology.orchestra.run.smithi076.stdout: "exists" 2021-03-15T22:28:48.755 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.756 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:28:48.756 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe", 2021-03-15T22:28:48.757 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:48.757 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:48.757 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:48.757 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:48.757 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:48.758 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:48.758 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 45, 2021-03-15T22:28:48.758 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 572, 2021-03-15T22:28:48.758 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:48.758 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:48.759 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:48.759 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.759 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.760 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6816", 2021-03-15T22:28:48.760 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:48.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.760 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.761 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6817", 2021-03-15T22:28:48.761 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:48.761 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.761 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.762 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:48.762 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.762 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.763 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6818", 2021-03-15T22:28:48.763 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:48.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.763 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.764 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6819", 2021-03-15T22:28:48.764 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:48.764 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.764 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.765 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:48.765 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.765 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.766 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6822", 2021-03-15T22:28:48.766 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:48.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.766 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.767 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6823", 2021-03-15T22:28:48.767 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:48.767 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.767 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.768 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:48.768 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.768 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.769 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6820", 2021-03-15T22:28:48.769 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:48.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.769 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.770 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6821", 2021-03-15T22:28:48.770 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:28:48.770 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.770 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.771 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6817/171378275", 2021-03-15T22:28:48.771 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6819/171378275", 2021-03-15T22:28:48.771 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6823/171378275", 2021-03-15T22:28:48.771 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6821/171378275", 2021-03-15T22:28:48.772 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:48.772 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:48.772 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:48.772 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.773 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:28:48.773 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614", 2021-03-15T22:28:48.773 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:28:48.773 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:28:48.774 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:28:48.774 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:28:48.774 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:28:48.774 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:28:48.775 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 52, 2021-03-15T22:28:48.775 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 570, 2021-03-15T22:28:48.775 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:28:48.775 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:28:48.775 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:28:48.776 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.776 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.776 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6824", 2021-03-15T22:28:48.776 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:48.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.777 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.777 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6825", 2021-03-15T22:28:48.778 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:48.778 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.778 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.778 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:28:48.779 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.779 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.779 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6826", 2021-03-15T22:28:48.779 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:48.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.780 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.780 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6827", 2021-03-15T22:28:48.781 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:48.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.781 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.781 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:28:48.782 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.782 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.782 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6830", 2021-03-15T22:28:48.783 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:48.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.783 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.783 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6831", 2021-03-15T22:28:48.784 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:48.784 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.784 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.784 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:28:48.785 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:28:48.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.785 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:28:48.785 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6828", 2021-03-15T22:28:48.786 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:48.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.786 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:28:48.786 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6829", 2021-03-15T22:28:48.787 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:28:48.787 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.787 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.787 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6825/652003948", 2021-03-15T22:28:48.788 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6827/652003948", 2021-03-15T22:28:48.788 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6831/652003948", 2021-03-15T22:28:48.788 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6829/652003948", 2021-03-15T22:28:48.788 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:28:48.789 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:28:48.789 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:28:48.789 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.789 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:48.790 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2021-03-15T22:28:48.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.790 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:28:48.790 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:48.791 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:48.791 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:48.791 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:48.791 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:48.791 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:27:55.678908+0000", 2021-03-15T22:28:48.792 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:48.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.792 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:28:48.792 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:48.793 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:48.793 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:48.793 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:48.793 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:48.794 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:28:12.905187+0000", 2021-03-15T22:28:48.794 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:48.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.794 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:28:48.795 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:48.795 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:48.795 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:48.795 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:48.795 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:48.796 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:26:26.403924+0000", 2021-03-15T22:28:48.796 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 490 2021-03-15T22:28:48.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.797 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:28:48.797 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:48.797 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:48.797 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:48.797 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:48.798 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 65536, 2021-03-15T22:28:48.798 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:28:41.442199+0000", 2021-03-15T22:28:48.798 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:48.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.799 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:28:48.799 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:48.799 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:48.799 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:48.800 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:48.800 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:48.800 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:28:01.554965+0000", 2021-03-15T22:28:48.800 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:48.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.801 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:28:48.801 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:48.801 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:48.801 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:48.802 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:48.802 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:48.802 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:27:17.335404+0000", 2021-03-15T22:28:48.802 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 540 2021-03-15T22:28:48.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.803 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:28:48.803 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:48.803 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:48.804 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:48.804 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:48.804 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:48.804 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:28:35.742710+0000", 2021-03-15T22:28:48.805 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:48.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.805 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:28:48.805 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:28:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:28:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:28:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:28:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:28:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:27:53.370478+0000", 2021-03-15T22:28:48.807 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:28:48.807 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.807 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:48.807 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [], 2021-03-15T22:28:48.808 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2021-03-15T22:28:48.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.808 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.3", 2021-03-15T22:28:48.808 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:48.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.809 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:28:48.809 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:28:48.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.809 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.810 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.14", 2021-03-15T22:28:48.810 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:48.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.811 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:28:48.811 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:28:48.811 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.811 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.812 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.a", 2021-03-15T22:28:48.812 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:48.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.813 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:28:48.813 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:28:48.813 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.813 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.814 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.b", 2021-03-15T22:28:48.814 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:48.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.814 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:28:48.815 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:28:48.815 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.816 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2021-03-15T22:28:48.816 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:48.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.816 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:28:48.817 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:28:48.817 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.817 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.818 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.17", 2021-03-15T22:28:48.818 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:48.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.818 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:28:48.819 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 5 2021-03-15T22:28:48.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.819 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.820 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.1b", 2021-03-15T22:28:48.820 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:28:48.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.820 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:28:48.821 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 5 2021-03-15T22:28:48.821 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.821 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.821 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.821 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:48.822 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [ 2021-03-15T22:28:48.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.822 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.1", 2021-03-15T22:28:48.822 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.822 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.823 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.823 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.823 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.824 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.1", 2021-03-15T22:28:48.824 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.824 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.825 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:48.825 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.825 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.826 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.2", 2021-03-15T22:28:48.826 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.826 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.826 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:48.826 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.827 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.827 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.3", 2021-03-15T22:28:48.828 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.828 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.828 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.828 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.828 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.829 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.5", 2021-03-15T22:28:48.829 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.829 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.830 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.830 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.831 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.7", 2021-03-15T22:28:48.831 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.831 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.831 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.831 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.832 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.832 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.c", 2021-03-15T22:28:48.832 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.833 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.833 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.833 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.833 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.834 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.10", 2021-03-15T22:28:48.834 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.834 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.835 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.835 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.835 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.836 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.17", 2021-03-15T22:28:48.836 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.836 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.836 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.837 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.837 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.837 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.18", 2021-03-15T22:28:48.838 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.838 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.838 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.838 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.838 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.839 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.1e", 2021-03-15T22:28:48.839 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.840 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.840 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.840 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.840 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.841 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.1f", 2021-03-15T22:28:48.841 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.841 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.841 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:28:48.842 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.842 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.3", 2021-03-15T22:28:48.842 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.843 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.843 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.843 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2021-03-15T22:28:48.843 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.4", 2021-03-15T22:28:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.846 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.6", 2021-03-15T22:28:48.846 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.846 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.846 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.847 INFO:teuthology.orchestra.run.smithi076.stdout: 7 2021-03-15T22:28:48.847 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.847 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.7", 2021-03-15T22:28:48.848 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.848 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.848 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.848 INFO:teuthology.orchestra.run.smithi076.stdout: 7 2021-03-15T22:28:48.848 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.849 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.b", 2021-03-15T22:28:48.849 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.849 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.850 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:48.850 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.850 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.851 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2021-03-15T22:28:48.851 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.851 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:48.851 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.852 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.852 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.853 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.11", 2021-03-15T22:28:48.853 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.853 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.853 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.853 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.854 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.854 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.14", 2021-03-15T22:28:48.854 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.855 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.855 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.855 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.855 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.856 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.15", 2021-03-15T22:28:48.856 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.856 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.857 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.857 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.857 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.16", 2021-03-15T22:28:48.858 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.858 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.858 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.859 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.859 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.859 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.17", 2021-03-15T22:28:48.860 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.860 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.860 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.860 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.861 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.19", 2021-03-15T22:28:48.861 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.861 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.861 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.862 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.862 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.1a", 2021-03-15T22:28:48.863 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.863 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:48.863 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.863 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.863 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.864 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.1b", 2021-03-15T22:28:48.864 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.864 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:48.865 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.866 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.1c", 2021-03-15T22:28:48.866 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.866 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.866 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.866 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.867 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.867 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.3", 2021-03-15T22:28:48.867 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.868 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.868 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.868 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.868 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.869 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.7", 2021-03-15T22:28:48.869 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.869 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.870 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.870 INFO:teuthology.orchestra.run.smithi076.stdout: 4 2021-03-15T22:28:48.870 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.871 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.a", 2021-03-15T22:28:48.871 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.871 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:48.871 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.871 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.872 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.872 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.b", 2021-03-15T22:28:48.873 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.873 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.873 INFO:teuthology.orchestra.run.smithi076.stdout: 7 2021-03-15T22:28:48.873 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.874 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.12", 2021-03-15T22:28:48.874 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.874 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.874 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.875 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.875 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.876 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.13", 2021-03-15T22:28:48.876 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.876 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.876 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.876 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.877 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.877 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.14", 2021-03-15T22:28:48.877 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.878 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.878 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.878 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.878 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.879 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.15", 2021-03-15T22:28:48.879 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.879 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.880 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.880 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.880 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.881 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.0", 2021-03-15T22:28:48.881 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.881 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.881 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.881 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.882 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.2", 2021-03-15T22:28:48.882 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.883 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:48.883 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.883 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.883 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.884 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.4", 2021-03-15T22:28:48.884 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.884 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.884 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.885 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.885 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.885 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.5", 2021-03-15T22:28:48.886 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.886 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.886 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.886 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.887 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.887 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.7", 2021-03-15T22:28:48.887 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.888 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:48.888 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.888 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.888 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.889 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.8", 2021-03-15T22:28:48.889 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.889 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.889 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.890 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.890 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.891 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.9", 2021-03-15T22:28:48.891 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.891 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.891 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.891 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.892 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.892 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.c", 2021-03-15T22:28:48.893 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.893 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.893 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.893 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.893 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.894 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.f", 2021-03-15T22:28:48.894 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.895 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:48.895 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.895 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.895 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.896 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.10", 2021-03-15T22:28:48.896 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.896 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.896 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.897 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:28:48.897 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.897 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.11", 2021-03-15T22:28:48.898 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.898 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.898 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.898 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.899 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.13", 2021-03-15T22:28:48.899 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.899 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.900 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.900 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.900 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.901 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.19", 2021-03-15T22:28:48.901 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.901 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.901 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.902 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.902 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.1f", 2021-03-15T22:28:48.902 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.903 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.903 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.903 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.903 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.904 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.0", 2021-03-15T22:28:48.904 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.904 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.905 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.905 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.905 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.2", 2021-03-15T22:28:48.906 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.906 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.906 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.906 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.907 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.907 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.3", 2021-03-15T22:28:48.907 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.908 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:28:48.908 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.908 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.909 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.5", 2021-03-15T22:28:48.909 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.909 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.910 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.910 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2021-03-15T22:28:48.910 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.911 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.6", 2021-03-15T22:28:48.911 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.911 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.911 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.911 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.912 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.912 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.a", 2021-03-15T22:28:48.912 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.913 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:28:48.913 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.913 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.914 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.c", 2021-03-15T22:28:48.914 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.914 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:28:48.914 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:28:48.915 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.915 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.915 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.e", 2021-03-15T22:28:48.916 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:28:48.916 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:28:48.916 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:28:48.916 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.917 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.917 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:48.917 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2021-03-15T22:28:48.917 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": { 2021-03-15T22:28:48.917 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:28:48.918 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4087324200": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:28:48.918 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4141143304": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:28:48.918 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:28:48.918 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2779879991": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:28:48.918 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:28:48.919 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:28:48.919 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2483102204": "2021-03-16T22:13:41.603652+0000" 2021-03-15T22:28:48.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.919 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2021-03-15T22:28:48.920 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2021-03-15T22:28:48.920 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2021-03-15T22:28:48.920 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2021-03-15T22:28:48.920 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2021-03-15T22:28:48.920 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2021-03-15T22:28:48.921 INFO:teuthology.orchestra.run.smithi076.stdout: "ruleset-failure-domain": "osd", 2021-03-15T22:28:48.921 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2021-03-15T22:28:48.921 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.921 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2021-03-15T22:28:48.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.922 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:28:48.922 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:28:48.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.923 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 258, 2021-03-15T22:28:48.923 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.923 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 303, 2021-03-15T22:28:48.924 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.924 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 318, 2021-03-15T22:28:48.924 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.925 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 347, 2021-03-15T22:28:48.925 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.926 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 358, 2021-03-15T22:28:48.926 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.927 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 360, 2021-03-15T22:28:48.927 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.928 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 365, 2021-03-15T22:28:48.928 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.929 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 368, 2021-03-15T22:28:48.929 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.929 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 371, 2021-03-15T22:28:48.930 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.930 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 373, 2021-03-15T22:28:48.930 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.931 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 375, 2021-03-15T22:28:48.931 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.932 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 382, 2021-03-15T22:28:48.932 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:28:48.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.933 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 388, 2021-03-15T22:28:48.933 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:28:48.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.934 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 391, 2021-03-15T22:28:48.934 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.935 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 395, 2021-03-15T22:28:48.935 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.935 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 399, 2021-03-15T22:28:48.936 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:28:48.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.936 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 409, 2021-03-15T22:28:48.937 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:28:48.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.937 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 416, 2021-03-15T22:28:48.937 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.938 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 419, 2021-03-15T22:28:48.938 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.939 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 423, 2021-03-15T22:28:48.939 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.940 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 425, 2021-03-15T22:28:48.940 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.941 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 428, 2021-03-15T22:28:48.941 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:28:48.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:28:48.942 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 433, 2021-03-15T22:28:48.942 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:28:48.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.942 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:28:48.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.943 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:28:48.943 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [], 2021-03-15T22:28:48.943 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2021-03-15T22:28:48.943 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2021-03-15T22:28:48.943 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2021-03-15T22:28:48.944 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2021-03-15T22:28:48.944 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2021-03-15T22:28:48.944 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2021-03-15T22:28:48.944 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2021-03-15T22:28:48.945 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2021-03-15T22:28:48.945 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2021-03-15T22:28:48.945 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:28:48.945 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:28:48.946 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:48.947 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: debug 2021-03-15T22:28:48.618+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:28:48.947 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:28:48.947 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:48.947 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:28:48.948 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:48.948 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:28:48.948 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:48.948 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:48.948 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:28:48.949 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: debug 2021-03-15T22:28:48.618+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:28:48.949 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:28:48.949 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:48.949 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:28:48.950 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:48.950 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:28:48.950 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:48.950 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:48.950 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:28:48.951 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: debug 2021-03-15T22:28:48.619+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:28:48.951 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:28:48.951 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:48.951 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:28:48.952 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:48.952 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:28:48.952 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:48.952 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:48.952 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:28:48.953 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: debug 2021-03-15T22:28:48.619+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:28:48.953 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:28:48.953 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:48.953 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:28:48.953 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:48.954 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:28:48.954 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:48.954 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:48.954 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:28:48.955 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: debug 2021-03-15T22:28:48.620+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:28:48.955 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:28:48.955 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:48.955 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:28:48.955 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:48.956 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:28:48.956 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:48.956 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:48.956 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:28:48.957 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: debug 2021-03-15T22:28:48.620+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:28:48.957 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:28:48.957 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:48.957 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:28:48.957 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:48.958 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:28:48.958 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:48.958 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:48.958 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:28:48.959 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: debug 2021-03-15T22:28:48.621+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:28:48.959 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:28:48.959 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:48.959 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:28:48.959 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:48.960 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:28:48.960 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:48.960 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:48.960 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:28:48.960 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:28:48 smithi076 conmon[56844]: 2021-03-15T22:28:48.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:48.962 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:48 smithi125 conmon[56739]: cluster 2021-03-15T22:28:47.113752 2021-03-15T22:28:48.962 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:48 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 767 : cluster [DBG] pgmap v1010: 138 pgs: 44 active+clean+remapped, 1 undersized+degraded+peered, 1 active+recovering+undersized+degraded+remapped, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 572 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 1.3 MiB/s wr, 10 op/s; 127/1758 objects degraded (7.224%); 402/1758 objects misplaced (22.867%); 689 KiB/s, 5 objects/s recovering 2021-03-15T22:28:48.963 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:48 smithi125 conmon[56739]: cluster 2021-03-15T22:28:47. 2021-03-15T22:28:48.963 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:48 smithi125 conmon[56739]: 140107+0000 osd.7 (osd.7) 113 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:28:48.963 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:48 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:28:48.963 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:48 smithi125 conmon[56739]: 03-15T22:28:48. 2021-03-15T22:28:48.963 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:48 smithi125 conmon[56739]: 188764+0000 mon.a (mon.0 2021-03-15T22:28:48.964 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:48 smithi125 conmon[56739]: ) 1615 : cluster [DBG] osdmap e579: 8 total, 6 up, 7 in 2021-03-15T22:28:48.964 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:48 smithi125 conmon[56739]: 2021-03-15T22:28:48.987 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:48.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:48.989 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:49.090 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:49.115 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:49.117 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:49.193 INFO:tasks.rados.rados.0.smithi076.stdout:2088: done (2 left) 2021-03-15T22:28:49.194 INFO:tasks.rados.rados.0.smithi076.stdout:2089: done (1 left) 2021-03-15T22:28:49.194 INFO:tasks.rados.rados.0.smithi076.stdout:2090: done (0 left) 2021-03-15T22:28:49.194 INFO:tasks.rados.rados.0.smithi076.stdout:2091: write oid 4 current snap is 238 2021-03-15T22:28:49.195 INFO:tasks.rados.rados.0.smithi076.stdout:2091: seq_num 727 ranges {618689=559558,1659115=559574} 2021-03-15T22:28:49.199 INFO:tasks.rados.rados.0.smithi076.stdout:2091: writing smithi07694878-4 from 618689 to 1178247 tid 1 2021-03-15T22:28:49.205 INFO:tasks.rados.rados.0.smithi076.stdout:2091: writing smithi07694878-4 from 1659115 to 2218689 tid 2 2021-03-15T22:28:49.206 INFO:tasks.rados.rados.0.smithi076.stdout:2092: delete oid 5 current snap is 238 2021-03-15T22:28:49.208 INFO:tasks.rados.rados.0.smithi076.stdout:2091: finishing write tid 1 to smithi07694878-4 2021-03-15T22:28:49.208 INFO:tasks.rados.rados.0.smithi076.stdout:2092: done (1 left) 2021-03-15T22:28:49.208 INFO:tasks.rados.rados.0.smithi076.stdout:2093: snap_remove snap 139 2021-03-15T22:28:49.219 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:49.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: cluster 2021-03-15T22:28:47.641182+0000 osd.1 (osd.1) 111 : cluster [DBG] 7.2 scrub ok 2021-03-15T22:28:49.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: cluster 2021-03-15T22:28:47.956913+0000 osd.4 (osd.4) 121 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:28:49.242 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: audit 2021 2021-03-15T22:28:49.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: -03-15T22:28:48.288498+0000 mon.a (mon.0) 1616 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:28:49.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: audit 2021-03-15T22:28:48.530767+0000 mon.a (mon.0) 1617 : audit [DBG] from='client.? 172.21.15.76:0/3286413490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:28:49.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:28:48.614124+0000 mgr.y (mgr.14144) 768 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:28:49.243 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: audit 2021-03-15T22:28:48.617938+0000 mon.a (mon.0) 1618 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:28:49.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:28:48.618844+0000 mgr.y (mgr.14144) 769 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:28:49.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:28:49.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:28:49.244 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:28:49.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.245 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:28:49.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:28:48.619357+0000 mgr.y (mgr.14144) 770 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:28:49.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:28:49.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.246 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:28:49.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:28:49.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:28:49.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:28:48.619832+0000 mgr.y (mgr.14144) 771 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:28:49.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:28:49.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.248 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:28:49.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:28:49.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.249 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:28:49.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:28:48.620296+0000 mgr.y (mgr.14144) 772 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:28:49.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:28:49.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.250 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:28:49.251 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.251 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:28:49.251 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.251 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.251 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:28:49.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:28:48.620793+0000 mgr.y (mgr.14144) 773 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:28:49.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:28:49.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.252 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:28:49.253 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.253 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:28:49.253 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.253 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.253 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:28:49.254 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:28:48.621231+0000 mgr.y (mgr.14144) 774 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:28:49.254 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:28:49.254 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.254 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:28:49.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:28:49.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:28:49.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: cephadm 2021-03-15T22:28:48.621699+0000 mgr.y (mgr.14144) 775 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:28:49.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:28:49.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.256 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:28:49.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:28:49.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.257 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:28:49.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: cluster 2021-03-15T22:28:49 2021-03-15T22:28:49.258 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[56514]: .190961+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e580: 8 total, 6 up, 7 in 2021-03-15T22:28:49.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cluster 2021-03-15T22:28:47. 2021-03-15T22:28:49.259 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: 641182+0000 osd.1 (osd.1) 111 : cluster [DBG] 7.2 scrub ok 2021-03-15T22:28:49.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:28:49.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: -03-15T22:28:47.956913+0000 osd.4 (osd.4) 121 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:28:49.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: audit 2021 2021-03-15T22:28:49.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: -03-15T22:28:48.288498+0000 2021-03-15T22:28:49.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: mon.a (mon.0) 1616 : audit [DBG] 2021-03-15T22:28:49.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:28:49.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: audit 2021-03-15T22:28:48 2021-03-15T22:28:49.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: .530767+0000 mon.a (mon.0) 1617 : audit 2021-03-15T22:28:49.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/3286413490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:28:49.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cephadm 2021-03-15T22: 2021-03-15T22:28:49.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: 28:48.614124+0000 mgr.y (mgr.14144 2021-03-15T22:28:49.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: ) 768 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:28:49.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:28:49.263 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: -15T22:28:48.617938+0000 mon.a 2021-03-15T22:28:49.263 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: (mon.0) 1618 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:28:49.263 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: 2021-03-15T22:28:49.263 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:28:49.263 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: 2021-03-15T22:28:48.618844+0000 mgr.y ( 2021-03-15T22:28:49.264 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: mgr.14144) 769 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:28:49.264 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:28:49.264 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.264 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:28:49.264 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:28:49.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:28:49.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:28:48.619357+0000 mgr.y (mgr.14144) 770 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:28:49.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:28:49.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:28:49.266 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:28:49.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:28:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:28:48.619832+0000 mgr.y (mgr.14144) 771 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:28:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:28:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:28:49.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:28:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:28:49.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:28 2021-03-15T22:28:49.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: :48.620296+0000 mgr.y (mgr.14144) 772 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:28:49.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:28:49.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:28:49.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:28:49.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:28:49.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:28:48.620793+0000 mgr.y (mgr.14144) 773 : 2021-03-15T22:28:49.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:28:49.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:28:49.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:28:49.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:28:49.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:28:49.274 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:28:48.621231+0000 mgr.y (mgr.14144) 774 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:28:49.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:28:49.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:28:49.275 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:28:49.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:28:49.276 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cephadm 2021-03-15T22:28:48.621699+0000 mgr.y (mgr.14144) 775 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:28:49.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:28:49.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:28:49.277 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:28:49.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.278 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:28:49.279 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:49 smithi076 conmon[62156]: cluster 2021-03-15T22:28:49.190961+0000 mon.a (mon.0) 1619 : cluster [DBG] osdmap e580: 8 total, 6 up, 7 in 2021-03-15T22:28:49.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:49.280 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:49.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:49.382 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:49.403 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:49.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:49.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:49.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:49.516 INFO:tasks.thrashosds.thrasher:j is {'epoch': 579, 'fsid': '8a572802-85db-11eb-9091-001a4aab830c', 'created': '2021-03-15T22:13:20.370795+0000', 'modified': '2021-03-15T22:28:48.184760+0000', 'last_up_change': '2021-03-15T22:27:35.183905+0000', 'last_in_change': '2021-03-15T22:28:38.336225+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-15T22:16:34.817191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-15T22:17:49.906400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-15T22:17:52.257970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-15T22:17:54.562498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-15T22:17:56.243806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 182, 'last_epoch_started': 181, 'last_epoch_clean': 181, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '182', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '182', 'last_force_op_resend_preluminous': '180', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-15T22:17:58.730534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-03-15T22:20:03.377294+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '579', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 445, 'snap_epoch': 579, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 572, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 188228176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 188228176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 188228176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 188228176}]}, 'public_addr': '172.21.15.76:6803/188228176', 'cluster_addr': '172.21.15.76:6805/188228176', 'heartbeat_back_addr': '172.21.15.76:6809/188228176', 'heartbeat_front_addr': '172.21.15.76:6807/188228176', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '97bd3ffc-15c8-4a6f-b37a-217813def526', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.999237060546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 576, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 3924955423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 3924955423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 3924955423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 3924955423}]}, 'public_addr': '172.21.15.76:6811/3924955423', 'cluster_addr': '172.21.15.76:6813/3924955423', 'heartbeat_back_addr': '172.21.15.76:6817/3924955423', 'heartbeat_front_addr': '172.21.15.76:6815/3924955423', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a7f5f115-e69e-47d9-b6c3-ff386ddfaa05', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 421, 'down_at': 490, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 1401263050}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 1401263050}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 1401263050}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 1401263050}]}, 'public_addr': '172.21.15.76:6819/1401263050', 'cluster_addr': '172.21.15.76:6821/1401263050', 'heartbeat_back_addr': '172.21.15.76:6825/1401263050', 'heartbeat_front_addr': '172.21.15.76:6823/1401263050', 'state': ['exists']}, {'osd': 3, 'uuid': 'a0bb7f88-0114-47cb-a82c-d8fb002aac56', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 571, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 3960968636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 3960968636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 3960968636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 3960968636}]}, 'public_addr': '172.21.15.76:6827/3960968636', 'cluster_addr': '172.21.15.76:6829/3960968636', 'heartbeat_back_addr': '172.21.15.76:6833/3960968636', 'heartbeat_front_addr': '172.21.15.76:6831/3960968636', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b650611c-6101-47e8-9ca9-94e02cf8b237', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 574, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6800', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6801', 'nonce': 37386122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6802', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6803', 'nonce': 37386122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6806', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6807', 'nonce': 37386122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6804', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6805', 'nonce': 37386122}]}, 'public_addr': '172.21.15.125:6801/37386122', 'cluster_addr': '172.21.15.125:6803/37386122', 'heartbeat_back_addr': '172.21.15.125:6807/37386122', 'heartbeat_front_addr': '172.21.15.125:6805/37386122', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7d6ceff2-7093-4ab9-8b96-ade448093bf2', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 38, 'up_thru': 490, 'down_at': 540, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6808', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6809', 'nonce': 3610838738}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6810', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6811', 'nonce': 3610838738}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6814', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6815', 'nonce': 3610838738}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6812', 'nonce': 3610838738}, {'type': 'v1', 'addr': '172.21.15.125:6813', 'nonce': 3610838738}]}, 'public_addr': '172.21.15.125:6809/3610838738', 'cluster_addr': '172.21.15.125:6811/3610838738', 'heartbeat_back_addr': '172.21.15.125:6815/3610838738', 'heartbeat_front_addr': '172.21.15.125:6813/3610838738', 'state': ['exists']}, {'osd': 6, 'uuid': '98b72022-a9a7-4ee2-89a1-cac037de7afe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 572, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6816', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6817', 'nonce': 171378275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6818', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6819', 'nonce': 171378275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6822', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6823', 'nonce': 171378275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6820', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6821', 'nonce': 171378275}]}, 'public_addr': '172.21.15.125:6817/171378275', 'cluster_addr': '172.21.15.125:6819/171378275', 'heartbeat_back_addr': '172.21.15.125:6823/171378275', 'heartbeat_front_addr': '172.21.15.125:6821/171378275', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f99ea650-b2dc-4d87-a44c-56a897edd614', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 570, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6824', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6825', 'nonce': 652003948}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6826', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6827', 'nonce': 652003948}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6830', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6831', 'nonce': 652003948}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6828', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6829', 'nonce': 652003948}]}, 'public_addr': '172.21.15.125:6825/652003948', 'cluster_addr': '172.21.15.125:6827/652003948', 'heartbeat_back_addr': '172.21.15.125:6831/652003948', 'heartbeat_front_addr': '172.21.15.125:6829/652003948', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:27:55.678908+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:28:12.905187+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:26:26.403924+0000', 'dead_epoch': 490}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 65536, 'last_purged_snaps_scrub': '2021-03-15T22:28:41.442199+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:28:01.554965+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:27:17.335404+0000', 'dead_epoch': 540}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:28:35.742710+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:27:53.370478+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.14', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.17', 'mappings': [{'from': 1, 'to': 5}]}, {'pgid': '4.1b', 'mappings': [{'from': 1, 'to': 5}]}], 'pg_temp': [{'pgid': '2.1', 'osds': [0, 1, 3]}, {'pgid': '3.1', 'osds': [0, 4, 3]}, {'pgid': '3.2', 'osds': [6, 4, 3]}, {'pgid': '3.3', 'osds': [0, 6, 3]}, {'pgid': '3.5', 'osds': [6, 3]}, {'pgid': '3.7', 'osds': [7, 0, 3]}, {'pgid': '3.c', 'osds': [6, 1, 3]}, {'pgid': '3.10', 'osds': [6, 0, 3]}, {'pgid': '3.17', 'osds': [0, 6, 3]}, {'pgid': '3.18', 'osds': [0, 1, 3]}, {'pgid': '3.1e', 'osds': [6, 1, 3]}, {'pgid': '3.1f', 'osds': [0, 6]}, {'pgid': '4.3', 'osds': [0, 7, 1]}, {'pgid': '4.4', 'osds': [1, 6, 3]}, {'pgid': '4.6', 'osds': [0, 1, 7]}, {'pgid': '4.7', 'osds': [1, 0, 7]}, {'pgid': '4.b', 'osds': [0, 4, 3]}, {'pgid': '4.f', 'osds': [4, 7, 3]}, {'pgid': '4.11', 'osds': [7, 6, 3]}, {'pgid': '4.14', 'osds': [1, 7, 3]}, {'pgid': '4.15', 'osds': [7, 6, 3]}, {'pgid': '4.16', 'osds': [0, 7, 3]}, {'pgid': '4.17', 'osds': [7, 3]}, {'pgid': '4.19', 'osds': [0, 3]}, {'pgid': '4.1a', 'osds': [4, 0, 3]}, {'pgid': '4.1b', 'osds': [4, 3]}, {'pgid': '4.1c', 'osds': [1, 0, 3]}, {'pgid': '5.3', 'osds': [0, 6, 3]}, {'pgid': '5.7', 'osds': [1, 0, 4]}, {'pgid': '5.a', 'osds': [4, 0, 3]}, {'pgid': '5.b', 'osds': [0, 7]}, {'pgid': '5.12', 'osds': [1, 7, 3]}, {'pgid': '5.13', 'osds': [0, 1, 3]}, {'pgid': '5.14', 'osds': [7, 1, 3]}, {'pgid': '5.15', 'osds': [1, 0, 3]}, {'pgid': '6.0', 'osds': [0, 3]}, {'pgid': '6.2', 'osds': [4, 6, 3]}, {'pgid': '6.4', 'osds': [1, 6, 3]}, {'pgid': '6.5', 'osds': [7, 6, 3]}, {'pgid': '6.7', 'osds': [4, 1, 3]}, {'pgid': '6.8', 'osds': [7, 6, 3]}, {'pgid': '6.9', 'osds': [0, 7, 3]}, {'pgid': '6.c', 'osds': [6, 7, 3]}, {'pgid': '6.f', 'osds': [4, 6, 3]}, {'pgid': '6.10', 'osds': [0, 1, 6]}, {'pgid': '6.11', 'osds': [0, 3]}, {'pgid': '6.13', 'osds': [0, 6, 3]}, {'pgid': '6.19', 'osds': [1, 3]}, {'pgid': '6.1f', 'osds': [6, 1, 3]}, {'pgid': '7.0', 'osds': [1, 3]}, {'pgid': '7.2', 'osds': [1, 7, 3]}, {'pgid': '7.3', 'osds': [4, 6, 3]}, {'pgid': '7.5', 'osds': [7, 0, 1]}, {'pgid': '7.6', 'osds': [0, 7, 3]}, {'pgid': '7.a', 'osds': [1, 3]}, {'pgid': '7.c', 'osds': [6, 7, 3]}, {'pgid': '7.e', 'osds': [0, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.76:6800/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4087324200': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4141143304': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:6801/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/2779879991': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6801/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6800/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:0/2483102204': '2021-03-16T22:13:41.603652+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 258, 'length': 1}, {'begin': 303, 'length': 1}, {'begin': 318, 'length': 1}, {'begin': 347, 'length': 1}, {'begin': 358, 'length': 1}, {'begin': 360, 'length': 1}, {'begin': 365, 'length': 1}, {'begin': 368, 'length': 1}, {'begin': 371, 'length': 1}, {'begin': 373, 'length': 1}, {'begin': 375, 'length': 1}, {'begin': 382, 'length': 2}, {'begin': 388, 'length': 2}, {'begin': 391, 'length': 1}, {'begin': 395, 'length': 1}, {'begin': 399, 'length': 2}, {'begin': 409, 'length': 2}, {'begin': 416, 'length': 1}, {'begin': 419, 'length': 1}, {'begin': 423, 'length': 1}, {'begin': 425, 'length': 1}, {'begin': 428, 'length': 1}, {'begin': 433, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-03-15T22:28:49.517 INFO:tasks.thrashosds.thrasher:No pg_upmap entries; doing nothing 2021-03-15T22:28:49.530 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:49.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:49.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:49.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cluster 2021-03-15T22:28:49.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 47 2021-03-15T22:28:49.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 641182 2021-03-15T22:28:49.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: osd.1 2021-03-15T22:28:49.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: osd 2021-03-15T22:28:49.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 1 2021-03-15T22:28:49.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 111 2021-03-15T22:28:49.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cluster 2021-03-15T22:28:49.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:49.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 7.2 scrub ok 2021-03-15T22:28:49.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cluster 2021-03-15T22:28:49.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 47 2021-03-15T22:28:49.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 956913 2021-03-15T22:28:49.547 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: osd.4 2021-03-15T22:28:49.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: osd 2021-03-15T22:28:49.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 4 2021-03-15T22:28:49.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 121 2021-03-15T22:28:49.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cluster 2021-03-15T22:28:49.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:49.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 4.1a scrub ok 2021-03-15T22:28:49.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: audit 2021-03-15T22:28:49.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.553 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 48 2021-03-15T22:28:49.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 288498 2021-03-15T22:28:49.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.554 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:49.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mon 2021-03-15T22:28:49.555 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 0 2021-03-15T22:28:49.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 1616 2021-03-15T22:28:49.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.556 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: audit 2021-03-15T22:28:49.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:49.557 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:28:49.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: audit 2021-03-15T22:28:49.559 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.560 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.561 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 48 2021-03-15T22:28:49.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 530767 2021-03-15T22:28:49.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:49.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mon 2021-03-15T22:28:49.564 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.564 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 0 2021-03-15T22:28:49.564 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.564 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 1617 2021-03-15T22:28:49.564 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.565 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: audit 2021-03-15T22:28:49.565 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.565 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:49.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/3286413490' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:28:49.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 48 2021-03-15T22:28:49.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 614124 2021-03-15T22:28:49.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:49.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr 2021-03-15T22:28:49.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 14144 2021-03-15T22:28:49.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 768 2021-03-15T22:28:49.572 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.572 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.572 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.572 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [INF] 2021-03-15T22:28:49.572 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:28:49.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: audit 2021-03-15T22:28:49.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.575 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.575 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.575 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.576 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.576 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.576 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 48 2021-03-15T22:28:49.576 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 617938 2021-03-15T22:28:49.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:49.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.578 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mon 2021-03-15T22:28:49.578 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.578 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 0 2021-03-15T22:28:49.578 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.579 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 1618 2021-03-15T22:28:49.579 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.579 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: audit 2021-03-15T22:28:49.579 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.580 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [INF] 2021-03-15T22:28:49.580 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.580 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:28:49.581 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.581 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.581 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.581 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.581 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.582 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.582 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.582 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.582 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.583 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.583 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.583 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.583 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 48 2021-03-15T22:28:49.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 618844 2021-03-15T22:28:49.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:49.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr 2021-03-15T22:28:49.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 14144 2021-03-15T22:28:49.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 769 2021-03-15T22:28:49.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [ERR] 2021-03-15T22:28:49.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:28:49.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:28:49.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:28:49.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:28:49.589 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.589 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.589 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:28:49.589 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.590 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.590 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.590 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.590 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.591 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.591 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.591 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.591 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.591 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.592 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.592 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.592 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.592 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 48 2021-03-15T22:28:49.593 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.593 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 619357 2021-03-15T22:28:49.593 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.593 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.593 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:49.594 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.594 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr 2021-03-15T22:28:49.594 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.594 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 14144 2021-03-15T22:28:49.595 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.595 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 770 2021-03-15T22:28:49.595 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.595 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.596 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.596 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [ERR] 2021-03-15T22:28:49.596 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.596 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:28:49.596 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:28:49.597 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.597 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:28:49.597 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.597 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:28:49.598 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.598 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.598 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:28:49.598 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.599 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.599 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.599 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.599 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.599 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.600 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.600 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.600 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.600 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.600 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.601 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.601 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.601 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 48 2021-03-15T22:28:49.601 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.602 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 619832 2021-03-15T22:28:49.602 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.602 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.606 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:49.606 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.607 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr 2021-03-15T22:28:49.607 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.607 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 14144 2021-03-15T22:28:49.607 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.608 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 771 2021-03-15T22:28:49.608 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.608 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.608 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.608 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [ERR] 2021-03-15T22:28:49.609 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.609 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:28:49.609 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:28:49.609 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.610 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:28:49.610 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.610 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:28:49.610 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.611 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.611 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:28:49.611 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.611 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.611 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.612 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.612 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.612 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.612 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.612 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.613 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.613 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.613 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.613 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.614 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.614 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 48 2021-03-15T22:28:49.614 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.614 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 620296 2021-03-15T22:28:49.614 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.615 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.615 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:49.615 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.615 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr 2021-03-15T22:28:49.616 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.616 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 14144 2021-03-15T22:28:49.616 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.616 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 772 2021-03-15T22:28:49.616 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.617 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.617 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.617 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [ERR] 2021-03-15T22:28:49.617 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.618 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:28:49.618 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:28:49.618 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.618 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:28:49.618 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.619 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:28:49.619 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.619 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.619 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:28:49.620 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.620 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.620 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.620 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.620 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.621 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.621 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.621 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.621 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.622 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.622 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.622 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.622 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.622 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 48 2021-03-15T22:28:49.623 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.623 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 620793 2021-03-15T22:28:49.623 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.623 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.623 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:49.624 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.624 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr 2021-03-15T22:28:49.624 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.624 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 14144 2021-03-15T22:28:49.625 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.625 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 773 2021-03-15T22:28:49.625 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.625 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.625 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.626 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [ERR] 2021-03-15T22:28:49.626 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.626 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:28:49.626 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:28:49.627 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.627 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:28:49.627 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.627 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:28:49.627 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.628 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.628 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:28:49.628 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.628 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.629 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.629 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.629 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.629 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.629 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.630 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.630 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.630 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.630 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.630 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.631 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.631 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 48 2021-03-15T22:28:49.631 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.631 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 621231 2021-03-15T22:28:49.632 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.632 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.632 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:49.632 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.632 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr 2021-03-15T22:28:49.633 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.633 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 14144 2021-03-15T22:28:49.633 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.633 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 774 2021-03-15T22:28:49.633 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.634 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.634 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.634 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [ERR] 2021-03-15T22:28:49.634 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.635 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:28:49.635 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:28:49.635 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.635 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:28:49.635 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.636 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:28:49.636 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.636 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.636 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:28:49.637 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.637 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.637 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.637 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.637 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.638 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.638 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.638 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.638 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.639 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.639 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.639 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.639 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.639 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 48 2021-03-15T22:28:49.640 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.640 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 621699 2021-03-15T22:28:49.640 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.640 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.640 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr.y 2021-03-15T22:28:49.641 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.641 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mgr 2021-03-15T22:28:49.641 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.641 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 14144 2021-03-15T22:28:49.642 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.642 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 775 2021-03-15T22:28:49.642 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.642 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cephadm 2021-03-15T22:28:49.642 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.643 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [ERR] 2021-03-15T22:28:49.643 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.643 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:28:49.643 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:28:49.644 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:28:49.644 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:28:49.644 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:28:49.644 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:28:49.644 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:28:49.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:28:49.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:28:49.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.645 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cluster 2021-03-15T22:28:49.646 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.646 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021 2021-03-15T22:28:49.646 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.646 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 03 2021-03-15T22:28:49.646 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: - 2021-03-15T22:28:49.647 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 15 2021-03-15T22:28:49.647 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: T 2021-03-15T22:28:49.647 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 22 2021-03-15T22:28:49.647 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.647 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 28 2021-03-15T22:28:49.648 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.648 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 49 2021-03-15T22:28:49.648 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.648 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 190961 2021-03-15T22:28:49.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: +0000 2021-03-15T22:28:49.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mon.a 2021-03-15T22:28:49.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ( 2021-03-15T22:28:49.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: mon 2021-03-15T22:28:49.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: . 2021-03-15T22:28:49.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 0 2021-03-15T22:28:49.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: ) 2021-03-15T22:28:49.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 1619 2021-03-15T22:28:49.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: : 2021-03-15T22:28:49.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: cluster 2021-03-15T22:28:49.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: [DBG] 2021-03-15T22:28:49.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.652 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: osdmap e580: 8 total, 6 up, 7 in 2021-03-15T22:28:49.652 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[56739]: 2021-03-15T22:28:49.652 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:49.675 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:49.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:49.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:49.757 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:28:49.757 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:28:49.757 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:28:49.758 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:28:49.777 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:49.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:49.800 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:49.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:49.902 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:49.923 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:49.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:49.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:50.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:50.047 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:50.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:50.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:50.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:50.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:50.174 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:50.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:50.185 INFO:tasks.rados.rados.0.smithi076.stdout:2093: done (1 left) 2021-03-15T22:28:50.186 INFO:tasks.rados.rados.0.smithi076.stdout:2094: write (excl) oid 8 current snap is 238 2021-03-15T22:28:50.186 INFO:tasks.rados.rados.0.smithi076.stdout:2094: seq_num 729 ranges {652303=710056,2059726=192577} 2021-03-15T22:28:50.193 INFO:tasks.rados.rados.0.smithi076.stdout:2094: writing smithi07694878-8 from 652303 to 1362359 tid 1 2021-03-15T22:28:50.199 INFO:tasks.rados.rados.0.smithi076.stdout:2094: writing smithi07694878-8 from 2059726 to 2252303 tid 2 2021-03-15T22:28:50.200 INFO:tasks.rados.rados.0.smithi076.stdout:2095: delete oid 20 current snap is 238 2021-03-15T22:28:50.200 INFO:tasks.rados.rados.0.smithi076.stdout:2091: finishing write tid 2 to smithi07694878-4 2021-03-15T22:28:50.200 INFO:tasks.rados.rados.0.smithi076.stdout:2091: finishing write tid 3 to smithi07694878-4 2021-03-15T22:28:50.200 INFO:tasks.rados.rados.0.smithi076.stdout:2091: finishing write tid 4 to smithi07694878-4 2021-03-15T22:28:50.200 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 560 (ObjNum 727 snap 238 seq_num 727) dirty exists 2021-03-15T22:28:50.201 INFO:tasks.rados.rados.0.smithi076.stdout:2091: left oid 4 (ObjNum 727 snap 238 seq_num 727) 2021-03-15T22:28:50.201 INFO:tasks.rados.rados.0.smithi076.stdout:2091: done (2 left) 2021-03-15T22:28:50.201 INFO:tasks.rados.rados.0.smithi076.stdout:2095: done (1 left) 2021-03-15T22:28:50.202 INFO:tasks.rados.rados.0.smithi076.stdout:2096: snap_create 2021-03-15T22:28:50.213 INFO:tasks.rados.rados.0.smithi076.stdout:2094: finishing write tid 1 to smithi07694878-8 2021-03-15T22:28:50.215 INFO:tasks.rados.rados.0.smithi076.stdout:2094: finishing write tid 2 to smithi07694878-8 2021-03-15T22:28:50.215 INFO:tasks.rados.rados.0.smithi076.stdout:2094: finishing write tid 3 to smithi07694878-8 2021-03-15T22:28:50.216 INFO:tasks.rados.rados.0.smithi076.stdout:2094: finishing write tid 4 to smithi07694878-8 2021-03-15T22:28:50.216 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1191 (ObjNum 729 snap 238 seq_num 729) dirty exists 2021-03-15T22:28:50.216 INFO:tasks.rados.rados.0.smithi076.stdout:2094: left oid 8 (ObjNum 729 snap 238 seq_num 729) 2021-03-15T22:28:50.231 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:28:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:28:50.276 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:50.298 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:50.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:50.400 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:50.421 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:50.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:50.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:50.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[62156]: cluster 2021-03-15T22:28:48.653961+0000 osd.1 (osd.1) 112 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:28:50.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[62156]: cluster 2021-03-15T22:28:49.114329+0000 mgr.y (mgr.14144) 776 : cluster [DBG] pgmap v1013: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 576 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 2.0 MiB/s wr, 11 op/s; 114/1764 objects degraded (6.463%); 411/1764 objects misplaced (23.299%); 4.0 MiB/s, 2 objects/s recovering 2021-03-15T22:28:50.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[62156]: cluster 2021-03-15T22:28:49.615547+0000 mon.a (mon.0) 1620 : cluster [WRN] Health check update: Degraded data redundancy: 114/1764 objects degraded (6.463%), 12 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:50.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[62156]: cluster 2021-03-15T22:28:49.778246+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:28:50.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[62156]: audit 2021-03-15T22:28:49.893419+0000 mgr.y (mgr.14144) 777 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:50.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[62156]: cluster 2021-03-15T22:28:50.169586+0000 mon.a (mon.0) 1621 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:50.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[62156]: cluster 2021-03-15T22:28:50.179568+0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e581: 8 total, 6 up, 7 in 2021-03-15T22:28:50.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[56514]: cluster 2021-03-15T22:28:48.653961+0000 osd.1 (osd.1) 112 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:28:50.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[56514]: cluster 2021-03-15T22:28:49.114329+0000 mgr.y (mgr.14144) 776 : cluster [DBG] pgmap v1013: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 576 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 2.0 MiB/s wr, 11 op/s; 114/1764 objects degraded (6.463%); 411/1764 objects misplaced (23.299%); 4.0 MiB/s, 2 objects/s recovering 2021-03-15T22:28:50.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[56514]: cluster 2021-03-15T22:28:49.615547+0000 mon.a (mon.0) 1620 : cluster [WRN] Health check update: Degraded data redundancy: 114/1764 objects degraded (6.463%), 12 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:50.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[56514]: cluster 2021-03-15T22:28:49.778246+0000 osd.0 (osd.0) 85 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:28:50.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[56514]: audit 2021-03-15T22:28:49.893419+0000 mgr.y (mgr.14144) 777 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:50.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[56514]: cluster 2021-03-15T22:28:50.169586+0000 mon.a (mon.0) 1621 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:50.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[56514]: cluster 2021-03-15T22:28:50.179568 2021-03-15T22:28:50.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:50 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1622 : cluster [DBG] osdmap e581: 8 total, 6 up, 7 in 2021-03-15T22:28:50.525 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: cluster 2021-03-15T22:28:48.653961+0000 2021-03-15T22:28:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: osd.1 (osd.1) 112 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:28:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: cluster 2021-03-15T22:28:49.114329+0000 mgr.y (mgr.14144) 776 : cluster [DBG] pgmap v1013: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 576 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 2.0 MiB/s wr, 11 op/s; 114/1764 objects degraded (6.463%); 411/1764 objects misplaced (23.299%); 4.0 MiB/s, 2 objects/s recovering 2021-03-15T22:28:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: cluster 2021-03-15T22:28:49.615547+0000 mon.a (mon 2021-03-15T22:28:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: .0) 1620 : cluster [WRN] Health check update: Degraded data redundancy: 114/1764 objects degraded (6.463%), 12 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:28:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: 15T22:28:49.778246+0000 osd.0 2021-03-15T22:28:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: (osd.0) 85 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:28:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: 2021-03-15T22:28:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: audit 2021-03-15T22:28:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: 2021-03-15T22:28:49.893419+0000 mgr.y 2021-03-15T22:28:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: (mgr.14144) 777 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:28:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: 2021-03-15T22:28:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:28:50.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: -03-15T22:28:50.169586+0000 mon.a ( 2021-03-15T22:28:50.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: mon.0) 1621 : cluster [WRN] Health check update: Reduced data availability: 6 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:28:50.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:28:50.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: 03-15T22:28:50.179568+0000 2021-03-15T22:28:50.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: mon.a (mon.0) 1622 : cluster 2021-03-15T22:28:50.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:50 smithi125 conmon[56739]: [DBG] osdmap e581: 8 total, 6 up, 7 in 2021-03-15T22:28:50.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:50.548 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:50.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:50.650 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:50.672 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:50.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:50.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:50.775 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:50.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:50.800 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:50.801 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:50.902 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:50.925 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:50.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:51.028 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:51.050 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:51.052 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:51.153 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:51.175 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:51.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:51.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:51.196 INFO:tasks.rados.rados.0.smithi076.stdout:2094: done (1 left) 2021-03-15T22:28:51.196 INFO:tasks.rados.rados.0.smithi076.stdout:2096: done (0 left) 2021-03-15T22:28:51.196 INFO:tasks.rados.rados.0.smithi076.stdout:2097: delete oid 33 current snap is 239 2021-03-15T22:28:51.201 INFO:tasks.rados.rados.0.smithi076.stdout:2097: done (0 left) 2021-03-15T22:28:51.201 INFO:tasks.rados.rados.0.smithi076.stdout:2098: write (excl) oid 38 current snap is 239 2021-03-15T22:28:51.201 INFO:tasks.rados.rados.0.smithi076.stdout:2098: seq_num 732 ranges {702724=652156,1951525=351199} 2021-03-15T22:28:51.209 INFO:tasks.rados.rados.0.smithi076.stdout:2098: writing smithi07694878-38 from 702724 to 1354880 tid 1 2021-03-15T22:28:51.217 INFO:tasks.rados.rados.0.smithi076.stdout:2098: writing smithi07694878-38 from 1951525 to 2302724 tid 2 2021-03-15T22:28:51.217 INFO:tasks.rados.rados.0.smithi076.stdout:2099: snap_remove snap 232 2021-03-15T22:28:51.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:51.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:51.301 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:51.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:51.402 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:51.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:51.425 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:51.426 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:51.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:51.550 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:51.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:51.551 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:51.640 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:28:51.652 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:51.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:51.686 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:51.686 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:51.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:51.810 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:51.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:51.913 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:51.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:51.945 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:51.946 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:52.047 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:52.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:52.071 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:52.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:52.090 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:52.125 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:28:52.143 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:28:52.162 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:28:52.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:52.180 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:28:52.194 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[56514]: cluster 2021-03-15T22:28:50.922660+0000 osd.4 2021-03-15T22:28:52.194 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[56514]: (osd.4) 122 : cluster [DBG] 5.a scrub ok 2021-03-15T22:28:52.194 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[56514]: cluster 2021-03-15T22:28:51.115176+0000 mgr.y (mgr.14144) 778 : cluster [DBG] pgmap v1016: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 554 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.2 MiB/s wr, 6 op/s; 114/1740 objects degraded (6.552%); 404/1740 objects misplaced (23.218%); 3.3 MiB/s, 2 objects/s recovering 2021-03-15T22:28:52.194 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[56514]: cluster 2021-03-15T22:28:51.193951+0000 mon.a (mon.0) 1623 : cluster [DBG] osdmap e582: 8 total, 6 up, 7 in 2021-03-15T22:28:52.195 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[56514]: cluster 2021-03-15T22:28:51.825147+0000 osd.0 (osd.0) 86 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:28:52.195 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[56514]: audit 2021-03-15T22:28:52.086639+0000 mon.a (mon.0) 1624 : audit [INF] from='client.? 172.21.15.76:0/2570303318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:28:52.195 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[56514]: audit 2021-03-15T22:28:52.104311+0000 mon.a (mon.0) 1625 : audit [DBG] from='client.? 172.21.15.76:0/3554586440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:52.199 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:52.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:52.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:52.202 INFO:tasks.rados.rados.0.smithi076.stdout:2099: done (1 left) 2021-03-15T22:28:52.203 INFO:tasks.rados.rados.0.smithi076.stdout:2100: write oid 49 current snap is 239 2021-03-15T22:28:52.203 INFO:tasks.rados.rados.0.smithi076.stdout:2100: seq_num 733 ranges {719531=727405,1893654=425877} 2021-03-15T22:28:52.204 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:28:52.212 INFO:tasks.rados.rados.0.smithi076.stdout:2100: writing smithi07694878-49 from 719531 to 1446936 tid 1 2021-03-15T22:28:52.218 INFO:tasks.rados.rados.0.smithi076.stdout:2100: writing smithi07694878-49 from 1893654 to 2319531 tid 2 2021-03-15T22:28:52.218 INFO:tasks.rados.rados.0.smithi076.stdout:2101: delete oid 44 current snap is 239 2021-03-15T22:28:52.218 INFO:tasks.rados.rados.0.smithi076.stdout:2098: finishing write tid 1 to smithi07694878-38 2021-03-15T22:28:52.218 INFO:tasks.rados.rados.0.smithi076.stdout:2098: finishing write tid 2 to smithi07694878-38 2021-03-15T22:28:52.219 INFO:tasks.rados.rados.0.smithi076.stdout:2098: finishing write tid 3 to smithi07694878-38 2021-03-15T22:28:52.219 INFO:tasks.rados.rados.0.smithi076.stdout:2098: finishing write tid 4 to smithi07694878-38 2021-03-15T22:28:52.219 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 1036 (ObjNum 732 snap 239 seq_num 732) dirty exists 2021-03-15T22:28:52.219 INFO:tasks.rados.rados.0.smithi076.stdout:2098: left oid 38 (ObjNum 732 snap 239 seq_num 732) 2021-03-15T22:28:52.220 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:28:52.220 INFO:tasks.rados.rados.0.smithi076.stdout:2098: done (2 left) 2021-03-15T22:28:52.221 INFO:tasks.rados.rados.0.smithi076.stdout:2101: done (1 left) 2021-03-15T22:28:52.221 INFO:tasks.rados.rados.0.smithi076.stdout:2102: read oid 1 snap -1 2021-03-15T22:28:52.221 INFO:tasks.rados.rados.0.smithi076.stdout:2102: expect (ObjNum 667 snap 211 seq_num 667) 2021-03-15T22:28:52.221 INFO:tasks.rados.rados.0.smithi076.stdout:2103: read oid 22 snap -1 2021-03-15T22:28:52.222 INFO:tasks.rados.rados.0.smithi076.stdout:2103: expect (ObjNum 674 snap 214 seq_num 674) 2021-03-15T22:28:52.222 INFO:tasks.rados.rados.0.smithi076.stdout:2104: read oid 44 snap -1 2021-03-15T22:28:52.222 INFO:tasks.rados.rados.0.smithi076.stdout:2104: expect deleted 2021-03-15T22:28:52.222 INFO:tasks.rados.rados.0.smithi076.stdout:2105: write (excl) oid 42 current snap is 239 2021-03-15T22:28:52.222 INFO:tasks.rados.rados.0.smithi076.stdout:2105: seq_num 735 ranges {753145=477903,2010618=342527} 2021-03-15T22:28:52.227 INFO:tasks.rados.rados.0.smithi076.stdout:2105: writing smithi07694878-42 from 753145 to 1231048 tid 1 2021-03-15T22:28:52.236 INFO:tasks.rados.rados.0.smithi076.stdout:2105: writing smithi07694878-42 from 2010618 to 2353145 tid 2 2021-03-15T22:28:52.238 INFO:tasks.rados.rados.0.smithi076.stdout:2106: snap_remove snap 203 2021-03-15T22:28:52.247 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:28:52.248 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:28:52.248 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:28:52.249 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:28:52.249 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:28:52.249 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:28:52.249 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:28:52.250 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:28:52.250 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:28:52.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:52.325 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:52.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:52.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:52.427 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:52.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:52.451 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:52.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:52.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[62156]: cluster 2021-03-15T22:28:50.922660+0000 osd.4 (osd.4) 122 : cluster [DBG] 5.a scrub ok 2021-03-15T22:28:52.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[62156]: cluster 2021-03-15T22:28:51.115176+0000 mgr.y (mgr.14144) 778 : cluster [DBG] pgmap v1016: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 554 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.2 MiB/s wr, 6 op/s; 114/1740 objects degraded (6.552%); 404/1740 objects misplaced (23.218%); 3.3 MiB/s, 2 objects/s recovering 2021-03-15T22:28:52.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[62156]: cluster 2021-03-15T22:28:51.193951+0000 mon.a (mon.0) 1623 : cluster [DBG] osdmap e582: 8 total, 6 up, 7 in 2021-03-15T22:28:52.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[62156]: cluster 2021-03-15T22:28:51.825147+0000 osd.0 (osd.0) 86 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:28:52.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[62156]: audit 2021-03-15T22:28:52.086639+0000 mon.a (mon.0) 1624 : audit [INF] from='client.? 172.21.15.76:0/2570303318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:28:52.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:52 smithi076 conmon[62156]: audit 2021-03-15T22:28:52.104311+0000 mon.a (mon.0) 1625 : audit [DBG] from='client.? 172.21.15.76:0/3554586440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:52.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: cluster 2021-03-15T22:28:50.922660+0000 osd.4 (osd.4) 2021-03-15T22:28:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: 122 : cluster [DBG] 5.a scrub ok 2021-03-15T22:28:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: cluster 2021-03-15T22:28:51.115176+0000 mgr.y (mgr.14144) 778 : cluster [DBG] pgmap v1016: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 554 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 2.0 KiB/s rd, 1.2 MiB/s wr, 6 op/s; 114/1740 objects degraded (6.552%); 404/1740 objects misplaced (23.218%); 3.3 MiB/s, 2 objects/s recovering 2021-03-15T22:28:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: cluster 2021-03-15T22:28:51.193951+0000 mon.a (mon 2021-03-15T22:28:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: .0) 1623 : cluster [DBG] osdmap e582: 8 total, 6 up, 7 in 2021-03-15T22:28:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:28:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: 03-15T22:28:51.825147+0000 2021-03-15T22:28:52.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: osd.0 (osd.0) 86 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:28:52.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: 2021-03-15T22:28:52.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:28:52.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: 28:52.086639+0000 mon.a (mon.0 2021-03-15T22:28:52.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: ) 1624 : audit [INF] from='client.? 172.21.15.76:0/2570303318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:28:52.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:28:52.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: 15T22:28:52.104311+0000 mon.a 2021-03-15T22:28:52.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:52 smithi125 conmon[56739]: (mon.0) 1625 : audit [DBG] from='client.? 172.21.15.76:0/3554586440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:28:52.553 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:52.576 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:52.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:52.578 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:52.679 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:52.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:52.702 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:52.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:52.804 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:52.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:52.828 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:52.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:52.930 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:52.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:52.955 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:52.956 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:53.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:53.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:53.082 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:53.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:53.184 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:53.212 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:53.213 INFO:tasks.rados.rados.0.smithi076.stdout:2106: done (5 left) 2021-03-15T22:28:53.214 INFO:tasks.rados.rados.0.smithi076.stdout:2107: read oid 12 snap 237 2021-03-15T22:28:53.214 INFO:tasks.rados.rados.0.smithi076.stdout:2107: expect deleted 2021-03-15T22:28:53.214 INFO:tasks.rados.rados.0.smithi076.stdout:2108: snap_create 2021-03-15T22:28:53.215 INFO:tasks.rados.rados.0.smithi076.stdout:2100: finishing write tid 1 to smithi07694878-49 2021-03-15T22:28:53.215 INFO:tasks.rados.rados.0.smithi076.stdout:2103: expect (ObjNum 674 snap 214 seq_num 674) 2021-03-15T22:28:53.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:53.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:53.317 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:53.330 INFO:tasks.rados.rados.0.smithi076.stdout:2100: finishing write tid 2 to smithi07694878-49 2021-03-15T22:28:53.331 INFO:tasks.rados.rados.0.smithi076.stdout:2100: finishing write tid 3 to smithi07694878-49 2021-03-15T22:28:53.331 INFO:tasks.rados.rados.0.smithi076.stdout:2100: finishing write tid 4 to smithi07694878-49 2021-03-15T22:28:53.331 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 1404 (ObjNum 733 snap 239 seq_num 733) dirty exists 2021-03-15T22:28:53.331 INFO:tasks.rados.rados.0.smithi076.stdout:2100: left oid 49 (ObjNum 733 snap 239 seq_num 733) 2021-03-15T22:28:53.332 INFO:tasks.rados.rados.0.smithi076.stdout:2105: finishing write tid 1 to smithi07694878-42 2021-03-15T22:28:53.332 INFO:tasks.rados.rados.0.smithi076.stdout:2102: expect (ObjNum 667 snap 211 seq_num 667) 2021-03-15T22:28:53.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:53.339 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:53.340 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:53.378 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[56514]: cluster 2021-03-15T22:28:51.588447+0000 osd.1 ( 2021-03-15T22:28:53.378 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[56514]: osd.1) 113 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:28:53.378 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[56514]: cluster 2021-03-15T22:28:51.970586+0000 osd.4 (osd.4) 123 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:28:53.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[56514]: cluster 2021-03-15T22:28:52.152870+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:28:53.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[56514]: cluster 2021-03-15T22:28:52.189268+0000 mon.a (mon.0) 1626 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:53.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[56514]: audit 2021-03-15T22:28:52.198528+0000 mon.a (mon.0) 1627 : audit [INF] from='client.? 172.21.15.76:0/2570303318' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:28:53.379 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[56514]: cluster 2021-03-15T22:28:52.198589+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e583: 8 total, 6 up, 7 in 2021-03-15T22:28:53.380 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[62156]: cluster 2021-03-15T22:28:51.588447+0000 osd.1 (osd.1) 113 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:28:53.380 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[62156]: cluster 2021-03-15T22:28:51.970586+0000 osd.4 (osd.4) 123 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:28:53.380 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[62156]: cluster 2021-03-15T22:28:52.152870+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:28:53.380 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[62156]: cluster 2021-03-15T22:28:52.189268+0000 mon.a (mon.0) 1626 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:53.381 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[62156]: audit 2021-03-15T22:28:52.198528+0000 mon.a (mon.0) 1627 : audit [INF] from='client.? 172.21.15.76:0/2570303318' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:28:53.381 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:53 smithi076 conmon[62156]: cluster 2021-03-15T22:28:52.198589+0000 mon.a (mon.0) 1628 : cluster [DBG] osdmap e583: 8 total, 6 up, 7 in 2021-03-15T22:28:53.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:53.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:53.464 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:53.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:53.467 INFO:tasks.rados.rados.0.smithi076.stdout:2105: finishing write tid 2 to smithi07694878-42 2021-03-15T22:28:53.467 INFO:tasks.rados.rados.0.smithi076.stdout:2105: finishing write tid 3 to smithi07694878-42 2021-03-15T22:28:53.467 INFO:tasks.rados.rados.0.smithi076.stdout:2105: finishing write tid 4 to smithi07694878-42 2021-03-15T22:28:53.467 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 1500 (ObjNum 735 snap 239 seq_num 735) dirty exists 2021-03-15T22:28:53.468 INFO:tasks.rados.rados.0.smithi076.stdout:2105: left oid 42 (ObjNum 735 snap 239 seq_num 735) 2021-03-15T22:28:53.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: cluster 2021-03-15T22:28:51.588447+0000 osd.1 (osd.1) 113 : cluster 2021-03-15T22:28:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: [DBG] 6.4 scrub ok 2021-03-15T22:28:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: cluster 2021-03-15T22:28:51.970586+0000 osd.4 (osd.4) 123 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:28:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:28:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: 28:52.152870+0000 osd.7 (osd.7) 114 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:28:53.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:28:53.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: -15T22:28:52.189268+0000 mon.a (mon.0) 1626 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:28:53.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:28:53.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: T22:28:52.198528+0000 mon.a (mon.0) 1627 : audit 2021-03-15T22:28:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/2570303318' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:28:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: cluster 2021-03-15T22:28:52.198589+0000 mon.a (mon. 2021-03-15T22:28:53.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:53 smithi125 conmon[56739]: 0) 1628 : cluster [DBG] osdmap e583: 8 total, 6 up, 7 in 2021-03-15T22:28:53.566 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:53.588 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:53.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:53.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:53.690 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:53.713 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:53.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:53.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:53.816 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:53.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:53.839 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:53.839 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:28:53.871 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:28:53.940 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:53.962 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:53.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:53.964 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:54.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:54.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:54.088 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:54.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:54.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:54.212 INFO:tasks.rados.rados.0.smithi076.stdout:2100: done (6 left) 2021-03-15T22:28:54.212 INFO:tasks.rados.rados.0.smithi076.stdout:2102: done (5 left) 2021-03-15T22:28:54.213 INFO:tasks.rados.rados.0.smithi076.stdout:2103: done (4 left) 2021-03-15T22:28:54.213 INFO:tasks.rados.rados.0.smithi076.stdout:2104: done (3 left) 2021-03-15T22:28:54.213 INFO:tasks.rados.rados.0.smithi076.stdout:2105: done (2 left) 2021-03-15T22:28:54.213 INFO:tasks.rados.rados.0.smithi076.stdout:2107: done (1 left) 2021-03-15T22:28:54.214 INFO:tasks.rados.rados.0.smithi076.stdout:2108: done (0 left) 2021-03-15T22:28:54.214 INFO:tasks.rados.rados.0.smithi076.stdout:2109: snap_create 2021-03-15T22:28:54.214 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:54.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:54.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:54.317 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:54.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:54.339 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:54.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:54.440 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:54.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:54.465 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:54.466 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:54.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[56514]: cluster 2021-03-15T22:28:53.115862+0000 mgr.y (mgr.14144 2021-03-15T22:28:54.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[56514]: ) 779 : cluster [DBG] pgmap v1019: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 550 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 10 op/s; 114/1737 objects degraded (6.563%); 406/1737 objects misplaced (23.374%) 2021-03-15T22:28:54.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[56514]: cluster 2021-03-15T22:28:53.207733+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e584: 8 total, 6 up, 7 in 2021-03-15T22:28:54.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[56514]: cluster 2021-03-15T22:28:53.439924+0000 osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:54.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[56514]: cluster 2021-03-15T22:28:53.440596+0000 osd.7 (osd.7) 116 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:54.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[62156]: cluster 2021-03-15T22:28:53.115862+0000 mgr.y (mgr.14144 2021-03-15T22:28:54.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[62156]: ) 779 : cluster [DBG] pgmap v1019: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 550 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 10 op/s; 114/1737 objects degraded (6.563%); 406/1737 objects misplaced (23.374%) 2021-03-15T22:28:54.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[62156]: cluster 2021-03-15T22:28:53.207733+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e584: 8 total, 6 up, 7 in 2021-03-15T22:28:54.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[62156]: cluster 2021-03-15T22:28:53.439924+0000 2021-03-15T22:28:54.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[62156]: osd.7 (osd.7) 115 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:54.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:28:54.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[62156]: 03-15T22:28:53.440596+0000 osd.7 (osd.7 2021-03-15T22:28:54.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:54 smithi076 conmon[62156]: ) 116 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:54.518 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 0] out_osds: [3] dead_osds: [2, 5] live_osds: [0, 1, 3, 4, 6, 7] 2021-03-15T22:28:54.518 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:28:54.519 INFO:tasks.thrashosds.thrasher:Adding osd 3 2021-03-15T22:28:54.520 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd in 3 2021-03-15T22:28:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:54 smithi125 conmon[56739]: cluster 2021-03-15T22:28:53.115862 2021-03-15T22:28:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:54 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 779 : cluster [DBG] pgmap v1019: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 550 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 1.5 MiB/s rd, 1.2 MiB/s wr, 10 op/s; 114/1737 objects degraded (6.563%); 406/1737 objects misplaced (23.374%) 2021-03-15T22:28:54.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:54 smithi125 conmon[56739]: cluster 2021-03-15T22:28: 2021-03-15T22:28:54.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:54 smithi125 conmon[56739]: 53.207733+0000 mon.a (mon.0) 1629 : cluster [DBG] osdmap e584: 8 total, 6 up, 7 in 2021-03-15T22:28:54.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:54 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:28:54.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:54 smithi125 conmon[56739]: -03-15T22:28:53.439924+0000 2021-03-15T22:28:54.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:54 smithi125 conmon[56739]: osd.7 (osd.7) 115 : cluster [DBG] 2021-03-15T22:28:54.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:54 smithi125 conmon[56739]: purged_snaps scrub starts 2021-03-15T22:28:54.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:54 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:28:54.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:54 smithi125 conmon[56739]: 22:28:53.440596+0000 osd.7 (osd. 2021-03-15T22:28:54.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:54 smithi125 conmon[56739]: 7) 116 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:54.567 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:54.591 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:54.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:54.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:54.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:54.718 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:54.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:54.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:54.755 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:28:54.755 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:28:54.807 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:54.821 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:54.849 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:54.850 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:54.951 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:54.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:54.977 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:54.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:55.079 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:55.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:55.103 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:55.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:55.205 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:55.222 INFO:tasks.rados.rados.0.smithi076.stdout:2109: done (0 left) 2021-03-15T22:28:55.222 INFO:tasks.rados.rados.0.smithi076.stdout:2110: read oid 14 snap -1 2021-03-15T22:28:55.222 INFO:tasks.rados.rados.0.smithi076.stdout:2110: expect deleted 2021-03-15T22:28:55.223 INFO:tasks.rados.rados.0.smithi076.stdout:2111: read oid 32 snap -1 2021-03-15T22:28:55.223 INFO:tasks.rados.rados.0.smithi076.stdout:2111: expect (ObjNum 717 snap 233 seq_num 717) 2021-03-15T22:28:55.223 INFO:tasks.rados.rados.0.smithi076.stdout:2112: read oid 24 snap 212 2021-03-15T22:28:55.224 INFO:tasks.rados.rados.0.smithi076.stdout:2112: expect (ObjNum 638 snap 200 seq_num 638) 2021-03-15T22:28:55.224 INFO:tasks.rados.rados.0.smithi076.stdout:2113: copy_from oid 29 from oid 36 current snap is 241 2021-03-15T22:28:55.224 INFO:tasks.rados.rados.0.smithi076.stdout:2114: read oid 15 snap -1 2021-03-15T22:28:55.224 INFO:tasks.rados.rados.0.smithi076.stdout:2114: expect (ObjNum 714 snap 231 seq_num 714) 2021-03-15T22:28:55.225 INFO:tasks.rados.rados.0.smithi076.stdout:2115: snap_create 2021-03-15T22:28:55.229 INFO:tasks.rados.rados.0.smithi076.stdout:2111: expect (ObjNum 717 snap 233 seq_num 717) 2021-03-15T22:28:55.231 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:55.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:55.233 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:55.313 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:28:55.334 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:55.345 INFO:tasks.rados.rados.0.smithi076.stdout:2112: expect (ObjNum 638 snap 200 seq_num 638) 2021-03-15T22:28:55.367 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:55.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:55.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:55.438 INFO:tasks.rados.rados.0.smithi076.stdout:2114: expect (ObjNum 714 snap 231 seq_num 714) 2021-03-15T22:28:55.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:55.493 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:55.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:55.495 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:55.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[56514]: cluster 2021-03-15T22:28:54.210135+0000 mon.a (mon.0) 1630 : cluster [DBG] osdmap e585: 8 total, 6 up, 7 in 2021-03-15T22:28:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[56514]: cluster 2021-03-15T22:28:55.171502+0000 mon.a (mon.0) 1631 : cluster [WRN] Health check update: Degraded data redundancy: 114/1737 objects degraded (6.563%), 12 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:55.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: cluster 2021-03-15T22:28:55.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 2021-03-15T22:28:55.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 2021 2021-03-15T22:28:55.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: - 2021-03-15T22:28:55.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 03 2021-03-15T22:28:55.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: - 2021-03-15T22:28:55.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 15 2021-03-15T22:28:55.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: T 2021-03-15T22:28:55.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 22 2021-03-15T22:28:55.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: : 2021-03-15T22:28:55.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 28 2021-03-15T22:28:55.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: : 2021-03-15T22:28:55.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 54 2021-03-15T22:28:55.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: . 2021-03-15T22:28:55.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 210135 2021-03-15T22:28:55.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: +0000 2021-03-15T22:28:55.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 2021-03-15T22:28:55.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: mon.a 2021-03-15T22:28:55.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: ( 2021-03-15T22:28:55.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: mon 2021-03-15T22:28:55.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: . 2021-03-15T22:28:55.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 0 2021-03-15T22:28:55.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: ) 2021-03-15T22:28:55.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 1630 2021-03-15T22:28:55.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: : 2021-03-15T22:28:55.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: cluster 2021-03-15T22:28:55.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 2021-03-15T22:28:55.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: [DBG] 2021-03-15T22:28:55.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 2021-03-15T22:28:55.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: osdmap e585: 8 total, 6 up, 7 in 2021-03-15T22:28:55.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:28:55.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: 28:55.171502+0000 mon.a (mon 2021-03-15T22:28:55.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: .0) 1631 : cluster [WRN] 2021-03-15T22:28:55.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:55 smithi076 conmon[62156]: Health check update: Degraded data redundancy: 114/1737 objects degraded (6.563%), 12 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:55.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:55 smithi125 conmon[56739]: cluster 2021-03-15T22:28:54.210135+0000 mon.a (mon. 2021-03-15T22:28:55.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:55 smithi125 conmon[56739]: 0) 1630 : cluster [DBG] osdmap e585: 8 total, 6 up, 7 in 2021-03-15T22:28:55.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:55 smithi125 conmon[56739]: cluster 2021-03-15T22:28:55.171502+0000 mon.a (mon.0) 1631 : cluster [WRN] Health check update: Degraded data redundancy: 114/1737 objects degraded (6.563%), 12 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:28:55.538 INFO:tasks.rados.rados.0.smithi076.stdout:2113: finishing copy_from to smithi07694878-29 2021-03-15T22:28:55.538 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 440 (ObjNum 387 snap 104 seq_num 387) dirty exists 2021-03-15T22:28:55.538 INFO:tasks.rados.rados.0.smithi076.stdout:2113: finishing copy_from racing read to smithi07694878-29 2021-03-15T22:28:55.596 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:55.617 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:55.619 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:55.677 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:28:55.720 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:55.744 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:55.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:55.745 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:55.846 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:55.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:55.873 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:55.873 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:55.874 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:28:55.889 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:55.975 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:55.998 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:55.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:56.000 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:56.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:56.125 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:56.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:56.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:56.224 INFO:tasks.rados.rados.0.smithi076.stdout:2110: done (5 left) 2021-03-15T22:28:56.225 INFO:tasks.rados.rados.0.smithi076.stdout:2111: done (4 left) 2021-03-15T22:28:56.225 INFO:tasks.rados.rados.0.smithi076.stdout:2112: done (3 left) 2021-03-15T22:28:56.225 INFO:tasks.rados.rados.0.smithi076.stdout:2113: done (2 left) 2021-03-15T22:28:56.226 INFO:tasks.rados.rados.0.smithi076.stdout:2114: done (1 left) 2021-03-15T22:28:56.226 INFO:tasks.rados.rados.0.smithi076.stdout:2115: done (0 left) 2021-03-15T22:28:56.226 INFO:tasks.rados.rados.0.smithi076.stdout:2116: write oid 33 current snap is 242 2021-03-15T22:28:56.226 INFO:tasks.rados.rados.0.smithi076.stdout:2116: seq_num 736 ranges {769952=553152,2069100=300852} 2021-03-15T22:28:56.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:56.232 INFO:tasks.rados.rados.0.smithi076.stdout:2116: writing smithi07694878-33 from 769952 to 1323104 tid 1 2021-03-15T22:28:56.238 INFO:tasks.rados.rados.0.smithi076.stdout:2116: writing smithi07694878-33 from 2069100 to 2369952 tid 2 2021-03-15T22:28:56.238 INFO:tasks.rados.rados.0.smithi076.stdout:2117: rollback oid 4 current snap is 242 2021-03-15T22:28:56.239 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 4 to 218 2021-03-15T22:28:56.239 INFO:tasks.rados.rados.0.smithi076.stdout:2116: finishing write tid 1 to smithi07694878-33 2021-03-15T22:28:56.239 INFO:tasks.rados.rados.0.smithi076.stdout:2118: snap_remove snap 162 2021-03-15T22:28:56.253 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:56.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:56.255 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:56.356 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:56.378 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:56.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:56.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:56.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:56.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[56514]: cluster 2021-03-15T22:28:55.116511+0000 mgr.y (mgr.14144) 780 : cluster [DBG] pgmap v1022: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 546 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 692 KiB/s wr, 9 op/s; 114/1734 objects degraded (6.574%); 405/1734 objects misplaced (23.356%) 2021-03-15T22:28:56.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[56514]: cluster 2021-03-15T22:28:55.219655+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e586: 8 total, 6 up, 7 in 2021-03-15T22:28:56.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[56514]: cluster 2021-03-15T22:28:55.734701+0000 osd.0 (osd.0) 87 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:56.498 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[56514]: cluster 2021-03-15T22:28:55.735244+0000 osd.0 (osd.0) 88 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:56.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[62156]: cluster 2021-03-15T22:28:55 2021-03-15T22:28:56.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[62156]: .116511+0000 mgr.y (mgr.14144) 780 : cluster [DBG] pgmap v1022: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 546 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 692 KiB/s wr, 9 op/s; 114/1734 objects degraded (6.574%); 405/1734 objects misplaced (23.356%) 2021-03-15T22:28:56.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:28:56.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[62156]: -03-15T22 2021-03-15T22:28:56.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[62156]: :28:55.219655+0000 mon.a (mon. 2021-03-15T22:28:56.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[62156]: 0) 1632 : cluster [DBG] osdmap e586: 8 total, 6 up, 7 in 2021-03-15T22:28:56.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[62156]: 2021-03-15T22:28:56.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[62156]: cluster 2021-03-15T22:28 2021-03-15T22:28:56.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[62156]: :55.734701+0000 osd.0 (osd.0) 87 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:56.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:28:56.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:56 smithi076 conmon[62156]: :28:55.735244+0000 osd.0 (osd.0) 88 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:56.502 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:56.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:56.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:56 smithi125 conmon[56739]: cluster 2021-03-15T22:28:55.116511+0000 mgr.y (mgr.14144) 780 : cluster 2021-03-15T22:28:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:56 smithi125 conmon[56739]: [DBG] pgmap v1022: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 546 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 4.6 MiB/s rd, 692 KiB/s wr, 9 op/s; 114/1734 objects degraded (6.574%); 405/1734 objects misplaced (23.356%) 2021-03-15T22:28:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:56 smithi125 conmon[56739]: cluster 2021-03-15T22:28:55.219655+0000 mon.a (mon.0) 1632 : cluster [DBG] osdmap e586: 8 total, 6 up, 7 in 2021-03-15T22:28:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:56 smithi125 conmon[56739]: cluster 2021-03-15T22:28: 2021-03-15T22:28:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:56 smithi125 conmon[56739]: 55.734701+0000 osd.0 (osd.0) 87 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:28:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:56 smithi125 conmon[56739]: cluster 2021-03-15T22:28:56.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:56 smithi125 conmon[56739]: 2021-03-15T22:28:55.735244+0000 osd.0 (osd.0) 2021-03-15T22:28:56.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:56 smithi125 conmon[56739]: 88 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:28:56.605 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:56.627 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:56.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:56.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:56.656 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:56.719 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:28:56.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:56.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:56.762 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:56.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:56.863 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:56.887 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:56.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:56.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:56.990 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:28:57.014 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:57.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:57.015 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:28:57.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:57.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:57.141 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:57.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:57.243 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:57.245 INFO:tasks.rados.rados.0.smithi076.stdout:2118: done (2 left) 2021-03-15T22:28:57.245 INFO:tasks.rados.rados.0.smithi076.stdout:2119: read oid 5 snap -1 2021-03-15T22:28:57.245 INFO:tasks.rados.rados.0.smithi076.stdout:2119: expect deleted 2021-03-15T22:28:57.246 INFO:tasks.rados.rados.0.smithi076.stdout:2120: snap_create 2021-03-15T22:28:57.246 INFO:tasks.rados.rados.0.smithi076.stdout:2116: finishing write tid 2 to smithi07694878-33 2021-03-15T22:28:57.246 INFO:tasks.rados.rados.0.smithi076.stdout:2117: finishing rollback tid 0 to smithi07694878-4 2021-03-15T22:28:57.247 INFO:tasks.rados.rados.0.smithi076.stdout:2117: finishing rollback tid 1 to smithi07694878-4 2021-03-15T22:28:57.247 INFO:tasks.rados.rados.0.smithi076.stdout:2116: finishing write tid 3 to smithi07694878-33 2021-03-15T22:28:57.247 INFO:tasks.rados.rados.0.smithi076.stdout:2117: finishing rollback tid 2 to smithi07694878-4 2021-03-15T22:28:57.247 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 572 (ObjNum 624 snap 196 seq_num 624) dirty exists 2021-03-15T22:28:57.248 INFO:tasks.rados.rados.0.smithi076.stdout:2116: finishing write tid 4 to smithi07694878-33 2021-03-15T22:28:57.248 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 858 (ObjNum 736 snap 242 seq_num 736) dirty exists 2021-03-15T22:28:57.248 INFO:tasks.rados.rados.0.smithi076.stdout:2116: left oid 33 (ObjNum 736 snap 242 seq_num 736) 2021-03-15T22:28:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:57.270 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:57.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:57.372 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:57.394 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:57.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:57.396 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:57.497 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:57.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:57 smithi076 conmon[56514]: cluster 2021-03-15T22:28:56.220007+0000 mon.a (mon 2021-03-15T22:28:57.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:57 smithi076 conmon[56514]: .0) 1633 : cluster [DBG] osdmap e587: 8 total, 6 up, 7 in 2021-03-15T22:28:57.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:57 smithi076 conmon[62156]: cluster 2021-03-15T22:28:56.220007+0000 2021-03-15T22:28:57.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:57 smithi076 conmon[62156]: mon.a (mon.0) 1633 : cluster [DBG] osdmap e587: 8 total, 6 up, 7 in 2021-03-15T22:28:57.518 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:57.520 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:57.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:57 smithi125 conmon[56739]: cluster 2021-03-15T22:28:56.220007+0000 mon.a ( 2021-03-15T22:28:57.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:57 smithi125 conmon[56739]: mon.0) 1633 : cluster [DBG] osdmap e587: 8 total, 6 up, 7 in 2021-03-15T22:28:57.621 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:57.643 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:57.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:57.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:57.746 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:57.768 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:57.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:57.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:57.870 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:57.893 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:57.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:57.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:57.900 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:57 smithi076 conmon[56514]: debug 2021-03-15T22:28:57.624+0000 7f0537c88700 -1 mon.a@0(leader).osd e588 definitely_dead 0 2021-03-15T22:28:57.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:58.022 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:58.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:58.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:58.125 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:58.147 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:58.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:58.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:58.242 INFO:tasks.rados.rados.0.smithi076.stdout:2116: done (3 left) 2021-03-15T22:28:58.242 INFO:tasks.rados.rados.0.smithi076.stdout:2117: done (2 left) 2021-03-15T22:28:58.242 INFO:tasks.rados.rados.0.smithi076.stdout:2119: done (1 left) 2021-03-15T22:28:58.243 INFO:tasks.rados.rados.0.smithi076.stdout:2120: done (0 left) 2021-03-15T22:28:58.243 INFO:tasks.rados.rados.0.smithi076.stdout:2121: read oid 49 snap -1 2021-03-15T22:28:58.243 INFO:tasks.rados.rados.0.smithi076.stdout:2121: expect (ObjNum 733 snap 239 seq_num 733) 2021-03-15T22:28:58.243 INFO:tasks.rados.rados.0.smithi076.stdout:2122: write oid 40 current snap is 243 2021-03-15T22:28:58.244 INFO:tasks.rados.rados.0.smithi076.stdout:2122: seq_num 737 ranges {786759=628401,2011229=375530} 2021-03-15T22:28:58.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:58.251 INFO:tasks.rados.rados.0.smithi076.stdout:2122: writing smithi07694878-40 from 786759 to 1415160 tid 1 2021-03-15T22:28:58.259 INFO:tasks.rados.rados.0.smithi076.stdout:2122: writing smithi07694878-40 from 2011229 to 2386759 tid 2 2021-03-15T22:28:58.260 INFO:tasks.rados.rados.0.smithi076.stdout:2123: write (excl) oid 45 current snap is 243 2021-03-15T22:28:58.260 INFO:tasks.rados.rados.0.smithi076.stdout:2123: seq_num 738 ranges {403566=420003,1386064=546918} 2021-03-15T22:28:58.260 INFO:teuthology.orchestra.run.smithi076.stderr:marked in osd.3. 2021-03-15T22:28:58.265 INFO:tasks.rados.rados.0.smithi076.stdout:2123: writing smithi07694878-45 from 403566 to 823569 tid 1 2021-03-15T22:28:58.275 INFO:tasks.rados.rados.0.smithi076.stdout:2123: writing smithi07694878-45 from 1386064 to 1932982 tid 2 2021-03-15T22:28:58.276 INFO:tasks.rados.rados.0.smithi076.stdout:2124: snap_create 2021-03-15T22:28:58.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:58.292 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:58.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:58.394 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:58.415 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:58.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:58.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:58.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:58 smithi076 conmon[56514]: cluster 2021-03-15T22:28:57.117117+0000 mgr.y (mgr.14144) 781 : cluster [DBG] pgmap v1025: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 547 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.7 MiB/s rd, 282 KiB/s wr, 9 op/s; 114/1737 objects degraded (6.563%); 406/1737 objects misplaced (23.374%) 2021-03-15T22:28:58.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:58 smithi076 conmon[56514]: cluster 2021-03-15T22:28:57.233887+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e588: 8 total, 6 up, 7 in 2021-03-15T22:28:58.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:58 smithi076 conmon[56514]: audit 2021-03-15T22:28:57.626449+0000 mon.a (mon.0) 1635 : audit [DBG] from='client.? 172.21.15.76:0/323829313' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2021-03-15T22:28:58.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:58 smithi076 conmon[62156]: cluster 2021-03-15T22:28:57.117117+0000 mgr.y (mgr.14144) 781 : cluster [DBG] pgmap v1025: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 547 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.7 MiB/s rd, 282 KiB/s wr, 9 op/s; 114/1737 objects degraded (6.563%); 406/1737 objects misplaced (23.374%) 2021-03-15T22:28:58.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:58 smithi076 conmon[62156]: cluster 2021-03-15T22:28:57.233887+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e588: 8 total, 6 up, 7 in 2021-03-15T22:28:58.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:58 smithi076 conmon[62156]: audit 2021-03-15T22:28:57.626449+0000 mon.a (mon.0) 1635 : audit [DBG] from='client.? 172.21.15.76:0/323829313' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2021-03-15T22:28:58.523 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:28:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:58 smithi125 conmon[56739]: cluster 2021-03-15T22:28:57.117117+0000 mgr.y (mgr.14144) 781 : cluster [DBG] pgmap v1025: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 547 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.7 MiB/s rd, 282 KiB/s wr, 9 op/s; 114/1737 objects degraded (6.563%); 406/1737 objects misplaced (23.374%) 2021-03-15T22:28:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:58 smithi125 conmon[56739]: cluster 2021-03-15T22:28:57. 2021-03-15T22:28:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:58 smithi125 conmon[56739]: 233887+0000 mon.a (mon.0) 1634 : cluster [DBG] osdmap e588: 8 total, 6 up, 7 in 2021-03-15T22:28:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:58 smithi125 conmon[56739]: audit 2021-03-15T22:28:57.626449+0000 mon.a (mon.0) 1635 : audit [DBG] from='client.? 172.21.15.76:0/323829313' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2021-03-15T22:28:58.544 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:58.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:58.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:28:58.647 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:58.669 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:58.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:58.670 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:58.771 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:58.795 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:58.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:58.797 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:58.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:58.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:58.923 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:58.923 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:59.025 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:59.048 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:59.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:59.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:59.151 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:28:59.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:59.175 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:59.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:28:59.249 INFO:tasks.rados.rados.0.smithi076.stdout:2124: done (3 left) 2021-03-15T22:28:59.250 INFO:tasks.rados.rados.0.smithi076.stdout:2125: snap_remove snap 238 2021-03-15T22:28:59.258 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:28:59.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:59.305 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:59.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:59.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:59.408 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:28:59.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:59.431 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:28:59.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:28:59.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[56514]: audit 2021-03-15T22:28:58.239081+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? 172.21.15.76:0/323829313' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2021-03-15T22:28:59.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[56514]: 2021-03-15T22:28:59.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[56514]: cluster 2021-03-15T22:28:58.239145+0000 mon.a (mon.0) 1637 : cluster [DBG] osdmap e589: 8 total, 6 up, 8 in 2021-03-15T22:28:59.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[56514]: audit 2021-03-15T22:28:58.546845+0000 mon.a (mon.0) 1638 : audit [INF] from='client.? 172.21.15.76:0/2616572085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:59.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[62156]: audit 2021-03-15T22:28:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[62156]: 2021-03-15T22:28:58. 2021-03-15T22:28:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[62156]: 239081+0000 mon.a (mon.0) 1636 : audit [INF] from='client.? 172.21.15.76:0/323829313' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2021-03-15T22:28:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[62156]: cluster 2021-03-15T22:28:58. 2021-03-15T22:28:59.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[62156]: 239145+0000 mon.a (mon.0) 1637 : cluster [DBG] osdmap e589: 8 total, 6 up, 8 in 2021-03-15T22:28:59.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[62156]: 2021-03-15T22:28:59.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[62156]: audit 2021-03-15T22:28:59.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[62156]: 2021-03-15T22:28:58.546845+0000 mon.a (mon.0) 1638 : audit [INF] 2021-03-15T22:28:59.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:28:59 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2616572085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:59.533 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:59.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:59 smithi125 conmon[56739]: audit 2021-03-15T22:28:58.239081+0000 mon.a (mon.0) 1636 : audit 2021-03-15T22:28:59.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:59 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/323829313' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2021-03-15T22:28:59.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:59 smithi125 conmon[56739]: cluster 2021-03-15T22:28:58.239145+0000 mon.a (mon.0) 1637 : cluster [DBG] osdmap e589: 8 total, 6 up, 8 in 2021-03-15T22:28:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:59 smithi125 conmon[56739]: audit 2021-03-15T22:28:58.546845+0000 2021-03-15T22:28:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:59 smithi125 conmon[56739]: mon.a (mon.0) 1638 : audit 2021-03-15T22:28:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:28:59 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/2616572085' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:28:59.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:59.556 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:59.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:59.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:59.680 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:59.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:59.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:59.782 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:28:59.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:59.805 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:59.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:28:59.907 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:28:59.929 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:28:59.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:28:59.931 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:28:59.972 INFO:tasks.thrashosds.thrasher:Added osd 3 2021-03-15T22:29:00.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:00.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:00.055 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:00.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:00.157 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:00.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:00.186 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:00.186 INFO:tasks.rados.rados.0.smithi076.stdout:2125: done (3 left) 2021-03-15T22:29:00.186 INFO:tasks.rados.rados.0.smithi076.stdout:2126: write oid 23 current snap is 244 2021-03-15T22:29:00.187 INFO:tasks.rados.rados.0.smithi076.stdout:2126: seq_num 739 ranges {420373=495252,1444546=690576} 2021-03-15T22:29:00.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:00.188 INFO:tasks.rados.rados.0.smithi076.stdout:2126: writing smithi07694878-23 from 420373 to 915625 tid 1 2021-03-15T22:29:00.201 INFO:tasks.rados.rados.0.smithi076.stdout:2126: writing smithi07694878-23 from 1444546 to 2135122 tid 2 2021-03-15T22:29:00.204 INFO:tasks.rados.rados.0.smithi076.stdout:2122: finishing write tid 1 to smithi07694878-40 2021-03-15T22:29:00.205 INFO:tasks.rados.rados.0.smithi076.stdout:2122: finishing write tid 2 to smithi07694878-40 2021-03-15T22:29:00.205 INFO:tasks.rados.rados.0.smithi076.stdout:2122: finishing write tid 3 to smithi07694878-40 2021-03-15T22:29:00.205 INFO:tasks.rados.rados.0.smithi076.stdout:2122: finishing write tid 4 to smithi07694878-40 2021-03-15T22:29:00.205 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1506 (ObjNum 737 snap 243 seq_num 737) dirty exists 2021-03-15T22:29:00.205 INFO:tasks.rados.rados.0.smithi076.stdout:2122: left oid 40 (ObjNum 737 snap 243 seq_num 737) 2021-03-15T22:29:00.206 INFO:tasks.rados.rados.0.smithi076.stdout:2121: expect (ObjNum 733 snap 239 seq_num 733) 2021-03-15T22:29:00.286 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:28:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:29:00.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:00.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:00.312 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:00.312 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:00.366 INFO:tasks.rados.rados.0.smithi076.stdout:2121: done (3 left) 2021-03-15T22:29:00.367 INFO:tasks.rados.rados.0.smithi076.stdout:2122: done (2 left) 2021-03-15T22:29:00.367 INFO:tasks.rados.rados.0.smithi076.stdout:2127: write oid 49 current snap is 244 2021-03-15T22:29:00.367 INFO:tasks.rados.rados.0.smithi076.stdout:2126: finishing write tid 1 to smithi07694878-23 2021-03-15T22:29:00.367 INFO:tasks.rados.rados.0.smithi076.stdout:2126: finishing write tid 2 to smithi07694878-23 2021-03-15T22:29:00.368 INFO:tasks.rados.rados.0.smithi076.stdout:2126: finishing write tid 3 to smithi07694878-23 2021-03-15T22:29:00.368 INFO:tasks.rados.rados.0.smithi076.stdout:2127: seq_num 740 ranges {437180=570501,1503028=550587} 2021-03-15T22:29:00.371 INFO:tasks.rados.rados.0.smithi076.stdout:2127: writing smithi07694878-49 from 437180 to 1007681 tid 1 2021-03-15T22:29:00.379 INFO:tasks.rados.rados.0.smithi076.stdout:2127: writing smithi07694878-49 from 1503028 to 2053615 tid 2 2021-03-15T22:29:00.379 INFO:tasks.rados.rados.0.smithi076.stdout:2128: snap_remove snap 243 2021-03-15T22:29:00.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:00.437 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:00.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:00.439 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:00.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[56514]: cluster 2021-03-15T22:28:59.117729+0000 mgr.y (mgr.14144) 782 : cluster [DBG] pgmap v1028: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 550 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 893 KiB/s wr, 9 op/s; 114/1737 objects degraded (6.563%); 407/1737 objects misplaced (23.431%) 2021-03-15T22:29:00.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[56514]: cluster 2021-03-15T22:28:59.234376+0000 mon.a (mon.0) 1639 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:00.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[56514]: audit 2021-03-15T22:28:59.242613+0000 mon.a (mon.0) 1640 : audit [INF] from='client.? 172.21.15.76:0/2616572085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:00.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[56514]: cluster 2021-03-15T22:28:59.242669+0000 mon.a (mon.0) 1641 : cluster [DBG] osdmap e590: 8 total, 6 up, 8 in 2021-03-15T22:29:00.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[56514]: audit 2021-03-15T22:28:59.904334+0000 mgr.y (mgr.14144) 783 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:00.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[56514]: cluster 2021-03-15T22:29:00.178609+0000 mon.a (mon.0) 1642 : cluster [DBG] osdmap e591: 8 total, 6 up, 8 in 2021-03-15T22:29:00.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: cluster 2021-03-15T22:28:59.117729 2021-03-15T22:29:00.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 782 : cluster [DBG] pgmap v1028: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 550 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 893 KiB/s wr, 9 op/s; 114/1737 objects degraded (6.563%); 407/1737 objects misplaced (23.431%) 2021-03-15T22:29:00.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: cluster 2021-03-15T22:28:59.234376+0000 mon.a 2021-03-15T22:29:00.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: (mon.0) 1639 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:00.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: audit 2021-03-15T22:29:00.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: 2021-03-15T22:28:59.242613+0000 mon.a (mon.0 2021-03-15T22:29:00.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: ) 1640 : audit [INF] from='client.? 172.21.15.76:0/2616572085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:00.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:29:00.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: 28:59.242669+0000 mon.a (mon.0) 1641 2021-03-15T22:29:00.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: : cluster [DBG] osdmap e590: 8 total, 6 up, 8 in 2021-03-15T22:29:00.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: audit 2021-03-15T22:28: 2021-03-15T22:29:00.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: 59.904334+0000 mgr.y (mgr.14144) 783 : audit 2021-03-15T22:29:00.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:00.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: cluster 2021-03-15T22:29:00. 2021-03-15T22:29:00.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: 178609+0000 mon.a (mon.0) 1642 : cluster [DBG] 2021-03-15T22:29:00.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:00 smithi076 conmon[62156]: osdmap e591: 8 total, 6 up, 8 in 2021-03-15T22:29:00.541 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:00.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:00.564 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:00.565 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:00.666 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:00.688 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:00.689 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:00.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: cluster 2021-03-15T22:28:59. 2021-03-15T22:29:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: 117729+0000 mgr.y (mgr.14144) 782 : cluster [DBG] pgmap v1028: 138 pgs: 45 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 22 active+undersized, 42 active+clean; 550 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 6.1 MiB/s rd, 893 KiB/s wr, 9 op/s; 114/1737 objects degraded (6.563%); 407/1737 objects misplaced (23.431%) 2021-03-15T22:29:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:29:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: :28:59.234376+0000 mon.a (mon.0) 1639 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: audit 2021-03-15T22:29:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: 2021-03-15T22:28:59. 2021-03-15T22:29:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: 242613+0000 mon.a (mon.0) 1640 : 2021-03-15T22:29:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: audit [INF] from='client.? 172.21.15.76:0/2616572085' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:29:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: -15T22:28:59.242669+0000 2021-03-15T22:29:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: mon.a (mon.0) 1641 : cluster [DBG] 2021-03-15T22:29:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: osdmap e590: 8 total, 6 up, 8 in 2021-03-15T22:29:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:29:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: T22:28:59.904334+0000 mgr.y ( 2021-03-15T22:29:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: mgr.14144) 783 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: 2021-03-15T22:29:00.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: cluster 2021-03-15T22:29:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: 2021-03-15T22:29:00.178609+0000 2021-03-15T22:29:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: mon.a (mon.0) 1642 : cluster 2021-03-15T22:29:00.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:00 smithi125 conmon[56739]: [DBG] osdmap e591: 8 total, 6 up, 8 in 2021-03-15T22:29:00.791 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:00.813 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:00.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:00.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:00.821 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:29:00.822 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:29:00.915 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:00.937 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:00.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:00.939 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:01.039 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:01.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:01.065 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:01.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:01.168 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:01.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:01.196 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:01.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:01.297 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:01.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:01.321 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:01.321 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:01.348 INFO:tasks.rados.rados.0.smithi076.stdout:2128: done (3 left) 2021-03-15T22:29:01.348 INFO:tasks.rados.rados.0.smithi076.stdout:2129: snap_remove snap 229 2021-03-15T22:29:01.422 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:01.449 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:01.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:01.552 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:01.576 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:01.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:01.677 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:01.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:01.705 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:01.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:01.762 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:29:01.762 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:29:01.762 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:29:01.762 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:29:01.763 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:29:01.763 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:29:01.763 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:29:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:29:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:29:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:29:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:29:01.764 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:29:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:29:01.765 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:29:01.807 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:01.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:01.830 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:01.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:01.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-15T22:29:01.931 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:01.954 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:01.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:01.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:02.056 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:02.074 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[56514]: audit 2021-03-15T22:29:01.065281+0000 mon.a (mon.0) 1643 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:02.075 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[56514]: audit 2021-03-15T22:29:01.101476+0000 mon.a (mon.0) 1644 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:29:02.075 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[56514]: audit 2021-03-15T22:29:01.101710+0000 mon.a (mon.0) 1645 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:29:02.075 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[56514]: cluster 2021-03-15T22:29:01.118362+0000 mgr.y (mgr.14144) 784 : cluster [DBG] pgmap v1031: 138 pgs: 2 peering, 6 remapped+peering, 35 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 23 active+undersized, 43 active+clean; 553 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 1.1 MiB/s wr, 9 op/s; 114/1740 objects degraded (6.552%); 324/1740 objects misplaced (18.621%); 11 MiB/s, 26 objects/s recovering 2021-03-15T22:29:02.075 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[56514]: cluster 2021-03-15T22:29:01.344047+0000 mon.a (mon.0) 1646 : cluster [DBG] osdmap e592: 8 total, 6 up, 8 in 2021-03-15T22:29:02.076 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[56514]: audit 2021-03-15T22:29:01.629296+0000 mon.a (mon.0) 1647 : audit [DBG] from='client.? 172.21.15.76:0/848939937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:02.078 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:02.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:02.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:02.181 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:02.203 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:02.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:02.204 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:02.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:02.327 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:02.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:02.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:02.359 INFO:tasks.rados.rados.0.smithi076.stdout:2129: done (3 left) 2021-03-15T22:29:02.359 INFO:tasks.rados.rados.0.smithi076.stdout:2130: read oid 4 snap -1 2021-03-15T22:29:02.359 INFO:tasks.rados.rados.0.smithi076.stdout:2130: expect (ObjNum 624 snap 196 seq_num 624) 2021-03-15T22:29:02.359 INFO:tasks.rados.rados.0.smithi076.stdout:2131: write oid 17 current snap is 244 2021-03-15T22:29:02.360 INFO:tasks.rados.rados.0.smithi076.stdout:2131: seq_num 741 ranges {453987=645750,1845157=608830} 2021-03-15T22:29:02.362 INFO:tasks.rados.rados.0.smithi076.stdout:2131: writing smithi07694878-17 from 453987 to 1099737 tid 1 2021-03-15T22:29:02.371 INFO:tasks.rados.rados.0.smithi076.stdout:2131: writing smithi07694878-17 from 1845157 to 2453987 tid 2 2021-03-15T22:29:02.371 INFO:tasks.rados.rados.0.smithi076.stdout:2132: copy_from oid 21 from oid 50 current snap is 244 2021-03-15T22:29:02.371 INFO:tasks.rados.rados.0.smithi076.stdout:2126: finishing write tid 4 to smithi07694878-23 2021-03-15T22:29:02.371 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 812 (ObjNum 739 snap 244 seq_num 739) dirty exists 2021-03-15T22:29:02.372 INFO:tasks.rados.rados.0.smithi076.stdout:2126: left oid 23 (ObjNum 739 snap 244 seq_num 739) 2021-03-15T22:29:02.372 INFO:tasks.rados.rados.0.smithi076.stdout:2126: done (5 left) 2021-03-15T22:29:02.372 INFO:tasks.rados.rados.0.smithi076.stdout:2133: write oid 13 current snap is 244 2021-03-15T22:29:02.372 INFO:tasks.rados.rados.0.smithi076.stdout:2133: seq_num 742 ranges {470794=720999,1903639=554256} 2021-03-15T22:29:02.378 INFO:tasks.rados.rados.0.smithi076.stdout:2133: writing smithi07694878-13 from 470794 to 1191793 tid 1 2021-03-15T22:29:02.386 INFO:tasks.rados.rados.0.smithi076.stdout:2133: writing smithi07694878-13 from 1903639 to 2457895 tid 2 2021-03-15T22:29:02.386 INFO:tasks.rados.rados.0.smithi076.stdout:2134: rollback oid 2 current snap is 244 2021-03-15T22:29:02.386 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 2 to 172 2021-03-15T22:29:02.387 INFO:tasks.rados.rados.0.smithi076.stdout:2123: finishing write tid 1 to smithi07694878-45 2021-03-15T22:29:02.387 INFO:tasks.rados.rados.0.smithi076.stdout:2123: finishing write tid 2 to smithi07694878-45 2021-03-15T22:29:02.387 INFO:tasks.rados.rados.0.smithi076.stdout:2123: finishing write tid 3 to smithi07694878-45 2021-03-15T22:29:02.387 INFO:tasks.rados.rados.0.smithi076.stdout:2123: finishing write tid 4 to smithi07694878-45 2021-03-15T22:29:02.388 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 1208 (ObjNum 738 snap 243 seq_num 738) dirty exists 2021-03-15T22:29:02.388 INFO:tasks.rados.rados.0.smithi076.stdout:2123: left oid 45 (ObjNum 738 snap 243 seq_num 738) 2021-03-15T22:29:02.388 INFO:tasks.rados.rados.0.smithi076.stdout:2127: finishing write tid 1 to smithi07694878-49 2021-03-15T22:29:02.388 INFO:tasks.rados.rados.0.smithi076.stdout:2127: finishing write tid 2 to smithi07694878-49 2021-03-15T22:29:02.388 INFO:tasks.rados.rados.0.smithi076.stdout:2127: finishing write tid 3 to smithi07694878-49 2021-03-15T22:29:02.389 INFO:tasks.rados.rados.0.smithi076.stdout:2127: finishing write tid 4 to smithi07694878-49 2021-03-15T22:29:02.389 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 1432 (ObjNum 740 snap 244 seq_num 740) dirty exists 2021-03-15T22:29:02.389 INFO:tasks.rados.rados.0.smithi076.stdout:2127: left oid 49 (ObjNum 740 snap 244 seq_num 740) 2021-03-15T22:29:02.389 INFO:tasks.rados.rados.0.smithi076.stdout:2123: done (6 left) 2021-03-15T22:29:02.390 INFO:tasks.rados.rados.0.smithi076.stdout:2127: done (5 left) 2021-03-15T22:29:02.390 INFO:tasks.rados.rados.0.smithi076.stdout:2135: write oid 43 current snap is 244 2021-03-15T22:29:02.390 INFO:tasks.rados.rados.0.smithi076.stdout:2130: expect (ObjNum 624 snap 196 seq_num 624) 2021-03-15T22:29:02.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:02.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:02.453 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:02.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:02.468 INFO:tasks.rados.rados.0.smithi076.stdout:2135: seq_num 743 ranges {487601=796248,1962121=525480} 2021-03-15T22:29:02.475 INFO:tasks.rados.rados.0.smithi076.stdout:2135: writing smithi07694878-43 from 487601 to 1283849 tid 1 2021-03-15T22:29:02.483 INFO:tasks.rados.rados.0.smithi076.stdout:2135: writing smithi07694878-43 from 1962121 to 2487601 tid 2 2021-03-15T22:29:02.483 INFO:tasks.rados.rados.0.smithi076.stdout:2130: done (5 left) 2021-03-15T22:29:02.484 INFO:tasks.rados.rados.0.smithi076.stdout:2136: copy_from oid 36 from oid 42 current snap is 244 2021-03-15T22:29:02.484 INFO:tasks.rados.rados.0.smithi076.stdout:2134: finishing rollback tid 0 to smithi07694878-2 2021-03-15T22:29:02.484 INFO:tasks.rados.rados.0.smithi076.stdout:2134: finishing rollback tid 1 to smithi07694878-2 2021-03-15T22:29:02.484 INFO:tasks.rados.rados.0.smithi076.stdout:2134: finishing rollback tid 2 to smithi07694878-2 2021-03-15T22:29:02.484 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1314 (ObjNum 539 snap 171 seq_num 539) dirty exists 2021-03-15T22:29:02.485 INFO:tasks.rados.rados.0.smithi076.stdout:2132: finishing copy_from to smithi07694878-21 2021-03-15T22:29:02.485 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 801 (ObjNum 658 snap 206 seq_num 658) dirty exists 2021-03-15T22:29:02.485 INFO:tasks.rados.rados.0.smithi076.stdout:2132: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:29:02.485 INFO:tasks.rados.rados.0.smithi076.stdout:2133: finishing write tid 1 to smithi07694878-13 2021-03-15T22:29:02.486 INFO:tasks.rados.rados.0.smithi076.stdout:2133: finishing write tid 2 to smithi07694878-13 2021-03-15T22:29:02.486 INFO:tasks.rados.rados.0.smithi076.stdout:2133: finishing write tid 3 to smithi07694878-13 2021-03-15T22:29:02.486 INFO:tasks.rados.rados.0.smithi076.stdout:2133: finishing write tid 4 to smithi07694878-13 2021-03-15T22:29:02.486 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 586 (ObjNum 742 snap 244 seq_num 742) dirty exists 2021-03-15T22:29:02.486 INFO:tasks.rados.rados.0.smithi076.stdout:2133: left oid 13 (ObjNum 742 snap 244 seq_num 742) 2021-03-15T22:29:02.487 INFO:tasks.rados.rados.0.smithi076.stdout:2132: done (5 left) 2021-03-15T22:29:02.487 INFO:tasks.rados.rados.0.smithi076.stdout:2133: done (4 left) 2021-03-15T22:29:02.487 INFO:tasks.rados.rados.0.smithi076.stdout:2134: done (3 left) 2021-03-15T22:29:02.487 INFO:tasks.rados.rados.0.smithi076.stdout:2137: snap_create 2021-03-15T22:29:02.487 INFO:tasks.rados.rados.0.smithi076.stdout:2135: finishing write tid 1 to smithi07694878-43 2021-03-15T22:29:02.489 INFO:tasks.rados.rados.0.smithi076.stdout:2135: finishing write tid 2 to smithi07694878-43 2021-03-15T22:29:02.489 INFO:tasks.rados.rados.0.smithi076.stdout:2135: finishing write tid 3 to smithi07694878-43 2021-03-15T22:29:02.489 INFO:tasks.rados.rados.0.smithi076.stdout:2135: finishing write tid 4 to smithi07694878-43 2021-03-15T22:29:02.489 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 815 (ObjNum 743 snap 244 seq_num 743) dirty exists 2021-03-15T22:29:02.490 INFO:tasks.rados.rados.0.smithi076.stdout:2135: left oid 43 (ObjNum 743 snap 244 seq_num 743) 2021-03-15T22:29:02.499 INFO:tasks.rados.rados.0.smithi076.stdout:2136: finishing copy_from to smithi07694878-36 2021-03-15T22:29:02.500 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 1500 (ObjNum 735 snap 239 seq_num 735) dirty exists 2021-03-15T22:29:02.500 INFO:tasks.rados.rados.0.smithi076.stdout:2136: finishing copy_from racing read to smithi07694878-36 2021-03-15T22:29:02.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: audit 2021-03-15T22:29:01.065281+0000 2021-03-15T22:29:02.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: mon.a (mon.0) 1643 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:02.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: audit 2021-03-15T22:29:01.101476+0000 mon.a (mon. 2021-03-15T22:29:02.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: 0) 1644 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:29:02.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: audit 2021-03-15T22:29:01.101710+0000 2021-03-15T22:29:02.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: mon.a (mon.0) 1645 : audit [INF] 2021-03-15T22:29:02.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:29:02.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: cluster 2021-03-15T22:29:01 2021-03-15T22:29:02.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: .118362+0000 mgr.y (mgr.14144) 784 : 2021-03-15T22:29:02.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: cluster [DBG] pgmap v1031: 138 pgs: 2 peering, 6 remapped+peering, 35 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 23 active+undersized, 43 active+clean; 553 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 1.1 MiB/s wr, 9 op/s; 114/1740 objects degraded (6.552%); 324/1740 objects misplaced (18.621%); 11 MiB/s, 26 objects/s recovering 2021-03-15T22:29:02.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:29:02.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: 29:01.344047+0000 mon.a (mon.0) 2021-03-15T22:29:02.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: 1646 : cluster [DBG] osdmap e592: 8 total, 6 up, 8 in 2021-03-15T22:29:02.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:29:02.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: 15T22:29:01.629296+0000 mon.a 2021-03-15T22:29:02.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: (mon.0) 1647 : audit [DBG] from='client.? 172.21.15.76:0/848939937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:02.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:02 smithi076 conmon[62156]: 2021-03-15T22:29:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:02 smithi125 conmon[56739]: audit 2021-03-15T22:29:01.065281+0000 mon.a (mon.0) 1643 2021-03-15T22:29:02.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:02 smithi125 conmon[56739]: : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:02.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:02 smithi125 conmon[56739]: audit 2021-03-15T22:29:01.101476+0000 mon.a (mon.0) 1644 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:29:02.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:02 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:29:02.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:02 smithi125 conmon[56739]: 03-15T22:29:01.101710+0000 mon.a (mon.0) 1645 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:29:02.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:02 smithi125 conmon[56739]: cluster 2021-03-15T22:29:01.118362+0000 mgr.y (mgr 2021-03-15T22:29:02.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:02 smithi125 conmon[56739]: .14144) 784 : cluster [DBG] pgmap v1031: 138 pgs: 2 peering, 6 remapped+peering, 35 active+clean+remapped, 1 undersized+degraded+peered, 12 active+undersized+remapped, 11 active+undersized+degraded, 5 undersized+peered, 23 active+undersized, 43 active+clean; 553 MiB data, 1.3 GiB used, 625 GiB / 626 GiB avail; 5.2 MiB/s rd, 1.1 MiB/s wr, 9 op/s; 114/1740 objects degraded (6.552%); 324/1740 objects misplaced (18.621%); 11 MiB/s, 26 objects/s recovering 2021-03-15T22:29:02.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:02 smithi125 conmon[56739]: cluster 2021-03-15T22:29:01.344047+0000 mon.a ( 2021-03-15T22:29:02.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:02 smithi125 conmon[56739]: mon.0) 1646 : cluster [DBG] osdmap e592: 8 total, 6 up, 8 in 2021-03-15T22:29:02.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:02 smithi125 conmon[56739]: audit 2021-03-15T22:29:01. 2021-03-15T22:29:02.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:02 smithi125 conmon[56739]: 629296+0000 mon.a (mon.0) 1647 : audit [DBG] from='client.? 172.21.15.76:0/848939937' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:02.555 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:02.573 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:02.579 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:02.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:02.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:02.592 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:29:02.593 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:29:02.682 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:02.704 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:02.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:02.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:02.807 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:02.831 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:02.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:02.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:02.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:02.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:02.958 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:02.959 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:03.038 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:29:03.060 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:03.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:03.084 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:03.084 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:03.186 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:03.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:03.212 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:03.213 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[56514]: cluster 2021-03-15T22:29:02.063432+0000 mon.a (mon 2021-03-15T22:29:03.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[56514]: .0) 1648 : cluster [WRN] Health check update: Degraded data redundancy: 114/1740 objects degraded (6.552%), 12 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:03.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[56514]: cluster 2021-03-15T22:29:02.350659+0000 mon.a (mon.0) 1649 : cluster [DBG] osdmap e593: 8 total, 6 up, 8 in 2021-03-15T22:29:03.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[56514]: cluster 2021-03-15T22:29:02.468706+0000 osd.4 (osd.4) 124 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:03.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[56514]: cluster 2021-03-15T22:29:02.469192+0000 osd.4 (osd.4) 125 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: cluster 2021-03-15T22:29:02. 2021-03-15T22:29:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: 063432+0000 mon.a (mon.0) 1648 : cluster [WRN] Health check update: Degraded data redundancy: 114/1740 objects degraded (6.552%), 12 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: cluster 2021-03-15T22:29:03.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:29:03.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: 22:29:02.350659+0000 2021-03-15T22:29:03.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: mon.a (mon.0) 1649 : cluster 2021-03-15T22:29:03.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: [DBG] osdmap e593: 8 total, 6 up, 8 in 2021-03-15T22:29:03.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: cluster 2021-03-15T22:29:03.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:29:03.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: 15T22:29:02.468706 2021-03-15T22:29:03.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 124 2021-03-15T22:29:03.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:03.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:03.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: 03-15T22:29:02.469192 2021-03-15T22:29:03.272 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 125 2021-03-15T22:29:03.273 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:03 smithi076 conmon[62156]: : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:03.315 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:03.337 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:03.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:03.339 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:03.353 INFO:tasks.rados.rados.0.smithi076.stdout:2135: done (3 left) 2021-03-15T22:29:03.353 INFO:tasks.rados.rados.0.smithi076.stdout:2136: done (2 left) 2021-03-15T22:29:03.353 INFO:tasks.rados.rados.0.smithi076.stdout:2137: done (1 left) 2021-03-15T22:29:03.354 INFO:tasks.rados.rados.0.smithi076.stdout:2138: delete oid 24 current snap is 245 2021-03-15T22:29:03.357 INFO:tasks.rados.rados.0.smithi076.stdout:2138: done (1 left) 2021-03-15T22:29:03.357 INFO:tasks.rados.rados.0.smithi076.stdout:2139: snap_remove snap 191 2021-03-15T22:29:03.441 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:03.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:03.464 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:03.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:03.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:03 smithi125 conmon[56739]: cluster 2021-03-15T22:29:02.063432+0000 mon.a ( 2021-03-15T22:29:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:03 smithi125 conmon[56739]: mon.0) 1648 : cluster [WRN] Health check update: Degraded data redundancy: 114/1740 objects degraded (6.552%), 12 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:03 smithi125 conmon[56739]: cluster 2021-03-15T22:29:02.350659+0000 mon.a 2021-03-15T22:29:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:03 smithi125 conmon[56739]: (mon.0) 1649 : cluster [DBG] osdmap e593: 8 total, 6 up, 8 in 2021-03-15T22:29:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:03 smithi125 conmon[56739]: cluster 2021-03-15T22:29 2021-03-15T22:29:03.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:03 smithi125 conmon[56739]: :02.468706+0000 osd.4 (osd.4) 124 2021-03-15T22:29:03.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:03 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:03.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:03 smithi125 conmon[56739]: cluster 2021-03-15T22:29 2021-03-15T22:29:03.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:03 smithi125 conmon[56739]: :02.469192+0000 osd.4 (osd.4) 125 2021-03-15T22:29:03.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:03 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:03.565 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:03.587 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:03.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:03.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:03.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:03.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:03.719 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:03.720 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:03.727 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:03.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:03.847 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:03.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:03.949 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:03.973 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:03.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:03.975 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:04.077 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:04.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:04.101 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:04.101 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:04.158 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:29:04.159 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:29:04.159 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:29:04.159 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:29:04.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:04.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:04.227 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:04.227 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:04.329 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:04.353 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:04.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:04.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:04.361 INFO:tasks.rados.rados.0.smithi076.stdout:2139: done (1 left) 2021-03-15T22:29:04.362 INFO:tasks.rados.rados.0.smithi076.stdout:2140: snap_remove snap 221 2021-03-15T22:29:04.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:04.477 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:04.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:04.478 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:04 smithi076 conmon[56514]: cluster 2021-03-15T22:29:03.119017+0000 mgr.y (mgr.14144) 785 : cluster [DBG] pgmap v1034: 138 pgs: 2 peering, 2 remapped+peering, 31 active+clean+remapped, 1 undersized+degraded+peered, 7 active+undersized+remapped, 15 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 25 active+undersized, 49 active+clean; 563 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 13 op/s; 180/1740 objects degraded (10.345%); 187/1740 objects misplaced (10.747%); 13 MiB/s, 37 objects/s recovering 2021-03-15T22:29:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:04 smithi076 conmon[56514]: cluster 2021-03-15T22:29:03.348307+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e594: 8 total, 6 up, 8 in 2021-03-15T22:29:04.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:04 smithi076 conmon[62156]: cluster 2021-03-15T22:29:03.119017+0000 mgr.y (mgr.14144) 785 : cluster [DBG] pgmap v1034: 138 pgs: 2 peering, 2 remapped+peering, 31 active+clean+remapped, 1 undersized+degraded+peered, 7 active+undersized+remapped, 15 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 25 active+undersized, 49 active+clean; 563 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 13 op/s; 180/1740 objects degraded (10.345%); 187/1740 objects misplaced (10.747%); 13 MiB/s, 37 objects/s recovering 2021-03-15T22:29:04.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:04 smithi076 conmon[62156]: cluster 2021-03-15T22:29:03.348307+0000 mon.a (mon.0) 1650 : cluster [DBG] osdmap e594: 8 total, 6 up, 8 in 2021-03-15T22:29:04.579 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:04.595 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:29:04.600 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:04.602 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:04.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:04.725 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:04.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:04.727 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:04.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:04 smithi125 conmon[56739]: cluster 2021-03-15T22:29:03.119017+0000 mgr.y (mgr 2021-03-15T22:29:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:04 smithi125 conmon[56739]: .14144) 785 : cluster [DBG] pgmap v1034: 138 pgs: 2 peering, 2 remapped+peering, 31 active+clean+remapped, 1 undersized+degraded+peered, 7 active+undersized+remapped, 15 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 25 active+undersized, 49 active+clean; 563 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.3 MiB/s wr, 13 op/s; 180/1740 objects degraded (10.345%); 187/1740 objects misplaced (10.747%); 13 MiB/s, 37 objects/s recovering 2021-03-15T22:29:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:04 smithi125 conmon[56739]: cluster 2021-03-15T22:29:03.348307+0000 mon.a (mon.0 2021-03-15T22:29:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:04 smithi125 conmon[56739]: ) 1650 : cluster [DBG] osdmap e594: 8 total, 6 up, 8 in 2021-03-15T22:29:04.828 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:04.851 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:04.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:04.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:04.953 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:04.974 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 5, 6, 7, 0, 3] out_osds: [] dead_osds: [2, 5] live_osds: [0, 1, 3, 4, 6, 7] 2021-03-15T22:29:04.974 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:29:04.974 INFO:tasks.thrashosds.thrasher:Removing osd 5, in_osds are: [1, 2, 4, 5, 6, 7, 0, 3] 2021-03-15T22:29:04.975 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd out 5 2021-03-15T22:29:04.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:04.977 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:04.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:05.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:05.101 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:05.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:05.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:05.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-15T22:29:05.203 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:05.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:05.230 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:05.231 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:05.332 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:05.355 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:05.356 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:05.372 INFO:tasks.rados.rados.0.smithi076.stdout:2140: done (1 left) 2021-03-15T22:29:05.373 INFO:tasks.rados.rados.0.smithi076.stdout:2141: copy_from oid 3 from oid 12 current snap is 245 2021-03-15T22:29:05.373 INFO:tasks.rados.rados.0.smithi076.stdout:2131: finishing write tid 1 to smithi07694878-17 2021-03-15T22:29:05.373 INFO:tasks.rados.rados.0.smithi076.stdout:2131: finishing write tid 2 to smithi07694878-17 2021-03-15T22:29:05.373 INFO:tasks.rados.rados.0.smithi076.stdout:2131: finishing write tid 3 to smithi07694878-17 2021-03-15T22:29:05.373 INFO:tasks.rados.rados.0.smithi076.stdout:2131: finishing write tid 4 to smithi07694878-17 2021-03-15T22:29:05.374 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 845 (ObjNum 741 snap 244 seq_num 741) dirty exists 2021-03-15T22:29:05.374 INFO:tasks.rados.rados.0.smithi076.stdout:2131: left oid 17 (ObjNum 741 snap 244 seq_num 741) 2021-03-15T22:29:05.374 INFO:tasks.rados.rados.0.smithi076.stdout:2131: done (1 left) 2021-03-15T22:29:05.374 INFO:tasks.rados.rados.0.smithi076.stdout:2142: write (excl) oid 36 current snap is 245 2021-03-15T22:29:05.375 INFO:tasks.rados.rados.0.smithi076.stdout:2142: seq_num 745 ranges {521215=546746,1562732=701583} 2021-03-15T22:29:05.379 INFO:tasks.rados.rados.0.smithi076.stdout:2142: writing smithi07694878-36 from 521215 to 1067961 tid 1 2021-03-15T22:29:05.388 INFO:tasks.rados.rados.0.smithi076.stdout:2142: writing smithi07694878-36 from 1562732 to 2264315 tid 2 2021-03-15T22:29:05.390 INFO:tasks.rados.rados.0.smithi076.stdout:2141: finishing copy_from to smithi07694878-3 2021-03-15T22:29:05.390 INFO:tasks.rados.rados.0.smithi076.stdout:2141: got expected ENOENT (src dne) 2021-03-15T22:29:05.390 INFO:tasks.rados.rados.0.smithi076.stdout:2141: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:29:05.390 INFO:tasks.rados.rados.0.smithi076.stdout:2141: done (1 left) 2021-03-15T22:29:05.391 INFO:tasks.rados.rados.0.smithi076.stdout:2143: copy_from oid 1 from oid 18 current snap is 245 2021-03-15T22:29:05.391 INFO:tasks.rados.rados.0.smithi076.stdout:2144: write oid 16 current snap is 245 2021-03-15T22:29:05.391 INFO:tasks.rados.rados.0.smithi076.stdout:2144: seq_num 746 ranges {538022=738348,1737567=561594} 2021-03-15T22:29:05.398 INFO:tasks.rados.rados.0.smithi076.stdout:2144: writing smithi07694878-16 from 538022 to 1276370 tid 1 2021-03-15T22:29:05.407 INFO:tasks.rados.rados.0.smithi076.stdout:2144: writing smithi07694878-16 from 1737567 to 2299161 tid 2 2021-03-15T22:29:05.407 INFO:tasks.rados.rados.0.smithi076.stdout:2145: write oid 44 current snap is 245 2021-03-15T22:29:05.407 INFO:tasks.rados.rados.0.smithi076.stdout:2142: finishing write tid 1 to smithi07694878-36 2021-03-15T22:29:05.407 INFO:tasks.rados.rados.0.smithi076.stdout:2145: seq_num 747 ranges {554829=413597,1396049=705252} 2021-03-15T22:29:05.413 INFO:tasks.rados.rados.0.smithi076.stdout:2145: writing smithi07694878-44 from 554829 to 968426 tid 1 2021-03-15T22:29:05.421 INFO:tasks.rados.rados.0.smithi076.stdout:2145: writing smithi07694878-44 from 1396049 to 2101301 tid 2 2021-03-15T22:29:05.421 INFO:tasks.rados.rados.0.smithi076.stdout:2144: finishing write tid 1 to smithi07694878-16 2021-03-15T22:29:05.422 INFO:tasks.rados.rados.0.smithi076.stdout:2142: finishing write tid 2 to smithi07694878-36 2021-03-15T22:29:05.422 INFO:tasks.rados.rados.0.smithi076.stdout:2142: finishing write tid 3 to smithi07694878-36 2021-03-15T22:29:05.422 INFO:tasks.rados.rados.0.smithi076.stdout:2142: finishing write tid 4 to smithi07694878-36 2021-03-15T22:29:05.422 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 1503 (ObjNum 745 snap 245 seq_num 745) dirty exists 2021-03-15T22:29:05.423 INFO:tasks.rados.rados.0.smithi076.stdout:2142: left oid 36 (ObjNum 745 snap 245 seq_num 745) 2021-03-15T22:29:05.423 INFO:tasks.rados.rados.0.smithi076.stdout:2142: done (3 left) 2021-03-15T22:29:05.423 INFO:tasks.rados.rados.0.smithi076.stdout:2146: snap_remove snap 240 2021-03-15T22:29:05.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:05.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:05.480 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:05.481 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:05.582 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:05.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:05.605 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:05.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:05.668 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:05.670 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:05.707 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:05.729 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:05.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:05.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:05.761 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:05.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:05 smithi076 conmon[56514]: cluster 2021-03-15T22:29:04.359205 2021-03-15T22:29:05.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:05 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1651 : cluster [DBG] osdmap e595: 8 total, 6 up, 8 in 2021-03-15T22:29:05.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:05 smithi076 conmon[62156]: cluster 2021-03-15T22:29:04.359205+0000 mon.a (mon.0) 1651 : cluster [DBG] osdmap e595: 8 total, 6 up, 8 in 2021-03-15T22:29:05.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:05 smithi125 conmon[56739]: cluster 2021-03-15T22:29:04.359205+0000 mon.a (mon 2021-03-15T22:29:05.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:05 smithi125 conmon[56739]: .0) 1651 : cluster [DBG] osdmap e595: 8 total, 6 up, 8 in 2021-03-15T22:29:05.831 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:05.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:05.855 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:05.855 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:05.957 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:05.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:05.982 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:05.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:06.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:06.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:06.107 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:06.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:06.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:06.233 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:06.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:06.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:06.336 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:06.357 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:06.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:06.358 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:06.366 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:06 smithi076 conmon[56514]: cluster 2021-03-15T22:29:05.119580+0000 mgr.y (mgr.14144) 786 : cluster [DBG] pgmap v1037: 138 pgs: 33 active+clean+remapped, 1 undersized+degraded+peered, 7 active+undersized+remapped, 15 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 25 active+undersized, 51 active+clean; 563 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 7 op/s; 180/1740 objects degraded (10.345%); 195/1740 objects misplaced (11.207%); 2.7 MiB/s, 10 objects/s recovering 2021-03-15T22:29:06.366 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:06 smithi076 conmon[56514]: cluster 2021-03-15T22:29:05.365059+0000 mon.a (mon.0) 1652 : cluster [DBG] osdmap e596: 8 total, 6 up, 8 in 2021-03-15T22:29:06.367 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:06 smithi076 conmon[56514]: audit 2021-03-15T22:29:06.254684+0000 mon.c (mon.1) 52 : audit [INF] from='client.? 172.21.15.76:0/3181136792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:06.367 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:06 smithi076 conmon[56514]: audit 2021-03-15T22:29:06.255021+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:06.394 INFO:tasks.rados.rados.0.smithi076.stdout:2146: done (3 left) 2021-03-15T22:29:06.395 INFO:tasks.rados.rados.0.smithi076.stdout:2147: write oid 43 current snap is 245 2021-03-15T22:29:06.395 INFO:tasks.rados.rados.0.smithi076.stdout:2147: seq_num 748 ranges {571636=488846,1854531=565263} 2021-03-15T22:29:06.395 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:29:06.397 INFO:tasks.rados.rados.0.smithi076.stdout:2147: writing smithi07694878-43 from 571636 to 1060482 tid 1 2021-03-15T22:29:06.406 INFO:tasks.rados.rados.0.smithi076.stdout:2147: writing smithi07694878-43 from 1854531 to 2419794 tid 2 2021-03-15T22:29:06.406 INFO:tasks.rados.rados.0.smithi076.stdout:2148: write oid 33 current snap is 245 2021-03-15T22:29:06.407 INFO:tasks.rados.rados.0.smithi076.stdout:2144: finishing write tid 2 to smithi07694878-16 2021-03-15T22:29:06.407 INFO:tasks.rados.rados.0.smithi076.stdout:2144: finishing write tid 3 to smithi07694878-16 2021-03-15T22:29:06.407 INFO:tasks.rados.rados.0.smithi076.stdout:2144: finishing write tid 4 to smithi07694878-16 2021-03-15T22:29:06.408 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1464 (ObjNum 746 snap 245 seq_num 746) dirty exists 2021-03-15T22:29:06.408 INFO:tasks.rados.rados.0.smithi076.stdout:2144: left oid 16 (ObjNum 746 snap 245 seq_num 746) 2021-03-15T22:29:06.408 INFO:tasks.rados.rados.0.smithi076.stdout:2143: finishing copy_from to smithi07694878-1 2021-03-15T22:29:06.408 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 809 (ObjNum 655 snap 206 seq_num 655) dirty exists 2021-03-15T22:29:06.408 INFO:tasks.rados.rados.0.smithi076.stdout:2143: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:29:06.409 INFO:tasks.rados.rados.0.smithi076.stdout:2145: finishing write tid 1 to smithi07694878-44 2021-03-15T22:29:06.409 INFO:tasks.rados.rados.0.smithi076.stdout:2145: finishing write tid 2 to smithi07694878-44 2021-03-15T22:29:06.409 INFO:tasks.rados.rados.0.smithi076.stdout:2145: finishing write tid 3 to smithi07694878-44 2021-03-15T22:29:06.409 INFO:tasks.rados.rados.0.smithi076.stdout:2145: finishing write tid 4 to smithi07694878-44 2021-03-15T22:29:06.409 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1194 (ObjNum 747 snap 245 seq_num 747) dirty exists 2021-03-15T22:29:06.410 INFO:tasks.rados.rados.0.smithi076.stdout:2145: left oid 44 (ObjNum 747 snap 245 seq_num 747) 2021-03-15T22:29:06.410 INFO:tasks.rados.rados.0.smithi076.stdout:2147: finishing write tid 1 to smithi07694878-43 2021-03-15T22:29:06.410 INFO:tasks.rados.rados.0.smithi076.stdout:2148: seq_num 749 ranges {588443=564095,1796660=708921} 2021-03-15T22:29:06.412 INFO:tasks.rados.rados.0.smithi076.stdout:2148: writing smithi07694878-33 from 588443 to 1152538 tid 1 2021-03-15T22:29:06.421 INFO:tasks.rados.rados.0.smithi076.stdout:2148: writing smithi07694878-33 from 1796660 to 2505581 tid 2 2021-03-15T22:29:06.422 INFO:tasks.rados.rados.0.smithi076.stdout:2147: finishing write tid 2 to smithi07694878-43 2021-03-15T22:29:06.423 INFO:tasks.rados.rados.0.smithi076.stdout:2147: finishing write tid 3 to smithi07694878-43 2021-03-15T22:29:06.423 INFO:tasks.rados.rados.0.smithi076.stdout:2147: finishing write tid 4 to smithi07694878-43 2021-03-15T22:29:06.423 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 818 (ObjNum 748 snap 245 seq_num 748) dirty exists 2021-03-15T22:29:06.423 INFO:tasks.rados.rados.0.smithi076.stdout:2147: left oid 43 (ObjNum 748 snap 245 seq_num 748) 2021-03-15T22:29:06.424 INFO:tasks.rados.rados.0.smithi076.stdout:2148: finishing write tid 1 to smithi07694878-33 2021-03-15T22:29:06.424 INFO:tasks.rados.rados.0.smithi076.stdout:2143: done (4 left) 2021-03-15T22:29:06.424 INFO:tasks.rados.rados.0.smithi076.stdout:2144: done (3 left) 2021-03-15T22:29:06.425 INFO:tasks.rados.rados.0.smithi076.stdout:2145: done (2 left) 2021-03-15T22:29:06.425 INFO:tasks.rados.rados.0.smithi076.stdout:2147: done (1 left) 2021-03-15T22:29:06.425 INFO:tasks.rados.rados.0.smithi076.stdout:2149: read oid 38 snap -1 2021-03-15T22:29:06.425 INFO:tasks.rados.rados.0.smithi076.stdout:2149: expect (ObjNum 732 snap 239 seq_num 732) 2021-03-15T22:29:06.426 INFO:tasks.rados.rados.0.smithi076.stdout:2150: read oid 13 snap -1 2021-03-15T22:29:06.426 INFO:tasks.rados.rados.0.smithi076.stdout:2150: expect (ObjNum 742 snap 244 seq_num 742) 2021-03-15T22:29:06.426 INFO:tasks.rados.rados.0.smithi076.stdout:2151: write (excl) oid 42 current snap is 245 2021-03-15T22:29:06.426 INFO:tasks.rados.rados.0.smithi076.stdout:2151: seq_num 750 ranges {605250=639344,1855142=452579} 2021-03-15T22:29:06.431 INFO:tasks.rados.rados.0.smithi076.stdout:2151: writing smithi07694878-42 from 605250 to 1244594 tid 1 2021-03-15T22:29:06.437 INFO:tasks.rados.rados.0.smithi076.stdout:2151: writing smithi07694878-42 from 1855142 to 2307721 tid 2 2021-03-15T22:29:06.438 INFO:tasks.rados.rados.0.smithi076.stdout:2152: snap_remove snap 207 2021-03-15T22:29:06.460 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:06.497 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:06.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:06.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:06.621 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:06.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:06.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:06.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:06 smithi076 conmon[62156]: cluster 2021-03-15T22:29:05.119580 2021-03-15T22:29:06.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:06 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 786 : cluster [DBG] pgmap v1037: 138 pgs: 33 active+clean+remapped, 1 undersized+degraded+peered, 7 active+undersized+remapped, 15 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 25 active+undersized, 51 active+clean; 563 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 7 op/s; 180/1740 objects degraded (10.345%); 195/1740 objects misplaced (11.207%); 2.7 MiB/s, 10 objects/s recovering 2021-03-15T22:29:06.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:06 smithi076 conmon[62156]: cluster 2021-03-15T22:29:05.365059+0000 mon.a (mon.0) 1652 : cluster [DBG] osdmap e596: 8 total, 6 up, 8 in 2021-03-15T22:29:06.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:06 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:29:06.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:06 smithi076 conmon[62156]: :29:06.254684+0000 mon.c (mon.1) 52 : audit [INF] from='client.? 172.21.15.76:0/3181136792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:06.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:06 smithi076 conmon[62156]: audit 2021-03-15T22:29:06.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:06 smithi076 conmon[62156]: 2021-03-15T22:29:06.255021+0000 mon.a (mon.0) 1653 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:06.725 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:06.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:06.750 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:06.750 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:06.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:06 smithi125 conmon[56739]: cluster 2021-03-15T22:29:05.119580+0000 2021-03-15T22:29:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:06 smithi125 conmon[56739]: mgr.y (mgr.14144) 786 : cluster [DBG] pgmap v1037: 138 pgs: 33 active+clean+remapped, 1 undersized+degraded+peered, 7 active+undersized+remapped, 15 active+undersized+degraded, 1 active+recovering+undersized+remapped, 5 undersized+peered, 25 active+undersized, 51 active+clean; 563 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.8 MiB/s wr, 7 op/s; 180/1740 objects degraded (10.345%); 195/1740 objects misplaced (11.207%); 2.7 MiB/s, 10 objects/s recovering 2021-03-15T22:29:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:06 smithi125 conmon[56739]: cluster 2021-03-15T22:29:05.365059+0000 mon.a 2021-03-15T22:29:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:06 smithi125 conmon[56739]: (mon.0) 1652 : cluster [DBG] osdmap e596: 8 total, 6 up, 8 in 2021-03-15T22:29:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:06 smithi125 conmon[56739]: audit 2021-03-15T22:29:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:06 smithi125 conmon[56739]: 2021-03-15T22:29:06.254684+0000 mon.c (mon.1) 2021-03-15T22:29:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:06 smithi125 conmon[56739]: 52 : audit [INF] from='client.? 172.21.15.76:0/3181136792' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:06 smithi125 conmon[56739]: audit 2021 2021-03-15T22:29:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:06 smithi125 conmon[56739]: -03-15T22:29:06.255021+0000 mon.a 2021-03-15T22:29:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:06 smithi125 conmon[56739]: (mon.0) 1653 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:06.852 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:06.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:06.875 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:06.875 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:06.977 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:06.999 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:06.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:07.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:07.101 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:07.123 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:07.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:07.125 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:07.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:07.249 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:07.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:07.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:07.351 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:07.374 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:07.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:07.376 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:07.392 INFO:tasks.rados.rados.0.smithi076.stdout:2152: done (4 left) 2021-03-15T22:29:07.393 INFO:tasks.rados.rados.0.smithi076.stdout:2153: write oid 37 current snap is 245 2021-03-15T22:29:07.393 INFO:tasks.rados.rados.0.smithi076.stdout:2153: seq_num 751 ranges {622057=714593,1913624=708433} 2021-03-15T22:29:07.399 INFO:tasks.rados.rados.0.smithi076.stdout:2153: writing smithi07694878-37 from 622057 to 1336650 tid 1 2021-03-15T22:29:07.407 INFO:tasks.rados.rados.0.smithi076.stdout:2153: writing smithi07694878-37 from 1913624 to 2622057 tid 2 2021-03-15T22:29:07.407 INFO:tasks.rados.rados.0.smithi076.stdout:2154: rollback oid 8 current snap is 245 2021-03-15T22:29:07.407 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 241 2021-03-15T22:29:07.408 INFO:tasks.rados.rados.0.smithi076.stdout:2149: expect (ObjNum 732 snap 239 seq_num 732) 2021-03-15T22:29:07.477 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:07.499 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:07.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:07.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:07.579 INFO:tasks.rados.rados.0.smithi076.stdout:2150: expect (ObjNum 742 snap 244 seq_num 742) 2021-03-15T22:29:07.602 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:07.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:07.625 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:07.625 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:07.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:07.733 INFO:tasks.rados.rados.0.smithi076.stdout:2148: finishing write tid 2 to smithi07694878-33 2021-03-15T22:29:07.733 INFO:tasks.rados.rados.0.smithi076.stdout:2149: done (5 left) 2021-03-15T22:29:07.733 INFO:tasks.rados.rados.0.smithi076.stdout:2150: done (4 left) 2021-03-15T22:29:07.733 INFO:tasks.rados.rados.0.smithi076.stdout:2155: write oid 46 current snap is 245 2021-03-15T22:29:07.734 INFO:tasks.rados.rados.0.smithi076.stdout:2148: finishing write tid 3 to smithi07694878-33 2021-03-15T22:29:07.734 INFO:tasks.rados.rados.0.smithi076.stdout:2151: finishing write tid 1 to smithi07694878-42 2021-03-15T22:29:07.734 INFO:tasks.rados.rados.0.smithi076.stdout:2148: finishing write tid 4 to smithi07694878-33 2021-03-15T22:29:07.734 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 861 (ObjNum 749 snap 245 seq_num 749) dirty exists 2021-03-15T22:29:07.734 INFO:tasks.rados.rados.0.smithi076.stdout:2148: left oid 33 (ObjNum 749 snap 245 seq_num 749) 2021-03-15T22:29:07.735 INFO:tasks.rados.rados.0.smithi076.stdout:2151: finishing write tid 2 to smithi07694878-42 2021-03-15T22:29:07.735 INFO:tasks.rados.rados.0.smithi076.stdout:2155: seq_num 752 ranges {638864=789842,1972106=456248} 2021-03-15T22:29:07.740 INFO:tasks.rados.rados.0.smithi076.stdout:2155: writing smithi07694878-46 from 638864 to 1428706 tid 1 2021-03-15T22:29:07.751 INFO:tasks.rados.rados.0.smithi076.stdout:2155: writing smithi07694878-46 from 1972106 to 2428354 tid 2 2021-03-15T22:29:07.752 INFO:tasks.rados.rados.0.smithi076.stdout:2148: done (4 left) 2021-03-15T22:29:07.752 INFO:tasks.rados.rados.0.smithi076.stdout:2156: copy_from oid 33 from oid 15 current snap is 245 2021-03-15T22:29:07.752 INFO:tasks.rados.rados.0.smithi076.stdout:2151: finishing write tid 3 to smithi07694878-42 2021-03-15T22:29:07.752 INFO:tasks.rados.rados.0.smithi076.stdout:2151: finishing write tid 4 to smithi07694878-42 2021-03-15T22:29:07.753 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 1512 (ObjNum 750 snap 245 seq_num 750) dirty exists 2021-03-15T22:29:07.754 INFO:tasks.rados.rados.0.smithi076.stdout:2151: left oid 42 (ObjNum 750 snap 245 seq_num 750) 2021-03-15T22:29:07.754 INFO:tasks.rados.rados.0.smithi076.stdout:2153: finishing write tid 1 to smithi07694878-37 2021-03-15T22:29:07.754 INFO:tasks.rados.rados.0.smithi076.stdout:2151: done (4 left) 2021-03-15T22:29:07.755 INFO:tasks.rados.rados.0.smithi076.stdout:2157: snap_create 2021-03-15T22:29:07.755 INFO:tasks.rados.rados.0.smithi076.stdout:2154: finishing rollback tid 0 to smithi07694878-8 2021-03-15T22:29:07.755 INFO:tasks.rados.rados.0.smithi076.stdout:2153: finishing write tid 2 to smithi07694878-37 2021-03-15T22:29:07.756 INFO:tasks.rados.rados.0.smithi076.stdout:2153: finishing write tid 3 to smithi07694878-37 2021-03-15T22:29:07.756 INFO:tasks.rados.rados.0.smithi076.stdout:2153: finishing write tid 4 to smithi07694878-37 2021-03-15T22:29:07.756 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 1516 (ObjNum 751 snap 245 seq_num 751) dirty exists 2021-03-15T22:29:07.757 INFO:tasks.rados.rados.0.smithi076.stdout:2153: left oid 37 (ObjNum 751 snap 245 seq_num 751) 2021-03-15T22:29:07.757 INFO:tasks.rados.rados.0.smithi076.stdout:2154: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:29:07.757 INFO:tasks.rados.rados.0.smithi076.stdout:2154: finishing rollback tid 2 to smithi07694878-8 2021-03-15T22:29:07.757 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1194 (ObjNum 729 snap 238 seq_num 729) dirty exists 2021-03-15T22:29:07.757 INFO:tasks.rados.rados.0.smithi076.stdout:2155: finishing write tid 1 to smithi07694878-46 2021-03-15T22:29:07.758 INFO:tasks.rados.rados.0.smithi076.stdout:2155: finishing write tid 2 to smithi07694878-46 2021-03-15T22:29:07.758 INFO:tasks.rados.rados.0.smithi076.stdout:2155: finishing write tid 3 to smithi07694878-46 2021-03-15T22:29:07.758 INFO:tasks.rados.rados.0.smithi076.stdout:2155: finishing write tid 4 to smithi07694878-46 2021-03-15T22:29:07.758 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 845 (ObjNum 752 snap 245 seq_num 752) dirty exists 2021-03-15T22:29:07.759 INFO:tasks.rados.rados.0.smithi076.stdout:2155: left oid 46 (ObjNum 752 snap 245 seq_num 752) 2021-03-15T22:29:07.760 INFO:tasks.rados.rados.0.smithi076.stdout:2156: finishing copy_from to smithi07694878-33 2021-03-15T22:29:07.760 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 1352 (ObjNum 714 snap 231 seq_num 714) dirty exists 2021-03-15T22:29:07.760 INFO:tasks.rados.rados.0.smithi076.stdout:2156: finishing copy_from racing read to smithi07694878-33 2021-03-15T22:29:07.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:07.761 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:07.762 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:07.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:07 smithi076 conmon[56514]: cluster 2021-03-15T22:29:06.362815+0000 mon.a (mon.0) 1654 2021-03-15T22:29:07.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:07 smithi076 conmon[56514]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:07.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:07 smithi076 conmon[56514]: audit 2021-03-15T22:29:06.377518+0000 mon.a (mon.0) 1655 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:29:07.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:07 smithi076 conmon[56514]: cluster 2021-03-15T22:29:06.377581+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e597: 8 total, 6 up, 8 in 2021-03-15T22:29:07.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:07 smithi076 conmon[62156]: cluster 2021-03-15T22:29:06.362815+0000 mon.a ( 2021-03-15T22:29:07.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:07 smithi076 conmon[62156]: mon.0) 1654 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:07.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:07 smithi076 conmon[62156]: audit 2021-03-15T22:29:06.377518+0000 mon.a (mon.0) 2021-03-15T22:29:07.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:07 smithi076 conmon[62156]: 1655 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:29:07.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:07 smithi076 conmon[62156]: cluster 2021-03-15T22:29:06 2021-03-15T22:29:07.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:07 smithi076 conmon[62156]: .377581+0000 mon.a (mon.0) 1656 : cluster [DBG] osdmap e597: 8 total, 6 up, 8 in 2021-03-15T22:29:07.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:07 smithi125 conmon[56739]: cluster 2021-03-15T22:29:06.362815+0000 mon.a (mon.0 2021-03-15T22:29:07.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:07 smithi125 conmon[56739]: ) 1654 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:07.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:07 smithi125 conmon[56739]: audit 2021-03-15T22:29:06.377518+0000 mon.a (mon.0) 1655 2021-03-15T22:29:07.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:07 smithi125 conmon[56739]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:29:07.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:07 smithi125 conmon[56739]: cluster 2021-03-15T22:29:06.377581+0000 2021-03-15T22:29:07.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:07 smithi125 conmon[56739]: mon.a (mon.0) 1656 : cluster [DBG] 2021-03-15T22:29:07.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:07 smithi125 conmon[56739]: osdmap e597: 8 total, 6 up, 8 in 2021-03-15T22:29:07.863 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:07.885 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:07.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:07.887 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:07.988 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:08.009 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:08.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:08.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:08.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:08.134 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:08.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:08.136 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:08.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:08.259 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:08.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:08.261 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:08.362 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:08.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:08.385 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:08.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:08.387 INFO:tasks.rados.rados.0.smithi076.stdout:2153: done (4 left) 2021-03-15T22:29:08.387 INFO:tasks.rados.rados.0.smithi076.stdout:2154: done (3 left) 2021-03-15T22:29:08.387 INFO:tasks.rados.rados.0.smithi076.stdout:2155: done (2 left) 2021-03-15T22:29:08.388 INFO:tasks.rados.rados.0.smithi076.stdout:2156: done (1 left) 2021-03-15T22:29:08.388 INFO:tasks.rados.rados.0.smithi076.stdout:2157: done (0 left) 2021-03-15T22:29:08.388 INFO:tasks.rados.rados.0.smithi076.stdout:2158: rollback oid 5 current snap is 246 2021-03-15T22:29:08.388 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 220 2021-03-15T22:29:08.388 INFO:tasks.rados.rados.0.smithi076.stdout:2159: snap_remove snap 234 2021-03-15T22:29:08.487 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:08.513 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:08.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:08.514 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:08.615 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:08.638 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:08.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:08.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:08.740 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:08.745 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: cluster 2021-03-15T22:29:07.120314+0000 mgr.y (mgr 2021-03-15T22:29:08.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: .14144) 787 : cluster [DBG] pgmap v1040: 138 pgs: 33 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+remapped, 15 active+undersized+degraded, 5 undersized+peered, 25 active+undersized, 54 active+clean; 568 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 1.8 MiB/s wr, 12 op/s; 180/1752 objects degraded (10.274%); 195/1752 objects misplaced (11.130%); 289 KiB/s, 0 objects/s recovering 2021-03-15T22:29:08.746 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: cluster 2021-03-15T22:29:07.148871+0000 osd.6 (osd.6) 92 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:29:08.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: cluster 2021-03-15T22:29: 2021-03-15T22:29:08.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: 07.163030+0000 osd.7 (osd. 2021-03-15T22:29:08.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: 7) 117 : cluster [DBG] 2021-03-15T22:29:08.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: 6.5 scrub ok 2021-03-15T22:29:08.747 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:29:08.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: :29:07.371548+0000 2021-03-15T22:29:08.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: mon.a (mon.0) 1657 : 2021-03-15T22:29:08.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: cluster [WRN] Health check update: Degraded data redundancy: 180/1752 objects degraded (10.274%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:08.748 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:29:08.749 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: 03-15T22:29:07.383117+0000 2021-03-15T22:29:08.749 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: mon.a (mon.0) 1658 : cluster [DBG] 2021-03-15T22:29:08.749 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[56514]: osdmap e598: 8 total, 6 up, 8 in 2021-03-15T22:29:08.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: cluster 2021-03-15T22:29:07.120314 2021-03-15T22:29:08.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 787 : cluster [DBG] pgmap v1040: 138 pgs: 33 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+remapped, 15 active+undersized+degraded, 5 undersized+peered, 25 active+undersized, 54 active+clean; 568 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 1.8 MiB/s wr, 12 op/s; 180/1752 objects degraded (10.274%); 195/1752 objects misplaced (11.130%); 289 KiB/s, 0 objects/s recovering 2021-03-15T22:29:08.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: cluster 2021-03-15T22:29:08.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: 2021-03-15T22:29:07.148871+0000 osd.6 (osd.6) 92 : cluster 2021-03-15T22:29:08.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: [DBG] 3.10 scrub ok 2021-03-15T22:29:08.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: cluster 2021-03-15T22:29:08.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:29:08.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: 15T22:29:07. 2021-03-15T22:29:08.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: 163030+0000 osd.7 (osd.7) 117 2021-03-15T22:29:08.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: : cluster [DBG] 6.5 scrub ok 2021-03-15T22:29:08.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:29:08.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: -15T22:29:07.371548+0000 2021-03-15T22:29:08.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: mon.a (mon.0) 1657 : cluster [WRN] 2021-03-15T22:29:08.753 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: Health check update: Degraded data redundancy: 180/1752 objects degraded (10.274%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:08.753 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: cluster 2021-03-15T22:29:08.753 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: 2021-03-15T22:29:07. 2021-03-15T22:29:08.753 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: 383117+0000 mon.a (mon.0) 1658 : cluster 2021-03-15T22:29:08.754 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:08 smithi076 conmon[62156]: [DBG] osdmap e598: 8 total, 6 up, 8 in 2021-03-15T22:29:08.763 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:08.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:08.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:08.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: cluster 2021-03-15T22:29:07.120314+0000 mgr.y (mgr.14144 2021-03-15T22:29:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: ) 787 : cluster [DBG] pgmap v1040: 138 pgs: 33 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+remapped, 15 active+undersized+degraded, 5 undersized+peered, 25 active+undersized, 54 active+clean; 568 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 1.8 MiB/s wr, 12 op/s; 180/1752 objects degraded (10.274%); 195/1752 objects misplaced (11.130%); 289 KiB/s, 0 objects/s recovering 2021-03-15T22:29:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: cluster 2021-03-15T22:29:07.148871+0000 osd.6 ( 2021-03-15T22:29:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: osd.6) 92 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:29:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:29:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: 03-15T22:29:07. 2021-03-15T22:29:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: 163030+0000 osd.7 (osd.7) 2021-03-15T22:29:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: 117 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:29:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:29:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: 03-15T22:29:07.371548 2021-03-15T22:29:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2021-03-15T22:29:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: 1657 : cluster [WRN] Health check update: Degraded data redundancy: 180/1752 objects degraded (10.274%), 16 pgs degraded, 34 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:29:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: -03-15T22:29: 2021-03-15T22:29:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: 07.383117+0000 mon.a (mon 2021-03-15T22:29:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: .0) 1658 : cluster [DBG] 2021-03-15T22:29:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:08 smithi125 conmon[56739]: osdmap e598: 8 total, 6 up, 8 in 2021-03-15T22:29:08.866 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:08.887 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:08.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:08.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:08.990 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:09.013 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:09.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:09.015 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:09.116 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:09.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:09.139 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:09.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:09.241 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:09.264 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:09.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:09.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:09.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:09.388 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:09.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:09.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:09.401 INFO:tasks.rados.rados.0.smithi076.stdout:2159: done (1 left) 2021-03-15T22:29:09.401 INFO:tasks.rados.rados.0.smithi076.stdout:2160: snap_remove snap 213 2021-03-15T22:29:09.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:09.515 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:09.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:09.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:09.589 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:29:09.607 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:29:09.618 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:09.625 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:29:09.640 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:09.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:09.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:09.643 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:29:09.661 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:29:09.679 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:29:09.679 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:29:09.679 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:29:09.680 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:29:09.680 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:29:09.680 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:29:09.680 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:29:09.681 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:29:09.681 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:29:09.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[56514]: cluster 2021-03-15T22:29:08.137981+0000 osd.7 (osd.7) 118 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:29:09.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[56514]: cluster 2021-03-15T22:29:08.154997+0000 osd.6 (osd.6) 93 : cluster [DBG] 7.d scrub ok 2021-03-15T22:29:09.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[56514]: cluster 2021-03-15T22:29:08.383223+0000 mon.a (mon.0) 1659 : cluster [DBG] osdmap e599: 8 total, 6 up, 8 in 2021-03-15T22:29:09.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:29:09.720 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[56514]: 22:29:08.872832+0000 osd.0 (osd.0) 89 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:29:09.720 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[56514]: debug 2021-03-15T22:29:09.716+0000 7f0537c88700 -1 mon.a@0(leader).osd e600 definitely_dead 0 2021-03-15T22:29:09.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: cluster 2021-03-15T22:29:09.721 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021-03-15T22:29:09.721 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021 2021-03-15T22:29:09.721 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: - 2021-03-15T22:29:09.721 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 03 2021-03-15T22:29:09.722 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: - 2021-03-15T22:29:09.722 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 15 2021-03-15T22:29:09.722 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: T 2021-03-15T22:29:09.722 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 22 2021-03-15T22:29:09.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: : 2021-03-15T22:29:09.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 29 2021-03-15T22:29:09.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: : 2021-03-15T22:29:09.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 08 2021-03-15T22:29:09.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: . 2021-03-15T22:29:09.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 137981 2021-03-15T22:29:09.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: +0000 2021-03-15T22:29:09.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021-03-15T22:29:09.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: osd.7 2021-03-15T22:29:09.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: ( 2021-03-15T22:29:09.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: osd 2021-03-15T22:29:09.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: . 2021-03-15T22:29:09.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 7 2021-03-15T22:29:09.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: ) 2021-03-15T22:29:09.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 118 2021-03-15T22:29:09.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: : 2021-03-15T22:29:09.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: cluster 2021-03-15T22:29:09.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021-03-15T22:29:09.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: [DBG] 2021-03-15T22:29:09.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021-03-15T22:29:09.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 6.8 scrub ok 2021-03-15T22:29:09.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021-03-15T22:29:09.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: cluster 2021-03-15T22:29:09.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021-03-15T22:29:09.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021 2021-03-15T22:29:09.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: - 2021-03-15T22:29:09.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 03 2021-03-15T22:29:09.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: - 2021-03-15T22:29:09.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 15 2021-03-15T22:29:09.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: T 2021-03-15T22:29:09.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 22 2021-03-15T22:29:09.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: : 2021-03-15T22:29:09.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 29 2021-03-15T22:29:09.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: : 2021-03-15T22:29:09.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 08 2021-03-15T22:29:09.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: . 2021-03-15T22:29:09.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 154997 2021-03-15T22:29:09.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: +0000 2021-03-15T22:29:09.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021-03-15T22:29:09.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: osd.6 2021-03-15T22:29:09.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: ( 2021-03-15T22:29:09.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: osd 2021-03-15T22:29:09.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: . 2021-03-15T22:29:09.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 6 2021-03-15T22:29:09.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: ) 2021-03-15T22:29:09.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 93 2021-03-15T22:29:09.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: : 2021-03-15T22:29:09.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: cluster 2021-03-15T22:29:09.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021-03-15T22:29:09.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: [DBG] 2021-03-15T22:29:09.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021-03-15T22:29:09.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 7.d scrub ok 2021-03-15T22:29:09.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021-03-15T22:29:09.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: cluster 2021-03-15T22:29:09.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021-03-15T22:29:09.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 2021 2021-03-15T22:29:09.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: - 2021-03-15T22:29:09.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 03 2021-03-15T22:29:09.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: - 2021-03-15T22:29:09.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 15 2021-03-15T22:29:09.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: T 2021-03-15T22:29:09.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 22 2021-03-15T22:29:09.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: : 2021-03-15T22:29:09.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 29 2021-03-15T22:29:09.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: : 2021-03-15T22:29:09.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: 08 2021-03-15T22:29:09.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: .383223+0000 mon.a (mon.0) 1659 : 2021-03-15T22:29:09.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: cluster [DBG] osdmap e599: 8 total, 6 up, 8 in 2021-03-15T22:29:09.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: cluster 2021-03-15T22:29:08.872832+0000 osd.0 ( 2021-03-15T22:29:09.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:09 smithi076 conmon[62156]: osd.0) 89 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:29:09.742 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:09.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:09.770 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:09.770 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[56739]: cluster 2021-03-15T22:29:08.137981+0000 osd.7 2021-03-15T22:29:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[56739]: (osd.7) 118 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:29:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[56739]: cluster 2021-03-15T22:29:08.154997+0000 osd.6 2021-03-15T22:29:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[56739]: (osd.6) 93 : cluster [DBG] 7.d scrub ok 2021-03-15T22:29:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:29:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[56739]: 22:29:08.383223+0000 mon.a (mon.0 2021-03-15T22:29:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[56739]: ) 1659 : cluster [DBG] osdmap e599: 8 total, 6 up, 8 in 2021-03-15T22:29:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:29:09.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[56739]: -03-15T22:29:08.872832+0000 2021-03-15T22:29:09.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[56739]: osd.0 (osd.0) 89 : cluster [DBG] 2021-03-15T22:29:09.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[56739]: 4.3 scrub ok 2021-03-15T22:29:09.871 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:09.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:09.896 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:09.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:09.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:10.020 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:10.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:10.021 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:10.056 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:29:10.122 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:10.145 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:10.146 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:10.190 INFO:tasks.rados.rados.0.smithi076.stdout:2160: done (1 left) 2021-03-15T22:29:10.191 INFO:tasks.rados.rados.0.smithi076.stdout:2161: snap_create 2021-03-15T22:29:10.191 INFO:tasks.rados.rados.0.smithi076.stdout:2158: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:29:10.191 INFO:tasks.rados.rados.0.smithi076.stdout:2158: finishing rollback tid 2 to smithi07694878-5 2021-03-15T22:29:10.191 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 617 (ObjNum 668 snap 212 seq_num 668) dirty exists 2021-03-15T22:29:10.196 INFO:teuthology.orchestra.run.smithi076.stderr:marked out osd.5. 2021-03-15T22:29:10.247 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:10.268 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:10.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:10.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:10.286 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:29:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:29:10.371 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:10.395 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:10.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:10.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:10.497 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:10.520 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:10.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:10.521 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:10.542 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:29:10.542 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:29:10.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:10.644 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:10.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:10.646 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:10.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: cluster 2021-03-15T22:29:09.120916+0000 mgr.y (mgr.14144) 788 : cluster [DBG] pgmap v1043: 138 pgs: 33 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+remapped, 15 active+undersized+degraded, 5 undersized+peered, 25 active+undersized, 54 active+clean; 572 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 183/1761 objects degraded (10.392%); 195/1761 objects misplaced (11.073%); 289 KiB/s, 0 objects/s recovering 2021-03-15T22:29:10.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: cluster 2021-03-15T22:29:09.154602+0000 osd.7 (osd.7 2021-03-15T22:29:10.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: ) 119 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:29:10.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: cluster 2021-03-15T22:29:10.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 2021-03-15T22:29:09. 2021-03-15T22:29:10.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 184793+0000 osd.3 (osd.3) 105 : cluster [DBG] 2021-03-15T22:29:10.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 3.1e scrub ok 2021-03-15T22:29:10.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:29:10.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: T22:29:09.390023+0000 mon.a (mon. 2021-03-15T22:29:10.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 0) 1660 : cluster [DBG] osdmap e600: 8 total, 6 up, 8 in 2021-03-15T22:29:10.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:29:10.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 22:29:09.569293+0000 mon.c (mon.1) 2021-03-15T22:29:10.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 53 : audit [DBG] from='client.? 172.21.15.76:0/1834438441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:10.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: audit 2021-03-15T22:29:10.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 2021-03-15T22:29 2021-03-15T22:29:10.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: :09.717006+0000 mon.a (mon.0) 1661 2021-03-15T22:29:10.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/1830620856' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2021-03-15T22:29:10.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: cluster 2021-03-15T22:29:10.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 2021-03-15T22:29:09. 2021-03-15T22:29:10.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 717093+0000 mon.a (mon.0) 1662 : cluster [INF] 2021-03-15T22:29:10.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: Client client.admin marked osd.5 out, after it was down for 94 seconds 2021-03-15T22:29:10.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:29:10.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 15T22:29:09.915108+0000 mgr.y (mgr. 2021-03-15T22:29:10.668 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 14144) 789 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:10.668 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: cluster 2021-03-15T22:29:10.668 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 2021-03-15T22:29:10.177223 2021-03-15T22:29:10.668 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1663 : cluster [WRN] 2021-03-15T22:29:10.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: Health check update: 1 osds down (OSD_DOWN) 2021-03-15T22:29:10.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: audit 2021-03-15T22:29:10.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 2021-03-15T22:29:10. 2021-03-15T22:29:10.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 181681+0000 mon.a (mon.0) 1664 : audit 2021-03-15T22:29:10.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/1830620856' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2021-03-15T22:29:10.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: cluster 2021-03-15T22:29:10.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:29:10.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: :29:10.181749+0000 mon.a (mon.0) 1665 : 2021-03-15T22:29:10.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[62156]: cluster [DBG] osdmap e601: 8 total, 6 up, 7 in 2021-03-15T22:29:10.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: cluster 2021-03-15T22:29:09.120916+0000 mgr.y (mgr.14144 2021-03-15T22:29:10.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: ) 788 : cluster [DBG] pgmap v1043: 138 pgs: 33 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+remapped, 15 active+undersized+degraded, 5 undersized+peered, 25 active+undersized, 54 active+clean; 572 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 183/1761 objects degraded (10.392%); 195/1761 objects misplaced (11.073%); 289 KiB/s, 0 objects/s recovering 2021-03-15T22:29:10.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: cluster 2021-03-15T22:29:09.154602+0000 osd.7 (osd.7) 119 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:29:10.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: cluster 2021-03-15T22:29:09.184793+0000 osd.3 (osd.3) 105 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:29:10.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: cluster 2021-03-15T22:29:09.390023+0000 mon.a (mon.0) 1660 : cluster [DBG] osdmap e600: 8 total, 6 up, 8 in 2021-03-15T22:29:10.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: audit 2021-03-15T22:29:09.569293+0000 mon.c (mon.1) 53 : audit 2021-03-15T22:29:10.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: [DBG] from='client.? 172.21.15.76:0/1834438441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:10.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: audit 2021-03-15T22:29:09.717006+0000 mon.a (mon.0) 1661 2021-03-15T22:29:10.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: : audit [INF] from='client.? 172.21.15.76:0/1830620856' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2021-03-15T22:29:10.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: cluster 2021-03-15T22:29: 2021-03-15T22:29:10.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: 09.717093+0000 mon.a (mon.0) 1662 : 2021-03-15T22:29:10.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: cluster [INF] Client client.admin marked osd.5 out, after it was down for 94 seconds 2021-03-15T22:29:10.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:29:10.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: 03-15T22:29:09.915108+0000 2021-03-15T22:29:10.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: mgr.y (mgr.14144) 789 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:10.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: 2021-03-15T22:29:10.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: cluster 2021-03-15T22:29: 2021-03-15T22:29:10.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: 10.177223+0000 mon.a (mon.0) 1663 2021-03-15T22:29:10.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: : cluster [WRN] Health check update: 1 osds down (OSD_DOWN) 2021-03-15T22:29:10.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: audit 2021-03-15T 2021-03-15T22:29:10.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: 22:29:10.181681+0000 mon.a (mon. 2021-03-15T22:29:10.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: 0) 1664 : audit [INF] from='client.? 172.21.15.76:0/1830620856' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2021-03-15T22:29:10.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:29:10.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:10 smithi076 conmon[56514]: -03-15T22:29:10.181749+0000 mon.a (mon.0) 1665 : cluster [DBG] osdmap e601: 8 total, 6 up, 7 in 2021-03-15T22:29:10.694 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:10.702 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:29:10.703 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:29:10.747 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:10.768 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:10.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:10.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:10.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: cluster 2021-03-15T22:29:09.120916+0000 mgr.y (mgr.14144) 788 : cluster [DBG] pgmap v1043: 138 pgs: 33 active+clean+remapped, 1 undersized+degraded+peered, 5 active+undersized+remapped, 15 active+undersized+degraded, 5 undersized+peered, 25 active+undersized, 54 active+clean; 572 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 183/1761 objects degraded (10.392%); 195/1761 objects misplaced (11.073%); 289 KiB/s, 0 objects/s recovering 2021-03-15T22:29:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: cluster 2021-03-15T22:29:09.154602+0000 osd.7 (osd.7) 119 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:29:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:29:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: 15T22:29:09.184793+0000 osd.3 (osd.3 2021-03-15T22:29:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: ) 105 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:29:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:29:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: 03-15T22:29:09.390023 2021-03-15T22:29:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1660 2021-03-15T22:29:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: : cluster [DBG] osdmap e600: 8 total, 6 up, 8 in 2021-03-15T22:29:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: audit 2021 2021-03-15T22:29:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: -03-15T22:29:09 2021-03-15T22:29:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: .569293+0000 mon.c (mon.1 2021-03-15T22:29:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: ) 53 : audit [DBG] from='client.? 172.21.15.76:0/1834438441' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: 2021-03-15T22:29:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: audit 2021-03-15T22:29:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: 2021-03-15T22:29 2021-03-15T22:29:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: :09.717006+0000 mon.a (mon 2021-03-15T22:29:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: .0) 1661 : audit [INF] from='client.? 172.21.15.76:0/1830620856' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["5"]}]: dispatch 2021-03-15T22:29:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: 2021-03-15T22:29:10.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:29:10.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: -03-15T22:29: 2021-03-15T22:29:10.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: 09.717093+0000 mon.a (mon 2021-03-15T22:29:10.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: .0) 1662 : cluster [INF] 2021-03-15T22:29:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: Client client.admin marked osd.5 out, after it was down for 94 seconds 2021-03-15T22:29:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: audit 2021 2021-03-15T22:29:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: -03-15T22:29: 2021-03-15T22:29:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: 09.915108+0000 mgr.y (mgr 2021-03-15T22:29:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: .14144) 789 : audit [DBG] 2021-03-15T22:29:10.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:10.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: cluster 2021-03-15T22:29:10.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:29:10.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: 29:10.177223+0000 mon.a 2021-03-15T22:29:10.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: (mon.0) 1663 : cluster 2021-03-15T22:29:10.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: [WRN] Health check update: 1 osds down (OSD_DOWN) 2021-03-15T22:29:10.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: audit 2021 2021-03-15T22:29:10.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: -03-15T22:29:10 2021-03-15T22:29:10.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: .181681+0000 mon.a (mon.0 2021-03-15T22:29:10.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: ) 1664 : audit [INF] from='client.? 172.21.15.76:0/1830620856' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["5"]}]': finished 2021-03-15T22:29:10.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: 2021-03-15T22:29:10.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:29:10.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: -03-15T22:29: 2021-03-15T22:29:10.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: 10.181749+0000 mon.a (mon 2021-03-15T22:29:10.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: .0) 1665 : cluster [DBG] 2021-03-15T22:29:10.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:10 smithi125 conmon[56739]: osdmap e601: 8 total, 6 up, 7 in 2021-03-15T22:29:10.871 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:10.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:10.894 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:10.894 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:10.995 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:11.017 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:11.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:11.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:11.120 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:11.143 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:11.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:11.144 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:11.196 INFO:tasks.rados.rados.0.smithi076.stdout:2158: done (1 left) 2021-03-15T22:29:11.197 INFO:tasks.rados.rados.0.smithi076.stdout:2161: done (0 left) 2021-03-15T22:29:11.197 INFO:tasks.rados.rados.0.smithi076.stdout:2162: write oid 22 current snap is 247 2021-03-15T22:29:11.197 INFO:tasks.rados.rados.0.smithi076.stdout:2162: seq_num 753 ranges {655671=581444,2030588=625083} 2021-03-15T22:29:11.201 INFO:tasks.rados.rados.0.smithi076.stdout:2162: writing smithi07694878-22 from 655671 to 1237115 tid 1 2021-03-15T22:29:11.210 INFO:tasks.rados.rados.0.smithi076.stdout:2162: writing smithi07694878-22 from 2030588 to 2655671 tid 2 2021-03-15T22:29:11.211 INFO:tasks.rados.rados.0.smithi076.stdout:2163: copy_from oid 28 from oid 34 current snap is 247 2021-03-15T22:29:11.211 INFO:tasks.rados.rados.0.smithi076.stdout:2164: read oid 20 snap 245 2021-03-15T22:29:11.211 INFO:tasks.rados.rados.0.smithi076.stdout:2164: expect deleted 2021-03-15T22:29:11.212 INFO:tasks.rados.rados.0.smithi076.stdout:2165: write oid 43 current snap is 247 2021-03-15T22:29:11.212 INFO:tasks.rados.rados.0.smithi076.stdout:2165: seq_num 754 ranges {672478=656693,2089070=459917} 2021-03-15T22:29:11.218 INFO:tasks.rados.rados.0.smithi076.stdout:2165: writing smithi07694878-43 from 672478 to 1329171 tid 1 2021-03-15T22:29:11.226 INFO:tasks.rados.rados.0.smithi076.stdout:2165: writing smithi07694878-43 from 2089070 to 2548987 tid 2 2021-03-15T22:29:11.226 INFO:tasks.rados.rados.0.smithi076.stdout:2166: snap_create 2021-03-15T22:29:11.226 INFO:tasks.rados.rados.0.smithi076.stdout:2162: finishing write tid 1 to smithi07694878-22 2021-03-15T22:29:11.226 INFO:tasks.rados.rados.0.smithi076.stdout:2163: finishing copy_from to smithi07694878-28 2021-03-15T22:29:11.227 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 849 (ObjNum 609 snap 189 seq_num 609) dirty exists 2021-03-15T22:29:11.227 INFO:tasks.rados.rados.0.smithi076.stdout:2162: finishing write tid 2 to smithi07694878-22 2021-03-15T22:29:11.227 INFO:tasks.rados.rados.0.smithi076.stdout:2162: finishing write tid 3 to smithi07694878-22 2021-03-15T22:29:11.227 INFO:tasks.rados.rados.0.smithi076.stdout:2162: finishing write tid 4 to smithi07694878-22 2021-03-15T22:29:11.228 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 1500 (ObjNum 753 snap 247 seq_num 753) dirty exists 2021-03-15T22:29:11.228 INFO:tasks.rados.rados.0.smithi076.stdout:2162: left oid 22 (ObjNum 753 snap 247 seq_num 753) 2021-03-15T22:29:11.228 INFO:tasks.rados.rados.0.smithi076.stdout:2165: finishing write tid 1 to smithi07694878-43 2021-03-15T22:29:11.228 INFO:tasks.rados.rados.0.smithi076.stdout:2163: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:29:11.229 INFO:tasks.rados.rados.0.smithi076.stdout:2165: finishing write tid 2 to smithi07694878-43 2021-03-15T22:29:11.229 INFO:tasks.rados.rados.0.smithi076.stdout:2165: finishing write tid 3 to smithi07694878-43 2021-03-15T22:29:11.230 INFO:tasks.rados.rados.0.smithi076.stdout:2165: finishing write tid 4 to smithi07694878-43 2021-03-15T22:29:11.230 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 821 (ObjNum 754 snap 247 seq_num 754) dirty exists 2021-03-15T22:29:11.230 INFO:tasks.rados.rados.0.smithi076.stdout:2165: left oid 43 (ObjNum 754 snap 247 seq_num 754) 2021-03-15T22:29:11.246 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:11.267 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:11.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:11.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:11.371 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:11.393 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:11.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:11.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:11.495 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:11.516 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:11.518 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:11.619 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:11.640 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:11.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:11.641 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:11.742 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:11.764 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:11.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:11.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:11.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:11 smithi076 conmon[56514]: cluster 2021-03-15T22:29:09.816955+0000 osd.4 (osd.4) 126 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:29:11.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:11 smithi076 conmon[56514]: cluster 2021-03-15T22:29:10.153318+0000 osd.6 (osd.6) 94 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:29:11.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:11 smithi076 conmon[56514]: audit 2021-03-15T22:29:11.069972+0000 mon.a (mon.0) 1666 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:11.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:11 smithi076 conmon[56514]: cluster 2021-03-15T22:29:11.183812+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e602: 8 total, 6 up, 7 in 2021-03-15T22:29:11.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:11 smithi076 conmon[62156]: cluster 2021-03-15T22:29:09.816955+0000 osd.4 (osd.4) 126 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:29:11.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:11 smithi076 conmon[62156]: cluster 2021-03-15T22:29:10.153318+0000 osd.6 (osd.6) 94 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:29:11.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:11 smithi076 conmon[62156]: audit 2021-03-15T22:29:11.069972+0000 mon.a (mon.0) 1666 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:11.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:11 smithi076 conmon[62156]: cluster 2021-03-15T22:29:11.183812+0000 mon.a (mon.0) 1667 : cluster [DBG] osdmap e602: 8 total, 6 up, 7 in 2021-03-15T22:29:11.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:11 smithi125 conmon[56739]: cluster 2021-03-15T22:29:09.816955+0000 osd.4 (osd.4) 2021-03-15T22:29:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:11 smithi125 conmon[56739]: 126 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:29:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:11 smithi125 conmon[56739]: cluster 2021-03-15T22:29:10.153318+0000 osd.6 (osd.6) 94 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:29:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:11 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:29:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:11 smithi125 conmon[56739]: -15T22:29:11. 2021-03-15T22:29:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:11 smithi125 conmon[56739]: 069972+0000 mon.a (mon.0) 1666 : audit 2021-03-15T22:29:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:11 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:11 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:29:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:11 smithi125 conmon[56739]: 03-15T22:29:11. 2021-03-15T22:29:11.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:11 smithi125 conmon[56739]: 183812+0000 mon.a (mon.0) 1667 : 2021-03-15T22:29:11.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:11 smithi125 conmon[56739]: cluster [DBG] osdmap e602: 8 total, 6 up, 7 in 2021-03-15T22:29:11.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:11.891 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:11.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:11.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:11.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-15T22:29:11.993 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:12.014 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:12.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:12.116 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:12.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:12.138 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:12.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:12.194 INFO:tasks.rados.rados.0.smithi076.stdout:2162: done (4 left) 2021-03-15T22:29:12.194 INFO:tasks.rados.rados.0.smithi076.stdout:2163: done (3 left) 2021-03-15T22:29:12.194 INFO:tasks.rados.rados.0.smithi076.stdout:2164: done (2 left) 2021-03-15T22:29:12.195 INFO:tasks.rados.rados.0.smithi076.stdout:2165: done (1 left) 2021-03-15T22:29:12.195 INFO:tasks.rados.rados.0.smithi076.stdout:2166: done (0 left) 2021-03-15T22:29:12.195 INFO:tasks.rados.rados.0.smithi076.stdout:2167: rollback oid 16 current snap is 248 2021-03-15T22:29:12.195 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 237 2021-03-15T22:29:12.196 INFO:tasks.rados.rados.0.smithi076.stdout:2168: read oid 29 snap -1 2021-03-15T22:29:12.196 INFO:tasks.rados.rados.0.smithi076.stdout:2168: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:29:12.196 INFO:tasks.rados.rados.0.smithi076.stdout:2169: write oid 26 current snap is 248 2021-03-15T22:29:12.196 INFO:tasks.rados.rados.0.smithi076.stdout:2169: seq_num 755 ranges {689285=731942,2147552=541733} 2021-03-15T22:29:12.200 INFO:tasks.rados.rados.0.smithi076.stdout:2169: writing smithi07694878-26 from 689285 to 1421227 tid 1 2021-03-15T22:29:12.212 INFO:tasks.rados.rados.0.smithi076.stdout:2169: writing smithi07694878-26 from 2147552 to 2689285 tid 2 2021-03-15T22:29:12.213 INFO:tasks.rados.rados.0.smithi076.stdout:2167: finishing rollback tid 0 to smithi07694878-16 2021-03-15T22:29:12.213 INFO:tasks.rados.rados.0.smithi076.stdout:2167: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:29:12.214 INFO:tasks.rados.rados.0.smithi076.stdout:2167: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:29:12.214 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1504 (ObjNum 699 snap 225 seq_num 699) dirty exists 2021-03-15T22:29:12.214 INFO:tasks.rados.rados.0.smithi076.stdout:2168: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:29:12.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:12.271 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:12.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:12.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:12.373 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:12.375 INFO:tasks.rados.rados.0.smithi076.stdout:2167: done (2 left) 2021-03-15T22:29:12.375 INFO:tasks.rados.rados.0.smithi076.stdout:2168: done (1 left) 2021-03-15T22:29:12.376 INFO:tasks.rados.rados.0.smithi076.stdout:2170: read oid 22 snap -1 2021-03-15T22:29:12.376 INFO:tasks.rados.rados.0.smithi076.stdout:2170: expect (ObjNum 753 snap 247 seq_num 753) 2021-03-15T22:29:12.376 INFO:tasks.rados.rados.0.smithi076.stdout:2171: snap_remove snap 236 2021-03-15T22:29:12.395 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:12.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:12.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:12.499 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:12.521 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:12.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:12.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:12.544 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:29:12.625 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:12.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:12.650 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:12.650 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:12.752 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:12.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[62156]: cluster 2021-03-15T22:29:10.861243+0000 osd.4 (osd 2021-03-15T22:29:12.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[62156]: .4) 127 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:29:12.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[62156]: cluster 2021-03-15T22:29:11.121539+0000 2021-03-15T22:29:12.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[62156]: mgr.y (mgr.14144) 790 : cluster [DBG] pgmap v1046: 138 pgs: 9 peering, 28 active+clean+remapped, 1 undersized+degraded+peered, 2 active+undersized+remapped, 14 active+undersized+degraded, 5 undersized+peered, 25 active+undersized, 54 active+clean; 582 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 12 op/s; 172/1767 objects degraded (9.734%); 169/1767 objects misplaced (9.564%) 2021-03-15T22:29:12.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[62156]: cluster 2021-03-15T22:29:12.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[62156]: 2021-03-15T22:29:11.129312+0000 osd.6 (osd.6) 95 : cluster [DBG] 3.a scrub ok 2021-03-15T22:29:12.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:12.764 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[62156]: 03-15T22:29:12.189441+0000 mon.a (mon.0 2021-03-15T22:29:12.764 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[62156]: ) 1668 : cluster [DBG] osdmap e603: 8 total, 6 up, 7 in 2021-03-15T22:29:12.764 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[56514]: cluster 2021-03-15T22:29:10.861243+0000 osd.4 (osd.4) 127 : cluster 2021-03-15T22:29:12.764 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[56514]: [DBG] 4.9 scrub ok 2021-03-15T22:29:12.765 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[56514]: cluster 2021-03-15T22:29:11.121539+0000 mgr.y (mgr.14144) 790 : cluster [DBG] pgmap v1046: 138 pgs: 9 peering, 28 active+clean+remapped, 1 undersized+degraded+peered, 2 active+undersized+remapped, 14 active+undersized+degraded, 5 undersized+peered, 25 active+undersized, 54 active+clean; 582 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 12 op/s; 172/1767 objects degraded (9.734%); 169/1767 objects misplaced (9.564%) 2021-03-15T22:29:12.765 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[56514]: cluster 2021-03-15T22:29:11.129312+0000 osd.6 (osd.6) 95 : cluster [DBG] 3.a scrub ok 2021-03-15T22:29:12.765 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:12 smithi076 conmon[56514]: cluster 2021-03-15T22:29:12.189441+0000 mon.a (mon.0) 1668 : cluster [DBG] osdmap e603: 8 total, 6 up, 7 in 2021-03-15T22:29:12.768 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:12.779 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:12.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:12.781 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:12.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: cluster 2021-03-15T22:29:10.861243+0000 2021-03-15T22:29:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: osd.4 (osd.4) 127 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:29:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:29:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: 03-15T22:29:11 2021-03-15T22:29:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: .121539+0000 mgr.y (mgr.14144) 790 2021-03-15T22:29:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: : cluster [DBG] pgmap v1046: 138 pgs: 9 peering, 28 active+clean+remapped, 1 undersized+degraded+peered, 2 active+undersized+remapped, 14 active+undersized+degraded, 5 undersized+peered, 25 active+undersized, 54 active+clean; 582 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 2.9 MiB/s wr, 12 op/s; 172/1767 objects degraded (9.734%); 169/1767 objects misplaced (9.564%) 2021-03-15T22:29:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:29:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: -03-15T22:29: 2021-03-15T22:29:12.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: 11.129312+0000 osd.6 (osd.6) 2021-03-15T22:29:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: 95 : cluster [DBG] 3.a scrub ok 2021-03-15T22:29:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: cluster 2021-03-15T22:29:12.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: 2021-03-15T22:29 2021-03-15T22:29:12.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: :12.189441+0000 mon.a (mon.0 2021-03-15T22:29:12.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:12 smithi125 conmon[56739]: ) 1668 : cluster [DBG] osdmap e603: 8 total, 6 up, 7 in 2021-03-15T22:29:12.882 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:12.904 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:12.905 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:12.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:12.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:13.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:13.030 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:13.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:13.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:13.133 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:13.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:13.160 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:13.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:13.195 INFO:tasks.rados.rados.0.smithi076.stdout:2171: done (2 left) 2021-03-15T22:29:13.195 INFO:tasks.rados.rados.0.smithi076.stdout:2172: read oid 40 snap -1 2021-03-15T22:29:13.196 INFO:tasks.rados.rados.0.smithi076.stdout:2172: expect (ObjNum 737 snap 243 seq_num 737) 2021-03-15T22:29:13.196 INFO:tasks.rados.rados.0.smithi076.stdout:2173: write (excl) oid 16 current snap is 248 2021-03-15T22:29:13.196 INFO:tasks.rados.rados.0.smithi076.stdout:2173: seq_num 756 ranges {706092=407191,1806034=463586} 2021-03-15T22:29:13.201 INFO:tasks.rados.rados.0.smithi076.stdout:2173: writing smithi07694878-16 from 706092 to 1113283 tid 1 2021-03-15T22:29:13.209 INFO:tasks.rados.rados.0.smithi076.stdout:2173: writing smithi07694878-16 from 1806034 to 2269620 tid 2 2021-03-15T22:29:13.209 INFO:tasks.rados.rados.0.smithi076.stdout:2174: snap_create 2021-03-15T22:29:13.209 INFO:tasks.rados.rados.0.smithi076.stdout:2169: finishing write tid 1 to smithi07694878-26 2021-03-15T22:29:13.209 INFO:tasks.rados.rados.0.smithi076.stdout:2169: finishing write tid 2 to smithi07694878-26 2021-03-15T22:29:13.209 INFO:tasks.rados.rados.0.smithi076.stdout:2169: finishing write tid 3 to smithi07694878-26 2021-03-15T22:29:13.210 INFO:tasks.rados.rados.0.smithi076.stdout:2169: finishing write tid 4 to smithi07694878-26 2021-03-15T22:29:13.210 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 814 (ObjNum 755 snap 248 seq_num 755) dirty exists 2021-03-15T22:29:13.210 INFO:tasks.rados.rados.0.smithi076.stdout:2169: left oid 26 (ObjNum 755 snap 248 seq_num 755) 2021-03-15T22:29:13.210 INFO:tasks.rados.rados.0.smithi076.stdout:2170: expect (ObjNum 753 snap 247 seq_num 753) 2021-03-15T22:29:13.262 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:13.284 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:13.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:13.286 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:13.344 INFO:tasks.rados.rados.0.smithi076.stdout:2172: expect (ObjNum 737 snap 243 seq_num 737) 2021-03-15T22:29:13.387 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:13.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:13.411 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:13.411 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:13.488 INFO:tasks.rados.rados.0.smithi076.stdout:2173: finishing write tid 1 to smithi07694878-16 2021-03-15T22:29:13.489 INFO:tasks.rados.rados.0.smithi076.stdout:2173: finishing write tid 2 to smithi07694878-16 2021-03-15T22:29:13.489 INFO:tasks.rados.rados.0.smithi076.stdout:2173: finishing write tid 3 to smithi07694878-16 2021-03-15T22:29:13.489 INFO:tasks.rados.rados.0.smithi076.stdout:2173: finishing write tid 4 to smithi07694878-16 2021-03-15T22:29:13.489 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1513 (ObjNum 756 snap 248 seq_num 756) dirty exists 2021-03-15T22:29:13.490 INFO:tasks.rados.rados.0.smithi076.stdout:2173: left oid 16 (ObjNum 756 snap 248 seq_num 756) 2021-03-15T22:29:13.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:13.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:13.536 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:13.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:13.638 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:13.661 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:13.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:13.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:13.764 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:13.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[56514]: cluster 2021-03-15T22:29:12. 2021-03-15T22:29:13.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[56514]: 186439+0000 osd.7 (osd.7) 120 : cluster [DBG] 5.d starting backfill to osd.0 from (0'0,0'0] MAX to 491'5 2021-03-15T22:29:13.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[56514]: cluster 2021-03-15T22:29:12.845797+0000 osd.0 (osd.0) 90 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:29:13.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[56514]: cluster 2021-03-15T22:29:13.188424+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e604: 8 total, 6 up, 7 in 2021-03-15T22:29:13.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[62156]: cluster 2021-03-15T22:29:12.186439 2021-03-15T22:29:13.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[62156]: +0000 osd.7 (osd.7) 120 : cluster [DBG] 5.d starting backfill to osd.0 from (0'0,0'0] MAX to 491'5 2021-03-15T22:29:13.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[62156]: cluster 2021-03-15T22:29 2021-03-15T22:29:13.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[62156]: :12.845797+0000 osd.0 (osd 2021-03-15T22:29:13.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[62156]: .0) 90 : cluster [DBG] 2021-03-15T22:29:13.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[62156]: 5.3 scrub ok 2021-03-15T22:29:13.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:13.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[62156]: 03-15T22:29:13.188424+0000 2021-03-15T22:29:13.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[62156]: mon.a (mon.0) 1669 : cluster 2021-03-15T22:29:13.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:13 smithi076 conmon[62156]: [DBG] osdmap e604: 8 total, 6 up, 7 in 2021-03-15T22:29:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:13 smithi125 conmon[56739]: cluster 2021-03-15T22:29:12.186439+0000 osd.7 (osd.7) 120 : cluster [DBG] 5.d starting backfill to osd.0 from (0'0,0'0] MAX to 491'5 2021-03-15T22:29:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:13 smithi125 conmon[56739]: cluster 2021-03-15T22:29:12.845797+0000 2021-03-15T22:29:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:13 smithi125 conmon[56739]: osd.0 (osd.0) 90 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:29:13.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:13 smithi125 conmon[56739]: cluster 2021-03-15T22:29:13.188424+0000 mon.a (mon.0) 1669 : cluster [DBG] osdmap e604: 8 total, 6 up, 7 in 2021-03-15T22:29:13.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:13.792 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:13.792 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:13.894 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:13.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:13.917 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:13.917 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:14.019 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:14.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:14.044 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:14.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:14.146 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:14.171 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:14.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:14.173 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:14.194 INFO:tasks.rados.rados.0.smithi076.stdout:2169: done (4 left) 2021-03-15T22:29:14.194 INFO:tasks.rados.rados.0.smithi076.stdout:2170: done (3 left) 2021-03-15T22:29:14.194 INFO:tasks.rados.rados.0.smithi076.stdout:2172: done (2 left) 2021-03-15T22:29:14.195 INFO:tasks.rados.rados.0.smithi076.stdout:2173: done (1 left) 2021-03-15T22:29:14.195 INFO:tasks.rados.rados.0.smithi076.stdout:2174: done (0 left) 2021-03-15T22:29:14.195 INFO:tasks.rados.rados.0.smithi076.stdout:2175: snap_remove snap 153 2021-03-15T22:29:14.197 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:29:14.274 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:14.298 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:14.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:14.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:14.401 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:14.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:14.424 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:14.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:14.525 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:14.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:14.551 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:14.552 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:14.653 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:14.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: cluster 2021-03-15T22:29:12.829447+0000 osd.4 (osd.4 2021-03-15T22:29:14.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: ) 128 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:29:14.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: cluster 2021-03-15T22:29:13.122117+0000 mgr.y (mgr.14144) 791 : cluster [DBG] pgmap v1049: 138 pgs: 1 active+recovering+degraded, 9 peering, 18 active+clean+remapped, 1 undersized+degraded+peered, 11 active+undersized+degraded, 3 undersized+peered, 23 active+undersized, 72 active+clean; 585 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.9 MiB/s rd, 2.9 MiB/s wr, 12 op/s; 180/1770 objects degraded (10.169%); 93/1770 objects misplaced (5.254%); 575 B/s, 6 objects/s recovering 2021-03-15T22:29:14.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: cluster 2021-03-15T22:29:13.250074+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:29:14.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: cluster 2021-03-15T22:29:13.284767+0000 osd.1 (osd.1) 114 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:29:14.665 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: cluster 2021-03-15T22:29:13.390571+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check update: Reduced data availability: 4 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:29:14.665 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: cluster 2021-03-15T22:29:13.390601+0000 mon.a (mon.0) 1671 : cluster [WRN] Health check update: Degraded data redundancy: 180/1770 objects degraded (10.169%), 13 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:14.665 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: audit 2021-03-15T22:29:13.589491+0000 mon.a (mon.0) 1672 : audit [INF] from='client.? 172.21.15.76:0/3281195256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:14.666 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:29:14.666 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: -15T22:29:14.185749+0000 mon.a (mon.0) 1673 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:29:14.667 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: audit 2021-03-15T22:29:14.189763+0000 mon.a (mon.0) 1674 : audit [INF] 2021-03-15T22:29:14.667 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/3281195256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:14.667 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: cluster 2021-03-15T22:29:14.668 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: 2021-03-15T22:29:14.189939+0000 mon.a (mon. 2021-03-15T22:29:14.668 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[56514]: 0) 1675 : cluster [DBG] osdmap e605: 8 total, 6 up, 7 in 2021-03-15T22:29:14.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: cluster 2021-03-15T22:29: 2021-03-15T22:29:14.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: 12.829447+0000 osd.4 (osd.4) 128 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:29:14.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:29:14.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: 22:29:13.122117+0000 mgr.y (mgr.14144) 791 : cluster [DBG] pgmap v1049: 138 pgs: 1 active+recovering+degraded, 9 peering, 18 active+clean+remapped, 1 undersized+degraded+peered, 11 active+undersized+degraded, 3 undersized+peered, 23 active+undersized, 72 active+clean; 585 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.9 MiB/s rd, 2.9 MiB/s wr, 12 op/s; 180/1770 objects degraded (10.169%); 93/1770 objects misplaced (5.254%); 575 B/s, 6 objects/s recovering 2021-03-15T22:29:14.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:29:14.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: 15T22:29:13.250074+0000 osd.3 (osd 2021-03-15T22:29:14.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: .3) 106 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:29:14.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:14.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: 03-15T22:29:13.284767+0000 2021-03-15T22:29:14.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: osd.1 (osd.1) 114 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:29:14.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: 2021-03-15T22:29:14.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: cluster 2021-03-15T22:29:14.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: 2021-03-15T22:29:13.390571+0000 mon.a 2021-03-15T22:29:14.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: (mon.0) 1670 : cluster [WRN] Health check update: Reduced data availability: 4 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:29:14.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:29:14.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: -15T22:29:13.390601+0000 mon.a ( 2021-03-15T22:29:14.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: mon.0) 1671 : cluster [WRN] Health check update: Degraded data redundancy: 180/1770 objects degraded (10.169%), 13 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:14.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:29:14.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: 15T22:29:13.589491+0000 mon.a (mon 2021-03-15T22:29:14.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: .0) 1672 : audit [INF] from='client.? 172.21.15.76:0/3281195256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:14.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:29:14.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: -03-15T22:29:14.185749 2021-03-15T22:29:14.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1673 : cluster [INF] 2021-03-15T22:29:14.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:29:14.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: audit 2021-03-15T22:29 2021-03-15T22:29:14.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: :14.189763+0000 mon.a (mon.0) 1674 : 2021-03-15T22:29:14.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/3281195256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:14.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: cluster 2021-03-15T22:29:14 2021-03-15T22:29:14.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: .189939+0000 mon.a (mon.0) 1675 : cluster 2021-03-15T22:29:14.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:14 smithi076 conmon[62156]: [DBG] osdmap e605: 8 total, 6 up, 7 in 2021-03-15T22:29:14.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:14.678 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:14.678 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:14.779 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: cluster 2021-03-15T22:29:12.829447+0000 osd.4 (osd.4) 128 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:29:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:29:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: 03-15T22:29:13.122117+0000 mgr.y (mgr.14144) 791 : cluster [DBG] pgmap v1049: 138 pgs: 1 active+recovering+degraded, 9 peering, 18 active+clean+remapped, 1 undersized+degraded+peered, 11 active+undersized+degraded, 3 undersized+peered, 23 active+undersized, 72 active+clean; 585 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.9 MiB/s rd, 2.9 MiB/s wr, 12 op/s; 180/1770 objects degraded (10.169%); 93/1770 objects misplaced (5.254%); 575 B/s, 6 objects/s recovering 2021-03-15T22:29:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: cluster 2021-03-15T22:29:13.250074+0000 osd.3 (osd.3) 106 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:29:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: cluster 2021-03-15T22:29:13.284767+0000 osd.1 (osd.1) 114 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:29:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: cluster 2021-03-15T22:29:13.390571+0000 mon.a (mon.0) 1670 : cluster [WRN] Health check update: Reduced data availability: 4 pgs inactive (PG_AVAILABILITY) 2021-03-15T22:29:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: cluster 2021-03-15T22:29:13.390601+0000 mon.a (mon.0) 1671 : 2021-03-15T22:29:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: cluster [WRN] Health check update: Degraded data redundancy: 180/1770 objects degraded (10.169%), 13 pgs degraded, 28 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: audit 2021-03-15T22:29:13.589491+0000 mon.a (mon.0) 1672 : audit [INF] 2021-03-15T22:29:14.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/3281195256' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:14.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:29:14.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: 22:29:14.185749+0000 mon.a (mon.0) 1673 : cluster 2021-03-15T22:29:14.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:29:14.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: audit 2021-03-15T22:29:14.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: 2021-03-15T22:29:14.189763+0000 2021-03-15T22:29:14.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: mon.a (mon.0) 1674 : audit [INF] 2021-03-15T22:29:14.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/3281195256' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:14.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:29:14.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: -03-15T22:29:14.189939+0000 mon.a ( 2021-03-15T22:29:14.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:14 smithi125 conmon[56739]: mon.0) 1675 : cluster [DBG] osdmap e605: 8 total, 6 up, 7 in 2021-03-15T22:29:14.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:14.803 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:14.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:14.905 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:14.926 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:14.928 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:15.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:15.050 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:15.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:15.052 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:15.153 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:15.175 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:15.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:15.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:15.204 INFO:tasks.rados.rados.0.smithi076.stdout:2175: done (0 left) 2021-03-15T22:29:15.204 INFO:tasks.rados.rados.0.smithi076.stdout:2176: read oid 12 snap -1 2021-03-15T22:29:15.205 INFO:tasks.rados.rados.0.smithi076.stdout:2176: expect deleted 2021-03-15T22:29:15.205 INFO:tasks.rados.rados.0.smithi076.stdout:2177: snap_create 2021-03-15T22:29:15.278 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:15.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:15.304 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:15.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:15.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:15.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:15.429 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:15.430 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:15.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[56514]: cluster 2021-03-15T22:29:13.911814+0000 osd.1 (osd.1) 115 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:15.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[56514]: cluster 2021-03-15T22:29:13.912636+0000 osd.1 (osd.1) 116 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:15.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[56514]: cluster 2021-03-15T22:29:14.052735+0000 osd.7 (osd.7) 121 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:29:15.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[56514]: cluster 2021-03-15T22:29:14.124820+0000 osd.6 (osd.6) 96 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:29:15.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[56514]: cluster 2021-03-15T22:29:14.776696+0000 osd.0 (osd.0) 91 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:29:15.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[56514]: cluster 2021-03-15T22:29:15.199394+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e606: 8 total, 6 up, 7 in 2021-03-15T22:29:15.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[62156]: cluster 2021-03-15T22:29:13.911814 2021-03-15T22:29:15.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 115 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:15.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[62156]: cluster 2021-03-15T22:29:13.912636+0000 osd.1 (osd.1) 116 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:15.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[62156]: cluster 2021-03-15T22:29:14.052735+0000 osd.7 (osd.7) 121 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:29:15.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[62156]: cluster 2021-03-15T22:29:14.124820+0000 osd.6 (osd.6) 96 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:29:15.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[62156]: cluster 2021-03-15T22:29:14.776696+0000 osd.0 (osd.0) 91 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:29:15.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:15 smithi076 conmon[62156]: cluster 2021-03-15T22:29:15.199394+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e606: 8 total, 6 up, 7 in 2021-03-15T22:29:15.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:15.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:15.554 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:15.555 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:15.656 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:15.678 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:15.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:15.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:15.780 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:15.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: cluster 2021-03-15T22:29:13.911814+0000 osd.1 (osd.1) 115 2021-03-15T22:29:15.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: cluster 2021-03-15T22:29:13.912636+0000 osd.1 (osd.1) 116 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: cluster 2021-03-15T22:29:14.052735+0000 osd.7 (osd.7) 121 : cluster 2021-03-15T22:29:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: [DBG] 4.8 scrub ok 2021-03-15T22:29:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:29:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: -03-15T22:29:14.124820+0000 osd.6 (osd.6) 96 : cluster [DBG] 2021-03-15T22:29:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: 4.1f scrub ok 2021-03-15T22:29:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:29:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: -15T22:29:14.776696+0000 osd.0 (osd.0) 91 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:29:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: 2021-03-15T22:29:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:29:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:15 smithi125 conmon[56739]: T22:29:15.199394+0000 mon.a (mon.0) 1676 : cluster [DBG] osdmap e606: 8 total, 6 up, 7 in 2021-03-15T22:29:15.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:15.805 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:15.806 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:15.873 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:29:15.873 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:29:15.907 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:15.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:15.932 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:15.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:15.976 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:29:15.994 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:29:16.011 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:29:16.031 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:29:16.034 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:16.050 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:29:16.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:16.065 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:16.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:16.069 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:29:16.069 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:29:16.069 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:29:16.070 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:29:16.070 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:29:16.070 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:29:16.071 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:29:16.071 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:29:16.071 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:29:16.169 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:16.191 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:16.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:16.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:16.203 INFO:tasks.rados.rados.0.smithi076.stdout:2176: done (1 left) 2021-03-15T22:29:16.203 INFO:tasks.rados.rados.0.smithi076.stdout:2177: done (0 left) 2021-03-15T22:29:16.204 INFO:tasks.rados.rados.0.smithi076.stdout:2178: write (excl) oid 48 current snap is 250 2021-03-15T22:29:16.204 INFO:tasks.rados.rados.0.smithi076.stdout:2178: seq_num 757 ranges {722899=482440,1748163=723597} 2021-03-15T22:29:16.210 INFO:tasks.rados.rados.0.smithi076.stdout:2178: writing smithi07694878-48 from 722899 to 1205339 tid 1 2021-03-15T22:29:16.219 INFO:tasks.rados.rados.0.smithi076.stdout:2178: writing smithi07694878-48 from 1748163 to 2471760 tid 2 2021-03-15T22:29:16.220 INFO:tasks.rados.rados.0.smithi076.stdout:2179: read oid 32 snap -1 2021-03-15T22:29:16.220 INFO:tasks.rados.rados.0.smithi076.stdout:2179: expect (ObjNum 717 snap 233 seq_num 717) 2021-03-15T22:29:16.220 INFO:tasks.rados.rados.0.smithi076.stdout:2180: copy_from oid 39 from oid 24 current snap is 250 2021-03-15T22:29:16.220 INFO:tasks.rados.rados.0.smithi076.stdout:2178: finishing write tid 1 to smithi07694878-48 2021-03-15T22:29:16.220 INFO:tasks.rados.rados.0.smithi076.stdout:2181: snap_remove snap 248 2021-03-15T22:29:16.294 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:16.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:16.320 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:16.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:16.421 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:16.443 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:16.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:16.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:16.546 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:16.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:16.569 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:16.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:16.670 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:16.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:16.694 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:16.694 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[56514]: cluster 2021-03-15T22:29:14.888006+0000 osd.4 (osd.4) 129 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:29:16.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[56514]: cluster 2021-03-15T22:29:15.122915+0000 mgr.y (mgr.14144) 792 : cluster [DBG] pgmap v1052: 138 pgs: 1 active+recovering+degraded, 9 peering, 17 active+clean+remapped, 11 active+undersized+degraded, 24 active+undersized, 76 active+clean; 585 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.8 MiB/s rd, 819 KiB/s wr, 4 op/s; 169/1770 objects degraded (9.548%); 85/1770 objects misplaced (4.802%); 719 B/s, 9 objects/s recovering 2021-03-15T22:29:16.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[56514]: cluster 2021-03-15T22:29:15.297469+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.f scrub ok 2021-03-15T22:29:16.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[56514]: cluster 2021-03-15T22:29:15.400961+0000 mon.a (mon.0) 1677 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive) 2021-03-15T22:29:16.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[56514]: audit 2021-03-15T22:29:15.957266+0000 mon.a (mon.0) 1678 : audit 2021-03-15T22:29:16.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[56514]: [DBG] from='client.? 172.21.15.76:0/1318102929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:16.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[56514]: cluster 2021-03-15T22:29:16.200256+0000 mon.a (mon.0) 1679 : cluster [DBG] osdmap e607: 8 total, 6 up, 7 in 2021-03-15T22:29:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[62156]: cluster 2021-03-15T22:29:14.888006+0000 osd.4 (osd.4) 129 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:29:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[62156]: cluster 2021-03-15T22:29:15.122915+0000 mgr.y (mgr.14144) 792 : cluster [DBG] pgmap v1052: 138 pgs: 1 active+recovering+degraded, 9 peering, 17 active+clean+remapped, 11 active+undersized+degraded, 24 active+undersized, 76 active+clean; 585 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.8 MiB/s rd, 819 KiB/s wr, 4 op/s; 169/1770 objects degraded (9.548%); 85/1770 objects misplaced (4.802%); 719 B/s, 9 objects/s recovering 2021-03-15T22:29:16.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[62156]: cluster 2021-03-15T22:29:15.297469+0000 osd.3 (osd.3) 107 : cluster [DBG] 4.f scrub ok 2021-03-15T22:29:16.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[62156]: cluster 2021-03-15T22:29:15.400961+0000 mon.a (mon.0) 1677 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive) 2021-03-15T22:29:16.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[62156]: audit 2021-03-15T22:29:15.957266+0000 mon.a (mon.0) 1678 : audit [DBG] from='client.? 172.21.15.76:0/1318102929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:16.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[62156]: cluster 2021-03-15T22:29:16.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:16 smithi076 conmon[62156]: 2021-03-15T22:29:16.200256+0000 mon.a (mon.0) 1679 : cluster [DBG] osdmap e607: 8 total, 6 up, 7 in 2021-03-15T22:29:16.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: cluster 2021-03-15T22:29:14.888006+0000 osd.4 (osd.4) 129 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:29:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: cluster 2021-03-15T22:29:15.122915+0000 mgr.y (mgr.14144) 792 : cluster [DBG] pgmap v1052: 138 pgs: 1 active+recovering+degraded, 9 peering, 17 active+clean+remapped, 11 active+undersized+degraded, 24 active+undersized, 76 active+clean; 585 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.8 MiB/s rd, 819 KiB/s wr, 4 op/s; 169/1770 objects degraded (9.548%); 85/1770 objects misplaced (4.802%); 719 B/s, 9 objects/s recovering 2021-03-15T22:29:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: cluster 2021-03-15T22:29: 2021-03-15T22:29:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: 15.297469+0000 osd.3 (osd.3 2021-03-15T22:29:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: ) 107 : cluster [DBG] 4.f scrub ok 2021-03-15T22:29:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:29:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: 03-15T22:29:15.400961 2021-03-15T22:29:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2021-03-15T22:29:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: 1677 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive) 2021-03-15T22:29:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: audit 2021 2021-03-15T22:29:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: -03-15T22:29:15 2021-03-15T22:29:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: .957266+0000 mon.a (mon.0 2021-03-15T22:29:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: ) 1678 : audit [DBG] from='client.? 172.21.15.76:0/1318102929' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: 2021-03-15T22:29:16.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:29:16.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: 22:29:16.200256+0000 2021-03-15T22:29:16.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: mon.a (mon.0) 1679 : cluster 2021-03-15T22:29:16.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:16 smithi125 conmon[56739]: [DBG] osdmap e607: 8 total, 6 up, 7 in 2021-03-15T22:29:16.795 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:16.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:16.818 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:16.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:16.920 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:16.943 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:16.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:16.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:17.021 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-15T22:29:17.021 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 0, 3] out_osds: [5] dead_osds: [2, 5] live_osds: [0, 1, 3, 4, 6, 7] 2021-03-15T22:29:17.022 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:29:17.022 INFO:tasks.thrashosds.thrasher:Reviving osd 5 2021-03-15T22:29:17.022 INFO:tasks.cephadm.osd.5:Restarting osd.5 (starting--it wasn't running)... 2021-03-15T22:29:17.023 DEBUG:teuthology.orchestra.run.smithi125:osd.5> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.5.service 2021-03-15T22:29:17.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo systemctl start ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.5 2021-03-15T22:29:17.046 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:17.055 INFO:journalctl@ceph.osd.5.smithi125.stdout:-- Logs begin at Mon 2021-03-15 21:44:18 UTC. -- 2021-03-15T22:29:17.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:17.071 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:17.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:17.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:17.182 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:29:17.183 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:29:17.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:17.197 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:17.197 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:17.207 INFO:tasks.rados.rados.0.smithi076.stdout:2181: done (3 left) 2021-03-15T22:29:17.207 INFO:tasks.rados.rados.0.smithi076.stdout:2182: rollback oid 37 current snap is 250 2021-03-15T22:29:17.207 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 37 to 247 2021-03-15T22:29:17.208 INFO:tasks.rados.rados.0.smithi076.stdout:2183: write oid 43 current snap is 250 2021-03-15T22:29:17.208 INFO:tasks.rados.rados.0.smithi076.stdout:2183: seq_num 758 ranges {739706=557689,1806645=467255} 2021-03-15T22:29:17.214 INFO:tasks.rados.rados.0.smithi076.stdout:2183: writing smithi07694878-43 from 739706 to 1297395 tid 1 2021-03-15T22:29:17.220 INFO:tasks.rados.rados.0.smithi076.stdout:2183: writing smithi07694878-43 from 1806645 to 2273900 tid 2 2021-03-15T22:29:17.221 INFO:tasks.rados.rados.0.smithi076.stdout:2184: write oid 46 current snap is 250 2021-03-15T22:29:17.221 INFO:tasks.rados.rados.0.smithi076.stdout:2180: finishing copy_from to smithi07694878-39 2021-03-15T22:29:17.221 INFO:tasks.rados.rados.0.smithi076.stdout:2180: got expected ENOENT (src dne) 2021-03-15T22:29:17.221 INFO:tasks.rados.rados.0.smithi076.stdout:2184: seq_num 759 ranges {756513=632938,1865127=727266} 2021-03-15T22:29:17.227 INFO:tasks.rados.rados.0.smithi076.stdout:2184: writing smithi07694878-46 from 756513 to 1389451 tid 1 2021-03-15T22:29:17.235 INFO:tasks.rados.rados.0.smithi076.stdout:2184: writing smithi07694878-46 from 1865127 to 2592393 tid 2 2021-03-15T22:29:17.235 INFO:tasks.rados.rados.0.smithi076.stdout:2185: read oid 8 snap -1 2021-03-15T22:29:17.235 INFO:tasks.rados.rados.0.smithi076.stdout:2185: expect (ObjNum 729 snap 238 seq_num 729) 2021-03-15T22:29:17.236 INFO:tasks.rados.rados.0.smithi076.stdout:2186: rollback oid 49 current snap is 250 2021-03-15T22:29:17.236 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 49 to 216 2021-03-15T22:29:17.236 INFO:tasks.rados.rados.0.smithi076.stdout:2180: finishing copy_from racing read to smithi07694878-39 2021-03-15T22:29:17.236 INFO:tasks.rados.rados.0.smithi076.stdout:2178: finishing write tid 2 to smithi07694878-48 2021-03-15T22:29:17.237 INFO:tasks.rados.rados.0.smithi076.stdout:2179: expect (ObjNum 717 snap 233 seq_num 717) 2021-03-15T22:29:17.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:17.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:17.328 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:17.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:17.350 INFO:tasks.rados.rados.0.smithi076.stdout:2178: finishing write tid 3 to smithi07694878-48 2021-03-15T22:29:17.350 INFO:tasks.rados.rados.0.smithi076.stdout:2178: finishing write tid 4 to smithi07694878-48 2021-03-15T22:29:17.351 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 1197 (ObjNum 757 snap 250 seq_num 757) dirty exists 2021-03-15T22:29:17.351 INFO:tasks.rados.rados.0.smithi076.stdout:2178: left oid 48 (ObjNum 757 snap 250 seq_num 757) 2021-03-15T22:29:17.351 INFO:tasks.rados.rados.0.smithi076.stdout:2182: finishing rollback tid 0 to smithi07694878-37 2021-03-15T22:29:17.351 INFO:tasks.rados.rados.0.smithi076.stdout:2182: finishing rollback tid 1 to smithi07694878-37 2021-03-15T22:29:17.351 INFO:tasks.rados.rados.0.smithi076.stdout:2182: finishing rollback tid 2 to smithi07694878-37 2021-03-15T22:29:17.352 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 1521 (ObjNum 751 snap 245 seq_num 751) dirty exists 2021-03-15T22:29:17.352 INFO:tasks.rados.rados.0.smithi076.stdout:2183: finishing write tid 1 to smithi07694878-43 2021-03-15T22:29:17.352 INFO:tasks.rados.rados.0.smithi076.stdout:2178: done (7 left) 2021-03-15T22:29:17.352 INFO:tasks.rados.rados.0.smithi076.stdout:2179: done (6 left) 2021-03-15T22:29:17.352 INFO:tasks.rados.rados.0.smithi076.stdout:2180: done (5 left) 2021-03-15T22:29:17.354 INFO:tasks.rados.rados.0.smithi076.stdout:2182: done (4 left) 2021-03-15T22:29:17.354 INFO:tasks.rados.rados.0.smithi076.stdout:2187: copy_from oid 40 from oid 11 current snap is 250 2021-03-15T22:29:17.354 INFO:tasks.rados.rados.0.smithi076.stdout:2183: finishing write tid 2 to smithi07694878-43 2021-03-15T22:29:17.354 INFO:tasks.rados.rados.0.smithi076.stdout:2183: finishing write tid 3 to smithi07694878-43 2021-03-15T22:29:17.355 INFO:tasks.rados.rados.0.smithi076.stdout:2183: finishing write tid 4 to smithi07694878-43 2021-03-15T22:29:17.355 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 824 (ObjNum 758 snap 250 seq_num 758) dirty exists 2021-03-15T22:29:17.355 INFO:tasks.rados.rados.0.smithi076.stdout:2183: left oid 43 (ObjNum 758 snap 250 seq_num 758) 2021-03-15T22:29:17.355 INFO:tasks.rados.rados.0.smithi076.stdout:2184: finishing write tid 1 to smithi07694878-46 2021-03-15T22:29:17.356 INFO:tasks.rados.rados.0.smithi076.stdout:2186: finishing rollback tid 0 to smithi07694878-49 2021-03-15T22:29:17.356 INFO:tasks.rados.rados.0.smithi076.stdout:2186: finishing rollback tid 1 to smithi07694878-49 2021-03-15T22:29:17.356 INFO:tasks.rados.rados.0.smithi076.stdout:2186: finishing rollback tid 2 to smithi07694878-49 2021-03-15T22:29:17.356 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 1529 (ObjNum 543 snap 172 seq_num 543) dirty exists 2021-03-15T22:29:17.356 INFO:tasks.rados.rados.0.smithi076.stdout:2185: expect (ObjNum 729 snap 238 seq_num 729) 2021-03-15T22:29:17.411 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:17.415 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 systemd[1]: Starting Ceph osd.5 for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:29:17.429 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:17.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:17.453 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:17.453 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:17.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:29:17.483 INFO:tasks.rados.rados.0.smithi076.stdout:2184: finishing write tid 2 to smithi07694878-46 2021-03-15T22:29:17.483 INFO:tasks.rados.rados.0.smithi076.stdout:2184: finishing write tid 3 to smithi07694878-46 2021-03-15T22:29:17.484 INFO:tasks.rados.rados.0.smithi076.stdout:2184: finishing write tid 4 to smithi07694878-46 2021-03-15T22:29:17.484 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 848 (ObjNum 759 snap 250 seq_num 759) dirty exists 2021-03-15T22:29:17.484 INFO:tasks.rados.rados.0.smithi076.stdout:2184: left oid 46 (ObjNum 759 snap 250 seq_num 759) 2021-03-15T22:29:17.484 INFO:tasks.rados.rados.0.smithi076.stdout:2183: done (4 left) 2021-03-15T22:29:17.485 INFO:tasks.rados.rados.0.smithi076.stdout:2184: done (3 left) 2021-03-15T22:29:17.485 INFO:tasks.rados.rados.0.smithi076.stdout:2185: done (2 left) 2021-03-15T22:29:17.485 INFO:tasks.rados.rados.0.smithi076.stdout:2186: done (1 left) 2021-03-15T22:29:17.485 INFO:tasks.rados.rados.0.smithi076.stdout:2188: snap_remove snap 245 2021-03-15T22:29:17.555 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:17.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:17.579 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:17.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:17.676 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:17 smithi125 conmon[56739]: cluster 2021-03-15T22:29:16.238753+0000 osd.1 (osd.1) 117 : 2021-03-15T22:29:17.677 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:17 smithi125 conmon[56739]: cluster [DBG] 5.7 scrub ok 2021-03-15T22:29:17.677 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:17 smithi125 conmon[56739]: cluster 2021-03-15T22:29:17.203362+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e608: 8 total, 6 up, 7 in 2021-03-15T22:29:17.677 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 podman[135450]: 2021-03-15 22:29:17.455109615 +0000 UTC m=+0.231502717 container create d6d62e25e8fb7976302a8ef8a2a66bca9ee738be0d466c628e97dcdc2db75eb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5-activate, maintainer=Dimitri Savineau , io.buildah.version=1.18.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) 2021-03-15T22:29:17.677 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 podman[135450]: 2021-03-15 22:29:17.61640834 +0000 UTC m=+0.392801431 container init d6d62e25e8fb7976302a8ef8a2a66bca9ee738be0d466c628e97dcdc2db75eb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5-activate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, maintainer=Dimitri Savineau , io.buildah.version=1.18.0) 2021-03-15T22:29:17.678 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 podman[135450]: 2021-03-15 22:29:17.675067329 +0000 UTC m=+0.451460325 container start d6d62e25e8fb7976302a8ef8a2a66bca9ee738be0d466c628e97dcdc2db75eb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5-activate, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20201204, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.18.0) 2021-03-15T22:29:17.678 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 podman[135450]: 2021-03-15 22:29:17.675185144 +0000 UTC m=+0.451578149 container attach d6d62e25e8fb7976302a8ef8a2a66bca9ee738be0d466c628e97dcdc2db75eb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5-activate, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, io.buildah.version=1.18.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) 2021-03-15T22:29:17.681 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:17.703 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:17.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:17.761 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:17.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:17 smithi076 conmon[56514]: cluster 2021-03-15T22:29:16.238753+0000 osd.1 ( 2021-03-15T22:29:17.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:17 smithi076 conmon[56514]: osd.1) 117 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:29:17.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:17 smithi076 conmon[56514]: cluster 2021-03-15T22:29:17.203362+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e608: 8 total, 6 up, 7 in 2021-03-15T22:29:17.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:17 smithi076 conmon[62156]: cluster 2021-03-15T22:29:16.238753+0000 osd.1 (osd.1) 117 2021-03-15T22:29:17.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:17 smithi076 conmon[62156]: : cluster [DBG] 5.7 scrub ok 2021-03-15T22:29:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:17 smithi076 conmon[62156]: cluster 2021-03-15T22:29:17.203362+0000 mon.a (mon.0) 1680 : cluster [DBG] osdmap e608: 8 total, 6 up, 7 in 2021-03-15T22:29:17.806 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:17.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:17.835 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:17.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:17.936 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:17.958 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:17.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:17.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:17.963 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 bash[135373]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-03-15T22:29:17.963 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 bash[135373]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_3 --path /var/lib/ceph/osd/ceph-5 --no-mon-config 2021-03-15T22:29:17.963 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 bash[135373]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-5/block 2021-03-15T22:29:17.964 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 bash[135373]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-5/block 2021-03-15T22:29:17.964 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 bash[135373]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-2 2021-03-15T22:29:17.964 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 bash[135373]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2021-03-15T22:29:17.964 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 bash[135373]: --> ceph-volume lvm activate successful for osd ID: 5 2021-03-15T22:29:17.965 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:17 smithi125 podman[135450]: 2021-03-15 22:29:17.96123387 +0000 UTC m=+0.737626878 container died d6d62e25e8fb7976302a8ef8a2a66bca9ee738be0d466c628e97dcdc2db75eb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5-activate) 2021-03-15T22:29:18.061 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:18.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:18.085 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:18.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:18.187 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:18.209 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:18.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:18.211 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:18.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:18.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:18.338 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:18.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:18.421 INFO:tasks.rados.rados.0.smithi076.stdout:2188: done (1 left) 2021-03-15T22:29:18.421 INFO:tasks.rados.rados.0.smithi076.stdout:2189: write (excl) oid 8 current snap is 250 2021-03-15T22:29:18.421 INFO:tasks.rados.rados.0.smithi076.stdout:2187: finishing copy_from to smithi07694878-40 2021-03-15T22:29:18.422 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 506 (ObjNum 702 snap 226 seq_num 702) dirty exists 2021-03-15T22:29:18.422 INFO:tasks.rados.rados.0.smithi076.stdout:2189: seq_num 760 ranges {773320=708187,1923609=470924} 2021-03-15T22:29:18.430 INFO:tasks.rados.rados.0.smithi076.stdout:2189: writing smithi07694878-8 from 773320 to 1481507 tid 1 2021-03-15T22:29:18.435 INFO:tasks.rados.rados.0.smithi076.stdout:2189: writing smithi07694878-8 from 1923609 to 2394533 tid 2 2021-03-15T22:29:18.435 INFO:tasks.rados.rados.0.smithi076.stdout:2190: snap_create 2021-03-15T22:29:18.435 INFO:tasks.rados.rados.0.smithi076.stdout:2187: finishing copy_from racing read to smithi07694878-40 2021-03-15T22:29:18.436 INFO:tasks.rados.rados.0.smithi076.stdout:2189: finishing write tid 1 to smithi07694878-8 2021-03-15T22:29:18.438 INFO:tasks.rados.rados.0.smithi076.stdout:2189: finishing write tid 2 to smithi07694878-8 2021-03-15T22:29:18.438 INFO:tasks.rados.rados.0.smithi076.stdout:2189: finishing write tid 3 to smithi07694878-8 2021-03-15T22:29:18.439 INFO:tasks.rados.rados.0.smithi076.stdout:2189: finishing write tid 4 to smithi07694878-8 2021-03-15T22:29:18.439 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1200 (ObjNum 760 snap 250 seq_num 760) dirty exists 2021-03-15T22:29:18.439 INFO:tasks.rados.rados.0.smithi076.stdout:2189: left oid 8 (ObjNum 760 snap 250 seq_num 760) 2021-03-15T22:29:18.440 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:18.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:18.463 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:18.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:18.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[56514]: cluster 2021-03-15T22:29:17.123588+0000 mgr.y ( 2021-03-15T22:29:18.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[56514]: mgr.14144) 793 : cluster [DBG] pgmap v1055: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 560 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.2 MiB/s rd, 818 KiB/s wr, 7 op/s; 173/1743 objects degraded (9.925%); 82/1743 objects misplaced (4.705%); 406 B/s, 6 objects/s recovering 2021-03-15T22:29:18.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[56514]: cluster 2021-03-15T22:29:17.132230+0000 osd.6 (osd.6) 97 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:29:18.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[56514]: cluster 2021-03-15T22:29:17.298039+0000 osd.3 (osd.3) 108 : cluster [DBG] 6.f scrub ok 2021-03-15T22:29:18.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[56514]: cluster 2021-03-15T22:29:17.782206+0000 osd.0 (osd.0) 92 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:29:18.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[62156]: cluster 2021-03-15T22:29:17. 2021-03-15T22:29:18.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[62156]: 123588+0000 mgr.y (mgr.14144) 793 : cluster [DBG] pgmap v1055: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 560 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.2 MiB/s rd, 818 KiB/s wr, 7 op/s; 173/1743 objects degraded (9.925%); 82/1743 objects misplaced (4.705%); 406 B/s, 6 objects/s recovering 2021-03-15T22:29:18.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:29:18.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[62156]: -15T22:29:17.132230+0000 osd.6 (osd.6) 97 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:29:18.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[62156]: cluster 2021-03-15T22:29:18.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[62156]: 2021-03-15T22:29:17.298039 2021-03-15T22:29:18.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[62156]: +0000 osd.3 (osd.3) 108 : cluster [DBG] 6.f scrub ok 2021-03-15T22:29:18.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[62156]: cluster 2021-03-15T22:29:18.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[62156]: 2021-03-15T22:29: 2021-03-15T22:29:18.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[62156]: 17.782206+0000 osd.0 (osd.0) 92 : cluster 2021-03-15T22:29:18.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:18 smithi076 conmon[62156]: [DBG] 3.12 scrub ok 2021-03-15T22:29:18.564 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:18.587 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:18.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:18.689 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:18.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:18.712 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:18.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:18.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:18 smithi125 conmon[56739]: cluster 2021-03-15T22:29:17.123588+0000 mgr.y (mgr.14144) 793 : cluster [DBG] pgmap v1055: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 560 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.2 MiB/s rd, 818 KiB/s wr, 7 op/s; 173/1743 objects degraded (9.925%); 82/1743 objects misplaced (4.705%); 406 B/s, 6 objects/s recovering 2021-03-15T22:29:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:18 smithi125 conmon[56739]: cluster 2021-03-15T22:29:17.132230+0000 osd.6 (osd.6) 97 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:29:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:18 smithi125 conmon[56739]: cluster 2021-03-15T22:29:17.298039+0000 osd.3 (osd.3) 108 : cluster [DBG] 6.f scrub ok 2021-03-15T22:29:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:18 smithi125 conmon[56739]: cluster 2021-03-15T22:29:17.782206+0000 osd.0 (osd.0) 92 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:29:18.813 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:18.835 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:18.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:18.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:18.938 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:18.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:18.964 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:18.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:19.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:19.091 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:19.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:19.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:19.184 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:29:19.194 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:19.217 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:19.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:19.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:19.287 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:19 smithi125 podman[135450]: 2021-03-15 22:29:19.054866306 +0000 UTC m=+1.831259397 container remove d6d62e25e8fb7976302a8ef8a2a66bca9ee738be0d466c628e97dcdc2db75eb3 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5-activate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, io.buildah.version=1.18.0, org.label-schema.name=CentOS Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2021-03-15T22:29:19.320 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:19.345 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:19.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:19.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:19.434 INFO:tasks.rados.rados.0.smithi076.stdout:2187: done (2 left) 2021-03-15T22:29:19.434 INFO:tasks.rados.rados.0.smithi076.stdout:2189: done (1 left) 2021-03-15T22:29:19.435 INFO:tasks.rados.rados.0.smithi076.stdout:2190: done (0 left) 2021-03-15T22:29:19.435 INFO:tasks.rados.rados.0.smithi076.stdout:2191: delete oid 20 current snap is 251 2021-03-15T22:29:19.441 INFO:tasks.rados.rados.0.smithi076.stdout:2191: done (0 left) 2021-03-15T22:29:19.442 INFO:tasks.rados.rados.0.smithi076.stdout:2192: snap_create 2021-03-15T22:29:19.447 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:19.473 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:19.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:19.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:19.496 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:19.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[56514]: cluster 2021-03-15T22:29:17.891847+0000 osd.4 (osd.4) 130 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:29:19.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[56514]: cluster 2021-03-15T22:29:18.056301+0000 osd.7 (osd.7) 122 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:29:19.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[56514]: cluster 2021-03-15T22:29:18.149199+0000 osd.1 (osd.1) 118 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:29:19.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[56514]: cluster 2021-03-15T22:29:18.266800+0000 osd.3 (osd.3) 109 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:29:19.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[56514]: cluster 2021-03-15T22:29:18.416786+0000 mon.a (mon.0) 1681 : cluster [DBG] osdmap e609: 8 total, 6 up, 7 in 2021-03-15T22:29:19.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[56514]: cluster 2021-03-15T22:29:18.753147+0000 osd.0 (osd.0) 93 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:29:19.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: cluster 2021-03-15T22:29:17.891847+0000 osd.4 2021-03-15T22:29:19.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: (osd.4) 130 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:29:19.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: cluster 2021-03-15T22:29:18.056301+0000 osd.7 (osd.7) 122 : cluster [DBG] 2021-03-15T22:29:19.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: 3.16 scrub ok 2021-03-15T22:29:19.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: cluster 2021-03-15T22:29: 2021-03-15T22:29:19.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: 18.149199+0000 osd.1 (osd.1) 118 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:29:19.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:19.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: 03-15T22:29:18.266800+0000 osd.3 2021-03-15T22:29:19.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: (osd.3) 109 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:29:19.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:19.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: 03-15T22:29:18.416786+0000 mon.a 2021-03-15T22:29:19.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: (mon.0) 1681 : cluster [DBG] osdmap e609: 8 total, 6 up, 7 in 2021-03-15T22:29:19.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:29:19.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: -15T22:29:18.753147+0000 osd.0 (osd 2021-03-15T22:29:19.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:19 smithi076 conmon[62156]: .0) 93 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:29:19.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:19.597 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:19.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:19.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:19.700 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:19.725 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:19.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:19.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: cluster 2021-03-15T22:29:17.891847+0000 2021-03-15T22:29:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: osd.4 (osd.4) 130 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:29:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: cluster 2021-03-15T22:29:18.056301+0000 osd.7 (osd.7) 122 : cluster 2021-03-15T22:29:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: [DBG] 3.16 scrub ok 2021-03-15T22:29:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:29:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: 03-15T22:29:18.149199+0000 osd.1 (osd.1) 2021-03-15T22:29:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: 118 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:29:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:29:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: :29:18.266800+0000 osd.3 (osd.3) 109 2021-03-15T22:29:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: : cluster [DBG] 6.7 scrub ok 2021-03-15T22:29:19.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:29:19.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: :29:18.416786+0000 mon.a (mon.0) 1681 : 2021-03-15T22:29:19.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: cluster [DBG] osdmap e609: 8 total, 6 up, 7 in 2021-03-15T22:29:19.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:29:19.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: :29:18.753147+0000 osd.0 (osd. 2021-03-15T22:29:19.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[56739]: 0) 93 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:29:19.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:19.850 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:19.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:19.851 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:19.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:19.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:19.976 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:19.976 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:20.073 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:19 smithi125 podman[135908]: 2021-03-15 22:29:19.938057773 +0000 UTC m=+0.312008871 container create c8ec55adedf486d0e00300e07623c1889995459763d0daf8a206b21292fddc2b (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Base Image, maintainer=Dimitri Savineau , ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.build-date=20201204, io.buildah.version=1.18.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4) 2021-03-15T22:29:20.073 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:29:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:29:20.077 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:20.100 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:20.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:20.102 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:20.152 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:29:20.153 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:29:20.153 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:29:20.153 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:29:20.191 INFO:tasks.rados.rados.0.smithi076.stdout:2192: done (0 left) 2021-03-15T22:29:20.192 INFO:tasks.rados.rados.0.smithi076.stdout:2193: snap_remove snap 212 2021-03-15T22:29:20.195 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:29:20.203 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:20.221 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:29:20.228 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:20.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:20.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:20.331 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:20.355 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:20.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:20.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:20.432 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:20 smithi125 podman[135908]: 2021-03-15 22:29:20.154610096 +0000 UTC m=+0.528561194 container init c8ec55adedf486d0e00300e07623c1889995459763d0daf8a206b21292fddc2b (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Base Image, GIT_CLEAN=True, org.label-schema.build-date=20201204, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2021-03-15T22:29:20.432 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:20 smithi125 podman[135908]: 2021-03-15 22:29:20.212961447 +0000 UTC m=+0.586912573 container start c8ec55adedf486d0e00300e07623c1889995459763d0daf8a206b21292fddc2b (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5, GIT_CLEAN=True, org.label-schema.build-date=20201204, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.name=CentOS Base Image, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2021-03-15T22:29:20.432 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:20 smithi125 bash[135373]: c8ec55adedf486d0e00300e07623c1889995459763d0daf8a206b21292fddc2b 2021-03-15T22:29:20.433 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:20 smithi125 systemd[1]: Started Ceph osd.5 for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:29:20.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:20.485 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:20.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:20.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:20.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:20.611 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:20.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:20.707 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: cluster 2021-03-15T22:29:19.039995 2021-03-15T22:29:20.707 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: +0000 osd.7 (osd.7) 123 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:29:20.707 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: cluster 2021-03-15T22:29:19.124092+0000 mgr.y (mgr.14144) 794 : cluster [DBG] pgmap v1058: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 565 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.4 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 176/1752 objects degraded (10.046%); 82/1752 objects misplaced (4.680%); 262 B/s, 3 objects/s recovering 2021-03-15T22:29:20.708 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: cluster 2021-03-15T22:29:19.420617+0000 mon.a (mon.0) 1682 : cluster [WRN] Health check update: Degraded data redundancy: 176/1752 objects degraded (10.046%), 13 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:20.708 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: cluster 2021-03-15T22:29:19.430394+0000 mon.a (mon.0) 1683 2021-03-15T22:29:20.708 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: : cluster [DBG] osdmap e610: 8 total, 6 up, 7 in 2021-03-15T22:29:20.708 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: audit 2021-03-15T22:29:19.604866 2021-03-15T22:29:20.709 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1684 : audit [INF] from='client.? 172.21.15.76:0/4292203143' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:29:20.709 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:29:20.709 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: -15T22:29:19.721283+0000 2021-03-15T22:29:20.709 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: osd.0 (osd.0) 94 : cluster [DBG] 2021-03-15T22:29:20.709 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: 6.10 scrub ok 2021-03-15T22:29:20.710 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: audit 2021-03-15T22:29:20.710 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: 2021-03-15T22:29:19.925847 2021-03-15T22:29:20.710 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 795 : audit 2021-03-15T22:29:20.710 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:20.711 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:29:20.711 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: 29:20.181269+0000 mon.a (mon.0 2021-03-15T22:29:20.711 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: ) 2021-03-15T22:29:20.711 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: 1685 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:20.711 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: 2021-03-15T22:29:20.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:29:20.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: 03-15T22:29:20.188520+0000 mon.a (mon 2021-03-15T22:29:20.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: .0) 1686 : audit [INF] from='client.? 172.21.15.76:0/4292203143' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:29:20.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:29:20.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: -15T22:29:20.188575+0000 mon.a (mon. 2021-03-15T22:29:20.713 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[56739]: 0) 1687 : cluster [DBG] osdmap e611: 8 total, 6 up, 7 in 2021-03-15T22:29:20.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:20.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:20.737 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:20.737 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:20.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[56514]: cluster 2021-03-15T22:29:19.039995+0000 osd.7 2021-03-15T22:29:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[56514]: (osd.7) 123 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:29:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[56514]: cluster 2021-03-15T22:29:19.124092+0000 mgr.y (mgr.14144) 794 : cluster [DBG] pgmap v1058: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 565 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.4 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 176/1752 objects degraded (10.046%); 82/1752 objects misplaced (4.680%); 262 B/s, 3 objects/s recovering 2021-03-15T22:29:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[56514]: cluster 2021-03-15T22:29:19.420617+0000 mon.a (mon.0) 1682 : cluster [WRN] Health check update: Degraded data redundancy: 176/1752 objects degraded (10.046%), 13 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[56514]: cluster 2021-03-15T22:29:19.430394+0000 mon.a (mon.0) 1683 : cluster [DBG] osdmap e610: 8 total, 6 up, 7 in 2021-03-15T22:29:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[56514]: audit 2021-03-15T22:29:19.604866+0000 mon.a (mon.0) 1684 : audit [INF] from='client.? 172.21.15.76:0/4292203143' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:29:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[56514]: cluster 2021-03-15T22:29:19.721283+0000 osd.0 (osd.0) 94 : cluster [DBG] 6.10 scrub ok 2021-03-15T22:29:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[56514]: audit 2021-03-15T22:29:19.925847+0000 mgr.y (mgr.14144) 795 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[56514]: cluster 2021-03-15T22:29:20.181269+0000 mon.a (mon.0) 1685 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:20.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[56514]: audit 2021-03-15T22:29:20.188520+0000 mon.a (mon.0) 1686 : audit [INF] from='client.? 172.21.15.76:0/4292203143' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:29:20.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[56514]: cluster 2021-03-15T22:29:20.188575+0000 mon.a (mon.0) 1687 : cluster [DBG] osdmap e611: 8 total, 6 up, 7 in 2021-03-15T22:29:20.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: cluster 2021-03-15T22:29:19. 2021-03-15T22:29:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: 039995+0000 osd.7 (osd.7) 123 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:29:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: 03-15T22:29:19.124092+0000 mgr.y (mgr.14144) 794 : cluster 2021-03-15T22:29:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: [DBG] pgmap v1058: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 565 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.4 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 176/1752 objects degraded (10.046%); 82/1752 objects misplaced (4.680%); 262 B/s, 3 objects/s recovering 2021-03-15T22:29:20.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:29:20.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: 29:19.420617+0000 mon.a (mon.0 2021-03-15T22:29:20.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: ) 1682 : cluster [WRN] Health check update: Degraded data redundancy: 176/1752 objects degraded (10.046%), 13 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:20.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:29:20.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: -03-15T22:29:19.430394 2021-03-15T22:29:20.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1683 : cluster 2021-03-15T22:29:20.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: [DBG] osdmap e610: 8 total, 6 up, 7 in 2021-03-15T22:29:20.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:29:20.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: 22:29:19.604866+0000 mon.a (mon 2021-03-15T22:29:20.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: .0) 1684 : audit [INF] from='client.? 172.21.15.76:0/4292203143' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:29:20.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:20.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: 03-15T22:29:19.721283 2021-03-15T22:29:20.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: +0000 osd.0 (osd.0) 94 : cluster 2021-03-15T22:29:20.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: [DBG] 6.10 scrub ok 2021-03-15T22:29:20.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:29:20.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: :29:19.925847+0000 mgr.y (mgr. 2021-03-15T22:29:20.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: 14144) 795 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:20.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:20.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: 03-15T22:29:20.181269+0000 2021-03-15T22:29:20.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: mon.a (mon.0) 1685 : cluster [WRN] 2021-03-15T22:29:20.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:20.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: audit 2021 2021-03-15T22:29:20.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: -03-15T22:29:20.188520 2021-03-15T22:29:20.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1686 : audit 2021-03-15T22:29:20.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/4292203143' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:29:20.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:29:20.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: 15T22:29:20.188575+0000 mon.a (mon 2021-03-15T22:29:20.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:20 smithi076 conmon[62156]: .0) 1687 : cluster [DBG] osdmap e611: 8 total, 6 up, 7 in 2021-03-15T22:29:20.838 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:20.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:20.862 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:20.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:20.928 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-15T22:29:20.962 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:20.964 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:20.987 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:20.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:20.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:21.036 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:20 smithi125 conmon[135985]: debug 2021-03-15T22:29:20.704+0000 7f4284983080 -1 Falling back to public interface 2021-03-15T22:29:21.089 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:21.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:21.113 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:21.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:21.201 INFO:tasks.rados.rados.0.smithi076.stdout:2193: done (0 left) 2021-03-15T22:29:21.201 INFO:tasks.rados.rados.0.smithi076.stdout:2194: rollback oid 11 current snap is 252 2021-03-15T22:29:21.201 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 11 to 183 2021-03-15T22:29:21.202 INFO:tasks.rados.rados.0.smithi076.stdout:2195: read oid 8 snap -1 2021-03-15T22:29:21.202 INFO:tasks.rados.rados.0.smithi076.stdout:2195: expect (ObjNum 760 snap 250 seq_num 760) 2021-03-15T22:29:21.202 INFO:tasks.rados.rados.0.smithi076.stdout:2196: snap_create 2021-03-15T22:29:21.214 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:21.216 INFO:tasks.rados.rados.0.smithi076.stdout:2194: finishing rollback tid 0 to smithi07694878-11 2021-03-15T22:29:21.216 INFO:tasks.rados.rados.0.smithi076.stdout:2194: finishing rollback tid 1 to smithi07694878-11 2021-03-15T22:29:21.216 INFO:tasks.rados.rados.0.smithi076.stdout:2194: finishing rollback tid 2 to smithi07694878-11 2021-03-15T22:29:21.216 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1506 (ObjNum 494 snap 146 seq_num 494) dirty exists 2021-03-15T22:29:21.217 INFO:tasks.rados.rados.0.smithi076.stdout:2195: expect (ObjNum 760 snap 250 seq_num 760) 2021-03-15T22:29:21.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:21.245 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:21.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:21.347 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:21.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:21.371 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:21.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:21.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:21 smithi125 conmon[56739]: cluster 2021-03-15T22:29:19.876001+0000 osd.4 (osd.4) 131 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:29:21.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:21 smithi125 conmon[56739]: cluster 2021-03-15T22:29:20.139310+0000 osd.1 (osd.1) 119 : cluster [DBG] 4.2 deep-scrub ok 2021-03-15T22:29:21.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:21 smithi125 conmon[56739]: cluster 2021-03-15T22:29:21.195328+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e612: 8 total, 6 up, 7 in 2021-03-15T22:29:21.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:21.495 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:21.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:21.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:21.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:21 smithi076 conmon[62156]: cluster 2021-03-15T22:29:19.876001+0000 osd.4 (osd.4) 131 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:29:21.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:21 smithi076 conmon[62156]: cluster 2021-03-15T22:29:20.139310+0000 osd.1 (osd.1) 119 : cluster [DBG] 4.2 deep-scrub ok 2021-03-15T22:29:21.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:21 smithi076 conmon[62156]: cluster 2021-03-15T22:29:21.195328+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e612: 8 total, 6 up, 7 in 2021-03-15T22:29:21.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:21 smithi076 conmon[56514]: cluster 2021-03-15T22:29:19.876001+0000 osd.4 (osd.4) 131 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:29:21.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:21 smithi076 conmon[56514]: cluster 2021-03-15T22:29:20.139310+0000 osd.1 (osd.1) 119 : cluster [DBG] 4.2 deep-scrub ok 2021-03-15T22:29:21.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:21 smithi076 conmon[56514]: cluster 2021-03-15T22:29:21.195328+0000 mon.a (mon.0) 1688 : cluster [DBG] osdmap e612: 8 total, 6 up, 7 in 2021-03-15T22:29:21.577 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:21.598 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:21.622 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:21.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:21.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:21.634 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:29:21.651 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:29:21.668 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:29:21.685 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:29:21.703 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:29:21.720 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:29:21.720 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:29:21.720 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:29:21.721 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:29:21.721 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:29:21.721 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:29:21.721 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.5 2021-03-15T22:29:21.722 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:29:21.722 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:29:21.724 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:21.746 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:21.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:21.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:21.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:21.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:21.873 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:21.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:21.975 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:21.998 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:21.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:21.999 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:22.100 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:22.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:22.126 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:22.126 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:22.203 INFO:tasks.rados.rados.0.smithi076.stdout:2194: done (2 left) 2021-03-15T22:29:22.203 INFO:tasks.rados.rados.0.smithi076.stdout:2195: done (1 left) 2021-03-15T22:29:22.203 INFO:tasks.rados.rados.0.smithi076.stdout:2196: done (0 left) 2021-03-15T22:29:22.203 INFO:tasks.rados.rados.0.smithi076.stdout:2197: read oid 39 snap -1 2021-03-15T22:29:22.204 INFO:tasks.rados.rados.0.smithi076.stdout:2197: expect deleted 2021-03-15T22:29:22.204 INFO:tasks.rados.rados.0.smithi076.stdout:2198: write (excl) oid 13 current snap is 253 2021-03-15T22:29:22.204 INFO:tasks.rados.rados.0.smithi076.stdout:2198: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2021-03-15T22:29:22.208 INFO:tasks.rados.rados.0.smithi076.stdout:2198: writing smithi07694878-13 from 406934 to 981972 tid 1 2021-03-15T22:29:22.214 INFO:tasks.rados.rados.0.smithi076.stdout:2198: writing smithi07694878-13 from 1640573 to 2115166 tid 2 2021-03-15T22:29:22.218 INFO:tasks.rados.rados.0.smithi076.stdout:2198: writing smithi07694878-13 from 2642791 to 2806934 tid 3 2021-03-15T22:29:22.219 INFO:tasks.rados.rados.0.smithi076.stdout:2199: rollback oid 28 current snap is 253 2021-03-15T22:29:22.219 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 250 2021-03-15T22:29:22.219 INFO:tasks.rados.rados.0.smithi076.stdout:2198: finishing write tid 1 to smithi07694878-13 2021-03-15T22:29:22.219 INFO:tasks.rados.rados.0.smithi076.stdout:2198: finishing write tid 2 to smithi07694878-13 2021-03-15T22:29:22.220 INFO:tasks.rados.rados.0.smithi076.stdout:2197: done (2 left) 2021-03-15T22:29:22.220 INFO:tasks.rados.rados.0.smithi076.stdout:2200: write oid 21 current snap is 253 2021-03-15T22:29:22.220 INFO:tasks.rados.rados.0.smithi076.stdout:2200: seq_num 763 ranges {423741=650287,1699055=618251} 2021-03-15T22:29:22.224 INFO:tasks.rados.rados.0.smithi076.stdout:2200: writing smithi07694878-21 from 423741 to 1074028 tid 1 2021-03-15T22:29:22.227 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:22.233 INFO:tasks.rados.rados.0.smithi076.stdout:2200: writing smithi07694878-21 from 1699055 to 2317306 tid 2 2021-03-15T22:29:22.233 INFO:tasks.rados.rados.0.smithi076.stdout:2201: read oid 3 snap -1 2021-03-15T22:29:22.234 INFO:tasks.rados.rados.0.smithi076.stdout:2201: expect deleted 2021-03-15T22:29:22.234 INFO:tasks.rados.rados.0.smithi076.stdout:2202: copy_from oid 1 from oid 38 current snap is 253 2021-03-15T22:29:22.234 INFO:tasks.rados.rados.0.smithi076.stdout:2198: finishing write tid 3 to smithi07694878-13 2021-03-15T22:29:22.234 INFO:tasks.rados.rados.0.smithi076.stdout:2199: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:29:22.235 INFO:tasks.rados.rados.0.smithi076.stdout:2199: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:29:22.235 INFO:tasks.rados.rados.0.smithi076.stdout:2199: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:29:22.236 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 1527 (ObjNum 609 snap 189 seq_num 609) dirty exists 2021-03-15T22:29:22.237 INFO:tasks.rados.rados.0.smithi076.stdout:2198: finishing write tid 4 to smithi07694878-13 2021-03-15T22:29:22.237 INFO:tasks.rados.rados.0.smithi076.stdout:2198: finishing write tid 5 to smithi07694878-13 2021-03-15T22:29:22.237 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1510 (ObjNum 762 snap 253 seq_num 762) dirty exists 2021-03-15T22:29:22.238 INFO:tasks.rados.rados.0.smithi076.stdout:2198: left oid 13 (ObjNum 762 snap 253 seq_num 762) 2021-03-15T22:29:22.238 INFO:tasks.rados.rados.0.smithi076.stdout:2200: finishing write tid 1 to smithi07694878-21 2021-03-15T22:29:22.238 INFO:tasks.rados.rados.0.smithi076.stdout:2198: done (4 left) 2021-03-15T22:29:22.238 INFO:tasks.rados.rados.0.smithi076.stdout:2199: done (3 left) 2021-03-15T22:29:22.239 INFO:tasks.rados.rados.0.smithi076.stdout:2203: snap_create 2021-03-15T22:29:22.239 INFO:tasks.rados.rados.0.smithi076.stdout:2200: finishing write tid 2 to smithi07694878-21 2021-03-15T22:29:22.240 INFO:tasks.rados.rados.0.smithi076.stdout:2200: finishing write tid 3 to smithi07694878-21 2021-03-15T22:29:22.240 INFO:tasks.rados.rados.0.smithi076.stdout:2200: finishing write tid 4 to smithi07694878-21 2021-03-15T22:29:22.240 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 1531 (ObjNum 763 snap 253 seq_num 763) dirty exists 2021-03-15T22:29:22.240 INFO:tasks.rados.rados.0.smithi076.stdout:2200: left oid 21 (ObjNum 763 snap 253 seq_num 763) 2021-03-15T22:29:22.244 INFO:tasks.rados.rados.0.smithi076.stdout:2202: finishing copy_from to smithi07694878-1 2021-03-15T22:29:22.244 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 1036 (ObjNum 732 snap 239 seq_num 732) dirty exists 2021-03-15T22:29:22.244 INFO:tasks.rados.rados.0.smithi076.stdout:2202: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:29:22.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:22.253 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:22.254 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:22.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:22.382 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:22.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:22.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:22.466 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:29:22.467 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:29:22.484 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:22.507 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:22.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:22.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:22 smithi125 conmon[56739]: cluster 2021-03-15T22:29:21.124670+0000 mgr.y (mgr.14144) 796 : cluster [DBG] pgmap v1061: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 569 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.3 MiB/s rd, 2.0 MiB/s wr, 14 op/s; 176/1758 objects degraded (10.011%); 80/1758 objects misplaced (4.551%) 2021-03-15T22:29:22.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:22 smithi125 conmon[56739]: audit 2021-03-15T22:29:21.615265+0000 mon.a (mon.0) 1689 : audit [DBG] from='client.? 172.21.15.76:0/2614969044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:22.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:22 smithi125 conmon[56739]: cluster 2021-03-15T22:29:22.199136+0000 mon.a (mon.0) 1690 : cluster [DBG] osdmap e613: 8 total, 6 up, 7 in 2021-03-15T22:29:22.609 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:22.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:22.634 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:22.635 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:22.736 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:22.761 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:22.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:22.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:22.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:22 smithi076 conmon[56514]: cluster 2021-03-15T22:29:21.124670+0000 2021-03-15T22:29:22.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:22 smithi076 conmon[56514]: mgr.y (mgr.14144) 796 : cluster [DBG] pgmap v1061: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 569 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.3 MiB/s rd, 2.0 MiB/s wr, 14 op/s; 176/1758 objects degraded (10.011%); 80/1758 objects misplaced (4.551%) 2021-03-15T22:29:22.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:22 smithi076 conmon[56514]: audit 2021-03-15T22:29:21.615265+0000 mon.a (mon.0) 1689 : audit [DBG] from='client.? 172.21.15.76:0/2614969044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:22.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:22 smithi076 conmon[56514]: cluster 2021-03-15T22:29:22.199136+0000 mon.a (mon.0) 1690 : cluster [DBG] osdmap e613: 8 total, 6 up, 7 in 2021-03-15T22:29:22.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:22 smithi076 conmon[62156]: cluster 2021-03-15T22:29:21.124670+0000 mgr.y ( 2021-03-15T22:29:22.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:22 smithi076 conmon[62156]: mgr.14144) 796 : cluster [DBG] pgmap v1061: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 569 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.3 MiB/s rd, 2.0 MiB/s wr, 14 op/s; 176/1758 objects degraded (10.011%); 80/1758 objects misplaced (4.551%) 2021-03-15T22:29:22.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:22 smithi076 conmon[62156]: audit 2021-03-15T22:29:21.615265+0000 mon.a (mon 2021-03-15T22:29:22.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:22 smithi076 conmon[62156]: .0) 1689 : audit [DBG] from='client.? 172.21.15.76:0/2614969044' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:22.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:22 smithi076 conmon[62156]: cluster 2021-03-15T22:29:22.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:22 smithi076 conmon[62156]: 2021-03-15T22:29:22.199136+0000 mon.a (mon. 2021-03-15T22:29:22.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:22 smithi076 conmon[62156]: 0) 1690 : cluster [DBG] osdmap e613: 8 total, 6 up, 7 in 2021-03-15T22:29:22.864 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:22.888 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:22.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:22.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:22.990 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:23.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:23.016 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:23.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:23.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:23.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:23.142 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:23.143 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:23.203 INFO:tasks.rados.rados.0.smithi076.stdout:2200: done (3 left) 2021-03-15T22:29:23.203 INFO:tasks.rados.rados.0.smithi076.stdout:2201: done (2 left) 2021-03-15T22:29:23.203 INFO:tasks.rados.rados.0.smithi076.stdout:2202: done (1 left) 2021-03-15T22:29:23.204 INFO:tasks.rados.rados.0.smithi076.stdout:2203: done (0 left) 2021-03-15T22:29:23.204 INFO:tasks.rados.rados.0.smithi076.stdout:2204: copy_from oid 5 from oid 11 current snap is 254 2021-03-15T22:29:23.204 INFO:tasks.rados.rados.0.smithi076.stdout:2205: write (excl) oid 34 current snap is 254 2021-03-15T22:29:23.204 INFO:tasks.rados.rados.0.smithi076.stdout:2205: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2021-03-15T22:29:23.209 INFO:tasks.rados.rados.0.smithi076.stdout:2205: writing smithi07694878-34 from 440548 to 1166084 tid 1 2021-03-15T22:29:23.215 INFO:tasks.rados.rados.0.smithi076.stdout:2205: writing smithi07694878-34 from 1757537 to 2235799 tid 2 2021-03-15T22:29:23.219 INFO:tasks.rados.rados.0.smithi076.stdout:2205: writing smithi07694878-34 from 2813990 to 2840548 tid 3 2021-03-15T22:29:23.219 INFO:tasks.rados.rados.0.smithi076.stdout:2206: read oid 10 snap -1 2021-03-15T22:29:23.219 INFO:tasks.rados.rados.0.smithi076.stdout:2206: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:29:23.219 INFO:tasks.rados.rados.0.smithi076.stdout:2207: snap_create 2021-03-15T22:29:23.220 INFO:tasks.rados.rados.0.smithi076.stdout:2204: finishing copy_from to smithi07694878-5 2021-03-15T22:29:23.220 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1506 (ObjNum 494 snap 146 seq_num 494) dirty exists 2021-03-15T22:29:23.220 INFO:tasks.rados.rados.0.smithi076.stdout:2204: finishing copy_from racing read to smithi07694878-5 2021-03-15T22:29:23.220 INFO:tasks.rados.rados.0.smithi076.stdout:2205: finishing write tid 1 to smithi07694878-34 2021-03-15T22:29:23.220 INFO:tasks.rados.rados.0.smithi076.stdout:2205: finishing write tid 2 to smithi07694878-34 2021-03-15T22:29:23.221 INFO:tasks.rados.rados.0.smithi076.stdout:2205: finishing write tid 3 to smithi07694878-34 2021-03-15T22:29:23.222 INFO:tasks.rados.rados.0.smithi076.stdout:2205: finishing write tid 4 to smithi07694878-34 2021-03-15T22:29:23.222 INFO:tasks.rados.rados.0.smithi076.stdout:2205: finishing write tid 5 to smithi07694878-34 2021-03-15T22:29:23.222 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 1356 (ObjNum 764 snap 254 seq_num 764) dirty exists 2021-03-15T22:29:23.222 INFO:tasks.rados.rados.0.smithi076.stdout:2205: left oid 34 (ObjNum 764 snap 254 seq_num 764) 2021-03-15T22:29:23.235 INFO:tasks.rados.rados.0.smithi076.stdout:2206: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:29:23.245 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:23.267 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:23.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:23.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:23.299 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:29:23.371 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:23.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:23.394 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:23.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:23.407 INFO:teuthology.orchestra.run.smithi125.stderr:no valid command found; 10 closest matches: 2021-03-15T22:29:23.408 INFO:teuthology.orchestra.run.smithi125.stderr:0 2021-03-15T22:29:23.408 INFO:teuthology.orchestra.run.smithi125.stderr:1 2021-03-15T22:29:23.408 INFO:teuthology.orchestra.run.smithi125.stderr:2 2021-03-15T22:29:23.408 INFO:teuthology.orchestra.run.smithi125.stderr:abort 2021-03-15T22:29:23.409 INFO:teuthology.orchestra.run.smithi125.stderr:assert 2021-03-15T22:29:23.409 INFO:teuthology.orchestra.run.smithi125.stderr:bluestore allocator dump block 2021-03-15T22:29:23.409 INFO:teuthology.orchestra.run.smithi125.stderr:bluestore allocator fragmentation block 2021-03-15T22:29:23.409 INFO:teuthology.orchestra.run.smithi125.stderr:bluestore allocator score block 2021-03-15T22:29:23.409 INFO:teuthology.orchestra.run.smithi125.stderr:config diff 2021-03-15T22:29:23.410 INFO:teuthology.orchestra.run.smithi125.stderr:config diff get 2021-03-15T22:29:23.410 INFO:teuthology.orchestra.run.smithi125.stderr:admin_socket: invalid command 2021-03-15T22:29:23.495 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:23.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:23.523 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:23.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:23.594 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:23.624 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:23.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:23.648 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:23.649 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:23.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:23.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:23.776 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:23.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:23.786 INFO:journalctl@ceph.osd.5.smithi125.stdout:Mar 15 22:29:23 smithi125 conmon[135985]: debug 2021-03-15T22:29:23.557+0000 7f4284983080 -1 osd.5 539 log_to_monitors {default=true} 2021-03-15T22:29:23.879 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:23.901 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:23.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:23.903 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:23.986 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-03-15T22:29:23.987 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-5, ['dump_ops_in_flight'] 2021-03-15T22:29:24.005 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:24.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:24.028 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:24.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:24.129 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:24.153 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:24.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:24.154 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:24.211 INFO:tasks.rados.rados.0.smithi076.stdout:2204: done (3 left) 2021-03-15T22:29:24.211 INFO:tasks.rados.rados.0.smithi076.stdout:2205: done (2 left) 2021-03-15T22:29:24.212 INFO:tasks.rados.rados.0.smithi076.stdout:2206: done (1 left) 2021-03-15T22:29:24.212 INFO:tasks.rados.rados.0.smithi076.stdout:2207: done (0 left) 2021-03-15T22:29:24.212 INFO:tasks.rados.rados.0.smithi076.stdout:2208: snap_remove snap 237 2021-03-15T22:29:24.256 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:24.280 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:24.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:24.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:24.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:24.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:24.406 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:24.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:24.407 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:29:24.407 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:29:24.463 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:24 smithi125 conmon[56739]: cluster 2021-03-15T22:29:23.125413+0000 mgr.y (mgr.14144) 797 : cluster [DBG] pgmap v1064: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 570 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 178/1764 objects degraded (10.091%); 80/1764 objects misplaced (4.535%) 2021-03-15T22:29:24.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:24 smithi125 conmon[56739]: cluster 2021-03-15T22:29:23.197519+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e614: 8 total, 6 up, 7 in 2021-03-15T22:29:24.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:24 smithi125 conmon[56739]: audit 2021-03-15T22:29:23.561284+0000 mon.b (mon.2) 26 : audit [INF] from='osd.5 [v2:172.21.15.125:6808/978017133,v1:172.21.15.125:6809/978017133]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:29:24.464 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:24 smithi125 conmon[56739]: audit 2021-03-15T22:29:23.561634+0000 mon.a (mon.0) 1692 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:29:24.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:29:24.489 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[56514]: cluster 2021-03-15T22:29:23.125413+0000 mgr.y (mgr.14144 2021-03-15T22:29:24.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[56514]: ) 797 : cluster [DBG] pgmap v1064: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 570 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 178/1764 objects degraded (10.091%); 80/1764 objects misplaced (4.535%) 2021-03-15T22:29:24.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[56514]: cluster 2021-03-15T22:29:23.197519+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e614: 8 total, 6 up, 7 in 2021-03-15T22:29:24.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[56514]: audit 2021-03-15T22:29:23.561284+0000 mon.b (mon.2) 26 : audit [INF] from='osd.5 [v2:172.21.15.125:6808/978017133,v1:172.21.15.125:6809/978017133]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:29:24.490 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[56514]: audit 2021-03-15T22:29:23.561634+0000 mon.a (mon.0) 1692 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:29:24.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: cluster 2021-03-15T22:29:23.125413+0000 2021-03-15T22:29:24.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: mgr.y (mgr.14144) 797 : cluster [DBG] pgmap v1064: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 570 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.2 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 178/1764 objects degraded (10.091%); 80/1764 objects misplaced (4.535%) 2021-03-15T22:29:24.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: cluster 2021-03-15T22:29 2021-03-15T22:29:24.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: :23.197519+0000 mon.a (mon.0) 1691 : cluster [DBG] osdmap e614: 8 total, 6 up, 7 in 2021-03-15T22:29:24.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: audit 2021 2021-03-15T22:29:24.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: -03-15T22:29:23.561284 2021-03-15T22:29:24.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: +0000 mon.b (mon.2) 26 : audit 2021-03-15T22:29:24.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: [INF] from='osd.5 [v2:172.21.15.125:6808/978017133,v1:172.21.15.125:6809/978017133]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:29:24.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:29:24.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: -15T22:29:23.561634+0000 2021-03-15T22:29:24.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: mon.a (mon.0) 1692 : audit [INF] 2021-03-15T22:29:24.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:24 smithi076 conmon[62156]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2021-03-15T22:29:24.508 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:24.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:24.534 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:24.534 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:24.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:24.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:24.659 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:24.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:24.761 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:24.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:24.784 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:24.784 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:24.868 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:24.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:24.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:24.911 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:24.911 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:25.012 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:25.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:25.036 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:25.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:25.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:25.162 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:25.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:25.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:25.165 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-15T22:29:25.220 INFO:tasks.rados.rados.0.smithi076.stdout:2208: done (0 left) 2021-03-15T22:29:25.221 INFO:tasks.rados.rados.0.smithi076.stdout:2209: read oid 10 snap 244 2021-03-15T22:29:25.221 INFO:tasks.rados.rados.0.smithi076.stdout:2209: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:29:25.221 INFO:tasks.rados.rados.0.smithi076.stdout:2210: copy_from oid 23 from oid 38 current snap is 255 2021-03-15T22:29:25.221 INFO:tasks.rados.rados.0.smithi076.stdout:2211: read oid 50 snap -1 2021-03-15T22:29:25.222 INFO:tasks.rados.rados.0.smithi076.stdout:2211: expect (ObjNum 658 snap 206 seq_num 658) 2021-03-15T22:29:25.222 INFO:tasks.rados.rados.0.smithi076.stdout:2212: read oid 28 snap -1 2021-03-15T22:29:25.222 INFO:tasks.rados.rados.0.smithi076.stdout:2212: expect (ObjNum 609 snap 189 seq_num 609) 2021-03-15T22:29:25.222 INFO:tasks.rados.rados.0.smithi076.stdout:2213: copy_from oid 13 from oid 36 current snap is 255 2021-03-15T22:29:25.223 INFO:tasks.rados.rados.0.smithi076.stdout:2214: write (excl) oid 29 current snap is 255 2021-03-15T22:29:25.223 INFO:tasks.rados.rados.0.smithi076.stdout:2214: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2021-03-15T22:29:25.225 INFO:tasks.rados.rados.0.smithi076.stdout:2214: writing smithi07694878-29 from 457355 to 858140 tid 1 2021-03-15T22:29:25.233 INFO:tasks.rados.rados.0.smithi076.stdout:2214: writing smithi07694878-29 from 1416019 to 2037939 tid 2 2021-03-15T22:29:25.238 INFO:tasks.rados.rados.0.smithi076.stdout:2214: writing smithi07694878-29 from 2725060 to 2857355 tid 3 2021-03-15T22:29:25.239 INFO:tasks.rados.rados.0.smithi076.stdout:2215: snap_remove snap 218 2021-03-15T22:29:25.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:25.295 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:25.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:25.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:25.348 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[56514]: audit 2021-03-15T22:29:24.206769+0000 mon.a (mon.0) 1693 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-15T22:29:25.349 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[56514]: cluster 2021-03-15T22:29:24.206853+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e615: 8 total, 6 up, 7 in 2021-03-15T22:29:25.349 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[56514]: audit 2021-03-15T22:29:24.207398+0000 mon.b (mon.2) 27 : audit [INF] from='osd.5 [v2:172.21.15.125:6808/978017133,v1:172.21.15.125:6809/978017133]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:29:25.349 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[56514]: audit 2021-03-15T22:29:24.207755+0000 mon.a (mon.0) 1695 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:29:25.349 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[56514]: cluster 2021-03-15T22:29:25.182472+0000 mon.a (mon.0) 1696 : cluster [WRN] Health check update: Degraded data redundancy: 178/1764 objects degraded (10.091%), 13 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:25.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: audit 2021-03-15T22:29:25.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: 2021-03-15T22:29:25.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: 2021 2021-03-15T22:29:25.350 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: - 2021-03-15T22:29:25.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: 03 2021-03-15T22:29:25.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: -15T22:29:24.206769+0000 2021-03-15T22:29:25.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: mon.a (mon.0) 1693 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-15T22:29:25.351 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:29:25.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: 22:29:24.206853+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e615: 8 total, 6 up, 7 in 2021-03-15T22:29:25.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: audit 2021-03-15T22:29:24.207398 2021-03-15T22:29:25.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: +0000 mon.b (mon.2) 27 : audit [INF] from='osd.5 [v2:172.21.15.125:6808/978017133,v1:172.21.15.125:6809/978017133]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:29:25.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:29:25.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: T22:29:24.207755+0000 mon.a (mon.0) 1695 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:29:25.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:25 smithi076 conmon[62156]: cluster 2021-03-15T22:29:25.182472+0000 mon.a (mon.0) 1696 : cluster [WRN] Health check update: Degraded data redundancy: 178/1764 objects degraded (10.091%), 13 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:25.397 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:25.419 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:25.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:25.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:25.508 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:25.522 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:25.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:25 smithi125 conmon[56739]: audit 2021-03-15T22:29:24.206769+0000 mon.a (mon.0 2021-03-15T22:29:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:25 smithi125 conmon[56739]: ) 1693 : audit [INF] from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2021-03-15T22:29:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:25 smithi125 conmon[56739]: cluster 2021-03-15T22:29:24.206853+0000 mon.a (mon.0) 1694 : cluster [DBG] osdmap e615: 8 total, 6 up, 7 in 2021-03-15T22:29:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:25 smithi125 conmon[56739]: audit 2021-03-15T22:29:24.207398+0000 mon.b (mon.2) 27 : audit [INF] from='osd.5 [v2:172.21.15.125:6808/978017133,v1:172.21.15.125:6809/978017133]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:29:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:25 smithi125 conmon[56739]: audit 2021-03-15T22:29:24.207755+0000 mon.a (mon.0) 1695 : audit [INF] from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi125", "root=default"]}]: dispatch 2021-03-15T22:29:25.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:25 smithi125 conmon[56739]: cluster 2021-03-15T22:29:25.182472+0000 mon.a (mon.0) 1696 : cluster [WRN] Health check update: Degraded data redundancy: 178/1764 objects degraded (10.091%), 13 pgs degraded, 22 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:25.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:25.547 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:25.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:25.649 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:25.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:25.672 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:25.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:25.774 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:25.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:25.797 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:25.798 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:25.899 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:25.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:25.923 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:25.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:26.025 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:26.050 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:26.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:26.052 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:26.153 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:26.175 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:26.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:26.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:26.233 INFO:tasks.rados.rados.0.smithi076.stdout:2215: done (6 left) 2021-03-15T22:29:26.234 INFO:tasks.rados.rados.0.smithi076.stdout:2216: write oid 40 current snap is 255 2021-03-15T22:29:26.234 INFO:tasks.rados.rados.0.smithi076.stdout:2216: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2021-03-15T22:29:26.234 INFO:tasks.rados.rados.0.smithi076.stdout:2216: writing smithi07694878-40 from 474162 to 950196 tid 1 2021-03-15T22:29:26.237 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:29:26.238 INFO:tasks.rados.rados.0.smithi076.stdout:2216: writing smithi07694878-40 from 1358148 to 1840079 tid 2 2021-03-15T22:29:26.245 INFO:tasks.rados.rados.0.smithi076.stdout:2216: writing smithi07694878-40 from 2352483 to 2817786 tid 3 2021-03-15T22:29:26.246 INFO:tasks.rados.rados.0.smithi076.stdout:2217: read oid 18 snap -1 2021-03-15T22:29:26.246 INFO:tasks.rados.rados.0.smithi076.stdout:2217: expect (ObjNum 655 snap 206 seq_num 655) 2021-03-15T22:29:26.246 INFO:tasks.rados.rados.0.smithi076.stdout:2218: write oid 32 current snap is 255 2021-03-15T22:29:26.247 INFO:tasks.rados.rados.0.smithi076.stdout:2218: seq_num 767 ranges {490969=551283,1816630=625589} 2021-03-15T22:29:26.251 INFO:tasks.rados.rados.0.smithi076.stdout:2218: writing smithi07694878-32 from 490969 to 1042252 tid 1 2021-03-15T22:29:26.259 INFO:tasks.rados.rados.0.smithi076.stdout:2218: writing smithi07694878-32 from 1816630 to 2442219 tid 2 2021-03-15T22:29:26.260 INFO:tasks.rados.rados.0.smithi076.stdout:2219: rollback oid 6 current snap is 255 2021-03-15T22:29:26.260 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 252 2021-03-15T22:29:26.260 INFO:tasks.rados.rados.0.smithi076.stdout:2212: expect (ObjNum 609 snap 189 seq_num 609) 2021-03-15T22:29:26.278 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:26.304 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:26.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:26.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:26.368 INFO:tasks.rados.rados.0.smithi076.stdout:2209: expect (ObjNum 444 snap 125 seq_num 444) 2021-03-15T22:29:26.407 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:26.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:26.431 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:26.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:26.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[56514]: cluster 2021-03-15T22:29:25.125995+0000 mgr.y (mgr.14144 2021-03-15T22:29:26.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[56514]: ) 798 : cluster [DBG] pgmap v1067: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 572 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 872 KiB/s wr, 3 op/s; 178/1767 objects degraded (10.074%); 81/1767 objects misplaced (4.584%) 2021-03-15T22:29:26.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[56514]: cluster 2021-03-15T22:29:25.214127+0000 mon.a (mon.0) 1697 : cluster [INF] osd.5 [v2:172.21.15.125:6808/978017133,v1:172.21.15.125:6809/978017133] boot 2021-03-15T22:29:26.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[56514]: cluster 2021-03-15T22:29:25.214172+0000 mon.a (mon.0) 1698 : cluster [DBG] osdmap e616: 8 total, 7 up, 7 in 2021-03-15T22:29:26.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[56514]: audit 2021-03-15T22:29:25.220663+0000 mon.a (mon.0) 1699 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:29:26.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[56514]: audit 2021-03-15T22:29:25.486098+0000 mon.c (mon.1) 54 : audit [INF] from='client.? 172.21.15.76:0/1668781779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:26.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[56514]: audit 2021-03-15T22:29:25.486379+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:26.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: cluster 2021-03-15T22:29:25.125995+0000 2021-03-15T22:29:26.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: mgr.y (mgr.14144) 798 : cluster [DBG] pgmap v1067: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 572 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 872 KiB/s wr, 3 op/s; 178/1767 objects degraded (10.074%); 81/1767 objects misplaced (4.584%) 2021-03-15T22:29:26.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:29:26.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: T22:29:25.214127+0000 mon.a (mon.0) 1697 : cluster [INF] osd.5 [v2:172.21.15.125:6808/978017133,v1:172.21.15.125:6809/978017133] boot 2021-03-15T22:29:26.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:29:26.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: 15T22:29:25.214172+0000 mon.a (mon.0) 2021-03-15T22:29:26.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: 1698 : cluster [DBG] osdmap e616: 8 total, 7 up, 7 in 2021-03-15T22:29:26.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: audit 2021-03-15T22:29:26.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: 2021-03-15T22:29 2021-03-15T22:29:26.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: :25.220663+0000 mon.a (mon.0) 2021-03-15T22:29:26.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: 1699 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:29:26.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: audit 2021-03-15T22:29:26.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: 2021-03-15T22:29 2021-03-15T22:29:26.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: :25.486098+0000 mon.c (mon.1) 2021-03-15T22:29:26.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: 54 : audit [INF] from='client.? 172.21.15.76:0/1668781779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: audit 2021-03-15T22:29:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:29:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: 29:25.486379+0000 mon.a (mon.0 2021-03-15T22:29:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:26 smithi076 conmon[62156]: ) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:26.533 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:26.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:26 smithi125 conmon[56739]: cluster 2021-03-15T22:29:25.125995+0000 mgr.y (mgr.14144) 798 : cluster [DBG] pgmap v1067: 138 pgs: 17 active+clean+remapped, 13 active+undersized+degraded, 25 active+undersized, 83 active+clean; 572 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 872 KiB/s wr, 3 op/s; 178/1767 objects degraded (10.074%); 81/1767 objects misplaced (4.584%) 2021-03-15T22:29:26.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:26 smithi125 conmon[56739]: cluster 2021-03-15T22:29:25.214127+0000 mon.a (mon.0) 1697 : cluster [INF] osd.5 [v2:172.21.15.125:6808/978017133,v1:172.21.15.125:6809/978017133] boot 2021-03-15T22:29:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:26 smithi125 conmon[56739]: cluster 2021-03-15T22:29:25.214172+0000 mon.a (mon.0) 1698 : cluster [DBG] osdmap e616: 8 total, 7 up, 7 in 2021-03-15T22:29:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:26 smithi125 conmon[56739]: audit 2021-03-15T22:29:25.220663+0000 mon.a (mon.0) 1699 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2021-03-15T22:29:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:26 smithi125 conmon[56739]: audit 2021-03-15T22:29:25.486098+0000 mon.c (mon.1) 54 : audit [INF] from='client.? 172.21.15.76:0/1668781779' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:26 smithi125 conmon[56739]: audit 2021-03-15T22:29:25.486379+0000 mon.a (mon.0) 1700 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:26.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:26.557 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:26.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:26.578 INFO:tasks.rados.rados.0.smithi076.stdout:2211: expect (ObjNum 658 snap 206 seq_num 658) 2021-03-15T22:29:26.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:26.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:26.683 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:26.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:26.708 INFO:tasks.rados.rados.0.smithi076.stdout:2210: finishing copy_from to smithi07694878-23 2021-03-15T22:29:26.708 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 1036 (ObjNum 732 snap 239 seq_num 732) dirty exists 2021-03-15T22:29:26.708 INFO:tasks.rados.rados.0.smithi076.stdout:2210: finishing copy_from racing read to smithi07694878-23 2021-03-15T22:29:26.709 INFO:tasks.rados.rados.0.smithi076.stdout:2214: finishing write tid 1 to smithi07694878-29 2021-03-15T22:29:26.709 INFO:tasks.rados.rados.0.smithi076.stdout:2209: done (9 left) 2021-03-15T22:29:26.709 INFO:tasks.rados.rados.0.smithi076.stdout:2210: done (8 left) 2021-03-15T22:29:26.709 INFO:tasks.rados.rados.0.smithi076.stdout:2211: done (7 left) 2021-03-15T22:29:26.710 INFO:tasks.rados.rados.0.smithi076.stdout:2212: done (6 left) 2021-03-15T22:29:26.710 INFO:tasks.rados.rados.0.smithi076.stdout:2220: delete oid 23 current snap is 255 2021-03-15T22:29:26.710 INFO:tasks.rados.rados.0.smithi076.stdout:2213: finishing copy_from to smithi07694878-13 2021-03-15T22:29:26.710 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1503 (ObjNum 745 snap 245 seq_num 745) dirty exists 2021-03-15T22:29:26.711 INFO:tasks.rados.rados.0.smithi076.stdout:2213: finishing copy_from racing read to smithi07694878-13 2021-03-15T22:29:26.711 INFO:tasks.rados.rados.0.smithi076.stdout:2214: finishing write tid 2 to smithi07694878-29 2021-03-15T22:29:26.711 INFO:tasks.rados.rados.0.smithi076.stdout:2214: finishing write tid 3 to smithi07694878-29 2021-03-15T22:29:26.711 INFO:tasks.rados.rados.0.smithi076.stdout:2214: finishing write tid 4 to smithi07694878-29 2021-03-15T22:29:26.712 INFO:tasks.rados.rados.0.smithi076.stdout:2214: finishing write tid 5 to smithi07694878-29 2021-03-15T22:29:26.712 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 1536 (ObjNum 765 snap 255 seq_num 765) dirty exists 2021-03-15T22:29:26.712 INFO:tasks.rados.rados.0.smithi076.stdout:2214: left oid 29 (ObjNum 765 snap 255 seq_num 765) 2021-03-15T22:29:26.712 INFO:tasks.rados.rados.0.smithi076.stdout:2219: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:29:26.712 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 1201 (ObjNum 261726112 snap 21896 seq_num 264924704) dirty dne 2021-03-15T22:29:26.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:26.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:26.810 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:26.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:26.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:26.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:26.936 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:26.936 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:27.037 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:27.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:27.063 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:27.063 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:27.129 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:29:27.130 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:29:27.159 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:29:27.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:27.176 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:29:27.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:27.189 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:27.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:27.194 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:29:27.214 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:29:27.217 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[56514]: cluster 2021-03-15T22:29:24.604267+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:27.218 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[56514]: cluster 2021-03-15T22:29:24.606372+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:27.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[56514]: cluster 2021-03-15T22:29:26.212379+0000 mon.a (mon.0) 1701 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:27.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[56514]: audit 2021-03-15T22:29:26.221054+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:27.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[56514]: cluster 2021-03-15T22:29:26.221121+0000 mon.a (mon.0) 1703 : cluster [DBG] osdmap e617: 8 total, 7 up, 7 in 2021-03-15T22:29:27.219 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[56514]: audit 2021-03-15T22:29:27.139961+0000 mon.a (mon.0) 1704 : audit [DBG] from='client.? 172.21.15.76:0/2165212003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:27.242 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:29:27.253 INFO:tasks.rados.rados.0.smithi076.stdout:2217: expect (ObjNum 655 snap 206 seq_num 655) 2021-03-15T22:29:27.268 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:29:27.291 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:27.300 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:29:27.301 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:29:27.301 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:29:27.302 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:29:27.302 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:29:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:29:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:29:27.303 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:29:27.304 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:29:27.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:27.328 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:27.329 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:27.430 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:27.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:27.453 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:27.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:27.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[62156]: cluster 2021-03-15T22:29:24.604267+0000 osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:27.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[62156]: cluster 2021-03-15T22:29:24.606372+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:27.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[62156]: cluster 2021-03-15T22:29:26.212379+0000 mon.a (mon.0) 1701 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:27.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[62156]: audit 2021-03-15T22:29:26.221054+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:27.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[62156]: cluster 2021-03-15T22:29:26.221121+0000 mon.a (mon.0) 1703 : cluster [DBG] osdmap e617: 8 total, 7 up, 7 in 2021-03-15T22:29:27.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[62156]: audit 2021-03-15T22:29:27.139961+0000 mon.a (mon. 2021-03-15T22:29:27.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:27 smithi076 conmon[62156]: 0) 1704 : audit [DBG] from='client.? 172.21.15.76:0/2165212003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:27.527 INFO:tasks.rados.rados.0.smithi076.stdout:2213: done (6 left) 2021-03-15T22:29:27.527 INFO:tasks.rados.rados.0.smithi076.stdout:2214: done (5 left) 2021-03-15T22:29:27.527 INFO:tasks.rados.rados.0.smithi076.stdout:2217: done (4 left) 2021-03-15T22:29:27.528 INFO:tasks.rados.rados.0.smithi076.stdout:2219: done (3 left) 2021-03-15T22:29:27.528 INFO:tasks.rados.rados.0.smithi076.stdout:2220: done (2 left) 2021-03-15T22:29:27.528 INFO:tasks.rados.rados.0.smithi076.stdout:2221: read oid 24 snap 172 2021-03-15T22:29:27.528 INFO:tasks.rados.rados.0.smithi076.stdout:2221: expect (ObjNum 514 snap 155 seq_num 514) 2021-03-15T22:29:27.529 INFO:tasks.rados.rados.0.smithi076.stdout:2222: read oid 43 snap 216 2021-03-15T22:29:27.529 INFO:tasks.rados.rados.0.smithi076.stdout:2222: expect deleted 2021-03-15T22:29:27.529 INFO:tasks.rados.rados.0.smithi076.stdout:2223: write (excl) oid 16 current snap is 255 2021-03-15T22:29:27.529 INFO:tasks.rados.rados.0.smithi076.stdout:2218: finishing write tid 1 to smithi07694878-32 2021-03-15T22:29:27.529 INFO:tasks.rados.rados.0.smithi076.stdout:2218: finishing write tid 2 to smithi07694878-32 2021-03-15T22:29:27.530 INFO:tasks.rados.rados.0.smithi076.stdout:2218: finishing write tid 3 to smithi07694878-32 2021-03-15T22:29:27.532 INFO:tasks.rados.rados.0.smithi076.stdout:2218: finishing write tid 4 to smithi07694878-32 2021-03-15T22:29:27.532 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 1540 (ObjNum 767 snap 255 seq_num 767) dirty exists 2021-03-15T22:29:27.533 INFO:tasks.rados.rados.0.smithi076.stdout:2218: left oid 32 (ObjNum 767 snap 255 seq_num 767) 2021-03-15T22:29:27.533 INFO:tasks.rados.rados.0.smithi076.stdout:2216: finishing write tid 1 to smithi07694878-40 2021-03-15T22:29:27.533 INFO:tasks.rados.rados.0.smithi076.stdout:2216: finishing write tid 2 to smithi07694878-40 2021-03-15T22:29:27.533 INFO:tasks.rados.rados.0.smithi076.stdout:2216: finishing write tid 3 to smithi07694878-40 2021-03-15T22:29:27.534 INFO:tasks.rados.rados.0.smithi076.stdout:2216: finishing write tid 4 to smithi07694878-40 2021-03-15T22:29:27.534 INFO:tasks.rados.rados.0.smithi076.stdout:2216: finishing write tid 5 to smithi07694878-40 2021-03-15T22:29:27.534 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1545 (ObjNum 766 snap 255 seq_num 766) dirty exists 2021-03-15T22:29:27.534 INFO:tasks.rados.rados.0.smithi076.stdout:2216: left oid 40 (ObjNum 766 snap 255 seq_num 766) 2021-03-15T22:29:27.539 INFO:tasks.rados.rados.0.smithi076.stdout:2223: seq_num 769 ranges {524583=418134,1649947=629258,2834752=89831} 2021-03-15T22:29:27.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:27 smithi125 conmon[56739]: cluster 2021-03-15T22:29:24.604267+0000 2021-03-15T22:29:27.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:27 smithi125 conmon[56739]: osd.5 (osd.5) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:27.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:27 smithi125 conmon[56739]: cluster 2021-03-15T22:29:24.606372+0000 osd.5 (osd.5) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:27.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:27 smithi125 conmon[56739]: cluster 2021-03-15T22:29:26.212379+0000 mon.a (mon.0) 1701 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:27.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:27 smithi125 conmon[56739]: audit 2021-03-15T22:29:26.221054+0000 mon.a (mon.0) 1702 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:27.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:27 smithi125 conmon[56739]: cluster 2021-03-15T22:29:26.221121+0000 mon.a (mon.0) 1703 : cluster [DBG] osdmap e617: 8 total, 7 up, 7 in 2021-03-15T22:29:27.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:27 smithi125 conmon[56739]: audit 2021-03-15T22:29:27.139961+0000 mon.a (mon.0) 1704 : audit [DBG] from='client.? 172.21.15.76:0/2165212003' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:27.543 INFO:tasks.rados.rados.0.smithi076.stdout:2223: writing smithi07694878-16 from 524583 to 942717 tid 1 2021-03-15T22:29:27.543 INFO:tasks.rados.rados.0.smithi076.stdout:2223: writing smithi07694878-16 from 1649947 to 2279205 tid 2 2021-03-15T22:29:27.544 INFO:tasks.rados.rados.0.smithi076.stdout:2223: writing smithi07694878-16 from 2834752 to 2924583 tid 3 2021-03-15T22:29:27.544 INFO:tasks.rados.rados.0.smithi076.stdout:2216: done (4 left) 2021-03-15T22:29:27.544 INFO:tasks.rados.rados.0.smithi076.stdout:2218: done (3 left) 2021-03-15T22:29:27.544 INFO:tasks.rados.rados.0.smithi076.stdout:2224: write oid 12 current snap is 255 2021-03-15T22:29:27.545 INFO:tasks.rados.rados.0.smithi076.stdout:2224: seq_num 770 ranges {541390=493383,1592076=489269,2862175=79215} 2021-03-15T22:29:27.548 INFO:tasks.rados.rados.0.smithi076.stdout:2224: writing smithi07694878-12 from 541390 to 1034773 tid 1 2021-03-15T22:29:27.553 INFO:tasks.rados.rados.0.smithi076.stdout:2224: writing smithi07694878-12 from 1592076 to 2081345 tid 2 2021-03-15T22:29:27.555 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:27.558 INFO:tasks.rados.rados.0.smithi076.stdout:2224: writing smithi07694878-12 from 2862175 to 2941390 tid 3 2021-03-15T22:29:27.559 INFO:tasks.rados.rados.0.smithi076.stdout:2225: copy_from oid 20 from oid 49 current snap is 255 2021-03-15T22:29:27.559 INFO:tasks.rados.rados.0.smithi076.stdout:2221: expect (ObjNum 514 snap 155 seq_num 514) 2021-03-15T22:29:27.578 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:27.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:27.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:27.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:27.703 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:27.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:27.704 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:27.745 INFO:tasks.rados.rados.0.smithi076.stdout:2223: finishing write tid 1 to smithi07694878-16 2021-03-15T22:29:27.745 INFO:tasks.rados.rados.0.smithi076.stdout:2221: done (4 left) 2021-03-15T22:29:27.745 INFO:tasks.rados.rados.0.smithi076.stdout:2222: done (3 left) 2021-03-15T22:29:27.746 INFO:tasks.rados.rados.0.smithi076.stdout:2226: delete oid 39 current snap is 255 2021-03-15T22:29:27.746 INFO:tasks.rados.rados.0.smithi076.stdout:2223: finishing write tid 2 to smithi07694878-16 2021-03-15T22:29:27.746 INFO:tasks.rados.rados.0.smithi076.stdout:2223: finishing write tid 3 to smithi07694878-16 2021-03-15T22:29:27.746 INFO:tasks.rados.rados.0.smithi076.stdout:2223: finishing write tid 4 to smithi07694878-16 2021-03-15T22:29:27.747 INFO:tasks.rados.rados.0.smithi076.stdout:2223: finishing write tid 5 to smithi07694878-16 2021-03-15T22:29:27.747 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1570 (ObjNum 769 snap 255 seq_num 769) dirty exists 2021-03-15T22:29:27.747 INFO:tasks.rados.rados.0.smithi076.stdout:2223: left oid 16 (ObjNum 769 snap 255 seq_num 769) 2021-03-15T22:29:27.747 INFO:tasks.rados.rados.0.smithi076.stdout:2224: finishing write tid 1 to smithi07694878-12 2021-03-15T22:29:27.747 INFO:tasks.rados.rados.0.smithi076.stdout:2224: finishing write tid 2 to smithi07694878-12 2021-03-15T22:29:27.748 INFO:tasks.rados.rados.0.smithi076.stdout:2224: finishing write tid 3 to smithi07694878-12 2021-03-15T22:29:27.748 INFO:tasks.rados.rados.0.smithi076.stdout:2224: finishing write tid 4 to smithi07694878-12 2021-03-15T22:29:27.748 INFO:tasks.rados.rados.0.smithi076.stdout:2224: finishing write tid 5 to smithi07694878-12 2021-03-15T22:29:27.748 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 1574 (ObjNum 770 snap 255 seq_num 770) dirty exists 2021-03-15T22:29:27.749 INFO:tasks.rados.rados.0.smithi076.stdout:2224: left oid 12 (ObjNum 770 snap 255 seq_num 770) 2021-03-15T22:29:27.749 INFO:tasks.rados.rados.0.smithi076.stdout:2225: finishing copy_from to smithi07694878-20 2021-03-15T22:29:27.749 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 1529 (ObjNum 543 snap 172 seq_num 543) dirty exists 2021-03-15T22:29:27.750 INFO:tasks.rados.rados.0.smithi076.stdout:2225: finishing copy_from racing read to smithi07694878-20 2021-03-15T22:29:27.751 INFO:tasks.rados.rados.0.smithi076.stdout:2223: done (3 left) 2021-03-15T22:29:27.752 INFO:tasks.rados.rados.0.smithi076.stdout:2224: done (2 left) 2021-03-15T22:29:27.752 INFO:tasks.rados.rados.0.smithi076.stdout:2225: done (1 left) 2021-03-15T22:29:27.752 INFO:tasks.rados.rados.0.smithi076.stdout:2226: done (0 left) 2021-03-15T22:29:27.752 INFO:tasks.rados.rados.0.smithi076.stdout:2227: read oid 5 snap 172 2021-03-15T22:29:27.752 INFO:tasks.rados.rados.0.smithi076.stdout:2227: expect deleted 2021-03-15T22:29:27.753 INFO:tasks.rados.rados.0.smithi076.stdout:2228: rollback oid 31 current snap is 255 2021-03-15T22:29:27.753 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 31 to 246 2021-03-15T22:29:27.753 INFO:tasks.rados.rados.0.smithi076.stdout:2229: rollback oid 22 current snap is 255 2021-03-15T22:29:27.753 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 22 to 242 2021-03-15T22:29:27.753 INFO:tasks.rados.rados.0.smithi076.stdout:2230: rollback oid 24 current snap is 255 2021-03-15T22:29:27.754 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 228 2021-03-15T22:29:27.754 INFO:tasks.rados.rados.0.smithi076.stdout:2231: write (excl) oid 17 current snap is 255 2021-03-15T22:29:27.754 INFO:tasks.rados.rados.0.smithi076.stdout:2231: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2021-03-15T22:29:27.755 INFO:tasks.rados.rados.0.smithi076.stdout:2231: writing smithi07694878-17 from 575004 to 1218885 tid 1 2021-03-15T22:29:27.758 INFO:tasks.rados.rados.0.smithi076.stdout:2231: writing smithi07694878-17 from 1709040 to 2201978 tid 2 2021-03-15T22:29:27.764 INFO:tasks.rados.rados.0.smithi076.stdout:2231: writing smithi07694878-17 from 2917021 to 2975004 tid 3 2021-03-15T22:29:27.764 INFO:tasks.rados.rados.0.smithi076.stdout:2232: rollback oid 37 current snap is 255 2021-03-15T22:29:27.765 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 37 to 251 2021-03-15T22:29:27.765 INFO:tasks.rados.rados.0.smithi076.stdout:2228: finishing rollback tid 0 to smithi07694878-31 2021-03-15T22:29:27.765 INFO:tasks.rados.rados.0.smithi076.stdout:2228: finishing rollback tid 1 to smithi07694878-31 2021-03-15T22:29:27.765 INFO:tasks.rados.rados.0.smithi076.stdout:2228: finishing rollback tid 2 to smithi07694878-31 2021-03-15T22:29:27.766 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 1039 (ObjNum 709 snap 227 seq_num 709) dirty exists 2021-03-15T22:29:27.766 INFO:tasks.rados.rados.0.smithi076.stdout:2229: finishing rollback tid 0 to smithi07694878-22 2021-03-15T22:29:27.766 INFO:tasks.rados.rados.0.smithi076.stdout:2229: finishing rollback tid 1 to smithi07694878-22 2021-03-15T22:29:27.766 INFO:tasks.rados.rados.0.smithi076.stdout:2229: finishing rollback tid 2 to smithi07694878-22 2021-03-15T22:29:27.767 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 1578 (ObjNum 674 snap 214 seq_num 674) dirty exists 2021-03-15T22:29:27.767 INFO:tasks.rados.rados.0.smithi076.stdout:2230: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:29:27.767 INFO:tasks.rados.rados.0.smithi076.stdout:2227: done (5 left) 2021-03-15T22:29:27.767 INFO:tasks.rados.rados.0.smithi076.stdout:2228: done (4 left) 2021-03-15T22:29:27.768 INFO:tasks.rados.rados.0.smithi076.stdout:2229: done (3 left) 2021-03-15T22:29:27.768 INFO:tasks.rados.rados.0.smithi076.stdout:2233: write oid 27 current snap is 255 2021-03-15T22:29:27.768 INFO:tasks.rados.rados.0.smithi076.stdout:2230: finishing rollback tid 2 to smithi07694878-24 2021-03-15T22:29:27.769 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 611 (ObjNum 638 snap 200 seq_num 638) dirty exists 2021-03-15T22:29:27.769 INFO:tasks.rados.rados.0.smithi076.stdout:2231: finishing write tid 1 to smithi07694878-17 2021-03-15T22:29:27.769 INFO:tasks.rados.rados.0.smithi076.stdout:2231: finishing write tid 2 to smithi07694878-17 2021-03-15T22:29:27.769 INFO:tasks.rados.rados.0.smithi076.stdout:2233: seq_num 773 ranges {591811=719130,1767522=636596,2828091=163720} 2021-03-15T22:29:27.770 INFO:tasks.rados.rados.0.smithi076.stdout:2233: writing smithi07694878-27 from 591811 to 1310941 tid 1 2021-03-15T22:29:27.777 INFO:tasks.rados.rados.0.smithi076.stdout:2233: writing smithi07694878-27 from 1767522 to 2404118 tid 2 2021-03-15T22:29:27.781 INFO:tasks.rados.rados.0.smithi076.stdout:2233: writing smithi07694878-27 from 2828091 to 2991811 tid 3 2021-03-15T22:29:27.781 INFO:tasks.rados.rados.0.smithi076.stdout:2230: done (3 left) 2021-03-15T22:29:27.781 INFO:tasks.rados.rados.0.smithi076.stdout:2234: read oid 41 snap -1 2021-03-15T22:29:27.782 INFO:tasks.rados.rados.0.smithi076.stdout:2234: expect (ObjNum 689 snap 220 seq_num 689) 2021-03-15T22:29:27.782 INFO:tasks.rados.rados.0.smithi076.stdout:2235: copy_from oid 11 from oid 49 current snap is 255 2021-03-15T22:29:27.782 INFO:tasks.rados.rados.0.smithi076.stdout:2231: finishing write tid 3 to smithi07694878-17 2021-03-15T22:29:27.782 INFO:tasks.rados.rados.0.smithi076.stdout:2236: read oid 34 snap -1 2021-03-15T22:29:27.783 INFO:tasks.rados.rados.0.smithi076.stdout:2236: expect (ObjNum 764 snap 254 seq_num 764) 2021-03-15T22:29:27.783 INFO:tasks.rados.rados.0.smithi076.stdout:2237: snap_create 2021-03-15T22:29:27.783 INFO:tasks.rados.rados.0.smithi076.stdout:2231: finishing write tid 4 to smithi07694878-17 2021-03-15T22:29:27.783 INFO:tasks.rados.rados.0.smithi076.stdout:2231: finishing write tid 5 to smithi07694878-17 2021-03-15T22:29:27.784 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 849 (ObjNum 772 snap 255 seq_num 772) dirty exists 2021-03-15T22:29:27.784 INFO:tasks.rados.rados.0.smithi076.stdout:2231: left oid 17 (ObjNum 772 snap 255 seq_num 772) 2021-03-15T22:29:27.784 INFO:tasks.rados.rados.0.smithi076.stdout:2233: finishing write tid 1 to smithi07694878-27 2021-03-15T22:29:27.784 INFO:tasks.rados.rados.0.smithi076.stdout:2232: finishing rollback tid 0 to smithi07694878-37 2021-03-15T22:29:27.785 INFO:tasks.rados.rados.0.smithi076.stdout:2232: finishing rollback tid 1 to smithi07694878-37 2021-03-15T22:29:27.785 INFO:tasks.rados.rados.0.smithi076.stdout:2232: finishing rollback tid 2 to smithi07694878-37 2021-03-15T22:29:27.785 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 1549 (ObjNum 751 snap 245 seq_num 751) dirty exists 2021-03-15T22:29:27.785 INFO:tasks.rados.rados.0.smithi076.stdout:2233: finishing write tid 2 to smithi07694878-27 2021-03-15T22:29:27.787 INFO:tasks.rados.rados.0.smithi076.stdout:2233: finishing write tid 3 to smithi07694878-27 2021-03-15T22:29:27.787 INFO:tasks.rados.rados.0.smithi076.stdout:2233: finishing write tid 4 to smithi07694878-27 2021-03-15T22:29:27.787 INFO:tasks.rados.rados.0.smithi076.stdout:2233: finishing write tid 5 to smithi07694878-27 2021-03-15T22:29:27.787 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1198 (ObjNum 773 snap 255 seq_num 773) dirty exists 2021-03-15T22:29:27.788 INFO:tasks.rados.rados.0.smithi076.stdout:2233: left oid 27 (ObjNum 773 snap 255 seq_num 773) 2021-03-15T22:29:27.788 INFO:tasks.rados.rados.0.smithi076.stdout:2234: expect (ObjNum 689 snap 220 seq_num 689) 2021-03-15T22:29:27.806 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:27.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:27.828 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:27.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:27.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-15T22:29:27.899 INFO:tasks.rados.rados.0.smithi076.stdout:2236: expect (ObjNum 764 snap 254 seq_num 764) 2021-03-15T22:29:27.930 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:27.951 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:27.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:27.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:28.026 INFO:tasks.rados.rados.0.smithi076.stdout:2235: finishing copy_from to smithi07694878-11 2021-03-15T22:29:28.026 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1529 (ObjNum 543 snap 172 seq_num 543) dirty exists 2021-03-15T22:29:28.026 INFO:tasks.rados.rados.0.smithi076.stdout:2235: finishing copy_from racing read to smithi07694878-11 2021-03-15T22:29:28.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:28.065 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:29:28.065 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:29:28.077 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:28.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:28.079 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:28.127 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:28.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:28.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:28.203 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:28.204 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:28.243 INFO:tasks.rados.rados.0.smithi076.stdout:2231: done (6 left) 2021-03-15T22:29:28.243 INFO:tasks.rados.rados.0.smithi076.stdout:2232: done (5 left) 2021-03-15T22:29:28.243 INFO:tasks.rados.rados.0.smithi076.stdout:2233: done (4 left) 2021-03-15T22:29:28.244 INFO:tasks.rados.rados.0.smithi076.stdout:2234: done (3 left) 2021-03-15T22:29:28.244 INFO:tasks.rados.rados.0.smithi076.stdout:2235: done (2 left) 2021-03-15T22:29:28.244 INFO:tasks.rados.rados.0.smithi076.stdout:2236: done (1 left) 2021-03-15T22:29:28.244 INFO:tasks.rados.rados.0.smithi076.stdout:2237: done (0 left) 2021-03-15T22:29:28.245 INFO:tasks.rados.rados.0.smithi076.stdout:2238: snap_remove snap 251 2021-03-15T22:29:28.305 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:28.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:28.329 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:28.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:28.431 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:28.453 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:28.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:28.455 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:28 smithi076 conmon[56514]: cluster 2021-03-15T22:29:27 2021-03-15T22:29:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:28 smithi076 conmon[56514]: .126718+0000 mgr.y (mgr.14144) 799 : cluster [DBG] pgmap v1070: 138 pgs: 7 remapped+peering, 17 active+clean+remapped, 9 active+undersized+degraded, 22 active+undersized, 83 active+clean; 582 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 14 op/s; 130/1782 objects degraded (7.295%); 81/1782 objects misplaced (4.545%) 2021-03-15T22:29:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:28 smithi076 conmon[56514]: cluster 2021-03-15T22:29:27.228033+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e618: 8 total, 7 up, 7 in 2021-03-15T22:29:28.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:28 smithi076 conmon[62156]: cluster 2021-03-15T22:29:27.126718+0000 mgr.y (mgr.14144 2021-03-15T22:29:28.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:28 smithi076 conmon[62156]: ) 799 : cluster [DBG] pgmap v1070: 138 pgs: 7 remapped+peering, 17 active+clean+remapped, 9 active+undersized+degraded, 22 active+undersized, 83 active+clean; 582 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 14 op/s; 130/1782 objects degraded (7.295%); 81/1782 objects misplaced (4.545%) 2021-03-15T22:29:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:28 smithi076 conmon[62156]: cluster 2021-03-15T22:29:27.228033+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e618: 8 total, 7 up, 7 in 2021-03-15T22:29:28.520 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:28 smithi125 conmon[56739]: cluster 2021-03-15T22:29:27.126718+0000 mgr.y 2021-03-15T22:29:28.520 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:28 smithi125 conmon[56739]: (mgr.14144) 799 : cluster [DBG] pgmap v1070: 138 pgs: 7 remapped+peering, 17 active+clean+remapped, 9 active+undersized+degraded, 22 active+undersized, 83 active+clean; 582 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 2.3 MiB/s wr, 14 op/s; 130/1782 objects degraded (7.295%); 81/1782 objects misplaced (4.545%) 2021-03-15T22:29:28.520 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:28 smithi125 conmon[56739]: cluster 2021-03-15T22:29:27.228033+0000 mon.a (mon.0) 1705 : cluster [DBG] osdmap e618: 8 total, 7 up, 7 in 2021-03-15T22:29:28.556 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:28.579 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:28.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:28.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:28.682 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:28.705 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:28.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:28.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:28.808 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:28.829 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:28.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:28.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:28.932 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:28.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:28.955 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:28.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:28.988 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:29:28.990 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:29:29.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:29.080 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:29.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:29.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:29.183 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:29.208 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:29.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:29.210 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:29.251 INFO:tasks.rados.rados.0.smithi076.stdout:2238: done (0 left) 2021-03-15T22:29:29.252 INFO:tasks.rados.rados.0.smithi076.stdout:2239: write (excl) oid 30 current snap is 256 2021-03-15T22:29:29.252 INFO:tasks.rados.rados.0.smithi076.stdout:2239: seq_num 774 ranges {608618=794379,2109651=780254} 2021-03-15T22:29:29.264 INFO:tasks.rados.rados.0.smithi076.stdout:2239: writing smithi07694878-30 from 608618 to 1402997 tid 1 2021-03-15T22:29:29.273 INFO:tasks.rados.rados.0.smithi076.stdout:2239: writing smithi07694878-30 from 2109651 to 2889905 tid 2 2021-03-15T22:29:29.273 INFO:tasks.rados.rados.0.smithi076.stdout:2240: rollback oid 50 current snap is 256 2021-03-15T22:29:29.273 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 50 to 223 2021-03-15T22:29:29.273 INFO:tasks.rados.rados.0.smithi076.stdout:2239: finishing write tid 1 to smithi07694878-30 2021-03-15T22:29:29.274 INFO:tasks.rados.rados.0.smithi076.stdout:2241: write oid 10 current snap is 256 2021-03-15T22:29:29.274 INFO:tasks.rados.rados.0.smithi076.stdout:2241: seq_num 775 ranges {625425=469628,1768133=640265} 2021-03-15T22:29:29.279 INFO:tasks.rados.rados.0.smithi076.stdout:2241: writing smithi07694878-10 from 625425 to 1095053 tid 1 2021-03-15T22:29:29.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:29 smithi125 conmon[56739]: cluster 2021-03-15T22:29:28.237981+0000 mon.a (mon.0) 1706 : cluster [DBG] osdmap e619: 8 total, 7 up, 7 in 2021-03-15T22:29:29.290 INFO:tasks.rados.rados.0.smithi076.stdout:2241: writing smithi07694878-10 from 1768133 to 2408398 tid 2 2021-03-15T22:29:29.291 INFO:tasks.rados.rados.0.smithi076.stdout:2242: copy_from oid 22 from oid 42 current snap is 256 2021-03-15T22:29:29.291 INFO:tasks.rados.rados.0.smithi076.stdout:2240: finishing rollback tid 0 to smithi07694878-50 2021-03-15T22:29:29.291 INFO:tasks.rados.rados.0.smithi076.stdout:2239: finishing write tid 2 to smithi07694878-30 2021-03-15T22:29:29.291 INFO:tasks.rados.rados.0.smithi076.stdout:2239: finishing write tid 3 to smithi07694878-30 2021-03-15T22:29:29.292 INFO:tasks.rados.rados.0.smithi076.stdout:2240: finishing rollback tid 1 to smithi07694878-50 2021-03-15T22:29:29.292 INFO:tasks.rados.rados.0.smithi076.stdout:2240: finishing rollback tid 2 to smithi07694878-50 2021-03-15T22:29:29.292 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 817 (ObjNum 658 snap 206 seq_num 658) dirty exists 2021-03-15T22:29:29.292 INFO:tasks.rados.rados.0.smithi076.stdout:2239: finishing write tid 4 to smithi07694878-30 2021-03-15T22:29:29.293 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 1586 (ObjNum 774 snap 256 seq_num 774) dirty exists 2021-03-15T22:29:29.293 INFO:tasks.rados.rados.0.smithi076.stdout:2239: left oid 30 (ObjNum 774 snap 256 seq_num 774) 2021-03-15T22:29:29.293 INFO:tasks.rados.rados.0.smithi076.stdout:2241: finishing write tid 1 to smithi07694878-10 2021-03-15T22:29:29.293 INFO:tasks.rados.rados.0.smithi076.stdout:2239: done (3 left) 2021-03-15T22:29:29.293 INFO:tasks.rados.rados.0.smithi076.stdout:2240: done (2 left) 2021-03-15T22:29:29.294 INFO:tasks.rados.rados.0.smithi076.stdout:2243: copy_from oid 5 from oid 39 current snap is 256 2021-03-15T22:29:29.294 INFO:tasks.rados.rados.0.smithi076.stdout:2244: read oid 4 snap -1 2021-03-15T22:29:29.294 INFO:tasks.rados.rados.0.smithi076.stdout:2244: expect (ObjNum 624 snap 196 seq_num 624) 2021-03-15T22:29:29.294 INFO:tasks.rados.rados.0.smithi076.stdout:2245: rollback oid 44 current snap is 256 2021-03-15T22:29:29.294 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 44 to 255 2021-03-15T22:29:29.295 INFO:tasks.rados.rados.0.smithi076.stdout:2246: rollback oid 35 current snap is 256 2021-03-15T22:29:29.295 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 35 to 211 2021-03-15T22:29:29.295 INFO:tasks.rados.rados.0.smithi076.stdout:2247: snap_create 2021-03-15T22:29:29.295 INFO:tasks.rados.rados.0.smithi076.stdout:2243: finishing copy_from to smithi07694878-5 2021-03-15T22:29:29.296 INFO:tasks.rados.rados.0.smithi076.stdout:2243: got expected ENOENT (src dne) 2021-03-15T22:29:29.296 INFO:tasks.rados.rados.0.smithi076.stdout:2243: finishing copy_from racing read to smithi07694878-5 2021-03-15T22:29:29.296 INFO:tasks.rados.rados.0.smithi076.stdout:2245: finishing rollback tid 0 to smithi07694878-44 2021-03-15T22:29:29.296 INFO:tasks.rados.rados.0.smithi076.stdout:2245: finishing rollback tid 1 to smithi07694878-44 2021-03-15T22:29:29.296 INFO:tasks.rados.rados.0.smithi076.stdout:2245: finishing rollback tid 2 to smithi07694878-44 2021-03-15T22:29:29.297 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1201 (ObjNum 747 snap 245 seq_num 747) dirty exists 2021-03-15T22:29:29.297 INFO:tasks.rados.rados.0.smithi076.stdout:2244: expect (ObjNum 624 snap 196 seq_num 624) 2021-03-15T22:29:29.298 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:29.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:29.333 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:29.335 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:29.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:29.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:29.381 INFO:tasks.rados.rados.0.smithi076.stdout:2246: finishing rollback tid 0 to smithi07694878-35 2021-03-15T22:29:29.381 INFO:tasks.rados.rados.0.smithi076.stdout:2246: finishing rollback tid 1 to smithi07694878-35 2021-03-15T22:29:29.382 INFO:tasks.rados.rados.0.smithi076.stdout:2241: finishing write tid 2 to smithi07694878-10 2021-03-15T22:29:29.382 INFO:tasks.rados.rados.0.smithi076.stdout:2241: finishing write tid 3 to smithi07694878-10 2021-03-15T22:29:29.382 INFO:tasks.rados.rados.0.smithi076.stdout:2241: finishing write tid 4 to smithi07694878-10 2021-03-15T22:29:29.382 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 1317 (ObjNum 775 snap 256 seq_num 775) dirty exists 2021-03-15T22:29:29.383 INFO:tasks.rados.rados.0.smithi076.stdout:2241: left oid 10 (ObjNum 775 snap 256 seq_num 775) 2021-03-15T22:29:29.383 INFO:tasks.rados.rados.0.smithi076.stdout:2246: finishing rollback tid 2 to smithi07694878-35 2021-03-15T22:29:29.383 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 1039 (ObjNum 629 snap 198 seq_num 629) dirty exists 2021-03-15T22:29:29.383 INFO:tasks.rados.rados.0.smithi076.stdout:2242: finishing copy_from to smithi07694878-22 2021-03-15T22:29:29.383 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 1512 (ObjNum 750 snap 245 seq_num 750) dirty exists 2021-03-15T22:29:29.384 INFO:tasks.rados.rados.0.smithi076.stdout:2242: finishing copy_from racing read to smithi07694878-22 2021-03-15T22:29:29.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:29.460 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:29.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:29.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:29 smithi076 conmon[56514]: cluster 2021-03-15T22:29:28.237981 2021-03-15T22:29:29.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:29 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1706 : cluster [DBG] osdmap e619: 8 total, 7 up, 7 in 2021-03-15T22:29:29.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:29 smithi076 conmon[62156]: cluster 2021-03-15T22:29:28.237981+0000 mon.a (mon.0) 1706 : cluster [DBG] osdmap e619: 8 total, 7 up, 7 in 2021-03-15T22:29:29.563 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:29.587 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:29.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:29.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:29.645 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:29:29.645 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:29:29.645 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:29:29.646 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:29:29.690 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:29.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:29.713 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:29.713 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:29.815 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:29.837 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:29.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:29.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:29.939 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:29.952 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:29:29 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:29:29.961 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:29.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:29.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:30.064 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:30.067 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:29:30.088 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:30.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:30.090 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:30.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:30.200 INFO:tasks.rados.rados.0.smithi076.stdout:2241: done (6 left) 2021-03-15T22:29:30.200 INFO:tasks.rados.rados.0.smithi076.stdout:2242: done (5 left) 2021-03-15T22:29:30.200 INFO:tasks.rados.rados.0.smithi076.stdout:2243: done (4 left) 2021-03-15T22:29:30.200 INFO:tasks.rados.rados.0.smithi076.stdout:2244: done (3 left) 2021-03-15T22:29:30.201 INFO:tasks.rados.rados.0.smithi076.stdout:2245: done (2 left) 2021-03-15T22:29:30.201 INFO:tasks.rados.rados.0.smithi076.stdout:2246: done (1 left) 2021-03-15T22:29:30.201 INFO:tasks.rados.rados.0.smithi076.stdout:2247: done (0 left) 2021-03-15T22:29:30.201 INFO:tasks.rados.rados.0.smithi076.stdout:2248: snap_create 2021-03-15T22:29:30.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:30.222 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:30.222 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:30.324 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:30.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:30.347 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:30.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:30.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[56514]: cluster 2021-03-15T22:29:29.127352+0000 mgr.y (mgr.14144 2021-03-15T22:29:30.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[56514]: ) 800 : cluster [DBG] pgmap v1073: 138 pgs: 7 remapped+peering, 22 active+clean+remapped, 7 active+undersized+degraded, 18 active+undersized, 1 active+recovering+degraded+remapped, 83 active+clean; 580 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 3.2 MiB/s wr, 22 op/s; 117/1788 objects degraded (6.544%); 96/1788 objects misplaced (5.369%); 8.8 MiB/s, 4 objects/s recovering 2021-03-15T22:29:30.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[56514]: cluster 2021-03-15T22:29:29.244027+0000 mon.a (mon.0) 1707 : cluster [DBG] osdmap e620: 8 total, 7 up, 7 in 2021-03-15T22:29:30.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[56514]: audit 2021-03-15T22:29:29.935511+0000 mgr.y (mgr.14144) 801 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:30.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[56514]: cluster 2021-03-15T22:29:30.183893+0000 mon.a (mon.0) 1708 : cluster [WRN] Health check update: Degraded data redundancy: 117/1788 objects degraded (6.544%), 8 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:30.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[56514]: cluster 2021-03-15T22:29:30.195455+0000 mon.a (mon.0) 1709 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2021-03-15T22:29:30.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[62156]: cluster 2021-03-15T22:29:29.127352+0000 mgr.y (mgr.14144 2021-03-15T22:29:30.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[62156]: ) 800 : cluster [DBG] pgmap v1073: 138 pgs: 7 remapped+peering, 22 active+clean+remapped, 7 active+undersized+degraded, 18 active+undersized, 1 active+recovering+degraded+remapped, 83 active+clean; 580 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 3.2 MiB/s wr, 22 op/s; 117/1788 objects degraded (6.544%); 96/1788 objects misplaced (5.369%); 8.8 MiB/s, 4 objects/s recovering 2021-03-15T22:29:30.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[62156]: cluster 2021-03-15T22:29:29.244027+0000 mon.a (mon.0) 1707 : cluster [DBG] osdmap e620: 8 total, 7 up, 7 in 2021-03-15T22:29:30.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[62156]: audit 2021-03-15T22:29:29.935511+0000 mgr.y (mgr.14144) 801 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:30.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[62156]: cluster 2021-03-15T22:29:30.183893+0000 mon.a (mon.0) 1708 : cluster [WRN] Health check update: Degraded data redundancy: 117/1788 objects degraded (6.544%), 8 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:30.421 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:30 smithi076 conmon[62156]: cluster 2021-03-15T22:29:30.195455+0000 mon.a (mon.0) 1709 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2021-03-15T22:29:30.449 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:30.459 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:30.474 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:30.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:30.476 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:30.488 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:30 smithi125 conmon[56739]: cluster 2021-03-15T22:29:29.127352+0000 mgr.y (mgr. 2021-03-15T22:29:30.488 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:30 smithi125 conmon[56739]: 14144) 800 : cluster [DBG] pgmap v1073: 138 pgs: 7 remapped+peering, 22 active+clean+remapped, 7 active+undersized+degraded, 18 active+undersized, 1 active+recovering+degraded+remapped, 83 active+clean; 580 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 3.2 MiB/s wr, 22 op/s; 117/1788 objects degraded (6.544%); 96/1788 objects misplaced (5.369%); 8.8 MiB/s, 4 objects/s recovering 2021-03-15T22:29:30.489 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:30 smithi125 conmon[56739]: cluster 2021-03-15T22:29:29.244027+0000 mon.a (mon.0) 1707 : cluster [DBG] osdmap e620: 8 total, 7 up, 7 in 2021-03-15T22:29:30.489 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:30 smithi125 conmon[56739]: audit 2021-03-15T22:29:29.935511+0000 mgr.y (mgr.14144) 801 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:30.489 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:30 smithi125 conmon[56739]: cluster 2021-03-15T22:29:30.183893+0000 mon.a (mon.0) 1708 : cluster [WRN] Health check update: Degraded data redundancy: 117/1788 objects degraded (6.544%), 8 pgs degraded, 17 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:30.489 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:30 smithi125 conmon[56739]: cluster 2021-03-15T22:29:30.195455+0000 mon.a 2021-03-15T22:29:30.490 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:30 smithi125 conmon[56739]: (mon.0) 1709 : cluster [DBG] osdmap e621: 8 total, 7 up, 7 in 2021-03-15T22:29:30.509 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-15T22:29:30.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:30.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:30.602 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:30.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:30.704 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:30.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:30.728 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:30.728 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:30.829 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:30.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:30.856 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:30.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:30.958 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:30.980 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:30.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:30.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:30.996 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:31.082 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:31.104 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:31.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:31.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:31.207 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:31.208 INFO:tasks.rados.rados.0.smithi076.stdout:2248: done (0 left) 2021-03-15T22:29:31.208 INFO:tasks.rados.rados.0.smithi076.stdout:2249: read oid 39 snap 216 2021-03-15T22:29:31.208 INFO:tasks.rados.rados.0.smithi076.stdout:2249: expect (ObjNum 664 snap 209 seq_num 664) 2021-03-15T22:29:31.209 INFO:tasks.rados.rados.0.smithi076.stdout:2250: copy_from oid 40 from oid 13 current snap is 258 2021-03-15T22:29:31.209 INFO:tasks.rados.rados.0.smithi076.stdout:2251: snap_remove snap 231 2021-03-15T22:29:31.214 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:29:31.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:31.234 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:31.234 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:31.335 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:31.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:31.361 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:31.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:31.434 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:29:31.434 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:29:31.435 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:29:31.435 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:29:31.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:31 smithi076 conmon[56514]: audit 2021-03-15T22:29:31.055140+0000 mon.c (mon.1) 55 : audit [INF] from='client.? 172.21.15.76:0/1465815681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:31.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:31 smithi076 conmon[56514]: audit 2021-03-15T22:29:31.055474+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:31.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:31 smithi076 conmon[56514]: audit 2021-03-15T22:29:31.205470+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:29:31.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:31 smithi076 conmon[56514]: cluster 2021-03-15T22:29:31.205539+0000 mon.a (mon.0) 1712 : cluster [DBG] osdmap e622: 8 total, 7 up, 7 in 2021-03-15T22:29:31.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:31 smithi076 conmon[62156]: audit 2021-03-15T22:29:31 2021-03-15T22:29:31.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:31 smithi076 conmon[62156]: .055140+0000 mon.c (mon.1) 55 : audit [INF] from='client.? 172.21.15.76:0/1465815681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:31.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:31 smithi076 conmon[62156]: audit 2021-03-15T22:29:31.055474+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:31.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:31 smithi076 conmon[62156]: audit 2021-03-15T22:29:31.205470+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:29:31.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:31 smithi076 conmon[62156]: cluster 2021-03-15T22:29:31.205539+0000 mon.a (mon.0) 1712 : cluster [DBG] osdmap e622: 8 total, 7 up, 7 in 2021-03-15T22:29:31.462 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:31.485 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:31.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:31.487 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:31.498 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:31 smithi125 conmon[56739]: audit 2021-03-15T22:29:31.055140+0000 mon.c (mon.1) 55 : audit [INF] from='client.? 172.21.15.76:0/1465815681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:31.499 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:31 smithi125 conmon[56739]: audit 2021-03-15T22:29:31.055474+0000 mon.a (mon.0) 1710 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:31.499 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:31 smithi125 conmon[56739]: audit 2021-03-15T22:29:31.205470+0000 mon.a (mon.0) 1711 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:29:31.499 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:31 smithi125 conmon[56739]: cluster 2021-03-15T22:29:31.205539+0000 mon.a (mon.0) 1712 : cluster [DBG] osdmap e622: 8 total, 7 up, 7 in 2021-03-15T22:29:31.588 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:31.611 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:31.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:31.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:31.713 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:31.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:31.737 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:31.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:31.839 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:31.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:31.863 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:31.863 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:31.965 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:31.988 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:31.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:31.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:32.091 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:32.114 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:32.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:32.115 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:32.212 INFO:tasks.rados.rados.0.smithi076.stdout:2251: done (2 left) 2021-03-15T22:29:32.212 INFO:tasks.rados.rados.0.smithi076.stdout:2252: read oid 6 snap -1 2021-03-15T22:29:32.213 INFO:tasks.rados.rados.0.smithi076.stdout:2252: expect deleted 2021-03-15T22:29:32.213 INFO:tasks.rados.rados.0.smithi076.stdout:2253: write oid 29 current snap is 258 2021-03-15T22:29:32.213 INFO:tasks.rados.rados.0.smithi076.stdout:2253: seq_num 776 ranges {642232=661230,1942968=783923} 2021-03-15T22:29:32.217 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:32.220 INFO:tasks.rados.rados.0.smithi076.stdout:2253: writing smithi07694878-29 from 642232 to 1303462 tid 1 2021-03-15T22:29:32.229 INFO:tasks.rados.rados.0.smithi076.stdout:2253: writing smithi07694878-29 from 1942968 to 2726891 tid 2 2021-03-15T22:29:32.230 INFO:tasks.rados.rados.0.smithi076.stdout:2254: read oid 17 snap -1 2021-03-15T22:29:32.230 INFO:tasks.rados.rados.0.smithi076.stdout:2254: expect (ObjNum 772 snap 255 seq_num 772) 2021-03-15T22:29:32.230 INFO:tasks.rados.rados.0.smithi076.stdout:2255: read oid 11 snap -1 2021-03-15T22:29:32.230 INFO:tasks.rados.rados.0.smithi076.stdout:2255: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:29:32.231 INFO:tasks.rados.rados.0.smithi076.stdout:2256: read oid 22 snap -1 2021-03-15T22:29:32.231 INFO:tasks.rados.rados.0.smithi076.stdout:2256: expect (ObjNum 750 snap 245 seq_num 750) 2021-03-15T22:29:32.231 INFO:tasks.rados.rados.0.smithi076.stdout:2257: rollback oid 3 current snap is 258 2021-03-15T22:29:32.231 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 235 2021-03-15T22:29:32.231 INFO:tasks.rados.rados.0.smithi076.stdout:2249: expect (ObjNum 664 snap 209 seq_num 664) 2021-03-15T22:29:32.254 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:32.254 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:32 smithi076 conmon[56514]: cluster 2021-03-15T22:29:31.128041+0000 mgr.y ( 2021-03-15T22:29:32.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:32 smithi076 conmon[56514]: mgr.14144) 802 : cluster [DBG] pgmap v1076: 138 pgs: 7 remapped+peering, 33 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 1 active+recovering+degraded+remapped, 83 active+clean; 597 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 5.1 MiB/s wr, 29 op/s; 109/1806 objects degraded (6.035%); 104/1806 objects misplaced (5.759%); 9.6 MiB/s, 4 objects/s recovering 2021-03-15T22:29:32.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:32 smithi076 conmon[56514]: cluster 2021-03-15T22:29:32.208130+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e623: 8 total, 7 up, 7 in 2021-03-15T22:29:32.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:32.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:32.357 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:32.379 INFO:tasks.rados.rados.0.smithi076.stdout:2250: finishing copy_from to smithi07694878-40 2021-03-15T22:29:32.379 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1503 (ObjNum 745 snap 245 seq_num 745) dirty exists 2021-03-15T22:29:32.379 INFO:tasks.rados.rados.0.smithi076.stdout:2250: finishing copy_from racing read to smithi07694878-40 2021-03-15T22:29:32.380 INFO:tasks.rados.rados.0.smithi076.stdout:2253: finishing write tid 1 to smithi07694878-29 2021-03-15T22:29:32.381 INFO:tasks.rados.rados.0.smithi076.stdout:2257: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:29:32.381 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 807 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:29:32.382 INFO:tasks.rados.rados.0.smithi076.stdout:2255: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:29:32.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:32.384 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:32.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:32.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:32.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:32.510 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:32.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:32 smithi076 conmon[62156]: cluster 2021-03-15T22:29:31.128041+0000 mgr.y ( 2021-03-15T22:29:32.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:32 smithi076 conmon[62156]: mgr.14144) 802 : cluster [DBG] pgmap v1076: 138 pgs: 7 remapped+peering, 33 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 1 active+recovering+degraded+remapped, 83 active+clean; 597 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 5.1 MiB/s wr, 29 op/s; 109/1806 objects degraded (6.035%); 104/1806 objects misplaced (5.759%); 9.6 MiB/s, 4 objects/s recovering 2021-03-15T22:29:32.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:32 smithi076 conmon[62156]: cluster 2021-03-15T22:29:32.208130+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e623: 8 total, 7 up, 7 in 2021-03-15T22:29:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:32 smithi125 conmon[56739]: cluster 2021-03-15T22:29:31.128041+0000 2021-03-15T22:29:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:32 smithi125 conmon[56739]: mgr.y (mgr.14144) 802 : cluster [DBG] pgmap v1076: 138 pgs: 7 remapped+peering, 33 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 1 active+recovering+degraded+remapped, 83 active+clean; 597 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 5.1 MiB/s wr, 29 op/s; 109/1806 objects degraded (6.035%); 104/1806 objects misplaced (5.759%); 9.6 MiB/s, 4 objects/s recovering 2021-03-15T22:29:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:32 smithi125 conmon[56739]: cluster 2021-03-15T22:29:32.208130+0000 mon.a (mon.0) 1713 : cluster [DBG] osdmap e623: 8 total, 7 up, 7 in 2021-03-15T22:29:32.549 INFO:tasks.rados.rados.0.smithi076.stdout:2253: finishing write tid 2 to smithi07694878-29 2021-03-15T22:29:32.549 INFO:tasks.rados.rados.0.smithi076.stdout:2253: finishing write tid 3 to smithi07694878-29 2021-03-15T22:29:32.549 INFO:tasks.rados.rados.0.smithi076.stdout:2253: finishing write tid 4 to smithi07694878-29 2021-03-15T22:29:32.549 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 1653 (ObjNum 776 snap 258 seq_num 776) dirty exists 2021-03-15T22:29:32.550 INFO:tasks.rados.rados.0.smithi076.stdout:2253: left oid 29 (ObjNum 776 snap 258 seq_num 776) 2021-03-15T22:29:32.550 INFO:tasks.rados.rados.0.smithi076.stdout:2249: done (7 left) 2021-03-15T22:29:32.550 INFO:tasks.rados.rados.0.smithi076.stdout:2250: done (6 left) 2021-03-15T22:29:32.550 INFO:tasks.rados.rados.0.smithi076.stdout:2252: done (5 left) 2021-03-15T22:29:32.551 INFO:tasks.rados.rados.0.smithi076.stdout:2253: done (4 left) 2021-03-15T22:29:32.551 INFO:tasks.rados.rados.0.smithi076.stdout:2255: done (3 left) 2021-03-15T22:29:32.551 INFO:tasks.rados.rados.0.smithi076.stdout:2257: done (2 left) 2021-03-15T22:29:32.551 INFO:tasks.rados.rados.0.smithi076.stdout:2258: snap_create 2021-03-15T22:29:32.552 INFO:tasks.rados.rados.0.smithi076.stdout:2256: expect (ObjNum 750 snap 245 seq_num 750) 2021-03-15T22:29:32.611 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:32.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:32.635 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:32.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:32.695 INFO:tasks.rados.rados.0.smithi076.stdout:2254: expect (ObjNum 772 snap 255 seq_num 772) 2021-03-15T22:29:32.737 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:32.760 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:32.761 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:32.818 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:29:32.831 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:29:32.849 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:29:32.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:32.866 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:29:32.885 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:29:32.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:32.893 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:32.894 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:32.902 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:29:32.918 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:29:32.919 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:29:32.919 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:29:32.919 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:29:32.920 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:29:32.920 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:29:32.920 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:29:32.920 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:29:32.921 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:29:32.996 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:33.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:33.020 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:33.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:33.121 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:33.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:33.145 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:33.145 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:33.247 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:33.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[62156]: cluster 2021-03-15T22:29:32.161337+0000 osd.1 (osd.1) 120 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:29:33.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[62156]: cluster 2021-03-15T22:29:32.224248+0000 osd.6 (osd.6) 98 : cluster [DBG] 6.a scrub ok 2021-03-15T22:29:33.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[62156]: cluster 2021-03-15T22:29:32.512480+0000 osd.3 (osd.3) 110 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:29:33.262 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[62156]: audit 2021-03-15T22:29:32.796036+0000 mon.a (mon.0) 1714 : audit [DBG] from='client.? 172.21.15.76:0/1882108803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:33.263 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: cluster 2021-03-15T22:29:33.263 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 2021-03-15T22:29:33.263 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 2021 2021-03-15T22:29:33.263 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: - 2021-03-15T22:29:33.264 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 03 2021-03-15T22:29:33.264 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: - 2021-03-15T22:29:33.264 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 15 2021-03-15T22:29:33.264 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: T 2021-03-15T22:29:33.265 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 22 2021-03-15T22:29:33.265 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: : 2021-03-15T22:29:33.265 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 29 2021-03-15T22:29:33.265 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: : 2021-03-15T22:29:33.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 32 2021-03-15T22:29:33.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: . 2021-03-15T22:29:33.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 161337 2021-03-15T22:29:33.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: +0000 2021-03-15T22:29:33.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 2021-03-15T22:29:33.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: osd.1 2021-03-15T22:29:33.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: (osd.1) 120 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:29:33.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 2021-03-15T22:29:33.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: cluster 2021-03-15T22:29: 2021-03-15T22:29:33.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 32.224248+0000 osd.6 (osd.6) 98 : cluster 2021-03-15T22:29:33.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: [DBG] 6.a scrub ok 2021-03-15T22:29:33.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:29:33.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 22:29:32.512480+0000 osd.3 (osd. 2021-03-15T22:29:33.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 3) 110 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:29:33.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: audit 2021-03-15T22:29:33.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 2021-03-15T 2021-03-15T22:29:33.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: 22:29:32.796036+0000 mon.a (mon 2021-03-15T22:29:33.270 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:33 smithi076 conmon[56514]: .0) 1714 : audit [DBG] from='client.? 172.21.15.76:0/1882108803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:33.272 INFO:tasks.rados.rados.0.smithi076.stdout:2254: done (2 left) 2021-03-15T22:29:33.272 INFO:tasks.rados.rados.0.smithi076.stdout:2256: done (1 left) 2021-03-15T22:29:33.273 INFO:tasks.rados.rados.0.smithi076.stdout:2258: done (0 left) 2021-03-15T22:29:33.273 INFO:tasks.rados.rados.0.smithi076.stdout:2259: snap_remove snap 249 2021-03-15T22:29:33.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:33.274 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:33.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:33.375 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:33.387 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:29:33.388 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:29:33.398 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:33.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:33.399 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:33.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:33 smithi125 conmon[56739]: cluster 2021-03-15T22:29:32.161337+0000 osd.1 (osd.1) 120 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:29:33.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:33 smithi125 conmon[56739]: cluster 2021-03-15T22:29:32.224248+0000 osd.6 (osd.6) 98 : cluster [DBG] 6.a scrub ok 2021-03-15T22:29:33.465 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:33 smithi125 conmon[56739]: cluster 2021-03-15T22:29:32.512480+0000 osd.3 (osd.3) 110 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:29:33.466 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:33 smithi125 conmon[56739]: audit 2021-03-15T22:29:32.796036+0000 mon.a (mon.0) 1714 : audit [DBG] from='client.? 172.21.15.76:0/1882108803' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:33.501 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:33.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:33.525 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:33.526 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:33.627 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:33.651 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:33.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:33.653 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:33.664 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:29:33.665 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:29:33.754 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:33.777 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:33.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:33.779 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:33.880 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:33.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:33.904 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:33.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:34.005 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:34.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:34.028 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:34.028 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:34.051 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-15T22:29:34.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:34.156 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:34.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:34.157 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:34.258 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:34.272 INFO:tasks.rados.rados.0.smithi076.stdout:2259: done (0 left) 2021-03-15T22:29:34.273 INFO:tasks.rados.rados.0.smithi076.stdout:2260: snap_remove snap 235 2021-03-15T22:29:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:34 smithi125 conmon[56739]: cluster 2021-03-15T22:29:33.128671+0000 mgr.y (mgr.14144) 803 : cluster [DBG] pgmap v1079: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 553 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 4.9 MiB/s wr, 26 op/s; 52/1749 objects degraded (2.973%); 189/1749 objects misplaced (10.806%); 5.8 MiB/s, 5 objects/s recovering 2021-03-15T22:29:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:34 smithi125 conmon[56739]: cluster 2021-03-15T22:29:33.191776+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:29:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:34 smithi125 conmon[56739]: cluster 2021-03-15T22:29:33.255239+0000 osd.6 (osd.6) 99 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:29:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:34 smithi125 conmon[56739]: cluster 2021-03-15T22:29:33.261279+0000 mon.a (mon.0) 1715 : cluster [DBG] osdmap e624: 8 total, 7 up, 7 in 2021-03-15T22:29:34.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:34 smithi125 conmon[56739]: cluster 2021-03-15T22:29:33.490622+0000 osd.3 (osd.3) 111 : cluster [DBG] 7.a scrub ok 2021-03-15T22:29:34.289 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:34.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:34.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:34.382 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:34.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:34.415 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:34.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:34.416 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:34.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[62156]: cluster 2021-03-15T22:29:33.128671+0000 mgr.y (mgr.14144) 803 2021-03-15T22:29:34.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[62156]: : cluster [DBG] pgmap v1079: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 553 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 4.9 MiB/s wr, 26 op/s; 52/1749 objects degraded (2.973%); 189/1749 objects misplaced (10.806%); 5.8 MiB/s, 5 objects/s recovering 2021-03-15T22:29:34.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[62156]: cluster 2021-03-15T22:29:33.191776+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:29:34.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[62156]: cluster 2021-03-15T22:29:33.255239+0000 osd.6 (osd.6) 99 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:29:34.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[62156]: cluster 2021-03-15T22:29: 2021-03-15T22:29:34.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[62156]: 33.261279+0000 mon.a (mon.0) 1715 : cluster [DBG] osdmap e624: 8 total, 7 up, 7 in 2021-03-15T22:29:34.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:29:34.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[62156]: :29:33.490622+0000 osd.3 (osd.3) 111 : cluster [DBG] 2021-03-15T22:29:34.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[62156]: 7.a scrub ok 2021-03-15T22:29:34.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[56514]: cluster 2021-03-15T22:29:33 2021-03-15T22:29:34.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[56514]: .128671+0000 mgr.y (mgr.14144) 803 : cluster [DBG] pgmap v1079: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 553 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 4.9 MiB/s wr, 26 op/s; 52/1749 objects degraded (2.973%); 189/1749 objects misplaced (10.806%); 5.8 MiB/s, 5 objects/s recovering 2021-03-15T22:29:34.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[56514]: cluster 2021-03-15T22:29:33.191776+0000 osd.1 (osd.1) 121 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:29:34.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[56514]: cluster 2021-03-15T22:29:34.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[56514]: 2021-03-15T22:29:33.255239+0000 osd.6 (osd.6) 99 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:29:34.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:29:34.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[56514]: -03-15T22:29:33.261279+0000 mon.a (mon.0) 1715 : cluster [DBG] osdmap e624: 8 total, 7 up, 7 in 2021-03-15T22:29:34.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[56514]: 2021-03-15T22:29:34.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[56514]: cluster 2021-03-15T22:29:33. 2021-03-15T22:29:34.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[56514]: 490622+0000 osd.3 (osd.3) 111 : cluster [DBG] 2021-03-15T22:29:34.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:34 smithi076 conmon[56514]: 7.a scrub ok 2021-03-15T22:29:34.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:34.537 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:29:34.545 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:34.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:34.547 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:34.647 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:34.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:34.671 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:34.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:34.774 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:34.796 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:34.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:34.798 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:34.814 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:34.899 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:34.923 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:34.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:34.924 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:35.025 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:35.049 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:35.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:35.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:35.151 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:35.173 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:35.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:35.175 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:35.273 INFO:tasks.rados.rados.0.smithi076.stdout:2260: done (0 left) 2021-03-15T22:29:35.274 INFO:tasks.rados.rados.0.smithi076.stdout:2261: write (excl) oid 41 current snap is 259 2021-03-15T22:29:35.274 INFO:tasks.rados.rados.0.smithi076.stdout:2261: seq_num 777 ranges {659039=736479,2001450=643934} 2021-03-15T22:29:35.275 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:35.282 INFO:tasks.rados.rados.0.smithi076.stdout:2261: writing smithi07694878-41 from 659039 to 1395518 tid 1 2021-03-15T22:29:35.295 INFO:tasks.rados.rados.0.smithi076.stdout:2261: writing smithi07694878-41 from 2001450 to 2645384 tid 2 2021-03-15T22:29:35.296 INFO:tasks.rados.rados.0.smithi076.stdout:2262: read oid 31 snap -1 2021-03-15T22:29:35.296 INFO:tasks.rados.rados.0.smithi076.stdout:2262: expect (ObjNum 709 snap 227 seq_num 709) 2021-03-15T22:29:35.296 INFO:tasks.rados.rados.0.smithi076.stdout:2263: write oid 24 current snap is 259 2021-03-15T22:29:35.297 INFO:tasks.rados.rados.0.smithi076.stdout:2261: finishing write tid 1 to smithi07694878-41 2021-03-15T22:29:35.297 INFO:tasks.rados.rados.0.smithi076.stdout:2263: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2021-03-15T22:29:35.303 INFO:tasks.rados.rados.0.smithi076.stdout:2263: writing smithi07694878-24 from 675846 to 1087574 tid 1 2021-03-15T22:29:35.307 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:35.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:35.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:35.310 INFO:tasks.rados.rados.0.smithi076.stdout:2263: writing smithi07694878-24 from 1543579 to 2331171 tid 2 2021-03-15T22:29:35.315 INFO:tasks.rados.rados.0.smithi076.stdout:2263: writing smithi07694878-24 from 2848853 to 3075846 tid 3 2021-03-15T22:29:35.315 INFO:tasks.rados.rados.0.smithi076.stdout:2264: read oid 13 snap -1 2021-03-15T22:29:35.315 INFO:tasks.rados.rados.0.smithi076.stdout:2264: expect (ObjNum 745 snap 245 seq_num 745) 2021-03-15T22:29:35.315 INFO:tasks.rados.rados.0.smithi076.stdout:2265: read oid 37 snap 202 2021-03-15T22:29:35.316 INFO:tasks.rados.rados.0.smithi076.stdout:2265: expect (ObjNum 536 snap 170 seq_num 536) 2021-03-15T22:29:35.316 INFO:tasks.rados.rados.0.smithi076.stdout:2266: snap_remove snap 216 2021-03-15T22:29:35.410 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:35.432 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:35.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:35.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:35.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:35 smithi125 conmon[56739]: cluster 2021-03-15T22:29:34.234638+0000 osd.1 (osd.1) 2021-03-15T22:29:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:35 smithi125 conmon[56739]: 122 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:29:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:35 smithi125 conmon[56739]: cluster 2021-03-15T22:29:34.269695+0000 mon.a (mon.0) 1716 : cluster [DBG] osdmap e625: 8 total, 7 up, 7 in 2021-03-15T22:29:35.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:35 smithi125 conmon[56739]: cluster 2021-03-15T22:29:35.186199+0000 mon.a (mon.0) 1717 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:35.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:35 smithi125 conmon[56739]: cluster 2021-03-15T22:29:35.186225+0000 mon.a (mon.0) 1718 : cluster [WRN] Health check update: Degraded data redundancy: 52/1749 objects degraded (2.973%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:35.559 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:35.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:35.661 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:35.667 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:29:35.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:35.686 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:35.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:35.706 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[56514]: cluster 2021-03-15T22:29:34.234638+0000 osd.1 ( 2021-03-15T22:29:35.706 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[56514]: osd.1) 122 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:29:35.706 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[56514]: cluster 2021-03-15T22:29:34.269695+0000 mon.a (mon.0) 1716 : cluster [DBG] osdmap e625: 8 total, 7 up, 7 in 2021-03-15T22:29:35.707 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[56514]: cluster 2021-03-15T22:29:35.186199+0000 mon.a (mon.0) 1717 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:35.707 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[56514]: cluster 2021-03-15T22:29:35.186225+0000 mon.a (mon.0) 1718 : cluster [WRN] Health check update: Degraded data redundancy: 52/1749 objects degraded (2.973%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:35.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: cluster 2021-03-15T22:29:34. 2021-03-15T22:29:35.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: 234638+0000 osd.1 (osd.1) 122 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:29:35.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:29:35.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: 15T22:29:34.269695+0000 mon.a (mon.0) 1716 : cluster [DBG] osdmap e625: 8 total, 7 up, 7 in 2021-03-15T22:29:35.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: 2021-03-15T22:29:35.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:29:35.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: T22:29:35.186199+0000 2021-03-15T22:29:35.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: mon.a (mon.0) 1717 2021-03-15T22:29:35.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:35.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:29:35.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: 15T22:29:35.186225+0000 mon.a (mon.0 2021-03-15T22:29:35.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:35 smithi076 conmon[62156]: ) 1718 : cluster [WRN] Health check update: Degraded data redundancy: 52/1749 objects degraded (2.973%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:35.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:35.812 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:35.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:35.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:35.915 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:35.935 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:29:35.936 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:29:35.939 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:35.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:35.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:35.966 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:36.041 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:36.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:36.065 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:36.066 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:36.167 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:36.192 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:36.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:36.193 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:36.279 INFO:tasks.rados.rados.0.smithi076.stdout:2266: done (5 left) 2021-03-15T22:29:36.280 INFO:tasks.rados.rados.0.smithi076.stdout:2267: snap_remove snap 254 2021-03-15T22:29:36.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:36.317 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:36.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:36.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:36.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[56514]: cluster 2021-03-15T22:29:34.939138+0000 osd.4 (osd.4) 132 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:29:36.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[56514]: cluster 2021-03-15T22:29:35.129156+0000 mgr.y (mgr.14144) 804 : cluster [DBG] pgmap v1082: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 553 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 8 op/s; 53/1752 objects degraded (3.025%); 189/1752 objects misplaced (10.788%); 5.0 MiB/s, 4 objects/s recovering 2021-03-15T22:29:36.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[56514]: cluster 2021-03-15T22:29:35.243839+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.d scrub ok 2021-03-15T22:29:36.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[56514]: cluster 2021-03-15T22:29:35.270722+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e626: 8 total, 7 up, 7 in 2021-03-15T22:29:36.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[56514]: cluster 2021-03-15T22:29:35.314959+0000 osd.6 (osd.6) 100 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:29:36.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[56514]: cluster 2021-03-15T22:29:35.496454+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.11 scrub ok 2021-03-15T22:29:36.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[56514]: audit 2021-03-15T22:29:36.074995+0000 mon.a (mon.0) 1720 : 2021-03-15T22:29:36.402 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[56514]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:36.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[62156]: cluster 2021-03-15T22:29:34.939138 2021-03-15T22:29:36.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 132 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:29:36.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[62156]: cluster 2021-03-15T22:29:35.129156+0000 mgr.y (mgr.14144) 804 : cluster [DBG] pgmap v1082: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 553 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 8 op/s; 53/1752 objects degraded (3.025%); 189/1752 objects misplaced (10.788%); 5.0 MiB/s, 4 objects/s recovering 2021-03-15T22:29:36.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[62156]: cluster 2021-03-15T22:29:35.243839+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.d scrub ok 2021-03-15T22:29:36.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[62156]: cluster 2021-03-15T22:29:35.270722+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e626: 8 total, 7 up, 7 in 2021-03-15T22:29:36.404 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[62156]: cluster 2021-03-15T22:29:35.314959+0000 osd.6 (osd.6) 100 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:29:36.404 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[62156]: cluster 2021-03-15T22:29:35.496454+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.11 scrub ok 2021-03-15T22:29:36.404 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:36 smithi076 conmon[62156]: audit 2021-03-15T22:29:36.074995+0000 mon.a (mon.0) 1720 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:36.419 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:36.442 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:36.442 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:36.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:36 smithi125 conmon[56739]: cluster 2021-03-15T22:29:34.939138+0000 osd.4 (osd.4) 132 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:29:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:36 smithi125 conmon[56739]: cluster 2021-03-15T22:29:35.129156+0000 mgr.y (mgr.14144) 804 : cluster [DBG] pgmap v1082: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 553 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 4.5 MiB/s rd, 1.2 MiB/s wr, 8 op/s; 53/1752 objects degraded (3.025%); 189/1752 objects misplaced (10.788%); 5.0 MiB/s, 4 objects/s recovering 2021-03-15T22:29:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:36 smithi125 conmon[56739]: cluster 2021-03-15T22:29:35.243839+0000 osd.1 (osd.1) 123 : cluster [DBG] 6.d scrub ok 2021-03-15T22:29:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:36 smithi125 conmon[56739]: cluster 2021-03-15T22:29:35.270722+0000 mon.a (mon.0) 1719 : cluster [DBG] osdmap e626: 8 total, 7 up, 7 in 2021-03-15T22:29:36.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:36 smithi125 conmon[56739]: cluster 2021-03-15T22:29:35.314959+0000 osd.6 (osd.6) 100 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:29:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:36 smithi125 conmon[56739]: cluster 2021-03-15T22:29:35.496454+0000 osd.3 (osd.3) 112 : cluster [DBG] 6.11 scrub ok 2021-03-15T22:29:36.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:36 smithi125 conmon[56739]: audit 2021-03-15T22:29:36.074995+0000 mon.a (mon.0) 1720 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:36.543 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:36.559 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-15T22:29:36.567 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:36.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:36.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:36.669 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:36.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:36.693 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:36.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:36.795 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:36.819 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:36.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:36.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:36.833 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:36.922 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:36.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:36.946 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:36.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:37.049 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:37.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:37.074 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:37.074 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:37.176 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:37.199 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:37.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:37.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:37.266 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 0, 3] out_osds: [5] dead_osds: [2] live_osds: [0, 1, 3, 4, 6, 7, 5] 2021-03-15T22:29:37.267 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:29:37.267 INFO:tasks.thrashosds.thrasher:inject_pause on 5 2021-03-15T22:29:37.267 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-03-15T22:29:37.268 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-03-15T22:29:37.268 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 config set filestore_inject_stall 3 2021-03-15T22:29:37.287 INFO:tasks.rados.rados.0.smithi076.stdout:2267: done (5 left) 2021-03-15T22:29:37.288 INFO:tasks.rados.rados.0.smithi076.stdout:2268: read oid 47 snap 211 2021-03-15T22:29:37.288 INFO:tasks.rados.rados.0.smithi076.stdout:2268: expect deleted 2021-03-15T22:29:37.288 INFO:tasks.rados.rados.0.smithi076.stdout:2269: snap_create 2021-03-15T22:29:37.288 INFO:tasks.rados.rados.0.smithi076.stdout:2261: finishing write tid 2 to smithi07694878-41 2021-03-15T22:29:37.288 INFO:tasks.rados.rados.0.smithi076.stdout:2261: finishing write tid 3 to smithi07694878-41 2021-03-15T22:29:37.289 INFO:tasks.rados.rados.0.smithi076.stdout:2261: finishing write tid 4 to smithi07694878-41 2021-03-15T22:29:37.289 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 1203 (ObjNum 777 snap 259 seq_num 777) dirty exists 2021-03-15T22:29:37.289 INFO:tasks.rados.rados.0.smithi076.stdout:2261: left oid 41 (ObjNum 777 snap 259 seq_num 777) 2021-03-15T22:29:37.289 INFO:tasks.rados.rados.0.smithi076.stdout:2262: expect (ObjNum 709 snap 227 seq_num 709) 2021-03-15T22:29:37.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:37.303 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:29:37.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:37.333 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:37.334 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:37.422 INFO:tasks.rados.rados.0.smithi076.stdout:2263: finishing write tid 1 to smithi07694878-24 2021-03-15T22:29:37.423 INFO:tasks.rados.rados.0.smithi076.stdout:2263: finishing write tid 2 to smithi07694878-24 2021-03-15T22:29:37.423 INFO:tasks.rados.rados.0.smithi076.stdout:2263: finishing write tid 3 to smithi07694878-24 2021-03-15T22:29:37.423 INFO:tasks.rados.rados.0.smithi076.stdout:2263: finishing write tid 4 to smithi07694878-24 2021-03-15T22:29:37.423 INFO:tasks.rados.rados.0.smithi076.stdout:2263: finishing write tid 5 to smithi07694878-24 2021-03-15T22:29:37.424 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 615 (ObjNum 778 snap 259 seq_num 778) dirty exists 2021-03-15T22:29:37.424 INFO:tasks.rados.rados.0.smithi076.stdout:2263: left oid 24 (ObjNum 778 snap 259 seq_num 778) 2021-03-15T22:29:37.424 INFO:tasks.rados.rados.0.smithi076.stdout:2264: expect (ObjNum 745 snap 245 seq_num 745) 2021-03-15T22:29:37.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:37.457 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:37.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:37.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:37.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:37 smithi125 conmon[56739]: cluster 2021-03-15T22:29:36.271636 2021-03-15T22:29:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:37 smithi125 conmon[56739]: +0000 osd.6 (osd.6) 101 : cluster [DBG] 7.c scrub ok 2021-03-15T22:29:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:37 smithi125 conmon[56739]: cluster 2021-03-15T22:29:36.276020+0000 mon.a (mon.0) 1721 : cluster [DBG] osdmap e627: 8 total, 7 up, 7 in 2021-03-15T22:29:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:37 smithi125 conmon[56739]: audit 2021-03-15T22:29:36.577099+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? 172.21.15.76:0/2375764867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:37.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:37 smithi125 conmon[56739]: cluster 2021-03-15T22:29:36.632674+0000 osd.6 (osd.6) 102 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:37.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:37 smithi125 conmon[56739]: cluster 2021-03-15T22:29:36.633568+0000 osd.6 (osd.6) 103 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:37.559 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:37.560 INFO:tasks.rados.rados.0.smithi076.stdout:2265: expect (ObjNum 536 snap 170 seq_num 536) 2021-03-15T22:29:37.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:37.582 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:37.583 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:37.684 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:37.707 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:37.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:37.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[56514]: cluster 2021-03-15T22:29:36.271636+0000 osd.6 ( 2021-03-15T22:29:37.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[56514]: osd.6) 101 : cluster [DBG] 7.c scrub ok 2021-03-15T22:29:37.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[56514]: cluster 2021-03-15T22:29:36.276020+0000 mon.a (mon.0) 1721 : cluster [DBG] osdmap e627: 8 total, 7 up, 7 in 2021-03-15T22:29:37.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[56514]: audit 2021-03-15T22:29:36.577099+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? 172.21.15.76:0/2375764867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:37.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[56514]: cluster 2021-03-15T22:29:36.632674+0000 osd.6 (osd.6) 102 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:37.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[56514]: cluster 2021-03-15T22:29:36.633568+0000 osd.6 (osd.6) 103 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:37.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: cluster 2021-03-15T22:29:37.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: 2021-03-15T22:29:37.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: 2021 2021-03-15T22:29:37.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: - 2021-03-15T22:29:37.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: 03 2021-03-15T22:29:37.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: - 2021-03-15T22:29:37.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: 15 2021-03-15T22:29:37.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: T 2021-03-15T22:29:37.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: 22 2021-03-15T22:29:37.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: : 2021-03-15T22:29:37.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: 29 2021-03-15T22:29:37.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: : 2021-03-15T22:29:37.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: 36 2021-03-15T22:29:37.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: . 2021-03-15T22:29:37.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: 271636 2021-03-15T22:29:37.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: +0000 2021-03-15T22:29:37.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: 2021-03-15T22:29:37.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: osd.6 2021-03-15T22:29:37.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: ( 2021-03-15T22:29:37.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: osd 2021-03-15T22:29:37.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: . 2021-03-15T22:29:37.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: 6) 101 : cluster [DBG] 7.c scrub ok 2021-03-15T22:29:37.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: cluster 2021-03-15T22:29:36.276020+0000 mon.a (mon.0) 1721 : cluster 2021-03-15T22:29:37.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: [DBG] osdmap e627: 8 total, 7 up, 7 in 2021-03-15T22:29:37.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:29:37.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: 22:29:36.577099+0000 mon.a (mon.0) 1722 : audit [INF] from='client.? 172.21.15.76:0/2375764867' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:37.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: cluster 2021-03-15T22:29:36 2021-03-15T22:29:37.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: .632674+0000 osd.6 (osd.6) 102 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:37.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: cluster 2021-03-15T22:29:36.633568 2021-03-15T22:29:37.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:37 smithi076 conmon[62156]: +0000 osd.6 (osd.6) 103 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:37.809 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:37.832 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:37.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:37.835 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:37.935 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:37.957 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:37.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:37.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:38.001 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:38.061 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:38.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:38.085 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:38.086 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:38.175 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:29:38.187 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:38.191 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:29:38.209 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:29:38.210 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:38.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:38.211 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:38.227 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:29:38.244 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:29:38.261 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:29:38.278 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:29:38.279 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:29:38.279 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:29:38.279 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:29:38.280 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:29:38.280 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:29:38.280 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:29:38.280 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:29:38.281 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:29:38.295 INFO:tasks.rados.rados.0.smithi076.stdout:2261: done (6 left) 2021-03-15T22:29:38.295 INFO:tasks.rados.rados.0.smithi076.stdout:2262: done (5 left) 2021-03-15T22:29:38.296 INFO:tasks.rados.rados.0.smithi076.stdout:2263: done (4 left) 2021-03-15T22:29:38.296 INFO:tasks.rados.rados.0.smithi076.stdout:2264: done (3 left) 2021-03-15T22:29:38.296 INFO:tasks.rados.rados.0.smithi076.stdout:2265: done (2 left) 2021-03-15T22:29:38.296 INFO:tasks.rados.rados.0.smithi076.stdout:2268: done (1 left) 2021-03-15T22:29:38.297 INFO:tasks.rados.rados.0.smithi076.stdout:2269: done (0 left) 2021-03-15T22:29:38.297 INFO:tasks.rados.rados.0.smithi076.stdout:2270: read oid 40 snap 196 2021-03-15T22:29:38.297 INFO:tasks.rados.rados.0.smithi076.stdout:2270: expect deleted 2021-03-15T22:29:38.297 INFO:tasks.rados.rados.0.smithi076.stdout:2271: read oid 6 snap -1 2021-03-15T22:29:38.298 INFO:tasks.rados.rados.0.smithi076.stdout:2271: expect deleted 2021-03-15T22:29:38.298 INFO:tasks.rados.rados.0.smithi076.stdout:2272: delete oid 46 current snap is 260 2021-03-15T22:29:38.299 INFO:tasks.rados.rados.0.smithi076.stdout:2270: done (2 left) 2021-03-15T22:29:38.299 INFO:tasks.rados.rados.0.smithi076.stdout:2271: done (1 left) 2021-03-15T22:29:38.299 INFO:tasks.rados.rados.0.smithi076.stdout:2272: done (0 left) 2021-03-15T22:29:38.300 INFO:tasks.rados.rados.0.smithi076.stdout:2273: delete oid 16 current snap is 260 2021-03-15T22:29:38.301 INFO:tasks.rados.rados.0.smithi076.stdout:2273: done (0 left) 2021-03-15T22:29:38.302 INFO:tasks.rados.rados.0.smithi076.stdout:2274: read oid 11 snap -1 2021-03-15T22:29:38.302 INFO:tasks.rados.rados.0.smithi076.stdout:2274: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:29:38.302 INFO:tasks.rados.rados.0.smithi076.stdout:2275: read oid 38 snap -1 2021-03-15T22:29:38.303 INFO:tasks.rados.rados.0.smithi076.stdout:2275: expect (ObjNum 732 snap 239 seq_num 732) 2021-03-15T22:29:38.303 INFO:tasks.rados.rados.0.smithi076.stdout:2276: delete oid 6 current snap is 260 2021-03-15T22:29:38.304 INFO:tasks.rados.rados.0.smithi076.stdout:2274: expect (ObjNum 543 snap 172 seq_num 543) 2021-03-15T22:29:38.313 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:38.337 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:38.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:38.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:38.440 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:38.459 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:29:38.460 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:29:38.460 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:29:38.460 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:29:38.462 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:38.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:38.464 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:38.473 INFO:tasks.rados.rados.0.smithi076.stdout:2274: done (2 left) 2021-03-15T22:29:38.473 INFO:tasks.rados.rados.0.smithi076.stdout:2276: done (1 left) 2021-03-15T22:29:38.474 INFO:tasks.rados.rados.0.smithi076.stdout:2277: rollback oid 47 current snap is 260 2021-03-15T22:29:38.474 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 47 to 239 2021-03-15T22:29:38.474 INFO:tasks.rados.rados.0.smithi076.stdout:2275: expect (ObjNum 732 snap 239 seq_num 732) 2021-03-15T22:29:38.521 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:38 smithi125 conmon[56739]: cluster 2021-03-15T22:29:36.937860+0000 osd.4 (osd.4) 133 : cluster [DBG] 5.e scrub ok 2021-03-15T22:29:38.521 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:38 smithi125 conmon[56739]: cluster 2021-03-15T22:29:37.129908+0000 mgr.y (mgr.14144) 805 : cluster [DBG] pgmap v1085: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 549 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 686 KiB/s wr, 13 op/s; 53/1746 objects degraded (3.036%); 188/1746 objects misplaced (10.767%) 2021-03-15T22:29:38.521 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:38 smithi125 conmon[56739]: cluster 2021-03-15T22:29:37.172393+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:29:38.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:38 smithi125 conmon[56739]: cluster 2021-03-15T22:29:37.274385+0000 mon.a (mon.0) 1723 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:29:38.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:38 smithi125 conmon[56739]: audit 2021-03-15T22:29:37.282108+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? 172.21.15.76:0/2375764867' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:38.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:38 smithi125 conmon[56739]: cluster 2021-03-15T22:29:37.282177+0000 mon.a (mon.0) 1725 : cluster [DBG] osdmap e628: 8 total, 7 up, 7 in 2021-03-15T22:29:38.522 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:38 smithi125 conmon[56739]: cluster 2021-03-15T22:29:37.724909+0000 osd.0 (osd.0) 95 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:29:38.523 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:38 smithi125 conmon[56739]: 2021-03-15T22:29:38.523 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:38 smithi125 conmon[56739]: audit 2021-03-15T22:29:38.156222+0000 mon.a (mon.0) 1726 : audit [DBG] from='client.? 172.21.15.76:0/695507057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:38.564 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:38.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:38.590 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:38.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:38.623 INFO:tasks.rados.rados.0.smithi076.stdout:2277: finishing rollback tid 0 to smithi07694878-47 2021-03-15T22:29:38.624 INFO:tasks.rados.rados.0.smithi076.stdout:2277: finishing rollback tid 1 to smithi07694878-47 2021-03-15T22:29:38.624 INFO:tasks.rados.rados.0.smithi076.stdout:2277: finishing rollback tid 2 to smithi07694878-47 2021-03-15T22:29:38.624 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1204 (ObjNum 669 snap 212 seq_num 669) dirty exists 2021-03-15T22:29:38.625 INFO:tasks.rados.rados.0.smithi076.stdout:2275: done (1 left) 2021-03-15T22:29:38.625 INFO:tasks.rados.rados.0.smithi076.stdout:2277: done (0 left) 2021-03-15T22:29:38.625 INFO:tasks.rados.rados.0.smithi076.stdout:2278: write (excl) oid 21 current snap is 260 2021-03-15T22:29:38.625 INFO:tasks.rados.rados.0.smithi076.stdout:2278: seq_num 782 ranges {743074=712724,2061154=794930} 2021-03-15T22:29:38.631 INFO:tasks.rados.rados.0.smithi076.stdout:2278: writing smithi07694878-21 from 743074 to 1455798 tid 1 2021-03-15T22:29:38.638 INFO:tasks.rados.rados.0.smithi076.stdout:2278: writing smithi07694878-21 from 2061154 to 2856084 tid 2 2021-03-15T22:29:38.639 INFO:tasks.rados.rados.0.smithi076.stdout:2279: snap_remove snap 185 2021-03-15T22:29:38.692 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:38.713 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:38.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:38.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:38.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[56514]: cluster 2021-03-15T22:29:36.937860+0000 osd.4 (osd 2021-03-15T22:29:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[56514]: .4) 133 : cluster [DBG] 5.e scrub ok 2021-03-15T22:29:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[56514]: cluster 2021-03-15T22:29:37.129908+0000 mgr.y (mgr.14144) 805 : cluster [DBG] pgmap v1085: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 549 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 686 KiB/s wr, 13 op/s; 53/1746 objects degraded (3.036%); 188/1746 objects misplaced (10.767%) 2021-03-15T22:29:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[56514]: cluster 2021-03-15T22:29:37.172393+0000 osd.1 (osd.1) 124 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:29:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[56514]: cluster 2021-03-15T22:29:37.274385+0000 mon.a (mon.0) 1723 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:29:38.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[56514]: audit 2021-03-15T22:29:37.282108+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? 172.21.15.76:0/2375764867' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:38.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[56514]: cluster 2021-03-15T22:29:37.282177+0000 mon.a (mon.0) 1725 : cluster [DBG] osdmap e628: 8 total, 7 up, 7 in 2021-03-15T22:29:38.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[56514]: cluster 2021-03-15T22:29:37.724909+0000 osd.0 (osd.0) 95 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:29:38.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[56514]: audit 2021-03-15T22:29:38.156222+0000 mon.a (mon.0) 1726 : audit [DBG] from='client.? 172.21.15.76:0/695507057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:38.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: cluster 2021-03-15T22:29:36.937860+0000 osd.4 2021-03-15T22:29:38.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: (osd.4) 133 : cluster [DBG] 5.e scrub ok 2021-03-15T22:29:38.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: cluster 2021-03-15T22:29:37.129908+0000 mgr.y (mgr.14144) 805 : cluster [DBG] pgmap v1085: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 549 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 686 KiB/s wr, 13 op/s; 53/1746 objects degraded (3.036%); 188/1746 objects misplaced (10.767%) 2021-03-15T22:29:38.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: cluster 2021-03-15T22:29:37.172393+0000 osd.1 (osd. 2021-03-15T22:29:38.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: 1) 124 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:29:38.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: cluster 2021-03-15T22:29:38.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:29:38.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: :29:37.274385+0000 mon.a (mon. 2021-03-15T22:29:38.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: 0) 1723 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:29:38.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: audit 2021-03-15T22:29:38.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: 2021-03-15T22:29:37. 2021-03-15T22:29:38.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: 282108+0000 mon.a (mon.0) 1724 : audit [INF] from='client.? 172.21.15.76:0/2375764867' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:38.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:29:38.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: T22:29:37.282177+0000 mon.a ( 2021-03-15T22:29:38.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: mon.0) 1725 : cluster [DBG] osdmap e628: 8 total, 7 up, 7 in 2021-03-15T22:29:38.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: 2021-03-15T22:29:38.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:38.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: 03-15T22:29: 2021-03-15T22:29:38.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: 37.724909+0000 osd.0 (osd 2021-03-15T22:29:38.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: .0) 95 : cluster [DBG] 2021-03-15T22:29:38.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: 6.0 scrub ok 2021-03-15T22:29:38.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:29:38.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: -15T22:29:38.156222+0000 2021-03-15T22:29:38.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: mon.a (mon.0) 1726 : audit [DBG] 2021-03-15T22:29:38.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:38 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/695507057' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:38.817 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:38.839 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:38.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:38.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:38.896 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:29:38.897 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:29:38.943 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:38.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:38.966 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:38.967 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:39.068 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:39.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:39.092 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:39.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:39.193 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:39.215 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:39.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:39.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:39.266 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-15T22:29:39.299 INFO:tasks.rados.rados.0.smithi076.stdout:2279: done (1 left) 2021-03-15T22:29:39.299 INFO:tasks.rados.rados.0.smithi076.stdout:2280: read oid 47 snap -1 2021-03-15T22:29:39.299 INFO:tasks.rados.rados.0.smithi076.stdout:2280: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:29:39.300 INFO:tasks.rados.rados.0.smithi076.stdout:2281: write oid 25 current snap is 260 2021-03-15T22:29:39.300 INFO:tasks.rados.rados.0.smithi076.stdout:2281: seq_num 783 ranges {759881=787973,2119636=654941} 2021-03-15T22:29:39.308 INFO:tasks.rados.rados.0.smithi076.stdout:2281: writing smithi07694878-25 from 759881 to 1547854 tid 1 2021-03-15T22:29:39.316 INFO:tasks.rados.rados.0.smithi076.stdout:2281: writing smithi07694878-25 from 2119636 to 2774577 tid 2 2021-03-15T22:29:39.317 INFO:tasks.rados.rados.0.smithi076.stdout:2282: read oid 28 snap -1 2021-03-15T22:29:39.318 INFO:tasks.rados.rados.0.smithi076.stdout:2282: expect (ObjNum 609 snap 189 seq_num 609) 2021-03-15T22:29:39.318 INFO:tasks.rados.rados.0.smithi076.stdout:2283: snap_create 2021-03-15T22:29:39.319 INFO:tasks.rados.rados.0.smithi076.stdout:2278: finishing write tid 1 to smithi07694878-21 2021-03-15T22:29:39.319 INFO:tasks.rados.rados.0.smithi076.stdout:2278: finishing write tid 2 to smithi07694878-21 2021-03-15T22:29:39.319 INFO:tasks.rados.rados.0.smithi076.stdout:2278: finishing write tid 3 to smithi07694878-21 2021-03-15T22:29:39.319 INFO:tasks.rados.rados.0.smithi076.stdout:2278: finishing write tid 4 to smithi07694878-21 2021-03-15T22:29:39.320 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 1703 (ObjNum 782 snap 260 seq_num 782) dirty exists 2021-03-15T22:29:39.320 INFO:tasks.rados.rados.0.smithi076.stdout:2278: left oid 21 (ObjNum 782 snap 260 seq_num 782) 2021-03-15T22:29:39.320 INFO:tasks.rados.rados.0.smithi076.stdout:2280: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:29:39.321 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:39.344 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:39.345 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:39.417 INFO:tasks.rados.rados.0.smithi076.stdout:2281: finishing write tid 1 to smithi07694878-25 2021-03-15T22:29:39.417 INFO:tasks.rados.rados.0.smithi076.stdout:2281: finishing write tid 2 to smithi07694878-25 2021-03-15T22:29:39.417 INFO:tasks.rados.rados.0.smithi076.stdout:2281: finishing write tid 3 to smithi07694878-25 2021-03-15T22:29:39.417 INFO:tasks.rados.rados.0.smithi076.stdout:2281: finishing write tid 4 to smithi07694878-25 2021-03-15T22:29:39.418 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 1320 (ObjNum 783 snap 260 seq_num 783) dirty exists 2021-03-15T22:29:39.418 INFO:tasks.rados.rados.0.smithi076.stdout:2281: left oid 25 (ObjNum 783 snap 260 seq_num 783) 2021-03-15T22:29:39.418 INFO:tasks.rados.rados.0.smithi076.stdout:2282: expect (ObjNum 609 snap 189 seq_num 609) 2021-03-15T22:29:39.446 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:39.468 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:39.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:39.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:39.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:39 smithi125 conmon[56739]: cluster 2021-03-15T22:29:38. 2021-03-15T22:29:39.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:39 smithi125 conmon[56739]: 210338+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.d scrub ok 2021-03-15T22:29:39.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:39 smithi125 conmon[56739]: cluster 2021-03-15T22:29:38.292610+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e629: 8 total, 7 up, 7 in 2021-03-15T22:29:39.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:39 smithi125 conmon[56739]: cluster 2021-03-15T22:29:38.513985+0000 osd.3 (osd.3) 113 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:29:39.570 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:39.592 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:39.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:39.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:39.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:39.716 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:39.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:39.717 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:39.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[56514]: cluster 2021-03-15T22:29:38.210338+0000 osd.7 (osd.7) 124 : cluster [DBG] 5.d scrub ok 2021-03-15T22:29:39.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[56514]: cluster 2021-03-15T22:29:38.292610+0000 mon.a (mon.0) 1727 : cluster [DBG] osdmap e629: 8 total, 7 up, 7 in 2021-03-15T22:29:39.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[56514]: cluster 2021-03-15T22:29:38.513985+0000 osd.3 (osd.3) 113 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:29:39.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: cluster 2021-03-15T22:29:39.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: 2021-03-15T22:29:39.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: 2021 2021-03-15T22:29:39.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: - 2021-03-15T22:29:39.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: 03 2021-03-15T22:29:39.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: - 2021-03-15T22:29:39.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: 15 2021-03-15T22:29:39.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: T 2021-03-15T22:29:39.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: 22 2021-03-15T22:29:39.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: : 2021-03-15T22:29:39.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: 29 2021-03-15T22:29:39.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: : 2021-03-15T22:29:39.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: 38 2021-03-15T22:29:39.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: . 2021-03-15T22:29:39.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: 210338 2021-03-15T22:29:39.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: +0000 2021-03-15T22:29:39.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: 2021-03-15T22:29:39.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: osd.7 (osd.7) 124 2021-03-15T22:29:39.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: : cluster [DBG] 5.d scrub ok 2021-03-15T22:29:39.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:29:39.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: T22:29:38.292610+0000 mon.a (mon 2021-03-15T22:29:39.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: .0) 1727 : cluster [DBG] osdmap e629: 8 total, 7 up, 7 in 2021-03-15T22:29:39.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: cluster 2021-03-15T22:29:39.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: 2021-03-15T22:29:38.513985+0000 2021-03-15T22:29:39.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: osd.3 (osd.3) 113 : cluster [DBG] 2021-03-15T22:29:39.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:39 smithi076 conmon[62156]: 6.1f scrub ok 2021-03-15T22:29:39.818 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:39.841 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:39.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:39.842 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:39.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:29:39.901 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:39.943 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:39.966 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:39.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:39.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:40.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:40.091 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:40.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:40.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:40.105 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:29:39 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:29:40.171 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:40.194 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:40.202 INFO:tasks.rados.rados.0.smithi076.stdout:2278: done (4 left) 2021-03-15T22:29:40.203 INFO:tasks.rados.rados.0.smithi076.stdout:2280: done (3 left) 2021-03-15T22:29:40.203 INFO:tasks.rados.rados.0.smithi076.stdout:2281: done (2 left) 2021-03-15T22:29:40.203 INFO:tasks.rados.rados.0.smithi076.stdout:2282: done (1 left) 2021-03-15T22:29:40.203 INFO:tasks.rados.rados.0.smithi076.stdout:2283: done (0 left) 2021-03-15T22:29:40.204 INFO:tasks.rados.rados.0.smithi076.stdout:2284: write (excl) oid 6 current snap is 261 2021-03-15T22:29:40.204 INFO:tasks.rados.rados.0.smithi076.stdout:2284: seq_num 784 ranges {776688=579575,1894471=798599} 2021-03-15T22:29:40.210 INFO:tasks.rados.rados.0.smithi076.stdout:2284: writing smithi07694878-6 from 776688 to 1356263 tid 1 2021-03-15T22:29:40.224 INFO:tasks.rados.rados.0.smithi076.stdout:2284: writing smithi07694878-6 from 1894471 to 2693070 tid 2 2021-03-15T22:29:40.226 INFO:tasks.rados.rados.0.smithi076.stdout:2285: read oid 27 snap -1 2021-03-15T22:29:40.226 INFO:tasks.rados.rados.0.smithi076.stdout:2285: expect (ObjNum 773 snap 255 seq_num 773) 2021-03-15T22:29:40.226 INFO:tasks.rados.rados.0.smithi076.stdout:2284: finishing write tid 1 to smithi07694878-6 2021-03-15T22:29:40.227 INFO:tasks.rados.rados.0.smithi076.stdout:2286: write (excl) oid 35 current snap is 261 2021-03-15T22:29:40.227 INFO:tasks.rados.rados.0.smithi076.stdout:2286: seq_num 785 ranges {793495=654824,1952953=658610,3157167=36328} 2021-03-15T22:29:40.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:40.229 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:40.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:40.235 INFO:tasks.rados.rados.0.smithi076.stdout:2286: writing smithi07694878-35 from 793495 to 1448319 tid 1 2021-03-15T22:29:40.243 INFO:tasks.rados.rados.0.smithi076.stdout:2286: writing smithi07694878-35 from 1952953 to 2611563 tid 2 2021-03-15T22:29:40.246 INFO:tasks.rados.rados.0.smithi076.stdout:2286: writing smithi07694878-35 from 3157167 to 3193495 tid 3 2021-03-15T22:29:40.247 INFO:tasks.rados.rados.0.smithi076.stdout:2287: copy_from oid 31 from oid 25 current snap is 261 2021-03-15T22:29:40.251 INFO:tasks.rados.rados.0.smithi076.stdout:2284: finishing write tid 2 to smithi07694878-6 2021-03-15T22:29:40.252 INFO:tasks.rados.rados.0.smithi076.stdout:2284: finishing write tid 3 to smithi07694878-6 2021-03-15T22:29:40.252 INFO:tasks.rados.rados.0.smithi076.stdout:2284: finishing write tid 4 to smithi07694878-6 2021-03-15T22:29:40.252 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 1211 (ObjNum 784 snap 261 seq_num 784) dirty exists 2021-03-15T22:29:40.252 INFO:tasks.rados.rados.0.smithi076.stdout:2284: left oid 6 (ObjNum 784 snap 261 seq_num 784) 2021-03-15T22:29:40.253 INFO:tasks.rados.rados.0.smithi076.stdout:2285: expect (ObjNum 773 snap 255 seq_num 773) 2021-03-15T22:29:40.331 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:40.349 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:29:40.349 INFO:teuthology.orchestra.run.smithi125.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-03-15T22:29:40.349 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:29:40.354 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:40.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:40.356 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:40.371 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:40 smithi125 conmon[56739]: cluster 2021-03-15T22:29:39.130540+0000 mgr.y (mgr. 2021-03-15T22:29:40.372 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:40 smithi125 conmon[56739]: 14144) 806 : cluster [DBG] pgmap v1088: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 547 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 15 MiB/s rd, 686 KiB/s wr, 13 op/s; 53/1743 objects degraded (3.041%); 187/1743 objects misplaced (10.729%) 2021-03-15T22:29:40.372 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:40 smithi125 conmon[56739]: cluster 2021-03-15T22:29:39.295554+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e630: 8 total, 7 up, 7 in 2021-03-15T22:29:40.372 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:40 smithi125 conmon[56739]: audit 2021-03-15T22:29:39.945444+0000 mgr.y (mgr.14144) 807 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:40.372 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:40 smithi125 conmon[56739]: cluster 2021-03-15T22:29:40.187468+0000 mon.a (mon.0) 1729 : cluster [WRN] Health check update: Degraded data redundancy: 53/1743 objects degraded (3.041%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:40.373 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:40 smithi125 conmon[56739]: cluster 2021-03-15T22:29:40.195990+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e631: 8 total, 7 up, 7 in 2021-03-15T22:29:40.423 INFO:tasks.rados.rados.0.smithi076.stdout:2286: finishing write tid 1 to smithi07694878-35 2021-03-15T22:29:40.424 INFO:tasks.rados.rados.0.smithi076.stdout:2286: finishing write tid 2 to smithi07694878-35 2021-03-15T22:29:40.424 INFO:tasks.rados.rados.0.smithi076.stdout:2284: done (3 left) 2021-03-15T22:29:40.424 INFO:tasks.rados.rados.0.smithi076.stdout:2285: done (2 left) 2021-03-15T22:29:40.424 INFO:tasks.rados.rados.0.smithi076.stdout:2288: rollback oid 20 current snap is 261 2021-03-15T22:29:40.425 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 20 to 196 2021-03-15T22:29:40.425 INFO:tasks.rados.rados.0.smithi076.stdout:2286: finishing write tid 3 to smithi07694878-35 2021-03-15T22:29:40.425 INFO:tasks.rados.rados.0.smithi076.stdout:2286: finishing write tid 4 to smithi07694878-35 2021-03-15T22:29:40.425 INFO:tasks.rados.rados.0.smithi076.stdout:2286: finishing write tid 5 to smithi07694878-35 2021-03-15T22:29:40.426 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 1043 (ObjNum 785 snap 261 seq_num 785) dirty exists 2021-03-15T22:29:40.426 INFO:tasks.rados.rados.0.smithi076.stdout:2286: left oid 35 (ObjNum 785 snap 261 seq_num 785) 2021-03-15T22:29:40.426 INFO:tasks.rados.rados.0.smithi076.stdout:2287: finishing copy_from to smithi07694878-31 2021-03-15T22:29:40.427 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 1320 (ObjNum 783 snap 260 seq_num 783) dirty exists 2021-03-15T22:29:40.427 INFO:tasks.rados.rados.0.smithi076.stdout:2287: finishing copy_from racing read to smithi07694878-31 2021-03-15T22:29:40.427 INFO:tasks.rados.rados.0.smithi076.stdout:2286: done (2 left) 2021-03-15T22:29:40.427 INFO:tasks.rados.rados.0.smithi076.stdout:2287: done (1 left) 2021-03-15T22:29:40.428 INFO:tasks.rados.rados.0.smithi076.stdout:2289: read oid 22 snap -1 2021-03-15T22:29:40.428 INFO:tasks.rados.rados.0.smithi076.stdout:2289: expect (ObjNum 750 snap 245 seq_num 750) 2021-03-15T22:29:40.428 INFO:tasks.rados.rados.0.smithi076.stdout:2290: write (excl) oid 2 current snap is 261 2021-03-15T22:29:40.428 INFO:tasks.rados.rados.0.smithi076.stdout:2290: seq_num 786 ranges {410302=730073,1895082=402268,2951884=258418} 2021-03-15T22:29:40.430 INFO:tasks.rados.rados.0.smithi076.stdout:2290: writing smithi07694878-2 from 410302 to 1140375 tid 1 2021-03-15T22:29:40.436 INFO:tasks.rados.rados.0.smithi076.stdout:2290: writing smithi07694878-2 from 1895082 to 2297350 tid 2 2021-03-15T22:29:40.441 INFO:tasks.rados.rados.0.smithi076.stdout:2290: writing smithi07694878-2 from 2951884 to 3210302 tid 3 2021-03-15T22:29:40.441 INFO:tasks.rados.rados.0.smithi076.stdout:2291: snap_remove snap 224 2021-03-15T22:29:40.457 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:40.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:40.481 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:40.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:40.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:40.606 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:40.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:40.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:40.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:40 smithi076 conmon[62156]: cluster 2021-03-15T22:29:39.130540+0000 mgr.y (mgr.14144) 806 : cluster [DBG] pgmap v1088: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 547 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 15 MiB/s rd, 686 KiB/s wr, 13 op/s; 53/1743 objects degraded (3.041%); 187/1743 objects misplaced (10.729%) 2021-03-15T22:29:40.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:40 smithi076 conmon[62156]: cluster 2021-03-15T22:29:39.295554+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e630: 8 total, 7 up, 7 in 2021-03-15T22:29:40.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:40 smithi076 conmon[62156]: audit 2021-03-15T22:29:39.945444+0000 mgr.y (mgr.14144) 807 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:40.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:40 smithi076 conmon[62156]: cluster 2021-03-15T22:29:40.187468+0000 mon.a (mon.0) 1729 : cluster [WRN] Health check update: Degraded data redundancy: 53/1743 objects degraded (3.041%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:40.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:40 smithi076 conmon[62156]: cluster 2021-03-15T22:29:40.195990+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e631: 8 total, 7 up, 7 in 2021-03-15T22:29:40.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:40 smithi076 conmon[56514]: cluster 2021-03-15T22:29:39.130540 2021-03-15T22:29:40.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:40 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 806 : cluster [DBG] pgmap v1088: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 547 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 15 MiB/s rd, 686 KiB/s wr, 13 op/s; 53/1743 objects degraded (3.041%); 187/1743 objects misplaced (10.729%) 2021-03-15T22:29:40.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:40 smithi076 conmon[56514]: cluster 2021-03-15T22:29:39.295554+0000 mon.a (mon.0) 1728 : cluster [DBG] osdmap e630: 8 total, 7 up, 7 in 2021-03-15T22:29:40.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:40 smithi076 conmon[56514]: audit 2021-03-15T22:29:39.945444+0000 mgr.y (mgr.14144) 807 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:40.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:40 smithi076 conmon[56514]: cluster 2021-03-15T22:29:40.187468+0000 mon.a (mon.0) 1729 : cluster [WRN] Health check update: Degraded data redundancy: 53/1743 objects degraded (3.041%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:40.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:40 smithi076 conmon[56514]: cluster 2021-03-15T22:29:40.195990+0000 mon.a (mon.0) 1730 : cluster [DBG] osdmap e631: 8 total, 7 up, 7 in 2021-03-15T22:29:40.709 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:40.738 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:40.739 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:40.840 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:40.863 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:40.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:40.864 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:40.898 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:29:40.965 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:40.988 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:40.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:40.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:41.091 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:41.114 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:41.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:41.116 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:41.217 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:41.239 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:41.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:41.240 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:41.329 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:41.341 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:41.364 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:41.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:41.365 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:41.391 INFO:tasks.rados.rados.0.smithi076.stdout:2291: done (3 left) 2021-03-15T22:29:41.392 INFO:tasks.rados.rados.0.smithi076.stdout:2292: delete oid 10 current snap is 261 2021-03-15T22:29:41.392 INFO:tasks.rados.rados.0.smithi076.stdout:2289: expect (ObjNum 750 snap 245 seq_num 750) 2021-03-15T22:29:41.466 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:41.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:41.490 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:41.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:41.541 INFO:tasks.rados.rados.0.smithi076.stdout:2288: finishing rollback tid 0 to smithi07694878-20 2021-03-15T22:29:41.541 INFO:tasks.rados.rados.0.smithi076.stdout:2288: finishing rollback tid 1 to smithi07694878-20 2021-03-15T22:29:41.541 INFO:tasks.rados.rados.0.smithi076.stdout:2288: finishing rollback tid 2 to smithi07694878-20 2021-03-15T22:29:41.542 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 1532 (ObjNum 554 snap 176 seq_num 554) dirty exists 2021-03-15T22:29:41.542 INFO:tasks.rados.rados.0.smithi076.stdout:2290: finishing write tid 1 to smithi07694878-2 2021-03-15T22:29:41.542 INFO:tasks.rados.rados.0.smithi076.stdout:2290: finishing write tid 2 to smithi07694878-2 2021-03-15T22:29:41.542 INFO:tasks.rados.rados.0.smithi076.stdout:2290: finishing write tid 3 to smithi07694878-2 2021-03-15T22:29:41.543 INFO:tasks.rados.rados.0.smithi076.stdout:2290: finishing write tid 4 to smithi07694878-2 2021-03-15T22:29:41.543 INFO:tasks.rados.rados.0.smithi076.stdout:2290: finishing write tid 5 to smithi07694878-2 2021-03-15T22:29:41.543 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1324 (ObjNum 786 snap 261 seq_num 786) dirty exists 2021-03-15T22:29:41.543 INFO:tasks.rados.rados.0.smithi076.stdout:2290: left oid 2 (ObjNum 786 snap 261 seq_num 786) 2021-03-15T22:29:41.543 INFO:tasks.rados.rados.0.smithi076.stdout:2288: done (3 left) 2021-03-15T22:29:41.544 INFO:tasks.rados.rados.0.smithi076.stdout:2289: done (2 left) 2021-03-15T22:29:41.544 INFO:tasks.rados.rados.0.smithi076.stdout:2290: done (1 left) 2021-03-15T22:29:41.544 INFO:tasks.rados.rados.0.smithi076.stdout:2292: done (0 left) 2021-03-15T22:29:41.545 INFO:tasks.rados.rados.0.smithi076.stdout:2293: read oid 15 snap -1 2021-03-15T22:29:41.545 INFO:tasks.rados.rados.0.smithi076.stdout:2293: expect (ObjNum 714 snap 231 seq_num 714) 2021-03-15T22:29:41.545 INFO:tasks.rados.rados.0.smithi076.stdout:2294: write (excl) oid 44 current snap is 261 2021-03-15T22:29:41.546 INFO:tasks.rados.rados.0.smithi076.stdout:2294: seq_num 788 ranges {443916=480571,1612046=405937,2606730=495052} 2021-03-15T22:29:41.546 INFO:tasks.rados.rados.0.smithi076.stdout:2294: writing smithi07694878-44 from 443916 to 924487 tid 1 2021-03-15T22:29:41.551 INFO:tasks.rados.rados.0.smithi076.stdout:2294: writing smithi07694878-44 from 1612046 to 2017983 tid 2 2021-03-15T22:29:41.558 INFO:tasks.rados.rados.0.smithi076.stdout:2294: writing smithi07694878-44 from 2606730 to 3101782 tid 3 2021-03-15T22:29:41.558 INFO:tasks.rados.rados.0.smithi076.stdout:2295: copy_from oid 38 from oid 1 current snap is 261 2021-03-15T22:29:41.559 INFO:tasks.rados.rados.0.smithi076.stdout:2293: expect (ObjNum 714 snap 231 seq_num 714) 2021-03-15T22:29:41.561 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:41 smithi076 conmon[62156]: cluster 2021-03-15T22:29:40.242102+0000 osd.1 (osd.1) 125 : cluster [DBG] 7.b scrub ok 2021-03-15T22:29:41.561 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:41 smithi076 conmon[62156]: cluster 2021-03-15T22:29:40.500800+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.c scrub ok 2021-03-15T22:29:41.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:41 smithi076 conmon[56514]: cluster 2021-03-15T22:29:40.242102+0000 2021-03-15T22:29:41.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:41 smithi076 conmon[56514]: osd.1 (osd.1) 125 : cluster [DBG] 7.b scrub ok 2021-03-15T22:29:41.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:41 smithi076 conmon[56514]: cluster 2021-03-15T22:29:40.500800+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.c scrub ok 2021-03-15T22:29:41.592 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:41.615 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:41.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:41.616 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:41.663 INFO:tasks.rados.rados.0.smithi076.stdout:2294: finishing write tid 1 to smithi07694878-44 2021-03-15T22:29:41.663 INFO:tasks.rados.rados.0.smithi076.stdout:2294: finishing write tid 2 to smithi07694878-44 2021-03-15T22:29:41.664 INFO:tasks.rados.rados.0.smithi076.stdout:2293: done (2 left) 2021-03-15T22:29:41.664 INFO:tasks.rados.rados.0.smithi076.stdout:2296: read oid 28 snap 223 2021-03-15T22:29:41.664 INFO:tasks.rados.rados.0.smithi076.stdout:2296: expect (ObjNum 654 snap 205 seq_num 654) 2021-03-15T22:29:41.664 INFO:tasks.rados.rados.0.smithi076.stdout:2297: read oid 48 snap -1 2021-03-15T22:29:41.665 INFO:tasks.rados.rados.0.smithi076.stdout:2297: expect (ObjNum 757 snap 250 seq_num 757) 2021-03-15T22:29:41.665 INFO:tasks.rados.rados.0.smithi076.stdout:2298: write (excl) oid 20 current snap is 261 2021-03-15T22:29:41.665 INFO:tasks.rados.rados.0.smithi076.stdout:2294: finishing write tid 3 to smithi07694878-44 2021-03-15T22:29:41.665 INFO:tasks.rados.rados.0.smithi076.stdout:2294: finishing write tid 4 to smithi07694878-44 2021-03-15T22:29:41.665 INFO:tasks.rados.rados.0.smithi076.stdout:2294: finishing write tid 5 to smithi07694878-44 2021-03-15T22:29:41.666 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1208 (ObjNum 788 snap 261 seq_num 788) dirty exists 2021-03-15T22:29:41.666 INFO:tasks.rados.rados.0.smithi076.stdout:2294: left oid 44 (ObjNum 788 snap 261 seq_num 788) 2021-03-15T22:29:41.666 INFO:tasks.rados.rados.0.smithi076.stdout:2295: finishing copy_from to smithi07694878-38 2021-03-15T22:29:41.666 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 1036 (ObjNum 732 snap 239 seq_num 732) dirty exists 2021-03-15T22:29:41.667 INFO:tasks.rados.rados.0.smithi076.stdout:2295: finishing copy_from racing read to smithi07694878-38 2021-03-15T22:29:41.667 INFO:tasks.rados.rados.0.smithi076.stdout:2298: seq_num 789 ranges {460723=555820,1670528=549595,2634153=626570} 2021-03-15T22:29:41.668 INFO:tasks.rados.rados.0.smithi076.stdout:2298: writing smithi07694878-20 from 460723 to 1016543 tid 1 2021-03-15T22:29:41.675 INFO:tasks.rados.rados.0.smithi076.stdout:2298: writing smithi07694878-20 from 1670528 to 2220123 tid 2 2021-03-15T22:29:41.681 INFO:tasks.rados.rados.0.smithi076.stdout:2298: writing smithi07694878-20 from 2634153 to 3260723 tid 3 2021-03-15T22:29:41.681 INFO:tasks.rados.rados.0.smithi076.stdout:2294: done (4 left) 2021-03-15T22:29:41.682 INFO:tasks.rados.rados.0.smithi076.stdout:2295: done (3 left) 2021-03-15T22:29:41.682 INFO:tasks.rados.rados.0.smithi076.stdout:2299: write oid 40 current snap is 261 2021-03-15T22:29:41.682 INFO:tasks.rados.rados.0.smithi076.stdout:2299: seq_num 790 ranges {477530=631069,1612657=409606,2545223=517596} 2021-03-15T22:29:41.687 INFO:tasks.rados.rados.0.smithi076.stdout:2299: writing smithi07694878-40 from 477530 to 1108599 tid 1 2021-03-15T22:29:41.692 INFO:tasks.rados.rados.0.smithi076.stdout:2299: writing smithi07694878-40 from 1612657 to 2022263 tid 2 2021-03-15T22:29:41.698 INFO:tasks.rados.rados.0.smithi076.stdout:2299: writing smithi07694878-40 from 2545223 to 3062819 tid 3 2021-03-15T22:29:41.699 INFO:tasks.rados.rados.0.smithi076.stdout:2300: read oid 6 snap -1 2021-03-15T22:29:41.699 INFO:tasks.rados.rados.0.smithi076.stdout:2300: expect (ObjNum 784 snap 261 seq_num 784) 2021-03-15T22:29:41.699 INFO:tasks.rados.rados.0.smithi076.stdout:2301: write (excl) oid 18 current snap is 261 2021-03-15T22:29:41.699 INFO:tasks.rados.rados.0.smithi076.stdout:2301: seq_num 791 ranges {494337=422671,1387492=553264,2688999=445221} 2021-03-15T22:29:41.704 INFO:tasks.rados.rados.0.smithi076.stdout:2301: writing smithi07694878-18 from 494337 to 917008 tid 1 2021-03-15T22:29:41.709 INFO:tasks.rados.rados.0.smithi076.stdout:2301: writing smithi07694878-18 from 1387492 to 1940756 tid 2 2021-03-15T22:29:41.716 INFO:tasks.rados.rados.0.smithi076.stdout:2301: writing smithi07694878-18 from 2688999 to 3134220 tid 3 2021-03-15T22:29:41.716 INFO:tasks.rados.rados.0.smithi076.stdout:2302: write (excl) oid 44 current snap is 261 2021-03-15T22:29:41.717 INFO:tasks.rados.rados.0.smithi076.stdout:2296: expect (ObjNum 654 snap 205 seq_num 654) 2021-03-15T22:29:41.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:41.727 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:41 smithi125 conmon[56739]: cluster 2021-03-15T22:29:40.242102+0000 osd.1 ( 2021-03-15T22:29:41.728 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:41 smithi125 conmon[56739]: osd.1) 125 : cluster [DBG] 7.b scrub ok 2021-03-15T22:29:41.728 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:41 smithi125 conmon[56739]: cluster 2021-03-15T22:29:40.500800+0000 osd.3 (osd.3) 114 : cluster [DBG] 6.c scrub ok 2021-03-15T22:29:41.740 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:41.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:41.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:41.842 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:41.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:41.867 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:41.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:41.917 INFO:tasks.rados.rados.0.smithi076.stdout:2297: expect (ObjNum 757 snap 250 seq_num 757) 2021-03-15T22:29:41.968 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:41.992 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:41.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:41.993 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:42.006 INFO:tasks.rados.rados.0.smithi076.stdout:2298: finishing write tid 1 to smithi07694878-20 2021-03-15T22:29:42.006 INFO:tasks.rados.rados.0.smithi076.stdout:2302: seq_num 792 ranges {511144=497920,1445974=413275,2316422=656493} 2021-03-15T22:29:42.011 INFO:tasks.rados.rados.0.smithi076.stdout:2302: writing smithi07694878-44 from 511144 to 1009064 tid 1 2021-03-15T22:29:42.016 INFO:tasks.rados.rados.0.smithi076.stdout:2302: writing smithi07694878-44 from 1445974 to 1859249 tid 2 2021-03-15T22:29:42.024 INFO:tasks.rados.rados.0.smithi076.stdout:2302: writing smithi07694878-44 from 2316422 to 2972915 tid 3 2021-03-15T22:29:42.024 INFO:tasks.rados.rados.0.smithi076.stdout:2296: done (6 left) 2021-03-15T22:29:42.025 INFO:tasks.rados.rados.0.smithi076.stdout:2297: done (5 left) 2021-03-15T22:29:42.025 INFO:tasks.rados.rados.0.smithi076.stdout:2303: rollback oid 16 current snap is 261 2021-03-15T22:29:42.025 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 223 2021-03-15T22:29:42.025 INFO:tasks.rados.rados.0.smithi076.stdout:2298: finishing write tid 2 to smithi07694878-20 2021-03-15T22:29:42.025 INFO:tasks.rados.rados.0.smithi076.stdout:2304: write (excl) oid 15 current snap is 261 2021-03-15T22:29:42.026 INFO:tasks.rados.rados.0.smithi076.stdout:2298: finishing write tid 3 to smithi07694878-20 2021-03-15T22:29:42.026 INFO:tasks.rados.rados.0.smithi076.stdout:2298: finishing write tid 4 to smithi07694878-20 2021-03-15T22:29:42.026 INFO:tasks.rados.rados.0.smithi076.stdout:2298: finishing write tid 5 to smithi07694878-20 2021-03-15T22:29:42.026 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 1536 (ObjNum 789 snap 261 seq_num 789) dirty exists 2021-03-15T22:29:42.027 INFO:tasks.rados.rados.0.smithi076.stdout:2298: left oid 20 (ObjNum 789 snap 261 seq_num 789) 2021-03-15T22:29:42.027 INFO:tasks.rados.rados.0.smithi076.stdout:2299: finishing write tid 1 to smithi07694878-40 2021-03-15T22:29:42.027 INFO:tasks.rados.rados.0.smithi076.stdout:2304: seq_num 793 ranges {527951=573169,1504456=556933,2743845=467765} 2021-03-15T22:29:42.028 INFO:tasks.rados.rados.0.smithi076.stdout:2304: writing smithi07694878-15 from 527951 to 1101120 tid 1 2021-03-15T22:29:42.033 INFO:tasks.rados.rados.0.smithi076.stdout:2304: writing smithi07694878-15 from 1504456 to 2061389 tid 2 2021-03-15T22:29:42.041 INFO:tasks.rados.rados.0.smithi076.stdout:2304: writing smithi07694878-15 from 2743845 to 3211610 tid 3 2021-03-15T22:29:42.041 INFO:tasks.rados.rados.0.smithi076.stdout:2298: done (6 left) 2021-03-15T22:29:42.041 INFO:tasks.rados.rados.0.smithi076.stdout:2305: write (excl) oid 42 current snap is 261 2021-03-15T22:29:42.042 INFO:tasks.rados.rados.0.smithi076.stdout:2299: finishing write tid 2 to smithi07694878-40 2021-03-15T22:29:42.042 INFO:tasks.rados.rados.0.smithi076.stdout:2305: seq_num 794 ranges {544758=648418,1846585=416944,2771268=573490} 2021-03-15T22:29:42.047 INFO:tasks.rados.rados.0.smithi076.stdout:2305: writing smithi07694878-42 from 544758 to 1193176 tid 1 2021-03-15T22:29:42.053 INFO:tasks.rados.rados.0.smithi076.stdout:2305: writing smithi07694878-42 from 1846585 to 2263529 tid 2 2021-03-15T22:29:42.059 INFO:tasks.rados.rados.0.smithi076.stdout:2305: writing smithi07694878-42 from 2771268 to 3344758 tid 3 2021-03-15T22:29:42.059 INFO:tasks.rados.rados.0.smithi076.stdout:2306: delete oid 45 current snap is 261 2021-03-15T22:29:42.059 INFO:tasks.rados.rados.0.smithi076.stdout:2299: finishing write tid 3 to smithi07694878-40 2021-03-15T22:29:42.060 INFO:tasks.rados.rados.0.smithi076.stdout:2300: expect (ObjNum 784 snap 261 seq_num 784) 2021-03-15T22:29:42.094 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:42.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:42.118 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:42.119 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:42.172 INFO:tasks.rados.rados.0.smithi076.stdout:2299: finishing write tid 4 to smithi07694878-40 2021-03-15T22:29:42.172 INFO:tasks.rados.rados.0.smithi076.stdout:2299: finishing write tid 5 to smithi07694878-40 2021-03-15T22:29:42.172 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1732 (ObjNum 790 snap 261 seq_num 790) dirty exists 2021-03-15T22:29:42.173 INFO:tasks.rados.rados.0.smithi076.stdout:2299: left oid 40 (ObjNum 790 snap 261 seq_num 790) 2021-03-15T22:29:42.173 INFO:tasks.rados.rados.0.smithi076.stdout:2301: finishing write tid 1 to smithi07694878-18 2021-03-15T22:29:42.173 INFO:tasks.rados.rados.0.smithi076.stdout:2301: finishing write tid 2 to smithi07694878-18 2021-03-15T22:29:42.173 INFO:tasks.rados.rados.0.smithi076.stdout:2299: done (7 left) 2021-03-15T22:29:42.173 INFO:tasks.rados.rados.0.smithi076.stdout:2300: done (6 left) 2021-03-15T22:29:42.174 INFO:tasks.rados.rados.0.smithi076.stdout:2306: done (5 left) 2021-03-15T22:29:42.174 INFO:tasks.rados.rados.0.smithi076.stdout:2307: read oid 5 snap -1 2021-03-15T22:29:42.174 INFO:tasks.rados.rados.0.smithi076.stdout:2307: expect (ObjNum 494 snap 146 seq_num 494) 2021-03-15T22:29:42.174 INFO:tasks.rados.rados.0.smithi076.stdout:2308: delete oid 14 current snap is 261 2021-03-15T22:29:42.175 INFO:tasks.rados.rados.0.smithi076.stdout:2301: finishing write tid 3 to smithi07694878-18 2021-03-15T22:29:42.175 INFO:tasks.rados.rados.0.smithi076.stdout:2301: finishing write tid 4 to smithi07694878-18 2021-03-15T22:29:42.175 INFO:tasks.rados.rados.0.smithi076.stdout:2301: finishing write tid 5 to smithi07694878-18 2021-03-15T22:29:42.175 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 1041 (ObjNum 791 snap 261 seq_num 791) dirty exists 2021-03-15T22:29:42.176 INFO:tasks.rados.rados.0.smithi076.stdout:2301: left oid 18 (ObjNum 791 snap 261 seq_num 791) 2021-03-15T22:29:42.176 INFO:tasks.rados.rados.0.smithi076.stdout:2302: finishing write tid 1 to smithi07694878-44 2021-03-15T22:29:42.176 INFO:tasks.rados.rados.0.smithi076.stdout:2302: finishing write tid 2 to smithi07694878-44 2021-03-15T22:29:42.176 INFO:tasks.rados.rados.0.smithi076.stdout:2303: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:29:42.176 INFO:tasks.rados.rados.0.smithi076.stdout:2302: finishing write tid 3 to smithi07694878-44 2021-03-15T22:29:42.177 INFO:tasks.rados.rados.0.smithi076.stdout:2303: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:29:42.177 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1654 (ObjNum 604 snap 188 seq_num 604) dirty exists 2021-03-15T22:29:42.177 INFO:tasks.rados.rados.0.smithi076.stdout:2302: finishing write tid 4 to smithi07694878-44 2021-03-15T22:29:42.177 INFO:tasks.rados.rados.0.smithi076.stdout:2302: finishing write tid 5 to smithi07694878-44 2021-03-15T22:29:42.178 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1212 (ObjNum 792 snap 261 seq_num 792) dirty exists 2021-03-15T22:29:42.178 INFO:tasks.rados.rados.0.smithi076.stdout:2302: left oid 44 (ObjNum 792 snap 261 seq_num 792) 2021-03-15T22:29:42.178 INFO:tasks.rados.rados.0.smithi076.stdout:2304: finishing write tid 1 to smithi07694878-15 2021-03-15T22:29:42.178 INFO:tasks.rados.rados.0.smithi076.stdout:2304: finishing write tid 2 to smithi07694878-15 2021-03-15T22:29:42.178 INFO:tasks.rados.rados.0.smithi076.stdout:2304: finishing write tid 3 to smithi07694878-15 2021-03-15T22:29:42.179 INFO:tasks.rados.rados.0.smithi076.stdout:2304: finishing write tid 4 to smithi07694878-15 2021-03-15T22:29:42.179 INFO:tasks.rados.rados.0.smithi076.stdout:2304: finishing write tid 5 to smithi07694878-15 2021-03-15T22:29:42.180 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 1659 (ObjNum 793 snap 261 seq_num 793) dirty exists 2021-03-15T22:29:42.180 INFO:tasks.rados.rados.0.smithi076.stdout:2304: left oid 15 (ObjNum 793 snap 261 seq_num 793) 2021-03-15T22:29:42.180 INFO:tasks.rados.rados.0.smithi076.stdout:2305: finishing write tid 1 to smithi07694878-42 2021-03-15T22:29:42.180 INFO:tasks.rados.rados.0.smithi076.stdout:2305: finishing write tid 2 to smithi07694878-42 2021-03-15T22:29:42.181 INFO:tasks.rados.rados.0.smithi076.stdout:2305: finishing write tid 3 to smithi07694878-42 2021-03-15T22:29:42.181 INFO:tasks.rados.rados.0.smithi076.stdout:2305: finishing write tid 4 to smithi07694878-42 2021-03-15T22:29:42.181 INFO:tasks.rados.rados.0.smithi076.stdout:2305: finishing write tid 5 to smithi07694878-42 2021-03-15T22:29:42.181 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 1737 (ObjNum 794 snap 261 seq_num 794) dirty exists 2021-03-15T22:29:42.182 INFO:tasks.rados.rados.0.smithi076.stdout:2305: left oid 42 (ObjNum 794 snap 261 seq_num 794) 2021-03-15T22:29:42.182 INFO:tasks.rados.rados.0.smithi076.stdout:2301: done (6 left) 2021-03-15T22:29:42.182 INFO:tasks.rados.rados.0.smithi076.stdout:2302: done (5 left) 2021-03-15T22:29:42.182 INFO:tasks.rados.rados.0.smithi076.stdout:2303: done (4 left) 2021-03-15T22:29:42.182 INFO:tasks.rados.rados.0.smithi076.stdout:2304: done (3 left) 2021-03-15T22:29:42.183 INFO:tasks.rados.rados.0.smithi076.stdout:2305: done (2 left) 2021-03-15T22:29:42.183 INFO:tasks.rados.rados.0.smithi076.stdout:2308: done (1 left) 2021-03-15T22:29:42.183 INFO:tasks.rados.rados.0.smithi076.stdout:2309: rollback oid 16 current snap is 261 2021-03-15T22:29:42.183 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 196 2021-03-15T22:29:42.184 INFO:tasks.rados.rados.0.smithi076.stdout:2310: read oid 50 snap -1 2021-03-15T22:29:42.184 INFO:tasks.rados.rados.0.smithi076.stdout:2310: expect (ObjNum 658 snap 206 seq_num 658) 2021-03-15T22:29:42.184 INFO:tasks.rados.rados.0.smithi076.stdout:2311: snap_remove snap 183 2021-03-15T22:29:42.194 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:29:42.194 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:29:42.194 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:29:42.194 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:29:42.195 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:29:42.220 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:42.244 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:42.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:42.245 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:42.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:42.369 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:42.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:42.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:42.379 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[62156]: cluster 2021-03-15T22:29:40.964743+0000 osd.4 (osd.4) 134 : cluster [DBG] 5.f scrub ok 2021-03-15T22:29:42.389 INFO:tasks.rados.rados.0.smithi076.stdout:2311: done (3 left) 2021-03-15T22:29:42.390 INFO:tasks.rados.rados.0.smithi076.stdout:2312: snap_create 2021-03-15T22:29:42.390 INFO:tasks.rados.rados.0.smithi076.stdout:2307: expect (ObjNum 494 snap 146 seq_num 494) 2021-03-15T22:29:42.410 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:29:42.470 INFO:tasks.rados.rados.0.smithi076.stdout:2310: expect (ObjNum 658 snap 206 seq_num 658) 2021-03-15T22:29:42.472 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:42.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:42.496 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:42.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:42.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:42 smithi125 conmon[56739]: cluster 2021-03-15T22:29:40.964743+0000 osd.4 (osd.4) 134 : cluster [DBG] 5.f scrub ok 2021-03-15T22:29:42.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:42 smithi125 conmon[56739]: cluster 2021-03-15T22:29:41.131100+0000 mgr.y (mgr.14144) 808 : cluster [DBG] pgmap v1091: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 554 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 692 KiB/s wr, 14 op/s; 54/1758 objects degraded (3.072%); 187/1758 objects misplaced (10.637%) 2021-03-15T22:29:42.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:42 smithi125 conmon[56739]: cluster 2021-03-15T22:29:41.190054+0000 osd.1 (osd.1) 126 : cluster [DBG] 5.c scrub ok 2021-03-15T22:29:42.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:42 smithi125 conmon[56739]: cluster 2021-03-15T22:29:41.377329+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e632: 8 total, 7 up, 7 in 2021-03-15T22:29:42.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:42 smithi125 conmon[56739]: cluster 2021-03-15T22:29:41.515189+0000 osd.3 (osd.3) 115 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:29:42.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:42 smithi125 conmon[56739]: audit 2021-03-15T22:29:41.939010+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? 172.21.15.76:0/2339639568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:29:42.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:42.602 INFO:tasks.rados.rados.0.smithi076.stdout:2309: finishing rollback tid 0 to smithi07694878-16 2021-03-15T22:29:42.602 INFO:tasks.rados.rados.0.smithi076.stdout:2309: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:29:42.603 INFO:tasks.rados.rados.0.smithi076.stdout:2309: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:29:42.603 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1662 (ObjNum 604 snap 188 seq_num 604) dirty exists 2021-03-15T22:29:42.622 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:42.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:42.623 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:42.725 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:42.747 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:42.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:42.749 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:42.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[56514]: cluster 2021-03-15T22:29:40.964743+0000 osd.4 (osd.4) 134 : cluster [DBG] 5.f scrub ok 2021-03-15T22:29:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[56514]: cluster 2021-03-15T22:29:41.131100+0000 mgr.y (mgr.14144) 808 : cluster [DBG] pgmap v1091: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 554 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 692 KiB/s wr, 14 op/s; 54/1758 objects degraded (3.072%); 187/1758 objects misplaced (10.637%) 2021-03-15T22:29:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[56514]: cluster 2021-03-15T22:29:41.190054+0000 osd.1 (osd.1) 126 : cluster [DBG] 5.c scrub ok 2021-03-15T22:29:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[56514]: cluster 2021-03-15T22:29:41.377329+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e632: 8 total, 7 up, 7 in 2021-03-15T22:29:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[56514]: cluster 2021-03-15T22:29:41.515189+0000 osd.3 (osd.3) 115 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:29:42.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[56514]: 2021-03-15T22:29:42.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[56514]: audit 2021-03-15T22:29:41.939010+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? 172.21.15.76:0/2339639568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:29:42.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[62156]: cluster 2021-03-15T22:29:41.131100+0000 mgr.y (mgr.14144) 808 : cluster [DBG] pgmap v1091: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 554 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 692 KiB/s wr, 14 op/s; 54/1758 objects degraded (3.072%); 187/1758 objects misplaced (10.637%) 2021-03-15T22:29:42.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[62156]: cluster 2021-03-15T22:29:41.190054+0000 osd.1 (osd.1) 126 : cluster [DBG] 5.c scrub ok 2021-03-15T22:29:42.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[62156]: cluster 2021-03-15T22:29:41.377329+0000 mon.a (mon.0) 1731 : cluster [DBG] osdmap e632: 8 total, 7 up, 7 in 2021-03-15T22:29:42.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[62156]: cluster 2021-03-15T22:29:41.515189+0000 osd.3 (osd.3) 115 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:29:42.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:42 smithi076 conmon[62156]: audit 2021-03-15T22:29:41.939010+0000 mon.a (mon.0) 1732 : audit [INF] from='client.? 172.21.15.76:0/2339639568' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:29:42.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:42.872 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:42.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:42.873 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:42.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-15T22:29:42.975 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:43.014 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:43.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:43.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:43.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:43.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:43.156 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:43.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:43.226 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:43.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:43.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:43.292 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:43.293 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:43.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:43.403 INFO:tasks.rados.rados.0.smithi076.stdout:2307: done (3 left) 2021-03-15T22:29:43.403 INFO:tasks.rados.rados.0.smithi076.stdout:2309: done (2 left) 2021-03-15T22:29:43.403 INFO:tasks.rados.rados.0.smithi076.stdout:2310: done (1 left) 2021-03-15T22:29:43.403 INFO:tasks.rados.rados.0.smithi076.stdout:2312: done (0 left) 2021-03-15T22:29:43.404 INFO:tasks.rados.rados.0.smithi076.stdout:2313: read oid 46 snap -1 2021-03-15T22:29:43.404 INFO:tasks.rados.rados.0.smithi076.stdout:2313: expect deleted 2021-03-15T22:29:43.404 INFO:tasks.rados.rados.0.smithi076.stdout:2314: write (excl) oid 13 current snap is 262 2021-03-15T22:29:43.404 INFO:tasks.rados.rados.0.smithi076.stdout:2314: seq_num 797 ranges {595179=474165,1622031=564271,2737184=629206} 2021-03-15T22:29:43.405 INFO:tasks.rados.rados.0.smithi076.stdout:2314: writing smithi07694878-13 from 595179 to 1069344 tid 1 2021-03-15T22:29:43.411 INFO:tasks.rados.rados.0.smithi076.stdout:2314: writing smithi07694878-13 from 1622031 to 2186302 tid 2 2021-03-15T22:29:43.419 INFO:tasks.rados.rados.0.smithi076.stdout:2314: writing smithi07694878-13 from 2737184 to 3366390 tid 3 2021-03-15T22:29:43.420 INFO:tasks.rados.rados.0.smithi076.stdout:2315: write oid 23 current snap is 262 2021-03-15T22:29:43.420 INFO:tasks.rados.rados.0.smithi076.stdout:2315: seq_num 798 ranges {611986=549414,1564160=424282,2764607=440478} 2021-03-15T22:29:43.424 INFO:tasks.rados.rados.0.smithi076.stdout:2315: writing smithi07694878-23 from 611986 to 1161400 tid 1 2021-03-15T22:29:43.428 INFO:tasks.rados.rados.0.smithi076.stdout:2315: writing smithi07694878-23 from 1564160 to 1988442 tid 2 2021-03-15T22:29:43.431 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:43.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:43.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:43.436 INFO:tasks.rados.rados.0.smithi076.stdout:2315: writing smithi07694878-23 from 2764607 to 3205085 tid 3 2021-03-15T22:29:43.436 INFO:tasks.rados.rados.0.smithi076.stdout:2316: snap_create 2021-03-15T22:29:43.436 INFO:tasks.rados.rados.0.smithi076.stdout:2314: finishing write tid 1 to smithi07694878-13 2021-03-15T22:29:43.436 INFO:tasks.rados.rados.0.smithi076.stdout:2314: finishing write tid 2 to smithi07694878-13 2021-03-15T22:29:43.437 INFO:tasks.rados.rados.0.smithi076.stdout:2314: finishing write tid 3 to smithi07694878-13 2021-03-15T22:29:43.437 INFO:tasks.rados.rados.0.smithi076.stdout:2314: finishing write tid 4 to smithi07694878-13 2021-03-15T22:29:43.437 INFO:tasks.rados.rados.0.smithi076.stdout:2314: finishing write tid 5 to smithi07694878-13 2021-03-15T22:29:43.437 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1533 (ObjNum 797 snap 262 seq_num 797) dirty exists 2021-03-15T22:29:43.437 INFO:tasks.rados.rados.0.smithi076.stdout:2314: left oid 13 (ObjNum 797 snap 262 seq_num 797) 2021-03-15T22:29:43.438 INFO:tasks.rados.rados.0.smithi076.stdout:2315: finishing write tid 1 to smithi07694878-23 2021-03-15T22:29:43.438 INFO:tasks.rados.rados.0.smithi076.stdout:2315: finishing write tid 2 to smithi07694878-23 2021-03-15T22:29:43.439 INFO:tasks.rados.rados.0.smithi076.stdout:2315: finishing write tid 3 to smithi07694878-23 2021-03-15T22:29:43.439 INFO:tasks.rados.rados.0.smithi076.stdout:2315: finishing write tid 4 to smithi07694878-23 2021-03-15T22:29:43.440 INFO:tasks.rados.rados.0.smithi076.stdout:2315: finishing write tid 5 to smithi07694878-23 2021-03-15T22:29:43.440 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 1045 (ObjNum 798 snap 262 seq_num 798) dirty exists 2021-03-15T22:29:43.440 INFO:tasks.rados.rados.0.smithi076.stdout:2315: left oid 23 (ObjNum 798 snap 262 seq_num 798) 2021-03-15T22:29:43.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:43.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:43.566 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:43.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[56514]: cluster 2021-03-15T22:29:41.941506+0000 osd.4 (osd.4 2021-03-15T22:29:43.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[56514]: ) 135 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:29:43.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[56514]: cluster 2021-03-15T22:29:42.375800+0000 mon.a (mon.0) 1733 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:43.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[56514]: audit 2021-03-15T22:29:42.383977+0000 mon.a (mon.0) 1734 : audit [INF] from='client.? 172.21.15.76:0/2339639568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:29:43.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[56514]: cluster 2021-03-15T22:29:42.384063+0000 mon.a (mon.0) 1735 : cluster [DBG] osdmap e633: 8 total, 7 up, 7 in 2021-03-15T22:29:43.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[56514]: cluster 2021-03-15T22:29:42.410047+0000 osd.3 (osd.3) 116 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:43.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[56514]: cluster 2021-03-15T22:29:42.410624+0000 osd.3 (osd.3) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:43.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: cluster 2021-03-15T22:29:41.941506+0000 2021-03-15T22:29:43.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: osd.4 (osd.4) 135 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:29:43.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: cluster 2021-03-15T22:29:42 2021-03-15T22:29:43.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: .375800+0000 mon.a (mon.0) 1733 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:43.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:29:43.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: -15T22:29:42.383977+0000 mon.a ( 2021-03-15T22:29:43.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: mon.0) 1734 : audit [INF] from='client.? 172.21.15.76:0/2339639568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:29:43.651 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:29:43.651 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: -03-15T22:29:42.384063+0000 mon.a 2021-03-15T22:29:43.651 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: (mon.0) 1735 : cluster [DBG] osdmap e633: 8 total, 7 up, 7 in 2021-03-15T22:29:43.651 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:29:43.651 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: -03-15T22:29:42.410047 2021-03-15T22:29:43.652 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: +0000 osd.3 (osd.3) 116 : cluster [DBG] 2021-03-15T22:29:43.652 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: purged_snaps scrub starts 2021-03-15T22:29:43.652 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: cluster 2021-03-15T22:29:42. 2021-03-15T22:29:43.652 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:43 smithi076 conmon[62156]: 410624+0000 osd.3 (osd.3) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:43.667 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:43.670 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:43 smithi125 conmon[56739]: cluster 2021-03-15T22:29:41.941506+0000 osd.4 (osd.4) 135 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:29:43.670 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:43 smithi125 conmon[56739]: cluster 2021-03-15T22:29:43.671 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:43 smithi125 conmon[56739]: 2021-03-15T22:29:42.375800+0000 mon.a (mon.0) 1733 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:43.671 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:43 smithi125 conmon[56739]: audit 2021-03-15T22:29:42.383977+0000 mon.a (mon.0) 1734 : audit [INF] from='client.? 172.21.15.76:0/2339639568' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:29:43.672 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:43 smithi125 conmon[56739]: cluster 2021-03-15T22:29:42.384063+0000 mon.a (mon.0) 1735 : cluster [DBG] osdmap e633: 8 total, 7 up, 7 in 2021-03-15T22:29:43.673 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:43 smithi125 conmon[56739]: cluster 2021-03-15T22:29:42.410047+0000 osd.3 (osd.3) 116 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:43.673 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:43 smithi125 conmon[56739]: cluster 2021-03-15T22:29:42 2021-03-15T22:29:43.674 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:43 smithi125 conmon[56739]: .410624+0000 osd.3 (osd.3) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:43.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:43.701 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:43.701 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:43.762 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:29:43.776 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:29:43.793 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:29:43.802 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:43.810 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:29:43.828 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:29:43.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:43.838 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:43.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:43.845 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:29:43.862 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:29:43.863 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:29:43.863 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:29:43.863 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:29:43.864 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:29:43.864 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:29:43.864 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:29:43.864 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:29:43.865 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:29:43.939 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:43.966 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:43.966 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:44.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:44.089 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:44.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:44.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:44.192 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:44.213 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:44.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:44.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:44.316 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:44.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:44.340 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:44.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:44.400 INFO:tasks.rados.rados.0.smithi076.stdout:2313: done (3 left) 2021-03-15T22:29:44.400 INFO:tasks.rados.rados.0.smithi076.stdout:2314: done (2 left) 2021-03-15T22:29:44.401 INFO:tasks.rados.rados.0.smithi076.stdout:2315: done (1 left) 2021-03-15T22:29:44.401 INFO:tasks.rados.rados.0.smithi076.stdout:2316: done (0 left) 2021-03-15T22:29:44.401 INFO:tasks.rados.rados.0.smithi076.stdout:2317: snap_remove snap 244 2021-03-15T22:29:44.441 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:44.466 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:44.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:44.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:44.513 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:29:44.514 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:29:44.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:44 smithi125 conmon[56739]: cluster 2021-03-15T22:29:43.131796+0000 mgr.y (mgr.14144) 809 : cluster [DBG] pgmap v1094: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 558 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 3.3 MiB/s wr, 27 op/s; 54/1761 objects degraded (3.066%); 188/1761 objects misplaced (10.676%) 2021-03-15T22:29:44.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:44 smithi125 conmon[56739]: cluster 2021-03-15T22:29:43.392660+0000 mon.a (mon.0) 1736 : cluster [DBG] osdmap e634: 8 total, 7 up, 7 in 2021-03-15T22:29:44.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:44 smithi125 conmon[56739]: audit 2021-03-15T22:29:43.741106+0000 mon.a (mon.0) 1737 : audit [DBG] from='client.? 172.21.15.76:0/3161615036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:44.569 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:44.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:44.593 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:44.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:44.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:44.717 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:44.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:44.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:44.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:44 smithi076 conmon[56514]: cluster 2021-03-15T22:29:43.131796+0000 mgr.y (mgr.14144) 809 : cluster [DBG] pgmap v1094: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 558 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 3.3 MiB/s wr, 27 op/s; 54/1761 objects degraded (3.066%); 188/1761 objects misplaced (10.676%) 2021-03-15T22:29:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:44 smithi076 conmon[56514]: cluster 2021-03-15T22:29:43.392660+0000 mon.a (mon.0) 1736 : cluster [DBG] osdmap e634: 8 total, 7 up, 7 in 2021-03-15T22:29:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:44 smithi076 conmon[56514]: audit 2021-03-15T22:29:43.741106+0000 mon.a (mon.0) 1737 : audit [DBG] from='client.? 172.21.15.76:0/3161615036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:44.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:44 smithi076 conmon[62156]: cluster 2021-03-15T22:29:43.131796+0000 mgr.y 2021-03-15T22:29:44.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:44 smithi076 conmon[62156]: (mgr.14144) 809 : cluster [DBG] pgmap v1094: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 558 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 14 MiB/s rd, 3.3 MiB/s wr, 27 op/s; 54/1761 objects degraded (3.066%); 188/1761 objects misplaced (10.676%) 2021-03-15T22:29:44.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:44 smithi076 conmon[62156]: cluster 2021-03-15T22:29:43.392660+0000 mon.a (mon.0) 1736 : cluster [DBG] osdmap e634: 8 total, 7 up, 7 in 2021-03-15T22:29:44.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:44 smithi076 conmon[62156]: audit 2021-03-15T22:29:43.741106+0000 mon.a (mon.0) 1737 : audit [DBG] from='client.? 172.21.15.76:0/3161615036' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:44.805 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:29:44.806 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:29:44.819 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:44.844 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:44.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:44.945 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:44.967 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:44.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:44.969 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:45.070 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:45.093 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:45.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:45.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:45.196 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:45.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:45.220 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:45.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:45.322 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:45.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:45.347 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:45.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:45.415 INFO:tasks.rados.rados.0.smithi076.stdout:2317: done (0 left) 2021-03-15T22:29:45.415 INFO:tasks.rados.rados.0.smithi076.stdout:2318: read oid 29 snap -1 2021-03-15T22:29:45.415 INFO:tasks.rados.rados.0.smithi076.stdout:2318: expect (ObjNum 776 snap 258 seq_num 776) 2021-03-15T22:29:45.415 INFO:tasks.rados.rados.0.smithi076.stdout:2319: write oid 8 current snap is 263 2021-03-15T22:29:45.416 INFO:tasks.rados.rados.0.smithi076.stdout:2319: seq_num 799 ranges {628793=741016,2138995=567940,3192030=236763} 2021-03-15T22:29:45.419 INFO:tasks.rados.rados.0.smithi076.stdout:2319: writing smithi07694878-8 from 628793 to 1369809 tid 1 2021-03-15T22:29:45.427 INFO:tasks.rados.rados.0.smithi076.stdout:2319: writing smithi07694878-8 from 2138995 to 2706935 tid 2 2021-03-15T22:29:45.432 INFO:tasks.rados.rados.0.smithi076.stdout:2319: writing smithi07694878-8 from 3192030 to 3428793 tid 3 2021-03-15T22:29:45.432 INFO:tasks.rados.rados.0.smithi076.stdout:2320: read oid 36 snap 223 2021-03-15T22:29:45.433 INFO:tasks.rados.rados.0.smithi076.stdout:2320: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:29:45.433 INFO:tasks.rados.rados.0.smithi076.stdout:2321: write (excl) oid 6 current snap is 263 2021-03-15T22:29:45.433 INFO:tasks.rados.rados.0.smithi076.stdout:2321: seq_num 800 ranges {645600=416265,1797477=711598,3219453=226147} 2021-03-15T22:29:45.438 INFO:tasks.rados.rados.0.smithi076.stdout:2321: writing smithi07694878-6 from 645600 to 1061865 tid 1 2021-03-15T22:29:45.446 INFO:tasks.rados.rados.0.smithi076.stdout:2321: writing smithi07694878-6 from 1797477 to 2509075 tid 2 2021-03-15T22:29:45.449 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:29:45.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:45.451 INFO:tasks.rados.rados.0.smithi076.stdout:2321: writing smithi07694878-6 from 3219453 to 3445600 tid 3 2021-03-15T22:29:45.452 INFO:tasks.rados.rados.0.smithi076.stdout:2322: snap_create 2021-03-15T22:29:45.452 INFO:tasks.rados.rados.0.smithi076.stdout:2318: expect (ObjNum 776 snap 258 seq_num 776) 2021-03-15T22:29:45.472 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:45.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:45.474 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:45.485 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:29:45.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:45.586 INFO:tasks.rados.rados.0.smithi076.stdout:2319: finishing write tid 1 to smithi07694878-8 2021-03-15T22:29:45.586 INFO:tasks.rados.rados.0.smithi076.stdout:2319: finishing write tid 2 to smithi07694878-8 2021-03-15T22:29:45.587 INFO:tasks.rados.rados.0.smithi076.stdout:2319: finishing write tid 3 to smithi07694878-8 2021-03-15T22:29:45.587 INFO:tasks.rados.rados.0.smithi076.stdout:2319: finishing write tid 4 to smithi07694878-8 2021-03-15T22:29:45.587 INFO:tasks.rados.rados.0.smithi076.stdout:2319: finishing write tid 5 to smithi07694878-8 2021-03-15T22:29:45.588 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1207 (ObjNum 799 snap 263 seq_num 799) dirty exists 2021-03-15T22:29:45.588 INFO:tasks.rados.rados.0.smithi076.stdout:2319: left oid 8 (ObjNum 799 snap 263 seq_num 799) 2021-03-15T22:29:45.589 INFO:tasks.rados.rados.0.smithi076.stdout:2320: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:29:45.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:45.598 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:45.599 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:45.700 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:45.724 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:45.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:45.726 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:45.727 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:45.741 INFO:tasks.rados.rados.0.smithi076.stdout:2321: finishing write tid 1 to smithi07694878-6 2021-03-15T22:29:45.742 INFO:tasks.rados.rados.0.smithi076.stdout:2321: finishing write tid 2 to smithi07694878-6 2021-03-15T22:29:45.742 INFO:tasks.rados.rados.0.smithi076.stdout:2321: finishing write tid 3 to smithi07694878-6 2021-03-15T22:29:45.742 INFO:tasks.rados.rados.0.smithi076.stdout:2321: finishing write tid 4 to smithi07694878-6 2021-03-15T22:29:45.743 INFO:tasks.rados.rados.0.smithi076.stdout:2321: finishing write tid 5 to smithi07694878-6 2021-03-15T22:29:45.743 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 1216 (ObjNum 800 snap 263 seq_num 800) dirty exists 2021-03-15T22:29:45.743 INFO:tasks.rados.rados.0.smithi076.stdout:2321: left oid 6 (ObjNum 800 snap 263 seq_num 800) 2021-03-15T22:29:45.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:45 smithi076 conmon[56514]: cluster 2021-03-15T22:29:44.397570+0000 mon.a (mon.0 2021-03-15T22:29:45.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:45 smithi076 conmon[56514]: ) 1738 : cluster [DBG] osdmap e635: 8 total, 7 up, 7 in 2021-03-15T22:29:45.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:45 smithi076 conmon[56514]: cluster 2021-03-15T22:29:45.189590+0000 mon.a (mon.0) 1739 : cluster [WRN] Health check update: Degraded data redundancy: 54/1761 objects degraded (3.066%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:45.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:45 smithi076 conmon[62156]: cluster 2021-03-15T22:29: 2021-03-15T22:29:45.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:45 smithi076 conmon[62156]: 44.397570+0000 mon.a (mon.0) 1738 : cluster [DBG] osdmap e635: 8 total, 7 up, 7 in 2021-03-15T22:29:45.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:45 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:45.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:45 smithi076 conmon[62156]: 03-15T22:29:45.189590+0000 mon.a (mon.0) 1739 2021-03-15T22:29:45.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:45 smithi076 conmon[62156]: : cluster [WRN] Health check update: Degraded data redundancy: 54/1761 objects degraded (3.066%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:45.772 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:45 smithi125 conmon[56739]: cluster 2021-03-15T22:29:44.397570+0000 mon.a (mon. 2021-03-15T22:29:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:45 smithi125 conmon[56739]: 0) 1738 : cluster [DBG] osdmap e635: 8 total, 7 up, 7 in 2021-03-15T22:29:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:45 smithi125 conmon[56739]: cluster 2021-03-15T22:29:45.189590+0000 mon.a (mon.0) 1739 : cluster [WRN] Health check update: Degraded data redundancy: 54/1761 objects degraded (3.066%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:45.827 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:45.850 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:45.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:45.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:45.954 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:45.977 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:45.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:45.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:46.081 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:46.100 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 0, 3] out_osds: [5] dead_osds: [2] live_osds: [0, 1, 3, 4, 6, 7, 5] 2021-03-15T22:29:46.101 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:29:46.101 INFO:tasks.thrashosds.thrasher:Adding osd 5 2021-03-15T22:29:46.102 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd in 5 2021-03-15T22:29:46.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:46.106 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:46.107 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:46.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:46.232 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:46.232 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:46.334 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:46.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:46.359 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:46.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:46.415 INFO:tasks.rados.rados.0.smithi076.stdout:2318: done (4 left) 2021-03-15T22:29:46.415 INFO:tasks.rados.rados.0.smithi076.stdout:2319: done (3 left) 2021-03-15T22:29:46.416 INFO:tasks.rados.rados.0.smithi076.stdout:2320: done (2 left) 2021-03-15T22:29:46.416 INFO:tasks.rados.rados.0.smithi076.stdout:2321: done (1 left) 2021-03-15T22:29:46.416 INFO:tasks.rados.rados.0.smithi076.stdout:2322: done (0 left) 2021-03-15T22:29:46.417 INFO:tasks.rados.rados.0.smithi076.stdout:2323: snap_create 2021-03-15T22:29:46.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:46.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:46.485 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:46.486 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:46.515 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:29:46.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:46 smithi076 conmon[56514]: cluster 2021-03-15T22:29:45.132474+0000 mgr.y (mgr.14144 2021-03-15T22:29:46.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:46 smithi076 conmon[56514]: ) 810 : cluster [DBG] pgmap v1097: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 560 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 25 op/s; 55/1764 objects degraded (3.118%); 188/1764 objects misplaced (10.658%) 2021-03-15T22:29:46.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:46 smithi076 conmon[56514]: cluster 2021-03-15T22:29:45.403871+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e636: 8 total, 7 up, 7 in 2021-03-15T22:29:46.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:46 smithi076 conmon[62156]: cluster 2021-03-15T22:29: 2021-03-15T22:29:46.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:46 smithi076 conmon[62156]: 45.132474+0000 mgr.y (mgr.14144) 810 : cluster [DBG] pgmap v1097: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 560 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 25 op/s; 55/1764 objects degraded (3.118%); 188/1764 objects misplaced (10.658%) 2021-03-15T22:29:46.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:46 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:29:46.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:46 smithi076 conmon[62156]: T22:29:45.403871+0000 2021-03-15T22:29:46.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:46 smithi076 conmon[62156]: mon.a (mon.0) 1740 2021-03-15T22:29:46.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:46 smithi076 conmon[62156]: : cluster [DBG] osdmap e636: 8 total, 7 up, 7 in 2021-03-15T22:29:46.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:46 smithi125 conmon[56739]: cluster 2021-03-15T22:29:45.132474+0000 mgr.y (mgr.14144) 810 : cluster [DBG] pgmap v1097: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 560 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 4.1 MiB/s wr, 25 op/s; 55/1764 objects degraded (3.118%); 188/1764 objects misplaced (10.658%) 2021-03-15T22:29:46.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:46 smithi125 conmon[56739]: cluster 2021-03-15T22:29:45.403871+0000 mon.a (mon.0) 1740 : cluster [DBG] osdmap e636: 8 total, 7 up, 7 in 2021-03-15T22:29:46.588 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:46.610 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:46.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:46.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:46.714 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:46.736 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:46.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:46.738 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:46.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:46.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:46.862 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:46.863 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:46.952 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:46.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:46.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:46.989 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:46.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:46.996 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:47.090 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:47.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:47.113 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:47.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:47.215 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:47.238 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:47.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:47.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:47.341 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:47.349 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:29:47.349 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:29:47.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:47.366 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:47.367 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:47.417 INFO:tasks.rados.rados.0.smithi076.stdout:2323: done (0 left) 2021-03-15T22:29:47.418 INFO:tasks.rados.rados.0.smithi076.stdout:2324: snap_create 2021-03-15T22:29:47.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:47.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:47.492 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:47.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:47.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:47.599 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:47 smithi076 conmon[56514]: cluster 2021-03-15T22:29:46.409351+0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e637: 8 total, 7 up, 7 in 2021-03-15T22:29:47.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:47 smithi076 conmon[62156]: cluster 2021-03-15T22:29:46.409351 2021-03-15T22:29:47.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:47 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1741 : cluster [DBG] osdmap e637: 8 total, 7 up, 7 in 2021-03-15T22:29:47.617 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:47.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:47.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:47.719 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:47.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:47.744 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:47.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:47.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:47 smithi125 conmon[56739]: cluster 2021-03-15T22:29:46.409351+0000 mon.a (mon 2021-03-15T22:29:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:47 smithi125 conmon[56739]: .0) 1741 : cluster [DBG] osdmap e637: 8 total, 7 up, 7 in 2021-03-15T22:29:47.846 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:47.867 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:47.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:47.869 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:47.970 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:47.993 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:47.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:47.994 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:47.999 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-15T22:29:48.095 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:48.119 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:48.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:48.121 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:48.222 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:48.244 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:48.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:48.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:48.280 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:48.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:48.368 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:48.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:48.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:48.420 INFO:tasks.rados.rados.0.smithi076.stdout:2324: done (0 left) 2021-03-15T22:29:48.421 INFO:tasks.rados.rados.0.smithi076.stdout:2325: snap_remove snap 233 2021-03-15T22:29:48.423 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:29:48.472 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:48.493 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:48.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:48.495 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:48.596 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:48.618 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:48.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:48.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:48.674 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:48 smithi125 conmon[56739]: cluster 2021-03-15T22:29:47.133060+0000 mgr.y (mgr.14144) 811 : cluster [DBG] pgmap v1100: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 530 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 3.3 MiB/s wr, 32 op/s; 55/1734 objects degraded (3.172%); 186/1734 objects misplaced (10.727%) 2021-03-15T22:29:48.675 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:48 smithi125 conmon[56739]: cluster 2021-03-15T22:29:47.414547+0000 mon.a (mon.0) 1742 : cluster [DBG] osdmap e638: 8 total, 7 up, 7 in 2021-03-15T22:29:48.676 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:48 smithi125 conmon[56739]: audit 2021-03-15T22:29:47.959666+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? 172.21.15.76:0/1097404987' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:48.720 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:48.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:48.746 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:48.747 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:48.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:48 smithi076 conmon[56514]: cluster 2021-03-15T22:29:47.133060+0000 2021-03-15T22:29:48.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:48 smithi076 conmon[56514]: mgr.y (mgr.14144) 811 : cluster [DBG] pgmap v1100: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 530 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 3.3 MiB/s wr, 32 op/s; 55/1734 objects degraded (3.172%); 186/1734 objects misplaced (10.727%) 2021-03-15T22:29:48.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:48 smithi076 conmon[56514]: cluster 2021-03-15T22:29:47.414547+0000 mon.a (mon.0) 1742 : cluster [DBG] osdmap e638: 8 total, 7 up, 7 in 2021-03-15T22:29:48.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:48 smithi076 conmon[56514]: audit 2021-03-15T22:29:47.959666+0000 mon.a (mon.0) 1743 : audit [INF] from='client.? 172.21.15.76:0/1097404987' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:48.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:48 smithi076 conmon[62156]: cluster 2021-03-15T22:29:47.133060 2021-03-15T22:29:48.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:48 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 811 : cluster [DBG] pgmap v1100: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 530 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 3.3 MiB/s wr, 32 op/s; 55/1734 objects degraded (3.172%); 186/1734 objects misplaced (10.727%) 2021-03-15T22:29:48.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:48 smithi076 conmon[62156]: cluster 2021-03-15T22:29:47.414547+0000 mon.a (mon.0) 1742 : cluster [DBG] osdmap e638: 8 total, 7 up, 7 in 2021-03-15T22:29:48.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:48 smithi076 conmon[62156]: audit 2021-03-15T22:29:47.959666+0000 mon.a ( 2021-03-15T22:29:48.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:48 smithi076 conmon[62156]: mon.0) 1743 : audit [INF] from='client.? 172.21.15.76:0/1097404987' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:29:48.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:48.874 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:48.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:48.875 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:48.976 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:48.999 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:49.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:49.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:49.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:49.123 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:49.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:49.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:49.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:49.248 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:49.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:49.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:49.351 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:49.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:49.378 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:49.378 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:49.440 INFO:tasks.rados.rados.0.smithi076.stdout:2325: done (0 left) 2021-03-15T22:29:49.441 INFO:tasks.rados.rados.0.smithi076.stdout:2326: snap_remove snap 202 2021-03-15T22:29:49.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:49.502 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:49.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:49.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:49.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:49 smithi125 conmon[56739]: cluster 2021-03-15T22:29:48.411332+0000 mon.a (mon. 2021-03-15T22:29:49.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:49 smithi125 conmon[56739]: 0) 1744 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:49.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:49 smithi125 conmon[56739]: audit 2021-03-15T22:29:48.416930+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? 172.21.15.76:0/1097404987' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:49.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:49 smithi125 conmon[56739]: cluster 2021-03-15T22:29:48.416996+0000 mon.a (mon.0) 1746 : cluster [DBG] osdmap e639: 8 total, 7 up, 7 in 2021-03-15T22:29:49.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:49 smithi125 conmon[56739]: audit 2021-03-15T22:29:48.622799+0000 mon.a (mon.0) 1747 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:29:49.604 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:49.626 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:49.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:49.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:49.698 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:49 smithi076 conmon[56514]: cluster 2021-03-15T22:29:49.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:49 smithi076 conmon[56514]: 2021-03-15T22:29:48.411332+0000 mon.a (mon.0) 1744 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:49.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:49 smithi076 conmon[56514]: audit 2021-03-15T22:29:48.416930+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? 172.21.15.76:0/1097404987' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:49.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:49 smithi076 conmon[56514]: cluster 2021-03-15T22:29:48.416996+0000 mon.a (mon.0) 1746 : cluster [DBG] osdmap e639: 8 total, 7 up, 7 in 2021-03-15T22:29:49.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:49 smithi076 conmon[56514]: audit 2021-03-15T22:29:48.622799+0000 mon.a (mon.0) 1747 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:29:49.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:49 smithi076 conmon[62156]: cluster 2021-03-15T22:29:48.411332+0000 mon.a (mon.0) 1744 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:49.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:49 smithi076 conmon[62156]: audit 2021-03-15T22:29:48.416930+0000 mon.a (mon.0) 1745 : audit [INF] from='client.? 172.21.15.76:0/1097404987' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:29:49.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:49 smithi076 conmon[62156]: cluster 2021-03-15T22:29:48.416996+0000 mon.a (mon.0) 1746 : cluster [DBG] osdmap e639: 8 total, 7 up, 7 in 2021-03-15T22:29:49.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:49 smithi076 conmon[62156]: audit 2021-03-15T22:29:48.622799+0000 mon.a (mon.0) 1747 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:29:49.729 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:49.752 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:49.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:49.754 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:49.855 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:49.879 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:49.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:49.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:49.883 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:29:49.884 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:29:49.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:50.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:50.005 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:50.006 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:50.107 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:50.137 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:50.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:50.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:50.205 INFO:tasks.rados.rados.0.smithi076.stdout:2326: done (0 left) 2021-03-15T22:29:50.205 INFO:tasks.rados.rados.0.smithi076.stdout:2327: read oid 7 snap 252 2021-03-15T22:29:50.205 INFO:tasks.rados.rados.0.smithi076.stdout:2327: expect (ObjNum 726 snap 237 seq_num 726) 2021-03-15T22:29:50.205 INFO:tasks.rados.rados.0.smithi076.stdout:2328: copy_from oid 45 from oid 30 current snap is 266 2021-03-15T22:29:50.206 INFO:tasks.rados.rados.0.smithi076.stdout:2329: rollback oid 8 current snap is 266 2021-03-15T22:29:50.206 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 220 2021-03-15T22:29:50.206 INFO:tasks.rados.rados.0.smithi076.stdout:2330: read oid 10 snap -1 2021-03-15T22:29:50.206 INFO:tasks.rados.rados.0.smithi076.stdout:2330: expect deleted 2021-03-15T22:29:50.207 INFO:tasks.rados.rados.0.smithi076.stdout:2331: write (excl) oid 11 current snap is 266 2021-03-15T22:29:50.207 INFO:tasks.rados.rados.0.smithi076.stdout:2331: seq_num 801 ranges {662407=491514,1855959=571609,2846876=615531} 2021-03-15T22:29:50.207 INFO:tasks.rados.rados.0.smithi076.stdout:2331: writing smithi07694878-11 from 662407 to 1153921 tid 1 2021-03-15T22:29:50.218 INFO:tasks.rados.rados.0.smithi076.stdout:2331: writing smithi07694878-11 from 1855959 to 2427568 tid 2 2021-03-15T22:29:50.229 INFO:tasks.rados.rados.0.smithi076.stdout:2331: writing smithi07694878-11 from 2846876 to 3462407 tid 3 2021-03-15T22:29:50.229 INFO:tasks.rados.rados.0.smithi076.stdout:2332: copy_from oid 44 from oid 42 current snap is 266 2021-03-15T22:29:50.230 INFO:tasks.rados.rados.0.smithi076.stdout:2330: done (5 left) 2021-03-15T22:29:50.230 INFO:tasks.rados.rados.0.smithi076.stdout:2333: snap_create 2021-03-15T22:29:50.231 INFO:teuthology.orchestra.run.smithi076.stderr:marked in osd.5. 2021-03-15T22:29:50.239 INFO:tasks.rados.rados.0.smithi076.stdout:2329: finishing rollback tid 0 to smithi07694878-8 2021-03-15T22:29:50.239 INFO:tasks.rados.rados.0.smithi076.stdout:2329: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:29:50.240 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:50.241 INFO:tasks.rados.rados.0.smithi076.stdout:2329: finishing rollback tid 2 to smithi07694878-8 2021-03-15T22:29:50.241 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1210 (ObjNum 649 snap 201 seq_num 649) dirty exists 2021-03-15T22:29:50.243 INFO:tasks.rados.rados.0.smithi076.stdout:2331: finishing write tid 1 to smithi07694878-11 2021-03-15T22:29:50.250 INFO:tasks.rados.rados.0.smithi076.stdout:2331: finishing write tid 2 to smithi07694878-11 2021-03-15T22:29:50.259 INFO:tasks.rados.rados.0.smithi076.stdout:2328: finishing copy_from to smithi07694878-45 2021-03-15T22:29:50.259 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 1586 (ObjNum 774 snap 256 seq_num 774) dirty exists 2021-03-15T22:29:50.260 INFO:tasks.rados.rados.0.smithi076.stdout:2328: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:29:50.262 INFO:tasks.rados.rados.0.smithi076.stdout:2331: finishing write tid 3 to smithi07694878-11 2021-03-15T22:29:50.262 INFO:tasks.rados.rados.0.smithi076.stdout:2331: finishing write tid 4 to smithi07694878-11 2021-03-15T22:29:50.263 INFO:tasks.rados.rados.0.smithi076.stdout:2331: finishing write tid 5 to smithi07694878-11 2021-03-15T22:29:50.263 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1537 (ObjNum 801 snap 266 seq_num 801) dirty exists 2021-03-15T22:29:50.263 INFO:tasks.rados.rados.0.smithi076.stdout:2331: left oid 11 (ObjNum 801 snap 266 seq_num 801) 2021-03-15T22:29:50.264 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:50.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:50.266 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:50.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[56514]: debug 2021-03-15T22:29:50.135+0000 7f0537c88700 -1 mon.a@0(leader).osd e640 definitely_dead 0 2021-03-15T22:29:50.286 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:29:50.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:29:49 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:29:50.304 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:29:50.324 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:29:50.344 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:29:50.362 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:29:50.367 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:50.381 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:29:50.389 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:50.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:50.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:50.405 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:29:50.405 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:29:50.406 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:29:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:29:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:29:50.407 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:29:50.408 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:29:50.408 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:29:50.408 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:29:50.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:50.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:50.517 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:50.517 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:50.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:50.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:50.643 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:50.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:50.710 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: cluster 2021-03-15T22:29:49.133541+0000 mgr.y (mgr.14144) 812 2021-03-15T22:29:50.711 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: : cluster [DBG] pgmap v1103: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 533 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 2.2 MiB/s wr, 27 op/s; 55/1731 objects degraded (3.177%); 185/1731 objects misplaced (10.687%) 2021-03-15T22:29:50.711 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: cluster 2021-03-15T22:29:49.432856+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e640: 8 total, 7 up, 7 in 2021-03-15T22:29:50.711 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: audit 2021-03-15T22:29:49.956265+0000 mgr.y (mgr.14144) 813 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:50.711 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: audit 2021-03-15T22:29:50.136532+0000 mon.a (mon.0) 1749 : audit [DBG] from='client.? 172.21.15.76:0/3063060067' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2021-03-15T22:29:50.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: cluster 2021-03-15T22:29:50.190833+0000 mon.a (mon.0) 1750 : cluster [WRN] Health check update: Degraded data redundancy: 55/1731 objects degraded (3.177%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:50.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: audit 2021-03-15T22:29:50.196820+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? 172.21.15.76:0/3063060067' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2021-03-15T22:29:50.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: cluster 2021-03-15T22:29:50.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: 2021-03-15T22:29:50.196884+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e641: 8 total, 7 up, 8 in 2021-03-15T22:29:50.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:29:50.713 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: 03-15T22:29:50.266151+0000 mon.b (mon.2 2021-03-15T22:29:50.713 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:50 smithi125 conmon[56739]: ) 28 : audit [DBG] from='client.? 172.21.15.76:0/1169784509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:50.735 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-15T22:29:50.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:50.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[56514]: cluster 2021-03-15T22:29:49.133541+0000 mgr.y (mgr.14144) 812 : cluster [DBG] pgmap v1103: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 533 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 2.2 MiB/s wr, 27 op/s; 55/1731 objects degraded (3.177%); 185/1731 objects misplaced (10.687%) 2021-03-15T22:29:50.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[56514]: cluster 2021-03-15T22:29:49.432856+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e640: 8 total, 7 up, 7 in 2021-03-15T22:29:50.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[56514]: audit 2021-03-15T22:29:49.956265+0000 mgr.y (mgr.14144) 813 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:50.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[56514]: audit 2021-03-15T22:29:50.136532+0000 mon.a (mon.0) 1749 : audit [DBG] from='client.? 172.21.15.76:0/3063060067' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2021-03-15T22:29:50.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[56514]: cluster 2021-03-15T22:29:50.190833+0000 mon.a (mon.0) 1750 : cluster [WRN] Health check update: Degraded data redundancy: 55/1731 objects degraded (3.177%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:50.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[56514]: audit 2021-03-15T22:29:50.196820+0000 mon.a (mon.0) 1751 : audit [INF] from='client.? 172.21.15.76:0/3063060067' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2021-03-15T22:29:50.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[56514]: cluster 2021-03-15T22:29:50.196884+0000 mon.a (mon.0) 1752 : cluster [DBG] osdmap e641: 8 total, 7 up, 8 in 2021-03-15T22:29:50.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[56514]: audit 2021-03-15T22:29:50.266151+0000 mon.b (mon.2) 28 : audit [DBG] from='client.? 172.21.15.76:0/1169784509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:50.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: cluster 2021-03-15T22:29:49.133541 2021-03-15T22:29:50.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 812 : cluster [DBG] pgmap v1103: 138 pgs: 41 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 83 active+clean; 533 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 2.2 MiB/s wr, 27 op/s; 55/1731 objects degraded (3.177%); 185/1731 objects misplaced (10.687%) 2021-03-15T22:29:50.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: cluster 2021-03-15T22:29:49.432856+0000 mon.a (mon.0) 1748 : cluster [DBG] osdmap e640: 8 total, 7 up, 7 in 2021-03-15T22:29:50.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: audit 2021-03-15T22:29:49.956265+0000 mgr.y (mgr.14144) 813 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:29:50.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: audit 2021-03-15T22:29:50.136532+0000 mon.a (mon.0) 1749 2021-03-15T22:29:50.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: : audit [DBG] from='client.? 172.21.15.76:0/3063060067' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["5"]}]: dispatch 2021-03-15T22:29:50.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:29:50.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: -15T22:29:50.190833+0000 mon.a (mon.0 2021-03-15T22:29:50.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: ) 1750 : cluster [WRN] Health check update: Degraded data redundancy: 55/1731 objects degraded (3.177%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:50.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:29:50.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: :29:50.196820+0000 mon.a (mon.0) 2021-03-15T22:29:50.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: 1751 : audit [INF] from='client.? 172.21.15.76:0/3063060067' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["5"]}]': finished 2021-03-15T22:29:50.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: cluster 2021-03-15T22:29:50.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: 2021-03-15T22:29 2021-03-15T22:29:50.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: :50.196884+0000 mon.a (mon.0) 2021-03-15T22:29:50.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: 1752 : cluster [DBG] osdmap e641: 8 total, 7 up, 8 in 2021-03-15T22:29:50.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:29:50.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: 15T22:29:50.266151+0000 2021-03-15T22:29:50.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:50 smithi076 conmon[62156]: mon.b (mon.2) 28 : audit [DBG] from='client.? 172.21.15.76:0/1169784509' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:50.800 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:50.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:50.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:50.903 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:50.925 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:50.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:50.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:51.028 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:51.050 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:51.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:51.052 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:51.096 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:29:51.153 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:51.175 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:51.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:51.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:51.206 INFO:tasks.rados.rados.0.smithi076.stdout:2328: done (5 left) 2021-03-15T22:29:51.207 INFO:tasks.rados.rados.0.smithi076.stdout:2329: done (4 left) 2021-03-15T22:29:51.207 INFO:tasks.rados.rados.0.smithi076.stdout:2331: done (3 left) 2021-03-15T22:29:51.207 INFO:tasks.rados.rados.0.smithi076.stdout:2333: done (2 left) 2021-03-15T22:29:51.208 INFO:tasks.rados.rados.0.smithi076.stdout:2334: snap_create 2021-03-15T22:29:51.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:51.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:51.309 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:51.309 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:51.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:51.416 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:29:51.432 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:51.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:51.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:51.495 INFO:tasks.thrashosds.thrasher:Added osd 5 2021-03-15T22:29:51.529 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:29:51.530 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:29:51.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:51.561 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:51.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:51.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:51.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:51.690 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:51.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:51.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:51.792 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:51.795 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:51.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:51.825 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:51.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:51.926 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:51.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:51.949 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:51.949 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:52.051 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:52.074 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:52.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:52.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:52.176 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:52.210 INFO:tasks.rados.rados.0.smithi076.stdout:2334: done (2 left) 2021-03-15T22:29:52.211 INFO:tasks.rados.rados.0.smithi076.stdout:2335: read oid 35 snap -1 2021-03-15T22:29:52.211 INFO:tasks.rados.rados.0.smithi076.stdout:2335: expect (ObjNum 785 snap 261 seq_num 785) 2021-03-15T22:29:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:52.213 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:52.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:52.214 INFO:tasks.rados.rados.0.smithi076.stdout:2336: read oid 37 snap -1 2021-03-15T22:29:52.214 INFO:tasks.rados.rados.0.smithi076.stdout:2336: expect (ObjNum 751 snap 245 seq_num 751) 2021-03-15T22:29:52.214 INFO:tasks.rados.rados.0.smithi076.stdout:2337: read oid 3 snap 261 2021-03-15T22:29:52.214 INFO:tasks.rados.rados.0.smithi076.stdout:2337: expect deleted 2021-03-15T22:29:52.215 INFO:tasks.rados.rados.0.smithi076.stdout:2338: copy_from oid 24 from oid 21 current snap is 268 2021-03-15T22:29:52.215 INFO:tasks.rados.rados.0.smithi076.stdout:2339: write (excl) oid 6 current snap is 268 2021-03-15T22:29:52.215 INFO:tasks.rados.rados.0.smithi076.stdout:2339: seq_num 802 ranges {679214=566763,1798088=715267,3157946=321268} 2021-03-15T22:29:52.218 INFO:tasks.rados.rados.0.smithi076.stdout:2339: writing smithi07694878-6 from 679214 to 1245977 tid 1 2021-03-15T22:29:52.227 INFO:tasks.rados.rados.0.smithi076.stdout:2339: writing smithi07694878-6 from 1798088 to 2513355 tid 2 2021-03-15T22:29:52.234 INFO:tasks.rados.rados.0.smithi076.stdout:2339: writing smithi07694878-6 from 3157946 to 3479214 tid 3 2021-03-15T22:29:52.234 INFO:tasks.rados.rados.0.smithi076.stdout:2340: write (excl) oid 1 current snap is 268 2021-03-15T22:29:52.234 INFO:tasks.rados.rados.0.smithi076.stdout:2339: finishing write tid 1 to smithi07694878-6 2021-03-15T22:29:52.235 INFO:tasks.rados.rados.0.smithi076.stdout:2327: expect (ObjNum 726 snap 237 seq_num 726) 2021-03-15T22:29:52.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:52 smithi076 conmon[56514]: cluster 2021-03-15T22:29:52.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:52 smithi076 conmon[56514]: 2021-03-15T22:29:52.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:52 smithi076 conmon[56514]: 2021-03-15T22:29:51. 2021-03-15T22:29:52.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:52 smithi076 conmon[56514]: 134345+0000 mgr.y (mgr.14144) 814 : cluster [DBG] pgmap v1106: 138 pgs: 4 remapped+peering, 7 peering, 36 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 77 active+clean; 535 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 4.4 MiB/s rd, 756 KiB/s wr, 7 op/s; 55/1734 objects degraded (3.172%); 178/1734 objects misplaced (10.265%) 2021-03-15T22:29:52.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:52 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:29:52.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:52 smithi076 conmon[56514]: 03-15T22:29:51.198146+0000 mon.a (mon 2021-03-15T22:29:52.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:52 smithi076 conmon[56514]: .0) 1753 : cluster [DBG] osdmap e642: 8 total, 7 up, 8 in 2021-03-15T22:29:52.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:52 smithi076 conmon[62156]: cluster 2021-03-15T22:29:51.134345+0000 2021-03-15T22:29:52.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:52 smithi076 conmon[62156]: mgr.y (mgr.14144) 814 : cluster [DBG] pgmap v1106: 138 pgs: 4 remapped+peering, 7 peering, 36 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 77 active+clean; 535 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 4.4 MiB/s rd, 756 KiB/s wr, 7 op/s; 55/1734 objects degraded (3.172%); 178/1734 objects misplaced (10.265%) 2021-03-15T22:29:52.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:52 smithi076 conmon[62156]: cluster 2021-03-15T22:29:51.198146+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e642: 8 total, 7 up, 8 in 2021-03-15T22:29:52.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:52.320 INFO:tasks.rados.rados.0.smithi076.stdout:2340: seq_num 803 ranges {696021=642012,1856570=575278,3185369=310652} 2021-03-15T22:29:52.327 INFO:tasks.rados.rados.0.smithi076.stdout:2340: writing smithi07694878-1 from 696021 to 1338033 tid 1 2021-03-15T22:29:52.334 INFO:tasks.rados.rados.0.smithi076.stdout:2340: writing smithi07694878-1 from 1856570 to 2431848 tid 2 2021-03-15T22:29:52.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:52.339 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:52.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:52.340 INFO:tasks.rados.rados.0.smithi076.stdout:2340: writing smithi07694878-1 from 3185369 to 3496021 tid 3 2021-03-15T22:29:52.340 INFO:tasks.rados.rados.0.smithi076.stdout:2327: done (7 left) 2021-03-15T22:29:52.340 INFO:tasks.rados.rados.0.smithi076.stdout:2337: done (6 left) 2021-03-15T22:29:52.341 INFO:tasks.rados.rados.0.smithi076.stdout:2341: snap_remove snap 172 2021-03-15T22:29:52.440 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:52.462 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:52.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:52.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:52.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:52 smithi125 conmon[56739]: cluster 2021-03-15T22:29:51.134345+0000 mgr.y (mgr.14144) 814 : cluster [DBG] pgmap v1106: 138 pgs: 4 remapped+peering, 7 peering, 36 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 77 active+clean; 535 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 4.4 MiB/s rd, 756 KiB/s wr, 7 op/s; 55/1734 objects degraded (3.172%); 178/1734 objects misplaced (10.265%) 2021-03-15T22:29:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:52 smithi125 conmon[56739]: cluster 2021-03-15T22:29:51.198146+0000 mon.a (mon.0) 1753 : cluster [DBG] osdmap e642: 8 total, 7 up, 8 in 2021-03-15T22:29:52.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:52.588 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:52.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:52.590 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:52.691 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:52.713 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:52.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:52.714 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:52.815 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:52.837 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:52.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:52.838 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:52.939 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:52.961 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:52.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:52.963 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:53.064 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:53.087 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:53.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:53.089 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:53.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:53.213 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:53.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:53.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:53.237 INFO:tasks.rados.rados.0.smithi076.stdout:2341: done (6 left) 2021-03-15T22:29:53.238 INFO:tasks.rados.rados.0.smithi076.stdout:2342: delete oid 11 current snap is 268 2021-03-15T22:29:53.238 INFO:tasks.rados.rados.0.smithi076.stdout:2335: expect (ObjNum 785 snap 261 seq_num 785) 2021-03-15T22:29:53.267 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:53 smithi076 conmon[62156]: cluster 2021-03-15T22:29:52.199551+0000 mon.a (mon.0) 1754 : cluster [DBG] osdmap e643: 8 total, 7 up, 8 in 2021-03-15T22:29:53.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:53 smithi076 conmon[56514]: cluster 2021-03-15T22:29:52.199551+0000 mon.a ( 2021-03-15T22:29:53.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:53 smithi076 conmon[56514]: mon.0) 1754 : cluster [DBG] osdmap e643: 8 total, 7 up, 8 in 2021-03-15T22:29:53.315 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:53.338 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:53.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:53.340 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:53.389 INFO:tasks.rados.rados.0.smithi076.stdout:2335: done (6 left) 2021-03-15T22:29:53.390 INFO:tasks.rados.rados.0.smithi076.stdout:2342: done (5 left) 2021-03-15T22:29:53.390 INFO:tasks.rados.rados.0.smithi076.stdout:2343: snap_remove snap 261 2021-03-15T22:29:53.441 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:53.448 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:53 smithi125 conmon[56739]: cluster 2021-03-15T22:29:52.199551+0000 2021-03-15T22:29:53.448 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:53 smithi125 conmon[56739]: mon.a (mon.0) 1754 : cluster [DBG] osdmap e643: 8 total, 7 up, 8 in 2021-03-15T22:29:53.459 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:29:53.459 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:29:53.460 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:29:53.460 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:29:53.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.460 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "MOSDPGPush(7.a 618/617 [PushOp(7:5bc7f316:::smithi07694878-38:1b4, version: 571'778, data_included: [2481817~146208,3258689~649107], data_size: 795315, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:5bc7f316:::smithi07694878-38:1b4@571'778, size: 3949777, copy_subset: [749777~722868,2156109~471916,3258689~691088], clone_subset: {7:5bc7f316:::smithi07694878-38:1ae=[0~749777,1472645~683464,2628025~630664]}, snapset: 1c0=[]:{10c=[10c,102],176=[176,175,173,170,16e,16d,168],181=[181,180,17f,17e,17a],1a2=[1a2,1a1,1a0,19f,19c,19b,19a,199,195,194,193,190,18f,18a,184],1ae=[1ae,1ac,1a9,1a7,1a6,1a4,1a3],1b4=[1b4,1b3,1b1,1b0],1c0=[1c0,1be,1bd,1bb,1b7,1b6]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3949777, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2481817, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-15T22:29:53.461 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:27.260617+0000", 2021-03-15T22:29:53.461 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 26.194941668999999, 2021-03-15T22:29:53.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070113240000000002, 2021-03-15T22:29:53.461 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.461 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:29:53.462 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.462 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.463 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:27.260617+0000", 2021-03-15T22:29:53.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.464 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.464 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.464 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:27.260617+0000", 2021-03-15T22:29:53.465 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021832290000000001 2021-03-15T22:29:53.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.465 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.465 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:27.262800+0000", 2021-03-15T22:29:53.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4236999999999999e-05 2021-03-15T22:29:53.466 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.466 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.466 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.467 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:27.262814+0000", 2021-03-15T22:29:53.467 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9040000000000001e-05 2021-03-15T22:29:53.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.467 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:27.262833+0000", 2021-03-15T22:29:53.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0677e-05 2021-03-15T22:29:53.468 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.468 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.469 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.469 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:27.262844+0000", 2021-03-15T22:29:53.469 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6129000000000002e-05 2021-03-15T22:29:53.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.470 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.470 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.470 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:27.262920+0000", 2021-03-15T22:29:53.470 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2490000000000001e-06 2021-03-15T22:29:53.470 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.471 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:27.262925+0000", 2021-03-15T22:29:53.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047027630000000004 2021-03-15T22:29:53.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:27.267628+0000", 2021-03-15T22:29:53.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070113240000000002 2021-03-15T22:29:53.473 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.473 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.473 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.474 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.1.0:743 7.0 e623/617 7:00c03349:::smithi07694878-40:1c7 v 623'1643, mlcod=620'1639)", 2021-03-15T22:29:53.474 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:32.216634+0000", 2021-03-15T22:29:53.474 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 21.238924601000001, 2021-03-15T22:29:53.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0097424629999999998, 2021-03-15T22:29:53.475 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.475 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:29:53.475 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216634+0000", 2021-03-15T22:29:53.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.476 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.477 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.477 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216634+0000", 2021-03-15T22:29:53.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2570000000000004e-06 2021-03-15T22:29:53.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.477 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.478 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216640+0000", 2021-03-15T22:29:53.478 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.024e-06 2021-03-15T22:29:53.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.479 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216642+0000", 2021-03-15T22:29:53.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6389999999999998e-06 2021-03-15T22:29:53.479 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216651+0000", 2021-03-15T22:29:53.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9669999999999998e-06 2021-03-15T22:29:53.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.482 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216657+0000", 2021-03-15T22:29:53.482 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1328000000000006e-05 2021-03-15T22:29:53.482 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.482 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.483 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.483 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216718+0000", 2021-03-15T22:29:53.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.156e-05 2021-03-15T22:29:53.483 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.484 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.484 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.484 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216739+0000", 2021-03-15T22:29:53.484 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0096119250000000003 2021-03-15T22:29:53.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.485 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:29:53.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.226351+0000", 2021-03-15T22:29:53.485 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4763e-05 2021-03-15T22:29:53.485 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.486 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.486 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.226376+0000", 2021-03-15T22:29:53.486 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0097424629999999998 2021-03-15T22:29:53.487 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.487 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.487 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.487 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.487 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.488 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.1.0:744 7.0 e623/617 7:08414d91:::smithi07694878-42:1b7 v 623'1645, mlcod=620'1639)", 2021-03-15T22:29:53.488 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:32.217680+0000", 2021-03-15T22:29:53.488 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 21.237878757000001, 2021-03-15T22:29:53.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0087403770000000006, 2021-03-15T22:29:53.489 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.489 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:29:53.489 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.489 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.489 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.490 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.217680+0000", 2021-03-15T22:29:53.490 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.490 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.490 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.491 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.491 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.217680+0000", 2021-03-15T22:29:53.491 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6109999999999998e-06 2021-03-15T22:29:53.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.491 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.492 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.492 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.217683+0000", 2021-03-15T22:29:53.492 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6189999999999998e-06 2021-03-15T22:29:53.492 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.492 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.493 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.493 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.217686+0000", 2021-03-15T22:29:53.494 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6220000000000001e-05 2021-03-15T22:29:53.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.495 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.495 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.495 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.217702+0000", 2021-03-15T22:29:53.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8479999999999997e-06 2021-03-15T22:29:53.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.496 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.217708+0000", 2021-03-15T22:29:53.497 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3078000000000002e-05 2021-03-15T22:29:53.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.217761+0000", 2021-03-15T22:29:53.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016628699999999999 2021-03-15T22:29:53.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.499 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.500 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.217927+0000", 2021-03-15T22:29:53.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0084511240000000008 2021-03-15T22:29:53.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:29:53.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.226378+0000", 2021-03-15T22:29:53.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.159e-05 2021-03-15T22:29:53.502 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.503 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.226420+0000", 2021-03-15T22:29:53.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0087403770000000006 2021-03-15T22:29:53.503 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.504 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.504 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.505 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.505 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.3.0:1326 7.a e623/617 7:5bc7f316:::smithi07694878-38:1b4 v 623'857, mlcod=623'853)", 2021-03-15T22:29:53.505 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:32.219398+0000", 2021-03-15T22:29:53.505 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 21.236160489, 2021-03-15T22:29:53.506 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060196449999999997, 2021-03-15T22:29:53.506 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.506 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:29:53.506 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.507 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.507 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.507 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.219398+0000", 2021-03-15T22:29:53.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.508 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.508 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.219398+0000", 2021-03-15T22:29:53.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4700000000000004e-06 2021-03-15T22:29:53.509 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.509 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.510 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.510 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.219402+0000", 2021-03-15T22:29:53.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.179e-06 2021-03-15T22:29:53.510 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.511 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.511 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.511 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.219405+0000", 2021-03-15T22:29:53.511 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5775999999999999e-05 2021-03-15T22:29:53.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.512 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.512 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.219421+0000", 2021-03-15T22:29:53.513 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4030000000000004e-06 2021-03-15T22:29:53.513 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.513 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.514 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.514 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.219428+0000", 2021-03-15T22:29:53.514 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8126000000000001e-05 2021-03-15T22:29:53.514 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.515 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.515 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.515 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.219466+0000", 2021-03-15T22:29:53.516 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1804e-05 2021-03-15T22:29:53.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.516 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.517 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.219488+0000", 2021-03-15T22:29:53.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0058955379999999996 2021-03-15T22:29:53.517 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.517 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.518 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:29:53.518 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.225383+0000", 2021-03-15T22:29:53.518 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4348999999999999e-05 2021-03-15T22:29:53.518 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.519 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.519 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.519 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.225417+0000", 2021-03-15T22:29:53.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060196449999999997 2021-03-15T22:29:53.520 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.520 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.520 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.521 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.0.0:1376 7.e e623/617 7:7ac381e0:::smithi07694878-24:1b3 v 623'196, mlcod=618'193)", 2021-03-15T22:29:53.521 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:32.245029+0000", 2021-03-15T22:29:53.522 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 21.210529265000002, 2021-03-15T22:29:53.522 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0091366350000000006, 2021-03-15T22:29:53.522 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.522 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:29:53.523 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.245029+0000", 2021-03-15T22:29:53.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.525 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.245029+0000", 2021-03-15T22:29:53.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1749999999999998e-06 2021-03-15T22:29:53.525 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.526 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.526 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.245033+0000", 2021-03-15T22:29:53.526 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9450000000000002e-06 2021-03-15T22:29:53.527 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.527 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.527 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.527 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.245036+0000", 2021-03-15T22:29:53.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5875999999999998e-05 2021-03-15T22:29:53.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.528 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.529 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.245052+0000", 2021-03-15T22:29:53.529 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0430000000000008e-06 2021-03-15T22:29:53.529 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.530 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.530 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.530 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.245060+0000", 2021-03-15T22:29:53.530 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7444e-05 2021-03-15T22:29:53.531 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.531 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.531 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.245127+0000", 2021-03-15T22:29:53.532 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5818999999999997e-05 2021-03-15T22:29:53.532 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.532 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.533 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.245163+0000", 2021-03-15T22:29:53.533 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0089581469999999996 2021-03-15T22:29:53.533 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.534 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.534 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:29:53.534 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.254121+0000", 2021-03-15T22:29:53.535 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4186e-05 2021-03-15T22:29:53.535 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.535 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.535 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.536 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.254166+0000", 2021-03-15T22:29:53.536 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0091366350000000006 2021-03-15T22:29:53.536 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.537 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.537 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.537 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.538 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(osd.7.52:26 7.0 7:08414d91:::smithi07694878-42:head [assert-version v1737,copy-get max 8388608] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e642)", 2021-03-15T22:29:53.538 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:51.199690+0000", 2021-03-15T22:29:53.538 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 2.2558686520000002, 2021-03-15T22:29:53.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0301155609999999, 2021-03-15T22:29:53.539 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.539 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:29:53.539 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.540 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199690+0000", 2021-03-15T22:29:53.540 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.541 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.541 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.541 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.542 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199690+0000", 2021-03-15T22:29:53.542 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.689e-06 2021-03-15T22:29:53.542 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.543 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.543 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199692+0000", 2021-03-15T22:29:53.543 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8900000000000002e-06 2021-03-15T22:29:53.544 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.544 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.544 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.545 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199702+0000", 2021-03-15T22:29:53.545 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1841999999999999e-05 2021-03-15T22:29:53.545 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.546 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.546 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199714+0000", 2021-03-15T22:29:53.546 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4039999999999998e-06 2021-03-15T22:29:53.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.547 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.547 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199721+0000", 2021-03-15T22:29:53.548 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8303000000000001e-05 2021-03-15T22:29:53.548 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.548 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.548 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.549 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199780+0000", 2021-03-15T22:29:53.549 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0937e-05 2021-03-15T22:29:53.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.550 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:29:53.551 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199791+0000", 2021-03-15T22:29:53.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010780039 2021-03-15T22:29:53.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.551 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.552 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.210571+0000", 2021-03-15T22:29:53.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0107e-05 2021-03-15T22:29:53.552 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.553 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:29:53.553 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.210581+0000", 2021-03-15T22:29:53.554 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0163172810000001 2021-03-15T22:29:53.554 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.554 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.554 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.555 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226898+0000", 2021-03-15T22:29:53.555 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012874800000000001 2021-03-15T22:29:53.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227027+0000", 2021-03-15T22:29:53.556 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027783209999999998 2021-03-15T22:29:53.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.557 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.557 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.229805+0000", 2021-03-15T22:29:53.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0301155609999999 2021-03-15T22:29:53.558 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.558 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.559 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.559 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2106 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94751378609664 gen 7] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:29:53.560 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.131311+0000", 2021-03-15T22:29:53.560 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.324247645, 2021-03-15T22:29:53.560 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.092023303000000001, 2021-03-15T22:29:53.560 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.561 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:29:53.561 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:29:53.561 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:29:53.566 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:29:53.566 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2106 2021-03-15T22:29:53.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.567 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.568 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.568 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131311+0000", 2021-03-15T22:29:53.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.569 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.569 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.570 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131311+0000", 2021-03-15T22:29:53.570 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9400000000000004e-07 2021-03-15T22:29:53.570 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.570 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.571 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.571 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131312+0000", 2021-03-15T22:29:53.571 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2810000000000001e-06 2021-03-15T22:29:53.572 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.572 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.572 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.572 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131313+0000", 2021-03-15T22:29:53.573 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6209999999999998e-06 2021-03-15T22:29:53.573 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.573 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.574 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.574 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131319+0000", 2021-03-15T22:29:53.574 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9940000000000002e-06 2021-03-15T22:29:53.574 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.575 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.575 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131323+0000", 2021-03-15T22:29:53.575 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0538999999999998e-05 2021-03-15T22:29:53.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.576 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.576 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.577 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131384+0000", 2021-03-15T22:29:53.577 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5080000000000005e-06 2021-03-15T22:29:53.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.578 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:29:53.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131392+0000", 2021-03-15T22:29:53.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.086493444000000003 2021-03-15T22:29:53.579 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.579 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.579 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.217886+0000", 2021-03-15T22:29:53.580 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010953 2021-03-15T22:29:53.580 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.580 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.581 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.581 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.218981+0000", 2021-03-15T22:29:53.581 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043527219999999998 2021-03-15T22:29:53.581 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.582 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.582 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.223334+0000", 2021-03-15T22:29:53.583 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.092023303000000001 2021-03-15T22:29:53.583 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.583 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.583 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.584 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.584 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.584 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2109 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94751378609664 gen 8] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:29:53.584 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:29:53.585 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.324224708, 2021-03-15T22:29:53.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.093324206000000007, 2021-03-15T22:29:53.585 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.585 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:29:53.586 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:29:53.586 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:29:53.586 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:29:53.586 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2109 2021-03-15T22:29:53.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.587 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:29:53.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.589 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.589 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.589 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:29:53.589 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0699999999999996e-07 2021-03-15T22:29:53.590 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.590 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.590 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:29:53.591 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.066e-06 2021-03-15T22:29:53.591 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.591 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.591 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.592 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131336+0000", 2021-03-15T22:29:53.592 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9439999999999999e-06 2021-03-15T22:29:53.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.592 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.593 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.593 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131342+0000", 2021-03-15T22:29:53.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1559999999999997e-06 2021-03-15T22:29:53.593 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.594 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.594 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131347+0000", 2021-03-15T22:29:53.595 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5375000000000003e-05 2021-03-15T22:29:53.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.596 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131412+0000", 2021-03-15T22:29:53.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2870000000000004e-06 2021-03-15T22:29:53.596 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.597 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:29:53.597 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131417+0000", 2021-03-15T22:29:53.597 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.087846846000000006 2021-03-15T22:29:53.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.598 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.598 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.598 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.219264+0000", 2021-03-15T22:29:53.599 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5641000000000003e-05 2021-03-15T22:29:53.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.599 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.599 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.600 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.219340+0000", 2021-03-15T22:29:53.600 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053179840000000004 2021-03-15T22:29:53.600 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.601 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.224658+0000", 2021-03-15T22:29:53.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.093324206000000007 2021-03-15T22:29:53.602 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.602 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.602 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.603 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "pg_update_log_missing(5.e epoch 643/642 rep_tid 601 entries 643'12 (0'0) error 5:7759931f:::notify.3:head by client.14397.0:2107 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2021-03-15T22:29:53.603 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.216974+0000", 2021-03-15T22:29:53.604 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.2385841870000001, 2021-03-15T22:29:53.604 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0092931279999999995, 2021-03-15T22:29:53.604 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.604 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-15T22:29:53.605 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.605 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.216974+0000", 2021-03-15T22:29:53.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.606 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.606 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.607 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.607 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.216974+0000", 2021-03-15T22:29:53.607 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5700000000000002e-07 2021-03-15T22:29:53.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.608 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.608 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.216975+0000", 2021-03-15T22:29:53.609 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0320000000000003e-06 2021-03-15T22:29:53.609 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.609 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.610 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.610 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.216980+0000", 2021-03-15T22:29:53.610 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2470000000000008e-06 2021-03-15T22:29:53.610 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.611 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.611 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.611 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.216988+0000", 2021-03-15T22:29:53.611 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9200000000000001e-06 2021-03-15T22:29:53.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.612 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.612 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.613 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.216994+0000", 2021-03-15T22:29:53.613 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00055924200000000005 2021-03-15T22:29:53.613 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.614 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.614 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.217553+0000", 2021-03-15T22:29:53.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0087139299999999999 2021-03-15T22:29:53.615 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.615 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.615 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.615 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226267+0000", 2021-03-15T22:29:53.616 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0092931279999999995 2021-03-15T22:29:53.616 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.616 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.616 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.617 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "pg_update_log_missing(5.e epoch 643/642 rep_tid 602 entries 643'13 (0'0) error 5:7759931f:::notify.3:head by client.14397.0:2110 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 0'0)", 2021-03-15T22:29:53.618 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.217000+0000", 2021-03-15T22:29:53.618 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.2385587010000001, 2021-03-15T22:29:53.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0092752310000000001, 2021-03-15T22:29:53.618 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.619 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-15T22:29:53.619 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.619 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.620 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.620 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.217000+0000", 2021-03-15T22:29:53.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.620 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.217000+0000", 2021-03-15T22:29:53.622 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0999999999999998e-07 2021-03-15T22:29:53.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.217000+0000", 2021-03-15T22:29:53.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7710000000000001e-06 2021-03-15T22:29:53.623 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.623 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.624 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.217002+0000", 2021-03-15T22:29:53.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1110000000000001e-06 2021-03-15T22:29:53.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.625 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.217006+0000", 2021-03-15T22:29:53.626 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0560999999999999e-05 2021-03-15T22:29:53.626 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.626 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.627 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.627 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.217017+0000", 2021-03-15T22:29:53.627 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00062421700000000004 2021-03-15T22:29:53.627 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.628 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.628 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.628 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.217641+0000", 2021-03-15T22:29:53.628 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0086339610000000008 2021-03-15T22:29:53.629 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.629 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.629 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226275+0000", 2021-03-15T22:29:53.630 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0092752310000000001 2021-03-15T22:29:53.630 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.630 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.630 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.631 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5968 7.0 7:071f7a04:::smithi07694878-37:head [read 0~2622057,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e643)", 2021-03-15T22:29:53.632 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.226628+0000", 2021-03-15T22:29:53.632 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.2289307199999999, 2021-03-15T22:29:53.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078271429999999999, 2021-03-15T22:29:53.632 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.633 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:29:53.633 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:29:53.633 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:29:53.633 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:29:53.634 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5968 2021-03-15T22:29:53.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.634 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226628+0000", 2021-03-15T22:29:53.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.635 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.636 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226628+0000", 2021-03-15T22:29:53.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1159999999999999e-06 2021-03-15T22:29:53.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.637 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.638 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226631+0000", 2021-03-15T22:29:53.638 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0199999999999996e-06 2021-03-15T22:29:53.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.639 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.639 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226635+0000", 2021-03-15T22:29:53.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0319000000000001e-05 2021-03-15T22:29:53.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.640 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226645+0000", 2021-03-15T22:29:53.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0379999999999996e-06 2021-03-15T22:29:53.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.641 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.641 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226651+0000", 2021-03-15T22:29:53.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047438949999999997 2021-03-15T22:29:53.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.642 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.643 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.643 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.231395+0000", 2021-03-15T22:29:53.643 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011444500000000001 2021-03-15T22:29:53.643 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.644 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.644 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.644 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.231509+0000", 2021-03-15T22:29:53.644 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00294531 2021-03-15T22:29:53.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.645 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.234455+0000", 2021-03-15T22:29:53.646 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0078271429999999999 2021-03-15T22:29:53.646 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.646 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.646 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.647 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.647 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5969 7.0 7:071f7a04:::smithi07694878-37:head [sparse-read 0~2120558] snapc 0=[] ondisk+read+known_if_redirected e643)", 2021-03-15T22:29:53.648 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.226661+0000", 2021-03-15T22:29:53.648 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.2288969350000001, 2021-03-15T22:29:53.648 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081348189999999997, 2021-03-15T22:29:53.648 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.649 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:29:53.649 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:29:53.649 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:29:53.649 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:29:53.650 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5969 2021-03-15T22:29:53.650 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.650 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.651 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.651 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226661+0000", 2021-03-15T22:29:53.651 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.651 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.652 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226661+0000", 2021-03-15T22:29:53.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0449999999999999e-06 2021-03-15T22:29:53.653 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.653 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.653 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.654 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226662+0000", 2021-03-15T22:29:53.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.497e-06 2021-03-15T22:29:53.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.655 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226664+0000", 2021-03-15T22:29:53.655 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3790000000000003e-06 2021-03-15T22:29:53.655 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.656 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.656 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226670+0000", 2021-03-15T22:29:53.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2810000000000002e-06 2021-03-15T22:29:53.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.657 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226675+0000", 2021-03-15T22:29:53.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0079447370000000003 2021-03-15T22:29:53.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.659 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.234619+0000", 2021-03-15T22:29:53.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4895999999999997e-05 2021-03-15T22:29:53.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.660 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.660 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.234684+0000", 2021-03-15T22:29:53.661 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111984 2021-03-15T22:29:53.661 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.661 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.661 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.662 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.234796+0000", 2021-03-15T22:29:53.662 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0081348189999999997 2021-03-15T22:29:53.662 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.662 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.663 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.663 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.663 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5970 7.0 7:071f7a04:::smithi07694878-37:head [read 0~2622057,checksum] snapc 0=[] ondisk+read+known_if_redirected e643)", 2021-03-15T22:29:53.664 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.226680+0000", 2021-03-15T22:29:53.664 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.2288785390000001, 2021-03-15T22:29:53.664 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012541424000000001, 2021-03-15T22:29:53.665 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.665 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:29:53.665 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:29:53.665 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:29:53.666 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:29:53.666 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5970 2021-03-15T22:29:53.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.666 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.667 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.667 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.667 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226680+0000", 2021-03-15T22:29:53.667 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.668 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.668 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.668 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.669 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226680+0000", 2021-03-15T22:29:53.669 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5719999999999999e-06 2021-03-15T22:29:53.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.670 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.670 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226681+0000", 2021-03-15T22:29:53.670 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.297e-06 2021-03-15T22:29:53.670 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.671 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.671 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.671 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226683+0000", 2021-03-15T22:29:53.671 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9460000000000003e-06 2021-03-15T22:29:53.672 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.672 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226689+0000", 2021-03-15T22:29:53.673 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7290000000000002e-06 2021-03-15T22:29:53.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.673 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.674 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.674 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226692+0000", 2021-03-15T22:29:53.674 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011147916000000001 2021-03-15T22:29:53.674 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.675 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.675 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.675 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.237840+0000", 2021-03-15T22:29:53.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000103939 2021-03-15T22:29:53.676 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.676 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.676 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.676 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.237944+0000", 2021-03-15T22:29:53.677 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001277025 2021-03-15T22:29:53.677 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.677 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.678 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.678 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.239221+0000", 2021-03-15T22:29:53.678 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012541424000000001 2021-03-15T22:29:53.678 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.679 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.679 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.680 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5974 7.e 7:7ac381e0:::smithi07694878-24:head [copy-from ver 1703] snapc 1fc=[1fc,1fb,1f8,1f7,1f6,1f4,1f3,1f0,1ee,1e9,1e7,1e6,1e4,1e0,1df,1dc,1d5,1d4,1c7,1c5,1c3,1ae,1a4,1a1,19c,193,17a,176,141] ondisk+write+known_if_redirected e643)", 2021-03-15T22:29:53.680 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.226697+0000", 2021-03-15T22:29:53.680 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.228861156, 2021-03-15T22:29:53.681 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031270606999999999, 2021-03-15T22:29:53.681 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.681 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:29:53.681 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:29:53.682 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:29:53.682 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:29:53.682 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5974 2021-03-15T22:29:53.683 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.683 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.683 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.683 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.684 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226697+0000", 2021-03-15T22:29:53.684 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.684 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.685 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.685 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226697+0000", 2021-03-15T22:29:53.685 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6269999999999999e-06 2021-03-15T22:29:53.686 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.686 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.686 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.686 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226702+0000", 2021-03-15T22:29:53.687 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.302e-06 2021-03-15T22:29:53.687 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.687 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.688 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.688 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226703+0000", 2021-03-15T22:29:53.688 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1020000000000002e-06 2021-03-15T22:29:53.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.689 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.689 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.689 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226709+0000", 2021-03-15T22:29:53.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6780000000000002e-06 2021-03-15T22:29:53.690 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226713+0000", 2021-03-15T22:29:53.691 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025251599999999999 2021-03-15T22:29:53.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.692 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.692 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226965+0000", 2021-03-15T22:29:53.692 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010260900000000001 2021-03-15T22:29:53.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.693 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.693 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227068+0000", 2021-03-15T22:29:53.693 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013635573999999999 2021-03-15T22:29:53.694 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.694 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-15T22:29:53.695 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.240704+0000", 2021-03-15T22:29:53.695 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010619755 2021-03-15T22:29:53.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.695 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.696 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:29:53.696 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.251323+0000", 2021-03-15T22:29:53.696 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024678149999999999 2021-03-15T22:29:53.696 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.697 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.697 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:29:53.697 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.253791+0000", 2021-03-15T22:29:53.698 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040414350000000003 2021-03-15T22:29:53.698 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.698 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:29:53.699 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.257833+0000", 2021-03-15T22:29:53.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9538e-05 2021-03-15T22:29:53.699 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.699 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:29:53.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.257872+0000", 2021-03-15T22:29:53.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5655999999999994e-05 2021-03-15T22:29:53.700 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.701 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.701 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.701 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.257968+0000", 2021-03-15T22:29:53.701 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031270606999999999 2021-03-15T22:29:53.702 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.702 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.702 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.703 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.703 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5975 7.e 7:7ac381e0:::smithi07694878-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e643)", 2021-03-15T22:29:53.703 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.226719+0000", 2021-03-15T22:29:53.704 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.2288389070000001, 2021-03-15T22:29:53.704 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031554302999999999, 2021-03-15T22:29:53.704 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.705 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:29:53.705 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:29:53.705 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:29:53.705 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:29:53.705 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5975 2021-03-15T22:29:53.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.706 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.706 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.707 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.707 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226719+0000", 2021-03-15T22:29:53.707 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.708 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.708 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.708 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226719+0000", 2021-03-15T22:29:53.708 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9899999999999999e-07 2021-03-15T22:29:53.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.709 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.709 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.709 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226720+0000", 2021-03-15T22:29:53.710 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3540000000000001e-06 2021-03-15T22:29:53.710 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.710 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.711 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.711 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226722+0000", 2021-03-15T22:29:53.711 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.049e-06 2021-03-15T22:29:53.711 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.712 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.712 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.712 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226728+0000", 2021-03-15T22:29:53.713 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9450000000000001e-06 2021-03-15T22:29:53.713 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.713 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.713 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.714 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226733+0000", 2021-03-15T22:29:53.714 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048394300000000003 2021-03-15T22:29:53.714 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.714 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.715 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.715 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227216+0000", 2021-03-15T22:29:53.715 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3120000000000001e-05 2021-03-15T22:29:53.715 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.716 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.716 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:29:53.716 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227260+0000", 2021-03-15T22:29:53.716 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020041761000000002 2021-03-15T22:29:53.717 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.717 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.717 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.718 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.247301+0000", 2021-03-15T22:29:53.718 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7263999999999997e-05 2021-03-15T22:29:53.718 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.718 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.719 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:29:53.719 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.247389+0000", 2021-03-15T22:29:53.719 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010630624999999999 2021-03-15T22:29:53.719 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.720 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.720 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.720 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258019+0000", 2021-03-15T22:29:53.721 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6768999999999995e-05 2021-03-15T22:29:53.721 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.721 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.721 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.722 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258116+0000", 2021-03-15T22:29:53.722 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015767400000000001 2021-03-15T22:29:53.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.722 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.723 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.723 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258274+0000", 2021-03-15T22:29:53.723 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031554302999999999 2021-03-15T22:29:53.723 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.724 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.724 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.724 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.725 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(osd.5.616:1 7.0 7:0cc37e3d:::smithi07694878-21:head [assert-version v1703,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e643)", 2021-03-15T22:29:53.725 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.227857+0000", 2021-03-15T22:29:53.725 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.227700926, 2021-03-15T22:29:53.726 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011428974999999999, 2021-03-15T22:29:53.726 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.726 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:29:53.726 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.727 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.727 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.727 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227857+0000", 2021-03-15T22:29:53.728 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.728 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.728 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.728 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.729 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227857+0000", 2021-03-15T22:29:53.729 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.481e-06 2021-03-15T22:29:53.729 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.729 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.730 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.730 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227860+0000", 2021-03-15T22:29:53.730 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8899999999999999e-06 2021-03-15T22:29:53.731 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.731 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.731 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.732 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227863+0000", 2021-03-15T22:29:53.732 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2099999999999999e-06 2021-03-15T22:29:53.732 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.733 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.733 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.734 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227872+0000", 2021-03-15T22:29:53.734 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5270000000000004e-06 2021-03-15T22:29:53.735 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.735 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.735 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.736 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227878+0000", 2021-03-15T22:29:53.736 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0069444199999999998 2021-03-15T22:29:53.736 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.736 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.737 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.737 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.234823+0000", 2021-03-15T22:29:53.737 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1724999999999993e-05 2021-03-15T22:29:53.738 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.738 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.738 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.739 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.234915+0000", 2021-03-15T22:29:53.739 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043717219999999998 2021-03-15T22:29:53.739 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.739 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.740 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.740 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.239286+0000", 2021-03-15T22:29:53.740 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011428974999999999 2021-03-15T22:29:53.740 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.741 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.741 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.741 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.742 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.742 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.3.0:1403 7.a e643/642 7:5595c26f:::smithi07694878-7:185 v 643'914, mlcod=643'910)", 2021-03-15T22:29:53.742 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.246076+0000", 2021-03-15T22:29:53.743 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.2094820630000001, 2021-03-15T22:29:53.743 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012195887000000001, 2021-03-15T22:29:53.743 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.743 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:29:53.744 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.744 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.744 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.744 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246076+0000", 2021-03-15T22:29:53.745 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.745 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.745 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.746 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.746 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246076+0000", 2021-03-15T22:29:53.746 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5759999999999999e-06 2021-03-15T22:29:53.746 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.747 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.747 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.747 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246081+0000", 2021-03-15T22:29:53.747 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4699999999999998e-06 2021-03-15T22:29:53.748 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.748 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.748 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.748 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246084+0000", 2021-03-15T22:29:53.749 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9805000000000001e-05 2021-03-15T22:29:53.749 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.749 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.749 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.750 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246104+0000", 2021-03-15T22:29:53.750 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7160000000000003e-06 2021-03-15T22:29:53.750 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.751 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.751 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.751 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246112+0000", 2021-03-15T22:29:53.751 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00092903500000000004 2021-03-15T22:29:53.752 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.752 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.752 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.753 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.247041+0000", 2021-03-15T22:29:53.753 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0318999999999999e-05 2021-03-15T22:29:53.753 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.753 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.754 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.754 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.247071+0000", 2021-03-15T22:29:53.754 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011167832000000001 2021-03-15T22:29:53.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.755 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:29:53.755 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258239+0000", 2021-03-15T22:29:53.756 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3133999999999998e-05 2021-03-15T22:29:53.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.757 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.757 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258272+0000", 2021-03-15T22:29:53.757 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012195887000000001 2021-03-15T22:29:53.757 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.758 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.758 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.759 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.3.0:1404 7.a e643/642 7:53dee572:::smithi07694878-35:1e4 v 643'916, mlcod=643'910)", 2021-03-15T22:29:53.759 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.249660+0000", 2021-03-15T22:29:53.759 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 1.2058987409999999, 2021-03-15T22:29:53.760 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0087786429999999992, 2021-03-15T22:29:53.760 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.760 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:29:53.760 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.761 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.761 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.761 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.249660+0000", 2021-03-15T22:29:53.761 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.762 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.762 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.762 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.763 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.249660+0000", 2021-03-15T22:29:53.763 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.287e-06 2021-03-15T22:29:53.763 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.763 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.764 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.764 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.249663+0000", 2021-03-15T22:29:53.764 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4419999999999998e-06 2021-03-15T22:29:53.764 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.765 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.765 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.765 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.249666+0000", 2021-03-15T22:29:53.765 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5651999999999999e-05 2021-03-15T22:29:53.766 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.766 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.766 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.766 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.249682+0000", 2021-03-15T22:29:53.767 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8540000000000002e-06 2021-03-15T22:29:53.767 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.767 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.768 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.768 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.249688+0000", 2021-03-15T22:29:53.768 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022266930000000001 2021-03-15T22:29:53.768 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.769 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.769 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.769 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.251914+0000", 2021-03-15T22:29:53.769 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5516000000000001e-05 2021-03-15T22:29:53.770 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.770 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.770 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.771 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.251940+0000", 2021-03-15T22:29:53.771 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00646636 2021-03-15T22:29:53.771 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.771 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.772 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:29:53.772 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258406+0000", 2021-03-15T22:29:53.772 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1838999999999998e-05 2021-03-15T22:29:53.772 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.773 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.773 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.773 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258438+0000", 2021-03-15T22:29:53.774 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0087786429999999992 2021-03-15T22:29:53.774 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.774 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.774 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.775 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.775 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.775 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.1.0:835 7.b e644/642 7:d4368cae:::smithi07694878-18:14b v 644'677, mlcod=640'673)", 2021-03-15T22:29:53.775 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:53.236270+0000", 2021-03-15T22:29:53.776 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.219288346, 2021-03-15T22:29:53.776 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0064595879999999996, 2021-03-15T22:29:53.776 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.776 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:29:53.777 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.777 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.777 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.778 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.236270+0000", 2021-03-15T22:29:53.778 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.778 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.778 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.779 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.779 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.236270+0000", 2021-03-15T22:29:53.779 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2770000000000001e-06 2021-03-15T22:29:53.779 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.780 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.780 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.780 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.236273+0000", 2021-03-15T22:29:53.780 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6429999999999999e-06 2021-03-15T22:29:53.781 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.781 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.781 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.781 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.236276+0000", 2021-03-15T22:29:53.782 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001057655 2021-03-15T22:29:53.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.783 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.783 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.237333+0000", 2021-03-15T22:29:53.783 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.579e-06 2021-03-15T22:29:53.783 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.784 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.784 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.784 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.237340+0000", 2021-03-15T22:29:53.784 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0158999999999999e-05 2021-03-15T22:29:53.785 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.785 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.785 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.786 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.237390+0000", 2021-03-15T22:29:53.786 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0319e-05 2021-03-15T22:29:53.786 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.786 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.787 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.787 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.237411+0000", 2021-03-15T22:29:53.787 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052793559999999998 2021-03-15T22:29:53.788 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.788 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.788 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:29:53.788 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.242690+0000", 2021-03-15T22:29:53.789 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.96e-05 2021-03-15T22:29:53.789 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.789 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.789 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.790 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.242730+0000", 2021-03-15T22:29:53.790 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0064595879999999996 2021-03-15T22:29:53.790 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.791 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.791 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.791 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.791 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.792 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.1.0:836 7.b e644/642 7:d10ba14d:::smithi07694878-43:14d v 644'679, mlcod=640'673)", 2021-03-15T22:29:53.792 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:53.238047+0000", 2021-03-15T22:29:53.792 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.21751161299999999, 2021-03-15T22:29:53.793 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0062480590000000003, 2021-03-15T22:29:53.793 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:29:53.793 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:29:53.793 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:29:53.794 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.794 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:29:53.794 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.238047+0000", 2021-03-15T22:29:53.794 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:29:53.795 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.795 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.795 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:29:53.795 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.238047+0000", 2021-03-15T22:29:53.796 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.807e-06 2021-03-15T22:29:53.796 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.796 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.796 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:29:53.797 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.238049+0000", 2021-03-15T22:29:53.797 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1129999999999999e-06 2021-03-15T22:29:53.797 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.798 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.798 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:29:53.798 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.238052+0000", 2021-03-15T22:29:53.798 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0600000000000008e-06 2021-03-15T22:29:53.799 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.799 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.799 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:29:53.800 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.238060+0000", 2021-03-15T22:29:53.800 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.014e-06 2021-03-15T22:29:53.800 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.800 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:29:53.801 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.238064+0000", 2021-03-15T22:29:53.801 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023028600000000001 2021-03-15T22:29:53.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.802 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.802 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:29:53.802 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.238294+0000", 2021-03-15T22:29:53.803 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8612000000000001e-05 2021-03-15T22:29:53.803 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.803 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.803 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:29:53.804 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.238313+0000", 2021-03-15T22:29:53.804 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0059510029999999998 2021-03-15T22:29:53.804 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.805 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.805 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:29:53.805 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.244264+0000", 2021-03-15T22:29:53.805 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1164000000000002e-05 2021-03-15T22:29:53.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:29:53.806 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:29:53.806 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:29:53.806 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:53.244295+0000", 2021-03-15T22:29:53.807 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0062480590000000003 2021-03-15T22:29:53.807 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.807 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.807 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.808 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:29:53.808 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:29:53.808 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:29:53.809 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:29:53.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:53.811 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:53.812 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:53.913 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:53.936 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:53.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:53.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:54.039 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:54.062 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:54.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:54.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:54.143 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:54.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:54.188 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:54.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:54.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:54.236 INFO:tasks.rados.rados.0.smithi076.stdout:2343: done (5 left) 2021-03-15T22:29:54.236 INFO:tasks.rados.rados.0.smithi076.stdout:2344: read oid 16 snap -1 2021-03-15T22:29:54.237 INFO:tasks.rados.rados.0.smithi076.stdout:2344: expect (ObjNum 604 snap 188 seq_num 604) 2021-03-15T22:29:54.237 INFO:tasks.rados.rados.0.smithi076.stdout:2345: snap_remove snap 267 2021-03-15T22:29:54.241 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:29:54.245 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:29:54.291 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:54.314 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:54.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:54.316 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:54.417 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:54.439 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:54.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:54.441 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:54.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56514]: cluster 2021-03-15T22:29:53.134881+0000 mgr.y (mgr.14144) 815 : cluster [DBG] pgmap v1109: 138 pgs: 6 remapped+peering, 8 peering, 30 active+clean+remapped, 1 active+recovering, 4 active+undersized+degraded, 10 active+undersized, 79 active+clean; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s; 55/1731 objects degraded (3.177%); 145/1731 objects misplaced (8.377%); 630 B/s, 6 objects/s recovering 2021-03-15T22:29:54.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56514]: cluster 2021-03-15T22:29:53.229662+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e644: 8 total, 7 up, 8 in 2021-03-15T22:29:54.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56514]: cluster 2021-03-15T22:29:53.642619+0000 osd.7 (osd.7) 125 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:54.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56514]: cluster 2021-03-15T22:29:53.643174+0000 osd.7 (osd.7) 126 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:54.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56514]: audit 2021-03-15T22:29:53.902351+0000 mon.a (mon.0) 1756 : audit [INF] from='client.? 172.21.15.76:0/377672286' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:54.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56514]: audit 2021-03-15T22:29:53.961105+0000 mon.a (mon.0) 1757 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:54.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:29:54.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56514]: -15T22:29:54.070911+0000 mon.a (mon.0) 1758 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:54.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[62156]: cluster 2021-03-15T22:29:53.134881+0000 mgr.y (mgr.14144) 815 : cluster [DBG] pgmap v1109: 138 pgs: 6 remapped+peering, 8 peering, 30 active+clean+remapped, 1 active+recovering, 4 active+undersized+degraded, 10 active+undersized, 79 active+clean; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s; 55/1731 objects degraded (3.177%); 145/1731 objects misplaced (8.377%); 630 B/s, 6 objects/s recovering 2021-03-15T22:29:54.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[62156]: cluster 2021-03-15T22:29:53.229662+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e644: 8 total, 7 up, 8 in 2021-03-15T22:29:54.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[62156]: cluster 2021-03-15T22:29:53.642619+0000 osd.7 (osd.7) 125 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:54.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[62156]: cluster 2021-03-15T22:29:53.643174+0000 osd.7 (osd.7) 126 : cluster [DBG] 2021-03-15T22:29:54.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[62156]: purged_snaps scrub ok 2021-03-15T22:29:54.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[62156]: audit 2021-03-15T22:29:53.902351+0000 mon.a (mon.0) 1756 : audit [INF] from='client.? 172.21.15.76:0/377672286' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:54.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[62156]: audit 2021-03-15T22:29:53.961105+0000 mon.a (mon.0) 1757 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:54.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[62156]: audit 2021-03-15T22:29:54.070911+0000 mon.a (mon.0) 1758 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:54.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: debug 2021-03-15T22:29:54.355+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:29:54.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:29:54.522 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:54.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:29:54.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:54.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:29:54.523 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:54.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:54.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:29:54.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: debug 2021-03-15T22:29:54.356+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:29:54.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:29:54.524 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:54.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:29:54.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:54.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:29:54.525 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:54.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:54.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:29:54.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: debug 2021-03-15T22:29:54.526 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: 2021-03-15T22:29:54.356+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:29:54.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:29:54.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:54.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:29:54.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:54.527 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:29:54.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:54.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:54.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:29:54.528 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: 2021-03-15T22:29:54.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: debug 2021-03-15T22:29:54.357+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:29:54.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:29:54.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:54.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:29:54.529 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:54.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:29:54.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:54.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:54.530 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:29:54.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: debug 2021-03-15T22:29:54.357+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:29:54.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:29:54.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:54.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:29:54.531 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:54.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:29:54.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:54.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:54.532 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:29:54.533 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: debug 2021-03-15T22:29:54.358+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:29:54.533 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:29:54.533 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:54.533 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:29:54.533 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:54.534 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:29:54.534 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:54.534 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:54.534 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:29:54.535 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: debug 2021-03-15T22:29:54.358+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:29:54.535 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:29:54.535 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:54.535 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:29:54.536 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:54.536 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:29:54.536 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:54.536 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:54.536 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:29:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:29:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:54 smithi125 conmon[56739]: cluster 2021-03-15T22:29:53.134881+0000 mgr.y (mgr.14144) 815 : cluster [DBG] pgmap v1109: 138 pgs: 6 remapped+peering, 8 peering, 30 active+clean+remapped, 1 active+recovering, 4 active+undersized+degraded, 10 active+undersized, 79 active+clean; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 1.1 MiB/s wr, 3 op/s; 55/1731 objects degraded (3.177%); 145/1731 objects misplaced (8.377%); 630 B/s, 6 objects/s recovering 2021-03-15T22:29:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:54 smithi125 conmon[56739]: cluster 2021-03-15T22:29:53.229662+0000 mon.a (mon.0) 1755 : cluster [DBG] osdmap e644: 8 total, 7 up, 8 in 2021-03-15T22:29:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:54 smithi125 conmon[56739]: cluster 2021-03-15T22:29:53.642619+0000 osd.7 (osd.7) 125 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:54 smithi125 conmon[56739]: cluster 2021-03-15T22:29:53.643174+0000 osd.7 (osd.7) 126 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:54 smithi125 conmon[56739]: audit 2021-03-15T22:29:53.902351+0000 mon.a (mon.0) 2021-03-15T22:29:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:54 smithi125 conmon[56739]: 1756 : audit [INF] from='client.? 172.21.15.76:0/377672286' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:29:54.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:54 smithi125 conmon[56739]: audit 2021-03-15T22:29:53.961105+0000 mon.a (mon.0) 1757 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:54.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:54 smithi125 conmon[56739]: audit 2021-03-15T22:29:54.070911+0000 mon.a (mon.0) 1758 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:54.542 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:54.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:54.566 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:54.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:54.667 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:54.689 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:54.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:54.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:54.792 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:54.818 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:54.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:54.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:54.921 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:54.945 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:54.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:54.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:55.047 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:55.070 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:55.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:55.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:55.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:55.196 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:55.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:55.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:55.250 INFO:tasks.rados.rados.0.smithi076.stdout:2345: done (6 left) 2021-03-15T22:29:55.250 INFO:tasks.rados.rados.0.smithi076.stdout:2346: snap_create 2021-03-15T22:29:55.250 INFO:tasks.rados.rados.0.smithi076.stdout:2332: finishing copy_from to smithi07694878-44 2021-03-15T22:29:55.251 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1737 (ObjNum 794 snap 261 seq_num 794) dirty exists 2021-03-15T22:29:55.251 INFO:tasks.rados.rados.0.smithi076.stdout:2332: finishing copy_from racing read to smithi07694878-44 2021-03-15T22:29:55.251 INFO:tasks.rados.rados.0.smithi076.stdout:2339: finishing write tid 2 to smithi07694878-6 2021-03-15T22:29:55.251 INFO:tasks.rados.rados.0.smithi076.stdout:2339: finishing write tid 3 to smithi07694878-6 2021-03-15T22:29:55.251 INFO:tasks.rados.rados.0.smithi076.stdout:2339: finishing write tid 4 to smithi07694878-6 2021-03-15T22:29:55.252 INFO:tasks.rados.rados.0.smithi076.stdout:2339: finishing write tid 5 to smithi07694878-6 2021-03-15T22:29:55.252 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 1590 (ObjNum 802 snap 268 seq_num 802) dirty exists 2021-03-15T22:29:55.252 INFO:tasks.rados.rados.0.smithi076.stdout:2339: left oid 6 (ObjNum 802 snap 268 seq_num 802) 2021-03-15T22:29:55.252 INFO:tasks.rados.rados.0.smithi076.stdout:2336: expect (ObjNum 751 snap 245 seq_num 751) 2021-03-15T22:29:55.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:55.322 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:55.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:55.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:55.387 INFO:tasks.rados.rados.0.smithi076.stdout:2338: finishing copy_from to smithi07694878-24 2021-03-15T22:29:55.388 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1703 (ObjNum 782 snap 260 seq_num 782) dirty exists 2021-03-15T22:29:55.388 INFO:tasks.rados.rados.0.smithi076.stdout:2338: finishing copy_from racing read to smithi07694878-24 2021-03-15T22:29:55.388 INFO:tasks.rados.rados.0.smithi076.stdout:2340: finishing write tid 1 to smithi07694878-1 2021-03-15T22:29:55.388 INFO:tasks.rados.rados.0.smithi076.stdout:2340: finishing write tid 2 to smithi07694878-1 2021-03-15T22:29:55.389 INFO:tasks.rados.rados.0.smithi076.stdout:2340: finishing write tid 3 to smithi07694878-1 2021-03-15T22:29:55.389 INFO:tasks.rados.rados.0.smithi076.stdout:2340: finishing write tid 4 to smithi07694878-1 2021-03-15T22:29:55.389 INFO:tasks.rados.rados.0.smithi076.stdout:2340: finishing write tid 5 to smithi07694878-1 2021-03-15T22:29:55.389 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 1324 (ObjNum 803 snap 268 seq_num 803) dirty exists 2021-03-15T22:29:55.389 INFO:tasks.rados.rados.0.smithi076.stdout:2340: left oid 1 (ObjNum 803 snap 268 seq_num 803) 2021-03-15T22:29:55.390 INFO:tasks.rados.rados.0.smithi076.stdout:2344: expect (ObjNum 604 snap 188 seq_num 604) 2021-03-15T22:29:55.396 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:55.424 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:55.447 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:55.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:55.449 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: cluster 2021-03-15T22:29:54.223062+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: audit 2021-03-15T22:29:54.233634+0000 mon.a (mon.0) 1760 : audit [INF] from='client.? 172.21.15.76:0/377672286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:29:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: cluster 2021-03-15T22:29:54.233693+0000 mon.a (mon.0) 1761 : cluster [DBG] osdmap e645: 8 total, 7 up, 8 in 2021-03-15T22:29:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:29:54.351496+0000 mgr.y (mgr.14144) 816 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:29:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: audit 2021-03-15T22:29:54.354884+0000 mon.a (mon.0) 1762 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:29:54.356244+0000 mgr.y (mgr.14144) 817 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:29:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:29:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:29:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:29:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:29:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:29:54.356695+0000 mgr.y (mgr.14144) 818 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:29:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:29:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:29:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:29:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:29:55.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:29:54.357166+0000 mgr.y (mgr.14144) 819 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:29:55.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:29:55.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:29:55.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:29:55.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:29:55.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:29:54.357714+0000 mgr.y (mgr.14144) 820 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:29:55.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:29:55.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:29:55.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:29:55.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:29:55.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:29:54.358282+0000 mgr.y (mgr.14144) 821 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:29:55.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:29:55.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:29:55.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:29:55.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:29:55.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:29:54.358829+0000 mgr.y (mgr.14144) 822 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:29:55.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:29:55.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:29:55.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:29:55.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:29:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: cephadm 2021-03-15T22: 2021-03-15T22:29:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: 29:54.359363+0000 mgr.y (mgr.14144) 823 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:29:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:29:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:29:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:29:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:29:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[56514]: cluster 2021-03-15T22:29:54.564111+0000 osd.3 (osd.3) 118 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:29:55.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cluster 2021-03-15T22:29:55.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021 2021-03-15T22:29:55.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: - 2021-03-15T22:29:55.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 03 2021-03-15T22:29:55.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: - 2021-03-15T22:29:55.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 15 2021-03-15T22:29:55.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: T 2021-03-15T22:29:55.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 22 2021-03-15T22:29:55.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: : 2021-03-15T22:29:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 29 2021-03-15T22:29:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: : 2021-03-15T22:29:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 54 2021-03-15T22:29:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: . 2021-03-15T22:29:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 223062 2021-03-15T22:29:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: +0000 2021-03-15T22:29:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: mon.a 2021-03-15T22:29:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: ( 2021-03-15T22:29:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: mon 2021-03-15T22:29:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: . 2021-03-15T22:29:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 0 2021-03-15T22:29:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: ) 2021-03-15T22:29:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 1759 2021-03-15T22:29:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: : 2021-03-15T22:29:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cluster 2021-03-15T22:29:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: [WRN] 2021-03-15T22:29:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: audit 2021-03-15T22:29:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021 2021-03-15T22:29:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: - 2021-03-15T22:29:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 03 2021-03-15T22:29:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: - 2021-03-15T22:29:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 15 2021-03-15T22:29:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: T 2021-03-15T22:29:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 22 2021-03-15T22:29:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: : 2021-03-15T22:29:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 29 2021-03-15T22:29:55.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: : 2021-03-15T22:29:55.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 54 2021-03-15T22:29:55.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: . 2021-03-15T22:29:55.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 233634 2021-03-15T22:29:55.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: +0000 2021-03-15T22:29:55.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: mon.a 2021-03-15T22:29:55.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: ( 2021-03-15T22:29:55.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: mon 2021-03-15T22:29:55.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: . 2021-03-15T22:29:55.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 0 2021-03-15T22:29:55.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: ) 2021-03-15T22:29:55.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 1760 2021-03-15T22:29:55.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: : 2021-03-15T22:29:55.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: audit 2021-03-15T22:29:55.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: [INF] 2021-03-15T22:29:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/377672286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:29:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cluster 2021-03-15T22:29:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021 2021-03-15T22:29:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: - 2021-03-15T22:29:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 03 2021-03-15T22:29:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: - 2021-03-15T22:29:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 15 2021-03-15T22:29:55.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: T 2021-03-15T22:29:55.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 22 2021-03-15T22:29:55.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: : 2021-03-15T22:29:55.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 29 2021-03-15T22:29:55.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: : 2021-03-15T22:29:55.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 54 2021-03-15T22:29:55.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: . 2021-03-15T22:29:55.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 233693+0000 mon.a (mon.0) 1761 : cluster 2021-03-15T22:29:55.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: [DBG] osdmap e645: 8 total, 7 up, 8 in 2021-03-15T22:29:55.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:29: 2021-03-15T22:29:55.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 54.351496+0000 mgr.y (mgr.14144) 816 : 2021-03-15T22:29:55.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:29:55.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:29:55.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 03-15T22:29:54.354884+0000 mon.a ( 2021-03-15T22:29:55.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: mon.0) 1762 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:55.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:29:55.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021-03-15T22:29:55.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021 2021-03-15T22:29:55.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: - 2021-03-15T22:29:55.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 03 2021-03-15T22:29:55.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: - 2021-03-15T22:29:55.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 15 2021-03-15T22:29:55.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: T22:29:54.356244+0000 2021-03-15T22:29:55.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: mgr.y (mgr.14144) 817 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:29:55.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:29:55.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:29:55.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:29:55.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:29:55.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:29:54.356695+0000 mgr.y (mgr.14144) 818 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:29:55.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:29:55.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:29:55.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:29:55.560 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.560 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.561 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:29:55.561 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:29:54.357166+0000 mgr.y (mgr.14144 2021-03-15T22:29:55.561 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: ) 819 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:29:55.561 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:29:55.562 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.562 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:29:55.562 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.563 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:29:55.563 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.563 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.563 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:29:55.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:29:55.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 03-15T22:29: 2021-03-15T22:29:55.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 54.357714+0000 2021-03-15T22:29:55.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: mgr.y (mgr.14144) 2021-03-15T22:29:55.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 820 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:29:55.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:29:55.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.568 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:29:55.568 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:29:55.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:29:55.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:29:55.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 2021- 2021-03-15T22:29:55.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 03- 2021-03-15T22:29:55.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: 15T22:29:54.358282+0000 mgr.y ( 2021-03-15T22:29:55.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: mgr.14144) 821 : cephadm 2021-03-15T22:29:55.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:29:55.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:29:55.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:29:55.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:29:55.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:29:55.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cephadm 2021-03-15 2021-03-15T22:29:55.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: T22:29:54.358829+0000 mgr.y (mgr.14144) 822 : cephadm [ERR] 2021-03-15T22:29:55.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:29:55.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:29:55.575 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.575 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:29:55.575 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.575 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:29:55.576 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.576 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.576 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:29:55.577 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:29:54.359363+0000 mgr.y (mgr.14144) 823 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:29:55.577 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:29:55.578 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.578 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:29:55.578 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:29:55.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:29:55.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:55 smithi076 conmon[62156]: cluster 2021-03-15T22:29:54.564111+0000 osd.3 (osd.3) 118 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:29:55.582 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:55.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: cluster 2021-03-15T22:29:54.223062+0000 mon.a (mon.0) 1759 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:29:55.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: audit 2021-03-15T22:29:54.233634+0000 mon.a (mon.0) 2021-03-15T22:29:55.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: 1760 : audit [INF] from='client.? 172.21.15.76:0/377672286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:29:55.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: cluster 2021-03-15T22:29:54.233693+0000 mon.a (mon.0) 1761 : cluster [DBG] osdmap e645: 8 total, 7 up, 8 in 2021-03-15T22:29:55.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:29:54.351496+0000 mgr.y (mgr.14144) 816 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:29:55.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: audit 2021-03-15T22:29:54.354884+0000 mon.a (mon.0) 1762 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:29:55.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:29:54.356244+0000 mgr.y (mgr.14144) 817 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:29:55.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:29:55.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:29:55.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:29:55.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.588 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:29:55.589 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:29:54.356695+0000 mgr.y (mgr.14144) 818 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:29:55.589 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:29:55.589 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.589 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:29:55.590 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.590 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:29:55.590 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.590 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.591 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:29:55.591 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:29:54.357166+0000 mgr.y (mgr.14144) 819 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:29:55.591 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:29:55.591 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.592 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:29:55.592 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.592 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:29:55.592 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.593 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.593 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:29:55.593 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:29:54.357714+0000 mgr.y (mgr.14144) 820 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:29:55.593 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:29:55.594 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.594 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:29:55.594 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.594 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:29:55.595 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.595 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.595 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:29:55.595 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: cephadm 2021- 2021-03-15T22:29:55.596 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: 03-15T22:29:54.358282+0000 mgr.y (mgr.14144) 821 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:29:55.596 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:29:55.596 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.596 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:29:55.597 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.597 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:29:55.597 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.597 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.598 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:29:55.598 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:29:54.358829+0000 mgr.y (mgr.14144) 822 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:29:55.598 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:29:55.598 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.599 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:29:55.599 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.599 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:29:55.599 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.600 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.600 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:29:55.600 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:29:54.359363+0000 mgr.y (mgr.14144) 823 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:29:55.600 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:29:55.600 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:29:55.601 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:29:55.601 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:29:55.601 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:29:55.601 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:29:55.602 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:29:55.602 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:29:55.602 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:55 smithi125 conmon[56739]: cluster 2021-03-15T22:29:54.564111+0000 osd.3 (osd.3) 118 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:29:55.623 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:55.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:55.627 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:55.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:55.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:55.752 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:55.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:55.854 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:55.876 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:55.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:55.877 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:55.979 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:56.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:56.002 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:56.002 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:56.104 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:56.128 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:56.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:56.130 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:56.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:56.241 INFO:tasks.rados.rados.0.smithi076.stdout:2332: done (6 left) 2021-03-15T22:29:56.242 INFO:tasks.rados.rados.0.smithi076.stdout:2336: done (5 left) 2021-03-15T22:29:56.242 INFO:tasks.rados.rados.0.smithi076.stdout:2338: done (4 left) 2021-03-15T22:29:56.242 INFO:tasks.rados.rados.0.smithi076.stdout:2339: done (3 left) 2021-03-15T22:29:56.242 INFO:tasks.rados.rados.0.smithi076.stdout:2340: done (2 left) 2021-03-15T22:29:56.242 INFO:tasks.rados.rados.0.smithi076.stdout:2344: done (1 left) 2021-03-15T22:29:56.243 INFO:tasks.rados.rados.0.smithi076.stdout:2346: done (0 left) 2021-03-15T22:29:56.243 INFO:tasks.rados.rados.0.smithi076.stdout:2347: write (excl) oid 50 current snap is 269 2021-03-15T22:29:56.243 INFO:tasks.rados.rados.0.smithi076.stdout:2347: seq_num 805 ranges {729635=792510,1973534=578947,3240215=289420} 2021-03-15T22:29:56.251 INFO:tasks.rados.rados.0.smithi076.stdout:2347: writing smithi07694878-50 from 729635 to 1522145 tid 1 2021-03-15T22:29:56.257 INFO:tasks.rados.rados.0.smithi076.stdout:2347: writing smithi07694878-50 from 1973534 to 2552481 tid 2 2021-03-15T22:29:56.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:56.263 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:56.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:56.265 INFO:tasks.rados.rados.0.smithi076.stdout:2347: writing smithi07694878-50 from 3240215 to 3529635 tid 3 2021-03-15T22:29:56.265 INFO:tasks.rados.rados.0.smithi076.stdout:2348: delete oid 14 current snap is 269 2021-03-15T22:29:56.266 INFO:tasks.rados.rados.0.smithi076.stdout:2347: finishing write tid 1 to smithi07694878-50 2021-03-15T22:29:56.266 INFO:tasks.rados.rados.0.smithi076.stdout:2347: finishing write tid 2 to smithi07694878-50 2021-03-15T22:29:56.267 INFO:tasks.rados.rados.0.smithi076.stdout:2347: finishing write tid 3 to smithi07694878-50 2021-03-15T22:29:56.267 INFO:tasks.rados.rados.0.smithi076.stdout:2347: finishing write tid 4 to smithi07694878-50 2021-03-15T22:29:56.268 INFO:tasks.rados.rados.0.smithi076.stdout:2347: finishing write tid 5 to smithi07694878-50 2021-03-15T22:29:56.268 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 821 (ObjNum 805 snap 269 seq_num 805) dirty exists 2021-03-15T22:29:56.268 INFO:tasks.rados.rados.0.smithi076.stdout:2347: left oid 50 (ObjNum 805 snap 269 seq_num 805) 2021-03-15T22:29:56.268 INFO:tasks.rados.rados.0.smithi076.stdout:2347: done (1 left) 2021-03-15T22:29:56.269 INFO:tasks.rados.rados.0.smithi076.stdout:2348: done (0 left) 2021-03-15T22:29:56.269 INFO:tasks.rados.rados.0.smithi076.stdout:2349: copy_from oid 40 from oid 24 current snap is 269 2021-03-15T22:29:56.269 INFO:tasks.rados.rados.0.smithi076.stdout:2350: rollback oid 26 current snap is 269 2021-03-15T22:29:56.269 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 259 2021-03-15T22:29:56.270 INFO:tasks.rados.rados.0.smithi076.stdout:2351: write oid 6 current snap is 269 2021-03-15T22:29:56.270 INFO:tasks.rados.rados.0.smithi076.stdout:2351: seq_num 807 ranges {763249=659361,2090498=582616,3295061=268188} 2021-03-15T22:29:56.274 INFO:tasks.rados.rados.0.smithi076.stdout:2351: writing smithi07694878-6 from 763249 to 1422610 tid 1 2021-03-15T22:29:56.282 INFO:tasks.rados.rados.0.smithi076.stdout:2351: writing smithi07694878-6 from 2090498 to 2673114 tid 2 2021-03-15T22:29:56.287 INFO:tasks.rados.rados.0.smithi076.stdout:2351: writing smithi07694878-6 from 3295061 to 3563249 tid 3 2021-03-15T22:29:56.287 INFO:tasks.rados.rados.0.smithi076.stdout:2352: write (excl) oid 36 current snap is 269 2021-03-15T22:29:56.287 INFO:tasks.rados.rados.0.smithi076.stdout:2352: seq_num 808 ranges {780056=734610,2148980=726274,3322484=257572} 2021-03-15T22:29:56.295 INFO:tasks.rados.rados.0.smithi076.stdout:2352: writing smithi07694878-36 from 780056 to 1514666 tid 1 2021-03-15T22:29:56.302 INFO:tasks.rados.rados.0.smithi076.stdout:2352: writing smithi07694878-36 from 2148980 to 2875254 tid 2 2021-03-15T22:29:56.307 INFO:tasks.rados.rados.0.smithi076.stdout:2352: writing smithi07694878-36 from 3322484 to 3580056 tid 3 2021-03-15T22:29:56.308 INFO:tasks.rados.rados.0.smithi076.stdout:2353: delete oid 46 current snap is 269 2021-03-15T22:29:56.308 INFO:tasks.rados.rados.0.smithi076.stdout:2350: finishing rollback tid 0 to smithi07694878-26 2021-03-15T22:29:56.308 INFO:tasks.rados.rados.0.smithi076.stdout:2350: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:29:56.308 INFO:tasks.rados.rados.0.smithi076.stdout:2350: finishing rollback tid 2 to smithi07694878-26 2021-03-15T22:29:56.309 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 824 (ObjNum 755 snap 248 seq_num 755) dirty exists 2021-03-15T22:29:56.309 INFO:tasks.rados.rados.0.smithi076.stdout:2351: finishing write tid 1 to smithi07694878-6 2021-03-15T22:29:56.309 INFO:tasks.rados.rados.0.smithi076.stdout:2349: finishing copy_from to smithi07694878-40 2021-03-15T22:29:56.309 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1703 (ObjNum 782 snap 260 seq_num 782) dirty exists 2021-03-15T22:29:56.309 INFO:tasks.rados.rados.0.smithi076.stdout:2349: finishing copy_from racing read to smithi07694878-40 2021-03-15T22:29:56.310 INFO:tasks.rados.rados.0.smithi076.stdout:2351: finishing write tid 2 to smithi07694878-6 2021-03-15T22:29:56.310 INFO:tasks.rados.rados.0.smithi076.stdout:2351: finishing write tid 3 to smithi07694878-6 2021-03-15T22:29:56.310 INFO:tasks.rados.rados.0.smithi076.stdout:2351: finishing write tid 4 to smithi07694878-6 2021-03-15T22:29:56.310 INFO:tasks.rados.rados.0.smithi076.stdout:2351: finishing write tid 5 to smithi07694878-6 2021-03-15T22:29:56.311 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 1594 (ObjNum 807 snap 269 seq_num 807) dirty exists 2021-03-15T22:29:56.311 INFO:tasks.rados.rados.0.smithi076.stdout:2351: left oid 6 (ObjNum 807 snap 269 seq_num 807) 2021-03-15T22:29:56.311 INFO:tasks.rados.rados.0.smithi076.stdout:2352: finishing write tid 1 to smithi07694878-36 2021-03-15T22:29:56.311 INFO:tasks.rados.rados.0.smithi076.stdout:2352: finishing write tid 2 to smithi07694878-36 2021-03-15T22:29:56.312 INFO:tasks.rados.rados.0.smithi076.stdout:2349: done (4 left) 2021-03-15T22:29:56.312 INFO:tasks.rados.rados.0.smithi076.stdout:2350: done (3 left) 2021-03-15T22:29:56.312 INFO:tasks.rados.rados.0.smithi076.stdout:2351: done (2 left) 2021-03-15T22:29:56.313 INFO:tasks.rados.rados.0.smithi076.stdout:2353: done (1 left) 2021-03-15T22:29:56.313 INFO:tasks.rados.rados.0.smithi076.stdout:2354: copy_from oid 28 from oid 31 current snap is 269 2021-03-15T22:29:56.313 INFO:tasks.rados.rados.0.smithi076.stdout:2355: snap_remove snap 252 2021-03-15T22:29:56.325 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[56514]: cluster 2021-03-15T22:29:55.135396+0000 mgr.y (mgr. 2021-03-15T22:29:56.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[56514]: 14144) 824 : cluster [DBG] pgmap v1112: 138 pgs: 4 remapped+peering, 8 peering, 27 active+clean+remapped, 1 active+recovering, 4 active+undersized+degraded, 10 active+undersized, 84 active+clean; 526 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 6 op/s; 56/1719 objects degraded (3.258%); 105/1719 objects misplaced (6.108%); 630 B/s, 6 objects/s recovering 2021-03-15T22:29:56.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[56514]: cluster 2021-03-15T22:29:55.239758+0000 mon.a (mon.0) 1763 : cluster [DBG] osdmap e646: 8 total, 7 up, 8 in 2021-03-15T22:29:56.326 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[56514]: cluster 2021-03-15T22:29:55.352989+0000 mon.a (mon.0) 1764 : cluster [WRN] Health check update: Degraded data redundancy: 56/1719 objects degraded (3.258%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:56.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[62156]: cluster 2021-03-15T22:29: 2021-03-15T22:29:56.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[62156]: 55.135396+0000 mgr.y (mgr.14144) 824 : cluster [DBG] pgmap v1112: 138 pgs: 4 remapped+peering, 8 peering, 27 active+clean+remapped, 1 active+recovering, 4 active+undersized+degraded, 10 active+undersized, 84 active+clean; 526 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 6 op/s; 56/1719 objects degraded (3.258%); 105/1719 objects misplaced (6.108%); 630 B/s, 6 objects/s recovering 2021-03-15T22:29:56.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[62156]: cluster 2021-03-15T22:29:56.327 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:29:56.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[62156]: T22:29:55.239758+0000 mon.a 2021-03-15T22:29:56.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[62156]: (mon.0) 1763 : cluster 2021-03-15T22:29:56.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[62156]: [DBG] osdmap e646: 8 total, 7 up, 8 in 2021-03-15T22:29:56.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:56.328 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[62156]: 03-15T22:29:55.352989+0000 mon.a 2021-03-15T22:29:56.329 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[62156]: (mon.0) 1764 : cluster [WRN] 2021-03-15T22:29:56.329 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:56 smithi076 conmon[62156]: Health check update: Degraded data redundancy: 56/1719 objects degraded (3.258%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:56.364 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:56.388 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:56.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:56.489 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:56.496 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 0, 3, 5] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 6, 7, 5] 2021-03-15T22:29:56.497 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:29:56.497 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump -f json-pretty 2021-03-15T22:29:56.511 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:56.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:56.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:56 smithi125 conmon[56739]: cluster 2021-03-15T22:29:55.135396+0000 mgr.y (mgr.14144 2021-03-15T22:29:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:56 smithi125 conmon[56739]: ) 824 : cluster [DBG] pgmap v1112: 138 pgs: 4 remapped+peering, 8 peering, 27 active+clean+remapped, 1 active+recovering, 4 active+undersized+degraded, 10 active+undersized, 84 active+clean; 526 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 2.6 MiB/s wr, 6 op/s; 56/1719 objects degraded (3.258%); 105/1719 objects misplaced (6.108%); 630 B/s, 6 objects/s recovering 2021-03-15T22:29:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:56 smithi125 conmon[56739]: cluster 2021-03-15T22:29:55.239758+0000 mon.a (mon.0) 1763 2021-03-15T22:29:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:56 smithi125 conmon[56739]: : cluster [DBG] osdmap e646: 8 total, 7 up, 8 in 2021-03-15T22:29:56.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:56 smithi125 conmon[56739]: cluster 2021-03-15T22:29:56.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:56 smithi125 conmon[56739]: 2021-03-15T22:29:55.352989+0000 mon.a (mon.0) 2021-03-15T22:29:56.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:56 smithi125 conmon[56739]: 1764 : cluster [WRN] Health check update: Degraded data redundancy: 56/1719 objects degraded (3.258%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:29:56.614 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:56.627 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:29:56.637 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:56.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:56.639 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:56.646 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:29:56.663 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:29:56.682 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:29:56.699 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:29:56.727 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:29:56.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:56.747 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:29:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:29:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:29:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:29:56.748 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:29:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:29:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:29:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:29:56.749 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:29:56.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:56.763 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:56.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:56.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:56.887 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:56.888 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:56.989 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:57.013 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:57.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:57.015 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:57.116 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:57.138 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:57.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:57.140 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:57.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:29:57.242 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:57.258 INFO:tasks.rados.rados.0.smithi076.stdout:2355: done (2 left) 2021-03-15T22:29:57.259 INFO:tasks.rados.rados.0.smithi076.stdout:2356: write oid 20 current snap is 269 2021-03-15T22:29:57.259 INFO:tasks.rados.rados.0.smithi076.stdout:2356: seq_num 810 ranges {413670=485108,1465944=729943,2977330=524801} 2021-03-15T22:29:57.264 INFO:tasks.rados.rados.0.smithi076.stdout:2356: writing smithi07694878-20 from 413670 to 898778 tid 1 2021-03-15T22:29:57.268 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:57.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:57.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:57.273 INFO:tasks.rados.rados.0.smithi076.stdout:2356: writing smithi07694878-20 from 1465944 to 2195887 tid 2 2021-03-15T22:29:57.280 INFO:tasks.rados.rados.0.smithi076.stdout:2356: writing smithi07694878-20 from 2977330 to 3502131 tid 3 2021-03-15T22:29:57.280 INFO:tasks.rados.rados.0.smithi076.stdout:2357: snap_remove snap 211 2021-03-15T22:29:57.371 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:57.393 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:57.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:57.394 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:57.496 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:57.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[56514]: cluster 2021-03-15T22:29:55.737367+0000 osd.5 2021-03-15T22:29:57.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[56514]: (osd.5) 3 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:29:57.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[56514]: cluster 2021-03-15T22:29:56.094978+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:57.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[56514]: cluster 2021-03-15T22:29:56.095502+0000 osd.0 (osd.0) 97 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:57.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[56514]: cluster 2021-03-15T22:29:56.187201+0000 osd.1 (osd.1) 127 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:29:57.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[56514]: cluster 2021-03-15T22:29:56.239165+0000 mon.a (mon.0) 1765 : cluster [DBG] osdmap e647: 8 total, 7 up, 8 in 2021-03-15T22:29:57.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[56514]: cluster 2021-03-15T22:29:56.479411+0000 osd.0 (osd.0) 98 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:29:57.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[56514]: cluster 2021-03-15T22:29:56.525818+0000 osd.3 (osd.3) 119 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:29:57.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[56514]: audit 2021-03-15T22:29:56.607028+0000 mon.a (mon.0) 1766 : audit [DBG] from='client.? 172.21.15.76:0/4153320506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:57.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: cluster 2021-03-15T22:29:57.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 2021-03-15T22:29:57.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 2021 2021-03-15T22:29:57.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: - 2021-03-15T22:29:57.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 03 2021-03-15T22:29:57.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: - 2021-03-15T22:29:57.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 15 2021-03-15T22:29:57.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: T 2021-03-15T22:29:57.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 22 2021-03-15T22:29:57.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: : 2021-03-15T22:29:57.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 29 2021-03-15T22:29:57.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: : 2021-03-15T22:29:57.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 55 2021-03-15T22:29:57.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: . 2021-03-15T22:29:57.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 737367 2021-03-15T22:29:57.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: +0000 2021-03-15T22:29:57.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 2021-03-15T22:29:57.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: osd.5 2021-03-15T22:29:57.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: ( 2021-03-15T22:29:57.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: osd 2021-03-15T22:29:57.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: . 2021-03-15T22:29:57.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 5 2021-03-15T22:29:57.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: ) 2021-03-15T22:29:57.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 3 2021-03-15T22:29:57.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: : 2021-03-15T22:29:57.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: cluster 2021-03-15T22:29:57.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 2021-03-15T22:29:57.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: [DBG] 2021-03-15T22:29:57.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 2021-03-15T22:29:57.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 5.6 scrub ok 2021-03-15T22:29:57.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 2021-03-15T22:29:57.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: cluster 2021-03-15T22:29:57.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 2021-03-15T22:29:57.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 2021 2021-03-15T22:29:57.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: - 2021-03-15T22:29:57.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 03-15T22:29: 2021-03-15T22:29:57.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 56.094978+0000 osd.0 (osd.0) 96 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:57.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:29:57.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: -15T22:29:56.095502+0000 osd.0 (osd.0 2021-03-15T22:29:57.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: ) 97 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:57.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:29:57.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 03-15T22:29:56.187201+0000 2021-03-15T22:29:57.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: osd.1 (osd.1) 127 : cluster [DBG] 2021-03-15T22:29:57.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 4.7 scrub ok 2021-03-15T22:29:57.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: cluster 2021-03-15T22:29:56.239165+0000 mon.a (mon.0) 1765 : cluster [DBG] osdmap e647: 8 total, 7 up, 8 in 2021-03-15T22:29:57.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:29:57.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 22:29:56.479411+0000 osd.0 (osd 2021-03-15T22:29:57.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: .0) 98 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:29:57.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:29:57.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: -03-15T22:29:56.525818 2021-03-15T22:29:57.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: +0000 osd.3 (osd.3) 119 : cluster 2021-03-15T22:29:57.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: [DBG] 5.13 scrub ok 2021-03-15T22:29:57.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:29:57.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: 29:56.607028+0000 mon.a (mon.0 2021-03-15T22:29:57.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:57 smithi076 conmon[62156]: ) 1766 : audit [DBG] from='client.? 172.21.15.76:0/4153320506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:57.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:57.537 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:57.537 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:57.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: cluster 2021-03-15T22:29:55.737367+0000 osd.5 ( 2021-03-15T22:29:57.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: osd.5) 3 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:29:57.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: cluster 2021-03-15T22:29:56.094978+0000 osd.0 (osd.0) 2021-03-15T22:29:57.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: 96 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:29:57.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: cluster 2021-03-15T22:29:56 2021-03-15T22:29:57.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: .095502+0000 osd.0 (osd.0) 97 : 2021-03-15T22:29:57.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: cluster [DBG] purged_snaps scrub ok 2021-03-15T22:29:57.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:29:57.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: -15T22:29:56.187201+0000 osd.1 2021-03-15T22:29:57.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: (osd.1) 127 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:29:57.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:29:57.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: 15T22:29:56.239165+0000 mon.a (mon.0 2021-03-15T22:29:57.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: ) 1765 : cluster [DBG] osdmap e647: 8 total, 7 up, 8 in 2021-03-15T22:29:57.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:29:57.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: -03-15T22:29:56.479411+0000 2021-03-15T22:29:57.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: osd.0 (osd.0) 98 : cluster [DBG] 2021-03-15T22:29:57.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: 5.8 scrub ok 2021-03-15T22:29:57.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: cluster 2021-03-15T22:29 2021-03-15T22:29:57.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: :56.525818+0000 osd.3 (osd.3) 2021-03-15T22:29:57.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: 119 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:29:57.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: audit 2021 2021-03-15T22:29:57.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: -03-15T22:29:56.607028+0000 2021-03-15T22:29:57.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: mon.a (mon.0) 1766 : audit 2021-03-15T22:29:57.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:57 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/4153320506' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:29:57.638 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:57.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:57.661 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:57.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:57.763 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:57.785 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:57.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:57.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:57.793 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:57.817 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:29:57.818 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:29:57.826 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:29:57.888 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:57.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:57.911 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:57.912 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:58.013 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:29:58.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:58.037 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:58.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:29:58.139 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:58.164 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:58.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:58.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:58.250 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:29:58.251 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:29:58.261 INFO:tasks.rados.rados.0.smithi076.stdout:2357: done (3 left) 2021-03-15T22:29:58.261 INFO:tasks.rados.rados.0.smithi076.stdout:2358: delete oid 4 current snap is 269 2021-03-15T22:29:58.261 INFO:tasks.rados.rados.0.smithi076.stdout:2352: finishing write tid 3 to smithi07694878-36 2021-03-15T22:29:58.261 INFO:tasks.rados.rados.0.smithi076.stdout:2352: finishing write tid 4 to smithi07694878-36 2021-03-15T22:29:58.262 INFO:tasks.rados.rados.0.smithi076.stdout:2352: finishing write tid 5 to smithi07694878-36 2021-03-15T22:29:58.262 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 1542 (ObjNum 808 snap 269 seq_num 808) dirty exists 2021-03-15T22:29:58.262 INFO:tasks.rados.rados.0.smithi076.stdout:2352: left oid 36 (ObjNum 808 snap 269 seq_num 808) 2021-03-15T22:29:58.262 INFO:tasks.rados.rados.0.smithi076.stdout:2354: finishing copy_from to smithi07694878-28 2021-03-15T22:29:58.263 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 1320 (ObjNum 783 snap 260 seq_num 783) dirty exists 2021-03-15T22:29:58.263 INFO:tasks.rados.rados.0.smithi076.stdout:2354: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:29:58.263 INFO:tasks.rados.rados.0.smithi076.stdout:2356: finishing write tid 1 to smithi07694878-20 2021-03-15T22:29:58.263 INFO:tasks.rados.rados.0.smithi076.stdout:2356: finishing write tid 2 to smithi07694878-20 2021-03-15T22:29:58.263 INFO:tasks.rados.rados.0.smithi076.stdout:2356: finishing write tid 3 to smithi07694878-20 2021-03-15T22:29:58.264 INFO:tasks.rados.rados.0.smithi076.stdout:2356: finishing write tid 4 to smithi07694878-20 2021-03-15T22:29:58.264 INFO:tasks.rados.rados.0.smithi076.stdout:2356: finishing write tid 5 to smithi07694878-20 2021-03-15T22:29:58.264 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 1540 (ObjNum 810 snap 269 seq_num 810) dirty exists 2021-03-15T22:29:58.264 INFO:tasks.rados.rados.0.smithi076.stdout:2356: left oid 20 (ObjNum 810 snap 269 seq_num 810) 2021-03-15T22:29:58.266 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:58.267 INFO:tasks.rados.rados.0.smithi076.stdout:2352: done (3 left) 2021-03-15T22:29:58.268 INFO:tasks.rados.rados.0.smithi076.stdout:2354: done (2 left) 2021-03-15T22:29:58.268 INFO:tasks.rados.rados.0.smithi076.stdout:2356: done (1 left) 2021-03-15T22:29:58.268 INFO:tasks.rados.rados.0.smithi076.stdout:2358: done (0 left) 2021-03-15T22:29:58.268 INFO:tasks.rados.rados.0.smithi076.stdout:2359: read oid 17 snap -1 2021-03-15T22:29:58.269 INFO:tasks.rados.rados.0.smithi076.stdout:2359: expect (ObjNum 772 snap 255 seq_num 772) 2021-03-15T22:29:58.269 INFO:tasks.rados.rados.0.smithi076.stdout:2360: snap_remove snap 217 2021-03-15T22:29:58.293 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:58.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:58.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:58.396 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:58.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:58.421 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:58.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:58.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[56514]: cluster 2021-03-15T22:29:56.742205+0000 osd.5 (osd.5) 4 : cluster [DBG] 7.e scrub ok 2021-03-15T22:29:58.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[56514]: cluster 2021-03-15T22:29:56.950490+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:29:58.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[56514]: cluster 2021-03-15T22:29:57.135954+0000 mgr.y (mgr.14144) 825 : cluster [DBG] pgmap v1115: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 483 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 3.0 MiB/s wr, 17 op/s; 56/1677 objects degraded (3.339%); 123/1677 objects misplaced (7.335%); 422 B/s, 6 objects/s recovering 2021-03-15T22:29:58.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[56514]: cluster 2021-03-15T22:29:57.169295+0000 osd.1 (osd.1) 128 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:29:58.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[56514]: cluster 2021-03-15T22:29:57.251208+0000 mon.a (mon.0) 1767 : cluster [DBG] osdmap e648: 8 total, 7 up, 8 in 2021-03-15T22:29:58.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: cluster 2021-03-15T22:29:56.742205+0000 osd.5 (osd 2021-03-15T22:29:58.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: .5) 4 : cluster [DBG] 7.e scrub ok 2021-03-15T22:29:58.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: cluster 2021-03-15T22:29:56.950490+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:29:58.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:29:58.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: T22:29:57.135954+0000 mgr.y (mgr.14144) 825 : cluster [DBG] pgmap v1115: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 483 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 3.0 MiB/s wr, 17 op/s; 56/1677 objects degraded (3.339%); 123/1677 objects misplaced (7.335%); 422 B/s, 6 objects/s recovering 2021-03-15T22:29:58.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:29:58.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: -03-15T22:29:57. 2021-03-15T22:29:58.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: 169295+0000 osd.1 (osd.1) 128 : cluster 2021-03-15T22:29:58.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: [DBG] 4.1c scrub ok 2021-03-15T22:29:58.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:29:58.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: 15T22:29:57.251208+0000 2021-03-15T22:29:58.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: mon.a (mon.0) 1767 : cluster [DBG] 2021-03-15T22:29:58.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:58 smithi076 conmon[62156]: osdmap e648: 8 total, 7 up, 8 in 2021-03-15T22:29:58.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:58 smithi125 conmon[56739]: cluster 2021-03-15T22:29:56.742205+0000 osd.5 (osd.5) 4 : cluster [DBG] 7.e scrub ok 2021-03-15T22:29:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:58 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:29:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:58 smithi125 conmon[56739]: 29:56.950490+0000 osd.4 (osd.4) 136 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:29:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:58 smithi125 conmon[56739]: cluster 2021-03-15T22:29:57.135954+0000 mgr.y (mgr.14144) 825 : cluster [DBG] pgmap v1115: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 483 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 9.5 MiB/s rd, 3.0 MiB/s wr, 17 op/s; 56/1677 objects degraded (3.339%); 123/1677 objects misplaced (7.335%); 422 B/s, 6 objects/s recovering 2021-03-15T22:29:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:58 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:29:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:58 smithi125 conmon[56739]: 29:57.169295+0000 osd.1 (osd.1 2021-03-15T22:29:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:58 smithi125 conmon[56739]: ) 128 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:29:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:58 smithi125 conmon[56739]: cluster 2021-03-15T22:29:58.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:58 smithi125 conmon[56739]: 2021-03-15T22:29 2021-03-15T22:29:58.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:58 smithi125 conmon[56739]: :57.251208+0000 mon.a (mon.0) 2021-03-15T22:29:58.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:58 smithi125 conmon[56739]: 1767 : cluster [DBG] osdmap e648: 8 total, 7 up, 8 in 2021-03-15T22:29:58.548 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:58.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:58.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:29:58.651 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:58.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:58.674 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:58.674 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:58.775 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:58.799 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:58.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:58.800 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:58.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:29:58.923 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:58.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:58.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:29:59.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:29:59.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:59.050 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:59.050 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:29:59.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:59.175 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:59.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:59.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:59.267 INFO:tasks.rados.rados.0.smithi076.stdout:2360: done (1 left) 2021-03-15T22:29:59.267 INFO:tasks.rados.rados.0.smithi076.stdout:2361: write (excl) oid 20 current snap is 269 2021-03-15T22:29:59.267 INFO:tasks.rados.rados.0.smithi076.stdout:2361: seq_num 812 ranges {447284=635606,1866555=733612,3315823=331461} 2021-03-15T22:29:59.273 INFO:tasks.rados.rados.0.smithi076.stdout:2361: writing smithi07694878-20 from 447284 to 1082890 tid 1 2021-03-15T22:29:59.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:59.282 INFO:tasks.rados.rados.0.smithi076.stdout:2361: writing smithi07694878-20 from 1866555 to 2600167 tid 2 2021-03-15T22:29:59.288 INFO:tasks.rados.rados.0.smithi076.stdout:2361: writing smithi07694878-20 from 3315823 to 3647284 tid 3 2021-03-15T22:29:59.288 INFO:tasks.rados.rados.0.smithi076.stdout:2362: write oid 11 current snap is 269 2021-03-15T22:29:59.288 INFO:tasks.rados.rados.0.smithi076.stdout:2359: expect (ObjNum 772 snap 255 seq_num 772) 2021-03-15T22:29:59.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:59.306 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:59.306 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:59.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:29:59.407 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:29:59.428 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:59.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:59.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:29:59.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:59 smithi076 conmon[56514]: cluster 2021-03-15T22:29:58.257470+0000 mon.a (mon.0) 1768 2021-03-15T22:29:59.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:59 smithi076 conmon[56514]: : cluster [DBG] osdmap e649: 8 total, 7 up, 8 in 2021-03-15T22:29:59.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:29:59 smithi076 conmon[56514]: cluster 2021-03-15T22:29:58.503530+0000 osd.3 (osd.3) 120 : cluster [DBG] 6.b scrub ok 2021-03-15T22:29:59.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:59 smithi076 conmon[62156]: cluster 2021-03-15T22:29:58.257470+0000 mon.a ( 2021-03-15T22:29:59.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:59 smithi076 conmon[62156]: mon.0) 1768 : cluster [DBG] osdmap e649: 8 total, 7 up, 8 in 2021-03-15T22:29:59.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:59 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:29:59.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:29:59 smithi076 conmon[62156]: 15T22:29:58.503530+0000 osd.3 (osd.3) 120 : cluster [DBG] 6.b scrub ok 2021-03-15T22:29:59.502 INFO:tasks.rados.rados.0.smithi076.stdout:2362: seq_num 813 ranges {464091=710855,1925037=477270,2943246=474970} 2021-03-15T22:29:59.508 INFO:tasks.rados.rados.0.smithi076.stdout:2362: writing smithi07694878-11 from 464091 to 1174946 tid 1 2021-03-15T22:29:59.515 INFO:tasks.rados.rados.0.smithi076.stdout:2362: writing smithi07694878-11 from 1925037 to 2402307 tid 2 2021-03-15T22:29:59.521 INFO:tasks.rados.rados.0.smithi076.stdout:2362: writing smithi07694878-11 from 2943246 to 3418216 tid 3 2021-03-15T22:29:59.521 INFO:tasks.rados.rados.0.smithi076.stdout:2359: done (2 left) 2021-03-15T22:29:59.521 INFO:tasks.rados.rados.0.smithi076.stdout:2363: copy_from oid 2 from oid 34 current snap is 269 2021-03-15T22:29:59.522 INFO:tasks.rados.rados.0.smithi076.stdout:2361: finishing write tid 1 to smithi07694878-20 2021-03-15T22:29:59.522 INFO:tasks.rados.rados.0.smithi076.stdout:2361: finishing write tid 2 to smithi07694878-20 2021-03-15T22:29:59.522 INFO:tasks.rados.rados.0.smithi076.stdout:2361: finishing write tid 3 to smithi07694878-20 2021-03-15T22:29:59.522 INFO:tasks.rados.rados.0.smithi076.stdout:2361: finishing write tid 4 to smithi07694878-20 2021-03-15T22:29:59.523 INFO:tasks.rados.rados.0.smithi076.stdout:2361: finishing write tid 5 to smithi07694878-20 2021-03-15T22:29:59.523 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 1544 (ObjNum 812 snap 269 seq_num 812) dirty exists 2021-03-15T22:29:59.523 INFO:tasks.rados.rados.0.smithi076.stdout:2361: left oid 20 (ObjNum 812 snap 269 seq_num 812) 2021-03-15T22:29:59.523 INFO:tasks.rados.rados.0.smithi076.stdout:2362: finishing write tid 1 to smithi07694878-11 2021-03-15T22:29:59.523 INFO:tasks.rados.rados.0.smithi076.stdout:2362: finishing write tid 2 to smithi07694878-11 2021-03-15T22:29:59.524 INFO:tasks.rados.rados.0.smithi076.stdout:2361: done (2 left) 2021-03-15T22:29:59.524 INFO:tasks.rados.rados.0.smithi076.stdout:2364: write (excl) oid 18 current snap is 269 2021-03-15T22:29:59.524 INFO:tasks.rados.rados.0.smithi076.stdout:2364: seq_num 814 ranges {480898=502457,1699872=737281,3087022=593876} 2021-03-15T22:29:59.526 INFO:tasks.rados.rados.0.smithi076.stdout:2364: writing smithi07694878-18 from 480898 to 983355 tid 1 2021-03-15T22:29:59.531 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:59.534 INFO:tasks.rados.rados.0.smithi076.stdout:2364: writing smithi07694878-18 from 1699872 to 2437153 tid 2 2021-03-15T22:29:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:59 smithi125 conmon[56739]: cluster 2021-03-15T22:29:58.257470+0000 mon.a 2021-03-15T22:29:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:59 smithi125 conmon[56739]: (mon.0) 1768 : cluster [DBG] osdmap e649: 8 total, 7 up, 8 in 2021-03-15T22:29:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:29:59 smithi125 conmon[56739]: cluster 2021-03-15T22:29:58.503530+0000 osd.3 (osd.3) 120 : cluster [DBG] 6.b scrub ok 2021-03-15T22:29:59.541 INFO:tasks.rados.rados.0.smithi076.stdout:2364: writing smithi07694878-18 from 3087022 to 3680898 tid 3 2021-03-15T22:29:59.542 INFO:tasks.rados.rados.0.smithi076.stdout:2365: snap_remove snap 246 2021-03-15T22:29:59.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:59.556 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:59.558 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:59.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:29:59.682 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:59.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:59.683 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:29:59.784 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:29:59.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:59.810 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:29:59.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:29:59.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:29:59.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:29:59.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:29:59.936 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:29:59.937 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:00.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:00.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:00.062 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:00.062 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:00.163 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:00.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:00.188 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:00.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:00.214 INFO:tasks.rados.rados.0.smithi076.stdout:2365: done (3 left) 2021-03-15T22:30:00.215 INFO:tasks.rados.rados.0.smithi076.stdout:2366: copy_from oid 46 from oid 22 current snap is 269 2021-03-15T22:30:00.215 INFO:tasks.rados.rados.0.smithi076.stdout:2367: rollback oid 28 current snap is 269 2021-03-15T22:30:00.215 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 260 2021-03-15T22:30:00.216 INFO:tasks.rados.rados.0.smithi076.stdout:2368: copy_from oid 3 from oid 32 current snap is 269 2021-03-15T22:30:00.216 INFO:tasks.rados.rados.0.smithi076.stdout:2369: snap_create 2021-03-15T22:30:00.216 INFO:tasks.rados.rados.0.smithi076.stdout:2362: finishing write tid 3 to smithi07694878-11 2021-03-15T22:30:00.216 INFO:tasks.rados.rados.0.smithi076.stdout:2362: finishing write tid 4 to smithi07694878-11 2021-03-15T22:30:00.217 INFO:tasks.rados.rados.0.smithi076.stdout:2362: finishing write tid 5 to smithi07694878-11 2021-03-15T22:30:00.217 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1547 (ObjNum 813 snap 269 seq_num 813) dirty exists 2021-03-15T22:30:00.217 INFO:tasks.rados.rados.0.smithi076.stdout:2362: left oid 11 (ObjNum 813 snap 269 seq_num 813) 2021-03-15T22:30:00.217 INFO:tasks.rados.rados.0.smithi076.stdout:2364: finishing write tid 1 to smithi07694878-18 2021-03-15T22:30:00.217 INFO:tasks.rados.rados.0.smithi076.stdout:2364: finishing write tid 2 to smithi07694878-18 2021-03-15T22:30:00.218 INFO:tasks.rados.rados.0.smithi076.stdout:2363: finishing copy_from to smithi07694878-2 2021-03-15T22:30:00.218 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1356 (ObjNum 764 snap 254 seq_num 764) dirty exists 2021-03-15T22:30:00.218 INFO:tasks.rados.rados.0.smithi076.stdout:2363: finishing copy_from racing read to smithi07694878-2 2021-03-15T22:30:00.219 INFO:tasks.rados.rados.0.smithi076.stdout:2364: finishing write tid 3 to smithi07694878-18 2021-03-15T22:30:00.219 INFO:tasks.rados.rados.0.smithi076.stdout:2364: finishing write tid 4 to smithi07694878-18 2021-03-15T22:30:00.219 INFO:tasks.rados.rados.0.smithi076.stdout:2364: finishing write tid 5 to smithi07694878-18 2021-03-15T22:30:00.219 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 1049 (ObjNum 814 snap 269 seq_num 814) dirty exists 2021-03-15T22:30:00.219 INFO:tasks.rados.rados.0.smithi076.stdout:2364: left oid 18 (ObjNum 814 snap 269 seq_num 814) 2021-03-15T22:30:00.233 INFO:tasks.rados.rados.0.smithi076.stdout:2367: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:30:00.233 INFO:tasks.rados.rados.0.smithi076.stdout:2367: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:30:00.233 INFO:tasks.rados.rados.0.smithi076.stdout:2367: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:30:00.233 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 1842 (ObjNum 609 snap 189 seq_num 609) dirty exists 2021-03-15T22:30:00.247 INFO:tasks.rados.rados.0.smithi076.stdout:2366: finishing copy_from to smithi07694878-46 2021-03-15T22:30:00.247 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 1512 (ObjNum 750 snap 245 seq_num 750) dirty exists 2021-03-15T22:30:00.248 INFO:tasks.rados.rados.0.smithi076.stdout:2368: finishing copy_from to smithi07694878-3 2021-03-15T22:30:00.248 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 1540 (ObjNum 767 snap 255 seq_num 767) dirty exists 2021-03-15T22:30:00.248 INFO:tasks.rados.rados.0.smithi076.stdout:2366: finishing copy_from racing read to smithi07694878-46 2021-03-15T22:30:00.249 INFO:tasks.rados.rados.0.smithi076.stdout:2368: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:30:00.287 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:29:59 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:30:00.290 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:00.312 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:00.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:00.314 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:00.364 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:29:58.713658+0000 osd.5 (osd.5) 5 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:30:00.365 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:29:59.136412+0000 mgr.y (mgr.14144) 826 : cluster [DBG] pgmap v1118: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 477 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 56/1668 objects degraded (3.357%); 123/1668 objects misplaced (7.374%); 422 B/s, 6 objects/s recovering 2021-03-15T22:30:00.365 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:29:59.264673+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e650: 8 total, 7 up, 8 in 2021-03-15T22:30:00.365 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:29:59.529152+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.b scrub ok 2021-03-15T22:30:00.365 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: audit 2021-03-15T22:29:59.967455+0000 mgr.y (mgr.14144) 827 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:00.366 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000123+0000 mon.a (mon.0) 1770 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 56/1668 objects degraded (3.357%), 4 pgs degraded, 14 pgs undersized 2021-03-15T22:30:00.366 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000150+0000 mon.a (mon.0) 1771 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2021-03-15T22:30:00.366 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000169+0000 mon.a (mon.0) 1772 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2021-03-15T22:30:00.366 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000181+0000 mon.a (mon.0) 1773 : cluster [WRN] osd.2 (root=default,host=smithi076) is down 2021-03-15T22:30:00.367 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000199+0000 mon.a (mon.0) 1774 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 56/1668 objects degraded (3.357%), 4 pgs degraded, 14 pgs undersized 2021-03-15T22:30:00.367 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000210+0000 mon.a (mon.0) 1775 : cluster [WRN] pg 3.0 is stuck undersized for 3m, current state active+undersized, last acting [1,6] 2021-03-15T22:30:00.367 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000223+0000 mon.a (mon.0) 1776 : cluster [WRN] pg 3.9 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2021-03-15T22:30:00.367 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000235+0000 mon.a (mon.0) 1777 : cluster [WRN] pg 3.13 is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2021-03-15T22:30:00.368 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000246+0000 mon.a (mon.0) 1778 : cluster [WRN] pg 3.14 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2021-03-15T22:30:00.368 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000258+0000 mon.a (mon.0) 1779 : cluster [WRN] pg 3.1a is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2021-03-15T22:30:00.368 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000269+0000 mon.a (mon.0) 1780 : cluster [WRN] pg 4.a is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,7] 2021-03-15T22:30:00.368 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000281+0000 mon.a (mon.0) 1781 : cluster [WRN] pg 4.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,1] 2021-03-15T22:30:00.368 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:30:00.369 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: 30:00.000290+0000 mon.a (mon.0 2021-03-15T22:30:00.369 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: ) 1782 : cluster [WRN] pg 4.12 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,6] 2021-03-15T22:30:00.369 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:30:00.369 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: 15T22:30:00.000301+0000 mon.a ( 2021-03-15T22:30:00.370 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: mon.0) 1783 : cluster [WRN] pg 6.1 is stuck undersized for 3m, current state active+undersized, last acting [1,6] 2021-03-15T22:30:00.370 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:30:00.370 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: 03-15T22:30:00.000311+0000 2021-03-15T22:30:00.370 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: mon.a (mon.0) 1784 : cluster [WRN] 2021-03-15T22:30:00.370 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: pg 6.3 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2021-03-15T22:30:00.371 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:30:00.371 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: 30:00.000324+0000 mon.a (mon. 2021-03-15T22:30:00.371 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: 0) 1785 : cluster [WRN] pg 6.e is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2021-03-15T22:30:00.371 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:30:00.372 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: 03-15T22:30:00.000336+0000 2021-03-15T22:30:00.372 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: mon.a (mon.0) 1786 : cluster [WRN] 2021-03-15T22:30:00.373 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: pg 6.12 is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2021-03-15T22:30:00.373 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.000347+0000 mon.a ( 2021-03-15T22:30:00.373 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: mon.0) 1787 : cluster [WRN] pg 6.14 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2021-03-15T22:30:00.373 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:30:00.374 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: 03-15T22:30:00.000360+0000 2021-03-15T22:30:00.375 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: mon.a (mon.0) 1788 : cluster [WRN] 2021-03-15T22:30:00.376 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: pg 7.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,4] 2021-03-15T22:30:00.377 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:30:00.377 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: 30:00.199432+0000 mon.a (mon.0 2021-03-15T22:30:00.378 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:00 smithi125 conmon[56739]: ) 1789 : cluster [DBG] osdmap e651: 8 total, 7 up, 8 in 2021-03-15T22:30:00.385 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:00.415 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:00.426 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:00.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:00.438 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:00.438 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:00.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:29:58.713658+0000 osd.5 (osd.5) 5 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:30:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:29:59.136412+0000 mgr.y (mgr.14144) 826 : cluster [DBG] pgmap v1118: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 477 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 56/1668 objects degraded (3.357%); 123/1668 objects misplaced (7.374%); 422 B/s, 6 objects/s recovering 2021-03-15T22:30:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:29:59.264673+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e650: 8 total, 7 up, 8 in 2021-03-15T22:30:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:29:59.529152+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.b scrub ok 2021-03-15T22:30:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: audit 2021-03-15T22:29:59.967455+0000 mgr.y (mgr.14144) 827 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000123+0000 mon.a (mon.0) 1770 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 56/1668 objects degraded (3.357%), 4 pgs degraded, 14 pgs undersized 2021-03-15T22:30:00.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000150+0000 mon.a (mon.0) 1771 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2021-03-15T22:30:00.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: 2021-03-15T22:30:00.000169+0000 mon.a (mon.0) 1772 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2021-03-15T22:30:00.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000181+0000 mon.a (mon.0) 1773 : cluster [WRN] osd.2 (root=default,host=smithi076) is down 2021-03-15T22:30:00.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000199+0000 mon.a (mon.0) 1774 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 56/1668 objects degraded (3.357%), 4 pgs degraded, 14 pgs undersized 2021-03-15T22:30:00.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000210+0000 mon.a (mon.0) 1775 : cluster [WRN] pg 3.0 is stuck undersized for 3m, current state active+undersized, last acting [1,6] 2021-03-15T22:30:00.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000223+0000 mon.a (mon.0) 1776 : cluster [WRN] pg 3.9 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2021-03-15T22:30:00.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000235+0000 mon.a (mon.0) 1777 : cluster [WRN] pg 3.13 is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2021-03-15T22:30:00.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000246+0000 mon.a (mon.0) 1778 : cluster [WRN] pg 3.14 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2021-03-15T22:30:00.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000258+0000 mon.a (mon.0) 1779 : cluster [WRN] pg 3.1a is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2021-03-15T22:30:00.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000269+0000 mon.a (mon.0) 1780 : cluster [WRN] pg 4.a is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,7] 2021-03-15T22:30:00.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000281+0000 mon.a (mon.0) 1781 : cluster [WRN] pg 4.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,1] 2021-03-15T22:30:00.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000290+0000 mon.a (mon.0) 1782 : cluster [WRN] pg 4.12 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,6] 2021-03-15T22:30:00.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000301+0000 mon.a (mon.0) 1783 : cluster [WRN] pg 6.1 is stuck undersized for 3m, current state active+undersized, last acting [1,6] 2021-03-15T22:30:00.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000311+0000 mon.a (mon.0) 1784 : cluster [WRN] pg 6.3 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2021-03-15T22:30:00.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000324+0000 mon.a (mon.0) 1785 : cluster [WRN] pg 6.e is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2021-03-15T22:30:00.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000336+0000 mon.a (mon.0) 1786 : cluster [WRN] pg 6.12 is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2021-03-15T22:30:00.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000347+0000 mon.a (mon.0) 1787 : cluster [WRN] pg 6.14 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2021-03-15T22:30:00.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.000360+0000 mon.a (mon.0) 1788 : cluster [WRN] pg 7.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,4] 2021-03-15T22:30:00.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.199432+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e651: 8 total, 7 up, 8 in 2021-03-15T22:30:00.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:29:58.713658+0000 osd.5 (osd.5) 5 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:30:00.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:29:59.136412+0000 mgr.y (mgr.14144) 826 : cluster [DBG] pgmap v1118: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 477 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.5 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 56/1668 objects degraded (3.357%); 123/1668 objects misplaced (7.374%); 422 B/s, 6 objects/s recovering 2021-03-15T22:30:00.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:29:59.264673+0000 mon.a (mon.0) 1769 : cluster [DBG] osdmap e650: 8 total, 7 up, 8 in 2021-03-15T22:30:00.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:29:59.529152+0000 osd.0 (osd.0) 99 : cluster [DBG] 5.b scrub ok 2021-03-15T22:30:00.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: audit 2021-03-15T22:29:59.967455+0000 mgr.y (mgr.14144) 827 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:00.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000123+0000 mon.a (mon.0) 1770 : cluster [WRN] Health detail: HEALTH_WARN nodeep-scrub flag(s) set; 1 osds down; Degraded data redundancy: 56/1668 objects degraded (3.357%), 4 pgs degraded, 14 pgs undersized 2021-03-15T22:30:00.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000150+0000 mon.a (mon.0) 1771 : cluster [WRN] [WRN] OSDMAP_FLAGS: nodeep-scrub flag(s) set 2021-03-15T22:30:00.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000169+0000 mon.a (mon.0) 1772 : cluster [WRN] [WRN] OSD_DOWN: 1 osds down 2021-03-15T22:30:00.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000181+0000 mon.a (mon.0) 1773 : cluster [WRN] osd.2 (root=default,host=smithi076) is down 2021-03-15T22:30:00.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000199+0000 mon.a (mon.0) 1774 : cluster [WRN] [WRN] PG_DEGRADED: Degraded data redundancy: 56/1668 objects degraded (3.357%), 4 pgs degraded, 14 pgs undersized 2021-03-15T22:30:00.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000210+0000 mon.a (mon.0) 1775 : cluster [WRN] pg 3.0 is stuck undersized for 3m, current state active+undersized, last acting [1,6] 2021-03-15T22:30:00.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000223+0000 mon.a (mon.0) 1776 : cluster [WRN] pg 3.9 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2021-03-15T22:30:00.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000235+0000 mon.a (mon.0) 1777 : cluster [WRN] pg 3.13 is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2021-03-15T22:30:00.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000246+0000 mon.a (mon.0) 1778 : cluster [WRN] pg 3.14 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2021-03-15T22:30:00.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000258+0000 mon.a (mon.0) 1779 : cluster [WRN] pg 3.1a is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2021-03-15T22:30:00.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000269+0000 mon.a (mon.0) 1780 : cluster [WRN] pg 4.a is stuck undersized for 3m, current state active+undersized+degraded, last acting [6,7] 2021-03-15T22:30:00.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000281+0000 mon.a (mon.0) 1781 : cluster [WRN] pg 4.d is stuck undersized for 3m, current state active+undersized+degraded, last acting [4,1] 2021-03-15T22:30:00.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000290+0000 mon.a (mon.0) 1782 : cluster [WRN] pg 4.12 is stuck undersized for 3m, current state active+undersized+degraded, last acting [1,6] 2021-03-15T22:30:00.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000301+0000 mon.a (mon.0) 1783 : cluster [WRN] pg 6.1 is stuck undersized for 3m, current state active+undersized, last acting [1,6] 2021-03-15T22:30:00.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000311+0000 mon.a (mon.0) 1784 : cluster [WRN] pg 6.3 is stuck undersized for 3m, current state active+undersized, last acting [7,6] 2021-03-15T22:30:00.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000324+0000 mon.a (mon.0) 1785 : cluster [WRN] pg 6.e is stuck undersized for 3m, current state active+undersized, last acting [4,1] 2021-03-15T22:30:00.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000336+0000 mon.a (mon.0) 1786 : cluster [WRN] pg 6.12 is stuck undersized for 3m, current state active+undersized, last acting [7,4] 2021-03-15T22:30:00.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000347+0000 mon.a (mon.0) 1787 : cluster [WRN] pg 6.14 is stuck undersized for 3m, current state active+undersized, last acting [4,7] 2021-03-15T22:30:00.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.000360+0000 mon.a (mon.0) 1788 : cluster [WRN] pg 7.8 is stuck undersized for 3m, current state active+undersized+degraded, last acting [7,4] 2021-03-15T22:30:00.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:00 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.199432+0000 mon.a (mon.0) 1789 : cluster [DBG] osdmap e651: 8 total, 7 up, 8 in 2021-03-15T22:30:00.540 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:00.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:00.563 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:00.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:00.665 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:00.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:00.690 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:00.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:00.792 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:00.814 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:00.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:00.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:00.917 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:00.939 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:00.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:00.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:01.042 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:01.066 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:01.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:01.169 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:01.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:01.193 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:01.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:01.210 INFO:tasks.rados.rados.0.smithi076.stdout:2362: done (6 left) 2021-03-15T22:30:01.211 INFO:tasks.rados.rados.0.smithi076.stdout:2363: done (5 left) 2021-03-15T22:30:01.211 INFO:tasks.rados.rados.0.smithi076.stdout:2364: done (4 left) 2021-03-15T22:30:01.211 INFO:tasks.rados.rados.0.smithi076.stdout:2366: done (3 left) 2021-03-15T22:30:01.211 INFO:tasks.rados.rados.0.smithi076.stdout:2367: done (2 left) 2021-03-15T22:30:01.211 INFO:tasks.rados.rados.0.smithi076.stdout:2368: done (1 left) 2021-03-15T22:30:01.212 INFO:tasks.rados.rados.0.smithi076.stdout:2369: done (0 left) 2021-03-15T22:30:01.212 INFO:tasks.rados.rados.0.smithi076.stdout:2370: copy_from oid 31 from oid 28 current snap is 270 2021-03-15T22:30:01.212 INFO:tasks.rados.rados.0.smithi076.stdout:2371: copy_from oid 43 from oid 8 current snap is 270 2021-03-15T22:30:01.212 INFO:tasks.rados.rados.0.smithi076.stdout:2372: snap_remove snap 260 2021-03-15T22:30:01.215 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:30:01.295 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:01.318 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:01.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:01.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:01.421 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:01.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:01.448 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:01.448 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:01.550 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:01.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:01.575 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:01.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:01.677 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:01.699 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:01.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:01.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:01.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[56514]: cluster 2021-03-15T22:29:59.763135+0000 osd.5 (osd.5) 6 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:30:01.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[56514]: cluster 2021-03-15T22:30:00.029946+0000 osd.4 (osd.4) 137 : cluster [DBG] 4.c scrub ok 2021-03-15T22:30:01.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[56514]: audit 2021-03-15T22:30:00.982671+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? 172.21.15.76:0/1321684610' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:01.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[56514]: audit 2021-03-15T22:30:01.079696+0000 mon.a (mon.0) 1791 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:30:01.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[56514]: audit 2021-03-15T22:30:01.104005+0000 mon.a (mon.0) 1792 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:30:01.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[56514]: audit 2021-03-15T22:30:01.105501+0000 mon.a (mon.0) 1793 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:30:01.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[56514]: cluster 2021-03-15T22:30:01.202278+0000 mon.a (mon.0) 1794 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:30:01.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[56514]: audit 2021-03-15T22:30:01.207869+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? 172.21.15.76:0/1321684610' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:01.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[56514]: cluster 2021-03-15T22:30:01.207926+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e652: 8 total, 7 up, 8 in 2021-03-15T22:30:01.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[62156]: cluster 2021-03-15T22:29:59.763135+0000 osd.5 2021-03-15T22:30:01.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[62156]: (osd.5) 6 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:30:01.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[62156]: cluster 2021-03-15T22:30:00.029946+0000 osd.4 (osd.4) 137 : cluster [DBG] 4.c scrub ok 2021-03-15T22:30:01.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[62156]: audit 2021-03-15T22:30:00.982671+0000 mon.a (mon.0) 1790 : audit [INF] from='client.? 172.21.15.76:0/1321684610' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:01.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[62156]: audit 2021-03-15T22:30:01.079696+0000 mon.a (mon.0) 1791 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:30:01.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[62156]: audit 2021-03-15T22:30:01.104005+0000 mon.a (mon.0) 1792 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:30:01.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[62156]: audit 2021-03-15T22:30:01.105501+0000 mon.a (mon.0) 1793 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:30:01.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[62156]: cluster 2021-03-15T22:30:01.202278+0000 mon.a (mon.0) 1794 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:30:01.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[62156]: audit 2021-03-15T22:30:01.207869+0000 mon.a (mon.0) 1795 : audit [INF] from='client.? 172.21.15.76:0/1321684610' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:01.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:01 smithi076 conmon[62156]: cluster 2021-03-15T22:30:01.207926+0000 mon.a (mon.0) 1796 : cluster [DBG] osdmap e652: 8 total, 7 up, 8 in 2021-03-15T22:30:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:30:01.781 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:30:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 652, 2021-03-15T22:30:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:30:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2021-03-15T22:13:20.370795+0000", 2021-03-15T22:30:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2021-03-15T22:30:01.202047+0000", 2021-03-15T22:30:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2021-03-15T22:29:25.202563+0000", 2021-03-15T22:30:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2021-03-15T22:29:50.191149+0000", 2021-03-15T22:30:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-03-15T22:30:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5799936, 2021-03-15T22:30:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2021-03-15T22:30:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2021-03-15T22:30:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2021-03-15T22:30:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2021-03-15T22:30:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2021-03-15T22:30:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 19, 2021-03-15T22:30:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2021-03-15T22:30:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-03-15T22:30:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2021-03-15T22:30:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2021-03-15T22:30:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 7, 2021-03-15T22:30:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2021-03-15T22:30:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2021-03-15T22:30:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2021-03-15T22:30:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2021-03-15T22:30:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2021-03-15T22:30:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2021-03-15T22:30:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "device_health_metrics", 2021-03-15T22:30:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:16:34.817191+0000", 2021-03-15T22:30:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2021-03-15T22:30:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2021-03-15T22:30:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2021-03-15T22:30:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2021-03-15T22:30:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2021-03-15T22:30:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "23", 2021-03-15T22:30:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:30:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2021-03-15T22:30:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_devicehealth": {} 2021-03-15T22:30:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2021-03-15T22:30:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "iscsi", 2021-03-15T22:30:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:49.906400+0000", 2021-03-15T22:30:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 3, 2021-03-15T22:30:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 3, 2021-03-15T22:30:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 3, 2021-03-15T22:30:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 3, 2021-03-15T22:30:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 3, 2021-03-15T22:30:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "56", 2021-03-15T22:30:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:30:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2021-03-15T22:30:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2021-03-15T22:30:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".rgw.root", 2021-03-15T22:30:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:52.257970+0000", 2021-03-15T22:30:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:30:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:30:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:30:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:30:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:30:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "58", 2021-03-15T22:30:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:01.830 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:01.830 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:01.830 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:01.830 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:01.830 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:01.831 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:01.831 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:01.831 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:01.831 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:01.831 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:01.832 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:01.832 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:01.832 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:01.833 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:01.833 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:01.833 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:01.833 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:01.833 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:01.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.834 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:01.834 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:01.834 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:01.835 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:01.835 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:01.835 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:01.835 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:01.835 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:01.836 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:01.836 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:01.836 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:01.836 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:30:01.837 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:01.837 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:30:01.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.838 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2021-03-15T22:30:01.838 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.log", 2021-03-15T22:30:01.838 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:54.562498+0000", 2021-03-15T22:30:01.838 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:01.839 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:01.839 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:01.839 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:01.839 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:01.839 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:01.840 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:01.840 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:01.840 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:01.840 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:01.840 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:01.841 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:01.841 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:30:01.841 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:30:01.841 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:30:01.842 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:30:01.842 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:30:01.842 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:01.842 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:01.843 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:01.843 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:01.843 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:01.843 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:01.843 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:01.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.844 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "60", 2021-03-15T22:30:01.844 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:01.844 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:01.844 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:01.845 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:01.845 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:01.845 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:01.845 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:01.845 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:01.846 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:01.846 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:01.846 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:01.846 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:01.847 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:01.847 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:01.847 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:01.847 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:01.847 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:01.848 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:01.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:01.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:01.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:01.848 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:01.849 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:01.849 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:01.849 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:01.849 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:01.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.850 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:01.850 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:01.850 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:01.850 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:01.851 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:01.851 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:01.851 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:01.851 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:01.851 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:01.852 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:01.852 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:01.852 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:30:01.852 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:01.852 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:30:01.853 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.854 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 5, 2021-03-15T22:30:01.854 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.control", 2021-03-15T22:30:01.854 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:56.243806+0000", 2021-03-15T22:30:01.854 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:01.854 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:01.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:01.855 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:01.855 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:01.855 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:01.856 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:01.856 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:01.856 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:01.856 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:01.856 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:01.857 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:01.857 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 22, 2021-03-15T22:30:01.857 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 22, 2021-03-15T22:30:01.857 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 22, 2021-03-15T22:30:01.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 22, 2021-03-15T22:30:01.858 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 22, 2021-03-15T22:30:01.858 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:01.858 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "5.16", 2021-03-15T22:30:01.858 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 182, 2021-03-15T22:30:01.859 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 181, 2021-03-15T22:30:01.859 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 181, 2021-03-15T22:30:01.859 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:01.859 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:01.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.860 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "182", 2021-03-15T22:30:01.860 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:01.860 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "182", 2021-03-15T22:30:01.860 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "180", 2021-03-15T22:30:01.861 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:01.861 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:01.861 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:01.861 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:01.862 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:01.862 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:01.862 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:01.862 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:01.862 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:01.863 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:01.863 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:01.863 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:01.863 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:01.864 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:01.864 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:01.864 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:01.864 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:01.864 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:01.865 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:01.865 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:01.865 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:01.865 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:01.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:01.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.866 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:01.866 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:01.866 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:01.867 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:01.867 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:01.867 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:01.867 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:01.867 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:01.868 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:01.868 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:01.868 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:01.868 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:30:01.868 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:01.869 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:30:01.869 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.870 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 6, 2021-03-15T22:30:01.870 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.meta", 2021-03-15T22:30:01.870 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:58.730534+0000", 2021-03-15T22:30:01.870 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:01.871 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:01.871 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:01.871 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:01.871 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:01.872 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:01.872 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:01.872 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:01.872 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:01.873 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:01.873 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:01.873 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:01.873 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:30:01.873 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:30:01.874 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:30:01.874 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:30:01.874 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:30:01.874 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:01.875 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:01.875 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:01.875 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:01.875 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:01.876 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:01.876 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:01.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.876 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "109", 2021-03-15T22:30:01.876 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:01.877 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:01.877 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:01.877 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:01.877 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:01.877 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:01.878 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:01.878 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:01.878 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:01.878 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:01.879 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:01.879 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:01.879 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:01.879 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:01.879 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:01.880 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:01.880 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:01.880 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:01.880 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:01.880 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:01.881 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:01.881 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:01.881 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:01.881 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:01.881 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:01.882 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:01.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.882 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:01.882 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:30:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_bias": 4, 2021-03-15T22:30:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 8 2021-03-15T22:30:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:30:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:30:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2021-03-15T22:30:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:20:03.377294+0000", 2021-03-15T22:30:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2021-03-15T22:30:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-03-15T22:30:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2021-03-15T22:30:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2021-03-15T22:30:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2021-03-15T22:30:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2021-03-15T22:30:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2021-03-15T22:30:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "652", 2021-03-15T22:30:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:01.894 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:01.894 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:01.894 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 517, 2021-03-15T22:30:01.894 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 652, 2021-03-15T22:30:01.894 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:01.895 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:01.895 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:01.895 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:01.895 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:01.896 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:01.896 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:01.896 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:01.896 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:01.896 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:01.897 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:01.897 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:01.897 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:01.897 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:01.897 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:01.898 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:01.898 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:01.898 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:01.898 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:01.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.899 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:01.899 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:01.899 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:01.899 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:01.900 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:01.900 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:01.900 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:01.900 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:01.900 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:01.901 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:01.901 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:01.901 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:30:01.901 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:01.901 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2021-03-15T22:30:01.902 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.902 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.902 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:01.902 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:01.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.903 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:30:01.903 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd", 2021-03-15T22:30:01.903 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:01.903 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:01.904 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:01.904 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:01.904 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:01.904 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:01.905 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 8, 2021-03-15T22:30:01.905 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 642, 2021-03-15T22:30:01.905 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:01.905 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:01.905 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:01.906 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.906 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2021-03-15T22:30:01.907 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:01.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.907 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.907 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2021-03-15T22:30:01.908 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:01.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.908 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:01.909 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.909 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.909 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2021-03-15T22:30:01.909 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:01.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.910 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2021-03-15T22:30:01.911 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:01.911 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.911 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.911 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:01.912 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.912 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.912 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2021-03-15T22:30:01.913 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:01.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.913 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.913 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2021-03-15T22:30:01.914 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:01.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.914 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.915 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:01.915 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.915 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.915 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2021-03-15T22:30:01.916 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:01.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.916 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.916 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2021-03-15T22:30:01.917 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:01.917 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.917 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.918 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/188228176", 2021-03-15T22:30:01.918 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/188228176", 2021-03-15T22:30:01.918 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/188228176", 2021-03-15T22:30:01.918 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/188228176", 2021-03-15T22:30:01.918 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:01.919 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:01.919 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:01.919 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.920 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:30:01.920 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526", 2021-03-15T22:30:01.920 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:01.920 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:01.921 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:01.921 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.999237060546875, 2021-03-15T22:30:01.921 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:01.921 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:01.921 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 13, 2021-03-15T22:30:01.922 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 642, 2021-03-15T22:30:01.922 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:01.922 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:01.922 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:01.922 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.923 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.923 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2021-03-15T22:30:01.923 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:01.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.924 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.924 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2021-03-15T22:30:01.925 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:01.925 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.925 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.925 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:01.926 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.926 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.926 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2021-03-15T22:30:01.927 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:01.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.927 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.927 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2021-03-15T22:30:01.928 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:01.928 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.928 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.928 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:01.929 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.929 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.929 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2021-03-15T22:30:01.930 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:01.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.930 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.930 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2021-03-15T22:30:01.931 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:01.931 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.931 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.932 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:01.932 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.932 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.933 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2021-03-15T22:30:01.933 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:01.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.933 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.934 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2021-03-15T22:30:01.934 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:01.934 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.934 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.935 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/3924955423", 2021-03-15T22:30:01.935 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/3924955423", 2021-03-15T22:30:01.935 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/3924955423", 2021-03-15T22:30:01.935 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/3924955423", 2021-03-15T22:30:01.936 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:01.936 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:01.936 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:01.936 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:30:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05", 2021-03-15T22:30:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 0, 2021-03-15T22:30:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:01.938 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:01.938 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:01.938 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:01.938 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:01.939 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 18, 2021-03-15T22:30:01.939 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 421, 2021-03-15T22:30:01.939 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 490, 2021-03-15T22:30:01.939 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:01.939 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:01.940 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.940 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.940 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2021-03-15T22:30:01.941 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:30:01.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.941 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.941 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2021-03-15T22:30:01.942 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:30:01.942 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.942 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.942 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:01.943 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.943 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.943 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2021-03-15T22:30:01.944 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:30:01.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.944 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.944 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2021-03-15T22:30:01.945 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:30:01.945 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.945 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.945 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:01.946 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.946 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.946 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2021-03-15T22:30:01.947 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:30:01.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.947 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.947 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2021-03-15T22:30:01.948 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:30:01.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.948 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.949 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:01.949 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.949 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.949 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2021-03-15T22:30:01.950 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:30:01.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.951 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2021-03-15T22:30:01.951 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1401263050 2021-03-15T22:30:01.951 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.951 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.952 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/1401263050", 2021-03-15T22:30:01.952 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/1401263050", 2021-03-15T22:30:01.952 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/1401263050", 2021-03-15T22:30:01.952 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/1401263050", 2021-03-15T22:30:01.953 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:01.953 INFO:teuthology.orchestra.run.smithi076.stdout: "exists" 2021-03-15T22:30:01.953 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.954 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:30:01.954 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56", 2021-03-15T22:30:01.954 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:01.954 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:01.954 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:01.955 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:01.955 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:01.955 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:01.955 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 26, 2021-03-15T22:30:01.956 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 642, 2021-03-15T22:30:01.956 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:01.956 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:01.956 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:01.956 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.957 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.957 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2021-03-15T22:30:01.957 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:01.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.958 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.958 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2021-03-15T22:30:01.958 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:01.959 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.959 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.959 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:01.959 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.960 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.960 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2021-03-15T22:30:01.960 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:01.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.961 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.961 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2021-03-15T22:30:01.961 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:01.962 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.962 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:01.962 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.963 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.963 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2021-03-15T22:30:01.963 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:01.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.964 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.964 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2021-03-15T22:30:01.964 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:01.965 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.965 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.965 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:01.965 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.966 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.966 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2021-03-15T22:30:01.966 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:01.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.967 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.967 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2021-03-15T22:30:01.967 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:01.968 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.968 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.968 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/3960968636", 2021-03-15T22:30:01.968 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/3960968636", 2021-03-15T22:30:01.969 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/3960968636", 2021-03-15T22:30:01.969 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/3960968636", 2021-03-15T22:30:01.969 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:01.969 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:01.969 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:01.970 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.970 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:30:01.971 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237", 2021-03-15T22:30:01.971 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:01.971 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:01.971 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:01.971 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:01.972 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:01.972 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:01.972 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 31, 2021-03-15T22:30:01.972 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 642, 2021-03-15T22:30:01.972 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:01.973 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:01.973 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:01.973 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.974 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.974 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6800", 2021-03-15T22:30:01.974 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:01.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.975 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.975 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6801", 2021-03-15T22:30:01.975 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:01.975 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.975 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.976 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:01.976 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.977 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.977 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6802", 2021-03-15T22:30:01.977 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:01.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.978 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.978 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6803", 2021-03-15T22:30:01.978 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:01.978 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.978 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.979 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:01.979 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.980 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.980 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6806", 2021-03-15T22:30:01.980 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:01.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.981 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.981 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6807", 2021-03-15T22:30:01.981 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:01.981 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.981 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.982 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:01.982 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.983 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.983 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6804", 2021-03-15T22:30:01.983 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:01.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6805", 2021-03-15T22:30:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.985 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.985 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6801/37386122", 2021-03-15T22:30:01.985 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6803/37386122", 2021-03-15T22:30:01.985 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6807/37386122", 2021-03-15T22:30:01.986 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6805/37386122", 2021-03-15T22:30:01.986 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:01.986 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:01.986 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:01.987 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.987 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:30:01.987 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2", 2021-03-15T22:30:01.988 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:01.988 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:01.988 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:01.988 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:01.988 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 38, 2021-03-15T22:30:01.989 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 539, 2021-03-15T22:30:01.989 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 616, 2021-03-15T22:30:01.989 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 642, 2021-03-15T22:30:01.989 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 540, 2021-03-15T22:30:01.989 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:01.990 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:01.990 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.990 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.990 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6808", 2021-03-15T22:30:01.991 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:01.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.991 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.992 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6809", 2021-03-15T22:30:01.992 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:01.992 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.992 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.993 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:01.993 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.993 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.993 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6810", 2021-03-15T22:30:01.994 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:01.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.994 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.995 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6811", 2021-03-15T22:30:01.995 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:01.995 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.995 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.996 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:01.996 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.996 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:01.997 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6814", 2021-03-15T22:30:01.997 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:01.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.997 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:01.998 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6815", 2021-03-15T22:30:01.998 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:01.998 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:01.998 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:01.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:01.999 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:01.999 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:01.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:01.999 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:02.000 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6812", 2021-03-15T22:30:02.000 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:02.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.001 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:02.001 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6813", 2021-03-15T22:30:02.001 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:02.001 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.001 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.002 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6809/978017133", 2021-03-15T22:30:02.002 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6811/978017133", 2021-03-15T22:30:02.002 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6815/978017133", 2021-03-15T22:30:02.003 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6813/978017133", 2021-03-15T22:30:02.003 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:02.003 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:02.003 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:02.003 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.004 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:30:02.004 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe", 2021-03-15T22:30:02.005 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:02.005 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:02.006 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:02.006 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:02.006 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:02.006 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:02.006 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 45, 2021-03-15T22:30:02.007 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 641, 2021-03-15T22:30:02.007 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:02.007 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:02.007 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:02.008 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:02.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.008 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:02.008 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6816", 2021-03-15T22:30:02.008 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:02.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.009 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:02.009 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6817", 2021-03-15T22:30:02.009 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:02.010 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.010 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.010 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:02.011 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:02.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.011 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:02.011 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6818", 2021-03-15T22:30:02.011 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:02.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.012 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:02.012 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6819", 2021-03-15T22:30:02.012 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:02.013 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.013 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.013 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:02.013 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:02.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.014 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:02.014 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6822", 2021-03-15T22:30:02.014 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:02.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.015 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:02.015 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6823", 2021-03-15T22:30:02.015 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:02.016 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.016 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.016 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:02.016 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:02.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.017 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:02.017 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6820", 2021-03-15T22:30:02.017 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:02.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.018 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:02.018 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6821", 2021-03-15T22:30:02.019 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:02.019 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.019 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.020 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6817/171378275", 2021-03-15T22:30:02.020 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6819/171378275", 2021-03-15T22:30:02.020 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6823/171378275", 2021-03-15T22:30:02.020 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6821/171378275", 2021-03-15T22:30:02.020 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:02.021 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:02.021 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:02.021 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.022 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:30:02.022 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614", 2021-03-15T22:30:02.022 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:02.023 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:02.023 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:02.023 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:02.023 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:02.023 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:02.024 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 52, 2021-03-15T22:30:02.024 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 642, 2021-03-15T22:30:02.025 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:02.025 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:02.025 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:02.025 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:02.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.026 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:02.026 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6824", 2021-03-15T22:30:02.026 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:02.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.027 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:02.027 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6825", 2021-03-15T22:30:02.028 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:02.028 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.028 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.029 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:02.029 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:02.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.030 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:02.030 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6826", 2021-03-15T22:30:02.030 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:02.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.031 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:02.031 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6827", 2021-03-15T22:30:02.031 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:02.031 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.032 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.032 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:02.032 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:02.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.033 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:02.033 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6830", 2021-03-15T22:30:02.033 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:02.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.034 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:02.034 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6831", 2021-03-15T22:30:02.034 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:02.034 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.035 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.035 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:02.035 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:02.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.036 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:02.036 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6828", 2021-03-15T22:30:02.036 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:02.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.037 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:02.037 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6829", 2021-03-15T22:30:02.037 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:02.037 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.038 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.038 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6825/652003948", 2021-03-15T22:30:02.038 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6827/652003948", 2021-03-15T22:30:02.039 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6831/652003948", 2021-03-15T22:30:02.039 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6829/652003948", 2021-03-15T22:30:02.039 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:02.039 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:02.039 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:02.040 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.040 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.040 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:02.040 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2021-03-15T22:30:02.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.041 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:30:02.041 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:02.041 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:02.041 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:02.041 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:02.042 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:02.042 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:29:56.095504+0000", 2021-03-15T22:30:02.042 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:02.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.043 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:30:02.043 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:02.043 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:02.043 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:02.044 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:02.044 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:02.044 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:29:13.912638+0000", 2021-03-15T22:30:02.044 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:02.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.045 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:30:02.045 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:02.045 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:02.046 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:02.046 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:02.046 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:02.046 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:26:26.403924+0000", 2021-03-15T22:30:02.046 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 490 2021-03-15T22:30:02.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.047 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:30:02.047 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:02.047 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:02.048 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:02.048 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:02.048 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:02.048 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:29:42.410626+0000", 2021-03-15T22:30:02.049 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:02.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.049 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:30:02.049 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:02.050 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:02.050 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:02.050 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:02.050 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:02.050 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:29:02.469194+0000", 2021-03-15T22:30:02.051 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:02.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.051 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:30:02.052 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:02.052 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:02.052 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:02.052 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:02.052 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:02.053 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:29:24.606375+0000", 2021-03-15T22:30:02.053 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 540 2021-03-15T22:30:02.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.054 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:30:02.054 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:02.054 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:02.054 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:02.055 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:02.055 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:02.055 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:29:36.633572+0000", 2021-03-15T22:30:02.055 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:02.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.056 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:30:02.056 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:02.056 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:02.056 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:02.057 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:02.057 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:02.057 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:29:53.643176+0000", 2021-03-15T22:30:02.057 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:02.058 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.058 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:02.058 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [], 2021-03-15T22:30:02.058 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2021-03-15T22:30:02.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.059 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.3", 2021-03-15T22:30:02.059 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:30:02.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.059 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:30:02.060 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:30:02.060 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.060 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.061 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.14", 2021-03-15T22:30:02.061 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:30:02.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.061 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:30:02.062 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:30:02.062 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.062 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.063 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.a", 2021-03-15T22:30:02.063 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:30:02.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.063 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:30:02.063 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:30:02.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.064 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.065 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.b", 2021-03-15T22:30:02.065 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:30:02.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.065 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:30:02.065 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:30:02.066 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.066 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.067 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2021-03-15T22:30:02.067 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:30:02.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.067 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:30:02.067 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:30:02.068 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.068 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.068 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.068 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:02.068 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [ 2021-03-15T22:30:02.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.069 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.1", 2021-03-15T22:30:02.069 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.069 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.070 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.070 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2021-03-15T22:30:02.070 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.071 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.3", 2021-03-15T22:30:02.071 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.071 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.071 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:30:02.072 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:30:02.072 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.072 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.4", 2021-03-15T22:30:02.073 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.073 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:30:02.073 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:30:02.073 INFO:teuthology.orchestra.run.smithi076.stdout: 0 2021-03-15T22:30:02.073 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.074 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.5", 2021-03-15T22:30:02.074 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.075 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:30:02.075 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.075 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:30:02.075 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.076 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.1e", 2021-03-15T22:30:02.076 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.076 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.077 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:30:02.077 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2021-03-15T22:30:02.077 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.078 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.1f", 2021-03-15T22:30:02.078 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.078 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.078 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:30:02.078 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:30:02.079 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.079 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.4", 2021-03-15T22:30:02.080 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.080 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:30:02.080 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.080 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:30:02.080 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.081 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.6", 2021-03-15T22:30:02.081 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.082 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.082 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:30:02.082 INFO:teuthology.orchestra.run.smithi076.stdout: 7 2021-03-15T22:30:02.082 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.083 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.b", 2021-03-15T22:30:02.083 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.083 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.084 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:30:02.084 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:30:02.084 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.085 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2021-03-15T22:30:02.085 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.085 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.085 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:30:02.086 INFO:teuthology.orchestra.run.smithi076.stdout: 7 2021-03-15T22:30:02.086 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.086 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.19", 2021-03-15T22:30:02.087 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.087 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.087 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.087 INFO:teuthology.orchestra.run.smithi076.stdout: 5 2021-03-15T22:30:02.087 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.088 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.1c", 2021-03-15T22:30:02.088 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:30:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: 0 2021-03-15T22:30:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.4", 2021-03-15T22:30:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:30:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:30:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: 0 2021-03-15T22:30:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.6", 2021-03-15T22:30:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:30:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:30:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: 4 2021-03-15T22:30:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.8", 2021-03-15T22:30:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:30:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: 5 2021-03-15T22:30:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.a", 2021-03-15T22:30:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:30:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: 0 2021-03-15T22:30:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.b", 2021-03-15T22:30:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:30:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: 7 2021-03-15T22:30:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.d", 2021-03-15T22:30:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:30:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:30:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: 0 2021-03-15T22:30:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.14", 2021-03-15T22:30:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:30:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2021-03-15T22:30:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.0", 2021-03-15T22:30:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: 5 2021-03-15T22:30:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.2", 2021-03-15T22:30:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:30:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:30:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.8", 2021-03-15T22:30:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:30:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:30:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.9", 2021-03-15T22:30:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:30:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:30:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.f", 2021-03-15T22:30:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:30:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:30:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.17", 2021-03-15T22:30:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:30:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:30:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: 0 2021-03-15T22:30:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.1c", 2021-03-15T22:30:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:30:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:30:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:30:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.2", 2021-03-15T22:30:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:30:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: 7 2021-03-15T22:30:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.3", 2021-03-15T22:30:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: 4, 2021-03-15T22:30:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: 6 2021-03-15T22:30:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.5", 2021-03-15T22:30:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:30:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2021-03-15T22:30:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.a", 2021-03-15T22:30:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: 3, 2021-03-15T22:30:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:30:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: 1 2021-03-15T22:30:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.e", 2021-03-15T22:30:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:30:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:30:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:30:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2021-03-15T22:30:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": { 2021-03-15T22:30:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:30:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4087324200": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:30:02.123 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4141143304": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:30:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:30:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2779879991": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:30:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:30:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:30:02.124 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2483102204": "2021-03-16T22:13:41.603652+0000" 2021-03-15T22:30:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2021-03-15T22:30:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2021-03-15T22:30:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2021-03-15T22:30:02.125 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2021-03-15T22:30:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2021-03-15T22:30:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2021-03-15T22:30:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "ruleset-failure-domain": "osd", 2021-03-15T22:30:02.126 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2021-03-15T22:30:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2021-03-15T22:30:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.127 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:30:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:30:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 258, 2021-03-15T22:30:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 268, 2021-03-15T22:30:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 291, 2021-03-15T22:30:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 303, 2021-03-15T22:30:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 305, 2021-03-15T22:30:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.132 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 318, 2021-03-15T22:30:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 321, 2021-03-15T22:30:02.133 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 347, 2021-03-15T22:30:02.134 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 351, 2021-03-15T22:30:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 353, 2021-03-15T22:30:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 358, 2021-03-15T22:30:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.137 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 360, 2021-03-15T22:30:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.138 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 365, 2021-03-15T22:30:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:30:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 368, 2021-03-15T22:30:02.139 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 371, 2021-03-15T22:30:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 373, 2021-03-15T22:30:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 375, 2021-03-15T22:30:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 382, 2021-03-15T22:30:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 4 2021-03-15T22:30:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.143 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 388, 2021-03-15T22:30:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:30:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 391, 2021-03-15T22:30:02.144 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 394, 2021-03-15T22:30:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:30:02.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 399, 2021-03-15T22:30:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:30:02.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 403, 2021-03-15T22:30:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 3 2021-03-15T22:30:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 409, 2021-03-15T22:30:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 4 2021-03-15T22:30:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.148 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 415, 2021-03-15T22:30:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:30:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 418, 2021-03-15T22:30:02.149 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:30:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 422, 2021-03-15T22:30:02.150 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:30:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 425, 2021-03-15T22:30:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 428, 2021-03-15T22:30:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 432, 2021-03-15T22:30:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:30:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.153 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 435, 2021-03-15T22:30:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:30:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.154 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 438, 2021-03-15T22:30:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:30:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.155 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 443, 2021-03-15T22:30:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 445, 2021-03-15T22:30:02.156 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 2 2021-03-15T22:30:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 448, 2021-03-15T22:30:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 452, 2021-03-15T22:30:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 456, 2021-03-15T22:30:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 460, 2021-03-15T22:30:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 465, 2021-03-15T22:30:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.161 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 468, 2021-03-15T22:30:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.162 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 471, 2021-03-15T22:30:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 473, 2021-03-15T22:30:02.163 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 477, 2021-03-15T22:30:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 479, 2021-03-15T22:30:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 481, 2021-03-15T22:30:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 499, 2021-03-15T22:30:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:02.167 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 508, 2021-03-15T22:30:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:02.168 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [], 2021-03-15T22:30:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2021-03-15T22:30:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2021-03-15T22:30:02.169 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2021-03-15T22:30:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2021-03-15T22:30:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2021-03-15T22:30:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2021-03-15T22:30:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2021-03-15T22:30:02.170 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2021-03-15T22:30:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2021-03-15T22:30:02.171 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:02.171 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:30:02.171 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:02.173 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: cluster 2021-03-15T22:29:59.763135+0000 osd.5 (osd.5) 6 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:30:02.173 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: cluster 2021-03-15T22:30:00.029946+0000 osd.4 (osd.4) 137 : cluster [DBG] 4.c scrub ok 2021-03-15T22:30:02.173 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: audit 2021-03-15T22:30:00.982671+0000 2021-03-15T22:30:02.173 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: mon.a (mon.0) 1790 : audit [INF] from='client.? 172.21.15.76:0/1321684610' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:02.174 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:30:02.174 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: -15T22:30:01.079696+0000 2021-03-15T22:30:02.174 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: mon.a (mon.0) 1791 : audit 2021-03-15T22:30:02.175 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:30:02.175 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:30:02.175 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: 03-15T22:30:01. 2021-03-15T22:30:02.175 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: 104005+0000 mon.a (mon.0) 2021-03-15T22:30:02.175 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: 1792 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:30:02.176 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: audit 2021 2021-03-15T22:30:02.176 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: -03-15T22:30: 2021-03-15T22:30:02.176 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: 01.105501+0000 mon.a (mon 2021-03-15T22:30:02.176 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: .0) 1793 : audit [INF] 2021-03-15T22:30:02.177 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:30:02.177 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:30:02.177 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: -03-15T22:30: 2021-03-15T22:30:02.177 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: 01.202278+0000 mon.a (mon. 2021-03-15T22:30:02.177 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: 0) 1794 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:30:02.178 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: 2021-03-15T22:30:02.178 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: audit 2021-03-15T22:30:02.178 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: 2021-03-15T22:30 2021-03-15T22:30:02.178 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: :01.207869+0000 mon.a (mon 2021-03-15T22:30:02.178 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: .0) 1795 : audit [INF] 2021-03-15T22:30:02.179 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1321684610' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:02.183 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:30:02.184 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: -03-15T22:30: 2021-03-15T22:30:02.184 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: 01.207926+0000 mon.a (mon 2021-03-15T22:30:02.184 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: .0) 1796 : cluster [DBG] 2021-03-15T22:30:02.185 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:01 smithi125 conmon[56739]: osdmap e652: 8 total, 7 up, 8 in 2021-03-15T22:30:02.208 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:02.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:02.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:02.213 INFO:tasks.rados.rados.0.smithi076.stdout:2372: done (2 left) 2021-03-15T22:30:02.213 INFO:tasks.rados.rados.0.smithi076.stdout:2373: delete oid 33 current snap is 270 2021-03-15T22:30:02.213 INFO:tasks.rados.rados.0.smithi076.stdout:2370: finishing copy_from to smithi07694878-31 2021-03-15T22:30:02.214 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 1842 (ObjNum 609 snap 189 seq_num 609) dirty exists 2021-03-15T22:30:02.214 INFO:tasks.rados.rados.0.smithi076.stdout:2370: finishing copy_from racing read to smithi07694878-31 2021-03-15T22:30:02.214 INFO:tasks.rados.rados.0.smithi076.stdout:2371: finishing copy_from to smithi07694878-43 2021-03-15T22:30:02.214 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 1210 (ObjNum 649 snap 201 seq_num 649) dirty exists 2021-03-15T22:30:02.215 INFO:tasks.rados.rados.0.smithi076.stdout:2371: finishing copy_from racing read to smithi07694878-43 2021-03-15T22:30:02.220 INFO:tasks.rados.rados.0.smithi076.stdout:2370: done (2 left) 2021-03-15T22:30:02.221 INFO:tasks.rados.rados.0.smithi076.stdout:2371: done (1 left) 2021-03-15T22:30:02.221 INFO:tasks.rados.rados.0.smithi076.stdout:2373: done (0 left) 2021-03-15T22:30:02.221 INFO:tasks.rados.rados.0.smithi076.stdout:2374: read oid 18 snap -1 2021-03-15T22:30:02.221 INFO:tasks.rados.rados.0.smithi076.stdout:2374: expect (ObjNum 814 snap 269 seq_num 814) 2021-03-15T22:30:02.222 INFO:tasks.rados.rados.0.smithi076.stdout:2375: delete oid 8 current snap is 270 2021-03-15T22:30:02.227 INFO:tasks.rados.rados.0.smithi076.stdout:2375: done (1 left) 2021-03-15T22:30:02.227 INFO:tasks.rados.rados.0.smithi076.stdout:2376: rollback oid 19 current snap is 270 2021-03-15T22:30:02.228 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 19 to 223 2021-03-15T22:30:02.228 INFO:tasks.rados.rados.0.smithi076.stdout:2377: rollback oid 33 current snap is 270 2021-03-15T22:30:02.228 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 33 to 262 2021-03-15T22:30:02.228 INFO:tasks.rados.rados.0.smithi076.stdout:2378: write oid 14 current snap is 270 2021-03-15T22:30:02.229 INFO:tasks.rados.rados.0.smithi076.stdout:2378: seq_num 817 ranges {531319=728204,1758965=484608,2652938=520058} 2021-03-15T22:30:02.234 INFO:tasks.rados.rados.0.smithi076.stdout:2378: writing smithi07694878-14 from 531319 to 1259523 tid 1 2021-03-15T22:30:02.239 INFO:tasks.rados.rados.0.smithi076.stdout:2378: writing smithi07694878-14 from 1758965 to 2243573 tid 2 2021-03-15T22:30:02.245 INFO:tasks.rados.rados.0.smithi076.stdout:2378: writing smithi07694878-14 from 2652938 to 3172996 tid 3 2021-03-15T22:30:02.245 INFO:tasks.rados.rados.0.smithi076.stdout:2379: snap_create 2021-03-15T22:30:02.246 INFO:tasks.rados.rados.0.smithi076.stdout:2376: finishing rollback tid 1 to smithi07694878-19 2021-03-15T22:30:02.246 INFO:tasks.rados.rados.0.smithi076.stdout:2376: finishing rollback tid 2 to smithi07694878-19 2021-03-15T22:30:02.247 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 1045 (ObjNum 495 snap 149 seq_num 495) dirty exists 2021-03-15T22:30:02.247 INFO:tasks.rados.rados.0.smithi076.stdout:2377: finishing rollback tid 1 to smithi07694878-33 2021-03-15T22:30:02.247 INFO:tasks.rados.rados.0.smithi076.stdout:2377: finishing rollback tid 2 to smithi07694878-33 2021-03-15T22:30:02.247 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 1547 (ObjNum 714 snap 231 seq_num 714) dirty exists 2021-03-15T22:30:02.248 INFO:tasks.rados.rados.0.smithi076.stdout:2374: expect (ObjNum 814 snap 269 seq_num 814) 2021-03-15T22:30:02.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:02.333 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:02.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:02.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:02.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:02.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:02.459 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:02.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:02.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: cluster 2021-03-15T22:30:01.066450+0000 osd.4 2021-03-15T22:30:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: (osd.4) 138 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:30:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: cluster 2021-03-15T22:30:01.137093+0000 mgr.y (mgr.14144) 828 : cluster [DBG] pgmap v1121: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.4 MiB/s wr, 9 op/s; 56/1659 objects degraded (3.376%); 123/1659 objects misplaced (7.414%) 2021-03-15T22:30:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: cluster 2021-03-15T22:30:01.208841+0000 osd.7 (osd.7) 127 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:30:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: audit 2021-03-15 2021-03-15T22:30:02.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: T22:30:01.626757+0000 mon.a (mon.0) 1797 : audit [DBG] from='client.? 172.21.15.76:0/3691780395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:30:02.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:30:02.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: -03-15T22:30: 2021-03-15T22:30:02.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: 02.077863+0000 mon.a (mon. 2021-03-15T22:30:02.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: 0) 1798 : cluster [WRN] Health check update: Degraded data redundancy: 56/1659 objects degraded (3.376%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:02.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:30:02.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: 15T22:30:02.208140+0000 mon.a 2021-03-15T22:30:02.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: (mon.0) 1799 : cluster [DBG] osdmap e653: 8 total, 7 up, 8 in 2021-03-15T22:30:02.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[56514]: 2021-03-15T22:30:02.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: cluster 2021-03-15T22:30:01. 2021-03-15T22:30:02.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: 066450+0000 osd.4 (osd.4) 138 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:30:02.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:30:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: 15T22:30:01.137093 2021-03-15T22:30:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 2021-03-15T22:30:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: 828 : cluster [DBG] pgmap v1121: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.4 MiB/s wr, 9 op/s; 56/1659 objects degraded (3.376%); 123/1659 objects misplaced (7.414%) 2021-03-15T22:30:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:30:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: 03-15T22:30:01.208841+0000 osd.7 (osd 2021-03-15T22:30:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: .7) 127 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:30:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: 2021-03-15T22:30:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:30:02.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: 03-15T22:30:01. 2021-03-15T22:30:02.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: 626757+0000 mon.a (mon.0 2021-03-15T22:30:02.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: ) 1797 : audit [DBG] from='client.? 172.21.15.76:0/3691780395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:30:02.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:30:02.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: -03-15T22:30:02.077863+0000 mon.a 2021-03-15T22:30:02.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: (mon.0) 1798 : cluster [WRN] Health check update: Degraded data redundancy: 56/1659 objects degraded (3.376%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:02.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: 2021-03-15T22:30:02.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: cluster 2021-03-15T22:30:02.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:30:02.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: :30:02.208140+0000 mon.a (mon. 2021-03-15T22:30:02.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:02 smithi076 conmon[62156]: 0) 1799 : cluster [DBG] osdmap e653: 8 total, 7 up, 8 in 2021-03-15T22:30:02.526 INFO:tasks.rados.rados.0.smithi076.stdout:2378: finishing write tid 1 to smithi07694878-14 2021-03-15T22:30:02.527 INFO:tasks.rados.rados.0.smithi076.stdout:2378: finishing write tid 2 to smithi07694878-14 2021-03-15T22:30:02.527 INFO:tasks.rados.rados.0.smithi076.stdout:2378: finishing write tid 3 to smithi07694878-14 2021-03-15T22:30:02.527 INFO:tasks.rados.rados.0.smithi076.stdout:2378: finishing write tid 4 to smithi07694878-14 2021-03-15T22:30:02.527 INFO:tasks.rados.rados.0.smithi076.stdout:2378: finishing write tid 5 to smithi07694878-14 2021-03-15T22:30:02.528 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 1544 (ObjNum 817 snap 270 seq_num 817) dirty exists 2021-03-15T22:30:02.528 INFO:tasks.rados.rados.0.smithi076.stdout:2378: left oid 14 (ObjNum 817 snap 270 seq_num 817) 2021-03-15T22:30:02.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:02.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:02.584 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:02.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:02.686 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:02.707 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:02.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: cluster 2021-03-15T22:30:01.066450+0000 osd.4 (osd.4) 138 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:30:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: cluster 2021-03-15T22:30:01.137093+0000 mgr.y (mgr.14144) 828 : cluster [DBG] pgmap v1121: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 4.4 MiB/s wr, 9 op/s; 56/1659 objects degraded (3.376%); 123/1659 objects misplaced (7.414%) 2021-03-15T22:30:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: cluster 2021-03-15T22:30:01 2021-03-15T22:30:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: .208841+0000 osd.7 (osd.7) 2021-03-15T22:30:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: 127 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:30:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:30:02.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: 03-15T22:30:01.626757+0000 2021-03-15T22:30:02.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: mon.a (mon.0) 1797 : audit 2021-03-15T22:30:02.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/3691780395' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:30:02.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: cluster 2021-03-15T22:30:02.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: 2021-03-15T22:30 2021-03-15T22:30:02.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: :02.077863+0000 mon.a (mon. 2021-03-15T22:30:02.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: 0) 1798 : cluster [WRN] Health check update: Degraded data redundancy: 56/1659 objects degraded (3.376%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:02.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: 2021-03-15T22:30:02.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:30:02.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: 15T22:30:02.208140 2021-03-15T22:30:02.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1799 : 2021-03-15T22:30:02.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:02 smithi125 conmon[56739]: cluster [DBG] osdmap e653: 8 total, 7 up, 8 in 2021-03-15T22:30:02.810 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:02.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:02.838 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:02.839 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:02.941 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:02.963 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:02.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:02.964 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:03.065 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:03.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:03.090 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:03.090 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:03.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:03.213 INFO:tasks.rados.rados.0.smithi076.stdout:2374: done (4 left) 2021-03-15T22:30:03.213 INFO:tasks.rados.rados.0.smithi076.stdout:2376: done (3 left) 2021-03-15T22:30:03.213 INFO:tasks.rados.rados.0.smithi076.stdout:2377: done (2 left) 2021-03-15T22:30:03.213 INFO:tasks.rados.rados.0.smithi076.stdout:2378: done (1 left) 2021-03-15T22:30:03.214 INFO:tasks.rados.rados.0.smithi076.stdout:2379: done (0 left) 2021-03-15T22:30:03.214 INFO:tasks.rados.rados.0.smithi076.stdout:2380: copy_from oid 33 from oid 42 current snap is 271 2021-03-15T22:30:03.214 INFO:tasks.rados.rados.0.smithi076.stdout:2381: snap_remove snap 228 2021-03-15T22:30:03.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:03.221 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:03.221 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:03.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:03.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:03.346 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:03.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:03.448 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:03.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:03.472 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:03.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:03.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:03.599 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:03.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:03.601 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:03.619 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[56514]: cluster 2021-03-15T22:30:02.064717+0000 osd.4 (osd.4 2021-03-15T22:30:03.619 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[56514]: ) 139 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:30:03.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[56514]: cluster 2021-03-15T22:30:02.548064+0000 osd.3 (osd.3) 121 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:30:03.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[56514]: cluster 2021-03-15T22:30:02.644641+0000 osd.4 (osd.4) 140 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:03.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[56514]: cluster 2021-03-15T22:30:02.645138+0000 osd.4 (osd.4) 141 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:03.621 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[56514]: cluster 2021-03-15T22:30:03.210571+0000 mon.a (mon.0) 1800 : cluster [DBG] osdmap e654: 8 total, 7 up, 8 in 2021-03-15T22:30:03.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: cluster 2021-03-15T22:30:02.064717 2021-03-15T22:30:03.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 139 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:30:03.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:30:03.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: 03-15T22:30:02.548064+0000 osd.3 (osd.3) 121 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:30:03.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:30:03.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: :30:02.644641+0000 osd.4 (osd.4) 140 : cluster 2021-03-15T22:30:03.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: [DBG] purged_snaps scrub starts 2021-03-15T22:30:03.624 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: cluster 2021-03-15T22:30:03.624 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: 2021-03-15T22:30:02.645138+0000 osd.4 (osd.4) 141 : cluster 2021-03-15T22:30:03.624 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: [DBG] purged_snaps scrub ok 2021-03-15T22:30:03.624 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:30:03.625 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: -03-15T22:30:03.210571+0000 mon.a (mon.0) 1800 : cluster 2021-03-15T22:30:03.625 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:03 smithi076 conmon[62156]: [DBG] osdmap e654: 8 total, 7 up, 8 in 2021-03-15T22:30:03.628 INFO:tasks.thrashosds.thrasher:j is {'epoch': 652, 'fsid': '8a572802-85db-11eb-9091-001a4aab830c', 'created': '2021-03-15T22:13:20.370795+0000', 'modified': '2021-03-15T22:30:01.202047+0000', 'last_up_change': '2021-03-15T22:29:25.202563+0000', 'last_in_change': '2021-03-15T22:29:50.191149+0000', 'flags': 'sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5799936, 'flags_set': ['pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 19, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-15T22:16:34.817191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-15T22:17:49.906400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-15T22:17:52.257970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-15T22:17:54.562498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-15T22:17:56.243806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 182, 'last_epoch_started': 181, 'last_epoch_clean': 181, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '182', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '182', 'last_force_op_resend_preluminous': '180', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-15T22:17:58.730534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-03-15T22:20:03.377294+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '652', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 517, 'snap_epoch': 652, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 642, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 188228176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 188228176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 188228176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 188228176}]}, 'public_addr': '172.21.15.76:6803/188228176', 'cluster_addr': '172.21.15.76:6805/188228176', 'heartbeat_back_addr': '172.21.15.76:6809/188228176', 'heartbeat_front_addr': '172.21.15.76:6807/188228176', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '97bd3ffc-15c8-4a6f-b37a-217813def526', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.999237060546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 642, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 3924955423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 3924955423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 3924955423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 3924955423}]}, 'public_addr': '172.21.15.76:6811/3924955423', 'cluster_addr': '172.21.15.76:6813/3924955423', 'heartbeat_back_addr': '172.21.15.76:6817/3924955423', 'heartbeat_front_addr': '172.21.15.76:6815/3924955423', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a7f5f115-e69e-47d9-b6c3-ff386ddfaa05', 'up': 0, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 421, 'down_at': 490, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 1401263050}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 1401263050}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 1401263050}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 1401263050}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 1401263050}]}, 'public_addr': '172.21.15.76:6819/1401263050', 'cluster_addr': '172.21.15.76:6821/1401263050', 'heartbeat_back_addr': '172.21.15.76:6825/1401263050', 'heartbeat_front_addr': '172.21.15.76:6823/1401263050', 'state': ['exists']}, {'osd': 3, 'uuid': 'a0bb7f88-0114-47cb-a82c-d8fb002aac56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 642, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 3960968636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 3960968636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 3960968636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 3960968636}]}, 'public_addr': '172.21.15.76:6827/3960968636', 'cluster_addr': '172.21.15.76:6829/3960968636', 'heartbeat_back_addr': '172.21.15.76:6833/3960968636', 'heartbeat_front_addr': '172.21.15.76:6831/3960968636', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b650611c-6101-47e8-9ca9-94e02cf8b237', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 642, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6800', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6801', 'nonce': 37386122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6802', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6803', 'nonce': 37386122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6806', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6807', 'nonce': 37386122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6804', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6805', 'nonce': 37386122}]}, 'public_addr': '172.21.15.125:6801/37386122', 'cluster_addr': '172.21.15.125:6803/37386122', 'heartbeat_back_addr': '172.21.15.125:6807/37386122', 'heartbeat_front_addr': '172.21.15.125:6805/37386122', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7d6ceff2-7093-4ab9-8b96-ade448093bf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 38, 'last_clean_end': 539, 'up_from': 616, 'up_thru': 642, 'down_at': 540, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6808', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6809', 'nonce': 978017133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6810', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6811', 'nonce': 978017133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6814', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6815', 'nonce': 978017133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6812', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6813', 'nonce': 978017133}]}, 'public_addr': '172.21.15.125:6809/978017133', 'cluster_addr': '172.21.15.125:6811/978017133', 'heartbeat_back_addr': '172.21.15.125:6815/978017133', 'heartbeat_front_addr': '172.21.15.125:6813/978017133', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98b72022-a9a7-4ee2-89a1-cac037de7afe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 641, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6816', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6817', 'nonce': 171378275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6818', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6819', 'nonce': 171378275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6822', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6823', 'nonce': 171378275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6820', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6821', 'nonce': 171378275}]}, 'public_addr': '172.21.15.125:6817/171378275', 'cluster_addr': '172.21.15.125:6819/171378275', 'heartbeat_back_addr': '172.21.15.125:6823/171378275', 'heartbeat_front_addr': '172.21.15.125:6821/171378275', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f99ea650-b2dc-4d87-a44c-56a897edd614', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 642, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6824', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6825', 'nonce': 652003948}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6826', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6827', 'nonce': 652003948}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6830', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6831', 'nonce': 652003948}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6828', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6829', 'nonce': 652003948}]}, 'public_addr': '172.21.15.125:6825/652003948', 'cluster_addr': '172.21.15.125:6827/652003948', 'heartbeat_back_addr': '172.21.15.125:6831/652003948', 'heartbeat_front_addr': '172.21.15.125:6829/652003948', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:29:56.095504+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:29:13.912638+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:26:26.403924+0000', 'dead_epoch': 490}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:29:42.410626+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:29:02.469194+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:29:24.606375+0000', 'dead_epoch': 540}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:29:36.633572+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:29:53.643176+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.14', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [{'pgid': '2.1', 'osds': [3, 0, 1]}, {'pgid': '3.3', 'osds': [0, 6, 3]}, {'pgid': '3.4', 'osds': [1, 5, 0]}, {'pgid': '3.5', 'osds': [5, 3, 6]}, {'pgid': '3.1e', 'osds': [3, 6, 1]}, {'pgid': '3.1f', 'osds': [0, 5, 6]}, {'pgid': '4.4', 'osds': [1, 3, 6]}, {'pgid': '4.6', 'osds': [0, 1, 7]}, {'pgid': '4.b', 'osds': [0, 4, 3]}, {'pgid': '4.f', 'osds': [3, 4, 7]}, {'pgid': '4.19', 'osds': [3, 0, 5]}, {'pgid': '4.1c', 'osds': [1, 3, 0]}, {'pgid': '5.4', 'osds': [7, 5, 0]}, {'pgid': '5.6', 'osds': [5, 7, 4]}, {'pgid': '5.8', 'osds': [0, 1, 5]}, {'pgid': '5.a', 'osds': [4, 3, 0]}, {'pgid': '5.b', 'osds': [0, 5, 7]}, {'pgid': '5.d', 'osds': [7, 5, 0]}, {'pgid': '5.14', 'osds': [3, 7, 1]}, {'pgid': '6.0', 'osds': [0, 3, 5]}, {'pgid': '6.2', 'osds': [4, 3, 6]}, {'pgid': '6.8', 'osds': [7, 3, 6]}, {'pgid': '6.9', 'osds': [0, 7, 3]}, {'pgid': '6.f', 'osds': [3, 4, 6]}, {'pgid': '6.17', 'osds': [4, 5, 0]}, {'pgid': '6.1c', 'osds': [7, 5, 3]}, {'pgid': '7.2', 'osds': [3, 1, 7]}, {'pgid': '7.3', 'osds': [3, 4, 6]}, {'pgid': '7.5', 'osds': [7, 0, 1]}, {'pgid': '7.a', 'osds': [3, 5, 1]}, {'pgid': '7.e', 'osds': [5, 0, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.76:6800/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4087324200': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4141143304': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:6801/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/2779879991': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6801/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6800/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:0/2483102204': '2021-03-16T22:13:41.603652+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 258, 'length': 1}, {'begin': 268, 'length': 1}, {'begin': 291, 'length': 1}, {'begin': 303, 'length': 1}, {'begin': 305, 'length': 1}, {'begin': 318, 'length': 1}, {'begin': 321, 'length': 1}, {'begin': 347, 'length': 1}, {'begin': 351, 'length': 1}, {'begin': 353, 'length': 1}, {'begin': 358, 'length': 1}, {'begin': 360, 'length': 1}, {'begin': 365, 'length': 2}, {'begin': 368, 'length': 1}, {'begin': 371, 'length': 1}, {'begin': 373, 'length': 1}, {'begin': 375, 'length': 1}, {'begin': 382, 'length': 4}, {'begin': 388, 'length': 2}, {'begin': 391, 'length': 1}, {'begin': 394, 'length': 2}, {'begin': 399, 'length': 2}, {'begin': 403, 'length': 3}, {'begin': 409, 'length': 4}, {'begin': 415, 'length': 2}, {'begin': 418, 'length': 2}, {'begin': 422, 'length': 2}, {'begin': 425, 'length': 1}, {'begin': 428, 'length': 1}, {'begin': 432, 'length': 2}, {'begin': 435, 'length': 2}, {'begin': 438, 'length': 2}, {'begin': 443, 'length': 1}, {'begin': 445, 'length': 2}, {'begin': 448, 'length': 1}, {'begin': 452, 'length': 1}, {'begin': 456, 'length': 1}, {'begin': 460, 'length': 1}, {'begin': 465, 'length': 1}, {'begin': 468, 'length': 1}, {'begin': 471, 'length': 1}, {'begin': 473, 'length': 1}, {'begin': 477, 'length': 1}, {'begin': 479, 'length': 1}, {'begin': 481, 'length': 1}, {'begin': 499, 'length': 1}, {'begin': 508, 'length': 1}]}], 'new_removed_snaps': [], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-03-15T22:30:03.629 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:30:03.702 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:03.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:03.736 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:03.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:03 smithi125 conmon[56739]: cluster 2021-03-15T22:30:02.064717+0000 osd.4 (osd.4 2021-03-15T22:30:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:03 smithi125 conmon[56739]: ) 139 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:30:03.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:03 smithi125 conmon[56739]: cluster 2021-03-15T22:30:02.548064+0000 osd.3 (osd.3) 121 : cluster [DBG] 4.14 scrub ok 2021-03-15T22:30:03.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:03 smithi125 conmon[56739]: cluster 2021-03-15T22:30:03.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:03 smithi125 conmon[56739]: 2021-03-15T22:30:02.644641+0000 osd.4 (osd.4) 140 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:03.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:03 smithi125 conmon[56739]: cluster 2021-03-15T22:30:02. 2021-03-15T22:30:03.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:03 smithi125 conmon[56739]: 645138+0000 osd.4 (osd.4) 141 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:03.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:03 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:30:03.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:03 smithi125 conmon[56739]: 22:30:03.210571+0000 mon.a (mon.0) 1800 : cluster 2021-03-15T22:30:03.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:03 smithi125 conmon[56739]: [DBG] osdmap e654: 8 total, 7 up, 8 in 2021-03-15T22:30:03.838 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:03.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:03.864 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:03.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:03.966 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:03.989 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:03.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:03.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:04.079 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:30:04.094 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:04.097 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:30:04.099 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:04.115 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:30:04.119 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:04.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:04.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:04.145 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:30:04.163 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:30:04.182 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:30:04.202 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:30:04.203 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:30:04.203 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:30:04.204 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:30:04.204 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:30:04.204 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:30:04.204 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:30:04.204 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:30:04.205 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:30:04.222 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:04.223 INFO:tasks.rados.rados.0.smithi076.stdout:2381: done (1 left) 2021-03-15T22:30:04.223 INFO:tasks.rados.rados.0.smithi076.stdout:2382: delete oid 21 current snap is 271 2021-03-15T22:30:04.224 INFO:tasks.rados.rados.0.smithi076.stdout:2380: finishing copy_from to smithi07694878-33 2021-03-15T22:30:04.224 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 1737 (ObjNum 794 snap 261 seq_num 794) dirty exists 2021-03-15T22:30:04.224 INFO:tasks.rados.rados.0.smithi076.stdout:2380: finishing copy_from racing read to smithi07694878-33 2021-03-15T22:30:04.231 INFO:tasks.rados.rados.0.smithi076.stdout:2380: done (1 left) 2021-03-15T22:30:04.231 INFO:tasks.rados.rados.0.smithi076.stdout:2382: done (0 left) 2021-03-15T22:30:04.232 INFO:tasks.rados.rados.0.smithi076.stdout:2383: read oid 30 snap -1 2021-03-15T22:30:04.232 INFO:tasks.rados.rados.0.smithi076.stdout:2383: expect (ObjNum 774 snap 256 seq_num 774) 2021-03-15T22:30:04.233 INFO:tasks.rados.rados.0.smithi076.stdout:2384: delete oid 15 current snap is 271 2021-03-15T22:30:04.241 INFO:tasks.rados.rados.0.smithi076.stdout:2383: expect (ObjNum 774 snap 256 seq_num 774) 2021-03-15T22:30:04.249 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:04.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:04.251 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:04.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:04.376 INFO:tasks.rados.rados.0.smithi076.stdout:2383: done (1 left) 2021-03-15T22:30:04.377 INFO:tasks.rados.rados.0.smithi076.stdout:2384: done (0 left) 2021-03-15T22:30:04.377 INFO:tasks.rados.rados.0.smithi076.stdout:2385: read oid 42 snap -1 2021-03-15T22:30:04.377 INFO:tasks.rados.rados.0.smithi076.stdout:2385: expect (ObjNum 794 snap 261 seq_num 794) 2021-03-15T22:30:04.377 INFO:tasks.rados.rados.0.smithi076.stdout:2386: delete oid 31 current snap is 271 2021-03-15T22:30:04.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:04.379 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:04.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:04.380 INFO:tasks.rados.rados.0.smithi076.stdout:2386: done (1 left) 2021-03-15T22:30:04.380 INFO:tasks.rados.rados.0.smithi076.stdout:2387: read oid 13 snap -1 2021-03-15T22:30:04.381 INFO:tasks.rados.rados.0.smithi076.stdout:2387: expect (ObjNum 797 snap 262 seq_num 797) 2021-03-15T22:30:04.381 INFO:tasks.rados.rados.0.smithi076.stdout:2388: snap_create 2021-03-15T22:30:04.384 INFO:tasks.rados.rados.0.smithi076.stdout:2387: expect (ObjNum 797 snap 262 seq_num 797) 2021-03-15T22:30:04.481 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:04.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:04.504 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:04.504 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[56514]: cluster 2021-03-15T22:30:02.813741+0000 osd.5 (osd.5) 7 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:30:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[56514]: cluster 2021-03-15T22:30:03.137605+0000 mgr.y (mgr.14144) 829 : cluster [DBG] pgmap v1124: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 480 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 6.2 MiB/s wr, 16 op/s; 56/1650 objects degraded (3.394%); 119/1650 objects misplaced (7.212%) 2021-03-15T22:30:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[56514]: cluster 2021-03-15T22:30:03.187380+0000 osd.1 (osd.1) 129 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:30:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[56514]: cluster 2021-03-15T22:30:03.257469+0000 osd.7 (osd.7) 128 : cluster [DBG] 5.4 deep-scrub ok 2021-03-15T22:30:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[56514]: cluster 2021-03-15T22:30:03.560119+0000 osd.0 (osd.0) 100 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:30:04.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[56514]: cluster 2021-03-15T22:30:03.567525+0000 osd.3 2021-03-15T22:30:04.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[56514]: (osd.3) 122 : cluster [DBG] 3.b deep-scrub ok 2021-03-15T22:30:04.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[56514]: audit 2021-03-15T22:30:04.059827+0000 mon.a (mon.0) 1801 : audit [DBG] from='client.? 172.21.15.76:0/371732983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:04.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[56514]: cluster 2021-03-15T22:30:04.212645+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e655: 8 total, 7 up, 8 in 2021-03-15T22:30:04.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: cluster 2021-03-15T22:30:04.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: 2021-03-15T22:30 2021-03-15T22:30:04.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: :02.813741+0000 osd.5 (osd.5) 7 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:30:04.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: cluster 2021-03-15T22:30:04.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: 2021-03-15T22:30 2021-03-15T22:30:04.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: :03.137605+0000 mgr.y (mgr.14144) 829 2021-03-15T22:30:04.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: : cluster [DBG] pgmap v1124: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 480 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 6.2 MiB/s wr, 16 op/s; 56/1650 objects degraded (3.394%); 119/1650 objects misplaced (7.212%) 2021-03-15T22:30:04.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: cluster 2021-03-15T22:30:04.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: 2021-03-15T22:30: 2021-03-15T22:30:04.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: 03.187380+0000 osd.1 (osd.1) 129 : cluster 2021-03-15T22:30:04.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: [DBG] 4.4 scrub ok 2021-03-15T22:30:04.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:30:04.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: T22:30:03.257469+0000 osd.7 (osd. 2021-03-15T22:30:04.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: 7) 128 : cluster [DBG] 5.4 deep-scrub ok 2021-03-15T22:30:04.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: cluster 2021-03-15T22:30:04.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: 2021-03-15T22:30:03 2021-03-15T22:30:04.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: .560119+0000 osd.0 (osd.0) 100 : cluster 2021-03-15T22:30:04.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: [DBG] 3.1f scrub ok 2021-03-15T22:30:04.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:30:04.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: :30:03.567525+0000 osd.3 (osd. 2021-03-15T22:30:04.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: 3) 122 : cluster [DBG] 3.b deep-scrub ok 2021-03-15T22:30:04.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: audit 2021-03-15T22:30:04.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: 2021-03-15T22:30: 2021-03-15T22:30:04.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: 04.059827+0000 mon.a (mon.0) 1801 : 2021-03-15T22:30:04.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: audit [DBG] from='client.? 172.21.15.76:0/371732983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:04.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:30:04.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: 22:30:04.212645+0000 mon.a (mon. 2021-03-15T22:30:04.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:04 smithi076 conmon[62156]: 0) 1802 : cluster [DBG] osdmap e655: 8 total, 7 up, 8 in 2021-03-15T22:30:04.605 INFO:tasks.rados.rados.0.smithi076.stdout:2385: expect (ObjNum 794 snap 261 seq_num 794) 2021-03-15T22:30:04.606 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:04.629 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:04.629 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:04.730 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:04.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:04.757 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:04.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: cluster 2021-03-15T22:30:02.813741+0000 osd.5 (osd.5) 7 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:30:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: cluster 2021-03-15T22:30:03 2021-03-15T22:30:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: .137605+0000 mgr.y (mgr.14144) 829 : cluster [DBG] pgmap v1124: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 480 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 6.2 MiB/s wr, 16 op/s; 56/1650 objects degraded (3.394%); 119/1650 objects misplaced (7.212%) 2021-03-15T22:30:04.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: cluster 2021-03-15T22:30:03.187380+0000 osd.1 (osd.1) 129 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:30:04.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: cluster 2021-03-15T22:30:03.257469+0000 osd.7 ( 2021-03-15T22:30:04.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: osd.7) 128 : cluster [DBG] 5.4 deep-scrub ok 2021-03-15T22:30:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: cluster 2021-03-15T22:30:03. 2021-03-15T22:30:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: 560119+0000 osd.0 (osd.0) 100 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:30:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:30:04.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: 03-15T22:30:03.567525+0000 osd.3 (osd.3 2021-03-15T22:30:04.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: ) 122 : cluster [DBG] 3.b deep-scrub ok 2021-03-15T22:30:04.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: audit 2021-03-15T22:30: 2021-03-15T22:30:04.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: 04.059827+0000 mon.a (mon.0) 1801 : audit [DBG] from='client.? 172.21.15.76:0/371732983' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:04.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: 2021-03-15T22:30:04.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:30:04.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:04 smithi125 conmon[56739]: T22:30:04.212645+0000 mon.a (mon.0) 1802 : cluster [DBG] osdmap e655: 8 total, 7 up, 8 in 2021-03-15T22:30:04.859 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:04.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:04.882 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:04.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:04.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:05.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:05.008 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:05.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:05.092 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:30:05.093 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:30:05.093 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:30:05.093 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:30:05.093 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:30:05.110 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:05.133 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:05.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:05.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:05.161 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:30:05.162 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:30:05.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:05.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:05.259 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:05.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:05.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:05.381 INFO:tasks.rados.rados.0.smithi076.stdout:2385: done (2 left) 2021-03-15T22:30:05.381 INFO:tasks.rados.rados.0.smithi076.stdout:2387: done (1 left) 2021-03-15T22:30:05.382 INFO:tasks.rados.rados.0.smithi076.stdout:2388: done (0 left) 2021-03-15T22:30:05.382 INFO:tasks.rados.rados.0.smithi076.stdout:2389: write oid 33 current snap is 272 2021-03-15T22:30:05.382 INFO:tasks.rados.rados.0.smithi076.stdout:2389: seq_num 821 ranges {598547=629200,1876540=491946,3046277=681499} 2021-03-15T22:30:05.384 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:05.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:05.385 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:05.388 INFO:tasks.rados.rados.0.smithi076.stdout:2389: writing smithi07694878-33 from 598547 to 1227747 tid 1 2021-03-15T22:30:05.394 INFO:tasks.rados.rados.0.smithi076.stdout:2389: writing smithi07694878-33 from 1876540 to 2368486 tid 2 2021-03-15T22:30:05.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:05 smithi076 conmon[56514]: cluster 2021-03-15T22:30:04.058159+0000 osd.4 2021-03-15T22:30:05.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:05 smithi076 conmon[56514]: (osd.4) 142 : cluster [DBG] 4.e scrub ok 2021-03-15T22:30:05.399 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:05 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:30:05.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:05 smithi076 conmon[62156]: 30:04.058159+0000 osd.4 (osd.4) 142 : cluster [DBG] 4.e scrub ok 2021-03-15T22:30:05.402 INFO:tasks.rados.rados.0.smithi076.stdout:2389: writing smithi07694878-33 from 3046277 to 3727776 tid 3 2021-03-15T22:30:05.402 INFO:tasks.rados.rados.0.smithi076.stdout:2390: snap_create 2021-03-15T22:30:05.402 INFO:tasks.rados.rados.0.smithi076.stdout:2389: finishing write tid 1 to smithi07694878-33 2021-03-15T22:30:05.403 INFO:tasks.rados.rados.0.smithi076.stdout:2389: finishing write tid 2 to smithi07694878-33 2021-03-15T22:30:05.404 INFO:tasks.rados.rados.0.smithi076.stdout:2389: finishing write tid 3 to smithi07694878-33 2021-03-15T22:30:05.405 INFO:tasks.rados.rados.0.smithi076.stdout:2389: finishing write tid 4 to smithi07694878-33 2021-03-15T22:30:05.405 INFO:tasks.rados.rados.0.smithi076.stdout:2389: finishing write tid 5 to smithi07694878-33 2021-03-15T22:30:05.406 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 1741 (ObjNum 821 snap 272 seq_num 821) dirty exists 2021-03-15T22:30:05.406 INFO:tasks.rados.rados.0.smithi076.stdout:2389: left oid 33 (ObjNum 821 snap 272 seq_num 821) 2021-03-15T22:30:05.487 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:05.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:05.512 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:05.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:05.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:05.637 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:05.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:05.739 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:05.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:05.762 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:05.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:05.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:05 smithi125 conmon[56739]: cluster 2021-03-15T22:30:04.058159 2021-03-15T22:30:05.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:05 smithi125 conmon[56739]: +0000 osd.4 (osd.4) 142 : cluster [DBG] 4.e scrub ok 2021-03-15T22:30:05.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:05.886 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:05.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:05.887 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:05.924 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:30:05.989 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:06.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:30:06.011 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:06.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:06.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:06.113 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:06.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:06.138 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:06.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:06.241 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:06.263 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:06.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:06.264 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:06.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:06.387 INFO:tasks.rados.rados.0.smithi076.stdout:2389: done (1 left) 2021-03-15T22:30:06.388 INFO:tasks.rados.rados.0.smithi076.stdout:2390: done (0 left) 2021-03-15T22:30:06.388 INFO:tasks.rados.rados.0.smithi076.stdout:2391: snap_remove snap 239 2021-03-15T22:30:06.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:06.390 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:06.390 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:06.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:06.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:06.515 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:06.516 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:06.617 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:06.639 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:06.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:06.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:06.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:06 smithi076 conmon[56514]: cluster 2021-03-15T22:30:05.138139 2021-03-15T22:30:06.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:06 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 830 : cluster [DBG] pgmap v1127: 138 pgs: 1 active+clean+snaptrim, 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 92 active+clean; 480 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.4 MiB/s wr, 9 op/s; 56/1653 objects degraded (3.388%); 120/1653 objects misplaced (7.260%) 2021-03-15T22:30:06.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:06 smithi076 conmon[56514]: cluster 2021-03-15T22:30:05.206219+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:30:06.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:06 smithi076 conmon[56514]: cluster 2021-03-15T22:30:05.376043+0000 mon.a (mon.0) 1803 : cluster [DBG] osdmap e656: 8 total, 7 up, 8 in 2021-03-15T22:30:06.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:06 smithi076 conmon[56514]: cluster 2021-03-15T22:30:05.601290+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.b scrub ok 2021-03-15T22:30:06.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:06 smithi076 conmon[62156]: cluster 2021-03-15T22:30:05.138139+0000 mgr.y (mgr.14144) 830 : cluster [DBG] pgmap v1127: 138 pgs: 1 active+clean+snaptrim, 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 92 active+clean; 480 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.4 MiB/s wr, 9 op/s; 56/1653 objects degraded (3.388%); 120/1653 objects misplaced (7.260%) 2021-03-15T22:30:06.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:06 smithi076 conmon[62156]: cluster 2021-03-15T22:30:05.206219+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:30:06.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:06 smithi076 conmon[62156]: cluster 2021-03-15T22:30:05.376043+0000 mon.a (mon.0) 1803 : cluster [DBG] osdmap e656: 8 total, 7 up, 8 in 2021-03-15T22:30:06.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:06 smithi076 conmon[62156]: cluster 2021-03-15T22:30:05.601290+0000 osd.0 (osd.0) 101 : cluster [DBG] 4.b scrub ok 2021-03-15T22:30:06.692 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:06.743 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:06.766 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:06.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:06.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:06.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:06 smithi125 conmon[56739]: cluster 2021-03-15T22:30:05.138139+0000 mgr.y (mgr.14144 2021-03-15T22:30:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:06 smithi125 conmon[56739]: ) 830 : cluster [DBG] pgmap v1127: 138 pgs: 1 active+clean+snaptrim, 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 92 active+clean; 480 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.2 MiB/s rd, 2.4 MiB/s wr, 9 op/s; 56/1653 objects degraded (3.388%); 120/1653 objects misplaced (7.260%) 2021-03-15T22:30:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:06 smithi125 conmon[56739]: cluster 2021-03-15T22:30:05.206219+0000 osd.7 (osd.7) 129 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:30:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:06 smithi125 conmon[56739]: cluster 2021-03-15T22:30: 2021-03-15T22:30:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:06 smithi125 conmon[56739]: 05.376043+0000 mon.a (mon.0) 1803 2021-03-15T22:30:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:06 smithi125 conmon[56739]: : cluster [DBG] osdmap e656: 8 total, 7 up, 8 in 2021-03-15T22:30:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:06 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:30:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:06 smithi125 conmon[56739]: T22:30:05.601290+0000 osd.0 ( 2021-03-15T22:30:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:06 smithi125 conmon[56739]: osd.0) 101 : cluster [DBG] 4.b scrub ok 2021-03-15T22:30:06.817 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:06.870 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:06.892 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:06.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:06.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:06.994 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:07.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:07.017 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:07.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:07.119 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:07.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:07.145 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:07.145 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:07.164 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:30:07.246 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:07.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:07.269 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:07.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:07.371 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:07.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:07 smithi076 conmon[56514]: cluster 2021-03-15T22:30:06.384426+0000 mon.a 2021-03-15T22:30:07.387 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:07 smithi076 conmon[56514]: (mon.0) 1804 : cluster [DBG] osdmap e657: 8 total, 7 up, 8 in 2021-03-15T22:30:07.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:07 smithi076 conmon[62156]: cluster 2021-03-15T22:30:07.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:07 smithi076 conmon[62156]: 2021-03-15T22:30:06.384426+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e657: 8 total, 7 up, 8 in 2021-03-15T22:30:07.396 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:07.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:07.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:07.398 INFO:tasks.rados.rados.0.smithi076.stdout:2391: done (0 left) 2021-03-15T22:30:07.398 INFO:tasks.rados.rados.0.smithi076.stdout:2392: delete oid 22 current snap is 273 2021-03-15T22:30:07.406 INFO:tasks.rados.rados.0.smithi076.stdout:2392: done (0 left) 2021-03-15T22:30:07.406 INFO:tasks.rados.rados.0.smithi076.stdout:2393: read oid 18 snap 268 2021-03-15T22:30:07.407 INFO:tasks.rados.rados.0.smithi076.stdout:2393: expect (ObjNum 791 snap 261 seq_num 791) 2021-03-15T22:30:07.407 INFO:tasks.rados.rados.0.smithi076.stdout:2394: read oid 26 snap 250 2021-03-15T22:30:07.407 INFO:tasks.rados.rados.0.smithi076.stdout:2394: expect (ObjNum 755 snap 248 seq_num 755) 2021-03-15T22:30:07.407 INFO:tasks.rados.rados.0.smithi076.stdout:2395: read oid 31 snap -1 2021-03-15T22:30:07.407 INFO:tasks.rados.rados.0.smithi076.stdout:2395: expect deleted 2021-03-15T22:30:07.408 INFO:tasks.rados.rados.0.smithi076.stdout:2396: read oid 6 snap -1 2021-03-15T22:30:07.408 INFO:tasks.rados.rados.0.smithi076.stdout:2396: expect (ObjNum 807 snap 269 seq_num 807) 2021-03-15T22:30:07.408 INFO:tasks.rados.rados.0.smithi076.stdout:2397: copy_from oid 2 from oid 1 current snap is 273 2021-03-15T22:30:07.408 INFO:tasks.rados.rados.0.smithi076.stdout:2398: read oid 40 snap 241 2021-03-15T22:30:07.409 INFO:tasks.rados.rados.0.smithi076.stdout:2398: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:30:07.409 INFO:tasks.rados.rados.0.smithi076.stdout:2399: copy_from oid 8 from oid 4 current snap is 273 2021-03-15T22:30:07.409 INFO:tasks.rados.rados.0.smithi076.stdout:2400: read oid 3 snap -1 2021-03-15T22:30:07.409 INFO:tasks.rados.rados.0.smithi076.stdout:2400: expect (ObjNum 767 snap 255 seq_num 767) 2021-03-15T22:30:07.410 INFO:tasks.rados.rados.0.smithi076.stdout:2401: snap_create 2021-03-15T22:30:07.410 INFO:tasks.rados.rados.0.smithi076.stdout:2399: finishing copy_from to smithi07694878-8 2021-03-15T22:30:07.411 INFO:tasks.rados.rados.0.smithi076.stdout:2399: got expected ENOENT (src dne) 2021-03-15T22:30:07.411 INFO:tasks.rados.rados.0.smithi076.stdout:2399: finishing copy_from racing read to smithi07694878-8 2021-03-15T22:30:07.411 INFO:tasks.rados.rados.0.smithi076.stdout:2393: expect (ObjNum 791 snap 261 seq_num 791) 2021-03-15T22:30:07.498 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:07.520 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:07.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:07.521 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:07.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:07.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:07.645 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:07.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:07.687 INFO:tasks.rados.rados.0.smithi076.stdout:2394: expect (ObjNum 755 snap 248 seq_num 755) 2021-03-15T22:30:07.747 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:07.770 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:07.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:07.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:07.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:07 smithi125 conmon[56739]: cluster 2021-03-15T22:30:06.384426+0000 mon.a (mon.0) 1804 : cluster [DBG] osdmap e657: 8 total, 7 up, 8 in 2021-03-15T22:30:07.796 INFO:tasks.rados.rados.0.smithi076.stdout:2400: expect (ObjNum 767 snap 255 seq_num 767) 2021-03-15T22:30:07.872 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:07.896 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:07.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:07.897 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:07.977 INFO:tasks.rados.rados.0.smithi076.stdout:2398: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:30:07.999 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:08.022 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:08.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:08.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:08.125 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:08.134 INFO:tasks.rados.rados.0.smithi076.stdout:2396: expect (ObjNum 807 snap 269 seq_num 807) 2021-03-15T22:30:08.150 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:08.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:08.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:08.253 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:08.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:08.278 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:08.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:08.310 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:30:08.314 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:30:08.341 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:08.379 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:08.380 INFO:tasks.rados.rados.0.smithi076.stdout:2397: finishing copy_from to smithi07694878-2 2021-03-15T22:30:08.381 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1324 (ObjNum 803 snap 268 seq_num 803) dirty exists 2021-03-15T22:30:08.381 INFO:tasks.rados.rados.0.smithi076.stdout:2397: finishing copy_from racing read to smithi07694878-2 2021-03-15T22:30:08.396 INFO:tasks.rados.rados.0.smithi076.stdout:2393: done (8 left) 2021-03-15T22:30:08.396 INFO:tasks.rados.rados.0.smithi076.stdout:2394: done (7 left) 2021-03-15T22:30:08.396 INFO:tasks.rados.rados.0.smithi076.stdout:2395: done (6 left) 2021-03-15T22:30:08.397 INFO:tasks.rados.rados.0.smithi076.stdout:2396: done (5 left) 2021-03-15T22:30:08.397 INFO:tasks.rados.rados.0.smithi076.stdout:2397: done (4 left) 2021-03-15T22:30:08.397 INFO:tasks.rados.rados.0.smithi076.stdout:2398: done (3 left) 2021-03-15T22:30:08.397 INFO:tasks.rados.rados.0.smithi076.stdout:2399: done (2 left) 2021-03-15T22:30:08.397 INFO:tasks.rados.rados.0.smithi076.stdout:2400: done (1 left) 2021-03-15T22:30:08.398 INFO:tasks.rados.rados.0.smithi076.stdout:2401: done (0 left) 2021-03-15T22:30:08.398 INFO:tasks.rados.rados.0.smithi076.stdout:2402: read oid 28 snap 264 2021-03-15T22:30:08.398 INFO:tasks.rados.rados.0.smithi076.stdout:2402: expect (ObjNum 609 snap 189 seq_num 609) 2021-03-15T22:30:08.399 INFO:tasks.rados.rados.0.smithi076.stdout:2403: rollback oid 6 current snap is 274 2021-03-15T22:30:08.399 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 273 2021-03-15T22:30:08.399 INFO:tasks.rados.rados.0.smithi076.stdout:2404: write (excl) oid 29 current snap is 274 2021-03-15T22:30:08.399 INFO:tasks.rados.rados.0.smithi076.stdout:2404: seq_num 823 ranges {632161=496051,1593504=495615,2701123=420396,3611645=220516} 2021-03-15T22:30:08.403 INFO:tasks.rados.rados.0.smithi076.stdout:2404: writing smithi07694878-29 from 632161 to 1128212 tid 1 2021-03-15T22:30:08.411 INFO:tasks.rados.rados.0.smithi076.stdout:2404: writing smithi07694878-29 from 1593504 to 2089119 tid 2 2021-03-15T22:30:08.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:08.413 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:08.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:08.416 INFO:tasks.rados.rados.0.smithi076.stdout:2404: writing smithi07694878-29 from 2701123 to 3121519 tid 3 2021-03-15T22:30:08.420 INFO:tasks.rados.rados.0.smithi076.stdout:2404: writing smithi07694878-29 from 3611645 to 3832161 tid 4 2021-03-15T22:30:08.421 INFO:tasks.rados.rados.0.smithi076.stdout:2405: write oid 36 current snap is 274 2021-03-15T22:30:08.422 INFO:tasks.rados.rados.0.smithi076.stdout:2405: seq_num 824 ranges {648968=571300,1651986=639273,2728546=631668} 2021-03-15T22:30:08.427 INFO:tasks.rados.rados.0.smithi076.stdout:2405: writing smithi07694878-36 from 648968 to 1220268 tid 1 2021-03-15T22:30:08.432 INFO:tasks.rados.rados.0.smithi076.stdout:2405: writing smithi07694878-36 from 1651986 to 2291259 tid 2 2021-03-15T22:30:08.439 INFO:tasks.rados.rados.0.smithi076.stdout:2405: writing smithi07694878-36 from 2728546 to 3360214 tid 3 2021-03-15T22:30:08.439 INFO:tasks.rados.rados.0.smithi076.stdout:2406: delete oid 3 current snap is 274 2021-03-15T22:30:08.439 INFO:tasks.rados.rados.0.smithi076.stdout:2402: expect (ObjNum 609 snap 189 seq_num 609) 2021-03-15T22:30:08.515 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[56514]: cluster 2021-03-15T22:30:06.920602+0000 osd.5 (osd.5) 8 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:30:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[56514]: cluster 2021-03-15T22:30:07.063221+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.a scrub ok 2021-03-15T22:30:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[56514]: cluster 2021-03-15T22:30:07.138955+0000 mgr.y (mgr.14144) 831 : cluster [DBG] pgmap v1130: 138 pgs: 1 active+clean+snaptrim, 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 92 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 2.2 MiB/s wr, 11 op/s; 56/1653 objects degraded (3.388%); 120/1653 objects misplaced (7.260%) 2021-03-15T22:30:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[56514]: cluster 2021-03-15T22:30:07.252954+0000 osd.7 (osd.7) 130 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:30:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[56514]: cluster 2021-03-15T22:30:07.389912+0000 mon.a (mon.0) 1805 : cluster [DBG] osdmap e658: 8 total, 7 up, 8 in 2021-03-15T22:30:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[56514]: cluster 2021-03-15T22:30:07.486181+0000 osd.3 (osd.3) 123 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:30:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: cluster 2021-03-15T22:30:06.920602 2021-03-15T22:30:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 8 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:30:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: cluster 2021-03-15T22:30:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: 2021-03-15T22:30:07.063221+0000 osd.4 2021-03-15T22:30:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: (osd.4) 2021-03-15T22:30:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: 143 : cluster [DBG] 5.a scrub ok 2021-03-15T22:30:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: cluster 2021-03-15T22:30:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: 2021-03-15T22:30:07.138955 2021-03-15T22:30:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 831 : cluster 2021-03-15T22:30:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: [DBG] pgmap v1130: 138 pgs: 1 active+clean+snaptrim, 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 92 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 2.2 MiB/s wr, 11 op/s; 56/1653 objects degraded (3.388%); 120/1653 objects misplaced (7.260%) 2021-03-15T22:30:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: cluster 2021-03-15T22:30:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: 2021-03-15T22:30: 2021-03-15T22:30:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: 07.252954+0000 osd.7 (osd.7) 130 2021-03-15T22:30:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: : cluster [DBG] 5.9 scrub ok 2021-03-15T22:30:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: cluster 2021-03-15T22:30:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: 2021-03-15T22:30:07.389912 2021-03-15T22:30:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1805 : cluster 2021-03-15T22:30:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: [DBG] osdmap e658: 8 total, 7 up, 8 in 2021-03-15T22:30:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:30:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: -15T22:30:07.486181+0000 2021-03-15T22:30:08.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: osd.3 (osd.3) 123 : cluster [DBG] 2021-03-15T22:30:08.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:08 smithi076 conmon[62156]: 4.10 scrub ok 2021-03-15T22:30:08.540 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:08.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:08.542 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:08.546 INFO:tasks.rados.rados.0.smithi076.stdout:2404: finishing write tid 1 to smithi07694878-29 2021-03-15T22:30:08.547 INFO:tasks.rados.rados.0.smithi076.stdout:2403: finishing rollback tid 0 to smithi07694878-6 2021-03-15T22:30:08.547 INFO:tasks.rados.rados.0.smithi076.stdout:2403: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:30:08.547 INFO:tasks.rados.rados.0.smithi076.stdout:2403: finishing rollback tid 2 to smithi07694878-6 2021-03-15T22:30:08.547 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 1597 (ObjNum 807 snap 269 seq_num 807) dirty exists 2021-03-15T22:30:08.548 INFO:tasks.rados.rados.0.smithi076.stdout:2404: finishing write tid 2 to smithi07694878-29 2021-03-15T22:30:08.548 INFO:tasks.rados.rados.0.smithi076.stdout:2402: done (4 left) 2021-03-15T22:30:08.548 INFO:tasks.rados.rados.0.smithi076.stdout:2403: done (3 left) 2021-03-15T22:30:08.549 INFO:tasks.rados.rados.0.smithi076.stdout:2406: done (2 left) 2021-03-15T22:30:08.549 INFO:tasks.rados.rados.0.smithi076.stdout:2407: read oid 4 snap -1 2021-03-15T22:30:08.549 INFO:tasks.rados.rados.0.smithi076.stdout:2407: expect deleted 2021-03-15T22:30:08.550 INFO:tasks.rados.rados.0.smithi076.stdout:2408: snap_remove snap 259 2021-03-15T22:30:08.644 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:08.667 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:08.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:08.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:08.770 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: cluster 2021-03-15T22:30:06.920602+0000 osd.5 (osd.5) 8 : cluster [DBG] 2021-03-15T22:30:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: 5.15 scrub ok 2021-03-15T22:30:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: cluster 2021-03-15T22:30:07.063221+0000 osd.4 (osd.4) 143 : cluster [DBG] 5.a scrub ok 2021-03-15T22:30:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:30:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: 22:30:07.138955+0000 mgr.y ( 2021-03-15T22:30:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: mgr.14144) 831 : cluster [DBG] pgmap v1130: 138 pgs: 1 active+clean+snaptrim, 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 92 active+clean; 482 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 2.2 MiB/s wr, 11 op/s; 56/1653 objects degraded (3.388%); 120/1653 objects misplaced (7.260%) 2021-03-15T22:30:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: cluster 2021-03-15T22:30:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: 2021-03-15T22:30: 2021-03-15T22:30:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: 07.252954+0000 osd.7 (osd.7) 2021-03-15T22:30:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: 130 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:30:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:30:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: -03-15T22:30: 2021-03-15T22:30:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: 07.389912+0000 mon.a (mon.0) 1805 2021-03-15T22:30:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: : cluster [DBG] osdmap e658: 8 total, 7 up, 8 in 2021-03-15T22:30:08.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:30:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: 03-15T22:30:07.486181 2021-03-15T22:30:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: +0000 osd.3 (osd.3) 123 : cluster 2021-03-15T22:30:08.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:08 smithi125 conmon[56739]: [DBG] 4.10 scrub ok 2021-03-15T22:30:08.793 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:08.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:08.795 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:08.896 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:08.938 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:08.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:08.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:09.040 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:09.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:09.064 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:09.064 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:09.166 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:09.188 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:09.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:09.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:09.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:09.317 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:09.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:09.318 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:09.402 INFO:tasks.rados.rados.0.smithi076.stdout:2408: done (3 left) 2021-03-15T22:30:09.403 INFO:tasks.rados.rados.0.smithi076.stdout:2409: delete oid 25 current snap is 274 2021-03-15T22:30:09.403 INFO:tasks.rados.rados.0.smithi076.stdout:2404: finishing write tid 3 to smithi07694878-29 2021-03-15T22:30:09.403 INFO:tasks.rados.rados.0.smithi076.stdout:2404: finishing write tid 4 to smithi07694878-29 2021-03-15T22:30:09.404 INFO:tasks.rados.rados.0.smithi076.stdout:2404: finishing write tid 5 to smithi07694878-29 2021-03-15T22:30:09.404 INFO:tasks.rados.rados.0.smithi076.stdout:2404: finishing write tid 6 to smithi07694878-29 2021-03-15T22:30:09.404 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 1888 (ObjNum 823 snap 274 seq_num 823) dirty exists 2021-03-15T22:30:09.404 INFO:tasks.rados.rados.0.smithi076.stdout:2404: left oid 29 (ObjNum 823 snap 274 seq_num 823) 2021-03-15T22:30:09.404 INFO:tasks.rados.rados.0.smithi076.stdout:2405: finishing write tid 1 to smithi07694878-36 2021-03-15T22:30:09.405 INFO:tasks.rados.rados.0.smithi076.stdout:2405: finishing write tid 2 to smithi07694878-36 2021-03-15T22:30:09.405 INFO:tasks.rados.rados.0.smithi076.stdout:2405: finishing write tid 3 to smithi07694878-36 2021-03-15T22:30:09.405 INFO:tasks.rados.rados.0.smithi076.stdout:2405: finishing write tid 4 to smithi07694878-36 2021-03-15T22:30:09.405 INFO:tasks.rados.rados.0.smithi076.stdout:2405: finishing write tid 5 to smithi07694878-36 2021-03-15T22:30:09.406 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 1551 (ObjNum 824 snap 274 seq_num 824) dirty exists 2021-03-15T22:30:09.406 INFO:tasks.rados.rados.0.smithi076.stdout:2405: left oid 36 (ObjNum 824 snap 274 seq_num 824) 2021-03-15T22:30:09.410 INFO:tasks.rados.rados.0.smithi076.stdout:2404: done (3 left) 2021-03-15T22:30:09.410 INFO:tasks.rados.rados.0.smithi076.stdout:2405: done (2 left) 2021-03-15T22:30:09.411 INFO:tasks.rados.rados.0.smithi076.stdout:2407: done (1 left) 2021-03-15T22:30:09.411 INFO:tasks.rados.rados.0.smithi076.stdout:2409: done (0 left) 2021-03-15T22:30:09.411 INFO:tasks.rados.rados.0.smithi076.stdout:2410: snap_create 2021-03-15T22:30:09.420 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:09.443 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:09.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:09.445 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:09.481 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1131,"stamp":"2021-03-15T22:30:07.384548+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":505020754,"num_objects":551,"num_object_clones":185,"num_object_copies":1653,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":56,"num_objects_misplaced":120,"num_objects_unfound":0,"num_objects_dirty":551,"num_whiteouts":6,"num_read":6842,"num_read_kb":4756322,"num_write":4166,"num_write_kb":1269754,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2613,"num_bytes_recovered":922702376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":12791,"ondisk_log_size":12791,"up":369,"acting":400,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":455,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1519808,"kb_used_data":1246080,"kb_used_omap":153,"kb_used_meta":273510,"kb_avail":748441408,"statfs":{"total":767960285184,"available":766404001792,"internally_reserved":0,"allocated":1275985920,"data_stored":1736769158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":280074769},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1893522,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":28,"num_read_kb":27496,"num_write":20,"num_write_kb":8907,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.176514"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"653","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389378+0000","last_change":"2021-03-15T22:29:51.225570+0000","last_active":"2021-03-15T22:30:05.389378+0000","last_peered":"2021-03-15T22:30:05.389378+0000","last_clean":"2021-03-15T22:30:05.389378+0000","last_became_active":"2021-03-15T22:29:51.225252+0000","last_became_peered":"2021-03-15T22:29:51.225252+0000","last_unstale":"2021-03-15T22:30:05.389378+0000","last_undegraded":"2021-03-15T22:30:05.389378+0000","last_fullsized":"2021-03-15T22:30:05.389378+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:17.891857+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:17.891857+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"145","reported_epoch":"655","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:04.216645+0000","last_change":"2021-03-15T22:30:03.560232+0000","last_active":"2021-03-15T22:30:04.216645+0000","last_peered":"2021-03-15T22:30:04.216645+0000","last_clean":"2021-03-15T22:30:04.216645+0000","last_became_active":"2021-03-15T22:29:52.209287+0000","last_became_peered":"2021-03-15T22:29:52.209287+0000","last_unstale":"2021-03-15T22:30:04.216645+0000","last_undegraded":"2021-03-15T22:30:04.216645+0000","last_fullsized":"2021-03-15T22:30:04.216645+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:30:03.560125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.560125+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,6],"avail_no_missing":["0","5","6"],"object_location_counts":[{"shards":"0,5,6","objects":1}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"554'21","reported_seq":"673","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389320+0000","last_change":"2021-03-15T22:30:01.066537+0000","last_active":"2021-03-15T22:30:05.389320+0000","last_peered":"2021-03-15T22:30:05.389320+0000","last_clean":"2021-03-15T22:30:05.389320+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:30:05.389320+0000","last_undegraded":"2021-03-15T22:30:05.389320+0000","last_fullsized":"2021-03-15T22:30:05.389320+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:30:01.066454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:01.066454+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"77","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515006+0000","last_change":"2021-03-15T22:28:59.271736+0000","last_active":"2021-03-15T22:30:05.515006+0000","last_peered":"2021-03-15T22:30:05.515006+0000","last_clean":"2021-03-15T22:30:05.515006+0000","last_became_active":"2021-03-15T22:28:59.271412+0000","last_became_peered":"2021-03-15T22:28:59.271412+0000","last_unstale":"2021-03-15T22:30:05.515006+0000","last_undegraded":"2021-03-15T22:30:05.515006+0000","last_fullsized":"2021-03-15T22:30:05.515006+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:39.160459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:27:39.160459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"354","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.515871+0000","last_change":"2021-03-15T22:29:09.184895+0000","last_active":"2021-03-15T22:30:05.515871+0000","last_peered":"2021-03-15T22:30:05.515871+0000","last_clean":"2021-03-15T22:30:05.515871+0000","last_became_active":"2021-03-15T22:29:00.220188+0000","last_became_peered":"2021-03-15T22:29:00.220188+0000","last_unstale":"2021-03-15T22:30:05.515871+0000","last_undegraded":"2021-03-15T22:30:05.515871+0000","last_fullsized":"2021-03-15T22:30:05.515871+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:09.184802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:09.184802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6,1],"avail_no_missing":["3","1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"556'32","reported_seq":"349","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.515301+0000","last_change":"2021-03-15T22:29:32.512589+0000","last_active":"2021-03-15T22:30:05.515301+0000","last_peered":"2021-03-15T22:30:05.515301+0000","last_clean":"2021-03-15T22:30:05.515301+0000","last_became_active":"2021-03-15T22:29:27.240491+0000","last_became_peered":"2021-03-15T22:29:27.240491+0000","last_unstale":"2021-03-15T22:30:05.515301+0000","last_undegraded":"2021-03-15T22:30:05.515301+0000","last_fullsized":"2021-03-15T22:30:05.515301+0000","mapping_epoch":617,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'32","last_scrub_stamp":"2021-03-15T22:29:32.512485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:32.512485+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":2670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,5],"avail_no_missing":["3","0","5"],"object_location_counts":[{"shards":"0,3,5","objects":12}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"118","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.214879+0000","last_change":"2021-03-15T22:29:18.753212+0000","last_active":"2021-03-15T22:30:04.214879+0000","last_peered":"2021-03-15T22:30:04.214879+0000","last_clean":"2021-03-15T22:30:04.214879+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:30:04.214879+0000","last_undegraded":"2021-03-15T22:30:04.214879+0000","last_fullsized":"2021-03-15T22:30:04.214879+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:18.753152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:29:18.753152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"23","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.216773+0000","last_change":"2021-03-15T22:29:51.227090+0000","last_active":"2021-03-15T22:30:03.216773+0000","last_peered":"2021-03-15T22:30:03.216773+0000","last_clean":"2021-03-15T22:30:03.216773+0000","last_became_active":"2021-03-15T22:29:51.226784+0000","last_became_peered":"2021-03-15T22:29:51.226784+0000","last_unstale":"2021-03-15T22:30:03.216773+0000","last_undegraded":"2021-03-15T22:30:03.216773+0000","last_fullsized":"2021-03-15T22:30:03.216773+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:12.829451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:12.829451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"796","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389527+0000","last_change":"2021-03-15T22:29:00.214607+0000","last_active":"2021-03-15T22:30:05.389527+0000","last_peered":"2021-03-15T22:30:05.389527+0000","last_clean":"2021-03-15T22:30:05.389527+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:30:05.389527+0000","last_undegraded":"2021-03-15T22:30:05.389527+0000","last_fullsized":"2021-03-15T22:30:05.389527+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:28:47.956917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:28:47.956917+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"268","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.216419+0000","last_change":"2021-03-15T22:29:52.224144+0000","last_active":"2021-03-15T22:30:03.216419+0000","last_peered":"2021-03-15T22:30:03.216419+0000","last_clean":"2021-03-15T22:30:03.216419+0000","last_became_active":"2021-03-15T22:29:52.223704+0000","last_became_peered":"2021-03-15T22:29:52.223704+0000","last_unstale":"2021-03-15T22:30:03.216419+0000","last_undegraded":"2021-03-15T22:30:03.216419+0000","last_fullsized":"2021-03-15T22:30:03.216419+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:37.172397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:29:37.172397+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"28","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.216674+0000","last_change":"2021-03-15T22:30:02.813811+0000","last_active":"2021-03-15T22:30:03.216674+0000","last_peered":"2021-03-15T22:30:03.216674+0000","last_clean":"2021-03-15T22:30:03.216674+0000","last_became_active":"2021-03-15T22:29:51.234868+0000","last_became_peered":"2021-03-15T22:29:51.234868+0000","last_unstale":"2021-03-15T22:30:03.216674+0000","last_undegraded":"2021-03-15T22:30:03.216674+0000","last_fullsized":"2021-03-15T22:30:03.216674+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:02.813745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.813745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"739","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389809+0000","last_change":"2021-03-15T22:29:56.950563+0000","last_active":"2021-03-15T22:30:05.389809+0000","last_peered":"2021-03-15T22:30:05.389809+0000","last_clean":"2021-03-15T22:30:05.389809+0000","last_became_active":"2021-03-15T22:29:11.197059+0000","last_became_peered":"2021-03-15T22:29:11.197059+0000","last_unstale":"2021-03-15T22:30:05.389809+0000","last_undegraded":"2021-03-15T22:30:05.389809+0000","last_fullsized":"2021-03-15T22:30:05.389809+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:29:56.950494+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.950494+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"651","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389890+0000","last_change":"2021-03-15T22:29:51.225493+0000","last_active":"2021-03-15T22:30:05.389890+0000","last_peered":"2021-03-15T22:30:05.389890+0000","last_clean":"2021-03-15T22:30:05.389890+0000","last_became_active":"2021-03-15T22:29:51.225147+0000","last_became_peered":"2021-03-15T22:29:51.225147+0000","last_unstale":"2021-03-15T22:30:05.389890+0000","last_undegraded":"2021-03-15T22:30:05.389890+0000","last_fullsized":"2021-03-15T22:30:05.389890+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:41.941510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.941510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"117","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.215104+0000","last_change":"2021-03-15T22:28:12.702035+0000","last_active":"2021-03-15T22:30:04.215104+0000","last_peered":"2021-03-15T22:30:04.215104+0000","last_clean":"2021-03-15T22:30:04.215104+0000","last_became_active":"2021-03-15T22:28:12.519619+0000","last_became_peered":"2021-03-15T22:28:12.519619+0000","last_unstale":"2021-03-15T22:30:04.215104+0000","last_undegraded":"2021-03-15T22:30:04.215104+0000","last_fullsized":"2021-03-15T22:30:04.215104+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:27:08.823978+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:27:08.823978+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"272","reported_epoch":"652","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:01.212027+0000","last_change":"2021-03-15T22:29:57.169405+0000","last_active":"2021-03-15T22:30:01.212027+0000","last_peered":"2021-03-15T22:30:01.212027+0000","last_clean":"2021-03-15T22:30:01.212027+0000","last_became_active":"2021-03-15T22:29:00.215216+0000","last_became_peered":"2021-03-15T22:29:00.215216+0000","last_unstale":"2021-03-15T22:30:01.212027+0000","last_undegraded":"2021-03-15T22:30:01.212027+0000","last_fullsized":"2021-03-15T22:30:01.212027+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:29:57.169300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:57.169300+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,0],"avail_no_missing":["1","0","3"],"object_location_counts":[{"shards":"0,1,3","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"468","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515924+0000","last_change":"2021-03-15T22:29:51.225542+0000","last_active":"2021-03-15T22:30:05.515924+0000","last_peered":"2021-03-15T22:30:05.515924+0000","last_clean":"2021-03-15T22:30:05.515924+0000","last_became_active":"2021-03-15T22:29:51.225288+0000","last_became_peered":"2021-03-15T22:29:51.225288+0000","last_unstale":"2021-03-15T22:30:05.515924+0000","last_undegraded":"2021-03-15T22:30:05.515924+0000","last_fullsized":"2021-03-15T22:30:05.515924+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:38.513993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:45.403769+0000","last_clean_scrub_stamp":"2021-03-15T22:29:38.513993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"632","reported_epoch":"656","state":"active+undersized","last_fresh":"2021-03-15T22:30:05.389770+0000","last_change":"2021-03-15T22:26:46.144450+0000","last_active":"2021-03-15T22:30:05.389770+0000","last_peered":"2021-03-15T22:30:05.389770+0000","last_clean":"2021-03-15T22:26:44.341675+0000","last_became_active":"2021-03-15T22:26:46.144450+0000","last_became_peered":"2021-03-15T22:26:46.144450+0000","last_unstale":"2021-03-15T22:30:05.389770+0000","last_undegraded":"2021-03-15T22:30:05.389770+0000","last_fullsized":"2021-03-15T22:26:46.140114+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:11.028422+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:11.028422+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"102","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515343+0000","last_change":"2021-03-15T22:29:54.564185+0000","last_active":"2021-03-15T22:30:05.515343+0000","last_peered":"2021-03-15T22:30:05.515343+0000","last_clean":"2021-03-15T22:30:05.515343+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:30:05.515343+0000","last_undegraded":"2021-03-15T22:30:05.515343+0000","last_fullsized":"2021-03-15T22:30:05.515343+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:29:54.564115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:54.564115+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"653","reported_epoch":"654","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:03.257066+0000","last_change":"2021-03-15T22:30:01.208944+0000","last_active":"2021-03-15T22:30:03.257066+0000","last_peered":"2021-03-15T22:30:03.257066+0000","last_clean":"2021-03-15T22:30:03.257066+0000","last_became_active":"2021-03-15T22:29:52.224351+0000","last_became_peered":"2021-03-15T22:29:52.224351+0000","last_unstale":"2021-03-15T22:30:03.257066+0000","last_undegraded":"2021-03-15T22:30:03.257066+0000","last_fullsized":"2021-03-15T22:30:03.257066+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:01.208847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:01.208847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,3],"avail_no_missing":["7","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"671","reported_epoch":"652","state":"active+clean","last_fresh":"2021-03-15T22:30:01.211345+0000","last_change":"2021-03-15T22:29:18.149267+0000","last_active":"2021-03-15T22:30:01.211345+0000","last_peered":"2021-03-15T22:30:01.211345+0000","last_clean":"2021-03-15T22:30:01.211345+0000","last_became_active":"2021-03-15T22:28:59.264559+0000","last_became_peered":"2021-03-15T22:28:59.264559+0000","last_unstale":"2021-03-15T22:30:01.211345+0000","last_undegraded":"2021-03-15T22:30:01.211345+0000","last_fullsized":"2021-03-15T22:30:01.211345+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:18.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:18.149204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"218","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.215482+0000","last_change":"2021-03-15T22:28:59.305604+0000","last_active":"2021-03-15T22:30:04.215482+0000","last_peered":"2021-03-15T22:30:04.215482+0000","last_clean":"2021-03-15T22:30:04.215482+0000","last_became_active":"2021-03-15T22:28:59.277385+0000","last_became_peered":"2021-03-15T22:28:59.277385+0000","last_unstale":"2021-03-15T22:30:04.215482+0000","last_undegraded":"2021-03-15T22:30:04.215482+0000","last_fullsized":"2021-03-15T22:30:04.215482+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"87'27","last_scrub_stamp":"2021-03-15T22:27:14.352631+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:27:14.352631+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"649","reported_epoch":"652","state":"active+clean","last_fresh":"2021-03-15T22:30:01.212453+0000","last_change":"2021-03-15T22:29:52.230266+0000","last_active":"2021-03-15T22:30:01.212453+0000","last_peered":"2021-03-15T22:30:01.212453+0000","last_clean":"2021-03-15T22:30:01.212453+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:30:01.212453+0000","last_undegraded":"2021-03-15T22:30:01.212453+0000","last_fullsized":"2021-03-15T22:30:01.212453+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:34.234643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:34.234643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"256","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515022+0000","last_change":"2021-03-15T22:29:00.220417+0000","last_active":"2021-03-15T22:30:05.515022+0000","last_peered":"2021-03-15T22:30:05.515022+0000","last_clean":"2021-03-15T22:30:05.515022+0000","last_became_active":"2021-03-15T22:29:00.220073+0000","last_became_peered":"2021-03-15T22:29:00.220073+0000","last_unstale":"2021-03-15T22:30:05.515022+0000","last_undegraded":"2021-03-15T22:30:05.515022+0000","last_fullsized":"2021-03-15T22:30:05.515022+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:34.099140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:28:34.099140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"741","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.213298+0000","last_change":"2021-03-15T22:29:52.273951+0000","last_active":"2021-03-15T22:30:04.213298+0000","last_peered":"2021-03-15T22:30:04.213298+0000","last_clean":"2021-03-15T22:30:04.213298+0000","last_became_active":"2021-03-15T22:29:51.232710+0000","last_became_peered":"2021-03-15T22:29:51.232710+0000","last_unstale":"2021-03-15T22:30:04.213298+0000","last_undegraded":"2021-03-15T22:30:04.213298+0000","last_fullsized":"2021-03-15T22:30:04.213298+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:29:14.124824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:14.124824+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"618","reported_epoch":"654","state":"active+undersized","last_fresh":"2021-03-15T22:30:03.256509+0000","last_change":"2021-03-15T22:26:46.146944+0000","last_active":"2021-03-15T22:30:03.256509+0000","last_peered":"2021-03-15T22:30:03.256509+0000","last_clean":"2021-03-15T22:26:44.403038+0000","last_became_active":"2021-03-15T22:26:46.146944+0000","last_became_peered":"2021-03-15T22:26:46.146944+0000","last_unstale":"2021-03-15T22:30:03.256509+0000","last_undegraded":"2021-03-15T22:30:03.256509+0000","last_fullsized":"2021-03-15T22:26:46.139288+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:09.177521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.177521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"636","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.212823+0000","last_change":"2021-03-15T22:29:35.315031+0000","last_active":"2021-03-15T22:30:04.212823+0000","last_peered":"2021-03-15T22:30:04.212823+0000","last_clean":"2021-03-15T22:30:04.212823+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:30:04.212823+0000","last_undegraded":"2021-03-15T22:30:04.212823+0000","last_fullsized":"2021-03-15T22:30:04.212823+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:35.314964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:35.314964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"157","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.216586+0000","last_change":"2021-03-15T22:29:51.226956+0000","last_active":"2021-03-15T22:30:04.216586+0000","last_peered":"2021-03-15T22:30:04.216586+0000","last_clean":"2021-03-15T22:30:04.216586+0000","last_became_active":"2021-03-15T22:29:51.226677+0000","last_became_peered":"2021-03-15T22:29:51.226677+0000","last_unstale":"2021-03-15T22:30:04.216586+0000","last_undegraded":"2021-03-15T22:30:04.216586+0000","last_fullsized":"2021-03-15T22:30:04.216586+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:14.776700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:29:14.776700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"99","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.514886+0000","last_change":"2021-03-15T22:28:59.309904+0000","last_active":"2021-03-15T22:30:05.514886+0000","last_peered":"2021-03-15T22:30:05.514886+0000","last_clean":"2021-03-15T22:30:05.514886+0000","last_became_active":"2021-03-15T22:28:59.269481+0000","last_became_peered":"2021-03-15T22:28:59.269481+0000","last_unstale":"2021-03-15T22:30:05.514886+0000","last_undegraded":"2021-03-15T22:30:05.514886+0000","last_fullsized":"2021-03-15T22:30:05.514886+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'12","last_scrub_stamp":"2021-03-15T22:27:34.103340+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:27:34.103340+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":6260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"256","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515303+0000","last_change":"2021-03-15T22:29:00.216337+0000","last_active":"2021-03-15T22:30:05.515303+0000","last_peered":"2021-03-15T22:30:05.515303+0000","last_clean":"2021-03-15T22:30:05.515303+0000","last_became_active":"2021-03-15T22:29:00.215311+0000","last_became_peered":"2021-03-15T22:29:00.215311+0000","last_unstale":"2021-03-15T22:30:05.515303+0000","last_undegraded":"2021-03-15T22:30:05.515303+0000","last_fullsized":"2021-03-15T22:30:05.515303+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:36.102000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:40.670674+0000","last_clean_scrub_stamp":"2021-03-15T22:28:36.102000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"614","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.256747+0000","last_change":"2021-03-15T22:28:09.088750+0000","last_active":"2021-03-15T22:30:03.256747+0000","last_peered":"2021-03-15T22:30:03.256747+0000","last_clean":"2021-03-15T22:30:03.256747+0000","last_became_active":"2021-03-15T22:23:52.262741+0000","last_became_peered":"2021-03-15T22:23:52.262741+0000","last_unstale":"2021-03-15T22:30:03.256747+0000","last_undegraded":"2021-03-15T22:30:03.256747+0000","last_fullsized":"2021-03-15T22:30:03.256747+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:09.088684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:28:09.088684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"23","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.216841+0000","last_change":"2021-03-15T22:29:51.220967+0000","last_active":"2021-03-15T22:30:03.216841+0000","last_peered":"2021-03-15T22:30:03.216841+0000","last_clean":"2021-03-15T22:30:03.216841+0000","last_became_active":"2021-03-15T22:29:51.220261+0000","last_became_peered":"2021-03-15T22:29:51.220261+0000","last_unstale":"2021-03-15T22:30:03.216841+0000","last_undegraded":"2021-03-15T22:30:03.216841+0000","last_fullsized":"2021-03-15T22:30:03.216841+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:18.056308+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:18.056308+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"438","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515590+0000","last_change":"2021-03-15T22:29:00.220920+0000","last_active":"2021-03-15T22:30:05.515590+0000","last_peered":"2021-03-15T22:30:05.515590+0000","last_clean":"2021-03-15T22:30:05.515590+0000","last_became_active":"2021-03-15T22:29:00.220100+0000","last_became_peered":"2021-03-15T22:29:00.220100+0000","last_unstale":"2021-03-15T22:30:05.515590+0000","last_undegraded":"2021-03-15T22:30:05.515590+0000","last_fullsized":"2021-03-15T22:30:05.515590+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:27:37.107464+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:27:37.107464+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"137","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.214642+0000","last_change":"2021-03-15T22:29:51.212768+0000","last_active":"2021-03-15T22:30:04.214642+0000","last_peered":"2021-03-15T22:30:04.214642+0000","last_clean":"2021-03-15T22:30:04.214642+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:30:04.214642+0000","last_undegraded":"2021-03-15T22:30:04.214642+0000","last_fullsized":"2021-03-15T22:30:04.214642+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:19.721291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:29:19.721291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"446","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515576+0000","last_change":"2021-03-15T22:29:56.525891+0000","last_active":"2021-03-15T22:30:05.515576+0000","last_peered":"2021-03-15T22:30:05.515576+0000","last_clean":"2021-03-15T22:30:05.515576+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:30:05.515576+0000","last_undegraded":"2021-03-15T22:30:05.515576+0000","last_fullsized":"2021-03-15T22:30:05.515576+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:56.525822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.525822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"654","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.257394+0000","last_change":"2021-03-15T22:29:09.154666+0000","last_active":"2021-03-15T22:30:03.257394+0000","last_peered":"2021-03-15T22:30:03.257394+0000","last_clean":"2021-03-15T22:30:03.257394+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:30:03.257394+0000","last_undegraded":"2021-03-15T22:30:03.257394+0000","last_fullsized":"2021-03-15T22:30:03.257394+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:09.154606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:29:09.154606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"323","reported_epoch":"652","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:30:01.211859+0000","last_change":"2021-03-15T22:26:46.149349+0000","last_active":"2021-03-15T22:30:01.211859+0000","last_peered":"2021-03-15T22:30:01.211859+0000","last_clean":"2021-03-15T22:26:44.404259+0000","last_became_active":"2021-03-15T22:26:46.149349+0000","last_became_peered":"2021-03-15T22:26:46.149349+0000","last_unstale":"2021-03-15T22:30:01.211859+0000","last_undegraded":"2021-03-15T22:26:46.140554+0000","last_fullsized":"2021-03-15T22:26:46.140451+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":381,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'15","last_scrub_stamp":"2021-03-15T22:26:42.456927+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:42.456927+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[{"shards":"1,6","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"278","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515194+0000","last_change":"2021-03-15T22:29:51.219273+0000","last_active":"2021-03-15T22:30:05.515194+0000","last_peered":"2021-03-15T22:30:05.515194+0000","last_clean":"2021-03-15T22:30:05.515194+0000","last_became_active":"2021-03-15T22:29:51.218982+0000","last_became_peered":"2021-03-15T22:29:51.218982+0000","last_unstale":"2021-03-15T22:30:05.515194+0000","last_undegraded":"2021-03-15T22:30:05.515194+0000","last_fullsized":"2021-03-15T22:30:05.515194+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:35.496457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:29:35.496457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"688","reported_epoch":"652","state":"active+clean","last_fresh":"2021-03-15T22:30:01.211760+0000","last_change":"2021-03-15T22:29:51.222565+0000","last_active":"2021-03-15T22:30:01.211760+0000","last_peered":"2021-03-15T22:30:01.211760+0000","last_clean":"2021-03-15T22:30:01.211760+0000","last_became_active":"2021-03-15T22:29:51.222254+0000","last_became_peered":"2021-03-15T22:29:51.222254+0000","last_unstale":"2021-03-15T22:30:01.211760+0000","last_undegraded":"2021-03-15T22:30:01.211760+0000","last_fullsized":"2021-03-15T22:30:01.211760+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:13.284771+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:25:11.794432+0000","last_clean_scrub_stamp":"2021-03-15T22:29:13.284771+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"171","reported_epoch":"654","state":"active+undersized","last_fresh":"2021-03-15T22:30:03.257011+0000","last_change":"2021-03-15T22:26:46.156620+0000","last_active":"2021-03-15T22:30:03.257011+0000","last_peered":"2021-03-15T22:30:03.257011+0000","last_clean":"2021-03-15T22:24:01.075356+0000","last_became_active":"2021-03-15T22:26:46.156620+0000","last_became_peered":"2021-03-15T22:26:46.156620+0000","last_unstale":"2021-03-15T22:30:03.257011+0000","last_undegraded":"2021-03-15T22:30:03.257011+0000","last_fullsized":"2021-03-15T22:26:46.141354+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":328,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:26:40.425545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:26:40.425545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"675","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389665+0000","last_change":"2021-03-15T22:30:02.064784+0000","last_active":"2021-03-15T22:30:05.389665+0000","last_peered":"2021-03-15T22:30:05.389665+0000","last_clean":"2021-03-15T22:30:05.389665+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:30:05.389665+0000","last_undegraded":"2021-03-15T22:30:05.389665+0000","last_fullsized":"2021-03-15T22:30:05.389665+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:30:02.064721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.064721+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"131","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.215791+0000","last_change":"2021-03-15T22:28:59.277756+0000","last_active":"2021-03-15T22:30:04.215791+0000","last_peered":"2021-03-15T22:30:04.215791+0000","last_clean":"2021-03-15T22:30:04.215791+0000","last_became_active":"2021-03-15T22:28:59.276812+0000","last_became_peered":"2021-03-15T22:28:59.276812+0000","last_unstale":"2021-03-15T22:30:04.215791+0000","last_undegraded":"2021-03-15T22:30:04.215791+0000","last_fullsized":"2021-03-15T22:30:04.215791+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:36.335532+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:27:36.335532+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"642'8","reported_seq":"30","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.216799+0000","last_change":"2021-03-15T22:29:51.581615+0000","last_active":"2021-03-15T22:30:03.216799+0000","last_peered":"2021-03-15T22:30:03.216799+0000","last_clean":"2021-03-15T22:30:03.216799+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:30:03.216799+0000","last_undegraded":"2021-03-15T22:30:03.216799+0000","last_fullsized":"2021-03-15T22:30:03.216799+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'6","last_scrub_stamp":"2021-03-15T22:28:33.647672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:28:33.647672+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"619","reported_epoch":"654","state":"active+undersized","last_fresh":"2021-03-15T22:30:03.256803+0000","last_change":"2021-03-15T22:26:46.155875+0000","last_active":"2021-03-15T22:30:03.256803+0000","last_peered":"2021-03-15T22:30:03.256803+0000","last_clean":"2021-03-15T22:26:44.403391+0000","last_became_active":"2021-03-15T22:26:46.155875+0000","last_became_peered":"2021-03-15T22:26:46.155875+0000","last_unstale":"2021-03-15T22:30:03.256803+0000","last_undegraded":"2021-03-15T22:30:03.256803+0000","last_fullsized":"2021-03-15T22:26:46.139933+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:01.148105+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:25:01.148105+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"556'24","reported_seq":"445","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515623+0000","last_change":"2021-03-15T22:30:02.548134+0000","last_active":"2021-03-15T22:30:05.515623+0000","last_peered":"2021-03-15T22:30:05.515623+0000","last_clean":"2021-03-15T22:30:05.515623+0000","last_became_active":"2021-03-15T22:29:00.226652+0000","last_became_peered":"2021-03-15T22:29:00.226652+0000","last_unstale":"2021-03-15T22:30:05.515623+0000","last_undegraded":"2021-03-15T22:30:05.515623+0000","last_fullsized":"2021-03-15T22:30:05.515623+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'24","last_scrub_stamp":"2021-03-15T22:30:02.548069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.548069+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"656","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.389973+0000","last_change":"2021-03-15T22:29:52.212163+0000","last_active":"2021-03-15T22:30:05.389973+0000","last_peered":"2021-03-15T22:30:05.389973+0000","last_clean":"2021-03-15T22:30:05.389973+0000","last_became_active":"2021-03-15T22:29:52.211703+0000","last_became_peered":"2021-03-15T22:29:52.211703+0000","last_unstale":"2021-03-15T22:30:05.389973+0000","last_undegraded":"2021-03-15T22:30:05.389973+0000","last_fullsized":"2021-03-15T22:30:05.389973+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:34.939142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:34.939142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,0],"avail_no_missing":["4","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"592'12","reported_seq":"297","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.515531+0000","last_change":"2021-03-15T22:29:00.227812+0000","last_active":"2021-03-15T22:30:05.515531+0000","last_peered":"2021-03-15T22:30:05.515531+0000","last_clean":"2021-03-15T22:30:05.515531+0000","last_became_active":"2021-03-15T22:29:00.227146+0000","last_became_peered":"2021-03-15T22:29:00.227146+0000","last_unstale":"2021-03-15T22:30:05.515531+0000","last_undegraded":"2021-03-15T22:30:05.515531+0000","last_fullsized":"2021-03-15T22:30:05.515531+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'10","last_scrub_stamp":"2021-03-15T22:28:47.140118+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:28:47.140118+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7,1],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":1}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"136","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.214510+0000","last_change":"2021-03-15T22:29:17.782276+0000","last_active":"2021-03-15T22:30:04.214510+0000","last_peered":"2021-03-15T22:30:04.214510+0000","last_clean":"2021-03-15T22:30:04.214510+0000","last_became_active":"2021-03-15T22:28:59.265822+0000","last_became_peered":"2021-03-15T22:28:59.265822+0000","last_unstale":"2021-03-15T22:30:04.214510+0000","last_undegraded":"2021-03-15T22:30:04.214510+0000","last_fullsized":"2021-03-15T22:30:04.214510+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:17.782209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:29:17.782209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"41","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.216665+0000","last_change":"2021-03-15T22:29:52.870643+0000","last_active":"2021-03-15T22:30:03.216665+0000","last_peered":"2021-03-15T22:30:03.216665+0000","last_clean":"2021-03-15T22:30:03.216665+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:30:03.216665+0000","last_undegraded":"2021-03-15T22:30:03.216665+0000","last_fullsized":"2021-03-15T22:30:03.216665+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:29:19.040000+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:29:19.040000+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"173","reported_epoch":"656","state":"active+undersized","last_fresh":"2021-03-15T22:30:05.390182+0000","last_change":"2021-03-15T22:26:46.147347+0000","last_active":"2021-03-15T22:30:05.390182+0000","last_peered":"2021-03-15T22:30:05.390182+0000","last_clean":"2021-03-15T22:17:58.999017+0000","last_became_active":"2021-03-15T22:26:46.147347+0000","last_became_peered":"2021-03-15T22:26:46.147347+0000","last_unstale":"2021-03-15T22:30:05.390182+0000","last_undegraded":"2021-03-15T22:30:05.390182+0000","last_fullsized":"2021-03-15T22:26:46.138696+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:00.614288+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:25:00.614288+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"620","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.256449+0000","last_change":"2021-03-15T22:27:33.386790+0000","last_active":"2021-03-15T22:30:03.256449+0000","last_peered":"2021-03-15T22:30:03.256449+0000","last_clean":"2021-03-15T22:30:03.256449+0000","last_became_active":"2021-03-15T22:23:52.263426+0000","last_became_peered":"2021-03-15T22:23:52.263426+0000","last_unstale":"2021-03-15T22:30:03.256449+0000","last_undegraded":"2021-03-15T22:30:03.256449+0000","last_fullsized":"2021-03-15T22:30:03.256449+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:33.386700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:33.386700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"176","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.215264+0000","last_change":"2021-03-15T22:29:00.214080+0000","last_active":"2021-03-15T22:30:04.215264+0000","last_peered":"2021-03-15T22:30:04.215264+0000","last_clean":"2021-03-15T22:30:04.215264+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:30:04.215264+0000","last_undegraded":"2021-03-15T22:30:04.215264+0000","last_fullsized":"2021-03-15T22:30:04.215264+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:28:44.765846+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"617","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.257353+0000","last_change":"2021-03-15T22:27:41.254554+0000","last_active":"2021-03-15T22:30:03.257353+0000","last_peered":"2021-03-15T22:30:03.257353+0000","last_clean":"2021-03-15T22:30:03.257353+0000","last_became_active":"2021-03-15T22:23:52.266133+0000","last_became_peered":"2021-03-15T22:23:52.266133+0000","last_unstale":"2021-03-15T22:30:03.257353+0000","last_undegraded":"2021-03-15T22:30:03.257353+0000","last_fullsized":"2021-03-15T22:30:03.257353+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:41.254469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:27:41.254469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"707","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.212580+0000","last_change":"2021-03-15T22:29:51.232146+0000","last_active":"2021-03-15T22:30:04.212580+0000","last_peered":"2021-03-15T22:30:04.212580+0000","last_clean":"2021-03-15T22:30:04.212580+0000","last_became_active":"2021-03-15T22:29:51.231818+0000","last_became_peered":"2021-03-15T22:29:51.231818+0000","last_unstale":"2021-03-15T22:30:04.212580+0000","last_undegraded":"2021-03-15T22:30:04.212580+0000","last_fullsized":"2021-03-15T22:30:04.212580+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:07.148878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:07.148878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"282","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515685+0000","last_change":"2021-03-15T22:29:13.250177+0000","last_active":"2021-03-15T22:30:05.515685+0000","last_peered":"2021-03-15T22:30:05.515685+0000","last_clean":"2021-03-15T22:30:05.515685+0000","last_became_active":"2021-03-15T22:29:11.204254+0000","last_became_peered":"2021-03-15T22:29:11.204254+0000","last_unstale":"2021-03-15T22:30:05.515685+0000","last_undegraded":"2021-03-15T22:30:05.515685+0000","last_fullsized":"2021-03-15T22:30:05.515685+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:29:13.250082+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:29:13.250082+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"653'706","reported_seq":"2575","reported_epoch":"653","state":"active+clean","last_fresh":"2021-03-15T22:30:02.232503+0000","last_change":"2021-03-15T22:30:02.232929+0000","last_active":"2021-03-15T22:30:02.232503+0000","last_peered":"2021-03-15T22:30:02.232503+0000","last_clean":"2021-03-15T22:30:02.232503+0000","last_became_active":"2021-03-15T22:29:52.227518+0000","last_became_peered":"2021-03-15T22:29:52.227518+0000","last_unstale":"2021-03-15T22:30:02.232503+0000","last_undegraded":"2021-03-15T22:30:02.232503+0000","last_fullsized":"2021-03-15T22:30:02.232503+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'650","last_scrub_stamp":"2021-03-15T22:29:40.242110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.242110+0000","log_size":706,"ondisk_log_size":706,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26548437,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":344,"num_read_kb":396845,"num_write":222,"num_write_kb":71077,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":30385610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"572","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.216609+0000","last_change":"2021-03-15T22:29:52.217756+0000","last_active":"2021-03-15T22:30:03.216609+0000","last_peered":"2021-03-15T22:30:03.216609+0000","last_clean":"2021-03-15T22:30:03.216609+0000","last_became_active":"2021-03-15T22:29:52.217465+0000","last_became_peered":"2021-03-15T22:29:52.217465+0000","last_unstale":"2021-03-15T22:30:03.216609+0000","last_undegraded":"2021-03-15T22:30:03.216609+0000","last_fullsized":"2021-03-15T22:30:03.216609+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:32.224256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:32.224256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"656","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.257140+0000","last_change":"2021-03-15T22:28:12.530786+0000","last_active":"2021-03-15T22:30:03.257140+0000","last_peered":"2021-03-15T22:30:03.257140+0000","last_clean":"2021-03-15T22:30:03.257140+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:30:03.257140+0000","last_undegraded":"2021-03-15T22:30:03.257140+0000","last_fullsized":"2021-03-15T22:30:03.257140+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:27:40.278709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:40.278709+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"555'30","reported_seq":"45","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.216296+0000","last_change":"2021-03-15T22:29:52.958056+0000","last_active":"2021-03-15T22:30:03.216296+0000","last_peered":"2021-03-15T22:30:03.216296+0000","last_clean":"2021-03-15T22:30:03.216296+0000","last_became_active":"2021-03-15T22:29:51.221013+0000","last_became_peered":"2021-03-15T22:29:51.221013+0000","last_unstale":"2021-03-15T22:30:03.216296+0000","last_undegraded":"2021-03-15T22:30:03.216296+0000","last_fullsized":"2021-03-15T22:30:03.216296+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'30","last_scrub_stamp":"2021-03-15T22:29:14.052742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:14.052742+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"317'7","reported_seq":"779","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.256622+0000","last_change":"2021-03-15T22:27:11.357502+0000","last_active":"2021-03-15T22:30:03.256622+0000","last_peered":"2021-03-15T22:30:03.256622+0000","last_clean":"2021-03-15T22:30:03.256622+0000","last_became_active":"2021-03-15T22:23:52.262878+0000","last_became_peered":"2021-03-15T22:23:52.262878+0000","last_unstale":"2021-03-15T22:30:03.256622+0000","last_undegraded":"2021-03-15T22:30:03.256622+0000","last_fullsized":"2021-03-15T22:30:03.256622+0000","mapping_epoch":316,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:27:11.357399+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:27:11.357399+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"655'960","reported_seq":"2767","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.515356+0000","last_change":"2021-03-15T22:30:04.232356+0000","last_active":"2021-03-15T22:30:05.515356+0000","last_peered":"2021-03-15T22:30:05.515356+0000","last_clean":"2021-03-15T22:30:05.515356+0000","last_became_active":"2021-03-15T22:29:52.224819+0000","last_became_peered":"2021-03-15T22:29:52.224819+0000","last_unstale":"2021-03-15T22:30:05.515356+0000","last_undegraded":"2021-03-15T22:30:05.515356+0000","last_fullsized":"2021-03-15T22:30:05.515356+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'857","last_scrub_stamp":"2021-03-15T22:29:33.490626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.490626+0000","log_size":960,"ondisk_log_size":960,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37522752,"num_objects":19,"num_object_clones":14,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":455,"num_read_kb":453678,"num_write":304,"num_write_kb":118978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":75,"num_bytes_recovered":48432527,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5,1],"avail_no_missing":["3","1","5"],"object_location_counts":[{"shards":"1,3,5","objects":19}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"82","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515091+0000","last_change":"2021-03-15T22:29:58.503593+0000","last_active":"2021-03-15T22:30:05.515091+0000","last_peered":"2021-03-15T22:30:05.515091+0000","last_clean":"2021-03-15T22:30:05.515091+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:30:05.515091+0000","last_undegraded":"2021-03-15T22:30:05.515091+0000","last_fullsized":"2021-03-15T22:30:05.515091+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:58.503533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:58.503533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"625","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.257012+0000","last_change":"2021-03-15T22:27:44.227979+0000","last_active":"2021-03-15T22:30:03.257012+0000","last_peered":"2021-03-15T22:30:03.257012+0000","last_clean":"2021-03-15T22:30:03.257012+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:30:03.257012+0000","last_undegraded":"2021-03-15T22:30:03.257012+0000","last_fullsized":"2021-03-15T22:30:03.257012+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:44.227918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:44.227918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"775","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389659+0000","last_change":"2021-03-15T22:29:10.861350+0000","last_active":"2021-03-15T22:30:05.389659+0000","last_peered":"2021-03-15T22:30:05.389659+0000","last_clean":"2021-03-15T22:30:05.389659+0000","last_became_active":"2021-03-15T22:28:59.267623+0000","last_became_peered":"2021-03-15T22:28:59.267623+0000","last_unstale":"2021-03-15T22:30:05.389659+0000","last_undegraded":"2021-03-15T22:30:05.389659+0000","last_fullsized":"2021-03-15T22:30:05.389659+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:29:10.861249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:10.861249+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"124","reported_epoch":"655","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:04.214736+0000","last_change":"2021-03-15T22:29:56.479503+0000","last_active":"2021-03-15T22:30:04.214736+0000","last_peered":"2021-03-15T22:30:04.214736+0000","last_clean":"2021-03-15T22:30:04.214736+0000","last_became_active":"2021-03-15T22:29:27.239512+0000","last_became_peered":"2021-03-15T22:29:27.239512+0000","last_unstale":"2021-03-15T22:30:04.214736+0000","last_undegraded":"2021-03-15T22:30:04.214736+0000","last_fullsized":"2021-03-15T22:30:04.214736+0000","mapping_epoch":617,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:56.479415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.479415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,5],"avail_no_missing":["0","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"655'277","reported_seq":"1990","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.222119+0000","last_change":"2021-03-15T22:30:04.222119+0000","last_active":"2021-03-15T22:30:04.222119+0000","last_peered":"2021-03-15T22:30:04.222119+0000","last_clean":"2021-03-15T22:30:04.222119+0000","last_became_active":"2021-03-15T22:28:16.177364+0000","last_became_peered":"2021-03-15T22:28:16.177364+0000","last_unstale":"2021-03-15T22:30:04.222119+0000","last_undegraded":"2021-03-15T22:30:04.222119+0000","last_fullsized":"2021-03-15T22:30:04.222119+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'183","last_scrub_stamp":"2021-03-15T22:27:09.383406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:27:09.383406+0000","log_size":277,"ondisk_log_size":277,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8756810,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":83,"num_read_kb":66680,"num_write":72,"num_write_kb":23091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":11148008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"661","reported_epoch":"654","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:03.256392+0000","last_change":"2021-03-15T22:29:08.138093+0000","last_active":"2021-03-15T22:30:03.256392+0000","last_peered":"2021-03-15T22:30:03.256392+0000","last_clean":"2021-03-15T22:30:03.256392+0000","last_became_active":"2021-03-15T22:29:00.214243+0000","last_became_peered":"2021-03-15T22:29:00.214243+0000","last_unstale":"2021-03-15T22:30:03.256392+0000","last_undegraded":"2021-03-15T22:30:03.256392+0000","last_fullsized":"2021-03-15T22:30:03.256392+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:08.137989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:08.137989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3,6],"avail_no_missing":["7","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"141","reported_epoch":"655","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:04.214956+0000","last_change":"2021-03-15T22:29:59.529320+0000","last_active":"2021-03-15T22:30:04.214956+0000","last_peered":"2021-03-15T22:30:04.214956+0000","last_clean":"2021-03-15T22:30:04.214956+0000","last_became_active":"2021-03-15T22:29:52.205954+0000","last_became_peered":"2021-03-15T22:29:52.205954+0000","last_unstale":"2021-03-15T22:30:04.214956+0000","last_undegraded":"2021-03-15T22:30:04.214956+0000","last_fullsized":"2021-03-15T22:30:04.214956+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:59.529159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:59.529159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5,7],"avail_no_missing":["0","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"662","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.257562+0000","last_change":"2021-03-15T22:29:51.222856+0000","last_active":"2021-03-15T22:30:03.257562+0000","last_peered":"2021-03-15T22:30:03.257562+0000","last_clean":"2021-03-15T22:30:03.257562+0000","last_became_active":"2021-03-15T22:29:51.222437+0000","last_became_peered":"2021-03-15T22:29:51.222437+0000","last_unstale":"2021-03-15T22:30:03.257562+0000","last_undegraded":"2021-03-15T22:30:03.257562+0000","last_fullsized":"2021-03-15T22:30:03.257562+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:12.404498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:27:12.404498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"727","reported_epoch":"655","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:30:04.212941+0000","last_change":"2021-03-15T22:26:46.158466+0000","last_active":"2021-03-15T22:30:04.212941+0000","last_peered":"2021-03-15T22:30:04.212941+0000","last_clean":"2021-03-15T22:26:44.404347+0000","last_became_active":"2021-03-15T22:26:46.158466+0000","last_became_peered":"2021-03-15T22:26:46.158466+0000","last_unstale":"2021-03-15T22:30:04.212941+0000","last_undegraded":"2021-03-15T22:26:46.143093+0000","last_fullsized":"2021-03-15T22:26:46.142970+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":329,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:26:11.821564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:11.821564+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":["6","7"],"object_location_counts":[{"shards":"6,7","objects":13}],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"643'513","reported_seq":"1923","reported_epoch":"654","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:30:03.256359+0000","last_change":"2021-03-15T22:26:46.147101+0000","last_active":"2021-03-15T22:30:03.256359+0000","last_peered":"2021-03-15T22:30:03.256359+0000","last_clean":"2021-03-15T22:26:41.176252+0000","last_became_active":"2021-03-15T22:26:46.147101+0000","last_became_peered":"2021-03-15T22:26:46.147101+0000","last_unstale":"2021-03-15T22:30:03.256359+0000","last_undegraded":"2021-03-15T22:26:46.139401+0000","last_fullsized":"2021-03-15T22:26:46.139252+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":92,"parent":"0.0","parent_split_bits":0,"last_scrub":"418'328","last_scrub_stamp":"2021-03-15T22:25:37.707812+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:25:37.707812+0000","log_size":513,"ondisk_log_size":513,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":68,"stat_sum":{"num_bytes":46007036,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":256,"num_read_kb":300311,"num_write":170,"num_write_kb":59847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":["7","4"],"object_location_counts":[{"shards":"4,7","objects":21}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"119","reported_epoch":"655","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:04.214535+0000","last_change":"2021-03-15T22:28:49.778347+0000","last_active":"2021-03-15T22:30:04.214535+0000","last_peered":"2021-03-15T22:30:04.214535+0000","last_clean":"2021-03-15T22:30:04.214535+0000","last_became_active":"2021-03-15T22:28:13.504327+0000","last_became_peered":"2021-03-15T22:28:13.504327+0000","last_unstale":"2021-03-15T22:30:04.214535+0000","last_undegraded":"2021-03-15T22:30:04.214535+0000","last_fullsized":"2021-03-15T22:30:04.214535+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:49.778250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:28:49.778250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7,3],"avail_no_missing":["0","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"619","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.389960+0000","last_change":"2021-03-15T22:29:00.214887+0000","last_active":"2021-03-15T22:30:05.389960+0000","last_peered":"2021-03-15T22:30:05.389960+0000","last_clean":"2021-03-15T22:30:05.389960+0000","last_became_active":"2021-03-15T22:29:00.214238+0000","last_became_peered":"2021-03-15T22:29:00.214238+0000","last_unstale":"2021-03-15T22:30:05.389960+0000","last_undegraded":"2021-03-15T22:30:05.389960+0000","last_fullsized":"2021-03-15T22:30:05.389960+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:50.922664+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:28:50.922664+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,0],"avail_no_missing":["4","0","3"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"23","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.216667+0000","last_change":"2021-03-15T22:29:51.232011+0000","last_active":"2021-03-15T22:30:03.216667+0000","last_peered":"2021-03-15T22:30:03.216667+0000","last_clean":"2021-03-15T22:30:03.216667+0000","last_became_active":"2021-03-15T22:29:51.231699+0000","last_became_peered":"2021-03-15T22:29:51.231699+0000","last_unstale":"2021-03-15T22:30:03.216667+0000","last_undegraded":"2021-03-15T22:30:03.216667+0000","last_fullsized":"2021-03-15T22:30:03.216667+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:44.484267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:28:44.484267+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"148","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.601396+0000","last_change":"2021-03-15T22:30:05.601396+0000","last_active":"2021-03-15T22:30:05.601396+0000","last_peered":"2021-03-15T22:30:05.601396+0000","last_clean":"2021-03-15T22:30:05.601396+0000","last_became_active":"2021-03-15T22:28:13.546532+0000","last_became_peered":"2021-03-15T22:28:13.546532+0000","last_unstale":"2021-03-15T22:30:05.601396+0000","last_undegraded":"2021-03-15T22:30:05.601396+0000","last_fullsized":"2021-03-15T22:30:05.601396+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:30:05.601295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:05.601295+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4,3],"avail_no_missing":["0","3","4"],"object_location_counts":[{"shards":"0,3,4","objects":9}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"655'762","reported_seq":"2906","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.224993+0000","last_change":"2021-03-15T22:30:04.224993+0000","last_active":"2021-03-15T22:30:04.224993+0000","last_peered":"2021-03-15T22:30:04.224993+0000","last_clean":"2021-03-15T22:30:04.224993+0000","last_became_active":"2021-03-15T22:29:02.359450+0000","last_became_peered":"2021-03-15T22:29:02.359450+0000","last_unstale":"2021-03-15T22:30:04.224993+0000","last_undegraded":"2021-03-15T22:30:04.224993+0000","last_fullsized":"2021-03-15T22:30:04.224993+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"606'654","last_scrub_stamp":"2021-03-15T22:29:17.132235+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:29:17.132235+0000","log_size":762,"ondisk_log_size":762,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32697268,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":204,"num_read_kb":162509,"num_write":228,"num_write_kb":72671,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":114286074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"77","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515716+0000","last_change":"2021-03-15T22:28:59.274160+0000","last_active":"2021-03-15T22:30:05.515716+0000","last_peered":"2021-03-15T22:30:05.515716+0000","last_clean":"2021-03-15T22:30:05.515716+0000","last_became_active":"2021-03-15T22:28:59.273785+0000","last_became_peered":"2021-03-15T22:28:59.273785+0000","last_unstale":"2021-03-15T22:30:05.515716+0000","last_undegraded":"2021-03-15T22:30:05.515716+0000","last_fullsized":"2021-03-15T22:30:05.515716+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:44.156498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:27:44.156498+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"23","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.216501+0000","last_change":"2021-03-15T22:29:51.236372+0000","last_active":"2021-03-15T22:30:03.216501+0000","last_peered":"2021-03-15T22:30:03.216501+0000","last_clean":"2021-03-15T22:30:03.216501+0000","last_became_active":"2021-03-15T22:29:51.236007+0000","last_became_peered":"2021-03-15T22:29:51.236007+0000","last_unstale":"2021-03-15T22:30:03.216501+0000","last_undegraded":"2021-03-15T22:30:03.216501+0000","last_fullsized":"2021-03-15T22:30:03.216501+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:33.191779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.191779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"107","reported_epoch":"655","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:04.214710+0000","last_change":"2021-03-15T22:28:28.430654+0000","last_active":"2021-03-15T22:30:04.214710+0000","last_peered":"2021-03-15T22:30:04.214710+0000","last_clean":"2021-03-15T22:30:04.214710+0000","last_became_active":"2021-03-15T22:28:28.430214+0000","last_became_peered":"2021-03-15T22:28:28.430214+0000","last_unstale":"2021-03-15T22:30:04.214710+0000","last_undegraded":"2021-03-15T22:30:04.214710+0000","last_fullsized":"2021-03-15T22:30:04.214710+0000","mapping_epoch":563,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":564,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:36.883125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:36.883125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6],"acting":[0,6,3],"avail_no_missing":["0","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"834","reported_epoch":"652","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:01.212274+0000","last_change":"2021-03-15T22:29:00.216301+0000","last_active":"2021-03-15T22:30:01.212274+0000","last_peered":"2021-03-15T22:30:01.212274+0000","last_clean":"2021-03-15T22:30:01.212274+0000","last_became_active":"2021-03-15T22:29:00.215768+0000","last_became_peered":"2021-03-15T22:29:00.215768+0000","last_unstale":"2021-03-15T22:30:01.212274+0000","last_undegraded":"2021-03-15T22:30:01.212274+0000","last_fullsized":"2021-03-15T22:30:01.212274+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:28:45.649404+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.649404+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3,6],"avail_no_missing":["1","3","6"],"object_location_counts":[{"shards":"1,3,6","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"113","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.215360+0000","last_change":"2021-03-15T22:28:12.519853+0000","last_active":"2021-03-15T22:30:04.215360+0000","last_peered":"2021-03-15T22:30:04.215360+0000","last_clean":"2021-03-15T22:30:04.215360+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:30:04.215360+0000","last_undegraded":"2021-03-15T22:30:04.215360+0000","last_fullsized":"2021-03-15T22:30:04.215360+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:10.399386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:27:10.399386+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"656'718","reported_seq":"2404","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.403910+0000","last_change":"2021-03-15T22:30:04.231227+0000","last_active":"2021-03-15T22:30:05.403910+0000","last_peered":"2021-03-15T22:30:05.403910+0000","last_clean":"2021-03-15T22:30:05.403910+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:30:05.403910+0000","last_undegraded":"2021-03-15T22:30:05.403910+0000","last_fullsized":"2021-03-15T22:30:05.403910+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"511'527","last_scrub_stamp":"2021-03-15T22:27:12.808295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:27:12.808295+0000","log_size":718,"ondisk_log_size":718,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37905268,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":197,"num_read_kb":176869,"num_write":196,"num_write_kb":75067,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"654","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.257636+0000","last_change":"2021-03-15T22:29:07.163096+0000","last_active":"2021-03-15T22:30:03.257636+0000","last_peered":"2021-03-15T22:30:03.257636+0000","last_clean":"2021-03-15T22:30:03.257636+0000","last_became_active":"2021-03-15T22:28:59.278695+0000","last_became_peered":"2021-03-15T22:28:59.278695+0000","last_unstale":"2021-03-15T22:30:03.257636+0000","last_undegraded":"2021-03-15T22:30:03.257636+0000","last_fullsized":"2021-03-15T22:30:03.257636+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:07.163034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:07.163034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"626","reported_epoch":"652","state":"active+undersized","last_fresh":"2021-03-15T22:30:01.211421+0000","last_change":"2021-03-15T22:26:46.145950+0000","last_active":"2021-03-15T22:30:01.211421+0000","last_peered":"2021-03-15T22:30:01.211421+0000","last_clean":"2021-03-15T22:26:44.403138+0000","last_became_active":"2021-03-15T22:26:46.145950+0000","last_became_peered":"2021-03-15T22:26:46.145950+0000","last_unstale":"2021-03-15T22:30:01.211421+0000","last_undegraded":"2021-03-15T22:30:01.211421+0000","last_fullsized":"2021-03-15T22:26:46.139516+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:39.929066+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:24:39.929066+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"586","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.515352+0000","last_change":"2021-03-15T22:29:00.214685+0000","last_active":"2021-03-15T22:30:05.515352+0000","last_peered":"2021-03-15T22:30:05.515352+0000","last_clean":"2021-03-15T22:30:05.515352+0000","last_became_active":"2021-03-15T22:29:00.214130+0000","last_became_peered":"2021-03-15T22:29:00.214130+0000","last_unstale":"2021-03-15T22:30:05.515352+0000","last_undegraded":"2021-03-15T22:30:05.515352+0000","last_fullsized":"2021-03-15T22:30:05.515352+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:38.142756+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:28:38.142756+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0,1],"avail_no_missing":["3","0","1"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"391","reported_epoch":"652","state":"active+clean","last_fresh":"2021-03-15T22:30:01.212228+0000","last_change":"2021-03-15T22:29:56.187294+0000","last_active":"2021-03-15T22:30:01.212228+0000","last_peered":"2021-03-15T22:30:01.212228+0000","last_clean":"2021-03-15T22:30:01.212228+0000","last_became_active":"2021-03-15T22:29:51.222262+0000","last_became_peered":"2021-03-15T22:29:51.222262+0000","last_unstale":"2021-03-15T22:30:01.212228+0000","last_undegraded":"2021-03-15T22:30:01.212228+0000","last_fullsized":"2021-03-15T22:30:01.212228+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:29:56.187210+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.187210+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"272","reported_epoch":"654","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:03.217014+0000","last_change":"2021-03-15T22:29:55.737465+0000","last_active":"2021-03-15T22:30:03.217014+0000","last_peered":"2021-03-15T22:30:03.217014+0000","last_clean":"2021-03-15T22:30:03.217014+0000","last_became_active":"2021-03-15T22:29:52.217331+0000","last_became_peered":"2021-03-15T22:29:52.217331+0000","last_unstale":"2021-03-15T22:30:03.217014+0000","last_undegraded":"2021-03-15T22:30:03.217014+0000","last_fullsized":"2021-03-15T22:30:03.217014+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:55.737371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:55.737371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7,4],"avail_no_missing":["5","4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"655'1768","reported_seq":"4131","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.242514+0000","last_change":"2021-03-15T22:30:04.242514+0000","last_active":"2021-03-15T22:30:04.242514+0000","last_peered":"2021-03-15T22:30:04.242514+0000","last_clean":"2021-03-15T22:30:04.242514+0000","last_became_active":"2021-03-15T22:28:59.276888+0000","last_became_peered":"2021-03-15T22:28:59.276888+0000","last_unstale":"2021-03-15T22:30:04.242514+0000","last_undegraded":"2021-03-15T22:30:04.242514+0000","last_fullsized":"2021-03-15T22:30:04.242514+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'1398","last_scrub_stamp":"2021-03-15T22:28:51.825151+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:28:51.825151+0000","log_size":1768,"ondisk_log_size":1768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":53251328,"num_objects":28,"num_object_clones":21,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":610,"num_read_kb":612585,"num_write":479,"num_write_kb":200603,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":205133613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"23","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.217030+0000","last_change":"2021-03-15T22:29:51.231275+0000","last_active":"2021-03-15T22:30:03.217030+0000","last_peered":"2021-03-15T22:30:03.217030+0000","last_clean":"2021-03-15T22:30:03.217030+0000","last_became_active":"2021-03-15T22:29:51.227896+0000","last_became_peered":"2021-03-15T22:29:51.227896+0000","last_unstale":"2021-03-15T22:30:03.217030+0000","last_undegraded":"2021-03-15T22:30:03.217030+0000","last_fullsized":"2021-03-15T22:30:03.217030+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:18.266805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:18.266805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"641","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.256614+0000","last_change":"2021-03-15T22:28:12.508332+0000","last_active":"2021-03-15T22:30:03.256614+0000","last_peered":"2021-03-15T22:30:03.256614+0000","last_clean":"2021-03-15T22:30:03.256614+0000","last_became_active":"2021-03-15T22:28:12.507833+0000","last_became_peered":"2021-03-15T22:28:12.507833+0000","last_unstale":"2021-03-15T22:30:03.256614+0000","last_undegraded":"2021-03-15T22:30:03.256614+0000","last_fullsized":"2021-03-15T22:30:03.256614+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:08.415630+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:27:08.415630+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"265","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515165+0000","last_change":"2021-03-15T22:29:51.223894+0000","last_active":"2021-03-15T22:30:05.515165+0000","last_peered":"2021-03-15T22:30:05.515165+0000","last_clean":"2021-03-15T22:30:05.515165+0000","last_became_active":"2021-03-15T22:29:51.223572+0000","last_became_peered":"2021-03-15T22:29:51.223572+0000","last_unstale":"2021-03-15T22:30:05.515165+0000","last_undegraded":"2021-03-15T22:30:05.515165+0000","last_fullsized":"2021-03-15T22:30:05.515165+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:45.454461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.454461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"673","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.213105+0000","last_change":"2021-03-15T22:29:10.153389+0000","last_active":"2021-03-15T22:30:04.213105+0000","last_peered":"2021-03-15T22:30:04.213105+0000","last_clean":"2021-03-15T22:30:04.213105+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:30:04.213105+0000","last_undegraded":"2021-03-15T22:30:04.213105+0000","last_fullsized":"2021-03-15T22:30:04.213105+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:29:10.153322+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:10.153322+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"657","reported_epoch":"654","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:03.257704+0000","last_change":"2021-03-15T22:30:03.257615+0000","last_active":"2021-03-15T22:30:03.257704+0000","last_peered":"2021-03-15T22:30:03.257704+0000","last_clean":"2021-03-15T22:30:03.257704+0000","last_became_active":"2021-03-15T22:29:52.223862+0000","last_became_peered":"2021-03-15T22:29:52.223862+0000","last_unstale":"2021-03-15T22:30:03.257704+0000","last_undegraded":"2021-03-15T22:30:03.257704+0000","last_fullsized":"2021-03-15T22:30:03.257704+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.257476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,0],"avail_no_missing":["7","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"655'505","reported_seq":"2313","reported_epoch":"655","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:04.223307+0000","last_change":"2021-03-15T22:30:04.223611+0000","last_active":"2021-03-15T22:30:04.223307+0000","last_peered":"2021-03-15T22:30:04.223307+0000","last_clean":"2021-03-15T22:30:04.223307+0000","last_became_active":"2021-03-15T22:28:20.178773+0000","last_became_peered":"2021-03-15T22:28:20.178773+0000","last_unstale":"2021-03-15T22:30:04.223307+0000","last_undegraded":"2021-03-15T22:30:04.223307+0000","last_fullsized":"2021-03-15T22:30:04.223307+0000","mapping_epoch":559,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":560,"parent":"0.0","parent_split_bits":0,"last_scrub":"560'396","last_scrub_stamp":"2021-03-15T22:28:45.222374+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.222374+0000","log_size":505,"ondisk_log_size":505,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27005649,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":158,"num_read_kb":157134,"num_write":143,"num_write_kb":65313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":51,"num_bytes_recovered":33674420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0,1],"avail_no_missing":["7","0","1"],"object_location_counts":[{"shards":"0,1,7","objects":10}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"686","reported_epoch":"652","state":"active+clean","last_fresh":"2021-03-15T22:30:01.211913+0000","last_change":"2021-03-15T22:29:51.222534+0000","last_active":"2021-03-15T22:30:01.211913+0000","last_peered":"2021-03-15T22:30:01.211913+0000","last_clean":"2021-03-15T22:30:01.211913+0000","last_became_active":"2021-03-15T22:29:51.222189+0000","last_became_peered":"2021-03-15T22:29:51.222189+0000","last_unstale":"2021-03-15T22:30:01.211913+0000","last_undegraded":"2021-03-15T22:30:01.211913+0000","last_fullsized":"2021-03-15T22:30:01.211913+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:51.588451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:28:51.588451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"132","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.214666+0000","last_change":"2021-03-15T22:28:59.276953+0000","last_active":"2021-03-15T22:30:04.214666+0000","last_peered":"2021-03-15T22:30:04.214666+0000","last_clean":"2021-03-15T22:30:04.214666+0000","last_became_active":"2021-03-15T22:28:59.276599+0000","last_became_peered":"2021-03-15T22:28:59.276599+0000","last_unstale":"2021-03-15T22:30:04.214666+0000","last_undegraded":"2021-03-15T22:30:04.214666+0000","last_fullsized":"2021-03-15T22:30:04.214666+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:17.909060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:17.909060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"1373","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.658483+0000","last_change":"2021-03-15T22:28:52.152942+0000","last_active":"2021-03-15T22:30:03.658483+0000","last_peered":"2021-03-15T22:30:03.658483+0000","last_clean":"2021-03-15T22:30:03.658483+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:30:03.658483+0000","last_undegraded":"2021-03-15T22:30:03.658483+0000","last_fullsized":"2021-03-15T22:30:03.658483+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:28:52.152875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:28:52.152875+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":714,"num_read_kb":709,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"156","reported_epoch":"655","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:04.215569+0000","last_change":"2021-03-15T22:28:45.780545+0000","last_active":"2021-03-15T22:30:04.215569+0000","last_peered":"2021-03-15T22:30:04.215569+0000","last_clean":"2021-03-15T22:30:04.215569+0000","last_became_active":"2021-03-15T22:28:13.532930+0000","last_became_peered":"2021-03-15T22:28:13.532930+0000","last_unstale":"2021-03-15T22:30:04.215569+0000","last_undegraded":"2021-03-15T22:30:04.215569+0000","last_fullsized":"2021-03-15T22:30:04.215569+0000","mapping_epoch":553,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":554,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:28:45.780434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.780434+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1,7],"avail_no_missing":["0","1","7"],"object_location_counts":[{"shards":"0,1,7","objects":10}],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"23","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.217033+0000","last_change":"2021-03-15T22:29:51.232258+0000","last_active":"2021-03-15T22:30:03.217033+0000","last_peered":"2021-03-15T22:30:03.217033+0000","last_clean":"2021-03-15T22:30:03.217033+0000","last_became_active":"2021-03-15T22:29:51.231915+0000","last_became_peered":"2021-03-15T22:29:51.231915+0000","last_unstale":"2021-03-15T22:30:03.217033+0000","last_undegraded":"2021-03-15T22:30:03.217033+0000","last_fullsized":"2021-03-15T22:30:03.217033+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:16.238757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:29:16.238757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"655'1862","reported_seq":"4198","reported_epoch":"655","state":"active+clean+snaptrim","last_fresh":"2021-03-15T22:30:04.229536+0000","last_change":"2021-03-15T22:30:04.222183+0000","last_active":"2021-03-15T22:30:04.229536+0000","last_peered":"2021-03-15T22:30:04.229536+0000","last_clean":"2021-03-15T22:30:04.229536+0000","last_became_active":"2021-03-15T22:29:52.225069+0000","last_became_peered":"2021-03-15T22:29:52.225069+0000","last_unstale":"2021-03-15T22:30:04.229536+0000","last_undegraded":"2021-03-15T22:30:04.229536+0000","last_fullsized":"2021-03-15T22:30:04.229536+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"650'1835","last_scrub_stamp":"2021-03-15T22:29:59.763141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:59.763141+0000","log_size":1862,"ondisk_log_size":1862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":83853761,"num_objects":40,"num_object_clones":33,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":1,"num_read":851,"num_read_kb":939386,"num_write":510,"num_write_kb":214105,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":165660176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"620","reported_epoch":"652","state":"active+undersized","last_fresh":"2021-03-15T22:30:01.211434+0000","last_change":"2021-03-15T22:26:46.145792+0000","last_active":"2021-03-15T22:30:01.211434+0000","last_peered":"2021-03-15T22:30:01.211434+0000","last_clean":"2021-03-15T22:26:44.402909+0000","last_became_active":"2021-03-15T22:26:46.145792+0000","last_became_peered":"2021-03-15T22:26:46.145792+0000","last_unstale":"2021-03-15T22:30:01.211434+0000","last_undegraded":"2021-03-15T22:30:01.211434+0000","last_fullsized":"2021-03-15T22:26:46.139336+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.950135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.950135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":["1","6"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"661","reported_epoch":"652","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:01.212115+0000","last_change":"2021-03-15T22:29:52.226769+0000","last_active":"2021-03-15T22:30:01.212115+0000","last_peered":"2021-03-15T22:30:01.212115+0000","last_clean":"2021-03-15T22:30:01.212115+0000","last_became_active":"2021-03-15T22:29:52.226341+0000","last_became_peered":"2021-03-15T22:29:52.226341+0000","last_unstale":"2021-03-15T22:30:01.212115+0000","last_undegraded":"2021-03-15T22:30:01.212115+0000","last_fullsized":"2021-03-15T22:30:01.212115+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:32.161342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:32.161342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5,0],"avail_no_missing":["1","0","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"193","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.215056+0000","last_change":"2021-03-15T22:29:52.258695+0000","last_active":"2021-03-15T22:30:04.215056+0000","last_peered":"2021-03-15T22:30:04.215056+0000","last_clean":"2021-03-15T22:30:04.215056+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:30:04.215056+0000","last_undegraded":"2021-03-15T22:30:04.215056+0000","last_fullsized":"2021-03-15T22:30:04.215056+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:29:08.872836+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:29:08.872836+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"670","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.212916+0000","last_change":"2021-03-15T22:29:51.233266+0000","last_active":"2021-03-15T22:30:04.212916+0000","last_peered":"2021-03-15T22:30:04.212916+0000","last_clean":"2021-03-15T22:30:04.212916+0000","last_became_active":"2021-03-15T22:29:51.232811+0000","last_became_peered":"2021-03-15T22:29:51.232811+0000","last_unstale":"2021-03-15T22:30:04.212916+0000","last_undegraded":"2021-03-15T22:30:04.212916+0000","last_fullsized":"2021-03-15T22:30:04.212916+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:33.255248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.255248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"655'318","reported_seq":"2186","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389272+0000","last_change":"2021-03-15T22:30:04.215445+0000","last_active":"2021-03-15T22:30:05.389272+0000","last_peered":"2021-03-15T22:30:05.389272+0000","last_clean":"2021-03-15T22:30:05.389272+0000","last_became_active":"2021-03-15T22:29:03.354952+0000","last_became_peered":"2021-03-15T22:29:03.354952+0000","last_unstale":"2021-03-15T22:30:05.389272+0000","last_undegraded":"2021-03-15T22:30:05.389272+0000","last_fullsized":"2021-03-15T22:30:05.389272+0000","mapping_epoch":593,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'265","last_scrub_stamp":"2021-03-15T22:29:09.816963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:09.816963+0000","log_size":318,"ondisk_log_size":318,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8847790,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":111,"num_read_kb":129149,"num_write":81,"num_write_kb":31147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":41911448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"137","reported_epoch":"655","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:04.214490+0000","last_change":"2021-03-15T22:29:37.725010+0000","last_active":"2021-03-15T22:30:04.214490+0000","last_peered":"2021-03-15T22:30:04.214490+0000","last_clean":"2021-03-15T22:30:04.214490+0000","last_became_active":"2021-03-15T22:29:27.238367+0000","last_became_peered":"2021-03-15T22:29:27.238367+0000","last_unstale":"2021-03-15T22:30:04.214490+0000","last_undegraded":"2021-03-15T22:30:04.214490+0000","last_fullsized":"2021-03-15T22:30:04.214490+0000","mapping_epoch":617,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":618,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:37.724915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:37.724915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3,5],"avail_no_missing":["0","3","5"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"579","reported_epoch":"654","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:03.216925+0000","last_change":"2021-03-15T22:29:58.713758+0000","last_active":"2021-03-15T22:30:03.216925+0000","last_peered":"2021-03-15T22:30:03.216925+0000","last_clean":"2021-03-15T22:30:03.216925+0000","last_became_active":"2021-03-15T22:29:52.224077+0000","last_became_peered":"2021-03-15T22:29:52.224077+0000","last_unstale":"2021-03-15T22:30:03.216925+0000","last_undegraded":"2021-03-15T22:30:03.216925+0000","last_fullsized":"2021-03-15T22:30:03.216925+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:58.713662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:29:58.713662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3,6],"avail_no_missing":["5","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"856","reported_epoch":"652","state":"active+clean","last_fresh":"2021-03-15T22:30:01.211844+0000","last_change":"2021-03-15T22:29:52.437785+0000","last_active":"2021-03-15T22:30:01.211844+0000","last_peered":"2021-03-15T22:30:01.211844+0000","last_clean":"2021-03-15T22:30:01.211844+0000","last_became_active":"2021-03-15T22:29:51.222356+0000","last_became_peered":"2021-03-15T22:29:51.222356+0000","last_unstale":"2021-03-15T22:30:01.211844+0000","last_undegraded":"2021-03-15T22:30:01.211844+0000","last_fullsized":"2021-03-15T22:30:01.211844+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:29:20.139319+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"642'8","reported_seq":"171","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.214883+0000","last_change":"2021-03-15T22:29:51.399268+0000","last_active":"2021-03-15T22:30:04.214883+0000","last_peered":"2021-03-15T22:30:04.214883+0000","last_clean":"2021-03-15T22:30:04.214883+0000","last_became_active":"2021-03-15T22:29:51.215636+0000","last_became_peered":"2021-03-15T22:29:51.215636+0000","last_unstale":"2021-03-15T22:30:04.214883+0000","last_undegraded":"2021-03-15T22:30:04.214883+0000","last_fullsized":"2021-03-15T22:30:04.214883+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'7","last_scrub_stamp":"2021-03-15T22:29:12.845802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:12.845802+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"655'480","reported_seq":"2195","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.515589+0000","last_change":"2021-03-15T22:30:04.232677+0000","last_active":"2021-03-15T22:30:05.515589+0000","last_peered":"2021-03-15T22:30:05.515589+0000","last_clean":"2021-03-15T22:30:05.515589+0000","last_became_active":"2021-03-15T22:29:00.225848+0000","last_became_peered":"2021-03-15T22:29:00.225848+0000","last_unstale":"2021-03-15T22:30:05.515589+0000","last_undegraded":"2021-03-15T22:30:05.515589+0000","last_fullsized":"2021-03-15T22:30:05.515589+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'390","last_scrub_stamp":"2021-03-15T22:28:47.641188+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:28:47.641188+0000","log_size":480,"ondisk_log_size":480,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20864307,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":170,"num_read_kb":155470,"num_write":142,"num_write_kb":56205,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":25727267,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1,7],"avail_no_missing":["3","1","7"],"object_location_counts":[{"shards":"1,3,7","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"618","reported_epoch":"654","state":"active+undersized","last_fresh":"2021-03-15T22:30:03.257047+0000","last_change":"2021-03-15T22:26:46.158212+0000","last_active":"2021-03-15T22:30:03.257047+0000","last_peered":"2021-03-15T22:30:03.257047+0000","last_clean":"2021-03-15T22:26:44.403963+0000","last_became_active":"2021-03-15T22:26:46.158212+0000","last_became_peered":"2021-03-15T22:26:46.158212+0000","last_unstale":"2021-03-15T22:30:03.257047+0000","last_undegraded":"2021-03-15T22:30:03.257047+0000","last_fullsized":"2021-03-15T22:26:46.146288+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.234565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.234565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":["7","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"113","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.215713+0000","last_change":"2021-03-15T22:28:12.526699+0000","last_active":"2021-03-15T22:30:04.215713+0000","last_peered":"2021-03-15T22:30:04.215713+0000","last_clean":"2021-03-15T22:30:04.215713+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:30:04.215713+0000","last_undegraded":"2021-03-15T22:30:04.215713+0000","last_fullsized":"2021-03-15T22:30:04.215713+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:08.372872+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:08.372872+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"765","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389651+0000","last_change":"2021-03-15T22:29:52.488194+0000","last_active":"2021-03-15T22:30:05.389651+0000","last_peered":"2021-03-15T22:30:05.389651+0000","last_clean":"2021-03-15T22:30:05.389651+0000","last_became_active":"2021-03-15T22:29:51.216039+0000","last_became_peered":"2021-03-15T22:29:51.216039+0000","last_unstale":"2021-03-15T22:30:05.389651+0000","last_undegraded":"2021-03-15T22:30:05.389651+0000","last_fullsized":"2021-03-15T22:30:05.389651+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:29:14.888009+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:14.888009+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"94","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.514821+0000","last_change":"2021-03-15T22:28:59.315468+0000","last_active":"2021-03-15T22:30:05.514821+0000","last_peered":"2021-03-15T22:30:05.514821+0000","last_clean":"2021-03-15T22:30:05.514821+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:30:05.514821+0000","last_undegraded":"2021-03-15T22:30:05.514821+0000","last_fullsized":"2021-03-15T22:30:05.514821+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"421'6","last_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:27:47.109636+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"655'823","reported_seq":"2691","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.515431+0000","last_change":"2021-03-15T22:30:04.227464+0000","last_active":"2021-03-15T22:30:05.515431+0000","last_peered":"2021-03-15T22:30:05.515431+0000","last_clean":"2021-03-15T22:30:05.515431+0000","last_became_active":"2021-03-15T22:29:00.214651+0000","last_became_peered":"2021-03-15T22:29:00.214651+0000","last_unstale":"2021-03-15T22:30:05.515431+0000","last_undegraded":"2021-03-15T22:30:05.515431+0000","last_fullsized":"2021-03-15T22:30:05.515431+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'662","last_scrub_stamp":"2021-03-15T22:28:45.043828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.043828+0000","log_size":823,"ondisk_log_size":823,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25794192,"num_objects":13,"num_object_clones":9,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":13,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":334,"num_read_kb":348790,"num_write":221,"num_write_kb":79373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":65272638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,6],"avail_no_missing":["3","4","6"],"object_location_counts":[{"shards":"3,4,6","objects":13}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"669","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.389447+0000","last_change":"2021-03-15T22:29:00.214632+0000","last_active":"2021-03-15T22:30:05.389447+0000","last_peered":"2021-03-15T22:30:05.389447+0000","last_clean":"2021-03-15T22:30:05.389447+0000","last_became_active":"2021-03-15T22:29:00.213052+0000","last_became_peered":"2021-03-15T22:29:00.213052+0000","last_unstale":"2021-03-15T22:30:05.389447+0000","last_undegraded":"2021-03-15T22:30:05.389447+0000","last_fullsized":"2021-03-15T22:30:05.389447+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:51.970593+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:28:51.970593+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3],"acting":[4,3,6],"avail_no_missing":["4","3","6"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"432","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515859+0000","last_change":"2021-03-15T22:29:00.238851+0000","last_active":"2021-03-15T22:30:05.515859+0000","last_peered":"2021-03-15T22:30:05.515859+0000","last_clean":"2021-03-15T22:30:05.515859+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:30:05.515859+0000","last_undegraded":"2021-03-15T22:30:05.515859+0000","last_fullsized":"2021-03-15T22:30:05.515859+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:37.136505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:28:37.136505+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"108","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.514911+0000","last_change":"2021-03-15T22:29:41.515256+0000","last_active":"2021-03-15T22:30:05.514911+0000","last_peered":"2021-03-15T22:30:05.514911+0000","last_clean":"2021-03-15T22:30:05.514911+0000","last_became_active":"2021-03-15T22:28:59.276755+0000","last_became_peered":"2021-03-15T22:28:59.276755+0000","last_unstale":"2021-03-15T22:30:05.514911+0000","last_undegraded":"2021-03-15T22:30:05.514911+0000","last_fullsized":"2021-03-15T22:30:05.514911+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:29:41.515193+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.515193+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"674","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389411+0000","last_change":"2021-03-15T22:29:19.876088+0000","last_active":"2021-03-15T22:30:05.389411+0000","last_peered":"2021-03-15T22:30:05.389411+0000","last_clean":"2021-03-15T22:30:05.389411+0000","last_became_active":"2021-03-15T22:28:59.266487+0000","last_became_peered":"2021-03-15T22:28:59.266487+0000","last_unstale":"2021-03-15T22:30:05.389411+0000","last_undegraded":"2021-03-15T22:30:05.389411+0000","last_fullsized":"2021-03-15T22:30:05.389411+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:19.876006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:29:19.876006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"655'811","reported_seq":"2607","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.223564+0000","last_change":"2021-03-15T22:30:04.223564+0000","last_active":"2021-03-15T22:30:04.223564+0000","last_peered":"2021-03-15T22:30:04.223564+0000","last_clean":"2021-03-15T22:30:04.223564+0000","last_became_active":"2021-03-15T22:29:00.220080+0000","last_became_peered":"2021-03-15T22:29:00.220080+0000","last_unstale":"2021-03-15T22:30:04.223564+0000","last_undegraded":"2021-03-15T22:30:04.223564+0000","last_fullsized":"2021-03-15T22:30:04.223564+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'732","last_scrub_stamp":"2021-03-15T22:29:36.271647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:36.271647+0000","log_size":811,"ondisk_log_size":811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26339498,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":287,"num_read_kb":275920,"num_write":203,"num_write_kb":74347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25937939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"574","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.216894+0000","last_change":"2021-03-15T22:29:52.227178+0000","last_active":"2021-03-15T22:30:03.216894+0000","last_peered":"2021-03-15T22:30:03.216894+0000","last_clean":"2021-03-15T22:30:03.216894+0000","last_became_active":"2021-03-15T22:29:52.223342+0000","last_became_peered":"2021-03-15T22:29:52.223342+0000","last_unstale":"2021-03-15T22:30:03.216894+0000","last_undegraded":"2021-03-15T22:30:03.216894+0000","last_fullsized":"2021-03-15T22:30:03.216894+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:35.243847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:35.243847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"643'13","reported_seq":"867","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389974+0000","last_change":"2021-03-15T22:29:52.212266+0000","last_active":"2021-03-15T22:30:05.389974+0000","last_peered":"2021-03-15T22:30:05.389974+0000","last_clean":"2021-03-15T22:30:05.389974+0000","last_became_active":"2021-03-15T22:29:52.211688+0000","last_became_peered":"2021-03-15T22:29:52.211688+0000","last_unstale":"2021-03-15T22:30:05.389974+0000","last_undegraded":"2021-03-15T22:30:05.389974+0000","last_fullsized":"2021-03-15T22:30:05.389974+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'11","last_scrub_stamp":"2021-03-15T22:29:36.937870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:36.937870+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"77","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.514787+0000","last_change":"2021-03-15T22:28:59.275938+0000","last_active":"2021-03-15T22:30:05.514787+0000","last_peered":"2021-03-15T22:30:05.514787+0000","last_clean":"2021-03-15T22:30:05.514787+0000","last_became_active":"2021-03-15T22:28:59.274186+0000","last_became_peered":"2021-03-15T22:28:59.274186+0000","last_unstale":"2021-03-15T22:30:05.514787+0000","last_undegraded":"2021-03-15T22:30:05.514787+0000","last_fullsized":"2021-03-15T22:30:05.514787+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:27:43.184429+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"437","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.515840+0000","last_change":"2021-03-15T22:29:15.297624+0000","last_active":"2021-03-15T22:30:05.515840+0000","last_peered":"2021-03-15T22:30:05.515840+0000","last_clean":"2021-03-15T22:30:05.515840+0000","last_became_active":"2021-03-15T22:29:00.232626+0000","last_became_peered":"2021-03-15T22:29:00.232626+0000","last_unstale":"2021-03-15T22:30:05.515840+0000","last_undegraded":"2021-03-15T22:30:05.515840+0000","last_fullsized":"2021-03-15T22:30:05.515840+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:29:15.297479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:15.297479+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,7],"avail_no_missing":["3","4","7"],"object_location_counts":[{"shards":"3,4,7","objects":9}],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.d","version":"651'262","reported_seq":"2015","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.213686+0000","last_change":"2021-03-15T22:30:04.213686+0000","last_active":"2021-03-15T22:30:04.213686+0000","last_peered":"2021-03-15T22:30:04.213686+0000","last_clean":"2021-03-15T22:30:04.213686+0000","last_became_active":"2021-03-15T22:28:15.172010+0000","last_became_peered":"2021-03-15T22:28:15.172010+0000","last_unstale":"2021-03-15T22:30:04.213686+0000","last_undegraded":"2021-03-15T22:30:04.213686+0000","last_fullsized":"2021-03-15T22:30:04.213686+0000","mapping_epoch":555,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"598'212","last_scrub_stamp":"2021-03-15T22:29:08.155000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:08.155000+0000","log_size":262,"ondisk_log_size":262,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13624200,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":105,"num_read_kb":100342,"num_write":55,"num_write_kb":20180,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":9579561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"371","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515421+0000","last_change":"2021-03-15T22:29:51.225632+0000","last_active":"2021-03-15T22:30:05.515421+0000","last_peered":"2021-03-15T22:30:05.515421+0000","last_clean":"2021-03-15T22:30:05.515421+0000","last_became_active":"2021-03-15T22:29:51.225325+0000","last_became_peered":"2021-03-15T22:29:51.225325+0000","last_unstale":"2021-03-15T22:30:05.515421+0000","last_undegraded":"2021-03-15T22:30:05.515421+0000","last_fullsized":"2021-03-15T22:30:05.515421+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:40.500803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.500803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"618","reported_epoch":"654","state":"active+clean","last_fresh":"2021-03-15T22:30:03.217177+0000","last_change":"2021-03-15T22:29:52.217795+0000","last_active":"2021-03-15T22:30:03.217177+0000","last_peered":"2021-03-15T22:30:03.217177+0000","last_clean":"2021-03-15T22:30:03.217177+0000","last_became_active":"2021-03-15T22:29:52.217407+0000","last_became_peered":"2021-03-15T22:29:52.217407+0000","last_unstale":"2021-03-15T22:30:03.217177+0000","last_undegraded":"2021-03-15T22:30:03.217177+0000","last_fullsized":"2021-03-15T22:30:03.217177+0000","mapping_epoch":642,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'7","last_scrub_stamp":"2021-03-15T22:29:40.964753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.964753+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"629","reported_epoch":"656","state":"active+undersized","last_fresh":"2021-03-15T22:30:05.389510+0000","last_change":"2021-03-15T22:26:46.147246+0000","last_active":"2021-03-15T22:30:05.389510+0000","last_peered":"2021-03-15T22:30:05.389510+0000","last_clean":"2021-03-15T22:26:44.341791+0000","last_became_active":"2021-03-15T22:26:46.147246+0000","last_became_peered":"2021-03-15T22:26:46.147246+0000","last_unstale":"2021-03-15T22:30:05.389510+0000","last_undegraded":"2021-03-15T22:30:05.389510+0000","last_fullsized":"2021-03-15T22:26:46.138599+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:34.991018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:55.098239+0000","last_clean_scrub_stamp":"2021-03-15T22:24:34.991018+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":["4","7"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"731","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.389802+0000","last_change":"2021-03-15T22:30:04.058254+0000","last_active":"2021-03-15T22:30:05.389802+0000","last_peered":"2021-03-15T22:30:05.389802+0000","last_clean":"2021-03-15T22:30:05.389802+0000","last_became_active":"2021-03-15T22:24:55.090544+0000","last_became_peered":"2021-03-15T22:24:55.090544+0000","last_unstale":"2021-03-15T22:30:05.389802+0000","last_undegraded":"2021-03-15T22:30:05.389802+0000","last_fullsized":"2021-03-15T22:30:05.389802+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:30:04.058167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:04.058167+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"655'229","reported_seq":"2081","reported_epoch":"655","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:04.226999+0000","last_change":"2021-03-15T22:30:04.227401+0000","last_active":"2021-03-15T22:30:04.226999+0000","last_peered":"2021-03-15T22:30:04.226999+0000","last_clean":"2021-03-15T22:30:04.226999+0000","last_became_active":"2021-03-15T22:29:52.222404+0000","last_became_peered":"2021-03-15T22:29:52.222404+0000","last_unstale":"2021-03-15T22:30:04.226999+0000","last_undegraded":"2021-03-15T22:30:04.226999+0000","last_fullsized":"2021-03-15T22:30:04.226999+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"646'221","last_scrub_stamp":"2021-03-15T22:29:56.742213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.742213+0000","log_size":229,"ondisk_log_size":229,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13368182,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":145,"num_read_kb":164356,"num_write":70,"num_write_kb":22842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":20001404,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0,3],"avail_no_missing":["5","0","3"],"object_location_counts":[{"shards":"0,3,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"554","reported_epoch":"656","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:05.515748+0000","last_change":"2021-03-15T22:29:17.298146+0000","last_active":"2021-03-15T22:30:05.515748+0000","last_peered":"2021-03-15T22:30:05.515748+0000","last_clean":"2021-03-15T22:30:05.515748+0000","last_became_active":"2021-03-15T22:29:00.233628+0000","last_became_peered":"2021-03-15T22:29:00.233628+0000","last_unstale":"2021-03-15T22:30:05.515748+0000","last_undegraded":"2021-03-15T22:30:05.515748+0000","last_fullsized":"2021-03-15T22:30:05.515748+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:17.298043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:29:17.298043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4,6],"avail_no_missing":["3","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"667","reported_epoch":"652","state":"active+clean","last_fresh":"2021-03-15T22:30:01.212446+0000","last_change":"2021-03-15T22:29:41.190121+0000","last_active":"2021-03-15T22:30:01.212446+0000","last_peered":"2021-03-15T22:30:01.212446+0000","last_clean":"2021-03-15T22:30:01.212446+0000","last_became_active":"2021-03-15T22:28:12.503247+0000","last_became_peered":"2021-03-15T22:28:12.503247+0000","last_unstale":"2021-03-15T22:30:01.212446+0000","last_undegraded":"2021-03-15T22:30:01.212446+0000","last_fullsized":"2021-03-15T22:30:01.212446+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:41.190058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.190058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"638","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.212827+0000","last_change":"2021-03-15T22:29:11.129378+0000","last_active":"2021-03-15T22:30:04.212827+0000","last_peered":"2021-03-15T22:30:04.212827+0000","last_clean":"2021-03-15T22:30:04.212827+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:30:04.212827+0000","last_undegraded":"2021-03-15T22:30:04.212827+0000","last_fullsized":"2021-03-15T22:30:04.212827+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:11.129316+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:11.129316+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"696","reported_epoch":"656","state":"active+undersized+degraded","last_fresh":"2021-03-15T22:30:05.389783+0000","last_change":"2021-03-15T22:26:46.147116+0000","last_active":"2021-03-15T22:30:05.389783+0000","last_peered":"2021-03-15T22:30:05.389783+0000","last_clean":"2021-03-15T22:26:44.341791+0000","last_became_active":"2021-03-15T22:26:46.147116+0000","last_became_peered":"2021-03-15T22:26:46.147116+0000","last_unstale":"2021-03-15T22:30:05.389783+0000","last_undegraded":"2021-03-15T22:26:46.138362+0000","last_fullsized":"2021-03-15T22:26:46.138236+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"115'33","last_scrub_stamp":"2021-03-15T22:25:04.028780+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:25:04.028780+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[{"shards":"1,4","objects":12}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"655'816","reported_seq":"2950","reported_epoch":"655","state":"active+clean","last_fresh":"2021-03-15T22:30:04.381093+0000","last_change":"2021-03-15T22:30:04.242292+0000","last_active":"2021-03-15T22:30:04.381093+0000","last_peered":"2021-03-15T22:30:04.381093+0000","last_clean":"2021-03-15T22:30:04.381093+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:30:04.381093+0000","last_undegraded":"2021-03-15T22:30:04.381093+0000","last_fullsized":"2021-03-15T22:30:04.381093+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"486'449","last_scrub_stamp":"2021-03-15T22:26:42.479453+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:26:42.479453+0000","log_size":816,"ondisk_log_size":816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42609040,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":332,"num_read_kb":314598,"num_write":234,"num_write_kb":84743,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"102","length":"1"},{"start":"10c","length":"1"},{"start":"123","length":"1"},{"start":"12f","length":"1"},{"start":"131","length":"1"},{"start":"13e","length":"1"},{"start":"141","length":"1"},{"start":"15b","length":"1"},{"start":"15f","length":"1"},{"start":"161","length":"1"},{"start":"166","length":"1"},{"start":"168","length":"1"},{"start":"16d","length":"2"},{"start":"170","length":"1"},{"start":"173","length":"1"},{"start":"175","length":"1"},{"start":"177","length":"1"},{"start":"17e","length":"4"},{"start":"184","length":"2"},{"start":"187","length":"1"},{"start":"18a","length":"2"},{"start":"18f","length":"2"},{"start":"193","length":"3"},{"start":"199","length":"4"},{"start":"19f","length":"2"},{"start":"1a2","length":"2"},{"start":"1a6","length":"2"},{"start":"1a9","length":"1"},{"start":"1ac","length":"1"},{"start":"1ae","length":"1"},{"start":"1b0","length":"2"},{"start":"1b3","length":"2"},{"start":"1b6","length":"2"},{"start":"1bb","length":"1"},{"start":"1bd","length":"2"},{"start":"1c0","length":"1"},{"start":"1c4","length":"1"},{"start":"1c8","length":"1"},{"start":"1cc","length":"1"},{"start":"1d1","length":"1"},{"start":"1d4","length":"1"},{"start":"1d7","length":"1"},{"start":"1d9","length":"1"},{"start":"1dd","length":"1"},{"start":"1df","length":"1"},{"start":"1e1","length":"1"},{"start":"1f0","length":"1"},{"start":"1f3","length":"1"},{"start":"1fc","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"621","reported_epoch":"656","state":"active+undersized","last_fresh":"2021-03-15T22:30:05.390056+0000","last_change":"2021-03-15T22:26:46.144801+0000","last_active":"2021-03-15T22:30:05.390056+0000","last_peered":"2021-03-15T22:30:05.390056+0000","last_clean":"2021-03-15T22:26:44.342145+0000","last_became_active":"2021-03-15T22:26:46.144801+0000","last_became_peered":"2021-03-15T22:26:46.144801+0000","last_unstale":"2021-03-15T22:30:05.390056+0000","last_undegraded":"2021-03-15T22:30:05.390056+0000","last_fullsized":"2021-03-15T22:26:46.138535+0000","mapping_epoch":490,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:40.910565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:24:40.910565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":["4","1"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"643'10","reported_seq":"251","reported_epoch":"654","state":"active+clean+remapped","last_fresh":"2021-03-15T22:30:03.257478+0000","last_change":"2021-03-15T22:29:52.229338+0000","last_active":"2021-03-15T22:30:03.257478+0000","last_peered":"2021-03-15T22:30:03.257478+0000","last_clean":"2021-03-15T22:30:03.257478+0000","last_became_active":"2021-03-15T22:29:52.227853+0000","last_became_peered":"2021-03-15T22:29:52.227853+0000","last_unstale":"2021-03-15T22:30:03.257478+0000","last_undegraded":"2021-03-15T22:30:03.257478+0000","last_fullsized":"2021-03-15T22:30:03.257478+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'8","last_scrub_stamp":"2021-03-15T22:29:38.210344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:38.210344+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5,0],"avail_no_missing":["7","0","5"],"object_location_counts":[{"shards":"0,5,7","objects":1}],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"82","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.515133+0000","last_change":"2021-03-15T22:30:03.567594+0000","last_active":"2021-03-15T22:30:05.515133+0000","last_peered":"2021-03-15T22:30:05.515133+0000","last_clean":"2021-03-15T22:30:05.515133+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:30:05.515133+0000","last_undegraded":"2021-03-15T22:30:05.515133+0000","last_fullsized":"2021-03-15T22:30:05.515133+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.567529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"780","reported_epoch":"656","state":"active+clean","last_fresh":"2021-03-15T22:30:05.390131+0000","last_change":"2021-03-15T22:30:00.030068+0000","last_active":"2021-03-15T22:30:05.390131+0000","last_peered":"2021-03-15T22:30:05.390131+0000","last_clean":"2021-03-15T22:30:05.390131+0000","last_became_active":"2021-03-15T22:28:59.268492+0000","last_became_peered":"2021-03-15T22:28:59.268492+0000","last_unstale":"2021-03-15T22:30:05.390131+0000","last_undegraded":"2021-03-15T22:30:05.390131+0000","last_fullsized":"2021-03-15T22:30:05.390131+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:30:00.029954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:00.029954+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":504995518,"num_objects":235,"num_object_clones":185,"num_object_copies":705,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":57,"num_objects_unfound":0,"num_objects_dirty":235,"num_whiteouts":6,"num_read":4642,"num_read_kb":4754622,"num_write":3330,"num_write_kb":1269589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1864,"num_bytes_recovered":922640243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1217019904,"data_stored":1702215245,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11810,"ondisk_log_size":11810,"up":42,"acting":47,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":714,"num_read_kb":709,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":8,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":86,"acting":91,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":35,"num_objects_misplaced":60,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1484,"num_read_kb":989,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":729,"num_bytes_recovered":59582,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2129920,"data_stored":76412,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":906,"ondisk_log_size":906,"up":87,"acting":93,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":65,"ondisk_log_size":65,"up":59,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":84,"acting":91,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299542,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":203348,"kb_used_data":163596,"kb_used_omap":0,"kb_used_meta":39744,"kb_avail":93541804,"statfs":{"total":95995035648,"available":95786807296,"internally_reserved":0,"allocated":167522304,"data_stored":237112439,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40697856},"hb_peers":[0,1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:29:10 2021","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.47099999999999997,"15min":0.48499999999999999},"min":{"1min":0.27500000000000002,"5min":0.215,"15min":0.215},"max":{"1min":0.55200000000000005,"5min":1.4750000000000001,"15min":2.73},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.41299999999999998,"5min":0.47299999999999998,"15min":0.49399999999999999},"min":{"1min":0.251,"5min":0.222,"15min":0.222},"max":{"1min":0.58199999999999996,"5min":2.0499999999999998,"15min":3.8799999999999999},"last":0.36799999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:29:10 2021","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.47499999999999998,"15min":0.48399999999999999},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.22600000000000001},"max":{"1min":0.60299999999999998,"5min":1.4350000000000001,"15min":2.6859999999999999},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.48999999999999999,"15min":0.49199999999999999},"min":{"1min":0.311,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.56399999999999995,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.51200000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.80700000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.502},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.32400000000000001,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":0.73099999999999998,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.48799999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:29:10 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.497,"15min":0.52200000000000002},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.59899999999999998,"5min":1.702,"15min":2.0339999999999998},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.40500000000000003,"5min":0.55600000000000005,"15min":0.55400000000000005},"min":{"1min":0.314,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.52000000000000002,"5min":5.415,"15min":5.415},"last":0.48999999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:29:10 2021","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.42799999999999999,"15min":0.44},"min":{"1min":0.246,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":0.53200000000000003,"5min":1.927,"15min":1.927},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.38200000000000001,"5min":0.44600000000000001,"15min":0.45400000000000001},"min":{"1min":0.253,"5min":0.253,"15min":0.253},"max":{"1min":0.57099999999999995,"5min":1.6779999999999999,"15min":2.5339999999999998},"last":0.621}]},{"osd":5,"last update":"Mon Mar 15 22:27:06 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.51300000000000001,"15min":0.47299999999999998},"min":{"1min":0.379,"5min":0.253,"15min":0.253},"max":{"1min":1.407,"5min":1.407,"15min":2.5110000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.66200000000000003,"5min":0.52900000000000003,"15min":0.51500000000000001},"min":{"1min":0.34999999999999998,"5min":0.255,"15min":0.255},"max":{"1min":2.0979999999999999,"5min":2.0979999999999999,"15min":2.0979999999999999},"last":0.58299999999999996}]},{"osd":6,"last update":"Mon Mar 15 22:29:10 2021","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.49399999999999999,"15min":0.51200000000000001},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.47299999999999998,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.39100000000000001,"5min":0.48899999999999999,"15min":0.50900000000000001},"min":{"1min":0.28499999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.56799999999999995,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":0.60599999999999998}]}]},{"osd":6,"up_from":45,"seq":193273528474,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":128120,"kb_used_data":95408,"kb_used_omap":0,"kb_used_meta":32704,"kb_avail":93617032,"statfs":{"total":95995035648,"available":95863840768,"internally_reserved":0,"allocated":97697792,"data_stored":140231317,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33488896},"hb_peers":[0,1,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:29:56 2021","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.47299999999999998,"15min":0.48299999999999998},"min":{"1min":0.253,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.52400000000000002,"5min":3.085,"15min":3.085},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.435,"15min":0.46999999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27100000000000002},"max":{"1min":0.53700000000000003,"5min":1.323,"15min":1.323},"last":0.41099999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:29:56 2021","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.60899999999999999,"15min":0.54200000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.254},"max":{"1min":0.55400000000000005,"5min":14.637,"15min":14.637},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.47699999999999998,"15min":0.48499999999999999},"min":{"1min":0.32800000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.47999999999999998,"5min":5.2450000000000001,"15min":5.2450000000000001},"last":0.316}]},{"osd":2,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.317,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65200000000000002,"5min":0.77600000000000002,"15min":0.80800000000000005},"last":0.625},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48899999999999999,"15min":0.498},"min":{"1min":0.35399999999999998,"5min":0.32800000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.42899999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:29:56 2021","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.503,"15min":0.51400000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.29899999999999999},"max":{"1min":0.51100000000000001,"5min":2.6150000000000002,"15min":2.6150000000000002},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.48899999999999999,"15min":0.51200000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.28699999999999998},"max":{"1min":0.56699999999999995,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.34799999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:29:56 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.41899999999999998,"15min":0.42299999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.497,"5min":4.306,"15min":4.306},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.252,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54800000000000004,"5min":4.181,"15min":4.181},"last":0.371}]},{"osd":5,"last update":"Mon Mar 15 22:26:47 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.498,"15min":0.46500000000000002},"min":{"1min":0.308,"5min":0.308,"15min":0.308},"max":{"1min":0.66700000000000004,"5min":0.89200000000000002,"15min":1.135},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.53700000000000003,"15min":0.495},"min":{"1min":0.371,"5min":0.33200000000000002,"15min":0.311},"max":{"1min":0.65100000000000002,"5min":0.99299999999999999,"15min":0.99299999999999999},"last":0.35699999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:29:08 2021","interfaces":[{"interface":"back","average":{"1min":1.109,"5min":0.61899999999999999,"15min":0.54600000000000004},"min":{"1min":0.255,"5min":0.249,"15min":0.249},"max":{"1min":15.340999999999999,"5min":15.340999999999999,"15min":15.340999999999999},"last":0.38500000000000001},{"interface":"front","average":{"1min":1.1519999999999999,"5min":0.65100000000000002,"15min":0.58199999999999996},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":15.356,"5min":15.356,"15min":15.356},"last":0.39400000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575019,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":236856,"kb_used_data":203568,"kb_used_omap":0,"kb_used_meta":33280,"kb_avail":93508296,"statfs":{"total":95995035648,"available":95752495104,"internally_reserved":0,"allocated":208453632,"data_stored":268976124,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34078720},"hb_peers":[0,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:29:38 2021","interfaces":[{"interface":"back","average":{"1min":0.27700000000000002,"5min":0.29299999999999998,"15min":0.51400000000000001},"min":{"1min":0.189,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.65100000000000002,"5min":0.65100000000000002,"15min":54.427999999999997},"last":0.26200000000000001},{"interface":"front","average":{"1min":0.313,"5min":0.32900000000000001,"15min":0.53300000000000003},"min":{"1min":0.20399999999999999,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.80000000000000004,"5min":0.80000000000000004,"15min":54.356999999999999},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.54300000000000004,"5min":1.1220000000000001,"15min":2.4100000000000001},"last":0.316},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.38600000000000001},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.502,"5min":0.69799999999999995,"15min":2.0760000000000001},"last":0.25900000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:29:07 2021","interfaces":[{"interface":"back","average":{"1min":0.318,"5min":0.36399999999999999,"15min":0.39500000000000002},"min":{"1min":0.224,"5min":0.215,"15min":0.215},"max":{"1min":0.73499999999999999,"5min":1.4910000000000001,"15min":1.796},"last":0.307},{"interface":"front","average":{"1min":0.33200000000000002,"5min":0.374,"15min":0.42799999999999999},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":0.69199999999999995,"5min":0.69199999999999995,"15min":1.7150000000000001},"last":0.32600000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:29:16 2021","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.48199999999999998,"15min":0.54600000000000004},"min":{"1min":0.247,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.76600000000000001,"5min":0.95199999999999996,"15min":3.3439999999999999},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.66000000000000003,"5min":0.54200000000000004,"15min":0.60199999999999998},"min":{"1min":0.27200000000000002,"5min":0.253,"15min":0.253},"max":{"1min":4.1790000000000003,"5min":4.1790000000000003,"15min":4.6139999999999999},"last":0.41299999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:27:33 2021","interfaces":[{"interface":"back","average":{"1min":0.49299999999999999,"5min":0.54700000000000004,"15min":0.60299999999999998},"min":{"1min":0.30499999999999999,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.76600000000000001,"5min":1.524,"15min":2.153},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56699999999999995,"15min":0.85999999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.82499999999999996,"5min":0.91300000000000003,"15min":15.472},"last":0.40699999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:29:48 2021","interfaces":[{"interface":"back","average":{"1min":0.47199999999999998,"5min":0.53200000000000003,"15min":0.60599999999999998},"min":{"1min":0.36899999999999999,"5min":0.314,"15min":0.314},"max":{"1min":0.66100000000000003,"5min":0.91800000000000004,"15min":1.75},"last":0.438},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.53200000000000003,"15min":0.58499999999999996},"min":{"1min":0.35099999999999998,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.65000000000000002,"5min":0.86599999999999999,"15min":1.3580000000000001},"last":0.50600000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:29:07 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.56100000000000005,"15min":0.74199999999999999},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.83299999999999996,"5min":0.96299999999999997,"15min":8.8100000000000005},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.54700000000000004,"15min":0.59299999999999997},"min":{"1min":0.37,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.83999999999999997,"5min":0.84799999999999998,"15min":1.325},"last":0.48399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738542,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":211352,"kb_used_data":176464,"kb_used_omap":0,"kb_used_meta":34880,"kb_avail":93533800,"statfs":{"total":95995035648,"available":95778611200,"internally_reserved":0,"allocated":180699136,"data_stored":239858316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35717120},"hb_peers":[1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:29:33 2021","interfaces":[{"interface":"back","average":{"1min":0.33000000000000002,"5min":0.35699999999999998,"15min":0.34799999999999998},"min":{"1min":0.19700000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.58099999999999996,"5min":1.7070000000000001,"15min":1.7070000000000001},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.374,"15min":0.36899999999999999},"min":{"1min":0.22,"5min":0.19900000000000001,"15min":0.19900000000000001},"max":{"1min":0.67000000000000004,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.436}]},{"osd":2,"last update":"Mon Mar 15 22:26:35 2021","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39200000000000002,"15min":0.39500000000000002},"min":{"1min":0.23899999999999999,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.76100000000000001,"5min":1.97,"15min":1.97},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.38500000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.20699999999999999},"max":{"1min":1.1419999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:29:11 2021","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.40500000000000003,"15min":0.39800000000000002},"min":{"1min":0.23000000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.538,"5min":1.538,"15min":1.538},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.39700000000000002,"5min":0.40899999999999997,"15min":0.41299999999999998},"min":{"1min":0.25,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":1.161,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.47899999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:29:13 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.53900000000000003,"15min":0.54400000000000004},"min":{"1min":0.34300000000000003,"5min":0.254,"15min":0.254},"max":{"1min":0.57599999999999996,"5min":1.583,"15min":1.583},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.55600000000000005,"15min":0.57899999999999996},"min":{"1min":0.28899999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.53900000000000003,"5min":1.645,"15min":1.645},"last":0.52000000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:26:31 2021","interfaces":[{"interface":"back","average":{"1min":0.57299999999999995,"5min":0.56999999999999995,"15min":0.59599999999999997},"min":{"1min":0.34100000000000003,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.93200000000000005,"5min":1.677,"15min":1.677},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.61599999999999999,"5min":0.56000000000000005,"15min":0.57299999999999995},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33500000000000002},"max":{"1min":0.997,"5min":0.997,"15min":1.0149999999999999},"last":0.58799999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:29:50 2021","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.55200000000000005,"15min":0.56100000000000005},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.80800000000000005,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.60399999999999998},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.874,"5min":0.98099999999999998,"15min":1.0049999999999999},"last":0.55100000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:29:13 2021","interfaces":[{"interface":"back","average":{"1min":0.44,"5min":0.55800000000000005,"15min":0.57099999999999995},"min":{"1min":0.34100000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.622,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.57499999999999996,"15min":0.57499999999999996},"min":{"1min":0.35199999999999998,"5min":0.317,"15min":0.317},"max":{"1min":0.61599999999999999,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.61899999999999999}]}]},{"osd":2,"up_from":18,"seq":77309411455,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":140056,"kb_used_data":107664,"kb_used_omap":0,"kb_used_meta":32384,"kb_avail":93605096,"statfs":{"total":95995035648,"available":95851618304,"internally_reserved":0,"allocated":110247936,"data_stored":155725916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33161216},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.34000000000000002,"15min":0.34799999999999998},"min":{"1min":0.246,"5min":0.18099999999999999,"15min":0.18099999999999999},"max":{"1min":0.45700000000000002,"5min":2.0019999999999998,"15min":2.0019999999999998},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.437,"15min":0.39500000000000002},"min":{"1min":0.246,"5min":0.21099999999999999,"15min":0.191},"max":{"1min":5.5250000000000004,"5min":5.5250000000000004,"15min":5.5250000000000004},"last":0.30599999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.42599999999999999,"15min":0.39600000000000002},"min":{"1min":0.22700000000000001,"5min":0.22,"15min":0.22},"max":{"1min":0.60999999999999999,"5min":5.4489999999999998,"15min":5.4489999999999998},"last":0.373},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.51800000000000002,"15min":0.46600000000000003},"min":{"1min":0.27400000000000002,"5min":0.19800000000000001,"15min":0.19800000000000001},"max":{"1min":2.3740000000000001,"5min":11.659000000000001,"15min":11.659000000000001},"last":0.38500000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:25:58 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.38400000000000001,"15min":0.41399999999999998},"min":{"1min":0.23699999999999999,"5min":0.23499999999999999,"15min":0.22},"max":{"1min":1.2010000000000001,"5min":2.0640000000000001,"15min":2.0640000000000001},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.44700000000000001,"15min":0.47699999999999998},"min":{"1min":0.30199999999999999,"5min":0.26400000000000001,"15min":0.24099999999999999},"max":{"1min":1.28,"5min":2.8759999999999999,"15min":2.8759999999999999},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:26:12 2021","interfaces":[{"interface":"back","average":{"1min":0.55000000000000004,"5min":0.55900000000000005,"15min":0.57299999999999995},"min":{"1min":0.38,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":1.3100000000000001,"5min":1.9850000000000001,"15min":1.9850000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53300000000000003,"15min":0.55100000000000005},"min":{"1min":0.39100000000000001,"5min":0.33400000000000002,"15min":0.33400000000000002},"max":{"1min":0.78400000000000003,"5min":1.8100000000000001,"15min":1.8100000000000001},"last":0.755}]},{"osd":5,"last update":"Mon Mar 15 22:26:37 2021","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.54800000000000004,"15min":0.57199999999999995},"min":{"1min":0.372,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.175,"5min":1.175,"15min":1.954},"last":0.71199999999999997},{"interface":"front","average":{"1min":0.5,"5min":0.54400000000000004,"15min":0.56599999999999995},"min":{"1min":0.40300000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66000000000000003,"5min":0.85799999999999998,"15min":2.2919999999999998},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:25:42 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.58499999999999996,"15min":0.57099999999999995},"min":{"1min":0.42699999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.72399999999999998,"5min":1.875,"15min":1.875},"last":0.67100000000000004},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.58899999999999997,"15min":0.57399999999999995},"min":{"1min":0.435,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":1.3520000000000001,"5min":1.609,"15min":1.609},"last":0.69099999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:26:12 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56999999999999995,"15min":0.55800000000000005},"min":{"1min":0.38900000000000001,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":1.3320000000000001,"5min":1.7190000000000001,"15min":1.7190000000000001},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.57799999999999996,"15min":0.58099999999999996},"min":{"1min":0.44800000000000001,"5min":0.40600000000000003,"15min":0.38700000000000001},"max":{"1min":0.68400000000000005,"5min":1.923,"15min":1.923},"last":0.625}]}]},{"osd":3,"up_from":26,"seq":111669149860,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":309676,"kb_used_data":264100,"kb_used_omap":0,"kb_used_meta":45568,"kb_avail":93435476,"statfs":{"total":95995035648,"available":95677927424,"internally_reserved":0,"allocated":270438400,"data_stored":375685848,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46661632},"hb_peers":[0,1,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:29:11 2021","interfaces":[{"interface":"back","average":{"1min":0.311,"5min":0.316,"15min":0.34399999999999997},"min":{"1min":0.184,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.56799999999999995,"5min":1.1499999999999999,"15min":2.625},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.309,"5min":0.315,"15min":0.35599999999999998},"min":{"1min":0.19800000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.51400000000000001,"5min":0.68100000000000005,"15min":2.6339999999999999},"last":0.25600000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:29:11 2021","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.35599999999999998,"15min":0.41199999999999998},"min":{"1min":0.23300000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.58699999999999997,"5min":0.67300000000000004,"15min":6.0199999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.372,"15min":0.41899999999999998},"min":{"1min":0.20699999999999999,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":2.5750000000000002,"5min":2.5750000000000002,"15min":7.6130000000000004},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51800000000000002,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.42199999999999999,"15min":0.39900000000000002},"min":{"1min":0.27000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54100000000000004,"5min":7.0860000000000003,"15min":7.0860000000000003},"last":0.36599999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:29:19 2021","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.504,"15min":0.52600000000000002},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67200000000000004,"5min":1.089,"15min":2.5920000000000001},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.46000000000000002,"5min":0.48699999999999999,"15min":0.52300000000000002},"min":{"1min":0.313,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.71399999999999997,"5min":1.0409999999999999,"15min":2.2890000000000001},"last":0.45400000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:27:32 2021","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.55900000000000005,"15min":0.54200000000000004},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.33400000000000002},"max":{"1min":0.78700000000000003,"5min":3.5259999999999998,"15min":3.5259999999999998},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.56599999999999995,"15min":0.56399999999999995},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.81699999999999995,"5min":2.6749999999999998,"15min":2.6749999999999998},"last":0.433}]},{"osd":6,"last update":"Mon Mar 15 22:29:51 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53600000000000003,"15min":0.57799999999999996},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.5299999999999998,"5min":2.5299999999999998,"15min":2.6499999999999999},"last":0.60799999999999998},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55700000000000005,"15min":0.57099999999999995},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.36099999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:29:11 2021","interfaces":[{"interface":"back","average":{"1min":0.68400000000000005,"5min":0.55400000000000005,"15min":0.57499999999999996},"min":{"1min":0.32700000000000001,"5min":0.32700000000000001,"15min":0.32700000000000001},"max":{"1min":4.4889999999999999,"5min":4.4889999999999999,"15min":4.4889999999999999},"last":0.443},{"interface":"front","average":{"1min":0.498,"5min":0.52800000000000002,"15min":0.56200000000000006},"min":{"1min":0.36399999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.70199999999999996,"5min":1.0129999999999999,"15min":2.0960000000000001},"last":0.41099999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986337,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":131324,"kb_used_data":99828,"kb_used_omap":0,"kb_used_meta":31488,"kb_avail":93613828,"statfs":{"total":95995035648,"available":95860559872,"internally_reserved":0,"allocated":102223872,"data_stored":138354080,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32243712},"hb_peers":[0,1,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:29:24 2021","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.48099999999999998,"15min":0.48199999999999998},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":0.86199999999999999,"5min":4.3769999999999998,"15min":4.3769999999999998},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.55900000000000005,"15min":0.51100000000000001},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.251},"max":{"1min":5.0209999999999999,"5min":5.0209999999999999,"15min":5.0209999999999999},"last":0.32600000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:29:24 2021","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.496,"15min":0.49399999999999999},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23599999999999999},"max":{"1min":0.81699999999999995,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.255},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.50700000000000001,"15min":0.51300000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.55800000000000005,"5min":4.718,"15min":4.718},"last":0.38100000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:26:18 2021","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48399999999999999,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.313},"max":{"1min":0.61299999999999999,"5min":0.70399999999999996,"15min":0.76100000000000001},"last":0.39300000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48299999999999998,"15min":0.502},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68799999999999994,"5min":0.71699999999999997,"15min":1.1910000000000001},"last":0.45000000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:29:24 2021","interfaces":[{"interface":"back","average":{"1min":0.70899999999999996,"5min":0.59199999999999997,"15min":0.56200000000000006},"min":{"1min":0.28499999999999998,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":5.0499999999999998,"5min":5.0499999999999998,"15min":5.0499999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.55700000000000005,"15min":0.55100000000000005},"min":{"1min":0.34200000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.84099999999999997,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.34000000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:27:32 2021","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.441,"15min":0.42199999999999999},"min":{"1min":0.28899999999999998,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.56499999999999995,"5min":0.745,"15min":0.745},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.47199999999999998,"15min":0.44700000000000001},"min":{"1min":0.27100000000000002,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.55300000000000005,"5min":0.72499999999999998,"15min":0.79200000000000004},"last":0.44}]},{"osd":6,"last update":"Mon Mar 15 22:29:51 2021","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.45500000000000002,"15min":0.44700000000000001},"min":{"1min":0.24199999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.88,"5min":4.3079999999999998,"15min":4.3079999999999998},"last":0.317},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.52200000000000002,"15min":0.52300000000000002},"min":{"1min":0.29999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.752,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.42299999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:29:06 2021","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.51400000000000001,"15min":0.502},"min":{"1min":0.27000000000000002,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":4.6929999999999996,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.82099999999999995,"5min":0.57499999999999996,"15min":0.54600000000000004},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":4.9900000000000002,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":0.36499999999999999}]}]},{"osd":5,"up_from":616,"seq":2645699854345,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159076,"kb_used_data":135452,"kb_used_omap":153,"kb_used_meta":23462,"kb_avail":93586076,"statfs":{"total":95995035648,"available":95832141824,"internally_reserved":0,"allocated":138702848,"data_stored":180825118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":24025617},"hb_peers":[0,1,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.55400000000000005}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.59099999999999997}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54400000000000004}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32800000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36899999999999999}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":7530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":12458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":233472,"data_stored":8356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":352256,"data_stored":12606,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":6418,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":290816,"data_stored":10582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":237568,"data_stored":8552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":173039616,"data_stored":234944230,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":200724480,"data_stored":267186513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":105123840,"data_stored":152421018,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262643712,"data_stored":370758415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":94511104,"data_stored":133438876,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":131125248,"data_stored":175933005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":89993216,"data_stored":135325001,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":159858688,"data_stored":232208187,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:30:09.492 INFO:tasks.thrashosds.thrasher:Setting 7.3 to [1, 6, 2] 2021-03-15T22:30:09.492 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '7.3', '1', '6', '2'] 2021-03-15T22:30:09.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pg-upmap 7.3 1 6 2 2021-03-15T22:30:09.546 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:09.568 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:09.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:09.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:09.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:09 smithi076 conmon[62156]: cluster 2021-03-15T22:30:07.891771+0000 osd.5 (osd.5) 9 : cluster [DBG] 3.c scrub ok 2021-03-15T22:30:09.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:09 smithi076 conmon[62156]: cluster 2021-03-15T22:30:08.279124+0000 osd.7 (osd.7) 131 : cluster [DBG] 3.d scrub ok 2021-03-15T22:30:09.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:09 smithi076 conmon[62156]: audit 2021-03-15T22:30:08.287655+0000 mgr.y (mgr.14144) 832 : audit [DBG] from='client.16074 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:30:09.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:09 smithi076 conmon[62156]: cluster 2021-03-15T22:30:08.392630+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e659: 8 total, 7 up, 8 in 2021-03-15T22:30:09.666 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:09 smithi076 conmon[56514]: cluster 2021-03-15T22:30:07.891771+0000 osd.5 (osd. 2021-03-15T22:30:09.666 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:09 smithi076 conmon[56514]: 5) 9 : cluster [DBG] 3.c scrub ok 2021-03-15T22:30:09.666 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:09 smithi076 conmon[56514]: cluster 2021-03-15T22:30:08.279124+0000 osd.7 (osd.7) 131 : cluster [DBG] 3.d scrub ok 2021-03-15T22:30:09.667 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:09 smithi076 conmon[56514]: audit 2021-03-15T22:30:08.287655+0000 mgr.y (mgr.14144) 832 : audit [DBG] from='client.16074 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:30:09.667 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:09 smithi076 conmon[56514]: cluster 2021-03-15T22:30:08.392630+0000 mon.a (mon.0) 1806 : cluster [DBG] osdmap e659: 8 total, 7 up, 8 in 2021-03-15T22:30:09.672 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:09.695 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:09.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:09.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:09.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:09 smithi125 conmon[56739]: cluster 2021-03-15T22:30:07.891771+0000 osd.5 (osd.5) 2021-03-15T22:30:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:09 smithi125 conmon[56739]: 9 : cluster [DBG] 3.c scrub ok 2021-03-15T22:30:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:09 smithi125 conmon[56739]: cluster 2021-03-15T22:30:08.279124+0000 osd.7 (osd.7 2021-03-15T22:30:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:09 smithi125 conmon[56739]: ) 131 : cluster [DBG] 3.d scrub ok 2021-03-15T22:30:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:09 smithi125 conmon[56739]: audit 2021-03-15T22:30:08.287655+0000 2021-03-15T22:30:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:09 smithi125 conmon[56739]: mgr.y (mgr.14144) 832 : audit [DBG] from='client.16074 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:30:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:09 smithi125 conmon[56739]: cluster 2021-03-15T22:30:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:09 smithi125 conmon[56739]: 2021-03-15T22:30:08.392630+0000 mon.a (mon.0) 1806 : cluster 2021-03-15T22:30:09.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:09 smithi125 conmon[56739]: [DBG] osdmap e659: 8 total, 7 up, 8 in 2021-03-15T22:30:09.797 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:09.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:09.821 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:09.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:09.922 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:09.945 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:09.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:09.947 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:10.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:10.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:10.074 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:10.074 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:10.176 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:10.202 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:10.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:10.203 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:10.211 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:30:10.212 INFO:tasks.rados.rados.0.smithi076.stdout:2410: done (0 left) 2021-03-15T22:30:10.212 INFO:tasks.rados.rados.0.smithi076.stdout:2411: write oid 17 current snap is 275 2021-03-15T22:30:10.213 INFO:tasks.rados.rados.0.smithi076.stdout:2411: seq_num 827 ranges {699389=797047,2111079=502953,3094462=581837} 2021-03-15T22:30:10.217 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:10.222 INFO:tasks.rados.rados.0.smithi076.stdout:2411: writing smithi07694878-17 from 699389 to 1496436 tid 1 2021-03-15T22:30:10.228 INFO:tasks.rados.rados.0.smithi076.stdout:2411: writing smithi07694878-17 from 2111079 to 2614032 tid 2 2021-03-15T22:30:10.235 INFO:tasks.rados.rados.0.smithi076.stdout:2411: writing smithi07694878-17 from 3094462 to 3676299 tid 3 2021-03-15T22:30:10.235 INFO:tasks.rados.rados.0.smithi076.stdout:2412: rollback oid 10 current snap is 275 2021-03-15T22:30:10.235 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 10 to 220 2021-03-15T22:30:10.236 INFO:tasks.rados.rados.0.smithi076.stdout:2411: finishing write tid 1 to smithi07694878-17 2021-03-15T22:30:10.236 INFO:tasks.rados.rados.0.smithi076.stdout:2411: finishing write tid 2 to smithi07694878-17 2021-03-15T22:30:10.236 INFO:tasks.rados.rados.0.smithi076.stdout:2413: read oid 35 snap -1 2021-03-15T22:30:10.236 INFO:tasks.rados.rados.0.smithi076.stdout:2413: expect (ObjNum 785 snap 261 seq_num 785) 2021-03-15T22:30:10.236 INFO:tasks.rados.rados.0.smithi076.stdout:2414: snap_remove snap 223 2021-03-15T22:30:10.286 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:30:09 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:30:10.305 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:30:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:30:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:30:10.305 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:30:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1191 7.6 7:61e0cc05:::smithi07694878-16:head [copy-from ver 101] snapc 4e=[4e,4d,3f] ondisk+write+known_if_redirected e175)", 2021-03-15T22:30:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:30:10.306 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 520.22523247300001, 2021-03-15T22:30:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.97834980800000004, 2021-03-15T22:30:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:10.307 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1191 2021-03-15T22:30:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.308 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:30:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067740+0000", 2021-03-15T22:30:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0440000000000003e-06 2021-03-15T22:30:10.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067745+0000", 2021-03-15T22:30:10.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4340000000000001e-06 2021-03-15T22:30:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067749+0000", 2021-03-15T22:30:10.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9241000000000001e-05 2021-03-15T22:30:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067768+0000", 2021-03-15T22:30:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3010000000000007e-06 2021-03-15T22:30:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067776+0000", 2021-03-15T22:30:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5014e-05 2021-03-15T22:30:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067821+0000", 2021-03-15T22:30:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9779999999999994e-06 2021-03-15T22:30:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067830+0000", 2021-03-15T22:30:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.95797833899999996 2021-03-15T22:30:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.025809+0000", 2021-03-15T22:30:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138412 2021-03-15T22:30:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.025947+0000", 2021-03-15T22:30:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.006355744 2021-03-15T22:30:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,7", 2021-03-15T22:30:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.032303+0000", 2021-03-15T22:30:10.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043109439999999997 2021-03-15T22:30:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:10.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.036614+0000", 2021-03-15T22:30:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0092878839999999997 2021-03-15T22:30:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.045902+0000", 2021-03-15T22:30:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8556999999999999e-05 2021-03-15T22:30:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.045990+0000", 2021-03-15T22:30:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6665999999999997e-05 2021-03-15T22:30:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046037+0000", 2021-03-15T22:30:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.325e-05 2021-03-15T22:30:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046090+0000", 2021-03-15T22:30:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.97834980800000004 2021-03-15T22:30:10.327 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:1192 7.6 7:61e0cc05:::smithi07694878-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e175)", 2021-03-15T22:30:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:30:10.329 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 520.22518315399998, 2021-03-15T22:30:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.978448604, 2021-03-15T22:30:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.330 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.331 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.331 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 1192 2021-03-15T22:30:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:30:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067790+0000", 2021-03-15T22:30:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.401e-06 2021-03-15T22:30:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067791+0000", 2021-03-15T22:30:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.083e-06 2021-03-15T22:30:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067794+0000", 2021-03-15T22:30:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4919999999999996e-06 2021-03-15T22:30:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067802+0000", 2021-03-15T22:30:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.383e-06 2021-03-15T22:30:10.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067807+0000", 2021-03-15T22:30:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8674000000000002e-05 2021-03-15T22:30:10.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067856+0000", 2021-03-15T22:30:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5890000000000003e-06 2021-03-15T22:30:10.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:30.067864+0000", 2021-03-15T22:30:10.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.95935619400000005 2021-03-15T22:30:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.027221+0000", 2021-03-15T22:30:10.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9426999999999998e-05 2021-03-15T22:30:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:30:10.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.027290+0000", 2021-03-15T22:30:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085250369999999992 2021-03-15T22:30:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.035815+0000", 2021-03-15T22:30:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9997999999999997e-05 2021-03-15T22:30:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:10.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.035875+0000", 2021-03-15T22:30:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010246089 2021-03-15T22:30:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046121+0000", 2021-03-15T22:30:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3338999999999998e-05 2021-03-15T22:30:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046184+0000", 2021-03-15T22:30:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3897999999999998e-05 2021-03-15T22:30:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:31.046238+0000", 2021-03-15T22:30:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.978448604 2021-03-15T22:30:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.348 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:548 7.6 e204/175 ondisk, result = 0)", 2021-03-15T22:30:10.349 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:30:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 490.21145024399999, 2021-03-15T22:30:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045793304999999999, 2021-03-15T22:30:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.350 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:30:10.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081523+0000", 2021-03-15T22:30:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6390000000000001e-06 2021-03-15T22:30:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081524+0000", 2021-03-15T22:30:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.863e-06 2021-03-15T22:30:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081526+0000", 2021-03-15T22:30:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1234e-05 2021-03-15T22:30:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081537+0000", 2021-03-15T22:30:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5210000000000002e-06 2021-03-15T22:30:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.081543+0000", 2021-03-15T22:30:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045750591 2021-03-15T22:30:10.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127293+0000", 2021-03-15T22:30:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6920000000000008e-06 2021-03-15T22:30:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127302+0000", 2021-03-15T22:30:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3764999999999999e-05 2021-03-15T22:30:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127316+0000", 2021-03-15T22:30:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045793304999999999 2021-03-15T22:30:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.361 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.0.0:548 7.6 e205/175 ondisk, result = 0)", 2021-03-15T22:30:10.362 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:30:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 490.20712676199997, 2021-03-15T22:30:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041608198999999998, 2021-03-15T22:30:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:30:10.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085846+0000", 2021-03-15T22:30:10.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4779999999999998e-06 2021-03-15T22:30:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085849+0000", 2021-03-15T22:30:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.046e-06 2021-03-15T22:30:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085851+0000", 2021-03-15T22:30:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4150000000000005e-06 2021-03-15T22:30:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085860+0000", 2021-03-15T22:30:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0900000000000001e-06 2021-03-15T22:30:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.085866+0000", 2021-03-15T22:30:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041511365000000001 2021-03-15T22:30:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127378+0000", 2021-03-15T22:30:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6030000000000001e-06 2021-03-15T22:30:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127384+0000", 2021-03-15T22:30:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0202000000000003e-05 2021-03-15T22:30:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.127454+0000", 2021-03-15T22:30:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041608198999999998 2021-03-15T22:30:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.373 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:61 7.5 e205/91 7:a9545d62:::smithi07694878-1:68 v 205'72, mlcod=202'69)", 2021-03-15T22:30:10.374 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:30:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 490.20210301600002, 2021-03-15T22:30:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034962251999999999, 2021-03-15T22:30:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.375 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:30:10.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090870+0000", 2021-03-15T22:30:10.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.798e-06 2021-03-15T22:30:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090877+0000", 2021-03-15T22:30:10.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3779999999999999e-06 2021-03-15T22:30:10.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090879+0000", 2021-03-15T22:30:10.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1902999999999999e-05 2021-03-15T22:30:10.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090891+0000", 2021-03-15T22:30:10.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7780000000000004e-06 2021-03-15T22:30:10.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090897+0000", 2021-03-15T22:30:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0942000000000002e-05 2021-03-15T22:30:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090938+0000", 2021-03-15T22:30:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6379e-05 2021-03-15T22:30:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.090954+0000", 2021-03-15T22:30:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034836759000000002 2021-03-15T22:30:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125791+0000", 2021-03-15T22:30:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1315e-05 2021-03-15T22:30:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.125832+0000", 2021-03-15T22:30:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034962251999999999 2021-03-15T22:30:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.386 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.387 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.388 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.6.0:214 7.d e205/91 7:bd183e1b:::smithi07694878-46:68 v 205'30, mlcod=202'27)", 2021-03-15T22:30:10.388 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:30:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 490.20166113699997, 2021-03-15T22:30:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034834256000000001, 2021-03-15T22:30:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.389 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:30:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091312+0000", 2021-03-15T22:30:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3590000000000002e-06 2021-03-15T22:30:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091314+0000", 2021-03-15T22:30:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.57e-06 2021-03-15T22:30:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091316+0000", 2021-03-15T22:30:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3749999999999999e-06 2021-03-15T22:30:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091322+0000", 2021-03-15T22:30:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2140000000000004e-06 2021-03-15T22:30:10.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091327+0000", 2021-03-15T22:30:10.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017519700000000001 2021-03-15T22:30:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091502+0000", 2021-03-15T22:30:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2955e-05 2021-03-15T22:30:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.091525+0000", 2021-03-15T22:30:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034582099999999998 2021-03-15T22:30:10.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.126108+0000", 2021-03-15T22:30:10.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8485999999999997e-05 2021-03-15T22:30:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:00.126146+0000", 2021-03-15T22:30:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034834256000000001 2021-03-15T22:30:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2103 7.f 7:fee9f235:::smithi07694878-11:head [copy-from ver 398] snapc 9a=[9a,97,93,92,8d,88,80,7d] ondisk+write+known_if_redirected e253)", 2021-03-15T22:30:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:30:10.403 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 442.08137815499998, 2021-03-15T22:30:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029938436999999998, 2021-03-15T22:30:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.404 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2103 2021-03-15T22:30:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.405 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:30:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211595+0000", 2021-03-15T22:30:10.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1350000000000002e-06 2021-03-15T22:30:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211599+0000", 2021-03-15T22:30:10.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1489999999999998e-06 2021-03-15T22:30:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211602+0000", 2021-03-15T22:30:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8119000000000001e-05 2021-03-15T22:30:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211620+0000", 2021-03-15T22:30:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9529999999999999e-06 2021-03-15T22:30:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211628+0000", 2021-03-15T22:30:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015272800000000001 2021-03-15T22:30:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211781+0000", 2021-03-15T22:30:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0270000000000001e-06 2021-03-15T22:30:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:30:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211790+0000", 2021-03-15T22:30:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035614209999999999 2021-03-15T22:30:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215351+0000", 2021-03-15T22:30:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8361999999999994e-05 2021-03-15T22:30:10.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215420+0000", 2021-03-15T22:30:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013426624999999999 2021-03-15T22:30:10.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:30:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.228846+0000", 2021-03-15T22:30:10.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041102990000000004 2021-03-15T22:30:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232957+0000", 2021-03-15T22:30:10.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013870029999999999 2021-03-15T22:30:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.234344+0000", 2021-03-15T22:30:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.007045791 2021-03-15T22:30:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241389+0000", 2021-03-15T22:30:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0001999999999998e-05 2021-03-15T22:30:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241459+0000", 2021-03-15T22:30:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3823000000000001e-05 2021-03-15T22:30:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241533+0000", 2021-03-15T22:30:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029938436999999998 2021-03-15T22:30:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.423 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2104 7.f 7:fee9f235:::smithi07694878-11:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e253)", 2021-03-15T22:30:10.424 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:30:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 442.08123641899999, 2021-03-15T22:30:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029977025000000001, 2021-03-15T22:30:10.425 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.426 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2104 2021-03-15T22:30:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:30:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211736+0000", 2021-03-15T22:30:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.491e-06 2021-03-15T22:30:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211738+0000", 2021-03-15T22:30:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.813e-06 2021-03-15T22:30:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211740+0000", 2021-03-15T22:30:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2709999999999992e-06 2021-03-15T22:30:10.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211748+0000", 2021-03-15T22:30:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8559999999999998e-06 2021-03-15T22:30:10.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211753+0000", 2021-03-15T22:30:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014630000000000001 2021-03-15T22:30:10.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211899+0000", 2021-03-15T22:30:10.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7549999999999997e-06 2021-03-15T22:30:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:30:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.211906+0000", 2021-03-15T22:30:10.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036139900000000001 2021-03-15T22:30:10.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215520+0000", 2021-03-15T22:30:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7696999999999999e-05 2021-03-15T22:30:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:30:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.215548+0000", 2021-03-15T22:30:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016795027000000001 2021-03-15T22:30:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232343+0000", 2021-03-15T22:30:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1192000000000005e-05 2021-03-15T22:30:10.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.232414+0000", 2021-03-15T22:30:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0091605279999999994 2021-03-15T22:30:10.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241574+0000", 2021-03-15T22:30:10.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5976000000000007e-05 2021-03-15T22:30:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241640+0000", 2021-03-15T22:30:10.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3128999999999997e-05 2021-03-15T22:30:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:48.241713+0000", 2021-03-15T22:30:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029977025000000001 2021-03-15T22:30:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2221 7.f 7:ffef77ba:::smithi07694878-13:head [sparse-read 0~2702901,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:30:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:30:10.445 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 432.24198480899997, 2021-03-15T22:30:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0065843969999999, 2021-03-15T22:30:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.446 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2221 2021-03-15T22:30:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.447 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:30:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050988+0000", 2021-03-15T22:30:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0599999999999999e-06 2021-03-15T22:30:10.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050991+0000", 2021-03-15T22:30:10.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0379999999999998e-06 2021-03-15T22:30:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.050993+0000", 2021-03-15T22:30:10.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6500000000000002e-06 2021-03-15T22:30:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051002+0000", 2021-03-15T22:30:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3650000000000003e-06 2021-03-15T22:30:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051007+0000", 2021-03-15T22:30:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015642 2021-03-15T22:30:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051164+0000", 2021-03-15T22:30:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7530000000000007e-06 2021-03-15T22:30:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051172+0000", 2021-03-15T22:30:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.004477362 2021-03-15T22:30:10.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.055650+0000", 2021-03-15T22:30:10.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134535 2021-03-15T22:30:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.055784+0000", 2021-03-15T22:30:10.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001788214 2021-03-15T22:30:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.057572+0000", 2021-03-15T22:30:10.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0065843969999999 2021-03-15T22:30:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2222 7.f 7:ffef77ba:::smithi07694878-13:head [read 0~2823609,checksum] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:30:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:30:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 432.24196018599997, 2021-03-15T22:30:10.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0081352180000001, 2021-03-15T22:30:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.462 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2222 2021-03-15T22:30:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:30:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051013+0000", 2021-03-15T22:30:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6190000000000001e-06 2021-03-15T22:30:10.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051014+0000", 2021-03-15T22:30:10.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.217e-06 2021-03-15T22:30:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051016+0000", 2021-03-15T22:30:10.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8270000000000003e-06 2021-03-15T22:30:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051021+0000", 2021-03-15T22:30:10.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4529999999999998e-06 2021-03-15T22:30:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051025+0000", 2021-03-15T22:30:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000168445 2021-03-15T22:30:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051193+0000", 2021-03-15T22:30:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5860000000000001e-06 2021-03-15T22:30:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051200+0000", 2021-03-15T22:30:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0070519229999999 2021-03-15T22:30:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.058252+0000", 2021-03-15T22:30:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1403000000000006e-05 2021-03-15T22:30:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.058333+0000", 2021-03-15T22:30:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00081474499999999997 2021-03-15T22:30:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.059148+0000", 2021-03-15T22:30:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0081352180000001 2021-03-15T22:30:10.475 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.476 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2223 7.f 7:ffef77ba:::smithi07694878-13:head [sparse-read 0~2702901] snapc 0=[] ondisk+read+known_if_redirected e263)", 2021-03-15T22:30:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:30:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 432.24194426299999, 2021-03-15T22:30:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.009479572, 2021-03-15T22:30:10.477 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.478 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2223 2021-03-15T22:30:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:30:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:30:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7300000000000005e-07 2021-03-15T22:30:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051029+0000", 2021-03-15T22:30:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.02e-06 2021-03-15T22:30:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051030+0000", 2021-03-15T22:30:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9280000000000001e-06 2021-03-15T22:30:10.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051035+0000", 2021-03-15T22:30:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.129e-06 2021-03-15T22:30:10.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051039+0000", 2021-03-15T22:30:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018268000000000001 2021-03-15T22:30:10.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051221+0000", 2021-03-15T22:30:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4270000000000002e-06 2021-03-15T22:30:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:58.051229+0000", 2021-03-15T22:30:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.007961946 2021-03-15T22:30:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.059191+0000", 2021-03-15T22:30:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010123370000000001 2021-03-15T22:30:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.060203+0000", 2021-03-15T22:30:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030523200000000001 2021-03-15T22:30:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:22:59.060508+0000", 2021-03-15T22:30:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.009479572 2021-03-15T22:30:10.491 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.492 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3021 7.6 7:6de8b070:::smithi07694878-15:head [copy-from ver 640] snapc e3=[e1,df,de,dd,db] ondisk+write+known_if_redirected e330)", 2021-03-15T22:30:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:30:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 365.21035442599998, 2021-03-15T22:30:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.124784161, 2021-03-15T22:30:10.493 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.494 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3021 2021-03-15T22:30:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:30:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:30:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7749999999999999e-06 2021-03-15T22:30:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082622+0000", 2021-03-15T22:30:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.072e-06 2021-03-15T22:30:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082625+0000", 2021-03-15T22:30:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2641e-05 2021-03-15T22:30:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082638+0000", 2021-03-15T22:30:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9170000000000006e-06 2021-03-15T22:30:10.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082646+0000", 2021-03-15T22:30:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1200999999999997e-05 2021-03-15T22:30:10.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082727+0000", 2021-03-15T22:30:10.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7440000000000007e-06 2021-03-15T22:30:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:30:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082735+0000", 2021-03-15T22:30:10.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022422739999999998 2021-03-15T22:30:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.084977+0000", 2021-03-15T22:30:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010620000000000001 2021-03-15T22:30:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085083+0000", 2021-03-15T22:30:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10953049500000001 2021-03-15T22:30:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,7", 2021-03-15T22:30:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.194614+0000", 2021-03-15T22:30:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043572510000000004 2021-03-15T22:30:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198971+0000", 2021-03-15T22:30:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002724242 2021-03-15T22:30:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.201695+0000", 2021-03-15T22:30:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054890260000000001 2021-03-15T22:30:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207184+0000", 2021-03-15T22:30:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4066000000000006e-05 2021-03-15T22:30:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207248+0000", 2021-03-15T22:30:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015425700000000001 2021-03-15T22:30:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207403+0000", 2021-03-15T22:30:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.124784161 2021-03-15T22:30:10.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.513 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3022 7.6 7:6de8b070:::smithi07694878-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e330)", 2021-03-15T22:30:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:30:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 365.21031405899998, 2021-03-15T22:30:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12494472, 2021-03-15T22:30:10.514 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.515 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3022 2021-03-15T22:30:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:30:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:30:10.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2669999999999999e-06 2021-03-15T22:30:10.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082660+0000", 2021-03-15T22:30:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7409999999999999e-06 2021-03-15T22:30:10.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082662+0000", 2021-03-15T22:30:10.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2030000000000003e-06 2021-03-15T22:30:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082669+0000", 2021-03-15T22:30:10.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1479999999999999e-06 2021-03-15T22:30:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082673+0000", 2021-03-15T22:30:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037115800000000001 2021-03-15T22:30:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.083044+0000", 2021-03-15T22:30:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1779999999999996e-06 2021-03-15T22:30:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:30:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.083049+0000", 2021-03-15T22:30:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021797460000000002 2021-03-15T22:30:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.525 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085228+0000", 2021-03-15T22:30:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5878999999999999e-05 2021-03-15T22:30:10.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:30:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085274+0000", 2021-03-15T22:30:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.112992462 2021-03-15T22:30:10.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198267+0000", 2021-03-15T22:30:10.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6129000000000002e-05 2021-03-15T22:30:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198343+0000", 2021-03-15T22:30:10.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090944800000000003 2021-03-15T22:30:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207437+0000", 2021-03-15T22:30:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1060000000000001e-05 2021-03-15T22:30:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207508+0000", 2021-03-15T22:30:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5268999999999999e-05 2021-03-15T22:30:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207604+0000", 2021-03-15T22:30:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12494472 2021-03-15T22:30:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.533 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:679 7.5 e560/559 7:ac84e656:::smithi07694878-31:1a1 v 560'396, mlcod=560'392)", 2021-03-15T22:30:10.534 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:28:20.276500+0000", 2021-03-15T22:30:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 110.016473033, 2021-03-15T22:30:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051465865, 2021-03-15T22:30:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:10.535 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276500+0000", 2021-03-15T22:30:10.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276500+0000", 2021-03-15T22:30:10.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5639999999999996e-06 2021-03-15T22:30:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276505+0000", 2021-03-15T22:30:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7400000000000002e-06 2021-03-15T22:30:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276509+0000", 2021-03-15T22:30:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4290999999999999e-05 2021-03-15T22:30:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276523+0000", 2021-03-15T22:30:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2339999999999997e-06 2021-03-15T22:30:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276531+0000", 2021-03-15T22:30:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0612999999999999e-05 2021-03-15T22:30:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276571+0000", 2021-03-15T22:30:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4697999999999999e-05 2021-03-15T22:30:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276606+0000", 2021-03-15T22:30:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051301986000000001 2021-03-15T22:30:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.327908+0000", 2021-03-15T22:30:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7738999999999997e-05 2021-03-15T22:30:10.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.327966+0000", 2021-03-15T22:30:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051465865 2021-03-15T22:30:10.546 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5431 7.f 7:ffef77ba:::smithi07694878-13:head [write 470794~720999] snapc 1c8=[1c7,1c5,1c4,1c3,1be,1bd,1bb,1b7,1b6,1b3,1ae,1a6,1a4,1a2,1a1,19f,19c,19b,195,194,193,18a,180,17a,176,16e,161,15f,141,123] ondisk+write+known_if_redirected e593)", 2021-03-15T22:30:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:02.377350+0000", 2021-03-15T22:30:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.915623361000002, 2021-03-15T22:30:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.031983472999999998, 2021-03-15T22:30:10.548 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.549 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5431 2021-03-15T22:30:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.377350+0000", 2021-03-15T22:30:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.377350+0000", 2021-03-15T22:30:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000173096 2021-03-15T22:30:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.377523+0000", 2021-03-15T22:30:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7239999999999998e-06 2021-03-15T22:30:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.377526+0000", 2021-03-15T22:30:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6858e-05 2021-03-15T22:30:10.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.377543+0000", 2021-03-15T22:30:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2520000000000002e-06 2021-03-15T22:30:10.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.377550+0000", 2021-03-15T22:30:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001252948 2021-03-15T22:30:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.378803+0000", 2021-03-15T22:30:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012431700000000001 2021-03-15T22:30:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.378928+0000", 2021-03-15T22:30:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026047899999999998 2021-03-15T22:30:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:30:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.379188+0000", 2021-03-15T22:30:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0087002629999999997 2021-03-15T22:30:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.387888+0000", 2021-03-15T22:30:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3208999999999997e-05 2021-03-15T22:30:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.387962+0000", 2021-03-15T22:30:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.02124295 2021-03-15T22:30:10.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.409205+0000", 2021-03-15T22:30:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4503000000000002e-05 2021-03-15T22:30:10.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.409269+0000", 2021-03-15T22:30:10.564 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3874000000000005e-05 2021-03-15T22:30:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.409333+0000", 2021-03-15T22:30:10.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.031983472999999998 2021-03-15T22:30:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5432 7.f 7:ffef77ba:::smithi07694878-13:head [write 1903639~554256] snapc 1c8=[1c7,1c5,1c4,1c3,1be,1bd,1bb,1b7,1b6,1b3,1ae,1a6,1a4,1a2,1a1,19f,19c,19b,195,194,193,18a,180,17a,176,16e,161,15f,141,123] ondisk+write+known_if_redirected e593)", 2021-03-15T22:30:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:02.384667+0000", 2021-03-15T22:30:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.908306144999997, 2021-03-15T22:30:10.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029570649000000001, 2021-03-15T22:30:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.568 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5432 2021-03-15T22:30:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384667+0000", 2021-03-15T22:30:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384667+0000", 2021-03-15T22:30:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000133697 2021-03-15T22:30:10.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384800+0000", 2021-03-15T22:30:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3639999999999999e-06 2021-03-15T22:30:10.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384804+0000", 2021-03-15T22:30:10.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3346000000000001e-05 2021-03-15T22:30:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384817+0000", 2021-03-15T22:30:10.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2300000000000008e-06 2021-03-15T22:30:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384825+0000", 2021-03-15T22:30:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1311999999999999e-05 2021-03-15T22:30:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.576 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384877+0000", 2021-03-15T22:30:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108891 2021-03-15T22:30:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384986+0000", 2021-03-15T22:30:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021888499999999999 2021-03-15T22:30:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.578 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:30:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.385204+0000", 2021-03-15T22:30:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027046190000000001 2021-03-15T22:30:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.387909+0000", 2021-03-15T22:30:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00091387700000000001 2021-03-15T22:30:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.388823+0000", 2021-03-15T22:30:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.025298417 2021-03-15T22:30:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.414121+0000", 2021-03-15T22:30:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9548e-05 2021-03-15T22:30:10.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.414181+0000", 2021-03-15T22:30:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6462999999999998e-05 2021-03-15T22:30:10.583 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.414237+0000", 2021-03-15T22:30:10.584 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029570649000000001 2021-03-15T22:30:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.585 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5433 7.f 7:ffef77ba:::smithi07694878-13:head [setxattr _header (58) in=7b,truncate 2470794] snapc 1c8=[1c7,1c5,1c4,1c3,1be,1bd,1bb,1b7,1b6,1b3,1ae,1a6,1a4,1a2,1a1,19f,19c,19b,195,194,193,18a,180,17a,176,16e,161,15f,141,123] ondisk+write+known_if_redirected e593)", 2021-03-15T22:30:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:02.384838+0000", 2021-03-15T22:30:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.908134442999994, 2021-03-15T22:30:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033394885999999999, 2021-03-15T22:30:10.586 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.587 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5433 2021-03-15T22:30:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384838+0000", 2021-03-15T22:30:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384838+0000", 2021-03-15T22:30:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0329999999999998e-06 2021-03-15T22:30:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384840+0000", 2021-03-15T22:30:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5990000000000001e-06 2021-03-15T22:30:10.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384842+0000", 2021-03-15T22:30:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7360000000000004e-06 2021-03-15T22:30:10.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384849+0000", 2021-03-15T22:30:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1150000000000002e-06 2021-03-15T22:30:10.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384854+0000", 2021-03-15T22:30:10.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0011906690000000001 2021-03-15T22:30:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.386045+0000", 2021-03-15T22:30:10.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010584100000000001 2021-03-15T22:30:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.386150+0000", 2021-03-15T22:30:10.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000177861 2021-03-15T22:30:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:30:10.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.386328+0000", 2021-03-15T22:30:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015859820000000001 2021-03-15T22:30:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:10.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.387914+0000", 2021-03-15T22:30:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00095111299999999996 2021-03-15T22:30:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.599 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.388865+0000", 2021-03-15T22:30:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.02924657 2021-03-15T22:30:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.418112+0000", 2021-03-15T22:30:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5361000000000001e-05 2021-03-15T22:30:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.418167+0000", 2021-03-15T22:30:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6006000000000005e-05 2021-03-15T22:30:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.418233+0000", 2021-03-15T22:30:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033394885999999999 2021-03-15T22:30:10.603 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.604 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5434 7.f 7:ffef77ba:::smithi07694878-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e593)", 2021-03-15T22:30:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:02.384871+0000", 2021-03-15T22:30:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 67.908101501000004, 2021-03-15T22:30:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033608708000000001, 2021-03-15T22:30:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.605 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:10.606 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5434 2021-03-15T22:30:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384871+0000", 2021-03-15T22:30:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384871+0000", 2021-03-15T22:30:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3179999999999999e-06 2021-03-15T22:30:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384873+0000", 2021-03-15T22:30:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3799999999999999e-06 2021-03-15T22:30:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384874+0000", 2021-03-15T22:30:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7569999999999998e-06 2021-03-15T22:30:10.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384881+0000", 2021-03-15T22:30:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5460000000000002e-06 2021-03-15T22:30:10.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.384885+0000", 2021-03-15T22:30:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001856482 2021-03-15T22:30:10.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.386742+0000", 2021-03-15T22:30:10.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021583999999999999 2021-03-15T22:30:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.386958+0000", 2021-03-15T22:30:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.031320977 2021-03-15T22:30:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.616 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.418279+0000", 2021-03-15T22:30:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4709000000000005e-05 2021-03-15T22:30:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.418363+0000", 2021-03-15T22:30:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000116699 2021-03-15T22:30:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:02.418480+0000", 2021-03-15T22:30:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033608708000000001 2021-03-15T22:30:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.619 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:5830 7.5 e631/559 7:ac84e656:::smithi07694878-31:head v 631'458, mlcod=630'454)", 2021-03-15T22:30:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:40.251563+0000", 2021-03-15T22:30:10.620 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 30.041409457, 2021-03-15T22:30:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071683063000000005, 2021-03-15T22:30:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:10.621 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251563+0000", 2021-03-15T22:30:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251563+0000", 2021-03-15T22:30:10.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052985690000000004 2021-03-15T22:30:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256862+0000", 2021-03-15T22:30:10.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2240000000000001e-06 2021-03-15T22:30:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256871+0000", 2021-03-15T22:30:10.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3723000000000002e-05 2021-03-15T22:30:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256905+0000", 2021-03-15T22:30:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2787e-05 2021-03-15T22:30:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.627 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256918+0000", 2021-03-15T22:30:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6076000000000006e-05 2021-03-15T22:30:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.628 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257014+0000", 2021-03-15T22:30:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2570999999999997e-05 2021-03-15T22:30:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257046+0000", 2021-03-15T22:30:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066154983000000001 2021-03-15T22:30:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.323201+0000", 2021-03-15T22:30:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5130000000000003e-05 2021-03-15T22:30:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.323246+0000", 2021-03-15T22:30:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071683063000000005 2021-03-15T22:30:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.632 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.633 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:755 7.5 e646/559 7:a9545d62:::smithi07694878-1:1fc v 646'483, mlcod=644'479)", 2021-03-15T22:30:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:55.246251+0000", 2021-03-15T22:30:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.046721476, 2021-03-15T22:30:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088387691000000004, 2021-03-15T22:30:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:10.634 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246251+0000", 2021-03-15T22:30:10.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:10.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246251+0000", 2021-03-15T22:30:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5550000000000001e-06 2021-03-15T22:30:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:10.637 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246254+0000", 2021-03-15T22:30:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3700000000000002e-06 2021-03-15T22:30:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.638 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246256+0000", 2021-03-15T22:30:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6309999999999998e-06 2021-03-15T22:30:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.639 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246266+0000", 2021-03-15T22:30:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6929999999999998e-06 2021-03-15T22:30:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246271+0000", 2021-03-15T22:30:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5197000000000001e-05 2021-03-15T22:30:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246316+0000", 2021-03-15T22:30:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3624e-05 2021-03-15T22:30:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246369+0000", 2021-03-15T22:30:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088224576999999998 2021-03-15T22:30:10.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334594+0000", 2021-03-15T22:30:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5043999999999998e-05 2021-03-15T22:30:10.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334639+0000", 2021-03-15T22:30:10.645 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088387691000000004 2021-03-15T22:30:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:10.646 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:10.647 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:30:10.647 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:10.670 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:10.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:10.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:10.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[56514]: cluster 2021-03-15T22:30:09.101163+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:30:10.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[56514]: cluster 2021-03-15T22:30:09.139569+0000 mgr.y (mgr.14144) 833 : cluster [DBG] pgmap v1133: 138 pgs: 1 active+clean+snaptrim, 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 92 active+clean; 482 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 1.6 MiB/s wr, 11 op/s; 56/1653 objects degraded (3.388%); 120/1653 objects misplaced (7.260%) 2021-03-15T22:30:10.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[56514]: cluster 2021-03-15T22:30:09.397617+0000 mon.a (mon.0) 1807 : cluster [DBG] osdmap e660: 8 total, 7 up, 8 in 2021-03-15T22:30:10.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[56514]: cluster 2021-03-15T22:30:09.506910+0000 osd.3 (osd.3) 124 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:30:10.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[56514]: audit 2021-03-15T22:30:09.978306+0000 mgr.y (mgr.14144) 834 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:10.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[56514]: audit 2021-03-15T22:30:10.163714+0000 mon.a (mon.0) 1808 : audit [INF] from='client.? 172.21.15.76:0/1238407287' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:30:10.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[56514]: cluster 2021-03-15T22:30:10.198370+0000 mon.a (mon.0) 1809 : cluster [WRN] Health check update: Degraded data redundancy: 56/1653 objects degraded (3.388%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:10.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[56514]: cluster 2021-03-15T22:30:10.198962+0000 mon.a (mon.0) 1810 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:10.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[56514]: audit 2021-03-15T22:30:10.203119+0000 mon.a (mon.0) 1811 : audit [INF] from='client.? 172.21.15.76:0/1238407287' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:30:10.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[56514]: cluster 2021-03-15T22:30:10.203191+0000 2021-03-15T22:30:10.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[56514]: mon.a (mon.0) 1812 : cluster [DBG] osdmap e661: 8 total, 7 up, 8 in 2021-03-15T22:30:10.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: cluster 2021-03-15T22:30:09.101163+0000 osd.4 (osd.4) 144 : 2021-03-15T22:30:10.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: cluster [DBG] 6.2 scrub ok 2021-03-15T22:30:10.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: cluster 2021-03-15T22:30:09.139569+0000 mgr.y ( 2021-03-15T22:30:10.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: mgr.14144) 833 : cluster [DBG] pgmap v1133: 138 pgs: 1 active+clean+snaptrim, 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 92 active+clean; 482 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 1.6 MiB/s wr, 11 op/s; 56/1653 objects degraded (3.388%); 120/1653 objects misplaced (7.260%) 2021-03-15T22:30:10.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: cluster 2021-03-15T22:30:10.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: 2021-03-15T22:30:09.397617+0000 mon.a (mon.0) 1807 : cluster 2021-03-15T22:30:10.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: [DBG] osdmap e660: 8 total, 7 up, 8 in 2021-03-15T22:30:10.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:30:10.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: 30 2021-03-15T22:30:10.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: :09.506910+0000 osd.3 (osd 2021-03-15T22:30:10.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: .3) 124 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:30:10.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: audit 2021-03-15T22:30:10.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: 2021-03-15T22:30:09.978306+0000 mgr.y (mgr. 2021-03-15T22:30:10.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: 14144) 834 : audit 2021-03-15T22:30:10.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:10.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: audit 2021-03-15T22:30 2021-03-15T22:30:10.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: :10.163714+0000 mon.a (mon.0) 1808 : 2021-03-15T22:30:10.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/1238407287' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:30:10.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:30:10.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: 03-15T22:30:10.198370+0000 mon.a (mon 2021-03-15T22:30:10.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: .0) 1809 : cluster [WRN] Health check update: Degraded data redundancy: 56/1653 objects degraded (3.388%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:10.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:30:10.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: -03-15T22:30:10.198962 2021-03-15T22:30:10.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2021-03-15T22:30:10.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: 1810 : cluster [WRN] 2021-03-15T22:30:10.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:10.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:30:10.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: 30:10.203119+0000 mon.a (mon.0) 2021-03-15T22:30:10.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: 1811 : audit [INF] from='client.? 172.21.15.76:0/1238407287' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:30:10.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: cluster 2021-03-15T22:30:10.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: 2021-03 2021-03-15T22:30:10.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: -15T22:30:10.203191+0000 mon.a 2021-03-15T22:30:10.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:10 smithi076 conmon[62156]: (mon.0) 1812 : cluster [DBG] osdmap e661: 8 total, 7 up, 8 in 2021-03-15T22:30:10.782 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:10.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: cluster 2021-03-15T22:30:09.101163+0000 osd.4 (osd.4) 144 : cluster [DBG] 6.2 scrub ok 2021-03-15T22:30:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: cluster 2021-03-15T22:30:09.139569+0000 mgr.y (mgr.14144) 833 : cluster [DBG] pgmap v1133: 138 pgs: 1 active+clean+snaptrim, 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 92 active+clean; 482 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 9.8 MiB/s rd, 1.6 MiB/s wr, 11 op/s; 56/1653 objects degraded (3.388%); 120/1653 objects misplaced (7.260%) 2021-03-15T22:30:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:30:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: T22:30:09.397617+0000 mon.a (mon.0) 1807 : cluster [DBG] 2021-03-15T22:30:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: osdmap e660: 8 total, 7 up, 8 in 2021-03-15T22:30:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: cluster 2021-03-15T22:30:09. 2021-03-15T22:30:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: 506910+0000 osd.3 (osd.3) 124 : cluster 2021-03-15T22:30:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: [DBG] 3.8 scrub ok 2021-03-15T22:30:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:30:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: T22:30:09.978306+0000 mgr.y ( 2021-03-15T22:30:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: mgr.14144) 834 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: 2021-03-15T22:30:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:30:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: 03-15T22:30:10.163714+0000 mon.a (mon. 2021-03-15T22:30:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: 0) 1808 : audit [INF] from='client.? 172.21.15.76:0/1238407287' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:30:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: cluster 2021-03-15T22:30:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: 2021-03-15T22:30:10. 2021-03-15T22:30:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: 198370+0000 mon.a (mon.0) 1809 : 2021-03-15T22:30:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: cluster [WRN] Health check update: Degraded data redundancy: 56/1653 objects degraded (3.388%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:30:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: 15T22:30:10.198962+0000 mon.a 2021-03-15T22:30:10.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: (mon.0) 1810 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:10.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: 2021-03-15T22:30:10.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: audit 2021-03-15T22:30:10.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: 2021-03-15T22:30:10.203119 2021-03-15T22:30:10.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1811 : audit 2021-03-15T22:30:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/1238407287' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:30:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:30:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: :30:10.203191+0000 mon.a (mon 2021-03-15T22:30:10.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:10 smithi125 conmon[56739]: .0) 1812 : cluster [DBG] osdmap e661: 8 total, 7 up, 8 in 2021-03-15T22:30:10.804 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:10.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:10.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:10.908 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:10.931 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:10.932 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:11.033 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:11.056 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:11.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:11.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:11.181 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:11.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:11.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:11.220 INFO:tasks.rados.rados.0.smithi076.stdout:2414: done (3 left) 2021-03-15T22:30:11.220 INFO:tasks.rados.rados.0.smithi076.stdout:2415: write oid 6 current snap is 275 2021-03-15T22:30:11.220 INFO:tasks.rados.rados.0.smithi076.stdout:2415: seq_num 828 ranges {716196=472296,1769561=646611,3121885=793109} 2021-03-15T22:30:11.227 INFO:tasks.rados.rados.0.smithi076.stdout:2415: writing smithi07694878-6 from 716196 to 1188492 tid 1 2021-03-15T22:30:11.234 INFO:tasks.rados.rados.0.smithi076.stdout:2415: writing smithi07694878-6 from 1769561 to 2416172 tid 2 2021-03-15T22:30:11.245 INFO:tasks.rados.rados.0.smithi076.stdout:2415: writing smithi07694878-6 from 3121885 to 3914994 tid 3 2021-03-15T22:30:11.245 INFO:tasks.rados.rados.0.smithi076.stdout:2416: copy_from oid 2 from oid 45 current snap is 275 2021-03-15T22:30:11.245 INFO:tasks.rados.rados.0.smithi076.stdout:2417: read oid 13 snap -1 2021-03-15T22:30:11.245 INFO:tasks.rados.rados.0.smithi076.stdout:2417: expect (ObjNum 797 snap 262 seq_num 797) 2021-03-15T22:30:11.246 INFO:tasks.rados.rados.0.smithi076.stdout:2418: rollback oid 29 current snap is 275 2021-03-15T22:30:11.246 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 29 to 270 2021-03-15T22:30:11.246 INFO:tasks.rados.rados.0.smithi076.stdout:2419: write (excl) oid 39 current snap is 275 2021-03-15T22:30:11.246 INFO:tasks.rados.rados.0.smithi076.stdout:2419: seq_num 829 ranges {733003=663898,1944396=506622,2865661=604381} 2021-03-15T22:30:11.253 INFO:tasks.rados.rados.0.smithi076.stdout:2419: writing smithi07694878-39 from 733003 to 1396901 tid 1 2021-03-15T22:30:11.260 INFO:tasks.rados.rados.0.smithi076.stdout:2419: writing smithi07694878-39 from 1944396 to 2451018 tid 2 2021-03-15T22:30:11.267 INFO:tasks.rados.rados.0.smithi076.stdout:2419: writing smithi07694878-39 from 2865661 to 3470042 tid 3 2021-03-15T22:30:11.267 INFO:tasks.rados.rados.0.smithi076.stdout:2412: finishing rollback tid 1 to smithi07694878-10 2021-03-15T22:30:11.267 INFO:tasks.rados.rados.0.smithi076.stdout:2412: finishing rollback tid 2 to smithi07694878-10 2021-03-15T22:30:11.267 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 1359 (ObjNum 444 snap 125 seq_num 444) dirty exists 2021-03-15T22:30:11.268 INFO:tasks.rados.rados.0.smithi076.stdout:2413: expect (ObjNum 785 snap 261 seq_num 785) 2021-03-15T22:30:11.284 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:11.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:11.307 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:11.307 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:11.409 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:11.414 INFO:tasks.rados.rados.0.smithi076.stdout:2411: finishing write tid 3 to smithi07694878-17 2021-03-15T22:30:11.414 INFO:tasks.rados.rados.0.smithi076.stdout:2411: finishing write tid 4 to smithi07694878-17 2021-03-15T22:30:11.414 INFO:tasks.rados.rados.0.smithi076.stdout:2411: finishing write tid 5 to smithi07694878-17 2021-03-15T22:30:11.415 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 853 (ObjNum 827 snap 275 seq_num 827) dirty exists 2021-03-15T22:30:11.415 INFO:tasks.rados.rados.0.smithi076.stdout:2411: left oid 17 (ObjNum 827 snap 275 seq_num 827) 2021-03-15T22:30:11.415 INFO:tasks.rados.rados.0.smithi076.stdout:2415: finishing write tid 1 to smithi07694878-6 2021-03-15T22:30:11.415 INFO:tasks.rados.rados.0.smithi076.stdout:2415: finishing write tid 2 to smithi07694878-6 2021-03-15T22:30:11.416 INFO:tasks.rados.rados.0.smithi076.stdout:2411: done (7 left) 2021-03-15T22:30:11.416 INFO:tasks.rados.rados.0.smithi076.stdout:2412: done (6 left) 2021-03-15T22:30:11.416 INFO:tasks.rados.rados.0.smithi076.stdout:2413: done (5 left) 2021-03-15T22:30:11.417 INFO:tasks.rados.rados.0.smithi076.stdout:2420: read oid 17 snap -1 2021-03-15T22:30:11.417 INFO:tasks.rados.rados.0.smithi076.stdout:2420: expect (ObjNum 827 snap 275 seq_num 827) 2021-03-15T22:30:11.418 INFO:tasks.rados.rados.0.smithi076.stdout:2421: read oid 44 snap 250 2021-03-15T22:30:11.418 INFO:tasks.rados.rados.0.smithi076.stdout:2421: expect (ObjNum 747 snap 245 seq_num 747) 2021-03-15T22:30:11.418 INFO:tasks.rados.rados.0.smithi076.stdout:2422: write oid 14 current snap is 275 2021-03-15T22:30:11.419 INFO:tasks.rados.rados.0.smithi076.stdout:2417: expect (ObjNum 797 snap 262 seq_num 797) 2021-03-15T22:30:11.434 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:11.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:11.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:11.536 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:11.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:11.561 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:11.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:11.625 INFO:tasks.rados.rados.0.smithi076.stdout:2415: finishing write tid 3 to smithi07694878-6 2021-03-15T22:30:11.626 INFO:tasks.rados.rados.0.smithi076.stdout:2415: finishing write tid 4 to smithi07694878-6 2021-03-15T22:30:11.626 INFO:tasks.rados.rados.0.smithi076.stdout:2415: finishing write tid 5 to smithi07694878-6 2021-03-15T22:30:11.626 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 1601 (ObjNum 828 snap 275 seq_num 828) dirty exists 2021-03-15T22:30:11.627 INFO:tasks.rados.rados.0.smithi076.stdout:2415: left oid 6 (ObjNum 828 snap 275 seq_num 828) 2021-03-15T22:30:11.627 INFO:tasks.rados.rados.0.smithi076.stdout:2418: finishing rollback tid 0 to smithi07694878-29 2021-03-15T22:30:11.627 INFO:tasks.rados.rados.0.smithi076.stdout:2419: finishing write tid 1 to smithi07694878-39 2021-03-15T22:30:11.627 INFO:tasks.rados.rados.0.smithi076.stdout:2418: finishing rollback tid 1 to smithi07694878-29 2021-03-15T22:30:11.627 INFO:tasks.rados.rados.0.smithi076.stdout:2418: finishing rollback tid 2 to smithi07694878-29 2021-03-15T22:30:11.628 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 1914 (ObjNum 776 snap 258 seq_num 776) dirty exists 2021-03-15T22:30:11.628 INFO:tasks.rados.rados.0.smithi076.stdout:2419: finishing write tid 2 to smithi07694878-39 2021-03-15T22:30:11.628 INFO:tasks.rados.rados.0.smithi076.stdout:2422: seq_num 830 ranges {749810=739147,2002878=650280,3293084=415653} 2021-03-15T22:30:11.633 INFO:tasks.rados.rados.0.smithi076.stdout:2422: writing smithi07694878-14 from 749810 to 1488957 tid 1 2021-03-15T22:30:11.639 INFO:tasks.rados.rados.0.smithi076.stdout:2422: writing smithi07694878-14 from 2002878 to 2653158 tid 2 2021-03-15T22:30:11.647 INFO:tasks.rados.rados.0.smithi076.stdout:2422: writing smithi07694878-14 from 3293084 to 3708737 tid 3 2021-03-15T22:30:11.647 INFO:tasks.rados.rados.0.smithi076.stdout:2415: done (7 left) 2021-03-15T22:30:11.647 INFO:tasks.rados.rados.0.smithi076.stdout:2417: done (6 left) 2021-03-15T22:30:11.648 INFO:tasks.rados.rados.0.smithi076.stdout:2418: done (5 left) 2021-03-15T22:30:11.648 INFO:tasks.rados.rados.0.smithi076.stdout:2423: rollback oid 50 current snap is 275 2021-03-15T22:30:11.649 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 50 to 196 2021-03-15T22:30:11.649 INFO:tasks.rados.rados.0.smithi076.stdout:2416: finishing copy_from to smithi07694878-2 2021-03-15T22:30:11.650 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1586 (ObjNum 774 snap 256 seq_num 774) dirty exists 2021-03-15T22:30:11.650 INFO:tasks.rados.rados.0.smithi076.stdout:2419: finishing write tid 3 to smithi07694878-39 2021-03-15T22:30:11.650 INFO:tasks.rados.rados.0.smithi076.stdout:2416: finishing copy_from racing read to smithi07694878-2 2021-03-15T22:30:11.651 INFO:tasks.rados.rados.0.smithi076.stdout:2419: finishing write tid 4 to smithi07694878-39 2021-03-15T22:30:11.651 INFO:tasks.rados.rados.0.smithi076.stdout:2419: finishing write tid 5 to smithi07694878-39 2021-03-15T22:30:11.652 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 1049 (ObjNum 829 snap 275 seq_num 829) dirty exists 2021-03-15T22:30:11.652 INFO:tasks.rados.rados.0.smithi076.stdout:2419: left oid 39 (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:30:11.653 INFO:tasks.rados.rados.0.smithi076.stdout:2421: expect (ObjNum 747 snap 245 seq_num 747) 2021-03-15T22:30:11.663 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:11.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:11.689 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:11.727 INFO:tasks.rados.rados.0.smithi076.stdout:2420: expect (ObjNum 827 snap 275 seq_num 827) 2021-03-15T22:30:11.790 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:11.812 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:11.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:11.813 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:11.914 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:11.936 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:11.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:11.938 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:12.019 INFO:tasks.rados.rados.0.smithi076.stdout:2422: finishing write tid 1 to smithi07694878-14 2021-03-15T22:30:12.019 INFO:tasks.rados.rados.0.smithi076.stdout:2422: finishing write tid 2 to smithi07694878-14 2021-03-15T22:30:12.019 INFO:tasks.rados.rados.0.smithi076.stdout:2422: finishing write tid 3 to smithi07694878-14 2021-03-15T22:30:12.020 INFO:tasks.rados.rados.0.smithi076.stdout:2422: finishing write tid 4 to smithi07694878-14 2021-03-15T22:30:12.020 INFO:tasks.rados.rados.0.smithi076.stdout:2422: finishing write tid 5 to smithi07694878-14 2021-03-15T22:30:12.020 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 1549 (ObjNum 830 snap 275 seq_num 830) dirty exists 2021-03-15T22:30:12.020 INFO:tasks.rados.rados.0.smithi076.stdout:2422: left oid 14 (ObjNum 830 snap 275 seq_num 830) 2021-03-15T22:30:12.021 INFO:tasks.rados.rados.0.smithi076.stdout:2423: finishing rollback tid 0 to smithi07694878-50 2021-03-15T22:30:12.021 INFO:tasks.rados.rados.0.smithi076.stdout:2423: finishing rollback tid 1 to smithi07694878-50 2021-03-15T22:30:12.021 INFO:tasks.rados.rados.0.smithi076.stdout:2416: done (5 left) 2021-03-15T22:30:12.021 INFO:tasks.rados.rados.0.smithi076.stdout:2419: done (4 left) 2021-03-15T22:30:12.022 INFO:tasks.rados.rados.0.smithi076.stdout:2420: done (3 left) 2021-03-15T22:30:12.022 INFO:tasks.rados.rados.0.smithi076.stdout:2421: done (2 left) 2021-03-15T22:30:12.022 INFO:tasks.rados.rados.0.smithi076.stdout:2422: done (1 left) 2021-03-15T22:30:12.022 INFO:tasks.rados.rados.0.smithi076.stdout:2424: snap_remove snap 270 2021-03-15T22:30:12.039 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:12.065 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:12.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:12.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:12.167 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:12.193 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:12.195 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:12.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:12 smithi076 conmon[56514]: cluster 2021-03-15T22:30:11.140166+0000 mgr.y (mgr.14144) 835 : cluster [DBG] pgmap v1136: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 485 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 14 op/s; 56/1665 objects degraded (3.363%); 121/1665 objects misplaced (7.267%) 2021-03-15T22:30:12.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:12 smithi076 conmon[56514]: cluster 2021-03-15T22:30:11.216713+0000 mon.a (mon.0) 1813 : cluster [DBG] osdmap e662: 8 total, 7 up, 8 in 2021-03-15T22:30:12.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:12 smithi076 conmon[62156]: cluster 2021-03-15T22:30:12.268 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:12 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:30:12.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:12 smithi076 conmon[62156]: 22:30:11.140166+0000 mgr.y (mgr.14144) 835 : cluster [DBG] pgmap v1136: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 485 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 14 op/s; 56/1665 objects degraded (3.363%); 121/1665 objects misplaced (7.267%) 2021-03-15T22:30:12.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:12 smithi076 conmon[62156]: cluster 2021-03-15T22:30:12.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:12 smithi076 conmon[62156]: 2021-03-15T22:30:11.216713+0000 2021-03-15T22:30:12.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:12 smithi076 conmon[62156]: mon.a (mon.0) 1813 : cluster [DBG] osdmap e662: 8 total, 7 up, 8 in 2021-03-15T22:30:12.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:12 smithi076 conmon[62156]: 2021-03-15T22:30:12.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:12.321 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:12.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:12.322 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:12.411 INFO:tasks.rados.rados.0.smithi076.stdout:2424: done (1 left) 2021-03-15T22:30:12.411 INFO:tasks.rados.rados.0.smithi076.stdout:2425: snap_create 2021-03-15T22:30:12.411 INFO:tasks.rados.rados.0.smithi076.stdout:2423: finishing rollback tid 2 to smithi07694878-50 2021-03-15T22:30:12.412 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 1552 (ObjNum 598 snap 183 seq_num 598) dirty exists 2021-03-15T22:30:12.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:12.426 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:30:12.447 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:12.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:12.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:12 smithi125 conmon[56739]: cluster 2021-03-15T22:30:11.140166+0000 mgr.y (mgr.14144) 835 : cluster [DBG] pgmap v1136: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 485 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 14 op/s; 56/1665 objects degraded (3.363%); 121/1665 objects misplaced (7.267%) 2021-03-15T22:30:12.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:12 smithi125 conmon[56739]: cluster 2021-03-15T22:30:11.216713+0000 mon.a (mon.0) 1813 : cluster [DBG] osdmap e662: 8 total, 7 up, 8 in 2021-03-15T22:30:12.550 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:12.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:12.574 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:12.575 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:12.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:12.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:12.700 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:12.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:12.802 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:12.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:12.826 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:12.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:12.928 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:12.951 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:12.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:12.952 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:13.053 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:13.075 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:13.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:13.077 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:13.150 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:30:13.150 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:30:13.151 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:30:13.151 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:30:13.151 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:30:13.151 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:30:13.151 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:30:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:30:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:30:13.152 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:30:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:30:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:30:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:30:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:30:13.153 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:30:13.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:13.203 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:13.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:13.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:13.306 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:13.316 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:13.330 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:13.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:13.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:13.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:13 smithi076 conmon[56514]: cluster 2021-03-15T22:30:12.402753 2021-03-15T22:30:13.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:13 smithi076 conmon[56514]: +0000 mon.a (mon.0) 1814 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2021-03-15T22:30:13.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:13 smithi076 conmon[56514]: audit 2021-03-15T22:30:12.958657+0000 mon.c (mon.1) 56 : audit [DBG] from='client.? 172.21.15.76:0/742635365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:13.410 INFO:tasks.rados.rados.0.smithi076.stdout:2423: done (1 left) 2021-03-15T22:30:13.410 INFO:tasks.rados.rados.0.smithi076.stdout:2425: done (0 left) 2021-03-15T22:30:13.410 INFO:tasks.rados.rados.0.smithi076.stdout:2426: snap_remove snap 271 2021-03-15T22:30:13.433 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:13.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:13.459 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:13.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:13.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:13.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:13.587 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:13.588 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:13.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:13.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:13.715 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:13.715 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:13.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:13 smithi076 conmon[62156]: cluster 2021-03-15T22:30:12.402753+0000 mon.a (mon.0) 1814 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2021-03-15T22:30:13.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:13 smithi076 conmon[62156]: audit 2021-03-15T22:30:12.958657+0000 mon.c (mon.1) 56 : audit [DBG] from='client.? 172.21.15.76:0/742635365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:13 smithi125 conmon[56739]: cluster 2021-03-15T22:30:12.402753+0000 mon.a (mon. 2021-03-15T22:30:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:13 smithi125 conmon[56739]: 0) 1814 : cluster [DBG] osdmap e663: 8 total, 7 up, 8 in 2021-03-15T22:30:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:13 smithi125 conmon[56739]: audit 2021-03-15T22:30:12.958657+0000 mon.c (mon.1) 56 : audit [DBG] from='client.? 172.21.15.76:0/742635365' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:13.816 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:13.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:13.840 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:13.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:13.942 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:13.968 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:13.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:14.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:30:14.070 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:14.165 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:14.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:14.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:14.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:14.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:14.311 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:14.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:14.335 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:30:14.335 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:30:14.412 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:14.427 INFO:tasks.rados.rados.0.smithi076.stdout:2426: done (0 left) 2021-03-15T22:30:14.428 INFO:tasks.rados.rados.0.smithi076.stdout:2427: read oid 25 snap -1 2021-03-15T22:30:14.429 INFO:tasks.rados.rados.0.smithi076.stdout:2427: expect deleted 2021-03-15T22:30:14.429 INFO:tasks.rados.rados.0.smithi076.stdout:2428: write (excl) oid 40 current snap is 276 2021-03-15T22:30:14.429 INFO:tasks.rados.rados.0.smithi076.stdout:2428: seq_num 831 ranges {766617=414396,1945007=510291,3204154=626925} 2021-03-15T22:30:14.432 INFO:tasks.rados.rados.0.smithi076.stdout:2428: writing smithi07694878-40 from 766617 to 1181013 tid 1 2021-03-15T22:30:14.441 INFO:tasks.rados.rados.0.smithi076.stdout:2428: writing smithi07694878-40 from 1945007 to 2455298 tid 2 2021-03-15T22:30:14.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:14.449 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:14.449 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:14.451 INFO:tasks.rados.rados.0.smithi076.stdout:2428: writing smithi07694878-40 from 3204154 to 3831079 tid 3 2021-03-15T22:30:14.452 INFO:tasks.rados.rados.0.smithi076.stdout:2429: snap_remove snap 274 2021-03-15T22:30:14.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:14 smithi076 conmon[56514]: cluster 2021-03-15T22:30:13.140725+0000 mgr.y (mgr.14144) 836 : cluster [DBG] pgmap v1139: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 491 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 3.0 MiB/s wr, 28 op/s; 56/1674 objects degraded (3.345%); 123/1674 objects misplaced (7.348%) 2021-03-15T22:30:14.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:14 smithi076 conmon[56514]: cluster 2021-03-15T22:30:13.408473+0000 mon.a (mon.0) 1815 : cluster [DBG] osdmap e664: 8 total, 7 up, 8 in 2021-03-15T22:30:14.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:14 smithi076 conmon[62156]: cluster 2021-03-15T22:30: 2021-03-15T22:30:14.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:14 smithi076 conmon[62156]: 13.140725+0000 mgr.y (mgr.14144) 836 : cluster [DBG] pgmap v1139: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 491 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 3.0 MiB/s wr, 28 op/s; 56/1674 objects degraded (3.345%); 123/1674 objects misplaced (7.348%) 2021-03-15T22:30:14.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:14 smithi076 conmon[62156]: cluster 2021-03-15T22:30:13.408473+0000 mon.a (mon.0) 1815 : cluster [DBG] osdmap e664: 8 total, 7 up, 8 in 2021-03-15T22:30:14.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:14.556 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:14.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:14.580 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:14.580 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:14.682 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:14.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:14.708 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:14.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:14 smithi125 conmon[56739]: cluster 2021-03-15T22:30:13.140725 2021-03-15T22:30:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:14 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 836 : cluster [DBG] pgmap v1139: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 491 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 22 MiB/s rd, 3.0 MiB/s wr, 28 op/s; 56/1674 objects degraded (3.345%); 123/1674 objects misplaced (7.348%) 2021-03-15T22:30:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:14 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:30:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:14 smithi125 conmon[56739]: -03-15T22:30:13.408473+0000 mon.a (mon.0) 1815 : cluster 2021-03-15T22:30:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:14 smithi125 conmon[56739]: [DBG] osdmap e664: 8 total, 7 up, 8 in 2021-03-15T22:30:14.810 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:14.835 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:14.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:14.837 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:14.938 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:14.963 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:14.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:15.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:15.087 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:15.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:15.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:15.190 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:15.217 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:15.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:15.218 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:15.319 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:15.342 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:15.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:15.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:15.441 INFO:tasks.rados.rados.0.smithi076.stdout:2429: done (2 left) 2021-03-15T22:30:15.441 INFO:tasks.rados.rados.0.smithi076.stdout:2430: snap_create 2021-03-15T22:30:15.442 INFO:tasks.rados.rados.0.smithi076.stdout:2428: finishing write tid 1 to smithi07694878-40 2021-03-15T22:30:15.442 INFO:tasks.rados.rados.0.smithi076.stdout:2428: finishing write tid 2 to smithi07694878-40 2021-03-15T22:30:15.442 INFO:tasks.rados.rados.0.smithi076.stdout:2428: finishing write tid 3 to smithi07694878-40 2021-03-15T22:30:15.442 INFO:tasks.rados.rados.0.smithi076.stdout:2428: finishing write tid 4 to smithi07694878-40 2021-03-15T22:30:15.443 INFO:tasks.rados.rados.0.smithi076.stdout:2428: finishing write tid 5 to smithi07694878-40 2021-03-15T22:30:15.443 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1927 (ObjNum 831 snap 276 seq_num 831) dirty exists 2021-03-15T22:30:15.443 INFO:tasks.rados.rados.0.smithi076.stdout:2428: left oid 40 (ObjNum 831 snap 276 seq_num 831) 2021-03-15T22:30:15.444 INFO:teuthology.orchestra.run.smithi076.stderr:set 7.3 pg_upmap mapping to [1,6,2] 2021-03-15T22:30:15.444 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:15.470 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:15.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:15.501 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[56514]: cluster 2021-03-15T22:30:14.133822+0000 2021-03-15T22:30:15.501 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[56514]: osd.1 (osd.1) 130 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:15.502 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[56514]: cluster 2021-03-15T22:30:14.134382+0000 osd.1 (osd.1) 131 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:15.502 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[56514]: cluster 2021-03-15T22:30:14.412401+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2021-03-15T22:30:15.502 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[56514]: audit 2021-03-15T22:30:14.457539+0000 mon.c (mon.1) 57 : audit [INF] from='client.? 172.21.15.76:0/206449103' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.3", "id": [1, 6, 2]}]: dispatch 2021-03-15T22:30:15.502 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[56514]: audit 2021-03-15T22:30:14.457889+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.3", "id": [1, 6, 2]}]: dispatch 2021-03-15T22:30:15.503 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[56514]: cluster 2021-03-15T22:30:15.200077+0000 mon.a (mon.0) 1818 : cluster [WRN] Health check update: Degraded data redundancy: 56/1674 objects degraded (3.345%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:15.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[62156]: cluster 2021-03-15T22:30:15.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[62156]: 2021-03 2021-03-15T22:30:15.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[62156]: -15T22:30:14.133822+0000 osd.1 (osd.1) 130 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:15.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[62156]: cluster 2021-03-15T22:30:15.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[62156]: 2021-03-15T22:30:15.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[62156]: 2021 2021-03-15T22:30:15.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[62156]: -03-15T22:30:14.134382+0000 osd.1 (osd.1) 131 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:15.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[62156]: cluster 2021-03-15T22:30:14.412401+0000 mon.a (mon.0) 1816 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2021-03-15T22:30:15.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[62156]: audit 2021-03-15T22:30:14.457539+0000 mon.c (mon.1) 57 : audit [INF] from='client.? 172.21.15.76:0/206449103' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.3", "id": [1, 6, 2]}]: dispatch 2021-03-15T22:30:15.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[62156]: audit 2021-03-15T22:30:14.457889+0000 mon.a (mon.0) 1817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.3", "id": [1, 6, 2]}]: dispatch 2021-03-15T22:30:15.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:15 smithi076 conmon[62156]: cluster 2021-03-15T22:30:15.200077+0000 mon.a (mon.0) 1818 : cluster [WRN] Health check update: Degraded data redundancy: 56/1674 objects degraded (3.345%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:15.571 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:15.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:15.596 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:15.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:15.698 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:15.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:15.722 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:15.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:15.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: cluster 2021-03-15T22:30:14.133822+0000 osd.1 (osd.1) 130 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: 2021-03-15T22:30:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: cluster 2021-03-15T22:30:14.134382+0000 osd.1 (osd.1) 131 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: cluster 2021-03-15T22:30:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: 2021-03-15T22:30:14.412401+0000 mon.a (mon.0) 2021-03-15T22:30:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: 1816 : cluster [DBG] osdmap e665: 8 total, 7 up, 8 in 2021-03-15T22:30:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:30:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: 15T22:30:14.457539+0000 2021-03-15T22:30:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: mon.c (mon.1) 57 : audit [INF] 2021-03-15T22:30:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/206449103' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.3", "id": [1, 6, 2]}]: dispatch 2021-03-15T22:30:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:30:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: 22:30:14.457889+0000 mon.a ( 2021-03-15T22:30:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: mon.0) 1817 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.3", "id": [1, 6, 2]}]: dispatch 2021-03-15T22:30:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:30:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: -03-15T22:30:15 2021-03-15T22:30:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: .200077+0000 mon.a (mon.0) 1818 : cluster 2021-03-15T22:30:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:15 smithi125 conmon[56739]: [WRN] Health check update: Degraded data redundancy: 56/1674 objects degraded (3.345%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:15.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:15.848 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:15.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:15.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:15.951 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:15.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:15.978 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:15.978 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:16.079 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:16.101 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:16.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:16.102 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:16.203 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:16.231 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:16.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:16.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:16.335 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:16.336 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:30:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:16.361 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:16.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:16.448 INFO:tasks.rados.rados.0.smithi076.stdout:2427: done (2 left) 2021-03-15T22:30:16.448 INFO:tasks.rados.rados.0.smithi076.stdout:2428: done (1 left) 2021-03-15T22:30:16.448 INFO:tasks.rados.rados.0.smithi076.stdout:2430: done (0 left) 2021-03-15T22:30:16.448 INFO:tasks.rados.rados.0.smithi076.stdout:2431: write (excl) oid 6 current snap is 277 2021-03-15T22:30:16.449 INFO:tasks.rados.rados.0.smithi076.stdout:2431: seq_num 832 ranges {783424=489645,2003489=653949,3231577=554550} 2021-03-15T22:30:16.455 INFO:tasks.rados.rados.0.smithi076.stdout:2431: writing smithi07694878-6 from 783424 to 1273069 tid 1 2021-03-15T22:30:16.462 INFO:tasks.rados.rados.0.smithi076.stdout:2431: writing smithi07694878-6 from 2003489 to 2657438 tid 2 2021-03-15T22:30:16.463 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:16.469 INFO:tasks.rados.rados.0.smithi076.stdout:2431: writing smithi07694878-6 from 3231577 to 3786127 tid 3 2021-03-15T22:30:16.469 INFO:tasks.rados.rados.0.smithi076.stdout:2432: write oid 16 current snap is 277 2021-03-15T22:30:16.469 INFO:tasks.rados.rados.0.smithi076.stdout:2431: finishing write tid 1 to smithi07694878-6 2021-03-15T22:30:16.470 INFO:tasks.rados.rados.0.smithi076.stdout:2432: seq_num 833 ranges {400231=564894,1661971=338260} 2021-03-15T22:30:16.474 INFO:tasks.rados.rados.0.smithi076.stdout:2432: writing smithi07694878-16 from 400231 to 965125 tid 1 2021-03-15T22:30:16.481 INFO:tasks.rados.rados.0.smithi076.stdout:2432: writing smithi07694878-16 from 1661971 to 2000231 tid 2 2021-03-15T22:30:16.481 INFO:tasks.rados.rados.0.smithi076.stdout:2433: delete oid 29 current snap is 277 2021-03-15T22:30:16.481 INFO:tasks.rados.rados.0.smithi076.stdout:2431: finishing write tid 2 to smithi07694878-6 2021-03-15T22:30:16.482 INFO:tasks.rados.rados.0.smithi076.stdout:2431: finishing write tid 3 to smithi07694878-6 2021-03-15T22:30:16.482 INFO:tasks.rados.rados.0.smithi076.stdout:2431: finishing write tid 4 to smithi07694878-6 2021-03-15T22:30:16.482 INFO:tasks.rados.rados.0.smithi076.stdout:2431: finishing write tid 5 to smithi07694878-6 2021-03-15T22:30:16.482 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 1605 (ObjNum 832 snap 277 seq_num 832) dirty exists 2021-03-15T22:30:16.483 INFO:tasks.rados.rados.0.smithi076.stdout:2431: left oid 6 (ObjNum 832 snap 277 seq_num 832) 2021-03-15T22:30:16.483 INFO:tasks.rados.rados.0.smithi076.stdout:2432: finishing write tid 1 to smithi07694878-16 2021-03-15T22:30:16.484 INFO:tasks.rados.rados.0.smithi076.stdout:2432: finishing write tid 2 to smithi07694878-16 2021-03-15T22:30:16.485 INFO:tasks.rados.rados.0.smithi076.stdout:2432: finishing write tid 3 to smithi07694878-16 2021-03-15T22:30:16.485 INFO:tasks.rados.rados.0.smithi076.stdout:2432: finishing write tid 4 to smithi07694878-16 2021-03-15T22:30:16.486 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1808 (ObjNum 833 snap 277 seq_num 833) dirty exists 2021-03-15T22:30:16.486 INFO:tasks.rados.rados.0.smithi076.stdout:2432: left oid 16 (ObjNum 833 snap 277 seq_num 833) 2021-03-15T22:30:16.487 INFO:tasks.rados.rados.0.smithi076.stdout:2431: done (2 left) 2021-03-15T22:30:16.488 INFO:tasks.rados.rados.0.smithi076.stdout:2432: done (1 left) 2021-03-15T22:30:16.488 INFO:tasks.rados.rados.0.smithi076.stdout:2433: done (0 left) 2021-03-15T22:30:16.488 INFO:tasks.rados.rados.0.smithi076.stdout:2434: rollback oid 25 current snap is 277 2021-03-15T22:30:16.488 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 257 2021-03-15T22:30:16.489 INFO:tasks.rados.rados.0.smithi076.stdout:2435: read oid 27 snap -1 2021-03-15T22:30:16.489 INFO:tasks.rados.rados.0.smithi076.stdout:2435: expect (ObjNum 773 snap 255 seq_num 773) 2021-03-15T22:30:16.489 INFO:tasks.rados.rados.0.smithi076.stdout:2436: read oid 16 snap -1 2021-03-15T22:30:16.489 INFO:tasks.rados.rados.0.smithi076.stdout:2436: expect (ObjNum 833 snap 277 seq_num 833) 2021-03-15T22:30:16.490 INFO:tasks.rados.rados.0.smithi076.stdout:2437: rollback oid 48 current snap is 277 2021-03-15T22:30:16.490 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 48 to 268 2021-03-15T22:30:16.490 INFO:tasks.rados.rados.0.smithi076.stdout:2438: snap_create 2021-03-15T22:30:16.491 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:16.491 INFO:tasks.rados.rados.0.smithi076.stdout:2437: finishing rollback tid 0 to smithi07694878-48 2021-03-15T22:30:16.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:16.493 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:16.493 INFO:tasks.rados.rados.0.smithi076.stdout:2437: finishing rollback tid 1 to smithi07694878-48 2021-03-15T22:30:16.493 INFO:tasks.rados.rados.0.smithi076.stdout:2436: expect (ObjNum 833 snap 277 seq_num 833) 2021-03-15T22:30:16.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:16.601 INFO:tasks.rados.rados.0.smithi076.stdout:2434: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:30:16.602 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 1357 (ObjNum 1 snap 0 seq_num 264835904) dirty dne 2021-03-15T22:30:16.602 INFO:tasks.rados.rados.0.smithi076.stdout:2437: finishing rollback tid 2 to smithi07694878-48 2021-03-15T22:30:16.603 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 1214 (ObjNum 757 snap 250 seq_num 757) dirty exists 2021-03-15T22:30:16.603 INFO:tasks.rados.rados.0.smithi076.stdout:2435: expect (ObjNum 773 snap 255 seq_num 773) 2021-03-15T22:30:16.616 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:16.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:16.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:16.637 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:30:16.637 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:30:16.637 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:16 smithi076 conmon[62156]: cluster 2021-03-15T22:30:15. 2021-03-15T22:30:16.638 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:16 smithi076 conmon[62156]: 141198+0000 mgr.y (mgr.14144) 837 : cluster [DBG] pgmap v1142: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 485 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 1.3 MiB/s wr, 19 op/s; 56/1665 objects degraded (3.363%); 121/1665 objects misplaced (7.267%) 2021-03-15T22:30:16.638 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:16 smithi076 conmon[62156]: audit 2021-03-15T22:30:15.431365+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.3", "id": [1, 6, 2]}]': finished 2021-03-15T22:30:16.638 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:16 smithi076 conmon[62156]: cluster 2021-03-15T22:30:15.431417+0000 mon.a (mon.0) 1820 : cluster [DBG] osdmap e666: 8 total, 7 up, 8 in 2021-03-15T22:30:16.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:16 smithi076 conmon[56514]: cluster 2021-03-15T22:30:15.141198 2021-03-15T22:30:16.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:16 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 837 : cluster [DBG] pgmap v1142: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 485 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 1.3 MiB/s wr, 19 op/s; 56/1665 objects degraded (3.363%); 121/1665 objects misplaced (7.267%) 2021-03-15T22:30:16.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:16 smithi076 conmon[56514]: audit 2021-03-15T22:30:15.431365+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.3", "id": [1, 6, 2]}]': finished 2021-03-15T22:30:16.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:16 smithi076 conmon[56514]: cluster 2021-03-15T22:30:15.431417+0000 mon.a (mon.0) 1820 : cluster [DBG] osdmap e666: 8 total, 7 up, 8 in 2021-03-15T22:30:16.720 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:16.745 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:16.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:16 smithi125 conmon[56739]: cluster 2021-03-15T22:30:15.141198+0000 mgr.y (mgr.14144) 837 : cluster [DBG] pgmap v1142: 138 pgs: 31 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 485 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 17 MiB/s rd, 1.3 MiB/s wr, 19 op/s; 56/1665 objects degraded (3.363%); 121/1665 objects misplaced (7.267%) 2021-03-15T22:30:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:16 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:30:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:16 smithi125 conmon[56739]: 30:15.431365+0000 mon.a (mon.0) 1819 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.3", "id": [1, 6, 2]}]': finished 2021-03-15T22:30:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:16 smithi125 conmon[56739]: cluster 2021-03-15T22:30:15.431417+0000 mon.a (mon.0) 1820 : cluster [DBG] osdmap e666: 8 total, 7 up, 8 in 2021-03-15T22:30:16.847 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:16.869 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:16.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:16.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:16.972 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:16.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:16.995 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:16.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:17.097 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:17.122 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:17.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:17.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:17.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:17.250 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:17.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:17.252 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:17.330 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:17.352 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:17.378 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:17.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:17.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:17.450 INFO:tasks.rados.rados.0.smithi076.stdout:2434: done (4 left) 2021-03-15T22:30:17.450 INFO:tasks.rados.rados.0.smithi076.stdout:2435: done (3 left) 2021-03-15T22:30:17.451 INFO:tasks.rados.rados.0.smithi076.stdout:2436: done (2 left) 2021-03-15T22:30:17.451 INFO:tasks.rados.rados.0.smithi076.stdout:2437: done (1 left) 2021-03-15T22:30:17.452 INFO:tasks.rados.rados.0.smithi076.stdout:2438: done (0 left) 2021-03-15T22:30:17.452 INFO:tasks.rados.rados.0.smithi076.stdout:2439: delete oid 3 current snap is 278 2021-03-15T22:30:17.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:17.511 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:17.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:17.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:17.521 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:30:17.522 INFO:tasks.rados.rados.0.smithi076.stdout:2439: done (0 left) 2021-03-15T22:30:17.522 INFO:tasks.rados.rados.0.smithi076.stdout:2440: copy_from oid 49 from oid 11 current snap is 278 2021-03-15T22:30:17.523 INFO:tasks.rados.rados.0.smithi076.stdout:2441: write (excl) oid 40 current snap is 278 2021-03-15T22:30:17.523 INFO:tasks.rados.rados.0.smithi076.stdout:2441: seq_num 836 ranges {450652=790641,1721064=329588} 2021-03-15T22:30:17.528 INFO:tasks.rados.rados.0.smithi076.stdout:2441: writing smithi07694878-40 from 450652 to 1241293 tid 1 2021-03-15T22:30:17.534 INFO:tasks.rados.rados.0.smithi076.stdout:2441: writing smithi07694878-40 from 1721064 to 2050652 tid 2 2021-03-15T22:30:17.534 INFO:tasks.rados.rados.0.smithi076.stdout:2442: write oid 27 current snap is 278 2021-03-15T22:30:17.534 INFO:tasks.rados.rados.0.smithi076.stdout:2442: seq_num 837 ranges {467459=582243,1495899=404945} 2021-03-15T22:30:17.539 INFO:tasks.rados.rados.0.smithi076.stdout:2442: writing smithi07694878-27 from 467459 to 1049702 tid 1 2021-03-15T22:30:17.545 INFO:tasks.rados.rados.0.smithi076.stdout:2442: writing smithi07694878-27 from 1495899 to 1900844 tid 2 2021-03-15T22:30:17.545 INFO:tasks.rados.rados.0.smithi076.stdout:2443: delete oid 46 current snap is 278 2021-03-15T22:30:17.545 INFO:tasks.rados.rados.0.smithi076.stdout:2440: finishing copy_from to smithi07694878-49 2021-03-15T22:30:17.545 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 1547 (ObjNum 813 snap 269 seq_num 813) dirty exists 2021-03-15T22:30:17.546 INFO:tasks.rados.rados.0.smithi076.stdout:2440: finishing copy_from racing read to smithi07694878-49 2021-03-15T22:30:17.546 INFO:tasks.rados.rados.0.smithi076.stdout:2442: finishing write tid 1 to smithi07694878-27 2021-03-15T22:30:17.546 INFO:tasks.rados.rados.0.smithi076.stdout:2441: finishing write tid 1 to smithi07694878-40 2021-03-15T22:30:17.548 INFO:tasks.rados.rados.0.smithi076.stdout:2441: finishing write tid 2 to smithi07694878-40 2021-03-15T22:30:17.548 INFO:tasks.rados.rados.0.smithi076.stdout:2440: done (3 left) 2021-03-15T22:30:17.549 INFO:tasks.rados.rados.0.smithi076.stdout:2443: done (2 left) 2021-03-15T22:30:17.549 INFO:tasks.rados.rados.0.smithi076.stdout:2444: rollback oid 26 current snap is 278 2021-03-15T22:30:17.549 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 257 2021-03-15T22:30:17.549 INFO:tasks.rados.rados.0.smithi076.stdout:2441: finishing write tid 3 to smithi07694878-40 2021-03-15T22:30:17.549 INFO:tasks.rados.rados.0.smithi076.stdout:2445: snap_create 2021-03-15T22:30:17.550 INFO:tasks.rados.rados.0.smithi076.stdout:2441: finishing write tid 4 to smithi07694878-40 2021-03-15T22:30:17.550 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1937 (ObjNum 836 snap 278 seq_num 836) dirty exists 2021-03-15T22:30:17.550 INFO:tasks.rados.rados.0.smithi076.stdout:2441: left oid 40 (ObjNum 836 snap 278 seq_num 836) 2021-03-15T22:30:17.550 INFO:tasks.rados.rados.0.smithi076.stdout:2442: finishing write tid 2 to smithi07694878-27 2021-03-15T22:30:17.550 INFO:tasks.rados.rados.0.smithi076.stdout:2442: finishing write tid 3 to smithi07694878-27 2021-03-15T22:30:17.551 INFO:tasks.rados.rados.0.smithi076.stdout:2442: finishing write tid 4 to smithi07694878-27 2021-03-15T22:30:17.551 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1740 (ObjNum 837 snap 278 seq_num 837) dirty exists 2021-03-15T22:30:17.551 INFO:tasks.rados.rados.0.smithi076.stdout:2442: left oid 27 (ObjNum 837 snap 278 seq_num 837) 2021-03-15T22:30:17.552 INFO:tasks.rados.rados.0.smithi076.stdout:2444: finishing rollback tid 0 to smithi07694878-26 2021-03-15T22:30:17.558 INFO:tasks.rados.rados.0.smithi076.stdout:2444: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:30:17.559 INFO:tasks.rados.rados.0.smithi076.stdout:2444: finishing rollback tid 2 to smithi07694878-26 2021-03-15T22:30:17.560 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 1555 (ObjNum 755 snap 248 seq_num 755) dirty exists 2021-03-15T22:30:17.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:17.636 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:17.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:17.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:17.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:17.761 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:17 smithi076 conmon[56514]: cluster 2021-03-15T22:30: 2021-03-15T22:30:17.762 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:17 smithi076 conmon[56514]: 16.441041+0000 mon.a (mon.0) 1821 : cluster [DBG] osdmap e667: 8 total, 7 up, 8 in 2021-03-15T22:30:17.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:17 smithi076 conmon[62156]: cluster 2021-03-15T22:30:16.441041+0000 mon.a (mon.0) 1821 : cluster [DBG] osdmap e667: 8 total, 7 up, 8 in 2021-03-15T22:30:17.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:17.763 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:17.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:17.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:17 smithi125 conmon[56739]: cluster 2021-03-15T22:30:16.441041+0000 mon.a (mon.0 2021-03-15T22:30:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:17 smithi125 conmon[56739]: ) 1821 : cluster [DBG] osdmap e667: 8 total, 7 up, 8 in 2021-03-15T22:30:17.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:17.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:17.889 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:17.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:17.990 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:18.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:18.015 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:18.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:18.074 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:18.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:18.141 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:18.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:18.243 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:18.266 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:18.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:18.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:18.369 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:18.392 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:18.394 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:18.458 INFO:tasks.rados.rados.0.smithi076.stdout:2441: done (3 left) 2021-03-15T22:30:18.459 INFO:tasks.rados.rados.0.smithi076.stdout:2442: done (2 left) 2021-03-15T22:30:18.459 INFO:tasks.rados.rados.0.smithi076.stdout:2444: done (1 left) 2021-03-15T22:30:18.459 INFO:tasks.rados.rados.0.smithi076.stdout:2445: done (0 left) 2021-03-15T22:30:18.460 INFO:tasks.rados.rados.0.smithi076.stdout:2446: snap_remove snap 257 2021-03-15T22:30:18.465 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:30:18.495 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:18.516 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:18.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:18.518 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:18.619 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:18.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:18.643 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:18.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:18.718 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:18 smithi076 conmon[56514]: cluster 2021-03-15T22:30:17.141818+0000 mgr.y (mgr.14144) 838 : cluster [DBG] pgmap v1145: 138 pgs: 1 remapped+peering, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 479 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.6 MiB/s wr, 9 op/s; 56/1656 objects degraded (3.382%); 107/1656 objects misplaced (6.461%) 2021-03-15T22:30:18.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:18 smithi076 conmon[56514]: cluster 2021-03-15T22:30:17.446591+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2021-03-15T22:30:18.719 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:18 smithi076 conmon[56514]: audit 2021-03-15T22:30:17.880427+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? 172.21.15.76:0/3563577487' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:18.719 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:18 smithi076 conmon[62156]: cluster 2021-03-15T22:30:17.141818+0000 mgr.y (mgr.14144) 838 : cluster [DBG] pgmap v1145: 138 pgs: 1 remapped+peering, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 479 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.6 MiB/s wr, 9 op/s; 56/1656 objects degraded (3.382%); 107/1656 objects misplaced (6.461%) 2021-03-15T22:30:18.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:18 smithi076 conmon[62156]: cluster 2021-03-15T22:30:17.446591+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2021-03-15T22:30:18.720 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:18 smithi076 conmon[62156]: audit 2021-03-15T22:30:17.880427+0000 mon.a (mon.0) 1823 : audit [INF] from='client.? 172.21.15.76:0/3563577487' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:18.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:18.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:18.769 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:18.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:18 smithi125 conmon[56739]: cluster 2021-03-15T22:30:17.141818+0000 mgr.y (mgr.14144) 838 : cluster [DBG] pgmap v1145: 138 pgs: 1 remapped+peering, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 479 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 4.9 MiB/s rd, 1.6 MiB/s wr, 9 op/s; 56/1656 objects degraded (3.382%); 107/1656 objects misplaced (6.461%) 2021-03-15T22:30:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:18 smithi125 conmon[56739]: cluster 2021-03-15T22:30:17.446591+0000 mon.a (mon.0) 1822 : cluster [DBG] osdmap e668: 8 total, 7 up, 8 in 2021-03-15T22:30:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:18 smithi125 conmon[56739]: audit 2021-03-15T22:30:17.880427+0000 2021-03-15T22:30:18.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:18 smithi125 conmon[56739]: mon.a (mon.0) 1823 : audit [INF] from='client.? 172.21.15.76:0/3563577487' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:18.871 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:18.897 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:18.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:18.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:19.000 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:19.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:19.028 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:19.029 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:19.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:19.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:19.158 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:19.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:19.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:19.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:19.285 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:19.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:19.387 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:19.411 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:19.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:19.463 INFO:tasks.rados.rados.0.smithi076.stdout:2446: done (0 left) 2021-03-15T22:30:19.464 INFO:tasks.rados.rados.0.smithi076.stdout:2447: rollback oid 40 current snap is 279 2021-03-15T22:30:19.464 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 40 to 276 2021-03-15T22:30:19.464 INFO:tasks.rados.rados.0.smithi076.stdout:2448: snap_create 2021-03-15T22:30:19.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[56514]: cluster 2021-03-15T22:30:18.444344+0000 mon.a (mon.0) 2021-03-15T22:30:19.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[56514]: 1824 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:19.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[56514]: audit 2021-03-15T22:30:18.452554+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? 172.21.15.76:0/3563577487' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:19.466 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[56514]: cluster 2021-03-15T22:30:18.452616+0000 mon.a (mon.0) 1826 : cluster [DBG] osdmap e669: 8 total, 7 up, 8 in 2021-03-15T22:30:19.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[62156]: cluster 2021-03-15T22:30:18 2021-03-15T22:30:19.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[62156]: .444344+0000 mon.a (mon.0) 1824 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:19.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[62156]: audit 2021 2021-03-15T22:30:19.467 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[62156]: -03-15T22:30:18.452554+0000 mon.a (mon.0) 1825 : 2021-03-15T22:30:19.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/3563577487' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:19.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:30:19.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[62156]: 15T22:30:18. 2021-03-15T22:30:19.468 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[62156]: 452616+0000 mon.a (mon.0 2021-03-15T22:30:19.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[62156]: ) 1826 : cluster [DBG] osdmap e669: 8 total, 7 up, 8 in 2021-03-15T22:30:19.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:19 smithi076 conmon[62156]: 2021-03-15T22:30:19.473 INFO:tasks.rados.rados.0.smithi076.stdout:2447: finishing rollback tid 0 to smithi07694878-40 2021-03-15T22:30:19.475 INFO:tasks.rados.rados.0.smithi076.stdout:2447: finishing rollback tid 1 to smithi07694878-40 2021-03-15T22:30:19.476 INFO:tasks.rados.rados.0.smithi076.stdout:2447: finishing rollback tid 2 to smithi07694878-40 2021-03-15T22:30:19.477 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1945 (ObjNum 831 snap 276 seq_num 831) dirty exists 2021-03-15T22:30:19.513 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:19.535 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:19.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:19.537 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:19.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:19.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:19.660 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:19.661 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:19.762 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:19.785 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:19.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:19 smithi125 conmon[56739]: cluster 2021-03-15T22:30:18.444344+0000 mon.a (mon.0) 2021-03-15T22:30:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:19 smithi125 conmon[56739]: 1824 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:19 smithi125 conmon[56739]: audit 2021-03-15T22:30:18.452554+0000 mon.a (mon.0) 1825 : audit [INF] from='client.? 172.21.15.76:0/3563577487' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:19 smithi125 conmon[56739]: cluster 2021-03-15T22:30:18.452616+0000 mon.a (mon.0) 1826 : cluster [DBG] 2021-03-15T22:30:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:19 smithi125 conmon[56739]: osdmap e669: 8 total, 7 up, 8 in 2021-03-15T22:30:19.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:19.889 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:19.912 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:19.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:19.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:20.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:20.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:20.039 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:20.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:20.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:20.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:20.163 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:20.164 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:20.214 INFO:tasks.rados.rados.0.smithi076.stdout:2447: done (1 left) 2021-03-15T22:30:20.214 INFO:tasks.rados.rados.0.smithi076.stdout:2448: done (0 left) 2021-03-15T22:30:20.215 INFO:tasks.rados.rados.0.smithi076.stdout:2449: write (excl) oid 5 current snap is 280 2021-03-15T22:30:20.215 INFO:tasks.rados.rados.0.smithi076.stdout:2449: seq_num 839 ranges {501073=732741,1896510=204563} 2021-03-15T22:30:20.221 INFO:tasks.rados.rados.0.smithi076.stdout:2449: writing smithi07694878-5 from 501073 to 1233814 tid 1 2021-03-15T22:30:20.226 INFO:tasks.rados.rados.0.smithi076.stdout:2449: writing smithi07694878-5 from 1896510 to 2101073 tid 2 2021-03-15T22:30:20.226 INFO:tasks.rados.rados.0.smithi076.stdout:2450: snap_create 2021-03-15T22:30:20.227 INFO:tasks.rados.rados.0.smithi076.stdout:2449: finishing write tid 1 to smithi07694878-5 2021-03-15T22:30:20.229 INFO:tasks.rados.rados.0.smithi076.stdout:2449: finishing write tid 2 to smithi07694878-5 2021-03-15T22:30:20.230 INFO:tasks.rados.rados.0.smithi076.stdout:2449: finishing write tid 3 to smithi07694878-5 2021-03-15T22:30:20.230 INFO:tasks.rados.rados.0.smithi076.stdout:2449: finishing write tid 4 to smithi07694878-5 2021-03-15T22:30:20.231 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1509 (ObjNum 839 snap 280 seq_num 839) dirty exists 2021-03-15T22:30:20.231 INFO:tasks.rados.rados.0.smithi076.stdout:2449: left oid 5 (ObjNum 839 snap 280 seq_num 839) 2021-03-15T22:30:20.266 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:20.286 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:30:19 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:30:20.288 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:20.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:20.289 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:20.361 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:30:20.361 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:30:20.361 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:30:20.367 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:30:20.385 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:30:20.391 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:20.402 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:30:20.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:20.413 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:20.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:20.419 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:30:20.420 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:30:20.420 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:30:20.420 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:30:20.421 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:30:20.421 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:30:20.421 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:30:20.421 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:30:20.421 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:30:20.515 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:20.539 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:20.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:20.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:20.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:20.664 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:20.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:20.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:20.767 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[56514]: cluster 2021-03-15T22:30:19.142492+0000 mgr.y (mgr.14144) 839 : cluster [DBG] pgmap v1148: 138 pgs: 1 remapped+peering, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 479 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 6 op/s; 56/1656 objects degraded (3.382%); 107/1656 objects misplaced (6.461%) 2021-03-15T22:30:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[56514]: cluster 2021-03-15T22:30:19.456386+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e670: 8 total, 7 up, 8 in 2021-03-15T22:30:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[56514]: audit 2021-03-15T22:30:19.989227+0000 mgr.y (mgr.14144) 840 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[56514]: cluster 2021-03-15T22:30:20.201386+0000 mon.a (mon.0) 1828 : cluster [WRN] Health check update: Degraded data redundancy: 56/1656 objects degraded (3.382%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[56514]: cluster 2021-03-15T22:30:20.209118+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e671: 8 total, 7 up, 8 in 2021-03-15T22:30:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[56514]: audit 2021-03-15T22:30:20.292649+0000 mon.a (mon.0) 1830 : audit [DBG] from='client.? 172.21.15.76:0/3449612820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:20.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[62156]: cluster 2021-03-15T22:30:19.142492+0000 mgr.y (mgr.14144) 839 : cluster [DBG] pgmap v1148: 138 pgs: 1 remapped+peering, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 479 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 6 op/s; 56/1656 objects degraded (3.382%); 107/1656 objects misplaced (6.461%) 2021-03-15T22:30:20.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[62156]: cluster 2021-03-15T22:30:19.456386+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e670: 8 total, 7 up, 8 in 2021-03-15T22:30:20.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[62156]: audit 2021-03-15T22:30:19.989227+0000 mgr.y (mgr.14144) 840 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:20.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[62156]: cluster 2021-03-15T22:30:20.201386+0000 mon.a (mon.0) 1828 : cluster [WRN] Health check update: Degraded data redundancy: 56/1656 objects degraded (3.382%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[62156]: cluster 2021-03-15T22:30:20.209118+0000 mon.a (mon.0) 1829 : cluster [DBG] osdmap e671: 8 total, 7 up, 8 in 2021-03-15T22:30:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[62156]: audit 2021-03-15T22:30:20.292649+0000 mon.a (mon 2021-03-15T22:30:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:20 smithi076 conmon[62156]: .0) 1830 : audit [DBG] from='client.? 172.21.15.76:0/3449612820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:20.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: cluster 2021-03-15T22:30:19.142492+0000 mgr.y (mgr.14144 2021-03-15T22:30:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: ) 839 : cluster [DBG] pgmap v1148: 138 pgs: 1 remapped+peering, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 479 MiB data, 1.4 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 1.6 MiB/s wr, 6 op/s; 56/1656 objects degraded (3.382%); 107/1656 objects misplaced (6.461%) 2021-03-15T22:30:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: cluster 2021-03-15T22:30:19.456386+0000 mon.a (mon.0) 1827 : cluster [DBG] osdmap e670: 8 total, 7 up, 8 in 2021-03-15T22:30:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: audit 2021 2021-03-15T22:30:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: -03-15T22:30:19.989227+0000 mgr.y (mgr.14144) 840 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:30:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: 03-15T22:30:20.201386+0000 mon.a (mon.0) 1828 : cluster [WRN] Health check update: Degraded data redundancy: 56/1656 objects degraded (3.382%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: 2021-03-15T22:30:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:30:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: -15T22:30:20.209118+0000 mon.a (mon.0) 1829 : cluster [DBG] 2021-03-15T22:30:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: osdmap e671: 8 total, 7 up, 8 in 2021-03-15T22:30:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:30:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: 03-15T22:30:20.292649+0000 mon.a (mon.0) 1830 : audit [DBG] from='client.? 172.21.15.76:0/3449612820' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:20 smithi125 conmon[56739]: 2021-03-15T22:30:20.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:20.796 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:20.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:20.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:20.920 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:20.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:20.922 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:21.023 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:21.045 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:21.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:21.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:21.134 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:30:21.135 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:30:21.148 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:21.172 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:21.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:21.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:21.225 INFO:tasks.rados.rados.0.smithi076.stdout:2449: done (1 left) 2021-03-15T22:30:21.225 INFO:tasks.rados.rados.0.smithi076.stdout:2450: done (0 left) 2021-03-15T22:30:21.225 INFO:tasks.rados.rados.0.smithi076.stdout:2451: delete oid 5 current snap is 281 2021-03-15T22:30:21.227 INFO:tasks.rados.rados.0.smithi076.stdout:2451: done (0 left) 2021-03-15T22:30:21.227 INFO:tasks.rados.rados.0.smithi076.stdout:2452: write oid 18 current snap is 281 2021-03-15T22:30:21.227 INFO:tasks.rados.rados.0.smithi076.stdout:2452: seq_num 841 ranges {534687=483239,1613474=412283} 2021-03-15T22:30:21.232 INFO:tasks.rados.rados.0.smithi076.stdout:2452: writing smithi07694878-18 from 534687 to 1017926 tid 1 2021-03-15T22:30:21.238 INFO:tasks.rados.rados.0.smithi076.stdout:2452: writing smithi07694878-18 from 1613474 to 2025757 tid 2 2021-03-15T22:30:21.239 INFO:tasks.rados.rados.0.smithi076.stdout:2453: write oid 14 current snap is 281 2021-03-15T22:30:21.239 INFO:tasks.rados.rados.0.smithi076.stdout:2452: finishing write tid 1 to smithi07694878-18 2021-03-15T22:30:21.239 INFO:tasks.rados.rados.0.smithi076.stdout:2453: seq_num 842 ranges {551494=558488,1671956=479538} 2021-03-15T22:30:21.244 INFO:tasks.rados.rados.0.smithi076.stdout:2453: writing smithi07694878-14 from 551494 to 1109982 tid 1 2021-03-15T22:30:21.251 INFO:tasks.rados.rados.0.smithi076.stdout:2453: writing smithi07694878-14 from 1671956 to 2151494 tid 2 2021-03-15T22:30:21.251 INFO:tasks.rados.rados.0.smithi076.stdout:2454: delete oid 23 current snap is 281 2021-03-15T22:30:21.251 INFO:tasks.rados.rados.0.smithi076.stdout:2452: finishing write tid 2 to smithi07694878-18 2021-03-15T22:30:21.251 INFO:tasks.rados.rados.0.smithi076.stdout:2452: finishing write tid 3 to smithi07694878-18 2021-03-15T22:30:21.252 INFO:tasks.rados.rados.0.smithi076.stdout:2452: finishing write tid 4 to smithi07694878-18 2021-03-15T22:30:21.252 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 1213 (ObjNum 841 snap 281 seq_num 841) dirty exists 2021-03-15T22:30:21.252 INFO:tasks.rados.rados.0.smithi076.stdout:2452: left oid 18 (ObjNum 841 snap 281 seq_num 841) 2021-03-15T22:30:21.253 INFO:tasks.rados.rados.0.smithi076.stdout:2452: done (2 left) 2021-03-15T22:30:21.253 INFO:tasks.rados.rados.0.smithi076.stdout:2454: done (1 left) 2021-03-15T22:30:21.253 INFO:tasks.rados.rados.0.smithi076.stdout:2455: write oid 45 current snap is 281 2021-03-15T22:30:21.253 INFO:tasks.rados.rados.0.smithi076.stdout:2455: seq_num 844 ranges {585108=425339,1788920=396188} 2021-03-15T22:30:21.258 INFO:tasks.rados.rados.0.smithi076.stdout:2455: writing smithi07694878-45 from 585108 to 1010447 tid 1 2021-03-15T22:30:21.264 INFO:tasks.rados.rados.0.smithi076.stdout:2455: writing smithi07694878-45 from 1788920 to 2185108 tid 2 2021-03-15T22:30:21.265 INFO:tasks.rados.rados.0.smithi076.stdout:2456: snap_remove snap 263 2021-03-15T22:30:21.276 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:21.285 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:30:21.285 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:30:21.286 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:30:21.286 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:30:21.286 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:30:21.298 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:21.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:21.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:21.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:21.423 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:21.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:21.424 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:21.495 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:30:21.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:21.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:21.550 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:21.550 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:21.651 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:21.673 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:21.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:21.675 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:21.776 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:21.792 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 0, 3, 5] out_osds: [] dead_osds: [2] live_osds: [0, 1, 3, 4, 6, 7, 5] 2021-03-15T22:30:21.793 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:30:21.793 INFO:tasks.thrashosds.thrasher:Reviving osd 2 2021-03-15T22:30:21.793 INFO:tasks.cephadm.osd.2:Restarting osd.2 (starting--it wasn't running)... 2021-03-15T22:30:21.794 DEBUG:teuthology.orchestra.run.smithi076:osd.2> sudo journalctl -f -n 0 -u ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.2.service 2021-03-15T22:30:21.797 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl start ceph-8a572802-85db-11eb-9091-001a4aab830c@osd.2 2021-03-15T22:30:21.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:21.800 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:21.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:21.822 INFO:journalctl@ceph.osd.2.smithi076.stdout:-- Logs begin at Mon 2021-03-15 21:44:23 UTC. -- 2021-03-15T22:30:21.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:21.922 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:21.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:21.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:21.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:30:21.984 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:22.025 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:22.052 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:22.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:22.053 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:22.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:22.178 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:21 smithi076 systemd[1]: Starting Ceph osd.2 for 8a572802-85db-11eb-9091-001a4aab830c... 2021-03-15T22:30:22.180 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:22.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:22.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:22.234 INFO:tasks.rados.rados.0.smithi076.stdout:2456: done (2 left) 2021-03-15T22:30:22.234 INFO:tasks.rados.rados.0.smithi076.stdout:2457: delete oid 46 current snap is 281 2021-03-15T22:30:22.234 INFO:tasks.rados.rados.0.smithi076.stdout:2455: finishing write tid 1 to smithi07694878-45 2021-03-15T22:30:22.235 INFO:tasks.rados.rados.0.smithi076.stdout:2455: finishing write tid 2 to smithi07694878-45 2021-03-15T22:30:22.235 INFO:tasks.rados.rados.0.smithi076.stdout:2455: finishing write tid 3 to smithi07694878-45 2021-03-15T22:30:22.235 INFO:tasks.rados.rados.0.smithi076.stdout:2455: finishing write tid 4 to smithi07694878-45 2021-03-15T22:30:22.235 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 1608 (ObjNum 844 snap 281 seq_num 844) dirty exists 2021-03-15T22:30:22.236 INFO:tasks.rados.rados.0.smithi076.stdout:2455: left oid 45 (ObjNum 844 snap 281 seq_num 844) 2021-03-15T22:30:22.239 INFO:tasks.rados.rados.0.smithi076.stdout:2455: done (2 left) 2021-03-15T22:30:22.239 INFO:tasks.rados.rados.0.smithi076.stdout:2457: done (1 left) 2021-03-15T22:30:22.239 INFO:tasks.rados.rados.0.smithi076.stdout:2458: write oid 18 current snap is 281 2021-03-15T22:30:22.239 INFO:tasks.rados.rados.0.smithi076.stdout:2458: seq_num 846 ranges {618722=575837,1905884=312838} 2021-03-15T22:30:22.247 INFO:tasks.rados.rados.0.smithi076.stdout:2458: writing smithi07694878-18 from 618722 to 1194559 tid 1 2021-03-15T22:30:22.255 INFO:tasks.rados.rados.0.smithi076.stdout:2458: writing smithi07694878-18 from 1905884 to 2218722 tid 2 2021-03-15T22:30:22.255 INFO:tasks.rados.rados.0.smithi076.stdout:2458: finishing write tid 1 to smithi07694878-18 2021-03-15T22:30:22.255 INFO:tasks.rados.rados.0.smithi076.stdout:2459: write (excl) oid 31 current snap is 281 2021-03-15T22:30:22.256 INFO:tasks.rados.rados.0.smithi076.stdout:2459: seq_num 847 ranges {635529=651086,1848013=387516} 2021-03-15T22:30:22.262 INFO:tasks.rados.rados.0.smithi076.stdout:2459: writing smithi07694878-31 from 635529 to 1286615 tid 1 2021-03-15T22:30:22.267 INFO:tasks.rados.rados.0.smithi076.stdout:2459: writing smithi07694878-31 from 1848013 to 2235529 tid 2 2021-03-15T22:30:22.268 INFO:tasks.rados.rados.0.smithi076.stdout:2460: snap_remove snap 276 2021-03-15T22:30:22.282 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:22.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:22.306 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:22.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:22.407 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:22.432 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:22.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:22.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:22 smithi076 conmon[56514]: cluster 2021-03-15T22:30:21.143072+0000 mgr.y (mgr.14144 2021-03-15T22:30:22.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:22 smithi076 conmon[56514]: ) 841 : cluster [DBG] pgmap v1151: 138 pgs: 1 remapped+peering, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 486 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 14 op/s; 57/1680 objects degraded (3.393%); 107/1680 objects misplaced (6.369%) 2021-03-15T22:30:22.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:22 smithi076 conmon[56514]: cluster 2021-03-15T22:30:21.222632+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e672: 8 total, 7 up, 8 in 2021-03-15T22:30:22.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:22 smithi076 conmon[62156]: cluster 2021-03-15T22:30:21.143072+0000 mgr.y 2021-03-15T22:30:22.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:22 smithi076 conmon[62156]: (mgr.14144) 841 : cluster [DBG] pgmap v1151: 138 pgs: 1 remapped+peering, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 486 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 14 op/s; 57/1680 objects degraded (3.393%); 107/1680 objects misplaced (6.369%) 2021-03-15T22:30:22.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:22 smithi076 conmon[62156]: cluster 2021-03-15T22:30:21.222632+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e672: 8 total, 7 up, 8 in 2021-03-15T22:30:22.472 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:22 smithi076 podman[222326]: 2021-03-15 22:30:22.325945831 +0000 UTC m=+0.177648907 container create 6be260ad6886df227ff2447783072409c7e45f3372f79389d9a7fe9a2aaa9702 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2-activate, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.18.0, ceph=True, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, org.label-schema.build-date=20201204, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Base Image) 2021-03-15T22:30:22.533 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:22.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:22 smithi125 conmon[56739]: cluster 2021-03-15T22:30:21.143072+0000 mgr.y (mgr.14144) 841 : cluster [DBG] pgmap v1151: 138 pgs: 1 remapped+peering, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 486 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.2 MiB/s wr, 14 op/s; 57/1680 objects degraded (3.393%); 107/1680 objects misplaced (6.369%) 2021-03-15T22:30:22.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:22 smithi125 conmon[56739]: cluster 2021-03-15T22:30:21.222632+0000 mon.a (mon.0) 1831 : cluster [DBG] osdmap e672: 8 total, 7 up, 8 in 2021-03-15T22:30:22.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:22.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:22.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:22.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:22.687 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:22.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:22.789 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:22.812 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:22.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:22.914 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:22.935 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:22.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:22.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:23.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:23.063 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:23.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:23.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:23.136 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:30:23.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:23.203 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:23.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:23.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:23.246 INFO:tasks.rados.rados.0.smithi076.stdout:2460: done (3 left) 2021-03-15T22:30:23.247 INFO:tasks.rados.rados.0.smithi076.stdout:2461: rollback oid 48 current snap is 281 2021-03-15T22:30:23.247 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 48 to 247 2021-03-15T22:30:23.247 INFO:tasks.rados.rados.0.smithi076.stdout:2458: finishing write tid 2 to smithi07694878-18 2021-03-15T22:30:23.247 INFO:tasks.rados.rados.0.smithi076.stdout:2458: finishing write tid 3 to smithi07694878-18 2021-03-15T22:30:23.247 INFO:tasks.rados.rados.0.smithi076.stdout:2458: finishing write tid 4 to smithi07694878-18 2021-03-15T22:30:23.248 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 1217 (ObjNum 846 snap 281 seq_num 846) dirty exists 2021-03-15T22:30:23.248 INFO:tasks.rados.rados.0.smithi076.stdout:2458: left oid 18 (ObjNum 846 snap 281 seq_num 846) 2021-03-15T22:30:23.248 INFO:tasks.rados.rados.0.smithi076.stdout:2459: finishing write tid 1 to smithi07694878-31 2021-03-15T22:30:23.248 INFO:tasks.rados.rados.0.smithi076.stdout:2459: finishing write tid 2 to smithi07694878-31 2021-03-15T22:30:23.249 INFO:tasks.rados.rados.0.smithi076.stdout:2459: finishing write tid 3 to smithi07694878-31 2021-03-15T22:30:23.249 INFO:tasks.rados.rados.0.smithi076.stdout:2459: finishing write tid 4 to smithi07694878-31 2021-03-15T22:30:23.249 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 1846 (ObjNum 847 snap 281 seq_num 847) dirty exists 2021-03-15T22:30:23.249 INFO:tasks.rados.rados.0.smithi076.stdout:2459: left oid 31 (ObjNum 847 snap 281 seq_num 847) 2021-03-15T22:30:23.249 INFO:tasks.rados.rados.0.smithi076.stdout:2458: done (3 left) 2021-03-15T22:30:23.250 INFO:tasks.rados.rados.0.smithi076.stdout:2459: done (2 left) 2021-03-15T22:30:23.250 INFO:tasks.rados.rados.0.smithi076.stdout:2462: snap_create 2021-03-15T22:30:23.251 INFO:tasks.rados.rados.0.smithi076.stdout:2461: finishing rollback tid 0 to smithi07694878-48 2021-03-15T22:30:23.252 INFO:tasks.rados.rados.0.smithi076.stdout:2461: finishing rollback tid 1 to smithi07694878-48 2021-03-15T22:30:23.252 INFO:tasks.rados.rados.0.smithi076.stdout:2461: finishing rollback tid 2 to smithi07694878-48 2021-03-15T22:30:23.252 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 1217 (ObjNum 641 snap 200 seq_num 641) dirty exists 2021-03-15T22:30:23.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:23 smithi076 conmon[56514]: cluster 2021-03-15T22:30:22.225695+0000 mon.a (mon. 2021-03-15T22:30:23.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:23 smithi076 conmon[56514]: 0) 1832 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2021-03-15T22:30:23.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:23 smithi076 conmon[62156]: cluster 2021-03-15T22:30:22.225695+0000 2021-03-15T22:30:23.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:23 smithi076 conmon[62156]: mon.a (mon.0) 1832 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2021-03-15T22:30:23.269 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:23 smithi076 podman[222326]: 2021-03-15 22:30:23.004270442 +0000 UTC m=+0.855973611 container init 6be260ad6886df227ff2447783072409c7e45f3372f79389d9a7fe9a2aaa9702 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2-activate, org.label-schema.build-date=20201204, org.label-schema.name=CentOS Base Image, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD) 2021-03-15T22:30:23.270 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:23 smithi076 podman[222326]: 2021-03-15 22:30:23.247370485 +0000 UTC m=+1.099073572 container start 6be260ad6886df227ff2447783072409c7e45f3372f79389d9a7fe9a2aaa9702 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2-activate, org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0, ceph=True, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, GIT_BRANCH=HEAD, org.label-schema.build-date=20201204, maintainer=Dimitri Savineau , org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.vendor=CentOS) 2021-03-15T22:30:23.270 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:23 smithi076 podman[222326]: 2021-03-15 22:30:23.247532128 +0000 UTC m=+1.099235238 container attach 6be260ad6886df227ff2447783072409c7e45f3372f79389d9a7fe9a2aaa9702 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2-activate, GIT_CLEAN=True, org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0, org.label-schema.build-date=20201204, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, maintainer=Dimitri Savineau ) 2021-03-15T22:30:23.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:23.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:23.329 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:23.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:23.430 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:23.452 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:23.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:23.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:23.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:23 smithi125 conmon[56739]: cluster 2021-03-15T22:30:22.225695+0000 mon.a (mon.0) 2021-03-15T22:30:23.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:23 smithi125 conmon[56739]: 1832 : cluster [DBG] osdmap e673: 8 total, 7 up, 8 in 2021-03-15T22:30:23.554 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:23.576 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:23.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:23.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:23.592 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:23 smithi076 bash[222148]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-03-15T22:30:23.593 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:23 smithi076 bash[222148]: Running command: /usr/bin/ceph-bluestore-tool --cluster=ceph prime-osd-dir --dev /dev/vg_nvme/lv_2 --path /var/lib/ceph/osd/ceph-2 --no-mon-config 2021-03-15T22:30:23.593 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:23 smithi076 bash[222148]: Running command: /usr/bin/ln -snf /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-2/block 2021-03-15T22:30:23.593 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:23 smithi076 bash[222148]: Running command: /usr/bin/chown -h ceph:ceph /var/lib/ceph/osd/ceph-2/block 2021-03-15T22:30:23.593 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:23 smithi076 bash[222148]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-1 2021-03-15T22:30:23.594 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:23 smithi076 bash[222148]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-03-15T22:30:23.594 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:23 smithi076 bash[222148]: --> ceph-volume lvm activate successful for osd ID: 2 2021-03-15T22:30:23.594 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:23 smithi076 podman[222326]: 2021-03-15 22:30:23.499126427 +0000 UTC m=+1.350829530 container died 6be260ad6886df227ff2447783072409c7e45f3372f79389d9a7fe9a2aaa9702 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2-activate) 2021-03-15T22:30:23.679 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:23.701 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:23.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:23.703 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:23.804 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:23.825 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:23.827 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:23.928 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:23.954 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:23.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:23.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:24.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:24.081 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:24.081 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:24.182 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:24.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:24.206 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:24.244 INFO:tasks.rados.rados.0.smithi076.stdout:2461: done (2 left) 2021-03-15T22:30:24.244 INFO:tasks.rados.rados.0.smithi076.stdout:2462: done (1 left) 2021-03-15T22:30:24.244 INFO:tasks.rados.rados.0.smithi076.stdout:2463: read oid 34 snap -1 2021-03-15T22:30:24.245 INFO:tasks.rados.rados.0.smithi076.stdout:2463: expect (ObjNum 764 snap 254 seq_num 764) 2021-03-15T22:30:24.245 INFO:tasks.rados.rados.0.smithi076.stdout:2464: delete oid 16 current snap is 282 2021-03-15T22:30:24.248 INFO:tasks.rados.rados.0.smithi076.stdout:2463: expect (ObjNum 764 snap 254 seq_num 764) 2021-03-15T22:30:24.308 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:24.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:24.331 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:24.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:24.374 INFO:tasks.rados.rados.0.smithi076.stdout:2463: done (2 left) 2021-03-15T22:30:24.374 INFO:tasks.rados.rados.0.smithi076.stdout:2464: done (1 left) 2021-03-15T22:30:24.374 INFO:tasks.rados.rados.0.smithi076.stdout:2465: rollback oid 25 current snap is 282 2021-03-15T22:30:24.375 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 262 2021-03-15T22:30:24.375 INFO:tasks.rados.rados.0.smithi076.stdout:2466: read oid 11 snap -1 2021-03-15T22:30:24.375 INFO:tasks.rados.rados.0.smithi076.stdout:2466: expect (ObjNum 813 snap 269 seq_num 813) 2021-03-15T22:30:24.375 INFO:tasks.rados.rados.0.smithi076.stdout:2467: write (excl) oid 21 current snap is 282 2021-03-15T22:30:24.376 INFO:tasks.rados.rados.0.smithi076.stdout:2467: seq_num 849 ranges {669143=401584,1564977=426959} 2021-03-15T22:30:24.379 INFO:tasks.rados.rados.0.smithi076.stdout:2467: writing smithi07694878-21 from 669143 to 1070727 tid 1 2021-03-15T22:30:24.385 INFO:tasks.rados.rados.0.smithi076.stdout:2467: writing smithi07694878-21 from 1564977 to 1991936 tid 2 2021-03-15T22:30:24.385 INFO:tasks.rados.rados.0.smithi076.stdout:2468: rollback oid 43 current snap is 282 2021-03-15T22:30:24.385 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 43 to 196 2021-03-15T22:30:24.385 INFO:tasks.rados.rados.0.smithi076.stdout:2466: expect (ObjNum 813 snap 269 seq_num 813) 2021-03-15T22:30:24.432 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:24.455 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:24.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:24.456 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:24.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:24 smithi076 conmon[56514]: cluster 2021-03-15T22:30:23.143583+0000 2021-03-15T22:30:24.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:24 smithi076 conmon[56514]: mgr.y (mgr.14144) 842 : cluster [DBG] pgmap v1154: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 488 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.8 MiB/s wr, 18 op/s; 57/1692 objects degraded (3.369%); 125/1692 objects misplaced (7.388%); 5.0 MiB/s, 16 objects/s recovering 2021-03-15T22:30:24.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:24 smithi076 conmon[56514]: cluster 2021-03-15T22:30:23.235966+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2021-03-15T22:30:24.469 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:24 smithi076 conmon[62156]: cluster 2021-03-15T22:30:23.143583+0000 mgr.y (mgr.14144 2021-03-15T22:30:24.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:24 smithi076 conmon[62156]: ) 842 : cluster [DBG] pgmap v1154: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 488 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.8 MiB/s wr, 18 op/s; 57/1692 objects degraded (3.369%); 125/1692 objects misplaced (7.388%); 5.0 MiB/s, 16 objects/s recovering 2021-03-15T22:30:24.470 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:24 smithi076 conmon[62156]: cluster 2021-03-15T22:30:23.235966+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2021-03-15T22:30:24.507 INFO:tasks.rados.rados.0.smithi076.stdout:2465: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:30:24.508 INFO:tasks.rados.rados.0.smithi076.stdout:2465: finishing rollback tid 2 to smithi07694878-25 2021-03-15T22:30:24.508 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 1588 (ObjNum 783 snap 260 seq_num 783) dirty exists 2021-03-15T22:30:24.509 INFO:tasks.rados.rados.0.smithi076.stdout:2467: finishing write tid 1 to smithi07694878-21 2021-03-15T22:30:24.509 INFO:tasks.rados.rados.0.smithi076.stdout:2465: done (4 left) 2021-03-15T22:30:24.510 INFO:tasks.rados.rados.0.smithi076.stdout:2466: done (3 left) 2021-03-15T22:30:24.510 INFO:tasks.rados.rados.0.smithi076.stdout:2469: write oid 26 current snap is 282 2021-03-15T22:30:24.510 INFO:tasks.rados.rados.0.smithi076.stdout:2468: finishing rollback tid 0 to smithi07694878-43 2021-03-15T22:30:24.511 INFO:tasks.rados.rados.0.smithi076.stdout:2468: finishing rollback tid 1 to smithi07694878-43 2021-03-15T22:30:24.511 INFO:tasks.rados.rados.0.smithi076.stdout:2468: finishing rollback tid 2 to smithi07694878-43 2021-03-15T22:30:24.512 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 1220 (ObjNum 613 snap 190 seq_num 613) dirty exists 2021-03-15T22:30:24.512 INFO:tasks.rados.rados.0.smithi076.stdout:2467: finishing write tid 2 to smithi07694878-21 2021-03-15T22:30:24.513 INFO:tasks.rados.rados.0.smithi076.stdout:2469: seq_num 850 ranges {685950=476833,1623459=570617} 2021-03-15T22:30:24.513 INFO:tasks.rados.rados.0.smithi076.stdout:2469: writing smithi07694878-26 from 685950 to 1162783 tid 1 2021-03-15T22:30:24.519 INFO:tasks.rados.rados.0.smithi076.stdout:2469: writing smithi07694878-26 from 1623459 to 2194076 tid 2 2021-03-15T22:30:24.520 INFO:tasks.rados.rados.0.smithi076.stdout:2468: done (3 left) 2021-03-15T22:30:24.520 INFO:tasks.rados.rados.0.smithi076.stdout:2470: copy_from oid 15 from oid 32 current snap is 282 2021-03-15T22:30:24.521 INFO:tasks.rados.rados.0.smithi076.stdout:2467: finishing write tid 3 to smithi07694878-21 2021-03-15T22:30:24.521 INFO:tasks.rados.rados.0.smithi076.stdout:2467: finishing write tid 4 to smithi07694878-21 2021-03-15T22:30:24.521 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 1966 (ObjNum 849 snap 282 seq_num 849) dirty exists 2021-03-15T22:30:24.522 INFO:tasks.rados.rados.0.smithi076.stdout:2467: left oid 21 (ObjNum 849 snap 282 seq_num 849) 2021-03-15T22:30:24.522 INFO:tasks.rados.rados.0.smithi076.stdout:2467: done (3 left) 2021-03-15T22:30:24.523 INFO:tasks.rados.rados.0.smithi076.stdout:2471: write (excl) oid 13 current snap is 282 2021-03-15T22:30:24.523 INFO:tasks.rados.rados.0.smithi076.stdout:2471: seq_num 851 ranges {702757=552082,1681941=430628} 2021-03-15T22:30:24.525 INFO:tasks.rados.rados.0.smithi076.stdout:2471: writing smithi07694878-13 from 702757 to 1254839 tid 1 2021-03-15T22:30:24.531 INFO:tasks.rados.rados.0.smithi076.stdout:2471: writing smithi07694878-13 from 1681941 to 2112569 tid 2 2021-03-15T22:30:24.532 INFO:tasks.rados.rados.0.smithi076.stdout:2472: snap_remove snap 262 2021-03-15T22:30:24.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:24 smithi125 conmon[56739]: cluster 2021-03-15T22:30:23.143583+0000 mgr.y (mgr.14144) 842 : cluster [DBG] pgmap v1154: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 488 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.8 MiB/s wr, 18 op/s; 57/1692 objects degraded (3.369%); 125/1692 objects misplaced (7.388%); 5.0 MiB/s, 16 objects/s recovering 2021-03-15T22:30:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:24 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:30:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:24 smithi125 conmon[56739]: 30:23.235966+0000 mon.a (mon.0) 1833 : cluster [DBG] osdmap e674: 8 total, 7 up, 8 in 2021-03-15T22:30:24.542 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:24.557 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:24.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:24.584 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:24.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:24.686 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:24.709 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:24.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:24.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:24.767 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:24 smithi076 podman[222326]: 2021-03-15 22:30:24.534014664 +0000 UTC m=+2.385717744 container remove 6be260ad6886df227ff2447783072409c7e45f3372f79389d9a7fe9a2aaa9702 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2-activate, org.label-schema.license=GPLv2, org.label-schema.build-date=20201204, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.name=CentOS Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.18.0, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, maintainer=Dimitri Savineau ) 2021-03-15T22:30:24.812 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:24.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:24.837 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:24.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:24.939 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:24.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:24.964 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:24.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:25.066 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:25.088 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:25.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:25.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:25.191 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:25.216 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:25.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:25.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:25.246 INFO:tasks.rados.rados.0.smithi076.stdout:2472: done (4 left) 2021-03-15T22:30:25.246 INFO:tasks.rados.rados.0.smithi076.stdout:2473: snap_remove snap 272 2021-03-15T22:30:25.250 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:30:25.318 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:25.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:25 smithi076 conmon[56514]: cluster 2021-03-15T22:30:24.241641+0000 2021-03-15T22:30:25.324 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:25 smithi076 conmon[56514]: mon.a (mon.0) 1834 : cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2021-03-15T22:30:25.325 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:25 smithi076 conmon[56514]: audit 2021-03-15T22:30:24.508299+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.76:0/2049795706' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:30:25.325 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:25 smithi076 conmon[56514]: cluster 2021-03-15T22:30:25.203772+0000 mon.a (mon.0) 1836 : cluster [WRN] Health check update: Degraded data redundancy: 57/1692 objects degraded (3.369%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:25.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:25 smithi076 conmon[62156]: cluster 2021-03-15T22:30:24.241641 2021-03-15T22:30:25.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:25 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1834 : cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2021-03-15T22:30:25.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:25 smithi076 conmon[62156]: audit 2021-03-15T22:30:24.508299+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.76:0/2049795706' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:30:25.326 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:25 smithi076 conmon[62156]: cluster 2021-03-15T22:30:25.203772+0000 mon.a (mon.0) 1836 : cluster [WRN] Health check update: Degraded data redundancy: 57/1692 objects degraded (3.369%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:25.343 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:25.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:25.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:25.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:25.468 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:25.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:25.470 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:25.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:25 smithi125 conmon[56739]: cluster 2021-03-15T22:30:24.241641+0000 mon.a (mon.0) 1834 : 2021-03-15T22:30:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:25 smithi125 conmon[56739]: cluster [DBG] osdmap e675: 8 total, 7 up, 8 in 2021-03-15T22:30:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:25 smithi125 conmon[56739]: audit 2021-03-15T22:30:24.508299+0000 mon.a (mon.0) 1835 : audit [INF] from='client.? 172.21.15.76:0/2049795706' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:30:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:25 smithi125 conmon[56739]: cluster 2021-03-15T22:30:25.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:25 smithi125 conmon[56739]: 2021-03-15T22:30:25.203772+0000 mon.a (mon.0) 1836 : cluster [WRN] Health check update: Degraded data redundancy: 57/1692 objects degraded (3.369%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:25.571 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:25.577 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:25 smithi076 podman[223064]: 2021-03-15 22:30:25.358875138 +0000 UTC m=+0.387302509 container create 00fce23501cc33ea48b399bf2ec4d0f7b2809866b1fc2b0f40850489a7b81432 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Dimitri Savineau , GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Base Image, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, org.label-schema.build-date=20201204) 2021-03-15T22:30:25.577 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:25 smithi076 podman[223064]: 2021-03-15 22:30:25.533935679 +0000 UTC m=+0.562363135 container init 00fce23501cc33ea48b399bf2ec4d0f7b2809866b1fc2b0f40850489a7b81432 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2, org.label-schema.name=CentOS Base Image, io.buildah.version=1.18.0, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.build-date=20201204, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Dimitri Savineau , ceph=True, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2021-03-15T22:30:25.577 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:25 smithi076 conmon[223218]: debug 2021-03-15T22:30:25.563+0000 7f9ec11c4700 1 -- 172.21.15.76:0/597778248 <== mon.1 v2:172.21.15.76:3301/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bdfae344e0 con 0x55bdfadfd400 2021-03-15T22:30:25.577 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:25 smithi076 podman[223064]: 2021-03-15 22:30:25.575512467 +0000 UTC m=+0.603939835 container start 00fce23501cc33ea48b399bf2ec4d0f7b2809866b1fc2b0f40850489a7b81432 (image=quay.ceph.io/ceph-ci/ceph@sha256:38274a03244dc68d38794e835023fc9be83edbf6e10e4487c12fff8ab9ef69bd, name=ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2, GIT_CLEAN=True, GIT_COMMIT=e2d54e961ffedf75484f4922f255352477e30cb4, org.label-schema.name=CentOS Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.18.0, maintainer=Dimitri Savineau , org.label-schema.vendor=CentOS, RELEASE=wip-sage-testing-2021-03-15-1507-b658018, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20201204) 2021-03-15T22:30:25.582 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-15T22:30:25.593 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:25.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:25.595 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:25.697 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:25.719 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:25.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:25.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:25.821 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:25.843 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:25.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:25.845 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:25.946 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:25.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:25.971 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:25.972 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:26.017 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:25 smithi076 bash[222148]: 00fce23501cc33ea48b399bf2ec4d0f7b2809866b1fc2b0f40850489a7b81432 2021-03-15T22:30:26.018 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:25 smithi076 systemd[1]: Started Ceph osd.2 for 8a572802-85db-11eb-9091-001a4aab830c. 2021-03-15T22:30:26.073 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:26.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:26.097 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:26.098 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:26.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:26.221 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:26.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:26.223 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:26.257 INFO:tasks.rados.rados.0.smithi076.stdout:2473: done (4 left) 2021-03-15T22:30:26.258 INFO:tasks.rados.rados.0.smithi076.stdout:2474: delete oid 35 current snap is 282 2021-03-15T22:30:26.258 INFO:tasks.rados.rados.0.smithi076.stdout:2471: finishing write tid 1 to smithi07694878-13 2021-03-15T22:30:26.258 INFO:tasks.rados.rados.0.smithi076.stdout:2470: finishing copy_from to smithi07694878-15 2021-03-15T22:30:26.258 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 1540 (ObjNum 767 snap 255 seq_num 767) dirty exists 2021-03-15T22:30:26.259 INFO:tasks.rados.rados.0.smithi076.stdout:2470: finishing copy_from racing read to smithi07694878-15 2021-03-15T22:30:26.260 INFO:tasks.rados.rados.0.smithi076.stdout:2471: finishing write tid 2 to smithi07694878-13 2021-03-15T22:30:26.260 INFO:tasks.rados.rados.0.smithi076.stdout:2471: finishing write tid 3 to smithi07694878-13 2021-03-15T22:30:26.260 INFO:tasks.rados.rados.0.smithi076.stdout:2471: finishing write tid 4 to smithi07694878-13 2021-03-15T22:30:26.260 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1554 (ObjNum 851 snap 282 seq_num 851) dirty exists 2021-03-15T22:30:26.261 INFO:tasks.rados.rados.0.smithi076.stdout:2471: left oid 13 (ObjNum 851 snap 282 seq_num 851) 2021-03-15T22:30:26.263 INFO:tasks.rados.rados.0.smithi076.stdout:2470: done (4 left) 2021-03-15T22:30:26.263 INFO:tasks.rados.rados.0.smithi076.stdout:2471: done (3 left) 2021-03-15T22:30:26.263 INFO:tasks.rados.rados.0.smithi076.stdout:2474: done (2 left) 2021-03-15T22:30:26.264 INFO:tasks.rados.rados.0.smithi076.stdout:2475: read oid 11 snap -1 2021-03-15T22:30:26.264 INFO:tasks.rados.rados.0.smithi076.stdout:2475: expect (ObjNum 813 snap 269 seq_num 813) 2021-03-15T22:30:26.264 INFO:tasks.rados.rados.0.smithi076.stdout:2476: delete oid 24 current snap is 282 2021-03-15T22:30:26.265 INFO:tasks.rados.rados.0.smithi076.stdout:2476: done (3 left) 2021-03-15T22:30:26.265 INFO:tasks.rados.rados.0.smithi076.stdout:2477: write (excl) oid 33 current snap is 282 2021-03-15T22:30:26.266 INFO:tasks.rados.rados.0.smithi076.stdout:2477: seq_num 854 ranges {753178=494182,1857387=495791} 2021-03-15T22:30:26.272 INFO:tasks.rados.rados.0.smithi076.stdout:2477: writing smithi07694878-33 from 753178 to 1247360 tid 1 2021-03-15T22:30:26.279 INFO:tasks.rados.rados.0.smithi076.stdout:2477: writing smithi07694878-33 from 1857387 to 2353178 tid 2 2021-03-15T22:30:26.280 INFO:tasks.rados.rados.0.smithi076.stdout:2478: read oid 50 snap -1 2021-03-15T22:30:26.280 INFO:tasks.rados.rados.0.smithi076.stdout:2478: expect (ObjNum 598 snap 183 seq_num 598) 2021-03-15T22:30:26.281 INFO:tasks.rados.rados.0.smithi076.stdout:2479: write (excl) oid 49 current snap is 282 2021-03-15T22:30:26.281 INFO:tasks.rados.rados.0.smithi076.stdout:2479: seq_num 855 ranges {769985=569431,1915869=437966} 2021-03-15T22:30:26.286 INFO:tasks.rados.rados.0.smithi076.stdout:2479: writing smithi07694878-49 from 769985 to 1339416 tid 1 2021-03-15T22:30:26.291 INFO:tasks.rados.rados.0.smithi076.stdout:2479: writing smithi07694878-49 from 1915869 to 2353835 tid 2 2021-03-15T22:30:26.292 INFO:tasks.rados.rados.0.smithi076.stdout:2480: write oid 30 current snap is 282 2021-03-15T22:30:26.292 INFO:tasks.rados.rados.0.smithi076.stdout:2480: seq_num 856 ranges {786792=644680,1857998=528794} 2021-03-15T22:30:26.299 INFO:tasks.rados.rados.0.smithi076.stdout:2480: writing smithi07694878-30 from 786792 to 1431472 tid 1 2021-03-15T22:30:26.305 INFO:tasks.rados.rados.0.smithi076.stdout:2480: writing smithi07694878-30 from 1857998 to 2386792 tid 2 2021-03-15T22:30:26.305 INFO:tasks.rados.rados.0.smithi076.stdout:2481: write oid 7 current snap is 282 2021-03-15T22:30:26.305 INFO:tasks.rados.rados.0.smithi076.stdout:2475: expect (ObjNum 813 snap 269 seq_num 813) 2021-03-15T22:30:26.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:26.346 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:26.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:26.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:26.424 INFO:tasks.rados.rados.0.smithi076.stdout:2477: finishing write tid 1 to smithi07694878-33 2021-03-15T22:30:26.424 INFO:tasks.rados.rados.0.smithi076.stdout:2481: seq_num 857 ranges {403599=719929,1916480=441635} 2021-03-15T22:30:26.430 INFO:tasks.rados.rados.0.smithi076.stdout:2481: writing smithi07694878-7 from 403599 to 1123528 tid 1 2021-03-15T22:30:26.437 INFO:tasks.rados.rados.0.smithi076.stdout:2481: writing smithi07694878-7 from 1916480 to 2358115 tid 2 2021-03-15T22:30:26.438 INFO:tasks.rados.rados.0.smithi076.stdout:2475: done (7 left) 2021-03-15T22:30:26.438 INFO:tasks.rados.rados.0.smithi076.stdout:2482: rollback oid 28 current snap is 282 2021-03-15T22:30:26.438 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 196 2021-03-15T22:30:26.438 INFO:tasks.rados.rados.0.smithi076.stdout:2477: finishing write tid 2 to smithi07694878-33 2021-03-15T22:30:26.439 INFO:tasks.rados.rados.0.smithi076.stdout:2477: finishing write tid 3 to smithi07694878-33 2021-03-15T22:30:26.439 INFO:tasks.rados.rados.0.smithi076.stdout:2477: finishing write tid 4 to smithi07694878-33 2021-03-15T22:30:26.439 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 1744 (ObjNum 854 snap 282 seq_num 854) dirty exists 2021-03-15T22:30:26.439 INFO:tasks.rados.rados.0.smithi076.stdout:2477: left oid 33 (ObjNum 854 snap 282 seq_num 854) 2021-03-15T22:30:26.440 INFO:tasks.rados.rados.0.smithi076.stdout:2477: done (7 left) 2021-03-15T22:30:26.440 INFO:tasks.rados.rados.0.smithi076.stdout:2483: snap_create 2021-03-15T22:30:26.440 INFO:tasks.rados.rados.0.smithi076.stdout:2479: finishing write tid 1 to smithi07694878-49 2021-03-15T22:30:26.440 INFO:tasks.rados.rados.0.smithi076.stdout:2479: finishing write tid 2 to smithi07694878-49 2021-03-15T22:30:26.440 INFO:tasks.rados.rados.0.smithi076.stdout:2479: finishing write tid 3 to smithi07694878-49 2021-03-15T22:30:26.441 INFO:tasks.rados.rados.0.smithi076.stdout:2479: finishing write tid 4 to smithi07694878-49 2021-03-15T22:30:26.441 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 1851 (ObjNum 855 snap 282 seq_num 855) dirty exists 2021-03-15T22:30:26.441 INFO:tasks.rados.rados.0.smithi076.stdout:2479: left oid 49 (ObjNum 855 snap 282 seq_num 855) 2021-03-15T22:30:26.441 INFO:tasks.rados.rados.0.smithi076.stdout:2480: finishing write tid 1 to smithi07694878-30 2021-03-15T22:30:26.442 INFO:tasks.rados.rados.0.smithi076.stdout:2480: finishing write tid 2 to smithi07694878-30 2021-03-15T22:30:26.442 INFO:tasks.rados.rados.0.smithi076.stdout:2480: finishing write tid 3 to smithi07694878-30 2021-03-15T22:30:26.442 INFO:tasks.rados.rados.0.smithi076.stdout:2480: finishing write tid 4 to smithi07694878-30 2021-03-15T22:30:26.442 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 1855 (ObjNum 856 snap 282 seq_num 856) dirty exists 2021-03-15T22:30:26.443 INFO:tasks.rados.rados.0.smithi076.stdout:2480: left oid 30 (ObjNum 856 snap 282 seq_num 856) 2021-03-15T22:30:26.443 INFO:tasks.rados.rados.0.smithi076.stdout:2481: finishing write tid 1 to smithi07694878-7 2021-03-15T22:30:26.444 INFO:tasks.rados.rados.0.smithi076.stdout:2481: finishing write tid 2 to smithi07694878-7 2021-03-15T22:30:26.444 INFO:tasks.rados.rados.0.smithi076.stdout:2481: finishing write tid 3 to smithi07694878-7 2021-03-15T22:30:26.444 INFO:tasks.rados.rados.0.smithi076.stdout:2482: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:30:26.445 INFO:tasks.rados.rados.0.smithi076.stdout:2481: finishing write tid 4 to smithi07694878-7 2021-03-15T22:30:26.446 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 1053 (ObjNum 857 snap 282 seq_num 857) dirty exists 2021-03-15T22:30:26.446 INFO:tasks.rados.rados.0.smithi076.stdout:2481: left oid 7 (ObjNum 857 snap 282 seq_num 857) 2021-03-15T22:30:26.446 INFO:tasks.rados.rados.0.smithi076.stdout:2482: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:30:26.446 INFO:tasks.rados.rados.0.smithi076.stdout:2482: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:30:26.447 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 1982 (ObjNum 620 snap 193 seq_num 620) dirty exists 2021-03-15T22:30:26.448 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:26.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:26.473 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:26.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:26.517 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[223218]: debug 2021-03-15T22:30:26.077+0000 7f9ec8c4f080 -1 Falling back to public interface 2021-03-15T22:30:26.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[56514]: cluster 2021-03-15T22:30:25.144153+0000 mgr.y (mgr.14144) 843 : cluster [DBG] pgmap v1157: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 493 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 890 KiB/s wr, 7 op/s; 57/1695 objects degraded (3.363%); 125/1695 objects misplaced (7.375%); 5.0 MiB/s, 16 objects/s recovering 2021-03-15T22:30:26.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[56514]: cluster 2021-03-15T22:30:25.165728+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:26.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[56514]: cluster 2021-03-15T22:30:25.166407+0000 osd.5 (osd.5) 11 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:26.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[56514]: cluster 2021-03-15T22:30:25.236993+0000 mon.a (mon.0) 1837 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:26.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[56514]: audit 2021 2021-03-15T22:30:26.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[56514]: -03-15T22:30:25.244357+0000 mon.a (mon.0) 1838 : audit [INF] from='client.? 172.21.15.76:0/2049795706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:30:26.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:30:26.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[56514]: 03-15T22:30:25 2021-03-15T22:30:26.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[56514]: .244412+0000 mon.a (mon.0 2021-03-15T22:30:26.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[56514]: ) 1839 : cluster [DBG] osdmap e676: 8 total, 7 up, 8 in 2021-03-15T22:30:26.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: cluster 2021-03-15T22:30:25 2021-03-15T22:30:26.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: .144153+0000 mgr.y (mgr.14144) 843 : cluster [DBG] pgmap v1157: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 493 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 890 KiB/s wr, 7 op/s; 57/1695 objects degraded (3.363%); 125/1695 objects misplaced (7.375%); 5.0 MiB/s, 16 objects/s recovering 2021-03-15T22:30:26.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:30:26.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: 03-15T22:30:25. 2021-03-15T22:30:26.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: 165728+0000 osd.5 (osd.5) 2021-03-15T22:30:26.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: 10 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:26.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:30:26.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: -03-15T22:30:25. 2021-03-15T22:30:26.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: 166407+0000 osd.5 (osd.5 2021-03-15T22:30:26.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: ) 11 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:26.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:30:26.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: 03-15T22:30:25. 2021-03-15T22:30:26.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: 236993+0000 mon.a (mon.0) 2021-03-15T22:30:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: 1837 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: audit 2021-03-15T22:30:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: 2021-03-15T22:30 2021-03-15T22:30:26.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: :25.244357+0000 mon.a (mon. 2021-03-15T22:30:26.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: 0) 1838 : audit [INF] 2021-03-15T22:30:26.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2049795706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:30:26.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:30:26.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: -03-15T22:30:25.244412+0000 mon.a 2021-03-15T22:30:26.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: (mon.0) 1839 : cluster 2021-03-15T22:30:26.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:26 smithi076 conmon[62156]: [DBG] osdmap e676: 8 total, 7 up, 8 in 2021-03-15T22:30:26.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: cluster 2021-03-15T22:30:25.144153+0000 mgr.y (mgr.14144) 843 : cluster [DBG] pgmap v1157: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 493 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 890 KiB/s wr, 7 op/s; 57/1695 objects degraded (3.363%); 125/1695 objects misplaced (7.375%); 5.0 MiB/s, 16 objects/s recovering 2021-03-15T22:30:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: cluster 2021-03-15T22:30:25.165728+0000 osd.5 (osd.5) 10 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:30:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: -15T22:30:25.166407+0000 osd.5 (osd.5) 11 : cluster 2021-03-15T22:30:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: [DBG] purged_snaps scrub ok 2021-03-15T22:30:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:30:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: :30:25.236993+0000 mon.a (mon. 2021-03-15T22:30:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: 0) 1837 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: audit 2021-03-15T22:30:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: 2021-03-15 2021-03-15T22:30:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: T22:30:25.244357+0000 mon.a ( 2021-03-15T22:30:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: mon.0) 1838 : audit [INF] from='client.? 172.21.15.76:0/2049795706' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:30:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:30:26.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: -03-15T22:30:25.244412 2021-03-15T22:30:26.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1839 : cluster 2021-03-15T22:30:26.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:26 smithi125 conmon[56739]: [DBG] osdmap e676: 8 total, 7 up, 8 in 2021-03-15T22:30:26.575 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:26.597 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:26.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:26.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:26.699 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:26.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:26.722 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:26.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:26.824 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:26.850 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:26.851 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:26.952 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:26.975 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:26.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:26.976 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:26.980 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:27.037 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:30:27.037 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:30:27.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:30:27.038 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:30:27.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.038 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDScrubReserve(6.1b REQUEST e194)", 2021-03-15T22:30:27.038 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:30:27.039 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 517.249418881, 2021-03-15T22:30:27.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.061386018000000001, 2021-03-15T22:30:27.039 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.039 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:27.040 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:30:27.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783092+0000", 2021-03-15T22:30:27.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6899999999999999e-06 2021-03-15T22:30:27.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783093+0000", 2021-03-15T22:30:27.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5640000000000001e-06 2021-03-15T22:30:27.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783097+0000", 2021-03-15T22:30:27.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7742e-05 2021-03-15T22:30:27.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783125+0000", 2021-03-15T22:30:27.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3793e-05 2021-03-15T22:30:27.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783138+0000", 2021-03-15T22:30:27.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8529999999999993e-05 2021-03-15T22:30:27.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783237+0000", 2021-03-15T22:30:27.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.241e-05 2021-03-15T22:30:27.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:30:27.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.783249+0000", 2021-03-15T22:30:27.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013943790000000001 2021-03-15T22:30:27.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.784644+0000", 2021-03-15T22:30:27.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2398000000000001e-05 2021-03-15T22:30:27.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.784656+0000", 2021-03-15T22:30:27.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.059821512 2021-03-15T22:30:27.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:21:49.844478+0000", 2021-03-15T22:30:27.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.061386018000000001 2021-03-15T22:30:27.052 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.053 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.053 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.054 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2768 7.b 7:d10ba14d:::smithi07694878-43:head [write 1689059~741123] snapc d2=[d2,d1,cf] ondisk+write+known_if_redirected e310)", 2021-03-15T22:30:27.054 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:30:27.054 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 401.92742051099998, 2021-03-15T22:30:27.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078094859000000003, 2021-03-15T22:30:27.055 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.055 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.055 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.055 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.056 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.056 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2768 2021-03-15T22:30:27.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.056 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:30:27.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.058 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:30:27.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033110399999999998 2021-03-15T22:30:27.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105421+0000", 2021-03-15T22:30:27.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4809999999999998e-06 2021-03-15T22:30:27.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105425+0000", 2021-03-15T22:30:27.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7363999999999998e-05 2021-03-15T22:30:27.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105442+0000", 2021-03-15T22:30:27.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9480000000000002e-06 2021-03-15T22:30:27.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105449+0000", 2021-03-15T22:30:27.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6847000000000001e-05 2021-03-15T22:30:27.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.064 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105496+0000", 2021-03-15T22:30:27.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8231000000000003e-05 2021-03-15T22:30:27.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.065 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105584+0000", 2021-03-15T22:30:27.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000198369 2021-03-15T22:30:27.066 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:30:27.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105782+0000", 2021-03-15T22:30:27.067 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003007292 2021-03-15T22:30:27.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108790+0000", 2021-03-15T22:30:27.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000200027 2021-03-15T22:30:27.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108990+0000", 2021-03-15T22:30:27.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074107929000000003 2021-03-15T22:30:27.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.069 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:27.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183098+0000", 2021-03-15T22:30:27.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4002000000000002e-05 2021-03-15T22:30:27.070 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183152+0000", 2021-03-15T22:30:27.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3265000000000001e-05 2021-03-15T22:30:27.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183185+0000", 2021-03-15T22:30:27.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078094859000000003 2021-03-15T22:30:27.073 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.074 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.074 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.075 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2769 7.b 7:d10ba14d:::smithi07694878-43:head [setxattr _header (58) in=7b,truncate 2588344] snapc d2=[d2,d1,cf] ondisk+write+known_if_redirected e310)", 2021-03-15T22:30:27.075 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:30:27.075 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 401.92704948300002, 2021-03-15T22:30:27.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077795487999999996, 2021-03-15T22:30:27.076 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.076 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.076 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.076 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.076 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.077 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2769 2021-03-15T22:30:27.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.077 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:30:27.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:30:27.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2639999999999998e-06 2021-03-15T22:30:27.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105463+0000", 2021-03-15T22:30:27.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.426e-06 2021-03-15T22:30:27.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.081 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.081 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105465+0000", 2021-03-15T22:30:27.081 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9349999999999999e-06 2021-03-15T22:30:27.082 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.082 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.082 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105471+0000", 2021-03-15T22:30:27.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2420000000000002e-06 2021-03-15T22:30:27.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105475+0000", 2021-03-15T22:30:27.084 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001413924 2021-03-15T22:30:27.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.106889+0000", 2021-03-15T22:30:27.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5030000000000003e-05 2021-03-15T22:30:27.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.106984+0000", 2021-03-15T22:30:27.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014550399999999999 2021-03-15T22:30:27.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:30:27.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107129+0000", 2021-03-15T22:30:27.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001729529 2021-03-15T22:30:27.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.088 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108859+0000", 2021-03-15T22:30:27.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023022799999999999 2021-03-15T22:30:27.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.109089+0000", 2021-03-15T22:30:27.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074107424000000005 2021-03-15T22:30:27.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:27.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183197+0000", 2021-03-15T22:30:27.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7657e-05 2021-03-15T22:30:27.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183224+0000", 2021-03-15T22:30:27.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2325e-05 2021-03-15T22:30:27.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183257+0000", 2021-03-15T22:30:27.094 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077795487999999996 2021-03-15T22:30:27.094 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.094 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.094 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.095 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2770 7.b 7:d10ba14d:::smithi07694878-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e310)", 2021-03-15T22:30:27.095 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:30:27.096 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 401.92701927500002, 2021-03-15T22:30:27.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077923611000000004, 2021-03-15T22:30:27.096 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.096 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:27.097 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.097 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.097 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.097 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2770 2021-03-15T22:30:27.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.098 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.098 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:30:27.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:30:27.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3200000000000001e-06 2021-03-15T22:30:27.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105493+0000", 2021-03-15T22:30:27.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3960000000000001e-06 2021-03-15T22:30:27.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105494+0000", 2021-03-15T22:30:27.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6490000000000002e-06 2021-03-15T22:30:27.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105501+0000", 2021-03-15T22:30:27.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9989999999999999e-06 2021-03-15T22:30:27.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.104 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105506+0000", 2021-03-15T22:30:27.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019212770000000001 2021-03-15T22:30:27.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107427+0000", 2021-03-15T22:30:27.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8102000000000005e-05 2021-03-15T22:30:27.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:27.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107515+0000", 2021-03-15T22:30:27.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075777318999999996 2021-03-15T22:30:27.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183292+0000", 2021-03-15T22:30:27.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9417999999999999e-05 2021-03-15T22:30:27.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.108 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183352+0000", 2021-03-15T22:30:27.109 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3131000000000003e-05 2021-03-15T22:30:27.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183415+0000", 2021-03-15T22:30:27.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077923611000000004 2021-03-15T22:30:27.110 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.110 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.110 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.111 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3010 7.b 7:d980be1f:::smithi07694878-23:head [write 726168~704991,stat] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:30:27.111 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:30:27.112 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 383.945997697, 2021-03-15T22:30:27.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0570805230000002, 2021-03-15T22:30:27.112 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.112 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.112 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.113 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.113 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.113 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3010 2021-03-15T22:30:27.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.114 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:30:27.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:30:27.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034716800000000002 2021-03-15T22:30:27.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086860+0000", 2021-03-15T22:30:27.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3299999999999999e-06 2021-03-15T22:30:27.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086863+0000", 2021-03-15T22:30:27.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1489e-05 2021-03-15T22:30:27.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086875+0000", 2021-03-15T22:30:27.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.605e-06 2021-03-15T22:30:27.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.120 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086882+0000", 2021-03-15T22:30:27.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5296999999999997e-05 2021-03-15T22:30:27.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.121 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.122 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.122 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086928+0000", 2021-03-15T22:30:27.122 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2879999999999998e-06 2021-03-15T22:30:27.122 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:27.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086937+0000", 2021-03-15T22:30:27.123 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99891912599999999 2021-03-15T22:30:27.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085856+0000", 2021-03-15T22:30:27.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1169999999999993e-06 2021-03-15T22:30:27.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:27.125 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085865+0000", 2021-03-15T22:30:27.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0140395639999999 2021-03-15T22:30:27.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.099905+0000", 2021-03-15T22:30:27.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022436600000000001 2021-03-15T22:30:27.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.128 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.100129+0000", 2021-03-15T22:30:27.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00043622700000000002 2021-03-15T22:30:27.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:30:27.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.100565+0000", 2021-03-15T22:30:27.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024326999999999999 2021-03-15T22:30:27.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:27.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102998+0000", 2021-03-15T22:30:27.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002982012 2021-03-15T22:30:27.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.105980+0000", 2021-03-15T22:30:27.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037236714999999997 2021-03-15T22:30:27.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.133 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143217+0000", 2021-03-15T22:30:27.133 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147832 2021-03-15T22:30:27.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.134 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.134 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143365+0000", 2021-03-15T22:30:27.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022868699999999999 2021-03-15T22:30:27.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143593+0000", 2021-03-15T22:30:27.135 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0570805230000002 2021-03-15T22:30:27.135 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.136 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.136 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.137 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3011 7.b 7:d980be1f:::smithi07694878-23:head [write 1895071~683474] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:30:27.137 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:30:27.137 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 383.93971745099998, 2021-03-15T22:30:27.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.09731113, 2021-03-15T22:30:27.137 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.138 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.138 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.138 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.138 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.139 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3011 2021-03-15T22:30:27.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.139 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.139 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:30:27.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:30:27.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027883500000000001 2021-03-15T22:30:27.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093072+0000", 2021-03-15T22:30:27.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.809e-06 2021-03-15T22:30:27.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.142 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.143 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093077+0000", 2021-03-15T22:30:27.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9491e-05 2021-03-15T22:30:27.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093096+0000", 2021-03-15T22:30:27.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.126e-06 2021-03-15T22:30:27.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093103+0000", 2021-03-15T22:30:27.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4002000000000001e-05 2021-03-15T22:30:27.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.146 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.146 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093167+0000", 2021-03-15T22:30:27.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0259999999999996e-06 2021-03-15T22:30:27.147 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.147 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:27.147 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093176+0000", 2021-03-15T22:30:27.148 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99270957599999998 2021-03-15T22:30:27.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.148 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.149 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085886+0000", 2021-03-15T22:30:27.149 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9440000000000002e-06 2021-03-15T22:30:27.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:27.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085892+0000", 2021-03-15T22:30:27.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.016173532 2021-03-15T22:30:27.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.151 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102065+0000", 2021-03-15T22:30:27.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000141037 2021-03-15T22:30:27.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:27.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102207+0000", 2021-03-15T22:30:27.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041447062 2021-03-15T22:30:27.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143654+0000", 2021-03-15T22:30:27.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014246199999999999 2021-03-15T22:30:27.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143796+0000", 2021-03-15T22:30:27.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000272444 2021-03-15T22:30:27.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:30:27.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.144068+0000", 2021-03-15T22:30:27.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018956929999999999 2021-03-15T22:30:27.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145964+0000", 2021-03-15T22:30:27.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041997686999999999 2021-03-15T22:30:27.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:27.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.187962+0000", 2021-03-15T22:30:27.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019156150000000001 2021-03-15T22:30:27.158 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.159 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.159 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.189877+0000", 2021-03-15T22:30:27.159 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8467999999999995e-05 2021-03-15T22:30:27.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.160 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.160 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.189956+0000", 2021-03-15T22:30:27.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014832100000000001 2021-03-15T22:30:27.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.161 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.161 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190104+0000", 2021-03-15T22:30:27.161 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.09731113 2021-03-15T22:30:27.162 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.162 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.162 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.163 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3012 7.b 7:d980be1f:::smithi07694878-23:head [setxattr _header (58) in=7b,truncate 3126168] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:30:27.163 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:30:27.163 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 383.93939461899998, 2021-03-15T22:30:27.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0972930829999998, 2021-03-15T22:30:27.164 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.164 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.164 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.164 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.164 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.165 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3012 2021-03-15T22:30:27.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.165 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:30:27.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:30:27.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.61e-06 2021-03-15T22:30:27.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093118+0000", 2021-03-15T22:30:27.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.234e-06 2021-03-15T22:30:27.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093119+0000", 2021-03-15T22:30:27.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7290000000000002e-06 2021-03-15T22:30:27.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093123+0000", 2021-03-15T22:30:27.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1649999999999999e-06 2021-03-15T22:30:27.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093127+0000", 2021-03-15T22:30:27.172 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6592e-05 2021-03-15T22:30:27.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.172 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.173 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093213+0000", 2021-03-15T22:30:27.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2364000000000001e-05 2021-03-15T22:30:27.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.173 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:27.174 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093226+0000", 2021-03-15T22:30:27.174 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99268559899999997 2021-03-15T22:30:27.174 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.175 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085911+0000", 2021-03-15T22:30:27.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1440000000000002e-06 2021-03-15T22:30:27.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:27.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085916+0000", 2021-03-15T22:30:27.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0164949109999999 2021-03-15T22:30:27.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102411+0000", 2021-03-15T22:30:27.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000116049 2021-03-15T22:30:27.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:27.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102527+0000", 2021-03-15T22:30:27.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042432780000000003 2021-03-15T22:30:27.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.179 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.144960+0000", 2021-03-15T22:30:27.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7323000000000003e-05 2021-03-15T22:30:27.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145057+0000", 2021-03-15T22:30:27.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018096599999999999 2021-03-15T22:30:27.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:30:27.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145238+0000", 2021-03-15T22:30:27.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077369500000000003 2021-03-15T22:30:27.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.146012+0000", 2021-03-15T22:30:27.183 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041960274999999998 2021-03-15T22:30:27.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:27.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.187972+0000", 2021-03-15T22:30:27.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021867610000000002 2021-03-15T22:30:27.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.185 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.185 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190159+0000", 2021-03-15T22:30:27.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9751000000000004e-05 2021-03-15T22:30:27.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.186 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.187 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190229+0000", 2021-03-15T22:30:27.187 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000180135 2021-03-15T22:30:27.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190409+0000", 2021-03-15T22:30:27.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0972930829999998 2021-03-15T22:30:27.188 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.188 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.189 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3013 7.b 7:d980be1f:::smithi07694878-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e328)", 2021-03-15T22:30:27.190 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:30:27.190 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 383.93936133099999, 2021-03-15T22:30:27.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.097631684, 2021-03-15T22:30:27.190 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.190 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:27.191 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.191 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.191 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.191 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3013 2021-03-15T22:30:27.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.192 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:30:27.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:30:27.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9600000000000008e-07 2021-03-15T22:30:27.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093150+0000", 2021-03-15T22:30:27.195 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.514e-06 2021-03-15T22:30:27.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093152+0000", 2021-03-15T22:30:27.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2985e-05 2021-03-15T22:30:27.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093165+0000", 2021-03-15T22:30:27.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4000000000000002e-06 2021-03-15T22:30:27.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.198 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.198 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093169+0000", 2021-03-15T22:30:27.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2490999999999993e-05 2021-03-15T22:30:27.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.199 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.199 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093252+0000", 2021-03-15T22:30:27.199 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7490000000000001e-06 2021-03-15T22:30:27.200 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.200 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:27.200 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093258+0000", 2021-03-15T22:30:27.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99267630200000001 2021-03-15T22:30:27.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.201 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085935+0000", 2021-03-15T22:30:27.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0580000000000003e-06 2021-03-15T22:30:27.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:30:27.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085941+0000", 2021-03-15T22:30:27.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.01663009 2021-03-15T22:30:27.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102571+0000", 2021-03-15T22:30:27.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012747 2021-03-15T22:30:27.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:27.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102698+0000", 2021-03-15T22:30:27.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042823333999999998 2021-03-15T22:30:27.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145522+0000", 2021-03-15T22:30:27.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7650000000000004e-05 2021-03-15T22:30:27.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:27.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145599+0000", 2021-03-15T22:30:27.207 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044843263000000001 2021-03-15T22:30:27.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190443+0000", 2021-03-15T22:30:27.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000117391 2021-03-15T22:30:27.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190560+0000", 2021-03-15T22:30:27.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000220991 2021-03-15T22:30:27.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.210 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190781+0000", 2021-03-15T22:30:27.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.097631684 2021-03-15T22:30:27.211 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.211 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.211 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.212 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:3668 7.4 e385/358 7:26337d05:::smithi07694878-20:head v 385'309, mlcod=377'306)", 2021-03-15T22:30:27.212 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:30:27.212 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 326.85412289499999, 2021-03-15T22:30:27.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054245017999999999, 2021-03-15T22:30:27.213 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.213 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.214 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:30:27.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:30:27.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00068176800000000002 2021-03-15T22:30:27.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179069+0000", 2021-03-15T22:30:27.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1430000000000001e-06 2021-03-15T22:30:27.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179077+0000", 2021-03-15T22:30:27.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9515000000000001e-05 2021-03-15T22:30:27.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179106+0000", 2021-03-15T22:30:27.219 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0193e-05 2021-03-15T22:30:27.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179116+0000", 2021-03-15T22:30:27.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8248000000000003e-05 2021-03-15T22:30:27.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179165+0000", 2021-03-15T22:30:27.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8050000000000001e-05 2021-03-15T22:30:27.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.222 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179193+0000", 2021-03-15T22:30:27.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053391015999999999 2021-03-15T22:30:27.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.223 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.232584+0000", 2021-03-15T22:30:27.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9085000000000002e-05 2021-03-15T22:30:27.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.225 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.225 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.232633+0000", 2021-03-15T22:30:27.225 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054245017999999999 2021-03-15T22:30:27.225 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.226 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.226 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.226 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4284 7.b 7:d4368cae:::smithi07694878-18:head [write 3203532~509848] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:30:27.227 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:30:27.227 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 266.86973236099999, 2021-03-15T22:30:27.227 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068311582999999995, 2021-03-15T22:30:27.227 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.228 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.228 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.228 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.228 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.228 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4284 2021-03-15T22:30:27.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.229 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.229 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.230 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:30:27.230 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.231 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.231 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:30:27.231 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015451700000000001 2021-03-15T22:30:27.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.232 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.232 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162933+0000", 2021-03-15T22:30:27.232 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.25e-06 2021-03-15T22:30:27.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.233 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.233 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162937+0000", 2021-03-15T22:30:27.233 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2824e-05 2021-03-15T22:30:27.234 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.234 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.234 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.234 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162960+0000", 2021-03-15T22:30:27.234 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.072e-06 2021-03-15T22:30:27.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.235 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.235 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162968+0000", 2021-03-15T22:30:27.236 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7661999999999999e-05 2021-03-15T22:30:27.236 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.236 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.237 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163016+0000", 2021-03-15T22:30:27.237 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108743 2021-03-15T22:30:27.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.237 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.238 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163124+0000", 2021-03-15T22:30:27.238 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021732500000000001 2021-03-15T22:30:27.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.239 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:30:27.239 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163342+0000", 2021-03-15T22:30:27.239 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043853354999999997 2021-03-15T22:30:27.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.240 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.240 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.207195+0000", 2021-03-15T22:30:27.240 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055661469999999996 2021-03-15T22:30:27.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.241 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.241 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.212761+0000", 2021-03-15T22:30:27.241 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018240684 2021-03-15T22:30:27.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.242 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:27.242 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231002+0000", 2021-03-15T22:30:27.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6122000000000003e-05 2021-03-15T22:30:27.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.243 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.243 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231048+0000", 2021-03-15T22:30:27.243 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1882000000000003e-05 2021-03-15T22:30:27.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.244 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.244 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231090+0000", 2021-03-15T22:30:27.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068311582999999995 2021-03-15T22:30:27.245 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.245 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.245 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.246 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4285 7.b 7:d4368cae:::smithi07694878-18:head [setxattr _header (58) in=7b,truncate 3865709] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:30:27.246 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:30:27.246 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 266.86952788899998, 2021-03-15T22:30:27.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068193614999999999, 2021-03-15T22:30:27.247 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.247 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.247 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.247 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.248 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.248 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4285 2021-03-15T22:30:27.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.248 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:30:27.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.250 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:30:27.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.633e-06 2021-03-15T22:30:27.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162985+0000", 2021-03-15T22:30:27.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6789999999999999e-06 2021-03-15T22:30:27.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.253 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162987+0000", 2021-03-15T22:30:27.253 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9500000000000001e-06 2021-03-15T22:30:27.253 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162995+0000", 2021-03-15T22:30:27.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.459e-06 2021-03-15T22:30:27.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.255 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162999+0000", 2021-03-15T22:30:27.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012207240000000001 2021-03-15T22:30:27.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164220+0000", 2021-03-15T22:30:27.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0790000000000004e-05 2021-03-15T22:30:27.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164301+0000", 2021-03-15T22:30:27.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014765399999999999 2021-03-15T22:30:27.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:30:27.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164449+0000", 2021-03-15T22:30:27.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042818110999999999 2021-03-15T22:30:27.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.259 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.207267+0000", 2021-03-15T22:30:27.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055270550000000003 2021-03-15T22:30:27.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.212794+0000", 2021-03-15T22:30:27.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018305267 2021-03-15T22:30:27.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.261 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:27.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231099+0000", 2021-03-15T22:30:27.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4928000000000002e-05 2021-03-15T22:30:27.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231134+0000", 2021-03-15T22:30:27.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2364999999999998e-05 2021-03-15T22:30:27.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231176+0000", 2021-03-15T22:30:27.264 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068193614999999999 2021-03-15T22:30:27.265 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.265 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.265 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.266 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4286 7.b 7:d4368cae:::smithi07694878-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e445)", 2021-03-15T22:30:27.266 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:30:27.266 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 266.86950624100001, 2021-03-15T22:30:27.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068364477000000007, 2021-03-15T22:30:27.267 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.267 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:30:27.267 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.267 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.268 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.268 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4286 2021-03-15T22:30:27.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.268 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:30:27.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:30:27.270 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6500000000000008e-07 2021-03-15T22:30:27.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163005+0000", 2021-03-15T22:30:27.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.218e-06 2021-03-15T22:30:27.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.272 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163007+0000", 2021-03-15T22:30:27.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3890000000000002e-06 2021-03-15T22:30:27.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163013+0000", 2021-03-15T22:30:27.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.053e-06 2021-03-15T22:30:27.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163017+0000", 2021-03-15T22:30:27.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017994630000000001 2021-03-15T22:30:27.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164816+0000", 2021-03-15T22:30:27.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0958000000000007e-05 2021-03-15T22:30:27.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:27.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164897+0000", 2021-03-15T22:30:27.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066313677000000001 2021-03-15T22:30:27.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231211+0000", 2021-03-15T22:30:27.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9739000000000004e-05 2021-03-15T22:30:27.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231281+0000", 2021-03-15T22:30:27.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8015000000000003e-05 2021-03-15T22:30:27.280 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231369+0000", 2021-03-15T22:30:27.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068364477000000007 2021-03-15T22:30:27.282 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.282 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.282 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.283 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4287 7.7 e445/420 7:ed4a01d2:::smithi07694878-2:head v 445'413, mlcod=444'410)", 2021-03-15T22:30:27.283 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:30:27.283 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 266.86848794500003, 2021-03-15T22:30:27.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066973671999999998, 2021-03-15T22:30:27.284 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.284 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.284 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:30:27.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.286 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:30:27.286 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.258e-06 2021-03-15T22:30:27.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164026+0000", 2021-03-15T22:30:27.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7980000000000001e-06 2021-03-15T22:30:27.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164029+0000", 2021-03-15T22:30:27.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0787e-05 2021-03-15T22:30:27.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164039+0000", 2021-03-15T22:30:27.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2829999999999998e-06 2021-03-15T22:30:27.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164046+0000", 2021-03-15T22:30:27.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5735000000000001e-05 2021-03-15T22:30:27.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164101+0000", 2021-03-15T22:30:27.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9206000000000001e-05 2021-03-15T22:30:27.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164131+0000", 2021-03-15T22:30:27.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066830296999999997 2021-03-15T22:30:27.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.230961+0000", 2021-03-15T22:30:27.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5308000000000001e-05 2021-03-15T22:30:27.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.230996+0000", 2021-03-15T22:30:27.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066973671999999998 2021-03-15T22:30:27.296 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.296 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.296 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.297 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4288 7.7 e445/420 7:ed4a01d2:::smithi07694878-2:head v 445'414, mlcod=444'410)", 2021-03-15T22:30:27.297 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.164500+0000", 2021-03-15T22:30:27.298 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 266.86801045099998, 2021-03-15T22:30:27.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066520592000000003, 2021-03-15T22:30:27.298 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.298 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.298 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.299 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164500+0000", 2021-03-15T22:30:27.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.300 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.300 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164500+0000", 2021-03-15T22:30:27.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9610000000000001e-06 2021-03-15T22:30:27.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.301 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.302 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164503+0000", 2021-03-15T22:30:27.302 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.407e-06 2021-03-15T22:30:27.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164505+0000", 2021-03-15T22:30:27.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8310000000000001e-06 2021-03-15T22:30:27.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164513+0000", 2021-03-15T22:30:27.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9269999999999998e-06 2021-03-15T22:30:27.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164517+0000", 2021-03-15T22:30:27.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6690999999999998e-05 2021-03-15T22:30:27.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.306 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.306 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164554+0000", 2021-03-15T22:30:27.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6207000000000002e-05 2021-03-15T22:30:27.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.307 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164580+0000", 2021-03-15T22:30:27.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066419645999999999 2021-03-15T22:30:27.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231000+0000", 2021-03-15T22:30:27.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0922000000000001e-05 2021-03-15T22:30:27.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.310 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231021+0000", 2021-03-15T22:30:27.310 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066520592000000003 2021-03-15T22:30:27.310 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.310 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.311 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.311 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4384 7.b 7:d980be1f:::smithi07694878-23:head [append 0~0] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:30:27.312 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:10.205239+0000", 2021-03-15T22:30:27.312 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 256.82727158500001, 2021-03-15T22:30:27.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044851078000000003, 2021-03-15T22:30:27.312 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.313 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.313 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.313 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.313 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.314 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4384 2021-03-15T22:30:27.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.314 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205239+0000", 2021-03-15T22:30:27.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205239+0000", 2021-03-15T22:30:27.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6599999999999999e-06 2021-03-15T22:30:27.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205242+0000", 2021-03-15T22:30:27.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4120000000000001e-06 2021-03-15T22:30:27.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205245+0000", 2021-03-15T22:30:27.318 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2353e-05 2021-03-15T22:30:27.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205257+0000", 2021-03-15T22:30:27.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.7439999999999996e-06 2021-03-15T22:30:27.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.320 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.320 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.321 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205266+0000", 2021-03-15T22:30:27.321 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00035602600000000001 2021-03-15T22:30:27.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.321 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.322 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205622+0000", 2021-03-15T22:30:27.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000115955 2021-03-15T22:30:27.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205738+0000", 2021-03-15T22:30:27.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00024261500000000001 2021-03-15T22:30:27.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:30:27.324 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205981+0000", 2021-03-15T22:30:27.324 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001692928 2021-03-15T22:30:27.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207674+0000", 2021-03-15T22:30:27.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9629e-05 2021-03-15T22:30:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207703+0000", 2021-03-15T22:30:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042243636000000001 2021-03-15T22:30:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:27.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.249947+0000", 2021-03-15T22:30:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8107000000000003e-05 2021-03-15T22:30:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250025+0000", 2021-03-15T22:30:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5012999999999996e-05 2021-03-15T22:30:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250090+0000", 2021-03-15T22:30:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044851078000000003 2021-03-15T22:30:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.330 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.331 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4385 7.b 7:d980be1f:::smithi07694878-23:head [rollback 128] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:30:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:10.205276+0000", 2021-03-15T22:30:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 256.82723460900002, 2021-03-15T22:30:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044905154000000003, 2021-03-15T22:30:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.332 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.333 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4385 2021-03-15T22:30:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205276+0000", 2021-03-15T22:30:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205276+0000", 2021-03-15T22:30:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3400000000000001e-06 2021-03-15T22:30:27.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205277+0000", 2021-03-15T22:30:27.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2640000000000001e-06 2021-03-15T22:30:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205279+0000", 2021-03-15T22:30:27.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0669999999999997e-06 2021-03-15T22:30:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205285+0000", 2021-03-15T22:30:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1370000000000001e-06 2021-03-15T22:30:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205290+0000", 2021-03-15T22:30:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013506429999999999 2021-03-15T22:30:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.206640+0000", 2021-03-15T22:30:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1700999999999998e-05 2021-03-15T22:30:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.206702+0000", 2021-03-15T22:30:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015400200000000001 2021-03-15T22:30:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:30:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.206856+0000", 2021-03-15T22:30:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010626349999999999 2021-03-15T22:30:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207919+0000", 2021-03-15T22:30:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0203000000000004e-05 2021-03-15T22:30:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207999+0000", 2021-03-15T22:30:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042099725999999997 2021-03-15T22:30:27.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250099+0000", 2021-03-15T22:30:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3871000000000001e-05 2021-03-15T22:30:27.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250133+0000", 2021-03-15T22:30:27.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8565e-05 2021-03-15T22:30:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250181+0000", 2021-03-15T22:30:27.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044905154000000003 2021-03-15T22:30:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4386 7.b 7:d980be1f:::smithi07694878-23:head [append 0~0] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:30:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:10.205294+0000", 2021-03-15T22:30:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 256.82721693500002, 2021-03-15T22:30:27.351 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044975109999999999, 2021-03-15T22:30:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:30:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:30:27.352 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4386 2021-03-15T22:30:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205294+0000", 2021-03-15T22:30:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205294+0000", 2021-03-15T22:30:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0389999999999999e-06 2021-03-15T22:30:27.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205295+0000", 2021-03-15T22:30:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.145e-06 2021-03-15T22:30:27.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205296+0000", 2021-03-15T22:30:27.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6250000000000004e-06 2021-03-15T22:30:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205301+0000", 2021-03-15T22:30:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9639999999999997e-06 2021-03-15T22:30:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.205305+0000", 2021-03-15T22:30:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019453750000000001 2021-03-15T22:30:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207251+0000", 2021-03-15T22:30:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8658000000000006e-05 2021-03-15T22:30:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207319+0000", 2021-03-15T22:30:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011772700000000001 2021-03-15T22:30:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:30:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.207437+0000", 2021-03-15T22:30:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00062831400000000004 2021-03-15T22:30:27.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.208065+0000", 2021-03-15T22:30:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2487000000000003e-05 2021-03-15T22:30:27.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.208098+0000", 2021-03-15T22:30:27.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042089556 2021-03-15T22:30:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:30:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250188+0000", 2021-03-15T22:30:27.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4434999999999998e-05 2021-03-15T22:30:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250222+0000", 2021-03-15T22:30:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6785e-05 2021-03-15T22:30:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.250269+0000", 2021-03-15T22:30:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044975109999999999 2021-03-15T22:30:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.369 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:679 7.5 e560/559 7:ac84e656:::smithi07694878-31:1a1 v 560'396, mlcod=560'392)", 2021-03-15T22:30:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:28:20.276511+0000", 2021-03-15T22:30:27.370 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 126.755999179, 2021-03-15T22:30:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053800807999999999, 2021-03-15T22:30:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.371 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276511+0000", 2021-03-15T22:30:27.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276511+0000", 2021-03-15T22:30:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5489999999999998e-06 2021-03-15T22:30:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276515+0000", 2021-03-15T22:30:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7549999999999999e-06 2021-03-15T22:30:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276518+0000", 2021-03-15T22:30:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2309e-05 2021-03-15T22:30:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276530+0000", 2021-03-15T22:30:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.844e-06 2021-03-15T22:30:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276537+0000", 2021-03-15T22:30:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.503e-05 2021-03-15T22:30:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276592+0000", 2021-03-15T22:30:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.885e-05 2021-03-15T22:30:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276621+0000", 2021-03-15T22:30:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053639340000000001 2021-03-15T22:30:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.330260+0000", 2021-03-15T22:30:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2130999999999999e-05 2021-03-15T22:30:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.330312+0000", 2021-03-15T22:30:27.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053800807999999999 2021-03-15T22:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:5830 7.5 e631/559 7:ac84e656:::smithi07694878-31:head v 631'458, mlcod=630'454)", 2021-03-15T22:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.780850729999997, 2021-03-15T22:30:27.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083826879000000007, 2021-03-15T22:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:30:27.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060822330000000003 2021-03-15T22:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257742+0000", 2021-03-15T22:30:27.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9649999999999996e-06 2021-03-15T22:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257748+0000", 2021-03-15T22:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0319e-05 2021-03-15T22:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257768+0000", 2021-03-15T22:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2021-03-15T22:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257777+0000", 2021-03-15T22:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0096000000000001e-05 2021-03-15T22:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257847+0000", 2021-03-15T22:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7812000000000001e-05 2021-03-15T22:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257875+0000", 2021-03-15T22:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077557744999999997 2021-03-15T22:30:27.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.335433+0000", 2021-03-15T22:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3674999999999997e-05 2021-03-15T22:30:27.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.335487+0000", 2021-03-15T22:30:27.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083826879000000007 2021-03-15T22:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:755 7.5 e646/559 7:a9545d62:::smithi07694878-1:1fc v 646'483, mlcod=644'479)", 2021-03-15T22:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.786226374000002, 2021-03-15T22:30:27.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087969915999999995, 2021-03-15T22:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:30:27.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1300000000000001e-06 2021-03-15T22:30:27.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246287+0000", 2021-03-15T22:30:27.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1730000000000002e-06 2021-03-15T22:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246290+0000", 2021-03-15T22:30:27.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1728e-05 2021-03-15T22:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:30:27.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246301+0000", 2021-03-15T22:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7710000000000001e-06 2021-03-15T22:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:30:27.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246308+0000", 2021-03-15T22:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5705999999999999e-05 2021-03-15T22:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246354+0000", 2021-03-15T22:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8252e-05 2021-03-15T22:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246382+0000", 2021-03-15T22:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087836030999999995 2021-03-15T22:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334218+0000", 2021-03-15T22:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6124999999999997e-05 2021-03-15T22:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:27.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334254+0000", 2021-03-15T22:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087969915999999995 2021-03-15T22:30:27.410 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:27.411 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:30:27.412 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:27.415 INFO:tasks.rados.rados.0.smithi076.stdout:2479: done (7 left) 2021-03-15T22:30:27.415 INFO:tasks.rados.rados.0.smithi076.stdout:2480: done (6 left) 2021-03-15T22:30:27.415 INFO:tasks.rados.rados.0.smithi076.stdout:2481: done (5 left) 2021-03-15T22:30:27.415 INFO:tasks.rados.rados.0.smithi076.stdout:2482: done (4 left) 2021-03-15T22:30:27.416 INFO:tasks.rados.rados.0.smithi076.stdout:2483: done (3 left) 2021-03-15T22:30:27.416 INFO:tasks.rados.rados.0.smithi076.stdout:2484: read oid 20 snap -1 2021-03-15T22:30:27.416 INFO:tasks.rados.rados.0.smithi076.stdout:2484: expect (ObjNum 812 snap 269 seq_num 812) 2021-03-15T22:30:27.416 INFO:tasks.rados.rados.0.smithi076.stdout:2485: copy_from oid 16 from oid 44 current snap is 283 2021-03-15T22:30:27.417 INFO:tasks.rados.rados.0.smithi076.stdout:2486: write (excl) oid 30 current snap is 283 2021-03-15T22:30:27.417 INFO:tasks.rados.rados.0.smithi076.stdout:2486: seq_num 858 ranges {420406=795178,1974962=445444} 2021-03-15T22:30:27.417 INFO:tasks.rados.rados.0.smithi076.stdout:2486: writing smithi07694878-30 from 420406 to 1215584 tid 1 2021-03-15T22:30:27.417 INFO:tasks.rados.rados.0.smithi076.stdout:2486: writing smithi07694878-30 from 1974962 to 2420406 tid 2 2021-03-15T22:30:27.417 INFO:tasks.rados.rados.0.smithi076.stdout:2487: write (excl) oid 25 current snap is 283 2021-03-15T22:30:27.418 INFO:tasks.rados.rados.0.smithi076.stdout:2487: seq_num 859 ranges {437213=470427,1633444=728951} 2021-03-15T22:30:27.418 INFO:tasks.rados.rados.0.smithi076.stdout:2487: writing smithi07694878-25 from 437213 to 907640 tid 1 2021-03-15T22:30:27.418 INFO:tasks.rados.rados.0.smithi076.stdout:2487: writing smithi07694878-25 from 1633444 to 2362395 tid 2 2021-03-15T22:30:27.418 INFO:tasks.rados.rados.0.smithi076.stdout:2488: write (excl) oid 19 current snap is 283 2021-03-15T22:30:27.419 INFO:tasks.rados.rados.0.smithi076.stdout:2488: seq_num 860 ranges {454020=662029,1691926=588962} 2021-03-15T22:30:27.419 INFO:tasks.rados.rados.0.smithi076.stdout:2488: writing smithi07694878-19 from 454020 to 1116049 tid 1 2021-03-15T22:30:27.419 INFO:tasks.rados.rados.0.smithi076.stdout:2488: writing smithi07694878-19 from 1691926 to 2280888 tid 2 2021-03-15T22:30:27.419 INFO:tasks.rados.rados.0.smithi076.stdout:2489: read oid 32 snap 220 2021-03-15T22:30:27.420 INFO:tasks.rados.rados.0.smithi076.stdout:2489: expect (ObjNum 682 snap 218 seq_num 682) 2021-03-15T22:30:27.420 INFO:tasks.rados.rados.0.smithi076.stdout:2490: write oid 38 current snap is 283 2021-03-15T22:30:27.420 INFO:tasks.rados.rados.0.smithi076.stdout:2484: expect (ObjNum 812 snap 269 seq_num 812) 2021-03-15T22:30:27.424 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:30:27.444 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:30:27.446 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:27.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:27.447 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:27.461 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:30:27.480 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:30:27.497 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:30:27.515 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:30:27.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:27 smithi076 conmon[56514]: cluster 2021-03-15T22:30:26.251222+0000 mon.a (mon.0) 1840 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2021-03-15T22:30:27.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:27 smithi076 conmon[62156]: cluster 2021-03-15T22:30:26.251222+0000 mon.a (mon.0) 1840 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2021-03-15T22:30:27.533 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:30:27.533 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:30:27.534 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:30:27.534 INFO:teuthology.orchestra.run.smithi076.stderr:Error ENXIO: problem getting command descriptions from osd.2 2021-03-15T22:30:27.535 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:30:27.535 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:30:27.536 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:30:27.536 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:30:27.536 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:30:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:27 smithi125 conmon[56739]: cluster 2021-03-15T22:30:26.251222+0000 mon.a (mon 2021-03-15T22:30:27.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:27 smithi125 conmon[56739]: .0) 1840 : cluster [DBG] osdmap e677: 8 total, 7 up, 8 in 2021-03-15T22:30:27.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:27.551 INFO:tasks.rados.rados.0.smithi076.stdout:2486: finishing write tid 1 to smithi07694878-30 2021-03-15T22:30:27.551 INFO:tasks.rados.rados.0.smithi076.stdout:2485: finishing copy_from to smithi07694878-16 2021-03-15T22:30:27.551 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1737 (ObjNum 794 snap 261 seq_num 794) dirty exists 2021-03-15T22:30:27.551 INFO:tasks.rados.rados.0.smithi076.stdout:2485: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:30:27.552 INFO:tasks.rados.rados.0.smithi076.stdout:2486: finishing write tid 2 to smithi07694878-30 2021-03-15T22:30:27.552 INFO:tasks.rados.rados.0.smithi076.stdout:2486: finishing write tid 3 to smithi07694878-30 2021-03-15T22:30:27.552 INFO:tasks.rados.rados.0.smithi076.stdout:2486: finishing write tid 4 to smithi07694878-30 2021-03-15T22:30:27.552 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 1860 (ObjNum 858 snap 283 seq_num 858) dirty exists 2021-03-15T22:30:27.553 INFO:tasks.rados.rados.0.smithi076.stdout:2486: left oid 30 (ObjNum 858 snap 283 seq_num 858) 2021-03-15T22:30:27.553 INFO:tasks.rados.rados.0.smithi076.stdout:2487: finishing write tid 1 to smithi07694878-25 2021-03-15T22:30:27.553 INFO:tasks.rados.rados.0.smithi076.stdout:2490: seq_num 861 ranges {470827=737278,1750408=720419} 2021-03-15T22:30:27.557 INFO:tasks.rados.rados.0.smithi076.stdout:2490: writing smithi07694878-38 from 470827 to 1208105 tid 1 2021-03-15T22:30:27.565 INFO:tasks.rados.rados.0.smithi076.stdout:2490: writing smithi07694878-38 from 1750408 to 2470827 tid 2 2021-03-15T22:30:27.565 INFO:tasks.rados.rados.0.smithi076.stdout:2484: done (9 left) 2021-03-15T22:30:27.565 INFO:tasks.rados.rados.0.smithi076.stdout:2485: done (8 left) 2021-03-15T22:30:27.566 INFO:tasks.rados.rados.0.smithi076.stdout:2486: done (7 left) 2021-03-15T22:30:27.566 INFO:tasks.rados.rados.0.smithi076.stdout:2491: snap_create 2021-03-15T22:30:27.566 INFO:tasks.rados.rados.0.smithi076.stdout:2487: finishing write tid 2 to smithi07694878-25 2021-03-15T22:30:27.566 INFO:tasks.rados.rados.0.smithi076.stdout:2487: finishing write tid 3 to smithi07694878-25 2021-03-15T22:30:27.566 INFO:tasks.rados.rados.0.smithi076.stdout:2487: finishing write tid 4 to smithi07694878-25 2021-03-15T22:30:27.567 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 1591 (ObjNum 859 snap 283 seq_num 859) dirty exists 2021-03-15T22:30:27.567 INFO:tasks.rados.rados.0.smithi076.stdout:2487: left oid 25 (ObjNum 859 snap 283 seq_num 859) 2021-03-15T22:30:27.567 INFO:tasks.rados.rados.0.smithi076.stdout:2488: finishing write tid 1 to smithi07694878-19 2021-03-15T22:30:27.568 INFO:tasks.rados.rados.0.smithi076.stdout:2488: finishing write tid 2 to smithi07694878-19 2021-03-15T22:30:27.568 INFO:tasks.rados.rados.0.smithi076.stdout:2488: finishing write tid 3 to smithi07694878-19 2021-03-15T22:30:27.568 INFO:tasks.rados.rados.0.smithi076.stdout:2488: finishing write tid 4 to smithi07694878-19 2021-03-15T22:30:27.568 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 1056 (ObjNum 860 snap 283 seq_num 860) dirty exists 2021-03-15T22:30:27.568 INFO:tasks.rados.rados.0.smithi076.stdout:2488: left oid 19 (ObjNum 860 snap 283 seq_num 860) 2021-03-15T22:30:27.569 INFO:tasks.rados.rados.0.smithi076.stdout:2489: expect (ObjNum 682 snap 218 seq_num 682) 2021-03-15T22:30:27.571 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:27.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:27.572 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:27.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:27.695 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:27.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:27.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:27.723 INFO:tasks.rados.rados.0.smithi076.stdout:2490: finishing write tid 1 to smithi07694878-38 2021-03-15T22:30:27.723 INFO:tasks.rados.rados.0.smithi076.stdout:2490: finishing write tid 2 to smithi07694878-38 2021-03-15T22:30:27.723 INFO:tasks.rados.rados.0.smithi076.stdout:2490: finishing write tid 3 to smithi07694878-38 2021-03-15T22:30:27.724 INFO:tasks.rados.rados.0.smithi076.stdout:2490: finishing write tid 4 to smithi07694878-38 2021-03-15T22:30:27.724 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 1059 (ObjNum 861 snap 283 seq_num 861) dirty exists 2021-03-15T22:30:27.724 INFO:tasks.rados.rados.0.smithi076.stdout:2490: left oid 38 (ObjNum 861 snap 283 seq_num 861) 2021-03-15T22:30:27.797 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:27.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:27.821 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:27.821 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:27.922 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:27.945 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:27.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:27.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:28.048 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:28.070 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:28.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:28.071 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:28.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:28.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:28.198 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:28.198 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:28.264 INFO:tasks.rados.rados.0.smithi076.stdout:2487: done (7 left) 2021-03-15T22:30:28.265 INFO:tasks.rados.rados.0.smithi076.stdout:2488: done (6 left) 2021-03-15T22:30:28.265 INFO:tasks.rados.rados.0.smithi076.stdout:2489: done (5 left) 2021-03-15T22:30:28.265 INFO:tasks.rados.rados.0.smithi076.stdout:2490: done (4 left) 2021-03-15T22:30:28.265 INFO:tasks.rados.rados.0.smithi076.stdout:2491: done (3 left) 2021-03-15T22:30:28.266 INFO:tasks.rados.rados.0.smithi076.stdout:2492: snap_create 2021-03-15T22:30:28.300 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:28.322 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:28.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:28.403 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-15T22:30:28.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:28.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:28.447 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:28.448 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:28 smithi076 conmon[56514]: cluster 2021-03-15T22:30:27.144929+0000 mgr.y (mgr.14144) 844 : cluster [DBG] pgmap v1160: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 501 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 12 op/s; 57/1710 objects degraded (3.333%); 126/1710 objects misplaced (7.368%) 2021-03-15T22:30:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:28 smithi076 conmon[56514]: cluster 2021-03-15T22:30:27.257273+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e678: 8 total, 7 up, 8 in 2021-03-15T22:30:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:28 smithi076 conmon[56514]: audit 2021-03-15T22:30:27.404346+0000 mon.a (mon.0) 1842 : audit [DBG] from='client.? 172.21.15.76:0/3659358699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:28 smithi076 conmon[56514]: cluster 2021-03-15T22:30:27.472312+0000 osd.3 (osd.3) 125 : cluster [DBG] 5.0 scrub ok 2021-03-15T22:30:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:28 smithi076 conmon[56514]: cluster 2021-03-15T22:30:27.732138+0000 osd.0 (osd.0) 102 : cluster [DBG] 7.f scrub ok 2021-03-15T22:30:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:28 smithi076 conmon[62156]: cluster 2021-03-15T22:30:27.144929+0000 mgr.y (mgr.14144) 844 : cluster [DBG] pgmap v1160: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 501 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 12 op/s; 57/1710 objects degraded (3.333%); 126/1710 objects misplaced (7.368%) 2021-03-15T22:30:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:28 smithi076 conmon[62156]: cluster 2021-03-15T22:30:27.257273+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e678: 8 total, 7 up, 8 in 2021-03-15T22:30:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:28 smithi076 conmon[62156]: audit 2021-03-15T22:30:27.404346+0000 mon.a (mon.0) 1842 : audit [DBG] from='client.? 172.21.15.76:0/3659358699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:28 smithi076 conmon[62156]: cluster 2021-03-15T22:30:27.472312+0000 osd.3 (osd.3) 125 : cluster [DBG] 5.0 scrub ok 2021-03-15T22:30:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:28 smithi076 conmon[62156]: cluster 2021-03-15T22:30:27.732138+0000 osd.0 (osd.0) 102 : cluster [DBG] 7.f scrub ok 2021-03-15T22:30:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: cluster 2021-03-15T22:30:27.144929+0000 mgr.y (mgr.14144) 2021-03-15T22:30:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: 844 : cluster [DBG] pgmap v1160: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 501 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.4 MiB/s wr, 12 op/s; 57/1710 objects degraded (3.333%); 126/1710 objects misplaced (7.368%) 2021-03-15T22:30:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: cluster 2021-03-15T22:30:27.257273+0000 mon.a (mon.0) 1841 : cluster [DBG] osdmap e678: 8 total, 7 up, 8 in 2021-03-15T22:30:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: audit 2021-03-15T22:30: 2021-03-15T22:30:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: 27.404346+0000 mon.a (mon.0) 1842 : 2021-03-15T22:30:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: audit [DBG] from='client.? 172.21.15.76:0/3659358699' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:30:28.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: T22:30:27.472312+0000 osd.3 (osd 2021-03-15T22:30:28.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: .3) 125 : cluster [DBG] 5.0 scrub ok 2021-03-15T22:30:28.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:30:28.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: 03-15T22:30:27.732138+0000 2021-03-15T22:30:28.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: osd.0 (osd.0) 102 : cluster [DBG] 7.f scrub ok 2021-03-15T22:30:28.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:28 smithi125 conmon[56739]: 2021-03-15T22:30:28.549 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:28.573 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:28.574 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:28.675 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:28.697 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:28.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:28.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:28.801 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:28.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:28.825 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:28.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:28.891 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:30:28.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:28.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:28.957 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:28.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:29.059 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:29.082 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:29.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:29.084 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:29.109 DEBUG:teuthology.orchestra.run:got remote process result: 6 2021-03-15T22:30:29.110 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2021-03-15T22:30:29.185 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:29.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:29.210 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:29.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:29.265 INFO:journalctl@ceph.osd.2.smithi076.stdout:Mar 15 22:30:28 smithi076 conmon[223218]: debug 2021-03-15T22:30:28.914+0000 7f9ec8c4f080 -1 osd.2 489 log_to_monitors {default=true} 2021-03-15T22:30:29.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[56514]: cluster 2021-03-15T22:30:28.176826+0000 osd.6 (osd.6) 104 2021-03-15T22:30:29.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[56514]: : cluster [DBG] 4.5 scrub ok 2021-03-15T22:30:29.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[56514]: cluster 2021-03-15T22:30:28.261082+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2021-03-15T22:30:29.266 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[56514]: cluster 2021-03-15T22:30:28.328454+0000 osd.4 (osd.4) 145 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:30:29.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[56514]: audit 2021-03-15T22:30:28.918492+0000 mon.a (mon.0) 1844 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-15T22:30:29.270 INFO:tasks.rados.rados.0.smithi076.stdout:2492: done (3 left) 2021-03-15T22:30:29.270 INFO:tasks.rados.rados.0.smithi076.stdout:2493: read oid 15 snap 250 2021-03-15T22:30:29.270 INFO:tasks.rados.rados.0.smithi076.stdout:2493: expect (ObjNum 714 snap 231 seq_num 714) 2021-03-15T22:30:29.271 INFO:tasks.rados.rados.0.smithi076.stdout:2494: read oid 11 snap -1 2021-03-15T22:30:29.271 INFO:tasks.rados.rados.0.smithi076.stdout:2494: expect (ObjNum 813 snap 269 seq_num 813) 2021-03-15T22:30:29.271 INFO:tasks.rados.rados.0.smithi076.stdout:2495: copy_from oid 24 from oid 12 current snap is 285 2021-03-15T22:30:29.271 INFO:tasks.rados.rados.0.smithi076.stdout:2496: write oid 49 current snap is 285 2021-03-15T22:30:29.272 INFO:tasks.rados.rados.0.smithi076.stdout:2496: seq_num 862 ranges {487634=412527,1408890=592631} 2021-03-15T22:30:29.275 INFO:tasks.rados.rados.0.smithi076.stdout:2496: writing smithi07694878-49 from 487634 to 900161 tid 1 2021-03-15T22:30:29.282 INFO:tasks.rados.rados.0.smithi076.stdout:2496: writing smithi07694878-49 from 1408890 to 2001521 tid 2 2021-03-15T22:30:29.283 INFO:tasks.rados.rados.0.smithi076.stdout:2497: rollback oid 3 current snap is 285 2021-03-15T22:30:29.283 INFO:tasks.rados.rados.0.smithi076.stdout:2493: expect (ObjNum 714 snap 231 seq_num 714) 2021-03-15T22:30:29.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:29.333 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:29.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:29.335 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:29.366 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:29.418 INFO:tasks.rados.rados.0.smithi076.stdout:2494: expect (ObjNum 813 snap 269 seq_num 813) 2021-03-15T22:30:29.426 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:29.436 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:29.458 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:29.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:29.459 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:29.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: cluster 2021-03-15T22:30:29.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 2021-03-15T22:30:29.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 2021 2021-03-15T22:30:29.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: - 2021-03-15T22:30:29.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 03 2021-03-15T22:30:29.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: - 2021-03-15T22:30:29.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 15 2021-03-15T22:30:29.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: T 2021-03-15T22:30:29.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 22 2021-03-15T22:30:29.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: : 2021-03-15T22:30:29.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 30 2021-03-15T22:30:29.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: : 2021-03-15T22:30:29.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 28 2021-03-15T22:30:29.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: . 2021-03-15T22:30:29.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 176826 2021-03-15T22:30:29.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: +0000 2021-03-15T22:30:29.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 2021-03-15T22:30:29.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: osd.6 2021-03-15T22:30:29.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: ( 2021-03-15T22:30:29.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: osd 2021-03-15T22:30:29.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: . 2021-03-15T22:30:29.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 6 2021-03-15T22:30:29.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: ) 2021-03-15T22:30:29.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 104 2021-03-15T22:30:29.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: : 2021-03-15T22:30:29.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: cluster 2021-03-15T22:30:29.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 2021-03-15T22:30:29.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: [DBG] 2021-03-15T22:30:29.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 2021-03-15T22:30:29.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 4.5 scrub ok 2021-03-15T22:30:29.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 2021-03-15T22:30:29.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: cluster 2021-03-15T22:30:29.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 2021-03-15T22:30:29.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 2021 2021-03-15T22:30:29.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: - 2021-03-15T22:30:29.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 03 2021-03-15T22:30:29.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: - 2021-03-15T22:30:29.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 15 2021-03-15T22:30:29.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: T 2021-03-15T22:30:29.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 22 2021-03-15T22:30:29.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: : 2021-03-15T22:30:29.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 30:28.261082+0000 2021-03-15T22:30:29.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: mon.a (mon.0) 1843 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2021-03-15T22:30:29.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: cluster 2021-03-15T22:30:29.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 2021-03-15T22:30:28.328454 2021-03-15T22:30:29.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 145 : cluster 2021-03-15T22:30:29.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: [DBG] 4.1a scrub ok 2021-03-15T22:30:29.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:30:29.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: T22:30:28.918492+0000 mon.a ( 2021-03-15T22:30:29.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: mon.0) 1844 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-15T22:30:29.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:29 smithi076 conmon[62156]: 2021-03-15T22:30:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:29 smithi125 conmon[56739]: cluster 2021-03-15T22:30:28.176826+0000 osd.6 (osd.6) 104 : cluster [DBG] 2021-03-15T22:30:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:29 smithi125 conmon[56739]: 4.5 scrub ok 2021-03-15T22:30:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:29 smithi125 conmon[56739]: cluster 2021-03-15T22:30:28.261082+0000 mon.a (mon.0) 1843 : cluster [DBG] osdmap e679: 8 total, 7 up, 8 in 2021-03-15T22:30:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:29 smithi125 conmon[56739]: cluster 2021-03-15T22:30:28. 2021-03-15T22:30:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:29 smithi125 conmon[56739]: 328454+0000 osd.4 (osd.4) 145 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:30:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:29 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:30:29.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:29 smithi125 conmon[56739]: 22:30:28.918492+0000 mon.a (mon.0) 1844 : 2021-03-15T22:30:29.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:29 smithi125 conmon[56739]: audit [INF] from='osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2021-03-15T22:30:29.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:29.575 INFO:tasks.rados.rados.0.smithi076.stdout:2496: finishing write tid 1 to smithi07694878-49 2021-03-15T22:30:29.575 INFO:tasks.rados.rados.0.smithi076.stdout:2496: finishing write tid 2 to smithi07694878-49 2021-03-15T22:30:29.575 INFO:tasks.rados.rados.0.smithi076.stdout:2496: finishing write tid 3 to smithi07694878-49 2021-03-15T22:30:29.576 INFO:tasks.rados.rados.0.smithi076.stdout:2496: finishing write tid 4 to smithi07694878-49 2021-03-15T22:30:29.576 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 1864 (ObjNum 862 snap 285 seq_num 862) dirty exists 2021-03-15T22:30:29.576 INFO:tasks.rados.rados.0.smithi076.stdout:2496: left oid 49 (ObjNum 862 snap 285 seq_num 862) 2021-03-15T22:30:29.576 INFO:tasks.rados.rados.0.smithi076.stdout:2495: finishing copy_from to smithi07694878-24 2021-03-15T22:30:29.577 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1574 (ObjNum 770 snap 255 seq_num 770) dirty exists 2021-03-15T22:30:29.577 INFO:tasks.rados.rados.0.smithi076.stdout:2495: finishing copy_from racing read to smithi07694878-24 2021-03-15T22:30:29.577 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 279 2021-03-15T22:30:29.577 INFO:tasks.rados.rados.0.smithi076.stdout:2493: done (7 left) 2021-03-15T22:30:29.578 INFO:tasks.rados.rados.0.smithi076.stdout:2494: done (6 left) 2021-03-15T22:30:29.578 INFO:tasks.rados.rados.0.smithi076.stdout:2495: done (5 left) 2021-03-15T22:30:29.578 INFO:tasks.rados.rados.0.smithi076.stdout:2496: done (4 left) 2021-03-15T22:30:29.578 INFO:tasks.rados.rados.0.smithi076.stdout:2498: write oid 6 current snap is 285 2021-03-15T22:30:29.578 INFO:tasks.rados.rados.0.smithi076.stdout:2498: seq_num 863 ranges {504441=487776,1467372=736289} 2021-03-15T22:30:29.579 INFO:tasks.rados.rados.0.smithi076.stdout:2498: writing smithi07694878-6 from 504441 to 992217 tid 1 2021-03-15T22:30:29.583 INFO:tasks.rados.rados.0.smithi076.stdout:2498: writing smithi07694878-6 from 1467372 to 2203661 tid 2 2021-03-15T22:30:29.583 INFO:tasks.rados.rados.0.smithi076.stdout:2499: snap_remove snap 269 2021-03-15T22:30:29.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:29.587 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:29.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:29.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:29.714 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:29.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:29.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:29.813 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:30:29.814 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:30:29.814 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:30:29.814 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:30:29.817 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:29.839 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:29.841 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:29.942 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:29.965 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:29.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:29.966 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:30.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:30.091 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:30.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:30.093 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:30.194 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:30.226 INFO:tasks.rados.rados.0.smithi076.stdout:2499: done (5 left) 2021-03-15T22:30:30.226 INFO:tasks.rados.rados.0.smithi076.stdout:2500: copy_from oid 30 from oid 1 current snap is 285 2021-03-15T22:30:30.227 INFO:tasks.rados.rados.0.smithi076.stdout:2501: write oid 24 current snap is 285 2021-03-15T22:30:30.227 INFO:tasks.rados.rados.0.smithi076.stdout:2501: seq_num 864 ranges {521248=563025,1809501=596300} 2021-03-15T22:30:30.229 INFO:tasks.rados.rados.0.smithi076.stdout:2501: writing smithi07694878-24 from 521248 to 1084273 tid 1 2021-03-15T22:30:30.230 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:30.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:30.232 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:30.237 INFO:tasks.rados.rados.0.smithi076.stdout:2501: writing smithi07694878-24 from 1809501 to 2405801 tid 2 2021-03-15T22:30:30.238 INFO:tasks.rados.rados.0.smithi076.stdout:2502: snap_create 2021-03-15T22:30:30.238 INFO:tasks.rados.rados.0.smithi076.stdout:2498: finishing write tid 1 to smithi07694878-6 2021-03-15T22:30:30.238 INFO:tasks.rados.rados.0.smithi076.stdout:2498: finishing write tid 2 to smithi07694878-6 2021-03-15T22:30:30.239 INFO:tasks.rados.rados.0.smithi076.stdout:2498: finishing write tid 3 to smithi07694878-6 2021-03-15T22:30:30.239 INFO:tasks.rados.rados.0.smithi076.stdout:2498: finishing write tid 4 to smithi07694878-6 2021-03-15T22:30:30.239 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 1611 (ObjNum 863 snap 285 seq_num 863) dirty exists 2021-03-15T22:30:30.239 INFO:tasks.rados.rados.0.smithi076.stdout:2498: left oid 6 (ObjNum 863 snap 285 seq_num 863) 2021-03-15T22:30:30.266 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:30:30.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:30.355 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:30.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:30.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:30.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:30.480 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:30.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:30.482 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:30.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: cluster 2021-03-15T22:30:28.967350+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:30:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: cluster 2021-03-15T22:30:29.145506+0000 mgr.y (mgr.14144) 845 : cluster [DBG] pgmap v1163: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 504 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 57/1713 objects degraded (3.327%); 126/1713 objects misplaced (7.356%) 2021-03-15T22:30:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: audit 2021-03-15T22:30:29.266064+0000 mon.a (mon.0) 1845 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-15T22:30:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: cluster 2021-03-15T22:30:29.266222+0000 mon.a (mon.0) 1846 : cluster [DBG] osdmap e680: 8 total, 7 up, 8 in 2021-03-15T22:30:30.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: audit 2021-03-15T22:30:29.268650+0000 mon.a (mon.0) 1847 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:30:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: cluster 2021-03-15T22:30:29.533412+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:30:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:30:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: 22:30:29.729229+0000 osd.0 (osd.0) 103 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:30:30.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: audit 2021-03-15T22:30:29.999970+0000 mgr.y (mgr.14144) 846 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:30.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: cluster 2021-03-15T22:30:30.204814+0000 mon.a (mon.0) 1848 : cluster [WRN] Health check update: Degraded data redundancy: 57/1713 objects degraded (3.327%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:30.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: cluster 2021-03-15T22:30:30.205567+0000 mon.a (mon.0) 1849 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-03-15T22:30:30.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: cluster 2021-03-15T22:30:30.209923+0000 mon.a (mon.0) 1850 : cluster [INF] osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618] boot 2021-03-15T22:30:30.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: cluster 2021-03-15T22:30:30.209991+0000 mon.a (mon.0) 1851 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2021-03-15T22:30:30.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[56514]: audit 2021-03-15T22:30:30.215513+0000 mon.a (mon.0) 1852 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:30:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: cluster 2021-03-15T22:30:28.967350 2021-03-15T22:30:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 12 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:30:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:30:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: 03-15T22:30:29.145506+0000 mgr.y (mgr.14144) 845 : cluster [DBG] pgmap v1163: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 504 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 57/1713 objects degraded (3.327%); 126/1713 objects misplaced (7.356%) 2021-03-15T22:30:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: audit 2021 2021-03-15T22:30:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: -03-15T22:30:29.266064+0000 mon.a ( 2021-03-15T22:30:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: mon.0) 1845 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-15T22:30:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:30:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: -03-15T22:30:29.266222+0000 2021-03-15T22:30:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: mon.a (mon.0) 1846 : cluster [DBG] osdmap e680: 8 total, 7 up, 8 in 2021-03-15T22:30:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: audit 2021 2021-03-15T22:30:30.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: -03-15T22:30:29.268650+0000 mon.a ( 2021-03-15T22:30:30.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: mon.0) 1847 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:30:30.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:30:30.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: :30:29.533412+0000 osd.3 (osd.3) 126 2021-03-15T22:30:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: : cluster [DBG] 6.6 scrub ok 2021-03-15T22:30:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:30:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: 22:30:29.729229+0000 osd.0 (osd.0) 2021-03-15T22:30:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: 103 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:30:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:30:30.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: 15T22:30:29.999970+0000 mgr.y (mgr. 2021-03-15T22:30:30.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: 14144) 846 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:30.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:30:30.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: T22:30:30.204814+0000 mon.a (mon.0 2021-03-15T22:30:30.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: ) 1848 : cluster [WRN] Health check update: Degraded data redundancy: 57/1713 objects degraded (3.327%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:30.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:30:30.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: T22:30:30.205567+0000 mon.a (mon.0) 2021-03-15T22:30:30.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: 1849 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-03-15T22:30:30.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:30:30.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: -15T22:30:30.209923+0000 mon.a ( 2021-03-15T22:30:30.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: mon.0) 1850 : cluster [INF] osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618] boot 2021-03-15T22:30:30.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:30:30.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: -15T22:30:30.209991+0000 mon.a (mon. 2021-03-15T22:30:30.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: 0) 1851 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2021-03-15T22:30:30.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:30:30.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: 03-15T22:30:30.215513+0000 mon.a (mon 2021-03-15T22:30:30.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:30 smithi076 conmon[62156]: .0) 1852 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:30:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: cluster 2021-03-15T22:30:28.967350+0000 osd.5 (osd.5) 12 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:30:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: cluster 2021-03-15T22:30:29 2021-03-15T22:30:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: .145506+0000 mgr.y (mgr.14144) 845 : cluster [DBG] pgmap v1163: 138 pgs: 1 active+clean+remapped+wait, 30 active+clean+remapped, 4 active+undersized+degraded, 10 active+undersized, 93 active+clean; 504 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.6 MiB/s rd, 1.2 MiB/s wr, 9 op/s; 57/1713 objects degraded (3.327%); 126/1713 objects misplaced (7.356%) 2021-03-15T22:30:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: audit 2021-03-15T22:30:29.266064+0000 mon.a (mon.0) 1845 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2021-03-15T22:30:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: cluster 2021-03-15T22:30:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: 2021-03-15T22:30:29.266222+0000 mon.a (mon.0) 1846 2021-03-15T22:30:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: : cluster [DBG] osdmap e680: 8 total, 7 up, 8 in 2021-03-15T22:30:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: audit 2021 2021-03-15T22:30:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: -03-15T22:30:29.268650+0000 mon.a 2021-03-15T22:30:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: (mon.0) 1847 : audit [INF] from='osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi076", "root=default"]}]: dispatch 2021-03-15T22:30:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: cluster 2021-03-15T22:30:29.533412+0000 osd.3 (osd.3) 126 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:30:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: cluster 2021-03-15T22:30:29.729229+0000 osd.0 (osd. 2021-03-15T22:30:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: 0) 103 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:30:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: audit 2021-03-15T22:30:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: 2021-03-15T22:30:29.999970+0000 mgr.y (mgr. 2021-03-15T22:30:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: 14144) 846 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: cluster 2021-03-15T22:30:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: 2021-03-15T22:30:30 2021-03-15T22:30:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: .204814+0000 mon.a (mon.0) 1848 : cluster [WRN] Health check update: Degraded data redundancy: 57/1713 objects degraded (3.327%), 4 pgs degraded, 14 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: 2021-03-15T22:30:30.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:30:30.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: 03-15T22:30:30.205567+0000 mon.a (mon.0) 1849 : cluster [INF] Health check cleared: OSD_DOWN (was: 1 osds down) 2021-03-15T22:30:30.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: cluster 2021-03-15T22:30 2021-03-15T22:30:30.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: :30.209923+0000 mon.a (mon.0) 1850 : 2021-03-15T22:30:30.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: cluster [INF] osd.2 [v2:172.21.15.76:6818/1676192618,v1:172.21.15.76:6819/1676192618] boot 2021-03-15T22:30:30.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:30:30.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: T22:30:30.209991+0000 mon.a ( 2021-03-15T22:30:30.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: mon.0) 1851 : cluster [DBG] osdmap e681: 8 total, 8 up, 8 in 2021-03-15T22:30:30.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: audit 2021-03-15T22:30:30.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: 2021-03-15T22:30:30 2021-03-15T22:30:30.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: .215513+0000 mon.a (mon.0) 1852 : 2021-03-15T22:30:30.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:30 smithi125 conmon[56739]: audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2021-03-15T22:30:30.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:30.606 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:30.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:30.607 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:30.708 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:30.732 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:30.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:30.733 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:30.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:30.858 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:30.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:30.961 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:30.984 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:30.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:30.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:31.086 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:31.108 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:31.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:31.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:31.111 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:30:31.211 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:31.218 INFO:tasks.rados.rados.0.smithi076.stdout:2498: done (7 left) 2021-03-15T22:30:31.219 INFO:tasks.rados.rados.0.smithi076.stdout:2502: done (6 left) 2021-03-15T22:30:31.219 INFO:tasks.rados.rados.0.smithi076.stdout:2503: delete oid 7 current snap is 286 2021-03-15T22:30:31.251 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:31.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:31.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:31.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:31.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:31.378 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:31.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:31.479 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:31.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:31.502 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:31.503 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:31.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:31 smithi125 conmon[56739]: cluster 2021-03-15T22:30:29.983037+0000 osd.1 (osd.1) 132 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:30:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:31 smithi125 conmon[56739]: cluster 2021-03-15T22:30:30. 2021-03-15T22:30:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:31 smithi125 conmon[56739]: 738099+0000 osd.0 (osd.0) 104 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:30:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:31 smithi125 conmon[56739]: cluster 2021-03-15T22:30:31.211543+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2021-03-15T22:30:31.605 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:31.628 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:31.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:31.629 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:31.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:31.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:31.753 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:31.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:31.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[56514]: cluster 2021-03-15T22:30:29.983037+0000 osd.1 (osd.1) 132 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:30:31.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[56514]: cluster 2021-03-15T22:30:30.738099+0000 osd.0 (osd.0) 104 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:30:31.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[56514]: cluster 2021-03-15T22:30:31.211543+0000 mon.a (mon.0) 1853 : cluster [DBG] osdmap e682: 8 total, 8 up, 8 in 2021-03-15T22:30:31.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[62156]: cluster 2021-03-15T22:30:29 2021-03-15T22:30:31.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[62156]: .983037+0000 osd.1 (osd.1) 132 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:30:31.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:30:31.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[62156]: 22:30:30.738099+0000 osd.0 (osd.0) 104 2021-03-15T22:30:31.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[62156]: : cluster [DBG] 5.5 scrub ok 2021-03-15T22:30:31.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:30:31.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[62156]: -03-15T22:30 2021-03-15T22:30:31.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[62156]: :31.211543+0000 mon.a ( 2021-03-15T22:30:31.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[62156]: mon.0) 1853 : cluster [DBG] 2021-03-15T22:30:31.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:31 smithi076 conmon[62156]: osdmap e682: 8 total, 8 up, 8 in 2021-03-15T22:30:31.854 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:31.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:31.877 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:31.877 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:31.978 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:32.001 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:32.003 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:32.104 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:32.127 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:32.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:32.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:32.188 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:32.229 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:32.276 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:32.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:32.278 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:32.289 INFO:tasks.rados.rados.0.smithi076.stdout:2500: finishing copy_from to smithi07694878-30 2021-03-15T22:30:32.289 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 1324 (ObjNum 803 snap 268 seq_num 803) dirty exists 2021-03-15T22:30:32.289 INFO:tasks.rados.rados.0.smithi076.stdout:2500: finishing copy_from racing read to smithi07694878-30 2021-03-15T22:30:32.291 INFO:tasks.rados.rados.0.smithi076.stdout:2453: finishing write tid 1 to smithi07694878-14 2021-03-15T22:30:32.292 INFO:tasks.rados.rados.0.smithi076.stdout:2453: finishing write tid 2 to smithi07694878-14 2021-03-15T22:30:32.292 INFO:tasks.rados.rados.0.smithi076.stdout:2453: finishing write tid 3 to smithi07694878-14 2021-03-15T22:30:32.293 INFO:tasks.rados.rados.0.smithi076.stdout:2453: finishing write tid 4 to smithi07694878-14 2021-03-15T22:30:32.293 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 1558 (ObjNum 842 snap 281 seq_num 842) dirty exists 2021-03-15T22:30:32.294 INFO:tasks.rados.rados.0.smithi076.stdout:2453: left oid 14 (ObjNum 842 snap 281 seq_num 842) 2021-03-15T22:30:32.294 INFO:tasks.rados.rados.0.smithi076.stdout:2469: finishing write tid 1 to smithi07694878-26 2021-03-15T22:30:32.294 INFO:tasks.rados.rados.0.smithi076.stdout:2469: finishing write tid 2 to smithi07694878-26 2021-03-15T22:30:32.295 INFO:tasks.rados.rados.0.smithi076.stdout:2453: done (6 left) 2021-03-15T22:30:32.296 INFO:tasks.rados.rados.0.smithi076.stdout:2500: done (5 left) 2021-03-15T22:30:32.296 INFO:tasks.rados.rados.0.smithi076.stdout:2503: done (4 left) 2021-03-15T22:30:32.296 INFO:tasks.rados.rados.0.smithi076.stdout:2504: read oid 36 snap 199 2021-03-15T22:30:32.297 INFO:tasks.rados.rados.0.smithi076.stdout:2504: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:30:32.297 INFO:tasks.rados.rados.0.smithi076.stdout:2505: write (excl) oid 11 current snap is 286 2021-03-15T22:30:32.297 INFO:tasks.rados.rados.0.smithi076.stdout:2505: seq_num 866 ranges {554862=713523,1926465=599969} 2021-03-15T22:30:32.303 INFO:tasks.rados.rados.0.smithi076.stdout:2505: writing smithi07694878-11 from 554862 to 1268385 tid 1 2021-03-15T22:30:32.313 INFO:tasks.rados.rados.0.smithi076.stdout:2505: writing smithi07694878-11 from 1926465 to 2526434 tid 2 2021-03-15T22:30:32.316 INFO:tasks.rados.rados.0.smithi076.stdout:2469: finishing write tid 3 to smithi07694878-26 2021-03-15T22:30:32.316 INFO:tasks.rados.rados.0.smithi076.stdout:2469: finishing write tid 4 to smithi07694878-26 2021-03-15T22:30:32.316 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 1561 (ObjNum 850 snap 282 seq_num 850) dirty exists 2021-03-15T22:30:32.316 INFO:tasks.rados.rados.0.smithi076.stdout:2469: left oid 26 (ObjNum 850 snap 282 seq_num 850) 2021-03-15T22:30:32.317 INFO:tasks.rados.rados.0.smithi076.stdout:2478: expect (ObjNum 598 snap 183 seq_num 598) 2021-03-15T22:30:32.379 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:32.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[56514]: cluster 2021-03-15T22:30:29.908471+0000 2021-03-15T22:30:32.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[56514]: osd.2 (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:32.393 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[56514]: cluster 2021-03-15T22:30:29.910312+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:32.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[56514]: cluster 2021-03-15T22:30:31.146089+0000 mgr.y (mgr.14144) 847 : cluster [DBG] pgmap v1166: 138 pgs: 1 peering, 8 remapped+peering, 1 active+clean+remapped+wait, 22 active+clean+remapped, 3 active+undersized+degraded, 10 active+undersized, 93 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 2.6 MiB/s wr, 24 op/s; 44/1734 objects degraded (2.537%); 107/1734 objects misplaced (6.171%) 2021-03-15T22:30:32.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[56514]: cluster 2021-03-15T22:30:32.215215+0000 mon.a (mon.0) 1854 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2021-03-15T22:30:32.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[62156]: cluster 2021-03-15T22:30:29.908471+0000 osd.2 2021-03-15T22:30:32.394 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[62156]: (osd.2) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:32.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[62156]: cluster 2021-03-15T22:30:29.910312+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:32.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[62156]: cluster 2021-03-15T22:30:31.146089+0000 mgr.y (mgr 2021-03-15T22:30:32.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[62156]: .14144) 847 : cluster [DBG] pgmap v1166: 138 pgs: 1 peering, 8 remapped+peering, 1 active+clean+remapped+wait, 22 active+clean+remapped, 3 active+undersized+degraded, 10 active+undersized, 93 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 2.6 MiB/s wr, 24 op/s; 44/1734 objects degraded (2.537%); 107/1734 objects misplaced (6.171%) 2021-03-15T22:30:32.395 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:30:32.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[62156]: 30:32.215215+0000 mon.a (mon.0) 2021-03-15T22:30:32.396 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:32 smithi076 conmon[62156]: 1854 : cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2021-03-15T22:30:32.400 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:32.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:32.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:32.414 INFO:tasks.rados.rados.0.smithi076.stdout:2497: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:30:32.415 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 1545 (ObjNum 265131200 snap 21896 seq_num 261725408) dirty dne 2021-03-15T22:30:32.415 INFO:tasks.rados.rados.0.smithi076.stdout:2469: done (5 left) 2021-03-15T22:30:32.415 INFO:tasks.rados.rados.0.smithi076.stdout:2478: done (4 left) 2021-03-15T22:30:32.416 INFO:tasks.rados.rados.0.smithi076.stdout:2497: done (3 left) 2021-03-15T22:30:32.416 INFO:tasks.rados.rados.0.smithi076.stdout:2506: read oid 14 snap 250 2021-03-15T22:30:32.416 INFO:tasks.rados.rados.0.smithi076.stdout:2506: expect deleted 2021-03-15T22:30:32.416 INFO:tasks.rados.rados.0.smithi076.stdout:2507: snap_remove snap 241 2021-03-15T22:30:32.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:32.526 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:32.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:32.528 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:32 smithi125 conmon[56739]: cluster 2021-03-15T22:30:29.908471+0000 osd.2 (osd.2 2021-03-15T22:30:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:32 smithi125 conmon[56739]: ) 1 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:32 smithi125 conmon[56739]: cluster 2021-03-15T22:30:29.910312+0000 osd.2 (osd.2) 2 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:32 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:30:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:32 smithi125 conmon[56739]: 15T22:30:31.146089+0000 mgr.y (mgr.14144) 847 : cluster [DBG] pgmap v1166: 138 pgs: 1 peering, 8 remapped+peering, 1 active+clean+remapped+wait, 22 active+clean+remapped, 3 active+undersized+degraded, 10 active+undersized, 93 active+clean; 515 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 19 MiB/s rd, 2.6 MiB/s wr, 24 op/s; 44/1734 objects degraded (2.537%); 107/1734 objects misplaced (6.171%) 2021-03-15T22:30:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:32 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:30:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:32 smithi125 conmon[56739]: 15T22:30:32.215215+0000 2021-03-15T22:30:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:32 smithi125 conmon[56739]: mon.a (mon.0) 1854 : 2021-03-15T22:30:32.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:32 smithi125 conmon[56739]: cluster [DBG] osdmap e683: 8 total, 8 up, 8 in 2021-03-15T22:30:32.629 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:32.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:32.659 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:32.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:30:32.695 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:30:32.761 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:32.792 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:32.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:32.793 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:32.894 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:32.916 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:32.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:32.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:33.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:33.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:33.041 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:33.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:33.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:33.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:33.165 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:33.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:33.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:33.284 INFO:tasks.rados.rados.0.smithi076.stdout:2507: done (4 left) 2021-03-15T22:30:33.285 INFO:tasks.rados.rados.0.smithi076.stdout:2508: copy_from oid 27 from oid 33 current snap is 286 2021-03-15T22:30:33.285 INFO:tasks.rados.rados.0.smithi076.stdout:2504: expect (ObjNum 387 snap 104 seq_num 387) 2021-03-15T22:30:33.293 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:33.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:33.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:33.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:33.417 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:33.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:33.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:33.457 INFO:tasks.rados.rados.0.smithi076.stdout:2505: finishing write tid 1 to smithi07694878-11 2021-03-15T22:30:33.457 INFO:tasks.rados.rados.0.smithi076.stdout:2505: finishing write tid 2 to smithi07694878-11 2021-03-15T22:30:33.457 INFO:tasks.rados.rados.0.smithi076.stdout:2504: done (4 left) 2021-03-15T22:30:33.458 INFO:tasks.rados.rados.0.smithi076.stdout:2509: snap_create 2021-03-15T22:30:33.458 INFO:tasks.rados.rados.0.smithi076.stdout:2505: finishing write tid 3 to smithi07694878-11 2021-03-15T22:30:33.458 INFO:tasks.rados.rados.0.smithi076.stdout:2505: finishing write tid 4 to smithi07694878-11 2021-03-15T22:30:33.458 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1557 (ObjNum 866 snap 286 seq_num 866) dirty exists 2021-03-15T22:30:33.459 INFO:tasks.rados.rados.0.smithi076.stdout:2505: left oid 11 (ObjNum 866 snap 286 seq_num 866) 2021-03-15T22:30:33.459 INFO:tasks.rados.rados.0.smithi076.stdout:2501: finishing write tid 1 to smithi07694878-24 2021-03-15T22:30:33.459 INFO:tasks.rados.rados.0.smithi076.stdout:2501: finishing write tid 2 to smithi07694878-24 2021-03-15T22:30:33.459 INFO:tasks.rados.rados.0.smithi076.stdout:2501: finishing write tid 3 to smithi07694878-24 2021-03-15T22:30:33.460 INFO:tasks.rados.rados.0.smithi076.stdout:2501: finishing write tid 4 to smithi07694878-24 2021-03-15T22:30:33.460 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1707 (ObjNum 864 snap 285 seq_num 864) dirty exists 2021-03-15T22:30:33.460 INFO:tasks.rados.rados.0.smithi076.stdout:2501: left oid 24 (ObjNum 864 snap 285 seq_num 864) 2021-03-15T22:30:33.520 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:33.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:33.543 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:33.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:33.644 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:33.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:33.668 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:33.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:33.770 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:33.792 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:33.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:33.793 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:33.853 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-15T22:30:33.895 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:33.917 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:33.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:33.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:34.020 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:34.043 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:34.045 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:34.146 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:34.169 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:34.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:34.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:34.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:34.290 INFO:tasks.rados.rados.0.smithi076.stdout:2501: done (4 left) 2021-03-15T22:30:34.290 INFO:tasks.rados.rados.0.smithi076.stdout:2505: done (3 left) 2021-03-15T22:30:34.290 INFO:tasks.rados.rados.0.smithi076.stdout:2506: done (2 left) 2021-03-15T22:30:34.291 INFO:tasks.rados.rados.0.smithi076.stdout:2509: done (1 left) 2021-03-15T22:30:34.291 INFO:tasks.rados.rados.0.smithi076.stdout:2510: delete oid 9 current snap is 287 2021-03-15T22:30:34.300 INFO:tasks.rados.rados.0.smithi076.stdout:2510: done (1 left) 2021-03-15T22:30:34.300 INFO:tasks.rados.rados.0.smithi076.stdout:2511: copy_from oid 28 from oid 42 current snap is 287 2021-03-15T22:30:34.301 INFO:tasks.rados.rados.0.smithi076.stdout:2512: delete oid 12 current snap is 287 2021-03-15T22:30:34.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:34.302 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:34.303 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:34.304 INFO:tasks.rados.rados.0.smithi076.stdout:2512: done (2 left) 2021-03-15T22:30:34.304 INFO:tasks.rados.rados.0.smithi076.stdout:2513: delete oid 32 current snap is 287 2021-03-15T22:30:34.305 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:30:34.310 INFO:tasks.rados.rados.0.smithi076.stdout:2513: done (2 left) 2021-03-15T22:30:34.310 INFO:tasks.rados.rados.0.smithi076.stdout:2514: write (excl) oid 21 current snap is 287 2021-03-15T22:30:34.311 INFO:tasks.rados.rados.0.smithi076.stdout:2514: seq_num 870 ranges {622090=730872,1760393=607307} 2021-03-15T22:30:34.317 INFO:tasks.rados.rados.0.smithi076.stdout:2514: writing smithi07694878-21 from 622090 to 1352962 tid 1 2021-03-15T22:30:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:2514: writing smithi07694878-21 from 1760393 to 2367700 tid 2 2021-03-15T22:30:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:2515: write (excl) oid 23 current snap is 287 2021-03-15T22:30:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:2511: finishing copy_from to smithi07694878-28 2021-03-15T22:30:34.325 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 1737 (ObjNum 794 snap 261 seq_num 794) dirty exists 2021-03-15T22:30:34.325 INFO:tasks.rados.rados.0.smithi076.stdout:2515: seq_num 871 ranges {638897=406121,1818875=750965} 2021-03-15T22:30:34.329 INFO:tasks.rados.rados.0.smithi076.stdout:2515: writing smithi07694878-23 from 638897 to 1045018 tid 1 2021-03-15T22:30:34.338 INFO:tasks.rados.rados.0.smithi076.stdout:2515: writing smithi07694878-23 from 1818875 to 2569840 tid 2 2021-03-15T22:30:34.338 INFO:tasks.rados.rados.0.smithi076.stdout:2516: write (excl) oid 41 current snap is 287 2021-03-15T22:30:34.338 INFO:tasks.rados.rados.0.smithi076.stdout:2516: seq_num 872 ranges {655704=481370,1761004=494623} 2021-03-15T22:30:34.344 INFO:tasks.rados.rados.0.smithi076.stdout:2516: writing smithi07694878-41 from 655704 to 1137074 tid 1 2021-03-15T22:30:34.351 INFO:tasks.rados.rados.0.smithi076.stdout:2516: writing smithi07694878-41 from 1761004 to 2255627 tid 2 2021-03-15T22:30:34.351 INFO:tasks.rados.rados.0.smithi076.stdout:2517: delete oid 3 current snap is 287 2021-03-15T22:30:34.351 INFO:tasks.rados.rados.0.smithi076.stdout:2511: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:30:34.352 INFO:tasks.rados.rados.0.smithi076.stdout:2514: finishing write tid 1 to smithi07694878-21 2021-03-15T22:30:34.352 INFO:tasks.rados.rados.0.smithi076.stdout:2514: finishing write tid 2 to smithi07694878-21 2021-03-15T22:30:34.352 INFO:tasks.rados.rados.0.smithi076.stdout:2514: finishing write tid 3 to smithi07694878-21 2021-03-15T22:30:34.352 INFO:tasks.rados.rados.0.smithi076.stdout:2514: finishing write tid 4 to smithi07694878-21 2021-03-15T22:30:34.352 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 2010 (ObjNum 870 snap 287 seq_num 870) dirty exists 2021-03-15T22:30:34.353 INFO:tasks.rados.rados.0.smithi076.stdout:2514: left oid 21 (ObjNum 870 snap 287 seq_num 870) 2021-03-15T22:30:34.353 INFO:tasks.rados.rados.0.smithi076.stdout:2515: finishing write tid 1 to smithi07694878-23 2021-03-15T22:30:34.353 INFO:tasks.rados.rados.0.smithi076.stdout:2515: finishing write tid 2 to smithi07694878-23 2021-03-15T22:30:34.353 INFO:tasks.rados.rados.0.smithi076.stdout:2515: finishing write tid 3 to smithi07694878-23 2021-03-15T22:30:34.354 INFO:tasks.rados.rados.0.smithi076.stdout:2515: finishing write tid 4 to smithi07694878-23 2021-03-15T22:30:34.354 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 1223 (ObjNum 871 snap 287 seq_num 871) dirty exists 2021-03-15T22:30:34.354 INFO:tasks.rados.rados.0.smithi076.stdout:2515: left oid 23 (ObjNum 871 snap 287 seq_num 871) 2021-03-15T22:30:34.354 INFO:tasks.rados.rados.0.smithi076.stdout:2516: finishing write tid 1 to smithi07694878-41 2021-03-15T22:30:34.355 INFO:tasks.rados.rados.0.smithi076.stdout:2511: done (5 left) 2021-03-15T22:30:34.355 INFO:tasks.rados.rados.0.smithi076.stdout:2514: done (4 left) 2021-03-15T22:30:34.355 INFO:tasks.rados.rados.0.smithi076.stdout:2515: done (3 left) 2021-03-15T22:30:34.355 INFO:tasks.rados.rados.0.smithi076.stdout:2517: done (2 left) 2021-03-15T22:30:34.356 INFO:tasks.rados.rados.0.smithi076.stdout:2518: write oid 20 current snap is 287 2021-03-15T22:30:34.356 INFO:tasks.rados.rados.0.smithi076.stdout:2518: seq_num 874 ranges {689318=631868,1877968=498292} 2021-03-15T22:30:34.359 INFO:tasks.rados.rados.0.smithi076.stdout:2518: writing smithi07694878-20 from 689318 to 1321186 tid 1 2021-03-15T22:30:34.365 INFO:tasks.rados.rados.0.smithi076.stdout:2518: writing smithi07694878-20 from 1877968 to 2376260 tid 2 2021-03-15T22:30:34.365 INFO:tasks.rados.rados.0.smithi076.stdout:2519: snap_create 2021-03-15T22:30:34.366 INFO:tasks.rados.rados.0.smithi076.stdout:2516: finishing write tid 2 to smithi07694878-41 2021-03-15T22:30:34.366 INFO:tasks.rados.rados.0.smithi076.stdout:2516: finishing write tid 3 to smithi07694878-41 2021-03-15T22:30:34.366 INFO:tasks.rados.rados.0.smithi076.stdout:2516: finishing write tid 4 to smithi07694878-41 2021-03-15T22:30:34.366 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 1220 (ObjNum 872 snap 287 seq_num 872) dirty exists 2021-03-15T22:30:34.367 INFO:tasks.rados.rados.0.smithi076.stdout:2516: left oid 41 (ObjNum 872 snap 287 seq_num 872) 2021-03-15T22:30:34.367 INFO:tasks.rados.rados.0.smithi076.stdout:2518: finishing write tid 1 to smithi07694878-20 2021-03-15T22:30:34.368 INFO:tasks.rados.rados.0.smithi076.stdout:2518: finishing write tid 2 to smithi07694878-20 2021-03-15T22:30:34.369 INFO:tasks.rados.rados.0.smithi076.stdout:2518: finishing write tid 3 to smithi07694878-20 2021-03-15T22:30:34.369 INFO:tasks.rados.rados.0.smithi076.stdout:2518: finishing write tid 4 to smithi07694878-20 2021-03-15T22:30:34.369 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 1747 (ObjNum 874 snap 287 seq_num 874) dirty exists 2021-03-15T22:30:34.369 INFO:tasks.rados.rados.0.smithi076.stdout:2518: left oid 20 (ObjNum 874 snap 287 seq_num 874) 2021-03-15T22:30:34.405 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:34.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:34.428 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:34.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:34.530 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:34.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:34 smithi125 conmon[56739]: cluster 2021-03-15T22:30:33.146704+0000 mgr.y (mgr.14144) 848 : cluster [DBG] pgmap v1169: 138 pgs: 6 peering, 10 remapped+peering, 13 active+clean+remapped, 1 active+undersized+remapped, 1 active+undersized+degraded, 5 active+undersized, 102 active+clean; 521 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 4.0 MiB/s wr, 32 op/s; 22/1749 objects degraded (1.258%); 33/1749 objects misplaced (1.887%); 5.5 MiB/s, 11 objects/s recovering 2021-03-15T22:30:34.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:34 smithi125 conmon[56739]: cluster 2021-03-15T22:30:33.280315+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2021-03-15T22:30:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:34 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:30:34.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:34 smithi125 conmon[56739]: 30:33.800602+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? 172.21.15.76:0/1615712455' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:34.551 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:34.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:34.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:34.599 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:34 smithi076 conmon[56514]: cluster 2021-03-15T22:30:33.146704+0000 mgr.y (mgr.14144) 848 : cluster [DBG] pgmap v1169: 138 pgs: 6 peering, 10 remapped+peering, 13 active+clean+remapped, 1 active+undersized+remapped, 1 active+undersized+degraded, 5 active+undersized, 102 active+clean; 521 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 4.0 MiB/s wr, 32 op/s; 22/1749 objects degraded (1.258%); 33/1749 objects misplaced (1.887%); 5.5 MiB/s, 11 objects/s recovering 2021-03-15T22:30:34.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:34 smithi076 conmon[56514]: cluster 2021-03-15T22:30:33.280315+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2021-03-15T22:30:34.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:34 smithi076 conmon[56514]: audit 2021-03-15T22:30:33.800602+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? 172.21.15.76:0/1615712455' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:34.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:34 smithi076 conmon[62156]: cluster 2021-03-15T22:30:33.146704+0000 2021-03-15T22:30:34.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:34 smithi076 conmon[62156]: mgr.y (mgr.14144) 848 : cluster [DBG] pgmap v1169: 138 pgs: 6 peering, 10 remapped+peering, 13 active+clean+remapped, 1 active+undersized+remapped, 1 active+undersized+degraded, 5 active+undersized, 102 active+clean; 521 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 20 MiB/s rd, 4.0 MiB/s wr, 32 op/s; 22/1749 objects degraded (1.258%); 33/1749 objects misplaced (1.887%); 5.5 MiB/s, 11 objects/s recovering 2021-03-15T22:30:34.601 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:34 smithi076 conmon[62156]: cluster 2021-03-15T22:30:33.280315+0000 mon.a (mon.0) 1855 : cluster [DBG] osdmap e684: 8 total, 8 up, 8 in 2021-03-15T22:30:34.602 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:34 smithi076 conmon[62156]: audit 2021-03-15T22:30:33.800602+0000 mon.a (mon.0) 1856 : audit [INF] from='client.? 172.21.15.76:0/1615712455' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:34.654 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:34.676 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:34.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:34.779 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:34.801 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:34.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:34.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:34.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:34.926 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:34.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:35.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:35.052 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:35.053 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:35.154 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:35.175 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:35.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:35.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:35.300 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:35.301 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:35.314 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:35.364 INFO:tasks.rados.rados.0.smithi076.stdout:2516: done (3 left) 2021-03-15T22:30:35.364 INFO:tasks.rados.rados.0.smithi076.stdout:2518: done (2 left) 2021-03-15T22:30:35.364 INFO:tasks.rados.rados.0.smithi076.stdout:2519: done (1 left) 2021-03-15T22:30:35.364 INFO:tasks.rados.rados.0.smithi076.stdout:2520: copy_from oid 18 from oid 46 current snap is 288 2021-03-15T22:30:35.365 INFO:tasks.rados.rados.0.smithi076.stdout:2521: copy_from oid 2 from oid 20 current snap is 288 2021-03-15T22:30:35.365 INFO:tasks.rados.rados.0.smithi076.stdout:2522: snap_create 2021-03-15T22:30:35.366 INFO:tasks.rados.rados.0.smithi076.stdout:2520: finishing copy_from to smithi07694878-18 2021-03-15T22:30:35.366 INFO:tasks.rados.rados.0.smithi076.stdout:2520: got expected ENOENT (src dne) 2021-03-15T22:30:35.366 INFO:tasks.rados.rados.0.smithi076.stdout:2520: finishing copy_from racing read to smithi07694878-18 2021-03-15T22:30:35.382 INFO:tasks.rados.rados.0.smithi076.stdout:2521: finishing copy_from to smithi07694878-2 2021-03-15T22:30:35.383 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1747 (ObjNum 874 snap 287 seq_num 874) dirty exists 2021-03-15T22:30:35.383 INFO:tasks.rados.rados.0.smithi076.stdout:2521: finishing copy_from racing read to smithi07694878-2 2021-03-15T22:30:35.403 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:35.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:35.426 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:35.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:35.528 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:35.552 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:35.553 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:35.611 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:30:35.628 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:30:35.644 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:30:35.655 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:35.662 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:30:35.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:35.678 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:35.679 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:35.681 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:30:35.699 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:30:35.700 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[56514]: cluster 2021-03-15T22:30:34.003113+0000 osd.2 (osd.2 2021-03-15T22:30:35.700 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[56514]: ) 3 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:30:35.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[56514]: cluster 2021-03-15T22:30:34.070595+0000 osd.1 (osd.1) 133 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:30:35.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[56514]: cluster 2021-03-15T22:30:34.278330+0000 mon.a (mon.0) 1857 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:30:35.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[56514]: audit 2021-03-15T22:30:34.287095+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? 172.21.15.76:0/1615712455' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:35.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[56514]: cluster 2021-03-15T22:30:34.287178+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2021-03-15T22:30:35.702 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[56514]: cluster 2021-03-15T22:30:34.363173+0000 osd.4 (osd.4) 146 : cluster [DBG] 3.9 deep-scrub ok 2021-03-15T22:30:35.702 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[56514]: cluster 2021-03-15T22:30:35.206399+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check update: Degraded data redundancy: 22/1749 objects degraded (1.258%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:35.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[62156]: cluster 2021-03-15T22:30:34.003113+0000 osd.2 (osd.2) 3 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:30:35.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[62156]: cluster 2021-03-15T22:30:34.070595+0000 osd.1 (osd.1) 133 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:30:35.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[62156]: cluster 2021-03-15T22:30:34.278330+0000 mon.a (mon.0) 1857 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:30:35.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[62156]: audit 2021-03-15T22:30:34.287095+0000 mon.a (mon.0) 1858 : audit [INF] from='client.? 172.21.15.76:0/1615712455' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:35.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[62156]: cluster 2021-03-15T22:30:34.287178+0000 mon.a (mon.0) 1859 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2021-03-15T22:30:35.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[62156]: cluster 2021-03-15T22:30:34.363173+0000 osd.4 (osd.4) 146 : cluster [DBG] 3.9 deep-scrub ok 2021-03-15T22:30:35.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:35 smithi076 conmon[62156]: cluster 2021-03-15T22:30:35.206399+0000 mon.a (mon.0) 1860 : cluster [WRN] Health check update: Degraded data redundancy: 22/1749 objects degraded (1.258%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:35.717 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:30:35.734 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:30:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:30:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:30:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:30:35.736 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:30:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:30:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:30:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:30:35.737 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:30:35.780 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:35.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: cluster 2021-03-15T22:30:34.003113+0000 osd.2 (osd.2) 3 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:30:35.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: cluster 2021-03-15T22:30 2021-03-15T22:30:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: :34.070595+0000 osd.1 (osd.1) 133 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:30:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: cluster 2021-03-15T22:30:34.278330+0000 mon.a (mon.0) 1857 : cluster [INF] 2021-03-15T22:30:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:30:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: audit 2021 2021-03-15T22:30:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: -03-15T22:30:34.287095+0000 2021-03-15T22:30:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: mon.a (mon.0) 1858 : audit [INF] 2021-03-15T22:30:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1615712455' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:30:35.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: 22:30:34.287178+0000 mon.a ( 2021-03-15T22:30:35.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: mon.0) 1859 : cluster [DBG] osdmap e685: 8 total, 8 up, 8 in 2021-03-15T22:30:35.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: cluster 2021-03-15T22:30:35.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: 2021-03-15T22:30 2021-03-15T22:30:35.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: :34.363173+0000 osd.4 (osd.4) 2021-03-15T22:30:35.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: 146 : cluster [DBG] 3.9 deep-scrub ok 2021-03-15T22:30:35.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:30:35.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: -03-15T22:30:35 2021-03-15T22:30:35.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: .206399+0000 mon.a (mon.0) 1860 2021-03-15T22:30:35.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:35 smithi125 conmon[56739]: : cluster [WRN] Health check update: Degraded data redundancy: 22/1749 objects degraded (1.258%), 1 pg degraded, 6 pgs undersized (PG_DEGRADED) 2021-03-15T22:30:35.803 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:35.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:35.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:35.906 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:35.930 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:35.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:35.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:36.033 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 0, 3, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:30:36.033 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:30:36.033 INFO:tasks.thrashosds.thrasher:inject_pause on 3 2021-03-15T22:30:36.034 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2021-03-15T22:30:36.034 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2021-03-15T22:30:36.034 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 config set filestore_inject_stall 3 2021-03-15T22:30:36.035 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:36.059 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:36.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:36.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:36.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:36.187 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:36.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:36.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:36.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:36.312 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:36.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:36.314 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:36.359 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[56514]: cluster 2021-03-15T22:30:34.990327+0000 osd.2 (osd.2) 4 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:30:36.360 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[56514]: cluster 2021-03-15T22:30:35.147221+0000 mgr.y (mgr.14144) 849 : cluster [DBG] pgmap v1172: 138 pgs: 6 peering, 7 remapped+peering, 3 active+clean+wait, 2 active+undersized+remapped, 120 active+clean; 502 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 10 op/s; 19 MiB/s, 23 objects/s recovering 2021-03-15T22:30:36.360 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[56514]: cluster 2021-03-15T22:30:35.351213+0000 mon.a (mon.0) 1861 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/1749 objects degraded (1.258%), 1 pg degraded, 6 pgs undersized) 2021-03-15T22:30:36.360 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[56514]: cluster 2021-03-15T22:30:35.351251+0000 mon.a (mon.0) 1862 : cluster [INF] Cluster is now healthy 2021-03-15T22:30:36.361 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[56514]: cluster 2021-03-15T22:30:35.360953+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2021-03-15T22:30:36.361 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[56514]: cluster 2021-03-15T22:30:35.423554+0000 osd.7 (osd.7) 132 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:30:36.361 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[56514]: audit 2021-03-15T22:30:35.592518+0000 mon.a (mon.0) 1864 : audit [DBG] from='client.? 172.21.15.76:0/2154560921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:36.366 INFO:tasks.rados.rados.0.smithi076.stdout:2520: done (3 left) 2021-03-15T22:30:36.366 INFO:tasks.rados.rados.0.smithi076.stdout:2521: done (2 left) 2021-03-15T22:30:36.367 INFO:tasks.rados.rados.0.smithi076.stdout:2522: done (1 left) 2021-03-15T22:30:36.367 INFO:tasks.rados.rados.0.smithi076.stdout:2523: snap_create 2021-03-15T22:30:36.415 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:36.440 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:36.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:36.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:36.543 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:36.568 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:36.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:36.569 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:36.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[62156]: cluster 2021-03-15T22:30:34.990327+0000 2021-03-15T22:30:36.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[62156]: osd.2 (osd.2) 4 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:30:36.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[62156]: cluster 2021-03-15T22:30:35.147221+0000 mgr.y (mgr.14144) 849 : cluster [DBG] pgmap v1172: 138 pgs: 6 peering, 7 remapped+peering, 3 active+clean+wait, 2 active+undersized+remapped, 120 active+clean; 502 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 10 op/s; 19 MiB/s, 23 objects/s recovering 2021-03-15T22:30:36.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[62156]: cluster 2021-03-15T22:30:35.351213+0000 mon.a (mon.0) 1861 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/1749 objects degraded (1.258%), 1 pg degraded, 6 pgs undersized) 2021-03-15T22:30:36.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[62156]: cluster 2021-03-15T22:30:35.351251+0000 mon.a (mon.0) 1862 : cluster [INF] Cluster is now healthy 2021-03-15T22:30:36.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[62156]: cluster 2021-03-15T22:30:35.360953+0000 mon.a (mon.0) 1863 : cluster [DBG] osdmap e686: 8 total, 8 up, 8 in 2021-03-15T22:30:36.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[62156]: cluster 2021-03-15T22:30:35.423554+0000 osd.7 (osd.7) 132 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:30:36.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:36 smithi076 conmon[62156]: audit 2021-03-15T22:30:35.592518+0000 mon.a (mon.0) 1864 : audit [DBG] from='client.? 172.21.15.76:0/2154560921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:36.670 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:36.695 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:36.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:36.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: cluster 2021-03-15T22:30:34.990327+0000 osd.2 (osd.2) 4 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:30:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: cluster 2021-03-15T22:30 2021-03-15T22:30:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: :35.147221+0000 mgr.y (mgr.14144) 849 : cluster [DBG] pgmap v1172: 138 pgs: 6 peering, 7 remapped+peering, 3 active+clean+wait, 2 active+undersized+remapped, 120 active+clean; 502 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.4 MiB/s wr, 10 op/s; 19 MiB/s, 23 objects/s recovering 2021-03-15T22:30:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: cluster 2021-03-15T22:30:35.351213+0000 mon.a (mon.0) 1861 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/1749 objects degraded (1.258%), 1 pg degraded, 6 pgs undersized) 2021-03-15T22:30:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:30:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: 03-15T22:30:35.351251+0000 mon.a (mon.0) 1862 : cluster [INF] Cluster is now healthy 2021-03-15T22:30:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:30:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: -03-15T22:30:35.360953 2021-03-15T22:30:36.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1863 : cluster [DBG] 2021-03-15T22:30:36.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: osdmap e686: 8 total, 8 up, 8 in 2021-03-15T22:30:36.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:30:36.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: 22:30:35.423554+0000 osd.7 (osd. 2021-03-15T22:30:36.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: 7) 132 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:30:36.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:30:36.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: 03-15T22:30:35.592518+0000 2021-03-15T22:30:36.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: mon.a (mon.0) 1864 : audit [DBG] from='client.? 172.21.15.76:0/2154560921' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:36.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:36 smithi125 conmon[56739]: 2021-03-15T22:30:36.798 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:36.822 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:36.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:36.825 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:36.926 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:36.951 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:36.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:36.953 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:37.054 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:37.077 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:37.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:37.078 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:37.179 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:37.206 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:37.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:37.208 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:37.307 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:30:37.308 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:37.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:37.335 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:37.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:37.372 INFO:tasks.rados.rados.0.smithi076.stdout:2523: done (1 left) 2021-03-15T22:30:37.372 INFO:tasks.rados.rados.0.smithi076.stdout:2524: snap_create 2021-03-15T22:30:37.392 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:37.436 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:37.463 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:37.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:37.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:37.565 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:37.583 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[56514]: cluster 2021-03-15T22:30:36.106275 2021-03-15T22:30:37.584 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[56514]: +0000 osd.5 (osd.5) 13 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:30:37.584 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[56514]: cluster 2021-03-15T22:30:36.290651+0000 osd.6 (osd.6) 105 : cluster [DBG] 7.d scrub ok 2021-03-15T22:30:37.584 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[56514]: cluster 2021-03-15T22:30:36.363895+0000 mon.a (mon.0) 1865 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2021-03-15T22:30:37.584 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[56514]: cluster 2021-03-15T22:30:36.417214+0000 osd.4 (osd.4) 147 : cluster [DBG] 6.e deep-scrub ok 2021-03-15T22:30:37.585 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: cluster 2021-03-15T22:30:36.106275 2021-03-15T22:30:37.585 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 13 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:30:37.585 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:30:37.586 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: -03-15T22:30:36.290651+0000 osd.6 (osd.6) 105 : 2021-03-15T22:30:37.586 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: cluster [DBG] 7.d scrub ok 2021-03-15T22:30:37.586 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:30:37.586 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: 15T22:30:36.363895+0000 mon.a (mon. 2021-03-15T22:30:37.586 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: 0) 1865 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2021-03-15T22:30:37.587 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: cluster 2021-03-15T22:30:37.587 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:30:37.587 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: 30:36.417214+0000 osd.4 (osd.4 2021-03-15T22:30:37.587 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:37 smithi076 conmon[62156]: ) 147 : cluster [DBG] 6.e deep-scrub ok 2021-03-15T22:30:37.591 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:37.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:37.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:37.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:37.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:37.721 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:37.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:37.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:37 smithi125 conmon[56739]: cluster 2021-03-15T22:30:36.106275+0000 osd.5 (osd.5) 13 : 2021-03-15T22:30:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:37 smithi125 conmon[56739]: cluster [DBG] 4.15 scrub ok 2021-03-15T22:30:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:37 smithi125 conmon[56739]: cluster 2021-03-15T22:30:36.290651+0000 osd.6 (osd.6) 105 : cluster [DBG] 7.d scrub ok 2021-03-15T22:30:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:37 smithi125 conmon[56739]: 2021-03-15T22:30:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:37 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:30:37.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:37 smithi125 conmon[56739]: T22:30:36.363895+0000 mon.a 2021-03-15T22:30:37.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:37 smithi125 conmon[56739]: (mon.0) 1865 : cluster [DBG] osdmap e687: 8 total, 8 up, 8 in 2021-03-15T22:30:37.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:37 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:30:37.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:37 smithi125 conmon[56739]: -15T22:30:36.417214+0000 2021-03-15T22:30:37.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:37 smithi125 conmon[56739]: osd.4 (osd.4) 147 : cluster [DBG] 6.e deep-scrub ok 2021-03-15T22:30:37.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:37 smithi125 conmon[56739]: 2021-03-15T22:30:37.823 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:37.847 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:37.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:37.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:37.950 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:37.977 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:37.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:37.978 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:37.984 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:38.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:38.105 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:38.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:38.107 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:38.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:38.233 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:38.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:38.235 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:38.336 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:38.361 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:38.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:38.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:38.372 INFO:tasks.rados.rados.0.smithi076.stdout:2524: done (1 left) 2021-03-15T22:30:38.373 INFO:tasks.rados.rados.0.smithi076.stdout:2525: write oid 22 current snap is 291 2021-03-15T22:30:38.373 INFO:tasks.rados.rados.0.smithi076.stdout:2525: seq_num 875 ranges {706125=423470,1652803=758303} 2021-03-15T22:30:38.378 INFO:tasks.rados.rados.0.smithi076.stdout:2525: writing smithi07694878-22 from 706125 to 1129595 tid 1 2021-03-15T22:30:38.385 INFO:tasks.rados.rados.0.smithi076.stdout:2525: writing smithi07694878-22 from 1652803 to 2411106 tid 2 2021-03-15T22:30:38.385 INFO:tasks.rados.rados.0.smithi076.stdout:2526: snap_remove snap 265 2021-03-15T22:30:38.464 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:38.488 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:38.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:38.590 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:38.613 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:38.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:38.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:38.658 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:30:38.659 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:30:38.659 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:30:38.659 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:30:38.659 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:30:38.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[56514]: cluster 2021-03-15T22:30:36.764735+0000 osd.6 (osd.6) 106 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:38.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[56514]: cluster 2021-03-15T22:30:36.765188+0000 osd.6 (osd.6) 107 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:38.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[56514]: cluster 2021-03-15T22:30:37.146404+0000 osd.5 (osd.5) 14 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:30:38.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[56514]: cluster 2021-03-15T22:30:37.147867+0000 mgr.y (mgr.14144) 850 : cluster [DBG] pgmap v1175: 138 pgs: 3 active+clean+wait, 2 active+undersized+remapped, 133 active+clean; 510 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 10 op/s; 27 MiB/s, 33 objects/s recovering 2021-03-15T22:30:38.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[56514]: cluster 2021-03-15T22:30:37.321710+0000 osd.6 (osd.6) 108 : cluster [DBG] 3.10 deep-scrub ok 2021-03-15T22:30:38.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[56514]: cluster 2021-03-15T22:30:37.368438+0000 mon.a (mon.0) 2021-03-15T22:30:38.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[56514]: 1866 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2021-03-15T22:30:38.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[56514]: cluster 2021-03-15T22:30:37.798383+0000 osd.0 (osd.0) 105 : cluster [DBG] 3.6 deep-scrub ok 2021-03-15T22:30:38.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[62156]: cluster 2021-03-15T22:30:36.764735+0000 osd.6 (osd.6) 106 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:38.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[62156]: cluster 2021-03-15T22:30:36.765188+0000 osd.6 (osd.6) 107 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:38.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[62156]: cluster 2021-03-15T22:30:37.146404+0000 osd.5 (osd.5) 14 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:30:38.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[62156]: cluster 2021-03-15T22:30:37.147867+0000 mgr.y (mgr.14144) 850 : cluster [DBG] pgmap v1175: 138 pgs: 3 active+clean+wait, 2 active+undersized+remapped, 133 active+clean; 510 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 10 op/s; 27 MiB/s, 33 objects/s recovering 2021-03-15T22:30:38.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[62156]: cluster 2021-03-15T22:30:37.321710+0000 osd.6 (osd.6) 108 : cluster 2021-03-15T22:30:38.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[62156]: [DBG] 3.10 deep-scrub ok 2021-03-15T22:30:38.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[62156]: cluster 2021-03-15T22:30:37.368438+0000 mon.a (mon.0) 1866 : cluster [DBG] osdmap e688: 8 total, 8 up, 8 in 2021-03-15T22:30:38.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:38 smithi076 conmon[62156]: cluster 2021-03-15T22:30:37.798383+0000 osd.0 (osd.0) 105 : cluster [DBG] 3.6 deep-scrub ok 2021-03-15T22:30:38.699 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:30:38.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:38.767 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:38.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:38.769 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: cluster 2021-03-15T22:30:36.764735+0000 osd.6 (osd.6) 106 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: cluster 2021-03-15T22:30:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: 2021-03-15T22:30:36.765188+0000 osd.6 (osd.6) 107 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: cluster 2021-03-15T22:30:37.146404+0000 osd.5 (osd.5) 14 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:30:38.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: cluster 2021-03-15T22:30:37.147867+0000 mgr.y 2021-03-15T22:30:38.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: (mgr.14144) 850 : cluster [DBG] pgmap v1175: 138 pgs: 3 active+clean+wait, 2 active+undersized+remapped, 133 active+clean; 510 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 3.3 MiB/s wr, 10 op/s; 27 MiB/s, 33 objects/s recovering 2021-03-15T22:30:38.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:30:38.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: 15T22:30:37.321710+0000 osd.6 ( 2021-03-15T22:30:38.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: osd.6) 108 : cluster [DBG] 3.10 deep-scrub ok 2021-03-15T22:30:38.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:30:38.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: -03-15T22:30:37.368438+0000 2021-03-15T22:30:38.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: mon.a (mon.0) 1866 : cluster [DBG] 2021-03-15T22:30:38.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: osdmap e688: 8 total, 8 up, 8 in 2021-03-15T22:30:38.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: cluster 2021-03-15T22:30: 2021-03-15T22:30:38.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: 37.798383+0000 osd.0 (osd.0) 105 2021-03-15T22:30:38.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:38 smithi125 conmon[56739]: : cluster [DBG] 3.6 deep-scrub ok 2021-03-15T22:30:38.870 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:38.895 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:38.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:38.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:38.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:39.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:39.022 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:39.023 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:39.124 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:39.150 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:39.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:39.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:39.253 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:39.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:39.276 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:39.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:39.378 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:39.391 INFO:tasks.rados.rados.0.smithi076.stdout:2526: done (2 left) 2021-03-15T22:30:39.391 INFO:tasks.rados.rados.0.smithi076.stdout:2527: read oid 15 snap -1 2021-03-15T22:30:39.391 INFO:tasks.rados.rados.0.smithi076.stdout:2527: expect (ObjNum 767 snap 255 seq_num 767) 2021-03-15T22:30:39.392 INFO:tasks.rados.rados.0.smithi076.stdout:2528: read oid 45 snap 247 2021-03-15T22:30:39.392 INFO:tasks.rados.rados.0.smithi076.stdout:2528: expect (ObjNum 738 snap 243 seq_num 738) 2021-03-15T22:30:39.392 INFO:tasks.rados.rados.0.smithi076.stdout:2529: delete oid 7 current snap is 291 2021-03-15T22:30:39.392 INFO:tasks.rados.rados.0.smithi076.stdout:2525: finishing write tid 1 to smithi07694878-22 2021-03-15T22:30:39.393 INFO:tasks.rados.rados.0.smithi076.stdout:2525: finishing write tid 2 to smithi07694878-22 2021-03-15T22:30:39.393 INFO:tasks.rados.rados.0.smithi076.stdout:2525: finishing write tid 3 to smithi07694878-22 2021-03-15T22:30:39.393 INFO:tasks.rados.rados.0.smithi076.stdout:2525: finishing write tid 4 to smithi07694878-22 2021-03-15T22:30:39.393 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 1893 (ObjNum 875 snap 291 seq_num 875) dirty exists 2021-03-15T22:30:39.393 INFO:tasks.rados.rados.0.smithi076.stdout:2525: left oid 22 (ObjNum 875 snap 291 seq_num 875) 2021-03-15T22:30:39.402 INFO:tasks.rados.rados.0.smithi076.stdout:2527: expect (ObjNum 767 snap 255 seq_num 767) 2021-03-15T22:30:39.409 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:39.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:39.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:39.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:39.533 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:39.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:39.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:39.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-15T22:30:39.587 INFO:tasks.rados.rados.0.smithi076.stdout:2528: expect (ObjNum 738 snap 243 seq_num 738) 2021-03-15T22:30:39.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:39.660 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:39.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:39.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:39.667 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:39.671 INFO:tasks.rados.rados.0.smithi076.stdout:2525: done (4 left) 2021-03-15T22:30:39.671 INFO:tasks.rados.rados.0.smithi076.stdout:2527: done (3 left) 2021-03-15T22:30:39.671 INFO:tasks.rados.rados.0.smithi076.stdout:2528: done (2 left) 2021-03-15T22:30:39.671 INFO:tasks.rados.rados.0.smithi076.stdout:2529: done (1 left) 2021-03-15T22:30:39.672 INFO:tasks.rados.rados.0.smithi076.stdout:2530: read oid 44 snap 285 2021-03-15T22:30:39.672 INFO:tasks.rados.rados.0.smithi076.stdout:2530: expect (ObjNum 794 snap 261 seq_num 794) 2021-03-15T22:30:39.672 INFO:tasks.rados.rados.0.smithi076.stdout:2531: snap_create 2021-03-15T22:30:39.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[56514]: cluster 2021-03-15T22:30:37.992345+0000 osd.2 (osd.2) 5 : cluster [DBG] 2021-03-15T22:30:39.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[56514]: 3.14 scrub ok 2021-03-15T22:30:39.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[56514]: cluster 2021-03-15T22:30:38.178591+0000 osd.1 (osd.1) 134 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:30:39.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[56514]: cluster 2021-03-15T22:30:38.369686+0000 mon.a (mon.0) 1867 : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2021-03-15T22:30:39.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[56514]: cluster 2021-03-15T22:30:38.419544+0000 osd.4 (osd.4) 148 : cluster [DBG] 4.d scrub ok 2021-03-15T22:30:39.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[56514]: cluster 2021-03-15T22:30:38.781720+0000 osd.0 (osd.0) 106 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:30:39.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: cluster 2021-03-15T22:30:37.992345+0000 osd.2 2021-03-15T22:30:39.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: (osd.2) 5 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:30:39.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: cluster 2021-03-15T22:30:38.178591+0000 osd.1 (osd 2021-03-15T22:30:39.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: .1) 134 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:30:39.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: cluster 2021-03-15T22:30:39.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: 2021-03-15T22:30:38.369686+0000 mon.a (mon.0) 1867 2021-03-15T22:30:39.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2021-03-15T22:30:39.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: cluster 2021-03-15T22:30 2021-03-15T22:30:39.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: :38.419544+0000 osd.4 (osd.4) 148 : cluster 2021-03-15T22:30:39.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: [DBG] 4.d scrub ok 2021-03-15T22:30:39.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:30:39.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: :30:38.781720+0000 osd.0 (osd.0 2021-03-15T22:30:39.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:39 smithi076 conmon[62156]: ) 106 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:30:39.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:39.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: cluster 2021-03-15T22:30:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: 2021-03-15T22:30:37 2021-03-15T22:30:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: .992345+0000 osd.2 (osd.2) 5 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:30:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:30:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: 15T22:30:38.178591+0000 osd.1 2021-03-15T22:30:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: (osd.1) 134 : cluster 2021-03-15T22:30:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: [DBG] 4.12 scrub ok 2021-03-15T22:30:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:30:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: 03-15T22:30:38.369686+0000 2021-03-15T22:30:39.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: mon.a (mon.0) 1867 2021-03-15T22:30:39.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: : cluster [DBG] osdmap e689: 8 total, 8 up, 8 in 2021-03-15T22:30:39.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:30:39.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: 03-15T22:30:38.419544+0000 2021-03-15T22:30:39.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: osd.4 (osd.4) 148 : cluster 2021-03-15T22:30:39.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: [DBG] 4.d scrub ok 2021-03-15T22:30:39.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:30:39.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: -15T22:30:38.781720 2021-03-15T22:30:39.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: +0000 osd.0 (osd.0) 106 : cluster [DBG] 2021-03-15T22:30:39.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:39 smithi125 conmon[56739]: 3.1b scrub ok 2021-03-15T22:30:39.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:39.792 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:39.793 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:39.894 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:40.206 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:30:40.206 INFO:teuthology.orchestra.run.smithi076.stdout: "success": "filestore_inject_stall = '3' (not observed, change may require restart) " 2021-03-15T22:30:40.207 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:30:40.217 INFO:tasks.rados.rados.0.smithi076.stdout:2531: done (2 left) 2021-03-15T22:30:40.218 INFO:tasks.rados.rados.0.smithi076.stdout:2532: read oid 23 snap 281 2021-03-15T22:30:40.218 INFO:tasks.rados.rados.0.smithi076.stdout:2532: expect deleted 2021-03-15T22:30:40.218 INFO:tasks.rados.rados.0.smithi076.stdout:2533: read oid 35 snap 264 2021-03-15T22:30:40.219 INFO:tasks.rados.rados.0.smithi076.stdout:2533: expect (ObjNum 785 snap 261 seq_num 785) 2021-03-15T22:30:40.219 INFO:tasks.rados.rados.0.smithi076.stdout:2534: write (excl) oid 5 current snap is 292 2021-03-15T22:30:40.219 INFO:tasks.rados.rados.0.smithi076.stdout:2534: seq_num 877 ranges {739739=573968,2053414=686325} 2021-03-15T22:30:40.223 INFO:tasks.rados.rados.0.smithi076.stdout:2534: writing smithi07694878-5 from 739739 to 1313707 tid 1 2021-03-15T22:30:40.232 INFO:tasks.rados.rados.0.smithi076.stdout:2534: writing smithi07694878-5 from 2053414 to 2739739 tid 2 2021-03-15T22:30:40.232 INFO:tasks.rados.rados.0.smithi076.stdout:2535: snap_remove snap 284 2021-03-15T22:30:40.233 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:40.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:40.234 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:40.286 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:30:40.335 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:40.359 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:40.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:40.360 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:40.461 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:40.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:40.486 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:40.491 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:40.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:40.618 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:40.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:40.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:40.720 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:40.744 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:40.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:40.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[56514]: cluster 2021-03-15T22:30:39.012671+0000 osd.2 (osd.2) 6 : cluster [DBG] 3.3 deep-scrub ok 2021-03-15T22:30:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[56514]: cluster 2021-03-15T22:30:39.148339+0000 mgr.y (mgr.14144) 851 : cluster [DBG] pgmap v1178: 138 pgs: 3 active+clean+wait, 1 active+undersized+remapped, 134 active+clean; 512 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 10 op/s; 13 MiB/s, 21 objects/s recovering 2021-03-15T22:30:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[56514]: cluster 2021-03-15T22:30:39.381778+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2021-03-15T22:30:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[56514]: cluster 2021-03-15T22:30:39.443609+0000 osd.4 (osd.4) 149 : cluster [DBG] 3.1a deep-scrub ok 2021-03-15T22:30:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[56514]: cluster 2021-03-15T22:30:39.836160+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.4 deep-scrub ok 2021-03-15T22:30:40.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[56514]: audit 2021-03-15T22:30:40.010538+0000 mgr.y (mgr.14144) 852 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:40.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[56514]: cluster 2021-03-15T22:30:40.214672+0000 mon.a (mon.0) 1869 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2021-03-15T22:30:40.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[62156]: cluster 2021-03-15T22:30:39.012671+0000 osd.2 (osd.2) 6 : cluster [DBG] 3.3 deep-scrub ok 2021-03-15T22:30:40.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[62156]: cluster 2021-03-15T22:30:39.148339+0000 mgr.y (mgr.14144) 851 : cluster [DBG] pgmap v1178: 138 pgs: 3 active+clean+wait, 1 active+undersized+remapped, 134 active+clean; 512 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 10 op/s; 13 MiB/s, 21 objects/s recovering 2021-03-15T22:30:40.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[62156]: cluster 2021-03-15T22:30:39.381778+0000 mon.a (mon.0) 1868 : cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2021-03-15T22:30:40.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[62156]: cluster 2021-03-15T22:30:39.443609+0000 osd.4 (osd.4) 149 : cluster [DBG] 3.1a deep-scrub ok 2021-03-15T22:30:40.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[62156]: cluster 2021-03-15T22:30:39.836160+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.4 deep-scrub ok 2021-03-15T22:30:40.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[62156]: audit 2021-03-15T22:30:40.010538+0000 mgr.y (mgr.14144) 852 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:40.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[62156]: cluster 2021-03-15T22:30:40.214672+0000 mon.a (mon.0) 2021-03-15T22:30:40.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:40 smithi076 conmon[62156]: 1869 : cluster [DBG] osdmap e691: 8 total, 8 up, 8 in 2021-03-15T22:30:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: cluster 2021-03-15T22:30:39.012671+0000 osd.2 (osd.2) 6 : cluster [DBG] 2021-03-15T22:30:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: 3.3 deep-scrub ok 2021-03-15T22:30:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: cluster 2021-03-15T22:30:39.148339+0000 mgr.y (mgr.14144) 851 : cluster [DBG] pgmap v1178: 138 pgs: 3 active+clean+wait, 1 active+undersized+remapped, 134 active+clean; 512 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.4 MiB/s rd, 2.6 MiB/s wr, 10 op/s; 13 MiB/s, 21 objects/s recovering 2021-03-15T22:30:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: cluster 2021-03-15T22:30:39.381778+0000 mon.a (mon.0) 1868 : 2021-03-15T22:30:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: cluster [DBG] osdmap e690: 8 total, 8 up, 8 in 2021-03-15T22:30:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: cluster 2021-03-15T22:30:39.443609+0000 osd.4 (osd.4) 149 : cluster 2021-03-15T22:30:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: [DBG] 3.1a deep-scrub ok 2021-03-15T22:30:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: cluster 2021-03-15T22:30:39. 2021-03-15T22:30:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: 836160+0000 osd.0 (osd.0) 107 : cluster [DBG] 7.4 deep-scrub ok 2021-03-15T22:30:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: audit 2021 2021-03-15T22:30:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: -03-15T22:30:40.010538+0000 mgr.y (mgr.14144) 852 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:30:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: T22:30:40.214672+0000 mon.a (mon.0) 1869 : cluster [DBG] 2021-03-15T22:30:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:40 smithi125 conmon[56739]: osdmap e691: 8 total, 8 up, 8 in 2021-03-15T22:30:40.847 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:40.874 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:40.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:40.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:40.978 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:41.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:41.001 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:41.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:41.103 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:41.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:41.126 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:41.127 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:41.225 INFO:tasks.rados.rados.0.smithi076.stdout:2535: done (5 left) 2021-03-15T22:30:41.225 INFO:tasks.rados.rados.0.smithi076.stdout:2536: copy_from oid 18 from oid 40 current snap is 292 2021-03-15T22:30:41.225 INFO:tasks.rados.rados.0.smithi076.stdout:2537: rollback oid 49 current snap is 292 2021-03-15T22:30:41.226 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 49 to 290 2021-03-15T22:30:41.226 INFO:tasks.rados.rados.0.smithi076.stdout:2538: delete oid 42 current snap is 292 2021-03-15T22:30:41.226 INFO:tasks.rados.rados.0.smithi076.stdout:2533: expect (ObjNum 785 snap 261 seq_num 785) 2021-03-15T22:30:41.228 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:41.254 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:41.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:41.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:41.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:41.368 INFO:tasks.rados.rados.0.smithi076.stdout:2534: finishing write tid 1 to smithi07694878-5 2021-03-15T22:30:41.369 INFO:tasks.rados.rados.0.smithi076.stdout:2534: finishing write tid 2 to smithi07694878-5 2021-03-15T22:30:41.369 INFO:tasks.rados.rados.0.smithi076.stdout:2534: finishing write tid 3 to smithi07694878-5 2021-03-15T22:30:41.369 INFO:tasks.rados.rados.0.smithi076.stdout:2534: finishing write tid 4 to smithi07694878-5 2021-03-15T22:30:41.369 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1513 (ObjNum 877 snap 292 seq_num 877) dirty exists 2021-03-15T22:30:41.370 INFO:tasks.rados.rados.0.smithi076.stdout:2534: left oid 5 (ObjNum 877 snap 292 seq_num 877) 2021-03-15T22:30:41.370 INFO:tasks.rados.rados.0.smithi076.stdout:2532: done (7 left) 2021-03-15T22:30:41.370 INFO:tasks.rados.rados.0.smithi076.stdout:2533: done (6 left) 2021-03-15T22:30:41.370 INFO:tasks.rados.rados.0.smithi076.stdout:2534: done (5 left) 2021-03-15T22:30:41.371 INFO:tasks.rados.rados.0.smithi076.stdout:2538: done (4 left) 2021-03-15T22:30:41.371 INFO:tasks.rados.rados.0.smithi076.stdout:2539: write (excl) oid 42 current snap is 292 2021-03-15T22:30:41.371 INFO:tasks.rados.rados.0.smithi076.stdout:2537: finishing rollback tid 0 to smithi07694878-49 2021-03-15T22:30:41.371 INFO:tasks.rados.rados.0.smithi076.stdout:2537: finishing rollback tid 1 to smithi07694878-49 2021-03-15T22:30:41.371 INFO:tasks.rados.rados.0.smithi076.stdout:2537: finishing rollback tid 2 to smithi07694878-49 2021-03-15T22:30:41.372 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 1911 (ObjNum 862 snap 285 seq_num 862) dirty exists 2021-03-15T22:30:41.372 INFO:tasks.rados.rados.0.smithi076.stdout:2536: finishing copy_from to smithi07694878-18 2021-03-15T22:30:41.372 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 1945 (ObjNum 831 snap 276 seq_num 831) dirty exists 2021-03-15T22:30:41.372 INFO:tasks.rados.rados.0.smithi076.stdout:2536: finishing copy_from racing read to smithi07694878-18 2021-03-15T22:30:41.373 INFO:tasks.rados.rados.0.smithi076.stdout:2539: seq_num 879 ranges {773353=724466,2170378=602975} 2021-03-15T22:30:41.376 INFO:tasks.rados.rados.0.smithi076.stdout:2539: writing smithi07694878-42 from 773353 to 1497819 tid 1 2021-03-15T22:30:41.381 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:41.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:41.384 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:41.385 INFO:tasks.rados.rados.0.smithi076.stdout:2539: writing smithi07694878-42 from 2170378 to 2773353 tid 2 2021-03-15T22:30:41.385 INFO:tasks.rados.rados.0.smithi076.stdout:2536: done (4 left) 2021-03-15T22:30:41.385 INFO:tasks.rados.rados.0.smithi076.stdout:2537: done (3 left) 2021-03-15T22:30:41.385 INFO:tasks.rados.rados.0.smithi076.stdout:2540: rollback oid 25 current snap is 292 2021-03-15T22:30:41.386 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 255 2021-03-15T22:30:41.386 INFO:tasks.rados.rados.0.smithi076.stdout:2539: finishing write tid 1 to smithi07694878-42 2021-03-15T22:30:41.386 INFO:tasks.rados.rados.0.smithi076.stdout:2541: write oid 22 current snap is 292 2021-03-15T22:30:41.386 INFO:tasks.rados.rados.0.smithi076.stdout:2541: seq_num 880 ranges {790160=799715,2112507=509299} 2021-03-15T22:30:41.392 INFO:tasks.rados.rados.0.smithi076.stdout:2541: writing smithi07694878-22 from 790160 to 1589875 tid 1 2021-03-15T22:30:41.398 INFO:tasks.rados.rados.0.smithi076.stdout:2541: writing smithi07694878-22 from 2112507 to 2621806 tid 2 2021-03-15T22:30:41.398 INFO:tasks.rados.rados.0.smithi076.stdout:2542: rollback oid 26 current snap is 292 2021-03-15T22:30:41.399 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 286 2021-03-15T22:30:41.399 INFO:tasks.rados.rados.0.smithi076.stdout:2540: finishing rollback tid 0 to smithi07694878-25 2021-03-15T22:30:41.399 INFO:tasks.rados.rados.0.smithi076.stdout:2540: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:30:41.399 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 1749 (ObjNum 1953066355 snap 925919592 seq_num 942946614) dirty dne 2021-03-15T22:30:41.400 INFO:tasks.rados.rados.0.smithi076.stdout:2539: finishing write tid 2 to smithi07694878-42 2021-03-15T22:30:41.400 INFO:tasks.rados.rados.0.smithi076.stdout:2539: finishing write tid 3 to smithi07694878-42 2021-03-15T22:30:41.400 INFO:tasks.rados.rados.0.smithi076.stdout:2539: finishing write tid 4 to smithi07694878-42 2021-03-15T22:30:41.400 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2031 (ObjNum 879 snap 292 seq_num 879) dirty exists 2021-03-15T22:30:41.401 INFO:tasks.rados.rados.0.smithi076.stdout:2539: left oid 42 (ObjNum 879 snap 292 seq_num 879) 2021-03-15T22:30:41.401 INFO:tasks.rados.rados.0.smithi076.stdout:2541: finishing write tid 1 to smithi07694878-22 2021-03-15T22:30:41.401 INFO:tasks.rados.rados.0.smithi076.stdout:2539: done (5 left) 2021-03-15T22:30:41.401 INFO:tasks.rados.rados.0.smithi076.stdout:2540: done (4 left) 2021-03-15T22:30:41.401 INFO:tasks.rados.rados.0.smithi076.stdout:2543: write oid 18 current snap is 292 2021-03-15T22:30:41.402 INFO:tasks.rados.rados.0.smithi076.stdout:2543: seq_num 881 ranges {406967=474964,1370989=769310,2677186=129781} 2021-03-15T22:30:41.403 INFO:tasks.rados.rados.0.smithi076.stdout:2543: writing smithi07694878-18 from 406967 to 881931 tid 1 2021-03-15T22:30:41.410 INFO:tasks.rados.rados.0.smithi076.stdout:2543: writing smithi07694878-18 from 1370989 to 2140299 tid 2 2021-03-15T22:30:41.414 INFO:tasks.rados.rados.0.smithi076.stdout:2543: writing smithi07694878-18 from 2677186 to 2806967 tid 3 2021-03-15T22:30:41.415 INFO:tasks.rados.rados.0.smithi076.stdout:2544: snap_create 2021-03-15T22:30:41.415 INFO:tasks.rados.rados.0.smithi076.stdout:2542: finishing rollback tid 0 to smithi07694878-26 2021-03-15T22:30:41.415 INFO:tasks.rados.rados.0.smithi076.stdout:2541: finishing write tid 2 to smithi07694878-22 2021-03-15T22:30:41.415 INFO:tasks.rados.rados.0.smithi076.stdout:2541: finishing write tid 3 to smithi07694878-22 2021-03-15T22:30:41.416 INFO:tasks.rados.rados.0.smithi076.stdout:2541: finishing write tid 4 to smithi07694878-22 2021-03-15T22:30:41.416 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 1923 (ObjNum 880 snap 292 seq_num 880) dirty exists 2021-03-15T22:30:41.416 INFO:tasks.rados.rados.0.smithi076.stdout:2541: left oid 22 (ObjNum 880 snap 292 seq_num 880) 2021-03-15T22:30:41.416 INFO:tasks.rados.rados.0.smithi076.stdout:2542: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:30:41.416 INFO:tasks.rados.rados.0.smithi076.stdout:2542: finishing rollback tid 2 to smithi07694878-26 2021-03-15T22:30:41.417 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 1564 (ObjNum 850 snap 282 seq_num 850) dirty exists 2021-03-15T22:30:41.417 INFO:tasks.rados.rados.0.smithi076.stdout:2543: finishing write tid 1 to smithi07694878-18 2021-03-15T22:30:41.417 INFO:tasks.rados.rados.0.smithi076.stdout:2543: finishing write tid 2 to smithi07694878-18 2021-03-15T22:30:41.418 INFO:tasks.rados.rados.0.smithi076.stdout:2543: finishing write tid 3 to smithi07694878-18 2021-03-15T22:30:41.418 INFO:tasks.rados.rados.0.smithi076.stdout:2543: finishing write tid 4 to smithi07694878-18 2021-03-15T22:30:41.419 INFO:tasks.rados.rados.0.smithi076.stdout:2543: finishing write tid 5 to smithi07694878-18 2021-03-15T22:30:41.419 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 1949 (ObjNum 881 snap 292 seq_num 881) dirty exists 2021-03-15T22:30:41.419 INFO:tasks.rados.rados.0.smithi076.stdout:2543: left oid 18 (ObjNum 881 snap 292 seq_num 881) 2021-03-15T22:30:41.485 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:41.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:41.509 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:41.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:41.563 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:41.608 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[56514]: cluster 2021-03-15T22:30:40.173512+0000 osd.5 ( 2021-03-15T22:30:41.609 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[56514]: osd.5) 15 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:30:41.609 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[56514]: audit 2021-03-15T22:30:41.085041+0000 mon.a (mon.0) 1870 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:30:41.609 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[56514]: cluster 2021-03-15T22:30:41.221419+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2021-03-15T22:30:41.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[62156]: cluster 2021-03-15T22:30:40 2021-03-15T22:30:41.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[62156]: .173512+0000 osd.5 (osd.5) 15 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:30:41.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[62156]: audit 2021-03-15T22:30:41.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[62156]: 2021-03-15T22:30:41. 2021-03-15T22:30:41.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[62156]: 085041+0000 mon.a (mon.0) 1870 : audit 2021-03-15T22:30:41.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:30:41.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[62156]: cluster 2021-03-15T22:30:41.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[62156]: 2021-03-15T22:30: 2021-03-15T22:30:41.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[62156]: 41.221419+0000 mon.a (mon.0) 1871 : 2021-03-15T22:30:41.612 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:41 smithi076 conmon[62156]: cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2021-03-15T22:30:41.612 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:41.635 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:41.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:41.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:41.737 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:41.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:41.765 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:41.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:41.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:41 smithi125 conmon[56739]: cluster 2021-03-15T22:30:40.173512+0000 osd.5 (osd.5) 15 : cluster [DBG] 2021-03-15T22:30:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:41 smithi125 conmon[56739]: 6.19 scrub ok 2021-03-15T22:30:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:41 smithi125 conmon[56739]: audit 2021-03-15T22:30:41.085041+0000 mon.a (mon.0) 1870 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:30:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:41 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:30:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:41 smithi125 conmon[56739]: :30:41.221419+0000 mon.a (mon.0) 1871 : cluster [DBG] osdmap e692: 8 total, 8 up, 8 in 2021-03-15T22:30:41.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:41.891 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:41.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:41.893 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:41.994 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:42.019 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:42.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:42.020 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:42.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:42.146 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:42.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:42.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:42.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:42.274 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:42.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:42.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:42.377 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:42.399 INFO:tasks.rados.rados.0.smithi076.stdout:2541: done (5 left) 2021-03-15T22:30:42.399 INFO:tasks.rados.rados.0.smithi076.stdout:2542: done (4 left) 2021-03-15T22:30:42.400 INFO:tasks.rados.rados.0.smithi076.stdout:2543: done (3 left) 2021-03-15T22:30:42.400 INFO:tasks.rados.rados.0.smithi076.stdout:2544: done (2 left) 2021-03-15T22:30:42.400 INFO:tasks.rados.rados.0.smithi076.stdout:2545: write (excl) oid 5 current snap is 293 2021-03-15T22:30:42.400 INFO:tasks.rados.rados.0.smithi076.stdout:2545: seq_num 882 ranges {423774=666566,1545824=512968,2704609=119165} 2021-03-15T22:30:42.404 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:42.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:42.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:42.406 INFO:tasks.rados.rados.0.smithi076.stdout:2545: writing smithi07694878-5 from 423774 to 1090340 tid 1 2021-03-15T22:30:42.411 INFO:tasks.rados.rados.0.smithi076.stdout:2545: writing smithi07694878-5 from 1545824 to 2058792 tid 2 2021-03-15T22:30:42.415 INFO:tasks.rados.rados.0.smithi076.stdout:2545: writing smithi07694878-5 from 2704609 to 2823774 tid 3 2021-03-15T22:30:42.416 INFO:tasks.rados.rados.0.smithi076.stdout:2546: snap_create 2021-03-15T22:30:42.416 INFO:tasks.rados.rados.0.smithi076.stdout:2545: finishing write tid 1 to smithi07694878-5 2021-03-15T22:30:42.416 INFO:tasks.rados.rados.0.smithi076.stdout:2545: finishing write tid 2 to smithi07694878-5 2021-03-15T22:30:42.417 INFO:tasks.rados.rados.0.smithi076.stdout:2545: finishing write tid 3 to smithi07694878-5 2021-03-15T22:30:42.417 INFO:tasks.rados.rados.0.smithi076.stdout:2545: finishing write tid 4 to smithi07694878-5 2021-03-15T22:30:42.418 INFO:tasks.rados.rados.0.smithi076.stdout:2545: finishing write tid 5 to smithi07694878-5 2021-03-15T22:30:42.418 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1517 (ObjNum 882 snap 293 seq_num 882) dirty exists 2021-03-15T22:30:42.418 INFO:tasks.rados.rados.0.smithi076.stdout:2545: left oid 5 (ObjNum 882 snap 293 seq_num 882) 2021-03-15T22:30:42.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:42 smithi076 conmon[56514]: cluster 2021-03-15T22:30:41.148857+0000 mgr.y (mgr.14144) 853 : cluster [DBG] pgmap v1181: 138 pgs: 3 active+clean+wait, 135 active+clean; 522 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2021-03-15T22:30:42.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:42 smithi076 conmon[56514]: cluster 2021-03-15T22:30:41.171916+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:30:42.498 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:42 smithi076 conmon[56514]: cluster 2021-03-15T22:30:41.227464+0000 osd.5 (osd.5) 16 : cluster [DBG] 4.8 deep-scrub ok 2021-03-15T22:30:42.498 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:42 smithi076 conmon[56514]: cluster 2021-03-15T22:30:41.431045+0000 osd.4 (osd.4) 150 : cluster [DBG] 6.1a deep-scrub ok 2021-03-15T22:30:42.498 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:42 smithi076 conmon[56514]: cluster 2021-03-15T22:30:41.759220+0000 osd.0 (osd.0) 108 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:30:42.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:42 smithi076 conmon[62156]: cluster 2021-03-15T22:30:41.148857+0000 mgr.y (mgr.14144) 853 : cluster [DBG] pgmap v1181: 138 pgs: 3 active+clean+wait, 135 active+clean; 522 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2021-03-15T22:30:42.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:42 smithi076 conmon[62156]: cluster 2021-03-15T22:30:41.171916+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:30:42.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:42 smithi076 conmon[62156]: cluster 2021-03-15T22:30:41.227464+0000 osd.5 (osd.5) 16 : cluster [DBG] 4.8 deep-scrub ok 2021-03-15T22:30:42.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:42 smithi076 conmon[62156]: cluster 2021-03-15T22:30:41.431045+0000 osd.4 (osd.4) 150 : cluster 2021-03-15T22:30:42.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:42 smithi076 conmon[62156]: [DBG] 6.1a deep-scrub ok 2021-03-15T22:30:42.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:42 smithi076 conmon[62156]: cluster 2021-03-15T22:30:41.759220+0000 osd.0 (osd.0) 108 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:30:42.510 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:42.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:42.533 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:42.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:42.635 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:42.658 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:42.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:42.660 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:42.761 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:42.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: cluster 2021-03-15T22:30:41.148857+0000 mgr.y (mgr.14144) 853 : cluster 2021-03-15T22:30:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: [DBG] pgmap v1181: 138 pgs: 3 active+clean+wait, 135 active+clean; 522 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 10 op/s 2021-03-15T22:30:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: cluster 2021-03-15T22:30:41.171916+0000 osd.1 (osd.1) 135 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:30:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:30:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: T22:30:41.227464+0000 osd.5 (osd.5) 16 : cluster [DBG] 4.8 deep-scrub ok 2021-03-15T22:30:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:30:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: 30:41.431045+0000 osd.4 (osd.4) 2021-03-15T22:30:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: 150 : cluster [DBG] 6.1a deep-scrub ok 2021-03-15T22:30:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:30:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: -15T22:30:41.759220+0000 2021-03-15T22:30:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: osd.0 (osd.0) 108 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:30:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:42 smithi125 conmon[56739]: 2021-03-15T22:30:42.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:42.791 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:42.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:42.893 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:42.919 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:42.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:42.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:43.021 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:43.046 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:43.047 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:43.148 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:43.173 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:43.174 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:43.275 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:43.302 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:43.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:43.304 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:43.402 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:30:43.403 INFO:tasks.rados.rados.0.smithi076.stdout:2545: done (3 left) 2021-03-15T22:30:43.404 INFO:tasks.rados.rados.0.smithi076.stdout:2546: done (2 left) 2021-03-15T22:30:43.404 INFO:tasks.rados.rados.0.smithi076.stdout:2547: delete oid 9 current snap is 294 2021-03-15T22:30:43.404 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:43.407 INFO:tasks.rados.rados.0.smithi076.stdout:2547: done (2 left) 2021-03-15T22:30:43.408 INFO:tasks.rados.rados.0.smithi076.stdout:2548: copy_from oid 45 from oid 29 current snap is 294 2021-03-15T22:30:43.408 INFO:tasks.rados.rados.0.smithi076.stdout:2549: rollback oid 37 current snap is 294 2021-03-15T22:30:43.408 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 37 to 293 2021-03-15T22:30:43.408 INFO:tasks.rados.rados.0.smithi076.stdout:2550: rollback oid 4 current snap is 294 2021-03-15T22:30:43.409 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 4 to 285 2021-03-15T22:30:43.409 INFO:tasks.rados.rados.0.smithi076.stdout:2551: write (excl) oid 46 current snap is 294 2021-03-15T22:30:43.409 INFO:tasks.rados.rados.0.smithi076.stdout:2551: seq_num 884 ranges {457388=417064,1546435=516637,2643102=214286} 2021-03-15T22:30:43.412 INFO:tasks.rados.rados.0.smithi076.stdout:2551: writing smithi07694878-46 from 457388 to 874452 tid 1 2021-03-15T22:30:43.421 INFO:tasks.rados.rados.0.smithi076.stdout:2551: writing smithi07694878-46 from 1546435 to 2063072 tid 2 2021-03-15T22:30:43.425 INFO:tasks.rados.rados.0.smithi076.stdout:2551: writing smithi07694878-46 from 2643102 to 2857388 tid 3 2021-03-15T22:30:43.426 INFO:tasks.rados.rados.0.smithi076.stdout:2552: rollback oid 3 current snap is 294 2021-03-15T22:30:43.426 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 292 2021-03-15T22:30:43.426 INFO:tasks.rados.rados.0.smithi076.stdout:2550: finishing rollback tid 1 to smithi07694878-4 2021-03-15T22:30:43.426 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 1543 (ObjNum 1902431536 snap 32564 seq_num 264928784) dirty dne 2021-03-15T22:30:43.427 INFO:tasks.rados.rados.0.smithi076.stdout:2549: finishing rollback tid 0 to smithi07694878-37 2021-03-15T22:30:43.427 INFO:tasks.rados.rados.0.smithi076.stdout:2548: finishing copy_from to smithi07694878-45 2021-03-15T22:30:43.427 INFO:tasks.rados.rados.0.smithi076.stdout:2548: got expected ENOENT (src dne) 2021-03-15T22:30:43.427 INFO:tasks.rados.rados.0.smithi076.stdout:2548: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:30:43.427 INFO:tasks.rados.rados.0.smithi076.stdout:2549: finishing rollback tid 1 to smithi07694878-37 2021-03-15T22:30:43.428 INFO:tasks.rados.rados.0.smithi076.stdout:2549: finishing rollback tid 2 to smithi07694878-37 2021-03-15T22:30:43.428 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 2035 (ObjNum 751 snap 245 seq_num 751) dirty exists 2021-03-15T22:30:43.428 INFO:tasks.rados.rados.0.smithi076.stdout:2551: finishing write tid 1 to smithi07694878-46 2021-03-15T22:30:43.428 INFO:tasks.rados.rados.0.smithi076.stdout:2551: finishing write tid 2 to smithi07694878-46 2021-03-15T22:30:43.429 INFO:tasks.rados.rados.0.smithi076.stdout:2548: done (6 left) 2021-03-15T22:30:43.429 INFO:tasks.rados.rados.0.smithi076.stdout:2549: done (5 left) 2021-03-15T22:30:43.429 INFO:tasks.rados.rados.0.smithi076.stdout:2550: done (4 left) 2021-03-15T22:30:43.430 INFO:tasks.rados.rados.0.smithi076.stdout:2553: snap_remove snap 278 2021-03-15T22:30:43.430 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:30:43.436 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:43.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:43.437 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:43.447 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:30:43.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[56514]: cluster 2021-03-15T22:30:42 2021-03-15T22:30:43.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[56514]: .233094+0000 osd.5 (osd.5) 17 : cluster [DBG] 3.16 deep-scrub ok 2021-03-15T22:30:43.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[56514]: cluster 2021-03-15T22:30:42.383508+0000 osd.4 (osd.4) 151 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:30:43.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[56514]: cluster 2021-03-15T22:30:42.394056+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2021-03-15T22:30:43.448 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[56514]: cluster 2021-03-15T22:30:42.770972+0000 osd.0 (osd.0) 109 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:30:43.449 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[56514]: audit 2021-03-15T22:30:43.380193+0000 mon.a (mon.0) 1873 : audit [DBG] from='client.? 172.21.15.76:0/413687296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:43.449 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[62156]: cluster 2021-03-15T22:30:42.233094+0000 osd.5 ( 2021-03-15T22:30:43.449 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[62156]: osd.5) 17 : cluster [DBG] 3.16 deep-scrub ok 2021-03-15T22:30:43.449 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[62156]: cluster 2021-03-15T22:30:42.383508+0000 osd.4 (osd.4) 151 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:30:43.450 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[62156]: cluster 2021-03-15T22:30 2021-03-15T22:30:43.450 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[62156]: :42.394056+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2021-03-15T22:30:43.452 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[62156]: cluster 2021-03-15T22:30 2021-03-15T22:30:43.452 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[62156]: :42.770972+0000 osd.0 (osd.0) 2021-03-15T22:30:43.452 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[62156]: 109 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:30:43.453 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:30:43.453 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[62156]: :30:43.380193+0000 mon.a (mon. 2021-03-15T22:30:43.453 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:43 smithi076 conmon[62156]: 0) 1873 : audit [DBG] from='client.? 172.21.15.76:0/413687296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:43.467 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:30:43.486 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:30:43.504 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:30:43.522 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:30:43.538 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:43.542 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:30:43.543 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:30:43.543 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:30:43.543 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:30:43.544 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:30:43.544 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:30:43.544 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:30:43.544 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:30:43.545 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:30:43.565 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:43.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:43.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:43.667 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:43.693 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:43.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:43.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:43.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:43 smithi125 conmon[56739]: cluster 2021-03-15T22:30:42.233094+0000 osd.5 (osd.5) 17 : cluster 2021-03-15T22:30:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:43 smithi125 conmon[56739]: [DBG] 3.16 deep-scrub ok 2021-03-15T22:30:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:43 smithi125 conmon[56739]: cluster 2021-03-15T22:30:42.383508+0000 osd.4 (osd.4) 151 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:30:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:43 smithi125 conmon[56739]: cluster 2021-03-15T22:30:42 2021-03-15T22:30:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:43 smithi125 conmon[56739]: .394056+0000 mon.a (mon.0) 1872 : cluster [DBG] osdmap e693: 8 total, 8 up, 8 in 2021-03-15T22:30:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:43 smithi125 conmon[56739]: cluster 2021-03-15T22:30:42.770972+0000 osd.0 (osd 2021-03-15T22:30:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:43 smithi125 conmon[56739]: .0) 109 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:30:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:43 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:30:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:43 smithi125 conmon[56739]: :30:43.380193+0000 mon.a (mon.0) 1873 : audit [DBG] from='client.? 172.21.15.76:0/413687296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:43.797 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:43.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:43.824 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:43.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:43.926 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:43.951 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:43.953 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:44.054 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:44.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:44.077 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:44.078 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:44.107 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:30:44.107 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:30:44.179 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:44.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:44.203 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:44.204 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:44.305 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:44.327 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:44.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:44.411 INFO:tasks.rados.rados.0.smithi076.stdout:2553: done (4 left) 2021-03-15T22:30:44.411 INFO:tasks.rados.rados.0.smithi076.stdout:2554: snap_remove snap 289 2021-03-15T22:30:44.417 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:30:44.431 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:44.455 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:44.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:44.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:44.557 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:44.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:44.585 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:44.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:44.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[56514]: cluster 2021-03-15T22:30:42.834806+0000 osd.3 (osd.3) 127 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:44.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[56514]: cluster 2021-03-15T22:30:42.836838+0000 osd.3 (osd.3) 128 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:44.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[56514]: cluster 2021-03-15T22:30:43.021626+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:30:44.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[56514]: cluster 2021-03-15T22:30:43.149296+0000 mgr.y (mgr.14144) 854 : cluster [DBG] pgmap v1184: 138 pgs: 3 active+clean+wait, 135 active+clean; 525 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2021-03-15T22:30:44.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[56514]: cluster 2021-03-15T22:30:43.259577+0000 osd.6 (osd.6) 109 : cluster [DBG] 3.a scrub ok 2021-03-15T22:30:44.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[56514]: cluster 2021-03-15T22:30:43.399227+0000 mon.a (mon.0) 1874 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2021-03-15T22:30:44.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[56514]: audit 2021-03-15T22:30:43.812059+0000 mon.a (mon.0) 1875 : audit [INF] from='client.? 172.21.15.76:0/2891218352' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:30:44.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[62156]: cluster 2021-03-15T22:30:42.834806+0000 osd.3 (osd.3) 127 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:44.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[62156]: cluster 2021-03-15T22:30:42.836838+0000 osd.3 (osd.3) 128 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:44.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[62156]: cluster 2021-03-15T22:30:43.021626+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:30:44.661 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[62156]: cluster 2021-03-15T22:30:43.149296+0000 mgr.y (mgr.14144) 854 : cluster [DBG] pgmap v1184: 138 pgs: 3 active+clean+wait, 135 active+clean; 525 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2021-03-15T22:30:44.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[62156]: cluster 2021-03-15T22:30:43.259577+0000 osd.6 (osd.6) 109 : cluster [DBG] 3.a scrub ok 2021-03-15T22:30:44.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[62156]: cluster 2021-03-15T22:30:43.399227+0000 mon.a (mon.0) 1874 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2021-03-15T22:30:44.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:44 smithi076 conmon[62156]: audit 2021-03-15T22:30:43.812059+0000 mon.a (mon.0) 1875 : audit [INF] from='client.? 172.21.15.76:0/2891218352' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:30:44.686 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:44.710 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:44.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:44.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:44.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: cluster 2021-03-15T22:30:42.834806+0000 osd.3 (osd.3) 127 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: cluster 2021-03-15T22:30:42.836838+0000 osd.3 (osd.3) 128 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: cluster 2021-03-15T22:30:43.021626+0000 osd.2 (osd.2) 7 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:30:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: cluster 2021-03-15T22:30:43 2021-03-15T22:30:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: .149296+0000 mgr.y (mgr.14144) 854 : 2021-03-15T22:30:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: cluster [DBG] pgmap v1184: 138 pgs: 3 active+clean+wait, 135 active+clean; 525 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 5.7 MiB/s rd, 3.0 MiB/s wr, 14 op/s 2021-03-15T22:30:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:30:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: 30:43.259577+0000 osd.6 (osd.6 2021-03-15T22:30:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: ) 109 : cluster [DBG] 3.a scrub ok 2021-03-15T22:30:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:30:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: -15T22:30:43.399227+0000 2021-03-15T22:30:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: mon.a (mon.0) 1874 : cluster [DBG] osdmap e694: 8 total, 8 up, 8 in 2021-03-15T22:30:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: 2021-03-15T22:30:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:30:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: 30:43.812059+0000 mon.a (mon.0 2021-03-15T22:30:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:44 smithi125 conmon[56739]: ) 1875 : audit [INF] from='client.? 172.21.15.76:0/2891218352' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:30:44.813 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:44.835 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:44.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:44.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:44.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:44.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:44.969 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:44.975 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:45.079 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:45.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:45.119 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:45.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:45.219 INFO:tasks.rados.rados.0.smithi076.stdout:2554: done (4 left) 2021-03-15T22:30:45.220 INFO:tasks.rados.rados.0.smithi076.stdout:2555: snap_remove snap 287 2021-03-15T22:30:45.220 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:45.253 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:45.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:45.255 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:45.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:45.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:45.382 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:45.382 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:45.484 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:45.508 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:45.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:45.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:45.611 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:45.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:45.635 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:45.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:45.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[56514]: cluster 2021-03-15T22:30:44.116928+0000 osd.1 (osd.1) 136 : cluster [DBG] 2021-03-15T22:30:45.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[56514]: 5.12 deep-scrub ok 2021-03-15T22:30:45.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[56514]: cluster 2021-03-15T22:30:44.396351+0000 mon.a (mon.0) 1876 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:45.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[56514]: audit 2021-03-15T22:30:44.408354+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? 172.21.15.76:0/2891218352' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:30:45.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[56514]: cluster 2021-03-15T22:30:44.408415+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2021-03-15T22:30:45.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[56514]: cluster 2021-03-15T22:30:45.217881+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2021-03-15T22:30:45.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[62156]: cluster 2021-03-15T22:30:44.116928+0000 osd.1 (osd. 2021-03-15T22:30:45.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[62156]: 1) 136 : cluster [DBG] 5.12 deep-scrub ok 2021-03-15T22:30:45.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[62156]: cluster 2021-03-15T22:30:44.396351+0000 mon.a (mon.0) 1876 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:45.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[62156]: audit 2021-03-15T22:30:44.408354+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? 172.21.15.76:0/2891218352' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:30:45.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[62156]: cluster 2021-03-15T22:30:44.408415+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2021-03-15T22:30:45.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:45 smithi076 conmon[62156]: cluster 2021-03-15T22:30:45.217881+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2021-03-15T22:30:45.687 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-15T22:30:45.711 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:45 smithi125 conmon[56739]: cluster 2021-03-15T22:30:44.116928+0000 osd.1 (osd.1) 136 : cluster [DBG] 5.12 deep-scrub ok 2021-03-15T22:30:45.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:45 smithi125 conmon[56739]: cluster 2021-03-15T22:30:44.396351+0000 mon.a (mon.0) 1876 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:45.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:45 smithi125 conmon[56739]: audit 2021-03-15T22:30:44.408354+0000 mon.a (mon.0) 1877 : audit [INF] from='client.? 172.21.15.76:0/2891218352' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:30:45.712 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:45 smithi125 conmon[56739]: cluster 2021-03-15T22:30:44.408415+0000 mon.a (mon.0) 1878 : cluster [DBG] osdmap e695: 8 total, 8 up, 8 in 2021-03-15T22:30:45.713 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:45 smithi125 conmon[56739]: cluster 2021-03-15T22:30:45.217881+0000 mon.a (mon.0) 1879 : cluster [DBG] osdmap e696: 8 total, 8 up, 8 in 2021-03-15T22:30:45.736 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:45.760 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:45.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:45.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:45.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:45.885 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:45.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:45.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:45.938 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 0, 3, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:30:45.939 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:30:45.939 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump -f json-pretty 2021-03-15T22:30:45.973 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:30:45.988 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:46.011 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:46.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:46.013 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:46.114 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:46.137 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:46.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:46.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:46.223 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:46.225 INFO:tasks.rados.rados.0.smithi076.stdout:2555: done (4 left) 2021-03-15T22:30:46.225 INFO:tasks.rados.rados.0.smithi076.stdout:2556: read oid 2 snap -1 2021-03-15T22:30:46.225 INFO:tasks.rados.rados.0.smithi076.stdout:2556: expect (ObjNum 874 snap 287 seq_num 874) 2021-03-15T22:30:46.226 INFO:tasks.rados.rados.0.smithi076.stdout:2557: snap_remove snap 247 2021-03-15T22:30:46.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:46.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:46.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:46.367 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:46.390 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:46.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:46.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:46.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:46.516 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:46.517 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:46 smithi125 conmon[56739]: cluster 2021-03-15T22:30:45.149745+0000 mgr.y (mgr.14144) 855 : cluster [DBG] pgmap v1187: 138 pgs: 3 active+clean+wait, 135 active+clean; 527 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2021-03-15T22:30:46.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:46 smithi125 conmon[56739]: cluster 2021-03-15T22:30:46.223639+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2021-03-15T22:30:46.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:46.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:46.642 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:46.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:46.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:46 smithi076 conmon[56514]: cluster 2021-03-15T22:30:45.149745+0000 mgr.y (mgr.14144) 855 : cluster 2021-03-15T22:30:46.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:46 smithi076 conmon[56514]: [DBG] pgmap v1187: 138 pgs: 3 active+clean+wait, 135 active+clean; 527 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2021-03-15T22:30:46.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:46 smithi076 conmon[56514]: cluster 2021-03-15T22:30:46.223639+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2021-03-15T22:30:46.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:46 smithi076 conmon[62156]: cluster 2021-03-15T22:30:45.149745+0000 mgr.y (mgr.14144) 855 : 2021-03-15T22:30:46.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:46 smithi076 conmon[62156]: cluster [DBG] pgmap v1187: 138 pgs: 3 active+clean+wait, 135 active+clean; 527 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 9 op/s 2021-03-15T22:30:46.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:46 smithi076 conmon[62156]: cluster 2021-03-15T22:30:46.223639+0000 mon.a (mon.0) 1880 : cluster [DBG] osdmap e697: 8 total, 8 up, 8 in 2021-03-15T22:30:46.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:46.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:46.768 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:46.768 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:46.870 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:46.892 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:46.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:46.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:46.897 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:30:46.995 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:47.017 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:47.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:47.019 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:47.120 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:47.144 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:47.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:47.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:47.232 INFO:tasks.rados.rados.0.smithi076.stdout:2557: done (5 left) 2021-03-15T22:30:47.232 INFO:tasks.rados.rados.0.smithi076.stdout:2558: rollback oid 36 current snap is 294 2021-03-15T22:30:47.232 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 36 to 288 2021-03-15T22:30:47.233 INFO:tasks.rados.rados.0.smithi076.stdout:2559: rollback oid 40 current snap is 294 2021-03-15T22:30:47.233 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 40 to 220 2021-03-15T22:30:47.233 INFO:tasks.rados.rados.0.smithi076.stdout:2560: snap_remove snap 280 2021-03-15T22:30:47.247 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:47.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:47.271 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:47.271 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:47.373 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:47.397 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:47.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:47.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:47.477 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:47.499 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:47.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:47.524 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:47.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:47.597 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:30:47.598 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:30:47.626 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:47.650 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:47.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:47.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:47.660 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:30:47.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:47.776 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:47.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:47.778 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:47.879 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:47.908 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:47.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:47.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:48.011 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:48.033 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:48.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:48.035 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:48.066 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:30:48.066 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:30:48.066 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 698, 2021-03-15T22:30:48.067 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:30:48.067 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2021-03-15T22:13:20.370795+0000", 2021-03-15T22:30:48.067 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2021-03-15T22:30:47.220013+0000", 2021-03-15T22:30:48.067 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2021-03-15T22:30:30.205134+0000", 2021-03-15T22:30:48.068 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2021-03-15T22:29:50.191149+0000", 2021-03-15T22:30:48.068 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-03-15T22:30:48.068 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5801984, 2021-03-15T22:30:48.068 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2021-03-15T22:30:48.068 INFO:teuthology.orchestra.run.smithi076.stdout: "noscrub", 2021-03-15T22:30:48.069 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2021-03-15T22:30:48.069 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2021-03-15T22:30:48.069 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2021-03-15T22:30:48.069 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2021-03-15T22:30:48.070 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:48.070 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 20, 2021-03-15T22:30:48.070 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2021-03-15T22:30:48.070 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-03-15T22:30:48.070 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2021-03-15T22:30:48.071 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2021-03-15T22:30:48.071 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 7, 2021-03-15T22:30:48.071 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2021-03-15T22:30:48.071 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2021-03-15T22:30:48.072 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2021-03-15T22:30:48.072 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2021-03-15T22:30:48.072 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2021-03-15T22:30:48.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.073 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2021-03-15T22:30:48.073 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "device_health_metrics", 2021-03-15T22:30:48.073 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:16:34.817191+0000", 2021-03-15T22:30:48.073 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:48.074 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:48.074 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:48.075 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:48.075 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:48.075 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:48.075 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:48.076 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:48.076 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:48.076 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:48.076 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:48.076 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:48.077 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2021-03-15T22:30:48.077 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2021-03-15T22:30:48.077 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2021-03-15T22:30:48.077 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2021-03-15T22:30:48.078 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2021-03-15T22:30:48.078 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:48.078 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:48.078 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:48.078 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:48.079 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:48.079 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:48.079 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:48.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.080 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "23", 2021-03-15T22:30:48.080 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:48.080 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:48.080 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:48.080 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:30:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2021-03-15T22:30:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_devicehealth": {} 2021-03-15T22:30:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2021-03-15T22:30:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "iscsi", 2021-03-15T22:30:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:49.906400+0000", 2021-03-15T22:30:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 3, 2021-03-15T22:30:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 3, 2021-03-15T22:30:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 3, 2021-03-15T22:30:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 3, 2021-03-15T22:30:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 3, 2021-03-15T22:30:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "56", 2021-03-15T22:30:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:48.107 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:30:48.107 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:48.107 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2021-03-15T22:30:48.107 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.108 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.108 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.108 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2021-03-15T22:30:48.108 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".rgw.root", 2021-03-15T22:30:48.108 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:52.257970+0000", 2021-03-15T22:30:48.109 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:48.109 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:48.109 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:48.109 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:48.110 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:48.110 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:48.110 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:48.110 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:48.110 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:48.111 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:48.111 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:48.111 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:48.111 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:30:48.112 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:30:48.112 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:30:48.112 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:30:48.112 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:30:48.112 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:48.113 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:48.113 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:48.113 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:48.113 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:48.114 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:48.114 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:48.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.114 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "58", 2021-03-15T22:30:48.114 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:48.115 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:48.115 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:48.115 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:48.115 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:48.116 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:48.116 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:48.116 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:48.117 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:48.117 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:48.117 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:48.117 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:48.118 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:48.118 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:48.118 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:48.118 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:48.118 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:48.119 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:48.119 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:48.119 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:48.119 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:48.120 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:48.120 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:48.120 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:48.120 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:48.120 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:48.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.121 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:48.121 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:48.121 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:48.122 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:48.122 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:48.122 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:48.122 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:48.122 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:48.123 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:48.123 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:48.123 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:48.123 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:30:48.124 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:48.124 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:30:48.124 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.125 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2021-03-15T22:30:48.125 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.log", 2021-03-15T22:30:48.125 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:54.562498+0000", 2021-03-15T22:30:48.125 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:48.125 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:48.126 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:48.126 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:48.126 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:48.127 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:48.127 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:48.127 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:48.128 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:48.128 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:48.128 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:48.128 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:48.129 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:30:48.129 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:30:48.129 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:30:48.129 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:30:48.129 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:30:48.130 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:48.130 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:48.130 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:48.130 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:48.131 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:48.131 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:48.131 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:48.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.132 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "60", 2021-03-15T22:30:48.132 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:48.132 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:48.133 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:48.133 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:48.133 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:48.133 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:48.133 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:48.138 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:48.138 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:48.139 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:48.139 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:48.139 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:48.139 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:48.140 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:48.140 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:48.140 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:48.140 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:48.141 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:48.141 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:48.141 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:48.141 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:48.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.142 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:48.142 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:48.142 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:48.142 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:48.143 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:48.143 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:48.143 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:48.143 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:48.144 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:48.144 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:48.144 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:48.144 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:30:48.145 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:48.145 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:30:48.145 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.146 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 5, 2021-03-15T22:30:48.146 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.control", 2021-03-15T22:30:48.146 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:56.243806+0000", 2021-03-15T22:30:48.146 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:48.146 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:48.147 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:48.147 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:48.148 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:48.148 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:48.150 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:48.150 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 22, 2021-03-15T22:30:48.150 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 22, 2021-03-15T22:30:48.150 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 22, 2021-03-15T22:30:48.151 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 22, 2021-03-15T22:30:48.151 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 22, 2021-03-15T22:30:48.151 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:48.151 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "5.16", 2021-03-15T22:30:48.152 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 182, 2021-03-15T22:30:48.152 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 181, 2021-03-15T22:30:48.152 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 181, 2021-03-15T22:30:48.152 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:48.152 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:48.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.153 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "182", 2021-03-15T22:30:48.153 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:48.153 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "182", 2021-03-15T22:30:48.154 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "180", 2021-03-15T22:30:48.154 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:48.154 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:48.154 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:48.154 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:48.155 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:48.155 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:48.155 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:48.155 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:48.156 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:48.156 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:48.156 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:48.157 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:48.157 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:48.157 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:48.157 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:48.158 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:48.158 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:48.158 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:48.159 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:48.159 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:48.159 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:48.159 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:48.160 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:48.160 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.160 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:48.161 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:48.161 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:48.161 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:48.161 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:48.161 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:48.162 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:48.162 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:48.162 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:48.162 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:48.162 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:48.163 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:30:48.163 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:48.163 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:30:48.163 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.164 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 6, 2021-03-15T22:30:48.164 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.meta", 2021-03-15T22:30:48.164 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:58.730534+0000", 2021-03-15T22:30:48.165 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:30:48.165 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:30:48.165 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:48.165 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:48.165 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:48.166 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:48.166 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:48.166 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:48.166 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:48.167 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:48.167 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:48.167 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:48.167 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:30:48.168 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:30:48.168 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:30:48.168 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:30:48.168 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:30:48.169 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:48.169 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:48.169 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:48.169 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:48.169 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:48.170 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:48.170 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:48.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.170 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "109", 2021-03-15T22:30:48.170 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:48.171 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:48.171 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:48.171 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:48.171 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:48.172 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:30:48.172 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:30:48.172 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:48.172 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:48.172 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:48.173 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:48.173 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:48.173 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:48.173 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:48.173 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:48.174 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:48.174 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:48.174 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:48.174 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:48.175 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:48.175 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:48.175 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:48.175 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:48.175 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:48.176 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:48.176 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:48.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.176 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:48.176 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:48.177 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:48.177 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:48.177 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:48.177 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:48.178 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:48.178 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:48.178 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:48.178 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:48.179 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:48.179 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:30:48.179 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_bias": 4, 2021-03-15T22:30:48.179 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 8 2021-03-15T22:30:48.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.180 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:48.180 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:30:48.180 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.181 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:30:48.181 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2021-03-15T22:30:48.181 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:20:03.377294+0000", 2021-03-15T22:30:48.181 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2021-03-15T22:30:48.182 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-03-15T22:30:48.182 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:30:48.182 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:30:48.182 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:30:48.183 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:30:48.183 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:30:48.183 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:30:48.183 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:30:48.183 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:30:48.184 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:30:48.184 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:30:48.184 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2021-03-15T22:30:48.184 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2021-03-15T22:30:48.185 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2021-03-15T22:30:48.185 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2021-03-15T22:30:48.185 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2021-03-15T22:30:48.185 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:30:48.185 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:30:48.186 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:30:48.186 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:30:48.186 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:30:48.186 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:30:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:30:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "698", 2021-03-15T22:30:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:30:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:30:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:30:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:30:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:30:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 562, 2021-03-15T22:30:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 698, 2021-03-15T22:30:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:30:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:30:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:30:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:30:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:30:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:30:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:30:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:30:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:30:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:30:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:30:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:30:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:30:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:30:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:30:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:30:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:30:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:30:48.193 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:30:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:30:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:30:48.194 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:30:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:30:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:30:48.195 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:30:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:30:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:30:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:30:48.196 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:30:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:30:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:30:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:30:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2021-03-15T22:30:48.197 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.198 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.198 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:48.198 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:48.198 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.199 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:30:48.199 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd", 2021-03-15T22:30:48.199 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:48.199 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:48.199 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:48.200 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:48.200 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:48.200 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:48.200 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 8, 2021-03-15T22:30:48.200 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 682, 2021-03-15T22:30:48.201 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:48.201 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:48.201 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:48.201 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.202 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.202 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2021-03-15T22:30:48.202 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:48.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.203 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.203 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2021-03-15T22:30:48.203 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:48.203 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.204 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.204 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:48.204 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.205 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.205 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2021-03-15T22:30:48.205 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:48.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.206 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.206 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2021-03-15T22:30:48.206 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:48.206 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.207 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.207 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:48.207 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.208 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.208 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2021-03-15T22:30:48.208 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:48.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.209 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.209 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2021-03-15T22:30:48.209 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:48.209 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.210 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.210 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:48.210 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.211 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.211 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2021-03-15T22:30:48.211 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:48.211 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.212 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.212 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2021-03-15T22:30:48.212 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:30:48.212 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.213 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.213 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/188228176", 2021-03-15T22:30:48.213 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/188228176", 2021-03-15T22:30:48.214 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/188228176", 2021-03-15T22:30:48.214 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/188228176", 2021-03-15T22:30:48.214 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:48.214 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:48.214 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:48.215 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.215 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:30:48.215 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526", 2021-03-15T22:30:48.216 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:48.216 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:48.216 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:48.216 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.999237060546875, 2021-03-15T22:30:48.217 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:48.217 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:48.217 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 13, 2021-03-15T22:30:48.217 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 684, 2021-03-15T22:30:48.217 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:48.218 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:48.218 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:48.218 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.218 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.219 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2021-03-15T22:30:48.219 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:48.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.220 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.222 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2021-03-15T22:30:48.222 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:48.222 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.223 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.223 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:48.223 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.224 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.224 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2021-03-15T22:30:48.224 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:48.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.225 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.225 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2021-03-15T22:30:48.225 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:48.226 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.226 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.226 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:48.226 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.227 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.227 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.227 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2021-03-15T22:30:48.228 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:48.228 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.228 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.228 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2021-03-15T22:30:48.229 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:48.229 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.229 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.229 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:48.230 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.230 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.230 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.230 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2021-03-15T22:30:48.231 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:48.231 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.231 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.231 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.231 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2021-03-15T22:30:48.232 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:30:48.232 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.232 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.233 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.233 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/3924955423", 2021-03-15T22:30:48.233 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/3924955423", 2021-03-15T22:30:48.233 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/3924955423", 2021-03-15T22:30:48.234 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/3924955423", 2021-03-15T22:30:48.234 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:48.234 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:48.234 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:48.234 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.235 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.235 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.235 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:30:48.235 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05", 2021-03-15T22:30:48.236 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:48.236 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:48.236 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:48.236 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:48.236 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 18, 2021-03-15T22:30:48.237 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 489, 2021-03-15T22:30:48.237 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 681, 2021-03-15T22:30:48.237 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 682, 2021-03-15T22:30:48.237 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 490, 2021-03-15T22:30:48.237 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:48.238 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:48.238 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.238 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.238 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.239 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2021-03-15T22:30:48.239 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:30:48.239 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.239 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.240 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2021-03-15T22:30:48.240 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:30:48.240 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.240 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.241 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.241 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:48.241 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.241 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.241 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.242 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2021-03-15T22:30:48.242 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:30:48.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.242 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.243 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2021-03-15T22:30:48.243 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:30:48.243 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.243 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.244 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:48.244 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.244 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.244 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.245 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2021-03-15T22:30:48.245 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:30:48.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.245 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.246 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2021-03-15T22:30:48.246 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:30:48.246 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.246 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.247 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:48.247 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.247 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.248 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2021-03-15T22:30:48.248 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:30:48.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.248 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.249 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2021-03-15T22:30:48.249 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:30:48.249 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.249 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.250 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/1676192618", 2021-03-15T22:30:48.250 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/1676192618", 2021-03-15T22:30:48.250 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/1676192618", 2021-03-15T22:30:48.250 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/1676192618", 2021-03-15T22:30:48.251 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:48.251 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:48.251 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:48.251 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.252 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.252 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:30:48.252 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56", 2021-03-15T22:30:48.252 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:48.252 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:48.253 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:48.253 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:48.253 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:48.253 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:48.254 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 26, 2021-03-15T22:30:48.254 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 684, 2021-03-15T22:30:48.254 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:48.254 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:48.254 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:48.255 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.255 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.255 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2021-03-15T22:30:48.256 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:48.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.256 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.256 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2021-03-15T22:30:48.257 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:48.257 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.257 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.257 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:48.258 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.258 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.258 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2021-03-15T22:30:48.259 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:48.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.259 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.259 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2021-03-15T22:30:48.260 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:48.260 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.260 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.261 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:48.261 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.262 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.262 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2021-03-15T22:30:48.262 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:48.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.263 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.263 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2021-03-15T22:30:48.263 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:48.263 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.264 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.264 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:48.264 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.265 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.265 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2021-03-15T22:30:48.265 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:48.265 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.265 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.266 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.266 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2021-03-15T22:30:48.266 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:30:48.266 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.267 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.267 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.267 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/3960968636", 2021-03-15T22:30:48.267 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/3960968636", 2021-03-15T22:30:48.267 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/3960968636", 2021-03-15T22:30:48.268 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/3960968636", 2021-03-15T22:30:48.268 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:48.268 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:48.268 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:48.268 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:30:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237", 2021-03-15T22:30:48.269 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:48.270 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 31, 2021-03-15T22:30:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 682, 2021-03-15T22:30:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:48.271 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.272 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6800", 2021-03-15T22:30:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.273 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6801", 2021-03-15T22:30:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.274 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.275 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6802", 2021-03-15T22:30:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.276 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6803", 2021-03-15T22:30:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.278 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6806", 2021-03-15T22:30:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.279 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6807", 2021-03-15T22:30:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.280 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.281 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:48.281 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.281 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.282 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6804", 2021-03-15T22:30:48.282 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:48.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.282 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.283 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6805", 2021-03-15T22:30:48.283 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:30:48.283 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.283 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.284 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6801/37386122", 2021-03-15T22:30:48.284 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6803/37386122", 2021-03-15T22:30:48.284 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6807/37386122", 2021-03-15T22:30:48.289 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6805/37386122", 2021-03-15T22:30:48.289 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:48.289 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:48.289 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:48.290 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:30:48.291 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2", 2021-03-15T22:30:48.291 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:48.291 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:48.291 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:48.291 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:48.292 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 38, 2021-03-15T22:30:48.292 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 539, 2021-03-15T22:30:48.292 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 616, 2021-03-15T22:30:48.292 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 682, 2021-03-15T22:30:48.293 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 540, 2021-03-15T22:30:48.293 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:48.293 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:48.293 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.294 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.294 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6808", 2021-03-15T22:30:48.294 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:48.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.295 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.295 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6809", 2021-03-15T22:30:48.295 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:48.295 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.296 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.296 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:48.296 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.297 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.297 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6810", 2021-03-15T22:30:48.297 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:48.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.298 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.298 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6811", 2021-03-15T22:30:48.298 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:48.298 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.299 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.299 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.299 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:48.300 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.300 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.300 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6814", 2021-03-15T22:30:48.300 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:48.301 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.302 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.302 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.302 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6815", 2021-03-15T22:30:48.303 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:48.303 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.303 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.304 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:48.304 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.305 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.305 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6812", 2021-03-15T22:30:48.305 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:48.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.306 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.306 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6813", 2021-03-15T22:30:48.306 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:30:48.306 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.307 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.307 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6809/978017133", 2021-03-15T22:30:48.308 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6811/978017133", 2021-03-15T22:30:48.308 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6815/978017133", 2021-03-15T22:30:48.308 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6813/978017133", 2021-03-15T22:30:48.309 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:48.309 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:48.309 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:48.309 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.310 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:30:48.310 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe", 2021-03-15T22:30:48.311 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:48.311 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:48.311 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:48.312 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:48.312 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:48.312 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:48.312 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 45, 2021-03-15T22:30:48.313 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 681, 2021-03-15T22:30:48.313 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:48.313 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:48.313 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:48.314 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.314 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.315 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.315 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6816", 2021-03-15T22:30:48.315 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:48.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.316 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.316 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6817", 2021-03-15T22:30:48.316 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:48.317 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.317 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.317 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:48.318 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.318 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.318 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6818", 2021-03-15T22:30:48.318 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:48.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.319 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.319 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6819", 2021-03-15T22:30:48.320 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:48.320 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.320 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.320 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:48.321 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.321 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.321 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6822", 2021-03-15T22:30:48.322 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:48.322 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.323 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.323 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6823", 2021-03-15T22:30:48.323 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:48.323 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.323 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.324 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:48.324 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.325 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.325 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6820", 2021-03-15T22:30:48.326 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:48.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.326 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.326 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6821", 2021-03-15T22:30:48.327 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:30:48.327 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.327 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.328 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6817/171378275", 2021-03-15T22:30:48.328 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6819/171378275", 2021-03-15T22:30:48.328 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6823/171378275", 2021-03-15T22:30:48.328 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6821/171378275", 2021-03-15T22:30:48.328 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:48.329 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:48.329 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:48.329 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.330 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:30:48.330 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614", 2021-03-15T22:30:48.330 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:30:48.330 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:30:48.331 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:30:48.331 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:30:48.331 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:30:48.331 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:30:48.331 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 52, 2021-03-15T22:30:48.332 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 685, 2021-03-15T22:30:48.332 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:30:48.332 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:30:48.333 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:30:48.333 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.333 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.333 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6824", 2021-03-15T22:30:48.334 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:48.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.335 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.335 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6825", 2021-03-15T22:30:48.335 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:48.336 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.336 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.336 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:30:48.336 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.337 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.337 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6826", 2021-03-15T22:30:48.337 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:48.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.338 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.338 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6827", 2021-03-15T22:30:48.338 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:48.339 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.339 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.339 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.339 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:30:48.339 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.340 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.340 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6830", 2021-03-15T22:30:48.340 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:48.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.341 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.341 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6831", 2021-03-15T22:30:48.341 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:48.341 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.342 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.343 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:30:48.343 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:30:48.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.343 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:30:48.343 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6828", 2021-03-15T22:30:48.344 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:48.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.345 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:30:48.345 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6829", 2021-03-15T22:30:48.345 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:30:48.346 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.346 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.346 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6825/652003948", 2021-03-15T22:30:48.347 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6827/652003948", 2021-03-15T22:30:48.347 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6831/652003948", 2021-03-15T22:30:48.347 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6829/652003948", 2021-03-15T22:30:48.347 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:30:48.347 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:30:48.348 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:30:48.348 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.348 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:48.348 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2021-03-15T22:30:48.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.349 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:30:48.349 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:48.349 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:29:56.095504+0000", 2021-03-15T22:30:48.350 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:30:48.351 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:48.352 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:30:14.134385+0000", 2021-03-15T22:30:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:30:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:48.354 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:30:29.910315+0000", 2021-03-15T22:30:48.355 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 490 2021-03-15T22:30:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:30:48.356 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:48.357 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:30:42.836840+0000", 2021-03-15T22:30:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.358 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:30:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:48.359 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:48.360 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:30:02.645140+0000", 2021-03-15T22:30:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:30:48.361 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:48.362 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:48.362 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:48.363 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:48.363 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:48.363 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:30:25.166411+0000", 2021-03-15T22:30:48.364 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 540 2021-03-15T22:30:48.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.364 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:30:48.365 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:48.365 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:48.365 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:48.365 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:48.366 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:48.366 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:30:36.765191+0000", 2021-03-15T22:30:48.366 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:48.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.368 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:30:48.368 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:30:48.368 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:30:48.368 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:30:48.369 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:30:48.369 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:30:48.369 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:29:53.643176+0000", 2021-03-15T22:30:48.369 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:30:48.370 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.370 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:48.370 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [ 2021-03-15T22:30:48.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.370 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.3", 2021-03-15T22:30:48.371 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:30:48.371 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:30:48.371 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:30:48.371 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2021-03-15T22:30:48.372 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.372 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.372 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:48.373 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2021-03-15T22:30:48.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.373 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.3", 2021-03-15T22:30:48.373 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:30:48.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:30:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:30:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.14", 2021-03-15T22:30:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:30:48.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:30:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:30:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.a", 2021-03-15T22:30:48.378 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:30:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:30:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:30:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.379 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.b", 2021-03-15T22:30:48.380 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:30:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:30:48.381 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:30:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.382 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2021-03-15T22:30:48.383 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:30:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:30:48.384 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:30:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:48.385 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2021-03-15T22:30:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2021-03-15T22:30:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": { 2021-03-15T22:30:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:30:48.386 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4087324200": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:30:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4141143304": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:30:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:30:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2779879991": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:30:48.387 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:30:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:30:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2483102204": "2021-03-16T22:13:41.603652+0000" 2021-03-15T22:30:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2021-03-15T22:30:48.388 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2021-03-15T22:30:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2021-03-15T22:30:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2021-03-15T22:30:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2021-03-15T22:30:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2021-03-15T22:30:48.389 INFO:teuthology.orchestra.run.smithi076.stdout: "ruleset-failure-domain": "osd", 2021-03-15T22:30:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2021-03-15T22:30:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.390 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2021-03-15T22:30:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:30:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:30:48.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 469, 2021-03-15T22:30:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.392 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 504, 2021-03-15T22:30:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 534, 2021-03-15T22:30:48.393 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:48.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 545, 2021-03-15T22:30:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 550, 2021-03-15T22:30:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:30:48.396 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 552, 2021-03-15T22:30:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [ 2021-03-15T22:30:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:30:48.398 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:30:48.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:30:48.399 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 469, 2021-03-15T22:30:48.399 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:30:48.399 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:30:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:30:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2021-03-15T22:30:48.400 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2021-03-15T22:30:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2021-03-15T22:30:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2021-03-15T22:30:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2021-03-15T22:30:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2021-03-15T22:30:48.401 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2021-03-15T22:30:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2021-03-15T22:30:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2021-03-15T22:30:48.402 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:30:48.402 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:30:48.403 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:48.404 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:48.407 INFO:tasks.rados.rados.0.smithi076.stdout:2560: done (7 left) 2021-03-15T22:30:48.407 INFO:tasks.rados.rados.0.smithi076.stdout:2561: rollback oid 22 current snap is 294 2021-03-15T22:30:48.407 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 22 to 264 2021-03-15T22:30:48.407 INFO:tasks.rados.rados.0.smithi076.stdout:2552: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:30:48.408 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 1545 (ObjNum 255149944 snap 21896 seq_num 1) dirty dne 2021-03-15T22:30:48.408 INFO:tasks.rados.rados.0.smithi076.stdout:2551: finishing write tid 3 to smithi07694878-46 2021-03-15T22:30:48.408 INFO:tasks.rados.rados.0.smithi076.stdout:2551: finishing write tid 4 to smithi07694878-46 2021-03-15T22:30:48.408 INFO:tasks.rados.rados.0.smithi076.stdout:2551: finishing write tid 5 to smithi07694878-46 2021-03-15T22:30:48.409 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 1517 (ObjNum 884 snap 294 seq_num 884) dirty exists 2021-03-15T22:30:48.409 INFO:tasks.rados.rados.0.smithi076.stdout:2551: left oid 46 (ObjNum 884 snap 294 seq_num 884) 2021-03-15T22:30:48.409 INFO:tasks.rados.rados.0.smithi076.stdout:2530: expect (ObjNum 794 snap 261 seq_num 794) 2021-03-15T22:30:48.410 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-15T22:30:48.448 INFO:tasks.rados.rados.0.smithi076.stdout:2508: finishing copy_from to smithi07694878-27 2021-03-15T22:30:48.449 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1744 (ObjNum 854 snap 282 seq_num 854) dirty exists 2021-03-15T22:30:48.449 INFO:tasks.rados.rados.0.smithi076.stdout:2508: finishing copy_from racing read to smithi07694878-27 2021-03-15T22:30:48.449 INFO:tasks.rados.rados.0.smithi076.stdout:2556: expect (ObjNum 874 snap 287 seq_num 874) 2021-03-15T22:30:48.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:48.469 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:48.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:48 smithi076 conmon[56514]: cluster 2021-03-15T22:30:47.150204+0000 mgr.y (mgr.14144) 856 : cluster [DBG] pgmap v1190: 138 pgs: 3 active+clean+wait, 135 active+clean; 533 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-03-15T22:30:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:48 smithi076 conmon[56514]: cluster 2021-03-15T22:30:47.223883+0000 mon.a (mon.0) 1881 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2021-03-15T22:30:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:48 smithi076 conmon[56514]: audit 2021-03-15T22:30:48.062233+0000 mon.c (mon.1) 58 : audit [DBG] from='client.? 172.21.15.76:0/3003885071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:30:48.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:48 smithi076 conmon[62156]: cluster 2021-03-15T22:30:47.150204+0000 mgr.y (mgr.14144) 856 : cluster [DBG] pgmap v1190: 138 pgs: 3 active+clean+wait, 135 active+clean; 533 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-03-15T22:30:48.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:48 smithi076 conmon[62156]: cluster 2021-03-15T22:30:47.223883+0000 mon.a (mon.0) 1881 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2021-03-15T22:30:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:48 smithi076 conmon[62156]: audit 2021-03-15T22:30:48.062233+0000 mon.c (mon.1) 58 : audit [DBG] from='client.? 172.21.15.76:0/3003885071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:30:48.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:48 smithi125 conmon[56739]: cluster 2021-03-15T22:30:47.150204+0000 mgr.y (mgr.14144) 856 : cluster [DBG] pgmap v1190: 138 pgs: 3 active+clean+wait, 135 active+clean; 533 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.2 MiB/s wr, 9 op/s 2021-03-15T22:30:48.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:48 smithi125 conmon[56739]: cluster 2021-03-15T22:30:47.223883+0000 mon.a (mon.0) 1881 : cluster [DBG] osdmap e698: 8 total, 8 up, 8 in 2021-03-15T22:30:48.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:48 smithi125 conmon[56739]: audit 2021-03-15T22:30:48.062233+0000 mon.c (mon.1) 58 : audit [DBG] from='client.? 172.21.15.76:0/3003885071' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:30:48.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:48.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:48.596 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:48.597 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:48.677 INFO:tasks.rados.rados.0.smithi076.stdout:2558: finishing rollback tid 0 to smithi07694878-36 2021-03-15T22:30:48.677 INFO:tasks.rados.rados.0.smithi076.stdout:2558: finishing rollback tid 1 to smithi07694878-36 2021-03-15T22:30:48.677 INFO:tasks.rados.rados.0.smithi076.stdout:2559: finishing rollback tid 0 to smithi07694878-40 2021-03-15T22:30:48.678 INFO:tasks.rados.rados.0.smithi076.stdout:2558: finishing rollback tid 2 to smithi07694878-36 2021-03-15T22:30:48.678 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 1560 (ObjNum 824 snap 274 seq_num 824) dirty exists 2021-03-15T22:30:48.678 INFO:tasks.rados.rados.0.smithi076.stdout:2559: finishing rollback tid 1 to smithi07694878-40 2021-03-15T22:30:48.679 INFO:tasks.rados.rados.0.smithi076.stdout:2559: finishing rollback tid 2 to smithi07694878-40 2021-03-15T22:30:48.679 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 2061 (ObjNum 681 snap 218 seq_num 681) dirty exists 2021-03-15T22:30:48.679 INFO:tasks.rados.rados.0.smithi076.stdout:2508: done (7 left) 2021-03-15T22:30:48.680 INFO:tasks.rados.rados.0.smithi076.stdout:2530: done (6 left) 2021-03-15T22:30:48.680 INFO:tasks.rados.rados.0.smithi076.stdout:2551: done (5 left) 2021-03-15T22:30:48.680 INFO:tasks.rados.rados.0.smithi076.stdout:2552: done (4 left) 2021-03-15T22:30:48.680 INFO:tasks.rados.rados.0.smithi076.stdout:2556: done (3 left) 2021-03-15T22:30:48.681 INFO:tasks.rados.rados.0.smithi076.stdout:2558: done (2 left) 2021-03-15T22:30:48.681 INFO:tasks.rados.rados.0.smithi076.stdout:2559: done (1 left) 2021-03-15T22:30:48.681 INFO:tasks.rados.rados.0.smithi076.stdout:2562: delete oid 10 current snap is 294 2021-03-15T22:30:48.681 INFO:tasks.rados.rados.0.smithi076.stdout:2561: finishing rollback tid 0 to smithi07694878-22 2021-03-15T22:30:48.682 INFO:tasks.rados.rados.0.smithi076.stdout:2561: finishing rollback tid 1 to smithi07694878-22 2021-03-15T22:30:48.682 INFO:tasks.rados.rados.0.smithi076.stdout:2561: finishing rollback tid 2 to smithi07694878-22 2021-03-15T22:30:48.682 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 1954 (ObjNum 750 snap 245 seq_num 750) dirty exists 2021-03-15T22:30:48.683 INFO:tasks.rados.rados.0.smithi076.stdout:2561: done (1 left) 2021-03-15T22:30:48.683 INFO:tasks.rados.rados.0.smithi076.stdout:2562: done (0 left) 2021-03-15T22:30:48.684 INFO:tasks.rados.rados.0.smithi076.stdout:2563: write oid 13 current snap is 294 2021-03-15T22:30:48.684 INFO:tasks.rados.rados.0.smithi076.stdout:2563: seq_num 886 ranges {491002=567562,1663399=520306,2814301=76701} 2021-03-15T22:30:48.685 INFO:tasks.rados.rados.0.smithi076.stdout:2563: writing smithi07694878-13 from 491002 to 1058564 tid 1 2021-03-15T22:30:48.690 INFO:tasks.rados.rados.0.smithi076.stdout:2563: writing smithi07694878-13 from 1663399 to 2183705 tid 2 2021-03-15T22:30:48.695 INFO:tasks.rados.rados.0.smithi076.stdout:2563: writing smithi07694878-13 from 2814301 to 2891002 tid 3 2021-03-15T22:30:48.695 INFO:tasks.rados.rados.0.smithi076.stdout:2564: rollback oid 44 current snap is 294 2021-03-15T22:30:48.696 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 44 to 242 2021-03-15T22:30:48.696 INFO:tasks.rados.rados.0.smithi076.stdout:2563: finishing write tid 1 to smithi07694878-13 2021-03-15T22:30:48.696 INFO:tasks.rados.rados.0.smithi076.stdout:2563: finishing write tid 2 to smithi07694878-13 2021-03-15T22:30:48.696 INFO:tasks.rados.rados.0.smithi076.stdout:2565: snap_create 2021-03-15T22:30:48.698 INFO:tasks.rados.rados.0.smithi076.stdout:2564: finishing rollback tid 0 to smithi07694878-44 2021-03-15T22:30:48.698 INFO:tasks.rados.rados.0.smithi076.stdout:2564: finishing rollback tid 1 to smithi07694878-44 2021-03-15T22:30:48.698 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1746 (ObjNum 1 snap 6 seq_num 2) dirty dne 2021-03-15T22:30:48.698 INFO:tasks.rados.rados.0.smithi076.stdout:2563: finishing write tid 3 to smithi07694878-13 2021-03-15T22:30:48.699 INFO:tasks.rados.rados.0.smithi076.stdout:2563: finishing write tid 4 to smithi07694878-13 2021-03-15T22:30:48.699 INFO:tasks.rados.rados.0.smithi076.stdout:2563: finishing write tid 5 to smithi07694878-13 2021-03-15T22:30:48.699 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1564 (ObjNum 886 snap 294 seq_num 886) dirty exists 2021-03-15T22:30:48.700 INFO:tasks.rados.rados.0.smithi076.stdout:2563: left oid 13 (ObjNum 886 snap 294 seq_num 886) 2021-03-15T22:30:48.700 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:48.722 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:30:48.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:48.724 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:48.724 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:48.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:48.847 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:48.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:48.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:48.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:48.971 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:48.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:48.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:49.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:49.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:49.097 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:49.097 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:49.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:49.223 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:49.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:49.224 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:49.241 INFO:tasks.rados.rados.0.smithi076.stdout:2563: done (2 left) 2021-03-15T22:30:49.241 INFO:tasks.rados.rados.0.smithi076.stdout:2564: done (1 left) 2021-03-15T22:30:49.241 INFO:tasks.rados.rados.0.smithi076.stdout:2565: done (0 left) 2021-03-15T22:30:49.242 INFO:tasks.rados.rados.0.smithi076.stdout:2566: read oid 25 snap -1 2021-03-15T22:30:49.242 INFO:tasks.rados.rados.0.smithi076.stdout:2566: expect deleted 2021-03-15T22:30:49.242 INFO:tasks.rados.rados.0.smithi076.stdout:2567: copy_from oid 39 from oid 2 current snap is 295 2021-03-15T22:30:49.242 INFO:tasks.rados.rados.0.smithi076.stdout:2568: rollback oid 28 current snap is 295 2021-03-15T22:30:49.243 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 253 2021-03-15T22:30:49.243 INFO:tasks.rados.rados.0.smithi076.stdout:2569: read oid 15 snap -1 2021-03-15T22:30:49.243 INFO:tasks.rados.rados.0.smithi076.stdout:2569: expect (ObjNum 767 snap 255 seq_num 767) 2021-03-15T22:30:49.244 INFO:tasks.rados.rados.0.smithi076.stdout:2570: read oid 33 snap -1 2021-03-15T22:30:49.244 INFO:tasks.rados.rados.0.smithi076.stdout:2570: expect (ObjNum 854 snap 282 seq_num 854) 2021-03-15T22:30:49.244 INFO:tasks.rados.rados.0.smithi076.stdout:2571: rollback oid 24 current snap is 295 2021-03-15T22:30:49.244 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 286 2021-03-15T22:30:49.245 INFO:tasks.rados.rados.0.smithi076.stdout:2572: snap_create 2021-03-15T22:30:49.245 INFO:tasks.rados.rados.0.smithi076.stdout:2571: finishing rollback tid 0 to smithi07694878-24 2021-03-15T22:30:49.246 INFO:tasks.rados.rados.0.smithi076.stdout:2570: expect (ObjNum 854 snap 282 seq_num 854) 2021-03-15T22:30:49.325 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:49.347 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:49.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:49.349 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:49.385 INFO:tasks.rados.rados.0.smithi076.stdout:2569: expect (ObjNum 767 snap 255 seq_num 767) 2021-03-15T22:30:49.437 INFO:tasks.thrashosds.thrasher:j is {'epoch': 698, 'fsid': '8a572802-85db-11eb-9091-001a4aab830c', 'created': '2021-03-15T22:13:20.370795+0000', 'modified': '2021-03-15T22:30:47.220013+0000', 'last_up_change': '2021-03-15T22:30:30.205134+0000', 'last_in_change': '2021-03-15T22:29:50.191149+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-15T22:16:34.817191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-15T22:17:49.906400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-15T22:17:52.257970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-15T22:17:54.562498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-15T22:17:56.243806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 22, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 22, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 182, 'last_epoch_started': 181, 'last_epoch_clean': 181, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '182', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '182', 'last_force_op_resend_preluminous': '180', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-15T22:17:58.730534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '109', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-03-15T22:20:03.377294+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '698', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 562, 'snap_epoch': 698, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 682, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 188228176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 188228176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 188228176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 188228176}]}, 'public_addr': '172.21.15.76:6803/188228176', 'cluster_addr': '172.21.15.76:6805/188228176', 'heartbeat_back_addr': '172.21.15.76:6809/188228176', 'heartbeat_front_addr': '172.21.15.76:6807/188228176', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '97bd3ffc-15c8-4a6f-b37a-217813def526', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.999237060546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 684, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 3924955423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 3924955423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 3924955423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 3924955423}]}, 'public_addr': '172.21.15.76:6811/3924955423', 'cluster_addr': '172.21.15.76:6813/3924955423', 'heartbeat_back_addr': '172.21.15.76:6817/3924955423', 'heartbeat_front_addr': '172.21.15.76:6815/3924955423', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a7f5f115-e69e-47d9-b6c3-ff386ddfaa05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 489, 'up_from': 681, 'up_thru': 682, 'down_at': 490, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 1676192618}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 1676192618}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 1676192618}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 1676192618}]}, 'public_addr': '172.21.15.76:6819/1676192618', 'cluster_addr': '172.21.15.76:6821/1676192618', 'heartbeat_back_addr': '172.21.15.76:6825/1676192618', 'heartbeat_front_addr': '172.21.15.76:6823/1676192618', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a0bb7f88-0114-47cb-a82c-d8fb002aac56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 684, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 3960968636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 3960968636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 3960968636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 3960968636}]}, 'public_addr': '172.21.15.76:6827/3960968636', 'cluster_addr': '172.21.15.76:6829/3960968636', 'heartbeat_back_addr': '172.21.15.76:6833/3960968636', 'heartbeat_front_addr': '172.21.15.76:6831/3960968636', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b650611c-6101-47e8-9ca9-94e02cf8b237', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 682, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6800', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6801', 'nonce': 37386122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6802', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6803', 'nonce': 37386122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6806', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6807', 'nonce': 37386122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6804', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6805', 'nonce': 37386122}]}, 'public_addr': '172.21.15.125:6801/37386122', 'cluster_addr': '172.21.15.125:6803/37386122', 'heartbeat_back_addr': '172.21.15.125:6807/37386122', 'heartbeat_front_addr': '172.21.15.125:6805/37386122', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7d6ceff2-7093-4ab9-8b96-ade448093bf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 38, 'last_clean_end': 539, 'up_from': 616, 'up_thru': 682, 'down_at': 540, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6808', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6809', 'nonce': 978017133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6810', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6811', 'nonce': 978017133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6814', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6815', 'nonce': 978017133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6812', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6813', 'nonce': 978017133}]}, 'public_addr': '172.21.15.125:6809/978017133', 'cluster_addr': '172.21.15.125:6811/978017133', 'heartbeat_back_addr': '172.21.15.125:6815/978017133', 'heartbeat_front_addr': '172.21.15.125:6813/978017133', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98b72022-a9a7-4ee2-89a1-cac037de7afe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 681, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6816', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6817', 'nonce': 171378275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6818', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6819', 'nonce': 171378275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6822', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6823', 'nonce': 171378275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6820', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6821', 'nonce': 171378275}]}, 'public_addr': '172.21.15.125:6817/171378275', 'cluster_addr': '172.21.15.125:6819/171378275', 'heartbeat_back_addr': '172.21.15.125:6823/171378275', 'heartbeat_front_addr': '172.21.15.125:6821/171378275', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f99ea650-b2dc-4d87-a44c-56a897edd614', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 685, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6824', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6825', 'nonce': 652003948}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6826', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6827', 'nonce': 652003948}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6830', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6831', 'nonce': 652003948}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6828', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6829', 'nonce': 652003948}]}, 'public_addr': '172.21.15.125:6825/652003948', 'cluster_addr': '172.21.15.125:6827/652003948', 'heartbeat_back_addr': '172.21.15.125:6831/652003948', 'heartbeat_front_addr': '172.21.15.125:6829/652003948', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:29:56.095504+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:30:14.134385+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:30:29.910315+0000', 'dead_epoch': 490}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:30:42.836840+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:30:02.645140+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:30:25.166411+0000', 'dead_epoch': 540}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:30:36.765191+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:29:53.643176+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '7.3', 'osds': [1, 6, 2]}], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.14', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.76:6800/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4087324200': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4141143304': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:6801/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/2779879991': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6801/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6800/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:0/2483102204': '2021-03-16T22:13:41.603652+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 469, 'length': 1}, {'begin': 504, 'length': 1}, {'begin': 534, 'length': 1}, {'begin': 545, 'length': 1}, {'begin': 550, 'length': 1}, {'begin': 552, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 469, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-03-15T22:30:49.438 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:30:49.449 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:49.494 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:49 smithi076 conmon[56514]: cluster 2021-03-15T22:30:48. 2021-03-15T22:30:49.494 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:49 smithi076 conmon[56514]: 231325+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2021-03-15T22:30:49.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:49 smithi076 conmon[62156]: cluster 2021-03-15T22:30:48.231325+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2021-03-15T22:30:49.499 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:49.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:49.501 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:49.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:49 smithi125 conmon[56739]: cluster 2021-03-15T22:30:48.231325+0000 mon.a (mon.0) 1882 : cluster [DBG] osdmap e699: 8 total, 8 up, 8 in 2021-03-15T22:30:49.579 INFO:tasks.rados.rados.0.smithi076.stdout:2571: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:30:49.579 INFO:tasks.rados.rados.0.smithi076.stdout:2571: finishing rollback tid 2 to smithi07694878-24 2021-03-15T22:30:49.580 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1710 (ObjNum 864 snap 285 seq_num 864) dirty exists 2021-03-15T22:30:49.580 INFO:tasks.rados.rados.0.smithi076.stdout:2568: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:30:49.580 INFO:tasks.rados.rados.0.smithi076.stdout:2568: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:30:49.580 INFO:tasks.rados.rados.0.smithi076.stdout:2568: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:30:49.581 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2085 (ObjNum 609 snap 189 seq_num 609) dirty exists 2021-03-15T22:30:49.581 INFO:tasks.rados.rados.0.smithi076.stdout:2567: finishing copy_from to smithi07694878-39 2021-03-15T22:30:49.581 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 1747 (ObjNum 874 snap 287 seq_num 874) dirty exists 2021-03-15T22:30:49.581 INFO:tasks.rados.rados.0.smithi076.stdout:2567: finishing copy_from racing read to smithi07694878-39 2021-03-15T22:30:49.602 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:49.624 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:49.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:49.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:49.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:49.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:49.751 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:49.751 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:49.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:49.875 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:49.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:49.877 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:49.978 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:50.001 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:50.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:50.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:50.026 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:30:50 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:30:50.103 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:50.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:50.126 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:50.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:50.226 INFO:tasks.rados.rados.0.smithi076.stdout:2566: done (6 left) 2021-03-15T22:30:50.226 INFO:tasks.rados.rados.0.smithi076.stdout:2567: done (5 left) 2021-03-15T22:30:50.227 INFO:tasks.rados.rados.0.smithi076.stdout:2568: done (4 left) 2021-03-15T22:30:50.227 INFO:tasks.rados.rados.0.smithi076.stdout:2569: done (3 left) 2021-03-15T22:30:50.227 INFO:tasks.rados.rados.0.smithi076.stdout:2570: done (2 left) 2021-03-15T22:30:50.227 INFO:tasks.rados.rados.0.smithi076.stdout:2571: done (1 left) 2021-03-15T22:30:50.228 INFO:tasks.rados.rados.0.smithi076.stdout:2572: done (0 left) 2021-03-15T22:30:50.228 INFO:tasks.rados.rados.0.smithi076.stdout:2573: delete oid 14 current snap is 296 2021-03-15T22:30:50.228 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:50.230 INFO:tasks.rados.rados.0.smithi076.stdout:2573: done (0 left) 2021-03-15T22:30:50.231 INFO:tasks.rados.rados.0.smithi076.stdout:2574: rollback oid 31 current snap is 296 2021-03-15T22:30:50.231 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 31 to 264 2021-03-15T22:30:50.231 INFO:tasks.rados.rados.0.smithi076.stdout:2575: write oid 15 current snap is 296 2021-03-15T22:30:50.231 INFO:tasks.rados.rados.0.smithi076.stdout:2575: seq_num 888 ranges {524616=718060,1664010=523975,2752794=171822} 2021-03-15T22:30:50.237 INFO:tasks.rados.rados.0.smithi076.stdout:2575: writing smithi07694878-15 from 524616 to 1242676 tid 1 2021-03-15T22:30:50.244 INFO:tasks.rados.rados.0.smithi076.stdout:2575: writing smithi07694878-15 from 1664010 to 2187985 tid 2 2021-03-15T22:30:50.248 INFO:tasks.rados.rados.0.smithi076.stdout:2575: writing smithi07694878-15 from 2752794 to 2924616 tid 3 2021-03-15T22:30:50.249 INFO:tasks.rados.rados.0.smithi076.stdout:2576: rollback oid 3 current snap is 296 2021-03-15T22:30:50.249 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 281 2021-03-15T22:30:50.249 INFO:tasks.rados.rados.0.smithi076.stdout:2574: finishing rollback tid 0 to smithi07694878-31 2021-03-15T22:30:50.250 INFO:tasks.rados.rados.0.smithi076.stdout:2574: finishing rollback tid 1 to smithi07694878-31 2021-03-15T22:30:50.251 INFO:tasks.rados.rados.0.smithi076.stdout:2574: finishing rollback tid 2 to smithi07694878-31 2021-03-15T22:30:50.252 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 1849 (ObjNum 783 snap 260 seq_num 783) dirty exists 2021-03-15T22:30:50.252 INFO:tasks.rados.rados.0.smithi076.stdout:2575: finishing write tid 1 to smithi07694878-15 2021-03-15T22:30:50.253 INFO:tasks.rados.rados.0.smithi076.stdout:2575: finishing write tid 2 to smithi07694878-15 2021-03-15T22:30:50.253 INFO:tasks.rados.rados.0.smithi076.stdout:2574: done (2 left) 2021-03-15T22:30:50.253 INFO:tasks.rados.rados.0.smithi076.stdout:2577: delete oid 27 current snap is 296 2021-03-15T22:30:50.256 INFO:tasks.rados.rados.0.smithi076.stdout:2576: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:30:50.256 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 1545 (ObjNum 1677764096 snap 32564 seq_num 1677723856) dirty dne 2021-03-15T22:30:50.257 INFO:tasks.rados.rados.0.smithi076.stdout:2575: finishing write tid 3 to smithi07694878-15 2021-03-15T22:30:50.258 INFO:tasks.rados.rados.0.smithi076.stdout:2575: finishing write tid 4 to smithi07694878-15 2021-03-15T22:30:50.258 INFO:tasks.rados.rados.0.smithi076.stdout:2575: finishing write tid 5 to smithi07694878-15 2021-03-15T22:30:50.259 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 1969 (ObjNum 888 snap 296 seq_num 888) dirty exists 2021-03-15T22:30:50.259 INFO:tasks.rados.rados.0.smithi076.stdout:2575: left oid 15 (ObjNum 888 snap 296 seq_num 888) 2021-03-15T22:30:50.259 INFO:tasks.rados.rados.0.smithi076.stdout:2575: done (2 left) 2021-03-15T22:30:50.259 INFO:tasks.rados.rados.0.smithi076.stdout:2576: done (1 left) 2021-03-15T22:30:50.260 INFO:tasks.rados.rados.0.smithi076.stdout:2577: done (0 left) 2021-03-15T22:30:50.260 INFO:tasks.rados.rados.0.smithi076.stdout:2578: read oid 4 snap -1 2021-03-15T22:30:50.260 INFO:tasks.rados.rados.0.smithi076.stdout:2578: expect deleted 2021-03-15T22:30:50.260 INFO:tasks.rados.rados.0.smithi076.stdout:2579: snap_remove snap 266 2021-03-15T22:30:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:50.262 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:50.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:50.278 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:30:50.278 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:30:50.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:50 smithi125 conmon[56739]: cluster 2021-03-15T22:30:49.150581 2021-03-15T22:30:50.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:50 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 857 : cluster [DBG] pgmap v1193: 138 pgs: 3 active+clean+wait, 135 active+clean; 532 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 918 KiB/s wr, 5 op/s 2021-03-15T22:30:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:50 smithi125 conmon[56739]: cluster 2021-03-15T22:30:49.236325+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2021-03-15T22:30:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:50 smithi125 conmon[56739]: audit 2021-03-15T22:30:50.021303+0000 mgr.y (mgr.14144) 858 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:50 smithi125 conmon[56739]: cluster 2021-03-15T22:30:50.222900+0000 mon.a (mon.0) 1884 : cluster 2021-03-15T22:30:50.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:50 smithi125 conmon[56739]: [DBG] osdmap e701: 8 total, 8 up, 8 in 2021-03-15T22:30:50.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:50.388 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:50.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:50.389 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:50.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:50.503 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:50 smithi076 conmon[56514]: cluster 2021-03-15T22:30:49.150581 2021-03-15T22:30:50.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:50 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 857 : cluster [DBG] pgmap v1193: 138 pgs: 3 active+clean+wait, 135 active+clean; 532 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 918 KiB/s wr, 5 op/s 2021-03-15T22:30:50.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:50 smithi076 conmon[56514]: cluster 2021-03-15T22:30:49.236325+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2021-03-15T22:30:50.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:50 smithi076 conmon[56514]: audit 2021-03-15T22:30:50.021303+0000 mgr.y (mgr.14144) 858 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:50.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:50 smithi076 conmon[56514]: cluster 2021-03-15T22:30:50.222900+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2021-03-15T22:30:50.505 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:30:50.505 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:30:50.505 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:30:50.506 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:30:50.506 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:30:50.506 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:30:50.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:50 smithi076 conmon[62156]: cluster 2021-03-15T22:30:49.150581+0000 2021-03-15T22:30:50.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:50 smithi076 conmon[62156]: mgr.y (mgr.14144) 857 : cluster [DBG] pgmap v1193: 138 pgs: 3 active+clean+wait, 135 active+clean; 532 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 918 KiB/s wr, 5 op/s 2021-03-15T22:30:50.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:50 smithi076 conmon[62156]: cluster 2021-03-15T22:30:49.236325+0000 mon.a (mon.0) 1883 : cluster [DBG] osdmap e700: 8 total, 8 up, 8 in 2021-03-15T22:30:50.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:50 smithi076 conmon[62156]: audit 2021-03-15T22:30:50.021303+0000 mgr.y (mgr.14144) 858 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:30:50.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:50 smithi076 conmon[62156]: cluster 2021-03-15T22:30:50.222900+0000 mon.a (mon.0) 1884 : cluster [DBG] osdmap e701: 8 total, 8 up, 8 in 2021-03-15T22:30:50.512 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:30:50.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:50.514 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:50.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:50.531 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:30:50.532 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:30:50.532 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:30:50.532 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:30:50.533 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:30:50.533 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:30:50.533 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:30:50.533 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:30:50.534 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:30:50.588 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:50.615 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:50.637 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:50.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:50.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:50.739 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:50.761 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:50.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:50.763 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:50.864 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:50.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:50.889 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:50.889 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:50.921 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-15T22:30:50.991 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:51.014 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:51.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:51.015 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:51.116 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:51.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:51.141 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:51.142 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:51.192 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:30:51.243 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:51.246 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:51 smithi125 conmon[56739]: audit 2021-03-15T22:30:50.384596+0000 mon.a (mon.0) 1885 : audit [DBG] from='client.? 172.21.15.76:0/1585564029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:51.251 INFO:tasks.rados.rados.0.smithi076.stdout:2579: done (1 left) 2021-03-15T22:30:51.252 INFO:tasks.rados.rados.0.smithi076.stdout:2580: delete oid 11 current snap is 296 2021-03-15T22:30:51.267 INFO:tasks.rados.rados.0.smithi076.stdout:2578: done (1 left) 2021-03-15T22:30:51.268 INFO:tasks.rados.rados.0.smithi076.stdout:2580: done (0 left) 2021-03-15T22:30:51.268 INFO:tasks.rados.rados.0.smithi076.stdout:2581: snap_create 2021-03-15T22:30:51.274 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:51.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:51.276 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:51.377 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:51.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:51.401 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:51.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:51.503 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:51.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:51 smithi076 conmon[56514]: audit 2021-03-15T22:30:50.384596+0000 mon.a (mon.0) 1885 : audit [DBG] from='client.? 172.21.15.76:0/1585564029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:51.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:51 smithi076 conmon[62156]: audit 2021-03-15T22:30:50.384596+0000 mon.a (mon.0) 1885 : audit [DBG] from='client.? 172.21.15.76:0/1585564029' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:51.525 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:51.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:51.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:51.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:51.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:51.652 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:51.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:51.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:51.776 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:51.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:51.777 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:51.878 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:51.903 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:51.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:51.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:52.006 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:52.029 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:52.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:52.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:52.131 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:52.154 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:52.155 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:52.250 INFO:tasks.rados.rados.0.smithi076.stdout:2581: done (0 left) 2021-03-15T22:30:52.251 INFO:tasks.rados.rados.0.smithi076.stdout:2582: snap_remove snap 283 2021-03-15T22:30:52.253 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:30:52.256 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:52.278 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:52.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:52.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:52.382 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:52.406 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:52.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:52.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:52.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:52.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:52 smithi076 conmon[56514]: cluster 2021-03-15T22:30:51.151237+0000 mgr.y (mgr.14144) 859 : 2021-03-15T22:30:52.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:52 smithi076 conmon[56514]: cluster [DBG] pgmap v1196: 138 pgs: 138 active+clean; 536 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2021-03-15T22:30:52.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:52 smithi076 conmon[56514]: cluster 2021-03-15T22:30:51.246209+0000 mon.a (mon.0) 1886 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2021-03-15T22:30:52.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:52 smithi076 conmon[56514]: audit 2021-03-15T22:30:51.655029+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? 172.21.15.76:0/3459291159' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:52.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:52 smithi076 conmon[62156]: cluster 2021-03-15T22:30:51.151237 2021-03-15T22:30:52.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:52 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 859 : cluster [DBG] pgmap v1196: 138 pgs: 138 active+clean; 536 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2021-03-15T22:30:52.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:52 smithi076 conmon[62156]: cluster 2021-03-15T22:30:51.246209+0000 mon.a (mon.0) 1886 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2021-03-15T22:30:52.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:52 smithi076 conmon[62156]: audit 2021-03-15T22:30:51.655029+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? 172.21.15.76:0/3459291159' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:52.533 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:52.533 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:52.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:52 smithi125 conmon[56739]: cluster 2021-03-15T22:30:51.151237+0000 mgr.y (mgr.14144) 859 : cluster [DBG] pgmap v1196: 138 pgs: 138 active+clean; 536 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.2 MiB/s wr, 15 op/s 2021-03-15T22:30:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:52 smithi125 conmon[56739]: cluster 2021-03-15T22:30:51.246209+0000 mon.a (mon.0) 1886 : cluster [DBG] osdmap e702: 8 total, 8 up, 8 in 2021-03-15T22:30:52.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:52 smithi125 conmon[56739]: audit 2021-03-15T22:30:51.655029+0000 mon.a (mon.0) 1887 : audit [INF] from='client.? 172.21.15.76:0/3459291159' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:30:52.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:52.659 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:52.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:52.761 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:52.784 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:52.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:52.806 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:30:52.807 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:30:52.807 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:30:52.807 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:30:52.807 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.808 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:1427 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94751391774336 gen 3] snapc 0=[] ondisk+write+known_if_redirected e356)", 2021-03-15T22:30:52.808 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:32.074509+0000", 2021-03-15T22:30:52.808 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 380.72683725299999, 2021-03-15T22:30:52.808 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027614613999999999, 2021-03-15T22:30:52.808 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:52.809 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:30:52.809 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:52.809 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:30:52.809 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:30:52.810 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1427 2021-03-15T22:30:52.810 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.810 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:52.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.811 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:52.811 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074509+0000", 2021-03-15T22:30:52.811 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:52.811 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.812 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:52.812 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074509+0000", 2021-03-15T22:30:52.813 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9180000000000001e-06 2021-03-15T22:30:52.813 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.813 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.813 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:52.813 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074511+0000", 2021-03-15T22:30:52.814 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9599999999999999e-06 2021-03-15T22:30:52.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.814 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:52.815 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074513+0000", 2021-03-15T22:30:52.815 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0105000000000001e-05 2021-03-15T22:30:52.815 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.816 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:52.816 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074523+0000", 2021-03-15T22:30:52.816 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0859999999999998e-06 2021-03-15T22:30:52.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.817 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:52.817 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074529+0000", 2021-03-15T22:30:52.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5707000000000001e-05 2021-03-15T22:30:52.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.818 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074575+0000", 2021-03-15T22:30:52.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9250000000000001e-06 2021-03-15T22:30:52.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:30:52.819 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074584+0000", 2021-03-15T22:30:52.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013803967 2021-03-15T22:30:52.820 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.820 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.821 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.088388+0000", 2021-03-15T22:30:52.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5370999999999994e-05 2021-03-15T22:30:52.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.821 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:52.822 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.088463+0000", 2021-03-15T22:30:52.822 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013660574999999999 2021-03-15T22:30:52.822 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.822 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.823 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:52.823 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.102124+0000", 2021-03-15T22:30:52.823 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027614613999999999 2021-03-15T22:30:52.823 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.823 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:52.824 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.824 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.824 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.824 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:1429 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94751391774336 gen 4] snapc 0=[] ondisk+write+known_if_redirected e356)", 2021-03-15T22:30:52.825 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:32.074540+0000", 2021-03-15T22:30:52.825 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 380.72680610399999, 2021-03-15T22:30:52.826 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027747608999999999, 2021-03-15T22:30:52.826 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:52.826 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:30:52.828 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:52.828 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:30:52.828 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:30:52.828 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1429 2021-03-15T22:30:52.829 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.829 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:52.829 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.829 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:52.829 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074540+0000", 2021-03-15T22:30:52.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:52.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.830 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.830 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:52.831 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074540+0000", 2021-03-15T22:30:52.831 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.201e-06 2021-03-15T22:30:52.831 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.831 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:52.832 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074541+0000", 2021-03-15T22:30:52.832 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3400000000000001e-06 2021-03-15T22:30:52.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:52.833 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074543+0000", 2021-03-15T22:30:52.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0380000000000001e-05 2021-03-15T22:30:52.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.834 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.834 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:52.834 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074563+0000", 2021-03-15T22:30:52.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.938e-06 2021-03-15T22:30:52.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.835 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.835 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:52.835 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074568+0000", 2021-03-15T22:30:52.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8131999999999999e-05 2021-03-15T22:30:52.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.836 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.837 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074616+0000", 2021-03-15T22:30:52.837 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3170000000000002e-06 2021-03-15T22:30:52.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.838 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:30:52.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.074625+0000", 2021-03-15T22:30:52.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014177538 2021-03-15T22:30:52.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.088802+0000", 2021-03-15T22:30:52.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9495999999999998e-05 2021-03-15T22:30:52.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.840 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.840 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:52.841 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.088892+0000", 2021-03-15T22:30:52.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013396267 2021-03-15T22:30:52.841 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.841 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:52.842 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.102288+0000", 2021-03-15T22:30:52.842 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.027747608999999999 2021-03-15T22:30:52.842 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.842 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:52.843 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.843 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:3661 7.3 e385/265 7:c7825e8f:::smithi07694878-26:head v 385'344, mlcod=385'342)", 2021-03-15T22:30:52.843 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.131866+0000", 2021-03-15T22:30:52.844 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.66948041699999, 2021-03-15T22:30:52.844 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038125187999999997, 2021-03-15T22:30:52.844 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:52.844 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:52.845 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:52.845 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.845 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:52.845 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131866+0000", 2021-03-15T22:30:52.845 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:52.846 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.846 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:52.846 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131866+0000", 2021-03-15T22:30:52.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00042573300000000003 2021-03-15T22:30:52.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:52.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.132292+0000", 2021-03-15T22:30:52.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5669999999999999e-06 2021-03-15T22:30:52.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:52.851 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.132295+0000", 2021-03-15T22:30:52.851 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5919000000000001e-05 2021-03-15T22:30:52.851 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.852 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.852 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:52.854 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.132311+0000", 2021-03-15T22:30:52.854 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7169999999999996e-06 2021-03-15T22:30:52.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.856 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.856 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:52.856 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.132317+0000", 2021-03-15T22:30:52.856 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5943999999999998e-05 2021-03-15T22:30:52.857 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.857 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.858 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.132373+0000", 2021-03-15T22:30:52.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3580000000000001e-05 2021-03-15T22:30:52.858 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.859 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.859 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:52.860 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.132396+0000", 2021-03-15T22:30:52.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037522251999999999 2021-03-15T22:30:52.860 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.861 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:52.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.169919+0000", 2021-03-15T22:30:52.862 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2476000000000002e-05 2021-03-15T22:30:52.862 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:52.863 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.169991+0000", 2021-03-15T22:30:52.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038125187999999997 2021-03-15T22:30:52.863 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.863 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:52.864 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.864 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.864 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.864 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:3662 7.3 e385/265 7:c7825e8f:::smithi07694878-26:head v 385'345, mlcod=385'342)", 2021-03-15T22:30:52.865 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.133242+0000", 2021-03-15T22:30:52.865 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.66810465600003, 2021-03-15T22:30:52.865 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036775236000000003, 2021-03-15T22:30:52.865 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:52.866 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:52.866 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:52.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.867 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:52.867 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.133242+0000", 2021-03-15T22:30:52.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:52.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.868 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.868 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:52.868 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.133242+0000", 2021-03-15T22:30:52.868 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.095e-06 2021-03-15T22:30:52.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.869 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.869 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:52.869 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.133245+0000", 2021-03-15T22:30:52.870 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5019999999999999e-06 2021-03-15T22:30:52.870 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.870 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.870 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:52.870 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.133247+0000", 2021-03-15T22:30:52.871 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1681999999999999e-05 2021-03-15T22:30:52.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.871 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:52.872 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.133259+0000", 2021-03-15T22:30:52.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1100000000000002e-06 2021-03-15T22:30:52.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.872 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.873 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:52.873 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.133264+0000", 2021-03-15T22:30:52.873 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0408000000000003e-05 2021-03-15T22:30:52.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.874 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.874 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.133314+0000", 2021-03-15T22:30:52.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6376999999999998e-05 2021-03-15T22:30:52.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.875 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.875 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:52.876 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.133341+0000", 2021-03-15T22:30:52.876 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036655449999999999 2021-03-15T22:30:52.876 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.876 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.876 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:52.877 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.169996+0000", 2021-03-15T22:30:52.877 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0611999999999998e-05 2021-03-15T22:30:52.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.878 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:52.878 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.170017+0000", 2021-03-15T22:30:52.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036775236000000003 2021-03-15T22:30:52.879 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.879 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:52.879 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.880 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3668 7.4 7:26337d05:::smithi07694878-20:head [write 437114~654296,stat] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:30:52.880 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.177315+0000", 2021-03-15T22:30:52.880 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.62403100799997, 2021-03-15T22:30:52.881 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.056009570000000002, 2021-03-15T22:30:52.881 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:52.881 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:52.881 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:52.882 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:30:52.882 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:52.882 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3668 2021-03-15T22:30:52.882 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.883 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:52.883 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.883 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:52.883 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.177315+0000", 2021-03-15T22:30:52.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:52.884 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.884 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.884 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:52.885 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.177315+0000", 2021-03-15T22:30:52.885 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00066170699999999999 2021-03-15T22:30:52.885 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.885 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.885 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:52.886 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.177977+0000", 2021-03-15T22:30:52.886 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.241e-06 2021-03-15T22:30:52.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.886 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.887 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:52.887 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.177980+0000", 2021-03-15T22:30:52.887 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5600999999999999e-05 2021-03-15T22:30:52.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.887 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:52.888 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.177996+0000", 2021-03-15T22:30:52.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0100000000000001e-06 2021-03-15T22:30:52.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.889 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.889 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:52.889 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.178002+0000", 2021-03-15T22:30:52.890 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9178999999999995e-05 2021-03-15T22:30:52.890 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.891 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.891 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.891 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.178081+0000", 2021-03-15T22:30:52.892 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8926999999999998e-05 2021-03-15T22:30:52.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.892 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:52.893 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.178160+0000", 2021-03-15T22:30:52.893 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016705600000000001 2021-03-15T22:30:52.893 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.893 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.894 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,6", 2021-03-15T22:30:52.894 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.178327+0000", 2021-03-15T22:30:52.894 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0022729590000000002 2021-03-15T22:30:52.894 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.894 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.895 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:30:52.895 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.180600+0000", 2021-03-15T22:30:52.895 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00047166500000000002 2021-03-15T22:30:52.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:52.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.181072+0000", 2021-03-15T22:30:52.896 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052022769000000003 2021-03-15T22:30:52.896 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.897 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:52.897 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.233094+0000", 2021-03-15T22:30:52.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104242 2021-03-15T22:30:52.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.898 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.898 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:52.898 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.233199+0000", 2021-03-15T22:30:52.899 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012621400000000001 2021-03-15T22:30:52.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.899 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.899 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:52.899 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.233325+0000", 2021-03-15T22:30:52.900 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.056009570000000002 2021-03-15T22:30:52.900 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.900 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:52.900 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.901 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.901 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3669 7.4 7:26337d05:::smithi07694878-20:head [write 1525843~477506] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:30:52.901 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.182635+0000", 2021-03-15T22:30:52.902 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.61871179600001, 2021-03-15T22:30:52.902 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.053293172999999999, 2021-03-15T22:30:52.902 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:52.902 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:52.903 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:52.903 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:30:52.903 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:52.903 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3669 2021-03-15T22:30:52.903 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.904 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:52.904 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.904 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:52.905 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.182635+0000", 2021-03-15T22:30:52.905 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:52.905 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.905 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.906 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:52.906 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.182635+0000", 2021-03-15T22:30:52.906 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00049118000000000002 2021-03-15T22:30:52.906 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.906 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.907 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:52.907 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183126+0000", 2021-03-15T22:30:52.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8669999999999998e-06 2021-03-15T22:30:52.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.908 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.908 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:52.908 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183129+0000", 2021-03-15T22:30:52.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5970000000000001e-05 2021-03-15T22:30:52.909 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:52.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183145+0000", 2021-03-15T22:30:52.910 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6210000000000001e-06 2021-03-15T22:30:52.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.910 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.910 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:52.911 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183150+0000", 2021-03-15T22:30:52.911 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9876000000000002e-05 2021-03-15T22:30:52.911 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.911 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.911 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.912 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183200+0000", 2021-03-15T22:30:52.912 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5669999999999999e-05 2021-03-15T22:30:52.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.912 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.913 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:52.913 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183276+0000", 2021-03-15T22:30:52.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.050082922000000002 2021-03-15T22:30:52.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.914 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.914 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.914 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.233359+0000", 2021-03-15T22:30:52.914 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.546e-05 2021-03-15T22:30:52.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.915 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:52.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.233424+0000", 2021-03-15T22:30:52.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000119673 2021-03-15T22:30:52.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.916 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,6", 2021-03-15T22:30:52.916 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.233544+0000", 2021-03-15T22:30:52.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001677448 2021-03-15T22:30:52.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.918 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:30:52.918 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.235221+0000", 2021-03-15T22:30:52.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021573499999999999 2021-03-15T22:30:52.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.919 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.919 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:52.920 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.235437+0000", 2021-03-15T22:30:52.920 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00038745499999999998 2021-03-15T22:30:52.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.921 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:52.921 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.235824+0000", 2021-03-15T22:30:52.921 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6147000000000003e-05 2021-03-15T22:30:52.921 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.921 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:52.922 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.235881+0000", 2021-03-15T22:30:52.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7148999999999999e-05 2021-03-15T22:30:52.922 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.923 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:52.923 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.235928+0000", 2021-03-15T22:30:52.923 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.053293172999999999 2021-03-15T22:30:52.923 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.924 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:52.924 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.925 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3670 7.4 7:26337d05:::smithi07694878-20:head [setxattr _header (58) in=7b,truncate 2437114] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:30:52.925 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.183159+0000", 2021-03-15T22:30:52.925 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.61818722700002, 2021-03-15T22:30:52.926 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052893004, 2021-03-15T22:30:52.926 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:52.926 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:52.927 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:52.927 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:30:52.927 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:52.928 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3670 2021-03-15T22:30:52.928 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.928 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:52.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.929 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:52.929 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183159+0000", 2021-03-15T22:30:52.929 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:52.930 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.930 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.930 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:52.930 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183159+0000", 2021-03-15T22:30:52.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5179999999999999e-06 2021-03-15T22:30:52.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.931 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:52.931 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183161+0000", 2021-03-15T22:30:52.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0920000000000001e-06 2021-03-15T22:30:52.932 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.932 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.933 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:52.933 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183162+0000", 2021-03-15T22:30:52.933 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6379999999999998e-06 2021-03-15T22:30:52.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.934 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.934 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:52.934 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183168+0000", 2021-03-15T22:30:52.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.791e-06 2021-03-15T22:30:52.934 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:52.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183171+0000", 2021-03-15T22:30:52.935 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000137069 2021-03-15T22:30:52.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.936 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.936 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.936 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183308+0000", 2021-03-15T22:30:52.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8690999999999997e-05 2021-03-15T22:30:52.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.937 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.937 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:52.937 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183377+0000", 2021-03-15T22:30:52.938 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.051298522999999999 2021-03-15T22:30:52.938 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.938 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.938 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.939 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234675+0000", 2021-03-15T22:30:52.939 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8233999999999995e-05 2021-03-15T22:30:52.939 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.939 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:52.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234774+0000", 2021-03-15T22:30:52.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000140531 2021-03-15T22:30:52.940 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.941 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,6", 2021-03-15T22:30:52.941 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234914+0000", 2021-03-15T22:30:52.941 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00058197100000000003 2021-03-15T22:30:52.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.942 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:30:52.942 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.235496+0000", 2021-03-15T22:30:52.942 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010804800000000001 2021-03-15T22:30:52.942 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.943 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:52.943 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.235604+0000", 2021-03-15T22:30:52.943 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00035556999999999999 2021-03-15T22:30:52.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.944 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.944 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:52.944 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.235960+0000", 2021-03-15T22:30:52.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9796000000000003e-05 2021-03-15T22:30:52.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.945 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:52.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.236000+0000", 2021-03-15T22:30:52.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2531999999999998e-05 2021-03-15T22:30:52.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.946 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:52.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.236052+0000", 2021-03-15T22:30:52.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.052893004 2021-03-15T22:30:52.947 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.947 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:52.947 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.948 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3671 7.4 7:26337d05:::smithi07694878-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e385)", 2021-03-15T22:30:52.948 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.183193+0000", 2021-03-15T22:30:52.948 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 352.61815324200001, 2021-03-15T22:30:52.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.053081970999999999, 2021-03-15T22:30:52.949 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:52.949 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:30:52.949 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:52.949 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:30:52.950 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:52.950 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3671 2021-03-15T22:30:52.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.950 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:52.951 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.951 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:52.951 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183193+0000", 2021-03-15T22:30:52.951 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:52.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.952 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.952 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:52.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183193+0000", 2021-03-15T22:30:52.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3399999999999998e-07 2021-03-15T22:30:52.952 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.953 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:52.953 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183194+0000", 2021-03-15T22:30:52.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.158e-06 2021-03-15T22:30:52.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.954 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:52.954 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183195+0000", 2021-03-15T22:30:52.955 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1289999999999998e-06 2021-03-15T22:30:52.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.955 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.955 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:52.955 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183199+0000", 2021-03-15T22:30:52.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1550000000000004e-06 2021-03-15T22:30:52.956 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.956 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.956 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:52.957 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183205+0000", 2021-03-15T22:30:52.957 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020032200000000001 2021-03-15T22:30:52.957 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.957 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.957 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.958 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183406+0000", 2021-03-15T22:30:52.958 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4864000000000003e-05 2021-03-15T22:30:52.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.959 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:52.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.183450+0000", 2021-03-15T22:30:52.959 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.051803707999999997 2021-03-15T22:30:52.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.235254+0000", 2021-03-15T22:30:52.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.368e-05 2021-03-15T22:30:52.961 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.961 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:52.962 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.235318+0000", 2021-03-15T22:30:52.962 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000777143 2021-03-15T22:30:52.962 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.236095+0000", 2021-03-15T22:30:52.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0381000000000002e-05 2021-03-15T22:30:52.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.963 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:52.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.236175+0000", 2021-03-15T22:30:52.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9697000000000004e-05 2021-03-15T22:30:52.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:52.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.236275+0000", 2021-03-15T22:30:52.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.053081970999999999 2021-03-15T22:30:52.965 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.966 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:52.966 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.967 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5160 7.4 7:26337d05:::smithi07694878-20:head [write 3121574~676394] snapc 1a9=[1a9,1a7,1a6,1a4,1a3,1a2,1a1,1a0,19f,19c,19b,195,194,193,190,18a,184,181,180,17a,176,16e,161,15f,141,131,123,10c] ondisk+write+known_if_redirected e531)", 2021-03-15T22:30:52.967 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:27:26.396034+0000", 2021-03-15T22:30:52.967 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 206.405312079, 2021-03-15T22:30:52.968 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015295948, 2021-03-15T22:30:52.968 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:52.968 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:52.968 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:52.969 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:30:52.969 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:52.969 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5160 2021-03-15T22:30:52.969 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.970 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:52.970 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.970 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:52.970 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396034+0000", 2021-03-15T22:30:52.970 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:52.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.971 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:52.971 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396034+0000", 2021-03-15T22:30:52.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00063068499999999997 2021-03-15T22:30:52.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.972 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:52.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396665+0000", 2021-03-15T22:30:52.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8310000000000001e-06 2021-03-15T22:30:52.973 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.973 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:52.974 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396668+0000", 2021-03-15T22:30:52.974 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4318000000000001e-05 2021-03-15T22:30:52.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.974 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:52.975 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396682+0000", 2021-03-15T22:30:52.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9129999999999997e-06 2021-03-15T22:30:52.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.976 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.976 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:52.976 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396689+0000", 2021-03-15T22:30:52.976 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0819e-05 2021-03-15T22:30:52.977 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.977 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.977 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396770+0000", 2021-03-15T22:30:52.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9481000000000006e-05 2021-03-15T22:30:52.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.978 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:52.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396859+0000", 2021-03-15T22:30:52.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000199741 2021-03-15T22:30:52.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,6", 2021-03-15T22:30:52.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.397059+0000", 2021-03-15T22:30:52.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0071089159999999998 2021-03-15T22:30:52.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:30:52.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.404168+0000", 2021-03-15T22:30:52.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0069656370000000002 2021-03-15T22:30:52.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:52.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.411134+0000", 2021-03-15T22:30:52.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0154999999999998e-05 2021-03-15T22:30:52.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.983 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:52.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.411174+0000", 2021-03-15T22:30:52.984 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8379000000000002e-05 2021-03-15T22:30:52.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:52.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.411262+0000", 2021-03-15T22:30:52.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8072999999999997e-05 2021-03-15T22:30:52.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.985 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:52.986 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.411330+0000", 2021-03-15T22:30:52.986 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015295948 2021-03-15T22:30:52.986 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.986 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:52.987 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:52.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.987 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5161 7.4 7:26337d05:::smithi07694878-20:head [setxattr _header (58) in=7b,truncate 3865742] snapc 1a9=[1a9,1a7,1a6,1a4,1a3,1a2,1a1,1a0,19f,19c,19b,195,194,193,190,18a,184,181,180,17a,176,16e,161,15f,141,131,123,10c] ondisk+write+known_if_redirected e531)", 2021-03-15T22:30:52.988 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:27:26.396700+0000", 2021-03-15T22:30:52.988 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 206.40464624699999, 2021-03-15T22:30:52.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014828872, 2021-03-15T22:30:52.988 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:52.988 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:52.989 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:52.989 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:30:52.989 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:52.989 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5161 2021-03-15T22:30:52.989 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.990 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:52.990 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.990 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:52.990 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396700+0000", 2021-03-15T22:30:52.991 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:52.991 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.991 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:52.992 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396700+0000", 2021-03-15T22:30:52.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1869999999999999e-06 2021-03-15T22:30:52.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.992 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.993 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:52.993 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396704+0000", 2021-03-15T22:30:52.993 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3149999999999999e-06 2021-03-15T22:30:52.993 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.993 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:52.994 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396706+0000", 2021-03-15T22:30:52.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6049999999999998e-06 2021-03-15T22:30:52.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.995 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:52.995 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396714+0000", 2021-03-15T22:30:52.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3250000000000001e-06 2021-03-15T22:30:52.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:52.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396718+0000", 2021-03-15T22:30:52.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00141292 2021-03-15T22:30:52.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.997 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:52.997 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.398131+0000", 2021-03-15T22:30:52.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9964999999999996e-05 2021-03-15T22:30:52.998 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.998 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.998 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:52.998 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.398231+0000", 2021-03-15T22:30:52.999 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016899800000000001 2021-03-15T22:30:52.999 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:52.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:52.999 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,6", 2021-03-15T22:30:52.999 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.398400+0000", 2021-03-15T22:30:53.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0057752819999999996 2021-03-15T22:30:53.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:30:53.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.404176+0000", 2021-03-15T22:30:53.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0071978119999999996 2021-03-15T22:30:53.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.001 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:53.002 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.411373+0000", 2021-03-15T22:30:53.002 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1977999999999998e-05 2021-03-15T22:30:53.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:53.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.411405+0000", 2021-03-15T22:30:53.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9568000000000003e-05 2021-03-15T22:30:53.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:53.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.411465+0000", 2021-03-15T22:30:53.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3917000000000005e-05 2021-03-15T22:30:53.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:53.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.411529+0000", 2021-03-15T22:30:53.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014828872 2021-03-15T22:30:53.005 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.006 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.006 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.007 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5162 7.4 7:26337d05:::smithi07694878-20:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e531)", 2021-03-15T22:30:53.007 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:27:26.396723+0000", 2021-03-15T22:30:53.007 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 206.404623504, 2021-03-15T22:30:53.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015055254000000001, 2021-03-15T22:30:53.008 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:53.008 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:30:53.008 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:53.008 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:30:53.009 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:53.009 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5162 2021-03-15T22:30:53.009 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.009 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:53.009 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.010 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:53.010 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396723+0000", 2021-03-15T22:30:53.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:53.010 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.011 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.011 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:53.011 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396723+0000", 2021-03-15T22:30:53.011 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7199999999999997e-07 2021-03-15T22:30:53.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.012 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.012 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:53.012 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396724+0000", 2021-03-15T22:30:53.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.048e-06 2021-03-15T22:30:53.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.013 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.013 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:53.013 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396725+0000", 2021-03-15T22:30:53.013 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1170000000000004e-06 2021-03-15T22:30:53.014 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.014 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:53.014 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396733+0000", 2021-03-15T22:30:53.015 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8229999999999997e-06 2021-03-15T22:30:53.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.015 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:53.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.396737+0000", 2021-03-15T22:30:53.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020977249999999999 2021-03-15T22:30:53.016 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.017 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.398834+0000", 2021-03-15T22:30:53.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8238000000000005e-05 2021-03-15T22:30:53.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:53.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.398923+0000", 2021-03-15T22:30:53.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01263621 2021-03-15T22:30:53.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.018 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.411559+0000", 2021-03-15T22:30:53.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000102988 2021-03-15T22:30:53.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:53.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.411662+0000", 2021-03-15T22:30:53.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011613299999999999 2021-03-15T22:30:53.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:53.021 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:27:26.411778+0000", 2021-03-15T22:30:53.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.015055254000000001 2021-03-15T22:30:53.021 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.022 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.022 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.022 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5425 7.1 7:8bc62bc6:::smithi07694878-17:head [write 453987~645750] snapc 1c8=[1c7,1c5,1c4,1c3,1be,1bd,1bb,1b7,1b6,1b3,1ae,1a6,1a4,1a2,1a1,19f,19c,19b,195,194,193,18a,180,17a,176,16e,161,15f,141,123] ondisk+write+known_if_redirected e593)", 2021-03-15T22:30:53.023 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:02.361609+0000", 2021-03-15T22:30:53.023 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 110.439737361, 2021-03-15T22:30:53.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99827899399999998, 2021-03-15T22:30:53.023 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:53.024 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:53.024 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:53.024 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:30:53.024 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:53.024 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5425 2021-03-15T22:30:53.025 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.025 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:53.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:53.026 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.361609+0000", 2021-03-15T22:30:53.026 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:53.026 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.027 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:53.027 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.361609+0000", 2021-03-15T22:30:53.027 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001187671 2021-03-15T22:30:53.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.028 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:53.028 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.362797+0000", 2021-03-15T22:30:53.028 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0879999999999999e-06 2021-03-15T22:30:53.029 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.029 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.029 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:53.029 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.362800+0000", 2021-03-15T22:30:53.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3922e-05 2021-03-15T22:30:53.030 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.030 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.030 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:53.030 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.362814+0000", 2021-03-15T22:30:53.031 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6619999999999999e-06 2021-03-15T22:30:53.031 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.031 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.031 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:53.031 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.362820+0000", 2021-03-15T22:30:53.032 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014633499999999999 2021-03-15T22:30:53.032 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.032 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.033 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.362967+0000", 2021-03-15T22:30:53.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1061e-05 2021-03-15T22:30:53.033 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.033 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.033 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:30:53.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.362988+0000", 2021-03-15T22:30:53.034 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99275288800000006 2021-03-15T22:30:53.034 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.034 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.355741+0000", 2021-03-15T22:30:53.035 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017865200000000001 2021-03-15T22:30:53.035 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:53.036 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.355919+0000", 2021-03-15T22:30:53.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00029376899999999999 2021-03-15T22:30:53.036 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.037 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,6", 2021-03-15T22:30:53.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.356213+0000", 2021-03-15T22:30:53.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003111384 2021-03-15T22:30:53.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.038 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:30:53.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.359324+0000", 2021-03-15T22:30:53.038 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00035514100000000001 2021-03-15T22:30:53.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:53.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.359680+0000", 2021-03-15T22:30:53.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6195999999999997e-05 2021-03-15T22:30:53.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.040 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:53.040 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.359716+0000", 2021-03-15T22:30:53.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000100499 2021-03-15T22:30:53.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:53.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.359816+0000", 2021-03-15T22:30:53.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1725999999999997e-05 2021-03-15T22:30:53.042 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.042 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:53.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.359888+0000", 2021-03-15T22:30:53.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99827899399999998 2021-03-15T22:30:53.043 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.043 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.043 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.044 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.044 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.044 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5426 7.1 7:8bc62bc6:::smithi07694878-17:head [write 1845157~608830] snapc 1c8=[1c7,1c5,1c4,1c3,1be,1bd,1bb,1b7,1b6,1b3,1ae,1a6,1a4,1a2,1a1,19f,19c,19b,195,194,193,18a,180,17a,176,16e,161,15f,141,123] ondisk+write+known_if_redirected e593)", 2021-03-15T22:30:53.044 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:02.369754+0000", 2021-03-15T22:30:53.044 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 110.431591944, 2021-03-15T22:30:53.045 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99039227399999996, 2021-03-15T22:30:53.045 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:53.045 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:53.045 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:53.046 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:30:53.046 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:53.046 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5426 2021-03-15T22:30:53.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.046 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:53.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:53.047 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.369754+0000", 2021-03-15T22:30:53.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:53.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:53.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.369754+0000", 2021-03-15T22:30:53.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00055287100000000003 2021-03-15T22:30:53.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.049 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:53.049 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370307+0000", 2021-03-15T22:30:53.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5659999999999998e-06 2021-03-15T22:30:53.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.050 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:53.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370311+0000", 2021-03-15T22:30:53.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6181e-05 2021-03-15T22:30:53.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.051 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:53.052 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370327+0000", 2021-03-15T22:30:53.052 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8920000000000003e-06 2021-03-15T22:30:53.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.052 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:53.053 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370334+0000", 2021-03-15T22:30:53.053 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1977999999999997e-05 2021-03-15T22:30:53.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.054 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370376+0000", 2021-03-15T22:30:53.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7290000000000006e-06 2021-03-15T22:30:53.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.055 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:30:53.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370385+0000", 2021-03-15T22:30:53.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.987266965 2021-03-15T22:30:53.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.357652+0000", 2021-03-15T22:30:53.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3595000000000001e-05 2021-03-15T22:30:53.056 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.057 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:53.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.357745+0000", 2021-03-15T22:30:53.057 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000169965 2021-03-15T22:30:53.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.058 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,6", 2021-03-15T22:30:53.058 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.357915+0000", 2021-03-15T22:30:53.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001719613 2021-03-15T22:30:53.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:30:53.059 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.359635+0000", 2021-03-15T22:30:53.060 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034181100000000002 2021-03-15T22:30:53.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.060 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:53.060 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.359977+0000", 2021-03-15T22:30:53.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000101073 2021-03-15T22:30:53.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.061 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:53.062 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.360078+0000", 2021-03-15T22:30:53.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4696000000000001e-05 2021-03-15T22:30:53.062 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.062 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.062 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:53.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.360112+0000", 2021-03-15T22:30:53.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4338999999999998e-05 2021-03-15T22:30:53.063 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:53.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.360147+0000", 2021-03-15T22:30:53.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99039227399999996 2021-03-15T22:30:53.064 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.064 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.065 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.065 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.065 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5427 7.1 7:8bc62bc6:::smithi07694878-17:head [setxattr _header (58) in=7b,truncate 2453987] snapc 1c8=[1c7,1c5,1c4,1c3,1be,1bd,1bb,1b7,1b6,1b3,1ae,1a6,1a4,1a2,1a1,19f,19c,19b,195,194,193,18a,180,17a,176,16e,161,15f,141,123] ondisk+write+known_if_redirected e593)", 2021-03-15T22:30:53.066 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:02.370348+0000", 2021-03-15T22:30:53.066 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 110.430998243, 2021-03-15T22:30:53.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98999518399999997, 2021-03-15T22:30:53.066 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:53.066 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:53.067 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:53.067 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:30:53.067 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:53.068 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5427 2021-03-15T22:30:53.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.068 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:53.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:53.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370348+0000", 2021-03-15T22:30:53.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:53.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.070 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:53.070 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370348+0000", 2021-03-15T22:30:53.071 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8049999999999997e-06 2021-03-15T22:30:53.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.071 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.071 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:53.071 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370353+0000", 2021-03-15T22:30:53.072 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2699999999999999e-06 2021-03-15T22:30:53.072 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.072 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:53.073 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370354+0000", 2021-03-15T22:30:53.073 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2450000000000003e-06 2021-03-15T22:30:53.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.073 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.073 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:53.074 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370360+0000", 2021-03-15T22:30:53.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0210000000000001e-06 2021-03-15T22:30:53.074 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.075 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:53.075 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370364+0000", 2021-03-15T22:30:53.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5448000000000002e-05 2021-03-15T22:30:53.075 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.075 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.076 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.076 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370420+0000", 2021-03-15T22:30:53.077 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4859999999999997e-06 2021-03-15T22:30:53.077 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.077 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.078 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:30:53.078 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370428+0000", 2021-03-15T22:30:53.078 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.988351282 2021-03-15T22:30:53.078 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.078 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.079 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.079 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.358780+0000", 2021-03-15T22:30:53.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8525999999999999e-05 2021-03-15T22:30:53.079 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.080 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.080 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:53.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.358868+0000", 2021-03-15T22:30:53.080 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012925600000000001 2021-03-15T22:30:53.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,6", 2021-03-15T22:30:53.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.358997+0000", 2021-03-15T22:30:53.081 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00064413999999999997 2021-03-15T22:30:53.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:30:53.082 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.359642+0000", 2021-03-15T22:30:53.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00040310600000000003 2021-03-15T22:30:53.083 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.083 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:53.084 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.360045+0000", 2021-03-15T22:30:53.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014433499999999999 2021-03-15T22:30:53.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.084 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:30:53.085 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.360189+0000", 2021-03-15T22:30:53.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7483000000000004e-05 2021-03-15T22:30:53.085 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.086 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:53.086 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.360256+0000", 2021-03-15T22:30:53.086 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6780999999999995e-05 2021-03-15T22:30:53.086 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.086 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.087 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:53.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.360343+0000", 2021-03-15T22:30:53.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98999518399999997 2021-03-15T22:30:53.087 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.088 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.088 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.088 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.088 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5428 7.1 7:8bc62bc6:::smithi07694878-17:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e593)", 2021-03-15T22:30:53.089 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:02.370384+0000", 2021-03-15T22:30:53.089 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 110.430962428, 2021-03-15T22:30:53.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99014543099999996, 2021-03-15T22:30:53.089 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:53.090 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:30:53.090 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:53.090 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:30:53.090 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:30:53.090 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5428 2021-03-15T22:30:53.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.091 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:53.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.091 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:53.092 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370384+0000", 2021-03-15T22:30:53.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:53.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.092 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:53.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370384+0000", 2021-03-15T22:30:53.093 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9999999999999995e-07 2021-03-15T22:30:53.093 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.093 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:53.094 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370385+0000", 2021-03-15T22:30:53.094 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.136e-06 2021-03-15T22:30:53.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.095 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:53.095 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370386+0000", 2021-03-15T22:30:53.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4849999999999997e-06 2021-03-15T22:30:53.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.096 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.096 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:53.096 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370393+0000", 2021-03-15T22:30:53.096 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1350000000000002e-06 2021-03-15T22:30:53.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.097 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:53.097 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370397+0000", 2021-03-15T22:30:53.097 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7643999999999999e-05 2021-03-15T22:30:53.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.098 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.098 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370454+0000", 2021-03-15T22:30:53.098 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.054e-06 2021-03-15T22:30:53.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.099 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.099 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:30:53.099 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:02.370460+0000", 2021-03-15T22:30:53.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98875252199999997 2021-03-15T22:30:53.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.100 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.100 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.359213+0000", 2021-03-15T22:30:53.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7296000000000002e-05 2021-03-15T22:30:53.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.101 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:30:53.101 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.359270+0000", 2021-03-15T22:30:53.102 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011120030000000001 2021-03-15T22:30:53.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.102 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.102 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.360382+0000", 2021-03-15T22:30:53.103 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3190999999999999e-05 2021-03-15T22:30:53.103 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:53.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.360435+0000", 2021-03-15T22:30:53.104 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3964999999999999e-05 2021-03-15T22:30:53.104 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.104 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.105 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:53.105 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:03.360529+0000", 2021-03-15T22:30:53.105 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99014543099999996 2021-03-15T22:30:53.105 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.105 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.106 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.106 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.106 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.106 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:5460 7.8 e596/490 7:1d74a1c3:::smithi07694878-44:head v 596'487, mlcod=541'485)", 2021-03-15T22:30:53.107 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:05.433070+0000", 2021-03-15T22:30:53.107 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 107.368276526, 2021-03-15T22:30:53.107 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016992131000000001, 2021-03-15T22:30:53.107 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:53.107 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:53.108 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:53.108 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.108 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:53.109 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:05.433070+0000", 2021-03-15T22:30:53.109 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:53.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.110 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:53.110 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:05.433070+0000", 2021-03-15T22:30:53.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000207104 2021-03-15T22:30:53.110 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.111 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:53.111 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:05.433277+0000", 2021-03-15T22:30:53.111 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3050000000000001e-06 2021-03-15T22:30:53.111 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.112 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:53.112 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:05.433280+0000", 2021-03-15T22:30:53.112 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8088e-05 2021-03-15T22:30:53.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.113 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.113 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:53.113 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:05.433298+0000", 2021-03-15T22:30:53.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.379e-06 2021-03-15T22:30:53.113 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.114 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.114 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:53.114 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:05.433306+0000", 2021-03-15T22:30:53.114 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2021000000000002e-05 2021-03-15T22:30:53.115 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.115 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.115 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.115 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:05.433358+0000", 2021-03-15T22:30:53.115 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2756999999999999e-05 2021-03-15T22:30:53.116 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.116 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.116 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:53.116 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:05.433380+0000", 2021-03-15T22:30:53.117 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016642825999999999 2021-03-15T22:30:53.117 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.117 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.117 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:53.117 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:05.450023+0000", 2021-03-15T22:30:53.118 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8651e-05 2021-03-15T22:30:53.118 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.118 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.118 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:53.119 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:05.450062+0000", 2021-03-15T22:30:53.119 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016992131000000001 2021-03-15T22:30:53.119 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.119 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.119 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.120 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.120 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.120 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2101 5.e 5:7759931f:::notify.3:head [watch ping cookie 94751391774336 gen 11] snapc 0=[] ondisk+write+known_if_redirected e637)", 2021-03-15T22:30:53.120 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.130247+0000", 2021-03-15T22:30:53.121 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 60.671099404000003, 2021-03-15T22:30:53.121 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.082516276999999999, 2021-03-15T22:30:53.121 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:53.121 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:30:53.121 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:53.122 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:30:53.122 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:30:53.122 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2101 2021-03-15T22:30:53.122 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.123 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:53.123 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.123 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:53.123 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130247+0000", 2021-03-15T22:30:53.123 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:53.124 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.124 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:53.124 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130247+0000", 2021-03-15T22:30:53.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5880000000000001e-06 2021-03-15T22:30:53.125 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.125 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.125 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:53.125 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130248+0000", 2021-03-15T22:30:53.126 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4339999999999999e-06 2021-03-15T22:30:53.126 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.126 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.126 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:53.127 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130250+0000", 2021-03-15T22:30:53.127 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.325e-05 2021-03-15T22:30:53.127 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.127 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.128 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:53.128 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130263+0000", 2021-03-15T22:30:53.128 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7740000000000004e-06 2021-03-15T22:30:53.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.128 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.129 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:53.129 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130268+0000", 2021-03-15T22:30:53.129 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000429956 2021-03-15T22:30:53.129 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.129 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.130 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.130 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130698+0000", 2021-03-15T22:30:53.130 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3700000000000001e-06 2021-03-15T22:30:53.130 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.131 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:30:53.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130707+0000", 2021-03-15T22:30:53.131 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.081714125999999998 2021-03-15T22:30:53.132 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.132 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.132 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.132 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.212421+0000", 2021-03-15T22:30:53.132 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025477699999999998 2021-03-15T22:30:53.133 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.133 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:53.133 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.212676+0000", 2021-03-15T22:30:53.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7002000000000005e-05 2021-03-15T22:30:53.134 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.134 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.134 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:53.134 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.212763+0000", 2021-03-15T22:30:53.135 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.082516276999999999 2021-03-15T22:30:53.135 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.135 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.135 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.136 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.136 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.136 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2107 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94751391774336 gen 12] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:30:53.136 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.130736+0000", 2021-03-15T22:30:53.136 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 60.670610730999996, 2021-03-15T22:30:53.137 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.095780959999999998, 2021-03-15T22:30:53.137 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:53.137 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:30:53.137 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:53.137 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:30:53.138 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:30:53.138 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2107 2021-03-15T22:30:53.138 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.138 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:53.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:53.139 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130736+0000", 2021-03-15T22:30:53.139 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:53.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.140 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.140 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:53.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130736+0000", 2021-03-15T22:30:53.140 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7570000000000001e-06 2021-03-15T22:30:53.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.141 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:53.141 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130737+0000", 2021-03-15T22:30:53.141 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2340000000000001e-06 2021-03-15T22:30:53.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.142 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.142 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:53.142 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130740+0000", 2021-03-15T22:30:53.143 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2480000000000001e-05 2021-03-15T22:30:53.143 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.143 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.143 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:53.143 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130752+0000", 2021-03-15T22:30:53.144 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4450000000000001e-06 2021-03-15T22:30:53.144 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.144 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.144 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:53.145 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130758+0000", 2021-03-15T22:30:53.145 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9721e-05 2021-03-15T22:30:53.145 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.145 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.145 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.146 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130808+0000", 2021-03-15T22:30:53.146 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5119999999999997e-06 2021-03-15T22:30:53.146 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.146 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.147 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:30:53.147 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130815+0000", 2021-03-15T22:30:53.147 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.082021700000000003 2021-03-15T22:30:53.147 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.147 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.148 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.148 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.212836+0000", 2021-03-15T22:30:53.148 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7723000000000002e-05 2021-03-15T22:30:53.148 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.149 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.149 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:53.149 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.212904+0000", 2021-03-15T22:30:53.149 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013612388 2021-03-15T22:30:53.149 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.150 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.150 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:53.150 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226517+0000", 2021-03-15T22:30:53.150 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.095780959999999998 2021-03-15T22:30:53.151 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.151 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.151 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.151 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.151 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.152 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2110 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94751391774336 gen 13] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:30:53.152 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.130775+0000", 2021-03-15T22:30:53.152 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 60.670571514000002, 2021-03-15T22:30:53.152 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.095838382, 2021-03-15T22:30:53.153 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:53.153 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:30:53.153 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:30:53.153 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:30:53.153 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:30:53.154 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2110 2021-03-15T22:30:53.154 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.154 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:53.154 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.154 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:53.155 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130775+0000", 2021-03-15T22:30:53.155 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:53.155 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.155 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.156 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:53.156 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130775+0000", 2021-03-15T22:30:53.156 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3090000000000001e-06 2021-03-15T22:30:53.156 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.156 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.157 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:53.157 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130776+0000", 2021-03-15T22:30:53.157 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5400000000000001e-06 2021-03-15T22:30:53.157 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.158 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.158 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:53.158 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130778+0000", 2021-03-15T22:30:53.158 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5300000000000002e-06 2021-03-15T22:30:53.158 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.159 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:53.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130784+0000", 2021-03-15T22:30:53.159 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9339999999999999e-06 2021-03-15T22:30:53.160 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.160 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.160 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:53.160 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130788+0000", 2021-03-15T22:30:53.160 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4913000000000002e-05 2021-03-15T22:30:53.161 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.161 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.161 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.161 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130833+0000", 2021-03-15T22:30:53.162 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4140000000000002e-06 2021-03-15T22:30:53.162 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.162 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.162 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:30:53.162 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130838+0000", 2021-03-15T22:30:53.163 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.082450009000000005 2021-03-15T22:30:53.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.163 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.164 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.213288+0000", 2021-03-15T22:30:53.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4061999999999998e-05 2021-03-15T22:30:53.164 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.164 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.164 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:53.165 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.213353+0000", 2021-03-15T22:30:53.165 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013260671 2021-03-15T22:30:53.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.165 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.166 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:53.166 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226613+0000", 2021-03-15T22:30:53.166 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.095838382 2021-03-15T22:30:53.166 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.166 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.167 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.167 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.167 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.167 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:6453 7.9 e691/556 7:91d654eb:::smithi07694878-5:head v 691'298, mlcod=681'295)", 2021-03-15T22:30:53.168 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:30:40.297583+0000", 2021-03-15T22:30:53.168 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 12.503763069, 2021-03-15T22:30:53.168 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017079403999999999, 2021-03-15T22:30:53.168 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:30:53.168 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:30:53.169 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:30:53.169 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.169 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:30:53.169 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:40.297583+0000", 2021-03-15T22:30:53.170 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:30:53.170 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.170 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.170 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:30:53.170 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:40.297583+0000", 2021-03-15T22:30:53.171 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00063136300000000002 2021-03-15T22:30:53.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.171 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.171 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:30:53.172 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:40.298215+0000", 2021-03-15T22:30:53.172 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4539999999999999e-06 2021-03-15T22:30:53.172 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.172 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.172 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:30:53.173 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:40.298218+0000", 2021-03-15T22:30:53.173 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1987999999999999e-05 2021-03-15T22:30:53.173 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.173 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.174 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:30:53.174 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:40.298230+0000", 2021-03-15T22:30:53.174 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.28e-06 2021-03-15T22:30:53.174 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.174 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.175 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:30:53.175 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:40.298236+0000", 2021-03-15T22:30:53.175 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3662999999999999e-05 2021-03-15T22:30:53.175 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.176 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.176 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:30:53.176 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:40.298280+0000", 2021-03-15T22:30:53.176 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5137e-05 2021-03-15T22:30:53.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.177 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.177 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:30:53.177 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:40.298305+0000", 2021-03-15T22:30:53.178 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016315036000000002 2021-03-15T22:30:53.178 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.178 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:30:53.179 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:40.314620+0000", 2021-03-15T22:30:53.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2483e-05 2021-03-15T22:30:53.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:30:53.179 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:30:53.179 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:30:53.180 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:40.314663+0000", 2021-03-15T22:30:53.180 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017079403999999999 2021-03-15T22:30:53.180 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.180 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.180 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.181 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:30:53.181 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:30:53.181 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:30:53.182 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:53.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:53.206 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:53.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:53.263 INFO:tasks.rados.rados.0.smithi076.stdout:2582: done (0 left) 2021-03-15T22:30:53.263 INFO:tasks.rados.rados.0.smithi076.stdout:2583: read oid 44 snap 268 2021-03-15T22:30:53.263 INFO:tasks.rados.rados.0.smithi076.stdout:2583: expect (ObjNum 794 snap 261 seq_num 794) 2021-03-15T22:30:53.264 INFO:tasks.rados.rados.0.smithi076.stdout:2584: read oid 29 snap -1 2021-03-15T22:30:53.264 INFO:tasks.rados.rados.0.smithi076.stdout:2584: expect deleted 2021-03-15T22:30:53.264 INFO:tasks.rados.rados.0.smithi076.stdout:2585: rollback oid 9 current snap is 297 2021-03-15T22:30:53.264 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 9 to 290 2021-03-15T22:30:53.264 INFO:tasks.rados.rados.0.smithi076.stdout:2586: snap_create 2021-03-15T22:30:53.272 INFO:tasks.rados.rados.0.smithi076.stdout:2585: finishing rollback tid 1 to smithi07694878-9 2021-03-15T22:30:53.272 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 1888 (ObjNum 255149944 snap 21896 seq_num 1) dirty dne 2021-03-15T22:30:53.273 INFO:tasks.rados.rados.0.smithi076.stdout:2583: expect (ObjNum 794 snap 261 seq_num 794) 2021-03-15T22:30:53.307 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:53.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:53.331 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:53.332 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:53.349 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:30:53.412 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-15T22:30:53.433 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:53.458 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:53.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:53.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:53.498 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:53 smithi076 conmon[56514]: cluster 2021-03-15T22:30:52.241916+0000 mon.a (mon.0) 1888 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:53.499 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:53 smithi076 conmon[56514]: audit 2021-03-15T22:30:52.248727+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? 172.21.15.76:0/3459291159' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:53.499 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:53 smithi076 conmon[56514]: cluster 2021-03-15T22:30:52.248781+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2021-03-15T22:30:53.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:53 smithi076 conmon[62156]: cluster 2021-03-15T22:30:52.241916+0000 mon.a (mon.0) 1888 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:53.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:53 smithi076 conmon[62156]: audit 2021-03-15T22:30:52.248727+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? 172.21.15.76:0/3459291159' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:53.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:53 smithi076 conmon[62156]: cluster 2021-03-15T22:30:52.248781+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2021-03-15T22:30:53.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:53 smithi125 conmon[56739]: cluster 2021-03-15T22:30:52.241916+0000 mon.a (mon.0) 1888 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:30:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:53 smithi125 conmon[56739]: audit 2021-03-15T22:30:52.248727+0000 mon.a (mon.0) 1889 : audit [INF] from='client.? 172.21.15.76:0/3459291159' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:30:53.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:53 smithi125 conmon[56739]: cluster 2021-03-15T22:30:52.248781+0000 mon.a (mon.0) 1890 : cluster [DBG] osdmap e703: 8 total, 8 up, 8 in 2021-03-15T22:30:53.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:53.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:53.585 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:53.586 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:53.666 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:53.687 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:53.688 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:30:53.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:53.711 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:53.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:53.725 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:30:53.728 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:30:53.813 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:53.836 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:53.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:53.838 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:53.939 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:53.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:53.962 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:53.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:54.063 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:54.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:54.086 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:54.087 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:54.188 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:54.211 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:54.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:54.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:54.267 INFO:tasks.rados.rados.0.smithi076.stdout:2583: done (3 left) 2021-03-15T22:30:54.268 INFO:tasks.rados.rados.0.smithi076.stdout:2584: done (2 left) 2021-03-15T22:30:54.268 INFO:tasks.rados.rados.0.smithi076.stdout:2585: done (1 left) 2021-03-15T22:30:54.268 INFO:tasks.rados.rados.0.smithi076.stdout:2586: done (0 left) 2021-03-15T22:30:54.269 INFO:tasks.rados.rados.0.smithi076.stdout:2587: snap_create 2021-03-15T22:30:54.314 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:54.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:54.338 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:54.339 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:54.440 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:54.462 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:54.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:54.463 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:54.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56514]: cluster 2021-03-15T22:30:53 2021-03-15T22:30:54.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56514]: .151824+0000 mgr.y (mgr.14144) 860 : cluster [DBG] pgmap v1199: 138 pgs: 138 active+clean; 539 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2021-03-15T22:30:54.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56514]: cluster 2021-03-15T22:30:53.253946+0000 mon.a (mon.0) 1891 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2021-03-15T22:30:54.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56514]: audit 2021-03-15T22:30:53.707902+0000 mgr.y (mgr.14144) 861 : audit [DBG] from='client.16167 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:30:54.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[62156]: cluster 2021-03-15T22:30:53.151824+0000 mgr.y (mgr 2021-03-15T22:30:54.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[62156]: .14144) 860 : cluster [DBG] pgmap v1199: 138 pgs: 138 active+clean; 539 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2021-03-15T22:30:54.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[62156]: cluster 2021-03-15T22:30:53.253946+0000 mon.a 2021-03-15T22:30:54.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[62156]: (mon.0) 1891 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2021-03-15T22:30:54.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:30:54.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[62156]: :30:53.707902+0000 mgr.y (mgr.14144) 2021-03-15T22:30:54.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[62156]: 861 : audit [DBG] from='client.16167 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:30:54.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:54 smithi125 conmon[56739]: cluster 2021-03-15T22:30:53.151824+0000 mgr.y (mgr.14144) 860 : cluster 2021-03-15T22:30:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:54 smithi125 conmon[56739]: [DBG] pgmap v1199: 138 pgs: 138 active+clean; 539 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 1.8 MiB/s wr, 16 op/s 2021-03-15T22:30:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:54 smithi125 conmon[56739]: cluster 2021-03-15T22:30:53.253946+0000 mon.a (mon.0) 1891 : cluster [DBG] osdmap e704: 8 total, 8 up, 8 in 2021-03-15T22:30:54.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:54 smithi125 conmon[56739]: audit 2021-03-15T22:30:53.707902+0000 mgr.y (mgr.14144) 861 : audit [DBG] from='client.16167 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:30:54.565 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:54.590 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:54.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:54.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:54.693 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:54.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:54.716 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:54.717 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:54.817 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:54.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:54.841 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:54.841 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:54.943 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:54.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:54.966 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:54.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:55.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: debug 2021-03-15T22:30:54.638+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:30:55.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:30:55.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.018 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:30:55.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:30:55.019 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:30:55.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: debug 2021-03-15T22:30:54.638+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:30:55.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:30:55.020 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:30:55.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:30:55.021 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:30:55.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: debug 2021-03-15T22:30:54.639+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:30:55.022 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:30:55.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:30:55.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:30:55.023 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:30:55.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: debug 2021-03-15T22:30:54.639+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:30:55.024 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:30:55.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:30:55.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:30:55.025 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:30:55.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: debug 2021-03-15T22:30:54.639+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:30:55.026 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:30:55.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:30:55.027 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.028 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:30:55.028 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:30:55.029 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: debug 2021-03-15T22:30:54.640+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:30:55.030 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:30:55.030 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.031 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:30:55.032 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.032 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:30:55.033 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.033 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.033 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:30:55.034 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: debug 2021-03-15T22:30:54.640+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:30:55.034 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:30:55.034 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.034 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:30:55.034 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.035 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:30:55.035 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.035 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.035 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:30:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:30:55.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:55.091 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:55.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:55.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:55.194 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:55.195 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:30:55.195 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:30:55.196 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:30:55.196 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:30:55.215 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1200,"stamp":"2021-03-15T22:30:53.251597+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":564940991,"num_objects":593,"num_object_clones":227,"num_object_copies":1779,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":593,"num_whiteouts":13,"num_read":7212,"num_read_kb":5076753,"num_write":4428,"num_write_kb":1361353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2862,"num_bytes_recovered":1078946714,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14175,"ondisk_log_size":14175,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1590492,"kb_used_data":1294108,"kb_used_omap":273,"kb_used_meta":296046,"kb_avail":748370724,"statfs":{"total":767960285184,"available":766331621376,"internally_reserved":0,"allocated":1325166592,"data_stored":1853826149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":303151994},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":7329915,"num_objects":3,"num_object_clones":3,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":3,"num_read":43,"num_read_kb":46390,"num_write":24,"num_write_kb":7447,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018082"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"703","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249183+0000","last_change":"2021-03-15T22:30:41.431150+0000","last_active":"2021-03-15T22:30:50.249183+0000","last_peered":"2021-03-15T22:30:50.249183+0000","last_clean":"2021-03-15T22:30:50.249183+0000","last_became_active":"2021-03-15T22:29:51.225252+0000","last_became_peered":"2021-03-15T22:29:51.225252+0000","last_unstale":"2021-03-15T22:30:50.249183+0000","last_undegraded":"2021-03-15T22:30:50.249183+0000","last_fullsized":"2021-03-15T22:30:50.249183+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.431052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"200","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241821+0000","last_change":"2021-03-15T22:30:32.251958+0000","last_active":"2021-03-15T22:30:49.241821+0000","last_peered":"2021-03-15T22:30:49.241821+0000","last_clean":"2021-03-15T22:30:49.241821+0000","last_became_active":"2021-03-15T22:30:32.251610+0000","last_became_peered":"2021-03-15T22:30:32.251610+0000","last_unstale":"2021-03-15T22:30:49.241821+0000","last_undegraded":"2021-03-15T22:30:49.241821+0000","last_fullsized":"2021-03-15T22:30:49.241821+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:30:03.560125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.560125+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"554'21","reported_seq":"718","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249384+0000","last_change":"2021-03-15T22:30:01.066537+0000","last_active":"2021-03-15T22:30:50.249384+0000","last_peered":"2021-03-15T22:30:50.249384+0000","last_clean":"2021-03-15T22:30:50.249384+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:30:50.249384+0000","last_undegraded":"2021-03-15T22:30:50.249384+0000","last_fullsized":"2021-03-15T22:30:50.249384+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:30:01.066454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:01.066454+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"122","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239496+0000","last_change":"2021-03-15T22:28:59.271736+0000","last_active":"2021-03-15T22:30:50.239496+0000","last_peered":"2021-03-15T22:30:50.239496+0000","last_clean":"2021-03-15T22:30:50.239496+0000","last_became_active":"2021-03-15T22:28:59.271412+0000","last_became_peered":"2021-03-15T22:28:59.271412+0000","last_unstale":"2021-03-15T22:30:50.239496+0000","last_undegraded":"2021-03-15T22:30:50.239496+0000","last_fullsized":"2021-03-15T22:30:50.239496+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:39.160459+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:27:39.160459+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"409","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.240644+0000","last_change":"2021-03-15T22:30:32.259108+0000","last_active":"2021-03-15T22:30:50.240644+0000","last_peered":"2021-03-15T22:30:50.240644+0000","last_clean":"2021-03-15T22:30:50.240644+0000","last_became_active":"2021-03-15T22:30:32.258785+0000","last_became_peered":"2021-03-15T22:30:32.258785+0000","last_unstale":"2021-03-15T22:30:50.240644+0000","last_undegraded":"2021-03-15T22:30:50.240644+0000","last_fullsized":"2021-03-15T22:30:50.240644+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:09.184802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:09.184802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"556'32","reported_seq":"419","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239572+0000","last_change":"2021-03-15T22:30:32.296239+0000","last_active":"2021-03-15T22:30:50.239572+0000","last_peered":"2021-03-15T22:30:50.239572+0000","last_clean":"2021-03-15T22:30:50.239572+0000","last_became_active":"2021-03-15T22:30:32.258079+0000","last_became_peered":"2021-03-15T22:30:32.258079+0000","last_unstale":"2021-03-15T22:30:50.239572+0000","last_undegraded":"2021-03-15T22:30:50.239572+0000","last_fullsized":"2021-03-15T22:30:50.239572+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'32","last_scrub_stamp":"2021-03-15T22:29:32.512485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:32.512485+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"163","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241060+0000","last_change":"2021-03-15T22:29:18.753212+0000","last_active":"2021-03-15T22:30:49.241060+0000","last_peered":"2021-03-15T22:30:49.241060+0000","last_clean":"2021-03-15T22:30:49.241060+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:30:49.241060+0000","last_undegraded":"2021-03-15T22:30:49.241060+0000","last_fullsized":"2021-03-15T22:30:49.241060+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:18.753152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:29:18.753152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"72","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225461+0000","last_change":"2021-03-15T22:30:37.146490+0000","last_active":"2021-03-15T22:30:47.225461+0000","last_peered":"2021-03-15T22:30:47.225461+0000","last_clean":"2021-03-15T22:30:47.225461+0000","last_became_active":"2021-03-15T22:29:51.226784+0000","last_became_peered":"2021-03-15T22:29:51.226784+0000","last_unstale":"2021-03-15T22:30:47.225461+0000","last_undegraded":"2021-03-15T22:30:47.225461+0000","last_fullsized":"2021-03-15T22:30:47.225461+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.146408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.146408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"846","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249322+0000","last_change":"2021-03-15T22:30:28.328526+0000","last_active":"2021-03-15T22:30:50.249322+0000","last_peered":"2021-03-15T22:30:50.249322+0000","last_clean":"2021-03-15T22:30:50.249322+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:30:50.249322+0000","last_undegraded":"2021-03-15T22:30:50.249322+0000","last_fullsized":"2021-03-15T22:30:50.249322+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:30:28.328458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.328458+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"317","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225172+0000","last_change":"2021-03-15T22:30:40.173603+0000","last_active":"2021-03-15T22:30:47.225172+0000","last_peered":"2021-03-15T22:30:47.225172+0000","last_clean":"2021-03-15T22:30:47.225172+0000","last_became_active":"2021-03-15T22:29:52.223704+0000","last_became_peered":"2021-03-15T22:29:52.223704+0000","last_unstale":"2021-03-15T22:30:47.225172+0000","last_undegraded":"2021-03-15T22:30:47.225172+0000","last_fullsized":"2021-03-15T22:30:47.225172+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:40.173519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:30:40.173519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"72","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225378+0000","last_change":"2021-03-15T22:30:02.813811+0000","last_active":"2021-03-15T22:30:47.225378+0000","last_peered":"2021-03-15T22:30:47.225378+0000","last_clean":"2021-03-15T22:30:47.225378+0000","last_became_active":"2021-03-15T22:29:51.234868+0000","last_became_peered":"2021-03-15T22:29:51.234868+0000","last_unstale":"2021-03-15T22:30:47.225378+0000","last_undegraded":"2021-03-15T22:30:47.225378+0000","last_fullsized":"2021-03-15T22:30:47.225378+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:02.813745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.813745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"784","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249547+0000","last_change":"2021-03-15T22:29:56.950563+0000","last_active":"2021-03-15T22:30:50.249547+0000","last_peered":"2021-03-15T22:30:50.249547+0000","last_clean":"2021-03-15T22:30:50.249547+0000","last_became_active":"2021-03-15T22:29:11.197059+0000","last_became_peered":"2021-03-15T22:29:11.197059+0000","last_unstale":"2021-03-15T22:30:50.249547+0000","last_undegraded":"2021-03-15T22:30:50.249547+0000","last_fullsized":"2021-03-15T22:30:50.249547+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:29:56.950494+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.950494+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"696","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.250325+0000","last_change":"2021-03-15T22:29:51.225493+0000","last_active":"2021-03-15T22:30:50.250325+0000","last_peered":"2021-03-15T22:30:50.250325+0000","last_clean":"2021-03-15T22:30:50.250325+0000","last_became_active":"2021-03-15T22:29:51.225147+0000","last_became_peered":"2021-03-15T22:29:51.225147+0000","last_unstale":"2021-03-15T22:30:50.250325+0000","last_undegraded":"2021-03-15T22:30:50.250325+0000","last_fullsized":"2021-03-15T22:30:50.250325+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:41.941510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.941510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"167","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241291+0000","last_change":"2021-03-15T22:30:38.781792+0000","last_active":"2021-03-15T22:30:49.241291+0000","last_peered":"2021-03-15T22:30:49.241291+0000","last_clean":"2021-03-15T22:30:49.241291+0000","last_became_active":"2021-03-15T22:28:12.519619+0000","last_became_peered":"2021-03-15T22:28:12.519619+0000","last_unstale":"2021-03-15T22:30:49.241291+0000","last_undegraded":"2021-03-15T22:30:49.241291+0000","last_fullsized":"2021-03-15T22:30:49.241291+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:30:38.781724+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.781724+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"567","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.236769+0000","last_change":"2021-03-15T22:30:32.299886+0000","last_active":"2021-03-15T22:30:48.236769+0000","last_peered":"2021-03-15T22:30:48.236769+0000","last_clean":"2021-03-15T22:30:48.236769+0000","last_became_active":"2021-03-15T22:30:32.250733+0000","last_became_peered":"2021-03-15T22:30:32.250733+0000","last_unstale":"2021-03-15T22:30:48.236769+0000","last_undegraded":"2021-03-15T22:30:48.236769+0000","last_fullsized":"2021-03-15T22:30:48.236769+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:29:57.169300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:57.169300+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"513","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.240846+0000","last_change":"2021-03-15T22:29:51.225542+0000","last_active":"2021-03-15T22:30:50.240846+0000","last_peered":"2021-03-15T22:30:50.240846+0000","last_clean":"2021-03-15T22:30:50.240846+0000","last_became_active":"2021-03-15T22:29:51.225288+0000","last_became_peered":"2021-03-15T22:29:51.225288+0000","last_unstale":"2021-03-15T22:30:50.240846+0000","last_undegraded":"2021-03-15T22:30:50.240846+0000","last_fullsized":"2021-03-15T22:30:50.240846+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:38.513993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:45.403769+0000","last_clean_scrub_stamp":"2021-03-15T22:29:38.513993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"691","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249332+0000","last_change":"2021-03-15T22:30:39.443682+0000","last_active":"2021-03-15T22:30:50.249332+0000","last_peered":"2021-03-15T22:30:50.249332+0000","last_clean":"2021-03-15T22:30:50.249332+0000","last_became_active":"2021-03-15T22:30:31.243315+0000","last_became_peered":"2021-03-15T22:30:31.243315+0000","last_unstale":"2021-03-15T22:30:50.249332+0000","last_undegraded":"2021-03-15T22:30:50.249332+0000","last_fullsized":"2021-03-15T22:30:50.249332+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.443613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"147","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.240119+0000","last_change":"2021-03-15T22:29:54.564185+0000","last_active":"2021-03-15T22:30:50.240119+0000","last_peered":"2021-03-15T22:30:50.240119+0000","last_clean":"2021-03-15T22:30:50.240119+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:30:50.240119+0000","last_undegraded":"2021-03-15T22:30:50.240119+0000","last_fullsized":"2021-03-15T22:30:50.240119+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:29:54.564115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:54.564115+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"706","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.233872+0000","last_change":"2021-03-15T22:30:32.243983+0000","last_active":"2021-03-15T22:30:47.233872+0000","last_peered":"2021-03-15T22:30:47.233872+0000","last_clean":"2021-03-15T22:30:47.233872+0000","last_became_active":"2021-03-15T22:30:32.243708+0000","last_became_peered":"2021-03-15T22:30:32.243708+0000","last_unstale":"2021-03-15T22:30:47.233872+0000","last_undegraded":"2021-03-15T22:30:47.233872+0000","last_fullsized":"2021-03-15T22:30:47.233872+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:01.208847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:01.208847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"720","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.221796+0000","last_change":"2021-03-15T22:29:18.149267+0000","last_active":"2021-03-15T22:30:50.221796+0000","last_peered":"2021-03-15T22:30:50.221796+0000","last_clean":"2021-03-15T22:30:50.221796+0000","last_became_active":"2021-03-15T22:28:59.264559+0000","last_became_peered":"2021-03-15T22:28:59.264559+0000","last_unstale":"2021-03-15T22:30:50.221796+0000","last_undegraded":"2021-03-15T22:30:50.221796+0000","last_fullsized":"2021-03-15T22:30:50.221796+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:18.149204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:18.149204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"268","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.242292+0000","last_change":"2021-03-15T22:30:41.759292+0000","last_active":"2021-03-15T22:30:49.242292+0000","last_peered":"2021-03-15T22:30:49.242292+0000","last_clean":"2021-03-15T22:30:49.242292+0000","last_became_active":"2021-03-15T22:28:59.277385+0000","last_became_peered":"2021-03-15T22:28:59.277385+0000","last_unstale":"2021-03-15T22:30:49.242292+0000","last_undegraded":"2021-03-15T22:30:49.242292+0000","last_fullsized":"2021-03-15T22:30:49.242292+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:30:41.759224+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.759224+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"698","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.223540+0000","last_change":"2021-03-15T22:29:52.230266+0000","last_active":"2021-03-15T22:30:50.223540+0000","last_peered":"2021-03-15T22:30:50.223540+0000","last_clean":"2021-03-15T22:30:50.223540+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:30:50.223540+0000","last_undegraded":"2021-03-15T22:30:50.223540+0000","last_fullsized":"2021-03-15T22:30:50.223540+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:34.234643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:34.234643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"301","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239527+0000","last_change":"2021-03-15T22:29:00.220417+0000","last_active":"2021-03-15T22:30:50.239527+0000","last_peered":"2021-03-15T22:30:50.239527+0000","last_clean":"2021-03-15T22:30:50.239527+0000","last_became_active":"2021-03-15T22:29:00.220073+0000","last_became_peered":"2021-03-15T22:29:00.220073+0000","last_unstale":"2021-03-15T22:30:50.239527+0000","last_undegraded":"2021-03-15T22:30:50.239527+0000","last_fullsized":"2021-03-15T22:30:50.239527+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:34.099140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:28:34.099140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"786","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.242906+0000","last_change":"2021-03-15T22:29:52.273951+0000","last_active":"2021-03-15T22:30:49.242906+0000","last_peered":"2021-03-15T22:30:49.242906+0000","last_clean":"2021-03-15T22:30:49.242906+0000","last_became_active":"2021-03-15T22:29:51.232710+0000","last_became_peered":"2021-03-15T22:29:51.232710+0000","last_unstale":"2021-03-15T22:30:49.242906+0000","last_undegraded":"2021-03-15T22:30:49.242906+0000","last_fullsized":"2021-03-15T22:30:49.242906+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:29:14.124824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:14.124824+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"670","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.232317+0000","last_change":"2021-03-15T22:30:31.230974+0000","last_active":"2021-03-15T22:30:47.232317+0000","last_peered":"2021-03-15T22:30:47.232317+0000","last_clean":"2021-03-15T22:30:47.232317+0000","last_became_active":"2021-03-15T22:30:31.230649+0000","last_became_peered":"2021-03-15T22:30:31.230649+0000","last_unstale":"2021-03-15T22:30:47.232317+0000","last_undegraded":"2021-03-15T22:30:47.232317+0000","last_fullsized":"2021-03-15T22:30:47.232317+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:09.177521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:25:09.177521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"681","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241562+0000","last_change":"2021-03-15T22:29:35.315031+0000","last_active":"2021-03-15T22:30:49.241562+0000","last_peered":"2021-03-15T22:30:49.241562+0000","last_clean":"2021-03-15T22:30:49.241562+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:30:49.241562+0000","last_undegraded":"2021-03-15T22:30:49.241562+0000","last_fullsized":"2021-03-15T22:30:49.241562+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:35.314964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:35.314964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"202","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241742+0000","last_change":"2021-03-15T22:29:51.226956+0000","last_active":"2021-03-15T22:30:49.241742+0000","last_peered":"2021-03-15T22:30:49.241742+0000","last_clean":"2021-03-15T22:30:49.241742+0000","last_became_active":"2021-03-15T22:29:51.226677+0000","last_became_peered":"2021-03-15T22:29:51.226677+0000","last_unstale":"2021-03-15T22:30:49.241742+0000","last_undegraded":"2021-03-15T22:30:49.241742+0000","last_fullsized":"2021-03-15T22:30:49.241742+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:14.776700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:29:14.776700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"149","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239406+0000","last_change":"2021-03-15T22:30:07.486292+0000","last_active":"2021-03-15T22:30:50.239406+0000","last_peered":"2021-03-15T22:30:50.239406+0000","last_clean":"2021-03-15T22:30:50.239406+0000","last_became_active":"2021-03-15T22:28:59.269481+0000","last_became_peered":"2021-03-15T22:28:59.269481+0000","last_unstale":"2021-03-15T22:30:50.239406+0000","last_undegraded":"2021-03-15T22:30:50.239406+0000","last_fullsized":"2021-03-15T22:30:50.239406+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:30:07.486190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.486190+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":6260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"301","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239772+0000","last_change":"2021-03-15T22:29:00.216337+0000","last_active":"2021-03-15T22:30:50.239772+0000","last_peered":"2021-03-15T22:30:50.239772+0000","last_clean":"2021-03-15T22:30:50.239772+0000","last_became_active":"2021-03-15T22:29:00.215311+0000","last_became_peered":"2021-03-15T22:29:00.215311+0000","last_unstale":"2021-03-15T22:30:50.239772+0000","last_undegraded":"2021-03-15T22:30:50.239772+0000","last_fullsized":"2021-03-15T22:30:50.239772+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:36.102000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:40.670674+0000","last_clean_scrub_stamp":"2021-03-15T22:28:36.102000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"657","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.232569+0000","last_change":"2021-03-15T22:28:09.088750+0000","last_active":"2021-03-15T22:30:47.232569+0000","last_peered":"2021-03-15T22:30:47.232569+0000","last_clean":"2021-03-15T22:30:47.232569+0000","last_became_active":"2021-03-15T22:23:52.262741+0000","last_became_peered":"2021-03-15T22:23:52.262741+0000","last_unstale":"2021-03-15T22:30:47.232569+0000","last_undegraded":"2021-03-15T22:30:47.232569+0000","last_fullsized":"2021-03-15T22:30:47.232569+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:09.088684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:28:09.088684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"72","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225478+0000","last_change":"2021-03-15T22:30:42.233176+0000","last_active":"2021-03-15T22:30:47.225478+0000","last_peered":"2021-03-15T22:30:47.225478+0000","last_clean":"2021-03-15T22:30:47.225478+0000","last_became_active":"2021-03-15T22:29:51.220261+0000","last_became_peered":"2021-03-15T22:29:51.220261+0000","last_unstale":"2021-03-15T22:30:47.225478+0000","last_undegraded":"2021-03-15T22:30:47.225478+0000","last_fullsized":"2021-03-15T22:30:47.225478+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.233098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"483","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239673+0000","last_change":"2021-03-15T22:29:00.220920+0000","last_active":"2021-03-15T22:30:50.239673+0000","last_peered":"2021-03-15T22:30:50.239673+0000","last_clean":"2021-03-15T22:30:50.239673+0000","last_became_active":"2021-03-15T22:29:00.220100+0000","last_became_peered":"2021-03-15T22:29:00.220100+0000","last_unstale":"2021-03-15T22:30:50.239673+0000","last_undegraded":"2021-03-15T22:30:50.239673+0000","last_fullsized":"2021-03-15T22:30:50.239673+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:27:37.107464+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:27:37.107464+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"182","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.240998+0000","last_change":"2021-03-15T22:29:51.212768+0000","last_active":"2021-03-15T22:30:49.240998+0000","last_peered":"2021-03-15T22:30:49.240998+0000","last_clean":"2021-03-15T22:30:49.240998+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:30:49.240998+0000","last_undegraded":"2021-03-15T22:30:49.240998+0000","last_fullsized":"2021-03-15T22:30:49.240998+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:19.721291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:29:19.721291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"491","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239722+0000","last_change":"2021-03-15T22:29:56.525891+0000","last_active":"2021-03-15T22:30:50.239722+0000","last_peered":"2021-03-15T22:30:50.239722+0000","last_clean":"2021-03-15T22:30:50.239722+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:30:50.239722+0000","last_undegraded":"2021-03-15T22:30:50.239722+0000","last_fullsized":"2021-03-15T22:30:50.239722+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:56.525822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.525822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"697","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.234593+0000","last_change":"2021-03-15T22:29:09.154666+0000","last_active":"2021-03-15T22:30:47.234593+0000","last_peered":"2021-03-15T22:30:47.234593+0000","last_clean":"2021-03-15T22:30:47.234593+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:30:47.234593+0000","last_undegraded":"2021-03-15T22:30:47.234593+0000","last_fullsized":"2021-03-15T22:30:47.234593+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:09.154606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:29:09.154606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"395","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.222443+0000","last_change":"2021-03-15T22:30:38.178660+0000","last_active":"2021-03-15T22:30:50.222443+0000","last_peered":"2021-03-15T22:30:50.222443+0000","last_clean":"2021-03-15T22:30:50.222443+0000","last_became_active":"2021-03-15T22:30:31.240826+0000","last_became_peered":"2021-03-15T22:30:31.240826+0000","last_unstale":"2021-03-15T22:30:50.222443+0000","last_undegraded":"2021-03-15T22:30:50.222443+0000","last_fullsized":"2021-03-15T22:30:50.222443+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:30:38.178596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.178596+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"323","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239486+0000","last_change":"2021-03-15T22:29:51.219273+0000","last_active":"2021-03-15T22:30:50.239486+0000","last_peered":"2021-03-15T22:30:50.239486+0000","last_clean":"2021-03-15T22:30:50.239486+0000","last_became_active":"2021-03-15T22:29:51.218982+0000","last_became_peered":"2021-03-15T22:29:51.218982+0000","last_unstale":"2021-03-15T22:30:50.239486+0000","last_undegraded":"2021-03-15T22:30:50.239486+0000","last_fullsized":"2021-03-15T22:30:50.239486+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:35.496457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:29:35.496457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"742","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.222331+0000","last_change":"2021-03-15T22:30:44.116997+0000","last_active":"2021-03-15T22:30:50.222331+0000","last_peered":"2021-03-15T22:30:50.222331+0000","last_clean":"2021-03-15T22:30:50.222331+0000","last_became_active":"2021-03-15T22:29:51.222254+0000","last_became_peered":"2021-03-15T22:29:51.222254+0000","last_unstale":"2021-03-15T22:30:50.222331+0000","last_undegraded":"2021-03-15T22:30:50.222331+0000","last_fullsized":"2021-03-15T22:30:50.222331+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:30:44.116934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"213","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.236877+0000","last_change":"2021-03-15T22:30:46.165561+0000","last_active":"2021-03-15T22:30:48.236877+0000","last_peered":"2021-03-15T22:30:48.236877+0000","last_clean":"2021-03-15T22:30:48.236877+0000","last_became_active":"2021-03-15T22:30:31.254001+0000","last_became_peered":"2021-03-15T22:30:31.254001+0000","last_unstale":"2021-03-15T22:30:48.236877+0000","last_undegraded":"2021-03-15T22:30:48.236877+0000","last_fullsized":"2021-03-15T22:30:48.236877+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.992352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.992352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"720","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249448+0000","last_change":"2021-03-15T22:30:02.064784+0000","last_active":"2021-03-15T22:30:50.249448+0000","last_peered":"2021-03-15T22:30:50.249448+0000","last_clean":"2021-03-15T22:30:50.249448+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:30:50.249448+0000","last_undegraded":"2021-03-15T22:30:50.249448+0000","last_fullsized":"2021-03-15T22:30:50.249448+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:30:02.064721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.064721+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"181","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.242448+0000","last_change":"2021-03-15T22:30:42.771039+0000","last_active":"2021-03-15T22:30:49.242448+0000","last_peered":"2021-03-15T22:30:49.242448+0000","last_clean":"2021-03-15T22:30:49.242448+0000","last_became_active":"2021-03-15T22:28:59.276812+0000","last_became_peered":"2021-03-15T22:28:59.276812+0000","last_unstale":"2021-03-15T22:30:49.242448+0000","last_undegraded":"2021-03-15T22:30:49.242448+0000","last_fullsized":"2021-03-15T22:30:49.242448+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:42.770976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.770976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"642'8","reported_seq":"88","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225557+0000","last_change":"2021-03-15T22:30:06.920670+0000","last_active":"2021-03-15T22:30:47.225557+0000","last_peered":"2021-03-15T22:30:47.225557+0000","last_clean":"2021-03-15T22:30:47.225557+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:30:47.225557+0000","last_undegraded":"2021-03-15T22:30:47.225557+0000","last_fullsized":"2021-03-15T22:30:47.225557+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'8","last_scrub_stamp":"2021-03-15T22:30:06.920606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:06.920606+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"676","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.231960+0000","last_change":"2021-03-15T22:30:35.423621+0000","last_active":"2021-03-15T22:30:47.231960+0000","last_peered":"2021-03-15T22:30:47.231960+0000","last_clean":"2021-03-15T22:30:47.231960+0000","last_became_active":"2021-03-15T22:30:31.230560+0000","last_became_peered":"2021-03-15T22:30:31.230560+0000","last_unstale":"2021-03-15T22:30:47.231960+0000","last_undegraded":"2021-03-15T22:30:47.231960+0000","last_fullsized":"2021-03-15T22:30:47.231960+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:35.423559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:35.423559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"556'24","reported_seq":"490","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.240006+0000","last_change":"2021-03-15T22:30:02.548134+0000","last_active":"2021-03-15T22:30:50.240006+0000","last_peered":"2021-03-15T22:30:50.240006+0000","last_clean":"2021-03-15T22:30:50.240006+0000","last_became_active":"2021-03-15T22:29:00.226652+0000","last_became_peered":"2021-03-15T22:29:00.226652+0000","last_unstale":"2021-03-15T22:30:50.240006+0000","last_undegraded":"2021-03-15T22:30:50.240006+0000","last_fullsized":"2021-03-15T22:30:50.240006+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'24","last_scrub_stamp":"2021-03-15T22:30:02.548069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.548069+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"711","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.250505+0000","last_change":"2021-03-15T22:30:32.248125+0000","last_active":"2021-03-15T22:30:50.250505+0000","last_peered":"2021-03-15T22:30:50.250505+0000","last_clean":"2021-03-15T22:30:50.250505+0000","last_became_active":"2021-03-15T22:30:32.247782+0000","last_became_peered":"2021-03-15T22:30:32.247782+0000","last_unstale":"2021-03-15T22:30:50.250505+0000","last_undegraded":"2021-03-15T22:30:50.250505+0000","last_fullsized":"2021-03-15T22:30:50.250505+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:34.939142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:34.939142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"683'14","reported_seq":"363","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239835+0000","last_change":"2021-03-15T22:30:32.252825+0000","last_active":"2021-03-15T22:30:50.239835+0000","last_peered":"2021-03-15T22:30:50.239835+0000","last_clean":"2021-03-15T22:30:50.239835+0000","last_became_active":"2021-03-15T22:30:32.252383+0000","last_became_peered":"2021-03-15T22:30:32.252383+0000","last_unstale":"2021-03-15T22:30:50.239835+0000","last_undegraded":"2021-03-15T22:30:50.239835+0000","last_fullsized":"2021-03-15T22:30:50.239835+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'10","last_scrub_stamp":"2021-03-15T22:28:47.140118+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:28:47.140118+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"181","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.240984+0000","last_change":"2021-03-15T22:29:17.782276+0000","last_active":"2021-03-15T22:30:49.240984+0000","last_peered":"2021-03-15T22:30:49.240984+0000","last_clean":"2021-03-15T22:30:49.240984+0000","last_became_active":"2021-03-15T22:28:59.265822+0000","last_became_peered":"2021-03-15T22:28:59.265822+0000","last_unstale":"2021-03-15T22:30:49.240984+0000","last_undegraded":"2021-03-15T22:30:49.240984+0000","last_fullsized":"2021-03-15T22:30:49.240984+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:17.782209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:29:17.782209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"90","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225361+0000","last_change":"2021-03-15T22:30:36.106413+0000","last_active":"2021-03-15T22:30:47.225361+0000","last_peered":"2021-03-15T22:30:47.225361+0000","last_clean":"2021-03-15T22:30:47.225361+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:30:47.225361+0000","last_undegraded":"2021-03-15T22:30:47.225361+0000","last_fullsized":"2021-03-15T22:30:47.225361+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:30:36.106285+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.106285+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"470","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.236562+0000","last_change":"2021-03-15T22:30:46.154245+0000","last_active":"2021-03-15T22:30:48.236562+0000","last_peered":"2021-03-15T22:30:48.236562+0000","last_clean":"2021-03-15T22:30:48.236562+0000","last_became_active":"2021-03-15T22:30:31.254532+0000","last_became_peered":"2021-03-15T22:30:31.254532+0000","last_unstale":"2021-03-15T22:30:48.236562+0000","last_undegraded":"2021-03-15T22:30:48.236562+0000","last_fullsized":"2021-03-15T22:30:48.236562+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.003120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.003120+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"666","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.232183+0000","last_change":"2021-03-15T22:30:10.271920+0000","last_active":"2021-03-15T22:30:47.232183+0000","last_peered":"2021-03-15T22:30:47.232183+0000","last_clean":"2021-03-15T22:30:47.232183+0000","last_became_active":"2021-03-15T22:23:52.263426+0000","last_became_peered":"2021-03-15T22:23:52.263426+0000","last_unstale":"2021-03-15T22:30:47.232183+0000","last_undegraded":"2021-03-15T22:30:47.232183+0000","last_fullsized":"2021-03-15T22:30:47.232183+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:33.386700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:33.386700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"221","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.242078+0000","last_change":"2021-03-15T22:29:00.214080+0000","last_active":"2021-03-15T22:30:49.242078+0000","last_peered":"2021-03-15T22:30:49.242078+0000","last_clean":"2021-03-15T22:30:49.242078+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:30:49.242078+0000","last_undegraded":"2021-03-15T22:30:49.242078+0000","last_fullsized":"2021-03-15T22:30:49.242078+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:28:44.765846+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"660","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.234668+0000","last_change":"2021-03-15T22:27:41.254554+0000","last_active":"2021-03-15T22:30:47.234668+0000","last_peered":"2021-03-15T22:30:47.234668+0000","last_clean":"2021-03-15T22:30:47.234668+0000","last_became_active":"2021-03-15T22:23:52.266133+0000","last_became_peered":"2021-03-15T22:23:52.266133+0000","last_unstale":"2021-03-15T22:30:47.234668+0000","last_undegraded":"2021-03-15T22:30:47.234668+0000","last_fullsized":"2021-03-15T22:30:47.234668+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:41.254469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:27:41.254469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"757","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241052+0000","last_change":"2021-03-15T22:30:37.321794+0000","last_active":"2021-03-15T22:30:49.241052+0000","last_peered":"2021-03-15T22:30:49.241052+0000","last_clean":"2021-03-15T22:30:49.241052+0000","last_became_active":"2021-03-15T22:29:51.231818+0000","last_became_peered":"2021-03-15T22:29:51.231818+0000","last_unstale":"2021-03-15T22:30:49.241052+0000","last_undegraded":"2021-03-15T22:30:49.241052+0000","last_fullsized":"2021-03-15T22:30:49.241052+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.321715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.321715+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.321715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"327","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.240517+0000","last_change":"2021-03-15T22:29:13.250177+0000","last_active":"2021-03-15T22:30:50.240517+0000","last_peered":"2021-03-15T22:30:50.240517+0000","last_clean":"2021-03-15T22:30:50.240517+0000","last_became_active":"2021-03-15T22:29:11.204254+0000","last_became_peered":"2021-03-15T22:29:11.204254+0000","last_unstale":"2021-03-15T22:30:50.240517+0000","last_undegraded":"2021-03-15T22:30:50.240517+0000","last_fullsized":"2021-03-15T22:30:50.240517+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:29:13.250082+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:29:13.250082+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"702'806","reported_seq":"2808","reported_epoch":"702","state":"active+clean","last_fresh":"2021-03-15T22:30:51.263505+0000","last_change":"2021-03-15T22:30:51.263814+0000","last_active":"2021-03-15T22:30:51.263505+0000","last_peered":"2021-03-15T22:30:51.263505+0000","last_clean":"2021-03-15T22:30:51.263505+0000","last_became_active":"2021-03-15T22:29:52.227518+0000","last_became_peered":"2021-03-15T22:29:52.227518+0000","last_unstale":"2021-03-15T22:30:51.263505+0000","last_undegraded":"2021-03-15T22:30:51.263505+0000","last_fullsized":"2021-03-15T22:30:51.263505+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'650","last_scrub_stamp":"2021-03-15T22:29:40.242110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.242110+0000","log_size":806,"ondisk_log_size":806,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34537168,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":359,"num_read_kb":409411,"num_write":243,"num_write_kb":79169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":30385610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"1fb","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"616","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225483+0000","last_change":"2021-03-15T22:29:52.217756+0000","last_active":"2021-03-15T22:30:47.225483+0000","last_peered":"2021-03-15T22:30:47.225483+0000","last_clean":"2021-03-15T22:30:47.225483+0000","last_became_active":"2021-03-15T22:29:52.217465+0000","last_became_peered":"2021-03-15T22:29:52.217465+0000","last_unstale":"2021-03-15T22:30:47.225483+0000","last_undegraded":"2021-03-15T22:30:47.225483+0000","last_fullsized":"2021-03-15T22:30:47.225483+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:32.224256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:32.224256+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"699","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.233664+0000","last_change":"2021-03-15T22:28:12.530786+0000","last_active":"2021-03-15T22:30:47.233664+0000","last_peered":"2021-03-15T22:30:47.233664+0000","last_clean":"2021-03-15T22:30:47.233664+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:30:47.233664+0000","last_undegraded":"2021-03-15T22:30:47.233664+0000","last_fullsized":"2021-03-15T22:30:47.233664+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:27:40.278709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:40.278709+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"104","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225079+0000","last_change":"2021-03-15T22:30:41.227569+0000","last_active":"2021-03-15T22:30:47.225079+0000","last_peered":"2021-03-15T22:30:47.225079+0000","last_clean":"2021-03-15T22:30:47.225079+0000","last_became_active":"2021-03-15T22:29:51.221013+0000","last_became_peered":"2021-03-15T22:29:51.221013+0000","last_unstale":"2021-03-15T22:30:47.225079+0000","last_undegraded":"2021-03-15T22:30:47.225079+0000","last_fullsized":"2021-03-15T22:30:47.225079+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.227473+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"317'7","reported_seq":"836","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.232410+0000","last_change":"2021-03-15T22:30:07.253037+0000","last_active":"2021-03-15T22:30:47.232410+0000","last_peered":"2021-03-15T22:30:47.232410+0000","last_clean":"2021-03-15T22:30:47.232410+0000","last_became_active":"2021-03-15T22:23:52.262878+0000","last_became_peered":"2021-03-15T22:23:52.262878+0000","last_unstale":"2021-03-15T22:30:47.232410+0000","last_undegraded":"2021-03-15T22:30:47.232410+0000","last_fullsized":"2021-03-15T22:30:47.232410+0000","mapping_epoch":316,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:30:07.252960+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.252960+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"702'1051","reported_seq":"3085","reported_epoch":"702","state":"active+clean","last_fresh":"2021-03-15T22:30:51.265146+0000","last_change":"2021-03-15T22:30:51.265402+0000","last_active":"2021-03-15T22:30:51.265146+0000","last_peered":"2021-03-15T22:30:51.265146+0000","last_clean":"2021-03-15T22:30:51.265146+0000","last_became_active":"2021-03-15T22:30:34.291079+0000","last_became_peered":"2021-03-15T22:30:34.291079+0000","last_unstale":"2021-03-15T22:30:51.265146+0000","last_undegraded":"2021-03-15T22:30:51.265146+0000","last_fullsized":"2021-03-15T22:30:51.265146+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'857","last_scrub_stamp":"2021-03-15T22:29:33.490626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.490626+0000","log_size":1051,"ondisk_log_size":1051,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":45974687,"num_objects":24,"num_object_clones":19,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":478,"num_read_kb":469288,"num_write":324,"num_write_kb":127123,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":80476435,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"1fb","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"127","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239575+0000","last_change":"2021-03-15T22:29:58.503593+0000","last_active":"2021-03-15T22:30:50.239575+0000","last_peered":"2021-03-15T22:30:50.239575+0000","last_clean":"2021-03-15T22:30:50.239575+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:30:50.239575+0000","last_undegraded":"2021-03-15T22:30:50.239575+0000","last_fullsized":"2021-03-15T22:30:50.239575+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:58.503533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:58.503533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"668","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.233778+0000","last_change":"2021-03-15T22:27:44.227979+0000","last_active":"2021-03-15T22:30:47.233778+0000","last_peered":"2021-03-15T22:30:47.233778+0000","last_clean":"2021-03-15T22:30:47.233778+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:30:47.233778+0000","last_undegraded":"2021-03-15T22:30:47.233778+0000","last_fullsized":"2021-03-15T22:30:47.233778+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:44.227918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:44.227918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"820","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249869+0000","last_change":"2021-03-15T22:29:10.861350+0000","last_active":"2021-03-15T22:30:50.249869+0000","last_peered":"2021-03-15T22:30:50.249869+0000","last_clean":"2021-03-15T22:30:50.249869+0000","last_became_active":"2021-03-15T22:28:59.267623+0000","last_became_peered":"2021-03-15T22:28:59.267623+0000","last_unstale":"2021-03-15T22:30:50.249869+0000","last_undegraded":"2021-03-15T22:30:50.249869+0000","last_fullsized":"2021-03-15T22:30:50.249869+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:29:10.861249+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:10.861249+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"511","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.238007+0000","last_change":"2021-03-15T22:30:32.239285+0000","last_active":"2021-03-15T22:30:48.238007+0000","last_peered":"2021-03-15T22:30:48.238007+0000","last_clean":"2021-03-15T22:30:48.238007+0000","last_became_active":"2021-03-15T22:30:32.239001+0000","last_became_peered":"2021-03-15T22:30:32.239001+0000","last_unstale":"2021-03-15T22:30:48.238007+0000","last_undegraded":"2021-03-15T22:30:48.238007+0000","last_fullsized":"2021-03-15T22:30:48.238007+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:56.479415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.479415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"699'311","reported_seq":"2157","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.222601+0000","last_change":"2021-03-15T22:30:48.238997+0000","last_active":"2021-03-15T22:30:50.222601+0000","last_peered":"2021-03-15T22:30:50.222601+0000","last_clean":"2021-03-15T22:30:50.222601+0000","last_became_active":"2021-03-15T22:28:16.177364+0000","last_became_peered":"2021-03-15T22:28:16.177364+0000","last_unstale":"2021-03-15T22:30:50.222601+0000","last_undegraded":"2021-03-15T22:30:50.222601+0000","last_fullsized":"2021-03-15T22:30:50.222601+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'293","last_scrub_stamp":"2021-03-15T22:30:29.729234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.729234+0000","log_size":311,"ondisk_log_size":311,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12795248,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":66683,"num_write":86,"num_write_kb":26508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":11148008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"714","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.232191+0000","last_change":"2021-03-15T22:30:32.237010+0000","last_active":"2021-03-15T22:30:47.232191+0000","last_peered":"2021-03-15T22:30:47.232191+0000","last_clean":"2021-03-15T22:30:47.232191+0000","last_became_active":"2021-03-15T22:30:32.236752+0000","last_became_peered":"2021-03-15T22:30:32.236752+0000","last_unstale":"2021-03-15T22:30:47.232191+0000","last_undegraded":"2021-03-15T22:30:47.232191+0000","last_fullsized":"2021-03-15T22:30:47.232191+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:08.137989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:08.137989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"523","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.237031+0000","last_change":"2021-03-15T22:30:32.267914+0000","last_active":"2021-03-15T22:30:48.237031+0000","last_peered":"2021-03-15T22:30:48.237031+0000","last_clean":"2021-03-15T22:30:48.237031+0000","last_became_active":"2021-03-15T22:30:32.263574+0000","last_became_peered":"2021-03-15T22:30:32.263574+0000","last_unstale":"2021-03-15T22:30:48.237031+0000","last_undegraded":"2021-03-15T22:30:48.237031+0000","last_fullsized":"2021-03-15T22:30:48.237031+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:59.529159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:59.529159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"710","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.234764+0000","last_change":"2021-03-15T22:30:08.279264+0000","last_active":"2021-03-15T22:30:47.234764+0000","last_peered":"2021-03-15T22:30:47.234764+0000","last_clean":"2021-03-15T22:30:47.234764+0000","last_became_active":"2021-03-15T22:29:51.222437+0000","last_became_peered":"2021-03-15T22:29:51.222437+0000","last_unstale":"2021-03-15T22:30:47.234764+0000","last_undegraded":"2021-03-15T22:30:47.234764+0000","last_fullsized":"2021-03-15T22:30:47.234764+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:08.279129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:30:08.279129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"802","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241848+0000","last_change":"2021-03-15T22:30:32.235249+0000","last_active":"2021-03-15T22:30:49.241848+0000","last_peered":"2021-03-15T22:30:49.241848+0000","last_clean":"2021-03-15T22:30:49.241848+0000","last_became_active":"2021-03-15T22:30:31.239271+0000","last_became_peered":"2021-03-15T22:30:31.239271+0000","last_unstale":"2021-03-15T22:30:49.241848+0000","last_undegraded":"2021-03-15T22:30:49.241848+0000","last_fullsized":"2021-03-15T22:30:49.241848+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"65'23","last_scrub_stamp":"2021-03-15T22:26:11.821564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:26:11.821564+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"699'572","reported_seq":"2151","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.695622+0000","last_change":"2021-03-15T22:30:48.245110+0000","last_active":"2021-03-15T22:30:48.695622+0000","last_peered":"2021-03-15T22:30:48.695622+0000","last_clean":"2021-03-15T22:30:48.695622+0000","last_became_active":"2021-03-15T22:30:31.242297+0000","last_became_peered":"2021-03-15T22:30:31.242297+0000","last_unstale":"2021-03-15T22:30:48.695622+0000","last_undegraded":"2021-03-15T22:30:48.695622+0000","last_fullsized":"2021-03-15T22:30:48.695622+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'555","last_scrub_stamp":"2021-03-15T22:30:34.990334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.990334+0000","log_size":572,"ondisk_log_size":572,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26841583,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":281,"num_read_kb":332033,"num_write":177,"num_write_kb":63111,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":24240388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"1f8","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"221","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"174","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241136+0000","last_change":"2021-03-15T22:30:32.238086+0000","last_active":"2021-03-15T22:30:49.241136+0000","last_peered":"2021-03-15T22:30:49.241136+0000","last_clean":"2021-03-15T22:30:49.241136+0000","last_became_active":"2021-03-15T22:30:32.237720+0000","last_became_peered":"2021-03-15T22:30:32.237720+0000","last_unstale":"2021-03-15T22:30:49.241136+0000","last_undegraded":"2021-03-15T22:30:49.241136+0000","last_fullsized":"2021-03-15T22:30:49.241136+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:49.778250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:28:49.778250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"501","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.236829+0000","last_change":"2021-03-15T22:30:32.267370+0000","last_active":"2021-03-15T22:30:48.236829+0000","last_peered":"2021-03-15T22:30:48.236829+0000","last_clean":"2021-03-15T22:30:48.236829+0000","last_became_active":"2021-03-15T22:30:32.267050+0000","last_became_peered":"2021-03-15T22:30:32.267050+0000","last_unstale":"2021-03-15T22:30:48.236829+0000","last_undegraded":"2021-03-15T22:30:48.236829+0000","last_fullsized":"2021-03-15T22:30:48.236829+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:07.063227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.063227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"72","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225378+0000","last_change":"2021-03-15T22:30:07.891846+0000","last_active":"2021-03-15T22:30:47.225378+0000","last_peered":"2021-03-15T22:30:47.225378+0000","last_clean":"2021-03-15T22:30:47.225378+0000","last_became_active":"2021-03-15T22:29:51.231699+0000","last_became_peered":"2021-03-15T22:29:51.231699+0000","last_unstale":"2021-03-15T22:30:47.225378+0000","last_undegraded":"2021-03-15T22:30:47.225378+0000","last_fullsized":"2021-03-15T22:30:47.225378+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:07.891775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.891775+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"212","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241363+0000","last_change":"2021-03-15T22:30:32.557348+0000","last_active":"2021-03-15T22:30:49.241363+0000","last_peered":"2021-03-15T22:30:49.241363+0000","last_clean":"2021-03-15T22:30:49.241363+0000","last_became_active":"2021-03-15T22:30:32.250712+0000","last_became_peered":"2021-03-15T22:30:32.250712+0000","last_unstale":"2021-03-15T22:30:49.241363+0000","last_undegraded":"2021-03-15T22:30:49.241363+0000","last_fullsized":"2021-03-15T22:30:49.241363+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:30:05.601295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:05.601295+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"699'852","reported_seq":"3111","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.228141+0000","last_change":"2021-03-15T22:30:48.245707+0000","last_active":"2021-03-15T22:30:50.228141+0000","last_peered":"2021-03-15T22:30:50.228141+0000","last_clean":"2021-03-15T22:30:50.228141+0000","last_became_active":"2021-03-15T22:29:02.359450+0000","last_became_peered":"2021-03-15T22:29:02.359450+0000","last_unstale":"2021-03-15T22:30:50.228141+0000","last_undegraded":"2021-03-15T22:30:50.228141+0000","last_fullsized":"2021-03-15T22:30:50.228141+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"606'654","last_scrub_stamp":"2021-03-15T22:29:17.132235+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:29:17.132235+0000","log_size":852,"ondisk_log_size":852,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43174572,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":217,"num_read_kb":173022,"num_write":241,"num_write_kb":82824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":114286074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"127","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.240411+0000","last_change":"2021-03-15T22:30:29.533509+0000","last_active":"2021-03-15T22:30:50.240411+0000","last_peered":"2021-03-15T22:30:50.240411+0000","last_clean":"2021-03-15T22:30:50.240411+0000","last_became_active":"2021-03-15T22:28:59.273785+0000","last_became_peered":"2021-03-15T22:28:59.273785+0000","last_unstale":"2021-03-15T22:30:50.240411+0000","last_undegraded":"2021-03-15T22:30:50.240411+0000","last_fullsized":"2021-03-15T22:30:50.240411+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:29.533421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.533421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"67","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225233+0000","last_change":"2021-03-15T22:29:51.236372+0000","last_active":"2021-03-15T22:30:47.225233+0000","last_peered":"2021-03-15T22:30:47.225233+0000","last_clean":"2021-03-15T22:30:47.225233+0000","last_became_active":"2021-03-15T22:29:51.236007+0000","last_became_peered":"2021-03-15T22:29:51.236007+0000","last_unstale":"2021-03-15T22:30:47.225233+0000","last_undegraded":"2021-03-15T22:30:47.225233+0000","last_fullsized":"2021-03-15T22:30:47.225233+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:33.191779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.191779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"32","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.237707+0000","last_change":"2021-03-15T22:30:39.012748+0000","last_active":"2021-03-15T22:30:48.237707+0000","last_peered":"2021-03-15T22:30:48.237707+0000","last_clean":"2021-03-15T22:30:48.237707+0000","last_became_active":"2021-03-15T22:30:32.262041+0000","last_became_peered":"2021-03-15T22:30:32.262041+0000","last_unstale":"2021-03-15T22:30:48.237707+0000","last_undegraded":"2021-03-15T22:30:48.237707+0000","last_fullsized":"2021-03-15T22:30:48.237707+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.012676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"913","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.223655+0000","last_change":"2021-03-15T22:30:32.294089+0000","last_active":"2021-03-15T22:30:50.223655+0000","last_peered":"2021-03-15T22:30:50.223655+0000","last_clean":"2021-03-15T22:30:50.223655+0000","last_became_active":"2021-03-15T22:30:32.244684+0000","last_became_peered":"2021-03-15T22:30:32.244684+0000","last_unstale":"2021-03-15T22:30:50.223655+0000","last_undegraded":"2021-03-15T22:30:50.223655+0000","last_fullsized":"2021-03-15T22:30:50.223655+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:30:03.187386+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.187386+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"163","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241439+0000","last_change":"2021-03-15T22:30:30.738196+0000","last_active":"2021-03-15T22:30:49.241439+0000","last_peered":"2021-03-15T22:30:49.241439+0000","last_clean":"2021-03-15T22:30:49.241439+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:30:49.241439+0000","last_undegraded":"2021-03-15T22:30:49.241439+0000","last_fullsized":"2021-03-15T22:30:49.241439+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:30.738108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:30.738108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"699'786","reported_seq":"2607","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.223792+0000","last_change":"2021-03-15T22:30:48.240780+0000","last_active":"2021-03-15T22:30:50.223792+0000","last_peered":"2021-03-15T22:30:50.223792+0000","last_clean":"2021-03-15T22:30:50.223792+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:30:50.223792+0000","last_undegraded":"2021-03-15T22:30:50.223792+0000","last_fullsized":"2021-03-15T22:30:50.223792+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"690'772","last_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.836165+0000","log_size":786,"ondisk_log_size":786,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33849096,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":226,"num_read_kb":216156,"num_write":204,"num_write_kb":77140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"697","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.235606+0000","last_change":"2021-03-15T22:29:07.163096+0000","last_active":"2021-03-15T22:30:47.235606+0000","last_peered":"2021-03-15T22:30:47.235606+0000","last_clean":"2021-03-15T22:30:47.235606+0000","last_became_active":"2021-03-15T22:28:59.278695+0000","last_became_peered":"2021-03-15T22:28:59.278695+0000","last_unstale":"2021-03-15T22:30:47.235606+0000","last_undegraded":"2021-03-15T22:30:47.235606+0000","last_fullsized":"2021-03-15T22:30:47.235606+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:07.163034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:07.163034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"689","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.221918+0000","last_change":"2021-03-15T22:30:34.070664+0000","last_active":"2021-03-15T22:30:50.221918+0000","last_peered":"2021-03-15T22:30:50.221918+0000","last_clean":"2021-03-15T22:30:50.221918+0000","last_became_active":"2021-03-15T22:30:31.243831+0000","last_became_peered":"2021-03-15T22:30:31.243831+0000","last_unstale":"2021-03-15T22:30:50.221918+0000","last_undegraded":"2021-03-15T22:30:50.221918+0000","last_fullsized":"2021-03-15T22:30:50.221918+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.070598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.070598+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"498","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.236467+0000","last_change":"2021-03-15T22:30:43.021718+0000","last_active":"2021-03-15T22:30:48.236467+0000","last_peered":"2021-03-15T22:30:48.236467+0000","last_clean":"2021-03-15T22:30:48.236467+0000","last_became_active":"2021-03-15T22:30:32.250316+0000","last_became_peered":"2021-03-15T22:30:32.250316+0000","last_unstale":"2021-03-15T22:30:48.236467+0000","last_undegraded":"2021-03-15T22:30:48.236467+0000","last_fullsized":"2021-03-15T22:30:48.236467+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:43.021632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:30:43.021632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"440","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.222880+0000","last_change":"2021-03-15T22:29:56.187294+0000","last_active":"2021-03-15T22:30:50.222880+0000","last_peered":"2021-03-15T22:30:50.222880+0000","last_clean":"2021-03-15T22:30:50.222880+0000","last_became_active":"2021-03-15T22:29:51.222262+0000","last_became_peered":"2021-03-15T22:29:51.222262+0000","last_unstale":"2021-03-15T22:30:50.222880+0000","last_undegraded":"2021-03-15T22:30:50.222880+0000","last_fullsized":"2021-03-15T22:30:50.222880+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:29:56.187210+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.187210+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"484","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.237314+0000","last_change":"2021-03-15T22:30:32.252681+0000","last_active":"2021-03-15T22:30:48.237314+0000","last_peered":"2021-03-15T22:30:48.237314+0000","last_clean":"2021-03-15T22:30:48.237314+0000","last_became_active":"2021-03-15T22:30:32.252163+0000","last_became_peered":"2021-03-15T22:30:32.252163+0000","last_unstale":"2021-03-15T22:30:48.237314+0000","last_undegraded":"2021-03-15T22:30:48.237314+0000","last_fullsized":"2021-03-15T22:30:48.237314+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:55.737371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:55.737371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"701'1969","reported_seq":"4439","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.250435+0000","last_change":"2021-03-15T22:30:48.261428+0000","last_active":"2021-03-15T22:30:50.250435+0000","last_peered":"2021-03-15T22:30:50.250435+0000","last_clean":"2021-03-15T22:30:50.250435+0000","last_became_active":"2021-03-15T22:28:59.276888+0000","last_became_peered":"2021-03-15T22:28:59.276888+0000","last_unstale":"2021-03-15T22:30:50.250435+0000","last_undegraded":"2021-03-15T22:30:50.250435+0000","last_fullsized":"2021-03-15T22:30:50.250435+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"581'1398","last_scrub_stamp":"2021-03-15T22:28:51.825151+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:28:51.825151+0000","log_size":1969,"ondisk_log_size":1969,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":77674223,"num_objects":41,"num_object_clones":34,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":2,"num_read":661,"num_read_kb":654741,"num_write":524,"num_write_kb":222714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":205133613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"72","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225925+0000","last_change":"2021-03-15T22:30:28.967439+0000","last_active":"2021-03-15T22:30:47.225925+0000","last_peered":"2021-03-15T22:30:47.225925+0000","last_clean":"2021-03-15T22:30:47.225925+0000","last_became_active":"2021-03-15T22:29:51.227896+0000","last_became_peered":"2021-03-15T22:29:51.227896+0000","last_unstale":"2021-03-15T22:30:47.225925+0000","last_undegraded":"2021-03-15T22:30:47.225925+0000","last_fullsized":"2021-03-15T22:30:47.225925+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:28.967356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.967356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"689","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.232366+0000","last_change":"2021-03-15T22:30:05.206297+0000","last_active":"2021-03-15T22:30:47.232366+0000","last_peered":"2021-03-15T22:30:47.232366+0000","last_clean":"2021-03-15T22:30:47.232366+0000","last_became_active":"2021-03-15T22:28:12.507833+0000","last_became_peered":"2021-03-15T22:28:12.507833+0000","last_unstale":"2021-03-15T22:30:47.232366+0000","last_undegraded":"2021-03-15T22:30:47.232366+0000","last_fullsized":"2021-03-15T22:30:47.232366+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:05.206223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:30:05.206223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"310","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.240036+0000","last_change":"2021-03-15T22:29:51.223894+0000","last_active":"2021-03-15T22:30:50.240036+0000","last_peered":"2021-03-15T22:30:50.240036+0000","last_clean":"2021-03-15T22:30:50.240036+0000","last_became_active":"2021-03-15T22:29:51.223572+0000","last_became_peered":"2021-03-15T22:29:51.223572+0000","last_unstale":"2021-03-15T22:30:50.240036+0000","last_undegraded":"2021-03-15T22:30:50.240036+0000","last_fullsized":"2021-03-15T22:30:50.240036+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:45.454461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.454461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"723","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.242605+0000","last_change":"2021-03-15T22:30:28.176902+0000","last_active":"2021-03-15T22:30:49.242605+0000","last_peered":"2021-03-15T22:30:49.242605+0000","last_clean":"2021-03-15T22:30:49.242605+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:30:49.242605+0000","last_undegraded":"2021-03-15T22:30:49.242605+0000","last_fullsized":"2021-03-15T22:30:49.242605+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:30:28.176831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.176831+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"710","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.233949+0000","last_change":"2021-03-15T22:30:32.244033+0000","last_active":"2021-03-15T22:30:47.233949+0000","last_peered":"2021-03-15T22:30:47.233949+0000","last_clean":"2021-03-15T22:30:47.233949+0000","last_became_active":"2021-03-15T22:30:32.243696+0000","last_became_peered":"2021-03-15T22:30:32.243696+0000","last_unstale":"2021-03-15T22:30:47.233949+0000","last_undegraded":"2021-03-15T22:30:47.233949+0000","last_fullsized":"2021-03-15T22:30:47.233949+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.257476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"698'546","reported_seq":"2531","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.241240+0000","last_change":"2021-03-15T22:30:48.242922+0000","last_active":"2021-03-15T22:30:48.241240+0000","last_peered":"2021-03-15T22:30:48.241240+0000","last_clean":"2021-03-15T22:30:48.241240+0000","last_became_active":"2021-03-15T22:30:35.366323+0000","last_became_peered":"2021-03-15T22:30:35.366323+0000","last_unstale":"2021-03-15T22:30:48.241240+0000","last_undegraded":"2021-03-15T22:30:48.241240+0000","last_fullsized":"2021-03-15T22:30:48.241240+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":0,"last_scrub":"560'396","last_scrub_stamp":"2021-03-15T22:28:45.222374+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.222374+0000","log_size":546,"ondisk_log_size":546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27005715,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":160,"num_read_kb":157135,"num_write":148,"num_write_kb":66328,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":54764539,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"1f8","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"221","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"740","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.223459+0000","last_change":"2021-03-15T22:30:41.171985+0000","last_active":"2021-03-15T22:30:50.223459+0000","last_peered":"2021-03-15T22:30:50.223459+0000","last_clean":"2021-03-15T22:30:50.223459+0000","last_became_active":"2021-03-15T22:29:51.222189+0000","last_became_peered":"2021-03-15T22:29:51.222189+0000","last_unstale":"2021-03-15T22:30:50.223459+0000","last_undegraded":"2021-03-15T22:30:50.223459+0000","last_fullsized":"2021-03-15T22:30:50.223459+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:41.171920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.171920+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"179","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241067+0000","last_change":"2021-03-15T22:30:31.762079+0000","last_active":"2021-03-15T22:30:49.241067+0000","last_peered":"2021-03-15T22:30:49.241067+0000","last_clean":"2021-03-15T22:30:49.241067+0000","last_became_active":"2021-03-15T22:28:59.276599+0000","last_became_peered":"2021-03-15T22:28:59.276599+0000","last_unstale":"2021-03-15T22:30:49.241067+0000","last_undegraded":"2021-03-15T22:30:49.241067+0000","last_fullsized":"2021-03-15T22:30:49.241067+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:17.909060+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:17.909060+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"1461","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.242602+0000","last_change":"2021-03-15T22:28:52.152942+0000","last_active":"2021-03-15T22:30:48.242602+0000","last_peered":"2021-03-15T22:30:48.242602+0000","last_clean":"2021-03-15T22:30:48.242602+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:30:48.242602+0000","last_undegraded":"2021-03-15T22:30:48.242602+0000","last_fullsized":"2021-03-15T22:30:48.242602+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:28:52.152875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:28:52.152875+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":759,"num_read_kb":754,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"226","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.242348+0000","last_change":"2021-03-15T22:30:32.617969+0000","last_active":"2021-03-15T22:30:49.242348+0000","last_peered":"2021-03-15T22:30:49.242348+0000","last_clean":"2021-03-15T22:30:49.242348+0000","last_became_active":"2021-03-15T22:30:32.251686+0000","last_became_peered":"2021-03-15T22:30:32.251686+0000","last_unstale":"2021-03-15T22:30:49.242348+0000","last_undegraded":"2021-03-15T22:30:49.242348+0000","last_fullsized":"2021-03-15T22:30:49.242348+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:28:45.780434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.780434+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"67","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225933+0000","last_change":"2021-03-15T22:29:51.232258+0000","last_active":"2021-03-15T22:30:47.225933+0000","last_peered":"2021-03-15T22:30:47.225933+0000","last_clean":"2021-03-15T22:30:47.225933+0000","last_became_active":"2021-03-15T22:29:51.231915+0000","last_became_peered":"2021-03-15T22:29:51.231915+0000","last_unstale":"2021-03-15T22:30:47.225933+0000","last_undegraded":"2021-03-15T22:30:47.225933+0000","last_fullsized":"2021-03-15T22:30:47.225933+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:16.238757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:29:16.238757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"699'2081","reported_seq":"4518","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.259974+0000","last_change":"2021-03-15T22:30:48.260247+0000","last_active":"2021-03-15T22:30:48.259974+0000","last_peered":"2021-03-15T22:30:48.259974+0000","last_clean":"2021-03-15T22:30:48.259974+0000","last_became_active":"2021-03-15T22:29:52.225069+0000","last_became_peered":"2021-03-15T22:29:52.225069+0000","last_unstale":"2021-03-15T22:30:48.259974+0000","last_undegraded":"2021-03-15T22:30:48.259974+0000","last_fullsized":"2021-03-15T22:30:48.259974+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"650'1835","last_scrub_stamp":"2021-03-15T22:29:59.763141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:59.763141+0000","log_size":2081,"ondisk_log_size":2081,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92141468,"num_objects":45,"num_object_clones":38,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":2,"num_read":896,"num_read_kb":979284,"num_write":551,"num_write_kb":224995,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":165660176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"1f8","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"221","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"680","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.221886+0000","last_change":"2021-03-15T22:30:33.027115+0000","last_active":"2021-03-15T22:30:50.221886+0000","last_peered":"2021-03-15T22:30:50.221886+0000","last_clean":"2021-03-15T22:30:50.221886+0000","last_became_active":"2021-03-15T22:30:31.240169+0000","last_became_peered":"2021-03-15T22:30:31.240169+0000","last_unstale":"2021-03-15T22:30:50.221886+0000","last_undegraded":"2021-03-15T22:30:50.221886+0000","last_fullsized":"2021-03-15T22:30:50.221886+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:24:30.950135+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:24:30.950135+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"720","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.223570+0000","last_change":"2021-03-15T22:30:32.244248+0000","last_active":"2021-03-15T22:30:50.223570+0000","last_peered":"2021-03-15T22:30:50.223570+0000","last_clean":"2021-03-15T22:30:50.223570+0000","last_became_active":"2021-03-15T22:30:32.243790+0000","last_became_peered":"2021-03-15T22:30:32.243790+0000","last_unstale":"2021-03-15T22:30:50.223570+0000","last_undegraded":"2021-03-15T22:30:50.223570+0000","last_fullsized":"2021-03-15T22:30:50.223570+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:32.161342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:32.161342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"238","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241281+0000","last_change":"2021-03-15T22:29:52.258695+0000","last_active":"2021-03-15T22:30:49.241281+0000","last_peered":"2021-03-15T22:30:49.241281+0000","last_clean":"2021-03-15T22:30:49.241281+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:30:49.241281+0000","last_undegraded":"2021-03-15T22:30:49.241281+0000","last_fullsized":"2021-03-15T22:30:49.241281+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:29:08.872836+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:29:08.872836+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"715","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241923+0000","last_change":"2021-03-15T22:29:51.233266+0000","last_active":"2021-03-15T22:30:49.241923+0000","last_peered":"2021-03-15T22:30:49.241923+0000","last_clean":"2021-03-15T22:30:49.241923+0000","last_became_active":"2021-03-15T22:29:51.232811+0000","last_became_peered":"2021-03-15T22:29:51.232811+0000","last_unstale":"2021-03-15T22:30:49.241923+0000","last_undegraded":"2021-03-15T22:30:49.241923+0000","last_fullsized":"2021-03-15T22:30:49.241923+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:33.255248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.255248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"702'351","reported_seq":"2354","reported_epoch":"702","state":"active+clean","last_fresh":"2021-03-15T22:30:51.246767+0000","last_change":"2021-03-15T22:30:51.247145+0000","last_active":"2021-03-15T22:30:51.246767+0000","last_peered":"2021-03-15T22:30:51.246767+0000","last_clean":"2021-03-15T22:30:51.246767+0000","last_became_active":"2021-03-15T22:29:03.354952+0000","last_became_peered":"2021-03-15T22:29:03.354952+0000","last_unstale":"2021-03-15T22:30:51.246767+0000","last_undegraded":"2021-03-15T22:30:51.246767+0000","last_fullsized":"2021-03-15T22:30:51.246767+0000","mapping_epoch":593,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"600'265","last_scrub_stamp":"2021-03-15T22:29:09.816963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:09.816963+0000","log_size":351,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11072175,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":120,"num_read_kb":143126,"num_write":86,"num_write_kb":32987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":41911448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"1fb","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"192","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.240931+0000","last_change":"2021-03-15T22:30:32.238279+0000","last_active":"2021-03-15T22:30:49.240931+0000","last_peered":"2021-03-15T22:30:49.240931+0000","last_clean":"2021-03-15T22:30:49.240931+0000","last_became_active":"2021-03-15T22:30:32.237925+0000","last_became_peered":"2021-03-15T22:30:32.237925+0000","last_unstale":"2021-03-15T22:30:49.240931+0000","last_undegraded":"2021-03-15T22:30:49.240931+0000","last_fullsized":"2021-03-15T22:30:49.240931+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:37.724915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:37.724915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"633","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225674+0000","last_change":"2021-03-15T22:30:32.257626+0000","last_active":"2021-03-15T22:30:47.225674+0000","last_peered":"2021-03-15T22:30:47.225674+0000","last_clean":"2021-03-15T22:30:47.225674+0000","last_became_active":"2021-03-15T22:30:32.257328+0000","last_became_peered":"2021-03-15T22:30:32.257328+0000","last_unstale":"2021-03-15T22:30:47.225674+0000","last_undegraded":"2021-03-15T22:30:47.225674+0000","last_fullsized":"2021-03-15T22:30:47.225674+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:58.713662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:29:58.713662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"905","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.222390+0000","last_change":"2021-03-15T22:29:52.437785+0000","last_active":"2021-03-15T22:30:50.222390+0000","last_peered":"2021-03-15T22:30:50.222390+0000","last_clean":"2021-03-15T22:30:50.222390+0000","last_became_active":"2021-03-15T22:29:51.222356+0000","last_became_peered":"2021-03-15T22:29:51.222356+0000","last_unstale":"2021-03-15T22:30:50.222390+0000","last_undegraded":"2021-03-15T22:30:50.222390+0000","last_fullsized":"2021-03-15T22:30:50.222390+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:29:20.139319+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"642'8","reported_seq":"225","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241179+0000","last_change":"2021-03-15T22:29:51.399268+0000","last_active":"2021-03-15T22:30:49.241179+0000","last_peered":"2021-03-15T22:30:49.241179+0000","last_clean":"2021-03-15T22:30:49.241179+0000","last_became_active":"2021-03-15T22:29:51.215636+0000","last_became_peered":"2021-03-15T22:29:51.215636+0000","last_unstale":"2021-03-15T22:30:49.241179+0000","last_undegraded":"2021-03-15T22:30:49.241179+0000","last_fullsized":"2021-03-15T22:30:49.241179+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'7","last_scrub_stamp":"2021-03-15T22:29:12.845802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:12.845802+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"702'548","reported_seq":"2460","reported_epoch":"702","state":"active+clean","last_fresh":"2021-03-15T22:30:51.257546+0000","last_change":"2021-03-15T22:30:51.257694+0000","last_active":"2021-03-15T22:30:51.257546+0000","last_peered":"2021-03-15T22:30:51.257546+0000","last_clean":"2021-03-15T22:30:51.257546+0000","last_became_active":"2021-03-15T22:30:34.297623+0000","last_became_peered":"2021-03-15T22:30:34.297623+0000","last_unstale":"2021-03-15T22:30:51.257546+0000","last_undegraded":"2021-03-15T22:30:51.257546+0000","last_fullsized":"2021-03-15T22:30:51.257546+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'390","last_scrub_stamp":"2021-03-15T22:28:47.641188+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:28:47.641188+0000","log_size":548,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25460626,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":191,"num_read_kb":173087,"num_write":162,"num_write_kb":61734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":47935099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"1fb","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"670","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.232825+0000","last_change":"2021-03-15T22:30:31.243805+0000","last_active":"2021-03-15T22:30:47.232825+0000","last_peered":"2021-03-15T22:30:47.232825+0000","last_clean":"2021-03-15T22:30:47.232825+0000","last_became_active":"2021-03-15T22:30:31.243392+0000","last_became_peered":"2021-03-15T22:30:31.243392+0000","last_unstale":"2021-03-15T22:30:47.232825+0000","last_undegraded":"2021-03-15T22:30:47.232825+0000","last_fullsized":"2021-03-15T22:30:47.232825+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.234565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.234565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"163","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.242390+0000","last_change":"2021-03-15T22:30:37.798454+0000","last_active":"2021-03-15T22:30:49.242390+0000","last_peered":"2021-03-15T22:30:49.242390+0000","last_clean":"2021-03-15T22:30:49.242390+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:30:49.242390+0000","last_undegraded":"2021-03-15T22:30:49.242390+0000","last_fullsized":"2021-03-15T22:30:49.242390+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.798388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"815","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249821+0000","last_change":"2021-03-15T22:30:42.383592+0000","last_active":"2021-03-15T22:30:50.249821+0000","last_peered":"2021-03-15T22:30:50.249821+0000","last_clean":"2021-03-15T22:30:50.249821+0000","last_became_active":"2021-03-15T22:29:51.216039+0000","last_became_peered":"2021-03-15T22:29:51.216039+0000","last_unstale":"2021-03-15T22:30:50.249821+0000","last_undegraded":"2021-03-15T22:30:50.249821+0000","last_fullsized":"2021-03-15T22:30:50.249821+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:30:42.383512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.383512+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"153","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239318+0000","last_change":"2021-03-15T22:30:27.472423+0000","last_active":"2021-03-15T22:30:50.239318+0000","last_peered":"2021-03-15T22:30:50.239318+0000","last_clean":"2021-03-15T22:30:50.239318+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:30:50.239318+0000","last_undegraded":"2021-03-15T22:30:50.239318+0000","last_fullsized":"2021-03-15T22:30:50.239318+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:30:27.472320+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:30:27.472320+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"702'941","reported_seq":"3065","reported_epoch":"702","state":"active+clean","last_fresh":"2021-03-15T22:30:51.259036+0000","last_change":"2021-03-15T22:30:51.259238+0000","last_active":"2021-03-15T22:30:51.259036+0000","last_peered":"2021-03-15T22:30:51.259036+0000","last_clean":"2021-03-15T22:30:51.259036+0000","last_became_active":"2021-03-15T22:30:34.298185+0000","last_became_peered":"2021-03-15T22:30:34.298185+0000","last_unstale":"2021-03-15T22:30:51.259036+0000","last_undegraded":"2021-03-15T22:30:51.259036+0000","last_fullsized":"2021-03-15T22:30:51.259036+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'889","last_scrub_stamp":"2021-03-15T22:30:29.983045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.983045+0000","log_size":941,"ondisk_log_size":941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30843430,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":364,"num_read_kb":380939,"num_write":242,"num_write_kb":83176,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":109136440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"1fb","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"729","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249231+0000","last_change":"2021-03-15T22:30:32.262270+0000","last_active":"2021-03-15T22:30:50.249231+0000","last_peered":"2021-03-15T22:30:50.249231+0000","last_clean":"2021-03-15T22:30:50.249231+0000","last_became_active":"2021-03-15T22:30:32.261995+0000","last_became_peered":"2021-03-15T22:30:32.261995+0000","last_unstale":"2021-03-15T22:30:50.249231+0000","last_undegraded":"2021-03-15T22:30:50.249231+0000","last_fullsized":"2021-03-15T22:30:50.249231+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:09.101167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:09.101167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"477","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.240750+0000","last_change":"2021-03-15T22:29:00.238851+0000","last_active":"2021-03-15T22:30:50.240750+0000","last_peered":"2021-03-15T22:30:50.240750+0000","last_clean":"2021-03-15T22:30:50.240750+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:30:50.240750+0000","last_undegraded":"2021-03-15T22:30:50.240750+0000","last_fullsized":"2021-03-15T22:30:50.240750+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:37.136505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:28:37.136505+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"153","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239433+0000","last_change":"2021-03-15T22:29:41.515256+0000","last_active":"2021-03-15T22:30:50.239433+0000","last_peered":"2021-03-15T22:30:50.239433+0000","last_clean":"2021-03-15T22:30:50.239433+0000","last_became_active":"2021-03-15T22:28:59.276755+0000","last_became_peered":"2021-03-15T22:28:59.276755+0000","last_unstale":"2021-03-15T22:30:50.239433+0000","last_undegraded":"2021-03-15T22:30:50.239433+0000","last_fullsized":"2021-03-15T22:30:50.239433+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:29:41.515193+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.515193+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"719","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249589+0000","last_change":"2021-03-15T22:29:19.876088+0000","last_active":"2021-03-15T22:30:50.249589+0000","last_peered":"2021-03-15T22:30:50.249589+0000","last_clean":"2021-03-15T22:30:50.249589+0000","last_became_active":"2021-03-15T22:28:59.266487+0000","last_became_peered":"2021-03-15T22:28:59.266487+0000","last_unstale":"2021-03-15T22:30:50.249589+0000","last_undegraded":"2021-03-15T22:30:50.249589+0000","last_fullsized":"2021-03-15T22:30:50.249589+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:19.876006+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:29:19.876006+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"699'887","reported_seq":"2798","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.228100+0000","last_change":"2021-03-15T22:30:48.248366+0000","last_active":"2021-03-15T22:30:50.228100+0000","last_peered":"2021-03-15T22:30:50.228100+0000","last_clean":"2021-03-15T22:30:50.228100+0000","last_became_active":"2021-03-15T22:29:00.220080+0000","last_became_peered":"2021-03-15T22:29:00.220080+0000","last_unstale":"2021-03-15T22:30:50.228100+0000","last_undegraded":"2021-03-15T22:30:50.228100+0000","last_fullsized":"2021-03-15T22:30:50.228100+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'732","last_scrub_stamp":"2021-03-15T22:29:36.271647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:36.271647+0000","log_size":887,"ondisk_log_size":887,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26493547,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":289,"num_read_kb":275921,"num_write":211,"num_write_kb":75302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25937939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"618","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.225634+0000","last_change":"2021-03-15T22:29:52.227178+0000","last_active":"2021-03-15T22:30:47.225634+0000","last_peered":"2021-03-15T22:30:47.225634+0000","last_clean":"2021-03-15T22:30:47.225634+0000","last_became_active":"2021-03-15T22:29:52.223342+0000","last_became_peered":"2021-03-15T22:29:52.223342+0000","last_unstale":"2021-03-15T22:30:47.225634+0000","last_undegraded":"2021-03-15T22:30:47.225634+0000","last_fullsized":"2021-03-15T22:30:47.225634+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:35.243847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:35.243847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"643'13","reported_seq":"921","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.250372+0000","last_change":"2021-03-15T22:29:52.212266+0000","last_active":"2021-03-15T22:30:50.250372+0000","last_peered":"2021-03-15T22:30:50.250372+0000","last_clean":"2021-03-15T22:30:50.250372+0000","last_became_active":"2021-03-15T22:29:52.211688+0000","last_became_peered":"2021-03-15T22:29:52.211688+0000","last_unstale":"2021-03-15T22:30:50.250372+0000","last_undegraded":"2021-03-15T22:30:50.250372+0000","last_fullsized":"2021-03-15T22:30:50.250372+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'11","last_scrub_stamp":"2021-03-15T22:29:36.937870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:36.937870+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"127","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239194+0000","last_change":"2021-03-15T22:30:09.506976+0000","last_active":"2021-03-15T22:30:50.239194+0000","last_peered":"2021-03-15T22:30:50.239194+0000","last_clean":"2021-03-15T22:30:50.239194+0000","last_became_active":"2021-03-15T22:28:59.274186+0000","last_became_peered":"2021-03-15T22:28:59.274186+0000","last_unstale":"2021-03-15T22:30:50.239194+0000","last_undegraded":"2021-03-15T22:30:50.239194+0000","last_fullsized":"2021-03-15T22:30:50.239194+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:09.506914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:30:09.506914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"546","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.236739+0000","last_change":"2021-03-15T22:30:33.332713+0000","last_active":"2021-03-15T22:30:48.236739+0000","last_peered":"2021-03-15T22:30:48.236739+0000","last_clean":"2021-03-15T22:30:48.236739+0000","last_became_active":"2021-03-15T22:30:32.262267+0000","last_became_peered":"2021-03-15T22:30:32.262267+0000","last_unstale":"2021-03-15T22:30:48.236739+0000","last_undegraded":"2021-03-15T22:30:48.236739+0000","last_fullsized":"2021-03-15T22:30:48.236739+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:29:15.297479+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:15.297479+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"698'287","reported_seq":"2173","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.228097+0000","last_change":"2021-03-15T22:30:48.236047+0000","last_active":"2021-03-15T22:30:50.228097+0000","last_peered":"2021-03-15T22:30:50.228097+0000","last_clean":"2021-03-15T22:30:50.228097+0000","last_became_active":"2021-03-15T22:28:15.172010+0000","last_became_peered":"2021-03-15T22:28:15.172010+0000","last_unstale":"2021-03-15T22:30:50.228097+0000","last_undegraded":"2021-03-15T22:30:50.228097+0000","last_fullsized":"2021-03-15T22:30:50.228097+0000","mapping_epoch":555,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'281","last_scrub_stamp":"2021-03-15T22:30:36.290657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.290657+0000","log_size":287,"ondisk_log_size":287,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15228363,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":108,"num_read_kb":100343,"num_write":61,"num_write_kb":21303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":9579561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"416","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239660+0000","last_change":"2021-03-15T22:29:51.225632+0000","last_active":"2021-03-15T22:30:50.239660+0000","last_peered":"2021-03-15T22:30:50.239660+0000","last_clean":"2021-03-15T22:30:50.239660+0000","last_became_active":"2021-03-15T22:29:51.225325+0000","last_became_peered":"2021-03-15T22:29:51.225325+0000","last_unstale":"2021-03-15T22:30:50.239660+0000","last_undegraded":"2021-03-15T22:30:50.239660+0000","last_fullsized":"2021-03-15T22:30:50.239660+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:40.500803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.500803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"671","reported_epoch":"698","state":"active+clean","last_fresh":"2021-03-15T22:30:47.226069+0000","last_change":"2021-03-15T22:29:52.217795+0000","last_active":"2021-03-15T22:30:47.226069+0000","last_peered":"2021-03-15T22:30:47.226069+0000","last_clean":"2021-03-15T22:30:47.226069+0000","last_became_active":"2021-03-15T22:29:52.217407+0000","last_became_peered":"2021-03-15T22:29:52.217407+0000","last_unstale":"2021-03-15T22:30:47.226069+0000","last_undegraded":"2021-03-15T22:30:47.226069+0000","last_fullsized":"2021-03-15T22:30:47.226069+0000","mapping_epoch":642,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'7","last_scrub_stamp":"2021-03-15T22:29:40.964753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.964753+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"688","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249735+0000","last_change":"2021-03-15T22:30:34.363233+0000","last_active":"2021-03-15T22:30:50.249735+0000","last_peered":"2021-03-15T22:30:50.249735+0000","last_clean":"2021-03-15T22:30:50.249735+0000","last_became_active":"2021-03-15T22:30:31.239376+0000","last_became_peered":"2021-03-15T22:30:31.239376+0000","last_unstale":"2021-03-15T22:30:50.249735+0000","last_undegraded":"2021-03-15T22:30:50.249735+0000","last_fullsized":"2021-03-15T22:30:50.249735+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.363177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"776","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.250284+0000","last_change":"2021-03-15T22:30:04.058254+0000","last_active":"2021-03-15T22:30:50.250284+0000","last_peered":"2021-03-15T22:30:50.250284+0000","last_clean":"2021-03-15T22:30:50.250284+0000","last_became_active":"2021-03-15T22:24:55.090544+0000","last_became_peered":"2021-03-15T22:24:55.090544+0000","last_unstale":"2021-03-15T22:30:50.250284+0000","last_undegraded":"2021-03-15T22:30:50.250284+0000","last_fullsized":"2021-03-15T22:30:50.250284+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:30:04.058167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:04.058167+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"700'261","reported_seq":"1641","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.248154+0000","last_change":"2021-03-15T22:30:48.245608+0000","last_active":"2021-03-15T22:30:49.248154+0000","last_peered":"2021-03-15T22:30:49.248154+0000","last_clean":"2021-03-15T22:30:49.248154+0000","last_became_active":"2021-03-15T22:30:32.257058+0000","last_became_peered":"2021-03-15T22:30:32.257058+0000","last_unstale":"2021-03-15T22:30:49.248154+0000","last_undegraded":"2021-03-15T22:30:49.248154+0000","last_fullsized":"2021-03-15T22:30:49.248154+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"646'221","last_scrub_stamp":"2021-03-15T22:29:56.742213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.742213+0000","log_size":261,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15889430,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":164357,"num_write":78,"num_write_kb":26848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":32799803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"1f8","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"221","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"503","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.236949+0000","last_change":"2021-03-15T22:30:32.267034+0000","last_active":"2021-03-15T22:30:48.236949+0000","last_peered":"2021-03-15T22:30:48.236949+0000","last_clean":"2021-03-15T22:30:48.236949+0000","last_became_active":"2021-03-15T22:30:32.266716+0000","last_became_peered":"2021-03-15T22:30:32.266716+0000","last_unstale":"2021-03-15T22:30:48.236949+0000","last_undegraded":"2021-03-15T22:30:48.236949+0000","last_fullsized":"2021-03-15T22:30:48.236949+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:17.298043+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:29:17.298043+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"716","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.223742+0000","last_change":"2021-03-15T22:29:41.190121+0000","last_active":"2021-03-15T22:30:50.223742+0000","last_peered":"2021-03-15T22:30:50.223742+0000","last_clean":"2021-03-15T22:30:50.223742+0000","last_became_active":"2021-03-15T22:28:12.503247+0000","last_became_peered":"2021-03-15T22:28:12.503247+0000","last_unstale":"2021-03-15T22:30:50.223742+0000","last_undegraded":"2021-03-15T22:30:50.223742+0000","last_fullsized":"2021-03-15T22:30:50.223742+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:41.190058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.190058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"688","reported_epoch":"700","state":"active+clean","last_fresh":"2021-03-15T22:30:49.241716+0000","last_change":"2021-03-15T22:30:43.259679+0000","last_active":"2021-03-15T22:30:49.241716+0000","last_peered":"2021-03-15T22:30:49.241716+0000","last_clean":"2021-03-15T22:30:49.241716+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:30:49.241716+0000","last_undegraded":"2021-03-15T22:30:49.241716+0000","last_fullsized":"2021-03-15T22:30:49.241716+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:43.259585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:43.259585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"772","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.250076+0000","last_change":"2021-03-15T22:30:38.419647+0000","last_active":"2021-03-15T22:30:50.250076+0000","last_peered":"2021-03-15T22:30:50.250076+0000","last_clean":"2021-03-15T22:30:50.250076+0000","last_became_active":"2021-03-15T22:30:31.239360+0000","last_became_peered":"2021-03-15T22:30:31.239360+0000","last_unstale":"2021-03-15T22:30:50.250076+0000","last_undegraded":"2021-03-15T22:30:50.250076+0000","last_fullsized":"2021-03-15T22:30:50.250076+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:30:38.419551+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.419551+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"699'931","reported_seq":"3193","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.223542+0000","last_change":"2021-03-15T22:30:48.255741+0000","last_active":"2021-03-15T22:30:50.223542+0000","last_peered":"2021-03-15T22:30:50.223542+0000","last_clean":"2021-03-15T22:30:50.223542+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:30:50.223542+0000","last_undegraded":"2021-03-15T22:30:50.223542+0000","last_fullsized":"2021-03-15T22:30:50.223542+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'877","last_scrub_stamp":"2021-03-15T22:30:27.732142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:27.732142+0000","log_size":931,"ondisk_log_size":931,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":45934424,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":380,"num_read_kb":379482,"num_write":254,"num_write_kb":89926,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d5","length":"1"},{"start":"216","length":"1"},{"start":"219","length":"1"},{"start":"226","length":"1"},{"start":"228","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"680","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.250381+0000","last_change":"2021-03-15T22:30:36.417325+0000","last_active":"2021-03-15T22:30:50.250381+0000","last_peered":"2021-03-15T22:30:50.250381+0000","last_clean":"2021-03-15T22:30:50.250381+0000","last_became_active":"2021-03-15T22:30:31.243269+0000","last_became_peered":"2021-03-15T22:30:31.243269+0000","last_unstale":"2021-03-15T22:30:50.250381+0000","last_undegraded":"2021-03-15T22:30:50.250381+0000","last_fullsized":"2021-03-15T22:30:50.250381+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"683'12","reported_seq":"603","reported_epoch":"699","state":"active+clean","last_fresh":"2021-03-15T22:30:48.237223+0000","last_change":"2021-03-15T22:30:32.259770+0000","last_active":"2021-03-15T22:30:48.237223+0000","last_peered":"2021-03-15T22:30:48.237223+0000","last_clean":"2021-03-15T22:30:48.237223+0000","last_became_active":"2021-03-15T22:30:32.256541+0000","last_became_peered":"2021-03-15T22:30:32.256541+0000","last_unstale":"2021-03-15T22:30:48.237223+0000","last_undegraded":"2021-03-15T22:30:48.237223+0000","last_fullsized":"2021-03-15T22:30:48.237223+0000","mapping_epoch":682,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'8","last_scrub_stamp":"2021-03-15T22:29:38.210344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:38.210344+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"127","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.239399+0000","last_change":"2021-03-15T22:30:03.567594+0000","last_active":"2021-03-15T22:30:50.239399+0000","last_peered":"2021-03-15T22:30:50.239399+0000","last_clean":"2021-03-15T22:30:50.239399+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:30:50.239399+0000","last_undegraded":"2021-03-15T22:30:50.239399+0000","last_fullsized":"2021-03-15T22:30:50.239399+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.567529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"825","reported_epoch":"701","state":"active+clean","last_fresh":"2021-03-15T22:30:50.249774+0000","last_change":"2021-03-15T22:30:00.030068+0000","last_active":"2021-03-15T22:30:50.249774+0000","last_peered":"2021-03-15T22:30:50.249774+0000","last_clean":"2021-03-15T22:30:50.249774+0000","last_became_active":"2021-03-15T22:28:59.268492+0000","last_became_peered":"2021-03-15T22:28:59.268492+0000","last_unstale":"2021-03-15T22:30:50.249774+0000","last_undegraded":"2021-03-15T22:30:50.249774+0000","last_fullsized":"2021-03-15T22:30:50.249774+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:30:00.029954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:00.029954+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":564915755,"num_objects":277,"num_object_clones":227,"num_object_copies":831,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":277,"num_whiteouts":13,"num_read":4967,"num_read_kb":5075008,"num_write":3592,"num_write_kb":1361188,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":1078884691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1262063616,"data_stored":1818678868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13180,"ondisk_log_size":13180,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":759,"num_read_kb":754,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1484,"num_read_kb":989,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":773,"num_bytes_recovered":59472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":916,"ondisk_log_size":916,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299551,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":196380,"kb_used_data":154452,"kb_used_omap":0,"kb_used_meta":41920,"kb_avail":93548772,"statfs":{"total":95995035648,"available":95793942528,"internally_reserved":0,"allocated":158158848,"data_stored":232412244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42926080},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.45900000000000002,"15min":0.48099999999999998},"min":{"1min":0.313,"5min":0.25,"15min":0.215},"max":{"1min":0.55200000000000005,"5min":1.4750000000000001,"15min":2.73},"last":0.498},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.48199999999999998,"15min":0.49199999999999999},"min":{"1min":0.36799999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.62,"5min":2.0499999999999998,"15min":3.8799999999999999},"last":0.56100000000000005}]},{"osd":1,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46999999999999997,"15min":0.48299999999999998},"min":{"1min":0.32000000000000001,"5min":0.27300000000000002,"15min":0.22600000000000001},"max":{"1min":0.629,"5min":1.4350000000000001,"15min":2.6859999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.497,"15min":0.49099999999999999},"min":{"1min":0.34699999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.64300000000000002,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.54200000000000004}]},{"osd":2,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.80700000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.32400000000000001,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":0.73099999999999998,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.505}]},{"osd":3,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.502,"15min":0.51600000000000001},"min":{"1min":0.33800000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.0249999999999999,"5min":1.702,"15min":2.0339999999999998},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.54500000000000004,"15min":0.54600000000000004},"min":{"1min":0.34699999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58499999999999996,"5min":5.415,"15min":5.415},"last":0.59599999999999997}]},{"osd":4,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":0.64400000000000002,"5min":1.927,"15min":1.927},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.44600000000000001,"15min":0.45400000000000001},"min":{"1min":0.29199999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.621,"5min":1.6779999999999999,"15min":2.5339999999999998},"last":0.31}]},{"osd":5,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.38200000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.48299999999999998,"15min":0.50900000000000001},"min":{"1min":0.313,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59299999999999997,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.48999999999999999,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.097,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":0.34899999999999998}]}]},{"osd":6,"up_from":45,"seq":193273528483,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":147432,"kb_used_data":108832,"kb_used_omap":0,"kb_used_meta":38592,"kb_avail":93597720,"statfs":{"total":95995035648,"available":95844065280,"internally_reserved":0,"allocated":111443968,"data_stored":166486689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39518208},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:29:56 2021","interfaces":[{"interface":"back","average":{"1min":0.36299999999999999,"5min":0.47299999999999998,"15min":0.48299999999999998},"min":{"1min":0.253,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.52400000000000002,"5min":3.085,"15min":3.085},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.435,"15min":0.46999999999999997},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27100000000000002},"max":{"1min":0.53700000000000003,"5min":1.323,"15min":1.323},"last":0.53700000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:29:56 2021","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.60899999999999999,"15min":0.54200000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.254},"max":{"1min":0.55400000000000005,"5min":14.637,"15min":14.637},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.38700000000000001,"5min":0.47699999999999998,"15min":0.48499999999999999},"min":{"1min":0.32800000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.47999999999999998,"5min":5.2450000000000001,"15min":5.2450000000000001},"last":0.375}]},{"osd":2,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.317,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65200000000000002,"5min":0.77600000000000002,"15min":0.80800000000000005},"last":0.503},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48899999999999999,"15min":0.498},"min":{"1min":0.35399999999999998,"5min":0.32800000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.47199999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:29:56 2021","interfaces":[{"interface":"back","average":{"1min":0.39500000000000002,"5min":0.503,"15min":0.51400000000000001},"min":{"1min":0.318,"5min":0.318,"15min":0.29899999999999999},"max":{"1min":0.51100000000000001,"5min":2.6150000000000002,"15min":2.6150000000000002},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.40899999999999997,"5min":0.48899999999999999,"15min":0.51200000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.28699999999999998},"max":{"1min":0.56699999999999995,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.44800000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:29:56 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.41899999999999998,"15min":0.42299999999999999},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.497,"5min":4.306,"15min":4.306},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.38400000000000001,"5min":0.48999999999999999,"15min":0.47699999999999998},"min":{"1min":0.252,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.54800000000000004,"5min":4.181,"15min":4.181},"last":0.41099999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.436}]},{"osd":7,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.59399999999999997,"15min":0.54000000000000004},"min":{"1min":0.307,"5min":0.249,"15min":0.249},"max":{"1min":0.58599999999999997,"5min":15.340999999999999,"15min":15.340999999999999},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.63300000000000001,"15min":0.57699999999999996},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63500000000000001,"5min":15.356,"15min":15.356},"last":0.51000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575029,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":231872,"kb_used_data":191096,"kb_used_omap":0,"kb_used_meta":40768,"kb_avail":93513280,"statfs":{"total":95995035648,"available":95757598720,"internally_reserved":0,"allocated":195682304,"data_stored":274533745,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41746432},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:40 2021","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.30399999999999999,"15min":0.51600000000000001},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.325,"5min":1.325,"15min":54.427999999999997},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.32800000000000001,"15min":0.53200000000000003},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":54.356999999999999},"last":0.378}]},{"osd":2,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.54300000000000004,"5min":1.1220000000000001,"15min":2.4100000000000001},"last":0.31},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.38600000000000001},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.502,"5min":0.69799999999999995,"15min":2.0760000000000001},"last":0.31900000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:30:07 2021","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.34000000000000002,"15min":0.38600000000000001},"min":{"1min":0.23100000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.57699999999999996,"5min":0.73499999999999999,"15min":1.796},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.31900000000000001,"5min":0.35699999999999998,"15min":0.41599999999999998},"min":{"1min":0.246,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":0.43099999999999999,"5min":0.69199999999999995,"15min":1.7150000000000001},"last":0.28799999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:30:17 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.53800000000000003},"min":{"1min":0.32700000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.69599999999999995,"5min":0.76600000000000001,"15min":3.3439999999999999},"last":0.621},{"interface":"front","average":{"1min":0.503,"5min":0.53000000000000003,"15min":0.59399999999999997},"min":{"1min":0.317,"5min":0.27200000000000002,"15min":0.253},"max":{"1min":0.77400000000000002,"5min":4.1790000000000003,"15min":4.6139999999999999},"last":0.52200000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.53300000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:30:50 2021","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52600000000000002,"15min":0.59699999999999998},"min":{"1min":0.35499999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.75},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52600000000000002,"15min":0.57899999999999996},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":1.3580000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:30:07 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.53700000000000003,"15min":0.70499999999999996},"min":{"1min":0.376,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.754,"5min":0.85599999999999998,"15min":8.8100000000000005},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.53000000000000003,"15min":0.58299999999999996},"min":{"1min":0.38200000000000001,"5min":0.37,"15min":0.35799999999999998},"max":{"1min":0.73099999999999998,"5min":0.83999999999999997,"15min":1.325},"last":0.67700000000000005}]}]},{"osd":0,"up_from":8,"seq":34359738551,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":246676,"kb_used_data":205388,"kb_used_omap":0,"kb_used_meta":41280,"kb_avail":93498476,"statfs":{"total":95995035648,"available":95742439424,"internally_reserved":0,"allocated":210317312,"data_stored":293828269,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42270720},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:30:33 2021","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.32800000000000001,"15min":0.34200000000000003},"min":{"1min":0.19800000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.435,"5min":1.024,"15min":1.7070000000000001},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.35799999999999998,"15min":0.36499999999999999},"min":{"1min":0.23300000000000001,"5min":0.20699999999999999,"15min":0.19900000000000001},"max":{"1min":0.55100000000000005,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.28499999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:26:35 2021","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39200000000000002,"15min":0.39500000000000002},"min":{"1min":0.23899999999999999,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.76100000000000001,"5min":1.97,"15min":1.97},"last":0.26700000000000002},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.38500000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.20699999999999999},"max":{"1min":1.1419999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.34000000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:30:16 2021","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40100000000000002,"15min":0.39300000000000002},"min":{"1min":0.25,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.82699999999999996,"5min":1.538,"15min":1.538},"last":0.377},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.40000000000000002,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.69799999999999995,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.32100000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:30:16 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52400000000000002,"15min":0.53900000000000003},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.88700000000000001,"5min":1.583,"15min":1.583},"last":0.60699999999999998},{"interface":"front","average":{"1min":0.5,"5min":0.53000000000000003,"15min":0.57099999999999995},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.84099999999999997,"5min":1.645,"15min":1.645},"last":0.629}]},{"osd":5,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.499},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.57099999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:29:50 2021","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.55200000000000005,"15min":0.56100000000000005},"min":{"1min":0.30299999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.80800000000000005,"5min":1.6180000000000001,"15min":1.6180000000000001},"last":0.64500000000000002},{"interface":"front","average":{"1min":0.48099999999999998,"5min":0.54400000000000004,"15min":0.58799999999999997},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.874,"5min":0.98099999999999998,"15min":1.0049999999999999},"last":0.50800000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:30:16 2021","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.52900000000000003,"15min":0.56499999999999995},"min":{"1min":0.34599999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71299999999999997,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.85299999999999998,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.66800000000000004}]}]},{"osd":2,"up_from":681,"seq":2924872728581,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":164052,"kb_used_data":141836,"kb_used_omap":119,"kb_used_meta":22088,"kb_avail":93581100,"statfs":{"total":95995035648,"available":95827046400,"internally_reserved":0,"allocated":145240064,"data_stored":174203451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":22618473},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.26600000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32400000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48599999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45900000000000002}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39600000000000002}]}]},{"osd":3,"up_from":26,"seq":111669149869,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327824,"kb_used_data":274632,"kb_used_omap":0,"kb_used_meta":53184,"kb_avail":93417328,"statfs":{"total":95995035648,"available":95659343872,"internally_reserved":0,"allocated":281223168,"data_stored":413109988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54460416},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:15 2021","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32400000000000001,"15min":0.34100000000000003},"min":{"1min":0.19900000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.95399999999999996,"5min":1.1499999999999999,"15min":2.625},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.316,"15min":0.35099999999999998},"min":{"1min":0.20899999999999999,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":1.419,"5min":1.419,"15min":2.6339999999999999},"last":0.47199999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:30:15 2021","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35699999999999998,"15min":0.40600000000000003},"min":{"1min":0.25,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":6.0199999999999996},"last":0.505},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.36899999999999999,"15min":0.41199999999999998},"min":{"1min":0.253,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.58699999999999997,"5min":2.5750000000000002,"15min":7.6130000000000004},"last":0.53900000000000003}]},{"osd":2,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51800000000000002,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.42199999999999999,"15min":0.39900000000000002},"min":{"1min":0.27000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54100000000000004,"5min":7.0860000000000003,"15min":7.0860000000000003},"last":0.55500000000000005}]},{"osd":4,"last update":"Mon Mar 15 22:30:19 2021","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.52400000000000002,"15min":0.53300000000000003},"min":{"1min":0.26300000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.3370000000000002,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.77500000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.495,"15min":0.52500000000000002},"min":{"1min":0.28499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.2890000000000001},"last":0.58099999999999996}]},{"osd":5,"last update":"Mon Mar 15 22:30:25 2021","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":9.4870000000000001,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":0.71999999999999997},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":15.776,"5min":15.776,"15min":15.776},"last":0.59499999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:29:51 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.53600000000000003,"15min":0.57799999999999996},"min":{"1min":0.30499999999999999,"5min":0.30099999999999999,"15min":0.30099999999999999},"max":{"1min":2.5299999999999998,"5min":2.5299999999999998,"15min":2.6499999999999999},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.61099999999999999,"5min":0.55700000000000005,"15min":0.57099999999999995},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.85299999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:30:15 2021","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.625,"5min":5.625,"15min":5.625},"last":0.81100000000000005},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54100000000000004,"15min":0.56200000000000006},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.5409999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":0.61599999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986346,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":107280,"kb_used_data":74312,"kb_used_omap":0,"kb_used_meta":32960,"kb_avail":93637872,"statfs":{"total":95995035648,"available":95885180928,"internally_reserved":0,"allocated":76095488,"data_stored":98704605,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33751040},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.28499999999999998,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":0.68999999999999995,"5min":4.3769999999999998,"15min":4.3769999999999998},"last":0.502},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.54600000000000004,"15min":0.50600000000000001},"min":{"1min":0.30399999999999999,"5min":0.28999999999999998,"15min":0.251},"max":{"1min":0.72599999999999998,"5min":5.0209999999999999,"15min":5.0209999999999999},"last":0.52000000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.49199999999999999,"15min":0.48799999999999999},"min":{"1min":0.255,"5min":0.23999999999999999,"15min":0.23599999999999999},"max":{"1min":0.71699999999999997,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.371},{"interface":"front","average":{"1min":0.44,"5min":0.5,"15min":0.50700000000000001},"min":{"1min":0.33900000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71099999999999997,"5min":4.718,"15min":4.718},"last":0.54500000000000004}]},{"osd":2,"last update":"Mon Mar 15 22:26:18 2021","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48399999999999999,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.313},"max":{"1min":0.61299999999999999,"5min":0.70399999999999996,"15min":0.76100000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48299999999999998,"15min":0.502},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68799999999999994,"5min":0.71699999999999997,"15min":1.1910000000000001},"last":0.52800000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.58599999999999997,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.754,"5min":5.0499999999999998,"15min":5.0499999999999998},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.55300000000000005,"15min":0.54400000000000004},"min":{"1min":0.34000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69899999999999995,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.56299999999999994}]},{"osd":5,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:29:51 2021","interfaces":[{"interface":"back","average":{"1min":0.377,"5min":0.45500000000000002,"15min":0.44700000000000001},"min":{"1min":0.24199999999999999,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.88,"5min":4.3079999999999998,"15min":4.3079999999999998},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.52200000000000002,"15min":0.52300000000000002},"min":{"1min":0.29999999999999999,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.752,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.61899999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:30:08 2021","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.498,"15min":0.498},"min":{"1min":0.29399999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58899999999999997,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.32400000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.59099999999999997,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":0.44}]}]},{"osd":5,"up_from":616,"seq":2645699854354,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":168976,"kb_used_data":143560,"kb_used_omap":153,"kb_used_meta":25254,"kb_avail":93576176,"statfs":{"total":95995035648,"available":95822004224,"internally_reserved":0,"allocated":147005440,"data_stored":200547158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":25860625},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.372}]},{"osd":1,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.442},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.39400000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41699999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.35699999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.51500000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.68500000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.65700000000000003},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":12458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":6416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":201965568,"data_stored":288483311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":187195392,"data_stored":269176711,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":139649024,"data_stored":174087600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":272760832,"data_stored":407754089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":67674112,"data_stored":93350058,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":138747904,"data_stored":195221629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":103059456,"data_stored":161141235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":151011328,"data_stored":229464235,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:30:55.229 INFO:tasks.thrashosds.thrasher:Setting 6.f to [1, 5, 4, 0, 2, 6] 2021-03-15T22:30:55.230 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '6.f', '1', '5', '4', '0', '2', '6'] 2021-03-15T22:30:55.231 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pg-upmap-items 6.f 1 5 4 0 2 6 2021-03-15T22:30:55.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:55.233 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:55.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:55.235 INFO:tasks.rados.rados.0.smithi076.stdout:2587: done (0 left) 2021-03-15T22:30:55.235 INFO:tasks.rados.rados.0.smithi076.stdout:2588: delete oid 46 current snap is 299 2021-03-15T22:30:55.241 INFO:tasks.rados.rados.0.smithi076.stdout:2588: done (0 left) 2021-03-15T22:30:55.241 INFO:tasks.rados.rados.0.smithi076.stdout:2589: delete oid 19 current snap is 299 2021-03-15T22:30:55.246 INFO:tasks.rados.rados.0.smithi076.stdout:2589: done (0 left) 2021-03-15T22:30:55.246 INFO:tasks.rados.rados.0.smithi076.stdout:2590: snap_remove snap 298 2021-03-15T22:30:55.288 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:30:55.335 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:55.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:55.374 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:55.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:55.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:55.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:55.501 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:55.501 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:55.503 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cluster 2021-03-15T22:30:54 2021-03-15T22:30:55.503 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: .262762+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2021-03-15T22:30:55.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: audit 2021-03-15T22:30:54.360567+0000 mon.a (mon.0) 1893 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:30:55.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cluster 2021-03-15T22:30:54.373513+0000 osd.7 (osd.7) 133 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:55.505 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cluster 2021-03-15T22:30:54.374678+0000 osd.7 (osd.7) 134 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:55.505 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:30:54.634731+0000 mgr.y (mgr.14144) 862 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:30:55.505 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: audit 2021-03-15T22:30:54.638138+0000 mon.a (mon.0) 1894 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:30:55.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:30:54.638976+0000 mgr.y (mgr.14144) 863 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:30:55.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:30:55.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:30:55.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:30:55.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.508 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:30:55.509 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:30:54.639350+0000 mgr.y (mgr.14144) 864 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:30:55.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:30:55.510 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:30:55.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.511 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:30:55.512 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.512 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:30:55.513 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:30:54.639704+0000 mgr.y (mgr.14144) 865 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:30:55.514 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:30:55.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:30:55.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:30:55.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:30:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:30:54.640065+0000 mgr.y (mgr.14144) 866 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:30:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:30:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:30:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:30:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:30:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:30:54.640466+0000 mgr.y (mgr.14144) 867 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:30:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:30:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:30:55.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:30:55.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:30:55.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:30:54.640849+0000 mgr.y (mgr.14144) 868 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:30:55.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:30:55.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:30:55.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:30:55.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:30:55.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:30:54.641186+0000 mgr.y (mgr.14144) 869 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:30:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:30:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:30:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:30:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:30:55.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: 2021-03-15T22:30:55.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[56514]: cluster 2021-03-15T22:30:55.223702+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2021-03-15T22:30:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cluster 2021-03-15T22:30:54.262762+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2021-03-15T22:30:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: audit 2021-03-15T22:30:54.360567+0000 mon.a (mon.0) 1893 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:30:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cluster 2021-03-15T22:30:54.373513+0000 osd.7 (osd.7) 133 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cluster 2021-03-15T22:30:54.374678+0000 osd.7 (osd.7) 134 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:30:54.634731+0000 mgr.y (mgr.14144) 862 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:30:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: audit 2021-03-15T22:30:54.638138+0000 mon.a (mon.0) 1894 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:30:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:30:54.638976+0000 mgr.y (mgr.14144) 863 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:30:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:30:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:30:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:30:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:30:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:30:54.639350+0000 mgr.y (mgr.14144) 864 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:30:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:30:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:30:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:30:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:30:55.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:30:54.639704+0000 mgr.y (mgr.14144) 865 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:30:55.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:30:55.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:30:55.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:30:55.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:30:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:30:54.640065+0000 mgr.y (mgr.14144) 866 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:30:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:30:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:30:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:30:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:30:55.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:30:54.640466+0000 mgr.y (mgr.14144) 867 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:30:55.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:30:55.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:30:55.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:30:55.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:30:55.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:30:54.640849+0000 mgr.y (mgr.14144) 868 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:30:55.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:30:55.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:30:55.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:30:55.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:30:55.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:30:54.641186+0000 mgr.y (mgr.14144) 869 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:30:55.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:30:55.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:30:55.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:30:55.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.560 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.560 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:30:55.561 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:55 smithi076 conmon[62156]: cluster 2021-03-15T22:30:55.223702+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2021-03-15T22:30:55.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cluster 2021-03-15T22:30:54.262762+0000 mon.a (mon.0) 1892 : cluster [DBG] osdmap e705: 8 total, 8 up, 8 in 2021-03-15T22:30:55.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: audit 2021-03-15T22:30:55.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: 2021-03-15T22:30:54.360567+0000 mon.a (mon.0) 1893 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:30:55.562 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cluster 2021-03-15T22:30:54.373513+0000 osd.7 (osd.7) 133 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:55.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cluster 2021-03-15T22:30:54.374678+0000 osd.7 (osd.7) 134 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:55.563 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:30:54.634731+0000 mgr.y (mgr.14144) 862 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:30:55.564 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: audit 2021-03-15T22:30:54.638138+0000 mon.a (mon.0) 1894 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:30:55.564 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:30:54.638976+0000 mgr.y (mgr.14144) 863 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:30:55.565 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:30:55.565 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:30:55.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:30:55.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.568 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:30:55.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:30:54.639350+0000 mgr.y (mgr.14144) 864 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:30:55.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:30:55.569 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:30:55.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.570 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:30:55.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:30:55.571 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:30:54.639704+0000 mgr.y (mgr.14144) 865 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:30:55.572 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:30:55.572 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:30:55.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:30:55.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:30:55.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:30:54.640065+0000 mgr.y (mgr.14144) 866 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:30:55.575 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:30:55.575 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.575 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:30:55.576 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.576 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:30:55.576 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:30:55.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:30:54.640466+0000 mgr.y (mgr.14144) 867 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:30:55.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:30:55.578 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.578 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:30:55.578 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.579 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:30:55.579 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.580 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.580 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:30:55.580 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:30:54.640849+0000 mgr.y (mgr.14144) 868 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:30:55.580 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:30:55.581 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.581 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:30:55.583 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.583 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:30:55.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:30:55.584 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:30:54.641186+0000 mgr.y (mgr.14144) 869 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:30:55.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:30:55.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:30:55.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:30:55.585 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:30:55.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:30:55.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:30:55.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:30:55.586 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:30:55.587 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:55 smithi125 conmon[56739]: cluster 2021-03-15T22:30:55.223702+0000 mon.a (mon.0) 1895 : cluster [DBG] osdmap e706: 8 total, 8 up, 8 in 2021-03-15T22:30:55.603 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:55.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:55.628 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:55.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:55.719 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:30:55.730 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:55.731 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:55.736 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:30:55.754 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:55.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:55.756 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:55.758 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:30:55.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-15T22:30:55.777 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:30:55.781 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:55.799 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:30:55.828 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:30:55.846 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:30:55.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:55.863 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:30:55.865 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:30:55.865 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:30:55.865 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:30:55.865 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:30:55.866 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:30:55.866 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:30:55.866 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:30:55.866 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:30:55.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:55.882 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:55.883 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:55.984 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:56.006 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:56.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:56.008 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:56.077 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:30:56.109 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:56.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:56.134 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:56.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:56.234 INFO:tasks.rados.rados.0.smithi076.stdout:2590: done (0 left) 2021-03-15T22:30:56.235 INFO:tasks.rados.rados.0.smithi076.stdout:2591: snap_create 2021-03-15T22:30:56.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:56.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:56.259 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:56.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:56.365 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:56.386 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:56.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:56.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:56.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:56.511 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:56.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:56.513 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:56.615 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:56.637 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:56.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:56.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:56.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:30:56.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:56.765 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:56.765 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:30:56.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:56 smithi076 conmon[56514]: cluster 2021-03-15T22:30:55.152297+0000 mgr.y (mgr.14144) 870 : cluster [DBG] pgmap v1202: 138 pgs: 138 active+clean; 544 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 657 KiB/s wr, 5 op/s 2021-03-15T22:30:56.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:56 smithi076 conmon[56514]: audit 2021-03-15T22:30:55.700023+0000 mon.a (mon.0) 1896 : audit [DBG] from='client.? 172.21.15.76:0/4205867960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:56.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:56 smithi076 conmon[56514]: cluster 2021-03-15T22:30:56.231989+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2021-03-15T22:30:56.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:56 smithi076 conmon[62156]: cluster 2021-03-15T22:30:55.152297+0000 2021-03-15T22:30:56.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:56 smithi076 conmon[62156]: mgr.y (mgr.14144) 870 : cluster [DBG] pgmap v1202: 138 pgs: 138 active+clean; 544 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 657 KiB/s wr, 5 op/s 2021-03-15T22:30:56.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:56 smithi076 conmon[62156]: audit 2021-03-15T22:30:55.700023+0000 mon.a (mon.0) 1896 : audit [DBG] from='client.? 172.21.15.76:0/4205867960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:56.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:56 smithi076 conmon[62156]: cluster 2021-03-15T22:30:56.231989+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2021-03-15T22:30:56.866 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:56.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:56.890 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:56.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:56.991 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:57.015 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:57.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:57.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:56 smithi125 conmon[56739]: cluster 2021-03-15T22:30:55.152297+0000 mgr.y (mgr.14144) 870 : cluster [DBG] pgmap v1202: 138 pgs: 138 active+clean; 544 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 657 KiB/s wr, 5 op/s 2021-03-15T22:30:57.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:56 smithi125 conmon[56739]: audit 2021-03-15T22:30:55.700023+0000 mon.a (mon.0) 1896 : audit [DBG] from='client.? 172.21.15.76:0/4205867960' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:30:57.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:56 smithi125 conmon[56739]: cluster 2021-03-15T22:30:56.231989+0000 mon.a (mon.0) 1897 : cluster [DBG] osdmap e707: 8 total, 8 up, 8 in 2021-03-15T22:30:57.119 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:57.142 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:57.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:57.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:57.236 INFO:tasks.rados.rados.0.smithi076.stdout:2591: done (0 left) 2021-03-15T22:30:57.236 INFO:tasks.rados.rados.0.smithi076.stdout:2592: write (excl) oid 10 current snap is 300 2021-03-15T22:30:57.236 INFO:tasks.rados.rados.0.smithi076.stdout:2592: seq_num 893 ranges {608651=410658,1556420=674971,2773556=235095} 2021-03-15T22:30:57.241 INFO:tasks.rados.rados.0.smithi076.stdout:2592: writing smithi07694878-10 from 608651 to 1019309 tid 1 2021-03-15T22:30:57.245 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:57.248 INFO:tasks.rados.rados.0.smithi076.stdout:2592: writing smithi07694878-10 from 1556420 to 2231391 tid 2 2021-03-15T22:30:57.252 INFO:tasks.rados.rados.0.smithi076.stdout:2592: writing smithi07694878-10 from 2773556 to 3008651 tid 3 2021-03-15T22:30:57.253 INFO:tasks.rados.rados.0.smithi076.stdout:2593: delete oid 13 current snap is 300 2021-03-15T22:30:57.253 INFO:tasks.rados.rados.0.smithi076.stdout:2592: finishing write tid 1 to smithi07694878-10 2021-03-15T22:30:57.253 INFO:tasks.rados.rados.0.smithi076.stdout:2592: finishing write tid 2 to smithi07694878-10 2021-03-15T22:30:57.257 INFO:tasks.rados.rados.0.smithi076.stdout:2592: finishing write tid 3 to smithi07694878-10 2021-03-15T22:30:57.257 INFO:tasks.rados.rados.0.smithi076.stdout:2592: finishing write tid 4 to smithi07694878-10 2021-03-15T22:30:57.258 INFO:tasks.rados.rados.0.smithi076.stdout:2593: done (1 left) 2021-03-15T22:30:57.258 INFO:tasks.rados.rados.0.smithi076.stdout:2594: delete oid 12 current snap is 300 2021-03-15T22:30:57.259 INFO:tasks.rados.rados.0.smithi076.stdout:2592: finishing write tid 5 to smithi07694878-10 2021-03-15T22:30:57.259 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 1754 (ObjNum 893 snap 300 seq_num 893) dirty exists 2021-03-15T22:30:57.260 INFO:tasks.rados.rados.0.smithi076.stdout:2592: left oid 10 (ObjNum 893 snap 300 seq_num 893) 2021-03-15T22:30:57.260 INFO:tasks.rados.rados.0.smithi076.stdout:2592: done (1 left) 2021-03-15T22:30:57.261 INFO:tasks.rados.rados.0.smithi076.stdout:2594: done (0 left) 2021-03-15T22:30:57.261 INFO:tasks.rados.rados.0.smithi076.stdout:2595: snap_create 2021-03-15T22:30:57.271 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:57.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:57.275 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:57.377 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:57.400 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:57.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:57.401 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:57.504 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:57.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:57.528 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:57.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:57.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:57.654 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:57.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:57.656 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:57.672 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:30:57.673 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:30:57.758 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:57.783 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:57.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:57.784 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:57.885 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:57.907 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:57.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:57.909 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:58.010 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:58.034 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:58.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:58.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:58.137 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:58.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:58.160 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:58.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:58.242 INFO:tasks.rados.rados.0.smithi076.stdout:2595: done (0 left) 2021-03-15T22:30:58.243 INFO:tasks.rados.rados.0.smithi076.stdout:2596: snap_create 2021-03-15T22:30:58.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-15T22:30:58.262 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:58.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:58.286 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:58.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:58.388 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:30:58.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:58.411 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:58.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:30:58.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[56514]: cluster 2021-03-15T22:30:57.021673+0000 osd.0 2021-03-15T22:30:58.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[56514]: (osd.0) 110 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:58.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[56514]: cluster 2021-03-15T22:30:57.022895+0000 osd.0 (osd.0) 111 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:58.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[56514]: cluster 2021-03-15T22:30:57.152880+0000 mgr.y (mgr.14144) 871 : cluster [DBG] pgmap v1205: 138 pgs: 138 active+clean; 541 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 0 B/s wr, 5 op/s 2021-03-15T22:30:58.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[56514]: cluster 2021-03-15T22:30:57.233286+0000 mon.a (mon.0) 1898 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2021-03-15T22:30:58.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[62156]: cluster 2021-03-15T22:30:57.021673 2021-03-15T22:30:58.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[62156]: +0000 osd.0 (osd.0) 110 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:58.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:30:58.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[62156]: -03-15T22:30:57.022895+0000 osd.0 (osd.0) 111 : cluster 2021-03-15T22:30:58.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[62156]: [DBG] purged_snaps scrub ok 2021-03-15T22:30:58.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[62156]: cluster 2021-03-15T22:30 2021-03-15T22:30:58.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[62156]: :57.152880+0000 mgr.y (mgr.14144) 871 2021-03-15T22:30:58.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[62156]: : cluster [DBG] pgmap v1205: 138 pgs: 138 active+clean; 541 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 0 B/s wr, 5 op/s 2021-03-15T22:30:58.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:30:58.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[62156]: 22:30:57.233286+0000 mon.a (mon. 2021-03-15T22:30:58.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:58 smithi076 conmon[62156]: 0) 1898 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2021-03-15T22:30:58.513 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:58.531 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:30:58.536 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:58 smithi125 conmon[56739]: cluster 2021-03-15T22:30:57.021673+0000 osd.0 (osd.0) 110 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:30:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:58 smithi125 conmon[56739]: cluster 2021-03-15T22:30:57.022895+0000 osd.0 (osd.0) 111 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:30:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:58 smithi125 conmon[56739]: cluster 2021-03-15T22:30:57.152880+0000 mgr.y (mgr.14144) 871 : cluster [DBG] pgmap v1205: 138 pgs: 138 active+clean; 541 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 4.0 MiB/s rd, 0 B/s wr, 5 op/s 2021-03-15T22:30:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:58 smithi125 conmon[56739]: cluster 2021-03-15T22:30:57.233286+0000 mon.a (mon.0) 1898 : cluster [DBG] osdmap e708: 8 total, 8 up, 8 in 2021-03-15T22:30:58.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:58.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:58.640 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:30:58.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:58.665 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:58.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:30:58.767 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:58.789 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:58.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:58.790 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:58.892 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:58.897 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:30:58.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:58.915 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:58.916 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:59.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:59.040 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:59.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:59.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:59.142 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:59.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:59.165 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:59.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:59.186 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:30:59.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:59 smithi076 conmon[56514]: cluster 2021-03-15T22:30:58.240542+0000 mon.a (mon.0) 1899 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2021-03-15T22:30:59.247 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:30:59 smithi076 conmon[56514]: audit 2021-03-15T22:30:58.823785+0000 mon.a (mon.0) 1900 : audit [INF] from='client.? 172.21.15.76:0/1204798681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:30:59.252 INFO:tasks.rados.rados.0.smithi076.stdout:2596: done (0 left) 2021-03-15T22:30:59.253 INFO:tasks.rados.rados.0.smithi076.stdout:2597: write oid 27 current snap is 302 2021-03-15T22:30:59.253 INFO:tasks.rados.rados.0.smithi076.stdout:2597: seq_num 896 ranges {659072=636405,1731866=422298,2855825=203247} 2021-03-15T22:30:59.255 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:30:59.260 INFO:tasks.rados.rados.0.smithi076.stdout:2597: writing smithi07694878-27 from 659072 to 1295477 tid 1 2021-03-15T22:30:59.268 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:30:59.270 INFO:tasks.rados.rados.0.smithi076.stdout:2597: writing smithi07694878-27 from 1731866 to 2154164 tid 2 2021-03-15T22:30:59.270 INFO:tasks.rados.rados.0.smithi076.stdout:2597: writing smithi07694878-27 from 2855825 to 3059072 tid 3 2021-03-15T22:30:59.271 INFO:tasks.rados.rados.0.smithi076.stdout:2597: finishing write tid 1 to smithi07694878-27 2021-03-15T22:30:59.271 INFO:tasks.rados.rados.0.smithi076.stdout:2597: finishing write tid 2 to smithi07694878-27 2021-03-15T22:30:59.271 INFO:tasks.rados.rados.0.smithi076.stdout:2598: snap_remove snap 250 2021-03-15T22:30:59.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:59.295 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:59.296 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:30:59.397 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:59.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:59.421 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:59.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:59.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:59 smithi076 conmon[62156]: cluster 2021-03-15T22:30:58.240542+0000 mon.a 2021-03-15T22:30:59.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:59 smithi076 conmon[62156]: (mon.0) 1899 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2021-03-15T22:30:59.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:59 smithi076 conmon[62156]: audit 2021-03-15T22:30:58.823785+0000 2021-03-15T22:30:59.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:30:59 smithi076 conmon[62156]: mon.a (mon.0) 1900 : audit [INF] from='client.? 172.21.15.76:0/1204798681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:30:59.522 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:30:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:59 smithi125 conmon[56739]: cluster 2021-03-15T22:30:58.240542+0000 mon.a (mon 2021-03-15T22:30:59.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:59 smithi125 conmon[56739]: .0) 1899 : cluster [DBG] osdmap e709: 8 total, 8 up, 8 in 2021-03-15T22:30:59.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:30:59 smithi125 conmon[56739]: audit 2021-03-15T22:30:58.823785+0000 mon.a (mon.0) 1900 : audit [INF] from='client.? 172.21.15.76:0/1204798681' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:30:59.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:59.545 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:59.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:30:59.648 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:30:59.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:59.671 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:59.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:30:59.773 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:30:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:59.799 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:30:59.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:30:59.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:30:59.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:30:59.925 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:30:59.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:30:59.953 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:30:59.954 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:31:00.027 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:00.050 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:00.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:00.051 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:00.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:00.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:00.176 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:00.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:00.225 INFO:tasks.rados.rados.0.smithi076.stdout:2598: done (1 left) 2021-03-15T22:31:00.226 INFO:tasks.rados.rados.0.smithi076.stdout:2599: write (excl) oid 11 current snap is 302 2021-03-15T22:31:00.226 INFO:tasks.rados.rados.0.smithi076.stdout:2599: seq_num 897 ranges {675879=711654,2073995=682309} 2021-03-15T22:31:00.233 INFO:teuthology.orchestra.run.smithi076.stderr:set 6.f pg_upmap_items mapping to [1->5,4->0,2->6] 2021-03-15T22:31:00.234 INFO:tasks.rados.rados.0.smithi076.stdout:2599: writing smithi07694878-11 from 675879 to 1387533 tid 1 2021-03-15T22:31:00.244 INFO:tasks.rados.rados.0.smithi076.stdout:2599: writing smithi07694878-11 from 2073995 to 2756304 tid 2 2021-03-15T22:31:00.244 INFO:tasks.rados.rados.0.smithi076.stdout:2600: write oid 26 current snap is 302 2021-03-15T22:31:00.245 INFO:tasks.rados.rados.0.smithi076.stdout:2597: finishing write tid 3 to smithi07694878-27 2021-03-15T22:31:00.245 INFO:tasks.rados.rados.0.smithi076.stdout:2597: finishing write tid 4 to smithi07694878-27 2021-03-15T22:31:00.245 INFO:tasks.rados.rados.0.smithi076.stdout:2597: finishing write tid 5 to smithi07694878-27 2021-03-15T22:31:00.245 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1751 (ObjNum 896 snap 302 seq_num 896) dirty exists 2021-03-15T22:31:00.246 INFO:tasks.rados.rados.0.smithi076.stdout:2597: left oid 27 (ObjNum 896 snap 302 seq_num 896) 2021-03-15T22:31:00.246 INFO:tasks.rados.rados.0.smithi076.stdout:2600: seq_num 898 ranges {692686=503256,1848830=425967,2910671=182015} 2021-03-15T22:31:00.249 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:31:00 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:31:00.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:59.153319+0000 mgr.y (mgr.14144) 872 : cluster [DBG] pgmap v1208: 138 pgs: 138 active+clean; 541 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-03-15T22:31:00.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:59.237876+0000 mon.a (mon.0) 1901 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:00.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:00 smithi125 conmon[56739]: audit 2021-03-15T22:30:59.247989+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? 172.21.15.76:0/1204798681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:31:00.250 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:59.248039+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2021-03-15T22:31:00.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:00 smithi125 conmon[56739]: audit 2021-03-15T22:30:59.253762+0000 mon.a (mon.0) 1904 : audit [INF] from='client.? 172.21.15.76:0/789293104' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.f", "id": [1, 5, 4, 0, 2, 6]}]: dispatch 2021-03-15T22:31:00.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:00 smithi125 conmon[56739]: cluster 2021-03-15T22:30:59.480711+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:31:00.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:00 smithi125 conmon[56739]: audit 2021-03-15T22:31:00.031353+0000 mgr.y (mgr.14144) 873 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:00.251 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:00 smithi125 conmon[56739]: audit 2021-03-15T22:31:00.223391+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? 172.21.15.76:0/789293104' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.f", "id": [1, 5, 4, 0, 2, 6]}]': finished 2021-03-15T22:31:00.252 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:00 smithi125 conmon[56739]: cluster 2021-03-15T22:31:00.223492+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2021-03-15T22:31:00.252 INFO:tasks.rados.rados.0.smithi076.stdout:2600: writing smithi07694878-26 from 692686 to 1195942 tid 1 2021-03-15T22:31:00.258 INFO:tasks.rados.rados.0.smithi076.stdout:2600: writing smithi07694878-26 from 1848830 to 2274797 tid 2 2021-03-15T22:31:00.264 INFO:tasks.rados.rados.0.smithi076.stdout:2600: writing smithi07694878-26 from 2910671 to 3092686 tid 3 2021-03-15T22:31:00.264 INFO:tasks.rados.rados.0.smithi076.stdout:2597: done (2 left) 2021-03-15T22:31:00.264 INFO:tasks.rados.rados.0.smithi076.stdout:2601: write oid 31 current snap is 302 2021-03-15T22:31:00.264 INFO:tasks.rados.rados.0.smithi076.stdout:2601: seq_num 899 ranges {709493=578505,1907312=685978} 2021-03-15T22:31:00.271 INFO:tasks.rados.rados.0.smithi076.stdout:2601: writing smithi07694878-31 from 709493 to 1287998 tid 1 2021-03-15T22:31:00.278 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:00.280 INFO:tasks.rados.rados.0.smithi076.stdout:2601: writing smithi07694878-31 from 1907312 to 2593290 tid 2 2021-03-15T22:31:00.282 INFO:tasks.rados.rados.0.smithi076.stdout:2602: copy_from oid 19 from oid 43 current snap is 302 2021-03-15T22:31:00.282 INFO:tasks.rados.rados.0.smithi076.stdout:2600: finishing write tid 1 to smithi07694878-26 2021-03-15T22:31:00.282 INFO:tasks.rados.rados.0.smithi076.stdout:2600: finishing write tid 2 to smithi07694878-26 2021-03-15T22:31:00.283 INFO:tasks.rados.rados.0.smithi076.stdout:2600: finishing write tid 3 to smithi07694878-26 2021-03-15T22:31:00.283 INFO:tasks.rados.rados.0.smithi076.stdout:2600: finishing write tid 4 to smithi07694878-26 2021-03-15T22:31:00.283 INFO:tasks.rados.rados.0.smithi076.stdout:2600: finishing write tid 5 to smithi07694878-26 2021-03-15T22:31:00.283 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 1569 (ObjNum 898 snap 302 seq_num 898) dirty exists 2021-03-15T22:31:00.284 INFO:tasks.rados.rados.0.smithi076.stdout:2600: left oid 26 (ObjNum 898 snap 302 seq_num 898) 2021-03-15T22:31:00.284 INFO:tasks.rados.rados.0.smithi076.stdout:2599: finishing write tid 1 to smithi07694878-11 2021-03-15T22:31:00.284 INFO:tasks.rados.rados.0.smithi076.stdout:2600: done (3 left) 2021-03-15T22:31:00.284 INFO:tasks.rados.rados.0.smithi076.stdout:2603: snap_remove snap 296 2021-03-15T22:31:00.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:00.308 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:00.308 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:00.409 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:00.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:00.434 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:00.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:00.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:59. 2021-03-15T22:31:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[56514]: 153319+0000 mgr.y (mgr.14144) 872 : cluster [DBG] pgmap v1208: 138 pgs: 138 active+clean; 541 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-03-15T22:31:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:59.237876+0000 mon.a (mon.0) 1901 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[56514]: audit 2021-03-15T22:30:59.247989+0000 mon.a (mon.0) 1902 : audit [INF] from='client.? 172.21.15.76:0/1204798681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:31:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:59.248039+0000 mon.a (mon.0) 1903 : cluster [DBG] osdmap e710: 8 total, 8 up, 8 in 2021-03-15T22:31:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[56514]: audit 2021-03-15T22:30:59.253762+0000 mon.a (mon.0) 1904 : audit [INF] from='client.? 172.21.15.76:0/789293104' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.f", "id": [1, 5, 4, 0, 2, 6]}]: dispatch 2021-03-15T22:31:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[56514]: cluster 2021-03-15T22:30:59.480711+0000 osd.7 (osd.7) 135 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:31:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[56514]: audit 2021-03-15T22:31:00.031353+0000 mgr.y (mgr.14144) 873 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:00.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[56514]: audit 2021-03-15T22:31:00.223391+0000 mon.a (mon.0) 1905 : audit [INF] from='client.? 172.21.15.76:0/789293104' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.f", "id": [1, 5, 4, 0, 2, 6]}]': finished 2021-03-15T22:31:00.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[56514]: cluster 2021-03-15T22:31:00.223492+0000 mon.a (mon.0) 1906 : cluster [DBG] osdmap e711: 8 total, 8 up, 8 in 2021-03-15T22:31:00.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:00.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021 2021-03-15T22:31:00.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 03 2021-03-15T22:31:00.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 15 2021-03-15T22:31:00.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: T 2021-03-15T22:31:00.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 22 2021-03-15T22:31:00.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 30 2021-03-15T22:31:00.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 59 2021-03-15T22:31:00.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 153319 2021-03-15T22:31:00.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: +0000 2021-03-15T22:31:00.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mgr.y 2021-03-15T22:31:00.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ( 2021-03-15T22:31:00.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mgr 2021-03-15T22:31:00.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 14144 2021-03-15T22:31:00.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ) 2021-03-15T22:31:00.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 872 2021-03-15T22:31:00.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:00.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:00.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: pgmap v1208: 138 pgs: 138 active+clean; 541 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 0 B/s wr, 0 op/s 2021-03-15T22:31:00.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:00.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021 2021-03-15T22:31:00.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 03 2021-03-15T22:31:00.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 15 2021-03-15T22:31:00.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: T 2021-03-15T22:31:00.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 22 2021-03-15T22:31:00.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 30 2021-03-15T22:31:00.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 59 2021-03-15T22:31:00.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 237876 2021-03-15T22:31:00.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: +0000 2021-03-15T22:31:00.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:00.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ( 2021-03-15T22:31:00.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon 2021-03-15T22:31:00.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 0 2021-03-15T22:31:00.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ) 2021-03-15T22:31:00.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 1901 2021-03-15T22:31:00.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:00.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: [WRN] 2021-03-15T22:31:00.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:00.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: audit 2021-03-15T22:31:00.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021 2021-03-15T22:31:00.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 03 2021-03-15T22:31:00.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 15 2021-03-15T22:31:00.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: T 2021-03-15T22:31:00.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 22 2021-03-15T22:31:00.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 30 2021-03-15T22:31:00.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 59 2021-03-15T22:31:00.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 247989 2021-03-15T22:31:00.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: +0000 2021-03-15T22:31:00.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:00.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ( 2021-03-15T22:31:00.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon 2021-03-15T22:31:00.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 0 2021-03-15T22:31:00.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ) 2021-03-15T22:31:00.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 1902 2021-03-15T22:31:00.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: audit 2021-03-15T22:31:00.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: [INF] 2021-03-15T22:31:00.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/1204798681' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:31:00.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:00.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021 2021-03-15T22:31:00.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 03 2021-03-15T22:31:00.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 15 2021-03-15T22:31:00.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: T 2021-03-15T22:31:00.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 22 2021-03-15T22:31:00.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 30 2021-03-15T22:31:00.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 59 2021-03-15T22:31:00.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 248039 2021-03-15T22:31:00.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: +0000 2021-03-15T22:31:00.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:00.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ( 2021-03-15T22:31:00.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon 2021-03-15T22:31:00.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 0 2021-03-15T22:31:00.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ) 2021-03-15T22:31:00.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 1903 2021-03-15T22:31:00.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:00.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:00.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: osdmap e710: 8 total, 8 up, 8 in 2021-03-15T22:31:00.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: audit 2021-03-15T22:31:00.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021 2021-03-15T22:31:00.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 03 2021-03-15T22:31:00.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 15 2021-03-15T22:31:00.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: T 2021-03-15T22:31:00.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 22 2021-03-15T22:31:00.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 30 2021-03-15T22:31:00.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 59 2021-03-15T22:31:00.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 253762 2021-03-15T22:31:00.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: +0000 2021-03-15T22:31:00.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:00.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ( 2021-03-15T22:31:00.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon 2021-03-15T22:31:00.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 0 2021-03-15T22:31:00.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ) 2021-03-15T22:31:00.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 1904 2021-03-15T22:31:00.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: audit 2021-03-15T22:31:00.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.560 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: [INF] 2021-03-15T22:31:00.560 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.560 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/789293104' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "6.f", "id": [1, 5, 4, 0, 2, 6]}]: dispatch 2021-03-15T22:31:00.560 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.561 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:00.561 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.561 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021 2021-03-15T22:31:00.561 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.562 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 03 2021-03-15T22:31:00.562 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.562 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 15 2021-03-15T22:31:00.562 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: T 2021-03-15T22:31:00.563 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 22 2021-03-15T22:31:00.563 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.563 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 30 2021-03-15T22:31:00.563 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 59 2021-03-15T22:31:00.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 480711 2021-03-15T22:31:00.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: +0000 2021-03-15T22:31:00.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: osd.7 2021-03-15T22:31:00.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ( 2021-03-15T22:31:00.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: osd 2021-03-15T22:31:00.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 7 2021-03-15T22:31:00.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ) 2021-03-15T22:31:00.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 135 2021-03-15T22:31:00.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:00.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:00.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 6.12 scrub ok 2021-03-15T22:31:00.568 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.568 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: audit 2021-03-15T22:31:00.568 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.568 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021 2021-03-15T22:31:00.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 03 2021-03-15T22:31:00.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 15 2021-03-15T22:31:00.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: T 2021-03-15T22:31:00.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 22 2021-03-15T22:31:00.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 31 2021-03-15T22:31:00.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 00 2021-03-15T22:31:00.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 031353 2021-03-15T22:31:00.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: +0000 2021-03-15T22:31:00.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mgr.y 2021-03-15T22:31:00.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ( 2021-03-15T22:31:00.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mgr 2021-03-15T22:31:00.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.572 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 14144 2021-03-15T22:31:00.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ) 2021-03-15T22:31:00.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 873 2021-03-15T22:31:00.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: audit 2021-03-15T22:31:00.573 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:00.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:00.574 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.575 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: audit 2021-03-15T22:31:00.575 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.575 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021 2021-03-15T22:31:00.575 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.575 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 03 2021-03-15T22:31:00.576 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.576 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 15 2021-03-15T22:31:00.576 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: T 2021-03-15T22:31:00.576 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 22 2021-03-15T22:31:00.577 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.577 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 31 2021-03-15T22:31:00.577 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.577 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 00 2021-03-15T22:31:00.577 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.578 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 223391 2021-03-15T22:31:00.578 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: +0000 2021-03-15T22:31:00.578 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.578 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:00.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ( 2021-03-15T22:31:00.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon 2021-03-15T22:31:00.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 0 2021-03-15T22:31:00.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ) 2021-03-15T22:31:00.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 1905 2021-03-15T22:31:00.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: audit 2021-03-15T22:31:00.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: [INF] 2021-03-15T22:31:00.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/789293104' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "6.f", "id": [1, 5, 4, 0, 2, 6]}]': finished 2021-03-15T22:31:00.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.582 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:00.582 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.582 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021 2021-03-15T22:31:00.582 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 03 2021-03-15T22:31:00.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: - 2021-03-15T22:31:00.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 15 2021-03-15T22:31:00.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: T 2021-03-15T22:31:00.583 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 22 2021-03-15T22:31:00.584 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.584 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 31 2021-03-15T22:31:00.584 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.584 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 00 2021-03-15T22:31:00.585 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.585 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 223492 2021-03-15T22:31:00.585 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: +0000 2021-03-15T22:31:00.585 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.586 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:00.586 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ( 2021-03-15T22:31:00.586 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: mon 2021-03-15T22:31:00.586 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: . 2021-03-15T22:31:00.587 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 0 2021-03-15T22:31:00.587 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: ) 2021-03-15T22:31:00.587 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 1906 2021-03-15T22:31:00.587 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: : 2021-03-15T22:31:00.588 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:00.588 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.588 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:00.588 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.588 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: osdmap e711: 8 total, 8 up, 8 in 2021-03-15T22:31:00.589 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:00 smithi076 conmon[62156]: 2021-03-15T22:31:00.589 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:00.591 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-15T22:31:00.613 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:00.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:00.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:00.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:00.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:00.740 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:00.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:00.842 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:00.865 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:00.866 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:31:00.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:00.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:00.969 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:00.991 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:00.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:00.992 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:01.093 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:01.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:01.118 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:01.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:01.219 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:01.239 INFO:tasks.rados.rados.0.smithi076.stdout:2603: done (3 left) 2021-03-15T22:31:01.240 INFO:tasks.rados.rados.0.smithi076.stdout:2604: snap_create 2021-03-15T22:31:01.240 INFO:tasks.rados.rados.0.smithi076.stdout:2601: finishing write tid 1 to smithi07694878-31 2021-03-15T22:31:01.240 INFO:tasks.rados.rados.0.smithi076.stdout:2601: finishing write tid 2 to smithi07694878-31 2021-03-15T22:31:01.240 INFO:tasks.rados.rados.0.smithi076.stdout:2601: finishing write tid 3 to smithi07694878-31 2021-03-15T22:31:01.241 INFO:tasks.rados.rados.0.smithi076.stdout:2599: finishing write tid 2 to smithi07694878-11 2021-03-15T22:31:01.241 INFO:tasks.rados.rados.0.smithi076.stdout:2599: finishing write tid 3 to smithi07694878-11 2021-03-15T22:31:01.241 INFO:tasks.rados.rados.0.smithi076.stdout:2601: finishing write tid 4 to smithi07694878-31 2021-03-15T22:31:01.241 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 1852 (ObjNum 899 snap 302 seq_num 899) dirty exists 2021-03-15T22:31:01.241 INFO:tasks.rados.rados.0.smithi076.stdout:2601: left oid 31 (ObjNum 899 snap 302 seq_num 899) 2021-03-15T22:31:01.242 INFO:tasks.rados.rados.0.smithi076.stdout:2599: finishing write tid 4 to smithi07694878-11 2021-03-15T22:31:01.242 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1569 (ObjNum 897 snap 302 seq_num 897) dirty exists 2021-03-15T22:31:01.242 INFO:tasks.rados.rados.0.smithi076.stdout:2599: left oid 11 (ObjNum 897 snap 302 seq_num 897) 2021-03-15T22:31:01.242 INFO:tasks.rados.rados.0.smithi076.stdout:2602: finishing copy_from to smithi07694878-19 2021-03-15T22:31:01.243 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 1220 (ObjNum 613 snap 190 seq_num 613) dirty exists 2021-03-15T22:31:01.243 INFO:tasks.rados.rados.0.smithi076.stdout:2602: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:31:01.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:01.247 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:01.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:01.314 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:01 smithi125 conmon[56739]: audit 2021-03-15T22:31:01.108230+0000 mon.a (mon. 2021-03-15T22:31:01.314 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:01 smithi125 conmon[56739]: 0) 1907 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:31:01.314 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:01 smithi125 conmon[56739]: audit 2021-03-15T22:31:01.108405+0000 mon.a (mon.0) 1908 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:31:01.315 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:01 smithi125 conmon[56739]: cluster 2021-03-15T22:31:01.233244+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2021-03-15T22:31:01.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:01.373 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:01.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:01.375 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:01.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:01 smithi076 conmon[56514]: audit 2021-03-15T22:31:01.108230+0000 mon.a (mon.0) 1907 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:31:01.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:01 smithi076 conmon[56514]: audit 2021-03-15T22:31:01.108405+0000 mon.a (mon.0) 1908 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:31:01.401 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:01 smithi076 conmon[56514]: cluster 2021-03-15T22:31:01.233244+0000 mon.a (mon.0) 1909 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2021-03-15T22:31:01.401 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:01 smithi076 conmon[62156]: audit 2021-03-15T22:31:01.108230+0000 2021-03-15T22:31:01.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:01 smithi076 conmon[62156]: mon.a (mon.0) 1907 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:31:01.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:01 smithi076 conmon[62156]: audit 2021-03-15T22:31: 2021-03-15T22:31:01.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:01 smithi076 conmon[62156]: 01.108405+0000 mon.a (mon.0) 1908 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:31:01.402 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:01 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:31:01.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:01 smithi076 conmon[62156]: 15T22:31:01.233244+0000 mon.a ( 2021-03-15T22:31:01.403 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:01 smithi076 conmon[62156]: mon.0) 1909 : cluster [DBG] osdmap e712: 8 total, 8 up, 8 in 2021-03-15T22:31:01.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:01.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:01.500 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:01.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:01.602 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:01.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:01.625 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:01.625 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:01.727 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:01.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:01.751 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:01.752 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:01.806 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:31:01.822 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:31:01.839 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:31:01.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:01.855 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:31:01.873 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:31:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:01.878 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:01.879 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:01.891 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:31:01.907 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:31:01.925 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:31:01.926 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:31:01.927 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:31:01.927 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:31:01.927 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:31:01.929 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:31:01.929 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:31:01.930 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:31:01.930 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:31:01.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:02.005 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:02.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:02.007 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:02.108 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:02.130 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:02.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:02.132 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:02.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:02.236 INFO:tasks.rados.rados.0.smithi076.stdout:2599: done (3 left) 2021-03-15T22:31:02.237 INFO:tasks.rados.rados.0.smithi076.stdout:2601: done (2 left) 2021-03-15T22:31:02.237 INFO:tasks.rados.rados.0.smithi076.stdout:2602: done (1 left) 2021-03-15T22:31:02.237 INFO:tasks.rados.rados.0.smithi076.stdout:2604: done (0 left) 2021-03-15T22:31:02.237 INFO:tasks.rados.rados.0.smithi076.stdout:2605: snap_remove snap 290 2021-03-15T22:31:02.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:02.258 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:02.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:02.324 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:02 smithi125 conmon[56739]: cluster 2021-03-15T22:31:01.010390+0000 osd.0 (osd.0 2021-03-15T22:31:02.325 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:02 smithi125 conmon[56739]: ) 112 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:31:02.325 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:02 smithi125 conmon[56739]: cluster 2021-03-15T22:31:01.153785+0000 mgr.y (mgr.14144) 874 : cluster [DBG] pgmap v1211: 138 pgs: 1 peering, 137 active+clean; 549 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 972 KiB/s wr, 6 op/s 2021-03-15T22:31:02.325 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:02 smithi125 conmon[56739]: cluster 2021-03-15T22:31:01.245495+0000 osd.6 (osd.6) 110 : cluster [DBG] 4.a scrub ok 2021-03-15T22:31:02.325 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:02 smithi125 conmon[56739]: audit 2021-03-15T22:31:01.786757+0000 mon.a (mon.0) 1910 : audit [DBG] from='client.? 172.21.15.76:0/1580956083' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:02.326 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:02 smithi125 conmon[56739]: cluster 2021-03-15T22:31:02.232082+0000 2021-03-15T22:31:02.326 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:02 smithi125 conmon[56739]: mon.a (mon.0) 1911 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2021-03-15T22:31:02.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:31:02.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:02.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:02.416 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:02.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:02.511 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:31:02.511 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:31:02.512 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:31:02.512 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:31:02.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[56514]: cluster 2021-03-15T22:31:01.010390+0000 2021-03-15T22:31:02.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[56514]: osd.0 (osd.0) 112 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:31:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[56514]: cluster 2021-03-15T22:31:01.153785+0000 mgr.y (mgr.14144) 874 : cluster [DBG] pgmap v1211: 138 pgs: 1 peering, 137 active+clean; 549 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 972 KiB/s wr, 6 op/s 2021-03-15T22:31:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[56514]: cluster 2021-03-15T22:31:01.245495+0000 osd.6 (osd.6) 110 : cluster [DBG] 4.a scrub ok 2021-03-15T22:31:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[56514]: audit 2021-03-15T22:31:01.786757+0000 mon.a (mon.0) 1910 : audit [DBG] from='client.? 172.21.15.76:0/1580956083' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:02.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[56514]: cluster 2021-03-15T22:31:02.232082+0000 mon.a (mon.0) 1911 : cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2021-03-15T22:31:02.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: cluster 2021-03-15T22:31:01.010390+0000 2021-03-15T22:31:02.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: osd.0 (osd.0) 112 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:31:02.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:31:02.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: -15T22:31:01.153785+0000 mgr.y (mgr.14144) 874 : cluster [DBG] 2021-03-15T22:31:02.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: pgmap v1211: 138 pgs: 1 peering, 137 active+clean; 549 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.0 KiB/s rd, 972 KiB/s wr, 6 op/s 2021-03-15T22:31:02.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: cluster 2021-03-15T22:31:01 2021-03-15T22:31:02.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: .245495+0000 osd.6 (osd.6) 110 : 2021-03-15T22:31:02.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: cluster [DBG] 4.a scrub ok 2021-03-15T22:31:02.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:31:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: 03-15T22:31:01.786757+0000 2021-03-15T22:31:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: mon.a (mon.0) 1910 : audit [DBG] 2021-03-15T22:31:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/1580956083' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:02.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: cluster 2021-03-15T22:31:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: 2021-03-15T22:31:02 2021-03-15T22:31:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: .232082+0000 mon.a (mon.0) 1911 : 2021-03-15T22:31:02.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:02 smithi076 conmon[62156]: cluster [DBG] osdmap e713: 8 total, 8 up, 8 in 2021-03-15T22:31:02.523 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:02.546 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:02.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:02.548 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:02.649 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:02.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:02.675 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:02.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:02.740 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:02.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:02.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:02.804 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:02.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:02.905 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:02.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:02.929 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:02.929 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:03.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:03.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:03.054 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:03.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:03.127 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-15T22:31:03.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:03.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:03.180 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:03.181 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:03.239 INFO:tasks.rados.rados.0.smithi076.stdout:2605: done (0 left) 2021-03-15T22:31:03.240 INFO:tasks.rados.rados.0.smithi076.stdout:2606: write (excl) oid 39 current snap is 303 2021-03-15T22:31:03.240 INFO:tasks.rados.rados.0.smithi076.stdout:2606: seq_num 900 ranges {726300=653754,1965794=429636,2965517=160783} 2021-03-15T22:31:03.248 INFO:tasks.rados.rados.0.smithi076.stdout:2606: writing smithi07694878-39 from 726300 to 1380054 tid 1 2021-03-15T22:31:03.253 INFO:tasks.rados.rados.0.smithi076.stdout:2606: writing smithi07694878-39 from 1965794 to 2395430 tid 2 2021-03-15T22:31:03.259 INFO:tasks.rados.rados.0.smithi076.stdout:2606: writing smithi07694878-39 from 2965517 to 3126300 tid 3 2021-03-15T22:31:03.259 INFO:tasks.rados.rados.0.smithi076.stdout:2607: copy_from oid 26 from oid 1 current snap is 303 2021-03-15T22:31:03.259 INFO:tasks.rados.rados.0.smithi076.stdout:2606: finishing write tid 1 to smithi07694878-39 2021-03-15T22:31:03.260 INFO:tasks.rados.rados.0.smithi076.stdout:2608: delete oid 35 current snap is 303 2021-03-15T22:31:03.260 INFO:tasks.rados.rados.0.smithi076.stdout:2606: finishing write tid 2 to smithi07694878-39 2021-03-15T22:31:03.264 INFO:tasks.rados.rados.0.smithi076.stdout:2606: finishing write tid 3 to smithi07694878-39 2021-03-15T22:31:03.264 INFO:tasks.rados.rados.0.smithi076.stdout:2606: finishing write tid 4 to smithi07694878-39 2021-03-15T22:31:03.264 INFO:tasks.rados.rados.0.smithi076.stdout:2608: done (2 left) 2021-03-15T22:31:03.265 INFO:tasks.rados.rados.0.smithi076.stdout:2609: copy_from oid 7 from oid 40 current snap is 303 2021-03-15T22:31:03.265 INFO:tasks.rados.rados.0.smithi076.stdout:2610: snap_create 2021-03-15T22:31:03.265 INFO:tasks.rados.rados.0.smithi076.stdout:2606: finishing write tid 5 to smithi07694878-39 2021-03-15T22:31:03.265 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 1752 (ObjNum 900 snap 303 seq_num 900) dirty exists 2021-03-15T22:31:03.266 INFO:tasks.rados.rados.0.smithi076.stdout:2606: left oid 39 (ObjNum 900 snap 303 seq_num 900) 2021-03-15T22:31:03.282 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:03.286 INFO:tasks.rados.rados.0.smithi076.stdout:2607: finishing copy_from to smithi07694878-26 2021-03-15T22:31:03.287 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 1324 (ObjNum 803 snap 268 seq_num 803) dirty exists 2021-03-15T22:31:03.287 INFO:tasks.rados.rados.0.smithi076.stdout:2607: finishing copy_from racing read to smithi07694878-26 2021-03-15T22:31:03.287 INFO:tasks.rados.rados.0.smithi076.stdout:2609: finishing copy_from to smithi07694878-7 2021-03-15T22:31:03.288 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2061 (ObjNum 681 snap 218 seq_num 681) dirty exists 2021-03-15T22:31:03.288 INFO:tasks.rados.rados.0.smithi076.stdout:2609: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:31:03.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:03.307 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:03.307 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:03.409 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:03.414 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:31:03.432 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:03.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:03.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:03.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: cluster 2021-03-15T22:31:03.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:31:03.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: 31:02.050387+0000 osd.0 (osd.0) 113 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:31:03.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: cluster 2021-03-15T22:31:03.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: 2021-03-15T22:31:02 2021-03-15T22:31:03.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: .092211+0000 osd.2 (osd.2) 8 : cluster 2021-03-15T22:31:03.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: [DBG] 4.f scrub ok 2021-03-15T22:31:03.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:31:03.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: 15T22:31:02.160810+0000 osd.1 ( 2021-03-15T22:31:03.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: osd.1) 137 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:31:03.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:31:03.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: 15T22:31:02.243391+0000 osd.6 (osd 2021-03-15T22:31:03.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: .6) 111 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:31:03.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:31:03.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: -15T22:31:02.426444+0000 osd.4 (osd.4) 152 : 2021-03-15T22:31:03.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: cluster [DBG] 7.1 scrub ok 2021-03-15T22:31:03.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: cluster 2021-03-15T22:31:03.233919+0000 mon.a (mon 2021-03-15T22:31:03.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[56514]: .0) 1912 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2021-03-15T22:31:03.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: cluster 2021-03-15T22:31:02 2021-03-15T22:31:03.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: .050387+0000 osd.0 (osd.0) 113 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:31:03.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: cluster 2021-03-15T22:31:03.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: 2021-03-15T22:31:02.092211+0000 osd.2 (osd.2) 8 2021-03-15T22:31:03.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: : cluster [DBG] 4.f scrub ok 2021-03-15T22:31:03.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: cluster 2021-03-15T22:31:03.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: 2021-03-15T22:31 2021-03-15T22:31:03.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: :02.160810+0000 osd.1 (osd.1 2021-03-15T22:31:03.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: ) 137 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:31:03.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: cluster 2021-03-15T22:31:03.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: 2021-03-15T22:31:02 2021-03-15T22:31:03.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: .243391+0000 osd.6 (osd.6) 111 : 2021-03-15T22:31:03.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: cluster [DBG] 7.7 scrub ok 2021-03-15T22:31:03.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:31:03.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: -15T22:31:02.426444+0000 osd.4 2021-03-15T22:31:03.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: (osd.4) 152 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:31:03.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: 2021-03-15T22:31:03.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: cluster 2021-03-15T22:31: 2021-03-15T22:31:03.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: 03.233919+0000 mon.a (mon.0) 1912 2021-03-15T22:31:03.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:03 smithi076 conmon[62156]: : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2021-03-15T22:31:03.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:03.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:03 smithi125 conmon[56739]: cluster 2021-03-15T22:31:02.050387+0000 osd.0 (osd.0) 113 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:31:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:03 smithi125 conmon[56739]: cluster 2021-03-15T22:31:02.092211+0000 osd.2 (osd.2) 8 : cluster [DBG] 4.f scrub ok 2021-03-15T22:31:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:03 smithi125 conmon[56739]: cluster 2021-03-15T22:31:02.160810+0000 osd.1 (osd.1) 137 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:31:03.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:03 smithi125 conmon[56739]: cluster 2021-03-15T22:31:02.243391+0000 osd.6 (osd.6) 111 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:31:03.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:03 smithi125 conmon[56739]: cluster 2021-03-15T22:31:02.426444+0000 osd.4 (osd.4) 152 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:31:03.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:03 smithi125 conmon[56739]: cluster 2021-03-15T22:31:03.233919+0000 mon.a (mon.0) 1912 : cluster [DBG] osdmap e714: 8 total, 8 up, 8 in 2021-03-15T22:31:03.556 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:03.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:03.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:03.659 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:03.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:03.684 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:03.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:03.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:03.808 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:03.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:03.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:03.910 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:03.932 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:03.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:03.934 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:04.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:04.058 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:04.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:04.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:04.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:04.184 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:04.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:04.185 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:04.279 INFO:tasks.rados.rados.0.smithi076.stdout:2606: done (3 left) 2021-03-15T22:31:04.279 INFO:tasks.rados.rados.0.smithi076.stdout:2607: done (2 left) 2021-03-15T22:31:04.279 INFO:tasks.rados.rados.0.smithi076.stdout:2609: done (1 left) 2021-03-15T22:31:04.279 INFO:tasks.rados.rados.0.smithi076.stdout:2610: done (0 left) 2021-03-15T22:31:04.280 INFO:tasks.rados.rados.0.smithi076.stdout:2611: delete oid 10 current snap is 304 2021-03-15T22:31:04.283 INFO:tasks.rados.rados.0.smithi076.stdout:2611: done (0 left) 2021-03-15T22:31:04.284 INFO:tasks.rados.rados.0.smithi076.stdout:2612: write oid 39 current snap is 304 2021-03-15T22:31:04.284 INFO:tasks.rados.rados.0.smithi076.stdout:2612: seq_num 903 ranges {776721=479501,2024887=693316} 2021-03-15T22:31:04.286 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:04.291 INFO:tasks.rados.rados.0.smithi076.stdout:2612: writing smithi07694878-39 from 776721 to 1256222 tid 1 2021-03-15T22:31:04.301 INFO:tasks.rados.rados.0.smithi076.stdout:2612: writing smithi07694878-39 from 2024887 to 2718203 tid 2 2021-03-15T22:31:04.301 INFO:tasks.rados.rados.0.smithi076.stdout:2612: finishing write tid 1 to smithi07694878-39 2021-03-15T22:31:04.302 INFO:tasks.rados.rados.0.smithi076.stdout:2613: rollback oid 28 current snap is 304 2021-03-15T22:31:04.302 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 268 2021-03-15T22:31:04.302 INFO:tasks.rados.rados.0.smithi076.stdout:2614: write oid 26 current snap is 304 2021-03-15T22:31:04.302 INFO:tasks.rados.rados.0.smithi076.stdout:2614: seq_num 904 ranges {793528=554750,2083369=436974,2958856=234672} 2021-03-15T22:31:04.310 INFO:tasks.rados.rados.0.smithi076.stdout:2614: writing smithi07694878-26 from 793528 to 1348278 tid 1 2021-03-15T22:31:04.315 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:04.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:04.317 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:04.317 INFO:tasks.rados.rados.0.smithi076.stdout:2614: writing smithi07694878-26 from 2083369 to 2520343 tid 2 2021-03-15T22:31:04.322 INFO:tasks.rados.rados.0.smithi076.stdout:2614: writing smithi07694878-26 from 2958856 to 3193528 tid 3 2021-03-15T22:31:04.322 INFO:tasks.rados.rados.0.smithi076.stdout:2615: write oid 24 current snap is 304 2021-03-15T22:31:04.322 INFO:tasks.rados.rados.0.smithi076.stdout:2612: finishing write tid 2 to smithi07694878-39 2021-03-15T22:31:04.322 INFO:tasks.rados.rados.0.smithi076.stdout:2612: finishing write tid 3 to smithi07694878-39 2021-03-15T22:31:04.323 INFO:tasks.rados.rados.0.smithi076.stdout:2615: seq_num 905 ranges {410335=746352,1741851=696985,3102632=107703} 2021-03-15T22:31:04.329 INFO:tasks.rados.rados.0.smithi076.stdout:2615: writing smithi07694878-24 from 410335 to 1156687 tid 1 2021-03-15T22:31:04.338 INFO:tasks.rados.rados.0.smithi076.stdout:2615: writing smithi07694878-24 from 1741851 to 2438836 tid 2 2021-03-15T22:31:04.342 INFO:tasks.rados.rados.0.smithi076.stdout:2615: writing smithi07694878-24 from 3102632 to 3210335 tid 3 2021-03-15T22:31:04.343 INFO:tasks.rados.rados.0.smithi076.stdout:2616: write (excl) oid 3 current snap is 304 2021-03-15T22:31:04.343 INFO:tasks.rados.rados.0.smithi076.stdout:2616: seq_num 906 ranges {427142=421601,1400333=440643,2613702=450326} 2021-03-15T22:31:04.347 INFO:tasks.rados.rados.0.smithi076.stdout:2616: writing smithi07694878-3 from 427142 to 848743 tid 1 2021-03-15T22:31:04.354 INFO:tasks.rados.rados.0.smithi076.stdout:2616: writing smithi07694878-3 from 1400333 to 1840976 tid 2 2021-03-15T22:31:04.362 INFO:tasks.rados.rados.0.smithi076.stdout:2616: writing smithi07694878-3 from 2613702 to 3064028 tid 3 2021-03-15T22:31:04.363 INFO:tasks.rados.rados.0.smithi076.stdout:2617: read oid 33 snap 242 2021-03-15T22:31:04.363 INFO:tasks.rados.rados.0.smithi076.stdout:2617: expect (ObjNum 736 snap 242 seq_num 736) 2021-03-15T22:31:04.363 INFO:tasks.rados.rados.0.smithi076.stdout:2618: rollback oid 14 current snap is 304 2021-03-15T22:31:04.363 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 14 to 196 2021-03-15T22:31:04.363 INFO:tasks.rados.rados.0.smithi076.stdout:2612: finishing write tid 4 to smithi07694878-39 2021-03-15T22:31:04.364 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2064 (ObjNum 903 snap 304 seq_num 903) dirty exists 2021-03-15T22:31:04.364 INFO:tasks.rados.rados.0.smithi076.stdout:2612: left oid 39 (ObjNum 903 snap 304 seq_num 903) 2021-03-15T22:31:04.364 INFO:tasks.rados.rados.0.smithi076.stdout:2613: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:31:04.364 INFO:tasks.rados.rados.0.smithi076.stdout:2613: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:31:04.365 INFO:tasks.rados.rados.0.smithi076.stdout:2613: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:31:04.365 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2137 (ObjNum 609 snap 189 seq_num 609) dirty exists 2021-03-15T22:31:04.365 INFO:tasks.rados.rados.0.smithi076.stdout:2614: finishing write tid 1 to smithi07694878-26 2021-03-15T22:31:04.365 INFO:tasks.rados.rados.0.smithi076.stdout:2614: finishing write tid 2 to smithi07694878-26 2021-03-15T22:31:04.365 INFO:tasks.rados.rados.0.smithi076.stdout:2612: done (6 left) 2021-03-15T22:31:04.366 INFO:tasks.rados.rados.0.smithi076.stdout:2613: done (5 left) 2021-03-15T22:31:04.366 INFO:tasks.rados.rados.0.smithi076.stdout:2619: write (excl) oid 39 current snap is 304 2021-03-15T22:31:04.366 INFO:tasks.rados.rados.0.smithi076.stdout:2614: finishing write tid 3 to smithi07694878-26 2021-03-15T22:31:04.366 INFO:tasks.rados.rados.0.smithi076.stdout:2619: seq_num 907 ranges {443949=496850,1458815=584301,2641125=602824} 2021-03-15T22:31:04.367 INFO:tasks.rados.rados.0.smithi076.stdout:2619: writing smithi07694878-39 from 443949 to 940799 tid 1 2021-03-15T22:31:04.374 INFO:tasks.rados.rados.0.smithi076.stdout:2619: writing smithi07694878-39 from 1458815 to 2043116 tid 2 2021-03-15T22:31:04.383 INFO:tasks.rados.rados.0.smithi076.stdout:2619: writing smithi07694878-39 from 2641125 to 3243949 tid 3 2021-03-15T22:31:04.384 INFO:tasks.rados.rados.0.smithi076.stdout:2620: write oid 31 current snap is 304 2021-03-15T22:31:04.385 INFO:tasks.rados.rados.0.smithi076.stdout:2614: finishing write tid 4 to smithi07694878-26 2021-03-15T22:31:04.385 INFO:tasks.rados.rados.0.smithi076.stdout:2614: finishing write tid 5 to smithi07694878-26 2021-03-15T22:31:04.385 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 1573 (ObjNum 904 snap 304 seq_num 904) dirty exists 2021-03-15T22:31:04.385 INFO:tasks.rados.rados.0.smithi076.stdout:2614: left oid 26 (ObjNum 904 snap 304 seq_num 904) 2021-03-15T22:31:04.386 INFO:tasks.rados.rados.0.smithi076.stdout:2615: finishing write tid 1 to smithi07694878-24 2021-03-15T22:31:04.386 INFO:tasks.rados.rados.0.smithi076.stdout:2615: finishing write tid 2 to smithi07694878-24 2021-03-15T22:31:04.386 INFO:tasks.rados.rados.0.smithi076.stdout:2615: finishing write tid 3 to smithi07694878-24 2021-03-15T22:31:04.386 INFO:tasks.rados.rados.0.smithi076.stdout:2615: finishing write tid 4 to smithi07694878-24 2021-03-15T22:31:04.386 INFO:tasks.rados.rados.0.smithi076.stdout:2615: finishing write tid 5 to smithi07694878-24 2021-03-15T22:31:04.387 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1714 (ObjNum 905 snap 304 seq_num 905) dirty exists 2021-03-15T22:31:04.387 INFO:tasks.rados.rados.0.smithi076.stdout:2615: left oid 24 (ObjNum 905 snap 304 seq_num 905) 2021-03-15T22:31:04.387 INFO:tasks.rados.rados.0.smithi076.stdout:2616: finishing write tid 1 to smithi07694878-3 2021-03-15T22:31:04.387 INFO:tasks.rados.rados.0.smithi076.stdout:2616: finishing write tid 2 to smithi07694878-3 2021-03-15T22:31:04.388 INFO:tasks.rados.rados.0.smithi076.stdout:2620: seq_num 908 ranges {460756=572099,1517297=444312,2668548=472870} 2021-03-15T22:31:04.388 INFO:tasks.rados.rados.0.smithi076.stdout:2620: writing smithi07694878-31 from 460756 to 1032855 tid 1 2021-03-15T22:31:04.391 INFO:tasks.rados.rados.0.smithi076.stdout:2620: writing smithi07694878-31 from 1517297 to 1961609 tid 2 2021-03-15T22:31:04.399 INFO:tasks.rados.rados.0.smithi076.stdout:2620: writing smithi07694878-31 from 2668548 to 3141418 tid 3 2021-03-15T22:31:04.399 INFO:tasks.rados.rados.0.smithi076.stdout:2614: done (6 left) 2021-03-15T22:31:04.399 INFO:tasks.rados.rados.0.smithi076.stdout:2615: done (5 left) 2021-03-15T22:31:04.400 INFO:tasks.rados.rados.0.smithi076.stdout:2621: read oid 12 snap -1 2021-03-15T22:31:04.400 INFO:tasks.rados.rados.0.smithi076.stdout:2621: expect deleted 2021-03-15T22:31:04.400 INFO:tasks.rados.rados.0.smithi076.stdout:2622: read oid 27 snap 294 2021-03-15T22:31:04.400 INFO:tasks.rados.rados.0.smithi076.stdout:2622: expect (ObjNum 854 snap 282 seq_num 854) 2021-03-15T22:31:04.401 INFO:tasks.rados.rados.0.smithi076.stdout:2623: snap_remove snap 281 2021-03-15T22:31:04.418 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:04.441 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:04.442 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:04 smithi076 conmon[56514]: cluster 2021-03-15T22:31:02.790529+0000 osd.4 (osd.4) 153 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:04 smithi076 conmon[56514]: cluster 2021-03-15T22:31:02.791342+0000 osd.4 (osd.4) 154 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:04 smithi076 conmon[56514]: cluster 2021-03-15T22:31:03.154208+0000 mgr.y (mgr.14144) 875 : cluster [DBG] pgmap v1214: 138 pgs: 1 peering, 137 active+clean; 553 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2021-03-15T22:31:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:04 smithi076 conmon[56514]: cluster 2021-03-15T22:31:03.281340+0000 osd.6 (osd.6) 112 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:31:04.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:04 smithi076 conmon[62156]: cluster 2021-03-15T22:31:02.790529+0000 osd.4 (osd.4) 153 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:04.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:04 smithi076 conmon[62156]: cluster 2021-03-15T22:31:02.791342+0000 osd.4 (osd.4) 154 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:04.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:04 smithi076 conmon[62156]: cluster 2021-03-15T22:31:03.154208+0000 mgr.y (mgr.14144) 875 : cluster [DBG] pgmap v1214: 138 pgs: 1 peering, 137 active+clean; 553 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2021-03-15T22:31:04.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:04 smithi076 conmon[62156]: cluster 2021-03-15T22:31:03.281340+0000 osd.6 (osd.6) 112 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:31:04.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:04 smithi125 conmon[56739]: cluster 2021-03-15T22:31:02.790529+0000 osd.4 (osd.4) 153 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:04 smithi125 conmon[56739]: cluster 2021-03-15T22:31 2021-03-15T22:31:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:04 smithi125 conmon[56739]: :02.791342+0000 osd.4 (osd.4) 154 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:04 smithi125 conmon[56739]: cluster 2021-03-15T22:31:03.154208+0000 mgr.y (mgr.14144) 875 : cluster [DBG] pgmap v1214: 138 pgs: 1 peering, 137 active+clean; 553 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.8 MiB/s wr, 8 op/s 2021-03-15T22:31:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:04 smithi125 conmon[56739]: cluster 2021-03-15T22:31:03.281340+0000 osd.6 (osd.6) 112 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:31:04.543 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:04.569 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:04.569 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:04.671 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:04.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:04.697 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:04.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:04.758 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:31:04.800 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:04.824 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:04.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:04.926 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:04.927 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:31:04.927 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:31:04.928 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:31:04.928 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:31:04.928 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:31:04.950 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:04.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:04.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:05.033 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:05.053 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:05.075 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:05.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:05.178 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:05.200 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:05.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:05.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:05.235 INFO:tasks.rados.rados.0.smithi076.stdout:2623: done (7 left) 2021-03-15T22:31:05.235 INFO:tasks.rados.rados.0.smithi076.stdout:2624: read oid 23 snap -1 2021-03-15T22:31:05.235 INFO:tasks.rados.rados.0.smithi076.stdout:2624: expect (ObjNum 871 snap 287 seq_num 871) 2021-03-15T22:31:05.236 INFO:tasks.rados.rados.0.smithi076.stdout:2625: snap_create 2021-03-15T22:31:05.236 INFO:tasks.rados.rados.0.smithi076.stdout:2617: expect (ObjNum 736 snap 242 seq_num 736) 2021-03-15T22:31:05.244 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:31:05.304 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:05.328 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:05.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:05.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:05.339 INFO:tasks.rados.rados.0.smithi076.stdout:2616: finishing write tid 3 to smithi07694878-3 2021-03-15T22:31:05.339 INFO:tasks.rados.rados.0.smithi076.stdout:2616: finishing write tid 4 to smithi07694878-3 2021-03-15T22:31:05.339 INFO:tasks.rados.rados.0.smithi076.stdout:2618: finishing rollback tid 1 to smithi07694878-14 2021-03-15T22:31:05.340 INFO:tasks.rados.rados.0.smithi076.stdout:2616: finishing write tid 5 to smithi07694878-3 2021-03-15T22:31:05.340 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 1577 (ObjNum 906 snap 304 seq_num 906) dirty exists 2021-03-15T22:31:05.340 INFO:tasks.rados.rados.0.smithi076.stdout:2616: left oid 3 (ObjNum 906 snap 304 seq_num 906) 2021-03-15T22:31:05.340 INFO:tasks.rados.rados.0.smithi076.stdout:2618: finishing rollback tid 2 to smithi07694878-14 2021-03-15T22:31:05.340 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 1579 (ObjNum 623 snap 193 seq_num 623) dirty exists 2021-03-15T22:31:05.341 INFO:tasks.rados.rados.0.smithi076.stdout:2619: finishing write tid 1 to smithi07694878-39 2021-03-15T22:31:05.341 INFO:tasks.rados.rados.0.smithi076.stdout:2619: finishing write tid 2 to smithi07694878-39 2021-03-15T22:31:05.341 INFO:tasks.rados.rados.0.smithi076.stdout:2619: finishing write tid 3 to smithi07694878-39 2021-03-15T22:31:05.341 INFO:tasks.rados.rados.0.smithi076.stdout:2619: finishing write tid 4 to smithi07694878-39 2021-03-15T22:31:05.342 INFO:tasks.rados.rados.0.smithi076.stdout:2619: finishing write tid 5 to smithi07694878-39 2021-03-15T22:31:05.342 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2068 (ObjNum 907 snap 304 seq_num 907) dirty exists 2021-03-15T22:31:05.342 INFO:tasks.rados.rados.0.smithi076.stdout:2619: left oid 39 (ObjNum 907 snap 304 seq_num 907) 2021-03-15T22:31:05.342 INFO:tasks.rados.rados.0.smithi076.stdout:2620: finishing write tid 1 to smithi07694878-31 2021-03-15T22:31:05.342 INFO:tasks.rados.rados.0.smithi076.stdout:2620: finishing write tid 2 to smithi07694878-31 2021-03-15T22:31:05.343 INFO:tasks.rados.rados.0.smithi076.stdout:2622: expect (ObjNum 854 snap 282 seq_num 854) 2021-03-15T22:31:05.431 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:05.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:05.456 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:05.457 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:05.480 INFO:tasks.rados.rados.0.smithi076.stdout:2620: finishing write tid 3 to smithi07694878-31 2021-03-15T22:31:05.481 INFO:tasks.rados.rados.0.smithi076.stdout:2620: finishing write tid 4 to smithi07694878-31 2021-03-15T22:31:05.481 INFO:tasks.rados.rados.0.smithi076.stdout:2620: finishing write tid 5 to smithi07694878-31 2021-03-15T22:31:05.481 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 1856 (ObjNum 908 snap 304 seq_num 908) dirty exists 2021-03-15T22:31:05.482 INFO:tasks.rados.rados.0.smithi076.stdout:2620: left oid 31 (ObjNum 908 snap 304 seq_num 908) 2021-03-15T22:31:05.482 INFO:tasks.rados.rados.0.smithi076.stdout:2624: expect (ObjNum 871 snap 287 seq_num 871) 2021-03-15T22:31:05.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-15T22:31:05.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[56514]: cluster 2021-03-15T22:31:04.009742+0000 osd.0 ( 2021-03-15T22:31:05.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[56514]: osd.0) 114 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:31:05.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[56514]: cluster 2021-03-15T22:31:04.276246+0000 mon.a (mon.0) 1913 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2021-03-15T22:31:05.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[56514]: cluster 2021-03-15T22:31:04.283351+0000 osd.6 (osd.6) 113 : cluster [DBG] 6.f scrub ok 2021-03-15T22:31:05.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[56514]: cluster 2021-03-15T22:31:04.380414+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.d scrub ok 2021-03-15T22:31:05.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[56514]: cluster 2021-03-15T22:31:04.495570+0000 osd.4 (osd.4) 155 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:31:05.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[56514]: audit 2021-03-15T22:31:04.577897+0000 mon.a (mon.0) 1914 : audit [INF] from='client.? 172.21.15.76:0/1307632643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:05.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[56514]: cluster 2021-03-15T22:31:05.220216+0000 mon.a (mon.0) 1915 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:31:05.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[56514]: cluster 2021-03-15T22:31:05.220244+0000 mon.a (mon.0) 1916 : cluster [INF] Cluster is now healthy 2021-03-15T22:31:05.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[56514]: audit 2021-03-15T22:31:05.227502+0000 mon.a (mon.0) 1917 : audit [INF] from='client.? 172.21.15.76:0/1307632643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:05.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[56514]: cluster 2021-03-15T22:31:05.227624+0000 mon.a (mon.0) 1918 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2021-03-15T22:31:05.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: cluster 2021-03-15T22:31:04. 2021-03-15T22:31:05.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: 009742+0000 osd.0 (osd.0) 114 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:31:05.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: cluster 2021-03-15T22:31:05.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: 2021-03-15T22:31:04.276246+0000 2021-03-15T22:31:05.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: mon.a (mon.0) 1913 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2021-03-15T22:31:05.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: 2021-03-15T22:31:05.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:31:05.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: T22:31:04.283351+0000 osd.6 (osd 2021-03-15T22:31:05.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: .6) 113 : cluster [DBG] 6.f scrub ok 2021-03-15T22:31:05.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:31:05.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: -03-15T22:31:04.380414+0000 2021-03-15T22:31:05.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: osd.5 (osd.5) 18 : cluster [DBG] 2021-03-15T22:31:05.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: 6.d scrub ok 2021-03-15T22:31:05.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: cluster 2021-03-15T22:31:04 2021-03-15T22:31:05.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: .495570+0000 osd.4 (osd.4) 155 : 2021-03-15T22:31:05.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: cluster [DBG] 5.1 scrub ok 2021-03-15T22:31:05.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:31:05.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: :31:04.577897+0000 mon.a (mon. 2021-03-15T22:31:05.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: 0) 1914 : audit [INF] from='client.? 172.21.15.76:0/1307632643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:05.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:31:05.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: T22:31:05.220216+0000 mon.a (mon 2021-03-15T22:31:05.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: .0) 1915 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:31:05.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:31:05.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: -03-15T22:31:05.220244+0000 2021-03-15T22:31:05.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: mon.a (mon.0) 1916 : cluster [INF] 2021-03-15T22:31:05.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: Cluster is now healthy 2021-03-15T22:31:05.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: audit 2021-03-15T22:31: 2021-03-15T22:31:05.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: 05.227502+0000 mon.a (mon.0) 1917 : 2021-03-15T22:31:05.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/1307632643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:05.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:31:05.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: 31:05.227624+0000 mon.a (mon.0) 2021-03-15T22:31:05.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:05 smithi076 conmon[62156]: 1918 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2021-03-15T22:31:05.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:05 smithi125 conmon[56739]: cluster 2021-03-15T22:31:04.009742+0000 osd.0 (osd.0) 114 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:31:05.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:05 smithi125 conmon[56739]: cluster 2021-03-15T22:31:04.276246+0000 mon.a (mon.0) 1913 : cluster [DBG] osdmap e715: 8 total, 8 up, 8 in 2021-03-15T22:31:05.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:05 smithi125 conmon[56739]: cluster 2021-03-15T22:31:04.283351+0000 osd.6 (osd.6) 113 : cluster [DBG] 6.f scrub ok 2021-03-15T22:31:05.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:05 smithi125 conmon[56739]: cluster 2021-03-15T22:31:04.380414+0000 osd.5 (osd.5) 18 : cluster [DBG] 6.d scrub ok 2021-03-15T22:31:05.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:05 smithi125 conmon[56739]: cluster 2021-03-15T22:31:04.495570+0000 osd.4 (osd.4) 155 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:31:05.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:05 smithi125 conmon[56739]: audit 2021-03-15T22:31:04.577897+0000 mon.a (mon.0) 1914 : audit [INF] from='client.? 172.21.15.76:0/1307632643' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:05.548 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:05 smithi125 conmon[56739]: cluster 2021-03-15T22:31:05.220216+0000 mon.a (mon.0) 1915 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:31:05.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:05 smithi125 conmon[56739]: cluster 2021-03-15T22:31:05.220244+0000 mon.a (mon.0) 1916 : cluster [INF] Cluster is now healthy 2021-03-15T22:31:05.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:05 smithi125 conmon[56739]: audit 2021-03-15T22:31:05.227502+0000 mon.a (mon.0) 1917 : audit [INF] from='client.? 172.21.15.76:0/1307632643' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:05.549 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:05 smithi125 conmon[56739]: cluster 2021-03-15T22:31:05.227624+0000 mon.a (mon.0) 1918 : cluster [DBG] osdmap e716: 8 total, 8 up, 8 in 2021-03-15T22:31:05.558 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:05.591 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:05.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:05.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:05.694 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:05.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:05.719 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:05.719 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:05.819 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:31:05.820 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:05.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:05.845 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:05.845 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:05.947 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:05.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:05.971 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:05.971 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:06.072 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:06.096 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:06.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:06.098 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:06.199 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:06.221 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:06.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:06.223 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:06.239 INFO:tasks.rados.rados.0.smithi076.stdout:2616: done (8 left) 2021-03-15T22:31:06.239 INFO:tasks.rados.rados.0.smithi076.stdout:2617: done (7 left) 2021-03-15T22:31:06.239 INFO:tasks.rados.rados.0.smithi076.stdout:2618: done (6 left) 2021-03-15T22:31:06.240 INFO:tasks.rados.rados.0.smithi076.stdout:2619: done (5 left) 2021-03-15T22:31:06.240 INFO:tasks.rados.rados.0.smithi076.stdout:2620: done (4 left) 2021-03-15T22:31:06.240 INFO:tasks.rados.rados.0.smithi076.stdout:2621: done (3 left) 2021-03-15T22:31:06.240 INFO:tasks.rados.rados.0.smithi076.stdout:2622: done (2 left) 2021-03-15T22:31:06.241 INFO:tasks.rados.rados.0.smithi076.stdout:2624: done (1 left) 2021-03-15T22:31:06.241 INFO:tasks.rados.rados.0.smithi076.stdout:2625: done (0 left) 2021-03-15T22:31:06.241 INFO:tasks.rados.rados.0.smithi076.stdout:2626: copy_from oid 40 from oid 5 current snap is 305 2021-03-15T22:31:06.241 INFO:tasks.rados.rados.0.smithi076.stdout:2627: delete oid 37 current snap is 305 2021-03-15T22:31:06.244 INFO:tasks.rados.rados.0.smithi076.stdout:2627: done (1 left) 2021-03-15T22:31:06.244 INFO:tasks.rados.rados.0.smithi076.stdout:2628: snap_remove snap 302 2021-03-15T22:31:06.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:06 smithi125 conmon[56739]: cluster 2021-03-15T22:31:05.123415+0000 osd.1 (osd.1) 138 2021-03-15T22:31:06.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:06 smithi125 conmon[56739]: : cluster [DBG] 3.19 scrub ok 2021-03-15T22:31:06.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:06 smithi125 conmon[56739]: cluster 2021-03-15T22:31:05.154645+0000 mgr.y (mgr.14144) 876 : cluster [DBG] pgmap v1217: 138 pgs: 1 peering, 137 active+clean; 550 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.1 MiB/s wr, 4 op/s 2021-03-15T22:31:06.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:06 smithi125 conmon[56739]: cluster 2021-03-15T22:31:05.262012+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:31:06.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:06 smithi125 conmon[56739]: cluster 2021-03-15T22:31:05.468996+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:31:06.288 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:06 smithi125 conmon[56739]: cluster 2021-03-15T22:31:06.235290+0000 mon.a (mon.0) 1919 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2021-03-15T22:31:06.324 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:06.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:06.347 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:06.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:06.405 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 0, 3, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:31:06.406 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:31:06.406 INFO:tasks.thrashosds.thrasher:Growing pool default.rgw.control 2021-03-15T22:31:06.407 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:31:06.429 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[56514]: cluster 2021-03-15T22:31:05.123415+0000 osd.1 ( 2021-03-15T22:31:06.430 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[56514]: osd.1) 138 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:31:06.430 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[56514]: cluster 2021-03-15T22:31:05.154645+0000 mgr.y (mgr.14144) 876 : cluster [DBG] pgmap v1217: 138 pgs: 1 peering, 137 active+clean; 550 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.1 MiB/s wr, 4 op/s 2021-03-15T22:31:06.430 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[56514]: cluster 2021-03-15T22:31:05.262012+0000 osd.3 (osd.3) 129 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:31:06.430 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[56514]: cluster 2021-03-15T22:31:05.468996+0000 osd.4 (osd.4) 156 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:31:06.431 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[56514]: cluster 2021-03-15T22:31:06.235290+0000 mon.a (mon.0) 1919 : cluster [DBG] osdmap e717: 8 total, 8 up, 8 in 2021-03-15T22:31:06.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: cluster 2021-03-15T22:31:05.123415+0000 2021-03-15T22:31:06.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: osd.1 (osd.1) 138 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:31:06.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:31:06.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: 22:31:05.154645+0000 mgr.y (mgr.14144) 876 : cluster [DBG] pgmap v1217: 138 pgs: 1 peering, 137 active+clean; 550 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 1.7 KiB/s rd, 1.1 MiB/s wr, 4 op/s 2021-03-15T22:31:06.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:31:06.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: -03-15T22:31:05.262012+0000 2021-03-15T22:31:06.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: osd.3 (osd.3) 129 : cluster [DBG] 2021-03-15T22:31:06.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: 6.1b scrub ok 2021-03-15T22:31:06.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:31:06.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: :31:05.468996+0000 osd.4 (osd.4 2021-03-15T22:31:06.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: ) 156 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:31:06.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:31:06.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: -03-15T22:31:06.235290+0000 2021-03-15T22:31:06.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: mon.a (mon.0) 1919 : cluster [DBG] 2021-03-15T22:31:06.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:06 smithi076 conmon[62156]: osdmap e717: 8 total, 8 up, 8 in 2021-03-15T22:31:06.449 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:06.471 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:06.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:06.473 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:06.574 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:06.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:06.598 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:06.598 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:06.700 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:06.722 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:06.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:06.724 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:06.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:06.847 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:06.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:06.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:06.949 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:06.972 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:06.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:06.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:06.982 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:07.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:07.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:07.098 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:07.098 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:07.199 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:07.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:07.225 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:07.226 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:07.242 INFO:tasks.rados.rados.0.smithi076.stdout:2628: done (1 left) 2021-03-15T22:31:07.243 INFO:tasks.rados.rados.0.smithi076.stdout:2629: snap_create 2021-03-15T22:31:07.243 INFO:tasks.rados.rados.0.smithi076.stdout:2626: finishing copy_from to smithi07694878-40 2021-03-15T22:31:07.243 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 1517 (ObjNum 882 snap 293 seq_num 882) dirty exists 2021-03-15T22:31:07.243 INFO:tasks.rados.rados.0.smithi076.stdout:2626: finishing copy_from racing read to smithi07694878-40 2021-03-15T22:31:07.327 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:07.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:07.354 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:07.355 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:07.432 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:31:07.432 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:31:07.432 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:31:07.433 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:31:07.433 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.433 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:1359 5.14 5:2b04a3e9:::notify.6:head [watch reconnect cookie 94751391777792 gen 3] snapc 0=[] ondisk+write+known_if_redirected e316)", 2021-03-15T22:31:07.433 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:52.068429+0000", 2021-03-15T22:31:07.434 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 435.359976663, 2021-03-15T22:31:07.434 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.20407250399999999, 2021-03-15T22:31:07.434 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.434 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.435 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.435 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:31:07.435 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:31:07.435 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1359 2021-03-15T22:31:07.436 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.436 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.436 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.436 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.437 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068429+0000", 2021-03-15T22:31:07.437 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.438 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.438 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.438 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.438 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068429+0000", 2021-03-15T22:31:07.439 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.123e-06 2021-03-15T22:31:07.439 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.439 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.439 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.440 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068432+0000", 2021-03-15T22:31:07.440 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.066e-06 2021-03-15T22:31:07.440 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.440 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.441 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.441 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068435+0000", 2021-03-15T22:31:07.441 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5502000000000002e-05 2021-03-15T22:31:07.441 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.441 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.442 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.442 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068451+0000", 2021-03-15T22:31:07.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.348e-06 2021-03-15T22:31:07.442 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.443 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.443 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068457+0000", 2021-03-15T22:31:07.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0260999999999998e-05 2021-03-15T22:31:07.444 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.445 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068517+0000", 2021-03-15T22:31:07.447 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1440999999999999e-05 2021-03-15T22:31:07.447 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.447 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:07.448 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068529+0000", 2021-03-15T22:31:07.448 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19618954299999999 2021-03-15T22:31:07.448 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.449 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.449 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.264718+0000", 2021-03-15T22:31:07.449 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000154158 2021-03-15T22:31:07.449 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.449 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.450 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.450 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.264872+0000", 2021-03-15T22:31:07.450 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0076290619999999998 2021-03-15T22:31:07.450 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.272501+0000", 2021-03-15T22:31:07.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.20407250399999999 2021-03-15T22:31:07.451 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.452 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.452 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.452 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:1360 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94751391778944 gen 4] snapc 0=[] ondisk+write+known_if_redirected e316)", 2021-03-15T22:31:07.453 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:52.068469+0000", 2021-03-15T22:31:07.453 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 435.35993604700002, 2021-03-15T22:31:07.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19939890299999999, 2021-03-15T22:31:07.453 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.453 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.454 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.454 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:31:07.454 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:31:07.454 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 1360 2021-03-15T22:31:07.455 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.455 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.455 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.455 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.455 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068469+0000", 2021-03-15T22:31:07.456 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.456 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.456 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068469+0000", 2021-03-15T22:31:07.457 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5300000000000003e-07 2021-03-15T22:31:07.457 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.457 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068470+0000", 2021-03-15T22:31:07.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1710000000000001e-06 2021-03-15T22:31:07.458 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.458 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.458 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.459 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068472+0000", 2021-03-15T22:31:07.459 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3089999999999997e-06 2021-03-15T22:31:07.459 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.459 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.459 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.460 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068476+0000", 2021-03-15T22:31:07.460 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.489e-06 2021-03-15T22:31:07.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.460 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.461 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068479+0000", 2021-03-15T22:31:07.461 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1756999999999998e-05 2021-03-15T22:31:07.461 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.461 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.462 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.462 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068541+0000", 2021-03-15T22:31:07.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.311e-05 2021-03-15T22:31:07.462 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.462 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.463 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:07.463 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.068554+0000", 2021-03-15T22:31:07.463 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19462378499999999 2021-03-15T22:31:07.463 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.463 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.464 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.464 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.263178+0000", 2021-03-15T22:31:07.464 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000179648 2021-03-15T22:31:07.464 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.465 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.465 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.263358+0000", 2021-03-15T22:31:07.465 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045107810000000002 2021-03-15T22:31:07.465 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.466 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.466 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.466 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.267868+0000", 2021-03-15T22:31:07.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.19939890299999999 2021-03-15T22:31:07.466 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.467 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.467 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.468 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34361.0:352 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e316)", 2021-03-15T22:31:07.468 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:23:52.097658+0000", 2021-03-15T22:31:07.468 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 435.33074737999999, 2021-03-15T22:31:07.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.17297211700000001, 2021-03-15T22:31:07.468 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.469 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.469 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.469 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34361", 2021-03-15T22:31:07.469 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.125:0/1442040869", 2021-03-15T22:31:07.469 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 352 2021-03-15T22:31:07.470 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.470 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.470 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.470 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.471 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097658+0000", 2021-03-15T22:31:07.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.471 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.471 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097658+0000", 2021-03-15T22:31:07.472 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2650000000000001e-06 2021-03-15T22:31:07.472 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.472 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.473 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097661+0000", 2021-03-15T22:31:07.473 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0199999999999999e-06 2021-03-15T22:31:07.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.473 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.473 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097664+0000", 2021-03-15T22:31:07.474 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6565000000000001e-05 2021-03-15T22:31:07.474 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.474 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.475 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.475 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097681+0000", 2021-03-15T22:31:07.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9939999999999997e-06 2021-03-15T22:31:07.475 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.475 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.476 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097687+0000", 2021-03-15T22:31:07.476 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2488000000000003e-05 2021-03-15T22:31:07.476 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.477 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.477 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.477 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097739+0000", 2021-03-15T22:31:07.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0113e-05 2021-03-15T22:31:07.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:07.478 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.097750+0000", 2021-03-15T22:31:07.478 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.172684848 2021-03-15T22:31:07.478 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.479 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.479 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.270434+0000", 2021-03-15T22:31:07.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9377999999999998e-05 2021-03-15T22:31:07.480 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.480 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.270514+0000", 2021-03-15T22:31:07.480 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000116446 2021-03-15T22:31:07.481 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.481 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.481 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.481 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:23:52.270630+0000", 2021-03-15T22:31:07.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.17297211700000001 2021-03-15T22:31:07.482 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.482 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.482 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.482 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.483 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.483 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3289 7.f 7:fc7cd984:::smithi07694878-4:head [delete] snapc fd=[fd,fc,fb,fa,f9,f7,f5,e9,e7,de] ondisk+write+peerstat_old+known_if_redirected e355)", 2021-03-15T22:31:07.483 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:30.295366+0000", 2021-03-15T22:31:07.483 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 397.13303920700002, 2021-03-15T22:31:07.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8092550220000001, 2021-03-15T22:31:07.484 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.484 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.484 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.484 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.484 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.485 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3289 2021-03-15T22:31:07.485 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.485 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.485 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.486 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.486 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295366+0000", 2021-03-15T22:31:07.486 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.487 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.487 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295366+0000", 2021-03-15T22:31:07.487 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8169999999999999e-06 2021-03-15T22:31:07.487 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.487 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.488 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.488 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295374+0000", 2021-03-15T22:31:07.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6029999999999999e-06 2021-03-15T22:31:07.488 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.488 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.489 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.489 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295382+0000", 2021-03-15T22:31:07.489 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1372000000000003e-05 2021-03-15T22:31:07.489 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.490 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.490 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.490 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295423+0000", 2021-03-15T22:31:07.490 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1537000000000001e-05 2021-03-15T22:31:07.490 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.491 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.491 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.491 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295445+0000", 2021-03-15T22:31:07.491 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012213400000000001 2021-03-15T22:31:07.491 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.492 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.492 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.492 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295567+0000", 2021-03-15T22:31:07.492 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1673000000000001e-05 2021-03-15T22:31:07.492 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.493 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:07.493 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.295588+0000", 2021-03-15T22:31:07.493 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79131791699999998 2021-03-15T22:31:07.494 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.494 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.494 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.494 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.086906+0000", 2021-03-15T22:31:07.494 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0113e-05 2021-03-15T22:31:07.495 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.495 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.495 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:07.495 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.086916+0000", 2021-03-15T22:31:07.495 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0123092789999999 2021-03-15T22:31:07.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.496 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.496 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.099226+0000", 2021-03-15T22:31:07.496 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016286 2021-03-15T22:31:07.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.497 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.497 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for missing object", 2021-03-15T22:31:07.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.099389+0000", 2021-03-15T22:31:07.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001614868 2021-03-15T22:31:07.498 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.498 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.101003+0000", 2021-03-15T22:31:07.499 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023411800000000001 2021-03-15T22:31:07.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.499 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.499 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.101238+0000", 2021-03-15T22:31:07.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033837310000000001 2021-03-15T22:31:07.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.500 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:32.104621+0000", 2021-03-15T22:31:07.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8092550220000001 2021-03-15T22:31:07.501 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.501 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.501 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.502 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.502 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34361.0:390 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e354)", 2021-03-15T22:31:07.502 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:30.317842+0000", 2021-03-15T22:31:07.502 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 397.11056336299998, 2021-03-15T22:31:07.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79261869600000001, 2021-03-15T22:31:07.503 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.503 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.503 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.503 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34361", 2021-03-15T22:31:07.504 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.125:0/1442040869", 2021-03-15T22:31:07.504 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 390 2021-03-15T22:31:07.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.504 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.505 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.505 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.505 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317842+0000", 2021-03-15T22:31:07.505 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.505 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.506 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.506 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.506 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317842+0000", 2021-03-15T22:31:07.506 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2830000000000004e-06 2021-03-15T22:31:07.506 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.507 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.507 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.507 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317849+0000", 2021-03-15T22:31:07.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2929999999999997e-06 2021-03-15T22:31:07.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.508 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.508 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317856+0000", 2021-03-15T22:31:07.508 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2073000000000003e-05 2021-03-15T22:31:07.508 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.509 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.509 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.509 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317888+0000", 2021-03-15T22:31:07.509 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7453e-05 2021-03-15T22:31:07.510 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.510 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.510 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.510 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.317905+0000", 2021-03-15T22:31:07.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015652 2021-03-15T22:31:07.511 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.511 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.511 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.511 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.318062+0000", 2021-03-15T22:31:07.511 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8180000000000004e-06 2021-03-15T22:31:07.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.512 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:07.512 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:30.318070+0000", 2021-03-15T22:31:07.512 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79211001599999997 2021-03-15T22:31:07.513 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.513 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.513 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.513 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.110180+0000", 2021-03-15T22:31:07.514 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000129782 2021-03-15T22:31:07.514 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.514 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.514 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.514 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.110309+0000", 2021-03-15T22:31:07.515 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015145800000000001 2021-03-15T22:31:07.515 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.515 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.515 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.515 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:31.110461+0000", 2021-03-15T22:31:07.516 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.79261869600000001 2021-03-15T22:31:07.516 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.516 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.516 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.517 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.517 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3664 7.6 7:608078b0:::smithi07694878-12:head [delete] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+peerstat_old+known_if_redirected e385)", 2021-03-15T22:31:07.517 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.131030+0000", 2021-03-15T22:31:07.517 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 367.29737577700001, 2021-03-15T22:31:07.518 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039032994000000001, 2021-03-15T22:31:07.518 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.518 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:07.518 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.518 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.519 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.519 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3664 2021-03-15T22:31:07.519 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.519 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.520 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131030+0000", 2021-03-15T22:31:07.520 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.521 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.521 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.521 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131030+0000", 2021-03-15T22:31:07.521 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9670000000000003e-06 2021-03-15T22:31:07.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131034+0000", 2021-03-15T22:31:07.522 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5580000000000002e-06 2021-03-15T22:31:07.522 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.523 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.523 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.523 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131038+0000", 2021-03-15T22:31:07.523 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0582e-05 2021-03-15T22:31:07.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.524 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.524 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131059+0000", 2021-03-15T22:31:07.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1455e-05 2021-03-15T22:31:07.525 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.525 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.525 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131070+0000", 2021-03-15T22:31:07.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3596999999999994e-05 2021-03-15T22:31:07.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.526 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.526 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131134+0000", 2021-03-15T22:31:07.527 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8918999999999994e-05 2021-03-15T22:31:07.527 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.527 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.527 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.527 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131213+0000", 2021-03-15T22:31:07.528 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000169662 2021-03-15T22:31:07.528 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.528 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:31:07.528 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.131382+0000", 2021-03-15T22:31:07.529 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001594575 2021-03-15T22:31:07.529 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.529 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.529 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.529 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.132977+0000", 2021-03-15T22:31:07.530 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035954851000000003 2021-03-15T22:31:07.530 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.530 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.530 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.530 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.168932+0000", 2021-03-15T22:31:07.531 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010296630000000001 2021-03-15T22:31:07.531 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.531 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:07.532 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.169962+0000", 2021-03-15T22:31:07.532 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2741000000000003e-05 2021-03-15T22:31:07.532 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.532 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.532 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:07.533 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.170004+0000", 2021-03-15T22:31:07.533 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8424000000000002e-05 2021-03-15T22:31:07.533 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.533 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.534 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.170063+0000", 2021-03-15T22:31:07.534 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039032994000000001 2021-03-15T22:31:07.534 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.534 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.534 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.535 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.535 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.535 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3679 7.6 7:6de8b070:::smithi07694878-15:head [append 0~0] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:31:07.535 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.195576+0000", 2021-03-15T22:31:07.536 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 367.23282942700001, 2021-03-15T22:31:07.536 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038774930999999999, 2021-03-15T22:31:07.536 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.536 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:07.536 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.537 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.537 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.537 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3679 2021-03-15T22:31:07.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.537 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.538 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.538 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195576+0000", 2021-03-15T22:31:07.538 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.539 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.539 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.539 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195576+0000", 2021-03-15T22:31:07.539 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1049999999999999e-06 2021-03-15T22:31:07.540 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.540 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.540 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195579+0000", 2021-03-15T22:31:07.540 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4559999999999999e-06 2021-03-15T22:31:07.541 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.541 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.541 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.541 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195582+0000", 2021-03-15T22:31:07.542 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2951e-05 2021-03-15T22:31:07.542 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.542 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.542 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195595+0000", 2021-03-15T22:31:07.543 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3730000000000001e-06 2021-03-15T22:31:07.543 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.543 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.543 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.543 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195600+0000", 2021-03-15T22:31:07.544 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6076999999999996e-05 2021-03-15T22:31:07.544 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.544 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.544 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.544 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195686+0000", 2021-03-15T22:31:07.545 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2466999999999996e-05 2021-03-15T22:31:07.545 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.545 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.546 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195759+0000", 2021-03-15T22:31:07.546 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016388000000000001 2021-03-15T22:31:07.546 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.546 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.546 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:31:07.547 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195922+0000", 2021-03-15T22:31:07.547 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0018566839999999999 2021-03-15T22:31:07.547 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.547 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.547 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:07.548 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197779+0000", 2021-03-15T22:31:07.548 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1146000000000001e-05 2021-03-15T22:31:07.548 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.548 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.548 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.549 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197830+0000", 2021-03-15T22:31:07.549 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036392299000000003 2021-03-15T22:31:07.549 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.549 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.550 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.550 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234223+0000", 2021-03-15T22:31:07.550 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6313e-05 2021-03-15T22:31:07.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.550 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:07.551 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234289+0000", 2021-03-15T22:31:07.551 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2180000000000004e-05 2021-03-15T22:31:07.551 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.551 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.552 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234351+0000", 2021-03-15T22:31:07.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038774930999999999 2021-03-15T22:31:07.552 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.552 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.553 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.553 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.553 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3680 7.6 7:6de8b070:::smithi07694878-15:head [rollback e7] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:31:07.554 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.195608+0000", 2021-03-15T22:31:07.554 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 367.23279747700002, 2021-03-15T22:31:07.554 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038942472999999998, 2021-03-15T22:31:07.554 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.554 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:07.555 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.555 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.555 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.555 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3680 2021-03-15T22:31:07.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.556 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.556 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195608+0000", 2021-03-15T22:31:07.556 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.557 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.557 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.557 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195608+0000", 2021-03-15T22:31:07.558 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0800000000000004e-07 2021-03-15T22:31:07.558 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.558 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.558 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195609+0000", 2021-03-15T22:31:07.559 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.015e-06 2021-03-15T22:31:07.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.559 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.559 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195610+0000", 2021-03-15T22:31:07.560 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4369999999999999e-06 2021-03-15T22:31:07.560 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.560 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.561 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195613+0000", 2021-03-15T22:31:07.561 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4880000000000001e-06 2021-03-15T22:31:07.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.561 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.562 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195616+0000", 2021-03-15T22:31:07.562 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001023368 2021-03-15T22:31:07.562 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.562 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.562 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.563 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.196639+0000", 2021-03-15T22:31:07.563 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5290000000000001e-05 2021-03-15T22:31:07.563 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.563 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.564 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.196694+0000", 2021-03-15T22:31:07.564 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014822899999999999 2021-03-15T22:31:07.564 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.564 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.565 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:31:07.565 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.196843+0000", 2021-03-15T22:31:07.565 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010197610000000001 2021-03-15T22:31:07.565 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.566 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.566 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197862+0000", 2021-03-15T22:31:07.566 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0350000000000001e-06 2021-03-15T22:31:07.566 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.566 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.567 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:07.567 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197867+0000", 2021-03-15T22:31:07.567 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036554689000000001 2021-03-15T22:31:07.567 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.567 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.568 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.568 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234422+0000", 2021-03-15T22:31:07.568 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2453000000000006e-05 2021-03-15T22:31:07.568 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.569 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.569 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:07.569 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234495+0000", 2021-03-15T22:31:07.569 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5999999999999999e-05 2021-03-15T22:31:07.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.570 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.570 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234551+0000", 2021-03-15T22:31:07.570 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.038942472999999998 2021-03-15T22:31:07.570 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.571 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.571 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.571 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.571 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.572 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3681 7.6 7:6de8b070:::smithi07694878-15:head [append 0~0] snapc 11b=[11b,118,117,116,115,114,111,10e,10d,10c,10b,108,102,100,f9,e9,e7,de] ondisk+write+known_if_redirected e385)", 2021-03-15T22:31:07.572 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:00.195641+0000", 2021-03-15T22:31:07.572 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 367.23276446599999, 2021-03-15T22:31:07.572 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039044739000000002, 2021-03-15T22:31:07.572 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.573 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:07.573 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.573 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.573 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.573 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3681 2021-03-15T22:31:07.574 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.574 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.574 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.574 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.574 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195641+0000", 2021-03-15T22:31:07.575 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.575 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.575 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.576 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195641+0000", 2021-03-15T22:31:07.576 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6000000000000002e-07 2021-03-15T22:31:07.576 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.576 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.576 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.577 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195642+0000", 2021-03-15T22:31:07.577 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.23e-07 2021-03-15T22:31:07.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.577 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195643+0000", 2021-03-15T22:31:07.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0450000000000001e-06 2021-03-15T22:31:07.578 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.578 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.579 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.579 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195646+0000", 2021-03-15T22:31:07.579 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0839999999999999e-06 2021-03-15T22:31:07.579 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.579 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.580 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.580 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.195648+0000", 2021-03-15T22:31:07.580 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015890559999999999 2021-03-15T22:31:07.580 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.580 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.581 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.581 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197237+0000", 2021-03-15T22:31:07.581 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4195000000000005e-05 2021-03-15T22:31:07.581 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.582 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.582 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.582 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197301+0000", 2021-03-15T22:31:07.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017653100000000001 2021-03-15T22:31:07.582 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.583 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.583 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:31:07.583 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197478+0000", 2021-03-15T22:31:07.583 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000394583 2021-03-15T22:31:07.583 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.584 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.584 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:07.584 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.197872+0000", 2021-03-15T22:31:07.584 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024062000000000001 2021-03-15T22:31:07.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.585 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.198113+0000", 2021-03-15T22:31:07.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036489966999999998 2021-03-15T22:31:07.586 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.586 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234603+0000", 2021-03-15T22:31:07.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0292e-05 2021-03-15T22:31:07.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:07.587 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234643+0000", 2021-03-15T22:31:07.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2783e-05 2021-03-15T22:31:07.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.588 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:00.234686+0000", 2021-03-15T22:31:07.589 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.039044739000000002 2021-03-15T22:31:07.589 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.589 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.589 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.590 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4380 7.6 7:6283974b:::smithi07694878-30:head [write 484200~624934,stat] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:31:07.590 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:31:07.590 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 297.29755484600003, 2021-03-15T22:31:07.590 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.122158787, 2021-03-15T22:31:07.591 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.591 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:07.591 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.591 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.591 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.592 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4380 2021-03-15T22:31:07.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.592 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.592 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.593 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.593 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:31:07.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.593 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.593 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.594 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:31:07.594 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00065488299999999998 2021-03-15T22:31:07.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.594 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131506+0000", 2021-03-15T22:31:07.595 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.037e-06 2021-03-15T22:31:07.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.596 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.596 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131512+0000", 2021-03-15T22:31:07.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9637000000000001e-05 2021-03-15T22:31:07.596 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.597 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.597 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131541+0000", 2021-03-15T22:31:07.597 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9325000000000001e-05 2021-03-15T22:31:07.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.598 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.598 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.598 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131571+0000", 2021-03-15T22:31:07.598 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6747000000000002e-05 2021-03-15T22:31:07.598 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.599 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.599 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.599 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131647+0000", 2021-03-15T22:31:07.599 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000176789 2021-03-15T22:31:07.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.600 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.600 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.600 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131824+0000", 2021-03-15T22:31:07.600 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00042344700000000003 2021-03-15T22:31:07.601 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.601 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:31:07.601 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.132248+0000", 2021-03-15T22:31:07.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0026234660000000001 2021-03-15T22:31:07.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.602 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.602 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.602 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.134871+0000", 2021-03-15T22:31:07.602 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031408100000000002 2021-03-15T22:31:07.603 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.603 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:07.603 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.135185+0000", 2021-03-15T22:31:07.603 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.117673658 2021-03-15T22:31:07.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.604 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.604 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.252859+0000", 2021-03-15T22:31:07.605 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6511999999999997e-05 2021-03-15T22:31:07.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.605 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:07.605 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.252915+0000", 2021-03-15T22:31:07.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4204999999999997e-05 2021-03-15T22:31:07.606 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.606 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253009+0000", 2021-03-15T22:31:07.607 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.122158787 2021-03-15T22:31:07.607 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.607 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.607 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.608 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4381 7.6 7:6283974b:::smithi07694878-30:head [write 1860843~223357] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:31:07.608 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:31:07.608 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 297.29228389899998, 2021-03-15T22:31:07.609 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119715842, 2021-03-15T22:31:07.609 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.609 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:07.609 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.609 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.610 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.610 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4381 2021-03-15T22:31:07.610 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.610 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.610 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.611 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.611 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:31:07.611 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.611 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.612 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.612 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.612 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:31:07.612 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019479299999999999 2021-03-15T22:31:07.612 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.613 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.613 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.613 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136316+0000", 2021-03-15T22:31:07.613 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7419999999999999e-06 2021-03-15T22:31:07.613 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.614 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.614 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136320+0000", 2021-03-15T22:31:07.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5999999999999999e-05 2021-03-15T22:31:07.614 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.615 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.615 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.615 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136336+0000", 2021-03-15T22:31:07.615 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4386e-05 2021-03-15T22:31:07.616 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.616 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.616 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.616 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136361+0000", 2021-03-15T22:31:07.616 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4256999999999999e-05 2021-03-15T22:31:07.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.617 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.617 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136415+0000", 2021-03-15T22:31:07.617 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000158257 2021-03-15T22:31:07.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.618 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.618 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:31:07.618 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136573+0000", 2021-03-15T22:31:07.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.116481746 2021-03-15T22:31:07.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.619 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.619 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.619 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253055+0000", 2021-03-15T22:31:07.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1245e-05 2021-03-15T22:31:07.620 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.620 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.620 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253126+0000", 2021-03-15T22:31:07.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000138727 2021-03-15T22:31:07.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:31:07.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253265+0000", 2021-03-15T22:31:07.622 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0017125980000000001 2021-03-15T22:31:07.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:07.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254977+0000", 2021-03-15T22:31:07.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000112898 2021-03-15T22:31:07.623 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.623 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.623 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.624 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255090+0000", 2021-03-15T22:31:07.624 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00066560399999999998 2021-03-15T22:31:07.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.624 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.625 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255756+0000", 2021-03-15T22:31:07.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8887999999999998e-05 2021-03-15T22:31:07.625 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.625 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.625 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:07.626 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255795+0000", 2021-03-15T22:31:07.626 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2701000000000003e-05 2021-03-15T22:31:07.626 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.626 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.627 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.627 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255837+0000", 2021-03-15T22:31:07.627 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119715842 2021-03-15T22:31:07.627 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.627 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.628 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.628 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.628 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.628 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4382 7.6 7:6283974b:::smithi07694878-30:head [setxattr _header (58) in=7b,truncate 2084200] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:31:07.628 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:31:07.629 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 297.29203481899998, 2021-03-15T22:31:07.629 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119633089, 2021-03-15T22:31:07.629 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.629 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:07.629 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.630 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.630 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.630 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4382 2021-03-15T22:31:07.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.631 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.631 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.631 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:31:07.631 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.632 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:31:07.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4609999999999998e-06 2021-03-15T22:31:07.633 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136373+0000", 2021-03-15T22:31:07.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7880000000000001e-06 2021-03-15T22:31:07.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.634 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136375+0000", 2021-03-15T22:31:07.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2340000000000002e-06 2021-03-15T22:31:07.635 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136380+0000", 2021-03-15T22:31:07.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3160000000000001e-06 2021-03-15T22:31:07.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.636 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136384+0000", 2021-03-15T22:31:07.637 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024020700000000001 2021-03-15T22:31:07.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.637 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.637 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136624+0000", 2021-03-15T22:31:07.638 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7928000000000003e-05 2021-03-15T22:31:07.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.638 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:31:07.639 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136702+0000", 2021-03-15T22:31:07.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.11744721399999999 2021-03-15T22:31:07.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.639 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.639 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.640 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254149+0000", 2021-03-15T22:31:07.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5769000000000001e-05 2021-03-15T22:31:07.640 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.641 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254205+0000", 2021-03-15T22:31:07.641 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000127489 2021-03-15T22:31:07.641 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.641 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:31:07.642 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254332+0000", 2021-03-15T22:31:07.642 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00068339200000000001 2021-03-15T22:31:07.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.642 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.643 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:07.643 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255016+0000", 2021-03-15T22:31:07.643 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000233143 2021-03-15T22:31:07.643 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.643 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.644 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.644 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255249+0000", 2021-03-15T22:31:07.644 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00063040000000000004 2021-03-15T22:31:07.644 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.644 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.645 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255879+0000", 2021-03-15T22:31:07.645 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9571000000000003e-05 2021-03-15T22:31:07.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.646 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:07.646 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255939+0000", 2021-03-15T22:31:07.646 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5177000000000003e-05 2021-03-15T22:31:07.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.647 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.647 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256004+0000", 2021-03-15T22:31:07.647 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119633089 2021-03-15T22:31:07.647 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.648 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.648 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.648 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.648 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.648 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4383 7.6 7:6283974b:::smithi07694878-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e455)", 2021-03-15T22:31:07.649 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:31:07.649 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 297.29201708800002, 2021-03-15T22:31:07.649 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119776914, 2021-03-15T22:31:07.649 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.649 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.650 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.650 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.650 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.650 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4383 2021-03-15T22:31:07.651 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.651 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.651 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.651 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.651 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:31:07.652 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.652 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:31:07.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9899999999999999e-07 2021-03-15T22:31:07.653 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.653 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.653 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.653 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136389+0000", 2021-03-15T22:31:07.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1799999999999999e-06 2021-03-15T22:31:07.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.654 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.655 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136390+0000", 2021-03-15T22:31:07.655 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0630000000000002e-06 2021-03-15T22:31:07.655 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.655 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.656 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136396+0000", 2021-03-15T22:31:07.656 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6900000000000001e-06 2021-03-15T22:31:07.656 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.656 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.657 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136399+0000", 2021-03-15T22:31:07.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034116099999999998 2021-03-15T22:31:07.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.657 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136740+0000", 2021-03-15T22:31:07.658 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000123552 2021-03-15T22:31:07.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.658 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.659 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:31:07.659 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136864+0000", 2021-03-15T22:31:07.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.118007994 2021-03-15T22:31:07.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.659 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.660 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.660 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254872+0000", 2021-03-15T22:31:07.660 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0631999999999996e-05 2021-03-15T22:31:07.660 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.661 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:31:07.661 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254963+0000", 2021-03-15T22:31:07.661 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010704379999999999 2021-03-15T22:31:07.661 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.661 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.662 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.662 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256033+0000", 2021-03-15T22:31:07.662 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2941e-05 2021-03-15T22:31:07.662 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.663 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.663 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.663 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256086+0000", 2021-03-15T22:31:07.663 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9463999999999997e-05 2021-03-15T22:31:07.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.664 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.664 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.664 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256165+0000", 2021-03-15T22:31:07.664 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119776914 2021-03-15T22:31:07.664 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.665 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.665 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.665 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4397 7.6 7:61e0cc05:::smithi07694878-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e457)", 2021-03-15T22:31:07.666 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:12.125625+0000", 2021-03-15T22:31:07.666 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 295.30278047000002, 2021-03-15T22:31:07.666 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035880710000000003, 2021-03-15T22:31:07.666 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.667 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.667 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.667 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.667 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.667 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4397 2021-03-15T22:31:07.668 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.668 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.668 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.668 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.668 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125625+0000", 2021-03-15T22:31:07.669 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.669 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.669 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125625+0000", 2021-03-15T22:31:07.670 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9230000000000001e-06 2021-03-15T22:31:07.670 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.670 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.670 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.671 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125627+0000", 2021-03-15T22:31:07.671 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.198e-06 2021-03-15T22:31:07.671 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.671 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.671 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.672 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125629+0000", 2021-03-15T22:31:07.672 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1629999999999996e-06 2021-03-15T22:31:07.672 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.672 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125637+0000", 2021-03-15T22:31:07.673 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5560000000000001e-06 2021-03-15T22:31:07.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.673 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.673 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.674 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.125642+0000", 2021-03-15T22:31:07.674 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053224839999999997 2021-03-15T22:31:07.674 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.674 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.674 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.675 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.130964+0000", 2021-03-15T22:31:07.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3690999999999996e-05 2021-03-15T22:31:07.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.675 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.676 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:31:07.676 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.131038+0000", 2021-03-15T22:31:07.676 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017722974999999998 2021-03-15T22:31:07.676 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.676 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.677 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.677 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.148761+0000", 2021-03-15T22:31:07.677 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5197000000000004e-05 2021-03-15T22:31:07.677 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.677 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.678 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:31:07.678 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.148856+0000", 2021-03-15T22:31:07.678 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012464467999999999 2021-03-15T22:31:07.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.679 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.679 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.679 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161321+0000", 2021-03-15T22:31:07.679 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3552000000000003e-05 2021-03-15T22:31:07.679 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.680 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.680 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.680 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161394+0000", 2021-03-15T22:31:07.680 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011150300000000001 2021-03-15T22:31:07.680 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.681 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.681 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:12.161506+0000", 2021-03-15T22:31:07.681 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035880710000000003 2021-03-15T22:31:07.681 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.682 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.682 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.682 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.682 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.683 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34361.0:611 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e551)", 2021-03-15T22:31:07.683 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:28:12.419483+0000", 2021-03-15T22:31:07.683 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 175.00892243000001, 2021-03-15T22:31:07.683 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.091628773999999996, 2021-03-15T22:31:07.683 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.684 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.684 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.684 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34361", 2021-03-15T22:31:07.684 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.125:0/1442040869", 2021-03-15T22:31:07.684 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 611 2021-03-15T22:31:07.685 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.685 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.685 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.685 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.685 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419483+0000", 2021-03-15T22:31:07.686 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.686 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.686 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.686 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.686 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419483+0000", 2021-03-15T22:31:07.687 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2217000000000001e-05 2021-03-15T22:31:07.687 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.687 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.687 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.688 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419505+0000", 2021-03-15T22:31:07.688 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9129999999999996e-06 2021-03-15T22:31:07.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.688 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.688 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.689 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419509+0000", 2021-03-15T22:31:07.689 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5500999999999999e-05 2021-03-15T22:31:07.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.689 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.689 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.690 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419525+0000", 2021-03-15T22:31:07.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3603000000000001e-05 2021-03-15T22:31:07.690 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419548+0000", 2021-03-15T22:31:07.691 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8461999999999996e-05 2021-03-15T22:31:07.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.692 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.692 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419617+0000", 2021-03-15T22:31:07.692 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.277e-05 2021-03-15T22:31:07.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.692 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.693 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:07.693 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419630+0000", 2021-03-15T22:31:07.693 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.091151858000000002 2021-03-15T22:31:07.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.694 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.694 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.510781+0000", 2021-03-15T22:31:07.694 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014667799999999999 2021-03-15T22:31:07.694 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.694 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.695 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.695 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.510928+0000", 2021-03-15T22:31:07.695 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018377199999999999 2021-03-15T22:31:07.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.695 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.696 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.696 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.511112+0000", 2021-03-15T22:31:07.696 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.091628773999999996 2021-03-15T22:31:07.696 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.697 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.697 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.697 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.697 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.697 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5830 7.5 7:ac84e656:::smithi07694878-31:head [copy-from ver 1320] snapc 1f0=[1f0,1ee,1e9,1e7,1e6,1e4,1e0,1df,1dc,1d5,1d4,1cc,1c7,1c5,1c3,1b6,1ae,1a6,1a4,1a1,19c,193,180,17a,176,15f,141] ondisk+write+known_if_redirected e631)", 2021-03-15T22:31:07.698 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:40.245985+0000", 2021-03-15T22:31:07.698 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 87.182420965000006, 2021-03-15T22:31:07.698 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.089874809, 2021-03-15T22:31:07.698 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.698 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:07.699 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.699 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.699 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.699 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5830 2021-03-15T22:31:07.699 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.700 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.245985+0000", 2021-03-15T22:31:07.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.701 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.701 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.701 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.701 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.245985+0000", 2021-03-15T22:31:07.702 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0769999999999998e-06 2021-03-15T22:31:07.702 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.702 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.702 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.702 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.245990+0000", 2021-03-15T22:31:07.703 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2939999999999999e-06 2021-03-15T22:31:07.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.703 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.703 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.703 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.245994+0000", 2021-03-15T22:31:07.704 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7839999999999999e-05 2021-03-15T22:31:07.704 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.704 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.704 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.705 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246012+0000", 2021-03-15T22:31:07.705 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2398000000000001e-05 2021-03-15T22:31:07.705 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.705 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.706 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246024+0000", 2021-03-15T22:31:07.706 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1279999999999997e-05 2021-03-15T22:31:07.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.706 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.706 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.707 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246075+0000", 2021-03-15T22:31:07.707 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3337000000000005e-05 2021-03-15T22:31:07.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.707 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.707 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.708 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246159+0000", 2021-03-15T22:31:07.708 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0051863150000000004 2021-03-15T22:31:07.708 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.708 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.708 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,1", 2021-03-15T22:31:07.709 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.251345+0000", 2021-03-15T22:31:07.709 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.054202781999999998 2021-03-15T22:31:07.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.709 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.710 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:07.710 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.305548+0000", 2021-03-15T22:31:07.710 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017903207000000001 2021-03-15T22:31:07.710 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.710 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.711 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.711 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.323451+0000", 2021-03-15T22:31:07.711 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012244708 2021-03-15T22:31:07.711 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.711 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.712 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:07.712 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.335696+0000", 2021-03-15T22:31:07.712 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6770000000000001e-05 2021-03-15T22:31:07.712 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.712 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.713 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:07.713 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.335743+0000", 2021-03-15T22:31:07.713 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000116801 2021-03-15T22:31:07.713 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.713 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.714 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.714 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.335859+0000", 2021-03-15T22:31:07.714 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.089874809 2021-03-15T22:31:07.714 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.715 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.715 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.715 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.715 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.715 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5831 7.5 7:ac84e656:::smithi07694878-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e631)", 2021-03-15T22:31:07.716 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:40.246040+0000", 2021-03-15T22:31:07.716 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 87.182365376000007, 2021-03-15T22:31:07.716 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.090216515999999997, 2021-03-15T22:31:07.716 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.716 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.717 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.717 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:07.717 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:07.717 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5831 2021-03-15T22:31:07.717 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.718 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.718 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.718 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.718 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246040+0000", 2021-03-15T22:31:07.719 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.719 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.719 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.719 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.719 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246040+0000", 2021-03-15T22:31:07.720 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1790000000000001e-06 2021-03-15T22:31:07.720 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.720 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.720 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.720 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246041+0000", 2021-03-15T22:31:07.721 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3319999999999999e-06 2021-03-15T22:31:07.721 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.721 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.721 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.721 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246043+0000", 2021-03-15T22:31:07.722 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6370000000000002e-06 2021-03-15T22:31:07.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.722 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.722 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.723 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246049+0000", 2021-03-15T22:31:07.723 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3510000000000004e-06 2021-03-15T22:31:07.723 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.723 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.723 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.724 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246054+0000", 2021-03-15T22:31:07.724 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024099399999999999 2021-03-15T22:31:07.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.724 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.724 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.725 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246295+0000", 2021-03-15T22:31:07.725 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7129999999999997e-05 2021-03-15T22:31:07.725 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.725 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.725 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:31:07.726 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246352+0000", 2021-03-15T22:31:07.726 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0096365749999999997 2021-03-15T22:31:07.726 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.726 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.727 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.727 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.255988+0000", 2021-03-15T22:31:07.727 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121428 2021-03-15T22:31:07.727 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.727 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.728 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:31:07.728 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.256110+0000", 2021-03-15T22:31:07.728 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.079801443 2021-03-15T22:31:07.728 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.728 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.729 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.729 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.335911+0000", 2021-03-15T22:31:07.729 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012181899999999999 2021-03-15T22:31:07.729 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.729 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.730 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.730 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.336033+0000", 2021-03-15T22:31:07.730 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022362799999999999 2021-03-15T22:31:07.730 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.731 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.731 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.731 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.336257+0000", 2021-03-15T22:31:07.731 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.090216515999999997 2021-03-15T22:31:07.731 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.732 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.732 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.732 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.732 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.732 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2102 5.d 5:b4812045:::notify.4:head [watch ping cookie 94751391775488 gen 9] snapc 0=[] ondisk+write+known_if_redirected e637)", 2021-03-15T22:31:07.733 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.130297+0000", 2021-03-15T22:31:07.733 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 75.298108927000001, 2021-03-15T22:31:07.733 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.098088540000000002, 2021-03-15T22:31:07.733 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.733 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.734 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.734 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:31:07.734 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:31:07.734 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2102 2021-03-15T22:31:07.734 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.735 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.735 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.735 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.735 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130297+0000", 2021-03-15T22:31:07.736 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.736 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.736 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.736 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.736 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130297+0000", 2021-03-15T22:31:07.737 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.739e-06 2021-03-15T22:31:07.737 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.737 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.737 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.737 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130298+0000", 2021-03-15T22:31:07.738 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6400000000000001e-06 2021-03-15T22:31:07.738 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.738 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.738 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.738 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130301+0000", 2021-03-15T22:31:07.739 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4290000000000007e-06 2021-03-15T22:31:07.739 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.739 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.739 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.739 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130310+0000", 2021-03-15T22:31:07.740 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0030000000000001e-05 2021-03-15T22:31:07.740 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.740 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.740 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.741 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130320+0000", 2021-03-15T22:31:07.741 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000322384 2021-03-15T22:31:07.741 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.741 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.741 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.742 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130643+0000", 2021-03-15T22:31:07.742 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5191000000000001e-05 2021-03-15T22:31:07.742 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.742 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.742 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:07.743 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130658+0000", 2021-03-15T22:31:07.743 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.097505270000000005 2021-03-15T22:31:07.743 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.743 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.743 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.744 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228163+0000", 2021-03-15T22:31:07.744 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014791500000000001 2021-03-15T22:31:07.744 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.744 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.745 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.745 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228311+0000", 2021-03-15T22:31:07.745 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3942000000000005e-05 2021-03-15T22:31:07.745 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.745 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.746 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.746 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228385+0000", 2021-03-15T22:31:07.746 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.098088540000000002 2021-03-15T22:31:07.746 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.746 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.747 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.747 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.747 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.747 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2108 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94751391775488 gen 10] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:31:07.747 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.130994+0000", 2021-03-15T22:31:07.748 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 75.297411314000001, 2021-03-15T22:31:07.748 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.100843209, 2021-03-15T22:31:07.748 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.748 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.748 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.749 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:31:07.749 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:31:07.749 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2108 2021-03-15T22:31:07.749 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.750 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.750 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.750 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.750 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130994+0000", 2021-03-15T22:31:07.750 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.751 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.751 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.751 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.751 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130994+0000", 2021-03-15T22:31:07.751 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.694e-06 2021-03-15T22:31:07.752 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.752 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.752 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.752 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130996+0000", 2021-03-15T22:31:07.752 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1129999999999999e-06 2021-03-15T22:31:07.753 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.753 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.753 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.753 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130998+0000", 2021-03-15T22:31:07.754 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4609999999999997e-06 2021-03-15T22:31:07.754 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.754 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.754 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.754 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131007+0000", 2021-03-15T22:31:07.755 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0089999999999998e-06 2021-03-15T22:31:07.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.755 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.755 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131012+0000", 2021-03-15T22:31:07.756 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4428e-05 2021-03-15T22:31:07.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.756 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.756 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131077+0000", 2021-03-15T22:31:07.757 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2669e-05 2021-03-15T22:31:07.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.757 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:07.757 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131090+0000", 2021-03-15T22:31:07.758 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.097317492000000005 2021-03-15T22:31:07.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.758 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.759 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228407+0000", 2021-03-15T22:31:07.759 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6538000000000006e-05 2021-03-15T22:31:07.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.759 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.760 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228474+0000", 2021-03-15T22:31:07.760 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033638050000000001 2021-03-15T22:31:07.760 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.760 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.760 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.761 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.231837+0000", 2021-03-15T22:31:07.761 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.100843209 2021-03-15T22:31:07.761 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.761 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.762 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.762 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.762 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.762 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2111 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94751391775488 gen 11] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:31:07.762 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.131034+0000", 2021-03-15T22:31:07.763 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 75.297371415000001, 2021-03-15T22:31:07.763 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.101203874, 2021-03-15T22:31:07.763 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:07.763 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:07.763 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:07.764 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:31:07.764 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:31:07.764 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2111 2021-03-15T22:31:07.764 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.765 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:07.765 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.765 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:07.765 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131034+0000", 2021-03-15T22:31:07.765 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:07.766 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.766 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.766 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:07.766 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131034+0000", 2021-03-15T22:31:07.766 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0070000000000001e-06 2021-03-15T22:31:07.767 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.767 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.767 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:07.767 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131035+0000", 2021-03-15T22:31:07.767 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.384e-06 2021-03-15T22:31:07.768 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.768 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.768 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:07.768 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131037+0000", 2021-03-15T22:31:07.768 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1930000000000004e-06 2021-03-15T22:31:07.769 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.769 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.769 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:07.769 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131044+0000", 2021-03-15T22:31:07.770 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3359999999999997e-06 2021-03-15T22:31:07.770 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.770 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.770 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:07.770 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131048+0000", 2021-03-15T22:31:07.771 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2633999999999996e-05 2021-03-15T22:31:07.771 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.771 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.771 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.771 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131121+0000", 2021-03-15T22:31:07.772 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4610000000000002e-06 2021-03-15T22:31:07.772 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.772 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.772 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:07.772 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131128+0000", 2021-03-15T22:31:07.773 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.097620176000000003 2021-03-15T22:31:07.773 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.773 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.773 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:07.773 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228748+0000", 2021-03-15T22:31:07.774 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0249999999999999e-05 2021-03-15T22:31:07.774 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.774 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.774 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:07.775 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228829+0000", 2021-03-15T22:31:07.775 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0034094329999999999 2021-03-15T22:31:07.775 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:07.775 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:07.775 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:07.776 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.232238+0000", 2021-03-15T22:31:07.776 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.101203874 2021-03-15T22:31:07.776 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.776 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.776 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.777 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:07.777 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:07.777 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:31:07.777 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:07.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[56514]: cluster 2021-03-15T22:31:05.998557+0000 osd.2 (osd.2) 9 : cluster [DBG] 5.d scrub ok 2021-03-15T22:31:07.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[56514]: cluster 2021-03-15T22:31:06.296658+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:31:07.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[56514]: cluster 2021-03-15T22:31:06.530062+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:31:07.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[56514]: cluster 2021-03-15T22:31:07.237399+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2021-03-15T22:31:07.780 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:07 smithi125 conmon[56739]: cluster 2021-03-15T22:31:05.998557+0000 osd.2 (osd.2) 9 : cluster 2021-03-15T22:31:07.780 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:07 smithi125 conmon[56739]: [DBG] 5.d scrub ok 2021-03-15T22:31:07.780 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:07 smithi125 conmon[56739]: cluster 2021-03-15T22:31:06.296658+0000 osd.6 (osd.6) 114 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:31:07.781 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:07 smithi125 conmon[56739]: cluster 2021-03-15T22:31:06.530062+0000 osd.7 (osd.7) 136 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:31:07.781 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:07 smithi125 conmon[56739]: cluster 2021-03-15T22:31:07.237399+0000 mon.a (mon.0) 1920 : cluster [DBG] osdmap e718: 8 total, 8 up, 8 in 2021-03-15T22:31:07.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:31:07.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: 31:05.998557+0000 osd.2 (osd.2) 9 : cluster [DBG] 5.d scrub ok 2021-03-15T22:31:07.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:31:07.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: -03-15T22:31: 2021-03-15T22:31:07.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: 06.296658+0000 osd.6 (osd.6) 114 : 2021-03-15T22:31:07.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: cluster [DBG] 5.11 scrub ok 2021-03-15T22:31:07.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: cluster 2021-03-15T22:31:07.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:31:07.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: 31:06.530062+0000 osd.7 2021-03-15T22:31:07.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: (osd.7) 136 : cluster 2021-03-15T22:31:07.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: [DBG] 6.3 scrub ok 2021-03-15T22:31:07.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: cluster 2021-03-15T22:31:07.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:31:07.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: 31:07.237399+0000 2021-03-15T22:31:07.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: mon.a (mon.0) 1920 : cluster 2021-03-15T22:31:07.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:07 smithi076 conmon[62156]: [DBG] osdmap e718: 8 total, 8 up, 8 in 2021-03-15T22:31:07.785 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:31:07.786 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:31:07.786 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:31:07.786 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:31:07.786 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:31:07.802 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:07.803 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:31:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:07.804 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:07.821 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:31:07.839 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:31:07.840 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:31:07.840 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:31:07.840 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:31:07.841 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:31:07.841 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:31:07.841 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:31:07.841 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:31:07.841 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:31:07.905 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:07.931 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:07.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:08.032 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:08.054 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:08.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:08.056 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:08.093 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:31:08.158 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:08.198 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:08.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:08.200 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:08.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:31:08.242 INFO:tasks.rados.rados.0.smithi076.stdout:2626: done (1 left) 2021-03-15T22:31:08.242 INFO:tasks.rados.rados.0.smithi076.stdout:2629: done (0 left) 2021-03-15T22:31:08.242 INFO:tasks.rados.rados.0.smithi076.stdout:2630: copy_from oid 20 from oid 37 current snap is 306 2021-03-15T22:31:08.243 INFO:tasks.rados.rados.0.smithi076.stdout:2631: rollback oid 25 current snap is 306 2021-03-15T22:31:08.243 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 277 2021-03-15T22:31:08.243 INFO:tasks.rados.rados.0.smithi076.stdout:2632: rollback oid 28 current snap is 306 2021-03-15T22:31:08.243 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 277 2021-03-15T22:31:08.243 INFO:tasks.rados.rados.0.smithi076.stdout:2633: read oid 14 snap -1 2021-03-15T22:31:08.244 INFO:tasks.rados.rados.0.smithi076.stdout:2633: expect (ObjNum 623 snap 193 seq_num 623) 2021-03-15T22:31:08.244 INFO:tasks.rados.rados.0.smithi076.stdout:2634: read oid 41 snap -1 2021-03-15T22:31:08.244 INFO:tasks.rados.rados.0.smithi076.stdout:2634: expect (ObjNum 872 snap 287 seq_num 872) 2021-03-15T22:31:08.244 INFO:tasks.rados.rados.0.smithi076.stdout:2635: snap_create 2021-03-15T22:31:08.245 INFO:tasks.rados.rados.0.smithi076.stdout:2630: finishing copy_from to smithi07694878-20 2021-03-15T22:31:08.246 INFO:tasks.rados.rados.0.smithi076.stdout:2630: got expected ENOENT (src dne) 2021-03-15T22:31:08.246 INFO:tasks.rados.rados.0.smithi076.stdout:2630: finishing copy_from racing read to smithi07694878-20 2021-03-15T22:31:08.247 INFO:tasks.rados.rados.0.smithi076.stdout:2632: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:31:08.248 INFO:tasks.rados.rados.0.smithi076.stdout:2632: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:31:08.248 INFO:tasks.rados.rados.0.smithi076.stdout:2632: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:31:08.248 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2161 (ObjNum 609 snap 189 seq_num 609) dirty exists 2021-03-15T22:31:08.251 INFO:tasks.rados.rados.0.smithi076.stdout:2633: expect (ObjNum 623 snap 193 seq_num 623) 2021-03-15T22:31:08.301 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:08.323 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:08.325 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:08.366 INFO:tasks.rados.rados.0.smithi076.stdout:2634: expect (ObjNum 872 snap 287 seq_num 872) 2021-03-15T22:31:08.416 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:31:08.426 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:08.448 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:08.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:08.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[56514]: cluster 2021-03-15T22:31:07.155125+0000 mgr.y (mgr. 2021-03-15T22:31:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[56514]: 14144) 877 : cluster [DBG] pgmap v1220: 138 pgs: 138 active+clean; 557 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 10 op/s 2021-03-15T22:31:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[56514]: cluster 2021-03-15T22:31:07.330956+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:31:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[56514]: cluster 2021-03-15T22:31:07.417731+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.a scrub ok 2021-03-15T22:31:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[56514]: audit 2021-03-15T22:31:07.697895+0000 mon.a (mon.0) 1921 : audit [DBG] from='client.? 172.21.15.76:0/2642776874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[56514]: cluster 2021-03-15T22:31:08.238369+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2021-03-15T22:31:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: cluster 2021-03-15T22:31:07 2021-03-15T22:31:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: .155125+0000 mgr.y (mgr.14144) 877 : cluster [DBG] pgmap v1220: 138 pgs: 138 active+clean; 557 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 10 op/s 2021-03-15T22:31:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: cluster 2021-03-15T22:31:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: 2021-03-15T22:31:07. 2021-03-15T22:31:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: 330956+0000 osd.3 (osd.3) 2021-03-15T22:31:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: 130 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:31:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:31:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: -03-15T22:31:07. 2021-03-15T22:31:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: 417731+0000 osd.5 (osd.5) 2021-03-15T22:31:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: 19 : cluster [DBG] 6.a scrub ok 2021-03-15T22:31:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: audit 2021 2021-03-15T22:31:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: -03-15T22:31:07 2021-03-15T22:31:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: .697895+0000 mon.a (mon. 2021-03-15T22:31:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: 0) 1921 : audit [DBG] from='client.? 172.21.15.76:0/2642776874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: 2021-03-15T22:31:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:31:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: -03-15T22:31: 2021-03-15T22:31:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: 08.238369+0000 mon.a (mon 2021-03-15T22:31:08.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: .0) 1922 : cluster [DBG] 2021-03-15T22:31:08.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:08 smithi076 conmon[62156]: osdmap e719: 8 total, 8 up, 8 in 2021-03-15T22:31:08.524 INFO:tasks.rados.rados.0.smithi076.stdout:2631: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:31:08.524 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 1749 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:31:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:08 smithi125 conmon[56739]: cluster 2021-03-15T22:31:07.155125+0000 mgr.y (mgr.14144) 877 : cluster [DBG] pgmap v1220: 138 pgs: 138 active+clean; 557 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1.5 MiB/s rd, 2.1 MiB/s wr, 10 op/s 2021-03-15T22:31:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:08 smithi125 conmon[56739]: cluster 2021-03-15T22:31:07.330956+0000 osd.3 (osd.3) 130 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:31:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:08 smithi125 conmon[56739]: cluster 2021-03-15T22:31:07.417731+0000 osd.5 (osd.5) 19 : cluster [DBG] 6.a scrub ok 2021-03-15T22:31:08.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:08 smithi125 conmon[56739]: audit 2021-03-15T22:31:07.697895+0000 mon.a (mon.0) 1921 : audit [DBG] from='client.? 172.21.15.76:0/2642776874' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:08.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:08 smithi125 conmon[56739]: cluster 2021-03-15T22:31:08.238369+0000 mon.a (mon.0) 1922 : cluster [DBG] osdmap e719: 8 total, 8 up, 8 in 2021-03-15T22:31:08.551 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:08.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:08.576 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:08.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:08.677 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:08.703 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:08.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:08.705 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:08.806 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:08.829 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:08.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:08.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:08.932 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:08.955 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:08.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:08.956 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:09.010 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:09.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:09.080 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:09.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:09.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:09.183 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:09.206 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:09.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:09.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:09.246 INFO:tasks.rados.rados.0.smithi076.stdout:2630: done (5 left) 2021-03-15T22:31:09.246 INFO:tasks.rados.rados.0.smithi076.stdout:2631: done (4 left) 2021-03-15T22:31:09.246 INFO:tasks.rados.rados.0.smithi076.stdout:2632: done (3 left) 2021-03-15T22:31:09.247 INFO:tasks.rados.rados.0.smithi076.stdout:2633: done (2 left) 2021-03-15T22:31:09.247 INFO:tasks.rados.rados.0.smithi076.stdout:2634: done (1 left) 2021-03-15T22:31:09.248 INFO:tasks.rados.rados.0.smithi076.stdout:2635: done (0 left) 2021-03-15T22:31:09.248 INFO:tasks.rados.rados.0.smithi076.stdout:2636: read oid 20 snap -1 2021-03-15T22:31:09.248 INFO:tasks.rados.rados.0.smithi076.stdout:2636: expect (ObjNum 874 snap 287 seq_num 874) 2021-03-15T22:31:09.249 INFO:tasks.rados.rados.0.smithi076.stdout:2637: delete oid 9 current snap is 307 2021-03-15T22:31:09.250 INFO:tasks.rados.rados.0.smithi076.stdout:2636: expect (ObjNum 874 snap 287 seq_num 874) 2021-03-15T22:31:09.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:09.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:09.335 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:09.336 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:09.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:09.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:09.461 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:09.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:09.497 INFO:tasks.rados.rados.0.smithi076.stdout:2636: done (1 left) 2021-03-15T22:31:09.497 INFO:tasks.rados.rados.0.smithi076.stdout:2637: done (0 left) 2021-03-15T22:31:09.497 INFO:tasks.rados.rados.0.smithi076.stdout:2638: write (excl) oid 48 current snap is 307 2021-03-15T22:31:09.498 INFO:tasks.rados.rados.0.smithi076.stdout:2638: seq_num 911 ranges {511177=797846,1976390=591639,3034464=276713} 2021-03-15T22:31:09.506 INFO:tasks.rados.rados.0.smithi076.stdout:2638: writing smithi07694878-48 from 511177 to 1309023 tid 1 2021-03-15T22:31:09.514 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:31:09.515 INFO:tasks.rados.rados.0.smithi076.stdout:2638: writing smithi07694878-48 from 1976390 to 2568029 tid 2 2021-03-15T22:31:09.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[56514]: cluster 2021-03-15T22:31:08.134634+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:31:09.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[56514]: cluster 2021-03-15T22:31:08.317808+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.13 deep-scrub ok 2021-03-15T22:31:09.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[56514]: cluster 2021-03-15T22:31:09.241859+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2021-03-15T22:31:09.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[62156]: cluster 2021-03-15T22:31:09.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[62156]: 2021-03-15T22:31:09.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[62156]: 2021-03-15T22:31:08.134634 2021-03-15T22:31:09.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 139 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:31:09.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:31:09.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[62156]: 31:08.317808+0000 osd.3 (osd.3) 131 : cluster [DBG] 2021-03-15T22:31:09.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[62156]: 6.13 deep-scrub ok 2021-03-15T22:31:09.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[62156]: cluster 2021-03-15T22:31:09.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[62156]: 2021-03-15T22:31:09.241859+0000 mon.a (mon.0) 1923 : 2021-03-15T22:31:09.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:09 smithi076 conmon[62156]: cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2021-03-15T22:31:09.526 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:31:09.528 INFO:tasks.rados.rados.0.smithi076.stdout:2638: writing smithi07694878-48 from 3034464 to 3311177 tid 3 2021-03-15T22:31:09.528 INFO:tasks.rados.rados.0.smithi076.stdout:2639: rollback oid 38 current snap is 307 2021-03-15T22:31:09.528 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 38 to 253 2021-03-15T22:31:09.528 INFO:tasks.rados.rados.0.smithi076.stdout:2638: finishing write tid 1 to smithi07694878-48 2021-03-15T22:31:09.529 INFO:tasks.rados.rados.0.smithi076.stdout:2638: finishing write tid 2 to smithi07694878-48 2021-03-15T22:31:09.529 INFO:tasks.rados.rados.0.smithi076.stdout:2640: delete oid 39 current snap is 307 2021-03-15T22:31:09.529 INFO:tasks.rados.rados.0.smithi076.stdout:2638: finishing write tid 3 to smithi07694878-48 2021-03-15T22:31:09.529 INFO:tasks.rados.rados.0.smithi076.stdout:2638: finishing write tid 4 to smithi07694878-48 2021-03-15T22:31:09.529 INFO:tasks.rados.rados.0.smithi076.stdout:2638: finishing write tid 5 to smithi07694878-48 2021-03-15T22:31:09.530 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 1224 (ObjNum 911 snap 307 seq_num 911) dirty exists 2021-03-15T22:31:09.530 INFO:tasks.rados.rados.0.smithi076.stdout:2638: left oid 48 (ObjNum 911 snap 307 seq_num 911) 2021-03-15T22:31:09.530 INFO:tasks.rados.rados.0.smithi076.stdout:2639: finishing rollback tid 0 to smithi07694878-38 2021-03-15T22:31:09.530 INFO:tasks.rados.rados.0.smithi076.stdout:2639: finishing rollback tid 1 to smithi07694878-38 2021-03-15T22:31:09.531 INFO:tasks.rados.rados.0.smithi076.stdout:2639: finishing rollback tid 2 to smithi07694878-38 2021-03-15T22:31:09.531 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2071 (ObjNum 732 snap 239 seq_num 732) dirty exists 2021-03-15T22:31:09.531 INFO:tasks.rados.rados.0.smithi076.stdout:2638: done (2 left) 2021-03-15T22:31:09.531 INFO:tasks.rados.rados.0.smithi076.stdout:2639: done (1 left) 2021-03-15T22:31:09.532 INFO:tasks.rados.rados.0.smithi076.stdout:2640: done (0 left) 2021-03-15T22:31:09.532 INFO:tasks.rados.rados.0.smithi076.stdout:2641: snap_remove snap 277 2021-03-15T22:31:09.563 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:09.585 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:09.587 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:09.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:09.711 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:09.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:09.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:09.745 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:09 smithi125 conmon[56739]: cluster 2021-03-15T22:31:08.134634+0000 osd.1 (osd.1) 139 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:31:09.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:09 smithi125 conmon[56739]: cluster 2021-03-15T22:31:08.317808+0000 osd.3 (osd.3) 131 : cluster [DBG] 6.13 deep-scrub ok 2021-03-15T22:31:09.746 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:09 smithi125 conmon[56739]: cluster 2021-03-15T22:31:09.241859+0000 mon.a (mon.0) 1923 : cluster [DBG] osdmap e720: 8 total, 8 up, 8 in 2021-03-15T22:31:09.815 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:09.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:09.839 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:09.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:09.941 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:09.967 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:09.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:10.014 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:31:10.015 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:31:10.069 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:10.092 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:10.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:10.093 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:10.194 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:10.220 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:10.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:10.233 INFO:tasks.rados.rados.0.smithi076.stdout:2641: done (0 left) 2021-03-15T22:31:10.233 INFO:tasks.rados.rados.0.smithi076.stdout:2642: delete oid 40 current snap is 307 2021-03-15T22:31:10.247 INFO:tasks.rados.rados.0.smithi076.stdout:2642: done (0 left) 2021-03-15T22:31:10.248 INFO:tasks.rados.rados.0.smithi076.stdout:2643: copy_from oid 16 from oid 30 current snap is 307 2021-03-15T22:31:10.248 INFO:tasks.rados.rados.0.smithi076.stdout:2644: delete oid 22 current snap is 307 2021-03-15T22:31:10.262 INFO:tasks.rados.rados.0.smithi076.stdout:2644: done (1 left) 2021-03-15T22:31:10.262 INFO:tasks.rados.rados.0.smithi076.stdout:2645: snap_create 2021-03-15T22:31:10.275 INFO:tasks.rados.rados.0.smithi076.stdout:2643: finishing copy_from to smithi07694878-16 2021-03-15T22:31:10.276 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1324 (ObjNum 803 snap 268 seq_num 803) dirty exists 2021-03-15T22:31:10.276 INFO:tasks.rados.rados.0.smithi076.stdout:2643: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:31:10.322 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:10.345 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:10.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:10.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:10.400 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:31:10.448 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:10.470 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:10.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:10.471 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:10.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:10.920 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[56514]: cluster 2021-03-15T22:31:08.968591+0000 osd.0 (osd.0) 2021-03-15T22:31:10.920 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[56514]: 115 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:31:10.920 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[56514]: cluster 2021-03-15T22:31:09.155590+0000 mgr.y (mgr.14144) 878 : cluster [DBG] pgmap v1223: 138 pgs: 138 active+clean; 565 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2021-03-15T22:31:10.921 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[56514]: audit 2021-03-15T22:31:09.500837+0000 mgr.y (mgr.14144) 879 : audit [DBG] from='client.25279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:31:10.921 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[56514]: audit 2021-03-15T22:31:10.041838+0000 mgr.y (mgr.14144) 880 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:10.921 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[56514]: cluster 2021-03-15T22:31:10.227650+0000 mon.a (mon.0) 1924 : cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2021-03-15T22:31:10.922 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:31:10.923 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: 31:08.968591+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:31:10.923 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:31:10.924 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: -03-15T22:31:09 2021-03-15T22:31:10.924 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: .155590+0000 mgr.y (mgr. 2021-03-15T22:31:10.925 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: 14144) 878 : cluster [DBG] 2021-03-15T22:31:10.926 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: pgmap v1223: 138 pgs: 138 active+clean; 565 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2021-03-15T22:31:10.929 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: audit 2021-03-15T22:31:10.929 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:31:10.930 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: 31:09.500837+0000 mgr.y 2021-03-15T22:31:10.930 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: (mgr.14144) 879 : audit 2021-03-15T22:31:10.930 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: [DBG] from='client.25279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:31:10.930 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: audit 2021 2021-03-15T22:31:10.931 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: -03-15T22:31: 2021-03-15T22:31:10.932 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: 10.041838+0000 mgr.y (mgr 2021-03-15T22:31:10.932 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: .14144) 880 : audit [DBG] 2021-03-15T22:31:10.932 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:10.932 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: cluster 2021-03-15T22:31:10.933 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:31:10.933 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: :31:10.227650+0000 2021-03-15T22:31:10.933 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: mon.a (mon.0) 1924 : 2021-03-15T22:31:10.933 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:10 smithi076 conmon[62156]: cluster [DBG] osdmap e721: 8 total, 8 up, 8 in 2021-03-15T22:31:10.934 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1224,"stamp":"2021-03-15T22:31:09.239705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600171108,"num_objects":604,"num_object_clones":238,"num_object_copies":1812,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":604,"num_whiteouts":13,"num_read":7274,"num_read_kb":5110865,"num_write":4495,"num_write_kb":1385841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2862,"num_bytes_recovered":1078946714,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14576,"ondisk_log_size":14576,"up":414,"acting":414,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":414,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1662548,"kb_used_data":1353684,"kb_used_omap":273,"kb_used_meta":308526,"kb_avail":748298668,"statfs":{"total":767960285184,"available":766257836032,"internally_reserved":0,"allocated":1386172416,"data_stored":1941837333,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":315931514},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":23542407,"num_objects":8,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":-1,"num_read":29,"num_read_kb":17753,"num_write":34,"num_write_kb":16030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.015143"},"pg_stats":[{"pgid":"6.1a","version":"0'0","reported_seq":"718","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229000+0000","last_change":"2021-03-15T22:30:41.431150+0000","last_active":"2021-03-15T22:31:05.229000+0000","last_peered":"2021-03-15T22:31:05.229000+0000","last_clean":"2021-03-15T22:31:05.229000+0000","last_became_active":"2021-03-15T22:29:51.225252+0000","last_became_peered":"2021-03-15T22:29:51.225252+0000","last_unstale":"2021-03-15T22:31:05.229000+0000","last_undegraded":"2021-03-15T22:31:05.229000+0000","last_fullsized":"2021-03-15T22:31:05.229000+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.431052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"215","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.280217+0000","last_change":"2021-03-15T22:30:32.251958+0000","last_active":"2021-03-15T22:31:04.280217+0000","last_peered":"2021-03-15T22:31:04.280217+0000","last_clean":"2021-03-15T22:31:04.280217+0000","last_became_active":"2021-03-15T22:30:32.251610+0000","last_became_peered":"2021-03-15T22:30:32.251610+0000","last_unstale":"2021-03-15T22:31:04.280217+0000","last_undegraded":"2021-03-15T22:31:04.280217+0000","last_fullsized":"2021-03-15T22:31:04.280217+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:30:03.560125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.560125+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"554'21","reported_seq":"733","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229131+0000","last_change":"2021-03-15T22:30:01.066537+0000","last_active":"2021-03-15T22:31:05.229131+0000","last_peered":"2021-03-15T22:31:05.229131+0000","last_clean":"2021-03-15T22:31:05.229131+0000","last_became_active":"2021-03-15T22:17:55.997546+0000","last_became_peered":"2021-03-15T22:17:55.997546+0000","last_unstale":"2021-03-15T22:31:05.229131+0000","last_undegraded":"2021-03-15T22:31:05.229131+0000","last_fullsized":"2021-03-15T22:31:05.229131+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:30:01.066454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:01.066454+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"142","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.262158+0000","last_change":"2021-03-15T22:31:05.262111+0000","last_active":"2021-03-15T22:31:05.262158+0000","last_peered":"2021-03-15T22:31:05.262158+0000","last_clean":"2021-03-15T22:31:05.262158+0000","last_became_active":"2021-03-15T22:28:59.271412+0000","last_became_peered":"2021-03-15T22:28:59.271412+0000","last_unstale":"2021-03-15T22:31:05.262158+0000","last_undegraded":"2021-03-15T22:31:05.262158+0000","last_fullsized":"2021-03-15T22:31:05.262158+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:05.262019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.262019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"424","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.231375+0000","last_change":"2021-03-15T22:30:32.259108+0000","last_active":"2021-03-15T22:31:05.231375+0000","last_peered":"2021-03-15T22:31:05.231375+0000","last_clean":"2021-03-15T22:31:05.231375+0000","last_became_active":"2021-03-15T22:30:32.258785+0000","last_became_peered":"2021-03-15T22:30:32.258785+0000","last_unstale":"2021-03-15T22:31:05.231375+0000","last_undegraded":"2021-03-15T22:31:05.231375+0000","last_fullsized":"2021-03-15T22:31:05.231375+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:09.184802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:09.184802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"556'32","reported_seq":"434","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.230318+0000","last_change":"2021-03-15T22:30:32.296239+0000","last_active":"2021-03-15T22:31:05.230318+0000","last_peered":"2021-03-15T22:31:05.230318+0000","last_clean":"2021-03-15T22:31:05.230318+0000","last_became_active":"2021-03-15T22:30:32.258079+0000","last_became_peered":"2021-03-15T22:30:32.258079+0000","last_unstale":"2021-03-15T22:31:05.230318+0000","last_undegraded":"2021-03-15T22:31:05.230318+0000","last_fullsized":"2021-03-15T22:31:05.230318+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'32","last_scrub_stamp":"2021-03-15T22:29:32.512485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:32.512485+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"178","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.279440+0000","last_change":"2021-03-15T22:29:18.753212+0000","last_active":"2021-03-15T22:31:04.279440+0000","last_peered":"2021-03-15T22:31:04.279440+0000","last_clean":"2021-03-15T22:31:04.279440+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:31:04.279440+0000","last_undegraded":"2021-03-15T22:31:04.279440+0000","last_fullsized":"2021-03-15T22:31:04.279440+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:18.753152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:29:18.753152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"92","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239859+0000","last_change":"2021-03-15T22:30:37.146490+0000","last_active":"2021-03-15T22:31:07.239859+0000","last_peered":"2021-03-15T22:31:07.239859+0000","last_clean":"2021-03-15T22:31:07.239859+0000","last_became_active":"2021-03-15T22:29:51.226784+0000","last_became_peered":"2021-03-15T22:29:51.226784+0000","last_unstale":"2021-03-15T22:31:07.239859+0000","last_undegraded":"2021-03-15T22:31:07.239859+0000","last_fullsized":"2021-03-15T22:31:07.239859+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.146408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.146408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"861","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229131+0000","last_change":"2021-03-15T22:30:28.328526+0000","last_active":"2021-03-15T22:31:05.229131+0000","last_peered":"2021-03-15T22:31:05.229131+0000","last_clean":"2021-03-15T22:31:05.229131+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:31:05.229131+0000","last_undegraded":"2021-03-15T22:31:05.229131+0000","last_fullsized":"2021-03-15T22:31:05.229131+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:30:28.328458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.328458+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"337","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239362+0000","last_change":"2021-03-15T22:30:40.173603+0000","last_active":"2021-03-15T22:31:07.239362+0000","last_peered":"2021-03-15T22:31:07.239362+0000","last_clean":"2021-03-15T22:31:07.239362+0000","last_became_active":"2021-03-15T22:29:52.223704+0000","last_became_peered":"2021-03-15T22:29:52.223704+0000","last_unstale":"2021-03-15T22:31:07.239362+0000","last_undegraded":"2021-03-15T22:31:07.239362+0000","last_fullsized":"2021-03-15T22:31:07.239362+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:40.173519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:30:40.173519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"92","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239544+0000","last_change":"2021-03-15T22:30:02.813811+0000","last_active":"2021-03-15T22:31:07.239544+0000","last_peered":"2021-03-15T22:31:07.239544+0000","last_clean":"2021-03-15T22:31:07.239544+0000","last_became_active":"2021-03-15T22:29:51.234868+0000","last_became_peered":"2021-03-15T22:29:51.234868+0000","last_unstale":"2021-03-15T22:31:07.239544+0000","last_undegraded":"2021-03-15T22:31:07.239544+0000","last_fullsized":"2021-03-15T22:31:07.239544+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:02.813745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.813745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"799","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229502+0000","last_change":"2021-03-15T22:29:56.950563+0000","last_active":"2021-03-15T22:31:05.229502+0000","last_peered":"2021-03-15T22:31:05.229502+0000","last_clean":"2021-03-15T22:31:05.229502+0000","last_became_active":"2021-03-15T22:29:11.197059+0000","last_became_peered":"2021-03-15T22:29:11.197059+0000","last_unstale":"2021-03-15T22:31:05.229502+0000","last_undegraded":"2021-03-15T22:31:05.229502+0000","last_fullsized":"2021-03-15T22:31:05.229502+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:29:56.950494+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.950494+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"711","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229400+0000","last_change":"2021-03-15T22:29:51.225493+0000","last_active":"2021-03-15T22:31:05.229400+0000","last_peered":"2021-03-15T22:31:05.229400+0000","last_clean":"2021-03-15T22:31:05.229400+0000","last_became_active":"2021-03-15T22:29:51.225147+0000","last_became_peered":"2021-03-15T22:29:51.225147+0000","last_unstale":"2021-03-15T22:31:05.229400+0000","last_undegraded":"2021-03-15T22:31:05.229400+0000","last_fullsized":"2021-03-15T22:31:05.229400+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:41.941510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.941510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"182","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.280049+0000","last_change":"2021-03-15T22:30:38.781792+0000","last_active":"2021-03-15T22:31:04.280049+0000","last_peered":"2021-03-15T22:31:04.280049+0000","last_clean":"2021-03-15T22:31:04.280049+0000","last_became_active":"2021-03-15T22:28:12.519619+0000","last_became_peered":"2021-03-15T22:28:12.519619+0000","last_unstale":"2021-03-15T22:31:04.280049+0000","last_undegraded":"2021-03-15T22:31:04.280049+0000","last_fullsized":"2021-03-15T22:31:04.280049+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:30:38.781724+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.781724+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"582","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.233777+0000","last_change":"2021-03-15T22:30:32.299886+0000","last_active":"2021-03-15T22:31:03.233777+0000","last_peered":"2021-03-15T22:31:03.233777+0000","last_clean":"2021-03-15T22:31:03.233777+0000","last_became_active":"2021-03-15T22:30:32.250733+0000","last_became_peered":"2021-03-15T22:30:32.250733+0000","last_unstale":"2021-03-15T22:31:03.233777+0000","last_undegraded":"2021-03-15T22:31:03.233777+0000","last_fullsized":"2021-03-15T22:31:03.233777+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:29:57.169300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:57.169300+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"528","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.230079+0000","last_change":"2021-03-15T22:29:51.225542+0000","last_active":"2021-03-15T22:31:05.230079+0000","last_peered":"2021-03-15T22:31:05.230079+0000","last_clean":"2021-03-15T22:31:05.230079+0000","last_became_active":"2021-03-15T22:29:51.225288+0000","last_became_peered":"2021-03-15T22:29:51.225288+0000","last_unstale":"2021-03-15T22:31:05.230079+0000","last_undegraded":"2021-03-15T22:31:05.230079+0000","last_fullsized":"2021-03-15T22:31:05.230079+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:38.513993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:45.403769+0000","last_clean_scrub_stamp":"2021-03-15T22:29:38.513993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"706","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229151+0000","last_change":"2021-03-15T22:30:39.443682+0000","last_active":"2021-03-15T22:31:05.229151+0000","last_peered":"2021-03-15T22:31:05.229151+0000","last_clean":"2021-03-15T22:31:05.229151+0000","last_became_active":"2021-03-15T22:30:31.243315+0000","last_became_peered":"2021-03-15T22:30:31.243315+0000","last_unstale":"2021-03-15T22:31:05.229151+0000","last_undegraded":"2021-03-15T22:31:05.229151+0000","last_fullsized":"2021-03-15T22:31:05.229151+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.443613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"162","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229400+0000","last_change":"2021-03-15T22:29:54.564185+0000","last_active":"2021-03-15T22:31:05.229400+0000","last_peered":"2021-03-15T22:31:05.229400+0000","last_clean":"2021-03-15T22:31:05.229400+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:31:05.229400+0000","last_undegraded":"2021-03-15T22:31:05.229400+0000","last_fullsized":"2021-03-15T22:31:05.229400+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:29:54.564115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:54.564115+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"721","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.374772+0000","last_change":"2021-03-15T22:30:32.243983+0000","last_active":"2021-03-15T22:31:02.374772+0000","last_peered":"2021-03-15T22:31:02.374772+0000","last_clean":"2021-03-15T22:31:02.374772+0000","last_became_active":"2021-03-15T22:30:32.243708+0000","last_became_peered":"2021-03-15T22:30:32.243708+0000","last_unstale":"2021-03-15T22:31:02.374772+0000","last_undegraded":"2021-03-15T22:31:02.374772+0000","last_fullsized":"2021-03-15T22:31:02.374772+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:01.208847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:01.208847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"741","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.240914+0000","last_change":"2021-03-15T22:31:05.123509+0000","last_active":"2021-03-15T22:31:06.240914+0000","last_peered":"2021-03-15T22:31:06.240914+0000","last_clean":"2021-03-15T22:31:06.240914+0000","last_became_active":"2021-03-15T22:28:59.264559+0000","last_became_peered":"2021-03-15T22:28:59.264559+0000","last_unstale":"2021-03-15T22:31:06.240914+0000","last_undegraded":"2021-03-15T22:31:06.240914+0000","last_fullsized":"2021-03-15T22:31:06.240914+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:05.123425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.123425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"283","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.280493+0000","last_change":"2021-03-15T22:30:41.759292+0000","last_active":"2021-03-15T22:31:04.280493+0000","last_peered":"2021-03-15T22:31:04.280493+0000","last_clean":"2021-03-15T22:31:04.280493+0000","last_became_active":"2021-03-15T22:28:59.277385+0000","last_became_peered":"2021-03-15T22:28:59.277385+0000","last_unstale":"2021-03-15T22:31:04.280493+0000","last_undegraded":"2021-03-15T22:31:04.280493+0000","last_fullsized":"2021-03-15T22:31:04.280493+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:30:41.759224+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.759224+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"714","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.242376+0000","last_change":"2021-03-15T22:29:52.230266+0000","last_active":"2021-03-15T22:31:06.242376+0000","last_peered":"2021-03-15T22:31:06.242376+0000","last_clean":"2021-03-15T22:31:06.242376+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:31:06.242376+0000","last_undegraded":"2021-03-15T22:31:06.242376+0000","last_fullsized":"2021-03-15T22:31:06.242376+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:34.234643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:34.234643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"316","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.230572+0000","last_change":"2021-03-15T22:29:00.220417+0000","last_active":"2021-03-15T22:31:05.230572+0000","last_peered":"2021-03-15T22:31:05.230572+0000","last_clean":"2021-03-15T22:31:05.230572+0000","last_became_active":"2021-03-15T22:29:00.220073+0000","last_became_peered":"2021-03-15T22:29:00.220073+0000","last_unstale":"2021-03-15T22:31:05.230572+0000","last_undegraded":"2021-03-15T22:31:05.230572+0000","last_fullsized":"2021-03-15T22:31:05.230572+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:34.099140+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:28:34.099140+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"806","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.272983+0000","last_change":"2021-03-15T22:31:03.281407+0000","last_active":"2021-03-15T22:31:04.272983+0000","last_peered":"2021-03-15T22:31:04.272983+0000","last_clean":"2021-03-15T22:31:04.272983+0000","last_became_active":"2021-03-15T22:29:51.232710+0000","last_became_peered":"2021-03-15T22:29:51.232710+0000","last_unstale":"2021-03-15T22:31:04.272983+0000","last_undegraded":"2021-03-15T22:31:04.272983+0000","last_fullsized":"2021-03-15T22:31:04.272983+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:31:03.281344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:03.281344+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"690","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.374038+0000","last_change":"2021-03-15T22:30:59.480779+0000","last_active":"2021-03-15T22:31:02.374038+0000","last_peered":"2021-03-15T22:31:02.374038+0000","last_clean":"2021-03-15T22:31:02.374038+0000","last_became_active":"2021-03-15T22:30:31.230649+0000","last_became_peered":"2021-03-15T22:30:31.230649+0000","last_unstale":"2021-03-15T22:31:02.374038+0000","last_undegraded":"2021-03-15T22:31:02.374038+0000","last_fullsized":"2021-03-15T22:31:02.374038+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:59.480717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:30:59.480717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"696","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.272470+0000","last_change":"2021-03-15T22:29:35.315031+0000","last_active":"2021-03-15T22:31:04.272470+0000","last_peered":"2021-03-15T22:31:04.272470+0000","last_clean":"2021-03-15T22:31:04.272470+0000","last_became_active":"2021-03-15T22:17:58.007827+0000","last_became_peered":"2021-03-15T22:17:58.007827+0000","last_unstale":"2021-03-15T22:31:04.272470+0000","last_undegraded":"2021-03-15T22:31:04.272470+0000","last_fullsized":"2021-03-15T22:31:04.272470+0000","mapping_epoch":60,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":61,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:35.314964+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:35.314964+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"217","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.280140+0000","last_change":"2021-03-15T22:29:51.226956+0000","last_active":"2021-03-15T22:31:04.280140+0000","last_peered":"2021-03-15T22:31:04.280140+0000","last_clean":"2021-03-15T22:31:04.280140+0000","last_became_active":"2021-03-15T22:29:51.226677+0000","last_became_peered":"2021-03-15T22:29:51.226677+0000","last_unstale":"2021-03-15T22:31:04.280140+0000","last_undegraded":"2021-03-15T22:31:04.280140+0000","last_fullsized":"2021-03-15T22:31:04.280140+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:14.776700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:29:14.776700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"164","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229970+0000","last_change":"2021-03-15T22:30:07.486292+0000","last_active":"2021-03-15T22:31:05.229970+0000","last_peered":"2021-03-15T22:31:05.229970+0000","last_clean":"2021-03-15T22:31:05.229970+0000","last_became_active":"2021-03-15T22:28:59.269481+0000","last_became_peered":"2021-03-15T22:28:59.269481+0000","last_unstale":"2021-03-15T22:31:05.229970+0000","last_undegraded":"2021-03-15T22:31:05.229970+0000","last_fullsized":"2021-03-15T22:31:05.229970+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:30:07.486190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.486190+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":50,"num_bytes_recovered":6260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"316","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229638+0000","last_change":"2021-03-15T22:29:00.216337+0000","last_active":"2021-03-15T22:31:05.229638+0000","last_peered":"2021-03-15T22:31:05.229638+0000","last_clean":"2021-03-15T22:31:05.229638+0000","last_became_active":"2021-03-15T22:29:00.215311+0000","last_became_peered":"2021-03-15T22:29:00.215311+0000","last_unstale":"2021-03-15T22:31:05.229638+0000","last_undegraded":"2021-03-15T22:31:05.229638+0000","last_fullsized":"2021-03-15T22:31:05.229638+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:36.102000+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:40.670674+0000","last_clean_scrub_stamp":"2021-03-15T22:28:36.102000+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"672","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.376114+0000","last_change":"2021-03-15T22:28:09.088750+0000","last_active":"2021-03-15T22:31:02.376114+0000","last_peered":"2021-03-15T22:31:02.376114+0000","last_clean":"2021-03-15T22:31:02.376114+0000","last_became_active":"2021-03-15T22:23:52.262741+0000","last_became_peered":"2021-03-15T22:23:52.262741+0000","last_unstale":"2021-03-15T22:31:02.376114+0000","last_undegraded":"2021-03-15T22:31:02.376114+0000","last_fullsized":"2021-03-15T22:31:02.376114+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:09.088684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:28:09.088684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"92","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239591+0000","last_change":"2021-03-15T22:30:42.233176+0000","last_active":"2021-03-15T22:31:07.239591+0000","last_peered":"2021-03-15T22:31:07.239591+0000","last_clean":"2021-03-15T22:31:07.239591+0000","last_became_active":"2021-03-15T22:29:51.220261+0000","last_became_peered":"2021-03-15T22:29:51.220261+0000","last_unstale":"2021-03-15T22:31:07.239591+0000","last_undegraded":"2021-03-15T22:31:07.239591+0000","last_fullsized":"2021-03-15T22:31:07.239591+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.233098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"498","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.231768+0000","last_change":"2021-03-15T22:29:00.220920+0000","last_active":"2021-03-15T22:31:05.231768+0000","last_peered":"2021-03-15T22:31:05.231768+0000","last_clean":"2021-03-15T22:31:05.231768+0000","last_became_active":"2021-03-15T22:29:00.220100+0000","last_became_peered":"2021-03-15T22:29:00.220100+0000","last_unstale":"2021-03-15T22:31:05.231768+0000","last_undegraded":"2021-03-15T22:31:05.231768+0000","last_fullsized":"2021-03-15T22:31:05.231768+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"90'22","last_scrub_stamp":"2021-03-15T22:27:37.107464+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:27:37.107464+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":580,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"197","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.279348+0000","last_change":"2021-03-15T22:29:51.212768+0000","last_active":"2021-03-15T22:31:04.279348+0000","last_peered":"2021-03-15T22:31:04.279348+0000","last_clean":"2021-03-15T22:31:04.279348+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:31:04.279348+0000","last_undegraded":"2021-03-15T22:31:04.279348+0000","last_fullsized":"2021-03-15T22:31:04.279348+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:19.721291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:29:19.721291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"506","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229563+0000","last_change":"2021-03-15T22:29:56.525891+0000","last_active":"2021-03-15T22:31:05.229563+0000","last_peered":"2021-03-15T22:31:05.229563+0000","last_clean":"2021-03-15T22:31:05.229563+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:31:05.229563+0000","last_undegraded":"2021-03-15T22:31:05.229563+0000","last_fullsized":"2021-03-15T22:31:05.229563+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:56.525822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.525822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"712","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.375501+0000","last_change":"2021-03-15T22:29:09.154666+0000","last_active":"2021-03-15T22:31:02.375501+0000","last_peered":"2021-03-15T22:31:02.375501+0000","last_clean":"2021-03-15T22:31:02.375501+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:31:02.375501+0000","last_undegraded":"2021-03-15T22:31:02.375501+0000","last_fullsized":"2021-03-15T22:31:02.375501+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:09.154606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:29:09.154606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"411","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.241507+0000","last_change":"2021-03-15T22:30:38.178660+0000","last_active":"2021-03-15T22:31:06.241507+0000","last_peered":"2021-03-15T22:31:06.241507+0000","last_clean":"2021-03-15T22:31:06.241507+0000","last_became_active":"2021-03-15T22:30:31.240826+0000","last_became_peered":"2021-03-15T22:30:31.240826+0000","last_unstale":"2021-03-15T22:31:06.241507+0000","last_undegraded":"2021-03-15T22:31:06.241507+0000","last_fullsized":"2021-03-15T22:31:06.241507+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:30:38.178596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.178596+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":1050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"338","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.230233+0000","last_change":"2021-03-15T22:29:51.219273+0000","last_active":"2021-03-15T22:31:05.230233+0000","last_peered":"2021-03-15T22:31:05.230233+0000","last_clean":"2021-03-15T22:31:05.230233+0000","last_became_active":"2021-03-15T22:29:51.218982+0000","last_became_peered":"2021-03-15T22:29:51.218982+0000","last_unstale":"2021-03-15T22:31:05.230233+0000","last_undegraded":"2021-03-15T22:31:05.230233+0000","last_fullsized":"2021-03-15T22:31:05.230233+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:35.496457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:29:35.496457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"758","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.241334+0000","last_change":"2021-03-15T22:30:44.116997+0000","last_active":"2021-03-15T22:31:06.241334+0000","last_peered":"2021-03-15T22:31:06.241334+0000","last_clean":"2021-03-15T22:31:06.241334+0000","last_became_active":"2021-03-15T22:29:51.222254+0000","last_became_peered":"2021-03-15T22:29:51.222254+0000","last_unstale":"2021-03-15T22:31:06.241334+0000","last_undegraded":"2021-03-15T22:31:06.241334+0000","last_fullsized":"2021-03-15T22:31:06.241334+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:30:44.116934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"228","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.234016+0000","last_change":"2021-03-15T22:30:46.165561+0000","last_active":"2021-03-15T22:31:03.234016+0000","last_peered":"2021-03-15T22:31:03.234016+0000","last_clean":"2021-03-15T22:31:03.234016+0000","last_became_active":"2021-03-15T22:30:31.254001+0000","last_became_peered":"2021-03-15T22:30:31.254001+0000","last_unstale":"2021-03-15T22:31:03.234016+0000","last_undegraded":"2021-03-15T22:31:03.234016+0000","last_fullsized":"2021-03-15T22:31:03.234016+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.992352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.992352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"735","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229356+0000","last_change":"2021-03-15T22:30:02.064784+0000","last_active":"2021-03-15T22:31:05.229356+0000","last_peered":"2021-03-15T22:31:05.229356+0000","last_clean":"2021-03-15T22:31:05.229356+0000","last_became_active":"2021-03-15T22:17:55.992556+0000","last_became_peered":"2021-03-15T22:17:55.992556+0000","last_unstale":"2021-03-15T22:31:05.229356+0000","last_undegraded":"2021-03-15T22:31:05.229356+0000","last_fullsized":"2021-03-15T22:31:05.229356+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:30:02.064721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.064721+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"196","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.280618+0000","last_change":"2021-03-15T22:30:42.771039+0000","last_active":"2021-03-15T22:31:04.280618+0000","last_peered":"2021-03-15T22:31:04.280618+0000","last_clean":"2021-03-15T22:31:04.280618+0000","last_became_active":"2021-03-15T22:28:59.276812+0000","last_became_peered":"2021-03-15T22:28:59.276812+0000","last_unstale":"2021-03-15T22:31:04.280618+0000","last_undegraded":"2021-03-15T22:31:04.280618+0000","last_fullsized":"2021-03-15T22:31:04.280618+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:42.770976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.770976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"642'8","reported_seq":"112","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239739+0000","last_change":"2021-03-15T22:30:06.920670+0000","last_active":"2021-03-15T22:31:07.239739+0000","last_peered":"2021-03-15T22:31:07.239739+0000","last_clean":"2021-03-15T22:31:07.239739+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:31:07.239739+0000","last_undegraded":"2021-03-15T22:31:07.239739+0000","last_fullsized":"2021-03-15T22:31:07.239739+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'8","last_scrub_stamp":"2021-03-15T22:30:06.920606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:06.920606+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"691","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.374260+0000","last_change":"2021-03-15T22:30:35.423621+0000","last_active":"2021-03-15T22:31:02.374260+0000","last_peered":"2021-03-15T22:31:02.374260+0000","last_clean":"2021-03-15T22:31:02.374260+0000","last_became_active":"2021-03-15T22:30:31.230560+0000","last_became_peered":"2021-03-15T22:30:31.230560+0000","last_unstale":"2021-03-15T22:31:02.374260+0000","last_undegraded":"2021-03-15T22:31:02.374260+0000","last_fullsized":"2021-03-15T22:31:02.374260+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:35.423559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:35.423559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"556'24","reported_seq":"505","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.234895+0000","last_change":"2021-03-15T22:30:02.548134+0000","last_active":"2021-03-15T22:31:05.234895+0000","last_peered":"2021-03-15T22:31:05.234895+0000","last_clean":"2021-03-15T22:31:05.234895+0000","last_became_active":"2021-03-15T22:29:00.226652+0000","last_became_peered":"2021-03-15T22:29:00.226652+0000","last_unstale":"2021-03-15T22:31:05.234895+0000","last_undegraded":"2021-03-15T22:31:05.234895+0000","last_fullsized":"2021-03-15T22:31:05.234895+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'24","last_scrub_stamp":"2021-03-15T22:30:02.548069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.548069+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"726","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229540+0000","last_change":"2021-03-15T22:30:32.248125+0000","last_active":"2021-03-15T22:31:05.229540+0000","last_peered":"2021-03-15T22:31:05.229540+0000","last_clean":"2021-03-15T22:31:05.229540+0000","last_became_active":"2021-03-15T22:30:32.247782+0000","last_became_peered":"2021-03-15T22:30:32.247782+0000","last_unstale":"2021-03-15T22:31:05.229540+0000","last_undegraded":"2021-03-15T22:31:05.229540+0000","last_fullsized":"2021-03-15T22:31:05.229540+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:34.939142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:34.939142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"683'14","reported_seq":"381","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.230095+0000","last_change":"2021-03-15T22:30:32.252825+0000","last_active":"2021-03-15T22:31:05.230095+0000","last_peered":"2021-03-15T22:31:05.230095+0000","last_clean":"2021-03-15T22:31:05.230095+0000","last_became_active":"2021-03-15T22:30:32.252383+0000","last_became_peered":"2021-03-15T22:30:32.252383+0000","last_unstale":"2021-03-15T22:31:05.230095+0000","last_undegraded":"2021-03-15T22:31:05.230095+0000","last_fullsized":"2021-03-15T22:31:05.230095+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'10","last_scrub_stamp":"2021-03-15T22:28:47.140118+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:28:47.140118+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"196","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.279555+0000","last_change":"2021-03-15T22:29:17.782276+0000","last_active":"2021-03-15T22:31:04.279555+0000","last_peered":"2021-03-15T22:31:04.279555+0000","last_clean":"2021-03-15T22:31:04.279555+0000","last_became_active":"2021-03-15T22:28:59.265822+0000","last_became_peered":"2021-03-15T22:28:59.265822+0000","last_unstale":"2021-03-15T22:31:04.279555+0000","last_undegraded":"2021-03-15T22:31:04.279555+0000","last_fullsized":"2021-03-15T22:31:04.279555+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:17.782209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:29:17.782209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"110","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239651+0000","last_change":"2021-03-15T22:30:36.106413+0000","last_active":"2021-03-15T22:31:07.239651+0000","last_peered":"2021-03-15T22:31:07.239651+0000","last_clean":"2021-03-15T22:31:07.239651+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:31:07.239651+0000","last_undegraded":"2021-03-15T22:31:07.239651+0000","last_fullsized":"2021-03-15T22:31:07.239651+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:30:36.106285+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.106285+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"485","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.233560+0000","last_change":"2021-03-15T22:30:46.154245+0000","last_active":"2021-03-15T22:31:03.233560+0000","last_peered":"2021-03-15T22:31:03.233560+0000","last_clean":"2021-03-15T22:31:03.233560+0000","last_became_active":"2021-03-15T22:30:31.254532+0000","last_became_peered":"2021-03-15T22:30:31.254532+0000","last_unstale":"2021-03-15T22:31:03.233560+0000","last_undegraded":"2021-03-15T22:31:03.233560+0000","last_fullsized":"2021-03-15T22:31:03.233560+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.003120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.003120+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"681","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.373627+0000","last_change":"2021-03-15T22:30:10.271920+0000","last_active":"2021-03-15T22:31:02.373627+0000","last_peered":"2021-03-15T22:31:02.373627+0000","last_clean":"2021-03-15T22:31:02.373627+0000","last_became_active":"2021-03-15T22:23:52.263426+0000","last_became_peered":"2021-03-15T22:23:52.263426+0000","last_unstale":"2021-03-15T22:31:02.373627+0000","last_undegraded":"2021-03-15T22:31:02.373627+0000","last_fullsized":"2021-03-15T22:31:02.373627+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:33.386700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:33.386700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"241","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.280263+0000","last_change":"2021-03-15T22:31:04.009824+0000","last_active":"2021-03-15T22:31:04.280263+0000","last_peered":"2021-03-15T22:31:04.280263+0000","last_clean":"2021-03-15T22:31:04.280263+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:31:04.280263+0000","last_undegraded":"2021-03-15T22:31:04.280263+0000","last_fullsized":"2021-03-15T22:31:04.280263+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:31:04.009748+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.009748+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"675","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.375427+0000","last_change":"2021-03-15T22:27:41.254554+0000","last_active":"2021-03-15T22:31:02.375427+0000","last_peered":"2021-03-15T22:31:02.375427+0000","last_clean":"2021-03-15T22:31:02.375427+0000","last_became_active":"2021-03-15T22:23:52.266133+0000","last_became_peered":"2021-03-15T22:23:52.266133+0000","last_unstale":"2021-03-15T22:31:02.375427+0000","last_undegraded":"2021-03-15T22:31:02.375427+0000","last_fullsized":"2021-03-15T22:31:02.375427+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:41.254469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:27:41.254469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"772","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.272224+0000","last_change":"2021-03-15T22:30:37.321794+0000","last_active":"2021-03-15T22:31:04.272224+0000","last_peered":"2021-03-15T22:31:04.272224+0000","last_clean":"2021-03-15T22:31:04.272224+0000","last_became_active":"2021-03-15T22:29:51.231818+0000","last_became_peered":"2021-03-15T22:29:51.231818+0000","last_unstale":"2021-03-15T22:31:04.272224+0000","last_undegraded":"2021-03-15T22:31:04.272224+0000","last_fullsized":"2021-03-15T22:31:04.272224+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.321715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.321715+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.321715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"342","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229698+0000","last_change":"2021-03-15T22:29:13.250177+0000","last_active":"2021-03-15T22:31:05.229698+0000","last_peered":"2021-03-15T22:31:05.229698+0000","last_clean":"2021-03-15T22:31:05.229698+0000","last_became_active":"2021-03-15T22:29:11.204254+0000","last_became_peered":"2021-03-15T22:29:11.204254+0000","last_unstale":"2021-03-15T22:31:05.229698+0000","last_undegraded":"2021-03-15T22:31:05.229698+0000","last_fullsized":"2021-03-15T22:31:05.229698+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:29:13.250082+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:29:13.250082+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"716'820","reported_seq":"2871","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.246219+0000","last_change":"2021-03-15T22:31:07.246342+0000","last_active":"2021-03-15T22:31:07.246219+0000","last_peered":"2021-03-15T22:31:07.246219+0000","last_clean":"2021-03-15T22:31:07.246219+0000","last_became_active":"2021-03-15T22:29:52.227518+0000","last_became_peered":"2021-03-15T22:29:52.227518+0000","last_unstale":"2021-03-15T22:31:07.246219+0000","last_undegraded":"2021-03-15T22:31:07.246219+0000","last_fullsized":"2021-03-15T22:31:07.246219+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'650","last_scrub_stamp":"2021-03-15T22:29:40.242110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.242110+0000","log_size":820,"ondisk_log_size":820,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34537168,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":368,"num_read_kb":420857,"num_write":243,"num_write_kb":79169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":30385610,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"},{"start":"241","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"641","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.417831+0000","last_change":"2021-03-15T22:31:07.417796+0000","last_active":"2021-03-15T22:31:07.417831+0000","last_peered":"2021-03-15T22:31:07.417831+0000","last_clean":"2021-03-15T22:31:07.417831+0000","last_became_active":"2021-03-15T22:29:52.217465+0000","last_became_peered":"2021-03-15T22:29:52.217465+0000","last_unstale":"2021-03-15T22:31:07.417831+0000","last_undegraded":"2021-03-15T22:31:07.417831+0000","last_fullsized":"2021-03-15T22:31:07.417831+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:07.417736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:07.417736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"714","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.375588+0000","last_change":"2021-03-15T22:28:12.530786+0000","last_active":"2021-03-15T22:31:02.375588+0000","last_peered":"2021-03-15T22:31:02.375588+0000","last_clean":"2021-03-15T22:31:02.375588+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:31:02.375588+0000","last_undegraded":"2021-03-15T22:31:02.375588+0000","last_fullsized":"2021-03-15T22:31:02.375588+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:27:40.278709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:40.278709+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"124","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239288+0000","last_change":"2021-03-15T22:30:41.227569+0000","last_active":"2021-03-15T22:31:07.239288+0000","last_peered":"2021-03-15T22:31:07.239288+0000","last_clean":"2021-03-15T22:31:07.239288+0000","last_became_active":"2021-03-15T22:29:51.221013+0000","last_became_peered":"2021-03-15T22:29:51.221013+0000","last_unstale":"2021-03-15T22:31:07.239288+0000","last_undegraded":"2021-03-15T22:31:07.239288+0000","last_fullsized":"2021-03-15T22:31:07.239288+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.227473+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1156,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"317'7","reported_seq":"854","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.373843+0000","last_change":"2021-03-15T22:30:07.253037+0000","last_active":"2021-03-15T22:31:02.373843+0000","last_peered":"2021-03-15T22:31:02.373843+0000","last_clean":"2021-03-15T22:31:02.373843+0000","last_became_active":"2021-03-15T22:23:52.262878+0000","last_became_peered":"2021-03-15T22:23:52.262878+0000","last_unstale":"2021-03-15T22:31:02.373843+0000","last_undegraded":"2021-03-15T22:31:02.373843+0000","last_fullsized":"2021-03-15T22:31:02.373843+0000","mapping_epoch":316,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:30:07.252960+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.252960+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"716'1097","reported_seq":"3170","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.255671+0000","last_change":"2021-03-15T22:31:05.255671+0000","last_active":"2021-03-15T22:31:05.255671+0000","last_peered":"2021-03-15T22:31:05.255671+0000","last_clean":"2021-03-15T22:31:05.255671+0000","last_became_active":"2021-03-15T22:30:34.291079+0000","last_became_peered":"2021-03-15T22:30:34.291079+0000","last_unstale":"2021-03-15T22:31:05.255671+0000","last_undegraded":"2021-03-15T22:31:05.255671+0000","last_fullsized":"2021-03-15T22:31:05.255671+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'857","last_scrub_stamp":"2021-03-15T22:29:33.490626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.490626+0000","log_size":1097,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55840724,"num_objects":27,"num_object_clones":22,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":1,"num_read":485,"num_read_kb":469291,"num_write":341,"num_write_kb":136747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":80476435,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"142","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229425+0000","last_change":"2021-03-15T22:29:58.503593+0000","last_active":"2021-03-15T22:31:05.229425+0000","last_peered":"2021-03-15T22:31:05.229425+0000","last_clean":"2021-03-15T22:31:05.229425+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:31:05.229425+0000","last_undegraded":"2021-03-15T22:31:05.229425+0000","last_fullsized":"2021-03-15T22:31:05.229425+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:58.503533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:58.503533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"683","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.375509+0000","last_change":"2021-03-15T22:27:44.227979+0000","last_active":"2021-03-15T22:31:02.375509+0000","last_peered":"2021-03-15T22:31:02.375509+0000","last_clean":"2021-03-15T22:31:02.375509+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:31:02.375509+0000","last_undegraded":"2021-03-15T22:31:02.375509+0000","last_fullsized":"2021-03-15T22:31:02.375509+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:44.227918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:44.227918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"840","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.469114+0000","last_change":"2021-03-15T22:31:05.469086+0000","last_active":"2021-03-15T22:31:05.469114+0000","last_peered":"2021-03-15T22:31:05.469114+0000","last_clean":"2021-03-15T22:31:05.469114+0000","last_became_active":"2021-03-15T22:28:59.267623+0000","last_became_peered":"2021-03-15T22:28:59.267623+0000","last_unstale":"2021-03-15T22:31:05.469114+0000","last_undegraded":"2021-03-15T22:31:05.469114+0000","last_fullsized":"2021-03-15T22:31:05.469114+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:31:05.469001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.469001+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"526","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.233003+0000","last_change":"2021-03-15T22:30:32.239285+0000","last_active":"2021-03-15T22:31:03.233003+0000","last_peered":"2021-03-15T22:31:03.233003+0000","last_clean":"2021-03-15T22:31:03.233003+0000","last_became_active":"2021-03-15T22:30:32.239001+0000","last_became_peered":"2021-03-15T22:30:32.239001+0000","last_unstale":"2021-03-15T22:31:03.233003+0000","last_undegraded":"2021-03-15T22:31:03.233003+0000","last_fullsized":"2021-03-15T22:31:03.233003+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:56.479415+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.479415+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"711'315","reported_seq":"2207","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.227486+0000","last_change":"2021-03-15T22:31:05.227666+0000","last_active":"2021-03-15T22:31:05.227486+0000","last_peered":"2021-03-15T22:31:05.227486+0000","last_clean":"2021-03-15T22:31:05.227486+0000","last_became_active":"2021-03-15T22:28:16.177364+0000","last_became_peered":"2021-03-15T22:28:16.177364+0000","last_unstale":"2021-03-15T22:31:05.227486+0000","last_undegraded":"2021-03-15T22:31:05.227486+0000","last_fullsized":"2021-03-15T22:31:05.227486+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'293","last_scrub_stamp":"2021-03-15T22:30:29.729234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.729234+0000","log_size":315,"ondisk_log_size":315,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12795248,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":90,"num_read_kb":66683,"num_write":86,"num_write_kb":26508,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":11148008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"729","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.375863+0000","last_change":"2021-03-15T22:30:32.237010+0000","last_active":"2021-03-15T22:31:02.375863+0000","last_peered":"2021-03-15T22:31:02.375863+0000","last_clean":"2021-03-15T22:31:02.375863+0000","last_became_active":"2021-03-15T22:30:32.236752+0000","last_became_peered":"2021-03-15T22:30:32.236752+0000","last_unstale":"2021-03-15T22:31:02.375863+0000","last_undegraded":"2021-03-15T22:31:02.375863+0000","last_fullsized":"2021-03-15T22:31:02.375863+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:08.137989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:08.137989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"538","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.233309+0000","last_change":"2021-03-15T22:30:32.267914+0000","last_active":"2021-03-15T22:31:03.233309+0000","last_peered":"2021-03-15T22:31:03.233309+0000","last_clean":"2021-03-15T22:31:03.233309+0000","last_became_active":"2021-03-15T22:30:32.263574+0000","last_became_peered":"2021-03-15T22:30:32.263574+0000","last_unstale":"2021-03-15T22:31:03.233309+0000","last_undegraded":"2021-03-15T22:31:03.233309+0000","last_fullsized":"2021-03-15T22:31:03.233309+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:59.529159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:59.529159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"725","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.375659+0000","last_change":"2021-03-15T22:30:08.279264+0000","last_active":"2021-03-15T22:31:02.375659+0000","last_peered":"2021-03-15T22:31:02.375659+0000","last_clean":"2021-03-15T22:31:02.375659+0000","last_became_active":"2021-03-15T22:29:51.222437+0000","last_became_peered":"2021-03-15T22:29:51.222437+0000","last_unstale":"2021-03-15T22:31:02.375659+0000","last_undegraded":"2021-03-15T22:31:02.375659+0000","last_fullsized":"2021-03-15T22:31:02.375659+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:08.279129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:30:08.279129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"822","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.272486+0000","last_change":"2021-03-15T22:31:01.245561+0000","last_active":"2021-03-15T22:31:04.272486+0000","last_peered":"2021-03-15T22:31:04.272486+0000","last_clean":"2021-03-15T22:31:04.272486+0000","last_became_active":"2021-03-15T22:30:31.239271+0000","last_became_peered":"2021-03-15T22:30:31.239271+0000","last_unstale":"2021-03-15T22:31:04.272486+0000","last_undegraded":"2021-03-15T22:31:04.272486+0000","last_fullsized":"2021-03-15T22:31:04.272486+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:31:01.245498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:01.245498+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":732,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"714'596","reported_seq":"2216","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.240369+0000","last_change":"2021-03-15T22:31:03.240369+0000","last_active":"2021-03-15T22:31:03.240369+0000","last_peered":"2021-03-15T22:31:03.240369+0000","last_clean":"2021-03-15T22:31:03.240369+0000","last_became_active":"2021-03-15T22:30:31.242297+0000","last_became_peered":"2021-03-15T22:30:31.242297+0000","last_unstale":"2021-03-15T22:31:03.240369+0000","last_undegraded":"2021-03-15T22:31:03.240369+0000","last_fullsized":"2021-03-15T22:31:03.240369+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'555","last_scrub_stamp":"2021-03-15T22:30:34.990334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.990334+0000","log_size":596,"ondisk_log_size":596,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29900655,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":292,"num_read_kb":348373,"num_write":183,"num_write_kb":64345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":24240388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"},{"start":"23a","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"189","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.279395+0000","last_change":"2021-03-15T22:30:32.238086+0000","last_active":"2021-03-15T22:31:04.279395+0000","last_peered":"2021-03-15T22:31:04.279395+0000","last_clean":"2021-03-15T22:31:04.279395+0000","last_became_active":"2021-03-15T22:30:32.237720+0000","last_became_peered":"2021-03-15T22:30:32.237720+0000","last_unstale":"2021-03-15T22:31:04.279395+0000","last_undegraded":"2021-03-15T22:31:04.279395+0000","last_fullsized":"2021-03-15T22:31:04.279395+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:49.778250+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:28:49.778250+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"516","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.235032+0000","last_change":"2021-03-15T22:30:32.267370+0000","last_active":"2021-03-15T22:31:03.235032+0000","last_peered":"2021-03-15T22:31:03.235032+0000","last_clean":"2021-03-15T22:31:03.235032+0000","last_became_active":"2021-03-15T22:30:32.267050+0000","last_became_peered":"2021-03-15T22:30:32.267050+0000","last_unstale":"2021-03-15T22:31:03.235032+0000","last_undegraded":"2021-03-15T22:31:03.235032+0000","last_fullsized":"2021-03-15T22:31:03.235032+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:07.063227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.063227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"92","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239539+0000","last_change":"2021-03-15T22:30:07.891846+0000","last_active":"2021-03-15T22:31:07.239539+0000","last_peered":"2021-03-15T22:31:07.239539+0000","last_clean":"2021-03-15T22:31:07.239539+0000","last_became_active":"2021-03-15T22:29:51.231699+0000","last_became_peered":"2021-03-15T22:29:51.231699+0000","last_unstale":"2021-03-15T22:31:07.239539+0000","last_undegraded":"2021-03-15T22:31:07.239539+0000","last_fullsized":"2021-03-15T22:31:07.239539+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:07.891775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.891775+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"227","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.280138+0000","last_change":"2021-03-15T22:30:32.557348+0000","last_active":"2021-03-15T22:31:04.280138+0000","last_peered":"2021-03-15T22:31:04.280138+0000","last_clean":"2021-03-15T22:31:04.280138+0000","last_became_active":"2021-03-15T22:30:32.250712+0000","last_became_peered":"2021-03-15T22:30:32.250712+0000","last_unstale":"2021-03-15T22:31:04.280138+0000","last_undegraded":"2021-03-15T22:31:04.280138+0000","last_fullsized":"2021-03-15T22:31:04.280138+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:30:05.601295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:05.601295+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"716'880","reported_seq":"3185","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.238061+0000","last_change":"2021-03-15T22:31:05.238249+0000","last_active":"2021-03-15T22:31:05.238061+0000","last_peered":"2021-03-15T22:31:05.238061+0000","last_clean":"2021-03-15T22:31:05.238061+0000","last_became_active":"2021-03-15T22:29:02.359450+0000","last_became_peered":"2021-03-15T22:29:02.359450+0000","last_unstale":"2021-03-15T22:31:05.238061+0000","last_undegraded":"2021-03-15T22:31:05.238061+0000","last_fullsized":"2021-03-15T22:31:05.238061+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'870","last_scrub_stamp":"2021-03-15T22:31:02.243396+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.243396+0000","log_size":880,"ondisk_log_size":880,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46183223,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":220,"num_read_kb":173023,"num_write":247,"num_write_kb":84116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":205,"num_bytes_recovered":114286074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"142","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229873+0000","last_change":"2021-03-15T22:30:29.533509+0000","last_active":"2021-03-15T22:31:05.229873+0000","last_peered":"2021-03-15T22:31:05.229873+0000","last_clean":"2021-03-15T22:31:05.229873+0000","last_became_active":"2021-03-15T22:28:59.273785+0000","last_became_peered":"2021-03-15T22:28:59.273785+0000","last_unstale":"2021-03-15T22:31:05.229873+0000","last_undegraded":"2021-03-15T22:31:05.229873+0000","last_fullsized":"2021-03-15T22:31:05.229873+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:29.533421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.533421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"87","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239533+0000","last_change":"2021-03-15T22:29:51.236372+0000","last_active":"2021-03-15T22:31:07.239533+0000","last_peered":"2021-03-15T22:31:07.239533+0000","last_clean":"2021-03-15T22:31:07.239533+0000","last_became_active":"2021-03-15T22:29:51.236007+0000","last_became_peered":"2021-03-15T22:29:51.236007+0000","last_unstale":"2021-03-15T22:31:07.239533+0000","last_undegraded":"2021-03-15T22:31:07.239533+0000","last_fullsized":"2021-03-15T22:31:07.239533+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:33.191779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.191779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"47","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.233111+0000","last_change":"2021-03-15T22:30:39.012748+0000","last_active":"2021-03-15T22:31:03.233111+0000","last_peered":"2021-03-15T22:31:03.233111+0000","last_clean":"2021-03-15T22:31:03.233111+0000","last_became_active":"2021-03-15T22:30:32.262041+0000","last_became_peered":"2021-03-15T22:30:32.262041+0000","last_unstale":"2021-03-15T22:31:03.233111+0000","last_undegraded":"2021-03-15T22:31:03.233111+0000","last_fullsized":"2021-03-15T22:31:03.233111+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.012676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,6],"acting":[2,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"929","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.241928+0000","last_change":"2021-03-15T22:30:32.294089+0000","last_active":"2021-03-15T22:31:06.241928+0000","last_peered":"2021-03-15T22:31:06.241928+0000","last_clean":"2021-03-15T22:31:06.241928+0000","last_became_active":"2021-03-15T22:30:32.244684+0000","last_became_peered":"2021-03-15T22:30:32.244684+0000","last_unstale":"2021-03-15T22:31:06.241928+0000","last_undegraded":"2021-03-15T22:31:06.241928+0000","last_fullsized":"2021-03-15T22:31:06.241928+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:30:03.187386+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.187386+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"178","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.280036+0000","last_change":"2021-03-15T22:30:30.738196+0000","last_active":"2021-03-15T22:31:04.280036+0000","last_peered":"2021-03-15T22:31:04.280036+0000","last_clean":"2021-03-15T22:31:04.280036+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:31:04.280036+0000","last_undegraded":"2021-03-15T22:31:04.280036+0000","last_fullsized":"2021-03-15T22:31:04.280036+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:30.738108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:30.738108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"716'800","reported_seq":"2667","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.238517+0000","last_change":"2021-03-15T22:31:05.238694+0000","last_active":"2021-03-15T22:31:05.238517+0000","last_peered":"2021-03-15T22:31:05.238517+0000","last_clean":"2021-03-15T22:31:05.238517+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:31:05.238517+0000","last_undegraded":"2021-03-15T22:31:05.238517+0000","last_fullsized":"2021-03-15T22:31:05.238517+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"690'772","last_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.836165+0000","log_size":800,"ondisk_log_size":800,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33849096,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":234,"num_read_kb":222458,"num_write":204,"num_write_kb":77140,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"712","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.375731+0000","last_change":"2021-03-15T22:29:07.163096+0000","last_active":"2021-03-15T22:31:02.375731+0000","last_peered":"2021-03-15T22:31:02.375731+0000","last_clean":"2021-03-15T22:31:02.375731+0000","last_became_active":"2021-03-15T22:28:59.278695+0000","last_became_peered":"2021-03-15T22:28:59.278695+0000","last_unstale":"2021-03-15T22:31:02.375731+0000","last_undegraded":"2021-03-15T22:31:02.375731+0000","last_fullsized":"2021-03-15T22:31:02.375731+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:07.163034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:07.163034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"705","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.241695+0000","last_change":"2021-03-15T22:30:34.070664+0000","last_active":"2021-03-15T22:31:06.241695+0000","last_peered":"2021-03-15T22:31:06.241695+0000","last_clean":"2021-03-15T22:31:06.241695+0000","last_became_active":"2021-03-15T22:30:31.243831+0000","last_became_peered":"2021-03-15T22:30:31.243831+0000","last_unstale":"2021-03-15T22:31:06.241695+0000","last_undegraded":"2021-03-15T22:31:06.241695+0000","last_fullsized":"2021-03-15T22:31:06.241695+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.070598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.070598+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"513","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.232809+0000","last_change":"2021-03-15T22:30:43.021718+0000","last_active":"2021-03-15T22:31:03.232809+0000","last_peered":"2021-03-15T22:31:03.232809+0000","last_clean":"2021-03-15T22:31:03.232809+0000","last_became_active":"2021-03-15T22:30:32.250316+0000","last_became_peered":"2021-03-15T22:30:32.250316+0000","last_unstale":"2021-03-15T22:31:03.232809+0000","last_undegraded":"2021-03-15T22:31:03.232809+0000","last_fullsized":"2021-03-15T22:31:03.232809+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:43.021632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:30:43.021632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"456","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.241186+0000","last_change":"2021-03-15T22:29:56.187294+0000","last_active":"2021-03-15T22:31:06.241186+0000","last_peered":"2021-03-15T22:31:06.241186+0000","last_clean":"2021-03-15T22:31:06.241186+0000","last_became_active":"2021-03-15T22:29:51.222262+0000","last_became_peered":"2021-03-15T22:29:51.222262+0000","last_unstale":"2021-03-15T22:31:06.241186+0000","last_undegraded":"2021-03-15T22:31:06.241186+0000","last_fullsized":"2021-03-15T22:31:06.241186+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:29:56.187210+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.187210+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"499","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.234529+0000","last_change":"2021-03-15T22:30:32.252681+0000","last_active":"2021-03-15T22:31:03.234529+0000","last_peered":"2021-03-15T22:31:03.234529+0000","last_clean":"2021-03-15T22:31:03.234529+0000","last_became_active":"2021-03-15T22:30:32.252163+0000","last_became_peered":"2021-03-15T22:30:32.252163+0000","last_unstale":"2021-03-15T22:31:03.234529+0000","last_undegraded":"2021-03-15T22:31:03.234529+0000","last_fullsized":"2021-03-15T22:31:03.234529+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:55.737371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:55.737371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"716'2021","reported_seq":"4531","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.252650+0000","last_change":"2021-03-15T22:31:05.253033+0000","last_active":"2021-03-15T22:31:05.252650+0000","last_peered":"2021-03-15T22:31:05.252650+0000","last_clean":"2021-03-15T22:31:05.252650+0000","last_became_active":"2021-03-15T22:28:59.276888+0000","last_became_peered":"2021-03-15T22:28:59.276888+0000","last_unstale":"2021-03-15T22:31:05.252650+0000","last_undegraded":"2021-03-15T22:31:05.252650+0000","last_fullsized":"2021-03-15T22:31:05.252650+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'2007","last_scrub_stamp":"2021-03-15T22:31:02.050395+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.050395+0000","log_size":2021,"ondisk_log_size":2021,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":76433601,"num_objects":40,"num_object_clones":33,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":2,"num_read":661,"num_read_kb":654741,"num_write":524,"num_write_kb":222714,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":205133613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"92","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239857+0000","last_change":"2021-03-15T22:30:28.967439+0000","last_active":"2021-03-15T22:31:07.239857+0000","last_peered":"2021-03-15T22:31:07.239857+0000","last_clean":"2021-03-15T22:31:07.239857+0000","last_became_active":"2021-03-15T22:29:51.227896+0000","last_became_peered":"2021-03-15T22:29:51.227896+0000","last_unstale":"2021-03-15T22:31:07.239857+0000","last_undegraded":"2021-03-15T22:31:07.239857+0000","last_fullsized":"2021-03-15T22:31:07.239857+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:28.967356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.967356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"704","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.375951+0000","last_change":"2021-03-15T22:30:05.206297+0000","last_active":"2021-03-15T22:31:02.375951+0000","last_peered":"2021-03-15T22:31:02.375951+0000","last_clean":"2021-03-15T22:31:02.375951+0000","last_became_active":"2021-03-15T22:28:12.507833+0000","last_became_peered":"2021-03-15T22:28:12.507833+0000","last_unstale":"2021-03-15T22:31:02.375951+0000","last_undegraded":"2021-03-15T22:31:02.375951+0000","last_fullsized":"2021-03-15T22:31:02.375951+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:05.206223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:30:05.206223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"325","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.231475+0000","last_change":"2021-03-15T22:29:51.223894+0000","last_active":"2021-03-15T22:31:05.231475+0000","last_peered":"2021-03-15T22:31:05.231475+0000","last_clean":"2021-03-15T22:31:05.231475+0000","last_became_active":"2021-03-15T22:29:51.223572+0000","last_became_peered":"2021-03-15T22:29:51.223572+0000","last_unstale":"2021-03-15T22:31:05.231475+0000","last_undegraded":"2021-03-15T22:31:05.231475+0000","last_fullsized":"2021-03-15T22:31:05.231475+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:45.454461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.454461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"738","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.272707+0000","last_change":"2021-03-15T22:30:28.176902+0000","last_active":"2021-03-15T22:31:04.272707+0000","last_peered":"2021-03-15T22:31:04.272707+0000","last_clean":"2021-03-15T22:31:04.272707+0000","last_became_active":"2021-03-15T22:17:56.290588+0000","last_became_peered":"2021-03-15T22:17:56.290588+0000","last_unstale":"2021-03-15T22:31:04.272707+0000","last_undegraded":"2021-03-15T22:31:04.272707+0000","last_fullsized":"2021-03-15T22:31:04.272707+0000","mapping_epoch":58,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":59,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:30:28.176831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.176831+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":45,"num_read_kb":33,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"725","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.374864+0000","last_change":"2021-03-15T22:30:32.244033+0000","last_active":"2021-03-15T22:31:02.374864+0000","last_peered":"2021-03-15T22:31:02.374864+0000","last_clean":"2021-03-15T22:31:02.374864+0000","last_became_active":"2021-03-15T22:30:32.243696+0000","last_became_peered":"2021-03-15T22:30:32.243696+0000","last_unstale":"2021-03-15T22:31:02.374864+0000","last_undegraded":"2021-03-15T22:31:02.374864+0000","last_fullsized":"2021-03-15T22:31:02.374864+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.257476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"714'568","reported_seq":"2593","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.242874+0000","last_change":"2021-03-15T22:31:03.242874+0000","last_active":"2021-03-15T22:31:03.242874+0000","last_peered":"2021-03-15T22:31:03.242874+0000","last_clean":"2021-03-15T22:31:03.242874+0000","last_became_active":"2021-03-15T22:30:35.366323+0000","last_became_peered":"2021-03-15T22:30:35.366323+0000","last_unstale":"2021-03-15T22:31:03.242874+0000","last_undegraded":"2021-03-15T22:31:03.242874+0000","last_fullsized":"2021-03-15T22:31:03.242874+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":0,"last_scrub":"560'396","last_scrub_stamp":"2021-03-15T22:28:45.222374+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.222374+0000","log_size":568,"ondisk_log_size":568,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28421494,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":161,"num_read_kb":157136,"num_write":154,"num_write_kb":67563,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":54764539,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"},{"start":"23a","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"756","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.241819+0000","last_change":"2021-03-15T22:30:41.171985+0000","last_active":"2021-03-15T22:31:06.241819+0000","last_peered":"2021-03-15T22:31:06.241819+0000","last_clean":"2021-03-15T22:31:06.241819+0000","last_became_active":"2021-03-15T22:29:51.222189+0000","last_became_peered":"2021-03-15T22:29:51.222189+0000","last_unstale":"2021-03-15T22:31:06.241819+0000","last_undegraded":"2021-03-15T22:31:06.241819+0000","last_fullsized":"2021-03-15T22:31:06.241819+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:41.171920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.171920+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"199","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.279442+0000","last_change":"2021-03-15T22:31:01.010456+0000","last_active":"2021-03-15T22:31:04.279442+0000","last_peered":"2021-03-15T22:31:04.279442+0000","last_clean":"2021-03-15T22:31:04.279442+0000","last_became_active":"2021-03-15T22:28:59.276599+0000","last_became_peered":"2021-03-15T22:28:59.276599+0000","last_unstale":"2021-03-15T22:31:04.279442+0000","last_undegraded":"2021-03-15T22:31:04.279442+0000","last_fullsized":"2021-03-15T22:31:04.279442+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:01.010393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:01.010393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"1491","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.237312+0000","last_change":"2021-03-15T22:28:52.152942+0000","last_active":"2021-03-15T22:31:03.237312+0000","last_peered":"2021-03-15T22:31:03.237312+0000","last_clean":"2021-03-15T22:31:03.237312+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:31:03.237312+0000","last_undegraded":"2021-03-15T22:31:03.237312+0000","last_fullsized":"2021-03-15T22:31:03.237312+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:28:52.152875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:28:52.152875+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":774,"num_read_kb":769,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"241","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.280537+0000","last_change":"2021-03-15T22:30:32.617969+0000","last_active":"2021-03-15T22:31:04.280537+0000","last_peered":"2021-03-15T22:31:04.280537+0000","last_clean":"2021-03-15T22:31:04.280537+0000","last_became_active":"2021-03-15T22:30:32.251686+0000","last_became_peered":"2021-03-15T22:30:32.251686+0000","last_unstale":"2021-03-15T22:31:04.280537+0000","last_undegraded":"2021-03-15T22:31:04.280537+0000","last_fullsized":"2021-03-15T22:31:04.280537+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:28:45.780434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.780434+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"87","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239833+0000","last_change":"2021-03-15T22:29:51.232258+0000","last_active":"2021-03-15T22:31:07.239833+0000","last_peered":"2021-03-15T22:31:07.239833+0000","last_clean":"2021-03-15T22:31:07.239833+0000","last_became_active":"2021-03-15T22:29:51.231915+0000","last_became_peered":"2021-03-15T22:29:51.231915+0000","last_unstale":"2021-03-15T22:31:07.239833+0000","last_undegraded":"2021-03-15T22:31:07.239833+0000","last_fullsized":"2021-03-15T22:31:07.239833+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:16.238757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:29:16.238757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"719'2161","reported_seq":"4637","reported_epoch":"719","state":"active+clean","last_fresh":"2021-03-15T22:31:08.246877+0000","last_change":"2021-03-15T22:31:07.248388+0000","last_active":"2021-03-15T22:31:08.246877+0000","last_peered":"2021-03-15T22:31:08.246877+0000","last_clean":"2021-03-15T22:31:08.246877+0000","last_became_active":"2021-03-15T22:29:52.225069+0000","last_became_peered":"2021-03-15T22:29:52.225069+0000","last_unstale":"2021-03-15T22:31:08.246877+0000","last_undegraded":"2021-03-15T22:31:08.246877+0000","last_fullsized":"2021-03-15T22:31:08.246877+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"650'1835","last_scrub_stamp":"2021-03-15T22:29:59.763141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:59.763141+0000","log_size":2161,"ondisk_log_size":2161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":100167828,"num_objects":48,"num_object_clones":41,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":3,"num_read":897,"num_read_kb":979284,"num_write":559,"num_write_kb":227753,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":165660176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"},{"start":"241","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"701","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.240915+0000","last_change":"2021-03-15T22:31:02.160887+0000","last_active":"2021-03-15T22:31:06.240915+0000","last_peered":"2021-03-15T22:31:06.240915+0000","last_clean":"2021-03-15T22:31:06.240915+0000","last_became_active":"2021-03-15T22:30:31.240169+0000","last_became_peered":"2021-03-15T22:30:31.240169+0000","last_unstale":"2021-03-15T22:31:06.240915+0000","last_undegraded":"2021-03-15T22:31:06.240915+0000","last_fullsized":"2021-03-15T22:31:06.240915+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:02.160813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.160813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"736","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.241893+0000","last_change":"2021-03-15T22:30:32.244248+0000","last_active":"2021-03-15T22:31:06.241893+0000","last_peered":"2021-03-15T22:31:06.241893+0000","last_clean":"2021-03-15T22:31:06.241893+0000","last_became_active":"2021-03-15T22:30:32.243790+0000","last_became_peered":"2021-03-15T22:30:32.243790+0000","last_unstale":"2021-03-15T22:31:06.241893+0000","last_undegraded":"2021-03-15T22:31:06.241893+0000","last_fullsized":"2021-03-15T22:31:06.241893+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:32.161342+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:32.161342+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"253","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.280105+0000","last_change":"2021-03-15T22:29:52.258695+0000","last_active":"2021-03-15T22:31:04.280105+0000","last_peered":"2021-03-15T22:31:04.280105+0000","last_clean":"2021-03-15T22:31:04.280105+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:31:04.280105+0000","last_undegraded":"2021-03-15T22:31:04.280105+0000","last_fullsized":"2021-03-15T22:31:04.280105+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:29:08.872836+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:29:08.872836+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"730","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.272526+0000","last_change":"2021-03-15T22:29:51.233266+0000","last_active":"2021-03-15T22:31:04.272526+0000","last_peered":"2021-03-15T22:31:04.272526+0000","last_clean":"2021-03-15T22:31:04.272526+0000","last_became_active":"2021-03-15T22:29:51.232811+0000","last_became_peered":"2021-03-15T22:29:51.232811+0000","last_unstale":"2021-03-15T22:31:04.272526+0000","last_undegraded":"2021-03-15T22:31:04.272526+0000","last_fullsized":"2021-03-15T22:31:04.272526+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:33.255248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.255248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"711'353","reported_seq":"2403","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.230015+0000","last_change":"2021-03-15T22:31:05.230015+0000","last_active":"2021-03-15T22:31:05.230015+0000","last_peered":"2021-03-15T22:31:05.230015+0000","last_clean":"2021-03-15T22:31:05.230015+0000","last_became_active":"2021-03-15T22:29:03.354952+0000","last_became_peered":"2021-03-15T22:29:03.354952+0000","last_unstale":"2021-03-15T22:31:05.230015+0000","last_undegraded":"2021-03-15T22:31:05.230015+0000","last_fullsized":"2021-03-15T22:31:05.230015+0000","mapping_epoch":593,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":594,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'353","last_scrub_stamp":"2021-03-15T22:31:02.426448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.426448+0000","log_size":353,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11072175,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":120,"num_read_kb":143126,"num_write":86,"num_write_kb":32987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":41911448,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"207","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.279346+0000","last_change":"2021-03-15T22:30:32.238279+0000","last_active":"2021-03-15T22:31:04.279346+0000","last_peered":"2021-03-15T22:31:04.279346+0000","last_clean":"2021-03-15T22:31:04.279346+0000","last_became_active":"2021-03-15T22:30:32.237925+0000","last_became_peered":"2021-03-15T22:30:32.237925+0000","last_unstale":"2021-03-15T22:31:04.279346+0000","last_undegraded":"2021-03-15T22:31:04.279346+0000","last_fullsized":"2021-03-15T22:31:04.279346+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:37.724915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:37.724915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"653","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239954+0000","last_change":"2021-03-15T22:30:32.257626+0000","last_active":"2021-03-15T22:31:07.239954+0000","last_peered":"2021-03-15T22:31:07.239954+0000","last_clean":"2021-03-15T22:31:07.239954+0000","last_became_active":"2021-03-15T22:30:32.257328+0000","last_became_peered":"2021-03-15T22:30:32.257328+0000","last_unstale":"2021-03-15T22:31:07.239954+0000","last_undegraded":"2021-03-15T22:31:07.239954+0000","last_fullsized":"2021-03-15T22:31:07.239954+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:58.713662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:29:58.713662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"921","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.241420+0000","last_change":"2021-03-15T22:29:52.437785+0000","last_active":"2021-03-15T22:31:06.241420+0000","last_peered":"2021-03-15T22:31:06.241420+0000","last_clean":"2021-03-15T22:31:06.241420+0000","last_became_active":"2021-03-15T22:29:51.222356+0000","last_became_peered":"2021-03-15T22:29:51.222356+0000","last_unstale":"2021-03-15T22:31:06.241420+0000","last_undegraded":"2021-03-15T22:31:06.241420+0000","last_fullsized":"2021-03-15T22:31:06.241420+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:29:20.139319+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"642'8","reported_seq":"243","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.279874+0000","last_change":"2021-03-15T22:29:51.399268+0000","last_active":"2021-03-15T22:31:04.279874+0000","last_peered":"2021-03-15T22:31:04.279874+0000","last_clean":"2021-03-15T22:31:04.279874+0000","last_became_active":"2021-03-15T22:29:51.215636+0000","last_became_peered":"2021-03-15T22:29:51.215636+0000","last_unstale":"2021-03-15T22:31:04.279874+0000","last_undegraded":"2021-03-15T22:31:04.279874+0000","last_fullsized":"2021-03-15T22:31:04.279874+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"602'7","last_scrub_stamp":"2021-03-15T22:29:12.845802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:12.845802+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"716'554","reported_seq":"2508","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.247658+0000","last_change":"2021-03-15T22:31:05.247658+0000","last_active":"2021-03-15T22:31:05.247658+0000","last_peered":"2021-03-15T22:31:05.247658+0000","last_clean":"2021-03-15T22:31:05.247658+0000","last_became_active":"2021-03-15T22:30:34.297623+0000","last_became_peered":"2021-03-15T22:30:34.297623+0000","last_unstale":"2021-03-15T22:31:05.247658+0000","last_undegraded":"2021-03-15T22:31:05.247658+0000","last_fullsized":"2021-03-15T22:31:05.247658+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"578'390","last_scrub_stamp":"2021-03-15T22:28:47.641188+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:28:47.641188+0000","log_size":554,"ondisk_log_size":554,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25460626,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":191,"num_read_kb":173087,"num_write":162,"num_write_kb":61734,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":47935099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"685","reported_epoch":"713","state":"active+clean","last_fresh":"2021-03-15T22:31:02.374400+0000","last_change":"2021-03-15T22:30:31.243805+0000","last_active":"2021-03-15T22:31:02.374400+0000","last_peered":"2021-03-15T22:31:02.374400+0000","last_clean":"2021-03-15T22:31:02.374400+0000","last_became_active":"2021-03-15T22:30:31.243392+0000","last_became_peered":"2021-03-15T22:30:31.243392+0000","last_unstale":"2021-03-15T22:31:02.374400+0000","last_undegraded":"2021-03-15T22:31:02.374400+0000","last_fullsized":"2021-03-15T22:31:02.374400+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:25:16.234565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:25:16.234565+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"178","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.280576+0000","last_change":"2021-03-15T22:30:37.798454+0000","last_active":"2021-03-15T22:31:04.280576+0000","last_peered":"2021-03-15T22:31:04.280576+0000","last_clean":"2021-03-15T22:31:04.280576+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:31:04.280576+0000","last_undegraded":"2021-03-15T22:31:04.280576+0000","last_fullsized":"2021-03-15T22:31:04.280576+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.798388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"830","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229871+0000","last_change":"2021-03-15T22:30:42.383592+0000","last_active":"2021-03-15T22:31:05.229871+0000","last_peered":"2021-03-15T22:31:05.229871+0000","last_clean":"2021-03-15T22:31:05.229871+0000","last_became_active":"2021-03-15T22:29:51.216039+0000","last_became_peered":"2021-03-15T22:29:51.216039+0000","last_unstale":"2021-03-15T22:31:05.229871+0000","last_undegraded":"2021-03-15T22:31:05.229871+0000","last_fullsized":"2021-03-15T22:31:05.229871+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:30:42.383512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.383512+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"171","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.230492+0000","last_change":"2021-03-15T22:30:27.472423+0000","last_active":"2021-03-15T22:31:05.230492+0000","last_peered":"2021-03-15T22:31:05.230492+0000","last_clean":"2021-03-15T22:31:05.230492+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:31:05.230492+0000","last_undegraded":"2021-03-15T22:31:05.230492+0000","last_fullsized":"2021-03-15T22:31:05.230492+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:30:27.472320+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:30:27.472320+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"718'979","reported_seq":"3147","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.248083+0000","last_change":"2021-03-15T22:31:07.248247+0000","last_active":"2021-03-15T22:31:07.248083+0000","last_peered":"2021-03-15T22:31:07.248083+0000","last_clean":"2021-03-15T22:31:07.248083+0000","last_became_active":"2021-03-15T22:30:34.298185+0000","last_became_peered":"2021-03-15T22:30:34.298185+0000","last_unstale":"2021-03-15T22:31:07.248083+0000","last_undegraded":"2021-03-15T22:31:07.248083+0000","last_fullsized":"2021-03-15T22:31:07.248083+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'889","last_scrub_stamp":"2021-03-15T22:30:29.983045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.983045+0000","log_size":979,"ondisk_log_size":979,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41263750,"num_objects":23,"num_object_clones":19,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":369,"num_read_kb":380942,"num_write":259,"num_write_kb":90159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":202,"num_bytes_recovered":109136440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"},{"start":"241","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"744","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229065+0000","last_change":"2021-03-15T22:30:32.262270+0000","last_active":"2021-03-15T22:31:05.229065+0000","last_peered":"2021-03-15T22:31:05.229065+0000","last_clean":"2021-03-15T22:31:05.229065+0000","last_became_active":"2021-03-15T22:30:32.261995+0000","last_became_peered":"2021-03-15T22:30:32.261995+0000","last_unstale":"2021-03-15T22:31:05.229065+0000","last_undegraded":"2021-03-15T22:31:05.229065+0000","last_fullsized":"2021-03-15T22:31:05.229065+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:09.101167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:09.101167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"492","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229904+0000","last_change":"2021-03-15T22:29:00.238851+0000","last_active":"2021-03-15T22:31:05.229904+0000","last_peered":"2021-03-15T22:31:05.229904+0000","last_clean":"2021-03-15T22:31:05.229904+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:31:05.229904+0000","last_undegraded":"2021-03-15T22:31:05.229904+0000","last_fullsized":"2021-03-15T22:31:05.229904+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:37.136505+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:28:37.136505+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"168","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.230056+0000","last_change":"2021-03-15T22:29:41.515256+0000","last_active":"2021-03-15T22:31:05.230056+0000","last_peered":"2021-03-15T22:31:05.230056+0000","last_clean":"2021-03-15T22:31:05.230056+0000","last_became_active":"2021-03-15T22:28:59.276755+0000","last_became_peered":"2021-03-15T22:28:59.276755+0000","last_unstale":"2021-03-15T22:31:05.230056+0000","last_undegraded":"2021-03-15T22:31:05.230056+0000","last_fullsized":"2021-03-15T22:31:05.230056+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:29:41.515193+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.515193+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"739","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229761+0000","last_change":"2021-03-15T22:31:04.495639+0000","last_active":"2021-03-15T22:31:05.229761+0000","last_peered":"2021-03-15T22:31:05.229761+0000","last_clean":"2021-03-15T22:31:05.229761+0000","last_became_active":"2021-03-15T22:28:59.266487+0000","last_became_peered":"2021-03-15T22:28:59.266487+0000","last_unstale":"2021-03-15T22:31:05.229761+0000","last_undegraded":"2021-03-15T22:31:05.229761+0000","last_fullsized":"2021-03-15T22:31:05.229761+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.495574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.495574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"716'903","reported_seq":"2858","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.231701+0000","last_change":"2021-03-15T22:31:05.231916+0000","last_active":"2021-03-15T22:31:05.231701+0000","last_peered":"2021-03-15T22:31:05.231701+0000","last_clean":"2021-03-15T22:31:05.231701+0000","last_became_active":"2021-03-15T22:29:00.220080+0000","last_became_peered":"2021-03-15T22:29:00.220080+0000","last_unstale":"2021-03-15T22:31:05.231701+0000","last_undegraded":"2021-03-15T22:31:05.231701+0000","last_fullsized":"2021-03-15T22:31:05.231701+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'732","last_scrub_stamp":"2021-03-15T22:29:36.271647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:36.271647+0000","log_size":903,"ondisk_log_size":903,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26493547,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":289,"num_read_kb":275921,"num_write":211,"num_write_kb":75302,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":25937939,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"643","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239898+0000","last_change":"2021-03-15T22:31:04.380476+0000","last_active":"2021-03-15T22:31:07.239898+0000","last_peered":"2021-03-15T22:31:07.239898+0000","last_clean":"2021-03-15T22:31:07.239898+0000","last_became_active":"2021-03-15T22:29:52.223342+0000","last_became_peered":"2021-03-15T22:29:52.223342+0000","last_unstale":"2021-03-15T22:31:07.239898+0000","last_undegraded":"2021-03-15T22:31:07.239898+0000","last_fullsized":"2021-03-15T22:31:07.239898+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.380417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.380417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"643'13","reported_seq":"939","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229536+0000","last_change":"2021-03-15T22:29:52.212266+0000","last_active":"2021-03-15T22:31:05.229536+0000","last_peered":"2021-03-15T22:31:05.229536+0000","last_clean":"2021-03-15T22:31:05.229536+0000","last_became_active":"2021-03-15T22:29:52.211688+0000","last_became_peered":"2021-03-15T22:29:52.211688+0000","last_unstale":"2021-03-15T22:31:05.229536+0000","last_undegraded":"2021-03-15T22:31:05.229536+0000","last_fullsized":"2021-03-15T22:31:05.229536+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'11","last_scrub_stamp":"2021-03-15T22:29:36.937870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:36.937870+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"142","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229735+0000","last_change":"2021-03-15T22:30:09.506976+0000","last_active":"2021-03-15T22:31:05.229735+0000","last_peered":"2021-03-15T22:31:05.229735+0000","last_clean":"2021-03-15T22:31:05.229735+0000","last_became_active":"2021-03-15T22:28:59.274186+0000","last_became_peered":"2021-03-15T22:28:59.274186+0000","last_unstale":"2021-03-15T22:31:05.229735+0000","last_undegraded":"2021-03-15T22:31:05.229735+0000","last_fullsized":"2021-03-15T22:31:05.229735+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:09.506914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:30:09.506914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"566","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.233105+0000","last_change":"2021-03-15T22:31:02.092287+0000","last_active":"2021-03-15T22:31:03.233105+0000","last_peered":"2021-03-15T22:31:03.233105+0000","last_clean":"2021-03-15T22:31:03.233105+0000","last_became_active":"2021-03-15T22:30:32.262267+0000","last_became_peered":"2021-03-15T22:30:32.262267+0000","last_unstale":"2021-03-15T22:31:03.233105+0000","last_undegraded":"2021-03-15T22:31:03.233105+0000","last_fullsized":"2021-03-15T22:31:03.233105+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:31:02.092217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.092217+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"712'295","reported_seq":"2226","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.224715+0000","last_change":"2021-03-15T22:31:05.224961+0000","last_active":"2021-03-15T22:31:05.224715+0000","last_peered":"2021-03-15T22:31:05.224715+0000","last_clean":"2021-03-15T22:31:05.224715+0000","last_became_active":"2021-03-15T22:28:15.172010+0000","last_became_peered":"2021-03-15T22:28:15.172010+0000","last_unstale":"2021-03-15T22:31:05.224715+0000","last_undegraded":"2021-03-15T22:31:05.224715+0000","last_fullsized":"2021-03-15T22:31:05.224715+0000","mapping_epoch":555,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":556,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'281","last_scrub_stamp":"2021-03-15T22:30:36.290657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.290657+0000","log_size":295,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15228363,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":108,"num_read_kb":100343,"num_write":62,"num_write_kb":21303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":9579561,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"431","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229930+0000","last_change":"2021-03-15T22:29:51.225632+0000","last_active":"2021-03-15T22:31:05.229930+0000","last_peered":"2021-03-15T22:31:05.229930+0000","last_clean":"2021-03-15T22:31:05.229930+0000","last_became_active":"2021-03-15T22:29:51.225325+0000","last_became_peered":"2021-03-15T22:29:51.225325+0000","last_unstale":"2021-03-15T22:31:05.229930+0000","last_undegraded":"2021-03-15T22:31:05.229930+0000","last_fullsized":"2021-03-15T22:31:05.229930+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:40.500803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.500803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"695","reported_epoch":"718","state":"active+clean","last_fresh":"2021-03-15T22:31:07.239977+0000","last_change":"2021-03-15T22:29:52.217795+0000","last_active":"2021-03-15T22:31:07.239977+0000","last_peered":"2021-03-15T22:31:07.239977+0000","last_clean":"2021-03-15T22:31:07.239977+0000","last_became_active":"2021-03-15T22:29:52.217407+0000","last_became_peered":"2021-03-15T22:29:52.217407+0000","last_unstale":"2021-03-15T22:31:07.239977+0000","last_undegraded":"2021-03-15T22:31:07.239977+0000","last_fullsized":"2021-03-15T22:31:07.239977+0000","mapping_epoch":642,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'7","last_scrub_stamp":"2021-03-15T22:29:40.964753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.964753+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"703","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229786+0000","last_change":"2021-03-15T22:30:34.363233+0000","last_active":"2021-03-15T22:31:05.229786+0000","last_peered":"2021-03-15T22:31:05.229786+0000","last_clean":"2021-03-15T22:31:05.229786+0000","last_became_active":"2021-03-15T22:30:31.239376+0000","last_became_peered":"2021-03-15T22:30:31.239376+0000","last_unstale":"2021-03-15T22:31:05.229786+0000","last_undegraded":"2021-03-15T22:31:05.229786+0000","last_fullsized":"2021-03-15T22:31:05.229786+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.363177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"791","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229448+0000","last_change":"2021-03-15T22:30:04.058254+0000","last_active":"2021-03-15T22:31:05.229448+0000","last_peered":"2021-03-15T22:31:05.229448+0000","last_clean":"2021-03-15T22:31:05.229448+0000","last_became_active":"2021-03-15T22:24:55.090544+0000","last_became_peered":"2021-03-15T22:24:55.090544+0000","last_unstale":"2021-03-15T22:31:05.229448+0000","last_undegraded":"2021-03-15T22:31:05.229448+0000","last_fullsized":"2021-03-15T22:31:05.229448+0000","mapping_epoch":379,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":380,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:30:04.058167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:04.058167+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1476,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"714'265","reported_seq":"1687","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.239324+0000","last_change":"2021-03-15T22:31:03.239324+0000","last_active":"2021-03-15T22:31:03.239324+0000","last_peered":"2021-03-15T22:31:03.239324+0000","last_clean":"2021-03-15T22:31:03.239324+0000","last_became_active":"2021-03-15T22:30:32.257058+0000","last_became_peered":"2021-03-15T22:30:32.257058+0000","last_unstale":"2021-03-15T22:31:03.239324+0000","last_undegraded":"2021-03-15T22:31:03.239324+0000","last_fullsized":"2021-03-15T22:31:03.239324+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"646'221","last_scrub_stamp":"2021-03-15T22:29:56.742213+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.742213+0000","log_size":265,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15889430,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":147,"num_read_kb":164357,"num_write":78,"num_write_kb":26848,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":32799803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"},{"start":"23a","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"19","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.283458+0000","last_change":"2021-03-15T22:31:04.283415+0000","last_active":"2021-03-15T22:31:04.283458+0000","last_peered":"2021-03-15T22:31:04.283458+0000","last_clean":"2021-03-15T22:31:04.283458+0000","last_became_active":"2021-03-15T22:31:01.240039+0000","last_became_peered":"2021-03-15T22:31:01.240039+0000","last_unstale":"2021-03-15T22:31:04.283458+0000","last_undegraded":"2021-03-15T22:31:04.283458+0000","last_fullsized":"2021-03-15T22:31:04.283458+0000","mapping_epoch":711,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":712,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.283355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.283355+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"732","reported_epoch":"717","state":"active+clean","last_fresh":"2021-03-15T22:31:06.241990+0000","last_change":"2021-03-15T22:29:41.190121+0000","last_active":"2021-03-15T22:31:06.241990+0000","last_peered":"2021-03-15T22:31:06.241990+0000","last_clean":"2021-03-15T22:31:06.241990+0000","last_became_active":"2021-03-15T22:28:12.503247+0000","last_became_peered":"2021-03-15T22:28:12.503247+0000","last_unstale":"2021-03-15T22:31:06.241990+0000","last_undegraded":"2021-03-15T22:31:06.241990+0000","last_fullsized":"2021-03-15T22:31:06.241990+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:41.190058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.190058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"703","reported_epoch":"715","state":"active+clean","last_fresh":"2021-03-15T22:31:04.272369+0000","last_change":"2021-03-15T22:30:43.259679+0000","last_active":"2021-03-15T22:31:04.272369+0000","last_peered":"2021-03-15T22:31:04.272369+0000","last_clean":"2021-03-15T22:31:04.272369+0000","last_became_active":"2021-03-15T22:17:53.976305+0000","last_became_peered":"2021-03-15T22:17:53.976305+0000","last_unstale":"2021-03-15T22:31:04.272369+0000","last_undegraded":"2021-03-15T22:31:04.272369+0000","last_fullsized":"2021-03-15T22:31:04.272369+0000","mapping_epoch":56,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":57,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:43.259585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:43.259585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"787","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229383+0000","last_change":"2021-03-15T22:30:38.419647+0000","last_active":"2021-03-15T22:31:05.229383+0000","last_peered":"2021-03-15T22:31:05.229383+0000","last_clean":"2021-03-15T22:31:05.229383+0000","last_became_active":"2021-03-15T22:30:31.239360+0000","last_became_peered":"2021-03-15T22:30:31.239360+0000","last_unstale":"2021-03-15T22:31:05.229383+0000","last_undegraded":"2021-03-15T22:31:05.229383+0000","last_fullsized":"2021-03-15T22:31:05.229383+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:30:38.419551+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.419551+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"716'974","reported_seq":"3276","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.254038+0000","last_change":"2021-03-15T22:31:05.254242+0000","last_active":"2021-03-15T22:31:05.254038+0000","last_peered":"2021-03-15T22:31:05.254038+0000","last_clean":"2021-03-15T22:31:05.254038+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:31:05.254038+0000","last_undegraded":"2021-03-15T22:31:05.254038+0000","last_fullsized":"2021-03-15T22:31:05.254038+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'877","last_scrub_stamp":"2021-03-15T22:30:27.732142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:27.732142+0000","log_size":974,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46608944,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":382,"num_read_kb":379483,"num_write":260,"num_write_kb":91288,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1dc","length":"1"},{"start":"21c","length":"1"},{"start":"229","length":"1"},{"start":"237","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"695","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229585+0000","last_change":"2021-03-15T22:30:36.417325+0000","last_active":"2021-03-15T22:31:05.229585+0000","last_peered":"2021-03-15T22:31:05.229585+0000","last_clean":"2021-03-15T22:31:05.229585+0000","last_became_active":"2021-03-15T22:30:31.243269+0000","last_became_peered":"2021-03-15T22:30:31.243269+0000","last_unstale":"2021-03-15T22:31:05.229585+0000","last_undegraded":"2021-03-15T22:31:05.229585+0000","last_fullsized":"2021-03-15T22:31:05.229585+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"683'12","reported_seq":"621","reported_epoch":"714","state":"active+clean","last_fresh":"2021-03-15T22:31:03.234395+0000","last_change":"2021-03-15T22:30:32.259770+0000","last_active":"2021-03-15T22:31:03.234395+0000","last_peered":"2021-03-15T22:31:03.234395+0000","last_clean":"2021-03-15T22:31:03.234395+0000","last_became_active":"2021-03-15T22:30:32.256541+0000","last_became_peered":"2021-03-15T22:30:32.256541+0000","last_unstale":"2021-03-15T22:31:03.234395+0000","last_undegraded":"2021-03-15T22:31:03.234395+0000","last_fullsized":"2021-03-15T22:31:03.234395+0000","mapping_epoch":682,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'8","last_scrub_stamp":"2021-03-15T22:29:38.210344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:38.210344+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"142","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229257+0000","last_change":"2021-03-15T22:30:03.567594+0000","last_active":"2021-03-15T22:31:05.229257+0000","last_peered":"2021-03-15T22:31:05.229257+0000","last_clean":"2021-03-15T22:31:05.229257+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:31:05.229257+0000","last_undegraded":"2021-03-15T22:31:05.229257+0000","last_fullsized":"2021-03-15T22:31:05.229257+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.567529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"840","reported_epoch":"716","state":"active+clean","last_fresh":"2021-03-15T22:31:05.229654+0000","last_change":"2021-03-15T22:30:00.030068+0000","last_active":"2021-03-15T22:31:05.229654+0000","last_peered":"2021-03-15T22:31:05.229654+0000","last_clean":"2021-03-15T22:31:05.229654+0000","last_became_active":"2021-03-15T22:28:59.268492+0000","last_became_peered":"2021-03-15T22:28:59.268492+0000","last_unstale":"2021-03-15T22:31:05.229654+0000","last_undegraded":"2021-03-15T22:31:05.229654+0000","last_fullsized":"2021-03-15T22:31:05.229654+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:30:00.029954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:00.029954+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":1872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":600145872,"num_objects":288,"num_object_clones":238,"num_object_copies":864,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":288,"num_whiteouts":13,"num_read":5014,"num_read_kb":5109105,"num_write":3659,"num_write_kb":1385676,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2069,"num_bytes_recovered":1078884691,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1317724160,"data_stored":1899053513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13581,"ondisk_log_size":13581,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":774,"num_read_kb":769,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1484,"num_read_kb":989,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":773,"num_bytes_recovered":59472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":916,"ondisk_log_size":916,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":22,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":69,"ondisk_log_size":69,"up":66,"acting":66,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299554,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":197016,"kb_used_data":154512,"kb_used_omap":0,"kb_used_meta":42496,"kb_avail":93548136,"statfs":{"total":95995035648,"available":95793291264,"internally_reserved":0,"allocated":158220288,"data_stored":235931701,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43515904},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.45900000000000002,"15min":0.48099999999999998},"min":{"1min":0.313,"5min":0.25,"15min":0.215},"max":{"1min":0.55200000000000005,"5min":1.4750000000000001,"15min":2.73},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.48199999999999998,"15min":0.49199999999999999},"min":{"1min":0.36799999999999999,"5min":0.222,"15min":0.222},"max":{"1min":0.62,"5min":2.0499999999999998,"15min":3.8799999999999999},"last":0.57399999999999995}]},{"osd":1,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46999999999999997,"15min":0.48299999999999998},"min":{"1min":0.32000000000000001,"5min":0.27300000000000002,"15min":0.22600000000000001},"max":{"1min":0.629,"5min":1.4350000000000001,"15min":2.6859999999999999},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.497,"15min":0.49099999999999999},"min":{"1min":0.34699999999999998,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.64300000000000002,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.34100000000000003}]},{"osd":2,"last update":"Mon Mar 15 22:26:04 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47599999999999998,"15min":0.47999999999999998},"min":{"1min":0.32000000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.80700000000000005,"5min":1.0029999999999999,"15min":1.0029999999999999},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.48799999999999999,"5min":0.51100000000000001,"15min":0.499},"min":{"1min":0.32400000000000001,"5min":0.311,"15min":0.28000000000000003},"max":{"1min":0.73099999999999998,"5min":2.5630000000000002,"15min":2.5630000000000002},"last":0.5}]},{"osd":3,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.502,"15min":0.51600000000000001},"min":{"1min":0.33800000000000002,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":1.0249999999999999,"5min":1.702,"15min":2.0339999999999998},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.54500000000000004,"15min":0.54600000000000004},"min":{"1min":0.34699999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58499999999999996,"5min":5.415,"15min":5.415},"last":0.46200000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.40999999999999998,"5min":0.42899999999999999,"15min":0.44},"min":{"1min":0.255,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":0.64400000000000002,"5min":1.927,"15min":1.927},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.43099999999999999,"5min":0.44600000000000001,"15min":0.45400000000000001},"min":{"1min":0.29199999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.621,"5min":1.6779999999999999,"15min":2.5339999999999998},"last":0.51800000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.71199999999999997,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.45000000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.56200000000000006}]},{"osd":6,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.48299999999999998,"15min":0.50900000000000001},"min":{"1min":0.313,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.59299999999999997,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.50700000000000001,"5min":0.48999999999999999,"15min":0.51000000000000001},"min":{"1min":0.33800000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":1.097,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":0.54300000000000004}]}]},{"osd":6,"up_from":45,"seq":193273528486,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":156444,"kb_used_data":117332,"kb_used_omap":0,"kb_used_meta":39104,"kb_avail":93588708,"statfs":{"total":95995035648,"available":95834836992,"internally_reserved":0,"allocated":120147968,"data_stored":179373484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40042496},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:59 2021","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47199999999999998,"15min":0.47999999999999998},"min":{"1min":0.251,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.92700000000000005,"5min":3.085,"15min":3.085},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.438,"15min":0.46800000000000003},"min":{"1min":0.312,"5min":0.27500000000000002,"15min":0.27100000000000002},"max":{"1min":0.81799999999999995,"5min":1.323,"15min":1.323},"last":0.53900000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:30:59 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.61199999999999999,"15min":0.54100000000000004},"min":{"1min":0.28599999999999998,"5min":0.28299999999999997,"15min":0.254},"max":{"1min":0.88100000000000001,"5min":14.637,"15min":14.637},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.47499999999999998,"15min":0.48299999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.84899999999999998,"5min":5.2450000000000001,"15min":5.2450000000000001},"last":0.433}]},{"osd":2,"last update":"Mon Mar 15 22:25:46 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.49299999999999999,"15min":0.501},"min":{"1min":0.317,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.65200000000000002,"5min":0.77600000000000002,"15min":0.80800000000000005},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.48899999999999999,"15min":0.498},"min":{"1min":0.35399999999999998,"5min":0.32800000000000001,"15min":0.31900000000000001},"max":{"1min":0.754,"5min":1.2170000000000001,"15min":1.2170000000000001},"last":0.42199999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:30:59 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.49399999999999999,"15min":0.51100000000000001},"min":{"1min":0.33000000000000002,"5min":0.318,"15min":0.29899999999999999},"max":{"1min":0.76700000000000002,"5min":2.6150000000000002,"15min":2.6150000000000002},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.48099999999999998,"15min":0.50800000000000001},"min":{"1min":0.33900000000000002,"5min":0.316,"15min":0.28699999999999998},"max":{"1min":0.80100000000000005,"5min":2.7890000000000001,"15min":2.7890000000000001},"last":0.38200000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:30:59 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.41399999999999998,"15min":0.42599999999999999},"min":{"1min":0.27900000000000003,"5min":0.224,"15min":0.224},"max":{"1min":0.70199999999999996,"5min":4.306,"15min":4.306},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.46999999999999997,"15min":0.47899999999999998},"min":{"1min":0.315,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":0.71399999999999997,"5min":4.181,"15min":4.181},"last":0.48099999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.441,"5min":0.441,"15min":0.441},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.60399999999999998,"5min":0.60399999999999998,"15min":0.60399999999999998},"last":0.56899999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:30:11 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.59399999999999997,"15min":0.54000000000000004},"min":{"1min":0.307,"5min":0.249,"15min":0.249},"max":{"1min":0.58599999999999997,"5min":15.340999999999999,"15min":15.340999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.63300000000000001,"15min":0.57699999999999996},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.63500000000000001,"5min":15.356,"15min":15.356},"last":0.51000000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575032,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":244980,"kb_used_data":199596,"kb_used_omap":0,"kb_used_meta":45376,"kb_avail":93500172,"statfs":{"total":95995035648,"available":95744176128,"internally_reserved":0,"allocated":204386304,"data_stored":285864533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46465024},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:40 2021","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.30399999999999999,"15min":0.51600000000000001},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.325,"5min":1.325,"15min":54.427999999999997},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.32800000000000001,"15min":0.53200000000000003},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":54.356999999999999},"last":0.27500000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.54300000000000004,"5min":1.1220000000000001,"15min":2.4100000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.38600000000000001},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.502,"5min":0.69799999999999995,"15min":2.0760000000000001},"last":0.379}]},{"osd":3,"last update":"Mon Mar 15 22:31:07 2021","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.33800000000000002,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":1.796},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.35899999999999999,"15min":0.41199999999999998},"min":{"1min":0.27100000000000002,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":0.35799999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:30:17 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.46899999999999997,"15min":0.53800000000000003},"min":{"1min":0.32700000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.69599999999999995,"5min":0.76600000000000001,"15min":3.3439999999999999},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.503,"5min":0.53000000000000003,"15min":0.59399999999999997},"min":{"1min":0.317,"5min":0.27200000000000002,"15min":0.253},"max":{"1min":0.77400000000000002,"5min":4.1790000000000003,"15min":4.6139999999999999},"last":0.56899999999999995}]},{"osd":5,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.52400000000000002,"15min":0.52400000000000002},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.73899999999999999,"5min":0.73899999999999999,"15min":0.73899999999999999},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.69599999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:30:50 2021","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52600000000000002,"15min":0.59699999999999998},"min":{"1min":0.35499999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.75},"last":0.67400000000000004},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52600000000000002,"15min":0.57899999999999996},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":1.3580000000000001},"last":0.52800000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:31:07 2021","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.52900000000000003,"15min":0.67400000000000004},"min":{"1min":0.40400000000000003,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":8.8100000000000005},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53000000000000003,"15min":0.57799999999999996},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.35799999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":1.325},"last":0.59999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738554,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":247880,"kb_used_data":205952,"kb_used_omap":0,"kb_used_meta":41920,"kb_avail":93497272,"statfs":{"total":95995035648,"available":95741206528,"internally_reserved":0,"allocated":210894848,"data_stored":294712948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42926080},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:30:33 2021","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.32800000000000001,"15min":0.34200000000000003},"min":{"1min":0.19800000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.435,"5min":1.024,"15min":1.7070000000000001},"last":0.251},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.35799999999999998,"15min":0.36499999999999999},"min":{"1min":0.23300000000000001,"5min":0.20699999999999999,"15min":0.19900000000000001},"max":{"1min":0.55100000000000005,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.40899999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:26:35 2021","interfaces":[{"interface":"back","average":{"1min":0.36399999999999999,"5min":0.39200000000000002,"15min":0.39500000000000002},"min":{"1min":0.23899999999999999,"5min":0.224,"15min":0.20799999999999999},"max":{"1min":0.76100000000000001,"5min":1.97,"15min":1.97},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.41799999999999998,"5min":0.38800000000000001,"15min":0.38500000000000001},"min":{"1min":0.251,"5min":0.251,"15min":0.20699999999999999},"max":{"1min":1.1419999999999999,"5min":2.0409999999999999,"15min":2.0409999999999999},"last":0.27700000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:30:16 2021","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.40100000000000002,"15min":0.39300000000000002},"min":{"1min":0.25,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.82699999999999996,"5min":1.538,"15min":1.538},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.34899999999999998,"5min":0.40000000000000002,"15min":0.40600000000000003},"min":{"1min":0.23899999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.69799999999999995,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.375}]},{"osd":4,"last update":"Mon Mar 15 22:30:16 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.52400000000000002,"15min":0.53900000000000003},"min":{"1min":0.27400000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.88700000000000001,"5min":1.583,"15min":1.583},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.5,"5min":0.53000000000000003,"15min":0.57099999999999995},"min":{"1min":0.33600000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.84099999999999997,"5min":1.645,"15min":1.645},"last":0.70599999999999996}]},{"osd":5,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.47299999999999998,"15min":0.47299999999999998},"min":{"1min":0.318,"5min":0.318,"15min":0.318},"max":{"1min":0.751,"5min":0.751,"15min":0.751},"last":0.61499999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:30:51 2021","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.52800000000000002,"15min":0.55600000000000005},"min":{"1min":0.35899999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71899999999999997,"5min":1.282,"15min":1.6180000000000001},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53100000000000003,"15min":0.58099999999999996},"min":{"1min":0.32000000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76000000000000001,"5min":0.98099999999999998,"15min":1.0049999999999999},"last":0.68300000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:30:16 2021","interfaces":[{"interface":"back","average":{"1min":0.48699999999999999,"5min":0.52900000000000003,"15min":0.56499999999999995},"min":{"1min":0.34599999999999997,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.71299999999999997,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.60299999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.55100000000000005,"15min":0.57199999999999995},"min":{"1min":0.38900000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.85299999999999998,"5min":1.6619999999999999,"15min":1.6619999999999999},"last":0.67000000000000004}]}]},{"osd":2,"up_from":681,"seq":2924872728584,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178276,"kb_used_data":155548,"kb_used_omap":119,"kb_used_meta":22600,"kb_avail":93566876,"statfs":{"total":95995035648,"available":95812481024,"internally_reserved":0,"allocated":159281152,"data_stored":198038484,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":23142761},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.25700000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.28000000000000003}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34599999999999997}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.443}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38900000000000001}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53100000000000003}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47799999999999998}]}]},{"osd":3,"up_from":26,"seq":111669149872,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":344164,"kb_used_data":286236,"kb_used_omap":0,"kb_used_meta":57920,"kb_avail":93400988,"statfs":{"total":95995035648,"available":95642611712,"internally_reserved":0,"allocated":293105664,"data_stored":423192822,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":59310080},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:15 2021","interfaces":[{"interface":"back","average":{"1min":0.32900000000000001,"5min":0.32400000000000001,"15min":0.34100000000000003},"min":{"1min":0.19900000000000001,"5min":0.17699999999999999,"15min":0.17699999999999999},"max":{"1min":0.95399999999999996,"5min":1.1499999999999999,"15min":2.625},"last":0.249},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.316,"15min":0.35099999999999998},"min":{"1min":0.20899999999999999,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":1.419,"5min":1.419,"15min":2.6339999999999999},"last":0.314}]},{"osd":1,"last update":"Mon Mar 15 22:30:15 2021","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.35699999999999998,"15min":0.40600000000000003},"min":{"1min":0.25,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":0.72599999999999998,"5min":0.72599999999999998,"15min":6.0199999999999996},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.33100000000000002,"5min":0.36899999999999999,"15min":0.41199999999999998},"min":{"1min":0.253,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.58699999999999997,"5min":2.5750000000000002,"15min":7.6130000000000004},"last":0.374}]},{"osd":2,"last update":"Mon Mar 15 22:26:03 2021","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.45200000000000001,"15min":0.42899999999999999},"min":{"1min":0.251,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.51800000000000002,"5min":7.5759999999999996,"15min":7.5759999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.33300000000000002,"5min":0.42199999999999999,"15min":0.39900000000000002},"min":{"1min":0.27000000000000002,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.54100000000000004,"5min":7.0860000000000003,"15min":7.0860000000000003},"last":0.38700000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:30:19 2021","interfaces":[{"interface":"back","average":{"1min":0.59799999999999998,"5min":0.52400000000000002,"15min":0.53300000000000003},"min":{"1min":0.26300000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":3.3370000000000002,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.52600000000000002},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.495,"15min":0.52500000000000002},"min":{"1min":0.28499999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":2.1059999999999999,"5min":2.1059999999999999,"15min":2.2890000000000001},"last":0.47899999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:30:25 2021","interfaces":[{"interface":"back","average":{"1min":0.86599999999999999,"5min":0.86599999999999999,"15min":0.86599999999999999},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":9.4870000000000001,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":0.56299999999999994},{"interface":"front","average":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.1679999999999999},"min":{"1min":0.32800000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":15.776,"5min":15.776,"15min":15.776},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:30:52 2021","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":0.73599999999999999,"5min":2.5299999999999998,"15min":2.6499999999999999},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.35899999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.86399999999999999,"5min":3.472,"15min":3.472},"last":0.59299999999999997}]},{"osd":7,"last update":"Mon Mar 15 22:30:15 2021","interfaces":[{"interface":"back","average":{"1min":0.70599999999999996,"5min":0.59499999999999997,"15min":0.58399999999999996},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":5.625,"5min":5.625,"15min":5.625},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.54100000000000004,"15min":0.56200000000000006},"min":{"1min":0.35699999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":2.5409999999999999,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":0.51600000000000001}]}]},{"osd":4,"up_from":31,"seq":133143986349,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":108960,"kb_used_data":75736,"kb_used_omap":0,"kb_used_meta":33216,"kb_avail":93636192,"statfs":{"total":95995035648,"available":95883460608,"internally_reserved":0,"allocated":77553664,"data_stored":100091736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":34013184},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.47899999999999998,"15min":0.47499999999999998},"min":{"1min":0.28499999999999998,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":0.68999999999999995,"5min":4.3769999999999998,"15min":4.3769999999999998},"last":0.439},{"interface":"front","average":{"1min":0.41999999999999998,"5min":0.54600000000000004,"15min":0.50600000000000001},"min":{"1min":0.30399999999999999,"5min":0.28999999999999998,"15min":0.251},"max":{"1min":0.72599999999999998,"5min":5.0209999999999999,"15min":5.0209999999999999},"last":0.42399999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.443,"5min":0.49199999999999999,"15min":0.48799999999999999},"min":{"1min":0.255,"5min":0.23999999999999999,"15min":0.23599999999999999},"max":{"1min":0.71699999999999997,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.44,"5min":0.5,"15min":0.50700000000000001},"min":{"1min":0.33900000000000002,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71099999999999997,"5min":4.718,"15min":4.718},"last":0.45900000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:26:18 2021","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.48399999999999999,"15min":0.50600000000000001},"min":{"1min":0.317,"5min":0.317,"15min":0.313},"max":{"1min":0.61299999999999999,"5min":0.70399999999999996,"15min":0.76100000000000001},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.48399999999999999,"5min":0.48299999999999998,"15min":0.502},"min":{"1min":0.30599999999999999,"5min":0.29399999999999998,"15min":0.29399999999999998},"max":{"1min":0.68799999999999994,"5min":0.71699999999999997,"15min":1.1910000000000001},"last":0.51100000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.58599999999999997,"15min":0.55500000000000005},"min":{"1min":0.36599999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.754,"5min":5.0499999999999998,"15min":5.0499999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.55300000000000005,"15min":0.54400000000000004},"min":{"1min":0.34000000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69899999999999995,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.58299999999999996}]},{"osd":5,"last update":"Mon Mar 15 22:30:27 2021","interfaces":[{"interface":"back","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.64800000000000002,"5min":0.64800000000000002,"15min":0.64800000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67100000000000004,"5min":0.67100000000000004,"15min":0.67100000000000004},"last":0.61599999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:30:54 2021","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.45900000000000002,"15min":0.45400000000000001},"min":{"1min":0.317,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.878,"5min":4.3079999999999998,"15min":4.3079999999999998},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.40500000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83799999999999997,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.65500000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:30:08 2021","interfaces":[{"interface":"back","average":{"1min":0.41499999999999998,"5min":0.498,"15min":0.498},"min":{"1min":0.29399999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.58899999999999997,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.42899999999999999,"5min":0.55700000000000005,"15min":0.54100000000000004},"min":{"1min":0.32400000000000001,"5min":0.25,"15min":0.25},"max":{"1min":0.59099999999999997,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":0.58899999999999997}]}]},{"osd":5,"up_from":616,"seq":2645699854358,"num_pgs":48,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":184828,"kb_used_data":158772,"kb_used_omap":153,"kb_used_meta":25894,"kb_avail":93560324,"statfs":{"total":95995035648,"available":95805771776,"internally_reserved":0,"allocated":162582528,"data_stored":224631625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":26515985},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.45200000000000001,"15min":0.45200000000000001},"min":{"1min":0.29199999999999998,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.443,"5min":0.443,"15min":0.443},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.59099999999999997,"5min":0.59099999999999997,"15min":0.59099999999999997},"last":0.47699999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.44600000000000001,"15min":0.44600000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.67600000000000005,"5min":0.67600000000000005,"15min":0.67600000000000005},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.36799999999999999,"5min":0.36799999999999999,"15min":0.36799999999999999},"max":{"1min":0.876,"5min":0.876,"15min":0.876},"last":0.434}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46100000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.46899999999999997,"15min":0.46899999999999997},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.64700000000000002,"5min":0.64700000000000002,"15min":0.64700000000000002},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.433,"5min":0.433,"15min":0.433},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.63300000000000001},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.57999999999999996,"5min":0.57999999999999996,"15min":0.57999999999999996},"last":0.38}]},{"osd":6,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"min":{"1min":0.23999999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.91200000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.45700000000000002,"15min":0.45700000000000002},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.44800000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:30:26 2021","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.43099999999999999,"15min":0.43099999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.73999999999999999,"5min":0.73999999999999999,"15min":0.73999999999999999},"last":0.51500000000000001},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.95499999999999996,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":12458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":6416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":11712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":202358784,"data_stored":289242758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":194592768,"data_stored":278500168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151994368,"data_stored":195465855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284459008,"data_stored":417711742,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":68947968,"data_stored":94612008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":154079232,"data_stored":219139148,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":111579136,"data_stored":173902798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":149712896,"data_stored":230479036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:31:10.943 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-03-15T22:31:10.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool set default.rgw.control pg_num 32 2021-03-15T22:31:10.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:31:10.945 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: cluster 2021-03-15T22:31:08.968591+0000 osd.0 (osd.0) 115 : cluster [DBG] 6.9 scrub ok 2021-03-15T22:31:10.946 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:31:10.946 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: 31:09.155590+0000 mgr.y (mgr.14144) 878 : cluster [DBG] pgmap v1223: 138 pgs: 138 active+clean; 565 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.3 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2021-03-15T22:31:10.946 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: audit 2021-03-15T22:31:09.500837+0000 mgr.y (mgr.14144) 879 : audit [DBG] from='client.25279 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:31:10.946 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: 2021-03-15T22:31:10.946 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:31:10.947 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: -15T22:31:10.041838+0000 mgr.y (mgr 2021-03-15T22:31:10.947 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: .14144) 880 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:10.947 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:31:10.947 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: 03-15T22:31:10.227650+0000 2021-03-15T22:31:10.948 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: mon.a (mon.0) 1924 : cluster [DBG] 2021-03-15T22:31:10.948 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:10 smithi125 conmon[56739]: osdmap e721: 8 total, 8 up, 8 in 2021-03-15T22:31:10.949 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-15T22:31:11.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:11.018 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:11.018 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:11.119 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:11.147 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:11.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:11.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:11.248 INFO:tasks.rados.rados.0.smithi076.stdout:2643: done (1 left) 2021-03-15T22:31:11.248 INFO:tasks.rados.rados.0.smithi076.stdout:2645: done (0 left) 2021-03-15T22:31:11.248 INFO:tasks.rados.rados.0.smithi076.stdout:2646: write oid 41 current snap is 308 2021-03-15T22:31:11.249 INFO:tasks.rados.rados.0.smithi076.stdout:2646: seq_num 915 ranges {578405=415195,1410318=598977,2744156=584480} 2021-03-15T22:31:11.249 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:11.255 INFO:tasks.rados.rados.0.smithi076.stdout:2646: writing smithi07694878-41 from 578405 to 993600 tid 1 2021-03-15T22:31:11.261 INFO:tasks.rados.rados.0.smithi076.stdout:2646: writing smithi07694878-41 from 1410318 to 2009295 tid 2 2021-03-15T22:31:11.269 INFO:tasks.rados.rados.0.smithi076.stdout:2646: writing smithi07694878-41 from 2744156 to 3328636 tid 3 2021-03-15T22:31:11.269 INFO:tasks.rados.rados.0.smithi076.stdout:2647: snap_create 2021-03-15T22:31:11.269 INFO:tasks.rados.rados.0.smithi076.stdout:2646: finishing write tid 1 to smithi07694878-41 2021-03-15T22:31:11.269 INFO:tasks.rados.rados.0.smithi076.stdout:2646: finishing write tid 2 to smithi07694878-41 2021-03-15T22:31:11.272 INFO:tasks.rados.rados.0.smithi076.stdout:2646: finishing write tid 3 to smithi07694878-41 2021-03-15T22:31:11.273 INFO:tasks.rados.rados.0.smithi076.stdout:2646: finishing write tid 4 to smithi07694878-41 2021-03-15T22:31:11.273 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:11.274 INFO:tasks.rados.rados.0.smithi076.stdout:2646: finishing write tid 5 to smithi07694878-41 2021-03-15T22:31:11.274 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 1228 (ObjNum 915 snap 308 seq_num 915) dirty exists 2021-03-15T22:31:11.274 INFO:tasks.rados.rados.0.smithi076.stdout:2646: left oid 41 (ObjNum 915 snap 308 seq_num 915) 2021-03-15T22:31:11.278 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:11.279 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:31:11.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:11.280 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:11.312 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:11.381 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:11.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:11.406 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:11.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:11.508 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:11.531 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:11.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:11.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:11.634 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:11.658 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:11.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:11.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:11 smithi076 conmon[56514]: cluster 2021-03-15T22:31:10.332315+0000 osd.3 (osd.3) 132 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:31:11.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:11 smithi076 conmon[56514]: cluster 2021-03-15T22:31:11.242299+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2021-03-15T22:31:11.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:11 smithi076 conmon[62156]: cluster 2021-03-15T22:31:10.332315+0000 osd.3 (osd.3) 2021-03-15T22:31:11.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:11 smithi076 conmon[62156]: 132 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:31:11.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:11 smithi076 conmon[62156]: cluster 2021-03-15T22:31:11.242299+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2021-03-15T22:31:11.703 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:11 smithi125 conmon[56739]: cluster 2021-03-15T22:31:10.332315+0000 osd.3 (osd.3) 132 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:31:11.703 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:11 smithi125 conmon[56739]: cluster 2021-03-15T22:31:11.242299+0000 mon.a (mon.0) 1925 : cluster [DBG] osdmap e722: 8 total, 8 up, 8 in 2021-03-15T22:31:11.760 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:11.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:11.784 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:11.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:11.885 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:11.907 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:11.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:11.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:12.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:12.032 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:12.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:12.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:12.135 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:12.156 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:12.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:12.160 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:12.247 INFO:tasks.rados.rados.0.smithi076.stdout:2646: done (1 left) 2021-03-15T22:31:12.247 INFO:tasks.rados.rados.0.smithi076.stdout:2647: done (0 left) 2021-03-15T22:31:12.248 INFO:tasks.rados.rados.0.smithi076.stdout:2648: write (excl) oid 2 current snap is 309 2021-03-15T22:31:12.248 INFO:tasks.rados.rados.0.smithi076.stdout:2648: seq_num 916 ranges {595212=490444,1868800=458988,2887932=507280} 2021-03-15T22:31:12.250 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:31:12.252 INFO:tasks.rados.rados.0.smithi076.stdout:2648: writing smithi07694878-2 from 595212 to 1085656 tid 1 2021-03-15T22:31:12.260 INFO:tasks.rados.rados.0.smithi076.stdout:2648: writing smithi07694878-2 from 1868800 to 2327788 tid 2 2021-03-15T22:31:12.261 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:12.267 INFO:tasks.rados.rados.0.smithi076.stdout:2648: writing smithi07694878-2 from 2887932 to 3395212 tid 3 2021-03-15T22:31:12.267 INFO:tasks.rados.rados.0.smithi076.stdout:2649: write oid 10 current snap is 309 2021-03-15T22:31:12.267 INFO:tasks.rados.rados.0.smithi076.stdout:2648: finishing write tid 1 to smithi07694878-2 2021-03-15T22:31:12.268 INFO:tasks.rados.rados.0.smithi076.stdout:2649: seq_num 917 ranges {612019=565693,1810929=602646,3082649=329370} 2021-03-15T22:31:12.273 INFO:tasks.rados.rados.0.smithi076.stdout:2649: writing smithi07694878-10 from 612019 to 1177712 tid 1 2021-03-15T22:31:12.280 INFO:tasks.rados.rados.0.smithi076.stdout:2649: writing smithi07694878-10 from 1810929 to 2413575 tid 2 2021-03-15T22:31:12.283 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:12.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:12.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:12.286 INFO:tasks.rados.rados.0.smithi076.stdout:2649: writing smithi07694878-10 from 3082649 to 3412019 tid 3 2021-03-15T22:31:12.287 INFO:tasks.rados.rados.0.smithi076.stdout:2650: snap_create 2021-03-15T22:31:12.287 INFO:tasks.rados.rados.0.smithi076.stdout:2648: finishing write tid 2 to smithi07694878-2 2021-03-15T22:31:12.287 INFO:tasks.rados.rados.0.smithi076.stdout:2648: finishing write tid 3 to smithi07694878-2 2021-03-15T22:31:12.287 INFO:tasks.rados.rados.0.smithi076.stdout:2648: finishing write tid 4 to smithi07694878-2 2021-03-15T22:31:12.288 INFO:tasks.rados.rados.0.smithi076.stdout:2648: finishing write tid 5 to smithi07694878-2 2021-03-15T22:31:12.288 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1759 (ObjNum 916 snap 309 seq_num 916) dirty exists 2021-03-15T22:31:12.288 INFO:tasks.rados.rados.0.smithi076.stdout:2648: left oid 2 (ObjNum 916 snap 309 seq_num 916) 2021-03-15T22:31:12.288 INFO:tasks.rados.rados.0.smithi076.stdout:2649: finishing write tid 1 to smithi07694878-10 2021-03-15T22:31:12.288 INFO:tasks.rados.rados.0.smithi076.stdout:2649: finishing write tid 2 to smithi07694878-10 2021-03-15T22:31:12.289 INFO:tasks.rados.rados.0.smithi076.stdout:2649: finishing write tid 3 to smithi07694878-10 2021-03-15T22:31:12.290 INFO:tasks.rados.rados.0.smithi076.stdout:2649: finishing write tid 4 to smithi07694878-10 2021-03-15T22:31:12.290 INFO:tasks.rados.rados.0.smithi076.stdout:2649: finishing write tid 5 to smithi07694878-10 2021-03-15T22:31:12.290 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 1763 (ObjNum 917 snap 309 seq_num 917) dirty exists 2021-03-15T22:31:12.290 INFO:tasks.rados.rados.0.smithi076.stdout:2649: left oid 10 (ObjNum 917 snap 309 seq_num 917) 2021-03-15T22:31:12.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:12.410 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:12.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:12.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:12.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:12.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: cluster 2021-03-15T22:31:10.971292+0000 2021-03-15T22:31:12.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: osd.0 (osd.0) 116 : cluster [DBG] 5.3 deep-scrub ok 2021-03-15T22:31:12.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: cluster 2021-03-15T22:31:11.029280+0000 osd.2 (osd.2) 10 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:31:12.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: cluster 2021-03-15T22:31:11.156211+0000 mgr.y (mgr.14144) 881 : cluster [DBG] pgmap v1226: 138 pgs: 138 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 21 op/s 2021-03-15T22:31:12.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: audit 2021-03-15T22:31:11.422736+0000 mon.c (mon.1) 59 : audit [INF] from='client.? 172.21.15.76:0/107670030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:31:12.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: audit 2021-03-15T22:31:11.423099+0000 mon.a (mon.0) 1926 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:31:12.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: cluster 2021-03-15T22:31:11.436717+0000 osd.5 (osd.5) 20 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:31:12.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: cluster 2021-03-15T22:31:12.237658+0000 mon.a 2021-03-15T22:31:12.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: (mon.0) 1927 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:12.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: audit 2021 2021-03-15T22:31:12.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: -03-15T22:31:12.244463+0000 mon.a (mon.0) 1928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:31:12.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: 2021-03-15T22:31:12.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: cluster 2021-03-15T22:31:12. 2021-03-15T22:31:12.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[56514]: 244543+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2021-03-15T22:31:12.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: cluster 2021-03-15T22:31:10.971292 2021-03-15T22:31:12.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: +0000 osd.0 (osd.0) 116 : cluster [DBG] 5.3 deep-scrub ok 2021-03-15T22:31:12.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:31:12.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: T22:31:11.029280+0000 osd.2 (osd.2) 10 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:31:12.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:31:12.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: 03-15T22:31: 2021-03-15T22:31:12.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: 11.156211+0000 mgr.y (mgr.14144) 881 : cluster [DBG] 2021-03-15T22:31:12.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: pgmap v1226: 138 pgs: 138 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 21 op/s 2021-03-15T22:31:12.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:31:12.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: 03-15T22:31:11.422736+0000 mon.c (mon.1) 59 : audit [INF] from='client.? 172.21.15.76:0/107670030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:31:12.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: audit 2021-03-15T22:31 2021-03-15T22:31:12.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: :11.423099+0000 mon.a (mon.0) 1926 : audit [INF] 2021-03-15T22:31:12.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:31:12.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:31:12.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: 22:31:11.436717+0000 osd.5 (osd.5) 20 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:31:12.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: cluster 2021-03-15T22:31:12. 2021-03-15T22:31:12.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: 237658+0000 mon.a (mon.0) 1927 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:12.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: audit 2021-03-15T22:31 2021-03-15T22:31:12.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: :12.244463+0000 mon.a (mon.0) 1928 : audit [INF] 2021-03-15T22:31:12.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:31:12.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:31:12.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:12 smithi076 conmon[62156]: 31:12.244543+0000 mon.a (mon.0) 1929 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2021-03-15T22:31:12.537 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:12.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:12.539 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:12.640 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:12.648 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: cluster 2021-03-15T22:31:10.971292+0000 osd.0 (osd. 2021-03-15T22:31:12.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: 0) 116 : cluster [DBG] 5.3 deep-scrub ok 2021-03-15T22:31:12.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: cluster 2021-03-15T22:31:11.029280+0000 osd.2 (osd.2) 10 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:31:12.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: cluster 2021-03-15T22:31:11.156211+0000 mgr.y (mgr.14144) 881 : cluster [DBG] pgmap v1226: 138 pgs: 138 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 21 op/s 2021-03-15T22:31:12.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: audit 2021-03-15T22:31:11.422736+0000 mon.c (mon.1 2021-03-15T22:31:12.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: ) 59 : audit [INF] from='client.? 172.21.15.76:0/107670030' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:31:12.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:31:12.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: :31:11.423099+0000 mon.a (mon.0) 1926 : audit [INF] 2021-03-15T22:31:12.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:31:12.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:31:12.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: 15T22:31:11.436717+0000 2021-03-15T22:31:12.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: osd.5 (osd.5) 20 : cluster [DBG] 2021-03-15T22:31:12.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: 5.7 scrub ok 2021-03-15T22:31:12.652 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:31:12.652 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: T22:31:12.237658+0000 mon.a 2021-03-15T22:31:12.652 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: (mon.0) 1927 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:12.652 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: 2021-03-15T22:31:12.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:31:12.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: 22:31:12.244463+0000 mon.a ( 2021-03-15T22:31:12.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: mon.0) 1928 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:31:12.653 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:31:12.654 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: 03-15T22:31:12.244543+0000 mon.a 2021-03-15T22:31:12.654 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: (mon.0) 1929 : cluster [DBG] osdmap e723: 8 total, 8 up, 8 in 2021-03-15T22:31:12.654 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:12 smithi125 conmon[56739]: 2021-03-15T22:31:12.661 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:12.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:12.663 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:12.765 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:12.787 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:12.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:12.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:12.832 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:31:12.832 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:31:12.890 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:12.917 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:12.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:12.918 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:13.019 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:13.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:13.045 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:13.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:13.147 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:13.169 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:13.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:13.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:13.250 INFO:tasks.rados.rados.0.smithi076.stdout:2648: done (2 left) 2021-03-15T22:31:13.250 INFO:tasks.rados.rados.0.smithi076.stdout:2649: done (1 left) 2021-03-15T22:31:13.250 INFO:tasks.rados.rados.0.smithi076.stdout:2650: done (0 left) 2021-03-15T22:31:13.251 INFO:tasks.rados.rados.0.smithi076.stdout:2651: snap_remove snap 291 2021-03-15T22:31:13.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:13.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:13.294 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:13.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:13.396 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:13.418 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:13.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:13.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:13.420 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:13 smithi076 conmon[62156]: cluster 2021-03-15T22:31:11.926265+0000 osd.0 (osd.0) 117 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:31:13.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-15T22:31:13.521 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:13.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:13.545 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:13.546 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:13.647 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:13.671 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:13.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:13.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:13.765 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:31:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:13 smithi076 conmon[56514]: cluster 2021-03-15T22:31:11.926265+0000 osd.0 (osd.0) 117 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:31:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:13 smithi076 conmon[56514]: cluster 2021-03-15T22:31:12.070840+0000 osd.1 (osd.1) 140 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:31:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:13 smithi076 conmon[56514]: cluster 2021-03-15T22:31:13.245937+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2021-03-15T22:31:13.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:13 smithi076 conmon[62156]: cluster 2021-03-15T22:31:12.070840+0000 osd.1 (osd.1) 140 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:31:13.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:13 smithi076 conmon[62156]: cluster 2021-03-15T22:31:13.245937+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2021-03-15T22:31:13.774 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:13.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:13 smithi125 conmon[56739]: cluster 2021-03-15T22:31:11.926265 2021-03-15T22:31:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:13 smithi125 conmon[56739]: +0000 osd.0 (osd.0) 117 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:31:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:13 smithi125 conmon[56739]: cluster 2021-03-15T22:31:12.070840+0000 osd.1 (osd.1) 140 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:31:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:13 smithi125 conmon[56739]: cluster 2021-03-15T22:31:13.245937+0000 mon.a (mon.0) 1930 : cluster [DBG] osdmap e724: 8 total, 8 up, 8 in 2021-03-15T22:31:13.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:13.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:13.898 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:13.921 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:13.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:13.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:14.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:14.047 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:14.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:14.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:14.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:14.172 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:14.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:14.173 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:14.249 INFO:tasks.rados.rados.0.smithi076.stdout:2651: done (0 left) 2021-03-15T22:31:14.249 INFO:tasks.rados.rados.0.smithi076.stdout:2652: snap_create 2021-03-15T22:31:14.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:14.298 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:14.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:14.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:14.400 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:14.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:14.424 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:14.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:14.505 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:31:14.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[56514]: cluster 2021-03-15T22:31:13.156823+0000 mgr.y (mgr 2021-03-15T22:31:14.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[56514]: .14144) 882 : cluster [DBG] pgmap v1229: 138 pgs: 138 active+clean; 583 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2021-03-15T22:31:14.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[56514]: cluster 2021-03-15T22:31:14.244606+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2021-03-15T22:31:14.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[56514]: audit 2021-03-15T22:31:14.307705+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? 172.21.15.76:0/4185851685' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]: dispatch 2021-03-15T22:31:14.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[62156]: cluster 2021-03-15T22:31:13 2021-03-15T22:31:14.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[62156]: .156823+0000 mgr.y (mgr.14144) 882 : cluster [DBG] pgmap v1229: 138 pgs: 138 active+clean; 583 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2021-03-15T22:31:14.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[62156]: cluster 2021-03-15T22:31:14.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:31:14.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[62156]: T22:31:14.244606+0000 2021-03-15T22:31:14.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[62156]: mon.a (mon.0) 1931 : cluster 2021-03-15T22:31:14.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[62156]: [DBG] osdmap e725: 8 total, 8 up, 8 in 2021-03-15T22:31:14.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[62156]: audit 2021 2021-03-15T22:31:14.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[62156]: -03-15T22:31:14 2021-03-15T22:31:14.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[62156]: .307705+0000 mon.a (mon.0) 1932 : 2021-03-15T22:31:14.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:14 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/4185851685' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]: dispatch 2021-03-15T22:31:14.522 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:31:14.525 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:14.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:14 smithi125 conmon[56739]: cluster 2021-03-15T22:31:13. 2021-03-15T22:31:14.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:14 smithi125 conmon[56739]: 156823+0000 mgr.y (mgr.14144) 882 : cluster [DBG] pgmap v1229: 138 pgs: 138 active+clean; 583 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 18 op/s 2021-03-15T22:31:14.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:14 smithi125 conmon[56739]: cluster 2021-03-15T22:31:14.244606+0000 mon.a (mon.0) 1931 : cluster [DBG] osdmap e725: 8 total, 8 up, 8 in 2021-03-15T22:31:14.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:14 smithi125 conmon[56739]: audit 2021-03-15T22:31:14.307705+0000 mon.a (mon.0) 1932 : audit [INF] from='client.? 172.21.15.76:0/4185851685' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]: dispatch 2021-03-15T22:31:14.538 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:31:14.547 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:14.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:14.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:14.554 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:31:14.571 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:31:14.588 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:31:14.605 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:31:14.622 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:31:14.623 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:31:14.623 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:31:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:31:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:31:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:31:14.624 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:31:14.625 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:31:14.625 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:31:14.649 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:14.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:14.674 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:14.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:14.776 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:14.798 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:14.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:14.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:14.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:14.923 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:14.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:14.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:15.027 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:15.052 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:15.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:15.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:15.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:15.178 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:15.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:15.179 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:15.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:31:15.232 INFO:tasks.rados.rados.0.smithi076.stdout:2652: done (0 left) 2021-03-15T22:31:15.233 INFO:tasks.rados.rados.0.smithi076.stdout:2653: write oid 36 current snap is 311 2021-03-15T22:31:15.233 INFO:tasks.rados.rados.0.smithi076.stdout:2653: seq_num 918 ranges {628826=640942,1869411=462657,2826425=534649} 2021-03-15T22:31:15.236 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 5 pg_num to 32 2021-03-15T22:31:15.239 INFO:tasks.rados.rados.0.smithi076.stdout:2653: writing smithi07694878-36 from 628826 to 1269768 tid 1 2021-03-15T22:31:15.245 INFO:tasks.rados.rados.0.smithi076.stdout:2653: writing smithi07694878-36 from 1869411 to 2332068 tid 2 2021-03-15T22:31:15.252 INFO:tasks.rados.rados.0.smithi076.stdout:2653: writing smithi07694878-36 from 2826425 to 3361074 tid 3 2021-03-15T22:31:15.253 INFO:tasks.rados.rados.0.smithi076.stdout:2654: snap_create 2021-03-15T22:31:15.253 INFO:tasks.rados.rados.0.smithi076.stdout:2653: finishing write tid 1 to smithi07694878-36 2021-03-15T22:31:15.253 INFO:tasks.rados.rados.0.smithi076.stdout:2653: finishing write tid 2 to smithi07694878-36 2021-03-15T22:31:15.255 INFO:tasks.rados.rados.0.smithi076.stdout:2653: finishing write tid 3 to smithi07694878-36 2021-03-15T22:31:15.256 INFO:tasks.rados.rados.0.smithi076.stdout:2653: finishing write tid 4 to smithi07694878-36 2021-03-15T22:31:15.256 INFO:tasks.rados.rados.0.smithi076.stdout:2653: finishing write tid 5 to smithi07694878-36 2021-03-15T22:31:15.257 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 1573 (ObjNum 918 snap 311 seq_num 918) dirty exists 2021-03-15T22:31:15.258 INFO:tasks.rados.rados.0.smithi076.stdout:2653: left oid 36 (ObjNum 918 snap 311 seq_num 918) 2021-03-15T22:31:15.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:15.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:15.309 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:15.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:15.410 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:15.435 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:15.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:15.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:15.442 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:31:15.442 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:31:15.442 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:31:15.442 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:31:15.443 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.443 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.1.0:743 7.0 e623/617 7:00c03349:::smithi07694878-40:1c7 v 623'1643, mlcod=620'1639)", 2021-03-15T22:31:15.443 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:32.216634+0000", 2021-03-15T22:31:15.443 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 103.221099675, 2021-03-15T22:31:15.443 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0097424629999999998, 2021-03-15T22:31:15.444 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.444 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:15.444 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.444 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.445 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.445 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216634+0000", 2021-03-15T22:31:15.445 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.445 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.446 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.446 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.446 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216634+0000", 2021-03-15T22:31:15.446 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2570000000000004e-06 2021-03-15T22:31:15.447 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.447 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.447 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.448 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216640+0000", 2021-03-15T22:31:15.448 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.024e-06 2021-03-15T22:31:15.448 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.448 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.449 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.449 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216642+0000", 2021-03-15T22:31:15.449 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6389999999999998e-06 2021-03-15T22:31:15.449 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.450 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.450 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.450 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216651+0000", 2021-03-15T22:31:15.450 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9669999999999998e-06 2021-03-15T22:31:15.451 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.451 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.451 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.451 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216657+0000", 2021-03-15T22:31:15.451 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1328000000000006e-05 2021-03-15T22:31:15.452 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.452 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.452 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.452 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216718+0000", 2021-03-15T22:31:15.453 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.156e-05 2021-03-15T22:31:15.453 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.453 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.453 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.453 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.216739+0000", 2021-03-15T22:31:15.454 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0096119250000000003 2021-03-15T22:31:15.454 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.454 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.455 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:15.455 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.226351+0000", 2021-03-15T22:31:15.456 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4763e-05 2021-03-15T22:31:15.456 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.456 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.457 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.458 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:32.226376+0000", 2021-03-15T22:31:15.458 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0097424629999999998 2021-03-15T22:31:15.458 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.459 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.459 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.460 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.460 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.461 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(osd.7.52:26 7.0 7:08414d91:::smithi07694878-42:head [assert-version v1737,copy-get max 8388608] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e642)", 2021-03-15T22:31:15.461 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:51.199690+0000", 2021-03-15T22:31:15.462 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 84.238043726000001, 2021-03-15T22:31:15.462 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0301155609999999, 2021-03-15T22:31:15.463 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.463 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:15.464 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.465 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.465 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.466 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199690+0000", 2021-03-15T22:31:15.466 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.467 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.467 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.468 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.468 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199690+0000", 2021-03-15T22:31:15.468 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.689e-06 2021-03-15T22:31:15.469 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.469 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.470 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.470 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199692+0000", 2021-03-15T22:31:15.471 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8900000000000002e-06 2021-03-15T22:31:15.471 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.471 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.472 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.472 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199702+0000", 2021-03-15T22:31:15.473 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1841999999999999e-05 2021-03-15T22:31:15.473 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.474 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.474 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.474 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199714+0000", 2021-03-15T22:31:15.475 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4039999999999998e-06 2021-03-15T22:31:15.475 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.476 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.476 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.477 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199721+0000", 2021-03-15T22:31:15.477 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8303000000000001e-05 2021-03-15T22:31:15.477 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.478 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.478 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.479 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199780+0000", 2021-03-15T22:31:15.479 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0937e-05 2021-03-15T22:31:15.479 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.480 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.480 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:31:15.481 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199791+0000", 2021-03-15T22:31:15.481 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010780039 2021-03-15T22:31:15.482 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.482 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.482 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.483 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.210571+0000", 2021-03-15T22:31:15.483 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0107e-05 2021-03-15T22:31:15.484 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.484 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.485 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:15.485 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.210581+0000", 2021-03-15T22:31:15.485 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0163172810000001 2021-03-15T22:31:15.486 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.486 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.487 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.487 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226898+0000", 2021-03-15T22:31:15.488 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012874800000000001 2021-03-15T22:31:15.488 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.488 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.489 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.489 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227027+0000", 2021-03-15T22:31:15.490 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027783209999999998 2021-03-15T22:31:15.490 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.490 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.491 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.491 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.229805+0000", 2021-03-15T22:31:15.491 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0301155609999999 2021-03-15T22:31:15.492 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.492 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.492 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.493 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.493 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.493 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2106 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94751378609664 gen 7] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:31:15.494 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.131311+0000", 2021-03-15T22:31:15.494 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 83.306422718999997, 2021-03-15T22:31:15.494 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.092023303000000001, 2021-03-15T22:31:15.495 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.495 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:15.495 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.495 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:31:15.495 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:31:15.496 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2106 2021-03-15T22:31:15.496 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.496 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.496 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.497 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.497 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131311+0000", 2021-03-15T22:31:15.497 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.497 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.497 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.498 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.498 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131311+0000", 2021-03-15T22:31:15.498 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9400000000000004e-07 2021-03-15T22:31:15.498 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.499 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.499 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.499 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131312+0000", 2021-03-15T22:31:15.499 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2810000000000001e-06 2021-03-15T22:31:15.499 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.500 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.500 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.500 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131313+0000", 2021-03-15T22:31:15.500 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6209999999999998e-06 2021-03-15T22:31:15.500 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.501 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.501 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.501 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131319+0000", 2021-03-15T22:31:15.501 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9940000000000002e-06 2021-03-15T22:31:15.501 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.502 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.502 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.502 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131323+0000", 2021-03-15T22:31:15.503 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0538999999999998e-05 2021-03-15T22:31:15.503 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.503 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.503 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.503 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131384+0000", 2021-03-15T22:31:15.504 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5080000000000005e-06 2021-03-15T22:31:15.504 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.504 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.504 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:15.505 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131392+0000", 2021-03-15T22:31:15.505 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.086493444000000003 2021-03-15T22:31:15.505 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.505 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.506 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.506 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.217886+0000", 2021-03-15T22:31:15.506 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010953 2021-03-15T22:31:15.506 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.506 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.507 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.507 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.218981+0000", 2021-03-15T22:31:15.507 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043527219999999998 2021-03-15T22:31:15.507 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.508 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.508 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.508 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.223334+0000", 2021-03-15T22:31:15.508 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.092023303000000001 2021-03-15T22:31:15.508 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.509 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.509 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.509 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.509 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.509 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2109 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94751378609664 gen 8] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:31:15.510 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:31:15.510 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 83.306399782, 2021-03-15T22:31:15.510 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.093324206000000007, 2021-03-15T22:31:15.510 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.511 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:15.511 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.511 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:31:15.511 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:31:15.511 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2109 2021-03-15T22:31:15.512 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.512 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.512 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.512 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.513 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:31:15.513 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.513 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.513 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.513 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.514 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:31:15.514 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0699999999999996e-07 2021-03-15T22:31:15.514 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.514 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.514 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.515 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:31:15.515 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.066e-06 2021-03-15T22:31:15.515 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.515 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.516 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.516 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131336+0000", 2021-03-15T22:31:15.516 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9439999999999999e-06 2021-03-15T22:31:15.516 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.516 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.517 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.517 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131342+0000", 2021-03-15T22:31:15.517 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1559999999999997e-06 2021-03-15T22:31:15.517 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.518 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.518 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.519 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131347+0000", 2021-03-15T22:31:15.519 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5375000000000003e-05 2021-03-15T22:31:15.520 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.520 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.520 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.520 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131412+0000", 2021-03-15T22:31:15.521 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2870000000000004e-06 2021-03-15T22:31:15.521 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.522 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.522 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:31:15.522 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131417+0000", 2021-03-15T22:31:15.523 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.087846846000000006 2021-03-15T22:31:15.523 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.524 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.524 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.524 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.219264+0000", 2021-03-15T22:31:15.524 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5641000000000003e-05 2021-03-15T22:31:15.524 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.525 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.525 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.525 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.219340+0000", 2021-03-15T22:31:15.525 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053179840000000004 2021-03-15T22:31:15.526 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.526 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.526 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.526 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.224658+0000", 2021-03-15T22:31:15.526 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.093324206000000007 2021-03-15T22:31:15.527 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.527 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.527 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.527 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.528 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.528 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5970 7.0 7:071f7a04:::smithi07694878-37:head [read 0~2622057,checksum] snapc 0=[] ondisk+read+known_if_redirected e643)", 2021-03-15T22:31:15.528 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.226680+0000", 2021-03-15T22:31:15.528 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 83.211053613000004, 2021-03-15T22:31:15.529 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012541424000000001, 2021-03-15T22:31:15.529 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.529 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:15.529 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.529 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.530 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.530 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5970 2021-03-15T22:31:15.530 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.530 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.531 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.531 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.531 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226680+0000", 2021-03-15T22:31:15.531 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.531 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.532 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.532 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.532 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226680+0000", 2021-03-15T22:31:15.533 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5719999999999999e-06 2021-03-15T22:31:15.533 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.533 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.533 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.533 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226681+0000", 2021-03-15T22:31:15.534 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.297e-06 2021-03-15T22:31:15.534 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.534 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.534 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.535 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226683+0000", 2021-03-15T22:31:15.535 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9460000000000003e-06 2021-03-15T22:31:15.535 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.535 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.535 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.536 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226689+0000", 2021-03-15T22:31:15.536 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7290000000000002e-06 2021-03-15T22:31:15.536 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.536 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.536 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.537 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226692+0000", 2021-03-15T22:31:15.537 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011147916000000001 2021-03-15T22:31:15.537 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.538 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.538 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.539 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.237840+0000", 2021-03-15T22:31:15.539 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000103939 2021-03-15T22:31:15.539 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.540 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.540 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.541 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.237944+0000", 2021-03-15T22:31:15.541 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001277025 2021-03-15T22:31:15.541 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.542 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.542 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.543 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.239221+0000", 2021-03-15T22:31:15.543 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012541424000000001 2021-03-15T22:31:15.544 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.544 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.544 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.545 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.545 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.546 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5974 7.e 7:7ac381e0:::smithi07694878-24:head [copy-from ver 1703] snapc 1fc=[1fc,1fb,1f8,1f7,1f6,1f4,1f3,1f0,1ee,1e9,1e7,1e6,1e4,1e0,1df,1dc,1d5,1d4,1c7,1c5,1c3,1ae,1a4,1a1,19c,193,17a,176,141] ondisk+write+known_if_redirected e643)", 2021-03-15T22:31:15.546 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.226697+0000", 2021-03-15T22:31:15.547 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 83.211036230000005, 2021-03-15T22:31:15.547 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031270606999999999, 2021-03-15T22:31:15.547 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.548 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:15.548 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.549 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.549 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.550 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5974 2021-03-15T22:31:15.550 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.551 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.551 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.551 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.552 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226697+0000", 2021-03-15T22:31:15.552 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.553 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.553 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.554 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.554 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226697+0000", 2021-03-15T22:31:15.554 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6269999999999999e-06 2021-03-15T22:31:15.555 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.555 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.556 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.556 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226702+0000", 2021-03-15T22:31:15.557 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.302e-06 2021-03-15T22:31:15.557 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.558 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.558 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.559 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226703+0000", 2021-03-15T22:31:15.559 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1020000000000002e-06 2021-03-15T22:31:15.559 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.560 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.560 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.560 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226709+0000", 2021-03-15T22:31:15.561 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6780000000000002e-06 2021-03-15T22:31:15.561 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.561 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.561 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.562 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226713+0000", 2021-03-15T22:31:15.562 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025251599999999999 2021-03-15T22:31:15.562 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.563 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.563 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.563 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226965+0000", 2021-03-15T22:31:15.564 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010260900000000001 2021-03-15T22:31:15.564 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.565 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.565 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.565 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227068+0000", 2021-03-15T22:31:15.566 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013635573999999999 2021-03-15T22:31:15.566 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.566 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.567 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-15T22:31:15.567 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.240704+0000", 2021-03-15T22:31:15.567 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010619755 2021-03-15T22:31:15.568 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.568 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.568 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:15.569 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.251323+0000", 2021-03-15T22:31:15.569 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024678149999999999 2021-03-15T22:31:15.569 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.570 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.570 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.571 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.253791+0000", 2021-03-15T22:31:15.572 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040414350000000003 2021-03-15T22:31:15.572 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.573 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.573 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.574 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.257833+0000", 2021-03-15T22:31:15.574 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9538e-05 2021-03-15T22:31:15.574 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.575 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.575 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:15.576 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.257872+0000", 2021-03-15T22:31:15.576 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5655999999999994e-05 2021-03-15T22:31:15.577 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.577 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.577 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.578 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.257968+0000", 2021-03-15T22:31:15.578 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031270606999999999 2021-03-15T22:31:15.579 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.579 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.580 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.580 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.581 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.581 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5975 7.e 7:7ac381e0:::smithi07694878-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e643)", 2021-03-15T22:31:15.581 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.226719+0000", 2021-03-15T22:31:15.582 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 83.211013980999994, 2021-03-15T22:31:15.582 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031554302999999999, 2021-03-15T22:31:15.583 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.583 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:15.583 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.584 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.584 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.584 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5975 2021-03-15T22:31:15.584 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.584 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.585 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.585 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.585 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226719+0000", 2021-03-15T22:31:15.585 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.585 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.586 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.586 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.586 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226719+0000", 2021-03-15T22:31:15.586 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9899999999999999e-07 2021-03-15T22:31:15.587 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.587 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.587 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.587 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226720+0000", 2021-03-15T22:31:15.587 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3540000000000001e-06 2021-03-15T22:31:15.588 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.588 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.588 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.588 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226722+0000", 2021-03-15T22:31:15.588 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.049e-06 2021-03-15T22:31:15.589 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.589 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.589 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.589 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226728+0000", 2021-03-15T22:31:15.590 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9450000000000001e-06 2021-03-15T22:31:15.590 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.590 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.590 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.590 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226733+0000", 2021-03-15T22:31:15.591 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048394300000000003 2021-03-15T22:31:15.591 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.591 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.591 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.591 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227216+0000", 2021-03-15T22:31:15.592 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3120000000000001e-05 2021-03-15T22:31:15.592 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.592 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.592 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:31:15.593 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227260+0000", 2021-03-15T22:31:15.593 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020041761000000002 2021-03-15T22:31:15.593 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.593 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.593 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.594 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.247301+0000", 2021-03-15T22:31:15.594 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7263999999999997e-05 2021-03-15T22:31:15.594 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.594 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.594 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:31:15.595 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.247389+0000", 2021-03-15T22:31:15.595 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010630624999999999 2021-03-15T22:31:15.595 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.595 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.595 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.596 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258019+0000", 2021-03-15T22:31:15.596 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6768999999999995e-05 2021-03-15T22:31:15.596 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.596 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.597 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.597 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258116+0000", 2021-03-15T22:31:15.597 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015767400000000001 2021-03-15T22:31:15.597 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.597 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.598 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.598 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258274+0000", 2021-03-15T22:31:15.598 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031554302999999999 2021-03-15T22:31:15.598 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.598 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.599 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.599 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.599 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.599 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(osd.5.616:1 7.0 7:0cc37e3d:::smithi07694878-21:head [assert-version v1703,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e643)", 2021-03-15T22:31:15.599 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.227857+0000", 2021-03-15T22:31:15.600 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 83.209875999999994, 2021-03-15T22:31:15.600 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011428974999999999, 2021-03-15T22:31:15.600 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.600 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:15.600 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.601 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.601 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.601 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227857+0000", 2021-03-15T22:31:15.601 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.602 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.602 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.602 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.602 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227857+0000", 2021-03-15T22:31:15.602 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.481e-06 2021-03-15T22:31:15.603 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.603 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.603 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.603 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227860+0000", 2021-03-15T22:31:15.603 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8899999999999999e-06 2021-03-15T22:31:15.604 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.604 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.604 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.604 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227863+0000", 2021-03-15T22:31:15.604 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2099999999999999e-06 2021-03-15T22:31:15.605 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.605 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.605 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.605 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227872+0000", 2021-03-15T22:31:15.606 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5270000000000004e-06 2021-03-15T22:31:15.606 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.606 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.606 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.606 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227878+0000", 2021-03-15T22:31:15.607 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0069444199999999998 2021-03-15T22:31:15.607 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.607 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.607 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.607 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.234823+0000", 2021-03-15T22:31:15.608 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1724999999999993e-05 2021-03-15T22:31:15.608 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.608 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.608 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.609 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.234915+0000", 2021-03-15T22:31:15.609 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043717219999999998 2021-03-15T22:31:15.609 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.609 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.609 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.610 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.239286+0000", 2021-03-15T22:31:15.610 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011428974999999999 2021-03-15T22:31:15.610 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.610 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.610 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.611 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.611 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.611 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.3.0:1403 7.a e643/642 7:5595c26f:::smithi07694878-7:185 v 643'914, mlcod=643'910)", 2021-03-15T22:31:15.611 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.246076+0000", 2021-03-15T22:31:15.612 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 83.191657137000007, 2021-03-15T22:31:15.612 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012195887000000001, 2021-03-15T22:31:15.612 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.612 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:15.612 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.613 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.613 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.613 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246076+0000", 2021-03-15T22:31:15.613 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.614 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.614 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.614 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.614 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246076+0000", 2021-03-15T22:31:15.614 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5759999999999999e-06 2021-03-15T22:31:15.615 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.615 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.615 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.615 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246081+0000", 2021-03-15T22:31:15.615 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4699999999999998e-06 2021-03-15T22:31:15.616 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.616 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.616 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.616 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246084+0000", 2021-03-15T22:31:15.616 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9805000000000001e-05 2021-03-15T22:31:15.617 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.617 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.617 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.617 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246104+0000", 2021-03-15T22:31:15.618 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7160000000000003e-06 2021-03-15T22:31:15.618 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.618 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.618 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.618 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.246112+0000", 2021-03-15T22:31:15.619 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00092903500000000004 2021-03-15T22:31:15.619 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.619 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.619 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.619 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.247041+0000", 2021-03-15T22:31:15.620 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0318999999999999e-05 2021-03-15T22:31:15.620 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.620 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.620 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.621 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.247071+0000", 2021-03-15T22:31:15.621 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011167832000000001 2021-03-15T22:31:15.621 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.621 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.621 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:15.622 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258239+0000", 2021-03-15T22:31:15.622 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3133999999999998e-05 2021-03-15T22:31:15.622 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.622 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.622 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.623 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258272+0000", 2021-03-15T22:31:15.623 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012195887000000001 2021-03-15T22:31:15.623 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.623 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.623 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.624 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.624 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.624 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6000 7.0 7:00c03349:::smithi07694878-40:head [copy-from ver 1703] snapc 200=[200,1fb,1f8,1f7,1f6,1f4,1f0,1ee,1e9,1e7,1e6,1e4,1e0,1df,1dc,1d5,1d4,1c7,1c5,1c3,1ae,1a4,1a1,19c,193,17a,176] ondisk+write+known_if_redirected e647)", 2021-03-15T22:31:15.624 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:56.265169+0000", 2021-03-15T22:31:15.625 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 79.172564805999997, 2021-03-15T22:31:15.625 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020808416999999999, 2021-03-15T22:31:15.625 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.625 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:15.625 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.626 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.626 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.626 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6000 2021-03-15T22:31:15.626 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.627 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.627 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.627 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.627 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265169+0000", 2021-03-15T22:31:15.627 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.628 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.628 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.628 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.628 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265169+0000", 2021-03-15T22:31:15.628 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.057e-06 2021-03-15T22:31:15.629 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.629 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.629 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.629 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265174+0000", 2021-03-15T22:31:15.629 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1699999999999999e-06 2021-03-15T22:31:15.630 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.630 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.630 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.630 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265178+0000", 2021-03-15T22:31:15.631 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5769000000000003e-05 2021-03-15T22:31:15.631 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.631 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.631 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.631 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265214+0000", 2021-03-15T22:31:15.632 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.092e-06 2021-03-15T22:31:15.632 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.632 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.632 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.632 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265221+0000", 2021-03-15T22:31:15.633 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.826e-05 2021-03-15T22:31:15.633 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.633 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.633 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.633 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265289+0000", 2021-03-15T22:31:15.634 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014873299999999999 2021-03-15T22:31:15.634 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.634 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.634 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.635 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265438+0000", 2021-03-15T22:31:15.635 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070524159999999997 2021-03-15T22:31:15.635 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.635 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.635 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:31:15.636 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.272490+0000", 2021-03-15T22:31:15.636 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052860520000000003 2021-03-15T22:31:15.636 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.636 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.636 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:15.637 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.277776+0000", 2021-03-15T22:31:15.637 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0080593580000000008 2021-03-15T22:31:15.637 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.637 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.638 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.638 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.285835+0000", 2021-03-15T22:31:15.638 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6378999999999998e-05 2021-03-15T22:31:15.638 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.638 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.639 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.639 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.285872+0000", 2021-03-15T22:31:15.639 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3061e-05 2021-03-15T22:31:15.639 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.640 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.640 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:15.640 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.285915+0000", 2021-03-15T22:31:15.640 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2069999999999994e-05 2021-03-15T22:31:15.640 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.641 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.641 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.641 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.285977+0000", 2021-03-15T22:31:15.641 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020808416999999999 2021-03-15T22:31:15.641 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.642 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.642 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.642 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.642 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.643 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6001 7.0 7:00c03349:::smithi07694878-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e647)", 2021-03-15T22:31:15.643 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:56.265234+0000", 2021-03-15T22:31:15.643 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 79.172499068999997, 2021-03-15T22:31:15.643 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021017456, 2021-03-15T22:31:15.643 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.644 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:15.644 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.644 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.644 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.644 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6001 2021-03-15T22:31:15.645 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.645 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.645 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.645 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.645 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265234+0000", 2021-03-15T22:31:15.646 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.646 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.646 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.646 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.647 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265234+0000", 2021-03-15T22:31:15.647 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2799999999999995e-07 2021-03-15T22:31:15.647 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.647 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.647 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.648 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265235+0000", 2021-03-15T22:31:15.648 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.062e-06 2021-03-15T22:31:15.648 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.648 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.648 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.649 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265236+0000", 2021-03-15T22:31:15.649 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5990000000000001e-06 2021-03-15T22:31:15.649 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.649 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.649 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.650 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265242+0000", 2021-03-15T22:31:15.650 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5379999999999998e-06 2021-03-15T22:31:15.650 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.650 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.651 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.651 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265245+0000", 2021-03-15T22:31:15.651 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000367549 2021-03-15T22:31:15.651 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.651 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.652 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.652 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265613+0000", 2021-03-15T22:31:15.652 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2518e-05 2021-03-15T22:31:15.652 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.652 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.653 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:31:15.653 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265675+0000", 2021-03-15T22:31:15.653 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011675506 2021-03-15T22:31:15.653 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.654 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.654 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.654 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.277351+0000", 2021-03-15T22:31:15.654 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111115 2021-03-15T22:31:15.654 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.655 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.655 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:31:15.655 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.277462+0000", 2021-03-15T22:31:15.655 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085610970000000002 2021-03-15T22:31:15.655 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.656 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.656 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.656 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.286023+0000", 2021-03-15T22:31:15.656 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6027999999999996e-05 2021-03-15T22:31:15.657 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.657 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.657 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.657 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.286119+0000", 2021-03-15T22:31:15.657 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013261600000000001 2021-03-15T22:31:15.658 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.658 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.658 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.658 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.286252+0000", 2021-03-15T22:31:15.659 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021017456 2021-03-15T22:31:15.659 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.659 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.659 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.659 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.660 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.660 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6016 7.0 7:0152dc37:::smithi07694878-28:head [copy-from ver 1320] snapc 200=[200,1fb,1f8,1f7,1f6,1f4,1f0,1ee,1e9,1e7,1e6,1e4,1e0,1df,1dc,1d5,1d4,1c7,1c5,1c3,1ae,1a4,1a1,19c,193,17a,176] ondisk+write+known_if_redirected e647)", 2021-03-15T22:31:15.660 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:56.308050+0000", 2021-03-15T22:31:15.660 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 79.129683432999997, 2021-03-15T22:31:15.661 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016297612999999999, 2021-03-15T22:31:15.661 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.661 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:15.661 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.661 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.662 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.662 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6016 2021-03-15T22:31:15.662 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.662 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.662 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.663 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.663 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308050+0000", 2021-03-15T22:31:15.663 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.663 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.664 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.664 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.664 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308050+0000", 2021-03-15T22:31:15.664 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.732e-06 2021-03-15T22:31:15.664 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.665 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.665 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.665 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308054+0000", 2021-03-15T22:31:15.665 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1010000000000002e-06 2021-03-15T22:31:15.665 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.666 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.666 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.666 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308058+0000", 2021-03-15T22:31:15.666 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3556e-05 2021-03-15T22:31:15.666 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.667 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.667 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.667 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308071+0000", 2021-03-15T22:31:15.667 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4110000000000004e-06 2021-03-15T22:31:15.668 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.668 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.668 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.668 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308077+0000", 2021-03-15T22:31:15.668 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1193999999999997e-05 2021-03-15T22:31:15.669 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.669 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.669 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.669 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308138+0000", 2021-03-15T22:31:15.669 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8767000000000001e-05 2021-03-15T22:31:15.670 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.670 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.670 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.670 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308207+0000", 2021-03-15T22:31:15.670 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0056250479999999997 2021-03-15T22:31:15.671 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.671 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.671 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:31:15.671 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.313832+0000", 2021-03-15T22:31:15.672 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0046612119999999996 2021-03-15T22:31:15.672 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.672 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.672 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:15.672 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.318493+0000", 2021-03-15T22:31:15.673 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.005504119 2021-03-15T22:31:15.673 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.673 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.673 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.673 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.323997+0000", 2021-03-15T22:31:15.674 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0002474 2021-03-15T22:31:15.674 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.674 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.674 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.674 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.324244+0000", 2021-03-15T22:31:15.675 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4350000000000001e-05 2021-03-15T22:31:15.675 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.675 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.675 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:15.676 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.324289+0000", 2021-03-15T22:31:15.676 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8723e-05 2021-03-15T22:31:15.676 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.676 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.676 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.677 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.324348+0000", 2021-03-15T22:31:15.677 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016297612999999999 2021-03-15T22:31:15.677 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.677 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.677 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.678 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.678 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.678 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6017 7.0 7:0152dc37:::smithi07694878-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e647)", 2021-03-15T22:31:15.678 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:56.308088+0000", 2021-03-15T22:31:15.678 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 79.129645191999998, 2021-03-15T22:31:15.679 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016500991999999999, 2021-03-15T22:31:15.679 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.679 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:15.679 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.680 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.680 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.680 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6017 2021-03-15T22:31:15.680 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.680 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.681 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.681 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.681 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308088+0000", 2021-03-15T22:31:15.681 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.681 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.682 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.682 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.682 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308088+0000", 2021-03-15T22:31:15.682 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1999999999999998e-07 2021-03-15T22:31:15.683 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.683 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.683 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.683 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308089+0000", 2021-03-15T22:31:15.683 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.33e-07 2021-03-15T22:31:15.684 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.684 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.684 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.684 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308090+0000", 2021-03-15T22:31:15.684 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5790000000000003e-06 2021-03-15T22:31:15.685 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.685 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.685 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.685 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308095+0000", 2021-03-15T22:31:15.685 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7889999999999999e-06 2021-03-15T22:31:15.686 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.686 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.686 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.686 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308098+0000", 2021-03-15T22:31:15.687 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000236276 2021-03-15T22:31:15.687 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.687 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.687 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.687 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308335+0000", 2021-03-15T22:31:15.688 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3000000000000001e-05 2021-03-15T22:31:15.688 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.688 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.688 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:31:15.688 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.308388+0000", 2021-03-15T22:31:15.689 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0092782669999999998 2021-03-15T22:31:15.689 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.689 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.689 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.689 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.317666+0000", 2021-03-15T22:31:15.690 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010539699999999999 2021-03-15T22:31:15.690 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.690 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.690 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:31:15.691 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.317771+0000", 2021-03-15T22:31:15.691 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066114540000000001 2021-03-15T22:31:15.691 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.691 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.691 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.692 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.324383+0000", 2021-03-15T22:31:15.692 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0736999999999996e-05 2021-03-15T22:31:15.692 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.692 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.692 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.693 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.324463+0000", 2021-03-15T22:31:15.693 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012574 2021-03-15T22:31:15.693 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.693 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.693 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.694 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.324589+0000", 2021-03-15T22:31:15.694 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016500991999999999 2021-03-15T22:31:15.694 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.694 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.695 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.695 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.695 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.695 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6168 7.0 7:054fa7db:::smithi07694878-29:head [append 0~0] snapc 20e=[20e,20c,20a,209,207,205,200,1fb,1f8,1f7,1f6,1f4,1e9,1e7,1e6,1e4,1e0,1dc,1d5,1c7,1c5,1a1,17a,176] ondisk+write+known_if_redirected e662)", 2021-03-15T22:31:15.695 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:30:11.244016+0000", 2021-03-15T22:31:15.696 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 64.193717098999997, 2021-03-15T22:31:15.696 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013920500000000001, 2021-03-15T22:31:15.696 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.696 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:15.696 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.697 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.697 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.697 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6168 2021-03-15T22:31:15.697 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.698 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.698 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.698 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.698 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244016+0000", 2021-03-15T22:31:15.698 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.699 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.699 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.699 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.699 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244016+0000", 2021-03-15T22:31:15.699 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.7369999999999998e-06 2021-03-15T22:31:15.700 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.700 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.700 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244019+0000", 2021-03-15T22:31:15.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0450000000000001e-06 2021-03-15T22:31:15.701 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.701 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.701 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.701 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244027+0000", 2021-03-15T22:31:15.701 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0474000000000001e-05 2021-03-15T22:31:15.702 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.702 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.702 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.702 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244047+0000", 2021-03-15T22:31:15.703 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8009999999999993e-06 2021-03-15T22:31:15.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.703 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.703 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.703 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244056+0000", 2021-03-15T22:31:15.704 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015393780000000001 2021-03-15T22:31:15.704 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.704 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.704 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.704 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.245596+0000", 2021-03-15T22:31:15.705 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9499000000000004e-05 2021-03-15T22:31:15.705 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.705 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.705 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.245695+0000", 2021-03-15T22:31:15.706 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025377199999999998 2021-03-15T22:31:15.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.706 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.706 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:31:15.707 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.245949+0000", 2021-03-15T22:31:15.707 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0030013700000000002 2021-03-15T22:31:15.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.707 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.707 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:15.708 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.248950+0000", 2021-03-15T22:31:15.708 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052695149999999998 2021-03-15T22:31:15.708 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.708 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.708 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.709 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.254220+0000", 2021-03-15T22:31:15.709 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0035665359999999999 2021-03-15T22:31:15.709 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.709 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.709 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.710 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.257786+0000", 2021-03-15T22:31:15.710 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3998000000000006e-05 2021-03-15T22:31:15.710 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.710 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.711 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:15.711 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.257850+0000", 2021-03-15T22:31:15.711 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6374999999999999e-05 2021-03-15T22:31:15.711 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.711 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.712 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.712 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.257937+0000", 2021-03-15T22:31:15.712 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013920500000000001 2021-03-15T22:31:15.712 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.712 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.713 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.713 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.713 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.713 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6169 7.0 7:054fa7db:::smithi07694878-29:head [rollback 207] snapc 20e=[20e,20c,20a,209,207,205,200,1fb,1f8,1f7,1f6,1f4,1e9,1e7,1e6,1e4,1e0,1dc,1d5,1c7,1c5,1a1,17a,176] ondisk+write+known_if_redirected e662)", 2021-03-15T22:31:15.713 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:30:11.244078+0000", 2021-03-15T22:31:15.714 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 64.193655566999993, 2021-03-15T22:31:15.714 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014036626, 2021-03-15T22:31:15.714 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.714 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:15.714 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.715 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.715 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.715 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6169 2021-03-15T22:31:15.715 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.716 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.716 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.716 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.716 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244078+0000", 2021-03-15T22:31:15.716 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.717 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.717 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.717 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.717 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244078+0000", 2021-03-15T22:31:15.717 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.435e-06 2021-03-15T22:31:15.718 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.718 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.718 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.718 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244079+0000", 2021-03-15T22:31:15.718 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4929999999999999e-06 2021-03-15T22:31:15.719 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.719 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.719 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.719 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244081+0000", 2021-03-15T22:31:15.720 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5189999999999998e-06 2021-03-15T22:31:15.720 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.720 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.720 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.720 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244090+0000", 2021-03-15T22:31:15.721 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6799999999999998e-06 2021-03-15T22:31:15.721 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.721 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.721 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.721 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244096+0000", 2021-03-15T22:31:15.722 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028994329999999999 2021-03-15T22:31:15.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.722 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.722 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.722 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.246995+0000", 2021-03-15T22:31:15.723 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2089999999999997e-05 2021-03-15T22:31:15.723 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.723 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.723 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.723 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.247057+0000", 2021-03-15T22:31:15.724 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016239600000000001 2021-03-15T22:31:15.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.724 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.724 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:31:15.725 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.247220+0000", 2021-03-15T22:31:15.725 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0017364679999999999 2021-03-15T22:31:15.725 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.725 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.725 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:15.726 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.248956+0000", 2021-03-15T22:31:15.726 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055487389999999996 2021-03-15T22:31:15.726 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.726 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.726 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.727 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.254505+0000", 2021-03-15T22:31:15.727 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003478054 2021-03-15T22:31:15.727 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.727 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.727 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.728 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.257983+0000", 2021-03-15T22:31:15.728 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9197999999999997e-05 2021-03-15T22:31:15.728 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.728 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.729 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:15.729 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.258052+0000", 2021-03-15T22:31:15.729 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2120999999999997e-05 2021-03-15T22:31:15.729 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.729 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.730 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.730 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.258114+0000", 2021-03-15T22:31:15.730 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014036626 2021-03-15T22:31:15.730 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.730 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.731 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.731 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.731 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.731 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6170 7.0 7:054fa7db:::smithi07694878-29:head [append 0~0] snapc 20e=[20e,20c,20a,209,207,205,200,1fb,1f8,1f7,1f6,1f4,1e9,1e7,1e6,1e4,1e0,1dc,1d5,1c7,1c5,1a1,17a,176] ondisk+write+known_if_redirected e662)", 2021-03-15T22:31:15.731 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:30:11.244105+0000", 2021-03-15T22:31:15.732 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 64.193628783999998, 2021-03-15T22:31:15.732 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014193417, 2021-03-15T22:31:15.732 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.732 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:15.733 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.733 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.733 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.733 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6170 2021-03-15T22:31:15.733 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.734 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.734 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.734 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.734 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244105+0000", 2021-03-15T22:31:15.735 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.735 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.735 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.735 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.735 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244105+0000", 2021-03-15T22:31:15.736 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.331e-06 2021-03-15T22:31:15.736 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.736 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.736 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.736 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244106+0000", 2021-03-15T22:31:15.737 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3209999999999999e-06 2021-03-15T22:31:15.737 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.737 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.737 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.737 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244107+0000", 2021-03-15T22:31:15.738 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9619999999999993e-06 2021-03-15T22:31:15.738 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.738 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.738 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.739 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244115+0000", 2021-03-15T22:31:15.739 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5799999999999999e-06 2021-03-15T22:31:15.739 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.739 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.739 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.740 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.244118+0000", 2021-03-15T22:31:15.740 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038183990000000001 2021-03-15T22:31:15.740 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.740 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.740 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.741 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.247936+0000", 2021-03-15T22:31:15.741 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9586000000000005e-05 2021-03-15T22:31:15.741 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.741 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.742 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.742 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.248036+0000", 2021-03-15T22:31:15.742 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020503900000000001 2021-03-15T22:31:15.742 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.742 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.743 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:31:15.743 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.248241+0000", 2021-03-15T22:31:15.743 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000801712 2021-03-15T22:31:15.743 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.743 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.744 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:15.744 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.249042+0000", 2021-03-15T22:31:15.744 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0054849219999999997 2021-03-15T22:31:15.744 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.745 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.745 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.745 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.254527+0000", 2021-03-15T22:31:15.745 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0036509680000000001 2021-03-15T22:31:15.745 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.746 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.746 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.746 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.258178+0000", 2021-03-15T22:31:15.746 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7742999999999999e-05 2021-03-15T22:31:15.746 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.747 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.747 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:15.747 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.258236+0000", 2021-03-15T22:31:15.747 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1853999999999994e-05 2021-03-15T22:31:15.747 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.748 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.748 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.748 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.258298+0000", 2021-03-15T22:31:15.748 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014193417 2021-03-15T22:31:15.748 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.749 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.749 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.749 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.749 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.750 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6463 7.0 7:08414d91:::smithi07694878-42:head [delete] snapc 22b=[22b,229,228,227,226,225,223,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1a1,17a,176] ondisk+write+peerstat_old+known_if_redirected e692)", 2021-03-15T22:31:15.750 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:30:41.223823+0000", 2021-03-15T22:31:15.750 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 34.213910528, 2021-03-15T22:31:15.750 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01168073, 2021-03-15T22:31:15.750 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.751 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:15.751 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.751 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.751 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.751 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6463 2021-03-15T22:31:15.752 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.752 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.752 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.752 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.752 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.223823+0000", 2021-03-15T22:31:15.753 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.753 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.753 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.753 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.754 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.223823+0000", 2021-03-15T22:31:15.754 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2229999999999999e-06 2021-03-15T22:31:15.754 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.754 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.754 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.755 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.223826+0000", 2021-03-15T22:31:15.755 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9110000000000002e-06 2021-03-15T22:31:15.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.755 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.756 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.223829+0000", 2021-03-15T22:31:15.756 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4603999999999999e-05 2021-03-15T22:31:15.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.757 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.757 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.223844+0000", 2021-03-15T22:31:15.757 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7260000000000002e-06 2021-03-15T22:31:15.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.758 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.758 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.223851+0000", 2021-03-15T22:31:15.758 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0043000000000002e-05 2021-03-15T22:31:15.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.759 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.759 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.223901+0000", 2021-03-15T22:31:15.759 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0139999999999998e-06 2021-03-15T22:31:15.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.760 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:31:15.760 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.223906+0000", 2021-03-15T22:31:15.760 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027157510000000002 2021-03-15T22:31:15.760 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.760 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.761 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.761 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.226622+0000", 2021-03-15T22:31:15.761 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1450000000000002e-05 2021-03-15T22:31:15.761 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.762 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.762 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.762 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.226694+0000", 2021-03-15T22:31:15.762 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017715799999999999 2021-03-15T22:31:15.762 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.763 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.763 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:31:15.763 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.226871+0000", 2021-03-15T22:31:15.763 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0083217039999999992 2021-03-15T22:31:15.763 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.764 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.764 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.764 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.235192+0000", 2021-03-15T22:31:15.764 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013983600000000001 2021-03-15T22:31:15.764 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.765 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.765 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:15.765 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.235332+0000", 2021-03-15T22:31:15.765 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4502999999999997e-05 2021-03-15T22:31:15.766 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.766 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.766 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.766 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.235377+0000", 2021-03-15T22:31:15.766 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9740999999999995e-05 2021-03-15T22:31:15.767 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.767 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.767 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:15.767 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.235446+0000", 2021-03-15T22:31:15.767 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7065999999999999e-05 2021-03-15T22:31:15.768 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.768 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.768 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.768 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:41.235504+0000", 2021-03-15T22:31:15.768 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01168073 2021-03-15T22:31:15.769 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.769 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.769 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.769 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.770 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.770 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:6618 7.a e714/684 7:5595c26f:::smithi07694878-7:head v 714'1079, mlcod=714'1077)", 2021-03-15T22:31:15.770 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:31:03.272075+0000", 2021-03-15T22:31:15.770 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 12.165658755999999, 2021-03-15T22:31:15.770 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0098718829999999997, 2021-03-15T22:31:15.771 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.771 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:15.771 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.771 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.771 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.772 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:03.272075+0000", 2021-03-15T22:31:15.772 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.772 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.772 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.773 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.773 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:03.272075+0000", 2021-03-15T22:31:15.773 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041210980000000001 2021-03-15T22:31:15.773 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.773 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.774 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.774 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:03.276196+0000", 2021-03-15T22:31:15.774 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5490000000000001e-06 2021-03-15T22:31:15.774 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.774 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.775 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.775 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:03.276203+0000", 2021-03-15T22:31:15.775 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3935999999999999e-05 2021-03-15T22:31:15.775 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.775 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.776 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.776 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:03.276227+0000", 2021-03-15T22:31:15.776 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5389999999999996e-06 2021-03-15T22:31:15.776 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.777 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.777 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.777 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:03.276237+0000", 2021-03-15T22:31:15.777 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7073000000000001e-05 2021-03-15T22:31:15.777 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.778 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.778 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.778 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:03.276294+0000", 2021-03-15T22:31:15.778 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4598000000000001e-05 2021-03-15T22:31:15.778 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.779 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.779 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.779 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:03.276318+0000", 2021-03-15T22:31:15.779 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055837999999999999 2021-03-15T22:31:15.779 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.780 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.780 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:15.780 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:03.281902+0000", 2021-03-15T22:31:15.780 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.4289999999999998e-05 2021-03-15T22:31:15.781 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.781 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.781 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.781 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:03.281946+0000", 2021-03-15T22:31:15.781 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0098718829999999997 2021-03-15T22:31:15.782 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.782 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.782 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.783 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6670 7.0 7:00c03349:::smithi07694878-40:head [copy-from ver 1517] snapc 245=[245,243,241,23e,23d,23c,239,235,234,22e,22d,22b,227,225,223,21f,218,215,211,20c,200,1f7,1e9,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e717)", 2021-03-15T22:31:15.783 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:31:06.237882+0000", 2021-03-15T22:31:15.783 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 9.1998510880000008, 2021-03-15T22:31:15.783 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020450328, 2021-03-15T22:31:15.783 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.784 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:31:15.784 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.784 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.784 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.785 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6670 2021-03-15T22:31:15.785 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.785 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.785 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.785 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.786 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237882+0000", 2021-03-15T22:31:15.786 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.786 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.786 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.786 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.787 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237882+0000", 2021-03-15T22:31:15.787 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5140000000000003e-06 2021-03-15T22:31:15.787 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.787 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.787 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.788 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237887+0000", 2021-03-15T22:31:15.788 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.371e-06 2021-03-15T22:31:15.788 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.788 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.789 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.789 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237890+0000", 2021-03-15T22:31:15.789 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7682000000000001e-05 2021-03-15T22:31:15.789 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.789 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.790 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.790 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237908+0000", 2021-03-15T22:31:15.790 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.002e-06 2021-03-15T22:31:15.790 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.790 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.791 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.791 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237915+0000", 2021-03-15T22:31:15.791 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1247000000000005e-05 2021-03-15T22:31:15.791 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.791 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.792 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.792 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237976+0000", 2021-03-15T22:31:15.792 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.081e-06 2021-03-15T22:31:15.792 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.793 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.793 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:31:15.793 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237985+0000", 2021-03-15T22:31:15.793 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001105314 2021-03-15T22:31:15.793 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.794 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.794 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.794 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.239090+0000", 2021-03-15T22:31:15.794 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000120964 2021-03-15T22:31:15.794 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.795 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.795 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.795 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.239211+0000", 2021-03-15T22:31:15.795 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055616399999999996 2021-03-15T22:31:15.795 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.796 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.796 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:31:15.796 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.244773+0000", 2021-03-15T22:31:15.796 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045633130000000003 2021-03-15T22:31:15.796 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.797 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.797 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:31:15.797 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.249336+0000", 2021-03-15T22:31:15.797 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0088461189999999995 2021-03-15T22:31:15.798 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.798 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.798 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.798 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258182+0000", 2021-03-15T22:31:15.798 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7354000000000002e-05 2021-03-15T22:31:15.799 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.799 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.799 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:31:15.799 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258220+0000", 2021-03-15T22:31:15.799 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5302000000000001e-05 2021-03-15T22:31:15.800 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.800 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.800 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:31:15.800 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258265+0000", 2021-03-15T22:31:15.800 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7424999999999999e-05 2021-03-15T22:31:15.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.801 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.801 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258333+0000", 2021-03-15T22:31:15.801 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020450328 2021-03-15T22:31:15.802 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.802 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.802 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.802 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.803 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.803 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6671 7.0 7:00c03349:::smithi07694878-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e717)", 2021-03-15T22:31:15.803 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:31:06.237929+0000", 2021-03-15T22:31:15.803 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 9.1998043739999993, 2021-03-15T22:31:15.803 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020548578000000001, 2021-03-15T22:31:15.804 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:31:15.804 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:31:15.804 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:31:15.804 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:31:15.804 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:31:15.805 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6671 2021-03-15T22:31:15.805 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.805 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:31:15.805 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.805 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:31:15.806 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237929+0000", 2021-03-15T22:31:15.806 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:31:15.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.806 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.807 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:31:15.807 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237929+0000", 2021-03-15T22:31:15.807 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.04e-06 2021-03-15T22:31:15.807 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.807 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.808 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:31:15.808 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237930+0000", 2021-03-15T22:31:15.808 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.491e-06 2021-03-15T22:31:15.808 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.808 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.809 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:31:15.809 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237931+0000", 2021-03-15T22:31:15.809 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1251e-05 2021-03-15T22:31:15.809 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.810 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:31:15.810 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237953+0000", 2021-03-15T22:31:15.810 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7879999999999997e-06 2021-03-15T22:31:15.810 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.811 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.811 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:31:15.811 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237958+0000", 2021-03-15T22:31:15.811 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016104300000000001 2021-03-15T22:31:15.811 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.812 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.812 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.238119+0000", 2021-03-15T22:31:15.812 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0128e-05 2021-03-15T22:31:15.812 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.813 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.813 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:31:15.813 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.238129+0000", 2021-03-15T22:31:15.813 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012166189999999999 2021-03-15T22:31:15.813 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.814 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.814 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.239345+0000", 2021-03-15T22:31:15.814 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3352e-05 2021-03-15T22:31:15.815 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.815 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:31:15.815 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.239399+0000", 2021-03-15T22:31:15.815 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0091340589999999999 2021-03-15T22:31:15.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.816 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.816 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.248533+0000", 2021-03-15T22:31:15.816 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5486e-05 2021-03-15T22:31:15.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.817 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.817 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:31:15.817 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.248628+0000", 2021-03-15T22:31:15.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0097252959999999996 2021-03-15T22:31:15.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.818 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:31:15.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258354+0000", 2021-03-15T22:31:15.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4283999999999997e-05 2021-03-15T22:31:15.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:31:15.819 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258408+0000", 2021-03-15T22:31:15.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9740999999999995e-05 2021-03-15T22:31:15.820 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:31:15.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:31:15.820 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:31:15.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258478+0000", 2021-03-15T22:31:15.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020548578000000001 2021-03-15T22:31:15.821 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.821 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.821 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.821 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:31:15.822 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:31:15.822 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:31:15.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:15.824 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[56514]: audit 2021-03-15T22:31:14.485361+0000 mon.a ( 2021-03-15T22:31:15.824 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[56514]: mon.0) 1933 : audit [DBG] from='client.? 172.21.15.76:0/2737908815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:15.824 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[56514]: audit 2021-03-15T22:31:15.228835+0000 mon.a (mon.0) 1934 : audit [INF] from='client.? 172.21.15.76:0/4185851685' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]': finished 2021-03-15T22:31:15.824 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[56514]: cluster 2021-03-15T22:31:15.228910+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2021-03-15T22:31:15.825 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:15 smithi125 conmon[56739]: audit 2021-03-15T22:31:14.485361+0000 mon.a (mon.0) 1933 : audit [DBG] from='client.? 172.21.15.76:0/2737908815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:15.825 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:15 smithi125 conmon[56739]: audit 2021-03-15T22:31:15.228835+0000 mon.a (mon.0) 1934 : audit [INF] from='client.? 172.21.15.76:0/4185851685' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]': finished 2021-03-15T22:31:15.825 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:15 smithi125 conmon[56739]: cluster 2021-03-15T22:31:15.228910+0000 mon.a (mon.0) 1935 : cluster [DBG] osdmap e726: 8 total, 8 up, 8 in 2021-03-15T22:31:15.826 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: audit 2021-03-15T22:31:15.826 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: 2021-03-15T22:31:15.826 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: 2021 2021-03-15T22:31:15.826 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: - 2021-03-15T22:31:15.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: 03-15T22:31:14.485361+0000 2021-03-15T22:31:15.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: mon.a (mon.0) 1933 : audit [DBG] from='client.? 172.21.15.76:0/2737908815' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:15.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: audit 2021-03-15T22:31:15.228835 2021-03-15T22:31:15.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1934 : audit [INF] from='client.? 172.21.15.76:0/4185851685' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num", "val": "32"}]': finished 2021-03-15T22:31:15.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: 2021-03-15T22:31:15.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: cluster 2021-03-15T22:31:15. 2021-03-15T22:31:15.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: 228910+0000 mon.a (mon.0) 1935 : cluster 2021-03-15T22:31:15.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:15 smithi076 conmon[62156]: [DBG] osdmap e726: 8 total, 8 up, 8 in 2021-03-15T22:31:15.890 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:15.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:15.892 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:15.950 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:15.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:16.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:16.020 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:16.020 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:16.034 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-15T22:31:16.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:16.145 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:16.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:16.146 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:16.243 INFO:tasks.rados.rados.0.smithi076.stdout:2653: done (1 left) 2021-03-15T22:31:16.243 INFO:tasks.rados.rados.0.smithi076.stdout:2654: done (0 left) 2021-03-15T22:31:16.243 INFO:tasks.rados.rados.0.smithi076.stdout:2655: delete oid 29 current snap is 312 2021-03-15T22:31:16.248 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:16.249 INFO:tasks.rados.rados.0.smithi076.stdout:2655: done (0 left) 2021-03-15T22:31:16.250 INFO:tasks.rados.rados.0.smithi076.stdout:2656: snap_remove snap 282 2021-03-15T22:31:16.272 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:16.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:16.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:16.375 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:16.397 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:16.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:16.399 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:16.434 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:16.500 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:16.524 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:16.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:16.525 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:16.626 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:16.649 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:16.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:16.651 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:16.752 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:16.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:16 smithi076 conmon[56514]: cluster 2021-03-15T22:31:15.065659+0000 osd.1 (osd.1) 141 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:16.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:16 smithi076 conmon[56514]: cluster 2021-03-15T22:31:15.067830+0000 osd.1 (osd.1) 142 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:16 smithi076 conmon[56514]: cluster 2021-03-15T22:31:15.157249+0000 mgr.y (mgr.14144) 883 : cluster [DBG] pgmap v1232: 138 pgs: 138 active+clean; 582 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 4 op/s 2021-03-15T22:31:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:16 smithi076 conmon[56514]: cluster 2021-03-15T22:31:16.239986+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2021-03-15T22:31:16.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:16 smithi076 conmon[62156]: cluster 2021-03-15T22:31:15.065659+0000 osd.1 (osd.1) 141 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:16.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:16 smithi076 conmon[62156]: cluster 2021-03-15T22:31:15.067830+0000 osd.1 (osd.1) 142 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:16.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:16 smithi076 conmon[62156]: cluster 2021-03-15T22:31:15.157249+0000 mgr.y (mgr.14144) 883 : cluster [DBG] pgmap v1232: 138 pgs: 138 active+clean; 582 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 4 op/s 2021-03-15T22:31:16.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:16 smithi076 conmon[62156]: cluster 2021-03-15T22:31:16.239986+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2021-03-15T22:31:16.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:16.783 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:16.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:16.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:16 smithi125 conmon[56739]: cluster 2021-03-15T22:31:15.065659+0000 osd.1 (osd.1) 141 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:16.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:16 smithi125 conmon[56739]: cluster 2021-03-15T22:31:15.067830+0000 osd.1 (osd.1) 142 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:16 smithi125 conmon[56739]: cluster 2021-03-15T22:31:15.157249+0000 mgr.y (mgr.14144) 883 : cluster [DBG] pgmap v1232: 138 pgs: 138 active+clean; 582 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 0 B/s wr, 4 op/s 2021-03-15T22:31:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:16 smithi125 conmon[56739]: cluster 2021-03-15T22:31:16.239986+0000 mon.a (mon.0) 1936 : cluster [DBG] osdmap e727: 8 total, 8 up, 8 in 2021-03-15T22:31:16.885 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:16.911 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:16.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:16.913 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:17.014 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:17.036 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:17.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:17.037 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:17.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:17.164 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:17.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:17.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:17.240 INFO:tasks.rados.rados.0.smithi076.stdout:2656: done (0 left) 2021-03-15T22:31:17.241 INFO:tasks.rados.rados.0.smithi076.stdout:2657: rollback oid 49 current snap is 312 2021-03-15T22:31:17.241 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 49 to 307 2021-03-15T22:31:17.242 INFO:tasks.rados.rados.0.smithi076.stdout:2658: rollback oid 48 current snap is 312 2021-03-15T22:31:17.242 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 48 to 255 2021-03-15T22:31:17.242 INFO:tasks.rados.rados.0.smithi076.stdout:2659: write oid 5 current snap is 312 2021-03-15T22:31:17.242 INFO:tasks.rados.rados.0.smithi076.stdout:2659: seq_num 920 ranges {662440=507793,1702728=749973,2881271=557193} 2021-03-15T22:31:17.246 INFO:tasks.rados.rados.0.smithi076.stdout:2659: writing smithi07694878-5 from 662440 to 1170233 tid 1 2021-03-15T22:31:17.254 INFO:tasks.rados.rados.0.smithi076.stdout:2659: writing smithi07694878-5 from 1702728 to 2452701 tid 2 2021-03-15T22:31:17.261 INFO:tasks.rados.rados.0.smithi076.stdout:2659: writing smithi07694878-5 from 2881271 to 3438464 tid 3 2021-03-15T22:31:17.261 INFO:tasks.rados.rados.0.smithi076.stdout:2660: rollback oid 35 current snap is 312 2021-03-15T22:31:17.261 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 35 to 273 2021-03-15T22:31:17.261 INFO:tasks.rados.rados.0.smithi076.stdout:2659: finishing write tid 1 to smithi07694878-5 2021-03-15T22:31:17.262 INFO:tasks.rados.rados.0.smithi076.stdout:2658: finishing rollback tid 0 to smithi07694878-48 2021-03-15T22:31:17.262 INFO:tasks.rados.rados.0.smithi076.stdout:2658: finishing rollback tid 1 to smithi07694878-48 2021-03-15T22:31:17.262 INFO:tasks.rados.rados.0.smithi076.stdout:2658: finishing rollback tid 2 to smithi07694878-48 2021-03-15T22:31:17.263 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 1231 (ObjNum 757 snap 250 seq_num 757) dirty exists 2021-03-15T22:31:17.263 INFO:tasks.rados.rados.0.smithi076.stdout:2658: done (3 left) 2021-03-15T22:31:17.263 INFO:tasks.rados.rados.0.smithi076.stdout:2661: write (excl) oid 7 current snap is 312 2021-03-15T22:31:17.263 INFO:tasks.rados.rados.0.smithi076.stdout:2661: seq_num 921 ranges {679247=583042,2044857=609984,3192341=286906} 2021-03-15T22:31:17.268 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:17.269 INFO:tasks.rados.rados.0.smithi076.stdout:2661: writing smithi07694878-7 from 679247 to 1262289 tid 1 2021-03-15T22:31:17.276 INFO:tasks.rados.rados.0.smithi076.stdout:2661: writing smithi07694878-7 from 2044857 to 2654841 tid 2 2021-03-15T22:31:17.281 INFO:tasks.rados.rados.0.smithi076.stdout:2661: writing smithi07694878-7 from 3192341 to 3479247 tid 3 2021-03-15T22:31:17.281 INFO:tasks.rados.rados.0.smithi076.stdout:2662: snap_remove snap 285 2021-03-15T22:31:17.293 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:17.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:17.396 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:17.419 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:17.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:17.421 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:17.522 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:17.545 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:17.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:17.547 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:17.648 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:17.670 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:17.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:17.672 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:17.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[56514]: audit 2021-03-15T22:31:17.158426+0000 mon.a (mon. 2021-03-15T22:31:17.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[56514]: 0) 1937 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]: dispatch 2021-03-15T22:31:17.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[56514]: audit 2021-03-15T22:31:17.235853+0000 mon.a (mon.0) 1938 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]': finished 2021-03-15T22:31:17.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[56514]: cluster 2021-03-15T22:31:17.235903+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2021-03-15T22:31:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[62156]: audit 2021-03-15T22:31 2021-03-15T22:31:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[62156]: :17.158426+0000 mon.a (mon.0) 1937 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]: dispatch 2021-03-15T22:31:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[62156]: audit 2021 2021-03-15T22:31:17.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[62156]: -03-15T22:31:17 2021-03-15T22:31:17.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[62156]: .235853+0000 mon.a (mon.0) 2021-03-15T22:31:17.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[62156]: 1938 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]': finished 2021-03-15T22:31:17.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:31:17.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[62156]: -03-15T22:31: 2021-03-15T22:31:17.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[62156]: 17.235903+0000 mon.a (mon 2021-03-15T22:31:17.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:17 smithi076 conmon[62156]: .0) 1939 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2021-03-15T22:31:17.773 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:17.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:17 smithi125 conmon[56739]: audit 2021-03-15T22:31:17.158426+0000 mon.a (mon.0) 1937 : 2021-03-15T22:31:17.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:17 smithi125 conmon[56739]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]: dispatch 2021-03-15T22:31:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:17 smithi125 conmon[56739]: audit 2021-03-15T22:31:17.235853+0000 mon.a (mon.0) 1938 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pg_num_actual", "val": "32"}]': finished 2021-03-15T22:31:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:17 smithi125 conmon[56739]: cluster 2021-03-15T22:31:17.235903+0000 mon.a (mon.0) 1939 : cluster [DBG] osdmap e728: 8 total, 8 up, 8 in 2021-03-15T22:31:17.795 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:17.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:17.796 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:17.830 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:31:17.898 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:17.920 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:17.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:17.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:18.023 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:18.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:18.048 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:18.049 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:18.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:18.176 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:18.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:18.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:18.247 INFO:tasks.rados.rados.0.smithi076.stdout:2662: done (4 left) 2021-03-15T22:31:18.247 INFO:tasks.rados.rados.0.smithi076.stdout:2663: write oid 2 current snap is 312 2021-03-15T22:31:18.248 INFO:tasks.rados.rados.0.smithi076.stdout:2663: seq_num 922 ranges {696054=658291,2103339=753642} 2021-03-15T22:31:18.255 INFO:tasks.rados.rados.0.smithi076.stdout:2663: writing smithi07694878-2 from 696054 to 1354345 tid 1 2021-03-15T22:31:18.265 INFO:tasks.rados.rados.0.smithi076.stdout:2663: writing smithi07694878-2 from 2103339 to 2856981 tid 2 2021-03-15T22:31:18.265 INFO:tasks.rados.rados.0.smithi076.stdout:2657: finishing rollback tid 0 to smithi07694878-49 2021-03-15T22:31:18.266 INFO:tasks.rados.rados.0.smithi076.stdout:2659: finishing write tid 2 to smithi07694878-5 2021-03-15T22:31:18.266 INFO:tasks.rados.rados.0.smithi076.stdout:2659: finishing write tid 3 to smithi07694878-5 2021-03-15T22:31:18.266 INFO:tasks.rados.rados.0.smithi076.stdout:2657: finishing rollback tid 1 to smithi07694878-49 2021-03-15T22:31:18.266 INFO:tasks.rados.rados.0.smithi076.stdout:2657: finishing rollback tid 2 to smithi07694878-49 2021-03-15T22:31:18.267 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 2048 (ObjNum 862 snap 285 seq_num 862) dirty exists 2021-03-15T22:31:18.268 INFO:tasks.rados.rados.0.smithi076.stdout:2659: finishing write tid 4 to smithi07694878-5 2021-03-15T22:31:18.268 INFO:tasks.rados.rados.0.smithi076.stdout:2659: finishing write tid 5 to smithi07694878-5 2021-03-15T22:31:18.268 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1521 (ObjNum 920 snap 312 seq_num 920) dirty exists 2021-03-15T22:31:18.269 INFO:tasks.rados.rados.0.smithi076.stdout:2659: left oid 5 (ObjNum 920 snap 312 seq_num 920) 2021-03-15T22:31:18.269 INFO:tasks.rados.rados.0.smithi076.stdout:2660: finishing rollback tid 1 to smithi07694878-35 2021-03-15T22:31:18.269 INFO:tasks.rados.rados.0.smithi076.stdout:2660: finishing rollback tid 2 to smithi07694878-35 2021-03-15T22:31:18.269 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 2074 (ObjNum 785 snap 261 seq_num 785) dirty exists 2021-03-15T22:31:18.270 INFO:tasks.rados.rados.0.smithi076.stdout:2661: finishing write tid 1 to smithi07694878-7 2021-03-15T22:31:18.270 INFO:tasks.rados.rados.0.smithi076.stdout:2661: finishing write tid 2 to smithi07694878-7 2021-03-15T22:31:18.270 INFO:tasks.rados.rados.0.smithi076.stdout:2661: finishing write tid 3 to smithi07694878-7 2021-03-15T22:31:18.270 INFO:tasks.rados.rados.0.smithi076.stdout:2661: finishing write tid 4 to smithi07694878-7 2021-03-15T22:31:18.270 INFO:tasks.rados.rados.0.smithi076.stdout:2661: finishing write tid 5 to smithi07694878-7 2021-03-15T22:31:18.271 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2078 (ObjNum 921 snap 312 seq_num 921) dirty exists 2021-03-15T22:31:18.271 INFO:tasks.rados.rados.0.smithi076.stdout:2661: left oid 7 (ObjNum 921 snap 312 seq_num 921) 2021-03-15T22:31:18.271 INFO:tasks.rados.rados.0.smithi076.stdout:2657: done (4 left) 2021-03-15T22:31:18.271 INFO:tasks.rados.rados.0.smithi076.stdout:2659: done (3 left) 2021-03-15T22:31:18.271 INFO:tasks.rados.rados.0.smithi076.stdout:2660: done (2 left) 2021-03-15T22:31:18.272 INFO:tasks.rados.rados.0.smithi076.stdout:2661: done (1 left) 2021-03-15T22:31:18.272 INFO:tasks.rados.rados.0.smithi076.stdout:2664: delete oid 9 current snap is 312 2021-03-15T22:31:18.272 INFO:tasks.rados.rados.0.smithi076.stdout:2663: finishing write tid 1 to smithi07694878-2 2021-03-15T22:31:18.273 INFO:tasks.rados.rados.0.smithi076.stdout:2663: finishing write tid 2 to smithi07694878-2 2021-03-15T22:31:18.273 INFO:tasks.rados.rados.0.smithi076.stdout:2663: finishing write tid 3 to smithi07694878-2 2021-03-15T22:31:18.274 INFO:tasks.rados.rados.0.smithi076.stdout:2663: finishing write tid 4 to smithi07694878-2 2021-03-15T22:31:18.274 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1766 (ObjNum 922 snap 312 seq_num 922) dirty exists 2021-03-15T22:31:18.274 INFO:tasks.rados.rados.0.smithi076.stdout:2663: left oid 2 (ObjNum 922 snap 312 seq_num 922) 2021-03-15T22:31:18.276 INFO:tasks.rados.rados.0.smithi076.stdout:2663: done (1 left) 2021-03-15T22:31:18.276 INFO:tasks.rados.rados.0.smithi076.stdout:2664: done (0 left) 2021-03-15T22:31:18.276 INFO:tasks.rados.rados.0.smithi076.stdout:2665: write (excl) oid 5 current snap is 312 2021-03-15T22:31:18.276 INFO:tasks.rados.rados.0.smithi076.stdout:2665: seq_num 924 ranges {729668=408789,1820303=757311,3274610=255058} 2021-03-15T22:31:18.278 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:18.282 INFO:tasks.rados.rados.0.smithi076.stdout:2665: writing smithi07694878-5 from 729668 to 1138457 tid 1 2021-03-15T22:31:18.290 INFO:tasks.rados.rados.0.smithi076.stdout:2665: writing smithi07694878-5 from 1820303 to 2577614 tid 2 2021-03-15T22:31:18.296 INFO:tasks.rados.rados.0.smithi076.stdout:2665: writing smithi07694878-5 from 3274610 to 3529668 tid 3 2021-03-15T22:31:18.296 INFO:tasks.rados.rados.0.smithi076.stdout:2666: read oid 36 snap -1 2021-03-15T22:31:18.297 INFO:tasks.rados.rados.0.smithi076.stdout:2666: expect (ObjNum 918 snap 311 seq_num 918) 2021-03-15T22:31:18.297 INFO:tasks.rados.rados.0.smithi076.stdout:2667: copy_from oid 4 from oid 6 current snap is 312 2021-03-15T22:31:18.297 INFO:tasks.rados.rados.0.smithi076.stdout:2665: finishing write tid 1 to smithi07694878-5 2021-03-15T22:31:18.297 INFO:tasks.rados.rados.0.smithi076.stdout:2665: finishing write tid 2 to smithi07694878-5 2021-03-15T22:31:18.298 INFO:tasks.rados.rados.0.smithi076.stdout:2668: rollback oid 25 current snap is 312 2021-03-15T22:31:18.298 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 305 2021-03-15T22:31:18.298 INFO:tasks.rados.rados.0.smithi076.stdout:2669: snap_create 2021-03-15T22:31:18.299 INFO:tasks.rados.rados.0.smithi076.stdout:2665: finishing write tid 3 to smithi07694878-5 2021-03-15T22:31:18.301 INFO:tasks.rados.rados.0.smithi076.stdout:2665: finishing write tid 4 to smithi07694878-5 2021-03-15T22:31:18.301 INFO:tasks.rados.rados.0.smithi076.stdout:2665: finishing write tid 5 to smithi07694878-5 2021-03-15T22:31:18.301 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1525 (ObjNum 924 snap 312 seq_num 924) dirty exists 2021-03-15T22:31:18.301 INFO:tasks.rados.rados.0.smithi076.stdout:2665: left oid 5 (ObjNum 924 snap 312 seq_num 924) 2021-03-15T22:31:18.304 INFO:tasks.rados.rados.0.smithi076.stdout:2668: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:31:18.304 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 1749 (ObjNum 1957169512 snap 32564 seq_num 265031520) dirty dne 2021-03-15T22:31:18.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:18.310 INFO:tasks.rados.rados.0.smithi076.stdout:2666: expect (ObjNum 918 snap 311 seq_num 918) 2021-03-15T22:31:18.311 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:18.311 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:18.329 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:18.413 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:18.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:18.440 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:18.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:18.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:18 smithi076 conmon[56514]: cluster 2021-03-15T22:31:17.157706+0000 mgr.y (mgr.14144) 884 : cluster [DBG] pgmap v1235: 138 pgs: 138 active+clean; 588 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 MiB/s wr, 7 op/s 2021-03-15T22:31:18.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:18 smithi076 conmon[56514]: cluster 2021-03-15T22:31:18.238428+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2021-03-15T22:31:18.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:18 smithi076 conmon[56514]: audit 2021-03-15T22:31:18.270829+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? 172.21.15.76:0/3304783079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:18.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:18 smithi076 conmon[62156]: cluster 2021-03-15T22:31:17.157706+0000 mgr.y (mgr.14144) 884 : cluster [DBG] pgmap v1235: 138 pgs: 138 active+clean; 588 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 MiB/s wr, 7 op/s 2021-03-15T22:31:18.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:18 smithi076 conmon[62156]: cluster 2021-03-15T22:31:18.238428+0000 mon.a (mon.0) 1940 : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2021-03-15T22:31:18.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:18 smithi076 conmon[62156]: audit 2021-03-15T22:31:18.270829+0000 mon.a (mon.0) 1941 : audit [INF] from='client.? 172.21.15.76:0/3304783079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:18.542 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:18.544 INFO:tasks.rados.rados.0.smithi076.stdout:2667: finishing copy_from to smithi07694878-4 2021-03-15T22:31:18.545 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 1611 (ObjNum 863 snap 285 seq_num 863) dirty exists 2021-03-15T22:31:18.545 INFO:tasks.rados.rados.0.smithi076.stdout:2667: finishing copy_from racing read to smithi07694878-4 2021-03-15T22:31:18.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:18.571 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:18.571 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:18.673 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:18.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:18.698 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:18.698 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:18.785 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:31:18.786 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:31:18.786 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:31:18.787 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:31:18.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:18 smithi125 conmon[56739]: cluster 2021-03-15T22:31:17.157706 2021-03-15T22:31:18.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:18 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 884 : cluster [DBG] pgmap v1235: 138 pgs: 138 active+clean; 588 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.2 KiB/s rd, 1.5 MiB/s wr, 7 op/s 2021-03-15T22:31:18.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:18 smithi125 conmon[56739]: cluster 2021-03-15T22:31:18.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:18 smithi125 conmon[56739]: 2021-03-15T22:31:18.238428+0000 mon.a (mon.0) 1940 2021-03-15T22:31:18.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:18 smithi125 conmon[56739]: : cluster [DBG] osdmap e729: 8 total, 8 up, 8 in 2021-03-15T22:31:18.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:18 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:31:18.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:18 smithi125 conmon[56739]: -15T22:31:18.270829+0000 mon.a (mon.0) 1941 : 2021-03-15T22:31:18.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:18 smithi125 conmon[56739]: audit [INF] from='client.? 172.21.15.76:0/3304783079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:18.800 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:18.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:18.828 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:18.828 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:18.929 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:18.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:18.959 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:18.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:19.061 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:19.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:19.087 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:19.088 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:19.189 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:19.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:19.219 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:19.219 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:19.241 INFO:tasks.rados.rados.0.smithi076.stdout:2665: done (4 left) 2021-03-15T22:31:19.241 INFO:tasks.rados.rados.0.smithi076.stdout:2666: done (3 left) 2021-03-15T22:31:19.241 INFO:tasks.rados.rados.0.smithi076.stdout:2667: done (2 left) 2021-03-15T22:31:19.242 INFO:tasks.rados.rados.0.smithi076.stdout:2668: done (1 left) 2021-03-15T22:31:19.242 INFO:tasks.rados.rados.0.smithi076.stdout:2669: done (0 left) 2021-03-15T22:31:19.242 INFO:tasks.rados.rados.0.smithi076.stdout:2670: snap_remove snap 293 2021-03-15T22:31:19.254 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:31:19.321 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:19.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:19.345 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:19.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:19.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:19.473 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:19.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:19.475 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:19.503 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-15T22:31:19.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:19.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:19.600 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:19.601 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:19.703 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:19.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:19.728 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:19.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:19.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[56514]: audit 2021-03-15T22:31:19.159018+0000 mon.a ( 2021-03-15T22:31:19.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[56514]: mon.0) 1942 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]: dispatch 2021-03-15T22:31:19.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[56514]: cluster 2021-03-15T22:31:19.234948+0000 mon.a (mon.0) 1943 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:19.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[56514]: audit 2021-03-15T22:31:19.238805+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? 172.21.15.76:0/3304783079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:19.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[56514]: audit 2021-03-15T22:31:19.238909+0000 mon.a (mon.0) 1945 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]': finished 2021-03-15T22:31:19.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[56514]: cluster 2021-03-15T22:31:19.238962+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2021-03-15T22:31:19.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: audit 2021-03-15T22:31:19.159018+0000 2021-03-15T22:31:19.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: mon.a (mon.0) 1942 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]: dispatch 2021-03-15T22:31:19.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: cluster 2021-03-15T22:31:19.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 2021-03-15T22:31:19.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 2021 2021-03-15T22:31:19.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: - 2021-03-15T22:31:19.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 03 2021-03-15T22:31:19.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: - 2021-03-15T22:31:19.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 15 2021-03-15T22:31:19.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: T 2021-03-15T22:31:19.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 22 2021-03-15T22:31:19.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: : 2021-03-15T22:31:19.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 31 2021-03-15T22:31:19.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: : 2021-03-15T22:31:19.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 19 2021-03-15T22:31:19.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: . 2021-03-15T22:31:19.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 234948 2021-03-15T22:31:19.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: +0000 2021-03-15T22:31:19.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 2021-03-15T22:31:19.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:19.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: ( 2021-03-15T22:31:19.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: mon 2021-03-15T22:31:19.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: . 2021-03-15T22:31:19.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 0 2021-03-15T22:31:19.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: ) 2021-03-15T22:31:19.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 1943 2021-03-15T22:31:19.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: : 2021-03-15T22:31:19.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:19.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: audit 2021 2021-03-15T22:31:19.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: -03-15T22:31:19.238805+0000 2021-03-15T22:31:19.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: mon.a (mon.0) 1944 : audit [INF] 2021-03-15T22:31:19.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3304783079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:19.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: audit 2021 2021-03-15T22:31:19.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: -03-15T22:31:19.238909 2021-03-15T22:31:19.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1945 : audit 2021-03-15T22:31:19.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]': finished 2021-03-15T22:31:19.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:31:19.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: :31:19.238962+0000 mon.a (mon. 2021-03-15T22:31:19.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:19 smithi076 conmon[62156]: 0) 1946 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2021-03-15T22:31:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:19 smithi125 conmon[56739]: audit 2021-03-15T22:31:19.159018+0000 mon.a (mon.0) 1942 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]: dispatch 2021-03-15T22:31:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:19 smithi125 conmon[56739]: cluster 2021-03-15T22:31:19.234948+0000 mon.a 2021-03-15T22:31:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:19 smithi125 conmon[56739]: (mon.0) 1943 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:19 smithi125 conmon[56739]: audit 2021-03-15T22:31:19.238805+0000 mon.a (mon.0) 1944 : audit [INF] from='client.? 172.21.15.76:0/3304783079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:19 smithi125 conmon[56739]: audit 2021-03-15T22:31:19.238909+0000 mon.a (mon.0) 1945 : audit 2021-03-15T22:31:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:19 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num_actual", "val": "32"}]': finished 2021-03-15T22:31:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:19 smithi125 conmon[56739]: cluster 2021-03-15T22:31: 2021-03-15T22:31:19.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:19 smithi125 conmon[56739]: 19.238962+0000 mon.a (mon.0) 1946 : cluster [DBG] osdmap e730: 8 total, 8 up, 8 in 2021-03-15T22:31:19.830 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:19.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:19.855 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:19.856 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:19.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:19.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:19.985 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:19.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:20.087 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:20.109 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:20.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:20.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:20.240 INFO:tasks.rados.rados.0.smithi076.stdout:2670: done (0 left) 2021-03-15T22:31:20.241 INFO:tasks.rados.rados.0.smithi076.stdout:2671: read oid 25 snap 294 2021-03-15T22:31:20.241 INFO:tasks.rados.rados.0.smithi076.stdout:2671: expect deleted 2021-03-15T22:31:20.241 INFO:tasks.rados.rados.0.smithi076.stdout:2672: write oid 23 current snap is 313 2021-03-15T22:31:20.242 INFO:tasks.rados.rados.0.smithi076.stdout:2672: seq_num 925 ranges {746475=484038,1762432=617322,2785680=529906} 2021-03-15T22:31:20.243 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:20.243 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:20.246 INFO:tasks.rados.rados.0.smithi076.stdout:2672: writing smithi07694878-23 from 746475 to 1230513 tid 1 2021-03-15T22:31:20.255 INFO:tasks.rados.rados.0.smithi076.stdout:2672: writing smithi07694878-23 from 1762432 to 2379754 tid 2 2021-03-15T22:31:20.262 INFO:tasks.rados.rados.0.smithi076.stdout:2672: writing smithi07694878-23 from 2785680 to 3315586 tid 3 2021-03-15T22:31:20.263 INFO:tasks.rados.rados.0.smithi076.stdout:2673: read oid 45 snap -1 2021-03-15T22:31:20.263 INFO:tasks.rados.rados.0.smithi076.stdout:2673: expect (ObjNum 844 snap 281 seq_num 844) 2021-03-15T22:31:20.263 INFO:tasks.rados.rados.0.smithi076.stdout:2674: snap_create 2021-03-15T22:31:20.264 INFO:tasks.rados.rados.0.smithi076.stdout:2672: finishing write tid 1 to smithi07694878-23 2021-03-15T22:31:20.267 INFO:tasks.rados.rados.0.smithi076.stdout:2672: finishing write tid 2 to smithi07694878-23 2021-03-15T22:31:20.267 INFO:tasks.rados.rados.0.smithi076.stdout:2672: finishing write tid 3 to smithi07694878-23 2021-03-15T22:31:20.268 INFO:tasks.rados.rados.0.smithi076.stdout:2672: finishing write tid 4 to smithi07694878-23 2021-03-15T22:31:20.268 INFO:tasks.rados.rados.0.smithi076.stdout:2672: finishing write tid 5 to smithi07694878-23 2021-03-15T22:31:20.268 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 1953 (ObjNum 925 snap 313 seq_num 925) dirty exists 2021-03-15T22:31:20.269 INFO:tasks.rados.rados.0.smithi076.stdout:2672: left oid 23 (ObjNum 925 snap 313 seq_num 925) 2021-03-15T22:31:20.270 INFO:tasks.rados.rados.0.smithi076.stdout:2673: expect (ObjNum 844 snap 281 seq_num 844) 2021-03-15T22:31:20.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:20.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:20.370 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:20.370 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:20.444 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:31:20 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:31:20.474 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:20.499 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:20.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:20.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:20.601 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:20.625 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:20.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:20.627 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:20.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:20 smithi076 conmon[62156]: cluster 2021-03-15T22:31:19.158176+0000 mgr.y 2021-03-15T22:31:20.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:20 smithi076 conmon[62156]: (mgr.14144) 885 : cluster [DBG] pgmap v1238: 148 pgs: 1 peering, 10 unknown, 137 active+clean; 588 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.5 MiB/s wr, 6 op/s 2021-03-15T22:31:20.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:20 smithi076 conmon[62156]: audit 2021-03-15T22:31:20.050740+0000 mgr.y (mgr.14144) 886 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:20.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:20 smithi076 conmon[62156]: cluster 2021-03-15T22:31:20.230693+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2021-03-15T22:31:20.705 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:20 smithi076 conmon[56514]: cluster 2021-03-15T22:31:19.158176+0000 mgr.y (mgr.14144) 885 : cluster [DBG] pgmap v1238: 148 pgs: 1 peering, 10 unknown, 137 active+clean; 588 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.5 MiB/s wr, 6 op/s 2021-03-15T22:31:20.705 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:20 smithi076 conmon[56514]: audit 2021-03-15T22:31:20.050740+0000 mgr.y (mgr.14144) 886 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:20.706 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:20 smithi076 conmon[56514]: cluster 2021-03-15T22:31:20.230693+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2021-03-15T22:31:20.712 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:20.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:20.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:20.752 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:20.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:20.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:20 smithi125 conmon[56739]: cluster 2021-03-15T22:31:19.158176+0000 mgr.y (mgr.14144) 885 : cluster [DBG] pgmap v1238: 148 pgs: 1 peering, 10 unknown, 137 active+clean; 588 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 1023 B/s rd, 1.5 MiB/s wr, 6 op/s 2021-03-15T22:31:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:20 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:31:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:20 smithi125 conmon[56739]: 31:20.050740+0000 mgr.y (mgr.14144) 886 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:20 smithi125 conmon[56739]: cluster 2021-03-15T22:31:20.230693+0000 mon.a (mon.0) 1947 : cluster [DBG] osdmap e731: 8 total, 8 up, 8 in 2021-03-15T22:31:20.854 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:20.879 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:20.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:20.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:20.975 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 6, 7, 0, 3, 5] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:31:20.975 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:31:20.976 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [1, 2, 4, 6, 7, 0, 3, 5] 2021-03-15T22:31:20.977 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd out 6 2021-03-15T22:31:20.982 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:21.043 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:21.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:21.046 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:21.148 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:21.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:21.172 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:21.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:21.239 INFO:tasks.rados.rados.0.smithi076.stdout:2671: done (3 left) 2021-03-15T22:31:21.240 INFO:tasks.rados.rados.0.smithi076.stdout:2672: done (2 left) 2021-03-15T22:31:21.240 INFO:tasks.rados.rados.0.smithi076.stdout:2673: done (1 left) 2021-03-15T22:31:21.240 INFO:tasks.rados.rados.0.smithi076.stdout:2674: done (0 left) 2021-03-15T22:31:21.240 INFO:tasks.rados.rados.0.smithi076.stdout:2675: read oid 11 snap -1 2021-03-15T22:31:21.241 INFO:tasks.rados.rados.0.smithi076.stdout:2675: expect (ObjNum 897 snap 302 seq_num 897) 2021-03-15T22:31:21.241 INFO:tasks.rados.rados.0.smithi076.stdout:2676: rollback oid 47 current snap is 314 2021-03-15T22:31:21.241 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 47 to 196 2021-03-15T22:31:21.241 INFO:tasks.rados.rados.0.smithi076.stdout:2677: snap_create 2021-03-15T22:31:21.243 INFO:tasks.rados.rados.0.smithi076.stdout:2676: finishing rollback tid 0 to smithi07694878-47 2021-03-15T22:31:21.243 INFO:tasks.rados.rados.0.smithi076.stdout:2676: finishing rollback tid 1 to smithi07694878-47 2021-03-15T22:31:21.244 INFO:tasks.rados.rados.0.smithi076.stdout:2676: finishing rollback tid 2 to smithi07694878-47 2021-03-15T22:31:21.244 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1754 (ObjNum 564 snap 177 seq_num 564) dirty exists 2021-03-15T22:31:21.248 INFO:tasks.rados.rados.0.smithi076.stdout:2675: expect (ObjNum 897 snap 302 seq_num 897) 2021-03-15T22:31:21.275 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:21.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:21.302 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:21.303 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:21.325 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:31:21.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:31:21.365 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:31:21.383 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:31:21.401 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:31:21.404 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:21.422 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:31:21.429 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:21.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:21.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:21.440 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:31:21.461 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:31:21.462 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:31:21.462 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:31:21.463 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:31:21.463 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:31:21.463 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:31:21.463 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:31:21.464 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:31:21.464 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:31:21.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:21.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:21.557 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:21.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:21.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:21.688 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:21.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:21.790 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:21.816 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:21.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:21.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:21.919 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:21.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:21.946 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:21.946 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:22.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:22.070 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:22.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:22.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:22.173 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:22.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:22.197 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:22.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:22.242 INFO:tasks.rados.rados.0.smithi076.stdout:2675: done (2 left) 2021-03-15T22:31:22.242 INFO:tasks.rados.rados.0.smithi076.stdout:2676: done (1 left) 2021-03-15T22:31:22.243 INFO:tasks.rados.rados.0.smithi076.stdout:2677: done (0 left) 2021-03-15T22:31:22.243 INFO:tasks.rados.rados.0.smithi076.stdout:2678: write oid 49 current snap is 315 2021-03-15T22:31:22.243 INFO:tasks.rados.rados.0.smithi076.stdout:2678: seq_num 926 ranges {763282=559287,1820914=760980,3213103=350179} 2021-03-15T22:31:22.249 INFO:tasks.rados.rados.0.smithi076.stdout:2678: writing smithi07694878-49 from 763282 to 1322569 tid 1 2021-03-15T22:31:22.256 INFO:tasks.rados.rados.0.smithi076.stdout:2678: writing smithi07694878-49 from 1820914 to 2581894 tid 2 2021-03-15T22:31:22.262 INFO:tasks.rados.rados.0.smithi076.stdout:2678: writing smithi07694878-49 from 3213103 to 3563282 tid 3 2021-03-15T22:31:22.262 INFO:tasks.rados.rados.0.smithi076.stdout:2679: write (excl) oid 34 current snap is 315 2021-03-15T22:31:22.263 INFO:tasks.rados.rados.0.smithi076.stdout:2678: finishing write tid 1 to smithi07694878-49 2021-03-15T22:31:22.263 INFO:tasks.rados.rados.0.smithi076.stdout:2679: seq_num 927 ranges {780089=634536,1879396=620991,3240526=339563} 2021-03-15T22:31:22.271 INFO:tasks.rados.rados.0.smithi076.stdout:2679: writing smithi07694878-34 from 780089 to 1414625 tid 1 2021-03-15T22:31:22.277 INFO:tasks.rados.rados.0.smithi076.stdout:2679: writing smithi07694878-34 from 1879396 to 2500387 tid 2 2021-03-15T22:31:22.284 INFO:tasks.rados.rados.0.smithi076.stdout:2679: writing smithi07694878-34 from 3240526 to 3580089 tid 3 2021-03-15T22:31:22.284 INFO:tasks.rados.rados.0.smithi076.stdout:2680: write oid 47 current snap is 315 2021-03-15T22:31:22.285 INFO:tasks.rados.rados.0.smithi076.stdout:2678: finishing write tid 2 to smithi07694878-49 2021-03-15T22:31:22.285 INFO:tasks.rados.rados.0.smithi076.stdout:2680: seq_num 928 ranges {796896=426138,1654231=764649,2984302=480075} 2021-03-15T22:31:22.290 INFO:tasks.rados.rados.0.smithi076.stdout:2680: writing smithi07694878-47 from 796896 to 1223034 tid 1 2021-03-15T22:31:22.297 INFO:tasks.rados.rados.0.smithi076.stdout:2680: writing smithi07694878-47 from 1654231 to 2418880 tid 2 2021-03-15T22:31:22.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:22.303 INFO:tasks.rados.rados.0.smithi076.stdout:2680: writing smithi07694878-47 from 2984302 to 3464377 tid 3 2021-03-15T22:31:22.303 INFO:tasks.rados.rados.0.smithi076.stdout:2681: snap_remove snap 258 2021-03-15T22:31:22.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:22.321 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:22.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:22.423 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:22.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:22.447 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:22.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:22.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: cluster 2021-03-15T22:31:21.158709+0000 mgr.y (mgr.14144) 887 : cluster [DBG] pgmap v1241: 148 pgs: 2 remapped+peering, 2 peering, 6 unknown, 138 active+clean; 593 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2021-03-15T22:31:22.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: cluster 2021-03-15T22:31:21.232617+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.1f starting backfill to osd.7 from (0'0,0'0] MAX to 643'9 2021-03-15T22:31:22.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: cluster 2021-03-15T22:31:21.232685+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2021-03-15T22:31:22.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: cluster 2021-03-15T22:31:21.233835+0000 osd.4 (osd.4) 157 : cluster [DBG] 5.19 starting backfill to osd.1 from (0'0,0'0] MAX to 317'7 2021-03-15T22:31:22.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: cluster 2021-03-15T22:31:22.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: 2021-03-15T22:31:21.233918+0000 osd.4 (osd.4) 158 : cluster [DBG] 5.19 starting backfill to osd.5 from (0'0,0'0] MAX to 317'7 2021-03-15T22:31:22.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: cluster 2021-03-15T22:31:21.235605+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.1d starting backfill to osd.0 from (0'0,0'0] MAX to 683'12 2021-03-15T22:31:22.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: cluster 2021-03-15T22:31:21.235685+0000 osd.2 (osd.2) 12 : cluster [DBG] 5.1d starting backfill to osd.1 from (0'0,0'0] MAX to 683'12 2021-03-15T22:31:22.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: cluster 2021-03-15T22:31:21.235709+0000 osd.0 (osd.0) 118 : cluster [DBG] 5.1e starting backfill to osd.2 from (0'0,0'0] MAX to 643'13 2021-03-15T22:31:22.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: cluster 2021-03-15T22:31:21.235762+0000 osd.2 (osd.2) 13 : cluster [DBG] 5.1d starting backfill to osd.4 from (0'0,0'0] MAX to 683'12 2021-03-15T22:31:22.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: cluster 2021-03-15T22:31:21.236531+0000 osd.0 (osd.0) 119 : cluster [DBG] 5.1e starting backfill to osd.7 from (0'0,0'0] MAX to 643'13 2021-03-15T22:31:22.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: audit 2021-03-15T22:31:21.304963+0000 mon.a (mon.0) 1949 : audit [DBG] from='client.? 172.21.15.76:0/3489742724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:22.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[62156]: cluster 2021-03-15T22:31:21.442140+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-15T22:31:22.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: cluster 2021-03-15T22:31:21.158709+0000 mgr.y (mgr.14144) 887 : cluster [DBG] pgmap v1241: 148 pgs: 2 remapped+peering, 2 peering, 6 unknown, 138 active+clean; 593 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2021-03-15T22:31:22.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: cluster 2021-03-15T22:31:21.232617+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.1f starting backfill to osd.7 from (0'0,0'0] MAX to 643'9 2021-03-15T22:31:22.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: cluster 2021-03-15T22:31:21.232685+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2021-03-15T22:31:22.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: cluster 2021-03-15T22:31:21.233835+0000 osd.4 (osd.4) 157 : cluster [DBG] 5.19 starting backfill to osd.1 from (0'0,0'0] MAX to 317'7 2021-03-15T22:31:22.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: cluster 2021-03-15T22:31:21.233918+0000 osd.4 (osd.4) 158 : cluster [DBG] 5.19 starting backfill to osd.5 from (0'0,0'0] MAX to 317'7 2021-03-15T22:31:22.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: cluster 2021-03-15T22:31:21.235605+0000 osd.2 (osd.2) 11 : cluster [DBG] 5.1d starting backfill to osd.0 from (0'0,0'0] MAX to 683'12 2021-03-15T22:31:22.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: cluster 2021-03-15T22:31:21.235685+0000 osd.2 (osd.2) 12 : cluster [DBG] 5.1d starting backfill to osd.1 from (0'0,0'0] MAX to 683'12 2021-03-15T22:31:22.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: cluster 2021-03-15T22:31:22.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: 2021-03-15T22:31:21.235709+0000 osd.0 (osd.0) 118 : cluster [DBG] 5.1e starting backfill to osd.2 from (0'0,0'0] MAX to 643'13 2021-03-15T22:31:22.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: cluster 2021-03-15T22:31:21.235762+0000 osd.2 (osd.2) 13 : cluster [DBG] 5.1d starting backfill to osd.4 from (0'0,0'0] MAX to 683'12 2021-03-15T22:31:22.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: cluster 2021-03-15T22:31:21.236531+0000 osd.0 (osd.0) 119 : cluster [DBG] 5.1e starting backfill to osd.7 from (0'0,0'0] MAX to 643'13 2021-03-15T22:31:22.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: audit 2021-03-15T22:31:21.304963+0000 mon.a (mon.0) 1949 : audit [DBG] from='client.? 172.21.15.76:0/3489742724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:22.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:22 smithi076 conmon[56514]: cluster 2021-03-15T22:31:21.442140+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-15T22:31:22.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: cluster 2021-03-15T22:31:21.158709+0000 mgr.y (mgr.14144) 887 2021-03-15T22:31:22.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: : cluster [DBG] pgmap v1241: 148 pgs: 2 remapped+peering, 2 peering, 6 unknown, 138 active+clean; 593 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 4.1 MiB/s rd, 1.7 MiB/s wr, 9 op/s 2021-03-15T22:31:22.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: cluster 2021-03-15T22:31:21.232617+0000 osd.6 (osd.6) 115 : cluster [DBG] 5.1f starting backfill to osd.7 from (0'0,0'0] MAX to 643'9 2021-03-15T22:31:22.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: cluster 2021-03-15T22:31:21.232685+0000 mon.a (mon.0) 1948 : cluster [DBG] osdmap e732: 8 total, 8 up, 8 in 2021-03-15T22:31:22.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: cluster 2021-03-15T22:31:21.233835+0000 osd.4 (osd.4 2021-03-15T22:31:22.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: ) 157 : cluster [DBG] 5.19 starting backfill to osd.1 from (0'0,0'0] MAX to 317'7 2021-03-15T22:31:22.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: cluster 2021-03-15T22:31:21.233918+0000 osd.4 (osd.4) 158 2021-03-15T22:31:22.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: : cluster [DBG] 5.19 starting backfill to osd.5 from (0'0,0'0] MAX to 317'7 2021-03-15T22:31:22.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:31:22.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: -03-15T22:31:21.235605+0000 osd.2 (osd.2) 11 : cluster [DBG] 2021-03-15T22:31:22.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: 5.1d starting backfill to osd.0 from (0'0,0'0] MAX to 683'12 2021-03-15T22:31:22.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: cluster 2021-03-15T22:31:22.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: 2021-03-15T22:31:21.235685+0000 osd.2 (osd.2) 12 : cluster [DBG] 2021-03-15T22:31:22.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: 5.1d starting backfill to osd.1 from (0'0,0'0] MAX to 683'12 2021-03-15T22:31:22.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: cluster 2021-03-15T22:31:22.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: 2021-03-15T22:31:21.235709+0000 osd.0 (osd.0) 2021-03-15T22:31:22.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: 118 : cluster [DBG] 5.1e starting backfill to osd.2 from (0'0,0'0] MAX to 643'13 2021-03-15T22:31:22.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: cluster 2021-03-15T22:31:21.235762+0000 osd.2 2021-03-15T22:31:22.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: (osd.2) 13 : cluster [DBG] 5.1d starting backfill to osd.4 from (0'0,0'0] MAX to 683'12 2021-03-15T22:31:22.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: cluster 2021-03-15T22:31 2021-03-15T22:31:22.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: :21.236531+0000 osd.0 (osd.0) 119 : cluster [DBG] 2021-03-15T22:31:22.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: 5.1e starting backfill to osd.7 from (0'0,0'0] MAX to 643'13 2021-03-15T22:31:22.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: audit 2021-03-15T22:31:22.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: 2021-03-15T22:31:21.304963+0000 mon.a (mon.0) 2021-03-15T22:31:22.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: 1949 : audit [DBG] from='client.? 172.21.15.76:0/3489742724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:22.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: cluster 2021-03-15T22:31:21 2021-03-15T22:31:22.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: .442140+0000 mon.a (mon.0) 1950 : cluster [WRN] Health check failed: Reduced data availability: 2 pgs inactive, 2 pgs peering (PG_AVAILABILITY) 2021-03-15T22:31:22.546 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:22 smithi125 conmon[56739]: 2021-03-15T22:31:22.548 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:22.570 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:22.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:22.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:22.657 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:22.672 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:22.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:31:22.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:22.699 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:22.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:22.801 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:22.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:22.825 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:22.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:22.874 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:31:22.874 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:31:22.927 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:22.954 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:22.956 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:23.057 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:23.080 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:23.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:23.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:23.182 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:23.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:23.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:23.250 INFO:tasks.rados.rados.0.smithi076.stdout:2681: done (3 left) 2021-03-15T22:31:23.250 INFO:tasks.rados.rados.0.smithi076.stdout:2682: read oid 5 snap -1 2021-03-15T22:31:23.250 INFO:tasks.rados.rados.0.smithi076.stdout:2682: expect (ObjNum 924 snap 312 seq_num 924) 2021-03-15T22:31:23.251 INFO:tasks.rados.rados.0.smithi076.stdout:2683: read oid 36 snap -1 2021-03-15T22:31:23.251 INFO:tasks.rados.rados.0.smithi076.stdout:2683: expect (ObjNum 918 snap 311 seq_num 918) 2021-03-15T22:31:23.251 INFO:tasks.rados.rados.0.smithi076.stdout:2684: copy_from oid 45 from oid 38 current snap is 315 2021-03-15T22:31:23.251 INFO:tasks.rados.rados.0.smithi076.stdout:2685: read oid 30 snap -1 2021-03-15T22:31:23.252 INFO:tasks.rados.rados.0.smithi076.stdout:2685: expect (ObjNum 803 snap 268 seq_num 803) 2021-03-15T22:31:23.252 INFO:tasks.rados.rados.0.smithi076.stdout:2686: delete oid 1 current snap is 315 2021-03-15T22:31:23.252 INFO:tasks.rados.rados.0.smithi076.stdout:2678: finishing write tid 3 to smithi07694878-49 2021-03-15T22:31:23.252 INFO:tasks.rados.rados.0.smithi076.stdout:2678: finishing write tid 4 to smithi07694878-49 2021-03-15T22:31:23.253 INFO:tasks.rados.rados.0.smithi076.stdout:2678: finishing write tid 5 to smithi07694878-49 2021-03-15T22:31:23.253 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 2082 (ObjNum 926 snap 315 seq_num 926) dirty exists 2021-03-15T22:31:23.253 INFO:tasks.rados.rados.0.smithi076.stdout:2678: left oid 49 (ObjNum 926 snap 315 seq_num 926) 2021-03-15T22:31:23.253 INFO:tasks.rados.rados.0.smithi076.stdout:2679: finishing write tid 1 to smithi07694878-34 2021-03-15T22:31:23.254 INFO:tasks.rados.rados.0.smithi076.stdout:2679: finishing write tid 2 to smithi07694878-34 2021-03-15T22:31:23.254 INFO:tasks.rados.rados.0.smithi076.stdout:2679: finishing write tid 3 to smithi07694878-34 2021-03-15T22:31:23.254 INFO:tasks.rados.rados.0.smithi076.stdout:2679: finishing write tid 4 to smithi07694878-34 2021-03-15T22:31:23.254 INFO:tasks.rados.rados.0.smithi076.stdout:2679: finishing write tid 5 to smithi07694878-34 2021-03-15T22:31:23.255 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 1751 (ObjNum 927 snap 315 seq_num 927) dirty exists 2021-03-15T22:31:23.255 INFO:tasks.rados.rados.0.smithi076.stdout:2679: left oid 34 (ObjNum 927 snap 315 seq_num 927) 2021-03-15T22:31:23.255 INFO:tasks.rados.rados.0.smithi076.stdout:2680: finishing write tid 1 to smithi07694878-47 2021-03-15T22:31:23.255 INFO:tasks.rados.rados.0.smithi076.stdout:2680: finishing write tid 2 to smithi07694878-47 2021-03-15T22:31:23.256 INFO:tasks.rados.rados.0.smithi076.stdout:2680: finishing write tid 3 to smithi07694878-47 2021-03-15T22:31:23.256 INFO:tasks.rados.rados.0.smithi076.stdout:2680: finishing write tid 4 to smithi07694878-47 2021-03-15T22:31:23.256 INFO:tasks.rados.rados.0.smithi076.stdout:2680: finishing write tid 5 to smithi07694878-47 2021-03-15T22:31:23.256 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1758 (ObjNum 928 snap 315 seq_num 928) dirty exists 2021-03-15T22:31:23.256 INFO:tasks.rados.rados.0.smithi076.stdout:2680: left oid 47 (ObjNum 928 snap 315 seq_num 928) 2021-03-15T22:31:23.264 INFO:tasks.rados.rados.0.smithi076.stdout:2682: expect (ObjNum 924 snap 312 seq_num 924) 2021-03-15T22:31:23.309 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:23.332 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:23.333 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:23.434 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:23.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:23.457 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:23.457 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:23.502 INFO:tasks.rados.rados.0.smithi076.stdout:2685: expect (ObjNum 803 snap 268 seq_num 803) 2021-03-15T22:31:23.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[56514]: cluster 2021-03-15T22:31:22.236732+0000 mon.a (mon.0) 1951 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2021-03-15T22:31:23.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: cluster 2021-03-15T22:31:23.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 2021-03-15T22:31:23.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 2021 2021-03-15T22:31:23.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: - 2021-03-15T22:31:23.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 03 2021-03-15T22:31:23.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: - 2021-03-15T22:31:23.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 15 2021-03-15T22:31:23.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: T 2021-03-15T22:31:23.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 22 2021-03-15T22:31:23.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: : 2021-03-15T22:31:23.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 31 2021-03-15T22:31:23.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: : 2021-03-15T22:31:23.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 22 2021-03-15T22:31:23.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: . 2021-03-15T22:31:23.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 236732 2021-03-15T22:31:23.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: +0000 2021-03-15T22:31:23.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 2021-03-15T22:31:23.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:23.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: ( 2021-03-15T22:31:23.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: mon 2021-03-15T22:31:23.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: . 2021-03-15T22:31:23.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 0 2021-03-15T22:31:23.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: ) 2021-03-15T22:31:23.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 1951 2021-03-15T22:31:23.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: : 2021-03-15T22:31:23.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: cluster 2021-03-15T22:31:23.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 2021-03-15T22:31:23.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:23.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 2021-03-15T22:31:23.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: osdmap e733: 8 total, 8 up, 8 in 2021-03-15T22:31:23.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:23 smithi076 conmon[62156]: 2021-03-15T22:31:23.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:23 smithi125 conmon[56739]: cluster 2021-03-15T22:31:22.236732+0000 mon.a (mon.0 2021-03-15T22:31:23.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:23 smithi125 conmon[56739]: ) 1951 : cluster [DBG] osdmap e733: 8 total, 8 up, 8 in 2021-03-15T22:31:23.558 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:23.580 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:23.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:23.582 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:23.683 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:23.705 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:23.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:23.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:23.726 INFO:tasks.rados.rados.0.smithi076.stdout:2683: expect (ObjNum 918 snap 311 seq_num 918) 2021-03-15T22:31:23.808 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:23.830 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:23.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:23.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:23.932 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:23.947 INFO:tasks.rados.rados.0.smithi076.stdout:2684: finishing copy_from to smithi07694878-45 2021-03-15T22:31:23.947 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 2071 (ObjNum 732 snap 239 seq_num 732) dirty exists 2021-03-15T22:31:23.948 INFO:tasks.rados.rados.0.smithi076.stdout:2684: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:31:23.948 INFO:tasks.rados.rados.0.smithi076.stdout:2678: done (7 left) 2021-03-15T22:31:23.948 INFO:tasks.rados.rados.0.smithi076.stdout:2679: done (6 left) 2021-03-15T22:31:23.948 INFO:tasks.rados.rados.0.smithi076.stdout:2680: done (5 left) 2021-03-15T22:31:23.949 INFO:tasks.rados.rados.0.smithi076.stdout:2682: done (4 left) 2021-03-15T22:31:23.949 INFO:tasks.rados.rados.0.smithi076.stdout:2683: done (3 left) 2021-03-15T22:31:23.949 INFO:tasks.rados.rados.0.smithi076.stdout:2684: done (2 left) 2021-03-15T22:31:23.949 INFO:tasks.rados.rados.0.smithi076.stdout:2685: done (1 left) 2021-03-15T22:31:23.949 INFO:tasks.rados.rados.0.smithi076.stdout:2686: done (0 left) 2021-03-15T22:31:23.950 INFO:tasks.rados.rados.0.smithi076.stdout:2687: rollback oid 4 current snap is 315 2021-03-15T22:31:23.950 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 4 to 300 2021-03-15T22:31:23.950 INFO:tasks.rados.rados.0.smithi076.stdout:2688: read oid 20 snap -1 2021-03-15T22:31:23.950 INFO:tasks.rados.rados.0.smithi076.stdout:2688: expect (ObjNum 874 snap 287 seq_num 874) 2021-03-15T22:31:23.951 INFO:tasks.rados.rados.0.smithi076.stdout:2689: rollback oid 35 current snap is 315 2021-03-15T22:31:23.951 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 35 to 273 2021-03-15T22:31:23.951 INFO:tasks.rados.rados.0.smithi076.stdout:2690: read oid 29 snap 312 2021-03-15T22:31:23.951 INFO:tasks.rados.rados.0.smithi076.stdout:2690: expect deleted 2021-03-15T22:31:23.952 INFO:tasks.rados.rados.0.smithi076.stdout:2691: rollback oid 7 current snap is 315 2021-03-15T22:31:23.952 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 7 to 301 2021-03-15T22:31:23.952 INFO:tasks.rados.rados.0.smithi076.stdout:2692: copy_from oid 45 from oid 39 current snap is 315 2021-03-15T22:31:23.953 INFO:tasks.rados.rados.0.smithi076.stdout:2693: read oid 14 snap 299 2021-03-15T22:31:23.953 INFO:tasks.rados.rados.0.smithi076.stdout:2693: expect deleted 2021-03-15T22:31:23.953 INFO:tasks.rados.rados.0.smithi076.stdout:2694: delete oid 36 current snap is 315 2021-03-15T22:31:23.954 INFO:tasks.rados.rados.0.smithi076.stdout:2689: finishing rollback tid 0 to smithi07694878-35 2021-03-15T22:31:23.954 INFO:tasks.rados.rados.0.smithi076.stdout:2689: finishing rollback tid 1 to smithi07694878-35 2021-03-15T22:31:23.955 INFO:tasks.rados.rados.0.smithi076.stdout:2688: expect (ObjNum 874 snap 287 seq_num 874) 2021-03-15T22:31:23.955 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:23.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:24.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-15T22:31:24.059 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:24.082 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:24.083 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:24.091 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:24.185 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:24.189 INFO:tasks.rados.rados.0.smithi076.stdout:2687: finishing rollback tid 0 to smithi07694878-4 2021-03-15T22:31:24.190 INFO:tasks.rados.rados.0.smithi076.stdout:2687: finishing rollback tid 1 to smithi07694878-4 2021-03-15T22:31:24.190 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 1613 (ObjNum 1953066355 snap 925919592 seq_num 942946614) dirty dne 2021-03-15T22:31:24.190 INFO:tasks.rados.rados.0.smithi076.stdout:2689: finishing rollback tid 2 to smithi07694878-35 2021-03-15T22:31:24.190 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 2081 (ObjNum 785 snap 261 seq_num 785) dirty exists 2021-03-15T22:31:24.191 INFO:tasks.rados.rados.0.smithi076.stdout:2691: finishing rollback tid 0 to smithi07694878-7 2021-03-15T22:31:24.191 INFO:tasks.rados.rados.0.smithi076.stdout:2687: done (7 left) 2021-03-15T22:31:24.191 INFO:tasks.rados.rados.0.smithi076.stdout:2688: done (6 left) 2021-03-15T22:31:24.191 INFO:tasks.rados.rados.0.smithi076.stdout:2689: done (5 left) 2021-03-15T22:31:24.191 INFO:tasks.rados.rados.0.smithi076.stdout:2690: done (4 left) 2021-03-15T22:31:24.192 INFO:tasks.rados.rados.0.smithi076.stdout:2693: done (3 left) 2021-03-15T22:31:24.192 INFO:tasks.rados.rados.0.smithi076.stdout:2694: done (2 left) 2021-03-15T22:31:24.192 INFO:tasks.rados.rados.0.smithi076.stdout:2695: write (excl) oid 2 current snap is 315 2021-03-15T22:31:24.193 INFO:tasks.rados.rados.0.smithi076.stdout:2691: finishing rollback tid 1 to smithi07694878-7 2021-03-15T22:31:24.193 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2083 (ObjNum 264849312 snap 21896 seq_num 0) dirty dne 2021-03-15T22:31:24.193 INFO:tasks.rados.rados.0.smithi076.stdout:2692: finishing copy_from to smithi07694878-45 2021-03-15T22:31:24.193 INFO:tasks.rados.rados.0.smithi076.stdout:2692: got expected ENOENT (src dne) 2021-03-15T22:31:24.193 INFO:tasks.rados.rados.0.smithi076.stdout:2695: seq_num 931 ranges {447317=651885,1713324=628329,3066571=580746} 2021-03-15T22:31:24.196 INFO:tasks.rados.rados.0.smithi076.stdout:2695: writing smithi07694878-2 from 447317 to 1099202 tid 1 2021-03-15T22:31:24.204 INFO:tasks.rados.rados.0.smithi076.stdout:2695: writing smithi07694878-2 from 1713324 to 2341653 tid 2 2021-03-15T22:31:24.208 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:24.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:24.209 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:24.214 INFO:tasks.rados.rados.0.smithi076.stdout:2695: writing smithi07694878-2 from 3066571 to 3647317 tid 3 2021-03-15T22:31:24.214 INFO:tasks.rados.rados.0.smithi076.stdout:2691: done (2 left) 2021-03-15T22:31:24.214 INFO:tasks.rados.rados.0.smithi076.stdout:2696: read oid 20 snap -1 2021-03-15T22:31:24.215 INFO:tasks.rados.rados.0.smithi076.stdout:2696: expect (ObjNum 874 snap 287 seq_num 874) 2021-03-15T22:31:24.215 INFO:tasks.rados.rados.0.smithi076.stdout:2697: write (excl) oid 42 current snap is 315 2021-03-15T22:31:24.215 INFO:tasks.rados.rados.0.smithi076.stdout:2692: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:31:24.216 INFO:tasks.rados.rados.0.smithi076.stdout:2695: finishing write tid 1 to smithi07694878-2 2021-03-15T22:31:24.216 INFO:tasks.rados.rados.0.smithi076.stdout:2695: finishing write tid 2 to smithi07694878-2 2021-03-15T22:31:24.216 INFO:tasks.rados.rados.0.smithi076.stdout:2697: seq_num 932 ranges {464124=727134,1771806=771987,2977641=641516} 2021-03-15T22:31:24.220 INFO:tasks.rados.rados.0.smithi076.stdout:2697: writing smithi07694878-42 from 464124 to 1191258 tid 1 2021-03-15T22:31:24.228 INFO:tasks.rados.rados.0.smithi076.stdout:2697: writing smithi07694878-42 from 1771806 to 2543793 tid 2 2021-03-15T22:31:24.235 INFO:tasks.rados.rados.0.smithi076.stdout:2697: writing smithi07694878-42 from 2977641 to 3619157 tid 3 2021-03-15T22:31:24.235 INFO:tasks.rados.rados.0.smithi076.stdout:2692: done (3 left) 2021-03-15T22:31:24.235 INFO:tasks.rados.rados.0.smithi076.stdout:2698: copy_from oid 14 from oid 7 current snap is 315 2021-03-15T22:31:24.236 INFO:tasks.rados.rados.0.smithi076.stdout:2696: expect (ObjNum 874 snap 287 seq_num 874) 2021-03-15T22:31:24.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:24.332 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:24.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:24.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:24.435 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:24.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:24.458 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:24.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:24.476 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:31:24.484 INFO:tasks.rados.rados.0.smithi076.stdout:2695: finishing write tid 3 to smithi07694878-2 2021-03-15T22:31:24.485 INFO:tasks.rados.rados.0.smithi076.stdout:2695: finishing write tid 4 to smithi07694878-2 2021-03-15T22:31:24.485 INFO:tasks.rados.rados.0.smithi076.stdout:2696: done (3 left) 2021-03-15T22:31:24.485 INFO:tasks.rados.rados.0.smithi076.stdout:2699: rollback oid 28 current snap is 315 2021-03-15T22:31:24.485 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 286 2021-03-15T22:31:24.486 INFO:tasks.rados.rados.0.smithi076.stdout:2695: finishing write tid 5 to smithi07694878-2 2021-03-15T22:31:24.486 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1770 (ObjNum 931 snap 315 seq_num 931) dirty exists 2021-03-15T22:31:24.486 INFO:tasks.rados.rados.0.smithi076.stdout:2695: left oid 2 (ObjNum 931 snap 315 seq_num 931) 2021-03-15T22:31:24.486 INFO:tasks.rados.rados.0.smithi076.stdout:2697: finishing write tid 1 to smithi07694878-42 2021-03-15T22:31:24.487 INFO:tasks.rados.rados.0.smithi076.stdout:2697: finishing write tid 2 to smithi07694878-42 2021-03-15T22:31:24.487 INFO:tasks.rados.rados.0.smithi076.stdout:2698: finishing copy_from to smithi07694878-14 2021-03-15T22:31:24.487 INFO:tasks.rados.rados.0.smithi076.stdout:2698: got expected ENOENT (src dne) 2021-03-15T22:31:24.487 INFO:tasks.rados.rados.0.smithi076.stdout:2698: finishing copy_from racing read to smithi07694878-14 2021-03-15T22:31:24.488 INFO:tasks.rados.rados.0.smithi076.stdout:2697: finishing write tid 3 to smithi07694878-42 2021-03-15T22:31:24.488 INFO:tasks.rados.rados.0.smithi076.stdout:2697: finishing write tid 4 to smithi07694878-42 2021-03-15T22:31:24.488 INFO:tasks.rados.rados.0.smithi076.stdout:2697: finishing write tid 5 to smithi07694878-42 2021-03-15T22:31:24.488 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2231 (ObjNum 932 snap 315 seq_num 932) dirty exists 2021-03-15T22:31:24.488 INFO:tasks.rados.rados.0.smithi076.stdout:2697: left oid 42 (ObjNum 932 snap 315 seq_num 932) 2021-03-15T22:31:24.489 INFO:tasks.rados.rados.0.smithi076.stdout:2695: done (3 left) 2021-03-15T22:31:24.489 INFO:tasks.rados.rados.0.smithi076.stdout:2697: done (2 left) 2021-03-15T22:31:24.489 INFO:tasks.rados.rados.0.smithi076.stdout:2698: done (1 left) 2021-03-15T22:31:24.489 INFO:tasks.rados.rados.0.smithi076.stdout:2700: snap_create 2021-03-15T22:31:24.490 INFO:tasks.rados.rados.0.smithi076.stdout:2699: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:31:24.490 INFO:tasks.rados.rados.0.smithi076.stdout:2699: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:31:24.491 INFO:tasks.rados.rados.0.smithi076.stdout:2699: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:31:24.491 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2235 (ObjNum 620 snap 193 seq_num 620) dirty exists 2021-03-15T22:31:24.499 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:24 smithi076 conmon[56514]: cluster 2021-03-15T22:31:23.159365+0000 mgr.y (mgr.14144) 888 : cluster [DBG] pgmap v1244: 148 pgs: 2 remapped+peering, 2 peering, 3 unknown, 141 active+clean; 598 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-03-15T22:31:24.499 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:24 smithi076 conmon[56514]: cluster 2021-03-15T22:31:23.243281+0000 mon.a (mon.0) 1952 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2021-03-15T22:31:24.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:24 smithi076 conmon[62156]: cluster 2021-03-15T22:31:23.159365+0000 mgr.y (mgr.14144) 888 : 2021-03-15T22:31:24.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:24 smithi076 conmon[62156]: cluster [DBG] pgmap v1244: 148 pgs: 2 remapped+peering, 2 peering, 3 unknown, 141 active+clean; 598 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-03-15T22:31:24.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:24 smithi076 conmon[62156]: cluster 2021-03-15T22:31:23.243281+0000 mon.a (mon.0) 1952 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2021-03-15T22:31:24.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:24 smithi125 conmon[56739]: cluster 2021-03-15T22:31:23.159365+0000 mgr.y (mgr.14144 2021-03-15T22:31:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:24 smithi125 conmon[56739]: ) 888 : cluster [DBG] pgmap v1244: 148 pgs: 2 remapped+peering, 2 peering, 3 unknown, 141 active+clean; 598 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 6.7 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-03-15T22:31:24.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:24 smithi125 conmon[56739]: cluster 2021-03-15T22:31:23.243281+0000 mon.a (mon.0) 1952 : cluster [DBG] osdmap e734: 8 total, 8 up, 8 in 2021-03-15T22:31:24.559 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:24.581 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:24.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:24.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:24.684 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:24.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:24.707 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:24.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:24.708 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:24.808 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:24.833 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:24.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:24.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:24.936 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:24.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:24.959 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:24.960 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:25.061 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:25.076 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:25.083 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:25.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:25.084 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:25.185 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:25.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:25.209 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:25.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:25.234 INFO:tasks.rados.rados.0.smithi076.stdout:2699: done (1 left) 2021-03-15T22:31:25.235 INFO:tasks.rados.rados.0.smithi076.stdout:2700: done (0 left) 2021-03-15T22:31:25.235 INFO:tasks.rados.rados.0.smithi076.stdout:2701: rollback oid 12 current snap is 316 2021-03-15T22:31:25.235 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 12 to 299 2021-03-15T22:31:25.235 INFO:tasks.rados.rados.0.smithi076.stdout:2702: copy_from oid 1 from oid 32 current snap is 316 2021-03-15T22:31:25.236 INFO:tasks.rados.rados.0.smithi076.stdout:2703: read oid 19 snap -1 2021-03-15T22:31:25.236 INFO:tasks.rados.rados.0.smithi076.stdout:2703: expect (ObjNum 613 snap 190 seq_num 613) 2021-03-15T22:31:25.236 INFO:tasks.rados.rados.0.smithi076.stdout:2704: rollback oid 21 current snap is 316 2021-03-15T22:31:25.236 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 21 to 242 2021-03-15T22:31:25.236 INFO:tasks.rados.rados.0.smithi076.stdout:2705: delete oid 34 current snap is 316 2021-03-15T22:31:25.240 INFO:tasks.rados.rados.0.smithi076.stdout:2701: finishing rollback tid 1 to smithi07694878-12 2021-03-15T22:31:25.240 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 1890 (ObjNum 1953066355 snap 925919592 seq_num 942946614) dirty dne 2021-03-15T22:31:25.241 INFO:tasks.rados.rados.0.smithi076.stdout:2701: done (4 left) 2021-03-15T22:31:25.241 INFO:tasks.rados.rados.0.smithi076.stdout:2705: done (3 left) 2021-03-15T22:31:25.241 INFO:tasks.rados.rados.0.smithi076.stdout:2706: rollback oid 4 current snap is 316 2021-03-15T22:31:25.242 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 4 to 305 2021-03-15T22:31:25.242 INFO:tasks.rados.rados.0.smithi076.stdout:2707: delete oid 8 current snap is 316 2021-03-15T22:31:25.244 INFO:tasks.rados.rados.0.smithi076.stdout:2706: finishing rollback tid 1 to smithi07694878-4 2021-03-15T22:31:25.244 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 1613 (ObjNum 288 snap 0 seq_num 33) dirty dne 2021-03-15T22:31:25.246 INFO:tasks.rados.rados.0.smithi076.stdout:2702: finishing copy_from to smithi07694878-1 2021-03-15T22:31:25.246 INFO:tasks.rados.rados.0.smithi076.stdout:2702: got expected ENOENT (src dne) 2021-03-15T22:31:25.247 INFO:tasks.rados.rados.0.smithi076.stdout:2704: finishing rollback tid 0 to smithi07694878-21 2021-03-15T22:31:25.247 INFO:tasks.rados.rados.0.smithi076.stdout:2704: finishing rollback tid 1 to smithi07694878-21 2021-03-15T22:31:25.247 INFO:tasks.rados.rados.0.smithi076.stdout:2702: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:31:25.248 INFO:tasks.rados.rados.0.smithi076.stdout:2704: finishing rollback tid 2 to smithi07694878-21 2021-03-15T22:31:25.248 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 2239 (ObjNum 724 snap 235 seq_num 724) dirty exists 2021-03-15T22:31:25.248 INFO:tasks.rados.rados.0.smithi076.stdout:2702: done (4 left) 2021-03-15T22:31:25.249 INFO:tasks.rados.rados.0.smithi076.stdout:2704: done (3 left) 2021-03-15T22:31:25.249 INFO:tasks.rados.rados.0.smithi076.stdout:2706: done (2 left) 2021-03-15T22:31:25.249 INFO:tasks.rados.rados.0.smithi076.stdout:2707: done (1 left) 2021-03-15T22:31:25.249 INFO:tasks.rados.rados.0.smithi076.stdout:2708: write (excl) oid 43 current snap is 316 2021-03-15T22:31:25.249 INFO:tasks.rados.rados.0.smithi076.stdout:2708: seq_num 935 ranges {514545=552881,1830899=519314,2943557=475332} 2021-03-15T22:31:25.253 INFO:tasks.rados.rados.0.smithi076.stdout:2708: writing smithi07694878-43 from 514545 to 1067426 tid 1 2021-03-15T22:31:25.262 INFO:tasks.rados.rados.0.smithi076.stdout:2708: writing smithi07694878-43 from 1830899 to 2350213 tid 2 2021-03-15T22:31:25.267 INFO:tasks.rados.rados.0.smithi076.stdout:2708: writing smithi07694878-43 from 2943557 to 3418889 tid 3 2021-03-15T22:31:25.267 INFO:tasks.rados.rados.0.smithi076.stdout:2709: write (excl) oid 44 current snap is 316 2021-03-15T22:31:25.267 INFO:tasks.rados.rados.0.smithi076.stdout:2703: expect (ObjNum 613 snap 190 seq_num 613) 2021-03-15T22:31:25.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:25.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:25.334 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:25.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:25.364 INFO:tasks.rados.rados.0.smithi076.stdout:2708: finishing write tid 1 to smithi07694878-43 2021-03-15T22:31:25.364 INFO:tasks.rados.rados.0.smithi076.stdout:2708: finishing write tid 2 to smithi07694878-43 2021-03-15T22:31:25.364 INFO:tasks.rados.rados.0.smithi076.stdout:2708: finishing write tid 3 to smithi07694878-43 2021-03-15T22:31:25.365 INFO:tasks.rados.rados.0.smithi076.stdout:2708: finishing write tid 4 to smithi07694878-43 2021-03-15T22:31:25.365 INFO:tasks.rados.rados.0.smithi076.stdout:2709: seq_num 936 ranges {531352=744483,2005734=779325,3487333=244019} 2021-03-15T22:31:25.370 INFO:tasks.rados.rados.0.smithi076.stdout:2709: writing smithi07694878-44 from 531352 to 1275835 tid 1 2021-03-15T22:31:25.379 INFO:tasks.rados.rados.0.smithi076.stdout:2709: writing smithi07694878-44 from 2005734 to 2785059 tid 2 2021-03-15T22:31:25.385 INFO:tasks.rados.rados.0.smithi076.stdout:2709: writing smithi07694878-44 from 3487333 to 3731352 tid 3 2021-03-15T22:31:25.385 INFO:tasks.rados.rados.0.smithi076.stdout:2703: done (2 left) 2021-03-15T22:31:25.385 INFO:tasks.rados.rados.0.smithi076.stdout:2710: snap_remove snap 312 2021-03-15T22:31:25.436 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:25.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:25.459 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:25.460 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:25.561 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:25.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:25.584 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:25.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:25.686 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:25.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:25.710 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:25.710 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:25.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:25.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:25.838 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:25.838 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:25.940 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:25.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:25.964 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:25.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:26.065 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:26.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:26.089 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:26.090 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:26.191 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:26.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:26.215 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:26.215 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:26.256 INFO:tasks.rados.rados.0.smithi076.stdout:2710: done (2 left) 2021-03-15T22:31:26.256 INFO:tasks.rados.rados.0.smithi076.stdout:2711: copy_from oid 12 from oid 37 current snap is 316 2021-03-15T22:31:26.257 INFO:tasks.rados.rados.0.smithi076.stdout:2712: copy_from oid 35 from oid 6 current snap is 316 2021-03-15T22:31:26.257 INFO:tasks.rados.rados.0.smithi076.stdout:2713: snap_remove snap 268 2021-03-15T22:31:26.316 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:26.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:26.340 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:26.340 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:26.425 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:26 smithi076 conmon[56514]: cluster 2021-03-15T22:31:25.159944+0000 mgr.y (mgr.14144) 889 : cluster [DBG] pgmap v1246: 148 pgs: 1 active+remapped+backfill_toofull, 2 remapped+peering, 1 peering, 144 active+clean; 603 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 2.6 MiB/s wr, 15 op/s; 3/1854 objects misplaced (0.162%); 0 B/s, 0 objects/s recovering 2021-03-15T22:31:26.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:26 smithi076 conmon[56514]: cluster 2021-03-15T22:31:25.184030+0000 osd.5 (osd.5) 21 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:26.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:26 smithi076 conmon[56514]: cluster 2021-03-15T22:31:25.186054+0000 osd.5 (osd.5) 22 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:26.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:26 smithi076 conmon[56514]: cluster 2021-03-15T22:31:25.232110+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2021-03-15T22:31:26.427 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:26 smithi076 conmon[56514]: cluster 2021-03-15T22:31:25.241270+0000 mon.a (mon.0) 1954 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:31:26.427 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:26 smithi076 conmon[56514]: debug 2021-03-15T22:31:26.275+0000 7f0537c88700 -1 mon.a@0(leader).osd e736 definitely_dead 0 2021-03-15T22:31:26.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:26 smithi076 conmon[62156]: cluster 2021-03-15T22:31:25.159944+0000 mgr.y (mgr.14144) 889 : cluster [DBG] pgmap v1246: 148 pgs: 1 active+remapped+backfill_toofull, 2 remapped+peering, 1 peering, 144 active+clean; 603 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 2.6 MiB/s wr, 15 op/s; 3/1854 objects misplaced (0.162%); 0 B/s, 0 objects/s recovering 2021-03-15T22:31:26.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:26 smithi076 conmon[62156]: cluster 2021-03-15T22:31:25.184030+0000 osd.5 (osd.5) 21 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:26.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:26 smithi076 conmon[62156]: cluster 2021-03-15T22:31:25.186054+0000 osd.5 (osd.5) 22 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:26.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:26 smithi076 conmon[62156]: cluster 2021-03-15T22:31:25.232110+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2021-03-15T22:31:26.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:26 smithi076 conmon[62156]: cluster 2021-03-15T22:31:25.241270+0000 mon.a (mon.0) 1954 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:31:26.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:26.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:26.467 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:26.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:26.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:26 smithi125 conmon[56739]: cluster 2021-03-15T22:31:25.159944+0000 mgr.y (mgr.14144) 889 2021-03-15T22:31:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:26 smithi125 conmon[56739]: : cluster [DBG] pgmap v1246: 148 pgs: 1 active+remapped+backfill_toofull, 2 remapped+peering, 1 peering, 144 active+clean; 603 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.4 MiB/s rd, 2.6 MiB/s wr, 15 op/s; 3/1854 objects misplaced (0.162%); 0 B/s, 0 objects/s recovering 2021-03-15T22:31:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:26 smithi125 conmon[56739]: cluster 2021-03-15T22:31:25.184030+0000 osd.5 (osd.5) 21 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:26.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:26 smithi125 conmon[56739]: cluster 2021-03-15T22:31:25.186054+0000 osd.5 (osd.5) 22 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:26 smithi125 conmon[56739]: cluster 2021-03-15T22:31 2021-03-15T22:31:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:26 smithi125 conmon[56739]: :25.232110+0000 mon.a (mon.0) 1953 : cluster [DBG] osdmap e735: 8 total, 8 up, 8 in 2021-03-15T22:31:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:26 smithi125 conmon[56739]: cluster 2021-03-15T22:31:26.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:26 smithi125 conmon[56739]: 2021-03-15T22:31:25.241270 2021-03-15T22:31:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:26 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1954 2021-03-15T22:31:26.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:26 smithi125 conmon[56739]: : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:31:26.569 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:26.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:26.592 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:26.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:26.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:26.717 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:26.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:26.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:26.820 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:26.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:26.844 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:26.845 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:26.946 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:26.970 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:26.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:26.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:27.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:27.097 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:27.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:27.098 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:27.200 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:27.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:27.226 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:27.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:27.258 INFO:tasks.rados.rados.0.smithi076.stdout:2713: done (4 left) 2021-03-15T22:31:27.259 INFO:tasks.rados.rados.0.smithi076.stdout:2714: delete oid 13 current snap is 316 2021-03-15T22:31:27.259 INFO:tasks.rados.rados.0.smithi076.stdout:2708: finishing write tid 5 to smithi07694878-43 2021-03-15T22:31:27.259 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 1957 (ObjNum 935 snap 316 seq_num 935) dirty exists 2021-03-15T22:31:27.260 INFO:tasks.rados.rados.0.smithi076.stdout:2708: left oid 43 (ObjNum 935 snap 316 seq_num 935) 2021-03-15T22:31:27.260 INFO:tasks.rados.rados.0.smithi076.stdout:2709: finishing write tid 1 to smithi07694878-44 2021-03-15T22:31:27.260 INFO:tasks.rados.rados.0.smithi076.stdout:2709: finishing write tid 2 to smithi07694878-44 2021-03-15T22:31:27.260 INFO:tasks.rados.rados.0.smithi076.stdout:2709: finishing write tid 3 to smithi07694878-44 2021-03-15T22:31:27.261 INFO:tasks.rados.rados.0.smithi076.stdout:2709: finishing write tid 4 to smithi07694878-44 2021-03-15T22:31:27.261 INFO:tasks.rados.rados.0.smithi076.stdout:2709: finishing write tid 5 to smithi07694878-44 2021-03-15T22:31:27.261 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1762 (ObjNum 936 snap 316 seq_num 936) dirty exists 2021-03-15T22:31:27.261 INFO:tasks.rados.rados.0.smithi076.stdout:2709: left oid 44 (ObjNum 936 snap 316 seq_num 936) 2021-03-15T22:31:27.261 INFO:tasks.rados.rados.0.smithi076.stdout:2711: finishing copy_from to smithi07694878-12 2021-03-15T22:31:27.262 INFO:tasks.rados.rados.0.smithi076.stdout:2711: got expected ENOENT (src dne) 2021-03-15T22:31:27.262 INFO:tasks.rados.rados.0.smithi076.stdout:2711: finishing copy_from racing read to smithi07694878-12 2021-03-15T22:31:27.262 INFO:tasks.rados.rados.0.smithi076.stdout:2712: finishing copy_from to smithi07694878-35 2021-03-15T22:31:27.262 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 1611 (ObjNum 863 snap 285 seq_num 863) dirty exists 2021-03-15T22:31:27.263 INFO:tasks.rados.rados.0.smithi076.stdout:2712: finishing copy_from racing read to smithi07694878-35 2021-03-15T22:31:27.275 INFO:teuthology.orchestra.run.smithi076.stderr:marked out osd.6. 2021-03-15T22:31:27.280 INFO:tasks.rados.rados.0.smithi076.stdout:2708: done (4 left) 2021-03-15T22:31:27.280 INFO:tasks.rados.rados.0.smithi076.stdout:2709: done (3 left) 2021-03-15T22:31:27.280 INFO:tasks.rados.rados.0.smithi076.stdout:2711: done (2 left) 2021-03-15T22:31:27.280 INFO:tasks.rados.rados.0.smithi076.stdout:2712: done (1 left) 2021-03-15T22:31:27.281 INFO:tasks.rados.rados.0.smithi076.stdout:2714: done (0 left) 2021-03-15T22:31:27.281 INFO:tasks.rados.rados.0.smithi076.stdout:2715: snap_create 2021-03-15T22:31:27.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:27.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:27.352 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:27.353 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:27.455 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:27.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:27.478 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:27.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:27.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[56514]: cluster 2021-03-15T22:31:26.248200+0000 mon.a (mon.0) 2021-03-15T22:31:27.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[56514]: 1955 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2021-03-15T22:31:27.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[56514]: audit 2021-03-15T22:31:26.276145+0000 mon.a (mon.0) 1956 : audit [INF] from='client.? 172.21.15.76:0/3831349895' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-03-15T22:31:27.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[56514]: cluster 2021-03-15T22:31:26.276246+0000 mon.a (mon.0) 1957 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2021-03-15T22:31:27.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: cluster 2021-03-15T22:31:27.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:31:27.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: 22:31:26.248200+0000 mon.a (mon.0) 1955 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2021-03-15T22:31:27.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: audit 2021-03-15T22:31:27.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: 2021-03-15T22:31:27.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: 2021 2021-03-15T22:31:27.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: - 2021-03-15T22:31:27.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: 03 2021-03-15T22:31:27.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: - 2021-03-15T22:31:27.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: 15 2021-03-15T22:31:27.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: T 2021-03-15T22:31:27.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: 22 2021-03-15T22:31:27.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: :31:26.276145+0000 mon.a 2021-03-15T22:31:27.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: (mon.0) 1956 : audit [INF] from='client.? 172.21.15.76:0/3831349895' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-03-15T22:31:27.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: cluster 2021-03-15T22:31:26.276246 2021-03-15T22:31:27.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1957 : cluster [INF] 2021-03-15T22:31:27.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:27 smithi076 conmon[62156]: Client client.admin marked osd.6 out, while it was still marked up 2021-03-15T22:31:27.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:27 smithi125 conmon[56739]: cluster 2021-03-15T22:31:26.248200+0000 mon.a (mon.0) 2021-03-15T22:31:27.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:27 smithi125 conmon[56739]: 1955 : cluster [DBG] osdmap e736: 8 total, 8 up, 8 in 2021-03-15T22:31:27.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:27 smithi125 conmon[56739]: audit 2021-03-15T22:31:26.276145+0000 mon.a (mon.0) 1956 : audit [INF] from='client.? 172.21.15.76:0/3831349895' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2021-03-15T22:31:27.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:27 smithi125 conmon[56739]: cluster 2021-03-15T22:31:26.276246+0000 2021-03-15T22:31:27.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:27 smithi125 conmon[56739]: mon.a (mon.0) 1957 : cluster [INF] Client client.admin marked osd.6 out, while it was still marked up 2021-03-15T22:31:27.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:27 smithi125 conmon[56739]: 2021-03-15T22:31:27.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:27.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:27.604 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:27.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:27.706 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:27.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:27.729 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:27.729 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:27.830 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:27.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:27.854 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:27.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:27.955 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:27.981 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:27.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:27.982 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:28.084 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:28.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:28.107 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:28.108 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:28.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:28.234 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:28.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:28.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:28.264 INFO:tasks.rados.rados.0.smithi076.stdout:2715: done (0 left) 2021-03-15T22:31:28.265 INFO:tasks.rados.rados.0.smithi076.stdout:2716: rollback oid 6 current snap is 317 2021-03-15T22:31:28.265 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 273 2021-03-15T22:31:28.265 INFO:tasks.rados.rados.0.smithi076.stdout:2717: snap_create 2021-03-15T22:31:28.286 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:31:28.290 INFO:tasks.rados.rados.0.smithi076.stdout:2716: finishing rollback tid 0 to smithi07694878-6 2021-03-15T22:31:28.291 INFO:tasks.rados.rados.0.smithi076.stdout:2716: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:31:28.291 INFO:tasks.rados.rados.0.smithi076.stdout:2716: finishing rollback tid 2 to smithi07694878-6 2021-03-15T22:31:28.292 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 2074 (ObjNum 807 snap 269 seq_num 807) dirty exists 2021-03-15T22:31:28.337 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:28.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:28.360 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:28.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:28.462 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:28.484 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:28.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[56514]: cluster 2021-03-15T22:31:27.160817+0000 mgr.y (mgr.14144) 890 : cluster [DBG] pgmap v1249: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 146 active+clean; 617 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.9 MiB/s wr, 23 op/s; 3/1884 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2021-03-15T22:31:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[56514]: cluster 2021-03-15T22:31:27.244722+0000 mon.a (mon.0) 1958 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-15T22:31:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[56514]: audit 2021-03-15T22:31:27.251971+0000 mon.a (mon.0) 1959 : audit [INF] from='client.? 172.21.15.76:0/3831349895' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2021-03-15T22:31:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[56514]: cluster 2021-03-15T22:31:27.252034+0000 mon.a (mon.0) 1960 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2021-03-15T22:31:28.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[56514]: audit 2021-03-15T22:31:28.228004+0000 mon.a (mon.0) 1961 : audit [INF] from='client.? 172.21.15.76:0/3175374479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:31:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: cluster 2021-03-15T22:31 2021-03-15T22:31:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: :27.160817+0000 mgr.y (mgr.14144) 890 : cluster [DBG] pgmap v1249: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 146 active+clean; 617 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.9 MiB/s wr, 23 op/s; 3/1884 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2021-03-15T22:31:28.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:31:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: 15T22:31:27.244722 2021-03-15T22:31:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1958 2021-03-15T22:31:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-15T22:31:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: audit 2021 2021-03-15T22:31:28.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: -03-15T22:31:27.251971+0000 2021-03-15T22:31:28.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: mon.a (mon.0) 1959 : audit [INF] 2021-03-15T22:31:28.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3831349895' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2021-03-15T22:31:28.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:31:28.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: 22:31:27.252034+0000 mon.a (mon. 2021-03-15T22:31:28.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: 0) 1960 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2021-03-15T22:31:28.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:31:28.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: 03-15T22:31:28.228004+0000 2021-03-15T22:31:28.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: mon.a (mon.0) 1961 : audit [INF] 2021-03-15T22:31:28.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:28 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3175374479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:31:28.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:28 smithi125 conmon[56739]: cluster 2021-03-15T22:31:27.160817+0000 mgr.y (mgr.14144 2021-03-15T22:31:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:28 smithi125 conmon[56739]: ) 890 : cluster [DBG] pgmap v1249: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 146 active+clean; 617 MiB data, 1.7 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 2.9 MiB/s wr, 23 op/s; 3/1884 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2021-03-15T22:31:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:28 smithi125 conmon[56739]: cluster 2021-03-15T22:31:27.244722+0000 mon.a (mon.0) 1958 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 2 pgs inactive, 2 pgs peering) 2021-03-15T22:31:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:28 smithi125 conmon[56739]: audit 2021-03-15T22:31:27.251971+0000 mon.a (mon.0) 1959 : audit 2021-03-15T22:31:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:28 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/3831349895' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2021-03-15T22:31:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:28 smithi125 conmon[56739]: cluster 2021-03-15T22:31:27.252034+0000 mon.a (mon. 2021-03-15T22:31:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:28 smithi125 conmon[56739]: 0) 1960 : cluster [DBG] osdmap e737: 8 total, 8 up, 7 in 2021-03-15T22:31:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:28 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:31:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:28 smithi125 conmon[56739]: -15T22:31:28.228004+0000 mon.a (mon.0 2021-03-15T22:31:28.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:28 smithi125 conmon[56739]: ) 1961 : audit [INF] from='client.? 172.21.15.76:0/3175374479' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:31:28.586 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:28.607 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:28.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:28.609 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:28.711 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:28.735 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:28.735 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:28.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:28.857 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:28.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:28.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:28.960 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:28.982 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:28.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:28.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:29.085 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:29.107 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:29.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:29.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:29.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:29.231 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:29.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:29.233 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:29.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:29 smithi076 conmon[56514]: cluster 2021-03-15T22:31:28.249858+0000 mon.a (mon.0) 1962 : cluster 2021-03-15T22:31:29.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:29 smithi076 conmon[56514]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:29.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:29 smithi076 conmon[56514]: audit 2021-03-15T22:31:28.262459+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? 172.21.15.76:0/3175374479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:31:29.255 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:29 smithi076 conmon[56514]: cluster 2021-03-15T22:31:28.262518+0000 mon.a (mon.0) 1964 : cluster [DBG] osdmap e738: 8 total, 8 up, 7 in 2021-03-15T22:31:29.268 INFO:tasks.rados.rados.0.smithi076.stdout:2716: done (1 left) 2021-03-15T22:31:29.268 INFO:tasks.rados.rados.0.smithi076.stdout:2717: done (0 left) 2021-03-15T22:31:29.269 INFO:tasks.rados.rados.0.smithi076.stdout:2718: snap_remove snap 313 2021-03-15T22:31:29.288 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:31:29.288 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:31:29.335 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:29.367 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:29.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:29.369 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:29.470 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:29.497 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:29.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:29.499 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:29.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:29 smithi076 conmon[62156]: cluster 2021-03-15T22:31:28.249858+0000 mon.a (mon 2021-03-15T22:31:29.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:29 smithi076 conmon[62156]: .0) 1962 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:29.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:29 smithi076 conmon[62156]: audit 2021-03-15T22:31:28.262459+0000 mon.a (mon.0) 1963 : audit [INF] from='client.? 172.21.15.76:0/3175374479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:31:29.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:29 smithi076 conmon[62156]: cluster 2021-03-15T22:31:28.262518+0000 mon.a (mon.0) 1964 : cluster [DBG] osdmap e738: 8 total, 8 up, 7 in 2021-03-15T22:31:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:29 smithi125 conmon[56739]: cluster 2021-03-15T22:31:28.249858+0000 mon.a (mon. 2021-03-15T22:31:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:29 smithi125 conmon[56739]: 0) 1962 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:29 smithi125 conmon[56739]: audit 2021-03-15T22:31:28.262459+0000 mon.a (mon.0) 1963 : audit [INF] 2021-03-15T22:31:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:29 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/3175374479' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:31:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:29 smithi125 conmon[56739]: cluster 2021-03-15T22:31:29.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:29 smithi125 conmon[56739]: 2021-03-15T22:31:28.262518+0000 mon.a 2021-03-15T22:31:29.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:29 smithi125 conmon[56739]: (mon.0) 1964 : cluster 2021-03-15T22:31:29.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:29 smithi125 conmon[56739]: [DBG] osdmap e738: 8 total, 8 up, 7 in 2021-03-15T22:31:29.601 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:29.622 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:29.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:29.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:29.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:29.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:29.750 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:29.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:29.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:29.873 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:29.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:29.875 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:29.976 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:29.999 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:29.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:30.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:30.101 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:30.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:30.124 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:30.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:30.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:30.237 INFO:tasks.rados.rados.0.smithi076.stdout:2718: done (0 left) 2021-03-15T22:31:30.237 INFO:tasks.rados.rados.0.smithi076.stdout:2719: read oid 7 snap -1 2021-03-15T22:31:30.237 INFO:tasks.rados.rados.0.smithi076.stdout:2719: expect deleted 2021-03-15T22:31:30.238 INFO:tasks.rados.rados.0.smithi076.stdout:2720: rollback oid 29 current snap is 318 2021-03-15T22:31:30.238 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 29 to 242 2021-03-15T22:31:30.241 INFO:tasks.rados.rados.0.smithi076.stdout:2721: write oid 40 current snap is 318 2021-03-15T22:31:30.242 INFO:tasks.rados.rados.0.smithi076.stdout:2721: seq_num 938 ranges {564966=494981,1606345=782994,3025826=425501} 2021-03-15T22:31:30.247 INFO:tasks.rados.rados.0.smithi076.stdout:2721: writing smithi07694878-40 from 564966 to 1059947 tid 1 2021-03-15T22:31:30.253 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:30.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:30.254 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:30.255 INFO:tasks.rados.rados.0.smithi076.stdout:2721: writing smithi07694878-40 from 1606345 to 2389339 tid 2 2021-03-15T22:31:30.260 INFO:tasks.rados.rados.0.smithi076.stdout:2721: writing smithi07694878-40 from 3025826 to 3451327 tid 3 2021-03-15T22:31:30.261 INFO:tasks.rados.rados.0.smithi076.stdout:2722: write (excl) oid 33 current snap is 318 2021-03-15T22:31:30.261 INFO:tasks.rados.rados.0.smithi076.stdout:2722: seq_num 939 ranges {581773=570230,1664827=526652,2653249=636773} 2021-03-15T22:31:30.268 INFO:tasks.rados.rados.0.smithi076.stdout:2722: writing smithi07694878-33 from 581773 to 1152003 tid 1 2021-03-15T22:31:30.275 INFO:tasks.rados.rados.0.smithi076.stdout:2722: writing smithi07694878-33 from 1664827 to 2191479 tid 2 2021-03-15T22:31:30.282 INFO:tasks.rados.rados.0.smithi076.stdout:2722: writing smithi07694878-33 from 2653249 to 3290022 tid 3 2021-03-15T22:31:30.283 INFO:tasks.rados.rados.0.smithi076.stdout:2723: copy_from oid 42 from oid 43 current snap is 318 2021-03-15T22:31:30.283 INFO:tasks.rados.rados.0.smithi076.stdout:2720: finishing rollback tid 1 to smithi07694878-29 2021-03-15T22:31:30.283 INFO:tasks.rados.rados.0.smithi076.stdout:2720: finishing rollback tid 2 to smithi07694878-29 2021-03-15T22:31:30.284 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 2265 (ObjNum 387 snap 104 seq_num 387) dirty exists 2021-03-15T22:31:30.285 INFO:tasks.rados.rados.0.smithi076.stdout:2721: finishing write tid 1 to smithi07694878-40 2021-03-15T22:31:30.286 INFO:tasks.rados.rados.0.smithi076.stdout:2722: finishing write tid 1 to smithi07694878-33 2021-03-15T22:31:30.286 INFO:tasks.rados.rados.0.smithi076.stdout:2721: finishing write tid 2 to smithi07694878-40 2021-03-15T22:31:30.286 INFO:tasks.rados.rados.0.smithi076.stdout:2721: finishing write tid 3 to smithi07694878-40 2021-03-15T22:31:30.286 INFO:tasks.rados.rados.0.smithi076.stdout:2719: done (4 left) 2021-03-15T22:31:30.287 INFO:tasks.rados.rados.0.smithi076.stdout:2720: done (3 left) 2021-03-15T22:31:30.287 INFO:tasks.rados.rados.0.smithi076.stdout:2724: read oid 17 snap -1 2021-03-15T22:31:30.287 INFO:tasks.rados.rados.0.smithi076.stdout:2724: expect (ObjNum 827 snap 275 seq_num 827) 2021-03-15T22:31:30.287 INFO:tasks.rados.rados.0.smithi076.stdout:2725: write (excl) oid 6 current snap is 318 2021-03-15T22:31:30.287 INFO:tasks.rados.rados.0.smithi076.stdout:2721: finishing write tid 4 to smithi07694878-40 2021-03-15T22:31:30.288 INFO:tasks.rados.rados.0.smithi076.stdout:2725: seq_num 940 ranges {598580=645479,1723309=786663,3080672=448045} 2021-03-15T22:31:30.288 INFO:tasks.rados.rados.0.smithi076.stdout:2725: writing smithi07694878-6 from 598580 to 1244059 tid 1 2021-03-15T22:31:30.296 INFO:tasks.rados.rados.0.smithi076.stdout:2725: writing smithi07694878-6 from 1723309 to 2509972 tid 2 2021-03-15T22:31:30.302 INFO:tasks.rados.rados.0.smithi076.stdout:2725: writing smithi07694878-6 from 3080672 to 3528717 tid 3 2021-03-15T22:31:30.302 INFO:tasks.rados.rados.0.smithi076.stdout:2726: read oid 14 snap 314 2021-03-15T22:31:30.303 INFO:tasks.rados.rados.0.smithi076.stdout:2726: expect (ObjNum 623 snap 193 seq_num 623) 2021-03-15T22:31:30.303 INFO:tasks.rados.rados.0.smithi076.stdout:2727: read oid 10 snap -1 2021-03-15T22:31:30.303 INFO:tasks.rados.rados.0.smithi076.stdout:2727: expect (ObjNum 917 snap 309 seq_num 917) 2021-03-15T22:31:30.303 INFO:tasks.rados.rados.0.smithi076.stdout:2728: delete oid 36 current snap is 318 2021-03-15T22:31:30.303 INFO:tasks.rados.rados.0.smithi076.stdout:2721: finishing write tid 5 to smithi07694878-40 2021-03-15T22:31:30.304 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 2269 (ObjNum 938 snap 318 seq_num 938) dirty exists 2021-03-15T22:31:30.304 INFO:tasks.rados.rados.0.smithi076.stdout:2721: left oid 40 (ObjNum 938 snap 318 seq_num 938) 2021-03-15T22:31:30.304 INFO:tasks.rados.rados.0.smithi076.stdout:2722: finishing write tid 2 to smithi07694878-33 2021-03-15T22:31:30.304 INFO:tasks.rados.rados.0.smithi076.stdout:2722: finishing write tid 3 to smithi07694878-33 2021-03-15T22:31:30.305 INFO:tasks.rados.rados.0.smithi076.stdout:2722: finishing write tid 4 to smithi07694878-33 2021-03-15T22:31:30.305 INFO:tasks.rados.rados.0.smithi076.stdout:2722: finishing write tid 5 to smithi07694878-33 2021-03-15T22:31:30.305 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 1756 (ObjNum 939 snap 318 seq_num 939) dirty exists 2021-03-15T22:31:30.305 INFO:tasks.rados.rados.0.smithi076.stdout:2722: left oid 33 (ObjNum 939 snap 318 seq_num 939) 2021-03-15T22:31:30.305 INFO:tasks.rados.rados.0.smithi076.stdout:2725: finishing write tid 1 to smithi07694878-6 2021-03-15T22:31:30.306 INFO:tasks.rados.rados.0.smithi076.stdout:2724: expect (ObjNum 827 snap 275 seq_num 827) 2021-03-15T22:31:30.355 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:30.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:30.379 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:30.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:30.407 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:31:30.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:30.515 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:30.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:30.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:30.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:31:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: cluster 2021-03-15T22:31:29.161191+0000 mgr.y (mgr.14144) 891 : 2021-03-15T22:31:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: cluster [DBG] pgmap v1252: 148 pgs: 1 active+remapped+backfill_toofull, 5 peering, 142 active+clean; 619 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 2.7 MiB/s wr, 25 op/s; 3/1887 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2021-03-15T22:31:30.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: cluster 2021-03-15T22:31:29.264870+0000 mon.a (mon.0) 1965 : cluster [DBG] osdmap e739: 8 total, 8 up, 7 in 2021-03-15T22:31:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:31:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: 15T22:31:29.265418+0000 osd.7 (osd.7) 137 : cluster [DBG] 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 732'8 2021-03-15T22:31:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:31:30.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: 03-15T22:31:29.267680 2021-03-15T22:31:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: +0000 osd.4 (osd.4) 2021-03-15T22:31:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: 159 : cluster [DBG] 5.1f starting backfill to osd.2 from (0'0,0'0] MAX to 736'13 2021-03-15T22:31:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: cluster 2021-03-15T22:31:30.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: 2021-03-15T22 2021-03-15T22:31:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: :31:29.268385+0000 osd.5 (osd.5) 23 : cluster [DBG] 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 643'9 2021-03-15T22:31:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: 2021-03-15T22:31:30.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:31:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: -15T22:31:30.061778+0000 2021-03-15T22:31:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: mgr.y (mgr.14144) 892 : audit [DBG] 2021-03-15T22:31:30.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:30.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:31:30.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: -03-15T22:31:30.234305 2021-03-15T22:31:30.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1966 : cluster 2021-03-15T22:31:30.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:30 smithi125 conmon[56739]: [DBG] osdmap e740: 8 total, 8 up, 7 in 2021-03-15T22:31:30.547 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:31:30.565 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:31:30.584 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:31:30.601 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:31:30.607 INFO:tasks.rados.rados.0.smithi076.stdout:2725: finishing write tid 2 to smithi07694878-6 2021-03-15T22:31:30.607 INFO:tasks.rados.rados.0.smithi076.stdout:2725: finishing write tid 3 to smithi07694878-6 2021-03-15T22:31:30.607 INFO:tasks.rados.rados.0.smithi076.stdout:2725: finishing write tid 4 to smithi07694878-6 2021-03-15T22:31:30.607 INFO:tasks.rados.rados.0.smithi076.stdout:2721: done (7 left) 2021-03-15T22:31:30.608 INFO:tasks.rados.rados.0.smithi076.stdout:2722: done (6 left) 2021-03-15T22:31:30.608 INFO:tasks.rados.rados.0.smithi076.stdout:2724: done (5 left) 2021-03-15T22:31:30.608 INFO:tasks.rados.rados.0.smithi076.stdout:2728: done (4 left) 2021-03-15T22:31:30.608 INFO:tasks.rados.rados.0.smithi076.stdout:2729: copy_from oid 5 from oid 15 current snap is 318 2021-03-15T22:31:30.608 INFO:tasks.rados.rados.0.smithi076.stdout:2725: finishing write tid 5 to smithi07694878-6 2021-03-15T22:31:30.609 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 2078 (ObjNum 940 snap 318 seq_num 940) dirty exists 2021-03-15T22:31:30.609 INFO:tasks.rados.rados.0.smithi076.stdout:2725: left oid 6 (ObjNum 940 snap 318 seq_num 940) 2021-03-15T22:31:30.609 INFO:tasks.rados.rados.0.smithi076.stdout:2725: done (4 left) 2021-03-15T22:31:30.609 INFO:tasks.rados.rados.0.smithi076.stdout:2730: delete oid 22 current snap is 318 2021-03-15T22:31:30.611 INFO:tasks.rados.rados.0.smithi076.stdout:2730: done (4 left) 2021-03-15T22:31:30.611 INFO:tasks.rados.rados.0.smithi076.stdout:2731: write (excl) oid 33 current snap is 318 2021-03-15T22:31:30.611 INFO:tasks.rados.rados.0.smithi076.stdout:2731: seq_num 943 ranges {649001=587579,1898755=533990,3162941=686060} 2021-03-15T22:31:30.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:30.619 INFO:tasks.rados.rados.0.smithi076.stdout:2731: writing smithi07694878-33 from 649001 to 1236580 tid 1 2021-03-15T22:31:30.620 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:31:30.626 INFO:tasks.rados.rados.0.smithi076.stdout:2731: writing smithi07694878-33 from 1898755 to 2432745 tid 2 2021-03-15T22:31:30.635 INFO:tasks.rados.rados.0.smithi076.stdout:2731: writing smithi07694878-33 from 3162941 to 3849001 tid 3 2021-03-15T22:31:30.636 INFO:tasks.rados.rados.0.smithi076.stdout:2732: delete oid 16 current snap is 318 2021-03-15T22:31:30.636 INFO:tasks.rados.rados.0.smithi076.stdout:2729: finishing copy_from to smithi07694878-5 2021-03-15T22:31:30.636 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1969 (ObjNum 888 snap 296 seq_num 888) dirty exists 2021-03-15T22:31:30.637 INFO:tasks.rados.rados.0.smithi076.stdout:2729: finishing copy_from racing read to smithi07694878-5 2021-03-15T22:31:30.637 INFO:tasks.rados.rados.0.smithi076.stdout:2731: finishing write tid 1 to smithi07694878-33 2021-03-15T22:31:30.637 INFO:tasks.rados.rados.0.smithi076.stdout:2731: finishing write tid 2 to smithi07694878-33 2021-03-15T22:31:30.639 INFO:tasks.rados.rados.0.smithi076.stdout:2729: done (5 left) 2021-03-15T22:31:30.639 INFO:tasks.rados.rados.0.smithi076.stdout:2732: done (4 left) 2021-03-15T22:31:30.639 INFO:tasks.rados.rados.0.smithi076.stdout:2733: read oid 44 snap -1 2021-03-15T22:31:30.640 INFO:tasks.rados.rados.0.smithi076.stdout:2733: expect (ObjNum 936 snap 316 seq_num 936) 2021-03-15T22:31:30.640 INFO:tasks.rados.rados.0.smithi076.stdout:2734: write oid 49 current snap is 318 2021-03-15T22:31:30.640 INFO:tasks.rados.rados.0.smithi076.stdout:2731: finishing write tid 3 to smithi07694878-33 2021-03-15T22:31:30.640 INFO:tasks.rados.rados.0.smithi076.stdout:2734: seq_num 945 ranges {682615=738077,2015719=537659,3217787=420758} 2021-03-15T22:31:30.641 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:31:30.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:30.643 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:30.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:30.648 INFO:tasks.rados.rados.0.smithi076.stdout:2734: writing smithi07694878-49 from 682615 to 1420692 tid 1 2021-03-15T22:31:30.655 INFO:tasks.rados.rados.0.smithi076.stdout:2734: writing smithi07694878-49 from 2015719 to 2553378 tid 2 2021-03-15T22:31:30.661 INFO:tasks.rados.rados.0.smithi076.stdout:2734: writing smithi07694878-49 from 3217787 to 3638545 tid 3 2021-03-15T22:31:30.662 INFO:tasks.rados.rados.0.smithi076.stdout:2735: write (excl) oid 24 current snap is 318 2021-03-15T22:31:30.662 INFO:tasks.rados.rados.0.smithi076.stdout:2731: finishing write tid 4 to smithi07694878-33 2021-03-15T22:31:30.663 INFO:tasks.rados.rados.0.smithi076.stdout:2735: seq_num 946 ranges {699422=413326,1557848=681317,3012504=748383} 2021-03-15T22:31:30.663 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:31:30.667 INFO:tasks.rados.rados.0.smithi076.stdout:2735: writing smithi07694878-24 from 699422 to 1112748 tid 1 2021-03-15T22:31:30.675 INFO:tasks.rados.rados.0.smithi076.stdout:2735: writing smithi07694878-24 from 1557848 to 2239165 tid 2 2021-03-15T22:31:30.686 INFO:tasks.rados.rados.0.smithi076.stdout:2735: writing smithi07694878-24 from 3012504 to 3760887 tid 3 2021-03-15T22:31:30.687 INFO:tasks.rados.rados.0.smithi076.stdout:2731: finishing write tid 5 to smithi07694878-33 2021-03-15T22:31:30.688 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 1760 (ObjNum 943 snap 318 seq_num 943) dirty exists 2021-03-15T22:31:30.688 INFO:tasks.rados.rados.0.smithi076.stdout:2731: left oid 33 (ObjNum 943 snap 318 seq_num 943) 2021-03-15T22:31:30.689 INFO:tasks.rados.rados.0.smithi076.stdout:2733: expect (ObjNum 936 snap 316 seq_num 936) 2021-03-15T22:31:30.691 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:31:30.692 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:31:30.693 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:31:30.693 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:31:30.694 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:31:30.694 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:31:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:31:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:31:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:31:30.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:30.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021 2021-03-15T22:31:30.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: - 2021-03-15T22:31:30.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 03 2021-03-15T22:31:30.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: - 2021-03-15T22:31:30.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 15 2021-03-15T22:31:30.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: T 2021-03-15T22:31:30.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 22 2021-03-15T22:31:30.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 31 2021-03-15T22:31:30.753 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.753 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 29 2021-03-15T22:31:30.753 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: . 2021-03-15T22:31:30.753 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 161191 2021-03-15T22:31:30.754 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: +0000 2021-03-15T22:31:30.754 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.754 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: mgr.y 2021-03-15T22:31:30.754 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: ( 2021-03-15T22:31:30.754 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: mgr 2021-03-15T22:31:30.755 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: . 2021-03-15T22:31:30.755 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 14144 2021-03-15T22:31:30.755 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: ) 2021-03-15T22:31:30.755 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 891 2021-03-15T22:31:30.755 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.756 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.756 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.756 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:30.756 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: pgmap v1252: 148 pgs: 1 active+remapped+backfill_toofull, 5 peering, 142 active+clean; 619 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 2.7 MiB/s wr, 25 op/s; 3/1887 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2021-03-15T22:31:30.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021 2021-03-15T22:31:30.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: - 2021-03-15T22:31:30.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 03 2021-03-15T22:31:30.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: - 2021-03-15T22:31:30.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 15 2021-03-15T22:31:30.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: T 2021-03-15T22:31:30.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 22 2021-03-15T22:31:30.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 31 2021-03-15T22:31:30.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 29 2021-03-15T22:31:30.760 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: . 2021-03-15T22:31:30.760 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 264870 2021-03-15T22:31:30.760 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: +0000 2021-03-15T22:31:30.760 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.761 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:30.761 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: ( 2021-03-15T22:31:30.761 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: mon 2021-03-15T22:31:30.761 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: . 2021-03-15T22:31:30.761 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 0 2021-03-15T22:31:30.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: ) 2021-03-15T22:31:30.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 1965 2021-03-15T22:31:30.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:30.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: osdmap e739: 8 total, 8 up, 7 in 2021-03-15T22:31:30.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.764 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.764 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021 2021-03-15T22:31:30.764 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: - 2021-03-15T22:31:30.764 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 03 2021-03-15T22:31:30.765 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: - 2021-03-15T22:31:30.765 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 15 2021-03-15T22:31:30.765 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: T 2021-03-15T22:31:30.765 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 22 2021-03-15T22:31:30.765 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.766 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 31 2021-03-15T22:31:30.766 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.766 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 29 2021-03-15T22:31:30.766 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: . 2021-03-15T22:31:30.766 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 265418 2021-03-15T22:31:30.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: +0000 2021-03-15T22:31:30.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: osd.7 2021-03-15T22:31:30.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: ( 2021-03-15T22:31:30.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: osd 2021-03-15T22:31:30.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: . 2021-03-15T22:31:30.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 7 2021-03-15T22:31:30.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: ) 2021-03-15T22:31:30.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 137 2021-03-15T22:31:30.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: cluster 2021-03-15T22:31:30.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021 2021-03-15T22:31:30.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: - 2021-03-15T22:31:30.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 03 2021-03-15T22:31:30.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: - 2021-03-15T22:31:30.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 15 2021-03-15T22:31:30.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: T 2021-03-15T22:31:30.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 22 2021-03-15T22:31:30.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 31 2021-03-15T22:31:30.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 29 2021-03-15T22:31:30.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: . 2021-03-15T22:31:30.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 161191 2021-03-15T22:31:30.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: +0000 2021-03-15T22:31:30.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: mgr.y 2021-03-15T22:31:30.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: ( 2021-03-15T22:31:30.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: mgr 2021-03-15T22:31:30.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: . 2021-03-15T22:31:30.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 14144 2021-03-15T22:31:30.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: ) 2021-03-15T22:31:30.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 891 2021-03-15T22:31:30.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: cluster 2021-03-15T22:31:30.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: [DBG] 2021-03-15T22:31:30.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: pgmap v1252: 148 pgs: 1 active+remapped+backfill_toofull, 5 peering, 142 active+clean; 619 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 22 MiB/s rd, 2.7 MiB/s wr, 25 op/s; 3/1887 objects misplaced (0.159%); 0 B/s, 0 objects/s recovering 2021-03-15T22:31:30.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: cluster 2021-03-15T22:31:30.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021 2021-03-15T22:31:30.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: - 2021-03-15T22:31:30.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 03 2021-03-15T22:31:30.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: - 2021-03-15T22:31:30.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 15 2021-03-15T22:31:30.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: T 2021-03-15T22:31:30.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 22 2021-03-15T22:31:30.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 31 2021-03-15T22:31:30.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 29 2021-03-15T22:31:30.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: . 2021-03-15T22:31:30.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 264870 2021-03-15T22:31:30.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: +0000 2021-03-15T22:31:30.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: mon.a 2021-03-15T22:31:30.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: ( 2021-03-15T22:31:30.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: mon 2021-03-15T22:31:30.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: . 2021-03-15T22:31:30.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 0 2021-03-15T22:31:30.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: ) 2021-03-15T22:31:30.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 1965 2021-03-15T22:31:30.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: cluster 2021-03-15T22:31:30.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: [DBG] 2021-03-15T22:31:30.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: osdmap e739: 8 total, 8 up, 7 in 2021-03-15T22:31:30.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: cluster 2021-03-15T22:31:30.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021 2021-03-15T22:31:30.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: - 2021-03-15T22:31:30.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 03 2021-03-15T22:31:30.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: - 2021-03-15T22:31:30.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 15 2021-03-15T22:31:30.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: T 2021-03-15T22:31:30.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 22 2021-03-15T22:31:30.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 31 2021-03-15T22:31:30.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 29 2021-03-15T22:31:30.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: . 2021-03-15T22:31:30.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 265418 2021-03-15T22:31:30.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: +0000 2021-03-15T22:31:30.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: osd.7 2021-03-15T22:31:30.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: ( 2021-03-15T22:31:30.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: osd 2021-03-15T22:31:30.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: . 2021-03-15T22:31:30.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 7 2021-03-15T22:31:30.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: ) 2021-03-15T22:31:30.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 137 2021-03-15T22:31:30.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: cluster 2021-03-15T22:31:30.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: [DBG] 2021-03-15T22:31:30.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 732'8 2021-03-15T22:31:30.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: cluster 2021-03-15T22:31:30.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021 2021-03-15T22:31:30.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: - 2021-03-15T22:31:30.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 03 2021-03-15T22:31:30.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: - 2021-03-15T22:31:30.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 15 2021-03-15T22:31:30.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: T 2021-03-15T22:31:30.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 22 2021-03-15T22:31:30.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 31 2021-03-15T22:31:30.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 29 2021-03-15T22:31:30.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: . 2021-03-15T22:31:30.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 267680 2021-03-15T22:31:30.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: +0000 2021-03-15T22:31:30.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22:31:30.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: osd.4 2021-03-15T22:31:30.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: ( 2021-03-15T22:31:30.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: osd 2021-03-15T22:31:30.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: . 2021-03-15T22:31:30.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 4 2021-03-15T22:31:30.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: ) 2021-03-15T22:31:30.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 159 2021-03-15T22:31:30.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: : 2021-03-15T22:31:30.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: cluster [DBG] 5.1f starting backfill to osd.2 from (0'0,0'0] MAX to 736'13 2021-03-15T22:31:30.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:31:30.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 03-15T22:31:29.268385+0000 osd.5 ( 2021-03-15T22:31:30.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: osd.5) 23 : cluster [DBG] 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 643'9 2021-03-15T22:31:30.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: audit 2021-03-15 2021-03-15T22:31:30.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: T22:31:30.061778+0000 mgr.y (mgr.14144) 892 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:30.795 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: cluster 2021-03-15T22:31:30.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: 2021-03-15T22 2021-03-15T22:31:30.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: :31:30.234305+0000 mon.a (mon.0 2021-03-15T22:31:30.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[56514]: ) 1966 : cluster [DBG] osdmap e740: 8 total, 8 up, 7 in 2021-03-15T22:31:30.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:30.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 5.9 starting backfill to osd.1 from (0'0,0'0] MAX to 732'8 2021-03-15T22:31:30.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021 2021-03-15T22:31:30.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: - 2021-03-15T22:31:30.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 03 2021-03-15T22:31:30.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: - 2021-03-15T22:31:30.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 15 2021-03-15T22:31:30.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: T 2021-03-15T22:31:30.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 22 2021-03-15T22:31:30.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 31 2021-03-15T22:31:30.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 29 2021-03-15T22:31:30.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: . 2021-03-15T22:31:30.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 267680 2021-03-15T22:31:30.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: +0000 2021-03-15T22:31:30.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: osd.4 2021-03-15T22:31:30.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: ( 2021-03-15T22:31:30.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: osd 2021-03-15T22:31:30.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: . 2021-03-15T22:31:30.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 4 2021-03-15T22:31:30.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: ) 2021-03-15T22:31:30.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 159 2021-03-15T22:31:30.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:30.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 5.1f starting backfill to osd.2 from (0'0,0'0] MAX to 736'13 2021-03-15T22:31:30.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021 2021-03-15T22:31:30.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: - 2021-03-15T22:31:30.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 03 2021-03-15T22:31:30.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: - 2021-03-15T22:31:30.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 15 2021-03-15T22:31:30.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: T 2021-03-15T22:31:30.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 22 2021-03-15T22:31:30.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 31 2021-03-15T22:31:30.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 29 2021-03-15T22:31:30.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: . 2021-03-15T22:31:30.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 268385 2021-03-15T22:31:30.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: +0000 2021-03-15T22:31:30.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: osd.5 2021-03-15T22:31:30.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: ( 2021-03-15T22:31:30.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: osd 2021-03-15T22:31:30.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: . 2021-03-15T22:31:30.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 5 2021-03-15T22:31:30.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: ) 2021-03-15T22:31:30.810 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 23 2021-03-15T22:31:30.810 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: : 2021-03-15T22:31:30.810 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.810 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.810 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:30.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 5.f starting backfill to osd.2 from (0'0,0'0] MAX to 643'9 2021-03-15T22:31:30.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: audit 2021-03-15T22:31:30.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: 2021-03-15T22:31:30 2021-03-15T22:31:30.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: .061778+0000 mgr.y (mgr.14144) 892 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:30.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:31:30.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: -03-15T22:31:30.234305 2021-03-15T22:31:30.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: +0000 mon.a (mon.0) 1966 : cluster 2021-03-15T22:31:30.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:30 smithi076 conmon[62156]: [DBG] osdmap e740: 8 total, 8 up, 7 in 2021-03-15T22:31:30.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:30.814 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:30.815 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:30.817 INFO:tasks.rados.rados.0.smithi076.stdout:2734: finishing write tid 1 to smithi07694878-49 2021-03-15T22:31:30.817 INFO:tasks.rados.rados.0.smithi076.stdout:2734: finishing write tid 2 to smithi07694878-49 2021-03-15T22:31:30.817 INFO:tasks.rados.rados.0.smithi076.stdout:2736: read oid 21 snap 305 2021-03-15T22:31:30.818 INFO:tasks.rados.rados.0.smithi076.stdout:2736: expect (ObjNum 870 snap 287 seq_num 870) 2021-03-15T22:31:30.818 INFO:tasks.rados.rados.0.smithi076.stdout:2731: done (7 left) 2021-03-15T22:31:30.818 INFO:tasks.rados.rados.0.smithi076.stdout:2733: done (6 left) 2021-03-15T22:31:30.818 INFO:tasks.rados.rados.0.smithi076.stdout:2737: rollback oid 40 current snap is 318 2021-03-15T22:31:30.819 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 40 to 256 2021-03-15T22:31:30.819 INFO:tasks.rados.rados.0.smithi076.stdout:2734: finishing write tid 3 to smithi07694878-49 2021-03-15T22:31:30.819 INFO:tasks.rados.rados.0.smithi076.stdout:2734: finishing write tid 4 to smithi07694878-49 2021-03-15T22:31:30.819 INFO:tasks.rados.rados.0.smithi076.stdout:2734: finishing write tid 5 to smithi07694878-49 2021-03-15T22:31:30.820 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 2123 (ObjNum 945 snap 318 seq_num 945) dirty exists 2021-03-15T22:31:30.820 INFO:tasks.rados.rados.0.smithi076.stdout:2734: left oid 49 (ObjNum 945 snap 318 seq_num 945) 2021-03-15T22:31:30.820 INFO:tasks.rados.rados.0.smithi076.stdout:2735: finishing write tid 1 to smithi07694878-24 2021-03-15T22:31:30.820 INFO:tasks.rados.rados.0.smithi076.stdout:2735: finishing write tid 2 to smithi07694878-24 2021-03-15T22:31:30.820 INFO:tasks.rados.rados.0.smithi076.stdout:2735: finishing write tid 3 to smithi07694878-24 2021-03-15T22:31:30.821 INFO:tasks.rados.rados.0.smithi076.stdout:2735: finishing write tid 4 to smithi07694878-24 2021-03-15T22:31:30.821 INFO:tasks.rados.rados.0.smithi076.stdout:2735: finishing write tid 5 to smithi07694878-24 2021-03-15T22:31:30.821 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1718 (ObjNum 946 snap 318 seq_num 946) dirty exists 2021-03-15T22:31:30.821 INFO:tasks.rados.rados.0.smithi076.stdout:2735: left oid 24 (ObjNum 946 snap 318 seq_num 946) 2021-03-15T22:31:30.821 INFO:tasks.rados.rados.0.smithi076.stdout:2734: done (6 left) 2021-03-15T22:31:30.822 INFO:tasks.rados.rados.0.smithi076.stdout:2735: done (5 left) 2021-03-15T22:31:30.822 INFO:tasks.rados.rados.0.smithi076.stdout:2738: rollback oid 11 current snap is 318 2021-03-15T22:31:30.822 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 11 to 311 2021-03-15T22:31:30.822 INFO:tasks.rados.rados.0.smithi076.stdout:2739: write (excl) oid 12 current snap is 318 2021-03-15T22:31:30.823 INFO:tasks.rados.rados.0.smithi076.stdout:2739: seq_num 947 ranges {716229=488575,1616330=541328,2756280=559655} 2021-03-15T22:31:30.823 INFO:tasks.rados.rados.0.smithi076.stdout:2739: writing smithi07694878-12 from 716229 to 1204804 tid 1 2021-03-15T22:31:30.829 INFO:tasks.rados.rados.0.smithi076.stdout:2739: writing smithi07694878-12 from 1616330 to 2157658 tid 2 2021-03-15T22:31:30.836 INFO:tasks.rados.rados.0.smithi076.stdout:2739: writing smithi07694878-12 from 2756280 to 3315935 tid 3 2021-03-15T22:31:30.836 INFO:tasks.rados.rados.0.smithi076.stdout:2740: write (excl) oid 48 current snap is 318 2021-03-15T22:31:30.836 INFO:tasks.rados.rados.0.smithi076.stdout:2738: finishing rollback tid 0 to smithi07694878-11 2021-03-15T22:31:30.837 INFO:tasks.rados.rados.0.smithi076.stdout:2740: seq_num 948 ranges {733036=563824,2074812=684986,3183703=749333} 2021-03-15T22:31:30.843 INFO:tasks.rados.rados.0.smithi076.stdout:2740: writing smithi07694878-48 from 733036 to 1296860 tid 1 2021-03-15T22:31:30.851 INFO:tasks.rados.rados.0.smithi076.stdout:2740: writing smithi07694878-48 from 2074812 to 2759798 tid 2 2021-03-15T22:31:30.858 INFO:tasks.rados.rados.0.smithi076.stdout:2740: writing smithi07694878-48 from 3183703 to 3933036 tid 3 2021-03-15T22:31:30.858 INFO:tasks.rados.rados.0.smithi076.stdout:2741: delete oid 37 current snap is 318 2021-03-15T22:31:30.858 INFO:tasks.rados.rados.0.smithi076.stdout:2738: finishing rollback tid 1 to smithi07694878-11 2021-03-15T22:31:30.858 INFO:tasks.rados.rados.0.smithi076.stdout:2738: finishing rollback tid 2 to smithi07694878-11 2021-03-15T22:31:30.859 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1617 (ObjNum 897 snap 302 seq_num 897) dirty exists 2021-03-15T22:31:30.859 INFO:tasks.rados.rados.0.smithi076.stdout:2739: finishing write tid 1 to smithi07694878-12 2021-03-15T22:31:30.859 INFO:tasks.rados.rados.0.smithi076.stdout:2736: expect (ObjNum 870 snap 287 seq_num 870) 2021-03-15T22:31:30.916 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:30.938 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:30.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:30.940 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:30.979 INFO:tasks.rados.rados.0.smithi076.stdout:2737: finishing rollback tid 0 to smithi07694878-40 2021-03-15T22:31:30.979 INFO:tasks.rados.rados.0.smithi076.stdout:2737: finishing rollback tid 1 to smithi07694878-40 2021-03-15T22:31:30.979 INFO:tasks.rados.rados.0.smithi076.stdout:2737: finishing rollback tid 2 to smithi07694878-40 2021-03-15T22:31:30.980 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 2274 (ObjNum 766 snap 255 seq_num 766) dirty exists 2021-03-15T22:31:30.980 INFO:tasks.rados.rados.0.smithi076.stdout:2739: finishing write tid 2 to smithi07694878-12 2021-03-15T22:31:30.980 INFO:tasks.rados.rados.0.smithi076.stdout:2736: done (8 left) 2021-03-15T22:31:30.980 INFO:tasks.rados.rados.0.smithi076.stdout:2737: done (7 left) 2021-03-15T22:31:30.981 INFO:tasks.rados.rados.0.smithi076.stdout:2738: done (6 left) 2021-03-15T22:31:30.981 INFO:tasks.rados.rados.0.smithi076.stdout:2741: done (5 left) 2021-03-15T22:31:30.981 INFO:tasks.rados.rados.0.smithi076.stdout:2742: read oid 2 snap -1 2021-03-15T22:31:30.981 INFO:tasks.rados.rados.0.smithi076.stdout:2742: expect (ObjNum 931 snap 315 seq_num 931) 2021-03-15T22:31:30.982 INFO:tasks.rados.rados.0.smithi076.stdout:2743: delete oid 36 current snap is 318 2021-03-15T22:31:30.982 INFO:tasks.rados.rados.0.smithi076.stdout:2739: finishing write tid 3 to smithi07694878-12 2021-03-15T22:31:30.982 INFO:tasks.rados.rados.0.smithi076.stdout:2739: finishing write tid 4 to smithi07694878-12 2021-03-15T22:31:30.982 INFO:tasks.rados.rados.0.smithi076.stdout:2739: finishing write tid 5 to smithi07694878-12 2021-03-15T22:31:30.982 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 2127 (ObjNum 947 snap 318 seq_num 947) dirty exists 2021-03-15T22:31:30.983 INFO:tasks.rados.rados.0.smithi076.stdout:2739: left oid 12 (ObjNum 947 snap 318 seq_num 947) 2021-03-15T22:31:30.983 INFO:tasks.rados.rados.0.smithi076.stdout:2739: done (6 left) 2021-03-15T22:31:30.984 INFO:tasks.rados.rados.0.smithi076.stdout:2743: done (5 left) 2021-03-15T22:31:30.984 INFO:tasks.rados.rados.0.smithi076.stdout:2744: copy_from oid 26 from oid 22 current snap is 318 2021-03-15T22:31:30.984 INFO:tasks.rados.rados.0.smithi076.stdout:2745: snap_create 2021-03-15T22:31:31.041 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:31.062 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:31.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:31.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:31.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:31.187 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:31.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:31.189 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:31.238 INFO:tasks.rados.rados.0.smithi076.stdout:2745: done (6 left) 2021-03-15T22:31:31.239 INFO:tasks.rados.rados.0.smithi076.stdout:2746: read oid 40 snap -1 2021-03-15T22:31:31.239 INFO:tasks.rados.rados.0.smithi076.stdout:2746: expect (ObjNum 766 snap 255 seq_num 766) 2021-03-15T22:31:31.239 INFO:tasks.rados.rados.0.smithi076.stdout:2747: snap_remove snap 256 2021-03-15T22:31:31.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:31.312 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:31.314 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:31.416 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:31.438 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:31.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:31.440 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:31.465 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:31:31.520 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:31.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:31 smithi125 conmon[56739]: cluster 2021-03-15T22:31:30.224606+0000 osd.2 (osd.2) 14 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:31 smithi125 conmon[56739]: cluster 2021-03-15T22:31:30.225947+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:31 smithi125 conmon[56739]: audit 2021-03-15T22:31:30.526962+0000 mon.c (mon.1) 60 : audit [DBG] from='client.? 172.21.15.76:0/4050327537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:31.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:31 smithi125 conmon[56739]: cluster 2021-03-15T22:31:31. 2021-03-15T22:31:31.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:31 smithi125 conmon[56739]: 235361+0000 mon.a (mon.0) 1967 : cluster 2021-03-15T22:31:31.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:31 smithi125 conmon[56739]: [DBG] osdmap e741: 8 total, 8 up, 7 in 2021-03-15T22:31:31.541 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:31.563 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:31.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:31.565 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:31.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:31.690 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:31.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:31.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:31.693 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:31 smithi076 conmon[56514]: cluster 2021-03-15T22:31:30.224606+0000 osd.2 (osd.2) 14 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:31.693 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:31 smithi076 conmon[56514]: cluster 2021-03-15T22:31:30.225947+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:31.693 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:31 smithi076 conmon[56514]: audit 2021-03-15T22:31:30.526962+0000 mon.c (mon.1) 60 : audit [DBG] from='client.? 172.21.15.76:0/4050327537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:31.694 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:31 smithi076 conmon[56514]: cluster 2021-03-15T22:31:31.235361+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2021-03-15T22:31:31.694 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:31 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.224606+0000 osd.2 (osd.2) 14 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:31.694 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:31 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.225947+0000 osd.2 (osd.2) 15 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:31.695 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:31 smithi076 conmon[62156]: audit 2021-03-15T22:31:30.526962+0000 mon.c (mon.1) 60 : audit [DBG] from='client.? 172.21.15.76:0/4050327537' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:31.696 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:31 smithi076 conmon[62156]: cluster 2021-03-15T22:31:31.235361+0000 mon.a (mon.0) 1967 : cluster [DBG] osdmap e741: 8 total, 8 up, 7 in 2021-03-15T22:31:31.793 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:31.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:31.817 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:31.817 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:31.919 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:31.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:31.944 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:31.944 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:32.037 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:32.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:32.071 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:32.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:32.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:32.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:32.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:32.197 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:32.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:32.246 INFO:tasks.rados.rados.0.smithi076.stdout:2747: done (7 left) 2021-03-15T22:31:32.246 INFO:tasks.rados.rados.0.smithi076.stdout:2748: delete oid 32 current snap is 319 2021-03-15T22:31:32.246 INFO:tasks.rados.rados.0.smithi076.stdout:2740: finishing write tid 1 to smithi07694878-48 2021-03-15T22:31:32.246 INFO:tasks.rados.rados.0.smithi076.stdout:2727: expect (ObjNum 917 snap 309 seq_num 917) 2021-03-15T22:31:32.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:32.323 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:32.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:32.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:32.360 INFO:tasks.rados.rados.0.smithi076.stdout:2746: expect (ObjNum 766 snap 255 seq_num 766) 2021-03-15T22:31:32.427 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:32.450 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:32.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:32.452 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:32.474 INFO:tasks.rados.rados.0.smithi076.stdout:2740: finishing write tid 2 to smithi07694878-48 2021-03-15T22:31:32.474 INFO:tasks.rados.rados.0.smithi076.stdout:2726: expect (ObjNum 623 snap 193 seq_num 623) 2021-03-15T22:31:32.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:32 smithi076 conmon[56514]: cluster 2021-03-15T22:31:30.979396+0000 osd.2 (osd.2) 16 : 2021-03-15T22:31:32.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:32 smithi076 conmon[56514]: cluster [DBG] 7.e scrub ok 2021-03-15T22:31:32.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:32 smithi076 conmon[56514]: cluster 2021-03-15T22:31:31.161672+0000 mgr.y (mgr.14144) 893 : cluster [DBG] pgmap v1255: 148 pgs: 1 active+remapped+backfill_toofull, 2 remapped+peering, 6 peering, 139 active+clean; 639 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 3.3 MiB/s wr, 14 op/s; 3/1908 objects misplaced (0.157%); 9.2 MiB/s, 19 objects/s recovering 2021-03-15T22:31:32.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:32 smithi076 conmon[56514]: cluster 2021-03-15T22:31:32.236765+0000 mon.a (mon.0) 1968 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2021-03-15T22:31:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:32 smithi076 conmon[62156]: cluster 2021-03-15T22:31:30.979396+0000 2021-03-15T22:31:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:32 smithi076 conmon[62156]: osd.2 (osd.2) 16 : cluster [DBG] 7.e scrub ok 2021-03-15T22:31:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:32 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:31:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:32 smithi076 conmon[62156]: T22:31:31.161672+0000 mgr.y (mgr.14144) 893 : cluster [DBG] pgmap v1255: 148 pgs: 1 active+remapped+backfill_toofull, 2 remapped+peering, 6 peering, 139 active+clean; 639 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 3.3 MiB/s wr, 14 op/s; 3/1908 objects misplaced (0.157%); 9.2 MiB/s, 19 objects/s recovering 2021-03-15T22:31:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:32 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:31:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:32 smithi076 conmon[62156]: 15T22:31:32.236765+0000 mon.a (mon.0 2021-03-15T22:31:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:32 smithi076 conmon[62156]: ) 1968 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2021-03-15T22:31:32.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:32 smithi125 conmon[56739]: cluster 2021-03-15T22:31:30.979396+0000 osd.2 (osd.2) 16 : cluster [DBG] 7.e scrub ok 2021-03-15T22:31:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:32 smithi125 conmon[56739]: 2021-03-15T22:31:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:32 smithi125 conmon[56739]: cluster 2021-03-15T22:31:31.161672+0000 mgr.y (mgr.14144) 893 : cluster [DBG] pgmap v1255: 148 pgs: 1 active+remapped+backfill_toofull, 2 remapped+peering, 6 peering, 139 active+clean; 639 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 3.3 MiB/s wr, 14 op/s; 3/1908 objects misplaced (0.157%); 9.2 MiB/s, 19 objects/s recovering 2021-03-15T22:31:32.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:32 smithi125 conmon[56739]: cluster 2021-03-15T22:31:32.236765+0000 mon.a (mon.0) 1968 : cluster [DBG] osdmap e742: 8 total, 8 up, 7 in 2021-03-15T22:31:32.553 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:32.577 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:32.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:32.578 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:32.583 INFO:tasks.rados.rados.0.smithi076.stdout:2740: finishing write tid 3 to smithi07694878-48 2021-03-15T22:31:32.583 INFO:tasks.rados.rados.0.smithi076.stdout:2740: finishing write tid 4 to smithi07694878-48 2021-03-15T22:31:32.583 INFO:tasks.rados.rados.0.smithi076.stdout:2740: finishing write tid 5 to smithi07694878-48 2021-03-15T22:31:32.584 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 1235 (ObjNum 948 snap 318 seq_num 948) dirty exists 2021-03-15T22:31:32.584 INFO:tasks.rados.rados.0.smithi076.stdout:2740: left oid 48 (ObjNum 948 snap 318 seq_num 948) 2021-03-15T22:31:32.584 INFO:tasks.rados.rados.0.smithi076.stdout:2742: expect (ObjNum 931 snap 315 seq_num 931) 2021-03-15T22:31:32.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:32.704 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:32.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:32.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:32.806 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:32.828 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:32.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:32.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:32.865 INFO:tasks.rados.rados.0.smithi076.stdout:2744: finishing copy_from to smithi07694878-26 2021-03-15T22:31:32.866 INFO:tasks.rados.rados.0.smithi076.stdout:2744: got expected ENOENT (src dne) 2021-03-15T22:31:32.866 INFO:tasks.rados.rados.0.smithi076.stdout:2744: finishing copy_from racing read to smithi07694878-26 2021-03-15T22:31:32.866 INFO:tasks.rados.rados.0.smithi076.stdout:2723: finishing copy_from to smithi07694878-42 2021-03-15T22:31:32.866 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 1957 (ObjNum 935 snap 316 seq_num 935) dirty exists 2021-03-15T22:31:32.867 INFO:tasks.rados.rados.0.smithi076.stdout:2723: finishing copy_from racing read to smithi07694878-42 2021-03-15T22:31:32.867 INFO:tasks.rados.rados.0.smithi076.stdout:2723: done (7 left) 2021-03-15T22:31:32.867 INFO:tasks.rados.rados.0.smithi076.stdout:2726: done (6 left) 2021-03-15T22:31:32.867 INFO:tasks.rados.rados.0.smithi076.stdout:2727: done (5 left) 2021-03-15T22:31:32.868 INFO:tasks.rados.rados.0.smithi076.stdout:2740: done (4 left) 2021-03-15T22:31:32.868 INFO:tasks.rados.rados.0.smithi076.stdout:2742: done (3 left) 2021-03-15T22:31:32.868 INFO:tasks.rados.rados.0.smithi076.stdout:2744: done (2 left) 2021-03-15T22:31:32.868 INFO:tasks.rados.rados.0.smithi076.stdout:2746: done (1 left) 2021-03-15T22:31:32.868 INFO:tasks.rados.rados.0.smithi076.stdout:2748: done (0 left) 2021-03-15T22:31:32.869 INFO:tasks.rados.rados.0.smithi076.stdout:2749: copy_from oid 19 from oid 35 current snap is 319 2021-03-15T22:31:32.869 INFO:tasks.rados.rados.0.smithi076.stdout:2750: rollback oid 32 current snap is 319 2021-03-15T22:31:32.870 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 32 to 242 2021-03-15T22:31:32.870 INFO:tasks.rados.rados.0.smithi076.stdout:2751: rollback oid 11 current snap is 319 2021-03-15T22:31:32.871 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 11 to 292 2021-03-15T22:31:32.871 INFO:tasks.rados.rados.0.smithi076.stdout:2752: snap_remove snap 275 2021-03-15T22:31:32.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:32.956 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:32.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:32.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:33.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:33.082 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:33.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:33.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:33.184 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:33.209 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:33.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:33.211 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:33.244 INFO:tasks.rados.rados.0.smithi076.stdout:2752: done (3 left) 2021-03-15T22:31:33.245 INFO:tasks.rados.rados.0.smithi076.stdout:2753: write (excl) oid 24 current snap is 319 2021-03-15T22:31:33.245 INFO:tasks.rados.rados.0.smithi076.stdout:2753: seq_num 952 ranges {400264=581173,1508740=491524} 2021-03-15T22:31:33.249 INFO:tasks.rados.rados.0.smithi076.stdout:2753: writing smithi07694878-24 from 400264 to 981437 tid 1 2021-03-15T22:31:33.255 INFO:tasks.rados.rados.0.smithi076.stdout:2753: writing smithi07694878-24 from 1508740 to 2000264 tid 2 2021-03-15T22:31:33.256 INFO:tasks.rados.rados.0.smithi076.stdout:2751: finishing rollback tid 0 to smithi07694878-11 2021-03-15T22:31:33.257 INFO:tasks.rados.rados.0.smithi076.stdout:2750: finishing rollback tid 1 to smithi07694878-32 2021-03-15T22:31:33.257 INFO:tasks.rados.rados.0.smithi076.stdout:2751: finishing rollback tid 1 to smithi07694878-11 2021-03-15T22:31:33.257 INFO:tasks.rados.rados.0.smithi076.stdout:2751: finishing rollback tid 2 to smithi07694878-11 2021-03-15T22:31:33.257 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1620 (ObjNum 866 snap 286 seq_num 866) dirty exists 2021-03-15T22:31:33.258 INFO:tasks.rados.rados.0.smithi076.stdout:2750: finishing rollback tid 2 to smithi07694878-32 2021-03-15T22:31:33.258 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 2294 (ObjNum 717 snap 233 seq_num 717) dirty exists 2021-03-15T22:31:33.258 INFO:tasks.rados.rados.0.smithi076.stdout:2749: finishing copy_from to smithi07694878-19 2021-03-15T22:31:33.258 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 1611 (ObjNum 863 snap 285 seq_num 863) dirty exists 2021-03-15T22:31:33.259 INFO:tasks.rados.rados.0.smithi076.stdout:2749: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:31:33.259 INFO:tasks.rados.rados.0.smithi076.stdout:2749: done (3 left) 2021-03-15T22:31:33.259 INFO:tasks.rados.rados.0.smithi076.stdout:2750: done (2 left) 2021-03-15T22:31:33.259 INFO:tasks.rados.rados.0.smithi076.stdout:2751: done (1 left) 2021-03-15T22:31:33.259 INFO:tasks.rados.rados.0.smithi076.stdout:2754: read oid 50 snap -1 2021-03-15T22:31:33.260 INFO:tasks.rados.rados.0.smithi076.stdout:2754: expect (ObjNum 598 snap 183 seq_num 598) 2021-03-15T22:31:33.260 INFO:tasks.rados.rados.0.smithi076.stdout:2755: snap_remove snap 253 2021-03-15T22:31:33.287 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:33 smithi076 conmon[56514]: cluster 2021-03-15T22:31:32.008400+0000 osd.2 (osd.2) 17 : cluster [DBG] 5.b scrub ok 2021-03-15T22:31:33.288 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:33 smithi076 conmon[56514]: cluster 2021-03-15T22:31:33.238185+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2021-03-15T22:31:33.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:33 smithi076 conmon[62156]: cluster 2021-03-15T22:31:32.008400+0000 osd.2 (osd.2) 17 : cluster [DBG] 5.b scrub ok 2021-03-15T22:31:33.288 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:33 smithi076 conmon[62156]: cluster 2021-03-15T22:31:33.238185+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2021-03-15T22:31:33.312 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:33.313 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:31:33.313 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:31:33.314 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:31:33.314 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:31:33.337 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:33.339 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:33.440 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:33.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:33.464 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:33.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:33.467 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 7, 0, 3, 5] out_osds: [6] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:31:33.467 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:31:33.468 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.meta 2021-03-15T22:31:33.469 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:31:33.484 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:31:33.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:33 smithi125 conmon[56739]: cluster 2021-03-15T22:31:32.008400+0000 osd.2 (osd.2) 17 : cluster [DBG] 2021-03-15T22:31:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:33 smithi125 conmon[56739]: 5.b scrub ok 2021-03-15T22:31:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:33 smithi125 conmon[56739]: cluster 2021-03-15T22:31:33.238185+0000 mon.a (mon.0) 1969 : cluster [DBG] osdmap e743: 8 total, 8 up, 7 in 2021-03-15T22:31:33.566 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:33.593 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:33.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:33.595 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:33.696 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:33.721 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:33.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:33.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:33.824 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:33.850 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:33.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:33.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:33.953 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:33.979 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:33.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:33.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:34.081 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:34.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:34.106 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:34.106 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:34.208 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:34.234 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:34.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:34.236 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:34.248 INFO:tasks.rados.rados.0.smithi076.stdout:2755: done (2 left) 2021-03-15T22:31:34.249 INFO:tasks.rados.rados.0.smithi076.stdout:2756: rollback oid 13 current snap is 319 2021-03-15T22:31:34.249 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 13 to 315 2021-03-15T22:31:34.249 INFO:tasks.rados.rados.0.smithi076.stdout:2757: snap_create 2021-03-15T22:31:34.249 INFO:tasks.rados.rados.0.smithi076.stdout:2754: expect (ObjNum 598 snap 183 seq_num 598) 2021-03-15T22:31:34.323 INFO:tasks.rados.rados.0.smithi076.stdout:2753: finishing write tid 1 to smithi07694878-24 2021-03-15T22:31:34.323 INFO:tasks.rados.rados.0.smithi076.stdout:2753: finishing write tid 2 to smithi07694878-24 2021-03-15T22:31:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:2753: finishing write tid 3 to smithi07694878-24 2021-03-15T22:31:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:2753: finishing write tid 4 to smithi07694878-24 2021-03-15T22:31:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1721 (ObjNum 952 snap 319 seq_num 952) dirty exists 2021-03-15T22:31:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:2753: left oid 24 (ObjNum 952 snap 319 seq_num 952) 2021-03-15T22:31:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:2756: finishing rollback tid 1 to smithi07694878-13 2021-03-15T22:31:34.325 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1566 (ObjNum 264916240 snap 21896 seq_num 262081120) dirty dne 2021-03-15T22:31:34.336 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:34.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:34.362 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:34.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:34.464 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:34.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:34.488 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:34.489 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:34.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[56514]: cluster 2021-03-15T22:31:33.136108+0000 osd.0 (osd.0 2021-03-15T22:31:34.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[56514]: ) 120 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:31:34.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[56514]: cluster 2021-03-15T22:31:33.162108+0000 mgr.y (mgr.14144) 894 : cluster [DBG] pgmap v1258: 148 pgs: 1 active+undersized+remapped, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 peering, 142 active+clean; 642 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 21 op/s; 4/1911 objects misplaced (0.209%); 34 MiB/s, 97 objects/s recovering 2021-03-15T22:31:34.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[56514]: cluster 2021-03-15T22:31:33.275477+0000 mon.a (mon.0) 1970 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:31:34.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[56514]: cluster 2021-03-15T22:31:34.243012+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2021-03-15T22:31:34.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[62156]: cluster 2021-03-15T22:31:33.136108+0000 2021-03-15T22:31:34.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[62156]: osd.0 (osd.0) 120 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:31:34.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:31:34.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[62156]: 31:33.162108+0000 mgr.y (mgr.14144) 894 : cluster [DBG] pgmap v1258: 148 pgs: 1 active+undersized+remapped, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 peering, 142 active+clean; 642 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 21 op/s; 4/1911 objects misplaced (0.209%); 34 MiB/s, 97 objects/s recovering 2021-03-15T22:31:34.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:31:34.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[62156]: -03-15T22:31:33.275477+0000 mon.a ( 2021-03-15T22:31:34.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[62156]: mon.0) 1970 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:31:34.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:31:34.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[62156]: T22:31:34.243012+0000 mon.a (mon.0) 2021-03-15T22:31:34.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:34 smithi076 conmon[62156]: 1971 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2021-03-15T22:31:34.590 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:31:34.591 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:34.617 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:34.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:34.619 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:34.676 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:34.677 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:34.720 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:34.743 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:34.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:34.745 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:34 smithi125 conmon[56739]: cluster 2021-03-15T22:31:33.136108+0000 osd.0 (osd.0) 120 : cluster [DBG] 3.17 scrub ok 2021-03-15T22:31:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:34 smithi125 conmon[56739]: 2021-03-15T22:31:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:34 smithi125 conmon[56739]: cluster 2021-03-15T22:31:33.162108+0000 mgr.y (mgr.14144) 894 : cluster [DBG] pgmap v1258: 148 pgs: 1 active+undersized+remapped, 2 active+remapped+backfill_toofull, 2 remapped+peering, 1 peering, 142 active+clean; 642 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 21 op/s; 4/1911 objects misplaced (0.209%); 34 MiB/s, 97 objects/s recovering 2021-03-15T22:31:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:34 smithi125 conmon[56739]: cluster 2021-03-15T22:31:33.275477+0000 mon.a (mon.0) 1970 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:31:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:34 smithi125 conmon[56739]: cluster 2021-03-15T22:31:34.243012+0000 mon.a (mon.0) 1971 : cluster [DBG] osdmap e744: 8 total, 8 up, 7 in 2021-03-15T22:31:34.846 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:34.872 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:34.872 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:34.973 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:34.996 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:34.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:34.997 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:35.100 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:35.122 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:35.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:35.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:35.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:35.242 INFO:tasks.rados.rados.0.smithi076.stdout:2753: done (3 left) 2021-03-15T22:31:35.242 INFO:tasks.rados.rados.0.smithi076.stdout:2754: done (2 left) 2021-03-15T22:31:35.243 INFO:tasks.rados.rados.0.smithi076.stdout:2756: done (1 left) 2021-03-15T22:31:35.243 INFO:tasks.rados.rados.0.smithi076.stdout:2757: done (0 left) 2021-03-15T22:31:35.243 INFO:tasks.rados.rados.0.smithi076.stdout:2758: delete oid 40 current snap is 320 2021-03-15T22:31:35.256 INFO:tasks.rados.rados.0.smithi076.stdout:2758: done (0 left) 2021-03-15T22:31:35.256 INFO:tasks.rados.rados.0.smithi076.stdout:2759: read oid 22 snap 301 2021-03-15T22:31:35.257 INFO:tasks.rados.rados.0.smithi076.stdout:2759: expect (ObjNum 750 snap 245 seq_num 750) 2021-03-15T22:31:35.257 INFO:tasks.rados.rados.0.smithi076.stdout:2760: read oid 18 snap 309 2021-03-15T22:31:35.257 INFO:tasks.rados.rados.0.smithi076.stdout:2760: expect (ObjNum 881 snap 292 seq_num 881) 2021-03-15T22:31:35.257 INFO:tasks.rados.rados.0.smithi076.stdout:2761: snap_remove snap 196 2021-03-15T22:31:35.264 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:35.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:35.266 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:35.268 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:31:35.303 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:35.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:35.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:35.393 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:35.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:35.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[56514]: cluster 2021-03-15T22:31:33.964074+0000 osd.2 (osd.2) 18 : cluster [DBG] 5.a scrub ok 2021-03-15T22:31:35.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[56514]: cluster 2021-03-15T22:31:34.511798+0000 osd.7 (osd.7) 138 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:31:35.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[56514]: audit 2021-03-15T22:31:35.162861+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? 172.21.15.76:0/2202070519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:35.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[56514]: cluster 2021-03-15T22:31:35.231672+0000 mon.a (mon.0) 1973 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:31:35.472 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[56514]: audit 2021-03-15T22:31:35.237135+0000 mon.a (mon.0) 1974 : audit [INF] from='client.? 172.21.15.76:0/2202070519' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:35.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[56514]: cluster 2021-03-15T22:31:35.239711+0000 mon.a (mon.0) 1975 : cluster [DBG] osdmap e745: 8 total, 8 up, 7 in 2021-03-15T22:31:35.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: cluster 2021-03-15T22:31:35.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021 2021-03-15T22:31:35.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 03 2021-03-15T22:31:35.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 15 2021-03-15T22:31:35.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: T 2021-03-15T22:31:35.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 22 2021-03-15T22:31:35.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 31 2021-03-15T22:31:35.479 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 33 2021-03-15T22:31:35.480 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 964074 2021-03-15T22:31:35.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: +0000 2021-03-15T22:31:35.481 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.482 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: osd.2 2021-03-15T22:31:35.482 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ( 2021-03-15T22:31:35.483 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: osd 2021-03-15T22:31:35.483 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2 2021-03-15T22:31:35.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ) 2021-03-15T22:31:35.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 18 2021-03-15T22:31:35.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: cluster 2021-03-15T22:31:35.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:35.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 5.a scrub ok 2021-03-15T22:31:35.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.488 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: cluster 2021-03-15T22:31:35.488 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021 2021-03-15T22:31:35.489 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 03 2021-03-15T22:31:35.490 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 15 2021-03-15T22:31:35.491 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: T 2021-03-15T22:31:35.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 22 2021-03-15T22:31:35.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 31 2021-03-15T22:31:35.493 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 34 2021-03-15T22:31:35.495 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 511798 2021-03-15T22:31:35.496 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: +0000 2021-03-15T22:31:35.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.497 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: osd.7 2021-03-15T22:31:35.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ( 2021-03-15T22:31:35.498 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: osd 2021-03-15T22:31:35.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 7 2021-03-15T22:31:35.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ) 2021-03-15T22:31:35.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 138 2021-03-15T22:31:35.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: cluster 2021-03-15T22:31:35.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:35.502 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 3.11 scrub ok 2021-03-15T22:31:35.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: audit 2021-03-15T22:31:35.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021 2021-03-15T22:31:35.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 03 2021-03-15T22:31:35.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 15 2021-03-15T22:31:35.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: T 2021-03-15T22:31:35.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 22 2021-03-15T22:31:35.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 31 2021-03-15T22:31:35.507 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 35 2021-03-15T22:31:35.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 162861 2021-03-15T22:31:35.508 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: +0000 2021-03-15T22:31:35.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:35.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ( 2021-03-15T22:31:35.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: mon 2021-03-15T22:31:35.509 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 0 2021-03-15T22:31:35.510 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ) 2021-03-15T22:31:35.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 1972 2021-03-15T22:31:35.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: audit 2021-03-15T22:31:35.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.511 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: [INF] 2021-03-15T22:31:35.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2202070519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:35.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.512 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: cluster 2021-03-15T22:31:35.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021 2021-03-15T22:31:35.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 03 2021-03-15T22:31:35.513 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 15 2021-03-15T22:31:35.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: T 2021-03-15T22:31:35.514 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 22 2021-03-15T22:31:35.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.515 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 31 2021-03-15T22:31:35.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 35 2021-03-15T22:31:35.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 231672 2021-03-15T22:31:35.516 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: +0000 2021-03-15T22:31:35.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:35.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ( 2021-03-15T22:31:35.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: mon 2021-03-15T22:31:35.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 0 2021-03-15T22:31:35.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ) 2021-03-15T22:31:35.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 1973 2021-03-15T22:31:35.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: cluster 2021-03-15T22:31:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: [INF] 2021-03-15T22:31:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:31:35.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: audit 2021-03-15T22:31:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021 2021-03-15T22:31:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 03 2021-03-15T22:31:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 15 2021-03-15T22:31:35.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: T 2021-03-15T22:31:35.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 22 2021-03-15T22:31:35.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 31 2021-03-15T22:31:35.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 35 2021-03-15T22:31:35.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 237135 2021-03-15T22:31:35.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: +0000 2021-03-15T22:31:35.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:35.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ( 2021-03-15T22:31:35.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: mon 2021-03-15T22:31:35.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 0 2021-03-15T22:31:35.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ) 2021-03-15T22:31:35.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 1974 2021-03-15T22:31:35.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: audit 2021-03-15T22:31:35.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: [INF] 2021-03-15T22:31:35.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2202070519' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:35.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: cluster 2021-03-15T22:31:35.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021 2021-03-15T22:31:35.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 03 2021-03-15T22:31:35.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: - 2021-03-15T22:31:35.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 15 2021-03-15T22:31:35.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: T 2021-03-15T22:31:35.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 22 2021-03-15T22:31:35.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 31 2021-03-15T22:31:35.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 35 2021-03-15T22:31:35.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 239711 2021-03-15T22:31:35.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: +0000 2021-03-15T22:31:35.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: mon.a 2021-03-15T22:31:35.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ( 2021-03-15T22:31:35.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: mon 2021-03-15T22:31:35.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: . 2021-03-15T22:31:35.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 0 2021-03-15T22:31:35.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: ) 2021-03-15T22:31:35.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 1975 2021-03-15T22:31:35.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: : 2021-03-15T22:31:35.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: cluster 2021-03-15T22:31:35.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: [DBG] 2021-03-15T22:31:35.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: osdmap e745: 8 total, 8 up, 7 in 2021-03-15T22:31:35.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:35 smithi076 conmon[62156]: 2021-03-15T22:31:35.540 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:35.566 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:35.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:35.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:35.669 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:35.691 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:35.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:35.692 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:35.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: cluster 2021-03-15T22:31:33.964074+0000 osd.2 (osd.2) 18 : cluster 2021-03-15T22:31:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: [DBG] 5.a scrub ok 2021-03-15T22:31:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: cluster 2021-03-15T22:31:34.511798+0000 osd.7 (osd.7) 138 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:31:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:31:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: :31:35.162861+0000 mon.a (mon.0) 1972 : audit [INF] from='client.? 172.21.15.76:0/2202070519' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:31:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: 31:35.231672+0000 mon.a (mon.0) 2021-03-15T22:31:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: 1973 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:31:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: audit 2021 2021-03-15T22:31:35.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: -03-15T22:31:35.237135+0000 2021-03-15T22:31:35.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: mon.a (mon.0) 1974 : audit [INF] from='client.? 172.21.15.76:0/2202070519' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:35.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: 2021-03-15T22:31:35.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: cluster 2021-03-15T22:31:35. 2021-03-15T22:31:35.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: 239711+0000 mon.a (mon.0) 1975 : cluster 2021-03-15T22:31:35.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:35 smithi125 conmon[56739]: [DBG] osdmap e745: 8 total, 8 up, 7 in 2021-03-15T22:31:35.793 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:35.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:35.821 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:35.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:35.922 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:35.946 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:35.946 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:36.048 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:36.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:36.076 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:36.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:36.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:36.203 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:36.205 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:36.249 INFO:tasks.rados.rados.0.smithi076.stdout:2761: done (2 left) 2021-03-15T22:31:36.249 INFO:tasks.rados.rados.0.smithi076.stdout:2762: snap_remove snap 299 2021-03-15T22:31:36.306 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:36.329 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:36.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:36.432 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:36.455 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:36.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:36.457 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:36.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[56514]: cluster 2021-03-15T22:31:34.960263+0000 osd.1 (osd 2021-03-15T22:31:36.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[56514]: .1) 143 : cluster [DBG] 7.b scrub ok 2021-03-15T22:31:36.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[56514]: cluster 2021-03-15T22:31:35.162592+0000 mgr.y (mgr.14144) 895 : cluster [DBG] pgmap v1261: 148 pgs: 1 active+clean+snaptrim, 1 active+undersized+remapped, 3 active+remapped+backfill_toofull, 1 peering, 142 active+clean; 655 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 15 MiB/s rd, 2.9 MiB/s wr, 22 op/s; 4/1914 objects misplaced (0.209%); 25 MiB/s, 78 objects/s recovering 2021-03-15T22:31:36.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[56514]: cluster 2021-03-15T22:31:35.474925+0000 osd.3 (osd.3) 133 : cluster 2021-03-15T22:31:36.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[56514]: [DBG] 7.2 scrub ok 2021-03-15T22:31:36.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[56514]: cluster 2021-03-15T22:31:35.559603+0000 2021-03-15T22:31:36.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[56514]: osd.7 (osd.7) 139 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:31:36.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:31:36.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[56514]: -03-15T22:31:36.245920+0000 2021-03-15T22:31:36.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[56514]: mon.a (mon.0) 1976 : cluster 2021-03-15T22:31:36.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[56514]: [DBG] osdmap e746: 8 total, 8 up, 7 in 2021-03-15T22:31:36.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[62156]: cluster 2021-03-15T22:31:34.960263+0000 osd.1 2021-03-15T22:31:36.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[62156]: (osd.1) 143 : cluster [DBG] 7.b scrub ok 2021-03-15T22:31:36.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[62156]: cluster 2021-03-15T22:31:35.162592+0000 mgr.y (mgr. 2021-03-15T22:31:36.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[62156]: 14144) 895 : cluster [DBG] pgmap v1261: 148 pgs: 1 active+clean+snaptrim, 1 active+undersized+remapped, 3 active+remapped+backfill_toofull, 1 peering, 142 active+clean; 655 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 15 MiB/s rd, 2.9 MiB/s wr, 22 op/s; 4/1914 objects misplaced (0.209%); 25 MiB/s, 78 objects/s recovering 2021-03-15T22:31:36.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:31:36.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[62156]: :31:35.474925+0000 osd.3 (osd.3) 133 : cluster [DBG] 7.2 scrub ok 2021-03-15T22:31:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:31:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[62156]: T22:31:35.559603+0000 osd.7 (osd.7) 139 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:31:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:31:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[62156]: T22:31:36.245920+0000 mon.a (mon.0) 1976 : 2021-03-15T22:31:36.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:36 smithi076 conmon[62156]: cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2021-03-15T22:31:36.558 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:36.580 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:36.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:36.582 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:36.683 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:36.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:36.706 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:36.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: cluster 2021-03-15T22:31:34.960263+0000 osd.1 (osd.1) 143 : cluster [DBG] 7.b scrub ok 2021-03-15T22:31:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: cluster 2021-03-15T22:31:35.162592+0000 mgr.y (mgr.14144) 895 : cluster [DBG] pgmap v1261: 148 pgs: 1 active+clean+snaptrim, 1 active+undersized+remapped, 3 active+remapped+backfill_toofull, 1 peering, 142 active+clean; 655 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 15 MiB/s rd, 2.9 MiB/s wr, 22 op/s; 4/1914 objects misplaced (0.209%); 25 MiB/s, 78 objects/s recovering 2021-03-15T22:31:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:31:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: -15T22:31:35.474925+0000 osd.3 (osd.3) 133 : cluster 2021-03-15T22:31:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: [DBG] 7.2 scrub ok 2021-03-15T22:31:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:31:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: -03-15T22:31:35. 2021-03-15T22:31:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: 559603+0000 osd.7 (osd.7) 2021-03-15T22:31:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: 139 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:31:36.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:31:36.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: 03-15T22:31:36. 2021-03-15T22:31:36.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: 245920+0000 mon.a (mon.0) 1976 2021-03-15T22:31:36.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:36 smithi125 conmon[56739]: : cluster [DBG] osdmap e746: 8 total, 8 up, 7 in 2021-03-15T22:31:36.807 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:36.833 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:36.835 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:36.936 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:36.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:36.961 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:36.962 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:37.064 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:37.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:37.088 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:37.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:37.190 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:37.213 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:37.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:37.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:37.254 INFO:tasks.rados.rados.0.smithi076.stdout:2762: done (2 left) 2021-03-15T22:31:37.254 INFO:tasks.rados.rados.0.smithi076.stdout:2763: delete oid 45 current snap is 320 2021-03-15T22:31:37.255 INFO:tasks.rados.rados.0.smithi076.stdout:2760: expect (ObjNum 881 snap 292 seq_num 881) 2021-03-15T22:31:37.315 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:37.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:37.342 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:37.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:37.425 INFO:tasks.rados.rados.0.smithi076.stdout:2759: expect (ObjNum 750 snap 245 seq_num 750) 2021-03-15T22:31:37.442 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:37 smithi076 conmon[56514]: cluster 2021-03-15T22:31:36.472643+0000 2021-03-15T22:31:37.443 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:37 smithi076 conmon[56514]: osd.3 (osd.3) 134 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:31:37.443 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:37 smithi076 conmon[56514]: cluster 2021-03-15T22:31:37.247642+0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2021-03-15T22:31:37.444 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:37 smithi076 conmon[62156]: cluster 2021-03-15T22:31:36.472643+0000 osd.3 (osd. 2021-03-15T22:31:37.444 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:37 smithi076 conmon[62156]: 3) 134 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:31:37.445 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:37 smithi076 conmon[62156]: cluster 2021-03-15T22:31:37.247642+0000 mon.a (mon.0) 1977 : 2021-03-15T22:31:37.445 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:37 smithi076 conmon[62156]: cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2021-03-15T22:31:37.446 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:37.470 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:37.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:37.472 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:37.573 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:37.575 INFO:tasks.rados.rados.0.smithi076.stdout:2759: done (2 left) 2021-03-15T22:31:37.575 INFO:tasks.rados.rados.0.smithi076.stdout:2760: done (1 left) 2021-03-15T22:31:37.575 INFO:tasks.rados.rados.0.smithi076.stdout:2763: done (0 left) 2021-03-15T22:31:37.576 INFO:tasks.rados.rados.0.smithi076.stdout:2764: read oid 28 snap -1 2021-03-15T22:31:37.576 INFO:tasks.rados.rados.0.smithi076.stdout:2764: expect (ObjNum 620 snap 193 seq_num 620) 2021-03-15T22:31:37.576 INFO:tasks.rados.rados.0.smithi076.stdout:2765: write (excl) oid 23 current snap is 320 2021-03-15T22:31:37.576 INFO:tasks.rados.rados.0.smithi076.stdout:2765: seq_num 955 ranges {450685=406920,1567833=482852} 2021-03-15T22:31:37.578 INFO:tasks.rados.rados.0.smithi076.stdout:2765: writing smithi07694878-23 from 450685 to 857605 tid 1 2021-03-15T22:31:37.586 INFO:tasks.rados.rados.0.smithi076.stdout:2765: writing smithi07694878-23 from 1567833 to 2050685 tid 2 2021-03-15T22:31:37.586 INFO:tasks.rados.rados.0.smithi076.stdout:2766: snap_remove snap 286 2021-03-15T22:31:37.602 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:37.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:37.604 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:37.705 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:37.731 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:37.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:37.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:37.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:37 smithi125 conmon[56739]: cluster 2021-03-15T22:31:36.472643+0000 osd.3 (osd.3) 134 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:31:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:37 smithi125 conmon[56739]: cluster 2021-03-15T22:31:37.247642 2021-03-15T22:31:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:37 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1977 : cluster [DBG] osdmap e747: 8 total, 8 up, 7 in 2021-03-15T22:31:37.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:37.858 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:37.860 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:37.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:37.987 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:37.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:37.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:38.003 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:31:38.006 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:31:38.090 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:38.116 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:38.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:38.118 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:38.219 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:38.242 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:38.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:38.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:38.318 INFO:tasks.rados.rados.0.smithi076.stdout:2766: done (2 left) 2021-03-15T22:31:38.318 INFO:tasks.rados.rados.0.smithi076.stdout:2767: write oid 38 current snap is 320 2021-03-15T22:31:38.318 INFO:tasks.rados.rados.0.smithi076.stdout:2767: seq_num 956 ranges {467492=482169,1626315=441177} 2021-03-15T22:31:38.326 INFO:tasks.rados.rados.0.smithi076.stdout:2767: writing smithi07694878-38 from 467492 to 949661 tid 1 2021-03-15T22:31:38.326 INFO:tasks.rados.rados.0.smithi076.stdout:2767: writing smithi07694878-38 from 1626315 to 2067492 tid 2 2021-03-15T22:31:38.326 INFO:tasks.rados.rados.0.smithi076.stdout:2768: snap_create 2021-03-15T22:31:38.327 INFO:tasks.rados.rados.0.smithi076.stdout:2764: expect (ObjNum 620 snap 193 seq_num 620) 2021-03-15T22:31:38.344 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:38.367 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:38.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:38.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:38.464 INFO:tasks.rados.rados.0.smithi076.stdout:2765: finishing write tid 1 to smithi07694878-23 2021-03-15T22:31:38.464 INFO:tasks.rados.rados.0.smithi076.stdout:2765: finishing write tid 2 to smithi07694878-23 2021-03-15T22:31:38.465 INFO:tasks.rados.rados.0.smithi076.stdout:2765: finishing write tid 3 to smithi07694878-23 2021-03-15T22:31:38.465 INFO:tasks.rados.rados.0.smithi076.stdout:2765: finishing write tid 4 to smithi07694878-23 2021-03-15T22:31:38.465 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 1960 (ObjNum 955 snap 320 seq_num 955) dirty exists 2021-03-15T22:31:38.465 INFO:tasks.rados.rados.0.smithi076.stdout:2765: left oid 23 (ObjNum 955 snap 320 seq_num 955) 2021-03-15T22:31:38.466 INFO:tasks.rados.rados.0.smithi076.stdout:2767: finishing write tid 1 to smithi07694878-38 2021-03-15T22:31:38.466 INFO:tasks.rados.rados.0.smithi076.stdout:2767: finishing write tid 2 to smithi07694878-38 2021-03-15T22:31:38.466 INFO:tasks.rados.rados.0.smithi076.stdout:2767: finishing write tid 3 to smithi07694878-38 2021-03-15T22:31:38.466 INFO:tasks.rados.rados.0.smithi076.stdout:2767: finishing write tid 4 to smithi07694878-38 2021-03-15T22:31:38.466 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2086 (ObjNum 956 snap 320 seq_num 956) dirty exists 2021-03-15T22:31:38.467 INFO:tasks.rados.rados.0.smithi076.stdout:2767: left oid 38 (ObjNum 956 snap 320 seq_num 956) 2021-03-15T22:31:38.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:38.496 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:38.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:38.504 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:38 smithi076 conmon[56514]: cluster 2021-03-15T22:31:37.163609+0000 mgr.y (mgr.14144) 896 : cluster [DBG] pgmap v1264: 148 pgs: 1 active+clean+snaptrim, 3 active+remapped+backfill_toofull, 144 active+clean; 641 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 17 MiB/s rd, 2.6 MiB/s wr, 22 op/s; 4/1881 objects misplaced (0.213%) 2021-03-15T22:31:38.505 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:38 smithi076 conmon[56514]: cluster 2021-03-15T22:31:37.466438+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.1e deep-scrub ok 2021-03-15T22:31:38.505 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:38 smithi076 conmon[56514]: audit 2021-03-15T22:31:37.915991+0000 mgr.y (mgr.14144) 897 : audit [DBG] from='client.16269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:31:38.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:38 smithi076 conmon[62156]: cluster 2021-03-15T22:31:37.163609+0000 mgr.y (mgr.14144) 896 : cluster [DBG] pgmap v1264: 148 pgs: 1 active+clean+snaptrim, 3 active+remapped+backfill_toofull, 144 active+clean; 641 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 17 MiB/s rd, 2.6 MiB/s wr, 22 op/s; 4/1881 objects misplaced (0.213%) 2021-03-15T22:31:38.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:38 smithi076 conmon[62156]: cluster 2021-03-15T22:31:37.466438+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.1e deep-scrub ok 2021-03-15T22:31:38.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:38 smithi076 conmon[62156]: audit 2021-03-15T22:31:37.915991+0000 mgr.y (mgr.14144) 897 : audit [DBG] from='client.16269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:31:38.554 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:31:38.598 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:38.624 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:38.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:38.626 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:38.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:38.753 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:38.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:38.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:38 smithi125 conmon[56739]: cluster 2021-03-15T22:31:37.163609+0000 mgr.y (mgr.14144) 896 : cluster [DBG] pgmap v1264: 148 pgs: 1 active+clean+snaptrim, 3 active+remapped+backfill_toofull, 144 active+clean; 641 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 17 MiB/s rd, 2.6 MiB/s wr, 22 op/s; 4/1881 objects misplaced (0.213%) 2021-03-15T22:31:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:38 smithi125 conmon[56739]: cluster 2021-03-15T22:31:37.466438+0000 osd.4 (osd.4) 160 : cluster [DBG] 6.1e deep-scrub ok 2021-03-15T22:31:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:38 smithi125 conmon[56739]: audit 2021-03-15T22:31:37.915991+0000 mgr.y ( 2021-03-15T22:31:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:38 smithi125 conmon[56739]: mgr.14144) 897 : audit [DBG] from='client.16269 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:31:38.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:38.882 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:38.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:38.883 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:38.958 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:31:38.959 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:31:38.959 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:31:38.978 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:31:38.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:38.997 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:31:39.008 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:39.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:39.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:39.016 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:31:39.034 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:31:39.052 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:31:39.053 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:31:39.053 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:31:39.053 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:31:39.053 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:31:39.054 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:31:39.054 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:31:39.054 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:31:39.054 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:31:39.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:39.137 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:39.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:39.168 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1265,"stamp":"2021-03-15T22:31:37.245055+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":672377503,"num_objects":627,"num_object_clones":261,"num_object_copies":1881,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":627,"num_whiteouts":14,"num_read":7548,"num_read_kb":5370365,"num_write":4676,"num_write_kb":1450518,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3255,"num_bytes_recovered":1223532281,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":15629,"ondisk_log_size":15629,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1782480,"kb_used_data":1504460,"kb_used_omap":273,"kb_used_meta":275694,"kb_avail":654433584,"statfs":{"total":671965249536,"available":670139990016,"internally_reserved":0,"allocated":1540567040,"data_stored":2161110340,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":282311546},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1090245,"num_objects":-10,"num_object_clones":-10,"num_object_copies":-30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-10,"num_whiteouts":-3,"num_read":68,"num_read_kb":67963,"num_write":22,"num_write_kb":10589,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.009339"},"pg_stats":[{"pgid":"5.19","version":"317'7","reported_seq":"95","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.238427+0000","last_change":"2021-03-15T22:31:23.256995+0000","last_active":"2021-03-15T22:31:31.238427+0000","last_peered":"2021-03-15T22:31:31.238427+0000","last_clean":"2021-03-15T22:31:31.238427+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:31:31.238427+0000","last_undegraded":"2021-03-15T22:31:31.238427+0000","last_fullsized":"2021-03-15T22:31:31.238427+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:30:07.252960+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.252960+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"747","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236576+0000","last_change":"2021-03-15T22:30:41.431150+0000","last_active":"2021-03-15T22:31:35.236576+0000","last_peered":"2021-03-15T22:31:35.236576+0000","last_clean":"2021-03-15T22:31:35.236576+0000","last_became_active":"2021-03-15T22:29:51.225252+0000","last_became_peered":"2021-03-15T22:29:51.225252+0000","last_unstale":"2021-03-15T22:31:35.236576+0000","last_undegraded":"2021-03-15T22:31:35.236576+0000","last_fullsized":"2021-03-15T22:31:35.236576+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.431052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"244","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.245147+0000","last_change":"2021-03-15T22:30:32.251958+0000","last_active":"2021-03-15T22:31:34.245147+0000","last_peered":"2021-03-15T22:31:34.245147+0000","last_clean":"2021-03-15T22:31:34.245147+0000","last_became_active":"2021-03-15T22:30:32.251610+0000","last_became_peered":"2021-03-15T22:30:32.251610+0000","last_unstale":"2021-03-15T22:31:34.245147+0000","last_undegraded":"2021-03-15T22:31:34.245147+0000","last_fullsized":"2021-03-15T22:31:34.245147+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:30:03.560125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.560125+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"554'21","reported_seq":"793","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236413+0000","last_change":"2021-03-15T22:31:29.175810+0000","last_active":"2021-03-15T22:31:35.236413+0000","last_peered":"2021-03-15T22:31:35.236413+0000","last_clean":"2021-03-15T22:31:35.236413+0000","last_became_active":"2021-03-15T22:31:28.287770+0000","last_became_peered":"2021-03-15T22:31:28.287770+0000","last_unstale":"2021-03-15T22:31:35.236413+0000","last_undegraded":"2021-03-15T22:31:35.236413+0000","last_fullsized":"2021-03-15T22:31:35.236413+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:30:01.066454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:01.066454+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"34","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236223+0000","last_change":"2021-03-15T22:31:28.274821+0000","last_active":"2021-03-15T22:31:35.236223+0000","last_peered":"2021-03-15T22:31:35.236223+0000","last_clean":"2021-03-15T22:31:35.236223+0000","last_became_active":"2021-03-15T22:31:28.274481+0000","last_became_peered":"2021-03-15T22:31:28.274481+0000","last_unstale":"2021-03-15T22:31:35.236223+0000","last_undegraded":"2021-03-15T22:31:35.236223+0000","last_fullsized":"2021-03-15T22:31:35.236223+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:11.029289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.029289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"180","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.240623+0000","last_change":"2021-03-15T22:31:28.285592+0000","last_active":"2021-03-15T22:31:35.240623+0000","last_peered":"2021-03-15T22:31:35.240623+0000","last_clean":"2021-03-15T22:31:35.240623+0000","last_became_active":"2021-03-15T22:31:28.285281+0000","last_became_peered":"2021-03-15T22:31:28.285281+0000","last_unstale":"2021-03-15T22:31:35.240623+0000","last_undegraded":"2021-03-15T22:31:35.240623+0000","last_fullsized":"2021-03-15T22:31:35.240623+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:05.262019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.262019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"462","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.239621+0000","last_change":"2021-03-15T22:31:28.290403+0000","last_active":"2021-03-15T22:31:35.239621+0000","last_peered":"2021-03-15T22:31:35.239621+0000","last_clean":"2021-03-15T22:31:35.239621+0000","last_became_active":"2021-03-15T22:31:28.289753+0000","last_became_peered":"2021-03-15T22:31:28.289753+0000","last_unstale":"2021-03-15T22:31:35.239621+0000","last_undegraded":"2021-03-15T22:31:35.239621+0000","last_fullsized":"2021-03-15T22:31:35.239621+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:09.184802+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:29:09.184802+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"556'32","reported_seq":"463","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.239661+0000","last_change":"2021-03-15T22:30:32.296239+0000","last_active":"2021-03-15T22:31:35.239661+0000","last_peered":"2021-03-15T22:31:35.239661+0000","last_clean":"2021-03-15T22:31:35.239661+0000","last_became_active":"2021-03-15T22:30:32.258079+0000","last_became_peered":"2021-03-15T22:30:32.258079+0000","last_unstale":"2021-03-15T22:31:35.239661+0000","last_undegraded":"2021-03-15T22:31:35.239661+0000","last_fullsized":"2021-03-15T22:31:35.239661+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'32","last_scrub_stamp":"2021-03-15T22:29:32.512485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:32.512485+0000","log_size":32,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"575","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.236471+0000","last_change":"2021-03-15T22:31:20.247091+0000","last_active":"2021-03-15T22:31:32.236471+0000","last_peered":"2021-03-15T22:31:32.236471+0000","last_clean":"2021-03-15T22:31:32.236471+0000","last_became_active":"2021-03-15T22:31:20.246770+0000","last_became_peered":"2021-03-15T22:31:20.246770+0000","last_unstale":"2021-03-15T22:31:32.236471+0000","last_undegraded":"2021-03-15T22:31:32.236471+0000","last_fullsized":"2021-03-15T22:31:32.236471+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:59.529159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:59.529159+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"207","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.242721+0000","last_change":"2021-03-15T22:29:18.753212+0000","last_active":"2021-03-15T22:31:34.242721+0000","last_peered":"2021-03-15T22:31:34.242721+0000","last_clean":"2021-03-15T22:31:34.242721+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:31:34.242721+0000","last_undegraded":"2021-03-15T22:31:34.242721+0000","last_fullsized":"2021-03-15T22:31:34.242721+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:18.753152+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:29:18.753152+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"125","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.237059+0000","last_change":"2021-03-15T22:31:28.288569+0000","last_active":"2021-03-15T22:31:32.237059+0000","last_peered":"2021-03-15T22:31:32.237059+0000","last_clean":"2021-03-15T22:31:32.237059+0000","last_became_active":"2021-03-15T22:31:28.288226+0000","last_became_peered":"2021-03-15T22:31:28.288226+0000","last_unstale":"2021-03-15T22:31:32.237059+0000","last_undegraded":"2021-03-15T22:31:32.237059+0000","last_fullsized":"2021-03-15T22:31:32.237059+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.146408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.146408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"890","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236807+0000","last_change":"2021-03-15T22:30:28.328526+0000","last_active":"2021-03-15T22:31:35.236807+0000","last_peered":"2021-03-15T22:31:35.236807+0000","last_clean":"2021-03-15T22:31:35.236807+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:31:35.236807+0000","last_undegraded":"2021-03-15T22:31:35.236807+0000","last_fullsized":"2021-03-15T22:31:35.236807+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:30:28.328458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.328458+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"23","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.239984+0000","last_change":"2021-03-15T22:31:20.250057+0000","last_active":"2021-03-15T22:31:33.239984+0000","last_peered":"2021-03-15T22:31:33.239984+0000","last_clean":"2021-03-15T22:31:33.239984+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:31:33.239984+0000","last_undegraded":"2021-03-15T22:31:33.239984+0000","last_fullsized":"2021-03-15T22:31:33.239984+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:07.063227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.063227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"361","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.236213+0000","last_change":"2021-03-15T22:30:40.173603+0000","last_active":"2021-03-15T22:31:32.236213+0000","last_peered":"2021-03-15T22:31:32.236213+0000","last_clean":"2021-03-15T22:31:32.236213+0000","last_became_active":"2021-03-15T22:29:52.223704+0000","last_became_peered":"2021-03-15T22:29:52.223704+0000","last_unstale":"2021-03-15T22:31:32.236213+0000","last_undegraded":"2021-03-15T22:31:32.236213+0000","last_fullsized":"2021-03-15T22:31:32.236213+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:40.173519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:30:40.173519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"116","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.236756+0000","last_change":"2021-03-15T22:30:02.813811+0000","last_active":"2021-03-15T22:31:32.236756+0000","last_peered":"2021-03-15T22:31:32.236756+0000","last_clean":"2021-03-15T22:31:32.236756+0000","last_became_active":"2021-03-15T22:29:51.234868+0000","last_became_peered":"2021-03-15T22:29:51.234868+0000","last_unstale":"2021-03-15T22:31:32.236756+0000","last_undegraded":"2021-03-15T22:31:32.236756+0000","last_fullsized":"2021-03-15T22:31:32.236756+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:02.813745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.813745+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"828","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237625+0000","last_change":"2021-03-15T22:29:56.950563+0000","last_active":"2021-03-15T22:31:35.237625+0000","last_peered":"2021-03-15T22:31:35.237625+0000","last_clean":"2021-03-15T22:31:35.237625+0000","last_became_active":"2021-03-15T22:29:11.197059+0000","last_became_peered":"2021-03-15T22:29:11.197059+0000","last_unstale":"2021-03-15T22:31:35.237625+0000","last_undegraded":"2021-03-15T22:31:35.237625+0000","last_fullsized":"2021-03-15T22:31:35.237625+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:29:56.950494+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.950494+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"678","reported_epoch":"743","state":"active+remapped+backfill_toofull","last_fresh":"2021-03-15T22:31:33.237886+0000","last_change":"2021-03-15T22:31:21.994721+0000","last_active":"2021-03-15T22:31:33.237886+0000","last_peered":"2021-03-15T22:31:33.237886+0000","last_clean":"2021-03-15T22:31:18.261662+0000","last_became_active":"2021-03-15T22:31:21.993742+0000","last_became_peered":"2021-03-15T22:31:21.993742+0000","last_unstale":"2021-03-15T22:31:33.237886+0000","last_undegraded":"2021-03-15T22:31:33.237886+0000","last_fullsized":"2021-03-15T22:31:33.237886+0000","mapping_epoch":731,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:31:05.998563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.998563+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":3,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[2,5,7],"avail_no_missing":["2","5","7"],"object_location_counts":[{"shards":"2,5,7","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"749","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237064+0000","last_change":"2021-03-15T22:31:28.282074+0000","last_active":"2021-03-15T22:31:35.237064+0000","last_peered":"2021-03-15T22:31:35.237064+0000","last_clean":"2021-03-15T22:31:35.237064+0000","last_became_active":"2021-03-15T22:31:28.281143+0000","last_became_peered":"2021-03-15T22:31:28.281143+0000","last_unstale":"2021-03-15T22:31:35.237064+0000","last_undegraded":"2021-03-15T22:31:35.237064+0000","last_fullsized":"2021-03-15T22:31:35.237064+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:41.941510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.941510+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"211","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.243418+0000","last_change":"2021-03-15T22:30:38.781792+0000","last_active":"2021-03-15T22:31:34.243418+0000","last_peered":"2021-03-15T22:31:34.243418+0000","last_clean":"2021-03-15T22:31:34.243418+0000","last_became_active":"2021-03-15T22:28:12.519619+0000","last_became_peered":"2021-03-15T22:28:12.519619+0000","last_unstale":"2021-03-15T22:31:34.243418+0000","last_undegraded":"2021-03-15T22:31:34.243418+0000","last_fullsized":"2021-03-15T22:31:34.243418+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:30:38.781724+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.781724+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"611","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.241012+0000","last_change":"2021-03-15T22:30:32.299886+0000","last_active":"2021-03-15T22:31:33.241012+0000","last_peered":"2021-03-15T22:31:33.241012+0000","last_clean":"2021-03-15T22:31:33.241012+0000","last_became_active":"2021-03-15T22:30:32.250733+0000","last_became_peered":"2021-03-15T22:30:32.250733+0000","last_unstale":"2021-03-15T22:31:33.241012+0000","last_undegraded":"2021-03-15T22:31:33.241012+0000","last_fullsized":"2021-03-15T22:31:33.241012+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:29:57.169300+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:57.169300+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"769","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236888+0000","last_change":"2021-03-15T22:31:20.241361+0000","last_active":"2021-03-15T22:31:35.236888+0000","last_peered":"2021-03-15T22:31:35.236888+0000","last_clean":"2021-03-15T22:31:35.236888+0000","last_became_active":"2021-03-15T22:31:20.240898+0000","last_became_peered":"2021-03-15T22:31:20.240898+0000","last_unstale":"2021-03-15T22:31:35.236888+0000","last_undegraded":"2021-03-15T22:31:35.236888+0000","last_fullsized":"2021-03-15T22:31:35.236888+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:41.190058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.190058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"566","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.240576+0000","last_change":"2021-03-15T22:31:28.286491+0000","last_active":"2021-03-15T22:31:35.240576+0000","last_peered":"2021-03-15T22:31:35.240576+0000","last_clean":"2021-03-15T22:31:35.240576+0000","last_became_active":"2021-03-15T22:31:28.286067+0000","last_became_peered":"2021-03-15T22:31:28.286067+0000","last_unstale":"2021-03-15T22:31:35.240576+0000","last_undegraded":"2021-03-15T22:31:35.240576+0000","last_fullsized":"2021-03-15T22:31:35.240576+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:38.513993+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:45.403769+0000","last_clean_scrub_stamp":"2021-03-15T22:29:38.513993+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"735","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236916+0000","last_change":"2021-03-15T22:30:39.443682+0000","last_active":"2021-03-15T22:31:35.236916+0000","last_peered":"2021-03-15T22:31:35.236916+0000","last_clean":"2021-03-15T22:31:35.236916+0000","last_became_active":"2021-03-15T22:30:31.243315+0000","last_became_peered":"2021-03-15T22:30:31.243315+0000","last_unstale":"2021-03-15T22:31:35.236916+0000","last_undegraded":"2021-03-15T22:31:35.236916+0000","last_fullsized":"2021-03-15T22:31:35.236916+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.443613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"191","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.239155+0000","last_change":"2021-03-15T22:29:54.564185+0000","last_active":"2021-03-15T22:31:35.239155+0000","last_peered":"2021-03-15T22:31:35.239155+0000","last_clean":"2021-03-15T22:31:35.239155+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:31:35.239155+0000","last_undegraded":"2021-03-15T22:31:35.239155+0000","last_fullsized":"2021-03-15T22:31:35.239155+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:29:54.564115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:54.564115+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"741'15","reported_seq":"279","reported_epoch":"745","state":"active+remapped+backfill_toofull","last_fresh":"2021-03-15T22:31:35.236993+0000","last_change":"2021-03-15T22:31:29.317296+0000","last_active":"2021-03-15T22:31:35.236993+0000","last_peered":"2021-03-15T22:31:35.236993+0000","last_clean":"2021-03-15T22:29:49.430566+0000","last_became_active":"2021-03-15T22:31:29.291043+0000","last_became_peered":"2021-03-15T22:31:29.291043+0000","last_unstale":"2021-03-15T22:31:35.236993+0000","last_undegraded":"2021-03-15T22:31:35.236993+0000","last_fullsized":"2021-03-15T22:31:35.236993+0000","mapping_epoch":738,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"618'7","last_scrub_stamp":"2021-03-15T22:29:40.964753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.964753+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,6],"avail_no_missing":["4","6","7"],"object_location_counts":[{"shards":"4,6,7","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"751","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.240418+0000","last_change":"2021-03-15T22:30:32.243983+0000","last_active":"2021-03-15T22:31:33.240418+0000","last_peered":"2021-03-15T22:31:33.240418+0000","last_clean":"2021-03-15T22:31:33.240418+0000","last_became_active":"2021-03-15T22:30:32.243708+0000","last_became_peered":"2021-03-15T22:30:32.243708+0000","last_unstale":"2021-03-15T22:31:33.240418+0000","last_undegraded":"2021-03-15T22:31:33.240418+0000","last_fullsized":"2021-03-15T22:31:33.240418+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:01.208847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:01.208847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"765","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.238538+0000","last_change":"2021-03-15T22:31:05.123509+0000","last_active":"2021-03-15T22:31:31.238538+0000","last_peered":"2021-03-15T22:31:31.238538+0000","last_clean":"2021-03-15T22:31:31.238538+0000","last_became_active":"2021-03-15T22:28:59.264559+0000","last_became_peered":"2021-03-15T22:28:59.264559+0000","last_unstale":"2021-03-15T22:31:31.238538+0000","last_undegraded":"2021-03-15T22:31:31.238538+0000","last_fullsized":"2021-03-15T22:31:31.238538+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:05.123425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.123425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"312","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.244369+0000","last_change":"2021-03-15T22:30:41.759292+0000","last_active":"2021-03-15T22:31:34.244369+0000","last_peered":"2021-03-15T22:31:34.244369+0000","last_clean":"2021-03-15T22:31:34.244369+0000","last_became_active":"2021-03-15T22:28:59.277385+0000","last_became_peered":"2021-03-15T22:28:59.277385+0000","last_unstale":"2021-03-15T22:31:34.244369+0000","last_undegraded":"2021-03-15T22:31:34.244369+0000","last_fullsized":"2021-03-15T22:31:34.244369+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:30:41.759224+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.759224+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"748","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.244026+0000","last_change":"2021-03-15T22:31:23.259094+0000","last_active":"2021-03-15T22:31:34.244026+0000","last_peered":"2021-03-15T22:31:34.244026+0000","last_clean":"2021-03-15T22:31:34.244026+0000","last_became_active":"2021-03-15T22:31:23.258435+0000","last_became_peered":"2021-03-15T22:31:23.258435+0000","last_unstale":"2021-03-15T22:31:34.244026+0000","last_undegraded":"2021-03-15T22:31:34.244026+0000","last_fullsized":"2021-03-15T22:31:34.244026+0000","mapping_epoch":733,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"618'11","last_scrub_stamp":"2021-03-15T22:29:36.937870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:36.937870+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"738","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.239961+0000","last_change":"2021-03-15T22:29:52.230266+0000","last_active":"2021-03-15T22:31:31.239961+0000","last_peered":"2021-03-15T22:31:31.239961+0000","last_clean":"2021-03-15T22:31:31.239961+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:31:31.239961+0000","last_undegraded":"2021-03-15T22:31:31.239961+0000","last_fullsized":"2021-03-15T22:31:31.239961+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:34.234643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:34.234643+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"350","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.241644+0000","last_change":"2021-03-15T22:31:10.332382+0000","last_active":"2021-03-15T22:31:35.241644+0000","last_peered":"2021-03-15T22:31:35.241644+0000","last_clean":"2021-03-15T22:31:35.241644+0000","last_became_active":"2021-03-15T22:29:00.220073+0000","last_became_peered":"2021-03-15T22:29:00.220073+0000","last_unstale":"2021-03-15T22:31:35.241644+0000","last_undegraded":"2021-03-15T22:31:35.241644+0000","last_fullsized":"2021-03-15T22:31:35.241644+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:10.332319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.332319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"735","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.237373+0000","last_change":"2021-03-15T22:31:28.320040+0000","last_active":"2021-03-15T22:31:32.237373+0000","last_peered":"2021-03-15T22:31:32.237373+0000","last_clean":"2021-03-15T22:31:32.237373+0000","last_became_active":"2021-03-15T22:31:28.287309+0000","last_became_peered":"2021-03-15T22:31:28.287309+0000","last_unstale":"2021-03-15T22:31:32.237373+0000","last_undegraded":"2021-03-15T22:31:32.237373+0000","last_fullsized":"2021-03-15T22:31:32.237373+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:31:03.281344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:03.281344+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"720","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.239364+0000","last_change":"2021-03-15T22:30:59.480779+0000","last_active":"2021-03-15T22:31:33.239364+0000","last_peered":"2021-03-15T22:31:33.239364+0000","last_clean":"2021-03-15T22:31:33.239364+0000","last_became_active":"2021-03-15T22:30:31.230649+0000","last_became_peered":"2021-03-15T22:30:31.230649+0000","last_unstale":"2021-03-15T22:31:33.239364+0000","last_undegraded":"2021-03-15T22:31:33.239364+0000","last_fullsized":"2021-03-15T22:31:33.239364+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:59.480717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:30:59.480717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"20","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236392+0000","last_change":"2021-03-15T22:31:28.278722+0000","last_active":"2021-03-15T22:31:35.236392+0000","last_peered":"2021-03-15T22:31:35.236392+0000","last_clean":"2021-03-15T22:31:35.236392+0000","last_became_active":"2021-03-15T22:31:28.278193+0000","last_became_peered":"2021-03-15T22:31:28.278193+0000","last_unstale":"2021-03-15T22:31:35.236392+0000","last_undegraded":"2021-03-15T22:31:35.236392+0000","last_fullsized":"2021-03-15T22:31:35.236392+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:06.296662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:06.296662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"251","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.245076+0000","last_change":"2021-03-15T22:31:33.136177+0000","last_active":"2021-03-15T22:31:34.245076+0000","last_peered":"2021-03-15T22:31:34.245076+0000","last_clean":"2021-03-15T22:31:34.245076+0000","last_became_active":"2021-03-15T22:29:51.226677+0000","last_became_peered":"2021-03-15T22:29:51.226677+0000","last_unstale":"2021-03-15T22:31:34.245076+0000","last_undegraded":"2021-03-15T22:31:34.245076+0000","last_fullsized":"2021-03-15T22:31:34.245076+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:33.136112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:31:33.136112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"224","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.241400+0000","last_change":"2021-03-15T22:31:28.306827+0000","last_active":"2021-03-15T22:31:35.241400+0000","last_peered":"2021-03-15T22:31:35.241400+0000","last_clean":"2021-03-15T22:31:35.241400+0000","last_became_active":"2021-03-15T22:31:28.284715+0000","last_became_peered":"2021-03-15T22:31:28.284715+0000","last_unstale":"2021-03-15T22:31:35.241400+0000","last_undegraded":"2021-03-15T22:31:35.241400+0000","last_fullsized":"2021-03-15T22:31:35.241400+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:30:07.486190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.486190+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":7512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"359","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.241141+0000","last_change":"2021-03-15T22:31:28.288399+0000","last_active":"2021-03-15T22:31:35.241141+0000","last_peered":"2021-03-15T22:31:35.241141+0000","last_clean":"2021-03-15T22:31:35.241141+0000","last_became_active":"2021-03-15T22:31:28.288084+0000","last_became_peered":"2021-03-15T22:31:28.288084+0000","last_unstale":"2021-03-15T22:31:35.241141+0000","last_undegraded":"2021-03-15T22:31:35.241141+0000","last_fullsized":"2021-03-15T22:31:35.241141+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.317817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"711","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.240995+0000","last_change":"2021-03-15T22:31:28.282286+0000","last_active":"2021-03-15T22:31:33.240995+0000","last_peered":"2021-03-15T22:31:33.240995+0000","last_clean":"2021-03-15T22:31:33.240995+0000","last_became_active":"2021-03-15T22:31:28.281921+0000","last_became_peered":"2021-03-15T22:31:28.281921+0000","last_unstale":"2021-03-15T22:31:33.240995+0000","last_undegraded":"2021-03-15T22:31:33.240995+0000","last_fullsized":"2021-03-15T22:31:33.240995+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:09.088684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:28:09.088684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"116","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.237151+0000","last_change":"2021-03-15T22:30:42.233176+0000","last_active":"2021-03-15T22:31:32.237151+0000","last_peered":"2021-03-15T22:31:32.237151+0000","last_clean":"2021-03-15T22:31:32.237151+0000","last_became_active":"2021-03-15T22:29:51.220261+0000","last_became_peered":"2021-03-15T22:29:51.220261+0000","last_unstale":"2021-03-15T22:31:32.237151+0000","last_undegraded":"2021-03-15T22:31:32.237151+0000","last_fullsized":"2021-03-15T22:31:32.237151+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.233098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"559","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.240211+0000","last_change":"2021-03-15T22:31:28.302192+0000","last_active":"2021-03-15T22:31:35.240211+0000","last_peered":"2021-03-15T22:31:35.240211+0000","last_clean":"2021-03-15T22:31:35.240211+0000","last_became_active":"2021-03-15T22:31:28.283474+0000","last_became_peered":"2021-03-15T22:31:28.283474+0000","last_unstale":"2021-03-15T22:31:35.240211+0000","last_undegraded":"2021-03-15T22:31:35.240211+0000","last_fullsized":"2021-03-15T22:31:35.240211+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:31:07.330960+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:31:07.330960+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"226","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.242573+0000","last_change":"2021-03-15T22:29:51.212768+0000","last_active":"2021-03-15T22:31:34.242573+0000","last_peered":"2021-03-15T22:31:34.242573+0000","last_clean":"2021-03-15T22:31:34.242573+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:31:34.242573+0000","last_undegraded":"2021-03-15T22:31:34.242573+0000","last_fullsized":"2021-03-15T22:31:34.242573+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:19.721291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:29:19.721291+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"535","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.240802+0000","last_change":"2021-03-15T22:29:56.525891+0000","last_active":"2021-03-15T22:31:35.240802+0000","last_peered":"2021-03-15T22:31:35.240802+0000","last_clean":"2021-03-15T22:31:35.240802+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:31:35.240802+0000","last_undegraded":"2021-03-15T22:31:35.240802+0000","last_fullsized":"2021-03-15T22:31:35.240802+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:56.525822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.525822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"742","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.240877+0000","last_change":"2021-03-15T22:29:09.154666+0000","last_active":"2021-03-15T22:31:33.240877+0000","last_peered":"2021-03-15T22:31:33.240877+0000","last_clean":"2021-03-15T22:31:33.240877+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:31:33.240877+0000","last_undegraded":"2021-03-15T22:31:33.240877+0000","last_fullsized":"2021-03-15T22:31:33.240877+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:09.154606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:29:09.154606+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"466","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.238779+0000","last_change":"2021-03-15T22:31:29.191723+0000","last_active":"2021-03-15T22:31:31.238779+0000","last_peered":"2021-03-15T22:31:31.238779+0000","last_clean":"2021-03-15T22:31:31.238779+0000","last_became_active":"2021-03-15T22:31:28.276464+0000","last_became_peered":"2021-03-15T22:31:28.276464+0000","last_unstale":"2021-03-15T22:31:31.238779+0000","last_undegraded":"2021-03-15T22:31:31.238779+0000","last_fullsized":"2021-03-15T22:31:31.238779+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:30:38.178596+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.178596+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"367","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.239565+0000","last_change":"2021-03-15T22:29:51.219273+0000","last_active":"2021-03-15T22:31:35.239565+0000","last_peered":"2021-03-15T22:31:35.239565+0000","last_clean":"2021-03-15T22:31:35.239565+0000","last_became_active":"2021-03-15T22:29:51.218982+0000","last_became_peered":"2021-03-15T22:29:51.218982+0000","last_unstale":"2021-03-15T22:31:35.239565+0000","last_undegraded":"2021-03-15T22:31:35.239565+0000","last_fullsized":"2021-03-15T22:31:35.239565+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:35.496457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:29:35.496457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"782","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.238672+0000","last_change":"2021-03-15T22:30:44.116997+0000","last_active":"2021-03-15T22:31:31.238672+0000","last_peered":"2021-03-15T22:31:31.238672+0000","last_clean":"2021-03-15T22:31:31.238672+0000","last_became_active":"2021-03-15T22:29:51.222254+0000","last_became_peered":"2021-03-15T22:29:51.222254+0000","last_unstale":"2021-03-15T22:31:31.238672+0000","last_undegraded":"2021-03-15T22:31:31.238672+0000","last_fullsized":"2021-03-15T22:31:31.238672+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:30:44.116934+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"18","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236658+0000","last_change":"2021-03-15T22:31:28.279172+0000","last_active":"2021-03-15T22:31:35.236658+0000","last_peered":"2021-03-15T22:31:35.236658+0000","last_clean":"2021-03-15T22:31:35.236658+0000","last_became_active":"2021-03-15T22:31:28.278787+0000","last_became_peered":"2021-03-15T22:31:28.278787+0000","last_unstale":"2021-03-15T22:31:35.236658+0000","last_undegraded":"2021-03-15T22:31:35.236658+0000","last_fullsized":"2021-03-15T22:31:35.236658+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.992352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.992352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"793","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237453+0000","last_change":"2021-03-15T22:31:28.320477+0000","last_active":"2021-03-15T22:31:35.237453+0000","last_peered":"2021-03-15T22:31:35.237453+0000","last_clean":"2021-03-15T22:31:35.237453+0000","last_became_active":"2021-03-15T22:31:28.282759+0000","last_became_peered":"2021-03-15T22:31:28.282759+0000","last_unstale":"2021-03-15T22:31:35.237453+0000","last_undegraded":"2021-03-15T22:31:35.237453+0000","last_fullsized":"2021-03-15T22:31:35.237453+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:30:02.064721+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.064721+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"225","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.244556+0000","last_change":"2021-03-15T22:30:42.771039+0000","last_active":"2021-03-15T22:31:34.244556+0000","last_peered":"2021-03-15T22:31:34.244556+0000","last_clean":"2021-03-15T22:31:34.244556+0000","last_became_active":"2021-03-15T22:28:59.276812+0000","last_became_peered":"2021-03-15T22:28:59.276812+0000","last_unstale":"2021-03-15T22:31:34.244556+0000","last_undegraded":"2021-03-15T22:31:34.244556+0000","last_fullsized":"2021-03-15T22:31:34.244556+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:42.770976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.770976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"642'8","reported_seq":"141","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.237070+0000","last_change":"2021-03-15T22:30:06.920670+0000","last_active":"2021-03-15T22:31:32.237070+0000","last_peered":"2021-03-15T22:31:32.237070+0000","last_clean":"2021-03-15T22:31:32.237070+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:31:32.237070+0000","last_undegraded":"2021-03-15T22:31:32.237070+0000","last_fullsized":"2021-03-15T22:31:32.237070+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'8","last_scrub_stamp":"2021-03-15T22:30:06.920606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:06.920606+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"721","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.240058+0000","last_change":"2021-03-15T22:30:35.423621+0000","last_active":"2021-03-15T22:31:33.240058+0000","last_peered":"2021-03-15T22:31:33.240058+0000","last_clean":"2021-03-15T22:31:33.240058+0000","last_became_active":"2021-03-15T22:30:31.230560+0000","last_became_peered":"2021-03-15T22:30:31.230560+0000","last_unstale":"2021-03-15T22:31:33.240058+0000","last_undegraded":"2021-03-15T22:31:33.240058+0000","last_fullsized":"2021-03-15T22:31:33.240058+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:35.423559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:35.423559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"556'24","reported_seq":"534","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.240314+0000","last_change":"2021-03-15T22:30:02.548134+0000","last_active":"2021-03-15T22:31:35.240314+0000","last_peered":"2021-03-15T22:31:35.240314+0000","last_clean":"2021-03-15T22:31:35.240314+0000","last_became_active":"2021-03-15T22:29:00.226652+0000","last_became_peered":"2021-03-15T22:29:00.226652+0000","last_unstale":"2021-03-15T22:31:35.240314+0000","last_undegraded":"2021-03-15T22:31:35.240314+0000","last_fullsized":"2021-03-15T22:31:35.240314+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'24","last_scrub_stamp":"2021-03-15T22:30:02.548069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.548069+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"755","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237841+0000","last_change":"2021-03-15T22:30:32.248125+0000","last_active":"2021-03-15T22:31:35.237841+0000","last_peered":"2021-03-15T22:31:35.237841+0000","last_clean":"2021-03-15T22:31:35.237841+0000","last_became_active":"2021-03-15T22:30:32.247782+0000","last_became_peered":"2021-03-15T22:30:32.247782+0000","last_unstale":"2021-03-15T22:31:35.237841+0000","last_undegraded":"2021-03-15T22:31:35.237841+0000","last_fullsized":"2021-03-15T22:31:35.237841+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:34.939142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:34.939142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"683'14","reported_seq":"416","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.239673+0000","last_change":"2021-03-15T22:30:32.252825+0000","last_active":"2021-03-15T22:31:35.239673+0000","last_peered":"2021-03-15T22:31:35.239673+0000","last_clean":"2021-03-15T22:31:35.239673+0000","last_became_active":"2021-03-15T22:30:32.252383+0000","last_became_peered":"2021-03-15T22:30:32.252383+0000","last_unstale":"2021-03-15T22:31:35.239673+0000","last_undegraded":"2021-03-15T22:31:35.239673+0000","last_fullsized":"2021-03-15T22:31:35.239673+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"572'10","last_scrub_stamp":"2021-03-15T22:28:47.140118+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:28:47.140118+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"225","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.242945+0000","last_change":"2021-03-15T22:29:17.782276+0000","last_active":"2021-03-15T22:31:34.242945+0000","last_peered":"2021-03-15T22:31:34.242945+0000","last_clean":"2021-03-15T22:31:34.242945+0000","last_became_active":"2021-03-15T22:28:59.265822+0000","last_became_peered":"2021-03-15T22:28:59.265822+0000","last_unstale":"2021-03-15T22:31:34.242945+0000","last_undegraded":"2021-03-15T22:31:34.242945+0000","last_fullsized":"2021-03-15T22:31:34.242945+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:17.782209+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:29:17.782209+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"134","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.236936+0000","last_change":"2021-03-15T22:30:36.106413+0000","last_active":"2021-03-15T22:31:32.236936+0000","last_peered":"2021-03-15T22:31:32.236936+0000","last_clean":"2021-03-15T22:31:32.236936+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:31:32.236936+0000","last_undegraded":"2021-03-15T22:31:32.236936+0000","last_fullsized":"2021-03-15T22:31:32.236936+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:30:36.106285+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.106285+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"25","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.240392+0000","last_change":"2021-03-15T22:31:20.257902+0000","last_active":"2021-03-15T22:31:35.240392+0000","last_peered":"2021-03-15T22:31:35.240392+0000","last_clean":"2021-03-15T22:31:35.240392+0000","last_became_active":"2021-03-15T22:31:20.256557+0000","last_became_peered":"2021-03-15T22:31:20.256557+0000","last_unstale":"2021-03-15T22:31:35.240392+0000","last_undegraded":"2021-03-15T22:31:35.240392+0000","last_fullsized":"2021-03-15T22:31:35.240392+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:11.436723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.436723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"514","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.240818+0000","last_change":"2021-03-15T22:30:46.154245+0000","last_active":"2021-03-15T22:31:33.240818+0000","last_peered":"2021-03-15T22:31:33.240818+0000","last_clean":"2021-03-15T22:31:33.240818+0000","last_became_active":"2021-03-15T22:30:31.254532+0000","last_became_peered":"2021-03-15T22:30:31.254532+0000","last_unstale":"2021-03-15T22:31:33.240818+0000","last_undegraded":"2021-03-15T22:31:33.240818+0000","last_fullsized":"2021-03-15T22:31:33.240818+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.003120+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.003120+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"720","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.239616+0000","last_change":"2021-03-15T22:31:28.277106+0000","last_active":"2021-03-15T22:31:33.239616+0000","last_peered":"2021-03-15T22:31:33.239616+0000","last_clean":"2021-03-15T22:31:33.239616+0000","last_became_active":"2021-03-15T22:31:28.276738+0000","last_became_peered":"2021-03-15T22:31:28.276738+0000","last_unstale":"2021-03-15T22:31:33.239616+0000","last_undegraded":"2021-03-15T22:31:33.239616+0000","last_fullsized":"2021-03-15T22:31:33.239616+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:33.386700+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:33.386700+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"270","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.244102+0000","last_change":"2021-03-15T22:31:04.009824+0000","last_active":"2021-03-15T22:31:34.244102+0000","last_peered":"2021-03-15T22:31:34.244102+0000","last_clean":"2021-03-15T22:31:34.244102+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:31:34.244102+0000","last_undegraded":"2021-03-15T22:31:34.244102+0000","last_fullsized":"2021-03-15T22:31:34.244102+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:31:04.009748+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.009748+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"536","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.237235+0000","last_change":"2021-03-15T22:31:20.261172+0000","last_active":"2021-03-15T22:31:32.237235+0000","last_peered":"2021-03-15T22:31:32.237235+0000","last_clean":"2021-03-15T22:31:32.237235+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:31:32.237235+0000","last_undegraded":"2021-03-15T22:31:32.237235+0000","last_fullsized":"2021-03-15T22:31:32.237235+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:55.737371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:55.737371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"714","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.240837+0000","last_change":"2021-03-15T22:31:28.286675+0000","last_active":"2021-03-15T22:31:33.240837+0000","last_peered":"2021-03-15T22:31:33.240837+0000","last_clean":"2021-03-15T22:31:33.240837+0000","last_became_active":"2021-03-15T22:31:28.286252+0000","last_became_peered":"2021-03-15T22:31:28.286252+0000","last_unstale":"2021-03-15T22:31:33.240837+0000","last_undegraded":"2021-03-15T22:31:33.240837+0000","last_fullsized":"2021-03-15T22:31:33.240837+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:41.254469+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:27:41.254469+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"703","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.242401+0000","last_change":"2021-03-15T22:31:28.287644+0000","last_active":"2021-03-15T22:31:34.242401+0000","last_peered":"2021-03-15T22:31:34.242401+0000","last_clean":"2021-03-15T22:31:34.242401+0000","last_became_active":"2021-03-15T22:31:28.287285+0000","last_became_peered":"2021-03-15T22:31:28.287285+0000","last_unstale":"2021-03-15T22:31:34.242401+0000","last_undegraded":"2021-03-15T22:31:34.242401+0000","last_fullsized":"2021-03-15T22:31:34.242401+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.321715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.321715+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.321715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"371","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.240366+0000","last_change":"2021-03-15T22:29:13.250177+0000","last_active":"2021-03-15T22:31:35.240366+0000","last_peered":"2021-03-15T22:31:35.240366+0000","last_clean":"2021-03-15T22:31:35.240366+0000","last_became_active":"2021-03-15T22:29:11.204254+0000","last_became_peered":"2021-03-15T22:29:11.204254+0000","last_unstale":"2021-03-15T22:31:35.240366+0000","last_undegraded":"2021-03-15T22:31:35.240366+0000","last_fullsized":"2021-03-15T22:31:35.240366+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:29:13.250082+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:29:13.250082+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"742'862","reported_seq":"3104","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.250193+0000","last_change":"2021-03-15T22:31:32.250572+0000","last_active":"2021-03-15T22:31:32.250193+0000","last_peered":"2021-03-15T22:31:32.250193+0000","last_clean":"2021-03-15T22:31:32.250193+0000","last_became_active":"2021-03-15T22:31:31.240075+0000","last_became_peered":"2021-03-15T22:31:31.240075+0000","last_unstale":"2021-03-15T22:31:32.250193+0000","last_undegraded":"2021-03-15T22:31:32.250193+0000","last_fullsized":"2021-03-15T22:31:32.250193+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"630'650","last_scrub_stamp":"2021-03-15T22:29:40.242110+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.242110+0000","log_size":862,"ondisk_log_size":862,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38982633,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":371,"num_read_kb":420859,"num_write":253,"num_write_kb":82276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":59821512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1e6","length":"1"},{"start":"1e9","length":"1"},{"start":"200","length":"1"},{"start":"252","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"674","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.236499+0000","last_change":"2021-03-15T22:31:28.283539+0000","last_active":"2021-03-15T22:31:32.236499+0000","last_peered":"2021-03-15T22:31:32.236499+0000","last_clean":"2021-03-15T22:31:32.236499+0000","last_became_active":"2021-03-15T22:31:28.283092+0000","last_became_peered":"2021-03-15T22:31:28.283092+0000","last_unstale":"2021-03-15T22:31:32.236499+0000","last_undegraded":"2021-03-15T22:31:32.236499+0000","last_fullsized":"2021-03-15T22:31:32.236499+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:07.417736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:07.417736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"744","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.241060+0000","last_change":"2021-03-15T22:28:12.530786+0000","last_active":"2021-03-15T22:31:33.241060+0000","last_peered":"2021-03-15T22:31:33.241060+0000","last_clean":"2021-03-15T22:31:33.241060+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:31:33.241060+0000","last_undegraded":"2021-03-15T22:31:33.241060+0000","last_fullsized":"2021-03-15T22:31:33.241060+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:27:40.278709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:40.278709+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"179","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.235968+0000","last_change":"2021-03-15T22:31:29.162156+0000","last_active":"2021-03-15T22:31:32.235968+0000","last_peered":"2021-03-15T22:31:32.235968+0000","last_clean":"2021-03-15T22:31:32.235968+0000","last_became_active":"2021-03-15T22:31:28.288606+0000","last_became_peered":"2021-03-15T22:31:28.288606+0000","last_unstale":"2021-03-15T22:31:32.235968+0000","last_undegraded":"2021-03-15T22:31:32.235968+0000","last_fullsized":"2021-03-15T22:31:32.235968+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.227473+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"741'10","reported_seq":"913","reported_epoch":"743","state":"active+remapped+backfill_toofull","last_fresh":"2021-03-15T22:31:33.239862+0000","last_change":"2021-03-15T22:31:29.277037+0000","last_active":"2021-03-15T22:31:33.239862+0000","last_peered":"2021-03-15T22:31:33.239862+0000","last_clean":"2021-03-15T22:31:27.146424+0000","last_became_active":"2021-03-15T22:31:29.271151+0000","last_became_peered":"2021-03-15T22:31:29.271151+0000","last_unstale":"2021-03-15T22:31:33.239862+0000","last_undegraded":"2021-03-15T22:31:33.239862+0000","last_fullsized":"2021-03-15T22:31:33.239862+0000","mapping_epoch":738,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:30:07.252960+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.252960+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,6],"avail_no_missing":["7","1","4","6"],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"746'1211","reported_seq":"3358","reported_epoch":"746","state":"active+clean","last_fresh":"2021-03-15T22:31:36.274450+0000","last_change":"2021-03-15T22:31:36.274698+0000","last_active":"2021-03-15T22:31:36.274450+0000","last_peered":"2021-03-15T22:31:36.274450+0000","last_clean":"2021-03-15T22:31:36.274450+0000","last_became_active":"2021-03-15T22:30:34.291079+0000","last_became_peered":"2021-03-15T22:30:34.291079+0000","last_unstale":"2021-03-15T22:31:36.274450+0000","last_undegraded":"2021-03-15T22:31:36.274450+0000","last_fullsized":"2021-03-15T22:31:36.274450+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"623'857","last_scrub_stamp":"2021-03-15T22:29:33.490626+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.490626+0000","log_size":1211,"ondisk_log_size":1211,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63795100,"num_objects":29,"num_object_clones":24,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":2,"num_read":498,"num_read_kb":480541,"num_write":356,"num_write_kb":143086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":80476435,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"176","length":"1"},{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"171","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.240713+0000","last_change":"2021-03-15T22:29:58.503593+0000","last_active":"2021-03-15T22:31:35.240713+0000","last_peered":"2021-03-15T22:31:35.240713+0000","last_clean":"2021-03-15T22:31:35.240713+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:31:35.240713+0000","last_undegraded":"2021-03-15T22:31:35.240713+0000","last_fullsized":"2021-03-15T22:31:35.240713+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:58.503533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:58.503533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"713","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.240851+0000","last_change":"2021-03-15T22:27:44.227979+0000","last_active":"2021-03-15T22:31:33.240851+0000","last_peered":"2021-03-15T22:31:33.240851+0000","last_clean":"2021-03-15T22:31:33.240851+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:31:33.240851+0000","last_undegraded":"2021-03-15T22:31:33.240851+0000","last_fullsized":"2021-03-15T22:31:33.240851+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:27:44.227918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:27:44.227918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"869","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237842+0000","last_change":"2021-03-15T22:31:05.469086+0000","last_active":"2021-03-15T22:31:35.237842+0000","last_peered":"2021-03-15T22:31:35.237842+0000","last_clean":"2021-03-15T22:31:35.237842+0000","last_became_active":"2021-03-15T22:28:59.267623+0000","last_became_peered":"2021-03-15T22:28:59.267623+0000","last_unstale":"2021-03-15T22:31:35.237842+0000","last_undegraded":"2021-03-15T22:31:35.237842+0000","last_fullsized":"2021-03-15T22:31:35.237842+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:31:05.469001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.469001+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"569","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.237062+0000","last_change":"2021-03-15T22:31:18.252819+0000","last_active":"2021-03-15T22:31:33.237062+0000","last_peered":"2021-03-15T22:31:33.237062+0000","last_clean":"2021-03-15T22:31:33.237062+0000","last_became_active":"2021-03-15T22:31:18.252524+0000","last_became_peered":"2021-03-15T22:31:18.252524+0000","last_unstale":"2021-03-15T22:31:33.237062+0000","last_undegraded":"2021-03-15T22:31:33.237062+0000","last_fullsized":"2021-03-15T22:31:33.237062+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:11.029289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.029289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"744'340","reported_seq":"2324","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.258475+0000","last_change":"2021-03-15T22:31:34.248027+0000","last_active":"2021-03-15T22:31:35.258475+0000","last_peered":"2021-03-15T22:31:35.258475+0000","last_clean":"2021-03-15T22:31:35.258475+0000","last_became_active":"2021-03-15T22:28:16.177364+0000","last_became_peered":"2021-03-15T22:28:16.177364+0000","last_unstale":"2021-03-15T22:31:35.258475+0000","last_undegraded":"2021-03-15T22:31:35.258475+0000","last_fullsized":"2021-03-15T22:31:35.258475+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'293","last_scrub_stamp":"2021-03-15T22:30:29.729234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.729234+0000","log_size":340,"ondisk_log_size":340,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14581361,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":102,"num_read_kb":79085,"num_write":97,"num_write_kb":32529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":11148008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"759","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.240550+0000","last_change":"2021-03-15T22:30:32.237010+0000","last_active":"2021-03-15T22:31:33.240550+0000","last_peered":"2021-03-15T22:31:33.240550+0000","last_clean":"2021-03-15T22:31:33.240550+0000","last_became_active":"2021-03-15T22:30:32.236752+0000","last_became_peered":"2021-03-15T22:30:32.236752+0000","last_unstale":"2021-03-15T22:31:33.240550+0000","last_undegraded":"2021-03-15T22:31:33.240550+0000","last_fullsized":"2021-03-15T22:31:33.240550+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:08.137989+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:08.137989+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"581","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.237700+0000","last_change":"2021-03-15T22:31:32.008461+0000","last_active":"2021-03-15T22:31:33.237700+0000","last_peered":"2021-03-15T22:31:33.237700+0000","last_clean":"2021-03-15T22:31:33.237700+0000","last_became_active":"2021-03-15T22:31:18.251403+0000","last_became_peered":"2021-03-15T22:31:18.251403+0000","last_unstale":"2021-03-15T22:31:33.237700+0000","last_undegraded":"2021-03-15T22:31:33.237700+0000","last_fullsized":"2021-03-15T22:31:33.237700+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:32.008403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:32.008403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"764","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.241056+0000","last_change":"2021-03-15T22:31:28.289675+0000","last_active":"2021-03-15T22:31:33.241056+0000","last_peered":"2021-03-15T22:31:33.241056+0000","last_clean":"2021-03-15T22:31:33.241056+0000","last_became_active":"2021-03-15T22:31:28.288901+0000","last_became_peered":"2021-03-15T22:31:28.288901+0000","last_unstale":"2021-03-15T22:31:33.241056+0000","last_undegraded":"2021-03-15T22:31:33.241056+0000","last_fullsized":"2021-03-15T22:31:33.241056+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:08.279129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:30:08.279129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"800","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.238354+0000","last_change":"2021-03-15T22:31:28.308317+0000","last_active":"2021-03-15T22:31:33.238354+0000","last_peered":"2021-03-15T22:31:33.238354+0000","last_clean":"2021-03-15T22:31:33.238354+0000","last_became_active":"2021-03-15T22:31:28.284158+0000","last_became_peered":"2021-03-15T22:31:28.284158+0000","last_unstale":"2021-03-15T22:31:33.238354+0000","last_undegraded":"2021-03-15T22:31:33.238354+0000","last_fullsized":"2021-03-15T22:31:33.238354+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:31:01.245498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:01.245498+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"744'667","reported_seq":"2373","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.261558+0000","last_change":"2021-03-15T22:31:34.261808+0000","last_active":"2021-03-15T22:31:34.261558+0000","last_peered":"2021-03-15T22:31:34.261558+0000","last_clean":"2021-03-15T22:31:34.261558+0000","last_became_active":"2021-03-15T22:30:31.242297+0000","last_became_peered":"2021-03-15T22:30:31.242297+0000","last_unstale":"2021-03-15T22:31:34.261558+0000","last_undegraded":"2021-03-15T22:31:34.261558+0000","last_fullsized":"2021-03-15T22:31:34.261558+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'555","last_scrub_stamp":"2021-03-15T22:30:34.990334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.990334+0000","log_size":667,"ondisk_log_size":667,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38899765,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":311,"num_read_kb":368321,"num_write":195,"num_write_kb":67707,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":24240388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"1e9","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"252","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"223","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.242751+0000","last_change":"2021-03-15T22:31:08.968657+0000","last_active":"2021-03-15T22:31:34.242751+0000","last_peered":"2021-03-15T22:31:34.242751+0000","last_clean":"2021-03-15T22:31:34.242751+0000","last_became_active":"2021-03-15T22:30:32.237720+0000","last_became_peered":"2021-03-15T22:30:32.237720+0000","last_unstale":"2021-03-15T22:31:34.242751+0000","last_undegraded":"2021-03-15T22:31:34.242751+0000","last_fullsized":"2021-03-15T22:31:34.242751+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"559","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.964203+0000","last_change":"2021-03-15T22:31:33.964163+0000","last_active":"2021-03-15T22:31:33.964203+0000","last_peered":"2021-03-15T22:31:33.964203+0000","last_clean":"2021-03-15T22:31:33.964203+0000","last_became_active":"2021-03-15T22:31:18.261916+0000","last_became_peered":"2021-03-15T22:31:18.261916+0000","last_unstale":"2021-03-15T22:31:33.964203+0000","last_undegraded":"2021-03-15T22:31:33.964203+0000","last_fullsized":"2021-03-15T22:31:33.964203+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:33.964082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:33.964082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"125","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.237165+0000","last_change":"2021-03-15T22:31:28.282450+0000","last_active":"2021-03-15T22:31:32.237165+0000","last_peered":"2021-03-15T22:31:32.237165+0000","last_clean":"2021-03-15T22:31:32.237165+0000","last_became_active":"2021-03-15T22:31:28.282098+0000","last_became_peered":"2021-03-15T22:31:28.282098+0000","last_unstale":"2021-03-15T22:31:32.237165+0000","last_undegraded":"2021-03-15T22:31:32.237165+0000","last_fullsized":"2021-03-15T22:31:32.237165+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:07.891775+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.891775+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"256","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.243467+0000","last_change":"2021-03-15T22:30:32.557348+0000","last_active":"2021-03-15T22:31:34.243467+0000","last_peered":"2021-03-15T22:31:34.243467+0000","last_clean":"2021-03-15T22:31:34.243467+0000","last_became_active":"2021-03-15T22:30:32.250712+0000","last_became_peered":"2021-03-15T22:30:32.250712+0000","last_unstale":"2021-03-15T22:31:34.243467+0000","last_undegraded":"2021-03-15T22:31:34.243467+0000","last_fullsized":"2021-03-15T22:31:34.243467+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:30:05.601295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:05.601295+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"744'952","reported_seq":"3480","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.262149+0000","last_change":"2021-03-15T22:31:34.261944+0000","last_active":"2021-03-15T22:31:35.262149+0000","last_peered":"2021-03-15T22:31:35.262149+0000","last_clean":"2021-03-15T22:31:35.262149+0000","last_became_active":"2021-03-15T22:31:31.245658+0000","last_became_peered":"2021-03-15T22:31:31.245658+0000","last_unstale":"2021-03-15T22:31:35.262149+0000","last_undegraded":"2021-03-15T22:31:35.262149+0000","last_fullsized":"2021-03-15T22:31:35.262149+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'870","last_scrub_stamp":"2021-03-15T22:31:02.243396+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.243396+0000","log_size":952,"ondisk_log_size":952,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51592364,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":240,"num_read_kb":188020,"num_write":266,"num_write_kb":90202,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":153074064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"171","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.239334+0000","last_change":"2021-03-15T22:30:29.533509+0000","last_active":"2021-03-15T22:31:35.239334+0000","last_peered":"2021-03-15T22:31:35.239334+0000","last_clean":"2021-03-15T22:31:35.239334+0000","last_became_active":"2021-03-15T22:28:59.273785+0000","last_became_peered":"2021-03-15T22:28:59.273785+0000","last_unstale":"2021-03-15T22:31:35.239334+0000","last_undegraded":"2021-03-15T22:31:35.239334+0000","last_fullsized":"2021-03-15T22:31:35.239334+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:29.533421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.533421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"122","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.236351+0000","last_change":"2021-03-15T22:31:30.660912+0000","last_active":"2021-03-15T22:31:32.236351+0000","last_peered":"2021-03-15T22:31:32.236351+0000","last_clean":"2021-03-15T22:31:32.236351+0000","last_became_active":"2021-03-15T22:31:28.287331+0000","last_became_peered":"2021-03-15T22:31:28.287331+0000","last_unstale":"2021-03-15T22:31:32.236351+0000","last_undegraded":"2021-03-15T22:31:32.236351+0000","last_fullsized":"2021-03-15T22:31:32.236351+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:33.191779+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.191779+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"85","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.237527+0000","last_change":"2021-03-15T22:31:28.278422+0000","last_active":"2021-03-15T22:31:33.237527+0000","last_peered":"2021-03-15T22:31:33.237527+0000","last_clean":"2021-03-15T22:31:33.237527+0000","last_became_active":"2021-03-15T22:31:28.278136+0000","last_became_peered":"2021-03-15T22:31:28.278136+0000","last_unstale":"2021-03-15T22:31:33.237527+0000","last_undegraded":"2021-03-15T22:31:33.237527+0000","last_fullsized":"2021-03-15T22:31:33.237527+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.012676+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"953","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.239261+0000","last_change":"2021-03-15T22:30:32.294089+0000","last_active":"2021-03-15T22:31:31.239261+0000","last_peered":"2021-03-15T22:31:31.239261+0000","last_clean":"2021-03-15T22:31:31.239261+0000","last_became_active":"2021-03-15T22:30:32.244684+0000","last_became_peered":"2021-03-15T22:30:32.244684+0000","last_unstale":"2021-03-15T22:31:31.239261+0000","last_undegraded":"2021-03-15T22:31:31.239261+0000","last_fullsized":"2021-03-15T22:31:31.239261+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:30:03.187386+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.187386+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"207","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.243223+0000","last_change":"2021-03-15T22:30:30.738196+0000","last_active":"2021-03-15T22:31:34.243223+0000","last_peered":"2021-03-15T22:31:34.243223+0000","last_clean":"2021-03-15T22:31:34.243223+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:31:34.243223+0000","last_undegraded":"2021-03-15T22:31:34.243223+0000","last_fullsized":"2021-03-15T22:31:34.243223+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:30.738108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:30.738108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"744'856","reported_seq":"2814","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.259901+0000","last_change":"2021-03-15T22:31:34.249263+0000","last_active":"2021-03-15T22:31:35.259901+0000","last_peered":"2021-03-15T22:31:35.259901+0000","last_clean":"2021-03-15T22:31:35.259901+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:31:35.259901+0000","last_undegraded":"2021-03-15T22:31:35.259901+0000","last_fullsized":"2021-03-15T22:31:35.259901+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"690'772","last_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.836165+0000","log_size":856,"ondisk_log_size":856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38683145,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":268,"num_read_kb":259251,"num_write":220,"num_write_kb":82159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"751","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.241151+0000","last_change":"2021-03-15T22:31:28.289881+0000","last_active":"2021-03-15T22:31:33.241151+0000","last_peered":"2021-03-15T22:31:33.241151+0000","last_clean":"2021-03-15T22:31:33.241151+0000","last_became_active":"2021-03-15T22:31:28.289352+0000","last_became_peered":"2021-03-15T22:31:28.289352+0000","last_unstale":"2021-03-15T22:31:33.241151+0000","last_undegraded":"2021-03-15T22:31:33.241151+0000","last_fullsized":"2021-03-15T22:31:33.241151+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:07.163034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:07.163034+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"738","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.238226+0000","last_change":"2021-03-15T22:31:28.277011+0000","last_active":"2021-03-15T22:31:31.238226+0000","last_peered":"2021-03-15T22:31:31.238226+0000","last_clean":"2021-03-15T22:31:31.238226+0000","last_became_active":"2021-03-15T22:31:28.276641+0000","last_became_peered":"2021-03-15T22:31:28.276641+0000","last_unstale":"2021-03-15T22:31:31.238226+0000","last_undegraded":"2021-03-15T22:31:31.238226+0000","last_fullsized":"2021-03-15T22:31:31.238226+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.070598+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.070598+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"542","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.237059+0000","last_change":"2021-03-15T22:30:43.021718+0000","last_active":"2021-03-15T22:31:33.237059+0000","last_peered":"2021-03-15T22:31:33.237059+0000","last_clean":"2021-03-15T22:31:33.237059+0000","last_became_active":"2021-03-15T22:30:32.250316+0000","last_became_peered":"2021-03-15T22:30:32.250316+0000","last_unstale":"2021-03-15T22:31:33.237059+0000","last_undegraded":"2021-03-15T22:31:33.237059+0000","last_fullsized":"2021-03-15T22:31:33.237059+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:43.021632+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:30:43.021632+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"480","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.239046+0000","last_change":"2021-03-15T22:29:56.187294+0000","last_active":"2021-03-15T22:31:31.239046+0000","last_peered":"2021-03-15T22:31:31.239046+0000","last_clean":"2021-03-15T22:31:31.239046+0000","last_became_active":"2021-03-15T22:29:51.222262+0000","last_became_peered":"2021-03-15T22:29:51.222262+0000","last_unstale":"2021-03-15T22:31:31.239046+0000","last_undegraded":"2021-03-15T22:31:31.239046+0000","last_fullsized":"2021-03-15T22:31:31.239046+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:29:56.187210+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.187210+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"537","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.238967+0000","last_change":"2021-03-15T22:31:18.249355+0000","last_active":"2021-03-15T22:31:33.238967+0000","last_peered":"2021-03-15T22:31:33.238967+0000","last_clean":"2021-03-15T22:31:33.238967+0000","last_became_active":"2021-03-15T22:31:18.249044+0000","last_became_peered":"2021-03-15T22:31:18.249044+0000","last_unstale":"2021-03-15T22:31:33.238967+0000","last_undegraded":"2021-03-15T22:31:33.238967+0000","last_fullsized":"2021-03-15T22:31:33.238967+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:55.737371+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:55.737371+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"744'2163","reported_seq":"4742","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.265049+0000","last_change":"2021-03-15T22:31:34.268993+0000","last_active":"2021-03-15T22:31:35.265049+0000","last_peered":"2021-03-15T22:31:35.265049+0000","last_clean":"2021-03-15T22:31:35.265049+0000","last_became_active":"2021-03-15T22:28:59.276888+0000","last_became_peered":"2021-03-15T22:28:59.276888+0000","last_unstale":"2021-03-15T22:31:35.265049+0000","last_undegraded":"2021-03-15T22:31:35.265049+0000","last_fullsized":"2021-03-15T22:31:35.265049+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'2007","last_scrub_stamp":"2021-03-15T22:31:02.050395+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.050395+0000","log_size":2163,"ondisk_log_size":2163,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":83236894,"num_objects":41,"num_object_clones":34,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":3,"num_read":683,"num_read_kb":678592,"num_write":544,"num_write_kb":230974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":205133613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"116","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.237180+0000","last_change":"2021-03-15T22:30:28.967439+0000","last_active":"2021-03-15T22:31:32.237180+0000","last_peered":"2021-03-15T22:31:32.237180+0000","last_clean":"2021-03-15T22:31:32.237180+0000","last_became_active":"2021-03-15T22:29:51.227896+0000","last_became_peered":"2021-03-15T22:29:51.227896+0000","last_unstale":"2021-03-15T22:31:32.237180+0000","last_undegraded":"2021-03-15T22:31:32.237180+0000","last_fullsized":"2021-03-15T22:31:32.237180+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:28.967356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.967356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"743","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.240644+0000","last_change":"2021-03-15T22:31:28.275169+0000","last_active":"2021-03-15T22:31:33.240644+0000","last_peered":"2021-03-15T22:31:33.240644+0000","last_clean":"2021-03-15T22:31:33.240644+0000","last_became_active":"2021-03-15T22:31:28.274813+0000","last_became_peered":"2021-03-15T22:31:28.274813+0000","last_unstale":"2021-03-15T22:31:33.240644+0000","last_undegraded":"2021-03-15T22:31:33.240644+0000","last_fullsized":"2021-03-15T22:31:33.240644+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:05.206223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:30:05.206223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"363","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.240570+0000","last_change":"2021-03-15T22:31:28.285973+0000","last_active":"2021-03-15T22:31:35.240570+0000","last_peered":"2021-03-15T22:31:35.240570+0000","last_clean":"2021-03-15T22:31:35.240570+0000","last_became_active":"2021-03-15T22:31:28.285608+0000","last_became_peered":"2021-03-15T22:31:28.285608+0000","last_unstale":"2021-03-15T22:31:35.240570+0000","last_undegraded":"2021-03-15T22:31:35.240570+0000","last_fullsized":"2021-03-15T22:31:35.240570+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:28:45.454461+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.454461+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"609","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237513+0000","last_change":"2021-03-15T22:31:28.303171+0000","last_active":"2021-03-15T22:31:35.237513+0000","last_peered":"2021-03-15T22:31:35.237513+0000","last_clean":"2021-03-15T22:31:35.237513+0000","last_became_active":"2021-03-15T22:31:28.277985+0000","last_became_peered":"2021-03-15T22:31:28.277985+0000","last_unstale":"2021-03-15T22:31:35.237513+0000","last_undegraded":"2021-03-15T22:31:35.237513+0000","last_fullsized":"2021-03-15T22:31:35.237513+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:30:28.176831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.176831+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"755","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.240411+0000","last_change":"2021-03-15T22:30:32.244033+0000","last_active":"2021-03-15T22:31:33.240411+0000","last_peered":"2021-03-15T22:31:33.240411+0000","last_clean":"2021-03-15T22:31:33.240411+0000","last_became_active":"2021-03-15T22:30:32.243696+0000","last_became_peered":"2021-03-15T22:30:32.243696+0000","last_unstale":"2021-03-15T22:31:33.240411+0000","last_undegraded":"2021-03-15T22:31:33.240411+0000","last_fullsized":"2021-03-15T22:31:33.240411+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.257476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"743'605","reported_seq":"2717","reported_epoch":"744","state":"active+clean+snaptrim","last_fresh":"2021-03-15T22:31:34.249970+0000","last_change":"2021-03-15T22:31:34.250432+0000","last_active":"2021-03-15T22:31:34.249970+0000","last_peered":"2021-03-15T22:31:34.249970+0000","last_clean":"2021-03-15T22:31:34.249970+0000","last_became_active":"2021-03-15T22:30:35.366323+0000","last_became_peered":"2021-03-15T22:30:35.366323+0000","last_unstale":"2021-03-15T22:31:34.249970+0000","last_undegraded":"2021-03-15T22:31:34.249970+0000","last_fullsized":"2021-03-15T22:31:34.249970+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":0,"last_scrub":"560'396","last_scrub_stamp":"2021-03-15T22:28:45.222374+0000","last_deep_scrub":"172'65","last_deep_scrub_stamp":"2021-03-15T22:21:27.654567+0000","last_clean_scrub_stamp":"2021-03-15T22:28:45.222374+0000","log_size":605,"ondisk_log_size":605,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":30030113,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":163,"num_read_kb":157137,"num_write":160,"num_write_kb":69018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":54764539,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1e6","length":"1"},{"start":"1e9","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"252","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"780","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.239037+0000","last_change":"2021-03-15T22:30:41.171985+0000","last_active":"2021-03-15T22:31:31.239037+0000","last_peered":"2021-03-15T22:31:31.239037+0000","last_clean":"2021-03-15T22:31:31.239037+0000","last_became_active":"2021-03-15T22:29:51.222189+0000","last_became_peered":"2021-03-15T22:29:51.222189+0000","last_unstale":"2021-03-15T22:31:31.239037+0000","last_undegraded":"2021-03-15T22:31:31.239037+0000","last_fullsized":"2021-03-15T22:31:31.239037+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:41.171920+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.171920+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"228","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.242837+0000","last_change":"2021-03-15T22:31:01.010456+0000","last_active":"2021-03-15T22:31:34.242837+0000","last_peered":"2021-03-15T22:31:34.242837+0000","last_clean":"2021-03-15T22:31:34.242837+0000","last_became_active":"2021-03-15T22:28:59.276599+0000","last_became_peered":"2021-03-15T22:28:59.276599+0000","last_unstale":"2021-03-15T22:31:34.242837+0000","last_undegraded":"2021-03-15T22:31:34.242837+0000","last_fullsized":"2021-03-15T22:31:34.242837+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:01.010393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:01.010393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"1551","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.782549+0000","last_change":"2021-03-15T22:28:52.152942+0000","last_active":"2021-03-15T22:31:33.782549+0000","last_peered":"2021-03-15T22:31:33.782549+0000","last_clean":"2021-03-15T22:31:33.782549+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:31:33.782549+0000","last_undegraded":"2021-03-15T22:31:33.782549+0000","last_fullsized":"2021-03-15T22:31:33.782549+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:28:52.152875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:28:52.152875+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":804,"num_read_kb":799,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"275","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.244427+0000","last_change":"2021-03-15T22:31:11.926326+0000","last_active":"2021-03-15T22:31:34.244427+0000","last_peered":"2021-03-15T22:31:34.244427+0000","last_clean":"2021-03-15T22:31:34.244427+0000","last_became_active":"2021-03-15T22:30:32.251686+0000","last_became_peered":"2021-03-15T22:30:32.251686+0000","last_unstale":"2021-03-15T22:31:34.244427+0000","last_undegraded":"2021-03-15T22:31:34.244427+0000","last_fullsized":"2021-03-15T22:31:34.244427+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:31:11.926268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.926268+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"125","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.237270+0000","last_change":"2021-03-15T22:31:18.270517+0000","last_active":"2021-03-15T22:31:32.237270+0000","last_peered":"2021-03-15T22:31:32.237270+0000","last_clean":"2021-03-15T22:31:32.237270+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:31:32.237270+0000","last_undegraded":"2021-03-15T22:31:32.237270+0000","last_fullsized":"2021-03-15T22:31:32.237270+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:11.436723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.436723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"743'2306","reported_seq":"4834","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.267246+0000","last_change":"2021-03-15T22:31:33.267468+0000","last_active":"2021-03-15T22:31:33.267246+0000","last_peered":"2021-03-15T22:31:33.267246+0000","last_clean":"2021-03-15T22:31:33.267246+0000","last_became_active":"2021-03-15T22:29:52.225069+0000","last_became_peered":"2021-03-15T22:29:52.225069+0000","last_unstale":"2021-03-15T22:31:33.267246+0000","last_undegraded":"2021-03-15T22:31:33.267246+0000","last_fullsized":"2021-03-15T22:31:33.267246+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"650'1835","last_scrub_stamp":"2021-03-15T22:29:59.763141+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:59.763141+0000","log_size":2306,"ondisk_log_size":2306,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114168283,"num_objects":50,"num_object_clones":43,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":1,"num_read":918,"num_read_kb":1002264,"num_write":579,"num_write_kb":235138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":165660176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1e6","length":"1"},{"start":"1e9","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"252","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"734","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.238624+0000","last_change":"2021-03-15T22:31:28.276749+0000","last_active":"2021-03-15T22:31:31.238624+0000","last_peered":"2021-03-15T22:31:31.238624+0000","last_clean":"2021-03-15T22:31:31.238624+0000","last_became_active":"2021-03-15T22:31:28.276431+0000","last_became_peered":"2021-03-15T22:31:28.276431+0000","last_unstale":"2021-03-15T22:31:31.238624+0000","last_undegraded":"2021-03-15T22:31:31.238624+0000","last_fullsized":"2021-03-15T22:31:31.238624+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:02.160813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.160813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"765","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.239197+0000","last_change":"2021-03-15T22:31:12.070899+0000","last_active":"2021-03-15T22:31:31.239197+0000","last_peered":"2021-03-15T22:31:31.239197+0000","last_clean":"2021-03-15T22:31:31.239197+0000","last_became_active":"2021-03-15T22:30:32.243790+0000","last_became_peered":"2021-03-15T22:30:32.243790+0000","last_unstale":"2021-03-15T22:31:31.239197+0000","last_undegraded":"2021-03-15T22:31:31.239197+0000","last_fullsized":"2021-03-15T22:31:31.239197+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:12.070843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:12.070843+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"282","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.243370+0000","last_change":"2021-03-15T22:29:52.258695+0000","last_active":"2021-03-15T22:31:34.243370+0000","last_peered":"2021-03-15T22:31:34.243370+0000","last_clean":"2021-03-15T22:31:34.243370+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:31:34.243370+0000","last_undegraded":"2021-03-15T22:31:34.243370+0000","last_fullsized":"2021-03-15T22:31:34.243370+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:29:08.872836+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:29:08.872836+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"666","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.242728+0000","last_change":"2021-03-15T22:31:28.279332+0000","last_active":"2021-03-15T22:31:34.242728+0000","last_peered":"2021-03-15T22:31:34.242728+0000","last_clean":"2021-03-15T22:31:34.242728+0000","last_became_active":"2021-03-15T22:31:28.278813+0000","last_became_peered":"2021-03-15T22:31:28.278813+0000","last_unstale":"2021-03-15T22:31:34.242728+0000","last_undegraded":"2021-03-15T22:31:34.242728+0000","last_fullsized":"2021-03-15T22:31:34.242728+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:33.255248+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:29:33.255248+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"746'361","reported_seq":"2568","reported_epoch":"746","state":"active+clean","last_fresh":"2021-03-15T22:31:36.248193+0000","last_change":"2021-03-15T22:31:36.248440+0000","last_active":"2021-03-15T22:31:36.248193+0000","last_peered":"2021-03-15T22:31:36.248193+0000","last_clean":"2021-03-15T22:31:36.248193+0000","last_became_active":"2021-03-15T22:31:33.250635+0000","last_became_peered":"2021-03-15T22:31:33.250635+0000","last_unstale":"2021-03-15T22:31:36.248193+0000","last_undegraded":"2021-03-15T22:31:36.248193+0000","last_fullsized":"2021-03-15T22:31:36.248193+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'353","last_scrub_stamp":"2021-03-15T22:31:02.426448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.426448+0000","log_size":361,"ondisk_log_size":361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10190621,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":129,"num_read_kb":161540,"num_write":86,"num_write_kb":32987,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":51449003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"176","length":"1"},{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"236","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.242793+0000","last_change":"2021-03-15T22:30:32.238279+0000","last_active":"2021-03-15T22:31:34.242793+0000","last_peered":"2021-03-15T22:31:34.242793+0000","last_clean":"2021-03-15T22:31:34.242793+0000","last_became_active":"2021-03-15T22:30:32.237925+0000","last_became_peered":"2021-03-15T22:30:32.237925+0000","last_unstale":"2021-03-15T22:31:34.242793+0000","last_undegraded":"2021-03-15T22:31:34.242793+0000","last_fullsized":"2021-03-15T22:31:34.242793+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:37.724915+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:37.724915+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"677","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.237219+0000","last_change":"2021-03-15T22:30:32.257626+0000","last_active":"2021-03-15T22:31:32.237219+0000","last_peered":"2021-03-15T22:31:32.237219+0000","last_clean":"2021-03-15T22:31:32.237219+0000","last_became_active":"2021-03-15T22:30:32.257328+0000","last_became_peered":"2021-03-15T22:30:32.257328+0000","last_unstale":"2021-03-15T22:31:32.237219+0000","last_undegraded":"2021-03-15T22:31:32.237219+0000","last_fullsized":"2021-03-15T22:31:32.237219+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:58.713662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:29:58.713662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"950","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.238700+0000","last_change":"2021-03-15T22:31:08.134727+0000","last_active":"2021-03-15T22:31:31.238700+0000","last_peered":"2021-03-15T22:31:31.238700+0000","last_clean":"2021-03-15T22:31:31.238700+0000","last_became_active":"2021-03-15T22:29:51.222356+0000","last_became_peered":"2021-03-15T22:29:51.222356+0000","last_unstale":"2021-03-15T22:31:31.238700+0000","last_undegraded":"2021-03-15T22:31:31.238700+0000","last_fullsized":"2021-03-15T22:31:31.238700+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:31:08.134638+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.134638+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"741'10","reported_seq":"297","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.243275+0000","last_change":"2021-03-15T22:31:28.389370+0000","last_active":"2021-03-15T22:31:34.243275+0000","last_peered":"2021-03-15T22:31:34.243275+0000","last_clean":"2021-03-15T22:31:34.243275+0000","last_became_active":"2021-03-15T22:31:28.280972+0000","last_became_peered":"2021-03-15T22:31:28.280972+0000","last_unstale":"2021-03-15T22:31:34.243275+0000","last_undegraded":"2021-03-15T22:31:34.243275+0000","last_fullsized":"2021-03-15T22:31:34.243275+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'9","last_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.971303+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"746'595","reported_seq":"2645","reported_epoch":"746","state":"active+clean","last_fresh":"2021-03-15T22:31:36.260943+0000","last_change":"2021-03-15T22:31:36.261300+0000","last_active":"2021-03-15T22:31:36.260943+0000","last_peered":"2021-03-15T22:31:36.260943+0000","last_clean":"2021-03-15T22:31:36.260943+0000","last_became_active":"2021-03-15T22:30:34.297623+0000","last_became_peered":"2021-03-15T22:30:34.297623+0000","last_unstale":"2021-03-15T22:31:36.260943+0000","last_undegraded":"2021-03-15T22:31:36.260943+0000","last_fullsized":"2021-03-15T22:31:36.260943+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'591","last_scrub_stamp":"2021-03-15T22:31:35.474935+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:31:35.474935+0000","log_size":595,"ondisk_log_size":595,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30281334,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":204,"num_read_kb":183762,"num_write":170,"num_write_kb":65821,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":47935099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"176","length":"1"},{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"729","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.240253+0000","last_change":"2021-03-15T22:31:28.282929+0000","last_active":"2021-03-15T22:31:33.240253+0000","last_peered":"2021-03-15T22:31:33.240253+0000","last_clean":"2021-03-15T22:31:33.240253+0000","last_became_active":"2021-03-15T22:31:28.282600+0000","last_became_peered":"2021-03-15T22:31:28.282600+0000","last_unstale":"2021-03-15T22:31:33.240253+0000","last_undegraded":"2021-03-15T22:31:33.240253+0000","last_fullsized":"2021-03-15T22:31:33.240253+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:06.530069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:06.530069+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"207","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.244504+0000","last_change":"2021-03-15T22:30:37.798454+0000","last_active":"2021-03-15T22:31:34.244504+0000","last_peered":"2021-03-15T22:31:34.244504+0000","last_clean":"2021-03-15T22:31:34.244504+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:31:34.244504+0000","last_undegraded":"2021-03-15T22:31:34.244504+0000","last_fullsized":"2021-03-15T22:31:34.244504+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.798388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"896","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237859+0000","last_change":"2021-03-15T22:31:29.720965+0000","last_active":"2021-03-15T22:31:35.237859+0000","last_peered":"2021-03-15T22:31:35.237859+0000","last_clean":"2021-03-15T22:31:35.237859+0000","last_became_active":"2021-03-15T22:31:28.279662+0000","last_became_peered":"2021-03-15T22:31:28.279662+0000","last_unstale":"2021-03-15T22:31:35.237859+0000","last_undegraded":"2021-03-15T22:31:35.237859+0000","last_fullsized":"2021-03-15T22:31:35.237859+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:30:42.383512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.383512+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"206","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.241311+0000","last_change":"2021-03-15T22:30:27.472423+0000","last_active":"2021-03-15T22:31:35.241311+0000","last_peered":"2021-03-15T22:31:35.241311+0000","last_clean":"2021-03-15T22:31:35.241311+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:31:35.241311+0000","last_undegraded":"2021-03-15T22:31:35.241311+0000","last_fullsized":"2021-03-15T22:31:35.241311+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:30:27.472320+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:30:27.472320+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"746'1027","reported_seq":"3289","reported_epoch":"746","state":"active+clean","last_fresh":"2021-03-15T22:31:36.281444+0000","last_change":"2021-03-15T22:31:36.281811+0000","last_active":"2021-03-15T22:31:36.281444+0000","last_peered":"2021-03-15T22:31:36.281444+0000","last_clean":"2021-03-15T22:31:36.281444+0000","last_became_active":"2021-03-15T22:31:31.245546+0000","last_became_peered":"2021-03-15T22:31:31.245546+0000","last_unstale":"2021-03-15T22:31:36.281444+0000","last_undegraded":"2021-03-15T22:31:36.281444+0000","last_fullsized":"2021-03-15T22:31:36.281444+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'889","last_scrub_stamp":"2021-03-15T22:30:29.983045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.983045+0000","log_size":1027,"ondisk_log_size":1027,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41263750,"num_objects":23,"num_object_clones":19,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":398,"num_read_kb":412350,"num_write":259,"num_write_kb":90159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":139702636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"176","length":"1"},{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"773","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236566+0000","last_change":"2021-03-15T22:30:32.262270+0000","last_active":"2021-03-15T22:31:35.236566+0000","last_peered":"2021-03-15T22:31:35.236566+0000","last_clean":"2021-03-15T22:31:35.236566+0000","last_became_active":"2021-03-15T22:30:32.261995+0000","last_became_peered":"2021-03-15T22:30:32.261995+0000","last_unstale":"2021-03-15T22:31:35.236566+0000","last_undegraded":"2021-03-15T22:31:35.236566+0000","last_fullsized":"2021-03-15T22:31:35.236566+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:09.101167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:09.101167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"526","reported_epoch":"746","state":"active+clean","last_fresh":"2021-03-15T22:31:36.472708+0000","last_change":"2021-03-15T22:31:36.472708+0000","last_active":"2021-03-15T22:31:36.472708+0000","last_peered":"2021-03-15T22:31:36.472708+0000","last_clean":"2021-03-15T22:31:36.472708+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:31:36.472708+0000","last_undegraded":"2021-03-15T22:31:36.472708+0000","last_fullsized":"2021-03-15T22:31:36.472708+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:36.472646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:31:36.472646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"197","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.241480+0000","last_change":"2021-03-15T22:29:41.515256+0000","last_active":"2021-03-15T22:31:35.241480+0000","last_peered":"2021-03-15T22:31:35.241480+0000","last_clean":"2021-03-15T22:31:35.241480+0000","last_became_active":"2021-03-15T22:28:59.276755+0000","last_became_peered":"2021-03-15T22:28:59.276755+0000","last_unstale":"2021-03-15T22:31:35.241480+0000","last_undegraded":"2021-03-15T22:31:35.241480+0000","last_fullsized":"2021-03-15T22:31:35.241480+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:29:41.515193+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.515193+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"768","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236472+0000","last_change":"2021-03-15T22:31:04.495639+0000","last_active":"2021-03-15T22:31:35.236472+0000","last_peered":"2021-03-15T22:31:35.236472+0000","last_clean":"2021-03-15T22:31:35.236472+0000","last_became_active":"2021-03-15T22:28:59.266487+0000","last_became_peered":"2021-03-15T22:28:59.266487+0000","last_unstale":"2021-03-15T22:31:35.236472+0000","last_undegraded":"2021-03-15T22:31:35.236472+0000","last_fullsized":"2021-03-15T22:31:35.236472+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.495574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.495574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"746'973","reported_seq":"3143","reported_epoch":"746","state":"active+clean","last_fresh":"2021-03-15T22:31:36.263998+0000","last_change":"2021-03-15T22:31:36.264240+0000","last_active":"2021-03-15T22:31:36.263998+0000","last_peered":"2021-03-15T22:31:36.263998+0000","last_clean":"2021-03-15T22:31:36.263998+0000","last_became_active":"2021-03-15T22:31:31.244986+0000","last_became_peered":"2021-03-15T22:31:31.244986+0000","last_unstale":"2021-03-15T22:31:36.263998+0000","last_undegraded":"2021-03-15T22:31:36.263998+0000","last_fullsized":"2021-03-15T22:31:36.263998+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"626'732","last_scrub_stamp":"2021-03-15T22:29:36.271647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:29:36.271647+0000","log_size":973,"ondisk_log_size":973,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31289812,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":302,"num_read_kb":284490,"num_write":228,"num_write_kb":80445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":50339417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"176","length":"1"},{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"667","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.237260+0000","last_change":"2021-03-15T22:31:04.380476+0000","last_active":"2021-03-15T22:31:32.237260+0000","last_peered":"2021-03-15T22:31:32.237260+0000","last_clean":"2021-03-15T22:31:32.237260+0000","last_became_active":"2021-03-15T22:29:52.223342+0000","last_became_peered":"2021-03-15T22:29:52.223342+0000","last_unstale":"2021-03-15T22:31:32.237260+0000","last_undegraded":"2021-03-15T22:31:32.237260+0000","last_fullsized":"2021-03-15T22:31:32.237260+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.380417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.380417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"984","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237345+0000","last_change":"2021-03-15T22:31:18.257612+0000","last_active":"2021-03-15T22:31:35.237345+0000","last_peered":"2021-03-15T22:31:35.237345+0000","last_clean":"2021-03-15T22:31:35.237345+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:31:35.237345+0000","last_undegraded":"2021-03-15T22:31:35.237345+0000","last_fullsized":"2021-03-15T22:31:35.237345+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'11","last_scrub_stamp":"2021-03-15T22:29:36.937870+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:36.937870+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"171","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.241224+0000","last_change":"2021-03-15T22:30:09.506976+0000","last_active":"2021-03-15T22:31:35.241224+0000","last_peered":"2021-03-15T22:31:35.241224+0000","last_clean":"2021-03-15T22:31:35.241224+0000","last_became_active":"2021-03-15T22:28:59.274186+0000","last_became_peered":"2021-03-15T22:28:59.274186+0000","last_unstale":"2021-03-15T22:31:35.241224+0000","last_undegraded":"2021-03-15T22:31:35.241224+0000","last_fullsized":"2021-03-15T22:31:35.241224+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:09.506914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:30:09.506914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"595","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.237886+0000","last_change":"2021-03-15T22:31:02.092287+0000","last_active":"2021-03-15T22:31:33.237886+0000","last_peered":"2021-03-15T22:31:33.237886+0000","last_clean":"2021-03-15T22:31:33.237886+0000","last_became_active":"2021-03-15T22:30:32.262267+0000","last_became_peered":"2021-03-15T22:30:32.262267+0000","last_unstale":"2021-03-15T22:31:33.237886+0000","last_undegraded":"2021-03-15T22:31:33.237886+0000","last_fullsized":"2021-03-15T22:31:33.237886+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:31:02.092217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.092217+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"742'301","reported_seq":"2351","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.243914+0000","last_change":"2021-03-15T22:31:32.244197+0000","last_active":"2021-03-15T22:31:32.243914+0000","last_peered":"2021-03-15T22:31:32.243914+0000","last_clean":"2021-03-15T22:31:32.243914+0000","last_became_active":"2021-03-15T22:31:31.242099+0000","last_became_peered":"2021-03-15T22:31:31.242099+0000","last_unstale":"2021-03-15T22:31:32.243914+0000","last_undegraded":"2021-03-15T22:31:32.243914+0000","last_fullsized":"2021-03-15T22:31:32.243914+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"684'281","last_scrub_stamp":"2021-03-15T22:30:36.290657+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.290657+0000","log_size":301,"ondisk_log_size":301,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15228363,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":108,"num_read_kb":100343,"num_write":62,"num_write_kb":21303,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":21427429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"1e6","length":"1"},{"start":"1e9","length":"1"},{"start":"200","length":"1"},{"start":"252","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"469","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.239530+0000","last_change":"2021-03-15T22:31:28.290105+0000","last_active":"2021-03-15T22:31:35.239530+0000","last_peered":"2021-03-15T22:31:35.239530+0000","last_clean":"2021-03-15T22:31:35.239530+0000","last_became_active":"2021-03-15T22:31:28.289518+0000","last_became_peered":"2021-03-15T22:31:28.289518+0000","last_unstale":"2021-03-15T22:31:35.239530+0000","last_undegraded":"2021-03-15T22:31:35.239530+0000","last_fullsized":"2021-03-15T22:31:35.239530+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:40.500803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.500803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"753","reported_epoch":"742","state":"active+clean","last_fresh":"2021-03-15T22:31:32.237441+0000","last_change":"2021-03-15T22:31:31.241376+0000","last_active":"2021-03-15T22:31:32.237441+0000","last_peered":"2021-03-15T22:31:32.237441+0000","last_clean":"2021-03-15T22:31:32.237441+0000","last_became_active":"2021-03-15T22:31:31.241017+0000","last_became_peered":"2021-03-15T22:31:31.241017+0000","last_unstale":"2021-03-15T22:31:32.237441+0000","last_undegraded":"2021-03-15T22:31:32.237441+0000","last_fullsized":"2021-03-15T22:31:32.237441+0000","mapping_epoch":740,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'7","last_scrub_stamp":"2021-03-15T22:29:40.964753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.964753+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"732","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237320+0000","last_change":"2021-03-15T22:30:34.363233+0000","last_active":"2021-03-15T22:31:35.237320+0000","last_peered":"2021-03-15T22:31:35.237320+0000","last_clean":"2021-03-15T22:31:35.237320+0000","last_became_active":"2021-03-15T22:30:31.239376+0000","last_became_peered":"2021-03-15T22:30:31.239376+0000","last_unstale":"2021-03-15T22:31:35.237320+0000","last_undegraded":"2021-03-15T22:31:35.237320+0000","last_fullsized":"2021-03-15T22:31:35.237320+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.363177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"849","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236958+0000","last_change":"2021-03-15T22:31:28.315261+0000","last_active":"2021-03-15T22:31:35.236958+0000","last_peered":"2021-03-15T22:31:35.236958+0000","last_clean":"2021-03-15T22:31:35.236958+0000","last_became_active":"2021-03-15T22:31:28.281662+0000","last_became_peered":"2021-03-15T22:31:28.281662+0000","last_unstale":"2021-03-15T22:31:35.236958+0000","last_undegraded":"2021-03-15T22:31:35.236958+0000","last_fullsized":"2021-03-15T22:31:35.236958+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:30:04.058167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:04.058167+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"743'299","reported_seq":"1817","reported_epoch":"744","state":"active+clean","last_fresh":"2021-03-15T22:31:34.250873+0000","last_change":"2021-03-15T22:31:34.256884+0000","last_active":"2021-03-15T22:31:34.250873+0000","last_peered":"2021-03-15T22:31:34.250873+0000","last_clean":"2021-03-15T22:31:34.250873+0000","last_became_active":"2021-03-15T22:30:32.257058+0000","last_became_peered":"2021-03-15T22:30:32.257058+0000","last_unstale":"2021-03-15T22:31:34.250873+0000","last_undegraded":"2021-03-15T22:31:34.250873+0000","last_fullsized":"2021-03-15T22:31:34.250873+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'291","last_scrub_stamp":"2021-03-15T22:31:30.979407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:31:30.979407+0000","log_size":299,"ondisk_log_size":299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21076112,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":152,"num_read_kb":164360,"num_write":92,"num_write_kb":31214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":32799803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"1e9","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"252","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"16","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.237794+0000","last_change":"2021-03-15T22:31:28.287568+0000","last_active":"2021-03-15T22:31:33.237794+0000","last_peered":"2021-03-15T22:31:33.237794+0000","last_clean":"2021-03-15T22:31:33.237794+0000","last_became_active":"2021-03-15T22:31:28.287218+0000","last_became_peered":"2021-03-15T22:31:28.287218+0000","last_unstale":"2021-03-15T22:31:33.237794+0000","last_undegraded":"2021-03-15T22:31:33.237794+0000","last_fullsized":"2021-03-15T22:31:33.237794+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.283355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.283355+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"765","reported_epoch":"741","state":"active+clean","last_fresh":"2021-03-15T22:31:31.239376+0000","last_change":"2021-03-15T22:31:18.250629+0000","last_active":"2021-03-15T22:31:31.239376+0000","last_peered":"2021-03-15T22:31:31.239376+0000","last_clean":"2021-03-15T22:31:31.239376+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:31:31.239376+0000","last_undegraded":"2021-03-15T22:31:31.239376+0000","last_fullsized":"2021-03-15T22:31:31.239376+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:41.190058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:29:41.190058+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"20","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.236678+0000","last_change":"2021-03-15T22:31:28.287806+0000","last_active":"2021-03-15T22:31:35.236678+0000","last_peered":"2021-03-15T22:31:35.236678+0000","last_clean":"2021-03-15T22:31:35.236678+0000","last_became_active":"2021-03-15T22:31:28.287437+0000","last_became_peered":"2021-03-15T22:31:28.287437+0000","last_unstale":"2021-03-15T22:31:35.236678+0000","last_undegraded":"2021-03-15T22:31:35.236678+0000","last_fullsized":"2021-03-15T22:31:35.236678+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:43.259585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:43.259585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"816","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237395+0000","last_change":"2021-03-15T22:30:38.419647+0000","last_active":"2021-03-15T22:31:35.237395+0000","last_peered":"2021-03-15T22:31:35.237395+0000","last_clean":"2021-03-15T22:31:35.237395+0000","last_became_active":"2021-03-15T22:30:31.239360+0000","last_became_peered":"2021-03-15T22:30:31.239360+0000","last_unstale":"2021-03-15T22:31:35.237395+0000","last_undegraded":"2021-03-15T22:31:35.237395+0000","last_fullsized":"2021-03-15T22:31:35.237395+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:30:38.419551+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.419551+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"744'1072","reported_seq":"3452","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.264744+0000","last_change":"2021-03-15T22:31:34.270933+0000","last_active":"2021-03-15T22:31:35.264744+0000","last_peered":"2021-03-15T22:31:35.264744+0000","last_clean":"2021-03-15T22:31:35.264744+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:31:35.264744+0000","last_undegraded":"2021-03-15T22:31:35.264744+0000","last_fullsized":"2021-03-15T22:31:35.264744+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'877","last_scrub_stamp":"2021-03-15T22:30:27.732142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:27.732142+0000","log_size":1072,"ondisk_log_size":1072,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49052617,"num_objects":27,"num_object_clones":23,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":3,"num_read":412,"num_read_kb":427661,"num_write":273,"num_write_kb":95335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1e0","length":"1"},{"start":"1e6","length":"1"},{"start":"200","length":"1"},{"start":"211","length":"1"},{"start":"254","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"724","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237567+0000","last_change":"2021-03-15T22:30:36.417325+0000","last_active":"2021-03-15T22:31:35.237567+0000","last_peered":"2021-03-15T22:31:35.237567+0000","last_clean":"2021-03-15T22:31:35.237567+0000","last_became_active":"2021-03-15T22:30:31.243269+0000","last_became_peered":"2021-03-15T22:30:31.243269+0000","last_unstale":"2021-03-15T22:31:35.237567+0000","last_undegraded":"2021-03-15T22:31:35.237567+0000","last_fullsized":"2021-03-15T22:31:35.237567+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"732'13","reported_seq":"671","reported_epoch":"743","state":"active+clean","last_fresh":"2021-03-15T22:31:33.238183+0000","last_change":"2021-03-15T22:31:18.249987+0000","last_active":"2021-03-15T22:31:33.238183+0000","last_peered":"2021-03-15T22:31:33.238183+0000","last_clean":"2021-03-15T22:31:33.238183+0000","last_became_active":"2021-03-15T22:31:18.249734+0000","last_became_peered":"2021-03-15T22:31:18.249734+0000","last_unstale":"2021-03-15T22:31:33.238183+0000","last_undegraded":"2021-03-15T22:31:33.238183+0000","last_fullsized":"2021-03-15T22:31:33.238183+0000","mapping_epoch":682,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:31:05.998563+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.998563+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"171","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.240635+0000","last_change":"2021-03-15T22:30:03.567594+0000","last_active":"2021-03-15T22:31:35.240635+0000","last_peered":"2021-03-15T22:31:35.240635+0000","last_clean":"2021-03-15T22:31:35.240635+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:31:35.240635+0000","last_undegraded":"2021-03-15T22:31:35.240635+0000","last_fullsized":"2021-03-15T22:31:35.240635+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.567529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"896","reported_epoch":"745","state":"active+clean","last_fresh":"2021-03-15T22:31:35.237050+0000","last_change":"2021-03-15T22:31:28.300885+0000","last_active":"2021-03-15T22:31:35.237050+0000","last_peered":"2021-03-15T22:31:35.237050+0000","last_clean":"2021-03-15T22:31:35.237050+0000","last_became_active":"2021-03-15T22:31:28.280812+0000","last_became_peered":"2021-03-15T22:31:28.280812+0000","last_unstale":"2021-03-15T22:31:35.237050+0000","last_undegraded":"2021-03-15T22:31:35.237050+0000","last_fullsized":"2021-03-15T22:31:35.237050+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:30:00.029954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:00.029954+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":672352267,"num_objects":311,"num_object_clones":261,"num_object_copies":933,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":311,"num_whiteouts":14,"num_read":5259,"num_read_kb":5368576,"num_write":3840,"num_write_kb":1450353,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2360,"num_bytes_recovered":1223461680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1600016384,"data_stored":2312109756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":14590,"ondisk_log_size":14590,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":804,"num_read_kb":799,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1483,"num_read_kb":988,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":874,"num_bytes_recovered":68050,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2228224,"data_stored":80200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":916,"ondisk_log_size":916,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":113,"ondisk_log_size":113,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299560,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":206932,"kb_used_data":175244,"kb_used_omap":0,"kb_used_meta":31680,"kb_avail":93538220,"statfs":{"total":95995035648,"available":95783137280,"internally_reserved":0,"allocated":179449856,"data_stored":267972751,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32440320},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:31:13 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.44800000000000001,"15min":0.47599999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.215},"max":{"1min":0.58799999999999997,"5min":1.4750000000000001,"15min":2.73},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.48999999999999999,"15min":0.49199999999999999},"min":{"1min":0.317,"5min":0.222,"15min":0.222},"max":{"1min":0.75800000000000001,"5min":2.0499999999999998,"15min":3.8799999999999999},"last":0.40100000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:31:13 2021","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.45100000000000001,"15min":0.47899999999999998},"min":{"1min":0.28599999999999998,"5min":0.27300000000000002,"15min":0.22600000000000001},"max":{"1min":0.56499999999999995,"5min":1.4350000000000001,"15min":2.6859999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48399999999999999,"15min":0.48999999999999999},"min":{"1min":0.34100000000000003,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.64600000000000002,"5min":2.0779999999999998,"15min":2.0779999999999998},"last":0.35199999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:31:31 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.58299999999999996,"5min":0.58299999999999996,"15min":0.58299999999999996},"last":0.435},{"interface":"front","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.29899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":1.119,"5min":1.119,"15min":1.119},"last":0.32100000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:31:13 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.501,"15min":0.51000000000000001},"min":{"1min":0.34100000000000003,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.67500000000000004,"5min":1.702,"15min":2.0339999999999998},"last":0.379},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.53900000000000003,"15min":0.53900000000000003},"min":{"1min":0.34799999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.66200000000000003,"5min":5.415,"15min":5.415},"last":0.41999999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:31:13 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.43099999999999999,"15min":0.441},"min":{"1min":0.27900000000000003,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":0.63900000000000001,"5min":1.927,"15min":1.927},"last":0.502},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.44500000000000001,"15min":0.45600000000000002},"min":{"1min":0.31,"5min":0.253,"15min":0.253},"max":{"1min":0.67500000000000004,"5min":1.6779999999999999,"15min":2.5339999999999998},"last":0.59999999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:31:31 2021","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.44700000000000001,"15min":0.44400000000000001},"min":{"1min":0.33300000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.67500000000000004,"5min":0.71199999999999997,"15min":0.71199999999999997},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.496,"15min":0.501},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.70299999999999996,"5min":1.071,"15min":1.071},"last":0.51500000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:31:13 2021","interfaces":[{"interface":"back","average":{"1min":0.52600000000000002,"5min":0.48599999999999999,"15min":0.51200000000000001},"min":{"1min":0.33900000000000002,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.68500000000000005,"5min":2.1179999999999999,"15min":2.1179999999999999},"last":0.57899999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.48999999999999999,"15min":0.51200000000000001},"min":{"1min":0.34899999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63900000000000001,"5min":1.8560000000000001,"15min":1.8560000000000001},"last":0.56599999999999995}]}]},{"osd":6,"up_from":0,"seq":193273528492,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575037,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":290336,"kb_used_data":236556,"kb_used_omap":0,"kb_used_meta":51776,"kb_avail":93454816,"statfs":{"total":95995035648,"available":95697731584,"internally_reserved":0,"allocated":242233344,"data_stored":333865861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53018624},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:30:40 2021","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.30399999999999999,"15min":0.51600000000000001},"min":{"1min":0.184,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":1.325,"5min":1.325,"15min":54.427999999999997},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.32800000000000001,"15min":0.53200000000000003},"min":{"1min":0.219,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.82799999999999996,"5min":0.82799999999999996,"15min":54.356999999999999},"last":0.56000000000000005}]},{"osd":2,"last update":"Mon Mar 15 22:25:44 2021","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.39900000000000002,"15min":0.41499999999999998},"min":{"1min":0.23999999999999999,"5min":0.23400000000000001,"15min":0.223},"max":{"1min":0.54300000000000004,"5min":1.1220000000000001,"15min":2.4100000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.33800000000000002,"15min":0.38600000000000001},"min":{"1min":0.25,"5min":0.215,"15min":0.215},"max":{"1min":0.502,"5min":0.69799999999999995,"15min":2.0760000000000001},"last":0.46999999999999997}]},{"osd":3,"last update":"Mon Mar 15 22:31:07 2021","interfaces":[{"interface":"back","average":{"1min":0.36499999999999999,"5min":0.33800000000000002,"15min":0.38100000000000001},"min":{"1min":0.26400000000000001,"5min":0.215,"15min":0.215},"max":{"1min":0.68400000000000005,"5min":0.73499999999999999,"15min":1.796},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.35899999999999999,"15min":0.41199999999999998},"min":{"1min":0.27100000000000002,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":2.0150000000000001,"5min":2.0150000000000001,"15min":2.0150000000000001},"last":0.57499999999999996}]},{"osd":4,"last update":"Mon Mar 15 22:31:17 2021","interfaces":[{"interface":"back","average":{"1min":0.53700000000000003,"5min":0.47899999999999998,"15min":0.53500000000000003},"min":{"1min":0.32300000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":0.90400000000000003,"5min":0.90400000000000003,"15min":3.3439999999999999},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53500000000000003,"15min":0.58799999999999997},"min":{"1min":0.34499999999999997,"5min":0.27200000000000002,"15min":0.253},"max":{"1min":1.284,"5min":4.1790000000000003,"15min":4.6139999999999999},"last":0.49299999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:31:28 2021","interfaces":[{"interface":"back","average":{"1min":0.58099999999999996,"5min":0.53500000000000003,"15min":0.52700000000000002},"min":{"1min":0.42399999999999999,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":1.226,"5min":1.226,"15min":1.226},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.504,"15min":0.49399999999999999},"min":{"1min":0.36799999999999999,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.86799999999999999,"5min":0.86799999999999999,"15min":0.86799999999999999},"last":0.78300000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:30:50 2021","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52600000000000002,"15min":0.59699999999999998},"min":{"1min":0.35499999999999998,"5min":0.314,"15min":0.314},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":1.75},"last":0.71599999999999997},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.52600000000000002,"15min":0.57899999999999996},"min":{"1min":0.40500000000000003,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":0.80900000000000005,"5min":0.86599999999999999,"15min":1.3580000000000001},"last":0.44700000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:31:07 2021","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.52900000000000003,"15min":0.67400000000000004},"min":{"1min":0.40400000000000003,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":0.91800000000000004,"5min":0.91800000000000004,"15min":8.8100000000000005},"last":0.76500000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53000000000000003,"15min":0.57799999999999996},"min":{"1min":0.40400000000000003,"5min":0.37,"15min":0.35799999999999998},"max":{"1min":0.85399999999999998,"5min":0.85399999999999998,"15min":1.325},"last":0.41999999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738560,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":276844,"kb_used_data":228964,"kb_used_omap":0,"kb_used_meta":47872,"kb_avail":93468308,"statfs":{"total":95995035648,"available":95711547392,"internally_reserved":0,"allocated":234459136,"data_stored":327560862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":49020928},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:30:33 2021","interfaces":[{"interface":"back","average":{"1min":0.27800000000000002,"5min":0.32800000000000001,"15min":0.34200000000000003},"min":{"1min":0.19800000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.435,"5min":1.024,"15min":1.7070000000000001},"last":0.27200000000000002},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.35799999999999998,"15min":0.36499999999999999},"min":{"1min":0.23300000000000001,"5min":0.20699999999999999,"15min":0.19900000000000001},"max":{"1min":0.55100000000000005,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.29999999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:31:32 2021","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.64900000000000002,"5min":0.64900000000000002,"15min":0.64900000000000002},"last":0.314},{"interface":"front","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.27700000000000002,"5min":0.27700000000000002,"15min":0.27700000000000002},"max":{"1min":0.55000000000000004,"5min":0.55000000000000004,"15min":0.55000000000000004},"last":0.34599999999999997}]},{"osd":3,"last update":"Mon Mar 15 22:31:16 2021","interfaces":[{"interface":"back","average":{"1min":0.37,"5min":0.39100000000000001,"15min":0.38800000000000001},"min":{"1min":0.23100000000000001,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":0.56899999999999995,"5min":1.538,"15min":1.538},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.38400000000000001,"15min":0.40200000000000002},"min":{"1min":0.26400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.72099999999999997,"5min":1.8340000000000001,"15min":1.8340000000000001},"last":0.33100000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:31:16 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.495,"15min":0.53700000000000003},"min":{"1min":0.28799999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.78100000000000003,"5min":0.88700000000000001,"15min":1.583},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.51500000000000001,"15min":0.56699999999999995},"min":{"1min":0.35899999999999999,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.78900000000000003,"5min":1.0620000000000001,"15min":1.645},"last":0.70399999999999996}]},{"osd":5,"last update":"Mon Mar 15 22:31:29 2021","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.32600000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.70999999999999996,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.48499999999999999,"15min":0.47699999999999998},"min":{"1min":0.41699999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.749,"5min":0.751,"15min":0.751},"last":0.54800000000000004}]},{"osd":6,"last update":"Mon Mar 15 22:30:51 2021","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.52800000000000002,"15min":0.55600000000000005},"min":{"1min":0.35899999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.71899999999999997,"5min":1.282,"15min":1.6180000000000001},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.53100000000000003,"15min":0.58099999999999996},"min":{"1min":0.32000000000000001,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76000000000000001,"5min":0.98099999999999998,"15min":1.0049999999999999},"last":0.68600000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:31:16 2021","interfaces":[{"interface":"back","average":{"1min":0.58199999999999996,"5min":0.51900000000000002,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77000000000000002,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.73699999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.51900000000000002,"15min":0.56999999999999995},"min":{"1min":0.39100000000000001,"5min":0.317,"15min":0.317},"max":{"1min":0.72299999999999998,"5min":0.875,"15min":1.6619999999999999},"last":0.74299999999999999}]}]},{"osd":2,"up_from":681,"seq":2924872728590,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":253688,"kb_used_data":228336,"kb_used_omap":119,"kb_used_meta":25224,"kb_avail":93491464,"statfs":{"total":95995035648,"available":95735259136,"internally_reserved":0,"allocated":233816064,"data_stored":308504214,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":25829737},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"last":0.37},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.30399999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.39600000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.432,"5min":0.432,"15min":0.432},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.25700000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.63200000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.57299999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.59199999999999997}]},{"osd":7,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.65100000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.504}]}]},{"osd":3,"up_from":26,"seq":111669149878,"num_pgs":73,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":396156,"kb_used_data":347380,"kb_used_omap":0,"kb_used_meta":48768,"kb_avail":93348996,"statfs":{"total":95995035648,"available":95589371904,"internally_reserved":0,"allocated":355717120,"data_stored":512262497,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":49938432},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:31:15 2021","interfaces":[{"interface":"back","average":{"1min":0.314,"5min":0.32000000000000001,"15min":0.33700000000000002},"min":{"1min":0.20899999999999999,"5min":0.184,"15min":0.17699999999999999},"max":{"1min":0.52000000000000002,"5min":1.1499999999999999,"15min":2.625},"last":0.29299999999999998},{"interface":"front","average":{"1min":0.32200000000000001,"5min":0.316,"15min":0.34499999999999997},"min":{"1min":0.23400000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.47199999999999998,"5min":1.419,"15min":2.6339999999999999},"last":0.32100000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:31:15 2021","interfaces":[{"interface":"back","average":{"1min":0.36799999999999999,"5min":0.36099999999999999,"15min":0.40100000000000002},"min":{"1min":0.25900000000000001,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":0.52500000000000002,"5min":0.72599999999999998,"15min":6.0199999999999996},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.36899999999999999,"15min":0.40500000000000003},"min":{"1min":0.249,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.53900000000000003,"5min":2.5750000000000002,"15min":7.6130000000000004},"last":0.35399999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:31:32 2021","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.31},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.39600000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:31:20 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.53300000000000003},"min":{"1min":0.32100000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.77500000000000002,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.58899999999999997},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48699999999999999,"15min":0.52500000000000002},"min":{"1min":0.41099999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.65600000000000003,"5min":2.1059999999999999,"15min":2.2890000000000001},"last":0.71599999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:31:25 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.79900000000000004,"15min":0.84299999999999997},"min":{"1min":0.378,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73399999999999999,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":0.73399999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":1.0369999999999999,"15min":1.1240000000000001},"min":{"1min":0.38200000000000001,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.72599999999999998,"5min":15.776,"15min":15.776},"last":0.59799999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:30:52 2021","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.53000000000000003,"15min":0.57299999999999995},"min":{"1min":0.315,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":0.73599999999999999,"5min":2.5299999999999998,"15min":2.6499999999999999},"last":0.746},{"interface":"front","average":{"1min":0.54800000000000004,"5min":0.55400000000000005,"15min":0.56799999999999995},"min":{"1min":0.35899999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.86399999999999999,"5min":3.472,"15min":3.472},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:31:15 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.59299999999999997,"15min":0.58099999999999996},"min":{"1min":0.308,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81100000000000005,"5min":5.625,"15min":5.625},"last":0.68000000000000005},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.53400000000000003,"15min":0.56100000000000005},"min":{"1min":0.371,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.75900000000000001,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":0.69299999999999995}]}]},{"osd":4,"up_from":31,"seq":133143986355,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":151984,"kb_used_data":112936,"kb_used_omap":0,"kb_used_meta":39040,"kb_avail":93593168,"statfs":{"total":95995035648,"available":95839404032,"internally_reserved":0,"allocated":115646464,"data_stored":155380349,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39976960},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:31:28 2021","interfaces":[{"interface":"back","average":{"1min":0.72499999999999998,"5min":0.53300000000000003,"15min":0.48799999999999999},"min":{"1min":0.33500000000000002,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":4.7050000000000001,"5min":4.7050000000000001,"15min":4.7050000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.54600000000000004,"15min":0.504},"min":{"1min":0.246,"5min":0.246,"15min":0.246},"max":{"1min":0.63100000000000001,"5min":5.0209999999999999,"15min":5.0209999999999999},"last":0.57799999999999996}]},{"osd":1,"last update":"Mon Mar 15 22:31:28 2021","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.28399999999999997,"5min":0.23999999999999999,"15min":0.23599999999999999},"max":{"1min":1.3720000000000001,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.51200000000000001,"15min":0.50900000000000001},"min":{"1min":0.308,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":1.494,"5min":4.718,"15min":4.718},"last":0.86599999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:31:31 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.45300000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:31:28 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.58999999999999997,"15min":0.55300000000000005},"min":{"1min":0.32200000000000001,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.80200000000000005,"5min":5.0499999999999998,"15min":5.0499999999999998},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.59099999999999997,"5min":0.56799999999999995,"15min":0.54400000000000004},"min":{"1min":0.41399999999999998,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.86299999999999999,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.89500000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:31:28 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.41599999999999998,"15min":0.40699999999999997},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.50900000000000001,"5min":0.435,"15min":0.42299999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.66000000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:30:54 2021","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.45900000000000002,"15min":0.45400000000000001},"min":{"1min":0.317,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.878,"5min":4.3079999999999998,"15min":4.3079999999999998},"last":0.504},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.52900000000000003,"15min":0.52800000000000002},"min":{"1min":0.40500000000000003,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.83799999999999997,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.71499999999999997}]},{"osd":7,"last update":"Mon Mar 15 22:31:10 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.50800000000000001,"15min":0.502},"min":{"1min":0.36199999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.80900000000000005,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.79900000000000004},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56100000000000005,"15min":0.54300000000000004},"min":{"1min":0.35599999999999998,"5min":0.25,"15min":0.25},"max":{"1min":0.81799999999999995,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":1.034}]}]},{"osd":5,"up_from":616,"seq":2645699854363,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":206540,"kb_used_data":175044,"kb_used_omap":153,"kb_used_meta":31334,"kb_avail":93538612,"statfs":{"total":95995035648,"available":95783538688,"internally_reserved":0,"allocated":179245056,"data_stored":255563806,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":32086545},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:31:28 2021","interfaces":[{"interface":"back","average":{"1min":0.50600000000000001,"5min":0.46200000000000002,"15min":0.45500000000000002},"min":{"1min":0.317,"5min":0.29199999999999998,"15min":0.29199999999999998},"max":{"1min":0.71199999999999997,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.34000000000000002,"5min":0.34000000000000002,"15min":0.34000000000000002},"max":{"1min":0.68899999999999995,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.33200000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:31:28 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.45500000000000002,"15min":0.44900000000000001},"min":{"1min":0.27500000000000002,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.44600000000000001},{"interface":"front","average":{"1min":0.51000000000000001,"5min":0.48999999999999999,"15min":0.48699999999999999},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.747,"5min":0.876,"15min":0.876},"last":0.35899999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:31:31 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"max":{"1min":0.871,"5min":0.871,"15min":0.871},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.53300000000000003},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.40100000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:31:28 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.47699999999999998,"15min":0.47099999999999997},"min":{"1min":0.36499999999999999,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.71499999999999997,"5min":0.71499999999999997,"15min":0.71499999999999997},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.44800000000000001,"15min":0.438},"min":{"1min":0.34899999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.38}]},{"osd":4,"last update":"Mon Mar 15 22:31:28 2021","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.38500000000000001,"15min":0.373},"min":{"1min":0.32800000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.66400000000000003,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.46400000000000002,"5min":0.39300000000000002,"15min":0.38100000000000001},"min":{"1min":0.312,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.82299999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.45700000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:31:28 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.45300000000000001,"15min":0.44700000000000001},"min":{"1min":0.30099999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.78400000000000003,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.53500000000000003,"5min":0.47199999999999998,"15min":0.46200000000000002},"min":{"1min":0.42399999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.69199999999999995,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.52600000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:31:28 2021","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.44700000000000001,"15min":0.436},"min":{"1min":0.378,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.78800000000000003,"5min":0.78800000000000003,"15min":0.78800000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.53400000000000003,"5min":0.49199999999999999,"15min":0.48499999999999999},"min":{"1min":0.40100000000000002,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.77600000000000002,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.51300000000000001}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":10926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":12926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":180224,"data_stored":6416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":376832,"data_stored":13488,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":11740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":7576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":225484800,"data_stored":321848536,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":233242624,"data_stored":328169860,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":223797248,"data_stored":301619914,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":346664960,"data_stored":506532327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":106635264,"data_stored":149651477,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":170389504,"data_stored":249864174,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":123215872,"data_stored":192150239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":170586112,"data_stored":262273229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:31:39.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool set default.rgw.meta pgp_num 32 2021-03-15T22:31:39.240 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:39.264 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:39.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:39.266 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:39.318 INFO:tasks.rados.rados.0.smithi076.stdout:2764: done (3 left) 2021-03-15T22:31:39.318 INFO:tasks.rados.rados.0.smithi076.stdout:2765: done (2 left) 2021-03-15T22:31:39.319 INFO:tasks.rados.rados.0.smithi076.stdout:2767: done (1 left) 2021-03-15T22:31:39.319 INFO:tasks.rados.rados.0.smithi076.stdout:2768: done (0 left) 2021-03-15T22:31:39.319 INFO:tasks.rados.rados.0.smithi076.stdout:2769: read oid 20 snap -1 2021-03-15T22:31:39.319 INFO:tasks.rados.rados.0.smithi076.stdout:2769: expect (ObjNum 874 snap 287 seq_num 874) 2021-03-15T22:31:39.319 INFO:tasks.rados.rados.0.smithi076.stdout:2770: write oid 18 current snap is 321 2021-03-15T22:31:39.320 INFO:tasks.rados.rados.0.smithi076.stdout:2770: seq_num 957 ranges {484299=557418,1684797=399502} 2021-03-15T22:31:39.325 INFO:tasks.rados.rados.0.smithi076.stdout:2770: writing smithi07694878-18 from 484299 to 1041717 tid 1 2021-03-15T22:31:39.330 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:39.331 INFO:tasks.rados.rados.0.smithi076.stdout:2770: writing smithi07694878-18 from 1684797 to 2084299 tid 2 2021-03-15T22:31:39.331 INFO:tasks.rados.rados.0.smithi076.stdout:2771: write oid 38 current snap is 321 2021-03-15T22:31:39.331 INFO:tasks.rados.rados.0.smithi076.stdout:2771: seq_num 958 ranges {501106=749020,1743279=357827} 2021-03-15T22:31:39.337 INFO:tasks.rados.rados.0.smithi076.stdout:2771: writing smithi07694878-38 from 501106 to 1250126 tid 1 2021-03-15T22:31:39.342 INFO:tasks.rados.rados.0.smithi076.stdout:2771: writing smithi07694878-38 from 1743279 to 2101106 tid 2 2021-03-15T22:31:39.342 INFO:tasks.rados.rados.0.smithi076.stdout:2772: delete oid 21 current snap is 321 2021-03-15T22:31:39.342 INFO:tasks.rados.rados.0.smithi076.stdout:2769: expect (ObjNum 874 snap 287 seq_num 874) 2021-03-15T22:31:39.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:39.392 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:39.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:39.395 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:39.436 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[56514]: cluster 2021-03-15T22:31:37.740148+0000 osd.6 (osd.6) 116 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:39.437 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[56514]: cluster 2021-03-15T22:31:37.740920+0000 osd.6 (osd.6) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:39.437 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[56514]: cluster 2021-03-15T22:31:38.308880+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2021-03-15T22:31:39.437 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[56514]: audit 2021-03-15T22:31:38.900542+0000 mon.a (mon.0) 1979 : audit [DBG] from='client.? 172.21.15.76:0/3181644148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:39.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: cluster 2021-03-15T22:31:37.740148+0000 2021-03-15T22:31:39.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: osd.6 (osd.6) 116 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:39.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: cluster 2021-03-15T22:31:37 2021-03-15T22:31:39.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: .740920+0000 osd.6 (osd.6) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:39.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: cluster 2021-03-15T22:31:39.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: 2021 2021-03-15T22:31:39.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: - 2021-03-15T22:31:39.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: 03 2021-03-15T22:31:39.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: - 2021-03-15T22:31:39.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: 15T22:31:38.308880+0000 mon.a (mon 2021-03-15T22:31:39.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: .0) 1978 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2021-03-15T22:31:39.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: audit 2021-03-15T22:31:39.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: 2021 2021-03-15T22:31:39.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: -03-15T22:31:38 2021-03-15T22:31:39.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:39 smithi076 conmon[62156]: .900542+0000 mon.a (mon.0) 1979 : audit [DBG] from='client.? 172.21.15.76:0/3181644148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:39.495 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:39.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:39.520 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:39.521 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:39.584 INFO:tasks.rados.rados.0.smithi076.stdout:2769: done (3 left) 2021-03-15T22:31:39.584 INFO:tasks.rados.rados.0.smithi076.stdout:2772: done (2 left) 2021-03-15T22:31:39.585 INFO:tasks.rados.rados.0.smithi076.stdout:2773: write (excl) oid 17 current snap is 321 2021-03-15T22:31:39.585 INFO:tasks.rados.rados.0.smithi076.stdout:2770: finishing write tid 1 to smithi07694878-18 2021-03-15T22:31:39.585 INFO:tasks.rados.rados.0.smithi076.stdout:2773: seq_num 960 ranges {534720=499518,1460243=674477} 2021-03-15T22:31:39.588 INFO:tasks.rados.rados.0.smithi076.stdout:2773: writing smithi07694878-17 from 534720 to 1034238 tid 1 2021-03-15T22:31:39.596 INFO:tasks.rados.rados.0.smithi076.stdout:2773: writing smithi07694878-17 from 1460243 to 2134720 tid 2 2021-03-15T22:31:39.596 INFO:tasks.rados.rados.0.smithi076.stdout:2774: write (excl) oid 27 current snap is 321 2021-03-15T22:31:39.597 INFO:tasks.rados.rados.0.smithi076.stdout:2770: finishing write tid 2 to smithi07694878-18 2021-03-15T22:31:39.597 INFO:tasks.rados.rados.0.smithi076.stdout:2770: finishing write tid 3 to smithi07694878-18 2021-03-15T22:31:39.598 INFO:tasks.rados.rados.0.smithi076.stdout:2770: finishing write tid 4 to smithi07694878-18 2021-03-15T22:31:39.598 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 1963 (ObjNum 957 snap 321 seq_num 957) dirty exists 2021-03-15T22:31:39.599 INFO:tasks.rados.rados.0.smithi076.stdout:2770: left oid 18 (ObjNum 957 snap 321 seq_num 957) 2021-03-15T22:31:39.599 INFO:tasks.rados.rados.0.smithi076.stdout:2771: finishing write tid 1 to smithi07694878-38 2021-03-15T22:31:39.599 INFO:tasks.rados.rados.0.smithi076.stdout:2774: seq_num 961 ranges {551527=574767,1918725=232802} 2021-03-15T22:31:39.601 INFO:tasks.rados.rados.0.smithi076.stdout:2774: writing smithi07694878-27 from 551527 to 1126294 tid 1 2021-03-15T22:31:39.608 INFO:tasks.rados.rados.0.smithi076.stdout:2774: writing smithi07694878-27 from 1918725 to 2151527 tid 2 2021-03-15T22:31:39.609 INFO:tasks.rados.rados.0.smithi076.stdout:2770: done (3 left) 2021-03-15T22:31:39.609 INFO:tasks.rados.rados.0.smithi076.stdout:2775: copy_from oid 46 from oid 47 current snap is 321 2021-03-15T22:31:39.610 INFO:tasks.rados.rados.0.smithi076.stdout:2771: finishing write tid 2 to smithi07694878-38 2021-03-15T22:31:39.610 INFO:tasks.rados.rados.0.smithi076.stdout:2771: finishing write tid 3 to smithi07694878-38 2021-03-15T22:31:39.611 INFO:tasks.rados.rados.0.smithi076.stdout:2771: finishing write tid 4 to smithi07694878-38 2021-03-15T22:31:39.611 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2089 (ObjNum 958 snap 321 seq_num 958) dirty exists 2021-03-15T22:31:39.611 INFO:tasks.rados.rados.0.smithi076.stdout:2771: left oid 38 (ObjNum 958 snap 321 seq_num 958) 2021-03-15T22:31:39.612 INFO:tasks.rados.rados.0.smithi076.stdout:2773: finishing write tid 1 to smithi07694878-17 2021-03-15T22:31:39.612 INFO:tasks.rados.rados.0.smithi076.stdout:2773: finishing write tid 2 to smithi07694878-17 2021-03-15T22:31:39.612 INFO:tasks.rados.rados.0.smithi076.stdout:2773: finishing write tid 3 to smithi07694878-17 2021-03-15T22:31:39.613 INFO:tasks.rados.rados.0.smithi076.stdout:2773: finishing write tid 4 to smithi07694878-17 2021-03-15T22:31:39.613 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 856 (ObjNum 960 snap 321 seq_num 960) dirty exists 2021-03-15T22:31:39.613 INFO:tasks.rados.rados.0.smithi076.stdout:2773: left oid 17 (ObjNum 960 snap 321 seq_num 960) 2021-03-15T22:31:39.614 INFO:tasks.rados.rados.0.smithi076.stdout:2774: finishing write tid 1 to smithi07694878-27 2021-03-15T22:31:39.614 INFO:tasks.rados.rados.0.smithi076.stdout:2771: done (3 left) 2021-03-15T22:31:39.614 INFO:tasks.rados.rados.0.smithi076.stdout:2773: done (2 left) 2021-03-15T22:31:39.615 INFO:tasks.rados.rados.0.smithi076.stdout:2776: write oid 20 current snap is 321 2021-03-15T22:31:39.615 INFO:tasks.rados.rados.0.smithi076.stdout:2776: seq_num 962 ranges {568334=650016,1860854=307480} 2021-03-15T22:31:39.616 INFO:tasks.rados.rados.0.smithi076.stdout:2776: writing smithi07694878-20 from 568334 to 1218350 tid 1 2021-03-15T22:31:39.620 INFO:tasks.rados.rados.0.smithi076.stdout:2776: writing smithi07694878-20 from 1860854 to 2168334 tid 2 2021-03-15T22:31:39.620 INFO:tasks.rados.rados.0.smithi076.stdout:2777: read oid 21 snap -1 2021-03-15T22:31:39.621 INFO:tasks.rados.rados.0.smithi076.stdout:2777: expect deleted 2021-03-15T22:31:39.621 INFO:tasks.rados.rados.0.smithi076.stdout:2778: snap_create 2021-03-15T22:31:39.621 INFO:tasks.rados.rados.0.smithi076.stdout:2774: finishing write tid 2 to smithi07694878-27 2021-03-15T22:31:39.621 INFO:tasks.rados.rados.0.smithi076.stdout:2774: finishing write tid 3 to smithi07694878-27 2021-03-15T22:31:39.621 INFO:tasks.rados.rados.0.smithi076.stdout:2774: finishing write tid 4 to smithi07694878-27 2021-03-15T22:31:39.622 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1765 (ObjNum 961 snap 321 seq_num 961) dirty exists 2021-03-15T22:31:39.622 INFO:tasks.rados.rados.0.smithi076.stdout:2774: left oid 27 (ObjNum 961 snap 321 seq_num 961) 2021-03-15T22:31:39.622 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:39.626 INFO:tasks.rados.rados.0.smithi076.stdout:2776: finishing write tid 1 to smithi07694878-20 2021-03-15T22:31:39.629 INFO:tasks.rados.rados.0.smithi076.stdout:2775: finishing copy_from to smithi07694878-46 2021-03-15T22:31:39.629 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 1758 (ObjNum 928 snap 315 seq_num 928) dirty exists 2021-03-15T22:31:39.629 INFO:tasks.rados.rados.0.smithi076.stdout:2776: finishing write tid 2 to smithi07694878-20 2021-03-15T22:31:39.630 INFO:tasks.rados.rados.0.smithi076.stdout:2775: finishing copy_from racing read to smithi07694878-46 2021-03-15T22:31:39.630 INFO:tasks.rados.rados.0.smithi076.stdout:2776: finishing write tid 3 to smithi07694878-20 2021-03-15T22:31:39.630 INFO:tasks.rados.rados.0.smithi076.stdout:2776: finishing write tid 4 to smithi07694878-20 2021-03-15T22:31:39.631 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 1763 (ObjNum 962 snap 321 seq_num 962) dirty exists 2021-03-15T22:31:39.631 INFO:tasks.rados.rados.0.smithi076.stdout:2776: left oid 20 (ObjNum 962 snap 321 seq_num 962) 2021-03-15T22:31:39.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:39.648 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:39.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:39.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:39.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:39.774 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:39.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:39 smithi125 conmon[56739]: cluster 2021-03-15T22:31:37.740148+0000 2021-03-15T22:31:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:39 smithi125 conmon[56739]: osd.6 (osd.6) 116 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:39 smithi125 conmon[56739]: cluster 2021-03-15T22:31:37.740920+0000 osd.6 (osd.6) 117 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:39 smithi125 conmon[56739]: cluster 2021-03-15T22:31:38.308880+0000 mon.a (mon.0) 1978 : cluster [DBG] osdmap e748: 8 total, 8 up, 7 in 2021-03-15T22:31:39.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:39 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:31:39.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:39 smithi125 conmon[56739]: 03-15T22:31:38.900542+0000 2021-03-15T22:31:39.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:39 smithi125 conmon[56739]: mon.a (mon.0) 1979 : audit [DBG] 2021-03-15T22:31:39.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:39 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/3181644148' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:39.877 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:39.903 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:39.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:39.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:40.006 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:40.031 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:40.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:40.033 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:40.075 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:31:40.075 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:31:40.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:40.160 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:40.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:40.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:40.241 INFO:tasks.rados.rados.0.smithi076.stdout:2774: done (4 left) 2021-03-15T22:31:40.241 INFO:tasks.rados.rados.0.smithi076.stdout:2775: done (3 left) 2021-03-15T22:31:40.241 INFO:tasks.rados.rados.0.smithi076.stdout:2776: done (2 left) 2021-03-15T22:31:40.242 INFO:tasks.rados.rados.0.smithi076.stdout:2777: done (1 left) 2021-03-15T22:31:40.242 INFO:tasks.rados.rados.0.smithi076.stdout:2778: done (0 left) 2021-03-15T22:31:40.242 INFO:tasks.rados.rados.0.smithi076.stdout:2779: write (excl) oid 42 current snap is 322 2021-03-15T22:31:40.242 INFO:tasks.rados.rados.0.smithi076.stdout:2779: seq_num 963 ranges {585141=725265,1919336=265805} 2021-03-15T22:31:40.249 INFO:tasks.rados.rados.0.smithi076.stdout:2779: writing smithi07694878-42 from 585141 to 1310406 tid 1 2021-03-15T22:31:40.256 INFO:tasks.rados.rados.0.smithi076.stdout:2779: writing smithi07694878-42 from 1919336 to 2185141 tid 2 2021-03-15T22:31:40.257 INFO:tasks.rados.rados.0.smithi076.stdout:2780: rollback oid 24 current snap is 322 2021-03-15T22:31:40.258 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 303 2021-03-15T22:31:40.258 INFO:tasks.rados.rados.0.smithi076.stdout:2781: write oid 11 current snap is 322 2021-03-15T22:31:40.258 INFO:tasks.rados.rados.0.smithi076.stdout:2781: seq_num 964 ranges {601948=400514,1577818=624130} 2021-03-15T22:31:40.263 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:40.264 INFO:tasks.rados.rados.0.smithi076.stdout:2781: writing smithi07694878-11 from 601948 to 1002462 tid 1 2021-03-15T22:31:40.273 INFO:tasks.rados.rados.0.smithi076.stdout:2781: writing smithi07694878-11 from 1577818 to 2201948 tid 2 2021-03-15T22:31:40.274 INFO:tasks.rados.rados.0.smithi076.stdout:2782: delete oid 12 current snap is 322 2021-03-15T22:31:40.274 INFO:tasks.rados.rados.0.smithi076.stdout:2779: finishing write tid 1 to smithi07694878-42 2021-03-15T22:31:40.274 INFO:tasks.rados.rados.0.smithi076.stdout:2780: finishing rollback tid 0 to smithi07694878-24 2021-03-15T22:31:40.274 INFO:tasks.rados.rados.0.smithi076.stdout:2780: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:31:40.274 INFO:tasks.rados.rados.0.smithi076.stdout:2780: finishing rollback tid 2 to smithi07694878-24 2021-03-15T22:31:40.275 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1724 (ObjNum 864 snap 285 seq_num 864) dirty exists 2021-03-15T22:31:40.275 INFO:tasks.rados.rados.0.smithi076.stdout:2779: finishing write tid 2 to smithi07694878-42 2021-03-15T22:31:40.275 INFO:tasks.rados.rados.0.smithi076.stdout:2779: finishing write tid 3 to smithi07694878-42 2021-03-15T22:31:40.275 INFO:tasks.rados.rados.0.smithi076.stdout:2779: finishing write tid 4 to smithi07694878-42 2021-03-15T22:31:40.276 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2360 (ObjNum 963 snap 322 seq_num 963) dirty exists 2021-03-15T22:31:40.276 INFO:tasks.rados.rados.0.smithi076.stdout:2779: left oid 42 (ObjNum 963 snap 322 seq_num 963) 2021-03-15T22:31:40.276 INFO:tasks.rados.rados.0.smithi076.stdout:2781: finishing write tid 1 to smithi07694878-11 2021-03-15T22:31:40.277 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:40.279 INFO:tasks.rados.rados.0.smithi076.stdout:2781: finishing write tid 2 to smithi07694878-11 2021-03-15T22:31:40.279 INFO:tasks.rados.rados.0.smithi076.stdout:2781: finishing write tid 3 to smithi07694878-11 2021-03-15T22:31:40.279 INFO:tasks.rados.rados.0.smithi076.stdout:2781: finishing write tid 4 to smithi07694878-11 2021-03-15T22:31:40.279 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1623 (ObjNum 964 snap 322 seq_num 964) dirty exists 2021-03-15T22:31:40.280 INFO:tasks.rados.rados.0.smithi076.stdout:2781: left oid 11 (ObjNum 964 snap 322 seq_num 964) 2021-03-15T22:31:40.280 INFO:tasks.rados.rados.0.smithi076.stdout:2779: done (3 left) 2021-03-15T22:31:40.281 INFO:tasks.rados.rados.0.smithi076.stdout:2780: done (2 left) 2021-03-15T22:31:40.281 INFO:tasks.rados.rados.0.smithi076.stdout:2781: done (1 left) 2021-03-15T22:31:40.281 INFO:tasks.rados.rados.0.smithi076.stdout:2782: done (0 left) 2021-03-15T22:31:40.281 INFO:tasks.rados.rados.0.smithi076.stdout:2783: copy_from oid 16 from oid 19 current snap is 322 2021-03-15T22:31:40.282 INFO:tasks.rados.rados.0.smithi076.stdout:2784: read oid 12 snap -1 2021-03-15T22:31:40.282 INFO:tasks.rados.rados.0.smithi076.stdout:2784: expect deleted 2021-03-15T22:31:40.282 INFO:tasks.rados.rados.0.smithi076.stdout:2785: delete oid 48 current snap is 322 2021-03-15T22:31:40.283 INFO:tasks.rados.rados.0.smithi076.stdout:2784: done (2 left) 2021-03-15T22:31:40.283 INFO:tasks.rados.rados.0.smithi076.stdout:2785: done (1 left) 2021-03-15T22:31:40.284 INFO:tasks.rados.rados.0.smithi076.stdout:2786: rollback oid 27 current snap is 322 2021-03-15T22:31:40.284 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 320 2021-03-15T22:31:40.284 INFO:tasks.rados.rados.0.smithi076.stdout:2787: read oid 45 snap -1 2021-03-15T22:31:40.284 INFO:tasks.rados.rados.0.smithi076.stdout:2787: expect deleted 2021-03-15T22:31:40.285 INFO:tasks.rados.rados.0.smithi076.stdout:2788: delete oid 35 current snap is 322 2021-03-15T22:31:40.289 INFO:tasks.rados.rados.0.smithi076.stdout:2786: finishing rollback tid 0 to smithi07694878-27 2021-03-15T22:31:40.289 INFO:tasks.rados.rados.0.smithi076.stdout:2786: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:31:40.290 INFO:tasks.rados.rados.0.smithi076.stdout:2786: finishing rollback tid 2 to smithi07694878-27 2021-03-15T22:31:40.290 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1768 (ObjNum 896 snap 302 seq_num 896) dirty exists 2021-03-15T22:31:40.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:40.292 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:40.293 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:40.334 INFO:tasks.rados.rados.0.smithi076.stdout:2786: done (3 left) 2021-03-15T22:31:40.335 INFO:tasks.rados.rados.0.smithi076.stdout:2787: done (2 left) 2021-03-15T22:31:40.335 INFO:tasks.rados.rados.0.smithi076.stdout:2788: done (1 left) 2021-03-15T22:31:40.335 INFO:tasks.rados.rados.0.smithi076.stdout:2789: write oid 13 current snap is 322 2021-03-15T22:31:40.335 INFO:tasks.rados.rados.0.smithi076.stdout:2789: seq_num 968 ranges {669176=417863,1811746=457430} 2021-03-15T22:31:40.341 INFO:tasks.rados.rados.0.smithi076.stdout:2789: writing smithi07694878-13 from 669176 to 1087039 tid 1 2021-03-15T22:31:40.349 INFO:tasks.rados.rados.0.smithi076.stdout:2789: writing smithi07694878-13 from 1811746 to 2269176 tid 2 2021-03-15T22:31:40.349 INFO:tasks.rados.rados.0.smithi076.stdout:2790: write (excl) oid 36 current snap is 322 2021-03-15T22:31:40.349 INFO:tasks.rados.rados.0.smithi076.stdout:2783: finishing copy_from to smithi07694878-16 2021-03-15T22:31:40.349 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 1611 (ObjNum 863 snap 285 seq_num 863) dirty exists 2021-03-15T22:31:40.350 INFO:tasks.rados.rados.0.smithi076.stdout:2790: seq_num 969 ranges {685983=493112,1870228=415755} 2021-03-15T22:31:40.356 INFO:tasks.rados.rados.0.smithi076.stdout:2790: writing smithi07694878-36 from 685983 to 1179095 tid 1 2021-03-15T22:31:40.363 INFO:tasks.rados.rados.0.smithi076.stdout:2790: writing smithi07694878-36 from 1870228 to 2285983 tid 2 2021-03-15T22:31:40.363 INFO:tasks.rados.rados.0.smithi076.stdout:2791: delete oid 8 current snap is 322 2021-03-15T22:31:40.363 INFO:tasks.rados.rados.0.smithi076.stdout:2783: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:31:40.394 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:40.402 INFO:tasks.rados.rados.0.smithi076.stdout:2783: done (3 left) 2021-03-15T22:31:40.402 INFO:tasks.rados.rados.0.smithi076.stdout:2791: done (2 left) 2021-03-15T22:31:40.402 INFO:tasks.rados.rados.0.smithi076.stdout:2792: write (excl) oid 15 current snap is 322 2021-03-15T22:31:40.403 INFO:tasks.rados.rados.0.smithi076.stdout:2792: seq_num 971 ranges {719597=643610,1870839=448758} 2021-03-15T22:31:40.406 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:31:40.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[56514]: cluster 2021-03-15T22:31: 2021-03-15T22:31:40.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[56514]: 39.163979+0000 mgr.y (mgr.14144) 898 : cluster [DBG] pgmap v1267: 148 pgs: 1 active+clean+snaptrim, 3 active+remapped+backfill_toofull, 144 active+clean; 638 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 829 KiB/s wr, 12 op/s; 4/1878 objects misplaced (0.213%) 2021-03-15T22:31:40.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[56514]: cluster 2021-03-15T22:31:39.312886+0000 mon.a (mon.0) 1980 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2021-03-15T22:31:40.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[56514]: cluster 2021-03-15T22:31:39.517043+0000 osd.7 (osd.7) 140 : cluster [DBG] 3.e scrub ok 2021-03-15T22:31:40.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[56514]: cluster 2021-03-15T22:31:39.555483+0000 osd.3 (osd.3) 135 : cluster [DBG] 3.2 deep-scrub ok 2021-03-15T22:31:40.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[56514]: audit 2021-03-15T22:31:40.072668+0000 mgr.y (mgr.14144) 899 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:40.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[56514]: cluster 2021-03-15T22:31:40.233808+0000 mon.a (mon.0) 1981 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:31:40.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[56514]: cluster 2021-03-15T22:31:40.238609+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2021-03-15T22:31:40.412 INFO:tasks.rados.rados.0.smithi076.stdout:2792: writing smithi07694878-15 from 719597 to 1363207 tid 1 2021-03-15T22:31:40.413 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: cluster 2021-03-15T22:31:40.413 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: 2021-03-15T22:31:40.413 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: 2021 2021-03-15T22:31:40.413 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: - 2021-03-15T22:31:40.413 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: 03-15T22:31:39.163979 2021-03-15T22:31:40.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 898 : cluster [DBG] pgmap v1267: 148 pgs: 1 active+clean+snaptrim, 3 active+remapped+backfill_toofull, 144 active+clean; 638 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 829 KiB/s wr, 12 op/s; 4/1878 objects misplaced (0.213%) 2021-03-15T22:31:40.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:31:40.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: -15T22:31:39.312886+0000 mon.a (mon.0) 2021-03-15T22:31:40.414 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: 1980 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2021-03-15T22:31:40.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:31:40.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: -15T22:31:39.517043+0000 osd.7 (osd.7 2021-03-15T22:31:40.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: ) 140 : cluster [DBG] 3.e scrub ok 2021-03-15T22:31:40.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:31:40.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: 03-15T22:31:39.555483+0000 osd.3 ( 2021-03-15T22:31:40.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: osd.3) 135 : cluster [DBG] 3.2 deep-scrub ok 2021-03-15T22:31:40.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:31:40.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: 03-15T22:31:40.072668+0000 mgr.y (mgr. 2021-03-15T22:31:40.417 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: 14144) 899 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:40.417 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:31:40.417 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: -15T22:31:40.233808+0000 mon.a (mon.0 2021-03-15T22:31:40.417 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: ) 1981 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:31:40.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:31:40.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: 03-15T22:31:40.238609+0000 mon.a ( 2021-03-15T22:31:40.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:40 smithi076 conmon[62156]: mon.0) 1982 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2021-03-15T22:31:40.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:40.420 INFO:tasks.rados.rados.0.smithi076.stdout:2792: writing smithi07694878-15 from 1870839 to 2319597 tid 2 2021-03-15T22:31:40.420 INFO:tasks.rados.rados.0.smithi076.stdout:2793: write (excl) oid 50 current snap is 322 2021-03-15T22:31:40.420 INFO:tasks.rados.rados.0.smithi076.stdout:2793: seq_num 972 ranges {736404=718859,1929321=407083} 2021-03-15T22:31:40.421 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:40.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:40.426 INFO:tasks.rados.rados.0.smithi076.stdout:2793: writing smithi07694878-50 from 736404 to 1455263 tid 1 2021-03-15T22:31:40.432 INFO:tasks.rados.rados.0.smithi076.stdout:2793: writing smithi07694878-50 from 1929321 to 2336404 tid 2 2021-03-15T22:31:40.432 INFO:tasks.rados.rados.0.smithi076.stdout:2794: write (excl) oid 28 current snap is 322 2021-03-15T22:31:40.433 INFO:tasks.rados.rados.0.smithi076.stdout:2794: seq_num 973 ranges {753211=794108,1987803=365408} 2021-03-15T22:31:40.442 INFO:tasks.rados.rados.0.smithi076.stdout:2794: writing smithi07694878-28 from 753211 to 1547319 tid 1 2021-03-15T22:31:40.448 INFO:tasks.rados.rados.0.smithi076.stdout:2794: writing smithi07694878-28 from 1987803 to 2353211 tid 2 2021-03-15T22:31:40.448 INFO:tasks.rados.rados.0.smithi076.stdout:2795: read oid 26 snap -1 2021-03-15T22:31:40.449 INFO:tasks.rados.rados.0.smithi076.stdout:2795: expect (ObjNum 904 snap 304 seq_num 904) 2021-03-15T22:31:40.449 INFO:tasks.rados.rados.0.smithi076.stdout:2796: snap_create 2021-03-15T22:31:40.450 INFO:tasks.rados.rados.0.smithi076.stdout:2789: finishing write tid 1 to smithi07694878-13 2021-03-15T22:31:40.450 INFO:tasks.rados.rados.0.smithi076.stdout:2789: finishing write tid 2 to smithi07694878-13 2021-03-15T22:31:40.451 INFO:tasks.rados.rados.0.smithi076.stdout:2789: finishing write tid 3 to smithi07694878-13 2021-03-15T22:31:40.451 INFO:tasks.rados.rados.0.smithi076.stdout:2789: finishing write tid 4 to smithi07694878-13 2021-03-15T22:31:40.452 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1626 (ObjNum 968 snap 322 seq_num 968) dirty exists 2021-03-15T22:31:40.452 INFO:tasks.rados.rados.0.smithi076.stdout:2789: left oid 13 (ObjNum 968 snap 322 seq_num 968) 2021-03-15T22:31:40.453 INFO:tasks.rados.rados.0.smithi076.stdout:2790: finishing write tid 1 to smithi07694878-36 2021-03-15T22:31:40.453 INFO:tasks.rados.rados.0.smithi076.stdout:2790: finishing write tid 2 to smithi07694878-36 2021-03-15T22:31:40.453 INFO:tasks.rados.rados.0.smithi076.stdout:2790: finishing write tid 3 to smithi07694878-36 2021-03-15T22:31:40.454 INFO:tasks.rados.rados.0.smithi076.stdout:2790: finishing write tid 4 to smithi07694878-36 2021-03-15T22:31:40.455 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 1629 (ObjNum 969 snap 322 seq_num 969) dirty exists 2021-03-15T22:31:40.455 INFO:tasks.rados.rados.0.smithi076.stdout:2790: left oid 36 (ObjNum 969 snap 322 seq_num 969) 2021-03-15T22:31:40.455 INFO:tasks.rados.rados.0.smithi076.stdout:2792: finishing write tid 1 to smithi07694878-15 2021-03-15T22:31:40.456 INFO:tasks.rados.rados.0.smithi076.stdout:2792: finishing write tid 2 to smithi07694878-15 2021-03-15T22:31:40.456 INFO:tasks.rados.rados.0.smithi076.stdout:2792: finishing write tid 3 to smithi07694878-15 2021-03-15T22:31:40.457 INFO:tasks.rados.rados.0.smithi076.stdout:2792: finishing write tid 4 to smithi07694878-15 2021-03-15T22:31:40.457 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 2196 (ObjNum 971 snap 322 seq_num 971) dirty exists 2021-03-15T22:31:40.458 INFO:tasks.rados.rados.0.smithi076.stdout:2792: left oid 15 (ObjNum 971 snap 322 seq_num 971) 2021-03-15T22:31:40.459 INFO:tasks.rados.rados.0.smithi076.stdout:2793: finishing write tid 1 to smithi07694878-50 2021-03-15T22:31:40.459 INFO:tasks.rados.rados.0.smithi076.stdout:2793: finishing write tid 2 to smithi07694878-50 2021-03-15T22:31:40.459 INFO:tasks.rados.rados.0.smithi076.stdout:2793: finishing write tid 3 to smithi07694878-50 2021-03-15T22:31:40.460 INFO:tasks.rados.rados.0.smithi076.stdout:2793: finishing write tid 4 to smithi07694878-50 2021-03-15T22:31:40.460 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 1582 (ObjNum 972 snap 322 seq_num 972) dirty exists 2021-03-15T22:31:40.461 INFO:tasks.rados.rados.0.smithi076.stdout:2793: left oid 50 (ObjNum 972 snap 322 seq_num 972) 2021-03-15T22:31:40.462 INFO:tasks.rados.rados.0.smithi076.stdout:2794: finishing write tid 1 to smithi07694878-28 2021-03-15T22:31:40.462 INFO:tasks.rados.rados.0.smithi076.stdout:2795: expect (ObjNum 904 snap 304 seq_num 904) 2021-03-15T22:31:40.522 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:40.546 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:40.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:40.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:40.649 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:40.671 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:40.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:40.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:40.715 INFO:tasks.rados.rados.0.smithi076.stdout:2794: finishing write tid 2 to smithi07694878-28 2021-03-15T22:31:40.715 INFO:tasks.rados.rados.0.smithi076.stdout:2794: finishing write tid 3 to smithi07694878-28 2021-03-15T22:31:40.715 INFO:tasks.rados.rados.0.smithi076.stdout:2794: finishing write tid 4 to smithi07694878-28 2021-03-15T22:31:40.715 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2364 (ObjNum 973 snap 322 seq_num 973) dirty exists 2021-03-15T22:31:40.716 INFO:tasks.rados.rados.0.smithi076.stdout:2794: left oid 28 (ObjNum 973 snap 322 seq_num 973) 2021-03-15T22:31:40.774 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:40.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: cluster 2021-03-15T22:31:39.163979+0000 mgr.y (mgr.14144) 898 : cluster 2021-03-15T22:31:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: [DBG] pgmap v1267: 148 pgs: 1 active+clean+snaptrim, 3 active+remapped+backfill_toofull, 144 active+clean; 638 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 829 KiB/s wr, 12 op/s; 4/1878 objects misplaced (0.213%) 2021-03-15T22:31:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: cluster 2021-03-15T22:31:39.312886+0000 mon.a (mon.0) 1980 : cluster [DBG] osdmap e749: 8 total, 8 up, 7 in 2021-03-15T22:31:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:31:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: -15T22:31:39.517043+0000 osd.7 (osd.7) 140 : cluster [DBG] 3.e scrub ok 2021-03-15T22:31:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:31:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: -03-15T22:31:39.555483+0000 osd.3 (osd.3) 135 : cluster [DBG] 3.2 deep-scrub ok 2021-03-15T22:31:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: audit 2021-03-15T22:31 2021-03-15T22:31:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: :40.072668+0000 mgr.y (mgr.14144) 899 2021-03-15T22:31:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:31:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: 31:40.233808+0000 mon.a (mon.0) 1981 : cluster [WRN] 2021-03-15T22:31:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 3 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:31:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:31:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:40 smithi125 conmon[56739]: 22:31:40.238609+0000 mon.a (mon.0) 1982 : cluster [DBG] osdmap e750: 8 total, 8 up, 7 in 2021-03-15T22:31:40.797 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:40.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:40.798 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:40.899 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:40.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:40.923 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:40.924 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:41.025 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:41.048 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:41.049 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:41.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:41.173 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:41.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:41.175 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:41.276 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:41.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:41.299 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:41.299 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:41.400 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:41.418 INFO:tasks.rados.rados.0.smithi076.stdout:2789: done (6 left) 2021-03-15T22:31:41.418 INFO:tasks.rados.rados.0.smithi076.stdout:2790: done (5 left) 2021-03-15T22:31:41.419 INFO:tasks.rados.rados.0.smithi076.stdout:2792: done (4 left) 2021-03-15T22:31:41.419 INFO:tasks.rados.rados.0.smithi076.stdout:2793: done (3 left) 2021-03-15T22:31:41.419 INFO:tasks.rados.rados.0.smithi076.stdout:2794: done (2 left) 2021-03-15T22:31:41.419 INFO:tasks.rados.rados.0.smithi076.stdout:2795: done (1 left) 2021-03-15T22:31:41.419 INFO:tasks.rados.rados.0.smithi076.stdout:2796: done (0 left) 2021-03-15T22:31:41.420 INFO:tasks.rados.rados.0.smithi076.stdout:2797: rollback oid 24 current snap is 323 2021-03-15T22:31:41.420 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 300 2021-03-15T22:31:41.420 INFO:tasks.rados.rados.0.smithi076.stdout:2798: read oid 50 snap -1 2021-03-15T22:31:41.420 INFO:tasks.rados.rados.0.smithi076.stdout:2798: expect (ObjNum 972 snap 322 seq_num 972) 2021-03-15T22:31:41.421 INFO:tasks.rados.rados.0.smithi076.stdout:2799: copy_from oid 48 from oid 4 current snap is 323 2021-03-15T22:31:41.421 INFO:tasks.rados.rados.0.smithi076.stdout:2800: rollback oid 37 current snap is 323 2021-03-15T22:31:41.421 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 37 to 307 2021-03-15T22:31:41.421 INFO:tasks.rados.rados.0.smithi076.stdout:2801: delete oid 3 current snap is 323 2021-03-15T22:31:41.422 INFO:tasks.rados.rados.0.smithi076.stdout:2797: finishing rollback tid 0 to smithi07694878-24 2021-03-15T22:31:41.426 INFO:tasks.rados.rados.0.smithi076.stdout:2798: expect (ObjNum 972 snap 322 seq_num 972) 2021-03-15T22:31:41.428 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:41.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:41.429 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:41.519 INFO:tasks.rados.rados.0.smithi076.stdout:2799: finishing copy_from to smithi07694878-48 2021-03-15T22:31:41.844 INFO:tasks.rados.rados.0.smithi076.stdout:2799: got expected ENOENT (src dne) 2021-03-15T22:31:41.845 INFO:tasks.rados.rados.0.smithi076.stdout:2799: finishing copy_from racing read to smithi07694878-48 2021-03-15T22:31:41.846 INFO:tasks.rados.rados.0.smithi076.stdout:2797: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:31:41.846 INFO:tasks.rados.rados.0.smithi076.stdout:2797: finishing rollback tid 2 to smithi07694878-24 2021-03-15T22:31:41.847 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1727 (ObjNum 864 snap 285 seq_num 864) dirty exists 2021-03-15T22:31:41.847 INFO:tasks.rados.rados.0.smithi076.stdout:2800: finishing rollback tid 1 to smithi07694878-37 2021-03-15T22:31:41.848 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 2149 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:31:41.848 INFO:tasks.rados.rados.0.smithi076.stdout:2797: done (4 left) 2021-03-15T22:31:41.849 INFO:tasks.rados.rados.0.smithi076.stdout:2798: done (3 left) 2021-03-15T22:31:41.849 INFO:tasks.rados.rados.0.smithi076.stdout:2799: done (2 left) 2021-03-15T22:31:41.850 INFO:tasks.rados.rados.0.smithi076.stdout:2800: done (1 left) 2021-03-15T22:31:41.850 INFO:tasks.rados.rados.0.smithi076.stdout:2801: done (0 left) 2021-03-15T22:31:41.851 INFO:tasks.rados.rados.0.smithi076.stdout:2802: rollback oid 39 current snap is 323 2021-03-15T22:31:41.852 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 39 to 292 2021-03-15T22:31:41.852 INFO:tasks.rados.rados.0.smithi076.stdout:2803: delete oid 30 current snap is 323 2021-03-15T22:31:41.853 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:41.856 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:41 smithi125 conmon[56739]: cluster 2021-03-15T22:31:40.359696+0000 osd.0 (osd.0) 121 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:31:41.858 INFO:tasks.rados.rados.0.smithi076.stdout:2802: finishing rollback tid 1 to smithi07694878-39 2021-03-15T22:31:41.859 INFO:tasks.rados.rados.0.smithi076.stdout:2802: finishing rollback tid 2 to smithi07694878-39 2021-03-15T22:31:41.859 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2092 (ObjNum 829 snap 275 seq_num 829) dirty exists 2021-03-15T22:31:41.859 INFO:tasks.rados.rados.0.smithi076.stdout:2802: done (1 left) 2021-03-15T22:31:41.859 INFO:tasks.rados.rados.0.smithi076.stdout:2803: done (0 left) 2021-03-15T22:31:41.859 INFO:tasks.rados.rados.0.smithi076.stdout:2804: delete oid 30 current snap is 323 2021-03-15T22:31:41.860 INFO:tasks.rados.rados.0.smithi076.stdout:2804: done (0 left) 2021-03-15T22:31:41.860 INFO:tasks.rados.rados.0.smithi076.stdout:2805: write (excl) oid 37 current snap is 323 2021-03-15T22:31:41.860 INFO:tasks.rados.rados.0.smithi076.stdout:2805: seq_num 977 ranges {420439=411457,1421731=480010} 2021-03-15T22:31:41.860 INFO:tasks.rados.rados.0.smithi076.stdout:2805: writing smithi07694878-37 from 420439 to 831896 tid 1 2021-03-15T22:31:41.861 INFO:tasks.rados.rados.0.smithi076.stdout:2805: writing smithi07694878-37 from 1421731 to 1901741 tid 2 2021-03-15T22:31:41.861 INFO:tasks.rados.rados.0.smithi076.stdout:2806: write (excl) oid 4 current snap is 323 2021-03-15T22:31:41.861 INFO:tasks.rados.rados.0.smithi076.stdout:2805: finishing write tid 1 to smithi07694878-37 2021-03-15T22:31:41.861 INFO:tasks.rados.rados.0.smithi076.stdout:2806: seq_num 978 ranges {437246=486706,1363860=623668} 2021-03-15T22:31:41.861 INFO:tasks.rados.rados.0.smithi076.stdout:2806: writing smithi07694878-4 from 437246 to 923952 tid 1 2021-03-15T22:31:41.862 INFO:tasks.rados.rados.0.smithi076.stdout:2806: writing smithi07694878-4 from 1363860 to 1987528 tid 2 2021-03-15T22:31:41.862 INFO:tasks.rados.rados.0.smithi076.stdout:2807: snap_create 2021-03-15T22:31:41.862 INFO:tasks.rados.rados.0.smithi076.stdout:2805: finishing write tid 2 to smithi07694878-37 2021-03-15T22:31:41.862 INFO:tasks.rados.rados.0.smithi076.stdout:2805: finishing write tid 3 to smithi07694878-37 2021-03-15T22:31:41.863 INFO:tasks.rados.rados.0.smithi076.stdout:2805: finishing write tid 4 to smithi07694878-37 2021-03-15T22:31:41.863 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 2368 (ObjNum 977 snap 323 seq_num 977) dirty exists 2021-03-15T22:31:41.863 INFO:tasks.rados.rados.0.smithi076.stdout:2805: left oid 37 (ObjNum 977 snap 323 seq_num 977) 2021-03-15T22:31:41.863 INFO:tasks.rados.rados.0.smithi076.stdout:2806: finishing write tid 1 to smithi07694878-4 2021-03-15T22:31:41.863 INFO:tasks.rados.rados.0.smithi076.stdout:2806: finishing write tid 2 to smithi07694878-4 2021-03-15T22:31:41.864 INFO:tasks.rados.rados.0.smithi076.stdout:2806: finishing write tid 3 to smithi07694878-4 2021-03-15T22:31:41.864 INFO:tasks.rados.rados.0.smithi076.stdout:2806: finishing write tid 4 to smithi07694878-4 2021-03-15T22:31:41.864 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 1632 (ObjNum 978 snap 323 seq_num 978) dirty exists 2021-03-15T22:31:41.864 INFO:tasks.rados.rados.0.smithi076.stdout:2806: left oid 4 (ObjNum 978 snap 323 seq_num 978) 2021-03-15T22:31:41.865 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:41 smithi076 conmon[56514]: cluster 2021-03-15T22:31:40.359696+0000 osd.0 (osd.0) 121 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:31:41.865 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:41 smithi076 conmon[62156]: cluster 2021-03-15T22:31:40.359696+0000 2021-03-15T22:31:41.865 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:41 smithi076 conmon[62156]: osd.0 (osd.0) 121 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:31:41.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:31:41.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:41.932 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:41.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:42.034 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:42.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:42.064 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:42.065 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:42.166 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:42.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:42.193 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:42.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:42.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:42.318 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:31:42.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:42.326 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:42.326 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:42.397 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:42.424 INFO:tasks.rados.rados.0.smithi076.stdout:2805: done (2 left) 2021-03-15T22:31:42.424 INFO:tasks.rados.rados.0.smithi076.stdout:2806: done (1 left) 2021-03-15T22:31:42.424 INFO:tasks.rados.rados.0.smithi076.stdout:2807: done (0 left) 2021-03-15T22:31:42.425 INFO:tasks.rados.rados.0.smithi076.stdout:2808: delete oid 30 current snap is 324 2021-03-15T22:31:42.427 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:42.430 INFO:tasks.rados.rados.0.smithi076.stdout:2808: done (0 left) 2021-03-15T22:31:42.430 INFO:tasks.rados.rados.0.smithi076.stdout:2809: copy_from oid 50 from oid 48 current snap is 324 2021-03-15T22:31:42.430 INFO:tasks.rados.rados.0.smithi076.stdout:2810: snap_create 2021-03-15T22:31:42.432 INFO:tasks.rados.rados.0.smithi076.stdout:2809: finishing copy_from to smithi07694878-50 2021-03-15T22:31:42.433 INFO:tasks.rados.rados.0.smithi076.stdout:2809: got expected ENOENT (src dne) 2021-03-15T22:31:42.433 INFO:tasks.rados.rados.0.smithi076.stdout:2809: finishing copy_from racing read to smithi07694878-50 2021-03-15T22:31:42.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:42.454 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:42.455 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:42.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[56514]: cluster 2021-03-15T22:31:40.870397+0000 osd.1 2021-03-15T22:31:42.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[56514]: (osd.1) 144 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:31:42.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[56514]: cluster 2021-03-15T22:31:41.164648+0000 mgr.y (mgr.14144) 900 : cluster [DBG] pgmap v1270: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 635 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 16 op/s; 4/1875 objects misplaced (0.213%) 2021-03-15T22:31:42.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[56514]: cluster 2021-03-15T22:31:41.407319+0000 osd.4 (osd.4) 161 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:31:42.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[56514]: cluster 2021-03-15T22:31:41.412850+0000 mon.a (mon.0) 1983 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2021-03-15T22:31:42.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[56514]: cluster 2021-03-15T22:31:41.551429+0000 osd.3 (osd.3) 136 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:31:42.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[62156]: cluster 2021-03-15T22:31:40. 2021-03-15T22:31:42.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[62156]: 870397+0000 osd.1 (osd.1) 144 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:31:42.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[62156]: cluster 2021-03-15T22:31:41.164648+0000 mgr.y (mgr.14144) 900 : cluster [DBG] pgmap v1270: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 635 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 16 op/s; 4/1875 objects misplaced (0.213%) 2021-03-15T22:31:42.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[62156]: cluster 2021-03-15T22:31 2021-03-15T22:31:42.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[62156]: :41.407319+0000 osd.4 (osd.4) 161 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:31:42.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:31:42.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[62156]: :31:41.412850+0000 mon.a (mon.0 2021-03-15T22:31:42.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[62156]: ) 1983 : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2021-03-15T22:31:42.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:31:42.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[62156]: -15T22:31:41.551429+0000 osd.3 ( 2021-03-15T22:31:42.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:42 smithi076 conmon[62156]: osd.3) 136 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:31:42.556 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:42.582 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:42.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:42.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:42.685 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:42.712 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:42.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:42.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:42.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:42 smithi125 conmon[56739]: cluster 2021-03-15T22:31:40.870397+0000 osd.1 (osd.1) 144 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:31:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:42 smithi125 conmon[56739]: cluster 2021-03-15T22:31:41. 2021-03-15T22:31:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:42 smithi125 conmon[56739]: 164648+0000 mgr.y (mgr.14144) 900 : cluster [DBG] pgmap v1270: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 635 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 16 op/s; 4/1875 objects misplaced (0.213%) 2021-03-15T22:31:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:42 smithi125 conmon[56739]: cluster 2021-03-15T22:31:41.407319+0000 osd.4 (osd.4) 161 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:31:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:42 smithi125 conmon[56739]: cluster 2021-03-15T22:31: 2021-03-15T22:31:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:42 smithi125 conmon[56739]: 41.412850+0000 mon.a (mon.0) 1983 2021-03-15T22:31:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:42 smithi125 conmon[56739]: : cluster [DBG] osdmap e751: 8 total, 8 up, 7 in 2021-03-15T22:31:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:42 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:31:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:42 smithi125 conmon[56739]: -03-15T22:31:41.551429 2021-03-15T22:31:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:42 smithi125 conmon[56739]: +0000 osd.3 (osd.3) 136 : cluster [DBG] 2021-03-15T22:31:42.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:42 smithi125 conmon[56739]: 5.14 scrub ok 2021-03-15T22:31:42.814 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:42.818 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:42.840 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:42.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:42.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:42.943 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:42.969 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:42.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:42.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:43.072 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:43.095 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:43.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:43.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:43.198 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:43.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:43.226 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:43.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:43.327 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:43.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:43.352 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:43.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:43.433 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:31:43.435 INFO:tasks.rados.rados.0.smithi076.stdout:2809: done (1 left) 2021-03-15T22:31:43.435 INFO:tasks.rados.rados.0.smithi076.stdout:2810: done (0 left) 2021-03-15T22:31:43.435 INFO:tasks.rados.rados.0.smithi076.stdout:2811: snap_remove snap 304 2021-03-15T22:31:43.437 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 6 pgp_num to 32 2021-03-15T22:31:43.453 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:43.476 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:43.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:43.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:43.578 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:43.604 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:43.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:43.606 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:43.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:43.733 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:43.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:43.735 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:43.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[56514]: cluster 2021-03-15T22:31:41.775521+0000 osd.5 (osd.5) 24 : cluster [DBG] 3.c scrub ok 2021-03-15T22:31:43.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[56514]: cluster 2021-03-15T22:31:42.055636+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:31:43.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[56514]: cluster 2021-03-15T22:31:42.396991+0000 osd.4 (osd.4) 162 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:31:43.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[56514]: cluster 2021-03-15T22:31:42.417778+0000 mon.a (mon.0) 1984 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2021-03-15T22:31:43.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[56514]: audit 2021-03-15T22:31:42.974859+0000 mon.a (mon.0) 1985 : audit [INF] from='client.? 172.21.15.76:0/3378265153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:31:43.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[56514]: audit 2021-03-15T22:31:43.377602+0000 mon.a (mon.0) 1986 : audit [INF] from='client.? 172.21.15.76:0/2732999353' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:31:43.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: cluster 2021-03-15T22:31:43.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: 2021-03-15T22:31:43.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: 2021 2021-03-15T22:31:43.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: -03-15T22:31:41. 2021-03-15T22:31:43.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: 775521+0000 osd.5 (osd.5) 24 : cluster [DBG] 3.c scrub ok 2021-03-15T22:31:43.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: cluster 2021-03-15T22:31:43.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:31:43.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: :31:42.055636+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:31:43.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: cluster 2021-03-15T22:31:43.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:31:43.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: 15T22:31:42.396991+0000 osd.4 (osd.4) 162 : cluster 2021-03-15T22:31:43.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: [DBG] 4.13 scrub ok 2021-03-15T22:31:43.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: cluster 2021-03-15T22:31:43.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: 2021-03-15T22:31:42.417778+0000 2021-03-15T22:31:43.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: mon.a (mon.0) 1984 : cluster [DBG] 2021-03-15T22:31:43.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: osdmap e752: 8 total, 8 up, 7 in 2021-03-15T22:31:43.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:31:43.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: 31:42.974859+0000 mon.a (mon.0) 2021-03-15T22:31:43.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: 1985 : audit [INF] from='client.? 172.21.15.76:0/3378265153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:31:43.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:31:43.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: 22:31:43.377602+0000 mon.a (mon. 2021-03-15T22:31:43.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:43 smithi076 conmon[62156]: 0) 1986 : audit [INF] from='client.? 172.21.15.76:0/2732999353' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:31:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: cluster 2021-03-15T22:31:41.775521+0000 osd.5 (osd.5) 2021-03-15T22:31:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: 24 : cluster [DBG] 3.c scrub ok 2021-03-15T22:31:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: cluster 2021-03-15T22:31:42.055636+0000 osd.2 (osd.2) 19 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:31:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: 2021-03-15T22:31:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: cluster 2021-03-15T22:31:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: 2021-03-15T22:31:42.396991+0000 2021-03-15T22:31:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: osd.4 (osd.4) 162 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:31:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: 2021-03-15T22:31:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:31:43.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: 31:42.417778+0000 mon.a (mon.0) 2021-03-15T22:31:43.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: 1984 : cluster [DBG] osdmap e752: 8 total, 8 up, 7 in 2021-03-15T22:31:43.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:31:43.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: -15T22:31:42.974859+0000 2021-03-15T22:31:43.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: mon.a (mon.0) 1985 : audit [INF] from='client.? 172.21.15.76:0/3378265153' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:31:43.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: 2021-03-15T22:31:43.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: audit 2021-03-15T22:31:43.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: 2021-03-15T22:31:43.377602+0000 2021-03-15T22:31:43.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: mon.a (mon.0) 1986 : audit [INF] 2021-03-15T22:31:43.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:43 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2732999353' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:31:43.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:43.859 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:43.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:43.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:43.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:43.988 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:43.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:43.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:44.091 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:44.116 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:44.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:44.118 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:44.219 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:44.244 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:44.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:44.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:44.347 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:44.371 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:44.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:44.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:44.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[56514]: cluster 2021-03-15T22:31:43 2021-03-15T22:31:44.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[56514]: .165102+0000 mgr.y (mgr.14144) 901 : cluster [DBG] pgmap v1273: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 647 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 3.9 MiB/s wr, 26 op/s; 4/1902 objects misplaced (0.210%) 2021-03-15T22:31:44.441 INFO:tasks.rados.rados.0.smithi076.stdout:2811: done (0 left) 2021-03-15T22:31:44.441 INFO:tasks.rados.rados.0.smithi076.stdout:2812: read oid 9 snap 308 2021-03-15T22:31:44.441 INFO:tasks.rados.rados.0.smithi076.stdout:2812: expect deleted 2021-03-15T22:31:44.442 INFO:tasks.rados.rados.0.smithi076.stdout:2813: delete oid 28 current snap is 325 2021-03-15T22:31:44.450 INFO:tasks.rados.rados.0.smithi076.stdout:2812: done (1 left) 2021-03-15T22:31:44.450 INFO:tasks.rados.rados.0.smithi076.stdout:2813: done (0 left) 2021-03-15T22:31:44.450 INFO:tasks.rados.rados.0.smithi076.stdout:2814: delete oid 13 current snap is 325 2021-03-15T22:31:44.455 INFO:tasks.rados.rados.0.smithi076.stdout:2814: done (0 left) 2021-03-15T22:31:44.455 INFO:tasks.rados.rados.0.smithi076.stdout:2815: snap_remove snap 273 2021-03-15T22:31:44.474 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:44.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:44.500 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:44.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:44.601 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:44.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:44.625 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:44.625 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:44.700 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[56514]: cluster 2021-03-15T22:31:43.405595+0000 osd.0 (osd.0) 122 : cluster [DBG] 6.10 scrub ok 2021-03-15T22:31:44.700 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[56514]: cluster 2021-03-15T22:31:43.415905+0000 mon.a (mon.0) 1987 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:44.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[56514]: audit 2021-03-15T22:31:43.428940+0000 mon.a (mon.0) 1988 : audit [INF] from='client.? 172.21.15.76:0/3378265153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:31:44.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[56514]: audit 2021-03-15T22:31:43.428993+0000 mon.a (mon.0) 1989 : audit [INF] from='client.? 172.21.15.76:0/2732999353' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:31:44.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[56514]: cluster 2021-03-15T22:31:43.429046+0000 mon.a (mon.0) 1990 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2021-03-15T22:31:44.701 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[56514]: cluster 2021-03-15T22:31:43.522284+0000 osd.3 (osd.3) 137 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:44.702 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[56514]: cluster 2021-03-15T22:31:43.524728+0000 osd.3 (osd.3) 138 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:44.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: cluster 2021-03-15T22:31:43. 2021-03-15T22:31:44.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: 165102+0000 mgr.y (mgr.14144) 901 : cluster [DBG] pgmap v1273: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 647 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 3.9 MiB/s wr, 26 op/s; 4/1902 objects misplaced (0.210%) 2021-03-15T22:31:44.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: cluster 2021-03-15T22:31:44.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: 2021-03-15T22:31:43.405595+0000 osd.0 (osd.0) 122 : cluster [DBG] 2021-03-15T22:31:44.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: 6.10 scrub ok 2021-03-15T22:31:44.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: cluster 2021-03-15T22:31:43 2021-03-15T22:31:44.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: .415905+0000 mon.a (mon.0) 1987 : cluster [WRN] 2021-03-15T22:31:44.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:44.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: audit 2021-03-15T22:31 2021-03-15T22:31:44.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: :43.428940+0000 mon.a (mon.0) 1988 : 2021-03-15T22:31:44.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/3378265153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:31:44.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: audit 2021-03-15T22:31 2021-03-15T22:31:44.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: :43.428993+0000 mon.a (mon.0) 1989 2021-03-15T22:31:44.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/2732999353' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:31:44.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:31:44.705 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: 03-15T22:31:43.429046+0000 2021-03-15T22:31:44.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: mon.a (mon.0) 1990 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2021-03-15T22:31:44.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: 2021-03-15T22:31:44.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: cluster 2021-03-15T22:31:43 2021-03-15T22:31:44.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: .522284+0000 osd.3 (osd.3) 137 : cluster 2021-03-15T22:31:44.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: [DBG] purged_snaps scrub starts 2021-03-15T22:31:44.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:31:44.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: 31:43.524728+0000 osd.3 (osd.3) 2021-03-15T22:31:44.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:44 smithi076 conmon[62156]: 138 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:44.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:44.753 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:44.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:44.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:44.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: cluster 2021-03-15T22:31:43.165102+0000 mgr.y (mgr.14144) 901 : cluster [DBG] pgmap v1273: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 647 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 3.9 MiB/s wr, 26 op/s; 4/1902 objects misplaced (0.210%) 2021-03-15T22:31:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: cluster 2021-03-15T22:31:43.405595+0000 osd.0 (osd.0) 122 : cluster [DBG] 6.10 scrub ok 2021-03-15T22:31:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: cluster 2021-03-15T22:31:43.415905+0000 2021-03-15T22:31:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: mon.a (mon.0) 1987 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:31:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: 22:31:43.428940+0000 mon.a (mon. 2021-03-15T22:31:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: 0) 1988 : audit [INF] from='client.? 172.21.15.76:0/3378265153' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:31:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: audit 2021 2021-03-15T22:31:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: -03-15T22:31:43.428993 2021-03-15T22:31:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1989 : audit [INF] 2021-03-15T22:31:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2732999353' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:31:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:31:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: :31:43.429046+0000 mon.a (mon. 2021-03-15T22:31:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: 0) 1990 : cluster [DBG] osdmap e753: 8 total, 8 up, 7 in 2021-03-15T22:31:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:31:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: -03-15T22:31:43.522284+0000 2021-03-15T22:31:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: osd.3 (osd.3) 137 : cluster [DBG] 2021-03-15T22:31:44.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: purged_snaps scrub starts 2021-03-15T22:31:44.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:31:44.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: 31:43.524728+0000 osd.3 (osd.3 2021-03-15T22:31:44.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:44 smithi125 conmon[56739]: ) 138 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:44.856 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:44.878 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:44.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:44.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:44.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:45.008 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:45.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:45.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:45.110 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:45.133 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:45.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:45.134 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:45.235 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:45.255 INFO:tasks.rados.rados.0.smithi076.stdout:2815: done (0 left) 2021-03-15T22:31:45.255 INFO:tasks.rados.rados.0.smithi076.stdout:2816: snap_create 2021-03-15T22:31:45.268 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:45.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:45.270 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:45.371 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:45.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:45.396 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:45.397 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:45.498 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:45.520 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:45.521 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:45.522 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:45.623 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:45.645 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:45.647 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:45.708 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:45 smithi076 conmon[56514]: cluster 2021-03-15T22:31:44. 2021-03-15T22:31:45.709 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:45 smithi076 conmon[56514]: 434774+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2021-03-15T22:31:45.709 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:45 smithi076 conmon[56514]: cluster 2021-03-15T22:31:45.241792+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2021-03-15T22:31:45.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:45 smithi076 conmon[62156]: cluster 2021-03-15T22:31:44. 2021-03-15T22:31:45.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:45 smithi076 conmon[62156]: 434774+0000 mon.a (mon.0) 1991 : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2021-03-15T22:31:45.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:45 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:31:45.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:45 smithi076 conmon[62156]: 03-15T22:31:45.241792+0000 mon.a ( 2021-03-15T22:31:45.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:45 smithi076 conmon[62156]: mon.0) 1992 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2021-03-15T22:31:45.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:45.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:45.774 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:45.774 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:45.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:45 smithi125 conmon[56739]: cluster 2021-03-15T22:31:44.434774+0000 mon.a (mon.0) 1991 2021-03-15T22:31:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:45 smithi125 conmon[56739]: : cluster [DBG] osdmap e754: 8 total, 8 up, 7 in 2021-03-15T22:31:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:45 smithi125 conmon[56739]: cluster 2021-03-15T22:31:45.241792+0000 mon.a (mon.0) 1992 : cluster [DBG] osdmap e755: 8 total, 8 up, 7 in 2021-03-15T22:31:45.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:45.900 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:45.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:45.901 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:46.002 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:46.003 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:31:46.004 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:31:46.027 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:46.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:46.029 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:46.130 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:46.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:46.156 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:46.157 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:46.228 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:31:46.244 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:31:46.258 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:46.259 INFO:tasks.rados.rados.0.smithi076.stdout:2816: done (0 left) 2021-03-15T22:31:46.260 INFO:tasks.rados.rados.0.smithi076.stdout:2817: read oid 28 snap -1 2021-03-15T22:31:46.260 INFO:tasks.rados.rados.0.smithi076.stdout:2817: expect deleted 2021-03-15T22:31:46.261 INFO:tasks.rados.rados.0.smithi076.stdout:2818: write oid 21 current snap is 326 2021-03-15T22:31:46.261 INFO:tasks.rados.rados.0.smithi076.stdout:2818: seq_num 982 ranges {504474=504055,1714141=631006} 2021-03-15T22:31:46.265 INFO:tasks.rados.rados.0.smithi076.stdout:2818: writing smithi07694878-21 from 504474 to 1008529 tid 1 2021-03-15T22:31:46.269 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:31:46.274 INFO:tasks.rados.rados.0.smithi076.stdout:2818: writing smithi07694878-21 from 1714141 to 2345147 tid 2 2021-03-15T22:31:46.274 INFO:tasks.rados.rados.0.smithi076.stdout:2819: snap_remove snap 220 2021-03-15T22:31:46.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:46.288 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:46.289 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:46.290 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:31:46.310 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:31:46.328 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:31:46.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:31:46.367 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:31:46.368 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:31:46.368 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:31:46.368 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:31:46.368 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:31:46.369 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:31:46.369 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:31:46.369 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:31:46.369 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:31:46.390 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:46.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:46.416 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:46.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:46.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:46.543 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:46.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:46.546 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:46.647 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:46.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:46.671 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:46.671 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:46.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:46 smithi076 conmon[56514]: cluster 2021-03-15T22:31:45.165556+0000 mgr.y (mgr.14144 2021-03-15T22:31:46.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:46 smithi076 conmon[56514]: ) 902 : cluster [DBG] pgmap v1276: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 657 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 5.8 MiB/s rd, 3.0 MiB/s wr, 22 op/s; 4/1923 objects misplaced (0.208%) 2021-03-15T22:31:46.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:46 smithi076 conmon[56514]: audit 2021-03-15T22:31:46.208213+0000 mon.c (mon.1) 61 : audit [DBG] from='client.? 172.21.15.76:0/2113901296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:46.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:46 smithi076 conmon[56514]: cluster 2021-03-15T22:31:46.251965+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2021-03-15T22:31:46.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:46 smithi076 conmon[62156]: cluster 2021-03-15T22:31:45.165556+0000 2021-03-15T22:31:46.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:46 smithi076 conmon[62156]: mgr.y (mgr.14144) 902 : cluster [DBG] pgmap v1276: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 657 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 5.8 MiB/s rd, 3.0 MiB/s wr, 22 op/s; 4/1923 objects misplaced (0.208%) 2021-03-15T22:31:46.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:46 smithi076 conmon[62156]: audit 2021-03-15T22:31:46.208213+0000 mon.c (mon.1) 61 : audit [DBG] from='client.? 172.21.15.76:0/2113901296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:46.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:46 smithi076 conmon[62156]: cluster 2021-03-15T22:31:46.251965+0000 mon.a (mon.0) 1993 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2021-03-15T22:31:46.773 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:46.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:46 smithi125 conmon[56739]: cluster 2021-03-15T22:31:45.165556+0000 mgr.y (mgr.14144) 902 : cluster [DBG] pgmap v1276: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 657 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 5.8 MiB/s rd, 3.0 MiB/s wr, 22 op/s; 4/1923 objects misplaced (0.208%) 2021-03-15T22:31:46.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:46 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:31:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:46 smithi125 conmon[56739]: 15T22:31:46.208213+0000 mon.c (mon.1) 61 : audit [DBG] from='client.? 172.21.15.76:0/2113901296' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:46 smithi125 conmon[56739]: cluster 2021-03-15T22:31:46.251965+0000 mon.a (mon. 2021-03-15T22:31:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:46 smithi125 conmon[56739]: 0) 1993 : cluster [DBG] osdmap e756: 8 total, 8 up, 7 in 2021-03-15T22:31:46.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:46.799 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:46.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:46.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:46.926 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:46.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:46.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:46.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:31:47.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:47.054 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:47.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:47.056 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:47.157 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:47.181 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:47.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:47.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:47.262 INFO:tasks.rados.rados.0.smithi076.stdout:2819: done (2 left) 2021-03-15T22:31:47.263 INFO:tasks.rados.rados.0.smithi076.stdout:2820: snap_create 2021-03-15T22:31:47.263 INFO:tasks.rados.rados.0.smithi076.stdout:2818: finishing write tid 1 to smithi07694878-21 2021-03-15T22:31:47.263 INFO:tasks.rados.rados.0.smithi076.stdout:2818: finishing write tid 2 to smithi07694878-21 2021-03-15T22:31:47.264 INFO:tasks.rados.rados.0.smithi076.stdout:2818: finishing write tid 3 to smithi07694878-21 2021-03-15T22:31:47.264 INFO:tasks.rados.rados.0.smithi076.stdout:2818: finishing write tid 4 to smithi07694878-21 2021-03-15T22:31:47.264 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 2395 (ObjNum 982 snap 326 seq_num 982) dirty exists 2021-03-15T22:31:47.264 INFO:tasks.rados.rados.0.smithi076.stdout:2818: left oid 21 (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:31:47.285 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:47.309 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:47.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:47.310 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:47.411 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:47.436 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:47.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:47.437 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:47.440 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:47.538 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:47.560 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:47.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:47.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:47.663 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:47.685 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:47.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:47.687 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:47.712 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:31:47.788 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:47.810 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:47.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:47.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:47.913 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:47.939 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:47.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:47.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:48.042 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:48.066 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:48.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:48.067 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:48.168 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:48.195 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:48.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:48.196 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:48.258 INFO:tasks.rados.rados.0.smithi076.stdout:2817: done (2 left) 2021-03-15T22:31:48.259 INFO:tasks.rados.rados.0.smithi076.stdout:2818: done (1 left) 2021-03-15T22:31:48.259 INFO:tasks.rados.rados.0.smithi076.stdout:2820: done (0 left) 2021-03-15T22:31:48.259 INFO:tasks.rados.rados.0.smithi076.stdout:2821: snap_create 2021-03-15T22:31:48.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:48 smithi076 conmon[62156]: cluster 2021-03-15T22:31:47.166131+0000 mgr.y (mgr.14144) 903 : cluster [DBG] pgmap v1279: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 647 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 1.2 MiB/s wr, 11 op/s; 4/1908 objects misplaced (0.210%) 2021-03-15T22:31:48.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:48 smithi076 conmon[62156]: cluster 2021-03-15T22:31:47.254349+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2021-03-15T22:31:48.297 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:48.323 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:48.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:48.325 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:48.426 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:48.452 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:48.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:48.454 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:48 smithi076 conmon[56514]: cluster 2021-03-15T22:31:47.166131+0000 mgr.y (mgr.14144) 903 : cluster [DBG] pgmap v1279: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 647 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 1.2 MiB/s wr, 11 op/s; 4/1908 objects misplaced (0.210%) 2021-03-15T22:31:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:48 smithi076 conmon[56514]: cluster 2021-03-15T22:31:47.254349+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2021-03-15T22:31:48.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:48 smithi125 conmon[56739]: cluster 2021-03-15T22:31:47.166131+0000 mgr.y (mgr.14144) 903 : cluster [DBG] pgmap v1279: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 647 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 1.2 MiB/s wr, 11 op/s; 4/1908 objects misplaced (0.210%) 2021-03-15T22:31:48.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:48 smithi125 conmon[56739]: cluster 2021-03-15T22:31:47.254349+0000 mon.a (mon.0) 1994 : cluster [DBG] osdmap e757: 8 total, 8 up, 7 in 2021-03-15T22:31:48.554 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:48.567 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:48.578 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:48.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:48.580 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:48.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:48.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:48.709 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:48.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:48.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:48.834 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:48.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:48.836 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:48.937 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:48.963 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:48.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:48.966 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:49.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:49.093 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:49.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:49.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:49.196 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:49.219 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:49.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:49.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:49.273 INFO:tasks.rados.rados.0.smithi076.stdout:2821: done (0 left) 2021-03-15T22:31:49.273 INFO:tasks.rados.rados.0.smithi076.stdout:2822: copy_from oid 38 from oid 23 current snap is 328 2021-03-15T22:31:49.273 INFO:tasks.rados.rados.0.smithi076.stdout:2823: read oid 15 snap -1 2021-03-15T22:31:49.273 INFO:tasks.rados.rados.0.smithi076.stdout:2823: expect (ObjNum 971 snap 322 seq_num 971) 2021-03-15T22:31:49.274 INFO:tasks.rados.rados.0.smithi076.stdout:2824: read oid 44 snap -1 2021-03-15T22:31:49.274 INFO:tasks.rados.rados.0.smithi076.stdout:2824: expect (ObjNum 936 snap 316 seq_num 936) 2021-03-15T22:31:49.275 INFO:tasks.rados.rados.0.smithi076.stdout:2825: read oid 7 snap -1 2021-03-15T22:31:49.275 INFO:tasks.rados.rados.0.smithi076.stdout:2825: expect deleted 2021-03-15T22:31:49.275 INFO:tasks.rados.rados.0.smithi076.stdout:2826: read oid 27 snap -1 2021-03-15T22:31:49.275 INFO:tasks.rados.rados.0.smithi076.stdout:2826: expect (ObjNum 896 snap 302 seq_num 896) 2021-03-15T22:31:49.276 INFO:tasks.rados.rados.0.smithi076.stdout:2827: rollback oid 33 current snap is 328 2021-03-15T22:31:49.276 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 33 to 255 2021-03-15T22:31:49.276 INFO:tasks.rados.rados.0.smithi076.stdout:2828: snap_create 2021-03-15T22:31:49.280 INFO:tasks.rados.rados.0.smithi076.stdout:2823: expect (ObjNum 971 snap 322 seq_num 971) 2021-03-15T22:31:49.322 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:49.348 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:49.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:49.350 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:49.433 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:31:49.452 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:49.464 INFO:tasks.rados.rados.0.smithi076.stdout:2827: finishing rollback tid 0 to smithi07694878-33 2021-03-15T22:31:49.464 INFO:tasks.rados.rados.0.smithi076.stdout:2827: finishing rollback tid 1 to smithi07694878-33 2021-03-15T22:31:49.464 INFO:tasks.rados.rados.0.smithi076.stdout:2827: finishing rollback tid 2 to smithi07694878-33 2021-03-15T22:31:49.465 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 1766 (ObjNum 714 snap 231 seq_num 714) dirty exists 2021-03-15T22:31:49.465 INFO:tasks.rados.rados.0.smithi076.stdout:2824: expect (ObjNum 936 snap 316 seq_num 936) 2021-03-15T22:31:49.478 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:49.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:49.479 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:49.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:49 smithi125 conmon[56739]: cluster 2021-03-15T22:31:49.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:49 smithi125 conmon[56739]: 2021-03-15T22:31:48.253454 2021-03-15T22:31:49.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:49 smithi125 conmon[56739]: +0000 mon.a (mon.0) 1995 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2021-03-15T22:31:49.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:49.584 INFO:tasks.rados.rados.0.smithi076.stdout:2826: expect (ObjNum 896 snap 302 seq_num 896) 2021-03-15T22:31:49.604 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:49.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:49.605 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:49.689 INFO:tasks.rados.rados.0.smithi076.stdout:2822: finishing copy_from to smithi07694878-38 2021-03-15T22:31:49.689 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 1960 (ObjNum 955 snap 320 seq_num 955) dirty exists 2021-03-15T22:31:49.690 INFO:tasks.rados.rados.0.smithi076.stdout:2822: finishing copy_from racing read to smithi07694878-38 2021-03-15T22:31:49.706 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:49.728 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:49.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:49.730 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:49.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:49 smithi076 conmon[56514]: cluster 2021-03-15T22:31:48.253454+0000 mon.a (mon.0) 1995 : cluster [DBG] 2021-03-15T22:31:49.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:49 smithi076 conmon[56514]: osdmap e758: 8 total, 8 up, 7 in 2021-03-15T22:31:49.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:49 smithi076 conmon[62156]: cluster 2021-03-15T22:31:48.253454+0000 2021-03-15T22:31:49.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:49 smithi076 conmon[62156]: mon.a (mon.0) 1995 : cluster [DBG] osdmap e758: 8 total, 8 up, 7 in 2021-03-15T22:31:49.788 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:31:49.789 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:31:49.789 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:31:49.789 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:31:49.831 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:49.857 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:49.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:49.858 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:49.959 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:49.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:49.984 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:49.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:50.086 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:50.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:50.110 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:50.110 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:50.212 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:50.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:50.239 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:50.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:50.283 INFO:tasks.rados.rados.0.smithi076.stdout:2822: done (6 left) 2021-03-15T22:31:50.283 INFO:tasks.rados.rados.0.smithi076.stdout:2823: done (5 left) 2021-03-15T22:31:50.283 INFO:tasks.rados.rados.0.smithi076.stdout:2824: done (4 left) 2021-03-15T22:31:50.284 INFO:tasks.rados.rados.0.smithi076.stdout:2825: done (3 left) 2021-03-15T22:31:50.284 INFO:tasks.rados.rados.0.smithi076.stdout:2826: done (2 left) 2021-03-15T22:31:50.284 INFO:tasks.rados.rados.0.smithi076.stdout:2827: done (1 left) 2021-03-15T22:31:50.285 INFO:tasks.rados.rados.0.smithi076.stdout:2828: done (0 left) 2021-03-15T22:31:50.285 INFO:tasks.rados.rados.0.smithi076.stdout:2829: snap_create 2021-03-15T22:31:50.341 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:50.366 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:50.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:50.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:50.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:50.493 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:50.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:50.495 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:50.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:50 smithi076 conmon[56514]: cluster 2021-03-15T22:31:49.166518+0000 mgr.y (mgr.14144) 904 : cluster [DBG] pgmap v1282: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 641 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 271 KiB/s wr, 2 op/s; 4/1902 objects misplaced (0.210%) 2021-03-15T22:31:50.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:50 smithi076 conmon[56514]: cluster 2021-03-15T22:31:49.267900+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2021-03-15T22:31:50.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:50 smithi076 conmon[62156]: cluster 2021-03-15T22:31:49 2021-03-15T22:31:50.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:50 smithi076 conmon[62156]: .166518+0000 mgr.y (mgr.14144) 904 : cluster [DBG] pgmap v1282: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 641 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 271 KiB/s wr, 2 op/s; 4/1902 objects misplaced (0.210%) 2021-03-15T22:31:50.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:50 smithi076 conmon[62156]: cluster 2021-03-15T22:31:49.267900+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2021-03-15T22:31:50.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:50 smithi125 conmon[56739]: cluster 2021-03-15T22:31:49.166518+0000 mgr.y (mgr.14144) 904 : cluster [DBG] pgmap v1282: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 641 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 271 KiB/s wr, 2 op/s; 4/1902 objects misplaced (0.210%) 2021-03-15T22:31:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:50 smithi125 conmon[56739]: cluster 2021-03-15T22:31 2021-03-15T22:31:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:50 smithi125 conmon[56739]: :49.267900+0000 mon.a (mon.0) 1996 : cluster [DBG] osdmap e759: 8 total, 8 up, 7 in 2021-03-15T22:31:50.538 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:31:50 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:31:50.570 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:50.595 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:31:50.596 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:50.602 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 7, 0, 3, 5] out_osds: [6] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:31:50.602 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 1 1 2021-03-15T22:31:50.622 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:50.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:50.624 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:50.725 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:50.749 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:50.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:50.751 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:50.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:50.882 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:50.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:50.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:50.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:51.006 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:51.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:51.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:51.109 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:51.131 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:51.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:51.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:51.158 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:51.159 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:51.234 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:51.256 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:51.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:51.257 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:51.287 INFO:tasks.rados.rados.0.smithi076.stdout:2829: done (0 left) 2021-03-15T22:31:51.288 INFO:tasks.rados.rados.0.smithi076.stdout:2830: write (excl) oid 48 current snap is 330 2021-03-15T22:31:51.288 INFO:tasks.rados.rados.0.smithi076.stdout:2830: seq_num 983 ranges {521281=579304,1656270=774664} 2021-03-15T22:31:51.288 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:31:51.293 INFO:tasks.rados.rados.0.smithi076.stdout:2830: writing smithi07694878-48 from 521281 to 1100585 tid 1 2021-03-15T22:31:51.301 INFO:tasks.rados.rados.0.smithi076.stdout:2830: writing smithi07694878-48 from 1656270 to 2430934 tid 2 2021-03-15T22:31:51.302 INFO:tasks.rados.rados.0.smithi076.stdout:2831: snap_remove snap 316 2021-03-15T22:31:51.358 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:51.384 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:51.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:51.386 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:51.423 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:51 smithi076 conmon[56514]: audit 2021-03-15T22:31:50.083503+0000 mgr.y (mgr.14144) 905 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:51.424 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:51 smithi076 conmon[56514]: cluster 2021-03-15T22:31:50.280592+0000 mon.a (mon.0) 1997 : cluster [DBG] osdmap e760: 8 total, 8 up, 7 in 2021-03-15T22:31:51.424 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:51 smithi076 conmon[56514]: audit 2021-03-15T22:31:50.922615+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? 172.21.15.76:0/1784447480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:51.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:51 smithi076 conmon[62156]: audit 2021-03-15T22:31:50.083503+0000 mgr.y (mgr.14144 2021-03-15T22:31:51.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:51 smithi076 conmon[62156]: ) 905 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:51.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:51 smithi076 conmon[62156]: cluster 2021-03-15T22:31:50.280592+0000 mon.a (mon.0) 1997 : cluster [DBG] osdmap e760: 8 total, 8 up, 7 in 2021-03-15T22:31:51.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:51 smithi076 conmon[62156]: audit 2021-03-15T22:31:50.922615+0000 mon.a (mon.0) 1998 : audit [INF] from='client.? 172.21.15.76:0/1784447480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:51.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:51.514 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:51.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:51.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:51.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:51.643 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:51.646 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:51.747 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:51.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:51.770 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:51.771 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:51.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:51 smithi125 conmon[56739]: audit 2021-03-15T22:31:50.083503+0000 mgr.y (mgr.14144) 905 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:31:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:51 smithi125 conmon[56739]: cluster 2021-03-15T22:31:50.280592+0000 mon.a (mon.0) 1997 : cluster [DBG] osdmap e760: 8 total, 8 up, 7 in 2021-03-15T22:31:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:51 smithi125 conmon[56739]: audit 2021-03-15T22:31:50.922615+0000 mon.a (mon.0) 1998 : audit 2021-03-15T22:31:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:51 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/1784447480' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:31:51.872 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:51.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:51.895 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:51.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:51.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:52.023 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:52.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:52.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:52.126 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:52.153 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:52.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:52.154 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:52.256 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:52.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:52.281 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:52.282 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:52.300 INFO:tasks.rados.rados.0.smithi076.stdout:2831: done (1 left) 2021-03-15T22:31:52.301 INFO:tasks.rados.rados.0.smithi076.stdout:2832: snap_remove snap 328 2021-03-15T22:31:52.382 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:52.405 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:52.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:52.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:52.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[62156]: cluster 2021-03-15T22:31:51 2021-03-15T22:31:52.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[62156]: .167042+0000 mgr.y (mgr.14144) 906 : cluster [DBG] pgmap v1285: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 607 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 277 KiB/s wr, 10 op/s; 4/1851 objects misplaced (0.216%) 2021-03-15T22:31:52.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[62156]: cluster 2021-03-15T22:31:51. 2021-03-15T22:31:52.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[62156]: 272980+0000 mon.a (mon.0) 1999 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:52.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[62156]: audit 2021-03-15T22:31:52.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[62156]: 2021-03-15T22:31:51.282061+0000 mon.a (mon.0) 2000 : audit 2021-03-15T22:31:52.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/1784447480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:52.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[62156]: cluster 2021-03-15T22:31:52.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[62156]: 2021-03-15T22:31:51.282117+0000 2021-03-15T22:31:52.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[62156]: mon.a (mon.0) 2001 : cluster [DBG] osdmap e761: 8 total, 8 up, 7 in 2021-03-15T22:31:52.443 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[56514]: cluster 2021-03-15T22:31:51.167042+0000 mgr.y (mgr 2021-03-15T22:31:52.443 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[56514]: .14144) 906 : cluster [DBG] pgmap v1285: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 607 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 277 KiB/s wr, 10 op/s; 4/1851 objects misplaced (0.216%) 2021-03-15T22:31:52.443 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[56514]: cluster 2021-03-15T22:31:51.272980+0000 mon.a (mon.0) 1999 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:52.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[56514]: audit 2021-03-15T22:31:51.282061+0000 mon.a (mon.0) 2000 : audit [INF] from='client.? 172.21.15.76:0/1784447480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:52.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:31:52.444 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:52 smithi076 conmon[56514]: 15T22:31:51.282117+0000 mon.a (mon.0) 2001 : cluster [DBG] osdmap e761: 8 total, 8 up, 7 in 2021-03-15T22:31:52.508 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:52.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:52.533 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:52.533 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:52.634 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:52.657 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:52.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:52.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:52.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:52.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:52.785 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:52.786 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:52.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: cluster 2021-03-15T22:31:51.167042+0000 mgr.y 2021-03-15T22:31:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: (mgr.14144) 906 : cluster [DBG] pgmap v1285: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 607 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 277 KiB/s wr, 10 op/s; 4/1851 objects misplaced (0.216%) 2021-03-15T22:31:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: cluster 2021-03-15T22:31:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: 2021-03-15T22:31:51.272980+0000 mon.a (mon.0) 1999 2021-03-15T22:31:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:52.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: audit 2021 2021-03-15T22:31:52.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: -03-15T22:31:51.282061+0000 2021-03-15T22:31:52.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: mon.a (mon.0) 2000 : audit [INF] from='client.? 172.21.15.76:0/1784447480' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:31:52.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: 2021-03-15T22:31:52.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: cluster 2021-03-15T22:31:51. 2021-03-15T22:31:52.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: 282117+0000 mon.a (mon.0) 2001 : cluster 2021-03-15T22:31:52.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:52 smithi125 conmon[56739]: [DBG] osdmap e761: 8 total, 8 up, 7 in 2021-03-15T22:31:52.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:52.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:52.910 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:52.910 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:53.011 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:53.039 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:53.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:53.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:53.142 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:53.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:53.170 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:53.170 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:53.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:53.299 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:53.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:53.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:53.307 INFO:tasks.rados.rados.0.smithi076.stdout:2832: done (1 left) 2021-03-15T22:31:53.307 INFO:tasks.rados.rados.0.smithi076.stdout:2833: snap_remove snap 199 2021-03-15T22:31:53.382 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:53 smithi076 conmon[56514]: cluster 2021-03-15T22:31:52.296416+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2021-03-15T22:31:53.383 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:53 smithi076 conmon[62156]: cluster 2021-03-15T22:31:52.296416+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2021-03-15T22:31:53.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:53.426 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:53.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:53.427 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:53.528 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:53.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:53.553 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:53.553 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:53.655 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:53.677 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:53.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:53.678 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:53.732 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:31:53.732 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:31:53.733 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:31:53.733 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:31:53.733 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:31:53.733 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:31:53.734 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:31:53.734 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:31:53.734 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:31:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:31:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:31:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:31:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:31:53.735 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:31:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:31:53.736 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:31:53.780 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:53.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:53 smithi125 conmon[56739]: cluster 2021-03-15T22:31:52.296416+0000 mon.a (mon.0) 2002 : cluster [DBG] osdmap e762: 8 total, 8 up, 7 in 2021-03-15T22:31:53.804 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:53.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:53.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:53.907 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:53.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:53.931 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:53.931 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:54.032 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:54.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:54.056 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:54.057 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:54.158 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:54.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:54.193 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:54.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:54.294 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:54.312 INFO:tasks.rados.rados.0.smithi076.stdout:2833: done (1 left) 2021-03-15T22:31:54.313 INFO:tasks.rados.rados.0.smithi076.stdout:2834: read oid 13 snap -1 2021-03-15T22:31:54.313 INFO:tasks.rados.rados.0.smithi076.stdout:2834: expect deleted 2021-03-15T22:31:54.313 INFO:tasks.rados.rados.0.smithi076.stdout:2835: snap_create 2021-03-15T22:31:54.314 INFO:tasks.rados.rados.0.smithi076.stdout:2830: finishing write tid 1 to smithi07694878-48 2021-03-15T22:31:54.314 INFO:tasks.rados.rados.0.smithi076.stdout:2830: finishing write tid 2 to smithi07694878-48 2021-03-15T22:31:54.314 INFO:tasks.rados.rados.0.smithi076.stdout:2830: finishing write tid 3 to smithi07694878-48 2021-03-15T22:31:54.314 INFO:tasks.rados.rados.0.smithi076.stdout:2830: finishing write tid 4 to smithi07694878-48 2021-03-15T22:31:54.314 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 1239 (ObjNum 983 snap 330 seq_num 983) dirty exists 2021-03-15T22:31:54.315 INFO:tasks.rados.rados.0.smithi076.stdout:2830: left oid 48 (ObjNum 983 snap 330 seq_num 983) 2021-03-15T22:31:54.322 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:54.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:54.323 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:54.424 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:54.449 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:54.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:54.452 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:54.553 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:54.578 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:54.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:54.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:54.648 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:31:54.672 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56514]: cluster 2021-03-15T22:31:53.167449+0000 mgr.y (mgr.14144) 907 : cluster 2021-03-15T22:31:54.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56514]: [DBG] pgmap v1288: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 595 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 4/1830 objects misplaced (0.219%) 2021-03-15T22:31:54.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56514]: cluster 2021-03-15T22:31:53.304211+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2021-03-15T22:31:54.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56514]: audit 2021-03-15T22:31:53.495575+0000 mon.a (mon.0) 2004 : audit [DBG] from='client.? 172.21.15.76:0/1990311796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:54.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[62156]: cluster 2021-03-15T22:31:53. 2021-03-15T22:31:54.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[62156]: 167449+0000 mgr.y (mgr.14144) 907 : cluster [DBG] pgmap v1288: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 595 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 4/1830 objects misplaced (0.219%) 2021-03-15T22:31:54.674 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[62156]: cluster 2021-03-15T22:31:53.304211+0000 mon.a (mon.0) 2003 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2021-03-15T22:31:54.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[62156]: audit 2021-03-15T22:31:53.495575+0000 mon.a (mon.0) 2004 : audit [DBG] from='client.? 172.21.15.76:0/1990311796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:54.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:54.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:54.706 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:54.707 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:54.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:54 smithi125 conmon[56739]: cluster 2021-03-15T22:31:53.167449+0000 mgr.y (mgr.14144) 907 : cluster [DBG] pgmap v1288: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 595 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 9.8 MiB/s rd, 1.1 MiB/s wr, 12 op/s; 4/1830 objects misplaced (0.219%) 2021-03-15T22:31:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:54 smithi125 conmon[56739]: 2021-03-15T22:31:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:54 smithi125 conmon[56739]: cluster 2021-03-15T22:31:53.304211+0000 mon.a (mon.0) 2021-03-15T22:31:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:54 smithi125 conmon[56739]: 2003 : cluster [DBG] osdmap e763: 8 total, 8 up, 7 in 2021-03-15T22:31:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:54 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:31:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:54 smithi125 conmon[56739]: 22:31:53.495575+0000 mon.a ( 2021-03-15T22:31:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:54 smithi125 conmon[56739]: mon.0) 2004 : audit [DBG] 2021-03-15T22:31:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:54 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1990311796' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:31:54.808 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:54.831 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:54.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:54.833 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:54.934 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:54.957 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:54.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:54.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:55.059 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:55.081 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:55.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:55.083 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:55.184 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:55.199 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: debug 2021-03-15T22:31:54.946+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:31:55.199 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:31:55.199 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.199 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:31:55.200 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.200 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:31:55.200 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.200 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.201 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:31:55.201 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: debug 2021-03-15T22:31:54.947+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:31:55.201 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:31:55.201 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.202 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:31:55.202 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.202 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:31:55.202 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.202 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.203 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:31:55.203 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: debug 2021-03-15T22:31:54.947+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:31:55.203 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:31:55.203 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.204 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:31:55.204 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.204 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:31:55.204 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.204 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.205 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:31:55.205 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: debug 2021-03-15T22:31:54.948+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:31:55.205 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:31:55.205 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:31:55.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:31:55.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.206 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:31:55.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: debug 2021-03-15T22:31:54.948+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:31:55.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:31:55.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.208 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:31:55.208 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.208 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:31:55.208 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.209 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.209 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:31:55.209 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: debug 2021-03-15T22:31:54.949+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:31:55.209 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:31:55.209 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.210 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:31:55.210 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.210 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:31:55.210 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.211 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.211 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:31:55.211 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: debug 2021-03-15T22:31:54.949+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:31:55.211 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:31:55.212 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.212 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:31:55.212 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.212 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:31:55.212 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.213 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.213 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:31:54 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:31:55.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:55.214 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:55.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:55.245 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:55.252 INFO:tasks.rados.rados.0.smithi076.stdout:2830: done (2 left) 2021-03-15T22:31:55.252 INFO:tasks.rados.rados.0.smithi076.stdout:2834: done (1 left) 2021-03-15T22:31:55.252 INFO:tasks.rados.rados.0.smithi076.stdout:2835: done (0 left) 2021-03-15T22:31:55.252 INFO:tasks.rados.rados.0.smithi076.stdout:2836: read oid 39 snap 330 2021-03-15T22:31:55.253 INFO:tasks.rados.rados.0.smithi076.stdout:2836: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:31:55.253 INFO:tasks.rados.rados.0.smithi076.stdout:2837: read oid 47 snap -1 2021-03-15T22:31:55.253 INFO:tasks.rados.rados.0.smithi076.stdout:2837: expect (ObjNum 928 snap 315 seq_num 928) 2021-03-15T22:31:55.253 INFO:tasks.rados.rados.0.smithi076.stdout:2838: copy_from oid 50 from oid 8 current snap is 331 2021-03-15T22:31:55.253 INFO:tasks.rados.rados.0.smithi076.stdout:2839: write oid 32 current snap is 331 2021-03-15T22:31:55.254 INFO:tasks.rados.rados.0.smithi076.stdout:2839: seq_num 984 ranges {538088=654553,1714752=634675} 2021-03-15T22:31:55.256 INFO:tasks.rados.rados.0.smithi076.stdout:2839: writing smithi07694878-32 from 538088 to 1192641 tid 1 2021-03-15T22:31:55.264 INFO:tasks.rados.rados.0.smithi076.stdout:2839: writing smithi07694878-32 from 1714752 to 2349427 tid 2 2021-03-15T22:31:55.265 INFO:tasks.rados.rados.0.smithi076.stdout:2840: read oid 38 snap -1 2021-03-15T22:31:55.265 INFO:tasks.rados.rados.0.smithi076.stdout:2840: expect (ObjNum 955 snap 320 seq_num 955) 2021-03-15T22:31:55.265 INFO:tasks.rados.rados.0.smithi076.stdout:2841: delete oid 17 current snap is 331 2021-03-15T22:31:55.267 INFO:tasks.rados.rados.0.smithi076.stdout:2838: finishing copy_from to smithi07694878-50 2021-03-15T22:31:55.268 INFO:tasks.rados.rados.0.smithi076.stdout:2838: got expected ENOENT (src dne) 2021-03-15T22:31:55.268 INFO:tasks.rados.rados.0.smithi076.stdout:2838: finishing copy_from racing read to smithi07694878-50 2021-03-15T22:31:55.275 INFO:tasks.rados.rados.0.smithi076.stdout:2836: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:31:55.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:55.345 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:55.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:55.347 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:55.409 INFO:tasks.rados.rados.0.smithi076.stdout:2836: done (5 left) 2021-03-15T22:31:55.409 INFO:tasks.rados.rados.0.smithi076.stdout:2838: done (4 left) 2021-03-15T22:31:55.410 INFO:tasks.rados.rados.0.smithi076.stdout:2841: done (3 left) 2021-03-15T22:31:55.410 INFO:tasks.rados.rados.0.smithi076.stdout:2842: copy_from oid 7 from oid 33 current snap is 331 2021-03-15T22:31:55.410 INFO:tasks.rados.rados.0.smithi076.stdout:2837: expect (ObjNum 928 snap 315 seq_num 928) 2021-03-15T22:31:55.434 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:31:55.435 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:31:55.448 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:55.470 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:55.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:55.472 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:55.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: cluster 2021-03-15T22:31:54.307430+0000 mon.a (mon.0) 2005 : cluster [DBG] osdmap e764: 8 total, 8 up, 7 in 2021-03-15T22:31:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: audit 2021-03-15T22:31:54.642109+0000 mon.a (mon.0) 2006 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:31:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:31:54.942283+0000 mgr.y (mgr.14144) 908 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:31:55.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: audit 2021-03-15T22:31:54.946618+0000 mon.a (mon.0) 2007 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:31:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:31:54.947670+0000 mgr.y (mgr.14144) 909 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:31:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:31:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:31:55.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:31:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:31:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:31:54.948136+0000 mgr.y (mgr.14144) 910 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:31:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:31:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:31:55.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:31:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:31:55.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:31:54.948683+0000 mgr.y (mgr.14144) 911 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:31:55.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:31:55.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: cluster 2021-03-15T22:31:55.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: 2021-03-15T22:31:54.307430+0000 mon.a (mon.0) 2005 : cluster [DBG] osdmap e764: 8 total, 8 up, 7 in 2021-03-15T22:31:55.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: audit 2021-03-15T22:31:54.642109+0000 mon.a (mon.0) 2006 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:31:55.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:31:54.942283+0000 mgr.y (mgr.14144) 908 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:31:55.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: audit 2021-03-15T22:31:54.946618+0000 mon.a (mon.0) 2007 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:31:55.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:31:54.947670+0000 mgr.y (mgr.14144) 909 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:31:55.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:31:55.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:31:55.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:31:55.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:31:55.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:31:54.948136+0000 mgr.y (mgr.14144) 910 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:31:55.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:31:55.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:31:55.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:31:55.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:31:55.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:31:54.948683+0000 mgr.y (mgr.14144) 911 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:31:55.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:31:55.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:31:55.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:31:55.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:31:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:31:54.949229+0000 mgr.y (mgr.14144) 912 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:31:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:31:55.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:31:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:31:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:31:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:31:54.949671+0000 mgr.y (mgr.14144) 913 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:31:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:31:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:31:55.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:31:55.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:31:55.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:31:54.950041+0000 mgr.y (mgr.14144) 914 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:31:55.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:31:55.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:31:55.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:31:55.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:31:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:31:55.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:31:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:31:54.949229+0000 mgr.y (mgr.14144) 912 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:31:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:31:55.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:31:55.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:31:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.540 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:31:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:31:54.949671+0000 mgr.y (mgr.14144) 913 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:31:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:31:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:31:55.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:31:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:31:55.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:31:54.950041+0000 mgr.y (mgr.14144) 914 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:31:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:31:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:31:55.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.544 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:31:55.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:31:55.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: cephadm 2021-03-15T22:31:54.950377+0000 mgr.y (mgr.14144) 915 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:31:55.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:31:55.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:31:55.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:31:55.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:31:55.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[56514]: cluster 2021-03-15T22:31:55.245568+0000 mon.a (mon.0) 2008 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2021-03-15T22:31:55.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:31:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: cephadm 2021-03-15T22:31:54.950377+0000 mgr.y (mgr.14144) 915 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:31:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:31:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:31:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:31:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:31:55.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:55 smithi076 conmon[62156]: cluster 2021-03-15T22:31:55.245568+0000 mon.a (mon.0) 2008 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2021-03-15T22:31:55.573 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:55.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:55.597 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:55.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:55.699 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:55.721 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:55.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:55.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:55.745 INFO:tasks.rados.rados.0.smithi076.stdout:2840: expect (ObjNum 955 snap 320 seq_num 955) 2021-03-15T22:31:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: cluster 2021-03-15T22:31:54.307430+0000 mon.a (mon.0) 2005 : cluster 2021-03-15T22:31:55.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: [DBG] osdmap e764: 8 total, 8 up, 7 in 2021-03-15T22:31:55.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: audit 2021-03-15T22:31:54.642109+0000 mon.a (mon.0) 2006 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:31:55.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:31:54.942283 2021-03-15T22:31:55.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 908 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:31:55.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:31:55.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: T22:31:54.946618+0000 mon.a (mon.0) 2007 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:31:55.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: cephadm 2021-03-15 2021-03-15T22:31:55.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: T22:31:54.947670+0000 mgr.y (mgr.14144) 909 : cephadm 2021-03-15T22:31:55.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:31:55.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:31:55.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:31:55.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:31:55.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:31:55.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:31:54.948136+0000 mgr.y (mgr.14144) 910 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:31:55.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:31:55.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:31:55.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:31:55.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:31:55.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:31:54.948683+0000 mgr.y 2021-03-15T22:31:55.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: (mgr.14144) 911 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:31:55.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:31:55.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:31:55.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:31:55.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:31:55.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:31:54.949229+0000 mgr.y (mgr.14144) 912 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:31:55.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:31:55.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:31:55.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:31:55.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:31:55.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: cephadm 2021-03- 2021-03-15T22:31:55.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: 15T22:31:54.949671+0000 mgr.y (mgr.14144) 913 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:31:55.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:31:55.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:31:55.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:31:55.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:31:55.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:31:54.950041+0000 mgr.y (mgr.14144) 914 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:31:55.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:31:55.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:31:55.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:31:55.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:31:55.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: cephadm 2021-03-15T22:31:54.950377+0000 mgr.y (mgr.14144) 915 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:31:55.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:31:55.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:31:55.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:31:55.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:31:55.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:31:55.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:31:55.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:31:55.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:31:55.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:55 smithi125 conmon[56739]: cluster 2021-03-15T22:31:55.245568+0000 mon.a (mon.0) 2008 : cluster [DBG] osdmap e765: 8 total, 8 up, 7 in 2021-03-15T22:31:55.824 INFO:tasks.rados.rados.0.smithi076.stdout:2839: finishing write tid 1 to smithi07694878-32 2021-03-15T22:31:55.825 INFO:tasks.rados.rados.0.smithi076.stdout:2839: finishing write tid 2 to smithi07694878-32 2021-03-15T22:31:55.825 INFO:tasks.rados.rados.0.smithi076.stdout:2839: finishing write tid 3 to smithi07694878-32 2021-03-15T22:31:55.825 INFO:tasks.rados.rados.0.smithi076.stdout:2839: finishing write tid 4 to smithi07694878-32 2021-03-15T22:31:55.826 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 2427 (ObjNum 984 snap 331 seq_num 984) dirty exists 2021-03-15T22:31:55.826 INFO:tasks.rados.rados.0.smithi076.stdout:2839: left oid 32 (ObjNum 984 snap 331 seq_num 984) 2021-03-15T22:31:55.827 INFO:tasks.rados.rados.0.smithi076.stdout:2842: finishing copy_from to smithi07694878-7 2021-03-15T22:31:55.827 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 1766 (ObjNum 714 snap 231 seq_num 714) dirty exists 2021-03-15T22:31:55.827 INFO:tasks.rados.rados.0.smithi076.stdout:2842: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:31:55.827 INFO:tasks.rados.rados.0.smithi076.stdout:2837: done (3 left) 2021-03-15T22:31:55.827 INFO:tasks.rados.rados.0.smithi076.stdout:2839: done (2 left) 2021-03-15T22:31:55.828 INFO:tasks.rados.rados.0.smithi076.stdout:2840: done (1 left) 2021-03-15T22:31:55.828 INFO:tasks.rados.rados.0.smithi076.stdout:2842: done (0 left) 2021-03-15T22:31:55.828 INFO:tasks.rados.rados.0.smithi076.stdout:2843: delete oid 6 current snap is 331 2021-03-15T22:31:55.829 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:55.830 INFO:tasks.rados.rados.0.smithi076.stdout:2843: done (0 left) 2021-03-15T22:31:55.830 INFO:tasks.rados.rados.0.smithi076.stdout:2844: copy_from oid 25 from oid 5 current snap is 331 2021-03-15T22:31:55.830 INFO:tasks.rados.rados.0.smithi076.stdout:2845: snap_create 2021-03-15T22:31:55.841 INFO:tasks.rados.rados.0.smithi076.stdout:2844: finishing copy_from to smithi07694878-25 2021-03-15T22:31:55.842 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 1969 (ObjNum 888 snap 296 seq_num 888) dirty exists 2021-03-15T22:31:55.842 INFO:tasks.rados.rados.0.smithi076.stdout:2844: finishing copy_from racing read to smithi07694878-25 2021-03-15T22:31:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:55.853 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:55.854 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:55.955 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:55.978 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:55.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:55.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:56.080 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:56.103 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:56.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:56.104 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:56.205 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:56.227 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:56.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:56.228 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:56.328 INFO:tasks.rados.rados.0.smithi076.stdout:2844: done (1 left) 2021-03-15T22:31:56.328 INFO:tasks.rados.rados.0.smithi076.stdout:2845: done (0 left) 2021-03-15T22:31:56.329 INFO:tasks.rados.rados.0.smithi076.stdout:2846: copy_from oid 38 from oid 2 current snap is 332 2021-03-15T22:31:56.329 INFO:tasks.rados.rados.0.smithi076.stdout:2847: copy_from oid 14 from oid 44 current snap is 332 2021-03-15T22:31:56.329 INFO:tasks.rados.rados.0.smithi076.stdout:2848: snap_remove snap 295 2021-03-15T22:31:56.330 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:56.333 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.1 to 1 (10000) 2021-03-15T22:31:56.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:56.363 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:56.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:56.366 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:31:56.465 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:56.491 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:56.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:56.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:56.594 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:56.621 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:56.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:56.622 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:56.724 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:56.749 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:56.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:56.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:56.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[56514]: cluster 2021-03-15T22:31:55.168021+0000 mgr.y (mgr.14144) 916 : cluster [DBG] pgmap v1291: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 595 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 831 KiB/s wr, 3 op/s; 4/1830 objects misplaced (0.219%) 2021-03-15T22:31:56.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[56514]: cluster 2021-03-15T22:31:55 2021-03-15T22:31:56.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[56514]: .354885+0000 osd.7 (osd.7) 141 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:56.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[56514]: cluster 2021-03-15T22:31:55.356190+0000 osd.7 (osd.7) 142 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:56.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[56514]: audit 2021-03-15T22:31:55.543752+0000 mon.c (mon.1) 62 : audit [INF] from='client.? 172.21.15.76:0/4095110867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:31:56.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[56514]: audit 2021-03-15T22:31:55.544080+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:31:56.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[62156]: cluster 2021-03-15T22:31:55.168021+0000 mgr.y (mgr.14144) 916 2021-03-15T22:31:56.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[62156]: : cluster [DBG] pgmap v1291: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 595 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 831 KiB/s wr, 3 op/s; 4/1830 objects misplaced (0.219%) 2021-03-15T22:31:56.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[62156]: cluster 2021-03-15T22:31:55.354885+0000 osd.7 (osd.7) 141 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:56.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[62156]: cluster 2021-03-15T22:31:55.356190+0000 osd.7 (osd.7) 142 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:56.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[62156]: audit 2021-03-15T22:31:55.543752+0000 mon.c (mon.1) 62 : audit [INF] from='client.? 172.21.15.76:0/4095110867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:31:56.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:56 smithi076 conmon[62156]: audit 2021-03-15T22:31:55.544080+0000 mon.a (mon.0) 2009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:31:56.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:56 smithi125 conmon[56739]: cluster 2021-03-15T22:31:55.168021+0000 mgr.y (mgr.14144) 916 : cluster [DBG] pgmap v1291: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 595 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 831 KiB/s wr, 3 op/s; 4/1830 objects misplaced (0.219%) 2021-03-15T22:31:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:56 smithi125 conmon[56739]: cluster 2021-03-15T22:31:55.354885+0000 osd.7 (osd.7) 141 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:56 smithi125 conmon[56739]: cluster 2021-03-15T22:31:55.356190+0000 osd.7 (osd.7) 142 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:56 smithi125 conmon[56739]: audit 2021-03-15T22:31:55.543752+0000 mon.c (mon.1) 62 : audit [INF] from='client.? 172.21.15.76:0/4095110867' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:31:56.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:56 smithi125 conmon[56739]: audit 2021-03-15T22:31:55.544080+0000 2021-03-15T22:31:56.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:56 smithi125 conmon[56739]: mon.a (mon.0) 2009 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2021-03-15T22:31:56.852 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:56.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:56.879 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:56.880 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:56.949 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:31:56.982 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:57.015 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:57.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:57.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:57.105 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 2 1 2021-03-15T22:31:57.119 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:31:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:57.176 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:57.176 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:31:57.278 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:57.305 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:57.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:57.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:57.337 INFO:tasks.rados.rados.0.smithi076.stdout:2848: done (2 left) 2021-03-15T22:31:57.337 INFO:tasks.rados.rados.0.smithi076.stdout:2849: read oid 28 snap -1 2021-03-15T22:31:57.338 INFO:tasks.rados.rados.0.smithi076.stdout:2849: expect deleted 2021-03-15T22:31:57.338 INFO:tasks.rados.rados.0.smithi076.stdout:2850: rollback oid 41 current snap is 332 2021-03-15T22:31:57.338 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 41 to 297 2021-03-15T22:31:57.338 INFO:tasks.rados.rados.0.smithi076.stdout:2851: copy_from oid 31 from oid 12 current snap is 332 2021-03-15T22:31:57.338 INFO:tasks.rados.rados.0.smithi076.stdout:2852: copy_from oid 5 from oid 6 current snap is 332 2021-03-15T22:31:57.339 INFO:tasks.rados.rados.0.smithi076.stdout:2853: snap_remove snap 310 2021-03-15T22:31:57.407 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:57.430 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:57.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:57.431 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:57.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:57 smithi076 conmon[56514]: audit 2021-03-15T22:31:56.323870+0000 mon.a (mon.0) 2010 : audit [INF] 2021-03-15T22:31:57.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:57 smithi076 conmon[56514]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-15T22:31:57.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:57 smithi076 conmon[56514]: cluster 2021-03-15T22:31:56.323957+0000 mon.a (mon.0) 2011 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2021-03-15T22:31:57.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:57 smithi076 conmon[62156]: audit 2021-03-15T22:31:56.323870 2021-03-15T22:31:57.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:57 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-15T22:31:57.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:57 smithi076 conmon[62156]: cluster 2021-03-15T22:31:56.323957+0000 mon.a ( 2021-03-15T22:31:57.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:57 smithi076 conmon[62156]: mon.0) 2011 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2021-03-15T22:31:57.532 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:57.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:57.555 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:57.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:57.657 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:57.681 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:57.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:57.683 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:57.683 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:57.695 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:57.780 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:31:57.784 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:57.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:57 smithi125 conmon[56739]: audit 2021-03-15T22:31:57.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:57 smithi125 conmon[56739]: 2021-03-15T22:31:56.323870+0000 mon.a ( 2021-03-15T22:31:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:57 smithi125 conmon[56739]: mon.0) 2010 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2021-03-15T22:31:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:57 smithi125 conmon[56739]: cluster 2021-03-15T22:31:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:57 smithi125 conmon[56739]: 2021-03-15T22:31:56.323957+0000 mon.a (mon.0) 2021-03-15T22:31:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:57 smithi125 conmon[56739]: 2011 : cluster [DBG] osdmap e766: 8 total, 8 up, 7 in 2021-03-15T22:31:57.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:57.813 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:57.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:57.915 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:57.942 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:57.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:57.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:58.044 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:58.069 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:58.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:58.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:58.171 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:58.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:58.197 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:58.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:58.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:58.325 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:58.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:58.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:58.350 INFO:tasks.rados.rados.0.smithi076.stdout:2853: done (6 left) 2021-03-15T22:31:58.350 INFO:tasks.rados.rados.0.smithi076.stdout:2854: copy_from oid 11 from oid 20 current snap is 332 2021-03-15T22:31:58.350 INFO:tasks.rados.rados.0.smithi076.stdout:2855: read oid 30 snap -1 2021-03-15T22:31:58.350 INFO:tasks.rados.rados.0.smithi076.stdout:2855: expect deleted 2021-03-15T22:31:58.351 INFO:tasks.rados.rados.0.smithi076.stdout:2856: read oid 29 snap 308 2021-03-15T22:31:58.351 INFO:tasks.rados.rados.0.smithi076.stdout:2856: expect deleted 2021-03-15T22:31:58.351 INFO:tasks.rados.rados.0.smithi076.stdout:2857: rollback oid 50 current snap is 332 2021-03-15T22:31:58.351 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 50 to 315 2021-03-15T22:31:58.352 INFO:tasks.rados.rados.0.smithi076.stdout:2858: delete oid 48 current snap is 332 2021-03-15T22:31:58.352 INFO:tasks.rados.rados.0.smithi076.stdout:2847: finishing copy_from to smithi07694878-14 2021-03-15T22:31:58.352 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 1762 (ObjNum 936 snap 316 seq_num 936) dirty exists 2021-03-15T22:31:58.352 INFO:tasks.rados.rados.0.smithi076.stdout:2847: finishing copy_from racing read to smithi07694878-14 2021-03-15T22:31:58.352 INFO:tasks.rados.rados.0.smithi076.stdout:2846: finishing copy_from to smithi07694878-38 2021-03-15T22:31:58.353 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 1770 (ObjNum 931 snap 315 seq_num 931) dirty exists 2021-03-15T22:31:58.353 INFO:tasks.rados.rados.0.smithi076.stdout:2846: finishing copy_from racing read to smithi07694878-38 2021-03-15T22:31:58.353 INFO:tasks.rados.rados.0.smithi076.stdout:2852: finishing copy_from to smithi07694878-5 2021-03-15T22:31:58.354 INFO:tasks.rados.rados.0.smithi076.stdout:2852: got expected ENOENT (src dne) 2021-03-15T22:31:58.354 INFO:tasks.rados.rados.0.smithi076.stdout:2851: finishing copy_from to smithi07694878-31 2021-03-15T22:31:58.354 INFO:tasks.rados.rados.0.smithi076.stdout:2851: got expected ENOENT (src dne) 2021-03-15T22:31:58.355 INFO:tasks.rados.rados.0.smithi076.stdout:2852: finishing copy_from racing read to smithi07694878-5 2021-03-15T22:31:58.355 INFO:tasks.rados.rados.0.smithi076.stdout:2850: finishing rollback tid 0 to smithi07694878-41 2021-03-15T22:31:58.355 INFO:tasks.rados.rados.0.smithi076.stdout:2851: finishing copy_from racing read to smithi07694878-31 2021-03-15T22:31:58.355 INFO:tasks.rados.rados.0.smithi076.stdout:2850: finishing rollback tid 1 to smithi07694878-41 2021-03-15T22:31:58.355 INFO:tasks.rados.rados.0.smithi076.stdout:2850: finishing rollback tid 2 to smithi07694878-41 2021-03-15T22:31:58.356 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 1242 (ObjNum 872 snap 287 seq_num 872) dirty exists 2021-03-15T22:31:58.359 INFO:tasks.rados.rados.0.smithi076.stdout:2846: done (10 left) 2021-03-15T22:31:58.359 INFO:tasks.rados.rados.0.smithi076.stdout:2847: done (9 left) 2021-03-15T22:31:58.361 INFO:tasks.rados.rados.0.smithi076.stdout:2849: done (8 left) 2021-03-15T22:31:58.361 INFO:tasks.rados.rados.0.smithi076.stdout:2850: done (7 left) 2021-03-15T22:31:58.361 INFO:tasks.rados.rados.0.smithi076.stdout:2851: done (6 left) 2021-03-15T22:31:58.362 INFO:tasks.rados.rados.0.smithi076.stdout:2852: done (5 left) 2021-03-15T22:31:58.362 INFO:tasks.rados.rados.0.smithi076.stdout:2855: done (4 left) 2021-03-15T22:31:58.362 INFO:tasks.rados.rados.0.smithi076.stdout:2856: done (3 left) 2021-03-15T22:31:58.362 INFO:tasks.rados.rados.0.smithi076.stdout:2858: done (2 left) 2021-03-15T22:31:58.362 INFO:tasks.rados.rados.0.smithi076.stdout:2859: rollback oid 29 current snap is 332 2021-03-15T22:31:58.363 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 29 to 307 2021-03-15T22:31:58.363 INFO:tasks.rados.rados.0.smithi076.stdout:2860: snap_remove snap 324 2021-03-15T22:31:58.369 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:31:58.427 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:58.451 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:58.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:58.453 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:31:58.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[56514]: cluster 2021-03-15T22:31:57.168672+0000 mgr.y (mgr.14144) 917 : cluster [DBG] pgmap v1294: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 2.9 MiB/s wr, 8 op/s; 4/1773 objects misplaced (0.226%) 2021-03-15T22:31:58.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[56514]: cluster 2021-03-15T22:31:57.182052+0000 osd.0 (osd.0) 123 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:58.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[56514]: cluster 2021-03-15T22:31:57.184107+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:58.475 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[56514]: cluster 2021-03-15T22:31:57.328809+0000 mon.a (mon.0) 2012 : cluster [DBG] osdmap e767: 8 total, 8 up, 7 in 2021-03-15T22:31:58.476 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[56514]: audit 2021-03-15T22:31:58.300268+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? 172.21.15.76:0/4166677798' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:31:58.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[62156]: cluster 2021-03-15T22:31:57.168672+0000 mgr.y 2021-03-15T22:31:58.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[62156]: (mgr.14144) 917 : cluster [DBG] pgmap v1294: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 2.9 MiB/s wr, 8 op/s; 4/1773 objects misplaced (0.226%) 2021-03-15T22:31:58.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[62156]: cluster 2021-03-15T22:31:57.182052+0000 osd.0 (osd.0) 123 : cluster [DBG] 2021-03-15T22:31:58.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[62156]: purged_snaps scrub starts 2021-03-15T22:31:58.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[62156]: cluster 2021-03-15T22:31:57.184107+0000 osd.0 (osd.0) 124 : cluster 2021-03-15T22:31:58.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[62156]: [DBG] purged_snaps scrub ok 2021-03-15T22:31:58.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[62156]: cluster 2021-03-15T22:31:57.328809+0000 2021-03-15T22:31:58.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[62156]: mon.a (mon.0) 2012 : cluster [DBG] osdmap e767: 8 total, 8 up, 7 in 2021-03-15T22:31:58.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:31:58.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[62156]: 03-15T22:31:58.300268+0000 mon.a (mon.0) 2013 : audit [INF] from='client.? 172.21.15.76:0/4166677798' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:31:58.478 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:58 smithi076 conmon[62156]: 2021-03-15T22:31:58.554 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:31:58.579 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:58.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:58.581 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:31:58.682 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:58.706 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:58.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:58.707 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:58.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:58 smithi125 conmon[56739]: cluster 2021-03-15T22:31:57.168672+0000 mgr.y (mgr.14144) 917 : cluster [DBG] pgmap v1294: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 562 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 2.9 MiB/s wr, 8 op/s; 4/1773 objects misplaced (0.226%) 2021-03-15T22:31:58.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:58 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:31:58.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:58 smithi125 conmon[56739]: 03-15T22:31:57.182052+0000 osd.0 (osd.0) 123 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:31:58.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:58 smithi125 conmon[56739]: cluster 2021-03-15T22:31:57.184107+0000 osd.0 (osd.0) 124 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:31:58.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:58 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:31:58.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:58 smithi125 conmon[56739]: 31:57.328809+0000 mon.a (mon.0) 2021-03-15T22:31:58.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:58 smithi125 conmon[56739]: 2012 : cluster [DBG] osdmap e767: 8 total, 8 up, 7 in 2021-03-15T22:31:58.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:58 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:31:58.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:58 smithi125 conmon[56739]: 03-15T22:31:58.300268+0000 2021-03-15T22:31:58.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:58 smithi125 conmon[56739]: mon.a (mon.0) 2013 : audit [INF] 2021-03-15T22:31:58.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:58 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/4166677798' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:31:58.809 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:31:58.831 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:58.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:58.834 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:31:58.936 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:58.960 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:58.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:58.962 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:59.063 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:59.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:59.086 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:59.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:59.188 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:59.214 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:59.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:59.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:59.316 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:59.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:59.343 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:59.344 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:59.348 INFO:tasks.rados.rados.0.smithi076.stdout:2860: done (3 left) 2021-03-15T22:31:59.348 INFO:tasks.rados.rados.0.smithi076.stdout:2861: delete oid 38 current snap is 332 2021-03-15T22:31:59.348 INFO:tasks.rados.rados.0.smithi076.stdout:2857: finishing rollback tid 0 to smithi07694878-50 2021-03-15T22:31:59.348 INFO:tasks.rados.rados.0.smithi076.stdout:2857: finishing rollback tid 1 to smithi07694878-50 2021-03-15T22:31:59.349 INFO:tasks.rados.rados.0.smithi076.stdout:2857: finishing rollback tid 2 to smithi07694878-50 2021-03-15T22:31:59.349 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 1765 (ObjNum 598 snap 183 seq_num 598) dirty exists 2021-03-15T22:31:59.349 INFO:tasks.rados.rados.0.smithi076.stdout:2859: finishing rollback tid 0 to smithi07694878-29 2021-03-15T22:31:59.349 INFO:tasks.rados.rados.0.smithi076.stdout:2859: finishing rollback tid 1 to smithi07694878-29 2021-03-15T22:31:59.349 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 2446 (ObjNum 1953066355 snap 925919592 seq_num 942946614) dirty dne 2021-03-15T22:31:59.350 INFO:tasks.rados.rados.0.smithi076.stdout:2854: finishing copy_from to smithi07694878-11 2021-03-15T22:31:59.350 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1763 (ObjNum 962 snap 321 seq_num 962) dirty exists 2021-03-15T22:31:59.350 INFO:tasks.rados.rados.0.smithi076.stdout:2854: finishing copy_from racing read to smithi07694878-11 2021-03-15T22:31:59.356 INFO:tasks.rados.rados.0.smithi076.stdout:2854: done (3 left) 2021-03-15T22:31:59.356 INFO:tasks.rados.rados.0.smithi076.stdout:2857: done (2 left) 2021-03-15T22:31:59.356 INFO:tasks.rados.rados.0.smithi076.stdout:2859: done (1 left) 2021-03-15T22:31:59.357 INFO:tasks.rados.rados.0.smithi076.stdout:2861: done (0 left) 2021-03-15T22:31:59.357 INFO:tasks.rados.rados.0.smithi076.stdout:2862: write oid 11 current snap is 332 2021-03-15T22:31:59.357 INFO:tasks.rados.rados.0.smithi076.stdout:2862: seq_num 989 ranges {622123=747151,2007162=614961} 2021-03-15T22:31:59.364 INFO:tasks.rados.rados.0.smithi076.stdout:2862: writing smithi07694878-11 from 622123 to 1369274 tid 1 2021-03-15T22:31:59.371 INFO:tasks.rados.rados.0.smithi076.stdout:2862: writing smithi07694878-11 from 2007162 to 2622123 tid 2 2021-03-15T22:31:59.372 INFO:tasks.rados.rados.0.smithi076.stdout:2863: rollback oid 16 current snap is 332 2021-03-15T22:31:59.372 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 315 2021-03-15T22:31:59.372 INFO:tasks.rados.rados.0.smithi076.stdout:2862: finishing write tid 1 to smithi07694878-11 2021-03-15T22:31:59.373 INFO:tasks.rados.rados.0.smithi076.stdout:2864: copy_from oid 1 from oid 33 current snap is 332 2021-03-15T22:31:59.373 INFO:tasks.rados.rados.0.smithi076.stdout:2865: snap_remove snap 288 2021-03-15T22:31:59.444 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:31:59.467 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:59.468 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:31:59.569 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:31:59.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[56514]: cluster 2021-03-15T22:31:58.326051+0000 mon.a (mon 2021-03-15T22:31:59.590 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[56514]: .0) 2014 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:59.590 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[56514]: audit 2021-03-15T22:31:58.338203+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? 172.21.15.76:0/4166677798' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:31:59.590 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[56514]: cluster 2021-03-15T22:31:58.338264+0000 mon.a (mon.0) 2016 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2021-03-15T22:31:59.590 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[56514]: cluster 2021-03-15T22:31:58.607610+0000 osd.7 (osd.7) 143 : cluster [DBG] 3.f scrub ok 2021-03-15T22:31:59.591 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[56514]: cluster 2021-03-15T22:31:58.617799+0000 osd.3 (osd.3) 139 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:31:59.591 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[62156]: cluster 2021-03-15T22:31:58.326051+0000 mon.a (mon. 2021-03-15T22:31:59.591 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[62156]: 0) 2014 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:59.592 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[62156]: audit 2021-03-15T22:31:58.338203+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? 172.21.15.76:0/4166677798' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:31:59.592 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[62156]: cluster 2021-03-15T22:31:58.338264+0000 mon.a (mon.0) 2016 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2021-03-15T22:31:59.592 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[62156]: cluster 2021-03-15T22:31:58.607610+0000 osd.7 (osd.7) 143 : cluster [DBG] 3.f scrub ok 2021-03-15T22:31:59.592 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:31:59 smithi076 conmon[62156]: cluster 2021-03-15T22:31:58.617799+0000 osd.3 (osd.3) 139 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:31:59.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:59.594 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:59.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:31:59.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:31:59.720 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:31:59.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:59.721 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:31:59.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:59 smithi125 conmon[56739]: cluster 2021-03-15T22:31:58.326051+0000 mon.a (mon.0) 2014 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:31:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:59 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:31:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:59 smithi125 conmon[56739]: :31:58.338203+0000 mon.a (mon.0) 2015 : audit [INF] from='client.? 172.21.15.76:0/4166677798' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:31:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:59 smithi125 conmon[56739]: cluster 2021-03-15T22:31:58.338264+0000 mon.a (mon.0) 2016 : cluster [DBG] osdmap e768: 8 total, 8 up, 7 in 2021-03-15T22:31:59.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:59 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:31:59.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:59 smithi125 conmon[56739]: 22:31:58.607610+0000 osd.7 (osd 2021-03-15T22:31:59.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:59 smithi125 conmon[56739]: .7) 143 : cluster [DBG] 3.f scrub ok 2021-03-15T22:31:59.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:59 smithi125 conmon[56739]: cluster 2021-03-15T22:31:59.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:59 smithi125 conmon[56739]: 2021-03-15T22:31: 2021-03-15T22:31:59.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:59 smithi125 conmon[56739]: 58.617799+0000 osd.3 (osd.3) 139 2021-03-15T22:31:59.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:31:59 smithi125 conmon[56739]: : cluster [DBG] 3.1e scrub ok 2021-03-15T22:31:59.823 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:31:59.845 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:59.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:59.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:31:59.947 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:31:59.969 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:31:59.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:31:59.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:00.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:00.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:00.095 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:00.095 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:00.196 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:00.219 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:00.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:00.220 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:00.250 INFO:tasks.rados.rados.0.smithi076.stdout:2865: done (3 left) 2021-03-15T22:32:00.251 INFO:tasks.rados.rados.0.smithi076.stdout:2866: write (excl) oid 41 current snap is 332 2021-03-15T22:32:00.251 INFO:tasks.rados.rados.0.smithi076.stdout:2866: seq_num 990 ranges {638930=422400,1665644=645682} 2021-03-15T22:32:00.256 INFO:tasks.rados.rados.0.smithi076.stdout:2866: writing smithi07694878-41 from 638930 to 1061330 tid 1 2021-03-15T22:32:00.265 INFO:tasks.rados.rados.0.smithi076.stdout:2866: writing smithi07694878-41 from 1665644 to 2311326 tid 2 2021-03-15T22:32:00.265 INFO:tasks.rados.rados.0.smithi076.stdout:2867: read oid 35 snap -1 2021-03-15T22:32:00.265 INFO:tasks.rados.rados.0.smithi076.stdout:2867: expect deleted 2021-03-15T22:32:00.266 INFO:tasks.rados.rados.0.smithi076.stdout:2868: read oid 27 snap -1 2021-03-15T22:32:00.266 INFO:tasks.rados.rados.0.smithi076.stdout:2868: expect (ObjNum 896 snap 302 seq_num 896) 2021-03-15T22:32:00.266 INFO:tasks.rados.rados.0.smithi076.stdout:2869: snap_create 2021-03-15T22:32:00.266 INFO:tasks.rados.rados.0.smithi076.stdout:2862: finishing write tid 2 to smithi07694878-11 2021-03-15T22:32:00.267 INFO:tasks.rados.rados.0.smithi076.stdout:2862: finishing write tid 3 to smithi07694878-11 2021-03-15T22:32:00.267 INFO:tasks.rados.rados.0.smithi076.stdout:2863: finishing rollback tid 0 to smithi07694878-16 2021-03-15T22:32:00.267 INFO:tasks.rados.rados.0.smithi076.stdout:2862: finishing write tid 4 to smithi07694878-11 2021-03-15T22:32:00.267 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1766 (ObjNum 989 snap 332 seq_num 989) dirty exists 2021-03-15T22:32:00.267 INFO:tasks.rados.rados.0.smithi076.stdout:2862: left oid 11 (ObjNum 989 snap 332 seq_num 989) 2021-03-15T22:32:00.268 INFO:tasks.rados.rados.0.smithi076.stdout:2863: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:32:00.268 INFO:tasks.rados.rados.0.smithi076.stdout:2863: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:32:00.268 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 2272 (ObjNum 803 snap 268 seq_num 803) dirty exists 2021-03-15T22:32:00.268 INFO:tasks.rados.rados.0.smithi076.stdout:2864: finishing copy_from to smithi07694878-1 2021-03-15T22:32:00.269 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 1766 (ObjNum 714 snap 231 seq_num 714) dirty exists 2021-03-15T22:32:00.269 INFO:tasks.rados.rados.0.smithi076.stdout:2864: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:32:00.270 INFO:tasks.rados.rados.0.smithi076.stdout:2866: finishing write tid 1 to smithi07694878-41 2021-03-15T22:32:00.276 INFO:tasks.rados.rados.0.smithi076.stdout:2866: finishing write tid 2 to smithi07694878-41 2021-03-15T22:32:00.277 INFO:tasks.rados.rados.0.smithi076.stdout:2866: finishing write tid 3 to smithi07694878-41 2021-03-15T22:32:00.277 INFO:tasks.rados.rados.0.smithi076.stdout:2866: finishing write tid 4 to smithi07694878-41 2021-03-15T22:32:00.277 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 1246 (ObjNum 990 snap 332 seq_num 990) dirty exists 2021-03-15T22:32:00.277 INFO:tasks.rados.rados.0.smithi076.stdout:2866: left oid 41 (ObjNum 990 snap 332 seq_num 990) 2021-03-15T22:32:00.282 INFO:tasks.rados.rados.0.smithi076.stdout:2868: expect (ObjNum 896 snap 302 seq_num 896) 2021-03-15T22:32:00.321 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:00.343 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:00.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:00.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:00.411 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:32:00 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:32:00.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:00.468 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:00.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:00.470 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:00.571 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:00.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:00.594 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:00.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:00.696 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:00.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:00.720 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:00.721 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:00.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[56514]: cluster 2021-03-15T22:31:58.785514+0000 osd.5 (osd.5) 25 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:32:00.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[56514]: cluster 2021-03-15T22:31:59.169239+0000 mgr.y (mgr.14144) 918 : cluster [DBG] pgmap v1297: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 555 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 2.9 MiB/s wr, 7 op/s; 4/1764 objects misplaced (0.227%) 2021-03-15T22:32:00.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[56514]: cluster 2021-03-15T22:31:59.346042+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e769: 8 total, 8 up, 7 in 2021-03-15T22:32:00.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[56514]: cluster 2021-03-15T22:32:00.247776+0000 mon.a (mon.0) 2018 : cluster [DBG] osdmap e770: 8 total, 8 up, 7 in 2021-03-15T22:32:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:58.785514 2021-03-15T22:32:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 25 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:32:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:32:00.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[62156]: 31:59.169239+0000 mgr.y (mgr.14144) 918 : cluster [DBG] pgmap v1297: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 555 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 2.9 MiB/s wr, 7 op/s; 4/1764 objects misplaced (0.227%) 2021-03-15T22:32:00.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[62156]: cluster 2021-03-15T22:31:59.346042+0000 2021-03-15T22:32:00.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[62156]: mon.a (mon.0) 2017 : cluster [DBG] osdmap e769: 8 total, 8 up, 7 in 2021-03-15T22:32:00.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[62156]: cluster 2021-03-15T22:32:00.247776+0000 mon.a (mon.0) 2018 : 2021-03-15T22:32:00.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:00 smithi076 conmon[62156]: cluster [DBG] osdmap e770: 8 total, 8 up, 7 in 2021-03-15T22:32:00.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:00 smithi125 conmon[56739]: cluster 2021-03-15T22:31:58.785514+0000 osd.5 (osd.5) 25 : cluster 2021-03-15T22:32:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:00 smithi125 conmon[56739]: [DBG] 2.2 scrub ok 2021-03-15T22:32:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:00 smithi125 conmon[56739]: cluster 2021-03-15T22:31:59.169239+0000 mgr.y (mgr.14144) 918 : cluster [DBG] pgmap v1297: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 555 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.6 MiB/s rd, 2.9 MiB/s wr, 7 op/s; 4/1764 objects misplaced (0.227%) 2021-03-15T22:32:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:00 smithi125 conmon[56739]: cluster 2021-03-15T22:31 2021-03-15T22:32:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:00 smithi125 conmon[56739]: :59.346042+0000 mon.a (mon.0) 2017 : cluster [DBG] osdmap e769: 8 total, 8 up, 7 in 2021-03-15T22:32:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:00 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:32:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:00 smithi125 conmon[56739]: -03-15T22:32:00.247776+0000 mon.a (mon.0) 2018 : cluster [DBG] 2021-03-15T22:32:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:00 smithi125 conmon[56739]: osdmap e770: 8 total, 8 up, 7 in 2021-03-15T22:32:00.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:00.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:00.846 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:00.846 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:00.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:00.971 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:00.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:00.972 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:01.073 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:01.098 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:01.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:01.100 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:01.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:01.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:01.229 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:01.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:01.253 INFO:tasks.rados.rados.0.smithi076.stdout:2862: done (6 left) 2021-03-15T22:32:01.253 INFO:tasks.rados.rados.0.smithi076.stdout:2863: done (5 left) 2021-03-15T22:32:01.253 INFO:tasks.rados.rados.0.smithi076.stdout:2864: done (4 left) 2021-03-15T22:32:01.253 INFO:tasks.rados.rados.0.smithi076.stdout:2866: done (3 left) 2021-03-15T22:32:01.254 INFO:tasks.rados.rados.0.smithi076.stdout:2867: done (2 left) 2021-03-15T22:32:01.254 INFO:tasks.rados.rados.0.smithi076.stdout:2868: done (1 left) 2021-03-15T22:32:01.254 INFO:tasks.rados.rados.0.smithi076.stdout:2869: done (0 left) 2021-03-15T22:32:01.254 INFO:tasks.rados.rados.0.smithi076.stdout:2870: write (excl) oid 2 current snap is 333 2021-03-15T22:32:01.255 INFO:tasks.rados.rados.0.smithi076.stdout:2870: seq_num 991 ranges {655737=497649,1607773=789340} 2021-03-15T22:32:01.259 INFO:tasks.rados.rados.0.smithi076.stdout:2870: writing smithi07694878-2 from 655737 to 1153386 tid 1 2021-03-15T22:32:01.267 INFO:tasks.rados.rados.0.smithi076.stdout:2870: writing smithi07694878-2 from 1607773 to 2397113 tid 2 2021-03-15T22:32:01.267 INFO:tasks.rados.rados.0.smithi076.stdout:2871: rollback oid 9 current snap is 333 2021-03-15T22:32:01.267 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 9 to 308 2021-03-15T22:32:01.267 INFO:tasks.rados.rados.0.smithi076.stdout:2870: finishing write tid 1 to smithi07694878-2 2021-03-15T22:32:01.268 INFO:tasks.rados.rados.0.smithi076.stdout:2872: rollback oid 17 current snap is 333 2021-03-15T22:32:01.269 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 17 to 330 2021-03-15T22:32:01.269 INFO:tasks.rados.rados.0.smithi076.stdout:2873: snap_remove snap 255 2021-03-15T22:32:01.331 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:01.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:01.355 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:01.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:01.457 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:01.480 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:01.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:01.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:01.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[56514]: cluster 2021-03-15T22:31:59.864454 2021-03-15T22:32:01.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[56514]: +0000 osd.1 (osd.1) 145 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:32:01.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[56514]: audit 2021-03-15T22:32:00.094006+0000 mgr.y (mgr.14144) 919 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:01.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[56514]: audit 2021-03-15T22:32:01.111074+0000 mon.a (mon.0) 2019 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:32:01.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[56514]: audit 2021-03-15T22:32:01.112487+0000 mon.a (mon.0) 2020 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:32:01.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[56514]: cluster 2021-03-15T22:32:01.250236+0000 mon.a (mon.0) 2021 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2021-03-15T22:32:01.541 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[62156]: cluster 2021-03-15T22:31:59. 2021-03-15T22:32:01.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[62156]: 864454+0000 osd.1 (osd.1) 145 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:32:01.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[62156]: audit 2021-03-15T22:32:00.094006+0000 mgr.y (mgr.14144) 919 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:01.542 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[62156]: audit 2021-03-15T22:32:01.111074+0000 mon.a (mon.0) 2019 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:32:01.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[62156]: audit 2021-03-15T22:32:01.112487+0000 mon.a (mon.0) 2020 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:32:01.543 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:01 smithi076 conmon[62156]: cluster 2021-03-15T22:32:01.250236+0000 mon.a (mon.0) 2021 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2021-03-15T22:32:01.583 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:01.606 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:01.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:01.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:01.709 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:01.733 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:01.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:01.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:01.764 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:32:01.764 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:32:01.767 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:32:01.778 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:32:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:32:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:32:01.778 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:32:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.779 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2768 7.b 7:d10ba14d:::smithi07694878-43:head [write 1689059~741123] snapc d2=[d2,d1,cf] ondisk+write+known_if_redirected e310)", 2021-03-15T22:32:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:32:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 496.66878636799999, 2021-03-15T22:32:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078094859000000003, 2021-03-15T22:32:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:01.780 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:01.781 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2768 2021-03-15T22:32:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:01.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:32:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:32:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033110399999999998 2021-03-15T22:32:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105421+0000", 2021-03-15T22:32:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4809999999999998e-06 2021-03-15T22:32:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105425+0000", 2021-03-15T22:32:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7363999999999998e-05 2021-03-15T22:32:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105442+0000", 2021-03-15T22:32:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9480000000000002e-06 2021-03-15T22:32:01.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105449+0000", 2021-03-15T22:32:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6847000000000001e-05 2021-03-15T22:32:01.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105496+0000", 2021-03-15T22:32:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8231000000000003e-05 2021-03-15T22:32:01.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105584+0000", 2021-03-15T22:32:01.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000198369 2021-03-15T22:32:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:32:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105782+0000", 2021-03-15T22:32:01.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003007292 2021-03-15T22:32:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108790+0000", 2021-03-15T22:32:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000200027 2021-03-15T22:32:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108990+0000", 2021-03-15T22:32:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074107929000000003 2021-03-15T22:32:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:32:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183098+0000", 2021-03-15T22:32:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4002000000000002e-05 2021-03-15T22:32:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183152+0000", 2021-03-15T22:32:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3265000000000001e-05 2021-03-15T22:32:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183185+0000", 2021-03-15T22:32:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078094859000000003 2021-03-15T22:32:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.797 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2769 7.b 7:d10ba14d:::smithi07694878-43:head [setxattr _header (58) in=7b,truncate 2588344] snapc d2=[d2,d1,cf] ondisk+write+known_if_redirected e310)", 2021-03-15T22:32:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:32:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 496.66841534000002, 2021-03-15T22:32:01.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077795487999999996, 2021-03-15T22:32:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:01.800 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:01.801 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2769 2021-03-15T22:32:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.802 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:32:01.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:01.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:32:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2639999999999998e-06 2021-03-15T22:32:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:01.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105463+0000", 2021-03-15T22:32:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.426e-06 2021-03-15T22:32:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105465+0000", 2021-03-15T22:32:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9349999999999999e-06 2021-03-15T22:32:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105471+0000", 2021-03-15T22:32:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2420000000000002e-06 2021-03-15T22:32:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105475+0000", 2021-03-15T22:32:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001413924 2021-03-15T22:32:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.106889+0000", 2021-03-15T22:32:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5030000000000003e-05 2021-03-15T22:32:01.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.106984+0000", 2021-03-15T22:32:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014550399999999999 2021-03-15T22:32:01.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:32:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107129+0000", 2021-03-15T22:32:01.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001729529 2021-03-15T22:32:01.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108859+0000", 2021-03-15T22:32:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023022799999999999 2021-03-15T22:32:01.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.109089+0000", 2021-03-15T22:32:01.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074107424000000005 2021-03-15T22:32:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:32:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183197+0000", 2021-03-15T22:32:01.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7657e-05 2021-03-15T22:32:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:01.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183224+0000", 2021-03-15T22:32:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2325e-05 2021-03-15T22:32:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:01.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183257+0000", 2021-03-15T22:32:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077795487999999996 2021-03-15T22:32:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:01.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2770 7.b 7:d10ba14d:::smithi07694878-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e310)", 2021-03-15T22:32:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:32:01.820 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 496.66838513200003, 2021-03-15T22:32:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077923611000000004, 2021-03-15T22:32:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:01.821 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:32:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:01.822 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2770 2021-03-15T22:32:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:01.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:32:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:32:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3200000000000001e-06 2021-03-15T22:32:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105493+0000", 2021-03-15T22:32:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3960000000000001e-06 2021-03-15T22:32:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105494+0000", 2021-03-15T22:32:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6490000000000002e-06 2021-03-15T22:32:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105501+0000", 2021-03-15T22:32:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9989999999999999e-06 2021-03-15T22:32:01.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105506+0000", 2021-03-15T22:32:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019212770000000001 2021-03-15T22:32:01.829 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107427+0000", 2021-03-15T22:32:01.831 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8102000000000005e-05 2021-03-15T22:32:01.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:01.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107515+0000", 2021-03-15T22:32:01.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075777318999999996 2021-03-15T22:32:01.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183292+0000", 2021-03-15T22:32:01.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9417999999999999e-05 2021-03-15T22:32:01.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:01.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183352+0000", 2021-03-15T22:32:01.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3131000000000003e-05 2021-03-15T22:32:01.834 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:01.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183415+0000", 2021-03-15T22:32:01.835 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077923611000000004 2021-03-15T22:32:01.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.836 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:01.836 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.837 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3010 7.b 7:d980be1f:::smithi07694878-23:head [write 726168~704991,stat] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:32:01.838 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:32:01.838 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 478.687363554, 2021-03-15T22:32:01.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0570805230000002, 2021-03-15T22:32:01.838 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:01.839 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:01.839 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:01.839 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:01.839 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:01.840 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3010 2021-03-15T22:32:01.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.840 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:01.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:01.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:32:01.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:01.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:01.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:32:01.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034716800000000002 2021-03-15T22:32:01.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:01.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086860+0000", 2021-03-15T22:32:01.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3299999999999999e-06 2021-03-15T22:32:01.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:01.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086863+0000", 2021-03-15T22:32:01.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1489e-05 2021-03-15T22:32:01.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:01.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086875+0000", 2021-03-15T22:32:01.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.605e-06 2021-03-15T22:32:01.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.847 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:01.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086882+0000", 2021-03-15T22:32:01.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5296999999999997e-05 2021-03-15T22:32:01.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086928+0000", 2021-03-15T22:32:01.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2879999999999998e-06 2021-03-15T22:32:01.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:32:01.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086937+0000", 2021-03-15T22:32:01.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99891912599999999 2021-03-15T22:32:01.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085856+0000", 2021-03-15T22:32:01.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1169999999999993e-06 2021-03-15T22:32:01.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:32:01.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085865+0000", 2021-03-15T22:32:01.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0140395639999999 2021-03-15T22:32:01.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.099905+0000", 2021-03-15T22:32:01.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022436600000000001 2021-03-15T22:32:01.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:01.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.100129+0000", 2021-03-15T22:32:01.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00043622700000000002 2021-03-15T22:32:01.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:32:01.855 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.100565+0000", 2021-03-15T22:32:01.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024326999999999999 2021-03-15T22:32:01.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:32:01.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102998+0000", 2021-03-15T22:32:01.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002982012 2021-03-15T22:32:01.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.105980+0000", 2021-03-15T22:32:01.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037236714999999997 2021-03-15T22:32:01.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143217+0000", 2021-03-15T22:32:01.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147832 2021-03-15T22:32:01.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:01.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143365+0000", 2021-03-15T22:32:01.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022868699999999999 2021-03-15T22:32:01.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:01.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143593+0000", 2021-03-15T22:32:01.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0570805230000002 2021-03-15T22:32:01.861 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.861 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:01.861 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.862 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3011 7.b 7:d980be1f:::smithi07694878-23:head [write 1895071~683474] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:32:01.862 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:32:01.862 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 478.68108330799998, 2021-03-15T22:32:01.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.09731113, 2021-03-15T22:32:01.863 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:01.863 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:01.863 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:01.863 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:01.864 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:01.864 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3011 2021-03-15T22:32:01.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.864 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:01.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:01.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:32:01.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:01.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:01.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:32:01.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027883500000000001 2021-03-15T22:32:01.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:01.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093072+0000", 2021-03-15T22:32:01.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.809e-06 2021-03-15T22:32:01.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:01.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093077+0000", 2021-03-15T22:32:01.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9491e-05 2021-03-15T22:32:01.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:01.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093096+0000", 2021-03-15T22:32:01.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.126e-06 2021-03-15T22:32:01.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:01.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093103+0000", 2021-03-15T22:32:01.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4002000000000001e-05 2021-03-15T22:32:01.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093167+0000", 2021-03-15T22:32:01.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0259999999999996e-06 2021-03-15T22:32:01.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:32:01.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093176+0000", 2021-03-15T22:32:01.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99270957599999998 2021-03-15T22:32:01.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085886+0000", 2021-03-15T22:32:01.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9440000000000002e-06 2021-03-15T22:32:01.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.874 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:32:01.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085892+0000", 2021-03-15T22:32:01.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.016173532 2021-03-15T22:32:01.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102065+0000", 2021-03-15T22:32:01.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000141037 2021-03-15T22:32:01.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:01.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102207+0000", 2021-03-15T22:32:01.877 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041447062 2021-03-15T22:32:01.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143654+0000", 2021-03-15T22:32:01.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014246199999999999 2021-03-15T22:32:01.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:01.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143796+0000", 2021-03-15T22:32:01.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000272444 2021-03-15T22:32:01.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:32:01.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.144068+0000", 2021-03-15T22:32:01.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018956929999999999 2021-03-15T22:32:01.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145964+0000", 2021-03-15T22:32:01.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041997686999999999 2021-03-15T22:32:01.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:32:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.187962+0000", 2021-03-15T22:32:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019156150000000001 2021-03-15T22:32:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.189877+0000", 2021-03-15T22:32:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8467999999999995e-05 2021-03-15T22:32:01.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.189956+0000", 2021-03-15T22:32:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014832100000000001 2021-03-15T22:32:01.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190104+0000", 2021-03-15T22:32:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.09731113 2021-03-15T22:32:01.886 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3012 7.b 7:d980be1f:::smithi07694878-23:head [setxattr _header (58) in=7b,truncate 3126168] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:32:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:32:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 478.68076047599999, 2021-03-15T22:32:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0972930829999998, 2021-03-15T22:32:01.888 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:01.889 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3012 2021-03-15T22:32:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:32:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:32:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.61e-06 2021-03-15T22:32:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093118+0000", 2021-03-15T22:32:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.234e-06 2021-03-15T22:32:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:01.894 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093119+0000", 2021-03-15T22:32:01.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7290000000000002e-06 2021-03-15T22:32:01.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:01.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093123+0000", 2021-03-15T22:32:01.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1649999999999999e-06 2021-03-15T22:32:01.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:01.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093127+0000", 2021-03-15T22:32:01.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6592e-05 2021-03-15T22:32:01.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093213+0000", 2021-03-15T22:32:01.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2364000000000001e-05 2021-03-15T22:32:01.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:32:01.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093226+0000", 2021-03-15T22:32:01.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99268559899999997 2021-03-15T22:32:01.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085911+0000", 2021-03-15T22:32:01.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1440000000000002e-06 2021-03-15T22:32:01.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:32:01.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085916+0000", 2021-03-15T22:32:01.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0164949109999999 2021-03-15T22:32:01.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102411+0000", 2021-03-15T22:32:01.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000116049 2021-03-15T22:32:01.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:01.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102527+0000", 2021-03-15T22:32:01.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042432780000000003 2021-03-15T22:32:01.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.144960+0000", 2021-03-15T22:32:01.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7323000000000003e-05 2021-03-15T22:32:01.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:01.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145057+0000", 2021-03-15T22:32:01.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018096599999999999 2021-03-15T22:32:01.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:32:01.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145238+0000", 2021-03-15T22:32:01.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077369500000000003 2021-03-15T22:32:01.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.146012+0000", 2021-03-15T22:32:01.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041960274999999998 2021-03-15T22:32:01.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:32:01.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.187972+0000", 2021-03-15T22:32:01.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021867610000000002 2021-03-15T22:32:01.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190159+0000", 2021-03-15T22:32:01.909 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9751000000000004e-05 2021-03-15T22:32:01.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.910 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:01.910 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190229+0000", 2021-03-15T22:32:01.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000180135 2021-03-15T22:32:01.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:01.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190409+0000", 2021-03-15T22:32:01.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0972930829999998 2021-03-15T22:32:01.911 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.912 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:01.912 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.912 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3013 7.b 7:d980be1f:::smithi07694878-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e328)", 2021-03-15T22:32:01.913 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:32:01.913 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 478.68072718799999, 2021-03-15T22:32:01.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.097631684, 2021-03-15T22:32:01.913 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:01.914 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:32:01.914 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:01.914 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:01.914 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:01.914 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3013 2021-03-15T22:32:01.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.915 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:01.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:01.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:32:01.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:01.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:01.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:32:01.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9600000000000008e-07 2021-03-15T22:32:01.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:01.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093150+0000", 2021-03-15T22:32:01.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.514e-06 2021-03-15T22:32:01.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:01.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093152+0000", 2021-03-15T22:32:01.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2985e-05 2021-03-15T22:32:01.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:01.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093165+0000", 2021-03-15T22:32:01.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4000000000000002e-06 2021-03-15T22:32:01.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:01.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093169+0000", 2021-03-15T22:32:01.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2490999999999993e-05 2021-03-15T22:32:01.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.922 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093252+0000", 2021-03-15T22:32:01.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7490000000000001e-06 2021-03-15T22:32:01.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:32:01.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093258+0000", 2021-03-15T22:32:01.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99267630200000001 2021-03-15T22:32:01.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.924 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085935+0000", 2021-03-15T22:32:01.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0580000000000003e-06 2021-03-15T22:32:01.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.925 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:32:01.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085941+0000", 2021-03-15T22:32:01.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.01663009 2021-03-15T22:32:01.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102571+0000", 2021-03-15T22:32:01.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012747 2021-03-15T22:32:01.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:01.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102698+0000", 2021-03-15T22:32:01.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042823333999999998 2021-03-15T22:32:01.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145522+0000", 2021-03-15T22:32:01.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7650000000000004e-05 2021-03-15T22:32:01.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:01.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145599+0000", 2021-03-15T22:32:01.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044843263000000001 2021-03-15T22:32:01.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190443+0000", 2021-03-15T22:32:01.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000117391 2021-03-15T22:32:01.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:01.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190560+0000", 2021-03-15T22:32:01.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000220991 2021-03-15T22:32:01.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:01.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190781+0000", 2021-03-15T22:32:01.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.097631684 2021-03-15T22:32:01.933 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.934 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:01.934 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.934 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:3668 7.4 e385/358 7:26337d05:::smithi07694878-20:head v 385'309, mlcod=377'306)", 2021-03-15T22:32:01.935 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:32:01.935 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 421.59548875199999, 2021-03-15T22:32:01.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054245017999999999, 2021-03-15T22:32:01.935 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:01.936 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:01.936 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:01.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:01.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:32:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:01.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.178388+0000", 2021-03-15T22:32:01.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00068176800000000002 2021-03-15T22:32:01.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:01.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179069+0000", 2021-03-15T22:32:01.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1430000000000001e-06 2021-03-15T22:32:01.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:01.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179077+0000", 2021-03-15T22:32:01.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9515000000000001e-05 2021-03-15T22:32:01.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:01.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179106+0000", 2021-03-15T22:32:01.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0193e-05 2021-03-15T22:32:01.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:01.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179116+0000", 2021-03-15T22:32:01.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8248000000000003e-05 2021-03-15T22:32:01.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179165+0000", 2021-03-15T22:32:01.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8050000000000001e-05 2021-03-15T22:32:01.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:01.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.179193+0000", 2021-03-15T22:32:01.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053391015999999999 2021-03-15T22:32:01.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:01.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.232584+0000", 2021-03-15T22:32:01.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9085000000000002e-05 2021-03-15T22:32:01.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:01.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:00.232633+0000", 2021-03-15T22:32:01.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.054245017999999999 2021-03-15T22:32:01.947 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.947 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:01.947 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.948 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4284 7.b 7:d4368cae:::smithi07694878-18:head [write 3203532~509848] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:32:01.948 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:32:01.949 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 361.611098218, 2021-03-15T22:32:01.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068311582999999995, 2021-03-15T22:32:01.949 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:01.949 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:01.949 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:01.950 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:01.950 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:01.950 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4284 2021-03-15T22:32:01.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.951 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:01.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:01.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:32:01.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:01.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:01.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:32:01.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015451700000000001 2021-03-15T22:32:01.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:01.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162933+0000", 2021-03-15T22:32:01.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.25e-06 2021-03-15T22:32:01.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:01.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162937+0000", 2021-03-15T22:32:01.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2824e-05 2021-03-15T22:32:01.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:01.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162960+0000", 2021-03-15T22:32:01.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.072e-06 2021-03-15T22:32:01.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:01.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162968+0000", 2021-03-15T22:32:01.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7661999999999999e-05 2021-03-15T22:32:01.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163016+0000", 2021-03-15T22:32:01.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108743 2021-03-15T22:32:01.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:01.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163124+0000", 2021-03-15T22:32:01.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021732500000000001 2021-03-15T22:32:01.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:32:01.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163342+0000", 2021-03-15T22:32:01.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043853354999999997 2021-03-15T22:32:01.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.207195+0000", 2021-03-15T22:32:01.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055661469999999996 2021-03-15T22:32:01.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.212761+0000", 2021-03-15T22:32:01.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018240684 2021-03-15T22:32:01.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:32:01.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231002+0000", 2021-03-15T22:32:01.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6122000000000003e-05 2021-03-15T22:32:01.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:01.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231048+0000", 2021-03-15T22:32:01.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1882000000000003e-05 2021-03-15T22:32:01.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:01.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231090+0000", 2021-03-15T22:32:01.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068311582999999995 2021-03-15T22:32:01.966 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.966 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:01.966 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.967 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4285 7.b 7:d4368cae:::smithi07694878-18:head [setxattr _header (58) in=7b,truncate 3865709] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:32:01.967 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:32:01.967 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 361.61089374599999, 2021-03-15T22:32:01.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068193614999999999, 2021-03-15T22:32:01.968 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:01.968 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:01.968 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:01.969 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:01.969 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:01.969 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4285 2021-03-15T22:32:01.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.969 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:01.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:01.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:32:01.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:01.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:01.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:32:01.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.633e-06 2021-03-15T22:32:01.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:01.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162985+0000", 2021-03-15T22:32:01.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6789999999999999e-06 2021-03-15T22:32:01.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:01.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162987+0000", 2021-03-15T22:32:01.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9500000000000001e-06 2021-03-15T22:32:01.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:01.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162995+0000", 2021-03-15T22:32:01.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.459e-06 2021-03-15T22:32:01.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:01.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162999+0000", 2021-03-15T22:32:01.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012207240000000001 2021-03-15T22:32:01.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164220+0000", 2021-03-15T22:32:01.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0790000000000004e-05 2021-03-15T22:32:01.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:01.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164301+0000", 2021-03-15T22:32:01.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014765399999999999 2021-03-15T22:32:01.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:32:01.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164449+0000", 2021-03-15T22:32:01.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042818110999999999 2021-03-15T22:32:01.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.207267+0000", 2021-03-15T22:32:01.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055270550000000003 2021-03-15T22:32:01.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:01.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.212794+0000", 2021-03-15T22:32:01.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018305267 2021-03-15T22:32:01.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:32:01.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231099+0000", 2021-03-15T22:32:01.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4928000000000002e-05 2021-03-15T22:32:01.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:01.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231134+0000", 2021-03-15T22:32:01.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2364999999999998e-05 2021-03-15T22:32:01.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231176+0000", 2021-03-15T22:32:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068193614999999999 2021-03-15T22:32:01.984 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.985 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:01.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:01.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.985 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4286 7.b 7:d4368cae:::smithi07694878-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e445)", 2021-03-15T22:32:01.986 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:32:01.986 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 361.61087209800002, 2021-03-15T22:32:01.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068364477000000007, 2021-03-15T22:32:01.986 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:01.987 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:32:01.987 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:01.987 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:01.987 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:01.987 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4286 2021-03-15T22:32:01.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.988 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:01.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:01.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:32:01.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:01.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:01.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:32:01.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6500000000000008e-07 2021-03-15T22:32:01.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:01.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163005+0000", 2021-03-15T22:32:01.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.218e-06 2021-03-15T22:32:01.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:01.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163007+0000", 2021-03-15T22:32:01.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3890000000000002e-06 2021-03-15T22:32:01.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:01.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163013+0000", 2021-03-15T22:32:01.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.053e-06 2021-03-15T22:32:01.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:01.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163017+0000", 2021-03-15T22:32:01.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017994630000000001 2021-03-15T22:32:01.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164816+0000", 2021-03-15T22:32:01.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0958000000000007e-05 2021-03-15T22:32:01.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:01.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164897+0000", 2021-03-15T22:32:01.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066313677000000001 2021-03-15T22:32:01.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:01.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231211+0000", 2021-03-15T22:32:01.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9739000000000004e-05 2021-03-15T22:32:01.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:01.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231281+0000", 2021-03-15T22:32:01.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8015000000000003e-05 2021-03-15T22:32:01.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:01.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:01.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:01.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231369+0000", 2021-03-15T22:32:01.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068364477000000007 2021-03-15T22:32:02.000 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.000 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:02.000 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.001 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4287 7.7 e445/420 7:ed4a01d2:::smithi07694878-2:head v 445'413, mlcod=444'410)", 2021-03-15T22:32:02.001 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:32:02.001 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 361.60985380199998, 2021-03-15T22:32:02.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066973671999999998, 2021-03-15T22:32:02.002 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:02.002 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:02.002 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:02.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:02.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:32:02.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:02.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:02.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:32:02.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.258e-06 2021-03-15T22:32:02.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:02.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164026+0000", 2021-03-15T22:32:02.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7980000000000001e-06 2021-03-15T22:32:02.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:02.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164029+0000", 2021-03-15T22:32:02.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0787e-05 2021-03-15T22:32:02.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:02.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164039+0000", 2021-03-15T22:32:02.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2829999999999998e-06 2021-03-15T22:32:02.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:02.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164046+0000", 2021-03-15T22:32:02.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5735000000000001e-05 2021-03-15T22:32:02.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164101+0000", 2021-03-15T22:32:02.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9206000000000001e-05 2021-03-15T22:32:02.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:02.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164131+0000", 2021-03-15T22:32:02.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066830296999999997 2021-03-15T22:32:02.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:02.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.230961+0000", 2021-03-15T22:32:02.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5308000000000001e-05 2021-03-15T22:32:02.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:02.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.230996+0000", 2021-03-15T22:32:02.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066973671999999998 2021-03-15T22:32:02.013 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.013 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:02.013 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.014 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4288 7.7 e445/420 7:ed4a01d2:::smithi07694878-2:head v 445'414, mlcod=444'410)", 2021-03-15T22:32:02.014 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.164500+0000", 2021-03-15T22:32:02.014 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 361.60937630799998, 2021-03-15T22:32:02.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066520592000000003, 2021-03-15T22:32:02.015 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:02.015 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:02.015 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:02.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:02.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164500+0000", 2021-03-15T22:32:02.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:02.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:02.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164500+0000", 2021-03-15T22:32:02.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9610000000000001e-06 2021-03-15T22:32:02.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:02.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164503+0000", 2021-03-15T22:32:02.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.407e-06 2021-03-15T22:32:02.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:02.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164505+0000", 2021-03-15T22:32:02.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8310000000000001e-06 2021-03-15T22:32:02.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:02.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164513+0000", 2021-03-15T22:32:02.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9269999999999998e-06 2021-03-15T22:32:02.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:02.021 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164517+0000", 2021-03-15T22:32:02.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6690999999999998e-05 2021-03-15T22:32:02.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164554+0000", 2021-03-15T22:32:02.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6207000000000002e-05 2021-03-15T22:32:02.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:02.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164580+0000", 2021-03-15T22:32:02.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066419645999999999 2021-03-15T22:32:02.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:02.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231000+0000", 2021-03-15T22:32:02.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0922000000000001e-05 2021-03-15T22:32:02.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:02.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231021+0000", 2021-03-15T22:32:02.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066520592000000003 2021-03-15T22:32:02.026 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.026 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:02.026 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.027 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:5830 7.5 e631/559 7:ac84e656:::smithi07694878-31:head v 631'458, mlcod=630'454)", 2021-03-15T22:32:02.027 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:32:02.027 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 141.522216587, 2021-03-15T22:32:02.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083826879000000007, 2021-03-15T22:32:02.028 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:02.028 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:02.028 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:02.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:02.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:32:02.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:02.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:02.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:32:02.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060822330000000003 2021-03-15T22:32:02.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:02.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257742+0000", 2021-03-15T22:32:02.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9649999999999996e-06 2021-03-15T22:32:02.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:02.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257748+0000", 2021-03-15T22:32:02.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0319e-05 2021-03-15T22:32:02.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:02.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257768+0000", 2021-03-15T22:32:02.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2021-03-15T22:32:02.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:02.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257777+0000", 2021-03-15T22:32:02.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0096000000000001e-05 2021-03-15T22:32:02.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257847+0000", 2021-03-15T22:32:02.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7812000000000001e-05 2021-03-15T22:32:02.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:02.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257875+0000", 2021-03-15T22:32:02.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077557744999999997 2021-03-15T22:32:02.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:02.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.335433+0000", 2021-03-15T22:32:02.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3674999999999997e-05 2021-03-15T22:32:02.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.038 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:02.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.335487+0000", 2021-03-15T22:32:02.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083826879000000007 2021-03-15T22:32:02.039 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.039 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:02.039 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.040 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:755 7.5 e646/559 7:a9545d62:::smithi07694878-1:1fc v 646'483, mlcod=644'479)", 2021-03-15T22:32:02.040 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:32:02.040 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 126.527592231, 2021-03-15T22:32:02.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087969915999999995, 2021-03-15T22:32:02.041 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:02.041 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:02.041 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:02.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:02.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:32:02.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:02.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:02.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:32:02.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1300000000000001e-06 2021-03-15T22:32:02.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:02.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246287+0000", 2021-03-15T22:32:02.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1730000000000002e-06 2021-03-15T22:32:02.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:02.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246290+0000", 2021-03-15T22:32:02.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1728e-05 2021-03-15T22:32:02.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:02.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246301+0000", 2021-03-15T22:32:02.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7710000000000001e-06 2021-03-15T22:32:02.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:02.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246308+0000", 2021-03-15T22:32:02.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5705999999999999e-05 2021-03-15T22:32:02.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246354+0000", 2021-03-15T22:32:02.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8252e-05 2021-03-15T22:32:02.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:02.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246382+0000", 2021-03-15T22:32:02.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087836030999999995 2021-03-15T22:32:02.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:02.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334218+0000", 2021-03-15T22:32:02.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6124999999999997e-05 2021-03-15T22:32:02.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:02.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334254+0000", 2021-03-15T22:32:02.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087969915999999995 2021-03-15T22:32:02.052 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.052 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:02.052 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.053 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6447 7.b 7:d980be1f:::smithi07694878-23:21c [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e691)", 2021-03-15T22:32:02.053 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.215534+0000", 2021-03-15T22:32:02.053 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 81.558342674000002, 2021-03-15T22:32:02.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079694523000000003, 2021-03-15T22:32:02.054 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:02.054 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:32:02.054 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:02.054 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:02.055 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:02.055 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6447 2021-03-15T22:32:02.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.055 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:02.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:02.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215534+0000", 2021-03-15T22:32:02.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:02.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:02.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215534+0000", 2021-03-15T22:32:02.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2140000000000004e-06 2021-03-15T22:32:02.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.058 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:02.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215539+0000", 2021-03-15T22:32:02.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8670000000000004e-06 2021-03-15T22:32:02.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:02.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215543+0000", 2021-03-15T22:32:02.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4408e-05 2021-03-15T22:32:02.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:02.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215557+0000", 2021-03-15T22:32:02.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4090000000000003e-06 2021-03-15T22:32:02.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:02.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215566+0000", 2021-03-15T22:32:02.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2050000000000002e-05 2021-03-15T22:32:02.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215648+0000", 2021-03-15T22:32:02.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8050000000000007e-06 2021-03-15T22:32:02.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:32:02.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215656+0000", 2021-03-15T22:32:02.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079466859000000001 2021-03-15T22:32:02.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.064 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.065 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295122+0000", 2021-03-15T22:32:02.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000105911 2021-03-15T22:32:02.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.066 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:02.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295228+0000", 2021-03-15T22:32:02.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079694523000000003 2021-03-15T22:32:02.066 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.066 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:02.067 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.067 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6448 7.b 7:d980be1f:::smithi07694878-23:21c [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e691)", 2021-03-15T22:32:02.068 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.215580+0000", 2021-03-15T22:32:02.068 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 81.558295939000004, 2021-03-15T22:32:02.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079736783000000006, 2021-03-15T22:32:02.068 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:02.068 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:32:02.069 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:02.069 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:02.069 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:02.069 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6448 2021-03-15T22:32:02.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.070 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:02.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:02.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215580+0000", 2021-03-15T22:32:02.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:02.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:02.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215580+0000", 2021-03-15T22:32:02.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.248e-06 2021-03-15T22:32:02.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:02.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215582+0000", 2021-03-15T22:32:02.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.646e-06 2021-03-15T22:32:02.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:02.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215583+0000", 2021-03-15T22:32:02.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4140000000000002e-06 2021-03-15T22:32:02.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:02.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215589+0000", 2021-03-15T22:32:02.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9349999999999996e-06 2021-03-15T22:32:02.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.076 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:02.076 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215593+0000", 2021-03-15T22:32:02.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000158023 2021-03-15T22:32:02.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215751+0000", 2021-03-15T22:32:02.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2480000000000004e-06 2021-03-15T22:32:02.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:32:02.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215756+0000", 2021-03-15T22:32:02.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079496878000000007 2021-03-15T22:32:02.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.079 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295253+0000", 2021-03-15T22:32:02.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4391000000000001e-05 2021-03-15T22:32:02.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:02.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295317+0000", 2021-03-15T22:32:02.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079736783000000006 2021-03-15T22:32:02.080 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.081 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:02.081 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.082 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6449 7.b 7:d980be1f:::smithi07694878-23:21c [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected e691)", 2021-03-15T22:32:02.082 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.215597+0000", 2021-03-15T22:32:02.082 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 81.558279647000006, 2021-03-15T22:32:02.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079805300999999995, 2021-03-15T22:32:02.082 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:02.083 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:32:02.083 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:32:02.083 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:32:02.083 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:02.083 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6449 2021-03-15T22:32:02.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.084 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:02.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:02.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215597+0000", 2021-03-15T22:32:02.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:02.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:02.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215597+0000", 2021-03-15T22:32:02.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.621e-06 2021-03-15T22:32:02.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:02.087 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215598+0000", 2021-03-15T22:32:02.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.468e-06 2021-03-15T22:32:02.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:02.088 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215600+0000", 2021-03-15T22:32:02.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7419999999999997e-06 2021-03-15T22:32:02.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215605+0000", 2021-03-15T22:32:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4640000000000002e-06 2021-03-15T22:32:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215608+0000", 2021-03-15T22:32:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016377699999999999 2021-03-15T22:32:02.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215772+0000", 2021-03-15T22:32:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3549999999999998e-06 2021-03-15T22:32:02.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:32:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215776+0000", 2021-03-15T22:32:02.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079562503000000007 2021-03-15T22:32:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295339+0000", 2021-03-15T22:32:02.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3371e-05 2021-03-15T22:32:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:02.094 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295402+0000", 2021-03-15T22:32:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079805300999999995 2021-03-15T22:32:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.095 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:6453 7.9 e691/556 7:91d654eb:::smithi07694878-5:head v 691'298, mlcod=681'295)", 2021-03-15T22:32:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.298138+0000", 2021-03-15T22:32:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 81.475738390000004, 2021-03-15T22:32:02.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080231284999999999, 2021-03-15T22:32:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:02.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298138+0000", 2021-03-15T22:32:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:02.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298138+0000", 2021-03-15T22:32:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012879800000000001 2021-03-15T22:32:02.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298267+0000", 2021-03-15T22:32:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6959999999999998e-06 2021-03-15T22:32:02.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298271+0000", 2021-03-15T22:32:02.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999994e-06 2021-03-15T22:32:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298280+0000", 2021-03-15T22:32:02.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2299999999999996e-06 2021-03-15T22:32:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:02.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298286+0000", 2021-03-15T22:32:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045678599999999999 2021-03-15T22:32:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298743+0000", 2021-03-15T22:32:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4567999999999999e-05 2021-03-15T22:32:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298768+0000", 2021-03-15T22:32:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079546108000000004 2021-03-15T22:32:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378314+0000", 2021-03-15T22:32:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5658999999999997e-05 2021-03-15T22:32:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.107 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378369+0000", 2021-03-15T22:32:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080231284999999999 2021-03-15T22:32:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.108 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.5.0:383 7.0 e762/642 7:08414d91:::smithi07694878-42:25b v 762'2409, mlcod=757'2405)", 2021-03-15T22:32:02.109 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:52.297106+0000", 2021-03-15T22:32:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 9.4767703569999995, 2021-03-15T22:32:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013731673, 2021-03-15T22:32:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:32:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:02.110 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:32:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:32:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:52.297106+0000", 2021-03-15T22:32:02.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:32:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:32:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:52.297106+0000", 2021-03-15T22:32:02.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3169999999999998e-06 2021-03-15T22:32:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:32:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:52.297112+0000", 2021-03-15T22:32:02.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.571e-06 2021-03-15T22:32:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:32:02.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:52.297115+0000", 2021-03-15T22:32:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8399999999999997e-06 2021-03-15T22:32:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:32:02.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:52.297122+0000", 2021-03-15T22:32:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7830000000000004e-06 2021-03-15T22:32:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:32:02.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:52.297128+0000", 2021-03-15T22:32:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0085726819999999999 2021-03-15T22:32:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:32:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:52.305700+0000", 2021-03-15T22:32:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6892999999999999e-05 2021-03-15T22:32:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:32:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:52.305717+0000", 2021-03-15T22:32:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050267560000000003 2021-03-15T22:32:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:32:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:52.310744+0000", 2021-03-15T22:32:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3831000000000003e-05 2021-03-15T22:32:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:02.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:32:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:52.310838+0000", 2021-03-15T22:32:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.013731673 2021-03-15T22:32:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.121 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:02.122 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:32:02.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:02.124 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:01 smithi125 conmon[56739]: cluster 2021-03-15T22:31:59.864454+0000 osd.1 (osd.1) 145 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:32:02.124 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:01 smithi125 conmon[56739]: audit 2021-03-15T22:32:00.094006+0000 mgr.y (mgr.14144) 919 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:02.124 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:01 smithi125 conmon[56739]: audit 2021-03-15T22:32:01.111074+0000 mon.a (mon.0) 2019 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:32:02.125 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:01 smithi125 conmon[56739]: audit 2021-03-15T22:32:01.112487+0000 mon.a (mon.0) 2020 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:32:02.125 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:01 smithi125 conmon[56739]: cluster 2021-03-15T22:32:01.250236+0000 mon.a (mon.0) 2021 : cluster [DBG] osdmap e771: 8 total, 8 up, 7 in 2021-03-15T22:32:02.126 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:32:02.126 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:32:02.126 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:32:02.127 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:32:02.127 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:32:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:32:02.127 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:32:02.128 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:32:02.128 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:32:02.128 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:32:02.128 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:32:02.129 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:32:02.129 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:32:02.151 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:02.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:02.153 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:02.254 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:02.255 INFO:tasks.rados.rados.0.smithi076.stdout:2873: done (3 left) 2021-03-15T22:32:02.256 INFO:tasks.rados.rados.0.smithi076.stdout:2874: write oid 37 current snap is 333 2021-03-15T22:32:02.256 INFO:tasks.rados.rados.0.smithi076.stdout:2872: finishing rollback tid 1 to smithi07694878-17 2021-03-15T22:32:02.256 INFO:tasks.rados.rados.0.smithi076.stdout:2874: seq_num 992 ranges {672544=572898,1666255=532998} 2021-03-15T22:32:02.259 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.2 to 1 (10000) 2021-03-15T22:32:02.261 INFO:tasks.rados.rados.0.smithi076.stdout:2874: writing smithi07694878-37 from 672544 to 1245442 tid 1 2021-03-15T22:32:02.267 INFO:tasks.rados.rados.0.smithi076.stdout:2874: writing smithi07694878-37 from 1666255 to 2199253 tid 2 2021-03-15T22:32:02.267 INFO:tasks.rados.rados.0.smithi076.stdout:2875: delete oid 4 current snap is 333 2021-03-15T22:32:02.267 INFO:tasks.rados.rados.0.smithi076.stdout:2872: finishing rollback tid 2 to smithi07694878-17 2021-03-15T22:32:02.268 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 859 (ObjNum 960 snap 321 seq_num 960) dirty exists 2021-03-15T22:32:02.268 INFO:tasks.rados.rados.0.smithi076.stdout:2870: finishing write tid 2 to smithi07694878-2 2021-03-15T22:32:02.268 INFO:tasks.rados.rados.0.smithi076.stdout:2870: finishing write tid 3 to smithi07694878-2 2021-03-15T22:32:02.268 INFO:tasks.rados.rados.0.smithi076.stdout:2870: finishing write tid 4 to smithi07694878-2 2021-03-15T22:32:02.268 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1972 (ObjNum 991 snap 333 seq_num 991) dirty exists 2021-03-15T22:32:02.269 INFO:tasks.rados.rados.0.smithi076.stdout:2870: left oid 2 (ObjNum 991 snap 333 seq_num 991) 2021-03-15T22:32:02.269 INFO:tasks.rados.rados.0.smithi076.stdout:2871: finishing rollback tid 1 to smithi07694878-9 2021-03-15T22:32:02.269 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 1888 (ObjNum 264850016 snap 21896 seq_num 0) dirty dne 2021-03-15T22:32:02.339 INFO:tasks.rados.rados.0.smithi076.stdout:2870: done (4 left) 2021-03-15T22:32:02.339 INFO:tasks.rados.rados.0.smithi076.stdout:2871: done (3 left) 2021-03-15T22:32:02.340 INFO:tasks.rados.rados.0.smithi076.stdout:2872: done (2 left) 2021-03-15T22:32:02.340 INFO:tasks.rados.rados.0.smithi076.stdout:2875: done (1 left) 2021-03-15T22:32:02.340 INFO:tasks.rados.rados.0.smithi076.stdout:2876: read oid 50 snap -1 2021-03-15T22:32:02.340 INFO:tasks.rados.rados.0.smithi076.stdout:2876: expect (ObjNum 598 snap 183 seq_num 598) 2021-03-15T22:32:02.340 INFO:tasks.rados.rados.0.smithi076.stdout:2877: write oid 7 current snap is 333 2021-03-15T22:32:02.341 INFO:tasks.rados.rados.0.smithi076.stdout:2877: seq_num 994 ranges {706158=723396,2183219=522939} 2021-03-15T22:32:02.347 INFO:tasks.rados.rados.0.smithi076.stdout:2877: writing smithi07694878-7 from 706158 to 1429554 tid 1 2021-03-15T22:32:02.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:02.354 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:02.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:02.356 INFO:tasks.rados.rados.0.smithi076.stdout:2877: writing smithi07694878-7 from 2183219 to 2706158 tid 2 2021-03-15T22:32:02.356 INFO:tasks.rados.rados.0.smithi076.stdout:2878: delete oid 43 current snap is 333 2021-03-15T22:32:02.356 INFO:tasks.rados.rados.0.smithi076.stdout:2876: expect (ObjNum 598 snap 183 seq_num 598) 2021-03-15T22:32:02.424 INFO:tasks.rados.rados.0.smithi076.stdout:2874: finishing write tid 1 to smithi07694878-37 2021-03-15T22:32:02.424 INFO:tasks.rados.rados.0.smithi076.stdout:2874: finishing write tid 2 to smithi07694878-37 2021-03-15T22:32:02.425 INFO:tasks.rados.rados.0.smithi076.stdout:2874: finishing write tid 3 to smithi07694878-37 2021-03-15T22:32:02.425 INFO:tasks.rados.rados.0.smithi076.stdout:2874: finishing write tid 4 to smithi07694878-37 2021-03-15T22:32:02.425 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 2470 (ObjNum 992 snap 333 seq_num 992) dirty exists 2021-03-15T22:32:02.425 INFO:tasks.rados.rados.0.smithi076.stdout:2874: left oid 37 (ObjNum 992 snap 333 seq_num 992) 2021-03-15T22:32:02.426 INFO:tasks.rados.rados.0.smithi076.stdout:2877: finishing write tid 1 to smithi07694878-7 2021-03-15T22:32:02.426 INFO:tasks.rados.rados.0.smithi076.stdout:2874: done (3 left) 2021-03-15T22:32:02.426 INFO:tasks.rados.rados.0.smithi076.stdout:2876: done (2 left) 2021-03-15T22:32:02.426 INFO:tasks.rados.rados.0.smithi076.stdout:2878: done (1 left) 2021-03-15T22:32:02.427 INFO:tasks.rados.rados.0.smithi076.stdout:2879: read oid 23 snap -1 2021-03-15T22:32:02.427 INFO:tasks.rados.rados.0.smithi076.stdout:2879: expect (ObjNum 955 snap 320 seq_num 955) 2021-03-15T22:32:02.427 INFO:tasks.rados.rados.0.smithi076.stdout:2880: copy_from oid 46 from oid 16 current snap is 333 2021-03-15T22:32:02.427 INFO:tasks.rados.rados.0.smithi076.stdout:2877: finishing write tid 2 to smithi07694878-7 2021-03-15T22:32:02.427 INFO:tasks.rados.rados.0.smithi076.stdout:2877: finishing write tid 3 to smithi07694878-7 2021-03-15T22:32:02.428 INFO:tasks.rados.rados.0.smithi076.stdout:2877: finishing write tid 4 to smithi07694878-7 2021-03-15T22:32:02.428 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2096 (ObjNum 994 snap 333 seq_num 994) dirty exists 2021-03-15T22:32:02.428 INFO:tasks.rados.rados.0.smithi076.stdout:2877: left oid 7 (ObjNum 994 snap 333 seq_num 994) 2021-03-15T22:32:02.428 INFO:tasks.rados.rados.0.smithi076.stdout:2877: done (2 left) 2021-03-15T22:32:02.429 INFO:tasks.rados.rados.0.smithi076.stdout:2881: read oid 3 snap -1 2021-03-15T22:32:02.429 INFO:tasks.rados.rados.0.smithi076.stdout:2881: expect deleted 2021-03-15T22:32:02.429 INFO:tasks.rados.rados.0.smithi076.stdout:2882: rollback oid 10 current snap is 333 2021-03-15T22:32:02.430 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 10 to 297 2021-03-15T22:32:02.430 INFO:tasks.rados.rados.0.smithi076.stdout:2883: delete oid 34 current snap is 333 2021-03-15T22:32:02.431 INFO:tasks.rados.rados.0.smithi076.stdout:2881: done (4 left) 2021-03-15T22:32:02.431 INFO:tasks.rados.rados.0.smithi076.stdout:2883: done (3 left) 2021-03-15T22:32:02.431 INFO:tasks.rados.rados.0.smithi076.stdout:2884: rollback oid 1 current snap is 333 2021-03-15T22:32:02.431 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 1 to 318 2021-03-15T22:32:02.432 INFO:tasks.rados.rados.0.smithi076.stdout:2885: delete oid 38 current snap is 333 2021-03-15T22:32:02.432 INFO:tasks.rados.rados.0.smithi076.stdout:2879: expect (ObjNum 955 snap 320 seq_num 955) 2021-03-15T22:32:02.445 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:32:02.455 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:02.480 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:02.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:02.482 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:02.513 INFO:tasks.rados.rados.0.smithi076.stdout:2882: finishing rollback tid 0 to smithi07694878-10 2021-03-15T22:32:02.513 INFO:tasks.rados.rados.0.smithi076.stdout:2882: finishing rollback tid 1 to smithi07694878-10 2021-03-15T22:32:02.513 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 1974 (ObjNum 262172032 snap 21896 seq_num 264952224) dirty dne 2021-03-15T22:32:02.514 INFO:tasks.rados.rados.0.smithi076.stdout:2884: finishing rollback tid 0 to smithi07694878-1 2021-03-15T22:32:02.514 INFO:tasks.rados.rados.0.smithi076.stdout:2884: finishing rollback tid 1 to smithi07694878-1 2021-03-15T22:32:02.514 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 1859 (ObjNum 1953066355 snap 925919592 seq_num 942946614) dirty dne 2021-03-15T22:32:02.514 INFO:tasks.rados.rados.0.smithi076.stdout:2880: finishing copy_from to smithi07694878-46 2021-03-15T22:32:02.515 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2272 (ObjNum 803 snap 268 seq_num 803) dirty exists 2021-03-15T22:32:02.515 INFO:tasks.rados.rados.0.smithi076.stdout:2880: finishing copy_from racing read to smithi07694878-46 2021-03-15T22:32:02.515 INFO:tasks.rados.rados.0.smithi076.stdout:2879: done (4 left) 2021-03-15T22:32:02.515 INFO:tasks.rados.rados.0.smithi076.stdout:2880: done (3 left) 2021-03-15T22:32:02.516 INFO:tasks.rados.rados.0.smithi076.stdout:2882: done (2 left) 2021-03-15T22:32:02.516 INFO:tasks.rados.rados.0.smithi076.stdout:2884: done (1 left) 2021-03-15T22:32:02.516 INFO:tasks.rados.rados.0.smithi076.stdout:2885: done (0 left) 2021-03-15T22:32:02.516 INFO:tasks.rados.rados.0.smithi076.stdout:2886: snap_create 2021-03-15T22:32:02.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:02.608 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:02.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:02.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:02.711 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:02.720 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[56514]: cluster 2021-03-15T22:32:00.816085+0000 osd.5 (osd.5) 26 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:32:02.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[56514]: cluster 2021-03-15T22:32:00.927833+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.d scrub ok 2021-03-15T22:32:02.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[56514]: cluster 2021-03-15T22:32:01.169716+0000 mgr.y (mgr.14144) 920 : cluster [DBG] pgmap v1300: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 543 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 7 op/s; 4/1752 objects misplaced (0.228%) 2021-03-15T22:32:02.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[56514]: cluster 2021-03-15T22:32:01.452882+0000 osd.0 (osd.0) 125 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:32:02.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[56514]: audit 2021-03-15T22:32:01.543953+0000 mon.a (mon.0) 2022 : audit [INF] from='client.? 172.21.15.76:0/2901003738' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-15T22:32:02.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[56514]: cluster 2021-03-15T22:32:01.586746+0000 osd.3 (osd.3) 140 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:32:02.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[56514]: audit 2021-03-15T22:32:01.711071+0000 mon.a (mon.0) 2023 : audit [DBG] from='client.? 172.21.15.76:0/1539220035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:02.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[56514]: audit 2021-03-15T22:32:02.093639+0000 mon.a (mon.0) 2024 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2021-03-15T22:32:02.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[56514]: audit 2021-03-15T22:32:02.252191+0000 mon.a (mon.0) 2025 : audit [INF] from='client.? 172.21.15.76:0/2901003738' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-15T22:32:02.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[56514]: audit 2021-03-15T22:32:02.252270+0000 mon.a (mon.0) 2026 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]': finished 2021-03-15T22:32:02.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[56514]: cluster 2021-03-15T22:32:02.252320+0000 mon.a (mon.0) 2027 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2021-03-15T22:32:02.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: cluster 2021-03-15T22:32:00.816085+0000 2021-03-15T22:32:02.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: osd.5 (osd.5) 26 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:32:02.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: cluster 2021-03-15T22:32:00. 2021-03-15T22:32:02.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 927833+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.d scrub ok 2021-03-15T22:32:02.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:32:02.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: -03-15T22:32:01.169716+0000 mgr.y ( 2021-03-15T22:32:02.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: mgr.14144) 920 : cluster [DBG] pgmap v1300: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 543 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 7 op/s; 4/1752 objects misplaced (0.228%) 2021-03-15T22:32:02.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 2021-03-15T22:32:02.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:32:02.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 32:01.452882+0000 osd.0 (osd.0) 2021-03-15T22:32:02.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 125 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:32:02.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: audit 2021-03-15T22:32:02.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 2021-03-15T22:32:01. 2021-03-15T22:32:02.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 543953+0000 mon.a (mon.0 2021-03-15T22:32:02.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: ) 2022 : audit [INF] from='client.? 172.21.15.76:0/2901003738' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-15T22:32:02.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 2021-03-15T22:32:02.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: cluster 2021-03-15T22:32:02.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:32:02.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 32:01.586746+0000 osd.3 ( 2021-03-15T22:32:02.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: osd.3) 140 : cluster 2021-03-15T22:32:02.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: [DBG] 4.17 scrub ok 2021-03-15T22:32:02.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: audit 2021-03-15T22:32:02.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:32:02.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: :32:01.711071+0000 mon.a ( 2021-03-15T22:32:02.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: mon.0) 2023 : audit 2021-03-15T22:32:02.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/1539220035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:02.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: audit 2021 2021-03-15T22:32:02.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: -03-15T22:32:02 2021-03-15T22:32:02.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: .093639+0000 mon.a (mon.0) 2024 : audit 2021-03-15T22:32:02.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2021-03-15T22:32:02.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:32:02.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 03-15T22:32:02.252191+0000 2021-03-15T22:32:02.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: mon.a (mon.0) 2025 : audit [INF] 2021-03-15T22:32:02.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2901003738' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-15T22:32:02.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: audit 2021-03-15T22:32:02.252270+0000 mon.a ( 2021-03-15T22:32:02.732 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: mon.0) 2026 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]': finished 2021-03-15T22:32:02.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:02.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 03-15T22:32:02 2021-03-15T22:32:02.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: .252320+0000 mon.a (mon. 2021-03-15T22:32:02.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 0) 2027 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2021-03-15T22:32:02.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:02 smithi076 conmon[62156]: 2021-03-15T22:32:02.736 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:02.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:02.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: cluster 2021-03-15T22:32:00.816085+0000 osd.5 (osd.5) 26 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:32:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: cluster 2021-03-15T22:32:00.927833+0000 osd.2 (osd.2) 20 : cluster [DBG] 5.d scrub ok 2021-03-15T22:32:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: cluster 2021-03-15T22:32:01.169716+0000 mgr.y (mgr.14144) 920 : cluster [DBG] pgmap v1300: 148 pgs: 3 active+remapped+backfill_toofull, 145 active+clean; 543 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.3 MiB/s rd, 1.1 MiB/s wr, 7 op/s; 4/1752 objects misplaced (0.228%) 2021-03-15T22:32:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: cluster 2021-03-15T22:32:02.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: 2021-03-15T22:32:01.452882+0000 osd.0 (osd.0) 125 : cluster [DBG] 2021-03-15T22:32:02.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: 6.18 scrub ok 2021-03-15T22:32:02.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:32:02.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: :32:01.543953+0000 mon.a (mon.0 2021-03-15T22:32:02.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: ) 2022 : audit [INF] from='client.? 172.21.15.76:0/2901003738' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2021-03-15T22:32:02.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:32:02.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: -03-15T22:32:01.586746 2021-03-15T22:32:02.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: +0000 osd.3 (osd.3) 140 : cluster 2021-03-15T22:32:02.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: [DBG] 4.17 scrub ok 2021-03-15T22:32:02.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:32:02.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: :32:01.711071+0000 mon.a (mon.0 2021-03-15T22:32:02.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: ) 2023 : audit [DBG] from='client.? 172.21.15.76:0/1539220035' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:02.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:32:02.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: -15T22:32:02.093639+0000 2021-03-15T22:32:02.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: mon.a (mon.0) 2024 : audit [INF] 2021-03-15T22:32:02.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]: dispatch 2021-03-15T22:32:02.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: audit 2021-03-15T22:32: 2021-03-15T22:32:02.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: 02.252191+0000 mon.a (mon.0) 2025 2021-03-15T22:32:02.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: : audit [INF] from='client.? 172.21.15.76:0/2901003738' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2021-03-15T22:32:02.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:32:02.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: T22:32:02.252270+0000 mon.a ( 2021-03-15T22:32:02.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: mon.0) 2026 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.9", "id": [3, 2]}]': finished 2021-03-15T22:32:02.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:32:02.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: 03-15T22:32:02.252320+0000 2021-03-15T22:32:02.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: mon.a (mon.0) 2027 : cluster [DBG] osdmap e772: 8 total, 8 up, 7 in 2021-03-15T22:32:02.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:02 smithi125 conmon[56739]: 2021-03-15T22:32:02.839 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:02.863 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:02.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:02.965 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:02.988 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:02.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:03.089 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:03.111 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:03.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:03.113 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:03.214 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:03.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:03.237 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:03.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:03.258 INFO:tasks.rados.rados.0.smithi076.stdout:2886: done (0 left) 2021-03-15T22:32:03.258 INFO:tasks.rados.rados.0.smithi076.stdout:2887: delete oid 28 current snap is 334 2021-03-15T22:32:03.263 INFO:tasks.rados.rados.0.smithi076.stdout:2887: done (0 left) 2021-03-15T22:32:03.263 INFO:tasks.rados.rados.0.smithi076.stdout:2888: snap_create 2021-03-15T22:32:03.339 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:03.363 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:03.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:03.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-15T22:32:03.465 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:03.509 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:03.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:03.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:03.612 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:03.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:03.635 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:03.636 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:03.737 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:03.759 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:03.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:03.862 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:03.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:03.885 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:03.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:03.936 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:03.958 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 4 1 2021-03-15T22:32:03.986 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:03.987 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:04.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:04.010 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:04.011 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:04.112 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:04.133 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:04.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:04.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:04.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:04.258 INFO:tasks.rados.rados.0.smithi076.stdout:2888: done (0 left) 2021-03-15T22:32:04.258 INFO:tasks.rados.rados.0.smithi076.stdout:2889: rollback oid 15 current snap is 335 2021-03-15T22:32:04.258 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 15 to 323 2021-03-15T22:32:04.259 INFO:tasks.rados.rados.0.smithi076.stdout:2890: write (excl) oid 12 current snap is 335 2021-03-15T22:32:04.259 INFO:tasks.rados.rados.0.smithi076.stdout:2890: seq_num 999 ranges {790193=415994,1959276=404016} 2021-03-15T22:32:04.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: cluster 2021-03-15T22:32:04.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: 2021-03-15T 2021-03-15T22:32:04.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: 22:32:02.933611+0000 osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:04.262 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:32:04.263 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: 22:32:02.934882+0000 osd.4 (osd.4) 2021-03-15T22:32:04.263 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: 164 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:04.263 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:32:04.263 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: -03-15T22:32:03.170167 2021-03-15T22:32:04.264 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[62156]: cluster 2021-03-15T22:32:02.933611+0000 2021-03-15T22:32:04.264 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[62156]: osd.4 (osd.4) 163 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:04.264 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[62156]: cluster 2021-03-15T22:32:02.934882+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:04.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[62156]: cluster 2021-03-15T22:32:03.170167+0000 mgr.y (mgr.14144) 921 : cluster [DBG] pgmap v1303: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.2 MiB/s wr, 14 op/s; 3/1758 objects misplaced (0.171%); 0 B/s, 0 objects/s recovering 2021-03-15T22:32:04.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[62156]: cluster 2021-03-15T22:32:03.254920+0000 mon.a (mon.0) 2028 : cluster [DBG] osdmap e773: 8 total, 8 up, 7 in 2021-03-15T22:32:04.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[62156]: cluster 2021-03-15T22:32:03.414921+0000 mon.a (mon.0) 2029 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:32:04.265 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[62156]: cluster 2021-03-15T22:32:03.502847+0000 osd.0 (osd.0) 126 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:32:04.268 INFO:tasks.rados.rados.0.smithi076.stdout:2890: writing smithi07694878-12 from 790193 to 1206187 tid 1 2021-03-15T22:32:04.269 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:04.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:04.271 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:04.275 INFO:tasks.rados.rados.0.smithi076.stdout:2890: writing smithi07694878-12 from 1959276 to 2363292 tid 2 2021-03-15T22:32:04.276 INFO:tasks.rados.rados.0.smithi076.stdout:2891: write oid 45 current snap is 335 2021-03-15T22:32:04.276 INFO:tasks.rados.rados.0.smithi076.stdout:2889: finishing rollback tid 0 to smithi07694878-15 2021-03-15T22:32:04.276 INFO:tasks.rados.rados.0.smithi076.stdout:2891: seq_num 1000 ranges {407000=491243,1617758=547674,2595228=211772} 2021-03-15T22:32:04.280 INFO:tasks.rados.rados.0.smithi076.stdout:2891: writing smithi07694878-45 from 407000 to 898243 tid 1 2021-03-15T22:32:04.290 INFO:tasks.rados.rados.0.smithi076.stdout:2891: writing smithi07694878-45 from 1617758 to 2165432 tid 2 2021-03-15T22:32:04.294 INFO:tasks.rados.rados.0.smithi076.stdout:2891: writing smithi07694878-45 from 2595228 to 2807000 tid 3 2021-03-15T22:32:04.294 INFO:tasks.rados.rados.0.smithi076.stdout:2892: read oid 8 snap -1 2021-03-15T22:32:04.294 INFO:tasks.rados.rados.0.smithi076.stdout:2892: expect deleted 2021-03-15T22:32:04.295 INFO:tasks.rados.rados.0.smithi076.stdout:2893: copy_from oid 27 from oid 38 current snap is 335 2021-03-15T22:32:04.295 INFO:tasks.rados.rados.0.smithi076.stdout:2889: finishing rollback tid 1 to smithi07694878-15 2021-03-15T22:32:04.295 INFO:tasks.rados.rados.0.smithi076.stdout:2889: finishing rollback tid 2 to smithi07694878-15 2021-03-15T22:32:04.295 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 2299 (ObjNum 971 snap 322 seq_num 971) dirty exists 2021-03-15T22:32:04.296 INFO:tasks.rados.rados.0.smithi076.stdout:2890: finishing write tid 1 to smithi07694878-12 2021-03-15T22:32:04.296 INFO:tasks.rados.rados.0.smithi076.stdout:2890: finishing write tid 2 to smithi07694878-12 2021-03-15T22:32:04.296 INFO:tasks.rados.rados.0.smithi076.stdout:2891: finishing write tid 1 to smithi07694878-45 2021-03-15T22:32:04.296 INFO:tasks.rados.rados.0.smithi076.stdout:2890: finishing write tid 3 to smithi07694878-12 2021-03-15T22:32:04.297 INFO:tasks.rados.rados.0.smithi076.stdout:2890: finishing write tid 4 to smithi07694878-12 2021-03-15T22:32:04.297 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 2302 (ObjNum 999 snap 335 seq_num 999) dirty exists 2021-03-15T22:32:04.297 INFO:tasks.rados.rados.0.smithi076.stdout:2890: left oid 12 (ObjNum 999 snap 335 seq_num 999) 2021-03-15T22:32:04.297 INFO:tasks.rados.rados.0.smithi076.stdout:2891: finishing write tid 2 to smithi07694878-45 2021-03-15T22:32:04.297 INFO:tasks.rados.rados.0.smithi076.stdout:2889: done (4 left) 2021-03-15T22:32:04.298 INFO:tasks.rados.rados.0.smithi076.stdout:2890: done (3 left) 2021-03-15T22:32:04.298 INFO:tasks.rados.rados.0.smithi076.stdout:2894: delete oid 34 current snap is 335 2021-03-15T22:32:04.299 INFO:tasks.rados.rados.0.smithi076.stdout:2892: done (3 left) 2021-03-15T22:32:04.299 INFO:tasks.rados.rados.0.smithi076.stdout:2894: done (2 left) 2021-03-15T22:32:04.299 INFO:tasks.rados.rados.0.smithi076.stdout:2895: write oid 47 current snap is 335 2021-03-15T22:32:04.299 INFO:tasks.rados.rados.0.smithi076.stdout:2895: seq_num 1002 ranges {440614=641741,1734722=551343} 2021-03-15T22:32:04.302 INFO:tasks.rados.rados.0.smithi076.stdout:2895: writing smithi07694878-47 from 440614 to 1082355 tid 1 2021-03-15T22:32:04.309 INFO:tasks.rados.rados.0.smithi076.stdout:2895: writing smithi07694878-47 from 1734722 to 2286065 tid 2 2021-03-15T22:32:04.310 INFO:tasks.rados.rados.0.smithi076.stdout:2896: read oid 17 snap 305 2021-03-15T22:32:04.310 INFO:tasks.rados.rados.0.smithi076.stdout:2896: expect (ObjNum 827 snap 275 seq_num 827) 2021-03-15T22:32:04.310 INFO:tasks.rados.rados.0.smithi076.stdout:2897: read oid 4 snap 297 2021-03-15T22:32:04.310 INFO:tasks.rados.rados.0.smithi076.stdout:2897: expect deleted 2021-03-15T22:32:04.311 INFO:tasks.rados.rados.0.smithi076.stdout:2898: snap_create 2021-03-15T22:32:04.311 INFO:tasks.rados.rados.0.smithi076.stdout:2891: finishing write tid 3 to smithi07694878-45 2021-03-15T22:32:04.311 INFO:tasks.rados.rados.0.smithi076.stdout:2893: finishing copy_from to smithi07694878-27 2021-03-15T22:32:04.311 INFO:tasks.rados.rados.0.smithi076.stdout:2893: got expected ENOENT (src dne) 2021-03-15T22:32:04.311 INFO:tasks.rados.rados.0.smithi076.stdout:2891: finishing write tid 4 to smithi07694878-45 2021-03-15T22:32:04.312 INFO:tasks.rados.rados.0.smithi076.stdout:2891: finishing write tid 5 to smithi07694878-45 2021-03-15T22:32:04.312 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 2084 (ObjNum 1000 snap 335 seq_num 1000) dirty exists 2021-03-15T22:32:04.312 INFO:tasks.rados.rados.0.smithi076.stdout:2891: left oid 45 (ObjNum 1000 snap 335 seq_num 1000) 2021-03-15T22:32:04.312 INFO:tasks.rados.rados.0.smithi076.stdout:2893: finishing copy_from racing read to smithi07694878-27 2021-03-15T22:32:04.313 INFO:tasks.rados.rados.0.smithi076.stdout:2895: finishing write tid 1 to smithi07694878-47 2021-03-15T22:32:04.313 INFO:tasks.rados.rados.0.smithi076.stdout:2895: finishing write tid 2 to smithi07694878-47 2021-03-15T22:32:04.314 INFO:tasks.rados.rados.0.smithi076.stdout:2895: finishing write tid 3 to smithi07694878-47 2021-03-15T22:32:04.314 INFO:tasks.rados.rados.0.smithi076.stdout:2895: finishing write tid 4 to smithi07694878-47 2021-03-15T22:32:04.314 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1771 (ObjNum 1002 snap 335 seq_num 1002) dirty exists 2021-03-15T22:32:04.314 INFO:tasks.rados.rados.0.smithi076.stdout:2895: left oid 47 (ObjNum 1002 snap 335 seq_num 1002) 2021-03-15T22:32:04.320 INFO:tasks.rados.rados.0.smithi076.stdout:2896: expect (ObjNum 827 snap 275 seq_num 827) 2021-03-15T22:32:04.373 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:04.394 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:04.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:04.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:04.496 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 921 : cluster [DBG] 2021-03-15T22:32:04.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: pgmap v1303: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.2 MiB/s wr, 14 op/s; 3/1758 objects misplaced (0.171%); 0 B/s, 0 objects/s recovering 2021-03-15T22:32:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:32:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: 32:03.254920+0000 mon.a (mon.0) 2021-03-15T22:32:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: 2028 : cluster [DBG] osdmap e773: 8 total, 8 up, 7 in 2021-03-15T22:32:04.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:32:04.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: 03-15T22:32:03.414921+0000 2021-03-15T22:32:04.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: mon.a (mon.0) 2029 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:32:04.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: 2021-03-15T22:32:04.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: cluster 2021-03-15T22:32:04.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: 2021-03-15T22:32:03.502847+0000 2021-03-15T22:32:04.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:04 smithi076 conmon[56514]: osd.0 (osd.0) 126 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:32:04.522 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:04.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:04.524 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: cluster 2021-03-15T22:32:02.933611+0000 osd.4 (osd.4) 2021-03-15T22:32:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: 163 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:04.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: cluster 2021-03-15T22:32:02.934882+0000 osd.4 (osd.4) 164 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: cluster 2021-03-15T22:32:03.170167+0000 mgr.y ( 2021-03-15T22:32:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: mgr.14144) 921 : cluster [DBG] pgmap v1303: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 544 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 6.2 MiB/s rd, 2.2 MiB/s wr, 14 op/s; 3/1758 objects misplaced (0.171%); 0 B/s, 0 objects/s recovering 2021-03-15T22:32:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: cluster 2021-03-15T22:32:03.254920+0000 mon.a 2021-03-15T22:32:04.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: (mon.0) 2028 : cluster [DBG] osdmap e773: 8 total, 8 up, 7 in 2021-03-15T22:32:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:32:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: -03-15T22:32:03.414921+0000 mon.a (mon.0) 2029 : cluster [WRN] 2021-03-15T22:32:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 2 pgs backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:32:04.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:32:04.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: 03-15T22:32:03.502847+0000 osd.0 (osd.0) 126 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:32:04.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:04 smithi125 conmon[56739]: 2021-03-15T22:32:04.625 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:04.647 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:04.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:04.648 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:04.750 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:04.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:04.773 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:04.773 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:04.874 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:04.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:04.898 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:04.898 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:05.000 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:05.021 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:05.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:05.023 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:05.124 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:05.146 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:05.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:05.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:05.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:05.256 INFO:tasks.rados.rados.0.smithi076.stdout:2891: done (5 left) 2021-03-15T22:32:05.256 INFO:tasks.rados.rados.0.smithi076.stdout:2893: done (4 left) 2021-03-15T22:32:05.256 INFO:tasks.rados.rados.0.smithi076.stdout:2895: done (3 left) 2021-03-15T22:32:05.256 INFO:tasks.rados.rados.0.smithi076.stdout:2896: done (2 left) 2021-03-15T22:32:05.257 INFO:tasks.rados.rados.0.smithi076.stdout:2897: done (1 left) 2021-03-15T22:32:05.257 INFO:tasks.rados.rados.0.smithi076.stdout:2898: done (0 left) 2021-03-15T22:32:05.257 INFO:tasks.rados.rados.0.smithi076.stdout:2899: read oid 21 snap -1 2021-03-15T22:32:05.257 INFO:tasks.rados.rados.0.smithi076.stdout:2899: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:32:05.258 INFO:tasks.rados.rados.0.smithi076.stdout:2900: snap_create 2021-03-15T22:32:05.262 INFO:tasks.rados.rados.0.smithi076.stdout:2899: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:32:05.274 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:05.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:05.276 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:05.377 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:05.400 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:05.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:05.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:32:05.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:05 smithi076 conmon[56514]: cluster 2021-03-15T22:32:04.254095+0000 2021-03-15T22:32:05.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:05 smithi076 conmon[56514]: mon.a (mon.0) 2030 : cluster [DBG] osdmap e774: 8 total, 8 up, 7 in 2021-03-15T22:32:05.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:05 smithi076 conmon[56514]: cluster 2021-03-15T22:32:05.251804+0000 mon.a (mon.0) 2031 : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2021-03-15T22:32:05.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:05 smithi076 conmon[62156]: cluster 2021-03-15T22:32:04.254095+0000 mon.a ( 2021-03-15T22:32:05.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:05 smithi076 conmon[62156]: mon.0) 2030 : cluster [DBG] osdmap e774: 8 total, 8 up, 7 in 2021-03-15T22:32:05.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:05 smithi076 conmon[62156]: cluster 2021-03-15T22:32:05.251804+0000 mon.a ( 2021-03-15T22:32:05.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:05 smithi076 conmon[62156]: mon.0) 2031 : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2021-03-15T22:32:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:05.527 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:05.527 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:05.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:05 smithi125 conmon[56739]: cluster 2021-03-15T22:32:04.254095+0000 mon.a (mon.0) 2030 : cluster [DBG] 2021-03-15T22:32:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:05 smithi125 conmon[56739]: osdmap e774: 8 total, 8 up, 7 in 2021-03-15T22:32:05.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:05 smithi125 conmon[56739]: cluster 2021-03-15T22:32:05.251804+0000 mon.a (mon.0) 2031 : cluster [DBG] osdmap e775: 8 total, 8 up, 7 in 2021-03-15T22:32:05.629 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:05.653 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:05.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:05.655 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:05.756 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:05.783 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:05.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:05.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:05.795 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:05.830 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:05.885 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:05.911 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:05.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:05.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:06.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:06.038 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:06.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:06.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:06.141 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:06.165 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:06.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:06.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:06.210 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:32:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:32:06.211 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:32:06.211 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:32:06.262 INFO:tasks.rados.rados.0.smithi076.stdout:2899: done (1 left) 2021-03-15T22:32:06.262 INFO:tasks.rados.rados.0.smithi076.stdout:2900: done (0 left) 2021-03-15T22:32:06.262 INFO:tasks.rados.rados.0.smithi076.stdout:2901: delete oid 4 current snap is 337 2021-03-15T22:32:06.267 INFO:tasks.rados.rados.0.smithi076.stdout:2901: done (0 left) 2021-03-15T22:32:06.267 INFO:tasks.rados.rados.0.smithi076.stdout:2902: read oid 45 snap -1 2021-03-15T22:32:06.267 INFO:tasks.rados.rados.0.smithi076.stdout:2902: expect (ObjNum 1000 snap 335 seq_num 1000) 2021-03-15T22:32:06.268 INFO:tasks.rados.rados.0.smithi076.stdout:2903: write oid 50 current snap is 337 2021-03-15T22:32:06.268 INFO:tasks.rados.rados.0.smithi076.stdout:2903: seq_num 1004 ranges {474228=508592,1451686=555012,2704920=169308} 2021-03-15T22:32:06.268 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:06.273 INFO:tasks.rados.rados.0.smithi076.stdout:2903: writing smithi07694878-50 from 474228 to 982820 tid 1 2021-03-15T22:32:06.278 INFO:tasks.rados.rados.0.smithi076.stdout:2903: writing smithi07694878-50 from 1451686 to 2006698 tid 2 2021-03-15T22:32:06.283 INFO:tasks.rados.rados.0.smithi076.stdout:2903: writing smithi07694878-50 from 2704920 to 2874228 tid 3 2021-03-15T22:32:06.283 INFO:tasks.rados.rados.0.smithi076.stdout:2904: read oid 25 snap -1 2021-03-15T22:32:06.284 INFO:tasks.rados.rados.0.smithi076.stdout:2904: expect (ObjNum 888 snap 296 seq_num 888) 2021-03-15T22:32:06.284 INFO:tasks.rados.rados.0.smithi076.stdout:2905: snap_remove snap 336 2021-03-15T22:32:06.295 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:06.297 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:06.398 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:06.420 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:06.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:06.422 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:06.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[56514]: cluster 2021-03-15T22:32:04.969487+0000 osd.2 ( 2021-03-15T22:32:06.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[56514]: osd.2) 21 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:32:06.515 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[56514]: cluster 2021-03-15T22:32:05.111017+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:32:06.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[56514]: cluster 2021-03-15T22:32:05.170909+0000 mgr.y (mgr.14144) 922 : cluster [DBG] pgmap v1306: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 540 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.1 MiB/s wr, 14 op/s; 3/1749 objects misplaced (0.172%); 0 B/s, 0 objects/s recovering 2021-03-15T22:32:06.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[56514]: cluster 2021-03-15T22:32:05.522366+0000 osd.7 (osd.7) 144 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:32:06.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[56514]: cluster 2021-03-15T22:32:05.639051+0000 osd.3 (osd.3) 141 : cluster [DBG] 7.a scrub ok 2021-03-15T22:32:06.516 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[56514]: cluster 2021-03-15T22:32:06.259902+0000 mon.a (mon.0) 2032 : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2021-03-15T22:32:06.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: cluster 2021-03-15T22:32:04.969487+0000 osd.2 2021-03-15T22:32:06.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: (osd.2) 21 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:32:06.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: cluster 2021-03-15T22:32:05.111017+0000 osd.4 (osd.4 2021-03-15T22:32:06.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: ) 165 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:32:06.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: cluster 2021-03-15T22:32:05 2021-03-15T22:32:06.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: .170909+0000 mgr.y (mgr.14144) 922 : 2021-03-15T22:32:06.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: cluster [DBG] pgmap v1306: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 540 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.1 MiB/s wr, 14 op/s; 3/1749 objects misplaced (0.172%); 0 B/s, 0 objects/s recovering 2021-03-15T22:32:06.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: cluster 2021-03-15T22:32: 2021-03-15T22:32:06.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: 05.522366+0000 osd.7 (osd.7) 144 2021-03-15T22:32:06.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: : cluster [DBG] 5.10 scrub ok 2021-03-15T22:32:06.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:32:06.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: :32:05.639051+0000 osd.3 (osd.3 2021-03-15T22:32:06.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: ) 141 : cluster [DBG] 7.a scrub ok 2021-03-15T22:32:06.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:32:06.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: -15T22:32:06.259902+0000 mon.a 2021-03-15T22:32:06.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:06 smithi076 conmon[62156]: (mon.0) 2032 : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2021-03-15T22:32:06.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:06.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: cluster 2021-03-15T22:32:04.969487+0000 osd.2 (osd.2) 21 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:32:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: 2021-03-15T22:32:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: cluster 2021-03-15T22:32:05.111017+0000 osd.4 (osd.4) 165 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:32:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: cluster 2021-03-15T22:32:05. 2021-03-15T22:32:06.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: 170909+0000 mgr.y (mgr.14144) 922 : cluster [DBG] pgmap v1306: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 540 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 2.1 MiB/s wr, 14 op/s; 3/1749 objects misplaced (0.172%); 0 B/s, 0 objects/s recovering 2021-03-15T22:32:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:32:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: 03-15T22:32:05.522366+0000 2021-03-15T22:32:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: osd.7 (osd.7) 144 : 2021-03-15T22:32:06.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: cluster [DBG] 5.10 scrub ok 2021-03-15T22:32:06.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:32:06.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: -03-15T22:32:05.639051 2021-03-15T22:32:06.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: +0000 osd.3 (osd.3) 141 2021-03-15T22:32:06.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: : cluster [DBG] 7.a scrub ok 2021-03-15T22:32:06.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:32:06.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: -03-15T22:32:06. 2021-03-15T22:32:06.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: 259902+0000 mon.a (mon.0 2021-03-15T22:32:06.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: ) 2032 : cluster [DBG] osdmap e776: 8 total, 8 up, 7 in 2021-03-15T22:32:06.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:06 smithi125 conmon[56739]: 2021-03-15T22:32:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:06.550 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:06.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:06.652 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:06.678 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:06.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:06.679 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:06.780 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:06.804 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:06.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:06.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:06.907 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:06.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:06.935 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:06.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:07.038 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:07.060 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:07.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:07.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:07.162 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:07.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:07.185 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:07.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:07.280 INFO:tasks.rados.rados.0.smithi076.stdout:2905: done (3 left) 2021-03-15T22:32:07.281 INFO:tasks.rados.rados.0.smithi076.stdout:2906: snap_create 2021-03-15T22:32:07.281 INFO:tasks.rados.rados.0.smithi076.stdout:2902: expect (ObjNum 1000 snap 335 seq_num 1000) 2021-03-15T22:32:07.283 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:32:07.287 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:07.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:07.313 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:07.313 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:07.381 INFO:tasks.rados.rados.0.smithi076.stdout:2903: finishing write tid 1 to smithi07694878-50 2021-03-15T22:32:07.381 INFO:tasks.rados.rados.0.smithi076.stdout:2903: finishing write tid 2 to smithi07694878-50 2021-03-15T22:32:07.382 INFO:tasks.rados.rados.0.smithi076.stdout:2903: finishing write tid 3 to smithi07694878-50 2021-03-15T22:32:07.382 INFO:tasks.rados.rados.0.smithi076.stdout:2903: finishing write tid 4 to smithi07694878-50 2021-03-15T22:32:07.382 INFO:tasks.rados.rados.0.smithi076.stdout:2903: finishing write tid 5 to smithi07694878-50 2021-03-15T22:32:07.382 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 1769 (ObjNum 1004 snap 337 seq_num 1004) dirty exists 2021-03-15T22:32:07.383 INFO:tasks.rados.rados.0.smithi076.stdout:2903: left oid 50 (ObjNum 1004 snap 337 seq_num 1004) 2021-03-15T22:32:07.383 INFO:tasks.rados.rados.0.smithi076.stdout:2904: expect (ObjNum 888 snap 296 seq_num 888) 2021-03-15T22:32:07.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: cluster 2021-03-15T22:32:07.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021-03-15T22:32 2021-03-15T22:32:07.407 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: :05.880841+0000 osd.1 (osd.1) 146 : cluster [DBG] 5.c scrub ok 2021-03-15T22:32:07.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: cluster 2021-03-15T22:32:07.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021-03-15T22:32:07.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021 2021-03-15T22:32:07.408 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: - 2021-03-15T22:32:07.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 03 2021-03-15T22:32:07.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: - 2021-03-15T22:32:07.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 15 2021-03-15T22:32:07.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: T 2021-03-15T22:32:07.409 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 22 2021-03-15T22:32:07.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: : 2021-03-15T22:32:07.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 32 2021-03-15T22:32:07.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: : 2021-03-15T22:32:07.410 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 06 2021-03-15T22:32:07.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: . 2021-03-15T22:32:07.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 670320 2021-03-15T22:32:07.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: +0000 2021-03-15T22:32:07.411 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021-03-15T22:32:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: osd.3 2021-03-15T22:32:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: ( 2021-03-15T22:32:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: osd 2021-03-15T22:32:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: . 2021-03-15T22:32:07.412 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 3 2021-03-15T22:32:07.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: ) 2021-03-15T22:32:07.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 142 2021-03-15T22:32:07.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: : 2021-03-15T22:32:07.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: cluster 2021-03-15T22:32:07.413 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021-03-15T22:32:07.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: [DBG] 2021-03-15T22:32:07.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021-03-15T22:32:07.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 6.11 scrub ok 2021-03-15T22:32:07.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021-03-15T22:32:07.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: audit 2021-03-15T22:32:07.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021-03-15T22:32:07.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021 2021-03-15T22:32:07.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: - 2021-03-15T22:32:07.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 03 2021-03-15T22:32:07.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: - 2021-03-15T22:32:07.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 15 2021-03-15T22:32:07.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: T 2021-03-15T22:32:07.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 22 2021-03-15T22:32:07.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: : 2021-03-15T22:32:07.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 32 2021-03-15T22:32:07.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: : 2021-03-15T22:32:07.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 06 2021-03-15T22:32:07.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: . 2021-03-15T22:32:07.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 941457 2021-03-15T22:32:07.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: +0000 2021-03-15T22:32:07.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021-03-15T22:32:07.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: mon.a 2021-03-15T22:32:07.418 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: ( 2021-03-15T22:32:07.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: mon 2021-03-15T22:32:07.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: . 2021-03-15T22:32:07.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 0 2021-03-15T22:32:07.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: ) 2021-03-15T22:32:07.419 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2033 2021-03-15T22:32:07.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: : 2021-03-15T22:32:07.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: audit 2021-03-15T22:32:07.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021-03-15T22:32:07.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: [INF] 2021-03-15T22:32:07.420 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021-03-15T22:32:07.421 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/2603181114' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:07.421 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[56514]: 2021-03-15T22:32:07.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: cluster 2021-03-15T22:32:07.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021 2021-03-15T22:32:07.422 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: - 2021-03-15T22:32:07.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 03 2021-03-15T22:32:07.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: - 2021-03-15T22:32:07.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 15 2021-03-15T22:32:07.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: T 2021-03-15T22:32:07.423 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 22 2021-03-15T22:32:07.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: : 2021-03-15T22:32:07.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 32 2021-03-15T22:32:07.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: : 2021-03-15T22:32:07.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 05 2021-03-15T22:32:07.424 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: . 2021-03-15T22:32:07.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 880841 2021-03-15T22:32:07.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: +0000 2021-03-15T22:32:07.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: osd.1 2021-03-15T22:32:07.425 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: ( 2021-03-15T22:32:07.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: osd 2021-03-15T22:32:07.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: . 2021-03-15T22:32:07.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 1 2021-03-15T22:32:07.426 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: ) 2021-03-15T22:32:07.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 146 2021-03-15T22:32:07.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: : 2021-03-15T22:32:07.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: cluster 2021-03-15T22:32:07.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: [DBG] 2021-03-15T22:32:07.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 5.c scrub ok 2021-03-15T22:32:07.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.428 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: cluster 2021-03-15T22:32:07.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021 2021-03-15T22:32:07.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: - 2021-03-15T22:32:07.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 03 2021-03-15T22:32:07.429 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: - 2021-03-15T22:32:07.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 15 2021-03-15T22:32:07.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: T 2021-03-15T22:32:07.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 22 2021-03-15T22:32:07.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: : 2021-03-15T22:32:07.430 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 32 2021-03-15T22:32:07.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: : 2021-03-15T22:32:07.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 06 2021-03-15T22:32:07.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: . 2021-03-15T22:32:07.431 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 670320 2021-03-15T22:32:07.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: +0000 2021-03-15T22:32:07.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: osd.3 2021-03-15T22:32:07.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: ( 2021-03-15T22:32:07.432 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: osd 2021-03-15T22:32:07.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: . 2021-03-15T22:32:07.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 3 2021-03-15T22:32:07.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: ) 2021-03-15T22:32:07.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 142 2021-03-15T22:32:07.433 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: : 2021-03-15T22:32:07.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: cluster 2021-03-15T22:32:07.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: [DBG] 2021-03-15T22:32:07.434 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.435 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 6.11 scrub ok 2021-03-15T22:32:07.435 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: audit 2021-03-15T22:32:07.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021 2021-03-15T22:32:07.436 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: - 2021-03-15T22:32:07.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 03 2021-03-15T22:32:07.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: - 2021-03-15T22:32:07.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 15 2021-03-15T22:32:07.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: T 2021-03-15T22:32:07.437 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 22 2021-03-15T22:32:07.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: : 2021-03-15T22:32:07.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 32 2021-03-15T22:32:07.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: : 2021-03-15T22:32:07.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 06 2021-03-15T22:32:07.438 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: . 2021-03-15T22:32:07.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 941457 2021-03-15T22:32:07.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: +0000 2021-03-15T22:32:07.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.439 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: mon.a 2021-03-15T22:32:07.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: ( 2021-03-15T22:32:07.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: mon 2021-03-15T22:32:07.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: . 2021-03-15T22:32:07.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 0 2021-03-15T22:32:07.440 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: ) 2021-03-15T22:32:07.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2033 2021-03-15T22:32:07.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: : 2021-03-15T22:32:07.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: audit 2021-03-15T22:32:07.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.441 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: [INF] 2021-03-15T22:32:07.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2603181114' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:07.442 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:07 smithi076 conmon[62156]: 2021-03-15T22:32:07.443 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:07.469 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:07.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:07.471 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:07 smithi125 conmon[56739]: cluster 2021-03-15T22:32:05.880841+0000 osd.1 (osd.1) 2021-03-15T22:32:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:07 smithi125 conmon[56739]: 146 : cluster [DBG] 5.c scrub ok 2021-03-15T22:32:07.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:07 smithi125 conmon[56739]: cluster 2021-03-15T22:32:06.670320+0000 osd.3 (osd.3) 142 : cluster [DBG] 6.11 scrub ok 2021-03-15T22:32:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:07 smithi125 conmon[56739]: audit 2021-03-15T22:32 2021-03-15T22:32:07.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:07 smithi125 conmon[56739]: :06.941457+0000 mon.a (mon.0) 2033 : audit [INF] from='client.? 172.21.15.76:0/2603181114' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:07.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-15T22:32:07.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:07.598 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:07.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:07.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:07.701 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:07.727 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:07.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:07.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:07.829 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:07.854 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:07.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:07.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:07.957 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:07.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:07.980 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:07.981 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:08.082 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:08.104 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:08.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:08.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:08.207 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:08.233 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:08.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:08.234 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:08.282 INFO:tasks.rados.rados.0.smithi076.stdout:2902: done (3 left) 2021-03-15T22:32:08.282 INFO:tasks.rados.rados.0.smithi076.stdout:2903: done (2 left) 2021-03-15T22:32:08.282 INFO:tasks.rados.rados.0.smithi076.stdout:2904: done (1 left) 2021-03-15T22:32:08.283 INFO:tasks.rados.rados.0.smithi076.stdout:2906: done (0 left) 2021-03-15T22:32:08.283 INFO:tasks.rados.rados.0.smithi076.stdout:2907: rollback oid 28 current snap is 338 2021-03-15T22:32:08.283 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 329 2021-03-15T22:32:08.283 INFO:tasks.rados.rados.0.smithi076.stdout:2908: snap_create 2021-03-15T22:32:08.288 INFO:tasks.rados.rados.0.smithi076.stdout:2907: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:32:08.288 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2374 (ObjNum 264853792 snap 21896 seq_num 262209296) dirty dne 2021-03-15T22:32:08.336 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:08.360 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:08.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:08.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:08.462 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:08.484 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:08.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[56514]: cluster 2021-03-15T22:32:06.881820+0000 osd.1 (osd.1) 2021-03-15T22:32:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[56514]: 147 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:32:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[56514]: cluster 2021-03-15T22:32:07.171459+0000 mgr.y (mgr.14144) 923 : cluster [DBG] pgmap v1309: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 2.4 MiB/s wr, 21 op/s; 3/1746 objects misplaced (0.172%); 172 B/s, 1 objects/s recovering 2021-03-15T22:32:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[56514]: cluster 2021-03-15T22:32:07.263360+0000 mon.a (mon.0) 2034 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:32:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[56514]: audit 2021-03-15T22:32:07.277239+0000 mon.a (mon.0) 2035 : audit [INF] from='client.? 172.21.15.76:0/2603181114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:08.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[56514]: cluster 2021-03-15T22:32:07.277288+0000 mon.a (mon.0) 2036 : cluster [DBG] osdmap e777: 8 total, 8 up, 7 in 2021-03-15T22:32:08.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[56514]: cluster 2021-03-15T22:32:07.632223+0000 osd.0 (osd.0) 127 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:32:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: cluster 2021-03-15T22:32: 2021-03-15T22:32:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: 06.881820+0000 osd.1 (osd.1) 147 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:32:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:32:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: 32:07.171459+0000 mgr.y (mgr.14144) 923 : cluster [DBG] pgmap v1309: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 2.4 MiB/s wr, 21 op/s; 3/1746 objects misplaced (0.172%); 172 B/s, 1 objects/s recovering 2021-03-15T22:32:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:32:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: 22:32:07.263360+0000 mon.a (mon.0) 2034 : cluster [INF] 2021-03-15T22:32:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:32:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: audit 2021-03-15T22:32:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: 2021-03-15T22:32:07.277239+0000 mon.a (mon.0) 2035 : audit [INF] 2021-03-15T22:32:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2603181114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:32:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: -03-15T22:32:07.277288+0000 mon.a ( 2021-03-15T22:32:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: mon.0) 2036 : cluster [DBG] 2021-03-15T22:32:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: osdmap e777: 8 total, 8 up, 7 in 2021-03-15T22:32:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: 03-15T22:32:07 2021-03-15T22:32:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: .632223+0000 osd.0 (osd.0 2021-03-15T22:32:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:08 smithi076 conmon[62156]: ) 127 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:32:08.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: cluster 2021-03-15T22:32:06.881820+0000 osd.1 (osd.1) 2021-03-15T22:32:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: 147 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:32:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: cluster 2021-03-15T22:32:07.171459+0000 mgr.y (mgr.14144) 923 : cluster [DBG] pgmap v1309: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 2.4 MiB/s wr, 21 op/s; 3/1746 objects misplaced (0.172%); 172 B/s, 1 objects/s recovering 2021-03-15T22:32:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:32:08.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: 32:07.263360+0000 mon.a (mon.0) 2034 : 2021-03-15T22:32:08.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:32:08.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: audit 2021 2021-03-15T22:32:08.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: -03-15T22:32:07.277239+0000 mon.a 2021-03-15T22:32:08.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: (mon.0) 2035 : audit [INF] from='client.? 172.21.15.76:0/2603181114' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:08.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: 2021-03-15T22:32:08.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: cluster 2021-03-15T22:32 2021-03-15T22:32:08.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: :07.277288+0000 mon.a (mon.0) 2036 2021-03-15T22:32:08.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: : cluster [DBG] osdmap e777: 8 total, 8 up, 7 in 2021-03-15T22:32:08.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:32:08.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: -15T22:32:07.632223+0000 osd.0 ( 2021-03-15T22:32:08.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:08 smithi125 conmon[56739]: osd.0) 127 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:32:08.586 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:08.608 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:08.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:08.610 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:08.711 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:08.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:08.734 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:08.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:08.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:08.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:08.859 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:08.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:08.960 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:08.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:08.988 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:08.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:09.091 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:09.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:09.119 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:09.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:09.196 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:09.221 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:09.247 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:09.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:09.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:09.287 INFO:tasks.rados.rados.0.smithi076.stdout:2907: done (1 left) 2021-03-15T22:32:09.287 INFO:tasks.rados.rados.0.smithi076.stdout:2908: done (0 left) 2021-03-15T22:32:09.287 INFO:tasks.rados.rados.0.smithi076.stdout:2909: delete oid 34 current snap is 339 2021-03-15T22:32:09.291 INFO:tasks.rados.rados.0.smithi076.stdout:2909: done (0 left) 2021-03-15T22:32:09.291 INFO:tasks.rados.rados.0.smithi076.stdout:2910: read oid 10 snap 333 2021-03-15T22:32:09.291 INFO:tasks.rados.rados.0.smithi076.stdout:2910: expect deleted 2021-03-15T22:32:09.291 INFO:tasks.rados.rados.0.smithi076.stdout:2911: snap_remove snap 329 2021-03-15T22:32:09.351 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:09.375 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:09.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:09.376 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:09.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[56514]: cluster 2021-03-15T22:32:07 2021-03-15T22:32:09.416 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[56514]: .910422+0000 osd.1 (osd.1) 148 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:32:09.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[56514]: cluster 2021-03-15T22:32:08.071205+0000 osd.4 (osd.4) 166 : cluster [DBG] 5.e scrub ok 2021-03-15T22:32:09.417 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[56514]: cluster 2021-03-15T22:32:08.277017+0000 mon.a (mon.0) 2037 : cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2021-03-15T22:32:09.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[62156]: cluster 2021-03-15T22:32:07.910422+0000 2021-03-15T22:32:09.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[62156]: osd.1 (osd.1) 148 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:32:09.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:32:09.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[62156]: T22:32:08.071205+0000 osd.4 (osd.4) 166 : cluster [DBG] 5.e scrub ok 2021-03-15T22:32:09.418 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[62156]: cluster 2021-03-15T22:32:09.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[62156]: 2021-03-15T22:32:08. 2021-03-15T22:32:09.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[62156]: 277017+0000 mon.a (mon.0) 2037 : 2021-03-15T22:32:09.419 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:09 smithi076 conmon[62156]: cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2021-03-15T22:32:09.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:09.505 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:09.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:09.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:09.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:09 smithi125 conmon[56739]: cluster 2021-03-15T22:32:07.910422+0000 osd.1 (osd. 2021-03-15T22:32:09.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:09 smithi125 conmon[56739]: 1) 148 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:32:09.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:09 smithi125 conmon[56739]: cluster 2021-03-15T22:32:08.071205+0000 osd.4 (osd.4) 166 : cluster [DBG] 5.e scrub ok 2021-03-15T22:32:09.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:09 smithi125 conmon[56739]: cluster 2021-03-15T22:32:08.277017+0000 2021-03-15T22:32:09.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:09 smithi125 conmon[56739]: mon.a (mon.0) 2037 : cluster [DBG] osdmap e778: 8 total, 8 up, 7 in 2021-03-15T22:32:09.608 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:09.634 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:09.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:09.636 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:09.737 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:09.761 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:09.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:09.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:09.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:09.890 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:09.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:09.892 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:09.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:10.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:10.019 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:10.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:10.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:10.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:10.145 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:10.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:10.237 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:32:10.247 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:10.255 INFO:tasks.rados.rados.0.smithi076.stdout:2911: done (1 left) 2021-03-15T22:32:10.255 INFO:tasks.rados.rados.0.smithi076.stdout:2912: write oid 19 current snap is 339 2021-03-15T22:32:10.255 INFO:tasks.rados.rados.0.smithi076.stdout:2912: seq_num 1006 ranges {507842=659090,1568650=558681,2759766=148076} 2021-03-15T22:32:10.263 INFO:tasks.rados.rados.0.smithi076.stdout:2912: writing smithi07694878-19 from 507842 to 1166932 tid 1 2021-03-15T22:32:10.272 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.4 to 1 (10000) 2021-03-15T22:32:10.274 INFO:tasks.rados.rados.0.smithi076.stdout:2912: writing smithi07694878-19 from 1568650 to 2127331 tid 2 2021-03-15T22:32:10.279 INFO:tasks.rados.rados.0.smithi076.stdout:2912: writing smithi07694878-19 from 2759766 to 2907842 tid 3 2021-03-15T22:32:10.279 INFO:tasks.rados.rados.0.smithi076.stdout:2910: done (1 left) 2021-03-15T22:32:10.280 INFO:tasks.rados.rados.0.smithi076.stdout:2913: copy_from oid 1 from oid 25 current snap is 339 2021-03-15T22:32:10.280 INFO:tasks.rados.rados.0.smithi076.stdout:2914: write (excl) oid 31 current snap is 339 2021-03-15T22:32:10.280 INFO:tasks.rados.rados.0.smithi076.stdout:2914: seq_num 1007 ranges {524649=734339,1910779=702339} 2021-03-15T22:32:10.287 INFO:tasks.rados.rados.0.smithi076.stdout:2914: writing smithi07694878-31 from 524649 to 1258988 tid 1 2021-03-15T22:32:10.295 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:10.296 INFO:tasks.rados.rados.0.smithi076.stdout:2914: writing smithi07694878-31 from 1910779 to 2613118 tid 2 2021-03-15T22:32:10.296 INFO:tasks.rados.rados.0.smithi076.stdout:2915: delete oid 27 current snap is 339 2021-03-15T22:32:10.296 INFO:tasks.rados.rados.0.smithi076.stdout:2912: finishing write tid 1 to smithi07694878-19 2021-03-15T22:32:10.297 INFO:tasks.rados.rados.0.smithi076.stdout:2912: finishing write tid 2 to smithi07694878-19 2021-03-15T22:32:10.297 INFO:tasks.rados.rados.0.smithi076.stdout:2912: finishing write tid 3 to smithi07694878-19 2021-03-15T22:32:10.297 INFO:tasks.rados.rados.0.smithi076.stdout:2912: finishing write tid 4 to smithi07694878-19 2021-03-15T22:32:10.298 INFO:tasks.rados.rados.0.smithi076.stdout:2912: finishing write tid 5 to smithi07694878-19 2021-03-15T22:32:10.298 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2100 (ObjNum 1006 snap 339 seq_num 1006) dirty exists 2021-03-15T22:32:10.298 INFO:tasks.rados.rados.0.smithi076.stdout:2912: left oid 19 (ObjNum 1006 snap 339 seq_num 1006) 2021-03-15T22:32:10.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:10.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:10.300 INFO:tasks.rados.rados.0.smithi076.stdout:2912: done (3 left) 2021-03-15T22:32:10.300 INFO:tasks.rados.rados.0.smithi076.stdout:2915: done (2 left) 2021-03-15T22:32:10.300 INFO:tasks.rados.rados.0.smithi076.stdout:2916: snap_create 2021-03-15T22:32:10.303 INFO:tasks.rados.rados.0.smithi076.stdout:2913: finishing copy_from to smithi07694878-1 2021-03-15T22:32:10.303 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 1969 (ObjNum 888 snap 296 seq_num 888) dirty exists 2021-03-15T22:32:10.304 INFO:tasks.rados.rados.0.smithi076.stdout:2913: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:32:10.304 INFO:tasks.rados.rados.0.smithi076.stdout:2914: finishing write tid 1 to smithi07694878-31 2021-03-15T22:32:10.307 INFO:tasks.rados.rados.0.smithi076.stdout:2914: finishing write tid 2 to smithi07694878-31 2021-03-15T22:32:10.307 INFO:tasks.rados.rados.0.smithi076.stdout:2914: finishing write tid 3 to smithi07694878-31 2021-03-15T22:32:10.308 INFO:tasks.rados.rados.0.smithi076.stdout:2914: finishing write tid 4 to smithi07694878-31 2021-03-15T22:32:10.308 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 1972 (ObjNum 1007 snap 339 seq_num 1007) dirty exists 2021-03-15T22:32:10.308 INFO:tasks.rados.rados.0.smithi076.stdout:2914: left oid 31 (ObjNum 1007 snap 339 seq_num 1007) 2021-03-15T22:32:10.385 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:32:10.385 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:32:10.386 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:32:10.386 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:32:10.386 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:32:10.386 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:32:10.387 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:32:10.387 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:32:10.387 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:32:10.387 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:32:10.388 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:32:10.388 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:32:10.388 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:32:10.388 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:32:10.389 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:32:10.389 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:32:10.400 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:10.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:10.424 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:10.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[56514]: cluster 2021-03-15T22:32:08.879991+0000 osd.1 (osd.1) 149 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:32:10.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[56514]: cluster 2021-03-15T22:32:09.171849+0000 mgr.y (mgr.14144) 924 : cluster [DBG] pgmap v1312: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.4 MiB/s wr, 14 op/s; 3/1743 objects misplaced (0.172%); 172 B/s, 1 objects/s recovering 2021-03-15T22:32:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[56514]: cluster 2021-03-15T22:32:09.284137+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e779: 8 total, 8 up, 7 in 2021-03-15T22:32:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[56514]: cluster 2021-03-15T22:32:09.615392+0000 osd.3 (osd.3) 143 : cluster [DBG] 7.c scrub ok 2021-03-15T22:32:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[56514]: audit 2021-03-15T22:32:09.791358+0000 mon.a (mon.0) 2039 : audit [INF] from='client.? 172.21.15.76:0/419389719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:32:10.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[56514]: audit 2021-03-15T22:32:10.059940+0000 mon.a (mon.0) 2040 : audit [DBG] from='client.? 172.21.15.76:0/476705013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:10.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[56514]: audit 2021-03-15T22:32:10.252093+0000 mon.a (mon.0) 2041 : audit 2021-03-15T22:32:10.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[56514]: [INF] from='client.? 172.21.15.76:0/419389719' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-15T22:32:10.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[56514]: cluster 2021-03-15T22:32:10.252211+0000 mon.a (mon.0) 2042 : cluster [DBG] osdmap e780: 8 total, 8 up, 7 in 2021-03-15T22:32:10.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: cluster 2021-03-15T22:32:08.879991+0000 osd.1 (osd.1) 149 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:32:10.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: cluster 2021-03-15T22:32:09.171849+0000 mgr.y (mgr.14144) 924 : cluster [DBG] pgmap v1312: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.4 MiB/s wr, 14 op/s; 3/1743 objects misplaced (0.172%); 172 B/s, 1 objects/s recovering 2021-03-15T22:32:10.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: cluster 2021-03-15T22:32:10.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: 2021-03-15T22:32:09.284137+0000 mon.a (mon.0) 2038 : cluster [DBG] osdmap e779: 8 total, 8 up, 7 in 2021-03-15T22:32:10.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:32:10.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: -15T22:32:09.615392+0000 osd.3 (osd.3) 2021-03-15T22:32:10.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: 143 : cluster [DBG] 7.c scrub ok 2021-03-15T22:32:10.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: audit 2021 2021-03-15T22:32:10.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: -03-15T22:32:09. 2021-03-15T22:32:10.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: 791358+0000 mon.a (mon.0) 2039 : audit 2021-03-15T22:32:10.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/419389719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:32:10.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:32:10.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: 15T22:32:10.059940+0000 2021-03-15T22:32:10.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: mon.a (mon.0) 2040 : audit [DBG] 2021-03-15T22:32:10.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/476705013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:10.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:32:10.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: 22:32:10.252093+0000 mon.a ( 2021-03-15T22:32:10.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: mon.0) 2041 : audit [INF] from='client.? 172.21.15.76:0/419389719' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-15T22:32:10.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:32:10.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: -03-15T22:32:10. 2021-03-15T22:32:10.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: 252211+0000 mon.a (mon.0) 2042 : cluster 2021-03-15T22:32:10.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:10 smithi076 conmon[62156]: [DBG] osdmap e780: 8 total, 8 up, 7 in 2021-03-15T22:32:10.526 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:10.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:32:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: cluster 2021-03-15T22:32:08.879991+0000 osd.1 (osd.1) 149 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:32:10.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 2021-03-15T22:32:10.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: cluster 2021-03-15T22:32:09.171849+0000 mgr.y (mgr.14144) 924 : cluster [DBG] pgmap v1312: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 538 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.8 MiB/s rd, 1.4 MiB/s wr, 14 op/s; 3/1743 objects misplaced (0.172%); 172 B/s, 1 objects/s recovering 2021-03-15T22:32:10.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: cluster 2021-03-15T22:32:09.284137+0000 mon.a (mon.0 2021-03-15T22:32:10.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: ) 2038 : cluster [DBG] osdmap e779: 8 total, 8 up, 7 in 2021-03-15T22:32:10.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: cluster 2021-03-15T22:32:10.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 2021-03-15T 2021-03-15T22:32:10.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 22:32:09.615392+0000 osd.3 (osd. 2021-03-15T22:32:10.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 3) 143 : cluster [DBG] 7.c scrub ok 2021-03-15T22:32:10.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 2021-03-15T22:32:10.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:32:10.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 03-15T22:32: 2021-03-15T22:32:10.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 09.791358+0000 mon.a (mon.0 2021-03-15T22:32:10.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: ) 2039 : audit [INF] from='client.? 172.21.15.76:0/419389719' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2021-03-15T22:32:10.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:32:10.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 03-15 2021-03-15T22:32:10.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: T22:32: 2021-03-15T22:32:10.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 10.059940+0000 mon.a (mon. 2021-03-15T22:32:10.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 0) 2040 : audit [DBG] from='client.? 172.21.15.76:0/476705013' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:10.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 2021-03-15T22:32:10.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:32:10.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 03-15T22:32:10. 2021-03-15T22:32:10.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 252093+0000 mon.a (mon.0) 2021-03-15T22:32:10.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 2041 : audit 2021-03-15T22:32:10.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/419389719' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2021-03-15T22:32:10.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 2021-03-15T22:32:10.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:32:10.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 03-15T22:32:10. 2021-03-15T22:32:10.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 252211+0000 mon.a (mon.0 2021-03-15T22:32:10.544 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: ) 2042 : cluster [DBG] osdmap e780: 8 total, 8 up, 7 in 2021-03-15T22:32:10.545 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:10 smithi125 conmon[56739]: 2021-03-15T22:32:10.545 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:10.548 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:10.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:10.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:10.651 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:10.679 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:10.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:10.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:10.781 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:10.807 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:10.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:10.809 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:10.910 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:10.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:10.933 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:10.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:11.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:11.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:11.058 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:11.059 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:11.161 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:11.187 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:11.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:11.189 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:11.292 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:11.296 INFO:tasks.rados.rados.0.smithi076.stdout:2913: done (2 left) 2021-03-15T22:32:11.296 INFO:tasks.rados.rados.0.smithi076.stdout:2914: done (1 left) 2021-03-15T22:32:11.296 INFO:tasks.rados.rados.0.smithi076.stdout:2916: done (0 left) 2021-03-15T22:32:11.297 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[56514]: cluster 2021-03-15T22:32:10.001230+0000 osd.2 (osd.2) 22 : cluster 2021-03-15T22:32:11.297 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[56514]: [DBG] 4.1c scrub ok 2021-03-15T22:32:11.297 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[56514]: audit 2021-03-15T22:32:10.104706+0000 mgr.y (mgr.14144) 925 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:11.297 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[56514]: cluster 2021-03-15T22:32:10.144128+0000 osd.4 (osd.4) 167 : cluster [DBG] 5.1c scrub ok 2021-03-15T22:32:11.298 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[56514]: cluster 2021-03-15T22:32:10.652890+0000 osd.3 (osd.3) 144 : cluster [DBG] 4.19 deep-scrub ok 2021-03-15T22:32:11.298 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[56514]: cluster 2021-03-15T22:32:10.674802+0000 osd.0 (osd.0) 128 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:32:11.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: cluster 2021-03-15T22:32:11.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 2021-03-15T22:32:11.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 2021 2021-03-15T22:32:11.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: - 2021-03-15T22:32:11.299 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 03 2021-03-15T22:32:11.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: - 2021-03-15T22:32:11.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 15 2021-03-15T22:32:11.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: T 2021-03-15T22:32:11.300 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 22 2021-03-15T22:32:11.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: : 2021-03-15T22:32:11.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 32 2021-03-15T22:32:11.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: : 2021-03-15T22:32:11.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 10 2021-03-15T22:32:11.301 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: . 2021-03-15T22:32:11.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 001230 2021-03-15T22:32:11.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: +0000 2021-03-15T22:32:11.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 2021-03-15T22:32:11.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: osd.2 2021-03-15T22:32:11.302 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: ( 2021-03-15T22:32:11.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: osd 2021-03-15T22:32:11.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: . 2021-03-15T22:32:11.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 2 2021-03-15T22:32:11.303 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: ) 22 : cluster [DBG] 2021-03-15T22:32:11.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 4.1c scrub ok 2021-03-15T22:32:11.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:32:11.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: T22:32:10.104706+0000 mgr.y ( 2021-03-15T22:32:11.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: mgr.14144) 925 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:11.304 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:32:11.305 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: T22:32:10.144128+0000 osd.4 ( 2021-03-15T22:32:11.305 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: osd.4) 167 : cluster [DBG] 5.1c scrub ok 2021-03-15T22:32:11.305 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:32:11.305 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: -03-15T22:32:10.652890+0000 2021-03-15T22:32:11.305 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: osd.3 (osd.3) 144 : cluster [DBG] 2021-03-15T22:32:11.306 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 4.19 deep-scrub ok 2021-03-15T22:32:11.306 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:32:11.306 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: 32:10.674802+0000 osd.0 (osd.0 2021-03-15T22:32:11.306 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:11 smithi076 conmon[62156]: ) 128 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:32:11.307 INFO:tasks.rados.rados.0.smithi076.stdout:2917: delete oid 13 current snap is 340 2021-03-15T22:32:11.307 INFO:tasks.rados.rados.0.smithi076.stdout:2917: done (0 left) 2021-03-15T22:32:11.307 INFO:tasks.rados.rados.0.smithi076.stdout:2918: read oid 48 snap -1 2021-03-15T22:32:11.308 INFO:tasks.rados.rados.0.smithi076.stdout:2918: expect deleted 2021-03-15T22:32:11.308 INFO:tasks.rados.rados.0.smithi076.stdout:2919: snap_remove snap 306 2021-03-15T22:32:11.316 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:11.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:11.317 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:11.419 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:11.443 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:11.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:11.445 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:11.546 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:11.572 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:11.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:11.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:11.674 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:11.700 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:11.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:11 smithi125 conmon[56739]: cluster 2021-03-15T22:32:10.001230+0000 osd.2 (osd.2) 22 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:32:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:11 smithi125 conmon[56739]: 2021-03-15T22:32:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:11 smithi125 conmon[56739]: audit 2021-03-15T22:32:10.104706+0000 mgr.y (mgr.14144) 925 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:11 smithi125 conmon[56739]: cluster 2021-03-15T22:32:10.144128+0000 osd.4 (osd.4) 167 : cluster [DBG] 5.1c scrub ok 2021-03-15T22:32:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:11 smithi125 conmon[56739]: cluster 2021-03-15T22:32:10.652890+0000 osd.3 (osd.3) 144 : cluster [DBG] 4.19 deep-scrub ok 2021-03-15T22:32:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:11 smithi125 conmon[56739]: cluster 2021-03-15T22:32:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:11 smithi125 conmon[56739]: 2021-03-15T22:32:10.674802+0000 osd.0 (osd.0) 128 : cluster [DBG] 6.0 scrub ok 2021-03-15T22:32:11.803 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:11.827 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:11.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:11.928 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:11.953 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:11.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:12.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:12.081 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:12.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:12.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:12.084 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 7 1 2021-03-15T22:32:12.184 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:12.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:12.209 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:12.209 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:12.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:12.314 INFO:tasks.rados.rados.0.smithi076.stdout:2919: done (1 left) 2021-03-15T22:32:12.314 INFO:tasks.rados.rados.0.smithi076.stdout:2920: write (excl) oid 5 current snap is 340 2021-03-15T22:32:12.314 INFO:tasks.rados.rados.0.smithi076.stdout:2920: seq_num 1010 ranges {575070=560086,1686225=566019,2869458=105612} 2021-03-15T22:32:12.315 INFO:tasks.rados.rados.0.smithi076.stdout:2920: writing smithi07694878-5 from 575070 to 1135156 tid 1 2021-03-15T22:32:12.319 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:12 smithi076 conmon[56514]: cluster 2021-03-15T22:32:11.172358+0000 mgr.y (mgr.14144) 926 : cluster [DBG] pgmap v1315: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 538 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 4 op/s; 3/1743 objects misplaced (0.172%) 2021-03-15T22:32:12.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:12 smithi076 conmon[56514]: cluster 2021-03-15T22:32:11.292780+0000 mon.a (mon.0) 2043 : cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2021-03-15T22:32:12.320 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:12 smithi076 conmon[56514]: cluster 2021-03-15T22:32:11.467511+0000 osd.7 (osd.7) 145 : cluster [DBG] 7.5 deep-scrub ok 2021-03-15T22:32:12.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:12 smithi076 conmon[62156]: cluster 2021-03-15T22:32:11.172358+0000 mgr.y (mgr.14144) 926 : cluster [DBG] pgmap v1315: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 538 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 4 op/s; 3/1743 objects misplaced (0.172%) 2021-03-15T22:32:12.320 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:12 smithi076 conmon[62156]: cluster 2021-03-15T22:32:11.292780+0000 mon.a (mon.0) 2043 : cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2021-03-15T22:32:12.321 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:12 smithi076 conmon[62156]: cluster 2021-03-15T22:32:11.467511+0000 osd.7 (osd.7) 145 : cluster [DBG] 7.5 deep-scrub ok 2021-03-15T22:32:12.324 INFO:tasks.rados.rados.0.smithi076.stdout:2920: writing smithi07694878-5 from 1686225 to 2252244 tid 2 2021-03-15T22:32:12.329 INFO:tasks.rados.rados.0.smithi076.stdout:2920: writing smithi07694878-5 from 2869458 to 2975070 tid 3 2021-03-15T22:32:12.329 INFO:tasks.rados.rados.0.smithi076.stdout:2921: write oid 40 current snap is 340 2021-03-15T22:32:12.329 INFO:tasks.rados.rados.0.smithi076.stdout:2921: seq_num 1011 ranges {591877=635335,1628354=709677} 2021-03-15T22:32:12.336 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:12.336 INFO:tasks.rados.rados.0.smithi076.stdout:2921: writing smithi07694878-40 from 591877 to 1227212 tid 1 2021-03-15T22:32:12.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:12.338 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:12.343 INFO:tasks.rados.rados.0.smithi076.stdout:2921: writing smithi07694878-40 from 1628354 to 2338031 tid 2 2021-03-15T22:32:12.344 INFO:tasks.rados.rados.0.smithi076.stdout:2922: read oid 36 snap 338 2021-03-15T22:32:12.344 INFO:tasks.rados.rados.0.smithi076.stdout:2922: expect (ObjNum 969 snap 322 seq_num 969) 2021-03-15T22:32:12.344 INFO:tasks.rados.rados.0.smithi076.stdout:2923: read oid 16 snap -1 2021-03-15T22:32:12.344 INFO:tasks.rados.rados.0.smithi076.stdout:2923: expect (ObjNum 803 snap 268 seq_num 803) 2021-03-15T22:32:12.345 INFO:tasks.rados.rados.0.smithi076.stdout:2924: write (excl) oid 37 current snap is 340 2021-03-15T22:32:12.345 INFO:tasks.rados.rados.0.smithi076.stdout:2920: finishing write tid 1 to smithi07694878-5 2021-03-15T22:32:12.345 INFO:tasks.rados.rados.0.smithi076.stdout:2920: finishing write tid 2 to smithi07694878-5 2021-03-15T22:32:12.345 INFO:tasks.rados.rados.0.smithi076.stdout:2920: finishing write tid 3 to smithi07694878-5 2021-03-15T22:32:12.346 INFO:tasks.rados.rados.0.smithi076.stdout:2920: finishing write tid 4 to smithi07694878-5 2021-03-15T22:32:12.346 INFO:tasks.rados.rados.0.smithi076.stdout:2920: finishing write tid 5 to smithi07694878-5 2021-03-15T22:32:12.346 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1973 (ObjNum 1010 snap 340 seq_num 1010) dirty exists 2021-03-15T22:32:12.346 INFO:tasks.rados.rados.0.smithi076.stdout:2920: left oid 5 (ObjNum 1010 snap 340 seq_num 1010) 2021-03-15T22:32:12.348 INFO:tasks.rados.rados.0.smithi076.stdout:2924: seq_num 1012 ranges {608684=426937,1803189=569688,2924304=84380} 2021-03-15T22:32:12.350 INFO:tasks.rados.rados.0.smithi076.stdout:2924: writing smithi07694878-37 from 608684 to 1035621 tid 1 2021-03-15T22:32:12.358 INFO:tasks.rados.rados.0.smithi076.stdout:2924: writing smithi07694878-37 from 1803189 to 2372877 tid 2 2021-03-15T22:32:12.363 INFO:tasks.rados.rados.0.smithi076.stdout:2924: writing smithi07694878-37 from 2924304 to 3008684 tid 3 2021-03-15T22:32:12.363 INFO:tasks.rados.rados.0.smithi076.stdout:2918: done (5 left) 2021-03-15T22:32:12.363 INFO:tasks.rados.rados.0.smithi076.stdout:2920: done (4 left) 2021-03-15T22:32:12.363 INFO:tasks.rados.rados.0.smithi076.stdout:2925: read oid 31 snap -1 2021-03-15T22:32:12.364 INFO:tasks.rados.rados.0.smithi076.stdout:2925: expect (ObjNum 1007 snap 339 seq_num 1007) 2021-03-15T22:32:12.364 INFO:tasks.rados.rados.0.smithi076.stdout:2926: read oid 11 snap -1 2021-03-15T22:32:12.364 INFO:tasks.rados.rados.0.smithi076.stdout:2926: expect (ObjNum 989 snap 332 seq_num 989) 2021-03-15T22:32:12.364 INFO:tasks.rados.rados.0.smithi076.stdout:2927: snap_remove snap 332 2021-03-15T22:32:12.439 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:12.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:12.464 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:12.464 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:12.537 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:32:13.124 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:32:13.125 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:13.126 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:12 smithi125 conmon[56739]: cluster 2021-03-15T22:32:11.172358+0000 mgr.y (mgr.14144) 926 : cluster [DBG] pgmap v1315: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 538 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 4 op/s; 3/1743 objects misplaced (0.172%) 2021-03-15T22:32:13.126 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:12 smithi125 conmon[56739]: cluster 2021-03-15T22:32:11.292780+0000 mon.a (mon.0) 2043 : cluster [DBG] osdmap e781: 8 total, 8 up, 7 in 2021-03-15T22:32:13.126 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:12 smithi125 conmon[56739]: cluster 2021-03-15T22:32:11.467511+0000 osd.7 (osd 2021-03-15T22:32:13.127 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:12 smithi125 conmon[56739]: .7) 145 : cluster [DBG] 7.5 deep-scrub ok 2021-03-15T22:32:13.127 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:13.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:13.154 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:13.155 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:13.256 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:13.282 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:13.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:13.284 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:13.321 INFO:tasks.rados.rados.0.smithi076.stdout:2927: done (6 left) 2021-03-15T22:32:13.321 INFO:tasks.rados.rados.0.smithi076.stdout:2928: rollback oid 38 current snap is 340 2021-03-15T22:32:13.321 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 38 to 331 2021-03-15T22:32:13.322 INFO:tasks.rados.rados.0.smithi076.stdout:2929: snap_remove snap 242 2021-03-15T22:32:13.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:13.412 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:13.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:13.414 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:13.515 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:13.541 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:13.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:13.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:13.631 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[56514]: cluster 2021-03-15T22:32:11.834098+0000 osd.5 (osd 2021-03-15T22:32:13.631 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[56514]: .5) 27 : cluster [DBG] 5.16 scrub ok 2021-03-15T22:32:13.631 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[56514]: cluster 2021-03-15T22:32:11.867806+0000 osd.1 (osd.1) 150 : cluster [DBG] 7.d scrub ok 2021-03-15T22:32:13.631 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[56514]: cluster 2021-03-15T22:32:12.036516+0000 osd.2 (osd.2) 23 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:32:13.632 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[56514]: cluster 2021-03-15T22:32:12.177015+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.18 deep-scrub ok 2021-03-15T22:32:13.632 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[56514]: cluster 2021-03-15T22:32:12.302945+0000 mon.a (mon.0) 2044 : cluster [DBG] osdmap e782: 8 total, 8 up, 7 in 2021-03-15T22:32:13.632 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[56514]: cluster 2021-03-15T22:32:12.653804+0000 osd.3 (osd.3) 145 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:32:13.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[62156]: cluster 2021-03-15T22:32:11.834098+0000 osd.5 (osd.5) 27 : cluster [DBG] 5.16 scrub ok 2021-03-15T22:32:13.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[62156]: cluster 2021-03-15T22:32:11.867806+0000 osd.1 (osd.1) 150 : cluster [DBG] 7.d scrub ok 2021-03-15T22:32:13.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[62156]: cluster 2021-03-15T22:32:12.036516+0000 osd.2 (osd.2) 23 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:32:13.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[62156]: cluster 2021-03-15T22:32:12.177015+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.18 deep-scrub ok 2021-03-15T22:32:13.634 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[62156]: cluster 2021-03-15T22:32:12.302945+0000 mon.a (mon.0) 2044 : cluster [DBG] osdmap e782: 8 total, 8 up, 7 in 2021-03-15T22:32:13.634 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:13 smithi076 conmon[62156]: cluster 2021-03-15T22:32:12.653804+0000 osd.3 (osd.3) 145 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:32:13.644 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:13.666 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:13.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:13.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:13.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-15T22:32:13.774 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:13.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: cluster 2021-03-15T22:32:11.834098+0000 osd.5 (osd.5) 27 2021-03-15T22:32:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: : cluster [DBG] 5.16 scrub ok 2021-03-15T22:32:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: cluster 2021-03-15T22:32:11.867806+0000 osd.1 (osd.1) 150 : cluster [DBG] 7.d scrub ok 2021-03-15T22:32:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: cluster 2021-03-15T22:32:12.036516+0000 2021-03-15T22:32:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: osd.2 (osd.2) 23 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:32:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:32:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: T22:32:12.177015+0000 osd.4 (osd.4) 168 : cluster [DBG] 4.18 deep-scrub ok 2021-03-15T22:32:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: 2021-03-15T22:32:13.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: cluster 2021-03-15T22:32:13.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: 2021-03-15T22:32:12.302945+0000 mon.a (mon. 2021-03-15T22:32:13.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: 0) 2044 : cluster [DBG] osdmap e782: 8 total, 8 up, 7 in 2021-03-15T22:32:13.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:32:13.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: 15T22:32:12.653804+0000 osd.3 (osd. 2021-03-15T22:32:13.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:13 smithi125 conmon[56739]: 3) 145 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:32:13.800 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:13.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:13.801 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:13.902 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:13.929 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:13.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:13.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:14.031 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:14.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:32:14.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:14.056 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:14.056 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:14.157 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:14.182 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:14.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:14.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:14.285 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:14.314 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:14.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:14.324 INFO:tasks.rados.rados.0.smithi076.stdout:2929: done (7 left) 2021-03-15T22:32:14.324 INFO:tasks.rados.rados.0.smithi076.stdout:2930: copy_from oid 34 from oid 18 current snap is 340 2021-03-15T22:32:14.324 INFO:tasks.rados.rados.0.smithi076.stdout:2931: rollback oid 8 current snap is 340 2021-03-15T22:32:14.325 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 333 2021-03-15T22:32:14.325 INFO:tasks.rados.rados.0.smithi076.stdout:2932: snap_remove snap 315 2021-03-15T22:32:14.364 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:14.416 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:14.441 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:14.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:14.443 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:14.544 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:14.570 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:14.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:14.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:14.673 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:14.698 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:14.700 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:14.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:14 smithi076 conmon[56514]: cluster 2021-03-15T22:32:12.794109+0000 osd.5 (osd.5) 28 : cluster [DBG] 5.1b scrub ok 2021-03-15T22:32:14.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:14 smithi076 conmon[56514]: cluster 2021-03-15T22:32:12.830481+0000 osd.1 (osd.1) 151 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:32:14.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:14 smithi076 conmon[56514]: cluster 2021-03-15T22:32:13.172810+0000 mgr.y (mgr.14144) 927 : cluster [DBG] pgmap v1318: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 539 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 334 KiB/s wr, 6 op/s; 3/1749 objects misplaced (0.172%) 2021-03-15T22:32:14.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:14 smithi076 conmon[56514]: cluster 2021-03-15T22:32:13.312458+0000 mon.a (mon.0) 2045 : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2021-03-15T22:32:14.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:14 smithi076 conmon[62156]: cluster 2021-03-15T22:32:12.794109+0000 osd.5 (osd.5) 28 : cluster [DBG] 5.1b scrub ok 2021-03-15T22:32:14.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:14 smithi076 conmon[62156]: cluster 2021-03-15T22:32:12.830481+0000 osd.1 (osd.1) 151 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:32:14.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:14 smithi076 conmon[62156]: cluster 2021-03-15T22:32:13.172810+0000 mgr.y (mgr.14144) 927 : cluster [DBG] pgmap v1318: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 539 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 334 KiB/s wr, 6 op/s; 3/1749 objects misplaced (0.172%) 2021-03-15T22:32:14.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:14 smithi076 conmon[62156]: cluster 2021-03-15T22:32:13.312458+0000 mon.a (mon.0) 2045 2021-03-15T22:32:14.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:14 smithi076 conmon[62156]: : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2021-03-15T22:32:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:14 smithi125 conmon[56739]: cluster 2021-03-15T22:32:12.794109+0000 osd.5 (osd 2021-03-15T22:32:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:14 smithi125 conmon[56739]: .5) 28 : cluster [DBG] 5.1b scrub ok 2021-03-15T22:32:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:14 smithi125 conmon[56739]: cluster 2021-03-15T22:32:12.830481+0000 osd.1 (osd.1) 151 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:32:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:14 smithi125 conmon[56739]: 2021-03-15T22:32:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:14 smithi125 conmon[56739]: cluster 2021-03-15T22:32:13 2021-03-15T22:32:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:14 smithi125 conmon[56739]: .172810+0000 mgr.y (mgr.14144) 927 : cluster [DBG] pgmap v1318: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 539 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 334 KiB/s wr, 6 op/s; 3/1749 objects misplaced (0.172%) 2021-03-15T22:32:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:14 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:32:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:14 smithi125 conmon[56739]: -03-15T22:32:13.312458+0000 mon.a (mon.0) 2021-03-15T22:32:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:14 smithi125 conmon[56739]: 2045 : cluster [DBG] osdmap e783: 8 total, 8 up, 7 in 2021-03-15T22:32:14.801 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:14.827 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:14.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:14.829 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:14.930 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:14.952 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:14.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:14.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:15.055 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:15.077 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:15.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:15.079 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:15.180 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:15.206 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:15.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:15.207 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:15.260 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:15.262 INFO:tasks.rados.rados.0.smithi076.stdout:2932: done (9 left) 2021-03-15T22:32:15.263 INFO:tasks.rados.rados.0.smithi076.stdout:2933: snap_remove snap 317 2021-03-15T22:32:15.288 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:32:15.308 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:15.349 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:15.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:15.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:15.452 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:15.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:15.478 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:15.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:15.580 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:15.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:15.606 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:15.607 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:15.617 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[56514]: cluster 2021-03-15T22:32:14.320334+0000 mon.a (mon.0) 2046 : cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2021-03-15T22:32:15.618 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[56514]: audit 2021-03-15T22:32:14.516968+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? 172.21.15.76:0/189415546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:32:15.618 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[56514]: cluster 2021-03-15T22:32:14.594713+0000 osd.3 (osd.3) 146 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:32:15.618 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[56514]: cluster 2021-03-15T22:32:15.249296+0000 mon.a (mon.0) 2048 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:15.618 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[56514]: audit 2021-03-15T22:32:15.258220+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? 172.21.15.76:0/189415546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:32:15.619 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[56514]: cluster 2021-03-15T22:32:15.258274+0000 mon.a (mon.0) 2050 : cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2021-03-15T22:32:15.619 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[62156]: cluster 2021-03-15T22:32:14.320334+0000 mon.a ( 2021-03-15T22:32:15.619 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[62156]: mon.0) 2046 : cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2021-03-15T22:32:15.619 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[62156]: audit 2021-03-15T22:32:14.516968+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? 172.21.15.76:0/189415546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:32:15.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[62156]: cluster 2021-03-15T22:32:14.594713+0000 osd.3 (osd.3) 146 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:32:15.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[62156]: cluster 2021-03-15T22:32:15.249296+0000 mon.a (mon.0) 2048 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:15.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[62156]: audit 2021-03-15T22:32:15.258220+0000 mon.a (mon.0) 2049 : audit [INF] from='client.? 172.21.15.76:0/189415546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:32:15.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:15 smithi076 conmon[62156]: cluster 2021-03-15T22:32:15.258274+0000 mon.a (mon.0) 2050 : cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2021-03-15T22:32:15.708 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:15.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:15.732 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:15.732 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:15.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: cluster 2021-03-15T22:32:14.320334+0000 mon.a (mon.0) 2046 2021-03-15T22:32:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: : cluster [DBG] osdmap e784: 8 total, 8 up, 7 in 2021-03-15T22:32:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: audit 2021-03-15T22:32:14.516968+0000 mon.a (mon.0) 2047 : audit [INF] from='client.? 172.21.15.76:0/189415546' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:32:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:32:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: 03-15T22:32:14.594713+0000 osd.3 (osd.3) 146 : cluster [DBG] 6.1f scrub ok 2021-03-15T22:32:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:32:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: -03-15T22:32:15.249296 2021-03-15T22:32:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2048 : cluster [WRN] 2021-03-15T22:32:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: audit 2021-03-15T22:32 2021-03-15T22:32:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: :15.258220+0000 mon.a (mon.0) 2021-03-15T22:32:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: 2049 : audit [INF] from='client.? 172.21.15.76:0/189415546' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:32:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: cluster 2021-03-15T22:32:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:32:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: 32:15.258274+0000 mon.a (mon.0) 2021-03-15T22:32:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:15 smithi125 conmon[56739]: 2050 : cluster [DBG] osdmap e785: 8 total, 8 up, 7 in 2021-03-15T22:32:15.834 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:15.858 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:15.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:15.961 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:15.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:15.985 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:15.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:16.087 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:16.111 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:16.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:16.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:16.214 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:16.238 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:16.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:16.240 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:16.274 INFO:tasks.rados.rados.0.smithi076.stdout:2933: done (9 left) 2021-03-15T22:32:16.274 INFO:tasks.rados.rados.0.smithi076.stdout:2934: rollback oid 26 current snap is 340 2021-03-15T22:32:16.274 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 264 2021-03-15T22:32:16.274 INFO:tasks.rados.rados.0.smithi076.stdout:2935: snap_remove snap 314 2021-03-15T22:32:16.341 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:16.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:16.365 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:16.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:16.467 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:16.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:16.491 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:16.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:16.593 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:16.621 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:16.621 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:16.720 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[56514]: cluster 2021-03-15T22:32:14.756454 2021-03-15T22:32:16.720 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[56514]: +0000 osd.5 (osd.5) 29 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:32:16.720 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[56514]: cluster 2021-03-15T22:32:14.793894+0000 osd.1 (osd.1) 152 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:32:16.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[56514]: cluster 2021-03-15T22:32:15.147629+0000 osd.1 (osd.1) 153 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:16.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[56514]: cluster 2021-03-15T22:32:15.149295+0000 osd.1 (osd.1) 154 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:16.721 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[56514]: cluster 2021-03-15T22:32:15.173165+0000 mgr.y (mgr.14144) 928 : cluster [DBG] pgmap v1321: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 546 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 9 op/s; 3/1755 objects misplaced (0.171%) 2021-03-15T22:32:16.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[56514]: cluster 2021-03-15T22:32:16.271093+0000 mon.a (mon.0) 2051 : cluster [DBG] osdmap e786: 8 total, 8 up, 7 in 2021-03-15T22:32:16.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[56514]: audit 2021-03-15 2021-03-15T22:32:16.722 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[56514]: T22:32:16.378570+0000 mon.a (mon.0) 2052 : audit [INF] from='client.? 172.21.15.76:0/1130592568' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:32:16.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: cluster 2021-03-15T22:32:16.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: 2021-03-15T22:32:14. 2021-03-15T22:32:16.723 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: 756454+0000 osd.5 (osd.5) 29 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:32:16.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: cluster 2021-03-15T22:32:16.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: 2021-03-15T22:32:14.793894+0000 osd.1 2021-03-15T22:32:16.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: (osd.1) 152 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:32:16.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:16.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: 03-15T22:32:15.147629+0000 osd.1 (osd.1 2021-03-15T22:32:16.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: ) 153 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:16.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: cluster 2021-03-15T22:32 2021-03-15T22:32:16.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: :15.149295+0000 osd.1 (osd.1) 154 : cluster 2021-03-15T22:32:16.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: [DBG] purged_snaps scrub ok 2021-03-15T22:32:16.726 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:32:16.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: :32:15.173165+0000 mgr.y (mgr.14144) 928 2021-03-15T22:32:16.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: : cluster [DBG] pgmap v1321: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 546 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 9 op/s; 3/1755 objects misplaced (0.171%) 2021-03-15T22:32:16.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:16.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: 03-15T22:32:16.271093+0000 2021-03-15T22:32:16.727 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: mon.a (mon.0) 2051 : cluster 2021-03-15T22:32:16.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: [DBG] osdmap e786: 8 total, 8 up, 7 in 2021-03-15T22:32:16.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: audit 2021-03-15T22:32:16.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: 2021-03-15T22:32: 2021-03-15T22:32:16.728 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: 16.378570+0000 mon.a (mon.0 2021-03-15T22:32:16.729 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:16 smithi076 conmon[62156]: ) 2052 : audit [INF] from='client.? 172.21.15.76:0/1130592568' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:32:16.729 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:16.753 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:16.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: cluster 2021-03-15T22:32:14.756454+0000 osd.5 (osd.5) 29 : 2021-03-15T22:32:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: cluster [DBG] 3.1c scrub ok 2021-03-15T22:32:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: cluster 2021-03-15T22:32:14.793894+0000 osd.1 (osd.1) 152 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:32:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: cluster 2021-03-15T22:32:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: 2021-03-15T22:32:15.147629+0000 osd.1 (osd.1) 153 : cluster [DBG] 2021-03-15T22:32:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: purged_snaps scrub starts 2021-03-15T22:32:16.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: cluster 2021-03-15T22:32 2021-03-15T22:32:16.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: :15.149295+0000 osd.1 (osd.1) 2021-03-15T22:32:16.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: 154 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:16.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:32:16.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: -03-15T22:32:15.173165+0000 2021-03-15T22:32:16.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: mgr.y (mgr.14144) 928 : cluster [DBG] 2021-03-15T22:32:16.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: pgmap v1321: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 546 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.9 MiB/s wr, 9 op/s; 3/1755 objects misplaced (0.171%) 2021-03-15T22:32:16.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: cluster 2021-03-15T22:32 2021-03-15T22:32:16.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: :16.271093+0000 mon.a (mon.0) 2021-03-15T22:32:16.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: 2051 : cluster [DBG] osdmap e786: 8 total, 8 up, 7 in 2021-03-15T22:32:16.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:32:16.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: -15T22:32:16.378570+0000 2021-03-15T22:32:16.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: mon.a (mon.0) 2052 : audit [INF] 2021-03-15T22:32:16.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:16 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1130592568' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2021-03-15T22:32:16.856 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:16.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:16.880 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:16.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:16.981 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:17.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:17.006 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:17.006 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:17.108 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:17.134 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:17.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:17.136 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:17.237 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:17.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:17.261 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:17.261 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:17.280 INFO:tasks.rados.rados.0.smithi076.stdout:2935: done (10 left) 2021-03-15T22:32:17.280 INFO:tasks.rados.rados.0.smithi076.stdout:2936: rollback oid 17 current snap is 340 2021-03-15T22:32:17.281 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 17 to 321 2021-03-15T22:32:17.281 INFO:tasks.rados.rados.0.smithi076.stdout:2923: expect (ObjNum 803 snap 268 seq_num 803) 2021-03-15T22:32:17.297 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.7 to 1 (10000) 2021-03-15T22:32:17.362 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:17.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:17.385 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:17.386 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:17.487 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:17.513 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:17.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:17.515 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:17.518 INFO:tasks.rados.rados.0.smithi076.stdout:2921: finishing write tid 1 to smithi07694878-40 2021-03-15T22:32:17.519 INFO:tasks.rados.rados.0.smithi076.stdout:2921: finishing write tid 2 to smithi07694878-40 2021-03-15T22:32:17.520 INFO:tasks.rados.rados.0.smithi076.stdout:2921: finishing write tid 3 to smithi07694878-40 2021-03-15T22:32:17.520 INFO:tasks.rados.rados.0.smithi076.stdout:2923: done (10 left) 2021-03-15T22:32:17.520 INFO:tasks.rados.rados.0.smithi076.stdout:2937: snap_remove snap 337 2021-03-15T22:32:17.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:17.640 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:17.640 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:17.742 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:17.764 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:17.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:17.766 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:17.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:17.893 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:17.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:17.895 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:17.996 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:18.017 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:18.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:18.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:18.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:18.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:18.147 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:18.148 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:18.249 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:18.272 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:18.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:18.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:18.274 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:18 smithi076 conmon[56514]: cluster 2021-03-15T22:32:17.173655+0000 mgr.y (mgr.14144) 929 : cluster [DBG] pgmap v1324: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 3/1692 objects misplaced (0.177%) 2021-03-15T22:32:18.275 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:18 smithi076 conmon[56514]: audit 2021-03-15T22:32:17.271043+0000 mon.a (mon.0) 2053 : audit [INF] from='client.? 172.21.15.76:0/1130592568' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:32:18.276 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:18 smithi076 conmon[56514]: cluster 2021-03-15T22:32:17.271100+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2021-03-15T22:32:18.375 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:18.401 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:18.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:18.403 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:18.451 INFO:tasks.rados.rados.0.smithi076.stdout:2937: done (10 left) 2021-03-15T22:32:18.451 INFO:tasks.rados.rados.0.smithi076.stdout:2938: rollback oid 24 current snap is 340 2021-03-15T22:32:18.451 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 331 2021-03-15T22:32:18.451 INFO:tasks.rados.rados.0.smithi076.stdout:2939: snap_create 2021-03-15T22:32:18.452 INFO:tasks.rados.rados.0.smithi076.stdout:2921: finishing write tid 4 to smithi07694878-40 2021-03-15T22:32:18.452 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 2499 (ObjNum 1011 snap 340 seq_num 1011) dirty exists 2021-03-15T22:32:18.452 INFO:tasks.rados.rados.0.smithi076.stdout:2921: left oid 40 (ObjNum 1011 snap 340 seq_num 1011) 2021-03-15T22:32:18.452 INFO:tasks.rados.rados.0.smithi076.stdout:2922: expect (ObjNum 969 snap 322 seq_num 969) 2021-03-15T22:32:18.478 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 0 1 2021-03-15T22:32:18.503 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:18.526 INFO:tasks.rados.rados.0.smithi076.stdout:2924: finishing write tid 1 to smithi07694878-37 2021-03-15T22:32:18.527 INFO:tasks.rados.rados.0.smithi076.stdout:2924: finishing write tid 2 to smithi07694878-37 2021-03-15T22:32:18.527 INFO:tasks.rados.rados.0.smithi076.stdout:2924: finishing write tid 3 to smithi07694878-37 2021-03-15T22:32:18.527 INFO:tasks.rados.rados.0.smithi076.stdout:2924: finishing write tid 4 to smithi07694878-37 2021-03-15T22:32:18.527 INFO:tasks.rados.rados.0.smithi076.stdout:2924: finishing write tid 5 to smithi07694878-37 2021-03-15T22:32:18.528 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 2504 (ObjNum 1012 snap 340 seq_num 1012) dirty exists 2021-03-15T22:32:18.528 INFO:tasks.rados.rados.0.smithi076.stdout:2924: left oid 37 (ObjNum 1012 snap 340 seq_num 1012) 2021-03-15T22:32:18.528 INFO:tasks.rados.rados.0.smithi076.stdout:2926: expect (ObjNum 989 snap 332 seq_num 989) 2021-03-15T22:32:18.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:18 smithi076 conmon[62156]: cluster 2021-03-15T22:32:17.173655+0000 mgr.y ( 2021-03-15T22:32:18.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:18 smithi076 conmon[62156]: mgr.14144) 929 : cluster [DBG] pgmap v1324: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 3/1692 objects misplaced (0.177%) 2021-03-15T22:32:18.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:18 smithi076 conmon[62156]: audit 2021-03-15T22:32:17.271043+0000 mon.a ( 2021-03-15T22:32:18.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:18 smithi076 conmon[62156]: mon.0) 2053 : audit [INF] from='client.? 172.21.15.76:0/1130592568' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:32:18.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:18 smithi076 conmon[62156]: cluster 2021-03-15T22:32 2021-03-15T22:32:18.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:18 smithi076 conmon[62156]: :17.271100+0000 mon.a (mon.0) 2054 : 2021-03-15T22:32:18.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:18 smithi076 conmon[62156]: cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2021-03-15T22:32:18.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:18 smithi125 conmon[56739]: cluster 2021-03-15T22:32:17.173655+0000 mgr.y (mgr.14144) 929 : cluster [DBG] pgmap v1324: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 508 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 13 MiB/s rd, 2.4 MiB/s wr, 18 op/s; 3/1692 objects misplaced (0.177%) 2021-03-15T22:32:18.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:18 smithi125 conmon[56739]: 2021-03-15T22:32:18.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:18 smithi125 conmon[56739]: audit 2021-03-15T22:32:17.271043+0000 mon.a (mon.0) 2053 : audit [INF] from='client.? 172.21.15.76:0/1130592568' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2021-03-15T22:32:18.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:18 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:32:18.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:18 smithi125 conmon[56739]: -03-15T22:32:17.271100+0000 mon.a (mon.0) 2054 : cluster [DBG] osdmap e787: 8 total, 8 up, 7 in 2021-03-15T22:32:18.547 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:18.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:18.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:18.614 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:32:18.615 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:32:18.650 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:18.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:18.675 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:18.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:18.676 INFO:tasks.rados.rados.0.smithi076.stdout:2925: expect (ObjNum 1007 snap 339 seq_num 1007) 2021-03-15T22:32:18.777 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:18.804 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:18.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:18.806 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:18.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:32:18.876 INFO:tasks.rados.rados.0.smithi076.stdout:2928: finishing rollback tid 1 to smithi07694878-38 2021-03-15T22:32:18.877 INFO:tasks.rados.rados.0.smithi076.stdout:2928: finishing rollback tid 2 to smithi07694878-38 2021-03-15T22:32:18.877 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2102 (ObjNum 955 snap 320 seq_num 955) dirty exists 2021-03-15T22:32:18.877 INFO:tasks.rados.rados.0.smithi076.stdout:2931: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:32:18.878 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1211 (ObjNum 265116640 snap 21896 seq_num 0) dirty dne 2021-03-15T22:32:18.878 INFO:tasks.rados.rados.0.smithi076.stdout:2930: finishing copy_from to smithi07694878-34 2021-03-15T22:32:18.879 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 1963 (ObjNum 957 snap 321 seq_num 957) dirty exists 2021-03-15T22:32:18.879 INFO:tasks.rados.rados.0.smithi076.stdout:2930: finishing copy_from racing read to smithi07694878-34 2021-03-15T22:32:18.880 INFO:tasks.rados.rados.0.smithi076.stdout:2934: finishing rollback tid 0 to smithi07694878-26 2021-03-15T22:32:18.880 INFO:tasks.rados.rados.0.smithi076.stdout:2934: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:32:18.881 INFO:tasks.rados.rados.0.smithi076.stdout:2934: finishing rollback tid 2 to smithi07694878-26 2021-03-15T22:32:18.881 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 1772 (ObjNum 755 snap 248 seq_num 755) dirty exists 2021-03-15T22:32:18.882 INFO:tasks.rados.rados.0.smithi076.stdout:2936: finishing rollback tid 0 to smithi07694878-17 2021-03-15T22:32:18.882 INFO:tasks.rados.rados.0.smithi076.stdout:2936: finishing rollback tid 1 to smithi07694878-17 2021-03-15T22:32:18.882 INFO:tasks.rados.rados.0.smithi076.stdout:2936: finishing rollback tid 2 to smithi07694878-17 2021-03-15T22:32:18.883 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 862 (ObjNum 960 snap 321 seq_num 960) dirty exists 2021-03-15T22:32:18.883 INFO:tasks.rados.rados.0.smithi076.stdout:2938: finishing rollback tid 0 to smithi07694878-24 2021-03-15T22:32:18.884 INFO:tasks.rados.rados.0.smithi076.stdout:2938: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:32:18.884 INFO:tasks.rados.rados.0.smithi076.stdout:2938: finishing rollback tid 2 to smithi07694878-24 2021-03-15T22:32:18.884 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1730 (ObjNum 864 snap 285 seq_num 864) dirty exists 2021-03-15T22:32:18.907 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:18.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:18.933 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:18.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:19.008 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:32:19.009 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:19.026 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:32:19.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:19.044 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:32:19.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:19.060 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:19.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:19.062 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:32:19.083 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:32:19.109 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:32:19.130 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:32:19.149 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:32:19.150 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:32:19.151 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:32:19.151 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:32:19.151 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:32:19.151 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:32:19.152 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:32:19.152 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:32:19.152 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:32:19.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:19.189 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:19.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:19.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:19.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:19.319 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:19.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:19.320 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:19.421 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:19.448 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:19.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:19.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:19.452 INFO:tasks.rados.rados.0.smithi076.stdout:2921: done (11 left) 2021-03-15T22:32:19.453 INFO:tasks.rados.rados.0.smithi076.stdout:2922: done (10 left) 2021-03-15T22:32:19.453 INFO:tasks.rados.rados.0.smithi076.stdout:2924: done (9 left) 2021-03-15T22:32:19.453 INFO:tasks.rados.rados.0.smithi076.stdout:2925: done (8 left) 2021-03-15T22:32:19.454 INFO:tasks.rados.rados.0.smithi076.stdout:2926: done (7 left) 2021-03-15T22:32:19.454 INFO:tasks.rados.rados.0.smithi076.stdout:2928: done (6 left) 2021-03-15T22:32:19.454 INFO:tasks.rados.rados.0.smithi076.stdout:2930: done (5 left) 2021-03-15T22:32:19.454 INFO:tasks.rados.rados.0.smithi076.stdout:2931: done (4 left) 2021-03-15T22:32:19.454 INFO:tasks.rados.rados.0.smithi076.stdout:2934: done (3 left) 2021-03-15T22:32:19.455 INFO:tasks.rados.rados.0.smithi076.stdout:2936: done (2 left) 2021-03-15T22:32:19.455 INFO:tasks.rados.rados.0.smithi076.stdout:2938: done (1 left) 2021-03-15T22:32:19.455 INFO:tasks.rados.rados.0.smithi076.stdout:2939: done (0 left) 2021-03-15T22:32:19.455 INFO:tasks.rados.rados.0.smithi076.stdout:2940: read oid 7 snap -1 2021-03-15T22:32:19.456 INFO:tasks.rados.rados.0.smithi076.stdout:2940: expect (ObjNum 994 snap 333 seq_num 994) 2021-03-15T22:32:19.456 INFO:tasks.rados.rados.0.smithi076.stdout:2941: rollback oid 36 current snap is 341 2021-03-15T22:32:19.456 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 36 to 292 2021-03-15T22:32:19.456 INFO:tasks.rados.rados.0.smithi076.stdout:2942: snap_remove snap 307 2021-03-15T22:32:19.550 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:19.575 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:19.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:19.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:19.678 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:19.703 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:19.705 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:19.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:19 smithi076 conmon[56514]: cluster 2021-03-15T22:32:18.441993+0000 mon.a (mon.0 2021-03-15T22:32:19.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:19 smithi076 conmon[56514]: ) 2055 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2021-03-15T22:32:19.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:19 smithi076 conmon[56514]: audit 2021-03-15T22:32:18.971553+0000 mon.a (mon.0) 2056 : audit [DBG] from='client.? 172.21.15.76:0/724565404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:19.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:19 smithi076 conmon[62156]: cluster 2021-03-15T22:32:19.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:19 smithi076 conmon[62156]: 2021-03-15T22:32:19.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:19 smithi076 conmon[62156]: 2021-03-15T22:32:18 2021-03-15T22:32:19.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:19 smithi076 conmon[62156]: .441993+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2021-03-15T22:32:19.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:19 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:32:19.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:19 smithi076 conmon[62156]: T22:32:18.971553+0000 mon.a (mon.0) 2056 : audit 2021-03-15T22:32:19.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:19 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/724565404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:19.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:19 smithi125 conmon[56739]: cluster 2021-03-15T22:32:18.441993+0000 mon.a (mon.0) 2055 : cluster [DBG] osdmap e788: 8 total, 8 up, 7 in 2021-03-15T22:32:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:19 smithi125 conmon[56739]: audit 2021-03-15T22:32:18.971553+0000 mon.a (mon.0) 2056 : audit [DBG] from='client.? 172.21.15.76:0/724565404' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:19.805 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:19.829 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:19.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:19.830 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:19.931 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:19.958 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:19.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:19.960 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:20.061 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:20.086 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:20.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:20.088 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:20.189 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:20.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-15T22:32:20.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:20.216 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:20.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:20.259 INFO:tasks.rados.rados.0.smithi076.stdout:2942: done (2 left) 2021-03-15T22:32:20.259 INFO:tasks.rados.rados.0.smithi076.stdout:2943: write oid 28 current snap is 341 2021-03-15T22:32:20.260 INFO:tasks.rados.rados.0.smithi076.stdout:2943: seq_num 1013 ranges {625491=502186,1861671=713346} 2021-03-15T22:32:20.270 INFO:tasks.rados.rados.0.smithi076.stdout:2943: writing smithi07694878-28 from 625491 to 1127677 tid 1 2021-03-15T22:32:20.271 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:20.279 INFO:tasks.rados.rados.0.smithi076.stdout:2943: writing smithi07694878-28 from 1861671 to 2575017 tid 2 2021-03-15T22:32:20.288 INFO:tasks.rados.rados.0.smithi076.stdout:2944: write oid 43 current snap is 341 2021-03-15T22:32:20.288 INFO:tasks.rados.rados.0.smithi076.stdout:2940: expect (ObjNum 994 snap 333 seq_num 994) 2021-03-15T22:32:20.318 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:20.342 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:20.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:20.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:20.423 INFO:tasks.rados.rados.0.smithi076.stdout:2941: finishing rollback tid 0 to smithi07694878-36 2021-03-15T22:32:20.424 INFO:tasks.rados.rados.0.smithi076.stdout:2941: finishing rollback tid 1 to smithi07694878-36 2021-03-15T22:32:20.424 INFO:tasks.rados.rados.0.smithi076.stdout:2941: finishing rollback tid 2 to smithi07694878-36 2021-03-15T22:32:20.424 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 1770 (ObjNum 824 snap 274 seq_num 824) dirty exists 2021-03-15T22:32:20.424 INFO:tasks.rados.rados.0.smithi076.stdout:2944: seq_num 1014 ranges {642298=577435,1920153=573357,2979150=63148} 2021-03-15T22:32:20.430 INFO:tasks.rados.rados.0.smithi076.stdout:2944: writing smithi07694878-43 from 642298 to 1219733 tid 1 2021-03-15T22:32:20.437 INFO:tasks.rados.rados.0.smithi076.stdout:2944: writing smithi07694878-43 from 1920153 to 2493510 tid 2 2021-03-15T22:32:20.441 INFO:tasks.rados.rados.0.smithi076.stdout:2944: writing smithi07694878-43 from 2979150 to 3042298 tid 3 2021-03-15T22:32:20.441 INFO:tasks.rados.rados.0.smithi076.stdout:2940: done (3 left) 2021-03-15T22:32:20.441 INFO:tasks.rados.rados.0.smithi076.stdout:2941: done (2 left) 2021-03-15T22:32:20.441 INFO:tasks.rados.rados.0.smithi076.stdout:2945: rollback oid 27 current snap is 341 2021-03-15T22:32:20.442 INFO:tasks.rados.rados.0.smithi076.stdout:2943: finishing write tid 1 to smithi07694878-28 2021-03-15T22:32:20.442 INFO:tasks.rados.rados.0.smithi076.stdout:2943: finishing write tid 2 to smithi07694878-28 2021-03-15T22:32:20.442 INFO:tasks.rados.rados.0.smithi076.stdout:2943: finishing write tid 3 to smithi07694878-28 2021-03-15T22:32:20.442 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 326 2021-03-15T22:32:20.443 INFO:tasks.rados.rados.0.smithi076.stdout:2943: finishing write tid 4 to smithi07694878-28 2021-03-15T22:32:20.443 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2545 (ObjNum 1013 snap 341 seq_num 1013) dirty exists 2021-03-15T22:32:20.443 INFO:tasks.rados.rados.0.smithi076.stdout:2943: left oid 28 (ObjNum 1013 snap 341 seq_num 1013) 2021-03-15T22:32:20.443 INFO:tasks.rados.rados.0.smithi076.stdout:2944: finishing write tid 1 to smithi07694878-43 2021-03-15T22:32:20.443 INFO:tasks.rados.rados.0.smithi076.stdout:2943: done (2 left) 2021-03-15T22:32:20.444 INFO:tasks.rados.rados.0.smithi076.stdout:2946: snap_remove snap 327 2021-03-15T22:32:20.444 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:20.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:20 smithi125 conmon[56739]: cluster 2021-03-15T22:32:19.174153+0000 mgr.y (mgr.14144) 930 : cluster 2021-03-15T22:32:20.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:20 smithi125 conmon[56739]: [DBG] pgmap v1327: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 504 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 810 KiB/s wr, 9 op/s; 3/1686 objects misplaced (0.178%) 2021-03-15T22:32:20.451 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:20 smithi125 conmon[56739]: cluster 2021-03-15T22:32:19.449418+0000 mon.a (mon.0) 2057 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2021-03-15T22:32:20.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:20 smithi125 conmon[56739]: cluster 2021-03-15T22:32:20.255445+0000 mon.a (mon. 2021-03-15T22:32:20.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:20 smithi125 conmon[56739]: 0) 2058 : cluster [DBG] osdmap e790: 8 total, 8 up, 7 in 2021-03-15T22:32:20.452 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:32:20 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:32:20.470 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:20.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:20.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:20.512 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:20.572 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:20.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:20 smithi076 conmon[56514]: cluster 2021-03-15T22:32:19.174153+0000 mgr.y (mgr.14144) 930 : cluster [DBG] pgmap v1327: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 504 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 810 KiB/s wr, 9 op/s; 3/1686 objects misplaced (0.178%) 2021-03-15T22:32:20.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:20 smithi076 conmon[56514]: cluster 2021-03-15T22:32:19.449418+0000 mon.a (mon.0) 2057 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2021-03-15T22:32:20.590 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:20 smithi076 conmon[56514]: cluster 2021-03-15T22:32:20.255445+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e790: 8 total, 8 up, 7 in 2021-03-15T22:32:20.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:20 smithi076 conmon[62156]: cluster 2021-03-15T22:32:19.174153+0000 mgr.y (mgr.14144) 930 : cluster [DBG] pgmap v1327: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 504 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 810 KiB/s wr, 9 op/s; 3/1686 objects misplaced (0.178%) 2021-03-15T22:32:20.590 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:20 smithi076 conmon[62156]: cluster 2021-03-15T22:32:19.449418+0000 mon.a (mon.0) 2057 : cluster [DBG] osdmap e789: 8 total, 8 up, 7 in 2021-03-15T22:32:20.591 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:20 smithi076 conmon[62156]: cluster 2021-03-15T22:32:20.255445+0000 mon.a (mon.0) 2058 : cluster [DBG] osdmap e790: 8 total, 8 up, 7 in 2021-03-15T22:32:20.594 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:20.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:20.596 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:20.697 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:20.720 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:20.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:20.722 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:20.823 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:20.847 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:20.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:20.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:20.950 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:20.972 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:20.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:20.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:21.075 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:21.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:21.099 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:21.100 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:21.201 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:21.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:21.225 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:21.226 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:21.272 INFO:tasks.rados.rados.0.smithi076.stdout:2946: done (2 left) 2021-03-15T22:32:21.272 INFO:tasks.rados.rados.0.smithi076.stdout:2947: write (excl) oid 18 current snap is 341 2021-03-15T22:32:21.272 INFO:tasks.rados.rados.0.smithi076.stdout:2947: seq_num 1015 ranges {659105=652684,1862282=717015} 2021-03-15T22:32:21.281 INFO:tasks.rados.rados.0.smithi076.stdout:2947: writing smithi07694878-18 from 659105 to 1311789 tid 1 2021-03-15T22:32:21.291 INFO:tasks.rados.rados.0.smithi076.stdout:2947: writing smithi07694878-18 from 1862282 to 2579297 tid 2 2021-03-15T22:32:21.291 INFO:tasks.rados.rados.0.smithi076.stdout:2948: write oid 5 current snap is 341 2021-03-15T22:32:21.291 INFO:tasks.rados.rados.0.smithi076.stdout:2945: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:32:21.292 INFO:tasks.rados.rados.0.smithi076.stdout:2944: finishing write tid 2 to smithi07694878-43 2021-03-15T22:32:21.292 INFO:tasks.rados.rados.0.smithi076.stdout:2945: finishing rollback tid 2 to smithi07694878-27 2021-03-15T22:32:21.292 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1774 (ObjNum 896 snap 302 seq_num 896) dirty exists 2021-03-15T22:32:21.292 INFO:tasks.rados.rados.0.smithi076.stdout:2944: finishing write tid 3 to smithi07694878-43 2021-03-15T22:32:21.292 INFO:tasks.rados.rados.0.smithi076.stdout:2944: finishing write tid 4 to smithi07694878-43 2021-03-15T22:32:21.293 INFO:tasks.rados.rados.0.smithi076.stdout:2944: finishing write tid 5 to smithi07694878-43 2021-03-15T22:32:21.293 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 1968 (ObjNum 1014 snap 341 seq_num 1014) dirty exists 2021-03-15T22:32:21.293 INFO:tasks.rados.rados.0.smithi076.stdout:2944: left oid 43 (ObjNum 1014 snap 341 seq_num 1014) 2021-03-15T22:32:21.293 INFO:tasks.rados.rados.0.smithi076.stdout:2947: finishing write tid 1 to smithi07694878-18 2021-03-15T22:32:21.294 INFO:tasks.rados.rados.0.smithi076.stdout:2948: seq_num 1016 ranges {675912=727933,1920764=460673,2801290=274622} 2021-03-15T22:32:21.298 INFO:tasks.rados.rados.0.smithi076.stdout:2948: writing smithi07694878-5 from 675912 to 1403845 tid 1 2021-03-15T22:32:21.304 INFO:tasks.rados.rados.0.smithi076.stdout:2948: writing smithi07694878-5 from 1920764 to 2381437 tid 2 2021-03-15T22:32:21.308 INFO:tasks.rados.rados.0.smithi076.stdout:2948: writing smithi07694878-5 from 2801290 to 3075912 tid 3 2021-03-15T22:32:21.308 INFO:tasks.rados.rados.0.smithi076.stdout:2944: done (3 left) 2021-03-15T22:32:21.309 INFO:tasks.rados.rados.0.smithi076.stdout:2945: done (2 left) 2021-03-15T22:32:21.309 INFO:tasks.rados.rados.0.smithi076.stdout:2949: write (excl) oid 8 current snap is 341 2021-03-15T22:32:21.309 INFO:tasks.rados.rados.0.smithi076.stdout:2949: seq_num 1017 ranges {692719=403182,1579246=720684,2828713=264006} 2021-03-15T22:32:21.314 INFO:tasks.rados.rados.0.smithi076.stdout:2949: writing smithi07694878-8 from 692719 to 1095901 tid 1 2021-03-15T22:32:21.321 INFO:tasks.rados.rados.0.smithi076.stdout:2949: writing smithi07694878-8 from 1579246 to 2299930 tid 2 2021-03-15T22:32:21.325 INFO:tasks.rados.rados.0.smithi076.stdout:2949: writing smithi07694878-8 from 2828713 to 3092719 tid 3 2021-03-15T22:32:21.325 INFO:tasks.rados.rados.0.smithi076.stdout:2950: write (excl) oid 45 current snap is 341 2021-03-15T22:32:21.326 INFO:tasks.rados.rados.0.smithi076.stdout:2950: seq_num 1018 ranges {709526=478431,1637728=464342,2856136=253390} 2021-03-15T22:32:21.326 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:21.331 INFO:tasks.rados.rados.0.smithi076.stdout:2950: writing smithi07694878-45 from 709526 to 1187957 tid 1 2021-03-15T22:32:21.336 INFO:tasks.rados.rados.0.smithi076.stdout:2950: writing smithi07694878-45 from 1637728 to 2102070 tid 2 2021-03-15T22:32:21.342 INFO:tasks.rados.rados.0.smithi076.stdout:2950: writing smithi07694878-45 from 2856136 to 3109526 tid 3 2021-03-15T22:32:21.343 INFO:tasks.rados.rados.0.smithi076.stdout:2951: write oid 42 current snap is 341 2021-03-15T22:32:21.343 INFO:tasks.rados.rados.0.smithi076.stdout:2947: finishing write tid 2 to smithi07694878-18 2021-03-15T22:32:21.343 INFO:tasks.rados.rados.0.smithi076.stdout:2951: seq_num 1019 ranges {726333=670033,2096210=724353} 2021-03-15T22:32:21.348 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:21.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:21.349 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:21.350 INFO:tasks.rados.rados.0.smithi076.stdout:2951: writing smithi07694878-42 from 726333 to 1396366 tid 1 2021-03-15T22:32:21.359 INFO:tasks.rados.rados.0.smithi076.stdout:2951: writing smithi07694878-42 from 2096210 to 2820563 tid 2 2021-03-15T22:32:21.359 INFO:tasks.rados.rados.0.smithi076.stdout:2952: read oid 33 snap -1 2021-03-15T22:32:21.360 INFO:tasks.rados.rados.0.smithi076.stdout:2952: expect (ObjNum 714 snap 231 seq_num 714) 2021-03-15T22:32:21.360 INFO:tasks.rados.rados.0.smithi076.stdout:2953: write (excl) oid 31 current snap is 341 2021-03-15T22:32:21.360 INFO:tasks.rados.rados.0.smithi076.stdout:2947: finishing write tid 3 to smithi07694878-18 2021-03-15T22:32:21.360 INFO:tasks.rados.rados.0.smithi076.stdout:2947: finishing write tid 4 to smithi07694878-18 2021-03-15T22:32:21.361 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 1971 (ObjNum 1015 snap 341 seq_num 1015) dirty exists 2021-03-15T22:32:21.361 INFO:tasks.rados.rados.0.smithi076.stdout:2947: left oid 18 (ObjNum 1015 snap 341 seq_num 1015) 2021-03-15T22:32:21.361 INFO:tasks.rados.rados.0.smithi076.stdout:2948: finishing write tid 1 to smithi07694878-5 2021-03-15T22:32:21.361 INFO:tasks.rados.rados.0.smithi076.stdout:2953: seq_num 1020 ranges {743140=745282,2154692=468011} 2021-03-15T22:32:21.367 INFO:tasks.rados.rados.0.smithi076.stdout:2953: writing smithi07694878-31 from 743140 to 1488422 tid 1 2021-03-15T22:32:21.376 INFO:tasks.rados.rados.0.smithi076.stdout:2953: writing smithi07694878-31 from 2154692 to 2622703 tid 2 2021-03-15T22:32:21.376 INFO:tasks.rados.rados.0.smithi076.stdout:2947: done (6 left) 2021-03-15T22:32:21.376 INFO:tasks.rados.rados.0.smithi076.stdout:2954: delete oid 9 current snap is 341 2021-03-15T22:32:21.376 INFO:tasks.rados.rados.0.smithi076.stdout:2948: finishing write tid 2 to smithi07694878-5 2021-03-15T22:32:21.377 INFO:tasks.rados.rados.0.smithi076.stdout:2948: finishing write tid 3 to smithi07694878-5 2021-03-15T22:32:21.377 INFO:tasks.rados.rados.0.smithi076.stdout:2948: finishing write tid 4 to smithi07694878-5 2021-03-15T22:32:21.377 INFO:tasks.rados.rados.0.smithi076.stdout:2948: finishing write tid 5 to smithi07694878-5 2021-03-15T22:32:21.377 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1977 (ObjNum 1016 snap 341 seq_num 1016) dirty exists 2021-03-15T22:32:21.378 INFO:tasks.rados.rados.0.smithi076.stdout:2948: left oid 5 (ObjNum 1016 snap 341 seq_num 1016) 2021-03-15T22:32:21.378 INFO:tasks.rados.rados.0.smithi076.stdout:2949: finishing write tid 1 to smithi07694878-8 2021-03-15T22:32:21.378 INFO:tasks.rados.rados.0.smithi076.stdout:2949: finishing write tid 2 to smithi07694878-8 2021-03-15T22:32:21.378 INFO:tasks.rados.rados.0.smithi076.stdout:2949: finishing write tid 3 to smithi07694878-8 2021-03-15T22:32:21.379 INFO:tasks.rados.rados.0.smithi076.stdout:2949: finishing write tid 4 to smithi07694878-8 2021-03-15T22:32:21.379 INFO:tasks.rados.rados.0.smithi076.stdout:2949: finishing write tid 5 to smithi07694878-8 2021-03-15T22:32:21.379 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1250 (ObjNum 1017 snap 341 seq_num 1017) dirty exists 2021-03-15T22:32:21.379 INFO:tasks.rados.rados.0.smithi076.stdout:2949: left oid 8 (ObjNum 1017 snap 341 seq_num 1017) 2021-03-15T22:32:21.380 INFO:tasks.rados.rados.0.smithi076.stdout:2950: finishing write tid 1 to smithi07694878-45 2021-03-15T22:32:21.380 INFO:tasks.rados.rados.0.smithi076.stdout:2950: finishing write tid 2 to smithi07694878-45 2021-03-15T22:32:21.380 INFO:tasks.rados.rados.0.smithi076.stdout:2950: finishing write tid 3 to smithi07694878-45 2021-03-15T22:32:21.380 INFO:tasks.rados.rados.0.smithi076.stdout:2950: finishing write tid 4 to smithi07694878-45 2021-03-15T22:32:21.381 INFO:tasks.rados.rados.0.smithi076.stdout:2950: finishing write tid 5 to smithi07694878-45 2021-03-15T22:32:21.381 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 2088 (ObjNum 1018 snap 341 seq_num 1018) dirty exists 2021-03-15T22:32:21.381 INFO:tasks.rados.rados.0.smithi076.stdout:2950: left oid 45 (ObjNum 1018 snap 341 seq_num 1018) 2021-03-15T22:32:21.381 INFO:tasks.rados.rados.0.smithi076.stdout:2951: finishing write tid 1 to smithi07694878-42 2021-03-15T22:32:21.382 INFO:tasks.rados.rados.0.smithi076.stdout:2952: expect (ObjNum 714 snap 231 seq_num 714) 2021-03-15T22:32:21.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:21.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:21.473 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:21.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:21.479 INFO:tasks.rados.rados.0.smithi076.stdout:2951: finishing write tid 2 to smithi07694878-42 2021-03-15T22:32:21.479 INFO:tasks.rados.rados.0.smithi076.stdout:2951: finishing write tid 3 to smithi07694878-42 2021-03-15T22:32:21.480 INFO:tasks.rados.rados.0.smithi076.stdout:2951: finishing write tid 4 to smithi07694878-42 2021-03-15T22:32:21.480 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2557 (ObjNum 1019 snap 341 seq_num 1019) dirty exists 2021-03-15T22:32:21.480 INFO:tasks.rados.rados.0.smithi076.stdout:2951: left oid 42 (ObjNum 1019 snap 341 seq_num 1019) 2021-03-15T22:32:21.480 INFO:tasks.rados.rados.0.smithi076.stdout:2953: finishing write tid 1 to smithi07694878-31 2021-03-15T22:32:21.481 INFO:tasks.rados.rados.0.smithi076.stdout:2948: done (6 left) 2021-03-15T22:32:21.481 INFO:tasks.rados.rados.0.smithi076.stdout:2949: done (5 left) 2021-03-15T22:32:21.481 INFO:tasks.rados.rados.0.smithi076.stdout:2950: done (4 left) 2021-03-15T22:32:21.481 INFO:tasks.rados.rados.0.smithi076.stdout:2951: done (3 left) 2021-03-15T22:32:21.482 INFO:tasks.rados.rados.0.smithi076.stdout:2952: done (2 left) 2021-03-15T22:32:21.482 INFO:tasks.rados.rados.0.smithi076.stdout:2954: done (1 left) 2021-03-15T22:32:21.482 INFO:tasks.rados.rados.0.smithi076.stdout:2955: copy_from oid 4 from oid 18 current snap is 341 2021-03-15T22:32:21.482 INFO:tasks.rados.rados.0.smithi076.stdout:2953: finishing write tid 2 to smithi07694878-31 2021-03-15T22:32:21.483 INFO:tasks.rados.rados.0.smithi076.stdout:2953: finishing write tid 3 to smithi07694878-31 2021-03-15T22:32:21.483 INFO:tasks.rados.rados.0.smithi076.stdout:2953: finishing write tid 4 to smithi07694878-31 2021-03-15T22:32:21.483 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 1975 (ObjNum 1020 snap 341 seq_num 1020) dirty exists 2021-03-15T22:32:21.483 INFO:tasks.rados.rados.0.smithi076.stdout:2953: left oid 31 (ObjNum 1020 snap 341 seq_num 1020) 2021-03-15T22:32:21.484 INFO:tasks.rados.rados.0.smithi076.stdout:2953: done (1 left) 2021-03-15T22:32:21.484 INFO:tasks.rados.rados.0.smithi076.stdout:2956: write (excl) oid 48 current snap is 341 2021-03-15T22:32:21.484 INFO:tasks.rados.rados.0.smithi076.stdout:2956: seq_num 1022 ranges {776754=495780,1871656=471680,2965828=210926} 2021-03-15T22:32:21.485 INFO:tasks.rados.rados.0.smithi076.stdout:2956: writing smithi07694878-48 from 776754 to 1272534 tid 1 2021-03-15T22:32:21.491 INFO:tasks.rados.rados.0.smithi076.stdout:2956: writing smithi07694878-48 from 1871656 to 2343336 tid 2 2021-03-15T22:32:21.497 INFO:tasks.rados.rados.0.smithi076.stdout:2956: writing smithi07694878-48 from 2965828 to 3176754 tid 3 2021-03-15T22:32:21.497 INFO:tasks.rados.rados.0.smithi076.stdout:2957: read oid 43 snap 326 2021-03-15T22:32:21.497 INFO:tasks.rados.rados.0.smithi076.stdout:2957: expect (ObjNum 935 snap 316 seq_num 935) 2021-03-15T22:32:21.498 INFO:tasks.rados.rados.0.smithi076.stdout:2958: snap_remove snap 319 2021-03-15T22:32:21.574 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:21.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:21.598 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:21.598 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:21.699 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:21.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:21.723 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:21.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:21.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:21 smithi076 conmon[56514]: audit 2021-03-15T22:32:20.115838+0000 mgr.y (mgr.14144) 931 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:21.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:21 smithi076 conmon[56514]: cluster 2021-03-15T22:32:21.265797+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2021-03-15T22:32:21.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:21 smithi076 conmon[62156]: audit 2021-03-15T22:32:20.115838+0000 mgr.y (mgr.14144 2021-03-15T22:32:21.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:21 smithi076 conmon[62156]: ) 931 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:21.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:21 smithi076 conmon[62156]: cluster 2021-03-15T22:32:21.265797+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2021-03-15T22:32:21.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:21 smithi125 conmon[56739]: audit 2021-03-15T22:32:20.115838+0000 mgr.y (mgr.14144) 931 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:21.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:21 smithi125 conmon[56739]: cluster 2021-03-15T22:32:21.265797+0000 mon.a (mon.0) 2059 : cluster [DBG] osdmap e791: 8 total, 8 up, 7 in 2021-03-15T22:32:21.825 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:21.849 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:21.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:21.851 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:21.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:21.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:21.975 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:21.975 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:22.048 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:32:22.048 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:32:22.049 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:32:22.049 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:32:22.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:22.101 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:22.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:22.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:22.204 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:22.227 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:22.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:22.228 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:22.329 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:22.338 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:32:22.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:22.355 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:22.356 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:22.457 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:22.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:22.483 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:22.483 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:22.484 INFO:tasks.rados.rados.0.smithi076.stdout:2958: done (3 left) 2021-03-15T22:32:22.484 INFO:tasks.rados.rados.0.smithi076.stdout:2959: rollback oid 30 current snap is 341 2021-03-15T22:32:22.484 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 30 to 279 2021-03-15T22:32:22.484 INFO:tasks.rados.rados.0.smithi076.stdout:2960: write (excl) oid 45 current snap is 341 2021-03-15T22:32:22.485 INFO:tasks.rados.rados.0.smithi076.stdout:2960: seq_num 1023 ranges {793561=571029,1813785=731691,2993251=200310} 2021-03-15T22:32:22.485 INFO:tasks.rados.rados.0.smithi076.stdout:2960: writing smithi07694878-45 from 793561 to 1364590 tid 1 2021-03-15T22:32:22.493 INFO:tasks.rados.rados.0.smithi076.stdout:2960: writing smithi07694878-45 from 1813785 to 2545476 tid 2 2021-03-15T22:32:22.498 INFO:tasks.rados.rados.0.smithi076.stdout:2960: writing smithi07694878-45 from 2993251 to 3193561 tid 3 2021-03-15T22:32:22.502 INFO:tasks.rados.rados.0.smithi076.stdout:2955: finishing copy_from to smithi07694878-4 2021-03-15T22:32:22.503 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 1971 (ObjNum 1015 snap 341 seq_num 1015) dirty exists 2021-03-15T22:32:22.503 INFO:tasks.rados.rados.0.smithi076.stdout:2955: finishing copy_from racing read to smithi07694878-4 2021-03-15T22:32:22.503 INFO:tasks.rados.rados.0.smithi076.stdout:2956: finishing write tid 1 to smithi07694878-48 2021-03-15T22:32:22.503 INFO:tasks.rados.rados.0.smithi076.stdout:2956: finishing write tid 2 to smithi07694878-48 2021-03-15T22:32:22.503 INFO:tasks.rados.rados.0.smithi076.stdout:2956: finishing write tid 3 to smithi07694878-48 2021-03-15T22:32:22.504 INFO:tasks.rados.rados.0.smithi076.stdout:2956: finishing write tid 4 to smithi07694878-48 2021-03-15T22:32:22.504 INFO:tasks.rados.rados.0.smithi076.stdout:2956: finishing write tid 5 to smithi07694878-48 2021-03-15T22:32:22.504 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 1254 (ObjNum 1022 snap 341 seq_num 1022) dirty exists 2021-03-15T22:32:22.504 INFO:tasks.rados.rados.0.smithi076.stdout:2956: left oid 48 (ObjNum 1022 snap 341 seq_num 1022) 2021-03-15T22:32:22.505 INFO:tasks.rados.rados.0.smithi076.stdout:2957: expect (ObjNum 935 snap 316 seq_num 935) 2021-03-15T22:32:22.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:22.607 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:22.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:22.610 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:22.672 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-15T22:32:22.697 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:22.711 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:22.716 INFO:tasks.rados.rados.0.smithi076.stdout:2955: done (4 left) 2021-03-15T22:32:22.717 INFO:tasks.rados.rados.0.smithi076.stdout:2956: done (3 left) 2021-03-15T22:32:22.717 INFO:tasks.rados.rados.0.smithi076.stdout:2957: done (2 left) 2021-03-15T22:32:22.717 INFO:tasks.rados.rados.0.smithi076.stdout:2961: rollback oid 47 current snap is 341 2021-03-15T22:32:22.717 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 47 to 297 2021-03-15T22:32:22.718 INFO:tasks.rados.rados.0.smithi076.stdout:2960: finishing write tid 1 to smithi07694878-45 2021-03-15T22:32:22.718 INFO:tasks.rados.rados.0.smithi076.stdout:2959: finishing rollback tid 1 to smithi07694878-30 2021-03-15T22:32:22.718 INFO:tasks.rados.rados.0.smithi076.stdout:2960: finishing write tid 2 to smithi07694878-45 2021-03-15T22:32:22.718 INFO:tasks.rados.rados.0.smithi076.stdout:2960: finishing write tid 3 to smithi07694878-45 2021-03-15T22:32:22.718 INFO:tasks.rados.rados.0.smithi076.stdout:2960: finishing write tid 4 to smithi07694878-45 2021-03-15T22:32:22.719 INFO:tasks.rados.rados.0.smithi076.stdout:2960: finishing write tid 5 to smithi07694878-45 2021-03-15T22:32:22.719 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 2092 (ObjNum 1023 snap 341 seq_num 1023) dirty exists 2021-03-15T22:32:22.719 INFO:tasks.rados.rados.0.smithi076.stdout:2960: left oid 45 (ObjNum 1023 snap 341 seq_num 1023) 2021-03-15T22:32:22.719 INFO:tasks.rados.rados.0.smithi076.stdout:2959: finishing rollback tid 2 to smithi07694878-30 2021-03-15T22:32:22.720 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 2369 (ObjNum 774 snap 256 seq_num 774) dirty exists 2021-03-15T22:32:22.720 INFO:tasks.rados.rados.0.smithi076.stdout:2959: done (2 left) 2021-03-15T22:32:22.720 INFO:tasks.rados.rados.0.smithi076.stdout:2960: done (1 left) 2021-03-15T22:32:22.720 INFO:tasks.rados.rados.0.smithi076.stdout:2962: snap_create 2021-03-15T22:32:22.721 INFO:tasks.rados.rados.0.smithi076.stdout:2961: finishing rollback tid 0 to smithi07694878-47 2021-03-15T22:32:22.722 INFO:tasks.rados.rados.0.smithi076.stdout:2961: finishing rollback tid 1 to smithi07694878-47 2021-03-15T22:32:22.722 INFO:tasks.rados.rados.0.smithi076.stdout:2961: finishing rollback tid 2 to smithi07694878-47 2021-03-15T22:32:22.722 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1777 (ObjNum 669 snap 212 seq_num 669) dirty exists 2021-03-15T22:32:22.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:22.743 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:22.744 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:22.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:22 smithi076 conmon[56514]: cluster 2021-03-15T22:32:21.174645+0000 mgr.y (mgr.14144) 932 : cluster [DBG] pgmap v1330: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 476 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s; 3/1638 objects misplaced (0.183%) 2021-03-15T22:32:22.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:22 smithi076 conmon[62156]: cluster 2021-03-15T22:32:21.174645+0000 mgr.y (mgr.14144) 932 : cluster [DBG] pgmap v1330: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 476 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s; 3/1638 objects misplaced (0.183%) 2021-03-15T22:32:22.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:22 smithi125 conmon[56739]: cluster 2021-03-15T22:32:21.174645+0000 mgr.y (mgr.14144) 932 2021-03-15T22:32:22.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:22 smithi125 conmon[56739]: : cluster [DBG] pgmap v1330: 148 pgs: 2 active+remapped+backfill_toofull, 146 active+clean; 476 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s; 3/1638 objects misplaced (0.183%) 2021-03-15T22:32:22.845 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:22.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:22.869 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:22.870 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:22.949 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:22.971 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:22.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:22.995 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:23.096 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:23.119 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:23.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:23.121 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:23.222 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:23.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:23.246 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:23.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:23.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:23.370 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:23.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:23.371 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:23.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[56514]: cluster 2021-03-15T22:32:22.464148+0000 mon.a (mon. 2021-03-15T22:32:23.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[56514]: 0) 2060 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2021-03-15T22:32:23.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[56514]: audit 2021-03-15T22:32:22.538022+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? 172.21.15.76:0/3068463048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-15T22:32:23.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[56514]: audit 2021-03-15T22:32:22.763515+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? 172.21.15.76:0/3251013155' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:23.473 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:23.475 INFO:tasks.rados.rados.0.smithi076.stdout:2961: done (1 left) 2021-03-15T22:32:23.475 INFO:tasks.rados.rados.0.smithi076.stdout:2962: done (0 left) 2021-03-15T22:32:23.476 INFO:tasks.rados.rados.0.smithi076.stdout:2963: read oid 3 snap -1 2021-03-15T22:32:23.476 INFO:tasks.rados.rados.0.smithi076.stdout:2963: expect deleted 2021-03-15T22:32:23.476 INFO:tasks.rados.rados.0.smithi076.stdout:2964: delete oid 29 current snap is 342 2021-03-15T22:32:23.478 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:32:23.481 INFO:tasks.rados.rados.0.smithi076.stdout:2963: done (1 left) 2021-03-15T22:32:23.482 INFO:tasks.rados.rados.0.smithi076.stdout:2964: done (0 left) 2021-03-15T22:32:23.482 INFO:tasks.rados.rados.0.smithi076.stdout:2965: rollback oid 2 current snap is 342 2021-03-15T22:32:23.482 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 2 to 321 2021-03-15T22:32:23.482 INFO:tasks.rados.rados.0.smithi076.stdout:2966: snap_create 2021-03-15T22:32:23.483 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.0 to 1 (10000) 2021-03-15T22:32:23.485 INFO:tasks.rados.rados.0.smithi076.stdout:2965: finishing rollback tid 0 to smithi07694878-2 2021-03-15T22:32:23.486 INFO:tasks.rados.rados.0.smithi076.stdout:2965: finishing rollback tid 1 to smithi07694878-2 2021-03-15T22:32:23.486 INFO:tasks.rados.rados.0.smithi076.stdout:2965: finishing rollback tid 2 to smithi07694878-2 2021-03-15T22:32:23.486 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1977 (ObjNum 931 snap 315 seq_num 931) dirty exists 2021-03-15T22:32:23.497 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:23.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:23.499 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:23.600 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:23.622 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:23.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:23.624 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:23.725 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:23.746 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:23.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:23.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:23.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[62156]: cluster 2021-03-15T22:32:23.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:32:23.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[62156]: T22:32:22.464148+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2021-03-15T22:32:23.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[62156]: audit 2021-03-15T22:32:23.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[62156]: 2021-03-15T22:32:22.538022+0000 2021-03-15T22:32:23.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[62156]: mon.a (mon.0) 2061 : audit [INF] from='client.? 172.21.15.76:0/3068463048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-15T22:32:23.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[62156]: 2021-03-15T22:32:23.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[62156]: audit 2021-03-15T22:32:22 2021-03-15T22:32:23.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[62156]: .763515+0000 mon.a (mon.0) 2062 : audit 2021-03-15T22:32:23.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:23 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/3251013155' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:23.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:23 smithi125 conmon[56739]: cluster 2021-03-15T22:32:22.464148+0000 mon.a (mon.0) 2060 : cluster [DBG] osdmap e792: 8 total, 8 up, 7 in 2021-03-15T22:32:23.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:23 smithi125 conmon[56739]: audit 2021-03-15T22:32:22.538022+0000 mon.a (mon.0) 2061 : audit [INF] from='client.? 172.21.15.76:0/3068463048' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]: dispatch 2021-03-15T22:32:23.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:23 smithi125 conmon[56739]: audit 2021-03-15T22:32:22.763515+0000 mon.a (mon.0) 2062 : audit [INF] from='client.? 172.21.15.76:0/3251013155' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:23.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:23.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:23.872 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:23.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:23.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:23.996 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:23.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:23.997 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:24.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:24.120 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:24.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:24.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:24.223 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:24.245 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:24.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:24.246 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:24.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:24.369 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:24.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:24.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:24.472 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:24.478 INFO:tasks.rados.rados.0.smithi076.stdout:2965: done (1 left) 2021-03-15T22:32:24.479 INFO:tasks.rados.rados.0.smithi076.stdout:2966: done (0 left) 2021-03-15T22:32:24.479 INFO:tasks.rados.rados.0.smithi076.stdout:2967: read oid 30 snap -1 2021-03-15T22:32:24.480 INFO:tasks.rados.rados.0.smithi076.stdout:2967: expect (ObjNum 774 snap 256 seq_num 774) 2021-03-15T22:32:24.480 INFO:tasks.rados.rados.0.smithi076.stdout:2968: snap_create 2021-03-15T22:32:24.484 INFO:tasks.rados.rados.0.smithi076.stdout:2967: expect (ObjNum 774 snap 256 seq_num 774) 2021-03-15T22:32:24.498 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:24.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:24.500 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:24.551 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:32:24.552 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:32:24.552 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:32:24.552 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:32:24.552 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:32:24.601 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:24.624 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:24.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:24.625 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:24.726 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:24.748 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: cluster 2021-03-15T22:32:23.175090+0000 2021-03-15T22:32:24.748 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: mgr.y (mgr.14144) 933 : cluster [DBG] pgmap v1333: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 146 active+clean; 484 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2021-03-15T22:32:24.748 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: cluster 2021-03-15T22:32:24.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: 2021-03-15T22:32:23.460573+0000 mon.a (mon.0) 2063 : cluster 2021-03-15T22:32:24.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:32:24.749 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: cluster 2021-03-15T22:32:23.461338 2021-03-15T22:32:24.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2064 : cluster [WRN] 2021-03-15T22:32:24.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:24.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:32:24.750 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: 22:32:23.468045+0000 mon.a (mon 2021-03-15T22:32:24.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: .0) 2065 : audit [INF] from='client.? 172.21.15.76:0/3068463048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-15T22:32:24.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: audit 2021-03-15T22:32:24.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: 2021-03-15T22:32:23.468156+0000 2021-03-15T22:32:24.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: mon.a (mon.0) 2066 : audit [INF] 2021-03-15T22:32:24.751 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3251013155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:24.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:32:24.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: :32:23.468208+0000 mon.a (mon. 2021-03-15T22:32:24.752 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[62156]: 0) 2067 : cluster [DBG] osdmap e793: 8 total, 8 up, 7 in 2021-03-15T22:32:24.753 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: cluster 2021-03-15T22:32:23.175090+0000 mgr.y (mgr.14144 2021-03-15T22:32:24.753 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: ) 933 : cluster [DBG] pgmap v1333: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 146 active+clean; 484 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2021-03-15T22:32:24.753 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: cluster 2021-03-15T22:32:23.460573+0000 mon.a (mon.0) 2063 : cluster 2021-03-15T22:32:24.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:32:24.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: cluster 2021-03-15T22:32:24.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: 2021-03-15T22:32:23. 2021-03-15T22:32:24.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: 461338+0000 mon.a (mon.0) 2064 2021-03-15T22:32:24.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:24.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:32:24.756 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: 03-15T22:32:23. 2021-03-15T22:32:24.756 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: 468045+0000 mon.a (mon.0) 2021-03-15T22:32:24.757 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: 2065 : audit [INF] from='client.? 172.21.15.76:0/3068463048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-15T22:32:24.757 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: audit 2021 2021-03-15T22:32:24.757 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: -03-15T22:32:23 2021-03-15T22:32:24.757 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: .468156+0000 mon.a (mon. 2021-03-15T22:32:24.758 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: 0) 2066 : audit [INF] from='client.? 172.21.15.76:0/3251013155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:24.758 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: 2021-03-15T22:32:24.758 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: cluster 2021-03-15T22:32:24.758 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:32:24.759 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: 32:23.468208+0000 mon.a ( 2021-03-15T22:32:24.759 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: mon.0) 2067 : cluster [DBG] 2021-03-15T22:32:24.759 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:24 smithi076 conmon[56514]: osdmap e793: 8 total, 8 up, 7 in 2021-03-15T22:32:24.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:24.760 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:24.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:24.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:24 smithi125 conmon[56739]: cluster 2021-03-15T22:32:23.175090+0000 mgr.y (mgr.14144) 933 : cluster [DBG] pgmap v1333: 148 pgs: 1 peering, 1 active+remapped+backfill_toofull, 146 active+clean; 484 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 1.5 MiB/s wr, 15 op/s 2021-03-15T22:32:24.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:24 smithi125 conmon[56739]: cluster 2021-03-15T22:32: 2021-03-15T22:32:24.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:24 smithi125 conmon[56739]: 23.460573+0000 mon.a (mon.0) 2063 : cluster [WRN] Health check update: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:32:24.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:24 smithi125 conmon[56739]: cluster 2021-03-15T22:32:23.461338+0000 mon.a (mon.0) 2064 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:24.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:24 smithi125 conmon[56739]: audit 2021-03-15T22:32:23.468045+0000 mon.a (mon.0) 2065 : audit [INF] from='client.? 172.21.15.76:0/3068463048' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 0, "weight": 1.0}]': finished 2021-03-15T22:32:24.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:24 smithi125 conmon[56739]: audit 2021-03-15T22:32:23.468156+0000 mon.a (mon.0) 2066 : audit [INF] from='client.? 172.21.15.76:0/3251013155' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:24.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:24 smithi125 conmon[56739]: cluster 2021-03-15T22:32:23.468208+0000 mon.a (mon.0) 2067 : cluster [DBG] osdmap e793: 8 total, 8 up, 7 in 2021-03-15T22:32:24.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:24.885 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:24.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:24.886 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:24.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 3 1 2021-03-15T22:32:24.988 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:25.032 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:25.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:25.034 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:25.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:25.155 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-15T22:32:25.157 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:25.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:25.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:25.260 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:25.261 INFO:tasks.rados.rados.0.smithi076.stdout:2967: done (1 left) 2021-03-15T22:32:25.261 INFO:tasks.rados.rados.0.smithi076.stdout:2968: done (0 left) 2021-03-15T22:32:25.261 INFO:tasks.rados.rados.0.smithi076.stdout:2969: snap_remove snap 342 2021-03-15T22:32:25.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:25.284 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:25.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:25.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:25.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:25.411 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:25.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:25.434 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:25.436 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:25.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:25.517 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:32:25.535 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:25.536 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:32:25.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:25.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:25.553 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:32:25.572 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:32:25.591 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:32:25.609 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:32:25.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:25 smithi076 conmon[56514]: cluster 2021-03-15T22:32:24.472721+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2021-03-15T22:32:25.611 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:25 smithi076 conmon[56514]: cluster 2021-03-15T22:32:25.258322+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e795: 8 total, 8 up, 7 in 2021-03-15T22:32:25.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:25 smithi076 conmon[62156]: cluster 2021-03-15T22:32:24.472721+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2021-03-15T22:32:25.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:25 smithi076 conmon[62156]: cluster 2021-03-15T22:32:25.258322+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e795: 8 total, 8 up, 7 in 2021-03-15T22:32:25.629 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:32:25.639 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:25.647 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:32:25.650 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:32:25.650 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:32:25.651 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:32:25.651 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:32:25.651 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:32:25.651 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:32:25.652 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:32:25.652 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:32:25.663 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:25.664 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:25.765 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:25.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:25 smithi125 conmon[56739]: cluster 2021-03-15T22:32:24.472721+0000 mon.a (mon.0) 2068 : cluster [DBG] osdmap e794: 8 total, 8 up, 7 in 2021-03-15T22:32:25.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:25 smithi125 conmon[56739]: cluster 2021-03-15T22:32:25.258322+0000 mon.a (mon.0) 2069 : cluster [DBG] osdmap e795: 8 total, 8 up, 7 in 2021-03-15T22:32:25.787 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:25.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:25.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:25.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:25.915 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:25.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:26.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:26.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:26.042 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:26.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:26.144 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:26.166 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:26.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:26.168 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:26.260 INFO:tasks.rados.rados.0.smithi076.stdout:2969: done (0 left) 2021-03-15T22:32:26.260 INFO:tasks.rados.rados.0.smithi076.stdout:2970: snap_create 2021-03-15T22:32:26.269 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:26.292 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:26.293 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:26.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:26.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:26.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:26.418 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:26.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:26.520 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:26.547 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:26.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:26.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:26.650 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:26.673 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:26.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:26.675 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:26.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:26 smithi125 conmon[56739]: cluster 2021-03-15T22:32:25.175393+0000 mgr.y (mgr.14144) 934 : cluster [DBG] 2021-03-15T22:32:26.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:26 smithi125 conmon[56739]: pgmap v1336: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 146 active+clean; 496 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2021-03-15T22:32:26.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:26 smithi125 conmon[56739]: cluster 2021-03-15T22:32:25.342466+0000 osd.5 (osd.5) 30 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:26.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:26 smithi125 conmon[56739]: cluster 2021-03-15T22:32:25.343726+0000 osd.5 (osd.5) 31 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:26.744 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:26 smithi125 conmon[56739]: audit 2021-03-15T22:32:25.498221+0000 mon.a (mon.0) 2070 : audit [DBG] from='client.? 172.21.15.76:0/2580352513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:26.745 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:26 smithi125 conmon[56739]: cluster 2021-03-15T22:32:26.258512+0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e796: 8 total, 8 up, 7 in 2021-03-15T22:32:26.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[56514]: cluster 2021-03-15T22:32:25.175393 2021-03-15T22:32:26.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 934 : cluster [DBG] pgmap v1336: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 146 active+clean; 496 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2021-03-15T22:32:26.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[56514]: cluster 2021-03-15T22:32:25.342466+0000 osd.5 (osd.5) 30 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:26.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[56514]: cluster 2021-03-15T22:32:25.343726+0000 osd.5 (osd.5) 31 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:26.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[56514]: audit 2021-03-15T22:32:25.498221+0000 mon.a (mon.0) 2070 : audit [DBG] from='client.? 172.21.15.76:0/2580352513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:26.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[56514]: cluster 2021-03-15T22:32:26.258512+0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e796: 8 total, 8 up, 7 in 2021-03-15T22:32:26.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: cluster 2021-03-15T22:32:25.175393+0000 mgr.y (mgr.14144 2021-03-15T22:32:26.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: ) 934 : cluster [DBG] pgmap v1336: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 146 active+clean; 496 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.0 MiB/s rd, 2.4 MiB/s wr, 18 op/s 2021-03-15T22:32:26.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:32:26.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: :32:25.342466+0000 osd.5 (osd.5) 30 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:26.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: cluster 2021-03-15T22:32:26.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:32:26.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: 32:25.343726+0000 osd.5 (osd.5 2021-03-15T22:32:26.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: ) 31 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:26.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: audit 2021-03-15T22:32:26.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: 2021-03-15T22:32:25.498221 2021-03-15T22:32:26.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2070 : audit 2021-03-15T22:32:26.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/2580352513' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:26.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: cluster 2021-03-15T22:32:26.258512 2021-03-15T22:32:26.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:26 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2071 : cluster [DBG] osdmap e796: 8 total, 8 up, 7 in 2021-03-15T22:32:26.774 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:32:26.776 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:26.797 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:26.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:26.799 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:26.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:26.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:26.925 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:26.926 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:26.929 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:32:26.929 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:32:27.027 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:27.049 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:27.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:27.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:27.152 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:27.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:27.175 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:27.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:27.221 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:27.266 INFO:tasks.rados.rados.0.smithi076.stdout:2970: done (0 left) 2021-03-15T22:32:27.266 INFO:tasks.rados.rados.0.smithi076.stdout:2971: delete oid 8 current snap is 345 2021-03-15T22:32:27.270 INFO:tasks.rados.rados.0.smithi076.stdout:2971: done (0 left) 2021-03-15T22:32:27.270 INFO:tasks.rados.rados.0.smithi076.stdout:2972: snap_create 2021-03-15T22:32:27.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:27.299 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:27.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:27.301 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:27.402 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:27.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:27.425 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:27.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:27.513 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-15T22:32:27.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:27.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:27.552 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:27.553 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:27.654 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:27.676 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:27.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:27.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:27.779 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:27.786 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:27.802 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:27.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:27.803 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:27.905 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:27.927 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:27.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:27.928 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:28.030 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:28.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:28.054 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:28.054 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:28.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:28.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:28.180 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:28.180 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:28.269 INFO:tasks.rados.rados.0.smithi076.stdout:2972: done (0 left) 2021-03-15T22:32:28.269 INFO:tasks.rados.rados.0.smithi076.stdout:2973: read oid 9 snap -1 2021-03-15T22:32:28.270 INFO:tasks.rados.rados.0.smithi076.stdout:2973: expect deleted 2021-03-15T22:32:28.270 INFO:tasks.rados.rados.0.smithi076.stdout:2974: read oid 27 snap -1 2021-03-15T22:32:28.270 INFO:tasks.rados.rados.0.smithi076.stdout:2974: expect (ObjNum 896 snap 302 seq_num 896) 2021-03-15T22:32:28.271 INFO:tasks.rados.rados.0.smithi076.stdout:2975: copy_from oid 7 from oid 20 current snap is 346 2021-03-15T22:32:28.271 INFO:tasks.rados.rados.0.smithi076.stdout:2976: write (excl) oid 4 current snap is 346 2021-03-15T22:32:28.272 INFO:tasks.rados.rados.0.smithi076.stdout:2976: seq_num 1026 ranges {443982=796776,1989231=479018,2959167=284815} 2021-03-15T22:32:28.275 INFO:tasks.rados.rados.0.smithi076.stdout:2976: writing smithi07694878-4 from 443982 to 1240758 tid 1 2021-03-15T22:32:28.276 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.3 to 1 (10000) 2021-03-15T22:32:28.282 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:28.283 INFO:tasks.rados.rados.0.smithi076.stdout:2976: writing smithi07694878-4 from 1989231 to 2468249 tid 2 2021-03-15T22:32:28.288 INFO:tasks.rados.rados.0.smithi076.stdout:2976: writing smithi07694878-4 from 2959167 to 3243982 tid 3 2021-03-15T22:32:28.288 INFO:tasks.rados.rados.0.smithi076.stdout:2977: rollback oid 24 current snap is 346 2021-03-15T22:32:28.289 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 333 2021-03-15T22:32:28.289 INFO:tasks.rados.rados.0.smithi076.stdout:2974: expect (ObjNum 896 snap 302 seq_num 896) 2021-03-15T22:32:28.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:28.308 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:28.308 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:28.387 INFO:tasks.rados.rados.0.smithi076.stdout:2975: finishing copy_from to smithi07694878-7 2021-03-15T22:32:28.388 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 1763 (ObjNum 962 snap 321 seq_num 962) dirty exists 2021-03-15T22:32:28.388 INFO:tasks.rados.rados.0.smithi076.stdout:2975: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:32:28.388 INFO:tasks.rados.rados.0.smithi076.stdout:2976: finishing write tid 1 to smithi07694878-4 2021-03-15T22:32:28.388 INFO:tasks.rados.rados.0.smithi076.stdout:2976: finishing write tid 2 to smithi07694878-4 2021-03-15T22:32:28.389 INFO:tasks.rados.rados.0.smithi076.stdout:2973: done (4 left) 2021-03-15T22:32:28.389 INFO:tasks.rados.rados.0.smithi076.stdout:2974: done (3 left) 2021-03-15T22:32:28.389 INFO:tasks.rados.rados.0.smithi076.stdout:2975: done (2 left) 2021-03-15T22:32:28.389 INFO:tasks.rados.rados.0.smithi076.stdout:2978: read oid 36 snap -1 2021-03-15T22:32:28.389 INFO:tasks.rados.rados.0.smithi076.stdout:2978: expect (ObjNum 824 snap 274 seq_num 824) 2021-03-15T22:32:28.390 INFO:tasks.rados.rados.0.smithi076.stdout:2979: write oid 37 current snap is 346 2021-03-15T22:32:28.390 INFO:tasks.rados.rados.0.smithi076.stdout:2977: finishing rollback tid 0 to smithi07694878-24 2021-03-15T22:32:28.390 INFO:tasks.rados.rados.0.smithi076.stdout:2977: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:32:28.390 INFO:tasks.rados.rados.0.smithi076.stdout:2977: finishing rollback tid 2 to smithi07694878-24 2021-03-15T22:32:28.391 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1733 (ObjNum 864 snap 285 seq_num 864) dirty exists 2021-03-15T22:32:28.391 INFO:tasks.rados.rados.0.smithi076.stdout:2976: finishing write tid 3 to smithi07694878-4 2021-03-15T22:32:28.391 INFO:tasks.rados.rados.0.smithi076.stdout:2979: seq_num 1027 ranges {460789=588378,1764066=739029,3219296=41493} 2021-03-15T22:32:28.393 INFO:tasks.rados.rados.0.smithi076.stdout:2979: writing smithi07694878-37 from 460789 to 1049167 tid 1 2021-03-15T22:32:28.401 INFO:tasks.rados.rados.0.smithi076.stdout:2979: writing smithi07694878-37 from 1764066 to 2503095 tid 2 2021-03-15T22:32:28.406 INFO:tasks.rados.rados.0.smithi076.stdout:2979: writing smithi07694878-37 from 3219296 to 3260789 tid 3 2021-03-15T22:32:28.406 INFO:tasks.rados.rados.0.smithi076.stdout:2977: done (3 left) 2021-03-15T22:32:28.406 INFO:tasks.rados.rados.0.smithi076.stdout:2980: delete oid 48 current snap is 346 2021-03-15T22:32:28.407 INFO:tasks.rados.rados.0.smithi076.stdout:2976: finishing write tid 4 to smithi07694878-4 2021-03-15T22:32:28.407 INFO:tasks.rados.rados.0.smithi076.stdout:2976: finishing write tid 5 to smithi07694878-4 2021-03-15T22:32:28.407 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 1975 (ObjNum 1026 snap 346 seq_num 1026) dirty exists 2021-03-15T22:32:28.407 INFO:tasks.rados.rados.0.smithi076.stdout:2976: left oid 4 (ObjNum 1026 snap 346 seq_num 1026) 2021-03-15T22:32:28.407 INFO:tasks.rados.rados.0.smithi076.stdout:2978: expect (ObjNum 824 snap 274 seq_num 824) 2021-03-15T22:32:28.410 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:28.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:28.433 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:28.434 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:28.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:32:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:28 smithi076 conmon[56514]: cluster 2021-03-15T22:32:27.175939+0000 mgr.y (mgr.14144) 935 : cluster [DBG] pgmap v1339: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 146 active+clean; 506 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.3 MiB/s rd, 2.3 MiB/s wr, 14 op/s 2021-03-15T22:32:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:28 smithi076 conmon[56514]: cluster 2021-03-15T22:32:27.262357+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2021-03-15T22:32:28.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:28 smithi076 conmon[56514]: audit 2021-03-15T22:32:27.985605+0000 mon.a (mon.0) 2073 : audit [INF] from='client.? 172.21.15.76:0/3391258797' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-15T22:32:28.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:28 smithi076 conmon[62156]: cluster 2021-03-15T22:32:27.175939+0000 mgr.y (mgr.14144) 935 : cluster [DBG] pgmap v1339: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 146 active+clean; 506 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.3 MiB/s rd, 2.3 MiB/s wr, 14 op/s 2021-03-15T22:32:28.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:28 smithi076 conmon[62156]: cluster 2021-03-15T22:32:27.262357+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2021-03-15T22:32:28.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:28 smithi076 conmon[62156]: audit 2021-03-15T22:32:27.985605+0000 mon.a (mon.0) 2073 : audit [INF] from='client.? 172.21.15.76:0/3391258797' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-15T22:32:28.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:28.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:28 smithi125 conmon[56739]: cluster 2021-03-15T22:32:27.175939+0000 mgr.y (mgr.14144) 935 : cluster [DBG] pgmap v1339: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 146 active+clean; 506 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.3 MiB/s rd, 2.3 MiB/s wr, 14 op/s 2021-03-15T22:32:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:28 smithi125 conmon[56739]: cluster 2021-03-15T22:32:27.262357+0000 mon.a (mon.0) 2072 : cluster [DBG] osdmap e797: 8 total, 8 up, 7 in 2021-03-15T22:32:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:28 smithi125 conmon[56739]: audit 2021-03-15T22:32:27.985605+0000 mon.a (mon.0) 2073 : audit [INF] from='client.? 172.21.15.76:0/3391258797' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]: dispatch 2021-03-15T22:32:28.558 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:28.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:28.560 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:28.608 INFO:tasks.rados.rados.0.smithi076.stdout:2979: finishing write tid 1 to smithi07694878-37 2021-03-15T22:32:28.609 INFO:tasks.rados.rados.0.smithi076.stdout:2976: done (3 left) 2021-03-15T22:32:28.609 INFO:tasks.rados.rados.0.smithi076.stdout:2978: done (2 left) 2021-03-15T22:32:28.609 INFO:tasks.rados.rados.0.smithi076.stdout:2980: done (1 left) 2021-03-15T22:32:28.609 INFO:tasks.rados.rados.0.smithi076.stdout:2981: snap_remove snap 305 2021-03-15T22:32:28.661 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:28.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:28.685 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:28.685 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:28.786 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:28.808 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:28.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:28.810 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:28.911 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:28.933 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:28.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:28.935 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:29.036 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:29.058 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:29.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:29.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:29.154 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 5 1 2021-03-15T22:32:29.161 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:29.220 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:29.221 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:29.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:29.222 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:29.227 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:32:29.227 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:32:29.274 INFO:tasks.rados.rados.0.smithi076.stdout:2981: done (1 left) 2021-03-15T22:32:29.274 INFO:tasks.rados.rados.0.smithi076.stdout:2982: read oid 8 snap 300 2021-03-15T22:32:29.274 INFO:tasks.rados.rados.0.smithi076.stdout:2982: expect deleted 2021-03-15T22:32:29.275 INFO:tasks.rados.rados.0.smithi076.stdout:2983: rollback oid 28 current snap is 346 2021-03-15T22:32:29.275 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 344 2021-03-15T22:32:29.275 INFO:tasks.rados.rados.0.smithi076.stdout:2979: finishing write tid 2 to smithi07694878-37 2021-03-15T22:32:29.275 INFO:tasks.rados.rados.0.smithi076.stdout:2979: finishing write tid 3 to smithi07694878-37 2021-03-15T22:32:29.276 INFO:tasks.rados.rados.0.smithi076.stdout:2979: finishing write tid 4 to smithi07694878-37 2021-03-15T22:32:29.276 INFO:tasks.rados.rados.0.smithi076.stdout:2979: finishing write tid 5 to smithi07694878-37 2021-03-15T22:32:29.276 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 2575 (ObjNum 1027 snap 346 seq_num 1027) dirty exists 2021-03-15T22:32:29.276 INFO:tasks.rados.rados.0.smithi076.stdout:2979: left oid 37 (ObjNum 1027 snap 346 seq_num 1027) 2021-03-15T22:32:29.276 INFO:tasks.rados.rados.0.smithi076.stdout:2979: done (2 left) 2021-03-15T22:32:29.277 INFO:tasks.rados.rados.0.smithi076.stdout:2984: rollback oid 26 current snap is 346 2021-03-15T22:32:29.277 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 26 to 308 2021-03-15T22:32:29.277 INFO:tasks.rados.rados.0.smithi076.stdout:2985: write oid 38 current snap is 346 2021-03-15T22:32:29.277 INFO:tasks.rados.rados.0.smithi076.stdout:2985: seq_num 1029 ranges {494403=738876,1764677=626345,3041436=252967} 2021-03-15T22:32:29.281 INFO:tasks.rados.rados.0.smithi076.stdout:2985: writing smithi07694878-38 from 494403 to 1233279 tid 1 2021-03-15T22:32:29.289 INFO:tasks.rados.rados.0.smithi076.stdout:2985: writing smithi07694878-38 from 1764677 to 2391022 tid 2 2021-03-15T22:32:29.300 INFO:tasks.rados.rados.0.smithi076.stdout:2985: writing smithi07694878-38 from 3041436 to 3294403 tid 3 2021-03-15T22:32:29.301 INFO:tasks.rados.rados.0.smithi076.stdout:2985: finishing write tid 1 to smithi07694878-38 2021-03-15T22:32:29.301 INFO:tasks.rados.rados.0.smithi076.stdout:2982: done (3 left) 2021-03-15T22:32:29.301 INFO:tasks.rados.rados.0.smithi076.stdout:2986: copy_from oid 36 from oid 47 current snap is 346 2021-03-15T22:32:29.301 INFO:tasks.rados.rados.0.smithi076.stdout:2987: read oid 49 snap 311 2021-03-15T22:32:29.302 INFO:tasks.rados.rados.0.smithi076.stdout:2987: expect (ObjNum 862 snap 285 seq_num 862) 2021-03-15T22:32:29.302 INFO:tasks.rados.rados.0.smithi076.stdout:2988: delete oid 20 current snap is 346 2021-03-15T22:32:29.302 INFO:tasks.rados.rados.0.smithi076.stdout:2984: finishing rollback tid 0 to smithi07694878-26 2021-03-15T22:32:29.303 INFO:tasks.rados.rados.0.smithi076.stdout:2984: finishing rollback tid 1 to smithi07694878-26 2021-03-15T22:32:29.303 INFO:tasks.rados.rados.0.smithi076.stdout:2984: finishing rollback tid 2 to smithi07694878-26 2021-03-15T22:32:29.303 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 1775 (ObjNum 904 snap 304 seq_num 904) dirty exists 2021-03-15T22:32:29.303 INFO:tasks.rados.rados.0.smithi076.stdout:2983: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:32:29.304 INFO:tasks.rados.rados.0.smithi076.stdout:2985: finishing write tid 2 to smithi07694878-38 2021-03-15T22:32:29.306 INFO:tasks.rados.rados.0.smithi076.stdout:2984: done (5 left) 2021-03-15T22:32:29.307 INFO:tasks.rados.rados.0.smithi076.stdout:2988: done (4 left) 2021-03-15T22:32:29.307 INFO:tasks.rados.rados.0.smithi076.stdout:2989: delete oid 11 current snap is 346 2021-03-15T22:32:29.307 INFO:tasks.rados.rados.0.smithi076.stdout:2983: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:32:29.307 INFO:tasks.rados.rados.0.smithi076.stdout:2983: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:32:29.308 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2579 (ObjNum 1013 snap 341 seq_num 1013) dirty exists 2021-03-15T22:32:29.319 INFO:tasks.rados.rados.0.smithi076.stdout:2987: expect (ObjNum 862 snap 285 seq_num 862) 2021-03-15T22:32:29.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:29.346 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:29.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:29.351 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:29.452 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:29.477 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:29.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:29.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:29.490 INFO:tasks.rados.rados.0.smithi076.stdout:2983: done (4 left) 2021-03-15T22:32:29.491 INFO:tasks.rados.rados.0.smithi076.stdout:2987: done (3 left) 2021-03-15T22:32:29.491 INFO:tasks.rados.rados.0.smithi076.stdout:2989: done (2 left) 2021-03-15T22:32:29.491 INFO:tasks.rados.rados.0.smithi076.stdout:2990: rollback oid 49 current snap is 346 2021-03-15T22:32:29.491 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 49 to 344 2021-03-15T22:32:29.492 INFO:tasks.rados.rados.0.smithi076.stdout:2985: finishing write tid 3 to smithi07694878-38 2021-03-15T22:32:29.492 INFO:tasks.rados.rados.0.smithi076.stdout:2985: finishing write tid 4 to smithi07694878-38 2021-03-15T22:32:29.492 INFO:tasks.rados.rados.0.smithi076.stdout:2985: finishing write tid 5 to smithi07694878-38 2021-03-15T22:32:29.492 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2106 (ObjNum 1029 snap 346 seq_num 1029) dirty exists 2021-03-15T22:32:29.493 INFO:tasks.rados.rados.0.smithi076.stdout:2985: left oid 38 (ObjNum 1029 snap 346 seq_num 1029) 2021-03-15T22:32:29.493 INFO:tasks.rados.rados.0.smithi076.stdout:2986: finishing copy_from to smithi07694878-36 2021-03-15T22:32:29.493 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 1777 (ObjNum 669 snap 212 seq_num 669) dirty exists 2021-03-15T22:32:29.493 INFO:tasks.rados.rados.0.smithi076.stdout:2986: finishing copy_from racing read to smithi07694878-36 2021-03-15T22:32:29.493 INFO:tasks.rados.rados.0.smithi076.stdout:2985: done (2 left) 2021-03-15T22:32:29.494 INFO:tasks.rados.rados.0.smithi076.stdout:2986: done (1 left) 2021-03-15T22:32:29.494 INFO:tasks.rados.rados.0.smithi076.stdout:2991: delete oid 25 current snap is 346 2021-03-15T22:32:29.495 INFO:tasks.rados.rados.0.smithi076.stdout:2991: done (1 left) 2021-03-15T22:32:29.495 INFO:tasks.rados.rados.0.smithi076.stdout:2992: copy_from oid 34 from oid 48 current snap is 346 2021-03-15T22:32:29.495 INFO:tasks.rados.rados.0.smithi076.stdout:2993: write (excl) oid 22 current snap is 346 2021-03-15T22:32:29.495 INFO:tasks.rados.rados.0.smithi076.stdout:2993: seq_num 1033 ranges {561631=639872,1882252=633683,3034775=326856} 2021-03-15T22:32:29.502 INFO:tasks.rados.rados.0.smithi076.stdout:2993: writing smithi07694878-22 from 561631 to 1201503 tid 1 2021-03-15T22:32:29.509 INFO:tasks.rados.rados.0.smithi076.stdout:2993: writing smithi07694878-22 from 1882252 to 2515935 tid 2 2021-03-15T22:32:29.514 INFO:tasks.rados.rados.0.smithi076.stdout:2993: writing smithi07694878-22 from 3034775 to 3361631 tid 3 2021-03-15T22:32:29.515 INFO:tasks.rados.rados.0.smithi076.stdout:2994: read oid 42 snap -1 2021-03-15T22:32:29.515 INFO:tasks.rados.rados.0.smithi076.stdout:2994: expect (ObjNum 1019 snap 341 seq_num 1019) 2021-03-15T22:32:29.515 INFO:tasks.rados.rados.0.smithi076.stdout:2995: snap_remove snap 323 2021-03-15T22:32:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:29 smithi125 conmon[56739]: audit 2021-03-15T22:32:28.265404+0000 mon.a (mon 2021-03-15T22:32:29.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:29 smithi125 conmon[56739]: .0) 2074 : audit [INF] from='client.? 172.21.15.76:0/3391258797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-15T22:32:29.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:29 smithi125 conmon[56739]: cluster 2021-03-15T22:32:28.265469+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e798: 8 total, 8 up, 7 in 2021-03-15T22:32:29.572 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:29.579 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:29.594 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:29 smithi076 conmon[56514]: audit 2021-03-15T22:32:28.265404+0000 mon.a 2021-03-15T22:32:29.594 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:29 smithi076 conmon[56514]: (mon.0) 2074 : audit [INF] from='client.? 172.21.15.76:0/3391258797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-15T22:32:29.595 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:29 smithi076 conmon[56514]: cluster 2021-03-15T22:32:28.265469+0000 mon.a (mon.0) 2075 : cluster [DBG] osdmap e798: 8 total, 8 up, 7 in 2021-03-15T22:32:29.595 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:29 smithi076 conmon[62156]: audit 2021-03-15T22:32 2021-03-15T22:32:29.595 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:29 smithi076 conmon[62156]: :28.265404+0000 mon.a (mon.0) 2074 : audit [INF] from='client.? 172.21.15.76:0/3391258797' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 1.0}]': finished 2021-03-15T22:32:29.596 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:29 smithi076 conmon[62156]: cluster 2021-03-15T22:32:28.265469+0000 mon.a 2021-03-15T22:32:29.596 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:29 smithi076 conmon[62156]: (mon.0) 2075 : cluster [DBG] osdmap e798: 8 total, 8 up, 7 in 2021-03-15T22:32:29.601 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:29.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:29.603 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:29.704 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:29.726 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:29.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:29.728 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:29.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-15T22:32:29.829 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:29.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:29.855 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:29.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:29.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:29.980 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:29.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:29.982 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:30.070 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:30.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:30.106 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:30.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:30.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:30.208 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:30.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:30.233 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:30.234 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:30.264 INFO:tasks.rados.rados.0.smithi076.stdout:2995: done (4 left) 2021-03-15T22:32:30.265 INFO:tasks.rados.rados.0.smithi076.stdout:2996: write (excl) oid 45 current snap is 346 2021-03-15T22:32:30.265 INFO:tasks.rados.rados.0.smithi076.stdout:2996: seq_num 1034 ranges {578438=431474,1657087=493694,2778551=599887} 2021-03-15T22:32:30.268 INFO:tasks.rados.rados.0.smithi076.stdout:2996: writing smithi07694878-45 from 578438 to 1009912 tid 1 2021-03-15T22:32:30.275 INFO:tasks.rados.rados.0.smithi076.stdout:2996: writing smithi07694878-45 from 1657087 to 2150781 tid 2 2021-03-15T22:32:30.283 INFO:tasks.rados.rados.0.smithi076.stdout:2996: writing smithi07694878-45 from 2778551 to 3378438 tid 3 2021-03-15T22:32:30.284 INFO:tasks.rados.rados.0.smithi076.stdout:2990: finishing rollback tid 0 to smithi07694878-49 2021-03-15T22:32:30.284 INFO:tasks.rados.rados.0.smithi076.stdout:2990: finishing rollback tid 1 to smithi07694878-49 2021-03-15T22:32:30.284 INFO:tasks.rados.rados.0.smithi076.stdout:2990: finishing rollback tid 2 to smithi07694878-49 2021-03-15T22:32:30.285 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 2389 (ObjNum 945 snap 318 seq_num 945) dirty exists 2021-03-15T22:32:30.285 INFO:tasks.rados.rados.0.smithi076.stdout:2992: finishing copy_from to smithi07694878-34 2021-03-15T22:32:30.285 INFO:tasks.rados.rados.0.smithi076.stdout:2992: got expected ENOENT (src dne) 2021-03-15T22:32:30.286 INFO:tasks.rados.rados.0.smithi076.stdout:2992: finishing copy_from racing read to smithi07694878-34 2021-03-15T22:32:30.286 INFO:tasks.rados.rados.0.smithi076.stdout:2993: finishing write tid 1 to smithi07694878-22 2021-03-15T22:32:30.286 INFO:tasks.rados.rados.0.smithi076.stdout:2993: finishing write tid 2 to smithi07694878-22 2021-03-15T22:32:30.286 INFO:tasks.rados.rados.0.smithi076.stdout:2993: finishing write tid 3 to smithi07694878-22 2021-03-15T22:32:30.286 INFO:tasks.rados.rados.0.smithi076.stdout:2993: finishing write tid 4 to smithi07694878-22 2021-03-15T22:32:30.287 INFO:tasks.rados.rados.0.smithi076.stdout:2993: finishing write tid 5 to smithi07694878-22 2021-03-15T22:32:30.287 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 2393 (ObjNum 1033 snap 346 seq_num 1033) dirty exists 2021-03-15T22:32:30.287 INFO:tasks.rados.rados.0.smithi076.stdout:2993: left oid 22 (ObjNum 1033 snap 346 seq_num 1033) 2021-03-15T22:32:30.287 INFO:tasks.rados.rados.0.smithi076.stdout:2994: expect (ObjNum 1019 snap 341 seq_num 1019) 2021-03-15T22:32:30.335 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:30.357 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:30.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:30.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:30.387 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:32:30.463 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:30.484 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:30 smithi125 conmon[56739]: cluster 2021-03-15T22:32:29.176330+0000 mgr.y (mgr.14144) 936 : cluster [DBG] pgmap v1342: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 506 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-03-15T22:32:30.485 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:30 smithi125 conmon[56739]: 2021-03-15T22:32:30.485 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:30 smithi125 conmon[56739]: cluster 2021-03-15T22:32:29.271154+0000 mon.a (mon.0) 2076 : cluster [DBG] osdmap e799: 8 total, 8 up, 7 in 2021-03-15T22:32:30.485 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:30 smithi125 conmon[56739]: audit 2021-03-15T22:32:29.716682+0000 mon.a (mon.0) 2077 : audit [INF] from='client.? 172.21.15.76:0/4269221038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:32:30.485 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:30 smithi125 conmon[56739]: cluster 2021-03-15T22:32:30.254782+0000 mon.a (mon.0) 2078 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:30.486 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:30 smithi125 conmon[56739]: audit 2021-03-15T22:32:30.259647+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? 172.21.15.76:0/4269221038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:32:30.486 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:30 smithi125 conmon[56739]: cluster 2021-03-15T22:32:30.259713+0000 mon.a (mon.0) 2080 : cluster [DBG] osdmap e800: 8 total, 8 up, 7 in 2021-03-15T22:32:30.486 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:32:30 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:32:30.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:30.488 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:30.488 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:30.491 INFO:tasks.rados.rados.0.smithi076.stdout:2996: finishing write tid 1 to smithi07694878-45 2021-03-15T22:32:30.491 INFO:tasks.rados.rados.0.smithi076.stdout:2996: finishing write tid 2 to smithi07694878-45 2021-03-15T22:32:30.491 INFO:tasks.rados.rados.0.smithi076.stdout:2996: finishing write tid 3 to smithi07694878-45 2021-03-15T22:32:30.491 INFO:tasks.rados.rados.0.smithi076.stdout:2990: done (4 left) 2021-03-15T22:32:30.492 INFO:tasks.rados.rados.0.smithi076.stdout:2992: done (3 left) 2021-03-15T22:32:30.492 INFO:tasks.rados.rados.0.smithi076.stdout:2993: done (2 left) 2021-03-15T22:32:30.492 INFO:tasks.rados.rados.0.smithi076.stdout:2994: done (1 left) 2021-03-15T22:32:30.492 INFO:tasks.rados.rados.0.smithi076.stdout:2997: delete oid 1 current snap is 346 2021-03-15T22:32:30.493 INFO:tasks.rados.rados.0.smithi076.stdout:2996: finishing write tid 4 to smithi07694878-45 2021-03-15T22:32:30.493 INFO:tasks.rados.rados.0.smithi076.stdout:2996: finishing write tid 5 to smithi07694878-45 2021-03-15T22:32:30.493 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 2096 (ObjNum 1034 snap 346 seq_num 1034) dirty exists 2021-03-15T22:32:30.493 INFO:tasks.rados.rados.0.smithi076.stdout:2996: left oid 45 (ObjNum 1034 snap 346 seq_num 1034) 2021-03-15T22:32:30.494 INFO:tasks.rados.rados.0.smithi076.stdout:2996: done (1 left) 2021-03-15T22:32:30.494 INFO:tasks.rados.rados.0.smithi076.stdout:2997: done (0 left) 2021-03-15T22:32:30.494 INFO:tasks.rados.rados.0.smithi076.stdout:2998: snap_create 2021-03-15T22:32:30.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: cluster 2021-03-15T22:32:29.176330 2021-03-15T22:32:30.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 936 : cluster [DBG] pgmap v1342: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 506 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-03-15T22:32:30.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: cluster 2021-03-15T22:32:29. 2021-03-15T22:32:30.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: 271154+0000 mon.a (mon.0) 2021-03-15T22:32:30.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: 2076 : cluster [DBG] osdmap e799: 8 total, 8 up, 7 in 2021-03-15T22:32:30.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: 2021-03-15T22:32:30.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:32:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: 03-15T22:32:29.716682 2021-03-15T22:32:30.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2077 : audit 2021-03-15T22:32:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/4269221038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:32:30.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: 03-15T22:32:30.254782+0000 mon.a ( 2021-03-15T22:32:30.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: mon.0) 2078 : cluster [WRN] 2021-03-15T22:32:30.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:30.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:32:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: T22:32:30.259647+0000 mon.a (mon.0) 2079 : audit [INF] from='client.? 172.21.15.76:0/4269221038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:32:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:32:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: -03-15T22:32:30.259713+0000 2021-03-15T22:32:30.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: mon.a (mon.0) 2080 : cluster [DBG] 2021-03-15T22:32:30.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[62156]: osdmap e800: 8 total, 8 up, 7 in 2021-03-15T22:32:30.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: cluster 2021-03-15T22:32:30.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021 2021-03-15T22:32:30.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 03 2021-03-15T22:32:30.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 15 2021-03-15T22:32:30.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: T 2021-03-15T22:32:30.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 22 2021-03-15T22:32:30.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 32 2021-03-15T22:32:30.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 29 2021-03-15T22:32:30.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 176330 2021-03-15T22:32:30.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: +0000 2021-03-15T22:32:30.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mgr.y 2021-03-15T22:32:30.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ( 2021-03-15T22:32:30.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mgr 2021-03-15T22:32:30.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 14144 2021-03-15T22:32:30.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ) 2021-03-15T22:32:30.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 936 2021-03-15T22:32:30.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: cluster 2021-03-15T22:32:30.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: [DBG] 2021-03-15T22:32:30.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: pgmap v1342: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 506 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 5.3 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-03-15T22:32:30.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: cluster 2021-03-15T22:32:30.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021 2021-03-15T22:32:30.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 03 2021-03-15T22:32:30.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 15 2021-03-15T22:32:30.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: T 2021-03-15T22:32:30.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 22 2021-03-15T22:32:30.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 32 2021-03-15T22:32:30.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 29 2021-03-15T22:32:30.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 271154 2021-03-15T22:32:30.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: +0000 2021-03-15T22:32:30.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mon.a 2021-03-15T22:32:30.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ( 2021-03-15T22:32:30.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mon 2021-03-15T22:32:30.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 0 2021-03-15T22:32:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ) 2021-03-15T22:32:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2076 2021-03-15T22:32:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: cluster 2021-03-15T22:32:30.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: [DBG] 2021-03-15T22:32:30.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: osdmap e799: 8 total, 8 up, 7 in 2021-03-15T22:32:30.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: audit 2021-03-15T22:32:30.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021 2021-03-15T22:32:30.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 03 2021-03-15T22:32:30.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 15 2021-03-15T22:32:30.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: T 2021-03-15T22:32:30.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 22 2021-03-15T22:32:30.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 32 2021-03-15T22:32:30.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 29 2021-03-15T22:32:30.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 716682 2021-03-15T22:32:30.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: +0000 2021-03-15T22:32:30.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mon.a 2021-03-15T22:32:30.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ( 2021-03-15T22:32:30.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mon 2021-03-15T22:32:30.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 0 2021-03-15T22:32:30.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ) 2021-03-15T22:32:30.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2077 2021-03-15T22:32:30.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: audit 2021-03-15T22:32:30.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: [INF] 2021-03-15T22:32:30.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/4269221038' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:32:30.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: cluster 2021-03-15T22:32:30.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.550 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021 2021-03-15T22:32:30.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 03 2021-03-15T22:32:30.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.551 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 15 2021-03-15T22:32:30.552 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: T 2021-03-15T22:32:30.552 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 22 2021-03-15T22:32:30.552 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.552 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 32 2021-03-15T22:32:30.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 30 2021-03-15T22:32:30.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 254782 2021-03-15T22:32:30.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: +0000 2021-03-15T22:32:30.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mon.a 2021-03-15T22:32:30.554 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ( 2021-03-15T22:32:30.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mon 2021-03-15T22:32:30.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 0 2021-03-15T22:32:30.555 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ) 2021-03-15T22:32:30.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2078 2021-03-15T22:32:30.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: cluster 2021-03-15T22:32:30.556 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: [WRN] 2021-03-15T22:32:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:30.557 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: audit 2021-03-15T22:32:30.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021 2021-03-15T22:32:30.558 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 03 2021-03-15T22:32:30.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 15 2021-03-15T22:32:30.559 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: T 2021-03-15T22:32:30.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 22 2021-03-15T22:32:30.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 32 2021-03-15T22:32:30.560 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 30 2021-03-15T22:32:30.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 259647 2021-03-15T22:32:30.561 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: +0000 2021-03-15T22:32:30.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mon.a 2021-03-15T22:32:30.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ( 2021-03-15T22:32:30.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mon 2021-03-15T22:32:30.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 0 2021-03-15T22:32:30.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ) 2021-03-15T22:32:30.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2079 2021-03-15T22:32:30.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.564 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: audit 2021-03-15T22:32:30.564 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.564 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: [INF] 2021-03-15T22:32:30.564 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.565 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/4269221038' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:32:30.565 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.565 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: cluster 2021-03-15T22:32:30.565 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021 2021-03-15T22:32:30.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 03 2021-03-15T22:32:30.566 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: - 2021-03-15T22:32:30.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 15 2021-03-15T22:32:30.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: T 2021-03-15T22:32:30.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 22 2021-03-15T22:32:30.567 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 32 2021-03-15T22:32:30.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 30 2021-03-15T22:32:30.568 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 259713 2021-03-15T22:32:30.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: +0000 2021-03-15T22:32:30.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.569 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mon.a 2021-03-15T22:32:30.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ( 2021-03-15T22:32:30.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: mon 2021-03-15T22:32:30.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: . 2021-03-15T22:32:30.570 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 0 2021-03-15T22:32:30.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: ) 2021-03-15T22:32:30.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2080 2021-03-15T22:32:30.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: : 2021-03-15T22:32:30.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: cluster 2021-03-15T22:32:30.571 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: [DBG] 2021-03-15T22:32:30.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: osdmap e800: 8 total, 8 up, 7 in 2021-03-15T22:32:30.572 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:30 smithi076 conmon[56514]: 2021-03-15T22:32:30.589 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:30.612 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:30.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:30.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:30.716 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:30.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:30.739 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:30.740 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:30.841 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:30.863 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:30.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:30.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:30.966 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:30.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:30.990 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:30.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:31.092 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:31.115 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:31.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:31.116 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:31.217 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:31.240 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:31.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:31.242 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:31.263 INFO:tasks.rados.rados.0.smithi076.stdout:2998: done (0 left) 2021-03-15T22:32:31.264 INFO:tasks.rados.rados.0.smithi076.stdout:2999: read oid 46 snap -1 2021-03-15T22:32:31.264 INFO:tasks.rados.rados.0.smithi076.stdout:2999: expect (ObjNum 803 snap 268 seq_num 803) 2021-03-15T22:32:31.264 INFO:tasks.rados.rados.0.smithi076.stdout:3000: delete oid 3 current snap is 347 2021-03-15T22:32:31.267 INFO:tasks.rados.rados.0.smithi076.stdout:3000: done (1 left) 2021-03-15T22:32:31.267 INFO:tasks.rados.rados.0.smithi076.stdout:3001: copy_from oid 50 from oid 20 current snap is 347 2021-03-15T22:32:31.267 INFO:tasks.rados.rados.0.smithi076.stdout:3002: read oid 3 snap -1 2021-03-15T22:32:31.268 INFO:tasks.rados.rados.0.smithi076.stdout:3002: expect deleted 2021-03-15T22:32:31.268 INFO:tasks.rados.rados.0.smithi076.stdout:3003: read oid 14 snap 343 2021-03-15T22:32:31.268 INFO:tasks.rados.rados.0.smithi076.stdout:3003: expect (ObjNum 936 snap 316 seq_num 936) 2021-03-15T22:32:31.268 INFO:tasks.rados.rados.0.smithi076.stdout:3004: rollback oid 16 current snap is 347 2021-03-15T22:32:31.268 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 264 2021-03-15T22:32:31.269 INFO:tasks.rados.rados.0.smithi076.stdout:3005: delete oid 37 current snap is 347 2021-03-15T22:32:31.275 INFO:tasks.rados.rados.0.smithi076.stdout:3002: done (5 left) 2021-03-15T22:32:31.275 INFO:tasks.rados.rados.0.smithi076.stdout:3005: done (4 left) 2021-03-15T22:32:31.276 INFO:tasks.rados.rados.0.smithi076.stdout:3006: snap_remove snap 301 2021-03-15T22:32:31.343 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:31.361 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:31 smithi125 conmon[56739]: audit 2021-03-15T22:32:30.126329+0000 mgr.y (mgr.14144) 937 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:31.362 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:31 smithi125 conmon[56739]: cluster 2021-03-15T22:32:30.365428+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:32:31.362 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:31 smithi125 conmon[56739]: cluster 2021-03-15T22:32:31.261421+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e801: 8 total, 8 up, 7 in 2021-03-15T22:32:31.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:31.368 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:31.368 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:31.469 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:31.494 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:31.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:31.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:31.541 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:32:31.541 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:32:31.597 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:31.618 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:31 smithi076 conmon[56514]: audit 2021-03-15T22:32:30.126329+0000 mgr.y (mgr 2021-03-15T22:32:31.619 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:31 smithi076 conmon[56514]: .14144) 937 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:31.619 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:31 smithi076 conmon[56514]: cluster 2021-03-15T22:32:30.365428+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:32:31.619 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:31 smithi076 conmon[56514]: cluster 2021-03-15T22:32:31.261421+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e801: 8 total, 8 up, 7 in 2021-03-15T22:32:31.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:31 smithi076 conmon[62156]: audit 2021-03-15T22:32:30.126329 2021-03-15T22:32:31.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:31 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 937 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:31.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:31 smithi076 conmon[62156]: cluster 2021-03-15T22:32:30.365428+0000 osd.7 (osd.7) 146 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:32:31.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:31 smithi076 conmon[62156]: cluster 2021-03-15T22:32:31.261421+0000 mon.a (mon.0) 2081 : cluster [DBG] osdmap e801: 8 total, 8 up, 7 in 2021-03-15T22:32:31.621 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:31.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:31.623 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:31.724 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:31.747 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:31.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:31.748 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:31.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:31.873 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:31.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:31.875 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:31.976 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:31.999 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:32.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:32.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:32.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:32.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-15T22:32:32.126 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:32.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:32.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:32.228 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:32.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:32.253 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:32.253 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:32.286 INFO:tasks.rados.rados.0.smithi076.stdout:3006: done (4 left) 2021-03-15T22:32:32.286 INFO:tasks.rados.rados.0.smithi076.stdout:3007: delete oid 3 current snap is 347 2021-03-15T22:32:32.291 INFO:tasks.rados.rados.0.smithi076.stdout:3004: finishing rollback tid 0 to smithi07694878-16 2021-03-15T22:32:32.292 INFO:tasks.rados.rados.0.smithi076.stdout:3004: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:32:32.292 INFO:tasks.rados.rados.0.smithi076.stdout:3003: expect (ObjNum 936 snap 316 seq_num 936) 2021-03-15T22:32:32.355 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:32.376 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:32:32.377 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:32.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:32.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:32.386 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:32.399 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:32:32.416 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:32:32.428 INFO:tasks.rados.rados.0.smithi076.stdout:3001: finishing copy_from to smithi07694878-50 2021-03-15T22:32:32.428 INFO:tasks.rados.rados.0.smithi076.stdout:3001: got expected ENOENT (src dne) 2021-03-15T22:32:32.428 INFO:tasks.rados.rados.0.smithi076.stdout:3001: finishing copy_from racing read to smithi07694878-50 2021-03-15T22:32:32.429 INFO:tasks.rados.rados.0.smithi076.stdout:3004: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:32:32.429 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 2403 (ObjNum 604 snap 188 seq_num 604) dirty exists 2021-03-15T22:32:32.429 INFO:tasks.rados.rados.0.smithi076.stdout:2999: expect (ObjNum 803 snap 268 seq_num 803) 2021-03-15T22:32:32.434 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:32:32.463 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:32:32.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:32.483 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:32:32.500 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:32:32.503 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:32.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:32.504 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:32.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: cluster 2021-03-15T22:32:32.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: 2021-03-15T22:32:32.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: 2021 2021-03-15T22:32:32.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: - 2021-03-15T22:32:32.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: 03 2021-03-15T22:32:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: - 2021-03-15T22:32:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: 15 2021-03-15T22:32:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: T 2021-03-15T22:32:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: 22 2021-03-15T22:32:32.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: : 2021-03-15T22:32:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: 32 2021-03-15T22:32:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: : 2021-03-15T22:32:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: 30 2021-03-15T22:32:32.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: . 2021-03-15T22:32:32.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: 719427 2021-03-15T22:32:32.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: +0000 2021-03-15T22:32:32.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: 2021-03-15T22:32:32.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: osd.1 2021-03-15T22:32:32.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: ( 2021-03-15T22:32:32.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: osd 2021-03-15T22:32:32.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: . 2021-03-15T22:32:32.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: 1 2021-03-15T22:32:32.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: ) 2021-03-15T22:32:32.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: 155 2021-03-15T22:32:32.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: : 2021-03-15T22:32:32.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: cluster [DBG] 5.12 scrub ok 2021-03-15T22:32:32.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: cluster 2021-03-15T22:32:30.909656+0000 osd.5 (osd.5) 32 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:32:32.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: cluster 2021-03-15T22:32:31.034625+0000 osd.2 (osd.2) 24 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:32:32.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: cluster 2021-03-15T22:32:31.176794+0000 mgr.y (mgr.14144) 938 : cluster [DBG] pgmap v1345: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 515 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2021-03-15T22:32:32.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: cluster 2021-03-15T22:32:31.226509+0000 osd.2 (osd.2) 25 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:32.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: cluster 2021-03-15T22:32:31.228002+0000 osd.2 (osd.2) 26 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:32.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: cluster 2021-03-15T22:32:31.390342+0000 osd.7 (osd.7) 147 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:32:32.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[62156]: cluster 2021-03-15T22:32:31.680106+0000 osd.0 (osd.0) 129 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:32:32.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster 2021-03-15T22:32:32.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021 2021-03-15T22:32:32.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: - 2021-03-15T22:32:32.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 03 2021-03-15T22:32:32.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: - 2021-03-15T22:32:32.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 15 2021-03-15T22:32:32.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: T 2021-03-15T22:32:32.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 22 2021-03-15T22:32:32.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: : 2021-03-15T22:32:32.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 32 2021-03-15T22:32:32.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: : 2021-03-15T22:32:32.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 30 2021-03-15T22:32:32.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: . 2021-03-15T22:32:32.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 719427 2021-03-15T22:32:32.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: +0000 2021-03-15T22:32:32.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: osd.1 2021-03-15T22:32:32.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: ( 2021-03-15T22:32:32.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: osd 2021-03-15T22:32:32.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: . 2021-03-15T22:32:32.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 1 2021-03-15T22:32:32.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: ) 2021-03-15T22:32:32.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 155 2021-03-15T22:32:32.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: : 2021-03-15T22:32:32.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster 2021-03-15T22:32:32.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: [DBG] 2021-03-15T22:32:32.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 5.12 scrub ok 2021-03-15T22:32:32.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster 2021-03-15T22:32:32.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021 2021-03-15T22:32:32.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: - 2021-03-15T22:32:32.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 03 2021-03-15T22:32:32.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: - 2021-03-15T22:32:32.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 15 2021-03-15T22:32:32.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: T 2021-03-15T22:32:32.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 22 2021-03-15T22:32:32.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: : 2021-03-15T22:32:32.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 32 2021-03-15T22:32:32.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: : 2021-03-15T22:32:32.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 30 2021-03-15T22:32:32.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: . 2021-03-15T22:32:32.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 909656 2021-03-15T22:32:32.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: +0000 2021-03-15T22:32:32.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: osd.5 2021-03-15T22:32:32.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: ( 2021-03-15T22:32:32.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: osd 2021-03-15T22:32:32.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: . 2021-03-15T22:32:32.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 5 2021-03-15T22:32:32.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: ) 2021-03-15T22:32:32.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 32 2021-03-15T22:32:32.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: : 2021-03-15T22:32:32.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster 2021-03-15T22:32:32.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: [DBG] 2021-03-15T22:32:32.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 7.0 scrub ok 2021-03-15T22:32:32.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster 2021-03-15T22:32:32.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021 2021-03-15T22:32:32.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: - 2021-03-15T22:32:32.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 03 2021-03-15T22:32:32.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: - 2021-03-15T22:32:32.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 15 2021-03-15T22:32:32.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: T 2021-03-15T22:32:32.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 22 2021-03-15T22:32:32.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: : 2021-03-15T22:32:32.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 32 2021-03-15T22:32:32.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: : 2021-03-15T22:32:32.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 31 2021-03-15T22:32:32.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: . 2021-03-15T22:32:32.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 034625 2021-03-15T22:32:32.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: +0000 2021-03-15T22:32:32.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: osd.2 2021-03-15T22:32:32.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: ( 2021-03-15T22:32:32.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: osd 2021-03-15T22:32:32.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: . 2021-03-15T22:32:32.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2 2021-03-15T22:32:32.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: ) 2021-03-15T22:32:32.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 24 2021-03-15T22:32:32.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: : 2021-03-15T22:32:32.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster 2021-03-15T22:32:32.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: [DBG] 2021-03-15T22:32:32.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 7.8 scrub ok 2021-03-15T22:32:32.545 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster 2021-03-15T22:32:32.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:31.176794+0000 2021-03-15T22:32:32.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: mgr.y (mgr.14144) 938 : cluster [DBG] pgmap v1345: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 515 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2021-03-15T22:32:32.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:32.546 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster 2021-03-15T22:32:31 2021-03-15T22:32:32.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: .226509+0000 osd.2 (osd.2) 25 : 2021-03-15T22:32:32.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:32.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:32:32.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: T22:32:31.228002+0000 osd.2 (osd 2021-03-15T22:32:32.547 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: .2) 26 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:32.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:32:32.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: -03-15T22:32:31.390342+0000 osd.7 2021-03-15T22:32:32.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: (osd.7) 147 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:32:32.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: cluster 2021-03-15T22:32:32.548 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 2021-03-15T22:32:31.680106+0000 2021-03-15T22:32:32.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: osd.0 (osd.0) 129 : cluster [DBG] 2021-03-15T22:32:32.549 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:32 smithi076 conmon[56514]: 3.1f scrub ok 2021-03-15T22:32:32.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:32 smithi125 conmon[56739]: cluster 2021-03-15T22:32:30.719427+0000 osd.1 2021-03-15T22:32:32.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:32 smithi125 conmon[56739]: (osd.1) 155 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:32:32.550 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:32 smithi125 conmon[56739]: cluster 2021-03-15T22:32:30.909656+0000 osd.5 (osd.5) 32 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:32:32.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:32 smithi125 conmon[56739]: cluster 2021-03-15T22:32:31.034625+0000 osd.2 (osd.2) 24 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:32:32.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:32 smithi125 conmon[56739]: cluster 2021-03-15T22:32:31.176794+0000 mgr.y (mgr.14144) 938 : cluster [DBG] pgmap v1345: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 515 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 1.9 MiB/s wr, 15 op/s 2021-03-15T22:32:32.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:32 smithi125 conmon[56739]: cluster 2021-03-15T22:32:31.226509+0000 osd.2 (osd.2) 25 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:32.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:32 smithi125 conmon[56739]: cluster 2021-03-15T22:32:31.228002+0000 osd.2 (osd.2) 26 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:32.551 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:32 smithi125 conmon[56739]: cluster 2021-03-15T22:32:31.390342+0000 osd.7 (osd.7) 147 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:32:32.552 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:32 smithi125 conmon[56739]: cluster 2021-03-15T22:32:31.680106+0000 osd.0 (osd.0) 129 : cluster [DBG] 3.1f scrub ok 2021-03-15T22:32:32.552 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:32:32.552 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:32:32.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:32:32.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:32:32.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:32:32.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:32:32.553 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:32:32.554 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:32:32.554 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:32:32.606 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:32.628 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:32.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:32.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:32.684 INFO:tasks.rados.rados.0.smithi076.stdout:2999: done (4 left) 2021-03-15T22:32:32.685 INFO:tasks.rados.rados.0.smithi076.stdout:3001: done (3 left) 2021-03-15T22:32:32.685 INFO:tasks.rados.rados.0.smithi076.stdout:3003: done (2 left) 2021-03-15T22:32:32.685 INFO:tasks.rados.rados.0.smithi076.stdout:3004: done (1 left) 2021-03-15T22:32:32.685 INFO:tasks.rados.rados.0.smithi076.stdout:3007: done (0 left) 2021-03-15T22:32:32.686 INFO:tasks.rados.rados.0.smithi076.stdout:3008: snap_remove snap 346 2021-03-15T22:32:32.733 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:32.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:32.758 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:32.758 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:32.860 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:32.883 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:32.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:32.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:32.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:33.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:33.008 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:33.009 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:33.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:33.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:33.133 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:33.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:33.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:33.259 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:33.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:33.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:33.295 INFO:tasks.rados.rados.0.smithi076.stdout:3008: done (0 left) 2021-03-15T22:32:33.295 INFO:tasks.rados.rados.0.smithi076.stdout:3009: rollback oid 34 current snap is 347 2021-03-15T22:32:33.295 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 34 to 339 2021-03-15T22:32:33.296 INFO:tasks.rados.rados.0.smithi076.stdout:3010: write (excl) oid 32 current snap is 347 2021-03-15T22:32:33.296 INFO:tasks.rados.rados.0.smithi076.stdout:3010: seq_num 1039 ranges {662473=407719,1833144=644690,3199313=263160} 2021-03-15T22:32:33.297 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.5 to 1 (10000) 2021-03-15T22:32:33.299 INFO:tasks.rados.rados.0.smithi076.stdout:3010: writing smithi07694878-32 from 662473 to 1070192 tid 1 2021-03-15T22:32:33.308 INFO:tasks.rados.rados.0.smithi076.stdout:3010: writing smithi07694878-32 from 1833144 to 2477834 tid 2 2021-03-15T22:32:33.315 INFO:tasks.rados.rados.0.smithi076.stdout:3010: writing smithi07694878-32 from 3199313 to 3462473 tid 3 2021-03-15T22:32:33.315 INFO:tasks.rados.rados.0.smithi076.stdout:3011: write oid 14 current snap is 347 2021-03-15T22:32:33.315 INFO:tasks.rados.rados.0.smithi076.stdout:3009: finishing rollback tid 0 to smithi07694878-34 2021-03-15T22:32:33.315 INFO:tasks.rados.rados.0.smithi076.stdout:3009: finishing rollback tid 1 to smithi07694878-34 2021-03-15T22:32:33.316 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 1966 (ObjNum 0 snap 0 seq_num 261593152) dirty dne 2021-03-15T22:32:33.316 INFO:tasks.rados.rados.0.smithi076.stdout:3010: finishing write tid 1 to smithi07694878-32 2021-03-15T22:32:33.316 INFO:tasks.rados.rados.0.smithi076.stdout:3011: seq_num 1040 ranges {679280=482968,1775273=788348,3110383=368897} 2021-03-15T22:32:33.321 INFO:tasks.rados.rados.0.smithi076.stdout:3011: writing smithi07694878-14 from 679280 to 1162248 tid 1 2021-03-15T22:32:33.329 INFO:tasks.rados.rados.0.smithi076.stdout:3011: writing smithi07694878-14 from 1775273 to 2563621 tid 2 2021-03-15T22:32:33.335 INFO:tasks.rados.rados.0.smithi076.stdout:3011: writing smithi07694878-14 from 3110383 to 3479280 tid 3 2021-03-15T22:32:33.335 INFO:tasks.rados.rados.0.smithi076.stdout:3009: done (2 left) 2021-03-15T22:32:33.336 INFO:tasks.rados.rados.0.smithi076.stdout:3012: copy_from oid 41 from oid 21 current snap is 347 2021-03-15T22:32:33.336 INFO:tasks.rados.rados.0.smithi076.stdout:3010: finishing write tid 2 to smithi07694878-32 2021-03-15T22:32:33.336 INFO:tasks.rados.rados.0.smithi076.stdout:3010: finishing write tid 3 to smithi07694878-32 2021-03-15T22:32:33.336 INFO:tasks.rados.rados.0.smithi076.stdout:3010: finishing write tid 4 to smithi07694878-32 2021-03-15T22:32:33.337 INFO:tasks.rados.rados.0.smithi076.stdout:3010: finishing write tid 5 to smithi07694878-32 2021-03-15T22:32:33.337 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 2616 (ObjNum 1039 snap 347 seq_num 1039) dirty exists 2021-03-15T22:32:33.337 INFO:tasks.rados.rados.0.smithi076.stdout:3010: left oid 32 (ObjNum 1039 snap 347 seq_num 1039) 2021-03-15T22:32:33.337 INFO:tasks.rados.rados.0.smithi076.stdout:3011: finishing write tid 1 to smithi07694878-14 2021-03-15T22:32:33.338 INFO:tasks.rados.rados.0.smithi076.stdout:3010: done (2 left) 2021-03-15T22:32:33.338 INFO:tasks.rados.rados.0.smithi076.stdout:3013: snap_create 2021-03-15T22:32:33.338 INFO:tasks.rados.rados.0.smithi076.stdout:3011: finishing write tid 2 to smithi07694878-14 2021-03-15T22:32:33.339 INFO:tasks.rados.rados.0.smithi076.stdout:3011: finishing write tid 3 to smithi07694878-14 2021-03-15T22:32:33.340 INFO:tasks.rados.rados.0.smithi076.stdout:3011: finishing write tid 4 to smithi07694878-14 2021-03-15T22:32:33.340 INFO:tasks.rados.rados.0.smithi076.stdout:3011: finishing write tid 5 to smithi07694878-14 2021-03-15T22:32:33.340 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 1779 (ObjNum 1040 snap 347 seq_num 1040) dirty exists 2021-03-15T22:32:33.340 INFO:tasks.rados.rados.0.smithi076.stdout:3011: left oid 14 (ObjNum 1040 snap 347 seq_num 1040) 2021-03-15T22:32:33.346 INFO:tasks.rados.rados.0.smithi076.stdout:3012: finishing copy_from to smithi07694878-41 2021-03-15T22:32:33.346 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 2395 (ObjNum 982 snap 326 seq_num 982) dirty exists 2021-03-15T22:32:33.347 INFO:tasks.rados.rados.0.smithi076.stdout:3012: finishing copy_from racing read to smithi07694878-41 2021-03-15T22:32:33.362 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:33.385 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:33.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:33.386 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:33.487 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:33.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:33.511 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:33.512 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:33.613 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:33.636 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:33.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:33.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:33.706 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:33 smithi125 conmon[56739]: cluster 2021-03-15T22:32:31.747579+0000 osd.1 (osd.1) 156 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:32:33.706 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:33 smithi125 conmon[56739]: cluster 2021-03-15T22:32:32.281748+0000 mon.a (mon.0) 2082 : cluster [DBG] osdmap e802: 8 total, 8 up, 7 in 2021-03-15T22:32:33.707 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:33 smithi125 conmon[56739]: cluster 2021-03-15T22:32:32.328523+0000 osd.4 (osd.4) 169 : cluster [DBG] 4.c scrub ok 2021-03-15T22:32:33.707 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:33 smithi125 conmon[56739]: audit 2021-03-15T22:32:32.348918+0000 mon.a (mon.0) 2083 : audit [DBG] from='client.? 172.21.15.76:0/2463899755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:33.708 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:33 smithi125 conmon[56739]: cluster 2021-03-15T22:32:32.417215+0000 osd.7 (osd.7) 148 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:32:33.708 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:33 smithi125 conmon[56739]: audit 2021-03-15T22:32:32.562624+0000 mon.c (mon.1) 63 : audit [INF] from='client.? 172.21.15.76:0/1546123786' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:32:33.709 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:33 smithi125 conmon[56739]: audit 2021-03-15T22:32:32.562956+0000 mon.a (mon.0) 2084 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:32:33.739 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:33.761 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:33.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:33.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:33.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: cluster 2021-03-15T22:32:31.747579 2021-03-15T22:32:33.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 156 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:32:33.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:33.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: 03-15T22:32:32.281748+0000 mon.a (mon.0) 2082 : 2021-03-15T22:32:33.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: cluster [DBG] osdmap e802: 8 total, 8 up, 7 in 2021-03-15T22:32:33.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:32:33.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: T22:32:32.328523+0000 osd.4 (osd 2021-03-15T22:32:33.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: .4) 169 : cluster [DBG] 4.c scrub ok 2021-03-15T22:32:33.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:32:33.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: -15T22:32:32.348918+0000 mon.a (mon. 2021-03-15T22:32:33.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: 0) 2083 : audit [DBG] from='client.? 172.21.15.76:0/2463899755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:33.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: cluster 2021-03-15T22:32:32.417215+0000 2021-03-15T22:32:33.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: osd.7 (osd.7) 148 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:32:33.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: audit 2021-03-15T22:32:32 2021-03-15T22:32:33.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: .562624+0000 mon.c (mon.1) 63 : audit [INF] from='client.? 172.21.15.76:0/1546123786' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:32:33.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: 2021-03-15T22:32:33.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:32:33.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[62156]: 32:32.562956+0000 mon.a (mon.0) 2084 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:32:33.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: cluster 2021-03-15T22:32:33.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021-03-15 2021-03-15T22:32:33.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: T22:32:31.747579+0000 osd.1 (osd.1) 156 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:32:33.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: cluster 2021-03-15T22:32:33.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021-03-15T22:32:33.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021 2021-03-15T22:32:33.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: - 2021-03-15T22:32:33.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 03 2021-03-15T22:32:33.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: - 2021-03-15T22:32:33.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 15 2021-03-15T22:32:33.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: T 2021-03-15T22:32:33.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 22 2021-03-15T22:32:33.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: : 2021-03-15T22:32:33.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 32 2021-03-15T22:32:33.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: : 2021-03-15T22:32:33.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 32 2021-03-15T22:32:33.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: . 2021-03-15T22:32:33.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 281748 2021-03-15T22:32:33.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: +0000 2021-03-15T22:32:33.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021-03-15T22:32:33.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: mon.a 2021-03-15T22:32:33.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: ( 2021-03-15T22:32:33.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: mon 2021-03-15T22:32:33.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: . 2021-03-15T22:32:33.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 0 2021-03-15T22:32:33.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: ) 2021-03-15T22:32:33.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2082 2021-03-15T22:32:33.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: : 2021-03-15T22:32:33.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: cluster 2021-03-15T22:32:33.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021-03-15T22:32:33.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: [DBG] 2021-03-15T22:32:33.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021-03-15T22:32:33.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: osdmap e802: 8 total, 8 up, 7 in 2021-03-15T22:32:33.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021-03-15T22:32:33.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: cluster 2021-03-15T22:32:33.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021-03-15T22:32:33.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021 2021-03-15T22:32:33.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: - 2021-03-15T22:32:33.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 03 2021-03-15T22:32:33.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: - 2021-03-15T22:32:33.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 15 2021-03-15T22:32:33.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: T 2021-03-15T22:32:33.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 22 2021-03-15T22:32:33.781 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: : 2021-03-15T22:32:33.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 32 2021-03-15T22:32:33.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: : 2021-03-15T22:32:33.782 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 32 2021-03-15T22:32:33.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: . 2021-03-15T22:32:33.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 328523 2021-03-15T22:32:33.783 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: +0000 2021-03-15T22:32:33.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021-03-15T22:32:33.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: osd.4 2021-03-15T22:32:33.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: ( 2021-03-15T22:32:33.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: osd 2021-03-15T22:32:33.784 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: . 2021-03-15T22:32:33.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 4 2021-03-15T22:32:33.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: ) 2021-03-15T22:32:33.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 169 2021-03-15T22:32:33.785 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: : 2021-03-15T22:32:33.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: cluster [DBG] 4.c scrub ok 2021-03-15T22:32:33.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: audit 2021-03-15T22:32:32.348918+0000 2021-03-15T22:32:33.786 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: mon.a (mon.0) 2083 : audit [DBG] from='client.? 172.21.15.76:0/2463899755' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:33.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: cluster 2021-03-15T22:32:32.417215+0000 2021-03-15T22:32:33.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: osd.7 (osd.7) 148 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:32:33.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: audit 2021-03-15T22:32:33.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021-03-15 2021-03-15T22:32:33.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: T22:32:32.562624+0000 mon.c (mon.1 2021-03-15T22:32:33.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: ) 63 : audit [INF] from='client.? 172.21.15.76:0/1546123786' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:32:33.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 2021-03-15T22:32:33.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:32:33.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: 03-15T22:32:32.562956+0000 2021-03-15T22:32:33.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: mon.a (mon.0) 2084 : audit 2021-03-15T22:32:33.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:33 smithi076 conmon[56514]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2021-03-15T22:32:33.846 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:32:33.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:33.888 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:33.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:33.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:33.910 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2021-03-15T22:32:33.910 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:32:33.944 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:32:33.944 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:32:33.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:32:33.945 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:32:33.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.945 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:3010 7.b e330/329 7:d980be1f:::smithi07694878-23:head v 330'208, mlcod=0'0)", 2021-03-15T22:32:33.945 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:24:05.105001+0000", 2021-03-15T22:32:33.945 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 508.83467269200003, 2021-03-15T22:32:33.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037948101999999997, 2021-03-15T22:32:33.946 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:33.946 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:33.946 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:33.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:33.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105001+0000", 2021-03-15T22:32:33.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:33.948 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.948 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:33.949 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105001+0000", 2021-03-15T22:32:33.950 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076767199999999995 2021-03-15T22:32:33.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.950 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:33.950 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105769+0000", 2021-03-15T22:32:33.951 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1410000000000001e-06 2021-03-15T22:32:33.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.951 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.951 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:33.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105772+0000", 2021-03-15T22:32:33.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2334000000000001e-05 2021-03-15T22:32:33.952 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.952 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.952 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:33.953 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105784+0000", 2021-03-15T22:32:33.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4559999999999999e-06 2021-03-15T22:32:33.953 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.954 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:33.954 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105790+0000", 2021-03-15T22:32:33.954 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.66e-05 2021-03-15T22:32:33.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.955 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:33.955 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105846+0000", 2021-03-15T22:32:33.955 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0828000000000001e-05 2021-03-15T22:32:33.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.956 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.956 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:33.956 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.105867+0000", 2021-03-15T22:32:33.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.036960346999999998 2021-03-15T22:32:33.956 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.957 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.957 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:33.957 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.142827+0000", 2021-03-15T22:32:33.957 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121724 2021-03-15T22:32:33.957 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.958 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:33.958 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:24:05.142949+0000", 2021-03-15T22:32:33.958 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.037948101999999997 2021-03-15T22:32:33.959 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:33.959 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:33.959 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:33.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.960 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3950 7.7 7:ed4a01d2:::smithi07694878-2:head [write 658973~703921,stat] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:32:33.960 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:32:33.960 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 418.81243244299998, 2021-03-15T22:32:33.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99727343300000004, 2021-03-15T22:32:33.961 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:33.961 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:33.961 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:33.962 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:33.962 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:33.962 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3950 2021-03-15T22:32:33.962 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.962 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:33.963 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:33.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:32:33.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:33.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.964 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:33.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:32:33.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076287800000000004 2021-03-15T22:32:33.965 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:33.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128004+0000", 2021-03-15T22:32:33.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8489999999999999e-06 2021-03-15T22:32:33.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:33.966 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128008+0000", 2021-03-15T22:32:33.967 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3244e-05 2021-03-15T22:32:33.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.967 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.967 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:33.967 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128031+0000", 2021-03-15T22:32:33.968 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2050000000000002e-06 2021-03-15T22:32:33.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.969 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.969 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:33.970 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128039+0000", 2021-03-15T22:32:33.970 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9277999999999997e-05 2021-03-15T22:32:33.970 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.971 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:33.971 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128099+0000", 2021-03-15T22:32:33.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2401999999999999e-05 2021-03-15T22:32:33.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.972 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:33.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128111+0000", 2021-03-15T22:32:33.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99084192299999996 2021-03-15T22:32:33.973 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.973 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:33.973 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.118953+0000", 2021-03-15T22:32:33.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000162363 2021-03-15T22:32:33.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.974 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:33.974 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.119115+0000", 2021-03-15T22:32:33.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023901900000000001 2021-03-15T22:32:33.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.975 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.975 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:33.975 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.119354+0000", 2021-03-15T22:32:33.976 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020445160000000001 2021-03-15T22:32:33.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.976 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.976 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:33.976 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121399+0000", 2021-03-15T22:32:33.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021134299999999999 2021-03-15T22:32:33.977 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.977 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:33.977 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123512+0000", 2021-03-15T22:32:33.978 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000816028 2021-03-15T22:32:33.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.978 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.978 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:33.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124328+0000", 2021-03-15T22:32:33.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2724000000000005e-05 2021-03-15T22:32:33.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:33.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124391+0000", 2021-03-15T22:32:33.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012357399999999999 2021-03-15T22:32:33.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:33.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124515+0000", 2021-03-15T22:32:33.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99727343300000004 2021-03-15T22:32:33.981 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:33.981 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:33.982 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:33.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.982 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3951 7.7 7:ed4a01d2:::smithi07694878-2:head [write 1907912~570853] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:32:33.982 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:32:33.983 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 418.80569030700002, 2021-03-15T22:32:33.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99540528800000005, 2021-03-15T22:32:33.983 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:33.983 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:33.984 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:33.984 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:33.984 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:33.984 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3951 2021-03-15T22:32:33.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.985 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:33.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.985 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:33.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:32:33.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:33.986 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.986 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.986 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:33.986 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:32:33.987 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00050036199999999996 2021-03-15T22:32:33.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.987 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:33.987 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134484+0000", 2021-03-15T22:32:33.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9979999999999999e-06 2021-03-15T22:32:33.988 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.988 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.988 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:33.989 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134487+0000", 2021-03-15T22:32:33.989 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3523000000000001e-05 2021-03-15T22:32:33.989 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.989 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.989 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:33.990 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134500+0000", 2021-03-15T22:32:33.990 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5979999999999996e-06 2021-03-15T22:32:33.990 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.990 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.990 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:33.991 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134506+0000", 2021-03-15T22:32:33.991 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0407000000000001e-05 2021-03-15T22:32:33.991 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.992 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:33.992 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134546+0000", 2021-03-15T22:32:33.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5650000000000002e-06 2021-03-15T22:32:33.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.992 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.993 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:33.993 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134555+0000", 2021-03-15T22:32:33.993 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98647964300000002 2021-03-15T22:32:33.993 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.993 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:33.994 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121034+0000", 2021-03-15T22:32:33.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104298 2021-03-15T22:32:33.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.995 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:33.995 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121139+0000", 2021-03-15T22:32:33.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0034252140000000002 2021-03-15T22:32:33.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:33.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124564+0000", 2021-03-15T22:32:33.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3158000000000005e-05 2021-03-15T22:32:33.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.997 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:33.997 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124657+0000", 2021-03-15T22:32:33.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000172645 2021-03-15T22:32:33.998 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.998 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.998 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:33.998 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124830+0000", 2021-03-15T22:32:33.999 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001768096 2021-03-15T22:32:33.999 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:33.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:33.999 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:34.000 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126598+0000", 2021-03-15T22:32:34.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020473919999999999 2021-03-15T22:32:34.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128645+0000", 2021-03-15T22:32:34.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00067562299999999998 2021-03-15T22:32:34.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.001 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.002 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129321+0000", 2021-03-15T22:32:34.002 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3652999999999998e-05 2021-03-15T22:32:34.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129354+0000", 2021-03-15T22:32:34.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4112999999999997e-05 2021-03-15T22:32:34.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129389+0000", 2021-03-15T22:32:34.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99540528800000005 2021-03-15T22:32:34.004 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.004 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.005 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.005 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.005 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3952 7.7 7:ed4a01d2:::smithi07694878-2:head [write 2985346~73627] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:32:34.006 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:32:34.006 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 418.802495808, 2021-03-15T22:32:34.006 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99236444199999996, 2021-03-15T22:32:34.006 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.006 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:34.007 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.007 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.007 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.007 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3952 2021-03-15T22:32:34.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.008 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.008 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.008 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.008 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:32:34.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.009 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.009 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.009 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.010 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:32:34.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5050000000000006e-05 2021-03-15T22:32:34.010 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.010 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.011 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.011 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137273+0000", 2021-03-15T22:32:34.011 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.883e-06 2021-03-15T22:32:34.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.011 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.012 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.012 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137275+0000", 2021-03-15T22:32:34.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0764000000000001e-05 2021-03-15T22:32:34.012 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.012 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.013 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.013 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137285+0000", 2021-03-15T22:32:34.013 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2190000000000001e-06 2021-03-15T22:32:34.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.013 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.014 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137292+0000", 2021-03-15T22:32:34.014 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5432999999999997e-05 2021-03-15T22:32:34.014 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.015 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137337+0000", 2021-03-15T22:32:34.015 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.738e-06 2021-03-15T22:32:34.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:34.016 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137346+0000", 2021-03-15T22:32:34.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98385600600000001 2021-03-15T22:32:34.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.017 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121202+0000", 2021-03-15T22:32:34.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9746e-05 2021-03-15T22:32:34.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.018 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.018 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:34.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121262+0000", 2021-03-15T22:32:34.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041684740000000001 2021-03-15T22:32:34.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125430+0000", 2021-03-15T22:32:34.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0578000000000001e-05 2021-03-15T22:32:34.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125501+0000", 2021-03-15T22:32:34.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8120000000000002e-05 2021-03-15T22:32:34.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.021 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:34.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125599+0000", 2021-03-15T22:32:34.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010039140000000001 2021-03-15T22:32:34.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:34.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126603+0000", 2021-03-15T22:32:34.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020828969999999998 2021-03-15T22:32:34.023 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.024 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128686+0000", 2021-03-15T22:32:34.024 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00075311200000000001 2021-03-15T22:32:34.024 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.024 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.025 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129439+0000", 2021-03-15T22:32:34.025 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7739999999999999e-05 2021-03-15T22:32:34.025 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.026 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129496+0000", 2021-03-15T22:32:34.026 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5768e-05 2021-03-15T22:32:34.026 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.026 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.027 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129542+0000", 2021-03-15T22:32:34.027 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99236444199999996 2021-03-15T22:32:34.027 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.028 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.028 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.028 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.028 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3953 7.7 7:ed4a01d2:::smithi07694878-2:head [setxattr _header (57) in=7b,truncate 3058973] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:32:34.029 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:32:34.029 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 418.80237095400003, 2021-03-15T22:32:34.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99574138599999995, 2021-03-15T22:32:34.029 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.029 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:34.030 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.030 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.030 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.030 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3953 2021-03-15T22:32:34.031 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.031 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.031 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.031 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.031 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:32:34.032 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.032 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.032 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.032 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:32:34.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9230000000000001e-06 2021-03-15T22:32:34.033 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.033 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.033 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137305+0000", 2021-03-15T22:32:34.034 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.252e-06 2021-03-15T22:32:34.034 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.034 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.034 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137306+0000", 2021-03-15T22:32:34.035 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9039999999999997e-06 2021-03-15T22:32:34.035 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.036 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137312+0000", 2021-03-15T22:32:34.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.856e-06 2021-03-15T22:32:34.036 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.037 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137316+0000", 2021-03-15T22:32:34.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6783000000000002e-05 2021-03-15T22:32:34.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.038 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137372+0000", 2021-03-15T22:32:34.038 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9349999999999999e-06 2021-03-15T22:32:34.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:34.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137378+0000", 2021-03-15T22:32:34.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98391964700000001 2021-03-15T22:32:34.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.040 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.040 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121298+0000", 2021-03-15T22:32:34.040 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9109000000000004e-05 2021-03-15T22:32:34.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:34.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121387+0000", 2021-03-15T22:32:34.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045047059999999998 2021-03-15T22:32:34.042 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.042 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125892+0000", 2021-03-15T22:32:34.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6956999999999998e-05 2021-03-15T22:32:34.043 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.043 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.043 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125939+0000", 2021-03-15T22:32:34.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012742400000000001 2021-03-15T22:32:34.044 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.044 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.044 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:34.044 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126066+0000", 2021-03-15T22:32:34.045 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000545302 2021-03-15T22:32:34.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.045 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:34.045 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126611+0000", 2021-03-15T22:32:34.046 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021128950000000001 2021-03-15T22:32:34.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.046 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.046 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.046 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128724+0000", 2021-03-15T22:32:34.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042103729999999999 2021-03-15T22:32:34.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.132935+0000", 2021-03-15T22:32:34.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5052000000000001e-05 2021-03-15T22:32:34.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.049 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.132990+0000", 2021-03-15T22:32:34.049 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4268000000000003e-05 2021-03-15T22:32:34.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.049 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133044+0000", 2021-03-15T22:32:34.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99574138599999995 2021-03-15T22:32:34.050 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.050 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.050 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.051 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3954 7.7 7:ed4a01d2:::smithi07694878-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e420)", 2021-03-15T22:32:34.051 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:32:34.052 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 418.80234178500001, 2021-03-15T22:32:34.052 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99588332599999996, 2021-03-15T22:32:34.052 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.052 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:34.052 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.053 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.053 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.053 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3954 2021-03-15T22:32:34.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.053 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.054 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:32:34.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.055 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:32:34.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.015e-06 2021-03-15T22:32:34.056 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137333+0000", 2021-03-15T22:32:34.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5290000000000001e-06 2021-03-15T22:32:34.057 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.057 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137334+0000", 2021-03-15T22:32:34.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.489e-06 2021-03-15T22:32:34.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.058 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.058 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137341+0000", 2021-03-15T22:32:34.059 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1989999999999997e-06 2021-03-15T22:32:34.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.059 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137346+0000", 2021-03-15T22:32:34.060 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0890999999999998e-05 2021-03-15T22:32:34.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.060 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.061 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137397+0000", 2021-03-15T22:32:34.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3159999999999997e-06 2021-03-15T22:32:34.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.061 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:34.062 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137405+0000", 2021-03-15T22:32:34.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98402323800000002 2021-03-15T22:32:34.062 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.062 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.062 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121428+0000", 2021-03-15T22:32:34.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2376999999999998e-05 2021-03-15T22:32:34.063 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:34.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121471+0000", 2021-03-15T22:32:34.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050435389999999997 2021-03-15T22:32:34.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.064 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126514+0000", 2021-03-15T22:32:34.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3162000000000003e-05 2021-03-15T22:32:34.065 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:34.066 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126578+0000", 2021-03-15T22:32:34.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0065105759999999997 2021-03-15T22:32:34.066 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.067 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133088+0000", 2021-03-15T22:32:34.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4956000000000006e-05 2021-03-15T22:32:34.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133153+0000", 2021-03-15T22:32:34.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2038999999999993e-05 2021-03-15T22:32:34.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133215+0000", 2021-03-15T22:32:34.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99588332599999996 2021-03-15T22:32:34.070 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.070 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.070 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.071 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3970 7.7 7:ecc0cb20:::smithi07694878-25:head [write 709394~646021] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:32:34.071 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:32:34.071 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 418.660404922, 2021-03-15T22:32:34.071 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84719129900000001, 2021-03-15T22:32:34.072 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.072 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:34.072 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.072 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.072 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.073 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3970 2021-03-15T22:32:34.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.073 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.073 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.073 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.074 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:32:34.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.074 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.075 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.075 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:32:34.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000550652 2021-03-15T22:32:34.075 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.075 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.076 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.076 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279819+0000", 2021-03-15T22:32:34.076 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8980000000000004e-06 2021-03-15T22:32:34.076 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.076 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.077 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.077 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279824+0000", 2021-03-15T22:32:34.077 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1767999999999999e-05 2021-03-15T22:32:34.077 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.078 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.078 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.078 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279846+0000", 2021-03-15T22:32:34.078 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2509999999999994e-06 2021-03-15T22:32:34.078 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.079 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.079 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.079 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279854+0000", 2021-03-15T22:32:34.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7908000000000001e-05 2021-03-15T22:32:34.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.080 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.080 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279922+0000", 2021-03-15T22:32:34.080 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2315e-05 2021-03-15T22:32:34.081 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.081 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:34.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279934+0000", 2021-03-15T22:32:34.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84156851799999999 2021-03-15T22:32:34.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.082 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121503+0000", 2021-03-15T22:32:34.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012479099999999999 2021-03-15T22:32:34.083 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.083 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121628+0000", 2021-03-15T22:32:34.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000208667 2021-03-15T22:32:34.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.084 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:34.085 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121836+0000", 2021-03-15T22:32:34.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023170439999999999 2021-03-15T22:32:34.085 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.085 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:34.086 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124153+0000", 2021-03-15T22:32:34.086 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012293359999999999 2021-03-15T22:32:34.086 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.086 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.087 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125383+0000", 2021-03-15T22:32:34.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00099435699999999997 2021-03-15T22:32:34.087 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.087 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.088 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.088 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126377+0000", 2021-03-15T22:32:34.088 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3995999999999997e-05 2021-03-15T22:32:34.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.089 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.089 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.089 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126421+0000", 2021-03-15T22:32:34.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8797999999999997e-05 2021-03-15T22:32:34.089 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.090 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.090 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.090 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126460+0000", 2021-03-15T22:32:34.090 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84719129900000001 2021-03-15T22:32:34.090 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.091 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.091 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.092 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3971 7.7 7:ecc0cb20:::smithi07694878-25:head [write 2083358~718180] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:32:34.092 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:32:34.092 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 418.65243929299999, 2021-03-15T22:32:34.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.840294447, 2021-03-15T22:32:34.092 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.093 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:34.093 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.093 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.093 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.094 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3971 2021-03-15T22:32:34.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.094 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.095 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:32:34.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.095 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.095 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.096 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:32:34.096 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00065142800000000001 2021-03-15T22:32:34.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.096 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.097 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.097 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287886+0000", 2021-03-15T22:32:34.097 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.418e-06 2021-03-15T22:32:34.097 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.098 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.098 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287888+0000", 2021-03-15T22:32:34.098 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2062000000000001e-05 2021-03-15T22:32:34.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.099 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.099 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.099 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287900+0000", 2021-03-15T22:32:34.099 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7250000000000002e-06 2021-03-15T22:32:34.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.100 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.100 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287906+0000", 2021-03-15T22:32:34.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9849000000000003e-05 2021-03-15T22:32:34.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.101 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.101 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287956+0000", 2021-03-15T22:32:34.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6460000000000005e-06 2021-03-15T22:32:34.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.102 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.102 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:34.102 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287965+0000", 2021-03-15T22:32:34.102 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83445325400000003 2021-03-15T22:32:34.103 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122419+0000", 2021-03-15T22:32:34.103 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7459e-05 2021-03-15T22:32:34.104 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.104 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.104 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122476+0000", 2021-03-15T22:32:34.105 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012662400000000001 2021-03-15T22:32:34.105 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.105 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.105 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:34.105 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122603+0000", 2021-03-15T22:32:34.106 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037072580000000002 2021-03-15T22:32:34.106 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.106 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.106 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.106 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126310+0000", 2021-03-15T22:32:34.107 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000275853 2021-03-15T22:32:34.107 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.107 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.107 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:34.108 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126586+0000", 2021-03-15T22:32:34.108 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00087237800000000004 2021-03-15T22:32:34.108 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.108 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.108 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.109 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127458+0000", 2021-03-15T22:32:34.109 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5484000000000001e-05 2021-03-15T22:32:34.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.109 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.110 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127494+0000", 2021-03-15T22:32:34.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5009000000000003e-05 2021-03-15T22:32:34.110 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.111 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.111 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127529+0000", 2021-03-15T22:32:34.111 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.840294447 2021-03-15T22:32:34.111 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.111 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.112 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.112 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3972 7.7 7:ecc0cb20:::smithi07694878-25:head [setxattr _header (58) in=7b,truncate 3109394] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:32:34.112 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:32:34.113 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 418.65175733500001, 2021-03-15T22:32:34.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84111847799999995, 2021-03-15T22:32:34.113 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.113 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:34.114 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.114 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.114 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.114 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3972 2021-03-15T22:32:34.114 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.115 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.115 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.115 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.115 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:32:34.116 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.116 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.116 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.116 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.116 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:32:34.117 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9489999999999999e-06 2021-03-15T22:32:34.117 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.117 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.117 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.118 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287918+0000", 2021-03-15T22:32:34.118 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.359e-06 2021-03-15T22:32:34.118 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.118 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.118 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.119 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287920+0000", 2021-03-15T22:32:34.119 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8050000000000003e-06 2021-03-15T22:32:34.119 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.119 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.120 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.120 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287925+0000", 2021-03-15T22:32:34.120 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5959999999999997e-06 2021-03-15T22:32:34.120 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.120 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.121 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.121 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287930+0000", 2021-03-15T22:32:34.121 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2366999999999993e-05 2021-03-15T22:32:34.121 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.122 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.122 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.122 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288002+0000", 2021-03-15T22:32:34.122 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2240000000000001e-06 2021-03-15T22:32:34.122 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.123 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.123 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:34.123 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288012+0000", 2021-03-15T22:32:34.123 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83555615299999997 2021-03-15T22:32:34.123 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.124 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.124 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123568+0000", 2021-03-15T22:32:34.124 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4648999999999995e-05 2021-03-15T22:32:34.125 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.125 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.125 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.125 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123662+0000", 2021-03-15T22:32:34.125 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020287300000000001 2021-03-15T22:32:34.126 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.126 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.126 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:34.126 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123865+0000", 2021-03-15T22:32:34.126 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002471843 2021-03-15T22:32:34.127 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.127 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.127 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.127 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126337+0000", 2021-03-15T22:32:34.128 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025485200000000002 2021-03-15T22:32:34.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.128 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.128 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:34.128 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126592+0000", 2021-03-15T22:32:34.129 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002302077 2021-03-15T22:32:34.129 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.129 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.129 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.129 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128894+0000", 2021-03-15T22:32:34.130 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6276999999999998e-05 2021-03-15T22:32:34.130 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.130 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.130 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128930+0000", 2021-03-15T22:32:34.131 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104454 2021-03-15T22:32:34.131 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.131 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.132 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129035+0000", 2021-03-15T22:32:34.132 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84111847799999995 2021-03-15T22:32:34.132 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.132 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.132 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.133 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.133 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3973 7.7 7:ecc0cb20:::smithi07694878-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e420)", 2021-03-15T22:32:34.133 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:32:34.134 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 418.65172606499999, 2021-03-15T22:32:34.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.841325871, 2021-03-15T22:32:34.134 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.134 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:34.134 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.135 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.135 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.135 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3973 2021-03-15T22:32:34.135 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.135 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.136 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.136 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.136 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:32:34.136 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.136 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.137 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.137 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.137 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:32:34.137 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0400000000000005e-07 2021-03-15T22:32:34.138 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.138 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.138 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.138 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:32:34.138 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3090000000000001e-06 2021-03-15T22:32:34.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.139 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287950+0000", 2021-03-15T22:32:34.139 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1809999999999998e-06 2021-03-15T22:32:34.140 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.140 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.140 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287956+0000", 2021-03-15T22:32:34.140 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9060000000000004e-06 2021-03-15T22:32:34.141 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.141 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.141 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287960+0000", 2021-03-15T22:32:34.142 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8666999999999997e-05 2021-03-15T22:32:34.142 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.142 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.142 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.142 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288039+0000", 2021-03-15T22:32:34.143 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7850000000000003e-06 2021-03-15T22:32:34.143 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.143 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.143 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:34.143 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288045+0000", 2021-03-15T22:32:34.144 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83615280199999997 2021-03-15T22:32:34.144 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.144 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.144 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.145 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124198+0000", 2021-03-15T22:32:34.145 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2801999999999998e-05 2021-03-15T22:32:34.145 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.145 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.145 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:34.146 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124281+0000", 2021-03-15T22:32:34.146 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047980210000000004 2021-03-15T22:32:34.146 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.146 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.146 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.147 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129079+0000", 2021-03-15T22:32:34.147 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2120000000000007e-05 2021-03-15T22:32:34.147 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.147 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.147 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.148 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129161+0000", 2021-03-15T22:32:34.148 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011237400000000001 2021-03-15T22:32:34.148 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.148 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.149 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.149 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129273+0000", 2021-03-15T22:32:34.149 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.841325871 2021-03-15T22:32:34.149 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.149 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.150 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.150 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.150 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.150 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4287 7.7 7:ed4a01d2:::smithi07694878-2:head [rollback 128] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:32:34.151 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.162901+0000", 2021-03-15T22:32:34.151 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 393.77677252900003, 2021-03-15T22:32:34.151 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068606072000000004, 2021-03-15T22:32:34.151 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.151 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:34.152 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.152 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.152 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.152 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4287 2021-03-15T22:32:34.152 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.153 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.153 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.153 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.153 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162901+0000", 2021-03-15T22:32:34.154 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.154 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.154 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.154 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.154 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162901+0000", 2021-03-15T22:32:34.155 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9990000000000002e-06 2021-03-15T22:32:34.155 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.155 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.155 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.155 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162905+0000", 2021-03-15T22:32:34.156 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8239999999999999e-06 2021-03-15T22:32:34.156 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.156 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.156 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.157 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162910+0000", 2021-03-15T22:32:34.157 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5261000000000001e-05 2021-03-15T22:32:34.157 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.157 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.157 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.158 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162935+0000", 2021-03-15T22:32:34.158 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6001000000000001e-05 2021-03-15T22:32:34.158 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.158 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.158 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.159 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162961+0000", 2021-03-15T22:32:34.159 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.522e-05 2021-03-15T22:32:34.159 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.159 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.159 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.160 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163056+0000", 2021-03-15T22:32:34.160 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000127291 2021-03-15T22:32:34.160 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.160 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.161 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.161 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163184+0000", 2021-03-15T22:32:34.161 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030452000000000002 2021-03-15T22:32:34.161 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.161 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.162 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:34.162 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163488+0000", 2021-03-15T22:32:34.162 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026637311 2021-03-15T22:32:34.162 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.162 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.163 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:34.163 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.190125+0000", 2021-03-15T22:32:34.163 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021491026999999999 2021-03-15T22:32:34.163 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.163 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.164 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.164 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211617+0000", 2021-03-15T22:32:34.164 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019763414 2021-03-15T22:32:34.164 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.165 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.165 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.165 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231380+0000", 2021-03-15T22:32:34.165 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2330000000000005e-05 2021-03-15T22:32:34.165 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.166 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.166 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.166 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231472+0000", 2021-03-15T22:32:34.166 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4873999999999998e-05 2021-03-15T22:32:34.166 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.167 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.167 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.167 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231507+0000", 2021-03-15T22:32:34.167 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068606072000000004 2021-03-15T22:32:34.168 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.168 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.168 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.168 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.168 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.169 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4288 7.7 7:ed4a01d2:::smithi07694878-2:head [append 0~0] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:32:34.169 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.162974+0000", 2021-03-15T22:32:34.169 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 393.77669931899999, 2021-03-15T22:32:34.169 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068646398999999997, 2021-03-15T22:32:34.169 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.170 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:34.170 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.170 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.170 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.170 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4288 2021-03-15T22:32:34.171 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.171 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.171 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.171 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.172 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162974+0000", 2021-03-15T22:32:34.172 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.172 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.172 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.172 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.173 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162974+0000", 2021-03-15T22:32:34.173 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6800000000000009e-07 2021-03-15T22:32:34.173 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.173 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.173 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.174 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162975+0000", 2021-03-15T22:32:34.174 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0729999999999999e-06 2021-03-15T22:32:34.174 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.174 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.174 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.175 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162976+0000", 2021-03-15T22:32:34.175 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5429999999999998e-06 2021-03-15T22:32:34.175 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.175 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.176 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.176 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162980+0000", 2021-03-15T22:32:34.176 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4399999999999999e-06 2021-03-15T22:32:34.176 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.176 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.177 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.177 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162982+0000", 2021-03-15T22:32:34.177 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011790209999999999 2021-03-15T22:32:34.177 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.177 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.178 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.178 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164161+0000", 2021-03-15T22:32:34.178 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104054 2021-03-15T22:32:34.178 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.178 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.179 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.179 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164265+0000", 2021-03-15T22:32:34.179 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019185799999999999 2021-03-15T22:32:34.179 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.180 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.180 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:34.180 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164457+0000", 2021-03-15T22:32:34.180 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025674285000000002 2021-03-15T22:32:34.180 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.181 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.181 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:34.181 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.190132+0000", 2021-03-15T22:32:34.181 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021520336000000001 2021-03-15T22:32:34.181 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.182 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.182 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.182 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211652+0000", 2021-03-15T22:32:34.182 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019892508999999999 2021-03-15T22:32:34.182 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.183 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.183 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.183 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231544+0000", 2021-03-15T22:32:34.183 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0618000000000003e-05 2021-03-15T22:32:34.184 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.184 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.184 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.184 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231585+0000", 2021-03-15T22:32:34.184 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5694000000000001e-05 2021-03-15T22:32:34.185 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.185 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.185 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.185 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231621+0000", 2021-03-15T22:32:34.185 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068646398999999997 2021-03-15T22:32:34.186 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.186 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.186 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.186 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.186 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.187 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4290 7.d 7:bd183e1b:::smithi07694878-46:head [stat,delete] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:32:34.187 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.163011+0000", 2021-03-15T22:32:34.187 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 393.77666232299998, 2021-03-15T22:32:34.187 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048696824999999999, 2021-03-15T22:32:34.188 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.188 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:34.188 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.188 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.188 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.189 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4290 2021-03-15T22:32:34.189 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.189 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.189 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.189 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.190 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163011+0000", 2021-03-15T22:32:34.190 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.190 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.190 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.191 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.191 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163011+0000", 2021-03-15T22:32:34.191 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.079e-06 2021-03-15T22:32:34.191 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.191 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.192 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.192 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163012+0000", 2021-03-15T22:32:34.192 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.043e-06 2021-03-15T22:32:34.192 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.192 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.193 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.193 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163013+0000", 2021-03-15T22:32:34.193 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7620000000000003e-06 2021-03-15T22:32:34.193 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.194 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.194 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.194 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163018+0000", 2021-03-15T22:32:34.194 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4980000000000003e-06 2021-03-15T22:32:34.194 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.195 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.195 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.195 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163025+0000", 2021-03-15T22:32:34.195 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6739999999999997e-05 2021-03-15T22:32:34.195 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.196 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.196 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.196 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163061+0000", 2021-03-15T22:32:34.196 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9894000000000003e-05 2021-03-15T22:32:34.196 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.197 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.197 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.197 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163161+0000", 2021-03-15T22:32:34.197 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017699600000000001 2021-03-15T22:32:34.198 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.198 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.198 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:34.198 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163338+0000", 2021-03-15T22:32:34.198 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001543619 2021-03-15T22:32:34.199 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.199 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.199 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:34.199 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164882+0000", 2021-03-15T22:32:34.199 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021337467999999998 2021-03-15T22:32:34.200 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.200 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.200 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.200 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.186219+0000", 2021-03-15T22:32:34.201 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025381363000000001 2021-03-15T22:32:34.201 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.201 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.201 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.201 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211601+0000", 2021-03-15T22:32:34.202 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9228999999999998e-05 2021-03-15T22:32:34.202 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.202 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.202 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.202 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211660+0000", 2021-03-15T22:32:34.203 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8133999999999997e-05 2021-03-15T22:32:34.203 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.203 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.203 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.204 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211708+0000", 2021-03-15T22:32:34.204 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048696824999999999 2021-03-15T22:32:34.204 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.204 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.204 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.205 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.205 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.205 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:4284 7.b e445/401 7:d4368cae:::smithi07694878-18:head v 445'396, mlcod=445'394)", 2021-03-15T22:32:34.205 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.164429+0000", 2021-03-15T22:32:34.205 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 393.77524410299998, 2021-03-15T22:32:34.206 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048217024999999997, 2021-03-15T22:32:34.206 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.206 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:34.206 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.206 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.207 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.207 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164429+0000", 2021-03-15T22:32:34.207 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.207 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.208 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.208 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.208 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164429+0000", 2021-03-15T22:32:34.208 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00035440200000000003 2021-03-15T22:32:34.208 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.209 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.209 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.209 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164784+0000", 2021-03-15T22:32:34.209 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.659e-06 2021-03-15T22:32:34.209 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.210 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.210 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.210 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164788+0000", 2021-03-15T22:32:34.210 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8576999999999998e-05 2021-03-15T22:32:34.210 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.211 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.211 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.211 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164806+0000", 2021-03-15T22:32:34.211 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0139999999999998e-06 2021-03-15T22:32:34.212 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.212 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.212 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.212 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164814+0000", 2021-03-15T22:32:34.212 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7796000000000001e-05 2021-03-15T22:32:34.213 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.213 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.213 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.213 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164862+0000", 2021-03-15T22:32:34.213 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2391e-05 2021-03-15T22:32:34.214 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.214 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.214 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.214 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164884+0000", 2021-03-15T22:32:34.215 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047722025000000001 2021-03-15T22:32:34.215 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.215 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.215 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.215 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.212606+0000", 2021-03-15T22:32:34.216 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0160999999999997e-05 2021-03-15T22:32:34.216 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.216 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.216 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.216 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.212647+0000", 2021-03-15T22:32:34.217 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048217024999999997 2021-03-15T22:32:34.217 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.217 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.217 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.217 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.218 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.218 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:4285 7.b e445/401 7:d4368cae:::smithi07694878-18:head v 445'397, mlcod=445'394)", 2021-03-15T22:32:34.218 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.165543+0000", 2021-03-15T22:32:34.218 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 393.77413016700001, 2021-03-15T22:32:34.219 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047128319000000002, 2021-03-15T22:32:34.219 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.219 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:34.219 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.219 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.220 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.220 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165543+0000", 2021-03-15T22:32:34.220 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.220 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.220 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.221 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.221 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165543+0000", 2021-03-15T22:32:34.221 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.305e-06 2021-03-15T22:32:34.221 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.222 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.222 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.222 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165546+0000", 2021-03-15T22:32:34.222 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9570000000000001e-06 2021-03-15T22:32:34.222 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.223 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.223 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.223 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165548+0000", 2021-03-15T22:32:34.223 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4930000000000003e-06 2021-03-15T22:32:34.224 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.224 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.224 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.224 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165554+0000", 2021-03-15T22:32:34.224 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5200000000000002e-06 2021-03-15T22:32:34.225 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.225 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.225 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.225 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165558+0000", 2021-03-15T22:32:34.225 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0366999999999999e-05 2021-03-15T22:32:34.226 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.226 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.226 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.226 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165588+0000", 2021-03-15T22:32:34.227 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3482999999999999e-05 2021-03-15T22:32:34.227 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.227 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.228 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.228 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.165602+0000", 2021-03-15T22:32:34.228 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047050275000000003 2021-03-15T22:32:34.228 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.229 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.229 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.229 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.212652+0000", 2021-03-15T22:32:34.229 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9919e-05 2021-03-15T22:32:34.230 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.230 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.230 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.230 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.212672+0000", 2021-03-15T22:32:34.230 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047128319000000002 2021-03-15T22:32:34.231 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.231 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.231 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.231 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.232 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.232 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6113 7.7 7:ed4a01d2:::smithi07694878-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e658)", 2021-03-15T22:32:34.232 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:30:07.405280+0000", 2021-03-15T22:32:34.232 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 146.53439388300001, 2021-03-15T22:32:34.232 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026961246000000001, 2021-03-15T22:32:34.233 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.233 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:34.233 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.233 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.234 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.234 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6113 2021-03-15T22:32:34.234 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.234 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.234 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.235 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.235 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.405280+0000", 2021-03-15T22:32:34.235 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.235 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.235 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.236 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.236 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.405280+0000", 2021-03-15T22:32:34.236 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.023e-06 2021-03-15T22:32:34.236 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.237 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.237 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.237 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.405281+0000", 2021-03-15T22:32:34.237 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1739999999999999e-06 2021-03-15T22:32:34.237 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.238 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.238 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.238 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.405282+0000", 2021-03-15T22:32:34.238 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2389999999999996e-06 2021-03-15T22:32:34.238 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.239 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.239 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.239 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.405289+0000", 2021-03-15T22:32:34.239 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2660000000000003e-06 2021-03-15T22:32:34.240 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.240 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.240 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.240 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.405293+0000", 2021-03-15T22:32:34.240 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018958199999999999 2021-03-15T22:32:34.241 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.241 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.241 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.405483+0000", 2021-03-15T22:32:34.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6331999999999997e-05 2021-03-15T22:32:34.242 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.242 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.242 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:32:34.242 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.405519+0000", 2021-03-15T22:32:34.243 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010336636 2021-03-15T22:32:34.243 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.243 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.243 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.243 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.415856+0000", 2021-03-15T22:32:34.244 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0116000000000003e-05 2021-03-15T22:32:34.244 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.244 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.244 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:34.244 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.415936+0000", 2021-03-15T22:32:34.245 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.016166621999999999 2021-03-15T22:32:34.245 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.245 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.245 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.246 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.432103+0000", 2021-03-15T22:32:34.246 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0446999999999999e-05 2021-03-15T22:32:34.246 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.246 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.246 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.247 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.432163+0000", 2021-03-15T22:32:34.247 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7809e-05 2021-03-15T22:32:34.247 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.247 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.247 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.248 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:07.432241+0000", 2021-03-15T22:32:34.248 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026961246000000001 2021-03-15T22:32:34.248 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.248 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.249 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.249 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.249 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.249 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6163 7.7 7:ed4a01d2:::smithi07694878-2:head [copy-from ver 1586] snapc 20e=[20e,20c,20a,209,207,205,200,1fb,1f8,1f7,1f6,1f4,1e9,1e7,1e6,1e4,1e0,1dc,1d5,1c7,1c5,1a1,17a,176] ondisk+write+known_if_redirected e662)", 2021-03-15T22:32:34.249 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:30:11.243588+0000", 2021-03-15T22:32:34.250 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 142.696086028, 2021-03-15T22:32:34.250 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028673536999999999, 2021-03-15T22:32:34.250 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.250 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:34.251 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.251 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.251 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.251 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6163 2021-03-15T22:32:34.251 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.252 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.252 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.252 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.252 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243588+0000", 2021-03-15T22:32:34.252 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.253 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.253 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.253 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.253 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243588+0000", 2021-03-15T22:32:34.254 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.73e-06 2021-03-15T22:32:34.254 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.254 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.254 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.254 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243589+0000", 2021-03-15T22:32:34.255 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3269999999999999e-06 2021-03-15T22:32:34.255 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.255 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.255 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.255 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243592+0000", 2021-03-15T22:32:34.256 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0662999999999999e-05 2021-03-15T22:32:34.256 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.256 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.256 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.257 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243602+0000", 2021-03-15T22:32:34.257 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6159999999999998e-06 2021-03-15T22:32:34.257 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.257 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.257 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.258 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243609+0000", 2021-03-15T22:32:34.258 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9159000000000001e-05 2021-03-15T22:32:34.258 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.258 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.258 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.259 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243658+0000", 2021-03-15T22:32:34.259 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010882099999999999 2021-03-15T22:32:34.259 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.259 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.260 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.260 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243767+0000", 2021-03-15T22:32:34.260 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.014486250000000001 2021-03-15T22:32:34.260 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.260 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.261 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-15T22:32:34.261 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.258253+0000", 2021-03-15T22:32:34.261 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0064858609999999999 2021-03-15T22:32:34.261 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.261 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.262 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:34.262 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.264739+0000", 2021-03-15T22:32:34.262 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0025981720000000002 2021-03-15T22:32:34.262 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.263 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.263 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.263 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.267337+0000", 2021-03-15T22:32:34.263 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0048257930000000001 2021-03-15T22:32:34.263 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.264 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.264 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:34.264 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.272163+0000", 2021-03-15T22:32:34.264 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8356999999999999e-05 2021-03-15T22:32:34.264 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.265 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.265 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:34.265 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.272201+0000", 2021-03-15T22:32:34.265 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9787999999999997e-05 2021-03-15T22:32:34.265 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.266 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.266 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.266 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.272261+0000", 2021-03-15T22:32:34.266 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028673536999999999 2021-03-15T22:32:34.267 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.267 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.267 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.267 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.267 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.268 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6164 7.7 7:ed4a01d2:::smithi07694878-2:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e662)", 2021-03-15T22:32:34.268 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:30:11.243626+0000", 2021-03-15T22:32:34.268 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 142.69604749199999, 2021-03-15T22:32:34.268 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028802066000000001, 2021-03-15T22:32:34.268 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:34.269 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:34.269 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:34.269 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:34.269 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:34.270 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6164 2021-03-15T22:32:34.270 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.270 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:34.270 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.270 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:34.271 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243626+0000", 2021-03-15T22:32:34.271 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:34.271 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.271 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.272 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:34.272 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243626+0000", 2021-03-15T22:32:34.272 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2759999999999999e-06 2021-03-15T22:32:34.272 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.272 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.273 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:34.273 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243627+0000", 2021-03-15T22:32:34.273 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7099999999999999e-06 2021-03-15T22:32:34.273 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.273 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.274 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:34.274 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243629+0000", 2021-03-15T22:32:34.274 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0320000000000003e-06 2021-03-15T22:32:34.274 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.275 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.275 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:34.275 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243637+0000", 2021-03-15T22:32:34.275 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7210000000000002e-06 2021-03-15T22:32:34.275 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.276 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.276 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:34.276 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243642+0000", 2021-03-15T22:32:34.276 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000284257 2021-03-15T22:32:34.276 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.277 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.277 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.277 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243926+0000", 2021-03-15T22:32:34.277 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1837e-05 2021-03-15T22:32:34.278 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.278 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.278 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:32:34.278 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.243958+0000", 2021-03-15T22:32:34.278 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018252641999999999 2021-03-15T22:32:34.279 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.279 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.279 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.279 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.262211+0000", 2021-03-15T22:32:34.279 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1899999999999999e-05 2021-03-15T22:32:34.280 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.280 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.280 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:34.280 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.262282+0000", 2021-03-15T22:32:34.281 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010014133999999999 2021-03-15T22:32:34.281 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.281 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.281 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:34.281 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.272297+0000", 2021-03-15T22:32:34.282 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5149999999999999e-05 2021-03-15T22:32:34.282 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.282 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.282 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:34.282 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.272352+0000", 2021-03-15T22:32:34.283 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6407000000000002e-05 2021-03-15T22:32:34.283 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:34.283 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:34.283 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:34.283 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:30:11.272428+0000", 2021-03-15T22:32:34.284 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028802066000000001 2021-03-15T22:32:34.284 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.284 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.284 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.285 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:34.285 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:34.285 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:32:34.285 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:34.287 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:34.309 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:34.310 INFO:tasks.rados.rados.0.smithi076.stdout:3011: done (2 left) 2021-03-15T22:32:34.311 INFO:tasks.rados.rados.0.smithi076.stdout:3012: done (1 left) 2021-03-15T22:32:34.311 INFO:tasks.rados.rados.0.smithi076.stdout:3013: done (0 left) 2021-03-15T22:32:34.311 INFO:tasks.rados.rados.0.smithi076.stdout:3014: copy_from oid 4 from oid 20 current snap is 348 2021-03-15T22:32:34.311 INFO:tasks.rados.rados.0.smithi076.stdout:3015: snap_remove snap 292 2021-03-15T22:32:34.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:34.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:34.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:34.435 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:34.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:34.539 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:34.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:34.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[56514]: cluster 2021-03-15T22:32:32.744707+0000 osd.1 (osd.1) 157 : cluster [DBG] 5.1d scrub ok 2021-03-15T22:32:34.562 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[56514]: cluster 2021-03-15T22:32:33.051958+0000 osd.2 (osd.2) 27 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:32:34.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[56514]: cluster 2021-03-15T22:32:33.177271+0000 mgr.y (mgr.14144) 939 : cluster [DBG] pgmap v1348: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 521 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2021-03-15T22:32:34.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[56514]: audit 2021-03-15T22:32:33.290182+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-15T22:32:34.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[56514]: cluster 2021-03-15T22:32:33.290296+0000 mon.a (mon.0) 2086 : cluster [DBG] osdmap e803: 8 total, 8 up, 7 in 2021-03-15T22:32:34.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[56514]: cluster 2021-03-15T22:32:33.338096+0000 osd.4 (osd 2021-03-15T22:32:34.563 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[56514]: .4) 170 : cluster [DBG] 5.18 scrub ok 2021-03-15T22:32:34.564 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[56514]: cluster 2021-03-15T22:32:33.438738+0000 osd.7 (osd.7) 149 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:32:34.564 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[56514]: cluster 2021-03-15T22:32:33.701671+0000 osd.0 (osd.0) 130 : cluster [DBG] 4.b scrub ok 2021-03-15T22:32:34.564 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: cluster 2021-03-15T22:32:32.744707+0000 2021-03-15T22:32:34.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: osd.1 (osd.1) 157 : cluster [DBG] 5.1d scrub ok 2021-03-15T22:32:34.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:32:34.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: :32:33.051958+0000 osd.2 (osd.2) 27 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:32:34.565 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:32:34.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: 32:33.177271+0000 mgr.y (mgr.14144) 939 : cluster [DBG] pgmap v1348: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 521 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2021-03-15T22:32:34.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:32:34.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: 03-15T22:32:33. 2021-03-15T22:32:34.566 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: 290182+0000 mon.a (mon.0) 2021-03-15T22:32:34.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: 2085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-15T22:32:34.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:32:34.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: -03-15T22:32:33 2021-03-15T22:32:34.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: .290296+0000 mon.a (mon. 2021-03-15T22:32:34.567 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: 0) 2086 : cluster [DBG] osdmap e803: 8 total, 8 up, 7 in 2021-03-15T22:32:34.568 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: 2021-03-15T22:32:34.568 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:32:34.568 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: -15T22:32:33. 2021-03-15T22:32:34.568 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: 338096+0000 osd.4 (osd.4 2021-03-15T22:32:34.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: ) 170 : cluster [DBG] 5.18 scrub ok 2021-03-15T22:32:34.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:32:34.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: -03-15T22:32:33 2021-03-15T22:32:34.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: .438738+0000 osd.7 (osd. 2021-03-15T22:32:34.569 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: 7) 149 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:32:34.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:32:34.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: -03-15T22:32: 2021-03-15T22:32:34.570 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: 33.701671+0000 osd.0 (osd 2021-03-15T22:32:34.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: .0) 130 : cluster [DBG] 2021-03-15T22:32:34.571 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:34 smithi076 conmon[62156]: 4.b scrub ok 2021-03-15T22:32:34.572 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:34.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:34.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: cluster 2021-03-15T22:32:32.744707 2021-03-15T22:32:34.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: +0000 osd.1 (osd.1) 157 : cluster [DBG] 5.1d scrub ok 2021-03-15T22:32:34.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: cluster 2021-03-15T22:32:33.051958+0000 osd.2 (osd.2) 27 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:32:34.573 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: cluster 2021-03-15T22:32:33.177271+0000 mgr.y (mgr.14144) 939 : cluster [DBG] pgmap v1348: 148 pgs: 1 active+remapped+backfill_toofull, 147 active+clean; 521 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 3.1 MiB/s wr, 24 op/s 2021-03-15T22:32:34.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: audit 2021-03-15T22:32:33.290182+0000 mon.a (mon.0) 2085 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2021-03-15T22:32:34.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: cluster 2021-03-15T22:32:33.290296+0000 2021-03-15T22:32:34.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: mon.a (mon.0) 2086 : cluster [DBG] osdmap e803: 8 total, 8 up, 7 in 2021-03-15T22:32:34.574 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:32:34.575 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: -15T22:32:33 2021-03-15T22:32:34.575 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: .338096+0000 osd.4 (osd. 2021-03-15T22:32:34.575 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: 4) 170 : cluster [DBG] 2021-03-15T22:32:34.575 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: 5.18 scrub ok 2021-03-15T22:32:34.575 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: cluster 2021-03-15T22:32:34.576 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: 2021-03-15T22 2021-03-15T22:32:34.576 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: :32:33.438738+0000 2021-03-15T22:32:34.576 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: osd.7 (osd.7) 149 2021-03-15T22:32:34.576 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: : cluster [DBG] 3.15 scrub ok 2021-03-15T22:32:34.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:32:34.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: -03-15T22:32: 2021-03-15T22:32:34.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: 33.701671+0000 osd.0 ( 2021-03-15T22:32:34.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: osd.0) 130 : cluster 2021-03-15T22:32:34.577 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:34 smithi125 conmon[56739]: [DBG] 4.b scrub ok 2021-03-15T22:32:34.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-15T22:32:34.674 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:34.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:34.699 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:34.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:34.801 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:34.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:34.826 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:34.826 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:34.906 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:34.928 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:34.950 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:34.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:34.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:35.053 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:35.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:35.077 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:35.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:35.179 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:35.201 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:35.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:35.203 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:35.265 INFO:tasks.rados.rados.0.smithi076.stdout:3015: done (1 left) 2021-03-15T22:32:35.265 INFO:tasks.rados.rados.0.smithi076.stdout:3016: snap_create 2021-03-15T22:32:35.266 INFO:tasks.rados.rados.0.smithi076.stdout:3014: finishing copy_from to smithi07694878-4 2021-03-15T22:32:35.266 INFO:tasks.rados.rados.0.smithi076.stdout:3014: got expected ENOENT (src dne) 2021-03-15T22:32:35.266 INFO:tasks.rados.rados.0.smithi076.stdout:3014: finishing copy_from racing read to smithi07694878-4 2021-03-15T22:32:35.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:35.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:35.328 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:35.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:35.342 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:32:35.381 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:35 smithi125 conmon[56739]: cluster 2021-03-15T22:32:33.883945+0000 osd.5 (osd.5) 33 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:32:35.381 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:35 smithi125 conmon[56739]: cluster 2021-03-15T22:32:34.095151+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:32:35.381 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:35 smithi125 conmon[56739]: cluster 2021-03-15T22:32:34.303910+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2021-03-15T22:32:35.382 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:35 smithi125 conmon[56739]: cluster 2021-03-15T22:32:34.739632+0000 osd.0 (osd.0) 131 : cluster [DBG] 5.1e scrub ok 2021-03-15T22:32:35.382 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:35 smithi125 conmon[56739]: cluster 2021-03-15T22:32:35.261548+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e805: 8 total, 8 up, 7 in 2021-03-15T22:32:35.430 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:35.451 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:35.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:35.454 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:35.556 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:35.578 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:35.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:35.579 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:35.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:35.703 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:35.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:35.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:35.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[56514]: cluster 2021-03-15T22:32:33.883945+0000 osd.5 (osd.5) 33 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:32:35.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[56514]: cluster 2021-03-15T22:32:34.095151+0000 osd.2 (osd.2) 28 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:32:35.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[56514]: cluster 2021-03-15T22:32:34.303910+0000 mon.a (mon.0) 2087 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2021-03-15T22:32:35.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[56514]: cluster 2021-03-15T22:32:34.739632+0000 osd.0 (osd.0) 131 : cluster [DBG] 5.1e scrub ok 2021-03-15T22:32:35.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[56514]: cluster 2021-03-15T22:32:35.261548+0000 mon.a (mon.0) 2088 : cluster [DBG] osdmap e805: 8 total, 8 up, 7 in 2021-03-15T22:32:35.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: cluster 2021-03-15T22:32:33.883945+0000 osd.5 (osd.5) 33 : cluster 2021-03-15T22:32:35.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: [DBG] 3.5 scrub ok 2021-03-15T22:32:35.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: cluster 2021-03-15T22:32:35.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: 2021-03-15T22:32:34.095151+0000 osd.2 (osd.2) 28 2021-03-15T22:32:35.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: : cluster [DBG] 2.1 scrub ok 2021-03-15T22:32:35.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:32:35.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: 32:34.303910+0000 mon.a (mon.0) 2021-03-15T22:32:35.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: 2087 : cluster [DBG] osdmap e804: 8 total, 8 up, 7 in 2021-03-15T22:32:35.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:32:35.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: 15T22:32:34.739632+0000 osd.0 ( 2021-03-15T22:32:35.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: osd.0) 131 : cluster [DBG] 5.1e scrub ok 2021-03-15T22:32:35.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:32:35.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: -03-15T22:32:35.261548 2021-03-15T22:32:35.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2088 : cluster [DBG] 2021-03-15T22:32:35.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:35 smithi076 conmon[62156]: osdmap e805: 8 total, 8 up, 7 in 2021-03-15T22:32:35.805 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:35.828 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:35.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:35.830 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:35.931 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:35.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:35.954 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:35.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:36.055 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:36.075 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:36.079 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:36.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:36.080 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:36.181 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:36.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:36.205 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:36.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:36.275 INFO:tasks.rados.rados.0.smithi076.stdout:3014: done (1 left) 2021-03-15T22:32:36.275 INFO:tasks.rados.rados.0.smithi076.stdout:3016: done (0 left) 2021-03-15T22:32:36.276 INFO:tasks.rados.rados.0.smithi076.stdout:3017: read oid 3 snap 343 2021-03-15T22:32:36.276 INFO:tasks.rados.rados.0.smithi076.stdout:3017: expect deleted 2021-03-15T22:32:36.276 INFO:tasks.rados.rados.0.smithi076.stdout:3018: rollback oid 6 current snap is 349 2021-03-15T22:32:36.276 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 347 2021-03-15T22:32:36.276 INFO:tasks.rados.rados.0.smithi076.stdout:3019: write (excl) oid 11 current snap is 349 2021-03-15T22:32:36.277 INFO:tasks.rados.rados.0.smithi076.stdout:3019: seq_num 1041 ranges {696087=558217,1833755=648359,3137806=358281} 2021-03-15T22:32:36.282 INFO:tasks.rados.rados.0.smithi076.stdout:3019: writing smithi07694878-11 from 696087 to 1254304 tid 1 2021-03-15T22:32:36.289 INFO:tasks.rados.rados.0.smithi076.stdout:3019: writing smithi07694878-11 from 1833755 to 2482114 tid 2 2021-03-15T22:32:36.295 INFO:tasks.rados.rados.0.smithi076.stdout:3019: writing smithi07694878-11 from 3137806 to 3496087 tid 3 2021-03-15T22:32:36.296 INFO:tasks.rados.rados.0.smithi076.stdout:3020: snap_create 2021-03-15T22:32:36.296 INFO:tasks.rados.rados.0.smithi076.stdout:3018: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:32:36.296 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 2080 (ObjNum 255155272 snap 21896 seq_num 3) dirty dne 2021-03-15T22:32:36.296 INFO:tasks.rados.rados.0.smithi076.stdout:3019: finishing write tid 1 to smithi07694878-11 2021-03-15T22:32:36.297 INFO:tasks.rados.rados.0.smithi076.stdout:3019: finishing write tid 2 to smithi07694878-11 2021-03-15T22:32:36.298 INFO:tasks.rados.rados.0.smithi076.stdout:3019: finishing write tid 3 to smithi07694878-11 2021-03-15T22:32:36.298 INFO:tasks.rados.rados.0.smithi076.stdout:3019: finishing write tid 4 to smithi07694878-11 2021-03-15T22:32:36.298 INFO:tasks.rados.rados.0.smithi076.stdout:3019: finishing write tid 5 to smithi07694878-11 2021-03-15T22:32:36.298 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1980 (ObjNum 1041 snap 349 seq_num 1041) dirty exists 2021-03-15T22:32:36.299 INFO:tasks.rados.rados.0.smithi076.stdout:3019: left oid 11 (ObjNum 1041 snap 349 seq_num 1041) 2021-03-15T22:32:36.306 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:36.330 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:36.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:36.433 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:36.437 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:32:36.437 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:32:36.437 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:32:36.438 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:32:36.457 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:36.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:36.458 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:36.513 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:36 smithi125 conmon[56739]: cluster 2021-03-15T22:32:35.067552+0000 osd.2 (osd.2 2021-03-15T22:32:36.513 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:36 smithi125 conmon[56739]: ) 29 : cluster [DBG] 4.f scrub ok 2021-03-15T22:32:36.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:36 smithi125 conmon[56739]: cluster 2021-03-15T22:32:35.177731+0000 mgr.y (mgr.14144) 940 : cluster [DBG] pgmap v1351: 148 pgs: 148 active+clean; 517 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.8 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-03-15T22:32:36.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:36 smithi125 conmon[56739]: cluster 2021-03-15T22:32:35.302123+0000 mon.a (mon.0) 2089 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:32:36.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:36 smithi125 conmon[56739]: audit 2021-03-15T22:32:36.091865+0000 mon.a (mon.0) 2090 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:32:36.514 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:36 smithi125 conmon[56739]: cluster 2021-03-15T22:32:36.272829+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e806: 8 total, 8 up, 7 in 2021-03-15T22:32:36.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[56514]: cluster 2021-03-15T22:32:35.067552+0000 osd.2 (osd.2) 29 : cluster 2021-03-15T22:32:36.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[56514]: [DBG] 4.f scrub ok 2021-03-15T22:32:36.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[56514]: cluster 2021-03-15T22:32:35.177731+0000 mgr.y (mgr.14144) 940 : cluster [DBG] pgmap v1351: 148 pgs: 148 active+clean; 517 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.8 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-03-15T22:32:36.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[56514]: cluster 2021-03-15T22:32:35.302123+0000 mon.a (mon.0) 2089 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:32:36.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[56514]: audit 2021-03-15T22:32:36.091865+0000 mon.a (mon.0) 2090 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:32:36.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[56514]: cluster 2021-03-15T22:32:36.272829+0000 mon.a (mon.0) 2091 : cluster [DBG] osdmap e806: 8 total, 8 up, 7 in 2021-03-15T22:32:36.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: cluster 2021-03-15T22:32:35.067552+0000 2021-03-15T22:32:36.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: osd.2 (osd.2) 29 : cluster [DBG] 4.f scrub ok 2021-03-15T22:32:36.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: cluster 2021-03-15T22:32:36.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:32:36.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: 22:32:35.177731+0000 mgr.y (mgr.14144) 940 : cluster [DBG] 2021-03-15T22:32:36.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: pgmap v1351: 148 pgs: 148 active+clean; 517 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.8 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2021-03-15T22:32:36.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: cluster 2021-03-15T22:32: 2021-03-15T22:32:36.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: 35.302123+0000 mon.a (mon.0) 2089 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:32:36.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: audit 2021-03-15T22:32: 2021-03-15T22:32:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: 36.091865+0000 mon.a (mon.0) 2090 : 2021-03-15T22:32:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:32:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: cluster 2021-03-15T22:32:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:32:36.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: 32:36.272829+0000 mon.a (mon.0) 2091 2021-03-15T22:32:36.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:36 smithi076 conmon[62156]: : cluster [DBG] osdmap e806: 8 total, 8 up, 7 in 2021-03-15T22:32:36.560 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:36.582 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:36.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:36.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:36.609 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:32:36.612 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:32:36.685 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:36.713 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:36.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:36.715 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:36.816 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:36.838 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:36.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:36.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:36.941 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:36.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:36.964 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:36.965 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:37.037 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-15T22:32:37.066 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:37.102 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:37.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:37.104 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:37.205 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:37.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:37.228 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:37.229 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:37.278 INFO:tasks.rados.rados.0.smithi076.stdout:3017: done (3 left) 2021-03-15T22:32:37.278 INFO:tasks.rados.rados.0.smithi076.stdout:3018: done (2 left) 2021-03-15T22:32:37.278 INFO:tasks.rados.rados.0.smithi076.stdout:3019: done (1 left) 2021-03-15T22:32:37.279 INFO:tasks.rados.rados.0.smithi076.stdout:3020: done (0 left) 2021-03-15T22:32:37.279 INFO:tasks.rados.rados.0.smithi076.stdout:3021: delete oid 44 current snap is 350 2021-03-15T22:32:37.280 INFO:tasks.rados.rados.0.smithi076.stdout:3021: done (0 left) 2021-03-15T22:32:37.281 INFO:tasks.rados.rados.0.smithi076.stdout:3022: delete oid 27 current snap is 350 2021-03-15T22:32:37.282 INFO:tasks.rados.rados.0.smithi076.stdout:3022: done (0 left) 2021-03-15T22:32:37.282 INFO:tasks.rados.rados.0.smithi076.stdout:3023: copy_from oid 33 from oid 41 current snap is 350 2021-03-15T22:32:37.283 INFO:tasks.rados.rados.0.smithi076.stdout:3024: snap_remove snap 325 2021-03-15T22:32:37.318 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:37.329 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:37.352 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:37.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:37.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:37.455 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:37.476 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:37.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:37.478 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:37.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:37 smithi076 conmon[56514]: audit 2021-03-15T22:32:36.592165+0000 mgr.y (mgr.14144) 941 : audit [DBG] from='client.16413 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:32:37.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:37 smithi076 conmon[56514]: cluster 2021-03-15T22:32:36.795296+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:32:37.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:37 smithi076 conmon[56514]: cluster 2021-03-15T22:32:37.275282+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e807: 8 total, 8 up, 7 in 2021-03-15T22:32:37.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:37 smithi076 conmon[62156]: audit 2021-03-15T22:32:36.592165+0000 mgr.y (mgr.14144) 941 : audit [DBG] from='client.16413 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:32:37.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:37 smithi076 conmon[62156]: cluster 2021-03-15T22:32:36.795296+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:32:37.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:37 smithi076 conmon[62156]: cluster 2021-03-15T22:32:37.275282+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e807: 8 total, 8 up, 7 in 2021-03-15T22:32:37.573 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1353,"stamp":"2021-03-15T22:32:36.270286+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":539418542,"num_objects":562,"num_object_clones":197,"num_object_copies":1686,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":562,"num_whiteouts":15,"num_read":7993,"num_read_kb":5752731,"num_write":4979,"num_write_kb":1548750,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3262,"num_bytes_recovered":1223532973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17756,"ondisk_log_size":17756,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1473732,"kb_used_data":1210316,"kb_used_omap":273,"kb_used_meta":263086,"kb_avail":654742332,"statfs":{"total":671965249536,"available":670456147968,"internally_reserved":0,"allocated":1239363584,"data_stored":1676104354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":269400954},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-7301786,"num_objects":-2,"num_object_clones":0,"num_object_copies":-6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-2,"num_whiteouts":3,"num_read":25,"num_read_kb":25292,"num_write":13,"num_write_kb":1286,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.989491"},"pg_stats":[{"pgid":"5.19","version":"317'7","reported_seq":"160","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.264199+0000","last_change":"2021-03-15T22:32:07.910488+0000","last_active":"2021-03-15T22:32:31.264199+0000","last_peered":"2021-03-15T22:32:31.264199+0000","last_clean":"2021-03-15T22:32:31.264199+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:32:31.264199+0000","last_undegraded":"2021-03-15T22:32:31.264199+0000","last_fullsized":"2021-03-15T22:32:31.264199+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:32:07.910427+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:32:07.910427+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"802","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.261392+0000","last_change":"2021-03-15T22:30:41.431150+0000","last_active":"2021-03-15T22:32:30.261392+0000","last_peered":"2021-03-15T22:32:30.261392+0000","last_clean":"2021-03-15T22:32:30.261392+0000","last_became_active":"2021-03-15T22:29:51.225252+0000","last_became_peered":"2021-03-15T22:29:51.225252+0000","last_unstale":"2021-03-15T22:32:30.261392+0000","last_undegraded":"2021-03-15T22:32:30.261392+0000","last_fullsized":"2021-03-15T22:32:30.261392+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.431052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"308","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.311153+0000","last_change":"2021-03-15T22:32:31.680174+0000","last_active":"2021-03-15T22:32:34.311153+0000","last_peered":"2021-03-15T22:32:34.311153+0000","last_clean":"2021-03-15T22:32:34.311153+0000","last_became_active":"2021-03-15T22:30:32.251610+0000","last_became_peered":"2021-03-15T22:30:32.251610+0000","last_unstale":"2021-03-15T22:32:34.311153+0000","last_undegraded":"2021-03-15T22:32:34.311153+0000","last_fullsized":"2021-03-15T22:32:34.311153+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:32:31.680111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.680111+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"863","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.262806+0000","last_change":"2021-03-15T22:32:12.177096+0000","last_active":"2021-03-15T22:32:30.262806+0000","last_peered":"2021-03-15T22:32:30.262806+0000","last_clean":"2021-03-15T22:32:30.262806+0000","last_became_active":"2021-03-15T22:31:28.287770+0000","last_became_peered":"2021-03-15T22:31:28.287770+0000","last_unstale":"2021-03-15T22:32:30.262806+0000","last_undegraded":"2021-03-15T22:32:30.262806+0000","last_fullsized":"2021-03-15T22:32:30.262806+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.177019+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"89","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.262618+0000","last_change":"2021-03-15T22:31:28.274821+0000","last_active":"2021-03-15T22:32:30.262618+0000","last_peered":"2021-03-15T22:32:30.262618+0000","last_clean":"2021-03-15T22:32:30.262618+0000","last_became_active":"2021-03-15T22:31:28.274481+0000","last_became_peered":"2021-03-15T22:31:28.274481+0000","last_unstale":"2021-03-15T22:32:30.262618+0000","last_undegraded":"2021-03-15T22:32:30.262618+0000","last_fullsized":"2021-03-15T22:32:30.262618+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:11.029289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.029289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"234","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.265445+0000","last_change":"2021-03-15T22:31:28.285592+0000","last_active":"2021-03-15T22:32:30.265445+0000","last_peered":"2021-03-15T22:32:30.265445+0000","last_clean":"2021-03-15T22:32:30.265445+0000","last_became_active":"2021-03-15T22:31:28.285281+0000","last_became_peered":"2021-03-15T22:31:28.285281+0000","last_unstale":"2021-03-15T22:32:30.265445+0000","last_undegraded":"2021-03-15T22:32:30.265445+0000","last_fullsized":"2021-03-15T22:32:30.265445+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:05.262019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.262019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"521","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.279801+0000","last_change":"2021-03-15T22:31:58.617873+0000","last_active":"2021-03-15T22:32:30.279801+0000","last_peered":"2021-03-15T22:32:30.279801+0000","last_clean":"2021-03-15T22:32:30.279801+0000","last_became_active":"2021-03-15T22:31:28.289753+0000","last_became_peered":"2021-03-15T22:31:28.289753+0000","last_unstale":"2021-03-15T22:32:30.279801+0000","last_undegraded":"2021-03-15T22:32:30.279801+0000","last_fullsized":"2021-03-15T22:32:30.279801+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:58.617803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:58.617803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"534","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.266127+0000","last_change":"2021-03-15T22:32:10.653022+0000","last_active":"2021-03-15T22:32:30.266127+0000","last_peered":"2021-03-15T22:32:30.266127+0000","last_clean":"2021-03-15T22:32:30.266127+0000","last_became_active":"2021-03-15T22:30:32.258079+0000","last_became_peered":"2021-03-15T22:30:32.258079+0000","last_unstale":"2021-03-15T22:32:30.266127+0000","last_undegraded":"2021-03-15T22:32:30.266127+0000","last_fullsized":"2021-03-15T22:32:30.266127+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.652898+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"638","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.295207+0000","last_change":"2021-03-15T22:32:12.794180+0000","last_active":"2021-03-15T22:32:32.295207+0000","last_peered":"2021-03-15T22:32:32.295207+0000","last_clean":"2021-03-15T22:32:32.295207+0000","last_became_active":"2021-03-15T22:31:20.246770+0000","last_became_peered":"2021-03-15T22:31:20.246770+0000","last_unstale":"2021-03-15T22:32:32.295207+0000","last_undegraded":"2021-03-15T22:32:32.295207+0000","last_fullsized":"2021-03-15T22:32:32.295207+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.794113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.794113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"271","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310304+0000","last_change":"2021-03-15T22:32:01.452949+0000","last_active":"2021-03-15T22:32:34.310304+0000","last_peered":"2021-03-15T22:32:34.310304+0000","last_clean":"2021-03-15T22:32:34.310304+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:32:34.310304+0000","last_undegraded":"2021-03-15T22:32:34.310304+0000","last_fullsized":"2021-03-15T22:32:34.310304+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:01.452885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:32:01.452885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"183","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.294712+0000","last_change":"2021-03-15T22:31:28.288569+0000","last_active":"2021-03-15T22:32:32.294712+0000","last_peered":"2021-03-15T22:32:32.294712+0000","last_clean":"2021-03-15T22:32:32.294712+0000","last_became_active":"2021-03-15T22:31:28.288226+0000","last_became_peered":"2021-03-15T22:31:28.288226+0000","last_unstale":"2021-03-15T22:32:32.294712+0000","last_undegraded":"2021-03-15T22:32:32.294712+0000","last_fullsized":"2021-03-15T22:32:32.294712+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.146408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.146408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"945","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.261641+0000","last_change":"2021-03-15T22:30:28.328526+0000","last_active":"2021-03-15T22:32:30.261641+0000","last_peered":"2021-03-15T22:32:30.261641+0000","last_clean":"2021-03-15T22:32:30.261641+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:32:30.261641+0000","last_undegraded":"2021-03-15T22:32:30.261641+0000","last_fullsized":"2021-03-15T22:32:30.261641+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:30:28.328458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.328458+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"81","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.305015+0000","last_change":"2021-03-15T22:31:20.250057+0000","last_active":"2021-03-15T22:32:32.305015+0000","last_peered":"2021-03-15T22:32:32.305015+0000","last_clean":"2021-03-15T22:32:32.305015+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:32:32.305015+0000","last_undegraded":"2021-03-15T22:32:32.305015+0000","last_fullsized":"2021-03-15T22:32:32.305015+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:07.063227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.063227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"419","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.295383+0000","last_change":"2021-03-15T22:30:40.173603+0000","last_active":"2021-03-15T22:32:32.295383+0000","last_peered":"2021-03-15T22:32:32.295383+0000","last_clean":"2021-03-15T22:32:32.295383+0000","last_became_active":"2021-03-15T22:29:52.223704+0000","last_became_peered":"2021-03-15T22:29:52.223704+0000","last_unstale":"2021-03-15T22:32:32.295383+0000","last_undegraded":"2021-03-15T22:32:32.295383+0000","last_fullsized":"2021-03-15T22:32:32.295383+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:40.173519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:30:40.173519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"179","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.295472+0000","last_change":"2021-03-15T22:32:14.756556+0000","last_active":"2021-03-15T22:32:32.295472+0000","last_peered":"2021-03-15T22:32:32.295472+0000","last_clean":"2021-03-15T22:32:32.295472+0000","last_became_active":"2021-03-15T22:29:51.234868+0000","last_became_peered":"2021-03-15T22:29:51.234868+0000","last_unstale":"2021-03-15T22:32:32.295472+0000","last_undegraded":"2021-03-15T22:32:32.295472+0000","last_fullsized":"2021-03-15T22:32:32.295472+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.756460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.756460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"888","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.266642+0000","last_change":"2021-03-15T22:32:05.111125+0000","last_active":"2021-03-15T22:32:30.266642+0000","last_peered":"2021-03-15T22:32:30.266642+0000","last_clean":"2021-03-15T22:32:30.266642+0000","last_became_active":"2021-03-15T22:29:11.197059+0000","last_became_peered":"2021-03-15T22:29:11.197059+0000","last_unstale":"2021-03-15T22:32:30.266642+0000","last_undegraded":"2021-03-15T22:32:30.266642+0000","last_fullsized":"2021-03-15T22:32:30.266642+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:32:05.111031+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.111031+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"757","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.744807+0000","last_change":"2021-03-15T22:32:32.744807+0000","last_active":"2021-03-15T22:32:32.744807+0000","last_peered":"2021-03-15T22:32:32.744807+0000","last_clean":"2021-03-15T22:32:32.744807+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:32:32.744807+0000","last_undegraded":"2021-03-15T22:32:32.744807+0000","last_fullsized":"2021-03-15T22:32:32.744807+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:32:32.744716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.744716+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"809","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.268132+0000","last_change":"2021-03-15T22:31:37.466525+0000","last_active":"2021-03-15T22:32:30.268132+0000","last_peered":"2021-03-15T22:32:30.268132+0000","last_clean":"2021-03-15T22:32:30.268132+0000","last_became_active":"2021-03-15T22:31:28.281143+0000","last_became_peered":"2021-03-15T22:31:28.281143+0000","last_unstale":"2021-03-15T22:32:30.268132+0000","last_undegraded":"2021-03-15T22:32:30.268132+0000","last_fullsized":"2021-03-15T22:32:30.268132+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:31:37.466443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"270","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310653+0000","last_change":"2021-03-15T22:30:38.781792+0000","last_active":"2021-03-15T22:32:34.310653+0000","last_peered":"2021-03-15T22:32:34.310653+0000","last_clean":"2021-03-15T22:32:34.310653+0000","last_became_active":"2021-03-15T22:28:12.519619+0000","last_became_peered":"2021-03-15T22:28:12.519619+0000","last_unstale":"2021-03-15T22:32:34.310653+0000","last_undegraded":"2021-03-15T22:32:34.310653+0000","last_fullsized":"2021-03-15T22:32:34.310653+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:30:38.781724+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.781724+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"676","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.323578+0000","last_change":"2021-03-15T22:32:10.001304+0000","last_active":"2021-03-15T22:32:33.323578+0000","last_peered":"2021-03-15T22:32:33.323578+0000","last_clean":"2021-03-15T22:32:33.323578+0000","last_became_active":"2021-03-15T22:30:32.250733+0000","last_became_peered":"2021-03-15T22:30:32.250733+0000","last_unstale":"2021-03-15T22:32:33.323578+0000","last_undegraded":"2021-03-15T22:32:33.323578+0000","last_fullsized":"2021-03-15T22:32:33.323578+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:32:10.001234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.001234+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"829","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.266287+0000","last_change":"2021-03-15T22:32:10.144209+0000","last_active":"2021-03-15T22:32:30.266287+0000","last_peered":"2021-03-15T22:32:30.266287+0000","last_clean":"2021-03-15T22:32:30.266287+0000","last_became_active":"2021-03-15T22:31:20.240898+0000","last_became_peered":"2021-03-15T22:31:20.240898+0000","last_unstale":"2021-03-15T22:32:30.266287+0000","last_undegraded":"2021-03-15T22:32:30.266287+0000","last_fullsized":"2021-03-15T22:32:30.266287+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.144132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.144132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"625","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.278723+0000","last_change":"2021-03-15T22:32:14.594783+0000","last_active":"2021-03-15T22:32:30.278723+0000","last_peered":"2021-03-15T22:32:30.278723+0000","last_clean":"2021-03-15T22:32:30.278723+0000","last_became_active":"2021-03-15T22:31:28.286067+0000","last_became_peered":"2021-03-15T22:31:28.286067+0000","last_unstale":"2021-03-15T22:32:30.278723+0000","last_undegraded":"2021-03-15T22:32:30.278723+0000","last_fullsized":"2021-03-15T22:32:30.278723+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.594717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:45.403769+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.594717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"790","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.261718+0000","last_change":"2021-03-15T22:30:39.443682+0000","last_active":"2021-03-15T22:32:30.261718+0000","last_peered":"2021-03-15T22:32:30.261718+0000","last_clean":"2021-03-15T22:32:30.261718+0000","last_became_active":"2021-03-15T22:30:31.243315+0000","last_became_peered":"2021-03-15T22:30:31.243315+0000","last_unstale":"2021-03-15T22:32:30.261718+0000","last_undegraded":"2021-03-15T22:32:30.261718+0000","last_fullsized":"2021-03-15T22:32:30.261718+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.443613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"245","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.270393+0000","last_change":"2021-03-15T22:29:54.564185+0000","last_active":"2021-03-15T22:32:30.270393+0000","last_peered":"2021-03-15T22:32:30.270393+0000","last_clean":"2021-03-15T22:32:30.270393+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:32:30.270393+0000","last_undegraded":"2021-03-15T22:32:30.270393+0000","last_fullsized":"2021-03-15T22:32:30.270393+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:29:54.564115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:54.564115+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"771'16","reported_seq":"362","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.265331+0000","last_change":"2021-03-15T22:32:01.253003+0000","last_active":"2021-03-15T22:32:30.265331+0000","last_peered":"2021-03-15T22:32:30.265331+0000","last_clean":"2021-03-15T22:32:30.265331+0000","last_became_active":"2021-03-15T22:32:01.252590+0000","last_became_peered":"2021-03-15T22:32:01.252590+0000","last_unstale":"2021-03-15T22:32:30.265331+0000","last_undegraded":"2021-03-15T22:32:30.265331+0000","last_fullsized":"2021-03-15T22:32:30.265331+0000","mapping_epoch":770,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":771,"parent":"0.0","parent_split_bits":5,"last_scrub":"618'7","last_scrub_stamp":"2021-03-15T22:29:40.964753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.964753+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"809","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.305452+0000","last_change":"2021-03-15T22:30:32.243983+0000","last_active":"2021-03-15T22:32:32.305452+0000","last_peered":"2021-03-15T22:32:32.305452+0000","last_clean":"2021-03-15T22:32:32.305452+0000","last_became_active":"2021-03-15T22:30:32.243708+0000","last_became_peered":"2021-03-15T22:30:32.243708+0000","last_unstale":"2021-03-15T22:32:32.305452+0000","last_undegraded":"2021-03-15T22:32:32.305452+0000","last_fullsized":"2021-03-15T22:32:32.305452+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:01.208847+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:01.208847+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"825","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.264234+0000","last_change":"2021-03-15T22:31:05.123509+0000","last_active":"2021-03-15T22:32:31.264234+0000","last_peered":"2021-03-15T22:32:31.264234+0000","last_clean":"2021-03-15T22:32:31.264234+0000","last_became_active":"2021-03-15T22:28:59.264559+0000","last_became_peered":"2021-03-15T22:28:59.264559+0000","last_unstale":"2021-03-15T22:32:31.264234+0000","last_undegraded":"2021-03-15T22:32:31.264234+0000","last_fullsized":"2021-03-15T22:32:31.264234+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:05.123425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.123425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"371","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310988+0000","last_change":"2021-03-15T22:30:41.759292+0000","last_active":"2021-03-15T22:32:34.310988+0000","last_peered":"2021-03-15T22:32:34.310988+0000","last_clean":"2021-03-15T22:32:34.310988+0000","last_became_active":"2021-03-15T22:28:59.277385+0000","last_became_peered":"2021-03-15T22:28:59.277385+0000","last_unstale":"2021-03-15T22:32:34.310988+0000","last_undegraded":"2021-03-15T22:32:34.310988+0000","last_fullsized":"2021-03-15T22:32:34.310988+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:30:41.759224+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.759224+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"812","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.739728+0000","last_change":"2021-03-15T22:32:34.739698+0000","last_active":"2021-03-15T22:32:34.739728+0000","last_peered":"2021-03-15T22:32:34.739728+0000","last_clean":"2021-03-15T22:32:34.739728+0000","last_became_active":"2021-03-15T22:31:23.258435+0000","last_became_peered":"2021-03-15T22:31:23.258435+0000","last_unstale":"2021-03-15T22:32:34.739728+0000","last_undegraded":"2021-03-15T22:32:34.739728+0000","last_fullsized":"2021-03-15T22:32:34.739728+0000","mapping_epoch":733,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:32:34.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.739637+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"803","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.264813+0000","last_change":"2021-03-15T22:31:40.870498+0000","last_active":"2021-03-15T22:32:31.264813+0000","last_peered":"2021-03-15T22:32:31.264813+0000","last_clean":"2021-03-15T22:32:31.264813+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:32:31.264813+0000","last_undegraded":"2021-03-15T22:32:31.264813+0000","last_fullsized":"2021-03-15T22:32:31.264813+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:40.870404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:40.870404+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"404","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.270248+0000","last_change":"2021-03-15T22:31:10.332382+0000","last_active":"2021-03-15T22:32:30.270248+0000","last_peered":"2021-03-15T22:32:30.270248+0000","last_clean":"2021-03-15T22:32:30.270248+0000","last_became_active":"2021-03-15T22:29:00.220073+0000","last_became_peered":"2021-03-15T22:29:00.220073+0000","last_unstale":"2021-03-15T22:32:30.270248+0000","last_undegraded":"2021-03-15T22:32:30.270248+0000","last_fullsized":"2021-03-15T22:32:30.270248+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:10.332319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.332319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"793","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.294729+0000","last_change":"2021-03-15T22:31:28.320040+0000","last_active":"2021-03-15T22:32:32.294729+0000","last_peered":"2021-03-15T22:32:32.294729+0000","last_clean":"2021-03-15T22:32:32.294729+0000","last_became_active":"2021-03-15T22:31:28.287309+0000","last_became_peered":"2021-03-15T22:31:28.287309+0000","last_unstale":"2021-03-15T22:32:32.294729+0000","last_undegraded":"2021-03-15T22:32:32.294729+0000","last_fullsized":"2021-03-15T22:32:32.294729+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:31:03.281344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:03.281344+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"778","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.304944+0000","last_change":"2021-03-15T22:30:59.480779+0000","last_active":"2021-03-15T22:32:32.304944+0000","last_peered":"2021-03-15T22:32:32.304944+0000","last_clean":"2021-03-15T22:32:32.304944+0000","last_became_active":"2021-03-15T22:30:31.230649+0000","last_became_peered":"2021-03-15T22:30:31.230649+0000","last_unstale":"2021-03-15T22:32:32.304944+0000","last_undegraded":"2021-03-15T22:32:32.304944+0000","last_fullsized":"2021-03-15T22:32:32.304944+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:59.480717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:30:59.480717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"75","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.263790+0000","last_change":"2021-03-15T22:31:28.278722+0000","last_active":"2021-03-15T22:32:30.263790+0000","last_peered":"2021-03-15T22:32:30.263790+0000","last_clean":"2021-03-15T22:32:30.263790+0000","last_became_active":"2021-03-15T22:31:28.278193+0000","last_became_peered":"2021-03-15T22:31:28.278193+0000","last_unstale":"2021-03-15T22:32:30.263790+0000","last_undegraded":"2021-03-15T22:32:30.263790+0000","last_fullsized":"2021-03-15T22:32:30.263790+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:06.296662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:06.296662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"310","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.311076+0000","last_change":"2021-03-15T22:31:33.136177+0000","last_active":"2021-03-15T22:32:34.311076+0000","last_peered":"2021-03-15T22:32:34.311076+0000","last_clean":"2021-03-15T22:32:34.311076+0000","last_became_active":"2021-03-15T22:29:51.226677+0000","last_became_peered":"2021-03-15T22:29:51.226677+0000","last_unstale":"2021-03-15T22:32:34.311076+0000","last_undegraded":"2021-03-15T22:32:34.311076+0000","last_fullsized":"2021-03-15T22:32:34.311076+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:33.136112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:31:33.136112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"278","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.275416+0000","last_change":"2021-03-15T22:31:28.306827+0000","last_active":"2021-03-15T22:32:30.275416+0000","last_peered":"2021-03-15T22:32:30.275416+0000","last_clean":"2021-03-15T22:32:30.275416+0000","last_became_active":"2021-03-15T22:31:28.284715+0000","last_became_peered":"2021-03-15T22:31:28.284715+0000","last_unstale":"2021-03-15T22:32:30.275416+0000","last_undegraded":"2021-03-15T22:32:30.275416+0000","last_fullsized":"2021-03-15T22:32:30.275416+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:30:07.486190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.486190+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":7512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"413","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.268069+0000","last_change":"2021-03-15T22:31:28.288399+0000","last_active":"2021-03-15T22:32:30.268069+0000","last_peered":"2021-03-15T22:32:30.268069+0000","last_clean":"2021-03-15T22:32:30.268069+0000","last_became_active":"2021-03-15T22:31:28.288084+0000","last_became_peered":"2021-03-15T22:31:28.288084+0000","last_unstale":"2021-03-15T22:32:30.268069+0000","last_undegraded":"2021-03-15T22:32:30.268069+0000","last_fullsized":"2021-03-15T22:32:30.268069+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.317817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"774","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.305429+0000","last_change":"2021-03-15T22:32:05.522466+0000","last_active":"2021-03-15T22:32:32.305429+0000","last_peered":"2021-03-15T22:32:32.305429+0000","last_clean":"2021-03-15T22:32:32.305429+0000","last_became_active":"2021-03-15T22:31:28.281921+0000","last_became_peered":"2021-03-15T22:31:28.281921+0000","last_unstale":"2021-03-15T22:32:32.305429+0000","last_undegraded":"2021-03-15T22:32:32.305429+0000","last_fullsized":"2021-03-15T22:32:32.305429+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:05.522372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.522372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"174","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.296037+0000","last_change":"2021-03-15T22:30:42.233176+0000","last_active":"2021-03-15T22:32:32.296037+0000","last_peered":"2021-03-15T22:32:32.296037+0000","last_clean":"2021-03-15T22:32:32.296037+0000","last_became_active":"2021-03-15T22:29:51.220261+0000","last_became_peered":"2021-03-15T22:29:51.220261+0000","last_unstale":"2021-03-15T22:32:32.296037+0000","last_undegraded":"2021-03-15T22:32:32.296037+0000","last_fullsized":"2021-03-15T22:32:32.296037+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.233098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"613","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.267726+0000","last_change":"2021-03-15T22:31:28.302192+0000","last_active":"2021-03-15T22:32:30.267726+0000","last_peered":"2021-03-15T22:32:30.267726+0000","last_clean":"2021-03-15T22:32:30.267726+0000","last_became_active":"2021-03-15T22:31:28.283474+0000","last_became_peered":"2021-03-15T22:31:28.283474+0000","last_unstale":"2021-03-15T22:32:30.267726+0000","last_undegraded":"2021-03-15T22:32:30.267726+0000","last_fullsized":"2021-03-15T22:32:30.267726+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:31:07.330960+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:31:07.330960+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"290","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310260+0000","last_change":"2021-03-15T22:31:43.405661+0000","last_active":"2021-03-15T22:32:34.310260+0000","last_peered":"2021-03-15T22:32:34.310260+0000","last_clean":"2021-03-15T22:32:34.310260+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:32:34.310260+0000","last_undegraded":"2021-03-15T22:32:34.310260+0000","last_fullsized":"2021-03-15T22:32:34.310260+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:43.405600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:31:43.405600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"589","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.265708+0000","last_change":"2021-03-15T22:29:56.525891+0000","last_active":"2021-03-15T22:32:30.265708+0000","last_peered":"2021-03-15T22:32:30.265708+0000","last_clean":"2021-03-15T22:32:30.265708+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:32:30.265708+0000","last_undegraded":"2021-03-15T22:32:30.265708+0000","last_fullsized":"2021-03-15T22:32:30.265708+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:56.525822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.525822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"805","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.438804+0000","last_change":"2021-03-15T22:32:33.438804+0000","last_active":"2021-03-15T22:32:33.438804+0000","last_peered":"2021-03-15T22:32:33.438804+0000","last_clean":"2021-03-15T22:32:33.438804+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:32:33.438804+0000","last_undegraded":"2021-03-15T22:32:33.438804+0000","last_fullsized":"2021-03-15T22:32:33.438804+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.438742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.438742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"531","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.263791+0000","last_change":"2021-03-15T22:32:08.880092+0000","last_active":"2021-03-15T22:32:31.263791+0000","last_peered":"2021-03-15T22:32:31.263791+0000","last_clean":"2021-03-15T22:32:31.263791+0000","last_became_active":"2021-03-15T22:31:28.276464+0000","last_became_peered":"2021-03-15T22:31:28.276464+0000","last_unstale":"2021-03-15T22:32:31.263791+0000","last_undegraded":"2021-03-15T22:32:31.263791+0000","last_fullsized":"2021-03-15T22:32:31.263791+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:32:08.879999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:08.879999+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"426","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.265959+0000","last_change":"2021-03-15T22:32:06.670385+0000","last_active":"2021-03-15T22:32:30.265959+0000","last_peered":"2021-03-15T22:32:30.265959+0000","last_clean":"2021-03-15T22:32:30.265959+0000","last_became_active":"2021-03-15T22:29:51.218982+0000","last_became_peered":"2021-03-15T22:29:51.218982+0000","last_unstale":"2021-03-15T22:32:30.265959+0000","last_undegraded":"2021-03-15T22:32:30.265959+0000","last_fullsized":"2021-03-15T22:32:30.265959+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:06.670324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:32:06.670324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"847","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.263606+0000","last_change":"2021-03-15T22:32:30.719489+0000","last_active":"2021-03-15T22:32:31.263606+0000","last_peered":"2021-03-15T22:32:31.263606+0000","last_clean":"2021-03-15T22:32:31.263606+0000","last_became_active":"2021-03-15T22:29:51.222254+0000","last_became_peered":"2021-03-15T22:29:51.222254+0000","last_unstale":"2021-03-15T22:32:31.263606+0000","last_undegraded":"2021-03-15T22:32:31.263606+0000","last_fullsized":"2021-03-15T22:32:31.263606+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:30.719431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.719431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"73","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.266093+0000","last_change":"2021-03-15T22:31:28.279172+0000","last_active":"2021-03-15T22:32:30.266093+0000","last_peered":"2021-03-15T22:32:30.266093+0000","last_clean":"2021-03-15T22:32:30.266093+0000","last_became_active":"2021-03-15T22:31:28.278787+0000","last_became_peered":"2021-03-15T22:31:28.278787+0000","last_unstale":"2021-03-15T22:32:30.266093+0000","last_undegraded":"2021-03-15T22:32:30.266093+0000","last_fullsized":"2021-03-15T22:32:30.266093+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.992352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.992352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"853","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.266552+0000","last_change":"2021-03-15T22:31:42.397085+0000","last_active":"2021-03-15T22:32:30.266552+0000","last_peered":"2021-03-15T22:32:30.266552+0000","last_clean":"2021-03-15T22:32:30.266552+0000","last_became_active":"2021-03-15T22:31:28.282759+0000","last_became_peered":"2021-03-15T22:31:28.282759+0000","last_unstale":"2021-03-15T22:32:30.266552+0000","last_undegraded":"2021-03-15T22:32:30.266552+0000","last_fullsized":"2021-03-15T22:32:30.266552+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:31:42.396996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.396996+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"284","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.311256+0000","last_change":"2021-03-15T22:30:42.771039+0000","last_active":"2021-03-15T22:32:34.311256+0000","last_peered":"2021-03-15T22:32:34.311256+0000","last_clean":"2021-03-15T22:32:34.311256+0000","last_became_active":"2021-03-15T22:28:59.276812+0000","last_became_peered":"2021-03-15T22:28:59.276812+0000","last_unstale":"2021-03-15T22:32:34.311256+0000","last_undegraded":"2021-03-15T22:32:34.311256+0000","last_fullsized":"2021-03-15T22:32:34.311256+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:42.770976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.770976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"642'8","reported_seq":"211","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.295844+0000","last_change":"2021-03-15T22:30:06.920670+0000","last_active":"2021-03-15T22:32:32.295844+0000","last_peered":"2021-03-15T22:32:32.295844+0000","last_clean":"2021-03-15T22:32:32.295844+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:32:32.295844+0000","last_undegraded":"2021-03-15T22:32:32.295844+0000","last_fullsized":"2021-03-15T22:32:32.295844+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"642'8","last_scrub_stamp":"2021-03-15T22:30:06.920606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:06.920606+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"779","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.304970+0000","last_change":"2021-03-15T22:30:35.423621+0000","last_active":"2021-03-15T22:32:32.304970+0000","last_peered":"2021-03-15T22:32:32.304970+0000","last_clean":"2021-03-15T22:32:32.304970+0000","last_became_active":"2021-03-15T22:30:31.230560+0000","last_became_peered":"2021-03-15T22:30:31.230560+0000","last_unstale":"2021-03-15T22:32:32.304970+0000","last_undegraded":"2021-03-15T22:32:32.304970+0000","last_fullsized":"2021-03-15T22:32:32.304970+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:35.423559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:35.423559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"556'24","reported_seq":"588","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.267624+0000","last_change":"2021-03-15T22:30:02.548134+0000","last_active":"2021-03-15T22:32:30.267624+0000","last_peered":"2021-03-15T22:32:30.267624+0000","last_clean":"2021-03-15T22:32:30.267624+0000","last_became_active":"2021-03-15T22:29:00.226652+0000","last_became_peered":"2021-03-15T22:29:00.226652+0000","last_unstale":"2021-03-15T22:32:30.267624+0000","last_undegraded":"2021-03-15T22:32:30.267624+0000","last_fullsized":"2021-03-15T22:32:30.267624+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'24","last_scrub_stamp":"2021-03-15T22:30:02.548069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.548069+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"815","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.272227+0000","last_change":"2021-03-15T22:31:41.407408+0000","last_active":"2021-03-15T22:32:30.272227+0000","last_peered":"2021-03-15T22:32:30.272227+0000","last_clean":"2021-03-15T22:32:30.272227+0000","last_became_active":"2021-03-15T22:30:32.247782+0000","last_became_peered":"2021-03-15T22:30:32.247782+0000","last_unstale":"2021-03-15T22:32:30.272227+0000","last_undegraded":"2021-03-15T22:32:30.272227+0000","last_fullsized":"2021-03-15T22:32:30.272227+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:41.407324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.407324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"683'14","reported_seq":"486","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.267502+0000","last_change":"2021-03-15T22:31:41.551496+0000","last_active":"2021-03-15T22:32:30.267502+0000","last_peered":"2021-03-15T22:32:30.267502+0000","last_clean":"2021-03-15T22:32:30.267502+0000","last_became_active":"2021-03-15T22:30:32.252383+0000","last_became_peered":"2021-03-15T22:30:32.252383+0000","last_unstale":"2021-03-15T22:32:30.267502+0000","last_undegraded":"2021-03-15T22:32:30.267502+0000","last_fullsized":"2021-03-15T22:32:30.267502+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'14","last_scrub_stamp":"2021-03-15T22:31:41.551435+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.551435+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"289","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310395+0000","last_change":"2021-03-15T22:32:03.502945+0000","last_active":"2021-03-15T22:32:34.310395+0000","last_peered":"2021-03-15T22:32:34.310395+0000","last_clean":"2021-03-15T22:32:34.310395+0000","last_became_active":"2021-03-15T22:28:59.265822+0000","last_became_peered":"2021-03-15T22:28:59.265822+0000","last_unstale":"2021-03-15T22:32:34.310395+0000","last_undegraded":"2021-03-15T22:32:34.310395+0000","last_fullsized":"2021-03-15T22:32:34.310395+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:03.502854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:32:03.502854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"192","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.295786+0000","last_change":"2021-03-15T22:30:36.106413+0000","last_active":"2021-03-15T22:32:32.295786+0000","last_peered":"2021-03-15T22:32:32.295786+0000","last_clean":"2021-03-15T22:32:32.295786+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:32:32.295786+0000","last_undegraded":"2021-03-15T22:32:32.295786+0000","last_fullsized":"2021-03-15T22:32:32.295786+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:30:36.106285+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.106285+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"79","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.278416+0000","last_change":"2021-03-15T22:31:20.257902+0000","last_active":"2021-03-15T22:32:30.278416+0000","last_peered":"2021-03-15T22:32:30.278416+0000","last_clean":"2021-03-15T22:32:30.278416+0000","last_became_active":"2021-03-15T22:31:20.256557+0000","last_became_peered":"2021-03-15T22:31:20.256557+0000","last_unstale":"2021-03-15T22:32:30.278416+0000","last_undegraded":"2021-03-15T22:32:30.278416+0000","last_fullsized":"2021-03-15T22:32:30.278416+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:11.436723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.436723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"579","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.323182+0000","last_change":"2021-03-15T22:32:12.036589+0000","last_active":"2021-03-15T22:32:33.323182+0000","last_peered":"2021-03-15T22:32:33.323182+0000","last_clean":"2021-03-15T22:32:33.323182+0000","last_became_active":"2021-03-15T22:30:31.254532+0000","last_became_peered":"2021-03-15T22:30:31.254532+0000","last_unstale":"2021-03-15T22:32:33.323182+0000","last_undegraded":"2021-03-15T22:32:33.323182+0000","last_fullsized":"2021-03-15T22:32:33.323182+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.036522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.036522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"783","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.307338+0000","last_change":"2021-03-15T22:31:34.511881+0000","last_active":"2021-03-15T22:32:32.307338+0000","last_peered":"2021-03-15T22:32:32.307338+0000","last_clean":"2021-03-15T22:32:32.307338+0000","last_became_active":"2021-03-15T22:31:28.276738+0000","last_became_peered":"2021-03-15T22:31:28.276738+0000","last_unstale":"2021-03-15T22:32:32.307338+0000","last_undegraded":"2021-03-15T22:32:32.307338+0000","last_fullsized":"2021-03-15T22:32:32.307338+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:34.511804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:34.511804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"329","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.311564+0000","last_change":"2021-03-15T22:31:04.009824+0000","last_active":"2021-03-15T22:32:34.311564+0000","last_peered":"2021-03-15T22:32:34.311564+0000","last_clean":"2021-03-15T22:32:34.311564+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:32:34.311564+0000","last_undegraded":"2021-03-15T22:32:34.311564+0000","last_fullsized":"2021-03-15T22:32:34.311564+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:31:04.009748+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.009748+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"599","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.296064+0000","last_change":"2021-03-15T22:32:11.834186+0000","last_active":"2021-03-15T22:32:32.296064+0000","last_peered":"2021-03-15T22:32:32.296064+0000","last_clean":"2021-03-15T22:32:32.296064+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:32:32.296064+0000","last_undegraded":"2021-03-15T22:32:32.296064+0000","last_fullsized":"2021-03-15T22:32:32.296064+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:11.834104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:11.834104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"777","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.306078+0000","last_change":"2021-03-15T22:31:35.559671+0000","last_active":"2021-03-15T22:32:32.306078+0000","last_peered":"2021-03-15T22:32:32.306078+0000","last_clean":"2021-03-15T22:32:32.306078+0000","last_became_active":"2021-03-15T22:31:28.286252+0000","last_became_peered":"2021-03-15T22:31:28.286252+0000","last_unstale":"2021-03-15T22:32:32.306078+0000","last_undegraded":"2021-03-15T22:32:32.306078+0000","last_fullsized":"2021-03-15T22:32:32.306078+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:35.559607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:35.559607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"762","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310111+0000","last_change":"2021-03-15T22:31:28.287644+0000","last_active":"2021-03-15T22:32:34.310111+0000","last_peered":"2021-03-15T22:32:34.310111+0000","last_clean":"2021-03-15T22:32:34.310111+0000","last_became_active":"2021-03-15T22:31:28.287285+0000","last_became_peered":"2021-03-15T22:31:28.287285+0000","last_unstale":"2021-03-15T22:32:34.310111+0000","last_undegraded":"2021-03-15T22:32:34.310111+0000","last_fullsized":"2021-03-15T22:32:34.310111+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.321715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.321715+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.321715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"430","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.278317+0000","last_change":"2021-03-15T22:32:01.586874+0000","last_active":"2021-03-15T22:32:30.278317+0000","last_peered":"2021-03-15T22:32:30.278317+0000","last_clean":"2021-03-15T22:32:30.278317+0000","last_became_active":"2021-03-15T22:29:11.204254+0000","last_became_peered":"2021-03-15T22:29:11.204254+0000","last_unstale":"2021-03-15T22:32:30.278317+0000","last_undegraded":"2021-03-15T22:32:30.278317+0000","last_fullsized":"2021-03-15T22:32:30.278317+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:32:01.586755+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:32:01.586755+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"802'1010","reported_seq":"3446","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.317768+0000","last_change":"2021-03-15T22:32:32.317985+0000","last_active":"2021-03-15T22:32:32.317768+0000","last_peered":"2021-03-15T22:32:32.317768+0000","last_clean":"2021-03-15T22:32:32.317768+0000","last_became_active":"2021-03-15T22:31:31.240075+0000","last_became_peered":"2021-03-15T22:31:31.240075+0000","last_unstale":"2021-03-15T22:32:32.317768+0000","last_undegraded":"2021-03-15T22:32:32.317768+0000","last_fullsized":"2021-03-15T22:32:32.317768+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'874","last_scrub_stamp":"2021-03-15T22:31:34.960268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:31:34.960268+0000","log_size":1010,"ondisk_log_size":1010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33119903,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":404,"num_read_kb":455535,"num_write":271,"num_write_kb":86607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":59821512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"292","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"732","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.294984+0000","last_change":"2021-03-15T22:31:28.283539+0000","last_active":"2021-03-15T22:32:32.294984+0000","last_peered":"2021-03-15T22:32:32.294984+0000","last_clean":"2021-03-15T22:32:32.294984+0000","last_became_active":"2021-03-15T22:31:28.283092+0000","last_became_peered":"2021-03-15T22:31:28.283092+0000","last_unstale":"2021-03-15T22:32:32.294984+0000","last_undegraded":"2021-03-15T22:32:32.294984+0000","last_fullsized":"2021-03-15T22:32:32.294984+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:07.417736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:07.417736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"807","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.305586+0000","last_change":"2021-03-15T22:31:58.607690+0000","last_active":"2021-03-15T22:32:32.305586+0000","last_peered":"2021-03-15T22:32:32.305586+0000","last_clean":"2021-03-15T22:32:32.305586+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:32:32.305586+0000","last_undegraded":"2021-03-15T22:32:32.305586+0000","last_fullsized":"2021-03-15T22:32:32.305586+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:31:58.607619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:58.607619+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"237","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.294651+0000","last_change":"2021-03-15T22:31:29.162156+0000","last_active":"2021-03-15T22:32:32.294651+0000","last_peered":"2021-03-15T22:32:32.294651+0000","last_clean":"2021-03-15T22:32:32.294651+0000","last_became_active":"2021-03-15T22:31:28.288606+0000","last_became_peered":"2021-03-15T22:31:28.288606+0000","last_unstale":"2021-03-15T22:32:32.294651+0000","last_undegraded":"2021-03-15T22:32:32.294651+0000","last_fullsized":"2021-03-15T22:32:32.294651+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.227473+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"801'11","reported_seq":"1004","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.307443+0000","last_change":"2021-03-15T22:32:31.265096+0000","last_active":"2021-03-15T22:32:32.307443+0000","last_peered":"2021-03-15T22:32:32.307443+0000","last_clean":"2021-03-15T22:32:32.307443+0000","last_became_active":"2021-03-15T22:32:31.264668+0000","last_became_peered":"2021-03-15T22:32:31.264668+0000","last_unstale":"2021-03-15T22:32:32.307443+0000","last_undegraded":"2021-03-15T22:32:32.307443+0000","last_fullsized":"2021-03-15T22:32:32.307443+0000","mapping_epoch":800,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:30:07.252960+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.252960+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"800'1396","reported_seq":"3707","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.387705+0000","last_change":"2021-03-15T22:32:30.387705+0000","last_active":"2021-03-15T22:32:30.387705+0000","last_peered":"2021-03-15T22:32:30.387705+0000","last_clean":"2021-03-15T22:32:30.387705+0000","last_became_active":"2021-03-15T22:30:34.291079+0000","last_became_peered":"2021-03-15T22:30:34.291079+0000","last_unstale":"2021-03-15T22:32:30.387705+0000","last_undegraded":"2021-03-15T22:32:30.387705+0000","last_fullsized":"2021-03-15T22:32:30.387705+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1312","last_scrub_stamp":"2021-03-15T22:32:05.639057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.639057+0000","log_size":1396,"ondisk_log_size":1396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":53615594,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":536,"num_read_kb":520708,"num_write":386,"num_write_kb":158842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":80476435,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"292","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"225","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.265572+0000","last_change":"2021-03-15T22:29:58.503593+0000","last_active":"2021-03-15T22:32:30.265572+0000","last_peered":"2021-03-15T22:32:30.265572+0000","last_clean":"2021-03-15T22:32:30.265572+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:32:30.265572+0000","last_undegraded":"2021-03-15T22:32:30.265572+0000","last_fullsized":"2021-03-15T22:32:30.265572+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:58.503533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:58.503533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"776","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.305451+0000","last_change":"2021-03-15T22:31:39.517144+0000","last_active":"2021-03-15T22:32:32.305451+0000","last_peered":"2021-03-15T22:32:32.305451+0000","last_clean":"2021-03-15T22:32:32.305451+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:32:32.305451+0000","last_undegraded":"2021-03-15T22:32:32.305451+0000","last_fullsized":"2021-03-15T22:32:32.305451+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:39.517052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:39.517052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"951","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.264396+0000","last_change":"2021-03-15T22:32:03.948964+0000","last_active":"2021-03-15T22:32:30.264396+0000","last_peered":"2021-03-15T22:32:30.264396+0000","last_clean":"2021-03-15T22:32:30.264396+0000","last_became_active":"2021-03-15T22:32:03.258590+0000","last_became_peered":"2021-03-15T22:32:03.258590+0000","last_unstale":"2021-03-15T22:32:30.264396+0000","last_undegraded":"2021-03-15T22:32:30.264396+0000","last_fullsized":"2021-03-15T22:32:30.264396+0000","mapping_epoch":772,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":773,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:31:05.469001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.469001+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":3460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"634","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.325024+0000","last_change":"2021-03-15T22:31:42.055712+0000","last_active":"2021-03-15T22:32:33.325024+0000","last_peered":"2021-03-15T22:32:33.325024+0000","last_clean":"2021-03-15T22:32:33.325024+0000","last_became_active":"2021-03-15T22:31:18.252524+0000","last_became_peered":"2021-03-15T22:31:18.252524+0000","last_unstale":"2021-03-15T22:32:33.325024+0000","last_undegraded":"2021-03-15T22:32:33.325024+0000","last_fullsized":"2021-03-15T22:32:33.325024+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"805'392","reported_seq":"2576","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.277703+0000","last_change":"2021-03-15T22:32:35.277933+0000","last_active":"2021-03-15T22:32:35.277703+0000","last_peered":"2021-03-15T22:32:35.277703+0000","last_clean":"2021-03-15T22:32:35.277703+0000","last_became_active":"2021-03-15T22:28:16.177364+0000","last_became_peered":"2021-03-15T22:28:16.177364+0000","last_unstale":"2021-03-15T22:32:35.277703+0000","last_undegraded":"2021-03-15T22:32:35.277703+0000","last_fullsized":"2021-03-15T22:32:35.277703+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'293","last_scrub_stamp":"2021-03-15T22:30:29.729234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.729234+0000","log_size":392,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10745993,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":79087,"num_write":107,"num_write_kb":35163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":11148008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"822","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.305698+0000","last_change":"2021-03-15T22:32:31.390410+0000","last_active":"2021-03-15T22:32:32.305698+0000","last_peered":"2021-03-15T22:32:32.305698+0000","last_clean":"2021-03-15T22:32:32.305698+0000","last_became_active":"2021-03-15T22:30:32.236752+0000","last_became_peered":"2021-03-15T22:30:32.236752+0000","last_unstale":"2021-03-15T22:32:32.305698+0000","last_undegraded":"2021-03-15T22:32:32.305698+0000","last_fullsized":"2021-03-15T22:32:32.305698+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"641","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.323928+0000","last_change":"2021-03-15T22:31:32.008461+0000","last_active":"2021-03-15T22:32:33.323928+0000","last_peered":"2021-03-15T22:32:33.323928+0000","last_clean":"2021-03-15T22:32:33.323928+0000","last_became_active":"2021-03-15T22:31:18.251403+0000","last_became_peered":"2021-03-15T22:31:18.251403+0000","last_unstale":"2021-03-15T22:32:33.323928+0000","last_undegraded":"2021-03-15T22:32:33.323928+0000","last_fullsized":"2021-03-15T22:32:33.323928+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:32.008403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:32.008403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"822","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.305293+0000","last_change":"2021-03-15T22:31:28.289675+0000","last_active":"2021-03-15T22:32:32.305293+0000","last_peered":"2021-03-15T22:32:32.305293+0000","last_clean":"2021-03-15T22:32:32.305293+0000","last_became_active":"2021-03-15T22:31:28.288901+0000","last_became_peered":"2021-03-15T22:31:28.288901+0000","last_unstale":"2021-03-15T22:32:32.305293+0000","last_undegraded":"2021-03-15T22:32:32.305293+0000","last_fullsized":"2021-03-15T22:32:32.305293+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:08.279129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:30:08.279129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"860","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.323365+0000","last_change":"2021-03-15T22:31:28.308317+0000","last_active":"2021-03-15T22:32:33.323365+0000","last_peered":"2021-03-15T22:32:33.323365+0000","last_clean":"2021-03-15T22:32:33.323365+0000","last_became_active":"2021-03-15T22:31:28.284158+0000","last_became_peered":"2021-03-15T22:31:28.284158+0000","last_unstale":"2021-03-15T22:32:33.323365+0000","last_undegraded":"2021-03-15T22:32:33.323365+0000","last_fullsized":"2021-03-15T22:32:33.323365+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:31:01.245498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:01.245498+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"802'785","reported_seq":"2688","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.325317+0000","last_change":"2021-03-15T22:32:33.325317+0000","last_active":"2021-03-15T22:32:33.325317+0000","last_peered":"2021-03-15T22:32:33.325317+0000","last_clean":"2021-03-15T22:32:33.325317+0000","last_became_active":"2021-03-15T22:30:31.242297+0000","last_became_peered":"2021-03-15T22:30:31.242297+0000","last_unstale":"2021-03-15T22:32:33.325317+0000","last_undegraded":"2021-03-15T22:32:33.325317+0000","last_fullsized":"2021-03-15T22:32:33.325317+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'783","last_scrub_stamp":"2021-03-15T22:32:31.034631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.034631+0000","log_size":785,"ondisk_log_size":785,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36638004,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":360,"num_read_kb":441676,"num_write":209,"num_write_kb":69663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":24240388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"292","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"282","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310297+0000","last_change":"2021-03-15T22:31:08.968657+0000","last_active":"2021-03-15T22:32:34.310297+0000","last_peered":"2021-03-15T22:32:34.310297+0000","last_clean":"2021-03-15T22:32:34.310297+0000","last_became_active":"2021-03-15T22:30:32.237720+0000","last_became_peered":"2021-03-15T22:30:32.237720+0000","last_unstale":"2021-03-15T22:32:34.310297+0000","last_undegraded":"2021-03-15T22:32:34.310297+0000","last_fullsized":"2021-03-15T22:32:34.310297+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"619","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.323521+0000","last_change":"2021-03-15T22:31:33.964163+0000","last_active":"2021-03-15T22:32:33.323521+0000","last_peered":"2021-03-15T22:32:33.323521+0000","last_clean":"2021-03-15T22:32:33.323521+0000","last_became_active":"2021-03-15T22:31:18.261916+0000","last_became_peered":"2021-03-15T22:31:18.261916+0000","last_unstale":"2021-03-15T22:32:33.323521+0000","last_undegraded":"2021-03-15T22:32:33.323521+0000","last_fullsized":"2021-03-15T22:32:33.323521+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:33.964082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:33.964082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"188","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.295402+0000","last_change":"2021-03-15T22:31:41.775620+0000","last_active":"2021-03-15T22:32:32.295402+0000","last_peered":"2021-03-15T22:32:32.295402+0000","last_clean":"2021-03-15T22:32:32.295402+0000","last_became_active":"2021-03-15T22:31:28.282098+0000","last_became_peered":"2021-03-15T22:31:28.282098+0000","last_unstale":"2021-03-15T22:32:32.295402+0000","last_undegraded":"2021-03-15T22:32:32.295402+0000","last_fullsized":"2021-03-15T22:32:32.295402+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:41.775530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.775530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"320","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310719+0000","last_change":"2021-03-15T22:32:33.701761+0000","last_active":"2021-03-15T22:32:34.310719+0000","last_peered":"2021-03-15T22:32:34.310719+0000","last_clean":"2021-03-15T22:32:34.310719+0000","last_became_active":"2021-03-15T22:30:32.250712+0000","last_became_peered":"2021-03-15T22:30:32.250712+0000","last_unstale":"2021-03-15T22:32:34.310719+0000","last_undegraded":"2021-03-15T22:32:34.310719+0000","last_fullsized":"2021-03-15T22:32:34.310719+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:32:33.701675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.701675+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"805'1058","reported_seq":"3769","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.289373+0000","last_change":"2021-03-15T22:32:35.289581+0000","last_active":"2021-03-15T22:32:35.289373+0000","last_peered":"2021-03-15T22:32:35.289373+0000","last_clean":"2021-03-15T22:32:35.289373+0000","last_became_active":"2021-03-15T22:31:31.245658+0000","last_became_peered":"2021-03-15T22:31:31.245658+0000","last_unstale":"2021-03-15T22:32:35.289373+0000","last_undegraded":"2021-03-15T22:32:35.289373+0000","last_fullsized":"2021-03-15T22:32:35.289373+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'870","last_scrub_stamp":"2021-03-15T22:31:02.243396+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.243396+0000","log_size":1058,"ondisk_log_size":1058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32728028,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":250,"num_read_kb":196596,"num_write":276,"num_write_kb":94316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":153074064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"225","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.279372+0000","last_change":"2021-03-15T22:30:29.533509+0000","last_active":"2021-03-15T22:32:30.279372+0000","last_peered":"2021-03-15T22:32:30.279372+0000","last_clean":"2021-03-15T22:32:30.279372+0000","last_became_active":"2021-03-15T22:28:59.273785+0000","last_became_peered":"2021-03-15T22:28:59.273785+0000","last_unstale":"2021-03-15T22:32:30.279372+0000","last_undegraded":"2021-03-15T22:32:30.279372+0000","last_fullsized":"2021-03-15T22:32:30.279372+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:29.533421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.533421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"185","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.295260+0000","last_change":"2021-03-15T22:31:58.785626+0000","last_active":"2021-03-15T22:32:32.295260+0000","last_peered":"2021-03-15T22:32:32.295260+0000","last_clean":"2021-03-15T22:32:32.295260+0000","last_became_active":"2021-03-15T22:31:28.287331+0000","last_became_peered":"2021-03-15T22:31:28.287331+0000","last_unstale":"2021-03-15T22:32:32.295260+0000","last_undegraded":"2021-03-15T22:32:32.295260+0000","last_fullsized":"2021-03-15T22:32:32.295260+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:58.785524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:31:58.785524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"150","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.323737+0000","last_change":"2021-03-15T22:32:33.052028+0000","last_active":"2021-03-15T22:32:33.323737+0000","last_peered":"2021-03-15T22:32:33.323737+0000","last_clean":"2021-03-15T22:32:33.323737+0000","last_became_active":"2021-03-15T22:31:28.278136+0000","last_became_peered":"2021-03-15T22:31:28.278136+0000","last_unstale":"2021-03-15T22:32:33.323737+0000","last_undegraded":"2021-03-15T22:32:33.323737+0000","last_fullsized":"2021-03-15T22:32:33.323737+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.051962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.051962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1018","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.264570+0000","last_change":"2021-03-15T22:32:06.881916+0000","last_active":"2021-03-15T22:32:31.264570+0000","last_peered":"2021-03-15T22:32:31.264570+0000","last_clean":"2021-03-15T22:32:31.264570+0000","last_became_active":"2021-03-15T22:30:32.244684+0000","last_became_peered":"2021-03-15T22:30:32.244684+0000","last_unstale":"2021-03-15T22:32:31.264570+0000","last_undegraded":"2021-03-15T22:32:31.264570+0000","last_fullsized":"2021-03-15T22:32:31.264570+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:32:06.881829+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:32:06.881829+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"266","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310541+0000","last_change":"2021-03-15T22:30:30.738196+0000","last_active":"2021-03-15T22:32:34.310541+0000","last_peered":"2021-03-15T22:32:34.310541+0000","last_clean":"2021-03-15T22:32:34.310541+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:32:34.310541+0000","last_undegraded":"2021-03-15T22:32:34.310541+0000","last_fullsized":"2021-03-15T22:32:34.310541+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:30.738108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:30.738108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"805'997","reported_seq":"3140","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.283242+0000","last_change":"2021-03-15T22:32:35.283473+0000","last_active":"2021-03-15T22:32:35.283242+0000","last_peered":"2021-03-15T22:32:35.283242+0000","last_clean":"2021-03-15T22:32:35.283242+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:32:35.283242+0000","last_undegraded":"2021-03-15T22:32:35.283242+0000","last_fullsized":"2021-03-15T22:32:35.283242+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"690'772","last_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.836165+0000","log_size":997,"ondisk_log_size":997,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":24374520,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":289,"num_read_kb":279538,"num_write":230,"num_write_kb":85131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"814","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.417321+0000","last_change":"2021-03-15T22:32:32.417288+0000","last_active":"2021-03-15T22:32:32.417321+0000","last_peered":"2021-03-15T22:32:32.417321+0000","last_clean":"2021-03-15T22:32:32.417321+0000","last_became_active":"2021-03-15T22:31:28.289352+0000","last_became_peered":"2021-03-15T22:31:28.289352+0000","last_unstale":"2021-03-15T22:32:32.417321+0000","last_undegraded":"2021-03-15T22:32:32.417321+0000","last_fullsized":"2021-03-15T22:32:32.417321+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:32.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"803","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.263563+0000","last_change":"2021-03-15T22:32:14.793976+0000","last_active":"2021-03-15T22:32:31.263563+0000","last_peered":"2021-03-15T22:32:31.263563+0000","last_clean":"2021-03-15T22:32:31.263563+0000","last_became_active":"2021-03-15T22:31:28.276641+0000","last_became_peered":"2021-03-15T22:31:28.276641+0000","last_unstale":"2021-03-15T22:32:31.263563+0000","last_undegraded":"2021-03-15T22:32:31.263563+0000","last_fullsized":"2021-03-15T22:32:31.263563+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.793900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.793900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"607","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:34.095292+0000","last_change":"2021-03-15T22:32:34.095255+0000","last_active":"2021-03-15T22:32:34.095292+0000","last_peered":"2021-03-15T22:32:34.095292+0000","last_clean":"2021-03-15T22:32:34.095292+0000","last_became_active":"2021-03-15T22:30:32.250316+0000","last_became_peered":"2021-03-15T22:30:32.250316+0000","last_unstale":"2021-03-15T22:32:34.095292+0000","last_undegraded":"2021-03-15T22:32:34.095292+0000","last_fullsized":"2021-03-15T22:32:34.095292+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:34.095158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.095158+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"545","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.265134+0000","last_change":"2021-03-15T22:31:59.864528+0000","last_active":"2021-03-15T22:32:31.265134+0000","last_peered":"2021-03-15T22:32:31.265134+0000","last_clean":"2021-03-15T22:32:31.265134+0000","last_became_active":"2021-03-15T22:29:51.222262+0000","last_became_peered":"2021-03-15T22:29:51.222262+0000","last_unstale":"2021-03-15T22:32:31.265134+0000","last_undegraded":"2021-03-15T22:32:31.265134+0000","last_fullsized":"2021-03-15T22:32:31.265134+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:31:59.864460+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:31:59.864460+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"602","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.323826+0000","last_change":"2021-03-15T22:32:04.969553+0000","last_active":"2021-03-15T22:32:33.323826+0000","last_peered":"2021-03-15T22:32:33.323826+0000","last_clean":"2021-03-15T22:32:33.323826+0000","last_became_active":"2021-03-15T22:31:18.249044+0000","last_became_peered":"2021-03-15T22:31:18.249044+0000","last_unstale":"2021-03-15T22:32:33.323826+0000","last_undegraded":"2021-03-15T22:32:33.323826+0000","last_fullsized":"2021-03-15T22:32:33.323826+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:04.969491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:04.969491+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"805'2425","reported_seq":"5164","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.293754+0000","last_change":"2021-03-15T22:32:35.293899+0000","last_active":"2021-03-15T22:32:35.293754+0000","last_peered":"2021-03-15T22:32:35.293754+0000","last_clean":"2021-03-15T22:32:35.293754+0000","last_became_active":"2021-03-15T22:28:59.276888+0000","last_became_peered":"2021-03-15T22:28:59.276888+0000","last_unstale":"2021-03-15T22:32:35.293754+0000","last_undegraded":"2021-03-15T22:32:35.293754+0000","last_fullsized":"2021-03-15T22:32:35.293754+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'2007","last_scrub_stamp":"2021-03-15T22:31:02.050395+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.050395+0000","log_size":2425,"ondisk_log_size":2425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":69589999,"num_objects":28,"num_object_clones":21,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":725,"num_read_kb":728262,"num_write":569,"num_write_kb":236854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":205133613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"174","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.296102+0000","last_change":"2021-03-15T22:30:28.967439+0000","last_active":"2021-03-15T22:32:32.296102+0000","last_peered":"2021-03-15T22:32:32.296102+0000","last_clean":"2021-03-15T22:32:32.296102+0000","last_became_active":"2021-03-15T22:29:51.227896+0000","last_became_peered":"2021-03-15T22:29:51.227896+0000","last_unstale":"2021-03-15T22:32:32.296102+0000","last_undegraded":"2021-03-15T22:32:32.296102+0000","last_fullsized":"2021-03-15T22:32:32.296102+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:28.967356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.967356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"801","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.305872+0000","last_change":"2021-03-15T22:31:28.275169+0000","last_active":"2021-03-15T22:32:32.305872+0000","last_peered":"2021-03-15T22:32:32.305872+0000","last_clean":"2021-03-15T22:32:32.305872+0000","last_became_active":"2021-03-15T22:31:28.274813+0000","last_became_peered":"2021-03-15T22:31:28.274813+0000","last_unstale":"2021-03-15T22:32:32.305872+0000","last_undegraded":"2021-03-15T22:32:32.305872+0000","last_fullsized":"2021-03-15T22:32:32.305872+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:05.206223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:30:05.206223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"422","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.277087+0000","last_change":"2021-03-15T22:31:39.555544+0000","last_active":"2021-03-15T22:32:30.277087+0000","last_peered":"2021-03-15T22:32:30.277087+0000","last_clean":"2021-03-15T22:32:30.277087+0000","last_became_active":"2021-03-15T22:31:28.285608+0000","last_became_peered":"2021-03-15T22:31:28.285608+0000","last_unstale":"2021-03-15T22:32:30.277087+0000","last_undegraded":"2021-03-15T22:32:30.277087+0000","last_fullsized":"2021-03-15T22:32:30.277087+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:31:39.555487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"664","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.265437+0000","last_change":"2021-03-15T22:31:28.303171+0000","last_active":"2021-03-15T22:32:30.265437+0000","last_peered":"2021-03-15T22:32:30.265437+0000","last_clean":"2021-03-15T22:32:30.265437+0000","last_became_active":"2021-03-15T22:31:28.277985+0000","last_became_peered":"2021-03-15T22:31:28.277985+0000","last_unstale":"2021-03-15T22:32:30.265437+0000","last_undegraded":"2021-03-15T22:32:30.265437+0000","last_fullsized":"2021-03-15T22:32:30.265437+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:30:28.176831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.176831+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"813","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.305504+0000","last_change":"2021-03-15T22:30:32.244033+0000","last_active":"2021-03-15T22:32:32.305504+0000","last_peered":"2021-03-15T22:32:32.305504+0000","last_clean":"2021-03-15T22:32:32.305504+0000","last_became_active":"2021-03-15T22:30:32.243696+0000","last_became_peered":"2021-03-15T22:30:32.243696+0000","last_unstale":"2021-03-15T22:32:32.305504+0000","last_undegraded":"2021-03-15T22:32:32.305504+0000","last_fullsized":"2021-03-15T22:32:32.305504+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.257476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"802'705","reported_seq":"3007","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.294803+0000","last_change":"2021-03-15T22:32:33.295043+0000","last_active":"2021-03-15T22:32:33.294803+0000","last_peered":"2021-03-15T22:32:33.294803+0000","last_clean":"2021-03-15T22:32:33.294803+0000","last_became_active":"2021-03-15T22:30:35.366323+0000","last_became_peered":"2021-03-15T22:30:35.366323+0000","last_unstale":"2021-03-15T22:32:33.294803+0000","last_undegraded":"2021-03-15T22:32:33.294803+0000","last_fullsized":"2021-03-15T22:32:33.294803+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'659","last_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:32:11.467520+0000","log_size":705,"ondisk_log_size":705,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21805658,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":178,"num_read_kb":168571,"num_write":173,"num_write_kb":76419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":54764539,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"292","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"845","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.264470+0000","last_change":"2021-03-15T22:32:12.830549+0000","last_active":"2021-03-15T22:32:31.264470+0000","last_peered":"2021-03-15T22:32:31.264470+0000","last_clean":"2021-03-15T22:32:31.264470+0000","last_became_active":"2021-03-15T22:29:51.222189+0000","last_became_peered":"2021-03-15T22:29:51.222189+0000","last_unstale":"2021-03-15T22:32:31.264470+0000","last_undegraded":"2021-03-15T22:32:31.264470+0000","last_fullsized":"2021-03-15T22:32:31.264470+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"287","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310345+0000","last_change":"2021-03-15T22:31:01.010456+0000","last_active":"2021-03-15T22:32:34.310345+0000","last_peered":"2021-03-15T22:32:34.310345+0000","last_clean":"2021-03-15T22:32:34.310345+0000","last_became_active":"2021-03-15T22:28:59.276599+0000","last_became_peered":"2021-03-15T22:28:59.276599+0000","last_unstale":"2021-03-15T22:32:34.310345+0000","last_undegraded":"2021-03-15T22:32:34.310345+0000","last_fullsized":"2021-03-15T22:32:34.310345+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:01.010393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:01.010393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"1674","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.293909+0000","last_change":"2021-03-15T22:32:30.365491+0000","last_active":"2021-03-15T22:32:33.293909+0000","last_peered":"2021-03-15T22:32:33.293909+0000","last_clean":"2021-03-15T22:32:33.293909+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:32:33.293909+0000","last_undegraded":"2021-03-15T22:32:33.293909+0000","last_fullsized":"2021-03-15T22:32:33.293909+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:32:30.365432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.365432+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":864,"num_read_kb":859,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"334","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.311070+0000","last_change":"2021-03-15T22:31:11.926326+0000","last_active":"2021-03-15T22:32:34.311070+0000","last_peered":"2021-03-15T22:32:34.311070+0000","last_clean":"2021-03-15T22:32:34.311070+0000","last_became_active":"2021-03-15T22:30:32.251686+0000","last_became_peered":"2021-03-15T22:30:32.251686+0000","last_unstale":"2021-03-15T22:32:34.311070+0000","last_undegraded":"2021-03-15T22:32:34.311070+0000","last_fullsized":"2021-03-15T22:32:34.311070+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:31:11.926268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.926268+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"188","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.296199+0000","last_change":"2021-03-15T22:32:00.816143+0000","last_active":"2021-03-15T22:32:32.296199+0000","last_peered":"2021-03-15T22:32:32.296199+0000","last_clean":"2021-03-15T22:32:32.296199+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:32:32.296199+0000","last_undegraded":"2021-03-15T22:32:32.296199+0000","last_fullsized":"2021-03-15T22:32:32.296199+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:00.816090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:32:00.816090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"803'2616","reported_seq":"5304","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.318314+0000","last_change":"2021-03-15T22:32:33.309953+0000","last_active":"2021-03-15T22:32:33.318314+0000","last_peered":"2021-03-15T22:32:33.318314+0000","last_clean":"2021-03-15T22:32:33.318314+0000","last_became_active":"2021-03-15T22:29:52.225069+0000","last_became_peered":"2021-03-15T22:29:52.225069+0000","last_unstale":"2021-03-15T22:32:33.318314+0000","last_undegraded":"2021-03-15T22:32:33.318314+0000","last_fullsized":"2021-03-15T22:32:33.318314+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'2597","last_scrub_stamp":"2021-03-15T22:32:30.909661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.909661+0000","log_size":2616,"ondisk_log_size":2616,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":80085572,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":2,"num_read":963,"num_read_kb":1030472,"num_write":638,"num_write_kb":249109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":165660176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"292","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"794","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.264285+0000","last_change":"2021-03-15T22:31:28.276749+0000","last_active":"2021-03-15T22:32:31.264285+0000","last_peered":"2021-03-15T22:32:31.264285+0000","last_clean":"2021-03-15T22:32:31.264285+0000","last_became_active":"2021-03-15T22:31:28.276431+0000","last_became_peered":"2021-03-15T22:31:28.276431+0000","last_unstale":"2021-03-15T22:32:31.264285+0000","last_undegraded":"2021-03-15T22:32:31.264285+0000","last_fullsized":"2021-03-15T22:32:31.264285+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:02.160813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.160813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"825","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.264518+0000","last_change":"2021-03-15T22:31:12.070899+0000","last_active":"2021-03-15T22:32:31.264518+0000","last_peered":"2021-03-15T22:32:31.264518+0000","last_clean":"2021-03-15T22:32:31.264518+0000","last_became_active":"2021-03-15T22:30:32.243790+0000","last_became_peered":"2021-03-15T22:30:32.243790+0000","last_unstale":"2021-03-15T22:32:31.264518+0000","last_undegraded":"2021-03-15T22:32:31.264518+0000","last_fullsized":"2021-03-15T22:32:31.264518+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:12.070843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:12.070843+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"346","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310617+0000","last_change":"2021-03-15T22:31:40.359761+0000","last_active":"2021-03-15T22:32:34.310617+0000","last_peered":"2021-03-15T22:32:34.310617+0000","last_clean":"2021-03-15T22:32:34.310617+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:32:34.310617+0000","last_undegraded":"2021-03-15T22:32:34.310617+0000","last_fullsized":"2021-03-15T22:32:34.310617+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:31:40.359701+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:31:40.359701+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"730","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310233+0000","last_change":"2021-03-15T22:32:07.632290+0000","last_active":"2021-03-15T22:32:34.310233+0000","last_peered":"2021-03-15T22:32:34.310233+0000","last_clean":"2021-03-15T22:32:34.310233+0000","last_became_active":"2021-03-15T22:31:28.278813+0000","last_became_peered":"2021-03-15T22:31:28.278813+0000","last_unstale":"2021-03-15T22:32:34.310233+0000","last_undegraded":"2021-03-15T22:32:34.310233+0000","last_fullsized":"2021-03-15T22:32:34.310233+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:07.632227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:32:07.632227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"800'417","reported_seq":"2802","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.290935+0000","last_change":"2021-03-15T22:32:30.290935+0000","last_active":"2021-03-15T22:32:30.290935+0000","last_peered":"2021-03-15T22:32:30.290935+0000","last_clean":"2021-03-15T22:32:30.290935+0000","last_became_active":"2021-03-15T22:31:33.250635+0000","last_became_peered":"2021-03-15T22:31:33.250635+0000","last_unstale":"2021-03-15T22:32:30.290935+0000","last_undegraded":"2021-03-15T22:32:30.290935+0000","last_fullsized":"2021-03-15T22:32:30.290935+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'353","last_scrub_stamp":"2021-03-15T22:31:02.426448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.426448+0000","log_size":417,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10642824,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":140,"num_read_kb":175517,"num_write":94,"num_write_kb":34134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":51449003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"292","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"300","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310191+0000","last_change":"2021-03-15T22:32:10.674869+0000","last_active":"2021-03-15T22:32:34.310191+0000","last_peered":"2021-03-15T22:32:34.310191+0000","last_clean":"2021-03-15T22:32:34.310191+0000","last_became_active":"2021-03-15T22:30:32.237925+0000","last_became_peered":"2021-03-15T22:30:32.237925+0000","last_unstale":"2021-03-15T22:32:34.310191+0000","last_undegraded":"2021-03-15T22:32:34.310191+0000","last_fullsized":"2021-03-15T22:32:34.310191+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"740","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.884013+0000","last_change":"2021-03-15T22:32:33.884013+0000","last_active":"2021-03-15T22:32:33.884013+0000","last_peered":"2021-03-15T22:32:33.884013+0000","last_clean":"2021-03-15T22:32:33.884013+0000","last_became_active":"2021-03-15T22:30:32.257328+0000","last_became_peered":"2021-03-15T22:30:32.257328+0000","last_unstale":"2021-03-15T22:32:33.884013+0000","last_undegraded":"2021-03-15T22:32:33.884013+0000","last_fullsized":"2021-03-15T22:32:33.884013+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.883949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.883949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"1015","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.747681+0000","last_change":"2021-03-15T22:32:31.747649+0000","last_active":"2021-03-15T22:32:31.747681+0000","last_peered":"2021-03-15T22:32:31.747681+0000","last_clean":"2021-03-15T22:32:31.747681+0000","last_became_active":"2021-03-15T22:29:51.222356+0000","last_became_peered":"2021-03-15T22:29:51.222356+0000","last_unstale":"2021-03-15T22:32:31.747681+0000","last_undegraded":"2021-03-15T22:32:31.747681+0000","last_fullsized":"2021-03-15T22:32:31.747681+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:32:31.747582+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.747582+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"741'10","reported_seq":"368","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.310550+0000","last_change":"2021-03-15T22:31:28.389370+0000","last_active":"2021-03-15T22:32:34.310550+0000","last_peered":"2021-03-15T22:32:34.310550+0000","last_clean":"2021-03-15T22:32:34.310550+0000","last_became_active":"2021-03-15T22:31:28.280972+0000","last_became_peered":"2021-03-15T22:31:28.280972+0000","last_unstale":"2021-03-15T22:32:34.310550+0000","last_undegraded":"2021-03-15T22:32:34.310550+0000","last_fullsized":"2021-03-15T22:32:34.310550+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'9","last_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.971303+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"800'693","reported_seq":"2908","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.431795+0000","last_change":"2021-03-15T22:32:30.431795+0000","last_active":"2021-03-15T22:32:30.431795+0000","last_peered":"2021-03-15T22:32:30.431795+0000","last_clean":"2021-03-15T22:32:30.431795+0000","last_became_active":"2021-03-15T22:30:34.297623+0000","last_became_peered":"2021-03-15T22:30:34.297623+0000","last_unstale":"2021-03-15T22:32:30.431795+0000","last_undegraded":"2021-03-15T22:32:30.431795+0000","last_fullsized":"2021-03-15T22:32:30.431795+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'591","last_scrub_stamp":"2021-03-15T22:31:35.474935+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:31:35.474935+0000","log_size":693,"ondisk_log_size":693,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22692286,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":219,"num_read_kb":187436,"num_write":192,"num_write_kb":71173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":47935099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"292","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"787","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.305193+0000","last_change":"2021-03-15T22:31:28.282929+0000","last_active":"2021-03-15T22:32:32.305193+0000","last_peered":"2021-03-15T22:32:32.305193+0000","last_clean":"2021-03-15T22:32:32.305193+0000","last_became_active":"2021-03-15T22:31:28.282600+0000","last_became_peered":"2021-03-15T22:31:28.282600+0000","last_unstale":"2021-03-15T22:32:32.305193+0000","last_undegraded":"2021-03-15T22:32:32.305193+0000","last_fullsized":"2021-03-15T22:32:32.305193+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:06.530069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:06.530069+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"266","reported_epoch":"804","state":"active+clean","last_fresh":"2021-03-15T22:32:34.311181+0000","last_change":"2021-03-15T22:30:37.798454+0000","last_active":"2021-03-15T22:32:34.311181+0000","last_peered":"2021-03-15T22:32:34.311181+0000","last_clean":"2021-03-15T22:32:34.311181+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:32:34.311181+0000","last_undegraded":"2021-03-15T22:32:34.311181+0000","last_fullsized":"2021-03-15T22:32:34.311181+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.798388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"951","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.264312+0000","last_change":"2021-03-15T22:31:29.720965+0000","last_active":"2021-03-15T22:32:30.264312+0000","last_peered":"2021-03-15T22:32:30.264312+0000","last_clean":"2021-03-15T22:32:30.264312+0000","last_became_active":"2021-03-15T22:31:28.279662+0000","last_became_peered":"2021-03-15T22:31:28.279662+0000","last_unstale":"2021-03-15T22:32:30.264312+0000","last_undegraded":"2021-03-15T22:32:30.264312+0000","last_fullsized":"2021-03-15T22:32:30.264312+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:30:42.383512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.383512+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"271","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.269846+0000","last_change":"2021-03-15T22:30:27.472423+0000","last_active":"2021-03-15T22:32:30.269846+0000","last_peered":"2021-03-15T22:32:30.269846+0000","last_clean":"2021-03-15T22:32:30.269846+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:32:30.269846+0000","last_undegraded":"2021-03-15T22:32:30.269846+0000","last_fullsized":"2021-03-15T22:32:30.269846+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:30:27.472320+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:30:27.472320+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"801'1189","reported_seq":"3610","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.272985+0000","last_change":"2021-03-15T22:32:30.298461+0000","last_active":"2021-03-15T22:32:31.272985+0000","last_peered":"2021-03-15T22:32:31.272985+0000","last_clean":"2021-03-15T22:32:31.272985+0000","last_became_active":"2021-03-15T22:31:31.245546+0000","last_became_peered":"2021-03-15T22:31:31.245546+0000","last_unstale":"2021-03-15T22:32:31.272985+0000","last_undegraded":"2021-03-15T22:32:31.272985+0000","last_fullsized":"2021-03-15T22:32:31.272985+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'889","last_scrub_stamp":"2021-03-15T22:30:29.983045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.983045+0000","log_size":1189,"ondisk_log_size":1189,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36611565,"num_objects":18,"num_object_clones":14,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":438,"num_read_kb":453225,"num_write":276,"num_write_kb":96110,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":139702636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"292","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"828","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.261469+0000","last_change":"2021-03-15T22:30:32.262270+0000","last_active":"2021-03-15T22:32:30.261469+0000","last_peered":"2021-03-15T22:32:30.261469+0000","last_clean":"2021-03-15T22:32:30.261469+0000","last_became_active":"2021-03-15T22:30:32.261995+0000","last_became_peered":"2021-03-15T22:30:32.261995+0000","last_unstale":"2021-03-15T22:32:30.261469+0000","last_undegraded":"2021-03-15T22:32:30.261469+0000","last_fullsized":"2021-03-15T22:32:30.261469+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:09.101167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:09.101167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"580","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.278607+0000","last_change":"2021-03-15T22:31:36.472708+0000","last_active":"2021-03-15T22:32:30.278607+0000","last_peered":"2021-03-15T22:32:30.278607+0000","last_clean":"2021-03-15T22:32:30.278607+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:32:30.278607+0000","last_undegraded":"2021-03-15T22:32:30.278607+0000","last_fullsized":"2021-03-15T22:32:30.278607+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:36.472646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:31:36.472646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"256","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.269956+0000","last_change":"2021-03-15T22:32:12.653877+0000","last_active":"2021-03-15T22:32:30.269956+0000","last_peered":"2021-03-15T22:32:30.269956+0000","last_clean":"2021-03-15T22:32:30.269956+0000","last_became_active":"2021-03-15T22:28:59.276755+0000","last_became_peered":"2021-03-15T22:28:59.276755+0000","last_unstale":"2021-03-15T22:32:30.269956+0000","last_undegraded":"2021-03-15T22:32:30.269956+0000","last_fullsized":"2021-03-15T22:32:30.269956+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:32:12.653809+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.653809+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"823","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.263843+0000","last_change":"2021-03-15T22:31:04.495639+0000","last_active":"2021-03-15T22:32:30.263843+0000","last_peered":"2021-03-15T22:32:30.263843+0000","last_clean":"2021-03-15T22:32:30.263843+0000","last_became_active":"2021-03-15T22:28:59.266487+0000","last_became_peered":"2021-03-15T22:28:59.266487+0000","last_unstale":"2021-03-15T22:32:30.263843+0000","last_undegraded":"2021-03-15T22:32:30.263843+0000","last_fullsized":"2021-03-15T22:32:30.263843+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.495574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.495574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"800'1087","reported_seq":"3420","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.295450+0000","last_change":"2021-03-15T22:32:30.295450+0000","last_active":"2021-03-15T22:32:30.295450+0000","last_peered":"2021-03-15T22:32:30.295450+0000","last_clean":"2021-03-15T22:32:30.295450+0000","last_became_active":"2021-03-15T22:31:31.244986+0000","last_became_peered":"2021-03-15T22:31:31.244986+0000","last_unstale":"2021-03-15T22:32:30.295450+0000","last_undegraded":"2021-03-15T22:32:30.295450+0000","last_fullsized":"2021-03-15T22:32:30.295450+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1034","last_scrub_stamp":"2021-03-15T22:32:09.615397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:09.615397+0000","log_size":1087,"ondisk_log_size":1087,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":28264803,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":313,"num_read_kb":284494,"num_write":252,"num_write_kb":85320,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":50339417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"292","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"725","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.295926+0000","last_change":"2021-03-15T22:31:04.380476+0000","last_active":"2021-03-15T22:32:32.295926+0000","last_peered":"2021-03-15T22:32:32.295926+0000","last_clean":"2021-03-15T22:32:32.295926+0000","last_became_active":"2021-03-15T22:29:52.223342+0000","last_became_peered":"2021-03-15T22:29:52.223342+0000","last_unstale":"2021-03-15T22:32:32.295926+0000","last_undegraded":"2021-03-15T22:32:32.295926+0000","last_fullsized":"2021-03-15T22:32:32.295926+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.380417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.380417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1055","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.262917+0000","last_change":"2021-03-15T22:32:08.071278+0000","last_active":"2021-03-15T22:32:30.262917+0000","last_peered":"2021-03-15T22:32:30.262917+0000","last_clean":"2021-03-15T22:32:30.262917+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:32:30.262917+0000","last_undegraded":"2021-03-15T22:32:30.262917+0000","last_fullsized":"2021-03-15T22:32:30.262917+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:32:08.071210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:08.071210+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"225","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.265449+0000","last_change":"2021-03-15T22:30:09.506976+0000","last_active":"2021-03-15T22:32:30.265449+0000","last_peered":"2021-03-15T22:32:30.265449+0000","last_clean":"2021-03-15T22:32:30.265449+0000","last_became_active":"2021-03-15T22:28:59.274186+0000","last_became_peered":"2021-03-15T22:28:59.274186+0000","last_unstale":"2021-03-15T22:32:30.265449+0000","last_undegraded":"2021-03-15T22:32:30.265449+0000","last_fullsized":"2021-03-15T22:32:30.265449+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:09.506914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:30:09.506914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"655","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.323895+0000","last_change":"2021-03-15T22:31:02.092287+0000","last_active":"2021-03-15T22:32:33.323895+0000","last_peered":"2021-03-15T22:32:33.323895+0000","last_clean":"2021-03-15T22:32:33.323895+0000","last_became_active":"2021-03-15T22:30:32.262267+0000","last_became_peered":"2021-03-15T22:30:32.262267+0000","last_unstale":"2021-03-15T22:32:33.323895+0000","last_undegraded":"2021-03-15T22:32:33.323895+0000","last_fullsized":"2021-03-15T22:32:33.323895+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:31:02.092217+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.092217+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"800'330","reported_seq":"2588","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.289242+0000","last_change":"2021-03-15T22:32:32.297934+0000","last_active":"2021-03-15T22:32:32.289242+0000","last_peered":"2021-03-15T22:32:32.289242+0000","last_clean":"2021-03-15T22:32:32.289242+0000","last_became_active":"2021-03-15T22:31:31.242099+0000","last_became_peered":"2021-03-15T22:31:31.242099+0000","last_unstale":"2021-03-15T22:32:32.289242+0000","last_undegraded":"2021-03-15T22:32:32.289242+0000","last_fullsized":"2021-03-15T22:32:32.289242+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'322","last_scrub_stamp":"2021-03-15T22:32:11.867811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:11.867811+0000","log_size":330,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9950305,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":114007,"num_write":64,"num_write_kb":28231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":21427429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"292","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"523","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.267363+0000","last_change":"2021-03-15T22:31:28.290105+0000","last_active":"2021-03-15T22:32:30.267363+0000","last_peered":"2021-03-15T22:32:30.267363+0000","last_clean":"2021-03-15T22:32:30.267363+0000","last_became_active":"2021-03-15T22:31:28.289518+0000","last_became_peered":"2021-03-15T22:31:28.289518+0000","last_unstale":"2021-03-15T22:32:30.267363+0000","last_undegraded":"2021-03-15T22:32:30.267363+0000","last_fullsized":"2021-03-15T22:32:30.267363+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:40.500803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.500803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"811","reported_epoch":"802","state":"active+clean","last_fresh":"2021-03-15T22:32:32.296322+0000","last_change":"2021-03-15T22:31:31.241376+0000","last_active":"2021-03-15T22:32:32.296322+0000","last_peered":"2021-03-15T22:32:32.296322+0000","last_clean":"2021-03-15T22:32:32.296322+0000","last_became_active":"2021-03-15T22:31:31.241017+0000","last_became_peered":"2021-03-15T22:31:31.241017+0000","last_unstale":"2021-03-15T22:32:32.296322+0000","last_undegraded":"2021-03-15T22:32:32.296322+0000","last_fullsized":"2021-03-15T22:32:32.296322+0000","mapping_epoch":740,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"618'7","last_scrub_stamp":"2021-03-15T22:29:40.964753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.964753+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"787","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.263947+0000","last_change":"2021-03-15T22:30:34.363233+0000","last_active":"2021-03-15T22:32:30.263947+0000","last_peered":"2021-03-15T22:32:30.263947+0000","last_clean":"2021-03-15T22:32:30.263947+0000","last_became_active":"2021-03-15T22:30:31.239376+0000","last_became_peered":"2021-03-15T22:30:31.239376+0000","last_unstale":"2021-03-15T22:32:30.263947+0000","last_undegraded":"2021-03-15T22:32:30.263947+0000","last_fullsized":"2021-03-15T22:32:30.263947+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.363177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"904","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.268247+0000","last_change":"2021-03-15T22:31:28.315261+0000","last_active":"2021-03-15T22:32:30.268247+0000","last_peered":"2021-03-15T22:32:30.268247+0000","last_clean":"2021-03-15T22:32:30.268247+0000","last_became_active":"2021-03-15T22:31:28.281662+0000","last_became_peered":"2021-03-15T22:31:28.281662+0000","last_unstale":"2021-03-15T22:32:30.268247+0000","last_undegraded":"2021-03-15T22:32:30.268247+0000","last_fullsized":"2021-03-15T22:32:30.268247+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:30:04.058167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:04.058167+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"802'357","reported_seq":"2066","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.324511+0000","last_change":"2021-03-15T22:32:33.324511+0000","last_active":"2021-03-15T22:32:33.324511+0000","last_peered":"2021-03-15T22:32:33.324511+0000","last_clean":"2021-03-15T22:32:33.324511+0000","last_became_active":"2021-03-15T22:30:32.257058+0000","last_became_peered":"2021-03-15T22:30:32.257058+0000","last_unstale":"2021-03-15T22:32:33.324511+0000","last_undegraded":"2021-03-15T22:32:33.324511+0000","last_fullsized":"2021-03-15T22:32:33.324511+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'291","last_scrub_stamp":"2021-03-15T22:31:30.979407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:31:30.979407+0000","log_size":357,"ondisk_log_size":357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17464712,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":152,"num_read_kb":164360,"num_write":100,"num_write_kb":31214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":32799803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"292","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"76","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.323809+0000","last_change":"2021-03-15T22:31:28.287568+0000","last_active":"2021-03-15T22:32:33.323809+0000","last_peered":"2021-03-15T22:32:33.323809+0000","last_clean":"2021-03-15T22:32:33.323809+0000","last_became_active":"2021-03-15T22:31:28.287218+0000","last_became_peered":"2021-03-15T22:31:28.287218+0000","last_unstale":"2021-03-15T22:32:33.323809+0000","last_undegraded":"2021-03-15T22:32:33.323809+0000","last_fullsized":"2021-03-15T22:32:33.323809+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.283355+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.283355+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"830","reported_epoch":"801","state":"active+clean","last_fresh":"2021-03-15T22:32:31.264652+0000","last_change":"2021-03-15T22:32:05.880915+0000","last_active":"2021-03-15T22:32:31.264652+0000","last_peered":"2021-03-15T22:32:31.264652+0000","last_clean":"2021-03-15T22:32:31.264652+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:32:31.264652+0000","last_undegraded":"2021-03-15T22:32:31.264652+0000","last_fullsized":"2021-03-15T22:32:31.264652+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:05.880846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.880846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"75","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.261561+0000","last_change":"2021-03-15T22:31:28.287806+0000","last_active":"2021-03-15T22:32:30.261561+0000","last_peered":"2021-03-15T22:32:30.261561+0000","last_clean":"2021-03-15T22:32:30.261561+0000","last_became_active":"2021-03-15T22:31:28.287437+0000","last_became_peered":"2021-03-15T22:31:28.287437+0000","last_unstale":"2021-03-15T22:32:30.261561+0000","last_undegraded":"2021-03-15T22:32:30.261561+0000","last_fullsized":"2021-03-15T22:32:30.261561+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:43.259585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:43.259585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"871","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.265568+0000","last_change":"2021-03-15T22:30:38.419647+0000","last_active":"2021-03-15T22:32:30.265568+0000","last_peered":"2021-03-15T22:32:30.265568+0000","last_clean":"2021-03-15T22:32:30.265568+0000","last_became_active":"2021-03-15T22:30:31.239360+0000","last_became_peered":"2021-03-15T22:30:31.239360+0000","last_unstale":"2021-03-15T22:32:30.265568+0000","last_undegraded":"2021-03-15T22:32:30.265568+0000","last_fullsized":"2021-03-15T22:32:30.265568+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:30:38.419551+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.419551+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"805'1236","reported_seq":"3801","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.286225+0000","last_change":"2021-03-15T22:32:35.286442+0000","last_active":"2021-03-15T22:32:35.286225+0000","last_peered":"2021-03-15T22:32:35.286225+0000","last_clean":"2021-03-15T22:32:35.286225+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:32:35.286225+0000","last_undegraded":"2021-03-15T22:32:35.286225+0000","last_fullsized":"2021-03-15T22:32:35.286225+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'877","last_scrub_stamp":"2021-03-15T22:30:27.732142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:27.732142+0000","log_size":1236,"ondisk_log_size":1236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51063540,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":453,"num_read_kb":471398,"num_write":306,"num_write_kb":110299,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"779","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.268229+0000","last_change":"2021-03-15T22:30:36.417325+0000","last_active":"2021-03-15T22:32:30.268229+0000","last_peered":"2021-03-15T22:32:30.268229+0000","last_clean":"2021-03-15T22:32:30.268229+0000","last_became_active":"2021-03-15T22:30:31.243269+0000","last_became_peered":"2021-03-15T22:30:31.243269+0000","last_unstale":"2021-03-15T22:32:30.268229+0000","last_undegraded":"2021-03-15T22:32:30.268229+0000","last_fullsized":"2021-03-15T22:32:30.268229+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"732'13","reported_seq":"748","reported_epoch":"803","state":"active+clean","last_fresh":"2021-03-15T22:32:33.323913+0000","last_change":"2021-03-15T22:32:00.927902+0000","last_active":"2021-03-15T22:32:33.323913+0000","last_peered":"2021-03-15T22:32:33.323913+0000","last_clean":"2021-03-15T22:32:33.323913+0000","last_became_active":"2021-03-15T22:31:18.249734+0000","last_became_peered":"2021-03-15T22:31:18.249734+0000","last_unstale":"2021-03-15T22:32:33.323913+0000","last_undegraded":"2021-03-15T22:32:33.323913+0000","last_fullsized":"2021-03-15T22:32:33.323913+0000","mapping_epoch":682,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'13","last_scrub_stamp":"2021-03-15T22:32:00.927837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:00.927837+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"225","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.265205+0000","last_change":"2021-03-15T22:30:03.567594+0000","last_active":"2021-03-15T22:32:30.265205+0000","last_peered":"2021-03-15T22:32:30.265205+0000","last_clean":"2021-03-15T22:32:30.265205+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:32:30.265205+0000","last_undegraded":"2021-03-15T22:32:30.265205+0000","last_fullsized":"2021-03-15T22:32:30.265205+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.567529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"951","reported_epoch":"800","state":"active+clean","last_fresh":"2021-03-15T22:32:30.266371+0000","last_change":"2021-03-15T22:31:28.300885+0000","last_active":"2021-03-15T22:32:30.266371+0000","last_peered":"2021-03-15T22:32:30.266371+0000","last_clean":"2021-03-15T22:32:30.266371+0000","last_became_active":"2021-03-15T22:31:28.280812+0000","last_became_peered":"2021-03-15T22:31:28.280812+0000","last_unstale":"2021-03-15T22:32:30.266371+0000","last_undegraded":"2021-03-15T22:32:30.266371+0000","last_fullsized":"2021-03-15T22:32:30.266371+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:30:00.029954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:00.029954+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":539393306,"num_objects":247,"num_object_clones":197,"num_object_copies":741,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":247,"num_whiteouts":15,"num_read":5644,"num_read_kb":5750882,"num_write":4143,"num_write_kb":1548585,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2360,"num_bytes_recovered":1223461680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1294970880,"data_stored":1826968451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16693,"ondisk_log_size":16693,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":864,"num_read_kb":859,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1483,"num_read_kb":988,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":68742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2228224,"data_stored":80200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":938,"ondisk_log_size":938,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":115,"ondisk_log_size":115,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299572,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":189068,"kb_used_data":150724,"kb_used_omap":0,"kb_used_meta":38336,"kb_avail":93556084,"statfs":{"total":95995035648,"available":95801430016,"internally_reserved":0,"allocated":154341376,"data_stored":211968410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39256064},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.44500000000000001,"15min":0.47599999999999998},"min":{"1min":0.371,"5min":0.216,"15min":0.215},"max":{"1min":0.70099999999999996,"5min":0.93999999999999995,"15min":2.73},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.47999999999999998,"15min":0.496},"min":{"1min":0.33300000000000002,"5min":0.251,"15min":0.222},"max":{"1min":0.71599999999999997,"5min":0.75800000000000001,"15min":3.8799999999999999},"last":0.44700000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.44800000000000001,"15min":0.47999999999999998},"min":{"1min":0.30199999999999999,"5min":0.27300000000000002,"15min":0.22600000000000001},"max":{"1min":0.73899999999999999,"5min":0.91300000000000003,"15min":2.6859999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.47399999999999998,"15min":0.49299999999999999},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.23400000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":2.0779999999999998},"last":0.52000000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:32:34 2021","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.49299999999999999,"15min":0.47999999999999998},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.505,"15min":0.503},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.753,"5min":1.119,"15min":1.119},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.501,"15min":0.50800000000000001},"min":{"1min":0.379,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79200000000000004,"5min":1.4299999999999999,"15min":2.0339999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.751,"5min":5.415,"15min":5.415},"last":0.435}]},{"osd":4,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.32600000000000001,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":0.753,"5min":0.753,"15min":1.927},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.43099999999999999,"15min":0.46300000000000002},"min":{"1min":0.28499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.68200000000000005,"5min":0.72399999999999998,"15min":2.5339999999999998},"last":0.39000000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:32:34 2021","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.44900000000000001},"min":{"1min":0.40200000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.499},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.41899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76900000000000002,"5min":1.071,"15min":1.071},"last":0.41899999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.46899999999999997,"15min":0.51700000000000002},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.48499999999999999,"15min":0.51800000000000002},"min":{"1min":0.36499999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77500000000000002,"5min":1.097,"15min":1.8560000000000001},"last":0.52800000000000002}]}]},{"osd":6,"up_from":0,"seq":193273528504,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575049,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":222412,"kb_used_data":175748,"kb_used_omap":0,"kb_used_meta":46656,"kb_avail":93522740,"statfs":{"total":95995035648,"available":95767285760,"internally_reserved":0,"allocated":179965952,"data_stored":230293490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47775744},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:31:43 2021","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.51400000000000001},"min":{"1min":0.20200000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.52300000000000002,"5min":1.325,"15min":54.427999999999997},"last":0.253},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.35099999999999998,"15min":0.54000000000000004},"min":{"1min":0.23000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.3500000000000001,"5min":3.3500000000000001,"15min":54.356999999999999},"last":0.32500000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:31:34 2021","interfaces":[{"interface":"back","average":{"1min":0.33400000000000002,"5min":0.33400000000000002,"15min":0.33400000000000002},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.51500000000000001,"5min":0.51500000000000001,"15min":0.51500000000000001},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.57599999999999996,"5min":0.57599999999999996,"15min":0.57599999999999996},"last":0.34999999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:32:13 2021","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.35699999999999998,"15min":0.38100000000000001},"min":{"1min":0.27200000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.647,"5min":1.647,"15min":1.796},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":3.2919999999999998,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.379}]},{"osd":4,"last update":"Mon Mar 15 22:32:18 2021","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.38700000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.0330000000000004,"5min":4.0330000000000004,"15min":4.0330000000000004},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.58299999999999996},"min":{"1min":0.35699999999999998,"5min":0.27200000000000002,"15min":0.253},"max":{"1min":0.78600000000000003,"5min":4.1790000000000003,"15min":4.6139999999999999},"last":0.41099999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:32:28 2021","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85599999999999998,"5min":1.226,"15min":1.226},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.55900000000000005,"15min":0.51200000000000001},"min":{"1min":0.45700000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Mar 15 22:31:55 2021","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.56399999999999995,"15min":0.59999999999999998},"min":{"1min":0.441,"5min":0.314,"15min":0.314},"max":{"1min":3.4209999999999998,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.59099999999999997},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.56000000000000005,"15min":0.58299999999999996},"min":{"1min":0.375,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":4.1210000000000004,"5min":4.1210000000000004,"15min":4.1210000000000004},"last":0.47999999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:32:13 2021","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.56599999999999995,"15min":0.65700000000000003},"min":{"1min":0.46800000000000003,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":8.8100000000000005},"last":0.624},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.57699999999999996,"15min":0.58699999999999997},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.35799999999999998},"max":{"1min":4.0860000000000003,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":0.44400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738572,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":218740,"kb_used_data":180140,"kb_used_omap":0,"kb_used_meta":38592,"kb_avail":93526412,"statfs":{"total":95995035648,"available":95771045888,"internally_reserved":0,"allocated":184463360,"data_stored":264292677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39518208},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:31:37 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32200000000000001,"15min":0.34000000000000002},"min":{"1min":0.23400000000000001,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":0.41599999999999998,"5min":1.024,"15min":1.7070000000000001},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.34100000000000003,"5min":0.34599999999999997,"15min":0.36299999999999999},"min":{"1min":0.23400000000000001,"5min":0.20699999999999999,"15min":0.19900000000000001},"max":{"1min":0.442,"5min":1.1950000000000001,"15min":1.1950000000000001},"last":0.377}]},{"osd":2,"last update":"Mon Mar 15 22:32:34 2021","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.375,"15min":0.36499999999999999},"min":{"1min":0.24099999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.33600000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.40200000000000002,"15min":0.38400000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:32:17 2021","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.39700000000000002,"15min":0.38700000000000001},"min":{"1min":0.27800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.41399999999999998,"15min":0.40699999999999997},"min":{"1min":0.254,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.589,"5min":2.589,"15min":2.589},"last":0.46400000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:32:17 2021","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.54900000000000004,"15min":0.57299999999999995},"min":{"1min":0.33000000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.645},"last":0.433}]},{"osd":5,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.54800000000000004,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.52100000000000002,"15min":0.48899999999999999},"min":{"1min":0.38900000000000001,"5min":0.318,"15min":0.318},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.45300000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:31:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.505,"15min":0.55300000000000005},"min":{"1min":0.36399999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.79800000000000004,"5min":0.84999999999999998,"15min":1.6180000000000001},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.51700000000000002,"15min":0.57699999999999996},"min":{"1min":0.40500000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76900000000000002,"5min":0.90100000000000002,"15min":1.0049999999999999},"last":0.47699999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:32:17 2021","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.443,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.6190000000000002,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.54200000000000004,"15min":0.57599999999999996},"min":{"1min":0.45200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.48599999999999999}]}]},{"osd":2,"up_from":681,"seq":2924872728602,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":203180,"kb_used_data":186788,"kb_used_omap":119,"kb_used_meta":16264,"kb_avail":93541972,"statfs":{"total":95995035648,"available":95786979328,"internally_reserved":0,"allocated":191270912,"data_stored":259160622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":16654697},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.307,"5min":0.307,"15min":0.307},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.40500000000000003,"5min":0.40500000000000003,"15min":0.40500000000000003},"last":0.44},{"interface":"front","average":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.32000000000000001},"min":{"1min":0.20100000000000001,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.33800000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"min":{"1min":0.26000000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"last":0.41899999999999998},{"interface":"front","average":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.47199999999999998,"5min":0.47199999999999998,"15min":0.47199999999999998},"last":0.379}]},{"osd":3,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.432,"5min":0.432,"15min":0.432},"last":0.432},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.49299999999999999,"5min":0.49299999999999999,"15min":0.49299999999999999},"last":0.40899999999999997}]},{"osd":4,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.46800000000000003,"15min":0.46800000000000003},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.61799999999999999,"5min":0.61799999999999999,"15min":0.61799999999999999},"last":0.82099999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.47999999999999998,"15min":0.47999999999999998},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.63200000000000001,"5min":0.63200000000000001,"15min":0.63200000000000001},"last":0.85599999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.496,"5min":0.496,"15min":0.496},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.65500000000000003},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.79500000000000004}]},{"osd":6,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"min":{"1min":0.33500000000000002,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.499,"5min":0.499,"15min":0.499},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.626,"5min":0.626,"15min":0.626},"last":0.83299999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:31:33 2021","interfaces":[{"interface":"back","average":{"1min":0.51800000000000002,"5min":0.51800000000000002,"15min":0.51800000000000002},"min":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.67300000000000004,"5min":0.67300000000000004,"15min":0.67300000000000004},"last":0.65500000000000003},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.41299999999999998,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":0.63900000000000001,"5min":0.63900000000000001,"15min":0.63900000000000001},"last":0.753}]}]},{"osd":3,"up_from":26,"seq":111669149889,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":345600,"kb_used_data":284664,"kb_used_omap":0,"kb_used_meta":60928,"kb_avail":93399552,"statfs":{"total":95995035648,"available":95641141248,"internally_reserved":0,"allocated":291495936,"data_stored":389418414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62390272},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32000000000000001,"15min":0.33300000000000002},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.17699999999999999},"max":{"1min":0.46700000000000003,"5min":1.1499999999999999,"15min":2.625},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.309,"5min":0.31,"15min":0.33900000000000002},"min":{"1min":0.20000000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.48499999999999999,"5min":1.419,"15min":2.6339999999999999},"last":0.253}]},{"osd":1,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.39500000000000002},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":0.49199999999999999,"5min":0.72599999999999998,"15min":6.0199999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.36299999999999999,"15min":0.39900000000000002},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.498,"5min":2.5750000000000002,"15min":7.6130000000000004},"last":0.36899999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:31:32 2021","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.309},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.29299999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:32:26 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.86899999999999999,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.497,"15min":0.52800000000000002},"min":{"1min":0.34599999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.82699999999999996,"5min":2.1059999999999999,"15min":2.2890000000000001},"last":0.41599999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:32:26 2021","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.73399999999999999,"15min":0.82199999999999995},"min":{"1min":0.311,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75800000000000001,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.91500000000000004,"15min":1.083},"min":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.79400000000000004,"5min":15.776,"15min":15.776},"last":0.38600000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:31:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":0.80000000000000004,"5min":2.5299999999999998,"15min":2.6499999999999999},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55800000000000005,"15min":0.56699999999999995},"min":{"1min":0.40799999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":3.472,"15min":3.472},"last":0.44900000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.57999999999999996},"min":{"1min":0.38600000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81100000000000005,"5min":5.625,"15min":5.625},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.56000000000000005},"min":{"1min":0.371,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78800000000000003,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":0.46600000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986366,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":135812,"kb_used_data":90748,"kb_used_omap":0,"kb_used_meta":45056,"kb_avail":93609340,"statfs":{"total":95995035648,"available":95855964160,"internally_reserved":0,"allocated":92925952,"data_stored":127578403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46137344},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.48499999999999999},"min":{"1min":0.24199999999999999,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":0.73199999999999998,"5min":4.7050000000000001,"15min":4.7050000000000001},"last":0.442},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54300000000000004,"15min":0.503},"min":{"1min":0.35799999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.65500000000000003,"5min":5.0209999999999999,"15min":5.0209999999999999},"last":0.503}]},{"osd":1,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.505,"15min":0.48499999999999999},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.23599999999999999},"max":{"1min":0.67900000000000005,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.28499999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.86599999999999999,"5min":4.718,"15min":4.718},"last":0.51800000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:31:31 2021","interfaces":[{"interface":"back","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.78000000000000003,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.5,"5min":0.5,"15min":0.5},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.53100000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.58899999999999997,"15min":0.55000000000000004},"min":{"1min":0.43099999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.88300000000000001,"5min":5.0499999999999998,"15min":5.0499999999999998},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89500000000000002,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.496}]},{"osd":5,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.42299999999999999,"15min":0.40899999999999997},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.44900000000000001,"15min":0.42699999999999999},"min":{"1min":0.30399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66000000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.30399999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:31:56 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.46100000000000002,"15min":0.46000000000000002},"min":{"1min":0.376,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67200000000000004,"5min":4.3079999999999998,"15min":4.3079999999999998},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.371,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72099999999999997,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.40100000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:32:14 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.32100000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79900000000000004,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.378},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56399999999999995,"15min":0.54700000000000004},"min":{"1min":0.35799999999999998,"5min":0.25,"15min":0.25},"max":{"1min":1.034,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":0.33300000000000002}]}]},{"osd":5,"up_from":616,"seq":2645699854375,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":158920,"kb_used_data":141504,"kb_used_omap":153,"kb_used_meta":17254,"kb_avail":93586232,"statfs":{"total":95995035648,"available":95832301568,"internally_reserved":0,"allocated":144900096,"data_stored":193392338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":17668625},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46400000000000002,"15min":0.45600000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65500000000000003,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.442,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.67700000000000005,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.46400000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46400000000000002,"15min":0.45200000000000001},"min":{"1min":0.312,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":0.876,"15min":0.876},"last":0.61199999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.622,"5min":0.871,"15min":0.871},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.53200000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.49199999999999999,"15min":0.47599999999999998},"min":{"1min":0.39200000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.45800000000000002,"15min":0.441},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.496}]},{"osd":4,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.378},"min":{"1min":0.32300000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.33000000000000002},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.40699999999999997,"15min":0.38600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.81499999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.35199999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45800000000000002,"15min":0.44800000000000001},"min":{"1min":0.32100000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.83999999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48599999999999999,"15min":0.46600000000000003},"min":{"1min":0.39000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.41599999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45500000000000002,"15min":0.439},"min":{"1min":0.29799999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.88700000000000001,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.40100000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":10926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":12926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":12796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":11740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":7576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":174739456,"data_stored":258063885,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":170237952,"data_stored":224089238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":181690368,"data_stored":252953293,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":281800704,"data_stored":385382869,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":83238912,"data_stored":121383663,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":135307264,"data_stored":187184541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":123215872,"data_stored":192150239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":144740352,"data_stored":205760723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:32:37.583 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:32:37.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:37.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:37.608 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:37.608 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:37.702 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:37 smithi125 conmon[56739]: audit 2021-03-15T22:32: 2021-03-15T22:32:37.703 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:37 smithi125 conmon[56739]: 36.592165+0000 mgr.y (mgr.14144) 941 : audit [DBG] from='client.16413 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:32:37.703 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:37 smithi125 conmon[56739]: cluster 2021-03-15T22:32:36.795296+0000 osd.0 (osd.0) 132 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:32:37.703 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:37 smithi125 conmon[56739]: cluster 2021-03-15T22:32:37.275282+0000 mon.a (mon.0) 2092 : cluster [DBG] osdmap e807: 8 total, 8 up, 7 in 2021-03-15T22:32:37.710 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:37.731 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:37.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:37.733 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:37.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:37.858 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:37.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:37.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:37.961 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:37.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:37.985 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:37.985 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:38.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:38.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:38.111 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:38.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:38.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:38.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:38.236 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:38.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:38.278 INFO:tasks.rados.rados.0.smithi076.stdout:3024: done (1 left) 2021-03-15T22:32:38.279 INFO:tasks.rados.rados.0.smithi076.stdout:3025: read oid 39 snap -1 2021-03-15T22:32:38.279 INFO:tasks.rados.rados.0.smithi076.stdout:3025: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:32:38.279 INFO:tasks.rados.rados.0.smithi076.stdout:3026: snap_remove snap 322 2021-03-15T22:32:38.293 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:32:38.338 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:38.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:38.362 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:38.363 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:38.465 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:38.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:38.489 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:38.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:38.591 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:38.609 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: cluster 2021-03-15T22:32:36.868925+0000 osd.5 (osd.5) 34 2021-03-15T22:32:38.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: : cluster [DBG] 5.f scrub ok 2021-03-15T22:32:38.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: cluster 2021-03-15T22:32:37.178196+0000 mgr.y (mgr.14144) 942 : cluster [DBG] pgmap v1354: 148 pgs: 148 active+clean; 518 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2021-03-15T22:32:38.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: cluster 2021-03-15T22:32:37.525686+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:32:38.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: cluster 2021-03-15T22:32:37.690522+0000 osd.3 (osd.3) 147 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:32:38.611 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: audit 2021-03-15T22:32:37.914900+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? 172.21.15.76:0/147535761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:38.611 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: cluster 2021-03-15T22:32:38.271440+0000 mon.a (mon.0) 2094 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:32:38.611 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:32:38.611 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: 22:32:38.271473+0000 mon.a (mon.0) 2095 : cluster [INF] Cluster is now healthy 2021-03-15T22:32:38.612 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: audit 2021-03-15T22:32:38.275892+0000 mon.a (mon.0) 2096 : audit 2021-03-15T22:32:38.612 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: [INF] from='client.? 172.21.15.76:0/147535761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:38.612 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:32:38.612 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[56514]: :32:38.276067+0000 mon.a (mon.0) 2097 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2021-03-15T22:32:38.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: cluster 2021-03-15T22:32:38.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 2021-03-15T22:32:38.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:32:38.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: :32:36.868925+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.f scrub ok 2021-03-15T22:32:38.614 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:38.614 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 03-15T22:32:37.178196+0000 mgr.y (mgr. 2021-03-15T22:32:38.614 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 14144) 942 : cluster [DBG] pgmap v1354: 148 pgs: 148 active+clean; 518 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2021-03-15T22:32:38.614 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:32:38.615 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: -15T22:32:37.525686+0000 osd.7 2021-03-15T22:32:38.615 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: (osd.7) 150 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:32:38.615 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:32:38.615 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: -03-15T22:32:37.690522+0000 2021-03-15T22:32:38.616 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: osd.3 (osd.3) 147 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:32:38.616 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 2021-03-15T22:32:38.616 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: audit 2021-03-15T22:32:38.616 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 2021-03-15T22:32:37.914900+0000 mon.a ( 2021-03-15T22:32:38.617 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: mon.0) 2093 : audit [INF] from='client.? 172.21.15.76:0/147535761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:38.617 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:38.617 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 03-15T22:32:38.271440+0000 mon.a 2021-03-15T22:32:38.618 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: (mon.0) 2094 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:32:38.619 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 2021-03-15T22:32:38.619 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: cluster 2021-03-15T22:32:38.619 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 2021-03-15T22:32:38.271473+0000 mon.a 2021-03-15T22:32:38.619 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: (mon.0) 2095 : cluster [INF] Cluster is now healthy 2021-03-15T22:32:38.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 2021-03-15T22:32:38.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: audit 2021-03-15T22:32:38.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 2021-03-15T22:32:38.275892+0000 mon.a (mon 2021-03-15T22:32:38.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: .0) 2096 : audit [INF] from='client.? 172.21.15.76:0/147535761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:38.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:38.621 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 03-15T22:32:38.276067+0000 2021-03-15T22:32:38.621 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: mon.a (mon.0) 2097 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2021-03-15T22:32:38.621 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[62156]: 2021-03-15T22:32:38.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:38.624 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:38.625 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:38.629 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:38 smithi125 conmon[56739]: cluster 2021-03-15T22:32:36.868925+0000 osd.5 (osd.5) 34 : cluster [DBG] 5.f scrub ok 2021-03-15T22:32:38.630 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:38 smithi125 conmon[56739]: cluster 2021-03-15T22:32:37.178196+0000 mgr.y (mgr.14144) 942 : cluster [DBG] pgmap v1354: 148 pgs: 148 active+clean; 518 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 1.3 MiB/s wr, 9 op/s 2021-03-15T22:32:38.630 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:38 smithi125 conmon[56739]: cluster 2021-03-15T22:32:37.525686+0000 osd.7 (osd.7) 150 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:32:38.630 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:38 smithi125 conmon[56739]: cluster 2021-03-15T22:32:37.690522+0000 osd.3 (osd.3) 147 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:32:38.631 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:38 smithi125 conmon[56739]: audit 2021-03-15T22:32:37.914900+0000 mon.a (mon.0) 2093 : audit [INF] from='client.? 172.21.15.76:0/147535761' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:38.631 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:38 smithi125 conmon[56739]: cluster 2021-03-15T22:32:38.271440+0000 mon.a (mon.0) 2094 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:32:38.631 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:38 smithi125 conmon[56739]: cluster 2021-03-15T22:32:38.271473 2021-03-15T22:32:38.631 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:38 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2095 : cluster [INF] Cluster is now healthy 2021-03-15T22:32:38.632 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:38 smithi125 conmon[56739]: audit 2021-03-15T22:32:38.275892+0000 mon.a (mon.0) 2096 : audit [INF] from='client.? 172.21.15.76:0/147535761' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:38.632 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:38 smithi125 conmon[56739]: cluster 2021-03-15T22:32:38.276067+0000 mon.a (mon.0) 2097 : cluster [DBG] osdmap e808: 8 total, 8 up, 7 in 2021-03-15T22:32:38.690 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:38.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:38.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:38.750 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:38.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:38.851 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:38.867 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:32:38.875 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:38.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:38.876 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:38.880 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:32:38.881 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:32:38.881 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:32:38.881 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:32:38.881 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:32:38.885 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:32:38.904 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:32:38.921 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:32:38.939 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:32:38.957 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:32:38.974 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:32:38.977 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:38.992 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:32:38.993 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:32:38.993 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:32:38.994 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:32:38.994 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:32:38.994 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:32:38.994 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:32:38.995 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:32:38.995 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:32:38.999 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:39.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:39.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:39.017 INFO:journalctl@ceph.osd.0.smithi076.stdout:Mar 15 22:32:38 smithi076 conmon[65511]: debug 2021-03-15T22:32:38.700+0000 7f1a26084700 -1 osd.0 pg_epoch: 808 pg[7.9( v 808'394 (0'0,808'394] local-lis/les=556/557 n=5 ec=91/91 lis/c=556/556 les/c/f=557/557/0 sis=556) [0,1,4] r=0 lpr=556 crt=808'394 lcod 805'392 mlcod 805'392 active+clean ps=[22d~1,23e~1,247~1,268~1,26c~1,299~1]] scrubber pg(7.9) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 4 (0x557ad6907ce0) 2021-03-15T22:32:39.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:39.126 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:39.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:39.128 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:39.229 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:39.251 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:39.252 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:39.287 INFO:tasks.rados.rados.0.smithi076.stdout:3026: done (2 left) 2021-03-15T22:32:39.287 INFO:tasks.rados.rados.0.smithi076.stdout:3027: write oid 46 current snap is 350 2021-03-15T22:32:39.287 INFO:tasks.rados.rados.0.smithi076.stdout:3027: seq_num 1044 ranges {746508=500317,2009201=795686,3220075=326433} 2021-03-15T22:32:39.293 INFO:tasks.rados.rados.0.smithi076.stdout:3027: writing smithi07694878-46 from 746508 to 1246825 tid 1 2021-03-15T22:32:39.303 INFO:tasks.rados.rados.0.smithi076.stdout:3027: writing smithi07694878-46 from 2009201 to 2804887 tid 2 2021-03-15T22:32:39.308 INFO:tasks.rados.rados.0.smithi076.stdout:3027: writing smithi07694878-46 from 3220075 to 3546508 tid 3 2021-03-15T22:32:39.311 INFO:tasks.rados.rados.0.smithi076.stdout:3023: finishing copy_from to smithi07694878-33 2021-03-15T22:32:39.312 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 2395 (ObjNum 982 snap 326 seq_num 982) dirty exists 2021-03-15T22:32:39.312 INFO:tasks.rados.rados.0.smithi076.stdout:3023: finishing copy_from racing read to smithi07694878-33 2021-03-15T22:32:39.312 INFO:tasks.rados.rados.0.smithi076.stdout:3025: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:32:39.353 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:39.375 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:39.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:39.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:39.435 INFO:tasks.rados.rados.0.smithi076.stdout:3027: finishing write tid 1 to smithi07694878-46 2021-03-15T22:32:39.436 INFO:tasks.rados.rados.0.smithi076.stdout:3027: finishing write tid 2 to smithi07694878-46 2021-03-15T22:32:39.436 INFO:tasks.rados.rados.0.smithi076.stdout:3027: finishing write tid 3 to smithi07694878-46 2021-03-15T22:32:39.436 INFO:tasks.rados.rados.0.smithi076.stdout:3027: finishing write tid 4 to smithi07694878-46 2021-03-15T22:32:39.436 INFO:tasks.rados.rados.0.smithi076.stdout:3027: finishing write tid 5 to smithi07694878-46 2021-03-15T22:32:39.437 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2276 (ObjNum 1044 snap 350 seq_num 1044) dirty exists 2021-03-15T22:32:39.437 INFO:tasks.rados.rados.0.smithi076.stdout:3027: left oid 46 (ObjNum 1044 snap 350 seq_num 1044) 2021-03-15T22:32:39.437 INFO:tasks.rados.rados.0.smithi076.stdout:3023: done (2 left) 2021-03-15T22:32:39.437 INFO:tasks.rados.rados.0.smithi076.stdout:3025: done (1 left) 2021-03-15T22:32:39.437 INFO:tasks.rados.rados.0.smithi076.stdout:3027: done (0 left) 2021-03-15T22:32:39.438 INFO:tasks.rados.rados.0.smithi076.stdout:3028: write oid 26 current snap is 350 2021-03-15T22:32:39.438 INFO:tasks.rados.rados.0.smithi076.stdout:3028: seq_num 1045 ranges {763315=575566,2067683=655697,3247498=315817} 2021-03-15T22:32:39.441 INFO:tasks.rados.rados.0.smithi076.stdout:3028: writing smithi07694878-26 from 763315 to 1338881 tid 1 2021-03-15T22:32:39.449 INFO:tasks.rados.rados.0.smithi076.stdout:3028: writing smithi07694878-26 from 2067683 to 2723380 tid 2 2021-03-15T22:32:39.455 INFO:tasks.rados.rados.0.smithi076.stdout:3028: writing smithi07694878-26 from 3247498 to 3563315 tid 3 2021-03-15T22:32:39.455 INFO:tasks.rados.rados.0.smithi076.stdout:3029: read oid 46 snap -1 2021-03-15T22:32:39.455 INFO:tasks.rados.rados.0.smithi076.stdout:3029: expect (ObjNum 1044 snap 350 seq_num 1044) 2021-03-15T22:32:39.456 INFO:tasks.rados.rados.0.smithi076.stdout:3030: read oid 29 snap -1 2021-03-15T22:32:39.456 INFO:tasks.rados.rados.0.smithi076.stdout:3030: expect deleted 2021-03-15T22:32:39.456 INFO:tasks.rados.rados.0.smithi076.stdout:3031: read oid 21 snap 335 2021-03-15T22:32:39.456 INFO:tasks.rados.rados.0.smithi076.stdout:3031: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:32:39.456 INFO:tasks.rados.rados.0.smithi076.stdout:3032: read oid 33 snap -1 2021-03-15T22:32:39.457 INFO:tasks.rados.rados.0.smithi076.stdout:3032: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:32:39.457 INFO:tasks.rados.rados.0.smithi076.stdout:3033: copy_from oid 41 from oid 20 current snap is 350 2021-03-15T22:32:39.457 INFO:tasks.rados.rados.0.smithi076.stdout:3028: finishing write tid 1 to smithi07694878-26 2021-03-15T22:32:39.457 INFO:tasks.rados.rados.0.smithi076.stdout:3028: finishing write tid 2 to smithi07694878-26 2021-03-15T22:32:39.458 INFO:tasks.rados.rados.0.smithi076.stdout:3034: snap_create 2021-03-15T22:32:39.458 INFO:tasks.rados.rados.0.smithi076.stdout:3028: finishing write tid 3 to smithi07694878-26 2021-03-15T22:32:39.458 INFO:tasks.rados.rados.0.smithi076.stdout:3028: finishing write tid 4 to smithi07694878-26 2021-03-15T22:32:39.461 INFO:tasks.rados.rados.0.smithi076.stdout:3029: expect (ObjNum 1044 snap 350 seq_num 1044) 2021-03-15T22:32:39.477 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:39.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:39.501 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:39.503 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:39.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:39 smithi076 conmon[56514]: cluster 2021-03-15T22:32:38.318682+0000 2021-03-15T22:32:39.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:39 smithi076 conmon[56514]: osd.4 (osd.4) 171 : cluster [DBG] 4.e scrub ok 2021-03-15T22:32:39.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:39 smithi076 conmon[56514]: cluster 2021-03-15T22:32:38.476260+0000 osd.7 (osd.7) 151 : cluster [DBG] 5.1a scrub ok 2021-03-15T22:32:39.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:39 smithi076 conmon[56514]: audit 2021-03-15T22:32:38.846650+0000 mon.a (mon.0) 2098 : audit [DBG] from='client.? 172.21.15.76:0/4127719102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:39.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:39 smithi076 conmon[56514]: cluster 2021-03-15T22:32:39.279109+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2021-03-15T22:32:39.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:39 smithi076 conmon[62156]: cluster 2021-03-15T22:32:38.318682+0000 osd.4 (osd 2021-03-15T22:32:39.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:39 smithi076 conmon[62156]: .4) 171 : cluster [DBG] 4.e scrub ok 2021-03-15T22:32:39.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:39 smithi076 conmon[62156]: cluster 2021-03-15T22:32:38.476260+0000 osd.7 (osd.7) 151 : cluster [DBG] 5.1a scrub ok 2021-03-15T22:32:39.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:39 smithi076 conmon[62156]: audit 2021-03-15T22:32:38.846650+0000 mon.a (mon.0) 2098 : audit [DBG] from='client.? 172.21.15.76:0/4127719102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:39.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:39 smithi076 conmon[62156]: cluster 2021-03-15T22:32:39.279109+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2021-03-15T22:32:39.536 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-15T22:32:39.598 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:39 smithi125 conmon[56739]: cluster 2021-03-15T22:32:38.318682+0000 osd.4 (osd.4) 171 : 2021-03-15T22:32:39.598 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:39 smithi125 conmon[56739]: cluster [DBG] 4.e scrub ok 2021-03-15T22:32:39.598 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:39 smithi125 conmon[56739]: cluster 2021-03-15T22:32:38.476260+0000 osd.7 (osd.7) 151 : cluster [DBG] 5.1a scrub ok 2021-03-15T22:32:39.599 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:39 smithi125 conmon[56739]: audit 2021-03-15T22:32:38.846650+0000 mon.a (mon.0) 2098 : audit [DBG] from='client.? 172.21.15.76:0/4127719102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:39.599 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:39 smithi125 conmon[56739]: cluster 2021-03-15T22:32:39.279109+0000 mon.a (mon.0) 2099 : cluster [DBG] osdmap e809: 8 total, 8 up, 7 in 2021-03-15T22:32:39.604 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:39.629 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:39.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:39.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:39.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:39.753 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:39.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:39.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:39.755 INFO:tasks.rados.rados.0.smithi076.stdout:3028: finishing write tid 5 to smithi07694878-26 2021-03-15T22:32:39.755 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 1783 (ObjNum 1045 snap 350 seq_num 1045) dirty exists 2021-03-15T22:32:39.755 INFO:tasks.rados.rados.0.smithi076.stdout:3028: left oid 26 (ObjNum 1045 snap 350 seq_num 1045) 2021-03-15T22:32:39.756 INFO:tasks.rados.rados.0.smithi076.stdout:3033: finishing copy_from to smithi07694878-41 2021-03-15T22:32:39.756 INFO:tasks.rados.rados.0.smithi076.stdout:3033: got expected ENOENT (src dne) 2021-03-15T22:32:39.756 INFO:tasks.rados.rados.0.smithi076.stdout:3033: finishing copy_from racing read to smithi07694878-41 2021-03-15T22:32:39.757 INFO:tasks.rados.rados.0.smithi076.stdout:3032: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:32:39.833 INFO:tasks.rados.rados.0.smithi076.stdout:3031: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:32:39.851 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:39.855 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:39.880 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:39.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:39.882 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:39.983 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:40.006 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:40.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:40.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:40.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:40.130 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:40.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:40.132 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:40.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:40.255 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:40.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:40.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:40.269 INFO:tasks.rados.rados.0.smithi076.stdout:3028: done (6 left) 2021-03-15T22:32:40.270 INFO:tasks.rados.rados.0.smithi076.stdout:3029: done (5 left) 2021-03-15T22:32:40.270 INFO:tasks.rados.rados.0.smithi076.stdout:3030: done (4 left) 2021-03-15T22:32:40.270 INFO:tasks.rados.rados.0.smithi076.stdout:3031: done (3 left) 2021-03-15T22:32:40.270 INFO:tasks.rados.rados.0.smithi076.stdout:3032: done (2 left) 2021-03-15T22:32:40.271 INFO:tasks.rados.rados.0.smithi076.stdout:3033: done (1 left) 2021-03-15T22:32:40.271 INFO:tasks.rados.rados.0.smithi076.stdout:3034: done (0 left) 2021-03-15T22:32:40.271 INFO:tasks.rados.rados.0.smithi076.stdout:3035: rollback oid 28 current snap is 351 2021-03-15T22:32:40.271 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 321 2021-03-15T22:32:40.272 INFO:tasks.rados.rados.0.smithi076.stdout:3036: snap_remove snap 344 2021-03-15T22:32:40.322 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:32:40.358 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:40.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:40.381 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:40.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:40.483 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:40.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:40.508 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:40.509 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:40.610 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:40.632 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:40.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:40.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:40.734 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:40.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:40.760 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:40.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:40.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: cluster 2021-03-15T22:32:38.701949+0000 2021-03-15T22:32:40.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: osd.1 (osd.1) 158 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:32:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: cluster 2021-03-15T22:32:38.710395+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: cluster 2021-03-15T22:32:38.711243+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: cluster 2021-03-15T22:32:38.880988+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.15 deep-scrub ok 2021-03-15T22:32:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: cluster 2021-03-15T22:32:38 2021-03-15T22:32:40.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: .958044+0000 osd.2 (osd.2) 30 : cluster [DBG] 6.f scrub ok 2021-03-15T22:32:40.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: cluster 2021-03-15T22:32:39.178718+0000 mgr.y ( 2021-03-15T22:32:40.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: mgr.14144) 943 : cluster [DBG] pgmap v1357: 148 pgs: 148 active+clean; 518 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 1012 KiB/s wr, 3 op/s 2021-03-15T22:32:40.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: cluster 2021-03-15T22:32:40.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: 2021-03-15T22:32:39.271258+0000 osd.4 (osd.4) 172 : 2021-03-15T22:32:40.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: cluster [DBG] 6.2 scrub ok 2021-03-15T22:32:40.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: cluster 2021-03-15T22:32:39. 2021-03-15T22:32:40.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: 451036+0000 osd.7 (osd.7) 152 : cluster [DBG] 2021-03-15T22:32:40.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: 5.4 scrub ok 2021-03-15T22:32:40.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: cluster 2021-03-15T22:32:40.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: 2021-03-15T22:32:39.697931+0000 osd.0 (osd.0) 133 : cluster 2021-03-15T22:32:40.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: [DBG] 3.10 deep-scrub ok 2021-03-15T22:32:40.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: cluster 2021-03-15T22:32:40.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: 2021-03-15T22:32:40.263874+0000 mon.a (mon 2021-03-15T22:32:40.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[56514]: .0) 2100 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2021-03-15T22:32:40.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: cluster 2021-03-15T22:32:38.701949+0000 osd.1 (osd 2021-03-15T22:32:40.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: .1) 158 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:32:40.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: cluster 2021-03-15T22:32:38.710395+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:40.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:40.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: 03-15T22:32:38.711243+0000 2021-03-15T22:32:40.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: osd.6 (osd.6) 119 : cluster 2021-03-15T22:32:40.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: [DBG] purged_snaps scrub ok 2021-03-15T22:32:40.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:40.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: 03-15T22:32:38.880988+0000 osd.5 2021-03-15T22:32:40.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: (osd.5) 35 : cluster 2021-03-15T22:32:40.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: [DBG] 5.15 deep-scrub ok 2021-03-15T22:32:40.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:40.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: 03-15T22:32:38.958044+0000 osd.2 2021-03-15T22:32:40.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: (osd.2) 30 : cluster 2021-03-15T22:32:40.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: [DBG] 6.f scrub ok 2021-03-15T22:32:40.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:40.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: 03-15T22:32:39.178718+0000 2021-03-15T22:32:40.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: mgr.y (mgr.14144) 943 : 2021-03-15T22:32:40.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: cluster [DBG] pgmap v1357: 148 pgs: 148 active+clean; 518 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 1012 KiB/s wr, 3 op/s 2021-03-15T22:32:40.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:40.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: 03-15T22:32:39.271258+0000 2021-03-15T22:32:40.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: osd.4 (osd.4) 172 : cluster 2021-03-15T22:32:40.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: [DBG] 6.2 scrub ok 2021-03-15T22:32:40.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:40.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: 03-15T22:32:39.451036+0000 2021-03-15T22:32:40.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: osd.7 (osd.7) 152 : cluster 2021-03-15T22:32:40.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: [DBG] 5.4 scrub ok 2021-03-15T22:32:40.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:40.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: 03-15T22:32:39.697931+0000 osd.0 (osd 2021-03-15T22:32:40.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: .0) 133 : cluster [DBG] 3.10 deep-scrub ok 2021-03-15T22:32:40.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: 2021-03-15T22:32:40.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: cluster 2021-03-15T22:32:40.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: 2021-03-15T22:32:40 2021-03-15T22:32:40.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: .263874+0000 mon.a (mon.0 2021-03-15T22:32:40.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:40 smithi076 conmon[62156]: ) 2100 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2021-03-15T22:32:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: cluster 2021-03-15T22:32:38.701949+0000 osd.1 (osd 2021-03-15T22:32:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: .1) 158 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:32:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: cluster 2021-03-15T22:32:38.710395+0000 osd.6 (osd.6) 118 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: cluster 2021-03-15T22:32:38.711243+0000 osd.6 (osd.6) 119 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: cluster 2021-03-15T22:32:38.880988+0000 osd.5 (osd.5) 35 : cluster [DBG] 5.15 deep-scrub ok 2021-03-15T22:32:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: cluster 2021-03-15T22:32:38.958044+0000 osd.2 (osd.2) 30 : cluster [DBG] 6.f scrub ok 2021-03-15T22:32:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:32:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: T22:32:39.178718+0000 mgr.y (mgr.14144) 943 : cluster [DBG] pgmap v1357: 148 pgs: 148 active+clean; 518 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 255 B/s rd, 1012 KiB/s wr, 3 op/s 2021-03-15T22:32:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:32:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: 22:32:39.271258+0000 osd.4 (osd.4) 172 : cluster [DBG] 2021-03-15T22:32:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: 6.2 scrub ok 2021-03-15T22:32:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:32:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: 32:39.451036+0000 osd.7 (osd.7) 2021-03-15T22:32:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: 152 : cluster [DBG] 5.4 scrub ok 2021-03-15T22:32:40.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:32:40.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: 03-15T22:32:39.697931+0000 2021-03-15T22:32:40.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: osd.0 (osd.0) 133 : cluster [DBG] 3.10 deep-scrub ok 2021-03-15T22:32:40.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: 2021-03-15T22:32:40.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: cluster 2021-03-15T22:32 2021-03-15T22:32:40.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: :40.263874+0000 mon.a (mon.0) 2021-03-15T22:32:40.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:40 smithi125 conmon[56739]: 2100 : cluster [DBG] osdmap e810: 8 total, 8 up, 7 in 2021-03-15T22:32:40.861 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:40.884 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:40.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:40.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:40.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:41.010 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:41.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:41.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:41.113 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:41.136 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:41.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:41.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:41.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:41.262 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:41.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:41.286 INFO:tasks.rados.rados.0.smithi076.stdout:3036: done (1 left) 2021-03-15T22:32:41.286 INFO:tasks.rados.rados.0.smithi076.stdout:3037: write oid 24 current snap is 351 2021-03-15T22:32:41.286 INFO:tasks.rados.rados.0.smithi076.stdout:3035: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:32:41.287 INFO:tasks.rados.rados.0.smithi076.stdout:3037: seq_num 1046 ranges {780122=650815,2126165=799355} 2021-03-15T22:32:41.294 INFO:tasks.rados.rados.0.smithi076.stdout:3037: writing smithi07694878-24 from 780122 to 1430937 tid 1 2021-03-15T22:32:41.302 INFO:tasks.rados.rados.0.smithi076.stdout:3037: writing smithi07694878-24 from 2126165 to 2925520 tid 2 2021-03-15T22:32:41.303 INFO:tasks.rados.rados.0.smithi076.stdout:3038: snap_remove snap 300 2021-03-15T22:32:41.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:41.388 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:41.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:41.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:41.420 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:32:41.420 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:32:41.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:41 smithi076 conmon[56514]: cluster 2021-03-15T22:32:39.977322+0000 osd.2 2021-03-15T22:32:41.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:41 smithi076 conmon[56514]: (osd.2) 31 : cluster [DBG] 4.a scrub ok 2021-03-15T22:32:41.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:41 smithi076 conmon[56514]: audit 2021-03-15T22:32:40.136500+0000 mgr.y (mgr.14144) 944 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:41.455 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:41 smithi076 conmon[56514]: cluster 2021-03-15T22:32:40.430035+0000 osd.7 (osd.7) 153 : cluster [DBG] 3.d scrub ok 2021-03-15T22:32:41.455 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:41 smithi076 conmon[56514]: cluster 2021-03-15T22:32:41.272737+0000 mon.a (mon.0) 2101 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2021-03-15T22:32:41.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:41 smithi076 conmon[62156]: cluster 2021-03-15T22:32:39.977322+0000 osd.2 ( 2021-03-15T22:32:41.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:41 smithi076 conmon[62156]: osd.2) 31 : cluster [DBG] 4.a scrub ok 2021-03-15T22:32:41.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:41 smithi076 conmon[62156]: audit 2021-03-15T22:32:40.136500+0000 mgr.y (mgr.14144) 944 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:41.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:41 smithi076 conmon[62156]: cluster 2021-03-15T22:32:40.430035+0000 osd.7 (osd.7) 153 : cluster [DBG] 3.d scrub ok 2021-03-15T22:32:41.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:41 smithi076 conmon[62156]: cluster 2021-03-15T22:32:41 2021-03-15T22:32:41.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:41 smithi076 conmon[62156]: .272737+0000 mon.a (mon.0) 2101 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2021-03-15T22:32:41.490 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:41.500 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:41 smithi125 conmon[56739]: cluster 2021-03-15T22:32:39.977322+0000 osd.2 (osd.2 2021-03-15T22:32:41.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:41 smithi125 conmon[56739]: ) 31 : cluster [DBG] 4.a scrub ok 2021-03-15T22:32:41.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:41 smithi125 conmon[56739]: audit 2021-03-15T22:32:40.136500+0000 mgr.y (mgr.14144) 944 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:41.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:41 smithi125 conmon[56739]: cluster 2021-03-15T22:32:40.430035+0000 osd.7 (osd.7) 153 : cluster [DBG] 3.d scrub ok 2021-03-15T22:32:41.501 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:41 smithi125 conmon[56739]: cluster 2021-03-15T22:32:41.272737+0000 mon.a (mon.0) 2101 : cluster [DBG] osdmap e811: 8 total, 8 up, 7 in 2021-03-15T22:32:41.513 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:41.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:41.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:41.562 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:32:41.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:41.643 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:41.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:41.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:41.745 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:41.768 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:41.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:41.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:41.810 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:32:41.813 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:32:41.847 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:41.871 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:41.895 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:41.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:41.897 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:41.998 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:42.020 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:42.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:42.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:42.029 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:32:42.123 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:42.146 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:42.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:42.147 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:42.216 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:32:42.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:42.271 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:42.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:42.272 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:42.283 INFO:tasks.rados.rados.0.smithi076.stdout:3038: done (2 left) 2021-03-15T22:32:42.283 INFO:tasks.rados.rados.0.smithi076.stdout:3039: read oid 35 snap -1 2021-03-15T22:32:42.283 INFO:tasks.rados.rados.0.smithi076.stdout:3039: expect deleted 2021-03-15T22:32:42.284 INFO:tasks.rados.rados.0.smithi076.stdout:3040: snap_remove snap 330 2021-03-15T22:32:42.345 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:42.374 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:42.397 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:42.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:42.398 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:42.499 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:42.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:42 smithi076 conmon[56514]: cluster 2021-03-15T22:32:41.179265+0000 mgr.y (mgr.14144) 945 : cluster [DBG] pgmap v1360: 148 pgs: 148 active+clean; 520 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.6 MiB/s rd, 994 KiB/s wr, 6 op/s 2021-03-15T22:32:42.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:42 smithi076 conmon[56514]: cluster 2021-03-15T22:32:41.294920+0000 osd.4 (osd.4) 173 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:32:42.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:42 smithi076 conmon[56514]: cluster 2021-03-15T22:32:41.673788+0000 osd.0 (osd.0) 134 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:32:42.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:42 smithi076 conmon[56514]: audit 2021-03-15T22:32:41.791085+0000 mgr.y (mgr.14144) 946 : audit [DBG] from='client.16431 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:32:42.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:42 smithi076 conmon[56514]: cluster 2021-03-15T22:32:42.274250+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2021-03-15T22:32:42.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:42 smithi076 conmon[62156]: cluster 2021-03-15T22:32:41.179265+0000 mgr.y (mgr.14144) 945 : cluster [DBG] pgmap v1360: 148 pgs: 148 active+clean; 520 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.6 MiB/s rd, 994 KiB/s wr, 6 op/s 2021-03-15T22:32:42.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:42 smithi076 conmon[62156]: cluster 2021-03-15T22:32:41.294920+0000 osd.4 (osd.4) 173 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:32:42.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:42 smithi076 conmon[62156]: cluster 2021-03-15T22:32:41.673788+0000 osd.0 (osd.0) 134 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:32:42.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:42 smithi076 conmon[62156]: audit 2021-03-15T22:32:41.791085+0000 mgr.y (mgr.14144) 946 : audit [DBG] from='client.16431 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:32:42.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:42 smithi076 conmon[62156]: cluster 2021-03-15T22:32:42.274250+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2021-03-15T22:32:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:42.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:42.522 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:42.623 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:42.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:42.646 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:42.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:42.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:42.753 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:42 smithi125 conmon[56739]: cluster 2021-03-15T22:32:41.179265+0000 mgr.y (mgr.14144) 945 : cluster [DBG] pgmap v1360: 148 pgs: 148 active+clean; 520 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.6 MiB/s rd, 994 KiB/s wr, 6 op/s 2021-03-15T22:32:42.754 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:42 smithi125 conmon[56739]: cluster 2021-03-15T22:32:41.294920+0000 osd.4 (osd.4) 173 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:32:42.754 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:42 smithi125 conmon[56739]: cluster 2021-03-15T22:32:41.673788+0000 osd.0 (osd.0) 134 : cluster [DBG] 3.1b scrub ok 2021-03-15T22:32:42.754 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:42 smithi125 conmon[56739]: audit 2021-03-15T22:32:41.791085+0000 mgr.y (mgr.14144) 946 : audit [DBG] from='client.16431 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:32:42.754 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:42 smithi125 conmon[56739]: cluster 2021-03-15T22:32:42.274250+0000 mon.a (mon.0) 2102 : cluster [DBG] osdmap e812: 8 total, 8 up, 7 in 2021-03-15T22:32:42.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:42.773 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:42.773 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:42.797 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1361,"stamp":"2021-03-15T22:32:41.270228+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":545525683,"num_objects":565,"num_object_clones":200,"num_object_copies":1695,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":565,"num_whiteouts":16,"num_read":8014,"num_read_kb":5767418,"num_write":4993,"num_write_kb":1556775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3262,"num_bytes_recovered":1223532973,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17916,"ondisk_log_size":17916,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1483924,"kb_used_data":1219036,"kb_used_omap":273,"kb_used_meta":264558,"kb_avail":654732140,"statfs":{"total":671965249536,"available":670445711360,"internally_reserved":0,"allocated":1248292864,"data_stored":1686886992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":270908282},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1962454,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":19,"num_read_kb":14686,"num_write":8,"num_write_kb":3976,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.997429"},"pg_stats":[{"pgid":"5.19","version":"317'7","reported_seq":"165","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.273556+0000","last_change":"2021-03-15T22:32:07.910488+0000","last_active":"2021-03-15T22:32:36.273556+0000","last_peered":"2021-03-15T22:32:36.273556+0000","last_clean":"2021-03-15T22:32:36.273556+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:32:36.273556+0000","last_undegraded":"2021-03-15T22:32:36.273556+0000","last_fullsized":"2021-03-15T22:32:36.273556+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:32:07.910427+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:32:07.910427+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"807","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263402+0000","last_change":"2021-03-15T22:30:41.431150+0000","last_active":"2021-03-15T22:32:35.263402+0000","last_peered":"2021-03-15T22:32:35.263402+0000","last_clean":"2021-03-15T22:32:35.263402+0000","last_became_active":"2021-03-15T22:29:51.225252+0000","last_became_peered":"2021-03-15T22:29:51.225252+0000","last_unstale":"2021-03-15T22:32:35.263402+0000","last_undegraded":"2021-03-15T22:32:35.263402+0000","last_fullsized":"2021-03-15T22:32:35.263402+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.431052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"313","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283819+0000","last_change":"2021-03-15T22:32:31.680174+0000","last_active":"2021-03-15T22:32:39.283819+0000","last_peered":"2021-03-15T22:32:39.283819+0000","last_clean":"2021-03-15T22:32:39.283819+0000","last_became_active":"2021-03-15T22:30:32.251610+0000","last_became_peered":"2021-03-15T22:30:32.251610+0000","last_unstale":"2021-03-15T22:32:39.283819+0000","last_undegraded":"2021-03-15T22:32:39.283819+0000","last_fullsized":"2021-03-15T22:32:39.283819+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:32:31.680111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.680111+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"868","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.264651+0000","last_change":"2021-03-15T22:32:12.177096+0000","last_active":"2021-03-15T22:32:35.264651+0000","last_peered":"2021-03-15T22:32:35.264651+0000","last_clean":"2021-03-15T22:32:35.264651+0000","last_became_active":"2021-03-15T22:31:28.287770+0000","last_became_peered":"2021-03-15T22:31:28.287770+0000","last_unstale":"2021-03-15T22:32:35.264651+0000","last_undegraded":"2021-03-15T22:32:35.264651+0000","last_fullsized":"2021-03-15T22:32:35.264651+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.177019+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1038,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"99","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.262729+0000","last_change":"2021-03-15T22:32:33.338150+0000","last_active":"2021-03-15T22:32:35.262729+0000","last_peered":"2021-03-15T22:32:35.262729+0000","last_clean":"2021-03-15T22:32:35.262729+0000","last_became_active":"2021-03-15T22:31:28.274481+0000","last_became_peered":"2021-03-15T22:31:28.274481+0000","last_unstale":"2021-03-15T22:32:35.262729+0000","last_undegraded":"2021-03-15T22:32:35.262729+0000","last_fullsized":"2021-03-15T22:32:35.262729+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.338100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.338100+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"239","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263762+0000","last_change":"2021-03-15T22:31:28.285592+0000","last_active":"2021-03-15T22:32:35.263762+0000","last_peered":"2021-03-15T22:32:35.263762+0000","last_clean":"2021-03-15T22:32:35.263762+0000","last_became_active":"2021-03-15T22:31:28.285281+0000","last_became_peered":"2021-03-15T22:31:28.285281+0000","last_unstale":"2021-03-15T22:32:35.263762+0000","last_undegraded":"2021-03-15T22:32:35.263762+0000","last_fullsized":"2021-03-15T22:32:35.263762+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:05.262019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.262019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"526","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.264890+0000","last_change":"2021-03-15T22:31:58.617873+0000","last_active":"2021-03-15T22:32:35.264890+0000","last_peered":"2021-03-15T22:32:35.264890+0000","last_clean":"2021-03-15T22:32:35.264890+0000","last_became_active":"2021-03-15T22:31:28.289753+0000","last_became_peered":"2021-03-15T22:31:28.289753+0000","last_unstale":"2021-03-15T22:32:35.264890+0000","last_undegraded":"2021-03-15T22:32:35.264890+0000","last_fullsized":"2021-03-15T22:32:35.264890+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:58.617803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:58.617803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,1],"acting":[3,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"539","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.260983+0000","last_change":"2021-03-15T22:32:10.653022+0000","last_active":"2021-03-15T22:32:35.260983+0000","last_peered":"2021-03-15T22:32:35.260983+0000","last_clean":"2021-03-15T22:32:35.260983+0000","last_became_active":"2021-03-15T22:30:32.258079+0000","last_became_peered":"2021-03-15T22:30:32.258079+0000","last_unstale":"2021-03-15T22:32:35.260983+0000","last_undegraded":"2021-03-15T22:32:35.260983+0000","last_fullsized":"2021-03-15T22:32:35.260983+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.652898+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"643","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523415+0000","last_change":"2021-03-15T22:32:12.794180+0000","last_active":"2021-03-15T22:32:37.523415+0000","last_peered":"2021-03-15T22:32:37.523415+0000","last_clean":"2021-03-15T22:32:37.523415+0000","last_became_active":"2021-03-15T22:31:20.246770+0000","last_became_peered":"2021-03-15T22:31:20.246770+0000","last_unstale":"2021-03-15T22:32:37.523415+0000","last_undegraded":"2021-03-15T22:32:37.523415+0000","last_fullsized":"2021-03-15T22:32:37.523415+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.794113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.794113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"276","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.282187+0000","last_change":"2021-03-15T22:32:01.452949+0000","last_active":"2021-03-15T22:32:39.282187+0000","last_peered":"2021-03-15T22:32:39.282187+0000","last_clean":"2021-03-15T22:32:39.282187+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:32:39.282187+0000","last_undegraded":"2021-03-15T22:32:39.282187+0000","last_fullsized":"2021-03-15T22:32:39.282187+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:01.452885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:32:01.452885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"188","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523769+0000","last_change":"2021-03-15T22:31:28.288569+0000","last_active":"2021-03-15T22:32:37.523769+0000","last_peered":"2021-03-15T22:32:37.523769+0000","last_clean":"2021-03-15T22:32:37.523769+0000","last_became_active":"2021-03-15T22:31:28.288226+0000","last_became_peered":"2021-03-15T22:31:28.288226+0000","last_unstale":"2021-03-15T22:32:37.523769+0000","last_undegraded":"2021-03-15T22:32:37.523769+0000","last_fullsized":"2021-03-15T22:32:37.523769+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.146408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.146408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"950","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263400+0000","last_change":"2021-03-15T22:30:28.328526+0000","last_active":"2021-03-15T22:32:35.263400+0000","last_peered":"2021-03-15T22:32:35.263400+0000","last_clean":"2021-03-15T22:32:35.263400+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:32:35.263400+0000","last_undegraded":"2021-03-15T22:32:35.263400+0000","last_fullsized":"2021-03-15T22:32:35.263400+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:30:28.328458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.328458+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"91","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.476370+0000","last_change":"2021-03-15T22:32:38.476370+0000","last_active":"2021-03-15T22:32:38.476370+0000","last_peered":"2021-03-15T22:32:38.476370+0000","last_clean":"2021-03-15T22:32:38.476370+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:32:38.476370+0000","last_undegraded":"2021-03-15T22:32:38.476370+0000","last_fullsized":"2021-03-15T22:32:38.476370+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.476269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.476269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"424","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523264+0000","last_change":"2021-03-15T22:30:40.173603+0000","last_active":"2021-03-15T22:32:37.523264+0000","last_peered":"2021-03-15T22:32:37.523264+0000","last_clean":"2021-03-15T22:32:37.523264+0000","last_became_active":"2021-03-15T22:29:52.223704+0000","last_became_peered":"2021-03-15T22:29:52.223704+0000","last_unstale":"2021-03-15T22:32:37.523264+0000","last_undegraded":"2021-03-15T22:32:37.523264+0000","last_fullsized":"2021-03-15T22:32:37.523264+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:40.173519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:30:40.173519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"184","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523608+0000","last_change":"2021-03-15T22:32:14.756556+0000","last_active":"2021-03-15T22:32:37.523608+0000","last_peered":"2021-03-15T22:32:37.523608+0000","last_clean":"2021-03-15T22:32:37.523608+0000","last_became_active":"2021-03-15T22:29:51.234868+0000","last_became_peered":"2021-03-15T22:29:51.234868+0000","last_unstale":"2021-03-15T22:32:37.523608+0000","last_undegraded":"2021-03-15T22:32:37.523608+0000","last_fullsized":"2021-03-15T22:32:37.523608+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.756460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.756460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"893","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.265096+0000","last_change":"2021-03-15T22:32:05.111125+0000","last_active":"2021-03-15T22:32:35.265096+0000","last_peered":"2021-03-15T22:32:35.265096+0000","last_clean":"2021-03-15T22:32:35.265096+0000","last_became_active":"2021-03-15T22:29:11.197059+0000","last_became_peered":"2021-03-15T22:29:11.197059+0000","last_unstale":"2021-03-15T22:32:35.265096+0000","last_undegraded":"2021-03-15T22:32:35.265096+0000","last_fullsized":"2021-03-15T22:32:35.265096+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:32:05.111031+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.111031+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"762","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.274066+0000","last_change":"2021-03-15T22:32:32.744807+0000","last_active":"2021-03-15T22:32:36.274066+0000","last_peered":"2021-03-15T22:32:36.274066+0000","last_clean":"2021-03-15T22:32:36.274066+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:32:36.274066+0000","last_undegraded":"2021-03-15T22:32:36.274066+0000","last_fullsized":"2021-03-15T22:32:36.274066+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:32:32.744716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.744716+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"814","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.262968+0000","last_change":"2021-03-15T22:31:37.466525+0000","last_active":"2021-03-15T22:32:35.262968+0000","last_peered":"2021-03-15T22:32:35.262968+0000","last_clean":"2021-03-15T22:32:35.262968+0000","last_became_active":"2021-03-15T22:31:28.281143+0000","last_became_peered":"2021-03-15T22:31:28.281143+0000","last_unstale":"2021-03-15T22:32:35.262968+0000","last_undegraded":"2021-03-15T22:32:35.262968+0000","last_fullsized":"2021-03-15T22:32:35.262968+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:31:37.466443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,3],"acting":[4,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"275","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283614+0000","last_change":"2021-03-15T22:30:38.781792+0000","last_active":"2021-03-15T22:32:39.283614+0000","last_peered":"2021-03-15T22:32:39.283614+0000","last_clean":"2021-03-15T22:32:39.283614+0000","last_became_active":"2021-03-15T22:28:12.519619+0000","last_became_peered":"2021-03-15T22:28:12.519619+0000","last_unstale":"2021-03-15T22:32:39.283614+0000","last_undegraded":"2021-03-15T22:32:39.283614+0000","last_fullsized":"2021-03-15T22:32:39.283614+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:30:38.781724+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.781724+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"681","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.277065+0000","last_change":"2021-03-15T22:32:10.001304+0000","last_active":"2021-03-15T22:32:38.277065+0000","last_peered":"2021-03-15T22:32:38.277065+0000","last_clean":"2021-03-15T22:32:38.277065+0000","last_became_active":"2021-03-15T22:30:32.250733+0000","last_became_peered":"2021-03-15T22:30:32.250733+0000","last_unstale":"2021-03-15T22:32:38.277065+0000","last_undegraded":"2021-03-15T22:32:38.277065+0000","last_fullsized":"2021-03-15T22:32:38.277065+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:32:10.001234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.001234+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"834","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263541+0000","last_change":"2021-03-15T22:32:10.144209+0000","last_active":"2021-03-15T22:32:35.263541+0000","last_peered":"2021-03-15T22:32:35.263541+0000","last_clean":"2021-03-15T22:32:35.263541+0000","last_became_active":"2021-03-15T22:31:20.240898+0000","last_became_peered":"2021-03-15T22:31:20.240898+0000","last_unstale":"2021-03-15T22:32:35.263541+0000","last_undegraded":"2021-03-15T22:32:35.263541+0000","last_fullsized":"2021-03-15T22:32:35.263541+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.144132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.144132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"630","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.265572+0000","last_change":"2021-03-15T22:32:14.594783+0000","last_active":"2021-03-15T22:32:35.265572+0000","last_peered":"2021-03-15T22:32:35.265572+0000","last_clean":"2021-03-15T22:32:35.265572+0000","last_became_active":"2021-03-15T22:31:28.286067+0000","last_became_peered":"2021-03-15T22:31:28.286067+0000","last_unstale":"2021-03-15T22:32:35.265572+0000","last_undegraded":"2021-03-15T22:32:35.265572+0000","last_fullsized":"2021-03-15T22:32:35.265572+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.594717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:45.403769+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.594717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"795","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263460+0000","last_change":"2021-03-15T22:30:39.443682+0000","last_active":"2021-03-15T22:32:35.263460+0000","last_peered":"2021-03-15T22:32:35.263460+0000","last_clean":"2021-03-15T22:32:35.263460+0000","last_became_active":"2021-03-15T22:30:31.243315+0000","last_became_peered":"2021-03-15T22:30:31.243315+0000","last_unstale":"2021-03-15T22:32:35.263460+0000","last_undegraded":"2021-03-15T22:32:35.263460+0000","last_fullsized":"2021-03-15T22:32:35.263460+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.443613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"250","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.261556+0000","last_change":"2021-03-15T22:29:54.564185+0000","last_active":"2021-03-15T22:32:35.261556+0000","last_peered":"2021-03-15T22:32:35.261556+0000","last_clean":"2021-03-15T22:32:35.261556+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:32:35.261556+0000","last_undegraded":"2021-03-15T22:32:35.261556+0000","last_fullsized":"2021-03-15T22:32:35.261556+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:29:54.564115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:54.564115+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"771'16","reported_seq":"368","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263107+0000","last_change":"2021-03-15T22:32:01.253003+0000","last_active":"2021-03-15T22:32:35.263107+0000","last_peered":"2021-03-15T22:32:35.263107+0000","last_clean":"2021-03-15T22:32:35.263107+0000","last_became_active":"2021-03-15T22:32:01.252590+0000","last_became_peered":"2021-03-15T22:32:01.252590+0000","last_unstale":"2021-03-15T22:32:35.263107+0000","last_undegraded":"2021-03-15T22:32:35.263107+0000","last_fullsized":"2021-03-15T22:32:35.263107+0000","mapping_epoch":770,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":771,"parent":"0.0","parent_split_bits":5,"last_scrub":"618'7","last_scrub_stamp":"2021-03-15T22:29:40.964753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.964753+0000","log_size":13,"ondisk_log_size":13,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"819","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.525806+0000","last_change":"2021-03-15T22:32:37.525778+0000","last_active":"2021-03-15T22:32:37.525806+0000","last_peered":"2021-03-15T22:32:37.525806+0000","last_clean":"2021-03-15T22:32:37.525806+0000","last_became_active":"2021-03-15T22:30:32.243708+0000","last_became_peered":"2021-03-15T22:30:32.243708+0000","last_unstale":"2021-03-15T22:32:37.525806+0000","last_undegraded":"2021-03-15T22:32:37.525806+0000","last_fullsized":"2021-03-15T22:32:37.525806+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:37.525692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:37.525692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"830","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.273503+0000","last_change":"2021-03-15T22:31:05.123509+0000","last_active":"2021-03-15T22:32:36.273503+0000","last_peered":"2021-03-15T22:32:36.273503+0000","last_clean":"2021-03-15T22:32:36.273503+0000","last_became_active":"2021-03-15T22:28:59.264559+0000","last_became_peered":"2021-03-15T22:28:59.264559+0000","last_unstale":"2021-03-15T22:32:36.273503+0000","last_undegraded":"2021-03-15T22:32:36.273503+0000","last_fullsized":"2021-03-15T22:32:36.273503+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:05.123425+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.123425+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"376","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283753+0000","last_change":"2021-03-15T22:30:41.759292+0000","last_active":"2021-03-15T22:32:39.283753+0000","last_peered":"2021-03-15T22:32:39.283753+0000","last_clean":"2021-03-15T22:32:39.283753+0000","last_became_active":"2021-03-15T22:28:59.277385+0000","last_became_peered":"2021-03-15T22:28:59.277385+0000","last_unstale":"2021-03-15T22:32:39.283753+0000","last_undegraded":"2021-03-15T22:32:39.283753+0000","last_fullsized":"2021-03-15T22:32:39.283753+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:30:41.759224+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.759224+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"817","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283200+0000","last_change":"2021-03-15T22:32:34.739698+0000","last_active":"2021-03-15T22:32:39.283200+0000","last_peered":"2021-03-15T22:32:39.283200+0000","last_clean":"2021-03-15T22:32:39.283200+0000","last_became_active":"2021-03-15T22:31:23.258435+0000","last_became_peered":"2021-03-15T22:31:23.258435+0000","last_unstale":"2021-03-15T22:32:39.283200+0000","last_undegraded":"2021-03-15T22:32:39.283200+0000","last_fullsized":"2021-03-15T22:32:39.283200+0000","mapping_epoch":733,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:32:34.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.739637+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"808","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.274330+0000","last_change":"2021-03-15T22:31:40.870498+0000","last_active":"2021-03-15T22:32:36.274330+0000","last_peered":"2021-03-15T22:32:36.274330+0000","last_clean":"2021-03-15T22:32:36.274330+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:32:36.274330+0000","last_undegraded":"2021-03-15T22:32:36.274330+0000","last_fullsized":"2021-03-15T22:32:36.274330+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:40.870404+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:40.870404+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"409","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.265809+0000","last_change":"2021-03-15T22:31:10.332382+0000","last_active":"2021-03-15T22:32:35.265809+0000","last_peered":"2021-03-15T22:32:35.265809+0000","last_clean":"2021-03-15T22:32:35.265809+0000","last_became_active":"2021-03-15T22:29:00.220073+0000","last_became_peered":"2021-03-15T22:29:00.220073+0000","last_unstale":"2021-03-15T22:32:35.265809+0000","last_undegraded":"2021-03-15T22:32:35.265809+0000","last_fullsized":"2021-03-15T22:32:35.265809+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:10.332319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.332319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"798","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523983+0000","last_change":"2021-03-15T22:31:28.320040+0000","last_active":"2021-03-15T22:32:37.523983+0000","last_peered":"2021-03-15T22:32:37.523983+0000","last_clean":"2021-03-15T22:32:37.523983+0000","last_became_active":"2021-03-15T22:31:28.287309+0000","last_became_peered":"2021-03-15T22:31:28.287309+0000","last_unstale":"2021-03-15T22:32:37.523983+0000","last_undegraded":"2021-03-15T22:32:37.523983+0000","last_fullsized":"2021-03-15T22:32:37.523983+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:31:03.281344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:03.281344+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"783","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.278485+0000","last_change":"2021-03-15T22:30:59.480779+0000","last_active":"2021-03-15T22:32:37.278485+0000","last_peered":"2021-03-15T22:32:37.278485+0000","last_clean":"2021-03-15T22:32:37.278485+0000","last_became_active":"2021-03-15T22:30:31.230649+0000","last_became_peered":"2021-03-15T22:30:31.230649+0000","last_unstale":"2021-03-15T22:32:37.278485+0000","last_undegraded":"2021-03-15T22:32:37.278485+0000","last_fullsized":"2021-03-15T22:32:37.278485+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:59.480717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:30:59.480717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"80","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.262521+0000","last_change":"2021-03-15T22:31:28.278722+0000","last_active":"2021-03-15T22:32:35.262521+0000","last_peered":"2021-03-15T22:32:35.262521+0000","last_clean":"2021-03-15T22:32:35.262521+0000","last_became_active":"2021-03-15T22:31:28.278193+0000","last_became_peered":"2021-03-15T22:31:28.278193+0000","last_unstale":"2021-03-15T22:32:35.262521+0000","last_undegraded":"2021-03-15T22:32:35.262521+0000","last_fullsized":"2021-03-15T22:32:35.262521+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:06.296662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:06.296662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"315","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283776+0000","last_change":"2021-03-15T22:31:33.136177+0000","last_active":"2021-03-15T22:32:39.283776+0000","last_peered":"2021-03-15T22:32:39.283776+0000","last_clean":"2021-03-15T22:32:39.283776+0000","last_became_active":"2021-03-15T22:29:51.226677+0000","last_became_peered":"2021-03-15T22:29:51.226677+0000","last_unstale":"2021-03-15T22:32:39.283776+0000","last_undegraded":"2021-03-15T22:32:39.283776+0000","last_fullsized":"2021-03-15T22:32:39.283776+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:33.136112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:31:33.136112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"283","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.265525+0000","last_change":"2021-03-15T22:31:28.306827+0000","last_active":"2021-03-15T22:32:35.265525+0000","last_peered":"2021-03-15T22:32:35.265525+0000","last_clean":"2021-03-15T22:32:35.265525+0000","last_became_active":"2021-03-15T22:31:28.284715+0000","last_became_peered":"2021-03-15T22:31:28.284715+0000","last_unstale":"2021-03-15T22:32:35.265525+0000","last_undegraded":"2021-03-15T22:32:35.265525+0000","last_fullsized":"2021-03-15T22:32:35.265525+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:30:07.486190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.486190+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":7512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"418","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.264337+0000","last_change":"2021-03-15T22:31:28.288399+0000","last_active":"2021-03-15T22:32:35.264337+0000","last_peered":"2021-03-15T22:32:35.264337+0000","last_clean":"2021-03-15T22:32:35.264337+0000","last_became_active":"2021-03-15T22:31:28.288084+0000","last_became_peered":"2021-03-15T22:31:28.288084+0000","last_unstale":"2021-03-15T22:32:35.264337+0000","last_undegraded":"2021-03-15T22:32:35.264337+0000","last_fullsized":"2021-03-15T22:32:35.264337+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.317817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"779","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.278909+0000","last_change":"2021-03-15T22:32:05.522466+0000","last_active":"2021-03-15T22:32:37.278909+0000","last_peered":"2021-03-15T22:32:37.278909+0000","last_clean":"2021-03-15T22:32:37.278909+0000","last_became_active":"2021-03-15T22:31:28.281921+0000","last_became_peered":"2021-03-15T22:31:28.281921+0000","last_unstale":"2021-03-15T22:32:37.278909+0000","last_undegraded":"2021-03-15T22:32:37.278909+0000","last_fullsized":"2021-03-15T22:32:37.278909+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:05.522372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.522372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"179","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523783+0000","last_change":"2021-03-15T22:30:42.233176+0000","last_active":"2021-03-15T22:32:37.523783+0000","last_peered":"2021-03-15T22:32:37.523783+0000","last_clean":"2021-03-15T22:32:37.523783+0000","last_became_active":"2021-03-15T22:29:51.220261+0000","last_became_peered":"2021-03-15T22:29:51.220261+0000","last_unstale":"2021-03-15T22:32:37.523783+0000","last_undegraded":"2021-03-15T22:32:37.523783+0000","last_fullsized":"2021-03-15T22:32:37.523783+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.233098+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"618","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.261122+0000","last_change":"2021-03-15T22:31:28.302192+0000","last_active":"2021-03-15T22:32:35.261122+0000","last_peered":"2021-03-15T22:32:35.261122+0000","last_clean":"2021-03-15T22:32:35.261122+0000","last_became_active":"2021-03-15T22:31:28.283474+0000","last_became_peered":"2021-03-15T22:31:28.283474+0000","last_unstale":"2021-03-15T22:32:35.261122+0000","last_undegraded":"2021-03-15T22:32:35.261122+0000","last_fullsized":"2021-03-15T22:32:35.261122+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:31:07.330960+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:31:07.330960+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"295","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.282067+0000","last_change":"2021-03-15T22:31:43.405661+0000","last_active":"2021-03-15T22:32:39.282067+0000","last_peered":"2021-03-15T22:32:39.282067+0000","last_clean":"2021-03-15T22:32:39.282067+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:32:39.282067+0000","last_undegraded":"2021-03-15T22:32:39.282067+0000","last_fullsized":"2021-03-15T22:32:39.282067+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:43.405600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:31:43.405600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"594","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263976+0000","last_change":"2021-03-15T22:29:56.525891+0000","last_active":"2021-03-15T22:32:35.263976+0000","last_peered":"2021-03-15T22:32:35.263976+0000","last_clean":"2021-03-15T22:32:35.263976+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:32:35.263976+0000","last_undegraded":"2021-03-15T22:32:35.263976+0000","last_fullsized":"2021-03-15T22:32:35.263976+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:56.525822+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:29:56.525822+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"810","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.279209+0000","last_change":"2021-03-15T22:32:33.438804+0000","last_active":"2021-03-15T22:32:37.279209+0000","last_peered":"2021-03-15T22:32:37.279209+0000","last_clean":"2021-03-15T22:32:37.279209+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:32:37.279209+0000","last_undegraded":"2021-03-15T22:32:37.279209+0000","last_fullsized":"2021-03-15T22:32:37.279209+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.438742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.438742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"536","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.273995+0000","last_change":"2021-03-15T22:32:08.880092+0000","last_active":"2021-03-15T22:32:36.273995+0000","last_peered":"2021-03-15T22:32:36.273995+0000","last_clean":"2021-03-15T22:32:36.273995+0000","last_became_active":"2021-03-15T22:31:28.276464+0000","last_became_peered":"2021-03-15T22:31:28.276464+0000","last_unstale":"2021-03-15T22:32:36.273995+0000","last_undegraded":"2021-03-15T22:32:36.273995+0000","last_fullsized":"2021-03-15T22:32:36.273995+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:32:08.879999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:08.879999+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"431","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.260894+0000","last_change":"2021-03-15T22:32:06.670385+0000","last_active":"2021-03-15T22:32:35.260894+0000","last_peered":"2021-03-15T22:32:35.260894+0000","last_clean":"2021-03-15T22:32:35.260894+0000","last_became_active":"2021-03-15T22:29:51.218982+0000","last_became_peered":"2021-03-15T22:29:51.218982+0000","last_unstale":"2021-03-15T22:32:35.260894+0000","last_undegraded":"2021-03-15T22:32:35.260894+0000","last_fullsized":"2021-03-15T22:32:35.260894+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:06.670324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:32:06.670324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"852","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.273767+0000","last_change":"2021-03-15T22:32:30.719489+0000","last_active":"2021-03-15T22:32:36.273767+0000","last_peered":"2021-03-15T22:32:36.273767+0000","last_clean":"2021-03-15T22:32:36.273767+0000","last_became_active":"2021-03-15T22:29:51.222254+0000","last_became_peered":"2021-03-15T22:29:51.222254+0000","last_unstale":"2021-03-15T22:32:36.273767+0000","last_undegraded":"2021-03-15T22:32:36.273767+0000","last_fullsized":"2021-03-15T22:32:36.273767+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:30.719431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.719431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"78","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263281+0000","last_change":"2021-03-15T22:31:28.279172+0000","last_active":"2021-03-15T22:32:35.263281+0000","last_peered":"2021-03-15T22:32:35.263281+0000","last_clean":"2021-03-15T22:32:35.263281+0000","last_became_active":"2021-03-15T22:31:28.278787+0000","last_became_peered":"2021-03-15T22:31:28.278787+0000","last_unstale":"2021-03-15T22:32:35.263281+0000","last_undegraded":"2021-03-15T22:32:35.263281+0000","last_fullsized":"2021-03-15T22:32:35.263281+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.992352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.992352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"858","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.265049+0000","last_change":"2021-03-15T22:31:42.397085+0000","last_active":"2021-03-15T22:32:35.265049+0000","last_peered":"2021-03-15T22:32:35.265049+0000","last_clean":"2021-03-15T22:32:35.265049+0000","last_became_active":"2021-03-15T22:31:28.282759+0000","last_became_peered":"2021-03-15T22:31:28.282759+0000","last_unstale":"2021-03-15T22:32:35.265049+0000","last_undegraded":"2021-03-15T22:32:35.265049+0000","last_fullsized":"2021-03-15T22:32:35.265049+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:31:42.396996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.396996+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"289","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.284019+0000","last_change":"2021-03-15T22:30:42.771039+0000","last_active":"2021-03-15T22:32:39.284019+0000","last_peered":"2021-03-15T22:32:39.284019+0000","last_clean":"2021-03-15T22:32:39.284019+0000","last_became_active":"2021-03-15T22:28:59.276812+0000","last_became_peered":"2021-03-15T22:28:59.276812+0000","last_unstale":"2021-03-15T22:32:39.284019+0000","last_undegraded":"2021-03-15T22:32:39.284019+0000","last_fullsized":"2021-03-15T22:32:39.284019+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:42.770976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.770976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"808'9","reported_seq":"223","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.881054+0000","last_change":"2021-03-15T22:32:38.881054+0000","last_active":"2021-03-15T22:32:38.881054+0000","last_peered":"2021-03-15T22:32:38.881054+0000","last_clean":"2021-03-15T22:32:38.881054+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:32:38.881054+0000","last_undegraded":"2021-03-15T22:32:38.881054+0000","last_fullsized":"2021-03-15T22:32:38.881054+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9","last_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_deep_scrub":"808'9","last_deep_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.880993+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"784","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.278719+0000","last_change":"2021-03-15T22:30:35.423621+0000","last_active":"2021-03-15T22:32:37.278719+0000","last_peered":"2021-03-15T22:32:37.278719+0000","last_clean":"2021-03-15T22:32:37.278719+0000","last_became_active":"2021-03-15T22:30:31.230560+0000","last_became_peered":"2021-03-15T22:30:31.230560+0000","last_unstale":"2021-03-15T22:32:37.278719+0000","last_undegraded":"2021-03-15T22:32:37.278719+0000","last_fullsized":"2021-03-15T22:32:37.278719+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:35.423559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:35.423559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"556'24","reported_seq":"593","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.262563+0000","last_change":"2021-03-15T22:30:02.548134+0000","last_active":"2021-03-15T22:32:35.262563+0000","last_peered":"2021-03-15T22:32:35.262563+0000","last_clean":"2021-03-15T22:32:35.262563+0000","last_became_active":"2021-03-15T22:29:00.226652+0000","last_became_peered":"2021-03-15T22:29:00.226652+0000","last_unstale":"2021-03-15T22:32:35.262563+0000","last_undegraded":"2021-03-15T22:32:35.262563+0000","last_fullsized":"2021-03-15T22:32:35.262563+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'24","last_scrub_stamp":"2021-03-15T22:30:02.548069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.548069+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"820","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263463+0000","last_change":"2021-03-15T22:31:41.407408+0000","last_active":"2021-03-15T22:32:35.263463+0000","last_peered":"2021-03-15T22:32:35.263463+0000","last_clean":"2021-03-15T22:32:35.263463+0000","last_became_active":"2021-03-15T22:30:32.247782+0000","last_became_peered":"2021-03-15T22:30:32.247782+0000","last_unstale":"2021-03-15T22:32:35.263463+0000","last_undegraded":"2021-03-15T22:32:35.263463+0000","last_fullsized":"2021-03-15T22:32:35.263463+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:41.407324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.407324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"683'14","reported_seq":"492","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.262417+0000","last_change":"2021-03-15T22:31:41.551496+0000","last_active":"2021-03-15T22:32:35.262417+0000","last_peered":"2021-03-15T22:32:35.262417+0000","last_clean":"2021-03-15T22:32:35.262417+0000","last_became_active":"2021-03-15T22:30:32.252383+0000","last_became_peered":"2021-03-15T22:30:32.252383+0000","last_unstale":"2021-03-15T22:32:35.262417+0000","last_undegraded":"2021-03-15T22:32:35.262417+0000","last_fullsized":"2021-03-15T22:32:35.262417+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'14","last_scrub_stamp":"2021-03-15T22:31:41.551435+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.551435+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"294","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.282796+0000","last_change":"2021-03-15T22:32:03.502945+0000","last_active":"2021-03-15T22:32:39.282796+0000","last_peered":"2021-03-15T22:32:39.282796+0000","last_clean":"2021-03-15T22:32:39.282796+0000","last_became_active":"2021-03-15T22:28:59.265822+0000","last_became_peered":"2021-03-15T22:28:59.265822+0000","last_unstale":"2021-03-15T22:32:39.282796+0000","last_undegraded":"2021-03-15T22:32:39.282796+0000","last_fullsized":"2021-03-15T22:32:39.282796+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:03.502854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:32:03.502854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"197","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523694+0000","last_change":"2021-03-15T22:30:36.106413+0000","last_active":"2021-03-15T22:32:37.523694+0000","last_peered":"2021-03-15T22:32:37.523694+0000","last_clean":"2021-03-15T22:32:37.523694+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:32:37.523694+0000","last_undegraded":"2021-03-15T22:32:37.523694+0000","last_fullsized":"2021-03-15T22:32:37.523694+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:30:36.106285+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.106285+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"84","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.265235+0000","last_change":"2021-03-15T22:31:20.257902+0000","last_active":"2021-03-15T22:32:35.265235+0000","last_peered":"2021-03-15T22:32:35.265235+0000","last_clean":"2021-03-15T22:32:35.265235+0000","last_became_active":"2021-03-15T22:31:20.256557+0000","last_became_peered":"2021-03-15T22:31:20.256557+0000","last_unstale":"2021-03-15T22:32:35.265235+0000","last_undegraded":"2021-03-15T22:32:35.265235+0000","last_fullsized":"2021-03-15T22:32:35.265235+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:11.436723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.436723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"584","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.276543+0000","last_change":"2021-03-15T22:32:12.036589+0000","last_active":"2021-03-15T22:32:38.276543+0000","last_peered":"2021-03-15T22:32:38.276543+0000","last_clean":"2021-03-15T22:32:38.276543+0000","last_became_active":"2021-03-15T22:30:31.254532+0000","last_became_peered":"2021-03-15T22:30:31.254532+0000","last_unstale":"2021-03-15T22:32:38.276543+0000","last_undegraded":"2021-03-15T22:32:38.276543+0000","last_fullsized":"2021-03-15T22:32:38.276543+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.036522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.036522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"788","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.278564+0000","last_change":"2021-03-15T22:31:34.511881+0000","last_active":"2021-03-15T22:32:37.278564+0000","last_peered":"2021-03-15T22:32:37.278564+0000","last_clean":"2021-03-15T22:32:37.278564+0000","last_became_active":"2021-03-15T22:31:28.276738+0000","last_became_peered":"2021-03-15T22:31:28.276738+0000","last_unstale":"2021-03-15T22:32:37.278564+0000","last_undegraded":"2021-03-15T22:32:37.278564+0000","last_fullsized":"2021-03-15T22:32:37.278564+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:34.511804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:34.511804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"334","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283366+0000","last_change":"2021-03-15T22:31:04.009824+0000","last_active":"2021-03-15T22:32:39.283366+0000","last_peered":"2021-03-15T22:32:39.283366+0000","last_clean":"2021-03-15T22:32:39.283366+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:32:39.283366+0000","last_undegraded":"2021-03-15T22:32:39.283366+0000","last_fullsized":"2021-03-15T22:32:39.283366+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:31:04.009748+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.009748+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"604","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523786+0000","last_change":"2021-03-15T22:32:11.834186+0000","last_active":"2021-03-15T22:32:37.523786+0000","last_peered":"2021-03-15T22:32:37.523786+0000","last_clean":"2021-03-15T22:32:37.523786+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:32:37.523786+0000","last_undegraded":"2021-03-15T22:32:37.523786+0000","last_fullsized":"2021-03-15T22:32:37.523786+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:11.834104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:11.834104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"782","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.279132+0000","last_change":"2021-03-15T22:31:35.559671+0000","last_active":"2021-03-15T22:32:37.279132+0000","last_peered":"2021-03-15T22:32:37.279132+0000","last_clean":"2021-03-15T22:32:37.279132+0000","last_became_active":"2021-03-15T22:31:28.286252+0000","last_became_peered":"2021-03-15T22:31:28.286252+0000","last_unstale":"2021-03-15T22:32:37.279132+0000","last_undegraded":"2021-03-15T22:32:37.279132+0000","last_fullsized":"2021-03-15T22:32:37.279132+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:35.559607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:35.559607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"772","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.698025+0000","last_change":"2021-03-15T22:32:39.697998+0000","last_active":"2021-03-15T22:32:39.698025+0000","last_peered":"2021-03-15T22:32:39.698025+0000","last_clean":"2021-03-15T22:32:39.698025+0000","last_became_active":"2021-03-15T22:31:28.287285+0000","last_became_peered":"2021-03-15T22:31:28.287285+0000","last_unstale":"2021-03-15T22:32:39.698025+0000","last_undegraded":"2021-03-15T22:32:39.698025+0000","last_fullsized":"2021-03-15T22:32:39.698025+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.697935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:39.697935+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.697935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"435","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.265111+0000","last_change":"2021-03-15T22:32:01.586874+0000","last_active":"2021-03-15T22:32:35.265111+0000","last_peered":"2021-03-15T22:32:35.265111+0000","last_clean":"2021-03-15T22:32:35.265111+0000","last_became_active":"2021-03-15T22:29:11.204254+0000","last_became_peered":"2021-03-15T22:29:11.204254+0000","last_unstale":"2021-03-15T22:32:35.265111+0000","last_undegraded":"2021-03-15T22:32:35.265111+0000","last_fullsized":"2021-03-15T22:32:35.265111+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:32:01.586755+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:32:01.586755+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"805'1016","reported_seq":"3465","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.274107+0000","last_change":"2021-03-15T22:32:35.281455+0000","last_active":"2021-03-15T22:32:36.274107+0000","last_peered":"2021-03-15T22:32:36.274107+0000","last_clean":"2021-03-15T22:32:36.274107+0000","last_became_active":"2021-03-15T22:31:31.240075+0000","last_became_peered":"2021-03-15T22:31:31.240075+0000","last_unstale":"2021-03-15T22:32:36.274107+0000","last_undegraded":"2021-03-15T22:32:36.274107+0000","last_fullsized":"2021-03-15T22:32:36.274107+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'874","last_scrub_stamp":"2021-03-15T22:31:34.960268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:31:34.960268+0000","log_size":1016,"ondisk_log_size":1016,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33119903,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":404,"num_read_kb":455535,"num_write":271,"num_write_kb":86607,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":108,"num_bytes_recovered":59821512,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"737","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523570+0000","last_change":"2021-03-15T22:31:28.283539+0000","last_active":"2021-03-15T22:32:37.523570+0000","last_peered":"2021-03-15T22:32:37.523570+0000","last_clean":"2021-03-15T22:32:37.523570+0000","last_became_active":"2021-03-15T22:31:28.283092+0000","last_became_peered":"2021-03-15T22:31:28.283092+0000","last_unstale":"2021-03-15T22:32:37.523570+0000","last_undegraded":"2021-03-15T22:32:37.523570+0000","last_fullsized":"2021-03-15T22:32:37.523570+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:07.417736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:07.417736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"812","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.279031+0000","last_change":"2021-03-15T22:31:58.607690+0000","last_active":"2021-03-15T22:32:37.279031+0000","last_peered":"2021-03-15T22:32:37.279031+0000","last_clean":"2021-03-15T22:32:37.279031+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:32:37.279031+0000","last_undegraded":"2021-03-15T22:32:37.279031+0000","last_fullsized":"2021-03-15T22:32:37.279031+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:31:58.607619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:58.607619+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"242","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523124+0000","last_change":"2021-03-15T22:31:29.162156+0000","last_active":"2021-03-15T22:32:37.523124+0000","last_peered":"2021-03-15T22:32:37.523124+0000","last_clean":"2021-03-15T22:32:37.523124+0000","last_became_active":"2021-03-15T22:31:28.288606+0000","last_became_peered":"2021-03-15T22:31:28.288606+0000","last_unstale":"2021-03-15T22:32:37.523124+0000","last_undegraded":"2021-03-15T22:32:37.523124+0000","last_fullsized":"2021-03-15T22:32:37.523124+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.227473+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"801'11","reported_seq":"1010","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.278765+0000","last_change":"2021-03-15T22:32:31.265096+0000","last_active":"2021-03-15T22:32:37.278765+0000","last_peered":"2021-03-15T22:32:37.278765+0000","last_clean":"2021-03-15T22:32:37.278765+0000","last_became_active":"2021-03-15T22:32:31.264668+0000","last_became_peered":"2021-03-15T22:32:31.264668+0000","last_unstale":"2021-03-15T22:32:37.278765+0000","last_undegraded":"2021-03-15T22:32:37.278765+0000","last_fullsized":"2021-03-15T22:32:37.278765+0000","mapping_epoch":800,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":801,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:30:07.252960+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.252960+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"805'1406","reported_seq":"3733","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.285905+0000","last_change":"2021-03-15T22:32:35.285905+0000","last_active":"2021-03-15T22:32:35.285905+0000","last_peered":"2021-03-15T22:32:35.285905+0000","last_clean":"2021-03-15T22:32:35.285905+0000","last_became_active":"2021-03-15T22:30:34.291079+0000","last_became_peered":"2021-03-15T22:30:34.291079+0000","last_unstale":"2021-03-15T22:32:35.285905+0000","last_undegraded":"2021-03-15T22:32:35.285905+0000","last_fullsized":"2021-03-15T22:32:35.285905+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1312","last_scrub_stamp":"2021-03-15T22:32:05.639057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.639057+0000","log_size":1406,"ondisk_log_size":1406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":53615594,"num_objects":25,"num_object_clones":20,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":536,"num_read_kb":520708,"num_write":386,"num_write_kb":158842,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":80476435,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"230","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263841+0000","last_change":"2021-03-15T22:29:58.503593+0000","last_active":"2021-03-15T22:32:35.263841+0000","last_peered":"2021-03-15T22:32:35.263841+0000","last_clean":"2021-03-15T22:32:35.263841+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:32:35.263841+0000","last_undegraded":"2021-03-15T22:32:35.263841+0000","last_fullsized":"2021-03-15T22:32:35.263841+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:58.503533+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:29:58.503533+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"781","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.279356+0000","last_change":"2021-03-15T22:31:39.517144+0000","last_active":"2021-03-15T22:32:37.279356+0000","last_peered":"2021-03-15T22:32:37.279356+0000","last_clean":"2021-03-15T22:32:37.279356+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:32:37.279356+0000","last_undegraded":"2021-03-15T22:32:37.279356+0000","last_fullsized":"2021-03-15T22:32:37.279356+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:39.517052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:39.517052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"956","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263059+0000","last_change":"2021-03-15T22:32:03.948964+0000","last_active":"2021-03-15T22:32:35.263059+0000","last_peered":"2021-03-15T22:32:35.263059+0000","last_clean":"2021-03-15T22:32:35.263059+0000","last_became_active":"2021-03-15T22:32:03.258590+0000","last_became_peered":"2021-03-15T22:32:03.258590+0000","last_unstale":"2021-03-15T22:32:35.263059+0000","last_undegraded":"2021-03-15T22:32:35.263059+0000","last_fullsized":"2021-03-15T22:32:35.263059+0000","mapping_epoch":772,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":773,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:31:05.469001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.469001+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":3460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"639","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.275948+0000","last_change":"2021-03-15T22:31:42.055712+0000","last_active":"2021-03-15T22:32:38.275948+0000","last_peered":"2021-03-15T22:32:38.275948+0000","last_clean":"2021-03-15T22:32:38.275948+0000","last_became_active":"2021-03-15T22:31:18.252524+0000","last_became_peered":"2021-03-15T22:31:18.252524+0000","last_unstale":"2021-03-15T22:32:38.275948+0000","last_undegraded":"2021-03-15T22:32:38.275948+0000","last_fullsized":"2021-03-15T22:32:38.275948+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"809'396","reported_seq":"2596","reported_epoch":"810","state":"active+clean","last_fresh":"2021-03-15T22:32:40.263546+0000","last_change":"2021-03-15T22:32:39.290275+0000","last_active":"2021-03-15T22:32:40.263546+0000","last_peered":"2021-03-15T22:32:40.263546+0000","last_clean":"2021-03-15T22:32:40.263546+0000","last_became_active":"2021-03-15T22:28:16.177364+0000","last_became_peered":"2021-03-15T22:28:16.177364+0000","last_unstale":"2021-03-15T22:32:40.263546+0000","last_undegraded":"2021-03-15T22:32:40.263546+0000","last_fullsized":"2021-03-15T22:32:40.263546+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'293","last_scrub_stamp":"2021-03-15T22:30:29.729234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.729234+0000","log_size":396,"ondisk_log_size":396,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10745993,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":106,"num_read_kb":79087,"num_write":107,"num_write_kb":35163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":11148008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22d","length":"1"},{"start":"267","length":"1"},{"start":"26c","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"827","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.278494+0000","last_change":"2021-03-15T22:32:31.390410+0000","last_active":"2021-03-15T22:32:37.278494+0000","last_peered":"2021-03-15T22:32:37.278494+0000","last_clean":"2021-03-15T22:32:37.278494+0000","last_became_active":"2021-03-15T22:30:32.236752+0000","last_became_peered":"2021-03-15T22:30:32.236752+0000","last_unstale":"2021-03-15T22:32:37.278494+0000","last_undegraded":"2021-03-15T22:32:37.278494+0000","last_fullsized":"2021-03-15T22:32:37.278494+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"646","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.276759+0000","last_change":"2021-03-15T22:31:32.008461+0000","last_active":"2021-03-15T22:32:38.276759+0000","last_peered":"2021-03-15T22:32:38.276759+0000","last_clean":"2021-03-15T22:32:38.276759+0000","last_became_active":"2021-03-15T22:31:18.251403+0000","last_became_peered":"2021-03-15T22:31:18.251403+0000","last_unstale":"2021-03-15T22:32:38.276759+0000","last_undegraded":"2021-03-15T22:32:38.276759+0000","last_fullsized":"2021-03-15T22:32:38.276759+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:32.008403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:32.008403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"827","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.279401+0000","last_change":"2021-03-15T22:31:28.289675+0000","last_active":"2021-03-15T22:32:37.279401+0000","last_peered":"2021-03-15T22:32:37.279401+0000","last_clean":"2021-03-15T22:32:37.279401+0000","last_became_active":"2021-03-15T22:31:28.288901+0000","last_became_peered":"2021-03-15T22:31:28.288901+0000","last_unstale":"2021-03-15T22:32:37.279401+0000","last_undegraded":"2021-03-15T22:32:37.279401+0000","last_fullsized":"2021-03-15T22:32:37.279401+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:08.279129+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:30:08.279129+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"865","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.277423+0000","last_change":"2021-03-15T22:31:28.308317+0000","last_active":"2021-03-15T22:32:38.277423+0000","last_peered":"2021-03-15T22:32:38.277423+0000","last_clean":"2021-03-15T22:32:38.277423+0000","last_became_active":"2021-03-15T22:31:28.284158+0000","last_became_peered":"2021-03-15T22:31:28.284158+0000","last_unstale":"2021-03-15T22:32:38.277423+0000","last_undegraded":"2021-03-15T22:32:38.277423+0000","last_fullsized":"2021-03-15T22:32:38.277423+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:31:01.245498+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:01.245498+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1464,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"809'807","reported_seq":"2723","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.296523+0000","last_change":"2021-03-15T22:32:39.296971+0000","last_active":"2021-03-15T22:32:39.296523+0000","last_peered":"2021-03-15T22:32:39.296523+0000","last_clean":"2021-03-15T22:32:39.296523+0000","last_became_active":"2021-03-15T22:30:31.242297+0000","last_became_peered":"2021-03-15T22:30:31.242297+0000","last_unstale":"2021-03-15T22:32:39.296523+0000","last_undegraded":"2021-03-15T22:32:39.296523+0000","last_fullsized":"2021-03-15T22:32:39.296523+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'783","last_scrub_stamp":"2021-03-15T22:32:31.034631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.034631+0000","log_size":807,"ondisk_log_size":807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36638004,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":361,"num_read_kb":441676,"num_write":211,"num_write_kb":69663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":24240388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"267","length":"2"},{"start":"26c","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"287","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.282238+0000","last_change":"2021-03-15T22:31:08.968657+0000","last_active":"2021-03-15T22:32:39.282238+0000","last_peered":"2021-03-15T22:32:39.282238+0000","last_clean":"2021-03-15T22:32:39.282238+0000","last_became_active":"2021-03-15T22:30:32.237720+0000","last_became_peered":"2021-03-15T22:30:32.237720+0000","last_unstale":"2021-03-15T22:32:39.282238+0000","last_undegraded":"2021-03-15T22:32:39.282238+0000","last_fullsized":"2021-03-15T22:32:39.282238+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"624","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.277600+0000","last_change":"2021-03-15T22:31:33.964163+0000","last_active":"2021-03-15T22:32:38.277600+0000","last_peered":"2021-03-15T22:32:38.277600+0000","last_clean":"2021-03-15T22:32:38.277600+0000","last_became_active":"2021-03-15T22:31:18.261916+0000","last_became_peered":"2021-03-15T22:31:18.261916+0000","last_unstale":"2021-03-15T22:32:38.277600+0000","last_undegraded":"2021-03-15T22:32:38.277600+0000","last_fullsized":"2021-03-15T22:32:38.277600+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:33.964082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:33.964082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"193","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523532+0000","last_change":"2021-03-15T22:31:41.775620+0000","last_active":"2021-03-15T22:32:37.523532+0000","last_peered":"2021-03-15T22:32:37.523532+0000","last_clean":"2021-03-15T22:32:37.523532+0000","last_became_active":"2021-03-15T22:31:28.282098+0000","last_became_peered":"2021-03-15T22:31:28.282098+0000","last_unstale":"2021-03-15T22:32:37.523532+0000","last_undegraded":"2021-03-15T22:32:37.523532+0000","last_fullsized":"2021-03-15T22:32:37.523532+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:41.775530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.775530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"325","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283710+0000","last_change":"2021-03-15T22:32:33.701761+0000","last_active":"2021-03-15T22:32:39.283710+0000","last_peered":"2021-03-15T22:32:39.283710+0000","last_clean":"2021-03-15T22:32:39.283710+0000","last_became_active":"2021-03-15T22:30:32.250712+0000","last_became_peered":"2021-03-15T22:30:32.250712+0000","last_unstale":"2021-03-15T22:32:39.283710+0000","last_undegraded":"2021-03-15T22:32:39.283710+0000","last_fullsized":"2021-03-15T22:32:39.283710+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:32:33.701675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.701675+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"809'1066","reported_seq":"3789","reported_epoch":"810","state":"active+clean","last_fresh":"2021-03-15T22:32:40.265556+0000","last_change":"2021-03-15T22:32:39.300499+0000","last_active":"2021-03-15T22:32:40.265556+0000","last_peered":"2021-03-15T22:32:40.265556+0000","last_clean":"2021-03-15T22:32:40.265556+0000","last_became_active":"2021-03-15T22:31:31.245658+0000","last_became_peered":"2021-03-15T22:31:31.245658+0000","last_unstale":"2021-03-15T22:32:40.265556+0000","last_undegraded":"2021-03-15T22:32:40.265556+0000","last_fullsized":"2021-03-15T22:32:40.265556+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'870","last_scrub_stamp":"2021-03-15T22:31:02.243396+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.243396+0000","log_size":1066,"ondisk_log_size":1066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32728028,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":250,"num_read_kb":196596,"num_write":276,"num_write_kb":94316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":267,"num_bytes_recovered":153074064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,1],"acting":[0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22d","length":"1"},{"start":"267","length":"1"},{"start":"26c","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"230","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.264595+0000","last_change":"2021-03-15T22:30:29.533509+0000","last_active":"2021-03-15T22:32:35.264595+0000","last_peered":"2021-03-15T22:32:35.264595+0000","last_clean":"2021-03-15T22:32:35.264595+0000","last_became_active":"2021-03-15T22:28:59.273785+0000","last_became_peered":"2021-03-15T22:28:59.273785+0000","last_unstale":"2021-03-15T22:32:35.264595+0000","last_undegraded":"2021-03-15T22:32:35.264595+0000","last_fullsized":"2021-03-15T22:32:35.264595+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:29.533421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.533421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"190","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523414+0000","last_change":"2021-03-15T22:31:58.785626+0000","last_active":"2021-03-15T22:32:37.523414+0000","last_peered":"2021-03-15T22:32:37.523414+0000","last_clean":"2021-03-15T22:32:37.523414+0000","last_became_active":"2021-03-15T22:31:28.287331+0000","last_became_peered":"2021-03-15T22:31:28.287331+0000","last_unstale":"2021-03-15T22:32:37.523414+0000","last_undegraded":"2021-03-15T22:32:37.523414+0000","last_fullsized":"2021-03-15T22:32:37.523414+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:58.785524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:31:58.785524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"155","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.276566+0000","last_change":"2021-03-15T22:32:33.052028+0000","last_active":"2021-03-15T22:32:38.276566+0000","last_peered":"2021-03-15T22:32:38.276566+0000","last_clean":"2021-03-15T22:32:38.276566+0000","last_became_active":"2021-03-15T22:31:28.278136+0000","last_became_peered":"2021-03-15T22:31:28.278136+0000","last_unstale":"2021-03-15T22:32:38.276566+0000","last_undegraded":"2021-03-15T22:32:38.276566+0000","last_fullsized":"2021-03-15T22:32:38.276566+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.051962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.051962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1023","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.274262+0000","last_change":"2021-03-15T22:32:06.881916+0000","last_active":"2021-03-15T22:32:36.274262+0000","last_peered":"2021-03-15T22:32:36.274262+0000","last_clean":"2021-03-15T22:32:36.274262+0000","last_became_active":"2021-03-15T22:30:32.244684+0000","last_became_peered":"2021-03-15T22:30:32.244684+0000","last_unstale":"2021-03-15T22:32:36.274262+0000","last_undegraded":"2021-03-15T22:32:36.274262+0000","last_fullsized":"2021-03-15T22:32:36.274262+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:32:06.881829+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:32:06.881829+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"271","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283218+0000","last_change":"2021-03-15T22:30:30.738196+0000","last_active":"2021-03-15T22:32:39.283218+0000","last_peered":"2021-03-15T22:32:39.283218+0000","last_clean":"2021-03-15T22:32:39.283218+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:32:39.283218+0000","last_undegraded":"2021-03-15T22:32:39.283218+0000","last_fullsized":"2021-03-15T22:32:39.283218+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:30.738108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:30.738108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"809'1007","reported_seq":"3170","reported_epoch":"810","state":"active+clean","last_fresh":"2021-03-15T22:32:40.265148+0000","last_change":"2021-03-15T22:32:39.293882+0000","last_active":"2021-03-15T22:32:40.265148+0000","last_peered":"2021-03-15T22:32:40.265148+0000","last_clean":"2021-03-15T22:32:40.265148+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:32:40.265148+0000","last_undegraded":"2021-03-15T22:32:40.265148+0000","last_fullsized":"2021-03-15T22:32:40.265148+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"805'997","last_scrub_stamp":"2021-03-15T22:32:36.795301+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.795301+0000","log_size":1007,"ondisk_log_size":1007,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26878994,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":300,"num_read_kb":294218,"num_write":231,"num_write_kb":87577,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22d","length":"1"},{"start":"267","length":"1"},{"start":"26c","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"819","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.279511+0000","last_change":"2021-03-15T22:32:32.417288+0000","last_active":"2021-03-15T22:32:37.279511+0000","last_peered":"2021-03-15T22:32:37.279511+0000","last_clean":"2021-03-15T22:32:37.279511+0000","last_became_active":"2021-03-15T22:31:28.289352+0000","last_became_peered":"2021-03-15T22:31:28.289352+0000","last_unstale":"2021-03-15T22:32:37.279511+0000","last_undegraded":"2021-03-15T22:32:37.279511+0000","last_fullsized":"2021-03-15T22:32:37.279511+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:32.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"808","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.274755+0000","last_change":"2021-03-15T22:32:14.793976+0000","last_active":"2021-03-15T22:32:36.274755+0000","last_peered":"2021-03-15T22:32:36.274755+0000","last_clean":"2021-03-15T22:32:36.274755+0000","last_became_active":"2021-03-15T22:31:28.276641+0000","last_became_peered":"2021-03-15T22:31:28.276641+0000","last_unstale":"2021-03-15T22:32:36.274755+0000","last_undegraded":"2021-03-15T22:32:36.274755+0000","last_fullsized":"2021-03-15T22:32:36.274755+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.793900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.793900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"612","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.276046+0000","last_change":"2021-03-15T22:32:34.095255+0000","last_active":"2021-03-15T22:32:38.276046+0000","last_peered":"2021-03-15T22:32:38.276046+0000","last_clean":"2021-03-15T22:32:38.276046+0000","last_became_active":"2021-03-15T22:30:32.250316+0000","last_became_peered":"2021-03-15T22:30:32.250316+0000","last_unstale":"2021-03-15T22:32:38.276046+0000","last_undegraded":"2021-03-15T22:32:38.276046+0000","last_fullsized":"2021-03-15T22:32:38.276046+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:34.095158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.095158+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"550","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.274593+0000","last_change":"2021-03-15T22:31:59.864528+0000","last_active":"2021-03-15T22:32:36.274593+0000","last_peered":"2021-03-15T22:32:36.274593+0000","last_clean":"2021-03-15T22:32:36.274593+0000","last_became_active":"2021-03-15T22:29:51.222262+0000","last_became_peered":"2021-03-15T22:29:51.222262+0000","last_unstale":"2021-03-15T22:32:36.274593+0000","last_undegraded":"2021-03-15T22:32:36.274593+0000","last_fullsized":"2021-03-15T22:32:36.274593+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:31:59.864460+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:31:59.864460+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"607","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.276216+0000","last_change":"2021-03-15T22:32:04.969553+0000","last_active":"2021-03-15T22:32:38.276216+0000","last_peered":"2021-03-15T22:32:38.276216+0000","last_clean":"2021-03-15T22:32:38.276216+0000","last_became_active":"2021-03-15T22:31:18.249044+0000","last_became_peered":"2021-03-15T22:31:18.249044+0000","last_unstale":"2021-03-15T22:32:38.276216+0000","last_undegraded":"2021-03-15T22:32:38.276216+0000","last_fullsized":"2021-03-15T22:32:38.276216+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:04.969491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:04.969491+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"809'2439","reported_seq":"5189","reported_epoch":"810","state":"active+clean","last_fresh":"2021-03-15T22:32:40.266478+0000","last_change":"2021-03-15T22:32:39.310791+0000","last_active":"2021-03-15T22:32:40.266478+0000","last_peered":"2021-03-15T22:32:40.266478+0000","last_clean":"2021-03-15T22:32:40.266478+0000","last_became_active":"2021-03-15T22:28:59.276888+0000","last_became_peered":"2021-03-15T22:28:59.276888+0000","last_unstale":"2021-03-15T22:32:40.266478+0000","last_undegraded":"2021-03-15T22:32:40.266478+0000","last_fullsized":"2021-03-15T22:32:40.266478+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'2007","last_scrub_stamp":"2021-03-15T22:31:02.050395+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.050395+0000","log_size":2439,"ondisk_log_size":2439,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":69589999,"num_objects":28,"num_object_clones":21,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":1,"num_read":725,"num_read_kb":728262,"num_write":569,"num_write_kb":236854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":205133613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22d","length":"1"},{"start":"267","length":"1"},{"start":"26c","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"179","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523866+0000","last_change":"2021-03-15T22:30:28.967439+0000","last_active":"2021-03-15T22:32:37.523866+0000","last_peered":"2021-03-15T22:32:37.523866+0000","last_clean":"2021-03-15T22:32:37.523866+0000","last_became_active":"2021-03-15T22:29:51.227896+0000","last_became_peered":"2021-03-15T22:29:51.227896+0000","last_unstale":"2021-03-15T22:32:37.523866+0000","last_undegraded":"2021-03-15T22:32:37.523866+0000","last_fullsized":"2021-03-15T22:32:37.523866+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:28.967356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.967356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"806","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.278647+0000","last_change":"2021-03-15T22:31:28.275169+0000","last_active":"2021-03-15T22:32:37.278647+0000","last_peered":"2021-03-15T22:32:37.278647+0000","last_clean":"2021-03-15T22:32:37.278647+0000","last_became_active":"2021-03-15T22:31:28.274813+0000","last_became_peered":"2021-03-15T22:31:28.274813+0000","last_unstale":"2021-03-15T22:32:37.278647+0000","last_undegraded":"2021-03-15T22:32:37.278647+0000","last_fullsized":"2021-03-15T22:32:37.278647+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:05.206223+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:30:05.206223+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"427","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.262128+0000","last_change":"2021-03-15T22:31:39.555544+0000","last_active":"2021-03-15T22:32:35.262128+0000","last_peered":"2021-03-15T22:32:35.262128+0000","last_clean":"2021-03-15T22:32:35.262128+0000","last_became_active":"2021-03-15T22:31:28.285608+0000","last_became_peered":"2021-03-15T22:31:28.285608+0000","last_unstale":"2021-03-15T22:32:35.262128+0000","last_undegraded":"2021-03-15T22:32:35.262128+0000","last_fullsized":"2021-03-15T22:32:35.262128+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:31:39.555487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,4],"acting":[3,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"669","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263121+0000","last_change":"2021-03-15T22:31:28.303171+0000","last_active":"2021-03-15T22:32:35.263121+0000","last_peered":"2021-03-15T22:32:35.263121+0000","last_clean":"2021-03-15T22:32:35.263121+0000","last_became_active":"2021-03-15T22:31:28.277985+0000","last_became_peered":"2021-03-15T22:31:28.277985+0000","last_unstale":"2021-03-15T22:32:35.263121+0000","last_undegraded":"2021-03-15T22:32:35.263121+0000","last_fullsized":"2021-03-15T22:32:35.263121+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:30:28.176831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.176831+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"818","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.279346+0000","last_change":"2021-03-15T22:30:32.244033+0000","last_active":"2021-03-15T22:32:37.279346+0000","last_peered":"2021-03-15T22:32:37.279346+0000","last_clean":"2021-03-15T22:32:37.279346+0000","last_became_active":"2021-03-15T22:30:32.243696+0000","last_became_peered":"2021-03-15T22:30:32.243696+0000","last_unstale":"2021-03-15T22:32:37.279346+0000","last_undegraded":"2021-03-15T22:32:37.279346+0000","last_fullsized":"2021-03-15T22:32:37.279346+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.257476+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"808'711","reported_seq":"3026","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.282904+0000","last_change":"2021-03-15T22:32:38.283125+0000","last_active":"2021-03-15T22:32:38.282904+0000","last_peered":"2021-03-15T22:32:38.282904+0000","last_clean":"2021-03-15T22:32:38.282904+0000","last_became_active":"2021-03-15T22:30:35.366323+0000","last_became_peered":"2021-03-15T22:30:35.366323+0000","last_unstale":"2021-03-15T22:32:38.282904+0000","last_undegraded":"2021-03-15T22:32:38.282904+0000","last_fullsized":"2021-03-15T22:32:38.282904+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'659","last_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:32:11.467520+0000","log_size":711,"ondisk_log_size":711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21805658,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":178,"num_read_kb":168571,"num_write":173,"num_write_kb":76419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":54764539,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"26c","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"850","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.274110+0000","last_change":"2021-03-15T22:32:12.830549+0000","last_active":"2021-03-15T22:32:36.274110+0000","last_peered":"2021-03-15T22:32:36.274110+0000","last_clean":"2021-03-15T22:32:36.274110+0000","last_became_active":"2021-03-15T22:29:51.222189+0000","last_became_peered":"2021-03-15T22:29:51.222189+0000","last_unstale":"2021-03-15T22:32:36.274110+0000","last_undegraded":"2021-03-15T22:32:36.274110+0000","last_fullsized":"2021-03-15T22:32:36.274110+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"292","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.282296+0000","last_change":"2021-03-15T22:31:01.010456+0000","last_active":"2021-03-15T22:32:39.282296+0000","last_peered":"2021-03-15T22:32:39.282296+0000","last_clean":"2021-03-15T22:32:39.282296+0000","last_became_active":"2021-03-15T22:28:59.276599+0000","last_became_peered":"2021-03-15T22:28:59.276599+0000","last_unstale":"2021-03-15T22:32:39.282296+0000","last_undegraded":"2021-03-15T22:32:39.282296+0000","last_fullsized":"2021-03-15T22:32:39.282296+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:01.010393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:01.010393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"1684","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.276972+0000","last_change":"2021-03-15T22:32:30.365491+0000","last_active":"2021-03-15T22:32:38.276972+0000","last_peered":"2021-03-15T22:32:38.276972+0000","last_clean":"2021-03-15T22:32:38.276972+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:32:38.276972+0000","last_undegraded":"2021-03-15T22:32:38.276972+0000","last_fullsized":"2021-03-15T22:32:38.276972+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:32:30.365432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.365432+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":869,"num_read_kb":864,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"339","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283838+0000","last_change":"2021-03-15T22:31:11.926326+0000","last_active":"2021-03-15T22:32:39.283838+0000","last_peered":"2021-03-15T22:32:39.283838+0000","last_clean":"2021-03-15T22:32:39.283838+0000","last_became_active":"2021-03-15T22:30:32.251686+0000","last_became_peered":"2021-03-15T22:30:32.251686+0000","last_unstale":"2021-03-15T22:32:39.283838+0000","last_undegraded":"2021-03-15T22:32:39.283838+0000","last_fullsized":"2021-03-15T22:32:39.283838+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:31:11.926268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.926268+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"193","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.523909+0000","last_change":"2021-03-15T22:32:00.816143+0000","last_active":"2021-03-15T22:32:37.523909+0000","last_peered":"2021-03-15T22:32:37.523909+0000","last_clean":"2021-03-15T22:32:37.523909+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:32:37.523909+0000","last_undegraded":"2021-03-15T22:32:37.523909+0000","last_fullsized":"2021-03-15T22:32:37.523909+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:00.816090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:32:00.816090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"808'2634","reported_seq":"5332","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.299710+0000","last_change":"2021-03-15T22:32:38.299932+0000","last_active":"2021-03-15T22:32:38.299710+0000","last_peered":"2021-03-15T22:32:38.299710+0000","last_clean":"2021-03-15T22:32:38.299710+0000","last_became_active":"2021-03-15T22:29:52.225069+0000","last_became_peered":"2021-03-15T22:29:52.225069+0000","last_unstale":"2021-03-15T22:32:38.299710+0000","last_undegraded":"2021-03-15T22:32:38.299710+0000","last_fullsized":"2021-03-15T22:32:38.299710+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'2597","last_scrub_stamp":"2021-03-15T22:32:30.909661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.909661+0000","log_size":2634,"ondisk_log_size":2634,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":80085572,"num_objects":36,"num_object_clones":29,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":2,"num_read":963,"num_read_kb":1030472,"num_write":638,"num_write_kb":249109,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":165660176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"26c","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"799","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.273590+0000","last_change":"2021-03-15T22:31:28.276749+0000","last_active":"2021-03-15T22:32:36.273590+0000","last_peered":"2021-03-15T22:32:36.273590+0000","last_clean":"2021-03-15T22:32:36.273590+0000","last_became_active":"2021-03-15T22:31:28.276431+0000","last_became_peered":"2021-03-15T22:31:28.276431+0000","last_unstale":"2021-03-15T22:32:36.273590+0000","last_undegraded":"2021-03-15T22:32:36.273590+0000","last_fullsized":"2021-03-15T22:32:36.273590+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:02.160813+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.160813+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"830","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.274216+0000","last_change":"2021-03-15T22:31:12.070899+0000","last_active":"2021-03-15T22:32:36.274216+0000","last_peered":"2021-03-15T22:32:36.274216+0000","last_clean":"2021-03-15T22:32:36.274216+0000","last_became_active":"2021-03-15T22:30:32.243790+0000","last_became_peered":"2021-03-15T22:30:32.243790+0000","last_unstale":"2021-03-15T22:32:36.274216+0000","last_undegraded":"2021-03-15T22:32:36.274216+0000","last_fullsized":"2021-03-15T22:32:36.274216+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:12.070843+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:12.070843+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"351","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283520+0000","last_change":"2021-03-15T22:31:40.359761+0000","last_active":"2021-03-15T22:32:39.283520+0000","last_peered":"2021-03-15T22:32:39.283520+0000","last_clean":"2021-03-15T22:32:39.283520+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:32:39.283520+0000","last_undegraded":"2021-03-15T22:32:39.283520+0000","last_fullsized":"2021-03-15T22:32:39.283520+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:31:40.359701+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:31:40.359701+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"735","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.282536+0000","last_change":"2021-03-15T22:32:07.632290+0000","last_active":"2021-03-15T22:32:39.282536+0000","last_peered":"2021-03-15T22:32:39.282536+0000","last_clean":"2021-03-15T22:32:39.282536+0000","last_became_active":"2021-03-15T22:31:28.278813+0000","last_became_peered":"2021-03-15T22:31:28.278813+0000","last_unstale":"2021-03-15T22:32:39.282536+0000","last_undegraded":"2021-03-15T22:32:39.282536+0000","last_fullsized":"2021-03-15T22:32:39.282536+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:07.632227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:32:07.632227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.1","version":"805'421","reported_seq":"2824","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.274404+0000","last_change":"2021-03-15T22:32:35.274404+0000","last_active":"2021-03-15T22:32:35.274404+0000","last_peered":"2021-03-15T22:32:35.274404+0000","last_clean":"2021-03-15T22:32:35.274404+0000","last_became_active":"2021-03-15T22:31:33.250635+0000","last_became_peered":"2021-03-15T22:31:33.250635+0000","last_unstale":"2021-03-15T22:32:35.274404+0000","last_undegraded":"2021-03-15T22:32:35.274404+0000","last_fullsized":"2021-03-15T22:32:35.274404+0000","mapping_epoch":742,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":743,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'353","last_scrub_stamp":"2021-03-15T22:31:02.426448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.426448+0000","log_size":421,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10642824,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":140,"num_read_kb":175517,"num_write":94,"num_write_kb":34134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":77,"num_bytes_recovered":51449003,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"305","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.281994+0000","last_change":"2021-03-15T22:32:10.674869+0000","last_active":"2021-03-15T22:32:39.281994+0000","last_peered":"2021-03-15T22:32:39.281994+0000","last_clean":"2021-03-15T22:32:39.281994+0000","last_became_active":"2021-03-15T22:30:32.237925+0000","last_became_peered":"2021-03-15T22:30:32.237925+0000","last_unstale":"2021-03-15T22:32:39.281994+0000","last_undegraded":"2021-03-15T22:32:39.281994+0000","last_fullsized":"2021-03-15T22:32:39.281994+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"745","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.524083+0000","last_change":"2021-03-15T22:32:33.884013+0000","last_active":"2021-03-15T22:32:37.524083+0000","last_peered":"2021-03-15T22:32:37.524083+0000","last_clean":"2021-03-15T22:32:37.524083+0000","last_became_active":"2021-03-15T22:30:32.257328+0000","last_became_peered":"2021-03-15T22:30:32.257328+0000","last_unstale":"2021-03-15T22:32:37.524083+0000","last_undegraded":"2021-03-15T22:32:37.524083+0000","last_fullsized":"2021-03-15T22:32:37.524083+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.883949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.883949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"1020","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.273856+0000","last_change":"2021-03-15T22:32:31.747649+0000","last_active":"2021-03-15T22:32:36.273856+0000","last_peered":"2021-03-15T22:32:36.273856+0000","last_clean":"2021-03-15T22:32:36.273856+0000","last_became_active":"2021-03-15T22:29:51.222356+0000","last_became_peered":"2021-03-15T22:29:51.222356+0000","last_unstale":"2021-03-15T22:32:36.273856+0000","last_undegraded":"2021-03-15T22:32:36.273856+0000","last_fullsized":"2021-03-15T22:32:36.273856+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:32:31.747582+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.747582+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"741'10","reported_seq":"374","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283345+0000","last_change":"2021-03-15T22:31:28.389370+0000","last_active":"2021-03-15T22:32:39.283345+0000","last_peered":"2021-03-15T22:32:39.283345+0000","last_clean":"2021-03-15T22:32:39.283345+0000","last_became_active":"2021-03-15T22:31:28.280972+0000","last_became_peered":"2021-03-15T22:31:28.280972+0000","last_unstale":"2021-03-15T22:32:39.283345+0000","last_undegraded":"2021-03-15T22:32:39.283345+0000","last_fullsized":"2021-03-15T22:32:39.283345+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'9","last_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.971303+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"806'702","reported_seq":"2933","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.276763+0000","last_change":"2021-03-15T22:32:35.282180+0000","last_active":"2021-03-15T22:32:36.276763+0000","last_peered":"2021-03-15T22:32:36.276763+0000","last_clean":"2021-03-15T22:32:36.276763+0000","last_became_active":"2021-03-15T22:30:34.297623+0000","last_became_peered":"2021-03-15T22:30:34.297623+0000","last_unstale":"2021-03-15T22:32:36.276763+0000","last_undegraded":"2021-03-15T22:32:36.276763+0000","last_fullsized":"2021-03-15T22:32:36.276763+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'591","last_scrub_stamp":"2021-03-15T22:31:35.474935+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:31:35.474935+0000","log_size":702,"ondisk_log_size":702,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":22692286,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":219,"num_read_kb":187436,"num_write":192,"num_write_kb":71173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":47935099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"792","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.278868+0000","last_change":"2021-03-15T22:31:28.282929+0000","last_active":"2021-03-15T22:32:37.278868+0000","last_peered":"2021-03-15T22:32:37.278868+0000","last_clean":"2021-03-15T22:32:37.278868+0000","last_became_active":"2021-03-15T22:31:28.282600+0000","last_became_peered":"2021-03-15T22:31:28.282600+0000","last_unstale":"2021-03-15T22:32:37.278868+0000","last_undegraded":"2021-03-15T22:32:37.278868+0000","last_fullsized":"2021-03-15T22:32:37.278868+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:06.530069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:06.530069+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"271","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.283926+0000","last_change":"2021-03-15T22:30:37.798454+0000","last_active":"2021-03-15T22:32:39.283926+0000","last_peered":"2021-03-15T22:32:39.283926+0000","last_clean":"2021-03-15T22:32:39.283926+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:32:39.283926+0000","last_undegraded":"2021-03-15T22:32:39.283926+0000","last_fullsized":"2021-03-15T22:32:39.283926+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.798388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"956","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.262970+0000","last_change":"2021-03-15T22:31:29.720965+0000","last_active":"2021-03-15T22:32:35.262970+0000","last_peered":"2021-03-15T22:32:35.262970+0000","last_clean":"2021-03-15T22:32:35.262970+0000","last_became_active":"2021-03-15T22:31:28.279662+0000","last_became_peered":"2021-03-15T22:31:28.279662+0000","last_unstale":"2021-03-15T22:32:35.262970+0000","last_undegraded":"2021-03-15T22:32:35.262970+0000","last_fullsized":"2021-03-15T22:32:35.262970+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":739,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:30:42.383512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.383512+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2820,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"277","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.265441+0000","last_change":"2021-03-15T22:30:27.472423+0000","last_active":"2021-03-15T22:32:35.265441+0000","last_peered":"2021-03-15T22:32:35.265441+0000","last_clean":"2021-03-15T22:32:35.265441+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:32:35.265441+0000","last_undegraded":"2021-03-15T22:32:35.265441+0000","last_fullsized":"2021-03-15T22:32:35.265441+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:30:27.472320+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:30:27.472320+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"805'1205","reported_seq":"3642","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.289439+0000","last_change":"2021-03-15T22:32:35.289439+0000","last_active":"2021-03-15T22:32:35.289439+0000","last_peered":"2021-03-15T22:32:35.289439+0000","last_clean":"2021-03-15T22:32:35.289439+0000","last_became_active":"2021-03-15T22:31:31.245546+0000","last_became_peered":"2021-03-15T22:31:31.245546+0000","last_unstale":"2021-03-15T22:32:35.289439+0000","last_undegraded":"2021-03-15T22:32:35.289439+0000","last_fullsized":"2021-03-15T22:32:35.289439+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'889","last_scrub_stamp":"2021-03-15T22:30:29.983045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.983045+0000","log_size":1205,"ondisk_log_size":1205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38251778,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":439,"num_read_kb":453226,"num_write":281,"num_write_kb":97713,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":139702636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,4],"acting":[3,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"833","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263222+0000","last_change":"2021-03-15T22:30:32.262270+0000","last_active":"2021-03-15T22:32:35.263222+0000","last_peered":"2021-03-15T22:32:35.263222+0000","last_clean":"2021-03-15T22:32:35.263222+0000","last_became_active":"2021-03-15T22:30:32.261995+0000","last_became_peered":"2021-03-15T22:30:32.261995+0000","last_unstale":"2021-03-15T22:32:35.263222+0000","last_undegraded":"2021-03-15T22:32:35.263222+0000","last_fullsized":"2021-03-15T22:32:35.263222+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:09.101167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:09.101167+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"585","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.265491+0000","last_change":"2021-03-15T22:31:36.472708+0000","last_active":"2021-03-15T22:32:35.265491+0000","last_peered":"2021-03-15T22:32:35.265491+0000","last_clean":"2021-03-15T22:32:35.265491+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:32:35.265491+0000","last_undegraded":"2021-03-15T22:32:35.265491+0000","last_fullsized":"2021-03-15T22:32:35.265491+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:36.472646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:31:36.472646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"261","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.265603+0000","last_change":"2021-03-15T22:32:12.653877+0000","last_active":"2021-03-15T22:32:35.265603+0000","last_peered":"2021-03-15T22:32:35.265603+0000","last_clean":"2021-03-15T22:32:35.265603+0000","last_became_active":"2021-03-15T22:28:59.276755+0000","last_became_peered":"2021-03-15T22:28:59.276755+0000","last_unstale":"2021-03-15T22:32:35.265603+0000","last_undegraded":"2021-03-15T22:32:35.265603+0000","last_fullsized":"2021-03-15T22:32:35.265603+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:32:12.653809+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.653809+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"828","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.262599+0000","last_change":"2021-03-15T22:31:04.495639+0000","last_active":"2021-03-15T22:32:35.262599+0000","last_peered":"2021-03-15T22:32:35.262599+0000","last_clean":"2021-03-15T22:32:35.262599+0000","last_became_active":"2021-03-15T22:28:59.266487+0000","last_became_peered":"2021-03-15T22:28:59.266487+0000","last_unstale":"2021-03-15T22:32:35.262599+0000","last_undegraded":"2021-03-15T22:32:35.262599+0000","last_fullsized":"2021-03-15T22:32:35.262599+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.495574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.495574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"805'1097","reported_seq":"3446","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.267004+0000","last_change":"2021-03-15T22:32:35.267004+0000","last_active":"2021-03-15T22:32:35.267004+0000","last_peered":"2021-03-15T22:32:35.267004+0000","last_clean":"2021-03-15T22:32:35.267004+0000","last_became_active":"2021-03-15T22:31:31.244986+0000","last_became_peered":"2021-03-15T22:31:31.244986+0000","last_unstale":"2021-03-15T22:32:35.267004+0000","last_undegraded":"2021-03-15T22:32:35.267004+0000","last_fullsized":"2021-03-15T22:32:35.267004+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1034","last_scrub_stamp":"2021-03-15T22:32:09.615397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:09.615397+0000","log_size":1097,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":30769277,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":314,"num_read_kb":284494,"num_write":253,"num_write_kb":87766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":110,"num_bytes_recovered":50339417,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"730","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.524004+0000","last_change":"2021-03-15T22:31:04.380476+0000","last_active":"2021-03-15T22:32:37.524004+0000","last_peered":"2021-03-15T22:32:37.524004+0000","last_clean":"2021-03-15T22:32:37.524004+0000","last_became_active":"2021-03-15T22:29:52.223342+0000","last_became_peered":"2021-03-15T22:29:52.223342+0000","last_unstale":"2021-03-15T22:32:37.524004+0000","last_undegraded":"2021-03-15T22:32:37.524004+0000","last_fullsized":"2021-03-15T22:32:37.524004+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.380417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.380417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1061","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263181+0000","last_change":"2021-03-15T22:32:08.071278+0000","last_active":"2021-03-15T22:32:35.263181+0000","last_peered":"2021-03-15T22:32:35.263181+0000","last_clean":"2021-03-15T22:32:35.263181+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:32:35.263181+0000","last_undegraded":"2021-03-15T22:32:35.263181+0000","last_fullsized":"2021-03-15T22:32:35.263181+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:32:08.071210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:08.071210+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"230","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.265356+0000","last_change":"2021-03-15T22:30:09.506976+0000","last_active":"2021-03-15T22:32:35.265356+0000","last_peered":"2021-03-15T22:32:35.265356+0000","last_clean":"2021-03-15T22:32:35.265356+0000","last_became_active":"2021-03-15T22:28:59.274186+0000","last_became_peered":"2021-03-15T22:28:59.274186+0000","last_unstale":"2021-03-15T22:32:35.265356+0000","last_undegraded":"2021-03-15T22:32:35.265356+0000","last_fullsized":"2021-03-15T22:32:35.265356+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:09.506914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:30:09.506914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"665","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.277639+0000","last_change":"2021-03-15T22:32:35.067624+0000","last_active":"2021-03-15T22:32:38.277639+0000","last_peered":"2021-03-15T22:32:38.277639+0000","last_clean":"2021-03-15T22:32:38.277639+0000","last_became_active":"2021-03-15T22:30:32.262267+0000","last_became_peered":"2021-03-15T22:30:32.262267+0000","last_unstale":"2021-03-15T22:32:38.277639+0000","last_undegraded":"2021-03-15T22:32:38.277639+0000","last_fullsized":"2021-03-15T22:32:38.277639+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:32:35.067557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:35.067557+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"800'330","reported_seq":"2602","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.274326+0000","last_change":"2021-03-15T22:32:35.270453+0000","last_active":"2021-03-15T22:32:36.274326+0000","last_peered":"2021-03-15T22:32:36.274326+0000","last_clean":"2021-03-15T22:32:36.274326+0000","last_became_active":"2021-03-15T22:31:31.242099+0000","last_became_peered":"2021-03-15T22:31:31.242099+0000","last_unstale":"2021-03-15T22:32:36.274326+0000","last_undegraded":"2021-03-15T22:32:36.274326+0000","last_fullsized":"2021-03-15T22:32:36.274326+0000","mapping_epoch":740,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'322","last_scrub_stamp":"2021-03-15T22:32:11.867811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:11.867811+0000","log_size":330,"ondisk_log_size":330,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9950305,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":118,"num_read_kb":114007,"num_write":64,"num_write_kb":28231,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":37,"num_bytes_recovered":21427429,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,3],"acting":[1,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"268","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"528","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.262264+0000","last_change":"2021-03-15T22:31:28.290105+0000","last_active":"2021-03-15T22:32:35.262264+0000","last_peered":"2021-03-15T22:32:35.262264+0000","last_clean":"2021-03-15T22:32:35.262264+0000","last_became_active":"2021-03-15T22:31:28.289518+0000","last_became_peered":"2021-03-15T22:31:28.289518+0000","last_unstale":"2021-03-15T22:32:35.262264+0000","last_undegraded":"2021-03-15T22:32:35.262264+0000","last_fullsized":"2021-03-15T22:32:35.262264+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:29:40.500803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.500803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"821","reported_epoch":"807","state":"active+clean","last_fresh":"2021-03-15T22:32:37.524035+0000","last_change":"2021-03-15T22:32:36.869015+0000","last_active":"2021-03-15T22:32:37.524035+0000","last_peered":"2021-03-15T22:32:37.524035+0000","last_clean":"2021-03-15T22:32:37.524035+0000","last_became_active":"2021-03-15T22:31:31.241017+0000","last_became_peered":"2021-03-15T22:31:31.241017+0000","last_unstale":"2021-03-15T22:32:37.524035+0000","last_undegraded":"2021-03-15T22:32:37.524035+0000","last_fullsized":"2021-03-15T22:32:37.524035+0000","mapping_epoch":740,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":741,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'9","last_scrub_stamp":"2021-03-15T22:32:36.868932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.868932+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"792","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.262728+0000","last_change":"2021-03-15T22:30:34.363233+0000","last_active":"2021-03-15T22:32:35.262728+0000","last_peered":"2021-03-15T22:32:35.262728+0000","last_clean":"2021-03-15T22:32:35.262728+0000","last_became_active":"2021-03-15T22:30:31.239376+0000","last_became_peered":"2021-03-15T22:30:31.239376+0000","last_unstale":"2021-03-15T22:32:35.262728+0000","last_undegraded":"2021-03-15T22:32:35.262728+0000","last_fullsized":"2021-03-15T22:32:35.262728+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.363177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"909","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.262866+0000","last_change":"2021-03-15T22:31:28.315261+0000","last_active":"2021-03-15T22:32:35.262866+0000","last_peered":"2021-03-15T22:32:35.262866+0000","last_clean":"2021-03-15T22:32:35.262866+0000","last_became_active":"2021-03-15T22:31:28.281662+0000","last_became_peered":"2021-03-15T22:31:28.281662+0000","last_unstale":"2021-03-15T22:32:35.262866+0000","last_undegraded":"2021-03-15T22:32:35.262866+0000","last_fullsized":"2021-03-15T22:32:35.262866+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:30:04.058167+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:04.058167+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":2214,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"809'363","reported_seq":"2090","reported_epoch":"809","state":"active+clean","last_fresh":"2021-03-15T22:32:39.299589+0000","last_change":"2021-03-15T22:32:39.300149+0000","last_active":"2021-03-15T22:32:39.299589+0000","last_peered":"2021-03-15T22:32:39.299589+0000","last_clean":"2021-03-15T22:32:39.299589+0000","last_became_active":"2021-03-15T22:30:32.257058+0000","last_became_peered":"2021-03-15T22:30:32.257058+0000","last_unstale":"2021-03-15T22:32:39.299589+0000","last_undegraded":"2021-03-15T22:32:39.299589+0000","last_fullsized":"2021-03-15T22:32:39.299589+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"740'291","last_scrub_stamp":"2021-03-15T22:31:30.979407+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:31:30.979407+0000","log_size":363,"ondisk_log_size":363,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15695781,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":152,"num_read_kb":164360,"num_write":100,"num_write_kb":31214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":32799803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"22d","length":"1"},{"start":"23e","length":"1"},{"start":"247","length":"1"},{"start":"267","length":"2"},{"start":"26c","length":"1"},{"start":"299","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"86","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.958166+0000","last_change":"2021-03-15T22:32:38.958119+0000","last_active":"2021-03-15T22:32:38.958166+0000","last_peered":"2021-03-15T22:32:38.958166+0000","last_clean":"2021-03-15T22:32:38.958166+0000","last_became_active":"2021-03-15T22:31:28.287218+0000","last_became_peered":"2021-03-15T22:31:28.287218+0000","last_unstale":"2021-03-15T22:32:38.958166+0000","last_undegraded":"2021-03-15T22:32:38.958166+0000","last_fullsized":"2021-03-15T22:32:38.958166+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.958050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.958050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"835","reported_epoch":"806","state":"active+clean","last_fresh":"2021-03-15T22:32:36.274323+0000","last_change":"2021-03-15T22:32:05.880915+0000","last_active":"2021-03-15T22:32:36.274323+0000","last_peered":"2021-03-15T22:32:36.274323+0000","last_clean":"2021-03-15T22:32:36.274323+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:32:36.274323+0000","last_undegraded":"2021-03-15T22:32:36.274323+0000","last_fullsized":"2021-03-15T22:32:36.274323+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:05.880846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.880846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"80","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263305+0000","last_change":"2021-03-15T22:31:28.287806+0000","last_active":"2021-03-15T22:32:35.263305+0000","last_peered":"2021-03-15T22:32:35.263305+0000","last_clean":"2021-03-15T22:32:35.263305+0000","last_became_active":"2021-03-15T22:31:28.287437+0000","last_became_peered":"2021-03-15T22:31:28.287437+0000","last_unstale":"2021-03-15T22:32:35.263305+0000","last_undegraded":"2021-03-15T22:32:35.263305+0000","last_fullsized":"2021-03-15T22:32:35.263305+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:43.259585+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:43.259585+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"876","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263049+0000","last_change":"2021-03-15T22:30:38.419647+0000","last_active":"2021-03-15T22:32:35.263049+0000","last_peered":"2021-03-15T22:32:35.263049+0000","last_clean":"2021-03-15T22:32:35.263049+0000","last_became_active":"2021-03-15T22:30:31.239360+0000","last_became_peered":"2021-03-15T22:30:31.239360+0000","last_unstale":"2021-03-15T22:32:35.263049+0000","last_undegraded":"2021-03-15T22:32:35.263049+0000","last_fullsized":"2021-03-15T22:32:35.263049+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:30:38.419551+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:30:38.419551+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"809'1252","reported_seq":"3830","reported_epoch":"810","state":"active+clean","last_fresh":"2021-03-15T22:32:40.266066+0000","last_change":"2021-03-15T22:32:39.310162+0000","last_active":"2021-03-15T22:32:40.266066+0000","last_peered":"2021-03-15T22:32:40.266066+0000","last_clean":"2021-03-15T22:32:40.266066+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:32:40.266066+0000","last_undegraded":"2021-03-15T22:32:40.266066+0000","last_fullsized":"2021-03-15T22:32:40.266066+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'877","last_scrub_stamp":"2021-03-15T22:30:27.732142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:27.732142+0000","log_size":1252,"ondisk_log_size":1252,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52290451,"num_objects":23,"num_object_clones":19,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":455,"num_read_kb":471399,"num_write":311,"num_write_kb":111829,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"22d","length":"1"},{"start":"267","length":"1"},{"start":"26c","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"784","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263123+0000","last_change":"2021-03-15T22:30:36.417325+0000","last_active":"2021-03-15T22:32:35.263123+0000","last_peered":"2021-03-15T22:32:35.263123+0000","last_clean":"2021-03-15T22:32:35.263123+0000","last_became_active":"2021-03-15T22:30:31.243269+0000","last_became_peered":"2021-03-15T22:30:31.243269+0000","last_unstale":"2021-03-15T22:32:35.263123+0000","last_undegraded":"2021-03-15T22:32:35.263123+0000","last_fullsized":"2021-03-15T22:32:35.263123+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"732'13","reported_seq":"754","reported_epoch":"808","state":"active+clean","last_fresh":"2021-03-15T22:32:38.276913+0000","last_change":"2021-03-15T22:32:00.927902+0000","last_active":"2021-03-15T22:32:38.276913+0000","last_peered":"2021-03-15T22:32:38.276913+0000","last_clean":"2021-03-15T22:32:38.276913+0000","last_became_active":"2021-03-15T22:31:18.249734+0000","last_became_peered":"2021-03-15T22:31:18.249734+0000","last_unstale":"2021-03-15T22:32:38.276913+0000","last_undegraded":"2021-03-15T22:32:38.276913+0000","last_fullsized":"2021-03-15T22:32:38.276913+0000","mapping_epoch":682,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'13","last_scrub_stamp":"2021-03-15T22:32:00.927837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:00.927837+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"230","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263673+0000","last_change":"2021-03-15T22:30:03.567594+0000","last_active":"2021-03-15T22:32:35.263673+0000","last_peered":"2021-03-15T22:32:35.263673+0000","last_clean":"2021-03-15T22:32:35.263673+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:32:35.263673+0000","last_undegraded":"2021-03-15T22:32:35.263673+0000","last_fullsized":"2021-03-15T22:32:35.263673+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.567529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"961","reported_epoch":"805","state":"active+clean","last_fresh":"2021-03-15T22:32:35.263624+0000","last_change":"2021-03-15T22:32:32.328591+0000","last_active":"2021-03-15T22:32:35.263624+0000","last_peered":"2021-03-15T22:32:35.263624+0000","last_clean":"2021-03-15T22:32:35.263624+0000","last_became_active":"2021-03-15T22:31:28.280812+0000","last_became_peered":"2021-03-15T22:31:28.280812+0000","last_unstale":"2021-03-15T22:32:35.263624+0000","last_undegraded":"2021-03-15T22:32:35.263624+0000","last_fullsized":"2021-03-15T22:32:35.263624+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:32:32.328526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.328526+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":545500447,"num_objects":250,"num_object_clones":200,"num_object_copies":750,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":250,"num_whiteouts":16,"num_read":5660,"num_read_kb":5765564,"num_write":4157,"num_write_kb":1556610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2360,"num_bytes_recovered":1223461680,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1305047040,"data_stored":1834349625,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":16852,"ondisk_log_size":16852,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":869,"num_read_kb":864,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1483,"num_read_kb":988,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":68742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2228224,"data_stored":80200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":938,"ondisk_log_size":938,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299573,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":188196,"kb_used_data":149724,"kb_used_omap":0,"kb_used_meta":38464,"kb_avail":93556956,"statfs":{"total":95995035648,"available":95802322944,"internally_reserved":0,"allocated":153317376,"data_stored":209419676,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39387136},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.44500000000000001,"15min":0.47599999999999998},"min":{"1min":0.371,"5min":0.216,"15min":0.215},"max":{"1min":0.70099999999999996,"5min":0.93999999999999995,"15min":2.73},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.47999999999999998,"15min":0.496},"min":{"1min":0.33300000000000002,"5min":0.251,"15min":0.222},"max":{"1min":0.71599999999999997,"5min":0.75800000000000001,"15min":3.8799999999999999},"last":0.44700000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.44800000000000001,"15min":0.47999999999999998},"min":{"1min":0.30199999999999999,"5min":0.27300000000000002,"15min":0.22600000000000001},"max":{"1min":0.73899999999999999,"5min":0.91300000000000003,"15min":2.6859999999999999},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.47399999999999998,"15min":0.49299999999999999},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.23400000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":2.0779999999999998},"last":0.52000000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:32:34 2021","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.49299999999999999,"15min":0.47999999999999998},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.505,"15min":0.503},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.753,"5min":1.119,"15min":1.119},"last":0.45700000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.501,"15min":0.50800000000000001},"min":{"1min":0.379,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79200000000000004,"5min":1.4299999999999999,"15min":2.0339999999999998},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.751,"5min":5.415,"15min":5.415},"last":0.435}]},{"osd":4,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.32600000000000001,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":0.753,"5min":0.753,"15min":1.927},"last":0.46600000000000003},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.43099999999999999,"15min":0.46300000000000002},"min":{"1min":0.28499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.68200000000000005,"5min":0.72399999999999998,"15min":2.5339999999999998},"last":0.39000000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:32:34 2021","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.44900000000000001},"min":{"1min":0.40200000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.499},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.41899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76900000000000002,"5min":1.071,"15min":1.071},"last":0.41899999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.46899999999999997,"15min":0.51700000000000002},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.48499999999999999,"15min":0.51800000000000002},"min":{"1min":0.36499999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77500000000000002,"5min":1.097,"15min":1.8560000000000001},"last":0.52800000000000002}]}]},{"osd":6,"up_from":0,"seq":193273528505,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":13,"seq":55834575050,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":224672,"kb_used_data":177816,"kb_used_omap":0,"kb_used_meta":46848,"kb_avail":93520480,"statfs":{"total":95995035648,"available":95764971520,"internally_reserved":0,"allocated":182083584,"data_stored":232288336,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47972352},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:31:43 2021","interfaces":[{"interface":"back","average":{"1min":0.30599999999999999,"5min":0.30299999999999999,"15min":0.51400000000000001},"min":{"1min":0.20200000000000001,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.52300000000000002,"5min":1.325,"15min":54.427999999999997},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.35099999999999998,"15min":0.54000000000000004},"min":{"1min":0.23000000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":3.3500000000000001,"5min":3.3500000000000001,"15min":54.356999999999999},"last":0.32300000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:32:36 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.379,"15min":0.34899999999999998},"min":{"1min":0.254,"5min":0.248,"15min":0.248},"max":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"last":0.254},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.39000000000000001,"15min":0.379},"min":{"1min":0.27200000000000002,"5min":0.254,"15min":0.254},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":0.36199999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:32:13 2021","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.35699999999999998,"15min":0.38100000000000001},"min":{"1min":0.27200000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.647,"5min":1.647,"15min":1.796},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":3.2919999999999998,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.46899999999999997}]},{"osd":4,"last update":"Mon Mar 15 22:32:18 2021","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.38700000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.0330000000000004,"5min":4.0330000000000004,"15min":4.0330000000000004},"last":0.372},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.58299999999999996},"min":{"1min":0.35699999999999998,"5min":0.27200000000000002,"15min":0.253},"max":{"1min":0.78600000000000003,"5min":4.1790000000000003,"15min":4.6139999999999999},"last":0.40300000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:32:28 2021","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85599999999999998,"5min":1.226,"15min":1.226},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.55900000000000005,"15min":0.51200000000000001},"min":{"1min":0.45700000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:31:55 2021","interfaces":[{"interface":"back","average":{"1min":0.749,"5min":0.56399999999999995,"15min":0.59999999999999998},"min":{"1min":0.441,"5min":0.314,"15min":0.314},"max":{"1min":3.4209999999999998,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.495},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.56000000000000005,"15min":0.58299999999999996},"min":{"1min":0.375,"5min":0.34300000000000003,"15min":0.33700000000000002},"max":{"1min":4.1210000000000004,"5min":4.1210000000000004,"15min":4.1210000000000004},"last":0.44}]},{"osd":7,"last update":"Mon Mar 15 22:32:13 2021","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.56599999999999995,"15min":0.65700000000000003},"min":{"1min":0.46800000000000003,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":8.8100000000000005},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.57699999999999996,"15min":0.58699999999999997},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.35799999999999998},"max":{"1min":4.0860000000000003,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":0.51400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738573,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":223776,"kb_used_data":184920,"kb_used_omap":0,"kb_used_meta":38848,"kb_avail":93521376,"statfs":{"total":95995035648,"available":95765889024,"internally_reserved":0,"allocated":189358080,"data_stored":268554024,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":39780352},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:32:40 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.32900000000000001,"15min":0.34200000000000003},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.7070000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.371,"15min":0.37},"min":{"1min":0.254,"5min":0.20699999999999999,"15min":0.19900000000000001},"max":{"1min":2.4830000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.30199999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:32:34 2021","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.375,"15min":0.36499999999999999},"min":{"1min":0.24099999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.40200000000000002,"15min":0.38400000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.376}]},{"osd":3,"last update":"Mon Mar 15 22:32:17 2021","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.39700000000000002,"15min":0.38700000000000001},"min":{"1min":0.27800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.34799999999999998},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.41399999999999998,"15min":0.40699999999999997},"min":{"1min":0.254,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.589,"5min":2.589,"15min":2.589},"last":0.35299999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:32:17 2021","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.54900000000000004,"15min":0.57299999999999995},"min":{"1min":0.33000000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.645},"last":0.50700000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.54800000000000004,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.52100000000000002,"15min":0.48899999999999999},"min":{"1min":0.38900000000000001,"5min":0.318,"15min":0.318},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.55300000000000005}]},{"osd":6,"last update":"Mon Mar 15 22:31:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.505,"15min":0.55300000000000005},"min":{"1min":0.36399999999999999,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.79800000000000004,"5min":0.84999999999999998,"15min":1.6180000000000001},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.51700000000000002,"15min":0.57699999999999996},"min":{"1min":0.40500000000000003,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.76900000000000002,"5min":0.90100000000000002,"15min":1.0049999999999999},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:32:17 2021","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.443,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.6190000000000002,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.54200000000000004,"15min":0.57599999999999996},"min":{"1min":0.45200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.47199999999999998}]}]},{"osd":2,"up_from":681,"seq":2924872728603,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":203432,"kb_used_data":186848,"kb_used_omap":119,"kb_used_meta":16456,"kb_avail":93541720,"statfs":{"total":95995035648,"available":95786721280,"internally_reserved":0,"allocated":191332352,"data_stored":259202541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":16851305},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.311,"15min":0.308},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.22600000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.32500000000000001,"15min":0.32100000000000001},"min":{"1min":0.221,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.31900000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32700000000000001,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.36399999999999999,"15min":0.35799999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.29899999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.38700000000000001,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.216,"15min":0.216},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":0.255},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.35899999999999999,"15min":0.35399999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.27000000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.50700000000000001,"15min":0.48099999999999998},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.623,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.40699999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51100000000000001,"15min":0.501},"min":{"1min":0.38300000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.52400000000000002,"15min":0.50600000000000001},"min":{"1min":0.40100000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.53200000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.52800000000000002,"15min":0.503},"min":{"1min":0.48599999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.42399999999999999,"5min":0.373,"15min":0.373},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.44600000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.56200000000000006,"15min":0.53200000000000003},"min":{"1min":0.44,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.60099999999999998},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.53900000000000003,"15min":0.52000000000000002},"min":{"1min":0.42799999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.42799999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149890,"num_pgs":72,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":347444,"kb_used_data":286124,"kb_used_omap":0,"kb_used_meta":61312,"kb_avail":93397708,"statfs":{"total":95995035648,"available":95639252992,"internally_reserved":0,"allocated":292990976,"data_stored":394148817,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":62783488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32000000000000001,"15min":0.33300000000000002},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.17699999999999999},"max":{"1min":0.46700000000000003,"5min":1.1499999999999999,"15min":2.625},"last":0.379},{"interface":"front","average":{"1min":0.309,"5min":0.31,"15min":0.33900000000000002},"min":{"1min":0.20000000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.48499999999999999,"5min":1.419,"15min":2.6339999999999999},"last":0.26000000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.39500000000000002},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":0.49199999999999999,"5min":0.72599999999999998,"15min":6.0199999999999996},"last":0.623},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.36299999999999999,"15min":0.39900000000000002},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.498,"5min":2.5750000000000002,"15min":7.6130000000000004},"last":0.61299999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:32:35 2021","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35699999999999998,"15min":0.35599999999999998},"min":{"1min":0.27100000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.29299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.36699999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:32:26 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.86899999999999999,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.497,"15min":0.52800000000000002},"min":{"1min":0.34599999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.82699999999999996,"5min":2.1059999999999999,"15min":2.2890000000000001},"last":0.56799999999999995}]},{"osd":5,"last update":"Mon Mar 15 22:32:26 2021","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.73399999999999999,"15min":0.82199999999999995},"min":{"1min":0.311,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75800000000000001,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.91500000000000004,"15min":1.083},"min":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.79400000000000004,"5min":15.776,"15min":15.776},"last":0.53000000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:31:52 2021","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.52800000000000002,"15min":0.56999999999999995},"min":{"1min":0.38400000000000001,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":0.80000000000000004,"5min":2.5299999999999998,"15min":2.6499999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.55800000000000005,"15min":0.56699999999999995},"min":{"1min":0.40799999999999997,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.76100000000000001,"5min":3.472,"15min":3.472},"last":0.50800000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.57999999999999996},"min":{"1min":0.38600000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81100000000000005,"5min":5.625,"15min":5.625},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.56000000000000005},"min":{"1min":0.371,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78800000000000003,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":0.498}]}]},{"osd":4,"up_from":31,"seq":133143986367,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":137296,"kb_used_data":92040,"kb_used_omap":0,"kb_used_meta":45248,"kb_avail":93607856,"statfs":{"total":95995035648,"available":95854444544,"internally_reserved":0,"allocated":94248960,"data_stored":129839341,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46333952},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.48499999999999999},"min":{"1min":0.24199999999999999,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":0.73199999999999998,"5min":4.7050000000000001,"15min":4.7050000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54300000000000004,"15min":0.503},"min":{"1min":0.35799999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.65500000000000003,"5min":5.0209999999999999,"15min":5.0209999999999999},"last":0.35499999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.505,"15min":0.48499999999999999},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.23599999999999999},"max":{"1min":0.67900000000000005,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.28499999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.86599999999999999,"5min":4.718,"15min":4.718},"last":0.40899999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:32:32 2021","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69099999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.36199999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73699999999999999,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.443}]},{"osd":3,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.58899999999999997,"15min":0.55000000000000004},"min":{"1min":0.43099999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.88300000000000001,"5min":5.0499999999999998,"15min":5.0499999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89500000000000002,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.47799999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.42299999999999999,"15min":0.40899999999999997},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.307},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.44900000000000001,"15min":0.42699999999999999},"min":{"1min":0.30399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66000000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.36899999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:31:56 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.46100000000000002,"15min":0.46000000000000002},"min":{"1min":0.376,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.67200000000000004,"5min":4.3079999999999998,"15min":4.3079999999999998},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.53200000000000003,"15min":0.53400000000000003},"min":{"1min":0.371,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.72099999999999997,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.46200000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:32:14 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.32100000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79900000000000004,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.433},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56399999999999995,"15min":0.54700000000000004},"min":{"1min":0.35799999999999998,"5min":0.25,"15min":0.25},"max":{"1min":1.034,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":0.42199999999999999}]}]},{"osd":5,"up_from":616,"seq":2645699854376,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":159108,"kb_used_data":141564,"kb_used_omap":153,"kb_used_meta":17382,"kb_avail":93586044,"statfs":{"total":95995035648,"available":95832109056,"internally_reserved":0,"allocated":144961536,"data_stored":193434257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":17799697},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46400000000000002,"15min":0.45600000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65500000000000003,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.67700000000000005,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.442}]},{"osd":1,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46400000000000002,"15min":0.45200000000000001},"min":{"1min":0.312,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":0.876,"15min":0.876},"last":0.46100000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.622,"5min":0.871,"15min":0.871},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.40300000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.49199999999999999,"15min":0.47599999999999998},"min":{"1min":0.39200000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.45800000000000002,"15min":0.441},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.39200000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.378},"min":{"1min":0.32300000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.40699999999999997,"15min":0.38600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.81499999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.45500000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45800000000000002,"15min":0.44800000000000001},"min":{"1min":0.32100000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.83999999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.318},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48599999999999999,"15min":0.46600000000000003},"min":{"1min":0.39000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.378}]},{"osd":7,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45500000000000002,"15min":0.439},"min":{"1min":0.29799999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.88700000000000001,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.434}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":303104,"data_stored":10926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":12926,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":356352,"data_stored":12796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":323584,"data_stored":11740,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":212992,"data_stored":7576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":179572736,"data_stored":262283183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":172294144,"data_stored":226042173,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":181690368,"data_stored":250432045,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":284811264,"data_stored":389484724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":84500480,"data_stored":123602650,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":135307264,"data_stored":187184541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":123215872,"data_stored":192150239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":143654912,"data_stored":203170070,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:32:42.806 INFO:tasks.cephadm.ceph_manager.ceph:recovered! 2021-03-15T22:32:42.807 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:32:42.807 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump -f json-pretty 2021-03-15T22:32:42.830 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:42.875 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:42.899 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:42.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:42.900 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:43.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:43.025 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:43.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:43.027 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:43.128 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:43.152 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:43.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:43.154 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:43.255 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:43.277 INFO:tasks.rados.rados.0.smithi076.stdout:3040: done (3 left) 2021-03-15T22:32:43.278 INFO:tasks.rados.rados.0.smithi076.stdout:3041: snap_create 2021-03-15T22:32:43.278 INFO:tasks.rados.rados.0.smithi076.stdout:3035: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:32:43.278 INFO:tasks.rados.rados.0.smithi076.stdout:3035: finishing rollback tid 2 to smithi07694878-28 2021-03-15T22:32:43.278 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2646 (ObjNum 620 snap 193 seq_num 620) dirty exists 2021-03-15T22:32:43.279 INFO:tasks.rados.rados.0.smithi076.stdout:3037: finishing write tid 1 to smithi07694878-24 2021-03-15T22:32:43.279 INFO:tasks.rados.rados.0.smithi076.stdout:3037: finishing write tid 2 to smithi07694878-24 2021-03-15T22:32:43.279 INFO:tasks.rados.rados.0.smithi076.stdout:3037: finishing write tid 3 to smithi07694878-24 2021-03-15T22:32:43.280 INFO:tasks.rados.rados.0.smithi076.stdout:3037: finishing write tid 4 to smithi07694878-24 2021-03-15T22:32:43.280 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1736 (ObjNum 1046 snap 351 seq_num 1046) dirty exists 2021-03-15T22:32:43.280 INFO:tasks.rados.rados.0.smithi076.stdout:3037: left oid 24 (ObjNum 1046 snap 351 seq_num 1046) 2021-03-15T22:32:43.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:43.282 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:43.282 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:43.384 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:43.405 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:43.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:43.407 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:43.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:43.547 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:43.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:43.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:43.650 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:43.665 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:43 smithi125 conmon[56739]: cluster 2021-03-15T22:32:41.878944+0000 osd.5 (osd.5) 36 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:32:43.666 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:43 smithi125 conmon[56739]: cluster 2021-03-15T22:32:42.621590+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.b scrub ok 2021-03-15T22:32:43.666 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:43 smithi125 conmon[56739]: cluster 2021-03-15T22:32:43.275966+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2021-03-15T22:32:43.678 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:43.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:43.680 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:43.735 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:43 smithi076 conmon[56514]: cluster 2021-03-15T22:32:41.878944+0000 osd.5 (osd. 2021-03-15T22:32:43.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:43 smithi076 conmon[56514]: 5) 36 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:32:43.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:43 smithi076 conmon[56514]: cluster 2021-03-15T22:32:42.621590+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.b scrub ok 2021-03-15T22:32:43.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:43 smithi076 conmon[56514]: cluster 2021-03-15T22:32:43.275966+0000 mon.a (mon.0) 2103 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2021-03-15T22:32:43.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:43 smithi076 conmon[62156]: cluster 2021-03-15T22:32:41.878944+0000 osd.5 (osd.5) 36 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:32:43.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:43 smithi076 conmon[62156]: cluster 2021-03-15T22:32:43.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:43 smithi076 conmon[62156]: 2021-03-15T22:32:42.621590+0000 osd.3 (osd.3) 148 : cluster [DBG] 6.b scrub ok 2021-03-15T22:32:43.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:43 smithi076 conmon[62156]: cluster 2021-03-15T22:32:43.275966+0000 mon.a 2021-03-15T22:32:43.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:43 smithi076 conmon[62156]: (mon.0) 2103 : cluster [DBG] osdmap e813: 8 total, 8 up, 7 in 2021-03-15T22:32:43.781 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:43.807 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:43.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:43.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:43.863 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:32:43.863 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:32:43.910 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:43.937 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:43.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:43.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:44.002 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:44.039 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:44.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:44.065 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:44.066 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:44.168 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:44.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:44.194 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:44.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:44.280 INFO:tasks.rados.rados.0.smithi076.stdout:3035: done (3 left) 2021-03-15T22:32:44.281 INFO:tasks.rados.rados.0.smithi076.stdout:3037: done (2 left) 2021-03-15T22:32:44.281 INFO:tasks.rados.rados.0.smithi076.stdout:3039: done (1 left) 2021-03-15T22:32:44.281 INFO:tasks.rados.rados.0.smithi076.stdout:3041: done (0 left) 2021-03-15T22:32:44.281 INFO:tasks.rados.rados.0.smithi076.stdout:3042: write oid 36 current snap is 352 2021-03-15T22:32:44.281 INFO:tasks.rados.rados.0.smithi076.stdout:3042: seq_num 1047 ranges {796929=726064,2184647=659366,3302344=294585} 2021-03-15T22:32:44.288 INFO:tasks.rados.rados.0.smithi076.stdout:3042: writing smithi07694878-36 from 796929 to 1522993 tid 1 2021-03-15T22:32:44.295 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:44.296 INFO:tasks.rados.rados.0.smithi076.stdout:3042: writing smithi07694878-36 from 2184647 to 2844013 tid 2 2021-03-15T22:32:44.300 INFO:tasks.rados.rados.0.smithi076.stdout:3042: writing smithi07694878-36 from 3302344 to 3596929 tid 3 2021-03-15T22:32:44.301 INFO:tasks.rados.rados.0.smithi076.stdout:3043: rollback oid 14 current snap is 352 2021-03-15T22:32:44.301 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 14 to 340 2021-03-15T22:32:44.302 INFO:tasks.rados.rados.0.smithi076.stdout:3044: copy_from oid 42 from oid 39 current snap is 352 2021-03-15T22:32:44.302 INFO:tasks.rados.rados.0.smithi076.stdout:3045: copy_from oid 17 from oid 4 current snap is 352 2021-03-15T22:32:44.302 INFO:tasks.rados.rados.0.smithi076.stdout:3046: snap_create 2021-03-15T22:32:44.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:44.320 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:44.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:44.416 INFO:tasks.rados.rados.0.smithi076.stdout:3042: finishing write tid 1 to smithi07694878-36 2021-03-15T22:32:44.417 INFO:tasks.rados.rados.0.smithi076.stdout:3043: finishing rollback tid 0 to smithi07694878-14 2021-03-15T22:32:44.417 INFO:tasks.rados.rados.0.smithi076.stdout:3042: finishing write tid 2 to smithi07694878-36 2021-03-15T22:32:44.417 INFO:tasks.rados.rados.0.smithi076.stdout:3043: finishing rollback tid 1 to smithi07694878-14 2021-03-15T22:32:44.417 INFO:tasks.rados.rados.0.smithi076.stdout:3042: finishing write tid 3 to smithi07694878-36 2021-03-15T22:32:44.418 INFO:tasks.rados.rados.0.smithi076.stdout:3042: finishing write tid 4 to smithi07694878-36 2021-03-15T22:32:44.418 INFO:tasks.rados.rados.0.smithi076.stdout:3043: finishing rollback tid 2 to smithi07694878-14 2021-03-15T22:32:44.419 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 1786 (ObjNum 936 snap 316 seq_num 936) dirty exists 2021-03-15T22:32:44.420 INFO:tasks.rados.rados.0.smithi076.stdout:3042: finishing write tid 5 to smithi07694878-36 2021-03-15T22:32:44.420 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 1984 (ObjNum 1047 snap 352 seq_num 1047) dirty exists 2021-03-15T22:32:44.420 INFO:tasks.rados.rados.0.smithi076.stdout:3042: left oid 36 (ObjNum 1047 snap 352 seq_num 1047) 2021-03-15T22:32:44.421 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:44.423 INFO:tasks.rados.rados.0.smithi076.stdout:3045: finishing copy_from to smithi07694878-17 2021-03-15T22:32:44.424 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 1975 (ObjNum 1026 snap 346 seq_num 1026) dirty exists 2021-03-15T22:32:44.424 INFO:tasks.rados.rados.0.smithi076.stdout:3045: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:32:44.434 INFO:tasks.rados.rados.0.smithi076.stdout:3044: finishing copy_from to smithi07694878-42 2021-03-15T22:32:44.435 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2092 (ObjNum 829 snap 275 seq_num 829) dirty exists 2021-03-15T22:32:44.435 INFO:tasks.rados.rados.0.smithi076.stdout:3044: finishing copy_from racing read to smithi07694878-42 2021-03-15T22:32:44.446 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:44.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:44.510 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-15T22:32:44.549 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:44.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:44.574 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:44.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:44.677 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:44.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:44.701 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:44.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:44.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[56514]: cluster 2021-03-15T22:32:43.179684+0000 mgr.y (mgr.14144) 947 : cluster [DBG] pgmap v1363: 148 pgs: 148 active+clean; 523 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-03-15T22:32:44.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[56514]: cluster 2021-03-15T22:32:43.307654+0000 osd.4 (osd.4) 174 : cluster [DBG] 4.d scrub ok 2021-03-15T22:32:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[56514]: cluster 2021-03-15T22:32:43.531343+0000 osd.7 (osd.7) 154 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:32:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[56514]: cluster 2021-03-15T22:32:43.672762+0000 osd.3 (osd.3) 149 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[56514]: cluster 2021-03-15T22:32:43.675085+0000 osd.3 (osd.3) 150 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[56514]: cluster 2021-03-15T22:32:44.276725+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2021-03-15T22:32:44.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[62156]: cluster 2021-03-15T22:32:43.179684+0000 mgr.y (mgr.14144) 947 : cluster [DBG] pgmap v1363: 148 pgs: 148 active+clean; 523 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-03-15T22:32:44.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[62156]: cluster 2021-03-15T22:32:43.307654+0000 osd.4 (osd.4) 174 : cluster [DBG] 4.d scrub ok 2021-03-15T22:32:44.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[62156]: cluster 2021-03-15T22:32:43.531343+0000 osd.7 (osd.7) 154 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:32:44.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[62156]: cluster 2021-03-15T22:32:43.672762+0000 osd.3 (osd.3) 149 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:44.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[62156]: cluster 2021-03-15T22:32:43.675085+0000 osd.3 (osd.3) 150 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:44.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:44 smithi076 conmon[62156]: cluster 2021-03-15T22:32:44.276725+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2021-03-15T22:32:44.784 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:44.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:44 smithi125 conmon[56739]: cluster 2021-03-15T22:32:43.179684+0000 mgr.y ( 2021-03-15T22:32:44.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:44 smithi125 conmon[56739]: mgr.14144) 947 : cluster [DBG] pgmap v1363: 148 pgs: 148 active+clean; 523 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.7 MiB/s wr, 14 op/s 2021-03-15T22:32:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:44 smithi125 conmon[56739]: cluster 2021-03-15T22:32:43.307654+0000 osd.4 (osd.4) 174 : cluster [DBG] 4.d scrub ok 2021-03-15T22:32:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:44 smithi125 conmon[56739]: cluster 2021-03-15T22:32:43.531343+0000 osd.7 (osd.7) 154 : cluster [DBG] 1.0 scrub ok 2021-03-15T22:32:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:44 smithi125 conmon[56739]: cluster 2021-03-15T22:32:43.672762+0000 osd.3 (osd.3) 149 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:44 smithi125 conmon[56739]: cluster 2021-03-15T22:32:43.675085+0000 osd.3 (osd.3) 150 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:44 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:32:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:44 smithi125 conmon[56739]: T22:32:44.276725+0000 mon.a (mon.0) 2104 : cluster [DBG] osdmap e814: 8 total, 8 up, 7 in 2021-03-15T22:32:44.803 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:44.826 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:44.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:44.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:44.928 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:44.953 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:44.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:44.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:45.056 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:45.078 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:45.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:45.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:45.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:45.204 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:45.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:45.206 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:45.270 INFO:tasks.rados.rados.0.smithi076.stdout:3042: done (4 left) 2021-03-15T22:32:45.271 INFO:tasks.rados.rados.0.smithi076.stdout:3043: done (3 left) 2021-03-15T22:32:45.271 INFO:tasks.rados.rados.0.smithi076.stdout:3044: done (2 left) 2021-03-15T22:32:45.271 INFO:tasks.rados.rados.0.smithi076.stdout:3045: done (1 left) 2021-03-15T22:32:45.271 INFO:tasks.rados.rados.0.smithi076.stdout:3046: done (0 left) 2021-03-15T22:32:45.272 INFO:tasks.rados.rados.0.smithi076.stdout:3047: read oid 34 snap -1 2021-03-15T22:32:45.272 INFO:tasks.rados.rados.0.smithi076.stdout:3047: expect deleted 2021-03-15T22:32:45.272 INFO:tasks.rados.rados.0.smithi076.stdout:3048: write oid 40 current snap is 353 2021-03-15T22:32:45.272 INFO:tasks.rados.rados.0.smithi076.stdout:3048: seq_num 1048 ranges {413736=401313,1326776=403024,2413414=457893,3438716=175020} 2021-03-15T22:32:45.275 INFO:tasks.rados.rados.0.smithi076.stdout:3048: writing smithi07694878-40 from 413736 to 815049 tid 1 2021-03-15T22:32:45.276 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:32:45.280 INFO:tasks.rados.rados.0.smithi076.stdout:3048: writing smithi07694878-40 from 1326776 to 1729800 tid 2 2021-03-15T22:32:45.289 INFO:tasks.rados.rados.0.smithi076.stdout:3048: writing smithi07694878-40 from 2413414 to 2871307 tid 3 2021-03-15T22:32:45.293 INFO:tasks.rados.rados.0.smithi076.stdout:3048: writing smithi07694878-40 from 3438716 to 3613736 tid 4 2021-03-15T22:32:45.293 INFO:tasks.rados.rados.0.smithi076.stdout:3049: write oid 44 current snap is 353 2021-03-15T22:32:45.294 INFO:tasks.rados.rados.0.smithi076.stdout:3049: seq_num 1049 ranges {430543=476562,1385258=663035,2840837=669165} 2021-03-15T22:32:45.298 INFO:tasks.rados.rados.0.smithi076.stdout:3049: writing smithi07694878-44 from 430543 to 907105 tid 1 2021-03-15T22:32:45.305 INFO:tasks.rados.rados.0.smithi076.stdout:3049: writing smithi07694878-44 from 1385258 to 2048293 tid 2 2021-03-15T22:32:45.307 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:45.313 INFO:tasks.rados.rados.0.smithi076.stdout:3049: writing smithi07694878-44 from 2840837 to 3510002 tid 3 2021-03-15T22:32:45.314 INFO:tasks.rados.rados.0.smithi076.stdout:3050: read oid 7 snap -1 2021-03-15T22:32:45.315 INFO:tasks.rados.rados.0.smithi076.stdout:3050: expect (ObjNum 962 snap 321 seq_num 962) 2021-03-15T22:32:45.315 INFO:tasks.rados.rados.0.smithi076.stdout:3051: snap_remove snap 264 2021-03-15T22:32:45.334 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:45.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:45.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:45.437 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:45.461 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:45.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:45.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:45.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:45 smithi125 conmon[56739]: cluster 2021-03-15T22:32:43.703014+0000 osd.1 (osd.1) 159 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:32:45.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:45 smithi125 conmon[56739]: audit 2021-03-15T22:32:45.217069+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? 172.21.15.76:0/3118499393' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:32:45.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:45 smithi125 conmon[56739]: cluster 2021-03-15T22:32:45.260871+0000 mon.a (mon.0) 2106 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:45.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:45 smithi125 conmon[56739]: audit 2021-03-15T22:32:45.266894+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? 172.21.15.76:0/3118499393' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:32:45.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:45 smithi125 conmon[56739]: cluster 2021-03-15T22:32:45.266967+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e815: 8 total, 8 up, 7 in 2021-03-15T22:32:45.565 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:45.574 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:32:45.575 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:32:45.575 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:32:45.575 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:32:45.576 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:32:45.589 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:45.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:45.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:45.593 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:32:45.611 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:32:45.629 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:32:45.630 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:32:45.630 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:32:45.630 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:32:45.631 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:32:45.631 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:32:45.631 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:32:45.631 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:32:45.631 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:32:45.692 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:45.715 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:45.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:45.716 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:45.728 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:45 smithi076 conmon[56514]: cluster 2021-03-15T22:32:43.703014+0000 osd.1 (osd.1) 159 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:32:45.729 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:45 smithi076 conmon[56514]: audit 2021-03-15T22:32:45.217069+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? 172.21.15.76:0/3118499393' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:32:45.729 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:45 smithi076 conmon[56514]: cluster 2021-03-15T22:32:45.260871+0000 mon.a (mon.0) 2106 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:45.729 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:45 smithi076 conmon[56514]: audit 2021-03-15T22:32:45.266894+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? 172.21.15.76:0/3118499393' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:32:45.729 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:45 smithi076 conmon[56514]: cluster 2021-03-15T22:32:45.266967+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e815: 8 total, 8 up, 7 in 2021-03-15T22:32:45.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:45 smithi076 conmon[62156]: cluster 2021-03-15T22:32:43.703014+0000 osd.1 (osd.1) 159 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:32:45.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:45 smithi076 conmon[62156]: audit 2021-03-15T22:32:45.217069+0000 mon.a (mon.0) 2105 : audit [INF] from='client.? 172.21.15.76:0/3118499393' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:32:45.730 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:45 smithi076 conmon[62156]: cluster 2021-03-15T22:32:45.260871+0000 mon.a (mon.0) 2106 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:45.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:45 smithi076 conmon[62156]: audit 2021-03-15T22:32:45.266894+0000 mon.a (mon.0) 2107 : audit [INF] from='client.? 172.21.15.76:0/3118499393' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:32:45.731 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:45 smithi076 conmon[62156]: cluster 2021-03-15T22:32:45.266967+0000 mon.a (mon.0) 2108 : cluster [DBG] osdmap e815: 8 total, 8 up, 7 in 2021-03-15T22:32:45.817 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:45.839 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:45.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:45.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:45.942 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:45.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:45.966 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:45.966 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:46.068 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:46.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:46.093 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:46.093 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:46.195 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:46.219 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:46.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:46.220 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:46.276 INFO:tasks.rados.rados.0.smithi076.stdout:3051: done (4 left) 2021-03-15T22:32:46.276 INFO:tasks.rados.rados.0.smithi076.stdout:3052: write oid 35 current snap is 353 2021-03-15T22:32:46.276 INFO:tasks.rados.rados.0.smithi076.stdout:3052: seq_num 1050 ranges {447350=668164,1560093=406693,2584613=480437} 2021-03-15T22:32:46.281 INFO:tasks.rados.rados.0.smithi076.stdout:3052: writing smithi07694878-35 from 447350 to 1115514 tid 1 2021-03-15T22:32:46.286 INFO:tasks.rados.rados.0.smithi076.stdout:3052: writing smithi07694878-35 from 1560093 to 1966786 tid 2 2021-03-15T22:32:46.292 INFO:tasks.rados.rados.0.smithi076.stdout:3052: writing smithi07694878-35 from 2584613 to 3065050 tid 3 2021-03-15T22:32:46.292 INFO:tasks.rados.rados.0.smithi076.stdout:3053: read oid 39 snap 320 2021-03-15T22:32:46.293 INFO:tasks.rados.rados.0.smithi076.stdout:3053: expect deleted 2021-03-15T22:32:46.293 INFO:tasks.rados.rados.0.smithi076.stdout:3054: read oid 43 snap -1 2021-03-15T22:32:46.293 INFO:tasks.rados.rados.0.smithi076.stdout:3054: expect (ObjNum 1014 snap 341 seq_num 1014) 2021-03-15T22:32:46.293 INFO:tasks.rados.rados.0.smithi076.stdout:3055: read oid 20 snap 350 2021-03-15T22:32:46.294 INFO:tasks.rados.rados.0.smithi076.stdout:3055: expect deleted 2021-03-15T22:32:46.294 INFO:tasks.rados.rados.0.smithi076.stdout:3056: read oid 46 snap -1 2021-03-15T22:32:46.294 INFO:tasks.rados.rados.0.smithi076.stdout:3056: expect (ObjNum 1044 snap 350 seq_num 1044) 2021-03-15T22:32:46.294 INFO:tasks.rados.rados.0.smithi076.stdout:3057: snap_remove snap 311 2021-03-15T22:32:46.321 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:46.323 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:32:46.323 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:32:46.324 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:32:46.324 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:32:46.324 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:32:46.343 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:46.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:46.344 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:46.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:46.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:46.471 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:46.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:46.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:46.594 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:46.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:46.596 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:46.697 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:46.705 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[56514]: cluster 2021-03-15T22:32:44.675885+0000 osd.1 (osd.1) 160 : cluster [DBG] 3.4 deep-scrub ok 2021-03-15T22:32:46.706 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[56514]: cluster 2021-03-15T22:32:45.180081+0000 mgr.y (mgr.14144) 948 : cluster [DBG] pgmap v1366: 148 pgs: 148 active+clean; 527 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2021-03-15T22:32:46.706 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[56514]: audit 2021-03-15T22:32:45.486349+0000 mon.a (mon.0) 2109 : audit [DBG] from='client.? 172.21.15.76:0/1212800760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:46.706 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[56514]: cluster 2021-03-15T22:32:46.273901+0000 mon.a (mon.0) 2110 : cluster [DBG] osdmap e816: 8 total, 8 up, 7 in 2021-03-15T22:32:46.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[62156]: cluster 2021-03-15T22:32:44.675885+0000 osd.1 ( 2021-03-15T22:32:46.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[62156]: osd.1) 160 : cluster [DBG] 3.4 deep-scrub ok 2021-03-15T22:32:46.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[62156]: cluster 2021-03-15T22:32:45.180081+0000 mgr.y 2021-03-15T22:32:46.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[62156]: (mgr.14144) 948 : cluster [DBG] pgmap v1366: 148 pgs: 148 active+clean; 527 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2021-03-15T22:32:46.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[62156]: audit 2021-03-15T22:32 2021-03-15T22:32:46.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[62156]: :45.486349+0000 mon.a (mon.0) 2109 2021-03-15T22:32:46.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[62156]: : audit [DBG] from='client.? 172.21.15.76:0/1212800760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:46.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:32:46.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[62156]: -15T22:32:46.273901+0000 2021-03-15T22:32:46.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[62156]: mon.a (mon.0) 2110 : cluster [DBG] 2021-03-15T22:32:46.709 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:46 smithi076 conmon[62156]: osdmap e816: 8 total, 8 up, 7 in 2021-03-15T22:32:46.719 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:46.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:46.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:46.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:46 smithi125 conmon[56739]: cluster 2021-03-15T22:32:44.675885+0000 osd.1 (osd.1) 160 : cluster [DBG] 3.4 deep-scrub ok 2021-03-15T22:32:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:46 smithi125 conmon[56739]: cluster 2021-03-15T22:32:45.180081+0000 mgr.y (mgr.14144) 948 : cluster [DBG] pgmap v1366: 148 pgs: 148 active+clean; 527 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 9.5 MiB/s rd, 1.1 MiB/s wr, 12 op/s 2021-03-15T22:32:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:46 smithi125 conmon[56739]: audit 2021-03-15T22:32:45.486349+0000 mon.a (mon.0) 2109 : audit [DBG] from='client.? 172.21.15.76:0/1212800760' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:46 smithi125 conmon[56739]: cluster 2021-03-15T22:32:46.273901+0000 mon.a (mon.0) 2110 : cluster [DBG] osdmap e816: 8 total, 8 up, 7 in 2021-03-15T22:32:46.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:46.844 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:46.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:46.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:46.884 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-15T22:32:46.947 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:46.969 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:46.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:47.072 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:47.094 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:47.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:47.096 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:47.156 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:32:47.198 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:47.220 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:47.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:47.278 INFO:tasks.rados.rados.0.smithi076.stdout:3057: done (9 left) 2021-03-15T22:32:47.279 INFO:tasks.rados.rados.0.smithi076.stdout:3058: read oid 13 snap -1 2021-03-15T22:32:47.279 INFO:tasks.rados.rados.0.smithi076.stdout:3058: expect deleted 2021-03-15T22:32:47.280 INFO:tasks.rados.rados.0.smithi076.stdout:3059: copy_from oid 1 from oid 21 current snap is 353 2021-03-15T22:32:47.280 INFO:tasks.rados.rados.0.smithi076.stdout:3060: copy_from oid 17 from oid 32 current snap is 353 2021-03-15T22:32:47.280 INFO:tasks.rados.rados.0.smithi076.stdout:3061: write oid 16 current snap is 353 2021-03-15T22:32:47.281 INFO:tasks.rados.rados.0.smithi076.stdout:3061: seq_num 1051 ranges {464157=743413,1618575=666704,3012036=408062} 2021-03-15T22:32:47.289 INFO:tasks.rados.rados.0.smithi076.stdout:3061: writing smithi07694878-16 from 464157 to 1207570 tid 1 2021-03-15T22:32:47.296 INFO:tasks.rados.rados.0.smithi076.stdout:3061: writing smithi07694878-16 from 1618575 to 2285279 tid 2 2021-03-15T22:32:47.303 INFO:tasks.rados.rados.0.smithi076.stdout:3061: writing smithi07694878-16 from 3012036 to 3420098 tid 3 2021-03-15T22:32:47.304 INFO:tasks.rados.rados.0.smithi076.stdout:3062: snap_remove snap 294 2021-03-15T22:32:47.304 INFO:tasks.rados.rados.0.smithi076.stdout:3048: finishing write tid 1 to smithi07694878-40 2021-03-15T22:32:47.304 INFO:tasks.rados.rados.0.smithi076.stdout:3048: finishing write tid 2 to smithi07694878-40 2021-03-15T22:32:47.305 INFO:tasks.rados.rados.0.smithi076.stdout:3048: finishing write tid 3 to smithi07694878-40 2021-03-15T22:32:47.305 INFO:tasks.rados.rados.0.smithi076.stdout:3048: finishing write tid 4 to smithi07694878-40 2021-03-15T22:32:47.305 INFO:tasks.rados.rados.0.smithi076.stdout:3048: finishing write tid 5 to smithi07694878-40 2021-03-15T22:32:47.305 INFO:tasks.rados.rados.0.smithi076.stdout:3048: finishing write tid 6 to smithi07694878-40 2021-03-15T22:32:47.306 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 2678 (ObjNum 1048 snap 353 seq_num 1048) dirty exists 2021-03-15T22:32:47.306 INFO:tasks.rados.rados.0.smithi076.stdout:3048: left oid 40 (ObjNum 1048 snap 353 seq_num 1048) 2021-03-15T22:32:47.306 INFO:tasks.rados.rados.0.smithi076.stdout:3049: finishing write tid 1 to smithi07694878-44 2021-03-15T22:32:47.306 INFO:tasks.rados.rados.0.smithi076.stdout:3049: finishing write tid 2 to smithi07694878-44 2021-03-15T22:32:47.307 INFO:tasks.rados.rados.0.smithi076.stdout:3050: expect (ObjNum 962 snap 321 seq_num 962) 2021-03-15T22:32:47.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:47.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:47.348 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:47.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:47.450 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:47.452 INFO:tasks.rados.rados.0.smithi076.stdout:3049: finishing write tid 3 to smithi07694878-44 2021-03-15T22:32:47.452 INFO:tasks.rados.rados.0.smithi076.stdout:3049: finishing write tid 4 to smithi07694878-44 2021-03-15T22:32:47.453 INFO:tasks.rados.rados.0.smithi076.stdout:3049: finishing write tid 5 to smithi07694878-44 2021-03-15T22:32:47.453 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 1783 (ObjNum 1049 snap 353 seq_num 1049) dirty exists 2021-03-15T22:32:47.454 INFO:tasks.rados.rados.0.smithi076.stdout:3049: left oid 44 (ObjNum 1049 snap 353 seq_num 1049) 2021-03-15T22:32:47.477 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:47.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:47.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:47.580 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:47.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:47.604 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:47.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:47.705 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:47.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:47.731 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:47.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:47.832 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:47.855 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:47.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:47.857 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:47.859 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:32:47.859 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:32:47.860 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 817, 2021-03-15T22:32:47.860 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:32:47.860 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2021-03-15T22:13:20.370795+0000", 2021-03-15T22:32:47.861 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2021-03-15T22:32:47.270408+0000", 2021-03-15T22:32:47.861 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2021-03-15T22:30:30.205134+0000", 2021-03-15T22:32:47.861 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2021-03-15T22:31:27.245203+0000", 2021-03-15T22:32:47.862 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-03-15T22:32:47.862 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5801984, 2021-03-15T22:32:47.862 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2021-03-15T22:32:47.863 INFO:teuthology.orchestra.run.smithi076.stdout: "noscrub", 2021-03-15T22:32:47.863 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2021-03-15T22:32:47.863 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2021-03-15T22:32:47.863 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2021-03-15T22:32:47.864 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2021-03-15T22:32:47.864 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:32:47.864 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 20, 2021-03-15T22:32:47.864 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2021-03-15T22:32:47.865 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-03-15T22:32:47.865 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2021-03-15T22:32:47.865 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2021-03-15T22:32:47.865 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 7, 2021-03-15T22:32:47.866 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2021-03-15T22:32:47.866 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2021-03-15T22:32:47.866 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2021-03-15T22:32:47.866 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2021-03-15T22:32:47.867 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2021-03-15T22:32:47.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:47.867 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2021-03-15T22:32:47.867 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "device_health_metrics", 2021-03-15T22:32:47.868 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:16:34.817191+0000", 2021-03-15T22:32:47.868 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:32:47.868 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:32:47.868 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:32:47.868 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:32:47.869 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:32:47.869 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:32:47.869 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:32:47.869 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:32:47.870 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:32:47.870 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:32:47.870 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:32:47.870 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:32:47.871 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2021-03-15T22:32:47.871 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2021-03-15T22:32:47.871 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2021-03-15T22:32:47.871 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2021-03-15T22:32:47.871 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2021-03-15T22:32:47.872 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:32:47.872 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:32:47.872 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:32:47.872 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:32:47.873 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:32:47.873 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:32:47.873 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:32:47.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.873 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "23", 2021-03-15T22:32:47.874 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:32:47.874 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:32:47.874 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:32:47.874 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:32:47.875 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:32:47.875 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:32:47.875 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:32:47.875 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:32:47.876 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:32:47.876 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:32:47.876 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:32:47.876 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:32:47.876 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:32:47.877 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:32:47.877 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:32:47.877 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:32:47.877 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:32:47.877 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:32:47.878 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:32:47.878 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:32:47.878 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:32:47.878 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:32:47.879 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:32:47.879 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:32:47.879 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:32:47.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:32:47.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.880 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:32:47.880 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:32:47.880 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:32:47.880 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:32:47.881 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:32:47.881 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:32:47.881 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:32:47.881 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:32:47.881 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:32:47.882 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:32:47.882 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:32:47.882 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:32:47.882 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2021-03-15T22:32:47.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.883 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:32:47.883 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_devicehealth": {} 2021-03-15T22:32:47.883 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:47.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:47.884 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2021-03-15T22:32:47.884 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "iscsi", 2021-03-15T22:32:47.884 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:49.906400+0000", 2021-03-15T22:32:47.885 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:32:47.885 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:32:47.885 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:32:47.885 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:32:47.886 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:32:47.886 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:32:47.886 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:32:47.886 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:32:47.886 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:32:47.887 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:32:47.887 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:32:47.887 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:32:47.887 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 3, 2021-03-15T22:32:47.888 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 3, 2021-03-15T22:32:47.888 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 3, 2021-03-15T22:32:47.888 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 3, 2021-03-15T22:32:47.889 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 3, 2021-03-15T22:32:47.889 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:32:47.889 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:32:47.889 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:32:47.889 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:32:47.890 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:32:47.890 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:32:47.890 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:32:47.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.891 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "56", 2021-03-15T22:32:47.891 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:32:47.892 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:32:47.892 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:32:47.892 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:32:47.892 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:32:47.893 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:32:47.893 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:32:47.893 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:32:47.893 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:32:47.893 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:32:47.894 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:32:47.894 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:32:47.894 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:32:47.894 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:32:47.895 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:32:47.895 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:32:47.895 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:32:47.895 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:32:47.896 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:32:47.896 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:32:47.896 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:32:47.896 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:32:47.896 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:32:47.897 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:32:47.897 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:32:47.897 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:32:47.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.898 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:32:47.899 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:32:47.899 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:32:47.900 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:32:47.900 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:32:47.900 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:32:47.901 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:32:47.901 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:32:47.902 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:32:47.902 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:32:47.903 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:32:47.903 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:32:47.904 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:32:47.904 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2021-03-15T22:32:47.904 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:47.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:47.906 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2021-03-15T22:32:47.906 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".rgw.root", 2021-03-15T22:32:47.907 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:52.257970+0000", 2021-03-15T22:32:47.907 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:32:47.908 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:32:47.908 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:32:47.909 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:32:47.909 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:32:47.909 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:32:47.910 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:32:47.910 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:32:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:32:47.911 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:32:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:32:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:32:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:32:47.913 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:32:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:32:47.914 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:32:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:32:47.915 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:32:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:32:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:32:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:32:47.916 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:32:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:32:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:32:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "58", 2021-03-15T22:32:47.917 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:32:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:32:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:32:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:32:47.918 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:32:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:32:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:32:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:32:47.919 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:32:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:32:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:32:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:32:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:32:47.920 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:32:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:32:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:32:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:32:47.921 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:32:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:32:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:32:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:32:47.922 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:32:47.923 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:32:47.923 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:32:47.923 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:32:47.923 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:32:47.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.924 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:32:47.924 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:32:47.925 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:32:47.925 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:32:47.926 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:32:47.926 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:32:47.926 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:32:47.926 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:32:47.927 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:32:47.927 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:32:47.928 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:32:47.928 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:32:47.929 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:32:47.929 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:32:47.929 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:47.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:47.930 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2021-03-15T22:32:47.930 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.log", 2021-03-15T22:32:47.930 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:54.562498+0000", 2021-03-15T22:32:47.931 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:32:47.931 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:32:47.931 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:32:47.931 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:32:47.932 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:32:47.932 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:32:47.932 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:32:47.932 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:32:47.933 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:32:47.933 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:32:47.933 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:32:47.933 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:32:47.933 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:32:47.934 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:32:47.934 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:32:47.934 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:32:47.934 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:32:47.935 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:32:47.935 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:32:47.935 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:32:47.935 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:32:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:32:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:32:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:32:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.936 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "60", 2021-03-15T22:32:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:32:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:32:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:32:47.937 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:32:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:32:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:32:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:32:47.938 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:32:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:32:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:32:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:32:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:32:47.939 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:32:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:32:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:32:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:32:47.940 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:32:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:32:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:32:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:32:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:32:47.941 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:32:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:32:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:32:47.942 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:32:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:32:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:32:47.943 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:32:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:32:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:32:47.944 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:32:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:32:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:32:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:32:47.945 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:32:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:32:47.946 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:32:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:32:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:32:47.947 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:32:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:47.948 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 5, 2021-03-15T22:32:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.control", 2021-03-15T22:32:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:56.243806+0000", 2021-03-15T22:32:47.949 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:32:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:32:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:32:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:32:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:32:47.950 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:32:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:32:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:32:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:32:47.951 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:32:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:32:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:32:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:32:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:32:47.952 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:32:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:32:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:32:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:32:47.953 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "5.16", 2021-03-15T22:32:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 182, 2021-03-15T22:32:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 181, 2021-03-15T22:32:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 181, 2021-03-15T22:32:47.954 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:32:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:32:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "730", 2021-03-15T22:32:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:32:47.955 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "182", 2021-03-15T22:32:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "728", 2021-03-15T22:32:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:32:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:32:47.956 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:32:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:32:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:32:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:32:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:32:47.957 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:32:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:32:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:32:47.958 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:32:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:32:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:32:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:32:47.959 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:32:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:32:47.960 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:32:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:32:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:32:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:32:47.961 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:32:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:32:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:32:47.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:32:47.963 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:32:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:32:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:32:47.964 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:32:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:32:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:32:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:32:47.965 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:32:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:32:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:32:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:32:47.966 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:32:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:32:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 6, 2021-03-15T22:32:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.meta", 2021-03-15T22:32:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:58.730534+0000", 2021-03-15T22:32:47.968 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:32:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:32:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:32:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:32:47.969 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:32:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:32:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:32:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:32:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:32:47.970 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:32:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:32:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:32:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:32:47.971 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:32:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:32:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:32:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:32:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:32:47.972 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:32:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:32:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:32:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:32:47.973 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:32:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:32:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "753", 2021-03-15T22:32:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:32:47.974 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:32:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:32:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:32:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:32:47.975 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:32:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:32:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:32:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:32:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:32:47.976 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:32:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:32:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:32:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:32:47.977 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:32:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:32:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:32:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:32:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:32:47.978 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:32:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:32:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:32:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:32:47.979 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:32:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:32:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:32:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:32:47.980 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:32:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:32:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:32:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:32:47.981 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:32:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:32:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:32:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:32:47.982 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:32:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:32:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:32:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_bias": 4, 2021-03-15T22:32:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 8 2021-03-15T22:32:47.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:32:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:32:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:47.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:32:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2021-03-15T22:32:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:20:03.377294+0000", 2021-03-15T22:32:47.985 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2021-03-15T22:32:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-03-15T22:32:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:32:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:32:47.986 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:32:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:32:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:32:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:32:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:32:47.987 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:32:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:32:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:32:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2021-03-15T22:32:47.988 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2021-03-15T22:32:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2021-03-15T22:32:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2021-03-15T22:32:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2021-03-15T22:32:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:32:47.989 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:32:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:32:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:32:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:32:47.990 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:32:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:32:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "817", 2021-03-15T22:32:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:32:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:32:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:32:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:32:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:32:47.992 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 681, 2021-03-15T22:32:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 817, 2021-03-15T22:32:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:32:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:32:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:32:47.993 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:32:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:32:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:32:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:32:47.994 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:32:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:32:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:32:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:32:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:32:47.995 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:32:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:32:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:32:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:32:47.996 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:32:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:32:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:32:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:32:47.997 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:32:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:32:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:32:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:32:47.998 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:32:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:32:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:32:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:32:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:32:47.999 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:32:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:32:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:32:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2021-03-15T22:32:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.000 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:32:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:32:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.001 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:32:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd", 2021-03-15T22:32:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:32:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:32:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:32:48.002 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:32:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:32:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:32:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 8, 2021-03-15T22:32:48.003 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 740, 2021-03-15T22:32:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:32:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:32:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:32:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2021-03-15T22:32:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:32:48.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2021-03-15T22:32:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:32:48.006 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:32:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2021-03-15T22:32:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:32:48.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2021-03-15T22:32:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:32:48.009 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:32:48.010 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2021-03-15T22:32:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:32:48.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2021-03-15T22:32:48.012 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:32:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:32:48.013 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2021-03-15T22:32:48.014 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:32:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2021-03-15T22:32:48.015 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:32:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.016 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/188228176", 2021-03-15T22:32:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/188228176", 2021-03-15T22:32:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/188228176", 2021-03-15T22:32:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/188228176", 2021-03-15T22:32:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:32:48.017 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:32:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:32:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:32:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526", 2021-03-15T22:32:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:32:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:32:48.019 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:32:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.999237060546875, 2021-03-15T22:32:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:32:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:32:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 13, 2021-03-15T22:32:48.020 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 792, 2021-03-15T22:32:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:32:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:32:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:32:48.021 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2021-03-15T22:32:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:32:48.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2021-03-15T22:32:48.023 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:32:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:32:48.024 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2021-03-15T22:32:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:32:48.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2021-03-15T22:32:48.026 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:32:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:32:48.027 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2021-03-15T22:32:48.028 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:32:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2021-03-15T22:32:48.029 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:32:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:32:48.030 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.031 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2021-03-15T22:32:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:32:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.032 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2021-03-15T22:32:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:32:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/3924955423", 2021-03-15T22:32:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/3924955423", 2021-03-15T22:32:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/3924955423", 2021-03-15T22:32:48.034 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/3924955423", 2021-03-15T22:32:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:32:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:32:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:32:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:32:48.036 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05", 2021-03-15T22:32:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:32:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:32:48.037 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:32:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:32:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 18, 2021-03-15T22:32:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 489, 2021-03-15T22:32:48.038 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 681, 2021-03-15T22:32:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 791, 2021-03-15T22:32:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 490, 2021-03-15T22:32:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:32:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:32:48.039 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2021-03-15T22:32:48.040 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:32:48.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.041 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.041 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2021-03-15T22:32:48.041 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:32:48.042 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.042 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.042 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:32:48.043 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.043 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.043 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2021-03-15T22:32:48.044 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:32:48.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.044 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.044 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2021-03-15T22:32:48.045 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:32:48.045 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.045 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.046 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:32:48.046 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.046 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.046 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2021-03-15T22:32:48.047 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:32:48.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.047 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.047 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2021-03-15T22:32:48.048 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:32:48.048 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.048 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.049 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:32:48.049 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.049 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.049 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2021-03-15T22:32:48.050 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:32:48.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.050 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.051 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2021-03-15T22:32:48.051 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:32:48.051 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.051 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.052 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/1676192618", 2021-03-15T22:32:48.052 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/1676192618", 2021-03-15T22:32:48.052 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/1676192618", 2021-03-15T22:32:48.052 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/1676192618", 2021-03-15T22:32:48.053 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:32:48.053 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:32:48.053 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:32:48.053 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.054 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:32:48.054 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56", 2021-03-15T22:32:48.054 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:32:48.055 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:32:48.055 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:32:48.055 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:32:48.055 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:32:48.055 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:32:48.056 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 26, 2021-03-15T22:32:48.056 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 740, 2021-03-15T22:32:48.056 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:32:48.056 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:32:48.057 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:32:48.057 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.057 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.057 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2021-03-15T22:32:48.058 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:32:48.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.058 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.059 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2021-03-15T22:32:48.059 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:32:48.059 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.059 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.060 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:32:48.060 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.060 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.060 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2021-03-15T22:32:48.061 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:32:48.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.061 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.062 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2021-03-15T22:32:48.062 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:32:48.062 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.062 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.063 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:32:48.063 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.063 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.064 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2021-03-15T22:32:48.064 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:32:48.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.064 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.065 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2021-03-15T22:32:48.065 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:32:48.065 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.065 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.066 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:32:48.066 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.066 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.066 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.067 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2021-03-15T22:32:48.067 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:32:48.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.067 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.068 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2021-03-15T22:32:48.068 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:32:48.068 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.068 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.069 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/3960968636", 2021-03-15T22:32:48.069 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/3960968636", 2021-03-15T22:32:48.069 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/3960968636", 2021-03-15T22:32:48.069 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/3960968636", 2021-03-15T22:32:48.070 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:32:48.070 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:32:48.070 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:32:48.070 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.071 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:32:48.071 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237", 2021-03-15T22:32:48.072 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:32:48.072 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:32:48.072 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:32:48.072 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:32:48.072 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:32:48.073 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:32:48.073 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 31, 2021-03-15T22:32:48.073 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 772, 2021-03-15T22:32:48.073 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:32:48.074 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:32:48.074 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:32:48.074 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.075 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.075 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6800", 2021-03-15T22:32:48.075 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:32:48.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.076 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.076 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6801", 2021-03-15T22:32:48.076 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:32:48.076 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.076 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.077 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:32:48.077 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.078 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.078 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6802", 2021-03-15T22:32:48.078 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:32:48.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.079 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.079 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6803", 2021-03-15T22:32:48.079 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:32:48.079 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.080 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.080 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:32:48.080 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6806", 2021-03-15T22:32:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:32:48.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6807", 2021-03-15T22:32:48.082 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:32:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:32:48.083 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6804", 2021-03-15T22:32:48.084 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:32:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6805", 2021-03-15T22:32:48.085 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:32:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.086 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6801/37386122", 2021-03-15T22:32:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6803/37386122", 2021-03-15T22:32:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6807/37386122", 2021-03-15T22:32:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6805/37386122", 2021-03-15T22:32:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:32:48.087 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:32:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:32:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:32:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2", 2021-03-15T22:32:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:32:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:32:48.089 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:32:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:32:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 38, 2021-03-15T22:32:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 539, 2021-03-15T22:32:48.090 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 616, 2021-03-15T22:32:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 740, 2021-03-15T22:32:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 540, 2021-03-15T22:32:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:32:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:32:48.091 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6808", 2021-03-15T22:32:48.092 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:32:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6809", 2021-03-15T22:32:48.093 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:32:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.094 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:32:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6810", 2021-03-15T22:32:48.095 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:32:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.096 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6811", 2021-03-15T22:32:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:32:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.097 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:32:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.098 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6814", 2021-03-15T22:32:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:32:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.099 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6815", 2021-03-15T22:32:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:32:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.100 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:32:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.101 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6812", 2021-03-15T22:32:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:32:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.102 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6813", 2021-03-15T22:32:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:32:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6809/978017133", 2021-03-15T22:32:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6811/978017133", 2021-03-15T22:32:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6815/978017133", 2021-03-15T22:32:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6813/978017133", 2021-03-15T22:32:48.104 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:32:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:32:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:32:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:32:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe", 2021-03-15T22:32:48.106 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:32:48.107 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 0, 2021-03-15T22:32:48.107 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0, 2021-03-15T22:32:48.107 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:32:48.107 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:32:48.107 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:32:48.108 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 45, 2021-03-15T22:32:48.108 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 733, 2021-03-15T22:32:48.108 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:32:48.108 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:32:48.108 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:32:48.109 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.109 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.109 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6816", 2021-03-15T22:32:48.110 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:32:48.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.110 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.111 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6817", 2021-03-15T22:32:48.111 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:32:48.111 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.111 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.112 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:32:48.112 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.112 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.112 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6818", 2021-03-15T22:32:48.113 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:32:48.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.113 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.114 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6819", 2021-03-15T22:32:48.114 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:32:48.114 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.114 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.115 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:32:48.115 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.115 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.116 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6822", 2021-03-15T22:32:48.116 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:32:48.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.117 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.117 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6823", 2021-03-15T22:32:48.117 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:32:48.117 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.117 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.118 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:32:48.118 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.118 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.119 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.119 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6820", 2021-03-15T22:32:48.119 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:32:48.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.120 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.120 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6821", 2021-03-15T22:32:48.120 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:32:48.120 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.121 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.121 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.121 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6817/171378275", 2021-03-15T22:32:48.121 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6819/171378275", 2021-03-15T22:32:48.121 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6823/171378275", 2021-03-15T22:32:48.122 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6821/171378275", 2021-03-15T22:32:48.122 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:32:48.122 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:32:48.122 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:32:48.123 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.123 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:32:48.123 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614", 2021-03-15T22:32:48.124 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:32:48.124 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:32:48.124 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:32:48.124 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:32:48.125 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:32:48.125 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:32:48.125 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 52, 2021-03-15T22:32:48.125 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 800, 2021-03-15T22:32:48.125 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:32:48.126 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:32:48.126 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:32:48.126 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.127 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.127 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6824", 2021-03-15T22:32:48.127 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:32:48.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.128 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.128 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6825", 2021-03-15T22:32:48.128 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:32:48.128 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.129 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.129 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:32:48.129 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.130 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.130 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6826", 2021-03-15T22:32:48.130 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:32:48.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.131 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.131 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.131 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6827", 2021-03-15T22:32:48.131 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:32:48.132 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.132 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.132 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.132 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:32:48.133 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.133 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.133 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6830", 2021-03-15T22:32:48.133 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:32:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.134 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6831", 2021-03-15T22:32:48.135 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:32:48.135 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.135 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.135 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.135 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:32:48.136 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:32:48.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.136 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:32:48.136 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6828", 2021-03-15T22:32:48.137 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:32:48.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.137 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:32:48.137 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6829", 2021-03-15T22:32:48.138 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:32:48.138 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.138 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.139 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6825/652003948", 2021-03-15T22:32:48.139 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6827/652003948", 2021-03-15T22:32:48.139 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6831/652003948", 2021-03-15T22:32:48.139 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6829/652003948", 2021-03-15T22:32:48.139 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:32:48.140 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:32:48.140 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:32:48.140 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.140 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.141 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:32:48.141 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2021-03-15T22:32:48.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.141 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:32:48.141 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:32:48.142 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:32:48.142 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:32:48.142 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:32:48.142 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:32:48.143 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:31:57.184113+0000", 2021-03-15T22:32:48.143 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:32:48.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.143 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:32:48.144 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:32:48.144 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:32:48.144 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:32:48.144 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:32:48.145 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:32:48.145 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:32:15.149298+0000", 2021-03-15T22:32:48.145 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:32:48.145 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.146 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:32:48.146 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:32:48.146 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:32:48.146 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:32:48.147 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:32:48.147 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:32:48.147 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:32:31.228005+0000", 2021-03-15T22:32:48.147 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 490 2021-03-15T22:32:48.148 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.148 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.148 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:32:48.148 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:32:48.148 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:32:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:32:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:32:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:32:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:32:43.675089+0000", 2021-03-15T22:32:48.149 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:32:48.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.150 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:32:48.150 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:32:48.151 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:32:48.151 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:32:48.151 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:32:48.151 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:32:48.152 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:32:02.934885+0000", 2021-03-15T22:32:48.152 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:32:48.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.152 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:32:48.153 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:32:48.153 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:32:48.153 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:32:48.153 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:32:48.153 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:32:48.154 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:32:25.343730+0000", 2021-03-15T22:32:48.154 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 540 2021-03-15T22:32:48.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.155 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:32:48.155 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:32:48.155 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:32:48.155 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:32:48.155 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:32:48.156 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 65536, 2021-03-15T22:32:48.156 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:32:38.711247+0000", 2021-03-15T22:32:48.156 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:32:48.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.157 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:32:48.157 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:32:48.157 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:32:48.157 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:32:48.157 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:32:48.158 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:32:48.158 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:31:55.356192+0000", 2021-03-15T22:32:48.158 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:32:48.158 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.159 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:32:48.159 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [ 2021-03-15T22:32:48.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.159 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.3", 2021-03-15T22:32:48.160 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:32:48.160 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:32:48.160 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:32:48.160 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2021-03-15T22:32:48.160 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.161 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.161 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:32:48.161 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2021-03-15T22:32:48.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.162 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.3", 2021-03-15T22:32:48.162 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:32:48.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.162 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:32:48.162 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:32:48.163 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.163 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.164 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "3.14", 2021-03-15T22:32:48.164 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:32:48.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.164 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:32:48.164 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:32:48.165 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.165 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.165 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.9", 2021-03-15T22:32:48.166 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:32:48.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.166 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 3, 2021-03-15T22:32:48.166 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:32:48.167 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.167 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.167 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.a", 2021-03-15T22:32:48.168 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:32:48.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.168 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:32:48.168 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:32:48.169 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.169 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.170 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.b", 2021-03-15T22:32:48.170 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:32:48.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.170 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:32:48.170 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:32:48.171 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.171 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.172 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "4.f", 2021-03-15T22:32:48.172 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:32:48.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.172 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:32:48.172 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 2 2021-03-15T22:32:48.173 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.173 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.173 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.173 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.f", 2021-03-15T22:32:48.174 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:32:48.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.174 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:32:48.174 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 0 2021-03-15T22:32:48.175 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.175 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.175 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.175 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:32:48.175 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2021-03-15T22:32:48.176 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2021-03-15T22:32:48.176 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": { 2021-03-15T22:32:48.176 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:32:48.176 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4087324200": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:32:48.177 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4141143304": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:32:48.177 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:32:48.177 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2779879991": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:32:48.177 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:32:48.177 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:32:48.178 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2483102204": "2021-03-16T22:13:41.603652+0000" 2021-03-15T22:32:48.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.178 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2021-03-15T22:32:48.178 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2021-03-15T22:32:48.178 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2021-03-15T22:32:48.179 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2021-03-15T22:32:48.179 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2021-03-15T22:32:48.179 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2021-03-15T22:32:48.179 INFO:teuthology.orchestra.run.smithi076.stdout: "ruleset-failure-domain": "osd", 2021-03-15T22:32:48.180 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2021-03-15T22:32:48.180 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.180 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.180 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2021-03-15T22:32:48.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.181 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:32:48.181 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:32:48.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.181 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 503, 2021-03-15T22:32:48.182 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:32:48.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.182 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 573, 2021-03-15T22:32:48.183 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:32:48.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.183 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 591, 2021-03-15T22:32:48.183 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:32:48.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.184 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 615, 2021-03-15T22:32:48.184 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:32:48.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.185 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 629, 2021-03-15T22:32:48.185 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:32:48.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:32:48.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.186 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 661, 2021-03-15T22:32:48.186 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:32:48.186 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.186 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:32:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [ 2021-03-15T22:32:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.187 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:32:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:32:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:32:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 591, 2021-03-15T22:32:48.188 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:32:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:32:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:32:48.189 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2021-03-15T22:32:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2021-03-15T22:32:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2021-03-15T22:32:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2021-03-15T22:32:48.190 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2021-03-15T22:32:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2021-03-15T22:32:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2021-03-15T22:32:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2021-03-15T22:32:48.191 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2021-03-15T22:32:48.192 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:32:48.192 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:32:48.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:48.214 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:48.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:48.216 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:48.317 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:48.339 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:48.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:48.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:48.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:48.458 INFO:tasks.rados.rados.0.smithi076.stdout:3047: done (13 left) 2021-03-15T22:32:48.459 INFO:tasks.rados.rados.0.smithi076.stdout:3048: done (12 left) 2021-03-15T22:32:48.459 INFO:tasks.rados.rados.0.smithi076.stdout:3049: done (11 left) 2021-03-15T22:32:48.459 INFO:tasks.rados.rados.0.smithi076.stdout:3050: done (10 left) 2021-03-15T22:32:48.459 INFO:tasks.rados.rados.0.smithi076.stdout:3062: done (9 left) 2021-03-15T22:32:48.460 INFO:tasks.rados.rados.0.smithi076.stdout:3063: read oid 42 snap -1 2021-03-15T22:32:48.460 INFO:tasks.rados.rados.0.smithi076.stdout:3063: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:32:48.460 INFO:tasks.rados.rados.0.smithi076.stdout:3064: rollback oid 3 current snap is 353 2021-03-15T22:32:48.461 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 318 2021-03-15T22:32:48.461 INFO:tasks.rados.rados.0.smithi076.stdout:3065: snap_create 2021-03-15T22:32:48.461 INFO:tasks.rados.rados.0.smithi076.stdout:3056: expect (ObjNum 1044 snap 350 seq_num 1044) 2021-03-15T22:32:48.472 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:48.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:48.474 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:48.498 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:48 smithi125 conmon[56739]: cluster 2021-03-15T22:32:47.180609+0000 mgr.y ( 2021-03-15T22:32:48.499 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:48 smithi125 conmon[56739]: mgr.14144) 949 : cluster [DBG] pgmap v1369: 148 pgs: 148 active+clean; 528 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2021-03-15T22:32:48.499 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:48 smithi125 conmon[56739]: cluster 2021-03-15T22:32:47.274449+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e817: 8 total, 8 up, 7 in 2021-03-15T22:32:48.499 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:48 smithi125 conmon[56739]: audit 2021-03-15T22:32:47.855805+0000 mon.c (mon.1) 64 : audit [DBG] from='client.? 172.21.15.76:0/3974150862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:32:48.575 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:48.596 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:48.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:48.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:48.599 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:48 smithi076 conmon[62156]: cluster 2021-03-15T22:32:47.180609+0000 mgr.y (mgr.14144) 949 : cluster [DBG] pgmap v1369: 148 pgs: 148 active+clean; 528 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2021-03-15T22:32:48.599 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:48 smithi076 conmon[62156]: cluster 2021-03-15T22:32:47.274449+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e817: 8 total, 8 up, 7 in 2021-03-15T22:32:48.600 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:48 smithi076 conmon[62156]: audit 2021-03-15T22:32:47.855805+0000 mon.c (mon.1) 64 : audit [DBG] from='client.? 172.21.15.76:0/3974150862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:32:48.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:48 smithi076 conmon[56514]: cluster 2021-03-15T22:32:47.180609+0000 mgr.y (mgr.14144) 949 : cluster [DBG] pgmap v1369: 148 pgs: 148 active+clean; 528 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 1.9 MiB/s wr, 10 op/s 2021-03-15T22:32:48.600 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:48 smithi076 conmon[56514]: cluster 2021-03-15T22:32:47.274449+0000 mon.a (mon.0) 2111 : cluster [DBG] osdmap e817: 8 total, 8 up, 7 in 2021-03-15T22:32:48.601 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:48 smithi076 conmon[56514]: audit 2021-03-15T22:32:47.855805+0000 mon.c (mon.1) 64 : audit [DBG] from='client.? 172.21.15.76:0/3974150862' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:32:48.629 INFO:tasks.thrashosds.thrasher:j is {'epoch': 817, 'fsid': '8a572802-85db-11eb-9091-001a4aab830c', 'created': '2021-03-15T22:13:20.370795+0000', 'modified': '2021-03-15T22:32:47.270408+0000', 'last_up_change': '2021-03-15T22:30:30.205134+0000', 'last_in_change': '2021-03-15T22:31:27.245203+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-15T22:16:34.817191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-15T22:17:49.906400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-15T22:17:52.257970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-15T22:17:54.562498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-15T22:17:56.243806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 182, 'last_epoch_started': 181, 'last_epoch_clean': 181, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '730', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '182', 'last_force_op_resend_preluminous': '728', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-15T22:17:58.730534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '753', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-03-15T22:20:03.377294+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '817', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 681, 'snap_epoch': 817, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 740, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 188228176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 188228176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 188228176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 188228176}]}, 'public_addr': '172.21.15.76:6803/188228176', 'cluster_addr': '172.21.15.76:6805/188228176', 'heartbeat_back_addr': '172.21.15.76:6809/188228176', 'heartbeat_front_addr': '172.21.15.76:6807/188228176', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '97bd3ffc-15c8-4a6f-b37a-217813def526', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.999237060546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 792, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 3924955423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 3924955423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 3924955423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 3924955423}]}, 'public_addr': '172.21.15.76:6811/3924955423', 'cluster_addr': '172.21.15.76:6813/3924955423', 'heartbeat_back_addr': '172.21.15.76:6817/3924955423', 'heartbeat_front_addr': '172.21.15.76:6815/3924955423', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a7f5f115-e69e-47d9-b6c3-ff386ddfaa05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 18, 'last_clean_end': 489, 'up_from': 681, 'up_thru': 791, 'down_at': 490, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 1676192618}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 1676192618}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 1676192618}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 1676192618}]}, 'public_addr': '172.21.15.76:6819/1676192618', 'cluster_addr': '172.21.15.76:6821/1676192618', 'heartbeat_back_addr': '172.21.15.76:6825/1676192618', 'heartbeat_front_addr': '172.21.15.76:6823/1676192618', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a0bb7f88-0114-47cb-a82c-d8fb002aac56', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 740, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 3960968636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 3960968636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 3960968636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 3960968636}]}, 'public_addr': '172.21.15.76:6827/3960968636', 'cluster_addr': '172.21.15.76:6829/3960968636', 'heartbeat_back_addr': '172.21.15.76:6833/3960968636', 'heartbeat_front_addr': '172.21.15.76:6831/3960968636', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b650611c-6101-47e8-9ca9-94e02cf8b237', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 772, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6800', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6801', 'nonce': 37386122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6802', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6803', 'nonce': 37386122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6806', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6807', 'nonce': 37386122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6804', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6805', 'nonce': 37386122}]}, 'public_addr': '172.21.15.125:6801/37386122', 'cluster_addr': '172.21.15.125:6803/37386122', 'heartbeat_back_addr': '172.21.15.125:6807/37386122', 'heartbeat_front_addr': '172.21.15.125:6805/37386122', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7d6ceff2-7093-4ab9-8b96-ade448093bf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 38, 'last_clean_end': 539, 'up_from': 616, 'up_thru': 740, 'down_at': 540, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6808', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6809', 'nonce': 978017133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6810', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6811', 'nonce': 978017133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6814', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6815', 'nonce': 978017133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6812', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6813', 'nonce': 978017133}]}, 'public_addr': '172.21.15.125:6809/978017133', 'cluster_addr': '172.21.15.125:6811/978017133', 'heartbeat_back_addr': '172.21.15.125:6815/978017133', 'heartbeat_front_addr': '172.21.15.125:6813/978017133', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98b72022-a9a7-4ee2-89a1-cac037de7afe', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 733, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6816', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6817', 'nonce': 171378275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6818', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6819', 'nonce': 171378275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6822', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6823', 'nonce': 171378275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6820', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6821', 'nonce': 171378275}]}, 'public_addr': '172.21.15.125:6817/171378275', 'cluster_addr': '172.21.15.125:6819/171378275', 'heartbeat_back_addr': '172.21.15.125:6823/171378275', 'heartbeat_front_addr': '172.21.15.125:6821/171378275', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f99ea650-b2dc-4d87-a44c-56a897edd614', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 800, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6824', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6825', 'nonce': 652003948}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6826', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6827', 'nonce': 652003948}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6830', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6831', 'nonce': 652003948}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6828', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6829', 'nonce': 652003948}]}, 'public_addr': '172.21.15.125:6825/652003948', 'cluster_addr': '172.21.15.125:6827/652003948', 'heartbeat_back_addr': '172.21.15.125:6831/652003948', 'heartbeat_front_addr': '172.21.15.125:6829/652003948', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:31:57.184113+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:32:15.149298+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:32:31.228005+0000', 'dead_epoch': 490}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:32:43.675089+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:32:02.934885+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:32:25.343730+0000', 'dead_epoch': 540}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 65536, 'last_purged_snaps_scrub': '2021-03-15T22:32:38.711247+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:31:55.356192+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '7.3', 'osds': [1, 6, 2]}], 'pg_upmap_items': [{'pgid': '3.3', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '3.14', 'mappings': [{'from': 4, 'to': 2}]}, {'pgid': '4.9', 'mappings': [{'from': 3, 'to': 2}]}, {'pgid': '4.a', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.b', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '4.f', 'mappings': [{'from': 1, 'to': 2}]}, {'pgid': '6.f', 'mappings': [{'from': 4, 'to': 0}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.76:6800/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4087324200': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4141143304': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:6801/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/2779879991': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6801/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6800/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:0/2483102204': '2021-03-16T22:13:41.603652+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 503, 'length': 1}, {'begin': 573, 'length': 1}, {'begin': 591, 'length': 1}, {'begin': 615, 'length': 1}, {'begin': 629, 'length': 1}, {'begin': 661, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 591, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-03-15T22:32:48.629 INFO:tasks.thrashosds.thrasher:Clearing pg_upmap on 3.3 2021-03-15T22:32:48.630 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd rm-pg-upmap-items 3.3 2021-03-15T22:32:48.699 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:48.711 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:32:48.717 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:32:48.717 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:32:48.718 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:32:48.718 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:32:48.718 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.718 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(osd.7.52:26 7.0 7:08414d91:::smithi07694878-42:head [assert-version v1737,copy-get max 8388608] snapc 0=[] RETRY=1 ondisk+retry+read+known_if_redirected e642)", 2021-03-15T22:32:48.719 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:51.199690+0000", 2021-03-15T22:32:48.719 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 177.51392099700001, 2021-03-15T22:32:48.719 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0301155609999999, 2021-03-15T22:32:48.719 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.720 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:48.720 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.720 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.720 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.720 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199690+0000", 2021-03-15T22:32:48.721 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.721 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.721 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.721 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.722 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199690+0000", 2021-03-15T22:32:48.722 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.689e-06 2021-03-15T22:32:48.722 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.722 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.722 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.723 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199692+0000", 2021-03-15T22:32:48.723 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8900000000000002e-06 2021-03-15T22:32:48.723 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.723 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.724 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.724 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199702+0000", 2021-03-15T22:32:48.724 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1841999999999999e-05 2021-03-15T22:32:48.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.724 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.725 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.725 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199714+0000", 2021-03-15T22:32:48.725 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4039999999999998e-06 2021-03-15T22:32:48.725 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.725 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.726 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.726 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199721+0000", 2021-03-15T22:32:48.726 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8303000000000001e-05 2021-03-15T22:32:48.726 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.727 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.727 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.727 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199780+0000", 2021-03-15T22:32:48.727 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0937e-05 2021-03-15T22:32:48.727 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.728 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.728 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:32:48.728 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.199791+0000", 2021-03-15T22:32:48.728 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010780039 2021-03-15T22:32:48.729 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.729 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.729 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.729 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.210571+0000", 2021-03-15T22:32:48.729 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0107e-05 2021-03-15T22:32:48.730 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.730 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.730 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:48.730 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:51.210581+0000", 2021-03-15T22:32:48.731 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0163172810000001 2021-03-15T22:32:48.731 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.731 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.731 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.731 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226898+0000", 2021-03-15T22:32:48.732 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012874800000000001 2021-03-15T22:32:48.732 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.732 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.732 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.732 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227027+0000", 2021-03-15T22:32:48.733 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0027783209999999998 2021-03-15T22:32:48.733 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.733 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.733 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.734 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.229805+0000", 2021-03-15T22:32:48.734 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0301155609999999 2021-03-15T22:32:48.734 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.734 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.734 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.735 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.735 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.735 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2106 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94751378609664 gen 7] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:32:48.735 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.131311+0000", 2021-03-15T22:32:48.736 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 176.58229999, 2021-03-15T22:32:48.736 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.092023303000000001, 2021-03-15T22:32:48.736 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.736 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:48.736 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.737 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:32:48.737 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:32:48.737 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2106 2021-03-15T22:32:48.737 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.737 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.738 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.738 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.738 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131311+0000", 2021-03-15T22:32:48.738 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.739 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.739 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.739 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.739 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131311+0000", 2021-03-15T22:32:48.739 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9400000000000004e-07 2021-03-15T22:32:48.740 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.740 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.740 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.740 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131312+0000", 2021-03-15T22:32:48.740 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2810000000000001e-06 2021-03-15T22:32:48.741 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.741 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.741 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.741 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131313+0000", 2021-03-15T22:32:48.742 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6209999999999998e-06 2021-03-15T22:32:48.742 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.742 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.742 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.742 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131319+0000", 2021-03-15T22:32:48.743 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9940000000000002e-06 2021-03-15T22:32:48.743 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.743 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.743 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.744 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131323+0000", 2021-03-15T22:32:48.744 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0538999999999998e-05 2021-03-15T22:32:48.744 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.744 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.744 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.745 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131384+0000", 2021-03-15T22:32:48.745 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5080000000000005e-06 2021-03-15T22:32:48.745 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.745 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.746 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:48.746 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131392+0000", 2021-03-15T22:32:48.746 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.086493444000000003 2021-03-15T22:32:48.746 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.746 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.747 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.747 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.217886+0000", 2021-03-15T22:32:48.748 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010953 2021-03-15T22:32:48.748 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.748 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.748 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.749 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.218981+0000", 2021-03-15T22:32:48.749 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043527219999999998 2021-03-15T22:32:48.749 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.749 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.750 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.750 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.223334+0000", 2021-03-15T22:32:48.750 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.092023303000000001 2021-03-15T22:32:48.750 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.750 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.751 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.751 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.751 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.751 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2109 5.f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94751378609664 gen 8] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:32:48.752 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:32:48.752 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 176.58227705300001, 2021-03-15T22:32:48.752 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.093324206000000007, 2021-03-15T22:32:48.752 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.752 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:48.753 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.753 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:32:48.753 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:32:48.753 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2109 2021-03-15T22:32:48.754 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.754 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.754 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.754 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.754 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:32:48.755 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.755 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.756 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:32:48.756 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0699999999999996e-07 2021-03-15T22:32:48.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.757 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.757 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131334+0000", 2021-03-15T22:32:48.758 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.066e-06 2021-03-15T22:32:48.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.758 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.758 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131336+0000", 2021-03-15T22:32:48.759 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9439999999999999e-06 2021-03-15T22:32:48.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.760 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.760 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.760 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131342+0000", 2021-03-15T22:32:48.760 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1559999999999997e-06 2021-03-15T22:32:48.760 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.761 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.761 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.761 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131347+0000", 2021-03-15T22:32:48.761 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5375000000000003e-05 2021-03-15T22:32:48.762 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.762 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.762 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.762 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131412+0000", 2021-03-15T22:32:48.763 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2870000000000004e-06 2021-03-15T22:32:48.763 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.764 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.765 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:32:48.765 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131417+0000", 2021-03-15T22:32:48.765 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.087846846000000006 2021-03-15T22:32:48.765 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.766 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.766 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.766 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.219264+0000", 2021-03-15T22:32:48.766 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.5641000000000003e-05 2021-03-15T22:32:48.767 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.767 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.767 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.767 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.219340+0000", 2021-03-15T22:32:48.767 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0053179840000000004 2021-03-15T22:32:48.768 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.768 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.768 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.768 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.224658+0000", 2021-03-15T22:32:48.769 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.093324206000000007 2021-03-15T22:32:48.769 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.769 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.769 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.769 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.770 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.770 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5974 7.e 7:7ac381e0:::smithi07694878-24:head [copy-from ver 1703] snapc 1fc=[1fc,1fb,1f8,1f7,1f6,1f4,1f3,1f0,1ee,1e9,1e7,1e6,1e4,1e0,1df,1dc,1d5,1d4,1c7,1c5,1c3,1ae,1a4,1a1,19c,193,17a,176,141] ondisk+write+known_if_redirected e643)", 2021-03-15T22:32:48.770 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.226697+0000", 2021-03-15T22:32:48.770 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 176.486913501, 2021-03-15T22:32:48.771 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031270606999999999, 2021-03-15T22:32:48.771 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.771 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:48.771 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.771 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.772 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.772 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5974 2021-03-15T22:32:48.772 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.772 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.773 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.773 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.773 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226697+0000", 2021-03-15T22:32:48.773 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.774 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.774 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.774 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.774 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226697+0000", 2021-03-15T22:32:48.775 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6269999999999999e-06 2021-03-15T22:32:48.775 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.775 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.775 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.775 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226702+0000", 2021-03-15T22:32:48.776 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.302e-06 2021-03-15T22:32:48.776 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.776 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.776 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.777 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226703+0000", 2021-03-15T22:32:48.777 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1020000000000002e-06 2021-03-15T22:32:48.777 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.777 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.777 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.778 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226709+0000", 2021-03-15T22:32:48.778 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6780000000000002e-06 2021-03-15T22:32:48.778 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.778 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.779 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.779 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226713+0000", 2021-03-15T22:32:48.779 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025251599999999999 2021-03-15T22:32:48.779 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.779 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.780 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.780 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226965+0000", 2021-03-15T22:32:48.780 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010260900000000001 2021-03-15T22:32:48.780 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.780 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.781 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.781 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227068+0000", 2021-03-15T22:32:48.781 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.013635573999999999 2021-03-15T22:32:48.781 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.782 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,3", 2021-03-15T22:32:48.782 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.240704+0000", 2021-03-15T22:32:48.782 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010619755 2021-03-15T22:32:48.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.783 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.783 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:48.783 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.251323+0000", 2021-03-15T22:32:48.783 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024678149999999999 2021-03-15T22:32:48.784 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.784 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.784 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.784 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.253791+0000", 2021-03-15T22:32:48.784 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0040414350000000003 2021-03-15T22:32:48.785 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.785 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.785 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.786 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.257833+0000", 2021-03-15T22:32:48.786 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9538e-05 2021-03-15T22:32:48.786 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.787 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.787 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:48.787 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.257872+0000", 2021-03-15T22:32:48.787 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5655999999999994e-05 2021-03-15T22:32:48.788 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.788 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.788 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.788 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.257968+0000", 2021-03-15T22:32:48.788 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031270606999999999 2021-03-15T22:32:48.789 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.789 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.789 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.789 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.790 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.790 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5975 7.e 7:7ac381e0:::smithi07694878-24:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e643)", 2021-03-15T22:32:48.790 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.226719+0000", 2021-03-15T22:32:48.790 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 176.48689125199999, 2021-03-15T22:32:48.790 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031554302999999999, 2021-03-15T22:32:48.791 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.791 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:48.792 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.792 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.792 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.792 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5975 2021-03-15T22:32:48.793 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.793 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.793 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.793 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.793 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226719+0000", 2021-03-15T22:32:48.794 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.794 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.794 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.794 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.795 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226719+0000", 2021-03-15T22:32:48.795 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9899999999999999e-07 2021-03-15T22:32:48.795 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.795 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.795 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.796 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226720+0000", 2021-03-15T22:32:48.796 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3540000000000001e-06 2021-03-15T22:32:48.796 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.796 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.796 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.797 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226722+0000", 2021-03-15T22:32:48.797 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.049e-06 2021-03-15T22:32:48.797 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.797 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.797 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.798 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226728+0000", 2021-03-15T22:32:48.798 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9450000000000001e-06 2021-03-15T22:32:48.798 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.798 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.799 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.799 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.226733+0000", 2021-03-15T22:32:48.799 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048394300000000003 2021-03-15T22:32:48.799 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.799 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.800 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.800 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227216+0000", 2021-03-15T22:32:48.800 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3120000000000001e-05 2021-03-15T22:32:48.800 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.800 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:32:48.801 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.227260+0000", 2021-03-15T22:32:48.801 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020041761000000002 2021-03-15T22:32:48.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.801 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.802 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.802 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.247301+0000", 2021-03-15T22:32:48.802 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7263999999999997e-05 2021-03-15T22:32:48.802 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.803 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.803 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:48.803 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.247389+0000", 2021-03-15T22:32:48.803 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.010630624999999999 2021-03-15T22:32:48.803 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.804 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.804 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.804 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258019+0000", 2021-03-15T22:32:48.804 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6768999999999995e-05 2021-03-15T22:32:48.804 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.805 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.805 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.805 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258116+0000", 2021-03-15T22:32:48.806 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015767400000000001 2021-03-15T22:32:48.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.806 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.807 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.807 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.258274+0000", 2021-03-15T22:32:48.807 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.031554302999999999 2021-03-15T22:32:48.807 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.807 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.808 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.808 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.808 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.808 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6000 7.0 7:00c03349:::smithi07694878-40:head [copy-from ver 1703] snapc 200=[200,1fb,1f8,1f7,1f6,1f4,1f0,1ee,1e9,1e7,1e6,1e4,1e0,1df,1dc,1d5,1d4,1c7,1c5,1c3,1ae,1a4,1a1,19c,193,17a,176] ondisk+write+known_if_redirected e647)", 2021-03-15T22:32:48.808 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:56.265169+0000", 2021-03-15T22:32:48.809 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 172.44844207700001, 2021-03-15T22:32:48.809 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020808416999999999, 2021-03-15T22:32:48.809 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.809 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:48.809 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.810 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.810 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.810 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6000 2021-03-15T22:32:48.810 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.811 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.811 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.811 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.811 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265169+0000", 2021-03-15T22:32:48.811 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.812 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.812 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.812 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265169+0000", 2021-03-15T22:32:48.812 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.057e-06 2021-03-15T22:32:48.813 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.813 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.813 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.813 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265174+0000", 2021-03-15T22:32:48.813 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1699999999999999e-06 2021-03-15T22:32:48.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.814 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.814 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.814 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265178+0000", 2021-03-15T22:32:48.815 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5769000000000003e-05 2021-03-15T22:32:48.815 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.815 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.815 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265214+0000", 2021-03-15T22:32:48.816 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.092e-06 2021-03-15T22:32:48.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.816 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.816 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265221+0000", 2021-03-15T22:32:48.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.826e-05 2021-03-15T22:32:48.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.817 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.817 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.817 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265289+0000", 2021-03-15T22:32:48.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014873299999999999 2021-03-15T22:32:48.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.818 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.819 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265438+0000", 2021-03-15T22:32:48.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0070524159999999997 2021-03-15T22:32:48.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:48.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.272490+0000", 2021-03-15T22:32:48.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0052860520000000003 2021-03-15T22:32:48.820 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.820 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:48.821 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.277776+0000", 2021-03-15T22:32:48.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0080593580000000008 2021-03-15T22:32:48.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.822 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.822 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.285835+0000", 2021-03-15T22:32:48.822 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6378999999999998e-05 2021-03-15T22:32:48.822 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.823 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.823 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.285872+0000", 2021-03-15T22:32:48.823 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3061e-05 2021-03-15T22:32:48.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.824 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.824 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:48.824 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.285915+0000", 2021-03-15T22:32:48.824 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2069999999999994e-05 2021-03-15T22:32:48.824 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.825 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.825 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.285977+0000", 2021-03-15T22:32:48.825 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020808416999999999 2021-03-15T22:32:48.825 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.826 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.826 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.826 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.827 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6001 7.0 7:00c03349:::smithi07694878-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e647)", 2021-03-15T22:32:48.827 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:56.265234+0000", 2021-03-15T22:32:48.827 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 172.44837634000001, 2021-03-15T22:32:48.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021017456, 2021-03-15T22:32:48.827 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.828 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:48.828 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.828 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.828 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.828 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6001 2021-03-15T22:32:48.829 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.829 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.829 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.829 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.829 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265234+0000", 2021-03-15T22:32:48.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.830 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.830 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.830 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265234+0000", 2021-03-15T22:32:48.831 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2799999999999995e-07 2021-03-15T22:32:48.831 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.831 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.831 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.831 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265235+0000", 2021-03-15T22:32:48.832 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.062e-06 2021-03-15T22:32:48.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.833 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265236+0000", 2021-03-15T22:32:48.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5990000000000001e-06 2021-03-15T22:32:48.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.834 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265242+0000", 2021-03-15T22:32:48.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5379999999999998e-06 2021-03-15T22:32:48.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.834 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.834 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.835 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265245+0000", 2021-03-15T22:32:48.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000367549 2021-03-15T22:32:48.835 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.835 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.835 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.836 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265613+0000", 2021-03-15T22:32:48.836 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2518e-05 2021-03-15T22:32:48.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.836 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:32:48.837 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.265675+0000", 2021-03-15T22:32:48.837 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.011675506 2021-03-15T22:32:48.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.277351+0000", 2021-03-15T22:32:48.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000111115 2021-03-15T22:32:48.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:48.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.277462+0000", 2021-03-15T22:32:48.839 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0085610970000000002 2021-03-15T22:32:48.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.840 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.840 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.286023+0000", 2021-03-15T22:32:48.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6027999999999996e-05 2021-03-15T22:32:48.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.840 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.841 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.841 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.286119+0000", 2021-03-15T22:32:48.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013261600000000001 2021-03-15T22:32:48.841 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.842 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.842 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:56.286252+0000", 2021-03-15T22:32:48.842 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021017456 2021-03-15T22:32:48.842 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.843 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.843 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.843 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6670 7.0 7:00c03349:::smithi07694878-40:head [copy-from ver 1517] snapc 245=[245,243,241,23e,23d,23c,239,235,234,22e,22d,22b,227,225,223,21f,218,215,211,20c,200,1f7,1e9,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e717)", 2021-03-15T22:32:48.844 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:31:06.237882+0000", 2021-03-15T22:32:48.844 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 102.475728359, 2021-03-15T22:32:48.844 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020450328, 2021-03-15T22:32:48.844 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.844 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:48.845 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.845 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.845 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.845 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6670 2021-03-15T22:32:48.845 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.846 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.846 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237882+0000", 2021-03-15T22:32:48.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.848 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237882+0000", 2021-03-15T22:32:48.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5140000000000003e-06 2021-03-15T22:32:48.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237887+0000", 2021-03-15T22:32:48.849 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.371e-06 2021-03-15T22:32:48.849 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.849 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.849 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.850 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237890+0000", 2021-03-15T22:32:48.850 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7682000000000001e-05 2021-03-15T22:32:48.850 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.850 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.851 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.851 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237908+0000", 2021-03-15T22:32:48.851 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.002e-06 2021-03-15T22:32:48.851 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.851 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.852 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.852 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237915+0000", 2021-03-15T22:32:48.852 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1247000000000005e-05 2021-03-15T22:32:48.852 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.852 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.853 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.853 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237976+0000", 2021-03-15T22:32:48.853 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.081e-06 2021-03-15T22:32:48.853 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.853 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.854 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:32:48.854 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237985+0000", 2021-03-15T22:32:48.854 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001105314 2021-03-15T22:32:48.854 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.854 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.855 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.855 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.239090+0000", 2021-03-15T22:32:48.855 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000120964 2021-03-15T22:32:48.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.856 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.856 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.856 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.239211+0000", 2021-03-15T22:32:48.856 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0055616399999999996 2021-03-15T22:32:48.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.857 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:48.857 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.244773+0000", 2021-03-15T22:32:48.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045633130000000003 2021-03-15T22:32:48.857 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.858 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.858 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:48.858 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.249336+0000", 2021-03-15T22:32:48.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0088461189999999995 2021-03-15T22:32:48.858 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.859 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.859 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.859 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258182+0000", 2021-03-15T22:32:48.859 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7354000000000002e-05 2021-03-15T22:32:48.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.860 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.860 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.860 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258220+0000", 2021-03-15T22:32:48.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5302000000000001e-05 2021-03-15T22:32:48.860 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.861 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:48.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258265+0000", 2021-03-15T22:32:48.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7424999999999999e-05 2021-03-15T22:32:48.862 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.862 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258333+0000", 2021-03-15T22:32:48.862 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020450328 2021-03-15T22:32:48.863 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.863 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.863 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.863 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.863 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.864 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:6671 7.0 7:00c03349:::smithi07694878-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e717)", 2021-03-15T22:32:48.864 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:31:06.237929+0000", 2021-03-15T22:32:48.864 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 102.47568164499999, 2021-03-15T22:32:48.864 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020548578000000001, 2021-03-15T22:32:48.864 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.865 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:48.865 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.865 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.865 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.865 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 6671 2021-03-15T22:32:48.866 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.866 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.866 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.867 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237929+0000", 2021-03-15T22:32:48.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.867 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.867 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.868 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237929+0000", 2021-03-15T22:32:48.868 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.04e-06 2021-03-15T22:32:48.868 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.868 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.868 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.869 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237930+0000", 2021-03-15T22:32:48.869 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.491e-06 2021-03-15T22:32:48.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.869 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.870 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.870 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237931+0000", 2021-03-15T22:32:48.871 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1251e-05 2021-03-15T22:32:48.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.871 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.871 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237953+0000", 2021-03-15T22:32:48.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7879999999999997e-06 2021-03-15T22:32:48.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.872 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.872 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.873 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.237958+0000", 2021-03-15T22:32:48.873 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016104300000000001 2021-03-15T22:32:48.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.874 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.874 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.238119+0000", 2021-03-15T22:32:48.874 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0128e-05 2021-03-15T22:32:48.874 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.875 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.875 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:32:48.875 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.238129+0000", 2021-03-15T22:32:48.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012166189999999999 2021-03-15T22:32:48.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.876 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.876 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.876 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.239345+0000", 2021-03-15T22:32:48.876 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3352e-05 2021-03-15T22:32:48.876 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.877 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:32:48.877 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.239399+0000", 2021-03-15T22:32:48.877 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0091340589999999999 2021-03-15T22:32:48.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.878 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.878 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.878 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.248533+0000", 2021-03-15T22:32:48.878 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5486e-05 2021-03-15T22:32:48.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.879 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.879 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:48.879 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.248628+0000", 2021-03-15T22:32:48.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0097252959999999996 2021-03-15T22:32:48.880 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.880 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.880 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258354+0000", 2021-03-15T22:32:48.880 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4283999999999997e-05 2021-03-15T22:32:48.881 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.881 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258408+0000", 2021-03-15T22:32:48.881 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9740999999999995e-05 2021-03-15T22:32:48.882 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.882 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:31:06.258478+0000", 2021-03-15T22:32:48.882 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.020548578000000001 2021-03-15T22:32:48.883 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.883 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.883 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.884 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.884 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7256 7.0 7:071f7a04:::smithi07694878-37:head [write 672544~572898] snapc 27b=[27b,276,275,271,26f,26e,26c,268,267,266,265,261,25d,25b,257,255,24f,24c,24b,24a,248,247,243,23e,23d,239,234,22d,218,1f7,1c7] ondisk+write+known_if_redirected e772)", 2021-03-15T22:32:48.884 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:32:02.260592+0000", 2021-03-15T22:32:48.884 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 46.453018497999999, 2021-03-15T22:32:48.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.083414811000000005, 2021-03-15T22:32:48.885 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.885 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:48.885 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.885 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.885 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.886 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7256 2021-03-15T22:32:48.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.886 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.886 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.886 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.887 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.260592+0000", 2021-03-15T22:32:48.887 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.887 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.888 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.260592+0000", 2021-03-15T22:32:48.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00077320299999999998 2021-03-15T22:32:48.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.888 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.889 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.889 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.261365+0000", 2021-03-15T22:32:48.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3009999999999997e-06 2021-03-15T22:32:48.889 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.889 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.890 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.890 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.261371+0000", 2021-03-15T22:32:48.890 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2044000000000001e-05 2021-03-15T22:32:48.890 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.890 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.891 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.891 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.261393+0000", 2021-03-15T22:32:48.891 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5684999999999999e-05 2021-03-15T22:32:48.891 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.892 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.892 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.261418+0000", 2021-03-15T22:32:48.892 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.3655e-05 2021-03-15T22:32:48.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.893 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.893 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.893 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.261472+0000", 2021-03-15T22:32:48.893 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3409999999999999e-06 2021-03-15T22:32:48.893 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.894 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.894 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:32:48.894 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.261478+0000", 2021-03-15T22:32:48.894 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015279639999999999 2021-03-15T22:32:48.894 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.895 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.895 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.895 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.263006+0000", 2021-03-15T22:32:48.895 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000206937 2021-03-15T22:32:48.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.263213+0000", 2021-03-15T22:32:48.896 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000277333 2021-03-15T22:32:48.897 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.897 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:48.897 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.263491+0000", 2021-03-15T22:32:48.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0060743560000000004 2021-03-15T22:32:48.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.898 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.898 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:48.898 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.269565+0000", 2021-03-15T22:32:48.898 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.071734908 2021-03-15T22:32:48.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.899 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.899 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.899 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.341300+0000", 2021-03-15T22:32:48.899 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024423840000000001 2021-03-15T22:32:48.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.900 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.900 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.900 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.343742+0000", 2021-03-15T22:32:48.900 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000188023 2021-03-15T22:32:48.901 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.901 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.901 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:48.901 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.343930+0000", 2021-03-15T22:32:48.901 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6676999999999998e-05 2021-03-15T22:32:48.902 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.902 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.902 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.902 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.344007+0000", 2021-03-15T22:32:48.903 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.083414811000000005 2021-03-15T22:32:48.903 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.903 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.903 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.903 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.904 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.904 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7257 7.0 7:071f7a04:::smithi07694878-37:head [write 1666255~532998] snapc 27b=[27b,276,275,271,26f,26e,26c,268,267,266,265,261,25d,25b,257,255,24f,24c,24b,24a,248,247,243,23e,23d,239,234,22d,218,1f7,1c7] ondisk+write+known_if_redirected e772)", 2021-03-15T22:32:48.904 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:32:02.265849+0000", 2021-03-15T22:32:48.904 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 46.44776169, 2021-03-15T22:32:48.904 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.078507039000000001, 2021-03-15T22:32:48.905 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.905 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:48.905 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.905 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.905 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.906 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7257 2021-03-15T22:32:48.906 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.906 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.906 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.906 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.907 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.265849+0000", 2021-03-15T22:32:48.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.907 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.907 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.907 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.908 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.265849+0000", 2021-03-15T22:32:48.908 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00056531599999999995 2021-03-15T22:32:48.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.908 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266414+0000", 2021-03-15T22:32:48.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3699999999999999e-06 2021-03-15T22:32:48.909 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.910 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.910 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266418+0000", 2021-03-15T22:32:48.910 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4975e-05 2021-03-15T22:32:48.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.910 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.911 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.911 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266433+0000", 2021-03-15T22:32:48.911 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3069999999999999e-06 2021-03-15T22:32:48.911 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.911 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.912 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.912 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266439+0000", 2021-03-15T22:32:48.912 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00088913299999999996 2021-03-15T22:32:48.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.912 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.913 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.913 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.267328+0000", 2021-03-15T22:32:48.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011092299999999999 2021-03-15T22:32:48.913 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.914 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.914 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.267439+0000", 2021-03-15T22:32:48.914 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014366800000000001 2021-03-15T22:32:48.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.915 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:48.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.267583+0000", 2021-03-15T22:32:48.915 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002128427 2021-03-15T22:32:48.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.916 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:48.916 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.269711+0000", 2021-03-15T22:32:48.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.073928355000000001 2021-03-15T22:32:48.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.917 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.917 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.343639+0000", 2021-03-15T22:32:48.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00062542900000000005 2021-03-15T22:32:48.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.918 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.918 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.344265+0000", 2021-03-15T22:32:48.918 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3739999999999998e-05 2021-03-15T22:32:48.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.919 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.919 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:48.919 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.344309+0000", 2021-03-15T22:32:48.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7395999999999998e-05 2021-03-15T22:32:48.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.920 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.920 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.344356+0000", 2021-03-15T22:32:48.920 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.078507039000000001 2021-03-15T22:32:48.920 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.921 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.921 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.921 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.921 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.922 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7258 7.0 7:071f7a04:::smithi07694878-37:head [setxattr _header (58) in=7b,truncate 2672544] snapc 27b=[27b,276,275,271,26f,26e,26c,268,267,266,265,261,25d,25b,257,255,24f,24c,24b,24a,248,247,243,23e,23d,239,234,22d,218,1f7,1c7] ondisk+write+known_if_redirected e772)", 2021-03-15T22:32:48.922 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:32:02.266448+0000", 2021-03-15T22:32:48.922 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 46.447162142000003, 2021-03-15T22:32:48.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.078113505999999999, 2021-03-15T22:32:48.923 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.923 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:48.923 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.923 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.923 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.924 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7258 2021-03-15T22:32:48.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.924 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.925 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.925 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266448+0000", 2021-03-15T22:32:48.925 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.925 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.925 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.926 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.926 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266448+0000", 2021-03-15T22:32:48.926 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2389999999999999e-06 2021-03-15T22:32:48.926 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.926 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.927 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266450+0000", 2021-03-15T22:32:48.927 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1119999999999999e-06 2021-03-15T22:32:48.927 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.928 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266451+0000", 2021-03-15T22:32:48.928 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0084999999999999e-05 2021-03-15T22:32:48.928 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.929 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.929 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266461+0000", 2021-03-15T22:32:48.929 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.844e-06 2021-03-15T22:32:48.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.930 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.930 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.930 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266465+0000", 2021-03-15T22:32:48.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023146859999999998 2021-03-15T22:32:48.930 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.931 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.931 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.268779+0000", 2021-03-15T22:32:48.931 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9202999999999994e-05 2021-03-15T22:32:48.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.932 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.932 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.932 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.268859+0000", 2021-03-15T22:32:48.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013743300000000001 2021-03-15T22:32:48.932 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.933 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.933 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:48.933 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.268996+0000", 2021-03-15T22:32:48.933 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00071907500000000001 2021-03-15T22:32:48.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.934 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.934 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:48.934 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.269715+0000", 2021-03-15T22:32:48.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.074420421 2021-03-15T22:32:48.935 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.344136+0000", 2021-03-15T22:32:48.935 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000315805 2021-03-15T22:32:48.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.936 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.936 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.936 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.344451+0000", 2021-03-15T22:32:48.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6499000000000003e-05 2021-03-15T22:32:48.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.937 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.937 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:48.937 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.344498+0000", 2021-03-15T22:32:48.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4103999999999995e-05 2021-03-15T22:32:48.938 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.938 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.938 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.938 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.344562+0000", 2021-03-15T22:32:48.938 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.078113505999999999 2021-03-15T22:32:48.939 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.939 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.939 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.939 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.940 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7259 7.0 7:071f7a04:::smithi07694878-37:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e772)", 2021-03-15T22:32:48.940 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:32:02.266468+0000", 2021-03-15T22:32:48.940 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 46.447142133, 2021-03-15T22:32:48.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.078377276999999995, 2021-03-15T22:32:48.941 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.941 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:48.941 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.941 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.941 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.942 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7259 2021-03-15T22:32:48.942 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.942 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.942 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.942 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.943 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266468+0000", 2021-03-15T22:32:48.943 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.943 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.944 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266468+0000", 2021-03-15T22:32:48.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4199999999999995e-07 2021-03-15T22:32:48.944 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.944 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.944 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266469+0000", 2021-03-15T22:32:48.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.78e-07 2021-03-15T22:32:48.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.945 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266470+0000", 2021-03-15T22:32:48.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3989999999999997e-06 2021-03-15T22:32:48.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266475+0000", 2021-03-15T22:32:48.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9210000000000001e-06 2021-03-15T22:32:48.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.948 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.948 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.266478+0000", 2021-03-15T22:32:48.948 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031171800000000002 2021-03-15T22:32:48.948 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.949 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.949 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.269595+0000", 2021-03-15T22:32:48.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1706000000000006e-05 2021-03-15T22:32:48.949 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.949 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:48.950 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.269686+0000", 2021-03-15T22:32:48.950 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.074924679999999994 2021-03-15T22:32:48.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.950 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.951 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.951 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.344611+0000", 2021-03-15T22:32:48.951 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010597799999999999 2021-03-15T22:32:48.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.951 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.952 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.344717+0000", 2021-03-15T22:32:48.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012869300000000001 2021-03-15T22:32:48.952 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.953 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.953 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:02.344846+0000", 2021-03-15T22:32:48.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.078377276999999995 2021-03-15T22:32:48.953 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.954 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.954 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.954 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7412 7.0 7:0152dc37:::smithi07694878-28:head [write 625491~502186] snapc 28d=[28d,285,283,27f,27e,27d,276,275,26f,26e,26c,268,267,266,265,261,25d,24f,24c,24b,247,243,23e,23d,239,234,22d,218,1f7] ondisk+write+known_if_redirected e790)", 2021-03-15T22:32:48.955 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:32:20.269346+0000", 2021-03-15T22:32:48.955 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 28.444264753999999, 2021-03-15T22:32:48.955 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.040611003, 2021-03-15T22:32:48.955 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.955 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:48.956 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.956 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.956 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.956 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7412 2021-03-15T22:32:48.957 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.957 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.957 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.957 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.957 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.269346+0000", 2021-03-15T22:32:48.958 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.958 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.958 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.269346+0000", 2021-03-15T22:32:48.959 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076832599999999995 2021-03-15T22:32:48.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.959 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.270114+0000", 2021-03-15T22:32:48.960 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7219999999999999e-06 2021-03-15T22:32:48.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.270119+0000", 2021-03-15T22:32:48.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4306000000000001e-05 2021-03-15T22:32:48.961 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.961 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.270143+0000", 2021-03-15T22:32:48.962 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7820000000000005e-06 2021-03-15T22:32:48.962 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.962 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.270153+0000", 2021-03-15T22:32:48.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3543999999999999e-05 2021-03-15T22:32:48.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.963 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.270227+0000", 2021-03-15T22:32:48.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0166999999999999e-05 2021-03-15T22:32:48.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.964 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:32:48.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.270237+0000", 2021-03-15T22:32:48.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028624140000000002 2021-03-15T22:32:48.965 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.966 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.273099+0000", 2021-03-15T22:32:48.966 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000133926 2021-03-15T22:32:48.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.967 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.273233+0000", 2021-03-15T22:32:48.967 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027083100000000002 2021-03-15T22:32:48.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.967 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.968 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:48.968 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.273504+0000", 2021-03-15T22:32:48.968 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.004093668 2021-03-15T22:32:48.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.968 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.969 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:48.969 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.277598+0000", 2021-03-15T22:32:48.969 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019921646000000001 2021-03-15T22:32:48.969 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.969 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.970 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.970 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.297519+0000", 2021-03-15T22:32:48.970 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012270191999999999 2021-03-15T22:32:48.970 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.970 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.971 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.309789+0000", 2021-03-15T22:32:48.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4802000000000006e-05 2021-03-15T22:32:48.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.971 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.972 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:48.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.309874+0000", 2021-03-15T22:32:48.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2676999999999995e-05 2021-03-15T22:32:48.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.973 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.309957+0000", 2021-03-15T22:32:48.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.040611003 2021-03-15T22:32:48.973 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.974 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.974 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.974 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7413 7.0 7:0152dc37:::smithi07694878-28:head [write 1861671~713346] snapc 28d=[28d,285,283,27f,27e,27d,276,275,26f,26e,26c,268,267,266,265,261,25d,24f,24c,24b,247,243,23e,23d,239,234,22d,218,1f7] ondisk+write+known_if_redirected e790)", 2021-03-15T22:32:48.975 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:32:20.279193+0000", 2021-03-15T22:32:48.975 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 28.434417332999999, 2021-03-15T22:32:48.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035056142999999998, 2021-03-15T22:32:48.975 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.975 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:48.976 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.976 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.976 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.976 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7413 2021-03-15T22:32:48.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.977 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.977 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.977 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.279193+0000", 2021-03-15T22:32:48.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.978 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.978 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.978 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.279193+0000", 2021-03-15T22:32:48.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00073461499999999999 2021-03-15T22:32:48.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.279928+0000", 2021-03-15T22:32:48.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5609999999999999e-06 2021-03-15T22:32:48.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.279931+0000", 2021-03-15T22:32:48.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6402999999999999e-05 2021-03-15T22:32:48.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.279948+0000", 2021-03-15T22:32:48.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0389999999999999e-06 2021-03-15T22:32:48.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:48.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.279955+0000", 2021-03-15T22:32:48.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9122999999999994e-05 2021-03-15T22:32:48.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.983 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:48.983 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.280034+0000", 2021-03-15T22:32:48.984 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017521400000000001 2021-03-15T22:32:48.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:48.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.280209+0000", 2021-03-15T22:32:48.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000274679 2021-03-15T22:32:48.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.985 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:48.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.280484+0000", 2021-03-15T22:32:48.986 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0024575109999999999 2021-03-15T22:32:48.986 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.986 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.986 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:48.987 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.282941+0000", 2021-03-15T22:32:48.987 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.022009035 2021-03-15T22:32:48.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.987 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.988 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.304950+0000", 2021-03-15T22:32:48.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0092190310000000008 2021-03-15T22:32:48.988 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.988 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.988 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:48.989 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.314169+0000", 2021-03-15T22:32:48.989 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5163000000000001e-05 2021-03-15T22:32:48.989 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.989 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.989 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:48.990 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.314215+0000", 2021-03-15T22:32:48.990 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4768999999999999e-05 2021-03-15T22:32:48.990 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.990 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.990 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:48.991 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.314249+0000", 2021-03-15T22:32:48.991 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.035056142999999998 2021-03-15T22:32:48.991 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.991 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:48.991 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:48.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.992 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.992 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7414 7.0 7:0152dc37:::smithi07694878-28:head [setxattr _header (58) in=7b,truncate 3025491] snapc 28d=[28d,285,283,27f,27e,27d,276,275,26f,26e,26c,268,267,266,265,261,25d,24f,24c,24b,247,243,23e,23d,239,234,22d,218,1f7] ondisk+write+known_if_redirected e790)", 2021-03-15T22:32:48.992 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:32:20.285856+0000", 2021-03-15T22:32:48.992 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 28.427755025, 2021-03-15T22:32:48.993 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028522190999999999, 2021-03-15T22:32:48.993 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:48.993 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:48.993 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:48.994 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:48.994 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:48.994 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7414 2021-03-15T22:32:48.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.994 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:48.995 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:48.995 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285856+0000", 2021-03-15T22:32:48.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:48.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:48.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285856+0000", 2021-03-15T22:32:48.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5650000000000003e-06 2021-03-15T22:32:48.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.997 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:48.997 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285860+0000", 2021-03-15T22:32:48.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5300000000000001e-06 2021-03-15T22:32:48.997 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.998 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.998 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:48.998 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285864+0000", 2021-03-15T22:32:48.998 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.908e-05 2021-03-15T22:32:48.998 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:48.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:48.999 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:48.999 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285883+0000", 2021-03-15T22:32:48.999 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4379999999999998e-06 2021-03-15T22:32:49.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:49.000 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285890+0000", 2021-03-15T22:32:49.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4698999999999997e-05 2021-03-15T22:32:49.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.001 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:49.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285965+0000", 2021-03-15T22:32:49.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00015629399999999999 2021-03-15T22:32:49.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.002 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:49.002 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.286121+0000", 2021-03-15T22:32:49.002 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00027146000000000001 2021-03-15T22:32:49.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:32:49.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.286393+0000", 2021-03-15T22:32:49.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0015424 2021-03-15T22:32:49.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.004 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:32:49.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.287935+0000", 2021-03-15T22:32:49.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017047749000000001 2021-03-15T22:32:49.005 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:49.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.304983+0000", 2021-03-15T22:32:49.006 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0093113689999999999 2021-03-15T22:32:49.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.006 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:32:49.006 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.314294+0000", 2021-03-15T22:32:49.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2006999999999999e-05 2021-03-15T22:32:49.007 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.007 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.007 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:49.007 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.314336+0000", 2021-03-15T22:32:49.008 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1600000000000002e-05 2021-03-15T22:32:49.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.008 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.008 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:49.008 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.314378+0000", 2021-03-15T22:32:49.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028522190999999999 2021-03-15T22:32:49.009 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:49.009 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:49.009 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:49.009 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.010 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.010 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7415 7.0 7:0152dc37:::smithi07694878-28:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e790)", 2021-03-15T22:32:49.010 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:32:20.285905+0000", 2021-03-15T22:32:49.010 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 28.427705364000001, 2021-03-15T22:32:49.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028661886000000001, 2021-03-15T22:32:49.011 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:49.011 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:49.011 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:49.011 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:49.011 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:49.012 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7415 2021-03-15T22:32:49.012 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.012 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:49.012 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.013 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:49.013 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285905+0000", 2021-03-15T22:32:49.013 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:49.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.013 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:49.014 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285905+0000", 2021-03-15T22:32:49.014 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.328e-06 2021-03-15T22:32:49.014 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.014 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.015 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:49.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285907+0000", 2021-03-15T22:32:49.015 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2060000000000004e-06 2021-03-15T22:32:49.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:49.016 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285913+0000", 2021-03-15T22:32:49.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2058e-05 2021-03-15T22:32:49.016 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:49.017 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285925+0000", 2021-03-15T22:32:49.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3989999999999993e-06 2021-03-15T22:32:49.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.018 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:49.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.285934+0000", 2021-03-15T22:32:49.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0018711400000000001 2021-03-15T22:32:49.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:49.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.287805+0000", 2021-03-15T22:32:49.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011013 2021-03-15T22:32:49.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:49.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.287916+0000", 2021-03-15T22:32:49.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026497525000000001 2021-03-15T22:32:49.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.021 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:49.021 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.314413+0000", 2021-03-15T22:32:49.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3033000000000004e-05 2021-03-15T22:32:49.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:49.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.314486+0000", 2021-03-15T22:32:49.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1067000000000002e-05 2021-03-15T22:32:49.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:49.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.314567+0000", 2021-03-15T22:32:49.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.028661886000000001 2021-03-15T22:32:49.023 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:49.024 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:49.024 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:49.024 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.024 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.3.0:1986 7.a e790/684 7:5e93ebe0:::smithi07694878-19:25d v 790'1363, mlcod=787'1359)", 2021-03-15T22:32:49.025 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:32:20.306211+0000", 2021-03-15T22:32:49.025 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 28.407399340000001, 2021-03-15T22:32:49.025 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029935024000000001, 2021-03-15T22:32:49.025 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:49.026 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:49.026 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:49.026 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:49.026 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.306211+0000", 2021-03-15T22:32:49.027 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:49.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.027 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:49.027 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.306211+0000", 2021-03-15T22:32:49.028 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1760000000000002e-06 2021-03-15T22:32:49.028 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.028 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:49.028 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.306214+0000", 2021-03-15T22:32:49.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.605e-06 2021-03-15T22:32:49.029 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.029 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.029 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:49.029 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.306217+0000", 2021-03-15T22:32:49.030 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.7999999999999993e-06 2021-03-15T22:32:49.030 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.030 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.030 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:49.030 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.306227+0000", 2021-03-15T22:32:49.031 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.693e-06 2021-03-15T22:32:49.031 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.031 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.031 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:49.032 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.306231+0000", 2021-03-15T22:32:49.032 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6121999999999997e-05 2021-03-15T22:32:49.032 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.032 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:49.033 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.306267+0000", 2021-03-15T22:32:49.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5648e-05 2021-03-15T22:32:49.033 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.033 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.033 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:49.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.306282+0000", 2021-03-15T22:32:49.034 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029794754 2021-03-15T22:32:49.034 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.034 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.034 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:49.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.336077+0000", 2021-03-15T22:32:49.035 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9226000000000004e-05 2021-03-15T22:32:49.035 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:49.036 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.336146+0000", 2021-03-15T22:32:49.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.029935024000000001 2021-03-15T22:32:49.036 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:49.036 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:49.036 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:49.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.037 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(osd.3.0:1987 7.a e790/684 7:5595c26f:::smithi07694878-7:24f v 790'1365, mlcod=787'1359)", 2021-03-15T22:32:49.037 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:32:20.307809+0000", 2021-03-15T22:32:49.037 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 28.405801635, 2021-03-15T22:32:49.038 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046112392000000002, 2021-03-15T22:32:49.038 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:49.038 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:32:49.038 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:49.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:49.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.307809+0000", 2021-03-15T22:32:49.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:49.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.040 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:49.040 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.307809+0000", 2021-03-15T22:32:49.040 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.2669999999999998e-06 2021-03-15T22:32:49.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:49.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.307812+0000", 2021-03-15T22:32:49.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.548e-06 2021-03-15T22:32:49.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.042 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:49.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.307815+0000", 2021-03-15T22:32:49.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2211999999999999e-05 2021-03-15T22:32:49.043 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.043 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.043 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:49.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.307827+0000", 2021-03-15T22:32:49.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9780000000000002e-06 2021-03-15T22:32:49.044 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.044 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.044 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:49.044 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.307833+0000", 2021-03-15T22:32:49.044 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5132000000000001e-05 2021-03-15T22:32:49.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.045 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:49.045 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.307878+0000", 2021-03-15T22:32:49.045 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.8438000000000001e-05 2021-03-15T22:32:49.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.046 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.046 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:49.046 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.307907+0000", 2021-03-15T22:32:49.046 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.045968780000000001 2021-03-15T22:32:49.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:32:49.047 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.353875+0000", 2021-03-15T22:32:49.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6037e-05 2021-03-15T22:32:49.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:49.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:20.353921+0000", 2021-03-15T22:32:49.049 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046112392000000002 2021-03-15T22:32:49.049 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:49.049 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:49.049 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:49.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.050 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7722 7.0 7:08414d91:::smithi07694878-42:head [read 0~3933003,checksum] snapc 0=[] ondisk+read+known_if_redirected e818)", 2021-03-15T22:32:49.050 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:32:48.460077+0000", 2021-03-15T22:32:49.050 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.253533593, 2021-03-15T22:32:49.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018734971999999999, 2021-03-15T22:32:49.051 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:32:49.051 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:32:49.051 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:32:49.051 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:32:49.051 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:32:49.052 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7722 2021-03-15T22:32:49.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.052 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:32:49.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.052 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:32:49.053 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:48.460077+0000", 2021-03-15T22:32:49.053 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:32:49.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:32:49.054 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:48.460077+0000", 2021-03-15T22:32:49.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8500000000000001e-06 2021-03-15T22:32:49.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.055 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:32:49.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:48.460079+0000", 2021-03-15T22:32:49.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6050000000000001e-06 2021-03-15T22:32:49.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:32:49.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:48.460080+0000", 2021-03-15T22:32:49.056 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7409999999999992e-06 2021-03-15T22:32:49.056 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.057 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:32:49.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:48.460088+0000", 2021-03-15T22:32:49.057 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9200000000000003e-06 2021-03-15T22:32:49.057 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.058 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:32:49.058 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:48.460093+0000", 2021-03-15T22:32:49.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000245727 2021-03-15T22:32:49.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:49.059 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:48.460339+0000", 2021-03-15T22:32:49.059 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1848999999999997e-05 2021-03-15T22:32:49.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:32:49.060 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:48.460401+0000", 2021-03-15T22:32:49.060 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017821182000000001 2021-03-15T22:32:49.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.060 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.061 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:32:49.061 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:48.478222+0000", 2021-03-15T22:32:49.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010379599999999999 2021-03-15T22:32:49.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.062 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:32:49.062 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:48.478326+0000", 2021-03-15T22:32:49.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00048630199999999998 2021-03-15T22:32:49.062 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:32:49.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:32:49.063 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:32:49.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:32:48.478812+0000", 2021-03-15T22:32:49.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.018734971999999999 2021-03-15T22:32:49.063 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:49.064 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:49.064 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:49.064 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:32:49.064 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:32:49.064 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:32:49.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:32:49.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:49.067 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:49.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:49.068 INFO:tasks.rados.rados.0.smithi076.stdout:3052: finishing write tid 1 to smithi07694878-35 2021-03-15T22:32:49.068 INFO:tasks.rados.rados.0.smithi076.stdout:3052: finishing write tid 2 to smithi07694878-35 2021-03-15T22:32:49.069 INFO:tasks.rados.rados.0.smithi076.stdout:3052: finishing write tid 3 to smithi07694878-35 2021-03-15T22:32:49.069 INFO:tasks.rados.rados.0.smithi076.stdout:3052: finishing write tid 4 to smithi07694878-35 2021-03-15T22:32:49.069 INFO:tasks.rados.rados.0.smithi076.stdout:3052: finishing write tid 5 to smithi07694878-35 2021-03-15T22:32:49.069 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 2110 (ObjNum 1050 snap 353 seq_num 1050) dirty exists 2021-03-15T22:32:49.069 INFO:tasks.rados.rados.0.smithi076.stdout:3052: left oid 35 (ObjNum 1050 snap 353 seq_num 1050) 2021-03-15T22:32:49.070 INFO:tasks.rados.rados.0.smithi076.stdout:3054: expect (ObjNum 1014 snap 341 seq_num 1014) 2021-03-15T22:32:49.070 INFO:tasks.rados.rados.0.smithi076.stdout:3059: finishing copy_from to smithi07694878-1 2021-03-15T22:32:49.070 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 2395 (ObjNum 982 snap 326 seq_num 982) dirty exists 2021-03-15T22:32:49.070 INFO:tasks.rados.rados.0.smithi076.stdout:3059: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:32:49.070 INFO:tasks.rados.rados.0.smithi076.stdout:3061: finishing write tid 1 to smithi07694878-16 2021-03-15T22:32:49.071 INFO:tasks.rados.rados.0.smithi076.stdout:3060: finishing copy_from to smithi07694878-17 2021-03-15T22:32:49.071 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 2616 (ObjNum 1039 snap 347 seq_num 1039) dirty exists 2021-03-15T22:32:49.071 INFO:tasks.rados.rados.0.smithi076.stdout:3060: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:32:49.071 INFO:tasks.rados.rados.0.smithi076.stdout:3061: finishing write tid 2 to smithi07694878-16 2021-03-15T22:32:49.071 INFO:tasks.rados.rados.0.smithi076.stdout:3061: finishing write tid 3 to smithi07694878-16 2021-03-15T22:32:49.072 INFO:tasks.rados.rados.0.smithi076.stdout:3061: finishing write tid 4 to smithi07694878-16 2021-03-15T22:32:49.072 INFO:tasks.rados.rados.0.smithi076.stdout:3061: finishing write tid 5 to smithi07694878-16 2021-03-15T22:32:49.072 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 2486 (ObjNum 1051 snap 353 seq_num 1051) dirty exists 2021-03-15T22:32:49.072 INFO:tasks.rados.rados.0.smithi076.stdout:3061: left oid 16 (ObjNum 1051 snap 353 seq_num 1051) 2021-03-15T22:32:49.073 INFO:tasks.rados.rados.0.smithi076.stdout:3064: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:32:49.073 INFO:tasks.rados.rados.0.smithi076.stdout:3064: finishing rollback tid 2 to smithi07694878-3 2021-03-15T22:32:49.073 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 1788 (ObjNum 906 snap 304 seq_num 906) dirty exists 2021-03-15T22:32:49.073 INFO:tasks.rados.rados.0.smithi076.stdout:3063: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:32:49.074 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:49.074 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:49.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:49.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:49.194 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:49.195 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:49.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:49.301 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-15T22:32:49.318 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:49.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:49.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:49.421 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:49.445 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:49.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:49.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:49.462 INFO:tasks.rados.rados.0.smithi076.stdout:3052: done (11 left) 2021-03-15T22:32:49.462 INFO:tasks.rados.rados.0.smithi076.stdout:3053: done (10 left) 2021-03-15T22:32:49.462 INFO:tasks.rados.rados.0.smithi076.stdout:3054: done (9 left) 2021-03-15T22:32:49.463 INFO:tasks.rados.rados.0.smithi076.stdout:3055: done (8 left) 2021-03-15T22:32:49.463 INFO:tasks.rados.rados.0.smithi076.stdout:3056: done (7 left) 2021-03-15T22:32:49.463 INFO:tasks.rados.rados.0.smithi076.stdout:3058: done (6 left) 2021-03-15T22:32:49.464 INFO:tasks.rados.rados.0.smithi076.stdout:3059: done (5 left) 2021-03-15T22:32:49.464 INFO:tasks.rados.rados.0.smithi076.stdout:3060: done (4 left) 2021-03-15T22:32:49.465 INFO:tasks.rados.rados.0.smithi076.stdout:3061: done (3 left) 2021-03-15T22:32:49.465 INFO:tasks.rados.rados.0.smithi076.stdout:3063: done (2 left) 2021-03-15T22:32:49.465 INFO:tasks.rados.rados.0.smithi076.stdout:3064: done (1 left) 2021-03-15T22:32:49.466 INFO:tasks.rados.rados.0.smithi076.stdout:3065: done (0 left) 2021-03-15T22:32:49.466 INFO:tasks.rados.rados.0.smithi076.stdout:3066: snap_create 2021-03-15T22:32:49.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:49 smithi125 conmon[56739]: cluster 2021-03-15T22:32:48.454449+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2021-03-15T22:32:49.548 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:49.573 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:49.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:49.575 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:49.678 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:49.701 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:49.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:49.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:49.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:49 smithi076 conmon[56514]: cluster 2021-03-15T22:32:48.454449+0000 mon.a ( 2021-03-15T22:32:49.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:49 smithi076 conmon[56514]: mon.0) 2112 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2021-03-15T22:32:49.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:49 smithi076 conmon[62156]: cluster 2021-03-15T22:32:48.454449+0000 mon.a (mon.0) 2112 : cluster [DBG] osdmap e818: 8 total, 8 up, 7 in 2021-03-15T22:32:49.804 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:49.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:49.828 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:49.828 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:49.930 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:49.952 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:49.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:49.954 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:50.055 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:50.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:50.087 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:50.087 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:50.189 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:50.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:50.213 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:50.213 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:50.270 INFO:tasks.rados.rados.0.smithi076.stdout:3066: done (0 left) 2021-03-15T22:32:50.271 INFO:tasks.rados.rados.0.smithi076.stdout:3067: delete oid 6 current snap is 355 2021-03-15T22:32:50.277 INFO:tasks.rados.rados.0.smithi076.stdout:3067: done (0 left) 2021-03-15T22:32:50.278 INFO:tasks.rados.rados.0.smithi076.stdout:3068: write oid 19 current snap is 355 2021-03-15T22:32:50.278 INFO:tasks.rados.rados.0.smithi076.stdout:3068: seq_num 1053 ranges {497771=493911,1619186=554020,2950529=430606} 2021-03-15T22:32:50.282 INFO:tasks.rados.rados.0.smithi076.stdout:3068: writing smithi07694878-19 from 497771 to 991682 tid 1 2021-03-15T22:32:50.290 INFO:tasks.rados.rados.0.smithi076.stdout:3068: writing smithi07694878-19 from 1619186 to 2173206 tid 2 2021-03-15T22:32:50.295 INFO:tasks.rados.rados.0.smithi076.stdout:3068: writing smithi07694878-19 from 2950529 to 3381135 tid 3 2021-03-15T22:32:50.296 INFO:tasks.rados.rados.0.smithi076.stdout:3069: write (excl) oid 26 current snap is 355 2021-03-15T22:32:50.296 INFO:tasks.rados.rados.0.smithi076.stdout:3069: seq_num 1054 ranges {514578=569160,1677668=414031,2577952=641878} 2021-03-15T22:32:50.301 INFO:tasks.rados.rados.0.smithi076.stdout:3069: writing smithi07694878-26 from 514578 to 1083738 tid 1 2021-03-15T22:32:50.307 INFO:tasks.rados.rados.0.smithi076.stdout:3069: writing smithi07694878-26 from 1677668 to 2091699 tid 2 2021-03-15T22:32:50.315 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:50.317 INFO:tasks.rados.rados.0.smithi076.stdout:3069: writing smithi07694878-26 from 2577952 to 3219830 tid 3 2021-03-15T22:32:50.317 INFO:tasks.rados.rados.0.smithi076.stdout:3070: write (excl) oid 38 current snap is 355 2021-03-15T22:32:50.317 INFO:tasks.rados.rados.0.smithi076.stdout:3068: finishing write tid 1 to smithi07694878-19 2021-03-15T22:32:50.318 INFO:tasks.rados.rados.0.smithi076.stdout:3070: seq_num 1055 ranges {531385=644409,1736150=557689,3005375=569503} 2021-03-15T22:32:50.321 INFO:tasks.rados.rados.0.smithi076.stdout:3070: writing smithi07694878-38 from 531385 to 1175794 tid 1 2021-03-15T22:32:50.328 INFO:tasks.rados.rados.0.smithi076.stdout:3070: writing smithi07694878-38 from 1736150 to 2293839 tid 2 2021-03-15T22:32:50.336 INFO:tasks.rados.rados.0.smithi076.stdout:3070: writing smithi07694878-38 from 3005375 to 3574878 tid 3 2021-03-15T22:32:50.336 INFO:tasks.rados.rados.0.smithi076.stdout:3071: copy_from oid 40 from oid 30 current snap is 355 2021-03-15T22:32:50.336 INFO:tasks.rados.rados.0.smithi076.stdout:3068: finishing write tid 2 to smithi07694878-19 2021-03-15T22:32:50.337 INFO:tasks.rados.rados.0.smithi076.stdout:3068: finishing write tid 3 to smithi07694878-19 2021-03-15T22:32:50.337 INFO:tasks.rados.rados.0.smithi076.stdout:3068: finishing write tid 4 to smithi07694878-19 2021-03-15T22:32:50.337 INFO:tasks.rados.rados.0.smithi076.stdout:3068: finishing write tid 5 to smithi07694878-19 2021-03-15T22:32:50.337 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2114 (ObjNum 1053 snap 355 seq_num 1053) dirty exists 2021-03-15T22:32:50.338 INFO:tasks.rados.rados.0.smithi076.stdout:3068: left oid 19 (ObjNum 1053 snap 355 seq_num 1053) 2021-03-15T22:32:50.338 INFO:tasks.rados.rados.0.smithi076.stdout:3069: finishing write tid 1 to smithi07694878-26 2021-03-15T22:32:50.338 INFO:tasks.rados.rados.0.smithi076.stdout:3069: finishing write tid 2 to smithi07694878-26 2021-03-15T22:32:50.338 INFO:tasks.rados.rados.0.smithi076.stdout:3069: finishing write tid 3 to smithi07694878-26 2021-03-15T22:32:50.339 INFO:tasks.rados.rados.0.smithi076.stdout:3069: finishing write tid 4 to smithi07694878-26 2021-03-15T22:32:50.339 INFO:tasks.rados.rados.0.smithi076.stdout:3069: finishing write tid 5 to smithi07694878-26 2021-03-15T22:32:50.339 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 26 v 1792 (ObjNum 1054 snap 355 seq_num 1054) dirty exists 2021-03-15T22:32:50.339 INFO:tasks.rados.rados.0.smithi076.stdout:3069: left oid 26 (ObjNum 1054 snap 355 seq_num 1054) 2021-03-15T22:32:50.340 INFO:tasks.rados.rados.0.smithi076.stdout:3070: finishing write tid 1 to smithi07694878-38 2021-03-15T22:32:50.340 INFO:tasks.rados.rados.0.smithi076.stdout:3070: finishing write tid 2 to smithi07694878-38 2021-03-15T22:32:50.340 INFO:tasks.rados.rados.0.smithi076.stdout:3068: done (3 left) 2021-03-15T22:32:50.340 INFO:tasks.rados.rados.0.smithi076.stdout:3069: done (2 left) 2021-03-15T22:32:50.340 INFO:tasks.rados.rados.0.smithi076.stdout:3072: write oid 19 current snap is 355 2021-03-15T22:32:50.341 INFO:tasks.rados.rados.0.smithi076.stdout:3072: seq_num 1056 ranges {548192=719658,1678279=417700,2516445=780775} 2021-03-15T22:32:50.343 INFO:tasks.rados.rados.0.smithi076.stdout:3072: writing smithi07694878-19 from 548192 to 1267850 tid 1 2021-03-15T22:32:50.348 INFO:tasks.rados.rados.0.smithi076.stdout:3072: writing smithi07694878-19 from 1678279 to 2095979 tid 2 2021-03-15T22:32:50.350 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:50.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:50.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:50.355 INFO:tasks.rados.rados.0.smithi076.stdout:3072: writing smithi07694878-19 from 2516445 to 3297220 tid 3 2021-03-15T22:32:50.356 INFO:tasks.rados.rados.0.smithi076.stdout:3070: finishing write tid 3 to smithi07694878-38 2021-03-15T22:32:50.356 INFO:tasks.rados.rados.0.smithi076.stdout:3070: finishing write tid 4 to smithi07694878-38 2021-03-15T22:32:50.356 INFO:tasks.rados.rados.0.smithi076.stdout:3070: finishing write tid 5 to smithi07694878-38 2021-03-15T22:32:50.356 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2118 (ObjNum 1055 snap 355 seq_num 1055) dirty exists 2021-03-15T22:32:50.356 INFO:tasks.rados.rados.0.smithi076.stdout:3070: left oid 38 (ObjNum 1055 snap 355 seq_num 1055) 2021-03-15T22:32:50.357 INFO:tasks.rados.rados.0.smithi076.stdout:3072: finishing write tid 1 to smithi07694878-19 2021-03-15T22:32:50.357 INFO:tasks.rados.rados.0.smithi076.stdout:3070: done (2 left) 2021-03-15T22:32:50.357 INFO:tasks.rados.rados.0.smithi076.stdout:3073: read oid 47 snap -1 2021-03-15T22:32:50.357 INFO:tasks.rados.rados.0.smithi076.stdout:3073: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:32:50.357 INFO:tasks.rados.rados.0.smithi076.stdout:3074: delete oid 43 current snap is 355 2021-03-15T22:32:50.361 INFO:tasks.rados.rados.0.smithi076.stdout:3074: done (3 left) 2021-03-15T22:32:50.361 INFO:tasks.rados.rados.0.smithi076.stdout:3075: copy_from oid 37 from oid 41 current snap is 355 2021-03-15T22:32:50.362 INFO:tasks.rados.rados.0.smithi076.stdout:3076: copy_from oid 23 from oid 24 current snap is 355 2021-03-15T22:32:50.363 INFO:tasks.rados.rados.0.smithi076.stdout:3073: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:32:50.452 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:50.454 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:50.459 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:32:50 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:32:50.474 INFO:tasks.rados.rados.0.smithi076.stdout:3073: done (4 left) 2021-03-15T22:32:50.474 INFO:tasks.rados.rados.0.smithi076.stdout:3077: snap_create 2021-03-15T22:32:50.474 INFO:tasks.rados.rados.0.smithi076.stdout:3071: finishing copy_from to smithi07694878-40 2021-03-15T22:32:50.475 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 2369 (ObjNum 774 snap 256 seq_num 774) dirty exists 2021-03-15T22:32:50.475 INFO:tasks.rados.rados.0.smithi076.stdout:3071: finishing copy_from racing read to smithi07694878-40 2021-03-15T22:32:50.475 INFO:tasks.rados.rados.0.smithi076.stdout:3072: finishing write tid 2 to smithi07694878-19 2021-03-15T22:32:50.475 INFO:tasks.rados.rados.0.smithi076.stdout:3072: finishing write tid 3 to smithi07694878-19 2021-03-15T22:32:50.475 INFO:tasks.rados.rados.0.smithi076.stdout:3072: finishing write tid 4 to smithi07694878-19 2021-03-15T22:32:50.476 INFO:tasks.rados.rados.0.smithi076.stdout:3072: finishing write tid 5 to smithi07694878-19 2021-03-15T22:32:50.476 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2122 (ObjNum 1056 snap 355 seq_num 1056) dirty exists 2021-03-15T22:32:50.476 INFO:tasks.rados.rados.0.smithi076.stdout:3072: left oid 19 (ObjNum 1056 snap 355 seq_num 1056) 2021-03-15T22:32:50.476 INFO:tasks.rados.rados.0.smithi076.stdout:3075: finishing copy_from to smithi07694878-37 2021-03-15T22:32:50.477 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 2395 (ObjNum 982 snap 326 seq_num 982) dirty exists 2021-03-15T22:32:50.477 INFO:tasks.rados.rados.0.smithi076.stdout:3075: finishing copy_from racing read to smithi07694878-37 2021-03-15T22:32:50.477 INFO:tasks.rados.rados.0.smithi076.stdout:3076: finishing copy_from to smithi07694878-23 2021-03-15T22:32:50.477 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 1736 (ObjNum 1046 snap 351 seq_num 1046) dirty exists 2021-03-15T22:32:50.477 INFO:tasks.rados.rados.0.smithi076.stdout:3076: finishing copy_from racing read to smithi07694878-23 2021-03-15T22:32:50.478 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:50.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:50.479 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:50.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:50 smithi076 conmon[56514]: cluster 2021-03-15T22:32:49.180930+0000 mgr.y (mgr.14144) 950 : cluster [DBG] pgmap v1372: 148 pgs: 148 active+clean; 524 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-03-15T22:32:50.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:50 smithi076 conmon[56514]: cluster 2021-03-15T22:32:49.459174+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2021-03-15T22:32:50.491 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:50 smithi076 conmon[56514]: cluster 2021-03-15T22:32:50.268268+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e820: 8 total, 8 up, 7 in 2021-03-15T22:32:50.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:50 smithi076 conmon[62156]: cluster 2021-03-15T22:32:49.180930+0000 mgr.y (mgr.14144) 950 : cluster [DBG] pgmap v1372: 148 pgs: 148 active+clean; 524 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-03-15T22:32:50.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:50 smithi076 conmon[62156]: cluster 2021-03-15T22:32:49.459174+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2021-03-15T22:32:50.492 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:50 smithi076 conmon[62156]: cluster 2021-03-15T22:32:50.268268+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e820: 8 total, 8 up, 7 in 2021-03-15T22:32:50.581 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:50.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:50.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:50.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:50.730 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:50.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:50.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:50.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:50 smithi125 conmon[56739]: cluster 2021-03-15T22:32:49.180930+0000 mgr.y (mgr.14144) 950 : cluster [DBG] pgmap v1372: 148 pgs: 148 active+clean; 524 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 8.6 MiB/s rd, 1.5 MiB/s wr, 10 op/s 2021-03-15T22:32:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:50 smithi125 conmon[56739]: cluster 2021-03-15T22:32:49.459174+0000 mon.a (mon.0) 2113 : cluster [DBG] osdmap e819: 8 total, 8 up, 7 in 2021-03-15T22:32:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:50 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:32:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:50 smithi125 conmon[56739]: :32:50.268268+0000 mon.a (mon.0) 2114 : cluster [DBG] osdmap e820: 8 total, 8 up, 7 in 2021-03-15T22:32:50.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:50 smithi125 conmon[56739]: 2021-03-15T22:32:50.833 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:50.858 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:50.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:50.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:50.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:50.984 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:50.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:50.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:51.086 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:51.109 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:51.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:51.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:51.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:51.234 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:51.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:51.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:51.240 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:51.336 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:51.359 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:51.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:51.360 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:51.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:51.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:51 smithi076 conmon[62156]: audit 2021-03-15T22:32: 2021-03-15T22:32:51.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:51 smithi076 conmon[62156]: 50.147512+0000 mgr.y (mgr.14144) 951 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:51.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:51 smithi076 conmon[62156]: audit 2021-03-15T22:32 2021-03-15T22:32:51.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:51 smithi076 conmon[62156]: :51.147647+0000 mon.a (mon. 2021-03-15T22:32:51.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:51 smithi076 conmon[62156]: 0) 2115 : audit [INF] from='client.? 172.21.15.76:0/3187456983' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "3.3"}]: dispatch 2021-03-15T22:32:51.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:51 smithi076 conmon[62156]: 2021-03-15T22:32:51.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:51 smithi076 conmon[56514]: audit 2021-03-15T22:32:50.147512 2021-03-15T22:32:51.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:51 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 951 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:51.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:51 smithi076 conmon[56514]: audit 2021-03-15T22:32:51.147647+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? 172.21.15.76:0/3187456983' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "3.3"}]: dispatch 2021-03-15T22:32:51.475 INFO:tasks.rados.rados.0.smithi076.stdout:3071: done (4 left) 2021-03-15T22:32:51.476 INFO:tasks.rados.rados.0.smithi076.stdout:3072: done (3 left) 2021-03-15T22:32:51.476 INFO:tasks.rados.rados.0.smithi076.stdout:3075: done (2 left) 2021-03-15T22:32:51.476 INFO:tasks.rados.rados.0.smithi076.stdout:3076: done (1 left) 2021-03-15T22:32:51.476 INFO:tasks.rados.rados.0.smithi076.stdout:3077: done (0 left) 2021-03-15T22:32:51.477 INFO:tasks.rados.rados.0.smithi076.stdout:3078: copy_from oid 3 from oid 28 current snap is 356 2021-03-15T22:32:51.477 INFO:tasks.rados.rados.0.smithi076.stdout:3079: write (excl) oid 18 current snap is 356 2021-03-15T22:32:51.477 INFO:tasks.rados.rados.0.smithi076.stdout:3079: seq_num 1058 ranges {581806=586509,1911596=421369,3087644=403319} 2021-03-15T22:32:51.479 INFO:teuthology.orchestra.run.smithi076.stderr:clear 3.3 pg_upmap_items mapping 2021-03-15T22:32:51.480 INFO:tasks.rados.rados.0.smithi076.stdout:3079: writing smithi07694878-18 from 581806 to 1168315 tid 1 2021-03-15T22:32:51.485 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:51.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:51.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:51.487 INFO:tasks.rados.rados.0.smithi076.stdout:3079: writing smithi07694878-18 from 1911596 to 2332965 tid 2 2021-03-15T22:32:51.494 INFO:tasks.rados.rados.0.smithi076.stdout:3079: writing smithi07694878-18 from 3087644 to 3490963 tid 3 2021-03-15T22:32:51.495 INFO:tasks.rados.rados.0.smithi076.stdout:3080: read oid 39 snap -1 2021-03-15T22:32:51.495 INFO:tasks.rados.rados.0.smithi076.stdout:3080: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:32:51.495 INFO:tasks.rados.rados.0.smithi076.stdout:3081: copy_from oid 48 from oid 33 current snap is 356 2021-03-15T22:32:51.495 INFO:tasks.rados.rados.0.smithi076.stdout:3079: finishing write tid 1 to smithi07694878-18 2021-03-15T22:32:51.496 INFO:tasks.rados.rados.0.smithi076.stdout:3078: finishing copy_from to smithi07694878-3 2021-03-15T22:32:51.496 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 2646 (ObjNum 620 snap 193 seq_num 620) dirty exists 2021-03-15T22:32:51.496 INFO:tasks.rados.rados.0.smithi076.stdout:3079: finishing write tid 2 to smithi07694878-18 2021-03-15T22:32:51.497 INFO:tasks.rados.rados.0.smithi076.stdout:3078: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:32:51.497 INFO:tasks.rados.rados.0.smithi076.stdout:3078: done (3 left) 2021-03-15T22:32:51.497 INFO:tasks.rados.rados.0.smithi076.stdout:3082: write oid 6 current snap is 356 2021-03-15T22:32:51.497 INFO:tasks.rados.rados.0.smithi076.stdout:3082: seq_num 1059 ranges {598613=661758,1970078=565027,3115067=614591} 2021-03-15T22:32:51.501 INFO:tasks.rados.rados.0.smithi076.stdout:3082: writing smithi07694878-6 from 598613 to 1260371 tid 1 2021-03-15T22:32:51.509 INFO:tasks.rados.rados.0.smithi076.stdout:3082: writing smithi07694878-6 from 1970078 to 2535105 tid 2 2021-03-15T22:32:51.515 INFO:tasks.rados.rados.0.smithi076.stdout:3082: writing smithi07694878-6 from 3115067 to 3729658 tid 3 2021-03-15T22:32:51.516 INFO:tasks.rados.rados.0.smithi076.stdout:3083: snap_remove snap 303 2021-03-15T22:32:51.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:51.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:51.612 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:51.613 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:51.714 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:51.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:51.737 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:51.738 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:51.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:51 smithi125 conmon[56739]: audit 2021-03-15T22:32:50.147512+0000 2021-03-15T22:32:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:51 smithi125 conmon[56739]: mgr.y (mgr.14144) 951 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:32:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:51 smithi125 conmon[56739]: audit 2021-03-15T22:32:51.147647+0000 mon.a (mon.0) 2115 : audit [INF] from='client.? 172.21.15.76:0/3187456983' entity='client.admin' cmd=[{"prefix": "osd rm-pg-upmap-items", "pgid": "3.3"}]: dispatch 2021-03-15T22:32:51.839 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:51.862 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:51.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:51.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:51.964 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:51.987 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:51.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:51.989 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:52.090 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:52.113 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:52.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:52.115 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:52.206 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:32:52.216 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:52.224 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:32:52.240 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:52.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:52.241 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:52.242 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:32:52.260 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:32:52.277 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:32:52.295 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:32:52.312 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:32:52.329 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:32:52.330 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:32:52.330 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:32:52.331 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:32:52.331 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:32:52.332 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:32:52.332 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:32:52.332 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:32:52.332 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:32:52.342 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:52.365 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:52.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:52.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:52.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:52.474 INFO:tasks.rados.rados.0.smithi076.stdout:3083: done (4 left) 2021-03-15T22:32:52.474 INFO:tasks.rados.rados.0.smithi076.stdout:3084: snap_create 2021-03-15T22:32:52.475 INFO:tasks.rados.rados.0.smithi076.stdout:3079: finishing write tid 3 to smithi07694878-18 2021-03-15T22:32:52.475 INFO:tasks.rados.rados.0.smithi076.stdout:3080: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:32:52.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[56514]: cluster 2021-03-15T22:32:51.181516+0000 mgr.y (mgr.14144) 952 : cluster [DBG] pgmap v1375: 148 pgs: 148 active+clean; 477 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-03-15T22:32:52.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[56514]: audit 2021-03-15T22:32:51.471356+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? 172.21.15.76:0/3187456983' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "3.3"}]': finished 2021-03-15T22:32:52.481 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[56514]: cluster 2021-03-15T22:32:51.471410+0000 mon.a (mon.0) 2117 : cluster [DBG] osdmap e821: 8 total, 8 up, 7 in 2021-03-15T22:32:52.482 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[56514]: audit 2021-03-15T22:32:52.187222+0000 mon.a (mon.0) 2118 : audit [DBG] from='client.? 172.21.15.76:0/96793452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:52.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: cluster 2021-03-15T22:32:52.511 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:52.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:52.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:52.622 INFO:tasks.rados.rados.0.smithi076.stdout:3079: finishing write tid 4 to smithi07694878-18 2021-03-15T22:32:52.622 INFO:tasks.rados.rados.0.smithi076.stdout:3079: finishing write tid 5 to smithi07694878-18 2021-03-15T22:32:52.622 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 1976 (ObjNum 1058 snap 356 seq_num 1058) dirty exists 2021-03-15T22:32:52.623 INFO:tasks.rados.rados.0.smithi076.stdout:3079: left oid 18 (ObjNum 1058 snap 356 seq_num 1058) 2021-03-15T22:32:52.623 INFO:tasks.rados.rados.0.smithi076.stdout:3081: finishing copy_from to smithi07694878-48 2021-03-15T22:32:52.623 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 2395 (ObjNum 982 snap 326 seq_num 982) dirty exists 2021-03-15T22:32:52.623 INFO:tasks.rados.rados.0.smithi076.stdout:3082: finishing write tid 1 to smithi07694878-6 2021-03-15T22:32:52.623 INFO:tasks.rados.rados.0.smithi076.stdout:3081: finishing copy_from racing read to smithi07694878-48 2021-03-15T22:32:52.624 INFO:tasks.rados.rados.0.smithi076.stdout:3082: finishing write tid 2 to smithi07694878-6 2021-03-15T22:32:52.624 INFO:tasks.rados.rados.0.smithi076.stdout:3082: finishing write tid 3 to smithi07694878-6 2021-03-15T22:32:52.624 INFO:tasks.rados.rados.0.smithi076.stdout:3082: finishing write tid 4 to smithi07694878-6 2021-03-15T22:32:52.624 INFO:tasks.rados.rados.0.smithi076.stdout:3082: finishing write tid 5 to smithi07694878-6 2021-03-15T22:32:52.625 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 2100 (ObjNum 1059 snap 356 seq_num 1059) dirty exists 2021-03-15T22:32:52.625 INFO:tasks.rados.rados.0.smithi076.stdout:3082: left oid 6 (ObjNum 1059 snap 356 seq_num 1059) 2021-03-15T22:32:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:52.638 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:52.638 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:52.739 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:52.761 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:52.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:52.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32 2021-03-15T22:32:52.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: :51.181516+0000 mgr.y (mgr.14144) 952 : cluster [DBG] 2021-03-15T22:32:52.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: pgmap v1375: 148 pgs: 148 active+clean; 477 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-03-15T22:32:52.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: audit 2021-03-15T22:32:52.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021 2021-03-15T22:32:52.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: - 2021-03-15T22:32:52.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 03 2021-03-15T22:32:52.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: - 2021-03-15T22:32:52.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 15 2021-03-15T22:32:52.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: T 2021-03-15T22:32:52.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 22 2021-03-15T22:32:52.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: : 2021-03-15T22:32:52.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 32 2021-03-15T22:32:52.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: : 2021-03-15T22:32:52.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 51 2021-03-15T22:32:52.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: . 2021-03-15T22:32:52.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 471356 2021-03-15T22:32:52.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: +0000 2021-03-15T22:32:52.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: mon.a 2021-03-15T22:32:52.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: ( 2021-03-15T22:32:52.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: mon 2021-03-15T22:32:52.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: . 2021-03-15T22:32:52.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 0 2021-03-15T22:32:52.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: ) 2021-03-15T22:32:52.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2116 2021-03-15T22:32:52.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: : 2021-03-15T22:32:52.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: audit 2021-03-15T22:32:52.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: [INF] 2021-03-15T22:32:52.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3187456983' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "3.3"}]': finished 2021-03-15T22:32:52.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: cluster 2021-03-15T22:32:52.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021 2021-03-15T22:32:52.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: - 2021-03-15T22:32:52.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 03 2021-03-15T22:32:52.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: - 2021-03-15T22:32:52.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 15 2021-03-15T22:32:52.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: T 2021-03-15T22:32:52.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 22 2021-03-15T22:32:52.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: : 2021-03-15T22:32:52.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 32 2021-03-15T22:32:52.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: : 2021-03-15T22:32:52.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 51 2021-03-15T22:32:52.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: . 2021-03-15T22:32:52.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 471410 2021-03-15T22:32:52.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: +0000 2021-03-15T22:32:52.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: mon.a 2021-03-15T22:32:52.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: ( 2021-03-15T22:32:52.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: mon 2021-03-15T22:32:52.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: . 2021-03-15T22:32:52.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 0 2021-03-15T22:32:52.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: ) 2021-03-15T22:32:52.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2117 2021-03-15T22:32:52.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: : 2021-03-15T22:32:52.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: cluster 2021-03-15T22:32:52.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: [DBG] 2021-03-15T22:32:52.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: osdmap e821: 8 total, 8 up, 7 in 2021-03-15T22:32:52.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: audit 2021-03-15T22:32:52.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021 2021-03-15T22:32:52.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: - 2021-03-15T22:32:52.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 03 2021-03-15T22:32:52.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: - 2021-03-15T22:32:52.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 15 2021-03-15T22:32:52.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: T 2021-03-15T22:32:52.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 22 2021-03-15T22:32:52.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: : 2021-03-15T22:32:52.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 32 2021-03-15T22:32:52.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: : 2021-03-15T22:32:52.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 52 2021-03-15T22:32:52.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: . 2021-03-15T22:32:52.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 187222 2021-03-15T22:32:52.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: +0000 2021-03-15T22:32:52.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: mon.a 2021-03-15T22:32:52.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: ( 2021-03-15T22:32:52.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: mon 2021-03-15T22:32:52.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: . 2021-03-15T22:32:52.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 0 2021-03-15T22:32:52.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: ) 2021-03-15T22:32:52.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2118 2021-03-15T22:32:52.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: : 2021-03-15T22:32:52.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: audit 2021-03-15T22:32:52.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: [DBG] 2021-03-15T22:32:52.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.788 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/96793452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:52.789 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:52 smithi076 conmon[62156]: 2021-03-15T22:32:52.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: cluster 2021-03-15T22:32:52.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: 2021-03-15T22:32:52.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: 2021-03-15T22:32:51. 2021-03-15T22:32:52.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: 181516+0000 mgr.y (mgr.14144) 952 : cluster [DBG] pgmap v1375: 148 pgs: 148 active+clean; 477 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 10 MiB/s rd, 2.7 MiB/s wr, 13 op/s 2021-03-15T22:32:52.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: audit 2021-03-15T22:32:52.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: 2021-03-15T22:32:51.471356+0000 mon.a (mon.0) 2116 : audit [INF] from='client.? 172.21.15.76:0/3187456983' entity='client.admin' cmd='[{"prefix": "osd rm-pg-upmap-items", "pgid": "3.3"}]': finished 2021-03-15T22:32:52.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:32:52.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: -03-15T22:32: 2021-03-15T22:32:52.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: 51.471410+0000 mon.a (mon.0 2021-03-15T22:32:52.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: ) 2117 : cluster [DBG] osdmap e821: 8 total, 8 up, 7 in 2021-03-15T22:32:52.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: audit 2021 2021-03-15T22:32:52.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: -03-15T22:32:52.187222+0000 2021-03-15T22:32:52.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: mon.a (mon.0) 2118 : audit [DBG] 2021-03-15T22:32:52.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:52 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/96793452' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:32:52.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:52.886 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:52.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:52.887 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:52.988 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:53.010 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:53.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:53.112 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:53.135 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:53.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:53.137 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:53.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:53.260 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:53.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:53.262 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:53.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:53.385 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:53.386 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:53.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:53.490 INFO:tasks.rados.rados.0.smithi076.stdout:3079: done (4 left) 2021-03-15T22:32:53.491 INFO:tasks.rados.rados.0.smithi076.stdout:3080: done (3 left) 2021-03-15T22:32:53.491 INFO:tasks.rados.rados.0.smithi076.stdout:3081: done (2 left) 2021-03-15T22:32:53.491 INFO:tasks.rados.rados.0.smithi076.stdout:3082: done (1 left) 2021-03-15T22:32:53.492 INFO:tasks.rados.rados.0.smithi076.stdout:3084: done (0 left) 2021-03-15T22:32:53.492 INFO:tasks.rados.rados.0.smithi076.stdout:3085: read oid 32 snap 326 2021-03-15T22:32:53.492 INFO:tasks.rados.rados.0.smithi076.stdout:3085: expect (ObjNum 717 snap 233 seq_num 717) 2021-03-15T22:32:53.492 INFO:tasks.rados.rados.0.smithi076.stdout:3086: read oid 27 snap -1 2021-03-15T22:32:53.493 INFO:tasks.rados.rados.0.smithi076.stdout:3086: expect deleted 2021-03-15T22:32:53.493 INFO:tasks.rados.rados.0.smithi076.stdout:3087: read oid 31 snap -1 2021-03-15T22:32:53.493 INFO:tasks.rados.rados.0.smithi076.stdout:3087: expect (ObjNum 1020 snap 341 seq_num 1020) 2021-03-15T22:32:53.494 INFO:tasks.rados.rados.0.smithi076.stdout:3088: delete oid 18 current snap is 357 2021-03-15T22:32:53.495 INFO:tasks.rados.rados.0.smithi076.stdout:3086: done (3 left) 2021-03-15T22:32:53.495 INFO:tasks.rados.rados.0.smithi076.stdout:3088: done (2 left) 2021-03-15T22:32:53.495 INFO:tasks.rados.rados.0.smithi076.stdout:3089: snap_remove snap 351 2021-03-15T22:32:53.511 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:53.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:53.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:53.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:53.636 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:53.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:53.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:53.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:53.760 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:53.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:53.762 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:53.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:53 smithi076 conmon[56514]: cluster 2021-03-15T22:32:52.470783 2021-03-15T22:32:53.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:53 smithi076 conmon[56514]: +0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e822: 8 total, 8 up, 7 in 2021-03-15T22:32:53.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:53 smithi076 conmon[62156]: cluster 2021-03-15T22:32:52.470783+0000 mon.a (mon.0) 2119 2021-03-15T22:32:53.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:53 smithi076 conmon[62156]: : cluster [DBG] osdmap e822: 8 total, 8 up, 7 in 2021-03-15T22:32:53.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:53 smithi125 conmon[56739]: cluster 2021-03-15T22:32:52.470783+0000 mon.a (mon.0) 2119 : cluster [DBG] osdmap e822: 8 total, 8 up, 7 in 2021-03-15T22:32:53.863 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:53.888 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:53.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:53.989 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:54.011 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:54.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:54.013 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:54.114 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:54.138 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:54.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:54.139 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:54.240 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:54.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:54.263 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:54.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:54.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:54.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:54.388 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:54.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:54.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:54.498 INFO:tasks.rados.rados.0.smithi076.stdout:3089: done (2 left) 2021-03-15T22:32:54.498 INFO:tasks.rados.rados.0.smithi076.stdout:3090: write oid 47 current snap is 357 2021-03-15T22:32:54.498 INFO:tasks.rados.rados.0.smithi076.stdout:3090: seq_num 1061 ranges {632227=412256,1570689=568696,2653560=753488} 2021-03-15T22:32:54.502 INFO:tasks.rados.rados.0.smithi076.stdout:3090: writing smithi07694878-47 from 632227 to 1044483 tid 1 2021-03-15T22:32:54.508 INFO:tasks.rados.rados.0.smithi076.stdout:3090: writing smithi07694878-47 from 1570689 to 2139385 tid 2 2021-03-15T22:32:54.516 INFO:tasks.rados.rados.0.smithi076.stdout:3090: writing smithi07694878-47 from 2653560 to 3407048 tid 3 2021-03-15T22:32:54.516 INFO:tasks.rados.rados.0.smithi076.stdout:3091: snap_create 2021-03-15T22:32:54.517 INFO:tasks.rados.rados.0.smithi076.stdout:3085: expect (ObjNum 717 snap 233 seq_num 717) 2021-03-15T22:32:54.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:54.525 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:54.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:54.527 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:32:54.527 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:32:54.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:54.638 INFO:tasks.rados.rados.0.smithi076.stdout:3087: expect (ObjNum 1020 snap 341 seq_num 1020) 2021-03-15T22:32:54.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:54 smithi076 conmon[56514]: cluster 2021-03-15T22:32:53.182142+0000 mgr.y (mgr.14144) 953 : cluster [DBG] pgmap v1378: 148 pgs: 1 peering, 147 active+clean; 502 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 8.0 MiB/s wr, 23 op/s 2021-03-15T22:32:54.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:54 smithi076 conmon[56514]: cluster 2021-03-15T22:32:53.482969+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e823: 8 total, 8 up, 7 in 2021-03-15T22:32:54.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:54 smithi076 conmon[62156]: cluster 2021-03-15T22:32:53.182142+0000 mgr.y (mgr.14144 2021-03-15T22:32:54.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:54 smithi076 conmon[62156]: ) 953 : cluster [DBG] pgmap v1378: 148 pgs: 1 peering, 147 active+clean; 502 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 8.0 MiB/s wr, 23 op/s 2021-03-15T22:32:54.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:54 smithi076 conmon[62156]: cluster 2021-03-15T22:32:53.482969+0000 mon.a ( 2021-03-15T22:32:54.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:54 smithi076 conmon[62156]: mon.0) 2120 : cluster [DBG] osdmap e823: 8 total, 8 up, 7 in 2021-03-15T22:32:54.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:54.652 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:54.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:54.776 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:54.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:54.777 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:54.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:54 smithi125 conmon[56739]: cluster 2021-03-15T22:32:53.182142+0000 mgr.y (mgr.14144) 953 : cluster [DBG] pgmap v1378: 148 pgs: 1 peering, 147 active+clean; 502 MiB data, 1.3 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 8.0 MiB/s wr, 23 op/s 2021-03-15T22:32:54.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:54 smithi125 conmon[56739]: 2021-03-15T22:32:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:54 smithi125 conmon[56739]: cluster 2021-03-15T22:32:53.482969+0000 mon.a (mon.0) 2120 : cluster [DBG] osdmap e823: 8 total, 8 up, 7 in 2021-03-15T22:32:54.861 INFO:tasks.rados.rados.0.smithi076.stdout:3090: finishing write tid 1 to smithi07694878-47 2021-03-15T22:32:54.862 INFO:tasks.rados.rados.0.smithi076.stdout:3090: finishing write tid 2 to smithi07694878-47 2021-03-15T22:32:54.862 INFO:tasks.rados.rados.0.smithi076.stdout:3090: finishing write tid 3 to smithi07694878-47 2021-03-15T22:32:54.862 INFO:tasks.rados.rados.0.smithi076.stdout:3090: finishing write tid 4 to smithi07694878-47 2021-03-15T22:32:54.863 INFO:tasks.rados.rados.0.smithi076.stdout:3090: finishing write tid 5 to smithi07694878-47 2021-03-15T22:32:54.863 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1787 (ObjNum 1061 snap 357 seq_num 1061) dirty exists 2021-03-15T22:32:54.863 INFO:tasks.rados.rados.0.smithi076.stdout:3090: left oid 47 (ObjNum 1061 snap 357 seq_num 1061) 2021-03-15T22:32:54.879 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:54.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:54.901 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:54.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:55.003 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:32:55.025 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:55.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:55.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:32:55.127 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:55.153 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:55.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:55.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:55.234 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: debug 2021-03-15T22:32:55.230+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:32:55.234 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:32:55.234 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:55.256 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:55.276 INFO:tasks.rados.rados.0.smithi076.stdout:3085: done (3 left) 2021-03-15T22:32:55.276 INFO:tasks.rados.rados.0.smithi076.stdout:3087: done (2 left) 2021-03-15T22:32:55.277 INFO:tasks.rados.rados.0.smithi076.stdout:3090: done (1 left) 2021-03-15T22:32:55.277 INFO:tasks.rados.rados.0.smithi076.stdout:3091: done (0 left) 2021-03-15T22:32:55.278 INFO:tasks.rados.rados.0.smithi076.stdout:3092: read oid 7 snap -1 2021-03-15T22:32:55.278 INFO:tasks.rados.rados.0.smithi076.stdout:3092: expect (ObjNum 962 snap 321 seq_num 962) 2021-03-15T22:32:55.278 INFO:tasks.rados.rados.0.smithi076.stdout:3093: copy_from oid 28 from oid 21 current snap is 358 2021-03-15T22:32:55.279 INFO:tasks.rados.rados.0.smithi076.stdout:3094: snap_remove snap 318 2021-03-15T22:32:55.284 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:55.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:55.286 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:55.332 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-15T22:32:55.388 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:55.410 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:55.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:55.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:55.491 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:32:55.492 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:55.492 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:32:55.492 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:55.492 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:55.493 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:32:55.493 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: debug 2021-03-15T22:32:55.230+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:32:55.493 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:32:55.493 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:55.493 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:32:55.494 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:55.494 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:32:55.494 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:55.494 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:55.495 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:32:55.495 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: debug 2021-03-15T22:32:55.231+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:32:55.495 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:32:55.496 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:55.496 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:32:55.496 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:55.496 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:32:55.496 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:55.497 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:55.497 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:32:55.497 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: debug 2021-03-15T22:32:55.231+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:32:55.497 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:32:55.497 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:55.498 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:32:55.498 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:55.498 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:32:55.499 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:55.499 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:55.499 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:32:55.499 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: debug 2021-03-15T22:32:55.232+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:32:55.499 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:32:55.500 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:55.500 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:32:55.500 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:55.500 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:32:55.500 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:55.501 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:55.501 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:32:55.501 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: debug 2021-03-15T22:32:55.232+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:32:55.501 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:32:55.502 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:55.502 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:32:55.502 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:55.502 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:32:55.503 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:55.503 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:55.503 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:32:55.503 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: debug 2021-03-15T22:32:55.233+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:32:55.504 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:32:55.504 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:55.504 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:32:55.504 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:55.504 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:32:55.505 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:55.505 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:55.505 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:32:55.505 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56514]: cluster 2021-03-15T22:32:54.489807+0000 mon.a (mon.0) 2121 2021-03-15T22:32:55.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56514]: : cluster [DBG] osdmap e824: 8 total, 8 up, 7 in 2021-03-15T22:32:55.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56514]: audit 2021-03-15T22:32:54.951369+0000 mon.a (mon.0) 2122 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:32:55.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56514]: audit 2021-03-15T22:32:55.230130+0000 mon.a (mon.0) 2123 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:32:55.506 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56514]: cluster 2021-03-15T22:32:55.270049+0000 mon.a (mon.0) 2124 : cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2021-03-15T22:32:55.507 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[56514]: audit 2021-03-15T22:32:55.293330+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? 172.21.15.76:0/2365746495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:55.513 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:55.536 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:55.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:55.538 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:55.616 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:55.638 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:55.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:55.663 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:55.663 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:55.764 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:55.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: cluster 2021-03-15T22:32:54.489807 2021-03-15T22:32:55.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2121 : cluster [DBG] osdmap e824: 8 total, 8 up, 7 in 2021-03-15T22:32:55.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:32:55.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: 15T22:32:54.951369+0000 mon.a (mon.0) 2122 : audit [DBG] 2021-03-15T22:32:55.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:32:55.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: audit 2021-03-15T22:32:55.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: 2021-03-15T22:32:55.230130+0000 mon.a (mon. 2021-03-15T22:32:55.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: 0) 2123 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:32:55.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:32:55.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: 15T22:32:55.270049+0000 mon.a (mon 2021-03-15T22:32:55.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: .0) 2124 : cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2021-03-15T22:32:55.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: audit 2021 2021-03-15T22:32:55.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: -03-15T22:32:55.293330 2021-03-15T22:32:55.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2125 : audit [INF] 2021-03-15T22:32:55.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:55 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2365746495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:55.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:55 smithi125 conmon[56739]: cluster 2021-03-15T22:32:54.489807+0000 mon.a (mon.0) 2021-03-15T22:32:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:55 smithi125 conmon[56739]: 2121 : cluster [DBG] osdmap e824: 8 total, 8 up, 7 in 2021-03-15T22:32:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:55 smithi125 conmon[56739]: audit 2021-03-15T22:32:54.951369+0000 mon.a (mon.0) 2122 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:32:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:55 smithi125 conmon[56739]: audit 2021-03-15T22:32:55.230130+0000 mon.a (mon.0) 2123 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:32:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:55 smithi125 conmon[56739]: cluster 2021-03-15T22:32:55.270049+0000 mon.a (mon.0) 2124 : cluster [DBG] osdmap e825: 8 total, 8 up, 7 in 2021-03-15T22:32:55.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:55 smithi125 conmon[56739]: audit 2021-03-15T22:32:55.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:55 smithi125 conmon[56739]: 2021-03-15T22:32:55.293330+0000 mon.a (mon.0) 2125 : audit [INF] from='client.? 172.21.15.76:0/2365746495' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:32:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:55.789 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:55.790 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:55.891 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:55.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:55.915 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:55.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:56.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:56.029 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:32:56.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:56.041 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:56.041 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:56.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:56.166 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:56.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:56.168 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:56.269 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:56.289 INFO:tasks.rados.rados.0.smithi076.stdout:3094: done (2 left) 2021-03-15T22:32:56.289 INFO:tasks.rados.rados.0.smithi076.stdout:3095: delete oid 26 current snap is 358 2021-03-15T22:32:56.291 INFO:tasks.rados.rados.0.smithi076.stdout:3092: expect (ObjNum 962 snap 321 seq_num 962) 2021-03-15T22:32:56.297 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:56.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:56.298 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:56.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:56.407 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:32:56.422 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:56.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:56.423 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:56.449 INFO:tasks.rados.rados.0.smithi076.stdout:3093: finishing copy_from to smithi07694878-28 2021-03-15T22:32:56.449 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2395 (ObjNum 982 snap 326 seq_num 982) dirty exists 2021-03-15T22:32:56.450 INFO:tasks.rados.rados.0.smithi076.stdout:3093: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:32:56.450 INFO:tasks.rados.rados.0.smithi076.stdout:3092: done (2 left) 2021-03-15T22:32:56.450 INFO:tasks.rados.rados.0.smithi076.stdout:3093: done (1 left) 2021-03-15T22:32:56.450 INFO:tasks.rados.rados.0.smithi076.stdout:3095: done (0 left) 2021-03-15T22:32:56.451 INFO:tasks.rados.rados.0.smithi076.stdout:3096: snap_create 2021-03-15T22:32:56.524 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:56.547 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:56.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:56.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:56.650 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:56.672 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:56.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:56.673 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:56.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cluster 2021-03-15T22:32:55.182479+0000 mgr.y (mgr.14144) 954 : cluster [DBG] pgmap v1381: 148 pgs: 1 peering, 147 active+clean; 507 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 22 op/s 2021-03-15T22:32:56.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:32:55.226923+0000 mgr.y (mgr.14144) 955 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:32:56.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:32:55.231056+0000 mgr.y (mgr.14144) 956 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:32:56.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:32:56.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:32:56.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:32:56.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:32:56.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:32:55.231572+0000 mgr.y (mgr.14144) 957 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:32:56.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:32:56.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:32:56.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:32:56.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:32:56.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:32:55.231980+0000 mgr.y (mgr.14144) 958 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:32:56.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:32:56.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:32:56.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cluster 2021-03-15T22:32:55.182479+0000 mgr.y (mgr.14144) 2021-03-15T22:32:56.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: 954 : cluster [DBG] pgmap v1381: 148 pgs: 1 peering, 147 active+clean; 507 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 22 op/s 2021-03-15T22:32:56.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:32:55.226923+0000 mgr.y (mgr.14144) 955 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:32:56.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:32:55.231056 2021-03-15T22:32:56.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 956 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:32:56.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:32:56.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:32:56.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:32:56.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:32:56.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:32:55.231572+0000 mgr.y (mgr.14144) 957 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:32:56.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:32:56.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:32:56.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:32:56.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:32:56.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:32:55.231980+0000 mgr.y (mgr.14144) 958 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:32:56.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:32:56.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:32:56.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:32:56.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:32:56.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:32:55.232487+0000 mgr.y (mgr.14144) 959 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:32:56.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:32:56.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:32:56.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:32:56.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:32:56.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:32:56.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: 2021-03-15T22:32:55.233054+0000 mgr.y (mgr.14144) 960 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:32:56.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:32:56.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:32:56.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:32:56.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:32:56.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:32:56.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: 2021-03-15T22:32:55.233388+0000 mgr.y (mgr.14144) 961 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:32:56.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:32:56.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:32:56.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.786 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:32:56.787 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:32:56.787 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:32:56.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:32:55.232487+0000 mgr.y (mgr.14144) 959 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:32:56.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:32:56.788 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:32:56.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:32:56.789 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:32:56.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:32:55.233054+0000 mgr.y (mgr.14144) 960 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:32:56.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:32:56.790 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:32:56.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:32:56.791 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:32:56.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:32:55.233388+0000 mgr.y (mgr.14144) 961 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:32:56.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:32:56.792 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:32:56.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:32:56.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.793 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.794 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:32:56.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.794 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:32:56.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cephadm 2021-03-15 2021-03-15T22:32:56.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: T22:32:55.233715+0000 mgr.y (mgr.14144) 962 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:32:56.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:32:56.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.795 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:32:56.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:32:56.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.796 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:32:56.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cluster 2021-03-15T22:32:55 2021-03-15T22:32:56.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: .514177+0000 osd.7 (osd.7) 155 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:56.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cluster 2021-03-15T22:32:55.515353+0000 osd.7 (osd.7 2021-03-15T22:32:56.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: ) 156 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:56.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cluster 2021-03-15T22:32:56.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: 2021-03-15T22:32:56. 2021-03-15T22:32:56.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: 274510+0000 mon.a (mon.0) 2126 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:56.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: audit 2021 2021-03-15T22:32:56.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: -03-15T22:32:56.279146+0000 mon.a ( 2021-03-15T22:32:56.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: mon.0) 2127 : audit [INF] from='client.? 172.21.15.76:0/2365746495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:56.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:32:56.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: T22:32:56.279228+0000 mon.a (mon.0) 2021-03-15T22:32:56.799 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[62156]: 2128 : cluster [DBG] osdmap e826: 8 total, 8 up, 7 in 2021-03-15T22:32:56.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:32:56.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: 2021-03-15T22:32:55.233715+0000 mgr.y (mgr.14144) 962 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:32:56.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:32:56.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:32:56.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:32:56.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:32:56.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cluster 2021-03-15T22:32:55.514177+0000 osd.7 (osd.7) 155 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:56.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cluster 2021-03-15T22:32:55.515353+0000 osd.7 (osd.7) 156 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:56.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cluster 2021-03-15T22:32:56.274510+0000 mon.a (mon.0) 2126 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:56.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: audit 2021-03-15T22:32:56.279146+0000 mon.a (mon.0) 2127 : audit [INF] from='client.? 172.21.15.76:0/2365746495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:56.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:56 smithi076 conmon[56514]: cluster 2021-03-15T22:32:56.279228+0000 mon.a (mon.0) 2128 : cluster [DBG] osdmap e826: 8 total, 8 up, 7 in 2021-03-15T22:32:56.803 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:56.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cluster 2021-03-15T22:32:55.182479+0000 mgr.y (mgr.14144 2021-03-15T22:32:56.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: ) 954 : cluster [DBG] pgmap v1381: 148 pgs: 1 peering, 147 active+clean; 507 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 6.6 MiB/s wr, 22 op/s 2021-03-15T22:32:56.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:32:55.226923+0000 mgr.y (mgr.14144) 955 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:32:56.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cephadm 2021-03-15T22: 2021-03-15T22:32:56.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 32:55.231056+0000 mgr.y (mgr 2021-03-15T22:32:56.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: .14144) 956 : cephadm [ERR] 2021-03-15T22:32:56.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:32:56.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:32:56.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:32:56.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:32:56.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:32:56.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:32:55.231572+0000 mgr.y (mgr.14144) 957 : cephadm [ERR] 2021-03-15T22:32:56.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:32:56.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:32:56.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:32:56.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:32:56.809 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:32:56.810 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:32:55.231980+0000 mgr.y (mgr.14144) 958 : cephadm 2021-03-15T22:32:56.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:32:56.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:32:56.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:32:56.811 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.812 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:32:56.812 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.812 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.812 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:32:56.813 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:32:55.232487+0000 mgr.y (mgr. 2021-03-15T22:32:56.813 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 14144) 959 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:32:56.813 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:32:56.813 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.813 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:32:56.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:32:56.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:32:56.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:32:55.233054+0000 mgr.y ( 2021-03-15T22:32:56.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: mgr.14144) 960 : cephadm 2021-03-15T22:32:56.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:32:56.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:32:56.816 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.816 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:32:56.816 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.816 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:32:56.816 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.817 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.817 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:32:56.817 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:32:55.233388+0000 mgr.y (mgr.14144) 2021-03-15T22:32:56.817 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 961 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:32:56.818 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:32:56.818 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.818 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:32:56.818 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.818 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:32:56.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:32:56.819 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:32:55.233715+0000 mgr.y (mgr 2021-03-15T22:32:56.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: .14144) 962 : cephadm [ERR] 2021-03-15T22:32:56.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:32:56.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:32:56.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:32:56.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:32:56.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:32:56.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:32:56.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:32:56.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:32:56.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:32:56.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cluster 2021-03-15T22:32:55.514177+0000 osd.7 (osd.7) 155 : 2021-03-15T22:32:56.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:56.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:32:56.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 03-15T22:32:55. 2021-03-15T22:32:56.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 515353+0000 osd.7 (osd.7 2021-03-15T22:32:56.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: ) 156 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:56.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:32:56.824 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 03-15T22:32:56. 2021-03-15T22:32:56.824 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 274510+0000 mon.a (mon.0) 2021-03-15T22:32:56.824 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 2126 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:32:56.824 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:32:56.824 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 03-15T22:32:56 2021-03-15T22:32:56.825 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: .279146+0000 mon.a (mon.0 2021-03-15T22:32:56.825 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: ) 2127 : audit [INF] from='client.? 172.21.15.76:0/2365746495' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:32:56.825 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 2021-03-15T22:32:56.825 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:32:56.826 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: -15T22:32:56. 2021-03-15T22:32:56.826 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 279228+0000 mon.a (mon.0 2021-03-15T22:32:56.826 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: ) 2128 : cluster [DBG] osdmap e826: 8 total, 8 up, 7 in 2021-03-15T22:32:56.826 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:56 smithi125 conmon[56739]: 2021-03-15T22:32:56.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:56.850 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:56.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:56.951 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:56.976 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:56.977 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:57.078 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:57.102 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:57.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:57.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:57.204 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:57.228 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:57.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:57.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:57.284 INFO:tasks.rados.rados.0.smithi076.stdout:3096: done (0 left) 2021-03-15T22:32:57.285 INFO:tasks.rados.rados.0.smithi076.stdout:3097: read oid 42 snap -1 2021-03-15T22:32:57.285 INFO:tasks.rados.rados.0.smithi076.stdout:3097: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:32:57.285 INFO:tasks.rados.rados.0.smithi076.stdout:3098: read oid 10 snap -1 2021-03-15T22:32:57.285 INFO:tasks.rados.rados.0.smithi076.stdout:3098: expect deleted 2021-03-15T22:32:57.286 INFO:tasks.rados.rados.0.smithi076.stdout:3099: write oid 19 current snap is 359 2021-03-15T22:32:57.286 INFO:tasks.rados.rados.0.smithi076.stdout:3099: seq_num 1063 ranges {665841=562754,1687653=572365,2708406=776032} 2021-03-15T22:32:57.291 INFO:tasks.rados.rados.0.smithi076.stdout:3099: writing smithi07694878-19 from 665841 to 1228595 tid 1 2021-03-15T22:32:57.297 INFO:tasks.rados.rados.0.smithi076.stdout:3099: writing smithi07694878-19 from 1687653 to 2260018 tid 2 2021-03-15T22:32:57.304 INFO:tasks.rados.rados.0.smithi076.stdout:3099: writing smithi07694878-19 from 2708406 to 3484438 tid 3 2021-03-15T22:32:57.304 INFO:tasks.rados.rados.0.smithi076.stdout:3100: rollback oid 1 current snap is 359 2021-03-15T22:32:57.304 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 1 to 355 2021-03-15T22:32:57.305 INFO:tasks.rados.rados.0.smithi076.stdout:3099: finishing write tid 1 to smithi07694878-19 2021-03-15T22:32:57.305 INFO:tasks.rados.rados.0.smithi076.stdout:3099: finishing write tid 2 to smithi07694878-19 2021-03-15T22:32:57.305 INFO:tasks.rados.rados.0.smithi076.stdout:3097: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:32:57.331 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:57.353 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:57.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:57.355 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:57.438 INFO:tasks.rados.rados.0.smithi076.stdout:3099: finishing write tid 3 to smithi07694878-19 2021-03-15T22:32:57.438 INFO:tasks.rados.rados.0.smithi076.stdout:3099: finishing write tid 4 to smithi07694878-19 2021-03-15T22:32:57.439 INFO:tasks.rados.rados.0.smithi076.stdout:3100: finishing rollback tid 0 to smithi07694878-1 2021-03-15T22:32:57.439 INFO:tasks.rados.rados.0.smithi076.stdout:3097: done (3 left) 2021-03-15T22:32:57.439 INFO:tasks.rados.rados.0.smithi076.stdout:3098: done (2 left) 2021-03-15T22:32:57.439 INFO:tasks.rados.rados.0.smithi076.stdout:3101: read oid 43 snap -1 2021-03-15T22:32:57.439 INFO:tasks.rados.rados.0.smithi076.stdout:3101: expect deleted 2021-03-15T22:32:57.440 INFO:tasks.rados.rados.0.smithi076.stdout:3102: read oid 26 snap -1 2021-03-15T22:32:57.440 INFO:tasks.rados.rados.0.smithi076.stdout:3102: expect deleted 2021-03-15T22:32:57.440 INFO:tasks.rados.rados.0.smithi076.stdout:3103: snap_create 2021-03-15T22:32:57.440 INFO:tasks.rados.rados.0.smithi076.stdout:3099: finishing write tid 5 to smithi07694878-19 2021-03-15T22:32:57.441 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2126 (ObjNum 1063 snap 359 seq_num 1063) dirty exists 2021-03-15T22:32:57.441 INFO:tasks.rados.rados.0.smithi076.stdout:3099: left oid 19 (ObjNum 1063 snap 359 seq_num 1063) 2021-03-15T22:32:57.441 INFO:tasks.rados.rados.0.smithi076.stdout:3100: finishing rollback tid 1 to smithi07694878-1 2021-03-15T22:32:57.441 INFO:tasks.rados.rados.0.smithi076.stdout:3100: finishing rollback tid 2 to smithi07694878-1 2021-03-15T22:32:57.441 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 2398 (ObjNum 982 snap 326 seq_num 982) dirty exists 2021-03-15T22:32:57.444 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:57.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:57.478 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:57.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:57.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:57.581 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:57.604 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:57.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:57.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:57.706 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:57.716 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:32:57.717 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:32:57.729 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:57.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:57.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:57.831 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:57.854 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:57.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:57.856 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:57.957 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:57.982 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:57.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:57.983 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:58.084 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:58.107 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:58.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:58.108 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:58.209 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:58.231 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:58.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:58.233 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:58.289 INFO:tasks.rados.rados.0.smithi076.stdout:3099: done (4 left) 2021-03-15T22:32:58.289 INFO:tasks.rados.rados.0.smithi076.stdout:3100: done (3 left) 2021-03-15T22:32:58.289 INFO:tasks.rados.rados.0.smithi076.stdout:3101: done (2 left) 2021-03-15T22:32:58.290 INFO:tasks.rados.rados.0.smithi076.stdout:3102: done (1 left) 2021-03-15T22:32:58.290 INFO:tasks.rados.rados.0.smithi076.stdout:3103: done (0 left) 2021-03-15T22:32:58.290 INFO:tasks.rados.rados.0.smithi076.stdout:3104: snap_create 2021-03-15T22:32:58.334 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:32:58.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:58.358 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:58.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:32:58.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:58.484 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:58.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:58.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:58.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:58 smithi076 conmon[56514]: cluster 2021-03-15T22:32:57.183032+0000 mgr.y (mgr.14144) 963 : cluster [DBG] 2021-03-15T22:32:58.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:58 smithi076 conmon[56514]: pgmap v1384: 148 pgs: 148 active+clean; 499 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-03-15T22:32:58.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:58 smithi076 conmon[56514]: cluster 2021-03-15T22:32:57.280408+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e827: 8 total, 8 up, 7 in 2021-03-15T22:32:58.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:58 smithi076 conmon[62156]: cluster 2021-03-15T22:32:57.183032 2021-03-15T22:32:58.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:58 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 963 : cluster [DBG] pgmap v1384: 148 pgs: 148 active+clean; 499 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-03-15T22:32:58.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:58 smithi076 conmon[62156]: cluster 2021-03-15T22:32:57.280408+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e827: 8 total, 8 up, 7 in 2021-03-15T22:32:58.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:58 smithi125 conmon[56739]: cluster 2021-03-15T22:32:57.183032+0000 mgr.y (mgr.14144) 963 : cluster [DBG] pgmap v1384: 148 pgs: 148 active+clean; 499 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 11 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-03-15T22:32:58.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:58 smithi125 conmon[56739]: cluster 2021-03-15T22:32:57.280408+0000 mon.a (mon.0) 2129 : cluster [DBG] osdmap e827: 8 total, 8 up, 7 in 2021-03-15T22:32:58.586 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:32:58.610 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:58.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:58.611 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:32:58.712 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:58.736 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:58.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:58.737 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:58.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:58.862 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:58.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:58.864 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:58.965 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:58.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:58.990 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:58.990 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:59.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:32:59.116 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:59.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:59.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:32:59.120 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 7, 0, 3, 5] out_osds: [6] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:32:59.120 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:32:59.121 INFO:tasks.thrashosds.thrasher:Adding osd 6 2021-03-15T22:32:59.121 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd in 6 2021-03-15T22:32:59.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-15T22:32:59.218 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:59.243 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:59.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:59.246 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:59.297 INFO:tasks.rados.rados.0.smithi076.stdout:3104: done (0 left) 2021-03-15T22:32:59.298 INFO:tasks.rados.rados.0.smithi076.stdout:3105: write (excl) oid 34 current snap is 361 2021-03-15T22:32:59.298 INFO:tasks.rados.rados.0.smithi076.stdout:3105: seq_num 1064 ranges {682648=638003,2029782=432376,3019476=587304} 2021-03-15T22:32:59.304 INFO:tasks.rados.rados.0.smithi076.stdout:3105: writing smithi07694878-34 from 682648 to 1320651 tid 1 2021-03-15T22:32:59.315 INFO:tasks.rados.rados.0.smithi076.stdout:3105: writing smithi07694878-34 from 2029782 to 2462158 tid 2 2021-03-15T22:32:59.323 INFO:tasks.rados.rados.0.smithi076.stdout:3105: writing smithi07694878-34 from 3019476 to 3606780 tid 3 2021-03-15T22:32:59.323 INFO:tasks.rados.rados.0.smithi076.stdout:3106: snap_create 2021-03-15T22:32:59.323 INFO:tasks.rados.rados.0.smithi076.stdout:3105: finishing write tid 1 to smithi07694878-34 2021-03-15T22:32:59.323 INFO:tasks.rados.rados.0.smithi076.stdout:3105: finishing write tid 2 to smithi07694878-34 2021-03-15T22:32:59.325 INFO:tasks.rados.rados.0.smithi076.stdout:3105: finishing write tid 3 to smithi07694878-34 2021-03-15T22:32:59.326 INFO:tasks.rados.rados.0.smithi076.stdout:3105: finishing write tid 4 to smithi07694878-34 2021-03-15T22:32:59.326 INFO:tasks.rados.rados.0.smithi076.stdout:3105: finishing write tid 5 to smithi07694878-34 2021-03-15T22:32:59.327 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 2399 (ObjNum 1064 snap 361 seq_num 1064) dirty exists 2021-03-15T22:32:59.327 INFO:tasks.rados.rados.0.smithi076.stdout:3105: left oid 34 (ObjNum 1064 snap 361 seq_num 1064) 2021-03-15T22:32:59.347 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:32:59.373 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:32:59.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:59.375 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:32:59.444 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:32:59.456 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[56514]: cluster 2021-03-15T22:32:58.049315+0000 osd.0 (osd.0) 135 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:59.457 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[56514]: cluster 2021-03-15T22:32:58.050601+0000 osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:59.457 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[56514]: cluster 2021-03-15T22:32:58.287027+0000 mon.a (mon.0) 2130 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2021-03-15T22:32:59.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: cluster 2021-03-15T22:32:59.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: 2021-03-15T22:32:59.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: 2021 2021-03-15T22:32:59.459 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: - 2021-03-15T22:32:59.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: 03 2021-03-15T22:32:59.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: - 2021-03-15T22:32:59.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: 15T22:32:58.049315+0000 osd.0 2021-03-15T22:32:59.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: (osd.0) 135 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:59.460 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: cluster 2021-03-15T22:32:58.050601+0000 2021-03-15T22:32:59.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:59.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:32:59.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: 03-15T22:32:58.287027+0000 mon.a ( 2021-03-15T22:32:59.461 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:32:59 smithi076 conmon[62156]: mon.0) 2130 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2021-03-15T22:32:59.476 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:59.499 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:59.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:59.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:59.601 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:32:59.627 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:59.628 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:32:59.630 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:59.647 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:59.729 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:32:59.740 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:32:59.752 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:59.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:59.753 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:32:59.758 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:32:59.777 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:32:59.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:59 smithi125 conmon[56739]: cluster 2021-03-15T22:32:58.049315+0000 osd.0 (osd 2021-03-15T22:32:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:59 smithi125 conmon[56739]: .0) 135 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:32:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:59 smithi125 conmon[56739]: cluster 2021-03-15T22:32:58.050601+0000 osd.0 (osd.0) 136 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:32:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:59 smithi125 conmon[56739]: cluster 2021-03-15T22:32:58 2021-03-15T22:32:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:59 smithi125 conmon[56739]: .287027+0000 mon.a (mon.0) 2021-03-15T22:32:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:32:59 smithi125 conmon[56739]: 2130 : cluster [DBG] osdmap e828: 8 total, 8 up, 7 in 2021-03-15T22:32:59.793 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:32:59.796 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:32:59.816 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:32:59.836 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:32:59.855 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:32:59.860 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:32:59.877 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:32:59.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:32:59.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:32:59.879 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:32:59.880 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:32:59.880 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:32:59.880 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:32:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:32:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:32:59.881 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:32:59.882 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:32:59.882 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:32:59.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:00.004 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:00.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:00.107 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:00.133 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:00.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:00.135 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:00.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:00.258 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:00.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:00.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:00.276 INFO:tasks.rados.rados.0.smithi076.stdout:3105: done (1 left) 2021-03-15T22:33:00.276 INFO:tasks.rados.rados.0.smithi076.stdout:3106: done (0 left) 2021-03-15T22:33:00.276 INFO:tasks.rados.rados.0.smithi076.stdout:3107: read oid 21 snap 345 2021-03-15T22:33:00.276 INFO:tasks.rados.rados.0.smithi076.stdout:3107: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:00.277 INFO:tasks.rados.rados.0.smithi076.stdout:3108: write oid 14 current snap is 362 2021-03-15T22:33:00.277 INFO:tasks.rados.rados.0.smithi076.stdout:3108: seq_num 1065 ranges {699455=429605,1804617=576034,3163252=514929} 2021-03-15T22:33:00.280 INFO:tasks.rados.rados.0.smithi076.stdout:3108: writing smithi07694878-14 from 699455 to 1129060 tid 1 2021-03-15T22:33:00.288 INFO:tasks.rados.rados.0.smithi076.stdout:3108: writing smithi07694878-14 from 1804617 to 2380651 tid 2 2021-03-15T22:33:00.300 INFO:tasks.rados.rados.0.smithi076.stdout:3108: writing smithi07694878-14 from 3163252 to 3678181 tid 3 2021-03-15T22:33:00.301 INFO:tasks.rados.rados.0.smithi076.stdout:3109: rollback oid 9 current snap is 362 2021-03-15T22:33:00.301 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 9 to 349 2021-03-15T22:33:00.301 INFO:tasks.rados.rados.0.smithi076.stdout:3107: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:00.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:00.385 INFO:tasks.rados.rados.0.smithi076.stdout:3108: finishing write tid 1 to smithi07694878-14 2021-03-15T22:33:00.386 INFO:tasks.rados.rados.0.smithi076.stdout:3108: finishing write tid 2 to smithi07694878-14 2021-03-15T22:33:00.386 INFO:tasks.rados.rados.0.smithi076.stdout:3107: done (2 left) 2021-03-15T22:33:00.386 INFO:tasks.rados.rados.0.smithi076.stdout:3110: rollback oid 47 current snap is 362 2021-03-15T22:33:00.387 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 47 to 360 2021-03-15T22:33:00.387 INFO:tasks.rados.rados.0.smithi076.stdout:3108: finishing write tid 3 to smithi07694878-14 2021-03-15T22:33:00.387 INFO:tasks.rados.rados.0.smithi076.stdout:3109: finishing rollback tid 1 to smithi07694878-9 2021-03-15T22:33:00.387 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 1888 (ObjNum 1 snap 21896 seq_num 264839376) dirty dne 2021-03-15T22:33:00.388 INFO:tasks.rados.rados.0.smithi076.stdout:3108: finishing write tid 4 to smithi07694878-14 2021-03-15T22:33:00.388 INFO:tasks.rados.rados.0.smithi076.stdout:3108: finishing write tid 5 to smithi07694878-14 2021-03-15T22:33:00.388 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 2651 (ObjNum 1065 snap 362 seq_num 1065) dirty exists 2021-03-15T22:33:00.389 INFO:tasks.rados.rados.0.smithi076.stdout:3108: left oid 14 (ObjNum 1065 snap 362 seq_num 1065) 2021-03-15T22:33:00.389 INFO:tasks.rados.rados.0.smithi076.stdout:3108: done (2 left) 2021-03-15T22:33:00.389 INFO:tasks.rados.rados.0.smithi076.stdout:3109: done (1 left) 2021-03-15T22:33:00.389 INFO:tasks.rados.rados.0.smithi076.stdout:3111: write oid 31 current snap is 362 2021-03-15T22:33:00.389 INFO:tasks.rados.rados.0.smithi076.stdout:3111: seq_num 1066 ranges {716262=504854,1863099=719692,3074322=726201} 2021-03-15T22:33:00.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:00.391 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:00.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:00.392 INFO:tasks.rados.rados.0.smithi076.stdout:3111: writing smithi07694878-31 from 716262 to 1221116 tid 1 2021-03-15T22:33:00.399 INFO:tasks.rados.rados.0.smithi076.stdout:3111: writing smithi07694878-31 from 1863099 to 2582791 tid 2 2021-03-15T22:33:00.406 INFO:tasks.rados.rados.0.smithi076.stdout:3111: writing smithi07694878-31 from 3074322 to 3800523 tid 3 2021-03-15T22:33:00.407 INFO:tasks.rados.rados.0.smithi076.stdout:3112: rollback oid 8 current snap is 362 2021-03-15T22:33:00.407 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 357 2021-03-15T22:33:00.407 INFO:tasks.rados.rados.0.smithi076.stdout:3113: read oid 17 snap -1 2021-03-15T22:33:00.407 INFO:tasks.rados.rados.0.smithi076.stdout:3113: expect (ObjNum 1039 snap 347 seq_num 1039) 2021-03-15T22:33:00.407 INFO:tasks.rados.rados.0.smithi076.stdout:3114: snap_create 2021-03-15T22:33:00.419 INFO:tasks.rados.rados.0.smithi076.stdout:3113: expect (ObjNum 1039 snap 347 seq_num 1039) 2021-03-15T22:33:00.492 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:00.515 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:00.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:00.517 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[56514]: cluster 2021-03-15T22:32:59.183404 2021-03-15T22:33:00.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 964 : cluster [DBG] pgmap v1387: 148 pgs: 148 active+clean; 497 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 0 B/s wr, 2 op/s 2021-03-15T22:33:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[56514]: cluster 2021-03-15T22:32:59.293600+0000 mon.a (mon.0) 2131 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2021-03-15T22:33:00.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[56514]: audit 2021-03-15T22:32:59.711269+0000 mon.c (mon.1) 65 : audit [DBG] from='client.? 172.21.15.76:0/672992219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:00.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[56514]: cluster 2021-03-15T22:33:00.271956+0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e830: 8 total, 8 up, 7 in 2021-03-15T22:33:00.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[62156]: cluster 2021-03-15T22:32:59.183404+0000 mgr.y (mgr.14144) 964 : cluster [DBG] 2021-03-15T22:33:00.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[62156]: pgmap v1387: 148 pgs: 148 active+clean; 497 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 0 B/s wr, 2 op/s 2021-03-15T22:33:00.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[62156]: cluster 2021-03-15T22:32:59.293600+0000 mon.a (mon.0) 2131 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2021-03-15T22:33:00.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[62156]: audit 2021-03-15T22:33:00.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[62156]: 2021-03-15T22:32:59.711269+0000 mon.c (mon.1) 65 : audit [DBG] from='client.? 172.21.15.76:0/672992219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:00.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[62156]: cluster 2021-03-15T22:33:00.271956 2021-03-15T22:33:00.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:00 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2132 : cluster [DBG] osdmap e830: 8 total, 8 up, 7 in 2021-03-15T22:33:00.537 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:33:00 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:33:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:00 smithi125 conmon[56739]: cluster 2021-03-15T22:32:59.183404+0000 mgr.y (mgr.14144) 964 : cluster [DBG] pgmap v1387: 148 pgs: 148 active+clean; 497 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 2.1 MiB/s rd, 0 B/s wr, 2 op/s 2021-03-15T22:33:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:00 smithi125 conmon[56739]: cluster 2021-03-15T22:32:59.293600+0000 2021-03-15T22:33:00.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:00 smithi125 conmon[56739]: mon.a (mon.0) 2131 : cluster [DBG] osdmap e829: 8 total, 8 up, 7 in 2021-03-15T22:33:00.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:00 smithi125 conmon[56739]: audit 2021-03-15T22:32:59.711269+0000 mon.c (mon.1) 65 : audit [DBG] from='client.? 172.21.15.76:0/672992219' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:00.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:00 smithi125 conmon[56739]: cluster 2021-03-15T22:33:00.271956+0000 mon.a (mon.0) 2132 : cluster [DBG] 2021-03-15T22:33:00.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:00 smithi125 conmon[56739]: osdmap e830: 8 total, 8 up, 7 in 2021-03-15T22:33:00.619 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:00.643 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:00.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:00.645 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:00.658 INFO:tasks.rados.rados.0.smithi076.stdout:3110: finishing rollback tid 0 to smithi07694878-47 2021-03-15T22:33:00.658 INFO:tasks.rados.rados.0.smithi076.stdout:3110: finishing rollback tid 1 to smithi07694878-47 2021-03-15T22:33:00.658 INFO:tasks.rados.rados.0.smithi076.stdout:3110: finishing rollback tid 2 to smithi07694878-47 2021-03-15T22:33:00.659 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1790 (ObjNum 1061 snap 357 seq_num 1061) dirty exists 2021-03-15T22:33:00.659 INFO:tasks.rados.rados.0.smithi076.stdout:3112: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:33:00.659 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1255 (ObjNum 1953066355 snap 925919592 seq_num 942946614) dirty dne 2021-03-15T22:33:00.659 INFO:tasks.rados.rados.0.smithi076.stdout:3111: finishing write tid 1 to smithi07694878-31 2021-03-15T22:33:00.660 INFO:tasks.rados.rados.0.smithi076.stdout:3111: finishing write tid 2 to smithi07694878-31 2021-03-15T22:33:00.660 INFO:tasks.rados.rados.0.smithi076.stdout:3111: finishing write tid 3 to smithi07694878-31 2021-03-15T22:33:00.660 INFO:tasks.rados.rados.0.smithi076.stdout:3111: finishing write tid 4 to smithi07694878-31 2021-03-15T22:33:00.660 INFO:tasks.rados.rados.0.smithi076.stdout:3111: finishing write tid 5 to smithi07694878-31 2021-03-15T22:33:00.660 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 2402 (ObjNum 1066 snap 362 seq_num 1066) dirty exists 2021-03-15T22:33:00.661 INFO:tasks.rados.rados.0.smithi076.stdout:3111: left oid 31 (ObjNum 1066 snap 362 seq_num 1066) 2021-03-15T22:33:00.746 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:00.770 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:00.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:00.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:00.873 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:00.894 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:00.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:00.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:00.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:01.021 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:01.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:01.022 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:01.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:01.146 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:01.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:01.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:01.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:01.273 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:01.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:01.274 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:01.308 INFO:tasks.rados.rados.0.smithi076.stdout:3110: done (4 left) 2021-03-15T22:33:01.309 INFO:tasks.rados.rados.0.smithi076.stdout:3111: done (3 left) 2021-03-15T22:33:01.309 INFO:tasks.rados.rados.0.smithi076.stdout:3112: done (2 left) 2021-03-15T22:33:01.309 INFO:tasks.rados.rados.0.smithi076.stdout:3113: done (1 left) 2021-03-15T22:33:01.309 INFO:tasks.rados.rados.0.smithi076.stdout:3114: done (0 left) 2021-03-15T22:33:01.310 INFO:tasks.rados.rados.0.smithi076.stdout:3115: rollback oid 5 current snap is 363 2021-03-15T22:33:01.310 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 352 2021-03-15T22:33:01.310 INFO:tasks.rados.rados.0.smithi076.stdout:3116: write (excl) oid 11 current snap is 363 2021-03-15T22:33:01.310 INFO:tasks.rados.rados.0.smithi076.stdout:3116: seq_num 1067 ranges {733069=580103,1921581=579703,3218098=537473} 2021-03-15T22:33:01.315 INFO:tasks.rados.rados.0.smithi076.stdout:3116: writing smithi07694878-11 from 733069 to 1313172 tid 1 2021-03-15T22:33:01.321 INFO:tasks.rados.rados.0.smithi076.stdout:3116: writing smithi07694878-11 from 1921581 to 2501284 tid 2 2021-03-15T22:33:01.329 INFO:tasks.rados.rados.0.smithi076.stdout:3116: writing smithi07694878-11 from 3218098 to 3755571 tid 3 2021-03-15T22:33:01.329 INFO:tasks.rados.rados.0.smithi076.stdout:3117: rollback oid 40 current snap is 363 2021-03-15T22:33:01.329 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 40 to 334 2021-03-15T22:33:01.330 INFO:tasks.rados.rados.0.smithi076.stdout:3115: finishing rollback tid 0 to smithi07694878-5 2021-03-15T22:33:01.330 INFO:tasks.rados.rados.0.smithi076.stdout:3115: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:33:01.330 INFO:tasks.rados.rados.0.smithi076.stdout:3115: finishing rollback tid 2 to smithi07694878-5 2021-03-15T22:33:01.330 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1980 (ObjNum 1016 snap 341 seq_num 1016) dirty exists 2021-03-15T22:33:01.330 INFO:tasks.rados.rados.0.smithi076.stdout:3116: finishing write tid 1 to smithi07694878-11 2021-03-15T22:33:01.331 INFO:tasks.rados.rados.0.smithi076.stdout:3115: done (2 left) 2021-03-15T22:33:01.331 INFO:tasks.rados.rados.0.smithi076.stdout:3118: write oid 25 current snap is 363 2021-03-15T22:33:01.331 INFO:tasks.rados.rados.0.smithi076.stdout:3116: finishing write tid 2 to smithi07694878-11 2021-03-15T22:33:01.331 INFO:tasks.rados.rados.0.smithi076.stdout:3118: seq_num 1068 ranges {749876=655352,1863710=723361,3012815=748745} 2021-03-15T22:33:01.336 INFO:tasks.rados.rados.0.smithi076.stdout:3118: writing smithi07694878-25 from 749876 to 1405228 tid 1 2021-03-15T22:33:01.343 INFO:tasks.rados.rados.0.smithi076.stdout:3118: writing smithi07694878-25 from 1863710 to 2587071 tid 2 2021-03-15T22:33:01.351 INFO:tasks.rados.rados.0.smithi076.stdout:3118: writing smithi07694878-25 from 3012815 to 3761560 tid 3 2021-03-15T22:33:01.351 INFO:tasks.rados.rados.0.smithi076.stdout:3119: snap_create 2021-03-15T22:33:01.351 INFO:tasks.rados.rados.0.smithi076.stdout:3116: finishing write tid 3 to smithi07694878-11 2021-03-15T22:33:01.351 INFO:tasks.rados.rados.0.smithi076.stdout:3116: finishing write tid 4 to smithi07694878-11 2021-03-15T22:33:01.352 INFO:tasks.rados.rados.0.smithi076.stdout:3116: finishing write tid 5 to smithi07694878-11 2021-03-15T22:33:01.352 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 1988 (ObjNum 1067 snap 363 seq_num 1067) dirty exists 2021-03-15T22:33:01.352 INFO:tasks.rados.rados.0.smithi076.stdout:3116: left oid 11 (ObjNum 1067 snap 363 seq_num 1067) 2021-03-15T22:33:01.352 INFO:tasks.rados.rados.0.smithi076.stdout:3117: finishing rollback tid 0 to smithi07694878-40 2021-03-15T22:33:01.352 INFO:tasks.rados.rados.0.smithi076.stdout:3117: finishing rollback tid 1 to smithi07694878-40 2021-03-15T22:33:01.353 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 2740 (ObjNum 261993792 snap 21896 seq_num 261715856) dirty dne 2021-03-15T22:33:01.353 INFO:tasks.rados.rados.0.smithi076.stdout:3118: finishing write tid 1 to smithi07694878-25 2021-03-15T22:33:01.353 INFO:tasks.rados.rados.0.smithi076.stdout:3118: finishing write tid 2 to smithi07694878-25 2021-03-15T22:33:01.355 INFO:tasks.rados.rados.0.smithi076.stdout:3118: finishing write tid 3 to smithi07694878-25 2021-03-15T22:33:01.355 INFO:tasks.rados.rados.0.smithi076.stdout:3118: finishing write tid 4 to smithi07694878-25 2021-03-15T22:33:01.355 INFO:tasks.rados.rados.0.smithi076.stdout:3118: finishing write tid 5 to smithi07694878-25 2021-03-15T22:33:01.355 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 1982 (ObjNum 1068 snap 363 seq_num 1068) dirty exists 2021-03-15T22:33:01.356 INFO:tasks.rados.rados.0.smithi076.stdout:3118: left oid 25 (ObjNum 1068 snap 363 seq_num 1068) 2021-03-15T22:33:01.375 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:01.403 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:01.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:01.404 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:01.506 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:01.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:01 smithi076 conmon[56514]: audit 2021-03-15T22:33:00.157455+0000 mgr.y (mgr.14144) 965 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:01 smithi076 conmon[56514]: audit 2021-03-15T22:33:01.113653+0000 mon.a (mon.0) 2133 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:33:01.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:01 smithi076 conmon[56514]: audit 2021-03-15T22:33:01.114699+0000 mon.a (mon.0) 2134 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:33:01.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:01 smithi076 conmon[62156]: audit 2021-03-15T22:33:00.157455+0000 mgr.y (mgr.14144) 965 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:01.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:01 smithi076 conmon[62156]: audit 2021-03-15T22:33:01.113653+0000 mon.a (mon.0) 2133 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:33:01.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:01 smithi076 conmon[62156]: audit 2021-03-15T22:33:01.114699+0000 mon.a (mon.0) 2134 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:33:01.529 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:01.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:01.632 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:01.655 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:01.658 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:01.759 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:01.782 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:01.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:01.784 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:01.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:01 smithi125 conmon[56739]: audit 2021-03-15T22:33:00.157455+0000 2021-03-15T22:33:01.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:01 smithi125 conmon[56739]: mgr.y (mgr.14144) 965 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:01 smithi125 conmon[56739]: audit 2021-03-15T22:33:01.113653+0000 mon.a (mon.0) 2133 : audit 2021-03-15T22:33:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:01 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:33:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:01 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:33:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:01 smithi125 conmon[56739]: T22:33:01.114699+0000 mon.a (mon.0) 2134 : 2021-03-15T22:33:01.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:01 smithi125 conmon[56739]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:33:01.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:01.908 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:01.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:01.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:02.010 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:02.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:02.034 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:02.034 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:02.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:02.157 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:02.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:02.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:02.260 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:02.286 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:02.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:02.288 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:02.310 INFO:tasks.rados.rados.0.smithi076.stdout:3116: done (3 left) 2021-03-15T22:33:02.310 INFO:tasks.rados.rados.0.smithi076.stdout:3117: done (2 left) 2021-03-15T22:33:02.310 INFO:tasks.rados.rados.0.smithi076.stdout:3118: done (1 left) 2021-03-15T22:33:02.310 INFO:tasks.rados.rados.0.smithi076.stdout:3119: done (0 left) 2021-03-15T22:33:02.311 INFO:tasks.rados.rados.0.smithi076.stdout:3120: snap_remove snap 343 2021-03-15T22:33:02.390 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:02.411 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:02.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:02.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:02.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:02.541 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:02.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:02.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:02.643 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:02.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:02.670 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:02.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:02.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:02 smithi076 conmon[56514]: cluster 2021-03-15T22:33:01.183879+0000 mgr.y 2021-03-15T22:33:02.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:02 smithi076 conmon[56514]: (mgr.14144) 966 : cluster [DBG] pgmap v1390: 148 pgs: 148 active+clean; 501 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2021-03-15T22:33:02.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:02 smithi076 conmon[56514]: cluster 2021-03-15T22:33:01.301968+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e831: 8 total, 8 up, 7 in 2021-03-15T22:33:02.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:02 smithi076 conmon[56514]: debug 2021-03-15T22:33:02.396+0000 7f0537c88700 -1 mon.a@0(leader).osd e832 definitely_dead 0 2021-03-15T22:33:02.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:02 smithi076 conmon[62156]: cluster 2021-03-15T22:33 2021-03-15T22:33:02.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:02 smithi076 conmon[62156]: :01.183879+0000 mgr.y (mgr.14144) 966 : cluster [DBG] pgmap v1390: 148 pgs: 148 active+clean; 501 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2021-03-15T22:33:02.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:02 smithi076 conmon[62156]: cluster 2021-03-15T22:33:01.301968+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e831: 8 total, 8 up, 7 in 2021-03-15T22:33:02.773 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:02.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:02 smithi125 conmon[56739]: cluster 2021-03-15T22:33:01.183879+0000 mgr.y (mgr.14144) 966 : 2021-03-15T22:33:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:02 smithi125 conmon[56739]: cluster [DBG] pgmap v1390: 148 pgs: 148 active+clean; 501 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 1.4 MiB/s wr, 7 op/s 2021-03-15T22:33:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:02 smithi125 conmon[56739]: cluster 2021-03-15T22:33:01.301968+0000 mon.a (mon.0) 2135 : cluster [DBG] osdmap e831: 8 total, 8 up, 7 in 2021-03-15T22:33:02.796 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:02.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:02.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:02.898 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:02.920 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:02.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:02.922 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:03.023 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:03.049 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:03.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:03.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:03.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:03.178 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:03.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:03.180 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:03.281 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:03.308 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:33:03.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:03.310 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:03.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:03.325 INFO:tasks.rados.rados.0.smithi076.stdout:3120: done (0 left) 2021-03-15T22:33:03.326 INFO:tasks.rados.rados.0.smithi076.stdout:3121: write (excl) oid 35 current snap is 364 2021-03-15T22:33:03.326 INFO:tasks.rados.rados.0.smithi076.stdout:3121: seq_num 1069 ranges {766683=730601,1922192=583372,3156591=676370} 2021-03-15T22:33:03.330 INFO:teuthology.orchestra.run.smithi076.stderr:marked in osd.6. 2021-03-15T22:33:03.332 INFO:tasks.rados.rados.0.smithi076.stdout:3121: writing smithi07694878-35 from 766683 to 1497284 tid 1 2021-03-15T22:33:03.337 INFO:tasks.rados.rados.0.smithi076.stdout:3121: writing smithi07694878-35 from 1922192 to 2505564 tid 2 2021-03-15T22:33:03.346 INFO:tasks.rados.rados.0.smithi076.stdout:3121: writing smithi07694878-35 from 3156591 to 3832961 tid 3 2021-03-15T22:33:03.346 INFO:tasks.rados.rados.0.smithi076.stdout:3122: rollback oid 9 current snap is 364 2021-03-15T22:33:03.347 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 9 to 356 2021-03-15T22:33:03.347 INFO:tasks.rados.rados.0.smithi076.stdout:3123: snap_remove snap 361 2021-03-15T22:33:03.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:03.425 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:33:03.425 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:33:03.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:33:03.426 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:33:03.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.426 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6381 7.e 7:7ac381e0:::smithi07694878-24:head [write 521248~563025] snapc 221=[221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f8,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1c5,1a1,17a,176] RETRY=1 ondisk+retry+write+known_if_redirected e682)", 2021-03-15T22:33:03.426 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:31.248957+0000", 2021-03-15T22:33:03.427 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 152.17042404899999, 2021-03-15T22:33:03.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.203159267, 2021-03-15T22:33:03.427 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.427 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:03.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:03.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:03.428 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:03.429 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6381 2021-03-15T22:33:03.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.429 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.248957+0000", 2021-03-15T22:33:03.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.248957+0000", 2021-03-15T22:33:03.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029079399999999998 2021-03-15T22:33:03.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.249248+0000", 2021-03-15T22:33:03.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7869999999999999e-06 2021-03-15T22:33:03.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.249252+0000", 2021-03-15T22:33:03.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024823800000000002 2021-03-15T22:33:03.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.251734+0000", 2021-03-15T22:33:03.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3850000000000002e-06 2021-03-15T22:33:03.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.251743+0000", 2021-03-15T22:33:03.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013638999999999999 2021-03-15T22:33:03.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.438 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253107+0000", 2021-03-15T22:33:03.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0289999999999994e-06 2021-03-15T22:33:03.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:03.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253116+0000", 2021-03-15T22:33:03.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0045301959999999 2021-03-15T22:33:03.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257646+0000", 2021-03-15T22:33:03.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134308 2021-03-15T22:33:03.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for missing object", 2021-03-15T22:33:03.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257780+0000", 2021-03-15T22:33:03.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036736790999999998 2021-03-15T22:33:03.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294517+0000", 2021-03-15T22:33:03.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000130782 2021-03-15T22:33:03.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:03.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294648+0000", 2021-03-15T22:33:03.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15022935500000001 2021-03-15T22:33:03.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.444877+0000", 2021-03-15T22:33:03.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011121299999999999 2021-03-15T22:33:03.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.444988+0000", 2021-03-15T22:33:03.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019712199999999999 2021-03-15T22:33:03.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 0,5", 2021-03-15T22:33:03.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.445185+0000", 2021-03-15T22:33:03.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066635430000000001 2021-03-15T22:33:03.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451849+0000", 2021-03-15T22:33:03.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3575999999999997e-05 2021-03-15T22:33:03.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:03.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451893+0000", 2021-03-15T22:33:03.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4709999999999999e-05 2021-03-15T22:33:03.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451917+0000", 2021-03-15T22:33:03.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7205000000000005e-05 2021-03-15T22:33:03.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:03.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451985+0000", 2021-03-15T22:33:03.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013219100000000001 2021-03-15T22:33:03.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452117+0000", 2021-03-15T22:33:03.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.203159267 2021-03-15T22:33:03.453 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.454 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.454 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.454 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6382 7.e 7:7ac381e0:::smithi07694878-24:head [write 1809501~596300] snapc 221=[221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f8,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1c5,1a1,17a,176] RETRY=1 ondisk+retry+write+known_if_redirected e682)", 2021-03-15T22:33:03.455 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:31.251755+0000", 2021-03-15T22:33:03.455 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 152.16762655100001, 2021-03-15T22:33:03.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2005434639999999, 2021-03-15T22:33:03.455 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.456 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:03.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:03.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:03.456 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:03.456 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6382 2021-03-15T22:33:03.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.457 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.251755+0000", 2021-03-15T22:33:03.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.251755+0000", 2021-03-15T22:33:03.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029779900000000001 2021-03-15T22:33:03.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252053+0000", 2021-03-15T22:33:03.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.745e-06 2021-03-15T22:33:03.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252055+0000", 2021-03-15T22:33:03.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0090000000000001e-06 2021-03-15T22:33:03.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252063+0000", 2021-03-15T22:33:03.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.49e-06 2021-03-15T22:33:03.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.463 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252066+0000", 2021-03-15T22:33:03.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001074237 2021-03-15T22:33:03.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253140+0000", 2021-03-15T22:33:03.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8170000000000004e-06 2021-03-15T22:33:03.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:03.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253146+0000", 2021-03-15T22:33:03.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0046597100000001 2021-03-15T22:33:03.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257806+0000", 2021-03-15T22:33:03.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6441999999999999e-05 2021-03-15T22:33:03.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for missing object", 2021-03-15T22:33:03.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257862+0000", 2021-03-15T22:33:03.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036841170999999999 2021-03-15T22:33:03.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294703+0000", 2021-03-15T22:33:03.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1587999999999994e-05 2021-03-15T22:33:03.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:03.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294765+0000", 2021-03-15T22:33:03.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15190767999999999 2021-03-15T22:33:03.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.446673+0000", 2021-03-15T22:33:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000120914 2021-03-15T22:33:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.446794+0000", 2021-03-15T22:33:03.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021026000000000001 2021-03-15T22:33:03.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 0,5", 2021-03-15T22:33:03.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.447004+0000", 2021-03-15T22:33:03.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048782030000000002 2021-03-15T22:33:03.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451882+0000", 2021-03-15T22:33:03.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028142400000000001 2021-03-15T22:33:03.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452163+0000", 2021-03-15T22:33:03.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0363999999999999e-05 2021-03-15T22:33:03.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:03.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452174+0000", 2021-03-15T22:33:03.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6031000000000001e-05 2021-03-15T22:33:03.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:03.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452210+0000", 2021-03-15T22:33:03.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8579999999999996e-05 2021-03-15T22:33:03.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452298+0000", 2021-03-15T22:33:03.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2005434639999999 2021-03-15T22:33:03.480 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.480 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.480 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.481 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6383 7.e 7:7ac381e0:::smithi07694878-24:head [setxattr _header (58) in=7b,truncate 2521248] snapc 221=[221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f8,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1c5,1a1,17a,176] RETRY=1 ondisk+retry+write+known_if_redirected e682)", 2021-03-15T22:33:03.481 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:31.252070+0000", 2021-03-15T22:33:03.481 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 152.167311082, 2021-03-15T22:33:03.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.200499961, 2021-03-15T22:33:03.482 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.482 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:03.482 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:03.482 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:03.482 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:03.483 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6383 2021-03-15T22:33:03.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.483 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252070+0000", 2021-03-15T22:33:03.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252070+0000", 2021-03-15T22:33:03.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.331e-06 2021-03-15T22:33:03.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252072+0000", 2021-03-15T22:33:03.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5899999999999995e-07 2021-03-15T22:33:03.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252073+0000", 2021-03-15T22:33:03.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2610000000000002e-06 2021-03-15T22:33:03.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252076+0000", 2021-03-15T22:33:03.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9029999999999997e-06 2021-03-15T22:33:03.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252080+0000", 2021-03-15T22:33:03.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010880149999999999 2021-03-15T22:33:03.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253168+0000", 2021-03-15T22:33:03.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8159999999999999e-06 2021-03-15T22:33:03.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:03.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253174+0000", 2021-03-15T22:33:03.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0047103100000001 2021-03-15T22:33:03.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257884+0000", 2021-03-15T22:33:03.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3068000000000001e-05 2021-03-15T22:33:03.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for missing object", 2021-03-15T22:33:03.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257937+0000", 2021-03-15T22:33:03.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036854445999999999 2021-03-15T22:33:03.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294791+0000", 2021-03-15T22:33:03.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147114 2021-03-15T22:33:03.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:03.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294938+0000", 2021-03-15T22:33:03.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.156267234 2021-03-15T22:33:03.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451206+0000", 2021-03-15T22:33:03.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6873999999999995e-05 2021-03-15T22:33:03.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451293+0000", 2021-03-15T22:33:03.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014486499999999999 2021-03-15T22:33:03.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 0,5", 2021-03-15T22:33:03.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451437+0000", 2021-03-15T22:33:03.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00087174500000000005 2021-03-15T22:33:03.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:03.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452309+0000", 2021-03-15T22:33:03.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8932e-05 2021-03-15T22:33:03.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452338+0000", 2021-03-15T22:33:03.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2397999999999997e-05 2021-03-15T22:33:03.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.502 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.502 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452370+0000", 2021-03-15T22:33:03.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9596000000000003e-05 2021-03-15T22:33:03.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:03.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452430+0000", 2021-03-15T22:33:03.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014029400000000001 2021-03-15T22:33:03.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452570+0000", 2021-03-15T22:33:03.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.200499961 2021-03-15T22:33:03.505 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.505 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.505 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.506 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6384 7.e 7:7ac381e0:::smithi07694878-24:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e682)", 2021-03-15T22:33:03.506 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:31.252084+0000", 2021-03-15T22:33:03.506 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 152.16729718299999, 2021-03-15T22:33:03.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.200996113, 2021-03-15T22:33:03.507 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.507 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:03.507 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:03.507 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:03.508 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:03.508 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6384 2021-03-15T22:33:03.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.508 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252084+0000", 2021-03-15T22:33:03.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252084+0000", 2021-03-15T22:33:03.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9100000000000002e-07 2021-03-15T22:33:03.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252085+0000", 2021-03-15T22:33:03.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3430000000000001e-06 2021-03-15T22:33:03.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252087+0000", 2021-03-15T22:33:03.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2700000000000004e-06 2021-03-15T22:33:03.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252092+0000", 2021-03-15T22:33:03.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.298e-06 2021-03-15T22:33:03.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252095+0000", 2021-03-15T22:33:03.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010992179999999999 2021-03-15T22:33:03.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253194+0000", 2021-03-15T22:33:03.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0279999999999997e-06 2021-03-15T22:33:03.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:03.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253200+0000", 2021-03-15T22:33:03.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0047584869999999 2021-03-15T22:33:03.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257959+0000", 2021-03-15T22:33:03.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.897e-05 2021-03-15T22:33:03.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for missing object", 2021-03-15T22:33:03.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.258008+0000", 2021-03-15T22:33:03.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037164928 2021-03-15T22:33:03.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.295173+0000", 2021-03-15T22:33:03.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8801999999999994e-05 2021-03-15T22:33:03.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:03.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.295272+0000", 2021-03-15T22:33:03.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15646575800000001 2021-03-15T22:33:03.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451737+0000", 2021-03-15T22:33:03.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2378000000000004e-05 2021-03-15T22:33:03.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:03.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451810+0000", 2021-03-15T22:33:03.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000797925 2021-03-15T22:33:03.524 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.524 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.525 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452608+0000", 2021-03-15T22:33:03.525 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000103439 2021-03-15T22:33:03.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452711+0000", 2021-03-15T22:33:03.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00036937800000000001 2021-03-15T22:33:03.526 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453080+0000", 2021-03-15T22:33:03.527 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.200996113 2021-03-15T22:33:03.527 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.528 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.528 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.528 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:2177 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94751391775488 gen 12] snapc 0=[] ondisk+write+known_if_redirected e682)", 2021-03-15T22:33:03.529 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.138277+0000", 2021-03-15T22:33:03.529 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 151.28110415, 2021-03-15T22:33:03.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.13344044399999999, 2021-03-15T22:33:03.529 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.529 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:03.530 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:03.530 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:33:03.530 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:33:03.530 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2177 2021-03-15T22:33:03.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.531 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138277+0000", 2021-03-15T22:33:03.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138277+0000", 2021-03-15T22:33:03.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5209999999999999e-06 2021-03-15T22:33:03.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138279+0000", 2021-03-15T22:33:03.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0579999999999999e-06 2021-03-15T22:33:03.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138283+0000", 2021-03-15T22:33:03.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1531e-05 2021-03-15T22:33:03.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138294+0000", 2021-03-15T22:33:03.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.02e-06 2021-03-15T22:33:03.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138303+0000", 2021-03-15T22:33:03.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2645999999999995e-05 2021-03-15T22:33:03.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138376+0000", 2021-03-15T22:33:03.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1537e-05 2021-03-15T22:33:03.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:03.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138388+0000", 2021-03-15T22:33:03.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12075796699999999 2021-03-15T22:33:03.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.259146+0000", 2021-03-15T22:33:03.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017358400000000001 2021-03-15T22:33:03.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.259319+0000", 2021-03-15T22:33:03.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012398579999999999 2021-03-15T22:33:03.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.271718+0000", 2021-03-15T22:33:03.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.13344044399999999 2021-03-15T22:33:03.543 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.543 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.544 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.544 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:2179 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94751391775488 gen 13] snapc 0=[] ondisk+write+known_if_redirected e682)", 2021-03-15T22:33:03.544 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.138324+0000", 2021-03-15T22:33:03.545 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 151.281057885, 2021-03-15T22:33:03.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.133677823, 2021-03-15T22:33:03.545 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.545 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:03.546 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:03.546 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:33:03.546 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:33:03.546 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2179 2021-03-15T22:33:03.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.547 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138324+0000", 2021-03-15T22:33:03.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138324+0000", 2021-03-15T22:33:03.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.001e-06 2021-03-15T22:33:03.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138325+0000", 2021-03-15T22:33:03.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.359e-06 2021-03-15T22:33:03.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138326+0000", 2021-03-15T22:33:03.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7190000000000001e-06 2021-03-15T22:33:03.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138334+0000", 2021-03-15T22:33:03.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1110000000000001e-06 2021-03-15T22:33:03.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138338+0000", 2021-03-15T22:33:03.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6509999999999998e-05 2021-03-15T22:33:03.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138414+0000", 2021-03-15T22:33:03.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0434999999999999e-05 2021-03-15T22:33:03.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:03.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138425+0000", 2021-03-15T22:33:03.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.121430683 2021-03-15T22:33:03.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.259855+0000", 2021-03-15T22:33:03.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3189000000000003e-05 2021-03-15T22:33:03.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.259899+0000", 2021-03-15T22:33:03.557 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012102816000000001 2021-03-15T22:33:03.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.272001+0000", 2021-03-15T22:33:03.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.133677823 2021-03-15T22:33:03.559 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.559 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.559 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.560 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGPush(4.6 683/682 [PushOp(4:6192870f:::obj_delete_at_hint.0000000053:head, version: 555'22, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6192870f:::obj_delete_at_hint.0000000053:head@555'22, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-15T22:33:03.560 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.282830+0000", 2021-03-15T22:33:03.560 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 151.136551387, 2021-03-15T22:33:03.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075199040999999994, 2021-03-15T22:33:03.560 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.561 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:03.561 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282830+0000", 2021-03-15T22:33:03.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.563 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282830+0000", 2021-03-15T22:33:03.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1109999999999999e-06 2021-03-15T22:33:03.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.564 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.564 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282833+0000", 2021-03-15T22:33:03.564 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0159999999999999e-05 2021-03-15T22:33:03.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282843+0000", 2021-03-15T22:33:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6823000000000001e-05 2021-03-15T22:33:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282860+0000", 2021-03-15T22:33:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8099999999999992e-06 2021-03-15T22:33:03.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.567 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282870+0000", 2021-03-15T22:33:03.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074966989999999997 2021-03-15T22:33:03.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.357837+0000", 2021-03-15T22:33:03.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2856e-05 2021-03-15T22:33:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.357850+0000", 2021-03-15T22:33:03.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017929099999999999 2021-03-15T22:33:03.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.570 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.358029+0000", 2021-03-15T22:33:03.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075199040999999994 2021-03-15T22:33:03.571 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.571 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.571 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.572 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGPush(4.6 683/682 [PushOp(4:63d917ed:::obj_delete_at_hint.0000000065:head, version: 555'24, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:63d917ed:::obj_delete_at_hint.0000000065:head@555'24, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-15T22:33:03.572 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.368712+0000", 2021-03-15T22:33:03.572 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 151.05066985799999, 2021-03-15T22:33:03.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083794849000000005, 2021-03-15T22:33:03.573 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.573 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:03.573 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368712+0000", 2021-03-15T22:33:03.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368712+0000", 2021-03-15T22:33:03.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.98e-06 2021-03-15T22:33:03.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.576 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368716+0000", 2021-03-15T22:33:03.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2875e-05 2021-03-15T22:33:03.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.577 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368729+0000", 2021-03-15T22:33:03.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1965000000000001e-05 2021-03-15T22:33:03.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.578 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.578 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368750+0000", 2021-03-15T22:33:03.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2337999999999999e-05 2021-03-15T22:33:03.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368763+0000", 2021-03-15T22:33:03.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083630726000000002 2021-03-15T22:33:03.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452394+0000", 2021-03-15T22:33:03.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3420000000000008e-06 2021-03-15T22:33:03.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.581 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.581 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452402+0000", 2021-03-15T22:33:03.582 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104623 2021-03-15T22:33:03.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.582 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.582 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.582 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452506+0000", 2021-03-15T22:33:03.583 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083794849000000005 2021-03-15T22:33:03.583 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.583 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.583 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.584 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGPush(4.6 683/682 [PushOp(4:64735f45:::obj_delete_at_hint.0000000083:head, version: 555'26, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:64735f45:::obj_delete_at_hint.0000000083:head@555'26, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-15T22:33:03.584 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.453341+0000", 2021-03-15T22:33:03.584 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 150.966040327, 2021-03-15T22:33:03.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077331792999999996, 2021-03-15T22:33:03.585 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.585 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:03.585 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.586 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.586 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453341+0000", 2021-03-15T22:33:03.586 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.587 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453341+0000", 2021-03-15T22:33:03.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1270000000000001e-06 2021-03-15T22:33:03.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453343+0000", 2021-03-15T22:33:03.588 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8979999999999996e-06 2021-03-15T22:33:03.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.589 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.589 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453351+0000", 2021-03-15T22:33:03.589 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4432e-05 2021-03-15T22:33:03.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.590 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.590 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.590 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453366+0000", 2021-03-15T22:33:03.591 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5170000000000002e-06 2021-03-15T22:33:03.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.591 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.592 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453373+0000", 2021-03-15T22:33:03.592 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077141928999999998 2021-03-15T22:33:03.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.530515+0000", 2021-03-15T22:33:03.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2335e-05 2021-03-15T22:33:03.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.530527+0000", 2021-03-15T22:33:03.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014555500000000001 2021-03-15T22:33:03.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.530673+0000", 2021-03-15T22:33:03.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077331792999999996 2021-03-15T22:33:03.595 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.595 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.596 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.596 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGPush(4.6 683/682 [PushOp(4:624175ac:::obj_delete_at_hint.0000000091:head, version: 555'28, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:624175ac:::obj_delete_at_hint.0000000091:head@555'28, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-15T22:33:03.597 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.531733+0000", 2021-03-15T22:33:03.597 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 150.88764820200001, 2021-03-15T22:33:03.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.085164504000000002, 2021-03-15T22:33:03.597 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.597 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:03.598 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531733+0000", 2021-03-15T22:33:03.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.599 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.599 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531733+0000", 2021-03-15T22:33:03.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5429999999999998e-06 2021-03-15T22:33:03.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531737+0000", 2021-03-15T22:33:03.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0711e-05 2021-03-15T22:33:03.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.602 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531748+0000", 2021-03-15T22:33:03.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0208e-05 2021-03-15T22:33:03.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531768+0000", 2021-03-15T22:33:03.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2381999999999999e-05 2021-03-15T22:33:03.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531780+0000", 2021-03-15T22:33:03.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.085010464999999993 2021-03-15T22:33:03.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.616791+0000", 2021-03-15T22:33:03.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2209999999999997e-06 2021-03-15T22:33:03.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.616799+0000", 2021-03-15T22:33:03.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8974000000000005e-05 2021-03-15T22:33:03.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.616898+0000", 2021-03-15T22:33:03.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.085164504000000002 2021-03-15T22:33:03.608 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.608 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.608 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.609 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:845 7.5 e772/685 7:ac84e656:::smithi07694878-31:1f0 v 772'640, mlcod=770'636)", 2021-03-15T22:33:03.609 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:02.252138+0000", 2021-03-15T22:33:03.609 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 61.167243775999999, 2021-03-15T22:33:03.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075996994999999998, 2021-03-15T22:33:03.609 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.610 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:03.610 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.611 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.252138+0000", 2021-03-15T22:33:03.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.252138+0000", 2021-03-15T22:33:03.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.019e-06 2021-03-15T22:33:03.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.613 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.252142+0000", 2021-03-15T22:33:03.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4530000000000004e-06 2021-03-15T22:33:03.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.252146+0000", 2021-03-15T22:33:03.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.072793821999999994 2021-03-15T22:33:03.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.615 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.615 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.324940+0000", 2021-03-15T22:33:03.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8179999999999998e-06 2021-03-15T22:33:03.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.616 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.616 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.324947+0000", 2021-03-15T22:33:03.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001865684 2021-03-15T22:33:03.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.326813+0000", 2021-03-15T22:33:03.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3147999999999999e-05 2021-03-15T22:33:03.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.326836+0000", 2021-03-15T22:33:03.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012622849999999999 2021-03-15T22:33:03.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.623 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:03.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328098+0000", 2021-03-15T22:33:03.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6766e-05 2021-03-15T22:33:03.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.624 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328135+0000", 2021-03-15T22:33:03.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075996994999999998 2021-03-15T22:33:03.625 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.625 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.625 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.626 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.2.0:161 7.8 e772/681 ondisk, result = 0)", 2021-03-15T22:33:03.626 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:02.254556+0000", 2021-03-15T22:33:03.626 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 61.164825186000002, 2021-03-15T22:33:03.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071612197000000002, 2021-03-15T22:33:03.627 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.627 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:03.627 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.627 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.628 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.254556+0000", 2021-03-15T22:33:03.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.628 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.254556+0000", 2021-03-15T22:33:03.629 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-06 2021-03-15T22:33:03.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.254560+0000", 2021-03-15T22:33:03.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9770000000000002e-06 2021-03-15T22:33:03.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.254564+0000", 2021-03-15T22:33:03.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068971258999999993 2021-03-15T22:33:03.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.632 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.632 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.323536+0000", 2021-03-15T22:33:03.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1450999999999999e-05 2021-03-15T22:33:03.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.633 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.633 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.323547+0000", 2021-03-15T22:33:03.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026004259999999999 2021-03-15T22:33:03.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.326147+0000", 2021-03-15T22:33:03.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5440000000000004e-06 2021-03-15T22:33:03.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.326157+0000", 2021-03-15T22:33:03.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.154e-05 2021-03-15T22:33:03.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.326169+0000", 2021-03-15T22:33:03.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071612197000000002 2021-03-15T22:33:03.637 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.637 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.637 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.638 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.3.0:1988 7.a e790/684 7:5bc7f316:::smithi07694878-38:261 v 790'1367, mlcod=790'1363)", 2021-03-15T22:33:03.638 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:20.354804+0000", 2021-03-15T22:33:03.638 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 43.064577423999999, 2021-03-15T22:33:03.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042477930999999997, 2021-03-15T22:33:03.639 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.639 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:03.639 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.639 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354804+0000", 2021-03-15T22:33:03.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.640 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.641 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.641 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354804+0000", 2021-03-15T22:33:03.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4660000000000002e-06 2021-03-15T22:33:03.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354812+0000", 2021-03-15T22:33:03.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5149999999999997e-06 2021-03-15T22:33:03.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.643 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354817+0000", 2021-03-15T22:33:03.643 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2300999999999999e-05 2021-03-15T22:33:03.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354839+0000", 2021-03-15T22:33:03.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4779999999999997e-06 2021-03-15T22:33:03.645 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.645 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.645 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.645 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354849+0000", 2021-03-15T22:33:03.645 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3566000000000002e-05 2021-03-15T22:33:03.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.646 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.646 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354902+0000", 2021-03-15T22:33:03.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0645999999999997e-05 2021-03-15T22:33:03.647 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.647 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.647 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354943+0000", 2021-03-15T22:33:03.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042287531000000003 2021-03-15T22:33:03.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.648 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:03.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.397231+0000", 2021-03-15T22:33:03.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1428000000000003e-05 2021-03-15T22:33:03.649 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.397282+0000", 2021-03-15T22:33:03.650 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042477930999999997 2021-03-15T22:33:03.650 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.650 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.650 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.651 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7840 7.8 7:1822d391:::smithi07694878-47:head [append 0~0] snapc 2b6=[2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,293,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e830)", 2021-03-15T22:33:03.651 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.384447+0000", 2021-03-15T22:33:03.652 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.0349343719999999, 2021-03-15T22:33:03.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043748464000000001, 2021-03-15T22:33:03.652 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.652 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:03.652 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:03.653 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:03.653 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:03.653 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7840 2021-03-15T22:33:03.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.653 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.654 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384447+0000", 2021-03-15T22:33:03.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384447+0000", 2021-03-15T22:33:03.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0560000000000001e-06 2021-03-15T22:33:03.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384453+0000", 2021-03-15T22:33:03.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.609e-06 2021-03-15T22:33:03.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384459+0000", 2021-03-15T22:33:03.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6245999999999999e-05 2021-03-15T22:33:03.658 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.658 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384485+0000", 2021-03-15T22:33:03.659 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7729999999999994e-06 2021-03-15T22:33:03.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.659 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.660 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384495+0000", 2021-03-15T22:33:03.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8068e-05 2021-03-15T22:33:03.660 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.660 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.660 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.661 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384553+0000", 2021-03-15T22:33:03.661 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6229000000000004e-05 2021-03-15T22:33:03.661 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384639+0000", 2021-03-15T22:33:03.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022088199999999999 2021-03-15T22:33:03.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:33:03.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384860+0000", 2021-03-15T22:33:03.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018978619999999999 2021-03-15T22:33:03.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.664 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.386758+0000", 2021-03-15T22:33:03.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8676e-05 2021-03-15T22:33:03.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.386797+0000", 2021-03-15T22:33:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041256679999999997 2021-03-15T22:33:03.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.666 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:03.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.428053+0000", 2021-03-15T22:33:03.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4635000000000005e-05 2021-03-15T22:33:03.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:03.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.428138+0000", 2021-03-15T22:33:03.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7747999999999997e-05 2021-03-15T22:33:03.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.428196+0000", 2021-03-15T22:33:03.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043748464000000001 2021-03-15T22:33:03.669 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.669 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.669 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.670 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7841 7.8 7:1822d391:::smithi07694878-47:head [rollback 2b5] snapc 2b6=[2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,293,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e830)", 2021-03-15T22:33:03.670 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.384512+0000", 2021-03-15T22:33:03.670 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.0348699880000001, 2021-03-15T22:33:03.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.055918190999999999, 2021-03-15T22:33:03.670 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.671 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:03.671 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:03.671 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:03.671 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:03.672 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7841 2021-03-15T22:33:03.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.672 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.673 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.673 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384512+0000", 2021-03-15T22:33:03.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.673 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.673 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384512+0000", 2021-03-15T22:33:03.674 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8169999999999999e-06 2021-03-15T22:33:03.674 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384513+0000", 2021-03-15T22:33:03.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6920000000000001e-06 2021-03-15T22:33:03.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384515+0000", 2021-03-15T22:33:03.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6319999999999996e-06 2021-03-15T22:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384521+0000", 2021-03-15T22:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8739999999999999e-06 2021-03-15T22:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384524+0000", 2021-03-15T22:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010356950000000001 2021-03-15T22:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.385559+0000", 2021-03-15T22:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010356700000000001 2021-03-15T22:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.680 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.385663+0000", 2021-03-15T22:33:03.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017394099999999999 2021-03-15T22:33:03.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:33:03.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.385837+0000", 2021-03-15T22:33:03.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001173363 2021-03-15T22:33:03.682 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.683 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.683 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.387010+0000", 2021-03-15T22:33:03.683 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3553999999999998e-05 2021-03-15T22:33:03.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.684 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.684 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.387034+0000", 2021-03-15T22:33:03.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053285379000000001 2021-03-15T22:33:03.684 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.685 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.685 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:03.685 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440319+0000", 2021-03-15T22:33:03.685 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3157000000000005e-05 2021-03-15T22:33:03.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:03.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440382+0000", 2021-03-15T22:33:03.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7519999999999999e-05 2021-03-15T22:33:03.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440430+0000", 2021-03-15T22:33:03.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.055918190999999999 2021-03-15T22:33:03.688 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.688 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.688 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.689 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7842 7.8 7:1822d391:::smithi07694878-47:head [append 0~0] snapc 2b6=[2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,293,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e830)", 2021-03-15T22:33:03.689 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.384542+0000", 2021-03-15T22:33:03.689 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.0348396910000002, 2021-03-15T22:33:03.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.055994378999999997, 2021-03-15T22:33:03.690 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.690 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:03.690 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:03.690 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:03.691 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:03.691 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7842 2021-03-15T22:33:03.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.691 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384542+0000", 2021-03-15T22:33:03.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384542+0000", 2021-03-15T22:33:03.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3289999999999999e-06 2021-03-15T22:33:03.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384543+0000", 2021-03-15T22:33:03.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1179999999999999e-06 2021-03-15T22:33:03.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.695 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384544+0000", 2021-03-15T22:33:03.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9259999999999997e-06 2021-03-15T22:33:03.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384550+0000", 2021-03-15T22:33:03.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.895e-06 2021-03-15T22:33:03.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384554+0000", 2021-03-15T22:33:03.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001725709 2021-03-15T22:33:03.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.386280+0000", 2021-03-15T22:33:03.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4659000000000006e-05 2021-03-15T22:33:03.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.386344+0000", 2021-03-15T22:33:03.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000121128 2021-03-15T22:33:03.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:33:03.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.386466+0000", 2021-03-15T22:33:03.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00072824599999999999 2021-03-15T22:33:03.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.387194+0000", 2021-03-15T22:33:03.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0037000000000001e-05 2021-03-15T22:33:03.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:03.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.387214+0000", 2021-03-15T22:33:03.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053224652999999997 2021-03-15T22:33:03.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:03.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440439+0000", 2021-03-15T22:33:03.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0268000000000001e-05 2021-03-15T22:33:03.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.705 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:03.705 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440489+0000", 2021-03-15T22:33:03.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7410999999999997e-05 2021-03-15T22:33:03.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.706 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440536+0000", 2021-03-15T22:33:03.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.055994378999999997 2021-03-15T22:33:03.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.707 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.707 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.708 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:7843 7.5 e830/685 7:ac84e656:::smithi07694878-31:head v 830'746, mlcod=827'743)", 2021-03-15T22:33:03.708 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.392156+0000", 2021-03-15T22:33:03.708 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.027225031, 2021-03-15T22:33:03.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048256610999999998, 2021-03-15T22:33:03.709 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.709 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:03.709 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.392156+0000", 2021-03-15T22:33:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.392156+0000", 2021-03-15T22:33:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00088703899999999995 2021-03-15T22:33:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393044+0000", 2021-03-15T22:33:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8550000000000002e-06 2021-03-15T22:33:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393046+0000", 2021-03-15T22:33:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8155000000000002e-05 2021-03-15T22:33:03.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393065+0000", 2021-03-15T22:33:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5960000000000003e-06 2021-03-15T22:33:03.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393070+0000", 2021-03-15T22:33:03.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0427999999999999e-05 2021-03-15T22:33:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393121+0000", 2021-03-15T22:33:03.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4057999999999999e-05 2021-03-15T22:33:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393145+0000", 2021-03-15T22:33:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047235906000000001 2021-03-15T22:33:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:03.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440381+0000", 2021-03-15T22:33:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2573999999999997e-05 2021-03-15T22:33:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440413+0000", 2021-03-15T22:33:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048256610999999998 2021-03-15T22:33:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.720 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:7844 7.5 e830/685 7:ac84e656:::smithi07694878-31:head v 830'747, mlcod=827'743)", 2021-03-15T22:33:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.399774+0000", 2021-03-15T22:33:03.721 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.0196078740000001, 2021-03-15T22:33:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040654153999999998, 2021-03-15T22:33:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:03.722 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.399774+0000", 2021-03-15T22:33:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.399774+0000", 2021-03-15T22:33:03.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001083351 2021-03-15T22:33:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400857+0000", 2021-03-15T22:33:03.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8080000000000001e-06 2021-03-15T22:33:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400863+0000", 2021-03-15T22:33:03.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3723000000000001e-05 2021-03-15T22:33:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400877+0000", 2021-03-15T22:33:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6489999999999996e-06 2021-03-15T22:33:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400882+0000", 2021-03-15T22:33:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1548000000000003e-05 2021-03-15T22:33:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400924+0000", 2021-03-15T22:33:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4221e-05 2021-03-15T22:33:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400938+0000", 2021-03-15T22:33:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039478164000000003 2021-03-15T22:33:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440416+0000", 2021-03-15T22:33:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.169e-05 2021-03-15T22:33:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440428+0000", 2021-03-15T22:33:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040654153999999998 2021-03-15T22:33:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.733 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.734 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "pg_update_log_missing(7.c epoch 830/740 rep_tid 2224 entries 830'1139 (0'0) error 7:3bc1c17e:::smithi07694878-8:head by client.24517.0:7848 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 826'1136)", 2021-03-15T22:33:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.406310+0000", 2021-03-15T22:33:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.0130710249999999, 2021-03-15T22:33:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034274735000000001, 2021-03-15T22:33:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.735 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406310+0000", 2021-03-15T22:33:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406310+0000", 2021-03-15T22:33:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3359999999999997e-06 2021-03-15T22:33:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406315+0000", 2021-03-15T22:33:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3603e-05 2021-03-15T22:33:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406328+0000", 2021-03-15T22:33:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7805999999999999e-05 2021-03-15T22:33:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406346+0000", 2021-03-15T22:33:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6640000000000001e-06 2021-03-15T22:33:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406353+0000", 2021-03-15T22:33:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2898999999999995e-05 2021-03-15T22:33:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406416+0000", 2021-03-15T22:33:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034169427000000002 2021-03-15T22:33:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440585+0000", 2021-03-15T22:33:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034274735000000001 2021-03-15T22:33:03.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.745 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:7845 7.5 e830/685 7:ac84e656:::smithi07694878-31:head v 830'748, mlcod=827'743)", 2021-03-15T22:33:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.406726+0000", 2021-03-15T22:33:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 3.0126550719999998, 2021-03-15T22:33:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034169436999999997, 2021-03-15T22:33:03.746 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:03.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406726+0000", 2021-03-15T22:33:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:03.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406726+0000", 2021-03-15T22:33:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010318860000000001 2021-03-15T22:33:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.407758+0000", 2021-03-15T22:33:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9380000000000002e-06 2021-03-15T22:33:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.407762+0000", 2021-03-15T22:33:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4844999999999999e-05 2021-03-15T22:33:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.407777+0000", 2021-03-15T22:33:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6789999999999996e-06 2021-03-15T22:33:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.407782+0000", 2021-03-15T22:33:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8760999999999997e-05 2021-03-15T22:33:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.407821+0000", 2021-03-15T22:33:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7261999999999998e-05 2021-03-15T22:33:03.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.407838+0000", 2021-03-15T22:33:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.033030605999999997 2021-03-15T22:33:03.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440868+0000", 2021-03-15T22:33:03.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7460000000000001e-05 2021-03-15T22:33:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440896+0000", 2021-03-15T22:33:03.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034169436999999997 2021-03-15T22:33:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:03.759 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:03.759 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:33:03.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:03.760 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[56514]: cluster 2021-03-15T22:33:02.305859+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2021-03-15T22:33:03.760 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[56514]: audit 2021-03-15T22:33:02.397225+0000 mon.a (mon.0) 2137 : audit [DBG] from='client.? 172.21.15.76:0/2814726716' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-03-15T22:33:03.761 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: cluster 2021-03-15T22:33:03.761 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 2021-03-15T22:33:03.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 2021 2021-03-15T22:33:03.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: -03-15T22:33 2021-03-15T22:33:03.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: :02.305859+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2021-03-15T22:33:03.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: audit 2021-03-15T22:33:03.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 2021-03-15T22:33:03.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 2021 2021-03-15T22:33:03.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: - 2021-03-15T22:33:03.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 03 2021-03-15T22:33:03.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: - 2021-03-15T22:33:03.764 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 15 2021-03-15T22:33:03.764 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: T 2021-03-15T22:33:03.764 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 22 2021-03-15T22:33:03.764 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: : 2021-03-15T22:33:03.764 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 33 2021-03-15T22:33:03.765 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: : 2021-03-15T22:33:03.765 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 02 2021-03-15T22:33:03.765 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: . 2021-03-15T22:33:03.765 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 397225 2021-03-15T22:33:03.765 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: +0000 2021-03-15T22:33:03.766 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 2021-03-15T22:33:03.766 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: mon.a 2021-03-15T22:33:03.766 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: ( 2021-03-15T22:33:03.766 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: mon 2021-03-15T22:33:03.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: . 2021-03-15T22:33:03.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 0 2021-03-15T22:33:03.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: ) 2021-03-15T22:33:03.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 2137 2021-03-15T22:33:03.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: : 2021-03-15T22:33:03.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: audit 2021-03-15T22:33:03.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 2021-03-15T22:33:03.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: [DBG] 2021-03-15T22:33:03.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 2021-03-15T22:33:03.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/2814726716' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-03-15T22:33:03.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:03 smithi076 conmon[62156]: 2021-03-15T22:33:03.769 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:03.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:03.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:03 smithi125 conmon[56739]: cluster 2021-03-15T22:33:02.305859+0000 mon.a (mon.0) 2136 : cluster [DBG] osdmap e832: 8 total, 8 up, 7 in 2021-03-15T22:33:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:03 smithi125 conmon[56739]: audit 2021-03-15T22:33:02.397225+0000 mon.a (mon 2021-03-15T22:33:03.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:03 smithi125 conmon[56739]: .0) 2137 : audit [DBG] from='client.? 172.21.15.76:0/2814726716' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["6"]}]: dispatch 2021-03-15T22:33:03.871 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:03.897 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:03.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:03.898 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:03.999 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:04.023 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:04.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:04.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:04.126 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:04.152 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:04.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:04.153 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:04.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:04.280 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:04.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:04.320 INFO:tasks.rados.rados.0.smithi076.stdout:3123: done (2 left) 2021-03-15T22:33:04.321 INFO:tasks.rados.rados.0.smithi076.stdout:3124: copy_from oid 4 from oid 2 current snap is 364 2021-03-15T22:33:04.321 INFO:tasks.rados.rados.0.smithi076.stdout:3125: write (excl) oid 28 current snap is 364 2021-03-15T22:33:04.321 INFO:tasks.rados.rados.0.smithi076.stdout:3125: seq_num 1070 ranges {783490=405850,1980674=727030,3184014=487642} 2021-03-15T22:33:04.322 INFO:tasks.rados.rados.0.smithi076.stdout:3125: writing smithi07694878-28 from 783490 to 1189340 tid 1 2021-03-15T22:33:04.331 INFO:tasks.rados.rados.0.smithi076.stdout:3125: writing smithi07694878-28 from 1980674 to 2707704 tid 2 2021-03-15T22:33:04.337 INFO:tasks.rados.rados.0.smithi076.stdout:3125: writing smithi07694878-28 from 3184014 to 3671656 tid 3 2021-03-15T22:33:04.338 INFO:tasks.rados.rados.0.smithi076.stdout:3126: delete oid 41 current snap is 364 2021-03-15T22:33:04.338 INFO:tasks.rados.rados.0.smithi076.stdout:3122: finishing rollback tid 1 to smithi07694878-9 2021-03-15T22:33:04.338 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 1888 (ObjNum 261752240 snap 21896 seq_num 262072176) dirty dne 2021-03-15T22:33:04.339 INFO:tasks.rados.rados.0.smithi076.stdout:3121: finishing write tid 1 to smithi07694878-35 2021-03-15T22:33:04.339 INFO:tasks.rados.rados.0.smithi076.stdout:3121: finishing write tid 2 to smithi07694878-35 2021-03-15T22:33:04.339 INFO:tasks.rados.rados.0.smithi076.stdout:3121: finishing write tid 3 to smithi07694878-35 2021-03-15T22:33:04.339 INFO:tasks.rados.rados.0.smithi076.stdout:3121: finishing write tid 4 to smithi07694878-35 2021-03-15T22:33:04.339 INFO:tasks.rados.rados.0.smithi076.stdout:3121: finishing write tid 5 to smithi07694878-35 2021-03-15T22:33:04.340 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 2130 (ObjNum 1069 snap 364 seq_num 1069) dirty exists 2021-03-15T22:33:04.340 INFO:tasks.rados.rados.0.smithi076.stdout:3121: left oid 35 (ObjNum 1069 snap 364 seq_num 1069) 2021-03-15T22:33:04.343 INFO:tasks.rados.rados.0.smithi076.stdout:3125: finishing write tid 1 to smithi07694878-28 2021-03-15T22:33:04.350 INFO:tasks.rados.rados.0.smithi076.stdout:3125: finishing write tid 2 to smithi07694878-28 2021-03-15T22:33:04.352 INFO:tasks.rados.rados.0.smithi076.stdout:3125: finishing write tid 3 to smithi07694878-28 2021-03-15T22:33:04.352 INFO:tasks.rados.rados.0.smithi076.stdout:3125: finishing write tid 4 to smithi07694878-28 2021-03-15T22:33:04.353 INFO:tasks.rados.rados.0.smithi076.stdout:3125: finishing write tid 5 to smithi07694878-28 2021-03-15T22:33:04.353 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2757 (ObjNum 1070 snap 364 seq_num 1070) dirty exists 2021-03-15T22:33:04.353 INFO:tasks.rados.rados.0.smithi076.stdout:3125: left oid 28 (ObjNum 1070 snap 364 seq_num 1070) 2021-03-15T22:33:04.382 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:04.403 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:04.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:04.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:04.507 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:04.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[62156]: cluster 2021-03-15T22:33:03.184317+0000 mgr.y (mgr.14144) 967 : cluster [DBG] 2021-03-15T22:33:04.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[62156]: pgmap v1393: 148 pgs: 148 active+clean; 504 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2021-03-15T22:33:04.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:33:04.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[62156]: 33:03.311383+0000 mon.a (mon.0) 2138 : audit [INF] from='client.? 172.21.15.76:0/2814726716' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2021-03-15T22:33:04.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:04.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[62156]: -03-15T22:33:03.311437+0000 2021-03-15T22:33:04.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[62156]: mon.a (mon.0) 2139 : cluster [DBG] osdmap e833: 8 total, 8 up, 8 in 2021-03-15T22:33:04.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[62156]: 2021-03-15T22:33:04.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: cluster 2021-03-15T22:33:04.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 2021-03-15T22:33:04.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 2021 2021-03-15T22:33:04.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: - 2021-03-15T22:33:04.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 03 2021-03-15T22:33:04.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: - 2021-03-15T22:33:04.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 15 2021-03-15T22:33:04.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: T 2021-03-15T22:33:04.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 22 2021-03-15T22:33:04.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: : 2021-03-15T22:33:04.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 33 2021-03-15T22:33:04.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: : 2021-03-15T22:33:04.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 03 2021-03-15T22:33:04.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: . 2021-03-15T22:33:04.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 184317 2021-03-15T22:33:04.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: +0000 2021-03-15T22:33:04.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 2021-03-15T22:33:04.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: mgr.y 2021-03-15T22:33:04.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: ( 2021-03-15T22:33:04.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: mgr 2021-03-15T22:33:04.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: . 2021-03-15T22:33:04.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 14144 2021-03-15T22:33:04.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: ) 2021-03-15T22:33:04.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 967 : cluster [DBG] pgmap v1393: 148 pgs: 148 active+clean; 504 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2021-03-15T22:33:04.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: audit 2021 2021-03-15T22:33:04.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: -03-15T22:33:03.311383+0000 mon.a (mon. 2021-03-15T22:33:04.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: 0) 2138 : audit [INF] from='client.? 172.21.15.76:0/2814726716' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2021-03-15T22:33:04.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:33:04.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: -03-15T22:33:03.311437+0000 2021-03-15T22:33:04.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: mon.a (mon.0) 2139 : cluster [DBG] 2021-03-15T22:33:04.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:04 smithi076 conmon[56514]: osdmap e833: 8 total, 8 up, 8 in 2021-03-15T22:33:04.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:04.534 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:04.535 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:04.636 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:04.656 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:04.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:04.759 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:04.780 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:04.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:04.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: cluster 2021-03-15T22:33:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021 2021-03-15T22:33:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: - 2021-03-15T22:33:04.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 03 2021-03-15T22:33:04.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: - 2021-03-15T22:33:04.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 15 2021-03-15T22:33:04.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: T 2021-03-15T22:33:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 22 2021-03-15T22:33:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: : 2021-03-15T22:33:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 33 2021-03-15T22:33:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: : 2021-03-15T22:33:04.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 03 2021-03-15T22:33:04.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: . 2021-03-15T22:33:04.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 184317 2021-03-15T22:33:04.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: +0000 2021-03-15T22:33:04.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: mgr.y 2021-03-15T22:33:04.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: ( 2021-03-15T22:33:04.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: mgr 2021-03-15T22:33:04.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: . 2021-03-15T22:33:04.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 14144 2021-03-15T22:33:04.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: ) 2021-03-15T22:33:04.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 967 2021-03-15T22:33:04.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: : 2021-03-15T22:33:04.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: cluster 2021-03-15T22:33:04.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: [DBG] 2021-03-15T22:33:04.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: pgmap v1393: 148 pgs: 148 active+clean; 504 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.2 MiB/s wr, 12 op/s 2021-03-15T22:33:04.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: audit 2021-03-15T22:33:04.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021 2021-03-15T22:33:04.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: - 2021-03-15T22:33:04.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 03 2021-03-15T22:33:04.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: - 2021-03-15T22:33:04.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 15 2021-03-15T22:33:04.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: T 2021-03-15T22:33:04.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 22 2021-03-15T22:33:04.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: : 2021-03-15T22:33:04.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 33 2021-03-15T22:33:04.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: : 2021-03-15T22:33:04.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 03 2021-03-15T22:33:04.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: . 2021-03-15T22:33:04.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 311383 2021-03-15T22:33:04.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: +0000 2021-03-15T22:33:04.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: mon.a 2021-03-15T22:33:04.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: ( 2021-03-15T22:33:04.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: mon 2021-03-15T22:33:04.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: . 2021-03-15T22:33:04.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 0 2021-03-15T22:33:04.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: ) 2021-03-15T22:33:04.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2138 2021-03-15T22:33:04.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: : 2021-03-15T22:33:04.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: audit 2021-03-15T22:33:04.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: [INF] 2021-03-15T22:33:04.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2814726716' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["6"]}]': finished 2021-03-15T22:33:04.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: cluster 2021-03-15T22:33:04.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021 2021-03-15T22:33:04.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: - 2021-03-15T22:33:04.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 03 2021-03-15T22:33:04.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: - 2021-03-15T22:33:04.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 15 2021-03-15T22:33:04.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: T 2021-03-15T22:33:04.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 22 2021-03-15T22:33:04.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: : 2021-03-15T22:33:04.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 33 2021-03-15T22:33:04.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: : 2021-03-15T22:33:04.804 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 03 2021-03-15T22:33:04.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: . 2021-03-15T22:33:04.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 311437 2021-03-15T22:33:04.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: +0000 2021-03-15T22:33:04.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.805 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: mon.a 2021-03-15T22:33:04.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: ( 2021-03-15T22:33:04.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: mon 2021-03-15T22:33:04.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: . 2021-03-15T22:33:04.806 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 0 2021-03-15T22:33:04.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: ) 2021-03-15T22:33:04.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2139 2021-03-15T22:33:04.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: : 2021-03-15T22:33:04.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: cluster 2021-03-15T22:33:04.807 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: [DBG] 2021-03-15T22:33:04.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: osdmap e833: 8 total, 8 up, 8 in 2021-03-15T22:33:04.808 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:04 smithi125 conmon[56739]: 2021-03-15T22:33:04.812 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:33:04.883 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:04.897 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:04.905 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:04.906 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:04.926 INFO:tasks.thrashosds.thrasher:Added osd 6 2021-03-15T22:33:05.007 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:05.029 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:05.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:05.030 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:05.131 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:05.152 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:05.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:05.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:05.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:05.279 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:05.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:05.280 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:05.296 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:33:05.349 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[56514]: cluster 2021-03-15T22:33:03.915232 2021-03-15T22:33:05.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[56514]: +0000 osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:05.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[56514]: cluster 2021-03-15T22:33:03.916360+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:05.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[56514]: cluster 2021-03-15T22:33:04.312397+0000 mon.a (mon.0) 2140 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2021-03-15T22:33:05.350 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[56514]: audit 2021-03-15T22:33:05.054661+0000 mon.a (mon.0) 2141 : audit [INF] from='client.? 172.21.15.76:0/1022577655' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:33:05.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[56514]: cluster 2021-03-15T22:33:05.268952+0000 mon.a (mon.0) 2142 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:05.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[56514]: audit 2021-03-15T22:33:05.278425+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? 172.21.15.76:0/1022577655' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:33:05.351 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[56514]: cluster 2021-03-15T22:33:05.278482+0000 mon.a (mon.0) 2144 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2021-03-15T22:33:05.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[62156]: cluster 2021-03-15T22:33:03. 2021-03-15T22:33:05.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[62156]: 915232+0000 osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:05.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[62156]: cluster 2021-03-15T22:33:03.916360+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:05.352 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[62156]: cluster 2021-03-15T22:33:04.312397+0000 mon.a (mon.0) 2140 : cluster [DBG] osdmap e834: 8 total, 8 up, 8 in 2021-03-15T22:33:05.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[62156]: audit 2021-03-15T22:33:05.054661+0000 mon.a (mon.0) 2141 : audit [INF] from='client.? 172.21.15.76:0/1022577655' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:33:05.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[62156]: cluster 2021-03-15T22:33:05.268952+0000 mon.a (mon.0) 2142 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:05.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[62156]: audit 2021-03-15T22:33:05.278425+0000 mon.a (mon.0) 2143 : audit [INF] from='client.? 172.21.15.76:0/1022577655' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:33:05.353 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:05 smithi076 conmon[62156]: cluster 2021-03-15T22:33:05.278482+0000 mon.a (mon.0) 2144 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2021-03-15T22:33:05.361 INFO:tasks.rados.rados.0.smithi076.stdout:3121: done (4 left) 2021-03-15T22:33:05.361 INFO:tasks.rados.rados.0.smithi076.stdout:3122: done (3 left) 2021-03-15T22:33:05.361 INFO:tasks.rados.rados.0.smithi076.stdout:3125: done (2 left) 2021-03-15T22:33:05.361 INFO:tasks.rados.rados.0.smithi076.stdout:3126: done (1 left) 2021-03-15T22:33:05.362 INFO:tasks.rados.rados.0.smithi076.stdout:3127: snap_create 2021-03-15T22:33:05.373 INFO:tasks.rados.rados.0.smithi076.stdout:3124: finishing copy_from to smithi07694878-4 2021-03-15T22:33:05.373 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 1977 (ObjNum 931 snap 315 seq_num 931) dirty exists 2021-03-15T22:33:05.373 INFO:tasks.rados.rados.0.smithi076.stdout:3124: finishing copy_from racing read to smithi07694878-4 2021-03-15T22:33:05.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:05.403 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:05.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:05.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:05.407 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:05.505 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:05.526 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:05.527 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:05.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:05.650 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:05.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:05.651 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:05.752 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:05.776 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:05.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:05.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:05.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: cluster 2021-03-15T22:33:03.915232+0000 2021-03-15T22:33:05.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: osd.4 (osd.4) 175 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:05.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: cluster 2021-03-15T22:33:03.916360+0000 osd.4 (osd.4) 176 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:05.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: cluster 2021-03-15T22:33:04 2021-03-15T22:33:05.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: .312397+0000 mon.a (mon.0) 2140 : cluster 2021-03-15T22:33:05.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: [DBG] osdmap e834: 8 total, 8 up, 8 in 2021-03-15T22:33:05.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: audit 2021-03-15T22:33 2021-03-15T22:33:05.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: :05.054661+0000 mon.a (mon.0) 2021-03-15T22:33:05.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: 2141 : audit [INF] from='client.? 172.21.15.76:0/1022577655' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:33:05.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:33:05.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: T22:33:05.268952+0000 mon.a ( 2021-03-15T22:33:05.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: mon.0) 2142 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:05.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: audit 2021 2021-03-15T22:33:05.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: -03-15T22:33:05.278425 2021-03-15T22:33:05.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2143 : audit 2021-03-15T22:33:05.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/1022577655' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:33:05.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: cluster 2021-03-15T22:33: 2021-03-15T22:33:05.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: 05.278482+0000 mon.a (mon.0) 2021-03-15T22:33:05.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:05 smithi125 conmon[56739]: 2144 : cluster [DBG] osdmap e835: 8 total, 8 up, 8 in 2021-03-15T22:33:05.879 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:05.901 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:05.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:05.902 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:06.003 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:06.025 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:06.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:06.027 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:06.128 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:06.151 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:06.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:06.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:06.253 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:06.274 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:06.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:06.276 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:06.331 INFO:tasks.rados.rados.0.smithi076.stdout:3124: done (1 left) 2021-03-15T22:33:06.332 INFO:tasks.rados.rados.0.smithi076.stdout:3127: done (0 left) 2021-03-15T22:33:06.332 INFO:tasks.rados.rados.0.smithi076.stdout:3128: snap_create 2021-03-15T22:33:06.377 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:06.399 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:06.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:06.401 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:06.502 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:06.524 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:06.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:06.526 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:06.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:06.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:06.650 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:06.650 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:06.697 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[56514]: cluster 2021-03-15T22:33:05.184741+0000 mgr.y (mgr. 2021-03-15T22:33:06.698 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[56514]: 14144) 968 : cluster [DBG] pgmap v1396: 148 pgs: 1 active+clean+snaptrim, 3 remapped+peering, 1 activating, 14 peering, 129 active+clean; 506 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.8 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2021-03-15T22:33:06.698 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[56514]: cluster 2021-03-15T22:33:05.275427+0000 osd.7 (osd.7) 157 : cluster [DBG] 5.9 starting backfill to osd.6 from (0'0,0'0] MAX to 801'11 2021-03-15T22:33:06.698 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[56514]: cluster 2021-03-15T22:33:05.276681+0000 osd.2 (osd.2) 32 : cluster [DBG] 5.1f starting backfill to osd.6 from (0'0,0'0] MAX to 771'16 2021-03-15T22:33:06.698 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[56514]: cluster 2021-03-15T22:33:05.281797+0000 osd.5 (osd.5) 37 : cluster [DBG] 5.f starting backfill to osd.6 from (0'0,0'0] MAX to 643'9 2021-03-15T22:33:06.698 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[56514]: cluster 2021-03-15T22:33:05.317812+0000 mon.a (mon.0) 2145 : cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:33:06.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: cluster 2021-03-15T22:33:05. 2021-03-15T22:33:06.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: 184741+0000 mgr.y (mgr.14144) 968 : cluster [DBG] pgmap v1396: 148 pgs: 1 active+clean+snaptrim, 3 remapped+peering, 1 activating, 14 peering, 129 active+clean; 506 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.8 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2021-03-15T22:33:06.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:06.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: -03-15T22:33: 2021-03-15T22:33:06.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: 05.275427+0000 osd.7 (osd 2021-03-15T22:33:06.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: .7) 157 : cluster 2021-03-15T22:33:06.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: [DBG] 5.9 starting backfill to osd.6 from (0'0,0'0] MAX to 801'11 2021-03-15T22:33:06.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:06.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: -03-15T22:33: 2021-03-15T22:33:06.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: 05.276681+0000 osd.2 (osd. 2021-03-15T22:33:06.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: 2) 32 : cluster [DBG] 5.1f starting backfill to osd.6 from (0'0,0'0] MAX to 771'16 2021-03-15T22:33:06.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: 2021-03-15T22:33:06.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: cluster 2021-03-15T22:33:06.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:33:06.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: 33:05.281797+0000 osd.5 2021-03-15T22:33:06.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: (osd.5) 37 : cluster 2021-03-15T22:33:06.702 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: [DBG] 5.f starting backfill to osd.6 from (0'0,0'0] MAX to 643'9 2021-03-15T22:33:06.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:33:06.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: 03-15T22:33:05. 2021-03-15T22:33:06.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: 317812+0000 mon.a (mon. 2021-03-15T22:33:06.703 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: 0) 2145 : cluster [WRN] 2021-03-15T22:33:06.704 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:06 smithi076 conmon[62156]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:33:06.751 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:06.773 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:06.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:06.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:06.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: cluster 2021-03-15T22:33:05.184741+0000 mgr.y (mgr.14144) 968 : cluster [DBG] pgmap v1396: 148 pgs: 1 active+clean+snaptrim, 3 remapped+peering, 1 activating, 14 peering, 129 active+clean; 506 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 6.8 MiB/s rd, 1.3 MiB/s wr, 10 op/s 2021-03-15T22:33:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: cluster 2021-03-15T22:33:05.275427+0000 osd.7 (osd.7) 157 : cluster [DBG] 5.9 starting backfill to osd.6 from (0'0,0'0] MAX to 801'11 2021-03-15T22:33:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:33:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: 03-15T22:33:05 2021-03-15T22:33:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: .276681+0000 osd.2 (osd. 2021-03-15T22:33:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: 2) 32 : cluster [DBG] 2021-03-15T22:33:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: 5.1f starting backfill to osd.6 from (0'0,0'0] MAX to 771'16 2021-03-15T22:33:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:33:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: -03-15T22:33 2021-03-15T22:33:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: :05.281797+0000 osd.5 ( 2021-03-15T22:33:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: osd.5) 37 : cluster 2021-03-15T22:33:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: [DBG] 5.f starting backfill to osd.6 from (0'0,0'0] MAX to 643'9 2021-03-15T22:33:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: cluster 2021-03-15T22:33:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: 2021-03-15T22 2021-03-15T22:33:06.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: :33:05.317812+0000 2021-03-15T22:33:06.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: mon.a (mon.0) 2145 : 2021-03-15T22:33:06.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:06 smithi125 conmon[56739]: cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:33:06.876 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:06.898 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:06.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:06.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:07.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:07.026 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:07.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:07.028 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:07.130 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:07.152 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:07.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:07.153 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:07.254 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:07.278 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:07.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:07.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:07.334 INFO:tasks.rados.rados.0.smithi076.stdout:3128: done (0 left) 2021-03-15T22:33:07.334 INFO:tasks.rados.rados.0.smithi076.stdout:3129: write oid 13 current snap is 366 2021-03-15T22:33:07.334 INFO:tasks.rados.rados.0.smithi076.stdout:3129: seq_num 1072 ranges {417104=672701,1813991=203113} 2021-03-15T22:33:07.340 INFO:tasks.rados.rados.0.smithi076.stdout:3129: writing smithi07694878-13 from 417104 to 1089805 tid 1 2021-03-15T22:33:07.346 INFO:tasks.rados.rados.0.smithi076.stdout:3129: writing smithi07694878-13 from 1813991 to 2017104 tid 2 2021-03-15T22:33:07.346 INFO:tasks.rados.rados.0.smithi076.stdout:3130: write (excl) oid 20 current snap is 366 2021-03-15T22:33:07.346 INFO:tasks.rados.rados.0.smithi076.stdout:3129: finishing write tid 1 to smithi07694878-13 2021-03-15T22:33:07.346 INFO:tasks.rados.rados.0.smithi076.stdout:3130: seq_num 1073 ranges {433911=747950,1756120=277791} 2021-03-15T22:33:07.353 INFO:tasks.rados.rados.0.smithi076.stdout:3130: writing smithi07694878-20 from 433911 to 1181861 tid 1 2021-03-15T22:33:07.358 INFO:tasks.rados.rados.0.smithi076.stdout:3130: writing smithi07694878-20 from 1756120 to 2033911 tid 2 2021-03-15T22:33:07.359 INFO:tasks.rados.rados.0.smithi076.stdout:3131: copy_from oid 47 from oid 44 current snap is 366 2021-03-15T22:33:07.359 INFO:tasks.rados.rados.0.smithi076.stdout:3129: finishing write tid 2 to smithi07694878-13 2021-03-15T22:33:07.359 INFO:tasks.rados.rados.0.smithi076.stdout:3129: finishing write tid 3 to smithi07694878-13 2021-03-15T22:33:07.360 INFO:tasks.rados.rados.0.smithi076.stdout:3129: finishing write tid 4 to smithi07694878-13 2021-03-15T22:33:07.361 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1991 (ObjNum 1072 snap 366 seq_num 1072) dirty exists 2021-03-15T22:33:07.361 INFO:tasks.rados.rados.0.smithi076.stdout:3129: left oid 13 (ObjNum 1072 snap 366 seq_num 1072) 2021-03-15T22:33:07.361 INFO:tasks.rados.rados.0.smithi076.stdout:3129: done (2 left) 2021-03-15T22:33:07.362 INFO:tasks.rados.rados.0.smithi076.stdout:3132: delete oid 42 current snap is 366 2021-03-15T22:33:07.362 INFO:tasks.rados.rados.0.smithi076.stdout:3130: finishing write tid 1 to smithi07694878-20 2021-03-15T22:33:07.364 INFO:tasks.rados.rados.0.smithi076.stdout:3130: finishing write tid 2 to smithi07694878-20 2021-03-15T22:33:07.365 INFO:tasks.rados.rados.0.smithi076.stdout:3130: finishing write tid 3 to smithi07694878-20 2021-03-15T22:33:07.365 INFO:tasks.rados.rados.0.smithi076.stdout:3130: finishing write tid 4 to smithi07694878-20 2021-03-15T22:33:07.365 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 2402 (ObjNum 1073 snap 366 seq_num 1073) dirty exists 2021-03-15T22:33:07.366 INFO:tasks.rados.rados.0.smithi076.stdout:3130: left oid 20 (ObjNum 1073 snap 366 seq_num 1073) 2021-03-15T22:33:07.366 INFO:tasks.rados.rados.0.smithi076.stdout:3130: done (2 left) 2021-03-15T22:33:07.367 INFO:tasks.rados.rados.0.smithi076.stdout:3132: done (1 left) 2021-03-15T22:33:07.367 INFO:tasks.rados.rados.0.smithi076.stdout:3133: write oid 19 current snap is 366 2021-03-15T22:33:07.368 INFO:tasks.rados.rados.0.smithi076.stdout:3133: seq_num 1075 ranges {467525=498448,1473084=594379} 2021-03-15T22:33:07.368 INFO:tasks.rados.rados.0.smithi076.stdout:3133: writing smithi07694878-19 from 467525 to 965973 tid 1 2021-03-15T22:33:07.375 INFO:tasks.rados.rados.0.smithi076.stdout:3133: writing smithi07694878-19 from 1473084 to 2067463 tid 2 2021-03-15T22:33:07.376 INFO:tasks.rados.rados.0.smithi076.stdout:3134: copy_from oid 43 from oid 37 current snap is 366 2021-03-15T22:33:07.376 INFO:tasks.rados.rados.0.smithi076.stdout:3133: finishing write tid 1 to smithi07694878-19 2021-03-15T22:33:07.377 INFO:tasks.rados.rados.0.smithi076.stdout:3135: read oid 32 snap -1 2021-03-15T22:33:07.377 INFO:tasks.rados.rados.0.smithi076.stdout:3135: expect (ObjNum 1039 snap 347 seq_num 1039) 2021-03-15T22:33:07.377 INFO:tasks.rados.rados.0.smithi076.stdout:3136: write oid 41 current snap is 366 2021-03-15T22:33:07.378 INFO:tasks.rados.rados.0.smithi076.stdout:3136: seq_num 1076 ranges {484332=573697,1531566=552766} 2021-03-15T22:33:07.380 INFO:tasks.rados.rados.0.smithi076.stdout:3136: writing smithi07694878-41 from 484332 to 1058029 tid 1 2021-03-15T22:33:07.381 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:07.386 INFO:tasks.rados.rados.0.smithi076.stdout:3136: writing smithi07694878-41 from 1531566 to 2084332 tid 2 2021-03-15T22:33:07.387 INFO:tasks.rados.rados.0.smithi076.stdout:3137: copy_from oid 27 from oid 25 current snap is 366 2021-03-15T22:33:07.387 INFO:tasks.rados.rados.0.smithi076.stdout:3131: finishing copy_from to smithi07694878-47 2021-03-15T22:33:07.387 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1783 (ObjNum 1049 snap 353 seq_num 1049) dirty exists 2021-03-15T22:33:07.387 INFO:tasks.rados.rados.0.smithi076.stdout:3133: finishing write tid 2 to smithi07694878-19 2021-03-15T22:33:07.388 INFO:tasks.rados.rados.0.smithi076.stdout:3131: finishing copy_from racing read to smithi07694878-47 2021-03-15T22:33:07.388 INFO:tasks.rados.rados.0.smithi076.stdout:3133: finishing write tid 3 to smithi07694878-19 2021-03-15T22:33:07.388 INFO:tasks.rados.rados.0.smithi076.stdout:3133: finishing write tid 4 to smithi07694878-19 2021-03-15T22:33:07.388 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2133 (ObjNum 1075 snap 366 seq_num 1075) dirty exists 2021-03-15T22:33:07.388 INFO:tasks.rados.rados.0.smithi076.stdout:3133: left oid 19 (ObjNum 1075 snap 366 seq_num 1075) 2021-03-15T22:33:07.389 INFO:tasks.rados.rados.0.smithi076.stdout:3131: done (5 left) 2021-03-15T22:33:07.389 INFO:tasks.rados.rados.0.smithi076.stdout:3133: done (4 left) 2021-03-15T22:33:07.389 INFO:tasks.rados.rados.0.smithi076.stdout:3138: write (excl) oid 10 current snap is 366 2021-03-15T22:33:07.389 INFO:tasks.rados.rados.0.smithi076.stdout:3138: seq_num 1077 ranges {501139=648946,1873695=227444} 2021-03-15T22:33:07.392 INFO:tasks.rados.rados.0.smithi076.stdout:3138: writing smithi07694878-10 from 501139 to 1150085 tid 1 2021-03-15T22:33:07.397 INFO:tasks.rados.rados.0.smithi076.stdout:3138: writing smithi07694878-10 from 1873695 to 2101139 tid 2 2021-03-15T22:33:07.398 INFO:tasks.rados.rados.0.smithi076.stdout:3139: snap_create 2021-03-15T22:33:07.398 INFO:tasks.rados.rados.0.smithi076.stdout:3135: expect (ObjNum 1039 snap 347 seq_num 1039) 2021-03-15T22:33:07.404 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:07.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:07.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:07.507 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:07.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:07.530 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:07.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:07.632 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:07.654 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:07.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:07.659 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:07.710 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:07 smithi076 conmon[56514]: cluster 2021-03-15T22:33:06.329833+0000 mon.a (mon.0) 2146 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2021-03-15T22:33:07.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:07 smithi076 conmon[62156]: cluster 2021-03-15T22:33:06.329833+0000 mon.a 2021-03-15T22:33:07.711 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:07 smithi076 conmon[62156]: (mon.0) 2146 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2021-03-15T22:33:07.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:07.784 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:07.784 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:07.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:07 smithi125 conmon[56739]: cluster 2021-03-15T22:33:06.329833+0000 2021-03-15T22:33:07.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:07 smithi125 conmon[56739]: mon.a (mon.0) 2146 : cluster [DBG] osdmap e836: 8 total, 8 up, 8 in 2021-03-15T22:33:07.885 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:07.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:07.908 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:07.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:07.991 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:33:07.991 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:33:07.991 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:33:07.992 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:33:07.992 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:33:07.992 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:33:07.998 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:33:08.010 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:08.015 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:33:08.016 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:33:08.017 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:33:08.017 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:33:08.017 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:33:08.017 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:33:08.018 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:33:08.018 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:33:08.018 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:33:08.032 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:08.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:08.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:08.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:08.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:08.161 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:08.262 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:08.284 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:08.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:08.285 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:08.350 INFO:tasks.rados.rados.0.smithi076.stdout:3135: done (5 left) 2021-03-15T22:33:08.351 INFO:tasks.rados.rados.0.smithi076.stdout:3139: done (4 left) 2021-03-15T22:33:08.351 INFO:tasks.rados.rados.0.smithi076.stdout:3140: read oid 35 snap 354 2021-03-15T22:33:08.351 INFO:tasks.rados.rados.0.smithi076.stdout:3140: expect (ObjNum 1050 snap 353 seq_num 1050) 2021-03-15T22:33:08.351 INFO:tasks.rados.rados.0.smithi076.stdout:3141: delete oid 40 current snap is 367 2021-03-15T22:33:08.356 INFO:tasks.rados.rados.0.smithi076.stdout:3141: done (5 left) 2021-03-15T22:33:08.356 INFO:tasks.rados.rados.0.smithi076.stdout:3142: rollback oid 47 current snap is 367 2021-03-15T22:33:08.356 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 47 to 366 2021-03-15T22:33:08.356 INFO:tasks.rados.rados.0.smithi076.stdout:3143: copy_from oid 5 from oid 18 current snap is 367 2021-03-15T22:33:08.357 INFO:tasks.rados.rados.0.smithi076.stdout:3144: snap_create 2021-03-15T22:33:08.360 INFO:tasks.rados.rados.0.smithi076.stdout:3143: finishing copy_from to smithi07694878-5 2021-03-15T22:33:08.360 INFO:tasks.rados.rados.0.smithi076.stdout:3143: got expected ENOENT (src dne) 2021-03-15T22:33:08.360 INFO:tasks.rados.rados.0.smithi076.stdout:3143: finishing copy_from racing read to smithi07694878-5 2021-03-15T22:33:08.362 INFO:tasks.rados.rados.0.smithi076.stdout:3142: finishing rollback tid 0 to smithi07694878-47 2021-03-15T22:33:08.362 INFO:tasks.rados.rados.0.smithi076.stdout:3142: finishing rollback tid 1 to smithi07694878-47 2021-03-15T22:33:08.363 INFO:tasks.rados.rados.0.smithi076.stdout:3140: expect (ObjNum 1050 snap 353 seq_num 1050) 2021-03-15T22:33:08.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:08.408 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:08.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:08.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:08.481 INFO:tasks.rados.rados.0.smithi076.stdout:3142: finishing rollback tid 2 to smithi07694878-47 2021-03-15T22:33:08.481 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1793 (ObjNum 1049 snap 353 seq_num 1049) dirty exists 2021-03-15T22:33:08.482 INFO:tasks.rados.rados.0.smithi076.stdout:3138: finishing write tid 1 to smithi07694878-10 2021-03-15T22:33:08.482 INFO:tasks.rados.rados.0.smithi076.stdout:3138: finishing write tid 2 to smithi07694878-10 2021-03-15T22:33:08.482 INFO:tasks.rados.rados.0.smithi076.stdout:3138: finishing write tid 3 to smithi07694878-10 2021-03-15T22:33:08.482 INFO:tasks.rados.rados.0.smithi076.stdout:3138: finishing write tid 4 to smithi07694878-10 2021-03-15T22:33:08.483 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 1985 (ObjNum 1077 snap 366 seq_num 1077) dirty exists 2021-03-15T22:33:08.483 INFO:tasks.rados.rados.0.smithi076.stdout:3138: left oid 10 (ObjNum 1077 snap 366 seq_num 1077) 2021-03-15T22:33:08.483 INFO:tasks.rados.rados.0.smithi076.stdout:3134: finishing copy_from to smithi07694878-43 2021-03-15T22:33:08.483 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 2395 (ObjNum 982 snap 326 seq_num 982) dirty exists 2021-03-15T22:33:08.484 INFO:tasks.rados.rados.0.smithi076.stdout:3134: finishing copy_from racing read to smithi07694878-43 2021-03-15T22:33:08.484 INFO:tasks.rados.rados.0.smithi076.stdout:3137: finishing copy_from to smithi07694878-27 2021-03-15T22:33:08.484 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1982 (ObjNum 1068 snap 363 seq_num 1068) dirty exists 2021-03-15T22:33:08.484 INFO:tasks.rados.rados.0.smithi076.stdout:3137: finishing copy_from racing read to smithi07694878-27 2021-03-15T22:33:08.510 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:08.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:08.533 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:08.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:08.635 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:08.657 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:08.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:08.658 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:08.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[56514]: cluster 2021-03-15T22:33:07.185232+0000 mgr.y (mgr.14144) 969 : 2021-03-15T22:33:08.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[56514]: cluster [DBG] pgmap v1399: 148 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 1 active+clean+snaptrim, 3 remapped+peering, 1 activating, 14 peering, 125 active+clean; 517 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 11 op/s; 66/1656 objects degraded (3.986%); 13/1656 objects misplaced (0.785%); 1.6 KiB/s, 25 objects/s recovering 2021-03-15T22:33:08.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[56514]: cluster 2021-03-15T22:33:07.328236+0000 mon.a (mon.0) 2147 : cluster [WRN] Health check failed: Degraded data redundancy: 66/1656 objects degraded (3.986%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:33:08.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[56514]: cluster 2021-03-15T22:33:07.332028+0000 mon.a (mon.0) 2148 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2021-03-15T22:33:08.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[56514]: audit 2021-03-15T22:33:07.873999+0000 mon.c (mon.1) 66 : audit [DBG] from='client.? 172.21.15.76:0/538834546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:08.688 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: cluster 2021-03-15T22:33:07.185232+0000 2021-03-15T22:33:08.688 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: mgr.y (mgr.14144) 969 : cluster [DBG] pgmap v1399: 148 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 1 active+clean+snaptrim, 3 remapped+peering, 1 activating, 14 peering, 125 active+clean; 517 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 11 op/s; 66/1656 objects degraded (3.986%); 13/1656 objects misplaced (0.785%); 1.6 KiB/s, 25 objects/s recovering 2021-03-15T22:33:08.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:33:08.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: T22:33:07.328236+0000 mon.a (mon.0) 2147 : cluster [WRN] Health check failed: Degraded data redundancy: 66/1656 objects degraded (3.986%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:33:08.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: 2021-03-15T22:33:08.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:33:08.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: 22:33:07.332028+0000 mon.a 2021-03-15T22:33:08.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: (mon.0) 2148 : cluster [DBG] osdmap e837: 8 total, 8 up, 8 in 2021-03-15T22:33:08.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: 2021-03-15T22:33:08.690 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:33:08.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: 22:33:07.873999+0000 mon.c (mon 2021-03-15T22:33:08.691 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:08 smithi076 conmon[62156]: .1) 66 : audit [DBG] from='client.? 172.21.15.76:0/538834546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:08.749 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:33:08.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:08.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:08 smithi125 conmon[56739]: cluster 2021-03-15T22:33:07.185232+0000 mgr.y (mgr.14144) 969 : cluster [DBG] pgmap v1399: 148 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 1 active+clean+snaptrim, 3 remapped+peering, 1 activating, 14 peering, 125 active+clean; 517 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 2.7 MiB/s rd, 2.7 MiB/s wr, 11 op/s; 66/1656 objects degraded (3.986%); 13/1656 objects misplaced (0.785%); 1.6 KiB/s, 25 objects/s recovering 2021-03-15T22:33:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:08 smithi125 conmon[56739]: cluster 2021-03-15T22:33:07.328236+0000 mon.a (mon.0) 2147 : cluster [WRN] Health check failed: Degraded data redundancy: 66/1656 objects degraded (3.986%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:33:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:08 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:33:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:08 smithi125 conmon[56739]: 03-15T22:33:07.332028+0000 mon.a (mon.0) 2148 : cluster 2021-03-15T22:33:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:08 smithi125 conmon[56739]: [DBG] osdmap e837: 8 total, 8 up, 8 in 2021-03-15T22:33:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:08 smithi125 conmon[56739]: audit 2021-03-15T22:33:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:08 smithi125 conmon[56739]: 2021-03-15T22:33 2021-03-15T22:33:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:08 smithi125 conmon[56739]: :07.873999+0000 mon.c (mon.1) 2021-03-15T22:33:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:08 smithi125 conmon[56739]: 66 : audit [DBG] from='client.? 172.21.15.76:0/538834546' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:08.851 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:08.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:08.852 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:08.926 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:33:08.926 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:33:08.954 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:08.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:08.977 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:08.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:09.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:09.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:09.102 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:09.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:09.204 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:09.226 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:09.228 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:09.329 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:09.352 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:09.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:09.354 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:09.355 INFO:tasks.rados.rados.0.smithi076.stdout:3134: done (7 left) 2021-03-15T22:33:09.355 INFO:tasks.rados.rados.0.smithi076.stdout:3137: done (6 left) 2021-03-15T22:33:09.355 INFO:tasks.rados.rados.0.smithi076.stdout:3138: done (5 left) 2021-03-15T22:33:09.355 INFO:tasks.rados.rados.0.smithi076.stdout:3140: done (4 left) 2021-03-15T22:33:09.356 INFO:tasks.rados.rados.0.smithi076.stdout:3142: done (3 left) 2021-03-15T22:33:09.356 INFO:tasks.rados.rados.0.smithi076.stdout:3143: done (2 left) 2021-03-15T22:33:09.356 INFO:tasks.rados.rados.0.smithi076.stdout:3144: done (1 left) 2021-03-15T22:33:09.356 INFO:tasks.rados.rados.0.smithi076.stdout:3145: read oid 48 snap -1 2021-03-15T22:33:09.357 INFO:tasks.rados.rados.0.smithi076.stdout:3145: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:09.357 INFO:tasks.rados.rados.0.smithi076.stdout:3146: read oid 1 snap -1 2021-03-15T22:33:09.357 INFO:tasks.rados.rados.0.smithi076.stdout:3146: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:09.358 INFO:tasks.rados.rados.0.smithi076.stdout:3147: copy_from oid 45 from oid 2 current snap is 368 2021-03-15T22:33:09.358 INFO:tasks.rados.rados.0.smithi076.stdout:3148: rollback oid 27 current snap is 368 2021-03-15T22:33:09.358 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 308 2021-03-15T22:33:09.359 INFO:tasks.rados.rados.0.smithi076.stdout:3149: read oid 4 snap 338 2021-03-15T22:33:09.359 INFO:tasks.rados.rados.0.smithi076.stdout:3149: expect deleted 2021-03-15T22:33:09.359 INFO:tasks.rados.rados.0.smithi076.stdout:3150: snap_remove snap 366 2021-03-15T22:33:09.455 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:09.478 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:09.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:09.479 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:09.580 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:09.602 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:09.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:09.604 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:09.638 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[56514]: cluster 2021-03-15T22:33:08.273373+0000 osd.6 (osd.6) 120 : cluster [DBG] 3.a scrub ok 2021-03-15T22:33:09.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[56514]: cluster 2021-03-15T22:33:08.345375+0000 mon.a (mon.0) 2149 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2021-03-15T22:33:09.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[56514]: cluster 2021-03-15T22:33:08.608661+0000 osd.0 (osd.0) 137 : cluster [DBG] 7.f scrub ok 2021-03-15T22:33:09.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[56514]: cluster 2021-03-15T22:33:08.884874+0000 osd.2 (osd.2) 33 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:33:09.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[62156]: cluster 2021-03-15T22:33:08. 2021-03-15T22:33:09.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[62156]: 273373+0000 osd.6 (osd.6) 120 : cluster [DBG] 3.a scrub ok 2021-03-15T22:33:09.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:33:09.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[62156]: -15T22:33:08.345375+0000 mon.a (mon.0) 2149 : 2021-03-15T22:33:09.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[62156]: cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2021-03-15T22:33:09.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[62156]: cluster 2021-03-15T22:33:08.608661+0000 osd.0 (osd 2021-03-15T22:33:09.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[62156]: .0) 137 : cluster [DBG] 7.f scrub ok 2021-03-15T22:33:09.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:33:09.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[62156]: 03-15T22:33:08.884874+0000 2021-03-15T22:33:09.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[62156]: osd.2 (osd.2) 33 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:33:09.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:09 smithi076 conmon[62156]: 2021-03-15T22:33:09.644 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:09.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:33:09.705 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:09.735 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:09.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:09.737 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:09.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: cluster 2021-03-15T22:33:08. 2021-03-15T22:33:09.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: 273373+0000 osd.6 (osd.6) 120 : cluster [DBG] 3.a scrub ok 2021-03-15T22:33:09.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: cluster 2021-03-15T22:33:09.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: 2021-03-15T22:33:08. 2021-03-15T22:33:09.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: 345375+0000 mon.a (mon.0) 2021-03-15T22:33:09.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: 2149 : cluster [DBG] osdmap e838: 8 total, 8 up, 8 in 2021-03-15T22:33:09.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:33:09.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: 03-15T22:33:08. 2021-03-15T22:33:09.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: 608661+0000 osd.0 (osd.0) 2021-03-15T22:33:09.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: 137 : cluster [DBG] 7.f scrub ok 2021-03-15T22:33:09.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:33:09.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: -03-15T22:33:08 2021-03-15T22:33:09.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: .884874+0000 osd.2 (osd.2 2021-03-15T22:33:09.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: ) 33 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:33:09.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:09 smithi125 conmon[56739]: 2021-03-15T22:33:09.838 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:09.862 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:09.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:09.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:09.927 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 7, 0, 3, 5, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:33:09.927 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:33:09.928 INFO:tasks.thrashosds.thrasher:fixing pg num pool unique_pool_0 2021-03-15T22:33:09.928 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:33:09.965 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:09.989 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:09.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:09.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:10.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:10.116 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:10.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:10.118 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:10.183 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:10.219 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:10.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:10.244 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:10.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:10.250 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:10.345 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:10.369 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:10.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:10.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:10.444 INFO:tasks.rados.rados.0.smithi076.stdout:3150: done (6 left) 2021-03-15T22:33:10.444 INFO:tasks.rados.rados.0.smithi076.stdout:3151: delete oid 36 current snap is 368 2021-03-15T22:33:10.445 INFO:tasks.rados.rados.0.smithi076.stdout:3148: finishing rollback tid 0 to smithi07694878-27 2021-03-15T22:33:10.445 INFO:tasks.rados.rados.0.smithi076.stdout:3148: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:33:10.445 INFO:tasks.rados.rados.0.smithi076.stdout:3148: finishing rollback tid 2 to smithi07694878-27 2021-03-15T22:33:10.445 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1985 (ObjNum 896 snap 302 seq_num 896) dirty exists 2021-03-15T22:33:10.445 INFO:tasks.rados.rados.0.smithi076.stdout:3146: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:10.452 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:33:10.452 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[56739]: cluster 2021-03-15T22:33:09.185619+0000 mgr.y ( 2021-03-15T22:33:10.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[56739]: mgr.14144) 970 : cluster [DBG] pgmap v1402: 148 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 1 active+clean+snaptrim, 2 remapped+peering, 1 activating, 16 peering, 124 active+clean; 517 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 2.3 MiB/s wr, 5 op/s; 66/1656 objects degraded (3.986%); 13/1656 objects misplaced (0.785%); 6.0 MiB/s, 43 objects/s recovering 2021-03-15T22:33:10.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[56739]: cluster 2021-03-15T22:33:09.245687+0000 osd.6 (osd.6) 121 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:33:10.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[56739]: cluster 2021-03-15T22:33:09.351479+0000 mon.a 2021-03-15T22:33:10.453 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[56739]: (mon.0) 2150 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2021-03-15T22:33:10.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:33:10.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[56739]: -15T22:33:09.468587+0000 osd.1 (osd 2021-03-15T22:33:10.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[56739]: .1) 161 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:33:10.454 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:33:10.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[56739]: T22:33:09.612119+0000 osd.3 (osd.3 2021-03-15T22:33:10.455 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:10 smithi125 conmon[56739]: ) 151 : cluster [DBG] 6.c scrub ok 2021-03-15T22:33:10.472 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:10.498 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:10.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:10.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:10.533 INFO:tasks.rados.rados.0.smithi076.stdout:3147: finishing copy_from to smithi07694878-45 2021-03-15T22:33:10.533 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 1977 (ObjNum 931 snap 315 seq_num 931) dirty exists 2021-03-15T22:33:10.533 INFO:tasks.rados.rados.0.smithi076.stdout:3147: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:33:10.534 INFO:tasks.rados.rados.0.smithi076.stdout:3146: done (6 left) 2021-03-15T22:33:10.534 INFO:tasks.rados.rados.0.smithi076.stdout:3147: done (5 left) 2021-03-15T22:33:10.534 INFO:tasks.rados.rados.0.smithi076.stdout:3148: done (4 left) 2021-03-15T22:33:10.534 INFO:tasks.rados.rados.0.smithi076.stdout:3149: done (3 left) 2021-03-15T22:33:10.535 INFO:tasks.rados.rados.0.smithi076.stdout:3151: done (2 left) 2021-03-15T22:33:10.535 INFO:tasks.rados.rados.0.smithi076.stdout:3152: read oid 35 snap 321 2021-03-15T22:33:10.535 INFO:tasks.rados.rados.0.smithi076.stdout:3152: expect (ObjNum 863 snap 285 seq_num 863) 2021-03-15T22:33:10.535 INFO:tasks.rados.rados.0.smithi076.stdout:3153: copy_from oid 33 from oid 47 current snap is 368 2021-03-15T22:33:10.535 INFO:tasks.rados.rados.0.smithi076.stdout:3154: rollback oid 6 current snap is 368 2021-03-15T22:33:10.536 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 334 2021-03-15T22:33:10.536 INFO:tasks.rados.rados.0.smithi076.stdout:3155: copy_from oid 4 from oid 29 current snap is 368 2021-03-15T22:33:10.536 INFO:tasks.rados.rados.0.smithi076.stdout:3156: rollback oid 46 current snap is 368 2021-03-15T22:33:10.536 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 46 to 352 2021-03-15T22:33:10.537 INFO:tasks.rados.rados.0.smithi076.stdout:3157: rollback oid 16 current snap is 368 2021-03-15T22:33:10.537 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 365 2021-03-15T22:33:10.537 INFO:tasks.rados.rados.0.smithi076.stdout:3158: delete oid 23 current snap is 368 2021-03-15T22:33:10.538 INFO:tasks.rados.rados.0.smithi076.stdout:3155: finishing copy_from to smithi07694878-4 2021-03-15T22:33:10.538 INFO:tasks.rados.rados.0.smithi076.stdout:3155: got expected ENOENT (src dne) 2021-03-15T22:33:10.538 INFO:tasks.rados.rados.0.smithi076.stdout:3155: finishing copy_from racing read to smithi07694878-4 2021-03-15T22:33:10.539 INFO:tasks.rados.rados.0.smithi076.stdout:3155: done (8 left) 2021-03-15T22:33:10.539 INFO:tasks.rados.rados.0.smithi076.stdout:3158: done (7 left) 2021-03-15T22:33:10.539 INFO:tasks.rados.rados.0.smithi076.stdout:3159: copy_from oid 40 from oid 42 current snap is 368 2021-03-15T22:33:10.539 INFO:tasks.rados.rados.0.smithi076.stdout:3160: snap_create 2021-03-15T22:33:10.540 INFO:tasks.rados.rados.0.smithi076.stdout:3152: expect (ObjNum 863 snap 285 seq_num 863) 2021-03-15T22:33:10.601 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:10.623 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:10.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:10.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:10.684 INFO:tasks.rados.rados.0.smithi076.stdout:3156: finishing rollback tid 0 to smithi07694878-46 2021-03-15T22:33:10.684 INFO:tasks.rados.rados.0.smithi076.stdout:3156: finishing rollback tid 1 to smithi07694878-46 2021-03-15T22:33:10.684 INFO:tasks.rados.rados.0.smithi076.stdout:3159: finishing copy_from to smithi07694878-40 2021-03-15T22:33:10.685 INFO:tasks.rados.rados.0.smithi076.stdout:3159: got expected ENOENT (src dne) 2021-03-15T22:33:10.685 INFO:tasks.rados.rados.0.smithi076.stdout:3156: finishing rollback tid 2 to smithi07694878-46 2021-03-15T22:33:10.685 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2279 (ObjNum 1044 snap 350 seq_num 1044) dirty exists 2021-03-15T22:33:10.685 INFO:tasks.rados.rados.0.smithi076.stdout:3154: finishing rollback tid 0 to smithi07694878-6 2021-03-15T22:33:10.686 INFO:tasks.rados.rados.0.smithi076.stdout:3157: finishing rollback tid 0 to smithi07694878-16 2021-03-15T22:33:10.687 INFO:tasks.rados.rados.0.smithi076.stdout:3157: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:33:10.687 INFO:tasks.rados.rados.0.smithi076.stdout:3157: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:33:10.687 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 2526 (ObjNum 1051 snap 353 seq_num 1051) dirty exists 2021-03-15T22:33:10.687 INFO:tasks.rados.rados.0.smithi076.stdout:3154: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:33:10.687 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 2102 (ObjNum 1957169032 snap 32564 seq_num 2521157712) dirty dne 2021-03-15T22:33:10.688 INFO:tasks.rados.rados.0.smithi076.stdout:3159: finishing copy_from racing read to smithi07694878-40 2021-03-15T22:33:10.688 INFO:tasks.rados.rados.0.smithi076.stdout:3153: finishing copy_from to smithi07694878-33 2021-03-15T22:33:10.688 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 1793 (ObjNum 1049 snap 353 seq_num 1049) dirty exists 2021-03-15T22:33:10.688 INFO:tasks.rados.rados.0.smithi076.stdout:3153: finishing copy_from racing read to smithi07694878-33 2021-03-15T22:33:10.726 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:10.749 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:10.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:10.750 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:10.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[56514]: cluster 2021-03-15T22:33:09.185619 2021-03-15T22:33:10.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 970 : cluster [DBG] pgmap v1402: 148 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 1 active+clean+snaptrim, 2 remapped+peering, 1 activating, 16 peering, 124 active+clean; 517 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 2.3 MiB/s wr, 5 op/s; 66/1656 objects degraded (3.986%); 13/1656 objects misplaced (0.785%); 6.0 MiB/s, 43 objects/s recovering 2021-03-15T22:33:10.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[56514]: cluster 2021-03-15T22:33:09.245687+0000 osd.6 (osd.6) 121 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:33:10.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[56514]: cluster 2021-03-15T22:33:09.351479+0000 mon.a (mon.0) 2150 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2021-03-15T22:33:10.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[56514]: cluster 2021-03-15T22:33:09.468587+0000 osd.1 (osd.1) 161 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:33:10.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[56514]: cluster 2021-03-15T22:33:09.612119+0000 osd.3 (osd.3) 151 : cluster [DBG] 6.c scrub ok 2021-03-15T22:33:10.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: cluster 2021-03-15T22:33:09.185619 2021-03-15T22:33:10.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 970 : cluster [DBG] pgmap v1402: 148 pgs: 2 active+recovering+undersized+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 1 active+recovering, 1 active+clean+snaptrim, 2 remapped+peering, 1 activating, 16 peering, 124 active+clean; 517 MiB data, 1.5 GiB used, 714 GiB / 715 GiB avail; 767 B/s rd, 2.3 MiB/s wr, 5 op/s; 66/1656 objects degraded (3.986%); 13/1656 objects misplaced (0.785%); 6.0 MiB/s, 43 objects/s recovering 2021-03-15T22:33:10.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: cluster 2021-03-15T22:33:09. 2021-03-15T22:33:10.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: 245687+0000 osd.6 (osd.6) 121 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:33:10.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: cluster 2021-03-15T22:33:10.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: 2021-03-15T22:33:09 2021-03-15T22:33:10.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: .351479+0000 mon.a (mon.0) 2150 : cluster [DBG] osdmap e839: 8 total, 8 up, 8 in 2021-03-15T22:33:10.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: cluster 2021-03-15T22:33:10.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: 2021-03-15T22:33: 2021-03-15T22:33:10.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: 09.468587+0000 osd.1 (osd.1) 161 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:33:10.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:33:10.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: 22:33:09.612119+0000 osd.3 (osd.3) 2021-03-15T22:33:10.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:10 smithi076 conmon[62156]: 151 : cluster [DBG] 6.c scrub ok 2021-03-15T22:33:10.791 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:33:10.852 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:10.875 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:10.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:10.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:10.977 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:11.002 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:11.002 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:11.103 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:11.126 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:11.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:11.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:11.228 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:11.252 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:11.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:11.254 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:11.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:11.378 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:11.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:11.379 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:11.463 INFO:tasks.rados.rados.0.smithi076.stdout:3152: done (8 left) 2021-03-15T22:33:11.464 INFO:tasks.rados.rados.0.smithi076.stdout:3153: done (7 left) 2021-03-15T22:33:11.464 INFO:tasks.rados.rados.0.smithi076.stdout:3154: done (6 left) 2021-03-15T22:33:11.464 INFO:tasks.rados.rados.0.smithi076.stdout:3156: done (5 left) 2021-03-15T22:33:11.464 INFO:tasks.rados.rados.0.smithi076.stdout:3157: done (4 left) 2021-03-15T22:33:11.465 INFO:tasks.rados.rados.0.smithi076.stdout:3159: done (3 left) 2021-03-15T22:33:11.465 INFO:tasks.rados.rados.0.smithi076.stdout:3160: done (2 left) 2021-03-15T22:33:11.465 INFO:tasks.rados.rados.0.smithi076.stdout:3161: copy_from oid 21 from oid 44 current snap is 369 2021-03-15T22:33:11.465 INFO:tasks.rados.rados.0.smithi076.stdout:3162: read oid 37 snap -1 2021-03-15T22:33:11.466 INFO:tasks.rados.rados.0.smithi076.stdout:3162: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:11.466 INFO:tasks.rados.rados.0.smithi076.stdout:3163: copy_from oid 29 from oid 40 current snap is 369 2021-03-15T22:33:11.466 INFO:tasks.rados.rados.0.smithi076.stdout:3164: rollback oid 27 current snap is 369 2021-03-15T22:33:11.466 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 368 2021-03-15T22:33:11.467 INFO:tasks.rados.rados.0.smithi076.stdout:3165: rollback oid 2 current snap is 369 2021-03-15T22:33:11.467 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 2 to 353 2021-03-15T22:33:11.467 INFO:tasks.rados.rados.0.smithi076.stdout:3166: snap_remove snap 363 2021-03-15T22:33:11.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:11.503 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:11.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:11.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:11.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[56514]: audit 2021-03-15T22:33:10.168385+0000 mgr.y (mgr.14144) 971 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:11.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[56514]: cluster 2021-03-15T22:33:10.438124+0000 mon.a (mon.0) 2151 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2021-03-15T22:33:11.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[56514]: cluster 2021-03-15T22:33:10.457071+0000 osd.1 (osd.1) 162 : cluster [DBG] 7.b scrub ok 2021-03-15T22:33:11.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: audit 2021-03-15T22:33:11.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:33:11.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: 22:33:10.168385+0000 mgr.y (mgr.14144) 971 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:11.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: cluster 2021-03-15T22:33:11.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:33:11.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: 33:10.438124+0000 mon.a 2021-03-15T22:33:11.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: (mon.0) 2151 : cluster 2021-03-15T22:33:11.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: [DBG] osdmap e840: 8 total, 8 up, 8 in 2021-03-15T22:33:11.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:11.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: -03-15T22:33:10.457071+0000 2021-03-15T22:33:11.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: osd.1 (osd.1) 162 : cluster [DBG] 7.b scrub ok 2021-03-15T22:33:11.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:11 smithi076 conmon[62156]: 2021-03-15T22:33:11.605 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:11.628 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:11.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:11.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:11.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:11.754 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:11.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:11.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:11.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:11 smithi125 conmon[56739]: audit 2021-03-15T22:33:10.168385+0000 mgr.y (mgr 2021-03-15T22:33:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:11 smithi125 conmon[56739]: .14144) 971 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:11 smithi125 conmon[56739]: cluster 2021-03-15T22:33:10.438124+0000 mon.a (mon. 2021-03-15T22:33:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:11 smithi125 conmon[56739]: 0) 2151 : cluster [DBG] osdmap e840: 8 total, 8 up, 8 in 2021-03-15T22:33:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:11 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:33:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:11 smithi125 conmon[56739]: -15T22:33:10.457071+0000 2021-03-15T22:33:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:11 smithi125 conmon[56739]: osd.1 (osd.1) 162 : 2021-03-15T22:33:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:11 smithi125 conmon[56739]: cluster [DBG] 7.b scrub ok 2021-03-15T22:33:11.856 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:11.878 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:11.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:11.880 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:11.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:12.005 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:12.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:12.007 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:12.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:12.132 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:12.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:12.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:12.205 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:12.235 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:12.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:12.259 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:12.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:12.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:12.610 INFO:tasks.rados.rados.0.smithi076.stdout:3166: done (7 left) 2021-03-15T22:33:12.611 INFO:tasks.rados.rados.0.smithi076.stdout:3167: snap_create 2021-03-15T22:33:12.611 INFO:tasks.rados.rados.0.smithi076.stdout:3164: finishing rollback tid 0 to smithi07694878-27 2021-03-15T22:33:12.611 INFO:tasks.rados.rados.0.smithi076.stdout:3164: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:33:12.611 INFO:tasks.rados.rados.0.smithi076.stdout:3164: finishing rollback tid 2 to smithi07694878-27 2021-03-15T22:33:12.611 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1988 (ObjNum 896 snap 302 seq_num 896) dirty exists 2021-03-15T22:33:12.612 INFO:tasks.rados.rados.0.smithi076.stdout:3165: finishing rollback tid 0 to smithi07694878-2 2021-03-15T22:33:12.612 INFO:tasks.rados.rados.0.smithi076.stdout:3165: finishing rollback tid 1 to smithi07694878-2 2021-03-15T22:33:12.612 INFO:tasks.rados.rados.0.smithi076.stdout:3165: finishing rollback tid 2 to smithi07694878-2 2021-03-15T22:33:12.612 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 1988 (ObjNum 931 snap 315 seq_num 931) dirty exists 2021-03-15T22:33:12.612 INFO:tasks.rados.rados.0.smithi076.stdout:3162: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:12.613 INFO:tasks.rados.rados.0.smithi076.stdout:3163: finishing copy_from to smithi07694878-29 2021-03-15T22:33:12.613 INFO:tasks.rados.rados.0.smithi076.stdout:3163: got expected ENOENT (src dne) 2021-03-15T22:33:12.613 INFO:tasks.rados.rados.0.smithi076.stdout:3163: finishing copy_from racing read to smithi07694878-29 2021-03-15T22:33:12.613 INFO:tasks.rados.rados.0.smithi076.stdout:3161: finishing copy_from to smithi07694878-21 2021-03-15T22:33:12.614 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 1783 (ObjNum 1049 snap 353 seq_num 1049) dirty exists 2021-03-15T22:33:12.614 INFO:tasks.rados.rados.0.smithi076.stdout:3161: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:33:12.635 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:12.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:12.636 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:12.737 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:12.763 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:12.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:12.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:12.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[56514]: cluster 2021-03-15T22:33: 2021-03-15T22:33:12.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[56514]: 11.186250+0000 mgr.y (mgr.14144) 972 : cluster [DBG] pgmap v1405: 148 pgs: 1 active+clean+wait, 1 active+recovering+undersized+remapped, 2 peering, 144 active+clean; 536 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 13 op/s; 6/1683 objects misplaced (0.357%); 19 MiB/s, 58 objects/s recovering 2021-03-15T22:33:12.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[56514]: cluster 2021-03-15T22:33:11.448918+0000 mon.a (mon.0) 2152 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-15T22:33:12.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[56514]: cluster 2021-03-15T22:33:11.448947+0000 mon.a (mon.0) 2153 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 66/1656 objects degraded (3.986%), 1 pg degraded) 2021-03-15T22:33:12.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[56514]: cluster 2021-03-15T22:33:11.458462+0000 mon.a (mon.0) 2154 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2021-03-15T22:33:12.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[56514]: cluster 2021-03-15T22:33:11.491192+0000 osd.1 (osd.1) 163 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:33:12.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[62156]: cluster 2021-03-15T22:33: 2021-03-15T22:33:12.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[62156]: 11.186250+0000 mgr.y (mgr.14144) 972 : cluster [DBG] pgmap v1405: 148 pgs: 1 active+clean+wait, 1 active+recovering+undersized+remapped, 2 peering, 144 active+clean; 536 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 13 op/s; 6/1683 objects misplaced (0.357%); 19 MiB/s, 58 objects/s recovering 2021-03-15T22:33:12.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[62156]: cluster 2021-03-15T22:33:11.448918+0000 mon.a (mon.0) 2152 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-15T22:33:12.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[62156]: cluster 2021-03-15T22:33:11.448947+0000 mon.a (mon.0) 2153 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 66/1656 objects degraded (3.986%), 1 pg degraded) 2021-03-15T22:33:12.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[62156]: cluster 2021-03-15T22:33:11.458462+0000 mon.a (mon.0) 2154 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2021-03-15T22:33:12.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:12 smithi076 conmon[62156]: cluster 2021-03-15T22:33:11.491192+0000 osd.1 (osd.1) 163 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:33:12.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:12 smithi125 conmon[56739]: cluster 2021-03-15T22:33:11.186250+0000 mgr.y (mgr.14144) 972 : cluster [DBG] pgmap v1405: 148 pgs: 1 active+clean+wait, 1 active+recovering+undersized+remapped, 2 peering, 144 active+clean; 536 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 2.0 MiB/s rd, 3.7 MiB/s wr, 13 op/s; 6/1683 objects misplaced (0.357%); 19 MiB/s, 58 objects/s recovering 2021-03-15T22:33:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:12 smithi125 conmon[56739]: cluster 2021-03-15T22:33:11.448918+0000 mon.a (mon.0) 2152 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-15T22:33:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:12 smithi125 conmon[56739]: cluster 2021-03-15T22:33:11.448947+0000 mon.a (mon. 2021-03-15T22:33:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:12 smithi125 conmon[56739]: 0) 2153 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 66/1656 objects degraded (3.986%), 1 pg degraded) 2021-03-15T22:33:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:12 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:33:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:12 smithi125 conmon[56739]: 22:33:11.458462+0000 mon.a ( 2021-03-15T22:33:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:12 smithi125 conmon[56739]: mon.0) 2154 : cluster [DBG] osdmap e841: 8 total, 8 up, 8 in 2021-03-15T22:33:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:12 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:33:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:12 smithi125 conmon[56739]: 15T22:33:11.491192+0000 osd.1 ( 2021-03-15T22:33:12.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:12 smithi125 conmon[56739]: osd.1) 163 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:33:12.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:12.889 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:12.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:12.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:12.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:13.018 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:13.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:13.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:13.120 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:13.145 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:13.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:13.147 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:13.151 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:33:13.151 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:33:13.248 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:13.270 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:13.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:13.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:13.373 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:13.395 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:13.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:13.396 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:13.482 INFO:tasks.rados.rados.0.smithi076.stdout:3161: done (7 left) 2021-03-15T22:33:13.483 INFO:tasks.rados.rados.0.smithi076.stdout:3162: done (6 left) 2021-03-15T22:33:13.483 INFO:tasks.rados.rados.0.smithi076.stdout:3163: done (5 left) 2021-03-15T22:33:13.483 INFO:tasks.rados.rados.0.smithi076.stdout:3164: done (4 left) 2021-03-15T22:33:13.483 INFO:tasks.rados.rados.0.smithi076.stdout:3165: done (3 left) 2021-03-15T22:33:13.484 INFO:tasks.rados.rados.0.smithi076.stdout:3167: done (2 left) 2021-03-15T22:33:13.484 INFO:tasks.rados.rados.0.smithi076.stdout:3168: snap_create 2021-03-15T22:33:13.497 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:13.524 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:13.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:13.526 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:13.627 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:13.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:13.654 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:13.655 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:13.756 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:13 smithi076 conmon[56514]: cluster 2021-03-15T22:33:12.429016+0000 osd.4 (osd.4) 177 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:33:13.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:13 smithi076 conmon[56514]: cluster 2021-03-15T22:33:12.467856+0000 mon.a (mon.0) 2155 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2021-03-15T22:33:13.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:13 smithi076 conmon[62156]: cluster 2021-03-15T22:33:12. 2021-03-15T22:33:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:13 smithi076 conmon[62156]: 429016+0000 osd.4 (osd.4) 177 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:33:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:13 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:33:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:13 smithi076 conmon[62156]: 22:33:12.467856+0000 mon.a (mon.0) 2155 : cluster 2021-03-15T22:33:13.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:13 smithi076 conmon[62156]: [DBG] osdmap e842: 8 total, 8 up, 8 in 2021-03-15T22:33:13.778 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:13.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:13.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:13.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:13 smithi125 conmon[56739]: cluster 2021-03-15T22:33:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:13 smithi125 conmon[56739]: 2021-03-15T22:33:12.429016+0000 osd.4 2021-03-15T22:33:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:13 smithi125 conmon[56739]: (osd.4) 177 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:33:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:13 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:33:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:13 smithi125 conmon[56739]: 22:33:12.467856+0000 mon.a (mon.0) 2155 : cluster [DBG] osdmap e842: 8 total, 8 up, 8 in 2021-03-15T22:33:13.881 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:13.906 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:13.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:13.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:14.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:14.036 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:14.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:14.037 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:14.053 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:33:14.139 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:14.165 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:14.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:14.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:14.269 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:14.292 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:14.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:14.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:14.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:14.421 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:14.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:14.423 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:14.481 INFO:tasks.rados.rados.0.smithi076.stdout:3168: done (2 left) 2021-03-15T22:33:14.481 INFO:tasks.rados.rados.0.smithi076.stdout:3169: snap_remove snap 355 2021-03-15T22:33:14.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[56514]: cluster 2021-03-15T22:33:12.882160+0000 osd.5 (osd.5) 38 2021-03-15T22:33:14.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[56514]: : cluster [DBG] 3.16 scrub ok 2021-03-15T22:33:14.484 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[56514]: cluster 2021-03-15T22:33:13.186745+0000 mgr.y (mgr.14144) 973 : cluster [DBG] pgmap v1408: 148 pgs: 1 active+clean+wait, 147 active+clean; 543 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 5.4 MiB/s wr, 20 op/s; 16 MiB/s, 42 objects/s recovering 2021-03-15T22:33:14.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[56514]: cluster 2021-03-15T22:33:13.477006+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2021-03-15T22:33:14.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[56514]: cluster 2021-03-15T22:33:13.869141+0000 osd.2 (osd.2) 34 : cluster [DBG] 7.e scrub ok 2021-03-15T22:33:14.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[56514]: audit 2021-03-15T22:33:14.094171+0000 mon.a (mon.0) 2157 : audit [INF] from='client.? 172.21.15.76:0/4217323487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:14.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[62156]: cluster 2021-03-15T22:33:12.882160+0000 osd.5 ( 2021-03-15T22:33:14.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[62156]: osd.5) 38 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:33:14.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[62156]: cluster 2021-03-15T22:33:13.186745+0000 mgr.y (mgr.14144) 973 : cluster [DBG] pgmap v1408: 148 pgs: 1 active+clean+wait, 147 active+clean; 543 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 5.4 MiB/s wr, 20 op/s; 16 MiB/s, 42 objects/s recovering 2021-03-15T22:33:14.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[62156]: cluster 2021-03-15T22:33:13.477006+0000 mon.a (mon.0) 2156 : cluster [DBG] osdmap e843: 8 total, 8 up, 8 in 2021-03-15T22:33:14.486 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[62156]: cluster 2021-03-15T22:33:13.869141+0000 osd.2 (osd.2) 34 2021-03-15T22:33:14.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[62156]: : cluster [DBG] 7.e scrub ok 2021-03-15T22:33:14.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[62156]: audit 2021-03-15T22:33:14.094171+0000 mon.a (mon. 2021-03-15T22:33:14.487 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:14 smithi076 conmon[62156]: 0) 2157 : audit [INF] from='client.? 172.21.15.76:0/4217323487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:14.488 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:33:14.524 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:14.550 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:14.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:14.552 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:14.653 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:14.675 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:14.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:14.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:14.778 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: cluster 2021-03-15T22:33:12.882160+0000 osd.5 (osd.5) 38 : cluster 2021-03-15T22:33:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: [DBG] 3.16 scrub ok 2021-03-15T22:33:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: cluster 2021-03-15T22:33:13.186745+0000 mgr.y (mgr.14144) 973 : cluster [DBG] pgmap v1408: 148 pgs: 1 active+clean+wait, 147 active+clean; 543 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 8.2 MiB/s rd, 5.4 MiB/s wr, 20 op/s; 16 MiB/s, 42 objects/s recovering 2021-03-15T22:33:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: cluster 2021-03-15T22:33:13.477006 2021-03-15T22:33:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2156 : cluster [DBG] 2021-03-15T22:33:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: osdmap e843: 8 total, 8 up, 8 in 2021-03-15T22:33:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:33:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: -15T22:33:13.869141 2021-03-15T22:33:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: +0000 osd.2 (osd.2) 34 : cluster [DBG] 2021-03-15T22:33:14.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: 7.e scrub ok 2021-03-15T22:33:14.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:33:14.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: -15T22:33:14.094171+0000 2021-03-15T22:33:14.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: mon.a (mon.0) 2157 : audit [INF] 2021-03-15T22:33:14.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:14 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/4217323487' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:14.803 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:14.805 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:14.906 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:14.932 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:14.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:14.933 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:15.034 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:15.061 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:15.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:15.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:15.163 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:15.188 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:15.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:15.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:15.289 INFO:tasks.rados.rados.0.smithi076.stdout:3169: done (2 left) 2021-03-15T22:33:15.289 INFO:tasks.rados.rados.0.smithi076.stdout:3170: read oid 38 snap -1 2021-03-15T22:33:15.289 INFO:tasks.rados.rados.0.smithi076.stdout:3170: expect (ObjNum 1055 snap 355 seq_num 1055) 2021-03-15T22:33:15.290 INFO:tasks.rados.rados.0.smithi076.stdout:3171: rollback oid 27 current snap is 371 2021-03-15T22:33:15.290 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 368 2021-03-15T22:33:15.290 INFO:tasks.rados.rados.0.smithi076.stdout:3172: delete oid 13 current snap is 371 2021-03-15T22:33:15.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:15.298 INFO:tasks.rados.rados.0.smithi076.stdout:3171: finishing rollback tid 0 to smithi07694878-27 2021-03-15T22:33:15.299 INFO:tasks.rados.rados.0.smithi076.stdout:3171: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:33:15.299 INFO:tasks.rados.rados.0.smithi076.stdout:3171: finishing rollback tid 2 to smithi07694878-27 2021-03-15T22:33:15.299 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1991 (ObjNum 896 snap 302 seq_num 896) dirty exists 2021-03-15T22:33:15.314 INFO:tasks.rados.rados.0.smithi076.stdout:3170: expect (ObjNum 1055 snap 355 seq_num 1055) 2021-03-15T22:33:15.330 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:15.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:15.332 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:15.433 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:15.459 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:15.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:15.461 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:15.488 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:15.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[56514]: cluster 2021-03-15T22:33:14.283758+0000 osd.6 (osd.6) 122 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:33:15.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[56514]: cluster 2021-03-15T22:33:14.474644+0000 mon.a (mon.0) 2158 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:33:15.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[56514]: cluster 2021-03-15T22:33:14.474666+0000 mon.a (mon.0) 2159 : cluster [INF] Cluster is now healthy 2021-03-15T22:33:15.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[56514]: audit 2021-03-15T22:33:14.478944+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? 172.21.15.76:0/4217323487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:33:15.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[56514]: cluster 2021-03-15T22:33:14.479006+0000 mon.a (mon.0) 2161 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2021-03-15T22:33:15.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[56514]: cluster 2021-03-15T22:33:14.622432+0000 osd.3 (osd.3) 152 : cluster [DBG] 4.14 deep-scrub ok 2021-03-15T22:33:15.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[56514]: cluster 2021-03-15T22:33:14.888114+0000 osd.2 (osd.2) 35 : cluster [DBG] 5.a scrub ok 2021-03-15T22:33:15.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[56514]: cluster 2021-03-15T22:33:15.286486+0000 mon.a (mon.0) 2162 : cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2021-03-15T22:33:15.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: cluster 2021-03-15T22:33:14. 2021-03-15T22:33:15.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: 283758+0000 osd.6 (osd.6) 122 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:33:15.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:33:15.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: T22:33:14.474644+0000 mon.a (mon.0) 2158 : cluster [INF] 2021-03-15T22:33:15.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:33:15.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:33:15.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: -15T22:33:14.474666+0000 2021-03-15T22:33:15.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: mon.a (mon.0) 2159 : cluster [INF] 2021-03-15T22:33:15.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: Cluster is now healthy 2021-03-15T22:33:15.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:33:15.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: T22:33:14.478944+0000 mon.a 2021-03-15T22:33:15.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: (mon.0) 2160 : audit [INF] from='client.? 172.21.15.76:0/4217323487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:33:15.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: 2021-03-15T22:33:15.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:33:15.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: :33:14.479006+0000 mon.a (mon. 2021-03-15T22:33:15.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: 0) 2161 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2021-03-15T22:33:15.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: cluster 2021-03-15T22:33:15.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: 2021-03-15T22:33:14. 2021-03-15T22:33:15.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: 622432+0000 osd.3 (osd.3) 152 : 2021-03-15T22:33:15.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: cluster [DBG] 4.14 deep-scrub ok 2021-03-15T22:33:15.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:15.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: -03-15T22:33:14.888114 2021-03-15T22:33:15.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: +0000 osd.2 (osd.2) 35 : cluster 2021-03-15T22:33:15.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: [DBG] 5.a scrub ok 2021-03-15T22:33:15.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:15.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: -03-15T22:33:15. 2021-03-15T22:33:15.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: 286486+0000 mon.a (mon.0) 2162 : 2021-03-15T22:33:15.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:15 smithi076 conmon[62156]: cluster [DBG] osdmap e845: 8 total, 8 up, 8 in 2021-03-15T22:33:15.536 INFO:tasks.rados.rados.0.smithi076.stdout:3170: done (4 left) 2021-03-15T22:33:15.536 INFO:tasks.rados.rados.0.smithi076.stdout:3171: done (3 left) 2021-03-15T22:33:15.536 INFO:tasks.rados.rados.0.smithi076.stdout:3172: done (2 left) 2021-03-15T22:33:15.536 INFO:tasks.rados.rados.0.smithi076.stdout:3173: read oid 34 snap -1 2021-03-15T22:33:15.537 INFO:tasks.rados.rados.0.smithi076.stdout:3173: expect (ObjNum 1064 snap 361 seq_num 1064) 2021-03-15T22:33:15.537 INFO:tasks.rados.rados.0.smithi076.stdout:3174: snap_remove snap 347 2021-03-15T22:33:15.563 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:15.587 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:15.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:15.589 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:15.689 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:33:15.692 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:15.693 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:33:15.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:15.722 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:15.725 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: cluster 2021-03-15T22:33:14.283758+0000 osd.6 (osd.6) 122 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:33:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: cluster 2021-03-15T22:33:14.474644+0000 mon.a (mon.0) 2158 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:33:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: cluster 2021-03-15T22:33:14.474666+0000 mon.a 2021-03-15T22:33:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: (mon.0) 2159 : cluster [INF] Cluster is now healthy 2021-03-15T22:33:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:33:15.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: 33:14.478944+0000 mon.a (mon.0) 2160 : audit [INF] from='client.? 172.21.15.76:0/4217323487' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:33:15.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: cluster 2021-03-15T22:33:14.479006+0000 2021-03-15T22:33:15.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: mon.a (mon.0) 2161 : cluster [DBG] osdmap e844: 8 total, 8 up, 8 in 2021-03-15T22:33:15.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:33:15.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: 33:14.622432+0000 osd.3 (osd.3) 152 : cluster [DBG] 2021-03-15T22:33:15.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: 4.14 deep-scrub ok 2021-03-15T22:33:15.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:33:15.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: -15T22:33:14.888114+0000 osd.2 (osd.2) 35 : cluster [DBG] 5.a scrub ok 2021-03-15T22:33:15.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:33:15.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: 22:33:15.286486+0000 mon.a (mon.0) 2162 : cluster 2021-03-15T22:33:15.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:15 smithi125 conmon[56739]: [DBG] osdmap e845: 8 total, 8 up, 8 in 2021-03-15T22:33:15.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:15.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:15.859 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:15.860 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:15.963 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:15.993 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:15.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:16.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:16.106 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:16.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:16.134 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:16.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:16.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:16.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:16.263 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:16.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:16.364 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:16.391 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:16.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:16.393 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:16.494 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:16.514 INFO:tasks.rados.rados.0.smithi076.stdout:3174: done (3 left) 2021-03-15T22:33:16.515 INFO:tasks.rados.rados.0.smithi076.stdout:3175: snap_remove snap 348 2021-03-15T22:33:16.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:16.528 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:16.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:16.630 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:16.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:16.657 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:16.658 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:16.759 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:16.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[56514]: cluster 2021-03-15T22:33:15.174206+0000 osd.1 (osd. 2021-03-15T22:33:16.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[56514]: 1) 164 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[56514]: cluster 2021-03-15T22:33:15.175898+0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[56514]: cluster 2021-03-15T22:33:15.187210+0000 mgr.y (mgr.14144) 974 : cluster [DBG] pgmap v1411: 148 pgs: 1 active+clean+wait, 147 active+clean; 549 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 3.7 MiB/s, 2 objects/s recovering 2021-03-15T22:33:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[56514]: cluster 2021-03-15T22:33:15.406489+0000 osd.1 (osd.1) 166 : cluster [DBG] 5.c scrub ok 2021-03-15T22:33:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[56514]: cluster 2021-03-15T22:33:15.469512+0000 osd.7 (osd.7) 158 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:33:16.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[56514]: cluster 2021-03-15T22:33:15.492008+0000 osd.0 (osd.0) 138 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:33:16.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[56514]: cluster 2021-03-15T22:33:15.617884+0000 osd.3 (osd.3) 153 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:33:16.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[56514]: audit 2021-03-15T22:33:15.670582+0000 mgr.y (mgr.14144) 975 : audit [DBG] from='client.25426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:33:16.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[56514]: audit 2021-03-15T22:33:16.097294+0000 mon.a (mon.0) 2163 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:33:16.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: cluster 2021-03-15T22:33:15.174206+0000 2021-03-15T22:33:16.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: osd.1 (osd.1) 164 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:16.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: cluster 2021-03-15T22:33:15.175898 2021-03-15T22:33:16.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:16.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:33:16.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: :33:15.187210+0000 mgr.y (mgr.14144) 2021-03-15T22:33:16.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: 974 : cluster [DBG] pgmap v1411: 148 pgs: 1 active+clean+wait, 147 active+clean; 549 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 3.7 MiB/s, 2 objects/s recovering 2021-03-15T22:33:16.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:33:16.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: 22:33:15.406489+0000 osd.1 (osd. 2021-03-15T22:33:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: 1) 166 : cluster [DBG] 5.c scrub ok 2021-03-15T22:33:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:33:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: -15T22:33:15.469512+0000 osd.7 2021-03-15T22:33:16.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: (osd.7) 158 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:33:16.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: 2021-03-15T22:33:16.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: cluster 2021-03-15T22:33:16.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: 2021-03-15T22:33:15.492008+0000 osd.0 ( 2021-03-15T22:33:16.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: osd.0) 138 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:33:16.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:33:16.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: 03-15T22:33:15.617884+0000 2021-03-15T22:33:16.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: osd.3 (osd.3) 153 : cluster [DBG] 2021-03-15T22:33:16.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: 4.1d scrub ok 2021-03-15T22:33:16.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: audit 2021-03-15T22:33:16.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: 2021-03-15T22:33:15.670582+0000 mgr.y 2021-03-15T22:33:16.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: (mgr.14144) 975 : audit [DBG] from='client.25426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:33:16.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: 2021-03-15T22:33:16.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: audit 2021-03-15T22:33:16.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: 2021-03-15T22:33:16.097294+0000 mon.a ( 2021-03-15T22:33:16.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:16 smithi076 conmon[62156]: mon.0) 2163 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:33:16.783 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:16.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:16.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:16.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: cluster 2021-03-15T22:33:15.174206+0000 osd.1 (osd.1) 164 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: cluster 2021-03-15T22:33:15.175898+0000 osd.1 (osd.1) 165 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: cluster 2021-03-15T22:33:15.187210+0000 mgr.y ( 2021-03-15T22:33:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: mgr.14144) 974 : cluster [DBG] pgmap v1411: 148 pgs: 1 active+clean+wait, 147 active+clean; 549 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 3.7 MiB/s, 2 objects/s recovering 2021-03-15T22:33:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:33:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: :33:15.406489+0000 osd.1 (osd.1) 166 : cluster [DBG] 5.c scrub ok 2021-03-15T22:33:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: cluster 2021-03-15T22:33:15.469512+0000 osd.7 (osd 2021-03-15T22:33:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: .7) 158 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:33:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: cluster 2021-03-15T22:33:15.492008+0000 osd.0 2021-03-15T22:33:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: (osd.0) 138 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:33:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: cluster 2021-03-15T22:33:15. 2021-03-15T22:33:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: 617884+0000 osd.3 (osd.3) 153 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:33:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:33:16.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: 33:15.670582+0000 mgr.y (mgr.14144) 975 : audit [DBG] from='client.25426 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:33:16.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:33:16.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: -15T22:33:16.097294+0000 mon.a (mon.0) 2021-03-15T22:33:16.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:16 smithi125 conmon[56739]: 2163 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:33:16.827 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1412,"stamp":"2021-03-15T22:33:15.283816+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":576106284,"num_objects":567,"num_object_clones":202,"num_object_copies":1701,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":567,"num_whiteouts":11,"num_read":8273,"num_read_kb":6013344,"num_write":5173,"num_write_kb":1651891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3610,"num_bytes_recovered":1316807880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":18876,"ondisk_log_size":18876,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1638592,"kb_used_data":1333000,"kb_used_omap":273,"kb_used_meta":301294,"kb_avail":748322624,"statfs":{"total":767960285184,"available":766282366976,"internally_reserved":0,"allocated":1364992000,"data_stored":1790477222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":308525946},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":13934233,"num_objects":6,"num_object_clones":6,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":-6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":43,"num_read_kb":47587,"num_write":14,"num_write_kb":10622,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":15403742,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.822072"},"pg_stats":[{"pgid":"5.19","version":"317'7","reported_seq":"200","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.475755+0000","last_change":"2021-03-15T22:32:07.910488+0000","last_active":"2021-03-15T22:33:11.475755+0000","last_peered":"2021-03-15T22:33:11.475755+0000","last_clean":"2021-03-15T22:33:11.475755+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:33:11.475755+0000","last_undegraded":"2021-03-15T22:33:11.475755+0000","last_fullsized":"2021-03-15T22:33:11.475755+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:32:07.910427+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:32:07.910427+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"842","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441721+0000","last_change":"2021-03-15T22:30:41.431150+0000","last_active":"2021-03-15T22:33:10.441721+0000","last_peered":"2021-03-15T22:33:10.441721+0000","last_clean":"2021-03-15T22:33:10.441721+0000","last_became_active":"2021-03-15T22:29:51.225252+0000","last_became_peered":"2021-03-15T22:29:51.225252+0000","last_unstale":"2021-03-15T22:33:10.441721+0000","last_undegraded":"2021-03-15T22:33:10.441721+0000","last_fullsized":"2021-03-15T22:33:10.441721+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.431052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"342","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.357260+0000","last_change":"2021-03-15T22:32:31.680174+0000","last_active":"2021-03-15T22:33:09.357260+0000","last_peered":"2021-03-15T22:33:09.357260+0000","last_clean":"2021-03-15T22:33:09.357260+0000","last_became_active":"2021-03-15T22:30:32.251610+0000","last_became_peered":"2021-03-15T22:30:32.251610+0000","last_unstale":"2021-03-15T22:33:09.357260+0000","last_undegraded":"2021-03-15T22:33:09.357260+0000","last_fullsized":"2021-03-15T22:33:09.357260+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:32:31.680111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.680111+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":873,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"934","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441633+0000","last_change":"2021-03-15T22:33:06.085397+0000","last_active":"2021-03-15T22:33:10.441633+0000","last_peered":"2021-03-15T22:33:10.441633+0000","last_clean":"2021-03-15T22:33:10.441633+0000","last_became_active":"2021-03-15T22:33:04.345393+0000","last_became_peered":"2021-03-15T22:33:04.345393+0000","last_unstale":"2021-03-15T22:33:10.441633+0000","last_undegraded":"2021-03-15T22:33:10.441633+0000","last_fullsized":"2021-03-15T22:33:10.441633+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.177019+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"143","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441454+0000","last_change":"2021-03-15T22:33:04.345933+0000","last_active":"2021-03-15T22:33:10.441454+0000","last_peered":"2021-03-15T22:33:10.441454+0000","last_clean":"2021-03-15T22:33:10.441454+0000","last_became_active":"2021-03-15T22:33:04.345518+0000","last_became_peered":"2021-03-15T22:33:04.345518+0000","last_unstale":"2021-03-15T22:33:10.441454+0000","last_undegraded":"2021-03-15T22:33:10.441454+0000","last_fullsized":"2021-03-15T22:33:10.441454+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.338100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.338100+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"283","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.446021+0000","last_change":"2021-03-15T22:33:04.332112+0000","last_active":"2021-03-15T22:33:10.446021+0000","last_peered":"2021-03-15T22:33:10.446021+0000","last_clean":"2021-03-15T22:33:10.446021+0000","last_became_active":"2021-03-15T22:33:04.331757+0000","last_became_peered":"2021-03-15T22:33:04.331757+0000","last_unstale":"2021-03-15T22:33:10.446021+0000","last_undegraded":"2021-03-15T22:33:10.446021+0000","last_fullsized":"2021-03-15T22:33:10.446021+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:05.262019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.262019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"570","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.447112+0000","last_change":"2021-03-15T22:33:04.334229+0000","last_active":"2021-03-15T22:33:10.447112+0000","last_peered":"2021-03-15T22:33:10.447112+0000","last_clean":"2021-03-15T22:33:10.447112+0000","last_became_active":"2021-03-15T22:33:04.333905+0000","last_became_peered":"2021-03-15T22:33:04.333905+0000","last_unstale":"2021-03-15T22:33:10.447112+0000","last_undegraded":"2021-03-15T22:33:10.447112+0000","last_fullsized":"2021-03-15T22:33:10.447112+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:58.617803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:58.617803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"574","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.445902+0000","last_change":"2021-03-15T22:32:10.653022+0000","last_active":"2021-03-15T22:33:10.445902+0000","last_peered":"2021-03-15T22:33:10.445902+0000","last_clean":"2021-03-15T22:33:10.445902+0000","last_became_active":"2021-03-15T22:30:32.258079+0000","last_became_peered":"2021-03-15T22:30:32.258079+0000","last_unstale":"2021-03-15T22:33:10.445902+0000","last_undegraded":"2021-03-15T22:33:10.445902+0000","last_fullsized":"2021-03-15T22:33:10.445902+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.652898+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"678","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.472692+0000","last_change":"2021-03-15T22:32:12.794180+0000","last_active":"2021-03-15T22:33:12.472692+0000","last_peered":"2021-03-15T22:33:12.472692+0000","last_clean":"2021-03-15T22:33:12.472692+0000","last_became_active":"2021-03-15T22:31:20.246770+0000","last_became_peered":"2021-03-15T22:31:20.246770+0000","last_unstale":"2021-03-15T22:33:12.472692+0000","last_undegraded":"2021-03-15T22:33:12.472692+0000","last_fullsized":"2021-03-15T22:33:12.472692+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.794113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.794113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"305","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.355429+0000","last_change":"2021-03-15T22:32:01.452949+0000","last_active":"2021-03-15T22:33:09.355429+0000","last_peered":"2021-03-15T22:33:09.355429+0000","last_clean":"2021-03-15T22:33:09.355429+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:33:09.355429+0000","last_undegraded":"2021-03-15T22:33:09.355429+0000","last_fullsized":"2021-03-15T22:33:09.355429+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:01.452885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:32:01.452885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"232","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.471687+0000","last_change":"2021-03-15T22:33:04.326563+0000","last_active":"2021-03-15T22:33:12.471687+0000","last_peered":"2021-03-15T22:33:12.471687+0000","last_clean":"2021-03-15T22:33:12.471687+0000","last_became_active":"2021-03-15T22:33:04.325611+0000","last_became_peered":"2021-03-15T22:33:04.325611+0000","last_unstale":"2021-03-15T22:33:12.471687+0000","last_undegraded":"2021-03-15T22:33:12.471687+0000","last_fullsized":"2021-03-15T22:33:12.471687+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.146408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.146408+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"985","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441856+0000","last_change":"2021-03-15T22:30:28.328526+0000","last_active":"2021-03-15T22:33:10.441856+0000","last_peered":"2021-03-15T22:33:10.441856+0000","last_clean":"2021-03-15T22:33:10.441856+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:33:10.441856+0000","last_undegraded":"2021-03-15T22:33:10.441856+0000","last_fullsized":"2021-03-15T22:33:10.441856+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:30:28.328458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.328458+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"125","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.458990+0000","last_change":"2021-03-15T22:32:38.476370+0000","last_active":"2021-03-15T22:33:11.458990+0000","last_peered":"2021-03-15T22:33:11.458990+0000","last_clean":"2021-03-15T22:33:11.458990+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:33:11.458990+0000","last_undegraded":"2021-03-15T22:33:11.458990+0000","last_fullsized":"2021-03-15T22:33:11.458990+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.476269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.476269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"459","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.471453+0000","last_change":"2021-03-15T22:30:40.173603+0000","last_active":"2021-03-15T22:33:12.471453+0000","last_peered":"2021-03-15T22:33:12.471453+0000","last_clean":"2021-03-15T22:33:12.471453+0000","last_became_active":"2021-03-15T22:29:52.223704+0000","last_became_peered":"2021-03-15T22:29:52.223704+0000","last_unstale":"2021-03-15T22:33:12.471453+0000","last_undegraded":"2021-03-15T22:33:12.471453+0000","last_fullsized":"2021-03-15T22:33:12.471453+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:40.173519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:30:40.173519+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"219","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.471768+0000","last_change":"2021-03-15T22:32:14.756556+0000","last_active":"2021-03-15T22:33:12.471768+0000","last_peered":"2021-03-15T22:33:12.471768+0000","last_clean":"2021-03-15T22:33:12.471768+0000","last_became_active":"2021-03-15T22:29:51.234868+0000","last_became_peered":"2021-03-15T22:29:51.234868+0000","last_unstale":"2021-03-15T22:33:12.471768+0000","last_undegraded":"2021-03-15T22:33:12.471768+0000","last_fullsized":"2021-03-15T22:33:12.471768+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.756460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.756460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"928","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441979+0000","last_change":"2021-03-15T22:32:05.111125+0000","last_active":"2021-03-15T22:33:10.441979+0000","last_peered":"2021-03-15T22:33:10.441979+0000","last_clean":"2021-03-15T22:33:10.441979+0000","last_became_active":"2021-03-15T22:29:11.197059+0000","last_became_peered":"2021-03-15T22:29:11.197059+0000","last_unstale":"2021-03-15T22:33:10.441979+0000","last_undegraded":"2021-03-15T22:33:10.441979+0000","last_fullsized":"2021-03-15T22:33:10.441979+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:32:05.111031+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.111031+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"797","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.476601+0000","last_change":"2021-03-15T22:32:32.744807+0000","last_active":"2021-03-15T22:33:11.476601+0000","last_peered":"2021-03-15T22:33:11.476601+0000","last_clean":"2021-03-15T22:33:11.476601+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:33:11.476601+0000","last_undegraded":"2021-03-15T22:33:11.476601+0000","last_fullsized":"2021-03-15T22:33:11.476601+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:32:32.744716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.744716+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"858","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.442222+0000","last_change":"2021-03-15T22:33:04.329427+0000","last_active":"2021-03-15T22:33:10.442222+0000","last_peered":"2021-03-15T22:33:10.442222+0000","last_clean":"2021-03-15T22:33:10.442222+0000","last_became_active":"2021-03-15T22:33:04.329078+0000","last_became_peered":"2021-03-15T22:33:04.329078+0000","last_unstale":"2021-03-15T22:33:10.442222+0000","last_undegraded":"2021-03-15T22:33:10.442222+0000","last_fullsized":"2021-03-15T22:33:10.442222+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:31:37.466443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"309","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.356064+0000","last_change":"2021-03-15T22:32:41.673856+0000","last_active":"2021-03-15T22:33:09.356064+0000","last_peered":"2021-03-15T22:33:09.356064+0000","last_clean":"2021-03-15T22:33:09.356064+0000","last_became_active":"2021-03-15T22:28:12.519619+0000","last_became_peered":"2021-03-15T22:28:12.519619+0000","last_unstale":"2021-03-15T22:33:09.356064+0000","last_undegraded":"2021-03-15T22:33:09.356064+0000","last_fullsized":"2021-03-15T22:33:09.356064+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:32:41.673792+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.673792+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"715","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.468241+0000","last_change":"2021-03-15T22:32:10.001304+0000","last_active":"2021-03-15T22:33:12.468241+0000","last_peered":"2021-03-15T22:33:12.468241+0000","last_clean":"2021-03-15T22:33:12.468241+0000","last_became_active":"2021-03-15T22:30:32.250733+0000","last_became_peered":"2021-03-15T22:30:32.250733+0000","last_unstale":"2021-03-15T22:33:12.468241+0000","last_undegraded":"2021-03-15T22:33:12.468241+0000","last_fullsized":"2021-03-15T22:33:12.468241+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:32:10.001234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.001234+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"869","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.442084+0000","last_change":"2021-03-15T22:32:10.144209+0000","last_active":"2021-03-15T22:33:10.442084+0000","last_peered":"2021-03-15T22:33:10.442084+0000","last_clean":"2021-03-15T22:33:10.442084+0000","last_became_active":"2021-03-15T22:31:20.240898+0000","last_became_peered":"2021-03-15T22:31:20.240898+0000","last_unstale":"2021-03-15T22:33:10.442084+0000","last_undegraded":"2021-03-15T22:33:10.442084+0000","last_fullsized":"2021-03-15T22:33:10.442084+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.144132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.144132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"674","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.447781+0000","last_change":"2021-03-15T22:33:04.337162+0000","last_active":"2021-03-15T22:33:10.447781+0000","last_peered":"2021-03-15T22:33:10.447781+0000","last_clean":"2021-03-15T22:33:10.447781+0000","last_became_active":"2021-03-15T22:33:04.336804+0000","last_became_peered":"2021-03-15T22:33:04.336804+0000","last_unstale":"2021-03-15T22:33:10.447781+0000","last_undegraded":"2021-03-15T22:33:10.447781+0000","last_fullsized":"2021-03-15T22:33:10.447781+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.594717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:45.403769+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.594717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"830","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441886+0000","last_change":"2021-03-15T22:30:39.443682+0000","last_active":"2021-03-15T22:33:10.441886+0000","last_peered":"2021-03-15T22:33:10.441886+0000","last_clean":"2021-03-15T22:33:10.441886+0000","last_became_active":"2021-03-15T22:30:31.243315+0000","last_became_peered":"2021-03-15T22:30:31.243315+0000","last_unstale":"2021-03-15T22:33:10.441886+0000","last_undegraded":"2021-03-15T22:33:10.441886+0000","last_fullsized":"2021-03-15T22:33:10.441886+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:30:39.443613+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"285","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.446042+0000","last_change":"2021-03-15T22:29:54.564185+0000","last_active":"2021-03-15T22:33:10.446042+0000","last_peered":"2021-03-15T22:33:10.446042+0000","last_clean":"2021-03-15T22:33:10.446042+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:33:10.446042+0000","last_undegraded":"2021-03-15T22:33:10.446042+0000","last_fullsized":"2021-03-15T22:33:10.446042+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:29:54.564115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:29:54.564115+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"836'18","reported_seq":"342","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.352762+0000","last_change":"2021-03-15T22:33:08.354002+0000","last_active":"2021-03-15T22:33:09.352762+0000","last_peered":"2021-03-15T22:33:09.352762+0000","last_clean":"2021-03-15T22:33:09.352762+0000","last_became_active":"2021-03-15T22:33:08.353440+0000","last_became_peered":"2021-03-15T22:33:08.353440+0000","last_unstale":"2021-03-15T22:33:09.352762+0000","last_undegraded":"2021-03-15T22:33:09.352762+0000","last_fullsized":"2021-03-15T22:33:09.352762+0000","mapping_epoch":837,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":5,"last_scrub":"618'7","last_scrub_stamp":"2021-03-15T22:29:40.964753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:29:40.964753+0000","log_size":15,"ondisk_log_size":15,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"853","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459533+0000","last_change":"2021-03-15T22:32:37.525778+0000","last_active":"2021-03-15T22:33:11.459533+0000","last_peered":"2021-03-15T22:33:11.459533+0000","last_clean":"2021-03-15T22:33:11.459533+0000","last_became_active":"2021-03-15T22:30:32.243708+0000","last_became_peered":"2021-03-15T22:30:32.243708+0000","last_unstale":"2021-03-15T22:33:11.459533+0000","last_undegraded":"2021-03-15T22:33:11.459533+0000","last_fullsized":"2021-03-15T22:33:11.459533+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:37.525692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:37.525692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"870","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.475895+0000","last_change":"2021-03-15T22:32:43.703085+0000","last_active":"2021-03-15T22:33:11.475895+0000","last_peered":"2021-03-15T22:33:11.475895+0000","last_clean":"2021-03-15T22:33:11.475895+0000","last_became_active":"2021-03-15T22:28:59.264559+0000","last_became_peered":"2021-03-15T22:28:59.264559+0000","last_unstale":"2021-03-15T22:33:11.475895+0000","last_undegraded":"2021-03-15T22:33:11.475895+0000","last_fullsized":"2021-03-15T22:33:11.475895+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:43.703019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.703019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"405","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.356129+0000","last_change":"2021-03-15T22:30:41.759292+0000","last_active":"2021-03-15T22:33:09.356129+0000","last_peered":"2021-03-15T22:33:09.356129+0000","last_clean":"2021-03-15T22:33:09.356129+0000","last_became_active":"2021-03-15T22:28:59.277385+0000","last_became_peered":"2021-03-15T22:28:59.277385+0000","last_unstale":"2021-03-15T22:33:09.356129+0000","last_undegraded":"2021-03-15T22:33:09.356129+0000","last_fullsized":"2021-03-15T22:33:09.356129+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:30:41.759224+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.759224+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":2802,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"846","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.355754+0000","last_change":"2021-03-15T22:32:34.739698+0000","last_active":"2021-03-15T22:33:09.355754+0000","last_peered":"2021-03-15T22:33:09.355754+0000","last_clean":"2021-03-15T22:33:09.355754+0000","last_became_active":"2021-03-15T22:31:23.258435+0000","last_became_peered":"2021-03-15T22:31:23.258435+0000","last_unstale":"2021-03-15T22:33:09.355754+0000","last_undegraded":"2021-03-15T22:33:09.355754+0000","last_fullsized":"2021-03-15T22:33:09.355754+0000","mapping_epoch":733,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:32:34.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.739637+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"848","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.491320+0000","last_change":"2021-03-15T22:33:11.491281+0000","last_active":"2021-03-15T22:33:11.491320+0000","last_peered":"2021-03-15T22:33:11.491320+0000","last_clean":"2021-03-15T22:33:11.491320+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:33:11.491320+0000","last_undegraded":"2021-03-15T22:33:11.491320+0000","last_fullsized":"2021-03-15T22:33:11.491320+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:11.491199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:11.491199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"444","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.447174+0000","last_change":"2021-03-15T22:31:10.332382+0000","last_active":"2021-03-15T22:33:10.447174+0000","last_peered":"2021-03-15T22:33:10.447174+0000","last_clean":"2021-03-15T22:33:10.447174+0000","last_became_active":"2021-03-15T22:29:00.220073+0000","last_became_peered":"2021-03-15T22:29:00.220073+0000","last_unstale":"2021-03-15T22:33:10.447174+0000","last_undegraded":"2021-03-15T22:33:10.447174+0000","last_fullsized":"2021-03-15T22:33:10.447174+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:10.332319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.332319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"40","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.352715+0000","last_change":"2021-03-15T22:33:04.362082+0000","last_active":"2021-03-15T22:33:09.352715+0000","last_peered":"2021-03-15T22:33:09.352715+0000","last_clean":"2021-03-15T22:33:09.352715+0000","last_became_active":"2021-03-15T22:33:04.336100+0000","last_became_peered":"2021-03-15T22:33:04.336100+0000","last_unstale":"2021-03-15T22:33:09.352715+0000","last_undegraded":"2021-03-15T22:33:09.352715+0000","last_fullsized":"2021-03-15T22:33:09.352715+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:31:03.281344+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:03.281344+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"817","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459002+0000","last_change":"2021-03-15T22:30:59.480779+0000","last_active":"2021-03-15T22:33:11.459002+0000","last_peered":"2021-03-15T22:33:11.459002+0000","last_clean":"2021-03-15T22:33:11.459002+0000","last_became_active":"2021-03-15T22:30:31.230649+0000","last_became_peered":"2021-03-15T22:30:31.230649+0000","last_unstale":"2021-03-15T22:33:11.459002+0000","last_undegraded":"2021-03-15T22:33:11.459002+0000","last_fullsized":"2021-03-15T22:33:11.459002+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:59.480717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:30:59.480717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"16","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.352578+0000","last_change":"2021-03-15T22:33:04.336068+0000","last_active":"2021-03-15T22:33:09.352578+0000","last_peered":"2021-03-15T22:33:09.352578+0000","last_clean":"2021-03-15T22:33:09.352578+0000","last_became_active":"2021-03-15T22:33:04.335734+0000","last_became_peered":"2021-03-15T22:33:04.335734+0000","last_unstale":"2021-03-15T22:33:09.352578+0000","last_undegraded":"2021-03-15T22:33:09.352578+0000","last_fullsized":"2021-03-15T22:33:09.352578+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:06.296662+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:06.296662+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"344","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.357180+0000","last_change":"2021-03-15T22:31:33.136177+0000","last_active":"2021-03-15T22:33:09.357180+0000","last_peered":"2021-03-15T22:33:09.357180+0000","last_clean":"2021-03-15T22:33:09.357180+0000","last_became_active":"2021-03-15T22:29:51.226677+0000","last_became_peered":"2021-03-15T22:29:51.226677+0000","last_unstale":"2021-03-15T22:33:09.357180+0000","last_undegraded":"2021-03-15T22:33:09.357180+0000","last_fullsized":"2021-03-15T22:33:09.357180+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:33.136112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:31:33.136112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"349","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.445898+0000","last_change":"2021-03-15T22:33:05.222611+0000","last_active":"2021-03-15T22:33:10.445898+0000","last_peered":"2021-03-15T22:33:10.445898+0000","last_clean":"2021-03-15T22:33:10.445898+0000","last_became_active":"2021-03-15T22:33:04.342749+0000","last_became_peered":"2021-03-15T22:33:04.342749+0000","last_unstale":"2021-03-15T22:33:10.445898+0000","last_undegraded":"2021-03-15T22:33:10.445898+0000","last_fullsized":"2021-03-15T22:33:10.445898+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:30:07.486190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.486190+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":8764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"462","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.446317+0000","last_change":"2021-03-15T22:33:04.333373+0000","last_active":"2021-03-15T22:33:10.446317+0000","last_peered":"2021-03-15T22:33:10.446317+0000","last_clean":"2021-03-15T22:33:10.446317+0000","last_became_active":"2021-03-15T22:33:04.333077+0000","last_became_peered":"2021-03-15T22:33:04.333077+0000","last_unstale":"2021-03-15T22:33:10.446317+0000","last_undegraded":"2021-03-15T22:33:10.446317+0000","last_fullsized":"2021-03-15T22:33:10.446317+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.317817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"822","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459313+0000","last_change":"2021-03-15T22:33:04.330630+0000","last_active":"2021-03-15T22:33:11.459313+0000","last_peered":"2021-03-15T22:33:11.459313+0000","last_clean":"2021-03-15T22:33:11.459313+0000","last_became_active":"2021-03-15T22:33:04.330300+0000","last_became_peered":"2021-03-15T22:33:04.330300+0000","last_unstale":"2021-03-15T22:33:11.459313+0000","last_undegraded":"2021-03-15T22:33:11.459313+0000","last_fullsized":"2021-03-15T22:33:11.459313+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:05.522372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.522372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"219","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.882277+0000","last_change":"2021-03-15T22:33:12.882245+0000","last_active":"2021-03-15T22:33:12.882277+0000","last_peered":"2021-03-15T22:33:12.882277+0000","last_clean":"2021-03-15T22:33:12.882277+0000","last_became_active":"2021-03-15T22:29:51.220261+0000","last_became_peered":"2021-03-15T22:29:51.220261+0000","last_unstale":"2021-03-15T22:33:12.882277+0000","last_undegraded":"2021-03-15T22:33:12.882277+0000","last_fullsized":"2021-03-15T22:33:12.882277+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.882164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.882164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"680","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.446038+0000","last_change":"2021-03-15T22:33:04.352497+0000","last_active":"2021-03-15T22:33:10.446038+0000","last_peered":"2021-03-15T22:33:10.446038+0000","last_clean":"2021-03-15T22:33:10.446038+0000","last_became_active":"2021-03-15T22:33:04.331607+0000","last_became_peered":"2021-03-15T22:33:04.331607+0000","last_unstale":"2021-03-15T22:33:10.446038+0000","last_undegraded":"2021-03-15T22:33:10.446038+0000","last_fullsized":"2021-03-15T22:33:10.446038+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:31:07.330960+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:31:07.330960+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"324","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.355346+0000","last_change":"2021-03-15T22:31:43.405661+0000","last_active":"2021-03-15T22:33:09.355346+0000","last_peered":"2021-03-15T22:33:09.355346+0000","last_clean":"2021-03-15T22:33:09.355346+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:33:09.355346+0000","last_undegraded":"2021-03-15T22:33:09.355346+0000","last_fullsized":"2021-03-15T22:33:09.355346+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:43.405600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:31:43.405600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"634","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.446239+0000","last_change":"2021-03-15T22:32:37.690585+0000","last_active":"2021-03-15T22:33:10.446239+0000","last_peered":"2021-03-15T22:33:10.446239+0000","last_clean":"2021-03-15T22:33:10.446239+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:33:10.446239+0000","last_undegraded":"2021-03-15T22:33:10.446239+0000","last_fullsized":"2021-03-15T22:33:10.446239+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:37.690525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:32:37.690525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"844","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459378+0000","last_change":"2021-03-15T22:32:33.438804+0000","last_active":"2021-03-15T22:33:11.459378+0000","last_peered":"2021-03-15T22:33:11.459378+0000","last_clean":"2021-03-15T22:33:11.459378+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:33:11.459378+0000","last_undegraded":"2021-03-15T22:33:11.459378+0000","last_fullsized":"2021-03-15T22:33:11.459378+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.438742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.438742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"602","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.476362+0000","last_change":"2021-03-15T22:33:05.212418+0000","last_active":"2021-03-15T22:33:11.476362+0000","last_peered":"2021-03-15T22:33:11.476362+0000","last_clean":"2021-03-15T22:33:11.476362+0000","last_became_active":"2021-03-15T22:33:04.336766+0000","last_became_peered":"2021-03-15T22:33:04.336766+0000","last_unstale":"2021-03-15T22:33:11.476362+0000","last_undegraded":"2021-03-15T22:33:11.476362+0000","last_fullsized":"2021-03-15T22:33:11.476362+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:32:08.879999+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:08.879999+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"466","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.445818+0000","last_change":"2021-03-15T22:32:06.670385+0000","last_active":"2021-03-15T22:33:10.445818+0000","last_peered":"2021-03-15T22:33:10.445818+0000","last_clean":"2021-03-15T22:33:10.445818+0000","last_became_active":"2021-03-15T22:29:51.218982+0000","last_became_peered":"2021-03-15T22:29:51.218982+0000","last_unstale":"2021-03-15T22:33:10.445818+0000","last_undegraded":"2021-03-15T22:33:10.445818+0000","last_fullsized":"2021-03-15T22:33:10.445818+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:06.670324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:32:06.670324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"887","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.476242+0000","last_change":"2021-03-15T22:32:30.719489+0000","last_active":"2021-03-15T22:33:11.476242+0000","last_peered":"2021-03-15T22:33:11.476242+0000","last_clean":"2021-03-15T22:33:11.476242+0000","last_became_active":"2021-03-15T22:29:51.222254+0000","last_became_peered":"2021-03-15T22:29:51.222254+0000","last_unstale":"2021-03-15T22:33:11.476242+0000","last_undegraded":"2021-03-15T22:33:11.476242+0000","last_fullsized":"2021-03-15T22:33:11.476242+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:30.719431+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.719431+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"274","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.468475+0000","last_change":"2021-03-15T22:33:08.884945+0000","last_active":"2021-03-15T22:33:12.468475+0000","last_peered":"2021-03-15T22:33:12.468475+0000","last_clean":"2021-03-15T22:33:12.468475+0000","last_became_active":"2021-03-15T22:33:04.341942+0000","last_became_peered":"2021-03-15T22:33:04.341942+0000","last_unstale":"2021-03-15T22:33:12.468475+0000","last_undegraded":"2021-03-15T22:33:12.468475+0000","last_fullsized":"2021-03-15T22:33:12.468475+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:08.884878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.884878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"922","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441892+0000","last_change":"2021-03-15T22:33:05.224785+0000","last_active":"2021-03-15T22:33:10.441892+0000","last_peered":"2021-03-15T22:33:10.441892+0000","last_clean":"2021-03-15T22:33:10.441892+0000","last_became_active":"2021-03-15T22:33:04.340460+0000","last_became_peered":"2021-03-15T22:33:04.340460+0000","last_unstale":"2021-03-15T22:33:10.441892+0000","last_undegraded":"2021-03-15T22:33:10.441892+0000","last_fullsized":"2021-03-15T22:33:10.441892+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:31:42.396996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.396996+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"318","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.356339+0000","last_change":"2021-03-15T22:30:42.771039+0000","last_active":"2021-03-15T22:33:09.356339+0000","last_peered":"2021-03-15T22:33:09.356339+0000","last_clean":"2021-03-15T22:33:09.356339+0000","last_became_active":"2021-03-15T22:28:59.276812+0000","last_became_peered":"2021-03-15T22:28:59.276812+0000","last_unstale":"2021-03-15T22:33:09.356339+0000","last_undegraded":"2021-03-15T22:33:09.356339+0000","last_fullsized":"2021-03-15T22:33:09.356339+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:42.770976+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.770976+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"808'9","reported_seq":"265","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.471793+0000","last_change":"2021-03-15T22:32:38.881054+0000","last_active":"2021-03-15T22:33:12.471793+0000","last_peered":"2021-03-15T22:33:12.471793+0000","last_clean":"2021-03-15T22:33:12.471793+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:33:12.471793+0000","last_undegraded":"2021-03-15T22:33:12.471793+0000","last_fullsized":"2021-03-15T22:33:12.471793+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9","last_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_deep_scrub":"808'9","last_deep_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.880993+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"820","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459141+0000","last_change":"2021-03-15T22:33:07.409793+0000","last_active":"2021-03-15T22:33:11.459141+0000","last_peered":"2021-03-15T22:33:11.459141+0000","last_clean":"2021-03-15T22:33:11.459141+0000","last_became_active":"2021-03-15T22:30:31.230560+0000","last_became_peered":"2021-03-15T22:30:31.230560+0000","last_unstale":"2021-03-15T22:33:11.459141+0000","last_undegraded":"2021-03-15T22:33:11.459141+0000","last_fullsized":"2021-03-15T22:33:11.459141+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:35.423559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:30:35.423559+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"556'24","reported_seq":"628","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.447000+0000","last_change":"2021-03-15T22:30:02.548134+0000","last_active":"2021-03-15T22:33:10.447000+0000","last_peered":"2021-03-15T22:33:10.447000+0000","last_clean":"2021-03-15T22:33:10.447000+0000","last_became_active":"2021-03-15T22:29:00.226652+0000","last_became_peered":"2021-03-15T22:29:00.226652+0000","last_unstale":"2021-03-15T22:33:10.447000+0000","last_undegraded":"2021-03-15T22:33:10.447000+0000","last_fullsized":"2021-03-15T22:33:10.447000+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'24","last_scrub_stamp":"2021-03-15T22:30:02.548069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:02.548069+0000","log_size":24,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"855","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.442475+0000","last_change":"2021-03-15T22:31:41.407408+0000","last_active":"2021-03-15T22:33:10.442475+0000","last_peered":"2021-03-15T22:33:10.442475+0000","last_clean":"2021-03-15T22:33:10.442475+0000","last_became_active":"2021-03-15T22:30:32.247782+0000","last_became_peered":"2021-03-15T22:30:32.247782+0000","last_unstale":"2021-03-15T22:33:10.442475+0000","last_undegraded":"2021-03-15T22:33:10.442475+0000","last_fullsized":"2021-03-15T22:33:10.442475+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:41.407324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.407324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"683'14","reported_seq":"534","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.446868+0000","last_change":"2021-03-15T22:31:41.551496+0000","last_active":"2021-03-15T22:33:10.446868+0000","last_peered":"2021-03-15T22:33:10.446868+0000","last_clean":"2021-03-15T22:33:10.446868+0000","last_became_active":"2021-03-15T22:30:32.252383+0000","last_became_peered":"2021-03-15T22:30:32.252383+0000","last_unstale":"2021-03-15T22:33:10.446868+0000","last_undegraded":"2021-03-15T22:33:10.446868+0000","last_fullsized":"2021-03-15T22:33:10.446868+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'14","last_scrub_stamp":"2021-03-15T22:31:41.551435+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.551435+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"323","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.355592+0000","last_change":"2021-03-15T22:32:03.502945+0000","last_active":"2021-03-15T22:33:09.355592+0000","last_peered":"2021-03-15T22:33:09.355592+0000","last_clean":"2021-03-15T22:33:09.355592+0000","last_became_active":"2021-03-15T22:28:59.265822+0000","last_became_peered":"2021-03-15T22:28:59.265822+0000","last_unstale":"2021-03-15T22:33:09.355592+0000","last_undegraded":"2021-03-15T22:33:09.355592+0000","last_fullsized":"2021-03-15T22:33:09.355592+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:03.502854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:32:03.502854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"237","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.471555+0000","last_change":"2021-03-15T22:32:41.879032+0000","last_active":"2021-03-15T22:33:12.471555+0000","last_peered":"2021-03-15T22:33:12.471555+0000","last_clean":"2021-03-15T22:33:12.471555+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:33:12.471555+0000","last_undegraded":"2021-03-15T22:33:12.471555+0000","last_fullsized":"2021-03-15T22:33:12.471555+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:32:41.878949+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.878949+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"119","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.447485+0000","last_change":"2021-03-15T22:31:20.257902+0000","last_active":"2021-03-15T22:33:10.447485+0000","last_peered":"2021-03-15T22:33:10.447485+0000","last_clean":"2021-03-15T22:33:10.447485+0000","last_became_active":"2021-03-15T22:31:20.256557+0000","last_became_peered":"2021-03-15T22:31:20.256557+0000","last_unstale":"2021-03-15T22:33:10.447485+0000","last_undegraded":"2021-03-15T22:33:10.447485+0000","last_fullsized":"2021-03-15T22:33:10.447485+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:11.436723+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.436723+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"618","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.468065+0000","last_change":"2021-03-15T22:32:12.036589+0000","last_active":"2021-03-15T22:33:12.468065+0000","last_peered":"2021-03-15T22:33:12.468065+0000","last_clean":"2021-03-15T22:33:12.468065+0000","last_became_active":"2021-03-15T22:30:31.254532+0000","last_became_peered":"2021-03-15T22:30:31.254532+0000","last_unstale":"2021-03-15T22:33:12.468065+0000","last_undegraded":"2021-03-15T22:33:12.468065+0000","last_fullsized":"2021-03-15T22:33:12.468065+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.036522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.036522+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"831","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.458989+0000","last_change":"2021-03-15T22:33:04.330570+0000","last_active":"2021-03-15T22:33:11.458989+0000","last_peered":"2021-03-15T22:33:11.458989+0000","last_clean":"2021-03-15T22:33:11.458989+0000","last_became_active":"2021-03-15T22:33:04.330195+0000","last_became_peered":"2021-03-15T22:33:04.330195+0000","last_unstale":"2021-03-15T22:33:11.458989+0000","last_undegraded":"2021-03-15T22:33:11.458989+0000","last_fullsized":"2021-03-15T22:33:11.458989+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:34.511804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:34.511804+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"363","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.355884+0000","last_change":"2021-03-15T22:31:04.009824+0000","last_active":"2021-03-15T22:33:09.355884+0000","last_peered":"2021-03-15T22:33:09.355884+0000","last_clean":"2021-03-15T22:33:09.355884+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:33:09.355884+0000","last_undegraded":"2021-03-15T22:33:09.355884+0000","last_fullsized":"2021-03-15T22:33:09.355884+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:31:04.009748+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.009748+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"639","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.472043+0000","last_change":"2021-03-15T22:32:11.834186+0000","last_active":"2021-03-15T22:33:12.472043+0000","last_peered":"2021-03-15T22:33:12.472043+0000","last_clean":"2021-03-15T22:33:12.472043+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:33:12.472043+0000","last_undegraded":"2021-03-15T22:33:12.472043+0000","last_fullsized":"2021-03-15T22:33:12.472043+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:11.834104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:11.834104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"825","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459263+0000","last_change":"2021-03-15T22:33:04.332819+0000","last_active":"2021-03-15T22:33:11.459263+0000","last_peered":"2021-03-15T22:33:11.459263+0000","last_clean":"2021-03-15T22:33:11.459263+0000","last_became_active":"2021-03-15T22:33:04.331013+0000","last_became_peered":"2021-03-15T22:33:04.331013+0000","last_unstale":"2021-03-15T22:33:11.459263+0000","last_undegraded":"2021-03-15T22:33:11.459263+0000","last_fullsized":"2021-03-15T22:33:11.459263+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:35.559607+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:35.559607+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"16","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.352183+0000","last_change":"2021-03-15T22:33:04.337202+0000","last_active":"2021-03-15T22:33:09.352183+0000","last_peered":"2021-03-15T22:33:09.352183+0000","last_clean":"2021-03-15T22:33:09.352183+0000","last_became_active":"2021-03-15T22:33:04.336886+0000","last_became_peered":"2021-03-15T22:33:04.336886+0000","last_unstale":"2021-03-15T22:33:09.352183+0000","last_undegraded":"2021-03-15T22:33:09.352183+0000","last_fullsized":"2021-03-15T22:33:09.352183+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.697935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:39.697935+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.697935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"470","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.447405+0000","last_change":"2021-03-15T22:32:01.586874+0000","last_active":"2021-03-15T22:33:10.447405+0000","last_peered":"2021-03-15T22:33:10.447405+0000","last_clean":"2021-03-15T22:33:10.447405+0000","last_became_active":"2021-03-15T22:29:11.204254+0000","last_became_peered":"2021-03-15T22:29:11.204254+0000","last_unstale":"2021-03-15T22:33:10.447405+0000","last_undegraded":"2021-03-15T22:33:10.447405+0000","last_fullsized":"2021-03-15T22:33:10.447405+0000","mapping_epoch":601,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":602,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:32:01.586755+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:32:01.586755+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":20,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"842'1092","reported_seq":"3779","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.475918+0000","last_change":"2021-03-15T22:33:12.476107+0000","last_active":"2021-03-15T22:33:12.475918+0000","last_peered":"2021-03-15T22:33:12.475918+0000","last_clean":"2021-03-15T22:33:12.475918+0000","last_became_active":"2021-03-15T22:33:08.355490+0000","last_became_peered":"2021-03-15T22:33:08.355490+0000","last_unstale":"2021-03-15T22:33:12.475918+0000","last_undegraded":"2021-03-15T22:33:12.475918+0000","last_fullsized":"2021-03-15T22:33:12.475918+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'1088","last_scrub_stamp":"2021-03-15T22:33:10.457078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:10.457078+0000","log_size":1092,"ondisk_log_size":1092,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35112289,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":417,"num_read_kb":468610,"num_write":281,"num_write_kb":93929,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":84869361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2b8","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"781","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.472548+0000","last_change":"2021-03-15T22:33:04.340976+0000","last_active":"2021-03-15T22:33:12.472548+0000","last_peered":"2021-03-15T22:33:12.472548+0000","last_clean":"2021-03-15T22:33:12.472548+0000","last_became_active":"2021-03-15T22:33:04.340352+0000","last_became_peered":"2021-03-15T22:33:04.340352+0000","last_unstale":"2021-03-15T22:33:12.472548+0000","last_undegraded":"2021-03-15T22:33:12.472548+0000","last_fullsized":"2021-03-15T22:33:12.472548+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:07.417736+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:07.417736+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"57'2","reported_seq":"846","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459565+0000","last_change":"2021-03-15T22:31:58.607690+0000","last_active":"2021-03-15T22:33:11.459565+0000","last_peered":"2021-03-15T22:33:11.459565+0000","last_clean":"2021-03-15T22:33:11.459565+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:33:11.459565+0000","last_undegraded":"2021-03-15T22:33:11.459565+0000","last_fullsized":"2021-03-15T22:33:11.459565+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'2","last_scrub_stamp":"2021-03-15T22:31:58.607619+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:58.607619+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"308","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.471235+0000","last_change":"2021-03-15T22:33:05.208555+0000","last_active":"2021-03-15T22:33:12.471235+0000","last_peered":"2021-03-15T22:33:12.471235+0000","last_clean":"2021-03-15T22:33:12.471235+0000","last_became_active":"2021-03-15T22:33:04.326070+0000","last_became_peered":"2021-03-15T22:33:04.326070+0000","last_unstale":"2021-03-15T22:33:12.471235+0000","last_undegraded":"2021-03-15T22:33:12.471235+0000","last_fullsized":"2021-03-15T22:33:12.471235+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:30:41.227473+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"841'14","reported_seq":"1077","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:12.165658+0000","last_change":"2021-03-15T22:33:08.354412+0000","last_active":"2021-03-15T22:33:12.165658+0000","last_peered":"2021-03-15T22:33:12.165658+0000","last_clean":"2021-03-15T22:33:12.165658+0000","last_became_active":"2021-03-15T22:33:08.354031+0000","last_became_peered":"2021-03-15T22:33:08.354031+0000","last_unstale":"2021-03-15T22:33:12.165658+0000","last_undegraded":"2021-03-15T22:33:12.165658+0000","last_fullsized":"2021-03-15T22:33:12.165658+0000","mapping_epoch":837,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:30:07.252960+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:30:07.252960+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"837'1502","reported_seq":"3934","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.534988+0000","last_change":"2021-03-15T22:33:10.448163+0000","last_active":"2021-03-15T22:33:10.534988+0000","last_peered":"2021-03-15T22:33:10.534988+0000","last_clean":"2021-03-15T22:33:10.534988+0000","last_became_active":"2021-03-15T22:30:34.291079+0000","last_became_peered":"2021-03-15T22:30:34.291079+0000","last_unstale":"2021-03-15T22:33:10.534988+0000","last_undegraded":"2021-03-15T22:33:10.534988+0000","last_fullsized":"2021-03-15T22:33:10.534988+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1312","last_scrub_stamp":"2021-03-15T22:32:05.639057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.639057+0000","log_size":1502,"ondisk_log_size":1502,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65472218,"num_objects":29,"num_object_clones":24,"num_object_copies":87,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":29,"num_whiteouts":0,"num_read":594,"num_read_kb":585534,"num_write":420,"num_write_kb":170295,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":117,"num_bytes_recovered":80476435,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"270","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.446102+0000","last_change":"2021-03-15T22:32:42.621654+0000","last_active":"2021-03-15T22:33:10.446102+0000","last_peered":"2021-03-15T22:33:10.446102+0000","last_clean":"2021-03-15T22:33:10.446102+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:33:10.446102+0000","last_undegraded":"2021-03-15T22:33:10.446102+0000","last_fullsized":"2021-03-15T22:33:10.446102+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:42.621594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:42.621594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"815","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459505+0000","last_change":"2021-03-15T22:31:39.517144+0000","last_active":"2021-03-15T22:33:11.459505+0000","last_peered":"2021-03-15T22:33:11.459505+0000","last_clean":"2021-03-15T22:33:11.459505+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:33:11.459505+0000","last_undegraded":"2021-03-15T22:33:11.459505+0000","last_fullsized":"2021-03-15T22:33:11.459505+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:39.517052+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:39.517052+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"991","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.442104+0000","last_change":"2021-03-15T22:32:03.948964+0000","last_active":"2021-03-15T22:33:10.442104+0000","last_peered":"2021-03-15T22:33:10.442104+0000","last_clean":"2021-03-15T22:33:10.442104+0000","last_became_active":"2021-03-15T22:32:03.258590+0000","last_became_peered":"2021-03-15T22:32:03.258590+0000","last_unstale":"2021-03-15T22:33:10.442104+0000","last_undegraded":"2021-03-15T22:33:10.442104+0000","last_fullsized":"2021-03-15T22:33:10.442104+0000","mapping_epoch":772,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":773,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:31:05.469001+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:05.469001+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":3460,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"673","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.469112+0000","last_change":"2021-03-15T22:31:42.055712+0000","last_active":"2021-03-15T22:33:12.469112+0000","last_peered":"2021-03-15T22:33:12.469112+0000","last_clean":"2021-03-15T22:33:12.469112+0000","last_became_active":"2021-03-15T22:31:18.252524+0000","last_became_peered":"2021-03-15T22:31:18.252524+0000","last_unstale":"2021-03-15T22:33:12.469112+0000","last_undegraded":"2021-03-15T22:33:12.469112+0000","last_fullsized":"2021-03-15T22:33:12.469112+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"834'418","reported_seq":"2702","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.438364+0000","last_change":"2021-03-15T22:33:10.438918+0000","last_active":"2021-03-15T22:33:10.438364+0000","last_peered":"2021-03-15T22:33:10.438364+0000","last_clean":"2021-03-15T22:33:10.438364+0000","last_became_active":"2021-03-15T22:28:16.177364+0000","last_became_peered":"2021-03-15T22:28:16.177364+0000","last_unstale":"2021-03-15T22:33:10.438364+0000","last_undegraded":"2021-03-15T22:33:10.438364+0000","last_fullsized":"2021-03-15T22:33:10.438364+0000","mapping_epoch":556,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":557,"parent":"0.0","parent_split_bits":0,"last_scrub":"677'293","last_scrub_stamp":"2021-03-15T22:30:29.729234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.729234+0000","log_size":418,"ondisk_log_size":418,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":10745993,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":107,"num_read_kb":79087,"num_write":109,"num_write_kb":35163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":11148008,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"861","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.458908+0000","last_change":"2021-03-15T22:32:31.390410+0000","last_active":"2021-03-15T22:33:11.458908+0000","last_peered":"2021-03-15T22:33:11.458908+0000","last_clean":"2021-03-15T22:33:11.458908+0000","last_became_active":"2021-03-15T22:30:32.236752+0000","last_became_peered":"2021-03-15T22:30:32.236752+0000","last_unstale":"2021-03-15T22:33:11.458908+0000","last_undegraded":"2021-03-15T22:33:11.458908+0000","last_fullsized":"2021-03-15T22:33:11.458908+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"680","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.468073+0000","last_change":"2021-03-15T22:31:32.008461+0000","last_active":"2021-03-15T22:33:12.468073+0000","last_peered":"2021-03-15T22:33:12.468073+0000","last_clean":"2021-03-15T22:33:12.468073+0000","last_became_active":"2021-03-15T22:31:18.251403+0000","last_became_peered":"2021-03-15T22:31:18.251403+0000","last_unstale":"2021-03-15T22:33:12.468073+0000","last_undegraded":"2021-03-15T22:33:12.468073+0000","last_fullsized":"2021-03-15T22:33:12.468073+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:32.008403+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:32.008403+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"875","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459723+0000","last_change":"2021-03-15T22:33:04.331301+0000","last_active":"2021-03-15T22:33:11.459723+0000","last_peered":"2021-03-15T22:33:11.459723+0000","last_clean":"2021-03-15T22:33:11.459723+0000","last_became_active":"2021-03-15T22:33:04.330771+0000","last_became_peered":"2021-03-15T22:33:04.330771+0000","last_unstale":"2021-03-15T22:33:11.459723+0000","last_undegraded":"2021-03-15T22:33:11.459723+0000","last_fullsized":"2021-03-15T22:33:11.459723+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:40.430038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:32:40.430038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"44","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.352349+0000","last_change":"2021-03-15T22:33:06.587448+0000","last_active":"2021-03-15T22:33:09.352349+0000","last_peered":"2021-03-15T22:33:09.352349+0000","last_clean":"2021-03-15T22:33:09.352349+0000","last_became_active":"2021-03-15T22:33:04.339508+0000","last_became_peered":"2021-03-15T22:33:04.339508+0000","last_unstale":"2021-03-15T22:33:09.352349+0000","last_undegraded":"2021-03-15T22:33:09.352349+0000","last_fullsized":"2021-03-15T22:33:09.352349+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":836,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:32:39.977326+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.977326+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":2196,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,7],"acting":[6,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"842'885","reported_seq":"2888","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.473992+0000","last_change":"2021-03-15T22:33:12.473992+0000","last_active":"2021-03-15T22:33:12.473992+0000","last_peered":"2021-03-15T22:33:12.473992+0000","last_clean":"2021-03-15T22:33:12.473992+0000","last_became_active":"2021-03-15T22:30:31.242297+0000","last_became_peered":"2021-03-15T22:30:31.242297+0000","last_unstale":"2021-03-15T22:33:12.473992+0000","last_undegraded":"2021-03-15T22:33:12.473992+0000","last_fullsized":"2021-03-15T22:33:12.473992+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'783","last_scrub_stamp":"2021-03-15T22:32:31.034631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.034631+0000","log_size":885,"ondisk_log_size":885,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":45491795,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":375,"num_read_kb":460690,"num_write":231,"num_write_kb":80530,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":24240388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2b8","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"316","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.355466+0000","last_change":"2021-03-15T22:31:08.968657+0000","last_active":"2021-03-15T22:33:09.355466+0000","last_peered":"2021-03-15T22:33:09.355466+0000","last_clean":"2021-03-15T22:33:09.355466+0000","last_became_active":"2021-03-15T22:30:32.237720+0000","last_became_peered":"2021-03-15T22:30:32.237720+0000","last_unstale":"2021-03-15T22:33:09.355466+0000","last_undegraded":"2021-03-15T22:33:09.355466+0000","last_fullsized":"2021-03-15T22:33:09.355466+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"658","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.468392+0000","last_change":"2021-03-15T22:31:33.964163+0000","last_active":"2021-03-15T22:33:12.468392+0000","last_peered":"2021-03-15T22:33:12.468392+0000","last_clean":"2021-03-15T22:33:12.468392+0000","last_became_active":"2021-03-15T22:31:18.261916+0000","last_became_peered":"2021-03-15T22:31:18.261916+0000","last_unstale":"2021-03-15T22:33:12.468392+0000","last_undegraded":"2021-03-15T22:33:12.468392+0000","last_fullsized":"2021-03-15T22:33:12.468392+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:33.964082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:33.964082+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"237","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.471675+0000","last_change":"2021-03-15T22:33:04.333680+0000","last_active":"2021-03-15T22:33:12.471675+0000","last_peered":"2021-03-15T22:33:12.471675+0000","last_clean":"2021-03-15T22:33:12.471675+0000","last_became_active":"2021-03-15T22:33:04.333291+0000","last_became_peered":"2021-03-15T22:33:04.333291+0000","last_unstale":"2021-03-15T22:33:12.471675+0000","last_undegraded":"2021-03-15T22:33:12.471675+0000","last_fullsized":"2021-03-15T22:33:12.471675+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:41.775530+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.775530+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"354","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.356148+0000","last_change":"2021-03-15T22:32:33.701761+0000","last_active":"2021-03-15T22:33:09.356148+0000","last_peered":"2021-03-15T22:33:09.356148+0000","last_clean":"2021-03-15T22:33:09.356148+0000","last_became_active":"2021-03-15T22:30:32.250712+0000","last_became_peered":"2021-03-15T22:30:32.250712+0000","last_unstale":"2021-03-15T22:33:09.356148+0000","last_undegraded":"2021-03-15T22:33:09.356148+0000","last_fullsized":"2021-03-15T22:33:09.356148+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:32:33.701675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.701675+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1788,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,4],"acting":[0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"838'1103","reported_seq":"3919","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.450720+0000","last_change":"2021-03-15T22:33:10.450971+0000","last_active":"2021-03-15T22:33:10.450720+0000","last_peered":"2021-03-15T22:33:10.450720+0000","last_clean":"2021-03-15T22:33:10.450720+0000","last_became_active":"2021-03-15T22:33:08.354856+0000","last_became_peered":"2021-03-15T22:33:08.354856+0000","last_unstale":"2021-03-15T22:33:10.450720+0000","last_undegraded":"2021-03-15T22:33:10.450720+0000","last_fullsized":"2021-03-15T22:33:10.450720+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'1103","last_scrub_stamp":"2021-03-15T22:33:09.245691+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.245691+0000","log_size":1103,"ondisk_log_size":1103,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29400209,"num_objects":11,"num_object_clones":8,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":253,"num_read_kb":196598,"num_write":285,"num_write_kb":97252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":333,"num_bytes_recovered":175537274,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"265","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.446496+0000","last_change":"2021-03-15T22:30:29.533509+0000","last_active":"2021-03-15T22:33:10.446496+0000","last_peered":"2021-03-15T22:33:10.446496+0000","last_clean":"2021-03-15T22:33:10.446496+0000","last_became_active":"2021-03-15T22:28:59.273785+0000","last_became_peered":"2021-03-15T22:28:59.273785+0000","last_unstale":"2021-03-15T22:33:10.446496+0000","last_undegraded":"2021-03-15T22:33:10.446496+0000","last_fullsized":"2021-03-15T22:33:10.446496+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:29.533421+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:29.533421+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"234","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.472265+0000","last_change":"2021-03-15T22:33:04.340785+0000","last_active":"2021-03-15T22:33:12.472265+0000","last_peered":"2021-03-15T22:33:12.472265+0000","last_clean":"2021-03-15T22:33:12.472265+0000","last_became_active":"2021-03-15T22:33:04.339912+0000","last_became_peered":"2021-03-15T22:33:04.339912+0000","last_unstale":"2021-03-15T22:33:12.472265+0000","last_undegraded":"2021-03-15T22:33:12.472265+0000","last_fullsized":"2021-03-15T22:33:12.472265+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:58.785524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:31:58.785524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"38","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441732+0000","last_change":"2021-03-15T22:33:04.336661+0000","last_active":"2021-03-15T22:33:10.441732+0000","last_peered":"2021-03-15T22:33:10.441732+0000","last_clean":"2021-03-15T22:33:10.441732+0000","last_became_active":"2021-03-15T22:33:04.335914+0000","last_became_peered":"2021-03-15T22:33:04.335914+0000","last_unstale":"2021-03-15T22:33:10.441732+0000","last_undegraded":"2021-03-15T22:33:10.441732+0000","last_fullsized":"2021-03-15T22:33:10.441732+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.051962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.051962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1058","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.476485+0000","last_change":"2021-03-15T22:32:06.881916+0000","last_active":"2021-03-15T22:33:11.476485+0000","last_peered":"2021-03-15T22:33:11.476485+0000","last_clean":"2021-03-15T22:33:11.476485+0000","last_became_active":"2021-03-15T22:30:32.244684+0000","last_became_peered":"2021-03-15T22:30:32.244684+0000","last_unstale":"2021-03-15T22:33:11.476485+0000","last_undegraded":"2021-03-15T22:33:11.476485+0000","last_fullsized":"2021-03-15T22:33:11.476485+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:32:06.881829+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:32:06.881829+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":2862,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"300","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.356413+0000","last_change":"2021-03-15T22:30:30.738196+0000","last_active":"2021-03-15T22:33:09.356413+0000","last_peered":"2021-03-15T22:33:09.356413+0000","last_clean":"2021-03-15T22:33:09.356413+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:33:09.356413+0000","last_undegraded":"2021-03-15T22:33:09.356413+0000","last_fullsized":"2021-03-15T22:33:09.356413+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:30.738108+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:30:30.738108+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"840'1066","reported_seq":"3305","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.551423+0000","last_change":"2021-03-15T22:33:10.439942+0000","last_active":"2021-03-15T22:33:10.551423+0000","last_peered":"2021-03-15T22:33:10.551423+0000","last_clean":"2021-03-15T22:33:10.551423+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:33:10.551423+0000","last_undegraded":"2021-03-15T22:33:10.551423+0000","last_fullsized":"2021-03-15T22:33:10.551423+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"805'997","last_scrub_stamp":"2021-03-15T22:32:36.795301+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.795301+0000","log_size":1066,"ondisk_log_size":1066,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32339062,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":305,"num_read_kb":294220,"num_write":241,"num_write_kb":93747,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"862","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459770+0000","last_change":"2021-03-15T22:33:04.334561+0000","last_active":"2021-03-15T22:33:11.459770+0000","last_peered":"2021-03-15T22:33:11.459770+0000","last_clean":"2021-03-15T22:33:11.459770+0000","last_became_active":"2021-03-15T22:33:04.334295+0000","last_became_peered":"2021-03-15T22:33:04.334295+0000","last_unstale":"2021-03-15T22:33:11.459770+0000","last_undegraded":"2021-03-15T22:33:11.459770+0000","last_fullsized":"2021-03-15T22:33:11.459770+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:32.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"852","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.475982+0000","last_change":"2021-03-15T22:33:04.342062+0000","last_active":"2021-03-15T22:33:11.475982+0000","last_peered":"2021-03-15T22:33:11.475982+0000","last_clean":"2021-03-15T22:33:11.475982+0000","last_became_active":"2021-03-15T22:33:04.341661+0000","last_became_peered":"2021-03-15T22:33:04.341661+0000","last_unstale":"2021-03-15T22:33:11.475982+0000","last_undegraded":"2021-03-15T22:33:11.475982+0000","last_fullsized":"2021-03-15T22:33:11.475982+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.793900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.793900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"646","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.467971+0000","last_change":"2021-03-15T22:32:34.095255+0000","last_active":"2021-03-15T22:33:12.467971+0000","last_peered":"2021-03-15T22:33:12.467971+0000","last_clean":"2021-03-15T22:33:12.467971+0000","last_became_active":"2021-03-15T22:30:32.250316+0000","last_became_peered":"2021-03-15T22:30:32.250316+0000","last_unstale":"2021-03-15T22:33:12.467971+0000","last_undegraded":"2021-03-15T22:33:12.467971+0000","last_fullsized":"2021-03-15T22:33:12.467971+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:34.095158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.095158+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"585","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.477166+0000","last_change":"2021-03-15T22:31:59.864528+0000","last_active":"2021-03-15T22:33:11.477166+0000","last_peered":"2021-03-15T22:33:11.477166+0000","last_clean":"2021-03-15T22:33:11.477166+0000","last_became_active":"2021-03-15T22:29:51.222262+0000","last_became_peered":"2021-03-15T22:29:51.222262+0000","last_unstale":"2021-03-15T22:33:11.477166+0000","last_undegraded":"2021-03-15T22:33:11.477166+0000","last_fullsized":"2021-03-15T22:33:11.477166+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:31:59.864460+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:31:59.864460+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"641","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.468800+0000","last_change":"2021-03-15T22:32:04.969553+0000","last_active":"2021-03-15T22:33:12.468800+0000","last_peered":"2021-03-15T22:33:12.468800+0000","last_clean":"2021-03-15T22:33:12.468800+0000","last_became_active":"2021-03-15T22:31:18.249044+0000","last_became_peered":"2021-03-15T22:31:18.249044+0000","last_unstale":"2021-03-15T22:33:12.468800+0000","last_undegraded":"2021-03-15T22:33:12.468800+0000","last_fullsized":"2021-03-15T22:33:12.468800+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:04.969491+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:04.969491+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"840'2526","reported_seq":"5342","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.539144+0000","last_change":"2021-03-15T22:33:10.439066+0000","last_active":"2021-03-15T22:33:10.539144+0000","last_peered":"2021-03-15T22:33:10.539144+0000","last_clean":"2021-03-15T22:33:10.539144+0000","last_became_active":"2021-03-15T22:28:59.276888+0000","last_became_peered":"2021-03-15T22:28:59.276888+0000","last_unstale":"2021-03-15T22:33:10.539144+0000","last_undegraded":"2021-03-15T22:33:10.539144+0000","last_fullsized":"2021-03-15T22:33:10.539144+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'2007","last_scrub_stamp":"2021-03-15T22:31:02.050395+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.050395+0000","log_size":2526,"ondisk_log_size":2526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60885031,"num_objects":25,"num_object_clones":18,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":726,"num_read_kb":728263,"num_write":576,"num_write_kb":238631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":469,"num_bytes_recovered":205133613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"214","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.472088+0000","last_change":"2021-03-15T22:30:28.967439+0000","last_active":"2021-03-15T22:33:12.472088+0000","last_peered":"2021-03-15T22:33:12.472088+0000","last_clean":"2021-03-15T22:33:12.472088+0000","last_became_active":"2021-03-15T22:29:51.227896+0000","last_became_peered":"2021-03-15T22:29:51.227896+0000","last_unstale":"2021-03-15T22:33:12.472088+0000","last_undegraded":"2021-03-15T22:33:12.472088+0000","last_fullsized":"2021-03-15T22:33:12.472088+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:28.967356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:30:28.967356+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"854","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459007+0000","last_change":"2021-03-15T22:33:04.338099+0000","last_active":"2021-03-15T22:33:11.459007+0000","last_peered":"2021-03-15T22:33:11.459007+0000","last_clean":"2021-03-15T22:33:11.459007+0000","last_became_active":"2021-03-15T22:33:04.337709+0000","last_became_peered":"2021-03-15T22:33:04.337709+0000","last_unstale":"2021-03-15T22:33:11.459007+0000","last_undegraded":"2021-03-15T22:33:11.459007+0000","last_fullsized":"2021-03-15T22:33:11.459007+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:43.531349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.531349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"471","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.447511+0000","last_change":"2021-03-15T22:33:04.338229+0000","last_active":"2021-03-15T22:33:10.447511+0000","last_peered":"2021-03-15T22:33:10.447511+0000","last_clean":"2021-03-15T22:33:10.447511+0000","last_became_active":"2021-03-15T22:33:04.337893+0000","last_became_peered":"2021-03-15T22:33:04.337893+0000","last_unstale":"2021-03-15T22:33:10.447511+0000","last_undegraded":"2021-03-15T22:33:10.447511+0000","last_fullsized":"2021-03-15T22:33:10.447511+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:31:39.555487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"36","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.352566+0000","last_change":"2021-03-15T22:33:04.356247+0000","last_active":"2021-03-15T22:33:09.352566+0000","last_peered":"2021-03-15T22:33:09.352566+0000","last_clean":"2021-03-15T22:33:09.352566+0000","last_became_active":"2021-03-15T22:33:04.330531+0000","last_became_peered":"2021-03-15T22:33:04.330531+0000","last_unstale":"2021-03-15T22:33:09.352566+0000","last_undegraded":"2021-03-15T22:33:09.352566+0000","last_fullsized":"2021-03-15T22:33:09.352566+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:32:41.294925+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.294925+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"857","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459590+0000","last_change":"2021-03-15T22:32:39.451103+0000","last_active":"2021-03-15T22:33:11.459590+0000","last_peered":"2021-03-15T22:33:11.459590+0000","last_clean":"2021-03-15T22:33:11.459590+0000","last_became_active":"2021-03-15T22:30:32.243696+0000","last_became_peered":"2021-03-15T22:30:32.243696+0000","last_unstale":"2021-03-15T22:33:11.459590+0000","last_undegraded":"2021-03-15T22:33:11.459590+0000","last_fullsized":"2021-03-15T22:33:11.459590+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.451041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.451041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"834'755","reported_seq":"3166","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.475343+0000","last_change":"2021-03-15T22:33:12.476094+0000","last_active":"2021-03-15T22:33:12.475343+0000","last_peered":"2021-03-15T22:33:12.475343+0000","last_clean":"2021-03-15T22:33:12.475343+0000","last_became_active":"2021-03-15T22:30:35.366323+0000","last_became_peered":"2021-03-15T22:30:35.366323+0000","last_unstale":"2021-03-15T22:33:12.475343+0000","last_undegraded":"2021-03-15T22:33:12.475343+0000","last_fullsized":"2021-03-15T22:33:12.475343+0000","mapping_epoch":685,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":686,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'659","last_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:32:11.467520+0000","log_size":755,"ondisk_log_size":755,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19515383,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":196,"num_read_kb":189381,"num_write":181,"num_write_kb":80772,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":54764539,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2b8","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"885","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.476228+0000","last_change":"2021-03-15T22:32:12.830549+0000","last_active":"2021-03-15T22:33:11.476228+0000","last_peered":"2021-03-15T22:33:11.476228+0000","last_clean":"2021-03-15T22:33:11.476228+0000","last_became_active":"2021-03-15T22:29:51.222189+0000","last_became_peered":"2021-03-15T22:29:51.222189+0000","last_unstale":"2021-03-15T22:33:11.476228+0000","last_undegraded":"2021-03-15T22:33:11.476228+0000","last_fullsized":"2021-03-15T22:33:11.476228+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"321","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.355535+0000","last_change":"2021-03-15T22:31:01.010456+0000","last_active":"2021-03-15T22:33:09.355535+0000","last_peered":"2021-03-15T22:33:09.355535+0000","last_clean":"2021-03-15T22:33:09.355535+0000","last_became_active":"2021-03-15T22:28:59.276599+0000","last_became_peered":"2021-03-15T22:28:59.276599+0000","last_unstale":"2021-03-15T22:33:09.355535+0000","last_undegraded":"2021-03-15T22:33:09.355535+0000","last_fullsized":"2021-03-15T22:33:09.355535+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:01.010393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:01.010393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"1753","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.918337+0000","last_change":"2021-03-15T22:32:30.365491+0000","last_active":"2021-03-15T22:33:12.918337+0000","last_peered":"2021-03-15T22:33:12.918337+0000","last_clean":"2021-03-15T22:33:12.918337+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:33:12.918337+0000","last_undegraded":"2021-03-15T22:33:12.918337+0000","last_fullsized":"2021-03-15T22:33:12.918337+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:32:30.365432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.365432+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":904,"num_read_kb":899,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"368","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.356204+0000","last_change":"2021-03-15T22:31:11.926326+0000","last_active":"2021-03-15T22:33:09.356204+0000","last_peered":"2021-03-15T22:33:09.356204+0000","last_clean":"2021-03-15T22:33:09.356204+0000","last_became_active":"2021-03-15T22:30:32.251686+0000","last_became_peered":"2021-03-15T22:30:32.251686+0000","last_unstale":"2021-03-15T22:33:09.356204+0000","last_undegraded":"2021-03-15T22:33:09.356204+0000","last_fullsized":"2021-03-15T22:33:09.356204+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:31:11.926268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.926268+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":1260,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"228","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.472167+0000","last_change":"2021-03-15T22:32:00.816143+0000","last_active":"2021-03-15T22:33:12.472167+0000","last_peered":"2021-03-15T22:33:12.472167+0000","last_clean":"2021-03-15T22:33:12.472167+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:33:12.472167+0000","last_undegraded":"2021-03-15T22:33:12.472167+0000","last_fullsized":"2021-03-15T22:33:12.472167+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:00.816090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:32:00.816090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"842'2768","reported_seq":"5566","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.486708+0000","last_change":"2021-03-15T22:33:12.486708+0000","last_active":"2021-03-15T22:33:12.486708+0000","last_peered":"2021-03-15T22:33:12.486708+0000","last_clean":"2021-03-15T22:33:12.486708+0000","last_became_active":"2021-03-15T22:29:52.225069+0000","last_became_peered":"2021-03-15T22:29:52.225069+0000","last_unstale":"2021-03-15T22:33:12.486708+0000","last_undegraded":"2021-03-15T22:33:12.486708+0000","last_fullsized":"2021-03-15T22:33:12.486708+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'2597","last_scrub_stamp":"2021-03-15T22:32:30.909661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.909661+0000","log_size":2768,"ondisk_log_size":2768,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":83303044,"num_objects":38,"num_object_clones":31,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":3,"num_read":1028,"num_read_kb":1107063,"num_write":659,"num_write_kb":267316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":298,"num_bytes_recovered":165660176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2b8","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"848","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.475921+0000","last_change":"2021-03-15T22:33:04.335240+0000","last_active":"2021-03-15T22:33:11.475921+0000","last_peered":"2021-03-15T22:33:11.475921+0000","last_clean":"2021-03-15T22:33:11.475921+0000","last_became_active":"2021-03-15T22:33:04.333607+0000","last_became_peered":"2021-03-15T22:33:04.333607+0000","last_unstale":"2021-03-15T22:33:11.475921+0000","last_undegraded":"2021-03-15T22:33:11.475921+0000","last_fullsized":"2021-03-15T22:33:11.475921+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.701955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.701955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"870","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.476394+0000","last_change":"2021-03-15T22:32:44.675954+0000","last_active":"2021-03-15T22:33:11.476394+0000","last_peered":"2021-03-15T22:33:11.476394+0000","last_clean":"2021-03-15T22:33:11.476394+0000","last_became_active":"2021-03-15T22:30:32.243790+0000","last_became_peered":"2021-03-15T22:30:32.243790+0000","last_unstale":"2021-03-15T22:33:11.476394+0000","last_undegraded":"2021-03-15T22:33:11.476394+0000","last_fullsized":"2021-03-15T22:33:11.476394+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_clean_scrub_stamp":"2021-03-15T22:32:44.675891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"380","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.355975+0000","last_change":"2021-03-15T22:31:40.359761+0000","last_active":"2021-03-15T22:33:09.355975+0000","last_peered":"2021-03-15T22:33:09.355975+0000","last_clean":"2021-03-15T22:33:09.355975+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:33:09.355975+0000","last_undegraded":"2021-03-15T22:33:09.355975+0000","last_fullsized":"2021-03-15T22:33:09.355975+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:31:40.359701+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:31:40.359701+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"16","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.352265+0000","last_change":"2021-03-15T22:33:04.338626+0000","last_active":"2021-03-15T22:33:09.352265+0000","last_peered":"2021-03-15T22:33:09.352265+0000","last_clean":"2021-03-15T22:33:09.352265+0000","last_became_active":"2021-03-15T22:33:04.337244+0000","last_became_peered":"2021-03-15T22:33:04.337244+0000","last_unstale":"2021-03-15T22:33:09.352265+0000","last_undegraded":"2021-03-15T22:33:09.352265+0000","last_fullsized":"2021-03-15T22:33:09.352265+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:07.632227+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:32:07.632227+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"839'447","reported_seq":"3022","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.442252+0000","last_change":"2021-03-15T22:33:10.442252+0000","last_active":"2021-03-15T22:33:10.442252+0000","last_peered":"2021-03-15T22:33:10.442252+0000","last_clean":"2021-03-15T22:33:10.442252+0000","last_became_active":"2021-03-15T22:33:09.353726+0000","last_became_peered":"2021-03-15T22:33:09.353726+0000","last_unstale":"2021-03-15T22:33:10.442252+0000","last_undegraded":"2021-03-15T22:33:10.442252+0000","last_fullsized":"2021-03-15T22:33:10.442252+0000","mapping_epoch":838,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":839,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'353","last_scrub_stamp":"2021-03-15T22:31:02.426448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.426448+0000","log_size":447,"ondisk_log_size":447,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":16049279,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":151,"num_read_kb":192431,"num_write":96,"num_write_kb":40684,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":104,"num_bytes_recovered":66852745,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"334","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.355329+0000","last_change":"2021-03-15T22:32:10.674869+0000","last_active":"2021-03-15T22:33:09.355329+0000","last_peered":"2021-03-15T22:33:09.355329+0000","last_clean":"2021-03-15T22:33:09.355329+0000","last_became_active":"2021-03-15T22:30:32.237925+0000","last_became_peered":"2021-03-15T22:30:32.237925+0000","last_unstale":"2021-03-15T22:33:09.355329+0000","last_undegraded":"2021-03-15T22:33:09.355329+0000","last_fullsized":"2021-03-15T22:33:09.355329+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"780","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.472017+0000","last_change":"2021-03-15T22:32:33.884013+0000","last_active":"2021-03-15T22:33:12.472017+0000","last_peered":"2021-03-15T22:33:12.472017+0000","last_clean":"2021-03-15T22:33:12.472017+0000","last_became_active":"2021-03-15T22:30:32.257328+0000","last_became_peered":"2021-03-15T22:30:32.257328+0000","last_unstale":"2021-03-15T22:33:12.472017+0000","last_undegraded":"2021-03-15T22:33:12.472017+0000","last_fullsized":"2021-03-15T22:33:12.472017+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.883949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.883949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"1055","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.476268+0000","last_change":"2021-03-15T22:32:31.747649+0000","last_active":"2021-03-15T22:33:11.476268+0000","last_peered":"2021-03-15T22:33:11.476268+0000","last_clean":"2021-03-15T22:33:11.476268+0000","last_became_active":"2021-03-15T22:29:51.222356+0000","last_became_peered":"2021-03-15T22:29:51.222356+0000","last_unstale":"2021-03-15T22:33:11.476268+0000","last_undegraded":"2021-03-15T22:33:11.476268+0000","last_fullsized":"2021-03-15T22:33:11.476268+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:32:31.747582+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.747582+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.3","version":"836'11","reported_seq":"422","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.355838+0000","last_change":"2021-03-15T22:33:04.427206+0000","last_active":"2021-03-15T22:33:09.355838+0000","last_peered":"2021-03-15T22:33:09.355838+0000","last_clean":"2021-03-15T22:33:09.355838+0000","last_became_active":"2021-03-15T22:33:04.336294+0000","last_became_peered":"2021-03-15T22:33:04.336294+0000","last_unstale":"2021-03-15T22:33:09.355838+0000","last_undegraded":"2021-03-15T22:33:09.355838+0000","last_fullsized":"2021-03-15T22:33:09.355838+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'9","last_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.971303+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"840'746","reported_seq":"3069","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.540226+0000","last_change":"2021-03-15T22:33:10.455212+0000","last_active":"2021-03-15T22:33:10.540226+0000","last_peered":"2021-03-15T22:33:10.540226+0000","last_clean":"2021-03-15T22:33:10.540226+0000","last_became_active":"2021-03-15T22:30:34.297623+0000","last_became_peered":"2021-03-15T22:30:34.297623+0000","last_unstale":"2021-03-15T22:33:10.540226+0000","last_undegraded":"2021-03-15T22:33:10.540226+0000","last_fullsized":"2021-03-15T22:33:10.540226+0000","mapping_epoch":684,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":685,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'591","last_scrub_stamp":"2021-03-15T22:31:35.474935+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:31:35.474935+0000","log_size":746,"ondisk_log_size":746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25864828,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":221,"num_read_kb":187437,"num_write":200,"num_write_kb":76535,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":79,"num_bytes_recovered":47935099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,2],"acting":[3,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"835","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.459393+0000","last_change":"2021-03-15T22:33:04.330560+0000","last_active":"2021-03-15T22:33:11.459393+0000","last_peered":"2021-03-15T22:33:11.459393+0000","last_clean":"2021-03-15T22:33:11.459393+0000","last_became_active":"2021-03-15T22:33:04.330219+0000","last_became_peered":"2021-03-15T22:33:04.330219+0000","last_unstale":"2021-03-15T22:33:11.459393+0000","last_undegraded":"2021-03-15T22:33:11.459393+0000","last_fullsized":"2021-03-15T22:33:11.459393+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:06.530069+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:06.530069+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"300","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.356270+0000","last_change":"2021-03-15T22:30:37.798454+0000","last_active":"2021-03-15T22:33:09.356270+0000","last_peered":"2021-03-15T22:33:09.356270+0000","last_clean":"2021-03-15T22:33:09.356270+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:33:09.356270+0000","last_undegraded":"2021-03-15T22:33:09.356270+0000","last_fullsized":"2021-03-15T22:33:09.356270+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:30:37.798388+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"1028","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441994+0000","last_change":"2021-03-15T22:33:05.544944+0000","last_active":"2021-03-15T22:33:10.441994+0000","last_peered":"2021-03-15T22:33:10.441994+0000","last_clean":"2021-03-15T22:33:10.441994+0000","last_became_active":"2021-03-15T22:33:04.336145+0000","last_became_peered":"2021-03-15T22:33:04.336145+0000","last_unstale":"2021-03-15T22:33:10.441994+0000","last_undegraded":"2021-03-15T22:33:10.441994+0000","last_fullsized":"2021-03-15T22:33:10.441994+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:30:42.383512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:30:42.383512+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"319","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.445806+0000","last_change":"2021-03-15T22:30:27.472423+0000","last_active":"2021-03-15T22:33:10.445806+0000","last_peered":"2021-03-15T22:33:10.445806+0000","last_clean":"2021-03-15T22:33:10.445806+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:33:10.445806+0000","last_undegraded":"2021-03-15T22:33:10.445806+0000","last_fullsized":"2021-03-15T22:33:10.445806+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:30:27.472320+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:30:27.472320+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"838'1298","reported_seq":"3827","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.471282+0000","last_change":"2021-03-15T22:33:12.471515+0000","last_active":"2021-03-15T22:33:12.471282+0000","last_peered":"2021-03-15T22:33:12.471282+0000","last_clean":"2021-03-15T22:33:12.471282+0000","last_became_active":"2021-03-15T22:33:08.354520+0000","last_became_peered":"2021-03-15T22:33:08.354520+0000","last_unstale":"2021-03-15T22:33:12.471282+0000","last_undegraded":"2021-03-15T22:33:12.471282+0000","last_fullsized":"2021-03-15T22:33:12.471282+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'1298","last_scrub_stamp":"2021-03-15T22:33:09.468592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.468592+0000","log_size":1298,"ondisk_log_size":1298,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46342072,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":444,"num_read_kb":453228,"num_write":301,"num_write_kb":104664,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":275,"num_bytes_recovered":139702636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2b8","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"873","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441734+0000","last_change":"2021-03-15T22:32:39.271324+0000","last_active":"2021-03-15T22:33:10.441734+0000","last_peered":"2021-03-15T22:33:10.441734+0000","last_clean":"2021-03-15T22:33:10.441734+0000","last_became_active":"2021-03-15T22:30:32.261995+0000","last_became_peered":"2021-03-15T22:30:32.261995+0000","last_unstale":"2021-03-15T22:33:10.441734+0000","last_undegraded":"2021-03-15T22:33:10.441734+0000","last_fullsized":"2021-03-15T22:33:10.441734+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":683,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"620","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.447679+0000","last_change":"2021-03-15T22:31:36.472708+0000","last_active":"2021-03-15T22:33:10.447679+0000","last_peered":"2021-03-15T22:33:10.447679+0000","last_clean":"2021-03-15T22:33:10.447679+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:33:10.447679+0000","last_undegraded":"2021-03-15T22:33:10.447679+0000","last_fullsized":"2021-03-15T22:33:10.447679+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:36.472646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:31:36.472646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"296","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.447038+0000","last_change":"2021-03-15T22:32:12.653877+0000","last_active":"2021-03-15T22:33:10.447038+0000","last_peered":"2021-03-15T22:33:10.447038+0000","last_clean":"2021-03-15T22:33:10.447038+0000","last_became_active":"2021-03-15T22:28:59.276755+0000","last_became_peered":"2021-03-15T22:28:59.276755+0000","last_unstale":"2021-03-15T22:33:10.447038+0000","last_undegraded":"2021-03-15T22:33:10.447038+0000","last_fullsized":"2021-03-15T22:33:10.447038+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:32:12.653809+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.653809+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2538,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"863","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441671+0000","last_change":"2021-03-15T22:31:04.495639+0000","last_active":"2021-03-15T22:33:10.441671+0000","last_peered":"2021-03-15T22:33:10.441671+0000","last_clean":"2021-03-15T22:33:10.441671+0000","last_became_active":"2021-03-15T22:28:59.266487+0000","last_became_peered":"2021-03-15T22:28:59.266487+0000","last_unstale":"2021-03-15T22:33:10.441671+0000","last_undegraded":"2021-03-15T22:33:10.441671+0000","last_fullsized":"2021-03-15T22:33:10.441671+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.495574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.495574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"837'1147","reported_seq":"3589","reported_epoch":"840","state":"active+clean+wait","last_fresh":"2021-03-15T22:33:10.450471+0000","last_change":"2021-03-15T22:33:10.450688+0000","last_active":"2021-03-15T22:33:10.450471+0000","last_peered":"2021-03-15T22:33:10.450471+0000","last_clean":"2021-03-15T22:33:10.450471+0000","last_became_active":"2021-03-15T22:33:07.345178+0000","last_became_peered":"2021-03-15T22:33:07.345178+0000","last_unstale":"2021-03-15T22:33:10.450471+0000","last_undegraded":"2021-03-15T22:33:10.450471+0000","last_fullsized":"2021-03-15T22:33:10.450471+0000","mapping_epoch":836,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":837,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1034","last_scrub_stamp":"2021-03-15T22:32:09.615397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:09.615397+0000","log_size":1147,"ondisk_log_size":1147,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":29844958,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":2,"num_read":316,"num_read_kb":284494,"num_write":255,"num_write_kb":90212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":180,"num_bytes_recovered":70828071,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"765","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.471931+0000","last_change":"2021-03-15T22:31:04.380476+0000","last_active":"2021-03-15T22:33:12.471931+0000","last_peered":"2021-03-15T22:33:12.471931+0000","last_clean":"2021-03-15T22:33:12.471931+0000","last_became_active":"2021-03-15T22:29:52.223342+0000","last_became_peered":"2021-03-15T22:29:52.223342+0000","last_unstale":"2021-03-15T22:33:12.471931+0000","last_undegraded":"2021-03-15T22:33:12.471931+0000","last_fullsized":"2021-03-15T22:33:12.471931+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.380417+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.380417+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1103","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.442290+0000","last_change":"2021-03-15T22:32:08.071278+0000","last_active":"2021-03-15T22:33:10.442290+0000","last_peered":"2021-03-15T22:33:10.442290+0000","last_clean":"2021-03-15T22:33:10.442290+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:33:10.442290+0000","last_undegraded":"2021-03-15T22:33:10.442290+0000","last_fullsized":"2021-03-15T22:33:10.442290+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:32:08.071210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:08.071210+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"265","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.445703+0000","last_change":"2021-03-15T22:30:09.506976+0000","last_active":"2021-03-15T22:33:10.445703+0000","last_peered":"2021-03-15T22:33:10.445703+0000","last_clean":"2021-03-15T22:33:10.445703+0000","last_became_active":"2021-03-15T22:28:59.274186+0000","last_became_peered":"2021-03-15T22:28:59.274186+0000","last_unstale":"2021-03-15T22:33:10.445703+0000","last_undegraded":"2021-03-15T22:33:10.445703+0000","last_fullsized":"2021-03-15T22:33:10.445703+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:09.506914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:30:09.506914+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"699","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.468797+0000","last_change":"2021-03-15T22:32:35.067624+0000","last_active":"2021-03-15T22:33:12.468797+0000","last_peered":"2021-03-15T22:33:12.468797+0000","last_clean":"2021-03-15T22:33:12.468797+0000","last_became_active":"2021-03-15T22:30:32.262267+0000","last_became_peered":"2021-03-15T22:30:32.262267+0000","last_unstale":"2021-03-15T22:33:12.468797+0000","last_undegraded":"2021-03-15T22:33:12.468797+0000","last_fullsized":"2021-03-15T22:33:12.468797+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:32:35.067557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:35.067557+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.d","version":"840'351","reported_seq":"2699","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.538039+0000","last_change":"2021-03-15T22:33:10.450669+0000","last_active":"2021-03-15T22:33:10.538039+0000","last_peered":"2021-03-15T22:33:10.538039+0000","last_clean":"2021-03-15T22:33:10.538039+0000","last_became_active":"2021-03-15T22:33:08.354872+0000","last_became_peered":"2021-03-15T22:33:08.354872+0000","last_unstale":"2021-03-15T22:33:10.538039+0000","last_undegraded":"2021-03-15T22:33:10.538039+0000","last_fullsized":"2021-03-15T22:33:10.538039+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'322","last_scrub_stamp":"2021-03-15T22:32:11.867811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:11.867811+0000","log_size":351,"ondisk_log_size":351,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11572741,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":137,"num_read_kb":148656,"num_write":71,"num_write_kb":29817,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":31290769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"577","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.446731+0000","last_change":"2021-03-15T22:33:09.612213+0000","last_active":"2021-03-15T22:33:10.446731+0000","last_peered":"2021-03-15T22:33:10.446731+0000","last_clean":"2021-03-15T22:33:10.446731+0000","last_became_active":"2021-03-15T22:33:04.331948+0000","last_became_peered":"2021-03-15T22:33:04.331948+0000","last_unstale":"2021-03-15T22:33:10.446731+0000","last_undegraded":"2021-03-15T22:33:10.446731+0000","last_fullsized":"2021-03-15T22:33:10.446731+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:09.612126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.612126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"879","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.472372+0000","last_change":"2021-03-15T22:33:08.352674+0000","last_active":"2021-03-15T22:33:12.472372+0000","last_peered":"2021-03-15T22:33:12.472372+0000","last_clean":"2021-03-15T22:33:12.472372+0000","last_became_active":"2021-03-15T22:33:08.352367+0000","last_became_peered":"2021-03-15T22:33:08.352367+0000","last_unstale":"2021-03-15T22:33:12.472372+0000","last_undegraded":"2021-03-15T22:33:12.472372+0000","last_fullsized":"2021-03-15T22:33:12.472372+0000","mapping_epoch":837,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'9","last_scrub_stamp":"2021-03-15T22:32:36.868932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.868932+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"827","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.441782+0000","last_change":"2021-03-15T22:30:34.363233+0000","last_active":"2021-03-15T22:33:10.441782+0000","last_peered":"2021-03-15T22:33:10.441782+0000","last_clean":"2021-03-15T22:33:10.441782+0000","last_became_active":"2021-03-15T22:30:31.239376+0000","last_became_peered":"2021-03-15T22:30:31.239376+0000","last_unstale":"2021-03-15T22:33:10.441782+0000","last_undegraded":"2021-03-15T22:33:10.441782+0000","last_fullsized":"2021-03-15T22:33:10.441782+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:34.363177+0000","last_clean_scrub_stamp":"2021-03-15T22:30:34.363177+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"978","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.442354+0000","last_change":"2021-03-15T22:33:05.231881+0000","last_active":"2021-03-15T22:33:10.442354+0000","last_peered":"2021-03-15T22:33:10.442354+0000","last_clean":"2021-03-15T22:33:10.442354+0000","last_became_active":"2021-03-15T22:33:04.340713+0000","last_became_peered":"2021-03-15T22:33:04.340713+0000","last_unstale":"2021-03-15T22:33:10.442354+0000","last_undegraded":"2021-03-15T22:33:10.442354+0000","last_fullsized":"2021-03-15T22:33:10.442354+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:32:38.318687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.318687+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2952,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"833'385","reported_seq":"2210","reported_epoch":"843","state":"active+clean","last_fresh":"2021-03-15T22:33:13.869212+0000","last_change":"2021-03-15T22:33:13.869212+0000","last_active":"2021-03-15T22:33:13.869212+0000","last_peered":"2021-03-15T22:33:13.869212+0000","last_clean":"2021-03-15T22:33:13.869212+0000","last_became_active":"2021-03-15T22:30:32.257058+0000","last_became_peered":"2021-03-15T22:30:32.257058+0000","last_unstale":"2021-03-15T22:33:13.869212+0000","last_undegraded":"2021-03-15T22:33:13.869212+0000","last_fullsized":"2021-03-15T22:33:13.869212+0000","mapping_epoch":682,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":684,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'385","last_scrub_stamp":"2021-03-15T22:33:13.869146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:13.869146+0000","log_size":385,"ondisk_log_size":385,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":12874457,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":153,"num_read_kb":164361,"num_write":104,"num_write_kb":32631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":47,"num_bytes_recovered":32799803,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2b8","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"120","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.468764+0000","last_change":"2021-03-15T22:32:38.958119+0000","last_active":"2021-03-15T22:33:12.468764+0000","last_peered":"2021-03-15T22:33:12.468764+0000","last_clean":"2021-03-15T22:33:12.468764+0000","last_became_active":"2021-03-15T22:31:28.287218+0000","last_became_peered":"2021-03-15T22:31:28.287218+0000","last_unstale":"2021-03-15T22:33:12.468764+0000","last_undegraded":"2021-03-15T22:33:12.468764+0000","last_fullsized":"2021-03-15T22:33:12.468764+0000","mapping_epoch":737,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":738,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.958050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.958050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"870","reported_epoch":"841","state":"active+clean","last_fresh":"2021-03-15T22:33:11.476632+0000","last_change":"2021-03-15T22:32:05.880915+0000","last_active":"2021-03-15T22:33:11.476632+0000","last_peered":"2021-03-15T22:33:11.476632+0000","last_clean":"2021-03-15T22:33:11.476632+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:33:11.476632+0000","last_undegraded":"2021-03-15T22:33:11.476632+0000","last_fullsized":"2021-03-15T22:33:11.476632+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:05.880846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.880846+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"21","reported_epoch":"839","state":"active+clean","last_fresh":"2021-03-15T22:33:09.352431+0000","last_change":"2021-03-15T22:33:08.273442+0000","last_active":"2021-03-15T22:33:09.352431+0000","last_peered":"2021-03-15T22:33:09.352431+0000","last_clean":"2021-03-15T22:33:09.352431+0000","last_became_active":"2021-03-15T22:33:04.327825+0000","last_became_peered":"2021-03-15T22:33:04.327825+0000","last_unstale":"2021-03-15T22:33:09.352431+0000","last_undegraded":"2021-03-15T22:33:09.352431+0000","last_fullsized":"2021-03-15T22:33:09.352431+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:08.273379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.273379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"916","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.442314+0000","last_change":"2021-03-15T22:32:43.307723+0000","last_active":"2021-03-15T22:33:10.442314+0000","last_peered":"2021-03-15T22:33:10.442314+0000","last_clean":"2021-03-15T22:33:10.442314+0000","last_became_active":"2021-03-15T22:30:31.239360+0000","last_became_peered":"2021-03-15T22:30:31.239360+0000","last_unstale":"2021-03-15T22:33:10.442314+0000","last_undegraded":"2021-03-15T22:33:10.442314+0000","last_fullsized":"2021-03-15T22:33:10.442314+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:32:43.307660+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.307660+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"840'1317","reported_seq":"3976","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.448356+0000","last_change":"2021-03-15T22:33:10.439992+0000","last_active":"2021-03-15T22:33:10.448356+0000","last_peered":"2021-03-15T22:33:10.448356+0000","last_clean":"2021-03-15T22:33:10.448356+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:33:10.448356+0000","last_undegraded":"2021-03-15T22:33:10.448356+0000","last_fullsized":"2021-03-15T22:33:10.448356+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"837'1315","last_scrub_stamp":"2021-03-15T22:33:08.608669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.608669+0000","log_size":1317,"ondisk_log_size":1317,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51267689,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":461,"num_read_kb":471402,"num_write":327,"num_write_kb":119548,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"293","length":"1"},{"start":"2b6","length":"1"},{"start":"2bd","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"819","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.442340+0000","last_change":"2021-03-15T22:30:36.417325+0000","last_active":"2021-03-15T22:33:10.442340+0000","last_peered":"2021-03-15T22:33:10.442340+0000","last_clean":"2021-03-15T22:33:10.442340+0000","last_became_active":"2021-03-15T22:30:31.243269+0000","last_became_peered":"2021-03-15T22:30:31.243269+0000","last_unstale":"2021-03-15T22:33:10.442340+0000","last_undegraded":"2021-03-15T22:33:10.442340+0000","last_fullsized":"2021-03-15T22:33:10.442340+0000","mapping_epoch":681,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":682,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:30:36.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"732'13","reported_seq":"795","reported_epoch":"842","state":"active+clean","last_fresh":"2021-03-15T22:33:12.468772+0000","last_change":"2021-03-15T22:32:00.927902+0000","last_active":"2021-03-15T22:33:12.468772+0000","last_peered":"2021-03-15T22:33:12.468772+0000","last_clean":"2021-03-15T22:33:12.468772+0000","last_became_active":"2021-03-15T22:31:18.249734+0000","last_became_peered":"2021-03-15T22:31:18.249734+0000","last_unstale":"2021-03-15T22:33:12.468772+0000","last_undegraded":"2021-03-15T22:33:12.468772+0000","last_fullsized":"2021-03-15T22:33:12.468772+0000","mapping_epoch":682,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'13","last_scrub_stamp":"2021-03-15T22:32:00.927837+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:00.927837+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"265","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.445928+0000","last_change":"2021-03-15T22:30:03.567594+0000","last_active":"2021-03-15T22:33:10.445928+0000","last_peered":"2021-03-15T22:33:10.445928+0000","last_clean":"2021-03-15T22:33:10.445928+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:33:10.445928+0000","last_undegraded":"2021-03-15T22:33:10.445928+0000","last_fullsized":"2021-03-15T22:33:10.445928+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.567529+0000","last_clean_scrub_stamp":"2021-03-15T22:30:03.567529+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"1023","reported_epoch":"840","state":"active+clean","last_fresh":"2021-03-15T22:33:10.442144+0000","last_change":"2021-03-15T22:33:05.216013+0000","last_active":"2021-03-15T22:33:10.442144+0000","last_peered":"2021-03-15T22:33:10.442144+0000","last_clean":"2021-03-15T22:33:10.442144+0000","last_became_active":"2021-03-15T22:33:04.337113+0000","last_became_peered":"2021-03-15T22:33:04.337113+0000","last_unstale":"2021-03-15T22:33:10.442144+0000","last_undegraded":"2021-03-15T22:33:10.442144+0000","last_fullsized":"2021-03-15T22:33:10.442144+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:32:32.328526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.328526+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":576081048,"num_objects":252,"num_object_clones":202,"num_object_copies":756,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":252,"num_whiteouts":11,"num_read":5884,"num_read_kb":6011455,"num_write":4337,"num_write_kb":1651726,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2605,"num_bytes_recovered":1316728475,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1290121216,"data_stored":1746989365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":17806,"ondisk_log_size":17806,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":904,"num_read_kb":899,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1773,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1483,"num_read_kb":988,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":982,"num_bytes_recovered":76854,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":938,"ondisk_log_size":938,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":122,"ondisk_log_size":122,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299580,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":194944,"kb_used_data":149888,"kb_used_omap":0,"kb_used_meta":43968,"kb_avail":93550208,"statfs":{"total":95995035648,"available":95795412992,"internally_reserved":0,"allocated":153485312,"data_stored":203862404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45023232},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.503,"5min":0.44500000000000001,"15min":0.47599999999999998},"min":{"1min":0.371,"5min":0.216,"15min":0.215},"max":{"1min":0.70099999999999996,"5min":0.93999999999999995,"15min":2.73},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.47999999999999998,"15min":0.496},"min":{"1min":0.33300000000000002,"5min":0.251,"15min":0.222},"max":{"1min":0.71599999999999997,"5min":0.75800000000000001,"15min":3.8799999999999999},"last":0.36299999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.44800000000000001,"15min":0.47999999999999998},"min":{"1min":0.30199999999999999,"5min":0.27300000000000002,"15min":0.22600000000000001},"max":{"1min":0.73899999999999999,"5min":0.91300000000000003,"15min":2.6859999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.47399999999999998,"15min":0.49299999999999999},"min":{"1min":0.35199999999999998,"5min":0.311,"15min":0.23400000000000001},"max":{"1min":0.81899999999999995,"5min":0.81899999999999995,"15min":2.0779999999999998},"last":0.47099999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:32:34 2021","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.49299999999999999,"15min":0.47999999999999998},"min":{"1min":0.39200000000000002,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.77400000000000002,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.505,"15min":0.503},"min":{"1min":0.32100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.753,"5min":1.119,"15min":1.119},"last":0.48499999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.501,"15min":0.50800000000000001},"min":{"1min":0.379,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.79200000000000004,"5min":1.4299999999999999,"15min":2.0339999999999998},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.54400000000000004,"5min":0.52900000000000003,"15min":0.53700000000000003},"min":{"1min":0.35499999999999998,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.751,"5min":5.415,"15min":5.415},"last":0.46000000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.41199999999999998,"15min":0.44500000000000001},"min":{"1min":0.32600000000000001,"5min":0.24199999999999999,"15min":0.23799999999999999},"max":{"1min":0.753,"5min":0.753,"15min":1.927},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.43099999999999999,"15min":0.46300000000000002},"min":{"1min":0.28499999999999998,"5min":0.253,"15min":0.253},"max":{"1min":0.68200000000000005,"5min":0.72399999999999998,"15min":2.5339999999999998},"last":0.44400000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:32:34 2021","interfaces":[{"interface":"back","average":{"1min":0.52400000000000002,"5min":0.46300000000000002,"15min":0.44900000000000001},"min":{"1min":0.40200000000000002,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.41899999999999998,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.76900000000000002,"5min":1.071,"15min":1.071},"last":0.53800000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.46899999999999997,"15min":0.51700000000000002},"min":{"1min":0.29999999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.78400000000000003,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.48499999999999999,"15min":0.51800000000000002},"min":{"1min":0.36499999999999999,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.77500000000000002,"5min":1.097,"15min":1.8560000000000001},"last":0.55800000000000005}]}]},{"osd":6,"up_from":45,"seq":193273528511,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172684,"kb_used_data":140356,"kb_used_omap":0,"kb_used_meta":32320,"kb_avail":93572468,"statfs":{"total":95995035648,"available":95818207232,"internally_reserved":0,"allocated":143724544,"data_stored":167309545,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33095680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,0,0,0,0,0,0,0,0,4],"upper_bound":8192},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:33:04 2021","interfaces":[{"interface":"back","average":{"1min":0.60999999999999999,"5min":0.48899999999999999,"15min":0.48399999999999999},"min":{"1min":0.29699999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.5470000000000002,"5min":3.085,"15min":3.085},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.27800000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":1.137,"5min":1.323,"15min":1.323},"last":0.51500000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:33:04 2021","interfaces":[{"interface":"back","average":{"1min":0.747,"5min":0.66600000000000004,"15min":0.55500000000000005},"min":{"1min":0.35899999999999999,"5min":0.27900000000000003,"15min":0.254},"max":{"1min":2.7839999999999998,"5min":14.637,"15min":14.637},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.48099999999999998,"15min":0.48099999999999998},"min":{"1min":0.35499999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.57799999999999996,"5min":5.2450000000000001,"15min":5.2450000000000001},"last":0.501}]},{"osd":2,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.46600000000000003,"15min":0.46200000000000002},"min":{"1min":0.312,"5min":0.312,"15min":0.312},"max":{"1min":0.76700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.51400000000000001,"15min":0.505},"min":{"1min":0.34300000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.94199999999999995,"5min":0.94199999999999995,"15min":0.94199999999999995},"last":0.61899999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:33:04 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.47799999999999998,"15min":0.51100000000000001},"min":{"1min":0.40799999999999997,"5min":0.318,"15min":0.29899999999999999},"max":{"1min":1.0609999999999999,"5min":1.0609999999999999,"15min":2.6150000000000002},"last":0.54900000000000004},{"interface":"front","average":{"1min":0.59699999999999998,"5min":0.47899999999999998,"15min":0.51100000000000001},"min":{"1min":0.36099999999999999,"5min":0.316,"15min":0.28699999999999998},"max":{"1min":2.4670000000000001,"5min":2.4670000000000001,"15min":2.7890000000000001},"last":0.628}]},{"osd":4,"last update":"Mon Mar 15 22:33:04 2021","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.45700000000000002,"15min":0.439},"min":{"1min":0.31,"5min":0.224,"15min":0.224},"max":{"1min":0.94099999999999995,"5min":4.306,"15min":4.306},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.47999999999999998,"15min":0.48999999999999999},"min":{"1min":0.34899999999999998,"5min":0.23499999999999999,"15min":0.23499999999999999},"max":{"1min":1.091,"5min":4.181,"15min":4.181},"last":0.29899999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.438,"15min":0.41399999999999998},"min":{"1min":0.35699999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.873,"5min":0.873,"15min":0.873},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.61899999999999999,"5min":0.48999999999999999,"15min":0.45700000000000002},"min":{"1min":0.45100000000000001,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":1.022,"5min":1.022,"15min":1.022},"last":0.317}]},{"osd":7,"last update":"Mon Mar 15 22:32:17 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.58599999999999997,"15min":0.54100000000000004},"min":{"1min":0.32500000000000001,"5min":0.249,"15min":0.249},"max":{"1min":0.76300000000000001,"5min":15.340999999999999,"15min":15.340999999999999},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.65200000000000002,"5min":0.64900000000000002,"15min":0.58599999999999997},"min":{"1min":0.42899999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.98799999999999999,"5min":15.356,"15min":15.356},"last":0.35799999999999998}]}]},{"osd":1,"up_from":13,"seq":55834575057,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":252136,"kb_used_data":198328,"kb_used_omap":0,"kb_used_meta":53440,"kb_avail":93493016,"statfs":{"total":95995035648,"available":95736848384,"internally_reserved":0,"allocated":203087872,"data_stored":256855493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54722560},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:46 2021","interfaces":[{"interface":"back","average":{"1min":0.32700000000000001,"5min":0.309,"15min":0.51400000000000001},"min":{"1min":0.23499999999999999,"5min":0.17599999999999999,"15min":0.17599999999999999},"max":{"1min":0.51400000000000001,"5min":1.325,"15min":54.427999999999997},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.33700000000000002,"5min":0.35699999999999998,"15min":0.53800000000000003},"min":{"1min":0.19500000000000001,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.41099999999999998,"5min":3.3500000000000001,"15min":54.356999999999999},"last":0.32200000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:32:36 2021","interfaces":[{"interface":"back","average":{"1min":0.55900000000000005,"5min":0.379,"15min":0.34899999999999998},"min":{"1min":0.254,"5min":0.248,"15min":0.248},"max":{"1min":4.1509999999999998,"5min":4.1509999999999998,"15min":4.1509999999999998},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.45600000000000002,"5min":0.39000000000000001,"15min":0.379},"min":{"1min":0.27200000000000002,"5min":0.254,"15min":0.254},"max":{"1min":1.891,"5min":1.891,"15min":1.891},"last":0.37}]},{"osd":3,"last update":"Mon Mar 15 22:32:13 2021","interfaces":[{"interface":"back","average":{"1min":0.442,"5min":0.35699999999999998,"15min":0.38100000000000001},"min":{"1min":0.27200000000000002,"5min":0.215,"15min":0.215},"max":{"1min":1.647,"5min":1.647,"15min":1.796},"last":0.27600000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.40300000000000002,"15min":0.41799999999999998},"min":{"1min":0.307,"5min":0.21199999999999999,"15min":0.20999999999999999},"max":{"1min":3.2919999999999998,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.35599999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:32:18 2021","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.53600000000000003,"15min":0.54800000000000004},"min":{"1min":0.38700000000000001,"5min":0.22800000000000001,"15min":0.22800000000000001},"max":{"1min":4.0330000000000004,"5min":4.0330000000000004,"15min":4.0330000000000004},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.54400000000000004,"15min":0.58299999999999996},"min":{"1min":0.35699999999999998,"5min":0.27200000000000002,"15min":0.253},"max":{"1min":0.78600000000000003,"5min":4.1790000000000003,"15min":4.6139999999999999},"last":0.57499999999999996}]},{"osd":5,"last update":"Mon Mar 15 22:32:28 2021","interfaces":[{"interface":"back","average":{"1min":0.57099999999999995,"5min":0.54400000000000004,"15min":0.53000000000000003},"min":{"1min":0.40999999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.85599999999999998,"5min":1.226,"15min":1.226},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.76700000000000002,"5min":0.55900000000000005,"15min":0.51200000000000001},"min":{"1min":0.45700000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":3.472,"5min":3.472,"15min":3.472},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:32:55 2021","interfaces":[{"interface":"back","average":{"1min":0.52900000000000003,"5min":0.56299999999999994,"15min":0.58799999999999997},"min":{"1min":0.39800000000000002,"5min":0.314,"15min":0.314},"max":{"1min":0.76400000000000001,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.56299999999999994,"15min":0.57599999999999996},"min":{"1min":0.44,"5min":0.34999999999999998,"15min":0.33700000000000002},"max":{"1min":0.80400000000000005,"5min":4.1210000000000004,"15min":4.1210000000000004},"last":0.58399999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:32:13 2021","interfaces":[{"interface":"back","average":{"1min":0.77600000000000002,"5min":0.56599999999999995,"15min":0.65700000000000003},"min":{"1min":0.46800000000000003,"5min":0.32100000000000001,"15min":0.316},"max":{"1min":3.5070000000000001,"5min":3.5070000000000001,"15min":8.8100000000000005},"last":0.68700000000000006},{"interface":"front","average":{"1min":0.78900000000000003,"5min":0.57699999999999996,"15min":0.58699999999999997},"min":{"1min":0.41999999999999998,"5min":0.37,"15min":0.35799999999999998},"max":{"1min":4.0860000000000003,"5min":4.0860000000000003,"15min":4.0860000000000003},"last":0.61399999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738579,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":223004,"kb_used_data":182444,"kb_used_omap":0,"kb_used_meta":40192,"kb_avail":93522148,"statfs":{"total":95995035648,"available":95766679552,"internally_reserved":0,"allocated":186822656,"data_stored":258217548,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41156608},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:32:40 2021","interfaces":[{"interface":"back","average":{"1min":0.40200000000000002,"5min":0.32900000000000001,"15min":0.34200000000000003},"min":{"1min":0.192,"5min":0.17999999999999999,"15min":0.17999999999999999},"max":{"1min":1.46,"5min":1.46,"15min":1.7070000000000001},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.48299999999999998,"5min":0.371,"15min":0.37},"min":{"1min":0.254,"5min":0.20699999999999999,"15min":0.19900000000000001},"max":{"1min":2.4830000000000001,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.32900000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:32:34 2021","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.375,"15min":0.36499999999999999},"min":{"1min":0.24099999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":1.4990000000000001,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.313},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.40200000000000002,"15min":0.38400000000000001},"min":{"1min":0.20999999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":2.5390000000000001,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.36499999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:32:17 2021","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.39700000000000002,"15min":0.38700000000000001},"min":{"1min":0.27800000000000002,"5min":0.22700000000000001,"15min":0.22700000000000001},"max":{"1min":1.544,"5min":1.544,"15min":1.544},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.52100000000000002,"5min":0.41399999999999998,"15min":0.40699999999999997},"min":{"1min":0.254,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":2.589,"5min":2.589,"15min":2.589},"last":0.26800000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:32:17 2021","interfaces":[{"interface":"back","average":{"1min":0.66600000000000004,"5min":0.52500000000000002,"15min":0.54300000000000004},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":1.861,"5min":1.861,"15min":1.861},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.70999999999999996,"5min":0.54900000000000004,"15min":0.57299999999999995},"min":{"1min":0.33000000000000002,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":1.5700000000000001,"5min":1.5700000000000001,"15min":1.645},"last":0.47099999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.69599999999999995,"5min":0.54800000000000004,"15min":0.51500000000000001},"min":{"1min":0.33900000000000002,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":2.6440000000000001,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.52100000000000002,"15min":0.48899999999999999},"min":{"1min":0.38900000000000001,"5min":0.318,"15min":0.318},"max":{"1min":1.909,"5min":1.909,"15min":1.909},"last":0.46200000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:32:52 2021","interfaces":[{"interface":"back","average":{"1min":0.63300000000000001,"5min":0.52600000000000002,"15min":0.55700000000000005},"min":{"1min":0.38700000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.823,"5min":1.823,"15min":1.823},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.66800000000000004,"5min":0.54800000000000004,"15min":0.57999999999999996},"min":{"1min":0.442,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":1.5880000000000001,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.498}]},{"osd":7,"last update":"Mon Mar 15 22:32:17 2021","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.55800000000000005,"15min":0.57699999999999996},"min":{"1min":0.443,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":2.6190000000000002,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.67300000000000004,"5min":0.54200000000000004,"15min":0.57599999999999996},"min":{"1min":0.45200000000000001,"5min":0.317,"15min":0.317},"max":{"1min":1.8859999999999999,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.42899999999999999}]}]},{"osd":2,"up_from":681,"seq":2924872728610,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":181540,"kb_used_data":158516,"kb_used_omap":119,"kb_used_meta":22536,"kb_avail":93563612,"statfs":{"total":95995035648,"available":95809138688,"internally_reserved":0,"allocated":162320384,"data_stored":229424274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":23077225},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.32800000000000001,"5min":0.311,"15min":0.308},"min":{"1min":0.19400000000000001,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"last":0.32000000000000001},{"interface":"front","average":{"1min":0.34599999999999997,"5min":0.32500000000000001,"15min":0.32100000000000001},"min":{"1min":0.221,"5min":0.20100000000000001,"15min":0.20100000000000001},"max":{"1min":0.49399999999999999,"5min":0.49399999999999999,"15min":0.49399999999999999},"last":0.255}]},{"osd":1,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.34599999999999997,"5min":0.32700000000000001,"15min":0.32400000000000001},"min":{"1min":0.23699999999999999,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"last":0.29099999999999998},{"interface":"front","average":{"1min":0.40100000000000002,"5min":0.36399999999999999,"15min":0.35799999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.221},"max":{"1min":1.3069999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.35899999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.58899999999999997,"5min":0.38700000000000001,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.216,"15min":0.216},"max":{"1min":3.496,"5min":3.496,"15min":3.496},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.39000000000000001,"5min":0.35899999999999999,"15min":0.35399999999999998},"min":{"1min":0.248,"5min":0.248,"15min":0.248},"max":{"1min":0.74299999999999999,"5min":0.74299999999999999,"15min":0.74299999999999999},"last":0.34899999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.66500000000000004,"5min":0.50700000000000001,"15min":0.48099999999999998},"min":{"1min":0.36699999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.2410000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.50800000000000001,"15min":0.48899999999999999},"min":{"1min":0.39100000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":1.4239999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.60299999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.51100000000000001,"15min":0.501},"min":{"1min":0.38300000000000001,"5min":0.36599999999999999,"15min":0.36599999999999999},"max":{"1min":1.0009999999999999,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.56899999999999995},{"interface":"front","average":{"1min":0.63200000000000001,"5min":0.52400000000000002,"15min":0.50600000000000001},"min":{"1min":0.40100000000000002,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":1.2849999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.623}]},{"osd":6,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.67900000000000005,"5min":0.52800000000000002,"15min":0.503},"min":{"1min":0.48599999999999999,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.3320000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.67800000000000005,"5min":0.53400000000000003,"15min":0.51000000000000001},"min":{"1min":0.42399999999999999,"5min":0.373,"15min":0.373},"max":{"1min":1.387,"5min":1.387,"15min":1.387},"last":0.66400000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:32:37 2021","interfaces":[{"interface":"back","average":{"1min":0.74099999999999999,"5min":0.56200000000000006,"15min":0.53200000000000003},"min":{"1min":0.44,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.9239999999999999,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.65300000000000002,"5min":0.53900000000000003,"15min":0.52000000000000002},"min":{"1min":0.42799999999999999,"5min":0.41299999999999998,"15min":0.41299999999999998},"max":{"1min":1.4590000000000001,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.63400000000000001}]}]},{"osd":3,"up_from":26,"seq":111669149897,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321920,"kb_used_data":265576,"kb_used_omap":0,"kb_used_meta":55616,"kb_avail":93423232,"statfs":{"total":95995035648,"available":95665389568,"internally_reserved":0,"allocated":271949824,"data_stored":359797735,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":56950784},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.32100000000000001,"5min":0.32000000000000001,"15min":0.33300000000000002},"min":{"1min":0.20799999999999999,"5min":0.184,"15min":0.17699999999999999},"max":{"1min":0.46700000000000003,"5min":1.1499999999999999,"15min":2.625},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.309,"5min":0.31,"15min":0.33900000000000002},"min":{"1min":0.20000000000000001,"5min":0.19800000000000001,"15min":0.189},"max":{"1min":0.48499999999999999,"5min":1.419,"15min":2.6339999999999999},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.39500000000000002},"min":{"1min":0.23599999999999999,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":0.49199999999999999,"5min":0.72599999999999998,"15min":6.0199999999999996},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.36299999999999999,"15min":0.39900000000000002},"min":{"1min":0.22500000000000001,"5min":0.20699999999999999,"15min":0.20699999999999999},"max":{"1min":0.498,"5min":2.5750000000000002,"15min":7.6130000000000004},"last":0.39900000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:32:35 2021","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.35699999999999998,"15min":0.35599999999999998},"min":{"1min":0.27100000000000002,"5min":0.216,"15min":0.216},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.39300000000000002,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.29299999999999998,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.56100000000000005,"5min":0.56100000000000005,"15min":0.56100000000000005},"last":0.35199999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:32:26 2021","interfaces":[{"interface":"back","average":{"1min":0.54900000000000004,"5min":0.51500000000000001,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.86899999999999999,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55400000000000005,"5min":0.497,"15min":0.52800000000000002},"min":{"1min":0.34599999999999997,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.82699999999999996,"5min":2.1059999999999999,"15min":2.2890000000000001},"last":0.69999999999999996}]},{"osd":5,"last update":"Mon Mar 15 22:32:26 2021","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.73399999999999999,"15min":0.82199999999999995},"min":{"1min":0.311,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.75800000000000001,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":0.68300000000000005},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.91500000000000004,"15min":1.083},"min":{"1min":0.38,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.79400000000000004,"5min":15.776,"15min":15.776},"last":0.54500000000000004}]},{"osd":6,"last update":"Mon Mar 15 22:32:55 2021","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.53600000000000003,"15min":0.56799999999999995},"min":{"1min":0.42899999999999999,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":0.88300000000000001,"5min":2.5299999999999998,"15min":2.6499999999999999},"last":0.70899999999999996},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.56000000000000005,"15min":0.56299999999999994},"min":{"1min":0.38500000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.77900000000000003,"5min":3.472,"15min":3.472},"last":0.61399999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:32:16 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.59699999999999998,"15min":0.57999999999999996},"min":{"1min":0.38600000000000001,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.81100000000000005,"5min":5.625,"15min":5.625},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.53000000000000003,"15min":0.56000000000000005},"min":{"1min":0.371,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.78800000000000003,"5min":2.5409999999999999,"15min":2.5409999999999999},"last":0.73199999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986374,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":114844,"kb_used_data":84140,"kb_used_omap":0,"kb_used_meta":30336,"kb_avail":93630308,"statfs":{"total":95995035648,"available":95877435392,"internally_reserved":0,"allocated":86159360,"data_stored":109081695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":31064064},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.52900000000000003,"15min":0.48499999999999999},"min":{"1min":0.24199999999999999,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":0.73199999999999998,"5min":4.7050000000000001,"15min":4.7050000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.54300000000000004,"15min":0.503},"min":{"1min":0.35799999999999998,"5min":0.246,"15min":0.246},"max":{"1min":0.65500000000000003,"5min":5.0209999999999999,"15min":5.0209999999999999},"last":0.47599999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.505,"15min":0.48499999999999999},"min":{"1min":0.29199999999999998,"5min":0.23999999999999999,"15min":0.23599999999999999},"max":{"1min":0.67900000000000005,"5min":4.3499999999999996,"15min":4.3499999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51200000000000001,"15min":0.50800000000000001},"min":{"1min":0.28499999999999998,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.86599999999999999,"5min":4.718,"15min":4.718},"last":0.46899999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:32:32 2021","interfaces":[{"interface":"back","average":{"1min":0.5,"5min":0.51000000000000001,"15min":0.51200000000000001},"min":{"1min":0.316,"5min":0.316,"15min":0.316},"max":{"1min":0.69099999999999995,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.497,"15min":0.499},"min":{"1min":0.36199999999999999,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.73699999999999999,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.44800000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.58899999999999997,"15min":0.55000000000000004},"min":{"1min":0.43099999999999999,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":0.88300000000000001,"5min":5.0499999999999998,"15min":5.0499999999999998},"last":0.53900000000000003},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.57099999999999995,"15min":0.54200000000000004},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.89500000000000002,"5min":4.4050000000000002,"15min":4.4050000000000002},"last":0.46500000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:32:30 2021","interfaces":[{"interface":"back","average":{"1min":0.439,"5min":0.42299999999999999,"15min":0.40899999999999997},"min":{"1min":0.28799999999999998,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.75900000000000001,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.44900000000000001,"15min":0.42699999999999999},"min":{"1min":0.30399999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66000000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.41299999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:32:57 2021","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.46700000000000003,"15min":0.46100000000000002},"min":{"1min":0.28399999999999997,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.63100000000000001,"5min":4.3079999999999998,"15min":4.3079999999999998},"last":0.38100000000000001},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.54600000000000004,"15min":0.53600000000000003},"min":{"1min":0.38800000000000001,"5min":0.28000000000000003,"15min":0.28000000000000003},"max":{"1min":0.71499999999999997,"5min":4.6600000000000001,"15min":4.6600000000000001},"last":0.45300000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:32:14 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.51700000000000002,"15min":0.50800000000000001},"min":{"1min":0.32100000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.79900000000000004,"5min":4.6929999999999996,"15min":4.6929999999999996},"last":0.438},{"interface":"front","average":{"1min":0.57599999999999996,"5min":0.56399999999999995,"15min":0.54700000000000004},"min":{"1min":0.35799999999999998,"5min":0.25,"15min":0.25},"max":{"1min":1.034,"5min":4.9900000000000002,"15min":4.9900000000000002},"last":0.39800000000000002}]}]},{"osd":5,"up_from":616,"seq":2645699854383,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":177520,"kb_used_data":153752,"kb_used_omap":153,"kb_used_meta":22886,"kb_avail":93567632,"statfs":{"total":95995035648,"available":95813255168,"internally_reserved":0,"allocated":157442048,"data_stored":205928528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":23435793},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.46000000000000002,"5min":0.46400000000000002,"15min":0.45600000000000002},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.65500000000000003,"5min":0.85399999999999998,"15min":0.85399999999999998},"last":0.5},{"interface":"front","average":{"1min":0.442,"5min":0.45600000000000002,"15min":0.44700000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.67700000000000005,"5min":0.68899999999999995,"15min":0.68899999999999995},"last":0.54200000000000004}]},{"osd":1,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.46400000000000002,"15min":0.45200000000000001},"min":{"1min":0.312,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.73199999999999998,"5min":0.73199999999999998,"15min":0.73199999999999998},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.501,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.74299999999999999,"5min":0.876,"15min":0.876},"last":0.52100000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.497,"15min":0.50700000000000001},"min":{"1min":0.29099999999999998,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":0.622,"5min":0.871,"15min":0.871},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52800000000000002,"15min":0.53100000000000003},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.49199999999999999,"15min":0.47599999999999998},"min":{"1min":0.39200000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.071,"5min":1.071,"15min":1.071},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.45800000000000002,"15min":0.441},"min":{"1min":0.315,"5min":0.315,"15min":0.315},"max":{"1min":0.998,"5min":0.998,"15min":0.998},"last":0.48799999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.40100000000000002,"15min":0.378},"min":{"1min":0.32300000000000001,"5min":0.26000000000000001,"15min":0.26000000000000001},"max":{"1min":0.86499999999999999,"5min":0.86499999999999999,"15min":0.86499999999999999},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.44700000000000001,"5min":0.40699999999999997,"15min":0.38600000000000001},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.81499999999999995,"5min":0.82299999999999995,"15min":0.82299999999999995},"last":0.42299999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.45800000000000002,"15min":0.44800000000000001},"min":{"1min":0.32100000000000001,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.83999999999999997,"5min":0.91200000000000003,"15min":0.91200000000000003},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.48599999999999999,"15min":0.46600000000000003},"min":{"1min":0.39000000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":0.89700000000000002,"5min":0.89700000000000002,"15min":0.89700000000000002},"last":0.44500000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:32:33 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.45500000000000002,"15min":0.439},"min":{"1min":0.29799999999999999,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.85199999999999998,"5min":0.85199999999999998,"15min":0.85199999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.53000000000000003,"5min":0.502,"15min":0.48799999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.88700000000000001,"5min":0.95499999999999996,"15min":0.95499999999999996},"last":0.41199999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7420,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":348160,"data_stored":12458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":307200,"data_stored":11020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9628,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":208896,"data_stored":7500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":177131520,"data_stored":251938721,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":193630208,"data_stored":250812133,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":153366528,"data_stored":223630255,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":262537216,"data_stored":353756202,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":76410880,"data_stored":104063590,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":148889600,"data_stored":200376704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":133644288,"data_stored":164343733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":144510976,"data_stored":198068027,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:33:16.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool set unique_pool_0 pgp_num 16 2021-03-15T22:33:16.886 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:16.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:16.943 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:16.944 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:17.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:17.073 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:17.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:17.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:17.176 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:17.207 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:17.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:17.208 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:17.309 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:17.319 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:33:17.334 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:17.336 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:17.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:17.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:17.418 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:33:17.435 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:33:17.438 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:17.457 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:33:17.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:17.467 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:17.468 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:17.476 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:33:17.495 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:33:17.524 INFO:tasks.rados.rados.0.smithi076.stdout:3175: done (3 left) 2021-03-15T22:33:17.524 INFO:tasks.rados.rados.0.smithi076.stdout:3176: write (excl) oid 25 current snap is 371 2021-03-15T22:33:17.524 INFO:tasks.rados.rados.0.smithi076.stdout:3176: seq_num 1082 ranges {585174=741544,1766105=419069} 2021-03-15T22:33:17.525 INFO:tasks.rados.rados.0.smithi076.stdout:3176: writing smithi07694878-25 from 585174 to 1326718 tid 1 2021-03-15T22:33:17.526 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:33:17.531 INFO:tasks.rados.rados.0.smithi076.stdout:3176: writing smithi07694878-25 from 1766105 to 2185174 tid 2 2021-03-15T22:33:17.533 INFO:tasks.rados.rados.0.smithi076.stdout:3173: expect (ObjNum 1064 snap 361 seq_num 1064) 2021-03-15T22:33:17.557 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:33:17.569 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:17.587 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:33:17.589 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:33:17.589 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:33:17.590 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:33:17.590 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:33:17.590 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:33:17.591 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:33:17.591 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:33:17.591 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:33:17.594 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:17.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:17.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:17.694 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:33:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:33:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:33:17.694 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:33:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3021 7.6 7:6de8b070:::smithi07694878-15:head [copy-from ver 640] snapc e3=[e1,df,de,dd,db] ondisk+write+known_if_redirected e330)", 2021-03-15T22:33:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:33:17.695 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 552.60722974299995, 2021-03-15T22:33:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.124784161, 2021-03-15T22:33:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.696 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3021 2021-03-15T22:33:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.697 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:33:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082618+0000", 2021-03-15T22:33:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7749999999999999e-06 2021-03-15T22:33:17.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082622+0000", 2021-03-15T22:33:17.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.072e-06 2021-03-15T22:33:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082625+0000", 2021-03-15T22:33:17.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2641e-05 2021-03-15T22:33:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082638+0000", 2021-03-15T22:33:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9170000000000006e-06 2021-03-15T22:33:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082646+0000", 2021-03-15T22:33:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1200999999999997e-05 2021-03-15T22:33:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.704 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082727+0000", 2021-03-15T22:33:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7440000000000007e-06 2021-03-15T22:33:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:33:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082735+0000", 2021-03-15T22:33:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022422739999999998 2021-03-15T22:33:17.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.084977+0000", 2021-03-15T22:33:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010620000000000001 2021-03-15T22:33:17.707 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085083+0000", 2021-03-15T22:33:17.708 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10953049500000001 2021-03-15T22:33:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,7", 2021-03-15T22:33:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.194614+0000", 2021-03-15T22:33:17.709 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043572510000000004 2021-03-15T22:33:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:17.710 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198971+0000", 2021-03-15T22:33:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002724242 2021-03-15T22:33:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.201695+0000", 2021-03-15T22:33:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0054890260000000001 2021-03-15T22:33:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207184+0000", 2021-03-15T22:33:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4066000000000006e-05 2021-03-15T22:33:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207248+0000", 2021-03-15T22:33:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015425700000000001 2021-03-15T22:33:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207403+0000", 2021-03-15T22:33:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.124784161 2021-03-15T22:33:17.715 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.716 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3022 7.6 7:6de8b070:::smithi07694878-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e330)", 2021-03-15T22:33:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:33:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 552.60718937599995, 2021-03-15T22:33:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12494472, 2021-03-15T22:33:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.717 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.718 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3022 2021-03-15T22:33:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:33:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082659+0000", 2021-03-15T22:33:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2669999999999999e-06 2021-03-15T22:33:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082660+0000", 2021-03-15T22:33:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7409999999999999e-06 2021-03-15T22:33:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082662+0000", 2021-03-15T22:33:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2030000000000003e-06 2021-03-15T22:33:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082669+0000", 2021-03-15T22:33:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1479999999999999e-06 2021-03-15T22:33:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.724 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.082673+0000", 2021-03-15T22:33:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00037115800000000001 2021-03-15T22:33:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.083044+0000", 2021-03-15T22:33:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1779999999999996e-06 2021-03-15T22:33:17.726 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:33:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.083049+0000", 2021-03-15T22:33:17.727 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021797460000000002 2021-03-15T22:33:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085228+0000", 2021-03-15T22:33:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5878999999999999e-05 2021-03-15T22:33:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:33:17.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.085274+0000", 2021-03-15T22:33:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.112992462 2021-03-15T22:33:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198267+0000", 2021-03-15T22:33:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6129000000000002e-05 2021-03-15T22:33:17.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:17.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.198343+0000", 2021-03-15T22:33:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0090944800000000003 2021-03-15T22:33:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207437+0000", 2021-03-15T22:33:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1060000000000001e-05 2021-03-15T22:33:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207508+0000", 2021-03-15T22:33:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5268999999999999e-05 2021-03-15T22:33:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.207604+0000", 2021-03-15T22:33:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12494472 2021-03-15T22:33:17.742 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.743 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:679 7.5 e560/559 7:ac84e656:::smithi07694878-31:1a1 v 560'396, mlcod=560'392)", 2021-03-15T22:33:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:28:20.276500+0000", 2021-03-15T22:33:17.744 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 297.41334834999998, 2021-03-15T22:33:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051465865, 2021-03-15T22:33:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.745 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276500+0000", 2021-03-15T22:33:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.747 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276500+0000", 2021-03-15T22:33:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5639999999999996e-06 2021-03-15T22:33:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276505+0000", 2021-03-15T22:33:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7400000000000002e-06 2021-03-15T22:33:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.749 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276509+0000", 2021-03-15T22:33:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4290999999999999e-05 2021-03-15T22:33:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276523+0000", 2021-03-15T22:33:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2339999999999997e-06 2021-03-15T22:33:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276531+0000", 2021-03-15T22:33:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0612999999999999e-05 2021-03-15T22:33:17.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276571+0000", 2021-03-15T22:33:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4697999999999999e-05 2021-03-15T22:33:17.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.276606+0000", 2021-03-15T22:33:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051301986000000001 2021-03-15T22:33:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:17.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.327908+0000", 2021-03-15T22:33:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7738999999999997e-05 2021-03-15T22:33:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:20.327966+0000", 2021-03-15T22:33:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.051465865 2021-03-15T22:33:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:5830 7.5 e631/559 7:ac84e656:::smithi07694878-31:head v 631'458, mlcod=630'454)", 2021-03-15T22:33:17.758 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:40.251563+0000", 2021-03-15T22:33:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 217.43828477400001, 2021-03-15T22:33:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071683063000000005, 2021-03-15T22:33:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.759 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251563+0000", 2021-03-15T22:33:17.760 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251563+0000", 2021-03-15T22:33:17.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052985690000000004 2021-03-15T22:33:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256862+0000", 2021-03-15T22:33:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2240000000000001e-06 2021-03-15T22:33:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256871+0000", 2021-03-15T22:33:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3723000000000002e-05 2021-03-15T22:33:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256905+0000", 2021-03-15T22:33:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2787e-05 2021-03-15T22:33:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256918+0000", 2021-03-15T22:33:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6076000000000006e-05 2021-03-15T22:33:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257014+0000", 2021-03-15T22:33:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2570999999999997e-05 2021-03-15T22:33:17.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257046+0000", 2021-03-15T22:33:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066154983000000001 2021-03-15T22:33:17.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:17.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.323201+0000", 2021-03-15T22:33:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5130000000000003e-05 2021-03-15T22:33:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.323246+0000", 2021-03-15T22:33:17.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071683063000000005 2021-03-15T22:33:17.771 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.771 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.771 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:755 7.5 e646/559 7:a9545d62:::smithi07694878-1:1fc v 646'483, mlcod=644'479)", 2021-03-15T22:33:17.772 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:55.246251+0000", 2021-03-15T22:33:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 202.44359679300001, 2021-03-15T22:33:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088387691000000004, 2021-03-15T22:33:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.773 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246251+0000", 2021-03-15T22:33:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246251+0000", 2021-03-15T22:33:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5550000000000001e-06 2021-03-15T22:33:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246254+0000", 2021-03-15T22:33:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3700000000000002e-06 2021-03-15T22:33:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246256+0000", 2021-03-15T22:33:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6309999999999998e-06 2021-03-15T22:33:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246266+0000", 2021-03-15T22:33:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6929999999999998e-06 2021-03-15T22:33:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246271+0000", 2021-03-15T22:33:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5197000000000001e-05 2021-03-15T22:33:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246316+0000", 2021-03-15T22:33:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3624e-05 2021-03-15T22:33:17.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.783 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246369+0000", 2021-03-15T22:33:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088224576999999998 2021-03-15T22:33:17.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334594+0000", 2021-03-15T22:33:17.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5043999999999998e-05 2021-03-15T22:33:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334639+0000", 2021-03-15T22:33:17.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088387691000000004 2021-03-15T22:33:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.789 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6453 7.9 7:91d654eb:::smithi07694878-5:head [write 739739~573968,stat] snapc 22b=[22b,229,228,227,226,225,223,221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1a1,17a,176] ondisk+write+known_if_redirected e691)", 2021-03-15T22:33:17.791 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.223418+0000", 2021-03-15T22:33:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 157.46642987000001, 2021-03-15T22:33:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15525217999999999, 2021-03-15T22:33:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.792 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.793 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6453 2021-03-15T22:33:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223418+0000", 2021-03-15T22:33:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223418+0000", 2021-03-15T22:33:17.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012317799999999999 2021-03-15T22:33:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223541+0000", 2021-03-15T22:33:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0259999999999998e-06 2021-03-15T22:33:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223547+0000", 2021-03-15T22:33:17.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9865999999999999e-05 2021-03-15T22:33:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223566+0000", 2021-03-15T22:33:17.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4069999999999993e-06 2021-03-15T22:33:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223575+0000", 2021-03-15T22:33:17.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0875000000000003e-05 2021-03-15T22:33:17.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223626+0000", 2021-03-15T22:33:17.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7320000000000003e-06 2021-03-15T22:33:17.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:33:17.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223631+0000", 2021-03-15T22:33:17.807 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022595760000000001 2021-03-15T22:33:17.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.808 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.225891+0000", 2021-03-15T22:33:17.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112502 2021-03-15T22:33:17.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.226003+0000", 2021-03-15T22:33:17.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000190173 2021-03-15T22:33:17.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,4", 2021-03-15T22:33:17.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.226194+0000", 2021-03-15T22:33:17.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088675127000000006 2021-03-15T22:33:17.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:17.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.314869+0000", 2021-03-15T22:33:17.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045894036999999999 2021-03-15T22:33:17.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:17.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.360763+0000", 2021-03-15T22:33:17.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017758818999999999 2021-03-15T22:33:17.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:17.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378522+0000", 2021-03-15T22:33:17.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9515999999999998e-05 2021-03-15T22:33:17.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:17.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378571+0000", 2021-03-15T22:33:17.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9345999999999994e-05 2021-03-15T22:33:17.817 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378670+0000", 2021-03-15T22:33:17.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15525217999999999 2021-03-15T22:33:17.818 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.819 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.819 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.820 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6454 7.9 7:91d654eb:::smithi07694878-5:head [write 2053414~686325] snapc 22b=[22b,229,228,227,226,225,223,221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1a1,17a,176] ondisk+write+known_if_redirected e691)", 2021-03-15T22:33:17.820 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.230948+0000", 2021-03-15T22:33:17.820 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 157.45889987000001, 2021-03-15T22:33:17.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15108200699999999, 2021-03-15T22:33:17.821 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.821 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:17.822 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.822 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.822 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.823 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6454 2021-03-15T22:33:17.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.824 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.230948+0000", 2021-03-15T22:33:17.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.230948+0000", 2021-03-15T22:33:17.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018821300000000001 2021-03-15T22:33:17.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.231137+0000", 2021-03-15T22:33:17.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5119999999999999e-06 2021-03-15T22:33:17.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.231142+0000", 2021-03-15T22:33:17.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066237401000000001 2021-03-15T22:33:17.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297379+0000", 2021-03-15T22:33:17.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2347999999999999e-05 2021-03-15T22:33:17.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297392+0000", 2021-03-15T22:33:17.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019934380000000002 2021-03-15T22:33:17.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299385+0000", 2021-03-15T22:33:17.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108266 2021-03-15T22:33:17.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:17.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299493+0000", 2021-03-15T22:33:17.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079232722000000005 2021-03-15T22:33:17.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378726+0000", 2021-03-15T22:33:17.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011364799999999999 2021-03-15T22:33:17.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378840+0000", 2021-03-15T22:33:17.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021629 2021-03-15T22:33:17.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,4", 2021-03-15T22:33:17.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.379056+0000", 2021-03-15T22:33:17.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021122210000000001 2021-03-15T22:33:17.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:17.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381168+0000", 2021-03-15T22:33:17.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0516000000000001e-05 2021-03-15T22:33:17.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:17.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381189+0000", 2021-03-15T22:33:17.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071458899999999996 2021-03-15T22:33:17.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:17.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381903+0000", 2021-03-15T22:33:17.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6925e-05 2021-03-15T22:33:17.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:17.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381970+0000", 2021-03-15T22:33:17.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9917999999999998e-05 2021-03-15T22:33:17.852 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382030+0000", 2021-03-15T22:33:17.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15108200699999999 2021-03-15T22:33:17.853 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.853 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.853 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6455 7.9 7:91d654eb:::smithi07694878-5:head [setxattr _header (57) in=7b,truncate 2739739] snapc 22b=[22b,229,228,227,226,225,223,221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1a1,17a,176] ondisk+write+known_if_redirected e691)", 2021-03-15T22:33:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.297411+0000", 2021-03-15T22:33:17.854 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 157.39243672399999, 2021-03-15T22:33:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.084805948000000006, 2021-03-15T22:33:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.855 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6455 2021-03-15T22:33:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.856 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297411+0000", 2021-03-15T22:33:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297411+0000", 2021-03-15T22:33:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8290000000000001e-06 2021-03-15T22:33:17.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297415+0000", 2021-03-15T22:33:17.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6919999999999998e-06 2021-03-15T22:33:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297418+0000", 2021-03-15T22:33:17.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5140000000000007e-06 2021-03-15T22:33:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297427+0000", 2021-03-15T22:33:17.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8450000000000002e-06 2021-03-15T22:33:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.862 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297432+0000", 2021-03-15T22:33:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020918960000000002 2021-03-15T22:33:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299524+0000", 2021-03-15T22:33:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3504000000000006e-05 2021-03-15T22:33:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:17.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299608+0000", 2021-03-15T22:33:17.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080542796999999999 2021-03-15T22:33:17.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.865 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.380151+0000", 2021-03-15T22:33:17.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4352999999999997e-05 2021-03-15T22:33:17.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.380235+0000", 2021-03-15T22:33:17.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013269599999999999 2021-03-15T22:33:17.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,4", 2021-03-15T22:33:17.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.380368+0000", 2021-03-15T22:33:17.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084666400000000001 2021-03-15T22:33:17.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.868 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:17.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381214+0000", 2021-03-15T22:33:17.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6737000000000003e-05 2021-03-15T22:33:17.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:17.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381261+0000", 2021-03-15T22:33:17.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00082662899999999995 2021-03-15T22:33:17.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:17.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382088+0000", 2021-03-15T22:33:17.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8632999999999999e-05 2021-03-15T22:33:17.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:17.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382146+0000", 2021-03-15T22:33:17.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1159000000000001e-05 2021-03-15T22:33:17.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382217+0000", 2021-03-15T22:33:17.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.084805948000000006 2021-03-15T22:33:17.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.874 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.875 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6456 7.9 7:91d654eb:::smithi07694878-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e691)", 2021-03-15T22:33:17.875 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.297438+0000", 2021-03-15T22:33:17.875 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 157.39241031700001, 2021-03-15T22:33:17.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.085000942999999995, 2021-03-15T22:33:17.876 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.876 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:17.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.877 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6456 2021-03-15T22:33:17.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.877 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297438+0000", 2021-03-15T22:33:17.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297438+0000", 2021-03-15T22:33:17.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.012e-06 2021-03-15T22:33:17.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297439+0000", 2021-03-15T22:33:17.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8310000000000003e-06 2021-03-15T22:33:17.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297446+0000", 2021-03-15T22:33:17.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7070000000000003e-06 2021-03-15T22:33:17.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.882 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297452+0000", 2021-03-15T22:33:17.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0090000000000001e-06 2021-03-15T22:33:17.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297456+0000", 2021-03-15T22:33:17.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002183955 2021-03-15T22:33:17.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.884 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299640+0000", 2021-03-15T22:33:17.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3368999999999993e-05 2021-03-15T22:33:17.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:17.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299734+0000", 2021-03-15T22:33:17.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080901084999999998 2021-03-15T22:33:17.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.380635+0000", 2021-03-15T22:33:17.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1402000000000003e-05 2021-03-15T22:33:17.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:17.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.380726+0000", 2021-03-15T22:33:17.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015328080000000001 2021-03-15T22:33:17.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382259+0000", 2021-03-15T22:33:17.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9952999999999997e-05 2021-03-15T22:33:17.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382349+0000", 2021-03-15T22:33:17.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9812e-05 2021-03-15T22:33:17.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382439+0000", 2021-03-15T22:33:17.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.085000942999999995 2021-03-15T22:33:17.891 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.891 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.892 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.892 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6965 7.7 7:ed4a01d2:::smithi07694878-2:head [sparse-read 0~3616366,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e740)", 2021-03-15T22:33:17.892 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.977912+0000", 2021-03-15T22:33:17.893 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 106.71193634399999, 2021-03-15T22:33:17.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27620877300000002, 2021-03-15T22:33:17.893 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.893 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:17.894 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.894 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.894 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.894 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6965 2021-03-15T22:33:17.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.895 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977912+0000", 2021-03-15T22:33:17.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.896 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977912+0000", 2021-03-15T22:33:17.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3100000000000002e-06 2021-03-15T22:33:17.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977916+0000", 2021-03-15T22:33:17.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1709999999999998e-06 2021-03-15T22:33:17.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977919+0000", 2021-03-15T22:33:17.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.258e-05 2021-03-15T22:33:17.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977932+0000", 2021-03-15T22:33:17.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4590000000000004e-06 2021-03-15T22:33:17.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977941+0000", 2021-03-15T22:33:17.901 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0978e-05 2021-03-15T22:33:17.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977982+0000", 2021-03-15T22:33:17.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2990000000000002e-06 2021-03-15T22:33:17.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:17.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977989+0000", 2021-03-15T22:33:17.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27114430499999997 2021-03-15T22:33:17.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249133+0000", 2021-03-15T22:33:17.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000114636 2021-03-15T22:33:17.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.905 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249248+0000", 2021-03-15T22:33:17.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048730350000000004 2021-03-15T22:33:17.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254121+0000", 2021-03-15T22:33:17.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27620877300000002 2021-03-15T22:33:17.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.907 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.907 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.908 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6966 7.7 7:ed4a01d2:::smithi07694878-2:head [sparse-read 0~3616366] snapc 0=[] ondisk+read+known_if_redirected e740)", 2021-03-15T22:33:17.908 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.977955+0000", 2021-03-15T22:33:17.908 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 106.71189364599999, 2021-03-15T22:33:17.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27650659300000002, 2021-03-15T22:33:17.909 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.909 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:17.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.909 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.910 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.910 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6966 2021-03-15T22:33:17.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.910 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977955+0000", 2021-03-15T22:33:17.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977955+0000", 2021-03-15T22:33:17.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3510000000000001e-06 2021-03-15T22:33:17.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977956+0000", 2021-03-15T22:33:17.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4020000000000001e-06 2021-03-15T22:33:17.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.914 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977957+0000", 2021-03-15T22:33:17.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1184e-05 2021-03-15T22:33:17.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.915 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977968+0000", 2021-03-15T22:33:17.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4050000000000002e-06 2021-03-15T22:33:17.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977973+0000", 2021-03-15T22:33:17.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8658000000000002e-05 2021-03-15T22:33:17.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.978012+0000", 2021-03-15T22:33:17.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8309999999999997e-06 2021-03-15T22:33:17.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:17.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.978017+0000", 2021-03-15T22:33:17.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27614046599999997 2021-03-15T22:33:17.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254158+0000", 2021-03-15T22:33:17.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010246700000000001 2021-03-15T22:33:17.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254260+0000", 2021-03-15T22:33:17.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000200829 2021-03-15T22:33:17.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.922 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.922 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254461+0000", 2021-03-15T22:33:17.922 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27650659300000002 2021-03-15T22:33:17.922 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.922 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.923 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.923 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.923 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6967 7.7 7:ed4a01d2:::smithi07694878-2:head [sparse-read 0~3616366] snapc 0=[] ondisk+read+known_if_redirected e740)", 2021-03-15T22:33:17.924 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.977987+0000", 2021-03-15T22:33:17.924 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 106.71186097099999, 2021-03-15T22:33:17.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27677133100000001, 2021-03-15T22:33:17.924 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.924 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:17.925 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.925 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.925 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.925 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6967 2021-03-15T22:33:17.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.926 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977987+0000", 2021-03-15T22:33:17.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977987+0000", 2021-03-15T22:33:17.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.184e-06 2021-03-15T22:33:17.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977988+0000", 2021-03-15T22:33:17.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4130000000000001e-06 2021-03-15T22:33:17.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977990+0000", 2021-03-15T22:33:17.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9290000000000003e-06 2021-03-15T22:33:17.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977996+0000", 2021-03-15T22:33:17.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2189999999999998e-06 2021-03-15T22:33:17.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.978000+0000", 2021-03-15T22:33:17.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0028999999999999e-05 2021-03-15T22:33:17.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.978040+0000", 2021-03-15T22:33:17.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4139999999999998e-06 2021-03-15T22:33:17.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:17.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.978043+0000", 2021-03-15T22:33:17.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27649842800000002 2021-03-15T22:33:17.935 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.935 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.935 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.935 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254542+0000", 2021-03-15T22:33:17.936 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5296000000000006e-05 2021-03-15T22:33:17.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.936 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.936 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254617+0000", 2021-03-15T22:33:17.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000141419 2021-03-15T22:33:17.937 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.937 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.937 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254759+0000", 2021-03-15T22:33:17.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27677133100000001 2021-03-15T22:33:17.938 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.938 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.938 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.939 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7260 7.f 7:fc7cd984:::smithi07694878-4:head [delete] snapc 27b=[27b,276,275,271,26f,26e,26c,268,267,266,265,261,25d,25b,257,255,24f,24c,24b,24a,248,247,243,23e,23d,239,234,22d,218,1f7,1c7] ondisk+write+peerstat_old+known_if_redirected e772)", 2021-03-15T22:33:17.939 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:02.265878+0000", 2021-03-15T22:33:17.940 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.423969948999996, 2021-03-15T22:33:17.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069591505999999997, 2021-03-15T22:33:17.940 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.940 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:17.940 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.941 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.941 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.941 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7260 2021-03-15T22:33:17.941 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.942 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.265878+0000", 2021-03-15T22:33:17.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.265878+0000", 2021-03-15T22:33:17.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.189e-06 2021-03-15T22:33:17.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.265882+0000", 2021-03-15T22:33:17.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4120000000000001e-06 2021-03-15T22:33:17.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.265886+0000", 2021-03-15T22:33:17.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058741452999999999 2021-03-15T22:33:17.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.324627+0000", 2021-03-15T22:33:17.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2182e-05 2021-03-15T22:33:17.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.324639+0000", 2021-03-15T22:33:17.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032676519999999998 2021-03-15T22:33:17.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.949 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.327907+0000", 2021-03-15T22:33:17.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000105665 2021-03-15T22:33:17.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:17.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328013+0000", 2021-03-15T22:33:17.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000235035 2021-03-15T22:33:17.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:33:17.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328248+0000", 2021-03-15T22:33:17.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00205624 2021-03-15T22:33:17.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:17.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.330304+0000", 2021-03-15T22:33:17.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5997000000000001e-05 2021-03-15T22:33:17.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:17.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.330340+0000", 2021-03-15T22:33:17.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050271780000000002 2021-03-15T22:33:17.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:17.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.335367+0000", 2021-03-15T22:33:17.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4549999999999998e-05 2021-03-15T22:33:17.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:17.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.335422+0000", 2021-03-15T22:33:17.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7953e-05 2021-03-15T22:33:17.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.335470+0000", 2021-03-15T22:33:17.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069591505999999997 2021-03-15T22:33:17.957 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.957 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.958 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7700 7.4 7:26337d05:::smithi07694878-20:2a2 [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e816)", 2021-03-15T22:33:17.958 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:46.291125+0000", 2021-03-15T22:33:17.959 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.398722795000001, 2021-03-15T22:33:17.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056649433999999999, 2021-03-15T22:33:17.959 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.959 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:17.960 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.960 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.960 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.960 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7700 2021-03-15T22:33:17.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.961 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291125+0000", 2021-03-15T22:33:17.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291125+0000", 2021-03-15T22:33:17.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8139999999999998e-06 2021-03-15T22:33:17.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291129+0000", 2021-03-15T22:33:17.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5050000000000002e-06 2021-03-15T22:33:17.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291132+0000", 2021-03-15T22:33:17.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5413999999999998e-05 2021-03-15T22:33:17.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291147+0000", 2021-03-15T22:33:17.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2910000000000004e-06 2021-03-15T22:33:17.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.966 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291153+0000", 2021-03-15T22:33:17.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4326999999999997e-05 2021-03-15T22:33:17.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.967 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291208+0000", 2021-03-15T22:33:17.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8760000000000002e-06 2021-03-15T22:33:17.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:33:17.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291214+0000", 2021-03-15T22:33:17.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056377700000000003 2021-03-15T22:33:17.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347591+0000", 2021-03-15T22:33:17.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018350699999999999 2021-03-15T22:33:17.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347775+0000", 2021-03-15T22:33:17.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056649433999999999 2021-03-15T22:33:17.972 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.972 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.972 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.973 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7701 7.4 7:26337d05:::smithi07694878-20:2a2 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e816)", 2021-03-15T22:33:17.973 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:46.291166+0000", 2021-03-15T22:33:17.973 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.398681774, 2021-03-15T22:33:17.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056734093999999999, 2021-03-15T22:33:17.973 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.974 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:17.974 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.974 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.974 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.975 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7701 2021-03-15T22:33:17.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.975 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291166+0000", 2021-03-15T22:33:17.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291166+0000", 2021-03-15T22:33:17.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4200000000000004e-07 2021-03-15T22:33:17.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291167+0000", 2021-03-15T22:33:17.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1009999999999999e-06 2021-03-15T22:33:17.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291168+0000", 2021-03-15T22:33:17.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.553e-06 2021-03-15T22:33:17.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291174+0000", 2021-03-15T22:33:17.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8510000000000001e-06 2021-03-15T22:33:17.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291178+0000", 2021-03-15T22:33:17.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0061000000000005e-05 2021-03-15T22:33:17.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291258+0000", 2021-03-15T22:33:17.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1459999999999998e-06 2021-03-15T22:33:17.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:33:17.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291262+0000", 2021-03-15T22:33:17.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056538214000000003 2021-03-15T22:33:17.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347800+0000", 2021-03-15T22:33:17.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100226 2021-03-15T22:33:17.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:17.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347901+0000", 2021-03-15T22:33:17.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056734093999999999 2021-03-15T22:33:17.986 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.986 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:17.987 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:17.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.987 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7702 7.4 7:26337d05:::smithi07694878-20:2a2 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e816)", 2021-03-15T22:33:17.987 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:33:17.988 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 31.398664782000001, 2021-03-15T22:33:17.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056836219, 2021-03-15T22:33:17.988 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:17.988 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:17.988 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:17.989 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:17.989 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:17.989 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7702 2021-03-15T22:33:17.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.990 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:17.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:17.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:33:17.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:17.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:17.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:33:17.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6599999999999994e-07 2021-03-15T22:33:17.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:17.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291184+0000", 2021-03-15T22:33:17.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.249e-06 2021-03-15T22:33:17.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:17.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291186+0000", 2021-03-15T22:33:17.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5099999999999998e-06 2021-03-15T22:33:17.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.994 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:17.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291191+0000", 2021-03-15T22:33:17.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7519999999999998e-06 2021-03-15T22:33:17.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:17.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291195+0000", 2021-03-15T22:33:17.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0414999999999995e-05 2021-03-15T22:33:17.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291275+0000", 2021-03-15T22:33:17.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2119999999999999e-06 2021-03-15T22:33:17.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:33:17.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291279+0000", 2021-03-15T22:33:17.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056643645999999999 2021-03-15T22:33:17.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:17.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:17.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:17.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347922+0000", 2021-03-15T22:33:17.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7468999999999998e-05 2021-03-15T22:33:17.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:18.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.348020+0000", 2021-03-15T22:33:18.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056836219 2021-03-15T22:33:18.001 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:18.001 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:18.001 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:18.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.002 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7877 7.f 7:fc7cd984:::smithi07694878-4:head [copy-from ver 1977] snapc 2b8=[2b8,2b7,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e834)", 2021-03-15T22:33:18.002 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:04.322675+0000", 2021-03-15T22:33:18.002 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.367173169000001, 2021-03-15T22:33:18.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0491417890000001, 2021-03-15T22:33:18.003 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:18.003 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:18.003 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:18.003 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:18.003 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:18.004 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7877 2021-03-15T22:33:18.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.004 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:18.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:18.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.322675+0000", 2021-03-15T22:33:18.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:18.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:18.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.322675+0000", 2021-03-15T22:33:18.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0920000000000002e-06 2021-03-15T22:33:18.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:18.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.322678+0000", 2021-03-15T22:33:18.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.898e-06 2021-03-15T22:33:18.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:18.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.322681+0000", 2021-03-15T22:33:18.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001547157 2021-03-15T22:33:18.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:18.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324228+0000", 2021-03-15T22:33:18.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0239e-05 2021-03-15T22:33:18.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:18.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324238+0000", 2021-03-15T22:33:18.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010807830000000001 2021-03-15T22:33:18.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:18.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.325319+0000", 2021-03-15T22:33:18.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012095 2021-03-15T22:33:18.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:18.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.325440+0000", 2021-03-15T22:33:18.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0357033609999999 2021-03-15T22:33:18.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:33:18.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.361143+0000", 2021-03-15T22:33:18.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010042997 2021-03-15T22:33:18.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:18.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371186+0000", 2021-03-15T22:33:18.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018398700000000001 2021-03-15T22:33:18.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:18.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371370+0000", 2021-03-15T22:33:18.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028334399999999999 2021-03-15T22:33:18.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:18.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371654+0000", 2021-03-15T22:33:18.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6285999999999993e-05 2021-03-15T22:33:18.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:18.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371740+0000", 2021-03-15T22:33:18.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6694999999999997e-05 2021-03-15T22:33:18.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:18.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371817+0000", 2021-03-15T22:33:18.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0491417890000001 2021-03-15T22:33:18.020 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:18.020 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:18.020 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:18.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.021 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7878 7.f 7:fc7cd984:::smithi07694878-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e834)", 2021-03-15T22:33:18.021 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:04.324891+0000", 2021-03-15T22:33:18.021 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.364957051999999, 2021-03-15T22:33:18.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0471670769999999, 2021-03-15T22:33:18.021 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:18.022 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7878 2021-03-15T22:33:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324891+0000", 2021-03-15T22:33:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324891+0000", 2021-03-15T22:33:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.869e-06 2021-03-15T22:33:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324893+0000", 2021-03-15T22:33:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5579999999999999e-06 2021-03-15T22:33:18.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324896+0000", 2021-03-15T22:33:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6606e-05 2021-03-15T22:33:18.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324952+0000", 2021-03-15T22:33:18.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1832999999999997e-05 2021-03-15T22:33:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324994+0000", 2021-03-15T22:33:18.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00094264299999999995 2021-03-15T22:33:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:18.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.325937+0000", 2021-03-15T22:33:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9849000000000002e-05 2021-03-15T22:33:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:33:18.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.325996+0000", 2021-03-15T22:33:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.041140036 2021-03-15T22:33:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:18.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.367137+0000", 2021-03-15T22:33:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000177877 2021-03-15T22:33:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:18.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.367314+0000", 2021-03-15T22:33:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045396300000000002 2021-03-15T22:33:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371854+0000", 2021-03-15T22:33:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010181 2021-03-15T22:33:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:18.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371956+0000", 2021-03-15T22:33:18.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102366 2021-03-15T22:33:18.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:18.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.372058+0000", 2021-03-15T22:33:18.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0471670769999999 2021-03-15T22:33:18.037 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:18.037 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:18.038 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:18.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.038 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(osd.0.8:76 7.7 7:ed4a01d2:::smithi07694878-2:head [assert-version v1977,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e834)", 2021-03-15T22:33:18.039 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:04.327331+0000", 2021-03-15T22:33:18.039 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 13.362517093999999, 2021-03-15T22:33:18.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.03086123, 2021-03-15T22:33:18.039 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:18.039 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:18.040 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:18.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:18.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327331+0000", 2021-03-15T22:33:18.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:18.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:18.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327331+0000", 2021-03-15T22:33:18.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0818999999999998e-05 2021-03-15T22:33:18.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:18.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327352+0000", 2021-03-15T22:33:18.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1210000000000001e-06 2021-03-15T22:33:18.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:18.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327354+0000", 2021-03-15T22:33:18.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9869000000000002e-05 2021-03-15T22:33:18.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:18.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327384+0000", 2021-03-15T22:33:18.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1650000000000003e-06 2021-03-15T22:33:18.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:18.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327390+0000", 2021-03-15T22:33:18.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000230375 2021-03-15T22:33:18.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:18.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327620+0000", 2021-03-15T22:33:18.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6136e-05 2021-03-15T22:33:18.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:18.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327667+0000", 2021-03-15T22:33:18.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.029658271 2021-03-15T22:33:18.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:18.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.357325+0000", 2021-03-15T22:33:18.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018968899999999999 2021-03-15T22:33:18.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:18.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.357515+0000", 2021-03-15T22:33:18.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067778499999999998 2021-03-15T22:33:18.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:18.051 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.358192+0000", 2021-03-15T22:33:18.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.03086123 2021-03-15T22:33:18.052 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:18.052 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:18.052 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:18.052 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.053 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e835,834 7:ecc0cb20:::smithi07694878-25:cd,308'179)", 2021-03-15T22:33:18.053 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:06.265008+0000", 2021-03-15T22:33:18.053 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 11.424840122999999, 2021-03-15T22:33:18.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.108627554, 2021-03-15T22:33:18.054 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:18.054 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:18.054 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:18.054 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:18.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.265008+0000", 2021-03-15T22:33:18.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:18.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:18.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.265008+0000", 2021-03-15T22:33:18.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3300000000000001e-06 2021-03-15T22:33:18.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:18.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.265010+0000", 2021-03-15T22:33:18.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4719999999999997e-06 2021-03-15T22:33:18.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.058 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:18.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.265015+0000", 2021-03-15T22:33:18.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10817592299999999 2021-03-15T22:33:18.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:18.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.373191+0000", 2021-03-15T22:33:18.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1566000000000001e-05 2021-03-15T22:33:18.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:18.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.373202+0000", 2021-03-15T22:33:18.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7701999999999997e-05 2021-03-15T22:33:18.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:18.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.373260+0000", 2021-03-15T22:33:18.062 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000375561 2021-03-15T22:33:18.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:18.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:18.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:18.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.373636+0000", 2021-03-15T22:33:18.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.108627554 2021-03-15T22:33:18.063 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:18.063 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:18.063 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:18.063 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:18.064 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:18.064 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:33:18.064 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:18.066 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:17 smithi125 conmon[56739]: cluster 2021-03-15T22:33:16.504667+0000 mon.a 2021-03-15T22:33:18.066 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:17 smithi125 conmon[56739]: (mon.0) 2164 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2021-03-15T22:33:18.066 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:17 smithi125 conmon[56739]: audit 2021-03-15T22:33:17.396124+0000 mon.a (mon.0) 2165 : audit [DBG] from='client.? 172.21.15.76:0/371232711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:18.067 INFO:tasks.rados.rados.0.smithi076.stdout:3173: done (3 left) 2021-03-15T22:33:18.068 INFO:tasks.rados.rados.0.smithi076.stdout:3177: delete oid 18 current snap is 371 2021-03-15T22:33:18.068 INFO:tasks.rados.rados.0.smithi076.stdout:3176: finishing write tid 1 to smithi07694878-25 2021-03-15T22:33:18.068 INFO:tasks.rados.rados.0.smithi076.stdout:3176: finishing write tid 2 to smithi07694878-25 2021-03-15T22:33:18.068 INFO:tasks.rados.rados.0.smithi076.stdout:3176: finishing write tid 3 to smithi07694878-25 2021-03-15T22:33:18.069 INFO:tasks.rados.rados.0.smithi076.stdout:3176: finishing write tid 4 to smithi07694878-25 2021-03-15T22:33:18.069 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 1991 (ObjNum 1082 snap 371 seq_num 1082) dirty exists 2021-03-15T22:33:18.069 INFO:tasks.rados.rados.0.smithi076.stdout:3176: left oid 25 (ObjNum 1082 snap 371 seq_num 1082) 2021-03-15T22:33:18.069 INFO:tasks.rados.rados.0.smithi076.stdout:3176: done (3 left) 2021-03-15T22:33:18.070 INFO:tasks.rados.rados.0.smithi076.stdout:3177: done (2 left) 2021-03-15T22:33:18.070 INFO:tasks.rados.rados.0.smithi076.stdout:3178: read oid 1 snap -1 2021-03-15T22:33:18.070 INFO:tasks.rados.rados.0.smithi076.stdout:3178: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:18.070 INFO:tasks.rados.rados.0.smithi076.stdout:3179: write (excl) oid 42 current snap is 371 2021-03-15T22:33:18.070 INFO:tasks.rados.rados.0.smithi076.stdout:3179: seq_num 1084 ranges {618788=492042,1883069=335719} 2021-03-15T22:33:18.071 INFO:tasks.rados.rados.0.smithi076.stdout:3179: writing smithi07694878-42 from 618788 to 1110830 tid 1 2021-03-15T22:33:18.071 INFO:tasks.rados.rados.0.smithi076.stdout:3179: writing smithi07694878-42 from 1883069 to 2218788 tid 2 2021-03-15T22:33:18.071 INFO:tasks.rados.rados.0.smithi076.stdout:3180: read oid 12 snap -1 2021-03-15T22:33:18.071 INFO:tasks.rados.rados.0.smithi076.stdout:3180: expect (ObjNum 999 snap 335 seq_num 999) 2021-03-15T22:33:18.072 INFO:tasks.rados.rados.0.smithi076.stdout:3181: copy_from oid 20 from oid 10 current snap is 371 2021-03-15T22:33:18.072 INFO:tasks.rados.rados.0.smithi076.stdout:3178: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:18.072 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:17 smithi076 conmon[56514]: cluster 2021-03-15T22:33:16.504667+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2021-03-15T22:33:18.072 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:17 smithi076 conmon[56514]: audit 2021-03-15T22:33:17.396124+0000 mon.a (mon.0) 2165 : audit [DBG] from='client.? 172.21.15.76:0/371232711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:18.073 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:17 smithi076 conmon[62156]: cluster 2021-03-15T22:33:16.504667+0000 mon.a (mon.0) 2164 : cluster [DBG] osdmap e846: 8 total, 8 up, 8 in 2021-03-15T22:33:18.073 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:17 smithi076 conmon[62156]: audit 2021-03-15T22:33:17.396124+0000 mon.a (mon.0) 2165 : audit [DBG] from='client.? 172.21.15.76:0/371232711' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:18.076 INFO:tasks.rados.rados.0.smithi076.stdout:3179: finishing write tid 1 to smithi07694878-42 2021-03-15T22:33:18.077 INFO:tasks.rados.rados.0.smithi076.stdout:3180: expect (ObjNum 999 snap 335 seq_num 999) 2021-03-15T22:33:18.077 INFO:tasks.rados.rados.0.smithi076.stdout:3179: finishing write tid 2 to smithi07694878-42 2021-03-15T22:33:18.077 INFO:tasks.rados.rados.0.smithi076.stdout:3179: finishing write tid 3 to smithi07694878-42 2021-03-15T22:33:18.077 INFO:tasks.rados.rados.0.smithi076.stdout:3179: finishing write tid 4 to smithi07694878-42 2021-03-15T22:33:18.077 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2795 (ObjNum 1084 snap 371 seq_num 1084) dirty exists 2021-03-15T22:33:18.078 INFO:tasks.rados.rados.0.smithi076.stdout:3179: left oid 42 (ObjNum 1084 snap 371 seq_num 1084) 2021-03-15T22:33:18.078 INFO:tasks.rados.rados.0.smithi076.stdout:3178: done (5 left) 2021-03-15T22:33:18.078 INFO:tasks.rados.rados.0.smithi076.stdout:3179: done (4 left) 2021-03-15T22:33:18.078 INFO:tasks.rados.rados.0.smithi076.stdout:3180: done (3 left) 2021-03-15T22:33:18.079 INFO:tasks.rados.rados.0.smithi076.stdout:3182: delete oid 6 current snap is 371 2021-03-15T22:33:18.079 INFO:tasks.rados.rados.0.smithi076.stdout:3181: finishing copy_from to smithi07694878-20 2021-03-15T22:33:18.079 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 1985 (ObjNum 1077 snap 366 seq_num 1077) dirty exists 2021-03-15T22:33:18.079 INFO:tasks.rados.rados.0.smithi076.stdout:3181: finishing copy_from racing read to smithi07694878-20 2021-03-15T22:33:18.079 INFO:tasks.rados.rados.0.smithi076.stdout:3181: done (3 left) 2021-03-15T22:33:18.080 INFO:tasks.rados.rados.0.smithi076.stdout:3182: done (2 left) 2021-03-15T22:33:18.080 INFO:tasks.rados.rados.0.smithi076.stdout:3183: read oid 3 snap -1 2021-03-15T22:33:18.080 INFO:tasks.rados.rados.0.smithi076.stdout:3183: expect (ObjNum 620 snap 193 seq_num 620) 2021-03-15T22:33:18.080 INFO:tasks.rados.rados.0.smithi076.stdout:3184: copy_from oid 15 from oid 28 current snap is 371 2021-03-15T22:33:18.081 INFO:tasks.rados.rados.0.smithi076.stdout:3185: write oid 32 current snap is 371 2021-03-15T22:33:18.081 INFO:tasks.rados.rados.0.smithi076.stdout:3185: seq_num 1086 ranges {652402=642540,1883680=368722} 2021-03-15T22:33:18.081 INFO:tasks.rados.rados.0.smithi076.stdout:3185: writing smithi07694878-32 from 652402 to 1294942 tid 1 2021-03-15T22:33:18.081 INFO:tasks.rados.rados.0.smithi076.stdout:3185: writing smithi07694878-32 from 1883680 to 2252402 tid 2 2021-03-15T22:33:18.082 INFO:tasks.rados.rados.0.smithi076.stdout:3186: rollback oid 13 current snap is 371 2021-03-15T22:33:18.082 INFO:tasks.rados.rados.0.smithi076.stdout:3183: expect (ObjNum 620 snap 193 seq_num 620) 2021-03-15T22:33:18.082 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 13 to 279 2021-03-15T22:33:18.082 INFO:tasks.rados.rados.0.smithi076.stdout:3185: finishing write tid 1 to smithi07694878-32 2021-03-15T22:33:18.082 INFO:tasks.rados.rados.0.smithi076.stdout:3184: finishing copy_from to smithi07694878-15 2021-03-15T22:33:18.083 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 2757 (ObjNum 1070 snap 364 seq_num 1070) dirty exists 2021-03-15T22:33:18.083 INFO:tasks.rados.rados.0.smithi076.stdout:3184: finishing copy_from racing read to smithi07694878-15 2021-03-15T22:33:18.083 INFO:tasks.rados.rados.0.smithi076.stdout:3185: finishing write tid 2 to smithi07694878-32 2021-03-15T22:33:18.083 INFO:tasks.rados.rados.0.smithi076.stdout:3185: finishing write tid 3 to smithi07694878-32 2021-03-15T22:33:18.084 INFO:tasks.rados.rados.0.smithi076.stdout:3185: finishing write tid 4 to smithi07694878-32 2021-03-15T22:33:18.084 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 2799 (ObjNum 1086 snap 371 seq_num 1086) dirty exists 2021-03-15T22:33:18.084 INFO:tasks.rados.rados.0.smithi076.stdout:3185: left oid 32 (ObjNum 1086 snap 371 seq_num 1086) 2021-03-15T22:33:18.084 INFO:tasks.rados.rados.0.smithi076.stdout:3183: done (5 left) 2021-03-15T22:33:18.084 INFO:tasks.rados.rados.0.smithi076.stdout:3184: done (4 left) 2021-03-15T22:33:18.085 INFO:tasks.rados.rados.0.smithi076.stdout:3185: done (3 left) 2021-03-15T22:33:18.085 INFO:tasks.rados.rados.0.smithi076.stdout:3187: rollback oid 31 current snap is 371 2021-03-15T22:33:18.085 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 31 to 350 2021-03-15T22:33:18.085 INFO:tasks.rados.rados.0.smithi076.stdout:3188: read oid 46 snap -1 2021-03-15T22:33:18.085 INFO:tasks.rados.rados.0.smithi076.stdout:3188: expect (ObjNum 1044 snap 350 seq_num 1044) 2021-03-15T22:33:18.086 INFO:tasks.rados.rados.0.smithi076.stdout:3189: snap_create 2021-03-15T22:33:18.086 INFO:tasks.rados.rados.0.smithi076.stdout:3186: finishing rollback tid 1 to smithi07694878-13 2021-03-15T22:33:18.086 INFO:tasks.rados.rados.0.smithi076.stdout:3186: finishing rollback tid 2 to smithi07694878-13 2021-03-15T22:33:18.086 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1995 (ObjNum 797 snap 262 seq_num 797) dirty exists 2021-03-15T22:33:18.087 INFO:tasks.rados.rados.0.smithi076.stdout:3187: finishing rollback tid 0 to smithi07694878-31 2021-03-15T22:33:18.087 INFO:tasks.rados.rados.0.smithi076.stdout:3187: finishing rollback tid 1 to smithi07694878-31 2021-03-15T22:33:18.087 INFO:tasks.rados.rados.0.smithi076.stdout:3187: finishing rollback tid 2 to smithi07694878-31 2021-03-15T22:33:18.087 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 2405 (ObjNum 1020 snap 341 seq_num 1020) dirty exists 2021-03-15T22:33:18.087 INFO:tasks.rados.rados.0.smithi076.stdout:3188: expect (ObjNum 1044 snap 350 seq_num 1044) 2021-03-15T22:33:18.110 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:18.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:18.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:18.213 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:18.234 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:18.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:18.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:18.337 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:18.359 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:18.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:18.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:18.462 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:18.485 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:18.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:18.486 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:18.519 INFO:tasks.rados.rados.0.smithi076.stdout:3186: done (5 left) 2021-03-15T22:33:18.520 INFO:tasks.rados.rados.0.smithi076.stdout:3187: done (4 left) 2021-03-15T22:33:18.520 INFO:tasks.rados.rados.0.smithi076.stdout:3188: done (3 left) 2021-03-15T22:33:18.520 INFO:tasks.rados.rados.0.smithi076.stdout:3189: done (2 left) 2021-03-15T22:33:18.520 INFO:tasks.rados.rados.0.smithi076.stdout:3190: delete oid 20 current snap is 372 2021-03-15T22:33:18.523 INFO:tasks.rados.rados.0.smithi076.stdout:3190: done (2 left) 2021-03-15T22:33:18.523 INFO:tasks.rados.rados.0.smithi076.stdout:3191: snap_create 2021-03-15T22:33:18.588 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:18.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:18.615 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:18.616 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:18.717 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:18.743 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:18.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:18.745 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:18.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: cluster 2021-03-15T22:33:17.187854+0000 2021-03-15T22:33:18.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: mgr.y (mgr.14144) 976 : cluster [DBG] pgmap v1414: 148 pgs: 1 active+clean+wait, 147 active+clean; 549 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 886 KiB/s wr, 11 op/s 2021-03-15T22:33:18.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:33:18.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: T22:33:17.305177+0000 osd.6 (osd.6) 123 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:33:18.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: -03-15T22:33:17.502699+0000 2021-03-15T22:33:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: osd.4 (osd.4) 178 : cluster [DBG] 2021-03-15T22:33:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: 4.1a scrub ok 2021-03-15T22:33:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: cluster 2021-03-15T22:33:17. 2021-03-15T22:33:18.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: 509901+0000 mon.a (mon.0) 2166 : cluster 2021-03-15T22:33:18.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: [DBG] osdmap e847: 8 total, 8 up, 8 in 2021-03-15T22:33:18.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:33:18.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: :33:17.621532+0000 osd.3 (osd. 2021-03-15T22:33:18.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[62156]: 3) 154 : cluster [DBG] 5.17 deep-scrub ok 2021-03-15T22:33:18.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: cluster 2021-03-15T22:33:18.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: 2021-03- 2021-03-15T22:33:18.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: 15T22:33:17.187854+0000 mgr.y (mgr.14144) 976 : cluster 2021-03-15T22:33:18.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: [DBG] pgmap v1414: 148 pgs: 1 active+clean+wait, 147 active+clean; 549 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 886 KiB/s wr, 11 op/s 2021-03-15T22:33:18.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:33:18.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: -03-15T22:33:17.305177+0000 2021-03-15T22:33:18.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: osd.6 (osd.6) 123 : cluster [DBG] 2021-03-15T22:33:18.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: 5.11 scrub ok 2021-03-15T22:33:18.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: cluster 2021-03-15T22:33 2021-03-15T22:33:18.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: :17.502699+0000 osd.4 (osd.4) 2021-03-15T22:33:18.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: 178 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:33:18.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:33:18.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: T22:33:17.509901+0000 mon.a (mon 2021-03-15T22:33:18.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: .0) 2166 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2021-03-15T22:33:18.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: cluster 2021-03-15T22:33:18.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: 2021-03-15T22:33:17.621532 2021-03-15T22:33:18.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: +0000 osd.3 (osd.3) 154 : cluster 2021-03-15T22:33:18.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:18 smithi076 conmon[56514]: [DBG] 5.17 deep-scrub ok 2021-03-15T22:33:18.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:18 smithi125 conmon[56739]: cluster 2021-03-15T22:33:17.187854+0000 mgr.y (mgr.14144) 976 : cluster [DBG] pgmap v1414: 148 pgs: 1 active+clean+wait, 147 active+clean; 549 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 886 KiB/s wr, 11 op/s 2021-03-15T22:33:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:18 smithi125 conmon[56739]: 2021-03-15T22:33:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:18 smithi125 conmon[56739]: cluster 2021-03-15T22:33:17.305177+0000 osd.6 (osd.6) 123 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:33:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:18 smithi125 conmon[56739]: cluster 2021-03-15T22:33:17.502699+0000 osd.4 (osd.4) 178 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:33:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:18 smithi125 conmon[56739]: cluster 2021-03-15T22:33:17.509901+0000 mon.a (mon.0) 2166 : cluster [DBG] osdmap e847: 8 total, 8 up, 8 in 2021-03-15T22:33:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:18 smithi125 conmon[56739]: cluster 2021-03-15T22:33 2021-03-15T22:33:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:18 smithi125 conmon[56739]: :17.621532+0000 osd.3 (osd.3) 154 : cluster [DBG] 5.17 deep-scrub ok 2021-03-15T22:33:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:18 smithi125 conmon[56739]: 2021-03-15T22:33:18.846 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:18.869 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:18.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:18.871 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:18.972 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:18.994 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:18.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:18.995 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:19.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:19.122 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:19.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:19.124 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:19.225 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:19.237 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:19.251 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:19.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:19.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:19.256 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:33:19.354 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:19.375 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:19.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:19.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:19.478 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:19.501 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:19.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:19.502 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:19.535 INFO:tasks.rados.rados.0.smithi076.stdout:3191: done (2 left) 2021-03-15T22:33:19.535 INFO:tasks.rados.rados.0.smithi076.stdout:3192: rollback oid 31 current snap is 373 2021-03-15T22:33:19.536 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 31 to 359 2021-03-15T22:33:19.536 INFO:tasks.rados.rados.0.smithi076.stdout:3193: rollback oid 9 current snap is 373 2021-03-15T22:33:19.536 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 9 to 349 2021-03-15T22:33:19.536 INFO:tasks.rados.rados.0.smithi076.stdout:3194: snap_create 2021-03-15T22:33:19.542 INFO:tasks.rados.rados.0.smithi076.stdout:3192: finishing rollback tid 0 to smithi07694878-31 2021-03-15T22:33:19.543 INFO:tasks.rados.rados.0.smithi076.stdout:3192: finishing rollback tid 1 to smithi07694878-31 2021-03-15T22:33:19.543 INFO:tasks.rados.rados.0.smithi076.stdout:3193: finishing rollback tid 1 to smithi07694878-9 2021-03-15T22:33:19.543 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 1888 (ObjNum 264718944 snap 21896 seq_num 261778912) dirty dne 2021-03-15T22:33:19.543 INFO:tasks.rados.rados.0.smithi076.stdout:3192: finishing rollback tid 2 to smithi07694878-31 2021-03-15T22:33:19.544 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 2408 (ObjNum 1020 snap 341 seq_num 1020) dirty exists 2021-03-15T22:33:19.603 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:19.628 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:19.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:19.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:19.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:19.754 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:19.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:19.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:19.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: cluster 2021-03-15T22:33:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: 2021-03-15T22:33:17. 2021-03-15T22:33:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: 761129+0000 osd.5 (osd.5) 39 : cluster [DBG] 6.7 deep-scrub ok 2021-03-15T22:33:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:33:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: 15T22:33:18.461096+0000 osd.4 ( 2021-03-15T22:33:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: osd.4) 179 : cluster [DBG] 2021-03-15T22:33:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: 3.9 deep-scrub ok 2021-03-15T22:33:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:33:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: -03-15T22:33:18.507197 2021-03-15T22:33:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: +0000 osd.7 (osd.7) 2021-03-15T22:33:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: 159 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:33:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:33:19.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: -03-15T22:33:18. 2021-03-15T22:33:19.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: 514805+0000 mon.a (mon.0 2021-03-15T22:33:19.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:19 smithi125 conmon[56739]: ) 2167 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2021-03-15T22:33:19.857 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:19.880 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:19 smithi076 conmon[56514]: cluster 2021-03-15T22:33:19.880 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:19 smithi076 conmon[56514]: 2021-03-15T22:33:17.761129+0000 osd.5 (osd.5) 39 : cluster [DBG] 6.7 deep-scrub ok 2021-03-15T22:33:19.880 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:19 smithi076 conmon[56514]: cluster 2021-03-15T22:33:18.461096+0000 osd.4 (osd.4) 179 : cluster [DBG] 3.9 deep-scrub ok 2021-03-15T22:33:19.881 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:19 smithi076 conmon[56514]: cluster 2021-03-15T22:33:18.507197+0000 osd.7 (osd.7) 159 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:33:19.881 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:19 smithi076 conmon[56514]: cluster 2021-03-15T22:33:18.514805+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2021-03-15T22:33:19.881 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:19 smithi076 conmon[62156]: cluster 2021-03-15T22:33:17.761129+0000 osd.5 (osd.5) 39 : cluster [DBG] 6.7 deep-scrub ok 2021-03-15T22:33:19.882 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:19 smithi076 conmon[62156]: cluster 2021-03-15T22:33:18.461096+0000 osd.4 (osd.4) 179 : cluster [DBG] 3.9 deep-scrub ok 2021-03-15T22:33:19.882 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:19 smithi076 conmon[62156]: cluster 2021-03-15T22:33:18.507197+0000 osd.7 (osd.7) 159 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:33:19.882 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:19 smithi076 conmon[62156]: cluster 2021-03-15T22:33:18 2021-03-15T22:33:19.882 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:19 smithi076 conmon[62156]: .514805+0000 mon.a (mon.0) 2167 : cluster [DBG] osdmap e848: 8 total, 8 up, 8 in 2021-03-15T22:33:19.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:19.884 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:19.884 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:19.985 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:20.009 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:20.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:20.010 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:20.111 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:20.137 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:20.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:20.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:20.200 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:20.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:20.262 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:20.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:20.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:20.288 INFO:tasks.rados.rados.0.smithi076.stdout:3192: done (4 left) 2021-03-15T22:33:20.289 INFO:tasks.rados.rados.0.smithi076.stdout:3193: done (3 left) 2021-03-15T22:33:20.289 INFO:tasks.rados.rados.0.smithi076.stdout:3194: done (2 left) 2021-03-15T22:33:20.289 INFO:tasks.rados.rados.0.smithi076.stdout:3195: rollback oid 16 current snap is 374 2021-03-15T22:33:20.289 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 364 2021-03-15T22:33:20.290 INFO:tasks.rados.rados.0.smithi076.stdout:3196: delete oid 22 current snap is 374 2021-03-15T22:33:20.303 INFO:tasks.rados.rados.0.smithi076.stdout:3195: finishing rollback tid 0 to smithi07694878-16 2021-03-15T22:33:20.303 INFO:tasks.rados.rados.0.smithi076.stdout:3195: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:33:20.304 INFO:tasks.rados.rados.0.smithi076.stdout:3195: finishing rollback tid 2 to smithi07694878-16 2021-03-15T22:33:20.304 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 2760 (ObjNum 1051 snap 353 seq_num 1051) dirty exists 2021-03-15T22:33:20.304 INFO:tasks.rados.rados.0.smithi076.stdout:3195: done (3 left) 2021-03-15T22:33:20.304 INFO:tasks.rados.rados.0.smithi076.stdout:3196: done (2 left) 2021-03-15T22:33:20.304 INFO:tasks.rados.rados.0.smithi076.stdout:3197: read oid 50 snap -1 2021-03-15T22:33:20.305 INFO:tasks.rados.rados.0.smithi076.stdout:3197: expect (ObjNum 1004 snap 337 seq_num 1004) 2021-03-15T22:33:20.305 INFO:tasks.rados.rados.0.smithi076.stdout:3198: read oid 36 snap -1 2021-03-15T22:33:20.305 INFO:tasks.rados.rados.0.smithi076.stdout:3198: expect deleted 2021-03-15T22:33:20.305 INFO:tasks.rados.rados.0.smithi076.stdout:3199: delete oid 37 current snap is 374 2021-03-15T22:33:20.309 INFO:tasks.rados.rados.0.smithi076.stdout:3198: done (4 left) 2021-03-15T22:33:20.309 INFO:tasks.rados.rados.0.smithi076.stdout:3199: done (3 left) 2021-03-15T22:33:20.310 INFO:tasks.rados.rados.0.smithi076.stdout:3200: rollback oid 47 current snap is 374 2021-03-15T22:33:20.310 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 47 to 339 2021-03-15T22:33:20.310 INFO:tasks.rados.rados.0.smithi076.stdout:3201: snap_remove snap 372 2021-03-15T22:33:20.365 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:20.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:20.388 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:20.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:20.511 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:20.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:20.513 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:20.533 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:33:20.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:20.637 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:20.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:20.639 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:20.740 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:20.763 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:20.765 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:20.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[56739]: cluster 2021-03-15T22:33:19.188434+0000 mgr.y (mgr.14144) 977 : 2021-03-15T22:33:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[56739]: cluster [DBG] pgmap v1417: 148 pgs: 1 active+clean+wait, 147 active+clean; 549 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 0 B/s wr, 6 op/s 2021-03-15T22:33:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[56739]: cluster 2021-03-15T22:33:19.496744+0000 osd.7 (osd.7) 160 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:33:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[56739]: cluster 2021-03-15T22:33: 2021-03-15T22:33:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[56739]: 19.531875+0000 mon.a (mon.0) 2168 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2021-03-15T22:33:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:33:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[56739]: :33:19.560382+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:33:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:33:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[56739]: 03-15T22:33:20.285601+0000 2021-03-15T22:33:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[56739]: mon.a (mon.0) 2169 : cluster [DBG] 2021-03-15T22:33:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:20 smithi125 conmon[56739]: osdmap e850: 8 total, 8 up, 8 in 2021-03-15T22:33:20.837 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:20 smithi076 conmon[56514]: cluster 2021-03-15T22:33:19.188434+0000 mgr.y (mgr.14144) 977 : cluster [DBG] pgmap v1417: 148 pgs: 1 active+clean+wait, 147 active+clean; 549 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 0 B/s wr, 6 op/s 2021-03-15T22:33:20.838 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:20 smithi076 conmon[56514]: cluster 2021-03-15T22:33:19.496744+0000 osd.7 (osd.7) 160 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:33:20.838 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:20 smithi076 conmon[56514]: cluster 2021-03-15T22:33:19.531875+0000 mon.a (mon.0) 2168 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2021-03-15T22:33:20.838 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:20 smithi076 conmon[56514]: cluster 2021-03-15T22:33:19.560382+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:33:20.838 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:20 smithi076 conmon[56514]: cluster 2021-03-15T22:33:20.285601+0000 mon.a (mon.0) 2169 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2021-03-15T22:33:20.839 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:20 smithi076 conmon[62156]: cluster 2021-03-15T22:33:19.188434+0000 mgr.y (mgr.14144) 977 : cluster [DBG] pgmap v1417: 148 pgs: 1 active+clean+wait, 147 active+clean; 549 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.1 MiB/s rd, 0 B/s wr, 6 op/s 2021-03-15T22:33:20.839 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:20 smithi076 conmon[62156]: cluster 2021-03-15T22:33:19.496744+0000 osd.7 (osd.7) 160 : cluster [DBG] 6.15 scrub ok 2021-03-15T22:33:20.839 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:20 smithi076 conmon[62156]: cluster 2021-03-15T22:33:19.531875+0000 mon.a (mon.0) 2168 : cluster [DBG] osdmap e849: 8 total, 8 up, 8 in 2021-03-15T22:33:20.839 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:20 smithi076 conmon[62156]: cluster 2021-03-15T22:33:19.560382+0000 osd.0 (osd.0) 139 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:33:20.840 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:20 smithi076 conmon[62156]: cluster 2021-03-15T22:33:20.285601+0000 mon.a 2021-03-15T22:33:20.840 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:20 smithi076 conmon[62156]: (mon.0) 2169 : cluster [DBG] osdmap e850: 8 total, 8 up, 8 in 2021-03-15T22:33:20.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:20.887 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:20.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:20.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:20.897 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:33:20.991 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:21.016 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:21.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:21.017 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:21.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:21.146 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:21.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:21.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:21.249 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:21.273 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:21.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:21.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:21.298 INFO:tasks.rados.rados.0.smithi076.stdout:3201: done (4 left) 2021-03-15T22:33:21.298 INFO:tasks.rados.rados.0.smithi076.stdout:3202: read oid 40 snap 365 2021-03-15T22:33:21.298 INFO:tasks.rados.rados.0.smithi076.stdout:3202: expect deleted 2021-03-15T22:33:21.298 INFO:tasks.rados.rados.0.smithi076.stdout:3203: copy_from oid 7 from oid 9 current snap is 374 2021-03-15T22:33:21.299 INFO:tasks.rados.rados.0.smithi076.stdout:3204: read oid 8 snap 370 2021-03-15T22:33:21.299 INFO:tasks.rados.rados.0.smithi076.stdout:3204: expect deleted 2021-03-15T22:33:21.299 INFO:tasks.rados.rados.0.smithi076.stdout:3205: snap_create 2021-03-15T22:33:21.299 INFO:tasks.rados.rados.0.smithi076.stdout:3197: expect (ObjNum 1004 snap 337 seq_num 1004) 2021-03-15T22:33:21.301 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 7 pgp_num to 16 2021-03-15T22:33:21.341 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:21.375 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:21.400 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:21.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:21.401 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:21.434 INFO:tasks.rados.rados.0.smithi076.stdout:3200: finishing rollback tid 0 to smithi07694878-47 2021-03-15T22:33:21.434 INFO:tasks.rados.rados.0.smithi076.stdout:3200: finishing rollback tid 1 to smithi07694878-47 2021-03-15T22:33:21.434 INFO:tasks.rados.rados.0.smithi076.stdout:3200: finishing rollback tid 2 to smithi07694878-47 2021-03-15T22:33:21.434 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 1994 (ObjNum 1002 snap 335 seq_num 1002) dirty exists 2021-03-15T22:33:21.435 INFO:tasks.rados.rados.0.smithi076.stdout:3203: finishing copy_from to smithi07694878-7 2021-03-15T22:33:21.435 INFO:tasks.rados.rados.0.smithi076.stdout:3203: got expected ENOENT (src dne) 2021-03-15T22:33:21.435 INFO:tasks.rados.rados.0.smithi076.stdout:3203: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:33:21.435 INFO:tasks.rados.rados.0.smithi076.stdout:3136: finishing write tid 1 to smithi07694878-41 2021-03-15T22:33:21.436 INFO:tasks.rados.rados.0.smithi076.stdout:3136: finishing write tid 2 to smithi07694878-41 2021-03-15T22:33:21.436 INFO:tasks.rados.rados.0.smithi076.stdout:3136: finishing write tid 3 to smithi07694878-41 2021-03-15T22:33:21.436 INFO:tasks.rados.rados.0.smithi076.stdout:3136: finishing write tid 4 to smithi07694878-41 2021-03-15T22:33:21.437 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 2399 (ObjNum 1076 snap 366 seq_num 1076) dirty exists 2021-03-15T22:33:21.437 INFO:tasks.rados.rados.0.smithi076.stdout:3136: left oid 41 (ObjNum 1076 snap 366 seq_num 1076) 2021-03-15T22:33:21.437 INFO:tasks.rados.rados.0.smithi076.stdout:3145: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:21.503 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:21.526 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:21.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:21.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:21.654 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:21.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:21.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:21.756 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:21.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:33:21.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[62156]: 22:33:20.178851+0000 mgr.y (mgr.14144) 978 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:21.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[62156]: cluster 2021-03-15T22:33:20.272516+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f scrub ok 2021-03-15T22:33:21.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[62156]: cluster 2021-03-15T22:33:20.329853+0000 osd.1 (osd.1) 167 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:33:21.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[62156]: cluster 2021-03-15T22:33:20.500976+0000 osd.4 (osd.4) 180 : cluster [DBG] 6.e scrub ok 2021-03-15T22:33:21.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[62156]: cluster 2021-03-15T22:33:20.684972+0000 osd.3 (osd.3) 155 : cluster [DBG] 5.0 scrub ok 2021-03-15T22:33:21.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[62156]: audit 2021-03-15T22:33:20.900525+0000 mon.a (mon.0) 2170 : audit [INF] from='client.? 172.21.15.76:0/3707627451' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2021-03-15T22:33:21.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[62156]: audit 2021-03-15T22:33:21.293910+0000 mon.a (mon.0) 2171 : audit [INF] from='client.? 172.21.15.76:0/3707627451' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2021-03-15T22:33:21.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[62156]: cluster 2021-03-15T22:33:21.293970+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2021-03-15T22:33:21.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[56514]: audit 2021-03-15T22:33:20. 2021-03-15T22:33:21.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[56514]: 178851+0000 mgr.y (mgr.14144) 978 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:21.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[56514]: cluster 2021-03-15T22:33:20.272516+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f scrub ok 2021-03-15T22:33:21.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[56514]: cluster 2021-03-15T22:33:20.329853+0000 osd.1 (osd.1) 167 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:33:21.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[56514]: cluster 2021-03-15T22:33:20.500976+0000 osd.4 (osd.4) 180 : cluster [DBG] 6.e scrub ok 2021-03-15T22:33:21.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[56514]: cluster 2021-03-15T22:33:20.684972+0000 osd.3 (osd.3) 155 : cluster [DBG] 5.0 scrub ok 2021-03-15T22:33:21.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[56514]: audit 2021-03-15T22:33:20.900525+0000 mon.a (mon.0) 2170 : audit [INF] from='client.? 172.21.15.76:0/3707627451' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2021-03-15T22:33:21.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[56514]: audit 2021-03-15T22:33:21.293910+0000 mon.a (mon.0) 2171 : audit [INF] from='client.? 172.21.15.76:0/3707627451' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2021-03-15T22:33:21.772 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:21 smithi076 conmon[56514]: cluster 2021-03-15T22:33:21.293970+0000 mon.a (mon.0) 2172 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2021-03-15T22:33:21.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:21.780 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:21.780 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:21.881 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:21.908 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:21.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:21.909 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:22.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:22.036 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:22.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: audit 2021-03-15T22:33:20.178851+0000 mgr.y (mgr.14144) 978 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:22.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: cluster 2021-03-15T22:33:20.272516+0000 osd.6 (osd.6) 124 : cluster [DBG] 5.1f scrub ok 2021-03-15T22:33:22.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: cluster 2021-03-15T22:33:20.329853+0000 osd.1 (osd.1) 167 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:33:22.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: cluster 2021-03-15T22:33:20.500976+0000 osd.4 (osd.4) 180 : cluster 2021-03-15T22:33:22.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: [DBG] 6.e scrub ok 2021-03-15T22:33:22.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: cluster 2021-03-15T22:33:22.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: 2021-03-15T22:33:20.684972+0000 osd.3 (osd.3 2021-03-15T22:33:22.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: ) 155 : cluster [DBG] 5.0 scrub ok 2021-03-15T22:33:22.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: audit 2021-03-15T22:33:20.900525+0000 mon.a (mon.0) 2170 2021-03-15T22:33:22.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: : audit [INF] from='client.? 172.21.15.76:0/3707627451' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]: dispatch 2021-03-15T22:33:22.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: audit 2021-03-15T22:33:21.293910 2021-03-15T22:33:22.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2171 : audit [INF] from='client.? 172.21.15.76:0/3707627451' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num", "val": "16"}]': finished 2021-03-15T22:33:22.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:33:22.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: T22:33:21.293970+0000 mon.a (mon.0) 2021-03-15T22:33:22.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:21 smithi125 conmon[56739]: 2172 : cluster [DBG] osdmap e851: 8 total, 8 up, 8 in 2021-03-15T22:33:22.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:22.041 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:22.142 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:22.168 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:22.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:22.169 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:22.270 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:22.293 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:22.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:22.294 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:22.298 INFO:tasks.rados.rados.0.smithi076.stdout:3136: done (7 left) 2021-03-15T22:33:22.298 INFO:tasks.rados.rados.0.smithi076.stdout:3145: done (6 left) 2021-03-15T22:33:22.299 INFO:tasks.rados.rados.0.smithi076.stdout:3197: done (5 left) 2021-03-15T22:33:22.299 INFO:tasks.rados.rados.0.smithi076.stdout:3200: done (4 left) 2021-03-15T22:33:22.299 INFO:tasks.rados.rados.0.smithi076.stdout:3202: done (3 left) 2021-03-15T22:33:22.299 INFO:tasks.rados.rados.0.smithi076.stdout:3203: done (2 left) 2021-03-15T22:33:22.299 INFO:tasks.rados.rados.0.smithi076.stdout:3204: done (1 left) 2021-03-15T22:33:22.300 INFO:tasks.rados.rados.0.smithi076.stdout:3205: done (0 left) 2021-03-15T22:33:22.300 INFO:tasks.rados.rados.0.smithi076.stdout:3206: snap_remove snap 335 2021-03-15T22:33:22.300 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:33:22.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:22.421 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:22.423 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:22.524 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:22.547 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:22.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:22.549 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:22.650 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:22.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:22.674 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:22.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:22.776 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:22.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[56514]: cluster 2021-03-15T22:33:21.189011+0000 mgr.y 2021-03-15T22:33:22.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[56514]: (mgr.14144) 979 : cluster [DBG] pgmap v1420: 148 pgs: 1 active+clean+wait, 147 active+clean; 564 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.2 MiB/s wr, 17 op/s 2021-03-15T22:33:22.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[56514]: cluster 2021-03-15T22:33:21.545820+0000 osd.7 (osd.7) 161 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:33:22.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[56514]: cluster 2021-03-15T22:33:21.690305+0000 osd.3 (osd.3) 156 : cluster [DBG] 3.b deep-scrub ok 2021-03-15T22:33:22.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[56514]: audit 2021-03-15T22:33:21.733075+0000 mon.a (mon.0) 2173 : audit [INF] from='client.? 172.21.15.76:0/3960407584' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:33:22.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[56514]: cluster 2021-03-15T22:33:21.809859+0000 osd.2 (osd.2) 36 : cluster [DBG] 5.b scrub ok 2021-03-15T22:33:22.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[56514]: cluster 2021-03-15T22:33:22.289239+0000 mon.a (mon.0) 2174 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:22.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[56514]: audit 2021-03-15T22:33:22.294114+0000 mon.a (mon.0) 2175 : audit [INF] from='client.? 172.21.15.76:0/3960407584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:33:22.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[56514]: cluster 2021-03-15T22:33:22.294178+0000 mon.a (mon.0) 2176 : cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2021-03-15T22:33:22.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: cluster 2021-03-15T22:33 2021-03-15T22:33:22.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: :21.189011+0000 mgr.y (mgr.14144) 979 : cluster [DBG] pgmap v1420: 148 pgs: 1 active+clean+wait, 147 active+clean; 564 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.2 MiB/s wr, 17 op/s 2021-03-15T22:33:22.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: cluster 2021-03-15T22:33:22.800 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:33:22.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: :33:21.545820+0000 osd.7 (osd.7 2021-03-15T22:33:22.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: ) 161 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:33:22.801 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:22.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: -03-15T22:33:21.690305 2021-03-15T22:33:22.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: +0000 osd.3 (osd.3) 156 2021-03-15T22:33:22.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: : cluster [DBG] 3.b deep-scrub ok 2021-03-15T22:33:22.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:33:22.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: 03-15T22:33:21.733075 2021-03-15T22:33:22.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2173 2021-03-15T22:33:22.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/3960407584' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:33:22.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:33:22.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: -15T22:33:21.809859 2021-03-15T22:33:22.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: +0000 osd.2 (osd.2) 36 : 2021-03-15T22:33:22.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: cluster [DBG] 5.b scrub ok 2021-03-15T22:33:22.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:33:22.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: 03-15T22:33:22.289239+0000 2021-03-15T22:33:22.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: mon.a (mon.0) 2174 : cluster 2021-03-15T22:33:22.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:22.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:33:22.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: T22:33:22.294114+0000 mon.a (mon 2021-03-15T22:33:22.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: .0) 2175 : audit [INF] from='client.? 172.21.15.76:0/3960407584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:33:22.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: 2021-03-15T22:33:22.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: cluster 2021-03-15T22:33:22.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: 2021-03-15T22:33:22. 2021-03-15T22:33:22.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: 294178+0000 mon.a (mon.0) 2176 : 2021-03-15T22:33:22.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:22 smithi076 conmon[62156]: cluster [DBG] osdmap e852: 8 total, 8 up, 8 in 2021-03-15T22:33:22.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:22.810 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:22.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:22.912 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:22.938 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:22.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:22.940 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:23.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: cluster 2021-03-15T22:33:21.189011+0000 mgr.y (mgr.14144) 979 : cluster [DBG] pgmap v1420: 148 pgs: 1 active+clean+wait, 147 active+clean; 564 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.2 MiB/s wr, 17 op/s 2021-03-15T22:33:23.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: cluster 2021-03-15T22:33:21.545820+0000 osd.7 (osd.7) 161 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:33:23.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: cluster 2021-03-15T22:33:21.690305+0000 osd.3 ( 2021-03-15T22:33:23.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: osd.3) 156 : cluster [DBG] 3.b deep-scrub ok 2021-03-15T22:33:23.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: audit 2021 2021-03-15T22:33:23.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: -03-15T22:33:21.733075+0000 mon.a (mon.0) 2173 : audit [INF] from='client.? 172.21.15.76:0/3960407584' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:33:23.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: 2021-03-15T22:33:23.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:33:23.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: -15T22:33:21.809859+0000 osd.2 (osd.2) 36 : cluster [DBG] 2021-03-15T22:33:23.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: 5.b scrub ok 2021-03-15T22:33:23.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: cluster 2021-03-15T22:33:23.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: 2021-03-15T22:33:22.289239+0000 mon.a (mon.0) 2021-03-15T22:33:23.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: 2174 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:23.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: audit 2021-03-15T22:33:22.294114 2021-03-15T22:33:23.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2175 : audit [INF] from='client.? 172.21.15.76:0/3960407584' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:33:23.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:33:23.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: :33:22.294178+0000 mon.a (mon.0) 2176 : cluster [DBG] 2021-03-15T22:33:23.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:22 smithi125 conmon[56739]: osdmap e852: 8 total, 8 up, 8 in 2021-03-15T22:33:23.041 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:23.067 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:23.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:23.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:23.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:23.195 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:23.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:23.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:23.298 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:23.300 INFO:tasks.rados.rados.0.smithi076.stdout:3206: done (0 left) 2021-03-15T22:33:23.300 INFO:tasks.rados.rados.0.smithi076.stdout:3207: snap_remove snap 350 2021-03-15T22:33:23.326 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:23.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:23.328 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:23.429 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:23.453 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:23.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:23.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:23.556 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:23.579 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:23.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:23.580 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:23.682 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:23.704 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:23.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:23.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:23.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[56514]: cluster 2021-03-15T22:33:22.282582+0000 osd.6 (osd.6) 125 : cluster [DBG] 2021-03-15T22:33:23.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[56514]: 5.2 scrub ok 2021-03-15T22:33:23.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[56514]: cluster 2021-03-15T22:33:22.542500+0000 osd.7 (osd.7) 162 : cluster [DBG] 3.e scrub ok 2021-03-15T22:33:23.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[56514]: cluster 2021-03-15T22:33:22.598169+0000 osd.0 (osd.0) 140 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:33:23.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[56514]: cluster 2021-03-15T22:33:23.295054+0000 mon.a (mon.0) 2177 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2021-03-15T22:33:23.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: cluster 2021-03-15T22:33: 2021-03-15T22:33:23.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: 22.282582+0000 osd.6 (osd.6) 125 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:33:23.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: cluster 2021-03-15T22:33:23.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: 2021-03-15T22:33: 2021-03-15T22:33:23.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: 22.542500+0000 osd.7 (osd.7 2021-03-15T22:33:23.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: ) 162 : cluster [DBG] 3.e scrub ok 2021-03-15T22:33:23.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:33:23.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: T22:33:22.598169+0000 osd.0 ( 2021-03-15T22:33:23.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: osd.0) 140 : cluster [DBG] 2021-03-15T22:33:23.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: 3.6 scrub ok 2021-03-15T22:33:23.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:23.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: -03-15T22:33:23 2021-03-15T22:33:23.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: .295054+0000 mon.a (mon.0) 2021-03-15T22:33:23.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:23 smithi076 conmon[62156]: 2177 : cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2021-03-15T22:33:23.807 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:23.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:23.835 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:23.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:23.936 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:23.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:23.961 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:23.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:24.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:23 smithi125 conmon[56739]: cluster 2021-03-15T22:33:22.282582+0000 osd.6 (osd.6) 125 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:33:24.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:23 smithi125 conmon[56739]: cluster 2021-03-15T22:33:22.542500+0000 osd.7 (osd.7) 162 : cluster [DBG] 3.e scrub ok 2021-03-15T22:33:24.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:23 smithi125 conmon[56739]: cluster 2021-03-15T22:33:22.598169+0000 2021-03-15T22:33:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:23 smithi125 conmon[56739]: osd.0 (osd.0) 140 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:33:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:23 smithi125 conmon[56739]: cluster 2021-03-15T22:33:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:23 smithi125 conmon[56739]: 2021-03-15T22:33:23.295054+0000 mon.a (mon.0) 2177 : 2021-03-15T22:33:24.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:23 smithi125 conmon[56739]: cluster [DBG] osdmap e853: 8 total, 8 up, 8 in 2021-03-15T22:33:24.063 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:24.090 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:24.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:24.092 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:24.166 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:33:24.166 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:33:24.166 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:33:24.167 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:33:24.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:24.215 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:24.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:24.217 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:24.300 INFO:tasks.rados.rados.0.smithi076.stdout:3207: done (0 left) 2021-03-15T22:33:24.301 INFO:tasks.rados.rados.0.smithi076.stdout:3208: rollback oid 13 current snap is 375 2021-03-15T22:33:24.301 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 13 to 352 2021-03-15T22:33:24.301 INFO:tasks.rados.rados.0.smithi076.stdout:3209: copy_from oid 40 from oid 12 current snap is 375 2021-03-15T22:33:24.302 INFO:tasks.rados.rados.0.smithi076.stdout:3210: snap_remove snap 338 2021-03-15T22:33:24.318 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:24.342 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:24.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:24.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:24.444 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:24.466 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:24.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:24.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:24.568 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:24.594 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:24.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:24.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:24.697 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:24.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:24.722 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:24.722 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:24.823 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:24.847 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:24.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:24.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:24.861 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:24 smithi076 conmon[56514]: cluster 2021-03-15T22:33:23.189469+0000 mgr.y (mgr.14144) 2021-03-15T22:33:24.861 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:24 smithi076 conmon[56514]: 980 : cluster [DBG] pgmap v1423: 148 pgs: 1 active+clean+wait, 147 active+clean; 564 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.2 MiB/s wr, 17 op/s 2021-03-15T22:33:24.862 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:24 smithi076 conmon[56514]: cluster 2021-03-15T22:33:24.296119+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2021-03-15T22:33:24.862 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:24 smithi076 conmon[62156]: cluster 2021-03-15T22:33:23.189469+0000 2021-03-15T22:33:24.862 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:24 smithi076 conmon[62156]: mgr.y (mgr.14144) 980 : cluster [DBG] pgmap v1423: 148 pgs: 1 active+clean+wait, 147 active+clean; 564 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.2 MiB/s wr, 17 op/s 2021-03-15T22:33:24.863 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:24 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:24.863 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:24 smithi076 conmon[62156]: -03-15T22:33:24.296119+0000 mon.a (mon.0) 2178 2021-03-15T22:33:24.863 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:24 smithi076 conmon[62156]: : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2021-03-15T22:33:24.884 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:33:24.949 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:24.975 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:24.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:24.977 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:25.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:24 smithi125 conmon[56739]: cluster 2021-03-15T22:33:23.189469+0000 mgr.y (mgr 2021-03-15T22:33:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:24 smithi125 conmon[56739]: .14144) 980 : cluster [DBG] pgmap v1423: 148 pgs: 1 active+clean+wait, 147 active+clean; 564 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 12 MiB/s rd, 2.2 MiB/s wr, 17 op/s 2021-03-15T22:33:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:24 smithi125 conmon[56739]: cluster 2021-03-15T22:33:24.296119+0000 mon.a (mon.0) 2178 : cluster [DBG] osdmap e854: 8 total, 8 up, 8 in 2021-03-15T22:33:25.078 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:25.103 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:25.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:25.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:25.205 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:25.229 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:25.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:25.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:25.262 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:25.298 INFO:tasks.rados.rados.0.smithi076.stdout:3210: done (2 left) 2021-03-15T22:33:25.299 INFO:tasks.rados.rados.0.smithi076.stdout:3211: snap_remove snap 334 2021-03-15T22:33:25.316 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:33:25.331 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:25.334 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:33:25.353 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:33:25.355 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:25.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:25.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:25.371 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:33:25.389 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:33:25.407 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:33:25.426 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:33:25.444 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:33:25.445 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:33:25.446 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:33:25.446 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:33:25.446 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:33:25.446 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:33:25.447 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:33:25.447 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:33:25.447 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:33:25.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:25.482 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:25.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:25.484 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:25.585 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:25.608 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:25.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:25.609 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:25.710 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:25.732 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:25.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:25.734 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:25.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:25 smithi076 conmon[56514]: audit 2021-03-15T22:33:25.280796+0000 mon.a (mon.0) 2179 : audit [DBG] from='client.? 172.21.15.76:0/3004058908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:25.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:25 smithi076 conmon[56514]: cluster 2021-03-15T22:33:25.287849+0000 mon.a (mon.0) 2180 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2021-03-15T22:33:25.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:25 smithi076 conmon[62156]: audit 2021-03-15T22:33:25.280796+0000 mon.a (mon.0) 2179 : audit [DBG] from='client.? 172.21.15.76:0/3004058908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:25.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:25 smithi076 conmon[62156]: cluster 2021-03-15T22:33:25.287849+0000 mon.a (mon.0) 2180 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2021-03-15T22:33:25.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:25.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:25.861 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:25.862 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:25.963 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:25.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:25.986 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:25.987 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:26.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:25 smithi125 conmon[56739]: audit 2021-03-15T22:33:25.280796+0000 mon.a (mon.0) 2179 2021-03-15T22:33:26.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:25 smithi125 conmon[56739]: : audit [DBG] from='client.? 172.21.15.76:0/3004058908' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:26.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:25 smithi125 conmon[56739]: cluster 2021-03-15T22:33:25.287849+0000 mon.a (mon.0) 2180 : cluster [DBG] osdmap e855: 8 total, 8 up, 8 in 2021-03-15T22:33:26.042 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:33:26.088 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:26.110 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:26.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:26.112 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:26.213 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:26.238 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:26.240 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:26.304 INFO:tasks.rados.rados.0.smithi076.stdout:3211: done (2 left) 2021-03-15T22:33:26.304 INFO:tasks.rados.rados.0.smithi076.stdout:3212: snap_remove snap 309 2021-03-15T22:33:26.341 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:26.365 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:26.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:26.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:26.468 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:26.491 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:26.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:26.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:26.558 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:26 smithi076 conmon[62156]: cluster 2021-03-15T22:33:25.189926+0000 2021-03-15T22:33:26.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:26 smithi076 conmon[62156]: mgr.y (mgr.14144) 981 : cluster [DBG] pgmap v1426: 148 pgs: 1 active+clean+wait, 147 active+clean; 568 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 0 B/s wr, 5 op/s 2021-03-15T22:33:26.559 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:26 smithi076 conmon[62156]: cluster 2021-03-15T22:33:26.301998+0000 mon.a (mon.0) 2181 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2021-03-15T22:33:26.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:26.616 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:26.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:26.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:26.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:26.744 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:26.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:26.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:26.847 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:26.872 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:26.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:26.874 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:26.975 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:26.999 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:27.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:27.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:26 smithi076 conmon[56514]: cluster 2021-03-15T22:33:25.189926+0000 mgr.y (mgr.14144) 981 : cluster [DBG] pgmap v1426: 148 pgs: 1 active+clean+wait, 147 active+clean; 568 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 0 B/s wr, 5 op/s 2021-03-15T22:33:27.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:26 smithi076 conmon[56514]: cluster 2021-03-15T22:33:26.301998+0000 mon.a (mon.0) 2181 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2021-03-15T22:33:27.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:26 smithi125 conmon[56739]: cluster 2021-03-15T22:33:25.189926+0000 mgr.y (mgr.14144) 2021-03-15T22:33:27.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:26 smithi125 conmon[56739]: 981 : cluster [DBG] pgmap v1426: 148 pgs: 1 active+clean+wait, 147 active+clean; 568 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.1 MiB/s rd, 0 B/s wr, 5 op/s 2021-03-15T22:33:27.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:26 smithi125 conmon[56739]: cluster 2021-03-15T22:33:26.301998+0000 mon.a (mon.0) 2181 : cluster [DBG] osdmap e856: 8 total, 8 up, 8 in 2021-03-15T22:33:27.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:26 smithi125 conmon[56739]: 2021-03-15T22:33:27.066 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:27.102 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:27.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:27.129 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:27.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:27.230 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:27.252 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:27.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:27.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:27.300 INFO:tasks.rados.rados.0.smithi076.stdout:3212: done (2 left) 2021-03-15T22:33:27.301 INFO:tasks.rados.rados.0.smithi076.stdout:3213: write oid 50 current snap is 375 2021-03-15T22:33:27.301 INFO:tasks.rados.rados.0.smithi076.stdout:3213: seq_num 1090 ranges {719630=659889,2117608=202022} 2021-03-15T22:33:27.315 INFO:tasks.rados.rados.0.smithi076.stdout:3213: writing smithi07694878-50 from 719630 to 1379519 tid 1 2021-03-15T22:33:27.321 INFO:tasks.rados.rados.0.smithi076.stdout:3213: writing smithi07694878-50 from 2117608 to 2319630 tid 2 2021-03-15T22:33:27.322 INFO:tasks.rados.rados.0.smithi076.stdout:3208: finishing rollback tid 0 to smithi07694878-13 2021-03-15T22:33:27.322 INFO:tasks.rados.rados.0.smithi076.stdout:3208: finishing rollback tid 1 to smithi07694878-13 2021-03-15T22:33:27.322 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1997 (ObjNum 1 snap 21896 seq_num 264924768) dirty dne 2021-03-15T22:33:27.322 INFO:tasks.rados.rados.0.smithi076.stdout:3209: finishing copy_from to smithi07694878-40 2021-03-15T22:33:27.323 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 2302 (ObjNum 999 snap 335 seq_num 999) dirty exists 2021-03-15T22:33:27.323 INFO:tasks.rados.rados.0.smithi076.stdout:3209: finishing copy_from racing read to smithi07694878-40 2021-03-15T22:33:27.323 INFO:tasks.rados.rados.0.smithi076.stdout:3208: done (2 left) 2021-03-15T22:33:27.323 INFO:tasks.rados.rados.0.smithi076.stdout:3209: done (1 left) 2021-03-15T22:33:27.323 INFO:tasks.rados.rados.0.smithi076.stdout:3214: write (excl) oid 27 current snap is 375 2021-03-15T22:33:27.324 INFO:tasks.rados.rados.0.smithi076.stdout:3214: seq_num 1091 ranges {736437=735138,2176090=160347} 2021-03-15T22:33:27.329 INFO:tasks.rados.rados.0.smithi076.stdout:3214: writing smithi07694878-27 from 736437 to 1471575 tid 1 2021-03-15T22:33:27.335 INFO:tasks.rados.rados.0.smithi076.stdout:3214: writing smithi07694878-27 from 2176090 to 2336437 tid 2 2021-03-15T22:33:27.335 INFO:tasks.rados.rados.0.smithi076.stdout:3215: read oid 14 snap -1 2021-03-15T22:33:27.336 INFO:tasks.rados.rados.0.smithi076.stdout:3215: expect (ObjNum 1065 snap 362 seq_num 1065) 2021-03-15T22:33:27.336 INFO:tasks.rados.rados.0.smithi076.stdout:3216: write oid 9 current snap is 375 2021-03-15T22:33:27.336 INFO:tasks.rados.rados.0.smithi076.stdout:3213: finishing write tid 1 to smithi07694878-50 2021-03-15T22:33:27.336 INFO:tasks.rados.rados.0.smithi076.stdout:3216: seq_num 1092 ranges {753244=410387,1834572=518672} 2021-03-15T22:33:27.341 INFO:tasks.rados.rados.0.smithi076.stdout:3216: writing smithi07694878-9 from 753244 to 1163631 tid 1 2021-03-15T22:33:27.348 INFO:tasks.rados.rados.0.smithi076.stdout:3216: writing smithi07694878-9 from 1834572 to 2353244 tid 2 2021-03-15T22:33:27.348 INFO:tasks.rados.rados.0.smithi076.stdout:3217: write (excl) oid 35 current snap is 375 2021-03-15T22:33:27.349 INFO:tasks.rados.rados.0.smithi076.stdout:3213: finishing write tid 2 to smithi07694878-50 2021-03-15T22:33:27.349 INFO:tasks.rados.rados.0.smithi076.stdout:3213: finishing write tid 3 to smithi07694878-50 2021-03-15T22:33:27.349 INFO:tasks.rados.rados.0.smithi076.stdout:3213: finishing write tid 4 to smithi07694878-50 2021-03-15T22:33:27.349 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 2654 (ObjNum 1090 snap 375 seq_num 1090) dirty exists 2021-03-15T22:33:27.349 INFO:tasks.rados.rados.0.smithi076.stdout:3213: left oid 50 (ObjNum 1090 snap 375 seq_num 1090) 2021-03-15T22:33:27.350 INFO:tasks.rados.rados.0.smithi076.stdout:3214: finishing write tid 1 to smithi07694878-27 2021-03-15T22:33:27.350 INFO:tasks.rados.rados.0.smithi076.stdout:3217: seq_num 1093 ranges {770051=485636,1776701=511047} 2021-03-15T22:33:27.354 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:27.355 INFO:tasks.rados.rados.0.smithi076.stdout:3217: writing smithi07694878-35 from 770051 to 1255687 tid 1 2021-03-15T22:33:27.361 INFO:tasks.rados.rados.0.smithi076.stdout:3217: writing smithi07694878-35 from 1776701 to 2287748 tid 2 2021-03-15T22:33:27.361 INFO:tasks.rados.rados.0.smithi076.stdout:3213: done (4 left) 2021-03-15T22:33:27.361 INFO:tasks.rados.rados.0.smithi076.stdout:3218: snap_create 2021-03-15T22:33:27.362 INFO:tasks.rados.rados.0.smithi076.stdout:3214: finishing write tid 2 to smithi07694878-27 2021-03-15T22:33:27.362 INFO:tasks.rados.rados.0.smithi076.stdout:3214: finishing write tid 3 to smithi07694878-27 2021-03-15T22:33:27.362 INFO:tasks.rados.rados.0.smithi076.stdout:3214: finishing write tid 4 to smithi07694878-27 2021-03-15T22:33:27.362 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 1997 (ObjNum 1091 snap 375 seq_num 1091) dirty exists 2021-03-15T22:33:27.363 INFO:tasks.rados.rados.0.smithi076.stdout:3214: left oid 27 (ObjNum 1091 snap 375 seq_num 1091) 2021-03-15T22:33:27.363 INFO:tasks.rados.rados.0.smithi076.stdout:3216: finishing write tid 1 to smithi07694878-9 2021-03-15T22:33:27.363 INFO:tasks.rados.rados.0.smithi076.stdout:3215: expect (ObjNum 1065 snap 362 seq_num 1065) 2021-03-15T22:33:27.379 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:27.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:27.380 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:27.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:27.503 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:27.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:27.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:27.543 INFO:tasks.rados.rados.0.smithi076.stdout:3216: finishing write tid 2 to smithi07694878-9 2021-03-15T22:33:27.543 INFO:tasks.rados.rados.0.smithi076.stdout:3216: finishing write tid 3 to smithi07694878-9 2021-03-15T22:33:27.543 INFO:tasks.rados.rados.0.smithi076.stdout:3216: finishing write tid 4 to smithi07694878-9 2021-03-15T22:33:27.543 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 2764 (ObjNum 1092 snap 375 seq_num 1092) dirty exists 2021-03-15T22:33:27.544 INFO:tasks.rados.rados.0.smithi076.stdout:3216: left oid 9 (ObjNum 1092 snap 375 seq_num 1092) 2021-03-15T22:33:27.544 INFO:tasks.rados.rados.0.smithi076.stdout:3217: finishing write tid 1 to smithi07694878-35 2021-03-15T22:33:27.544 INFO:tasks.rados.rados.0.smithi076.stdout:3217: finishing write tid 2 to smithi07694878-35 2021-03-15T22:33:27.544 INFO:tasks.rados.rados.0.smithi076.stdout:3217: finishing write tid 3 to smithi07694878-35 2021-03-15T22:33:27.544 INFO:tasks.rados.rados.0.smithi076.stdout:3217: finishing write tid 4 to smithi07694878-35 2021-03-15T22:33:27.545 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 2136 (ObjNum 1093 snap 375 seq_num 1093) dirty exists 2021-03-15T22:33:27.545 INFO:tasks.rados.rados.0.smithi076.stdout:3217: left oid 35 (ObjNum 1093 snap 375 seq_num 1093) 2021-03-15T22:33:27.595 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[56514]: cluster 2021-03-15T22:33:25.814408+0000 2021-03-15T22:33:27.596 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[56514]: osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:27.596 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[56514]: cluster 2021-03-15T22:33:25.816010+0000 osd.5 (osd.5) 41 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:27.596 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[56514]: cluster 2021-03-15T22:33:27.298308+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2021-03-15T22:33:27.596 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[62156]: cluster 2021-03-15T22:33: 2021-03-15T22:33:27.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[62156]: 25.814408+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:27.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[62156]: cluster 2021-03-15T22:33:25.816010 2021-03-15T22:33:27.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 41 : cluster [DBG] 2021-03-15T22:33:27.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[62156]: purged_snaps scrub ok 2021-03-15T22:33:27.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:33:27.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[62156]: T22:33:27.298308+0000 mon.a ( 2021-03-15T22:33:27.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:27 smithi076 conmon[62156]: mon.0) 2182 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2021-03-15T22:33:27.606 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:27.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:27.629 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:27.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:27.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:27.753 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:27.754 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:27.855 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:27.879 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:27.879 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:27.890 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:33:27.891 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:33:27.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:28.006 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:28.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:28.008 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:28.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:27 smithi125 conmon[56739]: cluster 2021-03-15T22:33:25.814408+0000 osd.5 (osd.5) 40 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:27 smithi125 conmon[56739]: cluster 2021-03-15T22:33:25.816010+0000 osd.5 (osd.5) 41 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:27 smithi125 conmon[56739]: cluster 2021-03-15T22:33:27.298308+0000 mon.a (mon.0) 2182 : cluster [DBG] osdmap e857: 8 total, 8 up, 8 in 2021-03-15T22:33:28.109 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:28.135 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:28.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:28.237 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:28.260 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:28.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:28.261 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:28.289 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 7, 0, 3, 5, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:33:28.289 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:33:28.289 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.533643 2021-03-15T22:33:28.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd primary-affinity 2 0.5336428333779778 2021-03-15T22:33:28.304 INFO:tasks.rados.rados.0.smithi076.stdout:3214: done (4 left) 2021-03-15T22:33:28.305 INFO:tasks.rados.rados.0.smithi076.stdout:3215: done (3 left) 2021-03-15T22:33:28.305 INFO:tasks.rados.rados.0.smithi076.stdout:3216: done (2 left) 2021-03-15T22:33:28.305 INFO:tasks.rados.rados.0.smithi076.stdout:3217: done (1 left) 2021-03-15T22:33:28.305 INFO:tasks.rados.rados.0.smithi076.stdout:3218: done (0 left) 2021-03-15T22:33:28.306 INFO:tasks.rados.rados.0.smithi076.stdout:3219: read oid 1 snap -1 2021-03-15T22:33:28.306 INFO:tasks.rados.rados.0.smithi076.stdout:3219: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:28.306 INFO:tasks.rados.rados.0.smithi076.stdout:3220: read oid 40 snap -1 2021-03-15T22:33:28.306 INFO:tasks.rados.rados.0.smithi076.stdout:3220: expect (ObjNum 999 snap 335 seq_num 999) 2021-03-15T22:33:28.307 INFO:tasks.rados.rados.0.smithi076.stdout:3221: copy_from oid 18 from oid 47 current snap is 376 2021-03-15T22:33:28.307 INFO:tasks.rados.rados.0.smithi076.stdout:3222: read oid 27 snap -1 2021-03-15T22:33:28.307 INFO:tasks.rados.rados.0.smithi076.stdout:3222: expect (ObjNum 1091 snap 375 seq_num 1091) 2021-03-15T22:33:28.307 INFO:tasks.rados.rados.0.smithi076.stdout:3223: copy_from oid 3 from oid 25 current snap is 376 2021-03-15T22:33:28.308 INFO:tasks.rados.rados.0.smithi076.stdout:3224: write (excl) oid 5 current snap is 376 2021-03-15T22:33:28.308 INFO:tasks.rados.rados.0.smithi076.stdout:3224: seq_num 1094 ranges {786858=560885,1835183=551675} 2021-03-15T22:33:28.312 INFO:tasks.rados.rados.0.smithi076.stdout:3224: writing smithi07694878-5 from 786858 to 1347743 tid 1 2021-03-15T22:33:28.319 INFO:tasks.rados.rados.0.smithi076.stdout:3224: writing smithi07694878-5 from 1835183 to 2386858 tid 2 2021-03-15T22:33:28.319 INFO:tasks.rados.rados.0.smithi076.stdout:3222: expect (ObjNum 1091 snap 375 seq_num 1091) 2021-03-15T22:33:28.342 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:33:28.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:28.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:28.387 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:28.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:28.429 INFO:tasks.rados.rados.0.smithi076.stdout:3219: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:28.489 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:28.511 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:28.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:28.512 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:28.513 INFO:tasks.rados.rados.0.smithi076.stdout:3224: finishing write tid 1 to smithi07694878-5 2021-03-15T22:33:28.513 INFO:tasks.rados.rados.0.smithi076.stdout:3220: expect (ObjNum 999 snap 335 seq_num 999) 2021-03-15T22:33:28.613 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:28.615 INFO:tasks.rados.rados.0.smithi076.stdout:3224: finishing write tid 2 to smithi07694878-5 2021-03-15T22:33:28.615 INFO:tasks.rados.rados.0.smithi076.stdout:3224: finishing write tid 3 to smithi07694878-5 2021-03-15T22:33:28.615 INFO:tasks.rados.rados.0.smithi076.stdout:3224: finishing write tid 4 to smithi07694878-5 2021-03-15T22:33:28.615 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1983 (ObjNum 1094 snap 376 seq_num 1094) dirty exists 2021-03-15T22:33:28.616 INFO:tasks.rados.rados.0.smithi076.stdout:3224: left oid 5 (ObjNum 1094 snap 376 seq_num 1094) 2021-03-15T22:33:28.616 INFO:tasks.rados.rados.0.smithi076.stdout:3221: finishing copy_from to smithi07694878-18 2021-03-15T22:33:28.616 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 1994 (ObjNum 1002 snap 335 seq_num 1002) dirty exists 2021-03-15T22:33:28.617 INFO:tasks.rados.rados.0.smithi076.stdout:3221: finishing copy_from racing read to smithi07694878-18 2021-03-15T22:33:28.617 INFO:tasks.rados.rados.0.smithi076.stdout:3223: finishing copy_from to smithi07694878-3 2021-03-15T22:33:28.617 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 1991 (ObjNum 1082 snap 371 seq_num 1082) dirty exists 2021-03-15T22:33:28.617 INFO:tasks.rados.rados.0.smithi076.stdout:3223: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:33:28.618 INFO:tasks.rados.rados.0.smithi076.stdout:3219: done (5 left) 2021-03-15T22:33:28.618 INFO:tasks.rados.rados.0.smithi076.stdout:3220: done (4 left) 2021-03-15T22:33:28.618 INFO:tasks.rados.rados.0.smithi076.stdout:3221: done (3 left) 2021-03-15T22:33:28.618 INFO:tasks.rados.rados.0.smithi076.stdout:3222: done (2 left) 2021-03-15T22:33:28.618 INFO:tasks.rados.rados.0.smithi076.stdout:3223: done (1 left) 2021-03-15T22:33:28.619 INFO:tasks.rados.rados.0.smithi076.stdout:3224: done (0 left) 2021-03-15T22:33:28.619 INFO:tasks.rados.rados.0.smithi076.stdout:3225: write (excl) oid 38 current snap is 376 2021-03-15T22:33:28.619 INFO:tasks.rados.rados.0.smithi076.stdout:3225: seq_num 1095 ranges {403665=752487,1610018=514716} 2021-03-15T22:33:28.621 INFO:tasks.rados.rados.0.smithi076.stdout:3225: writing smithi07694878-38 from 403665 to 1156152 tid 1 2021-03-15T22:33:28.626 INFO:tasks.rados.rados.0.smithi076.stdout:3225: writing smithi07694878-38 from 1610018 to 2124734 tid 2 2021-03-15T22:33:28.626 INFO:tasks.rados.rados.0.smithi076.stdout:3226: copy_from oid 48 from oid 39 current snap is 376 2021-03-15T22:33:28.626 INFO:tasks.rados.rados.0.smithi076.stdout:3225: finishing write tid 1 to smithi07694878-38 2021-03-15T22:33:28.627 INFO:tasks.rados.rados.0.smithi076.stdout:3227: read oid 10 snap -1 2021-03-15T22:33:28.627 INFO:tasks.rados.rados.0.smithi076.stdout:3227: expect (ObjNum 1077 snap 366 seq_num 1077) 2021-03-15T22:33:28.628 INFO:tasks.rados.rados.0.smithi076.stdout:3228: snap_create 2021-03-15T22:33:28.630 INFO:tasks.rados.rados.0.smithi076.stdout:3225: finishing write tid 2 to smithi07694878-38 2021-03-15T22:33:28.630 INFO:tasks.rados.rados.0.smithi076.stdout:3227: expect (ObjNum 1077 snap 366 seq_num 1077) 2021-03-15T22:33:28.643 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:28.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:28.644 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:28.699 INFO:tasks.rados.rados.0.smithi076.stdout:3225: finishing write tid 3 to smithi07694878-38 2021-03-15T22:33:28.699 INFO:tasks.rados.rados.0.smithi076.stdout:3225: finishing write tid 4 to smithi07694878-38 2021-03-15T22:33:28.699 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2139 (ObjNum 1095 snap 376 seq_num 1095) dirty exists 2021-03-15T22:33:28.700 INFO:tasks.rados.rados.0.smithi076.stdout:3225: left oid 38 (ObjNum 1095 snap 376 seq_num 1095) 2021-03-15T22:33:28.700 INFO:tasks.rados.rados.0.smithi076.stdout:3226: finishing copy_from to smithi07694878-48 2021-03-15T22:33:28.700 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 2092 (ObjNum 829 snap 275 seq_num 829) dirty exists 2021-03-15T22:33:28.700 INFO:tasks.rados.rados.0.smithi076.stdout:3226: finishing copy_from racing read to smithi07694878-48 2021-03-15T22:33:28.745 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:28.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:28 smithi076 conmon[56514]: cluster 2021-03-15T22:33:27.190513+0000 mgr.y (mgr.14144) 2021-03-15T22:33:28.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:28 smithi076 conmon[56514]: 982 : cluster [DBG] pgmap v1429: 148 pgs: 148 active+clean; 570 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 275 KiB/s wr, 6 op/s 2021-03-15T22:33:28.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:28 smithi076 conmon[56514]: cluster 2021-03-15T22:33:28.302011+0000 mon.a (mon.0) 2183 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2021-03-15T22:33:28.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:28 smithi076 conmon[62156]: cluster 2021-03-15T22:33:27.190513+0000 mgr.y (mgr.14144) 2021-03-15T22:33:28.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:28 smithi076 conmon[62156]: 982 : cluster [DBG] pgmap v1429: 148 pgs: 148 active+clean; 570 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 275 KiB/s wr, 6 op/s 2021-03-15T22:33:28.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:28 smithi076 conmon[62156]: cluster 2021-03-15T22:33:28.302011+0000 mon.a (mon.0) 2183 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2021-03-15T22:33:28.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:28.770 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:28.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:28.872 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:28.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:28.896 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:28.896 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:28.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:33:28.953 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:28.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:29.020 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:29.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:29.022 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:29.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:28 smithi125 conmon[56739]: cluster 2021-03-15T22:33:27.190513+0000 mgr.y (mgr. 2021-03-15T22:33:29.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:28 smithi125 conmon[56739]: 14144) 982 : cluster [DBG] pgmap v1429: 148 pgs: 148 active+clean; 570 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 3.0 MiB/s rd, 275 KiB/s wr, 6 op/s 2021-03-15T22:33:29.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:28 smithi125 conmon[56739]: cluster 2021-03-15T22:33:28.302011+0000 mon.a 2021-03-15T22:33:29.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:28 smithi125 conmon[56739]: (mon.0) 2183 : cluster [DBG] osdmap e858: 8 total, 8 up, 8 in 2021-03-15T22:33:29.045 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:29.123 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:29.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:29.148 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:29.148 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:29.249 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:29.272 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:29.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:29.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:29.374 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:29.397 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:29.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:29.399 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:29.500 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:29.527 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:29.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:29.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:29.581 INFO:tasks.rados.rados.0.smithi076.stdout:3225: done (3 left) 2021-03-15T22:33:29.582 INFO:tasks.rados.rados.0.smithi076.stdout:3226: done (2 left) 2021-03-15T22:33:29.582 INFO:tasks.rados.rados.0.smithi076.stdout:3227: done (1 left) 2021-03-15T22:33:29.582 INFO:tasks.rados.rados.0.smithi076.stdout:3228: done (0 left) 2021-03-15T22:33:29.582 INFO:tasks.rados.rados.0.smithi076.stdout:3229: snap_create 2021-03-15T22:33:29.583 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:33:29.629 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:29.655 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:29.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:29.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:29.668 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:29 smithi076 conmon[56514]: audit 2021-03-15T22:33:29.535021+0000 mon.a (mon.0) 2184 : audit [INF] from='client.? 172.21.15.76:0/3289555083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:29.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:29 smithi076 conmon[62156]: audit 2021-03-15T22:33:29.535021+0000 mon.a (mon.0) 2184 : audit 2021-03-15T22:33:29.669 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:29 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/3289555083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:29.758 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:29.783 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:29.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:29.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:29.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:29.911 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:29.912 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:30.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:30.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:29 smithi125 conmon[56739]: audit 2021-03-15T22:33:30.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:29 smithi125 conmon[56739]: 2021-03-15T22:33:29.535021+0000 2021-03-15T22:33:30.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:29 smithi125 conmon[56739]: mon.a (mon.0) 2184 : audit [INF] from='client.? 172.21.15.76:0/3289555083' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:30.038 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:30.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:30.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:30.141 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:30.164 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:30.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:30.166 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:30.267 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:30.294 INFO:tasks.rados.rados.0.smithi076.stdout:3229: done (0 left) 2021-03-15T22:33:30.294 INFO:tasks.rados.rados.0.smithi076.stdout:3230: write oid 32 current snap is 378 2021-03-15T22:33:30.295 INFO:tasks.rados.rados.0.smithi076.stdout:3230: seq_num 1096 ranges {420472=427736,1268500=658374,2347953=72519} 2021-03-15T22:33:30.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:30.296 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:30.296 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:30.299 INFO:tasks.rados.rados.0.smithi076.stdout:3230: writing smithi07694878-32 from 420472 to 848208 tid 1 2021-03-15T22:33:30.305 INFO:tasks.rados.rados.0.smithi076.stdout:3230: writing smithi07694878-32 from 1268500 to 1926874 tid 2 2021-03-15T22:33:30.308 INFO:tasks.rados.rados.0.smithi076.stdout:3230: writing smithi07694878-32 from 2347953 to 2420472 tid 3 2021-03-15T22:33:30.308 INFO:tasks.rados.rados.0.smithi076.stdout:3231: read oid 28 snap -1 2021-03-15T22:33:30.309 INFO:tasks.rados.rados.0.smithi076.stdout:3231: expect (ObjNum 1070 snap 364 seq_num 1070) 2021-03-15T22:33:30.309 INFO:tasks.rados.rados.0.smithi076.stdout:3232: snap_create 2021-03-15T22:33:30.327 INFO:tasks.rados.rados.0.smithi076.stdout:3231: expect (ObjNum 1070 snap 364 seq_num 1070) 2021-03-15T22:33:30.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:30.423 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:30.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:30.425 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:30.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:30.527 INFO:tasks.rados.rados.0.smithi076.stdout:3230: finishing write tid 1 to smithi07694878-32 2021-03-15T22:33:30.527 INFO:tasks.rados.rados.0.smithi076.stdout:3230: finishing write tid 2 to smithi07694878-32 2021-03-15T22:33:30.527 INFO:tasks.rados.rados.0.smithi076.stdout:3230: finishing write tid 3 to smithi07694878-32 2021-03-15T22:33:30.528 INFO:tasks.rados.rados.0.smithi076.stdout:3230: finishing write tid 4 to smithi07694878-32 2021-03-15T22:33:30.528 INFO:tasks.rados.rados.0.smithi076.stdout:3230: finishing write tid 5 to smithi07694878-32 2021-03-15T22:33:30.528 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 2849 (ObjNum 1096 snap 378 seq_num 1096) dirty exists 2021-03-15T22:33:30.528 INFO:tasks.rados.rados.0.smithi076.stdout:3230: left oid 32 (ObjNum 1096 snap 378 seq_num 1096) 2021-03-15T22:33:30.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:33:30.551 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:30.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:30.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:30.654 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:30.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:30.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:30.780 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:30.802 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:30.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:30.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:30.905 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:30.931 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:30.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:30.932 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:31.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[56514]: cluster 2021-03-15T22:33:29.190891+0000 mgr.y (mgr.14144 2021-03-15T22:33:31.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[56514]: ) 983 : cluster [DBG] pgmap v1432: 148 pgs: 148 active+clean; 571 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 486 KiB/s wr, 7 op/s 2021-03-15T22:33:31.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[56514]: cluster 2021-03-15T22:33:29.564476+0000 mon.a (mon.0) 2185 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:31.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[56514]: audit 2021-03-15T22:33:29.576160+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.76:0/3289555083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:33:31.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[56514]: cluster 2021-03-15T22:33:29.576224+0000 mon.a (mon.0) 2187 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2021-03-15T22:33:31.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[56514]: cluster 2021-03-15T22:33:30.292349+0000 mon.a (mon.0) 2188 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2021-03-15T22:33:31.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[62156]: cluster 2021-03-15T22:33:29.190891+0000 mgr.y 2021-03-15T22:33:31.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[62156]: (mgr.14144) 983 : cluster [DBG] pgmap v1432: 148 pgs: 148 active+clean; 571 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 486 KiB/s wr, 7 op/s 2021-03-15T22:33:31.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[62156]: cluster 2021-03-15T22:33:29.564476+0000 mon.a (mon.0) 2185 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:31.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[62156]: audit 2021-03-15T22:33:29.576160+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.76:0/3289555083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:33:31.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[62156]: cluster 2021-03-15T22:33:29.576224+0000 mon.a (mon.0) 2187 : cluster [DBG] osdmap e859: 8 total, 8 up, 8 in 2021-03-15T22:33:31.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:30 smithi076 conmon[62156]: cluster 2021-03-15T22:33:30.292349+0000 mon.a (mon.0) 2188 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2021-03-15T22:33:31.033 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:31.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[56739]: cluster 2021-03-15T22:33:29.190891+0000 mgr.y (mgr.14144) 983 : cluster [DBG] pgmap v1432: 148 pgs: 148 active+clean; 571 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.6 MiB/s rd, 486 KiB/s wr, 7 op/s 2021-03-15T22:33:31.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[56739]: cluster 2021-03-15T22:33:29.564476+0000 mon.a (mon.0) 2185 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:31.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:33:31.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[56739]: 33:29.576160+0000 mon.a (mon.0) 2186 : audit [INF] from='client.? 172.21.15.76:0/3289555083' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:33:31.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:33:31.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[56739]: -03-15T22:33:29.576224+0000 2021-03-15T22:33:31.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[56739]: mon.a (mon.0) 2187 : cluster [DBG] 2021-03-15T22:33:31.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[56739]: osdmap e859: 8 total, 8 up, 8 in 2021-03-15T22:33:31.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:33:31.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[56739]: 33:30.292349+0000 mon.a (mon.0 2021-03-15T22:33:31.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:30 smithi125 conmon[56739]: ) 2188 : cluster [DBG] osdmap e860: 8 total, 8 up, 8 in 2021-03-15T22:33:31.055 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:31.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:31.057 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:31.158 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:31.180 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:31.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:31.182 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:31.283 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:31.300 INFO:tasks.rados.rados.0.smithi076.stdout:3230: done (2 left) 2021-03-15T22:33:31.301 INFO:tasks.rados.rados.0.smithi076.stdout:3231: done (1 left) 2021-03-15T22:33:31.301 INFO:tasks.rados.rados.0.smithi076.stdout:3232: done (0 left) 2021-03-15T22:33:31.301 INFO:tasks.rados.rados.0.smithi076.stdout:3233: copy_from oid 44 from oid 48 current snap is 379 2021-03-15T22:33:31.302 INFO:tasks.rados.rados.0.smithi076.stdout:3234: copy_from oid 19 from oid 13 current snap is 379 2021-03-15T22:33:31.302 INFO:tasks.rados.rados.0.smithi076.stdout:3235: read oid 32 snap -1 2021-03-15T22:33:31.302 INFO:tasks.rados.rados.0.smithi076.stdout:3235: expect (ObjNum 1096 snap 378 seq_num 1096) 2021-03-15T22:33:31.303 INFO:tasks.rados.rados.0.smithi076.stdout:3236: snap_create 2021-03-15T22:33:31.304 INFO:tasks.rados.rados.0.smithi076.stdout:3234: finishing copy_from to smithi07694878-19 2021-03-15T22:33:31.305 INFO:tasks.rados.rados.0.smithi076.stdout:3234: got expected ENOENT (src dne) 2021-03-15T22:33:31.305 INFO:tasks.rados.rados.0.smithi076.stdout:3234: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:33:31.307 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:31.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:31.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:31.314 INFO:tasks.rados.rados.0.smithi076.stdout:3235: expect (ObjNum 1096 snap 378 seq_num 1096) 2021-03-15T22:33:31.410 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:31.415 INFO:tasks.rados.rados.0.smithi076.stdout:3233: finishing copy_from to smithi07694878-44 2021-03-15T22:33:31.416 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 2092 (ObjNum 829 snap 275 seq_num 829) dirty exists 2021-03-15T22:33:31.416 INFO:tasks.rados.rados.0.smithi076.stdout:3233: finishing copy_from racing read to smithi07694878-44 2021-03-15T22:33:31.432 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:31.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:31.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:31.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:31.561 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:31.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:31.563 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:31.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:31.690 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:31.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:31.692 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:31.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:31 smithi076 conmon[56514]: audit 2021-03-15T22:33:30.189730+0000 mgr.y (mgr.14144) 984 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:31.739 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:31 smithi076 conmon[56514]: cluster 2021-03-15T22:33:31.298505+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2021-03-15T22:33:31.739 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:31 smithi076 conmon[62156]: audit 2021-03-15T22:33:30.189730+0000 mgr.y (mgr.14144) 984 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:31.740 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:31 smithi076 conmon[62156]: cluster 2021-03-15T22:33:31.298505+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2021-03-15T22:33:31.792 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:31.816 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:31.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:31.818 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:31.898 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:31.919 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:31.946 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:31.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:31.947 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:32.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:31 smithi125 conmon[56739]: audit 2021-03-15T22:33:30.189730+0000 mgr.y (mgr.14144) 984 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:32.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:31 smithi125 conmon[56739]: cluster 2021-03-15T22:33:32.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:31 smithi125 conmon[56739]: 2021-03-15T22:33:31.298505+0000 mon.a (mon.0) 2189 : cluster [DBG] osdmap e861: 8 total, 8 up, 8 in 2021-03-15T22:33:32.049 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:32.074 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:32.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:32.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:32.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:32.199 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:32.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:32.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:32.284 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:33:32.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:32.307 INFO:tasks.rados.rados.0.smithi076.stdout:3233: done (3 left) 2021-03-15T22:33:32.307 INFO:tasks.rados.rados.0.smithi076.stdout:3234: done (2 left) 2021-03-15T22:33:32.307 INFO:tasks.rados.rados.0.smithi076.stdout:3235: done (1 left) 2021-03-15T22:33:32.307 INFO:tasks.rados.rados.0.smithi076.stdout:3236: done (0 left) 2021-03-15T22:33:32.308 INFO:tasks.rados.rados.0.smithi076.stdout:3237: snap_remove snap 373 2021-03-15T22:33:32.308 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:33:32.313 INFO:teuthology.orchestra.run.smithi076.stderr:set osd.2 primary-affinity to 0.533643 (8349722) 2021-03-15T22:33:32.329 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:32.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:32.331 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:32.332 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:33:32.350 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:33:32.369 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:33:32.386 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:33:32.405 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:33:32.425 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:33:32.426 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:33:32.427 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:33:32.427 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:33:32.427 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:33:32.427 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:33:32.428 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:33:32.428 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:33:32.428 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:33:32.431 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:32.454 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:32.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:32.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:32.556 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:32.578 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:32.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:32.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:32.707 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:32.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:32.709 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:32.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:32.835 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:32.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:32.937 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:32.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:32.959 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:32.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:33.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[56514]: cluster 2021-03-15T22:33:31.191352+0000 mgr.y (mgr.14144) 985 : cluster [DBG] pgmap v1435: 148 pgs: 148 active+clean; 580 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2021-03-15T22:33:33.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[56514]: cluster 2021-03-15T22:33:31.876107+0000 osd.2 (osd.2) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:33.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[56514]: cluster 2021-03-15T22:33:31.879276+0000 osd.2 (osd.2) 38 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:33.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[56514]: audit 2021-03-15T22:33:32.152219+0000 mon.c (mon.1) 67 : audit [INF] from='client.? 172.21.15.76:0/1367388409' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5336428333779778}]: dispatch 2021-03-15T22:33:33.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[56514]: audit 2021-03-15T22:33:32.152557+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5336428333779778}]: dispatch 2021-03-15T22:33:33.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[56514]: audit 2021-03-15T22:33:32.264603+0000 mon.a (mon.0) 2191 : audit [DBG] from='client.? 172.21.15.76:0/2630544966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:33.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[56514]: audit 2021-03-15T22:33:32.301415+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5336428333779778}]': finished 2021-03-15T22:33:33.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[56514]: cluster 2021-03-15T22:33:32.301490+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2021-03-15T22:33:33.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[62156]: cluster 2021-03-15T22:33:31.191352+0000 mgr.y (mgr.14144) 985 : cluster [DBG] pgmap v1435: 148 pgs: 148 active+clean; 580 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2021-03-15T22:33:33.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[62156]: cluster 2021-03-15T22:33:31.876107+0000 osd.2 (osd.2) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:33.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[62156]: cluster 2021-03-15T22:33:31.879276+0000 osd.2 (osd.2) 38 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:33.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[62156]: audit 2021-03-15T22:33:32.152219+0000 mon.c (mon.1) 67 : audit [INF] from='client.? 172.21.15.76:0/1367388409' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5336428333779778}]: dispatch 2021-03-15T22:33:33.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[62156]: audit 2021-03-15T22:33:32.152557+0000 mon.a (mon.0) 2190 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5336428333779778}]: dispatch 2021-03-15T22:33:33.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[62156]: audit 2021-03-15T22:33:32.264603+0000 mon.a (mon.0) 2191 : audit [DBG] from='client.? 172.21.15.76:0/2630544966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:33.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[62156]: audit 2021-03-15T22:33:32.301415+0000 mon.a (mon.0) 2192 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5336428333779778}]': finished 2021-03-15T22:33:33.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:32 smithi076 conmon[62156]: cluster 2021-03-15T22:33:32.301490+0000 mon.a (mon.0) 2193 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2021-03-15T22:33:33.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: cluster 2021-03-15T22:33:31.191352+0000 mgr.y (mgr.14144) 985 : cluster [DBG] pgmap v1435: 148 pgs: 148 active+clean; 580 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 14 MiB/s rd, 2.5 MiB/s wr, 18 op/s 2021-03-15T22:33:33.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: cluster 2021-03-15T22:33:31.876107+0000 osd.2 (osd.2) 37 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:33.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: cluster 2021-03-15T22:33:31 2021-03-15T22:33:33.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: .879276+0000 osd.2 (osd.2) 38 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:33.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:33:33.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: T22:33:32.152219+0000 mon.c ( 2021-03-15T22:33:33.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: mon.1) 67 : audit [INF] from='client.? 172.21.15.76:0/1367388409' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5336428333779778}]: dispatch 2021-03-15T22:33:33.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: audit 2021-03-15T22:33:33.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: 2021-03-15T22:33:32.152557 2021-03-15T22:33:33.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2190 : audit 2021-03-15T22:33:33.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5336428333779778}]: dispatch 2021-03-15T22:33:33.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:33:33.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: 22:33:32.264603+0000 mon.a (mon. 2021-03-15T22:33:33.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: 0) 2191 : audit [DBG] from='client.? 172.21.15.76:0/2630544966' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:33.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: audit 2021 2021-03-15T22:33:33.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: -03-15T22:33:32.301415+0000 2021-03-15T22:33:33.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: mon.a (mon.0) 2192 : audit [INF] 2021-03-15T22:33:33.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.5336428333779778}]': finished 2021-03-15T22:33:33.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:33:33.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: :33:32.301490+0000 mon.a (mon.0) 2021-03-15T22:33:33.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:32 smithi125 conmon[56739]: 2193 : cluster [DBG] osdmap e862: 8 total, 8 up, 8 in 2021-03-15T22:33:33.061 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:33.083 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:33.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:33.084 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:33.185 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:33.208 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:33.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:33.210 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:33.305 INFO:tasks.rados.rados.0.smithi076.stdout:3237: done (0 left) 2021-03-15T22:33:33.306 INFO:tasks.rados.rados.0.smithi076.stdout:3238: snap_remove snap 326 2021-03-15T22:33:33.311 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:33.337 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:33.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:33.339 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:33.440 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:33.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:33.464 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:33.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:33.566 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:33.592 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:33.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:33.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:33.602 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:33:33.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:33.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:33.718 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:33.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:33.843 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:33.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:33.845 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:33.947 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:33.968 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:33.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:33.970 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:34.071 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:34.098 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:34.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:34.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:34.201 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:34.227 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:34.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:34.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:34.312 INFO:tasks.rados.rados.0.smithi076.stdout:3238: done (0 left) 2021-03-15T22:33:34.313 INFO:tasks.rados.rados.0.smithi076.stdout:3239: delete oid 3 current snap is 380 2021-03-15T22:33:34.323 INFO:tasks.rados.rados.0.smithi076.stdout:3239: done (0 left) 2021-03-15T22:33:34.323 INFO:tasks.rados.rados.0.smithi076.stdout:3240: read oid 10 snap -1 2021-03-15T22:33:34.323 INFO:tasks.rados.rados.0.smithi076.stdout:3240: expect (ObjNum 1077 snap 366 seq_num 1077) 2021-03-15T22:33:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:3241: read oid 43 snap -1 2021-03-15T22:33:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:3241: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:3242: write (excl) oid 19 current snap is 380 2021-03-15T22:33:34.324 INFO:tasks.rados.rados.0.smithi076.stdout:3242: seq_num 1098 ranges {454086=578234,1669111=662043} 2021-03-15T22:33:34.329 INFO:tasks.rados.rados.0.smithi076.stdout:3242: writing smithi07694878-19 from 454086 to 1032320 tid 1 2021-03-15T22:33:34.330 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:34.337 INFO:tasks.rados.rados.0.smithi076.stdout:3242: writing smithi07694878-19 from 1669111 to 2331154 tid 2 2021-03-15T22:33:34.338 INFO:tasks.rados.rados.0.smithi076.stdout:3243: write oid 31 current snap is 380 2021-03-15T22:33:34.338 INFO:tasks.rados.rados.0.smithi076.stdout:3243: seq_num 1099 ranges {470893=653483,1727593=522054} 2021-03-15T22:33:34.344 INFO:tasks.rados.rados.0.smithi076.stdout:3243: writing smithi07694878-31 from 470893 to 1124376 tid 1 2021-03-15T22:33:34.350 INFO:tasks.rados.rados.0.smithi076.stdout:3243: writing smithi07694878-31 from 1727593 to 2249647 tid 2 2021-03-15T22:33:34.351 INFO:tasks.rados.rados.0.smithi076.stdout:3244: rollback oid 12 current snap is 380 2021-03-15T22:33:34.351 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 12 to 364 2021-03-15T22:33:34.351 INFO:tasks.rados.rados.0.smithi076.stdout:3241: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:33:34.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:34.356 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:34.357 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:34.370 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:34.437 INFO:tasks.rados.rados.0.smithi076.stdout:3240: expect (ObjNum 1077 snap 366 seq_num 1077) 2021-03-15T22:33:34.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:34.468 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: cluster 2021-03-15T22:33:34.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: 2021- 2021-03-15T22:33:34.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: 03-15T22:33:33.191882+0000 mgr.y (mgr.14144) 986 : cluster [DBG] pgmap v1438: 148 pgs: 1 peering, 147 active+clean; 587 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 4.0 MiB/s wr, 18 op/s; 27 B/s, 0 objects/s recovering 2021-03-15T22:33:34.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: cluster 2021-03-15T22:33:34.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: 2021-03-15T22:33:34.469 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: 2021 2021-03-15T22:33:34.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: - 2021-03-15T22:33:34.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: 03-15T22:33 2021-03-15T22:33:34.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: :33.302908+0000 mon.a (mon.0) 2194 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2021-03-15T22:33:34.470 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: cluster 2021-03-15T22:33:34.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: 2021-03-15 2021-03-15T22:33:34.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: T22:33:33.581393+0000 mon.a (mon.0) 2195 : cluster 2021-03-15T22:33:34.471 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[56514]: [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:33:34.471 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: cluster 2021-03-15T22:33:34.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 2021-03-15T22:33:34.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 2021 2021-03-15T22:33:34.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: - 2021-03-15T22:33:34.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 03 2021-03-15T22:33:34.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: - 2021-03-15T22:33:34.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 15 2021-03-15T22:33:34.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: T 2021-03-15T22:33:34.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 22 2021-03-15T22:33:34.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: : 2021-03-15T22:33:34.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 33 2021-03-15T22:33:34.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: : 2021-03-15T22:33:34.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 33 2021-03-15T22:33:34.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: . 2021-03-15T22:33:34.474 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 191882 2021-03-15T22:33:34.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: +0000 2021-03-15T22:33:34.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 2021-03-15T22:33:34.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: mgr.y (mgr.14144) 986 : cluster [DBG] 2021-03-15T22:33:34.475 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: pgmap v1438: 148 pgs: 1 peering, 147 active+clean; 587 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 4.0 MiB/s wr, 18 op/s; 27 B/s, 0 objects/s recovering 2021-03-15T22:33:34.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:33:34.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 03-15T22:33:33.302908+0000 2021-03-15T22:33:34.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: mon.a (mon.0) 2194 : cluster [DBG] osdmap e863: 8 total, 8 up, 8 in 2021-03-15T22:33:34.476 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 2021-03-15T22:33:34.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:34.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: -03-15T22:33:33. 2021-03-15T22:33:34.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: 581393+0000 mon.a (mon.0) 2195 : cluster 2021-03-15T22:33:34.477 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:34 smithi076 conmon[62156]: [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:33:34.482 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:34.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:34.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:34.512 INFO:tasks.rados.rados.0.smithi076.stdout:3242: finishing write tid 1 to smithi07694878-19 2021-03-15T22:33:34.512 INFO:tasks.rados.rados.0.smithi076.stdout:3242: finishing write tid 2 to smithi07694878-19 2021-03-15T22:33:34.513 INFO:tasks.rados.rados.0.smithi076.stdout:3242: finishing write tid 3 to smithi07694878-19 2021-03-15T22:33:34.513 INFO:tasks.rados.rados.0.smithi076.stdout:3242: finishing write tid 4 to smithi07694878-19 2021-03-15T22:33:34.513 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2142 (ObjNum 1098 snap 380 seq_num 1098) dirty exists 2021-03-15T22:33:34.513 INFO:tasks.rados.rados.0.smithi076.stdout:3242: left oid 19 (ObjNum 1098 snap 380 seq_num 1098) 2021-03-15T22:33:34.514 INFO:tasks.rados.rados.0.smithi076.stdout:3243: finishing write tid 1 to smithi07694878-31 2021-03-15T22:33:34.514 INFO:tasks.rados.rados.0.smithi076.stdout:3240: done (4 left) 2021-03-15T22:33:34.514 INFO:tasks.rados.rados.0.smithi076.stdout:3241: done (3 left) 2021-03-15T22:33:34.514 INFO:tasks.rados.rados.0.smithi076.stdout:3242: done (2 left) 2021-03-15T22:33:34.515 INFO:tasks.rados.rados.0.smithi076.stdout:3245: snap_remove snap 331 2021-03-15T22:33:34.584 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:34.608 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:34.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:34.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:34.704 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:33:34.704 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:33:34.705 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:33:34.705 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:33:34.705 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.706 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2768 7.b 7:d10ba14d:::smithi07694878-43:head [write 1689059~741123] snapc d2=[d2,d1,cf] ondisk+write+known_if_redirected e310)", 2021-03-15T22:33:34.706 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:33:34.707 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 589.59505808799997, 2021-03-15T22:33:34.707 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078094859000000003, 2021-03-15T22:33:34.707 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.707 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:34.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.708 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.708 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2768 2021-03-15T22:33:34.708 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.709 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.709 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.709 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.709 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:33:34.710 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.710 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.710 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.710 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.711 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105090+0000", 2021-03-15T22:33:34.711 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00033110399999999998 2021-03-15T22:33:34.711 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.711 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.711 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.712 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105421+0000", 2021-03-15T22:33:34.712 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4809999999999998e-06 2021-03-15T22:33:34.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.712 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.712 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.713 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105425+0000", 2021-03-15T22:33:34.713 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7363999999999998e-05 2021-03-15T22:33:34.713 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.713 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.714 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.714 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105442+0000", 2021-03-15T22:33:34.714 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9480000000000002e-06 2021-03-15T22:33:34.714 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.714 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.715 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.715 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105449+0000", 2021-03-15T22:33:34.715 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6847000000000001e-05 2021-03-15T22:33:34.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.715 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.716 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.716 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105496+0000", 2021-03-15T22:33:34.716 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8231000000000003e-05 2021-03-15T22:33:34.716 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.717 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.717 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.717 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105584+0000", 2021-03-15T22:33:34.717 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000198369 2021-03-15T22:33:34.717 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.718 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.718 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:33:34.718 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105782+0000", 2021-03-15T22:33:34.718 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003007292 2021-03-15T22:33:34.718 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.719 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.719 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.719 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108790+0000", 2021-03-15T22:33:34.719 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000200027 2021-03-15T22:33:34.720 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.720 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.720 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.720 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108990+0000", 2021-03-15T22:33:34.720 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074107929000000003 2021-03-15T22:33:34.721 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.721 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.721 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:34.721 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183098+0000", 2021-03-15T22:33:34.721 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4002000000000002e-05 2021-03-15T22:33:34.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.722 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.722 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:34.722 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183152+0000", 2021-03-15T22:33:34.722 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3265000000000001e-05 2021-03-15T22:33:34.723 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.723 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.723 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.723 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183185+0000", 2021-03-15T22:33:34.724 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078094859000000003 2021-03-15T22:33:34.724 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.724 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.724 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.724 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.725 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.725 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2769 7.b 7:d10ba14d:::smithi07694878-43:head [setxattr _header (58) in=7b,truncate 2588344] snapc d2=[d2,d1,cf] ondisk+write+known_if_redirected e310)", 2021-03-15T22:33:34.725 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:33:34.725 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 589.59468705999996, 2021-03-15T22:33:34.726 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077795487999999996, 2021-03-15T22:33:34.726 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.726 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:34.726 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.727 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.727 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.727 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2769 2021-03-15T22:33:34.727 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.727 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.728 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.728 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.728 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:33:34.728 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.729 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.729 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.729 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105461+0000", 2021-03-15T22:33:34.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2639999999999998e-06 2021-03-15T22:33:34.730 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.730 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.730 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.730 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105463+0000", 2021-03-15T22:33:34.731 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.426e-06 2021-03-15T22:33:34.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.731 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105465+0000", 2021-03-15T22:33:34.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9349999999999999e-06 2021-03-15T22:33:34.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.732 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105471+0000", 2021-03-15T22:33:34.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2420000000000002e-06 2021-03-15T22:33:34.733 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.733 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105475+0000", 2021-03-15T22:33:34.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001413924 2021-03-15T22:33:34.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.106889+0000", 2021-03-15T22:33:34.735 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5030000000000003e-05 2021-03-15T22:33:34.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.736 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.106984+0000", 2021-03-15T22:33:34.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014550399999999999 2021-03-15T22:33:34.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.737 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:33:34.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107129+0000", 2021-03-15T22:33:34.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001729529 2021-03-15T22:33:34.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.108859+0000", 2021-03-15T22:33:34.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00023022799999999999 2021-03-15T22:33:34.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.739 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.109089+0000", 2021-03-15T22:33:34.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074107424000000005 2021-03-15T22:33:34.740 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:34.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183197+0000", 2021-03-15T22:33:34.741 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7657e-05 2021-03-15T22:33:34.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:34.742 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183224+0000", 2021-03-15T22:33:34.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2325e-05 2021-03-15T22:33:34.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.743 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183257+0000", 2021-03-15T22:33:34.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077795487999999996 2021-03-15T22:33:34.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.743 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.744 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.744 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:2770 7.b 7:d10ba14d:::smithi07694878-43:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e310)", 2021-03-15T22:33:34.745 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:33:34.745 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 589.59465685199996, 2021-03-15T22:33:34.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077923611000000004, 2021-03-15T22:33:34.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.745 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:34.746 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.746 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.746 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.746 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2770 2021-03-15T22:33:34.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.747 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:33:34.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105491+0000", 2021-03-15T22:33:34.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3200000000000001e-06 2021-03-15T22:33:34.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105493+0000", 2021-03-15T22:33:34.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3960000000000001e-06 2021-03-15T22:33:34.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.750 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105494+0000", 2021-03-15T22:33:34.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6490000000000002e-06 2021-03-15T22:33:34.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105501+0000", 2021-03-15T22:33:34.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9989999999999999e-06 2021-03-15T22:33:34.752 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.105506+0000", 2021-03-15T22:33:34.753 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019212770000000001 2021-03-15T22:33:34.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.754 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107427+0000", 2021-03-15T22:33:34.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8102000000000005e-05 2021-03-15T22:33:34.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.755 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:34.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.107515+0000", 2021-03-15T22:33:34.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075777318999999996 2021-03-15T22:33:34.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.756 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183292+0000", 2021-03-15T22:33:34.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9417999999999999e-05 2021-03-15T22:33:34.757 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183352+0000", 2021-03-15T22:33:34.758 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3131000000000003e-05 2021-03-15T22:33:34.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.758 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.759 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:23:45.183415+0000", 2021-03-15T22:33:34.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077923611000000004 2021-03-15T22:33:34.759 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.759 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.759 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.760 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.760 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3010 7.b 7:d980be1f:::smithi07694878-23:head [write 726168~704991,stat] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:33:34.760 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:33:34.760 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 571.61363527399999, 2021-03-15T22:33:34.761 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0570805230000002, 2021-03-15T22:33:34.761 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.761 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:34.761 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.762 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.762 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.762 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3010 2021-03-15T22:33:34.762 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.762 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:33:34.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086513+0000", 2021-03-15T22:33:34.764 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00034716800000000002 2021-03-15T22:33:34.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.765 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086860+0000", 2021-03-15T22:33:34.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3299999999999999e-06 2021-03-15T22:33:34.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086863+0000", 2021-03-15T22:33:34.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1489e-05 2021-03-15T22:33:34.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.767 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086875+0000", 2021-03-15T22:33:34.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.605e-06 2021-03-15T22:33:34.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.768 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086882+0000", 2021-03-15T22:33:34.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5296999999999997e-05 2021-03-15T22:33:34.769 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086928+0000", 2021-03-15T22:33:34.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2879999999999998e-06 2021-03-15T22:33:34.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:34.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.086937+0000", 2021-03-15T22:33:34.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99891912599999999 2021-03-15T22:33:34.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085856+0000", 2021-03-15T22:33:34.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1169999999999993e-06 2021-03-15T22:33:34.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:34.772 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085865+0000", 2021-03-15T22:33:34.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0140395639999999 2021-03-15T22:33:34.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.773 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.099905+0000", 2021-03-15T22:33:34.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022436600000000001 2021-03-15T22:33:34.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.774 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.100129+0000", 2021-03-15T22:33:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00043622700000000002 2021-03-15T22:33:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:33:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.100565+0000", 2021-03-15T22:33:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024326999999999999 2021-03-15T22:33:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102998+0000", 2021-03-15T22:33:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002982012 2021-03-15T22:33:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.105980+0000", 2021-03-15T22:33:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037236714999999997 2021-03-15T22:33:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143217+0000", 2021-03-15T22:33:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147832 2021-03-15T22:33:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143365+0000", 2021-03-15T22:33:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022868699999999999 2021-03-15T22:33:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.780 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143593+0000", 2021-03-15T22:33:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0570805230000002 2021-03-15T22:33:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.781 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.782 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3011 7.b 7:d980be1f:::smithi07694878-23:head [write 1895071~683474] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:33:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:33:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 571.60735502800003, 2021-03-15T22:33:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.09731113, 2021-03-15T22:33:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.783 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3011 2021-03-15T22:33:34.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:33:34.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.092793+0000", 2021-03-15T22:33:34.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027883500000000001 2021-03-15T22:33:34.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.787 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093072+0000", 2021-03-15T22:33:34.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.809e-06 2021-03-15T22:33:34.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093077+0000", 2021-03-15T22:33:34.789 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9491e-05 2021-03-15T22:33:34.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.790 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093096+0000", 2021-03-15T22:33:34.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.126e-06 2021-03-15T22:33:34.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.791 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093103+0000", 2021-03-15T22:33:34.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4002000000000001e-05 2021-03-15T22:33:34.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093167+0000", 2021-03-15T22:33:34.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0259999999999996e-06 2021-03-15T22:33:34.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:34.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093176+0000", 2021-03-15T22:33:34.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99270957599999998 2021-03-15T22:33:34.793 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.794 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085886+0000", 2021-03-15T22:33:34.794 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9440000000000002e-06 2021-03-15T22:33:34.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:34.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085892+0000", 2021-03-15T22:33:34.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.016173532 2021-03-15T22:33:34.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.796 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.796 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102065+0000", 2021-03-15T22:33:34.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000141037 2021-03-15T22:33:34.797 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:34.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102207+0000", 2021-03-15T22:33:34.798 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041447062 2021-03-15T22:33:34.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143654+0000", 2021-03-15T22:33:34.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014246199999999999 2021-03-15T22:33:34.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.799 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.143796+0000", 2021-03-15T22:33:34.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000272444 2021-03-15T22:33:34.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.800 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:33:34.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.144068+0000", 2021-03-15T22:33:34.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018956929999999999 2021-03-15T22:33:34.801 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145964+0000", 2021-03-15T22:33:34.802 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041997686999999999 2021-03-15T22:33:34.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:34.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.187962+0000", 2021-03-15T22:33:34.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019156150000000001 2021-03-15T22:33:34.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.804 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.804 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.189877+0000", 2021-03-15T22:33:34.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8467999999999995e-05 2021-03-15T22:33:34.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.805 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:34.805 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.189956+0000", 2021-03-15T22:33:34.805 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014832100000000001 2021-03-15T22:33:34.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.806 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.806 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.806 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190104+0000", 2021-03-15T22:33:34.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.09731113 2021-03-15T22:33:34.806 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.807 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.807 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.808 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3012 7.b 7:d980be1f:::smithi07694878-23:head [setxattr _header (58) in=7b,truncate 3126168] snapc e3=[e1,e0,df,de,dd,dc,db] ondisk+write+known_if_redirected e328)", 2021-03-15T22:33:34.808 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:33:34.808 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 571.60703219599998, 2021-03-15T22:33:34.808 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0972930829999998, 2021-03-15T22:33:34.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.809 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:34.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.809 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.809 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3012 2021-03-15T22:33:34.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.810 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:33:34.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093116+0000", 2021-03-15T22:33:34.812 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.61e-06 2021-03-15T22:33:34.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.813 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093118+0000", 2021-03-15T22:33:34.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.234e-06 2021-03-15T22:33:34.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.814 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093119+0000", 2021-03-15T22:33:34.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7290000000000002e-06 2021-03-15T22:33:34.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093123+0000", 2021-03-15T22:33:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1649999999999999e-06 2021-03-15T22:33:34.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093127+0000", 2021-03-15T22:33:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6592e-05 2021-03-15T22:33:34.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093213+0000", 2021-03-15T22:33:34.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2364000000000001e-05 2021-03-15T22:33:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093226+0000", 2021-03-15T22:33:34.818 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99268559899999997 2021-03-15T22:33:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085911+0000", 2021-03-15T22:33:34.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1440000000000002e-06 2021-03-15T22:33:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:34.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085916+0000", 2021-03-15T22:33:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0164949109999999 2021-03-15T22:33:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102411+0000", 2021-03-15T22:33:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000116049 2021-03-15T22:33:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:34.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102527+0000", 2021-03-15T22:33:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042432780000000003 2021-03-15T22:33:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.144960+0000", 2021-03-15T22:33:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7323000000000003e-05 2021-03-15T22:33:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145057+0000", 2021-03-15T22:33:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018096599999999999 2021-03-15T22:33:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,6", 2021-03-15T22:33:34.826 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145238+0000", 2021-03-15T22:33:34.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00077369500000000003 2021-03-15T22:33:34.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.827 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.146012+0000", 2021-03-15T22:33:34.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041960274999999998 2021-03-15T22:33:34.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.828 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:34.828 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.187972+0000", 2021-03-15T22:33:34.828 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021867610000000002 2021-03-15T22:33:34.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.829 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.829 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.829 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190159+0000", 2021-03-15T22:33:34.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9751000000000004e-05 2021-03-15T22:33:34.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.830 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.830 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:34.830 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190229+0000", 2021-03-15T22:33:34.830 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000180135 2021-03-15T22:33:34.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.831 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190409+0000", 2021-03-15T22:33:34.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0972930829999998 2021-03-15T22:33:34.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.832 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.832 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.833 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.833 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3013 7.b 7:d980be1f:::smithi07694878-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e328)", 2021-03-15T22:33:34.833 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:33:34.833 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 571.60699890800004, 2021-03-15T22:33:34.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.097631684, 2021-03-15T22:33:34.834 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.834 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:34.834 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.835 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.835 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.835 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3013 2021-03-15T22:33:34.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.835 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.836 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:33:34.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093149+0000", 2021-03-15T22:33:34.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9600000000000008e-07 2021-03-15T22:33:34.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.838 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093150+0000", 2021-03-15T22:33:34.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.514e-06 2021-03-15T22:33:34.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093152+0000", 2021-03-15T22:33:34.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2985e-05 2021-03-15T22:33:34.840 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093165+0000", 2021-03-15T22:33:34.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4000000000000002e-06 2021-03-15T22:33:34.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093169+0000", 2021-03-15T22:33:34.842 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2490999999999993e-05 2021-03-15T22:33:34.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.843 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093252+0000", 2021-03-15T22:33:34.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7490000000000001e-06 2021-03-15T22:33:34.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.844 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:34.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:03.093258+0000", 2021-03-15T22:33:34.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99267630200000001 2021-03-15T22:33:34.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085935+0000", 2021-03-15T22:33:34.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0580000000000003e-06 2021-03-15T22:33:34.846 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:34.847 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:04.085941+0000", 2021-03-15T22:33:34.847 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.01663009 2021-03-15T22:33:34.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.848 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.848 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102571+0000", 2021-03-15T22:33:34.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012747 2021-03-15T22:33:34.848 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.848 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.849 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:34.849 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.102698+0000", 2021-03-15T22:33:34.849 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042823333999999998 2021-03-15T22:33:34.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.850 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.850 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145522+0000", 2021-03-15T22:33:34.850 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7650000000000004e-05 2021-03-15T22:33:34.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.851 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:34.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.145599+0000", 2021-03-15T22:33:34.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.044843263000000001 2021-03-15T22:33:34.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190443+0000", 2021-03-15T22:33:34.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000117391 2021-03-15T22:33:34.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190560+0000", 2021-03-15T22:33:34.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000220991 2021-03-15T22:33:34.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:24:05.190781+0000", 2021-03-15T22:33:34.854 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.097631684 2021-03-15T22:33:34.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.855 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.855 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.856 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4284 7.b 7:d4368cae:::smithi07694878-18:head [write 3203532~509848] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:33:34.856 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:33:34.856 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 454.53736993799998, 2021-03-15T22:33:34.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068311582999999995, 2021-03-15T22:33:34.857 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.857 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:34.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.857 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.858 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.858 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4284 2021-03-15T22:33:34.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.858 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:33:34.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:33:34.861 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015451700000000001 2021-03-15T22:33:34.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162933+0000", 2021-03-15T22:33:34.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.25e-06 2021-03-15T22:33:34.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162937+0000", 2021-03-15T22:33:34.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2824e-05 2021-03-15T22:33:34.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.863 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.863 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.864 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162960+0000", 2021-03-15T22:33:34.864 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.072e-06 2021-03-15T22:33:34.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.864 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.865 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162968+0000", 2021-03-15T22:33:34.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7661999999999999e-05 2021-03-15T22:33:34.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.865 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.866 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.866 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163016+0000", 2021-03-15T22:33:34.866 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108743 2021-03-15T22:33:34.866 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.867 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.867 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163124+0000", 2021-03-15T22:33:34.867 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021732500000000001 2021-03-15T22:33:34.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:33:34.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163342+0000", 2021-03-15T22:33:34.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043853354999999997 2021-03-15T22:33:34.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.207195+0000", 2021-03-15T22:33:34.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055661469999999996 2021-03-15T22:33:34.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.212761+0000", 2021-03-15T22:33:34.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018240684 2021-03-15T22:33:34.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:34.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231002+0000", 2021-03-15T22:33:34.871 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6122000000000003e-05 2021-03-15T22:33:34.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:34.872 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231048+0000", 2021-03-15T22:33:34.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1882000000000003e-05 2021-03-15T22:33:34.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231090+0000", 2021-03-15T22:33:34.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068311582999999995 2021-03-15T22:33:34.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.874 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.875 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4285 7.b 7:d4368cae:::smithi07694878-18:head [setxattr _header (58) in=7b,truncate 3865709] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:33:34.875 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:33:34.875 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 454.53716546599998, 2021-03-15T22:33:34.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068193614999999999, 2021-03-15T22:33:34.876 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.876 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:34.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.876 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.877 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.877 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4285 2021-03-15T22:33:34.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.877 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.878 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:33:34.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.879 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:33:34.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.633e-06 2021-03-15T22:33:34.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.880 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162985+0000", 2021-03-15T22:33:34.881 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6789999999999999e-06 2021-03-15T22:33:34.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.881 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.881 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162987+0000", 2021-03-15T22:33:34.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9500000000000001e-06 2021-03-15T22:33:34.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.882 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.882 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.883 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162995+0000", 2021-03-15T22:33:34.883 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.459e-06 2021-03-15T22:33:34.883 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.883 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.884 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162999+0000", 2021-03-15T22:33:34.884 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012207240000000001 2021-03-15T22:33:34.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164220+0000", 2021-03-15T22:33:34.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0790000000000004e-05 2021-03-15T22:33:34.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.885 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164301+0000", 2021-03-15T22:33:34.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014765399999999999 2021-03-15T22:33:34.886 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:33:34.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164449+0000", 2021-03-15T22:33:34.887 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042818110999999999 2021-03-15T22:33:34.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.888 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.207267+0000", 2021-03-15T22:33:34.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055270550000000003 2021-03-15T22:33:34.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.889 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:34.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.212794+0000", 2021-03-15T22:33:34.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018305267 2021-03-15T22:33:34.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:34.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231099+0000", 2021-03-15T22:33:34.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4928000000000002e-05 2021-03-15T22:33:34.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.891 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:34.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231134+0000", 2021-03-15T22:33:34.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2364999999999998e-05 2021-03-15T22:33:34.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231176+0000", 2021-03-15T22:33:34.893 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068193614999999999 2021-03-15T22:33:34.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.893 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.893 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.894 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4286 7.b 7:d4368cae:::smithi07694878-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e445)", 2021-03-15T22:33:34.894 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:33:34.894 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 454.537143818, 2021-03-15T22:33:34.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068364477000000007, 2021-03-15T22:33:34.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.895 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:34.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.895 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.896 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.896 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4286 2021-03-15T22:33:34.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.896 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.897 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:33:34.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.898 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.898 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:33:34.898 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6500000000000008e-07 2021-03-15T22:33:34.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.899 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.899 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163005+0000", 2021-03-15T22:33:34.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.218e-06 2021-03-15T22:33:34.900 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.900 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.900 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.900 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163007+0000", 2021-03-15T22:33:34.900 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3890000000000002e-06 2021-03-15T22:33:34.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.901 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.901 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163013+0000", 2021-03-15T22:33:34.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.053e-06 2021-03-15T22:33:34.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.902 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163017+0000", 2021-03-15T22:33:34.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017994630000000001 2021-03-15T22:33:34.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164816+0000", 2021-03-15T22:33:34.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0958000000000007e-05 2021-03-15T22:33:34.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:34.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164897+0000", 2021-03-15T22:33:34.905 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066313677000000001 2021-03-15T22:33:34.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231211+0000", 2021-03-15T22:33:34.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9739000000000004e-05 2021-03-15T22:33:34.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.906 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231281+0000", 2021-03-15T22:33:34.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8015000000000003e-05 2021-03-15T22:33:34.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231369+0000", 2021-03-15T22:33:34.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068364477000000007 2021-03-15T22:33:34.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.909 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.909 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4287 7.7 e445/420 7:ed4a01d2:::smithi07694878-2:head v 445'413, mlcod=444'410)", 2021-03-15T22:33:34.910 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:33:34.910 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 454.53612552200002, 2021-03-15T22:33:34.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066973671999999998, 2021-03-15T22:33:34.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.910 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:34.911 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.911 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:33:34.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.912 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:33:34.913 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.258e-06 2021-03-15T22:33:34.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164026+0000", 2021-03-15T22:33:34.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7980000000000001e-06 2021-03-15T22:33:34.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164029+0000", 2021-03-15T22:33:34.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0787e-05 2021-03-15T22:33:34.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164039+0000", 2021-03-15T22:33:34.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2829999999999998e-06 2021-03-15T22:33:34.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164046+0000", 2021-03-15T22:33:34.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5735000000000001e-05 2021-03-15T22:33:34.917 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164101+0000", 2021-03-15T22:33:34.918 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9206000000000001e-05 2021-03-15T22:33:34.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.919 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.919 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.919 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164131+0000", 2021-03-15T22:33:34.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066830296999999997 2021-03-15T22:33:34.919 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.920 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:34.920 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.230961+0000", 2021-03-15T22:33:34.920 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5308000000000001e-05 2021-03-15T22:33:34.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.921 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.921 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.230996+0000", 2021-03-15T22:33:34.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066973671999999998 2021-03-15T22:33:34.922 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.922 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.922 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.923 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:5830 7.5 e631/559 7:ac84e656:::smithi07694878-31:head v 631'458, mlcod=630'454)", 2021-03-15T22:33:34.923 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:33:34.923 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 234.44848830699999, 2021-03-15T22:33:34.923 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083826879000000007, 2021-03-15T22:33:34.924 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.924 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:34.924 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:33:34.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:33:34.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060822330000000003 2021-03-15T22:33:34.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257742+0000", 2021-03-15T22:33:34.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9649999999999996e-06 2021-03-15T22:33:34.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257748+0000", 2021-03-15T22:33:34.928 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0319e-05 2021-03-15T22:33:34.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257768+0000", 2021-03-15T22:33:34.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2021-03-15T22:33:34.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.930 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257777+0000", 2021-03-15T22:33:34.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0096000000000001e-05 2021-03-15T22:33:34.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.931 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.931 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257847+0000", 2021-03-15T22:33:34.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7812000000000001e-05 2021-03-15T22:33:34.932 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257875+0000", 2021-03-15T22:33:34.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077557744999999997 2021-03-15T22:33:34.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:34.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.335433+0000", 2021-03-15T22:33:34.934 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3674999999999997e-05 2021-03-15T22:33:34.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.335487+0000", 2021-03-15T22:33:34.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083826879000000007 2021-03-15T22:33:34.935 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.935 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.935 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:755 7.5 e646/559 7:a9545d62:::smithi07694878-1:1fc v 646'483, mlcod=644'479)", 2021-03-15T22:33:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:33:34.936 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 219.45386395099999, 2021-03-15T22:33:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087969915999999995, 2021-03-15T22:33:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.937 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:33:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:33:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1300000000000001e-06 2021-03-15T22:33:34.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246287+0000", 2021-03-15T22:33:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1730000000000002e-06 2021-03-15T22:33:34.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246290+0000", 2021-03-15T22:33:34.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1728e-05 2021-03-15T22:33:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246301+0000", 2021-03-15T22:33:34.942 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7710000000000001e-06 2021-03-15T22:33:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246308+0000", 2021-03-15T22:33:34.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5705999999999999e-05 2021-03-15T22:33:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246354+0000", 2021-03-15T22:33:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8252e-05 2021-03-15T22:33:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:34.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246382+0000", 2021-03-15T22:33:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087836030999999995 2021-03-15T22:33:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:34.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334218+0000", 2021-03-15T22:33:34.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6124999999999997e-05 2021-03-15T22:33:34.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334254+0000", 2021-03-15T22:33:34.948 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087969915999999995 2021-03-15T22:33:34.948 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.948 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.949 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.949 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6447 7.b 7:d980be1f:::smithi07694878-23:21c [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e691)", 2021-03-15T22:33:34.949 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.215534+0000", 2021-03-15T22:33:34.950 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 174.484614394, 2021-03-15T22:33:34.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079694523000000003, 2021-03-15T22:33:34.950 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.950 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:34.951 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.951 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.951 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.951 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6447 2021-03-15T22:33:34.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.952 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215534+0000", 2021-03-15T22:33:34.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215534+0000", 2021-03-15T22:33:34.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2140000000000004e-06 2021-03-15T22:33:34.954 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215539+0000", 2021-03-15T22:33:34.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8670000000000004e-06 2021-03-15T22:33:34.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215543+0000", 2021-03-15T22:33:34.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4408e-05 2021-03-15T22:33:34.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215557+0000", 2021-03-15T22:33:34.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4090000000000003e-06 2021-03-15T22:33:34.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215566+0000", 2021-03-15T22:33:34.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2050000000000002e-05 2021-03-15T22:33:34.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215648+0000", 2021-03-15T22:33:34.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8050000000000007e-06 2021-03-15T22:33:34.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:33:34.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215656+0000", 2021-03-15T22:33:34.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079466859000000001 2021-03-15T22:33:34.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295122+0000", 2021-03-15T22:33:34.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000105911 2021-03-15T22:33:34.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295228+0000", 2021-03-15T22:33:34.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079694523000000003 2021-03-15T22:33:34.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.963 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.963 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.964 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6448 7.b 7:d980be1f:::smithi07694878-23:21c [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e691)", 2021-03-15T22:33:34.964 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.215580+0000", 2021-03-15T22:33:34.964 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 174.48456765899999, 2021-03-15T22:33:34.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079736783000000006, 2021-03-15T22:33:34.965 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.965 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:34.965 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.965 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.966 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.966 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6448 2021-03-15T22:33:34.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.966 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215580+0000", 2021-03-15T22:33:34.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215580+0000", 2021-03-15T22:33:34.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.248e-06 2021-03-15T22:33:34.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215582+0000", 2021-03-15T22:33:34.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.646e-06 2021-03-15T22:33:34.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215583+0000", 2021-03-15T22:33:34.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4140000000000002e-06 2021-03-15T22:33:34.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215589+0000", 2021-03-15T22:33:34.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9349999999999996e-06 2021-03-15T22:33:34.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215593+0000", 2021-03-15T22:33:34.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000158023 2021-03-15T22:33:34.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215751+0000", 2021-03-15T22:33:34.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2480000000000004e-06 2021-03-15T22:33:34.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:33:34.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215756+0000", 2021-03-15T22:33:34.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079496878000000007 2021-03-15T22:33:34.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295253+0000", 2021-03-15T22:33:34.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4391000000000001e-05 2021-03-15T22:33:34.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295317+0000", 2021-03-15T22:33:34.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079736783000000006 2021-03-15T22:33:34.977 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.977 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.978 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6449 7.b 7:d980be1f:::smithi07694878-23:21c [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected e691)", 2021-03-15T22:33:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.215597+0000", 2021-03-15T22:33:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 174.48455136699999, 2021-03-15T22:33:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079805300999999995, 2021-03-15T22:33:34.979 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:34.980 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6449 2021-03-15T22:33:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215597+0000", 2021-03-15T22:33:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215597+0000", 2021-03-15T22:33:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.621e-06 2021-03-15T22:33:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.983 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215598+0000", 2021-03-15T22:33:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.468e-06 2021-03-15T22:33:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215600+0000", 2021-03-15T22:33:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7419999999999997e-06 2021-03-15T22:33:34.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215605+0000", 2021-03-15T22:33:34.986 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4640000000000002e-06 2021-03-15T22:33:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215608+0000", 2021-03-15T22:33:34.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016377699999999999 2021-03-15T22:33:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215772+0000", 2021-03-15T22:33:34.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3549999999999998e-06 2021-03-15T22:33:34.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:33:34.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215776+0000", 2021-03-15T22:33:34.990 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079562503000000007 2021-03-15T22:33:34.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:34.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295339+0000", 2021-03-15T22:33:34.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3371e-05 2021-03-15T22:33:34.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:34.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295402+0000", 2021-03-15T22:33:34.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079805300999999995 2021-03-15T22:33:34.992 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.992 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:34.992 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:34.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.993 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:6453 7.9 e691/556 7:91d654eb:::smithi07694878-5:head v 691'298, mlcod=681'295)", 2021-03-15T22:33:34.993 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.298138+0000", 2021-03-15T22:33:34.994 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 174.40201010999999, 2021-03-15T22:33:34.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080231284999999999, 2021-03-15T22:33:34.994 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:34.994 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:34.994 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:34.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:34.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298138+0000", 2021-03-15T22:33:34.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:34.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:34.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298138+0000", 2021-03-15T22:33:34.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012879800000000001 2021-03-15T22:33:34.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:34.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298267+0000", 2021-03-15T22:33:34.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6959999999999998e-06 2021-03-15T22:33:34.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:34.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298271+0000", 2021-03-15T22:33:34.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999994e-06 2021-03-15T22:33:34.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:34.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:34.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:35.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298280+0000", 2021-03-15T22:33:35.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2299999999999996e-06 2021-03-15T22:33:35.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:35.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298286+0000", 2021-03-15T22:33:35.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045678599999999999 2021-03-15T22:33:35.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:35.002 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298743+0000", 2021-03-15T22:33:35.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4567999999999999e-05 2021-03-15T22:33:35.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:35.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298768+0000", 2021-03-15T22:33:35.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079546108000000004 2021-03-15T22:33:35.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:35.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378314+0000", 2021-03-15T22:33:35.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5658999999999997e-05 2021-03-15T22:33:35.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:35.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378369+0000", 2021-03-15T22:33:35.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080231284999999999 2021-03-15T22:33:35.005 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:35.006 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:35.006 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:35.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.006 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:7256 7.0 e772/642 7:071f7a04:::smithi07694878-37:head v 772'2464, mlcod=770'2460)", 2021-03-15T22:33:35.007 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:02.263683+0000", 2021-03-15T22:33:35.007 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 92.436465341000002, 2021-03-15T22:33:35.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079578619000000003, 2021-03-15T22:33:35.007 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:35.008 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:35.008 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:35.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:35.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.263683+0000", 2021-03-15T22:33:35.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:35.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:35.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.263683+0000", 2021-03-15T22:33:35.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00093819800000000001 2021-03-15T22:33:35.010 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:35.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.264621+0000", 2021-03-15T22:33:35.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2779999999999999e-06 2021-03-15T22:33:35.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:35.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.264627+0000", 2021-03-15T22:33:35.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.062960655000000004 2021-03-15T22:33:35.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.012 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:35.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.327588+0000", 2021-03-15T22:33:35.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.593e-06 2021-03-15T22:33:35.013 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:35.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.327596+0000", 2021-03-15T22:33:35.014 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00059060600000000003 2021-03-15T22:33:35.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:35.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328186+0000", 2021-03-15T22:33:35.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4825e-05 2021-03-15T22:33:35.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.016 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:35.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328211+0000", 2021-03-15T22:33:35.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015016850999999999 2021-03-15T22:33:35.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:35.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.343228+0000", 2021-03-15T22:33:35.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3612999999999998e-05 2021-03-15T22:33:35.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.018 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:35.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.343261+0000", 2021-03-15T22:33:35.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079578619000000003 2021-03-15T22:33:35.019 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:35.019 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:35.019 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:35.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.020 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.4.0:684 7.1 e816/742 7:8bc62bc6:::smithi07694878-17:20e v 816'433, mlcod=813'429)", 2021-03-15T22:33:35.020 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:46.273595+0000", 2021-03-15T22:33:35.020 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 48.426553366999997, 2021-03-15T22:33:35.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078116977000000004, 2021-03-15T22:33:35.021 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:35.021 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:35.021 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:35.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:35.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273595+0000", 2021-03-15T22:33:35.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:35.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:35.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273595+0000", 2021-03-15T22:33:35.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9749999999999999e-06 2021-03-15T22:33:35.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:35.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273598+0000", 2021-03-15T22:33:35.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8880000000000001e-06 2021-03-15T22:33:35.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:35.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273601+0000", 2021-03-15T22:33:35.026 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1609999999999997e-06 2021-03-15T22:33:35.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:35.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273608+0000", 2021-03-15T22:33:35.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9379999999999997e-06 2021-03-15T22:33:35.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:35.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273614+0000", 2021-03-15T22:33:35.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074469357999999999 2021-03-15T22:33:35.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:35.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.348083+0000", 2021-03-15T22:33:35.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3329999999999999e-05 2021-03-15T22:33:35.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:35.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.348106+0000", 2021-03-15T22:33:35.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003527517 2021-03-15T22:33:35.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:35.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.351634+0000", 2021-03-15T22:33:35.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7810000000000002e-05 2021-03-15T22:33:35.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:35.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.351712+0000", 2021-03-15T22:33:35.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078116977000000004 2021-03-15T22:33:35.033 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:35.033 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:35.033 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:35.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.034 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7697 7.b 7:d10ba14d:::smithi07694878-43:head [read 0~3042298,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e816)", 2021-03-15T22:33:35.034 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:33:35.034 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 48.408964789000002, 2021-03-15T22:33:35.034 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066878664000000004, 2021-03-15T22:33:35.035 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:35.035 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:35.035 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:35.035 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:35.035 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:35.036 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7697 2021-03-15T22:33:35.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.036 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:35.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:35.037 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:33:35.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:35.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:35.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:33:35.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2300000000000002e-06 2021-03-15T22:33:35.038 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:35.039 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291188+0000", 2021-03-15T22:33:35.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9349999999999999e-06 2021-03-15T22:33:35.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:35.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291191+0000", 2021-03-15T22:33:35.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056407205000000002 2021-03-15T22:33:35.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.041 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:35.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347598+0000", 2021-03-15T22:33:35.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0958000000000001e-05 2021-03-15T22:33:35.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:35.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347609+0000", 2021-03-15T22:33:35.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032663129999999999 2021-03-15T22:33:35.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:35.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.350875+0000", 2021-03-15T22:33:35.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5338000000000002e-05 2021-03-15T22:33:35.044 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:35.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.350960+0000", 2021-03-15T22:33:35.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071016849999999999 2021-03-15T22:33:35.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:35.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:35.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:35.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.358062+0000", 2021-03-15T22:33:35.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066878664000000004 2021-03-15T22:33:35.046 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:35.046 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:35.046 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:35.046 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:35.046 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:35.047 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:33:35.047 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:35.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: cluster 2021-03-15T22:33:35.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 2021-03-15T22:33:35.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 2021 2021-03-15T22:33:35.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: - 2021-03-15T22:33:35.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 03 2021-03-15T22:33:35.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: - 2021-03-15T22:33:35.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 15 2021-03-15T22:33:35.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: T 2021-03-15T22:33:35.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 22 2021-03-15T22:33:35.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: : 2021-03-15T22:33:35.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 33 2021-03-15T22:33:35.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: : 2021-03-15T22:33:35.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 33 2021-03-15T22:33:35.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: . 2021-03-15T22:33:35.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 191882 2021-03-15T22:33:35.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: +0000 2021-03-15T22:33:35.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 2021-03-15T22:33:35.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: mgr.y (mgr.14144) 986 : cluster [DBG] pgmap v1438: 148 pgs: 1 peering, 147 active+clean; 587 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.0 MiB/s rd, 4.0 MiB/s wr, 18 op/s; 27 B/s, 0 objects/s recovering 2021-03-15T22:33:35.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: cluster 2021-03-15T22:33:35.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 2021-03-15T22:33:33.302908 2021-03-15T22:33:35.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2194 : cluster [DBG] 2021-03-15T22:33:35.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: osdmap e863: 8 total, 8 up, 8 in 2021-03-15T22:33:35.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: cluster 2021-03-15T22:33:35.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 2021-03-15T22:33: 2021-03-15T22:33:35.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: 33.581393+0000 mon.a (mon.0) 2195 : 2021-03-15T22:33:35.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:34 smithi125 conmon[56739]: cluster [WRN] Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:33:35.071 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:35.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:35.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:35.173 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:35.195 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:35.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:35.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:35.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:35.299 INFO:tasks.rados.rados.0.smithi076.stdout:3245: done (2 left) 2021-03-15T22:33:35.299 INFO:tasks.rados.rados.0.smithi076.stdout:3246: snap_create 2021-03-15T22:33:35.300 INFO:tasks.rados.rados.0.smithi076.stdout:3243: finishing write tid 2 to smithi07694878-31 2021-03-15T22:33:35.300 INFO:tasks.rados.rados.0.smithi076.stdout:3243: finishing write tid 3 to smithi07694878-31 2021-03-15T22:33:35.300 INFO:tasks.rados.rados.0.smithi076.stdout:3243: finishing write tid 4 to smithi07694878-31 2021-03-15T22:33:35.300 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 2411 (ObjNum 1099 snap 380 seq_num 1099) dirty exists 2021-03-15T22:33:35.301 INFO:tasks.rados.rados.0.smithi076.stdout:3243: left oid 31 (ObjNum 1099 snap 380 seq_num 1099) 2021-03-15T22:33:35.301 INFO:tasks.rados.rados.0.smithi076.stdout:3244: finishing rollback tid 0 to smithi07694878-12 2021-03-15T22:33:35.301 INFO:tasks.rados.rados.0.smithi076.stdout:3244: finishing rollback tid 1 to smithi07694878-12 2021-03-15T22:33:35.301 INFO:tasks.rados.rados.0.smithi076.stdout:3244: finishing rollback tid 2 to smithi07694878-12 2021-03-15T22:33:35.301 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 2767 (ObjNum 999 snap 335 seq_num 999) dirty exists 2021-03-15T22:33:35.332 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:35.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:35.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:35.434 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:35.459 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:35.461 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:35.562 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:35.584 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:35.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:35.585 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:35.686 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:35.708 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:35.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:35.710 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:35.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[62156]: cluster 2021-03-15T22:33:35.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[62156]: 2021-03-15T22:33: 2021-03-15T22:33:35.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[62156]: 34.303799+0000 mon.a (mon.0) 2196 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2021-03-15T22:33:35.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:33:35.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[62156]: 03-15T22:33:35.293383+0000 mon.a (mon.0) 2197 : cluster 2021-03-15T22:33:35.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[62156]: [DBG] osdmap e865: 8 total, 8 up, 8 in 2021-03-15T22:33:35.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[56514]: cluster 2021-03-15T22:33:35.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[56514]: 2021-03- 2021-03-15T22:33:35.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[56514]: 15T22:33:34.303799+0000 mon.a (mon.0) 2021-03-15T22:33:35.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[56514]: 2196 : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2021-03-15T22:33:35.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:33:35.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:35 smithi076 conmon[56514]: 15T22:33:35.293383+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2021-03-15T22:33:35.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:35 smithi125 conmon[56739]: cluster 2021-03-15T22:33:34.303799+0000 mon.a (mon.0) 2196 2021-03-15T22:33:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:35 smithi125 conmon[56739]: : cluster [DBG] osdmap e864: 8 total, 8 up, 8 in 2021-03-15T22:33:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:35 smithi125 conmon[56739]: cluster 2021-03-15T22:33:35.293383+0000 mon.a (mon.0) 2197 : cluster [DBG] osdmap e865: 8 total, 8 up, 8 in 2021-03-15T22:33:35.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:35.833 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:35.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:35.834 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:35.935 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:35.959 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:35.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:35.961 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:35.975 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-15T22:33:36.063 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:36.086 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:36.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:36.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:36.189 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:36.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:36.217 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:36.218 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:36.307 INFO:tasks.rados.rados.0.smithi076.stdout:3243: done (2 left) 2021-03-15T22:33:36.307 INFO:tasks.rados.rados.0.smithi076.stdout:3244: done (1 left) 2021-03-15T22:33:36.307 INFO:tasks.rados.rados.0.smithi076.stdout:3246: done (0 left) 2021-03-15T22:33:36.308 INFO:tasks.rados.rados.0.smithi076.stdout:3247: snap_remove snap 365 2021-03-15T22:33:36.314 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:33:36.319 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:36.380 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:36.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:36.381 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:36.432 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:36.484 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:36.509 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:36.510 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:36.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:36 smithi076 conmon[56514]: cluster 2021-03-15T22:33:35.192290+0000 mgr.y (mgr.14144) 987 : cluster 2021-03-15T22:33:36.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:36 smithi076 conmon[56514]: [DBG] pgmap v1441: 148 pgs: 1 peering, 147 active+clean; 592 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 3.0 MiB/s wr, 11 op/s; 27 B/s, 0 objects/s recovering 2021-03-15T22:33:36.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:36 smithi076 conmon[56514]: cluster 2021-03-15T22:33:36.301904+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2021-03-15T22:33:36.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:36 smithi076 conmon[62156]: cluster 2021-03-15T22:33:35.192290+0000 mgr.y (mgr.14144 2021-03-15T22:33:36.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:36 smithi076 conmon[62156]: ) 987 : cluster [DBG] pgmap v1441: 148 pgs: 1 peering, 147 active+clean; 592 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 3.0 MiB/s wr, 11 op/s; 27 B/s, 0 objects/s recovering 2021-03-15T22:33:36.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:36 smithi076 conmon[62156]: cluster 2021-03-15T22:33:36.301904+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2021-03-15T22:33:36.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:36 smithi076 conmon[62156]: 2021-03-15T22:33:36.611 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:36.614 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:36.636 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:36.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:36.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:36.738 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:36.763 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:36.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:36.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:36.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:36 smithi125 conmon[56739]: cluster 2021-03-15T22:33:35.192290+0000 mgr.y (mgr.14144) 2021-03-15T22:33:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:36 smithi125 conmon[56739]: 987 : cluster [DBG] pgmap v1441: 148 pgs: 1 peering, 147 active+clean; 592 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 7.4 MiB/s rd, 3.0 MiB/s wr, 11 op/s; 27 B/s, 0 objects/s recovering 2021-03-15T22:33:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:36 smithi125 conmon[56739]: cluster 2021-03-15T22:33:36.301904+0000 mon.a (mon.0) 2198 : cluster [DBG] osdmap e866: 8 total, 8 up, 8 in 2021-03-15T22:33:36.866 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:36.890 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:36.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:36.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:36.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:37.018 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:37.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:37.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:37.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:37.146 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:37.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:37.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:37.248 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:37.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:37.272 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:37.272 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:37.313 INFO:tasks.rados.rados.0.smithi076.stdout:3247: done (0 left) 2021-03-15T22:33:37.313 INFO:tasks.rados.rados.0.smithi076.stdout:3248: delete oid 33 current snap is 381 2021-03-15T22:33:37.322 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:33:37.323 INFO:tasks.rados.rados.0.smithi076.stdout:3248: done (0 left) 2021-03-15T22:33:37.323 INFO:tasks.rados.rados.0.smithi076.stdout:3249: snap_remove snap 376 2021-03-15T22:33:37.374 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:37.396 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:37.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:37.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:37.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:37.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:37.528 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:37.529 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:37.630 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:37.655 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:37.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:37.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:37.758 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:37.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[62156]: audit 2021-03-15T22:33:36.686420+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? 172.21.15.76:0/2749926115' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:33:37.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[62156]: cluster 2021-03-15T22:33:37.299892+0000 mon.a (mon.0) 2200 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:37.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[62156]: audit 2021-03-15T22:33:37.308852+0000 mon.a (mon.0) 2201 : audit [INF] from='client.? 172.21.15.76:0/2749926115' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:33:37.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[62156]: cluster 2021-03-15T22:33:37.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[62156]: 2021-03-15T22:33:37.308915+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2021-03-15T22:33:37.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: audit 2021-03-15T22:33:37.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: 2021-03-15 2021-03-15T22:33:37.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: T22:33:36.686420+0000 mon.a (mon.0) 2199 : audit [INF] from='client.? 172.21.15.76:0/2749926115' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:33:37.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: cluster 2021-03-15T22:33:37.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: 2021-03-15T22:33:37.299892+0000 mon.a (mon.0) 2200 : cluster 2021-03-15T22:33:37.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:37.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: audit 2021 2021-03-15T22:33:37.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: -03-15T22:33:37.308852+0000 mon.a (mon.0) 2201 : audit [INF] 2021-03-15T22:33:37.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/2749926115' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:33:37.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: cluster 2021-03-15T22:33:37.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: 2021-03-15T22:33:37.308915+0000 mon.a (mon.0) 2202 2021-03-15T22:33:37.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:37 smithi076 conmon[56514]: : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2021-03-15T22:33:37.784 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:37.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:37.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:37.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:37 smithi125 conmon[56739]: audit 2021-03-15T22:33:36.686420+0000 mon.a (mon.0) 2199 : audit [INF] 2021-03-15T22:33:37.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:37 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/2749926115' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:33:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:37 smithi125 conmon[56739]: cluster 2021-03-15T22:33:37.299892+0000 mon.a (mon.0) 2200 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:37 smithi125 conmon[56739]: audit 2021-03-15T22:33:37.308852+0000 mon.a (mon. 2021-03-15T22:33:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:37 smithi125 conmon[56739]: 0) 2201 : audit [INF] from='client.? 172.21.15.76:0/2749926115' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:33:37.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:37 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:33:37.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:37 smithi125 conmon[56739]: 22:33:37.308915+0000 mon.a (mon.0) 2202 : cluster [DBG] osdmap e867: 8 total, 8 up, 8 in 2021-03-15T22:33:37.886 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:37.911 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:37.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:38.013 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:38.036 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:38.038 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:38.139 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:38.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:38.166 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:38.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:38.268 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:38.290 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:38.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:38.328 INFO:tasks.rados.rados.0.smithi076.stdout:3249: done (0 left) 2021-03-15T22:33:38.328 INFO:tasks.rados.rados.0.smithi076.stdout:3250: read oid 9 snap 377 2021-03-15T22:33:38.328 INFO:tasks.rados.rados.0.smithi076.stdout:3250: expect (ObjNum 1092 snap 375 seq_num 1092) 2021-03-15T22:33:38.328 INFO:tasks.rados.rados.0.smithi076.stdout:3251: delete oid 21 current snap is 381 2021-03-15T22:33:38.336 INFO:tasks.rados.rados.0.smithi076.stdout:3251: done (1 left) 2021-03-15T22:33:38.336 INFO:tasks.rados.rados.0.smithi076.stdout:3252: rollback oid 12 current snap is 381 2021-03-15T22:33:38.336 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 12 to 339 2021-03-15T22:33:38.336 INFO:tasks.rados.rados.0.smithi076.stdout:3253: read oid 49 snap -1 2021-03-15T22:33:38.337 INFO:tasks.rados.rados.0.smithi076.stdout:3253: expect (ObjNum 945 snap 318 seq_num 945) 2021-03-15T22:33:38.337 INFO:tasks.rados.rados.0.smithi076.stdout:3254: read oid 7 snap -1 2021-03-15T22:33:38.337 INFO:tasks.rados.rados.0.smithi076.stdout:3254: expect (ObjNum 962 snap 321 seq_num 962) 2021-03-15T22:33:38.337 INFO:tasks.rados.rados.0.smithi076.stdout:3255: read oid 24 snap -1 2021-03-15T22:33:38.338 INFO:tasks.rados.rados.0.smithi076.stdout:3255: expect (ObjNum 1046 snap 351 seq_num 1046) 2021-03-15T22:33:38.338 INFO:tasks.rados.rados.0.smithi076.stdout:3256: read oid 34 snap 367 2021-03-15T22:33:38.338 INFO:tasks.rados.rados.0.smithi076.stdout:3256: expect (ObjNum 1064 snap 361 seq_num 1064) 2021-03-15T22:33:38.338 INFO:tasks.rados.rados.0.smithi076.stdout:3257: read oid 5 snap 345 2021-03-15T22:33:38.339 INFO:tasks.rados.rados.0.smithi076.stdout:3257: expect (ObjNum 1016 snap 341 seq_num 1016) 2021-03-15T22:33:38.339 INFO:tasks.rados.rados.0.smithi076.stdout:3258: read oid 10 snap 360 2021-03-15T22:33:38.339 INFO:tasks.rados.rados.0.smithi076.stdout:3258: expect deleted 2021-03-15T22:33:38.339 INFO:tasks.rados.rados.0.smithi076.stdout:3259: snap_create 2021-03-15T22:33:38.340 INFO:tasks.rados.rados.0.smithi076.stdout:3250: expect (ObjNum 1092 snap 375 seq_num 1092) 2021-03-15T22:33:38.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:38.414 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:38.416 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:38.419 INFO:tasks.rados.rados.0.smithi076.stdout:3255: expect (ObjNum 1046 snap 351 seq_num 1046) 2021-03-15T22:33:38.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:38.545 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:38.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:38.546 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:38.601 INFO:tasks.rados.rados.0.smithi076.stdout:3254: expect (ObjNum 962 snap 321 seq_num 962) 2021-03-15T22:33:38.647 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:38.673 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:38.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:38.675 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:38.747 INFO:tasks.rados.rados.0.smithi076.stdout:3257: expect (ObjNum 1016 snap 341 seq_num 1016) 2021-03-15T22:33:38.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[56514]: cluster 2021-03-15T22:33:37.192790+0000 mgr.y (mgr. 2021-03-15T22:33:38.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[56514]: 14144) 988 : cluster [DBG] pgmap v1444: 148 pgs: 1 peering, 147 active+clean; 582 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-03-15T22:33:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[56514]: cluster 2021-03-15T22:33:37.516663+0000 osd.7 (osd.7) 163 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:33:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[56514]: cluster 2021-03-15T22:33:37.752005+0000 osd.3 (osd.3) 157 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:33:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[56514]: cluster 2021-03-15T22:33:38.325857+0000 mon.a (mon.0) 2203 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2021-03-15T22:33:38.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: cluster 2021-03-15T22:33:38.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: 2021-03-15T22:33:38.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: 2021 2021-03-15T22:33:38.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: - 2021-03-15T22:33:38.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: 03 2021-03-15T22:33:38.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: - 2021-03-15T22:33:38.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: 15 2021-03-15T22:33:38.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: T 2021-03-15T22:33:38.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: 22 2021-03-15T22:33:38.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: : 2021-03-15T22:33:38.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: 33:37.192790+0000 mgr.y (mgr. 2021-03-15T22:33:38.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: 14144) 988 : cluster [DBG] pgmap v1444: 148 pgs: 1 peering, 147 active+clean; 582 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-03-15T22:33:38.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: cluster 2021-03-15T22:33:37.516663+0000 2021-03-15T22:33:38.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: osd.7 (osd.7) 163 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:33:38.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:33:38.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: 03-15T22:33:37.752005+0000 osd.3 ( 2021-03-15T22:33:38.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: osd.3) 157 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:33:38.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: cluster 2021-03-15T22:33:38.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: 2021-03-15T22:33:38.325857 2021-03-15T22:33:38.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2203 : cluster [DBG] 2021-03-15T22:33:38.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:38 smithi076 conmon[62156]: osdmap e868: 8 total, 8 up, 8 in 2021-03-15T22:33:38.775 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:38.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:38 smithi125 conmon[56739]: cluster 2021-03-15T22:33:37.192790+0000 mgr.y (mgr.14144) 988 : cluster 2021-03-15T22:33:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:38 smithi125 conmon[56739]: [DBG] pgmap v1444: 148 pgs: 1 peering, 147 active+clean; 582 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 9.6 MiB/s rd, 1.5 MiB/s wr, 11 op/s 2021-03-15T22:33:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:38 smithi125 conmon[56739]: cluster 2021-03-15T22:33:37.516663+0000 osd.7 (osd.7) 163 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:33:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:38 smithi125 conmon[56739]: cluster 2021-03-15T22:33: 2021-03-15T22:33:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:38 smithi125 conmon[56739]: 37.752005+0000 osd.3 (osd.3) 157 : cluster [DBG] 4.10 scrub ok 2021-03-15T22:33:38.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:38 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:33:38.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:38 smithi125 conmon[56739]: -15T22:33:38.325857+0000 mon.a (mon.0) 2021-03-15T22:33:38.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:38 smithi125 conmon[56739]: 2203 : cluster [DBG] osdmap e868: 8 total, 8 up, 8 in 2021-03-15T22:33:38.798 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:38.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:38.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:38.849 INFO:tasks.thrashosds.thrasher:in_osds: [1, 2, 4, 7, 0, 3, 5, 6] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:33:38.850 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:33:38.850 INFO:tasks.thrashosds.thrasher:Removing osd 2, in_osds are: [1, 2, 4, 7, 0, 3, 5, 6] 2021-03-15T22:33:38.850 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd out 2 2021-03-15T22:33:38.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:38.927 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:38.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:38.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:39.030 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:39.055 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:39.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:39.057 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:39.057 INFO:tasks.rados.rados.0.smithi076.stdout:3256: expect (ObjNum 1064 snap 361 seq_num 1064) 2021-03-15T22:33:39.157 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:39.180 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:39.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:39.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:39.190 INFO:tasks.rados.rados.0.smithi076.stdout:3252: finishing rollback tid 0 to smithi07694878-12 2021-03-15T22:33:39.191 INFO:tasks.rados.rados.0.smithi076.stdout:3252: finishing rollback tid 1 to smithi07694878-12 2021-03-15T22:33:39.191 INFO:tasks.rados.rados.0.smithi076.stdout:3252: finishing rollback tid 2 to smithi07694878-12 2021-03-15T22:33:39.191 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 2770 (ObjNum 999 snap 335 seq_num 999) dirty exists 2021-03-15T22:33:39.191 INFO:tasks.rados.rados.0.smithi076.stdout:3253: expect (ObjNum 945 snap 318 seq_num 945) 2021-03-15T22:33:39.268 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:33:39.269 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:33:39.269 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:33:39.269 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:33:39.283 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:39.308 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:39.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:39.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:39.411 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:39.423 INFO:tasks.rados.rados.0.smithi076.stdout:3250: done (8 left) 2021-03-15T22:33:39.423 INFO:tasks.rados.rados.0.smithi076.stdout:3252: done (7 left) 2021-03-15T22:33:39.423 INFO:tasks.rados.rados.0.smithi076.stdout:3253: done (6 left) 2021-03-15T22:33:39.424 INFO:tasks.rados.rados.0.smithi076.stdout:3254: done (5 left) 2021-03-15T22:33:39.424 INFO:tasks.rados.rados.0.smithi076.stdout:3255: done (4 left) 2021-03-15T22:33:39.424 INFO:tasks.rados.rados.0.smithi076.stdout:3256: done (3 left) 2021-03-15T22:33:39.424 INFO:tasks.rados.rados.0.smithi076.stdout:3257: done (2 left) 2021-03-15T22:33:39.425 INFO:tasks.rados.rados.0.smithi076.stdout:3258: done (1 left) 2021-03-15T22:33:39.425 INFO:tasks.rados.rados.0.smithi076.stdout:3259: done (0 left) 2021-03-15T22:33:39.425 INFO:tasks.rados.rados.0.smithi076.stdout:3260: read oid 31 snap 353 2021-03-15T22:33:39.425 INFO:tasks.rados.rados.0.smithi076.stdout:3260: expect (ObjNum 1020 snap 341 seq_num 1020) 2021-03-15T22:33:39.425 INFO:tasks.rados.rados.0.smithi076.stdout:3261: write (excl) oid 42 current snap is 382 2021-03-15T22:33:39.426 INFO:tasks.rados.rados.0.smithi076.stdout:3261: seq_num 1102 ranges {521314=479230,1786686=669381} 2021-03-15T22:33:39.428 INFO:tasks.rados.rados.0.smithi076.stdout:3261: writing smithi07694878-42 from 521314 to 1000544 tid 1 2021-03-15T22:33:39.437 INFO:tasks.rados.rados.0.smithi076.stdout:3261: writing smithi07694878-42 from 1786686 to 2456067 tid 2 2021-03-15T22:33:39.437 INFO:tasks.rados.rados.0.smithi076.stdout:3262: read oid 1 snap 297 2021-03-15T22:33:39.438 INFO:tasks.rados.rados.0.smithi076.stdout:3262: expect (ObjNum 803 snap 268 seq_num 803) 2021-03-15T22:33:39.438 INFO:tasks.rados.rados.0.smithi076.stdout:3263: write (excl) oid 21 current snap is 382 2021-03-15T22:33:39.438 INFO:tasks.rados.rados.0.smithi076.stdout:3260: expect (ObjNum 1020 snap 341 seq_num 1020) 2021-03-15T22:33:39.439 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:39.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:39.441 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:39.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[56514]: cluster 2021-03-15T22:33:39.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[56514]: 2021-03-15T22:33:37.774075+0000 osd.5 (osd.5) 42 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:33:39.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[56514]: cluster 2021-03-15T22:33:37.949870+0000 osd.2 (osd.2) 39 : cluster [DBG] 5.d scrub ok 2021-03-15T22:33:39.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[56514]: cluster 2021-03-15T 2021-03-15T22:33:39.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[56514]: 22:33:38.133613+0000 osd.1 (osd.1) 168 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:33:39.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[56514]: cluster 2021-03-15T22:33 2021-03-15T22:33:39.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[56514]: :38.907003+0000 osd.2 (osd.2) 2021-03-15T22:33:39.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[56514]: 40 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:33:39.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[62156]: cluster 2021-03-15T22:33:39.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[62156]: 2021-03-15T22:33:39.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[62156]: 2021 2021-03-15T22:33:39.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[62156]: - 2021-03-15T22:33:39.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[62156]: 03-15T22:33 2021-03-15T22:33:39.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[62156]: :37.774075+0000 osd.5 (osd.5) 42 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:33:39.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[62156]: cluster 2021-03-15T22:33 2021-03-15T22:33:39.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[62156]: :37.949870+0000 osd.2 (osd.2) 39 : cluster [DBG] 5.d scrub ok 2021-03-15T22:33:39.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:33:39.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[62156]: 33:38.133613+0000 osd.1 (osd.1) 168 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:33:39.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:39 smithi076 conmon[62156]: cluster 2021-03-15T22:33:38.907003+0000 osd.2 (osd.2) 40 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:33:39.542 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:39.566 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:39.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:39.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:39.663 INFO:tasks.rados.rados.0.smithi076.stdout:3261: finishing write tid 1 to smithi07694878-42 2021-03-15T22:33:39.663 INFO:tasks.rados.rados.0.smithi076.stdout:3261: finishing write tid 2 to smithi07694878-42 2021-03-15T22:33:39.663 INFO:tasks.rados.rados.0.smithi076.stdout:3261: finishing write tid 3 to smithi07694878-42 2021-03-15T22:33:39.664 INFO:tasks.rados.rados.0.smithi076.stdout:3261: finishing write tid 4 to smithi07694878-42 2021-03-15T22:33:39.664 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2889 (ObjNum 1102 snap 382 seq_num 1102) dirty exists 2021-03-15T22:33:39.664 INFO:tasks.rados.rados.0.smithi076.stdout:3261: left oid 42 (ObjNum 1102 snap 382 seq_num 1102) 2021-03-15T22:33:39.664 INFO:tasks.rados.rados.0.smithi076.stdout:3262: expect (ObjNum 803 snap 268 seq_num 803) 2021-03-15T22:33:39.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:39.692 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:39.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:39.693 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:39.779 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:39.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:39 smithi125 conmon[56739]: cluster 2021-03-15T22:33:37.774075+0000 osd.5 2021-03-15T22:33:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:39 smithi125 conmon[56739]: (osd.5) 42 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:33:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:39 smithi125 conmon[56739]: cluster 2021-03-15T22:33:37.949870+0000 osd.2 (osd. 2021-03-15T22:33:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:39 smithi125 conmon[56739]: 2) 39 : cluster [DBG] 5.d scrub ok 2021-03-15T22:33:39.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:39 smithi125 conmon[56739]: cluster 2021-03-15T22:33:38.133613+0000 osd.1 2021-03-15T22:33:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:39 smithi125 conmon[56739]: (osd.1) 168 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:33:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:39 smithi125 conmon[56739]: cluster 2021-03-15T22:33:38.907003 2021-03-15T22:33:39.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:39 smithi125 conmon[56739]: +0000 osd.2 (osd.2) 40 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:33:39.794 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:39.819 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:39.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:39.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:39.894 INFO:tasks.rados.rados.0.smithi076.stdout:3263: seq_num 1103 ranges {538121=670832,1961521=413039} 2021-03-15T22:33:39.900 INFO:tasks.rados.rados.0.smithi076.stdout:3263: writing smithi07694878-21 from 538121 to 1208953 tid 1 2021-03-15T22:33:39.911 INFO:tasks.rados.rados.0.smithi076.stdout:3263: writing smithi07694878-21 from 1961521 to 2374560 tid 2 2021-03-15T22:33:39.912 INFO:tasks.rados.rados.0.smithi076.stdout:3260: done (3 left) 2021-03-15T22:33:39.913 INFO:tasks.rados.rados.0.smithi076.stdout:3261: done (2 left) 2021-03-15T22:33:39.913 INFO:tasks.rados.rados.0.smithi076.stdout:3262: done (1 left) 2021-03-15T22:33:39.913 INFO:tasks.rados.rados.0.smithi076.stdout:3264: snap_create 2021-03-15T22:33:39.914 INFO:tasks.rados.rados.0.smithi076.stdout:3263: finishing write tid 1 to smithi07694878-21 2021-03-15T22:33:39.915 INFO:tasks.rados.rados.0.smithi076.stdout:3263: finishing write tid 2 to smithi07694878-21 2021-03-15T22:33:39.915 INFO:tasks.rados.rados.0.smithi076.stdout:3263: finishing write tid 3 to smithi07694878-21 2021-03-15T22:33:39.916 INFO:tasks.rados.rados.0.smithi076.stdout:3263: finishing write tid 4 to smithi07694878-21 2021-03-15T22:33:39.918 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 2892 (ObjNum 1103 snap 382 seq_num 1103) dirty exists 2021-03-15T22:33:39.919 INFO:tasks.rados.rados.0.smithi076.stdout:3263: left oid 21 (ObjNum 1103 snap 382 seq_num 1103) 2021-03-15T22:33:39.922 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:39.946 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:39.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:39.948 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:40.049 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:40.068 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:33:40.068 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:33:40.069 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:33:40.069 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:33:40.070 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:33:40.070 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:33:40.071 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:33:40.071 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:33:40.072 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:33:40.072 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:33:40.072 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:33:40.073 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:33:40.073 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:33:40.073 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:33:40.073 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:33:40.074 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:33:40.075 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:40.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:40.077 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:40.145 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-15T22:33:40.166 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:33:40.178 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:40.213 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:40.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:40.215 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:40.299 INFO:tasks.rados.rados.0.smithi076.stdout:3263: done (1 left) 2021-03-15T22:33:40.300 INFO:tasks.rados.rados.0.smithi076.stdout:3264: done (0 left) 2021-03-15T22:33:40.300 INFO:tasks.rados.rados.0.smithi076.stdout:3265: write (excl) oid 47 current snap is 383 2021-03-15T22:33:40.301 INFO:tasks.rados.rados.0.smithi076.stdout:3265: seq_num 1104 ranges {554928=746081,2020003=534925} 2021-03-15T22:33:40.306 INFO:tasks.rados.rados.0.smithi076.stdout:3265: writing smithi07694878-47 from 554928 to 1301009 tid 1 2021-03-15T22:33:40.314 INFO:tasks.rados.rados.0.smithi076.stdout:3265: writing smithi07694878-47 from 2020003 to 2554928 tid 2 2021-03-15T22:33:40.314 INFO:tasks.rados.rados.0.smithi076.stdout:3266: read oid 31 snap -1 2021-03-15T22:33:40.314 INFO:tasks.rados.rados.0.smithi076.stdout:3266: expect (ObjNum 1099 snap 380 seq_num 1099) 2021-03-15T22:33:40.315 INFO:tasks.rados.rados.0.smithi076.stdout:3267: snap_create 2021-03-15T22:33:40.316 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:40.320 INFO:tasks.rados.rados.0.smithi076.stdout:3265: finishing write tid 1 to smithi07694878-47 2021-03-15T22:33:40.325 INFO:tasks.rados.rados.0.smithi076.stdout:3265: finishing write tid 2 to smithi07694878-47 2021-03-15T22:33:40.328 INFO:tasks.rados.rados.0.smithi076.stdout:3265: finishing write tid 3 to smithi07694878-47 2021-03-15T22:33:40.329 INFO:tasks.rados.rados.0.smithi076.stdout:3265: finishing write tid 4 to smithi07694878-47 2021-03-15T22:33:40.329 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 2095 (ObjNum 1104 snap 383 seq_num 1104) dirty exists 2021-03-15T22:33:40.329 INFO:tasks.rados.rados.0.smithi076.stdout:3265: left oid 47 (ObjNum 1104 snap 383 seq_num 1104) 2021-03-15T22:33:40.330 INFO:tasks.rados.rados.0.smithi076.stdout:3266: expect (ObjNum 1099 snap 380 seq_num 1099) 2021-03-15T22:33:40.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: cluster 2021-03-15T22:33:38.759389+0000 2021-03-15T22:33:40.343 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: osd.5 (osd.5) 43 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:33:40.344 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:33:40.344 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: T22:33:38.911090+0000 osd.6 (osd.6) 126 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:40.344 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:40.344 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: -03-15T22:33:38.912488+0000 2021-03-15T22:33:40.344 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: osd.6 (osd.6) 127 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:40.345 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: 2021-03-15T22:33:40.345 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: cluster 2021-03-15T22:33:39 2021-03-15T22:33:40.345 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: .193121+0000 mgr.y (mgr.14144) 989 : 2021-03-15T22:33:40.345 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: cluster [DBG] pgmap v1447: 148 pgs: 148 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 303 KiB/s wr, 6 op/s 2021-03-15T22:33:40.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:33:40.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: 22:33:39.328615+0000 mon.a (mon 2021-03-15T22:33:40.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: .0) 2204 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-15T22:33:40.346 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: cluster 2021-03-15T22:33:40.347 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: 2021-03-15T22:33:39. 2021-03-15T22:33:40.347 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: 339219+0000 mon.a (mon.0) 2205 : cluster 2021-03-15T22:33:40.347 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: [DBG] osdmap e869: 8 total, 8 up, 8 in 2021-03-15T22:33:40.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:33:40.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: T22:33:39.355296+0000 osd.0 ( 2021-03-15T22:33:40.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: osd.0) 141 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:33:40.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: 2021-03-15T22:33:40.348 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:33:40.349 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: 33:39.564459+0000 osd.4 (osd.4 2021-03-15T22:33:40.349 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: ) 181 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:33:40.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:40.363 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:40.363 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:40.464 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:40.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:40.494 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:40.494 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:40.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:33:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: cluster 2021-03-15T22:33:38.759389+0000 osd.5 (osd.5) 43 : cluster [DBG] 2021-03-15T22:33:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: 4.8 scrub ok 2021-03-15T22:33:40.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: cluster 2021-03-15T22:33:38.911090+0000 osd.6 (osd.6) 126 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: cluster 2021-03-15T22:33:38.912488+0000 2021-03-15T22:33:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: osd.6 (osd.6) 127 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: cluster 2021-03-15T22:33:40.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: 2021-03-15T22:33:39.193121+0000 mgr.y (mgr 2021-03-15T22:33:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: .14144) 989 : cluster [DBG] pgmap v1447: 148 pgs: 148 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 303 KiB/s wr, 6 op/s 2021-03-15T22:33:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:33:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: -03-15T22:33:39.328615 2021-03-15T22:33:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2204 : cluster 2021-03-15T22:33:40.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-15T22:33:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:33:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: 22:33:39.339219+0000 mon.a (mon. 2021-03-15T22:33:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: 0) 2205 : cluster [DBG] osdmap e869: 8 total, 8 up, 8 in 2021-03-15T22:33:40.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:33:40.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: 03-15T22:33:39.355296+0000 2021-03-15T22:33:40.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: osd.0 (osd.0) 141 : cluster [DBG] 2021-03-15T22:33:40.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: 7.9 scrub ok 2021-03-15T22:33:40.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: cluster 2021-03-15T22:33 2021-03-15T22:33:40.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: :39.564459+0000 osd.4 (osd.4) 2021-03-15T22:33:40.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: 181 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:33:40.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:33:40.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: T22:33:39.884328+0000 mon.a 2021-03-15T22:33:40.542 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: (mon.0) 2206 : audit [DBG] from='client.? 172.21.15.76:0/888364064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:40.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: 2021-03-15T22:33:40.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: cluster 2021-03-15T22:33:40.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: 2021-03-15T22:33:40.294530+0000 2021-03-15T22:33:40.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: mon.a (mon.0) 2207 : cluster [DBG] 2021-03-15T22:33:40.543 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:40 smithi125 conmon[56739]: osdmap e870: 8 total, 8 up, 8 in 2021-03-15T22:33:40.595 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:40.619 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:40.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:40.620 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:40.645 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: audit 2021-03-15T22:33:40.645 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: 2021-03-15T22:33:39 2021-03-15T22:33:40.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: .884328+0000 mon.a (mon.0) 2206 2021-03-15T22:33:40.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: : audit [DBG] from='client.? 172.21.15.76:0/888364064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:40.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:33:40.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: 15T22:33:40.294530+0000 2021-03-15T22:33:40.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: mon.a (mon.0) 2207 : cluster [DBG] 2021-03-15T22:33:40.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[62156]: osdmap e870: 8 total, 8 up, 8 in 2021-03-15T22:33:40.647 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[56514]: cluster 2021-03-15T22:33:38.759389+0000 osd.5 (osd.5) 43 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:33:40.647 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[56514]: cluster 2021-03-15T22:33:38.911090+0000 osd.6 (osd.6) 126 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:40.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[56514]: cluster 2021-03-15T22:33:38.912488+0000 osd.6 (osd.6) 127 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:40.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[56514]: cluster 2021-03-15T22:33:39.193121+0000 mgr.y (mgr.14144) 989 : cluster [DBG] pgmap v1447: 148 pgs: 148 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 5.2 MiB/s rd, 303 KiB/s wr, 6 op/s 2021-03-15T22:33:40.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[56514]: cluster 2021-03-15T22:33:39.328615+0000 mon.a (mon.0) 2204 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2021-03-15T22:33:40.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[56514]: cluster 2021-03-15T22:33:39.339219+0000 mon.a (mon.0) 2205 : cluster 2021-03-15T22:33:40.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[56514]: [DBG] osdmap e869: 8 total, 8 up, 8 in 2021-03-15T22:33:40.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[56514]: cluster 2021-03-15T22:33:39.355296+0000 osd.0 (osd.0) 141 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:33:40.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[56514]: cluster 2021-03-15T22:33:39.564459+0000 osd.4 (osd.4) 181 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:33:40.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[56514]: audit 2021-03-15T22:33:39.884328+0000 mon.a (mon.0) 2206 : audit [DBG] from='client.? 172.21.15.76:0/888364064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:40.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:40 smithi076 conmon[56514]: cluster 2021-03-15T22:33:40.294530+0000 mon.a (mon.0) 2207 : cluster [DBG] osdmap e870: 8 total, 8 up, 8 in 2021-03-15T22:33:40.721 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:40.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:40.746 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:40.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:40.848 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:40.870 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:40.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:40.872 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:40.973 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:40.998 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:40.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:40.999 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:41.100 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:41.106 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:41.106 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:41.126 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:41.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:41.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:41.229 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:41.254 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:41.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:41.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:41.309 INFO:tasks.rados.rados.0.smithi076.stdout:3265: done (2 left) 2021-03-15T22:33:41.309 INFO:tasks.rados.rados.0.smithi076.stdout:3266: done (1 left) 2021-03-15T22:33:41.310 INFO:tasks.rados.rados.0.smithi076.stdout:3267: done (0 left) 2021-03-15T22:33:41.310 INFO:tasks.rados.rados.0.smithi076.stdout:3268: read oid 44 snap -1 2021-03-15T22:33:41.310 INFO:tasks.rados.rados.0.smithi076.stdout:3268: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:33:41.310 INFO:tasks.rados.rados.0.smithi076.stdout:3269: delete oid 23 current snap is 384 2021-03-15T22:33:41.313 INFO:tasks.rados.rados.0.smithi076.stdout:3269: done (1 left) 2021-03-15T22:33:41.314 INFO:tasks.rados.rados.0.smithi076.stdout:3270: snap_create 2021-03-15T22:33:41.319 INFO:tasks.rados.rados.0.smithi076.stdout:3268: expect (ObjNum 829 snap 275 seq_num 829) 2021-03-15T22:33:41.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:41.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:41.383 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:41.384 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:41.485 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:41.512 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:41.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:41.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:41.820 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: cluster 2021-03-15T22:33:39.756573+0000 osd.5 (osd.5) 44 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:33:41.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: audit 2021-03-15T22:33:40.193107+0000 mgr.y (mgr.14144) 990 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:41.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: cluster 2021-03-15T22:33:40.214698+0000 osd.1 (osd.1) 2021-03-15T22:33:41.821 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: 169 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:33:41.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: cluster 2021-03-15T22:33:41.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: 2021-03-15T22:33:40.563137+0000 osd.4 (osd 2021-03-15T22:33:41.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: .4) 182 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:33:41.822 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:33:41.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: 15T22:33:40.791524+0000 osd.3 ( 2021-03-15T22:33:41.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: osd.3) 158 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:33:41.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:33:41.823 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: 03-15T22:33:41.303582+0000 2021-03-15T22:33:41.824 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: mon.a (mon.0) 2208 : cluster [DBG] 2021-03-15T22:33:41.824 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:41 smithi125 conmon[56739]: osdmap e871: 8 total, 8 up, 8 in 2021-03-15T22:33:41.825 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[56514]: cluster 2021-03-15T22:33:39.756573+0000 osd.5 (osd.5) 44 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:33:41.826 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[56514]: audit 2021-03-15T22:33:40.193107+0000 mgr.y (mgr.14144) 990 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:41.826 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[56514]: cluster 2021-03-15T22:33:40.214698+0000 osd.1 (osd.1) 169 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:33:41.826 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[56514]: cluster 2021-03-15T22:33:40.563137+0000 osd.4 (osd.4) 182 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:33:41.827 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[56514]: cluster 2021-03-15T22:33:40.791524+0000 osd.3 (osd.3) 158 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:33:41.827 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[56514]: cluster 2021-03-15T22:33:41.303582+0000 mon.a (mon.0) 2208 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2021-03-15T22:33:41.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: cluster 2021-03-15T22:33:39.756573+0000 osd.5 2021-03-15T22:33:41.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: (osd.5) 44 : cluster [DBG] 3.1d scrub ok 2021-03-15T22:33:41.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: audit 2021-03-15T22:33:40.193107+0000 2021-03-15T22:33:41.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: mgr.y (mgr.14144) 990 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:41.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:41.829 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: -03-15T22:33:40. 2021-03-15T22:33:41.829 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: 214698+0000 osd.1 (osd.1) 2021-03-15T22:33:41.829 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: 169 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:33:41.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:41.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: -03-15T22:33: 2021-03-15T22:33:41.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: 40.563137+0000 osd.4 (osd. 2021-03-15T22:33:41.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: 4) 182 : cluster [DBG] 2021-03-15T22:33:41.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: 7.1 scrub ok 2021-03-15T22:33:41.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: cluster 2021-03-15T22:33:41.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: 2021-03-15T22:33 2021-03-15T22:33:41.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: :40.791524+0000 osd.3 (osd 2021-03-15T22:33:41.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: .3) 158 : cluster [DBG] 2021-03-15T22:33:41.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: 3.8 scrub ok 2021-03-15T22:33:41.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:33:41.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: 03-15T22:33:41 2021-03-15T22:33:41.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: .303582+0000 mon.a (mon.0 2021-03-15T22:33:41.834 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: ) 2208 : cluster [DBG] osdmap e871: 8 total, 8 up, 8 in 2021-03-15T22:33:41.834 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:41 smithi076 conmon[62156]: 2021-03-15T22:33:41.851 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:41.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:41.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:41.954 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:41.978 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:41.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:41.980 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:42.081 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:42.106 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:42.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:42.108 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:42.209 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:42.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:42.233 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:42.233 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:42.309 INFO:tasks.rados.rados.0.smithi076.stdout:3268: done (1 left) 2021-03-15T22:33:42.310 INFO:tasks.rados.rados.0.smithi076.stdout:3270: done (0 left) 2021-03-15T22:33:42.310 INFO:tasks.rados.rados.0.smithi076.stdout:3271: delete oid 28 current snap is 385 2021-03-15T22:33:42.315 INFO:tasks.rados.rados.0.smithi076.stdout:3271: done (0 left) 2021-03-15T22:33:42.315 INFO:tasks.rados.rados.0.smithi076.stdout:3272: read oid 6 snap -1 2021-03-15T22:33:42.315 INFO:tasks.rados.rados.0.smithi076.stdout:3272: expect deleted 2021-03-15T22:33:42.316 INFO:tasks.rados.rados.0.smithi076.stdout:3273: copy_from oid 7 from oid 40 current snap is 385 2021-03-15T22:33:42.316 INFO:tasks.rados.rados.0.smithi076.stdout:3274: write (excl) oid 3 current snap is 385 2021-03-15T22:33:42.316 INFO:tasks.rados.rados.0.smithi076.stdout:3274: seq_num 1107 ranges {605349=571828,1679096=420377} 2021-03-15T22:33:42.321 INFO:tasks.rados.rados.0.smithi076.stdout:3274: writing smithi07694878-3 from 605349 to 1177177 tid 1 2021-03-15T22:33:42.327 INFO:tasks.rados.rados.0.smithi076.stdout:3274: writing smithi07694878-3 from 1679096 to 2099473 tid 2 2021-03-15T22:33:42.328 INFO:tasks.rados.rados.0.smithi076.stdout:3275: rollback oid 8 current snap is 385 2021-03-15T22:33:42.328 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 362 2021-03-15T22:33:42.328 INFO:tasks.rados.rados.0.smithi076.stdout:3272: done (3 left) 2021-03-15T22:33:42.328 INFO:tasks.rados.rados.0.smithi076.stdout:3276: write oid 20 current snap is 385 2021-03-15T22:33:42.328 INFO:tasks.rados.rados.0.smithi076.stdout:3276: seq_num 1108 ranges {622156=647077,1737578=680388} 2021-03-15T22:33:42.334 INFO:tasks.rados.rados.0.smithi076.stdout:3276: writing smithi07694878-20 from 622156 to 1269233 tid 1 2021-03-15T22:33:42.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:42.341 INFO:tasks.rados.rados.0.smithi076.stdout:3276: writing smithi07694878-20 from 1737578 to 2417966 tid 2 2021-03-15T22:33:42.342 INFO:tasks.rados.rados.0.smithi076.stdout:3277: snap_create 2021-03-15T22:33:42.342 INFO:tasks.rados.rados.0.smithi076.stdout:3274: finishing write tid 1 to smithi07694878-3 2021-03-15T22:33:42.342 INFO:tasks.rados.rados.0.smithi076.stdout:3275: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:33:42.342 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1255 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:33:42.343 INFO:tasks.rados.rados.0.smithi076.stdout:3273: finishing copy_from to smithi07694878-7 2021-03-15T22:33:42.343 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2302 (ObjNum 999 snap 335 seq_num 999) dirty exists 2021-03-15T22:33:42.343 INFO:tasks.rados.rados.0.smithi076.stdout:3274: finishing write tid 2 to smithi07694878-3 2021-03-15T22:33:42.343 INFO:tasks.rados.rados.0.smithi076.stdout:3274: finishing write tid 3 to smithi07694878-3 2021-03-15T22:33:42.343 INFO:tasks.rados.rados.0.smithi076.stdout:3274: finishing write tid 4 to smithi07694878-3 2021-03-15T22:33:42.344 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 2658 (ObjNum 1107 snap 385 seq_num 1107) dirty exists 2021-03-15T22:33:42.344 INFO:tasks.rados.rados.0.smithi076.stdout:3274: left oid 3 (ObjNum 1107 snap 385 seq_num 1107) 2021-03-15T22:33:42.344 INFO:tasks.rados.rados.0.smithi076.stdout:3273: finishing copy_from racing read to smithi07694878-7 2021-03-15T22:33:42.344 INFO:tasks.rados.rados.0.smithi076.stdout:3276: finishing write tid 1 to smithi07694878-20 2021-03-15T22:33:42.345 INFO:tasks.rados.rados.0.smithi076.stdout:3276: finishing write tid 2 to smithi07694878-20 2021-03-15T22:33:42.345 INFO:tasks.rados.rados.0.smithi076.stdout:3276: finishing write tid 3 to smithi07694878-20 2021-03-15T22:33:42.346 INFO:tasks.rados.rados.0.smithi076.stdout:3276: finishing write tid 4 to smithi07694878-20 2021-03-15T22:33:42.346 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 2407 (ObjNum 1108 snap 385 seq_num 1108) dirty exists 2021-03-15T22:33:42.346 INFO:tasks.rados.rados.0.smithi076.stdout:3276: left oid 20 (ObjNum 1108 snap 385 seq_num 1108) 2021-03-15T22:33:42.358 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:42.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:42.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:42.461 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:42.483 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:42.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:42.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:42.586 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:42.607 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:42.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:42.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:42.710 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:42.735 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:42.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:42.736 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:42.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:42 smithi076 conmon[56514]: cluster 2021-03-15T22:33:40.729834+0000 osd.5 ( 2021-03-15T22:33:42.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:42 smithi076 conmon[56514]: osd.5) 45 : cluster [DBG] 6.d scrub ok 2021-03-15T22:33:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:42 smithi076 conmon[56514]: cluster 2021-03-15T22:33:41.193948+0000 mgr.y (mgr.14144) 991 : cluster [DBG] pgmap v1450: 148 pgs: 148 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 287 KiB/s wr, 16 op/s 2021-03-15T22:33:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:42 smithi076 conmon[56514]: cluster 2021-03-15T22:33:42.307195+0000 mon.a (mon.0) 2209 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2021-03-15T22:33:42.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:42 smithi076 conmon[62156]: cluster 2021-03-15T22:33: 2021-03-15T22:33:42.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:42 smithi076 conmon[62156]: 40.729834+0000 osd.5 (osd.5) 45 : cluster [DBG] 6.d scrub ok 2021-03-15T22:33:42.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:42 smithi076 conmon[62156]: cluster 2021-03-15T22:33:41.193948+0000 mgr.y (mgr.14144) 991 : cluster [DBG] pgmap v1450: 148 pgs: 148 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 287 KiB/s wr, 16 op/s 2021-03-15T22:33:42.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:42 smithi076 conmon[62156]: cluster 2021-03-15T22:33:42.307195+0000 mon.a (mon.0) 2209 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2021-03-15T22:33:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:42 smithi125 conmon[56739]: cluster 2021-03-15T22:33:40.729834+0000 osd.5 (osd.5) 45 : 2021-03-15T22:33:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:42 smithi125 conmon[56739]: cluster [DBG] 6.d scrub ok 2021-03-15T22:33:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:42 smithi125 conmon[56739]: cluster 2021-03-15T22:33:41.193948+0000 mgr.y (mgr.14144) 991 : cluster [DBG] pgmap v1450: 148 pgs: 148 active+clean; 581 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 18 MiB/s rd, 287 KiB/s wr, 16 op/s 2021-03-15T22:33:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:42 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:33:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:42 smithi125 conmon[56739]: T22:33:42.307195+0000 mon.a ( 2021-03-15T22:33:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:42 smithi125 conmon[56739]: mon.0) 2209 : cluster [DBG] osdmap e872: 8 total, 8 up, 8 in 2021-03-15T22:33:42.837 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:42.860 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:42.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:42.862 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:42.963 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:42.985 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:42.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:42.987 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:43.088 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:43.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:43.110 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:43.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:43.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:33:43.212 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:43.239 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:43.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:43.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:43.315 INFO:tasks.rados.rados.0.smithi076.stdout:3273: done (4 left) 2021-03-15T22:33:43.315 INFO:tasks.rados.rados.0.smithi076.stdout:3274: done (3 left) 2021-03-15T22:33:43.315 INFO:tasks.rados.rados.0.smithi076.stdout:3275: done (2 left) 2021-03-15T22:33:43.316 INFO:tasks.rados.rados.0.smithi076.stdout:3276: done (1 left) 2021-03-15T22:33:43.316 INFO:tasks.rados.rados.0.smithi076.stdout:3277: done (0 left) 2021-03-15T22:33:43.316 INFO:tasks.rados.rados.0.smithi076.stdout:3278: read oid 15 snap 352 2021-03-15T22:33:43.316 INFO:tasks.rados.rados.0.smithi076.stdout:3278: expect (ObjNum 971 snap 322 seq_num 971) 2021-03-15T22:33:43.317 INFO:tasks.rados.rados.0.smithi076.stdout:3279: write oid 4 current snap is 386 2021-03-15T22:33:43.317 INFO:tasks.rados.rados.0.smithi076.stdout:3279: seq_num 1109 ranges {638963=722326,2079707=424046} 2021-03-15T22:33:43.324 INFO:tasks.rados.rados.0.smithi076.stdout:3279: writing smithi07694878-4 from 638963 to 1361289 tid 1 2021-03-15T22:33:43.331 INFO:tasks.rados.rados.0.smithi076.stdout:3279: writing smithi07694878-4 from 2079707 to 2503753 tid 2 2021-03-15T22:33:43.332 INFO:tasks.rados.rados.0.smithi076.stdout:3280: rollback oid 33 current snap is 386 2021-03-15T22:33:43.332 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 33 to 378 2021-03-15T22:33:43.332 INFO:tasks.rados.rados.0.smithi076.stdout:3278: expect (ObjNum 971 snap 322 seq_num 971) 2021-03-15T22:33:43.342 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:43.368 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:43.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:43.370 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:43.425 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:43 smithi076 conmon[56514]: cluster 2021-03-15T22:33:42.894495+0000 osd.2 ( 2021-03-15T22:33:43.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:43 smithi076 conmon[56514]: osd.2) 41 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:33:43.426 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:43 smithi076 conmon[56514]: cluster 2021-03-15T22:33:43.308978+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2021-03-15T22:33:43.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:43 smithi076 conmon[62156]: cluster 2021-03-15T22:33:42.894495+0000 2021-03-15T22:33:43.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:43 smithi076 conmon[62156]: osd.2 (osd.2) 41 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:33:43.427 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:43 smithi076 conmon[62156]: cluster 2021-03-15T22:33:43.308978+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2021-03-15T22:33:43.471 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:43.498 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:43.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:43.500 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:43.531 INFO:tasks.rados.rados.0.smithi076.stdout:3279: finishing write tid 1 to smithi07694878-4 2021-03-15T22:33:43.531 INFO:tasks.rados.rados.0.smithi076.stdout:3280: finishing rollback tid 1 to smithi07694878-33 2021-03-15T22:33:43.531 INFO:tasks.rados.rados.0.smithi076.stdout:3280: finishing rollback tid 2 to smithi07694878-33 2021-03-15T22:33:43.532 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 2409 (ObjNum 1049 snap 353 seq_num 1049) dirty exists 2021-03-15T22:33:43.532 INFO:tasks.rados.rados.0.smithi076.stdout:3279: finishing write tid 2 to smithi07694878-4 2021-03-15T22:33:43.532 INFO:tasks.rados.rados.0.smithi076.stdout:3279: finishing write tid 3 to smithi07694878-4 2021-03-15T22:33:43.532 INFO:tasks.rados.rados.0.smithi076.stdout:3278: done (2 left) 2021-03-15T22:33:43.533 INFO:tasks.rados.rados.0.smithi076.stdout:3280: done (1 left) 2021-03-15T22:33:43.533 INFO:tasks.rados.rados.0.smithi076.stdout:3281: read oid 37 snap -1 2021-03-15T22:33:43.533 INFO:tasks.rados.rados.0.smithi076.stdout:3281: expect deleted 2021-03-15T22:33:43.533 INFO:tasks.rados.rados.0.smithi076.stdout:3282: copy_from oid 25 from oid 41 current snap is 386 2021-03-15T22:33:43.533 INFO:tasks.rados.rados.0.smithi076.stdout:3279: finishing write tid 4 to smithi07694878-4 2021-03-15T22:33:43.534 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 2000 (ObjNum 1109 snap 386 seq_num 1109) dirty exists 2021-03-15T22:33:43.534 INFO:tasks.rados.rados.0.smithi076.stdout:3279: left oid 4 (ObjNum 1109 snap 386 seq_num 1109) 2021-03-15T22:33:43.534 INFO:tasks.rados.rados.0.smithi076.stdout:3279: done (2 left) 2021-03-15T22:33:43.534 INFO:tasks.rados.rados.0.smithi076.stdout:3283: write (excl) oid 21 current snap is 386 2021-03-15T22:33:43.535 INFO:tasks.rados.rados.0.smithi076.stdout:3283: seq_num 1110 ranges {655770=513928,1854542=684057} 2021-03-15T22:33:43.537 INFO:tasks.rados.rados.0.smithi076.stdout:3283: writing smithi07694878-21 from 655770 to 1169698 tid 1 2021-03-15T22:33:43.546 INFO:tasks.rados.rados.0.smithi076.stdout:3283: writing smithi07694878-21 from 1854542 to 2538599 tid 2 2021-03-15T22:33:43.546 INFO:tasks.rados.rados.0.smithi076.stdout:3284: write oid 2 current snap is 386 2021-03-15T22:33:43.546 INFO:tasks.rados.rados.0.smithi076.stdout:3284: seq_num 1111 ranges {672577=589177,1913024=427715} 2021-03-15T22:33:43.553 INFO:tasks.rados.rados.0.smithi076.stdout:3284: writing smithi07694878-2 from 672577 to 1261754 tid 1 2021-03-15T22:33:43.560 INFO:tasks.rados.rados.0.smithi076.stdout:3284: writing smithi07694878-2 from 1913024 to 2340739 tid 2 2021-03-15T22:33:43.560 INFO:tasks.rados.rados.0.smithi076.stdout:3285: read oid 7 snap 370 2021-03-15T22:33:43.560 INFO:tasks.rados.rados.0.smithi076.stdout:3285: expect (ObjNum 962 snap 321 seq_num 962) 2021-03-15T22:33:43.560 INFO:tasks.rados.rados.0.smithi076.stdout:3286: write oid 31 current snap is 386 2021-03-15T22:33:43.561 INFO:tasks.rados.rados.0.smithi076.stdout:3282: finishing copy_from to smithi07694878-25 2021-03-15T22:33:43.561 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 2399 (ObjNum 1076 snap 366 seq_num 1076) dirty exists 2021-03-15T22:33:43.561 INFO:tasks.rados.rados.0.smithi076.stdout:3282: finishing copy_from racing read to smithi07694878-25 2021-03-15T22:33:43.561 INFO:tasks.rados.rados.0.smithi076.stdout:3283: finishing write tid 1 to smithi07694878-21 2021-03-15T22:33:43.561 INFO:tasks.rados.rados.0.smithi076.stdout:3283: finishing write tid 2 to smithi07694878-21 2021-03-15T22:33:43.562 INFO:tasks.rados.rados.0.smithi076.stdout:3283: finishing write tid 3 to smithi07694878-21 2021-03-15T22:33:43.562 INFO:tasks.rados.rados.0.smithi076.stdout:3283: finishing write tid 4 to smithi07694878-21 2021-03-15T22:33:43.562 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 2898 (ObjNum 1110 snap 386 seq_num 1110) dirty exists 2021-03-15T22:33:43.562 INFO:tasks.rados.rados.0.smithi076.stdout:3283: left oid 21 (ObjNum 1110 snap 386 seq_num 1110) 2021-03-15T22:33:43.563 INFO:tasks.rados.rados.0.smithi076.stdout:3284: finishing write tid 1 to smithi07694878-2 2021-03-15T22:33:43.563 INFO:tasks.rados.rados.0.smithi076.stdout:3286: seq_num 1112 ranges {689384=664426,1971506=687726} 2021-03-15T22:33:43.567 INFO:tasks.rados.rados.0.smithi076.stdout:3286: writing smithi07694878-31 from 689384 to 1353810 tid 1 2021-03-15T22:33:43.575 INFO:tasks.rados.rados.0.smithi076.stdout:3286: writing smithi07694878-31 from 1971506 to 2659232 tid 2 2021-03-15T22:33:43.575 INFO:tasks.rados.rados.0.smithi076.stdout:3281: done (5 left) 2021-03-15T22:33:43.575 INFO:tasks.rados.rados.0.smithi076.stdout:3282: done (4 left) 2021-03-15T22:33:43.575 INFO:tasks.rados.rados.0.smithi076.stdout:3283: done (3 left) 2021-03-15T22:33:43.576 INFO:tasks.rados.rados.0.smithi076.stdout:3287: copy_from oid 38 from oid 8 current snap is 386 2021-03-15T22:33:43.576 INFO:tasks.rados.rados.0.smithi076.stdout:3285: expect (ObjNum 962 snap 321 seq_num 962) 2021-03-15T22:33:43.601 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:43.626 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:43.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:43.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:43.701 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:43.729 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:43.731 INFO:tasks.rados.rados.0.smithi076.stdout:3284: finishing write tid 2 to smithi07694878-2 2021-03-15T22:33:43.731 INFO:tasks.rados.rados.0.smithi076.stdout:3285: done (3 left) 2021-03-15T22:33:43.732 INFO:tasks.rados.rados.0.smithi076.stdout:3288: read oid 28 snap -1 2021-03-15T22:33:43.732 INFO:tasks.rados.rados.0.smithi076.stdout:3288: expect deleted 2021-03-15T22:33:43.732 INFO:tasks.rados.rados.0.smithi076.stdout:3289: copy_from oid 50 from oid 41 current snap is 386 2021-03-15T22:33:43.732 INFO:tasks.rados.rados.0.smithi076.stdout:3284: finishing write tid 3 to smithi07694878-2 2021-03-15T22:33:43.732 INFO:tasks.rados.rados.0.smithi076.stdout:3284: finishing write tid 4 to smithi07694878-2 2021-03-15T22:33:43.733 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 2402 (ObjNum 1111 snap 386 seq_num 1111) dirty exists 2021-03-15T22:33:43.733 INFO:tasks.rados.rados.0.smithi076.stdout:3284: left oid 2 (ObjNum 1111 snap 386 seq_num 1111) 2021-03-15T22:33:43.733 INFO:tasks.rados.rados.0.smithi076.stdout:3286: finishing write tid 1 to smithi07694878-31 2021-03-15T22:33:43.733 INFO:tasks.rados.rados.0.smithi076.stdout:3287: finishing copy_from to smithi07694878-38 2021-03-15T22:33:43.734 INFO:tasks.rados.rados.0.smithi076.stdout:3287: got expected ENOENT (src dne) 2021-03-15T22:33:43.734 INFO:tasks.rados.rados.0.smithi076.stdout:3287: finishing copy_from racing read to smithi07694878-38 2021-03-15T22:33:43.734 INFO:tasks.rados.rados.0.smithi076.stdout:3286: finishing write tid 2 to smithi07694878-31 2021-03-15T22:33:43.734 INFO:tasks.rados.rados.0.smithi076.stdout:3286: finishing write tid 3 to smithi07694878-31 2021-03-15T22:33:43.734 INFO:tasks.rados.rados.0.smithi076.stdout:3286: finishing write tid 4 to smithi07694878-31 2021-03-15T22:33:43.735 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 2414 (ObjNum 1112 snap 386 seq_num 1112) dirty exists 2021-03-15T22:33:43.735 INFO:tasks.rados.rados.0.smithi076.stdout:3286: left oid 31 (ObjNum 1112 snap 386 seq_num 1112) 2021-03-15T22:33:43.735 INFO:tasks.rados.rados.0.smithi076.stdout:3284: done (4 left) 2021-03-15T22:33:43.735 INFO:tasks.rados.rados.0.smithi076.stdout:3286: done (3 left) 2021-03-15T22:33:43.735 INFO:tasks.rados.rados.0.smithi076.stdout:3287: done (2 left) 2021-03-15T22:33:43.736 INFO:tasks.rados.rados.0.smithi076.stdout:3290: snap_create 2021-03-15T22:33:43.742 INFO:tasks.rados.rados.0.smithi076.stdout:3289: finishing copy_from to smithi07694878-50 2021-03-15T22:33:43.742 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 2399 (ObjNum 1076 snap 366 seq_num 1076) dirty exists 2021-03-15T22:33:43.743 INFO:tasks.rados.rados.0.smithi076.stdout:3289: finishing copy_from racing read to smithi07694878-50 2021-03-15T22:33:43.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:43.754 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:43.755 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:43.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:43 smithi125 conmon[56739]: cluster 2021-03-15T22:33:42.894495+0000 osd.2 (osd.2) 41 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:33:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:43 smithi125 conmon[56739]: cluster 2021-03-15T22:33:43 2021-03-15T22:33:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:43 smithi125 conmon[56739]: .308978+0000 mon.a (mon.0) 2210 : cluster [DBG] osdmap e873: 8 total, 8 up, 8 in 2021-03-15T22:33:43.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:43.881 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:43.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:43.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:43.967 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:43 smithi076 conmon[56514]: debug 2021-03-15T22:33:43.935+0000 7f0537c88700 -1 mon.a@0(leader).osd e873 definitely_dead 0 2021-03-15T22:33:43.984 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:44.008 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:44.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:44.010 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:44.111 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:44.136 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:44.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:44.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:44.239 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:44.264 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:44.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:44.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:44.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:44.370 INFO:tasks.rados.rados.0.smithi076.stdout:3288: done (2 left) 2021-03-15T22:33:44.371 INFO:tasks.rados.rados.0.smithi076.stdout:3289: done (1 left) 2021-03-15T22:33:44.371 INFO:tasks.rados.rados.0.smithi076.stdout:3290: done (0 left) 2021-03-15T22:33:44.371 INFO:tasks.rados.rados.0.smithi076.stdout:3291: delete oid 39 current snap is 387 2021-03-15T22:33:44.382 INFO:teuthology.orchestra.run.smithi076.stderr:marked out osd.2. 2021-03-15T22:33:44.397 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:33:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:44.415 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:44.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:44.517 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:44.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: cluster 2021-03-15T22:33:42.741446 2021-03-15T22:33:44.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 46 : cluster [DBG] 6.a scrub ok 2021-03-15T22:33:44.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: cluster 2021-03-15T22:33:43. 2021-03-15T22:33:44.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: 194437+0000 mgr.y (mgr.14144) 992 : cluster [DBG] pgmap v1453: 148 pgs: 148 active+clean; 584 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 530 KiB/s wr, 17 op/s 2021-03-15T22:33:44.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:33:44.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: 33:43.209207+0000 osd.6 (osd.6 2021-03-15T22:33:44.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: ) 128 : cluster [DBG] 7.c scrub ok 2021-03-15T22:33:44.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:33:44.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: 03-15T22:33:43.268608+0000 2021-03-15T22:33:44.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: osd.1 (osd.1) 170 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:33:44.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: 2021-03-15T22:33:44.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: cluster 2021-03-15T22:33:44.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: 2021-03-15T22:33:43.476693+0000 osd.7 ( 2021-03-15T22:33:44.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: osd.7) 164 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:33:44.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:33:44.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: -03-15T22:33:43.514083+0000 2021-03-15T22:33:44.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: osd.4 (osd.4) 183 : cluster [DBG] 2021-03-15T22:33:44.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: 4.1 scrub ok 2021-03-15T22:33:44.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: audit 2021-03-15T22:33:44.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: 2021-03-15T22:33:43.936815+0000 2021-03-15T22:33:44.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: mon.a (mon.0) 2211 : audit [INF] 2021-03-15T22:33:44.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3812001979' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2021-03-15T22:33:44.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: cluster 2021-03-15T22:33:44.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: 2021-03-15T22:33:43.936901+0000 mon.a (mon.0) 2212 : 2021-03-15T22:33:44.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: cluster [INF] Client client.admin marked osd.2 out, while it was still marked up 2021-03-15T22:33:44.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: audit 2021-03-15T22:33: 2021-03-15T22:33:44.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: 44.246646+0000 mon.c (mon.1) 68 : 2021-03-15T22:33:44.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/686842779' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:44.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:33:44.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: 03-15T22:33:44.247003+0000 2021-03-15T22:33:44.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: mon.a (mon.0) 2213 : audit [INF] 2021-03-15T22:33:44.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[62156]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:44.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: cluster 2021-03-15T22:33:44.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33:44.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33:42 2021-03-15T22:33:44.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: .741446+0000 osd.5 (osd.5) 46 : cluster [DBG] 6.a scrub ok 2021-03-15T22:33:44.527 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: cluster 2021-03-15T22:33:44.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33:44.528 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33: 2021-03-15T22:33:44.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 43.194437+0000 mgr.y (mgr.14144) 992 : cluster [DBG] pgmap v1453: 148 pgs: 148 active+clean; 584 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 530 KiB/s wr, 17 op/s 2021-03-15T22:33:44.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: cluster 2021-03-15T22:33:44.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33:44.529 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021 2021-03-15T22:33:44.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: - 2021-03-15T22:33:44.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 03 2021-03-15T22:33:44.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: - 2021-03-15T22:33:44.530 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 15 2021-03-15T22:33:44.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: T 2021-03-15T22:33:44.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 22 2021-03-15T22:33:44.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: : 2021-03-15T22:33:44.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 33 2021-03-15T22:33:44.531 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: : 2021-03-15T22:33:44.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 43 2021-03-15T22:33:44.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: . 2021-03-15T22:33:44.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 209207 2021-03-15T22:33:44.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: +0000 2021-03-15T22:33:44.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33:44.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: osd.6 2021-03-15T22:33:44.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: ( 2021-03-15T22:33:44.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: osd.6) 128 : cluster [DBG] 7.c scrub ok 2021-03-15T22:33:44.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: cluster 2021-03-15T22:33:43.268608+0000 2021-03-15T22:33:44.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: osd.1 (osd.1) 170 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:33:44.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:33:44.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: -03-15T22:33:43.476693+0000 2021-03-15T22:33:44.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: osd.7 (osd.7) 164 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:33:44.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:33:44.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: -03-15T22:33:43.514083+0000 2021-03-15T22:33:44.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: osd.4 (osd.4) 183 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:33:44.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33:44.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: audit 2021-03-15T22:33:44.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33:43.936815+0000 mon.a (mon 2021-03-15T22:33:44.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: .0) 2211 : audit [INF] from='client.? 172.21.15.76:0/3812001979' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2021-03-15T22:33:44.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:33:44.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: -15T22:33:43.936901+0000 mon.a (mon.0) 2212 : cluster [INF] Client client.admin marked osd.2 out, while it was still marked up 2021-03-15T22:33:44.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33:44.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: audit 2021-03-15T22:33:44.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33:44.246646+0000 mon.c (mon. 2021-03-15T22:33:44.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 1) 68 : audit [INF] from='client.? 172.21.15.76:0/686842779' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:44.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: audit 2021-03-15T 2021-03-15T22:33:44.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 22:33:44. 2021-03-15T22:33:44.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 247003 2021-03-15T22:33:44.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: +0000 2021-03-15T22:33:44.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33:44.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: mon.a 2021-03-15T22:33:44.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: ( 2021-03-15T22:33:44.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: mon.0) 2213 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:44.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:44 smithi076 conmon[56514]: 2021-03-15T22:33:44.539 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:33:44.540 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:33:44.561 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:44.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:44.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:44.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:44.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:44.690 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:44.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: cluster 2021-03-15T22:33:42.741446+0000 osd.5 (osd.5) 46 : cluster 2021-03-15T22:33:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: [DBG] 6.a scrub ok 2021-03-15T22:33:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: cluster 2021-03-15T22:33:43.194437+0000 mgr.y (mgr.14144) 992 : cluster [DBG] pgmap v1453: 148 pgs: 148 active+clean; 584 MiB data, 1.6 GiB used, 714 GiB / 715 GiB avail; 16 MiB/s rd, 530 KiB/s wr, 17 op/s 2021-03-15T22:33:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:33:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: 22:33:43.209207+0000 osd.6 (osd.6) 128 : cluster [DBG] 7.c scrub ok 2021-03-15T22:33:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:33:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: 22:33:43.268608+0000 osd.1 (osd. 2021-03-15T22:33:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: 1) 170 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:33:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:33:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: -03-15T22:33:43.476693+0000 2021-03-15T22:33:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: osd.7 (osd.7) 164 : cluster [DBG] 2021-03-15T22:33:44.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: 5.9 scrub ok 2021-03-15T22:33:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: cluster 2021-03-15T22:33:43 2021-03-15T22:33:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: .514083+0000 osd.4 (osd.4) 183 : cluster 2021-03-15T22:33:44.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: [DBG] 4.1 scrub ok 2021-03-15T22:33:44.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:33:44.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: :33:43.936815+0000 mon.a (mon. 2021-03-15T22:33:44.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: 0) 2211 : audit [INF] from='client.? 172.21.15.76:0/3812001979' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["2"]}]: dispatch 2021-03-15T22:33:44.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: cluster 2021-03-15T22:33:44.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: 2021-03-15T22:33:43.936901 2021-03-15T22:33:44.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2212 : cluster 2021-03-15T22:33:44.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: [INF] Client client.admin marked osd.2 out, while it was still marked up 2021-03-15T22:33:44.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:33:44.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: :33:44.246646+0000 mon.c (mon.1 2021-03-15T22:33:44.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: ) 68 : audit [INF] from='client.? 172.21.15.76:0/686842779' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:44.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:33:44.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: 15T22:33:44.247003+0000 mon.a ( 2021-03-15T22:33:44.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: mon.0) 2213 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:44.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:44 smithi125 conmon[56739]: 2021-03-15T22:33:44.795 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:44.820 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:44.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:44.822 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:44.923 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:44.944 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:44.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:44.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:45.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:45.070 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:45.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:45.071 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:45.172 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:45.199 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:45.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:45.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:45.302 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:45.341 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:45.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:45.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:45.444 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:45.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:45.475 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:45.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:45.577 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:45.598 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:45.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:45.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:45.701 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:45.721 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:45.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:45.723 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:45.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: cluster 2021-03-15T22:33:44.219513+0000 osd.6 (osd.6) 129 : cluster [DBG] 7.d scrub ok 2021-03-15T22:33:45.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: cluster 2021-03-15T22:33:44.355113+0000 mon.a (mon.0) 2214 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:33:45.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: cluster 2021-03-15T22:33:44.355140+0000 mon.a (mon.0) 2215 : cluster [INF] Cluster is now healthy 2021-03-15T22:33:45.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: audit 2021-03-15T22:33:44.361981+0000 mon.a (mon.0) 2216 : audit [INF] from='client.? 172.21.15.76:0/3812001979' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2021-03-15T22:33:45.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: audit 2021-03-15T22:33:44.362214+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:33:45.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: cluster 2021-03-15T22:33:44.362315+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2021-03-15T22:33:45.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: cluster 2021-03-15T22:33:44.498257+0000 osd.7 (osd.7) 165 : cluster [DBG] 3.f deep-scrub ok 2021-03-15T22:33:45.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: cluster 2021-03-15T22:33:44.698897+0000 osd.3 (osd.3) 159 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:45.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: cluster 2021-03-15T22:33:44.701368+0000 osd.3 (osd.3) 160 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:45.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: cluster 2021-03-15T22:33:45.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: 2021-03-15T22:33:44.778563+0000 osd.3 (osd.3) 161 : cluster [DBG] 6.6 scrub ok 2021-03-15T22:33:45.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[56514]: cluster 2021-03-15T22:33:45.298734+0000 mon.a (mon.0) 2219 : cluster [DBG] osdmap e875: 8 total, 8 up, 7 in 2021-03-15T22:33:45.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: cluster 2021-03-15T22:33:45.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 2021-03-15T22:33:45.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 2021 2021-03-15T22:33:45.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: - 2021-03-15T22:33:45.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 03 2021-03-15T22:33:45.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: - 2021-03-15T22:33:45.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 15 2021-03-15T22:33:45.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: T 2021-03-15T22:33:45.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 22 2021-03-15T22:33:45.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: : 2021-03-15T22:33:45.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 33 2021-03-15T22:33:45.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: : 2021-03-15T22:33:45.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 44 2021-03-15T22:33:45.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: . 2021-03-15T22:33:45.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 219513 2021-03-15T22:33:45.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: +0000 2021-03-15T22:33:45.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 2021-03-15T22:33:45.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: osd.6 2021-03-15T22:33:45.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: ( 2021-03-15T22:33:45.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: osd 2021-03-15T22:33:45.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: . 2021-03-15T22:33:45.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 6 2021-03-15T22:33:45.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: ) 2021-03-15T22:33:45.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 129 2021-03-15T22:33:45.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: : 2021-03-15T22:33:45.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: cluster 2021-03-15T22:33:45.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 2021-03-15T22:33:45.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: [DBG] 2021-03-15T22:33:45.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 7.d scrub ok 2021-03-15T22:33:45.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: cluster 2021-03-15T22:33:44. 2021-03-15T22:33:45.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 355113+0000 mon.a (mon.0) 2214 : cluster [INF] 2021-03-15T22:33:45.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:33:45.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:33:45.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 22:33:44.355140+0000 mon.a (mon.0 2021-03-15T22:33:45.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: ) 2215 : cluster [INF] Cluster is now healthy 2021-03-15T22:33:45.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:33:45.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: -15T22:33:44.361981+0000 mon.a 2021-03-15T22:33:45.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: (mon.0) 2216 : audit [INF] from='client.? 172.21.15.76:0/3812001979' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2021-03-15T22:33:45.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:33:45.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: -15T22:33:44.362214+0000 mon.a (mon 2021-03-15T22:33:45.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: .0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:33:45.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:33:45.781 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 03-15T22:33:44.362315+0000 2021-03-15T22:33:45.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: mon.a (mon.0) 2218 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2021-03-15T22:33:45.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 2021-03-15T22:33:45.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: cluster 2021-03-15T22:33:45.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 2021-03-15T22:33:44.498257+0000 osd.7 2021-03-15T22:33:45.782 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: (osd.7) 165 : cluster [DBG] 3.f deep-scrub ok 2021-03-15T22:33:45.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: cluster 2021-03-15T22:33:45.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 2021-03-15T22:33:44.698897 2021-03-15T22:33:45.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: +0000 osd.3 (osd.3) 159 : cluster [DBG] 2021-03-15T22:33:45.783 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: purged_snaps scrub starts 2021-03-15T22:33:45.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: cluster 2021-03-15T22:33: 2021-03-15T22:33:45.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 44.701368+0000 osd.3 (osd.3) 160 : cluster 2021-03-15T22:33:45.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: [DBG] purged_snaps scrub ok 2021-03-15T22:33:45.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: cluster 2021-03-15T22:33 2021-03-15T22:33:45.784 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: :44.778563+0000 osd.3 (osd.3) 161 2021-03-15T22:33:45.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: : cluster [DBG] 6.6 scrub ok 2021-03-15T22:33:45.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:33:45.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 22:33:45.298734+0000 mon.a (mon. 2021-03-15T22:33:45.785 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:45 smithi076 conmon[62156]: 0) 2219 : cluster [DBG] osdmap e875: 8 total, 8 up, 7 in 2021-03-15T22:33:45.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: cluster 2021-03-15T22:33:44.219513+0000 osd.6 (osd.6) 129 2021-03-15T22:33:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: : cluster [DBG] 7.d scrub ok 2021-03-15T22:33:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: cluster 2021-03-15T22:33:44.355113+0000 mon.a (mon.0) 2214 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:33:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: cluster 2021-03-15T22:33:44.355140+0000 mon.a (mon.0) 2215 : cluster [INF] Cluster is now healthy 2021-03-15T22:33:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: audit 2021-03-15T22:33:44.361981+0000 mon.a (mon.0) 2216 : audit [INF] from='client.? 172.21.15.76:0/3812001979' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["2"]}]': finished 2021-03-15T22:33:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: audit 2021-03-15T22:33:44.362214+0000 mon.a (mon.0) 2217 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:33:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: cluster 2021-03-15T22:33:44.362315+0000 mon.a (mon.0) 2218 : cluster [DBG] osdmap e874: 8 total, 8 up, 7 in 2021-03-15T22:33:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: cluster 2021-03-15T22:33:44.498257+0000 osd.7 (osd.7) 165 : cluster [DBG] 3.f deep-scrub ok 2021-03-15T22:33:45.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: cluster 2021-03-15T22:33:45.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: 2021-03-15T22:33:44.698897+0000 osd.3 2021-03-15T22:33:45.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: (osd.3) 159 : cluster [DBG] 2021-03-15T22:33:45.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: purged_snaps scrub starts 2021-03-15T22:33:45.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:33:45.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: 03-15T22:33:44.701368+0000 osd.3 (osd 2021-03-15T22:33:45.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: .3) 160 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:45.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: cluster 2021-03-15T22:33:45.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: 2021-03-15T22:33:44.778563+0000 2021-03-15T22:33:45.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: osd.3 (osd.3) 161 : cluster 2021-03-15T22:33:45.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: [DBG] 6.6 scrub ok 2021-03-15T22:33:45.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: cluster 2021-03-15T22:33:45.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: 2021-03-15T22:33:45.298734+0000 mon.a 2021-03-15T22:33:45.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: (mon.0) 2219 : cluster [DBG] 2021-03-15T22:33:45.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:45 smithi125 conmon[56739]: osdmap e875: 8 total, 8 up, 7 in 2021-03-15T22:33:45.824 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:45.845 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:45.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:45.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:45.948 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:45.969 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:45.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:45.971 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:46.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:46.094 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:46.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:46.095 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:46.189 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-15T22:33:46.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:46.253 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:46.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:46.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:46.350 INFO:tasks.rados.rados.0.smithi076.stdout:3291: done (0 left) 2021-03-15T22:33:46.351 INFO:tasks.rados.rados.0.smithi076.stdout:3292: read oid 2 snap -1 2021-03-15T22:33:46.351 INFO:tasks.rados.rados.0.smithi076.stdout:3292: expect (ObjNum 1111 snap 386 seq_num 1111) 2021-03-15T22:33:46.351 INFO:tasks.rados.rados.0.smithi076.stdout:3293: write (excl) oid 14 current snap is 387 2021-03-15T22:33:46.352 INFO:tasks.rados.rados.0.smithi076.stdout:3293: seq_num 1114 ranges {722998=414924,1572117=575042} 2021-03-15T22:33:46.356 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:46.357 INFO:tasks.rados.rados.0.smithi076.stdout:3293: writing smithi07694878-14 from 722998 to 1137922 tid 1 2021-03-15T22:33:46.364 INFO:tasks.rados.rados.0.smithi076.stdout:3293: writing smithi07694878-14 from 1572117 to 2147159 tid 2 2021-03-15T22:33:46.364 INFO:tasks.rados.rados.0.smithi076.stdout:3294: delete oid 36 current snap is 387 2021-03-15T22:33:46.365 INFO:tasks.rados.rados.0.smithi076.stdout:3292: expect (ObjNum 1111 snap 386 seq_num 1111) 2021-03-15T22:33:46.385 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:46.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:46.386 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:46.487 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:46.512 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:46.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:46.513 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:46.584 INFO:tasks.rados.rados.0.smithi076.stdout:3293: finishing write tid 1 to smithi07694878-14 2021-03-15T22:33:46.585 INFO:tasks.rados.rados.0.smithi076.stdout:3292: done (2 left) 2021-03-15T22:33:46.585 INFO:tasks.rados.rados.0.smithi076.stdout:3294: done (1 left) 2021-03-15T22:33:46.585 INFO:tasks.rados.rados.0.smithi076.stdout:3295: read oid 8 snap 384 2021-03-15T22:33:46.585 INFO:tasks.rados.rados.0.smithi076.stdout:3295: expect deleted 2021-03-15T22:33:46.585 INFO:tasks.rados.rados.0.smithi076.stdout:3296: snap_create 2021-03-15T22:33:46.586 INFO:tasks.rados.rados.0.smithi076.stdout:3293: finishing write tid 2 to smithi07694878-14 2021-03-15T22:33:46.586 INFO:tasks.rados.rados.0.smithi076.stdout:3293: finishing write tid 3 to smithi07694878-14 2021-03-15T22:33:46.586 INFO:tasks.rados.rados.0.smithi076.stdout:3293: finishing write tid 4 to smithi07694878-14 2021-03-15T22:33:46.586 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 2661 (ObjNum 1114 snap 387 seq_num 1114) dirty exists 2021-03-15T22:33:46.587 INFO:tasks.rados.rados.0.smithi076.stdout:3293: left oid 14 (ObjNum 1114 snap 387 seq_num 1114) 2021-03-15T22:33:46.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:46.637 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:46.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:46.638 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:46.739 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:46.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:46 smithi076 conmon[56514]: cluster 2021-03-15T22:33:45.194801+0000 mgr.y (mgr 2021-03-15T22:33:46.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:46 smithi076 conmon[56514]: .14144) 993 : cluster [DBG] pgmap v1456: 148 pgs: 148 active+clean; 591 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2021-03-15T22:33:46.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:46 smithi076 conmon[56514]: cluster 2021-03-15T22:33:46.297572+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2021-03-15T22:33:46.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:46 smithi076 conmon[62156]: cluster 2021-03-15T22:33:45.194801+0000 2021-03-15T22:33:46.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:46 smithi076 conmon[62156]: mgr.y (mgr.14144) 993 : cluster [DBG] pgmap v1456: 148 pgs: 148 active+clean; 591 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2021-03-15T22:33:46.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:46 smithi076 conmon[62156]: cluster 2021-03-15T22:33:46.297572+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2021-03-15T22:33:46.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:46.770 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:46.770 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:46.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:46 smithi125 conmon[56739]: cluster 2021-03-15T22:33:45.194801+0000 mgr.y 2021-03-15T22:33:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:46 smithi125 conmon[56739]: (mgr.14144) 993 : cluster [DBG] pgmap v1456: 148 pgs: 148 active+clean; 591 MiB data, 1.4 GiB used, 624 GiB / 626 GiB avail; 16 MiB/s rd, 1.7 MiB/s wr, 20 op/s 2021-03-15T22:33:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:46 smithi125 conmon[56739]: cluster 2021-03-15T22:33:46.297572+0000 mon.a (mon.0) 2220 : cluster [DBG] osdmap e876: 8 total, 8 up, 7 in 2021-03-15T22:33:46.872 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:46.898 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:46.900 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:47.000 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:47.001 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:47.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:47.026 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:47.026 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:47.128 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:47.157 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:47.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:47.260 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:47.284 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:47.285 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:47.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:47.303 INFO:tasks.rados.rados.0.smithi076.stdout:3293: done (2 left) 2021-03-15T22:33:47.303 INFO:tasks.rados.rados.0.smithi076.stdout:3295: done (1 left) 2021-03-15T22:33:47.303 INFO:tasks.rados.rados.0.smithi076.stdout:3296: done (0 left) 2021-03-15T22:33:47.304 INFO:tasks.rados.rados.0.smithi076.stdout:3297: copy_from oid 37 from oid 20 current snap is 388 2021-03-15T22:33:47.304 INFO:tasks.rados.rados.0.smithi076.stdout:3298: write (excl) oid 38 current snap is 388 2021-03-15T22:33:47.304 INFO:tasks.rados.rados.0.smithi076.stdout:3298: seq_num 1116 ranges {756612=565422,2089081=578711} 2021-03-15T22:33:47.311 INFO:tasks.rados.rados.0.smithi076.stdout:3298: writing smithi07694878-38 from 756612 to 1322034 tid 1 2021-03-15T22:33:47.319 INFO:tasks.rados.rados.0.smithi076.stdout:3298: writing smithi07694878-38 from 2089081 to 2667792 tid 2 2021-03-15T22:33:47.319 INFO:tasks.rados.rados.0.smithi076.stdout:3299: snap_remove snap 356 2021-03-15T22:33:47.341 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:33:47.361 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:33:47.379 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:33:47.386 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:47.396 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:33:47.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:47.414 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:47.415 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:47.415 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:33:47.433 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:33:47.450 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:33:47.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:47 smithi076 conmon[62156]: cluster 2021-03-15T22:33:46.300702+0000 osd.7 (osd.7) 166 : cluster [DBG] 5.d starting backfill to osd.0 from (0'0,0'0] MAX to 732'13 2021-03-15T22:33:47.472 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:47 smithi076 conmon[62156]: cluster 2021-03-15T22:33:46.305753+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.1e starting backfill to osd.1 from (0'0,0'0] MAX to 643'13 2021-03-15T22:33:47.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:47 smithi076 conmon[62156]: cluster 2021-03-15T22:33:47.299755+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e877: 8 total, 8 up, 7 in 2021-03-15T22:33:47.473 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:47 smithi076 conmon[62156]: audit 2021-03-15T22:33:47.318935+0000 mon.c (mon.1) 69 : audit [DBG] from='client.? 172.21.15.76:0/3884563716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:47.473 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:47 smithi076 conmon[56514]: cluster 2021-03-15T22:33:46.300702+0000 osd.7 (osd.7) 166 : cluster [DBG] 5.d starting backfill to osd.0 from (0'0,0'0] MAX to 732'13 2021-03-15T22:33:47.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:47 smithi076 conmon[56514]: cluster 2021-03-15T22:33:46.305753+0000 osd.0 (osd.0) 142 : cluster [DBG] 5.1e starting backfill to osd.1 from (0'0,0'0] MAX to 643'13 2021-03-15T22:33:47.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:47 smithi076 conmon[56514]: cluster 2021-03-15T22:33:47.299755+0000 mon.a (mon.0) 2221 : cluster [DBG] osdmap e877: 8 total, 8 up, 7 in 2021-03-15T22:33:47.474 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:47 smithi076 conmon[56514]: audit 2021-03-15T22:33:47.318935+0000 mon.c (mon.1) 69 : audit [DBG] from='client.? 172.21.15.76:0/3884563716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:47.475 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:33:47.475 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:33:47.475 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:33:47.476 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:33:47.476 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:33:47.476 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:33:47.477 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:33:47.477 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:33:47.477 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:33:47.516 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:47.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:47.542 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:47.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:47.644 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:47.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:47.669 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:47.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:47.773 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:47.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:47 smithi125 conmon[56739]: cluster 2021-03-15T22:33:46.300702+0000 osd.7 (osd. 2021-03-15T22:33:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:47 smithi125 conmon[56739]: 7) 166 : cluster [DBG] 5.d starting backfill to osd.0 from (0'0,0'0] MAX to 732'13 2021-03-15T22:33:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:47 smithi125 conmon[56739]: cluster 2021-03-15T22:33:46.305753+0000 osd.0 (osd.0) 142 : cluster [DBG] 2021-03-15T22:33:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:47 smithi125 conmon[56739]: 5.1e starting backfill to osd.1 from (0'0,0'0] MAX to 643'13 2021-03-15T22:33:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:47 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:33:47.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:47 smithi125 conmon[56739]: -15T22:33:47.299755+0000 mon.a (mon.0) 2221 2021-03-15T22:33:47.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:47 smithi125 conmon[56739]: : cluster [DBG] osdmap e877: 8 total, 8 up, 7 in 2021-03-15T22:33:47.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:47 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:33:47.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:47 smithi125 conmon[56739]: 15T22:33:47.318935+0000 mon.c ( 2021-03-15T22:33:47.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:47 smithi125 conmon[56739]: mon.1) 69 : audit [DBG] from='client.? 172.21.15.76:0/3884563716' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:47.796 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:47.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:47.798 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:47.899 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:47.922 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:47.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:47.923 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:48.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:48.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:48.048 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:48.048 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:48.149 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:48.172 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:48.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:48.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:48.189 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:33:48.275 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:48.298 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:48.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:48.299 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:48.303 INFO:tasks.rados.rados.0.smithi076.stdout:3299: done (2 left) 2021-03-15T22:33:48.303 INFO:tasks.rados.rados.0.smithi076.stdout:3300: delete oid 6 current snap is 388 2021-03-15T22:33:48.303 INFO:tasks.rados.rados.0.smithi076.stdout:3297: finishing copy_from to smithi07694878-37 2021-03-15T22:33:48.303 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 2407 (ObjNum 1108 snap 385 seq_num 1108) dirty exists 2021-03-15T22:33:48.304 INFO:tasks.rados.rados.0.smithi076.stdout:3297: finishing copy_from racing read to smithi07694878-37 2021-03-15T22:33:48.324 INFO:tasks.rados.rados.0.smithi076.stdout:3297: done (2 left) 2021-03-15T22:33:48.324 INFO:tasks.rados.rados.0.smithi076.stdout:3300: done (1 left) 2021-03-15T22:33:48.324 INFO:tasks.rados.rados.0.smithi076.stdout:3301: read oid 42 snap -1 2021-03-15T22:33:48.325 INFO:tasks.rados.rados.0.smithi076.stdout:3301: expect (ObjNum 1102 snap 382 seq_num 1102) 2021-03-15T22:33:48.325 INFO:tasks.rados.rados.0.smithi076.stdout:3302: write oid 44 current snap is 388 2021-03-15T22:33:48.325 INFO:tasks.rados.rados.0.smithi076.stdout:3302: seq_num 1118 ranges {790226=432273,1806045=582380} 2021-03-15T22:33:48.331 INFO:tasks.rados.rados.0.smithi076.stdout:3302: writing smithi07694878-44 from 790226 to 1222499 tid 1 2021-03-15T22:33:48.338 INFO:tasks.rados.rados.0.smithi076.stdout:3302: writing smithi07694878-44 from 1806045 to 2388425 tid 2 2021-03-15T22:33:48.338 INFO:tasks.rados.rados.0.smithi076.stdout:3303: write oid 19 current snap is 388 2021-03-15T22:33:48.339 INFO:tasks.rados.rados.0.smithi076.stdout:3298: finishing write tid 1 to smithi07694878-38 2021-03-15T22:33:48.339 INFO:tasks.rados.rados.0.smithi076.stdout:3303: seq_num 1119 ranges {407033=507522,1464527=442391,2345976=461057} 2021-03-15T22:33:48.343 INFO:tasks.rados.rados.0.smithi076.stdout:3303: writing smithi07694878-19 from 407033 to 914555 tid 1 2021-03-15T22:33:48.349 INFO:tasks.rados.rados.0.smithi076.stdout:3303: writing smithi07694878-19 from 1464527 to 1906918 tid 2 2021-03-15T22:33:48.354 INFO:tasks.rados.rados.0.smithi076.stdout:3303: writing smithi07694878-19 from 2345976 to 2807033 tid 3 2021-03-15T22:33:48.354 INFO:tasks.rados.rados.0.smithi076.stdout:3304: write oid 6 current snap is 388 2021-03-15T22:33:48.354 INFO:tasks.rados.rados.0.smithi076.stdout:3298: finishing write tid 2 to smithi07694878-38 2021-03-15T22:33:48.355 INFO:tasks.rados.rados.0.smithi076.stdout:3301: expect (ObjNum 1102 snap 382 seq_num 1102) 2021-03-15T22:33:48.400 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:48.423 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:48.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:48.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:48.483 INFO:tasks.rados.rados.0.smithi076.stdout:3304: seq_num 1120 ranges {423840=582771,1523009=586049,2657046=166794} 2021-03-15T22:33:48.488 INFO:tasks.rados.rados.0.smithi076.stdout:3304: writing smithi07694878-6 from 423840 to 1006611 tid 1 2021-03-15T22:33:48.494 INFO:tasks.rados.rados.0.smithi076.stdout:3304: writing smithi07694878-6 from 1523009 to 2109058 tid 2 2021-03-15T22:33:48.499 INFO:tasks.rados.rados.0.smithi076.stdout:3304: writing smithi07694878-6 from 2657046 to 2823840 tid 3 2021-03-15T22:33:48.499 INFO:tasks.rados.rados.0.smithi076.stdout:3301: done (4 left) 2021-03-15T22:33:48.500 INFO:tasks.rados.rados.0.smithi076.stdout:3305: delete oid 35 current snap is 388 2021-03-15T22:33:48.500 INFO:tasks.rados.rados.0.smithi076.stdout:3302: finishing write tid 1 to smithi07694878-44 2021-03-15T22:33:48.500 INFO:tasks.rados.rados.0.smithi076.stdout:3298: finishing write tid 3 to smithi07694878-38 2021-03-15T22:33:48.500 INFO:tasks.rados.rados.0.smithi076.stdout:3298: finishing write tid 4 to smithi07694878-38 2021-03-15T22:33:48.501 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2306 (ObjNum 1116 snap 388 seq_num 1116) dirty exists 2021-03-15T22:33:48.501 INFO:tasks.rados.rados.0.smithi076.stdout:3298: left oid 38 (ObjNum 1116 snap 388 seq_num 1116) 2021-03-15T22:33:48.501 INFO:tasks.rados.rados.0.smithi076.stdout:3302: finishing write tid 2 to smithi07694878-44 2021-03-15T22:33:48.501 INFO:tasks.rados.rados.0.smithi076.stdout:3302: finishing write tid 3 to smithi07694878-44 2021-03-15T22:33:48.501 INFO:tasks.rados.rados.0.smithi076.stdout:3302: finishing write tid 4 to smithi07694878-44 2021-03-15T22:33:48.502 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 2098 (ObjNum 1118 snap 388 seq_num 1118) dirty exists 2021-03-15T22:33:48.502 INFO:tasks.rados.rados.0.smithi076.stdout:3302: left oid 44 (ObjNum 1118 snap 388 seq_num 1118) 2021-03-15T22:33:48.502 INFO:tasks.rados.rados.0.smithi076.stdout:3303: finishing write tid 1 to smithi07694878-19 2021-03-15T22:33:48.502 INFO:tasks.rados.rados.0.smithi076.stdout:3303: finishing write tid 2 to smithi07694878-19 2021-03-15T22:33:48.503 INFO:tasks.rados.rados.0.smithi076.stdout:3303: finishing write tid 3 to smithi07694878-19 2021-03-15T22:33:48.503 INFO:tasks.rados.rados.0.smithi076.stdout:3303: finishing write tid 4 to smithi07694878-19 2021-03-15T22:33:48.503 INFO:tasks.rados.rados.0.smithi076.stdout:3303: finishing write tid 5 to smithi07694878-19 2021-03-15T22:33:48.503 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2310 (ObjNum 1119 snap 388 seq_num 1119) dirty exists 2021-03-15T22:33:48.503 INFO:tasks.rados.rados.0.smithi076.stdout:3303: left oid 19 (ObjNum 1119 snap 388 seq_num 1119) 2021-03-15T22:33:48.504 INFO:tasks.rados.rados.0.smithi076.stdout:3304: finishing write tid 1 to smithi07694878-6 2021-03-15T22:33:48.504 INFO:tasks.rados.rados.0.smithi076.stdout:3304: finishing write tid 2 to smithi07694878-6 2021-03-15T22:33:48.505 INFO:tasks.rados.rados.0.smithi076.stdout:3304: finishing write tid 3 to smithi07694878-6 2021-03-15T22:33:48.505 INFO:tasks.rados.rados.0.smithi076.stdout:3304: finishing write tid 4 to smithi07694878-6 2021-03-15T22:33:48.505 INFO:tasks.rados.rados.0.smithi076.stdout:3304: finishing write tid 5 to smithi07694878-6 2021-03-15T22:33:48.506 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 2106 (ObjNum 1120 snap 388 seq_num 1120) dirty exists 2021-03-15T22:33:48.506 INFO:tasks.rados.rados.0.smithi076.stdout:3304: left oid 6 (ObjNum 1120 snap 388 seq_num 1120) 2021-03-15T22:33:48.506 INFO:tasks.rados.rados.0.smithi076.stdout:3298: done (4 left) 2021-03-15T22:33:48.506 INFO:tasks.rados.rados.0.smithi076.stdout:3302: done (3 left) 2021-03-15T22:33:48.506 INFO:tasks.rados.rados.0.smithi076.stdout:3303: done (2 left) 2021-03-15T22:33:48.507 INFO:tasks.rados.rados.0.smithi076.stdout:3304: done (1 left) 2021-03-15T22:33:48.507 INFO:tasks.rados.rados.0.smithi076.stdout:3305: done (0 left) 2021-03-15T22:33:48.507 INFO:tasks.rados.rados.0.smithi076.stdout:3306: snap_create 2021-03-15T22:33:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:48 smithi076 conmon[56514]: cluster 2021-03-15T22:33:47.195295 2021-03-15T22:33:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:48 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 994 : cluster [DBG] pgmap v1459: 148 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering, 143 active+clean; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 17 MiB/s rd, 3.4 MiB/s wr, 24 op/s; 2/1800 objects degraded (0.111%); 28/1800 objects misplaced (1.556%); 7.4 MiB/s, 51 objects/s recovering 2021-03-15T22:33:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:48 smithi076 conmon[56514]: cluster 2021-03-15T22:33:47.459028+0000 mon.a (mon.0) 2222 : cluster [WRN] Health check failed: Degraded data redundancy: 2/1800 objects degraded (0.111%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:33:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:48 smithi076 conmon[56514]: cluster 2021-03-15T22:33:48.299805+0000 mon.a (mon.0) 2223 : cluster [DBG] osdmap e878: 8 total, 8 up, 7 in 2021-03-15T22:33:48.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:48 smithi076 conmon[62156]: cluster 2021-03-15T22:33:47.195295+0000 2021-03-15T22:33:48.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:48 smithi076 conmon[62156]: mgr.y (mgr.14144) 994 : cluster [DBG] pgmap v1459: 148 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering, 143 active+clean; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 17 MiB/s rd, 3.4 MiB/s wr, 24 op/s; 2/1800 objects degraded (0.111%); 28/1800 objects misplaced (1.556%); 7.4 MiB/s, 51 objects/s recovering 2021-03-15T22:33:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:48 smithi076 conmon[62156]: cluster 2021-03-15T22:33:47.459028+0000 mon.a (mon.0) 2222 : cluster 2021-03-15T22:33:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:48 smithi076 conmon[62156]: [WRN] Health check failed: Degraded data redundancy: 2/1800 objects degraded (0.111%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:33:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:48 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:33:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:48 smithi076 conmon[62156]: 03-15T22:33:48.299805+0000 mon.a (mon.0) 2223 : cluster 2021-03-15T22:33:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:48 smithi076 conmon[62156]: [DBG] osdmap e878: 8 total, 8 up, 7 in 2021-03-15T22:33:48.525 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:48.547 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:48.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:48.548 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:48.649 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:48.670 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:48.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:48.672 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:48.773 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:48.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:48 smithi125 conmon[56739]: cluster 2021-03-15T22:33:47.195295 2021-03-15T22:33:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:48 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 994 : cluster [DBG] pgmap v1459: 148 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering, 143 active+clean; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 17 MiB/s rd, 3.4 MiB/s wr, 24 op/s; 2/1800 objects degraded (0.111%); 28/1800 objects misplaced (1.556%); 7.4 MiB/s, 51 objects/s recovering 2021-03-15T22:33:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:48 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:33:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:48 smithi125 conmon[56739]: 15T22:33:47.459028+0000 mon.a (mon.0) 2222 : cluster 2021-03-15T22:33:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:48 smithi125 conmon[56739]: [WRN] Health check failed: Degraded data redundancy: 2/1800 objects degraded (0.111%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:33:48.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:48 smithi125 conmon[56739]: cluster 2021-03-15T22:33:48.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:48 smithi125 conmon[56739]: 2021-03-15T22:33:48 2021-03-15T22:33:48.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:48 smithi125 conmon[56739]: .299805+0000 mon.a (mon.0) 2223 2021-03-15T22:33:48.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:48 smithi125 conmon[56739]: : cluster [DBG] osdmap e878: 8 total, 8 up, 7 in 2021-03-15T22:33:48.795 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:48.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:48.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:48.897 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:48.919 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:48.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:48.920 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:49.021 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:49.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:49.044 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:49.044 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:49.045 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:49.145 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:49.166 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:49.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:49.167 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:49.269 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:49.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:49.291 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:49.291 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:49.392 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:49.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:49.418 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:49.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:49.479 INFO:tasks.rados.rados.0.smithi076.stdout:3306: done (0 left) 2021-03-15T22:33:49.480 INFO:tasks.rados.rados.0.smithi076.stdout:3307: write (excl) oid 33 current snap is 389 2021-03-15T22:33:49.480 INFO:tasks.rados.rados.0.smithi076.stdout:3307: seq_num 1122 ranges {457454=733269,1923620=589718} 2021-03-15T22:33:49.485 INFO:tasks.rados.rados.0.smithi076.stdout:3307: writing smithi07694878-33 from 457454 to 1190723 tid 1 2021-03-15T22:33:49.494 INFO:tasks.rados.rados.0.smithi076.stdout:3307: writing smithi07694878-33 from 1923620 to 2513338 tid 2 2021-03-15T22:33:49.494 INFO:tasks.rados.rados.0.smithi076.stdout:3308: read oid 41 snap -1 2021-03-15T22:33:49.494 INFO:tasks.rados.rados.0.smithi076.stdout:3308: expect (ObjNum 1076 snap 366 seq_num 1076) 2021-03-15T22:33:49.495 INFO:tasks.rados.rados.0.smithi076.stdout:3309: delete oid 45 current snap is 389 2021-03-15T22:33:49.495 INFO:tasks.rados.rados.0.smithi076.stdout:3307: finishing write tid 1 to smithi07694878-33 2021-03-15T22:33:49.497 INFO:tasks.rados.rados.0.smithi076.stdout:3307: finishing write tid 2 to smithi07694878-33 2021-03-15T22:33:49.497 INFO:tasks.rados.rados.0.smithi076.stdout:3309: done (2 left) 2021-03-15T22:33:49.497 INFO:tasks.rados.rados.0.smithi076.stdout:3310: read oid 42 snap 370 2021-03-15T22:33:49.498 INFO:tasks.rados.rados.0.smithi076.stdout:3310: expect deleted 2021-03-15T22:33:49.498 INFO:tasks.rados.rados.0.smithi076.stdout:3311: snap_remove snap 333 2021-03-15T22:33:49.520 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:49.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:49.544 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:49.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:49.645 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:49.667 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:49.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:49.669 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:49.770 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:49.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:49.792 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:49.792 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:49.894 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:49.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:49.916 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:49.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:50.018 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:50.042 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:50.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:50.044 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:50.145 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:50.167 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:50.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:50.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:50.270 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:50.278 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:33:50.278 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:33:50.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:33:50.278 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:33:50.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.279 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3982 7.3 7:c458c800:::smithi07694878-3:head [copy-from ver 119] snapc 13f=[13f,13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e421)", 2021-03-15T22:33:50.279 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:36.115531+0000", 2021-03-15T22:33:50.279 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 494.00177036999997, 2021-03-15T22:33:50.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0139079740000001, 2021-03-15T22:33:50.280 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.280 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:50.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.280 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.281 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.281 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3982 2021-03-15T22:33:50.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.281 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115531+0000", 2021-03-15T22:33:50.282 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115531+0000", 2021-03-15T22:33:50.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4710000000000001e-06 2021-03-15T22:33:50.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115536+0000", 2021-03-15T22:33:50.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5420000000000001e-06 2021-03-15T22:33:50.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.285 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.285 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115539+0000", 2021-03-15T22:33:50.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3487e-05 2021-03-15T22:33:50.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.286 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.287 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115553+0000", 2021-03-15T22:33:50.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4500000000000004e-06 2021-03-15T22:33:50.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.287 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.287 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115561+0000", 2021-03-15T22:33:50.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0590000000000001e-05 2021-03-15T22:33:50.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115622+0000", 2021-03-15T22:33:50.289 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3449999999999995e-06 2021-03-15T22:33:50.289 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:33:50.290 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115631+0000", 2021-03-15T22:33:50.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00089869400000000001 2021-03-15T22:33:50.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.116530+0000", 2021-03-15T22:33:50.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8217000000000001e-05 2021-03-15T22:33:50.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.292 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.116598+0000", 2021-03-15T22:33:50.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.00515501 2021-03-15T22:33:50.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.293 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:33:50.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.121753+0000", 2021-03-15T22:33:50.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073058109999999997 2021-03-15T22:33:50.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129059+0000", 2021-03-15T22:33:50.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1652e-05 2021-03-15T22:33:50.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:50.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129071+0000", 2021-03-15T22:33:50.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025268099999999999 2021-03-15T22:33:50.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129323+0000", 2021-03-15T22:33:50.296 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5415999999999999e-05 2021-03-15T22:33:50.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:50.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129379+0000", 2021-03-15T22:33:50.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0608e-05 2021-03-15T22:33:50.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129439+0000", 2021-03-15T22:33:50.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0139079740000001 2021-03-15T22:33:50.299 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.299 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.300 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.300 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3983 7.3 7:c458c800:::smithi07694878-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e421)", 2021-03-15T22:33:50.300 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:36.115576+0000", 2021-03-15T22:33:50.301 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 494.00172606799998, 2021-03-15T22:33:50.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0140553489999999, 2021-03-15T22:33:50.301 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.301 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:50.301 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.302 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.302 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.302 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3983 2021-03-15T22:33:50.303 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.303 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115576+0000", 2021-03-15T22:33:50.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115576+0000", 2021-03-15T22:33:50.305 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2550000000000001e-06 2021-03-15T22:33:50.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.306 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.306 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115577+0000", 2021-03-15T22:33:50.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8390000000000001e-06 2021-03-15T22:33:50.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.307 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115579+0000", 2021-03-15T22:33:50.307 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0740000000000004e-06 2021-03-15T22:33:50.308 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.308 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115586+0000", 2021-03-15T22:33:50.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3220000000000003e-06 2021-03-15T22:33:50.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.309 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.309 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115590+0000", 2021-03-15T22:33:50.309 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108726 2021-03-15T22:33:50.310 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.310 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.311 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115699+0000", 2021-03-15T22:33:50.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9349999999999997e-06 2021-03-15T22:33:50.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.311 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:33:50.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115706+0000", 2021-03-15T22:33:50.312 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021999540000000001 2021-03-15T22:33:50.312 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.313 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.117906+0000", 2021-03-15T22:33:50.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6322e-05 2021-03-15T22:33:50.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:33:50.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.117962+0000", 2021-03-15T22:33:50.314 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.00844165 2021-03-15T22:33:50.314 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.315 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.315 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.126404+0000", 2021-03-15T22:33:50.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5353999999999996e-05 2021-03-15T22:33:50.316 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:50.316 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.126489+0000", 2021-03-15T22:33:50.316 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029835399999999998 2021-03-15T22:33:50.317 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.317 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.317 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129473+0000", 2021-03-15T22:33:50.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5050999999999997e-05 2021-03-15T22:33:50.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.318 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129548+0000", 2021-03-15T22:33:50.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3326999999999997e-05 2021-03-15T22:33:50.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129631+0000", 2021-03-15T22:33:50.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0140553489999999 2021-03-15T22:33:50.320 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.320 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.320 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.320 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.321 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4380 7.6 e455/421 7:6283974b:::smithi07694878-30:head v 455'1076, mlcod=454'1072)", 2021-03-15T22:33:50.321 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:10.132423+0000", 2021-03-15T22:33:50.321 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 459.98487853900002, 2021-03-15T22:33:50.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.120130079, 2021-03-15T22:33:50.322 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.322 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:50.322 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.323 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.323 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.132423+0000", 2021-03-15T22:33:50.323 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.132423+0000", 2021-03-15T22:33:50.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010757970000000001 2021-03-15T22:33:50.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.325 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.133499+0000", 2021-03-15T22:33:50.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5153e-05 2021-03-15T22:33:50.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.133514+0000", 2021-03-15T22:33:50.327 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.070374909999999999 2021-03-15T22:33:50.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.328 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.328 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.203889+0000", 2021-03-15T22:33:50.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6134000000000001e-05 2021-03-15T22:33:50.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.328 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.203915+0000", 2021-03-15T22:33:50.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8656000000000001e-05 2021-03-15T22:33:50.329 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.330 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.203984+0000", 2021-03-15T22:33:50.330 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5831999999999999e-05 2021-03-15T22:33:50.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.204010+0000", 2021-03-15T22:33:50.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048499043999999998 2021-03-15T22:33:50.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.332 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.332 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:50.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.252509+0000", 2021-03-15T22:33:50.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4552999999999998e-05 2021-03-15T22:33:50.332 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.252553+0000", 2021-03-15T22:33:50.334 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.120130079 2021-03-15T22:33:50.334 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.334 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.334 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.335 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5267 7.a 7.68cfe3da (undecoded) ondisk+write+known_if_redirected e540)", 2021-03-15T22:33:50.335 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.213130+0000", 2021-03-15T22:33:50.335 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 374.90417151299999, 2021-03-15T22:33:50.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99493337699999995, 2021-03-15T22:33:50.336 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.336 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:50.336 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.336 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.337 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.337 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5267 2021-03-15T22:33:50.337 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.337 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.337 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.338 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.338 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213130+0000", 2021-03-15T22:33:50.338 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.339 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.339 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.339 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213130+0000", 2021-03-15T22:33:50.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027025900000000002 2021-03-15T22:33:50.340 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.340 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.340 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.340 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213400+0000", 2021-03-15T22:33:50.340 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7129999999999999e-06 2021-03-15T22:33:50.341 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.341 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.341 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.341 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213404+0000", 2021-03-15T22:33:50.342 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3495e-05 2021-03-15T22:33:50.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.342 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.342 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.342 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213418+0000", 2021-03-15T22:33:50.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5329999999999997e-06 2021-03-15T22:33:50.343 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.344 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213425+0000", 2021-03-15T22:33:50.344 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5578999999999998e-05 2021-03-15T22:33:50.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.344 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213471+0000", 2021-03-15T22:33:50.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0062e-05 2021-03-15T22:33:50.345 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.345 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.346 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213481+0000", 2021-03-15T22:33:50.346 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99456081500000004 2021-03-15T22:33:50.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.347 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208042+0000", 2021-03-15T22:33:50.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1920999999999999e-05 2021-03-15T22:33:50.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208064+0000", 2021-03-15T22:33:50.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99493337699999995 2021-03-15T22:33:50.348 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.348 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.349 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.349 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.349 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5268 7.a 7.68cfe3da (undecoded) ondisk+write+known_if_redirected e540)", 2021-03-15T22:33:50.349 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.219467+0000", 2021-03-15T22:33:50.350 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 374.89783502799997, 2021-03-15T22:33:50.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98862604499999995, 2021-03-15T22:33:50.350 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.350 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:50.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.351 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.351 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5268 2021-03-15T22:33:50.351 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.352 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.352 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219467+0000", 2021-03-15T22:33:50.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219467+0000", 2021-03-15T22:33:50.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025054199999999997 2021-03-15T22:33:50.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.354 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219717+0000", 2021-03-15T22:33:50.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6829999999999999e-06 2021-03-15T22:33:50.355 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.355 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219722+0000", 2021-03-15T22:33:50.356 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1461e-05 2021-03-15T22:33:50.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.356 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.357 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219743+0000", 2021-03-15T22:33:50.357 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4905e-05 2021-03-15T22:33:50.357 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.357 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.357 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.358 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219758+0000", 2021-03-15T22:33:50.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6631000000000006e-05 2021-03-15T22:33:50.358 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.358 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.358 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.359 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219825+0000", 2021-03-15T22:33:50.359 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1799e-05 2021-03-15T22:33:50.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.359 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219837+0000", 2021-03-15T22:33:50.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98824981899999997 2021-03-15T22:33:50.360 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208087+0000", 2021-03-15T22:33:50.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2049999999999999e-06 2021-03-15T22:33:50.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208093+0000", 2021-03-15T22:33:50.362 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98862604499999995 2021-03-15T22:33:50.362 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.363 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.363 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.363 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5269 7.a 7.68cfe3da (undecoded) ondisk+write+known_if_redirected e540)", 2021-03-15T22:33:50.364 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.219771+0000", 2021-03-15T22:33:50.364 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 374.89753050000002, 2021-03-15T22:33:50.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98834424600000004, 2021-03-15T22:33:50.364 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.365 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:50.365 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.365 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.365 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.365 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5269 2021-03-15T22:33:50.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.366 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.366 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219771+0000", 2021-03-15T22:33:50.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219771+0000", 2021-03-15T22:33:50.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3630000000000001e-06 2021-03-15T22:33:50.368 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219774+0000", 2021-03-15T22:33:50.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2139999999999999e-06 2021-03-15T22:33:50.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219775+0000", 2021-03-15T22:33:50.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5919000000000001e-05 2021-03-15T22:33:50.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.371 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.371 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219791+0000", 2021-03-15T22:33:50.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0747999999999999e-05 2021-03-15T22:33:50.371 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.371 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.372 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.372 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219801+0000", 2021-03-15T22:33:50.372 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4376999999999996e-05 2021-03-15T22:33:50.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219876+0000", 2021-03-15T22:33:50.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.585e-06 2021-03-15T22:33:50.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.374 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.374 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219884+0000", 2021-03-15T22:33:50.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98822621899999996 2021-03-15T22:33:50.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208111+0000", 2021-03-15T22:33:50.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8210000000000001e-06 2021-03-15T22:33:50.375 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208115+0000", 2021-03-15T22:33:50.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98834424600000004 2021-03-15T22:33:50.376 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.377 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.377 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.377 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5270 7.a 7.68cfe3da (undecoded) ondisk+read+rwordered+known_if_redirected e540)", 2021-03-15T22:33:50.378 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.219812+0000", 2021-03-15T22:33:50.378 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 374.89748939200001, 2021-03-15T22:33:50.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98832836899999998, 2021-03-15T22:33:50.378 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.379 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:50.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.379 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.379 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5270 2021-03-15T22:33:50.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.380 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219812+0000", 2021-03-15T22:33:50.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219812+0000", 2021-03-15T22:33:50.382 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.88e-06 2021-03-15T22:33:50.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219814+0000", 2021-03-15T22:33:50.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2960000000000001e-06 2021-03-15T22:33:50.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219816+0000", 2021-03-15T22:33:50.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0669999999999998e-06 2021-03-15T22:33:50.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219826+0000", 2021-03-15T22:33:50.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.014e-06 2021-03-15T22:33:50.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219830+0000", 2021-03-15T22:33:50.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8069e-05 2021-03-15T22:33:50.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219918+0000", 2021-03-15T22:33:50.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6109999999999994e-06 2021-03-15T22:33:50.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219926+0000", 2021-03-15T22:33:50.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98820947100000001 2021-03-15T22:33:50.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208136+0000", 2021-03-15T22:33:50.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9609999999999996e-06 2021-03-15T22:33:50.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208141+0000", 2021-03-15T22:33:50.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98832836899999998 2021-03-15T22:33:50.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.391 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.391 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.392 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14427.0:604 4.19 4:9ed7f5b7:::obj_delete_at_hint.0000000004:head [call lock.lock in=8b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e550)", 2021-03-15T22:33:50.392 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:28:11.512449+0000", 2021-03-15T22:33:50.392 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 338.60485286400001, 2021-03-15T22:33:50.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0160361099999999, 2021-03-15T22:33:50.393 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.393 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:50.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.393 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14427", 2021-03-15T22:33:50.394 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2912955800", 2021-03-15T22:33:50.394 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 604 2021-03-15T22:33:50.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.394 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512449+0000", 2021-03-15T22:33:50.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512449+0000", 2021-03-15T22:33:50.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8390000000000001e-06 2021-03-15T22:33:50.396 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512451+0000", 2021-03-15T22:33:50.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0460000000000001e-06 2021-03-15T22:33:50.397 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512454+0000", 2021-03-15T22:33:50.398 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3121e-05 2021-03-15T22:33:50.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.399 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.399 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.399 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512467+0000", 2021-03-15T22:33:50.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8639999999999996e-06 2021-03-15T22:33:50.400 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.400 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512475+0000", 2021-03-15T22:33:50.400 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013658999999999999 2021-03-15T22:33:50.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512611+0000", 2021-03-15T22:33:50.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0465000000000001e-05 2021-03-15T22:33:50.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512622+0000", 2021-03-15T22:33:50.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.002971662 2021-03-15T22:33:50.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.515593+0000", 2021-03-15T22:33:50.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000250323 2021-03-15T22:33:50.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for degraded object", 2021-03-15T22:33:50.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.515844+0000", 2021-03-15T22:33:50.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096873329999999994 2021-03-15T22:33:50.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.525531+0000", 2021-03-15T22:33:50.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2487000000000001e-05 2021-03-15T22:33:50.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.525584+0000", 2021-03-15T22:33:50.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000216667 2021-03-15T22:33:50.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 0", 2021-03-15T22:33:50.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.525800+0000", 2021-03-15T22:33:50.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00229575 2021-03-15T22:33:50.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.528096+0000", 2021-03-15T22:33:50.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027043100000000001 2021-03-15T22:33:50.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:50.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.528366+0000", 2021-03-15T22:33:50.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4045000000000003e-05 2021-03-15T22:33:50.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:50.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.528410+0000", 2021-03-15T22:33:50.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4486999999999993e-05 2021-03-15T22:33:50.411 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.528485+0000", 2021-03-15T22:33:50.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0160361099999999 2021-03-15T22:33:50.412 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.413 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.413 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.413 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGPushReply(7.f 559/553 [PushReplyOp(7:ffef77ba:::smithi07694878-13:1b1)])", 2021-03-15T22:33:50.414 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:28:19.240832+0000", 2021-03-15T22:33:50.414 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 330.87647002800003, 2021-03-15T22:33:50.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.427929538, 2021-03-15T22:33:50.414 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.414 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:33:50.415 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240832+0000", 2021-03-15T22:33:50.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240832+0000", 2021-03-15T22:33:50.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.841e-06 2021-03-15T22:33:50.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240835+0000", 2021-03-15T22:33:50.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6820000000000002e-06 2021-03-15T22:33:50.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240839+0000", 2021-03-15T22:33:50.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1081999999999999e-05 2021-03-15T22:33:50.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240860+0000", 2021-03-15T22:33:50.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5779999999999999e-06 2021-03-15T22:33:50.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240869+0000", 2021-03-15T22:33:50.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.426681264 2021-03-15T22:33:50.421 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.667550+0000", 2021-03-15T22:33:50.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012110910000000001 2021-03-15T22:33:50.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.668761+0000", 2021-03-15T22:33:50.423 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.427929538 2021-03-15T22:33:50.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.423 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.424 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.424 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.424 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6450 7.a 7:53dee572:::smithi07694878-35:1f7 [sparse-read 0~1916845,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e691)", 2021-03-15T22:33:50.424 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.216790+0000", 2021-03-15T22:33:50.425 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 189.90051178499999, 2021-03-15T22:33:50.425 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095085135000000001, 2021-03-15T22:33:50.425 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.425 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:50.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.426 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.426 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.426 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6450 2021-03-15T22:33:50.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.427 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.216790+0000", 2021-03-15T22:33:50.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.216790+0000", 2021-03-15T22:33:50.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3280000000000002e-06 2021-03-15T22:33:50.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.429 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.216793+0000", 2021-03-15T22:33:50.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.931e-06 2021-03-15T22:33:50.430 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.216796+0000", 2021-03-15T22:33:50.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080014454999999998 2021-03-15T22:33:50.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.296811+0000", 2021-03-15T22:33:50.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0380000000000001e-05 2021-03-15T22:33:50.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.296821+0000", 2021-03-15T22:33:50.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071239899999999995 2021-03-15T22:33:50.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297533+0000", 2021-03-15T22:33:50.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119981 2021-03-15T22:33:50.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297653+0000", 2021-03-15T22:33:50.435 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014221661 2021-03-15T22:33:50.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.311875+0000", 2021-03-15T22:33:50.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095085135000000001 2021-03-15T22:33:50.436 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.436 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.437 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.437 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6959 7.c 7:37eb566e:::smithi07694878-48:head [write 733036~563824,stat] snapc 25b=[25b,25a,257,255,24f,24e,24c,24b,24a,248,247,245,243,23e,23d,23c,239,235,234,22d,227,225,218,211,20c,1f7,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e740)", 2021-03-15T22:33:50.437 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.842261+0000", 2021-03-15T22:33:50.438 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 139.27504075900001, 2021-03-15T22:33:50.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40759049800000002, 2021-03-15T22:33:50.438 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.438 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:50.438 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.439 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.439 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.439 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6959 2021-03-15T22:33:50.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.440 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.440 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842261+0000", 2021-03-15T22:33:50.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842261+0000", 2021-03-15T22:33:50.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016927899999999999 2021-03-15T22:33:50.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842430+0000", 2021-03-15T22:33:50.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.097e-06 2021-03-15T22:33:50.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842437+0000", 2021-03-15T22:33:50.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2824999999999998e-05 2021-03-15T22:33:50.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.444 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842470+0000", 2021-03-15T22:33:50.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2836999999999999e-05 2021-03-15T22:33:50.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842483+0000", 2021-03-15T22:33:50.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8919000000000003e-05 2021-03-15T22:33:50.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842542+0000", 2021-03-15T22:33:50.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1345999999999999e-05 2021-03-15T22:33:50.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.448 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842553+0000", 2021-03-15T22:33:50.448 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40354269599999998 2021-03-15T22:33:50.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.246096+0000", 2021-03-15T22:33:50.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018614400000000001 2021-03-15T22:33:50.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.449 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.246282+0000", 2021-03-15T22:33:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038109899999999998 2021-03-15T22:33:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:33:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.246663+0000", 2021-03-15T22:33:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024403139999999999 2021-03-15T22:33:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249103+0000", 2021-03-15T22:33:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9490000000000002e-06 2021-03-15T22:33:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249112+0000", 2021-03-15T22:33:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000559948 2021-03-15T22:33:50.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249672+0000", 2021-03-15T22:33:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0310999999999998e-05 2021-03-15T22:33:50.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249743+0000", 2021-03-15T22:33:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010873400000000001 2021-03-15T22:33:50.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249851+0000", 2021-03-15T22:33:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40759049800000002 2021-03-15T22:33:50.456 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6960 7.c 7:37eb566e:::smithi07694878-48:head [write 2074812~684986] snapc 25b=[25b,25a,257,255,24f,24e,24c,24b,24a,248,247,245,243,23e,23d,23c,239,235,234,22d,227,225,218,211,20c,1f7,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e740)", 2021-03-15T22:33:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.850305+0000", 2021-03-15T22:33:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 139.26699713599999, 2021-03-15T22:33:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40674566499999998, 2021-03-15T22:33:50.458 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.459 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6960 2021-03-15T22:33:50.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.460 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850305+0000", 2021-03-15T22:33:50.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850305+0000", 2021-03-15T22:33:50.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013730900000000001 2021-03-15T22:33:50.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850442+0000", 2021-03-15T22:33:50.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0299999999999999e-06 2021-03-15T22:33:50.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850448+0000", 2021-03-15T22:33:50.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3971000000000001e-05 2021-03-15T22:33:50.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850472+0000", 2021-03-15T22:33:50.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0040000000000005e-06 2021-03-15T22:33:50.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850481+0000", 2021-03-15T22:33:50.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6959000000000002e-05 2021-03-15T22:33:50.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850528+0000", 2021-03-15T22:33:50.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0190000000000006e-06 2021-03-15T22:33:50.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850536+0000", 2021-03-15T22:33:50.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.39859939 2021-03-15T22:33:50.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.469 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249135+0000", 2021-03-15T22:33:50.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5754999999999994e-05 2021-03-15T22:33:50.469 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.470 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:50.470 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249231+0000", 2021-03-15T22:33:50.470 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067494899999999999 2021-03-15T22:33:50.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.471 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.471 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.471 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249906+0000", 2021-03-15T22:33:50.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7687000000000003e-05 2021-03-15T22:33:50.472 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.472 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.472 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249984+0000", 2021-03-15T22:33:50.472 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015096900000000001 2021-03-15T22:33:50.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:33:50.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.250135+0000", 2021-03-15T22:33:50.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038623450000000001 2021-03-15T22:33:50.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:50.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.253997+0000", 2021-03-15T22:33:50.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6215000000000003e-05 2021-03-15T22:33:50.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254053+0000", 2021-03-15T22:33:50.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027848769999999998 2021-03-15T22:33:50.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.476 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.256838+0000", 2021-03-15T22:33:50.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138264 2021-03-15T22:33:50.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:50.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.256976+0000", 2021-03-15T22:33:50.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3922000000000001e-05 2021-03-15T22:33:50.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257050+0000", 2021-03-15T22:33:50.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40674566499999998 2021-03-15T22:33:50.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.479 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.479 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.480 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6961 7.c 7:37eb566e:::smithi07694878-48:head [write 3183703~749333] snapc 25b=[25b,25a,257,255,24f,24e,24c,24b,24a,248,247,245,243,23e,23d,23c,239,235,234,22d,227,225,218,211,20c,1f7,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e740)", 2021-03-15T22:33:50.480 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.856636+0000", 2021-03-15T22:33:50.480 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 139.26066596999999, 2021-03-15T22:33:50.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40065102200000002, 2021-03-15T22:33:50.481 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.481 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:50.481 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.481 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.482 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.482 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6961 2021-03-15T22:33:50.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.482 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856636+0000", 2021-03-15T22:33:50.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856636+0000", 2021-03-15T22:33:50.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028634199999999998 2021-03-15T22:33:50.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.485 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856922+0000", 2021-03-15T22:33:50.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4699999999999998e-06 2021-03-15T22:33:50.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856926+0000", 2021-03-15T22:33:50.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3896999999999999e-05 2021-03-15T22:33:50.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.487 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856939+0000", 2021-03-15T22:33:50.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3809999999999998e-06 2021-03-15T22:33:50.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856945+0000", 2021-03-15T22:33:50.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0936999999999997e-05 2021-03-15T22:33:50.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856996+0000", 2021-03-15T22:33:50.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1702000000000001e-05 2021-03-15T22:33:50.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.857007+0000", 2021-03-15T22:33:50.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.392417246 2021-03-15T22:33:50.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249425+0000", 2021-03-15T22:33:50.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5692999999999994e-05 2021-03-15T22:33:50.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249510+0000", 2021-03-15T22:33:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020279209999999998 2021-03-15T22:33:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.251538+0000", 2021-03-15T22:33:50.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012340800000000001 2021-03-15T22:33:50.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.251662+0000", 2021-03-15T22:33:50.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022522300000000001 2021-03-15T22:33:50.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:33:50.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.251887+0000", 2021-03-15T22:33:50.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052293959999999999 2021-03-15T22:33:50.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257116+0000", 2021-03-15T22:33:50.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0851e-05 2021-03-15T22:33:50.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:50.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257127+0000", 2021-03-15T22:33:50.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8776000000000002e-05 2021-03-15T22:33:50.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257176+0000", 2021-03-15T22:33:50.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2856999999999999e-05 2021-03-15T22:33:50.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.500 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.500 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:50.500 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257229+0000", 2021-03-15T22:33:50.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7921999999999999e-05 2021-03-15T22:33:50.500 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.501 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.501 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257287+0000", 2021-03-15T22:33:50.501 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40065102200000002 2021-03-15T22:33:50.501 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.502 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.502 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.502 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6962 7.c 7:37eb566e:::smithi07694878-48:head [setxattr _header (58) in=7b,truncate 3933036] snapc 25b=[25b,25a,257,255,24f,24e,24c,24b,24a,248,247,245,243,23e,23d,23c,239,235,234,22d,227,225,218,211,20c,1f7,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e740)", 2021-03-15T22:33:50.503 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.856954+0000", 2021-03-15T22:33:50.503 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 139.26034813999999, 2021-03-15T22:33:50.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40052243700000001, 2021-03-15T22:33:50.503 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.504 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:50.504 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.504 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.504 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.504 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6962 2021-03-15T22:33:50.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.505 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856954+0000", 2021-03-15T22:33:50.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856954+0000", 2021-03-15T22:33:50.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260000000000001e-06 2021-03-15T22:33:50.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856955+0000", 2021-03-15T22:33:50.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7999999999999993e-07 2021-03-15T22:33:50.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856956+0000", 2021-03-15T22:33:50.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5719999999999996e-06 2021-03-15T22:33:50.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856961+0000", 2021-03-15T22:33:50.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2320000000000001e-06 2021-03-15T22:33:50.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856964+0000", 2021-03-15T22:33:50.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6818999999999998e-05 2021-03-15T22:33:50.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.857041+0000", 2021-03-15T22:33:50.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5550000000000002e-06 2021-03-15T22:33:50.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.857046+0000", 2021-03-15T22:33:50.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.39249579499999998 2021-03-15T22:33:50.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249542+0000", 2021-03-15T22:33:50.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3127000000000006e-05 2021-03-15T22:33:50.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:50.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249625+0000", 2021-03-15T22:33:50.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035889419999999999 2021-03-15T22:33:50.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.253214+0000", 2021-03-15T22:33:50.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9442000000000002e-05 2021-03-15T22:33:50.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.253304+0000", 2021-03-15T22:33:50.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013094199999999999 2021-03-15T22:33:50.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:33:50.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.253434+0000", 2021-03-15T22:33:50.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036997000000000002 2021-03-15T22:33:50.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:50.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257134+0000", 2021-03-15T22:33:50.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019671599999999999 2021-03-15T22:33:50.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257331+0000", 2021-03-15T22:33:50.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2697999999999998e-05 2021-03-15T22:33:50.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257364+0000", 2021-03-15T22:33:50.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1310000000000002e-05 2021-03-15T22:33:50.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:50.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257415+0000", 2021-03-15T22:33:50.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1080999999999993e-05 2021-03-15T22:33:50.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.523 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257476+0000", 2021-03-15T22:33:50.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40052243700000001 2021-03-15T22:33:50.524 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.524 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.524 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.525 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6963 7.c 7:37eb566e:::smithi07694878-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e740)", 2021-03-15T22:33:50.525 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.856985+0000", 2021-03-15T22:33:50.525 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 139.260316303, 2021-03-15T22:33:50.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.400735811, 2021-03-15T22:33:50.526 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.526 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:50.526 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.526 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.527 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.527 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6963 2021-03-15T22:33:50.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.527 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856985+0000", 2021-03-15T22:33:50.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.529 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.529 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856985+0000", 2021-03-15T22:33:50.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2699999999999998e-07 2021-03-15T22:33:50.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856986+0000", 2021-03-15T22:33:50.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.023e-06 2021-03-15T22:33:50.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856987+0000", 2021-03-15T22:33:50.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0000000000000002e-06 2021-03-15T22:33:50.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856993+0000", 2021-03-15T22:33:50.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.694e-06 2021-03-15T22:33:50.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856996+0000", 2021-03-15T22:33:50.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1422999999999997e-05 2021-03-15T22:33:50.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.857067+0000", 2021-03-15T22:33:50.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4980000000000006e-06 2021-03-15T22:33:50.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.535 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.857076+0000", 2021-03-15T22:33:50.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.39701107000000002 2021-03-15T22:33:50.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254087+0000", 2021-03-15T22:33:50.537 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118583 2021-03-15T22:33:50.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:50.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254206+0000", 2021-03-15T22:33:50.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003334371 2021-03-15T22:33:50.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.539 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257540+0000", 2021-03-15T22:33:50.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4062999999999998e-05 2021-03-15T22:33:50.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257624+0000", 2021-03-15T22:33:50.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7158999999999996e-05 2021-03-15T22:33:50.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257721+0000", 2021-03-15T22:33:50.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.400735811 2021-03-15T22:33:50.541 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.541 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.542 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.542 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6969 7.3 7:c7825e8f:::smithi07694878-26:head [copy-from ver 0] snapc 25b=[25b,25a,257,255,24f,24e,24c,24b,24a,248,247,245,243,23e,23d,23c,239,235,234,22d,227,225,218,211,20c,1f7,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e740)", 2021-03-15T22:33:50.542 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.979796+0000", 2021-03-15T22:33:50.543 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 139.13750605000001, 2021-03-15T22:33:50.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.28205391499999999, 2021-03-15T22:33:50.543 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.543 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:50.543 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.544 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.544 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.544 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6969 2021-03-15T22:33:50.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.545 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979796+0000", 2021-03-15T22:33:50.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979796+0000", 2021-03-15T22:33:50.546 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6020000000000002e-06 2021-03-15T22:33:50.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979800+0000", 2021-03-15T22:33:50.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2289999999999999e-06 2021-03-15T22:33:50.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.548 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.548 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979803+0000", 2021-03-15T22:33:50.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990999999999999e-05 2021-03-15T22:33:50.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979815+0000", 2021-03-15T22:33:50.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0029999999999997e-06 2021-03-15T22:33:50.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979822+0000", 2021-03-15T22:33:50.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6368000000000001e-05 2021-03-15T22:33:50.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979869+0000", 2021-03-15T22:33:50.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1099999999999992e-06 2021-03-15T22:33:50.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979878+0000", 2021-03-15T22:33:50.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27729234800000002 2021-03-15T22:33:50.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.553 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257170+0000", 2021-03-15T22:33:50.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000218314 2021-03-15T22:33:50.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257389+0000", 2021-03-15T22:33:50.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00446095 2021-03-15T22:33:50.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.261850+0000", 2021-03-15T22:33:50.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.28205391499999999 2021-03-15T22:33:50.556 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.556 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.557 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.557 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6970 7.3 7:c7825e8f:::smithi07694878-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e740)", 2021-03-15T22:33:50.557 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.979836+0000", 2021-03-15T22:33:50.558 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 139.13746533, 2021-03-15T22:33:50.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.28649647299999997, 2021-03-15T22:33:50.558 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.558 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:50.559 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.559 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.559 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.559 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6970 2021-03-15T22:33:50.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.560 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979836+0000", 2021-03-15T22:33:50.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979836+0000", 2021-03-15T22:33:50.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.246e-06 2021-03-15T22:33:50.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979838+0000", 2021-03-15T22:33:50.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7039999999999999e-06 2021-03-15T22:33:50.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.563 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.563 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.563 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979839+0000", 2021-03-15T22:33:50.564 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5820000000000001e-06 2021-03-15T22:33:50.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.564 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.564 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979845+0000", 2021-03-15T22:33:50.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6890000000000003e-06 2021-03-15T22:33:50.565 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.565 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979850+0000", 2021-03-15T22:33:50.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9660999999999998e-05 2021-03-15T22:33:50.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.566 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.567 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979899+0000", 2021-03-15T22:33:50.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4100000000000001e-06 2021-03-15T22:33:50.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:33:50.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979904+0000", 2021-03-15T22:33:50.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27843821899999999 2021-03-15T22:33:50.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.258342+0000", 2021-03-15T22:33:50.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5978999999999999e-05 2021-03-15T22:33:50.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:33:50.570 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.258418+0000", 2021-03-15T22:33:50.570 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035069020000000001 2021-03-15T22:33:50.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.261925+0000", 2021-03-15T22:33:50.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2521000000000005e-05 2021-03-15T22:33:50.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:50.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.262007+0000", 2021-03-15T22:33:50.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041258969999999999 2021-03-15T22:33:50.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.266133+0000", 2021-03-15T22:33:50.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9773999999999997e-05 2021-03-15T22:33:50.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.266233+0000", 2021-03-15T22:33:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9889000000000005e-05 2021-03-15T22:33:50.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.266333+0000", 2021-03-15T22:33:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.28649647299999997 2021-03-15T22:33:50.575 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.576 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7848 7.c 7:3bc1c17e:::smithi07694878-8:head [rollback 2b1] snapc 2b6=[2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,293,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e830)", 2021-03-15T22:33:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.405277+0000", 2021-03-15T22:33:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 49.712024382999999, 2021-03-15T22:33:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035514907999999998, 2021-03-15T22:33:50.577 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:33:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.578 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7848 2021-03-15T22:33:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405277+0000", 2021-03-15T22:33:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405277+0000", 2021-03-15T22:33:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8029999999999996e-06 2021-03-15T22:33:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.581 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405282+0000", 2021-03-15T22:33:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4715e-05 2021-03-15T22:33:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.582 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405307+0000", 2021-03-15T22:33:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9646e-05 2021-03-15T22:33:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.583 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405336+0000", 2021-03-15T22:33:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8090000000000004e-06 2021-03-15T22:33:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.584 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.585 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405346+0000", 2021-03-15T22:33:50.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2713999999999994e-05 2021-03-15T22:33:50.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.585 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.586 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.586 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405429+0000", 2021-03-15T22:33:50.586 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6675999999999994e-05 2021-03-15T22:33:50.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.587 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405526+0000", 2021-03-15T22:33:50.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035266545000000003 2021-03-15T22:33:50.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440792+0000", 2021-03-15T22:33:50.588 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035514907999999998 2021-03-15T22:33:50.588 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.589 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.589 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.589 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7870 7.a 7:53dee572:::smithi07694878-35:head [write 766683~730601,stat] snapc 2b8=[2b8,2b7,2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e833)", 2021-03-15T22:33:50.590 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:03.332947+0000", 2021-03-15T22:33:50.590 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.784355103999999, 2021-03-15T22:33:50.590 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026195552, 2021-03-15T22:33:50.590 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.590 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7870 2021-03-15T22:33:50.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.592 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.592 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.332947+0000", 2021-03-15T22:33:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.332947+0000", 2021-03-15T22:33:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000387222 2021-03-15T22:33:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.333334+0000", 2021-03-15T22:33:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0460000000000001e-06 2021-03-15T22:33:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.333337+0000", 2021-03-15T22:33:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1156e-05 2021-03-15T22:33:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.333348+0000", 2021-03-15T22:33:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5389999999999998e-06 2021-03-15T22:33:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.333354+0000", 2021-03-15T22:33:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017828650000000001 2021-03-15T22:33:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.351182+0000", 2021-03-15T22:33:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9136999999999997e-05 2021-03-15T22:33:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.599 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.351271+0000", 2021-03-15T22:33:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000226386 2021-03-15T22:33:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:33:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.351498+0000", 2021-03-15T22:33:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0053001649999999999 2021-03-15T22:33:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.356798+0000", 2021-03-15T22:33:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000474687 2021-03-15T22:33:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.357273+0000", 2021-03-15T22:33:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001685689 2021-03-15T22:33:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.358958+0000", 2021-03-15T22:33:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2242999999999993e-05 2021-03-15T22:33:50.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:50.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.359021+0000", 2021-03-15T22:33:50.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000121632 2021-03-15T22:33:50.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.359142+0000", 2021-03-15T22:33:50.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.026195552 2021-03-15T22:33:50.606 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.607 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.607 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.607 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7871 7.a 7:53dee572:::smithi07694878-35:head [write 1922192~583372] snapc 2b8=[2b8,2b7,2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e833)", 2021-03-15T22:33:50.608 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:03.337378+0000", 2021-03-15T22:33:50.608 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 46.779923857999997, 2021-03-15T22:33:50.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029327857999999998, 2021-03-15T22:33:50.608 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:33:50.609 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:33:50.609 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:33:50.609 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:33:50.609 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:33:50.609 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7871 2021-03-15T22:33:50.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.610 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:33:50.610 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:33:50.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.337378+0000", 2021-03-15T22:33:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:33:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:33:50.611 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.337378+0000", 2021-03-15T22:33:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00030490299999999998 2021-03-15T22:33:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:33:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.337683+0000", 2021-03-15T22:33:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1939999999999998e-06 2021-03-15T22:33:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:33:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.337686+0000", 2021-03-15T22:33:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8063e-05 2021-03-15T22:33:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.614 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:33:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.337704+0000", 2021-03-15T22:33:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.25e-06 2021-03-15T22:33:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.615 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:33:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.337711+0000", 2021-03-15T22:33:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.019110309999999998 2021-03-15T22:33:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.356822+0000", 2021-03-15T22:33:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000105197 2021-03-15T22:33:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.617 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:33:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.356927+0000", 2021-03-15T22:33:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022825969999999999 2021-03-15T22:33:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:33:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.359209+0000", 2021-03-15T22:33:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8846999999999999e-05 2021-03-15T22:33:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.619 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:33:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.359288+0000", 2021-03-15T22:33:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021475500000000001 2021-03-15T22:33:50.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,5", 2021-03-15T22:33:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.359503+0000", 2021-03-15T22:33:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0046990139999999996 2021-03-15T22:33:50.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.622 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.364202+0000", 2021-03-15T22:33:50.622 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1623999999999999e-05 2021-03-15T22:33:50.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.623 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:33:50.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.364254+0000", 2021-03-15T22:33:50.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0023178690000000002 2021-03-15T22:33:50.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:33:50.624 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.366571+0000", 2021-03-15T22:33:50.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6674999999999993e-05 2021-03-15T22:33:50.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:33:50.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.366648+0000", 2021-03-15T22:33:50.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7559999999999998e-05 2021-03-15T22:33:50.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:33:50.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:33:50.626 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:33:50.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:03.366706+0000", 2021-03-15T22:33:50.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.029327857999999998 2021-03-15T22:33:50.627 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.627 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.627 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.627 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:33:50.628 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:33:50.628 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:33:50.629 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:50 smithi125 conmon[56739]: cluster 2021-03-15T22:33:49.195801+0000 mgr.y 2021-03-15T22:33:50.629 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:50 smithi125 conmon[56739]: (mgr.14144) 995 : cluster [DBG] pgmap v1462: 148 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering, 143 active+clean; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 7 op/s; 2/1800 objects degraded (0.111%); 28/1800 objects misplaced (1.556%); 7.4 MiB/s, 61 objects/s recovering 2021-03-15T22:33:50.629 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:50 smithi125 conmon[56739]: cluster 2021-03-15T22:33:49.476712+0000 mon.a (mon.0) 2224 : cluster [DBG] osdmap e879: 8 total, 8 up, 7 in 2021-03-15T22:33:50.630 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:33:50 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:33:50.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:50.631 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:50.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:50.635 INFO:tasks.rados.rados.0.smithi076.stdout:3311: done (3 left) 2021-03-15T22:33:50.635 INFO:tasks.rados.rados.0.smithi076.stdout:3312: copy_from oid 30 from oid 44 current snap is 389 2021-03-15T22:33:50.635 INFO:tasks.rados.rados.0.smithi076.stdout:3313: write (excl) oid 49 current snap is 389 2021-03-15T22:33:50.636 INFO:tasks.rados.rados.0.smithi076.stdout:3313: seq_num 1124 ranges {491068=483767,1640584=593387,2650385=240683} 2021-03-15T22:33:50.636 INFO:tasks.rados.rados.0.smithi076.stdout:3313: writing smithi07694878-49 from 491068 to 974835 tid 1 2021-03-15T22:33:50.636 INFO:tasks.rados.rados.0.smithi076.stdout:3313: writing smithi07694878-49 from 1640584 to 2233971 tid 2 2021-03-15T22:33:50.636 INFO:tasks.rados.rados.0.smithi076.stdout:3313: writing smithi07694878-49 from 2650385 to 2891068 tid 3 2021-03-15T22:33:50.636 INFO:tasks.rados.rados.0.smithi076.stdout:3314: rollback oid 17 current snap is 389 2021-03-15T22:33:50.637 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 17 to 339 2021-03-15T22:33:50.637 INFO:tasks.rados.rados.0.smithi076.stdout:3307: finishing write tid 3 to smithi07694878-33 2021-03-15T22:33:50.637 INFO:tasks.rados.rados.0.smithi076.stdout:3307: finishing write tid 4 to smithi07694878-33 2021-03-15T22:33:50.637 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 2412 (ObjNum 1122 snap 389 seq_num 1122) dirty exists 2021-03-15T22:33:50.638 INFO:tasks.rados.rados.0.smithi076.stdout:3307: left oid 33 (ObjNum 1122 snap 389 seq_num 1122) 2021-03-15T22:33:50.638 INFO:tasks.rados.rados.0.smithi076.stdout:3308: expect (ObjNum 1076 snap 366 seq_num 1076) 2021-03-15T22:33:50.638 INFO:tasks.rados.rados.0.smithi076.stdout:3313: finishing write tid 1 to smithi07694878-49 2021-03-15T22:33:50.638 INFO:tasks.rados.rados.0.smithi076.stdout:3307: done (5 left) 2021-03-15T22:33:50.638 INFO:tasks.rados.rados.0.smithi076.stdout:3308: done (4 left) 2021-03-15T22:33:50.639 INFO:tasks.rados.rados.0.smithi076.stdout:3310: done (3 left) 2021-03-15T22:33:50.639 INFO:tasks.rados.rados.0.smithi076.stdout:3315: write (excl) oid 23 current snap is 389 2021-03-15T22:33:50.639 INFO:tasks.rados.rados.0.smithi076.stdout:3312: finishing copy_from to smithi07694878-30 2021-03-15T22:33:50.639 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 2098 (ObjNum 1118 snap 388 seq_num 1118) dirty exists 2021-03-15T22:33:50.640 INFO:tasks.rados.rados.0.smithi076.stdout:3312: finishing copy_from racing read to smithi07694878-30 2021-03-15T22:33:50.640 INFO:tasks.rados.rados.0.smithi076.stdout:3313: finishing write tid 2 to smithi07694878-49 2021-03-15T22:33:50.640 INFO:tasks.rados.rados.0.smithi076.stdout:3314: finishing rollback tid 0 to smithi07694878-17 2021-03-15T22:33:50.640 INFO:tasks.rados.rados.0.smithi076.stdout:3314: finishing rollback tid 1 to smithi07694878-17 2021-03-15T22:33:50.640 INFO:tasks.rados.rados.0.smithi076.stdout:3314: finishing rollback tid 2 to smithi07694878-17 2021-03-15T22:33:50.641 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 2619 (ObjNum 960 snap 321 seq_num 960) dirty exists 2021-03-15T22:33:50.641 INFO:tasks.rados.rados.0.smithi076.stdout:3313: finishing write tid 3 to smithi07694878-49 2021-03-15T22:33:50.641 INFO:tasks.rados.rados.0.smithi076.stdout:3313: finishing write tid 4 to smithi07694878-49 2021-03-15T22:33:50.641 INFO:tasks.rados.rados.0.smithi076.stdout:3313: finishing write tid 5 to smithi07694878-49 2021-03-15T22:33:50.641 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 2774 (ObjNum 1124 snap 389 seq_num 1124) dirty exists 2021-03-15T22:33:50.642 INFO:tasks.rados.rados.0.smithi076.stdout:3313: left oid 49 (ObjNum 1124 snap 389 seq_num 1124) 2021-03-15T22:33:50.642 INFO:tasks.rados.rados.0.smithi076.stdout:3315: seq_num 1125 ranges {507875=559016,1699066=737045} 2021-03-15T22:33:50.642 INFO:tasks.rados.rados.0.smithi076.stdout:3315: writing smithi07694878-23 from 507875 to 1066891 tid 1 2021-03-15T22:33:50.642 INFO:tasks.rados.rados.0.smithi076.stdout:3315: writing smithi07694878-23 from 1699066 to 2436111 tid 2 2021-03-15T22:33:50.643 INFO:tasks.rados.rados.0.smithi076.stdout:3312: done (3 left) 2021-03-15T22:33:50.643 INFO:tasks.rados.rados.0.smithi076.stdout:3313: done (2 left) 2021-03-15T22:33:50.643 INFO:tasks.rados.rados.0.smithi076.stdout:3314: done (1 left) 2021-03-15T22:33:50.643 INFO:tasks.rados.rados.0.smithi076.stdout:3316: rollback oid 9 current snap is 389 2021-03-15T22:33:50.643 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 9 to 383 2021-03-15T22:33:50.644 INFO:tasks.rados.rados.0.smithi076.stdout:3315: finishing write tid 1 to smithi07694878-23 2021-03-15T22:33:50.644 INFO:tasks.rados.rados.0.smithi076.stdout:3317: rollback oid 8 current snap is 389 2021-03-15T22:33:50.644 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 381 2021-03-15T22:33:50.644 INFO:tasks.rados.rados.0.smithi076.stdout:3318: rollback oid 27 current snap is 389 2021-03-15T22:33:50.644 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 385 2021-03-15T22:33:50.645 INFO:tasks.rados.rados.0.smithi076.stdout:3319: write (excl) oid 5 current snap is 389 2021-03-15T22:33:50.645 INFO:tasks.rados.rados.0.smithi076.stdout:3319: seq_num 1126 ranges {524682=750618,1757548=597056} 2021-03-15T22:33:50.645 INFO:tasks.rados.rados.0.smithi076.stdout:3319: writing smithi07694878-5 from 524682 to 1275300 tid 1 2021-03-15T22:33:50.645 INFO:tasks.rados.rados.0.smithi076.stdout:3319: writing smithi07694878-5 from 1757548 to 2354604 tid 2 2021-03-15T22:33:50.646 INFO:tasks.rados.rados.0.smithi076.stdout:3320: delete oid 42 current snap is 389 2021-03-15T22:33:50.646 INFO:tasks.rados.rados.0.smithi076.stdout:3318: finishing rollback tid 0 to smithi07694878-27 2021-03-15T22:33:50.646 INFO:tasks.rados.rados.0.smithi076.stdout:3317: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:33:50.646 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 1255 (ObjNum 264886016 snap 21896 seq_num 261594768) dirty dne 2021-03-15T22:33:50.646 INFO:tasks.rados.rados.0.smithi076.stdout:3315: finishing write tid 2 to smithi07694878-23 2021-03-15T22:33:50.647 INFO:tasks.rados.rados.0.smithi076.stdout:3315: finishing write tid 3 to smithi07694878-23 2021-03-15T22:33:50.647 INFO:tasks.rados.rados.0.smithi076.stdout:3315: finishing write tid 4 to smithi07694878-23 2021-03-15T22:33:50.647 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 2399 (ObjNum 1125 snap 389 seq_num 1125) dirty exists 2021-03-15T22:33:50.647 INFO:tasks.rados.rados.0.smithi076.stdout:3315: left oid 23 (ObjNum 1125 snap 389 seq_num 1125) 2021-03-15T22:33:50.648 INFO:tasks.rados.rados.0.smithi076.stdout:3316: finishing rollback tid 0 to smithi07694878-9 2021-03-15T22:33:50.648 INFO:tasks.rados.rados.0.smithi076.stdout:3316: finishing rollback tid 1 to smithi07694878-9 2021-03-15T22:33:50.648 INFO:tasks.rados.rados.0.smithi076.stdout:3316: finishing rollback tid 2 to smithi07694878-9 2021-03-15T22:33:50.648 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 2777 (ObjNum 1092 snap 375 seq_num 1092) dirty exists 2021-03-15T22:33:50.648 INFO:tasks.rados.rados.0.smithi076.stdout:3319: finishing write tid 1 to smithi07694878-5 2021-03-15T22:33:50.649 INFO:tasks.rados.rados.0.smithi076.stdout:3319: finishing write tid 2 to smithi07694878-5 2021-03-15T22:33:50.649 INFO:tasks.rados.rados.0.smithi076.stdout:3319: finishing write tid 3 to smithi07694878-5 2021-03-15T22:33:50.649 INFO:tasks.rados.rados.0.smithi076.stdout:3319: finishing write tid 4 to smithi07694878-5 2021-03-15T22:33:50.649 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1986 (ObjNum 1126 snap 389 seq_num 1126) dirty exists 2021-03-15T22:33:50.649 INFO:tasks.rados.rados.0.smithi076.stdout:3319: left oid 5 (ObjNum 1126 snap 389 seq_num 1126) 2021-03-15T22:33:50.650 INFO:tasks.rados.rados.0.smithi076.stdout:3315: done (5 left) 2021-03-15T22:33:50.650 INFO:tasks.rados.rados.0.smithi076.stdout:3316: done (4 left) 2021-03-15T22:33:50.650 INFO:tasks.rados.rados.0.smithi076.stdout:3317: done (3 left) 2021-03-15T22:33:50.650 INFO:tasks.rados.rados.0.smithi076.stdout:3319: done (2 left) 2021-03-15T22:33:50.651 INFO:tasks.rados.rados.0.smithi076.stdout:3320: done (1 left) 2021-03-15T22:33:50.651 INFO:tasks.rados.rados.0.smithi076.stdout:3321: snap_remove snap 379 2021-03-15T22:33:50.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:50.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:50.756 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:50.756 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:50.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[56514]: cluster 2021-03-15T22:33:49.195801+0000 mgr.y (mgr.14144) 995 : cluster [DBG] pgmap v1462: 148 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering, 143 active+clean; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 7 op/s; 2/1800 objects degraded (0.111%); 28/1800 objects misplaced (1.556%); 7.4 MiB/s, 61 objects/s recovering 2021-03-15T22:33:50.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[56514]: cluster 2021-03-15T22:33:49.476712+0000 mon.a (mon.0) 2224 : cluster [DBG] osdmap e879: 8 total, 8 up, 7 in 2021-03-15T22:33:50.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[62156]: cluster 2021-03-15T22:33:50.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[62156]: 2021-03-15T22:33:50.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[62156]: 2021 2021-03-15T22:33:50.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[62156]: - 2021-03-15T22:33:50.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[62156]: 03 2021-03-15T22:33:50.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[62156]: -15T22:33 2021-03-15T22:33:50.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[62156]: :49.195801+0000 mgr.y (mgr.14144) 995 : cluster [DBG] pgmap v1462: 148 pgs: 1 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering, 143 active+clean; 604 MiB data, 1.5 GiB used, 624 GiB / 626 GiB avail; 2.8 MiB/s rd, 2.0 MiB/s wr, 7 op/s; 2/1800 objects degraded (0.111%); 28/1800 objects misplaced (1.556%); 7.4 MiB/s, 61 objects/s recovering 2021-03-15T22:33:50.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:33:50.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[62156]: -15T22:33:49.476712+0000 mon.a (mon.0) 2224 : cluster 2021-03-15T22:33:50.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:50 smithi076 conmon[62156]: [DBG] osdmap e879: 8 total, 8 up, 7 in 2021-03-15T22:33:50.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:50.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:50.883 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:50.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:50.912 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:33:50.970 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-15T22:33:50.984 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:51.037 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:51.039 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:51.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:51.162 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:51.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:51.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:51.265 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:33:51.265 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:51.268 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:51.290 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:51.290 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:51.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:51.414 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:51.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:51.517 INFO:tasks.rados.rados.0.smithi076.stdout:3321: done (1 left) 2021-03-15T22:33:51.517 INFO:tasks.rados.rados.0.smithi076.stdout:3322: read oid 16 snap -1 2021-03-15T22:33:51.518 INFO:tasks.rados.rados.0.smithi076.stdout:3322: expect (ObjNum 1051 snap 353 seq_num 1051) 2021-03-15T22:33:51.518 INFO:tasks.rados.rados.0.smithi076.stdout:3323: read oid 12 snap -1 2021-03-15T22:33:51.519 INFO:tasks.rados.rados.0.smithi076.stdout:3323: expect (ObjNum 999 snap 335 seq_num 999) 2021-03-15T22:33:51.519 INFO:tasks.rados.rados.0.smithi076.stdout:3324: copy_from oid 21 from oid 46 current snap is 389 2021-03-15T22:33:51.519 INFO:tasks.rados.rados.0.smithi076.stdout:3325: snap_remove snap 377 2021-03-15T22:33:51.520 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:51.522 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:33:51.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:51.559 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:51.559 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:51.661 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:51.683 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:51.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:51.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:51.726 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:51 smithi125 conmon[56739]: audit 2021-03-15T22:33:50.203847+0000 mgr.y (mgr.14144) 996 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:51.727 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:51 smithi125 conmon[56739]: cluster 2021-03-15T22:33:50.482715+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2021-03-15T22:33:51.727 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:51 smithi125 conmon[56739]: audit 2021-03-15T22:33:51.480739+0000 mon.a (mon.0) 2226 : audit [INF] from='client.? 172.21.15.76:0/724857443' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:33:51.757 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[56514]: audit 2021-03-15T22:33:51.758 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[56514]: 2021-03-15T22:33:50 2021-03-15T22:33:51.758 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[56514]: .203847+0000 mgr.y (mgr.14144) 996 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:51.759 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:33:51.759 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[56514]: 03-15T22:33:50.482715+0000 mon.a (mon 2021-03-15T22:33:51.759 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[56514]: .0) 2225 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2021-03-15T22:33:51.759 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[56514]: 2021-03-15T22:33:51.760 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:33:51.760 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[56514]: 03-15T22:33:51.480739+0000 mon.a ( 2021-03-15T22:33:51.760 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[56514]: mon.0) 2226 : audit [INF] from='client.? 172.21.15.76:0/724857443' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:33:51.760 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[56514]: 2021-03-15T22:33:51.761 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[62156]: audit 2021-03-15T22:33:50.203847 2021-03-15T22:33:51.761 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 996 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:33:51.761 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[62156]: cluster 2021-03-15T22:33:51.761 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[62156]: 2021-03-15T22:33:51.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[62156]: 2021-03-15T22:33: 2021-03-15T22:33:51.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[62156]: 50.482715+0000 mon.a (mon.0) 2225 : cluster [DBG] osdmap e880: 8 total, 8 up, 7 in 2021-03-15T22:33:51.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[62156]: audit 2021-03-15T22:33:51.762 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[62156]: 2021-03-15T22:33:51.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[62156]: 2021 2021-03-15T22:33:51.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[62156]: -03-15T22:33:51 2021-03-15T22:33:51.763 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:51 smithi076 conmon[62156]: .480739+0000 mon.a (mon.0) 2226 : audit [INF] from='client.? 172.21.15.76:0/724857443' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:33:51.787 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:51.810 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:51.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:51.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:51.913 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:51.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:51.937 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:51.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:51.938 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 7, 0, 3, 5, 6] out_osds: [2] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:33:51.938 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:33:51.939 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2021-03-15T22:33:51.939 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight-by-utilization 1000 3.0 2021-03-15T22:33:52.038 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:52.061 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:52.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:52.063 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:52.164 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:52.185 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:52.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:52.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:52.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:52.311 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:52.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:52.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:52.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:52.436 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:52.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:52.437 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:52.518 INFO:tasks.rados.rados.0.smithi076.stdout:3325: done (4 left) 2021-03-15T22:33:52.519 INFO:tasks.rados.rados.0.smithi076.stdout:3326: read oid 37 snap -1 2021-03-15T22:33:52.519 INFO:tasks.rados.rados.0.smithi076.stdout:3326: expect (ObjNum 1108 snap 385 seq_num 1108) 2021-03-15T22:33:52.519 INFO:tasks.rados.rados.0.smithi076.stdout:3327: rollback oid 13 current snap is 389 2021-03-15T22:33:52.520 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 13 to 362 2021-03-15T22:33:52.520 INFO:tasks.rados.rados.0.smithi076.stdout:3328: delete oid 34 current snap is 389 2021-03-15T22:33:52.520 INFO:tasks.rados.rados.0.smithi076.stdout:3322: expect (ObjNum 1051 snap 353 seq_num 1051) 2021-03-15T22:33:52.539 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:52.560 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:52.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:52.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:52.662 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:52.685 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:52.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:52.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:52.713 INFO:tasks.rados.rados.0.smithi076.stdout:3323: expect (ObjNum 999 snap 335 seq_num 999) 2021-03-15T22:33:52.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:52 smithi076 conmon[56514]: cluster 2021-03-15T22:33:51.196306+0000 mgr.y (mgr.14144) 997 : cluster [DBG] pgmap v1465: 148 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 143 active+clean; 614 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 10 op/s; 4/1812 objects degraded (0.221%); 57/1812 objects misplaced (3.146%); 3.5 MiB/s, 34 objects/s recovering 2021-03-15T22:33:52.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:52 smithi076 conmon[56514]: cluster 2021-03-15T22:33:51.494478+0000 mon.a (mon.0) 2227 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:52.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:52 smithi076 conmon[56514]: audit 2021-03-15T22:33:51.508700+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? 172.21.15.76:0/724857443' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:33:52.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:52 smithi076 conmon[56514]: cluster 2021-03-15T22:33:51.508792+0000 mon.a (mon.0) 2229 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2021-03-15T22:33:52.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:52 smithi076 conmon[62156]: cluster 2021-03-15T22:33:51.196306+0000 mgr.y (mgr.14144) 997 : cluster [DBG] pgmap v1465: 148 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 143 active+clean; 614 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 10 op/s; 4/1812 objects degraded (0.221%); 57/1812 objects misplaced (3.146%); 3.5 MiB/s, 34 objects/s recovering 2021-03-15T22:33:52.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:52 smithi076 conmon[62156]: cluster 2021-03-15T22:33:51.494478+0000 mon.a (mon.0) 2227 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:52.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:52 smithi076 conmon[62156]: audit 2021-03-15T22:33:51.508700+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? 172.21.15.76:0/724857443' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:33:52.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:52 smithi076 conmon[62156]: cluster 2021-03-15T22:33:51.508792+0000 mon.a (mon.0) 2229 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2021-03-15T22:33:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:52 smithi125 conmon[56739]: cluster 2021-03-15T22:33:51.196306+0000 mgr.y (mgr.14144) 997 : cluster [DBG] pgmap v1465: 148 pgs: 1 active+recovering+undersized+remapped, 2 active+recovering+undersized+degraded+remapped, 2 active+undersized+remapped, 143 active+clean; 614 MiB data, 1.6 GiB used, 624 GiB / 626 GiB avail; 1.4 MiB/s rd, 2.5 MiB/s wr, 10 op/s; 4/1812 objects degraded (0.221%); 57/1812 objects misplaced (3.146%); 3.5 MiB/s, 34 objects/s recovering 2021-03-15T22:33:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:52 smithi125 conmon[56739]: cluster 2021-03-15T22:33:51.494478+0000 mon.a (mon.0) 2227 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:33:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:52 smithi125 conmon[56739]: audit 2021-03-15T22:33:51.508700+0000 mon.a (mon.0) 2228 : audit [INF] from='client.? 172.21.15.76:0/724857443' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:33:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:52 smithi125 conmon[56739]: cluster 2021-03-15T22:33:51.508792+0000 mon.a (mon.0) 2229 : cluster [DBG] osdmap e881: 8 total, 8 up, 7 in 2021-03-15T22:33:52.788 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:52.809 INFO:tasks.rados.rados.0.smithi076.stdout:3324: finishing copy_from to smithi07694878-21 2021-03-15T22:33:52.809 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 2279 (ObjNum 1044 snap 350 seq_num 1044) dirty exists 2021-03-15T22:33:52.810 INFO:tasks.rados.rados.0.smithi076.stdout:3324: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:33:52.810 INFO:tasks.rados.rados.0.smithi076.stdout:3318: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:33:52.810 INFO:tasks.rados.rados.0.smithi076.stdout:3318: finishing rollback tid 2 to smithi07694878-27 2021-03-15T22:33:52.810 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 2101 (ObjNum 1091 snap 375 seq_num 1091) dirty exists 2021-03-15T22:33:52.811 INFO:tasks.rados.rados.0.smithi076.stdout:3318: done (6 left) 2021-03-15T22:33:52.811 INFO:tasks.rados.rados.0.smithi076.stdout:3322: done (5 left) 2021-03-15T22:33:52.811 INFO:tasks.rados.rados.0.smithi076.stdout:3323: done (4 left) 2021-03-15T22:33:52.811 INFO:tasks.rados.rados.0.smithi076.stdout:3324: done (3 left) 2021-03-15T22:33:52.811 INFO:tasks.rados.rados.0.smithi076.stdout:3328: done (2 left) 2021-03-15T22:33:52.812 INFO:tasks.rados.rados.0.smithi076.stdout:3329: write oid 31 current snap is 389 2021-03-15T22:33:52.812 INFO:tasks.rados.rados.0.smithi076.stdout:3327: finishing rollback tid 1 to smithi07694878-13 2021-03-15T22:33:52.812 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1997 (ObjNum 264887424 snap 21896 seq_num 262635312) dirty dne 2021-03-15T22:33:52.812 INFO:tasks.rados.rados.0.smithi076.stdout:3329: seq_num 1129 ranges {575103=576365,1932994=744383} 2021-03-15T22:33:52.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:52.814 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:52.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:52.815 INFO:tasks.rados.rados.0.smithi076.stdout:3329: writing smithi07694878-31 from 575103 to 1151468 tid 1 2021-03-15T22:33:52.825 INFO:tasks.rados.rados.0.smithi076.stdout:3329: writing smithi07694878-31 from 1932994 to 2677377 tid 2 2021-03-15T22:33:52.825 INFO:tasks.rados.rados.0.smithi076.stdout:3327: done (2 left) 2021-03-15T22:33:52.825 INFO:tasks.rados.rados.0.smithi076.stdout:3330: delete oid 25 current snap is 389 2021-03-15T22:33:52.826 INFO:tasks.rados.rados.0.smithi076.stdout:3326: expect (ObjNum 1108 snap 385 seq_num 1108) 2021-03-15T22:33:52.847 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:52.916 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:52.925 INFO:tasks.rados.rados.0.smithi076.stdout:3329: finishing write tid 1 to smithi07694878-31 2021-03-15T22:33:52.926 INFO:tasks.rados.rados.0.smithi076.stdout:3329: finishing write tid 2 to smithi07694878-31 2021-03-15T22:33:52.926 INFO:tasks.rados.rados.0.smithi076.stdout:3326: done (2 left) 2021-03-15T22:33:52.926 INFO:tasks.rados.rados.0.smithi076.stdout:3330: done (1 left) 2021-03-15T22:33:52.926 INFO:tasks.rados.rados.0.smithi076.stdout:3331: write (excl) oid 48 current snap is 389 2021-03-15T22:33:52.927 INFO:tasks.rados.rados.0.smithi076.stdout:3329: finishing write tid 3 to smithi07694878-31 2021-03-15T22:33:52.927 INFO:tasks.rados.rados.0.smithi076.stdout:3329: finishing write tid 4 to smithi07694878-31 2021-03-15T22:33:52.927 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 2417 (ObjNum 1129 snap 389 seq_num 1129) dirty exists 2021-03-15T22:33:52.927 INFO:tasks.rados.rados.0.smithi076.stdout:3329: left oid 31 (ObjNum 1129 snap 389 seq_num 1129) 2021-03-15T22:33:52.928 INFO:tasks.rados.rados.0.smithi076.stdout:3331: seq_num 1131 ranges {608717=726863,1933605=748052} 2021-03-15T22:33:52.932 INFO:tasks.rados.rados.0.smithi076.stdout:3331: writing smithi07694878-48 from 608717 to 1335580 tid 1 2021-03-15T22:33:52.939 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:52.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:52.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:52.941 INFO:tasks.rados.rados.0.smithi076.stdout:3331: writing smithi07694878-48 from 1933605 to 2681657 tid 2 2021-03-15T22:33:52.942 INFO:tasks.rados.rados.0.smithi076.stdout:3329: done (1 left) 2021-03-15T22:33:52.942 INFO:tasks.rados.rados.0.smithi076.stdout:3332: snap_create 2021-03-15T22:33:52.942 INFO:tasks.rados.rados.0.smithi076.stdout:3331: finishing write tid 1 to smithi07694878-48 2021-03-15T22:33:52.944 INFO:tasks.rados.rados.0.smithi076.stdout:3331: finishing write tid 2 to smithi07694878-48 2021-03-15T22:33:52.944 INFO:tasks.rados.rados.0.smithi076.stdout:3331: finishing write tid 3 to smithi07694878-48 2021-03-15T22:33:52.944 INFO:tasks.rados.rados.0.smithi076.stdout:3331: finishing write tid 4 to smithi07694878-48 2021-03-15T22:33:52.945 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 2402 (ObjNum 1131 snap 389 seq_num 1131) dirty exists 2021-03-15T22:33:52.945 INFO:tasks.rados.rados.0.smithi076.stdout:3331: left oid 48 (ObjNum 1131 snap 389 seq_num 1131) 2021-03-15T22:33:53.042 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:53.065 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:53.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:53.067 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:53.116 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:33:53.117 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:33:53.117 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:33:53.117 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:33:53.168 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:53.193 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:53.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:53.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:53.269 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:33:53.287 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:33:53.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:53.306 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:33:53.318 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:53.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:53.319 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:53.325 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:33:53.343 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:33:53.360 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:33:53.377 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:33:53.395 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:33:53.396 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:33:53.397 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:33:53.397 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:33:53.397 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:33:53.397 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:33:53.398 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:33:53.398 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:33:53.398 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:33:53.421 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:53.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:53.444 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:53.445 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:53.519 INFO:tasks.rados.rados.0.smithi076.stdout:3331: done (1 left) 2021-03-15T22:33:53.519 INFO:tasks.rados.rados.0.smithi076.stdout:3332: done (0 left) 2021-03-15T22:33:53.519 INFO:tasks.rados.rados.0.smithi076.stdout:3333: delete oid 3 current snap is 390 2021-03-15T22:33:53.546 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:53.571 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:53.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:53.591 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:53 smithi076 conmon[56514]: cluster 2021-03-15T22:33:52.510547+0000 mon.a (mon.0) 2021-03-15T22:33:53.591 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:53 smithi076 conmon[56514]: 2230 : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2021-03-15T22:33:53.592 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:53 smithi076 conmon[56514]: audit 2021-03-15T22:33:53.248361+0000 mon.a (mon.0) 2231 : audit [DBG] from='client.? 172.21.15.76:0/1522195549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:53.592 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:53 smithi076 conmon[62156]: cluster 2021-03-15T22:33:52.510547+0000 mon.a (mon.0) 2230 : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2021-03-15T22:33:53.592 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:53 smithi076 conmon[62156]: audit 2021-03-15T22:33:53.248361+0000 mon.a (mon.0) 2231 : audit [DBG] from='client.? 172.21.15.76:0/1522195549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:53.674 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:53.696 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:53.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:53.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:53.749 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-15T22:33:53.771 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:53 smithi125 conmon[56739]: cluster 2021-03-15T22:33:52.510547+0000 mon.a (mon.0) 2230 2021-03-15T22:33:53.771 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:53 smithi125 conmon[56739]: : cluster [DBG] osdmap e882: 8 total, 8 up, 7 in 2021-03-15T22:33:53.772 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:53 smithi125 conmon[56739]: audit 2021-03-15T22:33:53.248361+0000 mon.a (mon.0) 2231 : audit [DBG] from='client.? 172.21.15.76:0/1522195549' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:33:53.798 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:53.820 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:53.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:53.822 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:53.923 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:53.946 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:53.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:53.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:54.024 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:33:54.048 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:54.072 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:54.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:54.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:54.174 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:54.196 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:54.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:54.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:54.298 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:54.320 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:54.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:54.321 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:54.422 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:54.445 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:54.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:54.446 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:54.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:54.571 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:54.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:54.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:54.673 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:54.695 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:54.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:54.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:54.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:54 smithi076 conmon[62156]: cluster 2021-03-15T22:33:53.196762+0000 mgr.y (mgr.14144) 998 : cluster [DBG] pgmap v1468: 148 pgs: 1 active+clean+wait, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 144 active+clean; 625 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 19 op/s; 2/1827 objects degraded (0.109%); 33/1827 objects misplaced (1.806%); 25 MiB/s, 54 objects/s recovering 2021-03-15T22:33:54.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:54 smithi076 conmon[62156]: cluster 2021-03-15T22:33:53.507215+0000 mon.a (mon.0) 2232 : cluster [WRN] Health check update: Degraded data redundancy: 2/1827 objects degraded (0.109%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:33:54.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:54 smithi076 conmon[62156]: cluster 2021-03-15T22:33:53.515384+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2021-03-15T22:33:54.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:54 smithi076 conmon[56514]: cluster 2021-03-15T22:33:53.196762+0000 mgr.y (mgr.14144) 998 : cluster [DBG] pgmap v1468: 148 pgs: 1 active+clean+wait, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 144 active+clean; 625 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 19 op/s; 2/1827 objects degraded (0.109%); 33/1827 objects misplaced (1.806%); 25 MiB/s, 54 objects/s recovering 2021-03-15T22:33:54.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:54 smithi076 conmon[56514]: cluster 2021-03-15T22:33:53.507215+0000 mon.a (mon.0) 2232 : cluster [WRN] Health check update: Degraded data redundancy: 2/1827 objects degraded (0.109%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:33:54.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:54 smithi076 conmon[56514]: cluster 2021-03-15T22:33:53.515384+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2021-03-15T22:33:54.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:54 smithi125 conmon[56739]: cluster 2021-03-15T22:33:53.196762+0000 mgr.y (mgr.14144) 998 : cluster [DBG] pgmap v1468: 148 pgs: 1 active+clean+wait, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 144 active+clean; 625 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 1.4 MiB/s rd, 4.5 MiB/s wr, 19 op/s; 2/1827 objects degraded (0.109%); 33/1827 objects misplaced (1.806%); 25 MiB/s, 54 objects/s recovering 2021-03-15T22:33:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:54 smithi125 conmon[56739]: cluster 2021-03-15T22:33:53.507215+0000 mon.a (mon.0) 2232 : cluster [WRN] Health check update: Degraded data redundancy: 2/1827 objects degraded (0.109%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:33:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:54 smithi125 conmon[56739]: cluster 2021-03-15T22:33:53.515384+0000 mon.a (mon.0) 2233 : cluster [DBG] osdmap e883: 8 total, 8 up, 7 in 2021-03-15T22:33:54.797 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:54.819 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:54.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:54.820 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:54.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:33:54.921 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:54.942 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:54.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:54.944 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:55.045 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:55.070 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:55.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:55.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:55.150 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:55.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:55.196 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:55.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:55.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:55.299 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:55.320 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:55.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:55.322 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:55.328 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 444 (0%) 2021-03-15T22:33:55.329 INFO:teuthology.orchestra.run.smithi076.stdout:avg 63.4286 2021-03-15T22:33:55.329 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 5.55125 -> 5.55125 (expected baseline 7.37342) 2021-03-15T22:33:55.329 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.5 with 55 -> 55 pgs (0.867117 -> 0.867117 * mean) 2021-03-15T22:33:55.329 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 71 -> 71 pgs (1.11937 -> 1.11937 * mean) 2021-03-15T22:33:55.330 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:33:55.330 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:33:55.330 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 3 2021-03-15T22:33:55.331 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:33:55.331 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0027 2021-03-15T22:33:55.331 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0270 2021-03-15T22:33:55.332 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:33:55.423 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:55.450 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:55.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:55.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:55.552 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:55.577 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:55.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:55.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:55.631 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56514]: audit 2021-03-15T22:33:55.234768+0000 mon.a ( 2021-03-15T22:33:55.631 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56514]: mon.0) 2234 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:33:55.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[62156]: audit 2021-03-15T22:33:55.234768+0000 mon.a (mon.0 2021-03-15T22:33:55.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[62156]: ) 2234 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:33:55.633 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: debug 2021-03-15T22:33:55.535+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:33:55.633 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:33:55.633 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:55.633 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:33:55.634 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:55.634 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:33:55.634 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:55.634 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:55.635 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:33:55.635 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: debug 2021-03-15T22:33:55.535+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:33:55.635 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:33:55.635 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:55.635 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:33:55.636 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:55.636 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:33:55.636 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:55.636 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:55.636 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:33:55.637 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: debug 2021-03-15T22:33:55.536+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:33:55.637 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:33:55.637 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:55.637 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:33:55.638 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:55.638 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:33:55.638 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:55.638 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:55.638 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:33:55.639 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: debug 2021-03-15T22:33:55.536+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:33:55.639 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:33:55.639 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:55.639 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:33:55.640 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:55.640 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:33:55.640 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:55.640 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:55.640 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:33:55.641 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: debug 2021-03-15T22:33:55.537+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:33:55.641 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:33:55.641 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:55.641 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:33:55.642 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:55.642 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:33:55.642 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:55.642 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:55.642 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:33:55.643 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: debug 2021-03-15T22:33:55.537+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:33:55.643 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:33:55.643 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:55.643 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:33:55.644 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:55.644 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:33:55.644 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:55.644 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:55.644 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:33:55.645 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: debug 2021-03-15T22:33:55.537+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:33:55.645 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:33:55.645 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:55.645 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:33:55.645 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:55.646 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:33:55.646 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:55.646 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:55.646 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:33:55 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:33:55.668 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:33:55.669 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:33:55.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:55.701 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:55.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:55.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:55.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:55 smithi125 conmon[56739]: audit 2021-03-15T22:33:55.234768+0000 mon.a (mon.0) 2021-03-15T22:33:55.743 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:55 smithi125 conmon[56739]: 2234 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:33:55.803 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:55.825 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:55.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:55.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:55.927 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:55.950 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:55.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:55.951 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:56.052 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:56.076 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:56.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:56.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:56.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:56.200 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:56.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:56.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:56.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-15T22:33:56.303 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:56.313 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '0.05', 'overage': '1000', 'type': 'test-reweight-by-utilization'} 2021-03-15T22:33:56.314 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd test-reweight-by-utilization 1000 0.05 2021-03-15T22:33:56.325 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:56.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:56.327 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:56.429 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:56.451 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:56.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:56.453 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:56.545 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:33:56.554 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:56.577 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:56.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:56.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:56.679 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:56.702 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:56.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:56.704 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:56.764 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:33:56.781 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: cluster 2021-03-15T22:33:55.197379+0000 mgr.y (mgr.14144) 999 : cluster [DBG] pgmap v1470: 148 pgs: 1 active+clean+wait, 147 active+clean; 629 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 4.0 MiB/s wr, 21 op/s; 30 MiB/s, 38 objects/s recovering 2021-03-15T22:33:56.781 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: audit 2021-03-15T22:33:55.326379+0000 mgr.y (mgr.14144) 1000 : audit [DBG] from='client.16593 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:33:56.781 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: cluster 2021-03-15T22:33:55.513832+0000 mon.a (mon.0) 2235 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/1827 objects degraded (0.109%), 1 pg degraded) 2021-03-15T22:33:56.782 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:33:55.529574+0000 mgr.y (mgr.14144) 1001 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:33:56.782 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: audit 2021-03-15T22:33:55.535112+0000 mon.a (mon.0) 2236 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:33:56.782 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:33:55.536021+0000 mgr.y (mgr.14144) 1002 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:33:56.782 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:33:56.783 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.783 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:33:56.783 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.783 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:33:56.784 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.784 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.784 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:33:56.784 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:33:55.536490+0000 mgr.y (mgr.14144) 1003 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:33:56.784 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:33:56.785 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.785 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:33:56.785 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.785 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:33:56.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:33:56.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:33:55.536916+0000 mgr.y (mgr.14144) 1004 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:33:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:33:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:33:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:33:56.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:33:56.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:33:55.537313+0000 mgr.y (mgr.14144) 1005 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:33:56.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:33:56.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:33:56.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:33:56.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:33:56.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:33:55.537735+0000 mgr.y (mgr.14144) 1006 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:33:56.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:33:56.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:33:56.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:33:56.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:33:56.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:33:55.538136+0000 mgr.y (mgr.14144) 1007 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:33:56.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:33:56.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:33:56.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:33:56.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:33:56.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:33:55.538530+0000 mgr.y (mgr.14144) 1008 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:33:56.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:33:56.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:33:56.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:33:56.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:33:56.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:56 smithi125 conmon[56739]: audit 2021-03-15T22:33:56.102612+0000 mon.a (mon.0) 2237 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:33:56.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: cluster 2021-03-15T22:33:55.197379+0000 mgr.y (mgr.14144) 999 : cluster [DBG] pgmap v1470: 148 pgs: 1 active+clean+wait, 147 active+clean; 629 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 4.0 MiB/s wr, 21 op/s; 30 MiB/s, 38 objects/s recovering 2021-03-15T22:33:56.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: audit 2021-03-15T22:33:55.326379+0000 mgr.y (mgr.14144) 1000 : audit [DBG] from='client.16593 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:33:56.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: cluster 2021-03-15T22:33:55.513832+0000 mon.a (mon.0) 2235 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/1827 objects degraded (0.109%), 1 pg degraded) 2021-03-15T22:33:56.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:33:55.529574+0000 mgr.y (mgr.14144) 1001 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:33:56.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: audit 2021-03-15T22:33:55.535112+0000 mon.a (mon.0) 2236 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:33:56.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:33:55.536021+0000 mgr.y (mgr.14144) 1002 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:33:56.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:33:56.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:33:56.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:33:56.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:33:56.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:33:55.536490+0000 mgr.y (mgr.14144) 1003 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:33:56.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:33:56.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:33:56.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:33:56.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:33:56.802 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: cluster 2021-03-15T22:33:55. 2021-03-15T22:33:56.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: 197379+0000 mgr.y (mgr.14144) 999 : cluster [DBG] pgmap v1470: 148 pgs: 1 active+clean+wait, 147 active+clean; 629 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 7.3 MiB/s rd, 4.0 MiB/s wr, 21 op/s; 30 MiB/s, 38 objects/s recovering 2021-03-15T22:33:56.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: audit 2021-03-15T22:33:56.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: 2021-03-15T22:33:55.326379+0000 mgr.y (mgr.14144 2021-03-15T22:33:56.803 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: ) 1000 : audit [DBG] from='client.16593 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:33:56.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: cluster 2021-03-15T22:33:56.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:33:56.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: 22:33:55.513832+0000 mon.a (mon.0) 2235 2021-03-15T22:33:56.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/1827 objects degraded (0.109%), 1 pg degraded) 2021-03-15T22:33:56.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:33:56.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: -03-15T22:33:55.529574+0000 mgr.y ( 2021-03-15T22:33:56.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: mgr.14144) 1001 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:33:56.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: audit 2021-03-15T22:33:56.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: 2021-03-15T22:33:55.535112+0000 mon.a 2021-03-15T22:33:56.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: (mon.0) 2236 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:33:56.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:33:56.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: 2021-03-15T22:33:55.536021+0000 2021-03-15T22:33:56.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: mgr.y (mgr.14144) 1002 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:33:56.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:33:56.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:33:56.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:33:56.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:33:56.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:33:55.536490+0000 mgr.y (mgr.14144) 1003 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:33:56.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:33:56.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:33:56.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:33:56.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:33:55.536916+0000 mgr.y (mgr.14144) 1004 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:33:56.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:33:56.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:33:56.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:33:56.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:33:56.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:33:55 2021-03-15T22:33:56.813 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: .537313+0000 mgr.y (mgr.14144) 1005 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:33:56.813 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:33:56.813 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.813 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:33:56.813 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.814 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:33:56.814 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.814 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.814 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:33:56.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:33:55.537735+0000 mgr.y (mgr.14144) 1006 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:33:56.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:33:56.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:33:56.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:33:56.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:33:56.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:33:55.538136+0000 mgr.y (mgr.14144) 1007 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:33:56.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:33:56.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:33:56.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:33:56.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:33:56.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:33:55.538530+0000 mgr.y (mgr.14144) 1008 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:33:56.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:33:56.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:33:56.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.820 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:33:56.820 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.820 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.820 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:33:56.821 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[56514]: audit 2021-03-15T22:33:56.102612+0000 mon.a (mon.0) 2237 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:33:56.821 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.821 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:33:56.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:33:56.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: 2021-03-15T22:33:55.536916+0000 mgr.y (mgr.14144) 1004 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:33:56.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:33:56.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:33:56.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:33:56.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:33:56.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:33:55.537313+0000 mgr.y (mgr. 2021-03-15T22:33:56.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: 14144) 1005 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:33:56.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:33:56.825 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.825 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:33:56.825 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.825 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:33:56.826 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.826 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.826 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:33:56.826 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: cephadm 2021-03-15T22 2021-03-15T22:33:56.826 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: :33:55.537735+0000 mgr.y (mgr.14144) 1006 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:33:56.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:33:56.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:33:56.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:33:56.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:33:56.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: cephadm 2021- 2021-03-15T22:33:56.829 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: 03-15T22:33:55.538136+0000 mgr.y (mgr.14144) 1007 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:33:56.829 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:33:56.829 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.829 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:33:56.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:33:56.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:33:56.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:33:55 2021-03-15T22:33:56.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: .538530+0000 mgr.y (mgr.14144) 1008 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:33:56.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:33:56.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:33:56.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:33:56.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:33:56.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:33:56.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:33:56.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:33:56.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:33:56.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: audit 2021-03-15T22:33:56.102612+0000 mon.a (mon.0) 2021-03-15T22:33:56.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:56 smithi076 conmon[62156]: 2237 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:33:56.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:56.856 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:56.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:56.857 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:56.958 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:56.980 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:56.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:57.071 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:57.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:57.110 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:57.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:57.112 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:57.121 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:33:57.213 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:57.235 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:57.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:57.236 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:57.337 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:57.359 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:57.360 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:57.461 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:57.483 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:57.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:57.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:57.585 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:57.592 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:57 smithi076 conmon[56514]: cluster 2021-03-15T22:33:56.322330+0000 2021-03-15T22:33:57.592 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:57 smithi076 conmon[56514]: osd.7 (osd.7) 167 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:57.592 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:57 smithi076 conmon[56514]: cluster 2021-03-15T22:33:56.324046+0000 osd.7 (osd.7) 168 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:57.592 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:57 smithi076 conmon[56514]: cluster 2021-03-15T22:33:57.105126+0000 mon.a (mon.0) 2238 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2021-03-15T22:33:57.593 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:57 smithi076 conmon[62156]: cluster 2021-03-15T22:33:56. 2021-03-15T22:33:57.593 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:57 smithi076 conmon[62156]: 322330+0000 osd.7 (osd.7) 167 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:57.593 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:57 smithi076 conmon[62156]: cluster 2021-03-15T22:33:56.324046+0000 osd.7 (osd.7) 168 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:57.594 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:57 smithi076 conmon[62156]: cluster 2021-03-15T22:33:57.105126+0000 mon.a (mon.0) 2238 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2021-03-15T22:33:57.609 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:57.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:57.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:57.711 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:33:57.734 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:57.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:57.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:33:57.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:57 smithi125 conmon[56739]: cluster 2021-03-15T22:33:56.322330+0000 osd.7 (osd.7) 167 2021-03-15T22:33:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:57 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:57 smithi125 conmon[56739]: cluster 2021-03-15T22:33:56.324046+0000 osd.7 (osd.7) 168 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:57 smithi125 conmon[56739]: cluster 2021-03-15T22:33:57.105126+0000 mon.a (mon.0) 2238 : cluster [DBG] osdmap e884: 8 total, 8 up, 7 in 2021-03-15T22:33:57.837 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:33:57.859 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:57.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:33:57.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:57.984 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:57.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:57.985 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:58.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:58.110 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:58.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:58.111 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:58.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:58.236 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:58.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:58.237 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:58.251 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:33:58.251 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:33:58.338 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:58.360 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:58.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:58.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:58.462 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:58.485 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:58.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:58.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:58.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:58.610 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:58.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:58.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:58.712 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:58.734 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:58.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:58.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:58.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:58 smithi076 conmon[56514]: cluster 2021-03-15T22:33:57.197773+0000 mgr.y (mgr.14144) 1009 : cluster [DBG] pgmap v1472: 148 pgs: 1 active+clean+wait, 147 active+clean; 630 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.4 MiB/s wr, 15 op/s; 25 MiB/s, 30 objects/s recovering 2021-03-15T22:33:58.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:58 smithi076 conmon[62156]: cluster 2021-03-15T22:33:57.197773 2021-03-15T22:33:58.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:58 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 1009 : cluster [DBG] pgmap v1472: 148 pgs: 1 active+clean+wait, 147 active+clean; 630 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.4 MiB/s wr, 15 op/s; 25 MiB/s, 30 objects/s recovering 2021-03-15T22:33:58.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:58 smithi125 conmon[56739]: cluster 2021-03-15T22:33:57.197773+0000 mgr.y (mgr.14144) 1009 : cluster [DBG] pgmap v1472: 148 pgs: 1 active+clean+wait, 147 active+clean; 630 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 7.9 MiB/s rd, 2.4 MiB/s wr, 15 op/s; 25 MiB/s, 30 objects/s recovering 2021-03-15T22:33:58.837 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:58.858 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:58.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:58.859 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:58.889 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-15T22:33:58.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:33:58.994 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:58.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:58.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:33:59.097 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:59.120 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:59.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:59.121 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:59.204 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:33:59.222 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:33:59.246 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:59.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:59.248 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:33:59.349 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:59.372 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:59.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:59.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:59.475 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:33:59.498 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:59.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:59.499 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:33:59.542 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:33:59.601 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:33:59.604 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[56514]: audit 2021 2021-03-15T22:33:59.604 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[56514]: -03-15T22:33:58.672752+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? 172.21.15.76:0/3516555532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:59.605 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[56514]: cluster 2021-03-15T22:33:58.879321+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:59.605 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[56514]: cluster 2021-03-15T22:33:58.881191+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:59.605 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[62156]: audit 2021-03-15T22:33:59.606 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[62156]: 2021-03-15T22:33:59.606 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[62156]: 2021 2021-03-15T22:33:59.606 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[62156]: -03-15T22:33:58.672752+0000 mon.a (mon.0) 2239 : audit [INF] from='client.? 172.21.15.76:0/3516555532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:59.606 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[62156]: cluster 2021-03-15T22:33:59.607 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[62156]: 2021-03-15T22:33:58.879321+0000 osd.0 2021-03-15T22:33:59.607 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[62156]: (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:59.607 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:33:59 smithi076 conmon[62156]: cluster 2021-03-15T22:33:58.881191+0000 osd.0 (osd.0) 144 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:33:59.627 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:33:59.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:59.628 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:33:59.666 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:59 smithi125 conmon[56739]: audit 2021-03-15T22:33:59.667 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:59 smithi125 conmon[56739]: 2021-03-15T22:33:58.672752+0000 mon.a (mon.0 2021-03-15T22:33:59.667 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:59 smithi125 conmon[56739]: ) 2239 : audit [INF] from='client.? 172.21.15.76:0/3516555532' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:33:59.667 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:59 smithi125 conmon[56739]: cluster 2021-03-15T22:33:58 2021-03-15T22:33:59.667 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:59 smithi125 conmon[56739]: .879321+0000 osd.0 (osd.0) 143 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:33:59.667 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:59 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:33:59.668 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:59 smithi125 conmon[56739]: 03-15T22:33:58.881191+0000 osd.0 (osd.0) 144 : cluster [DBG] 2021-03-15T22:33:59.668 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:33:59 smithi125 conmon[56739]: purged_snaps scrub ok 2021-03-15T22:33:59.729 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 444 (0%) 2021-03-15T22:33:59.729 INFO:teuthology.orchestra.run.smithi076.stdout:avg 63.4286 2021-03-15T22:33:59.729 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 5.55125 -> 5.55125 (expected baseline 7.37342) 2021-03-15T22:33:59.729 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.5 with 55 -> 55 pgs (0.867117 -> 0.867117 * mean) 2021-03-15T22:33:59.730 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.1 with 71 -> 71 pgs (1.11937 -> 1.11937 * mean) 2021-03-15T22:33:59.730 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:33:59.730 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:33:59.730 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 0.05 2021-03-15T22:33:59.731 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:33:59.731 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0028 2021-03-15T22:33:59.731 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0275 2021-03-15T22:33:59.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:33:59.732 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:33:59.755 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:59.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:59.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:33:59.858 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:33:59.880 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:33:59.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:33:59.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:33:59.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:00.004 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:00.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:00.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:00.107 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:00.131 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:00.133 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:00.234 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:00.256 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:00.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:00.258 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:00.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:00.380 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:00.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:00.382 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:00.483 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:00.499 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:34:00.507 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:00.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:00.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:00.609 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:00.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:00.633 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:00.634 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:00.735 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:00.757 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:00.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:00.759 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:00.768 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:34:00.769 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:34:00.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[56739]: cluster 2021-03-15T22:33:59.198159+0000 mgr.y (mgr.14144) 1010 : cluster [DBG] pgmap v1473: 148 pgs: 1 active+clean+wait, 147 active+clean; 630 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 8.8 MiB/s, 8 objects/s recovering 2021-03-15T22:34:00.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:34:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[56739]: -03-15T22:33:59.528279+0000 mon.a (mon.0) 2240 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[56739]: audit 2021-03-15T22:33:59.535364+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? 172.21.15.76:0/3516555532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[56739]: cluster 2021-03-15T22:33:59.535411+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2021-03-15T22:34:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:34:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[56739]: 15T22:33:59.726771+0000 mgr.y (mgr.14144) 1011 : audit [DBG] from='client.16605 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:34:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[56739]: 03-15T22:34:00 2021-03-15T22:34:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[56739]: .304264+0000 mon.a (mon.0) 2243 : 2021-03-15T22:34:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:00 smithi125 conmon[56739]: cluster [DBG] osdmap e886: 8 total, 8 up, 7 in 2021-03-15T22:34:00.794 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:34:00.795 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:34:00.795 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:34:00.795 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:34:00.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[56514]: cluster 2021-03-15T22:33:59.198159+0000 mgr.y (mgr.14144) 1010 : cluster [DBG] pgmap v1473: 148 pgs: 1 active+clean+wait, 147 active+clean; 630 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 8.8 MiB/s, 8 objects/s recovering 2021-03-15T22:34:00.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[56514]: cluster 2021-03-15T22:33:59.528279+0000 mon.a (mon.0) 2240 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:00.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[56514]: audit 2021-03-15T22:33:59.535364+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? 172.21.15.76:0/3516555532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:00.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[56514]: cluster 2021-03-15T22:33:59.535411+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2021-03-15T22:34:00.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[56514]: audit 2021-03-15T22:33:59.726771+0000 mgr.y (mgr.14144) 1011 : audit [DBG] from='client.16605 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:00.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[56514]: cluster 2021-03-15T22:34:00.304264+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e886: 8 total, 8 up, 7 in 2021-03-15T22:34:00.797 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[62156]: cluster 2021-03-15T22:33:59.198159+0000 mgr.y (mgr.14144) 1010 : cluster [DBG] pgmap v1473: 148 pgs: 1 active+clean+wait, 147 active+clean; 630 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 6.7 MiB/s rd, 1.1 MiB/s wr, 8 op/s; 8.8 MiB/s, 8 objects/s recovering 2021-03-15T22:34:00.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[62156]: cluster 2021-03-15T22:33:59.528279+0000 mon.a (mon.0) 2240 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:00.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[62156]: audit 2021-03-15T22:33:59.535364+0000 mon.a (mon.0) 2241 : audit [INF] from='client.? 172.21.15.76:0/3516555532' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:00.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[62156]: cluster 2021-03-15T22:33:59.535411+0000 mon.a (mon.0) 2242 : cluster [DBG] osdmap e885: 8 total, 8 up, 7 in 2021-03-15T22:34:00.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[62156]: audit 2021-03-15T22:33:59.726771+0000 mgr.y (mgr.14144) 1011 : audit [DBG] from='client.16605 -' entity='client.admin' cmd=[{"prefix": "osd test-reweight-by-utilization", "oload": 1000, "max_change": 0.05, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:00.798 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:00 smithi076 conmon[62156]: cluster 2021-03-15T22:34:00.304264+0000 mon.a (mon.0) 2243 : cluster [DBG] osdmap e886: 8 total, 8 up, 7 in 2021-03-15T22:34:00.807 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:34:00.824 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:34:00.842 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:34:00.853 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2021-03-15T22:34:00.853 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight-by-utilization 1000 1.0 2021-03-15T22:34:00.860 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:00.861 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:34:00.861 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:34:00.862 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:34:00.862 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:34:00.862 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:34:00.862 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:34:00.863 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:34:00.863 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:34:00.863 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:34:00.882 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:00.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:00.884 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:00.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:01.009 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:01.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:01.011 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:01.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:01.135 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:01.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:01.138 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:01.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:01.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:01.264 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:01.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:01.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:01.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:01.390 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:01.391 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:01.447 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-15T22:34:01.491 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:01.514 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:01.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:01.516 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:01.565 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:01.617 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:01.640 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:01.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:01.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:01.723 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:34:01.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:01.765 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[56514]: audit 2021-03-15T22:34:00.214768 2021-03-15T22:34:01.765 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 1012 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:01.765 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[56514]: audit 2021-03-15T22:34:00.717086+0000 mon.a (mon.0) 2244 : audit [DBG] from='client.? 172.21.15.76:0/404344447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:01.765 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[56514]: audit 2021-03-15T22:34:01.116431+0000 mon.a (mon.0) 2245 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:34:01.766 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[56514]: audit 2021-03-15T22:34:01.116834+0000 mon.a (mon.0) 2246 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:34:01.766 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[62156]: audit 2021-03-15T22:34:00.214768+0000 2021-03-15T22:34:01.766 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[62156]: mgr.y (mgr.14144) 1012 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:01.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[62156]: audit 2021-03-15T22:34:00.717086+0000 mon.a (mon.0) 2244 : audit [DBG] from='client.? 172.21.15.76:0/404344447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:01.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[62156]: audit 2021-03-15T22:34:01.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[62156]: 2021-03-15T22:34:01.116431+0000 mon.a (mon.0) 2245 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:34:01.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[62156]: audit 2021 2021-03-15T22:34:01.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[62156]: -03-15T22:34:01.116834+0000 2021-03-15T22:34:01.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[62156]: mon.a (mon.0) 2246 : audit [INF] 2021-03-15T22:34:01.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:01 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:34:01.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:01.770 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:01.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:01.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:01 smithi125 conmon[56739]: audit 2021-03-15T22:34:00.214768+0000 mgr.y (mgr.14144) 1012 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:01 smithi125 conmon[56739]: audit 2021-03-15T22:34:00.717086+0000 mon.a (mon.0) 2244 : audit [DBG] from='client.? 172.21.15.76:0/404344447' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:01 smithi125 conmon[56739]: audit 2021-03-15T22:34:01.116431+0000 mon.a (mon.0) 2245 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:34:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:01 smithi125 conmon[56739]: audit 2021-03-15T22:34:01.116834+0000 mon.a (mon.0) 2246 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:34:01.871 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:01.894 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:01.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:01.896 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:01.997 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:02.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:02.021 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:02.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:02.123 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:02.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:02.146 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:02.147 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:02.248 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:02.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:02.272 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:02.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:02.360 INFO:tasks.rados.rados.0.smithi076.stdout:3333: done (0 left) 2021-03-15T22:34:02.361 INFO:tasks.rados.rados.0.smithi076.stdout:3334: write (excl) oid 23 current snap is 390 2021-03-15T22:34:02.361 INFO:tasks.rados.rados.0.smithi076.stdout:3334: seq_num 1133 ranges {642331=593714,1766922=751721} 2021-03-15T22:34:02.367 INFO:tasks.rados.rados.0.smithi076.stdout:3334: writing smithi07694878-23 from 642331 to 1236045 tid 1 2021-03-15T22:34:02.373 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:02.375 INFO:tasks.rados.rados.0.smithi076.stdout:3334: writing smithi07694878-23 from 1766922 to 2518643 tid 2 2021-03-15T22:34:02.375 INFO:tasks.rados.rados.0.smithi076.stdout:3335: snap_remove snap 371 2021-03-15T22:34:02.395 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:02.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:02.397 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:02.498 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:02.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:02.522 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:02.522 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:02.550 INFO:tasks.rados.rados.0.smithi076.stdout:3335: done (1 left) 2021-03-15T22:34:02.550 INFO:tasks.rados.rados.0.smithi076.stdout:3336: read oid 18 snap -1 2021-03-15T22:34:02.550 INFO:tasks.rados.rados.0.smithi076.stdout:3336: expect (ObjNum 1002 snap 335 seq_num 1002) 2021-03-15T22:34:02.550 INFO:tasks.rados.rados.0.smithi076.stdout:3337: read oid 43 snap -1 2021-03-15T22:34:02.551 INFO:tasks.rados.rados.0.smithi076.stdout:3337: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:34:02.551 INFO:tasks.rados.rados.0.smithi076.stdout:3338: copy_from oid 36 from oid 8 current snap is 390 2021-03-15T22:34:02.551 INFO:tasks.rados.rados.0.smithi076.stdout:3339: write (excl) oid 24 current snap is 390 2021-03-15T22:34:02.552 INFO:tasks.rados.rados.0.smithi076.stdout:3339: seq_num 1134 ranges {659138=668963,2109051=611732} 2021-03-15T22:34:02.557 INFO:tasks.rados.rados.0.smithi076.stdout:3339: writing smithi07694878-24 from 659138 to 1328101 tid 1 2021-03-15T22:34:02.566 INFO:tasks.rados.rados.0.smithi076.stdout:3339: writing smithi07694878-24 from 2109051 to 2720783 tid 2 2021-03-15T22:34:02.567 INFO:tasks.rados.rados.0.smithi076.stdout:3334: finishing write tid 1 to smithi07694878-23 2021-03-15T22:34:02.567 INFO:tasks.rados.rados.0.smithi076.stdout:3334: finishing write tid 2 to smithi07694878-23 2021-03-15T22:34:02.567 INFO:tasks.rados.rados.0.smithi076.stdout:3334: finishing write tid 3 to smithi07694878-23 2021-03-15T22:34:02.568 INFO:tasks.rados.rados.0.smithi076.stdout:3334: finishing write tid 4 to smithi07694878-23 2021-03-15T22:34:02.568 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 2402 (ObjNum 1133 snap 390 seq_num 1133) dirty exists 2021-03-15T22:34:02.568 INFO:tasks.rados.rados.0.smithi076.stdout:3334: left oid 23 (ObjNum 1133 snap 390 seq_num 1133) 2021-03-15T22:34:02.568 INFO:tasks.rados.rados.0.smithi076.stdout:3338: finishing copy_from to smithi07694878-36 2021-03-15T22:34:02.569 INFO:tasks.rados.rados.0.smithi076.stdout:3338: got expected ENOENT (src dne) 2021-03-15T22:34:02.569 INFO:tasks.rados.rados.0.smithi076.stdout:3338: finishing copy_from racing read to smithi07694878-36 2021-03-15T22:34:02.569 INFO:tasks.rados.rados.0.smithi076.stdout:3334: done (4 left) 2021-03-15T22:34:02.569 INFO:tasks.rados.rados.0.smithi076.stdout:3338: done (3 left) 2021-03-15T22:34:02.570 INFO:tasks.rados.rados.0.smithi076.stdout:3340: snap_create 2021-03-15T22:34:02.570 INFO:tasks.rados.rados.0.smithi076.stdout:3339: finishing write tid 1 to smithi07694878-24 2021-03-15T22:34:02.571 INFO:tasks.rados.rados.0.smithi076.stdout:3339: finishing write tid 2 to smithi07694878-24 2021-03-15T22:34:02.572 INFO:tasks.rados.rados.0.smithi076.stdout:3339: finishing write tid 3 to smithi07694878-24 2021-03-15T22:34:02.572 INFO:tasks.rados.rados.0.smithi076.stdout:3339: finishing write tid 4 to smithi07694878-24 2021-03-15T22:34:02.572 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1739 (ObjNum 1134 snap 390 seq_num 1134) dirty exists 2021-03-15T22:34:02.573 INFO:tasks.rados.rados.0.smithi076.stdout:3339: left oid 24 (ObjNum 1134 snap 390 seq_num 1134) 2021-03-15T22:34:02.574 INFO:tasks.rados.rados.0.smithi076.stdout:3336: expect (ObjNum 1002 snap 335 seq_num 1002) 2021-03-15T22:34:02.623 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:02.646 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:02.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:02.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:02.668 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:34:02.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:02.774 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:02.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:02.775 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:02.876 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:02.891 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:02 smithi125 conmon[56739]: cluster 2021-03-15T22:34:01.198596+0000 mgr.y (mgr.14144) 1013 : cluster [DBG] pgmap v1476: 148 pgs: 1 active+clean+wait, 147 active+clean; 630 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 5.4 MiB/s rd, 240 KiB/s wr, 5 op/s 2021-03-15T22:34:02.891 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:02 smithi125 conmon[56739]: audit 2021-03-15T22:34:02.258436+0000 mon.a (mon.0) 2247 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2021-03-15T22:34:02.891 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:02 smithi125 conmon[56739]: audit 2021-03-15T22:34:02.258685+0000 mon.a (mon.0) 2248 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2021-03-15T22:34:02.892 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:02 smithi125 conmon[56739]: audit 2021-03-15T22:34:02.258875+0000 mon.a (mon.0) 2249 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]: dispatch 2021-03-15T22:34:02.892 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:02 smithi125 conmon[56739]: audit 2021-03-15T22:34:02.259030+0000 mon.a (mon.0) 2250 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]: dispatch 2021-03-15T22:34:02.892 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:02 smithi125 conmon[56739]: audit 2021-03-15T22:34:02.259180+0000 mon.a (mon.0) 2251 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2021-03-15T22:34:02.892 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:02 smithi125 conmon[56739]: audit 2021-03-15T22:34:02.259335+0000 mon.a (mon.0) 2021-03-15T22:34:02.893 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:02 smithi125 conmon[56739]: 2252 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]}]: dispatch 2021-03-15T22:34:02.893 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:02 smithi125 conmon[56739]: audit 2021-03-15T22:34:02.259503+0000 mon.a (mon.0) 2253 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]: dispatch 2021-03-15T22:34:02.900 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:02.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:02.901 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:02.903 INFO:tasks.rados.rados.0.smithi076.stdout:3337: expect (ObjNum 982 snap 326 seq_num 982) 2021-03-15T22:34:02.972 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:03.002 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:03.006 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: cluster 2021-03-15T22:34:01.198596+0000 mgr.y ( 2021-03-15T22:34:03.006 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: mgr.14144) 1013 : cluster [DBG] pgmap v1476: 148 pgs: 1 active+clean+wait, 147 active+clean; 630 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 5.4 MiB/s rd, 240 KiB/s wr, 5 op/s 2021-03-15T22:34:03.007 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: audit 2021-03-15T22:34:02.258436+0000 mon.a (mon.0) 2247 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2021-03-15T22:34:03.007 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: audit 2021-03-15T22:34:03.007 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: 2021-03-15T22:34:02.258685+0000 mon.a (mon.0) 2248 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2021-03-15T22:34:03.007 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: audit 2021-03-15T22:34:02 2021-03-15T22:34:03.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: .258875+0000 mon.a (mon.0) 2249 : audit [INF] 2021-03-15T22:34:03.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]: dispatch 2021-03-15T22:34:03.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:34:03.008 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: 03-15T22:34:02.259030+0000 mon.a (mon.0) 2250 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]: dispatch 2021-03-15T22:34:03.009 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:34:03.009 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: T22:34:02.259180+0000 mon.a (mon.0) 2251 : audit 2021-03-15T22:34:03.009 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2021-03-15T22:34:03.009 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:34:03.010 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: :34:02.259335+0000 mon.a (mon. 2021-03-15T22:34:03.010 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: 0) 2252 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]}]: dispatch 2021-03-15T22:34:03.010 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:34:03.010 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: -15T22:34:02.259503+0000 2021-03-15T22:34:03.011 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: mon.a (mon.0) 2253 : audit [INF] 2021-03-15T22:34:03.011 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]: dispatch 2021-03-15T22:34:03.011 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[56514]: cluster 2021-03-15T22:34:01.198596+0000 2021-03-15T22:34:03.011 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[56514]: mgr.y (mgr.14144) 1013 : cluster [DBG] pgmap v1476: 148 pgs: 1 active+clean+wait, 147 active+clean; 630 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 5.4 MiB/s rd, 240 KiB/s wr, 5 op/s 2021-03-15T22:34:03.012 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.258436+0000 mon.a (mon.0) 2247 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]: dispatch 2021-03-15T22:34:03.012 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.258685+0000 mon.a (mon.0) 2248 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]: dispatch 2021-03-15T22:34:03.012 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.258875+0000 mon.a (mon.0) 2249 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]: dispatch 2021-03-15T22:34:03.012 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.259030+0000 mon.a (mon.0) 2250 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]: dispatch 2021-03-15T22:34:03.013 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.259180+0000 mon.a (mon.0) 2251 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]: dispatch 2021-03-15T22:34:03.013 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.259335+0000 mon.a (mon.0) 2252 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]}]: dispatch 2021-03-15T22:34:03.013 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:02 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.259503+0000 mon.a (mon.0) 2253 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]: dispatch 2021-03-15T22:34:03.026 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:03.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:03.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:03.128 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:03.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:03.153 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:03.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:03.254 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:03.278 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:03.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:03.280 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:03.299 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:34:03.299 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:34:03.299 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:34:03.299 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:34:03.300 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:34:03.381 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:03.404 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:03.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:03.406 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:03.438 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 444 (0%) 2021-03-15T22:34:03.439 INFO:teuthology.orchestra.run.smithi076.stdout:avg 63.4286 2021-03-15T22:34:03.439 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 5.0669 -> 5.0669 (expected baseline 7.37342) 2021-03-15T22:34:03.439 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.5 with 57 -> 57 pgs (0.898649 -> 0.898649 * mean) 2021-03-15T22:34:03.439 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.3 with 72 -> 72 pgs (1.13514 -> 1.13514 * mean) 2021-03-15T22:34:03.440 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:34:03.440 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:34:03.440 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 1 2021-03-15T22:34:03.440 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:34:03.441 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0027 2021-03-15T22:34:03.441 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0274 2021-03-15T22:34:03.441 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:34:03.507 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:03.530 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:34:03.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:03.532 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:03.532 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:03.569 INFO:tasks.rados.rados.0.smithi076.stdout:3336: done (3 left) 2021-03-15T22:34:03.570 INFO:tasks.rados.rados.0.smithi076.stdout:3337: done (2 left) 2021-03-15T22:34:03.570 INFO:tasks.rados.rados.0.smithi076.stdout:3339: done (1 left) 2021-03-15T22:34:03.570 INFO:tasks.rados.rados.0.smithi076.stdout:3340: done (0 left) 2021-03-15T22:34:03.570 INFO:tasks.rados.rados.0.smithi076.stdout:3341: copy_from oid 44 from oid 25 current snap is 391 2021-03-15T22:34:03.571 INFO:tasks.rados.rados.0.smithi076.stdout:3342: delete oid 33 current snap is 391 2021-03-15T22:34:03.571 INFO:tasks.rados.rados.0.smithi076.stdout:3341: finishing copy_from to smithi07694878-44 2021-03-15T22:34:03.572 INFO:tasks.rados.rados.0.smithi076.stdout:3341: got expected ENOENT (src dne) 2021-03-15T22:34:03.573 INFO:tasks.rados.rados.0.smithi076.stdout:3341: finishing copy_from racing read to smithi07694878-44 2021-03-15T22:34:03.573 INFO:tasks.rados.rados.0.smithi076.stdout:3341: done (1 left) 2021-03-15T22:34:03.573 INFO:tasks.rados.rados.0.smithi076.stdout:3342: done (0 left) 2021-03-15T22:34:03.573 INFO:tasks.rados.rados.0.smithi076.stdout:3343: read oid 28 snap 339 2021-03-15T22:34:03.574 INFO:tasks.rados.rados.0.smithi076.stdout:3343: expect deleted 2021-03-15T22:34:03.574 INFO:tasks.rados.rados.0.smithi076.stdout:3344: rollback oid 45 current snap is 391 2021-03-15T22:34:03.574 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 45 to 341 2021-03-15T22:34:03.574 INFO:tasks.rados.rados.0.smithi076.stdout:3345: delete oid 7 current snap is 391 2021-03-15T22:34:03.577 INFO:tasks.rados.rados.0.smithi076.stdout:3344: finishing rollback tid 1 to smithi07694878-45 2021-03-15T22:34:03.577 INFO:tasks.rados.rados.0.smithi076.stdout:3344: finishing rollback tid 2 to smithi07694878-45 2021-03-15T22:34:03.578 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 2109 (ObjNum 1023 snap 341 seq_num 1023) dirty exists 2021-03-15T22:34:03.578 INFO:tasks.rados.rados.0.smithi076.stdout:3343: done (2 left) 2021-03-15T22:34:03.578 INFO:tasks.rados.rados.0.smithi076.stdout:3344: done (1 left) 2021-03-15T22:34:03.578 INFO:tasks.rados.rados.0.smithi076.stdout:3345: done (0 left) 2021-03-15T22:34:03.579 INFO:tasks.rados.rados.0.smithi076.stdout:3346: snap_create 2021-03-15T22:34:03.633 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:03.656 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:03.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:03.657 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:03.758 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:03.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[62156]: audit 2021-03-15T22:34:02.544015+0000 mon.a (mon.0) 2254 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2021-03-15T22:34:03.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[62156]: audit 2021-03-15T22:34:02.544096+0000 mon.a (mon.0) 2255 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2021-03-15T22:34:03.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[62156]: audit 2021-03-15T22:34:02.544157+0000 mon.a (mon.0) 2256 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]': finished 2021-03-15T22:34:03.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[62156]: audit 2021-03-15T22:34:02.544217+0000 mon.a (mon.0) 2257 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]': finished 2021-03-15T22:34:03.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[62156]: audit 2021-03-15T22:34:02.544275+0000 mon.a (mon.0) 2258 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]': finished 2021-03-15T22:34:03.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[62156]: audit 2021-03-15T22:34:02.544331+0000 mon.a (mon.0) 2259 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]}]': finished 2021-03-15T22:34:03.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[62156]: audit 2021-03-15T22:34:02.544393+0000 mon.a (mon.0) 2260 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]': finished 2021-03-15T22:34:03.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[62156]: cluster 2021-03-15T22:34:02.544462+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e887: 8 total, 8 up, 7 in 2021-03-15T22:34:03.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.544015+0000 mon.a (mon.0) 2254 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2021-03-15T22:34:03.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.544096+0000 mon.a (mon.0) 2255 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2021-03-15T22:34:03.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.544157+0000 mon.a (mon.0) 2256 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]': finished 2021-03-15T22:34:03.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.544217+0000 mon.a (mon.0) 2257 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]': finished 2021-03-15T22:34:03.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.544275+0000 mon.a (mon.0) 2258 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]': finished 2021-03-15T22:34:03.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.544331+0000 mon.a (mon.0) 2021-03-15T22:34:03.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[56514]: 2259 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]}]': finished 2021-03-15T22:34:03.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[56514]: audit 2021-03-15T22:34:02.544393+0000 mon.a (mon.0) 2260 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]': finished 2021-03-15T22:34:03.771 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:03 smithi076 conmon[56514]: cluster 2021-03-15T22:34:02.544462+0000 mon.a (mon.0) 2261 : cluster [DBG] osdmap e887: 8 total, 8 up, 7 in 2021-03-15T22:34:03.782 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:03.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:03.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:03.884 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:03.900 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: audit 2021-03-15T22:34:03.900 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: 2021-03-15T22:34:02.544015+0000 2021-03-15T22:34:03.900 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: mon.a (mon.0) 2254 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [1, 5]}]': finished 2021-03-15T22:34:03.901 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:34:03.901 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: 22:34:02.544096+0000 mon.a (mon.0) 2255 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1b", "id": [1, 5]}]': finished 2021-03-15T22:34:03.901 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: audit 2021-03-15T22:34:02.544157+0000 mon.a (mon. 2021-03-15T22:34:03.901 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: 0) 2256 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.10", "id": [7, 3]}]': finished 2021-03-15T22:34:03.901 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: audit 2021-03-15T22:34:03.902 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: 2021-03-15T22:34:02.544217+0000 mon.a (mon 2021-03-15T22:34:03.902 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: .0) 2257 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.17", "id": [1, 6]}]': finished 2021-03-15T22:34:03.902 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: audit 2021-03-15T22:34:02.544275+0000 2021-03-15T22:34:03.902 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: mon.a (mon.0) 2258 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1b", "id": [0, 3]}]': finished 2021-03-15T22:34:03.903 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: audit 2021-03-15T22:34:02.544331+0000 2021-03-15T22:34:03.903 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: mon.a (mon.0) 2259 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "5.1f", "id": [4, 3]}]': finished 2021-03-15T22:34:03.903 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: audit 2021-03-15T22:34:02.544393+0000 2021-03-15T22:34:03.904 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: mon.a (mon.0) 2260 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "6.c", "id": [3, 1]}]': finished 2021-03-15T22:34:03.904 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: 2021-03-15T22:34:03.904 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:34:03.904 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: -15T22:34:02. 2021-03-15T22:34:03.905 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: 544462+0000 mon.a (mon.0 2021-03-15T22:34:03.905 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: ) 2261 : cluster [DBG] osdmap e887: 8 total, 8 up, 7 in 2021-03-15T22:34:03.905 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:03 smithi125 conmon[56739]: 2021-03-15T22:34:03.906 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:03.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:03.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:03.922 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-15T22:34:04.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:04.030 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:04.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:04.032 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:04.133 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:04.155 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:04.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:04.157 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:04.201 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:34:04.258 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:04.280 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:04.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:04.282 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:04.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:04.406 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:04.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:04.407 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:04.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:04.530 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:04.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:04.532 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:04.565 INFO:tasks.rados.rados.0.smithi076.stdout:3346: done (0 left) 2021-03-15T22:34:04.565 INFO:tasks.rados.rados.0.smithi076.stdout:3347: snap_create 2021-03-15T22:34:04.633 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:04.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: cluster 2021-03-15T22:34:04.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: 2021-03-15T 2021-03-15T22:34:04.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: 22:34:03.199017+0000 mgr.y (mgr.14144) 1014 : cluster [DBG] pgmap v1478: 148 pgs: 1 peering, 1 active+clean+wait, 146 active+clean; 631 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 219 KiB/s wr, 4 op/s 2021-03-15T22:34:04.649 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: audit 2021-03-15T22:34:04.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: 2021-03- 2021-03-15T22:34:04.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: 15T22:34:03.436708+0000 mgr.y ( 2021-03-15T22:34:04.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: mgr.14144) 1015 : audit [DBG] 2021-03-15T22:34:04.650 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: from='client.16617 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:04.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: cluster 2021-03-15T22:34:04.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: 2021-03-15T22:34 2021-03-15T22:34:04.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: :03.560801+0000 mon.a (mon 2021-03-15T22:34:04.651 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: .0) 2262 : cluster [DBG] 2021-03-15T22:34:04.652 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:04 smithi125 conmon[56739]: osdmap e888: 8 total, 8 up, 7 in 2021-03-15T22:34:04.656 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:04.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:04.658 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:04.759 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:04.781 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:04.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:04.782 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:04.801 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '3.0', 'overage': '110', 'type': 'reweight-by-utilization'} 2021-03-15T22:34:04.801 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight-by-utilization 110 3.0 2021-03-15T22:34:04.863 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[56514]: cluster 2021-03-15T22:34:03.199017+0000 mgr.y 2021-03-15T22:34:04.863 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[56514]: (mgr.14144) 1014 : cluster [DBG] pgmap v1478: 148 pgs: 1 peering, 1 active+clean+wait, 146 active+clean; 631 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 219 KiB/s wr, 4 op/s 2021-03-15T22:34:04.863 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[56514]: audit 2021-03-15T22:34:03.436708+0000 mgr.y (mgr.14144) 1015 : audit [DBG] from='client.16617 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:04.864 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[56514]: cluster 2021-03-15T22:34:03.560801+0000 mon.a (mon.0) 2262 : cluster [DBG] osdmap e888: 8 total, 8 up, 7 in 2021-03-15T22:34:04.864 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[62156]: cluster 2021-03-15T22:34:03.199017+0000 mgr.y 2021-03-15T22:34:04.864 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[62156]: (mgr.14144) 1014 : cluster [DBG] pgmap v1478: 148 pgs: 1 peering, 1 active+clean+wait, 146 active+clean; 631 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 3.8 MiB/s rd, 219 KiB/s wr, 4 op/s 2021-03-15T22:34:04.864 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:34:04.865 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[62156]: 34:03.436708+0000 mgr.y (mgr.14144) 1015 : audit [DBG] from='client.16617 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:04.865 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[62156]: cluster 2021-03-15T22:34:04.865 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[62156]: 2021-03-15T22:34 2021-03-15T22:34:04.865 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[62156]: :03.560801+0000 mon.a (mon.0) 2021-03-15T22:34:04.866 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:04 smithi076 conmon[62156]: 2262 : cluster [DBG] osdmap e888: 8 total, 8 up, 7 in 2021-03-15T22:34:04.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:04.905 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:04.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:04.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:04.972 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:05.008 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:05.032 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:05.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:05.033 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:05.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:05.157 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:05.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:05.158 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:05.182 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:05.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:05.280 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:05.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:05.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:05.315 INFO:tasks.rados.rados.0.smithi076.stdout:3347: done (0 left) 2021-03-15T22:34:05.316 INFO:tasks.rados.rados.0.smithi076.stdout:3348: write oid 42 current snap is 393 2021-03-15T22:34:05.316 INFO:tasks.rados.rados.0.smithi076.stdout:3348: seq_num 1137 ranges {709559=494710,1884497=759059} 2021-03-15T22:34:05.320 INFO:tasks.rados.rados.0.smithi076.stdout:3348: writing smithi07694878-42 from 709559 to 1204269 tid 1 2021-03-15T22:34:05.330 INFO:tasks.rados.rados.0.smithi076.stdout:3348: writing smithi07694878-42 from 1884497 to 2643556 tid 2 2021-03-15T22:34:05.330 INFO:tasks.rados.rados.0.smithi076.stdout:3349: write (excl) oid 18 current snap is 393 2021-03-15T22:34:05.331 INFO:tasks.rados.rados.0.smithi076.stdout:3349: seq_num 1138 ranges {726366=569959,1826626=502717,2801601=324765} 2021-03-15T22:34:05.338 INFO:tasks.rados.rados.0.smithi076.stdout:3349: writing smithi07694878-18 from 726366 to 1296325 tid 1 2021-03-15T22:34:05.345 INFO:tasks.rados.rados.0.smithi076.stdout:3349: writing smithi07694878-18 from 1826626 to 2329343 tid 2 2021-03-15T22:34:05.350 INFO:tasks.rados.rados.0.smithi076.stdout:3349: writing smithi07694878-18 from 2801601 to 3126366 tid 3 2021-03-15T22:34:05.350 INFO:tasks.rados.rados.0.smithi076.stdout:3350: write (excl) oid 48 current snap is 393 2021-03-15T22:34:05.350 INFO:tasks.rados.rados.0.smithi076.stdout:3348: finishing write tid 1 to smithi07694878-42 2021-03-15T22:34:05.351 INFO:tasks.rados.rados.0.smithi076.stdout:3350: seq_num 1139 ranges {743173=645208,1885108=762728} 2021-03-15T22:34:05.358 INFO:tasks.rados.rados.0.smithi076.stdout:3350: writing smithi07694878-48 from 743173 to 1388381 tid 1 2021-03-15T22:34:05.367 INFO:tasks.rados.rados.0.smithi076.stdout:3350: writing smithi07694878-48 from 1885108 to 2647836 tid 2 2021-03-15T22:34:05.368 INFO:tasks.rados.rados.0.smithi076.stdout:3351: write (excl) oid 14 current snap is 393 2021-03-15T22:34:05.368 INFO:tasks.rados.rados.0.smithi076.stdout:3348: finishing write tid 2 to smithi07694878-42 2021-03-15T22:34:05.368 INFO:tasks.rados.rados.0.smithi076.stdout:3348: finishing write tid 3 to smithi07694878-42 2021-03-15T22:34:05.368 INFO:tasks.rados.rados.0.smithi076.stdout:3348: finishing write tid 4 to smithi07694878-42 2021-03-15T22:34:05.369 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2950 (ObjNum 1137 snap 393 seq_num 1137) dirty exists 2021-03-15T22:34:05.369 INFO:tasks.rados.rados.0.smithi076.stdout:3348: left oid 42 (ObjNum 1137 snap 393 seq_num 1137) 2021-03-15T22:34:05.369 INFO:tasks.rados.rados.0.smithi076.stdout:3349: finishing write tid 1 to smithi07694878-18 2021-03-15T22:34:05.369 INFO:tasks.rados.rados.0.smithi076.stdout:3351: seq_num 1140 ranges {759980=720457,1943590=506386,2856447=303533} 2021-03-15T22:34:05.376 INFO:tasks.rados.rados.0.smithi076.stdout:3351: writing smithi07694878-14 from 759980 to 1480437 tid 1 2021-03-15T22:34:05.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:05.384 INFO:tasks.rados.rados.0.smithi076.stdout:3351: writing smithi07694878-14 from 1943590 to 2449976 tid 2 2021-03-15T22:34:05.388 INFO:tasks.rados.rados.0.smithi076.stdout:3351: writing smithi07694878-14 from 2856447 to 3159980 tid 3 2021-03-15T22:34:05.388 INFO:tasks.rados.rados.0.smithi076.stdout:3348: done (3 left) 2021-03-15T22:34:05.388 INFO:tasks.rados.rados.0.smithi076.stdout:3352: read oid 19 snap -1 2021-03-15T22:34:05.389 INFO:tasks.rados.rados.0.smithi076.stdout:3352: expect (ObjNum 1119 snap 388 seq_num 1119) 2021-03-15T22:34:05.389 INFO:tasks.rados.rados.0.smithi076.stdout:3353: snap_create 2021-03-15T22:34:05.389 INFO:tasks.rados.rados.0.smithi076.stdout:3349: finishing write tid 2 to smithi07694878-18 2021-03-15T22:34:05.389 INFO:tasks.rados.rados.0.smithi076.stdout:3349: finishing write tid 3 to smithi07694878-18 2021-03-15T22:34:05.390 INFO:tasks.rados.rados.0.smithi076.stdout:3349: finishing write tid 4 to smithi07694878-18 2021-03-15T22:34:05.390 INFO:tasks.rados.rados.0.smithi076.stdout:3349: finishing write tid 5 to smithi07694878-18 2021-03-15T22:34:05.390 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 2406 (ObjNum 1138 snap 393 seq_num 1138) dirty exists 2021-03-15T22:34:05.390 INFO:tasks.rados.rados.0.smithi076.stdout:3349: left oid 18 (ObjNum 1138 snap 393 seq_num 1138) 2021-03-15T22:34:05.390 INFO:tasks.rados.rados.0.smithi076.stdout:3350: finishing write tid 1 to smithi07694878-48 2021-03-15T22:34:05.391 INFO:tasks.rados.rados.0.smithi076.stdout:3350: finishing write tid 2 to smithi07694878-48 2021-03-15T22:34:05.391 INFO:tasks.rados.rados.0.smithi076.stdout:3350: finishing write tid 3 to smithi07694878-48 2021-03-15T22:34:05.391 INFO:tasks.rados.rados.0.smithi076.stdout:3350: finishing write tid 4 to smithi07694878-48 2021-03-15T22:34:05.391 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 2405 (ObjNum 1139 snap 393 seq_num 1139) dirty exists 2021-03-15T22:34:05.392 INFO:tasks.rados.rados.0.smithi076.stdout:3350: left oid 48 (ObjNum 1139 snap 393 seq_num 1139) 2021-03-15T22:34:05.392 INFO:tasks.rados.rados.0.smithi076.stdout:3351: finishing write tid 1 to smithi07694878-14 2021-03-15T22:34:05.392 INFO:tasks.rados.rados.0.smithi076.stdout:3351: finishing write tid 2 to smithi07694878-14 2021-03-15T22:34:05.393 INFO:tasks.rados.rados.0.smithi076.stdout:3351: finishing write tid 3 to smithi07694878-14 2021-03-15T22:34:05.393 INFO:tasks.rados.rados.0.smithi076.stdout:3351: finishing write tid 4 to smithi07694878-14 2021-03-15T22:34:05.393 INFO:tasks.rados.rados.0.smithi076.stdout:3351: finishing write tid 5 to smithi07694878-14 2021-03-15T22:34:05.393 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 2666 (ObjNum 1140 snap 393 seq_num 1140) dirty exists 2021-03-15T22:34:05.394 INFO:tasks.rados.rados.0.smithi076.stdout:3351: left oid 14 (ObjNum 1140 snap 393 seq_num 1140) 2021-03-15T22:34:05.395 INFO:tasks.rados.rados.0.smithi076.stdout:3352: expect (ObjNum 1119 snap 388 seq_num 1119) 2021-03-15T22:34:05.411 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:05.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:05.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:05.514 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:05.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:05.537 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:05.537 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:05.639 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:05.663 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:05.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:05.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:05.743 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:34:05.743 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:34:05.743 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:34:05.744 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:34:05.744 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.744 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3950 7.7 7:ed4a01d2:::smithi07694878-2:head [write 658973~703921,stat] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:34:05.744 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:34:05.745 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 510.61115892999999, 2021-03-15T22:34:05.745 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99727343300000004, 2021-03-15T22:34:05.745 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.745 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:05.746 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.746 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.746 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.746 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3950 2021-03-15T22:34:05.747 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.747 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.747 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.747 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.747 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:34:05.748 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.748 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.748 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.748 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.749 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.127241+0000", 2021-03-15T22:34:05.749 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00076287800000000004 2021-03-15T22:34:05.749 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.749 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.749 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.750 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128004+0000", 2021-03-15T22:34:05.750 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8489999999999999e-06 2021-03-15T22:34:05.750 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.750 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.751 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.751 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128008+0000", 2021-03-15T22:34:05.751 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3244e-05 2021-03-15T22:34:05.751 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.751 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.752 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.752 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128031+0000", 2021-03-15T22:34:05.752 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2050000000000002e-06 2021-03-15T22:34:05.752 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.753 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.753 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.753 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128039+0000", 2021-03-15T22:34:05.753 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9277999999999997e-05 2021-03-15T22:34:05.753 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.754 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.754 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.754 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128099+0000", 2021-03-15T22:34:05.754 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2401999999999999e-05 2021-03-15T22:34:05.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.755 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:34:05.755 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.128111+0000", 2021-03-15T22:34:05.755 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99084192299999996 2021-03-15T22:34:05.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.756 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.756 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.118953+0000", 2021-03-15T22:34:05.757 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000162363 2021-03-15T22:34:05.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.757 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.757 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.119115+0000", 2021-03-15T22:34:05.758 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00023901900000000001 2021-03-15T22:34:05.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.758 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:34:05.759 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.119354+0000", 2021-03-15T22:34:05.759 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020445160000000001 2021-03-15T22:34:05.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.759 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.759 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:05.760 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121399+0000", 2021-03-15T22:34:05.760 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021134299999999999 2021-03-15T22:34:05.760 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.760 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.761 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.761 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123512+0000", 2021-03-15T22:34:05.761 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000816028 2021-03-15T22:34:05.761 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.761 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.762 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.762 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124328+0000", 2021-03-15T22:34:05.762 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2724000000000005e-05 2021-03-15T22:34:05.762 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.763 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.763 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:05.763 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124391+0000", 2021-03-15T22:34:05.763 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012357399999999999 2021-03-15T22:34:05.763 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.764 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.764 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.764 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124515+0000", 2021-03-15T22:34:05.764 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99727343300000004 2021-03-15T22:34:05.765 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.765 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.765 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.765 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.766 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.766 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3951 7.7 7:ed4a01d2:::smithi07694878-2:head [write 1907912~570853] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:34:05.766 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:34:05.766 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 510.60441679399997, 2021-03-15T22:34:05.766 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99540528800000005, 2021-03-15T22:34:05.767 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.767 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:05.767 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.767 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.768 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.768 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3951 2021-03-15T22:34:05.768 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.768 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.768 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.769 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.769 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:34:05.769 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.769 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.769 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.770 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.770 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.133983+0000", 2021-03-15T22:34:05.770 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00050036199999999996 2021-03-15T22:34:05.770 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.771 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.771 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.771 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134484+0000", 2021-03-15T22:34:05.771 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9979999999999999e-06 2021-03-15T22:34:05.771 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.772 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.772 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.772 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134487+0000", 2021-03-15T22:34:05.772 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3523000000000001e-05 2021-03-15T22:34:05.773 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.773 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.773 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.773 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134500+0000", 2021-03-15T22:34:05.773 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5979999999999996e-06 2021-03-15T22:34:05.774 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.774 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.774 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.774 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134506+0000", 2021-03-15T22:34:05.775 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0407000000000001e-05 2021-03-15T22:34:05.775 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.775 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.775 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.775 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134546+0000", 2021-03-15T22:34:05.776 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5650000000000002e-06 2021-03-15T22:34:05.776 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.776 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.776 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:34:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.134555+0000", 2021-03-15T22:34:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98647964300000002 2021-03-15T22:34:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.777 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121034+0000", 2021-03-15T22:34:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104298 2021-03-15T22:34:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.778 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121139+0000", 2021-03-15T22:34:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0034252140000000002 2021-03-15T22:34:05.779 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124564+0000", 2021-03-15T22:34:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.3158000000000005e-05 2021-03-15T22:34:05.780 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124657+0000", 2021-03-15T22:34:05.781 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000172645 2021-03-15T22:34:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:34:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124830+0000", 2021-03-15T22:34:05.782 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001768096 2021-03-15T22:34:05.783 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.783 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.783 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:05.783 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126598+0000", 2021-03-15T22:34:05.783 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020473919999999999 2021-03-15T22:34:05.784 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.784 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.784 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.784 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128645+0000", 2021-03-15T22:34:05.785 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00067562299999999998 2021-03-15T22:34:05.785 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.785 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.785 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.785 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129321+0000", 2021-03-15T22:34:05.786 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3652999999999998e-05 2021-03-15T22:34:05.786 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.786 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.786 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:05.787 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129354+0000", 2021-03-15T22:34:05.787 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4112999999999997e-05 2021-03-15T22:34:05.787 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.787 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.788 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.788 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129389+0000", 2021-03-15T22:34:05.788 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99540528800000005 2021-03-15T22:34:05.788 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.788 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.789 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.789 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.789 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.789 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3952 7.7 7:ed4a01d2:::smithi07694878-2:head [write 2985346~73627] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:34:05.790 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:34:05.790 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 510.60122229500001, 2021-03-15T22:34:05.790 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99236444199999996, 2021-03-15T22:34:05.790 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.790 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:05.791 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.791 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.791 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.791 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3952 2021-03-15T22:34:05.791 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.792 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.792 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.792 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.792 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:34:05.793 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.793 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.793 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.793 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.793 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137178+0000", 2021-03-15T22:34:05.794 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.5050000000000006e-05 2021-03-15T22:34:05.794 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.794 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.794 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.795 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137273+0000", 2021-03-15T22:34:05.795 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.883e-06 2021-03-15T22:34:05.795 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.795 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.795 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.796 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137275+0000", 2021-03-15T22:34:05.796 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0764000000000001e-05 2021-03-15T22:34:05.796 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.796 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.797 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.797 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137285+0000", 2021-03-15T22:34:05.797 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2190000000000001e-06 2021-03-15T22:34:05.797 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.797 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.798 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.798 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137292+0000", 2021-03-15T22:34:05.798 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5432999999999997e-05 2021-03-15T22:34:05.798 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.799 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.799 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.799 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137337+0000", 2021-03-15T22:34:05.799 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.738e-06 2021-03-15T22:34:05.799 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.800 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.800 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:34:05.800 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137346+0000", 2021-03-15T22:34:05.800 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98385600600000001 2021-03-15T22:34:05.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.801 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.801 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121202+0000", 2021-03-15T22:34:05.801 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9746e-05 2021-03-15T22:34:05.802 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.802 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.802 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:05.802 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121262+0000", 2021-03-15T22:34:05.803 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041684740000000001 2021-03-15T22:34:05.803 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.803 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.803 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.803 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125430+0000", 2021-03-15T22:34:05.804 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.0578000000000001e-05 2021-03-15T22:34:05.804 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.804 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.804 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.804 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125501+0000", 2021-03-15T22:34:05.805 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8120000000000002e-05 2021-03-15T22:34:05.805 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.805 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.805 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:34:05.806 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125599+0000", 2021-03-15T22:34:05.806 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010039140000000001 2021-03-15T22:34:05.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.806 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.806 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:05.807 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126603+0000", 2021-03-15T22:34:05.807 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020828969999999998 2021-03-15T22:34:05.807 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.807 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.808 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.808 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128686+0000", 2021-03-15T22:34:05.808 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00075311200000000001 2021-03-15T22:34:05.808 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.808 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.809 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.809 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129439+0000", 2021-03-15T22:34:05.809 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7739999999999999e-05 2021-03-15T22:34:05.809 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.810 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:05.810 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129496+0000", 2021-03-15T22:34:05.810 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5768e-05 2021-03-15T22:34:05.810 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.811 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.811 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.811 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129542+0000", 2021-03-15T22:34:05.811 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99236444199999996 2021-03-15T22:34:05.812 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.812 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.812 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.812 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.813 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3953 7.7 7:ed4a01d2:::smithi07694878-2:head [setxattr _header (57) in=7b,truncate 3058973] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:34:05.813 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:34:05.813 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 510.60109744099998, 2021-03-15T22:34:05.813 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99574138599999995, 2021-03-15T22:34:05.814 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.814 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:05.814 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.814 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.814 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.815 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3953 2021-03-15T22:34:05.815 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.815 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.816 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.816 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:34:05.816 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.817 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.817 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137303+0000", 2021-03-15T22:34:05.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9230000000000001e-06 2021-03-15T22:34:05.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.818 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137305+0000", 2021-03-15T22:34:05.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.252e-06 2021-03-15T22:34:05.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.819 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137306+0000", 2021-03-15T22:34:05.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9039999999999997e-06 2021-03-15T22:34:05.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.820 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137312+0000", 2021-03-15T22:34:05.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.856e-06 2021-03-15T22:34:05.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.821 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.821 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137316+0000", 2021-03-15T22:34:05.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6783000000000002e-05 2021-03-15T22:34:05.822 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.822 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.822 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.822 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137372+0000", 2021-03-15T22:34:05.823 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9349999999999999e-06 2021-03-15T22:34:05.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.823 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:34:05.823 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137378+0000", 2021-03-15T22:34:05.824 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98391964700000001 2021-03-15T22:34:05.824 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.824 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.824 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.824 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121298+0000", 2021-03-15T22:34:05.825 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.9109000000000004e-05 2021-03-15T22:34:05.825 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.825 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:05.826 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121387+0000", 2021-03-15T22:34:05.826 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045047059999999998 2021-03-15T22:34:05.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.826 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.827 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.827 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125892+0000", 2021-03-15T22:34:05.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6956999999999998e-05 2021-03-15T22:34:05.827 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.827 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.828 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.828 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125939+0000", 2021-03-15T22:34:05.828 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012742400000000001 2021-03-15T22:34:05.828 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.829 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.829 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:34:05.829 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126066+0000", 2021-03-15T22:34:05.829 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000545302 2021-03-15T22:34:05.829 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.830 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.830 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:05.830 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126611+0000", 2021-03-15T22:34:05.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021128950000000001 2021-03-15T22:34:05.830 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.831 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.831 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.831 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128724+0000", 2021-03-15T22:34:05.831 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0042103729999999999 2021-03-15T22:34:05.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.832 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.132935+0000", 2021-03-15T22:34:05.832 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5052000000000001e-05 2021-03-15T22:34:05.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:05.833 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.132990+0000", 2021-03-15T22:34:05.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4268000000000003e-05 2021-03-15T22:34:05.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.834 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.834 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.835 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133044+0000", 2021-03-15T22:34:05.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99574138599999995 2021-03-15T22:34:05.835 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.835 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.835 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3954 7.7 7:ed4a01d2:::smithi07694878-2:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e420)", 2021-03-15T22:34:05.836 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:34:05.837 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 510.60106827200002, 2021-03-15T22:34:05.837 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99588332599999996, 2021-03-15T22:34:05.837 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.837 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:34:05.837 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.838 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.838 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.838 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3954 2021-03-15T22:34:05.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.839 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:34:05.839 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.840 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.840 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.840 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137332+0000", 2021-03-15T22:34:05.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.015e-06 2021-03-15T22:34:05.841 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.841 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.841 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137333+0000", 2021-03-15T22:34:05.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5290000000000001e-06 2021-03-15T22:34:05.842 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.842 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.842 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.842 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137334+0000", 2021-03-15T22:34:05.843 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.489e-06 2021-03-15T22:34:05.843 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.843 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.843 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137341+0000", 2021-03-15T22:34:05.844 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1989999999999997e-06 2021-03-15T22:34:05.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.844 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.844 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.845 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137346+0000", 2021-03-15T22:34:05.845 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0890999999999998e-05 2021-03-15T22:34:05.845 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.845 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.845 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.846 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137397+0000", 2021-03-15T22:34:05.846 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3159999999999997e-06 2021-03-15T22:34:05.846 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.846 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:34:05.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.137405+0000", 2021-03-15T22:34:05.847 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.98402323800000002 2021-03-15T22:34:05.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.848 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121428+0000", 2021-03-15T22:34:05.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2376999999999998e-05 2021-03-15T22:34:05.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.849 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.849 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:05.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121471+0000", 2021-03-15T22:34:05.849 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0050435389999999997 2021-03-15T22:34:05.849 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.850 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.850 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.850 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126514+0000", 2021-03-15T22:34:05.850 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3162000000000003e-05 2021-03-15T22:34:05.851 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.851 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.851 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:05.851 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126578+0000", 2021-03-15T22:34:05.851 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0065105759999999997 2021-03-15T22:34:05.852 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.852 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.852 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.852 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133088+0000", 2021-03-15T22:34:05.853 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4956000000000006e-05 2021-03-15T22:34:05.853 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.853 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.853 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.853 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133153+0000", 2021-03-15T22:34:05.854 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2038999999999993e-05 2021-03-15T22:34:05.854 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.854 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.854 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.854 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.133215+0000", 2021-03-15T22:34:05.855 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99588332599999996 2021-03-15T22:34:05.855 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.855 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.855 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.856 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.856 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3970 7.7 7:ecc0cb20:::smithi07694878-25:head [write 709394~646021] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:34:05.856 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:34:05.856 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 510.45913140900001, 2021-03-15T22:34:05.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84719129900000001, 2021-03-15T22:34:05.857 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.857 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:05.857 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.858 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.858 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.858 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3970 2021-03-15T22:34:05.858 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.858 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.859 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.859 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.859 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:34:05.859 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.860 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.860 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.860 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279269+0000", 2021-03-15T22:34:05.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000550652 2021-03-15T22:34:05.861 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.861 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.861 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.861 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279819+0000", 2021-03-15T22:34:05.861 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8980000000000004e-06 2021-03-15T22:34:05.862 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.862 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279824+0000", 2021-03-15T22:34:05.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1767999999999999e-05 2021-03-15T22:34:05.863 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.863 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.863 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.864 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279846+0000", 2021-03-15T22:34:05.864 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2509999999999994e-06 2021-03-15T22:34:05.864 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.864 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.864 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.865 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279854+0000", 2021-03-15T22:34:05.865 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7908000000000001e-05 2021-03-15T22:34:05.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.865 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.866 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.866 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279922+0000", 2021-03-15T22:34:05.866 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2315e-05 2021-03-15T22:34:05.866 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.867 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:34:05.867 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.279934+0000", 2021-03-15T22:34:05.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84156851799999999 2021-03-15T22:34:05.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.868 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.868 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.868 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121503+0000", 2021-03-15T22:34:05.868 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012479099999999999 2021-03-15T22:34:05.868 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.869 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.869 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.869 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121628+0000", 2021-03-15T22:34:05.869 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000208667 2021-03-15T22:34:05.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.870 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.870 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:34:05.870 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.121836+0000", 2021-03-15T22:34:05.870 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023170439999999999 2021-03-15T22:34:05.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.871 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:05.871 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124153+0000", 2021-03-15T22:34:05.871 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0012293359999999999 2021-03-15T22:34:05.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.872 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.872 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.872 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.125383+0000", 2021-03-15T22:34:05.873 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00099435699999999997 2021-03-15T22:34:05.873 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.873 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.873 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126377+0000", 2021-03-15T22:34:05.874 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3995999999999997e-05 2021-03-15T22:34:05.874 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.874 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.874 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:05.875 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126421+0000", 2021-03-15T22:34:05.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8797999999999997e-05 2021-03-15T22:34:05.875 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.875 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.875 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.876 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126460+0000", 2021-03-15T22:34:05.876 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84719129900000001 2021-03-15T22:34:05.876 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.876 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.876 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.877 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.877 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3971 7.7 7:ecc0cb20:::smithi07694878-25:head [write 2083358~718180] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:34:05.877 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:34:05.878 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 510.45116578, 2021-03-15T22:34:05.878 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.840294447, 2021-03-15T22:34:05.878 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.878 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:05.879 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.879 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.879 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.879 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3971 2021-03-15T22:34:05.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.880 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.880 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.880 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:34:05.880 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.881 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.881 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287234+0000", 2021-03-15T22:34:05.882 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00065142800000000001 2021-03-15T22:34:05.882 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.882 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287886+0000", 2021-03-15T22:34:05.883 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.418e-06 2021-03-15T22:34:05.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.883 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.883 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.884 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287888+0000", 2021-03-15T22:34:05.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2062000000000001e-05 2021-03-15T22:34:05.884 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.884 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.884 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.885 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287900+0000", 2021-03-15T22:34:05.885 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7250000000000002e-06 2021-03-15T22:34:05.885 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.885 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.885 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.886 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287906+0000", 2021-03-15T22:34:05.886 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9849000000000003e-05 2021-03-15T22:34:05.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.886 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.887 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.887 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287956+0000", 2021-03-15T22:34:05.887 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6460000000000005e-06 2021-03-15T22:34:05.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.887 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:34:05.888 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287965+0000", 2021-03-15T22:34:05.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83445325400000003 2021-03-15T22:34:05.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.889 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.889 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.889 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122419+0000", 2021-03-15T22:34:05.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7459e-05 2021-03-15T22:34:05.889 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.890 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.890 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.890 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122476+0000", 2021-03-15T22:34:05.890 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012662400000000001 2021-03-15T22:34:05.891 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.891 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.891 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:34:05.891 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.122603+0000", 2021-03-15T22:34:05.891 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037072580000000002 2021-03-15T22:34:05.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.892 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.892 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126310+0000", 2021-03-15T22:34:05.893 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000275853 2021-03-15T22:34:05.893 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.893 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.893 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:05.893 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126586+0000", 2021-03-15T22:34:05.894 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00087237800000000004 2021-03-15T22:34:05.894 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.894 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.894 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.894 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127458+0000", 2021-03-15T22:34:05.895 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5484000000000001e-05 2021-03-15T22:34:05.895 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.895 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.895 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:05.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127494+0000", 2021-03-15T22:34:05.896 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5009000000000003e-05 2021-03-15T22:34:05.896 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.897 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.127529+0000", 2021-03-15T22:34:05.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.840294447 2021-03-15T22:34:05.897 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.897 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.898 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.898 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.898 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3972 7.7 7:ecc0cb20:::smithi07694878-25:head [setxattr _header (58) in=7b,truncate 3109394] snapc 13e=[13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e420)", 2021-03-15T22:34:05.898 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:34:05.899 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 510.45048382200002, 2021-03-15T22:34:05.899 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84111847799999995, 2021-03-15T22:34:05.899 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.899 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:05.900 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.900 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.900 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.900 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3972 2021-03-15T22:34:05.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.901 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.901 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.901 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.901 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:34:05.902 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.902 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.902 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.902 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.902 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287916+0000", 2021-03-15T22:34:05.903 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9489999999999999e-06 2021-03-15T22:34:05.903 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.903 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.903 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.904 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287918+0000", 2021-03-15T22:34:05.904 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.359e-06 2021-03-15T22:34:05.904 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.904 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.904 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.905 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287920+0000", 2021-03-15T22:34:05.905 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8050000000000003e-06 2021-03-15T22:34:05.905 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.905 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.905 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.906 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287925+0000", 2021-03-15T22:34:05.906 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5959999999999997e-06 2021-03-15T22:34:05.906 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.906 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.907 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.907 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287930+0000", 2021-03-15T22:34:05.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2366999999999993e-05 2021-03-15T22:34:05.907 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.907 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.908 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.908 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288002+0000", 2021-03-15T22:34:05.908 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2240000000000001e-06 2021-03-15T22:34:05.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.909 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:34:05.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288012+0000", 2021-03-15T22:34:05.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83555615299999997 2021-03-15T22:34:05.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.910 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.910 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.910 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123568+0000", 2021-03-15T22:34:05.910 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4648999999999995e-05 2021-03-15T22:34:05.911 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.911 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.911 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.911 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123662+0000", 2021-03-15T22:34:05.912 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00020287300000000001 2021-03-15T22:34:05.912 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.912 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.912 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:34:05.913 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.123865+0000", 2021-03-15T22:34:05.913 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002471843 2021-03-15T22:34:05.913 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.913 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.914 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126337+0000", 2021-03-15T22:34:05.914 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00025485200000000002 2021-03-15T22:34:05.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.914 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.915 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:05.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.126592+0000", 2021-03-15T22:34:05.915 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.002302077 2021-03-15T22:34:05.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.916 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.916 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128894+0000", 2021-03-15T22:34:05.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6276999999999998e-05 2021-03-15T22:34:05.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.917 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:05.917 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.128930+0000", 2021-03-15T22:34:05.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104454 2021-03-15T22:34:05.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.918 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.918 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129035+0000", 2021-03-15T22:34:05.918 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.84111847799999995 2021-03-15T22:34:05.919 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.919 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.919 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.919 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.920 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:3973 7.7 7:ecc0cb20:::smithi07694878-25:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e420)", 2021-03-15T22:34:05.920 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:34:05.920 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 510.450452552, 2021-03-15T22:34:05.920 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.841325871, 2021-03-15T22:34:05.921 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.921 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:34:05.921 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.921 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.921 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.922 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 3973 2021-03-15T22:34:05.922 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.922 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.922 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.923 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.923 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:34:05.923 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.923 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.924 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.924 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:34:05.924 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0400000000000005e-07 2021-03-15T22:34:05.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.925 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.925 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287948+0000", 2021-03-15T22:34:05.925 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3090000000000001e-06 2021-03-15T22:34:05.925 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.926 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.926 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.926 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287950+0000", 2021-03-15T22:34:05.926 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1809999999999998e-06 2021-03-15T22:34:05.926 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.927 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287956+0000", 2021-03-15T22:34:05.927 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9060000000000004e-06 2021-03-15T22:34:05.928 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.928 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.928 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.287960+0000", 2021-03-15T22:34:05.928 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8666999999999997e-05 2021-03-15T22:34:05.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.929 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.929 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288039+0000", 2021-03-15T22:34:05.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7850000000000003e-06 2021-03-15T22:34:05.930 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.930 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.930 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:34:05.930 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:35.288045+0000", 2021-03-15T22:34:05.931 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.83615280199999997 2021-03-15T22:34:05.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.931 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.931 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.932 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124198+0000", 2021-03-15T22:34:05.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2801999999999998e-05 2021-03-15T22:34:05.932 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.932 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.932 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:05.933 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.124281+0000", 2021-03-15T22:34:05.933 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0047980210000000004 2021-03-15T22:34:05.933 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.933 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.933 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.934 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129079+0000", 2021-03-15T22:34:05.934 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2120000000000007e-05 2021-03-15T22:34:05.934 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.934 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129161+0000", 2021-03-15T22:34:05.935 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00011237400000000001 2021-03-15T22:34:05.935 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.936 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.936 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:25:36.129273+0000", 2021-03-15T22:34:05.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.841325871 2021-03-15T22:34:05.936 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.937 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.937 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.937 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.937 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4287 7.7 7:ed4a01d2:::smithi07694878-2:head [rollback 128] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:34:05.938 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.162901+0000", 2021-03-15T22:34:05.938 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 485.57549901599998, 2021-03-15T22:34:05.938 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068606072000000004, 2021-03-15T22:34:05.938 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.939 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:05.939 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.939 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.939 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.939 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4287 2021-03-15T22:34:05.940 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.940 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.940 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.941 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162901+0000", 2021-03-15T22:34:05.941 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.941 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.942 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162901+0000", 2021-03-15T22:34:05.942 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9990000000000002e-06 2021-03-15T22:34:05.942 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.942 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.942 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.943 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162905+0000", 2021-03-15T22:34:05.943 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8239999999999999e-06 2021-03-15T22:34:05.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.944 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.944 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162910+0000", 2021-03-15T22:34:05.944 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5261000000000001e-05 2021-03-15T22:34:05.944 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.945 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162935+0000", 2021-03-15T22:34:05.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6001000000000001e-05 2021-03-15T22:34:05.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.946 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162961+0000", 2021-03-15T22:34:05.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.522e-05 2021-03-15T22:34:05.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163056+0000", 2021-03-15T22:34:05.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000127291 2021-03-15T22:34:05.948 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.948 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.948 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163184+0000", 2021-03-15T22:34:05.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00030452000000000002 2021-03-15T22:34:05.949 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.949 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.949 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:34:05.949 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163488+0000", 2021-03-15T22:34:05.950 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.026637311 2021-03-15T22:34:05.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.950 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:05.951 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.190125+0000", 2021-03-15T22:34:05.951 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021491026999999999 2021-03-15T22:34:05.951 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.951 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.951 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.952 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211617+0000", 2021-03-15T22:34:05.952 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019763414 2021-03-15T22:34:05.952 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.952 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.953 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.953 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231380+0000", 2021-03-15T22:34:05.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2330000000000005e-05 2021-03-15T22:34:05.953 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.953 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.954 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:05.954 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231472+0000", 2021-03-15T22:34:05.954 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4873999999999998e-05 2021-03-15T22:34:05.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.954 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.955 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.955 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231507+0000", 2021-03-15T22:34:05.955 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068606072000000004 2021-03-15T22:34:05.955 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.956 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.956 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.956 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.956 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.956 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4288 7.7 7:ed4a01d2:::smithi07694878-2:head [append 0~0] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:34:05.957 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.162974+0000", 2021-03-15T22:34:05.957 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 485.575425806, 2021-03-15T22:34:05.957 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068646398999999997, 2021-03-15T22:34:05.957 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.958 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:05.958 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.958 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.958 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.958 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4288 2021-03-15T22:34:05.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.959 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.959 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162974+0000", 2021-03-15T22:34:05.960 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162974+0000", 2021-03-15T22:34:05.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6800000000000009e-07 2021-03-15T22:34:05.961 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.962 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.962 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162975+0000", 2021-03-15T22:34:05.962 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0729999999999999e-06 2021-03-15T22:34:05.962 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162976+0000", 2021-03-15T22:34:05.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5429999999999998e-06 2021-03-15T22:34:05.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.963 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162980+0000", 2021-03-15T22:34:05.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4399999999999999e-06 2021-03-15T22:34:05.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.162982+0000", 2021-03-15T22:34:05.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011790209999999999 2021-03-15T22:34:05.965 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.966 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164161+0000", 2021-03-15T22:34:05.966 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000104054 2021-03-15T22:34:05.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.967 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.967 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.967 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164265+0000", 2021-03-15T22:34:05.967 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019185799999999999 2021-03-15T22:34:05.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.968 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.968 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:34:05.968 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164457+0000", 2021-03-15T22:34:05.969 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025674285000000002 2021-03-15T22:34:05.969 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.969 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.969 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:05.969 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.190132+0000", 2021-03-15T22:34:05.970 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021520336000000001 2021-03-15T22:34:05.970 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.970 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.970 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.971 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211652+0000", 2021-03-15T22:34:05.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019892508999999999 2021-03-15T22:34:05.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.971 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.971 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.972 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231544+0000", 2021-03-15T22:34:05.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0618000000000003e-05 2021-03-15T22:34:05.972 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.973 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:05.973 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231585+0000", 2021-03-15T22:34:05.973 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5694000000000001e-05 2021-03-15T22:34:05.973 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.973 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.974 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.974 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.231621+0000", 2021-03-15T22:34:05.974 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068646398999999997 2021-03-15T22:34:05.974 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.975 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.975 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.975 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.975 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4290 7.d 7:bd183e1b:::smithi07694878-46:head [stat,delete] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:34:05.976 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.163011+0000", 2021-03-15T22:34:05.976 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 485.57538880999999, 2021-03-15T22:34:05.976 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048696824999999999, 2021-03-15T22:34:05.976 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.977 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:05.977 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:05.977 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:05.977 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:05.977 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4290 2021-03-15T22:34:05.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.978 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.978 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.978 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163011+0000", 2021-03-15T22:34:05.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.979 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163011+0000", 2021-03-15T22:34:05.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.079e-06 2021-03-15T22:34:05.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.980 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163012+0000", 2021-03-15T22:34:05.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.043e-06 2021-03-15T22:34:05.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:05.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163013+0000", 2021-03-15T22:34:05.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7620000000000003e-06 2021-03-15T22:34:05.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.983 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:05.983 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163018+0000", 2021-03-15T22:34:05.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4980000000000003e-06 2021-03-15T22:34:05.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:05.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163025+0000", 2021-03-15T22:34:05.984 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6739999999999997e-05 2021-03-15T22:34:05.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.985 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:05.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163061+0000", 2021-03-15T22:34:05.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9894000000000003e-05 2021-03-15T22:34:05.986 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.986 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.986 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:05.986 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163161+0000", 2021-03-15T22:34:05.987 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017699600000000001 2021-03-15T22:34:05.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.987 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 1,3", 2021-03-15T22:34:05.987 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.163338+0000", 2021-03-15T22:34:05.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001543619 2021-03-15T22:34:05.988 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.988 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.988 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:05.988 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164882+0000", 2021-03-15T22:34:05.989 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.021337467999999998 2021-03-15T22:34:05.989 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.989 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.989 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.990 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.186219+0000", 2021-03-15T22:34:05.990 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.025381363000000001 2021-03-15T22:34:05.990 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.990 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.990 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:05.991 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211601+0000", 2021-03-15T22:34:05.991 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9228999999999998e-05 2021-03-15T22:34:05.991 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.992 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:05.992 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211660+0000", 2021-03-15T22:34:05.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8133999999999997e-05 2021-03-15T22:34:05.992 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.992 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.993 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:05.993 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.211708+0000", 2021-03-15T22:34:05.993 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048696824999999999 2021-03-15T22:34:05.993 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.994 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:05.994 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:05.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.994 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.995 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_repop(client.24517.0:4284 7.b e445/401 7:d4368cae:::smithi07694878-18:head v 445'396, mlcod=445'394)", 2021-03-15T22:34:05.995 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:00.164429+0000", 2021-03-15T22:34:05.995 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 485.57397058999999, 2021-03-15T22:34:05.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048217024999999997, 2021-03-15T22:34:05.995 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:05.996 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:05.996 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:05.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:05.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164429+0000", 2021-03-15T22:34:05.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:05.997 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.997 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:05.998 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164429+0000", 2021-03-15T22:34:05.998 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00035440200000000003 2021-03-15T22:34:05.998 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.998 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:05.998 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:05.999 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164784+0000", 2021-03-15T22:34:05.999 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.659e-06 2021-03-15T22:34:05.999 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:05.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:06.000 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164788+0000", 2021-03-15T22:34:06.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8576999999999998e-05 2021-03-15T22:34:06.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:06.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164806+0000", 2021-03-15T22:34:06.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0139999999999998e-06 2021-03-15T22:34:06.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.002 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:06.002 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164814+0000", 2021-03-15T22:34:06.002 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.7796000000000001e-05 2021-03-15T22:34:06.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164862+0000", 2021-03-15T22:34:06.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2391e-05 2021-03-15T22:34:06.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.004 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:06.004 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.164884+0000", 2021-03-15T22:34:06.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.047722025000000001 2021-03-15T22:34:06.005 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:06.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.212606+0000", 2021-03-15T22:34:06.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0160999999999997e-05 2021-03-15T22:34:06.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.006 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:06.006 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:00.212647+0000", 2021-03-15T22:34:06.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.048217024999999997 2021-03-15T22:34:06.007 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.007 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:06.007 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.007 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.008 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.008 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7907 7.c 7:3fac7ed7:::smithi07694878-41:head [write 484332~573697] snapc 2bc=[2bc,2bb,2b8,2b7,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e837)", 2021-03-15T22:34:06.008 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:33:07.379424+0000", 2021-03-15T22:34:06.008 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 58.358975932, 2021-03-15T22:34:06.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.979796202999999, 2021-03-15T22:34:06.009 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:06.009 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:06.009 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:06.009 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:06.010 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:06.010 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7907 2021-03-15T22:34:06.010 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.010 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:06.011 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.011 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:06.011 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.379424+0000", 2021-03-15T22:34:06.011 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:06.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.012 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.012 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:06.012 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.379424+0000", 2021-03-15T22:34:06.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000574477 2021-03-15T22:34:06.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.013 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.013 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:06.013 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.379999+0000", 2021-03-15T22:34:06.013 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.4659999999999999e-06 2021-03-15T22:34:06.014 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.014 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:06.014 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.380002+0000", 2021-03-15T22:34:06.014 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1892e-05 2021-03-15T22:34:06.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.015 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:06.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.380024+0000", 2021-03-15T22:34:06.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.799e-06 2021-03-15T22:34:06.016 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:06.016 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.380033+0000", 2021-03-15T22:34:06.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6356e-05 2021-03-15T22:34:06.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.380099+0000", 2021-03-15T22:34:06.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1007999999999998e-05 2021-03-15T22:34:06.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.018 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.018 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for readable", 2021-03-15T22:34:06.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.380140+0000", 2021-03-15T22:34:06.019 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.974887803 2021-03-15T22:34:06.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.355028+0000", 2021-03-15T22:34:06.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4281000000000001e-05 2021-03-15T22:34:06.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:06.021 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.355122+0000", 2021-03-15T22:34:06.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000210161 2021-03-15T22:34:06.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:34:06.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.355332+0000", 2021-03-15T22:34:06.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0037114209999999999 2021-03-15T22:34:06.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:06.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.359044+0000", 2021-03-15T22:34:06.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.224e-05 2021-03-15T22:34:06.024 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.024 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:06.024 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.359106+0000", 2021-03-15T22:34:06.024 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0154e-05 2021-03-15T22:34:06.025 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.025 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:06.025 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.359116+0000", 2021-03-15T22:34:06.026 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6609e-05 2021-03-15T22:34:06.026 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.026 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:06.026 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.359163+0000", 2021-03-15T22:34:06.027 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7535999999999999e-05 2021-03-15T22:34:06.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.027 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:06.028 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.359220+0000", 2021-03-15T22:34:06.028 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.979796202999999 2021-03-15T22:34:06.028 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.028 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:06.028 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.029 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.029 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.029 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7908 7.c 7:3fac7ed7:::smithi07694878-41:head [write 1531566~552766] snapc 2bc=[2bc,2bb,2b8,2b7,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e837)", 2021-03-15T22:34:06.029 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:33:07.385084+0000", 2021-03-15T22:34:06.029 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 58.353316407000001, 2021-03-15T22:34:06.030 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.978276144000001, 2021-03-15T22:34:06.030 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:06.030 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:06.030 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:06.031 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:06.031 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:06.031 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7908 2021-03-15T22:34:06.031 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.031 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:06.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.032 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:06.032 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385084+0000", 2021-03-15T22:34:06.032 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:06.033 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.033 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.033 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:06.033 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385084+0000", 2021-03-15T22:34:06.033 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00051728799999999997 2021-03-15T22:34:06.034 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.034 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.034 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:06.034 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385601+0000", 2021-03-15T22:34:06.035 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5499999999999996e-06 2021-03-15T22:34:06.035 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.035 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.035 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:06.035 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385606+0000", 2021-03-15T22:34:06.036 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7550999999999998e-05 2021-03-15T22:34:06.036 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:06.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385623+0000", 2021-03-15T22:34:06.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1160000000000005e-06 2021-03-15T22:34:06.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.037 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:06.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385632+0000", 2021-03-15T22:34:06.038 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4653999999999995e-05 2021-03-15T22:34:06.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385697+0000", 2021-03-15T22:34:06.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6267000000000002e-05 2021-03-15T22:34:06.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for readable", 2021-03-15T22:34:06.040 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385753+0000", 2021-03-15T22:34:06.040 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.971051794999999 2021-03-15T22:34:06.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.356805+0000", 2021-03-15T22:34:06.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.4081999999999999e-05 2021-03-15T22:34:06.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.042 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:06.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.356889+0000", 2021-03-15T22:34:06.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018215899999999999 2021-03-15T22:34:06.042 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.043 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.043 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:34:06.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.357071+0000", 2021-03-15T22:34:06.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021676289999999999 2021-03-15T22:34:06.044 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.044 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.044 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:06.044 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.359239+0000", 2021-03-15T22:34:06.044 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0038719340000000001 2021-03-15T22:34:06.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.045 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:06.045 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.363111+0000", 2021-03-15T22:34:06.045 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8928e-05 2021-03-15T22:34:06.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.046 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.046 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:06.046 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.363200+0000", 2021-03-15T22:34:06.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8856000000000004e-05 2021-03-15T22:34:06.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:06.047 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.363288+0000", 2021-03-15T22:34:06.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1334999999999994e-05 2021-03-15T22:34:06.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:06.049 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.363360+0000", 2021-03-15T22:34:06.049 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.978276144000001 2021-03-15T22:34:06.049 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.049 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:06.049 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.050 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7909 7.c 7:3fac7ed7:::smithi07694878-41:head [setxattr _header (58) in=7b,truncate 2084332] snapc 2bc=[2bc,2bb,2b8,2b7,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e837)", 2021-03-15T22:34:06.050 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:33:07.385646+0000", 2021-03-15T22:34:06.051 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 58.352753761999999, 2021-03-15T22:34:06.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.97803442, 2021-03-15T22:34:06.051 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:06.051 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:06.051 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:06.052 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:06.052 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:06.052 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7909 2021-03-15T22:34:06.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.052 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:06.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.053 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:06.053 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385646+0000", 2021-03-15T22:34:06.053 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:06.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.054 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:06.054 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385646+0000", 2021-03-15T22:34:06.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1440000000000001e-06 2021-03-15T22:34:06.055 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.055 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.055 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:06.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385648+0000", 2021-03-15T22:34:06.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4899999999999999e-06 2021-03-15T22:34:06.056 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.056 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:06.056 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385650+0000", 2021-03-15T22:34:06.057 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6699999999999997e-06 2021-03-15T22:34:06.057 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.057 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.057 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:06.057 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385657+0000", 2021-03-15T22:34:06.058 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3490000000000002e-06 2021-03-15T22:34:06.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.058 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.058 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:06.059 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385661+0000", 2021-03-15T22:34:06.059 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000137424 2021-03-15T22:34:06.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.060 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385798+0000", 2021-03-15T22:34:06.060 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0439000000000001e-05 2021-03-15T22:34:06.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.060 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for readable", 2021-03-15T22:34:06.061 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385829+0000", 2021-03-15T22:34:06.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.972378213000001 2021-03-15T22:34:06.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.062 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.062 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.358207+0000", 2021-03-15T22:34:06.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9380000000000001e-05 2021-03-15T22:34:06.062 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.062 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.063 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:06.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.358306+0000", 2021-03-15T22:34:06.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000189784 2021-03-15T22:34:06.063 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.064 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:34:06.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.358496+0000", 2021-03-15T22:34:06.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0041577029999999996 2021-03-15T22:34:06.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:34:06.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.362654+0000", 2021-03-15T22:34:06.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00078554199999999997 2021-03-15T22:34:06.066 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:06.066 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.363439+0000", 2021-03-15T22:34:06.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.2874000000000006e-05 2021-03-15T22:34:06.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.067 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.067 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:06.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.363522+0000", 2021-03-15T22:34:06.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7093000000000003e-05 2021-03-15T22:34:06.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:34:06.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.363589+0000", 2021-03-15T22:34:06.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.1315000000000002e-05 2021-03-15T22:34:06.069 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:06.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.363681+0000", 2021-03-15T22:34:06.070 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.97803442 2021-03-15T22:34:06.070 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.070 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:06.070 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.071 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.071 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7910 7.c 7:3fac7ed7:::smithi07694878-41:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e837)", 2021-03-15T22:34:06.071 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:33:07.385679+0000", 2021-03-15T22:34:06.071 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 58.352721267, 2021-03-15T22:34:06.072 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.978203475000001, 2021-03-15T22:34:06.072 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:06.072 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:34:06.072 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:06.073 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:06.073 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:06.073 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7910 2021-03-15T22:34:06.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.073 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:06.074 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.074 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:06.074 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385679+0000", 2021-03-15T22:34:06.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:06.075 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.075 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.075 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:06.075 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385679+0000", 2021-03-15T22:34:06.075 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2750000000000001e-06 2021-03-15T22:34:06.076 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.076 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.076 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:06.076 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385680+0000", 2021-03-15T22:34:06.077 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6109999999999999e-06 2021-03-15T22:34:06.077 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.077 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.077 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:06.077 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385682+0000", 2021-03-15T22:34:06.078 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5679999999999996e-06 2021-03-15T22:34:06.078 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.078 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.078 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:06.079 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385688+0000", 2021-03-15T22:34:06.079 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5229999999999997e-06 2021-03-15T22:34:06.079 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.079 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.079 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:06.080 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385693+0000", 2021-03-15T22:34:06.080 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000163747 2021-03-15T22:34:06.080 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.080 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.080 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.081 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385857+0000", 2021-03-15T22:34:06.081 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0587000000000003e-05 2021-03-15T22:34:06.081 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.081 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.082 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for readable", 2021-03-15T22:34:06.082 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:07.385897+0000", 2021-03-15T22:34:06.082 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.973386563 2021-03-15T22:34:06.082 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.082 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.083 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.083 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.359284+0000", 2021-03-15T22:34:06.083 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8235000000000001e-05 2021-03-15T22:34:06.083 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.084 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.084 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:06.084 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.359342+0000", 2021-03-15T22:34:06.084 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0043886480000000002 2021-03-15T22:34:06.084 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.085 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.085 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.085 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.363731+0000", 2021-03-15T22:34:06.085 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.3676999999999994e-05 2021-03-15T22:34:06.085 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.086 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.086 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:06.086 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.363794+0000", 2021-03-15T22:34:06.086 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.8041000000000006e-05 2021-03-15T22:34:06.087 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.087 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.087 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:06.087 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.363882+0000", 2021-03-15T22:34:06.087 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 13.978203475000001 2021-03-15T22:34:06.088 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.088 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:06.088 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.088 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.089 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.089 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:8055 7.c 7:3bc1c17e:::smithi07694878-8:2c3 [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e851)", 2021-03-15T22:34:06.089 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:33:21.296923+0000", 2021-03-15T22:34:06.089 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 44.441477452000001, 2021-03-15T22:34:06.089 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.067892806, 2021-03-15T22:34:06.090 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:06.090 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-15T22:34:06.090 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:06.090 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:06.090 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:06.091 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 8055 2021-03-15T22:34:06.091 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.091 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:06.091 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.092 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:06.092 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.296923+0000", 2021-03-15T22:34:06.092 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:06.092 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.092 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.093 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:06.093 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.296923+0000", 2021-03-15T22:34:06.093 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.797e-06 2021-03-15T22:34:06.093 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.094 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.094 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:06.094 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.296927+0000", 2021-03-15T22:34:06.094 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.303e-06 2021-03-15T22:34:06.094 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.095 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.095 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:06.095 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.296931+0000", 2021-03-15T22:34:06.095 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.8484000000000001e-05 2021-03-15T22:34:06.096 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.096 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.096 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:06.096 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.296949+0000", 2021-03-15T22:34:06.096 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0129999999999999e-06 2021-03-15T22:34:06.097 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.097 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.097 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:06.097 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.296958+0000", 2021-03-15T22:34:06.097 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000110318 2021-03-15T22:34:06.098 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.098 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.098 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.098 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297069+0000", 2021-03-15T22:34:06.098 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1387e-05 2021-03-15T22:34:06.099 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.099 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.099 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:34:06.099 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297080+0000", 2021-03-15T22:34:06.100 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.003179248 2021-03-15T22:34:06.100 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.100 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.100 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.100 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.300259+0000", 2021-03-15T22:34:06.101 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4898999999999997e-05 2021-03-15T22:34:06.101 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.101 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.101 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for readable", 2021-03-15T22:34:06.102 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.300314+0000", 2021-03-15T22:34:06.102 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.064327753000000001 2021-03-15T22:34:06.102 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.102 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.103 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.103 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.364642+0000", 2021-03-15T22:34:06.103 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000173604 2021-03-15T22:34:06.103 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.103 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.104 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:06.104 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.364815+0000", 2021-03-15T22:34:06.104 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.067892806 2021-03-15T22:34:06.104 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.104 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:06.105 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.105 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.105 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.105 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:8056 7.c 7:3bc1c17e:::smithi07694878-8:2c3 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e851)", 2021-03-15T22:34:06.105 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:33:21.296989+0000", 2021-03-15T22:34:06.106 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 44.441410933, 2021-03-15T22:34:06.106 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068053519000000007, 2021-03-15T22:34:06.106 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:06.106 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-15T22:34:06.107 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:06.107 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:06.107 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:06.107 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 8056 2021-03-15T22:34:06.108 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.108 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:06.108 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.108 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:06.108 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.296989+0000", 2021-03-15T22:34:06.109 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:06.109 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.109 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.109 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:06.109 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.296989+0000", 2021-03-15T22:34:06.110 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.857e-06 2021-03-15T22:34:06.110 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.110 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.110 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:06.111 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.296991+0000", 2021-03-15T22:34:06.111 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.227e-06 2021-03-15T22:34:06.111 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.111 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.111 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:06.112 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.296993+0000", 2021-03-15T22:34:06.112 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.6612e-05 2021-03-15T22:34:06.112 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.112 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.112 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:06.113 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297010+0000", 2021-03-15T22:34:06.113 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4740000000000002e-06 2021-03-15T22:34:06.113 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.113 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.114 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:06.114 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297016+0000", 2021-03-15T22:34:06.114 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000134416 2021-03-15T22:34:06.114 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.114 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.115 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.119 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297151+0000", 2021-03-15T22:34:06.119 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9670000000000004e-06 2021-03-15T22:34:06.119 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.119 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.120 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:34:06.120 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297158+0000", 2021-03-15T22:34:06.120 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0031900909999999999 2021-03-15T22:34:06.120 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.121 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.121 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.121 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.300348+0000", 2021-03-15T22:34:06.121 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0672e-05 2021-03-15T22:34:06.121 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.122 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.122 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for readable", 2021-03-15T22:34:06.122 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.300388+0000", 2021-03-15T22:34:06.122 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.064464932000000003 2021-03-15T22:34:06.123 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.123 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.123 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.123 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.364853+0000", 2021-03-15T22:34:06.123 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000189271 2021-03-15T22:34:06.124 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.124 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.124 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:06.124 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.365043+0000", 2021-03-15T22:34:06.125 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068053519000000007 2021-03-15T22:34:06.125 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.125 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:06.125 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.126 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.126 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.126 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:8057 7.c 7:3bc1c17e:::smithi07694878-8:2c3 [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected e851)", 2021-03-15T22:34:06.126 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:33:21.297106+0000", 2021-03-15T22:34:06.126 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 44.441294585000001, 2021-03-15T22:34:06.127 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068078825999999995, 2021-03-15T22:34:06.127 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:06.127 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "reached pg", 2021-03-15T22:34:06.127 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:34:06.127 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:34:06.128 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:06.128 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 8057 2021-03-15T22:34:06.128 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.128 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:06.129 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.129 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:06.129 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297106+0000", 2021-03-15T22:34:06.129 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:06.129 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.130 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.130 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:06.130 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297106+0000", 2021-03-15T22:34:06.130 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.194e-06 2021-03-15T22:34:06.131 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.131 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.131 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:06.131 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297110+0000", 2021-03-15T22:34:06.131 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5959999999999998e-06 2021-03-15T22:34:06.132 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.132 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.132 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:06.132 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297112+0000", 2021-03-15T22:34:06.133 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1398e-05 2021-03-15T22:34:06.133 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.133 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.133 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:06.133 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297124+0000", 2021-03-15T22:34:06.134 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5089999999999999e-06 2021-03-15T22:34:06.134 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.134 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.134 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:06.135 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297130+0000", 2021-03-15T22:34:06.135 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4178999999999997e-05 2021-03-15T22:34:06.135 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.135 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.135 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.136 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297184+0000", 2021-03-15T22:34:06.136 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6890000000000004e-06 2021-03-15T22:34:06.136 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.136 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.137 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:34:06.137 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.297192+0000", 2021-03-15T22:34:06.137 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0032254179999999999 2021-03-15T22:34:06.137 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.137 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.138 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.138 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.300417+0000", 2021-03-15T22:34:06.138 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8952000000000002e-05 2021-03-15T22:34:06.138 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.139 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.139 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for readable", 2021-03-15T22:34:06.139 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.300466+0000", 2021-03-15T22:34:06.139 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.064625771999999998 2021-03-15T22:34:06.139 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.140 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.140 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:06.140 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.365092+0000", 2021-03-15T22:34:06.140 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.2119000000000004e-05 2021-03-15T22:34:06.140 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:06.141 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:06.141 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:34:06.141 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:21.365184+0000", 2021-03-15T22:34:06.141 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.068078825999999995 2021-03-15T22:34:06.142 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.142 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:06.142 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.142 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:06.142 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:06.143 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:34:06.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:06.145 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[56514]: cluster 2021-03-15T22:34:04.216770+0000 osd.4 (osd.4) 184 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:06.145 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[56514]: cluster 2021-03-15T22:34:04.218580+0000 osd.4 (osd.4) 185 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:06.145 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[56514]: cluster 2021-03-15T22:34:04.561451+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e889: 8 total, 8 up, 7 in 2021-03-15T22:34:06.145 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[56514]: cluster 2021-03-15T22:34:04.563478+0000 osd.6 (osd.6) 130 : cluster [DBG] 5.1f starting backfill to osd.3 from (0'0,0'0] MAX to 841'19 2021-03-15T22:34:06.146 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[56514]: cluster 2021-03-15T22:34:05.307903+0000 mon.a (mon.0 2021-03-15T22:34:06.146 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[56514]: ) 2264 : cluster [DBG] osdmap e890: 8 total, 8 up, 7 in 2021-03-15T22:34:06.146 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:05 smithi125 conmon[56739]: cluster 2021-03-15T22:34:04.216770+0000 osd.4 (osd.4) 184 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:06.146 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:05 smithi125 conmon[56739]: cluster 2021-03-15T22:34:04.218580+0000 osd.4 (osd.4) 185 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:06.147 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:05 smithi125 conmon[56739]: cluster 2021-03-15T22:34:04.561451+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e889: 8 total, 8 up, 7 in 2021-03-15T22:34:06.147 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:05 smithi125 conmon[56739]: cluster 2021-03-15T22:34:04.563478+0000 osd.6 (osd.6) 130 : cluster [DBG] 5.1f starting backfill to osd.3 from (0'0,0'0] MAX to 841'19 2021-03-15T22:34:06.147 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:05 smithi125 conmon[56739]: cluster 2021-03-15T22:34:05.307903+0000 mon.a (mon.0) 2264 : cluster [DBG] osdmap e890: 8 total, 8 up, 7 in 2021-03-15T22:34:06.148 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[62156]: cluster 2021-03-15T22:34:04.216770+0000 osd.4 (osd.4) 184 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:06.148 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[62156]: cluster 2021-03-15T22:34:04.218580+0000 osd.4 (osd.4) 185 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:06.148 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[62156]: cluster 2021-03-15T22:34:04.561451+0000 mon.a (mon.0) 2263 : cluster [DBG] osdmap e889: 8 total, 8 up, 7 in 2021-03-15T22:34:06.148 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[62156]: cluster 2021-03-15T22:34:04.563478+0000 osd.6 (osd.6) 130 : cluster [DBG] 5.1f starting backfill to osd.3 from (0'0,0'0] MAX to 841'19 2021-03-15T22:34:06.149 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[62156]: cluster 2021-03-15T22:34:05.307903+0000 mon.a ( 2021-03-15T22:34:06.149 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:05 smithi076 conmon[62156]: mon.0) 2264 : cluster [DBG] osdmap e890: 8 total, 8 up, 7 in 2021-03-15T22:34:06.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:06.170 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:06.171 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:06.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:06.295 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:06.297 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:06.310 INFO:tasks.rados.rados.0.smithi076.stdout:3349: done (4 left) 2021-03-15T22:34:06.311 INFO:tasks.rados.rados.0.smithi076.stdout:3350: done (3 left) 2021-03-15T22:34:06.311 INFO:tasks.rados.rados.0.smithi076.stdout:3351: done (2 left) 2021-03-15T22:34:06.311 INFO:tasks.rados.rados.0.smithi076.stdout:3352: done (1 left) 2021-03-15T22:34:06.311 INFO:tasks.rados.rados.0.smithi076.stdout:3353: done (0 left) 2021-03-15T22:34:06.311 INFO:tasks.rados.rados.0.smithi076.stdout:3354: rollback oid 31 current snap is 394 2021-03-15T22:34:06.312 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 31 to 386 2021-03-15T22:34:06.312 INFO:tasks.rados.rados.0.smithi076.stdout:3355: copy_from oid 47 from oid 2 current snap is 394 2021-03-15T22:34:06.312 INFO:tasks.rados.rados.0.smithi076.stdout:3356: snap_create 2021-03-15T22:34:06.313 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:34:06.318 INFO:tasks.rados.rados.0.smithi076.stdout:3354: finishing rollback tid 0 to smithi07694878-31 2021-03-15T22:34:06.318 INFO:tasks.rados.rados.0.smithi076.stdout:3354: finishing rollback tid 1 to smithi07694878-31 2021-03-15T22:34:06.319 INFO:tasks.rados.rados.0.smithi076.stdout:3354: finishing rollback tid 2 to smithi07694878-31 2021-03-15T22:34:06.319 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 2420 (ObjNum 1112 snap 386 seq_num 1112) dirty exists 2021-03-15T22:34:06.322 INFO:tasks.rados.rados.0.smithi076.stdout:3355: finishing copy_from to smithi07694878-47 2021-03-15T22:34:06.322 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 2402 (ObjNum 1111 snap 386 seq_num 1111) dirty exists 2021-03-15T22:34:06.322 INFO:tasks.rados.rados.0.smithi076.stdout:3355: finishing copy_from racing read to smithi07694878-47 2021-03-15T22:34:06.398 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:06.404 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-15T22:34:06.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:06.421 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:06.421 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:06.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:06.545 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:06.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:06.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:06.648 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:06.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[56514]: cluster 2021-03-15T22:34:05.199426+0000 mgr.y (mgr 2021-03-15T22:34:06.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[56514]: .14144) 1016 : cluster [DBG] pgmap v1481: 148 pgs: 1 active+recovering, 1 peering, 1 active+clean+wait, 145 active+clean; 631 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 4.7 MiB/s rd, 524 KiB/s wr, 7 op/s; 6/1848 objects misplaced (0.325%) 2021-03-15T22:34:06.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[56514]: audit 2021-03-15T22:34:05.784468+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? 172.21.15.76:0/3675799870' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:34:06.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[56514]: cluster 2021-03-15T22:34:06.302994+0000 mon.a (mon.0) 2266 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:06.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[56514]: audit 2021-03-15T22:34:06.307454+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? 172.21.15.76:0/3675799870' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:34:06.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[56514]: cluster 2021-03-15T22:34:06.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[56514]: 2021-03-15T22:34:06.307509+0000 mon.a (mon.0) 2268 : cluster [DBG] osdmap e891: 8 total, 8 up, 7 in 2021-03-15T22:34:06.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: cluster 2021-03-15T22:34:05.199426+0000 mgr.y (mgr 2021-03-15T22:34:06.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: .14144) 1016 : cluster [DBG] pgmap v1481: 148 pgs: 1 active+recovering, 1 peering, 1 active+clean+wait, 145 active+clean; 631 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 4.7 MiB/s rd, 524 KiB/s wr, 7 op/s; 6/1848 objects misplaced (0.325%) 2021-03-15T22:34:06.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: audit 2021-03-15T22:34:05.784468+0000 mon.a 2021-03-15T22:34:06.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: (mon.0) 2265 : audit [INF] from='client.? 172.21.15.76:0/3675799870' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:34:06.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: cluster 2021-03-15T22:34:06.302994+0000 2021-03-15T22:34:06.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: mon.a (mon.0) 2266 : cluster [WRN] 2021-03-15T22:34:06.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:06.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:34:06.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: 22:34:06.307454+0000 mon.a (mon 2021-03-15T22:34:06.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: .0) 2267 : audit [INF] from='client.? 172.21.15.76:0/3675799870' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:34:06.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:34:06.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: -03-15T22:34 2021-03-15T22:34:06.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: :06.307509+0000 mon.a (mon.0 2021-03-15T22:34:06.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:06 smithi076 conmon[62156]: ) 2268 : cluster [DBG] osdmap e891: 8 total, 8 up, 7 in 2021-03-15T22:34:06.670 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:06.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:06.671 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:06.696 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:34:06.772 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:06.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:06 smithi125 conmon[56739]: cluster 2021-03-15T22:34:05.199426+0000 mgr.y (mgr.14144) 1016 : 2021-03-15T22:34:06.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:06 smithi125 conmon[56739]: cluster [DBG] pgmap v1481: 148 pgs: 1 active+recovering, 1 peering, 1 active+clean+wait, 145 active+clean; 631 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 4.7 MiB/s rd, 524 KiB/s wr, 7 op/s; 6/1848 objects misplaced (0.325%) 2021-03-15T22:34:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:06 smithi125 conmon[56739]: audit 2021-03-15T22:34:05.784468+0000 mon.a (mon.0) 2265 : audit [INF] from='client.? 172.21.15.76:0/3675799870' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:34:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:06 smithi125 conmon[56739]: cluster 2021-03-15T22:34:06.302994+0000 mon.a (mon.0) 2266 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:06 smithi125 conmon[56739]: audit 2021-03-15T22:34:06.307454+0000 mon.a (mon.0) 2267 : audit [INF] from='client.? 172.21.15.76:0/3675799870' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:34:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:06 smithi125 conmon[56739]: cluster 2021-03-15T22:34:06.307509+0000 mon.a (mon.0) 2268 : cluster 2021-03-15T22:34:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:06 smithi125 conmon[56739]: [DBG] osdmap e891: 8 total, 8 up, 7 in 2021-03-15T22:34:06.794 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:06.796 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:06.897 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:06.919 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:06.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:06.921 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:07.022 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:07.044 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:07.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:07.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:07.146 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:07.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:07.170 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:07.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:07.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:07.295 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:07.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:07.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:07.314 INFO:tasks.rados.rados.0.smithi076.stdout:3354: done (2 left) 2021-03-15T22:34:07.315 INFO:tasks.rados.rados.0.smithi076.stdout:3355: done (1 left) 2021-03-15T22:34:07.315 INFO:tasks.rados.rados.0.smithi076.stdout:3356: done (0 left) 2021-03-15T22:34:07.315 INFO:tasks.rados.rados.0.smithi076.stdout:3357: snap_create 2021-03-15T22:34:07.398 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:07.422 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:07.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:07.424 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:07.525 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:07.547 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:07.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:07.549 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:07.650 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:07.673 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:07.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:07.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:07.775 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:07.798 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:07.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:07.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:07.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:07.924 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:07.925 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:08.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:08.039 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:34:08.039 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:34:08.040 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:34:08.040 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:34:08.040 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:34:08.041 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:34:08.041 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:34:08.041 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:34:08.042 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:34:08.042 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:34:08.042 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:34:08.043 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:34:08.043 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:34:08.043 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:34:08.043 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:34:08.043 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:34:08.050 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:08.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:08.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:08.148 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:34:08.149 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:34:08.152 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:08.174 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:08.176 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:08.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:08.299 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:08.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:08.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:08.324 INFO:tasks.rados.rados.0.smithi076.stdout:3357: done (0 left) 2021-03-15T22:34:08.324 INFO:tasks.rados.rados.0.smithi076.stdout:3358: read oid 21 snap 387 2021-03-15T22:34:08.324 INFO:tasks.rados.rados.0.smithi076.stdout:3358: expect (ObjNum 1110 snap 386 seq_num 1110) 2021-03-15T22:34:08.325 INFO:tasks.rados.rados.0.smithi076.stdout:3359: snap_remove snap 353 2021-03-15T22:34:08.401 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:08.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:08.425 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:08.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:08.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[56514]: cluster 2021-03-15T22:34:07.125151+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:34:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[56514]: cluster 2021-03-15T22:34:07.199941+0000 mgr.y (mgr.14144) 1017 : cluster [DBG] pgmap v1484: 148 pgs: 1 active+recovering, 2 peering, 145 active+clean; 635 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.7 MiB/s rd, 1.0 MiB/s wr, 9 op/s; 6/1860 objects misplaced (0.323%) 2021-03-15T22:34:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[56514]: cluster 2021-03-15T22:34:07.203635+0000 osd.6 (osd.6) 131 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:34:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[56514]: cluster 2021-03-15T22:34:07.311027+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e892: 8 total, 8 up, 7 in 2021-03-15T22:34:08.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[56514]: cluster 2021-03-15T22:34:07.452568+0000 osd.4 (osd.4) 186 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:34:08.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[56514]: audit 2021-03-15T22:34:07.893799+0000 mon.a (mon.0) 2270 : audit [DBG] from='client.? 172.21.15.76:0/1481619536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: cluster 2021-03-15T22:34:08.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: 2021-03-15T22:34:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: 2021 2021-03-15T22:34:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: - 2021-03-15T22:34:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: 03 2021-03-15T22:34:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: - 2021-03-15T22:34:08.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: 15T22:34:07 2021-03-15T22:34:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: .125151+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:34:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:34:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: :34:07.199941+0000 mgr.y (mgr.14144) 2021-03-15T22:34:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: 1017 : cluster [DBG] pgmap v1484: 148 pgs: 1 active+recovering, 2 peering, 145 active+clean; 635 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.7 MiB/s rd, 1.0 MiB/s wr, 9 op/s; 6/1860 objects misplaced (0.323%) 2021-03-15T22:34:08.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: cluster 2021-03-15T22:34:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: 2021-03-15 2021-03-15T22:34:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: T22:34:07.203635+0000 osd.6 (osd.6) 131 : 2021-03-15T22:34:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: cluster [DBG] 3.10 scrub ok 2021-03-15T22:34:08.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:34:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: T22:34:07.311027+0000 mon.a (mon.0) 2269 : cluster [DBG] 2021-03-15T22:34:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: osdmap e892: 8 total, 8 up, 7 in 2021-03-15T22:34:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:34:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: 03-15T22:34:07.452568+0000 osd.4 2021-03-15T22:34:08.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: (osd.4) 186 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:34:08.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: 2021-03-15T22:34:08.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: audit 2021-03-15T22:34:08.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: 2021-03-15T22:34:07.893799+0000 2021-03-15T22:34:08.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: mon.a (mon.0) 2270 : audit [DBG] 2021-03-15T22:34:08.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:08 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/1481619536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:08.527 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:08.549 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:08.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:08.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:08.651 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:08.675 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:08.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:08.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:08.740 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:08 smithi125 conmon[56739]: cluster 2021-03-15T22:34:07.125151+0000 osd.0 (osd.0) 145 : cluster [DBG] 4.1e scrub ok 2021-03-15T22:34:08.740 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:08 smithi125 conmon[56739]: cluster 2021-03-15T22:34:07.199941+0000 mgr.y (mgr.14144) 1017 : cluster [DBG] pgmap v1484: 148 pgs: 1 active+recovering, 2 peering, 145 active+clean; 635 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.7 MiB/s rd, 1.0 MiB/s wr, 9 op/s; 6/1860 objects misplaced (0.323%) 2021-03-15T22:34:08.741 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:08 smithi125 conmon[56739]: cluster 2021-03-15T22:34:07.203635+0000 osd.6 (osd.6) 131 : cluster [DBG] 3.10 scrub ok 2021-03-15T22:34:08.741 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:08 smithi125 conmon[56739]: cluster 2021-03-15T22:34:07.311027+0000 mon.a (mon.0) 2269 : cluster [DBG] osdmap e892: 8 total, 8 up, 7 in 2021-03-15T22:34:08.741 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:08 smithi125 conmon[56739]: cluster 2021-03-15T22:34:07.452568+0000 osd.4 (osd.4) 186 : cluster [DBG] 4.9 scrub ok 2021-03-15T22:34:08.741 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:08 smithi125 conmon[56739]: audit 2021-03-15T22:34:07.893799+0000 mon.a (mon.0) 2270 : audit [DBG] from='client.? 172.21.15.76:0/1481619536' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:08.763 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-15T22:34:08.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:08.801 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:08.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:08.802 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:08.903 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:08.925 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:08.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:08.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:09.028 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:09.038 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:34:09.052 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:09.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:09.054 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:09.155 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:09.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:09.179 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:09.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:09.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:09.303 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:09.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:09.305 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:09.329 INFO:teuthology.orchestra.run.smithi076.stdout:moved 49 / 444 (11.036%) 2021-03-15T22:34:09.330 INFO:teuthology.orchestra.run.smithi076.stdout:avg 63.4286 2021-03-15T22:34:09.330 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 5.0669 -> 9.78128 (expected baseline 7.37342) 2021-03-15T22:34:09.331 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.5 with 57 -> 60 pgs (0.898649 -> 0.945946 * mean) 2021-03-15T22:34:09.331 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.3 with 72 -> 42 pgs (1.13514 -> 0.662162 * mean) 2021-03-15T22:34:09.331 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:34:09.331 INFO:teuthology.orchestra.run.smithi076.stdout:oload 110 2021-03-15T22:34:09.332 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 3 2021-03-15T22:34:09.332 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:34:09.332 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0028 2021-03-15T22:34:09.333 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0030 2021-03-15T22:34:09.333 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3 weight 1.0000 -> 0.5592 2021-03-15T22:34:09.333 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1 weight 1.0000 -> 0.8249 2021-03-15T22:34:09.334 INFO:tasks.rados.rados.0.smithi076.stdout:3359: done (1 left) 2021-03-15T22:34:09.334 INFO:tasks.rados.rados.0.smithi076.stdout:3360: snap_remove snap 341 2021-03-15T22:34:09.406 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:09.428 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:09.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:09.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:09.490 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:09 smithi125 conmon[56739]: cluster 2021-03-15T22:34:07.807407+0000 2021-03-15T22:34:09.490 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:09 smithi125 conmon[56739]: osd.5 (osd.5) 47 : cluster [DBG] 3.c scrub ok 2021-03-15T22:34:09.490 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:09 smithi125 conmon[56739]: cluster 2021-03-15T22:34:08.321501+0000 mon.a (mon.0) 2271 : cluster [DBG] osdmap e893: 8 total, 8 up, 7 in 2021-03-15T22:34:09.491 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:09 smithi125 conmon[56739]: cluster 2021-03-15T22:34:08.508820+0000 osd.7 (osd.7) 169 : cluster [DBG] 7.5 scrub ok 2021-03-15T22:34:09.491 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:09 smithi125 conmon[56739]: cluster 2021-03-15T22:34:08.652726+0000 osd.3 (osd.3) 162 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:34:09.491 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:09 smithi125 conmon[56739]: audit 2021-03-15T22:34:09.184654+0000 mon.a (mon.0) 2272 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'54059','3':'36650'}"}]: dispatch 2021-03-15T22:34:09.528 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:34:09.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:09.593 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:09.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:09.595 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:09.696 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:09.717 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:09.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:09.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[56514]: cluster 2021-03-15T22:34:07.807407+0000 osd.5 (osd 2021-03-15T22:34:09.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[56514]: .5) 47 : cluster [DBG] 3.c scrub ok 2021-03-15T22:34:09.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[56514]: cluster 2021-03-15T22:34:08.321501+0000 mon.a (mon.0) 2271 : cluster [DBG] osdmap e893: 8 total, 8 up, 7 in 2021-03-15T22:34:09.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[56514]: cluster 2021-03-15T22:34:08.508820+0000 osd.7 (osd.7) 169 : cluster [DBG] 7.5 scrub ok 2021-03-15T22:34:09.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[56514]: cluster 2021-03-15T22:34:08.652726+0000 osd.3 (osd.3) 162 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:34:09.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[56514]: audit 2021-03-15T22:34:09.184654+0000 mon.a (mon.0) 2272 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'54059','3':'36650'}"}]: dispatch 2021-03-15T22:34:09.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: cluster 2021-03-15T22:34:07.807407 2021-03-15T22:34:09.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 47 : cluster [DBG] 3.c scrub ok 2021-03-15T22:34:09.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:34:09.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: -15T22:34:08.321501+0000 mon.a (mon.0) 2271 : cluster 2021-03-15T22:34:09.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: [DBG] osdmap e893: 8 total, 8 up, 7 in 2021-03-15T22:34:09.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: cluster 2021-03-15T22:34: 2021-03-15T22:34:09.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: 08.508820+0000 osd.7 (osd.7) 169 : cluster 2021-03-15T22:34:09.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: [DBG] 7.5 scrub ok 2021-03-15T22:34:09.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:34:09.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: 34:08.652726+0000 osd.3 (osd.3) 2021-03-15T22:34:09.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: 162 : cluster [DBG] 6.1b scrub ok 2021-03-15T22:34:09.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:34:09.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: 22:34:09.184654+0000 mon.a (mon. 2021-03-15T22:34:09.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:09 smithi076 conmon[62156]: 0) 2272 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd reweightn", "weights": "{'1':'54059','3':'36650'}"}]: dispatch 2021-03-15T22:34:09.820 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:09.842 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:09.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:09.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:09.944 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:09.966 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:09.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:09.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:10.036 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:10.059 INFO:tasks.thrashosds.thrasher:Reweighting by: {'max_change': '1.0', 'overage': '1000', 'type': 'reweight-by-utilization'} 2021-03-15T22:34:10.059 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight-by-utilization 1000 1.0 2021-03-15T22:34:10.069 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:10.123 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:10.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:10.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:10.226 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:10.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:10.251 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:10.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:10.323 INFO:tasks.rados.rados.0.smithi076.stdout:3360: done (1 left) 2021-03-15T22:34:10.324 INFO:tasks.rados.rados.0.smithi076.stdout:3361: snap_create 2021-03-15T22:34:10.324 INFO:tasks.rados.rados.0.smithi076.stdout:3358: expect (ObjNum 1110 snap 386 seq_num 1110) 2021-03-15T22:34:10.352 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:10.377 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:10.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:10.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:10.399 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:10 smithi125 conmon[56739]: audit 2021-03-15T22:34:09.183763+0000 mgr.y (mgr. 2021-03-15T22:34:10.399 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:10 smithi125 conmon[56739]: 14144) 1018 : audit [DBG] from='client.25492 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:10.400 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:10 smithi125 conmon[56739]: cluster 2021-03-15T22:34:09.200290+0000 mgr.y (mgr.14144) 1019 : cluster [DBG] pgmap v1487: 148 pgs: 1 active+recovering, 1 peering, 146 active+clean; 637 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.7 MiB/s rd, 1.0 MiB/s wr, 8 op/s; 6/1863 objects misplaced (0.322%); 65 B/s, 0 objects/s recovering 2021-03-15T22:34:10.400 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:10 smithi125 conmon[56739]: cluster 2021-03-15T22:34:09.215026+0000 osd.6 (osd.6) 132 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:34:10.400 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:10 smithi125 conmon[56739]: audit 2021-03-15T22:34:09.327674+0000 mon.a (mon.0) 2273 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'54059','3':'36650'}"}]': finished 2021-03-15T22:34:10.400 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:10 smithi125 conmon[56739]: cluster 2021-03-15T22:34:09.327745+0000 mon.a (mon.0) 2274 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2021-03-15T22:34:10.400 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:10 smithi125 conmon[56739]: cluster 2021-03-15T22:34:10.308155+0000 mon.a (mon.0) 2275 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2021-03-15T22:34:10.401 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:34:10 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:34:10.481 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:10.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:10.505 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:10.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:10.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: audit 2021-03-15T22:34:09.183763+0000 mgr.y (mgr.14144) 1018 : audit [DBG] from='client.25492 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:10.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: cluster 2021-03-15T22:34:09.200290+0000 mgr.y (mgr.14144) 1019 : cluster [DBG] pgmap v1487: 148 pgs: 1 active+recovering, 1 peering, 146 active+clean; 637 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.7 MiB/s rd, 1.0 MiB/s wr, 8 op/s; 6/1863 objects misplaced (0.322%); 65 B/s, 0 objects/s recovering 2021-03-15T22:34:10.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: cluster 2021-03-15T22:34:09.215026+0000 osd.6 (osd.6) 132 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:34:10.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: audit 2021-03-15T22:34:09.327674+0000 mon.a (mon.0) 2273 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'54059','3':'36650'}"}]': finished 2021-03-15T22:34:10.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: cluster 2021-03-15T22:34:10.589 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: 2021 2021-03-15T22:34:10.590 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: - 2021-03-15T22:34:10.590 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: 03 2021-03-15T22:34:10.590 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: - 2021-03-15T22:34:10.590 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: 15 2021-03-15T22:34:10.591 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: T 2021-03-15T22:34:10.591 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: 22 2021-03-15T22:34:10.591 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: : 2021-03-15T22:34:10.591 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: 34 2021-03-15T22:34:10.591 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: : 2021-03-15T22:34:10.592 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: 09 2021-03-15T22:34:10.592 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: . 2021-03-15T22:34:10.592 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: 327745 2021-03-15T22:34:10.592 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: +0000 mon.a (mon.0 2021-03-15T22:34:10.593 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: ) 2274 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2021-03-15T22:34:10.593 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: cluster 2021- 2021-03-15T22:34:10.593 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: 03-15T22:34:10.308155+0000 mon.a 2021-03-15T22:34:10.593 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: (mon.0) 2275 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2021-03-15T22:34:10.593 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[56514]: 2021-03-15T22:34:10.594 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: audit 2021-03-15T22:34:09. 2021-03-15T22:34:10.594 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: 183763+0000 mgr.y (mgr.14144) 1018 : audit [DBG] from='client.25492 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 110, "max_change": 3.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:10.594 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: cluster 2021-03-15T22:34:09. 2021-03-15T22:34:10.595 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: 200290+0000 mgr.y (mgr.14144) 1019 : cluster 2021-03-15T22:34:10.595 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: [DBG] pgmap v1487: 148 pgs: 1 active+recovering, 1 peering, 146 active+clean; 637 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 2.7 MiB/s rd, 1.0 MiB/s wr, 8 op/s; 6/1863 objects misplaced (0.322%); 65 B/s, 0 objects/s recovering 2021-03-15T22:34:10.595 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:34:10.595 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: -15T22:34:09.215026+0000 2021-03-15T22:34:10.596 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: osd.6 (osd.6) 132 : cluster 2021-03-15T22:34:10.596 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: [DBG] 4.5 scrub ok 2021-03-15T22:34:10.596 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:34:10.596 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: -15T22:34:09.327674+0000 2021-03-15T22:34:10.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: mon.a (mon.0) 2273 : audit [INF] 2021-03-15T22:34:10.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd reweightn", "weights": "{'1':'54059','3':'36650'}"}]': finished 2021-03-15T22:34:10.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:34:10.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: T22:34:09.327745+0000 mon.a 2021-03-15T22:34:10.597 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: (mon.0) 2274 : cluster [DBG] osdmap e894: 8 total, 8 up, 7 in 2021-03-15T22:34:10.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: 2021-03-15T22:34:10.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:34:10.598 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: :34:10.308155+0000 mon.a (mon.0 2021-03-15T22:34:10.599 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:10 smithi076 conmon[62156]: ) 2275 : cluster [DBG] osdmap e895: 8 total, 8 up, 7 in 2021-03-15T22:34:10.599 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:10.605 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:10.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:10.628 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:10.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:10.642 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:34:10.643 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:34:10.730 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:34:10.731 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:10.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:10.755 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:10.755 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:10.856 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:10.879 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:10.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:10.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:10.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:11.003 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:11.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:11.004 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:11.105 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:11.128 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:11.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:11.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:11.230 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:11.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:11.254 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:11.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:11.313 INFO:tasks.rados.rados.0.smithi076.stdout:3358: done (1 left) 2021-03-15T22:34:11.314 INFO:tasks.rados.rados.0.smithi076.stdout:3361: done (0 left) 2021-03-15T22:34:11.314 INFO:tasks.rados.rados.0.smithi076.stdout:3362: read oid 39 snap -1 2021-03-15T22:34:11.314 INFO:tasks.rados.rados.0.smithi076.stdout:3362: expect deleted 2021-03-15T22:34:11.315 INFO:tasks.rados.rados.0.smithi076.stdout:3363: copy_from oid 42 from oid 4 current snap is 397 2021-03-15T22:34:11.315 INFO:tasks.rados.rados.0.smithi076.stdout:3364: write (excl) oid 16 current snap is 397 2021-03-15T22:34:11.315 INFO:tasks.rados.rados.0.smithi076.stdout:3364: seq_num 1141 ranges {776787=512059,1718425=766397,3000223=176564} 2021-03-15T22:34:11.321 INFO:tasks.rados.rados.0.smithi076.stdout:3364: writing smithi07694878-16 from 776787 to 1288846 tid 1 2021-03-15T22:34:11.329 INFO:tasks.rados.rados.0.smithi076.stdout:3364: writing smithi07694878-16 from 1718425 to 2484822 tid 2 2021-03-15T22:34:11.334 INFO:tasks.rados.rados.0.smithi076.stdout:3364: writing smithi07694878-16 from 3000223 to 3176787 tid 3 2021-03-15T22:34:11.334 INFO:tasks.rados.rados.0.smithi076.stdout:3362: done (2 left) 2021-03-15T22:34:11.334 INFO:tasks.rados.rados.0.smithi076.stdout:3365: snap_create 2021-03-15T22:34:11.355 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:11.356 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:11.384 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-15T22:34:11.387 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:11.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:11.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:11.401 INFO:tasks.rados.rados.0.smithi076.stdout:3364: finishing write tid 1 to smithi07694878-16 2021-03-15T22:34:11.410 INFO:tasks.rados.rados.0.smithi076.stdout:3364: finishing write tid 2 to smithi07694878-16 2021-03-15T22:34:11.410 INFO:tasks.rados.rados.0.smithi076.stdout:3364: finishing write tid 3 to smithi07694878-16 2021-03-15T22:34:11.411 INFO:tasks.rados.rados.0.smithi076.stdout:3364: finishing write tid 4 to smithi07694878-16 2021-03-15T22:34:11.411 INFO:tasks.rados.rados.0.smithi076.stdout:3364: finishing write tid 5 to smithi07694878-16 2021-03-15T22:34:11.411 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 2781 (ObjNum 1141 snap 397 seq_num 1141) dirty exists 2021-03-15T22:34:11.411 INFO:tasks.rados.rados.0.smithi076.stdout:3364: left oid 16 (ObjNum 1141 snap 397 seq_num 1141) 2021-03-15T22:34:11.414 INFO:tasks.rados.rados.0.smithi076.stdout:3363: finishing copy_from to smithi07694878-42 2021-03-15T22:34:11.414 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2000 (ObjNum 1109 snap 386 seq_num 1109) dirty exists 2021-03-15T22:34:11.414 INFO:tasks.rados.rados.0.smithi076.stdout:3363: finishing copy_from racing read to smithi07694878-42 2021-03-15T22:34:11.489 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:11.517 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:11.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:11.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:11.619 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:11.629 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[56514]: cluster 2021-03-15T22:34:10.222517 2021-03-15T22:34:11.630 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[56514]: +0000 osd.1 (osd.1) 171 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:34:11.630 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[56514]: audit 2021-03-15T22:34:10.225286+0000 mgr.y (mgr.14144) 1020 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:11.630 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[56514]: cluster 2021-03-15T22:34:10.249436+0000 osd.6 (osd.6) 133 : cluster [DBG] 4.a scrub ok 2021-03-15T22:34:11.630 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[56514]: cluster 2021-03-15T22:34:11.309387+0000 mon.a (mon.0) 2276 : cluster [DBG] osdmap e896: 8 total, 8 up, 7 in 2021-03-15T22:34:11.631 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: cluster 2021-03-15T22:34:11.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021 2021-03-15T22:34:11.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: - 2021-03-15T22:34:11.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 03 2021-03-15T22:34:11.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: - 2021-03-15T22:34:11.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 15 2021-03-15T22:34:11.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: T 2021-03-15T22:34:11.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 22 2021-03-15T22:34:11.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.634 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 34 2021-03-15T22:34:11.634 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.634 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 10 2021-03-15T22:34:11.634 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: . 2021-03-15T22:34:11.634 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 222517 2021-03-15T22:34:11.635 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: +0000 2021-03-15T22:34:11.635 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.635 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: osd.1 2021-03-15T22:34:11.635 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: ( 2021-03-15T22:34:11.635 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: osd 2021-03-15T22:34:11.636 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: . 2021-03-15T22:34:11.636 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 1 2021-03-15T22:34:11.636 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: ) 2021-03-15T22:34:11.636 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 171 2021-03-15T22:34:11.637 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.637 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: cluster 2021-03-15T22:34:11.637 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.637 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: [DBG] 2021-03-15T22:34:11.637 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.638 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 3.0 scrub ok 2021-03-15T22:34:11.638 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.638 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: audit 2021-03-15T22:34:11.638 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.638 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021 2021-03-15T22:34:11.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: - 2021-03-15T22:34:11.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 03 2021-03-15T22:34:11.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: - 2021-03-15T22:34:11.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 15 2021-03-15T22:34:11.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: T 2021-03-15T22:34:11.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 22 2021-03-15T22:34:11.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 34 2021-03-15T22:34:11.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 10 2021-03-15T22:34:11.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: . 2021-03-15T22:34:11.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 225286 2021-03-15T22:34:11.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: +0000 2021-03-15T22:34:11.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: mgr.y 2021-03-15T22:34:11.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: ( 2021-03-15T22:34:11.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: mgr 2021-03-15T22:34:11.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: . 2021-03-15T22:34:11.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 14144 2021-03-15T22:34:11.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: ) 2021-03-15T22:34:11.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 1020 2021-03-15T22:34:11.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: audit 2021-03-15T22:34:11.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: [DBG] 2021-03-15T22:34:11.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:11.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: cluster 2021-03-15T22:34:11.645 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.645 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021 2021-03-15T22:34:11.645 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: - 2021-03-15T22:34:11.645 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 03 2021-03-15T22:34:11.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: - 2021-03-15T22:34:11.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 15 2021-03-15T22:34:11.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: T 2021-03-15T22:34:11.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 22 2021-03-15T22:34:11.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 34 2021-03-15T22:34:11.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 10 2021-03-15T22:34:11.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: . 2021-03-15T22:34:11.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 249436 2021-03-15T22:34:11.648 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: +0000 2021-03-15T22:34:11.648 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.648 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: osd.6 2021-03-15T22:34:11.648 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: ( 2021-03-15T22:34:11.648 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: osd 2021-03-15T22:34:11.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: . 2021-03-15T22:34:11.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 6 2021-03-15T22:34:11.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: ) 2021-03-15T22:34:11.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 133 2021-03-15T22:34:11.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: cluster 2021-03-15T22:34:11.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: [DBG] 2021-03-15T22:34:11.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.651 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 4.a scrub ok 2021-03-15T22:34:11.651 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.651 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: cluster 2021-03-15T22:34:11.651 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.651 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021 2021-03-15T22:34:11.652 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: - 2021-03-15T22:34:11.652 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 03 2021-03-15T22:34:11.652 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: - 2021-03-15T22:34:11.652 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 15 2021-03-15T22:34:11.652 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: T 2021-03-15T22:34:11.653 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 22 2021-03-15T22:34:11.653 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.653 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 34 2021-03-15T22:34:11.653 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.653 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 11 2021-03-15T22:34:11.654 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: . 2021-03-15T22:34:11.654 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 309387 2021-03-15T22:34:11.654 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: +0000 2021-03-15T22:34:11.654 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.655 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: mon.a 2021-03-15T22:34:11.655 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: ( 2021-03-15T22:34:11.655 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: mon 2021-03-15T22:34:11.655 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: . 2021-03-15T22:34:11.655 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 0 2021-03-15T22:34:11.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: ) 2021-03-15T22:34:11.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2276 2021-03-15T22:34:11.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: : 2021-03-15T22:34:11.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: cluster 2021-03-15T22:34:11.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: [DBG] 2021-03-15T22:34:11.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: osdmap e896: 8 total, 8 up, 7 in 2021-03-15T22:34:11.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:11 smithi076 conmon[62156]: 2021-03-15T22:34:11.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:11.659 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:11.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:11.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:11.783 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:11.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:11.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:11 smithi125 conmon[56739]: cluster 2021-03-15T22:34:10.222517+0000 osd.1 (osd. 2021-03-15T22:34:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:11 smithi125 conmon[56739]: 1) 171 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:34:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:11 smithi125 conmon[56739]: audit 2021-03-15T22:34:10.225286+0000 mgr.y (mgr.14144) 1020 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:11 smithi125 conmon[56739]: cluster 2021-03-15T22:34:10.249436+0000 osd.6 (osd.6) 133 : cluster [DBG] 4.a scrub ok 2021-03-15T22:34:11.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:11 smithi125 conmon[56739]: cluster 2021-03-15T22:34:11.309387+0000 mon.a (mon.0) 2276 : cluster [DBG] osdmap e896: 8 total, 8 up, 7 in 2021-03-15T22:34:11.801 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:34:11.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:11.909 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:11.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:11.911 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:12.012 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:12.035 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:12.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:12.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:12.137 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:12.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:12.168 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:12.168 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:12.270 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:12.292 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:12.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:12.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:12.312 INFO:tasks.rados.rados.0.smithi076.stdout:3363: done (2 left) 2021-03-15T22:34:12.313 INFO:tasks.rados.rados.0.smithi076.stdout:3364: done (1 left) 2021-03-15T22:34:12.313 INFO:tasks.rados.rados.0.smithi076.stdout:3365: done (0 left) 2021-03-15T22:34:12.313 INFO:tasks.rados.rados.0.smithi076.stdout:3366: snap_remove snap 374 2021-03-15T22:34:12.395 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:12.417 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:12.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:12.418 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:12.519 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:12.547 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:12.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:12.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:12.734 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:12 smithi076 conmon[56514]: cluster 2021-03-15T22:34:11.201050+0000 mgr.y (mgr.14144) 1021 : cluster [DBG] pgmap v1490: 148 pgs: 5 remapped+peering, 143 active+clean; 644 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2021-03-15T22:34:12.734 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:12 smithi076 conmon[56514]: cluster 2021-03-15T22:34:11.311807+0000 osd.0 (osd.0) 146 : cluster [DBG] 5.1e starting backfill to osd.3 from (0'0,0'0] MAX to 643'13 2021-03-15T22:34:12.735 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:12 smithi076 conmon[56514]: cluster 2021-03-15T22:34:12.309993+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2021-03-15T22:34:12.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:12 smithi076 conmon[62156]: cluster 2021-03-15T22:34:11.201050+0000 mgr.y (mgr.14144) 1021 : cluster [DBG] pgmap v1490: 148 pgs: 5 remapped+peering, 143 active+clean; 644 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2021-03-15T22:34:12.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:12 smithi076 conmon[62156]: cluster 2021-03-15T22:34:11.311807+0000 osd.0 (osd.0) 146 : cluster [DBG] 5.1e starting backfill to osd.3 from (0'0,0'0] MAX to 643'13 2021-03-15T22:34:12.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:12 smithi076 conmon[62156]: cluster 2021-03-15T22:34:12.309993+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2021-03-15T22:34:12.753 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:12.777 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:12.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:12.779 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:12.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:12 smithi125 conmon[56739]: cluster 2021-03-15T22:34:11.201050+0000 mgr.y (mgr.14144) 1021 : cluster [DBG] pgmap v1490: 148 pgs: 5 remapped+peering, 143 active+clean; 644 MiB data, 1.7 GiB used, 624 GiB / 626 GiB avail; 1.9 MiB/s rd, 1.3 MiB/s wr, 7 op/s 2021-03-15T22:34:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:12 smithi125 conmon[56739]: cluster 2021-03-15T22:34:11.311807+0000 osd.0 (osd.0) 146 : cluster [DBG] 5.1e starting backfill to osd.3 from (0'0,0'0] MAX to 643'13 2021-03-15T22:34:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:12 smithi125 conmon[56739]: cluster 2021-03-15T22:34:12.309993+0000 mon.a (mon.0) 2277 : cluster [DBG] osdmap e897: 8 total, 8 up, 7 in 2021-03-15T22:34:12.880 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:12.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:12.905 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:12.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:13.007 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:13.031 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:13.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:13.033 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:13.134 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:13.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:13.157 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:13.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:13.259 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:13.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:13.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:13.322 INFO:tasks.rados.rados.0.smithi076.stdout:3366: done (0 left) 2021-03-15T22:34:13.322 INFO:tasks.rados.rados.0.smithi076.stdout:3367: delete oid 12 current snap is 398 2021-03-15T22:34:13.325 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:34:13.340 INFO:tasks.rados.rados.0.smithi076.stdout:3367: done (0 left) 2021-03-15T22:34:13.340 INFO:tasks.rados.rados.0.smithi076.stdout:3368: write (excl) oid 40 current snap is 398 2021-03-15T22:34:13.341 INFO:tasks.rados.rados.0.smithi076.stdout:3368: seq_num 1143 ranges {410401=662557,1719036=770066,2938716=271685} 2021-03-15T22:34:13.346 INFO:tasks.rados.rados.0.smithi076.stdout:3368: writing smithi07694878-40 from 410401 to 1072958 tid 1 2021-03-15T22:34:13.355 INFO:tasks.rados.rados.0.smithi076.stdout:3368: writing smithi07694878-40 from 1719036 to 2489102 tid 2 2021-03-15T22:34:13.361 INFO:tasks.rados.rados.0.smithi076.stdout:3368: writing smithi07694878-40 from 2938716 to 3210401 tid 3 2021-03-15T22:34:13.361 INFO:tasks.rados.rados.0.smithi076.stdout:3368: finishing write tid 1 to smithi07694878-40 2021-03-15T22:34:13.361 INFO:tasks.rados.rados.0.smithi076.stdout:3369: read oid 42 snap 391 2021-03-15T22:34:13.361 INFO:tasks.rados.rados.0.smithi076.stdout:3369: expect deleted 2021-03-15T22:34:13.362 INFO:tasks.rados.rados.0.smithi076.stdout:3370: read oid 35 snap -1 2021-03-15T22:34:13.362 INFO:tasks.rados.rados.0.smithi076.stdout:3370: expect deleted 2021-03-15T22:34:13.362 INFO:tasks.rados.rados.0.smithi076.stdout:3371: copy_from oid 31 from oid 16 current snap is 398 2021-03-15T22:34:13.362 INFO:tasks.rados.rados.0.smithi076.stdout:3372: rollback oid 10 current snap is 398 2021-03-15T22:34:13.363 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 10 to 380 2021-03-15T22:34:13.363 INFO:tasks.rados.rados.0.smithi076.stdout:3373: copy_from oid 37 from oid 9 current snap is 398 2021-03-15T22:34:13.363 INFO:tasks.rados.rados.0.smithi076.stdout:3374: copy_from oid 3 from oid 1 current snap is 398 2021-03-15T22:34:13.363 INFO:tasks.rados.rados.0.smithi076.stdout:3375: snap_create 2021-03-15T22:34:13.365 INFO:tasks.rados.rados.0.smithi076.stdout:3372: finishing rollback tid 0 to smithi07694878-10 2021-03-15T22:34:13.365 INFO:tasks.rados.rados.0.smithi076.stdout:3372: finishing rollback tid 1 to smithi07694878-10 2021-03-15T22:34:13.365 INFO:tasks.rados.rados.0.smithi076.stdout:3372: finishing rollback tid 2 to smithi07694878-10 2021-03-15T22:34:13.366 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 2406 (ObjNum 1077 snap 366 seq_num 1077) dirty exists 2021-03-15T22:34:13.367 INFO:tasks.rados.rados.0.smithi076.stdout:3368: finishing write tid 2 to smithi07694878-40 2021-03-15T22:34:13.367 INFO:tasks.rados.rados.0.smithi076.stdout:3368: finishing write tid 3 to smithi07694878-40 2021-03-15T22:34:13.368 INFO:tasks.rados.rados.0.smithi076.stdout:3368: finishing write tid 4 to smithi07694878-40 2021-03-15T22:34:13.369 INFO:tasks.rados.rados.0.smithi076.stdout:3368: finishing write tid 5 to smithi07694878-40 2021-03-15T22:34:13.369 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 2957 (ObjNum 1143 snap 398 seq_num 1143) dirty exists 2021-03-15T22:34:13.369 INFO:tasks.rados.rados.0.smithi076.stdout:3368: left oid 40 (ObjNum 1143 snap 398 seq_num 1143) 2021-03-15T22:34:13.379 INFO:tasks.rados.rados.0.smithi076.stdout:3374: finishing copy_from to smithi07694878-3 2021-03-15T22:34:13.379 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 2398 (ObjNum 982 snap 326 seq_num 982) dirty exists 2021-03-15T22:34:13.379 INFO:tasks.rados.rados.0.smithi076.stdout:3374: finishing copy_from racing read to smithi07694878-3 2021-03-15T22:34:13.384 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:13.391 INFO:tasks.rados.rados.0.smithi076.stdout:3371: finishing copy_from to smithi07694878-31 2021-03-15T22:34:13.391 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 2781 (ObjNum 1141 snap 397 seq_num 1141) dirty exists 2021-03-15T22:34:13.392 INFO:tasks.rados.rados.0.smithi076.stdout:3373: finishing copy_from to smithi07694878-37 2021-03-15T22:34:13.392 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 2777 (ObjNum 1092 snap 375 seq_num 1092) dirty exists 2021-03-15T22:34:13.392 INFO:tasks.rados.rados.0.smithi076.stdout:3371: finishing copy_from racing read to smithi07694878-31 2021-03-15T22:34:13.392 INFO:tasks.rados.rados.0.smithi076.stdout:3373: finishing copy_from racing read to smithi07694878-37 2021-03-15T22:34:13.396 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[67289]: debug 2021-03-15T22:34:13.396 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[67289]: 2021-03-15T22:34:13.321+0000 7fd4d602c700 -1 osd.6 pg_epoch: 898 pg[7.c( v 890'1200 (0'0,890'1200] local-lis/les=897/898 n=17 ec=91/91 lis/c=897/897 les/c/f=898/898/0 sis=897) [6,7,0] r=0 lpr=897 crt=890'1200 lcod 890'1199 mlcod 0'0 active+clean+snaptrim trimq=[291~1,2ab~1,2cb~1]] removing snap head 2021-03-15T22:34:13.397 INFO:journalctl@ceph.osd.6.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[67289]: 2021-03-15T22:34:13.397 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: audit 2021-03-15T22:34:13.118741+0000 mon.a (mon. 2021-03-15T22:34:13.397 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: 0) 2278 : audit [INF] from='client.? 172.21.15.76:0/2264083864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:13.398 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: cluster 2021-03-15T22:34:13.307545+0000 mon.a (mon.0) 2279 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:34:13.398 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: cluster 2021-03-15T22:34:13.307572+0000 mon.a (mon.0 2021-03-15T22:34:13.398 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: ) 2280 : cluster [INF] Cluster is now healthy 2021-03-15T22:34:13.398 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:34:13.399 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: 15T22:34:13.313016+0000 mon.a ( 2021-03-15T22:34:13.399 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: mon.0) 2281 : audit [INF] from='client.? 172.21.15.76:0/2264083864' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:13.399 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:34:13.399 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: -03-15T22:34:13.313091+0000 2021-03-15T22:34:13.399 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: mon.a (mon.0) 2282 : cluster [DBG] 2021-03-15T22:34:13.400 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:13 smithi125 conmon[56739]: osdmap e898: 8 total, 8 up, 7 in 2021-03-15T22:34:13.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:13.406 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:13.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:13.508 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:13.529 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:13.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:13.531 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:13.616 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:34:13.617 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:34:13.631 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:13.655 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:13.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:13.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:13.757 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:13 smithi076 conmon[56514]: audit 2021-03-15T22:34:13.118741+0000 mon.a (mon.0) 2278 : 2021-03-15T22:34:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:13 smithi076 conmon[56514]: audit [INF] from='client.? 172.21.15.76:0/2264083864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:13.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:13 smithi076 conmon[56514]: cluster 2021-03-15T22:34:13.307545+0000 mon.a (mon.0) 2279 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:34:13.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:13 smithi076 conmon[56514]: cluster 2021-03-15T22:34:13.307572+0000 mon.a (mon.0) 2280 : cluster [INF] Cluster is now healthy 2021-03-15T22:34:13.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:13 smithi076 conmon[56514]: audit 2021-03-15T22:34:13.313016+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? 172.21.15.76:0/2264083864' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:13.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:13 smithi076 conmon[56514]: cluster 2021-03-15T22:34:13.313091+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e898: 8 total, 8 up, 7 in 2021-03-15T22:34:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:13 smithi076 conmon[62156]: audit 2021-03-15T22:34:13.118741+0000 mon.a (mon.0) 2278 : audit [INF] from='client.? 172.21.15.76:0/2264083864' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:13 smithi076 conmon[62156]: cluster 2021-03-15T22:34:13.307545+0000 mon.a (mon.0) 2279 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:34:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:13 smithi076 conmon[62156]: cluster 2021-03-15T22:34:13.307572+0000 mon.a (mon.0) 2280 : cluster [INF] Cluster is now healthy 2021-03-15T22:34:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:13 smithi076 conmon[62156]: audit 2021-03-15T22:34:13.313016+0000 mon.a (mon.0) 2281 : audit [INF] from='client.? 172.21.15.76:0/2264083864' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:13.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:13 smithi076 conmon[62156]: cluster 2021-03-15T22:34:13.313091+0000 mon.a (mon.0) 2282 : cluster [DBG] osdmap e898: 8 total, 8 up, 7 in 2021-03-15T22:34:13.784 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:13.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:13.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:13.886 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:13.908 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:13.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:13.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:14.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:14.031 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:14.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:14.032 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:14.133 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:14.154 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:14.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:14.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:14.257 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:14.278 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:14.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:14.280 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:14.320 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:34:14.328 INFO:teuthology.orchestra.run.smithi076.stdout:moved 0 / 444 (0%) 2021-03-15T22:34:14.329 INFO:teuthology.orchestra.run.smithi076.stdout:avg 63.4286 2021-03-15T22:34:14.329 INFO:teuthology.orchestra.run.smithi076.stdout:stddev 9.78128 -> 9.78128 (expected baseline 7.37342) 2021-03-15T22:34:14.329 INFO:teuthology.orchestra.run.smithi076.stdout:min osd.3 with 42 -> 42 pgs (0.662162 -> 0.662162 * mean) 2021-03-15T22:34:14.329 INFO:teuthology.orchestra.run.smithi076.stdout:max osd.7 with 74 -> 74 pgs (1.16667 -> 1.16667 * mean) 2021-03-15T22:34:14.330 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:34:14.330 INFO:teuthology.orchestra.run.smithi076.stdout:oload 1000 2021-03-15T22:34:14.330 INFO:teuthology.orchestra.run.smithi076.stdout:max_change 1 2021-03-15T22:34:14.330 INFO:teuthology.orchestra.run.smithi076.stdout:max_change_osds 4 2021-03-15T22:34:14.331 INFO:teuthology.orchestra.run.smithi076.stdout:average_utilization 0.0028 2021-03-15T22:34:14.331 INFO:teuthology.orchestra.run.smithi076.stdout:overload_utilization 0.0283 2021-03-15T22:34:14.331 INFO:teuthology.orchestra.run.smithi076.stderr:no change 2021-03-15T22:34:14.358 INFO:tasks.rados.rados.0.smithi076.stdout:3368: done (7 left) 2021-03-15T22:34:14.358 INFO:tasks.rados.rados.0.smithi076.stdout:3369: done (6 left) 2021-03-15T22:34:14.359 INFO:tasks.rados.rados.0.smithi076.stdout:3370: done (5 left) 2021-03-15T22:34:14.359 INFO:tasks.rados.rados.0.smithi076.stdout:3371: done (4 left) 2021-03-15T22:34:14.359 INFO:tasks.rados.rados.0.smithi076.stdout:3372: done (3 left) 2021-03-15T22:34:14.359 INFO:tasks.rados.rados.0.smithi076.stdout:3373: done (2 left) 2021-03-15T22:34:14.359 INFO:tasks.rados.rados.0.smithi076.stdout:3374: done (1 left) 2021-03-15T22:34:14.360 INFO:tasks.rados.rados.0.smithi076.stdout:3375: done (0 left) 2021-03-15T22:34:14.360 INFO:tasks.rados.rados.0.smithi076.stdout:3376: snap_create 2021-03-15T22:34:14.381 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:14.425 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:14.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:14.426 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:14.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:14.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:14.554 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:14.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:14.654 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:34:14.655 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:14.680 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:14.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:14.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:14.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:14 smithi076 conmon[56514]: cluster 2021-03-15T22:34:13.201423+0000 mgr.y (mgr.14144) 1022 : cluster [DBG] pgmap v1493: 148 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 5 remapped+peering, 141 active+clean; 644 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.9 MiB/s rd, 959 KiB/s wr, 5 op/s; 7/1869 objects misplaced (0.375%); 5.6 MiB/s, 26 objects/s recovering 2021-03-15T22:34:14.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:14 smithi076 conmon[62156]: cluster 2021-03-15T22:34:13.201423+0000 2021-03-15T22:34:14.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:14 smithi076 conmon[62156]: mgr.y (mgr.14144) 1022 : cluster [DBG] pgmap v1493: 148 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 5 remapped+peering, 141 active+clean; 644 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.9 MiB/s rd, 959 KiB/s wr, 5 op/s; 7/1869 objects misplaced (0.375%); 5.6 MiB/s, 26 objects/s recovering 2021-03-15T22:34:14.783 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:14 smithi125 conmon[56739]: cluster 2021-03-15T22:34:13.201423+0000 mgr.y (mgr.14144 2021-03-15T22:34:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:14 smithi125 conmon[56739]: ) 1022 : cluster [DBG] pgmap v1493: 148 pgs: 1 active+undersized+remapped, 1 active+recovering+undersized+remapped, 5 remapped+peering, 141 active+clean; 644 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 1.9 MiB/s rd, 959 KiB/s wr, 5 op/s; 7/1869 objects misplaced (0.375%); 5.6 MiB/s, 26 objects/s recovering 2021-03-15T22:34:14.804 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:14.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:14.805 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:14.906 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:14.928 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:14.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:14.929 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:15.030 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:15.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:15.054 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:15.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:15.156 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:15.180 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:15.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:15.182 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:15.230 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:34:15.230 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:34:15.230 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:34:15.231 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:34:15.231 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:34:15.231 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:34:15.231 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:34:15.231 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:34:15.232 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:34:15.232 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:34:15.232 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:34:15.232 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:34:15.233 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:34:15.233 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:34:15.233 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:34:15.233 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:34:15.283 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:15.305 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:15.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:15.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:15.408 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:15.431 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:15.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:15.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:15.463 INFO:tasks.rados.rados.0.smithi076.stdout:3376: done (0 left) 2021-03-15T22:34:15.464 INFO:tasks.rados.rados.0.smithi076.stdout:3377: write (excl) oid 47 current snap is 400 2021-03-15T22:34:15.464 INFO:tasks.rados.rados.0.smithi076.stdout:3377: seq_num 1144 ranges {427208=737806,1777518=513724,2966139=261069} 2021-03-15T22:34:15.470 INFO:tasks.rados.rados.0.smithi076.stdout:3377: writing smithi07694878-47 from 427208 to 1165014 tid 1 2021-03-15T22:34:15.476 INFO:tasks.rados.rados.0.smithi076.stdout:3377: writing smithi07694878-47 from 1777518 to 2291242 tid 2 2021-03-15T22:34:15.481 INFO:tasks.rados.rados.0.smithi076.stdout:3377: writing smithi07694878-47 from 2966139 to 3227208 tid 3 2021-03-15T22:34:15.482 INFO:tasks.rados.rados.0.smithi076.stdout:3378: copy_from oid 1 from oid 42 current snap is 400 2021-03-15T22:34:15.482 INFO:tasks.rados.rados.0.smithi076.stdout:3379: write (excl) oid 17 current snap is 400 2021-03-15T22:34:15.482 INFO:tasks.rados.rados.0.smithi076.stdout:3379: seq_num 1145 ranges {444015=413055,1436000=773735,2709915=534100} 2021-03-15T22:34:15.486 INFO:tasks.rados.rados.0.smithi076.stdout:3379: writing smithi07694878-17 from 444015 to 857070 tid 1 2021-03-15T22:34:15.493 INFO:tasks.rados.rados.0.smithi076.stdout:3379: writing smithi07694878-17 from 1436000 to 2209735 tid 2 2021-03-15T22:34:15.500 INFO:tasks.rados.rados.0.smithi076.stdout:3379: writing smithi07694878-17 from 2709915 to 3244015 tid 3 2021-03-15T22:34:15.501 INFO:tasks.rados.rados.0.smithi076.stdout:3380: snap_create 2021-03-15T22:34:15.502 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[56514]: audit 2021-03-15T22:34:14. 2021-03-15T22:34:15.502 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[56514]: 326239+0000 mgr.y (mgr.14144) 1023 : audit [DBG] from='client.16647 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:15.503 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[56514]: cluster 2021-03-15T22:34:14.355939+0000 mon.a (mon.0) 2283 : cluster [DBG] osdmap e899: 8 total, 8 up, 7 in 2021-03-15T22:34:15.503 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[56514]: audit 2021-03-15T22:34:14.928948+0000 mon.c (mon.1) 70 : audit [DBG] from='client.? 172.21.15.76:0/13008832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:15.503 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: audit 2021-03-15T22:34:15.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: 2021-03-15T22:34:15.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:34:15.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: :34:14.326239+0000 mgr.y (mgr.14144) 1023 : audit [DBG] from='client.16647 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:15.504 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: cluster 2021-03-15T22:34:15.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: 2021-03-15T22:34:15.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: 2021 2021-03-15T22:34:15.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: - 2021-03-15T22:34:15.505 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: 03 2021-03-15T22:34:15.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: -15T22:34:14.355939+0000 mon.a (mon. 2021-03-15T22:34:15.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: 0) 2283 : cluster [DBG] osdmap e899: 8 total, 8 up, 7 in 2021-03-15T22:34:15.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: audit 2021-03-15T22:34:14 2021-03-15T22:34:15.506 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:15 smithi076 conmon[62156]: .928948+0000 mon.c (mon.1) 70 : audit [DBG] from='client.? 172.21.15.76:0/13008832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:15.507 INFO:tasks.rados.rados.0.smithi076.stdout:3377: finishing write tid 1 to smithi07694878-47 2021-03-15T22:34:15.507 INFO:tasks.rados.rados.0.smithi076.stdout:3379: finishing write tid 1 to smithi07694878-17 2021-03-15T22:34:15.510 INFO:tasks.rados.rados.0.smithi076.stdout:3377: finishing write tid 2 to smithi07694878-47 2021-03-15T22:34:15.513 INFO:tasks.rados.rados.0.smithi076.stdout:3379: finishing write tid 2 to smithi07694878-17 2021-03-15T22:34:15.513 INFO:tasks.rados.rados.0.smithi076.stdout:3379: finishing write tid 3 to smithi07694878-17 2021-03-15T22:34:15.513 INFO:tasks.rados.rados.0.smithi076.stdout:3379: finishing write tid 4 to smithi07694878-17 2021-03-15T22:34:15.514 INFO:tasks.rados.rados.0.smithi076.stdout:3379: finishing write tid 5 to smithi07694878-17 2021-03-15T22:34:15.514 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 2623 (ObjNum 1145 snap 400 seq_num 1145) dirty exists 2021-03-15T22:34:15.514 INFO:tasks.rados.rados.0.smithi076.stdout:3379: left oid 17 (ObjNum 1145 snap 400 seq_num 1145) 2021-03-15T22:34:15.514 INFO:tasks.rados.rados.0.smithi076.stdout:3377: finishing write tid 3 to smithi07694878-47 2021-03-15T22:34:15.515 INFO:tasks.rados.rados.0.smithi076.stdout:3377: finishing write tid 4 to smithi07694878-47 2021-03-15T22:34:15.515 INFO:tasks.rados.rados.0.smithi076.stdout:3377: finishing write tid 5 to smithi07694878-47 2021-03-15T22:34:15.515 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 2406 (ObjNum 1144 snap 400 seq_num 1144) dirty exists 2021-03-15T22:34:15.515 INFO:tasks.rados.rados.0.smithi076.stdout:3377: left oid 47 (ObjNum 1144 snap 400 seq_num 1144) 2021-03-15T22:34:15.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:15.558 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:15.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:15.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:15.615 INFO:tasks.rados.rados.0.smithi076.stdout:3378: finishing copy_from to smithi07694878-1 2021-03-15T22:34:15.615 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 2000 (ObjNum 1109 snap 386 seq_num 1109) dirty exists 2021-03-15T22:34:15.615 INFO:tasks.rados.rados.0.smithi076.stdout:3378: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:34:15.660 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:15.683 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:15.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:15.684 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:15.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: audit 2021-03-15T22:34:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 2021-03-15T22:34:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 2021 2021-03-15T22:34:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: - 2021-03-15T22:34:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 03 2021-03-15T22:34:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: - 2021-03-15T22:34:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 15 2021-03-15T22:34:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: T 2021-03-15T22:34:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 22 2021-03-15T22:34:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: : 2021-03-15T22:34:15.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 34 2021-03-15T22:34:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: : 2021-03-15T22:34:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 14 2021-03-15T22:34:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: . 2021-03-15T22:34:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 326239 2021-03-15T22:34:15.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: +0000 2021-03-15T22:34:15.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 2021-03-15T22:34:15.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: mgr.y 2021-03-15T22:34:15.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: ( 2021-03-15T22:34:15.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: mgr 2021-03-15T22:34:15.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: . 2021-03-15T22:34:15.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 14144 2021-03-15T22:34:15.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: ) 2021-03-15T22:34:15.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 1023 2021-03-15T22:34:15.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: : 2021-03-15T22:34:15.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: audit 2021-03-15T22:34:15.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 2021-03-15T22:34:15.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: [DBG] 2021-03-15T22:34:15.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 2021-03-15T22:34:15.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: from='client.16647 -' entity='client.admin' cmd=[{"prefix": "osd reweight-by-utilization", "oload": 1000, "max_change": 1.0, "target": ["mon-mgr", ""]}]: dispatch 2021-03-15T22:34:15.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 2021-03-15T22:34:15.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: cluster 2021-03-15T22:34:15.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 2021-03-15T22:34:15.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 2021 2021-03-15T22:34:15.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: - 2021-03-15T22:34:15.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 03 2021-03-15T22:34:15.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: - 2021-03-15T22:34:15.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 15 2021-03-15T22:34:15.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: T 2021-03-15T22:34:15.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 22 2021-03-15T22:34:15.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: : 2021-03-15T22:34:15.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 34 2021-03-15T22:34:15.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: : 2021-03-15T22:34:15.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 14 2021-03-15T22:34:15.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: . 2021-03-15T22:34:15.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 355939 2021-03-15T22:34:15.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: +0000 2021-03-15T22:34:15.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: 2021-03-15T22:34:15.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: mon.a (mon.0) 2283 : cluster [DBG] 2021-03-15T22:34:15.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: osdmap e899: 8 total, 8 up, 7 in 2021-03-15T22:34:15.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:34:15.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: T22:34:14.928948+0000 mon.c (mon.1) 70 : audit [DBG] 2021-03-15T22:34:15.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:15 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/13008832' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:15.811 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:15.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:15.813 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:15.914 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:15.936 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:15.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:15.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:16.039 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:16.061 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:16.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:16.163 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:16.186 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:16.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:16.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:16.238 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:34:16.239 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:34:16.239 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:16.239 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "MOSDPGPush(7.8 900/895 [PushOp(7:1822d391:::smithi07694878-47:27e, version: 883'976, data_included: [2141557~144508,2840614~638534], data_size: 783042, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:1822d391:::smithi07694878-47:27e@883'976, size: 3596896, copy_subset: [440614~641741,1734722~551343,2840614~756282], clone_subset: {7:1822d391:::smithi07694878-47:254=[],7:1822d391:::smithi07694878-47:28d=[0~440614,1082355~652367,2286065~554549]}, snapset: 2f6=[]:{254=[24b,239,218],27e=[266,265],28d=[28d,285],2af=[2ac,2ab,2a7,29f,296,291],2b6=[2b5,2b4,2b3,2b1],2bc=[2bb,2b7],2c9=[2c3,2c2,2c0,2be],2dd=[2dd,2dc,2d9,2d4,2d1,2cb],2ed=[2ed,2ec,2eb,2ea,2e8,2e4,2e2,2e1,2e0,2df,2de],2f6=[2f6,2f5,2f3,2f2,2ef,2ee]}, object_exist: 0), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:3596896, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to:2141557, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-15T22:34:16.239 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:34:16.235102+0000", 2021-03-15T22:34:16.240 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 0.001135921, 2021-03-15T22:34:16.240 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001185066, 2021-03-15T22:34:16.240 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:34:16.240 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:34:16.241 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:34:16.241 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:16.241 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:34:16.241 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:16.235102+0000", 2021-03-15T22:34:16.241 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:34:16.242 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:16.242 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:16.242 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:34:16.242 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:16.235102+0000", 2021-03-15T22:34:16.243 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00039202900000000001 2021-03-15T22:34:16.243 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:16.243 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:16.243 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:34:16.243 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:16.235494+0000", 2021-03-15T22:34:16.244 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.9414000000000001e-05 2021-03-15T22:34:16.244 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:16.244 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:16.244 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:34:16.244 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:16.235513+0000", 2021-03-15T22:34:16.245 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1690000000000002e-05 2021-03-15T22:34:16.245 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:16.245 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:16.245 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:34:16.245 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:16.235555+0000", 2021-03-15T22:34:16.246 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.0489e-05 2021-03-15T22:34:16.246 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:16.246 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:16.246 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:34:16.247 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:16.235575+0000", 2021-03-15T22:34:16.247 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0537000000000004e-05 2021-03-15T22:34:16.247 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:16.247 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:16.247 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:34:16.248 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:16.235666+0000", 2021-03-15T22:34:16.248 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7570000000000001e-06 2021-03-15T22:34:16.248 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:34:16.248 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:34:16.248 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:34:16.249 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:16.235672+0000", 2021-03-15T22:34:16.249 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00056991599999999996 2021-03-15T22:34:16.249 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:16.249 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:34:16.250 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:16.250 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:34:16.250 INFO:teuthology.orchestra.run.smithi125.stdout: ], 2021-03-15T22:34:16.250 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 1 2021-03-15T22:34:16.251 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:34:16.288 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:16.314 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:16.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:16.316 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:16.416 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:16.437 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:16.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:16.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:16.506 INFO:tasks.rados.rados.0.smithi076.stdout:3377: done (3 left) 2021-03-15T22:34:16.507 INFO:tasks.rados.rados.0.smithi076.stdout:3378: done (2 left) 2021-03-15T22:34:16.507 INFO:tasks.rados.rados.0.smithi076.stdout:3379: done (1 left) 2021-03-15T22:34:16.507 INFO:tasks.rados.rados.0.smithi076.stdout:3380: done (0 left) 2021-03-15T22:34:16.507 INFO:tasks.rados.rados.0.smithi076.stdout:3381: rollback oid 33 current snap is 401 2021-03-15T22:34:16.508 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 33 to 390 2021-03-15T22:34:16.508 INFO:tasks.rados.rados.0.smithi076.stdout:3382: copy_from oid 39 from oid 45 current snap is 401 2021-03-15T22:34:16.508 INFO:tasks.rados.rados.0.smithi076.stdout:3383: read oid 36 snap 349 2021-03-15T22:34:16.508 INFO:tasks.rados.rados.0.smithi076.stdout:3383: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:34:16.509 INFO:tasks.rados.rados.0.smithi076.stdout:3384: snap_create 2021-03-15T22:34:16.513 INFO:tasks.rados.rados.0.smithi076.stdout:3381: finishing rollback tid 1 to smithi07694878-33 2021-03-15T22:34:16.513 INFO:tasks.rados.rados.0.smithi076.stdout:3381: finishing rollback tid 2 to smithi07694878-33 2021-03-15T22:34:16.514 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 2416 (ObjNum 1122 snap 389 seq_num 1122) dirty exists 2021-03-15T22:34:16.517 INFO:tasks.rados.rados.0.smithi076.stdout:3383: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:34:16.540 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:16.561 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:16.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:16.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:16.618 INFO:tasks.rados.rados.0.smithi076.stdout:3382: finishing copy_from to smithi07694878-39 2021-03-15T22:34:16.619 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2109 (ObjNum 1023 snap 341 seq_num 1023) dirty exists 2021-03-15T22:34:16.619 INFO:tasks.rados.rados.0.smithi076.stdout:3382: finishing copy_from racing read to smithi07694878-39 2021-03-15T22:34:16.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:16.688 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:16.688 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:16.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: cluster 2021-03-15T22:34: 2021-03-15T22:34:16.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: 15.201974+0000 mgr.y (mgr.14144) 1024 : cluster [DBG] pgmap v1496: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 5 remapped+peering, 139 active+clean; 653 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s rd, 2.4 MiB/s wr, 2 op/s; 6/1881 objects degraded (0.319%); 74/1881 objects misplaced (3.934%); 6.1 MiB/s, 47 objects/s recovering 2021-03-15T22:34:16.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: cluster 2021-03-15T22:34:15.281259+0000 2021-03-15T22:34:16.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: osd.1 (osd.1) 172 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:16.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: cluster 2021-03-15T22:34:16.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: 2021-03-15T22:34:15. 2021-03-15T22:34:16.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: 283243+0000 osd.1 (osd.1 2021-03-15T22:34:16.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: ) 173 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:16.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: cluster 2021-03-15T22:34: 2021-03-15T22:34:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: 15.401112+0000 mon.a (mon.0) 2284 : cluster [WRN] Health check failed: Degraded data redundancy: 6/1881 objects degraded (0.319%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:34:16.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[62156]: -03-15T22:34:15.461346+0000 mon.a (mon.0) 2285 : cluster [DBG] osdmap e900: 8 total, 8 up, 7 in 2021-03-15T22:34:16.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: cluster 2021-03-15T22:34:16.773 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: 2021-03-15T22:34 2021-03-15T22:34:16.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: :15.201974+0000 mgr.y (mgr.14144) 1024 : cluster [DBG] pgmap v1496: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 5 remapped+peering, 139 active+clean; 653 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s rd, 2.4 MiB/s wr, 2 op/s; 6/1881 objects degraded (0.319%); 74/1881 objects misplaced (3.934%); 6.1 MiB/s, 47 objects/s recovering 2021-03-15T22:34:16.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: cluster 2021-03-15T22:34:16.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: 2021-03-15T22 2021-03-15T22:34:16.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: :34:15.281259+0000 osd.1 2021-03-15T22:34:16.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: (osd.1) 172 : cluster 2021-03-15T22:34:16.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: [DBG] purged_snaps scrub starts 2021-03-15T22:34:16.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: cluster 2021-03-15T22:34:16.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: 2021-03-15T22:34:15. 2021-03-15T22:34:16.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: 283243+0000 osd.1 (osd.1) 173 : 2021-03-15T22:34:16.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:16.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: cluster 2021-03-15T22:34:15.401112+0000 mon.a (mon 2021-03-15T22:34:16.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: .0) 2284 : cluster [WRN] Health check failed: Degraded data redundancy: 6/1881 objects degraded (0.319%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:16.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: cluster 2021-03-15T22:34:16.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: 2021-03-15T22:34:15.461346+0000 mon.a (mon. 2021-03-15T22:34:16.780 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:16 smithi076 conmon[56514]: 0) 2285 : cluster [DBG] osdmap e900: 8 total, 8 up, 7 in 2021-03-15T22:34:16.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:16 smithi125 conmon[56739]: cluster 2021-03-15T22:34:15.201974+0000 mgr.y (mgr.14144 2021-03-15T22:34:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:16 smithi125 conmon[56739]: ) 1024 : cluster [DBG] pgmap v1496: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 2 active+recovering+undersized+remapped, 5 remapped+peering, 139 active+clean; 653 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 0 B/s rd, 2.4 MiB/s wr, 2 op/s; 6/1881 objects degraded (0.319%); 74/1881 objects misplaced (3.934%); 6.1 MiB/s, 47 objects/s recovering 2021-03-15T22:34:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:16 smithi125 conmon[56739]: cluster 2021-03-15T22:34:15.281259+0000 osd.1 (osd.1) 172 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:16 smithi125 conmon[56739]: cluster 2021-03-15T22:34:15.283243+0000 osd.1 (osd.1) 173 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:16 smithi125 conmon[56739]: cluster 2021-03-15T22:34:15.401112+0000 mon.a (mon.0) 2284 : cluster [WRN] Health check failed: Degraded data redundancy: 6/1881 objects degraded (0.319%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:16.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:16 smithi125 conmon[56739]: cluster 2021-03-15T22:34:15.461346+0000 mon.a (mon.0) 2285 : cluster [DBG] osdmap e900: 8 total, 8 up, 7 in 2021-03-15T22:34:16.789 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:16.812 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:16.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:16.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:16.882 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-15T22:34:16.914 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:16.936 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:16.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:16.938 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:17.039 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:17.060 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:17.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:17.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:17.163 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:17.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:34:17.183 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:34:17.188 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:17.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:17.190 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:17.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:17.312 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:17.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:17.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:17.415 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:17.438 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:17.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:17.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:17.459 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:17.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:17 smithi076 conmon[56514]: cluster 2021-03-15T22:34:16.504401+0000 mon.a (mon 2021-03-15T22:34:17.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:17 smithi076 conmon[56514]: .0) 2286 : cluster [DBG] osdmap e901: 8 total, 8 up, 7 in 2021-03-15T22:34:17.521 INFO:tasks.rados.rados.0.smithi076.stdout:3381: done (3 left) 2021-03-15T22:34:17.522 INFO:tasks.rados.rados.0.smithi076.stdout:3382: done (2 left) 2021-03-15T22:34:17.522 INFO:tasks.rados.rados.0.smithi076.stdout:3383: done (1 left) 2021-03-15T22:34:17.522 INFO:tasks.rados.rados.0.smithi076.stdout:3384: done (0 left) 2021-03-15T22:34:17.523 INFO:tasks.rados.rados.0.smithi076.stdout:3385: delete oid 32 current snap is 402 2021-03-15T22:34:17.526 INFO:tasks.rados.rados.0.smithi076.stdout:3385: done (0 left) 2021-03-15T22:34:17.527 INFO:tasks.rados.rados.0.smithi076.stdout:3386: write (excl) oid 30 current snap is 402 2021-03-15T22:34:17.527 INFO:tasks.rados.rados.0.smithi076.stdout:3386: seq_num 1147 ranges {477629=563553,1836611=777404,3048408=229221} 2021-03-15T22:34:17.532 INFO:tasks.rados.rados.0.smithi076.stdout:3386: writing smithi07694878-30 from 477629 to 1041182 tid 1 2021-03-15T22:34:17.541 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:17.542 INFO:tasks.rados.rados.0.smithi076.stdout:3386: writing smithi07694878-30 from 1836611 to 2614015 tid 2 2021-03-15T22:34:17.545 INFO:tasks.rados.rados.0.smithi076.stdout:3386: writing smithi07694878-30 from 3048408 to 3277629 tid 3 2021-03-15T22:34:17.545 INFO:tasks.rados.rados.0.smithi076.stdout:3387: copy_from oid 25 from oid 50 current snap is 402 2021-03-15T22:34:17.545 INFO:tasks.rados.rados.0.smithi076.stdout:3386: finishing write tid 1 to smithi07694878-30 2021-03-15T22:34:17.546 INFO:tasks.rados.rados.0.smithi076.stdout:3388: snap_create 2021-03-15T22:34:17.548 INFO:tasks.rados.rados.0.smithi076.stdout:3386: finishing write tid 2 to smithi07694878-30 2021-03-15T22:34:17.549 INFO:tasks.rados.rados.0.smithi076.stdout:3386: finishing write tid 3 to smithi07694878-30 2021-03-15T22:34:17.549 INFO:tasks.rados.rados.0.smithi076.stdout:3386: finishing write tid 4 to smithi07694878-30 2021-03-15T22:34:17.549 INFO:tasks.rados.rados.0.smithi076.stdout:3386: finishing write tid 5 to smithi07694878-30 2021-03-15T22:34:17.549 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 2786 (ObjNum 1147 snap 402 seq_num 1147) dirty exists 2021-03-15T22:34:17.550 INFO:tasks.rados.rados.0.smithi076.stdout:3386: left oid 30 (ObjNum 1147 snap 402 seq_num 1147) 2021-03-15T22:34:17.557 INFO:tasks.rados.rados.0.smithi076.stdout:3387: finishing copy_from to smithi07694878-25 2021-03-15T22:34:17.557 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 2399 (ObjNum 1076 snap 366 seq_num 1076) dirty exists 2021-03-15T22:34:17.557 INFO:tasks.rados.rados.0.smithi076.stdout:3387: finishing copy_from racing read to smithi07694878-25 2021-03-15T22:34:17.565 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:17.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:17.566 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:17.655 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: cluster 2021-03-15T22:34:17.655 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: 2021-03-15T22:34:17.655 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: 2021 2021-03-15T22:34:17.656 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: - 2021-03-15T22:34:17.656 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: 03 2021-03-15T22:34:17.656 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: - 2021-03-15T22:34:17.656 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: 15 2021-03-15T22:34:17.657 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: T 2021-03-15T22:34:17.657 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: 22 2021-03-15T22:34:17.657 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: : 2021-03-15T22:34:17.657 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: 34 2021-03-15T22:34:17.658 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: : 2021-03-15T22:34:17.658 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: 16.504401+0000 mon.a (mon.0) 2021-03-15T22:34:17.658 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:17 smithi125 conmon[56739]: 2286 : cluster [DBG] osdmap e901: 8 total, 8 up, 7 in 2021-03-15T22:34:17.668 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:17.692 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:17.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:17.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:17.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:17 smithi076 conmon[62156]: cluster 2021-03-15T22:34:16.504401+0000 mon.a (mon.0) 2286 : cluster [DBG] osdmap e901: 8 total, 8 up, 7 in 2021-03-15T22:34:17.795 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:17.819 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:17.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:17.820 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:17.921 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:17.946 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:17.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:17.947 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:18.023 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:34:18.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:18.095 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:18.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:18.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:18.197 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:18.221 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:18.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:18.223 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:18.324 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:18.348 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:18.349 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:18.451 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:18.475 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:18.477 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:18.528 INFO:tasks.rados.rados.0.smithi076.stdout:3386: done (2 left) 2021-03-15T22:34:18.528 INFO:tasks.rados.rados.0.smithi076.stdout:3387: done (1 left) 2021-03-15T22:34:18.529 INFO:tasks.rados.rados.0.smithi076.stdout:3388: done (0 left) 2021-03-15T22:34:18.529 INFO:tasks.rados.rados.0.smithi076.stdout:3389: delete oid 24 current snap is 403 2021-03-15T22:34:18.532 INFO:tasks.rados.rados.0.smithi076.stdout:3389: done (0 left) 2021-03-15T22:34:18.532 INFO:tasks.rados.rados.0.smithi076.stdout:3390: delete oid 3 current snap is 403 2021-03-15T22:34:18.534 INFO:tasks.rados.rados.0.smithi076.stdout:3390: done (0 left) 2021-03-15T22:34:18.534 INFO:tasks.rados.rados.0.smithi076.stdout:3391: snap_create 2021-03-15T22:34:18.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:18 smithi125 conmon[56739]: cluster 2021-03-15T22:34:17.202538+0000 mgr.y (mgr.14144 2021-03-15T22:34:18.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:18 smithi125 conmon[56739]: ) 1025 : cluster [DBG] pgmap v1499: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 145 active+clean; 656 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s rd, 4.5 MiB/s wr, 7 op/s; 6/1884 objects degraded (0.318%); 67/1884 objects misplaced (3.556%); 34 MiB/s, 110 objects/s recovering 2021-03-15T22:34:18.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:18 smithi125 conmon[56739]: cluster 2021-03-15T22:34:17.505773+0000 mon.a (mon.0) 2287 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:34:18.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:18 smithi125 conmon[56739]: cluster 2021-03-15T22:34:17.517587+0000 mon.a (mon.0) 2288 : cluster [DBG] osdmap e902: 8 total, 8 up, 7 in 2021-03-15T22:34:18.578 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:18.602 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:18.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:18.604 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:18.661 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:18.704 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:18.726 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:18.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:18.728 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:18.760 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:34:18.760 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:34:18.761 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:34:18.761 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:34:18.761 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:34:18.821 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[56514]: cluster 2021-03-15T22:34:17.202538+0000 mgr.y (mgr.14144) 1025 : cluster [DBG] pgmap v1499: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 145 active+clean; 656 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s rd, 4.5 MiB/s wr, 7 op/s; 6/1884 objects degraded (0.318%); 67/1884 objects misplaced (3.556%); 34 MiB/s, 110 objects/s recovering 2021-03-15T22:34:18.821 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[56514]: cluster 2021-03-15T22:34:17.505773+0000 mon.a (mon.0) 2287 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:34:18.821 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[56514]: cluster 2021-03-15T22:34:17.517587+0000 mon.a (mon.0) 2288 : cluster [DBG] osdmap e902: 8 total, 8 up, 7 in 2021-03-15T22:34:18.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: cluster 2021-03-15T22:34:17 2021-03-15T22:34:18.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: .202538+0000 mgr.y (mgr.14144) 1025 : cluster [DBG] pgmap v1499: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 145 active+clean; 656 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 0 B/s rd, 4.5 MiB/s wr, 7 op/s; 6/1884 objects degraded (0.318%); 67/1884 objects misplaced (3.556%); 34 MiB/s, 110 objects/s recovering 2021-03-15T22:34:18.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: cluster 2021-03-15T22:34:18.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:34:18.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: 15T22:34:17. 2021-03-15T22:34:18.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: 505773+0000 mon.a (mon.0 2021-03-15T22:34:18.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: ) 2287 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:34:18.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: 2021-03-15T22:34:18.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: cluster 2021-03-15T22:34:18.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: 2021-03-15T22:34: 2021-03-15T22:34:18.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: 17.517587+0000 mon.a (mon.0) 2288 : cluster 2021-03-15T22:34:18.824 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:18 smithi076 conmon[62156]: [DBG] osdmap e902: 8 total, 8 up, 7 in 2021-03-15T22:34:18.829 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:18.851 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:18.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:18.853 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:18.954 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:18.977 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:18.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:18.979 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:19.080 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:19.102 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:19.103 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:19.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:19.204 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:19.228 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:19.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:19.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:19.330 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:19.345 INFO:journalctl@ceph.osd.1.smithi076.stdout:Mar 15 22:34:19 smithi076 conmon[68886]: debug 2021-03-15T22:34:19.273+0000 7fae32da9700 -1 osd.1 pg_epoch: 903 pg[4.f( v 556'29 (0'0,556'29] local-lis/les=874/875 n=9 ec=58/58 lis/c=874/874 les/c/f=875/875/0 sis=874) [1,3,4] r=0 lpr=874 crt=556'29 mlcod 0'0 active+clean] scrubber pg(4.f) handle_scrub_reserve_grant: received unsolicited reservation grant from osd 4 (0x5638186f7760) 2021-03-15T22:34:19.354 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:19.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:19.355 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:19.444 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-15T22:34:19.456 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:19.508 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:19.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:19.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:19.535 INFO:tasks.rados.rados.0.smithi076.stdout:3391: done (0 left) 2021-03-15T22:34:19.535 INFO:tasks.rados.rados.0.smithi076.stdout:3392: write oid 23 current snap is 404 2021-03-15T22:34:19.535 INFO:tasks.rados.rados.0.smithi076.stdout:3392: seq_num 1150 ranges {528050=505653,1612057=524731,2614324=567403} 2021-03-15T22:34:19.537 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:34:19.540 INFO:tasks.rados.rados.0.smithi076.stdout:3392: writing smithi07694878-23 from 528050 to 1033703 tid 1 2021-03-15T22:34:19.546 INFO:tasks.rados.rados.0.smithi076.stdout:3392: writing smithi07694878-23 from 1612057 to 2136788 tid 2 2021-03-15T22:34:19.552 INFO:tasks.rados.rados.0.smithi076.stdout:3392: writing smithi07694878-23 from 2614324 to 3181727 tid 3 2021-03-15T22:34:19.553 INFO:tasks.rados.rados.0.smithi076.stdout:3393: write (excl) oid 11 current snap is 404 2021-03-15T22:34:19.553 INFO:tasks.rados.rados.0.smithi076.stdout:3393: seq_num 1151 ranges {544857=580902,1670539=668389,3041747=303110} 2021-03-15T22:34:19.558 INFO:tasks.rados.rados.0.smithi076.stdout:3393: writing smithi07694878-11 from 544857 to 1125759 tid 1 2021-03-15T22:34:19.566 INFO:tasks.rados.rados.0.smithi076.stdout:3393: writing smithi07694878-11 from 1670539 to 2338928 tid 2 2021-03-15T22:34:19.572 INFO:tasks.rados.rados.0.smithi076.stdout:3393: writing smithi07694878-11 from 3041747 to 3344857 tid 3 2021-03-15T22:34:19.572 INFO:tasks.rados.rados.0.smithi076.stdout:3394: copy_from oid 17 from oid 4 current snap is 404 2021-03-15T22:34:19.572 INFO:tasks.rados.rados.0.smithi076.stdout:3392: finishing write tid 1 to smithi07694878-23 2021-03-15T22:34:19.573 INFO:tasks.rados.rados.0.smithi076.stdout:3392: finishing write tid 2 to smithi07694878-23 2021-03-15T22:34:19.573 INFO:tasks.rados.rados.0.smithi076.stdout:3392: finishing write tid 3 to smithi07694878-23 2021-03-15T22:34:19.573 INFO:tasks.rados.rados.0.smithi076.stdout:3392: finishing write tid 4 to smithi07694878-23 2021-03-15T22:34:19.573 INFO:tasks.rados.rados.0.smithi076.stdout:3392: finishing write tid 5 to smithi07694878-23 2021-03-15T22:34:19.574 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 2410 (ObjNum 1150 snap 404 seq_num 1150) dirty exists 2021-03-15T22:34:19.574 INFO:tasks.rados.rados.0.smithi076.stdout:3392: left oid 23 (ObjNum 1150 snap 404 seq_num 1150) 2021-03-15T22:34:19.574 INFO:tasks.rados.rados.0.smithi076.stdout:3393: finishing write tid 1 to smithi07694878-11 2021-03-15T22:34:19.574 INFO:tasks.rados.rados.0.smithi076.stdout:3393: finishing write tid 2 to smithi07694878-11 2021-03-15T22:34:19.574 INFO:tasks.rados.rados.0.smithi076.stdout:3392: done (2 left) 2021-03-15T22:34:19.575 INFO:tasks.rados.rados.0.smithi076.stdout:3395: write oid 14 current snap is 404 2021-03-15T22:34:19.575 INFO:tasks.rados.rados.0.smithi076.stdout:3395: seq_num 1152 ranges {561664=656151,1729021=528400,2669170=692494} 2021-03-15T22:34:19.578 INFO:tasks.rados.rados.0.smithi076.stdout:3395: writing smithi07694878-14 from 561664 to 1217815 tid 1 2021-03-15T22:34:19.584 INFO:tasks.rados.rados.0.smithi076.stdout:3395: writing smithi07694878-14 from 1729021 to 2257421 tid 2 2021-03-15T22:34:19.591 INFO:tasks.rados.rados.0.smithi076.stdout:3395: writing smithi07694878-14 from 2669170 to 3361664 tid 3 2021-03-15T22:34:19.591 INFO:tasks.rados.rados.0.smithi076.stdout:3396: read oid 28 snap -1 2021-03-15T22:34:19.592 INFO:tasks.rados.rados.0.smithi076.stdout:3396: expect deleted 2021-03-15T22:34:19.592 INFO:tasks.rados.rados.0.smithi076.stdout:3397: rollback oid 5 current snap is 404 2021-03-15T22:34:19.592 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 321 2021-03-15T22:34:19.592 INFO:tasks.rados.rados.0.smithi076.stdout:3393: finishing write tid 3 to smithi07694878-11 2021-03-15T22:34:19.592 INFO:tasks.rados.rados.0.smithi076.stdout:3393: finishing write tid 4 to smithi07694878-11 2021-03-15T22:34:19.593 INFO:tasks.rados.rados.0.smithi076.stdout:3393: finishing write tid 5 to smithi07694878-11 2021-03-15T22:34:19.593 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 2004 (ObjNum 1151 snap 404 seq_num 1151) dirty exists 2021-03-15T22:34:19.593 INFO:tasks.rados.rados.0.smithi076.stdout:3393: left oid 11 (ObjNum 1151 snap 404 seq_num 1151) 2021-03-15T22:34:19.593 INFO:tasks.rados.rados.0.smithi076.stdout:3394: finishing copy_from to smithi07694878-17 2021-03-15T22:34:19.594 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 2000 (ObjNum 1109 snap 386 seq_num 1109) dirty exists 2021-03-15T22:34:19.594 INFO:tasks.rados.rados.0.smithi076.stdout:3395: finishing write tid 1 to smithi07694878-14 2021-03-15T22:34:19.594 INFO:tasks.rados.rados.0.smithi076.stdout:3394: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:34:19.594 INFO:tasks.rados.rados.0.smithi076.stdout:3395: finishing write tid 2 to smithi07694878-14 2021-03-15T22:34:19.594 INFO:tasks.rados.rados.0.smithi076.stdout:3393: done (4 left) 2021-03-15T22:34:19.595 INFO:tasks.rados.rados.0.smithi076.stdout:3394: done (3 left) 2021-03-15T22:34:19.595 INFO:tasks.rados.rados.0.smithi076.stdout:3398: write oid 35 current snap is 404 2021-03-15T22:34:19.595 INFO:tasks.rados.rados.0.smithi076.stdout:3398: seq_num 1153 ranges {578471=731400,1787503=672058,3096593=281878} 2021-03-15T22:34:19.598 INFO:tasks.rados.rados.0.smithi076.stdout:3398: writing smithi07694878-35 from 578471 to 1309871 tid 1 2021-03-15T22:34:19.604 INFO:tasks.rados.rados.0.smithi076.stdout:3398: writing smithi07694878-35 from 1787503 to 2459561 tid 2 2021-03-15T22:34:19.610 INFO:tasks.rados.rados.0.smithi076.stdout:3398: writing smithi07694878-35 from 3096593 to 3378471 tid 3 2021-03-15T22:34:19.610 INFO:tasks.rados.rados.0.smithi076.stdout:3399: snap_create 2021-03-15T22:34:19.610 INFO:tasks.rados.rados.0.smithi076.stdout:3397: finishing rollback tid 0 to smithi07694878-5 2021-03-15T22:34:19.611 INFO:tasks.rados.rados.0.smithi076.stdout:3397: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:34:19.611 INFO:tasks.rados.rados.0.smithi076.stdout:3397: finishing rollback tid 2 to smithi07694878-5 2021-03-15T22:34:19.611 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 1989 (ObjNum 888 snap 296 seq_num 888) dirty exists 2021-03-15T22:34:19.611 INFO:tasks.rados.rados.0.smithi076.stdout:3395: finishing write tid 3 to smithi07694878-14 2021-03-15T22:34:19.611 INFO:tasks.rados.rados.0.smithi076.stdout:3395: finishing write tid 4 to smithi07694878-14 2021-03-15T22:34:19.612 INFO:tasks.rados.rados.0.smithi076.stdout:3395: finishing write tid 5 to smithi07694878-14 2021-03-15T22:34:19.612 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 2671 (ObjNum 1152 snap 404 seq_num 1152) dirty exists 2021-03-15T22:34:19.612 INFO:tasks.rados.rados.0.smithi076.stdout:3395: left oid 14 (ObjNum 1152 snap 404 seq_num 1152) 2021-03-15T22:34:19.612 INFO:tasks.rados.rados.0.smithi076.stdout:3398: finishing write tid 1 to smithi07694878-35 2021-03-15T22:34:19.612 INFO:tasks.rados.rados.0.smithi076.stdout:3398: finishing write tid 2 to smithi07694878-35 2021-03-15T22:34:19.613 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:19.614 INFO:tasks.rados.rados.0.smithi076.stdout:3398: finishing write tid 3 to smithi07694878-35 2021-03-15T22:34:19.614 INFO:tasks.rados.rados.0.smithi076.stdout:3398: finishing write tid 4 to smithi07694878-35 2021-03-15T22:34:19.615 INFO:tasks.rados.rados.0.smithi076.stdout:3398: finishing write tid 5 to smithi07694878-35 2021-03-15T22:34:19.615 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 2316 (ObjNum 1153 snap 404 seq_num 1153) dirty exists 2021-03-15T22:34:19.615 INFO:tasks.rados.rados.0.smithi076.stdout:3398: left oid 35 (ObjNum 1153 snap 404 seq_num 1153) 2021-03-15T22:34:19.636 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:19.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:19.638 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:19.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:19 smithi076 conmon[56514]: cluster 2021-03-15T22:34:18.525300+0000 mon.a (mon.0) 2021-03-15T22:34:19.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:19 smithi076 conmon[56514]: 2289 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2021-03-15T22:34:19.677 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:19 smithi076 conmon[56514]: audit 2021-03-15T22:34:19.269351+0000 mon.a (mon.0) 2290 : audit [INF] from='client.? 172.21.15.76:0/154842786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:34:19.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:19 smithi076 conmon[62156]: cluster 2021-03-15T22:34:18.525300+0000 mon.a 2021-03-15T22:34:19.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:19 smithi076 conmon[62156]: (mon.0) 2289 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2021-03-15T22:34:19.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:19 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:34:19.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:19 smithi076 conmon[62156]: 22:34:19.269351+0000 mon.a (mon.0) 2290 : audit [INF] from='client.? 172.21.15.76:0/154842786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:34:19.738 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:19.739 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:34:19.760 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:19.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:19.761 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:19.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:19 smithi125 conmon[56739]: cluster 2021-03-15T22:34:18.525300+0000 mon.a (mon.0) 2289 : cluster [DBG] osdmap e903: 8 total, 8 up, 7 in 2021-03-15T22:34:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:19 smithi125 conmon[56739]: audit 2021-03-15T22:34:19.269351+0000 mon.a (mon.0) 2290 : audit [INF] from='client.? 172.21.15.76:0/154842786' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:34:19.862 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:19.884 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:19.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:19.885 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:19.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:20.008 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:20.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:20.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:20.110 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:20.133 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:20.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:20.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:20.235 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:20.258 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:20.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:20.260 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:20.361 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:20.370 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 7, 0, 3, 5, 6] out_osds: [2] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:34:20.370 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:34:20.370 INFO:tasks.thrashosds.thrasher:Growing pool default.rgw.meta 2021-03-15T22:34:20.371 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:34:20.382 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:20.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:20.384 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:20.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:20.510 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:20.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:20.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:20.533 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:34:20 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:34:20.545 INFO:tasks.rados.rados.0.smithi076.stdout:3395: done (4 left) 2021-03-15T22:34:20.545 INFO:tasks.rados.rados.0.smithi076.stdout:3396: done (3 left) 2021-03-15T22:34:20.545 INFO:tasks.rados.rados.0.smithi076.stdout:3397: done (2 left) 2021-03-15T22:34:20.546 INFO:tasks.rados.rados.0.smithi076.stdout:3398: done (1 left) 2021-03-15T22:34:20.546 INFO:tasks.rados.rados.0.smithi076.stdout:3399: done (0 left) 2021-03-15T22:34:20.547 INFO:tasks.rados.rados.0.smithi076.stdout:3400: delete oid 2 current snap is 405 2021-03-15T22:34:20.551 INFO:tasks.rados.rados.0.smithi076.stdout:3400: done (0 left) 2021-03-15T22:34:20.551 INFO:tasks.rados.rados.0.smithi076.stdout:3401: read oid 36 snap -1 2021-03-15T22:34:20.551 INFO:tasks.rados.rados.0.smithi076.stdout:3401: expect deleted 2021-03-15T22:34:20.552 INFO:tasks.rados.rados.0.smithi076.stdout:3402: write oid 43 current snap is 405 2021-03-15T22:34:20.552 INFO:tasks.rados.rados.0.smithi076.stdout:3402: seq_num 1155 ranges {612085=481898,1788114=675727,3035086=376999} 2021-03-15T22:34:20.557 INFO:tasks.rados.rados.0.smithi076.stdout:3402: writing smithi07694878-43 from 612085 to 1093983 tid 1 2021-03-15T22:34:20.564 INFO:tasks.rados.rados.0.smithi076.stdout:3402: writing smithi07694878-43 from 1788114 to 2463841 tid 2 2021-03-15T22:34:20.570 INFO:tasks.rados.rados.0.smithi076.stdout:3402: writing smithi07694878-43 from 3035086 to 3412085 tid 3 2021-03-15T22:34:20.571 INFO:tasks.rados.rados.0.smithi076.stdout:3403: read oid 11 snap -1 2021-03-15T22:34:20.571 INFO:tasks.rados.rados.0.smithi076.stdout:3403: expect (ObjNum 1151 snap 404 seq_num 1151) 2021-03-15T22:34:20.571 INFO:tasks.rados.rados.0.smithi076.stdout:3404: write oid 46 current snap is 405 2021-03-15T22:34:20.571 INFO:tasks.rados.rados.0.smithi076.stdout:3404: seq_num 1156 ranges {628892=673500,1962949=535738,3178862=250030} 2021-03-15T22:34:20.577 INFO:tasks.rados.rados.0.smithi076.stdout:3404: writing smithi07694878-46 from 628892 to 1302392 tid 1 2021-03-15T22:34:20.584 INFO:tasks.rados.rados.0.smithi076.stdout:3404: writing smithi07694878-46 from 1962949 to 2498687 tid 2 2021-03-15T22:34:20.590 INFO:tasks.rados.rados.0.smithi076.stdout:3404: writing smithi07694878-46 from 3178862 to 3428892 tid 3 2021-03-15T22:34:20.590 INFO:tasks.rados.rados.0.smithi076.stdout:3405: snap_create 2021-03-15T22:34:20.590 INFO:tasks.rados.rados.0.smithi076.stdout:3402: finishing write tid 1 to smithi07694878-43 2021-03-15T22:34:20.590 INFO:tasks.rados.rados.0.smithi076.stdout:3402: finishing write tid 2 to smithi07694878-43 2021-03-15T22:34:20.591 INFO:tasks.rados.rados.0.smithi076.stdout:3402: finishing write tid 3 to smithi07694878-43 2021-03-15T22:34:20.591 INFO:tasks.rados.rados.0.smithi076.stdout:3402: finishing write tid 4 to smithi07694878-43 2021-03-15T22:34:20.591 INFO:tasks.rados.rados.0.smithi076.stdout:3402: finishing write tid 5 to smithi07694878-43 2021-03-15T22:34:20.591 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 2414 (ObjNum 1155 snap 405 seq_num 1155) dirty exists 2021-03-15T22:34:20.592 INFO:tasks.rados.rados.0.smithi076.stdout:3402: left oid 43 (ObjNum 1155 snap 405 seq_num 1155) 2021-03-15T22:34:20.592 INFO:tasks.rados.rados.0.smithi076.stdout:3403: expect (ObjNum 1151 snap 404 seq_num 1151) 2021-03-15T22:34:20.612 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:20.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:20.637 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:20.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:20.718 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:20.738 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:20.749 INFO:tasks.rados.rados.0.smithi076.stdout:3404: finishing write tid 1 to smithi07694878-46 2021-03-15T22:34:20.749 INFO:tasks.rados.rados.0.smithi076.stdout:3404: finishing write tid 2 to smithi07694878-46 2021-03-15T22:34:20.750 INFO:tasks.rados.rados.0.smithi076.stdout:3404: finishing write tid 3 to smithi07694878-46 2021-03-15T22:34:20.750 INFO:tasks.rados.rados.0.smithi076.stdout:3404: finishing write tid 4 to smithi07694878-46 2021-03-15T22:34:20.750 INFO:tasks.rados.rados.0.smithi076.stdout:3404: finishing write tid 5 to smithi07694878-46 2021-03-15T22:34:20.750 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2283 (ObjNum 1156 snap 405 seq_num 1156) dirty exists 2021-03-15T22:34:20.751 INFO:tasks.rados.rados.0.smithi076.stdout:3404: left oid 46 (ObjNum 1156 snap 405 seq_num 1156) 2021-03-15T22:34:20.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[56514]: cluster 2021-03-15T22:34:19.202914+0000 mgr.y (mgr.14144) 1026 : cluster [DBG] pgmap v1502: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 145 active+clean; 656 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 2.1 MiB/s wr, 5 op/s; 6/1884 objects degraded (0.318%); 67/1884 objects misplaced (3.556%); 34 MiB/s, 90 objects/s recovering 2021-03-15T22:34:20.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[56514]: cluster 2021-03-15T22:34:19.400185+0000 osd.7 (osd.7) 170 : cluster [DBG] 5.14 deep-scrub ok 2021-03-15T22:34:20.754 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[56514]: cluster 2021-03-15T22:34:19.520437+0000 mon.a (mon.0) 2291 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:20.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[56514]: audit 2021-03-15T22:34:19.532772+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? 172.21.15.76:0/154842786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:34:20.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[56514]: cluster 2021-03-15T22:34:19.532829+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2021-03-15T22:34:20.755 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: cluster 2021-03-15T22:34:20.756 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: 2021-03-15T22:34:20.756 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: 2021-03-15T22:34: 2021-03-15T22:34:20.756 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: 19.202914+0000 mgr.y (mgr.14144) 1026 : cluster [DBG] pgmap v1502: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 145 active+clean; 656 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 2.1 MiB/s wr, 5 op/s; 6/1884 objects degraded (0.318%); 67/1884 objects misplaced (3.556%); 34 MiB/s, 90 objects/s recovering 2021-03-15T22:34:20.756 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: cluster 2021-03-15T22:34:20.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:34:20.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: 15T22:34:19.400185+0000 osd.7 (osd.7) 2021-03-15T22:34:20.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: 170 : cluster [DBG] 5.14 deep-scrub ok 2021-03-15T22:34:20.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:34:20.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: 03-15T22:34:19.520437+0000 mon.a 2021-03-15T22:34:20.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: (mon.0) 2291 : cluster [WRN] 2021-03-15T22:34:20.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:20.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:34:20.758 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: 15T22:34:19.532772+0000 mon.a ( 2021-03-15T22:34:20.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: mon.0) 2292 : audit [INF] from='client.? 172.21.15.76:0/154842786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:34:20.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:34:20.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: 03-15T22:34:19.532829+0000 mon.a 2021-03-15T22:34:20.759 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: (mon.0) 2293 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2021-03-15T22:34:20.760 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:20 smithi076 conmon[62156]: 2021-03-15T22:34:20.763 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:20.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:20.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:20.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:20 smithi125 conmon[56739]: cluster 2021-03-15T22:34:19.202914+0000 mgr.y (mgr.14144) 1026 : cluster [DBG] pgmap v1502: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 1 active+recovering+undersized+remapped, 145 active+clean; 656 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 511 B/s rd, 2.1 MiB/s wr, 5 op/s; 6/1884 objects degraded (0.318%); 67/1884 objects misplaced (3.556%); 34 MiB/s, 90 objects/s recovering 2021-03-15T22:34:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:20 smithi125 conmon[56739]: cluster 2021-03-15T22:34:19.400185+0000 osd.7 (osd.7) 170 : cluster [DBG] 5.14 deep-scrub ok 2021-03-15T22:34:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:20 smithi125 conmon[56739]: cluster 2021-03-15T22:34:19.520437+0000 mon.a (mon.0) 2291 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:20 smithi125 conmon[56739]: audit 2021-03-15T22:34:19.532772+0000 mon.a (mon.0) 2292 : audit [INF] from='client.? 172.21.15.76:0/154842786' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:34:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:20 smithi125 conmon[56739]: cluster 2021-03-15T22:34:19.532829+0000 mon.a (mon.0) 2293 : cluster [DBG] osdmap e904: 8 total, 8 up, 7 in 2021-03-15T22:34:20.865 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:20.888 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:20.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:20.889 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:20.990 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:21.014 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:21.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:21.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:21.033 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:34:21.049 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:34:21.067 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:34:21.088 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:34:21.106 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:34:21.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:21.123 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:34:21.139 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:21.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:21.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:21.142 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:34:21.160 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:34:21.161 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:34:21.161 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:34:21.161 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:34:21.162 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:34:21.162 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:34:21.162 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:34:21.162 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:34:21.163 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:34:21.241 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:21.263 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:21.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:21.265 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:21.345 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:34:21.345 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:34:21.366 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:21.388 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:21.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:21.389 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:21.490 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:21.512 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:21.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:21.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:21.548 INFO:tasks.rados.rados.0.smithi076.stdout:3401: done (4 left) 2021-03-15T22:34:21.548 INFO:tasks.rados.rados.0.smithi076.stdout:3402: done (3 left) 2021-03-15T22:34:21.548 INFO:tasks.rados.rados.0.smithi076.stdout:3403: done (2 left) 2021-03-15T22:34:21.549 INFO:tasks.rados.rados.0.smithi076.stdout:3404: done (1 left) 2021-03-15T22:34:21.549 INFO:tasks.rados.rados.0.smithi076.stdout:3405: done (0 left) 2021-03-15T22:34:21.549 INFO:tasks.rados.rados.0.smithi076.stdout:3406: rollback oid 41 current snap is 406 2021-03-15T22:34:21.549 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 41 to 403 2021-03-15T22:34:21.550 INFO:tasks.rados.rados.0.smithi076.stdout:3407: write (excl) oid 23 current snap is 406 2021-03-15T22:34:21.550 INFO:tasks.rados.rados.0.smithi076.stdout:3407: seq_num 1157 ranges {645699=748749,2021431=679396,3206285=239414} 2021-03-15T22:34:21.555 INFO:tasks.rados.rados.0.smithi076.stdout:3407: writing smithi07694878-23 from 645699 to 1394448 tid 1 2021-03-15T22:34:21.563 INFO:tasks.rados.rados.0.smithi076.stdout:3407: writing smithi07694878-23 from 2021431 to 2700827 tid 2 2021-03-15T22:34:21.568 INFO:tasks.rados.rados.0.smithi076.stdout:3407: writing smithi07694878-23 from 3206285 to 3445699 tid 3 2021-03-15T22:34:21.568 INFO:tasks.rados.rados.0.smithi076.stdout:3408: copy_from oid 8 from oid 20 current snap is 406 2021-03-15T22:34:21.568 INFO:tasks.rados.rados.0.smithi076.stdout:3406: finishing rollback tid 0 to smithi07694878-41 2021-03-15T22:34:21.568 INFO:tasks.rados.rados.0.smithi076.stdout:3406: finishing rollback tid 1 to smithi07694878-41 2021-03-15T22:34:21.569 INFO:tasks.rados.rados.0.smithi076.stdout:3406: finishing rollback tid 2 to smithi07694878-41 2021-03-15T22:34:21.569 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 2408 (ObjNum 1076 snap 366 seq_num 1076) dirty exists 2021-03-15T22:34:21.569 INFO:tasks.rados.rados.0.smithi076.stdout:3406: done (2 left) 2021-03-15T22:34:21.569 INFO:tasks.rados.rados.0.smithi076.stdout:3409: write oid 28 current snap is 406 2021-03-15T22:34:21.569 INFO:tasks.rados.rados.0.smithi076.stdout:3407: finishing write tid 1 to smithi07694878-23 2021-03-15T22:34:21.570 INFO:tasks.rados.rados.0.smithi076.stdout:3407: finishing write tid 2 to smithi07694878-23 2021-03-15T22:34:21.570 INFO:tasks.rados.rados.0.smithi076.stdout:3409: seq_num 1158 ranges {662506=423998,1563560=539407,2717355=490285} 2021-03-15T22:34:21.573 INFO:tasks.rados.rados.0.smithi076.stdout:3409: writing smithi07694878-28 from 662506 to 1086504 tid 1 2021-03-15T22:34:21.579 INFO:tasks.rados.rados.0.smithi076.stdout:3409: writing smithi07694878-28 from 1563560 to 2102967 tid 2 2021-03-15T22:34:21.585 INFO:tasks.rados.rados.0.smithi076.stdout:3409: writing smithi07694878-28 from 2717355 to 3207640 tid 3 2021-03-15T22:34:21.586 INFO:tasks.rados.rados.0.smithi076.stdout:3410: snap_remove snap 386 2021-03-15T22:34:21.615 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:21.637 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:21.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:21.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:21.739 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:21.761 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:21.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:21.763 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:21.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:21 smithi125 conmon[56739]: audit 2021-03-15T22:34:20.235464 2021-03-15T22:34:21.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:21 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 1027 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:21.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:21 smithi125 conmon[56739]: cluster 2021-03-15T22:34:20.539660+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e905: 8 total, 8 up, 7 in 2021-03-15T22:34:21.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:21 smithi125 conmon[56739]: audit 2021-03-15T22:34:21.012749+0000 mon.a (mon.0) 2295 : audit [DBG] from='client.? 172.21.15.76:0/124637011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:21.864 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:21.886 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:21.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:21.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:21.978 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[56514]: audit 2021-03-15T22:34: 2021-03-15T22:34:21.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[56514]: 20.235464+0000 mgr.y (mgr.14144) 1027 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:21.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[56514]: cluster 2021-03-15T22:34:20.539660+0000 mon.a (mon.0) 2294 : cluster [DBG] osdmap e905: 8 total, 8 up, 7 in 2021-03-15T22:34:21.979 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[56514]: audit 2021-03-15T22:34:21.012749+0000 mon.a (mon.0) 2295 : audit [DBG] from='client.? 172.21.15.76:0/124637011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:21.980 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: audit 2021-03-15T22:34:21.980 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: 2021-03-15T22:34:21.980 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: 2021 2021-03-15T22:34:21.980 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: -03-15T22:34:20.235464 2021-03-15T22:34:21.981 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144 2021-03-15T22:34:21.981 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: ) 1027 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:21.981 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:34:21.981 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: T22:34:20.539660+0000 mon.a (mon. 2021-03-15T22:34:21.981 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: 0) 2294 : cluster [DBG] osdmap e905: 8 total, 8 up, 7 in 2021-03-15T22:34:21.982 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:34:21.982 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: 03-15T22:34:21.012749+0000 mon.a 2021-03-15T22:34:21.982 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: (mon.0) 2295 : audit [DBG] from='client.? 172.21.15.76:0/124637011' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:21.982 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:21 smithi076 conmon[62156]: 2021-03-15T22:34:21.988 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:22.010 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:22.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:22.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:22.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-15T22:34:22.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:22.135 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:22.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:22.136 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:22.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:22.259 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:22.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:22.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:22.362 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:22.383 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:22.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:22.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:22.485 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:22.507 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:22.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:22.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:22.561 INFO:tasks.rados.rados.0.smithi076.stdout:3410: done (3 left) 2021-03-15T22:34:22.561 INFO:tasks.rados.rados.0.smithi076.stdout:3411: write oid 44 current snap is 406 2021-03-15T22:34:22.561 INFO:tasks.rados.rados.0.smithi076.stdout:3407: finishing write tid 3 to smithi07694878-23 2021-03-15T22:34:22.561 INFO:tasks.rados.rados.0.smithi076.stdout:3407: finishing write tid 4 to smithi07694878-23 2021-03-15T22:34:22.562 INFO:tasks.rados.rados.0.smithi076.stdout:3407: finishing write tid 5 to smithi07694878-23 2021-03-15T22:34:22.562 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 2418 (ObjNum 1157 snap 406 seq_num 1157) dirty exists 2021-03-15T22:34:22.562 INFO:tasks.rados.rados.0.smithi076.stdout:3407: left oid 23 (ObjNum 1157 snap 406 seq_num 1157) 2021-03-15T22:34:22.562 INFO:tasks.rados.rados.0.smithi076.stdout:3409: finishing write tid 1 to smithi07694878-28 2021-03-15T22:34:22.563 INFO:tasks.rados.rados.0.smithi076.stdout:3408: finishing copy_from to smithi07694878-8 2021-03-15T22:34:22.563 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 2407 (ObjNum 1108 snap 385 seq_num 1108) dirty exists 2021-03-15T22:34:22.563 INFO:tasks.rados.rados.0.smithi076.stdout:3408: finishing copy_from racing read to smithi07694878-8 2021-03-15T22:34:22.563 INFO:tasks.rados.rados.0.smithi076.stdout:3409: finishing write tid 2 to smithi07694878-28 2021-03-15T22:34:22.564 INFO:tasks.rados.rados.0.smithi076.stdout:3409: finishing write tid 3 to smithi07694878-28 2021-03-15T22:34:22.564 INFO:tasks.rados.rados.0.smithi076.stdout:3409: finishing write tid 4 to smithi07694878-28 2021-03-15T22:34:22.564 INFO:tasks.rados.rados.0.smithi076.stdout:3409: finishing write tid 5 to smithi07694878-28 2021-03-15T22:34:22.564 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2965 (ObjNum 1158 snap 406 seq_num 1158) dirty exists 2021-03-15T22:34:22.564 INFO:tasks.rados.rados.0.smithi076.stdout:3409: left oid 28 (ObjNum 1158 snap 406 seq_num 1158) 2021-03-15T22:34:22.565 INFO:tasks.rados.rados.0.smithi076.stdout:3411: seq_num 1159 ranges {679313=499247,1622042=683065,2744778=701557} 2021-03-15T22:34:22.566 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:34:22.570 INFO:tasks.rados.rados.0.smithi076.stdout:3411: writing smithi07694878-44 from 679313 to 1178560 tid 1 2021-03-15T22:34:22.578 INFO:tasks.rados.rados.0.smithi076.stdout:3411: writing smithi07694878-44 from 1622042 to 2305107 tid 2 2021-03-15T22:34:22.578 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:22.586 INFO:tasks.rados.rados.0.smithi076.stdout:3411: writing smithi07694878-44 from 2744778 to 3446335 tid 3 2021-03-15T22:34:22.586 INFO:tasks.rados.rados.0.smithi076.stdout:3407: done (3 left) 2021-03-15T22:34:22.586 INFO:tasks.rados.rados.0.smithi076.stdout:3408: done (2 left) 2021-03-15T22:34:22.587 INFO:tasks.rados.rados.0.smithi076.stdout:3409: done (1 left) 2021-03-15T22:34:22.587 INFO:tasks.rados.rados.0.smithi076.stdout:3412: copy_from oid 28 from oid 14 current snap is 406 2021-03-15T22:34:22.587 INFO:tasks.rados.rados.0.smithi076.stdout:3411: finishing write tid 1 to smithi07694878-44 2021-03-15T22:34:22.587 INFO:tasks.rados.rados.0.smithi076.stdout:3411: finishing write tid 2 to smithi07694878-44 2021-03-15T22:34:22.588 INFO:tasks.rados.rados.0.smithi076.stdout:3413: snap_create 2021-03-15T22:34:22.589 INFO:tasks.rados.rados.0.smithi076.stdout:3411: finishing write tid 3 to smithi07694878-44 2021-03-15T22:34:22.590 INFO:tasks.rados.rados.0.smithi076.stdout:3411: finishing write tid 4 to smithi07694878-44 2021-03-15T22:34:22.590 INFO:tasks.rados.rados.0.smithi076.stdout:3411: finishing write tid 5 to smithi07694878-44 2021-03-15T22:34:22.590 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 2410 (ObjNum 1159 snap 406 seq_num 1159) dirty exists 2021-03-15T22:34:22.590 INFO:tasks.rados.rados.0.smithi076.stdout:3411: left oid 44 (ObjNum 1159 snap 406 seq_num 1159) 2021-03-15T22:34:22.603 INFO:tasks.rados.rados.0.smithi076.stdout:3412: finishing copy_from to smithi07694878-28 2021-03-15T22:34:22.604 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 2671 (ObjNum 1152 snap 404 seq_num 1152) dirty exists 2021-03-15T22:34:22.604 INFO:tasks.rados.rados.0.smithi076.stdout:3412: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:34:22.608 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:22.630 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:22.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:22.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:22.732 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:22.756 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:22.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:22.757 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:22.829 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:22 smithi076 conmon[56514]: cluster 2021-03-15T22:34:21.203425+0000 mgr.y (mgr.14144) 1028 : cluster [DBG] pgmap v1505: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 671 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 8/1911 objects degraded (0.419%); 42/1911 objects misplaced (2.198%); 10 MiB/s, 23 objects/s recovering 2021-03-15T22:34:22.830 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:22 smithi076 conmon[56514]: cluster 2021-03-15T22:34:21.536090+0000 mon.a (mon.0) 2296 : cluster [WRN] Health check update: Degraded data redundancy: 8/1911 objects degraded (0.419%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:22.830 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:22 smithi076 conmon[56514]: cluster 2021-03-15T22:34:21.545014+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e906: 8 total, 8 up, 7 in 2021-03-15T22:34:22.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:22 smithi076 conmon[62156]: cluster 2021-03-15T22:34:21.203425+0000 mgr.y (mgr.14144) 1028 : cluster [DBG] pgmap v1505: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 671 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 8/1911 objects degraded (0.419%); 42/1911 objects misplaced (2.198%); 10 MiB/s, 23 objects/s recovering 2021-03-15T22:34:22.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:22 smithi076 conmon[62156]: cluster 2021-03-15T22:34:21.536090+0000 mon.a (mon.0) 2296 : cluster [WRN] Health check update: Degraded data redundancy: 8/1911 objects degraded (0.419%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:22.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:22 smithi076 conmon[62156]: cluster 2021-03-15T22:34:21.545014+0000 mon.a (mon.0) 2297 : cluster [DBG] osdmap e906: 8 total, 8 up, 7 in 2021-03-15T22:34:22.859 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:22.880 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:22.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:22.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:22.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:23.004 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:23.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:23.005 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:23.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:22 smithi125 conmon[56739]: cluster 2021-03-15T22:34:21.203425+0000 mgr.y (mgr.14144) 1028 : cluster [DBG] pgmap v1505: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 671 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 2.6 MiB/s wr, 14 op/s; 8/1911 objects degraded (0.419%); 42/1911 objects misplaced (2.198%); 10 MiB/s, 23 objects/s recovering 2021-03-15T22:34:23.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:22 smithi125 conmon[56739]: cluster 2021-03-15T22:34:21.536090+0000 mon.a (mon.0) 2296 : cluster [WRN] Health check update: Degraded data redundancy: 8/1911 objects degraded (0.419%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:23.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:22 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:34:23.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:22 smithi125 conmon[56739]: 03-15T22:34:21.545014 2021-03-15T22:34:23.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:22 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2297 2021-03-15T22:34:23.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:22 smithi125 conmon[56739]: : cluster [DBG] osdmap e906: 8 total, 8 up, 7 in 2021-03-15T22:34:23.106 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:23.125 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:23.132 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:23.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:23.133 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:23.234 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:23.256 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:23.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:23.257 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:23.358 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:23.382 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:23.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:23.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:23.484 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:23.506 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:23.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:23.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:23.566 INFO:tasks.rados.rados.0.smithi076.stdout:3411: done (2 left) 2021-03-15T22:34:23.567 INFO:tasks.rados.rados.0.smithi076.stdout:3412: done (1 left) 2021-03-15T22:34:23.567 INFO:tasks.rados.rados.0.smithi076.stdout:3413: done (0 left) 2021-03-15T22:34:23.567 INFO:tasks.rados.rados.0.smithi076.stdout:3414: write (excl) oid 12 current snap is 407 2021-03-15T22:34:23.567 INFO:tasks.rados.rados.0.smithi076.stdout:3414: seq_num 1160 ranges {696120=574496,1680524=543076,2888554=512829} 2021-03-15T22:34:23.573 INFO:tasks.rados.rados.0.smithi076.stdout:3414: writing smithi07694878-12 from 696120 to 1270616 tid 1 2021-03-15T22:34:23.578 INFO:tasks.rados.rados.0.smithi076.stdout:3414: writing smithi07694878-12 from 1680524 to 2223600 tid 2 2021-03-15T22:34:23.585 INFO:tasks.rados.rados.0.smithi076.stdout:3414: writing smithi07694878-12 from 2888554 to 3401383 tid 3 2021-03-15T22:34:23.586 INFO:tasks.rados.rados.0.smithi076.stdout:3415: copy_from oid 5 from oid 1 current snap is 407 2021-03-15T22:34:23.586 INFO:tasks.rados.rados.0.smithi076.stdout:3414: finishing write tid 1 to smithi07694878-12 2021-03-15T22:34:23.586 INFO:tasks.rados.rados.0.smithi076.stdout:3414: finishing write tid 2 to smithi07694878-12 2021-03-15T22:34:23.586 INFO:tasks.rados.rados.0.smithi076.stdout:3416: snap_remove snap 396 2021-03-15T22:34:23.609 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:23.631 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:23.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:23.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:23.652 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:34:23.655 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:34:23.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:23.755 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:23.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:23.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:23.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:23 smithi076 conmon[56514]: cluster 2021-03-15T22:34:22 2021-03-15T22:34:23.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:23 smithi076 conmon[56514]: .554978+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2021-03-15T22:34:23.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:23 smithi076 conmon[62156]: cluster 2021-03-15T22:34:23.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:23 smithi076 conmon[62156]: 2021-03-15T22:34:23.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:23 smithi076 conmon[62156]: 2021 2021-03-15T22:34:23.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:23 smithi076 conmon[62156]: - 2021-03-15T22:34:23.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:23 smithi076 conmon[62156]: 03 2021-03-15T22:34:23.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:23 smithi076 conmon[62156]: - 2021-03-15T22:34:23.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:23 smithi076 conmon[62156]: 15 2021-03-15T22:34:23.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:23 smithi076 conmon[62156]: T22:34:22. 2021-03-15T22:34:23.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:23 smithi076 conmon[62156]: 554978+0000 mon.a (mon.0) 2298 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2021-03-15T22:34:23.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:23.880 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:23.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:23.881 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:23.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:24.002 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:34:24.004 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:24.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:24.005 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:24.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:23 smithi125 conmon[56739]: cluster 2021-03-15T22:34:22.554978+0000 2021-03-15T22:34:24.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:23 smithi125 conmon[56739]: mon.a (mon.0) 2298 : cluster [DBG] osdmap e907: 8 total, 8 up, 7 in 2021-03-15T22:34:24.106 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:24.128 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:24.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:24.130 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:24.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:24.253 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:24.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:24.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:24.355 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:24.378 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:24.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:24.380 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:24.481 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:24.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:24.504 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:24.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:24.569 INFO:tasks.rados.rados.0.smithi076.stdout:3416: done (2 left) 2021-03-15T22:34:24.569 INFO:tasks.rados.rados.0.smithi076.stdout:3417: write oid 3 current snap is 407 2021-03-15T22:34:24.569 INFO:tasks.rados.rados.0.smithi076.stdout:3417: seq_num 1161 ranges {712927=649745,2139006=686734} 2021-03-15T22:34:24.575 INFO:tasks.rados.rados.0.smithi076.stdout:3417: writing smithi07694878-3 from 712927 to 1362672 tid 1 2021-03-15T22:34:24.586 INFO:tasks.rados.rados.0.smithi076.stdout:3417: writing smithi07694878-3 from 2139006 to 2825740 tid 2 2021-03-15T22:34:24.586 INFO:tasks.rados.rados.0.smithi076.stdout:3418: read oid 26 snap -1 2021-03-15T22:34:24.586 INFO:tasks.rados.rados.0.smithi076.stdout:3418: expect deleted 2021-03-15T22:34:24.586 INFO:tasks.rados.rados.0.smithi076.stdout:3419: rollback oid 4 current snap is 407 2021-03-15T22:34:24.587 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 4 to 406 2021-03-15T22:34:24.587 INFO:tasks.rados.rados.0.smithi076.stdout:3414: finishing write tid 3 to smithi07694878-12 2021-03-15T22:34:24.587 INFO:tasks.rados.rados.0.smithi076.stdout:3414: finishing write tid 4 to smithi07694878-12 2021-03-15T22:34:24.588 INFO:tasks.rados.rados.0.smithi076.stdout:3414: finishing write tid 5 to smithi07694878-12 2021-03-15T22:34:24.588 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 2790 (ObjNum 1160 snap 407 seq_num 1160) dirty exists 2021-03-15T22:34:24.589 INFO:tasks.rados.rados.0.smithi076.stdout:3414: left oid 12 (ObjNum 1160 snap 407 seq_num 1160) 2021-03-15T22:34:24.589 INFO:tasks.rados.rados.0.smithi076.stdout:3415: finishing copy_from to smithi07694878-5 2021-03-15T22:34:24.590 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 2000 (ObjNum 1109 snap 386 seq_num 1109) dirty exists 2021-03-15T22:34:24.590 INFO:tasks.rados.rados.0.smithi076.stdout:3415: finishing copy_from racing read to smithi07694878-5 2021-03-15T22:34:24.590 INFO:tasks.rados.rados.0.smithi076.stdout:3417: finishing write tid 1 to smithi07694878-3 2021-03-15T22:34:24.591 INFO:tasks.rados.rados.0.smithi076.stdout:3414: done (4 left) 2021-03-15T22:34:24.591 INFO:tasks.rados.rados.0.smithi076.stdout:3415: done (3 left) 2021-03-15T22:34:24.592 INFO:tasks.rados.rados.0.smithi076.stdout:3420: rollback oid 44 current snap is 407 2021-03-15T22:34:24.592 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 44 to 384 2021-03-15T22:34:24.592 INFO:tasks.rados.rados.0.smithi076.stdout:3421: read oid 34 snap -1 2021-03-15T22:34:24.593 INFO:tasks.rados.rados.0.smithi076.stdout:3421: expect deleted 2021-03-15T22:34:24.593 INFO:tasks.rados.rados.0.smithi076.stdout:3422: read oid 42 snap -1 2021-03-15T22:34:24.593 INFO:tasks.rados.rados.0.smithi076.stdout:3422: expect (ObjNum 1109 snap 386 seq_num 1109) 2021-03-15T22:34:24.594 INFO:tasks.rados.rados.0.smithi076.stdout:3423: rollback oid 19 current snap is 407 2021-03-15T22:34:24.594 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 19 to 354 2021-03-15T22:34:24.594 INFO:tasks.rados.rados.0.smithi076.stdout:3424: snap_create 2021-03-15T22:34:24.596 INFO:tasks.rados.rados.0.smithi076.stdout:3417: finishing write tid 2 to smithi07694878-3 2021-03-15T22:34:24.597 INFO:tasks.rados.rados.0.smithi076.stdout:3419: finishing rollback tid 0 to smithi07694878-4 2021-03-15T22:34:24.597 INFO:tasks.rados.rados.0.smithi076.stdout:3420: finishing rollback tid 0 to smithi07694878-44 2021-03-15T22:34:24.597 INFO:tasks.rados.rados.0.smithi076.stdout:3420: finishing rollback tid 1 to smithi07694878-44 2021-03-15T22:34:24.597 INFO:tasks.rados.rados.0.smithi076.stdout:3420: finishing rollback tid 2 to smithi07694878-44 2021-03-15T22:34:24.598 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 2413 (ObjNum 829 snap 275 seq_num 829) dirty exists 2021-03-15T22:34:24.598 INFO:tasks.rados.rados.0.smithi076.stdout:3419: finishing rollback tid 1 to smithi07694878-4 2021-03-15T22:34:24.598 INFO:tasks.rados.rados.0.smithi076.stdout:3419: finishing rollback tid 2 to smithi07694878-4 2021-03-15T22:34:24.598 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 2007 (ObjNum 1109 snap 386 seq_num 1109) dirty exists 2021-03-15T22:34:24.599 INFO:tasks.rados.rados.0.smithi076.stdout:3417: finishing write tid 3 to smithi07694878-3 2021-03-15T22:34:24.599 INFO:tasks.rados.rados.0.smithi076.stdout:3417: finishing write tid 4 to smithi07694878-3 2021-03-15T22:34:24.599 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 2674 (ObjNum 1161 snap 407 seq_num 1161) dirty exists 2021-03-15T22:34:24.599 INFO:tasks.rados.rados.0.smithi076.stdout:3417: left oid 3 (ObjNum 1161 snap 407 seq_num 1161) 2021-03-15T22:34:24.600 INFO:tasks.rados.rados.0.smithi076.stdout:3423: finishing rollback tid 0 to smithi07694878-19 2021-03-15T22:34:24.600 INFO:tasks.rados.rados.0.smithi076.stdout:3423: finishing rollback tid 1 to smithi07694878-19 2021-03-15T22:34:24.600 INFO:tasks.rados.rados.0.smithi076.stdout:3423: finishing rollback tid 2 to smithi07694878-19 2021-03-15T22:34:24.600 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2319 (ObjNum 1006 snap 339 seq_num 1006) dirty exists 2021-03-15T22:34:24.601 INFO:tasks.rados.rados.0.smithi076.stdout:3422: expect (ObjNum 1109 snap 386 seq_num 1109) 2021-03-15T22:34:24.605 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:24.627 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:24.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:24.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:24.729 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:24.750 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:24.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:24.752 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:24.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:24 smithi076 conmon[56514]: cluster 2021-03-15T22:34:23.203868+0000 mgr.y (mgr.14144) 1029 : cluster [DBG] pgmap v1508: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 684 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 5.2 MiB/s wr, 23 op/s; 8/1929 objects degraded (0.415%); 42/1929 objects misplaced (2.177%); 10 MiB/s, 23 objects/s recovering 2021-03-15T22:34:24.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:24 smithi076 conmon[56514]: cluster 2021-03-15T22:34:23.561748+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2021-03-15T22:34:24.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:24 smithi076 conmon[56514]: 2021-03-15T22:34:24.805 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:24 smithi076 conmon[56514]: audit 2021-03-15T22:34:23.631400+0000 mgr.y (mgr.14144) 1030 : audit [DBG] from='client.16671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:34:24.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:24 smithi076 conmon[62156]: cluster 2021-03-15T22:34:23.203868 2021-03-15T22:34:24.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:24 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 1029 : cluster [DBG] pgmap v1508: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 684 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 5.2 MiB/s wr, 23 op/s; 8/1929 objects degraded (0.415%); 42/1929 objects misplaced (2.177%); 10 MiB/s, 23 objects/s recovering 2021-03-15T22:34:24.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:24 smithi076 conmon[62156]: cluster 2021-03-15T22:34:23.561748+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2021-03-15T22:34:24.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:24 smithi076 conmon[62156]: audit 2021-03-15T22:34:23.631400+0000 mgr.y (mgr.14144) 1030 : audit [DBG] from='client.16671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:34:24.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:24.875 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:24.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:24.876 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:24.888 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1509,"stamp":"2021-03-15T22:34:23.558810+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":717420398,"num_objects":643,"num_object_clones":279,"num_object_copies":1929,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":42,"num_objects_unfound":0,"num_objects_dirty":643,"num_whiteouts":13,"num_read":8770,"num_read_kb":6460598,"num_write":5473,"num_write_kb":1776403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4770,"num_bytes_recovered":1723160565,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":20440,"ondisk_log_size":20440,"up":444,"acting":443,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":447,"num_osds":7,"num_per_pool_osds":7,"num_per_pool_omap_osds":7,"kb":656216064,"kb_used":1954704,"kb_used_data":1616920,"kb_used_omap":153,"kb_used_meta":337574,"kb_avail":654261360,"statfs":{"total":671965249536,"available":669963632640,"internally_reserved":0,"allocated":1655726080,"data_stored":2261333411,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":345676305},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":25270753,"num_objects":11,"num_object_clones":11,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":-1,"num_read":30,"num_read_kb":19770,"num_write":47,"num_write_kb":17269,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.028882"},"pg_stats":[{"pgid":"5.19","version":"317'7","reported_seq":"266","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.556900+0000","last_change":"2021-03-15T22:34:17.260472+0000","last_active":"2021-03-15T22:34:21.556900+0000","last_peered":"2021-03-15T22:34:21.556900+0000","last_clean":"2021-03-15T22:34:21.556900+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:34:21.556900+0000","last_undegraded":"2021-03-15T22:34:21.556900+0000","last_fullsized":"2021-03-15T22:34:21.556900+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:32:07.910427+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:32:07.910427+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"921","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.537749+0000","last_change":"2021-03-15T22:34:10.331764+0000","last_active":"2021-03-15T22:34:20.537749+0000","last_peered":"2021-03-15T22:34:20.537749+0000","last_clean":"2021-03-15T22:34:20.537749+0000","last_became_active":"2021-03-15T22:34:10.331424+0000","last_became_peered":"2021-03-15T22:34:10.331424+0000","last_unstale":"2021-03-15T22:34:20.537749+0000","last_undegraded":"2021-03-15T22:34:20.537749+0000","last_fullsized":"2021-03-15T22:34:20.537749+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.564464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.564464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"419","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560671+0000","last_change":"2021-03-15T22:33:45.335440+0000","last_active":"2021-03-15T22:34:19.560671+0000","last_peered":"2021-03-15T22:34:19.560671+0000","last_clean":"2021-03-15T22:34:19.560671+0000","last_became_active":"2021-03-15T22:33:45.318577+0000","last_became_peered":"2021-03-15T22:33:45.318577+0000","last_unstale":"2021-03-15T22:34:19.560671+0000","last_undegraded":"2021-03-15T22:34:19.560671+0000","last_fullsized":"2021-03-15T22:34:19.560671+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:32:31.680111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.680111+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1746,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"999","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.537654+0000","last_change":"2021-03-15T22:33:06.085397+0000","last_active":"2021-03-15T22:34:20.537654+0000","last_peered":"2021-03-15T22:34:20.537654+0000","last_clean":"2021-03-15T22:34:20.537654+0000","last_became_active":"2021-03-15T22:33:04.345393+0000","last_became_peered":"2021-03-15T22:33:04.345393+0000","last_unstale":"2021-03-15T22:34:20.537654+0000","last_undegraded":"2021-03-15T22:34:20.537654+0000","last_fullsized":"2021-03-15T22:34:20.537654+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.177019+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"208","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.537471+0000","last_change":"2021-03-15T22:33:04.345933+0000","last_active":"2021-03-15T22:34:20.537471+0000","last_peered":"2021-03-15T22:34:20.537471+0000","last_clean":"2021-03-15T22:34:20.537471+0000","last_became_active":"2021-03-15T22:33:04.345518+0000","last_became_peered":"2021-03-15T22:33:04.345518+0000","last_unstale":"2021-03-15T22:34:20.537471+0000","last_undegraded":"2021-03-15T22:34:20.537471+0000","last_fullsized":"2021-03-15T22:34:20.537471+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.338100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.338100+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"309","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.517572+0000","last_change":"2021-03-15T22:34:10.332423+0000","last_active":"2021-03-15T22:34:17.517572+0000","last_peered":"2021-03-15T22:34:17.517572+0000","last_clean":"2021-03-15T22:34:17.517572+0000","last_became_active":"2021-03-15T22:34:10.332054+0000","last_became_peered":"2021-03-15T22:34:10.332054+0000","last_unstale":"2021-03-15T22:34:17.517572+0000","last_undegraded":"2021-03-15T22:34:17.517572+0000","last_fullsized":"2021-03-15T22:34:17.517572+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:08.652741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.652741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"625","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.546194+0000","last_change":"2021-03-15T22:34:10.349707+0000","last_active":"2021-03-15T22:34:19.546194+0000","last_peered":"2021-03-15T22:34:19.546194+0000","last_clean":"2021-03-15T22:34:19.546194+0000","last_became_active":"2021-03-15T22:34:10.349377+0000","last_became_peered":"2021-03-15T22:34:10.349377+0000","last_unstale":"2021-03-15T22:34:19.546194+0000","last_undegraded":"2021-03-15T22:34:19.546194+0000","last_fullsized":"2021-03-15T22:34:19.546194+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:58.617803+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:58.617803+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"546","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.559676+0000","last_change":"2021-03-15T22:34:10.372328+0000","last_active":"2021-03-15T22:34:19.559676+0000","last_peered":"2021-03-15T22:34:19.559676+0000","last_clean":"2021-03-15T22:34:19.559676+0000","last_became_active":"2021-03-15T22:34:10.333633+0000","last_became_peered":"2021-03-15T22:34:10.333633+0000","last_unstale":"2021-03-15T22:34:19.559676+0000","last_undegraded":"2021-03-15T22:34:19.559676+0000","last_fullsized":"2021-03-15T22:34:19.559676+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.652898+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":60,"num_bytes_recovered":3560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"746","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523452+0000","last_change":"2021-03-15T22:34:03.570696+0000","last_active":"2021-03-15T22:34:17.523452+0000","last_peered":"2021-03-15T22:34:17.523452+0000","last_clean":"2021-03-15T22:34:17.523452+0000","last_became_active":"2021-03-15T22:34:03.570306+0000","last_became_peered":"2021-03-15T22:34:03.570306+0000","last_unstale":"2021-03-15T22:34:17.523452+0000","last_undegraded":"2021-03-15T22:34:17.523452+0000","last_fullsized":"2021-03-15T22:34:17.523452+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.794113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.794113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"369","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.559922+0000","last_change":"2021-03-15T22:32:01.452949+0000","last_active":"2021-03-15T22:34:19.559922+0000","last_peered":"2021-03-15T22:34:19.559922+0000","last_clean":"2021-03-15T22:34:19.559922+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:34:19.559922+0000","last_undegraded":"2021-03-15T22:34:19.559922+0000","last_fullsized":"2021-03-15T22:34:19.559922+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:01.452885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:32:01.452885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"296","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523422+0000","last_change":"2021-03-15T22:33:39.756677+0000","last_active":"2021-03-15T22:34:17.523422+0000","last_peered":"2021-03-15T22:34:17.523422+0000","last_clean":"2021-03-15T22:34:17.523422+0000","last_became_active":"2021-03-15T22:33:04.325611+0000","last_became_peered":"2021-03-15T22:33:04.325611+0000","last_unstale":"2021-03-15T22:34:17.523422+0000","last_undegraded":"2021-03-15T22:34:17.523422+0000","last_fullsized":"2021-03-15T22:34:17.523422+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.756581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"1055","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.537915+0000","last_change":"2021-03-15T22:33:17.502781+0000","last_active":"2021-03-15T22:34:20.537915+0000","last_peered":"2021-03-15T22:34:20.537915+0000","last_clean":"2021-03-15T22:34:20.537915+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:34:20.537915+0000","last_undegraded":"2021-03-15T22:34:20.537915+0000","last_fullsized":"2021-03-15T22:34:20.537915+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:33:17.502703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.502703+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"183","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.517449+0000","last_change":"2021-03-15T22:32:38.476370+0000","last_active":"2021-03-15T22:34:17.517449+0000","last_peered":"2021-03-15T22:34:17.517449+0000","last_clean":"2021-03-15T22:34:17.517449+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:34:17.517449+0000","last_undegraded":"2021-03-15T22:34:17.517449+0000","last_fullsized":"2021-03-15T22:34:17.517449+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.476269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.476269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"532","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.522961+0000","last_change":"2021-03-15T22:34:10.348386+0000","last_active":"2021-03-15T22:34:17.522961+0000","last_peered":"2021-03-15T22:34:17.522961+0000","last_clean":"2021-03-15T22:34:17.522961+0000","last_became_active":"2021-03-15T22:34:10.348126+0000","last_became_peered":"2021-03-15T22:34:10.348126+0000","last_unstale":"2021-03-15T22:34:17.522961+0000","last_undegraded":"2021-03-15T22:34:17.522961+0000","last_fullsized":"2021-03-15T22:34:17.522961+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.774079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.774079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"278","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523323+0000","last_change":"2021-03-15T22:32:14.756556+0000","last_active":"2021-03-15T22:34:17.523323+0000","last_peered":"2021-03-15T22:34:17.523323+0000","last_clean":"2021-03-15T22:34:17.523323+0000","last_became_active":"2021-03-15T22:29:51.234868+0000","last_became_peered":"2021-03-15T22:29:51.234868+0000","last_unstale":"2021-03-15T22:34:17.523323+0000","last_undegraded":"2021-03-15T22:34:17.523323+0000","last_fullsized":"2021-03-15T22:34:17.523323+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.756460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.756460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"1027","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538276+0000","last_change":"2021-03-15T22:34:10.338165+0000","last_active":"2021-03-15T22:34:20.538276+0000","last_peered":"2021-03-15T22:34:20.538276+0000","last_clean":"2021-03-15T22:34:20.538276+0000","last_became_active":"2021-03-15T22:34:10.326323+0000","last_became_peered":"2021-03-15T22:34:10.326323+0000","last_unstale":"2021-03-15T22:34:20.538276+0000","last_undegraded":"2021-03-15T22:34:20.538276+0000","last_fullsized":"2021-03-15T22:34:20.538276+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:32:05.111031+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.111031+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"861","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.557136+0000","last_change":"2021-03-15T22:32:32.744807+0000","last_active":"2021-03-15T22:34:21.557136+0000","last_peered":"2021-03-15T22:34:21.557136+0000","last_clean":"2021-03-15T22:34:21.557136+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:34:21.557136+0000","last_undegraded":"2021-03-15T22:34:21.557136+0000","last_fullsized":"2021-03-15T22:34:21.557136+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:32:32.744716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.744716+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"923","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538325+0000","last_change":"2021-03-15T22:33:04.329427+0000","last_active":"2021-03-15T22:34:20.538325+0000","last_peered":"2021-03-15T22:34:20.538325+0000","last_clean":"2021-03-15T22:34:20.538325+0000","last_became_active":"2021-03-15T22:33:04.329078+0000","last_became_peered":"2021-03-15T22:33:04.329078+0000","last_unstale":"2021-03-15T22:34:20.538325+0000","last_undegraded":"2021-03-15T22:34:20.538325+0000","last_fullsized":"2021-03-15T22:34:20.538325+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:31:37.466443+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"373","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560186+0000","last_change":"2021-03-15T22:32:41.673856+0000","last_active":"2021-03-15T22:34:19.560186+0000","last_peered":"2021-03-15T22:34:19.560186+0000","last_clean":"2021-03-15T22:34:19.560186+0000","last_became_active":"2021-03-15T22:28:12.519619+0000","last_became_peered":"2021-03-15T22:28:12.519619+0000","last_unstale":"2021-03-15T22:34:19.560186+0000","last_undegraded":"2021-03-15T22:34:19.560186+0000","last_fullsized":"2021-03-15T22:34:19.560186+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:32:41.673792+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.673792+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"410","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.556745+0000","last_change":"2021-03-15T22:34:13.226382+0000","last_active":"2021-03-15T22:34:21.556745+0000","last_peered":"2021-03-15T22:34:21.556745+0000","last_clean":"2021-03-15T22:34:21.556745+0000","last_became_active":"2021-03-15T22:34:10.342146+0000","last_became_peered":"2021-03-15T22:34:10.342146+0000","last_unstale":"2021-03-15T22:34:21.556745+0000","last_undegraded":"2021-03-15T22:34:21.556745+0000","last_fullsized":"2021-03-15T22:34:21.556745+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:32:10.001234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.001234+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,4],"acting":[1,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"943","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538504+0000","last_change":"2021-03-15T22:33:45.324534+0000","last_active":"2021-03-15T22:34:20.538504+0000","last_peered":"2021-03-15T22:34:20.538504+0000","last_clean":"2021-03-15T22:34:20.538504+0000","last_became_active":"2021-03-15T22:33:45.323935+0000","last_became_peered":"2021-03-15T22:33:45.323935+0000","last_unstale":"2021-03-15T22:34:20.538504+0000","last_undegraded":"2021-03-15T22:34:20.538504+0000","last_fullsized":"2021-03-15T22:34:20.538504+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.144132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.144132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"739","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.403122+0000","last_change":"2021-03-15T22:33:04.337162+0000","last_active":"2021-03-15T22:34:21.403122+0000","last_peered":"2021-03-15T22:34:21.403122+0000","last_clean":"2021-03-15T22:34:21.403122+0000","last_became_active":"2021-03-15T22:33:04.336804+0000","last_became_peered":"2021-03-15T22:33:04.336804+0000","last_unstale":"2021-03-15T22:34:21.403122+0000","last_undegraded":"2021-03-15T22:34:21.403122+0000","last_fullsized":"2021-03-15T22:34:21.403122+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.594717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:45.403769+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.594717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"909","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.537913+0000","last_change":"2021-03-15T22:33:45.318571+0000","last_active":"2021-03-15T22:34:20.537913+0000","last_peered":"2021-03-15T22:34:20.537913+0000","last_clean":"2021-03-15T22:34:20.537913+0000","last_became_active":"2021-03-15T22:33:45.318115+0000","last_became_peered":"2021-03-15T22:33:45.318115+0000","last_unstale":"2021-03-15T22:34:20.537913+0000","last_undegraded":"2021-03-15T22:34:20.537913+0000","last_fullsized":"2021-03-15T22:34:20.537913+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.429029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.429029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,0],"acting":[4,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"355","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402585+0000","last_change":"2021-03-15T22:33:15.617975+0000","last_active":"2021-03-15T22:34:21.402585+0000","last_peered":"2021-03-15T22:34:21.402585+0000","last_clean":"2021-03-15T22:34:21.402585+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:34:21.402585+0000","last_undegraded":"2021-03-15T22:34:21.402585+0000","last_fullsized":"2021-03-15T22:34:21.402585+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:33:15.617891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.617891+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"891'22","reported_seq":"451","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.546950+0000","last_change":"2021-03-15T22:34:06.311981+0000","last_active":"2021-03-15T22:34:19.546950+0000","last_peered":"2021-03-15T22:34:19.546950+0000","last_clean":"2021-03-15T22:34:19.546950+0000","last_became_active":"2021-03-15T22:34:06.311478+0000","last_became_peered":"2021-03-15T22:34:06.311478+0000","last_unstale":"2021-03-15T22:34:19.546950+0000","last_undegraded":"2021-03-15T22:34:19.546950+0000","last_fullsized":"2021-03-15T22:34:19.546950+0000","mapping_epoch":890,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"841'19","last_scrub_stamp":"2021-03-15T22:33:20.272520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.272520+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"920","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.519108+0000","last_change":"2021-03-15T22:33:45.324672+0000","last_active":"2021-03-15T22:34:17.519108+0000","last_peered":"2021-03-15T22:34:17.519108+0000","last_clean":"2021-03-15T22:34:17.519108+0000","last_became_active":"2021-03-15T22:33:45.323105+0000","last_became_peered":"2021-03-15T22:33:45.323105+0000","last_unstale":"2021-03-15T22:34:17.519108+0000","last_undegraded":"2021-03-15T22:34:17.519108+0000","last_fullsized":"2021-03-15T22:34:17.519108+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:37.525692+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:37.525692+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"934","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.556797+0000","last_change":"2021-03-15T22:32:43.703085+0000","last_active":"2021-03-15T22:34:21.556797+0000","last_peered":"2021-03-15T22:34:21.556797+0000","last_clean":"2021-03-15T22:34:21.556797+0000","last_became_active":"2021-03-15T22:28:59.264559+0000","last_became_peered":"2021-03-15T22:28:59.264559+0000","last_unstale":"2021-03-15T22:34:21.556797+0000","last_undegraded":"2021-03-15T22:34:21.556797+0000","last_fullsized":"2021-03-15T22:34:21.556797+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:43.703019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.703019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"507","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560562+0000","last_change":"2021-03-15T22:34:10.385041+0000","last_active":"2021-03-15T22:34:19.560562+0000","last_peered":"2021-03-15T22:34:19.560562+0000","last_clean":"2021-03-15T22:34:19.560562+0000","last_became_active":"2021-03-15T22:34:10.347227+0000","last_became_peered":"2021-03-15T22:34:10.347227+0000","last_unstale":"2021-03-15T22:34:19.560562+0000","last_undegraded":"2021-03-15T22:34:19.560562+0000","last_fullsized":"2021-03-15T22:34:19.560562+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:34:07.125155+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.125155+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":3736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"945","reported_epoch":"904","state":"active+remapped+backfill_toofull","last_fresh":"2021-03-15T22:34:19.560331+0000","last_change":"2021-03-15T22:34:11.339947+0000","last_active":"2021-03-15T22:34:19.560331+0000","last_peered":"2021-03-15T22:34:19.560331+0000","last_clean":"2021-03-15T22:34:08.322665+0000","last_became_active":"2021-03-15T22:34:11.325468+0000","last_became_peered":"2021-03-15T22:34:11.325468+0000","last_unstale":"2021-03-15T22:34:19.560331+0000","last_undegraded":"2021-03-15T22:34:19.560331+0000","last_fullsized":"2021-03-15T22:34:19.560331+0000","mapping_epoch":895,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:32:34.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.739637+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,1],"avail_no_missing":["0","1","3","7"],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"912","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.557309+0000","last_change":"2021-03-15T22:33:11.491281+0000","last_active":"2021-03-15T22:34:21.557309+0000","last_peered":"2021-03-15T22:34:21.557309+0000","last_clean":"2021-03-15T22:34:21.557309+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:34:21.557309+0000","last_undegraded":"2021-03-15T22:34:21.557309+0000","last_fullsized":"2021-03-15T22:34:21.557309+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:11.491199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:11.491199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"518","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402432+0000","last_change":"2021-03-15T22:34:10.325480+0000","last_active":"2021-03-15T22:34:21.402432+0000","last_peered":"2021-03-15T22:34:21.402432+0000","last_clean":"2021-03-15T22:34:21.402432+0000","last_became_active":"2021-03-15T22:34:10.325232+0000","last_became_peered":"2021-03-15T22:34:10.325232+0000","last_unstale":"2021-03-15T22:34:21.402432+0000","last_undegraded":"2021-03-15T22:34:21.402432+0000","last_fullsized":"2021-03-15T22:34:21.402432+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:10.332319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.332319+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"108","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.546716+0000","last_change":"2021-03-15T22:33:14.283830+0000","last_active":"2021-03-15T22:34:19.546716+0000","last_peered":"2021-03-15T22:34:19.546716+0000","last_clean":"2021-03-15T22:34:19.546716+0000","last_became_active":"2021-03-15T22:33:04.336100+0000","last_became_peered":"2021-03-15T22:33:04.336100+0000","last_unstale":"2021-03-15T22:34:19.546716+0000","last_undegraded":"2021-03-15T22:34:19.546716+0000","last_fullsized":"2021-03-15T22:34:19.546716+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:33:14.283762+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.283762+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"898","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.517961+0000","last_change":"2021-03-15T22:34:10.350754+0000","last_active":"2021-03-15T22:34:17.517961+0000","last_peered":"2021-03-15T22:34:17.517961+0000","last_clean":"2021-03-15T22:34:17.517961+0000","last_became_active":"2021-03-15T22:34:10.349575+0000","last_became_peered":"2021-03-15T22:34:10.349575+0000","last_unstale":"2021-03-15T22:34:17.517961+0000","last_undegraded":"2021-03-15T22:34:17.517961+0000","last_fullsized":"2021-03-15T22:34:17.517961+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.469521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.469521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"84","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.544704+0000","last_change":"2021-03-15T22:33:17.305262+0000","last_active":"2021-03-15T22:34:19.544704+0000","last_peered":"2021-03-15T22:34:19.544704+0000","last_clean":"2021-03-15T22:34:19.544704+0000","last_became_active":"2021-03-15T22:33:04.335734+0000","last_became_peered":"2021-03-15T22:33:04.335734+0000","last_unstale":"2021-03-15T22:34:19.544704+0000","last_undegraded":"2021-03-15T22:34:19.544704+0000","last_fullsized":"2021-03-15T22:34:19.544704+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.305181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.305181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"417","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560652+0000","last_change":"2021-03-15T22:34:10.325949+0000","last_active":"2021-03-15T22:34:19.560652+0000","last_peered":"2021-03-15T22:34:19.560652+0000","last_clean":"2021-03-15T22:34:19.560652+0000","last_became_active":"2021-03-15T22:34:10.325671+0000","last_became_peered":"2021-03-15T22:34:10.325671+0000","last_unstale":"2021-03-15T22:34:19.560652+0000","last_undegraded":"2021-03-15T22:34:19.560652+0000","last_fullsized":"2021-03-15T22:34:19.560652+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:33.136112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:31:33.136112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"419","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402270+0000","last_change":"2021-03-15T22:33:37.752077+0000","last_active":"2021-03-15T22:34:21.402270+0000","last_peered":"2021-03-15T22:34:21.402270+0000","last_clean":"2021-03-15T22:34:21.402270+0000","last_became_active":"2021-03-15T22:33:04.342749+0000","last_became_peered":"2021-03-15T22:33:04.342749+0000","last_unstale":"2021-03-15T22:34:21.402270+0000","last_undegraded":"2021-03-15T22:34:21.402270+0000","last_fullsized":"2021-03-15T22:34:21.402270+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:33:37.752011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.752011+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":8764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"527","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402956+0000","last_change":"2021-03-15T22:33:04.333373+0000","last_active":"2021-03-15T22:34:21.402956+0000","last_peered":"2021-03-15T22:34:21.402956+0000","last_clean":"2021-03-15T22:34:21.402956+0000","last_became_active":"2021-03-15T22:33:04.333077+0000","last_became_peered":"2021-03-15T22:33:04.333077+0000","last_unstale":"2021-03-15T22:34:21.402956+0000","last_undegraded":"2021-03-15T22:34:21.402956+0000","last_fullsized":"2021-03-15T22:34:21.402956+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.317817+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"28","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402341+0000","last_change":"2021-03-15T22:34:03.571320+0000","last_active":"2021-03-15T22:34:21.402341+0000","last_peered":"2021-03-15T22:34:21.402341+0000","last_clean":"2021-03-15T22:34:21.402341+0000","last_became_active":"2021-03-15T22:34:03.571049+0000","last_became_peered":"2021-03-15T22:34:03.571049+0000","last_unstale":"2021-03-15T22:34:21.402341+0000","last_undegraded":"2021-03-15T22:34:21.402341+0000","last_fullsized":"2021-03-15T22:34:21.402341+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:05.522372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.522372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"278","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523696+0000","last_change":"2021-03-15T22:33:12.882245+0000","last_active":"2021-03-15T22:34:17.523696+0000","last_peered":"2021-03-15T22:34:17.523696+0000","last_clean":"2021-03-15T22:34:17.523696+0000","last_became_active":"2021-03-15T22:29:51.220261+0000","last_became_peered":"2021-03-15T22:29:51.220261+0000","last_unstale":"2021-03-15T22:34:17.523696+0000","last_undegraded":"2021-03-15T22:34:17.523696+0000","last_fullsized":"2021-03-15T22:34:17.523696+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.882164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.882164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"745","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402380+0000","last_change":"2021-03-15T22:33:04.352497+0000","last_active":"2021-03-15T22:34:21.402380+0000","last_peered":"2021-03-15T22:34:21.402380+0000","last_clean":"2021-03-15T22:34:21.402380+0000","last_became_active":"2021-03-15T22:33:04.331607+0000","last_became_peered":"2021-03-15T22:33:04.331607+0000","last_unstale":"2021-03-15T22:34:21.402380+0000","last_undegraded":"2021-03-15T22:34:21.402380+0000","last_fullsized":"2021-03-15T22:34:21.402380+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:31:07.330960+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:31:07.330960+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"388","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.559826+0000","last_change":"2021-03-15T22:31:43.405661+0000","last_active":"2021-03-15T22:34:19.559826+0000","last_peered":"2021-03-15T22:34:19.559826+0000","last_clean":"2021-03-15T22:34:19.559826+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:34:19.559826+0000","last_undegraded":"2021-03-15T22:34:19.559826+0000","last_fullsized":"2021-03-15T22:34:19.559826+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:43.405600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:31:43.405600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"699","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402763+0000","last_change":"2021-03-15T22:32:37.690585+0000","last_active":"2021-03-15T22:34:21.402763+0000","last_peered":"2021-03-15T22:34:21.402763+0000","last_clean":"2021-03-15T22:34:21.402763+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:34:21.402763+0000","last_undegraded":"2021-03-15T22:34:21.402763+0000","last_fullsized":"2021-03-15T22:34:21.402763+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:37.690525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:32:37.690525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"902","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.518439+0000","last_change":"2021-03-15T22:32:33.438804+0000","last_active":"2021-03-15T22:34:17.518439+0000","last_peered":"2021-03-15T22:34:17.518439+0000","last_clean":"2021-03-15T22:34:17.518439+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:34:17.518439+0000","last_undegraded":"2021-03-15T22:34:17.518439+0000","last_fullsized":"2021-03-15T22:34:17.518439+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.438742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.438742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"584","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.544800+0000","last_change":"2021-03-15T22:34:10.334670+0000","last_active":"2021-03-15T22:34:19.544800+0000","last_peered":"2021-03-15T22:34:19.544800+0000","last_clean":"2021-03-15T22:34:19.544800+0000","last_became_active":"2021-03-15T22:34:10.334322+0000","last_became_peered":"2021-03-15T22:34:10.334322+0000","last_unstale":"2021-03-15T22:34:19.544800+0000","last_undegraded":"2021-03-15T22:34:19.544800+0000","last_fullsized":"2021-03-15T22:34:19.544800+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:33:20.329858+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.329858+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"276","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.559893+0000","last_change":"2021-03-15T22:34:10.329272+0000","last_active":"2021-03-15T22:34:19.559893+0000","last_peered":"2021-03-15T22:34:19.559893+0000","last_clean":"2021-03-15T22:34:19.559893+0000","last_became_active":"2021-03-15T22:34:10.328943+0000","last_became_peered":"2021-03-15T22:34:10.328943+0000","last_unstale":"2021-03-15T22:34:19.559893+0000","last_undegraded":"2021-03-15T22:34:19.559893+0000","last_fullsized":"2021-03-15T22:34:19.559893+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:06.670324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:32:06.670324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,6],"acting":[0,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"965","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.557063+0000","last_change":"2021-03-15T22:34:10.353649+0000","last_active":"2021-03-15T22:34:21.557063+0000","last_peered":"2021-03-15T22:34:21.557063+0000","last_clean":"2021-03-15T22:34:21.557063+0000","last_became_active":"2021-03-15T22:34:10.353314+0000","last_became_peered":"2021-03-15T22:34:10.353314+0000","last_unstale":"2021-03-15T22:34:21.557063+0000","last_undegraded":"2021-03-15T22:34:21.557063+0000","last_fullsized":"2021-03-15T22:34:21.557063+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:43.268612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.268612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"41","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538282+0000","last_change":"2021-03-15T22:33:45.314503+0000","last_active":"2021-03-15T22:34:20.538282+0000","last_peered":"2021-03-15T22:34:20.538282+0000","last_clean":"2021-03-15T22:34:20.538282+0000","last_became_active":"2021-03-15T22:33:45.314042+0000","last_became_peered":"2021-03-15T22:33:45.314042+0000","last_unstale":"2021-03-15T22:34:20.538282+0000","last_undegraded":"2021-03-15T22:34:20.538282+0000","last_fullsized":"2021-03-15T22:34:20.538282+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:08.884878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.884878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"987","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538118+0000","last_change":"2021-03-15T22:33:05.224785+0000","last_active":"2021-03-15T22:34:20.538118+0000","last_peered":"2021-03-15T22:34:20.538118+0000","last_clean":"2021-03-15T22:34:20.538118+0000","last_became_active":"2021-03-15T22:33:04.340460+0000","last_became_peered":"2021-03-15T22:33:04.340460+0000","last_unstale":"2021-03-15T22:34:20.538118+0000","last_undegraded":"2021-03-15T22:34:20.538118+0000","last_fullsized":"2021-03-15T22:34:20.538118+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:31:42.396996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.396996+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"396","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560656+0000","last_change":"2021-03-15T22:34:10.327350+0000","last_active":"2021-03-15T22:34:19.560656+0000","last_peered":"2021-03-15T22:34:19.560656+0000","last_clean":"2021-03-15T22:34:19.560656+0000","last_became_active":"2021-03-15T22:34:10.326881+0000","last_became_peered":"2021-03-15T22:34:10.326881+0000","last_unstale":"2021-03-15T22:34:19.560656+0000","last_undegraded":"2021-03-15T22:34:19.560656+0000","last_fullsized":"2021-03-15T22:34:19.560656+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:19.560387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:19.560387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"808'9","reported_seq":"337","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523515+0000","last_change":"2021-03-15T22:32:38.881054+0000","last_active":"2021-03-15T22:34:17.523515+0000","last_peered":"2021-03-15T22:34:17.523515+0000","last_clean":"2021-03-15T22:34:17.523515+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:34:17.523515+0000","last_undegraded":"2021-03-15T22:34:17.523515+0000","last_fullsized":"2021-03-15T22:34:17.523515+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9","last_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_deep_scrub":"808'9","last_deep_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.880993+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"892","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.517649+0000","last_change":"2021-03-15T22:33:45.314935+0000","last_active":"2021-03-15T22:34:17.517649+0000","last_peered":"2021-03-15T22:34:17.517649+0000","last_clean":"2021-03-15T22:34:17.517649+0000","last_became_active":"2021-03-15T22:33:45.314581+0000","last_became_peered":"2021-03-15T22:33:45.314581+0000","last_unstale":"2021-03-15T22:34:17.517649+0000","last_undegraded":"2021-03-15T22:34:17.517649+0000","last_fullsized":"2021-03-15T22:34:17.517649+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.516668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.516668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"844'33","reported_seq":"736","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402607+0000","last_change":"2021-03-15T22:34:10.358943+0000","last_active":"2021-03-15T22:34:21.402607+0000","last_peered":"2021-03-15T22:34:21.402607+0000","last_clean":"2021-03-15T22:34:21.402607+0000","last_became_active":"2021-03-15T22:34:10.322836+0000","last_became_peered":"2021-03-15T22:34:10.322836+0000","last_unstale":"2021-03-15T22:34:21.402607+0000","last_undegraded":"2021-03-15T22:34:21.402607+0000","last_fullsized":"2021-03-15T22:34:21.402607+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"844'33","last_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_deep_scrub":"844'33","last_deep_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.622437+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":1872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"929","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538509+0000","last_change":"2021-03-15T22:33:45.319789+0000","last_active":"2021-03-15T22:34:20.538509+0000","last_peered":"2021-03-15T22:34:20.538509+0000","last_clean":"2021-03-15T22:34:20.538509+0000","last_became_active":"2021-03-15T22:33:45.319467+0000","last_became_peered":"2021-03-15T22:33:45.319467+0000","last_unstale":"2021-03-15T22:34:20.538509+0000","last_undegraded":"2021-03-15T22:34:20.538509+0000","last_fullsized":"2021-03-15T22:34:20.538509+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:41.407324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.407324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"896'16","reported_seq":"394","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.519124+0000","last_change":"2021-03-15T22:34:10.443822+0000","last_active":"2021-03-15T22:34:17.519124+0000","last_peered":"2021-03-15T22:34:17.519124+0000","last_clean":"2021-03-15T22:34:17.519124+0000","last_became_active":"2021-03-15T22:34:10.348856+0000","last_became_peered":"2021-03-15T22:34:10.348856+0000","last_unstale":"2021-03-15T22:34:17.519124+0000","last_undegraded":"2021-03-15T22:34:17.519124+0000","last_fullsized":"2021-03-15T22:34:17.519124+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"683'14","last_scrub_stamp":"2021-03-15T22:31:41.551435+0000","last_deep_scrub":"361'6","last_deep_scrub_stamp":"2021-03-15T22:24:36.098511+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.551435+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"396","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.559900+0000","last_change":"2021-03-15T22:34:10.347844+0000","last_active":"2021-03-15T22:34:19.559900+0000","last_peered":"2021-03-15T22:34:19.559900+0000","last_clean":"2021-03-15T22:34:19.559900+0000","last_became_active":"2021-03-15T22:34:10.347512+0000","last_became_peered":"2021-03-15T22:34:10.347512+0000","last_unstale":"2021-03-15T22:34:19.559900+0000","last_undegraded":"2021-03-15T22:34:19.559900+0000","last_fullsized":"2021-03-15T22:34:19.559900+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:03.502854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:32:03.502854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"296","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523345+0000","last_change":"2021-03-15T22:32:41.879032+0000","last_active":"2021-03-15T22:34:17.523345+0000","last_peered":"2021-03-15T22:34:17.523345+0000","last_clean":"2021-03-15T22:34:17.523345+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:34:17.523345+0000","last_undegraded":"2021-03-15T22:34:17.523345+0000","last_fullsized":"2021-03-15T22:34:17.523345+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:32:41.878949+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.878949+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"192","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.546319+0000","last_change":"2021-03-15T22:34:10.328735+0000","last_active":"2021-03-15T22:34:19.546319+0000","last_peered":"2021-03-15T22:34:19.546319+0000","last_clean":"2021-03-15T22:34:19.546319+0000","last_became_active":"2021-03-15T22:34:10.328416+0000","last_became_peered":"2021-03-15T22:34:10.328416+0000","last_unstale":"2021-03-15T22:34:19.546319+0000","last_undegraded":"2021-03-15T22:34:19.546319+0000","last_fullsized":"2021-03-15T22:34:19.546319+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.621539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"480","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538596+0000","last_change":"2021-03-15T22:33:45.316783+0000","last_active":"2021-03-15T22:34:20.538596+0000","last_peered":"2021-03-15T22:34:20.538596+0000","last_clean":"2021-03-15T22:34:20.538596+0000","last_became_active":"2021-03-15T22:33:45.316449+0000","last_became_peered":"2021-03-15T22:33:45.316449+0000","last_unstale":"2021-03-15T22:34:20.538596+0000","last_undegraded":"2021-03-15T22:34:20.538596+0000","last_fullsized":"2021-03-15T22:34:20.538596+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:42.894499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:33:42.894499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"894","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.516966+0000","last_change":"2021-03-15T22:33:21.545910+0000","last_active":"2021-03-15T22:34:17.516966+0000","last_peered":"2021-03-15T22:34:17.516966+0000","last_clean":"2021-03-15T22:34:17.516966+0000","last_became_active":"2021-03-15T22:33:04.330195+0000","last_became_peered":"2021-03-15T22:33:04.330195+0000","last_unstale":"2021-03-15T22:34:17.516966+0000","last_undegraded":"2021-03-15T22:34:17.516966+0000","last_fullsized":"2021-03-15T22:34:17.516966+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.545826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.545826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"427","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560452+0000","last_change":"2021-03-15T22:31:04.009824+0000","last_active":"2021-03-15T22:34:19.560452+0000","last_peered":"2021-03-15T22:34:19.560452+0000","last_clean":"2021-03-15T22:34:19.560452+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:34:19.560452+0000","last_undegraded":"2021-03-15T22:34:19.560452+0000","last_fullsized":"2021-03-15T22:34:19.560452+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:31:04.009748+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.009748+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"698","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523775+0000","last_change":"2021-03-15T22:32:11.834186+0000","last_active":"2021-03-15T22:34:17.523775+0000","last_peered":"2021-03-15T22:34:17.523775+0000","last_clean":"2021-03-15T22:34:17.523775+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:34:17.523775+0000","last_undegraded":"2021-03-15T22:34:17.523775+0000","last_fullsized":"2021-03-15T22:34:17.523775+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:11.834104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:11.834104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"888","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.518364+0000","last_change":"2021-03-15T22:33:19.496857+0000","last_active":"2021-03-15T22:34:17.518364+0000","last_peered":"2021-03-15T22:34:17.518364+0000","last_clean":"2021-03-15T22:34:17.518364+0000","last_became_active":"2021-03-15T22:33:04.331013+0000","last_became_peered":"2021-03-15T22:33:04.331013+0000","last_unstale":"2021-03-15T22:34:17.518364+0000","last_undegraded":"2021-03-15T22:34:17.518364+0000","last_fullsized":"2021-03-15T22:34:17.518364+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:19.496753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:19.496753+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"84","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.544708+0000","last_change":"2021-03-15T22:34:07.203702+0000","last_active":"2021-03-15T22:34:19.544708+0000","last_peered":"2021-03-15T22:34:19.544708+0000","last_clean":"2021-03-15T22:34:19.544708+0000","last_became_active":"2021-03-15T22:33:04.336886+0000","last_became_peered":"2021-03-15T22:33:04.336886+0000","last_unstale":"2021-03-15T22:34:19.544708+0000","last_undegraded":"2021-03-15T22:34:19.544708+0000","last_fullsized":"2021-03-15T22:34:19.544708+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.203639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:39.697935+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.203639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"228","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.518965+0000","last_change":"2021-03-15T22:34:10.356382+0000","last_active":"2021-03-15T22:34:17.518965+0000","last_peered":"2021-03-15T22:34:17.518965+0000","last_clean":"2021-03-15T22:34:17.518965+0000","last_became_active":"2021-03-15T22:34:10.333956+0000","last_became_peered":"2021-03-15T22:34:10.333956+0000","last_unstale":"2021-03-15T22:34:17.518965+0000","last_undegraded":"2021-03-15T22:34:17.518965+0000","last_fullsized":"2021-03-15T22:34:17.518965+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:32:01.586755+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:32:01.586755+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"907'1200","reported_seq":"4048","reported_epoch":"907","state":"active+clean","last_fresh":"2021-03-15T22:34:22.565780+0000","last_change":"2021-03-15T22:34:22.565952+0000","last_active":"2021-03-15T22:34:22.565780+0000","last_peered":"2021-03-15T22:34:22.565780+0000","last_clean":"2021-03-15T22:34:22.565780+0000","last_became_active":"2021-03-15T22:33:08.355490+0000","last_became_peered":"2021-03-15T22:33:08.355490+0000","last_unstale":"2021-03-15T22:34:22.565780+0000","last_undegraded":"2021-03-15T22:34:22.565780+0000","last_fullsized":"2021-03-15T22:34:22.565780+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'1088","last_scrub_stamp":"2021-03-15T22:33:10.457078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:10.457078+0000","log_size":1200,"ondisk_log_size":1200,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47484436,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":454,"num_read_kb":504188,"num_write":310,"num_write_kb":105343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":84869361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"},{"start":"2e1","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"845","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523281+0000","last_change":"2021-03-15T22:33:42.741529+0000","last_active":"2021-03-15T22:34:17.523281+0000","last_peered":"2021-03-15T22:34:17.523281+0000","last_clean":"2021-03-15T22:34:17.523281+0000","last_became_active":"2021-03-15T22:33:04.340352+0000","last_became_peered":"2021-03-15T22:33:04.340352+0000","last_unstale":"2021-03-15T22:34:17.523281+0000","last_undegraded":"2021-03-15T22:34:17.523281+0000","last_fullsized":"2021-03-15T22:34:17.523281+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:42.741451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:42.741451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"874'4","reported_seq":"911","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.517784+0000","last_change":"2021-03-15T22:33:44.498327+0000","last_active":"2021-03-15T22:34:17.517784+0000","last_peered":"2021-03-15T22:34:17.517784+0000","last_clean":"2021-03-15T22:34:17.517784+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:34:17.517784+0000","last_undegraded":"2021-03-15T22:34:17.517784+0000","last_fullsized":"2021-03-15T22:34:17.517784+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"874'4","last_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_deep_scrub":"874'4","last_deep_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.498263+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"372","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.522887+0000","last_change":"2021-03-15T22:33:38.759477+0000","last_active":"2021-03-15T22:34:17.522887+0000","last_peered":"2021-03-15T22:34:17.522887+0000","last_clean":"2021-03-15T22:34:17.522887+0000","last_became_active":"2021-03-15T22:33:04.326070+0000","last_became_peered":"2021-03-15T22:33:04.326070+0000","last_unstale":"2021-03-15T22:34:17.522887+0000","last_undegraded":"2021-03-15T22:34:17.522887+0000","last_fullsized":"2021-03-15T22:34:17.522887+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:33:38.759395+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.759395+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"841'14","reported_seq":"1153","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.517085+0000","last_change":"2021-03-15T22:33:43.476783+0000","last_active":"2021-03-15T22:34:17.517085+0000","last_peered":"2021-03-15T22:34:17.517085+0000","last_clean":"2021-03-15T22:34:17.517085+0000","last_became_active":"2021-03-15T22:33:08.354031+0000","last_became_peered":"2021-03-15T22:33:08.354031+0000","last_unstale":"2021-03-15T22:34:17.517085+0000","last_undegraded":"2021-03-15T22:34:17.517085+0000","last_fullsized":"2021-03-15T22:34:17.517085+0000","mapping_epoch":837,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'14","last_scrub_stamp":"2021-03-15T22:33:43.476701+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.476701+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"904'1670","reported_seq":"4519","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402440+0000","last_change":"2021-03-15T22:34:13.335301+0000","last_active":"2021-03-15T22:34:21.402440+0000","last_peered":"2021-03-15T22:34:21.402440+0000","last_clean":"2021-03-15T22:34:21.402440+0000","last_became_active":"2021-03-15T22:33:48.313152+0000","last_became_peered":"2021-03-15T22:33:48.313152+0000","last_unstale":"2021-03-15T22:34:21.402440+0000","last_undegraded":"2021-03-15T22:34:21.402440+0000","last_fullsized":"2021-03-15T22:34:21.402440+0000","mapping_epoch":877,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1312","last_scrub_stamp":"2021-03-15T22:32:05.639057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.639057+0000","log_size":1670,"ondisk_log_size":1670,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":78331380,"num_objects":36,"num_object_clones":31,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":1,"num_read":631,"num_read_kb":611520,"num_write":451,"num_write_kb":183711,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":137371477,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"335","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402609+0000","last_change":"2021-03-15T22:32:42.621654+0000","last_active":"2021-03-15T22:34:21.402609+0000","last_peered":"2021-03-15T22:34:21.402609+0000","last_clean":"2021-03-15T22:34:21.402609+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:34:21.402609+0000","last_undegraded":"2021-03-15T22:34:21.402609+0000","last_fullsized":"2021-03-15T22:34:21.402609+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:42.621594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:42.621594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"878","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.518700+0000","last_change":"2021-03-15T22:33:22.542596+0000","last_active":"2021-03-15T22:34:17.518700+0000","last_peered":"2021-03-15T22:34:17.518700+0000","last_clean":"2021-03-15T22:34:17.518700+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:34:17.518700+0000","last_undegraded":"2021-03-15T22:34:17.518700+0000","last_fullsized":"2021-03-15T22:34:17.518700+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.542506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.542506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"1115","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538084+0000","last_change":"2021-03-15T22:34:11.031994+0000","last_active":"2021-03-15T22:34:20.538084+0000","last_peered":"2021-03-15T22:34:20.538084+0000","last_clean":"2021-03-15T22:34:20.538084+0000","last_became_active":"2021-03-15T22:34:10.342120+0000","last_became_peered":"2021-03-15T22:34:10.342120+0000","last_unstale":"2021-03-15T22:34:20.538084+0000","last_undegraded":"2021-03-15T22:34:20.538084+0000","last_fullsized":"2021-03-15T22:34:20.538084+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:34:07.452572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.452572+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"608","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.559901+0000","last_change":"2021-03-15T22:33:45.318595+0000","last_active":"2021-03-15T22:34:19.559901+0000","last_peered":"2021-03-15T22:34:19.559901+0000","last_clean":"2021-03-15T22:34:19.559901+0000","last_became_active":"2021-03-15T22:33:45.318247+0000","last_became_peered":"2021-03-15T22:33:45.318247+0000","last_unstale":"2021-03-15T22:34:19.559901+0000","last_undegraded":"2021-03-15T22:34:19.559901+0000","last_fullsized":"2021-03-15T22:34:19.559901+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"904'468","reported_seq":"2998","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.593598+0000","last_change":"2021-03-15T22:34:13.339268+0000","last_active":"2021-03-15T22:34:19.593598+0000","last_peered":"2021-03-15T22:34:19.593598+0000","last_clean":"2021-03-15T22:34:19.593598+0000","last_became_active":"2021-03-15T22:34:13.324667+0000","last_became_peered":"2021-03-15T22:34:13.324667+0000","last_unstale":"2021-03-15T22:34:19.593598+0000","last_undegraded":"2021-03-15T22:34:19.593598+0000","last_fullsized":"2021-03-15T22:34:19.593598+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'446","last_scrub_stamp":"2021-03-15T22:33:39.355306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.355306+0000","log_size":468,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15861605,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":120,"num_read_kb":94113,"num_write":119,"num_write_kb":37568,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":23698344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"928","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.516808+0000","last_change":"2021-03-15T22:33:45.325203+0000","last_active":"2021-03-15T22:34:17.516808+0000","last_peered":"2021-03-15T22:34:17.516808+0000","last_clean":"2021-03-15T22:34:17.516808+0000","last_became_active":"2021-03-15T22:33:45.323552+0000","last_became_peered":"2021-03-15T22:33:45.323552+0000","last_unstale":"2021-03-15T22:34:17.516808+0000","last_undegraded":"2021-03-15T22:34:17.516808+0000","last_fullsized":"2021-03-15T22:34:17.516808+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,6],"acting":[7,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"594","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560106+0000","last_change":"2021-03-15T22:33:45.313059+0000","last_active":"2021-03-15T22:34:19.560106+0000","last_peered":"2021-03-15T22:34:19.560106+0000","last_clean":"2021-03-15T22:34:19.560106+0000","last_became_active":"2021-03-15T22:33:45.312722+0000","last_became_peered":"2021-03-15T22:33:45.312722+0000","last_unstale":"2021-03-15T22:34:19.560106+0000","last_undegraded":"2021-03-15T22:34:19.560106+0000","last_fullsized":"2021-03-15T22:34:19.560106+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.809863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.809863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"933","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.518659+0000","last_change":"2021-03-15T22:33:04.331301+0000","last_active":"2021-03-15T22:34:17.518659+0000","last_peered":"2021-03-15T22:34:17.518659+0000","last_clean":"2021-03-15T22:34:17.518659+0000","last_became_active":"2021-03-15T22:33:04.330771+0000","last_became_peered":"2021-03-15T22:33:04.330771+0000","last_unstale":"2021-03-15T22:34:17.518659+0000","last_undegraded":"2021-03-15T22:34:17.518659+0000","last_fullsized":"2021-03-15T22:34:17.518659+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:40.430038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:32:40.430038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"149","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.544890+0000","last_change":"2021-03-15T22:34:10.249538+0000","last_active":"2021-03-15T22:34:19.544890+0000","last_peered":"2021-03-15T22:34:19.544890+0000","last_clean":"2021-03-15T22:34:19.544890+0000","last_became_active":"2021-03-15T22:33:45.318867+0000","last_became_peered":"2021-03-15T22:33:45.318867+0000","last_unstale":"2021-03-15T22:34:19.544890+0000","last_undegraded":"2021-03-15T22:34:19.544890+0000","last_fullsized":"2021-03-15T22:34:19.544890+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:34:10.249444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:10.249444+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":2928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"900'1004","reported_seq":"3438","reported_epoch":"903","state":"peering","last_fresh":"2021-03-15T22:34:18.523070+0000","last_change":"2021-03-15T22:34:18.523426+0000","last_active":"2021-03-15T22:34:18.523070+0000","last_peered":"2021-03-15T22:34:17.518192+0000","last_clean":"2021-03-15T22:34:08.505725+0000","last_became_active":"2021-03-15T22:34:11.344939+0000","last_became_peered":"2021-03-15T22:34:11.344939+0000","last_unstale":"2021-03-15T22:34:18.523070+0000","last_undegraded":"2021-03-15T22:34:18.523070+0000","last_fullsized":"2021-03-15T22:34:18.523070+0000","mapping_epoch":903,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":883,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'783","last_scrub_stamp":"2021-03-15T22:32:31.034631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.034631+0000","log_size":1004,"ondisk_log_size":1004,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":58220180,"num_objects":26,"num_object_clones":23,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":393,"num_read_kb":471188,"num_write":256,"num_write_kb":91578,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":152,"num_bytes_recovered":110432026,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[3,4,5],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"389","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.559972+0000","last_change":"2021-03-15T22:33:45.327914+0000","last_active":"2021-03-15T22:34:19.559972+0000","last_peered":"2021-03-15T22:34:19.559972+0000","last_clean":"2021-03-15T22:34:19.559972+0000","last_became_active":"2021-03-15T22:33:45.327557+0000","last_became_peered":"2021-03-15T22:33:45.327557+0000","last_unstale":"2021-03-15T22:34:19.559972+0000","last_undegraded":"2021-03-15T22:34:19.559972+0000","last_fullsized":"2021-03-15T22:34:19.559972+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"721","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538016+0000","last_change":"2021-03-15T22:34:10.345265+0000","last_active":"2021-03-15T22:34:20.538016+0000","last_peered":"2021-03-15T22:34:20.538016+0000","last_clean":"2021-03-15T22:34:20.538016+0000","last_became_active":"2021-03-15T22:34:10.345016+0000","last_became_peered":"2021-03-15T22:34:10.345016+0000","last_unstale":"2021-03-15T22:34:20.538016+0000","last_undegraded":"2021-03-15T22:34:20.538016+0000","last_fullsized":"2021-03-15T22:34:20.538016+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:14.888119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.888119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,1],"acting":[4,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"303","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523279+0000","last_change":"2021-03-15T22:34:07.807466+0000","last_active":"2021-03-15T22:34:17.523279+0000","last_peered":"2021-03-15T22:34:17.523279+0000","last_clean":"2021-03-15T22:34:17.523279+0000","last_became_active":"2021-03-15T22:33:04.333291+0000","last_became_peered":"2021-03-15T22:33:04.333291+0000","last_unstale":"2021-03-15T22:34:17.523279+0000","last_undegraded":"2021-03-15T22:34:17.523279+0000","last_fullsized":"2021-03-15T22:34:17.523279+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.807411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.807411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"447","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560231+0000","last_change":"2021-03-15T22:33:46.129541+0000","last_active":"2021-03-15T22:34:19.560231+0000","last_peered":"2021-03-15T22:34:19.560231+0000","last_clean":"2021-03-15T22:34:19.560231+0000","last_became_active":"2021-03-15T22:33:45.326747+0000","last_became_peered":"2021-03-15T22:33:45.326747+0000","last_unstale":"2021-03-15T22:34:19.560231+0000","last_undegraded":"2021-03-15T22:34:19.560231+0000","last_fullsized":"2021-03-15T22:34:19.560231+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:32:33.701675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.701675+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":2682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"905'1194","reported_seq":"4330","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.546100+0000","last_change":"2021-03-15T22:34:13.337220+0000","last_active":"2021-03-15T22:34:20.546100+0000","last_peered":"2021-03-15T22:34:20.546100+0000","last_clean":"2021-03-15T22:34:20.546100+0000","last_became_active":"2021-03-15T22:34:13.319812+0000","last_became_peered":"2021-03-15T22:34:13.319812+0000","last_unstale":"2021-03-15T22:34:20.546100+0000","last_undegraded":"2021-03-15T22:34:20.546100+0000","last_fullsized":"2021-03-15T22:34:20.546100+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'1103","last_scrub_stamp":"2021-03-15T22:33:09.245691+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.245691+0000","log_size":1194,"ondisk_log_size":1194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34300642,"num_objects":17,"num_object_clones":14,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":285,"num_read_kb":224542,"num_write":301,"num_write_kb":103453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":404,"num_bytes_recovered":201860638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"315","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538170+0000","last_change":"2021-03-15T22:34:10.330710+0000","last_active":"2021-03-15T22:34:20.538170+0000","last_peered":"2021-03-15T22:34:20.538170+0000","last_clean":"2021-03-15T22:34:20.538170+0000","last_became_active":"2021-03-15T22:34:10.330386+0000","last_became_peered":"2021-03-15T22:34:10.330386+0000","last_unstale":"2021-03-15T22:34:20.538170+0000","last_undegraded":"2021-03-15T22:34:20.538170+0000","last_fullsized":"2021-03-15T22:34:20.538170+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"295","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523472+0000","last_change":"2021-03-15T22:33:50.846420+0000","last_active":"2021-03-15T22:34:17.523472+0000","last_peered":"2021-03-15T22:34:17.523472+0000","last_clean":"2021-03-15T22:34:17.523472+0000","last_became_active":"2021-03-15T22:33:04.339912+0000","last_became_peered":"2021-03-15T22:33:04.339912+0000","last_unstale":"2021-03-15T22:34:17.523472+0000","last_undegraded":"2021-03-15T22:34:17.523472+0000","last_fullsized":"2021-03-15T22:34:17.523472+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:58.785524+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:31:58.785524+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"103","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.537850+0000","last_change":"2021-03-15T22:33:04.336661+0000","last_active":"2021-03-15T22:34:20.537850+0000","last_peered":"2021-03-15T22:34:20.537850+0000","last_clean":"2021-03-15T22:34:20.537850+0000","last_became_active":"2021-03-15T22:33:04.335914+0000","last_became_peered":"2021-03-15T22:33:04.335914+0000","last_unstale":"2021-03-15T22:34:20.537850+0000","last_undegraded":"2021-03-15T22:34:20.537850+0000","last_fullsized":"2021-03-15T22:34:20.537850+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.051962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.051962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1201","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.556850+0000","last_change":"2021-03-15T22:34:10.389157+0000","last_active":"2021-03-15T22:34:21.556850+0000","last_peered":"2021-03-15T22:34:21.556850+0000","last_clean":"2021-03-15T22:34:21.556850+0000","last_became_active":"2021-03-15T22:34:10.350383+0000","last_became_peered":"2021-03-15T22:34:10.350383+0000","last_unstale":"2021-03-15T22:34:21.556850+0000","last_undegraded":"2021-03-15T22:34:21.556850+0000","last_fullsized":"2021-03-15T22:34:21.556850+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:33:38.133617+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.133617+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":71,"num_bytes_recovered":4770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"369","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560256+0000","last_change":"2021-03-15T22:33:15.492116+0000","last_active":"2021-03-15T22:34:19.560256+0000","last_peered":"2021-03-15T22:34:19.560256+0000","last_clean":"2021-03-15T22:34:19.560256+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:34:19.560256+0000","last_undegraded":"2021-03-15T22:34:19.560256+0000","last_fullsized":"2021-03-15T22:34:19.560256+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.492014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.492014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"901'1155","reported_seq":"3554","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560190+0000","last_change":"2021-03-15T22:34:13.329771+0000","last_active":"2021-03-15T22:34:19.560190+0000","last_peered":"2021-03-15T22:34:19.560190+0000","last_clean":"2021-03-15T22:34:19.560190+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:34:19.560190+0000","last_undegraded":"2021-03-15T22:34:19.560190+0000","last_fullsized":"2021-03-15T22:34:19.560190+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"805'997","last_scrub_stamp":"2021-03-15T22:32:36.795301+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.795301+0000","log_size":1155,"ondisk_log_size":1155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42772202,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":331,"num_read_kb":327804,"num_write":256,"num_write_kb":98389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"920","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.518797+0000","last_change":"2021-03-15T22:33:04.334561+0000","last_active":"2021-03-15T22:34:17.518797+0000","last_peered":"2021-03-15T22:34:17.518797+0000","last_clean":"2021-03-15T22:34:17.518797+0000","last_became_active":"2021-03-15T22:33:04.334295+0000","last_became_peered":"2021-03-15T22:33:04.334295+0000","last_unstale":"2021-03-15T22:34:17.518797+0000","last_undegraded":"2021-03-15T22:34:17.518797+0000","last_fullsized":"2021-03-15T22:34:17.518797+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:32.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"930","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.556551+0000","last_change":"2021-03-15T22:34:10.222593+0000","last_active":"2021-03-15T22:34:21.556551+0000","last_peered":"2021-03-15T22:34:21.556551+0000","last_clean":"2021-03-15T22:34:21.556551+0000","last_became_active":"2021-03-15T22:33:45.322214+0000","last_became_peered":"2021-03-15T22:33:45.322214+0000","last_unstale":"2021-03-15T22:34:21.556551+0000","last_undegraded":"2021-03-15T22:34:21.556551+0000","last_fullsized":"2021-03-15T22:34:21.556551+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:10.222521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:10.222521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,0],"acting":[1,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"635","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.559577+0000","last_change":"2021-03-15T22:34:10.334436+0000","last_active":"2021-03-15T22:34:19.559577+0000","last_peered":"2021-03-15T22:34:19.559577+0000","last_clean":"2021-03-15T22:34:19.559577+0000","last_became_active":"2021-03-15T22:34:10.334081+0000","last_became_peered":"2021-03-15T22:34:10.334081+0000","last_unstale":"2021-03-15T22:34:19.559577+0000","last_undegraded":"2021-03-15T22:34:19.559577+0000","last_fullsized":"2021-03-15T22:34:19.559577+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:34.095158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.095158+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,6],"acting":[0,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"386","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523757+0000","last_change":"2021-03-15T22:34:10.343282+0000","last_active":"2021-03-15T22:34:17.523757+0000","last_peered":"2021-03-15T22:34:17.523757+0000","last_clean":"2021-03-15T22:34:17.523757+0000","last_became_active":"2021-03-15T22:34:10.322666+0000","last_became_peered":"2021-03-15T22:34:10.322666+0000","last_unstale":"2021-03-15T22:34:17.523757+0000","last_undegraded":"2021-03-15T22:34:17.523757+0000","last_fullsized":"2021-03-15T22:34:17.523757+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:33:40.214707+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.214707+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"553","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523236+0000","last_change":"2021-03-15T22:33:45.308301+0000","last_active":"2021-03-15T22:34:17.523236+0000","last_peered":"2021-03-15T22:34:17.523236+0000","last_clean":"2021-03-15T22:34:17.523236+0000","last_became_active":"2021-03-15T22:33:45.307632+0000","last_became_peered":"2021-03-15T22:33:45.307632+0000","last_unstale":"2021-03-15T22:34:17.523236+0000","last_undegraded":"2021-03-15T22:34:17.523236+0000","last_fullsized":"2021-03-15T22:34:17.523236+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:38.907011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.907011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"902'2697","reported_seq":"6032","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560505+0000","last_change":"2021-03-15T22:34:13.366832+0000","last_active":"2021-03-15T22:34:19.560505+0000","last_peered":"2021-03-15T22:34:19.560505+0000","last_clean":"2021-03-15T22:34:19.560505+0000","last_became_active":"2021-03-15T22:34:13.331560+0000","last_became_peered":"2021-03-15T22:34:13.331560+0000","last_unstale":"2021-03-15T22:34:19.560505+0000","last_undegraded":"2021-03-15T22:34:19.560505+0000","last_fullsized":"2021-03-15T22:34:19.560505+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'2007","last_scrub_stamp":"2021-03-15T22:31:02.050395+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.050395+0000","log_size":2697,"ondisk_log_size":2697,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":72108767,"num_objects":34,"num_object_clones":27,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":2,"num_read":790,"num_read_kb":804491,"num_write":607,"num_write_kb":250402,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":264088494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"278","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523837+0000","last_change":"2021-03-15T22:33:17.761226+0000","last_active":"2021-03-15T22:34:17.523837+0000","last_peered":"2021-03-15T22:34:17.523837+0000","last_clean":"2021-03-15T22:34:17.523837+0000","last_became_active":"2021-03-15T22:29:51.227896+0000","last_became_peered":"2021-03-15T22:29:51.227896+0000","last_unstale":"2021-03-15T22:34:17.523837+0000","last_undegraded":"2021-03-15T22:34:17.523837+0000","last_fullsized":"2021-03-15T22:34:17.523837+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.761137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"912","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.518208+0000","last_change":"2021-03-15T22:33:04.338099+0000","last_active":"2021-03-15T22:34:17.518208+0000","last_peered":"2021-03-15T22:34:17.518208+0000","last_clean":"2021-03-15T22:34:17.518208+0000","last_became_active":"2021-03-15T22:33:04.337709+0000","last_became_peered":"2021-03-15T22:33:04.337709+0000","last_unstale":"2021-03-15T22:34:17.518208+0000","last_undegraded":"2021-03-15T22:34:17.518208+0000","last_fullsized":"2021-03-15T22:34:17.518208+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:43.531349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.531349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"475","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.522876+0000","last_change":"2021-03-15T22:34:10.326358+0000","last_active":"2021-03-15T22:34:17.522876+0000","last_peered":"2021-03-15T22:34:17.522876+0000","last_clean":"2021-03-15T22:34:17.522876+0000","last_became_active":"2021-03-15T22:34:10.322476+0000","last_became_peered":"2021-03-15T22:34:10.322476+0000","last_unstale":"2021-03-15T22:34:17.522876+0000","last_undegraded":"2021-03-15T22:34:17.522876+0000","last_fullsized":"2021-03-15T22:34:17.522876+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:31:39.555487+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"104","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.546484+0000","last_change":"2021-03-15T22:34:09.215097+0000","last_active":"2021-03-15T22:34:19.546484+0000","last_peered":"2021-03-15T22:34:19.546484+0000","last_clean":"2021-03-15T22:34:19.546484+0000","last_became_active":"2021-03-15T22:33:04.330531+0000","last_became_peered":"2021-03-15T22:33:04.330531+0000","last_unstale":"2021-03-15T22:34:19.546484+0000","last_undegraded":"2021-03-15T22:34:19.546484+0000","last_fullsized":"2021-03-15T22:34:19.546484+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:34:09.215034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:09.215034+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"924","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.519181+0000","last_change":"2021-03-15T22:33:45.312186+0000","last_active":"2021-03-15T22:34:17.519181+0000","last_peered":"2021-03-15T22:34:17.519181+0000","last_clean":"2021-03-15T22:34:17.519181+0000","last_became_active":"2021-03-15T22:33:45.310317+0000","last_became_peered":"2021-03-15T22:33:45.310317+0000","last_unstale":"2021-03-15T22:34:17.519181+0000","last_undegraded":"2021-03-15T22:34:17.519181+0000","last_fullsized":"2021-03-15T22:34:17.519181+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.451041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.451041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"900'833","reported_seq":"3536","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.518275+0000","last_change":"2021-03-15T22:34:13.330465+0000","last_active":"2021-03-15T22:34:17.518275+0000","last_peered":"2021-03-15T22:34:17.518275+0000","last_clean":"2021-03-15T22:34:17.518275+0000","last_became_active":"2021-03-15T22:33:52.518138+0000","last_became_peered":"2021-03-15T22:33:52.518138+0000","last_unstale":"2021-03-15T22:34:17.518275+0000","last_undegraded":"2021-03-15T22:34:17.518275+0000","last_fullsized":"2021-03-15T22:34:17.518275+0000","mapping_epoch":881,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":882,"parent":"0.0","parent_split_bits":0,"last_scrub":"891'819","last_scrub_stamp":"2021-03-15T22:34:08.508824+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.508824+0000","log_size":833,"ondisk_log_size":833,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32109553,"num_objects":16,"num_object_clones":14,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":242,"num_read_kb":248361,"num_write":201,"num_write_kb":90213,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":123,"num_bytes_recovered":78049614,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"686","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523186+0000","last_change":"2021-03-15T22:34:10.319049+0000","last_active":"2021-03-15T22:34:17.523186+0000","last_peered":"2021-03-15T22:34:17.523186+0000","last_clean":"2021-03-15T22:34:17.523186+0000","last_became_active":"2021-03-15T22:34:10.318669+0000","last_became_peered":"2021-03-15T22:34:10.318669+0000","last_unstale":"2021-03-15T22:34:17.523186+0000","last_undegraded":"2021-03-15T22:34:17.523186+0000","last_fullsized":"2021-03-15T22:34:17.523186+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,4],"acting":[5,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"396","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560016+0000","last_change":"2021-03-15T22:34:16.297619+0000","last_active":"2021-03-15T22:34:19.560016+0000","last_peered":"2021-03-15T22:34:19.560016+0000","last_clean":"2021-03-15T22:34:19.560016+0000","last_became_active":"2021-03-15T22:34:10.321100+0000","last_became_peered":"2021-03-15T22:34:10.321100+0000","last_unstale":"2021-03-15T22:34:19.560016+0000","last_undegraded":"2021-03-15T22:34:19.560016+0000","last_fullsized":"2021-03-15T22:34:19.560016+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:01.010393+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:31:01.010393+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"1876","reported_epoch":"903","state":"active+clean","last_fresh":"2021-03-15T22:34:18.522736+0000","last_change":"2021-03-15T22:32:30.365491+0000","last_active":"2021-03-15T22:34:18.522736+0000","last_peered":"2021-03-15T22:34:18.522736+0000","last_clean":"2021-03-15T22:34:18.522736+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:34:18.522736+0000","last_undegraded":"2021-03-15T22:34:18.522736+0000","last_fullsized":"2021-03-15T22:34:18.522736+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:32:30.365432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.365432+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":969,"num_read_kb":964,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"463","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560623+0000","last_change":"2021-03-15T22:33:46.027676+0000","last_active":"2021-03-15T22:34:19.560623+0000","last_peered":"2021-03-15T22:34:19.560623+0000","last_clean":"2021-03-15T22:34:19.560623+0000","last_became_active":"2021-03-15T22:33:45.329123+0000","last_became_peered":"2021-03-15T22:33:45.329123+0000","last_unstale":"2021-03-15T22:34:19.560623+0000","last_undegraded":"2021-03-15T22:34:19.560623+0000","last_fullsized":"2021-03-15T22:34:19.560623+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:31:11.926268+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:31:11.926268+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":1890,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"287","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523904+0000","last_change":"2021-03-15T22:32:00.816143+0000","last_active":"2021-03-15T22:34:17.523904+0000","last_peered":"2021-03-15T22:34:17.523904+0000","last_clean":"2021-03-15T22:34:17.523904+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:34:17.523904+0000","last_undegraded":"2021-03-15T22:34:17.523904+0000","last_fullsized":"2021-03-15T22:34:17.523904+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:00.816090+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:32:00.816090+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"902'2961","reported_seq":"6084","reported_epoch":"903","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-03-15T22:34:19.217615+0000","last_change":"2021-03-15T22:34:11.413378+0000","last_active":"2021-03-15T22:34:19.217615+0000","last_peered":"2021-03-15T22:34:19.217615+0000","last_clean":"2021-03-15T22:34:08.329717+0000","last_became_active":"2021-03-15T22:34:11.388402+0000","last_became_peered":"2021-03-15T22:34:11.388402+0000","last_unstale":"2021-03-15T22:34:19.217615+0000","last_undegraded":"2021-03-15T22:34:11.412860+0000","last_fullsized":"2021-03-15T22:34:11.310331+0000","mapping_epoch":895,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'2597","last_scrub_stamp":"2021-03-15T22:32:30.909661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.909661+0000","log_size":2961,"ondisk_log_size":2961,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"stat_sum":{"num_bytes":110661967,"num_objects":50,"num_object_clones":43,"num_object_copies":150,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":42,"num_objects_unfound":0,"num_objects_dirty":50,"num_whiteouts":3,"num_read":1092,"num_read_kb":1172037,"num_write":703,"num_write_kb":290119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":390,"num_bytes_recovered":165660176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3],"avail_no_missing":["5","1","3"],"object_location_counts":[{"shards":"1,3,5","objects":42},{"shards":"3,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"921","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.556890+0000","last_change":"2021-03-15T22:33:45.317311+0000","last_active":"2021-03-15T22:34:21.556890+0000","last_peered":"2021-03-15T22:34:21.556890+0000","last_clean":"2021-03-15T22:34:21.556890+0000","last_became_active":"2021-03-15T22:33:45.316930+0000","last_became_peered":"2021-03-15T22:33:45.316930+0000","last_unstale":"2021-03-15T22:34:21.556890+0000","last_undegraded":"2021-03-15T22:34:21.556890+0000","last_fullsized":"2021-03-15T22:34:21.556890+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.701955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.701955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,7],"acting":[1,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"943","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.556747+0000","last_change":"2021-03-15T22:33:45.319292+0000","last_active":"2021-03-15T22:34:21.556747+0000","last_peered":"2021-03-15T22:34:21.556747+0000","last_clean":"2021-03-15T22:34:21.556747+0000","last_became_active":"2021-03-15T22:33:45.318944+0000","last_became_peered":"2021-03-15T22:33:45.318944+0000","last_unstale":"2021-03-15T22:34:21.556747+0000","last_undegraded":"2021-03-15T22:34:21.556747+0000","last_fullsized":"2021-03-15T22:34:21.556747+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_clean_scrub_stamp":"2021-03-15T22:32:44.675891+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"444","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560148+0000","last_change":"2021-03-15T22:31:40.359761+0000","last_active":"2021-03-15T22:34:19.560148+0000","last_peered":"2021-03-15T22:34:19.560148+0000","last_clean":"2021-03-15T22:34:19.560148+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:34:19.560148+0000","last_undegraded":"2021-03-15T22:34:19.560148+0000","last_fullsized":"2021-03-15T22:34:19.560148+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:31:40.359701+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:31:40.359701+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"84","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.544823+0000","last_change":"2021-03-15T22:33:22.282663+0000","last_active":"2021-03-15T22:34:19.544823+0000","last_peered":"2021-03-15T22:34:19.544823+0000","last_clean":"2021-03-15T22:34:19.544823+0000","last_became_active":"2021-03-15T22:33:04.337244+0000","last_became_peered":"2021-03-15T22:33:04.337244+0000","last_unstale":"2021-03-15T22:34:19.544823+0000","last_undegraded":"2021-03-15T22:34:19.544823+0000","last_fullsized":"2021-03-15T22:34:19.544823+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.282586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.282586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"904'486","reported_seq":"3307","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.537592+0000","last_change":"2021-03-15T22:34:15.615893+0000","last_active":"2021-03-15T22:34:20.537592+0000","last_peered":"2021-03-15T22:34:20.537592+0000","last_clean":"2021-03-15T22:34:20.537592+0000","last_became_active":"2021-03-15T22:34:15.498612+0000","last_became_peered":"2021-03-15T22:34:15.498612+0000","last_unstale":"2021-03-15T22:34:20.537592+0000","last_undegraded":"2021-03-15T22:34:20.537592+0000","last_fullsized":"2021-03-15T22:34:20.537592+0000","mapping_epoch":899,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":900,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'463","last_scrub_stamp":"2021-03-15T22:33:40.563144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.563144+0000","log_size":486,"ondisk_log_size":486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23044032,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":154,"num_read_kb":192432,"num_write":104,"num_write_kb":44944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":84391207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"407","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.559745+0000","last_change":"2021-03-15T22:33:45.330597+0000","last_active":"2021-03-15T22:34:19.559745+0000","last_peered":"2021-03-15T22:34:19.559745+0000","last_clean":"2021-03-15T22:34:19.559745+0000","last_became_active":"2021-03-15T22:33:45.327560+0000","last_became_peered":"2021-03-15T22:33:45.327560+0000","last_unstale":"2021-03-15T22:34:19.559745+0000","last_undegraded":"2021-03-15T22:34:19.559745+0000","last_fullsized":"2021-03-15T22:34:19.559745+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"857","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523681+0000","last_change":"2021-03-15T22:34:10.326577+0000","last_active":"2021-03-15T22:34:17.523681+0000","last_peered":"2021-03-15T22:34:17.523681+0000","last_clean":"2021-03-15T22:34:17.523681+0000","last_became_active":"2021-03-15T22:34:10.326083+0000","last_became_peered":"2021-03-15T22:34:10.326083+0000","last_unstale":"2021-03-15T22:34:17.523681+0000","last_undegraded":"2021-03-15T22:34:17.523681+0000","last_fullsized":"2021-03-15T22:34:17.523681+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.883949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.883949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,7],"acting":[5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"856","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.522972+0000","last_change":"2021-03-15T22:34:11.330201+0000","last_active":"2021-03-15T22:34:17.522972+0000","last_peered":"2021-03-15T22:34:17.522972+0000","last_clean":"2021-03-15T22:34:17.522972+0000","last_became_active":"2021-03-15T22:34:10.321886+0000","last_became_peered":"2021-03-15T22:34:10.321886+0000","last_unstale":"2021-03-15T22:34:17.522972+0000","last_undegraded":"2021-03-15T22:34:17.522972+0000","last_fullsized":"2021-03-15T22:34:17.522972+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:32:31.747582+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.747582+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"836'11","reported_seq":"500","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560066+0000","last_change":"2021-03-15T22:33:04.427206+0000","last_active":"2021-03-15T22:34:19.560066+0000","last_peered":"2021-03-15T22:34:19.560066+0000","last_clean":"2021-03-15T22:34:19.560066+0000","last_became_active":"2021-03-15T22:33:04.336294+0000","last_became_peered":"2021-03-15T22:33:04.336294+0000","last_unstale":"2021-03-15T22:34:19.560066+0000","last_undegraded":"2021-03-15T22:34:19.560066+0000","last_fullsized":"2021-03-15T22:34:19.560066+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'9","last_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.971303+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"898'798","reported_seq":"3530","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402192+0000","last_change":"2021-03-15T22:34:13.340100+0000","last_active":"2021-03-15T22:34:21.402192+0000","last_peered":"2021-03-15T22:34:21.402192+0000","last_clean":"2021-03-15T22:34:21.402192+0000","last_became_active":"2021-03-15T22:34:13.328973+0000","last_became_peered":"2021-03-15T22:34:13.328973+0000","last_unstale":"2021-03-15T22:34:21.402192+0000","last_undegraded":"2021-03-15T22:34:21.402192+0000","last_fullsized":"2021-03-15T22:34:21.402192+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'591","last_scrub_stamp":"2021-03-15T22:31:35.474935+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:31:35.474935+0000","log_size":798,"ondisk_log_size":798,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31882229,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":222,"num_read_kb":187438,"num_write":207,"num_write_kb":77841,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":184,"num_bytes_recovered":91383896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"907","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.517995+0000","last_change":"2021-03-15T22:33:45.320646+0000","last_active":"2021-03-15T22:34:17.517995+0000","last_peered":"2021-03-15T22:34:17.517995+0000","last_clean":"2021-03-15T22:34:17.517995+0000","last_became_active":"2021-03-15T22:33:45.320269+0000","last_became_peered":"2021-03-15T22:33:45.320269+0000","last_unstale":"2021-03-15T22:34:17.517995+0000","last_undegraded":"2021-03-15T22:34:17.517995+0000","last_fullsized":"2021-03-15T22:34:17.517995+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.507204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.507204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"369","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.560610+0000","last_change":"2021-03-15T22:33:22.598240+0000","last_active":"2021-03-15T22:34:19.560610+0000","last_peered":"2021-03-15T22:34:19.560610+0000","last_clean":"2021-03-15T22:34:19.560610+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:34:19.560610+0000","last_undegraded":"2021-03-15T22:34:19.560610+0000","last_fullsized":"2021-03-15T22:34:19.560610+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.598174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.598174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"1098","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538007+0000","last_change":"2021-03-15T22:33:43.514160+0000","last_active":"2021-03-15T22:34:20.538007+0000","last_peered":"2021-03-15T22:34:20.538007+0000","last_clean":"2021-03-15T22:34:20.538007+0000","last_became_active":"2021-03-15T22:33:04.336145+0000","last_became_peered":"2021-03-15T22:33:04.336145+0000","last_unstale":"2021-03-15T22:34:20.538007+0000","last_undegraded":"2021-03-15T22:34:20.538007+0000","last_fullsized":"2021-03-15T22:34:20.538007+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:33:43.514088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.514088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"403","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402245+0000","last_change":"2021-03-15T22:33:20.685041+0000","last_active":"2021-03-15T22:34:21.402245+0000","last_peered":"2021-03-15T22:34:21.402245+0000","last_clean":"2021-03-15T22:34:21.402245+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:34:21.402245+0000","last_undegraded":"2021-03-15T22:34:21.402245+0000","last_fullsized":"2021-03-15T22:34:21.402245+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:33:20.684976+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.684976+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"904'1446","reported_seq":"4140","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402860+0000","last_change":"2021-03-15T22:34:13.328362+0000","last_active":"2021-03-15T22:34:21.402860+0000","last_peered":"2021-03-15T22:34:21.402860+0000","last_clean":"2021-03-15T22:34:21.402860+0000","last_became_active":"2021-03-15T22:33:48.311983+0000","last_became_peered":"2021-03-15T22:33:48.311983+0000","last_unstale":"2021-03-15T22:34:21.402860+0000","last_undegraded":"2021-03-15T22:34:21.402860+0000","last_fullsized":"2021-03-15T22:34:21.402860+0000","mapping_epoch":877,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'1298","last_scrub_stamp":"2021-03-15T22:33:09.468592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.468592+0000","log_size":1446,"ondisk_log_size":1446,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":59660836,"num_objects":28,"num_object_clones":24,"num_object_copies":84,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":28,"num_whiteouts":2,"num_read":483,"num_read_kb":496506,"num_write":329,"num_write_kb":117392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":367,"num_bytes_recovered":182140835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"956","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.537822+0000","last_change":"2021-03-15T22:34:10.330961+0000","last_active":"2021-03-15T22:34:20.537822+0000","last_peered":"2021-03-15T22:34:20.537822+0000","last_clean":"2021-03-15T22:34:20.537822+0000","last_became_active":"2021-03-15T22:34:10.330626+0000","last_became_peered":"2021-03-15T22:34:10.330626+0000","last_unstale":"2021-03-15T22:34:20.537822+0000","last_undegraded":"2021-03-15T22:34:20.537822+0000","last_fullsized":"2021-03-15T22:34:20.537822+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,7],"acting":[4,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"685","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.403044+0000","last_change":"2021-03-15T22:31:36.472708+0000","last_active":"2021-03-15T22:34:21.403044+0000","last_peered":"2021-03-15T22:34:21.403044+0000","last_clean":"2021-03-15T22:34:21.403044+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:34:21.403044+0000","last_undegraded":"2021-03-15T22:34:21.403044+0000","last_fullsized":"2021-03-15T22:34:21.403044+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:36.472646+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:31:36.472646+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"332","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.517106+0000","last_change":"2021-03-15T22:34:11.416796+0000","last_active":"2021-03-15T22:34:17.517106+0000","last_peered":"2021-03-15T22:34:17.517106+0000","last_clean":"2021-03-15T22:34:17.517106+0000","last_became_active":"2021-03-15T22:34:10.349731+0000","last_became_peered":"2021-03-15T22:34:10.349731+0000","last_unstale":"2021-03-15T22:34:17.517106+0000","last_undegraded":"2021-03-15T22:34:17.517106+0000","last_fullsized":"2021-03-15T22:34:17.517106+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:32:12.653809+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.653809+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"937","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.537726+0000","last_change":"2021-03-15T22:34:10.327330+0000","last_active":"2021-03-15T22:34:20.537726+0000","last_peered":"2021-03-15T22:34:20.537726+0000","last_clean":"2021-03-15T22:34:20.537726+0000","last_became_active":"2021-03-15T22:34:10.327006+0000","last_became_peered":"2021-03-15T22:34:10.327006+0000","last_unstale":"2021-03-15T22:34:20.537726+0000","last_undegraded":"2021-03-15T22:34:20.537726+0000","last_fullsized":"2021-03-15T22:34:20.537726+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:04.495574+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:31:04.495574+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,0],"acting":[4,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"898'1210","reported_seq":"3992","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.546299+0000","last_change":"2021-03-15T22:34:13.336855+0000","last_active":"2021-03-15T22:34:19.546299+0000","last_peered":"2021-03-15T22:34:19.546299+0000","last_clean":"2021-03-15T22:34:19.546299+0000","last_became_active":"2021-03-15T22:34:13.319391+0000","last_became_peered":"2021-03-15T22:34:13.319391+0000","last_unstale":"2021-03-15T22:34:19.546299+0000","last_undegraded":"2021-03-15T22:34:19.546299+0000","last_fullsized":"2021-03-15T22:34:19.546299+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"872'1183","last_scrub_stamp":"2021-03-15T22:33:43.209215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.209215+0000","log_size":1210,"ondisk_log_size":1210,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33265600,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":340,"num_read_kb":305980,"num_write":268,"num_write_kb":97971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":253,"num_bytes_recovered":97905074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"838","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.523635+0000","last_change":"2021-03-15T22:34:10.318236+0000","last_active":"2021-03-15T22:34:17.523635+0000","last_peered":"2021-03-15T22:34:17.523635+0000","last_clean":"2021-03-15T22:34:17.523635+0000","last_became_active":"2021-03-15T22:34:10.317937+0000","last_became_peered":"2021-03-15T22:34:10.317937+0000","last_unstale":"2021-03-15T22:34:17.523635+0000","last_undegraded":"2021-03-15T22:34:17.523635+0000","last_fullsized":"2021-03-15T22:34:17.523635+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.729838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.729838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,6],"acting":[5,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1182","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538346+0000","last_change":"2021-03-15T22:32:08.071278+0000","last_active":"2021-03-15T22:34:20.538346+0000","last_peered":"2021-03-15T22:34:20.538346+0000","last_clean":"2021-03-15T22:34:20.538346+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:34:20.538346+0000","last_undegraded":"2021-03-15T22:34:20.538346+0000","last_fullsized":"2021-03-15T22:34:20.538346+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:32:08.071210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:08.071210+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"314","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.556637+0000","last_change":"2021-03-15T22:34:10.332407+0000","last_active":"2021-03-15T22:34:21.556637+0000","last_peered":"2021-03-15T22:34:21.556637+0000","last_clean":"2021-03-15T22:34:21.556637+0000","last_became_active":"2021-03-15T22:34:10.332089+0000","last_became_peered":"2021-03-15T22:34:10.332089+0000","last_unstale":"2021-03-15T22:34:21.556637+0000","last_undegraded":"2021-03-15T22:34:21.556637+0000","last_fullsized":"2021-03-15T22:34:21.556637+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.791527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.791527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"64","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.557349+0000","last_change":"2021-03-15T22:34:19.274154+0000","last_active":"2021-03-15T22:34:21.557349+0000","last_peered":"2021-03-15T22:34:21.557349+0000","last_clean":"2021-03-15T22:34:21.557349+0000","last_became_active":"2021-03-15T22:33:45.317888+0000","last_became_peered":"2021-03-15T22:33:45.317888+0000","last_unstale":"2021-03-15T22:34:21.557349+0000","last_undegraded":"2021-03-15T22:34:21.557349+0000","last_fullsized":"2021-03-15T22:34:21.557349+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:32:35.067557+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:35.067557+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"905'386","reported_seq":"2907","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.591190+0000","last_change":"2021-03-15T22:34:13.315820+0000","last_active":"2021-03-15T22:34:20.591190+0000","last_peered":"2021-03-15T22:34:20.591190+0000","last_clean":"2021-03-15T22:34:20.591190+0000","last_became_active":"2021-03-15T22:33:08.354872+0000","last_became_peered":"2021-03-15T22:33:08.354872+0000","last_unstale":"2021-03-15T22:34:20.591190+0000","last_undegraded":"2021-03-15T22:34:20.591190+0000","last_fullsized":"2021-03-15T22:34:20.591190+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'373","last_scrub_stamp":"2021-03-15T22:33:44.219517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.219517+0000","log_size":386,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13032009,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":146,"num_read_kb":162517,"num_write":76,"num_write_kb":31244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":31290769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"577","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.546140+0000","last_change":"2021-03-15T22:34:10.329853+0000","last_active":"2021-03-15T22:34:19.546140+0000","last_peered":"2021-03-15T22:34:19.546140+0000","last_clean":"2021-03-15T22:34:19.546140+0000","last_became_active":"2021-03-15T22:34:10.328556+0000","last_became_peered":"2021-03-15T22:34:10.328556+0000","last_unstale":"2021-03-15T22:34:19.546140+0000","last_undegraded":"2021-03-15T22:34:19.546140+0000","last_fullsized":"2021-03-15T22:34:19.546140+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:09.612126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.612126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,7],"acting":[6,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"938","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.524071+0000","last_change":"2021-03-15T22:33:08.352674+0000","last_active":"2021-03-15T22:34:17.524071+0000","last_peered":"2021-03-15T22:34:17.524071+0000","last_clean":"2021-03-15T22:34:17.524071+0000","last_became_active":"2021-03-15T22:33:08.352367+0000","last_became_peered":"2021-03-15T22:33:08.352367+0000","last_unstale":"2021-03-15T22:34:17.524071+0000","last_undegraded":"2021-03-15T22:34:17.524071+0000","last_fullsized":"2021-03-15T22:34:17.524071+0000","mapping_epoch":837,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'9","last_scrub_stamp":"2021-03-15T22:32:36.868932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.868932+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"906","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.537840+0000","last_change":"2021-03-15T22:33:45.320886+0000","last_active":"2021-03-15T22:34:20.537840+0000","last_peered":"2021-03-15T22:34:20.537840+0000","last_clean":"2021-03-15T22:34:20.537840+0000","last_became_active":"2021-03-15T22:33:45.320470+0000","last_became_peered":"2021-03-15T22:33:45.320470+0000","last_unstale":"2021-03-15T22:34:20.537840+0000","last_undegraded":"2021-03-15T22:34:20.537840+0000","last_fullsized":"2021-03-15T22:34:20.537840+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.461102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"1072","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538288+0000","last_change":"2021-03-15T22:34:10.369701+0000","last_active":"2021-03-15T22:34:20.538288+0000","last_peered":"2021-03-15T22:34:20.538288+0000","last_clean":"2021-03-15T22:34:20.538288+0000","last_became_active":"2021-03-15T22:34:10.330608+0000","last_became_peered":"2021-03-15T22:34:10.330608+0000","last_unstale":"2021-03-15T22:34:20.538288+0000","last_undegraded":"2021-03-15T22:34:20.538288+0000","last_fullsized":"2021-03-15T22:34:20.538288+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:32:38.318687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.318687+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"903'417","reported_seq":"2450","reported_epoch":"903","state":"active+clean","last_fresh":"2021-03-15T22:34:18.526831+0000","last_change":"2021-03-15T22:34:13.334186+0000","last_active":"2021-03-15T22:34:18.526831+0000","last_peered":"2021-03-15T22:34:18.526831+0000","last_clean":"2021-03-15T22:34:18.526831+0000","last_became_active":"2021-03-15T22:33:50.490721+0000","last_became_peered":"2021-03-15T22:33:50.490721+0000","last_unstale":"2021-03-15T22:34:18.526831+0000","last_undegraded":"2021-03-15T22:34:18.526831+0000","last_fullsized":"2021-03-15T22:34:18.526831+0000","mapping_epoch":879,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":880,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'385","last_scrub_stamp":"2021-03-15T22:33:13.869146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:13.869146+0000","log_size":417,"ondisk_log_size":417,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14155152,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":155,"num_read_kb":164362,"num_write":109,"num_write_kb":33883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":74,"num_bytes_recovered":44438274,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"642","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402934+0000","last_change":"2021-03-15T22:33:45.324193+0000","last_active":"2021-03-15T22:34:21.402934+0000","last_peered":"2021-03-15T22:34:21.402934+0000","last_clean":"2021-03-15T22:34:21.402934+0000","last_became_active":"2021-03-15T22:33:45.323862+0000","last_became_peered":"2021-03-15T22:33:45.323862+0000","last_unstale":"2021-03-15T22:34:21.402934+0000","last_undegraded":"2021-03-15T22:34:21.402934+0000","last_fullsized":"2021-03-15T22:34:21.402934+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.958050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.958050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"939","reported_epoch":"906","state":"active+clean","last_fresh":"2021-03-15T22:34:21.556886+0000","last_change":"2021-03-15T22:33:15.406572+0000","last_active":"2021-03-15T22:34:21.556886+0000","last_peered":"2021-03-15T22:34:21.556886+0000","last_clean":"2021-03-15T22:34:21.556886+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:34:21.556886+0000","last_undegraded":"2021-03-15T22:34:21.556886+0000","last_fullsized":"2021-03-15T22:34:21.556886+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.406493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.406493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"93","reported_epoch":"904","state":"active+clean","last_fresh":"2021-03-15T22:34:19.544912+0000","last_change":"2021-03-15T22:34:10.332595+0000","last_active":"2021-03-15T22:34:19.544912+0000","last_peered":"2021-03-15T22:34:19.544912+0000","last_clean":"2021-03-15T22:34:19.544912+0000","last_became_active":"2021-03-15T22:34:10.332236+0000","last_became_peered":"2021-03-15T22:34:10.332236+0000","last_unstale":"2021-03-15T22:34:19.544912+0000","last_undegraded":"2021-03-15T22:34:19.544912+0000","last_fullsized":"2021-03-15T22:34:19.544912+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:08.273379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.273379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"1016","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538788+0000","last_change":"2021-03-15T22:33:45.355215+0000","last_active":"2021-03-15T22:34:20.538788+0000","last_peered":"2021-03-15T22:34:20.538788+0000","last_clean":"2021-03-15T22:34:20.538788+0000","last_became_active":"2021-03-15T22:33:45.315193+0000","last_became_peered":"2021-03-15T22:33:45.315193+0000","last_unstale":"2021-03-15T22:34:20.538788+0000","last_undegraded":"2021-03-15T22:34:20.538788+0000","last_fullsized":"2021-03-15T22:34:20.538788+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:32:43.307660+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.307660+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":734,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"904'1425","reported_seq":"4238","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.572909+0000","last_change":"2021-03-15T22:34:13.333356+0000","last_active":"2021-03-15T22:34:20.572909+0000","last_peered":"2021-03-15T22:34:20.572909+0000","last_clean":"2021-03-15T22:34:20.572909+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:34:20.572909+0000","last_undegraded":"2021-03-15T22:34:20.572909+0000","last_fullsized":"2021-03-15T22:34:20.572909+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"837'1315","last_scrub_stamp":"2021-03-15T22:33:08.608669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.608669+0000","log_size":1425,"ondisk_log_size":1425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50504572,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":479,"num_read_kb":491166,"num_write":340,"num_write_kb":122187,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"291","length":"1"},{"start":"2ab","length":"1"},{"start":"2cb","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"907","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538405+0000","last_change":"2021-03-15T22:34:10.326535+0000","last_active":"2021-03-15T22:34:20.538405+0000","last_peered":"2021-03-15T22:34:20.538405+0000","last_clean":"2021-03-15T22:34:20.538405+0000","last_became_active":"2021-03-15T22:34:10.326211+0000","last_became_peered":"2021-03-15T22:34:10.326211+0000","last_unstale":"2021-03-15T22:34:20.538405+0000","last_undegraded":"2021-03-15T22:34:20.538405+0000","last_fullsized":"2021-03-15T22:34:20.538405+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:20.500979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.500979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"881'16","reported_seq":"344","reported_epoch":"902","state":"active+clean","last_fresh":"2021-03-15T22:34:17.518580+0000","last_change":"2021-03-15T22:33:48.308057+0000","last_active":"2021-03-15T22:34:17.518580+0000","last_peered":"2021-03-15T22:34:17.518580+0000","last_clean":"2021-03-15T22:34:17.518580+0000","last_became_active":"2021-03-15T22:33:48.307475+0000","last_became_peered":"2021-03-15T22:33:48.307475+0000","last_unstale":"2021-03-15T22:34:17.518580+0000","last_undegraded":"2021-03-15T22:34:17.518580+0000","last_fullsized":"2021-03-15T22:34:17.518580+0000","mapping_epoch":877,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'13","last_scrub_stamp":"2021-03-15T22:33:37.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.949876+0000","log_size":14,"ondisk_log_size":14,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"335","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:21.402452+0000","last_change":"2021-03-15T22:33:21.690369+0000","last_active":"2021-03-15T22:34:21.402452+0000","last_peered":"2021-03-15T22:34:21.402452+0000","last_clean":"2021-03-15T22:34:21.402452+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:34:21.402452+0000","last_undegraded":"2021-03-15T22:34:21.402452+0000","last_fullsized":"2021-03-15T22:34:21.402452+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.690309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"1088","reported_epoch":"905","state":"active+clean","last_fresh":"2021-03-15T22:34:20.538552+0000","last_change":"2021-03-15T22:33:05.216013+0000","last_active":"2021-03-15T22:34:20.538552+0000","last_peered":"2021-03-15T22:34:20.538552+0000","last_clean":"2021-03-15T22:34:20.538552+0000","last_became_active":"2021-03-15T22:33:04.337113+0000","last_became_peered":"2021-03-15T22:33:04.337113+0000","last_unstale":"2021-03-15T22:34:20.538552+0000","last_undegraded":"2021-03-15T22:34:20.538552+0000","last_fullsized":"2021-03-15T22:34:20.538552+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:32:32.328526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.328526+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":717395162,"num_objects":328,"num_object_clones":279,"num_object_copies":984,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":42,"num_objects_unfound":0,"num_objects_dirty":328,"num_whiteouts":13,"num_read":6317,"num_read_kb":6458645,"num_write":4637,"num_write_kb":1776238,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3606,"num_bytes_recovered":1723069743,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1717829632,"data_stored":2376571454,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":19350,"ondisk_log_size":19350,"up":48,"acting":47,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":969,"num_read_kb":964,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2646,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1482,"num_read_kb":987,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1141,"num_bytes_recovered":87398,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2166784,"data_stored":77680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":947,"ondisk_log_size":947,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":131,"ondisk_log_size":131,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299593,"num_pgs":74,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":264976,"kb_used_data":209288,"kb_used_omap":0,"kb_used_meta":55680,"kb_avail":93480176,"statfs":{"total":95995035648,"available":95723700224,"internally_reserved":0,"allocated":214310912,"data_stored":312089079,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57016320},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:33:20 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.44400000000000001,"15min":0.47399999999999998},"min":{"1min":0.316,"5min":0.216,"15min":0.215},"max":{"1min":0.59999999999999998,"5min":0.70099999999999996,"15min":2.73},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.48499999999999999,"5min":0.48399999999999999,"15min":0.49399999999999999},"min":{"1min":0.32600000000000001,"5min":0.251,"15min":0.222},"max":{"1min":0.629,"5min":0.75800000000000001,"15min":3.8799999999999999},"last":0.45100000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:33:20 2021","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44700000000000001,"15min":0.47799999999999998},"min":{"1min":0.249,"5min":0.249,"15min":0.22600000000000001},"max":{"1min":0.53500000000000003,"5min":0.73899999999999999,"15min":2.6859999999999999},"last":0.46899999999999997},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.48499999999999999,"15min":0.49399999999999999},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.23400000000000001},"max":{"1min":0.72199999999999998,"5min":0.81899999999999995,"15min":2.0779999999999998},"last":0.56299999999999994}]},{"osd":2,"last update":"Mon Mar 15 22:33:37 2021","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.502,"15min":0.48299999999999998},"min":{"1min":0.42999999999999999,"5min":0.33800000000000002,"15min":0.33800000000000002},"max":{"1min":0.65500000000000003,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.48599999999999999,"5min":0.502,"15min":0.502},"min":{"1min":0.307,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.67000000000000004,"5min":1.119,"15min":1.119},"last":0.61799999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:33:20 2021","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.49199999999999999,"15min":0.503},"min":{"1min":0.34599999999999997,"5min":0.28899999999999998,"15min":0.28899999999999998},"max":{"1min":0.68799999999999994,"5min":1.0249999999999999,"15min":2.0339999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.47499999999999998,"15min":0.53300000000000003},"min":{"1min":0.41299999999999998,"5min":0.314,"15min":0.27300000000000002},"max":{"1min":0.78300000000000003,"5min":0.78300000000000003,"15min":5.415},"last":0.51400000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:33:20 2021","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.42299999999999999,"15min":0.44600000000000001},"min":{"1min":0.28599999999999998,"5min":0.246,"15min":0.23799999999999999},"max":{"1min":0.66200000000000003,"5min":0.753,"15min":1.927},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.44800000000000001,"15min":0.46500000000000002},"min":{"1min":0.29699999999999999,"5min":0.253,"15min":0.253},"max":{"1min":0.621,"5min":0.68200000000000005,"15min":2.5339999999999998},"last":0.41399999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:33:37 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.47499999999999998,"15min":0.45300000000000001},"min":{"1min":0.32100000000000001,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.69599999999999995,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.51500000000000001,"15min":0.50700000000000001},"min":{"1min":0.38400000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.77700000000000002,"5min":1.071,"15min":1.071},"last":0.34200000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:33:20 2021","interfaces":[{"interface":"back","average":{"1min":0.49399999999999999,"5min":0.47999999999999998,"15min":0.51800000000000002},"min":{"1min":0.35399999999999998,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.70699999999999996,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.36199999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.50800000000000001,"15min":0.52200000000000002},"min":{"1min":0.34399999999999997,"5min":0.28499999999999998,"15min":0.28399999999999997},"max":{"1min":0.73899999999999999,"5min":1.097,"15min":1.8560000000000001},"last":0.38500000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528525,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":228196,"kb_used_data":187868,"kb_used_omap":0,"kb_used_meta":40320,"kb_avail":93516956,"statfs":{"total":95995035648,"available":95761362944,"internally_reserved":0,"allocated":192376832,"data_stored":241371129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41287680},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:05 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47099999999999997,"15min":0.48399999999999999},"min":{"1min":0.30399999999999999,"5min":0.251,"15min":0.24299999999999999},"max":{"1min":0.70799999999999996,"5min":2.5470000000000002,"15min":3.085},"last":0.623},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46700000000000003,"15min":0.46800000000000003},"min":{"1min":0.35299999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.64100000000000001,"5min":1.137,"15min":1.323},"last":0.51900000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:34:05 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53600000000000003,"15min":0.55500000000000005},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.254},"max":{"1min":0.68200000000000005,"5min":2.7839999999999998,"15min":14.637},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.44400000000000001,"15min":0.47999999999999998},"min":{"1min":0.35799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73999999999999999,"5min":0.84899999999999998,"15min":5.2450000000000001},"last":0.48399999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:33:40 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.34899999999999998,"5min":0.312,"15min":0.312},"max":{"1min":0.64700000000000002,"5min":0.83099999999999996,"15min":0.83099999999999996},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.61199999999999999,"5min":0.53600000000000003,"15min":0.51200000000000001},"min":{"1min":0.33300000000000002,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":2.2210000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.63300000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:34:05 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49099999999999999,"15min":0.50900000000000001},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.29899999999999999},"max":{"1min":0.58399999999999996,"5min":1.0609999999999999,"15min":2.6150000000000002},"last":0.68999999999999995},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.49199999999999999,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.316,"15min":0.28699999999999998},"max":{"1min":0.69199999999999995,"5min":2.4670000000000001,"15min":2.7890000000000001},"last":0.51300000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:34:05 2021","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.432,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.68600000000000005,"5min":0.94099999999999995,"15min":4.306},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.498,"5min":0.46999999999999997,"15min":0.495},"min":{"1min":0.28799999999999998,"5min":0.252,"15min":0.23499999999999999},"max":{"1min":0.67800000000000005,"5min":1.091,"15min":4.181},"last":0.33600000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:33:33 2021","interfaces":[{"interface":"back","average":{"1min":0.52500000000000002,"5min":0.46200000000000002,"15min":0.42199999999999999},"min":{"1min":0.35399999999999998,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.89300000000000002,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.439},{"interface":"front","average":{"1min":0.57499999999999996,"5min":0.51700000000000002,"15min":0.46600000000000003},"min":{"1min":0.317,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.98099999999999998,"5min":1.022,"15min":1.022},"last":0.55300000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:34:19 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47399999999999998,"15min":0.53900000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.249},"max":{"1min":0.72299999999999998,"5min":0.76300000000000001,"15min":15.340999999999999},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55500000000000005,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.28999999999999998,"15min":0.27200000000000002},"max":{"1min":0.65500000000000003,"5min":1.0109999999999999,"15min":15.356},"last":0.46999999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575071,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":329660,"kb_used_data":274804,"kb_used_omap":0,"kb_used_meta":54848,"kb_avail":93415492,"statfs":{"total":95995035648,"available":95657463808,"internally_reserved":0,"allocated":281399296,"data_stored":374530368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":56164352},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:33:48 2021","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.32300000000000001,"15min":0.501},"min":{"1min":0.23699999999999999,"5min":0.184,"15min":0.17599999999999999},"max":{"1min":0.52000000000000002,"5min":1.325,"15min":54.427999999999997},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.35999999999999999,"5min":0.36899999999999999,"15min":0.52900000000000003},"min":{"1min":0.254,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.56299999999999994,"5min":3.3500000000000001,"15min":54.356999999999999},"last":0.32000000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:33:40 2021","interfaces":[{"interface":"back","average":{"1min":0.34100000000000003,"5min":0.38,"15min":0.34899999999999998},"min":{"1min":0.26400000000000001,"5min":0.248,"15min":0.248},"max":{"1min":0.52000000000000002,"5min":4.1509999999999998,"15min":4.1509999999999998},"last":0.26100000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.39300000000000002,"15min":0.38},"min":{"1min":0.27500000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.53200000000000003,"5min":1.891,"15min":1.891},"last":0.35499999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:34:17 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.374,"15min":0.36899999999999999},"min":{"1min":0.28100000000000003,"5min":0.23100000000000001,"15min":0.215},"max":{"1min":0.501,"5min":1.647,"15min":1.647},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.42699999999999999,"15min":0.40200000000000002},"min":{"1min":0.26000000000000001,"5min":0.246,"15min":0.20999999999999999},"max":{"1min":0.61699999999999999,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.28399999999999997}]},{"osd":4,"last update":"Mon Mar 15 22:34:21 2021","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.60699999999999998,"15min":0.55300000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.22800000000000001},"max":{"1min":4.593,"5min":4.593,"15min":4.593},"last":0.375},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53900000000000003,"15min":0.56399999999999995},"min":{"1min":0.35499999999999998,"5min":0.317,"15min":0.253},"max":{"1min":0.874,"5min":1.284,"15min":4.6139999999999999},"last":0.51600000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:33:30 2021","interfaces":[{"interface":"back","average":{"1min":0.56100000000000005,"5min":0.55200000000000005,"15min":0.53300000000000003},"min":{"1min":0.34899999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.89700000000000002,"5min":1.226,"15min":1.226},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.60099999999999998,"5min":0.58199999999999996,"15min":0.52000000000000002},"min":{"1min":0.47399999999999998,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.2130000000000001,"5min":3.472,"15min":3.472},"last":0.40600000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:33:58 2021","interfaces":[{"interface":"back","average":{"1min":0.64600000000000002,"5min":0.59199999999999997,"15min":0.58399999999999996},"min":{"1min":0.47999999999999998,"5min":0.35499999999999998,"15min":0.314},"max":{"1min":1.2509999999999999,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.441},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.61799999999999999,"15min":0.58399999999999996},"min":{"1min":0.40500000000000003,"5min":0.35099999999999998,"15min":0.33700000000000002},"max":{"1min":4.6529999999999996,"5min":4.6529999999999996,"15min":4.6529999999999996},"last":0.48599999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:34:17 2021","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.60899999999999999,"15min":0.59899999999999998},"min":{"1min":0.42099999999999999,"5min":0.376,"15min":0.316},"max":{"1min":1.476,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.64500000000000002,"15min":0.59099999999999997},"min":{"1min":0.45600000000000002,"5min":0.38200000000000001,"15min":0.35799999999999998},"max":{"1min":4.625,"5min":4.625,"15min":4.625},"last":0.52900000000000003}]}]},{"osd":0,"up_from":8,"seq":34359738593,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":299644,"kb_used_data":251316,"kb_used_omap":0,"kb_used_meta":48320,"kb_avail":93445508,"statfs":{"total":95995035648,"available":95688200192,"internally_reserved":0,"allocated":257347584,"data_stored":376383000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":49479680},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:33:42 2021","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.33300000000000002,"15min":0.33600000000000002},"min":{"1min":0.23699999999999999,"5min":0.192,"15min":0.17999999999999999},"max":{"1min":0.50600000000000001,"5min":1.46,"15min":1.7070000000000001},"last":0.55400000000000005},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.36899999999999999,"15min":0.36399999999999999},"min":{"1min":0.22800000000000001,"5min":0.22,"15min":0.19900000000000001},"max":{"1min":0.54400000000000004,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.496}]},{"osd":2,"last update":"Mon Mar 15 22:33:37 2021","interfaces":[{"interface":"back","average":{"1min":0.34999999999999998,"5min":0.373,"15min":0.36399999999999999},"min":{"1min":0.24399999999999999,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.56200000000000006,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.40100000000000002,"15min":0.38400000000000001},"min":{"1min":0.252,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.59299999999999997,"5min":2.5390000000000001,"15min":2.5390000000000001},"last":0.38700000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:33:19 2021","interfaces":[{"interface":"back","average":{"1min":0.39700000000000002,"5min":0.39400000000000002,"15min":0.38400000000000001},"min":{"1min":0.26900000000000002,"5min":0.23000000000000001,"15min":0.22700000000000001},"max":{"1min":0.72999999999999998,"5min":1.544,"15min":1.544},"last":0.30499999999999999},{"interface":"front","average":{"1min":0.39900000000000002,"5min":0.41299999999999998,"15min":0.40300000000000002},"min":{"1min":0.26800000000000002,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.80200000000000005,"5min":2.589,"15min":2.589},"last":0.32700000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:33:19 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.53700000000000003,"15min":0.54100000000000004},"min":{"1min":0.38200000000000001,"5min":0.27400000000000002,"15min":0.254},"max":{"1min":0.748,"5min":1.861,"15min":1.861},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.57399999999999995,"5min":0.55800000000000005,"15min":0.56999999999999995},"min":{"1min":0.39500000000000002,"5min":0.28899999999999998,"15min":0.28799999999999998},"max":{"1min":0.79100000000000004,"5min":1.5700000000000001,"15min":1.645},"last":0.53500000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:33:31 2021","interfaces":[{"interface":"back","average":{"1min":0.55400000000000005,"5min":0.55900000000000005,"15min":0.51900000000000002},"min":{"1min":0.36899999999999999,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.77800000000000002,"5min":2.6440000000000001,"15min":2.6440000000000001},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.53700000000000003,"15min":0.49399999999999999},"min":{"1min":0.41899999999999998,"5min":0.318,"15min":0.318},"max":{"1min":0.77900000000000003,"5min":1.909,"15min":1.909},"last":0.50700000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:33:57 2021","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.53800000000000003,"15min":0.55400000000000005},"min":{"1min":0.38200000000000001,"5min":0.30299999999999999,"15min":0.26200000000000001},"max":{"1min":0.77000000000000002,"5min":1.823,"15min":1.823},"last":0.441},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.56899999999999995,"15min":0.57699999999999996},"min":{"1min":0.43099999999999999,"5min":0.27100000000000002,"15min":0.27100000000000002},"max":{"1min":0.754,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.56999999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:33:19 2021","interfaces":[{"interface":"back","average":{"1min":0.61599999999999999,"5min":0.57499999999999996,"15min":0.57999999999999996},"min":{"1min":0.45400000000000001,"5min":0.34100000000000003,"15min":0.27900000000000003},"max":{"1min":0.83699999999999997,"5min":2.6190000000000002,"15min":2.6190000000000002},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58999999999999997,"5min":0.55900000000000005,"15min":0.57799999999999996},"min":{"1min":0.42899999999999999,"5min":0.35199999999999998,"15min":0.317},"max":{"1min":0.81100000000000005,"5min":1.8859999999999999,"15min":1.8859999999999999},"last":0.46999999999999997}]}]},{"osd":2,"up_from":0,"seq":2924872728623,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":3,"up_from":26,"seq":111669149911,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":330124,"kb_used_data":260548,"kb_used_omap":0,"kb_used_meta":69568,"kb_avail":93415028,"statfs":{"total":95995035648,"available":95656988672,"internally_reserved":0,"allocated":266801152,"data_stored":355425261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":71237632},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:19 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.34599999999999997,"15min":0.33800000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17699999999999999},"max":{"1min":2.7509999999999999,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.32500000000000001,"15min":0.33500000000000002},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.189},"max":{"1min":0.76900000000000002,"5min":1.419,"15min":2.6339999999999999},"last":0.29299999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:34:19 2021","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36599999999999999,"15min":0.39200000000000002},"min":{"1min":0.249,"5min":0.23599999999999999,"15min":0.23000000000000001},"max":{"1min":0.59999999999999998,"5min":0.72599999999999998,"15min":6.0199999999999996},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.36299999999999999,"15min":0.40000000000000002},"min":{"1min":0.253,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":7.6130000000000004},"last":0.27200000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:33:37 2021","interfaces":[{"interface":"back","average":{"1min":0.34200000000000003,"5min":0.35399999999999998,"15min":0.35499999999999998},"min":{"1min":0.23699999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.54400000000000004,"5min":0.54400000000000004,"15min":0.54400000000000004},"last":0.435},{"interface":"front","average":{"1min":0.38600000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.377}]},{"osd":4,"last update":"Mon Mar 15 22:33:28 2021","interfaces":[{"interface":"back","average":{"1min":0.56000000000000005,"5min":0.53400000000000003,"15min":0.54100000000000004},"min":{"1min":0.31900000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.78500000000000003,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.59399999999999997,"5min":0.53000000000000003,"15min":0.53300000000000003},"min":{"1min":0.371,"5min":0.28499999999999998,"15min":0.26700000000000002},"max":{"1min":0.82799999999999996,"5min":2.1059999999999999,"15min":2.2890000000000001},"last":0.45700000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:33:28 2021","interfaces":[{"interface":"back","average":{"1min":0.57399999999999995,"5min":0.67600000000000005,"15min":0.80200000000000005},"min":{"1min":0.36299999999999999,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":0.58299999999999996},{"interface":"front","average":{"1min":0.58599999999999997,"5min":0.79900000000000004,"15min":1.0449999999999999},"min":{"1min":0.40999999999999998,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.84799999999999998,"5min":15.776,"15min":15.776},"last":0.51700000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:33:58 2021","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.56499999999999995,"15min":0.56999999999999995},"min":{"1min":0.41599999999999998,"5min":0.30499999999999999,"15min":0.30099999999999999},"max":{"1min":1.359,"5min":2.5299999999999998,"15min":2.6499999999999999},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.57299999999999995,"15min":0.56399999999999995},"min":{"1min":0.38800000000000001,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":1.3400000000000001,"5min":3.472,"15min":3.472},"last":0.56699999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:34:19 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.34799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69899999999999995,"5min":5.625,"15min":5.625},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.58199999999999996,"15min":0.56599999999999995},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35499999999999998},"max":{"1min":2.621,"5min":2.621,"15min":2.621},"last":0.48599999999999999}]}]},{"osd":4,"up_from":31,"seq":133143986388,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":180876,"kb_used_data":143364,"kb_used_omap":0,"kb_used_meta":37504,"kb_avail":93564276,"statfs":{"total":95995035648,"available":95809818624,"internally_reserved":0,"allocated":146804736,"data_stored":213833486,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38404096},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:33:31 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.5,"15min":0.47999999999999998},"min":{"1min":0.25600000000000001,"5min":0.20000000000000001,"15min":0.19500000000000001},"max":{"1min":0.67300000000000004,"5min":4.7050000000000001,"15min":4.7050000000000001},"last":0.56499999999999995},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.51000000000000001,"15min":0.503},"min":{"1min":0.33600000000000002,"5min":0.246,"15min":0.246},"max":{"1min":1.4179999999999999,"5min":5.0209999999999999,"15min":5.0209999999999999},"last":0.47199999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:33:31 2021","interfaces":[{"interface":"back","average":{"1min":0.47699999999999998,"5min":0.46800000000000003,"15min":0.48199999999999998},"min":{"1min":0.25,"5min":0.23999999999999999,"15min":0.23599999999999999},"max":{"1min":0.73299999999999998,"5min":1.3720000000000001,"15min":4.3499999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.49399999999999999,"5min":0.48499999999999999,"15min":0.505},"min":{"1min":0.28399999999999997,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.94599999999999995,"5min":1.494,"15min":4.718},"last":0.48699999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:33:35 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.504,"15min":0.51000000000000001},"min":{"1min":0.27000000000000002,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.748,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.49099999999999999,"15min":0.497},"min":{"1min":0.35699999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.621,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.497}]},{"osd":3,"last update":"Mon Mar 15 22:33:31 2021","interfaces":[{"interface":"back","average":{"1min":0.54300000000000004,"5min":0.55800000000000005,"15min":0.54900000000000004},"min":{"1min":0.377,"5min":0.28499999999999998,"15min":0.28499999999999998},"max":{"1min":1.486,"5min":5.0499999999999998,"15min":5.0499999999999998},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.53700000000000003,"15min":0.54000000000000004},"min":{"1min":0.34100000000000003,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":1.4950000000000001,"5min":1.4950000000000001,"15min":4.4050000000000002},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:33:31 2021","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.41899999999999998,"15min":0.40799999999999997},"min":{"1min":0.28000000000000003,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.56399999999999995,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.22900000000000001},{"interface":"front","average":{"1min":0.44800000000000001,"5min":0.45500000000000002,"15min":0.42899999999999999},"min":{"1min":0.33900000000000002,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.69099999999999995,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.31}]},{"osd":6,"last update":"Mon Mar 15 22:33:57 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.46500000000000002,"15min":0.46700000000000003},"min":{"1min":0.38100000000000001,"5min":0.24199999999999999,"15min":0.22600000000000001},"max":{"1min":0.73399999999999999,"5min":0.88,"15min":4.3079999999999998},"last":0.247},{"interface":"front","average":{"1min":0.56599999999999995,"5min":0.53500000000000003,"15min":0.54100000000000004},"min":{"1min":0.435,"5min":0.29999999999999999,"15min":0.28000000000000003},"max":{"1min":0.71999999999999997,"5min":0.83799999999999997,"15min":4.6600000000000001},"last":0.33000000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:34:15 2021","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.497,"15min":0.51000000000000001},"min":{"1min":0.35099999999999998,"5min":0.29399999999999998,"15min":0.26300000000000001},"max":{"1min":0.66500000000000004,"5min":0.80900000000000005,"15min":4.6929999999999996},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51200000000000001,"15min":0.54700000000000004},"min":{"1min":0.35499999999999998,"5min":0.32400000000000001,"15min":0.25},"max":{"1min":0.65800000000000003,"5min":1.034,"15min":4.9900000000000002},"last":0.29999999999999999}]}]},{"osd":5,"up_from":616,"seq":2645699854396,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321228,"kb_used_data":289732,"kb_used_omap":153,"kb_used_meta":31334,"kb_avail":93423924,"statfs":{"total":95995035648,"available":95666098176,"internally_reserved":0,"allocated":296685568,"data_stored":387701088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":32086545},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:33:37 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.48499999999999999,"15min":0.46300000000000002},"min":{"1min":0.25800000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.2130000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":1.0780000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.48099999999999998,"15min":0.45500000000000002},"min":{"1min":0.28000000000000003,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":2.5030000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":1.002}]},{"osd":1,"last update":"Mon Mar 15 22:33:37 2021","interfaces":[{"interface":"back","average":{"1min":0.90200000000000002,"5min":0.55600000000000005,"15min":0.48199999999999998},"min":{"1min":0.34100000000000003,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":6.4359999999999999,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":1.046},{"interface":"front","average":{"1min":0.5,"5min":0.496,"15min":0.48899999999999999},"min":{"1min":0.34100000000000003,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":0.71099999999999997,"5min":0.876,"15min":0.876},"last":1.161}]},{"osd":2,"last update":"Mon Mar 15 22:33:37 2021","interfaces":[{"interface":"back","average":{"1min":1.026,"5min":0.59999999999999998,"15min":0.54200000000000004},"min":{"1min":0.32000000000000001,"5min":0.29099999999999998,"15min":0.29099999999999998},"max":{"1min":10.952,"5min":10.952,"15min":10.952},"last":1.121},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.52500000000000002,"15min":0.53000000000000003},"min":{"1min":0.33600000000000002,"5min":0.33100000000000002,"15min":0.33100000000000002},"max":{"1min":1.125,"5min":1.125,"15min":1.125},"last":1.103}]},{"osd":3,"last update":"Mon Mar 15 22:33:37 2021","interfaces":[{"interface":"back","average":{"1min":1.371,"5min":0.67300000000000004,"15min":0.53700000000000003},"min":{"1min":0.32500000000000001,"5min":0.32500000000000001,"15min":0.32500000000000001},"max":{"1min":10.355,"5min":10.355,"15min":10.355},"last":1.1379999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.48799999999999999,"15min":0.45100000000000001},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":2.476,"5min":2.476,"15min":2.476},"last":1.1839999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:33:37 2021","interfaces":[{"interface":"back","average":{"1min":0.80900000000000005,"5min":0.48999999999999999,"15min":0.40799999999999997},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":7.4000000000000004,"5min":7.4000000000000004,"15min":7.4000000000000004},"last":1.204},{"interface":"front","average":{"1min":0.56699999999999995,"5min":0.44600000000000001,"15min":0.39900000000000002},"min":{"1min":0.30599999999999999,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":2.4100000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.38800000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:33:37 2021","interfaces":[{"interface":"back","average":{"1min":0.67100000000000004,"5min":0.503,"15min":0.46300000000000002},"min":{"1min":0.318,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":3.9060000000000001,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.98899999999999999},{"interface":"front","average":{"1min":0.71699999999999997,"5min":0.53800000000000003,"15min":0.48399999999999999},"min":{"1min":0.377,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":4.0190000000000001,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":1.0860000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:33:37 2021","interfaces":[{"interface":"back","average":{"1min":0.64000000000000001,"5min":0.496,"15min":0.45200000000000001},"min":{"1min":0.34100000000000003,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":2.5699999999999998,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":1.0169999999999999},{"interface":"front","average":{"1min":0.81299999999999994,"5min":0.56799999999999995,"15min":0.51000000000000001},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":5.944,"5min":5.944,"15min":5.944},"last":1.0269999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":9520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":360448,"data_stored":12916,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":200704,"data_stored":7108,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":217088,"data_stored":7866,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":278528,"data_stored":10020,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":225280,"data_stored":8092,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":12166,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":9992,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":257736704,"data_stored":379264270,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":280215552,"data_stored":373964035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":63741952,"data_stored":107883808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":267259904,"data_stored":356334695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":145694720,"data_stored":213269988,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":295673856,"data_stored":387164060,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":194265088,"data_stored":247132065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":213241856,"data_stored":311558533,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:34:24.897 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-03-15T22:34:24.897 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool set default.rgw.meta pg_num 42 2021-03-15T22:34:24.925 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:24.977 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:24.998 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:24.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:25.000 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:25.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:24 smithi125 conmon[56739]: cluster 2021-03-15T22:34:23.203868+0000 mgr.y (mgr.14144) 1029 : cluster 2021-03-15T22:34:25.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:24 smithi125 conmon[56739]: [DBG] pgmap v1508: 148 pgs: 1 active+remapped+backfill_toofull, 1 peering, 1 active+recovering+undersized+degraded+remapped, 145 active+clean; 684 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 4.8 MiB/s rd, 5.2 MiB/s wr, 23 op/s; 8/1929 objects degraded (0.415%); 42/1929 objects misplaced (2.177%); 10 MiB/s, 23 objects/s recovering 2021-03-15T22:34:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:24 smithi125 conmon[56739]: cluster 2021-03-15T22:34:23.561748+0000 mon.a (mon.0) 2299 : cluster [DBG] osdmap e908: 8 total, 8 up, 7 in 2021-03-15T22:34:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:24 smithi125 conmon[56739]: audit 2021-03-15T22:34:23.631400+0000 mgr.y (mgr.14144) 2021-03-15T22:34:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:24 smithi125 conmon[56739]: 1030 : audit [DBG] from='client.16671 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:34:25.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:25.124 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:25.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:25.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:25.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:25.248 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:25.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:25.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:25.351 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:25.373 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:25.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:25.374 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:25.475 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:25.497 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:25.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:25.499 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:25.576 INFO:tasks.rados.rados.0.smithi076.stdout:3417: done (7 left) 2021-03-15T22:34:25.576 INFO:tasks.rados.rados.0.smithi076.stdout:3418: done (6 left) 2021-03-15T22:34:25.576 INFO:tasks.rados.rados.0.smithi076.stdout:3419: done (5 left) 2021-03-15T22:34:25.576 INFO:tasks.rados.rados.0.smithi076.stdout:3420: done (4 left) 2021-03-15T22:34:25.577 INFO:tasks.rados.rados.0.smithi076.stdout:3421: done (3 left) 2021-03-15T22:34:25.577 INFO:tasks.rados.rados.0.smithi076.stdout:3422: done (2 left) 2021-03-15T22:34:25.577 INFO:tasks.rados.rados.0.smithi076.stdout:3423: done (1 left) 2021-03-15T22:34:25.577 INFO:tasks.rados.rados.0.smithi076.stdout:3424: done (0 left) 2021-03-15T22:34:25.578 INFO:tasks.rados.rados.0.smithi076.stdout:3425: read oid 49 snap -1 2021-03-15T22:34:25.578 INFO:tasks.rados.rados.0.smithi076.stdout:3425: expect (ObjNum 1124 snap 389 seq_num 1124) 2021-03-15T22:34:25.578 INFO:tasks.rados.rados.0.smithi076.stdout:3426: write oid 15 current snap is 408 2021-03-15T22:34:25.579 INFO:tasks.rados.rados.0.smithi076.stdout:3426: seq_num 1162 ranges {729734=724994,2197488=430392,3227047=302687} 2021-03-15T22:34:25.584 INFO:tasks.rados.rados.0.smithi076.stdout:3426: writing smithi07694878-15 from 729734 to 1454728 tid 1 2021-03-15T22:34:25.590 INFO:tasks.rados.rados.0.smithi076.stdout:3426: writing smithi07694878-15 from 2197488 to 2627880 tid 2 2021-03-15T22:34:25.595 INFO:tasks.rados.rados.0.smithi076.stdout:3426: writing smithi07694878-15 from 3227047 to 3529734 tid 3 2021-03-15T22:34:25.595 INFO:tasks.rados.rados.0.smithi076.stdout:3427: copy_from oid 23 from oid 44 current snap is 408 2021-03-15T22:34:25.596 INFO:tasks.rados.rados.0.smithi076.stdout:3425: expect (ObjNum 1124 snap 389 seq_num 1124) 2021-03-15T22:34:25.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:25.622 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:25.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:25.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:25.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:25 smithi076 conmon[56514]: cluster 2021-03-15T22:34:24.564521+0000 mon.a 2021-03-15T22:34:25.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:25 smithi076 conmon[56514]: (mon.0) 2300 : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2021-03-15T22:34:25.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:25 smithi076 conmon[62156]: cluster 2021-03-15T22:34:24.564521 2021-03-15T22:34:25.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:25 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2021-03-15T22:34:25.724 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:25.746 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:25.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:25.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:25.813 INFO:tasks.rados.rados.0.smithi076.stdout:3426: finishing write tid 1 to smithi07694878-15 2021-03-15T22:34:25.813 INFO:tasks.rados.rados.0.smithi076.stdout:3426: finishing write tid 2 to smithi07694878-15 2021-03-15T22:34:25.814 INFO:tasks.rados.rados.0.smithi076.stdout:3426: finishing write tid 3 to smithi07694878-15 2021-03-15T22:34:25.814 INFO:tasks.rados.rados.0.smithi076.stdout:3426: finishing write tid 4 to smithi07694878-15 2021-03-15T22:34:25.814 INFO:tasks.rados.rados.0.smithi076.stdout:3426: finishing write tid 5 to smithi07694878-15 2021-03-15T22:34:25.814 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 2794 (ObjNum 1162 snap 408 seq_num 1162) dirty exists 2021-03-15T22:34:25.815 INFO:tasks.rados.rados.0.smithi076.stdout:3426: left oid 15 (ObjNum 1162 snap 408 seq_num 1162) 2021-03-15T22:34:25.815 INFO:tasks.rados.rados.0.smithi076.stdout:3427: finishing copy_from to smithi07694878-23 2021-03-15T22:34:25.815 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 2413 (ObjNum 829 snap 275 seq_num 829) dirty exists 2021-03-15T22:34:25.815 INFO:tasks.rados.rados.0.smithi076.stdout:3427: finishing copy_from racing read to smithi07694878-23 2021-03-15T22:34:25.816 INFO:tasks.rados.rados.0.smithi076.stdout:3425: done (2 left) 2021-03-15T22:34:25.816 INFO:tasks.rados.rados.0.smithi076.stdout:3426: done (1 left) 2021-03-15T22:34:25.816 INFO:tasks.rados.rados.0.smithi076.stdout:3427: done (0 left) 2021-03-15T22:34:25.816 INFO:tasks.rados.rados.0.smithi076.stdout:3428: rollback oid 45 current snap is 408 2021-03-15T22:34:25.816 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 45 to 401 2021-03-15T22:34:25.817 INFO:tasks.rados.rados.0.smithi076.stdout:3429: read oid 23 snap 388 2021-03-15T22:34:25.817 INFO:tasks.rados.rados.0.smithi076.stdout:3429: expect deleted 2021-03-15T22:34:25.817 INFO:tasks.rados.rados.0.smithi076.stdout:3430: snap_remove snap 320 2021-03-15T22:34:25.849 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:25.872 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:25.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:25.873 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:25.974 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:25.987 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:34:25.987 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:34:25.996 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:25.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:25.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:26.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:25 smithi125 conmon[56739]: cluster 2021-03-15T22:34:24. 2021-03-15T22:34:26.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:25 smithi125 conmon[56739]: 564521+0000 mon.a (mon.0) 2300 : cluster [DBG] osdmap e909: 8 total, 8 up, 7 in 2021-03-15T22:34:26.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:26.122 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:26.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:26.123 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:26.224 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:26.248 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:26.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:26.249 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:26.350 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:26.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:26.374 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:26.374 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:26.475 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:26.499 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:26.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:26.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:26.588 INFO:tasks.rados.rados.0.smithi076.stdout:3430: done (2 left) 2021-03-15T22:34:26.588 INFO:tasks.rados.rados.0.smithi076.stdout:3431: copy_from oid 21 from oid 13 current snap is 408 2021-03-15T22:34:26.588 INFO:tasks.rados.rados.0.smithi076.stdout:3432: read oid 41 snap -1 2021-03-15T22:34:26.589 INFO:tasks.rados.rados.0.smithi076.stdout:3432: expect (ObjNum 1076 snap 366 seq_num 1076) 2021-03-15T22:34:26.589 INFO:tasks.rados.rados.0.smithi076.stdout:3433: snap_create 2021-03-15T22:34:26.589 INFO:tasks.rados.rados.0.smithi076.stdout:3428: finishing rollback tid 0 to smithi07694878-45 2021-03-15T22:34:26.589 INFO:tasks.rados.rados.0.smithi076.stdout:3428: finishing rollback tid 1 to smithi07694878-45 2021-03-15T22:34:26.590 INFO:tasks.rados.rados.0.smithi076.stdout:3428: finishing rollback tid 2 to smithi07694878-45 2021-03-15T22:34:26.590 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 2112 (ObjNum 1023 snap 341 seq_num 1023) dirty exists 2021-03-15T22:34:26.597 INFO:tasks.rados.rados.0.smithi076.stdout:3431: finishing copy_from to smithi07694878-21 2021-03-15T22:34:26.598 INFO:tasks.rados.rados.0.smithi076.stdout:3431: got expected ENOENT (src dne) 2021-03-15T22:34:26.598 INFO:tasks.rados.rados.0.smithi076.stdout:3431: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:34:26.599 INFO:tasks.rados.rados.0.smithi076.stdout:3432: expect (ObjNum 1076 snap 366 seq_num 1076) 2021-03-15T22:34:26.602 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:26.618 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:26.625 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:26.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:26.626 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:26.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:26 smithi076 conmon[56514]: cluster 2021-03-15T22:34:25.204261+0000 mgr.y (mgr.14144) 1031 : cluster [DBG] pgmap v1511: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 146 active+clean; 689 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 4.2 MiB/s wr, 13 op/s; 9/1929 objects degraded (0.467%); 40/1929 objects misplaced (2.074%); 546 KiB/s, 14 objects/s recovering 2021-03-15T22:34:26.723 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:26 smithi076 conmon[56514]: cluster 2021-03-15T22:34:25.573509+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e910: 8 total, 8 up, 7 in 2021-03-15T22:34:26.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:26 smithi076 conmon[62156]: cluster 2021-03-15T22:34:25.204261+0000 mgr.y (mgr.14144) 1031 : cluster [DBG] pgmap v1511: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 146 active+clean; 689 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 4.2 MiB/s wr, 13 op/s; 9/1929 objects degraded (0.467%); 40/1929 objects misplaced (2.074%); 546 KiB/s, 14 objects/s recovering 2021-03-15T22:34:26.724 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:26 smithi076 conmon[62156]: cluster 2021-03-15T22:34:25.573509+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e910: 8 total, 8 up, 7 in 2021-03-15T22:34:26.728 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:26.751 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:26.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:26.752 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:26.853 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:26.875 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:26.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:26.877 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:26.978 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:26.994 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-15T22:34:26.999 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:27.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:27.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:27.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:26 smithi125 conmon[56739]: cluster 2021-03-15T22:34:25.204261+0000 mgr.y (mgr. 2021-03-15T22:34:27.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:26 smithi125 conmon[56739]: 14144) 1031 : cluster [DBG] pgmap v1511: 148 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 146 active+clean; 689 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.0 KiB/s rd, 4.2 MiB/s wr, 13 op/s; 9/1929 objects degraded (0.467%); 40/1929 objects misplaced (2.074%); 546 KiB/s, 14 objects/s recovering 2021-03-15T22:34:27.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:26 smithi125 conmon[56739]: cluster 2021-03-15T22:34:25.573509+0000 mon.a (mon.0) 2301 : cluster [DBG] osdmap e910: 8 total, 8 up, 7 in 2021-03-15T22:34:27.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:27.125 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:27.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:27.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:27.249 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:27.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:27.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:27.352 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:27.373 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:27.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:27.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:27.476 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:27.498 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:27.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:27.499 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:27.587 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:27 smithi076 conmon[56514]: cluster 2021-03-15T22:34:26.563570+0000 osd.5 (osd.5) 48 : cluster 2021-03-15T22:34:27.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:27 smithi076 conmon[56514]: [DBG] purged_snaps scrub starts 2021-03-15T22:34:27.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:27 smithi076 conmon[56514]: cluster 2021-03-15T22:34:26.565555+0000 osd.5 (osd.5) 49 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:27.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:27 smithi076 conmon[56514]: cluster 2021-03-15T22:34:26.581903+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e911: 8 total, 8 up, 7 in 2021-03-15T22:34:27.588 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:27 smithi076 conmon[56514]: audit 2021-03-15T22:34:27.224868+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? 172.21.15.76:0/3856835326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:27.590 INFO:tasks.rados.rados.0.smithi076.stdout:3428: done (4 left) 2021-03-15T22:34:27.591 INFO:tasks.rados.rados.0.smithi076.stdout:3429: done (3 left) 2021-03-15T22:34:27.591 INFO:tasks.rados.rados.0.smithi076.stdout:3431: done (2 left) 2021-03-15T22:34:27.591 INFO:tasks.rados.rados.0.smithi076.stdout:3432: done (1 left) 2021-03-15T22:34:27.591 INFO:tasks.rados.rados.0.smithi076.stdout:3433: done (0 left) 2021-03-15T22:34:27.592 INFO:tasks.rados.rados.0.smithi076.stdout:3434: copy_from oid 38 from oid 16 current snap is 409 2021-03-15T22:34:27.592 INFO:tasks.rados.rados.0.smithi076.stdout:3435: write (excl) oid 49 current snap is 409 2021-03-15T22:34:27.592 INFO:tasks.rados.rados.0.smithi076.stdout:3435: seq_num 1163 ranges {746541=400243,1739617=690403,3138117=408424} 2021-03-15T22:34:27.596 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:34:27.597 INFO:tasks.rados.rados.0.smithi076.stdout:3435: writing smithi07694878-49 from 746541 to 1146784 tid 1 2021-03-15T22:34:27.600 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:27.604 INFO:tasks.rados.rados.0.smithi076.stdout:3435: writing smithi07694878-49 from 1739617 to 2430020 tid 2 2021-03-15T22:34:27.610 INFO:tasks.rados.rados.0.smithi076.stdout:3435: writing smithi07694878-49 from 3138117 to 3546541 tid 3 2021-03-15T22:34:27.611 INFO:tasks.rados.rados.0.smithi076.stdout:3436: delete oid 33 current snap is 409 2021-03-15T22:34:27.611 INFO:tasks.rados.rados.0.smithi076.stdout:3435: finishing write tid 1 to smithi07694878-49 2021-03-15T22:34:27.612 INFO:tasks.rados.rados.0.smithi076.stdout:3434: finishing copy_from to smithi07694878-38 2021-03-15T22:34:27.612 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2781 (ObjNum 1141 snap 397 seq_num 1141) dirty exists 2021-03-15T22:34:27.612 INFO:tasks.rados.rados.0.smithi076.stdout:3434: finishing copy_from racing read to smithi07694878-38 2021-03-15T22:34:27.613 INFO:tasks.rados.rados.0.smithi076.stdout:3435: finishing write tid 2 to smithi07694878-49 2021-03-15T22:34:27.614 INFO:tasks.rados.rados.0.smithi076.stdout:3434: done (2 left) 2021-03-15T22:34:27.614 INFO:tasks.rados.rados.0.smithi076.stdout:3436: done (1 left) 2021-03-15T22:34:27.615 INFO:tasks.rados.rados.0.smithi076.stdout:3437: rollback oid 17 current snap is 409 2021-03-15T22:34:27.615 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 17 to 394 2021-03-15T22:34:27.615 INFO:tasks.rados.rados.0.smithi076.stdout:3438: delete oid 33 current snap is 409 2021-03-15T22:34:27.616 INFO:tasks.rados.rados.0.smithi076.stdout:3435: finishing write tid 3 to smithi07694878-49 2021-03-15T22:34:27.616 INFO:tasks.rados.rados.0.smithi076.stdout:3435: finishing write tid 4 to smithi07694878-49 2021-03-15T22:34:27.617 INFO:tasks.rados.rados.0.smithi076.stdout:3435: finishing write tid 5 to smithi07694878-49 2021-03-15T22:34:27.617 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 2798 (ObjNum 1163 snap 409 seq_num 1163) dirty exists 2021-03-15T22:34:27.617 INFO:tasks.rados.rados.0.smithi076.stdout:3435: left oid 49 (ObjNum 1163 snap 409 seq_num 1163) 2021-03-15T22:34:27.617 INFO:tasks.rados.rados.0.smithi076.stdout:3435: done (2 left) 2021-03-15T22:34:27.618 INFO:tasks.rados.rados.0.smithi076.stdout:3438: done (1 left) 2021-03-15T22:34:27.618 INFO:tasks.rados.rados.0.smithi076.stdout:3439: snap_remove snap 382 2021-03-15T22:34:27.625 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:27.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:27.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:27.728 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:27.751 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:27.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:27.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:27.854 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:27 smithi076 conmon[62156]: cluster 2021-03-15T22:34:26.563570+0000 osd.5 (osd.5) 48 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:27.855 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:27 smithi076 conmon[62156]: cluster 2021-03-15T22:34:26.565555+0000 osd.5 (osd.5) 49 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:27 smithi076 conmon[62156]: cluster 2021-03-15T22:34:26.581903+0000 mon.a (mon.0) 2302 : cluster [DBG] osdmap e911: 8 total, 8 up, 7 in 2021-03-15T22:34:27.856 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:27 smithi076 conmon[62156]: audit 2021-03-15T22:34:27.224868+0000 mon.a (mon.0) 2303 : audit [INF] from='client.? 172.21.15.76:0/3856835326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:27.877 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:27.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:27.884 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:27.979 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:28.001 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:28.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:28.003 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: cluster 2021-03-15T22:34:26.563570+0000 2021-03-15T22:34:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: osd.5 (osd.5) 48 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:34:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: :34:26.565555+0000 osd.5 (osd.5) 49 2021-03-15T22:34:28.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:28.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:34:28.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: -03-15T22:34: 2021-03-15T22:34:28.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: 26.581903+0000 mon.a ( 2021-03-15T22:34:28.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: mon.0) 2302 : cluster 2021-03-15T22:34:28.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: [DBG] osdmap e911: 8 total, 8 up, 7 in 2021-03-15T22:34:28.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:34:28.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: 03-15T22:34: 2021-03-15T22:34:28.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: 27.224868+0000 mon.a ( 2021-03-15T22:34:28.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: mon.0) 2303 : audit 2021-03-15T22:34:28.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:27 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/3856835326' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:28.104 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:28.127 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:28.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:28.129 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:28.230 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:28.252 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:28.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:28.254 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:28.355 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:28.377 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:28.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:28.379 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:28.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:28.502 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:28.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:28.504 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:28.596 INFO:tasks.rados.rados.0.smithi076.stdout:3439: done (1 left) 2021-03-15T22:34:28.596 INFO:tasks.rados.rados.0.smithi076.stdout:3440: read oid 4 snap -1 2021-03-15T22:34:28.597 INFO:tasks.rados.rados.0.smithi076.stdout:3440: expect (ObjNum 1109 snap 386 seq_num 1109) 2021-03-15T22:34:28.597 INFO:tasks.rados.rados.0.smithi076.stdout:3441: delete oid 33 current snap is 409 2021-03-15T22:34:28.597 INFO:tasks.rados.rados.0.smithi076.stdout:3437: finishing rollback tid 0 to smithi07694878-17 2021-03-15T22:34:28.597 INFO:tasks.rados.rados.0.smithi076.stdout:3437: finishing rollback tid 1 to smithi07694878-17 2021-03-15T22:34:28.598 INFO:tasks.rados.rados.0.smithi076.stdout:3437: finishing rollback tid 2 to smithi07694878-17 2021-03-15T22:34:28.598 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 2626 (ObjNum 960 snap 321 seq_num 960) dirty exists 2021-03-15T22:34:28.603 INFO:tasks.rados.rados.0.smithi076.stdout:3437: done (2 left) 2021-03-15T22:34:28.604 INFO:tasks.rados.rados.0.smithi076.stdout:3441: done (1 left) 2021-03-15T22:34:28.604 INFO:tasks.rados.rados.0.smithi076.stdout:3442: read oid 20 snap 404 2021-03-15T22:34:28.604 INFO:tasks.rados.rados.0.smithi076.stdout:3442: expect (ObjNum 1108 snap 385 seq_num 1108) 2021-03-15T22:34:28.604 INFO:tasks.rados.rados.0.smithi076.stdout:3443: read oid 46 snap -1 2021-03-15T22:34:28.605 INFO:tasks.rados.rados.0.smithi076.stdout:3443: expect (ObjNum 1156 snap 405 seq_num 1156) 2021-03-15T22:34:28.605 INFO:tasks.rados.rados.0.smithi076.stdout:3444: snap_remove snap 354 2021-03-15T22:34:28.605 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:28.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:28.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:28.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:28 smithi076 conmon[56514]: cluster 2021-03-15T22:34:27.204813+0000 mgr.y (mgr.14144) 1032 : cluster [DBG] pgmap v1514: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 704 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 14 op/s; 9/1947 objects degraded (0.462%); 40/1947 objects misplaced (2.054%); 546 KiB/s, 14 objects/s recovering 2021-03-15T22:34:28.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:28 smithi076 conmon[56514]: cluster 2021-03-15T22:34:27.579911+0000 mon.a (mon.0) 2304 : cluster [WRN] Health check update: Degraded data redundancy: 9/1947 objects degraded (0.462%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:28.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:28 smithi076 conmon[56514]: cluster 2021-03-15T22:34:27.580384+0000 mon.a (mon.0) 2305 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:28.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:28 smithi076 conmon[56514]: audit 2021-03-15T22:34:27.588162+0000 mon.a (mon.0) 2306 : audit [INF] from='client.? 172.21.15.76:0/3856835326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:28.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:28 smithi076 conmon[56514]: cluster 2021-03-15T22:34:27.588243+0000 mon.a (mon.0) 2307 : cluster [DBG] osdmap e912: 8 total, 8 up, 7 in 2021-03-15T22:34:28.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:28 smithi076 conmon[62156]: cluster 2021-03-15T22:34:27.204813+0000 mgr.y (mgr.14144) 1032 : cluster [DBG] pgmap v1514: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 704 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 14 op/s; 9/1947 objects degraded (0.462%); 40/1947 objects misplaced (2.054%); 546 KiB/s, 14 objects/s recovering 2021-03-15T22:34:28.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:28 smithi076 conmon[62156]: cluster 2021-03-15T22:34:27.579911+0000 mon.a (mon.0) 2304 : cluster [WRN] Health check update: Degraded data redundancy: 9/1947 objects degraded (0.462%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:28.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:28 smithi076 conmon[62156]: cluster 2021-03-15T22:34:27.580384+0000 mon.a (mon.0) 2305 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:28.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:28 smithi076 conmon[62156]: audit 2021-03-15T22:34:27.588162+0000 mon.a (mon.0) 2306 : audit [INF] from='client.? 172.21.15.76:0/3856835326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:28.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:28 smithi076 conmon[62156]: cluster 2021-03-15T22:34:27.588243+0000 mon.a (mon.0) 2307 : cluster [DBG] osdmap e912: 8 total, 8 up, 7 in 2021-03-15T22:34:28.736 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:28.761 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:28.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:28.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:28.863 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:28.886 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:28.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:28.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:28.989 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:29.011 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:29.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:29.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:29.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: cluster 2021-03-15T22:34:27.204813+0000 mgr.y (mgr.14144) 2021-03-15T22:34:29.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: 1032 : cluster [DBG] pgmap v1514: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 704 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 3.9 MiB/s wr, 14 op/s; 9/1947 objects degraded (0.462%); 40/1947 objects misplaced (2.054%); 546 KiB/s, 14 objects/s recovering 2021-03-15T22:34:29.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: cluster 2021-03-15T22:34:27.579911+0000 mon.a (mon.0) 2304 : cluster [WRN] Health check update: Degraded data redundancy: 9/1947 objects degraded (0.462%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:29.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: cluster 2021-03-15T22:34:27.580384+0000 mon.a (mon. 2021-03-15T22:34:29.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: 0) 2305 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:29.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: audit 2021-03-15T22:34:29.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: 2021-03-15T22:34:27. 2021-03-15T22:34:29.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: 588162+0000 mon.a (mon.0) 2306 : audit 2021-03-15T22:34:29.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/3856835326' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:29.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: cluster 2021-03-15T22:34:29.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: 2021-03-15T22:34: 2021-03-15T22:34:29.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: 27.588243+0000 mon.a (mon.0) 2307 2021-03-15T22:34:29.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:28 smithi125 conmon[56739]: : cluster [DBG] osdmap e912: 8 total, 8 up, 7 in 2021-03-15T22:34:29.092 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:34:29.110 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:34:29.114 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:29.129 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:34:29.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:29.138 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:29.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:29.146 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:34:29.164 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:34:29.182 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:34:29.202 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:34:29.229 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:34:29.230 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:34:29.230 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:34:29.230 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:34:29.230 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:34:29.231 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:34:29.231 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:34:29.231 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:34:29.232 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:34:29.240 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:29.262 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:29.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:29.264 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:29.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:29.386 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:29.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:29.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:29.489 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:29.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:29.512 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:29.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:29.601 INFO:tasks.rados.rados.0.smithi076.stdout:3444: done (3 left) 2021-03-15T22:34:29.601 INFO:tasks.rados.rados.0.smithi076.stdout:3445: snap_remove snap 399 2021-03-15T22:34:29.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:29.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:29.637 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:29.637 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:29.739 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:29.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:29.762 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:29.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:29.864 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:29.887 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:29.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:29.889 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:29.990 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:30.012 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:30.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:30.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:30.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:29 smithi076 conmon[56514]: cluster 2021-03-15T22:34:28.593323+0000 mon.a (mon.0) 2308 : cluster [DBG] osdmap e913: 8 total, 8 up, 7 in 2021-03-15T22:34:30.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:29 smithi076 conmon[56514]: 2021-03-15T22:34:30.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:29 smithi076 conmon[56514]: audit 2021-03-15T22:34:29.072112+0000 mon.c (mon.1) 71 : audit [DBG] from='client.? 172.21.15.76:0/1955449263' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:30.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:29 smithi076 conmon[62156]: cluster 2021-03-15T22:34:28.593323+0000 2021-03-15T22:34:30.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:29 smithi076 conmon[62156]: mon.a (mon.0) 2308 : cluster [DBG] osdmap e913: 8 total, 8 up, 7 in 2021-03-15T22:34:30.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:29 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:34:30.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:29 smithi076 conmon[62156]: 15T22:34:29.072112+0000 mon.c (mon.1) 71 : audit 2021-03-15T22:34:30.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:29 smithi076 conmon[62156]: [DBG] from='client.? 172.21.15.76:0/1955449263' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:30.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:29 smithi125 conmon[56739]: cluster 2021-03-15T22:34:28.593323+0000 mon.a (mon 2021-03-15T22:34:30.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:29 smithi125 conmon[56739]: .0) 2308 : cluster [DBG] osdmap e913: 8 total, 8 up, 7 in 2021-03-15T22:34:30.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:29 smithi125 conmon[56739]: audit 2021-03-15T22:34:29.072112+0000 mon.c (mon.1) 71 : audit [DBG] from='client.? 172.21.15.76:0/1955449263' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:30.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:29 smithi125 conmon[56739]: 2021-03-15T22:34:30.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:30.139 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:30.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:30.140 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:30.241 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:30.263 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:30.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:30.265 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:30.321 INFO:tasks.rados.rados.0.smithi076.stdout:3445: done (3 left) 2021-03-15T22:34:30.322 INFO:tasks.rados.rados.0.smithi076.stdout:3446: snap_remove snap 408 2021-03-15T22:34:30.366 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:30.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:30.390 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:30.391 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:30.492 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:30.514 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:30.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:30.515 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:30.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:34:30 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:34:30.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:30.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:30.641 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:30.642 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:30.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:30.744 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:34:30.766 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:30.767 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:30.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:30 smithi076 conmon[56514]: cluster 2021-03-15T22:34:29.205269+0000 mgr.y (mgr.14144) 1033 : cluster [DBG] pgmap v1517: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 708 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 10 op/s; 9/1950 objects degraded (0.462%); 40/1950 objects misplaced (2.051%) 2021-03-15T22:34:30.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:30 smithi076 conmon[56514]: cluster 2021-03-15T22:34:29.598668+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e914: 8 total, 8 up, 7 in 2021-03-15T22:34:30.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:30 smithi076 conmon[56514]: cluster 2021-03-15T22:34:30.314076+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e915: 8 total, 8 up, 7 in 2021-03-15T22:34:30.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:30 smithi076 conmon[62156]: cluster 2021-03-15T22:34:29.205269+0000 mgr.y (mgr.14144) 1033 : cluster [DBG] pgmap v1517: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 708 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 10 op/s; 9/1950 objects degraded (0.462%); 40/1950 objects misplaced (2.051%) 2021-03-15T22:34:30.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:30 smithi076 conmon[62156]: cluster 2021-03-15T22:34:29.598668+0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e914: 8 total, 8 up, 7 in 2021-03-15T22:34:30.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:30 smithi076 conmon[62156]: cluster 2021-03-15T22:34:30.314076+0000 mon.a (mon.0) 2310 : cluster [DBG] osdmap e915: 8 total, 8 up, 7 in 2021-03-15T22:34:30.868 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:30.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:30.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:30.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:31.017 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:31.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:31.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:31.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:30 smithi125 conmon[56739]: cluster 2021-03-15T22:34:29.205269+0000 mgr.y (mgr 2021-03-15T22:34:31.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:30 smithi125 conmon[56739]: .14144) 1033 : cluster [DBG] pgmap v1517: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 146 active+clean; 708 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.4 MiB/s rd, 3.2 MiB/s wr, 10 op/s; 9/1950 objects degraded (0.462%); 40/1950 objects misplaced (2.051%) 2021-03-15T22:34:31.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:30 smithi125 conmon[56739]: cluster 2021-03-15T22:34:29.598668 2021-03-15T22:34:31.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:30 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2309 : cluster [DBG] osdmap e914: 8 total, 8 up, 7 in 2021-03-15T22:34:31.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:30 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:34:31.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:30 smithi125 conmon[56739]: 03-15T22:34:30 2021-03-15T22:34:31.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:30 smithi125 conmon[56739]: .314076+0000 mon.a (mon.0) 2310 : 2021-03-15T22:34:31.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:30 smithi125 conmon[56739]: cluster [DBG] osdmap e915: 8 total, 8 up, 7 in 2021-03-15T22:34:31.119 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:31.142 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:31.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:31.144 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:31.231 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:31.245 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:31.268 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:31.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:31.269 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:31.313 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:34:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:34:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:34:31.314 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:34:31.314 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:34:31.327 INFO:tasks.rados.rados.0.smithi076.stdout:3446: done (3 left) 2021-03-15T22:34:31.328 INFO:tasks.rados.rados.0.smithi076.stdout:3447: write (excl) oid 33 current snap is 409 2021-03-15T22:34:31.328 INFO:tasks.rados.rados.0.smithi076.stdout:3447: seq_num 1167 ranges {413769=417592,1573545=697741,2847809=624439} 2021-03-15T22:34:31.333 INFO:tasks.rados.rados.0.smithi076.stdout:3447: writing smithi07694878-33 from 413769 to 831361 tid 1 2021-03-15T22:34:31.333 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 6 pg_num to 42 2021-03-15T22:34:31.341 INFO:tasks.rados.rados.0.smithi076.stdout:3447: writing smithi07694878-33 from 1573545 to 2271286 tid 2 2021-03-15T22:34:31.348 INFO:tasks.rados.rados.0.smithi076.stdout:3447: writing smithi07694878-33 from 2847809 to 3472248 tid 3 2021-03-15T22:34:31.349 INFO:tasks.rados.rados.0.smithi076.stdout:3448: rollback oid 9 current snap is 409 2021-03-15T22:34:31.349 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 9 to 384 2021-03-15T22:34:31.349 INFO:tasks.rados.rados.0.smithi076.stdout:3440: expect (ObjNum 1109 snap 386 seq_num 1109) 2021-03-15T22:34:31.370 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:31.393 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:31.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:31.394 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:31.495 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:31.519 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:31.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:31.520 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:31.539 INFO:tasks.rados.rados.0.smithi076.stdout:3442: expect (ObjNum 1108 snap 385 seq_num 1108) 2021-03-15T22:34:31.621 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:31.623 INFO:tasks.rados.rados.0.smithi076.stdout:3443: expect (ObjNum 1156 snap 405 seq_num 1156) 2021-03-15T22:34:31.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:31.646 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:31.647 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:31.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[56514]: audit 2021-03-15T22:34:30.246288+0000 mgr.y (mgr.14144) 1034 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:31.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[56514]: audit 2021-03-15T22:34:31.260504+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? 172.21.15.76:0/3102488636' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2021-03-15T22:34:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[56514]: audit 2021-03-15T22:34:31.324214+0000 mon.a (mon.0) 2312 : audit [INF] from='client.? 172.21.15.76:0/3102488636' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]': finished 2021-03-15T22:34:31.662 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[56514]: cluster 2021-03-15T22:34:31.324285+0000 mon.a (mon.0) 2313 : cluster [DBG] osdmap e916: 8 total, 8 up, 7 in 2021-03-15T22:34:31.662 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: audit 2021-03-15T22:34:31.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: 2021-03-15T22:34:31.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: 2021 2021-03-15T22:34:31.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: -03-15T22:34:30.246288 2021-03-15T22:34:31.663 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 1034 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:31.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: audit 2021-03-15T22:34:31.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:34:31.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: :34:31.260504+0000 mon.a (mon. 2021-03-15T22:34:31.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: 0) 2311 : audit [INF] from='client.? 172.21.15.76:0/3102488636' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2021-03-15T22:34:31.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:34:31.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: -15T22:34:31.324214+0000 mon.a 2021-03-15T22:34:31.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: (mon.0) 2312 : audit [INF] from='client.? 172.21.15.76:0/3102488636' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]': finished 2021-03-15T22:34:31.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: 2021-03-15T22:34:31.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: cluster 2021-03-15T22:34:31.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:34:31.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: :34:31.324285+0000 mon.a (mon. 2021-03-15T22:34:31.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:31 smithi076 conmon[62156]: 0) 2313 : cluster [DBG] osdmap e916: 8 total, 8 up, 7 in 2021-03-15T22:34:31.748 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:31.769 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:31.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:31.771 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:31.872 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:31.894 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:31.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:31.896 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:31.931 INFO:tasks.rados.rados.0.smithi076.stdout:3447: finishing write tid 1 to smithi07694878-33 2021-03-15T22:34:31.931 INFO:tasks.rados.rados.0.smithi076.stdout:3447: finishing write tid 2 to smithi07694878-33 2021-03-15T22:34:31.931 INFO:tasks.rados.rados.0.smithi076.stdout:3447: finishing write tid 3 to smithi07694878-33 2021-03-15T22:34:31.931 INFO:tasks.rados.rados.0.smithi076.stdout:3447: finishing write tid 4 to smithi07694878-33 2021-03-15T22:34:31.932 INFO:tasks.rados.rados.0.smithi076.stdout:3447: finishing write tid 5 to smithi07694878-33 2021-03-15T22:34:31.932 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 2421 (ObjNum 1167 snap 409 seq_num 1167) dirty exists 2021-03-15T22:34:31.932 INFO:tasks.rados.rados.0.smithi076.stdout:3447: left oid 33 (ObjNum 1167 snap 409 seq_num 1167) 2021-03-15T22:34:31.932 INFO:tasks.rados.rados.0.smithi076.stdout:3448: finishing rollback tid 0 to smithi07694878-9 2021-03-15T22:34:31.933 INFO:tasks.rados.rados.0.smithi076.stdout:3448: finishing rollback tid 1 to smithi07694878-9 2021-03-15T22:34:31.933 INFO:tasks.rados.rados.0.smithi076.stdout:3448: finishing rollback tid 2 to smithi07694878-9 2021-03-15T22:34:31.933 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 2801 (ObjNum 1092 snap 375 seq_num 1092) dirty exists 2021-03-15T22:34:31.933 INFO:tasks.rados.rados.0.smithi076.stdout:3440: done (4 left) 2021-03-15T22:34:31.933 INFO:tasks.rados.rados.0.smithi076.stdout:3442: done (3 left) 2021-03-15T22:34:31.934 INFO:tasks.rados.rados.0.smithi076.stdout:3443: done (2 left) 2021-03-15T22:34:31.934 INFO:tasks.rados.rados.0.smithi076.stdout:3447: done (1 left) 2021-03-15T22:34:31.934 INFO:tasks.rados.rados.0.smithi076.stdout:3448: done (0 left) 2021-03-15T22:34:31.934 INFO:tasks.rados.rados.0.smithi076.stdout:3449: write (excl) oid 10 current snap is 409 2021-03-15T22:34:31.935 INFO:tasks.rados.rados.0.smithi076.stdout:3449: seq_num 1168 ranges {430576=492841,1632027=441399,2475232=435711,3365397=265179} 2021-03-15T22:34:31.935 INFO:tasks.rados.rados.0.smithi076.stdout:3449: writing smithi07694878-10 from 430576 to 923417 tid 1 2021-03-15T22:34:31.941 INFO:tasks.rados.rados.0.smithi076.stdout:3449: writing smithi07694878-10 from 1632027 to 2073426 tid 2 2021-03-15T22:34:31.946 INFO:tasks.rados.rados.0.smithi076.stdout:3449: writing smithi07694878-10 from 2475232 to 2910943 tid 3 2021-03-15T22:34:31.950 INFO:tasks.rados.rados.0.smithi076.stdout:3449: writing smithi07694878-10 from 3365397 to 3630576 tid 4 2021-03-15T22:34:31.951 INFO:tasks.rados.rados.0.smithi076.stdout:3450: delete oid 46 current snap is 409 2021-03-15T22:34:31.951 INFO:tasks.rados.rados.0.smithi076.stdout:3449: finishing write tid 1 to smithi07694878-10 2021-03-15T22:34:31.951 INFO:tasks.rados.rados.0.smithi076.stdout:3449: finishing write tid 2 to smithi07694878-10 2021-03-15T22:34:31.951 INFO:tasks.rados.rados.0.smithi076.stdout:3449: finishing write tid 3 to smithi07694878-10 2021-03-15T22:34:31.953 INFO:tasks.rados.rados.0.smithi076.stdout:3450: done (1 left) 2021-03-15T22:34:31.953 INFO:tasks.rados.rados.0.smithi076.stdout:3451: delete oid 45 current snap is 409 2021-03-15T22:34:31.954 INFO:tasks.rados.rados.0.smithi076.stdout:3449: finishing write tid 4 to smithi07694878-10 2021-03-15T22:34:31.954 INFO:tasks.rados.rados.0.smithi076.stdout:3449: finishing write tid 5 to smithi07694878-10 2021-03-15T22:34:31.954 INFO:tasks.rados.rados.0.smithi076.stdout:3449: finishing write tid 6 to smithi07694878-10 2021-03-15T22:34:31.954 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 2412 (ObjNum 1168 snap 409 seq_num 1168) dirty exists 2021-03-15T22:34:31.955 INFO:tasks.rados.rados.0.smithi076.stdout:3449: left oid 10 (ObjNum 1168 snap 409 seq_num 1168) 2021-03-15T22:34:31.955 INFO:tasks.rados.rados.0.smithi076.stdout:3449: done (1 left) 2021-03-15T22:34:31.956 INFO:tasks.rados.rados.0.smithi076.stdout:3451: done (0 left) 2021-03-15T22:34:31.956 INFO:tasks.rados.rados.0.smithi076.stdout:3452: write oid 19 current snap is 409 2021-03-15T22:34:31.956 INFO:tasks.rados.rados.0.smithi076.stdout:3452: seq_num 1171 ranges {480997=434941,1407473=705079,2557501=785880} 2021-03-15T22:34:31.959 INFO:tasks.rados.rados.0.smithi076.stdout:3452: writing smithi07694878-19 from 480997 to 915938 tid 1 2021-03-15T22:34:31.966 INFO:tasks.rados.rados.0.smithi076.stdout:3452: writing smithi07694878-19 from 1407473 to 2112552 tid 2 2021-03-15T22:34:31.973 INFO:tasks.rados.rados.0.smithi076.stdout:3452: writing smithi07694878-19 from 2557501 to 3343381 tid 3 2021-03-15T22:34:31.974 INFO:tasks.rados.rados.0.smithi076.stdout:3453: write oid 33 current snap is 409 2021-03-15T22:34:31.974 INFO:tasks.rados.rados.0.smithi076.stdout:3452: finishing write tid 1 to smithi07694878-19 2021-03-15T22:34:31.974 INFO:tasks.rados.rados.0.smithi076.stdout:3452: finishing write tid 2 to smithi07694878-19 2021-03-15T22:34:31.974 INFO:tasks.rados.rados.0.smithi076.stdout:3453: seq_num 1172 ranges {497804=510190,1465955=448737,2584924=597152} 2021-03-15T22:34:31.979 INFO:tasks.rados.rados.0.smithi076.stdout:3453: writing smithi07694878-33 from 497804 to 1007994 tid 1 2021-03-15T22:34:31.984 INFO:tasks.rados.rados.0.smithi076.stdout:3453: writing smithi07694878-33 from 1465955 to 1914692 tid 2 2021-03-15T22:34:31.991 INFO:tasks.rados.rados.0.smithi076.stdout:3453: writing smithi07694878-33 from 2584924 to 3182076 tid 3 2021-03-15T22:34:31.991 INFO:tasks.rados.rados.0.smithi076.stdout:3454: rollback oid 13 current snap is 409 2021-03-15T22:34:31.992 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 13 to 405 2021-03-15T22:34:31.992 INFO:tasks.rados.rados.0.smithi076.stdout:3452: finishing write tid 3 to smithi07694878-19 2021-03-15T22:34:31.992 INFO:tasks.rados.rados.0.smithi076.stdout:3452: finishing write tid 4 to smithi07694878-19 2021-03-15T22:34:31.992 INFO:tasks.rados.rados.0.smithi076.stdout:3452: finishing write tid 5 to smithi07694878-19 2021-03-15T22:34:31.993 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2785 (ObjNum 1171 snap 409 seq_num 1171) dirty exists 2021-03-15T22:34:31.993 INFO:tasks.rados.rados.0.smithi076.stdout:3452: left oid 19 (ObjNum 1171 snap 409 seq_num 1171) 2021-03-15T22:34:31.993 INFO:tasks.rados.rados.0.smithi076.stdout:3453: finishing write tid 1 to smithi07694878-33 2021-03-15T22:34:31.993 INFO:tasks.rados.rados.0.smithi076.stdout:3453: finishing write tid 2 to smithi07694878-33 2021-03-15T22:34:31.993 INFO:tasks.rados.rados.0.smithi076.stdout:3452: done (2 left) 2021-03-15T22:34:31.994 INFO:tasks.rados.rados.0.smithi076.stdout:3455: snap_create 2021-03-15T22:34:31.994 INFO:tasks.rados.rados.0.smithi076.stdout:3454: finishing rollback tid 1 to smithi07694878-13 2021-03-15T22:34:31.995 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 1997 (ObjNum 255149944 snap 21896 seq_num 1) dirty dne 2021-03-15T22:34:31.995 INFO:tasks.rados.rados.0.smithi076.stdout:3453: finishing write tid 3 to smithi07694878-33 2021-03-15T22:34:31.995 INFO:tasks.rados.rados.0.smithi076.stdout:3453: finishing write tid 4 to smithi07694878-33 2021-03-15T22:34:31.995 INFO:tasks.rados.rados.0.smithi076.stdout:3453: finishing write tid 5 to smithi07694878-33 2021-03-15T22:34:31.996 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 2425 (ObjNum 1172 snap 409 seq_num 1172) dirty exists 2021-03-15T22:34:31.996 INFO:tasks.rados.rados.0.smithi076.stdout:3453: left oid 33 (ObjNum 1172 snap 409 seq_num 1172) 2021-03-15T22:34:31.996 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:32.018 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:32.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:32.019 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:32.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:31 smithi125 conmon[56739]: audit 2021-03-15T22:34:30.246288+0000 mgr.y (mgr.14144) 1034 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:32.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:31 smithi125 conmon[56739]: audit 2021-03-15T22:34: 2021-03-15T22:34:32.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:31 smithi125 conmon[56739]: 31.260504+0000 mon.a (mon.0) 2311 : audit [INF] from='client.? 172.21.15.76:0/3102488636' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]: dispatch 2021-03-15T22:34:32.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:31 smithi125 conmon[56739]: audit 2021-03-15T22:34:31.324214+0000 mon.a (mon.0) 2312 : audit [INF] from='client.? 172.21.15.76:0/3102488636' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "42"}]': finished 2021-03-15T22:34:32.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:31 smithi125 conmon[56739]: cluster 2021-03-15T22:34:32.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:31 smithi125 conmon[56739]: 2021-03-15T22:34:31.324285+0000 mon.a (mon 2021-03-15T22:34:32.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:31 smithi125 conmon[56739]: .0) 2313 : cluster [DBG] osdmap e916: 8 total, 8 up, 7 in 2021-03-15T22:34:32.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:31 smithi125 conmon[56739]: 2021-03-15T22:34:32.120 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:32.146 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:32.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:32.148 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:32.249 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:32.275 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:32.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:32.277 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:32.378 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:32.404 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:32.405 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:32.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:32.528 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:32.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:32.530 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:32.614 INFO:tasks.rados.rados.0.smithi076.stdout:3453: done (2 left) 2021-03-15T22:34:32.615 INFO:tasks.rados.rados.0.smithi076.stdout:3454: done (1 left) 2021-03-15T22:34:32.615 INFO:tasks.rados.rados.0.smithi076.stdout:3455: done (0 left) 2021-03-15T22:34:32.615 INFO:tasks.rados.rados.0.smithi076.stdout:3456: delete oid 6 current snap is 410 2021-03-15T22:34:32.617 INFO:tasks.rados.rados.0.smithi076.stdout:3456: done (0 left) 2021-03-15T22:34:32.618 INFO:tasks.rados.rados.0.smithi076.stdout:3457: rollback oid 35 current snap is 410 2021-03-15T22:34:32.618 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 35 to 392 2021-03-15T22:34:32.618 INFO:tasks.rados.rados.0.smithi076.stdout:3458: snap_remove snap 385 2021-03-15T22:34:32.631 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:32.655 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:32.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:32.656 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:32.659 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-15T22:34:32.758 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:32.784 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:32.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:32.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:32.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:32.913 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:32.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:32.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:32.927 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:34:32.950 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:32 smithi076 conmon[56514]: cluster 2021-03-15T22:34:31. 2021-03-15T22:34:32.951 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:32 smithi076 conmon[56514]: 205840+0000 mgr.y (mgr.14144) 1035 : cluster [DBG] pgmap v1520: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 146 active+clean; 712 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 14 op/s; 22 MiB/s, 14 objects/s recovering 2021-03-15T22:34:32.951 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:32 smithi076 conmon[56514]: cluster 2021-03-15T22:34:31.597517+0000 mon.a (mon.0) 2314 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/1950 objects degraded (0.462%), 1 pg degraded) 2021-03-15T22:34:32.951 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:32 smithi076 conmon[62156]: cluster 2021-03-15T22:34:31. 2021-03-15T22:34:32.952 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:32 smithi076 conmon[62156]: 205840+0000 mgr.y (mgr.14144) 1035 : cluster [DBG] pgmap v1520: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 146 active+clean; 712 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 14 op/s; 22 MiB/s, 14 objects/s recovering 2021-03-15T22:34:32.952 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:32 smithi076 conmon[62156]: cluster 2021-03-15T22:34:32.952 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:32 smithi076 conmon[62156]: 2021-03-15T22:34:31.597517 2021-03-15T22:34:32.952 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:32 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2314 2021-03-15T22:34:32.953 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:32 smithi076 conmon[62156]: : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/1950 objects degraded (0.462%), 1 pg degraded) 2021-03-15T22:34:33.016 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:33.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:32 smithi125 conmon[56739]: cluster 2021-03-15T22:34:31.205840+0000 mgr.y (mgr.14144) 1035 : cluster [DBG] pgmap v1520: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 146 active+clean; 712 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 14 op/s; 22 MiB/s, 14 objects/s recovering 2021-03-15T22:34:33.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:32 smithi125 conmon[56739]: cluster 2021-03-15T22:34:31 2021-03-15T22:34:33.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:32 smithi125 conmon[56739]: .597517+0000 mon.a (mon.0) 2314 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 9/1950 objects degraded (0.462%), 1 pg degraded) 2021-03-15T22:34:33.042 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:33.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:33.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:33.144 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:33.168 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:33.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:33.169 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:33.205 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:33.270 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:33.296 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:33.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:33.327 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:33.398 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:33.420 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:33.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:33.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:33.523 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:33.549 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:33.550 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:33.551 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:33.620 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[62156]: cluster 2021-03-15T22:34:33.621 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[62156]: 2021-03-15T22:34:33.621 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[62156]: 2021-03-15T22:34:31.972444 2021-03-15T22:34:33.621 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[62156]: +0000 osd.2 (osd.2) 42 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:33.621 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[62156]: cluster 2021-03-15T22:34:31.974612+0000 osd.2 (osd.2 2021-03-15T22:34:33.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[62156]: ) 43 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:33.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[62156]: cluster 2021-03-15T22:34:32. 2021-03-15T22:34:33.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[62156]: 612953+0000 mon.a (mon.0) 2315 : cluster [DBG] osdmap e917: 8 total, 8 up, 7 in 2021-03-15T22:34:33.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:34:33.622 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[62156]: -15T22:34:33.207236+0000 mon.a ( 2021-03-15T22:34:33.623 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[62156]: mon.0) 2316 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-03-15T22:34:33.623 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: cluster 2021-03-15T22:34:33.623 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: 2021-03-15T22:34:33.624 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: 2021 2021-03-15T22:34:33.624 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: - 2021-03-15T22:34:33.624 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: 03 2021-03-15T22:34:33.624 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: - 2021-03-15T22:34:33.625 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: 15 2021-03-15T22:34:33.625 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: T 2021-03-15T22:34:33.625 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: 22 2021-03-15T22:34:33.625 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: :34:31.972444+0000 osd.2 ( 2021-03-15T22:34:33.626 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: osd.2) 42 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:33.626 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:34:33.626 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: -15T22:34:31.974612+0000 osd.2 (osd.2 2021-03-15T22:34:33.626 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: ) 43 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:33.626 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:34:33.627 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: :34:32.612953+0000 mon.a (mon.0) 2021-03-15T22:34:33.627 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: 2315 : cluster [DBG] osdmap e917: 8 total, 8 up, 7 in 2021-03-15T22:34:33.627 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: audit 2021-03-15 2021-03-15T22:34:33.627 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: T22:34:33.207236+0000 mon.a (mon. 2021-03-15T22:34:33.628 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:33 smithi076 conmon[56514]: 0) 2316 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-03-15T22:34:33.628 INFO:tasks.rados.rados.0.smithi076.stdout:3458: done (1 left) 2021-03-15T22:34:33.628 INFO:tasks.rados.rados.0.smithi076.stdout:3459: snap_remove snap 395 2021-03-15T22:34:33.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:33.675 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:33.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:33.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:33.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:33.801 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:33.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:33.802 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:33.903 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:33.925 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:33.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:33.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:34.027 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:34.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:33 smithi125 conmon[56739]: cluster 2021-03-15T22:34:31.972444+0000 osd.2 (osd.2 2021-03-15T22:34:34.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:33 smithi125 conmon[56739]: ) 42 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:34.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:33 smithi125 conmon[56739]: cluster 2021-03-15T22:34:31.974612+0000 osd.2 (osd.2) 43 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:34.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:33 smithi125 conmon[56739]: cluster 2021-03-15T22:34:32.612953 2021-03-15T22:34:34.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:33 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2315 : cluster [DBG] osdmap e917: 8 total, 8 up, 7 in 2021-03-15T22:34:34.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:33 smithi125 conmon[56739]: 2021-03-15T22:34:34.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:33 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:34:34.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:33 smithi125 conmon[56739]: -15T22:34:33.207236+0000 mon.a ( 2021-03-15T22:34:34.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:33 smithi125 conmon[56739]: mon.0) 2316 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]: dispatch 2021-03-15T22:34:34.053 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:34.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:34.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:34.156 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:34.179 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:34.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:34.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:34.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:34.302 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:34.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:34.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:34.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:34.426 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:34.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:34.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:34.529 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:34.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:34.557 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:34.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:34.631 INFO:tasks.rados.rados.0.smithi076.stdout:3459: done (1 left) 2021-03-15T22:34:34.631 INFO:tasks.rados.rados.0.smithi076.stdout:3460: rollback oid 14 current snap is 410 2021-03-15T22:34:34.631 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 14 to 403 2021-03-15T22:34:34.631 INFO:tasks.rados.rados.0.smithi076.stdout:3457: finishing rollback tid 0 to smithi07694878-35 2021-03-15T22:34:34.632 INFO:tasks.rados.rados.0.smithi076.stdout:3457: finishing rollback tid 1 to smithi07694878-35 2021-03-15T22:34:34.632 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 35 v 2787 (ObjNum 262635472 snap 21896 seq_num 262658464) dirty dne 2021-03-15T22:34:34.632 INFO:tasks.rados.rados.0.smithi076.stdout:3457: done (1 left) 2021-03-15T22:34:34.632 INFO:tasks.rados.rados.0.smithi076.stdout:3461: write oid 33 current snap is 410 2021-03-15T22:34:34.633 INFO:tasks.rados.rados.0.smithi076.stdout:3461: seq_num 1174 ranges {531418=660688,1982919=452406,3039770=691648} 2021-03-15T22:34:34.633 INFO:tasks.rados.rados.0.smithi076.stdout:3461: writing smithi07694878-33 from 531418 to 1192106 tid 1 2021-03-15T22:34:34.638 INFO:tasks.rados.rados.0.smithi076.stdout:3461: writing smithi07694878-33 from 1982919 to 2435325 tid 2 2021-03-15T22:34:34.639 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:34:34.647 INFO:tasks.rados.rados.0.smithi076.stdout:3461: writing smithi07694878-33 from 3039770 to 3731418 tid 3 2021-03-15T22:34:34.647 INFO:tasks.rados.rados.0.smithi076.stdout:3462: snap_create 2021-03-15T22:34:34.648 INFO:tasks.rados.rados.0.smithi076.stdout:3460: finishing rollback tid 0 to smithi07694878-14 2021-03-15T22:34:34.648 INFO:tasks.rados.rados.0.smithi076.stdout:3460: finishing rollback tid 1 to smithi07694878-14 2021-03-15T22:34:34.648 INFO:tasks.rados.rados.0.smithi076.stdout:3460: finishing rollback tid 2 to smithi07694878-14 2021-03-15T22:34:34.648 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 2677 (ObjNum 1140 snap 393 seq_num 1140) dirty exists 2021-03-15T22:34:34.648 INFO:tasks.rados.rados.0.smithi076.stdout:3461: finishing write tid 1 to smithi07694878-33 2021-03-15T22:34:34.649 INFO:tasks.rados.rados.0.smithi076.stdout:3461: finishing write tid 2 to smithi07694878-33 2021-03-15T22:34:34.651 INFO:tasks.rados.rados.0.smithi076.stdout:3461: finishing write tid 3 to smithi07694878-33 2021-03-15T22:34:34.652 INFO:tasks.rados.rados.0.smithi076.stdout:3461: finishing write tid 4 to smithi07694878-33 2021-03-15T22:34:34.652 INFO:tasks.rados.rados.0.smithi076.stdout:3461: finishing write tid 5 to smithi07694878-33 2021-03-15T22:34:34.652 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 2429 (ObjNum 1174 snap 410 seq_num 1174) dirty exists 2021-03-15T22:34:34.652 INFO:tasks.rados.rados.0.smithi076.stdout:3461: left oid 33 (ObjNum 1174 snap 410 seq_num 1174) 2021-03-15T22:34:34.658 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:34.681 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:34.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:34.682 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:34.783 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:34.805 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:34.808 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:34.908 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:34.930 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:34.932 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:35.017 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[62156]: cluster 2021-03-15T22:34:33.206403+0000 mgr.y (mgr.14144) 2021-03-15T22:34:35.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[62156]: 1036 : cluster [DBG] pgmap v1523: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 146 active+clean; 714 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 15 op/s; 22 MiB/s, 14 objects/s recovering 2021-03-15T22:34:35.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[62156]: audit 2021-03-15T22:34:33.620317+0000 mon.a (mon.0) 2317 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]': finished 2021-03-15T22:34:35.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[62156]: cluster 2021-03-15T22:34:33.620401+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2021-03-15T22:34:35.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[62156]: 2021-03-15T22:34:35.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[62156]: audit 2021-03-15T22:34:33.846729+0000 mon.c (mon.1) 72 : audit [INF] from='client.? 172.21.15.76:0/822047795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:34:35.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[62156]: audit 2021-03-15T22:34:33 2021-03-15T22:34:35.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[62156]: .847086+0000 mon.a (mon.0) 2319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:34:35.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: cluster 2021-03-15T22:34:35.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: 2021-03-15T 2021-03-15T22:34:35.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: 22:34:33.206403+0000 mgr.y (mgr.14144) 1036 : cluster [DBG] pgmap v1523: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 146 active+clean; 714 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 15 op/s; 22 MiB/s, 14 objects/s recovering 2021-03-15T22:34:35.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:34:35.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: 03-15T22:34:33.620317+0000 mon.a (mon 2021-03-15T22:34:35.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: .0) 2317 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]': finished 2021-03-15T22:34:35.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: 2021-03-15T22:34:35.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: cluster 2021-03-15T22:34:35.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: 2021-03 2021-03-15T22:34:35.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: -15T22:34:33.620401 2021-03-15T22:34:35.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: +0000 mon.a (mon.0) 2021-03-15T22:34:35.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: 2318 : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2021-03-15T22:34:35.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: 2021-03-15T22:34:35.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:34:35.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: 03-15T22:34:33.846729 2021-03-15T22:34:35.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: +0000 mon.c (mon.1) 2021-03-15T22:34:35.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: 72 : audit [INF] from='client.? 172.21.15.76:0/822047795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:34:35.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:34:35.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: 03-15T22:34:33.847086+0000 mon.a (mon 2021-03-15T22:34:35.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: .0) 2319 : audit [INF] 2021-03-15T22:34:35.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:34 smithi076 conmon[56514]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:34:35.033 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:35.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:34 smithi125 conmon[56739]: cluster 2021-03-15T22:34:33.206403+0000 mgr.y (mgr.14144 2021-03-15T22:34:35.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:34 smithi125 conmon[56739]: ) 1036 : cluster [DBG] pgmap v1523: 148 pgs: 1 active+remapped+backfill_toofull, 1 active+undersized+remapped, 146 active+clean; 714 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 12 MiB/s rd, 1.1 MiB/s wr, 15 op/s; 22 MiB/s, 14 objects/s recovering 2021-03-15T22:34:35.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:34 smithi125 conmon[56739]: audit 2021-03-15T22:34:33.620317+0000 mon.a (mon.0) 2317 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "42"}]': finished 2021-03-15T22:34:35.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:34 smithi125 conmon[56739]: cluster 2021-03-15T22:34:33. 2021-03-15T22:34:35.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:34 smithi125 conmon[56739]: 620401+0000 mon.a (mon.0) 2318 : cluster [DBG] osdmap e918: 8 total, 8 up, 7 in 2021-03-15T22:34:35.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:34 smithi125 conmon[56739]: audit 2021-03-15T22:34:33.846729 2021-03-15T22:34:35.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:34 smithi125 conmon[56739]: +0000 mon.c (mon.1) 72 : audit [INF] from='client.? 172.21.15.76:0/822047795' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:34:35.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:34 smithi125 conmon[56739]: audit 2021-03-15T22:34:33.847086+0000 2021-03-15T22:34:35.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:34 smithi125 conmon[56739]: mon.a (mon.0) 2319 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:34:35.059 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:35.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:35.060 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:35.161 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:35.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:35.185 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:35.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:35.286 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:35.312 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:35.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:35.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:35.415 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:35.436 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:35.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:35.438 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:35.539 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:35.561 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:35.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:35.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:35.643 INFO:tasks.rados.rados.0.smithi076.stdout:3460: done (2 left) 2021-03-15T22:34:35.644 INFO:tasks.rados.rados.0.smithi076.stdout:3461: done (1 left) 2021-03-15T22:34:35.644 INFO:tasks.rados.rados.0.smithi076.stdout:3462: done (0 left) 2021-03-15T22:34:35.644 INFO:tasks.rados.rados.0.smithi076.stdout:3463: delete oid 1 current snap is 411 2021-03-15T22:34:35.648 INFO:tasks.rados.rados.0.smithi076.stdout:3463: done (0 left) 2021-03-15T22:34:35.648 INFO:tasks.rados.rados.0.smithi076.stdout:3464: write (excl) oid 47 current snap is 411 2021-03-15T22:34:35.649 INFO:tasks.rados.rados.0.smithi076.stdout:3464: seq_num 1176 ranges {565032=411186,1583530=456075,2578263=758593} 2021-03-15T22:34:35.654 INFO:tasks.rados.rados.0.smithi076.stdout:3464: writing smithi07694878-47 from 565032 to 976218 tid 1 2021-03-15T22:34:35.660 INFO:tasks.rados.rados.0.smithi076.stdout:3464: writing smithi07694878-47 from 1583530 to 2039605 tid 2 2021-03-15T22:34:35.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:35.667 INFO:tasks.rados.rados.0.smithi076.stdout:3464: writing smithi07694878-47 from 2578263 to 3336856 tid 3 2021-03-15T22:34:35.667 INFO:tasks.rados.rados.0.smithi076.stdout:3465: write (excl) oid 32 current snap is 411 2021-03-15T22:34:35.668 INFO:tasks.rados.rados.0.smithi076.stdout:3465: seq_num 1177 ranges {581839=486435,1642012=599733,3005686=569865} 2021-03-15T22:34:35.672 INFO:tasks.rados.rados.0.smithi076.stdout:3465: writing smithi07694878-32 from 581839 to 1068274 tid 1 2021-03-15T22:34:35.679 INFO:tasks.rados.rados.0.smithi076.stdout:3465: writing smithi07694878-32 from 1642012 to 2241745 tid 2 2021-03-15T22:34:35.688 INFO:tasks.rados.rados.0.smithi076.stdout:3465: writing smithi07694878-32 from 3005686 to 3575551 tid 3 2021-03-15T22:34:35.688 INFO:tasks.rados.rados.0.smithi076.stdout:3466: snap_create 2021-03-15T22:34:35.688 INFO:tasks.rados.rados.0.smithi076.stdout:3464: finishing write tid 1 to smithi07694878-47 2021-03-15T22:34:35.689 INFO:tasks.rados.rados.0.smithi076.stdout:3464: finishing write tid 2 to smithi07694878-47 2021-03-15T22:34:35.689 INFO:tasks.rados.rados.0.smithi076.stdout:3464: finishing write tid 3 to smithi07694878-47 2021-03-15T22:34:35.689 INFO:tasks.rados.rados.0.smithi076.stdout:3464: finishing write tid 4 to smithi07694878-47 2021-03-15T22:34:35.689 INFO:tasks.rados.rados.0.smithi076.stdout:3464: finishing write tid 5 to smithi07694878-47 2021-03-15T22:34:35.690 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 2417 (ObjNum 1176 snap 411 seq_num 1176) dirty exists 2021-03-15T22:34:35.690 INFO:tasks.rados.rados.0.smithi076.stdout:3464: left oid 47 (ObjNum 1176 snap 411 seq_num 1176) 2021-03-15T22:34:35.690 INFO:tasks.rados.rados.0.smithi076.stdout:3465: finishing write tid 1 to smithi07694878-32 2021-03-15T22:34:35.691 INFO:tasks.rados.rados.0.smithi076.stdout:3465: finishing write tid 2 to smithi07694878-32 2021-03-15T22:34:35.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:35.692 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:35.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:35.694 INFO:tasks.rados.rados.0.smithi076.stdout:3465: finishing write tid 3 to smithi07694878-32 2021-03-15T22:34:35.694 INFO:tasks.rados.rados.0.smithi076.stdout:3465: finishing write tid 4 to smithi07694878-32 2021-03-15T22:34:35.695 INFO:tasks.rados.rados.0.smithi076.stdout:3465: finishing write tid 5 to smithi07694878-32 2021-03-15T22:34:35.695 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 3038 (ObjNum 1177 snap 411 seq_num 1177) dirty exists 2021-03-15T22:34:35.696 INFO:tasks.rados.rados.0.smithi076.stdout:3465: left oid 32 (ObjNum 1177 snap 411 seq_num 1177) 2021-03-15T22:34:35.794 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:35.819 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:35.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:35.821 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:35.922 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:35.945 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:35.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:35.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:36.007 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[56514]: cluster 2021-03-15T22:34:34.614647+0000 mon.a ( 2021-03-15T22:34:36.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[56514]: mon.0) 2320 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:36.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[56514]: audit 2021-03-15T22:34:34.619000+0000 mon.a (mon.0) 2321 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:34:36.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[56514]: cluster 2021-03-15T22:34:34.619076+0000 mon.a (mon.0) 2322 : cluster [DBG] osdmap e919: 8 total, 8 up, 7 in 2021-03-15T22:34:36.008 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[56514]: cluster 2021-03-15T22:34:34.783041+0000 osd.3 (osd.3) 163 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:34:36.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[56514]: cluster 2021-03-15T22:34:34.843474+0000 osd.5 (osd.5) 50 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:34:36.009 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[56514]: audit 2021-03-15T22:34:35.207697+0000 mon.a (mon.0) 2323 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-03-15T22:34:36.009 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: cluster 2021-03-15T22:34:34.614647 2021-03-15T22:34:36.009 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2320 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:36.010 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:34:36.010 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: 34:34.619000+0000 mon.a (mon.0) 2321 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:34:36.010 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:34:36.010 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: -15T22:34:34.619076+0000 mon.a 2021-03-15T22:34:36.011 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: (mon.0) 2322 : cluster [DBG] osdmap e919: 8 total, 8 up, 7 in 2021-03-15T22:34:36.011 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: 2021-03-15T22:34:36.011 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:34:36.011 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: 22:34:34.783041+0000 osd.3 (osd.3 2021-03-15T22:34:36.011 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: ) 163 : cluster [DBG] 6.13 scrub ok 2021-03-15T22:34:36.012 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:34:36.012 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: 15T22:34:34.843474 2021-03-15T22:34:36.012 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 50 2021-03-15T22:34:36.012 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: : cluster [DBG] 3.2 scrub ok 2021-03-15T22:34:36.013 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: audit 2021-03-15T22:34:36.013 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:34:36.013 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: 34:35.207697+0000 mon.a 2021-03-15T22:34:36.013 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: (mon.0) 2323 : audit 2021-03-15T22:34:36.014 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:35 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-03-15T22:34:36.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: cluster 2021-03-15T22:34:34.614647+0000 mon.a (mon.0 2021-03-15T22:34:36.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: ) 2320 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:36.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:34:36.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: 03-15T22:34:34.619000+0000 mon.a ( 2021-03-15T22:34:36.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: mon.0) 2321 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:34:36.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: cluster 2021-03-15T22:34:36.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: 2021-03-15T22:34:34.619076+0000 2021-03-15T22:34:36.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: mon.a (mon.0) 2322 : cluster 2021-03-15T22:34:36.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: [DBG] osdmap e919: 8 total, 8 up, 7 in 2021-03-15T22:34:36.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:34:36.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: 03-15T22:34:34.783041+0000 2021-03-15T22:34:36.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: osd.3 (osd.3) 163 : cluster [DBG] 2021-03-15T22:34:36.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: 6.13 scrub ok 2021-03-15T22:34:36.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:34:36.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: -15T22:34:34. 2021-03-15T22:34:36.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: 843474+0000 osd.5 (osd.5) 2021-03-15T22:34:36.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: 50 : cluster [DBG] 3.2 scrub ok 2021-03-15T22:34:36.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: audit 2021-03-15T22:34:36.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: 2021-03-15T22:34 2021-03-15T22:34:36.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: :35.207697+0000 mon.a (mon.0) 2021-03-15T22:34:36.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:35 smithi125 conmon[56739]: 2323 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]: dispatch 2021-03-15T22:34:36.047 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:36.072 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:36.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:36.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:36.175 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:36.200 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:36.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:36.202 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:36.303 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:36.329 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:36.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:36.332 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:36.369 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:34:36.369 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:34:36.370 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:34:36.370 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:34:36.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.370 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6381 7.e 7:7ac381e0:::smithi07694878-24:head [write 521248~563025] snapc 221=[221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f8,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1c5,1a1,17a,176] RETRY=1 ondisk+retry+write+known_if_redirected e682)", 2021-03-15T22:34:36.371 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:31.248957+0000", 2021-03-15T22:34:36.371 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 244.89021868699999, 2021-03-15T22:34:36.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.203159267, 2021-03-15T22:34:36.371 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.371 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:36.372 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:36.372 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:36.372 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:36.372 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6381 2021-03-15T22:34:36.373 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.373 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.373 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.373 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.373 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.248957+0000", 2021-03-15T22:34:36.374 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.374 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.248957+0000", 2021-03-15T22:34:36.375 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029079399999999998 2021-03-15T22:34:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.249248+0000", 2021-03-15T22:34:36.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7869999999999999e-06 2021-03-15T22:34:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.249252+0000", 2021-03-15T22:34:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024823800000000002 2021-03-15T22:34:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.251734+0000", 2021-03-15T22:34:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3850000000000002e-06 2021-03-15T22:34:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.251743+0000", 2021-03-15T22:34:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0013638999999999999 2021-03-15T22:34:36.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253107+0000", 2021-03-15T22:34:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0289999999999994e-06 2021-03-15T22:34:36.381 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:34:36.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253116+0000", 2021-03-15T22:34:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0045301959999999 2021-03-15T22:34:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257646+0000", 2021-03-15T22:34:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000134308 2021-03-15T22:34:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for missing object", 2021-03-15T22:34:36.384 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257780+0000", 2021-03-15T22:34:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036736790999999998 2021-03-15T22:34:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294517+0000", 2021-03-15T22:34:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000130782 2021-03-15T22:34:36.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294648+0000", 2021-03-15T22:34:36.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15022935500000001 2021-03-15T22:34:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.444877+0000", 2021-03-15T22:34:36.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011121299999999999 2021-03-15T22:34:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.444988+0000", 2021-03-15T22:34:36.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019712199999999999 2021-03-15T22:34:36.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 0,5", 2021-03-15T22:34:36.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.445185+0000", 2021-03-15T22:34:36.391 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0066635430000000001 2021-03-15T22:34:36.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451849+0000", 2021-03-15T22:34:36.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3575999999999997e-05 2021-03-15T22:34:36.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:36.393 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451893+0000", 2021-03-15T22:34:36.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4709999999999999e-05 2021-03-15T22:34:36.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451917+0000", 2021-03-15T22:34:36.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7205000000000005e-05 2021-03-15T22:34:36.394 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.394 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.395 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:36.395 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451985+0000", 2021-03-15T22:34:36.395 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013219100000000001 2021-03-15T22:34:36.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.396 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.396 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452117+0000", 2021-03-15T22:34:36.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.203159267 2021-03-15T22:34:36.397 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.397 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.398 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.398 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6382 7.e 7:7ac381e0:::smithi07694878-24:head [write 1809501~596300] snapc 221=[221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f8,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1c5,1a1,17a,176] RETRY=1 ondisk+retry+write+known_if_redirected e682)", 2021-03-15T22:34:36.399 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:31.251755+0000", 2021-03-15T22:34:36.399 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 244.88742118900001, 2021-03-15T22:34:36.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2005434639999999, 2021-03-15T22:34:36.399 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.399 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:36.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:36.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:36.400 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:36.400 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6382 2021-03-15T22:34:36.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.401 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.401 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.251755+0000", 2021-03-15T22:34:36.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.402 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.251755+0000", 2021-03-15T22:34:36.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00029779900000000001 2021-03-15T22:34:36.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252053+0000", 2021-03-15T22:34:36.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.745e-06 2021-03-15T22:34:36.404 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252055+0000", 2021-03-15T22:34:36.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0090000000000001e-06 2021-03-15T22:34:36.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252063+0000", 2021-03-15T22:34:36.406 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.49e-06 2021-03-15T22:34:36.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.407 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.407 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252066+0000", 2021-03-15T22:34:36.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001074237 2021-03-15T22:34:36.407 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.407 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.408 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.408 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253140+0000", 2021-03-15T22:34:36.408 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8170000000000004e-06 2021-03-15T22:34:36.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.409 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.409 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:34:36.409 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253146+0000", 2021-03-15T22:34:36.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0046597100000001 2021-03-15T22:34:36.409 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.410 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.410 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.410 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257806+0000", 2021-03-15T22:34:36.410 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6441999999999999e-05 2021-03-15T22:34:36.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for missing object", 2021-03-15T22:34:36.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257862+0000", 2021-03-15T22:34:36.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036841170999999999 2021-03-15T22:34:36.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294703+0000", 2021-03-15T22:34:36.413 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1587999999999994e-05 2021-03-15T22:34:36.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:36.414 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294765+0000", 2021-03-15T22:34:36.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15190767999999999 2021-03-15T22:34:36.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.415 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.446673+0000", 2021-03-15T22:34:36.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000120914 2021-03-15T22:34:36.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.446794+0000", 2021-03-15T22:34:36.416 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021026000000000001 2021-03-15T22:34:36.416 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.416 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.417 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 0,5", 2021-03-15T22:34:36.417 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.447004+0000", 2021-03-15T22:34:36.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048782030000000002 2021-03-15T22:34:36.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.417 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.418 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451882+0000", 2021-03-15T22:34:36.418 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028142400000000001 2021-03-15T22:34:36.418 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.419 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.419 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452163+0000", 2021-03-15T22:34:36.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0363999999999999e-05 2021-03-15T22:34:36.419 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.420 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:36.420 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452174+0000", 2021-03-15T22:34:36.420 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6031000000000001e-05 2021-03-15T22:34:36.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.421 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:36.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452210+0000", 2021-03-15T22:34:36.421 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8579999999999996e-05 2021-03-15T22:34:36.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.422 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.422 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.422 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452298+0000", 2021-03-15T22:34:36.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2005434639999999 2021-03-15T22:34:36.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.423 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.423 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.423 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.424 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6383 7.e 7:7ac381e0:::smithi07694878-24:head [setxattr _header (58) in=7b,truncate 2521248] snapc 221=[221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f8,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1c5,1a1,17a,176] RETRY=1 ondisk+retry+write+known_if_redirected e682)", 2021-03-15T22:34:36.424 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:31.252070+0000", 2021-03-15T22:34:36.424 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 244.88710571999999, 2021-03-15T22:34:36.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.200499961, 2021-03-15T22:34:36.425 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.425 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:36.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:36.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:36.425 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:36.426 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6383 2021-03-15T22:34:36.426 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.426 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.426 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.427 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.427 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252070+0000", 2021-03-15T22:34:36.427 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.427 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.427 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.428 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.428 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252070+0000", 2021-03-15T22:34:36.428 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.331e-06 2021-03-15T22:34:36.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.429 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.429 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252072+0000", 2021-03-15T22:34:36.429 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5899999999999995e-07 2021-03-15T22:34:36.429 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.430 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.430 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.430 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252073+0000", 2021-03-15T22:34:36.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2610000000000002e-06 2021-03-15T22:34:36.431 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.431 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.431 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.431 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252076+0000", 2021-03-15T22:34:36.431 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9029999999999997e-06 2021-03-15T22:34:36.432 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.432 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.432 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.432 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252080+0000", 2021-03-15T22:34:36.432 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010880149999999999 2021-03-15T22:34:36.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.433 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.433 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.433 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253168+0000", 2021-03-15T22:34:36.433 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8159999999999999e-06 2021-03-15T22:34:36.434 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.434 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:34:36.434 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253174+0000", 2021-03-15T22:34:36.434 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0047103100000001 2021-03-15T22:34:36.435 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.435 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257884+0000", 2021-03-15T22:34:36.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3068000000000001e-05 2021-03-15T22:34:36.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.436 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for missing object", 2021-03-15T22:34:36.436 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257937+0000", 2021-03-15T22:34:36.437 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.036854445999999999 2021-03-15T22:34:36.437 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.437 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294791+0000", 2021-03-15T22:34:36.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000147114 2021-03-15T22:34:36.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.438 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.438 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:36.439 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.294938+0000", 2021-03-15T22:34:36.439 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.156267234 2021-03-15T22:34:36.439 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451206+0000", 2021-03-15T22:34:36.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6873999999999995e-05 2021-03-15T22:34:36.440 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.440 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.441 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451293+0000", 2021-03-15T22:34:36.441 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014486499999999999 2021-03-15T22:34:36.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.442 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 0,5", 2021-03-15T22:34:36.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451437+0000", 2021-03-15T22:34:36.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00087174500000000005 2021-03-15T22:34:36.442 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.442 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.443 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:36.443 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452309+0000", 2021-03-15T22:34:36.443 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8932e-05 2021-03-15T22:34:36.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452338+0000", 2021-03-15T22:34:36.444 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2397999999999997e-05 2021-03-15T22:34:36.444 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.445 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.445 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452370+0000", 2021-03-15T22:34:36.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9596000000000003e-05 2021-03-15T22:34:36.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.446 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:36.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452430+0000", 2021-03-15T22:34:36.446 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014029400000000001 2021-03-15T22:34:36.446 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.447 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.447 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.447 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452570+0000", 2021-03-15T22:34:36.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.200499961 2021-03-15T22:34:36.448 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.448 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.448 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.448 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.449 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6384 7.e 7:7ac381e0:::smithi07694878-24:head [read 0~1] snapc 0=[] RETRY=1 ondisk+retry+read+rwordered+known_if_redirected e682)", 2021-03-15T22:34:36.449 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:31.252084+0000", 2021-03-15T22:34:36.449 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 244.88709182100001, 2021-03-15T22:34:36.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.200996113, 2021-03-15T22:34:36.449 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.450 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:36.450 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:36.450 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:36.450 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:36.450 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6384 2021-03-15T22:34:36.451 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.451 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.451 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.451 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.452 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252084+0000", 2021-03-15T22:34:36.452 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252084+0000", 2021-03-15T22:34:36.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9100000000000002e-07 2021-03-15T22:34:36.453 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.453 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.454 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.454 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252085+0000", 2021-03-15T22:34:36.454 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3430000000000001e-06 2021-03-15T22:34:36.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.455 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252087+0000", 2021-03-15T22:34:36.455 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2700000000000004e-06 2021-03-15T22:34:36.455 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.456 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.456 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.456 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252092+0000", 2021-03-15T22:34:36.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.298e-06 2021-03-15T22:34:36.457 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.252095+0000", 2021-03-15T22:34:36.457 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010992179999999999 2021-03-15T22:34:36.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.458 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.458 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.458 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253194+0000", 2021-03-15T22:34:36.459 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0279999999999997e-06 2021-03-15T22:34:36.459 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:34:36.459 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:31.253200+0000", 2021-03-15T22:34:36.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0047584869999999 2021-03-15T22:34:36.460 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.460 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.460 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.461 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.257959+0000", 2021-03-15T22:34:36.461 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.897e-05 2021-03-15T22:34:36.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for missing object", 2021-03-15T22:34:36.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.258008+0000", 2021-03-15T22:34:36.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.037164928 2021-03-15T22:34:36.462 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.462 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.462 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.463 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.295173+0000", 2021-03-15T22:34:36.463 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8801999999999994e-05 2021-03-15T22:34:36.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:36.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.295272+0000", 2021-03-15T22:34:36.464 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15646575800000001 2021-03-15T22:34:36.464 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.464 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.465 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.465 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451737+0000", 2021-03-15T22:34:36.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2378000000000004e-05 2021-03-15T22:34:36.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.465 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:36.466 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.451810+0000", 2021-03-15T22:34:36.466 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000797925 2021-03-15T22:34:36.466 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.467 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452608+0000", 2021-03-15T22:34:36.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000103439 2021-03-15T22:34:36.467 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.467 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.468 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452711+0000", 2021-03-15T22:34:36.468 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00036937800000000001 2021-03-15T22:34:36.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.469 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453080+0000", 2021-03-15T22:34:36.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.200996113 2021-03-15T22:34:36.469 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.470 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.470 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.470 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.470 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.470 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:2177 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94751391775488 gen 12] snapc 0=[] ondisk+write+known_if_redirected e682)", 2021-03-15T22:34:36.471 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.138277+0000", 2021-03-15T22:34:36.471 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 244.000898788, 2021-03-15T22:34:36.471 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.13344044399999999, 2021-03-15T22:34:36.471 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.472 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:36.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:36.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:34:36.472 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:34:36.472 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2177 2021-03-15T22:34:36.473 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.473 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.473 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.473 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.473 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138277+0000", 2021-03-15T22:34:36.474 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.474 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.474 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.474 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.474 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138277+0000", 2021-03-15T22:34:36.475 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5209999999999999e-06 2021-03-15T22:34:36.475 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.475 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.475 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.475 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138279+0000", 2021-03-15T22:34:36.476 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0579999999999999e-06 2021-03-15T22:34:36.476 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.476 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.476 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.477 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138283+0000", 2021-03-15T22:34:36.477 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1531e-05 2021-03-15T22:34:36.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.477 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.477 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.478 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138294+0000", 2021-03-15T22:34:36.478 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.02e-06 2021-03-15T22:34:36.478 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.479 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138303+0000", 2021-03-15T22:34:36.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2645999999999995e-05 2021-03-15T22:34:36.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.479 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138376+0000", 2021-03-15T22:34:36.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1537e-05 2021-03-15T22:34:36.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:34:36.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138388+0000", 2021-03-15T22:34:36.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.12075796699999999 2021-03-15T22:34:36.481 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.259146+0000", 2021-03-15T22:34:36.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017358400000000001 2021-03-15T22:34:36.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.259319+0000", 2021-03-15T22:34:36.483 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012398579999999999 2021-03-15T22:34:36.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.271718+0000", 2021-03-15T22:34:36.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.13344044399999999 2021-03-15T22:34:36.485 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.485 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.485 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.486 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14397.0:2179 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94751391775488 gen 13] snapc 0=[] ondisk+write+known_if_redirected e682)", 2021-03-15T22:34:36.486 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.138324+0000", 2021-03-15T22:34:36.486 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 244.00085252299999, 2021-03-15T22:34:36.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.133677823, 2021-03-15T22:34:36.487 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.487 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:36.487 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:36.487 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14397", 2021-03-15T22:34:36.488 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:34:36.488 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 2179 2021-03-15T22:34:36.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.488 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138324+0000", 2021-03-15T22:34:36.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.489 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138324+0000", 2021-03-15T22:34:36.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.001e-06 2021-03-15T22:34:36.490 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138325+0000", 2021-03-15T22:34:36.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.359e-06 2021-03-15T22:34:36.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138326+0000", 2021-03-15T22:34:36.492 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7190000000000001e-06 2021-03-15T22:34:36.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.493 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138334+0000", 2021-03-15T22:34:36.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1110000000000001e-06 2021-03-15T22:34:36.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138338+0000", 2021-03-15T22:34:36.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6509999999999998e-05 2021-03-15T22:34:36.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138414+0000", 2021-03-15T22:34:36.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0434999999999999e-05 2021-03-15T22:34:36.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.496 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:34:36.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.138425+0000", 2021-03-15T22:34:36.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.121430683 2021-03-15T22:34:36.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.497 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.259855+0000", 2021-03-15T22:34:36.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3189000000000003e-05 2021-03-15T22:34:36.498 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.259899+0000", 2021-03-15T22:34:36.499 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.012102816000000001 2021-03-15T22:34:36.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.499 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.499 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.272001+0000", 2021-03-15T22:34:36.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.133677823 2021-03-15T22:34:36.500 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.500 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.500 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.501 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.501 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.501 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGPush(4.6 683/682 [PushOp(4:6192870f:::obj_delete_at_hint.0000000053:head, version: 555'22, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:6192870f:::obj_delete_at_hint.0000000053:head@555'22, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-15T22:34:36.501 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.282830+0000", 2021-03-15T22:34:36.501 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 243.85634602499999, 2021-03-15T22:34:36.502 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075199040999999994, 2021-03-15T22:34:36.502 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.502 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:36.502 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282830+0000", 2021-03-15T22:34:36.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.504 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282830+0000", 2021-03-15T22:34:36.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1109999999999999e-06 2021-03-15T22:34:36.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282833+0000", 2021-03-15T22:34:36.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0159999999999999e-05 2021-03-15T22:34:36.506 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282843+0000", 2021-03-15T22:34:36.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6823000000000001e-05 2021-03-15T22:34:36.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282860+0000", 2021-03-15T22:34:36.508 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8099999999999992e-06 2021-03-15T22:34:36.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.282870+0000", 2021-03-15T22:34:36.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074966989999999997 2021-03-15T22:34:36.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.510 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.357837+0000", 2021-03-15T22:34:36.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2856e-05 2021-03-15T22:34:36.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.357850+0000", 2021-03-15T22:34:36.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017929099999999999 2021-03-15T22:34:36.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.512 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.358029+0000", 2021-03-15T22:34:36.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075199040999999994 2021-03-15T22:34:36.512 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.512 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.513 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.513 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGPush(4.6 683/682 [PushOp(4:63d917ed:::obj_delete_at_hint.0000000065:head, version: 555'24, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:63d917ed:::obj_delete_at_hint.0000000065:head@555'24, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-15T22:34:36.513 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.368712+0000", 2021-03-15T22:34:36.514 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 243.77046449599999, 2021-03-15T22:34:36.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083794849000000005, 2021-03-15T22:34:36.514 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.514 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:36.515 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368712+0000", 2021-03-15T22:34:36.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368712+0000", 2021-03-15T22:34:36.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.98e-06 2021-03-15T22:34:36.517 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368716+0000", 2021-03-15T22:34:36.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2875e-05 2021-03-15T22:34:36.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368729+0000", 2021-03-15T22:34:36.519 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1965000000000001e-05 2021-03-15T22:34:36.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368750+0000", 2021-03-15T22:34:36.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2337999999999999e-05 2021-03-15T22:34:36.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.521 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.368763+0000", 2021-03-15T22:34:36.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083630726000000002 2021-03-15T22:34:36.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452394+0000", 2021-03-15T22:34:36.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3420000000000008e-06 2021-03-15T22:34:36.522 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.522 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.522 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.523 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452402+0000", 2021-03-15T22:34:36.523 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000104623 2021-03-15T22:34:36.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.524 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.524 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.452506+0000", 2021-03-15T22:34:36.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083794849000000005 2021-03-15T22:34:36.524 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.525 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.525 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.525 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.525 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGPush(4.6 683/682 [PushOp(4:64735f45:::obj_delete_at_hint.0000000083:head, version: 555'26, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:64735f45:::obj_delete_at_hint.0000000083:head@555'26, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-15T22:34:36.526 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.453341+0000", 2021-03-15T22:34:36.526 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 243.685834965, 2021-03-15T22:34:36.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077331792999999996, 2021-03-15T22:34:36.526 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.527 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:36.527 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453341+0000", 2021-03-15T22:34:36.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.529 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453341+0000", 2021-03-15T22:34:36.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1270000000000001e-06 2021-03-15T22:34:36.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.529 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453343+0000", 2021-03-15T22:34:36.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8979999999999996e-06 2021-03-15T22:34:36.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453351+0000", 2021-03-15T22:34:36.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4432e-05 2021-03-15T22:34:36.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.532 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453366+0000", 2021-03-15T22:34:36.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5170000000000002e-06 2021-03-15T22:34:36.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.533 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.453373+0000", 2021-03-15T22:34:36.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077141928999999998 2021-03-15T22:34:36.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.530515+0000", 2021-03-15T22:34:36.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2335e-05 2021-03-15T22:34:36.535 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.535 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.530527+0000", 2021-03-15T22:34:36.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014555500000000001 2021-03-15T22:34:36.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.530673+0000", 2021-03-15T22:34:36.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077331792999999996 2021-03-15T22:34:36.537 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.537 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.537 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.538 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGPush(4.6 683/682 [PushOp(4:624175ac:::obj_delete_at_hint.0000000091:head, version: 555'28, data_included: [], data_size: 0, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(4:624175ac:::obj_delete_at_hint.0000000091:head@555'28, size: 0, copy_subset: [], clone_subset: {}, snapset: 0=[]:{}, object_exist: 1), after_progress: ObjectRecoveryProgress(!first, data_recovered_to:0, data_complete:true, omap_recovered_to:, omap_complete:true, error:false), before_progress: ObjectRecoveryProgress(first, data_recovered_to:0, data_complete:false, omap_recovered_to:, omap_complete:true, error:false))])", 2021-03-15T22:34:36.538 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:32.531733+0000", 2021-03-15T22:34:36.538 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 243.60744284, 2021-03-15T22:34:36.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.085164504000000002, 2021-03-15T22:34:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531733+0000", 2021-03-15T22:34:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531733+0000", 2021-03-15T22:34:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5429999999999998e-06 2021-03-15T22:34:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531737+0000", 2021-03-15T22:34:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0711e-05 2021-03-15T22:34:36.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531748+0000", 2021-03-15T22:34:36.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0208e-05 2021-03-15T22:34:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531768+0000", 2021-03-15T22:34:36.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2381999999999999e-05 2021-03-15T22:34:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.531780+0000", 2021-03-15T22:34:36.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.085010464999999993 2021-03-15T22:34:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.546 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.616791+0000", 2021-03-15T22:34:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2209999999999997e-06 2021-03-15T22:34:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.547 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.616799+0000", 2021-03-15T22:34:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8974000000000005e-05 2021-03-15T22:34:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.548 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:32.616898+0000", 2021-03-15T22:34:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.085164504000000002 2021-03-15T22:34:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.549 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.550 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.550 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:845 7.5 e772/685 7:ac84e656:::smithi07694878-31:1f0 v 772'640, mlcod=770'636)", 2021-03-15T22:34:36.550 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:02.252138+0000", 2021-03-15T22:34:36.551 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 153.88703841399999, 2021-03-15T22:34:36.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075996994999999998, 2021-03-15T22:34:36.551 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.551 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:36.552 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.552 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.252138+0000", 2021-03-15T22:34:36.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.553 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.252138+0000", 2021-03-15T22:34:36.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.019e-06 2021-03-15T22:34:36.554 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.252142+0000", 2021-03-15T22:34:36.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4530000000000004e-06 2021-03-15T22:34:36.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.252146+0000", 2021-03-15T22:34:36.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.072793821999999994 2021-03-15T22:34:36.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.324940+0000", 2021-03-15T22:34:36.557 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8179999999999998e-06 2021-03-15T22:34:36.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.324947+0000", 2021-03-15T22:34:36.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001865684 2021-03-15T22:34:36.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.559 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.326813+0000", 2021-03-15T22:34:36.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3147999999999999e-05 2021-03-15T22:34:36.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.326836+0000", 2021-03-15T22:34:36.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012622849999999999 2021-03-15T22:34:36.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.561 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:36.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328098+0000", 2021-03-15T22:34:36.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6766e-05 2021-03-15T22:34:36.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328135+0000", 2021-03-15T22:34:36.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.075996994999999998 2021-03-15T22:34:36.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.563 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.563 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.564 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.2.0:161 7.8 e772/681 ondisk, result = 0)", 2021-03-15T22:34:36.564 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:02.254556+0000", 2021-03-15T22:34:36.564 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 153.884619824, 2021-03-15T22:34:36.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071612197000000002, 2021-03-15T22:34:36.565 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.565 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:36.565 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.565 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.566 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.566 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.254556+0000", 2021-03-15T22:34:36.566 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.566 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.566 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.567 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.254556+0000", 2021-03-15T22:34:36.567 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9999999999999998e-06 2021-03-15T22:34:36.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.568 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.254560+0000", 2021-03-15T22:34:36.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9770000000000002e-06 2021-03-15T22:34:36.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.569 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.254564+0000", 2021-03-15T22:34:36.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068971258999999993 2021-03-15T22:34:36.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.570 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.570 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.323536+0000", 2021-03-15T22:34:36.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1450999999999999e-05 2021-03-15T22:34:36.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.323547+0000", 2021-03-15T22:34:36.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0026004259999999999 2021-03-15T22:34:36.572 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.326147+0000", 2021-03-15T22:34:36.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5440000000000004e-06 2021-03-15T22:34:36.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.326157+0000", 2021-03-15T22:34:36.574 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.154e-05 2021-03-15T22:34:36.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.326169+0000", 2021-03-15T22:34:36.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071612197000000002 2021-03-15T22:34:36.575 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.576 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.576 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.577 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.3.0:1988 7.a e790/684 7:5bc7f316:::smithi07694878-38:261 v 790'1367, mlcod=790'1363)", 2021-03-15T22:34:36.577 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:20.354804+0000", 2021-03-15T22:34:36.577 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 135.78437206199999, 2021-03-15T22:34:36.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042477930999999997, 2021-03-15T22:34:36.577 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.578 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:36.578 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.578 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354804+0000", 2021-03-15T22:34:36.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.579 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.579 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.579 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.580 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354804+0000", 2021-03-15T22:34:36.580 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4660000000000002e-06 2021-03-15T22:34:36.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.580 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.581 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354812+0000", 2021-03-15T22:34:36.581 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5149999999999997e-06 2021-03-15T22:34:36.581 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.581 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.582 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.582 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354817+0000", 2021-03-15T22:34:36.582 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2300999999999999e-05 2021-03-15T22:34:36.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.582 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.583 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.583 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354839+0000", 2021-03-15T22:34:36.583 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4779999999999997e-06 2021-03-15T22:34:36.583 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.583 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.584 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.584 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354849+0000", 2021-03-15T22:34:36.584 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3566000000000002e-05 2021-03-15T22:34:36.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.585 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.585 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354902+0000", 2021-03-15T22:34:36.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0645999999999997e-05 2021-03-15T22:34:36.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.585 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.586 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.586 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.354943+0000", 2021-03-15T22:34:36.586 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042287531000000003 2021-03-15T22:34:36.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.587 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:36.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.397231+0000", 2021-03-15T22:34:36.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1428000000000003e-05 2021-03-15T22:34:36.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.588 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:20.397282+0000", 2021-03-15T22:34:36.588 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042477930999999997 2021-03-15T22:34:36.589 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.589 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.589 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.590 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7840 7.8 7:1822d391:::smithi07694878-47:head [append 0~0] snapc 2b6=[2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,293,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e830)", 2021-03-15T22:34:36.590 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.384447+0000", 2021-03-15T22:34:36.590 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.754729010000005, 2021-03-15T22:34:36.590 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043748464000000001, 2021-03-15T22:34:36.591 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.591 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:36.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:36.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:36.591 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:36.592 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7840 2021-03-15T22:34:36.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.592 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384447+0000", 2021-03-15T22:34:36.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.593 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384447+0000", 2021-03-15T22:34:36.594 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0560000000000001e-06 2021-03-15T22:34:36.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384453+0000", 2021-03-15T22:34:36.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.609e-06 2021-03-15T22:34:36.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.596 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384459+0000", 2021-03-15T22:34:36.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6245999999999999e-05 2021-03-15T22:34:36.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384485+0000", 2021-03-15T22:34:36.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7729999999999994e-06 2021-03-15T22:34:36.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.598 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384495+0000", 2021-03-15T22:34:36.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8068e-05 2021-03-15T22:34:36.599 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.599 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.599 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.599 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384553+0000", 2021-03-15T22:34:36.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6229000000000004e-05 2021-03-15T22:34:36.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.600 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.600 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384639+0000", 2021-03-15T22:34:36.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022088199999999999 2021-03-15T22:34:36.601 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.601 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.601 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:34:36.601 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384860+0000", 2021-03-15T22:34:36.602 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0018978619999999999 2021-03-15T22:34:36.602 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.602 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.602 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.386758+0000", 2021-03-15T22:34:36.603 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8676e-05 2021-03-15T22:34:36.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.604 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.386797+0000", 2021-03-15T22:34:36.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.041256679999999997 2021-03-15T22:34:36.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:36.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.428053+0000", 2021-03-15T22:34:36.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4635000000000005e-05 2021-03-15T22:34:36.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.606 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:36.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.428138+0000", 2021-03-15T22:34:36.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7747999999999997e-05 2021-03-15T22:34:36.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.428196+0000", 2021-03-15T22:34:36.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043748464000000001 2021-03-15T22:34:36.607 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.608 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.608 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.609 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7841 7.8 7:1822d391:::smithi07694878-47:head [rollback 2b5] snapc 2b6=[2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,293,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e830)", 2021-03-15T22:34:36.609 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.384512+0000", 2021-03-15T22:34:36.609 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.754664625999993, 2021-03-15T22:34:36.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.055918190999999999, 2021-03-15T22:34:36.609 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.610 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:36.610 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:36.610 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:36.610 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:36.611 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7841 2021-03-15T22:34:36.611 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.611 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.611 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384512+0000", 2021-03-15T22:34:36.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.613 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384512+0000", 2021-03-15T22:34:36.613 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8169999999999999e-06 2021-03-15T22:34:36.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.613 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.614 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384513+0000", 2021-03-15T22:34:36.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6920000000000001e-06 2021-03-15T22:34:36.614 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.614 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.615 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.615 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384515+0000", 2021-03-15T22:34:36.615 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6319999999999996e-06 2021-03-15T22:34:36.615 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.616 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.616 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384521+0000", 2021-03-15T22:34:36.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8739999999999999e-06 2021-03-15T22:34:36.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384524+0000", 2021-03-15T22:34:36.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010356950000000001 2021-03-15T22:34:36.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.385559+0000", 2021-03-15T22:34:36.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010356700000000001 2021-03-15T22:34:36.618 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.619 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.619 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.385663+0000", 2021-03-15T22:34:36.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00017394099999999999 2021-03-15T22:34:36.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.620 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:34:36.620 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.385837+0000", 2021-03-15T22:34:36.620 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001173363 2021-03-15T22:34:36.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.621 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.621 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.387010+0000", 2021-03-15T22:34:36.621 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3553999999999998e-05 2021-03-15T22:34:36.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.622 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.387034+0000", 2021-03-15T22:34:36.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053285379000000001 2021-03-15T22:34:36.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.623 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:36.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440319+0000", 2021-03-15T22:34:36.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3157000000000005e-05 2021-03-15T22:34:36.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:36.624 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440382+0000", 2021-03-15T22:34:36.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7519999999999999e-05 2021-03-15T22:34:36.625 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.625 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.626 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440430+0000", 2021-03-15T22:34:36.626 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.055918190999999999 2021-03-15T22:34:36.626 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.626 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.627 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.627 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.627 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.627 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7842 7.8 7:1822d391:::smithi07694878-47:head [append 0~0] snapc 2b6=[2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,293,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e830)", 2021-03-15T22:34:36.627 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.384542+0000", 2021-03-15T22:34:36.628 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.754634328999998, 2021-03-15T22:34:36.628 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.055994378999999997, 2021-03-15T22:34:36.628 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.628 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:36.628 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:36.629 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:36.629 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:36.629 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7842 2021-03-15T22:34:36.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.630 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.630 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384542+0000", 2021-03-15T22:34:36.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384542+0000", 2021-03-15T22:34:36.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3289999999999999e-06 2021-03-15T22:34:36.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.632 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.632 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384543+0000", 2021-03-15T22:34:36.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1179999999999999e-06 2021-03-15T22:34:36.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.633 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.633 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384544+0000", 2021-03-15T22:34:36.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9259999999999997e-06 2021-03-15T22:34:36.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384550+0000", 2021-03-15T22:34:36.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.895e-06 2021-03-15T22:34:36.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.635 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.384554+0000", 2021-03-15T22:34:36.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001725709 2021-03-15T22:34:36.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.637 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.386280+0000", 2021-03-15T22:34:36.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4659000000000006e-05 2021-03-15T22:34:36.637 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.637 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.638 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.386344+0000", 2021-03-15T22:34:36.638 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000121128 2021-03-15T22:34:36.638 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.638 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.639 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 4,7", 2021-03-15T22:34:36.639 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.386466+0000", 2021-03-15T22:34:36.639 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00072824599999999999 2021-03-15T22:34:36.639 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.640 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.640 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.640 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.387194+0000", 2021-03-15T22:34:36.640 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0037000000000001e-05 2021-03-15T22:34:36.641 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.641 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.641 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.641 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.387214+0000", 2021-03-15T22:34:36.641 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.053224652999999997 2021-03-15T22:34:36.642 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.642 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.642 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:36.642 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440439+0000", 2021-03-15T22:34:36.642 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0268000000000001e-05 2021-03-15T22:34:36.643 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.643 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.643 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:36.643 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440489+0000", 2021-03-15T22:34:36.644 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7410999999999997e-05 2021-03-15T22:34:36.644 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.644 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.644 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.644 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440536+0000", 2021-03-15T22:34:36.645 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.055994378999999997 2021-03-15T22:34:36.645 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.645 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.645 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.646 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.646 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.646 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:7843 7.5 e830/685 7:ac84e656:::smithi07694878-31:head v 830'746, mlcod=827'743)", 2021-03-15T22:34:36.646 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.392156+0000", 2021-03-15T22:34:36.646 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.747019668999997, 2021-03-15T22:34:36.647 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048256610999999998, 2021-03-15T22:34:36.647 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.647 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:36.647 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.647 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.648 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.648 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.392156+0000", 2021-03-15T22:34:36.648 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.648 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.649 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.649 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.649 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.392156+0000", 2021-03-15T22:34:36.649 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00088703899999999995 2021-03-15T22:34:36.650 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.650 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.650 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.650 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393044+0000", 2021-03-15T22:34:36.651 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8550000000000002e-06 2021-03-15T22:34:36.651 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.651 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.651 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.651 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393046+0000", 2021-03-15T22:34:36.652 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8155000000000002e-05 2021-03-15T22:34:36.652 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.652 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.652 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.653 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393065+0000", 2021-03-15T22:34:36.653 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5960000000000003e-06 2021-03-15T22:34:36.653 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.653 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.653 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.654 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393070+0000", 2021-03-15T22:34:36.654 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0427999999999999e-05 2021-03-15T22:34:36.654 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.654 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.655 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.655 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393121+0000", 2021-03-15T22:34:36.655 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4057999999999999e-05 2021-03-15T22:34:36.655 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.655 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.656 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.656 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.393145+0000", 2021-03-15T22:34:36.656 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.047235906000000001 2021-03-15T22:34:36.656 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.656 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.657 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:36.657 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440381+0000", 2021-03-15T22:34:36.657 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2573999999999997e-05 2021-03-15T22:34:36.657 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.657 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.658 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.658 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440413+0000", 2021-03-15T22:34:36.658 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048256610999999998 2021-03-15T22:34:36.658 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.659 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.659 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.659 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.659 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.660 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:7844 7.5 e830/685 7:ac84e656:::smithi07694878-31:head v 830'747, mlcod=827'743)", 2021-03-15T22:34:36.660 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.399774+0000", 2021-03-15T22:34:36.660 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.739402511999998, 2021-03-15T22:34:36.660 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040654153999999998, 2021-03-15T22:34:36.660 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.661 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:36.661 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.661 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.661 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.662 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.399774+0000", 2021-03-15T22:34:36.662 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.662 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.662 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.662 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.663 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.399774+0000", 2021-03-15T22:34:36.663 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001083351 2021-03-15T22:34:36.663 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.663 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.663 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.664 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400857+0000", 2021-03-15T22:34:36.664 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8080000000000001e-06 2021-03-15T22:34:36.664 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.664 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.665 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.665 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400863+0000", 2021-03-15T22:34:36.665 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3723000000000001e-05 2021-03-15T22:34:36.665 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.665 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.666 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.666 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400877+0000", 2021-03-15T22:34:36.666 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6489999999999996e-06 2021-03-15T22:34:36.666 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.667 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.667 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.667 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400882+0000", 2021-03-15T22:34:36.667 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1548000000000003e-05 2021-03-15T22:34:36.667 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.668 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.668 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.668 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400924+0000", 2021-03-15T22:34:36.668 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4221e-05 2021-03-15T22:34:36.668 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.669 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.669 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.669 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.400938+0000", 2021-03-15T22:34:36.669 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.039478164000000003 2021-03-15T22:34:36.669 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.670 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.670 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:36.670 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440416+0000", 2021-03-15T22:34:36.670 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.169e-05 2021-03-15T22:34:36.670 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.671 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.671 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.671 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440428+0000", 2021-03-15T22:34:36.671 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.040654153999999998 2021-03-15T22:34:36.672 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.672 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.672 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.672 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.672 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.673 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "pg_update_log_missing(7.c epoch 830/740 rep_tid 2224 entries 830'1139 (0'0) error 7:3bc1c17e:::smithi07694878-8:head by client.24517.0:7848 0.000000 0 ObjectCleanRegions clean_offsets: [0~18446744073709551615], clean_omap: 1, new_object: 0 trim_to 0'0 roll_forward_to 826'1136)", 2021-03-15T22:34:36.673 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.406310+0000", 2021-03-15T22:34:36.673 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 95.732865662999998, 2021-03-15T22:34:36.673 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034274735000000001, 2021-03-15T22:34:36.673 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.674 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:34:36.674 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.674 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.674 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.674 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406310+0000", 2021-03-15T22:34:36.675 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.675 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.675 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.675 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.675 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406310+0000", 2021-03-15T22:34:36.676 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3359999999999997e-06 2021-03-15T22:34:36.676 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.676 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.676 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.677 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406315+0000", 2021-03-15T22:34:36.677 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3603e-05 2021-03-15T22:34:36.677 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.677 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.677 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.678 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406328+0000", 2021-03-15T22:34:36.678 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7805999999999999e-05 2021-03-15T22:34:36.678 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.678 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.678 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.679 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406346+0000", 2021-03-15T22:34:36.679 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6640000000000001e-06 2021-03-15T22:34:36.679 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.679 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.679 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.680 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406353+0000", 2021-03-15T22:34:36.680 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2898999999999995e-05 2021-03-15T22:34:36.680 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.680 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.681 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.681 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.406416+0000", 2021-03-15T22:34:36.681 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034169427000000002 2021-03-15T22:34:36.681 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.681 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.682 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.682 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440585+0000", 2021-03-15T22:34:36.682 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.034274735000000001 2021-03-15T22:34:36.682 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.682 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7884 7.c 7:3fac7ed7:::smithi07694878-41:head [delete] snapc 2b8=[2b8,2b7,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+peerstat_old+known_if_redirected e834)", 2021-03-15T22:34:36.683 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:04.336744+0000", 2021-03-15T22:34:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 91.802432272999994, 2021-03-15T22:34:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0232414750000001, 2021-03-15T22:34:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:36.684 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7884 2021-03-15T22:34:36.685 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.336744+0000", 2021-03-15T22:34:36.686 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.336744+0000", 2021-03-15T22:34:36.687 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4490000000000002e-06 2021-03-15T22:34:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.336746+0000", 2021-03-15T22:34:36.688 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.717e-06 2021-03-15T22:34:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.336749+0000", 2021-03-15T22:34:36.689 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1549000000000001e-05 2021-03-15T22:34:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:36.690 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.336761+0000", 2021-03-15T22:34:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.5010000000000001e-06 2021-03-15T22:34:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:36.691 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.336767+0000", 2021-03-15T22:34:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8868e-05 2021-03-15T22:34:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.692 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.336816+0000", 2021-03-15T22:34:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1041e-05 2021-03-15T22:34:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:34:36.693 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.336827+0000", 2021-03-15T22:34:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.018411696 2021-03-15T22:34:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:36.694 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.355239+0000", 2021-03-15T22:34:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00026677899999999997 2021-03-15T22:34:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.695 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.355505+0000", 2021-03-15T22:34:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000304433 2021-03-15T22:34:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.696 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,6,7", 2021-03-15T22:34:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.355810+0000", 2021-03-15T22:34:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001384279 2021-03-15T22:34:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.697 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.357194+0000", 2021-03-15T22:34:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00035986999999999998 2021-03-15T22:34:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.698 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.357554+0000", 2021-03-15T22:34:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011468699999999999 2021-03-15T22:34:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.699 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.357669+0000", 2021-03-15T22:34:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021349530000000002 2021-03-15T22:34:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.700 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.359804+0000", 2021-03-15T22:34:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4161e-05 2021-03-15T22:34:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.701 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.359848+0000", 2021-03-15T22:34:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013749199999999999 2021-03-15T22:34:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:36.702 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.359985+0000", 2021-03-15T22:34:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0232414750000001 2021-03-15T22:34:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.703 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:36.704 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:36.704 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:34:36.705 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:36.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[62156]: cluster 2021-03-15T22:34:35. 2021-03-15T22:34:36.706 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[62156]: 206890+0000 mgr.y (mgr.14144) 1037 : cluster [DBG] pgmap v1526: 158 pgs: 5 peering, 10 unknown, 1 active+remapped+backfill_toofull, 142 active+clean; 710 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 776 KiB/s wr, 2 op/s 2021-03-15T22:34:36.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:34:36.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[62156]: -03-15T22:34:35.410481+0000 osd.0 (osd.0) 147 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:34:36.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[62156]: audit 2021-03-15T22:34:35 2021-03-15T22:34:36.707 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[62156]: .634388+0000 mon.a (mon.0) 2324 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]': finished 2021-03-15T22:34:36.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:34:36.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[62156]: T22:34:35.634458+0000 mon.a (mon.0) 2325 : 2021-03-15T22:34:36.708 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[62156]: cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2021-03-15T22:34:36.708 INFO:tasks.rados.rados.0.smithi076.stdout:3464: done (2 left) 2021-03-15T22:34:36.709 INFO:tasks.rados.rados.0.smithi076.stdout:3465: done (1 left) 2021-03-15T22:34:36.709 INFO:tasks.rados.rados.0.smithi076.stdout:3466: done (0 left) 2021-03-15T22:34:36.709 INFO:tasks.rados.rados.0.smithi076.stdout:3467: rollback oid 15 current snap is 412 2021-03-15T22:34:36.709 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 15 to 394 2021-03-15T22:34:36.710 INFO:tasks.rados.rados.0.smithi076.stdout:3468: snap_remove snap 339 2021-03-15T22:34:36.730 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:36.732 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:36.833 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:36.857 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:36.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:36.859 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:36.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[56514]: cluster 2021-03-15T22:34:35.206890+0000 mgr.y (mgr.14144) 1037 : cluster [DBG] pgmap v1526: 158 pgs: 5 peering, 10 unknown, 1 active+remapped+backfill_toofull, 142 active+clean; 710 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 776 KiB/s wr, 2 op/s 2021-03-15T22:34:36.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[56514]: cluster 2021-03-15T22:34:35.410481+0000 osd.0 (osd.0) 147 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:34:36.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[56514]: audit 2021-03-15T22:34:35.634388+0000 mon.a (mon.0) 2324 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]': finished 2021-03-15T22:34:36.934 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:36 smithi076 conmon[56514]: cluster 2021-03-15T22:34:35.634458+0000 mon.a (mon.0) 2325 : cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2021-03-15T22:34:36.960 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:36.983 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:36.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:36.985 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:37.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:36 smithi125 conmon[56739]: cluster 2021-03-15T22:34:35.206890+0000 mgr.y (mgr.14144) 1037 : cluster [DBG] pgmap v1526: 158 pgs: 5 peering, 10 unknown, 1 active+remapped+backfill_toofull, 142 active+clean; 710 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 776 KiB/s wr, 2 op/s 2021-03-15T22:34:37.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:36 smithi125 conmon[56739]: cluster 2021-03-15T22:34:35.410481+0000 osd.0 (osd.0) 147 : cluster [DBG] 3.1 scrub ok 2021-03-15T22:34:37.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:36 smithi125 conmon[56739]: audit 2021-03-15T22:34:35.634388+0000 mon.a (mon.0) 2324 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "42"}]': finished 2021-03-15T22:34:37.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:36 smithi125 conmon[56739]: cluster 2021-03-15T22:34:35.634458+0000 mon.a (mon.0 2021-03-15T22:34:37.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:36 smithi125 conmon[56739]: ) 2325 : cluster [DBG] osdmap e920: 8 total, 8 up, 7 in 2021-03-15T22:34:37.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:34:37.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:37.112 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:37.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:37.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:37.214 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:37.240 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:37.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:37.241 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:34:37.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:37.258 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:34:37.276 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:34:37.294 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:34:37.318 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:34:37.337 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:34:37.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:37.349 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:37.356 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:34:37.369 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:37.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:37.370 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:37.374 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:34:37.376 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:34:37.376 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:34:37.376 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:34:37.376 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:34:37.377 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:34:37.377 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:34:37.377 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:34:37.377 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:34:37.407 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:34:37.471 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:37.496 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:37.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:37.498 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:37.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:37.624 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:37.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:37.626 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:37.653 INFO:tasks.rados.rados.0.smithi076.stdout:3468: done (1 left) 2021-03-15T22:34:37.654 INFO:tasks.rados.rados.0.smithi076.stdout:3469: snap_create 2021-03-15T22:34:37.654 INFO:tasks.rados.rados.0.smithi076.stdout:3467: finishing rollback tid 0 to smithi07694878-15 2021-03-15T22:34:37.655 INFO:tasks.rados.rados.0.smithi076.stdout:3467: finishing rollback tid 1 to smithi07694878-15 2021-03-15T22:34:37.655 INFO:tasks.rados.rados.0.smithi076.stdout:3467: finishing rollback tid 2 to smithi07694878-15 2021-03-15T22:34:37.655 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 2804 (ObjNum 1070 snap 364 seq_num 1070) dirty exists 2021-03-15T22:34:37.675 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 7, 0, 3, 5, 6] out_osds: [2] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:34:37.676 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:34:37.676 INFO:tasks.thrashosds.thrasher:Reweighting osd 3 to 0.7640074395265278 2021-03-15T22:34:37.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd reweight 3 0.7640074395265278 2021-03-15T22:34:37.698 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:37 smithi076 conmon[56514]: cluster 2021-03-15T22:34:36.438536+0000 osd.7 2021-03-15T22:34:37.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:37 smithi076 conmon[56514]: (osd.7) 171 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:34:37.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:37 smithi076 conmon[56514]: cluster 2021-03-15T22:34:36.642302+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e921: 8 total, 8 up, 7 in 2021-03-15T22:34:37.699 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:37 smithi076 conmon[56514]: audit 2021-03-15T22:34:37.221309+0000 mon.c (mon.1) 73 : audit [DBG] from='client.? 172.21.15.76:0/993910096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:37.699 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:37 smithi076 conmon[62156]: cluster 2021-03-15T22:34:36.438536+0000 osd.7 (osd.7) 2021-03-15T22:34:37.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:37 smithi076 conmon[62156]: 171 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:34:37.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:37 smithi076 conmon[62156]: cluster 2021-03-15T22:34:36.642302+0000 mon.a (mon.0) 2326 : cluster [DBG] 2021-03-15T22:34:37.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:37 smithi076 conmon[62156]: osdmap e921: 8 total, 8 up, 7 in 2021-03-15T22:34:37.700 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:37 smithi076 conmon[62156]: audit 2021-03-15T22:34: 2021-03-15T22:34:37.701 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:37 smithi076 conmon[62156]: 37.221309+0000 mon.c (mon.1) 73 : audit [DBG] from='client.? 172.21.15.76:0/993910096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:37.727 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:37.749 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:37.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:37.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:37.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:37.878 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:37.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:37.880 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:37.981 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:38.002 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:38.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:38.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:37 smithi125 conmon[56739]: cluster 2021-03-15T22:34:36.438536+0000 osd.7 ( 2021-03-15T22:34:38.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:37 smithi125 conmon[56739]: osd.7) 171 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:34:38.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:37 smithi125 conmon[56739]: cluster 2021-03-15T22:34:36.642302+0000 mon.a (mon.0) 2326 : cluster [DBG] osdmap e921: 8 total, 8 up, 7 in 2021-03-15T22:34:38.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:37 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:34:38.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:37 smithi125 conmon[56739]: T22:34:37.221309+0000 2021-03-15T22:34:38.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:37 smithi125 conmon[56739]: mon.c (mon.1) 73 2021-03-15T22:34:38.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:37 smithi125 conmon[56739]: : audit [DBG] from='client.? 172.21.15.76:0/993910096' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:38.105 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:38.132 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:38.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:38.133 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:38.234 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:38.256 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:38.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:38.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:38.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:38.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:38.383 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:38.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:38.385 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:38.386 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:38.485 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:38.511 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:38.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:38.513 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:38.614 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:38.636 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:38.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:38.638 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:38.654 INFO:tasks.rados.rados.0.smithi076.stdout:3467: done (1 left) 2021-03-15T22:34:38.654 INFO:tasks.rados.rados.0.smithi076.stdout:3469: done (0 left) 2021-03-15T22:34:38.654 INFO:tasks.rados.rados.0.smithi076.stdout:3470: write (excl) oid 40 current snap is 413 2021-03-15T22:34:38.655 INFO:tasks.rados.rados.0.smithi076.stdout:3470: seq_num 1178 ranges {598646=561684,1700494=459744,2633109=781137} 2021-03-15T22:34:38.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:38 smithi076 conmon[62156]: cluster 2021-03-15T22:34:37.207449+0000 mgr.y (mgr.14144) 1038 : cluster [DBG] pgmap v1529: 158 pgs: 1 active+remapped+backfill_toofull, 9 peering, 6 unknown, 142 active+clean; 721 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 2.1 MiB/s wr, 11 op/s 2021-03-15T22:34:38.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:38 smithi076 conmon[62156]: cluster 2021-03-15T22:34:37.435001+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:34:38.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:38 smithi076 conmon[62156]: cluster 2021-03-15T22:34:37.646769+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e922: 8 total, 8 up, 7 in 2021-03-15T22:34:38.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:38 smithi076 conmon[62156]: cluster 2021-03-15T22:34:37.787230+0000 osd.3 (osd.3) 164 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:34:38.660 INFO:tasks.rados.rados.0.smithi076.stdout:3470: writing smithi07694878-40 from 598646 to 1160330 tid 1 2021-03-15T22:34:38.666 INFO:tasks.rados.rados.0.smithi076.stdout:3470: writing smithi07694878-40 from 1700494 to 2160238 tid 2 2021-03-15T22:34:38.674 INFO:tasks.rados.rados.0.smithi076.stdout:3470: writing smithi07694878-40 from 2633109 to 3414246 tid 3 2021-03-15T22:34:38.674 INFO:tasks.rados.rados.0.smithi076.stdout:3471: delete oid 47 current snap is 413 2021-03-15T22:34:38.674 INFO:tasks.rados.rados.0.smithi076.stdout:3470: finishing write tid 1 to smithi07694878-40 2021-03-15T22:34:38.674 INFO:tasks.rados.rados.0.smithi076.stdout:3470: finishing write tid 2 to smithi07694878-40 2021-03-15T22:34:38.680 INFO:tasks.rados.rados.0.smithi076.stdout:3470: finishing write tid 3 to smithi07694878-40 2021-03-15T22:34:38.681 INFO:tasks.rados.rados.0.smithi076.stdout:3470: finishing write tid 4 to smithi07694878-40 2021-03-15T22:34:38.681 INFO:tasks.rados.rados.0.smithi076.stdout:3471: done (1 left) 2021-03-15T22:34:38.682 INFO:tasks.rados.rados.0.smithi076.stdout:3472: read oid 33 snap -1 2021-03-15T22:34:38.682 INFO:tasks.rados.rados.0.smithi076.stdout:3472: expect (ObjNum 1174 snap 410 seq_num 1174) 2021-03-15T22:34:38.682 INFO:tasks.rados.rados.0.smithi076.stdout:3473: delete oid 11 current snap is 413 2021-03-15T22:34:38.682 INFO:tasks.rados.rados.0.smithi076.stdout:3470: finishing write tid 5 to smithi07694878-40 2021-03-15T22:34:38.682 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 40 v 3051 (ObjNum 1178 snap 413 seq_num 1178) dirty exists 2021-03-15T22:34:38.683 INFO:tasks.rados.rados.0.smithi076.stdout:3470: left oid 40 (ObjNum 1178 snap 413 seq_num 1178) 2021-03-15T22:34:38.685 INFO:tasks.rados.rados.0.smithi076.stdout:3470: done (2 left) 2021-03-15T22:34:38.686 INFO:tasks.rados.rados.0.smithi076.stdout:3473: done (1 left) 2021-03-15T22:34:38.686 INFO:tasks.rados.rados.0.smithi076.stdout:3474: snap_remove snap 406 2021-03-15T22:34:38.739 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:38.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:38.762 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:38.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:38.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:38.889 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:38.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:38.991 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:39.016 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:39.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:38 smithi076 conmon[56514]: cluster 2021-03-15T22:34:37.207449+0000 mgr.y (mgr.14144) 1038 : cluster [DBG] pgmap v1529: 158 pgs: 1 active+remapped+backfill_toofull, 9 peering, 6 unknown, 142 active+clean; 721 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 2.1 MiB/s wr, 11 op/s 2021-03-15T22:34:39.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:38 smithi076 conmon[56514]: cluster 2021-03-15T22:34:37.435001+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:34:39.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:38 smithi076 conmon[56514]: cluster 2021-03-15T22:34:37.646769+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e922: 8 total, 8 up, 7 in 2021-03-15T22:34:39.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:38 smithi076 conmon[56514]: cluster 2021-03-15T22:34:37.787230+0000 osd.3 (osd.3) 164 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:34:39.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:39.019 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:39.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:38 smithi125 conmon[56739]: cluster 2021-03-15T22:34:37.207449+0000 mgr.y (mgr.14144) 1038 : cluster [DBG] pgmap v1529: 158 pgs: 1 active+remapped+backfill_toofull, 9 peering, 6 unknown, 142 active+clean; 721 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 2.5 KiB/s rd, 2.1 MiB/s wr, 11 op/s 2021-03-15T22:34:39.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:38 smithi125 conmon[56739]: 2021-03-15T22:34:39.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:38 smithi125 conmon[56739]: cluster 2021-03-15T22:34:37.435001+0000 osd.0 (osd.0) 148 : cluster [DBG] 4.16 scrub ok 2021-03-15T22:34:39.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:38 smithi125 conmon[56739]: cluster 2021-03-15T22:34:37.646769+0000 mon.a (mon.0) 2327 : cluster [DBG] osdmap e922: 8 total, 8 up, 7 in 2021-03-15T22:34:39.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:38 smithi125 conmon[56739]: cluster 2021-03-15T22:34:37.787230+0000 osd.3 (osd.3) 164 : cluster [DBG] 4.11 scrub ok 2021-03-15T22:34:39.120 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:39.143 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:39.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:39.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:39.245 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:39.271 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:39.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:39.273 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:39.374 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:39.400 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:39.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:39.402 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:39.503 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:39.526 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:39.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:39.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:39.629 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:39.651 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:39.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:39.653 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:39.668 INFO:tasks.rados.rados.0.smithi076.stdout:3474: done (1 left) 2021-03-15T22:34:39.668 INFO:tasks.rados.rados.0.smithi076.stdout:3475: snap_create 2021-03-15T22:34:39.668 INFO:tasks.rados.rados.0.smithi076.stdout:3472: expect (ObjNum 1174 snap 410 seq_num 1174) 2021-03-15T22:34:39.754 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:39.779 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:39.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:39.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:39.881 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:39.906 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:39.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:39.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:40.009 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:40.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:39 smithi076 conmon[56514]: cluster 2021-03-15T22:34:38.119152+0000 osd.1 (osd. 2021-03-15T22:34:40.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:39 smithi076 conmon[56514]: 1) 174 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:34:40.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:39 smithi076 conmon[56514]: cluster 2021-03-15T22:34:38.649848+0000 mon.a (mon.0) 2328 : cluster [DBG] osdmap e923: 8 total, 8 up, 7 in 2021-03-15T22:34:40.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:39 smithi076 conmon[62156]: cluster 2021-03-15T22:34:38.119152+0000 osd.1 (osd.1) 174 : 2021-03-15T22:34:40.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:39 smithi076 conmon[62156]: cluster [DBG] 4.1c scrub ok 2021-03-15T22:34:40.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:39 smithi076 conmon[62156]: cluster 2021-03-15T22:34:38.649848+0000 mon.a (mon.0) 2328 : cluster [DBG] osdmap e923: 8 total, 8 up, 7 in 2021-03-15T22:34:40.032 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:40.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:40.034 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:40.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:39 smithi125 conmon[56739]: cluster 2021-03-15T22:34:38.119152+0000 osd.1 (osd.1 2021-03-15T22:34:40.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:39 smithi125 conmon[56739]: ) 174 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:34:40.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:39 smithi125 conmon[56739]: cluster 2021-03-15T22:34:38.649848+0000 mon.a (mon.0) 2328 : cluster [DBG] osdmap e923: 8 total, 8 up, 7 in 2021-03-15T22:34:40.135 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:40.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:40.159 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:40.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:40.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:34:40.260 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:40.282 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:40.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:40.284 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:40.322 INFO:tasks.rados.rados.0.smithi076.stdout:3472: done (1 left) 2021-03-15T22:34:40.323 INFO:tasks.rados.rados.0.smithi076.stdout:3475: done (0 left) 2021-03-15T22:34:40.323 INFO:tasks.rados.rados.0.smithi076.stdout:3476: write (excl) oid 22 current snap is 414 2021-03-15T22:34:40.323 INFO:tasks.rados.rados.0.smithi076.stdout:3476: seq_num 1181 ranges {649067=503784,1875940=607071,3115378=731306} 2021-03-15T22:34:40.329 INFO:tasks.rados.rados.0.smithi076.stdout:3476: writing smithi07694878-22 from 649067 to 1152851 tid 1 2021-03-15T22:34:40.337 INFO:tasks.rados.rados.0.smithi076.stdout:3476: writing smithi07694878-22 from 1875940 to 2483011 tid 2 2021-03-15T22:34:40.345 INFO:tasks.rados.rados.0.smithi076.stdout:3476: writing smithi07694878-22 from 3115378 to 3846684 tid 3 2021-03-15T22:34:40.345 INFO:tasks.rados.rados.0.smithi076.stdout:3477: write oid 41 current snap is 414 2021-03-15T22:34:40.345 INFO:tasks.rados.rados.0.smithi076.stdout:3477: seq_num 1182 ranges {665874=579033,1934422=467082,3142801=542578} 2021-03-15T22:34:40.352 INFO:tasks.rados.rados.0.smithi076.stdout:3477: writing smithi07694878-41 from 665874 to 1244907 tid 1 2021-03-15T22:34:40.358 INFO:tasks.rados.rados.0.smithi076.stdout:3477: writing smithi07694878-41 from 1934422 to 2401504 tid 2 2021-03-15T22:34:40.366 INFO:tasks.rados.rados.0.smithi076.stdout:3477: writing smithi07694878-41 from 3142801 to 3685379 tid 3 2021-03-15T22:34:40.366 INFO:tasks.rados.rados.0.smithi076.stdout:3478: write oid 10 current snap is 414 2021-03-15T22:34:40.366 INFO:tasks.rados.rados.0.smithi076.stdout:3476: finishing write tid 1 to smithi07694878-22 2021-03-15T22:34:40.366 INFO:tasks.rados.rados.0.smithi076.stdout:3478: seq_num 1183 ranges {682681=654282,1876551=610740,3053871=470203} 2021-03-15T22:34:40.373 INFO:tasks.rados.rados.0.smithi076.stdout:3478: writing smithi07694878-10 from 682681 to 1336963 tid 1 2021-03-15T22:34:40.379 INFO:tasks.rados.rados.0.smithi076.stdout:3478: writing smithi07694878-10 from 1876551 to 2487291 tid 2 2021-03-15T22:34:40.385 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:40.386 INFO:tasks.rados.rados.0.smithi076.stdout:3478: writing smithi07694878-10 from 3053871 to 3524074 tid 3 2021-03-15T22:34:40.386 INFO:tasks.rados.rados.0.smithi076.stdout:3479: write oid 2 current snap is 414 2021-03-15T22:34:40.387 INFO:tasks.rados.rados.0.smithi076.stdout:3476: finishing write tid 2 to smithi07694878-22 2021-03-15T22:34:40.387 INFO:tasks.rados.rados.0.smithi076.stdout:3479: seq_num 1184 ranges {699488=729531,1935033=470751,3081294=681475} 2021-03-15T22:34:40.393 INFO:tasks.rados.rados.0.smithi076.stdout:3479: writing smithi07694878-2 from 699488 to 1429019 tid 1 2021-03-15T22:34:40.399 INFO:tasks.rados.rados.0.smithi076.stdout:3479: writing smithi07694878-2 from 1935033 to 2405784 tid 2 2021-03-15T22:34:40.406 INFO:tasks.rados.rados.0.smithi076.stdout:3479: writing smithi07694878-2 from 3081294 to 3762769 tid 3 2021-03-15T22:34:40.407 INFO:tasks.rados.rados.0.smithi076.stdout:3480: snap_remove snap 279 2021-03-15T22:34:40.415 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:40.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:40.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:40.518 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:40.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:34:40.541 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:40.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:40.542 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:40.643 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:40.665 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:40.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:40.666 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:40.767 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:40.789 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:40.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:40.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:40.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: cluster 2021-03-15T22:34:38.931273+0000 osd.6 (osd.6) 2021-03-15T22:34:40.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: 134 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:40.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: cluster 2021-03-15T22:34:38.932971+0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:40.807 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: cluster 2021-03-15T22:34:39.122659+0000 osd.1 (osd.1) 175 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:34:40.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: cluster 2021-03-15T22:34:39.207863+0000 mgr.y (mgr.14144) 1039 : cluster [DBG] pgmap v1532: 158 pgs: 1 active+remapped+backfill_toofull, 9 peering, 4 unknown, 144 active+clean; 721 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.1 MiB/s wr, 10 op/s 2021-03-15T22:34:40.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: cluster 2021-03-15T22:34:39.663034+0000 mon.a (mon.0) 2329 : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2021-03-15T22:34:40.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: cluster 2021-03-15T22:34:39.673331+0000 osd.4 (osd.4) 187 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:34:40.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:34:40.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: -15T22:34:39.831360+0000 osd.5 (osd 2021-03-15T22:34:40.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: .5) 51 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:34:40.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: 2021-03-15T22:34:40.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: cluster 2021-03-15T22:34:40.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: 2021-03-15T22:34: 2021-03-15T22:34:40.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: 40.319302+0000 mon.a (mon.0 2021-03-15T22:34:40.811 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[56514]: ) 2330 : cluster [DBG] osdmap e925: 8 total, 8 up, 7 in 2021-03-15T22:34:40.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: cluster 2021-03-15T22:34:38.931273+0000 2021-03-15T22:34:40.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: osd.6 (osd.6) 134 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:40.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:34:40.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: 15T22:34:38.932971+0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:40.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: 2021-03-15T22:34:40.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: cluster 2021-03-15T22:34 2021-03-15T22:34:40.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: :39.122659+0000 osd.1 (osd.1) 2021-03-15T22:34:40.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: 175 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:34:40.813 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:34:40.814 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: 03-15T22:34:39.207863+0000 2021-03-15T22:34:40.814 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: mgr.y (mgr.14144) 1039 : cluster [DBG] 2021-03-15T22:34:40.814 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: pgmap v1532: 158 pgs: 1 active+remapped+backfill_toofull, 9 peering, 4 unknown, 144 active+clean; 721 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.1 MiB/s wr, 10 op/s 2021-03-15T22:34:40.814 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:34:40.814 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: 34:39.663034+0000 mon.a (mon.0 2021-03-15T22:34:40.815 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: ) 2329 : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2021-03-15T22:34:40.815 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:34:40.816 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: -03-15T22:34:39.673331+0000 2021-03-15T22:34:40.816 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: osd.4 (osd.4) 187 : cluster [DBG] 2021-03-15T22:34:40.816 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: 5.1 scrub ok 2021-03-15T22:34:40.816 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:34:40.817 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: 22:34:39.831360+0000 osd.5 (osd 2021-03-15T22:34:40.817 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: .5) 51 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:34:40.817 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: cluster 2021-03-15T22:34:40.817 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: 2021-03-15T22:34:40.319302 2021-03-15T22:34:40.817 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2330 : cluster 2021-03-15T22:34:40.818 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:40 smithi076 conmon[62156]: [DBG] osdmap e925: 8 total, 8 up, 7 in 2021-03-15T22:34:40.892 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:40.915 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:40.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:40.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:40.991 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:41.006 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:34:41.006 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:34:41.019 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:41.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: cluster 2021-03-15T22:34:38.931273+0000 osd.6 (osd.6) 134 : 2021-03-15T22:34:41.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:41.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: cluster 2021-03-15T22:34:38.932971+0000 osd.6 (osd.6) 135 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:41.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: 2021-03-15T22:34:41.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:34:41.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: 15T22:34:39.122659+0000 osd.1 ( 2021-03-15T22:34:41.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: osd.1) 175 : cluster [DBG] 2021-03-15T22:34:41.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: 5.19 scrub ok 2021-03-15T22:34:41.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:34:41.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: 03-15T22:34: 2021-03-15T22:34:41.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: 39.207863+0000 mgr.y (mgr.14144) 1039 2021-03-15T22:34:41.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: : cluster [DBG] pgmap v1532: 158 pgs: 1 active+remapped+backfill_toofull, 9 peering, 4 unknown, 144 active+clean; 721 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 1.2 KiB/s rd, 2.1 MiB/s wr, 10 op/s 2021-03-15T22:34:41.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:34:41.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: -03-15T22:34: 2021-03-15T22:34:41.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: 39.663034+0000 mon.a (mon.0) 2329 2021-03-15T22:34:41.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: : cluster [DBG] osdmap e924: 8 total, 8 up, 7 in 2021-03-15T22:34:41.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: cluster 2021-03-15T22:34:41.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: 2021-03-15T22:34 2021-03-15T22:34:41.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: :39.673331+0000 osd.4 (osd 2021-03-15T22:34:41.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: .4) 187 : cluster [DBG] 5.1 scrub ok 2021-03-15T22:34:41.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:34:41.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: 15T22:34:39.831360+0000 osd.5 ( 2021-03-15T22:34:41.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: osd.5) 51 : cluster [DBG] 2.2 scrub ok 2021-03-15T22:34:41.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:34:41.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: -03-15T22:34:40 2021-03-15T22:34:41.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: .319302+0000 mon.a (mon.0) 2330 : 2021-03-15T22:34:41.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:40 smithi125 conmon[56739]: cluster [DBG] osdmap e925: 8 total, 8 up, 7 in 2021-03-15T22:34:41.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:41.045 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:41.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:41.146 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:41.171 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:41.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:41.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:41.274 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:41.299 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:41.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:41.300 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:41.335 INFO:tasks.rados.rados.0.smithi076.stdout:3480: done (4 left) 2021-03-15T22:34:41.335 INFO:tasks.rados.rados.0.smithi076.stdout:3481: delete oid 11 current snap is 414 2021-03-15T22:34:41.336 INFO:tasks.rados.rados.0.smithi076.stdout:3476: finishing write tid 3 to smithi07694878-22 2021-03-15T22:34:41.336 INFO:tasks.rados.rados.0.smithi076.stdout:3476: finishing write tid 4 to smithi07694878-22 2021-03-15T22:34:41.336 INFO:tasks.rados.rados.0.smithi076.stdout:3476: finishing write tid 5 to smithi07694878-22 2021-03-15T22:34:41.336 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 2817 (ObjNum 1181 snap 414 seq_num 1181) dirty exists 2021-03-15T22:34:41.337 INFO:tasks.rados.rados.0.smithi076.stdout:3476: left oid 22 (ObjNum 1181 snap 414 seq_num 1181) 2021-03-15T22:34:41.337 INFO:tasks.rados.rados.0.smithi076.stdout:3477: finishing write tid 1 to smithi07694878-41 2021-03-15T22:34:41.337 INFO:tasks.rados.rados.0.smithi076.stdout:3477: finishing write tid 2 to smithi07694878-41 2021-03-15T22:34:41.337 INFO:tasks.rados.rados.0.smithi076.stdout:3477: finishing write tid 3 to smithi07694878-41 2021-03-15T22:34:41.337 INFO:tasks.rados.rados.0.smithi076.stdout:3477: finishing write tid 4 to smithi07694878-41 2021-03-15T22:34:41.338 INFO:tasks.rados.rados.0.smithi076.stdout:3477: finishing write tid 5 to smithi07694878-41 2021-03-15T22:34:41.338 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 2412 (ObjNum 1182 snap 414 seq_num 1182) dirty exists 2021-03-15T22:34:41.338 INFO:tasks.rados.rados.0.smithi076.stdout:3477: left oid 41 (ObjNum 1182 snap 414 seq_num 1182) 2021-03-15T22:34:41.338 INFO:tasks.rados.rados.0.smithi076.stdout:3478: finishing write tid 1 to smithi07694878-10 2021-03-15T22:34:41.339 INFO:tasks.rados.rados.0.smithi076.stdout:3478: finishing write tid 2 to smithi07694878-10 2021-03-15T22:34:41.339 INFO:tasks.rados.rados.0.smithi076.stdout:3478: finishing write tid 3 to smithi07694878-10 2021-03-15T22:34:41.339 INFO:tasks.rados.rados.0.smithi076.stdout:3478: finishing write tid 4 to smithi07694878-10 2021-03-15T22:34:41.339 INFO:tasks.rados.rados.0.smithi076.stdout:3478: finishing write tid 5 to smithi07694878-10 2021-03-15T22:34:41.339 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 2416 (ObjNum 1183 snap 414 seq_num 1183) dirty exists 2021-03-15T22:34:41.340 INFO:tasks.rados.rados.0.smithi076.stdout:3478: left oid 10 (ObjNum 1183 snap 414 seq_num 1183) 2021-03-15T22:34:41.340 INFO:tasks.rados.rados.0.smithi076.stdout:3479: finishing write tid 1 to smithi07694878-2 2021-03-15T22:34:41.340 INFO:tasks.rados.rados.0.smithi076.stdout:3479: finishing write tid 2 to smithi07694878-2 2021-03-15T22:34:41.340 INFO:tasks.rados.rados.0.smithi076.stdout:3479: finishing write tid 3 to smithi07694878-2 2021-03-15T22:34:41.340 INFO:tasks.rados.rados.0.smithi076.stdout:3479: finishing write tid 4 to smithi07694878-2 2021-03-15T22:34:41.341 INFO:tasks.rados.rados.0.smithi076.stdout:3479: finishing write tid 5 to smithi07694878-2 2021-03-15T22:34:41.341 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 2420 (ObjNum 1184 snap 414 seq_num 1184) dirty exists 2021-03-15T22:34:41.341 INFO:tasks.rados.rados.0.smithi076.stdout:3479: left oid 2 (ObjNum 1184 snap 414 seq_num 1184) 2021-03-15T22:34:41.342 INFO:tasks.rados.rados.0.smithi076.stdout:3476: done (4 left) 2021-03-15T22:34:41.342 INFO:tasks.rados.rados.0.smithi076.stdout:3477: done (3 left) 2021-03-15T22:34:41.342 INFO:tasks.rados.rados.0.smithi076.stdout:3478: done (2 left) 2021-03-15T22:34:41.343 INFO:tasks.rados.rados.0.smithi076.stdout:3479: done (1 left) 2021-03-15T22:34:41.343 INFO:tasks.rados.rados.0.smithi076.stdout:3481: done (0 left) 2021-03-15T22:34:41.343 INFO:tasks.rados.rados.0.smithi076.stdout:3482: read oid 1 snap -1 2021-03-15T22:34:41.343 INFO:tasks.rados.rados.0.smithi076.stdout:3482: expect deleted 2021-03-15T22:34:41.343 INFO:tasks.rados.rados.0.smithi076.stdout:3483: snap_remove snap 378 2021-03-15T22:34:41.401 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:41.424 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:41.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:41.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:41.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:41.548 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:41.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:41.550 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:41.651 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:41.674 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:41.675 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:41.777 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:41.798 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:41.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:41.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[56514]: audit 2021-03-15T22:34:40.256343+0000 mgr.y (mgr.14144) 1040 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:41.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[56514]: cluster 2021-03-15T22:34:40.472791+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:34:41.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[56514]: cluster 2021-03-15T22:34:40.475847+0000 osd.6 (osd.6) 136 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:34:41.809 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[56514]: cluster 2021-03-15T22:34:40.799149+0000 osd.5 (osd.5) 52 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:34:41.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[56514]: cluster 2021-03-15T22:34:41.325335+0000 mon.a (mon 2021-03-15T22:34:41.810 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[56514]: .0) 2331 : cluster [DBG] osdmap e926: 8 total, 8 up, 7 in 2021-03-15T22:34:41.810 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[62156]: audit 2021-03-15T22:34:40.256343+0000 mgr.y (mgr.14144) 2021-03-15T22:34:41.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[62156]: 1040 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:41.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[62156]: cluster 2021-03-15T22:34:40.472791+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:34:41.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:34:41.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[62156]: -03-15T22:34:40.475847+0000 osd.6 (osd.6 2021-03-15T22:34:41.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[62156]: ) 136 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:34:41.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[62156]: cluster 2021-03-15T22:34:40.799149+0000 osd.5 2021-03-15T22:34:41.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[62156]: (osd.5) 52 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:34:41.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[62156]: cluster 2021-03-15T22:34:41.325335 2021-03-15T22:34:41.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:41 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2331 : cluster [DBG] osdmap e926: 8 total, 8 up, 7 in 2021-03-15T22:34:41.901 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:41.924 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:41.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:41.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:41.958 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:34:42.027 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:42.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: audit 2021-03-15T22:34:40.256343+0000 mgr.y (mgr.14144) 1040 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:42.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: cluster 2021-03-15T22:34:40.472791+0000 osd.0 (osd.0) 149 : cluster [DBG] 4.6 scrub ok 2021-03-15T22:34:42.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:34:42.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: -15T22:34:40.475847+0000 osd.6 (osd.6 2021-03-15T22:34:42.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: ) 136 : cluster [DBG] 3.1e scrub ok 2021-03-15T22:34:42.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: cluster 2021-03-15T22:34:42.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: 2021-03-15T22:34 2021-03-15T22:34:42.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: :40.799149+0000 osd.5 (osd.5) 2021-03-15T22:34:42.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: 52 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:34:42.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:34:42.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: -03-15T22:34: 2021-03-15T22:34:42.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: 41.325335+0000 mon.a (mon.0) 2331 2021-03-15T22:34:42.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:41 smithi125 conmon[56739]: : cluster [DBG] osdmap e926: 8 total, 8 up, 7 in 2021-03-15T22:34:42.063 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:42.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:42.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:42.166 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:42.190 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:42.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:42.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:42.293 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:42.319 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:42.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:42.320 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:42.339 INFO:tasks.rados.rados.0.smithi076.stdout:3483: done (1 left) 2021-03-15T22:34:42.340 INFO:tasks.rados.rados.0.smithi076.stdout:3484: write oid 37 current snap is 414 2021-03-15T22:34:42.340 INFO:tasks.rados.rados.0.smithi076.stdout:3484: seq_num 1186 ranges {733102=596382,1768350=758067,3136140=704019} 2021-03-15T22:34:42.350 INFO:teuthology.orchestra.run.smithi076.stderr:reweighted osd.3 to 0.764007 (c395) 2021-03-15T22:34:42.353 INFO:tasks.rados.rados.0.smithi076.stdout:3484: writing smithi07694878-37 from 733102 to 1329484 tid 1 2021-03-15T22:34:42.361 INFO:tasks.rados.rados.0.smithi076.stdout:3484: writing smithi07694878-37 from 1768350 to 2526417 tid 2 2021-03-15T22:34:42.370 INFO:tasks.rados.rados.0.smithi076.stdout:3484: writing smithi07694878-37 from 3136140 to 3840159 tid 3 2021-03-15T22:34:42.370 INFO:tasks.rados.rados.0.smithi076.stdout:3485: rollback oid 27 current snap is 414 2021-03-15T22:34:42.370 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 369 2021-03-15T22:34:42.371 INFO:tasks.rados.rados.0.smithi076.stdout:3482: done (2 left) 2021-03-15T22:34:42.371 INFO:tasks.rados.rados.0.smithi076.stdout:3486: snap_create 2021-03-15T22:34:42.376 INFO:tasks.rados.rados.0.smithi076.stdout:3484: finishing write tid 1 to smithi07694878-37 2021-03-15T22:34:42.376 INFO:tasks.rados.rados.0.smithi076.stdout:3484: finishing write tid 2 to smithi07694878-37 2021-03-15T22:34:42.379 INFO:tasks.rados.rados.0.smithi076.stdout:3484: finishing write tid 3 to smithi07694878-37 2021-03-15T22:34:42.379 INFO:tasks.rados.rados.0.smithi076.stdout:3484: finishing write tid 4 to smithi07694878-37 2021-03-15T22:34:42.379 INFO:tasks.rados.rados.0.smithi076.stdout:3484: finishing write tid 5 to smithi07694878-37 2021-03-15T22:34:42.379 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 3076 (ObjNum 1186 snap 414 seq_num 1186) dirty exists 2021-03-15T22:34:42.380 INFO:tasks.rados.rados.0.smithi076.stdout:3484: left oid 37 (ObjNum 1186 snap 414 seq_num 1186) 2021-03-15T22:34:42.422 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:42.444 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:42.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:42.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:42.547 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:42.569 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:42.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:42.571 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:42.671 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:42.694 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:42.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:42.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:42.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[56514]: cluster 2021-03-15T22:34:40.844177+0000 osd.3 (osd.3) 165 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:34:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[56514]: cluster 2021-03-15T22:34:41.208423+0000 mgr.y (mgr.14144) 1041 : cluster [DBG] pgmap v1535: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 741 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-03-15T22:34:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[56514]: cluster 2021-03-15T22:34:41.436834+0000 osd.7 (osd.7) 172 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:34:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[56514]: cluster 2021-03-15T22:34:41.490875+0000 osd.6 (osd.6) 137 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:34:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[56514]: cluster 2021-03-15T22:34:41.614679+0000 osd.4 (osd.4) 188 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:34:42.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[56514]: audit 2021-03-15T22:34:41.940178+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? 172.21.15.76:0/1481828302' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.7640074395265278}]: dispatch 2021-03-15T22:34:42.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[56514]: audit 2021-03-15T22:34:42.331821+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? 172.21.15.76:0/1481828302' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.7640074395265278}]': finished 2021-03-15T22:34:42.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[56514]: cluster 2021-03-15T22:34:42.331920+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e927: 8 total, 8 up, 7 in 2021-03-15T22:34:42.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[56514]: audit 2021-03-15T22:34:42.414794+0000 mon.c (mon.1) 74 : audit [INF] from='client.? 172.21.15.76:0/419148246' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:42.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[56514]: audit 2021-03-15T22:34:42.415104+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:42.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[62156]: cluster 2021-03-15T22:34:40.844177+0000 osd.3 (osd. 2021-03-15T22:34:42.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[62156]: 3) 165 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:34:42.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[62156]: cluster 2021-03-15T22:34:41.208423+0000 mgr.y (mgr.14144) 1041 : cluster [DBG] pgmap v1535: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 741 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-03-15T22:34:42.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[62156]: cluster 2021-03-15T22:34:41.436834+0000 osd.7 (osd.7) 172 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:34:42.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[62156]: cluster 2021-03-15T22:34:41.490875+0000 osd.6 (osd.6) 137 : cluster [DBG] 7.7 scrub ok 2021-03-15T22:34:42.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[62156]: cluster 2021-03-15T22:34:41.614679+0000 osd.4 (osd.4) 188 : cluster [DBG] 6.1e scrub ok 2021-03-15T22:34:42.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[62156]: audit 2021-03-15T22:34:41.940178+0000 mon.a (mon.0) 2332 : audit [INF] from='client.? 172.21.15.76:0/1481828302' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.7640074395265278}]: dispatch 2021-03-15T22:34:42.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[62156]: audit 2021-03-15T22:34:42.331821+0000 mon.a (mon.0) 2333 : audit [INF] from='client.? 172.21.15.76:0/1481828302' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.7640074395265278}]': finished 2021-03-15T22:34:42.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[62156]: cluster 2021-03-15T22:34:42.331920+0000 mon.a (mon.0) 2334 : cluster [DBG] osdmap e927: 8 total, 8 up, 7 in 2021-03-15T22:34:42.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[62156]: audit 2021-03-15T22:34:42.414794+0000 mon.c (mon.1) 74 : audit [INF] from='client.? 172.21.15.76:0/419148246' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:42.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:42 smithi076 conmon[62156]: audit 2021-03-15T22:34:42.415104+0000 mon.a (mon.0) 2335 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:42.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:42.818 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:42.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:42.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:42.920 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:42.942 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:42.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:42.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:43.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: cluster 2021-03-15T22:34:40.844177+0000 osd.3 (osd.3) 165 2021-03-15T22:34:43.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: : cluster [DBG] 3.18 scrub ok 2021-03-15T22:34:43.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: cluster 2021-03-15T22:34:41.208423+0000 mgr.y (mgr.14144) 1041 : cluster [DBG] pgmap v1535: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 741 MiB data, 2.0 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-03-15T22:34:43.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: cluster 2021-03-15T22:34:41.436834+0000 osd.7 (osd.7) 2021-03-15T22:34:43.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: 172 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:34:43.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: cluster 2021-03-15T22:34:43.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: 2021-03-15T22:34:41.490875+0000 2021-03-15T22:34:43.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: osd.6 (osd.6) 137 : cluster 2021-03-15T22:34:43.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: [DBG] 7.7 scrub ok 2021-03-15T22:34:43.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:34:43.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: -15T22:34:41.614679+0000 osd.4 ( 2021-03-15T22:34:43.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: osd.4) 188 : cluster [DBG] 2021-03-15T22:34:43.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: 6.1e scrub ok 2021-03-15T22:34:43.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:34:43.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: -15T22:34:41.940178+0000 mon.a ( 2021-03-15T22:34:43.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: mon.0) 2332 : audit [INF] 2021-03-15T22:34:43.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1481828302' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 3, "weight": 0.7640074395265278}]: dispatch 2021-03-15T22:34:43.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:34:43.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: 03-15T22:34:42.331821+0000 mon.a (mon. 2021-03-15T22:34:43.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: 0) 2333 : audit [INF] from='client.? 172.21.15.76:0/1481828302' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 3, "weight": 0.7640074395265278}]': finished 2021-03-15T22:34:43.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:34:43.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: -15T22:34:42.331920+0000 mon.a (mon 2021-03-15T22:34:43.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: .0) 2334 : cluster [DBG] 2021-03-15T22:34:43.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: osdmap e927: 8 total, 8 up, 7 in 2021-03-15T22:34:43.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: audit 2021 2021-03-15T22:34:43.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: -03-15T22:34:42.414794+0000 2021-03-15T22:34:43.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: mon.c (mon.1) 74 : audit 2021-03-15T22:34:43.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/419148246' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:43.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:34:43.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: -15T22:34:42.415104+0000 mon.a 2021-03-15T22:34:43.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: (mon.0) 2335 : audit 2021-03-15T22:34:43.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:42 smithi125 conmon[56739]: [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:43.045 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:43.066 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:43.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:43.067 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:43.163 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:43.168 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:43.192 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:43.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:43.193 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:43.294 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:43.318 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:43.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:43.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:43.348 INFO:tasks.rados.rados.0.smithi076.stdout:3484: done (2 left) 2021-03-15T22:34:43.349 INFO:tasks.rados.rados.0.smithi076.stdout:3486: done (1 left) 2021-03-15T22:34:43.349 INFO:tasks.rados.rados.0.smithi076.stdout:3487: read oid 26 snap -1 2021-03-15T22:34:43.349 INFO:tasks.rados.rados.0.smithi076.stdout:3487: expect deleted 2021-03-15T22:34:43.349 INFO:tasks.rados.rados.0.smithi076.stdout:3488: write oid 45 current snap is 415 2021-03-15T22:34:43.350 INFO:tasks.rados.rados.0.smithi076.stdout:3488: seq_num 1187 ranges {749909=671631,2110479=618078,3163563=515291} 2021-03-15T22:34:43.351 INFO:tasks.rados.rados.0.smithi076.stdout:3488: writing smithi07694878-45 from 749909 to 1421540 tid 1 2021-03-15T22:34:43.359 INFO:tasks.rados.rados.0.smithi076.stdout:3488: writing smithi07694878-45 from 2110479 to 2728557 tid 2 2021-03-15T22:34:43.365 INFO:tasks.rados.rados.0.smithi076.stdout:3488: writing smithi07694878-45 from 3163563 to 3678854 tid 3 2021-03-15T22:34:43.365 INFO:tasks.rados.rados.0.smithi076.stdout:3489: snap_create 2021-03-15T22:34:43.366 INFO:tasks.rados.rados.0.smithi076.stdout:3488: finishing write tid 1 to smithi07694878-45 2021-03-15T22:34:43.366 INFO:tasks.rados.rados.0.smithi076.stdout:3488: finishing write tid 2 to smithi07694878-45 2021-03-15T22:34:43.366 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:34:43.368 INFO:tasks.rados.rados.0.smithi076.stdout:3488: finishing write tid 3 to smithi07694878-45 2021-03-15T22:34:43.368 INFO:tasks.rados.rados.0.smithi076.stdout:3488: finishing write tid 4 to smithi07694878-45 2021-03-15T22:34:43.369 INFO:tasks.rados.rados.0.smithi076.stdout:3488: finishing write tid 5 to smithi07694878-45 2021-03-15T22:34:43.369 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 2118 (ObjNum 1187 snap 415 seq_num 1187) dirty exists 2021-03-15T22:34:43.369 INFO:tasks.rados.rados.0.smithi076.stdout:3488: left oid 45 (ObjNum 1187 snap 415 seq_num 1187) 2021-03-15T22:34:43.420 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:43.442 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:43.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:43.443 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:43.544 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:43.576 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:43.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:43.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:43.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:43.700 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:43.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:43.702 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:43.803 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:43.825 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:43.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:43.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:43.928 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:43.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:43.956 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:43.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:44.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: cluster 2021-03-15T22:34:41.870360+0000 osd.3 2021-03-15T22:34:44.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: (osd.3) 166 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:34:44.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: cluster 2021-03-15T22:34:42.447623+0000 osd.6 (osd.6) 138 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:34:44.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: cluster 2021-03-15T22:34:42.631957+0000 osd.4 (osd.4) 189 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:34:44.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: cluster 2021-03-15T22:34:43.325046+0000 2021-03-15T22:34:44.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: mon.a (mon.0) 2336 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:34:44.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: audit 2021 2021-03-15T22:34:44.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: -03-15T22:34:43.334421+0000 mon.a (mon.0 2021-03-15T22:34:44.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: ) 2337 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:44.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:34:44.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: -03-15T22:34: 2021-03-15T22:34:44.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: 43.334638+0000 mon.a (mon. 2021-03-15T22:34:44.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[56514]: 0) 2338 : cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2021-03-15T22:34:44.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: cluster 2021-03-15T22:34:41.870360 2021-03-15T22:34:44.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: +0000 osd.3 (osd.3) 166 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:34:44.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: cluster 2021-03-15T22:34:42.447623 2021-03-15T22:34:44.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: +0000 osd.6 (osd.6) 138 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:34:44.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: 2021-03-15T22:34:44.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: cluster 2021-03-15T22:34:44.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: 2021-03-15T22:34:42.631957+0000 osd.4 (osd 2021-03-15T22:34:44.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: .4) 189 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:34:44.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:34:44.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: T22:34:43.325046+0000 mon.a (mon.0 2021-03-15T22:34:44.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: ) 2336 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:34:44.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:34:44.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: 22:34:43.334421+0000 mon.a (mon. 2021-03-15T22:34:44.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: 0) 2337 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:44.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:34:44.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: 03-15T22:34:43.334638+0000 mon.a (mon 2021-03-15T22:34:44.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:43 smithi076 conmon[62156]: .0) 2338 : cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2021-03-15T22:34:44.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:43 smithi125 conmon[56739]: cluster 2021-03-15T22:34:41.870360+0000 osd.3 (osd.3) 166 : cluster [DBG] 3.7 scrub ok 2021-03-15T22:34:44.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:43 smithi125 conmon[56739]: 2021-03-15T22:34:44.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:43 smithi125 conmon[56739]: cluster 2021-03-15T22:34:42.447623+0000 osd.6 (osd.6) 138 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:34:44.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:43 smithi125 conmon[56739]: cluster 2021-03-15T22:34:42.631957+0000 osd.4 (osd.4) 189 : cluster [DBG] 4.13 scrub ok 2021-03-15T22:34:44.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:43 smithi125 conmon[56739]: 2021-03-15T22:34:44.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:43 smithi125 conmon[56739]: cluster 2021-03-15T22:34:43.325046+0000 mon.a (mon.0) 2336 : cluster 2021-03-15T22:34:44.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:43 smithi125 conmon[56739]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:34:44.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:43 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:34:44.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:43 smithi125 conmon[56739]: -15T22:34:43.334421+0000 mon.a (mon.0) 2337 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:44.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:43 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:34:44.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:43 smithi125 conmon[56739]: :34:43.334638+0000 mon.a (mon.0) 2338 : cluster [DBG] osdmap e928: 8 total, 8 up, 7 in 2021-03-15T22:34:44.059 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:44.084 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:44.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:44.085 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:44.186 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:44.212 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:44.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:44.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:44.315 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:44.333 INFO:tasks.rados.rados.0.smithi076.stdout:3487: done (3 left) 2021-03-15T22:34:44.334 INFO:tasks.rados.rados.0.smithi076.stdout:3488: done (2 left) 2021-03-15T22:34:44.334 INFO:tasks.rados.rados.0.smithi076.stdout:3489: done (1 left) 2021-03-15T22:34:44.334 INFO:tasks.rados.rados.0.smithi076.stdout:3490: read oid 9 snap 391 2021-03-15T22:34:44.335 INFO:tasks.rados.rados.0.smithi076.stdout:3490: expect (ObjNum 1092 snap 375 seq_num 1092) 2021-03-15T22:34:44.335 INFO:tasks.rados.rados.0.smithi076.stdout:3491: read oid 33 snap -1 2021-03-15T22:34:44.335 INFO:tasks.rados.rados.0.smithi076.stdout:3491: expect (ObjNum 1174 snap 410 seq_num 1174) 2021-03-15T22:34:44.335 INFO:tasks.rados.rados.0.smithi076.stdout:3492: delete oid 4 current snap is 416 2021-03-15T22:34:44.340 INFO:tasks.rados.rados.0.smithi076.stdout:3491: expect (ObjNum 1174 snap 410 seq_num 1174) 2021-03-15T22:34:44.343 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:44.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:44.345 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:44.446 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:44.469 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:44.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:44.471 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:44.546 INFO:tasks.rados.rados.0.smithi076.stdout:3491: done (3 left) 2021-03-15T22:34:44.546 INFO:tasks.rados.rados.0.smithi076.stdout:3492: done (2 left) 2021-03-15T22:34:44.547 INFO:tasks.rados.rados.0.smithi076.stdout:3493: rollback oid 21 current snap is 416 2021-03-15T22:34:44.547 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 21 to 402 2021-03-15T22:34:44.547 INFO:tasks.rados.rados.0.smithi076.stdout:3490: expect (ObjNum 1092 snap 375 seq_num 1092) 2021-03-15T22:34:44.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:44.598 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:44.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:44.599 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:44.629 INFO:tasks.rados.rados.0.smithi076.stdout:3485: finishing rollback tid 0 to smithi07694878-27 2021-03-15T22:34:44.629 INFO:tasks.rados.rados.0.smithi076.stdout:3485: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:34:44.629 INFO:tasks.rados.rados.0.smithi076.stdout:3485: finishing rollback tid 2 to smithi07694878-27 2021-03-15T22:34:44.630 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 2421 (ObjNum 896 snap 302 seq_num 896) dirty exists 2021-03-15T22:34:44.630 INFO:tasks.rados.rados.0.smithi076.stdout:3493: finishing rollback tid 0 to smithi07694878-21 2021-03-15T22:34:44.630 INFO:tasks.rados.rados.0.smithi076.stdout:3493: finishing rollback tid 1 to smithi07694878-21 2021-03-15T22:34:44.630 INFO:tasks.rados.rados.0.smithi076.stdout:3485: done (2 left) 2021-03-15T22:34:44.631 INFO:tasks.rados.rados.0.smithi076.stdout:3490: done (1 left) 2021-03-15T22:34:44.631 INFO:tasks.rados.rados.0.smithi076.stdout:3494: read oid 35 snap -1 2021-03-15T22:34:44.631 INFO:tasks.rados.rados.0.smithi076.stdout:3494: expect deleted 2021-03-15T22:34:44.633 INFO:tasks.rados.rados.0.smithi076.stdout:3495: write oid 2 current snap is 416 2021-03-15T22:34:44.633 INFO:tasks.rados.rados.0.smithi076.stdout:3493: finishing rollback tid 2 to smithi07694878-21 2021-03-15T22:34:44.634 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 3082 (ObjNum 1044 snap 350 seq_num 1044) dirty exists 2021-03-15T22:34:44.634 INFO:tasks.rados.rados.0.smithi076.stdout:3495: seq_num 1189 ranges {783523=422129,1827443=621747,3218409=654188} 2021-03-15T22:34:44.635 INFO:tasks.rados.rados.0.smithi076.stdout:3495: writing smithi07694878-2 from 783523 to 1205652 tid 1 2021-03-15T22:34:44.642 INFO:tasks.rados.rados.0.smithi076.stdout:3495: writing smithi07694878-2 from 1827443 to 2449190 tid 2 2021-03-15T22:34:44.651 INFO:tasks.rados.rados.0.smithi076.stdout:3495: writing smithi07694878-2 from 3218409 to 3872597 tid 3 2021-03-15T22:34:44.651 INFO:tasks.rados.rados.0.smithi076.stdout:3493: done (2 left) 2021-03-15T22:34:44.652 INFO:tasks.rados.rados.0.smithi076.stdout:3496: copy_from oid 19 from oid 1 current snap is 416 2021-03-15T22:34:44.652 INFO:tasks.rados.rados.0.smithi076.stdout:3495: finishing write tid 1 to smithi07694878-2 2021-03-15T22:34:44.652 INFO:tasks.rados.rados.0.smithi076.stdout:3495: finishing write tid 2 to smithi07694878-2 2021-03-15T22:34:44.653 INFO:tasks.rados.rados.0.smithi076.stdout:3494: done (2 left) 2021-03-15T22:34:44.653 INFO:tasks.rados.rados.0.smithi076.stdout:3497: delete oid 39 current snap is 416 2021-03-15T22:34:44.656 INFO:tasks.rados.rados.0.smithi076.stdout:3496: finishing copy_from to smithi07694878-19 2021-03-15T22:34:44.657 INFO:tasks.rados.rados.0.smithi076.stdout:3496: got expected ENOENT (src dne) 2021-03-15T22:34:44.657 INFO:tasks.rados.rados.0.smithi076.stdout:3496: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:34:44.657 INFO:tasks.rados.rados.0.smithi076.stdout:3496: done (2 left) 2021-03-15T22:34:44.658 INFO:tasks.rados.rados.0.smithi076.stdout:3497: done (1 left) 2021-03-15T22:34:44.658 INFO:tasks.rados.rados.0.smithi076.stdout:3498: read oid 3 snap -1 2021-03-15T22:34:44.659 INFO:tasks.rados.rados.0.smithi076.stdout:3498: expect (ObjNum 1161 snap 407 seq_num 1161) 2021-03-15T22:34:44.659 INFO:tasks.rados.rados.0.smithi076.stdout:3499: write (excl) oid 45 current snap is 416 2021-03-15T22:34:44.660 INFO:tasks.rados.rados.0.smithi076.stdout:3499: seq_num 1191 ranges {417137=572627,1428054=589083} 2021-03-15T22:34:44.661 INFO:tasks.rados.rados.0.smithi076.stdout:3499: writing smithi07694878-45 from 417137 to 989764 tid 1 2021-03-15T22:34:44.666 INFO:tasks.rados.rados.0.smithi076.stdout:3499: writing smithi07694878-45 from 1428054 to 2017137 tid 2 2021-03-15T22:34:44.666 INFO:tasks.rados.rados.0.smithi076.stdout:3495: finishing write tid 3 to smithi07694878-2 2021-03-15T22:34:44.666 INFO:tasks.rados.rados.0.smithi076.stdout:3500: read oid 19 snap -1 2021-03-15T22:34:44.667 INFO:tasks.rados.rados.0.smithi076.stdout:3500: expect (ObjNum 1171 snap 409 seq_num 1171) 2021-03-15T22:34:44.667 INFO:tasks.rados.rados.0.smithi076.stdout:3501: delete oid 29 current snap is 416 2021-03-15T22:34:44.667 INFO:tasks.rados.rados.0.smithi076.stdout:3495: finishing write tid 4 to smithi07694878-2 2021-03-15T22:34:44.667 INFO:tasks.rados.rados.0.smithi076.stdout:3495: finishing write tid 5 to smithi07694878-2 2021-03-15T22:34:44.668 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 2424 (ObjNum 1189 snap 416 seq_num 1189) dirty exists 2021-03-15T22:34:44.668 INFO:tasks.rados.rados.0.smithi076.stdout:3495: left oid 2 (ObjNum 1189 snap 416 seq_num 1189) 2021-03-15T22:34:44.668 INFO:tasks.rados.rados.0.smithi076.stdout:3498: expect (ObjNum 1161 snap 407 seq_num 1161) 2021-03-15T22:34:44.701 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:44.723 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:44.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:44.725 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:44.733 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[56514]: cluster 2021-03-15T22:34:43.208884+0000 mgr.y (mgr.14144) 1042 : cluster [DBG] pgmap v1538: 158 pgs: 2 peering, 1 active+remapped+backfill_toofull, 155 active+clean; 709 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-03-15T22:34:44.734 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[56514]: cluster 2021-03-15T22:34:43.393270+0000 osd.7 (osd.7) 173 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:34:44.734 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[56514]: cluster 2021-03-15T22:34:43.447599+0000 osd.6 (osd.6) 139 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:34:44.734 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[56514]: cluster 2021-03-15T22:34:44.050362+0000 osd.1 (osd.1) 176 : cluster [DBG] 4.f scrub ok 2021-03-15T22:34:44.734 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[56514]: cluster 2021-03-15T22:34:44.331632+0000 mon.a (mon.0) 2339 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2021-03-15T22:34:44.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: cluster 2021-03-15T22:34:43.208884+0000 2021-03-15T22:34:44.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: mgr.y (mgr.14144) 1042 : cluster [DBG] pgmap v1538: 158 pgs: 2 peering, 1 active+remapped+backfill_toofull, 155 active+clean; 709 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-03-15T22:34:44.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: cluster 2021-03-15T22:34:43.393270+0000 osd.7 (osd.7) 173 : 2021-03-15T22:34:44.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: cluster [DBG] 6.1c scrub ok 2021-03-15T22:34:44.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:34:44.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: 15T22:34:43.447599+0000 osd.6 (osd.6) 139 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:34:44.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:34:44.738 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: 15T22:34:44.050362+0000 osd.1 (osd.1 2021-03-15T22:34:44.738 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: ) 176 : cluster [DBG] 4.f scrub ok 2021-03-15T22:34:44.739 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:34:44.739 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: 15T22:34:44.331632+0000 mon.a 2021-03-15T22:34:44.740 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: (mon.0) 2339 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2021-03-15T22:34:44.740 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:44 smithi076 conmon[62156]: 2021-03-15T22:34:44.786 INFO:tasks.rados.rados.0.smithi076.stdout:3499: finishing write tid 1 to smithi07694878-45 2021-03-15T22:34:44.786 INFO:tasks.rados.rados.0.smithi076.stdout:3495: done (4 left) 2021-03-15T22:34:44.787 INFO:tasks.rados.rados.0.smithi076.stdout:3498: done (3 left) 2021-03-15T22:34:44.787 INFO:tasks.rados.rados.0.smithi076.stdout:3501: done (2 left) 2021-03-15T22:34:44.787 INFO:tasks.rados.rados.0.smithi076.stdout:3502: snap_create 2021-03-15T22:34:44.787 INFO:tasks.rados.rados.0.smithi076.stdout:3499: finishing write tid 2 to smithi07694878-45 2021-03-15T22:34:44.788 INFO:tasks.rados.rados.0.smithi076.stdout:3499: finishing write tid 3 to smithi07694878-45 2021-03-15T22:34:44.788 INFO:tasks.rados.rados.0.smithi076.stdout:3499: finishing write tid 4 to smithi07694878-45 2021-03-15T22:34:44.788 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 2121 (ObjNum 1191 snap 416 seq_num 1191) dirty exists 2021-03-15T22:34:44.788 INFO:tasks.rados.rados.0.smithi076.stdout:3499: left oid 45 (ObjNum 1191 snap 416 seq_num 1191) 2021-03-15T22:34:44.789 INFO:tasks.rados.rados.0.smithi076.stdout:3500: expect (ObjNum 1171 snap 409 seq_num 1171) 2021-03-15T22:34:44.826 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:44.851 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:44.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:44.853 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:44.955 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:44.977 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:44.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:44.978 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:45.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: cluster 2021-03-15T22:34:43.208884+0000 mgr.y (mgr.14144) 1042 2021-03-15T22:34:45.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: : cluster [DBG] pgmap v1538: 158 pgs: 2 peering, 1 active+remapped+backfill_toofull, 155 active+clean; 709 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.9 MiB/s wr, 16 op/s 2021-03-15T22:34:45.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: cluster 2021-03-15T22:34:43.393270+0000 osd.7 (osd.7) 173 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:34:45.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:34:45.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: -03-15T22:34:43.447599+0000 osd.6 (osd.6) 139 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:34:45.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:34:45.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: -03-15T22:34:44.050362 2021-03-15T22:34:45.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: +0000 osd.1 (osd.1) 176 : cluster [DBG] 2021-03-15T22:34:45.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: 4.f scrub ok 2021-03-15T22:34:45.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: cluster 2021-03-15T22:34 2021-03-15T22:34:45.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: :44.331632+0000 mon.a (mon.0) 2021-03-15T22:34:45.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:44 smithi125 conmon[56739]: 2339 : cluster [DBG] osdmap e929: 8 total, 8 up, 7 in 2021-03-15T22:34:45.080 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:45.103 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:45.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:45.105 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:45.206 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:45.227 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:45.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:45.229 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:45.326 INFO:tasks.rados.rados.0.smithi076.stdout:3499: done (2 left) 2021-03-15T22:34:45.326 INFO:tasks.rados.rados.0.smithi076.stdout:3500: done (1 left) 2021-03-15T22:34:45.326 INFO:tasks.rados.rados.0.smithi076.stdout:3502: done (0 left) 2021-03-15T22:34:45.327 INFO:tasks.rados.rados.0.smithi076.stdout:3503: rollback oid 25 current snap is 417 2021-03-15T22:34:45.327 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 358 2021-03-15T22:34:45.327 INFO:tasks.rados.rados.0.smithi076.stdout:3504: write (excl) oid 39 current snap is 417 2021-03-15T22:34:45.327 INFO:tasks.rados.rados.0.smithi076.stdout:3504: seq_num 1193 ranges {450751=723125,1945018=105733} 2021-03-15T22:34:45.330 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:45.333 INFO:tasks.rados.rados.0.smithi076.stdout:3504: writing smithi07694878-39 from 450751 to 1173876 tid 1 2021-03-15T22:34:45.338 INFO:tasks.rados.rados.0.smithi076.stdout:3504: writing smithi07694878-39 from 1945018 to 2050751 tid 2 2021-03-15T22:34:45.339 INFO:tasks.rados.rados.0.smithi076.stdout:3505: write (excl) oid 8 current snap is 417 2021-03-15T22:34:45.339 INFO:tasks.rados.rados.0.smithi076.stdout:3505: seq_num 1194 ranges {467558=514727,1719853=347705} 2021-03-15T22:34:45.344 INFO:tasks.rados.rados.0.smithi076.stdout:3505: writing smithi07694878-8 from 467558 to 982285 tid 1 2021-03-15T22:34:45.352 INFO:tasks.rados.rados.0.smithi076.stdout:3505: writing smithi07694878-8 from 1719853 to 2067558 tid 2 2021-03-15T22:34:45.352 INFO:tasks.rados.rados.0.smithi076.stdout:3506: read oid 13 snap -1 2021-03-15T22:34:45.353 INFO:tasks.rados.rados.0.smithi076.stdout:3506: expect deleted 2021-03-15T22:34:45.353 INFO:tasks.rados.rados.0.smithi076.stdout:3507: rollback oid 18 current snap is 417 2021-03-15T22:34:45.353 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 18 to 297 2021-03-15T22:34:45.353 INFO:tasks.rados.rados.0.smithi076.stdout:3504: finishing write tid 1 to smithi07694878-39 2021-03-15T22:34:45.354 INFO:tasks.rados.rados.0.smithi076.stdout:3504: finishing write tid 2 to smithi07694878-39 2021-03-15T22:34:45.354 INFO:tasks.rados.rados.0.smithi076.stdout:3504: finishing write tid 3 to smithi07694878-39 2021-03-15T22:34:45.354 INFO:tasks.rados.rados.0.smithi076.stdout:3504: finishing write tid 4 to smithi07694878-39 2021-03-15T22:34:45.354 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2791 (ObjNum 1193 snap 417 seq_num 1193) dirty exists 2021-03-15T22:34:45.355 INFO:tasks.rados.rados.0.smithi076.stdout:3504: left oid 39 (ObjNum 1193 snap 417 seq_num 1193) 2021-03-15T22:34:45.355 INFO:tasks.rados.rados.0.smithi076.stdout:3505: finishing write tid 1 to smithi07694878-8 2021-03-15T22:34:45.356 INFO:tasks.rados.rados.0.smithi076.stdout:3504: done (4 left) 2021-03-15T22:34:45.356 INFO:tasks.rados.rados.0.smithi076.stdout:3508: read oid 41 snap -1 2021-03-15T22:34:45.356 INFO:tasks.rados.rados.0.smithi076.stdout:3508: expect (ObjNum 1182 snap 414 seq_num 1182) 2021-03-15T22:34:45.356 INFO:tasks.rados.rados.0.smithi076.stdout:3509: read oid 50 snap 357 2021-03-15T22:34:45.357 INFO:tasks.rados.rados.0.smithi076.stdout:3509: expect (ObjNum 1004 snap 337 seq_num 1004) 2021-03-15T22:34:45.357 INFO:tasks.rados.rados.0.smithi076.stdout:3510: snap_remove snap 308 2021-03-15T22:34:45.357 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:45.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:45.359 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:45.460 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:45.482 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:45.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:45.484 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:45.545 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:34:45.546 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:34:45.546 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:34:45.546 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:34:45.547 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:34:45.547 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:34:45.547 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:34:45.547 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:34:45.548 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:34:45.548 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:34:45.548 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:34:45.548 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:34:45.549 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:34:45.549 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:34:45.549 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:34:45.549 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:34:45.585 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:45.609 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:45.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:45.611 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:45.712 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:45.734 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:45.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:45.735 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:45.836 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:45.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:45.859 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:45.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:45.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:45.982 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:45.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:45.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:46.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:46.107 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:46.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:46.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:46.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:46.230 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:46.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:46.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:46.333 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:46.334 INFO:tasks.rados.rados.0.smithi076.stdout:3510: done (6 left) 2021-03-15T22:34:46.334 INFO:tasks.rados.rados.0.smithi076.stdout:3511: rollback oid 28 current snap is 417 2021-03-15T22:34:46.335 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 28 to 389 2021-03-15T22:34:46.335 INFO:tasks.rados.rados.0.smithi076.stdout:3512: write oid 46 current snap is 417 2021-03-15T22:34:46.335 INFO:tasks.rados.rados.0.smithi076.stdout:3512: seq_num 1195 ranges {484365=589976,1661982=422383} 2021-03-15T22:34:46.340 INFO:tasks.rados.rados.0.smithi076.stdout:3512: writing smithi07694878-46 from 484365 to 1074341 tid 1 2021-03-15T22:34:46.346 INFO:tasks.rados.rados.0.smithi076.stdout:3512: writing smithi07694878-46 from 1661982 to 2084365 tid 2 2021-03-15T22:34:46.346 INFO:tasks.rados.rados.0.smithi076.stdout:3509: expect (ObjNum 1004 snap 337 seq_num 1004) 2021-03-15T22:34:46.364 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:46.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:46.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:46.466 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:46.476 INFO:tasks.rados.rados.0.smithi076.stdout:3507: finishing rollback tid 0 to smithi07694878-18 2021-03-15T22:34:46.477 INFO:tasks.rados.rados.0.smithi076.stdout:3507: finishing rollback tid 1 to smithi07694878-18 2021-03-15T22:34:46.477 INFO:tasks.rados.rados.0.smithi076.stdout:3507: finishing rollback tid 2 to smithi07694878-18 2021-03-15T22:34:46.477 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 2421 (ObjNum 881 snap 292 seq_num 881) dirty exists 2021-03-15T22:34:46.478 INFO:tasks.rados.rados.0.smithi076.stdout:3506: done (7 left) 2021-03-15T22:34:46.478 INFO:tasks.rados.rados.0.smithi076.stdout:3507: done (6 left) 2021-03-15T22:34:46.478 INFO:tasks.rados.rados.0.smithi076.stdout:3509: done (5 left) 2021-03-15T22:34:46.478 INFO:tasks.rados.rados.0.smithi076.stdout:3513: read oid 2 snap 392 2021-03-15T22:34:46.479 INFO:tasks.rados.rados.0.smithi076.stdout:3513: expect (ObjNum 1111 snap 386 seq_num 1111) 2021-03-15T22:34:46.479 INFO:tasks.rados.rados.0.smithi076.stdout:3514: snap_remove snap 394 2021-03-15T22:34:46.483 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[62156]: cluster 2021-03-15T22:34:45.209261+0000 2021-03-15T22:34:46.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[62156]: mgr.y (mgr.14144) 1043 : cluster [DBG] pgmap v1541: 158 pgs: 1 remapped+peering, 2 peering, 1 active+remapped+backfill_toofull, 154 active+clean; 690 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 503 KiB/s wr, 2 op/s 2021-03-15T22:34:46.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[62156]: cluster 2021-03-15T22:34:45.323314+0000 mon.a (mon.0) 2340 : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2021-03-15T22:34:46.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[62156]: audit 2021-03-15T22:34:45.377821+0000 mon.a (mon.0) 2341 : audit [DBG] from='client.? 172.21.15.76:0/3204763934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:46.484 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[62156]: cluster 2021-03-15T22:34:45.540676+0000 osd.3 (osd.3) 167 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:46.485 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[62156]: cluster 2021-03-15T22:34:45.543745+0000 osd.3 (osd.3) 168 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:46.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[56514]: cluster 2021-03-15T22:34:45.209261+0000 mgr.y ( 2021-03-15T22:34:46.485 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[56514]: mgr.14144) 1043 : cluster [DBG] pgmap v1541: 158 pgs: 1 remapped+peering, 2 peering, 1 active+remapped+backfill_toofull, 154 active+clean; 690 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 503 KiB/s wr, 2 op/s 2021-03-15T22:34:46.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[56514]: cluster 2021-03-15T22:34:45.323314+0000 mon.a (mon.0) 2340 : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2021-03-15T22:34:46.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[56514]: audit 2021-03-15T22:34:45.377821+0000 mon.a (mon.0) 2341 : audit [DBG] from='client.? 172.21.15.76:0/3204763934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:46.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[56514]: cluster 2021-03-15T22:34:45.540676+0000 osd.3 (osd.3) 167 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:46.486 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:46 smithi076 conmon[56514]: cluster 2021-03-15T22:34:45.543745+0000 osd.3 (osd.3) 168 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:46.491 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:46.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:46.492 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:46.518 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:34:46.561 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:34:46.561 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:34:46.562 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:34:46.562 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:34:46.562 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:34:46.594 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:46.615 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:46.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:46.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:46.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:46.741 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:46.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:46.744 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:46.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:46 smithi125 conmon[56739]: cluster 2021-03-15T22:34:45.209261+0000 mgr.y (mgr.14144) 1043 : cluster [DBG] pgmap v1541: 158 pgs: 1 remapped+peering, 2 peering, 1 active+remapped+backfill_toofull, 154 active+clean; 690 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 1.5 KiB/s rd, 503 KiB/s wr, 2 op/s 2021-03-15T22:34:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:46 smithi125 conmon[56739]: cluster 2021-03-15T22:34:45.323314+0000 mon.a (mon.0) 2340 : cluster [DBG] osdmap e930: 8 total, 8 up, 7 in 2021-03-15T22:34:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:46 smithi125 conmon[56739]: audit 2021-03-15T22:34:45.377821+0000 mon.a (mon.0) 2341 : audit [DBG] from='client.? 172.21.15.76:0/3204763934' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:46 smithi125 conmon[56739]: cluster 2021-03-15T22:34:45.540676+0000 osd.3 (osd.3) 167 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:46.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:46 smithi125 conmon[56739]: cluster 2021-03-15T22:34:45.543745+0000 osd.3 (osd.3) 168 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:46.845 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:46.868 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:46.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:46.869 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:46.970 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:46.992 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:46.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:46.993 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:47.094 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:47.116 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:47.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:47.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:47.218 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:47.240 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:47.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:47.242 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:47.309 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:47.328 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:34:47.341 INFO:tasks.rados.rados.0.smithi076.stdout:3514: done (6 left) 2021-03-15T22:34:47.341 INFO:tasks.rados.rados.0.smithi076.stdout:3515: write oid 48 current snap is 417 2021-03-15T22:34:47.342 INFO:tasks.rados.rados.0.smithi076.stdout:3515: seq_num 1196 ranges {501172=665225,1720464=380708} 2021-03-15T22:34:47.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:47.349 INFO:tasks.rados.rados.0.smithi076.stdout:3515: writing smithi07694878-48 from 501172 to 1166397 tid 1 2021-03-15T22:34:47.356 INFO:tasks.rados.rados.0.smithi076.stdout:3515: writing smithi07694878-48 from 1720464 to 2101172 tid 2 2021-03-15T22:34:47.357 INFO:tasks.rados.rados.0.smithi076.stdout:3516: write oid 7 current snap is 417 2021-03-15T22:34:47.357 INFO:tasks.rados.rados.0.smithi076.stdout:3516: seq_num 1197 ranges {517979=740474,1778946=339033} 2021-03-15T22:34:47.365 INFO:tasks.rados.rados.0.smithi076.stdout:3516: writing smithi07694878-7 from 517979 to 1258453 tid 1 2021-03-15T22:34:47.370 INFO:tasks.rados.rados.0.smithi076.stdout:3516: writing smithi07694878-7 from 1778946 to 2117979 tid 2 2021-03-15T22:34:47.370 INFO:tasks.rados.rados.0.smithi076.stdout:3517: write oid 49 current snap is 417 2021-03-15T22:34:47.370 INFO:tasks.rados.rados.0.smithi076.stdout:3505: finishing write tid 2 to smithi07694878-8 2021-03-15T22:34:47.370 INFO:tasks.rados.rados.0.smithi076.stdout:3517: seq_num 1198 ranges {534786=415723,1437428=697358} 2021-03-15T22:34:47.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:47.372 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:47.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:47.375 INFO:tasks.rados.rados.0.smithi076.stdout:3517: writing smithi07694878-49 from 534786 to 950509 tid 1 2021-03-15T22:34:47.381 INFO:tasks.rados.rados.0.smithi076.stdout:3517: writing smithi07694878-49 from 1437428 to 2134786 tid 2 2021-03-15T22:34:47.382 INFO:tasks.rados.rados.0.smithi076.stdout:3518: snap_remove snap 390 2021-03-15T22:34:47.473 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:47.496 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:47.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:47.553 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:47 smithi076 conmon[56514]: cluster 2021-03-15T22:34:46.331467+0000 mon.a (mon.0) 2342 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2021-03-15T22:34:47.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:47 smithi076 conmon[62156]: cluster 2021-03-15T22:34:46. 2021-03-15T22:34:47.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:47 smithi076 conmon[62156]: 331467+0000 mon.a (mon.0) 2342 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2021-03-15T22:34:47.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:47.620 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:47.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:47.622 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:47.723 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:47.745 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:47.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:47.747 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:47.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:47 smithi125 conmon[56739]: cluster 2021-03-15T22:34:47.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:47 smithi125 conmon[56739]: 2021-03-15T22:34:46.331467+0000 2021-03-15T22:34:47.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:47 smithi125 conmon[56739]: mon.a (mon.0) 2342 : cluster [DBG] osdmap e931: 8 total, 8 up, 7 in 2021-03-15T22:34:47.834 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:47.848 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:47.869 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:47.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:47.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:47.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:47.994 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:47.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:47.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:48.096 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:48.118 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:48.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:48.120 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:48.221 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:48.240 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:34:48.243 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:48.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:48.244 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:48.345 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:48.349 INFO:tasks.rados.rados.0.smithi076.stdout:3518: done (9 left) 2021-03-15T22:34:48.349 INFO:tasks.rados.rados.0.smithi076.stdout:3519: read oid 45 snap -1 2021-03-15T22:34:48.349 INFO:tasks.rados.rados.0.smithi076.stdout:3519: expect (ObjNum 1191 snap 416 seq_num 1191) 2021-03-15T22:34:48.350 INFO:tasks.rados.rados.0.smithi076.stdout:3520: write (excl) oid 39 current snap is 417 2021-03-15T22:34:48.350 INFO:tasks.rados.rados.0.smithi076.stdout:3520: seq_num 1199 ranges {551593=490972,1779557=372036} 2021-03-15T22:34:48.355 INFO:tasks.rados.rados.0.smithi076.stdout:3520: writing smithi07694878-39 from 551593 to 1042565 tid 1 2021-03-15T22:34:48.363 INFO:tasks.rados.rados.0.smithi076.stdout:3520: writing smithi07694878-39 from 1779557 to 2151593 tid 2 2021-03-15T22:34:48.364 INFO:tasks.rados.rados.0.smithi076.stdout:3521: read oid 1 snap -1 2021-03-15T22:34:48.364 INFO:tasks.rados.rados.0.smithi076.stdout:3521: expect deleted 2021-03-15T22:34:48.364 INFO:tasks.rados.rados.0.smithi076.stdout:3505: finishing write tid 3 to smithi07694878-8 2021-03-15T22:34:48.364 INFO:tasks.rados.rados.0.smithi076.stdout:3505: finishing write tid 4 to smithi07694878-8 2021-03-15T22:34:48.365 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 2415 (ObjNum 1194 snap 417 seq_num 1194) dirty exists 2021-03-15T22:34:48.365 INFO:tasks.rados.rados.0.smithi076.stdout:3505: left oid 8 (ObjNum 1194 snap 417 seq_num 1194) 2021-03-15T22:34:48.365 INFO:tasks.rados.rados.0.smithi076.stdout:3508: expect (ObjNum 1182 snap 414 seq_num 1182) 2021-03-15T22:34:48.375 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:48.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:48.377 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:48.478 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:48.502 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:48.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:48.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:48.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[56514]: cluster 2021-03-15T22:34:47.209809+0000 2021-03-15T22:34:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[56514]: mgr.y (mgr.14144) 1044 : cluster [DBG] pgmap v1544: 158 pgs: 1 remapped+peering, 3 peering, 154 active+clean; 655 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 2.0 MiB/s wr, 24 op/s; 6.7 MiB/s, 21 objects/s recovering 2021-03-15T22:34:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[56514]: cluster 2021-03-15T22:34:47.328153+0000 mon.a (mon.0) 2343 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:34:48.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[56514]: cluster 2021-03-15T22:34:47.328197+0000 mon.a (mon.0) 2344 : cluster [INF] Cluster is now healthy 2021-03-15T22:34:48.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[56514]: cluster 2021-03-15T22:34:47.336669+0000 mon.a (mon.0) 2345 : cluster [DBG] osdmap e932: 8 total, 8 up, 7 in 2021-03-15T22:34:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: cluster 2021-03-15T22:34:47. 2021-03-15T22:34:48.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: 209809+0000 mgr.y (mgr.14144) 1044 : cluster [DBG] pgmap v1544: 158 pgs: 1 remapped+peering, 3 peering, 154 active+clean; 655 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 2.0 MiB/s wr, 24 op/s; 6.7 MiB/s, 21 objects/s recovering 2021-03-15T22:34:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: cluster 2021-03-15T22:34:47 2021-03-15T22:34:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: .328153+0000 mon.a (mon.0) 2343 2021-03-15T22:34:48.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:34:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:34:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: -03-15T22:34:47. 2021-03-15T22:34:48.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: 328197+0000 mon.a (mon.0) 2344 : 2021-03-15T22:34:48.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: cluster [INF] Cluster is now healthy 2021-03-15T22:34:48.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:34:48.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: -03-15T22:34:47.336669+0000 2021-03-15T22:34:48.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: mon.a (mon.0) 2345 : cluster [DBG] 2021-03-15T22:34:48.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:48 smithi076 conmon[62156]: osdmap e932: 8 total, 8 up, 7 in 2021-03-15T22:34:48.555 INFO:tasks.rados.rados.0.smithi076.stdout:3512: finishing write tid 1 to smithi07694878-46 2021-03-15T22:34:48.555 INFO:tasks.rados.rados.0.smithi076.stdout:3511: finishing rollback tid 0 to smithi07694878-28 2021-03-15T22:34:48.556 INFO:tasks.rados.rados.0.smithi076.stdout:3505: done (11 left) 2021-03-15T22:34:48.556 INFO:tasks.rados.rados.0.smithi076.stdout:3508: done (10 left) 2021-03-15T22:34:48.556 INFO:tasks.rados.rados.0.smithi076.stdout:3522: rollback oid 36 current snap is 417 2021-03-15T22:34:48.557 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 36 to 349 2021-03-15T22:34:48.557 INFO:tasks.rados.rados.0.smithi076.stdout:3511: finishing rollback tid 1 to smithi07694878-28 2021-03-15T22:34:48.558 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 3086 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:34:48.558 INFO:tasks.rados.rados.0.smithi076.stdout:3503: finishing rollback tid 0 to smithi07694878-25 2021-03-15T22:34:48.558 INFO:tasks.rados.rados.0.smithi076.stdout:3503: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:34:48.559 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 2426 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:34:48.559 INFO:tasks.rados.rados.0.smithi076.stdout:3512: finishing write tid 2 to smithi07694878-46 2021-03-15T22:34:48.559 INFO:tasks.rados.rados.0.smithi076.stdout:3512: finishing write tid 3 to smithi07694878-46 2021-03-15T22:34:48.560 INFO:tasks.rados.rados.0.smithi076.stdout:3512: finishing write tid 4 to smithi07694878-46 2021-03-15T22:34:48.560 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2287 (ObjNum 1195 snap 417 seq_num 1195) dirty exists 2021-03-15T22:34:48.561 INFO:tasks.rados.rados.0.smithi076.stdout:3512: left oid 46 (ObjNum 1195 snap 417 seq_num 1195) 2021-03-15T22:34:48.561 INFO:tasks.rados.rados.0.smithi076.stdout:3513: expect (ObjNum 1111 snap 386 seq_num 1111) 2021-03-15T22:34:48.605 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:48.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:48.723 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 7, 0, 3, 5, 6] out_osds: [2] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:34:48.724 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:34:48.724 INFO:tasks.thrashosds.thrasher:Adding osd 2 2021-03-15T22:34:48.725 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd in 2 2021-03-15T22:34:48.731 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:48.754 INFO:tasks.rados.rados.0.smithi076.stdout:3515: finishing write tid 1 to smithi07694878-48 2021-03-15T22:34:48.754 INFO:tasks.rados.rados.0.smithi076.stdout:3515: finishing write tid 2 to smithi07694878-48 2021-03-15T22:34:48.754 INFO:tasks.rados.rados.0.smithi076.stdout:3503: done (10 left) 2021-03-15T22:34:48.755 INFO:tasks.rados.rados.0.smithi076.stdout:3511: done (9 left) 2021-03-15T22:34:48.755 INFO:tasks.rados.rados.0.smithi076.stdout:3512: done (8 left) 2021-03-15T22:34:48.755 INFO:tasks.rados.rados.0.smithi076.stdout:3513: done (7 left) 2021-03-15T22:34:48.755 INFO:tasks.rados.rados.0.smithi076.stdout:3523: read oid 17 snap -1 2021-03-15T22:34:48.755 INFO:tasks.rados.rados.0.smithi076.stdout:3523: expect (ObjNum 960 snap 321 seq_num 960) 2021-03-15T22:34:48.756 INFO:tasks.rados.rados.0.smithi076.stdout:3524: delete oid 31 current snap is 417 2021-03-15T22:34:48.756 INFO:tasks.rados.rados.0.smithi076.stdout:3515: finishing write tid 3 to smithi07694878-48 2021-03-15T22:34:48.756 INFO:tasks.rados.rados.0.smithi076.stdout:3515: finishing write tid 4 to smithi07694878-48 2021-03-15T22:34:48.756 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 2418 (ObjNum 1196 snap 417 seq_num 1196) dirty exists 2021-03-15T22:34:48.756 INFO:tasks.rados.rados.0.smithi076.stdout:3515: left oid 48 (ObjNum 1196 snap 417 seq_num 1196) 2021-03-15T22:34:48.757 INFO:tasks.rados.rados.0.smithi076.stdout:3516: finishing write tid 1 to smithi07694878-7 2021-03-15T22:34:48.757 INFO:tasks.rados.rados.0.smithi076.stdout:3516: finishing write tid 2 to smithi07694878-7 2021-03-15T22:34:48.757 INFO:tasks.rados.rados.0.smithi076.stdout:3516: finishing write tid 3 to smithi07694878-7 2021-03-15T22:34:48.757 INFO:tasks.rados.rados.0.smithi076.stdout:3516: finishing write tid 4 to smithi07694878-7 2021-03-15T22:34:48.758 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2794 (ObjNum 1197 snap 417 seq_num 1197) dirty exists 2021-03-15T22:34:48.758 INFO:tasks.rados.rados.0.smithi076.stdout:3516: left oid 7 (ObjNum 1197 snap 417 seq_num 1197) 2021-03-15T22:34:48.758 INFO:tasks.rados.rados.0.smithi076.stdout:3517: finishing write tid 1 to smithi07694878-49 2021-03-15T22:34:48.758 INFO:tasks.rados.rados.0.smithi076.stdout:3517: finishing write tid 2 to smithi07694878-49 2021-03-15T22:34:48.758 INFO:tasks.rados.rados.0.smithi076.stdout:3517: finishing write tid 3 to smithi07694878-49 2021-03-15T22:34:48.759 INFO:tasks.rados.rados.0.smithi076.stdout:3517: finishing write tid 4 to smithi07694878-49 2021-03-15T22:34:48.759 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 2863 (ObjNum 1198 snap 417 seq_num 1198) dirty exists 2021-03-15T22:34:48.759 INFO:tasks.rados.rados.0.smithi076.stdout:3517: left oid 49 (ObjNum 1198 snap 417 seq_num 1198) 2021-03-15T22:34:48.759 INFO:tasks.rados.rados.0.smithi076.stdout:3519: expect (ObjNum 1191 snap 416 seq_num 1191) 2021-03-15T22:34:48.773 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:48.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:48.774 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:48.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:48 smithi125 conmon[56739]: cluster 2021-03-15T22:34:47.209809+0000 2021-03-15T22:34:48.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:48 smithi125 conmon[56739]: mgr.y (mgr.14144) 1044 : cluster [DBG] pgmap v1544: 158 pgs: 1 remapped+peering, 3 peering, 154 active+clean; 655 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 2.0 MiB/s wr, 24 op/s; 6.7 MiB/s, 21 objects/s recovering 2021-03-15T22:34:48.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:48 smithi125 conmon[56739]: cluster 2021-03-15T22:34:47.328153+0000 mon.a (mon.0) 2343 2021-03-15T22:34:48.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:48 smithi125 conmon[56739]: : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:34:48.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:48 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:34:48.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:48 smithi125 conmon[56739]: :34:47.328197+0000 mon.a (mon.0) 2344 : cluster [INF] Cluster is now healthy 2021-03-15T22:34:48.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:48 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:34:48.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:48 smithi125 conmon[56739]: -15T22:34:47.336669+0000 mon.a ( 2021-03-15T22:34:48.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:48 smithi125 conmon[56739]: mon.0) 2345 : cluster [DBG] osdmap e932: 8 total, 8 up, 7 in 2021-03-15T22:34:48.840 INFO:tasks.rados.rados.0.smithi076.stdout:3520: finishing write tid 1 to smithi07694878-39 2021-03-15T22:34:48.843 INFO:tasks.rados.rados.0.smithi076.stdout:3520: finishing write tid 2 to smithi07694878-39 2021-03-15T22:34:48.844 INFO:tasks.rados.rados.0.smithi076.stdout:3520: finishing write tid 3 to smithi07694878-39 2021-03-15T22:34:48.844 INFO:tasks.rados.rados.0.smithi076.stdout:3515: done (8 left) 2021-03-15T22:34:48.844 INFO:tasks.rados.rados.0.smithi076.stdout:3516: done (7 left) 2021-03-15T22:34:48.844 INFO:tasks.rados.rados.0.smithi076.stdout:3517: done (6 left) 2021-03-15T22:34:48.845 INFO:tasks.rados.rados.0.smithi076.stdout:3519: done (5 left) 2021-03-15T22:34:48.845 INFO:tasks.rados.rados.0.smithi076.stdout:3521: done (4 left) 2021-03-15T22:34:48.845 INFO:tasks.rados.rados.0.smithi076.stdout:3524: done (3 left) 2021-03-15T22:34:48.845 INFO:tasks.rados.rados.0.smithi076.stdout:3525: write (excl) oid 44 current snap is 417 2021-03-15T22:34:48.845 INFO:tasks.rados.rados.0.smithi076.stdout:3520: finishing write tid 4 to smithi07694878-39 2021-03-15T22:34:48.846 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2797 (ObjNum 1199 snap 417 seq_num 1199) dirty exists 2021-03-15T22:34:48.846 INFO:tasks.rados.rados.0.smithi076.stdout:3520: left oid 39 (ObjNum 1199 snap 417 seq_num 1199) 2021-03-15T22:34:48.846 INFO:tasks.rados.rados.0.smithi076.stdout:3525: seq_num 1201 ranges {585207=641470,1896521=288686} 2021-03-15T22:34:48.847 INFO:tasks.rados.rados.0.smithi076.stdout:3525: writing smithi07694878-44 from 585207 to 1226677 tid 1 2021-03-15T22:34:48.849 INFO:tasks.rados.rados.0.smithi076.stdout:3525: writing smithi07694878-44 from 1896521 to 2185207 tid 2 2021-03-15T22:34:48.849 INFO:tasks.rados.rados.0.smithi076.stdout:3520: done (3 left) 2021-03-15T22:34:48.850 INFO:tasks.rados.rados.0.smithi076.stdout:3526: snap_create 2021-03-15T22:34:48.850 INFO:tasks.rados.rados.0.smithi076.stdout:3522: finishing rollback tid 1 to smithi07694878-36 2021-03-15T22:34:48.850 INFO:tasks.rados.rados.0.smithi076.stdout:3522: finishing rollback tid 2 to smithi07694878-36 2021-03-15T22:34:48.850 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 2011 (ObjNum 669 snap 212 seq_num 669) dirty exists 2021-03-15T22:34:48.851 INFO:tasks.rados.rados.0.smithi076.stdout:3523: expect (ObjNum 960 snap 321 seq_num 960) 2021-03-15T22:34:48.876 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:48.892 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:48.903 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:48.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:48.905 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:48.961 INFO:tasks.rados.rados.0.smithi076.stdout:3525: finishing write tid 1 to smithi07694878-44 2021-03-15T22:34:48.962 INFO:tasks.rados.rados.0.smithi076.stdout:3525: finishing write tid 2 to smithi07694878-44 2021-03-15T22:34:48.962 INFO:tasks.rados.rados.0.smithi076.stdout:3525: finishing write tid 3 to smithi07694878-44 2021-03-15T22:34:48.962 INFO:tasks.rados.rados.0.smithi076.stdout:3525: finishing write tid 4 to smithi07694878-44 2021-03-15T22:34:48.963 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 2424 (ObjNum 1201 snap 417 seq_num 1201) dirty exists 2021-03-15T22:34:48.963 INFO:tasks.rados.rados.0.smithi076.stdout:3525: left oid 44 (ObjNum 1201 snap 417 seq_num 1201) 2021-03-15T22:34:49.006 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:49.035 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:49.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:49.036 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:49.137 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:49.162 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:49.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:49.163 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:49.264 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:49.287 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:49.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:49.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:49.355 INFO:tasks.rados.rados.0.smithi076.stdout:3522: done (3 left) 2021-03-15T22:34:49.356 INFO:tasks.rados.rados.0.smithi076.stdout:3523: done (2 left) 2021-03-15T22:34:49.356 INFO:tasks.rados.rados.0.smithi076.stdout:3525: done (1 left) 2021-03-15T22:34:49.356 INFO:tasks.rados.rados.0.smithi076.stdout:3526: done (0 left) 2021-03-15T22:34:49.356 INFO:tasks.rados.rados.0.smithi076.stdout:3527: read oid 42 snap 375 2021-03-15T22:34:49.357 INFO:tasks.rados.rados.0.smithi076.stdout:3527: expect (ObjNum 1084 snap 371 seq_num 1084) 2021-03-15T22:34:49.357 INFO:tasks.rados.rados.0.smithi076.stdout:3528: write (excl) oid 48 current snap is 418 2021-03-15T22:34:49.357 INFO:tasks.rados.rados.0.smithi076.stdout:3528: seq_num 1202 ranges {602014=433072,1671356=530658} 2021-03-15T22:34:49.361 INFO:tasks.rados.rados.0.smithi076.stdout:3528: writing smithi07694878-48 from 602014 to 1035086 tid 1 2021-03-15T22:34:49.368 INFO:tasks.rados.rados.0.smithi076.stdout:3528: writing smithi07694878-48 from 1671356 to 2202014 tid 2 2021-03-15T22:34:49.368 INFO:tasks.rados.rados.0.smithi076.stdout:3529: write (excl) oid 8 current snap is 418 2021-03-15T22:34:49.368 INFO:tasks.rados.rados.0.smithi076.stdout:3529: seq_num 1203 ranges {618821=508321,1613485=531077} 2021-03-15T22:34:49.374 INFO:tasks.rados.rados.0.smithi076.stdout:3529: writing smithi07694878-8 from 618821 to 1127142 tid 1 2021-03-15T22:34:49.380 INFO:tasks.rados.rados.0.smithi076.stdout:3529: writing smithi07694878-8 from 1613485 to 2144562 tid 2 2021-03-15T22:34:49.380 INFO:tasks.rados.rados.0.smithi076.stdout:3530: copy_from oid 25 from oid 33 current snap is 418 2021-03-15T22:34:49.380 INFO:tasks.rados.rados.0.smithi076.stdout:3527: expect (ObjNum 1084 snap 371 seq_num 1084) 2021-03-15T22:34:49.389 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:49.415 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:49.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:49.416 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:49.459 INFO:tasks.rados.rados.0.smithi076.stdout:3528: finishing write tid 1 to smithi07694878-48 2021-03-15T22:34:49.460 INFO:tasks.rados.rados.0.smithi076.stdout:3528: finishing write tid 2 to smithi07694878-48 2021-03-15T22:34:49.460 INFO:tasks.rados.rados.0.smithi076.stdout:3528: finishing write tid 3 to smithi07694878-48 2021-03-15T22:34:49.460 INFO:tasks.rados.rados.0.smithi076.stdout:3528: finishing write tid 4 to smithi07694878-48 2021-03-15T22:34:49.460 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 2421 (ObjNum 1202 snap 418 seq_num 1202) dirty exists 2021-03-15T22:34:49.461 INFO:tasks.rados.rados.0.smithi076.stdout:3528: left oid 48 (ObjNum 1202 snap 418 seq_num 1202) 2021-03-15T22:34:49.461 INFO:tasks.rados.rados.0.smithi076.stdout:3529: finishing write tid 1 to smithi07694878-8 2021-03-15T22:34:49.461 INFO:tasks.rados.rados.0.smithi076.stdout:3527: done (3 left) 2021-03-15T22:34:49.461 INFO:tasks.rados.rados.0.smithi076.stdout:3528: done (2 left) 2021-03-15T22:34:49.462 INFO:tasks.rados.rados.0.smithi076.stdout:3531: snap_remove snap 352 2021-03-15T22:34:49.517 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:49.540 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:49.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:49.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:49.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:49.664 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:49.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:49.665 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:49.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:49 smithi076 conmon[62156]: cluster 2021-03-15T22:34:48.037742+0000 osd.1 (osd.1) 177 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:34:49.725 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:49 smithi076 conmon[62156]: cluster 2021-03-15T22:34:48.345270+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e933: 8 total, 8 up, 7 in 2021-03-15T22:34:49.726 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:49 smithi076 conmon[56514]: cluster 2021-03-15T22:34:49.726 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:49 smithi076 conmon[56514]: 2021-03-15T22: 2021-03-15T22:34:49.730 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:49 smithi076 conmon[56514]: 34:48.037742+0000 osd.1 (osd.1) 177 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:34:49.730 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:49 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:34:49.731 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:49 smithi076 conmon[56514]: -03-15T22:34:48.345270+0000 2021-03-15T22:34:49.732 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:49 smithi076 conmon[56514]: mon.a (mon.0) 2346 : cluster [DBG] 2021-03-15T22:34:49.732 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:49 smithi076 conmon[56514]: osdmap e933: 8 total, 8 up, 7 in 2021-03-15T22:34:49.762 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:49.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:49.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:49 smithi125 conmon[56739]: cluster 2021-03-15T22:34:48.037742+0000 osd.1 2021-03-15T22:34:49.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:49 smithi125 conmon[56739]: (osd.1) 177 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:34:49.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:49 smithi125 conmon[56739]: cluster 2021-03-15T22:34:48.345270+0000 mon.a (mon.0) 2346 : cluster [DBG] osdmap e933: 8 total, 8 up, 7 in 2021-03-15T22:34:49.788 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:49.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:49.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:49.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:49.912 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:49.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:50.015 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:50.037 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:50.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:50.038 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:50.139 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:50.161 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:50.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:50.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:50.264 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:50.289 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:50.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:50.291 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:50.330 INFO:tasks.rados.rados.0.smithi076.stdout:3531: done (2 left) 2021-03-15T22:34:50.330 INFO:tasks.rados.rados.0.smithi076.stdout:3532: write (excl) oid 19 current snap is 418 2021-03-15T22:34:50.331 INFO:tasks.rados.rados.0.smithi076.stdout:3529: finishing write tid 2 to smithi07694878-8 2021-03-15T22:34:50.331 INFO:tasks.rados.rados.0.smithi076.stdout:3532: seq_num 1204 ranges {635628=583570,1671967=563661} 2021-03-15T22:34:50.337 INFO:tasks.rados.rados.0.smithi076.stdout:3532: writing smithi07694878-19 from 635628 to 1219198 tid 1 2021-03-15T22:34:50.344 INFO:tasks.rados.rados.0.smithi076.stdout:3532: writing smithi07694878-19 from 1671967 to 2235628 tid 2 2021-03-15T22:34:50.344 INFO:tasks.rados.rados.0.smithi076.stdout:3533: copy_from oid 28 from oid 47 current snap is 418 2021-03-15T22:34:50.345 INFO:tasks.rados.rados.0.smithi076.stdout:3529: finishing write tid 3 to smithi07694878-8 2021-03-15T22:34:50.345 INFO:tasks.rados.rados.0.smithi076.stdout:3529: finishing write tid 4 to smithi07694878-8 2021-03-15T22:34:50.345 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 2424 (ObjNum 1203 snap 418 seq_num 1203) dirty exists 2021-03-15T22:34:50.345 INFO:tasks.rados.rados.0.smithi076.stdout:3529: left oid 8 (ObjNum 1203 snap 418 seq_num 1203) 2021-03-15T22:34:50.346 INFO:tasks.rados.rados.0.smithi076.stdout:3530: finishing copy_from to smithi07694878-25 2021-03-15T22:34:50.346 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 2429 (ObjNum 1174 snap 410 seq_num 1174) dirty exists 2021-03-15T22:34:50.346 INFO:tasks.rados.rados.0.smithi076.stdout:3530: finishing copy_from racing read to smithi07694878-25 2021-03-15T22:34:50.346 INFO:tasks.rados.rados.0.smithi076.stdout:3529: done (3 left) 2021-03-15T22:34:50.347 INFO:tasks.rados.rados.0.smithi076.stdout:3530: done (2 left) 2021-03-15T22:34:50.347 INFO:tasks.rados.rados.0.smithi076.stdout:3534: copy_from oid 42 from oid 48 current snap is 418 2021-03-15T22:34:50.347 INFO:tasks.rados.rados.0.smithi076.stdout:3535: read oid 31 snap -1 2021-03-15T22:34:50.347 INFO:tasks.rados.rados.0.smithi076.stdout:3535: expect deleted 2021-03-15T22:34:50.348 INFO:tasks.rados.rados.0.smithi076.stdout:3536: delete oid 49 current snap is 418 2021-03-15T22:34:50.348 INFO:tasks.rados.rados.0.smithi076.stdout:3533: finishing copy_from to smithi07694878-28 2021-03-15T22:34:50.349 INFO:tasks.rados.rados.0.smithi076.stdout:3533: got expected ENOENT (src dne) 2021-03-15T22:34:50.349 INFO:tasks.rados.rados.0.smithi076.stdout:3533: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:34:50.349 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:34:50.357 INFO:tasks.rados.rados.0.smithi076.stdout:3533: done (4 left) 2021-03-15T22:34:50.357 INFO:tasks.rados.rados.0.smithi076.stdout:3535: done (3 left) 2021-03-15T22:34:50.357 INFO:tasks.rados.rados.0.smithi076.stdout:3536: done (2 left) 2021-03-15T22:34:50.358 INFO:tasks.rados.rados.0.smithi076.stdout:3537: write (excl) oid 50 current snap is 418 2021-03-15T22:34:50.358 INFO:tasks.rados.rados.0.smithi076.stdout:3537: seq_num 1206 ranges {669242=734068,2188931=80311} 2021-03-15T22:34:50.365 INFO:tasks.rados.rados.0.smithi076.stdout:3537: writing smithi07694878-50 from 669242 to 1403310 tid 1 2021-03-15T22:34:50.371 INFO:tasks.rados.rados.0.smithi076.stdout:3537: writing smithi07694878-50 from 2188931 to 2269242 tid 2 2021-03-15T22:34:50.371 INFO:tasks.rados.rados.0.smithi076.stdout:3538: snap_remove snap 358 2021-03-15T22:34:50.391 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:50.413 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:50.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:50.415 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:50.516 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:50.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:34:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: cluster 2021-03-15T22:34:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: 2021-03 2021-03-15T22:34:50.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: -15T22:34:49.210365+0000 mgr.y (mgr.14144) 1045 : cluster [DBG] pgmap v1547: 158 pgs: 1 remapped+peering, 1 peering, 156 active+clean; 658 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 1.5 MiB/s wr, 21 op/s; 6.7 MiB/s, 26 objects/s recovering 2021-03-15T22:34:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: 2021-03-15T22:34:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:34:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: -03-15T22:34:49.353765+0000 2021-03-15T22:34:50.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: mon.a (mon.0) 2347 : cluster [DBG] 2021-03-15T22:34:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: osdmap e934: 8 total, 8 up, 7 in 2021-03-15T22:34:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:34:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: :34:49.671291+0000 mon.a (mon. 2021-03-15T22:34:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: 0) 2348 : audit [INF] from='client.? 172.21.15.76:0/1923463945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:34:50.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: cluster 2021-03-15T22:34:50.318073+0000 mon.a (mon.0) 2349 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:50.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: audit 2021-03-15T22:34:50.327746+0000 2021-03-15T22:34:50.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: mon.a (mon.0) 2350 : audit [INF] from='client.? 172.21.15.76:0/1923463945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:34:50.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:34:50.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: -15T22:34:50.327805+0000 mon.a 2021-03-15T22:34:50.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: (mon.0) 2351 : cluster [DBG] osdmap e935: 8 total, 8 up, 7 in 2021-03-15T22:34:50.541 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:50 smithi125 conmon[56739]: 2021-03-15T22:34:50.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:50.542 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:50.543 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:50.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[62156]: cluster 2021-03-15T22:34:49.210365+0000 mgr.y (mgr.14144) 1045 : cluster [DBG] pgmap v1547: 158 pgs: 1 remapped+peering, 1 peering, 156 active+clean; 658 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 1.5 MiB/s wr, 21 op/s; 6.7 MiB/s, 26 objects/s recovering 2021-03-15T22:34:50.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[62156]: cluster 2021-03-15T22:34:49.353765+0000 mon.a (mon.0) 2347 : cluster [DBG] osdmap e934: 8 total, 8 up, 7 in 2021-03-15T22:34:50.634 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[62156]: audit 2021-03-15T22:34:49.671291+0000 mon.a (mon.0) 2348 : audit [INF] from='client.? 172.21.15.76:0/1923463945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:34:50.634 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[62156]: cluster 2021-03-15T22:34:50.318073+0000 mon.a (mon.0) 2349 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:50.634 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[62156]: audit 2021-03-15T22:34:50.327746+0000 mon.a (mon.0) 2350 : audit [INF] from='client.? 172.21.15.76:0/1923463945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:34:50.634 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[62156]: cluster 2021-03-15T22:34:50.327805+0000 mon.a (mon.0) 2351 : cluster [DBG] osdmap e935: 8 total, 8 up, 7 in 2021-03-15T22:34:50.635 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: cluster 2021-03-15T22:34:50.635 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021-03-15T22:34:50.635 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021 2021-03-15T22:34:50.635 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: - 2021-03-15T22:34:50.636 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 03 2021-03-15T22:34:50.636 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: - 2021-03-15T22:34:50.636 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 15 2021-03-15T22:34:50.636 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: T 2021-03-15T22:34:50.637 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 22 2021-03-15T22:34:50.637 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: : 2021-03-15T22:34:50.637 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 34 2021-03-15T22:34:50.637 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: : 2021-03-15T22:34:50.637 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 49 2021-03-15T22:34:50.638 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: . 2021-03-15T22:34:50.638 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 210365 2021-03-15T22:34:50.638 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: +0000 2021-03-15T22:34:50.638 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021-03-15T22:34:50.638 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: mgr.y 2021-03-15T22:34:50.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: ( 2021-03-15T22:34:50.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: mgr 2021-03-15T22:34:50.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: . 2021-03-15T22:34:50.639 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 14144 2021-03-15T22:34:50.640 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: ) 2021-03-15T22:34:50.640 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 1045 2021-03-15T22:34:50.640 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: : 2021-03-15T22:34:50.640 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: cluster 2021-03-15T22:34:50.640 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021-03-15T22:34:50.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: [DBG] 2021-03-15T22:34:50.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021-03-15T22:34:50.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: pgmap v1547: 158 pgs: 1 remapped+peering, 1 peering, 156 active+clean; 658 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 21 MiB/s rd, 1.5 MiB/s wr, 21 op/s; 6.7 MiB/s, 26 objects/s recovering 2021-03-15T22:34:50.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021-03-15T22:34:50.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: cluster 2021-03-15T22:34:50.642 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021-03-15T22:34:50.642 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021 2021-03-15T22:34:50.642 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: - 2021-03-15T22:34:50.642 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 03 2021-03-15T22:34:50.642 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: - 2021-03-15T22:34:50.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 15 2021-03-15T22:34:50.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: T 2021-03-15T22:34:50.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 22 2021-03-15T22:34:50.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: : 2021-03-15T22:34:50.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 34 2021-03-15T22:34:50.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: : 2021-03-15T22:34:50.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 49 2021-03-15T22:34:50.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: . 2021-03-15T22:34:50.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 353765 2021-03-15T22:34:50.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: +0000 2021-03-15T22:34:50.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021-03-15T22:34:50.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: mon.a 2021-03-15T22:34:50.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: ( 2021-03-15T22:34:50.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: mon 2021-03-15T22:34:50.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: . 2021-03-15T22:34:50.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 0 2021-03-15T22:34:50.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: ) 2021-03-15T22:34:50.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2347 2021-03-15T22:34:50.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: : 2021-03-15T22:34:50.647 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: cluster 2021-03-15T22:34:50.647 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021-03-15T22:34:50.647 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: [DBG] 2021-03-15T22:34:50.647 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021-03-15T22:34:50.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: osdmap e934: 8 total, 8 up, 7 in 2021-03-15T22:34:50.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021-03-15T22:34:50.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: audit 2021-03-15T22:34:50.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 2021-03-15T22:34 2021-03-15T22:34:50.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: :49.671291+0000 mon.a (mon.0) 2348 : audit [INF] from='client.? 172.21.15.76:0/1923463945' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:34:50.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: cluster 2021-03-15T22:34: 2021-03-15T22:34:50.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 50.318073+0000 mon.a (mon.0) 2349 : 2021-03-15T22:34:50.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:50.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: audit 2021- 2021-03-15T22:34:50.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 03-15T22:34:50.327746+0000 2021-03-15T22:34:50.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: mon.a (mon.0) 2350 : audit [INF] 2021-03-15T22:34:50.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/1923463945' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:34:50.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:34:50.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 34:50.327805+0000 mon.a (mon. 2021-03-15T22:34:50.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:50 smithi076 conmon[56514]: 0) 2351 : cluster [DBG] osdmap e935: 8 total, 8 up, 7 in 2021-03-15T22:34:50.651 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:50.675 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:50.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:50.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:50.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:50.799 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:50.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:50.801 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:50.902 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:50.924 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:50.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:50.925 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:51.026 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:51.048 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:51.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:51.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:51.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:51.171 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:51.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:51.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:51.273 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:51.295 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:51.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:51.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:51.365 INFO:tasks.rados.rados.0.smithi076.stdout:3538: done (3 left) 2021-03-15T22:34:51.366 INFO:tasks.rados.rados.0.smithi076.stdout:3539: read oid 46 snap -1 2021-03-15T22:34:51.366 INFO:tasks.rados.rados.0.smithi076.stdout:3539: expect (ObjNum 1195 snap 417 seq_num 1195) 2021-03-15T22:34:51.366 INFO:tasks.rados.rados.0.smithi076.stdout:3540: rollback oid 30 current snap is 418 2021-03-15T22:34:51.366 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 30 to 368 2021-03-15T22:34:51.367 INFO:tasks.rados.rados.0.smithi076.stdout:3532: finishing write tid 1 to smithi07694878-19 2021-03-15T22:34:51.367 INFO:tasks.rados.rados.0.smithi076.stdout:3534: finishing copy_from to smithi07694878-42 2021-03-15T22:34:51.367 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2421 (ObjNum 1202 snap 418 seq_num 1202) dirty exists 2021-03-15T22:34:51.367 INFO:tasks.rados.rados.0.smithi076.stdout:3534: finishing copy_from racing read to smithi07694878-42 2021-03-15T22:34:51.368 INFO:tasks.rados.rados.0.smithi076.stdout:3532: finishing write tid 2 to smithi07694878-19 2021-03-15T22:34:51.368 INFO:tasks.rados.rados.0.smithi076.stdout:3534: done (4 left) 2021-03-15T22:34:51.368 INFO:tasks.rados.rados.0.smithi076.stdout:3541: read oid 8 snap -1 2021-03-15T22:34:51.368 INFO:tasks.rados.rados.0.smithi076.stdout:3541: expect (ObjNum 1203 snap 418 seq_num 1203) 2021-03-15T22:34:51.369 INFO:tasks.rados.rados.0.smithi076.stdout:3542: write oid 3 current snap is 418 2021-03-15T22:34:51.369 INFO:tasks.rados.rados.0.smithi076.stdout:3532: finishing write tid 3 to smithi07694878-19 2021-03-15T22:34:51.369 INFO:tasks.rados.rados.0.smithi076.stdout:3532: finishing write tid 4 to smithi07694878-19 2021-03-15T22:34:51.369 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2800 (ObjNum 1204 snap 418 seq_num 1204) dirty exists 2021-03-15T22:34:51.369 INFO:tasks.rados.rados.0.smithi076.stdout:3532: left oid 19 (ObjNum 1204 snap 418 seq_num 1204) 2021-03-15T22:34:51.370 INFO:tasks.rados.rados.0.smithi076.stdout:3537: finishing write tid 1 to smithi07694878-50 2021-03-15T22:34:51.370 INFO:tasks.rados.rados.0.smithi076.stdout:3537: finishing write tid 2 to smithi07694878-50 2021-03-15T22:34:51.370 INFO:tasks.rados.rados.0.smithi076.stdout:3537: finishing write tid 3 to smithi07694878-50 2021-03-15T22:34:51.370 INFO:tasks.rados.rados.0.smithi076.stdout:3537: finishing write tid 4 to smithi07694878-50 2021-03-15T22:34:51.371 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 2680 (ObjNum 1206 snap 418 seq_num 1206) dirty exists 2021-03-15T22:34:51.371 INFO:tasks.rados.rados.0.smithi076.stdout:3537: left oid 50 (ObjNum 1206 snap 418 seq_num 1206) 2021-03-15T22:34:51.371 INFO:tasks.rados.rados.0.smithi076.stdout:3542: seq_num 1207 ranges {686049=409317,1847413=438636} 2021-03-15T22:34:51.371 INFO:tasks.rados.rados.0.smithi076.stdout:3542: writing smithi07694878-3 from 686049 to 1095366 tid 1 2021-03-15T22:34:51.379 INFO:tasks.rados.rados.0.smithi076.stdout:3542: writing smithi07694878-3 from 1847413 to 2286049 tid 2 2021-03-15T22:34:51.379 INFO:tasks.rados.rados.0.smithi076.stdout:3532: done (5 left) 2021-03-15T22:34:51.379 INFO:tasks.rados.rados.0.smithi076.stdout:3537: done (4 left) 2021-03-15T22:34:51.379 INFO:tasks.rados.rados.0.smithi076.stdout:3543: write oid 34 current snap is 418 2021-03-15T22:34:51.380 INFO:tasks.rados.rados.0.smithi076.stdout:3543: seq_num 1208 ranges {702856=484566,1789542=513314} 2021-03-15T22:34:51.384 INFO:tasks.rados.rados.0.smithi076.stdout:3543: writing smithi07694878-34 from 702856 to 1187422 tid 1 2021-03-15T22:34:51.390 INFO:tasks.rados.rados.0.smithi076.stdout:3543: writing smithi07694878-34 from 1789542 to 2302856 tid 2 2021-03-15T22:34:51.390 INFO:tasks.rados.rados.0.smithi076.stdout:3544: snap_remove snap 409 2021-03-15T22:34:51.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:51.423 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:51.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:51.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:51.526 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:51.548 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:51.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:51.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:51.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:51 smithi076 conmon[56514]: audit 2021-03-15T22:34:50.266993+0000 mgr.y (mgr.14144) 1046 2021-03-15T22:34:51.641 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:51 smithi076 conmon[56514]: : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:51.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:51 smithi076 conmon[62156]: audit 2021-03-15T22:34:50.266993 2021-03-15T22:34:51.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:51 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 1046 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:51.650 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:51.671 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:51.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:51.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:51.728 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:34:51.728 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:34:51.729 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:34:51.729 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:34:51.729 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.729 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:5830 7.5 e631/559 7:ac84e656:::smithi07694878-31:head v 631'458, mlcod=630'454)", 2021-03-15T22:34:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:40.251563+0000", 2021-03-15T22:34:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 311.47260683000002, 2021-03-15T22:34:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071683063000000005, 2021-03-15T22:34:51.730 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.731 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:51.731 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.731 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.731 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.732 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251563+0000", 2021-03-15T22:34:51.732 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.732 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.733 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.733 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251563+0000", 2021-03-15T22:34:51.733 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052985690000000004 2021-03-15T22:34:51.734 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.734 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.734 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.734 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256862+0000", 2021-03-15T22:34:51.734 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2240000000000001e-06 2021-03-15T22:34:51.735 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.735 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.735 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.735 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256871+0000", 2021-03-15T22:34:51.736 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3723000000000002e-05 2021-03-15T22:34:51.736 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.736 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.736 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.737 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256905+0000", 2021-03-15T22:34:51.737 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2787e-05 2021-03-15T22:34:51.737 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.737 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.738 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.738 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.256918+0000", 2021-03-15T22:34:51.738 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6076000000000006e-05 2021-03-15T22:34:51.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.738 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.739 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.739 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257014+0000", 2021-03-15T22:34:51.739 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2570999999999997e-05 2021-03-15T22:34:51.739 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.740 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.740 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.740 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257046+0000", 2021-03-15T22:34:51.740 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066154983000000001 2021-03-15T22:34:51.741 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.741 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.741 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:51.741 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.323201+0000", 2021-03-15T22:34:51.742 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5130000000000003e-05 2021-03-15T22:34:51.742 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.742 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.742 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.743 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.323246+0000", 2021-03-15T22:34:51.743 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.071683063000000005 2021-03-15T22:34:51.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.743 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.743 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.744 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.744 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:755 7.5 e646/559 7:a9545d62:::smithi07694878-1:1fc v 646'483, mlcod=644'479)", 2021-03-15T22:34:51.744 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:55.246251+0000", 2021-03-15T22:34:51.745 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 296.47791884899999, 2021-03-15T22:34:51.745 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088387691000000004, 2021-03-15T22:34:51.745 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.745 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:51.746 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.746 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.746 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.746 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246251+0000", 2021-03-15T22:34:51.747 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.747 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.747 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.747 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.748 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246251+0000", 2021-03-15T22:34:51.748 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5550000000000001e-06 2021-03-15T22:34:51.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.748 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.749 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246254+0000", 2021-03-15T22:34:51.749 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3700000000000002e-06 2021-03-15T22:34:51.749 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.749 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.750 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.750 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246256+0000", 2021-03-15T22:34:51.750 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6309999999999998e-06 2021-03-15T22:34:51.750 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.751 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.751 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.751 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246266+0000", 2021-03-15T22:34:51.751 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6929999999999998e-06 2021-03-15T22:34:51.751 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.752 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.752 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.752 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246271+0000", 2021-03-15T22:34:51.752 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5197000000000001e-05 2021-03-15T22:34:51.753 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.753 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.753 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.753 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246316+0000", 2021-03-15T22:34:51.754 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3624e-05 2021-03-15T22:34:51.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.754 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.754 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.755 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246369+0000", 2021-03-15T22:34:51.755 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088224576999999998 2021-03-15T22:34:51.755 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.755 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.756 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:51.756 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334594+0000", 2021-03-15T22:34:51.756 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5043999999999998e-05 2021-03-15T22:34:51.756 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.757 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.757 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.757 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334639+0000", 2021-03-15T22:34:51.757 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088387691000000004 2021-03-15T22:34:51.757 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.758 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.758 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.758 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.758 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.759 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6453 7.9 7:91d654eb:::smithi07694878-5:head [write 739739~573968,stat] snapc 22b=[22b,229,228,227,226,225,223,221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1a1,17a,176] ondisk+write+known_if_redirected e691)", 2021-03-15T22:34:51.759 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.223418+0000", 2021-03-15T22:34:51.759 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 251.50075192599999, 2021-03-15T22:34:51.759 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15525217999999999, 2021-03-15T22:34:51.760 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.760 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:51.760 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.760 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.761 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.761 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6453 2021-03-15T22:34:51.761 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.761 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.762 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.762 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.762 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223418+0000", 2021-03-15T22:34:51.762 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.763 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.763 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.763 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.763 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223418+0000", 2021-03-15T22:34:51.763 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012317799999999999 2021-03-15T22:34:51.764 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.764 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.764 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.764 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223541+0000", 2021-03-15T22:34:51.765 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0259999999999998e-06 2021-03-15T22:34:51.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.765 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.766 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223547+0000", 2021-03-15T22:34:51.766 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9865999999999999e-05 2021-03-15T22:34:51.766 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.766 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.766 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.767 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223566+0000", 2021-03-15T22:34:51.767 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4069999999999993e-06 2021-03-15T22:34:51.767 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.767 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.768 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.768 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223575+0000", 2021-03-15T22:34:51.768 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0875000000000003e-05 2021-03-15T22:34:51.768 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.769 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.769 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.769 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223626+0000", 2021-03-15T22:34:51.769 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7320000000000003e-06 2021-03-15T22:34:51.770 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.770 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.770 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:34:51.770 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.223631+0000", 2021-03-15T22:34:51.771 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022595760000000001 2021-03-15T22:34:51.771 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.771 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.771 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.771 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.225891+0000", 2021-03-15T22:34:51.772 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000112502 2021-03-15T22:34:51.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.772 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.772 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.226003+0000", 2021-03-15T22:34:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000190173 2021-03-15T22:34:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.773 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,4", 2021-03-15T22:34:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.226194+0000", 2021-03-15T22:34:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.088675127000000006 2021-03-15T22:34:51.774 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.314869+0000", 2021-03-15T22:34:51.775 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.045894036999999999 2021-03-15T22:34:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.360763+0000", 2021-03-15T22:34:51.776 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.017758818999999999 2021-03-15T22:34:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:51.777 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378522+0000", 2021-03-15T22:34:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9515999999999998e-05 2021-03-15T22:34:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.778 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378571+0000", 2021-03-15T22:34:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9345999999999994e-05 2021-03-15T22:34:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.779 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378670+0000", 2021-03-15T22:34:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15525217999999999 2021-03-15T22:34:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.780 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.781 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6454 7.9 7:91d654eb:::smithi07694878-5:head [write 2053414~686325] snapc 22b=[22b,229,228,227,226,225,223,221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1a1,17a,176] ondisk+write+known_if_redirected e691)", 2021-03-15T22:34:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.230948+0000", 2021-03-15T22:34:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 251.49322192599999, 2021-03-15T22:34:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15108200699999999, 2021-03-15T22:34:51.782 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.783 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6454 2021-03-15T22:34:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.784 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.785 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.230948+0000", 2021-03-15T22:34:51.785 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.785 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.785 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.786 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.786 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.230948+0000", 2021-03-15T22:34:51.786 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018821300000000001 2021-03-15T22:34:51.786 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.786 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.787 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.787 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.231137+0000", 2021-03-15T22:34:51.787 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5119999999999999e-06 2021-03-15T22:34:51.787 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.788 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.788 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.788 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.231142+0000", 2021-03-15T22:34:51.788 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066237401000000001 2021-03-15T22:34:51.789 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.789 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.789 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.789 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297379+0000", 2021-03-15T22:34:51.790 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2347999999999999e-05 2021-03-15T22:34:51.790 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.790 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.790 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.791 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297392+0000", 2021-03-15T22:34:51.791 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0019934380000000002 2021-03-15T22:34:51.791 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.791 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.792 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.792 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299385+0000", 2021-03-15T22:34:51.792 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108266 2021-03-15T22:34:51.792 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.793 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.793 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:51.793 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299493+0000", 2021-03-15T22:34:51.793 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079232722000000005 2021-03-15T22:34:51.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.794 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.794 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.795 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378726+0000", 2021-03-15T22:34:51.795 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00011364799999999999 2021-03-15T22:34:51.795 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.795 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.795 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.796 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378840+0000", 2021-03-15T22:34:51.796 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021629 2021-03-15T22:34:51.796 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.797 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.797 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,4", 2021-03-15T22:34:51.797 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.379056+0000", 2021-03-15T22:34:51.797 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021122210000000001 2021-03-15T22:34:51.798 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.798 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:51.798 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381168+0000", 2021-03-15T22:34:51.799 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0516000000000001e-05 2021-03-15T22:34:51.799 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.799 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.800 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:51.800 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381189+0000", 2021-03-15T22:34:51.800 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071458899999999996 2021-03-15T22:34:51.800 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.801 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:51.801 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381903+0000", 2021-03-15T22:34:51.801 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6925e-05 2021-03-15T22:34:51.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.802 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:51.802 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381970+0000", 2021-03-15T22:34:51.803 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9917999999999998e-05 2021-03-15T22:34:51.803 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.803 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.803 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.803 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382030+0000", 2021-03-15T22:34:51.804 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.15108200699999999 2021-03-15T22:34:51.804 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.804 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.805 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.805 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6455 7.9 7:91d654eb:::smithi07694878-5:head [setxattr _header (57) in=7b,truncate 2739739] snapc 22b=[22b,229,228,227,226,225,223,221,220,21f,21c,219,218,216,215,211,20c,200,1fb,1f7,1e9,1e6,1e4,1e0,1dc,1d5,1c7,1a1,17a,176] ondisk+write+known_if_redirected e691)", 2021-03-15T22:34:51.806 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.297411+0000", 2021-03-15T22:34:51.806 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 251.42675878, 2021-03-15T22:34:51.806 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.084805948000000006, 2021-03-15T22:34:51.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.807 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:51.807 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.807 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.807 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.807 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6455 2021-03-15T22:34:51.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.808 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.808 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.809 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297411+0000", 2021-03-15T22:34:51.809 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.809 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.809 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.810 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.810 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297411+0000", 2021-03-15T22:34:51.810 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8290000000000001e-06 2021-03-15T22:34:51.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.811 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.811 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.811 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297415+0000", 2021-03-15T22:34:51.811 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6919999999999998e-06 2021-03-15T22:34:51.812 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.812 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.812 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297418+0000", 2021-03-15T22:34:51.813 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5140000000000007e-06 2021-03-15T22:34:51.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.813 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.813 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.814 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297427+0000", 2021-03-15T22:34:51.814 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8450000000000002e-06 2021-03-15T22:34:51.814 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.814 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.815 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.815 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297432+0000", 2021-03-15T22:34:51.815 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020918960000000002 2021-03-15T22:34:51.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.816 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.816 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299524+0000", 2021-03-15T22:34:51.816 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3504000000000006e-05 2021-03-15T22:34:51.816 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.817 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.817 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:51.817 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299608+0000", 2021-03-15T22:34:51.817 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080542796999999999 2021-03-15T22:34:51.818 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.818 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.818 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.380151+0000", 2021-03-15T22:34:51.819 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4352999999999997e-05 2021-03-15T22:34:51.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.819 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.820 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.380235+0000", 2021-03-15T22:34:51.820 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013269599999999999 2021-03-15T22:34:51.820 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.820 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.820 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 1,4", 2021-03-15T22:34:51.821 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.380368+0000", 2021-03-15T22:34:51.821 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00084666400000000001 2021-03-15T22:34:51.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.822 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:51.822 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381214+0000", 2021-03-15T22:34:51.822 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6737000000000003e-05 2021-03-15T22:34:51.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.823 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.823 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:51.823 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.381261+0000", 2021-03-15T22:34:51.823 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00082662899999999995 2021-03-15T22:34:51.823 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.824 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:51.824 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382088+0000", 2021-03-15T22:34:51.824 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8632999999999999e-05 2021-03-15T22:34:51.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.825 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:51.825 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382146+0000", 2021-03-15T22:34:51.826 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1159000000000001e-05 2021-03-15T22:34:51.826 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.826 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.826 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.827 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382217+0000", 2021-03-15T22:34:51.827 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.084805948000000006 2021-03-15T22:34:51.827 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.827 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.827 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.828 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6456 7.9 7:91d654eb:::smithi07694878-5:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e691)", 2021-03-15T22:34:51.828 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.297438+0000", 2021-03-15T22:34:51.829 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 251.42673237299999, 2021-03-15T22:34:51.829 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.085000942999999995, 2021-03-15T22:34:51.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.829 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:51.830 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.830 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.830 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.830 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6456 2021-03-15T22:34:51.831 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.831 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.831 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.831 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.832 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297438+0000", 2021-03-15T22:34:51.832 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.833 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.833 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297438+0000", 2021-03-15T22:34:51.833 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.012e-06 2021-03-15T22:34:51.833 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.834 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.834 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.834 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297439+0000", 2021-03-15T22:34:51.834 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8310000000000003e-06 2021-03-15T22:34:51.835 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.835 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.835 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297446+0000", 2021-03-15T22:34:51.836 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7070000000000003e-06 2021-03-15T22:34:51.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.836 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.837 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297452+0000", 2021-03-15T22:34:51.837 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0090000000000001e-06 2021-03-15T22:34:51.837 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.837 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.837 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.838 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297456+0000", 2021-03-15T22:34:51.838 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002183955 2021-03-15T22:34:51.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.839 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.839 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299640+0000", 2021-03-15T22:34:51.839 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3368999999999993e-05 2021-03-15T22:34:51.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.840 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.840 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:51.840 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.299734+0000", 2021-03-15T22:34:51.840 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080901084999999998 2021-03-15T22:34:51.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.841 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.841 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.380635+0000", 2021-03-15T22:34:51.841 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1402000000000003e-05 2021-03-15T22:34:51.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.842 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:51.842 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.380726+0000", 2021-03-15T22:34:51.843 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0015328080000000001 2021-03-15T22:34:51.843 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.843 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.843 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.844 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382259+0000", 2021-03-15T22:34:51.844 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9952999999999997e-05 2021-03-15T22:34:51.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.845 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.845 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382349+0000", 2021-03-15T22:34:51.845 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9812e-05 2021-03-15T22:34:51.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.846 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.846 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.846 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.382439+0000", 2021-03-15T22:34:51.846 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.085000942999999995 2021-03-15T22:34:51.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.847 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.847 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.847 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.848 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6965 7.7 7:ed4a01d2:::smithi07694878-2:head [sparse-read 0~3616366,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e740)", 2021-03-15T22:34:51.848 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.977912+0000", 2021-03-15T22:34:51.848 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 200.74625839999999, 2021-03-15T22:34:51.848 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27620877300000002, 2021-03-15T22:34:51.849 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.849 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:51.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.849 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.850 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.850 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6965 2021-03-15T22:34:51.850 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.850 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.850 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.851 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.851 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977912+0000", 2021-03-15T22:34:51.851 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.851 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.852 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.852 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977912+0000", 2021-03-15T22:34:51.852 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3100000000000002e-06 2021-03-15T22:34:51.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.853 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.853 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977916+0000", 2021-03-15T22:34:51.853 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1709999999999998e-06 2021-03-15T22:34:51.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.854 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.854 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.854 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977919+0000", 2021-03-15T22:34:51.855 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.258e-05 2021-03-15T22:34:51.855 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.855 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.856 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977932+0000", 2021-03-15T22:34:51.856 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4590000000000004e-06 2021-03-15T22:34:51.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.856 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.857 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977941+0000", 2021-03-15T22:34:51.857 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0978e-05 2021-03-15T22:34:51.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.857 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.858 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.858 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977982+0000", 2021-03-15T22:34:51.858 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2990000000000002e-06 2021-03-15T22:34:51.858 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.859 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:34:51.859 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977989+0000", 2021-03-15T22:34:51.859 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27114430499999997 2021-03-15T22:34:51.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.860 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.860 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.860 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249133+0000", 2021-03-15T22:34:51.860 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000114636 2021-03-15T22:34:51.861 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.861 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.861 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249248+0000", 2021-03-15T22:34:51.862 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0048730350000000004 2021-03-15T22:34:51.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.862 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.863 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254121+0000", 2021-03-15T22:34:51.863 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27620877300000002 2021-03-15T22:34:51.863 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.863 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.864 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.864 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.864 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.864 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6966 7.7 7:ed4a01d2:::smithi07694878-2:head [sparse-read 0~3616366] snapc 0=[] ondisk+read+known_if_redirected e740)", 2021-03-15T22:34:51.864 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.977955+0000", 2021-03-15T22:34:51.865 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 200.746215702, 2021-03-15T22:34:51.865 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27650659300000002, 2021-03-15T22:34:51.865 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.866 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:51.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.866 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.867 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6966 2021-03-15T22:34:51.867 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.867 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.867 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.868 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.868 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977955+0000", 2021-03-15T22:34:51.868 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.868 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977955+0000", 2021-03-15T22:34:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3510000000000001e-06 2021-03-15T22:34:51.869 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977956+0000", 2021-03-15T22:34:51.870 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4020000000000001e-06 2021-03-15T22:34:51.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.871 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.871 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.871 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977957+0000", 2021-03-15T22:34:51.872 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1184e-05 2021-03-15T22:34:51.872 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.872 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977968+0000", 2021-03-15T22:34:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4050000000000002e-06 2021-03-15T22:34:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.873 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977973+0000", 2021-03-15T22:34:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8658000000000002e-05 2021-03-15T22:34:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.874 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.875 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.875 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.978012+0000", 2021-03-15T22:34:51.875 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8309999999999997e-06 2021-03-15T22:34:51.875 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.876 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:34:51.876 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.978017+0000", 2021-03-15T22:34:51.876 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27614046599999997 2021-03-15T22:34:51.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.877 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.877 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.877 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254158+0000", 2021-03-15T22:34:51.878 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010246700000000001 2021-03-15T22:34:51.878 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.878 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.879 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254260+0000", 2021-03-15T22:34:51.879 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000200829 2021-03-15T22:34:51.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.880 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.880 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254461+0000", 2021-03-15T22:34:51.880 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27650659300000002 2021-03-15T22:34:51.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.881 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.881 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.881 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.881 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.882 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6967 7.7 7:ed4a01d2:::smithi07694878-2:head [sparse-read 0~3616366] snapc 0=[] ondisk+read+known_if_redirected e740)", 2021-03-15T22:34:51.882 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.977987+0000", 2021-03-15T22:34:51.882 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 200.746183027, 2021-03-15T22:34:51.882 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27677133100000001, 2021-03-15T22:34:51.883 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.883 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:51.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.883 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.884 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6967 2021-03-15T22:34:51.884 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.884 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.884 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.885 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.885 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977987+0000", 2021-03-15T22:34:51.885 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.885 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.886 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.886 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977987+0000", 2021-03-15T22:34:51.886 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.184e-06 2021-03-15T22:34:51.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.887 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.887 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977988+0000", 2021-03-15T22:34:51.888 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4130000000000001e-06 2021-03-15T22:34:51.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.888 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.888 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.889 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977990+0000", 2021-03-15T22:34:51.889 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9290000000000003e-06 2021-03-15T22:34:51.889 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.890 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.890 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.977996+0000", 2021-03-15T22:34:51.890 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2189999999999998e-06 2021-03-15T22:34:51.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.891 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.891 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.978000+0000", 2021-03-15T22:34:51.891 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.0028999999999999e-05 2021-03-15T22:34:51.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.892 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.892 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.978040+0000", 2021-03-15T22:34:51.892 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4139999999999998e-06 2021-03-15T22:34:51.892 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.893 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:34:51.893 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.978043+0000", 2021-03-15T22:34:51.894 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27649842800000002 2021-03-15T22:34:51.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.894 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.894 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.895 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254542+0000", 2021-03-15T22:34:51.895 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5296000000000006e-05 2021-03-15T22:34:51.895 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.895 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.896 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254617+0000", 2021-03-15T22:34:51.896 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000141419 2021-03-15T22:34:51.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.897 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.897 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254759+0000", 2021-03-15T22:34:51.897 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27677133100000001 2021-03-15T22:34:51.897 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.898 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7260 7.f 7:fc7cd984:::smithi07694878-4:head [delete] snapc 27b=[27b,276,275,271,26f,26e,26c,268,267,266,265,261,25d,25b,257,255,24f,24c,24b,24a,248,247,243,23e,23d,239,234,22d,218,1f7,1c7] ondisk+write+peerstat_old+known_if_redirected e772)", 2021-03-15T22:34:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:02.265878+0000", 2021-03-15T22:34:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 169.458292005, 2021-03-15T22:34:51.899 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069591505999999997, 2021-03-15T22:34:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.900 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7260 2021-03-15T22:34:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.901 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.265878+0000", 2021-03-15T22:34:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.902 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.265878+0000", 2021-03-15T22:34:51.903 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.189e-06 2021-03-15T22:34:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.265882+0000", 2021-03-15T22:34:51.904 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4120000000000001e-06 2021-03-15T22:34:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.905 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.265886+0000", 2021-03-15T22:34:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.058741452999999999 2021-03-15T22:34:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.324627+0000", 2021-03-15T22:34:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2182e-05 2021-03-15T22:34:51.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.324639+0000", 2021-03-15T22:34:51.908 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032676519999999998 2021-03-15T22:34:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.909 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.327907+0000", 2021-03-15T22:34:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000105665 2021-03-15T22:34:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328013+0000", 2021-03-15T22:34:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000235035 2021-03-15T22:34:51.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:34:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328248+0000", 2021-03-15T22:34:51.912 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00205624 2021-03-15T22:34:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:51.913 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.330304+0000", 2021-03-15T22:34:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5997000000000001e-05 2021-03-15T22:34:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.914 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.330340+0000", 2021-03-15T22:34:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0050271780000000002 2021-03-15T22:34:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.915 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.335367+0000", 2021-03-15T22:34:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4549999999999998e-05 2021-03-15T22:34:51.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.335422+0000", 2021-03-15T22:34:51.917 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7953e-05 2021-03-15T22:34:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.918 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.335470+0000", 2021-03-15T22:34:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.069591505999999997 2021-03-15T22:34:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.919 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7701 7.4 7:26337d05:::smithi07694878-20:2a2 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e816)", 2021-03-15T22:34:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:46.291166+0000", 2021-03-15T22:34:51.920 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 125.43300383, 2021-03-15T22:34:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056734093999999999, 2021-03-15T22:34:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:34:51.921 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7701 2021-03-15T22:34:51.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.923 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291166+0000", 2021-03-15T22:34:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.924 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291166+0000", 2021-03-15T22:34:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4200000000000004e-07 2021-03-15T22:34:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.925 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291167+0000", 2021-03-15T22:34:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1009999999999999e-06 2021-03-15T22:34:51.926 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291168+0000", 2021-03-15T22:34:51.927 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.553e-06 2021-03-15T22:34:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.928 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291174+0000", 2021-03-15T22:34:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8510000000000001e-06 2021-03-15T22:34:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.929 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291178+0000", 2021-03-15T22:34:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0061000000000005e-05 2021-03-15T22:34:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.930 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291258+0000", 2021-03-15T22:34:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1459999999999998e-06 2021-03-15T22:34:51.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:34:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291262+0000", 2021-03-15T22:34:51.932 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056538214000000003 2021-03-15T22:34:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347800+0000", 2021-03-15T22:34:51.933 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000100226 2021-03-15T22:34:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.934 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347901+0000", 2021-03-15T22:34:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056734093999999999 2021-03-15T22:34:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.935 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.936 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7702 7.4 7:26337d05:::smithi07694878-20:2a2 [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e816)", 2021-03-15T22:34:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:34:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 125.43298683800001, 2021-03-15T22:34:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056836219, 2021-03-15T22:34:51.937 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:34:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.938 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7702 2021-03-15T22:34:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.939 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:34:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:34:51.941 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6599999999999994e-07 2021-03-15T22:34:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.942 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291184+0000", 2021-03-15T22:34:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.249e-06 2021-03-15T22:34:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.943 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291186+0000", 2021-03-15T22:34:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5099999999999998e-06 2021-03-15T22:34:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.944 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291191+0000", 2021-03-15T22:34:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7519999999999998e-06 2021-03-15T22:34:51.945 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291195+0000", 2021-03-15T22:34:51.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0414999999999995e-05 2021-03-15T22:34:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291275+0000", 2021-03-15T22:34:51.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2119999999999999e-06 2021-03-15T22:34:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:34:51.948 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291279+0000", 2021-03-15T22:34:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056643645999999999 2021-03-15T22:34:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347922+0000", 2021-03-15T22:34:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7468999999999998e-05 2021-03-15T22:34:51.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.348020+0000", 2021-03-15T22:34:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056836219 2021-03-15T22:34:51.951 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7877 7.f 7:fc7cd984:::smithi07694878-4:head [copy-from ver 1977] snapc 2b8=[2b8,2b7,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e834)", 2021-03-15T22:34:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:04.322675+0000", 2021-03-15T22:34:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 107.40149522500001, 2021-03-15T22:34:51.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0491417890000001, 2021-03-15T22:34:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.954 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7877 2021-03-15T22:34:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.955 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.322675+0000", 2021-03-15T22:34:51.956 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.322675+0000", 2021-03-15T22:34:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0920000000000002e-06 2021-03-15T22:34:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.958 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.322678+0000", 2021-03-15T22:34:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.898e-06 2021-03-15T22:34:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.322681+0000", 2021-03-15T22:34:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001547157 2021-03-15T22:34:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324228+0000", 2021-03-15T22:34:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0239e-05 2021-03-15T22:34:51.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324238+0000", 2021-03-15T22:34:51.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010807830000000001 2021-03-15T22:34:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.325319+0000", 2021-03-15T22:34:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012095 2021-03-15T22:34:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.325440+0000", 2021-03-15T22:34:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0357033609999999 2021-03-15T22:34:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.965 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 3,7", 2021-03-15T22:34:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.361143+0000", 2021-03-15T22:34:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.010042997 2021-03-15T22:34:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.966 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:34:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371186+0000", 2021-03-15T22:34:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018398700000000001 2021-03-15T22:34:51.967 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371370+0000", 2021-03-15T22:34:51.968 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028334399999999999 2021-03-15T22:34:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:34:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371654+0000", 2021-03-15T22:34:51.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6285999999999993e-05 2021-03-15T22:34:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:51.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371740+0000", 2021-03-15T22:34:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6694999999999997e-05 2021-03-15T22:34:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.971 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371817+0000", 2021-03-15T22:34:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0491417890000001 2021-03-15T22:34:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.972 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.973 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7878 7.f 7:fc7cd984:::smithi07694878-4:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e834)", 2021-03-15T22:34:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:04.324891+0000", 2021-03-15T22:34:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 107.399279108, 2021-03-15T22:34:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0471670769999999, 2021-03-15T22:34:51.974 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:34:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:34:51.975 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:34:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7878 2021-03-15T22:34:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324891+0000", 2021-03-15T22:34:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324891+0000", 2021-03-15T22:34:51.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.869e-06 2021-03-15T22:34:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324893+0000", 2021-03-15T22:34:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5579999999999999e-06 2021-03-15T22:34:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324896+0000", 2021-03-15T22:34:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6606e-05 2021-03-15T22:34:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324952+0000", 2021-03-15T22:34:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1832999999999997e-05 2021-03-15T22:34:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.324994+0000", 2021-03-15T22:34:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00094264299999999995 2021-03-15T22:34:51.983 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.325937+0000", 2021-03-15T22:34:51.984 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9849000000000002e-05 2021-03-15T22:34:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:34:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.325996+0000", 2021-03-15T22:34:51.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.041140036 2021-03-15T22:34:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.986 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.367137+0000", 2021-03-15T22:34:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000177877 2021-03-15T22:34:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:34:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.367314+0000", 2021-03-15T22:34:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0045396300000000002 2021-03-15T22:34:51.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371854+0000", 2021-03-15T22:34:51.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010181 2021-03-15T22:34:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:51.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.371956+0000", 2021-03-15T22:34:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000102366 2021-03-15T22:34:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.372058+0000", 2021-03-15T22:34:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0471670769999999 2021-03-15T22:34:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.992 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.993 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(osd.0.8:76 7.7 7:ed4a01d2:::smithi07694878-2:head [assert-version v1977,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected e834)", 2021-03-15T22:34:51.994 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:04.327331+0000", 2021-03-15T22:34:51.994 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 107.39683915000001, 2021-03-15T22:34:51.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.03086123, 2021-03-15T22:34:51.994 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:51.995 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:34:51.995 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:51.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327331+0000", 2021-03-15T22:34:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327331+0000", 2021-03-15T22:34:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0818999999999998e-05 2021-03-15T22:34:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.997 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327352+0000", 2021-03-15T22:34:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1210000000000001e-06 2021-03-15T22:34:51.998 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327354+0000", 2021-03-15T22:34:51.999 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9869000000000002e-05 2021-03-15T22:34:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:52.000 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327384+0000", 2021-03-15T22:34:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1650000000000003e-06 2021-03-15T22:34:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:52.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327390+0000", 2021-03-15T22:34:52.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000230375 2021-03-15T22:34:52.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:52.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327620+0000", 2021-03-15T22:34:52.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6136e-05 2021-03-15T22:34:52.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:34:52.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:04.327667+0000", 2021-03-15T22:34:52.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.029658271 2021-03-15T22:34:52.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.005 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:52.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.357325+0000", 2021-03-15T22:34:52.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018968899999999999 2021-03-15T22:34:52.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:52.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.357515+0000", 2021-03-15T22:34:52.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067778499999999998 2021-03-15T22:34:52.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.007 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:52.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:05.358192+0000", 2021-03-15T22:34:52.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.03086123 2021-03-15T22:34:52.008 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.008 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:52.009 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.010 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGRecoveryDeleteReply(7.7 e835,834 7:ecc0cb20:::smithi07694878-25:cd,308'179)", 2021-03-15T22:34:52.010 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:06.265008+0000", 2021-03-15T22:34:52.010 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 105.459162179, 2021-03-15T22:34:52.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.108627554, 2021-03-15T22:34:52.011 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:52.011 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:34:52.011 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:52.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.012 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:52.012 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.265008+0000", 2021-03-15T22:34:52.012 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:52.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.013 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:52.013 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.265008+0000", 2021-03-15T22:34:52.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3300000000000001e-06 2021-03-15T22:34:52.014 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.014 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:52.014 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.265010+0000", 2021-03-15T22:34:52.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4719999999999997e-06 2021-03-15T22:34:52.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:52.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.265015+0000", 2021-03-15T22:34:52.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.10817592299999999 2021-03-15T22:34:52.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.016 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:52.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.373191+0000", 2021-03-15T22:34:52.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1566000000000001e-05 2021-03-15T22:34:52.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:52.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.373202+0000", 2021-03-15T22:34:52.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7701999999999997e-05 2021-03-15T22:34:52.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:52.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.373260+0000", 2021-03-15T22:34:52.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000375561 2021-03-15T22:34:52.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:52.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:06.373636+0000", 2021-03-15T22:34:52.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.108627554 2021-03-15T22:34:52.021 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.021 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:52.021 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.021 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.022 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:8473 7.5 e900/881 7:a9545d62:::smithi07694878-1:head v 900'833, mlcod=898'829)", 2021-03-15T22:34:52.022 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:15.513894+0000", 2021-03-15T22:34:52.022 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.210275805000002, 2021-03-15T22:34:52.023 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.099574201000000001, 2021-03-15T22:34:52.023 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:52.023 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:52.023 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:52.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:52.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.513894+0000", 2021-03-15T22:34:52.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:52.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.025 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.025 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:52.025 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.513894+0000", 2021-03-15T22:34:52.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0040718849999999999 2021-03-15T22:34:52.026 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.026 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:52.026 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.517966+0000", 2021-03-15T22:34:52.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2101000000000001e-05 2021-03-15T22:34:52.027 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.027 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:52.027 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.517988+0000", 2021-03-15T22:34:52.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8615999999999999e-05 2021-03-15T22:34:52.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.028 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:52.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.518007+0000", 2021-03-15T22:34:52.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0790000000000003e-06 2021-03-15T22:34:52.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:52.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.518014+0000", 2021-03-15T22:34:52.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4459e-05 2021-03-15T22:34:52.030 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:52.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.518089+0000", 2021-03-15T22:34:52.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8683e-05 2021-03-15T22:34:52.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:52.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.518117+0000", 2021-03-15T22:34:52.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095304312000000002 2021-03-15T22:34:52.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:52.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.613422+0000", 2021-03-15T22:34:52.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7066e-05 2021-03-15T22:34:52.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:52.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.613469+0000", 2021-03-15T22:34:52.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.099574201000000001 2021-03-15T22:34:52.035 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.035 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:52.035 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.036 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.4.0:714 7.1 e900/899 7:8bc62bc6:::smithi07694878-17:2e4 v 900'484, mlcod=900'480)", 2021-03-15T22:34:52.036 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:15.542430+0000", 2021-03-15T22:34:52.036 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 36.181740617000003, 2021-03-15T22:34:52.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.072600789999999998, 2021-03-15T22:34:52.037 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:52.037 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:52.037 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:52.038 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.038 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:52.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.542430+0000", 2021-03-15T22:34:52.039 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:52.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.039 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:52.040 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.542430+0000", 2021-03-15T22:34:52.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0699999999999998e-06 2021-03-15T22:34:52.040 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.040 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.040 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:52.041 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.542433+0000", 2021-03-15T22:34:52.041 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2490000000000001e-06 2021-03-15T22:34:52.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.041 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:52.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.542436+0000", 2021-03-15T22:34:52.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2114e-05 2021-03-15T22:34:52.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:52.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.542448+0000", 2021-03-15T22:34:52.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0098999999999999e-05 2021-03-15T22:34:52.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:52.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.542458+0000", 2021-03-15T22:34:52.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7843000000000001e-05 2021-03-15T22:34:52.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:52.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.542486+0000", 2021-03-15T22:34:52.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5135e-05 2021-03-15T22:34:52.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:52.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.542501+0000", 2021-03-15T22:34:52.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.072492398999999999 2021-03-15T22:34:52.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:52.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.614994+0000", 2021-03-15T22:34:52.048 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6881000000000001e-05 2021-03-15T22:34:52.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:52.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.615030+0000", 2021-03-15T22:34:52.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.072600789999999998 2021-03-15T22:34:52.049 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.050 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:52.050 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.051 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.4.0:723 7.1 e914/899 7:8bc62bc6:::smithi07694878-17:2e4 v 914'500, mlcod=912'496)", 2021-03-15T22:34:52.051 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:29.599759+0000", 2021-03-15T22:34:52.051 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.124410814000001, 2021-03-15T22:34:52.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078064415999999998, 2021-03-15T22:34:52.052 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:52.052 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:52.052 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:52.052 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.052 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:52.053 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.599759+0000", 2021-03-15T22:34:52.053 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:52.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.054 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:52.054 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.599759+0000", 2021-03-15T22:34:52.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1379999999999999e-06 2021-03-15T22:34:52.054 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.055 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.055 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:52.055 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.599763+0000", 2021-03-15T22:34:52.055 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6919999999999998e-06 2021-03-15T22:34:52.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:52.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.599765+0000", 2021-03-15T22:34:52.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073809586999999996 2021-03-15T22:34:52.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:52.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.673575+0000", 2021-03-15T22:34:52.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3869999999999997e-06 2021-03-15T22:34:52.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:52.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.673582+0000", 2021-03-15T22:34:52.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027220970000000001 2021-03-15T22:34:52.059 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:52.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.676304+0000", 2021-03-15T22:34:52.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3706e-05 2021-03-15T22:34:52.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:52.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.676328+0000", 2021-03-15T22:34:52.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.001474618 2021-03-15T22:34:52.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:52.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.677803+0000", 2021-03-15T22:34:52.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1191000000000001e-05 2021-03-15T22:34:52.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:52.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.677824+0000", 2021-03-15T22:34:52.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078064415999999998 2021-03-15T22:34:52.064 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.064 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:52.065 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.065 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.5.0:709 7.e e914/879 7:7ac381e0:::smithi07694878-24:2e8 v 914'427, mlcod=911'423)", 2021-03-15T22:34:52.066 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:29.606630+0000", 2021-03-15T22:34:52.066 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 22.117540201000001, 2021-03-15T22:34:52.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.072714138999999997, 2021-03-15T22:34:52.066 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:34:52.067 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:34:52.067 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:34:52.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.067 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:34:52.068 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.606630+0000", 2021-03-15T22:34:52.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:34:52.068 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.068 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.069 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:34:52.069 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.606630+0000", 2021-03-15T22:34:52.069 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8550000000000004e-06 2021-03-15T22:34:52.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:34:52.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.606634+0000", 2021-03-15T22:34:52.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.939e-06 2021-03-15T22:34:52.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:34:52.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.606636+0000", 2021-03-15T22:34:52.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066909977999999995 2021-03-15T22:34:52.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:34:52.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.673546+0000", 2021-03-15T22:34:52.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6950000000000006e-06 2021-03-15T22:34:52.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:34:52.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.673555+0000", 2021-03-15T22:34:52.074 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017501179999999999 2021-03-15T22:34:52.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:34:52.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.675305+0000", 2021-03-15T22:34:52.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3093000000000001e-05 2021-03-15T22:34:52.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.076 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:34:52.076 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.675328+0000", 2021-03-15T22:34:52.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0039880899999999997 2021-03-15T22:34:52.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:34:52.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.679316+0000", 2021-03-15T22:34:52.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8371e-05 2021-03-15T22:34:52.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:34:52.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:34:52.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:34:52.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.679344+0000", 2021-03-15T22:34:52.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.072714138999999997 2021-03-15T22:34:52.079 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.079 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:52.080 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.080 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:34:52.080 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:34:52.080 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:34:52.081 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:52.082 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:51 smithi125 conmon[56739]: audit 2021-03-15T22:34:50.266993 2021-03-15T22:34:52.083 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:51 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 1046 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:34:52.146 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:52.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:52.148 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:52.249 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:52.271 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:52.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:52.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:52.366 INFO:tasks.rados.rados.0.smithi076.stdout:3544: done (5 left) 2021-03-15T22:34:52.366 INFO:tasks.rados.rados.0.smithi076.stdout:3545: write oid 13 current snap is 418 2021-03-15T22:34:52.366 INFO:tasks.rados.rados.0.smithi076.stdout:3539: expect (ObjNum 1195 snap 417 seq_num 1195) 2021-03-15T22:34:52.373 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:52.399 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:52.400 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:52.440 INFO:tasks.rados.rados.0.smithi076.stdout:3541: expect (ObjNum 1203 snap 418 seq_num 1203) 2021-03-15T22:34:52.501 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:52.522 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:52.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:52.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:52.560 INFO:tasks.rados.rados.0.smithi076.stdout:3543: finishing write tid 1 to smithi07694878-34 2021-03-15T22:34:52.561 INFO:tasks.rados.rados.0.smithi076.stdout:3543: finishing write tid 2 to smithi07694878-34 2021-03-15T22:34:52.561 INFO:tasks.rados.rados.0.smithi076.stdout:3543: finishing write tid 3 to smithi07694878-34 2021-03-15T22:34:52.561 INFO:tasks.rados.rados.0.smithi076.stdout:3545: seq_num 1209 ranges {719663=676168,1964377=355286} 2021-03-15T22:34:52.568 INFO:tasks.rados.rados.0.smithi076.stdout:3545: writing smithi07694878-13 from 719663 to 1395831 tid 1 2021-03-15T22:34:52.573 INFO:tasks.rados.rados.0.smithi076.stdout:3545: writing smithi07694878-13 from 1964377 to 2319663 tid 2 2021-03-15T22:34:52.574 INFO:tasks.rados.rados.0.smithi076.stdout:3539: done (5 left) 2021-03-15T22:34:52.574 INFO:tasks.rados.rados.0.smithi076.stdout:3541: done (4 left) 2021-03-15T22:34:52.574 INFO:tasks.rados.rados.0.smithi076.stdout:3546: delete oid 12 current snap is 418 2021-03-15T22:34:52.575 INFO:tasks.rados.rados.0.smithi076.stdout:3543: finishing write tid 4 to smithi07694878-34 2021-03-15T22:34:52.575 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 2432 (ObjNum 1208 snap 418 seq_num 1208) dirty exists 2021-03-15T22:34:52.575 INFO:tasks.rados.rados.0.smithi076.stdout:3543: left oid 34 (ObjNum 1208 snap 418 seq_num 1208) 2021-03-15T22:34:52.575 INFO:tasks.rados.rados.0.smithi076.stdout:3542: finishing write tid 1 to smithi07694878-3 2021-03-15T22:34:52.575 INFO:tasks.rados.rados.0.smithi076.stdout:3542: finishing write tid 2 to smithi07694878-3 2021-03-15T22:34:52.576 INFO:tasks.rados.rados.0.smithi076.stdout:3542: finishing write tid 3 to smithi07694878-3 2021-03-15T22:34:52.576 INFO:tasks.rados.rados.0.smithi076.stdout:3542: finishing write tid 4 to smithi07694878-3 2021-03-15T22:34:52.576 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 2683 (ObjNum 1207 snap 418 seq_num 1207) dirty exists 2021-03-15T22:34:52.576 INFO:tasks.rados.rados.0.smithi076.stdout:3542: left oid 3 (ObjNum 1207 snap 418 seq_num 1207) 2021-03-15T22:34:52.577 INFO:tasks.rados.rados.0.smithi076.stdout:3540: finishing rollback tid 0 to smithi07694878-30 2021-03-15T22:34:52.577 INFO:tasks.rados.rados.0.smithi076.stdout:3540: finishing rollback tid 1 to smithi07694878-30 2021-03-15T22:34:52.577 INFO:tasks.rados.rados.0.smithi076.stdout:3540: finishing rollback tid 2 to smithi07694878-30 2021-03-15T22:34:52.577 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 2893 (ObjNum 774 snap 256 seq_num 774) dirty exists 2021-03-15T22:34:52.577 INFO:tasks.rados.rados.0.smithi076.stdout:3545: finishing write tid 1 to smithi07694878-13 2021-03-15T22:34:52.579 INFO:tasks.rados.rados.0.smithi076.stdout:3545: finishing write tid 2 to smithi07694878-13 2021-03-15T22:34:52.579 INFO:tasks.rados.rados.0.smithi076.stdout:3545: finishing write tid 3 to smithi07694878-13 2021-03-15T22:34:52.579 INFO:tasks.rados.rados.0.smithi076.stdout:3540: done (4 left) 2021-03-15T22:34:52.579 INFO:tasks.rados.rados.0.smithi076.stdout:3542: done (3 left) 2021-03-15T22:34:52.579 INFO:tasks.rados.rados.0.smithi076.stdout:3543: done (2 left) 2021-03-15T22:34:52.580 INFO:tasks.rados.rados.0.smithi076.stdout:3546: done (1 left) 2021-03-15T22:34:52.580 INFO:tasks.rados.rados.0.smithi076.stdout:3547: snap_create 2021-03-15T22:34:52.580 INFO:tasks.rados.rados.0.smithi076.stdout:3545: finishing write tid 4 to smithi07694878-13 2021-03-15T22:34:52.580 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 2014 (ObjNum 1209 snap 418 seq_num 1209) dirty exists 2021-03-15T22:34:52.581 INFO:tasks.rados.rados.0.smithi076.stdout:3545: left oid 13 (ObjNum 1209 snap 418 seq_num 1209) 2021-03-15T22:34:52.624 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:52.648 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:52.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:52.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:52.716 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:52 smithi076 conmon[56514]: cluster 2021-03-15T22:34:51.210869+0000 mgr.y (mgr.14144) 1047 : cluster [DBG] pgmap v1550: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 661 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 4/1914 objects degraded (0.209%); 11/1914 objects misplaced (0.575%); 5.6 MiB/s, 19 objects/s recovering 2021-03-15T22:34:52.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:52 smithi076 conmon[56514]: cluster 2021-03-15T22:34:51.350049+0000 mon.a (mon.0) 2352 : cluster [WRN] Health check failed: Degraded data redundancy: 4/1914 objects degraded (0.209%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:52.717 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:52 smithi076 conmon[56514]: cluster 2021-03-15T22:34:51.361333+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2021-03-15T22:34:52.717 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:52 smithi076 conmon[62156]: cluster 2021-03-15T22:34:51.210869+0000 mgr.y (mgr.14144) 1047 : cluster [DBG] pgmap v1550: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 661 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 4/1914 objects degraded (0.209%); 11/1914 objects misplaced (0.575%); 5.6 MiB/s, 19 objects/s recovering 2021-03-15T22:34:52.718 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:52 smithi076 conmon[62156]: cluster 2021-03-15T22:34:51.350049+0000 mon.a (mon.0) 2352 : cluster [WRN] Health check failed: Degraded data redundancy: 4/1914 objects degraded (0.209%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:52.718 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:52 smithi076 conmon[62156]: cluster 2021-03-15T22:34:51.361333+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2021-03-15T22:34:52.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:52.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:34:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:52.774 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:52.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:52.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:52 smithi125 conmon[56739]: cluster 2021-03-15T22:34:51.210869+0000 mgr.y (mgr.14144) 1047 : cluster [DBG] pgmap v1550: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 661 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 3.2 MiB/s rd, 2.3 MiB/s wr, 15 op/s; 4/1914 objects degraded (0.209%); 11/1914 objects misplaced (0.575%); 5.6 MiB/s, 19 objects/s recovering 2021-03-15T22:34:52.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:52 smithi125 conmon[56739]: 2021-03-15T22:34:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:52 smithi125 conmon[56739]: cluster 2021-03-15T22:34:51.350049+0000 mon.a (mon.0) 2352 : cluster [WRN] Health check failed: Degraded data redundancy: 4/1914 objects degraded (0.209%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:34:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:52 smithi125 conmon[56739]: cluster 2021-03-15T22:34: 2021-03-15T22:34:52.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:52 smithi125 conmon[56739]: 51.361333+0000 mon.a (mon.0) 2353 : cluster [DBG] osdmap e936: 8 total, 8 up, 7 in 2021-03-15T22:34:52.875 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:52.901 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:52.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:52.902 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:53.003 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:53.029 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:53.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:53.030 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:53.131 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:53.153 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:53.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:53.155 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:53.255 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:53.265 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:53.280 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:53.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:53.281 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:53.316 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:34:53.334 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:34:53.356 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:34:53.377 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:34:53.377 INFO:tasks.rados.rados.0.smithi076.stdout:3545: done (1 left) 2021-03-15T22:34:53.378 INFO:tasks.rados.rados.0.smithi076.stdout:3547: done (0 left) 2021-03-15T22:34:53.378 INFO:tasks.rados.rados.0.smithi076.stdout:3548: delete oid 24 current snap is 419 2021-03-15T22:34:53.379 INFO:tasks.rados.rados.0.smithi076.stdout:3548: done (0 left) 2021-03-15T22:34:53.380 INFO:tasks.rados.rados.0.smithi076.stdout:3549: read oid 17 snap -1 2021-03-15T22:34:53.380 INFO:tasks.rados.rados.0.smithi076.stdout:3549: expect (ObjNum 960 snap 321 seq_num 960) 2021-03-15T22:34:53.382 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:53.383 INFO:tasks.rados.rados.0.smithi076.stdout:3550: rollback oid 5 current snap is 419 2021-03-15T22:34:53.383 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 357 2021-03-15T22:34:53.383 INFO:tasks.rados.rados.0.smithi076.stdout:3551: read oid 27 snap -1 2021-03-15T22:34:53.383 INFO:tasks.rados.rados.0.smithi076.stdout:3551: expect (ObjNum 896 snap 302 seq_num 896) 2021-03-15T22:34:53.384 INFO:tasks.rados.rados.0.smithi076.stdout:3552: write oid 21 current snap is 419 2021-03-15T22:34:53.384 INFO:tasks.rados.rados.0.smithi076.stdout:3552: seq_num 1212 ranges {770084=501915,2023470=346614} 2021-03-15T22:34:53.387 INFO:tasks.rados.rados.0.smithi076.stdout:3552: writing smithi07694878-21 from 770084 to 1271999 tid 1 2021-03-15T22:34:53.394 INFO:tasks.rados.rados.0.smithi076.stdout:3552: writing smithi07694878-21 from 2023470 to 2370084 tid 2 2021-03-15T22:34:53.394 INFO:tasks.rados.rados.0.smithi076.stdout:3553: copy_from oid 23 from oid 34 current snap is 419 2021-03-15T22:34:53.394 INFO:tasks.rados.rados.0.smithi076.stdout:3550: finishing rollback tid 0 to smithi07694878-5 2021-03-15T22:34:53.394 INFO:tasks.rados.rados.0.smithi076.stdout:3550: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:34:53.395 INFO:tasks.rados.rados.0.smithi076.stdout:3550: finishing rollback tid 2 to smithi07694878-5 2021-03-15T22:34:53.395 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 2003 (ObjNum 1016 snap 341 seq_num 1016) dirty exists 2021-03-15T22:34:53.395 INFO:tasks.rados.rados.0.smithi076.stdout:3549: expect (ObjNum 960 snap 321 seq_num 960) 2021-03-15T22:34:53.403 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:34:53.410 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:53.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:53.411 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:53.422 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:34:53.443 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:34:53.461 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:34:53.462 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:34:53.463 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:34:53.463 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:34:53.463 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:34:53.463 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:34:53.463 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:34:53.464 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:34:53.464 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:34:53.497 INFO:tasks.rados.rados.0.smithi076.stdout:3551: expect (ObjNum 896 snap 302 seq_num 896) 2021-03-15T22:34:53.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:53.534 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:53.536 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:53.601 INFO:tasks.rados.rados.0.smithi076.stdout:3552: finishing write tid 1 to smithi07694878-21 2021-03-15T22:34:53.601 INFO:tasks.rados.rados.0.smithi076.stdout:3552: finishing write tid 2 to smithi07694878-21 2021-03-15T22:34:53.601 INFO:tasks.rados.rados.0.smithi076.stdout:3549: done (4 left) 2021-03-15T22:34:53.602 INFO:tasks.rados.rados.0.smithi076.stdout:3550: done (3 left) 2021-03-15T22:34:53.602 INFO:tasks.rados.rados.0.smithi076.stdout:3551: done (2 left) 2021-03-15T22:34:53.602 INFO:tasks.rados.rados.0.smithi076.stdout:3554: delete oid 1 current snap is 419 2021-03-15T22:34:53.602 INFO:tasks.rados.rados.0.smithi076.stdout:3552: finishing write tid 3 to smithi07694878-21 2021-03-15T22:34:53.603 INFO:tasks.rados.rados.0.smithi076.stdout:3552: finishing write tid 4 to smithi07694878-21 2021-03-15T22:34:53.603 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 3148 (ObjNum 1212 snap 419 seq_num 1212) dirty exists 2021-03-15T22:34:53.603 INFO:tasks.rados.rados.0.smithi076.stdout:3552: left oid 21 (ObjNum 1212 snap 419 seq_num 1212) 2021-03-15T22:34:53.603 INFO:tasks.rados.rados.0.smithi076.stdout:3553: finishing copy_from to smithi07694878-23 2021-03-15T22:34:53.603 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 2432 (ObjNum 1208 snap 418 seq_num 1208) dirty exists 2021-03-15T22:34:53.604 INFO:tasks.rados.rados.0.smithi076.stdout:3553: finishing copy_from racing read to smithi07694878-23 2021-03-15T22:34:53.604 INFO:tasks.rados.rados.0.smithi076.stdout:3552: done (2 left) 2021-03-15T22:34:53.605 INFO:tasks.rados.rados.0.smithi076.stdout:3553: done (1 left) 2021-03-15T22:34:53.605 INFO:tasks.rados.rados.0.smithi076.stdout:3554: done (0 left) 2021-03-15T22:34:53.605 INFO:tasks.rados.rados.0.smithi076.stdout:3555: copy_from oid 44 from oid 32 current snap is 419 2021-03-15T22:34:53.605 INFO:tasks.rados.rados.0.smithi076.stdout:3556: snap_create 2021-03-15T22:34:53.618 INFO:tasks.rados.rados.0.smithi076.stdout:3555: finishing copy_from to smithi07694878-44 2021-03-15T22:34:53.618 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 3038 (ObjNum 1177 snap 411 seq_num 1177) dirty exists 2021-03-15T22:34:53.618 INFO:tasks.rados.rados.0.smithi076.stdout:3555: finishing copy_from racing read to smithi07694878-44 2021-03-15T22:34:53.634 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:53 smithi076 conmon[56514]: cluster 2021-03-15T22:34:52.363933 2021-03-15T22:34:53.634 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:53 smithi076 conmon[56514]: +0000 mon.a (mon.0) 2354 : cluster [DBG] osdmap e937: 8 total, 8 up, 7 in 2021-03-15T22:34:53.634 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:53 smithi076 conmon[56514]: audit 2021-03-15T22:34:53.295645+0000 mon.c (mon.1) 75 : audit [DBG] from='client.? 172.21.15.76:0/3798773761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:53.635 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:53 smithi076 conmon[62156]: cluster 2021-03-15T22:34:52 2021-03-15T22:34:53.635 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:53 smithi076 conmon[62156]: .363933+0000 mon.a (mon.0) 2354 : cluster [DBG] osdmap e937: 8 total, 8 up, 7 in 2021-03-15T22:34:53.635 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:53 smithi076 conmon[62156]: audit 2021-03-15T22:34:53.295645+0000 mon.c (mon.1) 75 : audit [DBG] from='client.? 172.21.15.76:0/3798773761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:53.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:53.660 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:53.661 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:53.762 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:53.784 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:53.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:53.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:53.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:53 smithi125 conmon[56739]: cluster 2021-03-15T22:34:52.363933+0000 mon.a (mon.0) 2354 : cluster 2021-03-15T22:34:53.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:53 smithi125 conmon[56739]: [DBG] osdmap e937: 8 total, 8 up, 7 in 2021-03-15T22:34:53.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:53 smithi125 conmon[56739]: audit 2021-03-15T22:34:53.295645+0000 mon.c (mon.1) 75 : audit [DBG] from='client.? 172.21.15.76:0/3798773761' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:34:53.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:53.908 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:53.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:53.910 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:53.969 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:53 smithi076 conmon[56514]: debug 2021-03-15T22:34:53.830+0000 7f0537c88700 -1 mon.a@0(leader).osd e938 definitely_dead 0 2021-03-15T22:34:54.010 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:54.031 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:54.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:54.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:54.084 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:34:54.134 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:54.155 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:54.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:54.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:54.258 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:54.281 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:54.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:54.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:54.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:54 smithi076 conmon[62156]: cluster 2021-03-15T22:34:53.211353+0000 mgr.y (mgr.14144) 2021-03-15T22:34:54.373 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:54 smithi076 conmon[62156]: 1048 : cluster [DBG] pgmap v1553: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 662 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 4/1917 objects degraded (0.209%); 11/1917 objects misplaced (0.574%); 5.6 MiB/s, 14 objects/s recovering 2021-03-15T22:34:54.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:54 smithi076 conmon[62156]: cluster 2021-03-15T22:34:53.372381+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2021-03-15T22:34:54.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:54 smithi076 conmon[62156]: audit 2021-03-15T22:34:53.830696+0000 mon.c (mon.1) 76 : audit [DBG] from='client.? 172.21.15.76:0/3554232903' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2021-03-15T22:34:54.374 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:54 smithi076 conmon[62156]: audit 2021-03-15T22:34:53.831257+0000 mon.a (mon.0) 2356 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2021-03-15T22:34:54.384 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:54.387 INFO:tasks.rados.rados.0.smithi076.stdout:3555: done (1 left) 2021-03-15T22:34:54.388 INFO:tasks.rados.rados.0.smithi076.stdout:3556: done (0 left) 2021-03-15T22:34:54.388 INFO:tasks.rados.rados.0.smithi076.stdout:3557: delete oid 49 current snap is 420 2021-03-15T22:34:54.399 INFO:teuthology.orchestra.run.smithi076.stderr:marked in osd.2. 2021-03-15T22:34:54.407 INFO:tasks.rados.rados.0.smithi076.stdout:3557: done (0 left) 2021-03-15T22:34:54.408 INFO:tasks.rados.rados.0.smithi076.stdout:3558: snap_remove snap 405 2021-03-15T22:34:54.413 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:54.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:54.415 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:54.516 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:54.539 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:54.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:54.540 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:54.632 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:54 smithi076 conmon[56514]: cluster 2021-03-15T22:34:53. 2021-03-15T22:34:54.633 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:54 smithi076 conmon[56514]: 211353+0000 mgr.y (mgr.14144) 1048 : cluster [DBG] pgmap v1553: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 662 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 4/1917 objects degraded (0.209%); 11/1917 objects misplaced (0.574%); 5.6 MiB/s, 14 objects/s recovering 2021-03-15T22:34:54.633 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:54 smithi076 conmon[56514]: cluster 2021-03-15T22:34:53.372381+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2021-03-15T22:34:54.633 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:54 smithi076 conmon[56514]: audit 2021-03-15T22:34:53.830696+0000 mon.c (mon.1) 76 : audit [DBG] from='client.? 172.21.15.76:0/3554232903' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2021-03-15T22:34:54.633 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:54 smithi076 conmon[56514]: audit 2021-03-15T22:34:53.831257+0000 mon.a (mon.0) 2356 : audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2021-03-15T22:34:54.641 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:54.663 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:54.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:54.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:54.766 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:54.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:54 smithi125 conmon[56739]: cluster 2021-03-15T22:34:53.211353+0000 mgr.y (mgr.14144) 1048 : cluster 2021-03-15T22:34:54.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:54 smithi125 conmon[56739]: [DBG] pgmap v1553: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 157 active+clean; 662 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 8.0 MiB/s rd, 3.5 MiB/s wr, 26 op/s; 4/1917 objects degraded (0.209%); 11/1917 objects misplaced (0.574%); 5.6 MiB/s, 14 objects/s recovering 2021-03-15T22:34:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:54 smithi125 conmon[56739]: cluster 2021-03-15T22:34:53.372381+0000 mon.a (mon.0) 2355 : cluster [DBG] osdmap e938: 8 total, 8 up, 7 in 2021-03-15T22:34:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:54 smithi125 conmon[56739]: audit 2021-03-15T22:34 2021-03-15T22:34:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:54 smithi125 conmon[56739]: :53.830696+0000 mon.c (mon.1) 76 : audit [DBG] from='client.? 172.21.15.76:0/3554232903' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2021-03-15T22:34:54.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:54 smithi125 conmon[56739]: audit 2021-03-15T22:34: 2021-03-15T22:34:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:54 smithi125 conmon[56739]: 53.831257+0000 mon.a (mon.0) 2356 : 2021-03-15T22:34:54.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:54 smithi125 conmon[56739]: audit [DBG] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["2"]}]: dispatch 2021-03-15T22:34:54.791 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:54.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:54.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:54.893 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:54.915 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:54.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:54.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:55.017 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:55.042 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:55.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:55.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:55.144 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:55.165 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:55.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:55.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:55.267 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:55.289 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:55.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:55.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:55.328 INFO:tasks.rados.rados.0.smithi076.stdout:3558: done (0 left) 2021-03-15T22:34:55.328 INFO:tasks.rados.rados.0.smithi076.stdout:3559: snap_create 2021-03-15T22:34:55.392 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:55.414 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:55.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:55.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:55.516 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:55.539 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:55.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:55.540 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:55.641 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:55.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[62156]: audit 2021-03-15T22:34:54.378816+0000 mon.a (mon.0 2021-03-15T22:34:55.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[62156]: ) 2357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2021-03-15T22:34:55.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[62156]: cluster 2021-03-15T22:34:54.378901+0000 mon.a (mon.0) 2358 : cluster [DBG] osdmap e939: 8 total, 8 up, 8 in 2021-03-15T22:34:55.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[62156]: cluster 2021-03-15T22:34:55.324060+0000 mon.a (mon.0) 2359 : 2021-03-15T22:34:55.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[62156]: cluster [DBG] osdmap e940: 8 total, 8 up, 8 in 2021-03-15T22:34:55.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[56514]: audit 2021-03-15T22:34:55.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[56514]: 2021-03 2021-03-15T22:34:55.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[56514]: -15T22:34:54.378816+0000 mon.a (mon.0) 2357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2021-03-15T22:34:55.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:34:55.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[56514]: -15T22:34:54.378901+0000 mon.a (mon.0) 2021-03-15T22:34:55.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[56514]: 2358 : cluster [DBG] osdmap e939: 8 total, 8 up, 8 in 2021-03-15T22:34:55.660 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:34:55.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[56514]: -15T22:34:55.324060+0000 mon.a (mon. 2021-03-15T22:34:55.661 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:55 smithi076 conmon[56514]: 0) 2359 : cluster [DBG] osdmap e940: 8 total, 8 up, 8 in 2021-03-15T22:34:55.662 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:55.663 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:55.682 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:55.685 INFO:tasks.thrashosds.thrasher:Added osd 2 2021-03-15T22:34:55.764 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:55.786 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:55 smithi125 conmon[56739]: audit 2021-03-15T22:34:54.378816+0000 2021-03-15T22:34:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:55 smithi125 conmon[56739]: mon.a (mon.0) 2357 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["2"]}]': finished 2021-03-15T22:34:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:55 smithi125 conmon[56739]: cluster 2021-03-15T22:34:54.378901+0000 mon.a (mon.0) 2358 : cluster 2021-03-15T22:34:55.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:55 smithi125 conmon[56739]: [DBG] osdmap e939: 8 total, 8 up, 8 in 2021-03-15T22:34:55.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:55 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:34:55.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:55 smithi125 conmon[56739]: :34:55.324060+0000 mon.a (mon.0) 2359 : cluster [DBG] osdmap e940: 8 total, 8 up, 8 in 2021-03-15T22:34:55.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:55 smithi125 conmon[56739]: 2021-03-15T22:34:55.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:55.789 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:55.890 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:55.912 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:55.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:55.914 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:56.015 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:56.036 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:56.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:56.038 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:56.138 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:34:56.138 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:34:56.140 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:56.164 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:56.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:56.165 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:56.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: debug 2021-03-15T22:34:56.083+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:34:56.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:34:56.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.207 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:34:56.208 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.208 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:34:56.208 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.208 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.208 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:34:56.209 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: debug 2021-03-15T22:34:56.084+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:34:56.209 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:34:56.209 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.209 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:34:56.210 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.210 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:34:56.210 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.210 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.210 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:34:56.211 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: debug 2021-03-15T22:34:56.084+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:34:56.211 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:34:56.211 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.211 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:34:56.212 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.212 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:34:56.212 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.212 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.212 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:34:56.213 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: debug 2021-03-15T22:34:56.084+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:34:56.213 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:34:56.213 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.213 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:34:56.214 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.214 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:34:56.214 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.214 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.215 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:34:56.215 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: debug 2021-03-15T22:34:56.085+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:34:56.215 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:34:56.215 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.215 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:34:56.216 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.216 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:34:56.216 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.216 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.217 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:34:56.217 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: debug 2021-03-15T22:34:56.085+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:34:56.217 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:34:56.217 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.217 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:34:56.218 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.218 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:34:56.218 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.218 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.219 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:34:56.219 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: debug 2021-03-15T22:34:56.085+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:34:56.219 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:34:56.219 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.219 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:34:56.220 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.220 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:34:56.220 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.220 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.220 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:34:56.266 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:56.288 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:56.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:56.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:56.337 INFO:tasks.rados.rados.0.smithi076.stdout:3559: done (0 left) 2021-03-15T22:34:56.337 INFO:tasks.rados.rados.0.smithi076.stdout:3560: write oid 39 current snap is 421 2021-03-15T22:34:56.337 INFO:tasks.rados.rados.0.smithi076.stdout:3560: seq_num 1215 ranges {420505=727662,1798916=553091} 2021-03-15T22:34:56.340 INFO:tasks.rados.rados.0.smithi076.stdout:3560: writing smithi07694878-39 from 420505 to 1148167 tid 1 2021-03-15T22:34:56.348 INFO:tasks.rados.rados.0.smithi076.stdout:3560: writing smithi07694878-39 from 1798916 to 2352007 tid 2 2021-03-15T22:34:56.348 INFO:tasks.rados.rados.0.smithi076.stdout:3561: write (excl) oid 31 current snap is 421 2021-03-15T22:34:56.348 INFO:tasks.rados.rados.0.smithi076.stdout:3561: seq_num 1216 ranges {437312=402911,1341045=696749} 2021-03-15T22:34:56.353 INFO:tasks.rados.rados.0.smithi076.stdout:3561: writing smithi07694878-31 from 437312 to 840223 tid 1 2021-03-15T22:34:56.361 INFO:tasks.rados.rados.0.smithi076.stdout:3561: writing smithi07694878-31 from 1341045 to 2037794 tid 2 2021-03-15T22:34:56.362 INFO:tasks.rados.rados.0.smithi076.stdout:3562: read oid 21 snap -1 2021-03-15T22:34:56.362 INFO:tasks.rados.rados.0.smithi076.stdout:3562: expect (ObjNum 1212 snap 419 seq_num 1212) 2021-03-15T22:34:56.362 INFO:tasks.rados.rados.0.smithi076.stdout:3563: snap_remove snap 413 2021-03-15T22:34:56.391 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:56.413 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:56.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:56.414 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:56.515 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:56.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: cluster 2021-03-15T22:34:55.211750+0000 mgr.y (mgr.14144) 1049 : cluster [DBG] pgmap v1556: 158 pgs: 158 active+clean; 667 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.8 MiB/s wr, 18 op/s; 5.2 MiB/s, 4 objects/s recovering 2021-03-15T22:34:56.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: cluster 2021-03-15T22:34:55.373457+0000 mon.a (mon.0) 2360 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/1917 objects degraded (0.209%), 1 pg degraded) 2021-03-15T22:34:56.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: audit 2021-03-15T22:34:55.539836+0000 mon.a (mon.0) 2361 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:34:56.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: audit 2021-03-15T22:34:55.803241+0000 mon.a (mon.0) 2362 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:34:56.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:34:56.078621+0000 mgr.y (mgr.14144) 1050 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:34:56.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: audit 2021-03-15T22:34:56.083275+0000 mon.a (mon.0) 2363 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:34:56.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:34:56.084177+0000 mgr.y (mgr.14144) 1051 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:34:56.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:34:56.520 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:34:56.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:34:56.521 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:34:56.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:34:56.084717+0000 mgr.y (mgr.14144) 1052 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:34:56.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:34:56.522 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:34:56.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:34:56.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:34:56.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: cluster 2021-03-15T22:34:55.211750+0000 mgr.y (mgr 2021-03-15T22:34:56.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: .14144) 1049 : cluster [DBG] pgmap v1556: 158 pgs: 158 active+clean; 667 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.8 MiB/s wr, 18 op/s; 5.2 MiB/s, 4 objects/s recovering 2021-03-15T22:34:56.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: cluster 2021-03-15T22:34:55 2021-03-15T22:34:56.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: .373457+0000 mon.a (mon.0) 2360 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/1917 objects degraded (0.209%), 1 pg degraded) 2021-03-15T22:34:56.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: audit 2021-03-15T22:34:56.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: 2021-03-15T22:34:55. 2021-03-15T22:34:56.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: 539836+0000 mon.a (mon.0) 2361 : audit 2021-03-15T22:34:56.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:34:56.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:34:56.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: T22:34:55.803241+0000 mon.a ( 2021-03-15T22:34:56.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: mon.0) 2362 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:34:56.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: cephadm 2021-03 2021-03-15T22:34:56.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: -15T22:34:56.078621+0000 mgr.y ( 2021-03-15T22:34:56.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: mgr.14144) 1050 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:34:56.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: audit 2021-03-15T22:34:56.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: 2021-03-15T22:34:56 2021-03-15T22:34:56.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: .083275+0000 mon.a (mon.0) 2363 : 2021-03-15T22:34:56.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:34:56.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: cephadm 2021 2021-03-15T22:34:56.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: -03-15T22:34:56.084177 2021-03-15T22:34:56.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 1051 : cephadm 2021-03-15T22:34:56.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:34:56.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:34:56.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:34:56.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:34:56.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:34:56.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:34:56.084717+0000 mgr.y (mgr.14144) 1052 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:34:56.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:34:56.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:34:56.532 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:34:56.085117+0000 mgr.y (mgr.14144) 1053 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:34:56.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:34:56.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:34:56.533 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:34:56.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.534 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:34:56.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: cephadm 2021 2021-03-15T22:34:56.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: -03-15T22:34:56.085486+0000 mgr.y (mgr.14144) 1054 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:34:56.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:34:56.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.535 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:34:56.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:34:56.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.536 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:34:56.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:34:56.085893+0000 mgr.y (mgr.14144) 1055 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:34:56.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:34:56.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.537 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:34:56.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:34:56.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.538 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:34:56.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:34:56.086264+0000 mgr.y (mgr.14144) 1056 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:34:56.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:34:56.539 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:34:56.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:34:56.540 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:34:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: cephadm 2021-03-15T22:34:56.086628+0000 mgr.y (mgr.14144) 1057 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:34:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:34:56.541 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:34:56.542 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:34:56.543 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:34:56.544 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[56514]: cluster 2021-03-15T22:34:56.326148+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e941: 8 total, 8 up, 8 in 2021-03-15T22:34:56.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:34:56.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.545 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:34:56.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:34:56.085117+0000 mgr.y (mgr.14144) 1053 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:34:56.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:34:56.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.546 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:34:56.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:34:56.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.547 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:34:56.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: cephadm 2021-03- 2021-03-15T22:34:56.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: 15T22:34:56.085486+0000 mgr.y (mgr.14144) 1054 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:34:56.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:34:56.548 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:34:56.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:34:56.549 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:34:56.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:34:56.085893 2021-03-15T22:34:56.550 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 1055 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:34:56.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:34:56.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:34:56.551 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:34:56.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:34:56.552 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:34:56.086264+0000 mgr.y (mgr.14144) 1056 : cephadm 2021-03-15T22:34:56.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:34:56.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:34:56.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:34:56.553 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:34:56.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.554 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:34:56.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: cephadm 2021-03-15T22:34:56.086628+0000 mgr.y (mgr.14144) 1057 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:34:56.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:34:56.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:34:56.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:34:56.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:34:56.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: cluster 2021-03-15T22:34:56.557 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:56 smithi076 conmon[62156]: 2021-03-15T22:34:56.326148+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e941: 8 total, 8 up, 8 in 2021-03-15T22:34:56.580 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:56.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:56.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:56.655 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:34:56.682 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:56.726 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:56.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:56.727 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: cluster 2021-03-15T22:34:55.211750+0000 mgr.y (mgr.14144) 1049 : cluster 2021-03-15T22:34:56.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: [DBG] pgmap v1556: 158 pgs: 158 active+clean; 667 MiB data, 1.8 GiB used, 624 GiB / 626 GiB avail; 8.4 MiB/s rd, 2.8 MiB/s wr, 18 op/s; 5.2 MiB/s, 4 objects/s recovering 2021-03-15T22:34:56.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: cluster 2021-03-15T22:34:55.373457+0000 mon.a (mon.0) 2360 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 4/1917 objects degraded (0.209%), 1 pg degraded) 2021-03-15T22:34:56.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: audit 2021-03-15T22:34:55.539836+0000 mon.a (mon.0) 2361 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:34:56.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: audit 2021-03-15T22:34:55.803241+0000 mon.a (mon.0) 2362 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:34:56.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:34:56.078621+0000 mgr.y (mgr.14144) 1050 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:34:56.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: audit 2021-03-15T22:34:56.083275+0000 mon.a (mon.0) 2363 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:34:56.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:34:56.084177+0000 mgr.y (mgr.14144) 1051 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:34:56.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:34:56.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:34:56.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:34:56.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:34:56.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:34:56.084717+0000 mgr.y (mgr.14144) 1052 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:34:56.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:34:56.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:34:56.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:34:56.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:34:56.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:34:56.085117+0000 mgr.y (mgr.14144) 1053 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:34:56.793 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:34:56.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:34:56.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.794 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:34:56.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:34:56.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:34:56.085486+0000 mgr.y (mgr.14144) 1054 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:34:56.795 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:34:56.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:34:56.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.796 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:34:56.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:34:56.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:34:56.085893+0000 mgr.y (mgr.14144) 1055 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:34:56.797 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:34:56.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:34:56.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.798 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:34:56.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:34:56.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:34:56.086264+0000 mgr.y (mgr.14144) 1056 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:34:56.799 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:34:56.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:34:56.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.800 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:34:56.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:34:56.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: cephadm 2021-03-15T22:34:56.086628+0000 mgr.y (mgr.14144) 1057 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:34:56.801 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:34:56.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:34:56.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:34:56.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:34:56.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:34:56.802 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:34:56.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:34:56.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:34:56.803 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:56 smithi125 conmon[56739]: cluster 2021-03-15T22:34:56.326148+0000 mon.a (mon.0) 2364 : cluster [DBG] osdmap e941: 8 total, 8 up, 8 in 2021-03-15T22:34:56.829 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:56.850 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:56.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:56.852 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:56.953 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:34:56.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:56.982 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:56.982 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:34:57.083 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:57.107 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:57.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:57.109 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:57.206 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:34:57.209 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:57.230 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:57.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:57.231 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:57.303 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:57.332 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:57.333 INFO:tasks.rados.rados.0.smithi076.stdout:3563: done (3 left) 2021-03-15T22:34:57.334 INFO:tasks.rados.rados.0.smithi076.stdout:3564: snap_remove snap 383 2021-03-15T22:34:57.357 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:57.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:57.358 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:57.394 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[56514]: cluster 2021-03-15T22:34:56.327834+0000 osd.0 ( 2021-03-15T22:34:57.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[56514]: osd.0) 150 : cluster [DBG] 5.d starting backfill to osd.2 from (0'0,0'0] MAX to 881'16 2021-03-15T22:34:57.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[56514]: cluster 2021-03-15T22:34:56.329563+0000 osd.0 (osd.0) 151 : cluster [DBG] 5.1e starting backfill to osd.2 from (0'0,0'0] MAX to 643'13 2021-03-15T22:34:57.395 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[56514]: cluster 2021-03-15T22:34:56.467136+0000 osd.7 (osd.7) 174 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:57.396 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[56514]: cluster 2021-03-15T22:34:56.469324+0000 osd.7 (osd.7) 175 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:57.396 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[56514]: cluster 2021-03-15T22:34:57.331605+0000 2021-03-15T22:34:57.396 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[56514]: mon.a (mon.0) 2365 : cluster [DBG] osdmap e942: 8 total, 8 up, 8 in 2021-03-15T22:34:57.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:57.480 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:57.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:57.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:57.582 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:57.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:57.607 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:57.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:57.708 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:57.729 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:57.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:57.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:57.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: cluster 2021-03-15T22:34:56.327834+0000 2021-03-15T22:34:57.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: osd.0 (osd.0) 150 : cluster [DBG] 5.d starting backfill to osd.2 from (0'0,0'0] MAX to 881'16 2021-03-15T22:34:57.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: cluster 2021-03-15T22:34:56.329563+0000 osd.0 (osd. 2021-03-15T22:34:57.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: 0) 151 : cluster [DBG] 5.1e starting backfill to osd.2 from (0'0,0'0] MAX to 643'13 2021-03-15T22:34:57.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: cluster 2021-03-15T22:34 2021-03-15T22:34:57.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: :56.467136+0000 osd.7 (osd.7) 174 2021-03-15T22:34:57.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:57.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:34:57.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: -03-15T22:34:56.469324+0000 2021-03-15T22:34:57.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: osd.7 (osd.7) 175 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:57.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: 2021-03-15T22:34:57.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:34:57.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: 22:34:57.331605+0000 mon.a (mon 2021-03-15T22:34:57.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:57 smithi076 conmon[62156]: .0) 2365 : cluster [DBG] osdmap e942: 8 total, 8 up, 8 in 2021-03-15T22:34:57.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: cluster 2021-03-15T22:34:56.327834+0000 osd.0 (osd 2021-03-15T22:34:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: .0) 150 : cluster [DBG] 5.d starting backfill to osd.2 from (0'0,0'0] MAX to 881'16 2021-03-15T22:34:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: cluster 2021-03-15T22:34:56.329563+0000 osd.0 (osd.0) 151 : cluster 2021-03-15T22:34:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: [DBG] 5.1e starting backfill to osd.2 from (0'0,0'0] MAX to 643'13 2021-03-15T22:34:57.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:34:57.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: T22:34:56.467136+0000 osd.7 (osd.7) 174 2021-03-15T22:34:57.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:57.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:34:57.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: T22:34:56.469324+0000 osd.7 ( 2021-03-15T22:34:57.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: osd.7) 175 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:57.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:34:57.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: 03-15T22:34:57.331605+0000 2021-03-15T22:34:57.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: mon.a (mon.0) 2365 : cluster [DBG] 2021-03-15T22:34:57.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:57 smithi125 conmon[56739]: osdmap e942: 8 total, 8 up, 8 in 2021-03-15T22:34:57.832 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:57.852 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:57.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:57.854 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:57.955 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:57.975 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:57.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:57.977 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:58.078 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:58.099 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:58.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:58.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:58.202 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:58.219 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:34:58.225 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:58.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:58.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:58.328 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:58.345 INFO:tasks.rados.rados.0.smithi076.stdout:3564: done (3 left) 2021-03-15T22:34:58.345 INFO:tasks.rados.rados.0.smithi076.stdout:3565: read oid 10 snap -1 2021-03-15T22:34:58.346 INFO:tasks.rados.rados.0.smithi076.stdout:3565: expect (ObjNum 1183 snap 414 seq_num 1183) 2021-03-15T22:34:58.346 INFO:tasks.rados.rados.0.smithi076.stdout:3566: read oid 15 snap 387 2021-03-15T22:34:58.346 INFO:tasks.rados.rados.0.smithi076.stdout:3566: expect (ObjNum 1070 snap 364 seq_num 1070) 2021-03-15T22:34:58.346 INFO:tasks.rados.rados.0.smithi076.stdout:3567: snap_remove snap 349 2021-03-15T22:34:58.376 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:58.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:58.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:58.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[56514]: cluster 2021-03-15T22:34:57.212316+0000 mgr.y (mgr.14144) 1058 : cluster 2021-03-15T22:34:58.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[56514]: [DBG] pgmap v1559: 158 pgs: 1 active+recovery_wait+remapped, 4 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 149 active+clean; 677 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 2.4 MiB/s wr, 16 op/s; 2/1932 objects degraded (0.104%); 162/1932 objects misplaced (8.385%); 11 MiB/s, 37 objects/s recovering 2021-03-15T22:34:58.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[56514]: cluster 2021-03-15T22:34:57.386803+0000 mon.a (mon.0) 2366 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:34:58.399 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[56514]: cluster 2021-03-15T22:34:57.386853+0000 mon.a (mon.0) 2367 : cluster [WRN] Health check failed: Degraded data redundancy: 2/1932 objects degraded (0.104%), 2 pgs degraded (PG_DEGRADED) 2021-03-15T22:34:58.400 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[56514]: cluster 2021-03-15T22:34:58.334875+0000 mon.a (mon.0) 2368 : cluster [DBG] osdmap e943: 8 total, 8 up, 8 in 2021-03-15T22:34:58.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[62156]: cluster 2021-03-15T22:34:57. 2021-03-15T22:34:58.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[62156]: 212316+0000 mgr.y (mgr.14144) 1058 : cluster [DBG] pgmap v1559: 158 pgs: 1 active+recovery_wait+remapped, 4 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 149 active+clean; 677 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 2.4 MiB/s wr, 16 op/s; 2/1932 objects degraded (0.104%); 162/1932 objects misplaced (8.385%); 11 MiB/s, 37 objects/s recovering 2021-03-15T22:34:58.400 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[62156]: cluster 2021-03-15T22:34:58.479 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:58.501 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:58.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:58.502 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:58.603 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:34:58.626 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:58.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:58.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:34:58.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[62156]: 2021-03-15T22:34:57.386803+0000 mon.a (mon.0) 2366 : cluster 2021-03-15T22:34:58.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[62156]: [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:34:58.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[62156]: cluster 2021-03-15T22:34: 2021-03-15T22:34:58.664 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[62156]: 57.386853+0000 mon.a (mon.0) 2367 : cluster 2021-03-15T22:34:58.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[62156]: [WRN] Health check failed: Degraded data redundancy: 2/1932 objects degraded (0.104%), 2 pgs degraded (PG_DEGRADED) 2021-03-15T22:34:58.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[62156]: cluster 2021-03-15T22:34:58. 2021-03-15T22:34:58.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[62156]: 334875+0000 mon.a (mon.0) 2368 : cluster [DBG] 2021-03-15T22:34:58.665 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:58 smithi076 conmon[62156]: osdmap e943: 8 total, 8 up, 8 in 2021-03-15T22:34:58.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:58.749 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:58.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:58.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:58.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:58 smithi125 conmon[56739]: cluster 2021-03-15T22:34:57.212316+0000 mgr.y (mgr 2021-03-15T22:34:58.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:58 smithi125 conmon[56739]: .14144) 1058 : cluster [DBG] pgmap v1559: 158 pgs: 1 active+recovery_wait+remapped, 4 remapped+peering, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 149 active+clean; 677 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 6.5 MiB/s rd, 2.4 MiB/s wr, 16 op/s; 2/1932 objects degraded (0.104%); 162/1932 objects misplaced (8.385%); 11 MiB/s, 37 objects/s recovering 2021-03-15T22:34:58.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:58 smithi125 conmon[56739]: cluster 2021-03-15T22:34:57.386803+0000 mon.a (mon.0) 2366 : cluster [WRN] Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2021-03-15T22:34:58.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:58 smithi125 conmon[56739]: cluster 2021-03-15T22:34:57.386853+0000 mon.a ( 2021-03-15T22:34:58.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:58 smithi125 conmon[56739]: mon.0) 2367 : cluster [WRN] Health check failed: Degraded data redundancy: 2/1932 objects degraded (0.104%), 2 pgs degraded (PG_DEGRADED) 2021-03-15T22:34:58.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:58 smithi125 conmon[56739]: cluster 2021-03-15T22:34:58.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:58 smithi125 conmon[56739]: 2021-03-15T22:34:58.334875+0000 2021-03-15T22:34:58.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:58 smithi125 conmon[56739]: mon.a (mon.0) 2368 : cluster [DBG] 2021-03-15T22:34:58.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:58 smithi125 conmon[56739]: osdmap e943: 8 total, 8 up, 8 in 2021-03-15T22:34:58.852 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:58.873 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:58.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:58.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:58.975 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:34:58.996 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:58.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:58.997 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:34:59.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:59.121 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:59.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:59.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:59.224 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:34:59.246 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:59.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:59.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:34:59.334 INFO:tasks.rados.rados.0.smithi076.stdout:3567: done (5 left) 2021-03-15T22:34:59.334 INFO:tasks.rados.rados.0.smithi076.stdout:3568: read oid 28 snap -1 2021-03-15T22:34:59.335 INFO:tasks.rados.rados.0.smithi076.stdout:3568: expect deleted 2021-03-15T22:34:59.335 INFO:tasks.rados.rados.0.smithi076.stdout:3569: write (excl) oid 5 current snap is 421 2021-03-15T22:34:59.335 INFO:tasks.rados.rados.0.smithi076.stdout:3561: finishing write tid 1 to smithi07694878-31 2021-03-15T22:34:59.335 INFO:tasks.rados.rados.0.smithi076.stdout:3569: seq_num 1217 ranges {454119=594513,1515880=556760} 2021-03-15T22:34:59.341 INFO:tasks.rados.rados.0.smithi076.stdout:3569: writing smithi07694878-5 from 454119 to 1048632 tid 1 2021-03-15T22:34:59.344 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:34:59.348 INFO:tasks.rados.rados.0.smithi076.stdout:3569: writing smithi07694878-5 from 1515880 to 2072640 tid 2 2021-03-15T22:34:59.348 INFO:tasks.rados.rados.0.smithi076.stdout:3570: snap_remove snap 380 2021-03-15T22:34:59.348 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:59.376 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:59.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:59.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:59.479 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:34:59.500 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:59.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:59.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:34:59.602 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:34:59.623 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:59.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:59.624 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:34:59.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[56514]: audit 2021-03-15T22:34:58.821528+0000 mon.c (mon.1) 77 : audit [INF] from='client.? 172.21.15.76:0/304688975' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:59.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[56514]: audit 2021-03-15T22:34:58.821845+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:59.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[56514]: cluster 2021-03-15T22:34:58.943526+0000 osd.0 (osd.0) 152 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:59.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[56514]: cluster 2021-03-15T22:34:58.945591+0000 osd.0 (osd.0) 153 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:59.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[56514]: cluster 2021-03-15T22:34:59.325371+0000 mon.a (mon.0) 2370 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:59.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[56514]: audit 2021-03-15T22:34:59.328707+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:59.658 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[56514]: cluster 2021-03-15T22:34:59.328787+0000 mon.a (mon.0) 2372 : cluster [DBG] 2021-03-15T22:34:59.659 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[56514]: osdmap e944: 8 total, 8 up, 8 in 2021-03-15T22:34:59.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[62156]: audit 2021-03-15T22:34:58.821528+0000 mon.c (mon.1) 77 : audit [INF] from='client.? 172.21.15.76:0/304688975' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:59.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[62156]: audit 2021-03-15T22:34:58.821845+0000 mon.a (mon.0) 2369 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:59.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[62156]: cluster 2021-03-15T22:34:58.943526+0000 osd.0 (osd.0) 152 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:59.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[62156]: cluster 2021-03-15T22:34:58.945591+0000 osd.0 (osd.0) 153 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:34:59.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[62156]: cluster 2021-03-15T22:34:59.325371+0000 mon.a (mon.0) 2370 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:59.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[62156]: audit 2021-03-15T22:34:59.328707+0000 mon.a (mon.0) 2371 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:59.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:34:59 smithi076 conmon[62156]: cluster 2021-03-15T22:34:59.328787+0000 mon.a (mon.0) 2372 : cluster [DBG] osdmap e944: 8 total, 8 up, 8 in 2021-03-15T22:34:59.725 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:34:59.747 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:34:59.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:59.749 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:34:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: audit 2021-03-15T22:34:58. 2021-03-15T22:34:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: 821528+0000 mon.c (mon.1) 77 : audit [INF] from='client.? 172.21.15.76:0/304688975' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: audit 2021-03-15T22:34:58.821845+0000 2021-03-15T22:34:59.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: mon.a (mon.0) 2369 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:34:59.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:34:59.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: T22:34:58. 2021-03-15T22:34:59.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: 943526+0000 osd.0 (osd.0) 152 : 2021-03-15T22:34:59.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:34:59.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:34:59.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: -03-15T22:34:58.945591 2021-03-15T22:34:59.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: +0000 osd.0 (osd.0) 153 : cluster 2021-03-15T22:34:59.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: [DBG] purged_snaps scrub ok 2021-03-15T22:34:59.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:34:59.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: T22:34:59.325371+0000 mon.a ( 2021-03-15T22:34:59.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: mon.0) 2370 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:34:59.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: 2021-03-15T22:34:59.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: audit 2021-03-15T22:34:59.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: 2021-03-15T22:34:59.328707 2021-03-15T22:34:59.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2371 : audit 2021-03-15T22:34:59.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:34:59.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:34:59.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: 22:34:59.328787+0000 mon.a 2021-03-15T22:34:59.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: (mon.0) 2372 : cluster [DBG] 2021-03-15T22:34:59.792 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:34:59 smithi125 conmon[56739]: osdmap e944: 8 total, 8 up, 8 in 2021-03-15T22:34:59.850 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:59.870 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:59.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:59.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:34:59.972 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:34:59.993 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:34:59.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:34:59.995 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:00.096 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:00.117 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:00.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:00.118 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:00.219 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:00.242 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:00.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:00.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:00.331 INFO:tasks.rados.rados.0.smithi076.stdout:3570: done (7 left) 2021-03-15T22:35:00.331 INFO:tasks.rados.rados.0.smithi076.stdout:3571: read oid 42 snap 397 2021-03-15T22:35:00.332 INFO:tasks.rados.rados.0.smithi076.stdout:3571: expect (ObjNum 1109 snap 386 seq_num 1109) 2021-03-15T22:35:00.332 INFO:tasks.rados.rados.0.smithi076.stdout:3572: snap_create 2021-03-15T22:35:00.333 INFO:tasks.rados.rados.0.smithi076.stdout:3561: finishing write tid 2 to smithi07694878-31 2021-03-15T22:35:00.333 INFO:tasks.rados.rados.0.smithi076.stdout:3561: finishing write tid 3 to smithi07694878-31 2021-03-15T22:35:00.333 INFO:tasks.rados.rados.0.smithi076.stdout:3561: finishing write tid 4 to smithi07694878-31 2021-03-15T22:35:00.334 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 2786 (ObjNum 1216 snap 421 seq_num 1216) dirty exists 2021-03-15T22:35:00.334 INFO:tasks.rados.rados.0.smithi076.stdout:3561: left oid 31 (ObjNum 1216 snap 421 seq_num 1216) 2021-03-15T22:35:00.334 INFO:tasks.rados.rados.0.smithi076.stdout:3560: finishing write tid 1 to smithi07694878-39 2021-03-15T22:35:00.334 INFO:tasks.rados.rados.0.smithi076.stdout:3560: finishing write tid 2 to smithi07694878-39 2021-03-15T22:35:00.334 INFO:tasks.rados.rados.0.smithi076.stdout:3560: finishing write tid 3 to smithi07694878-39 2021-03-15T22:35:00.335 INFO:tasks.rados.rados.0.smithi076.stdout:3560: finishing write tid 4 to smithi07694878-39 2021-03-15T22:35:00.335 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2803 (ObjNum 1215 snap 421 seq_num 1215) dirty exists 2021-03-15T22:35:00.335 INFO:tasks.rados.rados.0.smithi076.stdout:3560: left oid 39 (ObjNum 1215 snap 421 seq_num 1215) 2021-03-15T22:35:00.335 INFO:tasks.rados.rados.0.smithi076.stdout:3562: expect (ObjNum 1212 snap 419 seq_num 1212) 2021-03-15T22:35:00.345 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:00.381 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:00.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:00.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:00.484 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:00.506 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:00.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:00.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:00.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:35:00 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:35:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:00 smithi125 conmon[56739]: cluster 2021-03-15T22:34:59.212694+0000 mgr.y (mgr 2021-03-15T22:35:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:00 smithi125 conmon[56739]: .14144) 1059 : cluster [DBG] pgmap v1562: 158 pgs: 1 peering, 1 active+recovery_wait+remapped, 4 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 149 active+clean; 679 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.3 MiB/s wr, 9 op/s; 2/1935 objects degraded (0.103%); 110/1935 objects misplaced (5.685%); 5.5 MiB/s, 36 objects/s recovering 2021-03-15T22:35:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:00 smithi125 conmon[56739]: cluster 2021-03-15T22:35:00.326867+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e945: 8 total, 8 up, 8 in 2021-03-15T22:35:00.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:00 smithi125 conmon[56739]: 2021-03-15T22:35:00.588 INFO:tasks.rados.rados.0.smithi076.stdout:3565: expect (ObjNum 1183 snap 414 seq_num 1183) 2021-03-15T22:35:00.608 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:00.631 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:00.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:00.632 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:00.687 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 7, 0, 3, 5, 6, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:35:00.687 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:35:00.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump -f json-pretty 2021-03-15T22:35:00.709 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:00 smithi076 conmon[56514]: cluster 2021-03-15T22:34:59.212694+0000 mgr.y (mgr.14144) 1059 : 2021-03-15T22:35:00.709 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:00 smithi076 conmon[56514]: cluster [DBG] pgmap v1562: 158 pgs: 1 peering, 1 active+recovery_wait+remapped, 4 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 149 active+clean; 679 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.3 MiB/s wr, 9 op/s; 2/1935 objects degraded (0.103%); 110/1935 objects misplaced (5.685%); 5.5 MiB/s, 36 objects/s recovering 2021-03-15T22:35:00.710 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:00 smithi076 conmon[56514]: cluster 2021-03-15T22:35:00.326867+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e945: 8 total, 8 up, 8 in 2021-03-15T22:35:00.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:00 smithi076 conmon[62156]: cluster 2021-03-15T22:34:59.212694+0000 mgr.y (mgr.14144) 2021-03-15T22:35:00.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:00 smithi076 conmon[62156]: 1059 : cluster [DBG] pgmap v1562: 158 pgs: 1 peering, 1 active+recovery_wait+remapped, 4 remapped+peering, 1 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 1 active+recovering, 149 active+clean; 679 MiB data, 1.9 GiB used, 624 GiB / 626 GiB avail; 3.0 MiB/s rd, 1.3 MiB/s wr, 9 op/s; 2/1935 objects degraded (0.103%); 110/1935 objects misplaced (5.685%); 5.5 MiB/s, 36 objects/s recovering 2021-03-15T22:35:00.710 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:00 smithi076 conmon[62156]: cluster 2021-03-15T22:35:00.326867+0000 mon.a (mon.0) 2373 : cluster [DBG] osdmap e945: 8 total, 8 up, 8 in 2021-03-15T22:35:00.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:00.753 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:00.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:00.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:00.806 INFO:tasks.rados.rados.0.smithi076.stdout:3566: expect (ObjNum 1070 snap 364 seq_num 1070) 2021-03-15T22:35:00.855 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:00.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:00.881 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:00.882 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:00.983 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:01.001 INFO:tasks.rados.rados.0.smithi076.stdout:3569: finishing write tid 1 to smithi07694878-5 2021-03-15T22:35:01.001 INFO:tasks.rados.rados.0.smithi076.stdout:3569: finishing write tid 2 to smithi07694878-5 2021-03-15T22:35:01.001 INFO:tasks.rados.rados.0.smithi076.stdout:3569: finishing write tid 3 to smithi07694878-5 2021-03-15T22:35:01.001 INFO:tasks.rados.rados.0.smithi076.stdout:3569: finishing write tid 4 to smithi07694878-5 2021-03-15T22:35:01.002 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 2006 (ObjNum 1217 snap 421 seq_num 1217) dirty exists 2021-03-15T22:35:01.002 INFO:tasks.rados.rados.0.smithi076.stdout:3569: left oid 5 (ObjNum 1217 snap 421 seq_num 1217) 2021-03-15T22:35:01.002 INFO:tasks.rados.rados.0.smithi076.stdout:3571: expect (ObjNum 1109 snap 386 seq_num 1109) 2021-03-15T22:35:01.005 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:01.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:01.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:01.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:01.132 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:01.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:01.134 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:01.140 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:01.193 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:35:01.210 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:35:01.227 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:35:01.235 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:01.246 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:35:01.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:01.260 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:01.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:01.264 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:35:01.284 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:35:01.301 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:35:01.321 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:35:01.323 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:35:01.323 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:35:01.324 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:35:01.324 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:35:01.324 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:35:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:35:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:35:01.325 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:35:01.343 INFO:tasks.rados.rados.0.smithi076.stdout:3560: done (8 left) 2021-03-15T22:35:01.343 INFO:tasks.rados.rados.0.smithi076.stdout:3561: done (7 left) 2021-03-15T22:35:01.344 INFO:tasks.rados.rados.0.smithi076.stdout:3562: done (6 left) 2021-03-15T22:35:01.344 INFO:tasks.rados.rados.0.smithi076.stdout:3565: done (5 left) 2021-03-15T22:35:01.344 INFO:tasks.rados.rados.0.smithi076.stdout:3566: done (4 left) 2021-03-15T22:35:01.345 INFO:tasks.rados.rados.0.smithi076.stdout:3568: done (3 left) 2021-03-15T22:35:01.345 INFO:tasks.rados.rados.0.smithi076.stdout:3569: done (2 left) 2021-03-15T22:35:01.345 INFO:tasks.rados.rados.0.smithi076.stdout:3571: done (1 left) 2021-03-15T22:35:01.346 INFO:tasks.rados.rados.0.smithi076.stdout:3572: done (0 left) 2021-03-15T22:35:01.346 INFO:tasks.rados.rados.0.smithi076.stdout:3573: delete oid 43 current snap is 422 2021-03-15T22:35:01.349 INFO:tasks.rados.rados.0.smithi076.stdout:3573: done (0 left) 2021-03-15T22:35:01.349 INFO:tasks.rados.rados.0.smithi076.stdout:3574: read oid 47 snap -1 2021-03-15T22:35:01.350 INFO:tasks.rados.rados.0.smithi076.stdout:3574: expect deleted 2021-03-15T22:35:01.350 INFO:tasks.rados.rados.0.smithi076.stdout:3575: snap_create 2021-03-15T22:35:01.362 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:01.383 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:01.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:01.385 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:01.487 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:01.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:01.512 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:01.513 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:01.558 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:35:01.558 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:35:01.559 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:35:01.559 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:35:01.559 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:35:01.614 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:01.638 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:01.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:01.641 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:01.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:01 smithi076 conmon[56514]: audit 2021-03-15T22:35:00.277776+0000 mgr.y (mgr.14144) 1060 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:01.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:01 smithi076 conmon[56514]: audit 2021-03-15T22:35:01.120145+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:35:01.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:01 smithi076 conmon[56514]: audit 2021-03-15T22:35:01.121859+0000 mon.a (mon.0) 2375 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:35:01.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:01 smithi076 conmon[56514]: audit 2021-03-15T22:35:01.173615+0000 mon.c (mon.1) 78 : audit [DBG] from='client.? 172.21.15.76:0/2513435002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:01.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:01 smithi076 conmon[56514]: cluster 2021-03-15T22:35:01.338506+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e946: 8 total, 8 up, 8 in 2021-03-15T22:35:01.649 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:01 smithi076 conmon[62156]: audit 2021-03-15T22:35:00.277776+0000 mgr.y (mgr.14144) 1060 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:01.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:01 smithi076 conmon[62156]: audit 2021-03-15T22:35:01.120145+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:35:01.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:01 smithi076 conmon[62156]: audit 2021-03-15T22:35:01.121859+0000 mon.a (mon.0) 2375 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:35:01.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:01 smithi076 conmon[62156]: audit 2021-03-15T22:35:01.173615+0000 mon.c (mon.1) 78 : audit [DBG] from='client.? 172.21.15.76:0/2513435002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:01.650 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:01 smithi076 conmon[62156]: cluster 2021-03-15T22:35:01.338506+0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e946: 8 total, 8 up, 8 in 2021-03-15T22:35:01.742 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:01.764 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:01.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:01.766 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:01.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:01 smithi125 conmon[56739]: audit 2021-03-15T22:35:00.277776+0000 mgr.y (mgr. 2021-03-15T22:35:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:01 smithi125 conmon[56739]: 14144) 1060 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:01 smithi125 conmon[56739]: audit 2021-03-15T22:35:01.120145+0000 mon.a (mon.0) 2374 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:35:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:01 smithi125 conmon[56739]: audit 2021-03-15T22:35:01.121859+0000 mon.a (mon 2021-03-15T22:35:01.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:01 smithi125 conmon[56739]: .0) 2375 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:35:01.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:01 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:35:01.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:01 smithi125 conmon[56739]: 22:35:01.173615+0000 mon.c (mon. 2021-03-15T22:35:01.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:01 smithi125 conmon[56739]: 1) 78 : audit [DBG] from='client.? 172.21.15.76:0/2513435002' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:01.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:01 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:35:01.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:01 smithi125 conmon[56739]: -03-15T22:35:01.338506 2021-03-15T22:35:01.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:01 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2376 : cluster [DBG] osdmap e946: 8 total, 8 up, 8 in 2021-03-15T22:35:01.867 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:01.888 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:01.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:01.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:01.991 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:02.014 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:02.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:02.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:02.117 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:02.138 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:02.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:02.140 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:02.241 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:02.262 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:02.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:02.264 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:02.313 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:35:02.339 INFO:tasks.rados.rados.0.smithi076.stdout:3574: done (1 left) 2021-03-15T22:35:02.339 INFO:tasks.rados.rados.0.smithi076.stdout:3575: done (0 left) 2021-03-15T22:35:02.340 INFO:tasks.rados.rados.0.smithi076.stdout:3576: write (excl) oid 1 current snap is 423 2021-03-15T22:35:02.340 INFO:tasks.rados.rados.0.smithi076.stdout:3576: seq_num 1219 ranges {487733=745011,1632844=560429} 2021-03-15T22:35:02.346 INFO:tasks.rados.rados.0.smithi076.stdout:3576: writing smithi07694878-1 from 487733 to 1232744 tid 1 2021-03-15T22:35:02.352 INFO:tasks.rados.rados.0.smithi076.stdout:3576: writing smithi07694878-1 from 1632844 to 2193273 tid 2 2021-03-15T22:35:02.352 INFO:tasks.rados.rados.0.smithi076.stdout:3577: write (excl) oid 29 current snap is 423 2021-03-15T22:35:02.353 INFO:tasks.rados.rados.0.smithi076.stdout:3577: seq_num 1220 ranges {504540=420260,1574973=704087} 2021-03-15T22:35:02.357 INFO:tasks.rados.rados.0.smithi076.stdout:3577: writing smithi07694878-29 from 504540 to 924800 tid 1 2021-03-15T22:35:02.364 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:02.366 INFO:tasks.rados.rados.0.smithi076.stdout:3577: writing smithi07694878-29 from 1574973 to 2279060 tid 2 2021-03-15T22:35:02.366 INFO:tasks.rados.rados.0.smithi076.stdout:3578: snap_remove snap 398 2021-03-15T22:35:02.391 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:02.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:02.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:02.494 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:02.516 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:02.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:02.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:02.618 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:02.641 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:02.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:02.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:02.732 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[56514]: cluster 2021-03-15T22:35:01.213379+0000 mgr.y (mgr.14144) 1061 : cluster [DBG] pgmap v1565: 158 pgs: 1 active+remapped+backfill_toofull, 1 active, 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 151 active+clean; 680 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 3/1926 objects degraded (0.156%); 96/1926 objects misplaced (4.984%); 21 MiB/s, 66 objects/s recovering 2021-03-15T22:35:02.732 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[56514]: cluster 2021-03-15T22:35:01.450921+0000 mon.a (mon.0) 2377 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:35:02.732 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[56514]: cluster 2021-03-15T22:35:01.450951+0000 mon.a (mon.0) 2378 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-03-15T22:35:02.733 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[56514]: cluster 2021-03-15T22:35:02.337550+0000 mon.a (mon.0) 2379 : cluster [DBG] osdmap e947: 8 total, 8 up, 8 in 2021-03-15T22:35:02.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[62156]: cluster 2021-03-15T22:35:01. 2021-03-15T22:35:02.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[62156]: 213379+0000 mgr.y (mgr.14144) 1061 : cluster [DBG] pgmap v1565: 158 pgs: 1 active+remapped+backfill_toofull, 1 active, 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 151 active+clean; 680 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 3/1926 objects degraded (0.156%); 96/1926 objects misplaced (4.984%); 21 MiB/s, 66 objects/s recovering 2021-03-15T22:35:02.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[62156]: cluster 2021-03-15T22:35:02.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[62156]: 2021-03-15T22:35:01.450921+0000 mon.a (mon.0) 2377 : cluster [WRN] 2021-03-15T22:35:02.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[62156]: Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:35:02.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:35:02.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[62156]: T22:35:01.450951+0000 mon.a ( 2021-03-15T22:35:02.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[62156]: mon.0) 2378 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-03-15T22:35:02.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:35:02.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[62156]: T22:35:02.337550+0000 mon.a (mon.0 2021-03-15T22:35:02.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:02 smithi076 conmon[62156]: ) 2379 : cluster [DBG] osdmap e947: 8 total, 8 up, 8 in 2021-03-15T22:35:02.744 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:02.759 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:35:02.765 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:02.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:02.766 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:02.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:02 smithi125 conmon[56739]: cluster 2021-03-15T22:35:01.213379+0000 mgr.y (mgr.14144) 1061 : cluster [DBG] pgmap v1565: 158 pgs: 1 active+remapped+backfill_toofull, 1 active, 1 active+recovering+undersized+remapped, 1 peering, 2 active+recovering+undersized+degraded+remapped, 1 active+undersized+remapped, 151 active+clean; 680 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 3/1926 objects degraded (0.156%); 96/1926 objects misplaced (4.984%); 21 MiB/s, 66 objects/s recovering 2021-03-15T22:35:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:02 smithi125 conmon[56739]: cluster 2021-03-15T22:35:01 2021-03-15T22:35:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:02 smithi125 conmon[56739]: .450921+0000 mon.a (mon.0) 2377 : cluster [WRN] Health check failed: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull (PG_BACKFILL_FULL) 2021-03-15T22:35:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:02 smithi125 conmon[56739]: cluster 2021-03-15T22:35:01.450951+0000 mon.a (mon.0) 2378 : cluster [INF] Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2021-03-15T22:35:02.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:02 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:35:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:02 smithi125 conmon[56739]: 03-15T22:35:02.337550+0000 mon.a (mon.0) 2379 : cluster [DBG] 2021-03-15T22:35:02.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:02 smithi125 conmon[56739]: osdmap e947: 8 total, 8 up, 8 in 2021-03-15T22:35:02.867 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:02.889 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:02.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:02.890 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:02.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:03.016 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:03.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:03.017 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:03.046 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:03.118 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:03.140 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:03.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:03.142 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:03.243 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:03.266 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:03.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:03.267 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:03.342 INFO:tasks.rados.rados.0.smithi076.stdout:3578: done (2 left) 2021-03-15T22:35:03.343 INFO:tasks.rados.rados.0.smithi076.stdout:3579: snap_remove snap 381 2021-03-15T22:35:03.368 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:03.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:03.392 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:03.392 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:03.493 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:03.515 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:03.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:03.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:03.618 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:03.639 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:03.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:03.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:03.742 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:03.763 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:03.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:03.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:03.822 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:03.865 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:03.887 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:03.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:03.888 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:03.989 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:04.013 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:04.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:04.014 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:04.115 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:04.137 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:04.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:04.138 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:04.239 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:04.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:04.262 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:04.263 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:04.348 INFO:tasks.rados.rados.0.smithi076.stdout:3579: done (2 left) 2021-03-15T22:35:04.348 INFO:tasks.rados.rados.0.smithi076.stdout:3580: read oid 42 snap -1 2021-03-15T22:35:04.349 INFO:tasks.rados.rados.0.smithi076.stdout:3580: expect (ObjNum 1202 snap 418 seq_num 1202) 2021-03-15T22:35:04.349 INFO:tasks.rados.rados.0.smithi076.stdout:3581: write oid 32 current snap is 423 2021-03-15T22:35:04.349 INFO:tasks.rados.rados.0.smithi076.stdout:3581: seq_num 1221 ranges {521347=495509,1633455=564098} 2021-03-15T22:35:04.354 INFO:tasks.rados.rados.0.smithi076.stdout:3581: writing smithi07694878-32 from 521347 to 1016856 tid 1 2021-03-15T22:35:04.362 INFO:tasks.rados.rados.0.smithi076.stdout:3581: writing smithi07694878-32 from 1633455 to 2197553 tid 2 2021-03-15T22:35:04.362 INFO:tasks.rados.rados.0.smithi076.stdout:3582: snap_create 2021-03-15T22:35:04.362 INFO:tasks.rados.rados.0.smithi076.stdout:3577: finishing write tid 1 to smithi07694878-29 2021-03-15T22:35:04.363 INFO:tasks.rados.rados.0.smithi076.stdout:3577: finishing write tid 2 to smithi07694878-29 2021-03-15T22:35:04.363 INFO:tasks.rados.rados.0.smithi076.stdout:3577: finishing write tid 3 to smithi07694878-29 2021-03-15T22:35:04.363 INFO:tasks.rados.rados.0.smithi076.stdout:3577: finishing write tid 4 to smithi07694878-29 2021-03-15T22:35:04.363 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 3151 (ObjNum 1220 snap 423 seq_num 1220) dirty exists 2021-03-15T22:35:04.363 INFO:tasks.rados.rados.0.smithi076.stdout:3577: left oid 29 (ObjNum 1220 snap 423 seq_num 1220) 2021-03-15T22:35:04.364 INFO:tasks.rados.rados.0.smithi076.stdout:3576: finishing write tid 1 to smithi07694878-1 2021-03-15T22:35:04.364 INFO:tasks.rados.rados.0.smithi076.stdout:3576: finishing write tid 2 to smithi07694878-1 2021-03-15T22:35:04.364 INFO:tasks.rados.rados.0.smithi076.stdout:3576: finishing write tid 3 to smithi07694878-1 2021-03-15T22:35:04.364 INFO:tasks.rados.rados.0.smithi076.stdout:3576: finishing write tid 4 to smithi07694878-1 2021-03-15T22:35:04.365 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 2789 (ObjNum 1219 snap 423 seq_num 1219) dirty exists 2021-03-15T22:35:04.365 INFO:tasks.rados.rados.0.smithi076.stdout:3576: left oid 1 (ObjNum 1219 snap 423 seq_num 1219) 2021-03-15T22:35:04.365 INFO:tasks.rados.rados.0.smithi076.stdout:3580: expect (ObjNum 1202 snap 418 seq_num 1202) 2021-03-15T22:35:04.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:04.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:04.392 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:04.392 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:04.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:04 smithi076 conmon[56514]: cluster 2021-03-15T22:35:03.213924+0000 mgr.y (mgr.14144) 1062 : cluster [DBG] pgmap v1568: 158 pgs: 1 active+remapped+backfill_toofull, 1 active, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 675 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 550 KiB/s wr, 11 op/s; 1/1920 objects degraded (0.052%); 64/1920 objects misplaced (3.333%); 46 MiB/s, 117 objects/s recovering 2021-03-15T22:35:04.414 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:04 smithi076 conmon[56514]: cluster 2021-03-15T22:35:03.338626+0000 mon.a (mon.0) 2380 : cluster [DBG] osdmap e948: 8 total, 8 up, 8 in 2021-03-15T22:35:04.415 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:04 smithi076 conmon[56514]: cluster 2021-03-15T22:35:03.464471+0000 mon.a (mon.0) 2381 : cluster [WRN] Health check update: Degraded data redundancy: 1/1920 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:04.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:04 smithi076 conmon[62156]: cluster 2021-03-15T22:35:03.213924+0000 2021-03-15T22:35:04.415 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:04 smithi076 conmon[62156]: mgr.y (mgr.14144) 1062 : cluster [DBG] pgmap v1568: 158 pgs: 1 active+remapped+backfill_toofull, 1 active, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 675 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 550 KiB/s wr, 11 op/s; 1/1920 objects degraded (0.052%); 64/1920 objects misplaced (3.333%); 46 MiB/s, 117 objects/s recovering 2021-03-15T22:35:04.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:04 smithi076 conmon[62156]: cluster 2021-03-15T22:35:03.338626+0000 2021-03-15T22:35:04.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:04 smithi076 conmon[62156]: mon.a (mon.0) 2380 : cluster [DBG] osdmap e948: 8 total, 8 up, 8 in 2021-03-15T22:35:04.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:04 smithi076 conmon[62156]: cluster 2021-03-15T22:35:03. 2021-03-15T22:35:04.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:04 smithi076 conmon[62156]: 464471+0000 mon.a (mon.0) 2381 : cluster [WRN] 2021-03-15T22:35:04.416 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:04 smithi076 conmon[62156]: Health check update: Degraded data redundancy: 1/1920 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:04.493 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:04.497 INFO:tasks.rados.rados.0.smithi076.stdout:3581: finishing write tid 1 to smithi07694878-32 2021-03-15T22:35:04.497 INFO:tasks.rados.rados.0.smithi076.stdout:3581: finishing write tid 2 to smithi07694878-32 2021-03-15T22:35:04.497 INFO:tasks.rados.rados.0.smithi076.stdout:3581: finishing write tid 3 to smithi07694878-32 2021-03-15T22:35:04.498 INFO:tasks.rados.rados.0.smithi076.stdout:3581: finishing write tid 4 to smithi07694878-32 2021-03-15T22:35:04.498 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 32 v 3155 (ObjNum 1221 snap 423 seq_num 1221) dirty exists 2021-03-15T22:35:04.498 INFO:tasks.rados.rados.0.smithi076.stdout:3581: left oid 32 (ObjNum 1221 snap 423 seq_num 1221) 2021-03-15T22:35:04.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:04.519 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:04.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:04.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:04.642 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:04.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:04.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:04.745 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:04.766 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:04.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:04.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:04.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:04 smithi125 conmon[56739]: cluster 2021-03-15T22:35:03.213924+0000 mgr.y ( 2021-03-15T22:35:04.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:04 smithi125 conmon[56739]: mgr.14144) 1062 : cluster [DBG] pgmap v1568: 158 pgs: 1 active+remapped+backfill_toofull, 1 active, 1 active+recovering+undersized+remapped, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 675 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 550 KiB/s wr, 11 op/s; 1/1920 objects degraded (0.052%); 64/1920 objects misplaced (3.333%); 46 MiB/s, 117 objects/s recovering 2021-03-15T22:35:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:04 smithi125 conmon[56739]: cluster 2021-03-15T22:35:03.338626+0000 mon.a (mon.0) 2380 : cluster [DBG] 2021-03-15T22:35:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:04 smithi125 conmon[56739]: osdmap e948: 8 total, 8 up, 8 in 2021-03-15T22:35:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:04 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:35:04.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:04 smithi125 conmon[56739]: -03-15T22:35:03.464471+0000 mon.a ( 2021-03-15T22:35:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:04 smithi125 conmon[56739]: mon.0) 2381 : cluster [WRN] Health check update: Degraded data redundancy: 1/1920 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:04.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:04 smithi125 conmon[56739]: 2021-03-15T22:35:04.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:35:04.870 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:04.891 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:04.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:04.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:04.993 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:05.017 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:05.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:05.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:05.027 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:35:05.027 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:35:05.028 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 949, 2021-03-15T22:35:05.028 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:35:05.028 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2021-03-15T22:13:20.370795+0000", 2021-03-15T22:35:05.028 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2021-03-15T22:35:04.335601+0000", 2021-03-15T22:35:05.028 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2021-03-15T22:30:30.205134+0000", 2021-03-15T22:35:05.029 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2021-03-15T22:34:54.369110+0000", 2021-03-15T22:35:05.029 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-03-15T22:35:05.029 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5806080, 2021-03-15T22:35:05.029 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2021-03-15T22:35:05.030 INFO:teuthology.orchestra.run.smithi076.stdout: "nodeep-scrub", 2021-03-15T22:35:05.030 INFO:teuthology.orchestra.run.smithi076.stdout: "noscrub", 2021-03-15T22:35:05.030 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2021-03-15T22:35:05.030 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2021-03-15T22:35:05.030 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2021-03-15T22:35:05.031 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2021-03-15T22:35:05.031 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:35:05.031 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 20, 2021-03-15T22:35:05.031 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2021-03-15T22:35:05.031 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-03-15T22:35:05.032 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2021-03-15T22:35:05.032 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2021-03-15T22:35:05.032 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 7, 2021-03-15T22:35:05.032 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2021-03-15T22:35:05.032 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2021-03-15T22:35:05.033 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2021-03-15T22:35:05.033 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2021-03-15T22:35:05.033 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2021-03-15T22:35:05.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.033 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2021-03-15T22:35:05.034 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "device_health_metrics", 2021-03-15T22:35:05.034 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:16:34.817191+0000", 2021-03-15T22:35:05.034 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:35:05.034 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:35:05.035 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:35:05.035 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:35:05.035 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:35:05.035 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:35:05.035 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:35:05.036 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:35:05.036 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:35:05.036 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:35:05.036 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:35:05.036 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:35:05.037 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 1, 2021-03-15T22:35:05.037 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 1, 2021-03-15T22:35:05.037 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 1, 2021-03-15T22:35:05.037 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 1, 2021-03-15T22:35:05.038 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 1, 2021-03-15T22:35:05.038 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:35:05.038 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:35:05.038 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:35:05.038 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:35:05.039 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:35:05.039 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:35:05.039 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:35:05.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.039 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "23", 2021-03-15T22:35:05.040 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:35:05.040 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:35:05.040 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:35:05.040 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:35:05.040 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:35:05.041 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:35:05.041 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:35:05.041 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:35:05.041 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:35:05.041 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:35:05.042 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:35:05.042 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:35:05.042 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:35:05.042 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:35:05.042 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:35:05.043 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:35:05.043 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:35:05.043 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:35:05.043 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:35:05.043 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:35:05.044 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:35:05.044 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:35:05.044 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:35:05.044 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:35:05.045 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:35:05.045 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:35:05.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.045 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:35:05.045 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:35:05.046 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:35:05.046 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:35:05.046 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:35:05.046 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:35:05.046 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:35:05.047 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:35:05.047 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:35:05.047 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:35:05.047 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:35:05.047 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:35:05.048 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2021-03-15T22:35:05.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.048 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:35:05.048 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_devicehealth": {} 2021-03-15T22:35:05.048 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.049 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2021-03-15T22:35:05.049 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "iscsi", 2021-03-15T22:35:05.049 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:49.906400+0000", 2021-03-15T22:35:05.050 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:35:05.050 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:35:05.050 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:35:05.051 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:35:05.051 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:35:05.052 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:35:05.052 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:35:05.052 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:35:05.052 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:35:05.053 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:35:05.053 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:35:05.053 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:35:05.053 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 3, 2021-03-15T22:35:05.053 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 3, 2021-03-15T22:35:05.054 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 3, 2021-03-15T22:35:05.054 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 3, 2021-03-15T22:35:05.054 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 3, 2021-03-15T22:35:05.054 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:35:05.054 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:35:05.055 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:35:05.055 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:35:05.055 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:35:05.055 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:35:05.055 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:35:05.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.056 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "56", 2021-03-15T22:35:05.056 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:35:05.056 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:35:05.056 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:35:05.057 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:35:05.057 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:35:05.057 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:35:05.057 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:35:05.057 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:35:05.058 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:35:05.058 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:35:05.058 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:35:05.058 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:35:05.059 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:35:05.059 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:35:05.059 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:35:05.059 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:35:05.059 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:35:05.060 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:35:05.060 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:35:05.060 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:35:05.060 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:35:05.060 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:35:05.061 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:35:05.061 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:35:05.061 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:35:05.061 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:35:05.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.062 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:35:05.062 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:35:05.062 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:35:05.062 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:35:05.063 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:35:05.063 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:35:05.063 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:35:05.063 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:35:05.063 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:35:05.064 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:35:05.064 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:35:05.064 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:35:05.064 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:35:05.064 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2021-03-15T22:35:05.065 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.065 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2021-03-15T22:35:05.065 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".rgw.root", 2021-03-15T22:35:05.066 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:52.257970+0000", 2021-03-15T22:35:05.066 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:35:05.066 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:35:05.066 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:35:05.067 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:35:05.067 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:35:05.067 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:35:05.067 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:35:05.067 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:35:05.068 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:35:05.068 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:35:05.068 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:35:05.068 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:35:05.068 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:35:05.069 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:35:05.069 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:35:05.069 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:35:05.069 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:35:05.069 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:35:05.070 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:35:05.070 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:35:05.070 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:35:05.070 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:35:05.071 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:35:05.071 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:35:05.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.071 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "58", 2021-03-15T22:35:05.071 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:35:05.072 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:35:05.072 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:35:05.072 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:35:05.072 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:35:05.072 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:35:05.073 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:35:05.073 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:35:05.073 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:35:05.073 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:35:05.073 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:35:05.074 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:35:05.074 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:35:05.074 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:35:05.074 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:35:05.074 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:35:05.075 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:35:05.075 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:35:05.075 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:35:05.075 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:35:05.076 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:35:05.076 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:35:05.076 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:35:05.076 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:35:05.076 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:35:05.077 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:35:05.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.077 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:35:05.077 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:35:05.077 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:35:05.078 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:35:05.078 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:35:05.078 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:35:05.078 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:35:05.078 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:35:05.079 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:35:05.079 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:35:05.079 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:35:05.079 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:35:05.080 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:35:05.080 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:35:05.080 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.080 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.080 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.081 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2021-03-15T22:35:05.081 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.log", 2021-03-15T22:35:05.081 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:54.562498+0000", 2021-03-15T22:35:05.081 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:35:05.081 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:35:05.082 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:35:05.082 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:35:05.082 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:35:05.082 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:35:05.082 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:35:05.083 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:35:05.083 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:35:05.083 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:35:05.083 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:35:05.084 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:35:05.084 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:35:05.084 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:35:05.084 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:35:05.084 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:35:05.085 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:35:05.085 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:35:05.085 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:35:05.085 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:35:05.085 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:35:05.086 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:35:05.086 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:35:05.086 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:35:05.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.087 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "60", 2021-03-15T22:35:05.087 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:35:05.087 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:35:05.087 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:35:05.088 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:35:05.088 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:35:05.088 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:35:05.088 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:35:05.088 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:35:05.089 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:35:05.089 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:35:05.089 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:35:05.089 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:35:05.089 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:35:05.090 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:35:05.090 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:35:05.090 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:35:05.090 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:35:05.091 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:35:05.091 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:35:05.091 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:35:05.091 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:35:05.091 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:35:05.092 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:35:05.092 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:35:05.092 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:35:05.092 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:35:05.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.093 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:35:05.093 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:35:05.093 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:35:05.093 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:35:05.094 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:35:05.094 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:35:05.094 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:35:05.094 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:35:05.094 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:35:05.095 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:35:05.095 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:35:05.095 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:35:05.095 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:35:05.095 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:35:05.096 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.097 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 5, 2021-03-15T22:35:05.097 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.control", 2021-03-15T22:35:05.098 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:56.243806+0000", 2021-03-15T22:35:05.098 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:35:05.098 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:35:05.098 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:35:05.098 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:35:05.099 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:35:05.099 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:35:05.099 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:35:05.099 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:35:05.100 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:35:05.100 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:35:05.100 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:35:05.100 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:35:05.100 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:35:05.101 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:35:05.101 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:35:05.101 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:35:05.101 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:35:05.102 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:35:05.102 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "5.16", 2021-03-15T22:35:05.102 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 182, 2021-03-15T22:35:05.102 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 181, 2021-03-15T22:35:05.102 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 181, 2021-03-15T22:35:05.103 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:35:05.103 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:35:05.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.103 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "730", 2021-03-15T22:35:05.103 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:35:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "182", 2021-03-15T22:35:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "728", 2021-03-15T22:35:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:35:05.104 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:35:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:35:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:35:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:35:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:35:05.105 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:35:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:35:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:35:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:35:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:35:05.106 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:35:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:35:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:35:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:35:05.107 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:35:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:35:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:35:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:35:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:35:05.108 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:35:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:35:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:35:05.109 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:35:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:35:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:35:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:35:05.110 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:35:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:35:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:35:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:35:05.111 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:35:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:35:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:35:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:35:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:35:05.112 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:35:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 6, 2021-03-15T22:35:05.113 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.meta", 2021-03-15T22:35:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:58.730534+0000", 2021-03-15T22:35:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:35:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:35:05.114 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:35:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:35:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:35:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:35:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:35:05.115 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:35:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:35:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:35:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:35:05.116 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:35:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 42, 2021-03-15T22:35:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 42, 2021-03-15T22:35:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 42, 2021-03-15T22:35:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 42, 2021-03-15T22:35:05.117 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 42, 2021-03-15T22:35:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:35:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:35:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:35:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:35:05.118 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:35:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:35:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:35:05.119 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "920", 2021-03-15T22:35:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:35:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:35:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "918", 2021-03-15T22:35:05.120 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:35:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:35:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:35:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:35:05.121 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:35:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:35:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:35:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:35:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:35:05.122 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:35:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:35:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:35:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:35:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:35:05.123 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:35:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:35:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:35:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:35:05.124 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:35:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:35:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:35:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:35:05.125 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:35:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:35:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:35:05.126 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:35:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:35:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:35:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:35:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:35:05.127 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:35:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:35:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:35:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:35:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:35:05.128 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_bias": 4, 2021-03-15T22:35:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 8 2021-03-15T22:35:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:35:05.129 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:35:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.130 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:35:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2021-03-15T22:35:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:20:03.377294+0000", 2021-03-15T22:35:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2021-03-15T22:35:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-03-15T22:35:05.131 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:35:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:35:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:35:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:35:05.132 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:35:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:35:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:35:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:35:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:35:05.133 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:35:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2021-03-15T22:35:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2021-03-15T22:35:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2021-03-15T22:35:05.134 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2021-03-15T22:35:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2021-03-15T22:35:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:35:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:35:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:35:05.135 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:35:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:35:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:35:05.136 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:35:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "949", 2021-03-15T22:35:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:35:05.137 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:35:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:35:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:35:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:35:05.138 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 807, 2021-03-15T22:35:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 949, 2021-03-15T22:35:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:35:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:35:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:35:05.139 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:35:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:35:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:35:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:35:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:35:05.140 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:35:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:35:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:35:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:35:05.141 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:35:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:35:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:35:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:35:05.142 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:35:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:35:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:35:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:35:05.143 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:35:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:35:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:35:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:35:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:35:05.144 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:35:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:35:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:35:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:35:05.145 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:35:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:35:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:35:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2021-03-15T22:35:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.146 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:35:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:35:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.147 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:35:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd", 2021-03-15T22:35:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:35:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:35:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:35:05.148 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:35:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:35:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:35:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 8, 2021-03-15T22:35:05.149 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 942, 2021-03-15T22:35:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:35:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:35:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:35:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2021-03-15T22:35:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:35:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2021-03-15T22:35:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:35:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.152 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:35:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.153 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2021-03-15T22:35:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:35:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.154 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2021-03-15T22:35:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:35:05.155 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:35:05.156 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2021-03-15T22:35:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:35:05.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.158 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2021-03-15T22:35:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:35:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:35:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.160 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2021-03-15T22:35:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:35:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.161 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2021-03-15T22:35:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:35:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.162 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/188228176", 2021-03-15T22:35:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/188228176", 2021-03-15T22:35:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/188228176", 2021-03-15T22:35:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/188228176", 2021-03-15T22:35:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:35:05.163 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:35:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:35:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:35:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526", 2021-03-15T22:35:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:35:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:35:05.165 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0.8248748779296875, 2021-03-15T22:35:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.999237060546875, 2021-03-15T22:35:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:35:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:35:05.166 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 13, 2021-03-15T22:35:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 942, 2021-03-15T22:35:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:35:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:35:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:35:05.167 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2021-03-15T22:35:05.168 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:35:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2021-03-15T22:35:05.169 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:35:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.170 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:35:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.171 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2021-03-15T22:35:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:35:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.172 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2021-03-15T22:35:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:35:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.173 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:35:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.174 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2021-03-15T22:35:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:35:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.175 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2021-03-15T22:35:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:35:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:35:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.177 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2021-03-15T22:35:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:35:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.178 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2021-03-15T22:35:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:35:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/3924955423", 2021-03-15T22:35:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/3924955423", 2021-03-15T22:35:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/3924955423", 2021-03-15T22:35:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/3924955423", 2021-03-15T22:35:05.180 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:35:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:35:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:35:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:35:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05", 2021-03-15T22:35:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:35:05.182 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:35:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:35:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.53363037109375, 2021-03-15T22:35:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 18, 2021-03-15T22:35:05.183 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 489, 2021-03-15T22:35:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 681, 2021-03-15T22:35:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 943, 2021-03-15T22:35:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 490, 2021-03-15T22:35:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:35:05.184 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:35:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.185 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2021-03-15T22:35:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:35:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.186 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2021-03-15T22:35:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:35:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.187 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:35:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.188 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2021-03-15T22:35:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:35:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.189 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2021-03-15T22:35:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:35:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:35:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.191 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2021-03-15T22:35:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:35:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.192 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2021-03-15T22:35:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:35:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.193 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:35:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.194 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2021-03-15T22:35:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:35:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.195 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2021-03-15T22:35:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:35:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.196 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/1676192618", 2021-03-15T22:35:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/1676192618", 2021-03-15T22:35:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/1676192618", 2021-03-15T22:35:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/1676192618", 2021-03-15T22:35:05.197 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:35:05.198 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:35:05.198 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:35:05.198 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.198 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.199 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:35:05.199 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56", 2021-03-15T22:35:05.199 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:35:05.199 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:35:05.200 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0.7639923095703125, 2021-03-15T22:35:05.200 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:35:05.200 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:35:05.200 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:35:05.201 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 26, 2021-03-15T22:35:05.201 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 942, 2021-03-15T22:35:05.201 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:35:05.201 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:35:05.201 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:35:05.202 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.202 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.202 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.202 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2021-03-15T22:35:05.203 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:35:05.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.203 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.203 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2021-03-15T22:35:05.204 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:35:05.204 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.204 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.204 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.204 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:35:05.205 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.205 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.205 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2021-03-15T22:35:05.205 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:35:05.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.206 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.206 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2021-03-15T22:35:05.207 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:35:05.207 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.207 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.207 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:35:05.208 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.208 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.208 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2021-03-15T22:35:05.209 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:35:05.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.209 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.209 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2021-03-15T22:35:05.210 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:35:05.210 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.210 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.210 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:35:05.211 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.211 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.211 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.211 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2021-03-15T22:35:05.212 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:35:05.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.212 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.212 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2021-03-15T22:35:05.213 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:35:05.213 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.213 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.213 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.213 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/3960968636", 2021-03-15T22:35:05.214 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/3960968636", 2021-03-15T22:35:05.214 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/3960968636", 2021-03-15T22:35:05.214 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/3960968636", 2021-03-15T22:35:05.214 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:35:05.215 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:35:05.215 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:35:05.215 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.215 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.216 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:35:05.216 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237", 2021-03-15T22:35:05.216 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:35:05.216 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:35:05.217 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:35:05.217 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:35:05.217 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:35:05.217 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:35:05.218 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 31, 2021-03-15T22:35:05.218 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 939, 2021-03-15T22:35:05.218 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:35:05.218 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:35:05.219 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:35:05.219 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.219 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.219 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6800", 2021-03-15T22:35:05.220 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:35:05.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.220 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.220 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6801", 2021-03-15T22:35:05.221 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:35:05.221 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.221 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.221 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:35:05.222 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.222 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.222 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6802", 2021-03-15T22:35:05.223 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:35:05.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.223 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.223 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.223 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6803", 2021-03-15T22:35:05.224 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:35:05.224 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.224 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.224 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.225 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:35:05.225 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.225 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.225 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.225 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6806", 2021-03-15T22:35:05.226 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:35:05.226 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.226 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.226 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.226 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6807", 2021-03-15T22:35:05.227 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:35:05.227 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.227 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.227 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.228 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:35:05.228 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.228 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.228 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.228 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6804", 2021-03-15T22:35:05.229 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:35:05.229 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.229 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.229 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.230 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6805", 2021-03-15T22:35:05.230 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:35:05.230 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.230 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.230 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.231 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6801/37386122", 2021-03-15T22:35:05.231 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6803/37386122", 2021-03-15T22:35:05.231 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6807/37386122", 2021-03-15T22:35:05.231 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6805/37386122", 2021-03-15T22:35:05.232 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:35:05.232 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:35:05.232 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:35:05.232 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.232 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.233 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.233 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:35:05.233 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2", 2021-03-15T22:35:05.233 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:35:05.233 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:35:05.234 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:35:05.234 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:35:05.234 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 38, 2021-03-15T22:35:05.234 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 539, 2021-03-15T22:35:05.235 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 616, 2021-03-15T22:35:05.235 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 940, 2021-03-15T22:35:05.235 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 540, 2021-03-15T22:35:05.235 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:35:05.235 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:35:05.236 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.236 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.236 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.236 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6808", 2021-03-15T22:35:05.236 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:35:05.237 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.237 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.237 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.237 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6809", 2021-03-15T22:35:05.238 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:35:05.238 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.238 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.238 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.239 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:35:05.239 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.239 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.239 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.240 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6810", 2021-03-15T22:35:05.240 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:35:05.240 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.240 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.240 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.241 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6811", 2021-03-15T22:35:05.241 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:35:05.241 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.241 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.242 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.242 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:35:05.242 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.242 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.242 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.243 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6814", 2021-03-15T22:35:05.243 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:35:05.243 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.243 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.244 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6815", 2021-03-15T22:35:05.244 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:35:05.244 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.244 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.244 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.245 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:35:05.245 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.245 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.245 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.245 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6812", 2021-03-15T22:35:05.246 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:35:05.246 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.246 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.247 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6813", 2021-03-15T22:35:05.247 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:35:05.247 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.247 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.248 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6809/978017133", 2021-03-15T22:35:05.248 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6811/978017133", 2021-03-15T22:35:05.248 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6815/978017133", 2021-03-15T22:35:05.248 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6813/978017133", 2021-03-15T22:35:05.248 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:35:05.249 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:35:05.249 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:35:05.249 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.250 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:35:05.250 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe", 2021-03-15T22:35:05.250 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:35:05.250 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:35:05.251 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:35:05.251 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:35:05.251 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:35:05.251 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:35:05.252 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 45, 2021-03-15T22:35:05.252 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 942, 2021-03-15T22:35:05.252 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:35:05.252 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:35:05.252 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:35:05.253 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.253 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.253 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6816", 2021-03-15T22:35:05.253 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:35:05.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.254 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.254 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6817", 2021-03-15T22:35:05.255 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:35:05.255 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.255 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.256 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:35:05.256 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.256 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.256 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6818", 2021-03-15T22:35:05.257 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:35:05.257 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.257 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.257 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.258 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6819", 2021-03-15T22:35:05.258 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:35:05.258 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.258 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.259 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:35:05.259 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.259 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.260 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6822", 2021-03-15T22:35:05.260 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:35:05.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.260 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.261 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6823", 2021-03-15T22:35:05.261 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:35:05.261 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.261 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.262 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:35:05.262 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.262 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.262 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.263 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6820", 2021-03-15T22:35:05.263 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:35:05.263 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.263 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.264 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6821", 2021-03-15T22:35:05.264 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:35:05.264 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.264 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.265 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6817/171378275", 2021-03-15T22:35:05.265 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6819/171378275", 2021-03-15T22:35:05.265 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6823/171378275", 2021-03-15T22:35:05.265 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6821/171378275", 2021-03-15T22:35:05.265 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:35:05.266 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:35:05.266 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:35:05.266 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.267 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.267 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:35:05.267 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614", 2021-03-15T22:35:05.267 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:35:05.267 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:35:05.268 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:35:05.268 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:35:05.268 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:35:05.268 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:35:05.268 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 52, 2021-03-15T22:35:05.269 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 947, 2021-03-15T22:35:05.269 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:35:05.269 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:35:05.269 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:35:05.269 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.270 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.270 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6824", 2021-03-15T22:35:05.270 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:35:05.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.271 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.271 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6825", 2021-03-15T22:35:05.271 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:35:05.271 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.272 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.272 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:35:05.272 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.273 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.273 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6826", 2021-03-15T22:35:05.273 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:35:05.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.274 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.274 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6827", 2021-03-15T22:35:05.274 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:35:05.275 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.275 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.275 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:35:05.275 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.276 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.276 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6830", 2021-03-15T22:35:05.276 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:35:05.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.277 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.277 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6831", 2021-03-15T22:35:05.277 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:35:05.278 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.278 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.278 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:35:05.278 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:35:05.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.279 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:35:05.279 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6828", 2021-03-15T22:35:05.279 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:35:05.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.280 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:35:05.280 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6829", 2021-03-15T22:35:05.280 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:35:05.281 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.281 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.281 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6825/652003948", 2021-03-15T22:35:05.281 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6827/652003948", 2021-03-15T22:35:05.282 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6831/652003948", 2021-03-15T22:35:05.282 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6829/652003948", 2021-03-15T22:35:05.282 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:35:05.282 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:35:05.282 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:35:05.283 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.283 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.283 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:35:05.283 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2021-03-15T22:35:05.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.284 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:35:05.284 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:35:05.284 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:35:05.284 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:35:05.284 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:35:05.285 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:35:05.285 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:34:58.945593+0000", 2021-03-15T22:35:05.285 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:35:05.285 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.285 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.286 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:35:05.286 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:35:05.286 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:35:05.286 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:35:05.286 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:35:05.287 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:35:05.287 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:34:15.283244+0000", 2021-03-15T22:35:05.287 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:35:05.287 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.288 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:35:05.288 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:35:05.288 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:35:05.288 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:35:05.289 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:35:05.289 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:35:05.289 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:34:31.974617+0000", 2021-03-15T22:35:05.289 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 490 2021-03-15T22:35:05.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.290 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:35:05.290 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:35:05.290 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:35:05.291 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:35:05.291 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:35:05.291 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:35:05.291 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:34:45.543749+0000", 2021-03-15T22:35:05.292 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:35:05.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.292 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:35:05.292 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:35:05.293 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:35:05.293 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:35:05.293 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:35:05.293 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:35:05.294 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:34:04.218583+0000", 2021-03-15T22:35:05.294 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:35:05.294 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.294 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:35:05.295 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:35:05.295 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:35:05.295 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:35:05.295 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:35:05.295 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:35:05.296 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:34:26.565557+0000", 2021-03-15T22:35:05.296 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 540 2021-03-15T22:35:05.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.297 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:35:05.297 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:35:05.297 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:35:05.297 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:35:05.297 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:35:05.298 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:35:05.298 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:34:38.932973+0000", 2021-03-15T22:35:05.298 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:35:05.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.299 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.299 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:35:05.299 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:35:05.299 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:35:05.299 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:35:05.300 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:35:05.300 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:35:05.300 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:34:56.469328+0000", 2021-03-15T22:35:05.300 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:35:05.300 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.301 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:35:05.301 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [ 2021-03-15T22:35:05.301 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.301 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.3", 2021-03-15T22:35:05.301 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:35:05.302 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:35:05.302 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:35:05.302 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2021-03-15T22:35:05.302 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.303 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.303 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:35:05.303 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2021-03-15T22:35:05.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.303 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.10", 2021-03-15T22:35:05.304 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:35:05.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.304 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 7, 2021-03-15T22:35:05.304 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 3 2021-03-15T22:35:05.305 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.305 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.305 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.17", 2021-03-15T22:35:05.306 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:35:05.306 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.306 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:35:05.306 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 6 2021-03-15T22:35:05.306 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.307 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.307 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.307 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.1b", 2021-03-15T22:35:05.308 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:35:05.308 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.308 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 0, 2021-03-15T22:35:05.308 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 3 2021-03-15T22:35:05.308 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.309 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.309 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.1f", 2021-03-15T22:35:05.310 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:35:05.310 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.310 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:35:05.310 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 3 2021-03-15T22:35:05.310 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.311 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.311 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.f", 2021-03-15T22:35:05.311 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:35:05.312 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.312 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:35:05.312 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 0 2021-03-15T22:35:05.312 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.312 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.313 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.313 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:35:05.313 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [ 2021-03-15T22:35:05.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.313 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.d", 2021-03-15T22:35:05.314 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:35:05.314 INFO:teuthology.orchestra.run.smithi076.stdout: 0, 2021-03-15T22:35:05.314 INFO:teuthology.orchestra.run.smithi076.stdout: 7, 2021-03-15T22:35:05.314 INFO:teuthology.orchestra.run.smithi076.stdout: 5 2021-03-15T22:35:05.314 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.315 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.315 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.0", 2021-03-15T22:35:05.315 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:35:05.316 INFO:teuthology.orchestra.run.smithi076.stdout: 5, 2021-03-15T22:35:05.316 INFO:teuthology.orchestra.run.smithi076.stdout: 3 2021-03-15T22:35:05.316 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.316 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.316 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:35:05.317 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2021-03-15T22:35:05.317 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": { 2021-03-15T22:35:05.317 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:35:05.317 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4087324200": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:35:05.317 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4141143304": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:35:05.318 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:35:05.318 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2779879991": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:35:05.318 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:35:05.318 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:35:05.319 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2483102204": "2021-03-16T22:13:41.603652+0000" 2021-03-15T22:35:05.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.319 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2021-03-15T22:35:05.319 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2021-03-15T22:35:05.320 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2021-03-15T22:35:05.320 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2021-03-15T22:35:05.320 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2021-03-15T22:35:05.320 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2021-03-15T22:35:05.320 INFO:teuthology.orchestra.run.smithi076.stdout: "ruleset-failure-domain": "osd", 2021-03-15T22:35:05.321 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2021-03-15T22:35:05.321 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.321 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2021-03-15T22:35:05.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.322 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:35:05.322 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:35:05.322 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.322 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 671, 2021-03-15T22:35:05.323 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:35:05.323 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.323 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.323 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 729, 2021-03-15T22:35:05.323 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:35:05.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.324 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 732, 2021-03-15T22:35:05.324 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:35:05.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.325 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 734, 2021-03-15T22:35:05.325 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:35:05.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.326 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 757, 2021-03-15T22:35:05.326 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:35:05.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.327 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 764, 2021-03-15T22:35:05.327 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:35:05.327 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:05.327 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.327 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 783, 2021-03-15T22:35:05.328 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:35:05.328 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.328 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.328 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.328 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:35:05.329 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [ 2021-03-15T22:35:05.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.329 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:35:05.329 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:35:05.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:05.330 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 732, 2021-03-15T22:35:05.330 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:35:05.330 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.330 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:05.331 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.331 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:35:05.331 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [], 2021-03-15T22:35:05.331 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2021-03-15T22:35:05.331 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2021-03-15T22:35:05.332 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2021-03-15T22:35:05.332 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2021-03-15T22:35:05.332 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2021-03-15T22:35:05.332 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2021-03-15T22:35:05.332 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2021-03-15T22:35:05.333 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2021-03-15T22:35:05.333 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:05.333 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:35:05.334 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:05.357 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:05.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:05.359 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:05.361 INFO:tasks.rados.rados.0.smithi076.stdout:3576: done (4 left) 2021-03-15T22:35:05.361 INFO:tasks.rados.rados.0.smithi076.stdout:3577: done (3 left) 2021-03-15T22:35:05.362 INFO:tasks.rados.rados.0.smithi076.stdout:3580: done (2 left) 2021-03-15T22:35:05.362 INFO:tasks.rados.rados.0.smithi076.stdout:3581: done (1 left) 2021-03-15T22:35:05.362 INFO:tasks.rados.rados.0.smithi076.stdout:3582: done (0 left) 2021-03-15T22:35:05.362 INFO:tasks.rados.rados.0.smithi076.stdout:3583: delete oid 20 current snap is 424 2021-03-15T22:35:05.364 INFO:tasks.rados.rados.0.smithi076.stdout:3583: done (0 left) 2021-03-15T22:35:05.364 INFO:tasks.rados.rados.0.smithi076.stdout:3584: read oid 46 snap -1 2021-03-15T22:35:05.364 INFO:tasks.rados.rados.0.smithi076.stdout:3584: expect (ObjNum 1195 snap 417 seq_num 1195) 2021-03-15T22:35:05.365 INFO:tasks.rados.rados.0.smithi076.stdout:3585: delete oid 9 current snap is 424 2021-03-15T22:35:05.370 INFO:tasks.rados.rados.0.smithi076.stdout:3585: done (1 left) 2021-03-15T22:35:05.370 INFO:tasks.rados.rados.0.smithi076.stdout:3586: rollback oid 16 current snap is 424 2021-03-15T22:35:05.370 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 16 to 321 2021-03-15T22:35:05.370 INFO:tasks.rados.rados.0.smithi076.stdout:3587: copy_from oid 45 from oid 37 current snap is 424 2021-03-15T22:35:05.371 INFO:tasks.rados.rados.0.smithi076.stdout:3588: snap_remove snap 384 2021-03-15T22:35:05.459 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:05.482 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:05.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:05.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:05.584 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:05.607 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:05.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:05.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:05.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:05.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:05.732 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:05.732 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:05.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:05 smithi076 conmon[56514]: cluster 2021-03-15T22:35:04.340227+0000 mon.a (mon.0) 2382 : cluster [DBG] osdmap e949: 8 total, 8 up, 8 in 2021-03-15T22:35:05.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:05 smithi076 conmon[56514]: audit 2021-03-15T22:35:04.676876+0000 mon.a (mon.0) 2383 : audit [DBG] from='client.? 172.21.15.76:0/3301031263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:35:05.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:05 smithi076 conmon[62156]: cluster 2021-03-15T22:35:04.340227+0000 mon.a (mon.0) 2382 2021-03-15T22:35:05.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:05 smithi076 conmon[62156]: : cluster [DBG] osdmap e949: 8 total, 8 up, 8 in 2021-03-15T22:35:05.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:05 smithi076 conmon[62156]: audit 2021-03-15T22:35:04.676876+0000 mon.a (mon.0) 2383 : audit [DBG] from='client.? 172.21.15.76:0/3301031263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:35:05.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:05 smithi125 conmon[56739]: cluster 2021-03-15T22:35:04.340227+0000 2021-03-15T22:35:05.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:05 smithi125 conmon[56739]: mon.a (mon.0) 2382 : cluster [DBG] osdmap e949: 8 total, 8 up, 8 in 2021-03-15T22:35:05.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:05 smithi125 conmon[56739]: audit 2021-03-15T22:35: 2021-03-15T22:35:05.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:05 smithi125 conmon[56739]: 04.676876+0000 mon.a (mon.0) 2383 : audit [DBG] from='client.? 172.21.15.76:0/3301031263' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:35:05.834 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:05.855 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:05.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:05.857 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:05.958 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:05.980 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:05.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:05.982 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:06.083 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:06.104 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:06.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:06.106 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:06.207 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:06.228 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:06.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:06.229 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:06.330 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:06.355 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:06.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:06.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:06.364 INFO:tasks.rados.rados.0.smithi076.stdout:3588: done (3 left) 2021-03-15T22:35:06.364 INFO:tasks.rados.rados.0.smithi076.stdout:3589: rollback oid 41 current snap is 424 2021-03-15T22:35:06.364 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 41 to 392 2021-03-15T22:35:06.365 INFO:tasks.rados.rados.0.smithi076.stdout:3584: expect (ObjNum 1195 snap 417 seq_num 1195) 2021-03-15T22:35:06.448 INFO:tasks.rados.rados.0.smithi076.stdout:3586: finishing rollback tid 0 to smithi07694878-16 2021-03-15T22:35:06.448 INFO:tasks.rados.rados.0.smithi076.stdout:3586: finishing rollback tid 1 to smithi07694878-16 2021-03-15T22:35:06.448 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 2989 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:35:06.448 INFO:tasks.rados.rados.0.smithi076.stdout:3587: finishing copy_from to smithi07694878-45 2021-03-15T22:35:06.449 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 3076 (ObjNum 1186 snap 414 seq_num 1186) dirty exists 2021-03-15T22:35:06.449 INFO:tasks.rados.rados.0.smithi076.stdout:3587: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:35:06.449 INFO:tasks.rados.rados.0.smithi076.stdout:3589: finishing rollback tid 0 to smithi07694878-41 2021-03-15T22:35:06.449 INFO:tasks.rados.rados.0.smithi076.stdout:3589: finishing rollback tid 1 to smithi07694878-41 2021-03-15T22:35:06.450 INFO:tasks.rados.rados.0.smithi076.stdout:3589: finishing rollback tid 2 to smithi07694878-41 2021-03-15T22:35:06.450 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 2427 (ObjNum 1076 snap 366 seq_num 1076) dirty exists 2021-03-15T22:35:06.450 INFO:tasks.rados.rados.0.smithi076.stdout:3584: done (3 left) 2021-03-15T22:35:06.450 INFO:tasks.rados.rados.0.smithi076.stdout:3586: done (2 left) 2021-03-15T22:35:06.451 INFO:tasks.rados.rados.0.smithi076.stdout:3587: done (1 left) 2021-03-15T22:35:06.451 INFO:tasks.rados.rados.0.smithi076.stdout:3589: done (0 left) 2021-03-15T22:35:06.451 INFO:tasks.rados.rados.0.smithi076.stdout:3590: copy_from oid 46 from oid 26 current snap is 424 2021-03-15T22:35:06.451 INFO:tasks.rados.rados.0.smithi076.stdout:3591: write (excl) oid 20 current snap is 424 2021-03-15T22:35:06.451 INFO:tasks.rados.rados.0.smithi076.stdout:3591: seq_num 1224 ranges {571768=437609,1808901=711425} 2021-03-15T22:35:06.453 INFO:tasks.rados.rados.0.smithi076.stdout:3591: writing smithi07694878-20 from 571768 to 1009377 tid 1 2021-03-15T22:35:06.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:06.462 INFO:tasks.rados.rados.0.smithi076.stdout:3591: writing smithi07694878-20 from 1808901 to 2520326 tid 2 2021-03-15T22:35:06.462 INFO:tasks.rados.rados.0.smithi076.stdout:3592: rollback oid 25 current snap is 424 2021-03-15T22:35:06.463 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 25 to 345 2021-03-15T22:35:06.463 INFO:tasks.rados.rados.0.smithi076.stdout:3590: finishing copy_from to smithi07694878-46 2021-03-15T22:35:06.463 INFO:tasks.rados.rados.0.smithi076.stdout:3590: got expected ENOENT (src dne) 2021-03-15T22:35:06.463 INFO:tasks.rados.rados.0.smithi076.stdout:3590: finishing copy_from racing read to smithi07694878-46 2021-03-15T22:35:06.464 INFO:tasks.rados.rados.0.smithi076.stdout:3591: finishing write tid 1 to smithi07694878-20 2021-03-15T22:35:06.464 INFO:tasks.rados.rados.0.smithi076.stdout:3590: done (2 left) 2021-03-15T22:35:06.464 INFO:tasks.rados.rados.0.smithi076.stdout:3593: write (excl) oid 21 current snap is 424 2021-03-15T22:35:06.464 INFO:tasks.rados.rados.0.smithi076.stdout:3593: seq_num 1225 ranges {588575=512858,1867383=455083} 2021-03-15T22:35:06.468 INFO:tasks.rados.rados.0.smithi076.stdout:3593: writing smithi07694878-21 from 588575 to 1101433 tid 1 2021-03-15T22:35:06.476 INFO:tasks.thrashosds.thrasher:j is {'epoch': 949, 'fsid': '8a572802-85db-11eb-9091-001a4aab830c', 'created': '2021-03-15T22:13:20.370795+0000', 'modified': '2021-03-15T22:35:04.335601+0000', 'last_up_change': '2021-03-15T22:30:30.205134+0000', 'last_in_change': '2021-03-15T22:34:54.369110+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-15T22:16:34.817191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-15T22:17:49.906400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-15T22:17:52.257970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '58', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-15T22:17:54.562498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-15T22:17:56.243806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 182, 'last_epoch_started': 181, 'last_epoch_clean': 181, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '730', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '182', 'last_force_op_resend_preluminous': '728', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-15T22:17:58.730534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 42, 'pg_placement_num': 42, 'pg_placement_num_target': 42, 'pg_num_target': 42, 'pg_num_pending': 42, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '920', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '918', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-03-15T22:20:03.377294+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '949', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 807, 'snap_epoch': 949, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 942, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 188228176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 188228176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 188228176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 188228176}]}, 'public_addr': '172.21.15.76:6803/188228176', 'cluster_addr': '172.21.15.76:6805/188228176', 'heartbeat_back_addr': '172.21.15.76:6809/188228176', 'heartbeat_front_addr': '172.21.15.76:6807/188228176', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '97bd3ffc-15c8-4a6f-b37a-217813def526', 'up': 1, 'in': 1, 'weight': 0.8248748779296875, 'primary_affinity': 0.999237060546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 942, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 3924955423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 3924955423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 3924955423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 3924955423}]}, 'public_addr': '172.21.15.76:6811/3924955423', 'cluster_addr': '172.21.15.76:6813/3924955423', 'heartbeat_back_addr': '172.21.15.76:6817/3924955423', 'heartbeat_front_addr': '172.21.15.76:6815/3924955423', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a7f5f115-e69e-47d9-b6c3-ff386ddfaa05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.53363037109375, 'last_clean_begin': 18, 'last_clean_end': 489, 'up_from': 681, 'up_thru': 943, 'down_at': 490, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 1676192618}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 1676192618}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 1676192618}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 1676192618}]}, 'public_addr': '172.21.15.76:6819/1676192618', 'cluster_addr': '172.21.15.76:6821/1676192618', 'heartbeat_back_addr': '172.21.15.76:6825/1676192618', 'heartbeat_front_addr': '172.21.15.76:6823/1676192618', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a0bb7f88-0114-47cb-a82c-d8fb002aac56', 'up': 1, 'in': 1, 'weight': 0.7639923095703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 942, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 3960968636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 3960968636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 3960968636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 3960968636}]}, 'public_addr': '172.21.15.76:6827/3960968636', 'cluster_addr': '172.21.15.76:6829/3960968636', 'heartbeat_back_addr': '172.21.15.76:6833/3960968636', 'heartbeat_front_addr': '172.21.15.76:6831/3960968636', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b650611c-6101-47e8-9ca9-94e02cf8b237', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 939, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6800', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6801', 'nonce': 37386122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6802', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6803', 'nonce': 37386122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6806', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6807', 'nonce': 37386122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6804', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6805', 'nonce': 37386122}]}, 'public_addr': '172.21.15.125:6801/37386122', 'cluster_addr': '172.21.15.125:6803/37386122', 'heartbeat_back_addr': '172.21.15.125:6807/37386122', 'heartbeat_front_addr': '172.21.15.125:6805/37386122', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7d6ceff2-7093-4ab9-8b96-ade448093bf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 38, 'last_clean_end': 539, 'up_from': 616, 'up_thru': 940, 'down_at': 540, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6808', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6809', 'nonce': 978017133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6810', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6811', 'nonce': 978017133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6814', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6815', 'nonce': 978017133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6812', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6813', 'nonce': 978017133}]}, 'public_addr': '172.21.15.125:6809/978017133', 'cluster_addr': '172.21.15.125:6811/978017133', 'heartbeat_back_addr': '172.21.15.125:6815/978017133', 'heartbeat_front_addr': '172.21.15.125:6813/978017133', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98b72022-a9a7-4ee2-89a1-cac037de7afe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 942, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6816', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6817', 'nonce': 171378275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6818', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6819', 'nonce': 171378275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6822', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6823', 'nonce': 171378275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6820', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6821', 'nonce': 171378275}]}, 'public_addr': '172.21.15.125:6817/171378275', 'cluster_addr': '172.21.15.125:6819/171378275', 'heartbeat_back_addr': '172.21.15.125:6823/171378275', 'heartbeat_front_addr': '172.21.15.125:6821/171378275', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f99ea650-b2dc-4d87-a44c-56a897edd614', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 947, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6824', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6825', 'nonce': 652003948}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6826', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6827', 'nonce': 652003948}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6830', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6831', 'nonce': 652003948}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6828', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6829', 'nonce': 652003948}]}, 'public_addr': '172.21.15.125:6825/652003948', 'cluster_addr': '172.21.15.125:6827/652003948', 'heartbeat_back_addr': '172.21.15.125:6831/652003948', 'heartbeat_front_addr': '172.21.15.125:6829/652003948', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:34:58.945593+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:34:15.283244+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:34:31.974617+0000', 'dead_epoch': 490}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:34:45.543749+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:34:04.218583+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:34:26.565557+0000', 'dead_epoch': 540}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:34:38.932973+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:34:56.469328+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '7.3', 'osds': [1, 6, 2]}], 'pg_upmap_items': [{'pgid': '5.10', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.17', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '5.1b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '5.1f', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '6.f', 'mappings': [{'from': 4, 'to': 0}]}], 'pg_temp': [{'pgid': '5.d', 'osds': [0, 7, 5]}, {'pgid': '7.0', 'osds': [5, 3]}], 'primary_temp': [], 'blocklist': {'172.21.15.76:6800/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4087324200': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4141143304': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:6801/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/2779879991': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6801/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6800/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:0/2483102204': '2021-03-16T22:13:41.603652+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 671, 'length': 1}, {'begin': 729, 'length': 1}, {'begin': 732, 'length': 1}, {'begin': 734, 'length': 1}, {'begin': 757, 'length': 1}, {'begin': 764, 'length': 1}, {'begin': 783, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 732, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-03-15T22:35:06.476 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:35:06.477 INFO:tasks.rados.rados.0.smithi076.stdout:3593: writing smithi07694878-21 from 1867383 to 2322466 tid 2 2021-03-15T22:35:06.477 INFO:tasks.rados.rados.0.smithi076.stdout:3594: snap_remove snap 368 2021-03-15T22:35:06.480 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:06.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:06.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:06.503 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:06.583 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:06.604 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:06.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:06.606 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:06.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: cluster 2021-03-15T22:35: 2021-03-15T22:35:06.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: 05.067009+0000 osd.4 (osd.4) 190 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:06.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:35:06.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: T22:35:05.068361+0000 osd.4 (osd. 2021-03-15T22:35:06.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: 4) 191 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:06.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: 2021-03-15T22:35:06.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: cluster 2021-03-15T22:35:06.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: 2021-03-15T22:35:05 2021-03-15T22:35:06.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: .214436+0000 mgr.y (mgr. 2021-03-15T22:35:06.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: 14144) 1063 : cluster [DBG] pgmap v1571: 158 pgs: 2 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 680 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 594 KiB/s wr, 6 op/s; 1/1920 objects degraded (0.052%); 46/1920 objects misplaced (2.396%); 33 MiB/s, 75 objects/s recovering 2021-03-15T22:35:06.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: 2021-03-15T22:35:06.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:35:06.642 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: -03-15T22:35:05.357511+0000 mon.a (mon.0) 2021-03-15T22:35:06.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[62156]: 2384 : cluster [DBG] osdmap e950: 8 total, 8 up, 8 in 2021-03-15T22:35:06.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[56514]: cluster 2021-03-15T22:35:05.067009+0000 2021-03-15T22:35:06.643 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[56514]: osd.4 (osd.4) 190 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:06.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[56514]: cluster 2021-03-15T22:35:05.068361+0000 osd.4 (osd.4) 191 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:06.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[56514]: cluster 2021-03-15T22:35:05.214436+0000 mgr.y (mgr.14144) 1063 : cluster [DBG] pgmap v1571: 158 pgs: 2 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 680 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 594 KiB/s wr, 6 op/s; 1/1920 objects degraded (0.052%); 46/1920 objects misplaced (2.396%); 33 MiB/s, 75 objects/s recovering 2021-03-15T22:35:06.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[56514]: cluster 2021-03-15T22:35:05.357511 2021-03-15T22:35:06.644 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:06 smithi076 conmon[56514]: +0000 mon.a (mon.0) 2384 : cluster [DBG] osdmap e950: 8 total, 8 up, 8 in 2021-03-15T22:35:06.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:06.728 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:06.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:06.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:06.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:06 smithi125 conmon[56739]: cluster 2021-03-15T22:35:05.067009+0000 osd.4 (osd.4) 2021-03-15T22:35:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:06 smithi125 conmon[56739]: 190 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:06 smithi125 conmon[56739]: cluster 2021-03-15T22:35:05.068361+0000 osd.4 (osd.4) 191 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:06 smithi125 conmon[56739]: 2021-03-15T22:35:06.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:06 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:35:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:06 smithi125 conmon[56739]: 22:35:05.214436+0000 mgr.y (mgr.14144) 1063 : cluster 2021-03-15T22:35:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:06 smithi125 conmon[56739]: [DBG] pgmap v1571: 158 pgs: 2 active+clean+snaptrim, 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 680 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 594 KiB/s wr, 6 op/s; 1/1920 objects degraded (0.052%); 46/1920 objects misplaced (2.396%); 33 MiB/s, 75 objects/s recovering 2021-03-15T22:35:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:06 smithi125 conmon[56739]: cluster 2021-03-15T22:35:05 2021-03-15T22:35:06.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:06 smithi125 conmon[56739]: .357511+0000 mon.a (mon.0) 2384 : cluster 2021-03-15T22:35:06.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:06 smithi125 conmon[56739]: [DBG] osdmap e950: 8 total, 8 up, 8 in 2021-03-15T22:35:06.831 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:06.854 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:06.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:06.855 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:06.904 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:06.945 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:35:06.946 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:35:06.946 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:35:06.946 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:35:06.946 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.947 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4284 7.b 7:d4368cae:::smithi07694878-18:head [write 3203532~509848] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:35:06.947 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:35:06.947 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 546.77914411799998, 2021-03-15T22:35:06.947 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068311582999999995, 2021-03-15T22:35:06.948 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:06.948 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:06.948 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:06.948 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:06.948 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:06.949 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4284 2021-03-15T22:35:06.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.949 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:06.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.949 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:06.950 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:35:06.950 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:06.950 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.950 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.951 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:06.951 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162778+0000", 2021-03-15T22:35:06.951 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015451700000000001 2021-03-15T22:35:06.951 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.951 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.952 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:06.952 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162933+0000", 2021-03-15T22:35:06.952 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.25e-06 2021-03-15T22:35:06.952 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162937+0000", 2021-03-15T22:35:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2824e-05 2021-03-15T22:35:06.953 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162960+0000", 2021-03-15T22:35:06.954 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.072e-06 2021-03-15T22:35:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162968+0000", 2021-03-15T22:35:06.955 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7661999999999999e-05 2021-03-15T22:35:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.956 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163016+0000", 2021-03-15T22:35:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108743 2021-03-15T22:35:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.957 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163124+0000", 2021-03-15T22:35:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00021732500000000001 2021-03-15T22:35:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:35:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163342+0000", 2021-03-15T22:35:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.043853354999999997 2021-03-15T22:35:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.207195+0000", 2021-03-15T22:35:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055661469999999996 2021-03-15T22:35:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.212761+0000", 2021-03-15T22:35:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018240684 2021-03-15T22:35:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:35:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231002+0000", 2021-03-15T22:35:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6122000000000003e-05 2021-03-15T22:35:06.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231048+0000", 2021-03-15T22:35:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1882000000000003e-05 2021-03-15T22:35:06.963 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.964 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:06.964 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231090+0000", 2021-03-15T22:35:06.964 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068311582999999995 2021-03-15T22:35:06.964 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:06.965 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:06.965 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:06.965 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.966 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4285 7.b 7:d4368cae:::smithi07694878-18:head [setxattr _header (58) in=7b,truncate 3865709] snapc 153=[153,151,14d,148,143,141,13f,13e,133,131,130,12f,12e,128,126,123,117,114,10c,102,100,e7] ondisk+write+known_if_redirected e445)", 2021-03-15T22:35:06.966 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:35:06.966 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 546.77893964600003, 2021-03-15T22:35:06.966 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068193614999999999, 2021-03-15T22:35:06.966 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:06.967 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:06.967 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:06.967 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:06.967 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:06.967 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4285 2021-03-15T22:35:06.968 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.968 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:06.968 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.968 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:06.969 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:35:06.969 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:06.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.969 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.969 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:06.970 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162983+0000", 2021-03-15T22:35:06.970 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.633e-06 2021-03-15T22:35:06.970 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.970 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.970 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:06.971 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162985+0000", 2021-03-15T22:35:06.971 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.6789999999999999e-06 2021-03-15T22:35:06.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.972 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:06.972 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162987+0000", 2021-03-15T22:35:06.972 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.9500000000000001e-06 2021-03-15T22:35:06.972 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.973 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:06.973 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162995+0000", 2021-03-15T22:35:06.973 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.459e-06 2021-03-15T22:35:06.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.974 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:06.974 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.162999+0000", 2021-03-15T22:35:06.974 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012207240000000001 2021-03-15T22:35:06.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.975 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:06.975 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164220+0000", 2021-03-15T22:35:06.975 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0790000000000004e-05 2021-03-15T22:35:06.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.976 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.976 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:06.976 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164301+0000", 2021-03-15T22:35:06.976 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00014765399999999999 2021-03-15T22:35:06.976 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.977 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 5,6", 2021-03-15T22:35:06.977 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164449+0000", 2021-03-15T22:35:06.977 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.042818110999999999 2021-03-15T22:35:06.977 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.978 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:06.978 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.207267+0000", 2021-03-15T22:35:06.978 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0055270550000000003 2021-03-15T22:35:06.979 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.979 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:06.979 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.212794+0000", 2021-03-15T22:35:06.979 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018305267 2021-03-15T22:35:06.980 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.980 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:35:06.980 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231099+0000", 2021-03-15T22:35:06.981 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4928000000000002e-05 2021-03-15T22:35:06.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.981 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:06.981 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231134+0000", 2021-03-15T22:35:06.982 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2364999999999998e-05 2021-03-15T22:35:06.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.982 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:06.983 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231176+0000", 2021-03-15T22:35:06.983 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068193614999999999 2021-03-15T22:35:06.983 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:06.983 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:06.983 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:06.984 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.984 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.984 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:4286 7.b 7:d4368cae:::smithi07694878-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e445)", 2021-03-15T22:35:06.984 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:35:06.984 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 546.778917998, 2021-03-15T22:35:06.985 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068364477000000007, 2021-03-15T22:35:06.985 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:06.985 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:35:06.985 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:06.985 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:06.986 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:06.986 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 4286 2021-03-15T22:35:06.986 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.986 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:06.987 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.987 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:06.987 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:35:06.987 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:06.988 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.988 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.988 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:06.988 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163004+0000", 2021-03-15T22:35:06.988 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6500000000000008e-07 2021-03-15T22:35:06.989 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.989 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.989 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:06.989 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163005+0000", 2021-03-15T22:35:06.989 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.218e-06 2021-03-15T22:35:06.990 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.990 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.990 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:06.990 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163007+0000", 2021-03-15T22:35:06.991 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3890000000000002e-06 2021-03-15T22:35:06.991 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.991 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.991 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:06.991 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163013+0000", 2021-03-15T22:35:06.992 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.053e-06 2021-03-15T22:35:06.992 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.992 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.992 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:06.993 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.163017+0000", 2021-03-15T22:35:06.993 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0017994630000000001 2021-03-15T22:35:06.993 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.993 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.993 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:06.994 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164816+0000", 2021-03-15T22:35:06.994 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0958000000000007e-05 2021-03-15T22:35:06.994 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.994 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.995 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:06.995 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164897+0000", 2021-03-15T22:35:06.995 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066313677000000001 2021-03-15T22:35:06.995 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.995 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.996 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:06.996 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231211+0000", 2021-03-15T22:35:06.996 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9739000000000004e-05 2021-03-15T22:35:06.996 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.996 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.997 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:06.997 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231281+0000", 2021-03-15T22:35:06.997 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8015000000000003e-05 2021-03-15T22:35:06.997 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.998 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.998 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:06.998 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.231369+0000", 2021-03-15T22:35:06.998 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.068364477000000007 2021-03-15T22:35:06.998 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:06.999 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:06.999 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:06.999 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:06.999 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:06.999 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4287 7.7 e445/420 7:ed4a01d2:::smithi07694878-2:head v 445'413, mlcod=444'410)", 2021-03-15T22:35:07.000 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:35:07.000 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 546.77789970200001, 2021-03-15T22:35:07.000 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066973671999999998, 2021-03-15T22:35:07.000 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.001 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.001 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.001 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.001 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.001 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:35:07.002 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.002 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.002 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.002 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.003 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164023+0000", 2021-03-15T22:35:07.003 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.258e-06 2021-03-15T22:35:07.003 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.003 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.003 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.004 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164026+0000", 2021-03-15T22:35:07.004 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7980000000000001e-06 2021-03-15T22:35:07.004 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.004 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.004 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.005 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164029+0000", 2021-03-15T22:35:07.005 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0787e-05 2021-03-15T22:35:07.005 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.005 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.006 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.006 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164039+0000", 2021-03-15T22:35:07.006 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2829999999999998e-06 2021-03-15T22:35:07.006 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.006 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.007 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.007 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164046+0000", 2021-03-15T22:35:07.007 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5735000000000001e-05 2021-03-15T22:35:07.007 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.008 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.008 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.008 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164101+0000", 2021-03-15T22:35:07.008 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9206000000000001e-05 2021-03-15T22:35:07.008 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.009 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.009 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.009 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.164131+0000", 2021-03-15T22:35:07.009 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066830296999999997 2021-03-15T22:35:07.009 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.010 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.010 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.010 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.230961+0000", 2021-03-15T22:35:07.010 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5308000000000001e-05 2021-03-15T22:35:07.011 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.011 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.011 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.011 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:00.230996+0000", 2021-03-15T22:35:07.011 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066973671999999998 2021-03-15T22:35:07.012 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.012 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.012 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.012 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.013 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.013 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:5830 7.5 e631/559 7:ac84e656:::smithi07694878-31:head v 631'458, mlcod=630'454)", 2021-03-15T22:35:07.013 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:35:07.013 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 326.69026248699998, 2021-03-15T22:35:07.013 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083826879000000007, 2021-03-15T22:35:07.014 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.014 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.014 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.014 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.015 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.015 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:35:07.015 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.015 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.015 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.016 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.016 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.251660+0000", 2021-03-15T22:35:07.016 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0060822330000000003 2021-03-15T22:35:07.016 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.016 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.017 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.017 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257742+0000", 2021-03-15T22:35:07.017 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9649999999999996e-06 2021-03-15T22:35:07.017 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.017 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.018 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.018 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257748+0000", 2021-03-15T22:35:07.018 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0319e-05 2021-03-15T22:35:07.018 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.019 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.019 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.019 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257768+0000", 2021-03-15T22:35:07.019 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0340000000000002e-06 2021-03-15T22:35:07.019 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.020 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.020 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.020 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257777+0000", 2021-03-15T22:35:07.020 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0096000000000001e-05 2021-03-15T22:35:07.020 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.021 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.021 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.021 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257847+0000", 2021-03-15T22:35:07.021 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7812000000000001e-05 2021-03-15T22:35:07.022 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.022 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.022 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.022 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.257875+0000", 2021-03-15T22:35:07.022 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077557744999999997 2021-03-15T22:35:07.023 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.023 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.023 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.023 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.335433+0000", 2021-03-15T22:35:07.024 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3674999999999997e-05 2021-03-15T22:35:07.024 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.024 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.024 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.024 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:40.335487+0000", 2021-03-15T22:35:07.025 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.083826879000000007 2021-03-15T22:35:07.025 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.025 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.025 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.025 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.026 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.026 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:755 7.5 e646/559 7:a9545d62:::smithi07694878-1:1fc v 646'483, mlcod=644'479)", 2021-03-15T22:35:07.026 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:35:07.026 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 311.69563813100001, 2021-03-15T22:35:07.027 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087969915999999995, 2021-03-15T22:35:07.027 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.027 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.027 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.027 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.028 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.028 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:35:07.028 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.028 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.029 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.029 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.029 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246284+0000", 2021-03-15T22:35:07.029 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1300000000000001e-06 2021-03-15T22:35:07.029 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.030 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.030 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.030 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246287+0000", 2021-03-15T22:35:07.030 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1730000000000002e-06 2021-03-15T22:35:07.031 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.031 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.031 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.031 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246290+0000", 2021-03-15T22:35:07.031 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1728e-05 2021-03-15T22:35:07.032 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.032 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.032 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.032 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246301+0000", 2021-03-15T22:35:07.032 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.7710000000000001e-06 2021-03-15T22:35:07.033 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.033 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.033 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.033 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246308+0000", 2021-03-15T22:35:07.033 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5705999999999999e-05 2021-03-15T22:35:07.034 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.034 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.034 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.034 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246354+0000", 2021-03-15T22:35:07.035 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8252e-05 2021-03-15T22:35:07.035 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.035 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.035 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.035 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.246382+0000", 2021-03-15T22:35:07.036 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087836030999999995 2021-03-15T22:35:07.036 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.036 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.036 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.036 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334218+0000", 2021-03-15T22:35:07.037 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6124999999999997e-05 2021-03-15T22:35:07.037 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.037 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.037 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.038 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:29:55.334254+0000", 2021-03-15T22:35:07.038 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.087969915999999995 2021-03-15T22:35:07.038 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.038 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.038 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.039 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.039 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.039 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6447 7.b 7:d980be1f:::smithi07694878-23:21c [read 0~0,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e691)", 2021-03-15T22:35:07.039 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.215534+0000", 2021-03-15T22:35:07.039 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 266.726388574, 2021-03-15T22:35:07.040 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079694523000000003, 2021-03-15T22:35:07.040 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.040 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:35:07.040 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:07.041 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:07.041 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:07.041 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6447 2021-03-15T22:35:07.041 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.041 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.042 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.042 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.042 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215534+0000", 2021-03-15T22:35:07.042 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.042 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.043 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.043 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.043 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215534+0000", 2021-03-15T22:35:07.043 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2140000000000004e-06 2021-03-15T22:35:07.043 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.044 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.044 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.044 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215539+0000", 2021-03-15T22:35:07.044 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.8670000000000004e-06 2021-03-15T22:35:07.045 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.045 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.045 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.045 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215543+0000", 2021-03-15T22:35:07.045 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4408e-05 2021-03-15T22:35:07.046 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.046 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.046 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.046 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215557+0000", 2021-03-15T22:35:07.046 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4090000000000003e-06 2021-03-15T22:35:07.047 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.047 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.047 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.047 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215566+0000", 2021-03-15T22:35:07.047 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2050000000000002e-05 2021-03-15T22:35:07.048 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.048 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.048 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.048 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215648+0000", 2021-03-15T22:35:07.049 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8050000000000007e-06 2021-03-15T22:35:07.049 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.049 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.049 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:35:07.049 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215656+0000", 2021-03-15T22:35:07.050 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079466859000000001 2021-03-15T22:35:07.050 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.050 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.050 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.050 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295122+0000", 2021-03-15T22:35:07.051 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000105911 2021-03-15T22:35:07.051 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.051 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.051 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.052 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295228+0000", 2021-03-15T22:35:07.052 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079694523000000003 2021-03-15T22:35:07.052 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.052 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.052 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.053 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.053 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.053 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6448 7.b 7:d980be1f:::smithi07694878-23:21c [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected e691)", 2021-03-15T22:35:07.053 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.215580+0000", 2021-03-15T22:35:07.053 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 266.72634183899999, 2021-03-15T22:35:07.054 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079736783000000006, 2021-03-15T22:35:07.054 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.054 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:35:07.054 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:07.054 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:07.055 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:07.055 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6448 2021-03-15T22:35:07.055 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.055 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.056 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.056 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.056 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215580+0000", 2021-03-15T22:35:07.056 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.056 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.057 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.057 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.057 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215580+0000", 2021-03-15T22:35:07.057 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.248e-06 2021-03-15T22:35:07.057 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.058 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.058 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.058 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215582+0000", 2021-03-15T22:35:07.058 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.646e-06 2021-03-15T22:35:07.058 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.059 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.059 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.059 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215583+0000", 2021-03-15T22:35:07.059 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.4140000000000002e-06 2021-03-15T22:35:07.060 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.060 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.060 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.060 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215589+0000", 2021-03-15T22:35:07.060 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9349999999999996e-06 2021-03-15T22:35:07.061 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.061 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.061 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.061 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215593+0000", 2021-03-15T22:35:07.061 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000158023 2021-03-15T22:35:07.062 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.062 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.062 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.062 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215751+0000", 2021-03-15T22:35:07.063 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2480000000000004e-06 2021-03-15T22:35:07.063 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.063 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.063 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:35:07.063 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215756+0000", 2021-03-15T22:35:07.064 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079496878000000007 2021-03-15T22:35:07.064 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.064 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.064 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.064 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295253+0000", 2021-03-15T22:35:07.065 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.4391000000000001e-05 2021-03-15T22:35:07.065 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.065 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.065 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.066 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295317+0000", 2021-03-15T22:35:07.066 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079736783000000006 2021-03-15T22:35:07.066 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.066 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.066 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.067 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.067 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.067 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6449 7.b 7:d980be1f:::smithi07694878-23:21c [read 0~0,checksum] snapc 0=[] ondisk+read+known_if_redirected e691)", 2021-03-15T22:35:07.067 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.215597+0000", 2021-03-15T22:35:07.067 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 266.72632554699999, 2021-03-15T22:35:07.068 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079805300999999995, 2021-03-15T22:35:07.068 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.068 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:35:07.068 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:07.069 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:07.069 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:07.069 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6449 2021-03-15T22:35:07.069 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.069 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.070 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.070 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.070 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215597+0000", 2021-03-15T22:35:07.070 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.071 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.071 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.071 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.071 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215597+0000", 2021-03-15T22:35:07.071 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.621e-06 2021-03-15T22:35:07.072 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.072 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.072 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.072 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215598+0000", 2021-03-15T22:35:07.072 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.468e-06 2021-03-15T22:35:07.073 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.073 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.073 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.073 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215600+0000", 2021-03-15T22:35:07.073 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7419999999999997e-06 2021-03-15T22:35:07.074 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.074 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.074 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.074 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215605+0000", 2021-03-15T22:35:07.075 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4640000000000002e-06 2021-03-15T22:35:07.075 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.075 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.075 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.075 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215608+0000", 2021-03-15T22:35:07.076 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016377699999999999 2021-03-15T22:35:07.076 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.076 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.076 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.077 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215772+0000", 2021-03-15T22:35:07.077 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3549999999999998e-06 2021-03-15T22:35:07.077 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.077 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.077 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:35:07.078 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.215776+0000", 2021-03-15T22:35:07.078 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079562503000000007 2021-03-15T22:35:07.078 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.078 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.078 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.079 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295339+0000", 2021-03-15T22:35:07.079 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.3371e-05 2021-03-15T22:35:07.079 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.079 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.080 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.080 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.295402+0000", 2021-03-15T22:35:07.080 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079805300999999995 2021-03-15T22:35:07.080 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.080 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.081 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.081 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.081 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.081 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:6453 7.9 e691/556 7:91d654eb:::smithi07694878-5:head v 691'298, mlcod=681'295)", 2021-03-15T22:35:07.082 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.298138+0000", 2021-03-15T22:35:07.082 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 266.64378428999999, 2021-03-15T22:35:07.082 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080231284999999999, 2021-03-15T22:35:07.082 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.082 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.083 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.083 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.083 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.083 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298138+0000", 2021-03-15T22:35:07.083 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.084 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.084 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.084 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.084 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298138+0000", 2021-03-15T22:35:07.085 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012879800000000001 2021-03-15T22:35:07.085 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.085 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.085 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.085 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298267+0000", 2021-03-15T22:35:07.086 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.6959999999999998e-06 2021-03-15T22:35:07.086 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.086 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.086 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.086 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298271+0000", 2021-03-15T22:35:07.087 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4399999999999994e-06 2021-03-15T22:35:07.087 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.087 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.087 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.088 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298280+0000", 2021-03-15T22:35:07.088 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2299999999999996e-06 2021-03-15T22:35:07.088 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.088 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.088 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.089 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298286+0000", 2021-03-15T22:35:07.089 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00045678599999999999 2021-03-15T22:35:07.089 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.089 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.089 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.090 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298743+0000", 2021-03-15T22:35:07.090 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4567999999999999e-05 2021-03-15T22:35:07.090 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.090 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.091 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.091 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.298768+0000", 2021-03-15T22:35:07.091 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079546108000000004 2021-03-15T22:35:07.091 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.091 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.092 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.092 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378314+0000", 2021-03-15T22:35:07.092 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5658999999999997e-05 2021-03-15T22:35:07.092 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.092 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.093 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.093 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.378369+0000", 2021-03-15T22:35:07.093 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080231284999999999 2021-03-15T22:35:07.093 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.094 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.094 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.094 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.094 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.094 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:7256 7.0 e772/642 7:071f7a04:::smithi07694878-37:head v 772'2464, mlcod=770'2460)", 2021-03-15T22:35:07.095 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:02.263683+0000", 2021-03-15T22:35:07.095 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 184.67823952099999, 2021-03-15T22:35:07.095 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079578619000000003, 2021-03-15T22:35:07.095 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.095 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.096 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.096 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.096 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.096 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.263683+0000", 2021-03-15T22:35:07.096 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.097 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.097 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.097 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.097 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.263683+0000", 2021-03-15T22:35:07.098 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00093819800000000001 2021-03-15T22:35:07.098 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.098 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.098 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.098 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.264621+0000", 2021-03-15T22:35:07.099 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2779999999999999e-06 2021-03-15T22:35:07.099 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.099 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.099 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.100 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.264627+0000", 2021-03-15T22:35:07.100 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.062960655000000004 2021-03-15T22:35:07.100 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.100 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.100 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.101 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.327588+0000", 2021-03-15T22:35:07.101 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.593e-06 2021-03-15T22:35:07.101 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.101 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.101 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.102 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.327596+0000", 2021-03-15T22:35:07.102 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00059060600000000003 2021-03-15T22:35:07.102 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.102 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.103 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.103 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328186+0000", 2021-03-15T22:35:07.103 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4825e-05 2021-03-15T22:35:07.103 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.103 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.104 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.104 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.328211+0000", 2021-03-15T22:35:07.104 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015016850999999999 2021-03-15T22:35:07.104 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.104 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.105 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.105 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.343228+0000", 2021-03-15T22:35:07.105 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3612999999999998e-05 2021-03-15T22:35:07.105 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.106 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.106 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.106 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:02.343261+0000", 2021-03-15T22:35:07.106 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.079578619000000003 2021-03-15T22:35:07.106 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.107 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.107 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.107 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.107 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.107 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.4.0:684 7.1 e816/742 7:8bc62bc6:::smithi07694878-17:20e v 816'433, mlcod=813'429)", 2021-03-15T22:35:07.108 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:46.273595+0000", 2021-03-15T22:35:07.108 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 140.66832754699999, 2021-03-15T22:35:07.108 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078116977000000004, 2021-03-15T22:35:07.108 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.108 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.109 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.109 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.109 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.109 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273595+0000", 2021-03-15T22:35:07.110 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.110 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.110 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.110 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.110 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273595+0000", 2021-03-15T22:35:07.111 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9749999999999999e-06 2021-03-15T22:35:07.111 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.111 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.111 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.111 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273598+0000", 2021-03-15T22:35:07.112 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.8880000000000001e-06 2021-03-15T22:35:07.112 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.112 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.112 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.113 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273601+0000", 2021-03-15T22:35:07.113 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1609999999999997e-06 2021-03-15T22:35:07.113 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.113 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.113 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273608+0000", 2021-03-15T22:35:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.9379999999999997e-06 2021-03-15T22:35:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.114 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.273614+0000", 2021-03-15T22:35:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.074469357999999999 2021-03-15T22:35:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.115 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.348083+0000", 2021-03-15T22:35:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3329999999999999e-05 2021-03-15T22:35:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.116 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.348106+0000", 2021-03-15T22:35:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003527517 2021-03-15T22:35:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.117 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.351634+0000", 2021-03-15T22:35:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7810000000000002e-05 2021-03-15T22:35:07.118 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.351712+0000", 2021-03-15T22:35:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.078116977000000004 2021-03-15T22:35:07.119 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.120 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7697 7.b 7:d10ba14d:::smithi07694878-43:head [read 0~3042298,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e816)", 2021-03-15T22:35:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:35:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 140.650738969, 2021-03-15T22:35:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066878664000000004, 2021-03-15T22:35:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.121 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:35:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:07.122 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7697 2021-03-15T22:35:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.123 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:35:07.124 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.124 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.124 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.124 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.124 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291183+0000", 2021-03-15T22:35:07.125 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2300000000000002e-06 2021-03-15T22:35:07.125 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.125 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.125 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.126 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291188+0000", 2021-03-15T22:35:07.126 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9349999999999999e-06 2021-03-15T22:35:07.126 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.126 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.126 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.127 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.291191+0000", 2021-03-15T22:35:07.127 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.056407205000000002 2021-03-15T22:35:07.127 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.127 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.127 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.128 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347598+0000", 2021-03-15T22:35:07.128 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0958000000000001e-05 2021-03-15T22:35:07.128 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.128 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.129 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.129 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.347609+0000", 2021-03-15T22:35:07.129 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0032663129999999999 2021-03-15T22:35:07.129 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.129 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.130 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.130 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.350875+0000", 2021-03-15T22:35:07.130 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5338000000000002e-05 2021-03-15T22:35:07.130 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.130 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.131 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.131 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.350960+0000", 2021-03-15T22:35:07.131 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0071016849999999999 2021-03-15T22:35:07.131 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.132 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.132 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.132 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:32:46.358062+0000", 2021-03-15T22:35:07.132 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.066878664000000004 2021-03-15T22:35:07.132 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.133 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.133 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.133 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.133 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.133 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:8350 7.0 e881/642 7:0cc37e3d:::smithi07694878-21:head v 881'2927, mlcod=880'2919)", 2021-03-15T22:35:07.134 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:51.525868+0000", 2021-03-15T22:35:07.134 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 75.416054799999998, 2021-03-15T22:35:07.134 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016512084, 2021-03-15T22:35:07.134 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.135 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.135 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.135 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.135 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.135 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:51.525868+0000", 2021-03-15T22:35:07.136 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.136 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.136 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.136 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.136 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:51.525868+0000", 2021-03-15T22:35:07.137 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0075251670000000001 2021-03-15T22:35:07.137 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.137 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.137 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.138 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:51.533393+0000", 2021-03-15T22:35:07.138 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.757e-06 2021-03-15T22:35:07.138 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.138 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.138 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.139 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:51.533395+0000", 2021-03-15T22:35:07.139 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2310000000000001e-05 2021-03-15T22:35:07.139 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.139 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.140 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.140 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:51.533408+0000", 2021-03-15T22:35:07.140 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0610000000000003e-06 2021-03-15T22:35:07.140 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.140 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.141 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.141 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:51.533413+0000", 2021-03-15T22:35:07.141 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.7685999999999997e-05 2021-03-15T22:35:07.141 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.141 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.142 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.142 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:51.533461+0000", 2021-03-15T22:35:07.142 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2470999999999999e-05 2021-03-15T22:35:07.142 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.143 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.143 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.143 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:51.533483+0000", 2021-03-15T22:35:07.143 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0088630830000000008 2021-03-15T22:35:07.143 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.144 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.144 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.144 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:51.542346+0000", 2021-03-15T22:35:07.144 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3549e-05 2021-03-15T22:35:07.144 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.145 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.145 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.145 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:51.542380+0000", 2021-03-15T22:35:07.145 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.016512084 2021-03-15T22:35:07.146 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.146 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.146 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.146 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.146 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.147 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:8374 7.b 7:d4368cae:::smithi07694878-18:head [sparse-read 0~2840614,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e887)", 2021-03-15T22:35:07.147 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:02.548208+0000", 2021-03-15T22:35:07.147 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 64.393714704999994, 2021-03-15T22:35:07.147 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015975698999999999, 2021-03-15T22:35:07.147 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.148 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:35:07.148 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:07.148 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:07.148 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:07.149 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 8374 2021-03-15T22:35:07.149 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.149 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.149 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.149 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.150 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:02.548208+0000", 2021-03-15T22:35:07.150 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.150 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.150 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.150 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.151 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:02.548208+0000", 2021-03-15T22:35:07.151 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.853e-06 2021-03-15T22:35:07.151 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.151 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.152 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.152 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:02.548212+0000", 2021-03-15T22:35:07.152 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2569999999999999e-06 2021-03-15T22:35:07.152 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.152 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.153 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.153 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:02.548216+0000", 2021-03-15T22:35:07.153 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5254e-05 2021-03-15T22:35:07.153 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.153 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.154 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.154 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:02.548231+0000", 2021-03-15T22:35:07.154 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.2400000000000001e-06 2021-03-15T22:35:07.154 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.155 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.155 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.155 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:02.548238+0000", 2021-03-15T22:35:07.155 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.011524680000000001 2021-03-15T22:35:07.155 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.156 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.156 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.156 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:02.559763+0000", 2021-03-15T22:35:07.156 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8666999999999997e-05 2021-03-15T22:35:07.156 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.157 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.157 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.157 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:02.559852+0000", 2021-03-15T22:35:07.157 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043317479999999998 2021-03-15T22:35:07.157 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.158 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.158 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.158 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:02.564183+0000", 2021-03-15T22:35:07.158 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015975698999999999 2021-03-15T22:35:07.159 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.159 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.159 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.159 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.159 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.160 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:8473 7.5 e900/881 7:a9545d62:::smithi07694878-1:head v 900'833, mlcod=898'829)", 2021-03-15T22:35:07.160 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:15.514361+0000", 2021-03-15T22:35:07.160 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 51.427561681, 2021-03-15T22:35:07.160 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081667280999999994, 2021-03-15T22:35:07.161 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.161 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.161 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.161 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.161 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.162 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.514361+0000", 2021-03-15T22:35:07.162 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.162 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.162 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.162 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.163 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.514361+0000", 2021-03-15T22:35:07.163 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0043657820000000003 2021-03-15T22:35:07.163 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.163 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.163 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.164 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.518726+0000", 2021-03-15T22:35:07.164 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.2529999999999998e-06 2021-03-15T22:35:07.164 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.164 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.165 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.165 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.518731+0000", 2021-03-15T22:35:07.165 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5979999999999999e-05 2021-03-15T22:35:07.165 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.165 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.166 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.166 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.518747+0000", 2021-03-15T22:35:07.166 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.49e-06 2021-03-15T22:35:07.166 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.166 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.167 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.167 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.518752+0000", 2021-03-15T22:35:07.167 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8075999999999998e-05 2021-03-15T22:35:07.167 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.167 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.168 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.168 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.518800+0000", 2021-03-15T22:35:07.168 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9457e-05 2021-03-15T22:35:07.168 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.169 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.169 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.169 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.518820+0000", 2021-03-15T22:35:07.169 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.077169143999999995 2021-03-15T22:35:07.169 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.170 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.170 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.170 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.595989+0000", 2021-03-15T22:35:07.170 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.9099e-05 2021-03-15T22:35:07.171 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.171 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.171 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.171 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:15.596028+0000", 2021-03-15T22:35:07.171 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.081667280999999994 2021-03-15T22:35:07.172 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.172 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.172 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.172 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.172 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.173 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:1024 7.5 e914/881 7:ac84e656:::smithi07694878-31:2b6 v 914'849, mlcod=913'845)", 2021-03-15T22:35:07.173 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:29.607577+0000", 2021-03-15T22:35:07.173 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 37.334344973, 2021-03-15T22:35:07.173 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.067915246999999998, 2021-03-15T22:35:07.174 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.174 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.174 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.174 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.174 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.175 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.607577+0000", 2021-03-15T22:35:07.175 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.175 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.175 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.176 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.176 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.607577+0000", 2021-03-15T22:35:07.176 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7599999999999997e-06 2021-03-15T22:35:07.176 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.176 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.177 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.177 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.607587+0000", 2021-03-15T22:35:07.177 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.743e-06 2021-03-15T22:35:07.177 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.177 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.178 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.178 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.607591+0000", 2021-03-15T22:35:07.178 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.065740641000000002 2021-03-15T22:35:07.178 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.178 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.179 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.179 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.673332+0000", 2021-03-15T22:35:07.179 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3540000000000002e-06 2021-03-15T22:35:07.179 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.180 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.180 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.180 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.673339+0000", 2021-03-15T22:35:07.180 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0257e-05 2021-03-15T22:35:07.181 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.181 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.181 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.181 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.673429+0000", 2021-03-15T22:35:07.181 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4343999999999999e-05 2021-03-15T22:35:07.182 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.182 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.182 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.182 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.673453+0000", 2021-03-15T22:35:07.182 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020062869999999998 2021-03-15T22:35:07.183 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.183 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.183 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.183 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.675460+0000", 2021-03-15T22:35:07.184 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2861000000000003e-05 2021-03-15T22:35:07.184 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.184 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.184 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.184 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.675493+0000", 2021-03-15T22:35:07.185 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.067915246999999998 2021-03-15T22:35:07.185 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.185 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.185 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.185 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.186 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.186 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.6.0:1608 7.d e935/837 7:bd183e1b:::smithi07694878-46:2be v 935'417, mlcod=932'413)", 2021-03-15T22:35:07.186 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:50.325741+0000", 2021-03-15T22:35:07.186 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 16.61618159, 2021-03-15T22:35:07.186 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018137021999999999, 2021-03-15T22:35:07.187 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.187 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.187 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.187 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.188 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.188 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:50.325741+0000", 2021-03-15T22:35:07.188 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.188 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.188 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.189 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.189 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:50.325741+0000", 2021-03-15T22:35:07.189 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8599999999999998e-06 2021-03-15T22:35:07.189 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.190 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.190 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.190 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:50.325747+0000", 2021-03-15T22:35:07.190 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.7580000000000001e-06 2021-03-15T22:35:07.190 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.191 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.191 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.191 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:50.325749+0000", 2021-03-15T22:35:07.191 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8789999999999997e-06 2021-03-15T22:35:07.191 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.192 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.192 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.192 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:50.325756+0000", 2021-03-15T22:35:07.192 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6779999999999997e-06 2021-03-15T22:35:07.192 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:50.325762+0000", 2021-03-15T22:35:07.193 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.002303785 2021-03-15T22:35:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:50.328066+0000", 2021-03-15T22:35:07.194 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.0822000000000002e-05 2021-03-15T22:35:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.195 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:50.328087+0000", 2021-03-15T22:35:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.015759097 2021-03-15T22:35:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.196 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:50.343846+0000", 2021-03-15T22:35:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2143e-05 2021-03-15T22:35:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.197 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:50.343878+0000", 2021-03-15T22:35:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.018137021999999999 2021-03-15T22:35:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.198 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.7.0:1072 7.5 e936/881 7:a9545d62:::smithi07694878-1:2b3 v 936'895, mlcod=933'889)", 2021-03-15T22:35:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:51.368516+0000", 2021-03-15T22:35:07.199 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.573405874000001, 2021-03-15T22:35:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024707060999999999, 2021-03-15T22:35:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.200 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.368516+0000", 2021-03-15T22:35:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.201 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.368516+0000", 2021-03-15T22:35:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.1810000000000003e-06 2021-03-15T22:35:07.202 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.368521+0000", 2021-03-15T22:35:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0180000000000002e-06 2021-03-15T22:35:07.203 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.368524+0000", 2021-03-15T22:35:07.204 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.008e-05 2021-03-15T22:35:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.368534+0000", 2021-03-15T22:35:07.205 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0081000000000001e-05 2021-03-15T22:35:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.368544+0000", 2021-03-15T22:35:07.206 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.1791000000000001e-05 2021-03-15T22:35:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.207 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.368576+0000", 2021-03-15T22:35:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1670000000000001e-05 2021-03-15T22:35:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.208 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.368597+0000", 2021-03-15T22:35:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024592737999999999 2021-03-15T22:35:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.209 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.393190+0000", 2021-03-15T22:35:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3501999999999999e-05 2021-03-15T22:35:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.210 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.393224+0000", 2021-03-15T22:35:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.024707060999999999 2021-03-15T22:35:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.211 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(osd.6.0:1611 7.d e936/837 7:bd183e1b:::smithi07694878-46:2be v 936'419, mlcod=933'415)", 2021-03-15T22:35:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:51.370498+0000", 2021-03-15T22:35:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 15.571423999, 2021-03-15T22:35:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023183448999999998, 2021-03-15T22:35:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:07.213 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.370498+0000", 2021-03-15T22:35:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:07.214 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.370498+0000", 2021-03-15T22:35:07.215 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.066e-06 2021-03-15T22:35:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.370501+0000", 2021-03-15T22:35:07.216 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.419e-06 2021-03-15T22:35:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.370504+0000", 2021-03-15T22:35:07.217 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.021342188000000002 2021-03-15T22:35:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.391846+0000", 2021-03-15T22:35:07.218 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.1720000000000007e-06 2021-03-15T22:35:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:07.219 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.391854+0000", 2021-03-15T22:35:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00090126300000000002 2021-03-15T22:35:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:07.220 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.392755+0000", 2021-03-15T22:35:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3521999999999999e-05 2021-03-15T22:35:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:07.221 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.392779+0000", 2021-03-15T22:35:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00087303100000000002 2021-03-15T22:35:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.222 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.393652+0000", 2021-03-15T22:35:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9788e-05 2021-03-15T22:35:07.223 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:51.393682+0000", 2021-03-15T22:35:07.224 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.023183448999999998 2021-03-15T22:35:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:07.225 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:07.226 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:35:07.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:07.248 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:07.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:07.351 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:07.377 INFO:tasks.rados.rados.0.smithi076.stdout:3594: done (3 left) 2021-03-15T22:35:07.377 INFO:tasks.rados.rados.0.smithi076.stdout:3595: rollback oid 11 current snap is 424 2021-03-15T22:35:07.378 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 11 to 411 2021-03-15T22:35:07.378 INFO:tasks.rados.rados.0.smithi076.stdout:3596: read oid 47 snap -1 2021-03-15T22:35:07.378 INFO:tasks.rados.rados.0.smithi076.stdout:3596: expect deleted 2021-03-15T22:35:07.378 INFO:tasks.rados.rados.0.smithi076.stdout:3597: snap_create 2021-03-15T22:35:07.379 INFO:tasks.rados.rados.0.smithi076.stdout:3591: finishing write tid 2 to smithi07694878-20 2021-03-15T22:35:07.379 INFO:tasks.rados.rados.0.smithi076.stdout:3591: finishing write tid 3 to smithi07694878-20 2021-03-15T22:35:07.379 INFO:tasks.rados.rados.0.smithi076.stdout:3592: finishing rollback tid 0 to smithi07694878-25 2021-03-15T22:35:07.379 INFO:tasks.rados.rados.0.smithi076.stdout:3591: finishing write tid 4 to smithi07694878-20 2021-03-15T22:35:07.379 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 2436 (ObjNum 1224 snap 424 seq_num 1224) dirty exists 2021-03-15T22:35:07.380 INFO:tasks.rados.rados.0.smithi076.stdout:3591: left oid 20 (ObjNum 1224 snap 424 seq_num 1224) 2021-03-15T22:35:07.380 INFO:tasks.rados.rados.0.smithi076.stdout:3592: finishing rollback tid 1 to smithi07694878-25 2021-03-15T22:35:07.380 INFO:tasks.rados.rados.0.smithi076.stdout:3592: finishing rollback tid 2 to smithi07694878-25 2021-03-15T22:35:07.380 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 2432 (ObjNum 888 snap 296 seq_num 888) dirty exists 2021-03-15T22:35:07.380 INFO:tasks.rados.rados.0.smithi076.stdout:3593: finishing write tid 1 to smithi07694878-21 2021-03-15T22:35:07.381 INFO:tasks.rados.rados.0.smithi076.stdout:3593: finishing write tid 2 to smithi07694878-21 2021-03-15T22:35:07.381 INFO:tasks.rados.rados.0.smithi076.stdout:3593: finishing write tid 3 to smithi07694878-21 2021-03-15T22:35:07.381 INFO:tasks.rados.rados.0.smithi076.stdout:3593: finishing write tid 4 to smithi07694878-21 2021-03-15T22:35:07.381 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 3159 (ObjNum 1225 snap 424 seq_num 1225) dirty exists 2021-03-15T22:35:07.382 INFO:tasks.rados.rados.0.smithi076.stdout:3593: left oid 21 (ObjNum 1225 snap 424 seq_num 1225) 2021-03-15T22:35:07.382 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:35:07.383 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:07.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:07.385 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:07.388 INFO:tasks.rados.rados.0.smithi076.stdout:3595: finishing rollback tid 1 to smithi07694878-11 2021-03-15T22:35:07.389 INFO:tasks.rados.rados.0.smithi076.stdout:3595: finishing rollback tid 2 to smithi07694878-11 2021-03-15T22:35:07.389 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 2016 (ObjNum 1151 snap 404 seq_num 1151) dirty exists 2021-03-15T22:35:07.486 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:07.507 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:07.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:07.509 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:07.610 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:07.631 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:07.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:07.632 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:07.733 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:07.755 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:07.756 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:07.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[56514]: cluster 2021-03-15T22:35:06.362255+0000 mon.a (mon.0) 2385 : cluster [DBG] osdmap e951: 8 total, 8 up, 8 in 2021-03-15T22:35:07.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[56514]: audit 2021-03-15T22:35:06.796542+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? 172.21.15.76:0/1955427939' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:35:07.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: cluster 2021-03-15T22:35:07.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 2021-03-15T22:35:07.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 2021 2021-03-15T22:35:07.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: - 2021-03-15T22:35:07.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 03 2021-03-15T22:35:07.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: - 2021-03-15T22:35:07.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 15 2021-03-15T22:35:07.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: T 2021-03-15T22:35:07.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 22 2021-03-15T22:35:07.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: : 2021-03-15T22:35:07.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 35 2021-03-15T22:35:07.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: : 2021-03-15T22:35:07.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 06 2021-03-15T22:35:07.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: . 2021-03-15T22:35:07.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 362255 2021-03-15T22:35:07.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: +0000 2021-03-15T22:35:07.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 2021-03-15T22:35:07.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: mon.a 2021-03-15T22:35:07.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: ( 2021-03-15T22:35:07.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: mon 2021-03-15T22:35:07.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: . 2021-03-15T22:35:07.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 0 2021-03-15T22:35:07.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: ) 2021-03-15T22:35:07.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 2385 2021-03-15T22:35:07.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: : 2021-03-15T22:35:07.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: cluster 2021-03-15T22:35:07.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 2021-03-15T22:35:07.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: [DBG] 2021-03-15T22:35:07.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 2021-03-15T22:35:07.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: osdmap e951: 8 total, 8 up, 8 in 2021-03-15T22:35:07.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 2021-03-15T22:35:07.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:35:07.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:07 smithi076 conmon[62156]: 35:06.796542+0000 mon.a (mon.0) 2386 : audit [INF] from='client.? 172.21.15.76:0/1955427939' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:35:07.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:07 smithi125 conmon[56739]: cluster 2021-03-15T22:35:06.362255+0000 2021-03-15T22:35:07.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:07 smithi125 conmon[56739]: mon.a (mon.0) 2385 : cluster [DBG] osdmap e951: 8 total, 8 up, 8 in 2021-03-15T22:35:07.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:07 smithi125 conmon[56739]: audit 2021-03-15T22:35:06.796542+0000 mon.a 2021-03-15T22:35:07.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:07 smithi125 conmon[56739]: (mon.0) 2386 : audit [INF] from='client.? 172.21.15.76:0/1955427939' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:35:07.857 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:07.880 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:07.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:07.882 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:07.983 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:08.004 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:08.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:08.006 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:08.107 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:08.129 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:08.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:08.130 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:08.231 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:08.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-15T22:35:08.253 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:08.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:08.254 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:08.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:08.377 INFO:tasks.rados.rados.0.smithi076.stdout:3591: done (5 left) 2021-03-15T22:35:08.377 INFO:tasks.rados.rados.0.smithi076.stdout:3592: done (4 left) 2021-03-15T22:35:08.377 INFO:tasks.rados.rados.0.smithi076.stdout:3593: done (3 left) 2021-03-15T22:35:08.378 INFO:tasks.rados.rados.0.smithi076.stdout:3595: done (2 left) 2021-03-15T22:35:08.378 INFO:tasks.rados.rados.0.smithi076.stdout:3596: done (1 left) 2021-03-15T22:35:08.378 INFO:tasks.rados.rados.0.smithi076.stdout:3597: done (0 left) 2021-03-15T22:35:08.378 INFO:tasks.rados.rados.0.smithi076.stdout:3598: rollback oid 12 current snap is 425 2021-03-15T22:35:08.379 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 12 to 415 2021-03-15T22:35:08.379 INFO:tasks.rados.rados.0.smithi076.stdout:3599: copy_from oid 11 from oid 2 current snap is 425 2021-03-15T22:35:08.379 INFO:tasks.rados.rados.0.smithi076.stdout:3600: snap_remove snap 421 2021-03-15T22:35:08.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:08.380 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:08.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:08.482 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:08.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[56514]: cluster 2021-03-15T22:35:07.214935+0000 mgr.y (mgr.14144) 1064 : cluster 2021-03-15T22:35:08.497 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[56514]: [DBG] pgmap v1574: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+snaptrim, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 683 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 9 op/s; 1/1932 objects degraded (0.052%); 46/1932 objects misplaced (2.381%); 7.5 MiB/s, 20 objects/s recovering 2021-03-15T22:35:08.498 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[56514]: cluster 2021-03-15T22:35:07.358877+0000 mon.a (mon.0) 2387 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:08.498 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[56514]: audit 2021-03-15T22:35:07.366394+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? 172.21.15.76:0/1955427939' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:35:08.498 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[56514]: cluster 2021-03-15T22:35:07.366441+0000 mon.a (mon.0) 2389 : cluster [DBG] osdmap e952: 8 total, 8 up, 8 in 2021-03-15T22:35:08.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:35:08.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[62156]: 35:07.214935+0000 mgr.y (mgr.14144) 1064 : cluster [DBG] pgmap v1574: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+snaptrim, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 683 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 9 op/s; 1/1932 objects degraded (0.052%); 46/1932 objects misplaced (2.381%); 7.5 MiB/s, 20 objects/s recovering 2021-03-15T22:35:08.499 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[62156]: cluster 2021-03-15T22:35:08.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[62156]: 2021-03-15T22:35:07.358877+0000 mon.a (mon.0) 2387 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:08.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[62156]: 2021-03-15T22:35:08.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[62156]: audit 2021-03-15T22:35:07 2021-03-15T22:35:08.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[62156]: .366394+0000 mon.a (mon.0) 2388 : audit [INF] from='client.? 172.21.15.76:0/1955427939' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:35:08.500 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:35:08.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[62156]: :35:07.366441+0000 mon.a (mon.0) 2389 2021-03-15T22:35:08.501 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:08 smithi076 conmon[62156]: : cluster [DBG] osdmap e952: 8 total, 8 up, 8 in 2021-03-15T22:35:08.503 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:08.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:08.506 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:08.607 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:08.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:08.632 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:08.632 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:08.734 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:08.755 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:08.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:08.757 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:08.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: cluster 2021-03-15T22:35:07.214935+0000 mgr.y (mgr. 2021-03-15T22:35:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: 14144) 1064 : cluster [DBG] pgmap v1574: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+snaptrim, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 683 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.5 MiB/s wr, 9 op/s; 1/1932 objects degraded (0.052%); 46/1932 objects misplaced (2.381%); 7.5 MiB/s, 20 objects/s recovering 2021-03-15T22:35:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: cluster 2021-03-15T22:35:07.358877+0000 mon.a (mon. 2021-03-15T22:35:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: 0) 2387 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:08.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: audit 2021 2021-03-15T22:35:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: -03-15T22:35: 2021-03-15T22:35:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: 07.366394+0000 mon.a (mon.0 2021-03-15T22:35:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: ) 2388 : audit [INF] from='client.? 172.21.15.76:0/1955427939' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:35:08.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: 2021-03-15T22:35:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: cluster 2021-03-15T22:35:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: 2021-03-15T22:35 2021-03-15T22:35:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: :07.366441+0000 mon.a (mon 2021-03-15T22:35:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: .0) 2389 : cluster [DBG] 2021-03-15T22:35:08.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:08 smithi125 conmon[56739]: osdmap e952: 8 total, 8 up, 8 in 2021-03-15T22:35:08.858 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:08.881 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:08.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:08.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:08.984 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:09.005 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:09.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:09.007 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:09.108 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:09.130 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:09.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:09.132 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:09.233 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:09.254 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:09.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:09.256 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:09.356 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:09.378 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:09.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:09.380 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:09.385 INFO:tasks.rados.rados.0.smithi076.stdout:3600: done (2 left) 2021-03-15T22:35:09.385 INFO:tasks.rados.rados.0.smithi076.stdout:3601: rollback oid 1 current snap is 425 2021-03-15T22:35:09.386 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 1 to 359 2021-03-15T22:35:09.386 INFO:tasks.rados.rados.0.smithi076.stdout:3598: finishing rollback tid 1 to smithi07694878-12 2021-03-15T22:35:09.386 INFO:tasks.rados.rados.0.smithi076.stdout:3598: finishing rollback tid 2 to smithi07694878-12 2021-03-15T22:35:09.386 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 3015 (ObjNum 1160 snap 407 seq_num 1160) dirty exists 2021-03-15T22:35:09.386 INFO:tasks.rados.rados.0.smithi076.stdout:3599: finishing copy_from to smithi07694878-11 2021-03-15T22:35:09.387 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 2424 (ObjNum 1189 snap 416 seq_num 1189) dirty exists 2021-03-15T22:35:09.387 INFO:tasks.rados.rados.0.smithi076.stdout:3599: finishing copy_from racing read to smithi07694878-11 2021-03-15T22:35:09.387 INFO:tasks.rados.rados.0.smithi076.stdout:3598: done (2 left) 2021-03-15T22:35:09.387 INFO:tasks.rados.rados.0.smithi076.stdout:3599: done (1 left) 2021-03-15T22:35:09.388 INFO:tasks.rados.rados.0.smithi076.stdout:3602: delete oid 31 current snap is 425 2021-03-15T22:35:09.394 INFO:tasks.rados.rados.0.smithi076.stdout:3601: finishing rollback tid 0 to smithi07694878-1 2021-03-15T22:35:09.395 INFO:tasks.rados.rados.0.smithi076.stdout:3601: finishing rollback tid 1 to smithi07694878-1 2021-03-15T22:35:09.395 INFO:tasks.rados.rados.0.smithi076.stdout:3601: finishing rollback tid 2 to smithi07694878-1 2021-03-15T22:35:09.395 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 2792 (ObjNum 982 snap 326 seq_num 982) dirty exists 2021-03-15T22:35:09.396 INFO:tasks.rados.rados.0.smithi076.stdout:3601: done (1 left) 2021-03-15T22:35:09.396 INFO:tasks.rados.rados.0.smithi076.stdout:3602: done (0 left) 2021-03-15T22:35:09.396 INFO:tasks.rados.rados.0.smithi076.stdout:3603: write oid 7 current snap is 425 2021-03-15T22:35:09.396 INFO:tasks.rados.rados.0.smithi076.stdout:3603: seq_num 1227 ranges {622189=663356,1984347=458752} 2021-03-15T22:35:09.401 INFO:tasks.rados.rados.0.smithi076.stdout:3603: writing smithi07694878-7 from 622189 to 1285545 tid 1 2021-03-15T22:35:09.408 INFO:tasks.rados.rados.0.smithi076.stdout:3603: writing smithi07694878-7 from 1984347 to 2443099 tid 2 2021-03-15T22:35:09.408 INFO:tasks.rados.rados.0.smithi076.stdout:3604: write oid 29 current snap is 425 2021-03-15T22:35:09.409 INFO:tasks.rados.rados.0.smithi076.stdout:3603: finishing write tid 1 to smithi07694878-7 2021-03-15T22:35:09.409 INFO:tasks.rados.rados.0.smithi076.stdout:3604: seq_num 1228 ranges {638996=738605,1926476=712520} 2021-03-15T22:35:09.415 INFO:tasks.rados.rados.0.smithi076.stdout:3604: writing smithi07694878-29 from 638996 to 1377601 tid 1 2021-03-15T22:35:09.423 INFO:tasks.rados.rados.0.smithi076.stdout:3604: writing smithi07694878-29 from 1926476 to 2638996 tid 2 2021-03-15T22:35:09.423 INFO:tasks.rados.rados.0.smithi076.stdout:3605: read oid 47 snap 340 2021-03-15T22:35:09.423 INFO:tasks.rados.rados.0.smithi076.stdout:3605: expect (ObjNum 1002 snap 335 seq_num 1002) 2021-03-15T22:35:09.423 INFO:tasks.rados.rados.0.smithi076.stdout:3606: read oid 3 snap -1 2021-03-15T22:35:09.424 INFO:tasks.rados.rados.0.smithi076.stdout:3606: expect (ObjNum 1207 snap 418 seq_num 1207) 2021-03-15T22:35:09.424 INFO:tasks.rados.rados.0.smithi076.stdout:3607: snap_remove snap 297 2021-03-15T22:35:09.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:09.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:09.505 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:09.506 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:09.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[56514]: cluster 2021-03-15T22:35:08.374880+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e953: 8 total, 8 up, 8 in 2021-03-15T22:35:09.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[56514]: cluster 2021-03-15T22:35:08.402636+0000 osd.7 (osd.7) 176 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:35:09.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: cluster 2021-03-15T22:35:09.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: 2021-03-15T22:35:09.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: 2021 2021-03-15T22:35:09.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: - 2021-03-15T22:35:09.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: 03 2021-03-15T22:35:09.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: - 2021-03-15T22:35:09.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: 15 2021-03-15T22:35:09.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: T 2021-03-15T22:35:09.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: 22:35:08.374880 2021-03-15T22:35:09.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e953: 8 total, 8 up, 8 in 2021-03-15T22:35:09.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:35:09.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: 03-15T22:35:08.402636+0000 osd.7 (osd 2021-03-15T22:35:09.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:09 smithi076 conmon[62156]: .7) 176 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:35:09.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:09.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:09.630 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:09.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:09.732 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:09.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:09.754 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:09.755 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:09.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:09 smithi125 conmon[56739]: cluster 2021-03-15T22:35:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:09 smithi125 conmon[56739]: 2021-03-15T22:35: 2021-03-15T22:35:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:09 smithi125 conmon[56739]: 08.374880+0000 mon.a (mon.0) 2390 : cluster [DBG] osdmap e953: 8 total, 8 up, 8 in 2021-03-15T22:35:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:09 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:35:09.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:09 smithi125 conmon[56739]: 03-15T22:35:08.402636+0000 osd.7 ( 2021-03-15T22:35:09.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:09 smithi125 conmon[56739]: osd.7) 176 : cluster [DBG] 4.0 scrub ok 2021-03-15T22:35:09.855 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:09.877 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:09.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:09.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:09.880 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:35:09.880 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:35:09.881 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:35:09.881 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:35:09.881 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:35:09.881 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:35:09.882 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:35:09.882 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:35:09.882 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:35:09.882 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:35:09.883 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:35:09.883 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:35:09.883 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:35:09.883 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:35:09.883 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:35:09.884 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:35:09.979 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:10.001 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:10.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:10.002 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:10.043 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:10.103 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:10.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:10.130 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:10.131 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:10.232 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:10.255 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:10.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:10.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:10.341 INFO:tasks.rados.rados.0.smithi076.stdout:3607: done (4 left) 2021-03-15T22:35:10.342 INFO:tasks.rados.rados.0.smithi076.stdout:3608: write (excl) oid 46 current snap is 425 2021-03-15T22:35:10.342 INFO:tasks.rados.rados.0.smithi076.stdout:3608: seq_num 1229 ranges {655803=413854,1584958=462421} 2021-03-15T22:35:10.344 INFO:tasks.rados.rados.0.smithi076.stdout:3608: writing smithi07694878-46 from 655803 to 1069657 tid 1 2021-03-15T22:35:10.350 INFO:tasks.rados.rados.0.smithi076.stdout:3608: writing smithi07694878-46 from 1584958 to 2047379 tid 2 2021-03-15T22:35:10.351 INFO:tasks.rados.rados.0.smithi076.stdout:3609: rollback oid 13 current snap is 425 2021-03-15T22:35:10.351 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 13 to 360 2021-03-15T22:35:10.354 INFO:tasks.rados.rados.0.smithi076.stdout:3603: finishing write tid 2 to smithi07694878-7 2021-03-15T22:35:10.355 INFO:tasks.rados.rados.0.smithi076.stdout:3603: finishing write tid 3 to smithi07694878-7 2021-03-15T22:35:10.355 INFO:tasks.rados.rados.0.smithi076.stdout:3603: finishing write tid 4 to smithi07694878-7 2021-03-15T22:35:10.355 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2806 (ObjNum 1227 snap 425 seq_num 1227) dirty exists 2021-03-15T22:35:10.355 INFO:tasks.rados.rados.0.smithi076.stdout:3603: left oid 7 (ObjNum 1227 snap 425 seq_num 1227) 2021-03-15T22:35:10.355 INFO:tasks.rados.rados.0.smithi076.stdout:3604: finishing write tid 1 to smithi07694878-29 2021-03-15T22:35:10.356 INFO:tasks.rados.rados.0.smithi076.stdout:3606: expect (ObjNum 1207 snap 418 seq_num 1207) 2021-03-15T22:35:10.358 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:10.380 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:10.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:10.381 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:10.455 INFO:tasks.rados.rados.0.smithi076.stdout:3605: expect (ObjNum 1002 snap 335 seq_num 1002) 2021-03-15T22:35:10.483 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:10.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:10.506 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:10.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:10.607 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:10.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:10.633 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:10.633 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:10.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[56514]: cluster 2021-03-15T22:35:09.215451+0000 mgr.y (mgr.14144) 1065 : cluster [DBG] pgmap v1577: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+snaptrim, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 683 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 743 KiB/s rd, 960 KiB/s wr, 3 op/s; 1/1932 objects degraded (0.052%); 46/1932 objects misplaced (2.381%) 2021-03-15T22:35:10.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[56514]: 2021-03-15T22:35:10.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[56514]: cluster 2021-03-15T22:35:09.379501+0000 mon.a (mon.0) 2391 : cluster [DBG] osdmap e954: 8 total, 8 up, 8 in 2021-03-15T22:35:10.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[56514]: audit 2021-03-15T22:35:09.637368+0000 mon.a (mon.0) 2392 : audit [DBG] from='client.? 172.21.15.76:0/3831564037' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:10.677 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[56514]: cluster 2021-03-15T22:35:10.324706+0000 mon.a (mon.0) 2393 : cluster [WRN] Health check update: Degraded data redundancy: 1/1932 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:10.677 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[56514]: cluster 2021-03-15T22:35:10.328657+0000 mon.a (mon.0) 2394 : cluster [DBG] osdmap e955: 8 total, 8 up, 8 in 2021-03-15T22:35:10.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: cluster 2021-03-15T22:35:09.215451+0000 mgr.y 2021-03-15T22:35:10.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: (mgr.14144) 1065 : cluster [DBG] pgmap v1577: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+snaptrim, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 683 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 743 KiB/s rd, 960 KiB/s wr, 3 op/s; 1/1932 objects degraded (0.052%); 46/1932 objects misplaced (2.381%) 2021-03-15T22:35:10.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: cluster 2021-03-15T22:35:09.379501+0000 mon.a (mon 2021-03-15T22:35:10.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: .0) 2391 : cluster [DBG] osdmap e954: 8 total, 8 up, 8 in 2021-03-15T22:35:10.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:35:10.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: 03-15T22:35:09.637368+0000 mon.a (mon.0) 2021-03-15T22:35:10.679 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: 2392 : audit [DBG] from='client.? 172.21.15.76:0/3831564037' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:10.679 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:35:10.679 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: T22:35:10.324706+0000 mon.a (mon 2021-03-15T22:35:10.679 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: .0) 2393 : cluster [WRN] Health check update: Degraded data redundancy: 1/1932 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:10.680 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:35:10.680 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: 03-15T22:35:10.328657+0000 2021-03-15T22:35:10.680 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: mon.a (mon.0) 2394 : cluster [DBG] 2021-03-15T22:35:10.680 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:10 smithi076 conmon[62156]: osdmap e955: 8 total, 8 up, 8 in 2021-03-15T22:35:10.734 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:10.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:10.757 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:10.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:10.779 INFO:tasks.rados.rados.0.smithi076.stdout:3604: finishing write tid 2 to smithi07694878-29 2021-03-15T22:35:10.779 INFO:tasks.rados.rados.0.smithi076.stdout:3604: finishing write tid 3 to smithi07694878-29 2021-03-15T22:35:10.780 INFO:tasks.rados.rados.0.smithi076.stdout:3603: done (5 left) 2021-03-15T22:35:10.780 INFO:tasks.rados.rados.0.smithi076.stdout:3605: done (4 left) 2021-03-15T22:35:10.780 INFO:tasks.rados.rados.0.smithi076.stdout:3606: done (3 left) 2021-03-15T22:35:10.780 INFO:tasks.rados.rados.0.smithi076.stdout:3610: copy_from oid 1 from oid 45 current snap is 425 2021-03-15T22:35:10.781 INFO:tasks.rados.rados.0.smithi076.stdout:3604: finishing write tid 4 to smithi07694878-29 2021-03-15T22:35:10.781 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 3163 (ObjNum 1228 snap 425 seq_num 1228) dirty exists 2021-03-15T22:35:10.781 INFO:tasks.rados.rados.0.smithi076.stdout:3604: left oid 29 (ObjNum 1228 snap 425 seq_num 1228) 2021-03-15T22:35:10.781 INFO:tasks.rados.rados.0.smithi076.stdout:3608: finishing write tid 1 to smithi07694878-46 2021-03-15T22:35:10.782 INFO:tasks.rados.rados.0.smithi076.stdout:3608: finishing write tid 2 to smithi07694878-46 2021-03-15T22:35:10.782 INFO:tasks.rados.rados.0.smithi076.stdout:3608: finishing write tid 3 to smithi07694878-46 2021-03-15T22:35:10.782 INFO:tasks.rados.rados.0.smithi076.stdout:3608: finishing write tid 4 to smithi07694878-46 2021-03-15T22:35:10.782 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2290 (ObjNum 1229 snap 425 seq_num 1229) dirty exists 2021-03-15T22:35:10.783 INFO:tasks.rados.rados.0.smithi076.stdout:3608: left oid 46 (ObjNum 1229 snap 425 seq_num 1229) 2021-03-15T22:35:10.783 INFO:tasks.rados.rados.0.smithi076.stdout:3609: finishing rollback tid 0 to smithi07694878-13 2021-03-15T22:35:10.783 INFO:tasks.rados.rados.0.smithi076.stdout:3609: finishing rollback tid 1 to smithi07694878-13 2021-03-15T22:35:10.783 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 2426 (ObjNum 1902432464 snap 32564 seq_num 264691200) dirty dne 2021-03-15T22:35:10.783 INFO:tasks.rados.rados.0.smithi076.stdout:3604: done (3 left) 2021-03-15T22:35:10.784 INFO:tasks.rados.rados.0.smithi076.stdout:3608: done (2 left) 2021-03-15T22:35:10.784 INFO:tasks.rados.rados.0.smithi076.stdout:3609: done (1 left) 2021-03-15T22:35:10.784 INFO:tasks.rados.rados.0.smithi076.stdout:3611: copy_from oid 28 from oid 9 current snap is 425 2021-03-15T22:35:10.784 INFO:tasks.rados.rados.0.smithi076.stdout:3612: snap_remove snap 392 2021-03-15T22:35:10.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:35:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: cluster 2021-03-15T22:35:09.215451+0000 2021-03-15T22:35:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: mgr.y (mgr.14144) 1065 : cluster [DBG] pgmap v1577: 158 pgs: 1 active+remapped+backfill_toofull, 2 active+clean+snaptrim, 1 active+recovering+undersized+degraded+remapped, 154 active+clean; 683 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 743 KiB/s rd, 960 KiB/s wr, 3 op/s; 1/1932 objects degraded (0.052%); 46/1932 objects misplaced (2.381%) 2021-03-15T22:35:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: cluster 2021-03-15T22:35:10.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: 2021-03 2021-03-15T22:35:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: -15T22:35:09.379501+0000 2021-03-15T22:35:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: mon.a (mon.0) 2391 : 2021-03-15T22:35:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: cluster [DBG] osdmap e954: 8 total, 8 up, 8 in 2021-03-15T22:35:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:35:10.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: 03-15T22:35:09.637368+0000 2021-03-15T22:35:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: mon.a (mon.0) 2392 : 2021-03-15T22:35:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: audit [DBG] from='client.? 172.21.15.76:0/3831564037' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:35:10.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: 03-15T22:35:10.324706+0000 2021-03-15T22:35:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: mon.a (mon.0) 2393 2021-03-15T22:35:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: : cluster [WRN] Health check update: Degraded data redundancy: 1/1932 objects degraded (0.052%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:35:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: 03-15T22:35:10.328657+0000 2021-03-15T22:35:10.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: mon.a (mon.0) 2394 2021-03-15T22:35:10.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:10 smithi125 conmon[56739]: : cluster [DBG] osdmap e955: 8 total, 8 up, 8 in 2021-03-15T22:35:10.858 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:10.880 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:10.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:10.881 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:10.982 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:11.006 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:11.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:11.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:11.109 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:11.133 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:11.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:11.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:11.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:11.258 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:11.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:11.260 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:11.361 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:11.383 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:11.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:11.384 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:11.399 INFO:tasks.rados.rados.0.smithi076.stdout:3612: done (2 left) 2021-03-15T22:35:11.400 INFO:tasks.rados.rados.0.smithi076.stdout:3613: read oid 15 snap 404 2021-03-15T22:35:11.400 INFO:tasks.rados.rados.0.smithi076.stdout:3613: expect (ObjNum 1070 snap 364 seq_num 1070) 2021-03-15T22:35:11.400 INFO:tasks.rados.rados.0.smithi076.stdout:3614: delete oid 17 current snap is 425 2021-03-15T22:35:11.400 INFO:tasks.rados.rados.0.smithi076.stdout:3611: finishing copy_from to smithi07694878-28 2021-03-15T22:35:11.401 INFO:tasks.rados.rados.0.smithi076.stdout:3611: got expected ENOENT (src dne) 2021-03-15T22:35:11.401 INFO:tasks.rados.rados.0.smithi076.stdout:3611: finishing copy_from racing read to smithi07694878-28 2021-03-15T22:35:11.401 INFO:tasks.rados.rados.0.smithi076.stdout:3610: finishing copy_from to smithi07694878-1 2021-03-15T22:35:11.401 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 3076 (ObjNum 1186 snap 414 seq_num 1186) dirty exists 2021-03-15T22:35:11.401 INFO:tasks.rados.rados.0.smithi076.stdout:3610: finishing copy_from racing read to smithi07694878-1 2021-03-15T22:35:11.404 INFO:tasks.rados.rados.0.smithi076.stdout:3613: expect (ObjNum 1070 snap 364 seq_num 1070) 2021-03-15T22:35:11.485 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:11.507 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:11.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:11.510 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:11.595 INFO:tasks.rados.rados.0.smithi076.stdout:3610: done (3 left) 2021-03-15T22:35:11.596 INFO:tasks.rados.rados.0.smithi076.stdout:3611: done (2 left) 2021-03-15T22:35:11.596 INFO:tasks.rados.rados.0.smithi076.stdout:3613: done (1 left) 2021-03-15T22:35:11.596 INFO:tasks.rados.rados.0.smithi076.stdout:3614: done (0 left) 2021-03-15T22:35:11.596 INFO:tasks.rados.rados.0.smithi076.stdout:3615: snap_remove snap 391 2021-03-15T22:35:11.611 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:11.634 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:11.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:11.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:11.685 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:35:11.688 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:35:11.737 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:11.759 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:11.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:11.760 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:11.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:11 smithi076 conmon[56514]: cluster 2021-03-15T22:35:10.068024+0000 osd.1 (osd.1) 178 : cluster [DBG] 5.1d scrub ok 2021-03-15T22:35:11.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:11 smithi076 conmon[56514]: audit 2021-03-15T22:35:10.287505+0000 mgr.y (mgr.14144) 1066 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:11.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:11 smithi076 conmon[56514]: cluster 2021-03-15T22:35:10.452800+0000 osd.7 (osd.7) 177 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:35:11.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:11 smithi076 conmon[62156]: cluster 2021-03-15T22:35:10.068024+0000 osd.1 (osd.1) 178 : cluster [DBG] 5.1d scrub ok 2021-03-15T22:35:11.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:11 smithi076 conmon[62156]: audit 2021-03-15T22:35:10.287505+0000 mgr.y (mgr.14144) 1066 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:11.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:11 smithi076 conmon[62156]: cluster 2021-03-15T22:35:10.452800+0000 osd.7 (osd.7) 177 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:35:11.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:11 smithi125 conmon[56739]: cluster 2021-03-15T22:35:10.068024 2021-03-15T22:35:11.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:11 smithi125 conmon[56739]: +0000 osd.1 (osd.1) 178 : cluster [DBG] 5.1d scrub ok 2021-03-15T22:35:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:11 smithi125 conmon[56739]: audit 2021-03-15T22:35 2021-03-15T22:35:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:11 smithi125 conmon[56739]: :10.287505+0000 mgr.y (mgr.14144) 1066 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:11 smithi125 conmon[56739]: cluster 2021-03-15T22:35:10.452800+0000 2021-03-15T22:35:11.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:11 smithi125 conmon[56739]: osd.7 (osd.7) 177 : cluster [DBG] 2.0 scrub ok 2021-03-15T22:35:11.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:11.883 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:11.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:11.885 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:11.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:12.009 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:12.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:12.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:12.067 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:35:12.112 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:12.135 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:12.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:12.136 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:12.237 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:12.259 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:12.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:12.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:12.362 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:12.387 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:12.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:12.389 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:12.401 INFO:tasks.rados.rados.0.smithi076.stdout:3615: done (0 left) 2021-03-15T22:35:12.402 INFO:tasks.rados.rados.0.smithi076.stdout:3616: delete oid 25 current snap is 425 2021-03-15T22:35:12.409 INFO:tasks.rados.rados.0.smithi076.stdout:3616: done (0 left) 2021-03-15T22:35:12.410 INFO:tasks.rados.rados.0.smithi076.stdout:3617: write (excl) oid 46 current snap is 425 2021-03-15T22:35:12.410 INFO:tasks.rados.rados.0.smithi076.stdout:3617: seq_num 1232 ranges {706224=755954,2160404=545820} 2021-03-15T22:35:12.418 INFO:tasks.rados.rados.0.smithi076.stdout:3617: writing smithi07694878-46 from 706224 to 1462178 tid 1 2021-03-15T22:35:12.425 INFO:tasks.rados.rados.0.smithi076.stdout:3617: writing smithi07694878-46 from 2160404 to 2706224 tid 2 2021-03-15T22:35:12.425 INFO:tasks.rados.rados.0.smithi076.stdout:3618: delete oid 34 current snap is 425 2021-03-15T22:35:12.425 INFO:tasks.rados.rados.0.smithi076.stdout:3617: finishing write tid 1 to smithi07694878-46 2021-03-15T22:35:12.427 INFO:tasks.rados.rados.0.smithi076.stdout:3618: done (1 left) 2021-03-15T22:35:12.427 INFO:tasks.rados.rados.0.smithi076.stdout:3619: write (excl) oid 39 current snap is 425 2021-03-15T22:35:12.427 INFO:tasks.rados.rados.0.smithi076.stdout:3619: seq_num 1234 ranges {739838=506452,1877368=613417} 2021-03-15T22:35:12.434 INFO:tasks.rados.rados.0.smithi076.stdout:3619: writing smithi07694878-39 from 739838 to 1246290 tid 1 2021-03-15T22:35:12.441 INFO:tasks.rados.rados.0.smithi076.stdout:3619: writing smithi07694878-39 from 1877368 to 2490785 tid 2 2021-03-15T22:35:12.441 INFO:tasks.rados.rados.0.smithi076.stdout:3620: snap_create 2021-03-15T22:35:12.441 INFO:tasks.rados.rados.0.smithi076.stdout:3617: finishing write tid 2 to smithi07694878-46 2021-03-15T22:35:12.442 INFO:tasks.rados.rados.0.smithi076.stdout:3617: finishing write tid 3 to smithi07694878-46 2021-03-15T22:35:12.442 INFO:tasks.rados.rados.0.smithi076.stdout:3617: finishing write tid 4 to smithi07694878-46 2021-03-15T22:35:12.442 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2293 (ObjNum 1232 snap 425 seq_num 1232) dirty exists 2021-03-15T22:35:12.442 INFO:tasks.rados.rados.0.smithi076.stdout:3617: left oid 46 (ObjNum 1232 snap 425 seq_num 1232) 2021-03-15T22:35:12.443 INFO:tasks.rados.rados.0.smithi076.stdout:3619: finishing write tid 1 to smithi07694878-39 2021-03-15T22:35:12.444 INFO:tasks.rados.rados.0.smithi076.stdout:3619: finishing write tid 2 to smithi07694878-39 2021-03-15T22:35:12.444 INFO:tasks.rados.rados.0.smithi076.stdout:3619: finishing write tid 3 to smithi07694878-39 2021-03-15T22:35:12.445 INFO:tasks.rados.rados.0.smithi076.stdout:3619: finishing write tid 4 to smithi07694878-39 2021-03-15T22:35:12.445 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2809 (ObjNum 1234 snap 425 seq_num 1234) dirty exists 2021-03-15T22:35:12.445 INFO:tasks.rados.rados.0.smithi076.stdout:3619: left oid 39 (ObjNum 1234 snap 425 seq_num 1234) 2021-03-15T22:35:12.490 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:12.513 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:12.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:12.514 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:12.615 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:12.636 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1581,"stamp":"2021-03-15T22:35:11.390965+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":706413128,"num_objects":636,"num_object_clones":271,"num_object_copies":1908,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":44,"num_objects_unfound":0,"num_objects_dirty":636,"num_whiteouts":12,"num_read":9130,"num_read_kb":6751870,"num_write":5753,"num_write_kb":1872058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5825,"num_bytes_recovered":2139876644,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22532,"ondisk_log_size":22532,"up":474,"acting":473,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":476,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1997636,"kb_used_data":1603652,"kb_used_omap":273,"kb_used_meta":393646,"kb_avail":747963580,"statfs":{"total":767960285184,"available":765914705920,"internally_reserved":0,"allocated":1642139648,"data_stored":2277338250,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":403094394},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-10127841,"num_objects":-8,"num_object_clones":-8,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":-2,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":-2,"num_read":23,"num_read_kb":10768,"num_write":25,"num_write_kb":7852,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.016511"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"455","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.369214+0000","last_change":"2021-03-15T22:34:55.350872+0000","last_active":"2021-03-15T22:35:07.369214+0000","last_peered":"2021-03-15T22:35:07.369214+0000","last_clean":"2021-03-15T22:35:07.369214+0000","last_became_active":"2021-03-15T22:34:55.350524+0000","last_became_peered":"2021-03-15T22:34:55.350524+0000","last_unstale":"2021-03-15T22:35:07.369214+0000","last_undegraded":"2021-03-15T22:35:07.369214+0000","last_fullsized":"2021-03-15T22:35:07.369214+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"23","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.374359+0000","last_change":"2021-03-15T22:34:55.349985+0000","last_active":"2021-03-15T22:35:07.374359+0000","last_peered":"2021-03-15T22:35:07.374359+0000","last_clean":"2021-03-15T22:35:07.374359+0000","last_became_active":"2021-03-15T22:34:55.348740+0000","last_became_peered":"2021-03-15T22:34:55.348740+0000","last_unstale":"2021-03-15T22:35:07.374359+0000","last_undegraded":"2021-03-15T22:35:07.374359+0000","last_fullsized":"2021-03-15T22:35:07.374359+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"44","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.384372+0000","last_change":"2021-03-15T22:34:36.653657+0000","last_active":"2021-03-15T22:35:09.384372+0000","last_peered":"2021-03-15T22:35:09.384372+0000","last_clean":"2021-03-15T22:35:09.384372+0000","last_became_active":"2021-03-15T22:34:36.653285+0000","last_became_peered":"2021-03-15T22:34:36.653285+0000","last_unstale":"2021-03-15T22:35:09.384372+0000","last_undegraded":"2021-03-15T22:35:09.384372+0000","last_fullsized":"2021-03-15T22:35:09.384372+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.761137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"41","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.369210+0000","last_change":"2021-03-15T22:34:36.657886+0000","last_active":"2021-03-15T22:35:06.369210+0000","last_peered":"2021-03-15T22:35:06.369210+0000","last_clean":"2021-03-15T22:35:06.369210+0000","last_became_active":"2021-03-15T22:34:36.657561+0000","last_became_peered":"2021-03-15T22:34:36.657561+0000","last_unstale":"2021-03-15T22:35:06.369210+0000","last_undegraded":"2021-03-15T22:35:06.369210+0000","last_fullsized":"2021-03-15T22:35:06.369210+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"990","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.386724+0000","last_change":"2021-03-15T22:34:55.346316+0000","last_active":"2021-03-15T22:35:09.386724+0000","last_peered":"2021-03-15T22:35:09.386724+0000","last_clean":"2021-03-15T22:35:09.386724+0000","last_became_active":"2021-03-15T22:34:55.345943+0000","last_became_peered":"2021-03-15T22:34:55.345943+0000","last_unstale":"2021-03-15T22:35:09.386724+0000","last_undegraded":"2021-03-15T22:35:09.386724+0000","last_fullsized":"2021-03-15T22:35:09.386724+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:32.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"762","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.378994+0000","last_change":"2021-03-15T22:34:55.355408+0000","last_active":"2021-03-15T22:35:07.378994+0000","last_peered":"2021-03-15T22:35:07.378994+0000","last_clean":"2021-03-15T22:35:07.378994+0000","last_became_active":"2021-03-15T22:34:55.355062+0000","last_became_peered":"2021-03-15T22:34:55.355062+0000","last_unstale":"2021-03-15T22:35:07.378994+0000","last_undegraded":"2021-03-15T22:35:07.378994+0000","last_fullsized":"2021-03-15T22:35:07.378994+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"968","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.384141+0000","last_change":"2021-03-15T22:34:36.652890+0000","last_active":"2021-03-15T22:35:09.384141+0000","last_peered":"2021-03-15T22:35:09.384141+0000","last_clean":"2021-03-15T22:35:09.384141+0000","last_became_active":"2021-03-15T22:34:36.652531+0000","last_became_peered":"2021-03-15T22:34:36.652531+0000","last_unstale":"2021-03-15T22:35:09.384141+0000","last_undegraded":"2021-03-15T22:35:09.384141+0000","last_fullsized":"2021-03-15T22:35:09.384141+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.507204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.507204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1023","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.384876+0000","last_change":"2021-03-15T22:34:55.344965+0000","last_active":"2021-03-15T22:35:09.384876+0000","last_peered":"2021-03-15T22:35:09.384876+0000","last_clean":"2021-03-15T22:35:09.384876+0000","last_became_active":"2021-03-15T22:34:55.343726+0000","last_became_peered":"2021-03-15T22:34:55.343726+0000","last_unstale":"2021-03-15T22:35:09.384876+0000","last_undegraded":"2021-03-15T22:35:09.384876+0000","last_fullsized":"2021-03-15T22:35:09.384876+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"985","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.369896+0000","last_change":"2021-03-15T22:34:55.351171+0000","last_active":"2021-03-15T22:35:07.369896+0000","last_peered":"2021-03-15T22:35:07.369896+0000","last_clean":"2021-03-15T22:35:07.369896+0000","last_became_active":"2021-03-15T22:34:55.350597+0000","last_became_peered":"2021-03-15T22:34:55.350597+0000","last_unstale":"2021-03-15T22:35:07.369896+0000","last_undegraded":"2021-03-15T22:35:07.369896+0000","last_fullsized":"2021-03-15T22:35:07.369896+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.701955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.701955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"50","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.368714+0000","last_change":"2021-03-15T22:34:55.350719+0000","last_active":"2021-03-15T22:35:06.368714+0000","last_peered":"2021-03-15T22:35:06.368714+0000","last_clean":"2021-03-15T22:35:06.368714+0000","last_became_active":"2021-03-15T22:34:55.350384+0000","last_became_peered":"2021-03-15T22:34:55.350384+0000","last_unstale":"2021-03-15T22:35:06.368714+0000","last_undegraded":"2021-03-15T22:35:06.368714+0000","last_fullsized":"2021-03-15T22:35:06.368714+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"317'7","reported_seq":"316","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.368984+0000","last_change":"2021-03-15T22:34:39.122745+0000","last_active":"2021-03-15T22:35:06.368984+0000","last_peered":"2021-03-15T22:35:06.368984+0000","last_clean":"2021-03-15T22:35:06.368984+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:35:06.368984+0000","last_undegraded":"2021-03-15T22:35:06.368984+0000","last_fullsized":"2021-03-15T22:35:06.368984+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:34:39.122665+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.122665+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"966","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362122+0000","last_change":"2021-03-15T22:34:10.331764+0000","last_active":"2021-03-15T22:35:05.362122+0000","last_peered":"2021-03-15T22:35:05.362122+0000","last_clean":"2021-03-15T22:35:05.362122+0000","last_became_active":"2021-03-15T22:34:10.331424+0000","last_became_peered":"2021-03-15T22:34:10.331424+0000","last_unstale":"2021-03-15T22:35:05.362122+0000","last_undegraded":"2021-03-15T22:35:05.362122+0000","last_fullsized":"2021-03-15T22:35:05.362122+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.564464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.564464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"482","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.380901+0000","last_change":"2021-03-15T22:34:55.389531+0000","last_active":"2021-03-15T22:35:09.380901+0000","last_peered":"2021-03-15T22:35:09.380901+0000","last_clean":"2021-03-15T22:35:09.380901+0000","last_became_active":"2021-03-15T22:34:55.356022+0000","last_became_peered":"2021-03-15T22:34:55.356022+0000","last_unstale":"2021-03-15T22:35:09.380901+0000","last_undegraded":"2021-03-15T22:35:09.380901+0000","last_fullsized":"2021-03-15T22:35:09.380901+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:32:31.680111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.680111+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"1044","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362051+0000","last_change":"2021-03-15T22:33:06.085397+0000","last_active":"2021-03-15T22:35:05.362051+0000","last_peered":"2021-03-15T22:35:05.362051+0000","last_clean":"2021-03-15T22:35:05.362051+0000","last_became_active":"2021-03-15T22:33:04.345393+0000","last_became_peered":"2021-03-15T22:33:04.345393+0000","last_unstale":"2021-03-15T22:35:05.362051+0000","last_undegraded":"2021-03-15T22:35:05.362051+0000","last_fullsized":"2021-03-15T22:35:05.362051+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.177019+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"253","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.361841+0000","last_change":"2021-03-15T22:33:04.345933+0000","last_active":"2021-03-15T22:35:05.361841+0000","last_peered":"2021-03-15T22:35:05.361841+0000","last_clean":"2021-03-15T22:35:05.361841+0000","last_became_active":"2021-03-15T22:33:04.345518+0000","last_became_peered":"2021-03-15T22:33:04.345518+0000","last_unstale":"2021-03-15T22:35:05.361841+0000","last_undegraded":"2021-03-15T22:35:05.361841+0000","last_fullsized":"2021-03-15T22:35:05.361841+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.338100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.338100+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"33","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370336+0000","last_change":"2021-03-15T22:34:43.345093+0000","last_active":"2021-03-15T22:35:05.370336+0000","last_peered":"2021-03-15T22:35:05.370336+0000","last_clean":"2021-03-15T22:35:05.370336+0000","last_became_active":"2021-03-15T22:34:43.344764+0000","last_became_peered":"2021-03-15T22:34:43.344764+0000","last_unstale":"2021-03-15T22:35:05.370336+0000","last_undegraded":"2021-03-15T22:35:05.370336+0000","last_fullsized":"2021-03-15T22:35:05.370336+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:08.652741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.652741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"42","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370714+0000","last_change":"2021-03-15T22:34:55.347147+0000","last_active":"2021-03-15T22:35:05.370714+0000","last_peered":"2021-03-15T22:35:05.370714+0000","last_clean":"2021-03-15T22:35:05.370714+0000","last_became_active":"2021-03-15T22:34:55.346711+0000","last_became_peered":"2021-03-15T22:34:55.346711+0000","last_unstale":"2021-03-15T22:35:05.370714+0000","last_undegraded":"2021-03-15T22:35:05.370714+0000","last_fullsized":"2021-03-15T22:35:05.370714+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.475850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.475850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"94","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370220+0000","last_change":"2021-03-15T22:34:55.399761+0000","last_active":"2021-03-15T22:35:05.370220+0000","last_peered":"2021-03-15T22:35:05.370220+0000","last_clean":"2021-03-15T22:35:05.370220+0000","last_became_active":"2021-03-15T22:34:55.346148+0000","last_became_peered":"2021-03-15T22:34:55.346148+0000","last_unstale":"2021-03-15T22:35:05.370220+0000","last_undegraded":"2021-03-15T22:35:05.370220+0000","last_fullsized":"2021-03-15T22:35:05.370220+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.652898+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":5340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"798","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.378858+0000","last_change":"2021-03-15T22:34:44.798780+0000","last_active":"2021-03-15T22:35:07.378858+0000","last_peered":"2021-03-15T22:35:07.378858+0000","last_clean":"2021-03-15T22:35:07.378858+0000","last_became_active":"2021-03-15T22:34:03.570306+0000","last_became_peered":"2021-03-15T22:34:03.570306+0000","last_unstale":"2021-03-15T22:35:07.378858+0000","last_undegraded":"2021-03-15T22:35:07.378858+0000","last_fullsized":"2021-03-15T22:35:07.378858+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.794113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.794113+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"419","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.378179+0000","last_change":"2021-03-15T22:32:01.452949+0000","last_active":"2021-03-15T22:35:09.378179+0000","last_peered":"2021-03-15T22:35:09.378179+0000","last_clean":"2021-03-15T22:35:09.378179+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:35:09.378179+0000","last_undegraded":"2021-03-15T22:35:09.378179+0000","last_fullsized":"2021-03-15T22:35:09.378179+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:01.452885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:32:01.452885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"346","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.379414+0000","last_change":"2021-03-15T22:33:39.756677+0000","last_active":"2021-03-15T22:35:07.379414+0000","last_peered":"2021-03-15T22:35:07.379414+0000","last_clean":"2021-03-15T22:35:07.379414+0000","last_became_active":"2021-03-15T22:33:04.325611+0000","last_became_peered":"2021-03-15T22:33:04.325611+0000","last_unstale":"2021-03-15T22:35:07.379414+0000","last_undegraded":"2021-03-15T22:35:07.379414+0000","last_fullsized":"2021-03-15T22:35:07.379414+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.756581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"1100","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362292+0000","last_change":"2021-03-15T22:33:17.502781+0000","last_active":"2021-03-15T22:35:05.362292+0000","last_peered":"2021-03-15T22:35:05.362292+0000","last_clean":"2021-03-15T22:35:05.362292+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:35:05.362292+0000","last_undegraded":"2021-03-15T22:35:05.362292+0000","last_fullsized":"2021-03-15T22:35:05.362292+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:33:17.502703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.502703+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"233","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.370230+0000","last_change":"2021-03-15T22:32:38.476370+0000","last_active":"2021-03-15T22:35:07.370230+0000","last_peered":"2021-03-15T22:35:07.370230+0000","last_clean":"2021-03-15T22:35:07.370230+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:35:07.370230+0000","last_undegraded":"2021-03-15T22:35:07.370230+0000","last_fullsized":"2021-03-15T22:35:07.370230+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.476269+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.476269+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"591","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.378959+0000","last_change":"2021-03-15T22:34:43.344692+0000","last_active":"2021-03-15T22:35:07.378959+0000","last_peered":"2021-03-15T22:35:07.378959+0000","last_clean":"2021-03-15T22:35:07.378959+0000","last_became_active":"2021-03-15T22:34:43.344401+0000","last_became_peered":"2021-03-15T22:34:43.344401+0000","last_unstale":"2021-03-15T22:35:07.378959+0000","last_undegraded":"2021-03-15T22:35:07.378959+0000","last_fullsized":"2021-03-15T22:35:07.378959+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.774079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.774079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"328","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.380324+0000","last_change":"2021-03-15T22:32:14.756556+0000","last_active":"2021-03-15T22:35:07.380324+0000","last_peered":"2021-03-15T22:35:07.380324+0000","last_clean":"2021-03-15T22:35:07.380324+0000","last_became_active":"2021-03-15T22:29:51.234868+0000","last_became_peered":"2021-03-15T22:29:51.234868+0000","last_unstale":"2021-03-15T22:35:07.380324+0000","last_undegraded":"2021-03-15T22:35:07.380324+0000","last_fullsized":"2021-03-15T22:35:07.380324+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.756460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.756460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"1089","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362610+0000","last_change":"2021-03-15T22:34:55.357888+0000","last_active":"2021-03-15T22:35:05.362610+0000","last_peered":"2021-03-15T22:35:05.362610+0000","last_clean":"2021-03-15T22:35:05.362610+0000","last_became_active":"2021-03-15T22:34:55.335550+0000","last_became_peered":"2021-03-15T22:34:55.335550+0000","last_unstale":"2021-03-15T22:35:05.362610+0000","last_undegraded":"2021-03-15T22:35:05.362610+0000","last_fullsized":"2021-03-15T22:35:05.362610+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:32:05.111031+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.111031+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"908","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.369615+0000","last_change":"2021-03-15T22:34:45.001265+0000","last_active":"2021-03-15T22:35:06.369615+0000","last_peered":"2021-03-15T22:35:06.369615+0000","last_clean":"2021-03-15T22:35:06.369615+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:35:06.369615+0000","last_undegraded":"2021-03-15T22:35:06.369615+0000","last_fullsized":"2021-03-15T22:35:06.369615+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:32:32.744716+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.744716+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"973","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362787+0000","last_change":"2021-03-15T22:34:41.614750+0000","last_active":"2021-03-15T22:35:05.362787+0000","last_peered":"2021-03-15T22:35:05.362787+0000","last_clean":"2021-03-15T22:35:05.362787+0000","last_became_active":"2021-03-15T22:33:04.329078+0000","last_became_peered":"2021-03-15T22:33:04.329078+0000","last_unstale":"2021-03-15T22:35:05.362787+0000","last_undegraded":"2021-03-15T22:35:05.362787+0000","last_fullsized":"2021-03-15T22:35:05.362787+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.614684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.614684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"423","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.379909+0000","last_change":"2021-03-15T22:32:41.673856+0000","last_active":"2021-03-15T22:35:09.379909+0000","last_peered":"2021-03-15T22:35:09.379909+0000","last_clean":"2021-03-15T22:35:09.379909+0000","last_became_active":"2021-03-15T22:28:12.519619+0000","last_became_peered":"2021-03-15T22:28:12.519619+0000","last_unstale":"2021-03-15T22:35:09.379909+0000","last_undegraded":"2021-03-15T22:35:09.379909+0000","last_fullsized":"2021-03-15T22:35:09.379909+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:32:41.673792+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.673792+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"510","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.369922+0000","last_change":"2021-03-15T22:34:55.366147+0000","last_active":"2021-03-15T22:35:06.369922+0000","last_peered":"2021-03-15T22:35:06.369922+0000","last_clean":"2021-03-15T22:35:06.369922+0000","last_became_active":"2021-03-15T22:34:55.350325+0000","last_became_peered":"2021-03-15T22:34:55.350325+0000","last_unstale":"2021-03-15T22:35:06.369922+0000","last_undegraded":"2021-03-15T22:35:06.369922+0000","last_fullsized":"2021-03-15T22:35:06.369922+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:34:38.119156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:38.119156+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"997","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362940+0000","last_change":"2021-03-15T22:34:55.338366+0000","last_active":"2021-03-15T22:35:05.362940+0000","last_peered":"2021-03-15T22:35:05.362940+0000","last_clean":"2021-03-15T22:35:05.362940+0000","last_became_active":"2021-03-15T22:34:55.338001+0000","last_became_peered":"2021-03-15T22:34:55.338001+0000","last_unstale":"2021-03-15T22:35:05.362940+0000","last_undegraded":"2021-03-15T22:35:05.362940+0000","last_fullsized":"2021-03-15T22:35:05.362940+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.144132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.144132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"783","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.371013+0000","last_change":"2021-03-15T22:33:04.337162+0000","last_active":"2021-03-15T22:35:05.371013+0000","last_peered":"2021-03-15T22:35:05.371013+0000","last_clean":"2021-03-15T22:35:05.371013+0000","last_became_active":"2021-03-15T22:33:04.336804+0000","last_became_peered":"2021-03-15T22:33:04.336804+0000","last_unstale":"2021-03-15T22:35:05.371013+0000","last_undegraded":"2021-03-15T22:35:05.371013+0000","last_fullsized":"2021-03-15T22:35:05.371013+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.594717+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:45.403769+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.594717+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"963","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362390+0000","last_change":"2021-03-15T22:34:55.343334+0000","last_active":"2021-03-15T22:35:05.362390+0000","last_peered":"2021-03-15T22:35:05.362390+0000","last_clean":"2021-03-15T22:35:05.362390+0000","last_became_active":"2021-03-15T22:34:55.342358+0000","last_became_peered":"2021-03-15T22:34:55.342358+0000","last_unstale":"2021-03-15T22:35:05.362390+0000","last_undegraded":"2021-03-15T22:35:05.362390+0000","last_fullsized":"2021-03-15T22:35:05.362390+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.429029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.429029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"399","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370595+0000","last_change":"2021-03-15T22:33:15.617975+0000","last_active":"2021-03-15T22:35:05.370595+0000","last_peered":"2021-03-15T22:35:05.370595+0000","last_clean":"2021-03-15T22:35:05.370595+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:35:05.370595+0000","last_undegraded":"2021-03-15T22:35:05.370595+0000","last_fullsized":"2021-03-15T22:35:05.370595+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:33:15.617891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.617891+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"891'22","reported_seq":"511","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.384977+0000","last_change":"2021-03-15T22:34:06.311981+0000","last_active":"2021-03-15T22:35:09.384977+0000","last_peered":"2021-03-15T22:35:09.384977+0000","last_clean":"2021-03-15T22:35:09.384977+0000","last_became_active":"2021-03-15T22:34:06.311478+0000","last_became_peered":"2021-03-15T22:34:06.311478+0000","last_unstale":"2021-03-15T22:35:09.384977+0000","last_undegraded":"2021-03-15T22:35:09.384977+0000","last_fullsized":"2021-03-15T22:35:09.384977+0000","mapping_epoch":890,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"841'19","last_scrub_stamp":"2021-03-15T22:33:20.272520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.272520+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"984","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.370843+0000","last_change":"2021-03-15T22:34:55.352256+0000","last_active":"2021-03-15T22:35:07.370843+0000","last_peered":"2021-03-15T22:35:07.370843+0000","last_clean":"2021-03-15T22:35:07.370843+0000","last_became_active":"2021-03-15T22:34:55.351597+0000","last_became_peered":"2021-03-15T22:34:55.351597+0000","last_unstale":"2021-03-15T22:35:07.370843+0000","last_undegraded":"2021-03-15T22:35:07.370843+0000","last_fullsized":"2021-03-15T22:35:07.370843+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:43.393275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:43.393275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"981","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.369065+0000","last_change":"2021-03-15T22:34:49.015597+0000","last_active":"2021-03-15T22:35:06.369065+0000","last_peered":"2021-03-15T22:35:06.369065+0000","last_clean":"2021-03-15T22:35:06.369065+0000","last_became_active":"2021-03-15T22:28:59.264559+0000","last_became_peered":"2021-03-15T22:28:59.264559+0000","last_unstale":"2021-03-15T22:35:06.369065+0000","last_undegraded":"2021-03-15T22:35:06.369065+0000","last_fullsized":"2021-03-15T22:35:06.369065+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:43.703019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.703019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"557","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.382353+0000","last_change":"2021-03-15T22:34:10.385041+0000","last_active":"2021-03-15T22:35:09.382353+0000","last_peered":"2021-03-15T22:35:09.382353+0000","last_clean":"2021-03-15T22:35:09.382353+0000","last_became_active":"2021-03-15T22:34:10.347227+0000","last_became_peered":"2021-03-15T22:34:10.347227+0000","last_unstale":"2021-03-15T22:35:09.382353+0000","last_undegraded":"2021-03-15T22:35:09.382353+0000","last_fullsized":"2021-03-15T22:35:09.382353+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:34:07.125155+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.125155+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":3736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"1033","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.381781+0000","last_change":"2021-03-15T22:34:58.356466+0000","last_active":"2021-03-15T22:35:09.381781+0000","last_peered":"2021-03-15T22:35:09.381781+0000","last_clean":"2021-03-15T22:35:09.381781+0000","last_became_active":"2021-03-15T22:34:58.353086+0000","last_became_peered":"2021-03-15T22:34:58.353086+0000","last_unstale":"2021-03-15T22:35:09.381781+0000","last_undegraded":"2021-03-15T22:35:09.381781+0000","last_fullsized":"2021-03-15T22:35:09.381781+0000","mapping_epoch":942,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:32:34.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.739637+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"957","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.370055+0000","last_change":"2021-03-15T22:33:11.491281+0000","last_active":"2021-03-15T22:35:06.370055+0000","last_peered":"2021-03-15T22:35:06.370055+0000","last_clean":"2021-03-15T22:35:06.370055+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:35:06.370055+0000","last_undegraded":"2021-03-15T22:35:06.370055+0000","last_fullsized":"2021-03-15T22:35:06.370055+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:11.491199+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:11.491199+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"576","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370243+0000","last_change":"2021-03-15T22:34:55.345698+0000","last_active":"2021-03-15T22:35:05.370243+0000","last_peered":"2021-03-15T22:35:05.370243+0000","last_clean":"2021-03-15T22:35:05.370243+0000","last_became_active":"2021-03-15T22:34:55.345277+0000","last_became_peered":"2021-03-15T22:34:55.345277+0000","last_unstale":"2021-03-15T22:35:05.370243+0000","last_undegraded":"2021-03-15T22:35:05.370243+0000","last_fullsized":"2021-03-15T22:35:05.370243+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.844183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.844183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"163","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.384899+0000","last_change":"2021-03-15T22:34:43.447677+0000","last_active":"2021-03-15T22:35:09.384899+0000","last_peered":"2021-03-15T22:35:09.384899+0000","last_clean":"2021-03-15T22:35:09.384899+0000","last_became_active":"2021-03-15T22:33:04.336100+0000","last_became_peered":"2021-03-15T22:33:04.336100+0000","last_unstale":"2021-03-15T22:35:09.384899+0000","last_undegraded":"2021-03-15T22:35:09.384899+0000","last_fullsized":"2021-03-15T22:35:09.384899+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:34:43.447602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:43.447602+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"966","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.370301+0000","last_change":"2021-03-15T22:34:55.351849+0000","last_active":"2021-03-15T22:35:07.370301+0000","last_peered":"2021-03-15T22:35:07.370301+0000","last_clean":"2021-03-15T22:35:07.370301+0000","last_became_active":"2021-03-15T22:34:55.351089+0000","last_became_peered":"2021-03-15T22:34:55.351089+0000","last_unstale":"2021-03-15T22:35:07.370301+0000","last_undegraded":"2021-03-15T22:35:07.370301+0000","last_fullsized":"2021-03-15T22:35:07.370301+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.469521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.469521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"134","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.386172+0000","last_change":"2021-03-15T22:33:17.305262+0000","last_active":"2021-03-15T22:35:09.386172+0000","last_peered":"2021-03-15T22:35:09.386172+0000","last_clean":"2021-03-15T22:35:09.386172+0000","last_became_active":"2021-03-15T22:33:04.335734+0000","last_became_peered":"2021-03-15T22:33:04.335734+0000","last_unstale":"2021-03-15T22:35:09.386172+0000","last_undegraded":"2021-03-15T22:35:09.386172+0000","last_fullsized":"2021-03-15T22:35:09.386172+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.305181+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.305181+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"476","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.380820+0000","last_change":"2021-03-15T22:34:43.350810+0000","last_active":"2021-03-15T22:35:09.380820+0000","last_peered":"2021-03-15T22:35:09.380820+0000","last_clean":"2021-03-15T22:35:09.380820+0000","last_became_active":"2021-03-15T22:34:43.350301+0000","last_became_peered":"2021-03-15T22:34:43.350301+0000","last_unstale":"2021-03-15T22:35:09.380820+0000","last_undegraded":"2021-03-15T22:35:09.380820+0000","last_fullsized":"2021-03-15T22:35:09.380820+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:33.136112+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:05.736770+0000","last_clean_scrub_stamp":"2021-03-15T22:31:33.136112+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"463","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370107+0000","last_change":"2021-03-15T22:33:37.752077+0000","last_active":"2021-03-15T22:35:05.370107+0000","last_peered":"2021-03-15T22:35:05.370107+0000","last_clean":"2021-03-15T22:35:05.370107+0000","last_became_active":"2021-03-15T22:33:04.342749+0000","last_became_peered":"2021-03-15T22:33:04.342749+0000","last_unstale":"2021-03-15T22:35:05.370107+0000","last_undegraded":"2021-03-15T22:35:05.370107+0000","last_fullsized":"2021-03-15T22:35:05.370107+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:33:37.752011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.752011+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":8764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"576","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370882+0000","last_change":"2021-03-15T22:34:34.783107+0000","last_active":"2021-03-15T22:35:05.370882+0000","last_peered":"2021-03-15T22:35:05.370882+0000","last_clean":"2021-03-15T22:35:05.370882+0000","last_became_active":"2021-03-15T22:33:04.333077+0000","last_became_peered":"2021-03-15T22:33:04.333077+0000","last_unstale":"2021-03-15T22:35:05.370882+0000","last_undegraded":"2021-03-15T22:35:05.370882+0000","last_fullsized":"2021-03-15T22:35:05.370882+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.783047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.783047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"72","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370138+0000","last_change":"2021-03-15T22:34:03.571320+0000","last_active":"2021-03-15T22:35:05.370138+0000","last_peered":"2021-03-15T22:35:05.370138+0000","last_clean":"2021-03-15T22:35:05.370138+0000","last_became_active":"2021-03-15T22:34:03.571049+0000","last_became_peered":"2021-03-15T22:34:03.571049+0000","last_unstale":"2021-03-15T22:35:05.370138+0000","last_undegraded":"2021-03-15T22:35:05.370138+0000","last_fullsized":"2021-03-15T22:35:05.370138+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:05.522372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.522372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"328","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.381083+0000","last_change":"2021-03-15T22:33:12.882245+0000","last_active":"2021-03-15T22:35:07.381083+0000","last_peered":"2021-03-15T22:35:07.381083+0000","last_clean":"2021-03-15T22:35:07.381083+0000","last_became_active":"2021-03-15T22:29:51.220261+0000","last_became_peered":"2021-03-15T22:29:51.220261+0000","last_unstale":"2021-03-15T22:35:07.381083+0000","last_undegraded":"2021-03-15T22:35:07.381083+0000","last_fullsized":"2021-03-15T22:35:07.381083+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.882164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.882164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"794","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370320+0000","last_change":"2021-03-15T22:34:37.787302+0000","last_active":"2021-03-15T22:35:05.370320+0000","last_peered":"2021-03-15T22:35:05.370320+0000","last_clean":"2021-03-15T22:35:05.370320+0000","last_became_active":"2021-03-15T22:33:04.331607+0000","last_became_peered":"2021-03-15T22:33:04.331607+0000","last_unstale":"2021-03-15T22:35:05.370320+0000","last_undegraded":"2021-03-15T22:35:05.370320+0000","last_fullsized":"2021-03-15T22:35:05.370320+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:34:37.787235+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.787235+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"438","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.378086+0000","last_change":"2021-03-15T22:31:43.405661+0000","last_active":"2021-03-15T22:35:09.378086+0000","last_peered":"2021-03-15T22:35:09.378086+0000","last_clean":"2021-03-15T22:35:09.378086+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:35:09.378086+0000","last_undegraded":"2021-03-15T22:35:09.378086+0000","last_fullsized":"2021-03-15T22:35:09.378086+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:43.405600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:31:43.405600+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"743","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370803+0000","last_change":"2021-03-15T22:32:37.690585+0000","last_active":"2021-03-15T22:35:05.370803+0000","last_peered":"2021-03-15T22:35:05.370803+0000","last_clean":"2021-03-15T22:35:05.370803+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:35:05.370803+0000","last_undegraded":"2021-03-15T22:35:05.370803+0000","last_fullsized":"2021-03-15T22:35:05.370803+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:37.690525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:32:37.690525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"954","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.430778+0000","last_change":"2021-03-15T22:35:07.430778+0000","last_active":"2021-03-15T22:35:07.430778+0000","last_peered":"2021-03-15T22:35:07.430778+0000","last_clean":"2021-03-15T22:35:07.430778+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:35:07.430778+0000","last_undegraded":"2021-03-15T22:35:07.430778+0000","last_fullsized":"2021-03-15T22:35:07.430778+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.438742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.438742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"670","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.385151+0000","last_change":"2021-03-15T22:34:56.210802+0000","last_active":"2021-03-15T22:35:09.385151+0000","last_peered":"2021-03-15T22:35:09.385151+0000","last_clean":"2021-03-15T22:35:09.385151+0000","last_became_active":"2021-03-15T22:34:55.344555+0000","last_became_peered":"2021-03-15T22:34:55.344555+0000","last_unstale":"2021-03-15T22:35:09.385151+0000","last_undegraded":"2021-03-15T22:35:09.385151+0000","last_fullsized":"2021-03-15T22:35:09.385151+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:34:42.447631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.447631+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"335","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.379422+0000","last_change":"2021-03-15T22:34:55.355928+0000","last_active":"2021-03-15T22:35:09.379422+0000","last_peered":"2021-03-15T22:35:09.379422+0000","last_clean":"2021-03-15T22:35:09.379422+0000","last_became_active":"2021-03-15T22:34:55.355119+0000","last_became_peered":"2021-03-15T22:34:55.355119+0000","last_unstale":"2021-03-15T22:35:09.379422+0000","last_undegraded":"2021-03-15T22:35:09.379422+0000","last_fullsized":"2021-03-15T22:35:09.379422+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:06.670324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:32:06.670324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"1010","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.369683+0000","last_change":"2021-03-15T22:34:10.353649+0000","last_active":"2021-03-15T22:35:06.369683+0000","last_peered":"2021-03-15T22:35:06.369683+0000","last_clean":"2021-03-15T22:35:06.369683+0000","last_became_active":"2021-03-15T22:34:10.353314+0000","last_became_peered":"2021-03-15T22:34:10.353314+0000","last_unstale":"2021-03-15T22:35:06.369683+0000","last_undegraded":"2021-03-15T22:35:06.369683+0000","last_fullsized":"2021-03-15T22:35:06.369683+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:43.268612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.268612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"86","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362540+0000","last_change":"2021-03-15T22:33:45.314503+0000","last_active":"2021-03-15T22:35:05.362540+0000","last_peered":"2021-03-15T22:35:05.362540+0000","last_clean":"2021-03-15T22:35:05.362540+0000","last_became_active":"2021-03-15T22:33:45.314042+0000","last_became_peered":"2021-03-15T22:33:45.314042+0000","last_unstale":"2021-03-15T22:35:05.362540+0000","last_undegraded":"2021-03-15T22:35:05.362540+0000","last_fullsized":"2021-03-15T22:35:05.362540+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:08.884878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.884878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"1037","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362505+0000","last_change":"2021-03-15T22:34:42.632034+0000","last_active":"2021-03-15T22:35:05.362505+0000","last_peered":"2021-03-15T22:35:05.362505+0000","last_clean":"2021-03-15T22:35:05.362505+0000","last_became_active":"2021-03-15T22:33:04.340460+0000","last_became_peered":"2021-03-15T22:33:04.340460+0000","last_unstale":"2021-03-15T22:35:05.362505+0000","last_undegraded":"2021-03-15T22:35:05.362505+0000","last_fullsized":"2021-03-15T22:35:05.362505+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:34:42.631962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.631962+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"446","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.382605+0000","last_change":"2021-03-15T22:34:10.327350+0000","last_active":"2021-03-15T22:35:09.382605+0000","last_peered":"2021-03-15T22:35:09.382605+0000","last_clean":"2021-03-15T22:35:09.382605+0000","last_became_active":"2021-03-15T22:34:10.326881+0000","last_became_peered":"2021-03-15T22:34:10.326881+0000","last_unstale":"2021-03-15T22:35:09.382605+0000","last_undegraded":"2021-03-15T22:35:09.382605+0000","last_fullsized":"2021-03-15T22:35:09.382605+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:19.560387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:19.560387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"808'9","reported_seq":"397","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.379505+0000","last_change":"2021-03-15T22:32:38.881054+0000","last_active":"2021-03-15T22:35:07.379505+0000","last_peered":"2021-03-15T22:35:07.379505+0000","last_clean":"2021-03-15T22:35:07.379505+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:35:07.379505+0000","last_undegraded":"2021-03-15T22:35:07.379505+0000","last_fullsized":"2021-03-15T22:35:07.379505+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9","last_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_deep_scrub":"808'9","last_deep_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.880993+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"951","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.370188+0000","last_change":"2021-03-15T22:34:55.351414+0000","last_active":"2021-03-15T22:35:07.370188+0000","last_peered":"2021-03-15T22:35:07.370188+0000","last_clean":"2021-03-15T22:35:07.370188+0000","last_became_active":"2021-03-15T22:34:55.351042+0000","last_became_peered":"2021-03-15T22:34:55.351042+0000","last_unstale":"2021-03-15T22:35:07.370188+0000","last_undegraded":"2021-03-15T22:35:07.370188+0000","last_fullsized":"2021-03-15T22:35:07.370188+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.516668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.516668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"844'33","reported_seq":"809","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.371119+0000","last_change":"2021-03-15T22:34:55.387712+0000","last_active":"2021-03-15T22:35:05.371119+0000","last_peered":"2021-03-15T22:35:05.371119+0000","last_clean":"2021-03-15T22:35:05.371119+0000","last_became_active":"2021-03-15T22:34:55.351721+0000","last_became_peered":"2021-03-15T22:34:55.351721+0000","last_unstale":"2021-03-15T22:35:05.371119+0000","last_undegraded":"2021-03-15T22:35:05.371119+0000","last_fullsized":"2021-03-15T22:35:05.371119+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"844'33","last_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_deep_scrub":"844'33","last_deep_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.622437+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"983","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362910+0000","last_change":"2021-03-15T22:34:55.340800+0000","last_active":"2021-03-15T22:35:05.362910+0000","last_peered":"2021-03-15T22:35:05.362910+0000","last_clean":"2021-03-15T22:35:05.362910+0000","last_became_active":"2021-03-15T22:34:55.340440+0000","last_became_peered":"2021-03-15T22:34:55.340440+0000","last_unstale":"2021-03-15T22:35:05.362910+0000","last_undegraded":"2021-03-15T22:35:05.362910+0000","last_fullsized":"2021-03-15T22:35:05.362910+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:41.407324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.407324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"941'17","reported_seq":"472","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.370760+0000","last_change":"2021-03-15T22:34:55.473459+0000","last_active":"2021-03-15T22:35:07.370760+0000","last_peered":"2021-03-15T22:35:07.370760+0000","last_clean":"2021-03-15T22:35:07.370760+0000","last_became_active":"2021-03-15T22:34:55.351682+0000","last_became_peered":"2021-03-15T22:34:55.351682+0000","last_unstale":"2021-03-15T22:35:07.370760+0000","last_undegraded":"2021-03-15T22:35:07.370760+0000","last_fullsized":"2021-03-15T22:35:07.370760+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"896'16","last_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_deep_scrub":"896'16","last_deep_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_clean_scrub_stamp":"2021-03-15T22:34:19.400191+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"446","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.379611+0000","last_change":"2021-03-15T22:34:10.347844+0000","last_active":"2021-03-15T22:35:09.379611+0000","last_peered":"2021-03-15T22:35:09.379611+0000","last_clean":"2021-03-15T22:35:09.379611+0000","last_became_active":"2021-03-15T22:34:10.347512+0000","last_became_peered":"2021-03-15T22:34:10.347512+0000","last_unstale":"2021-03-15T22:35:09.379611+0000","last_undegraded":"2021-03-15T22:35:09.379611+0000","last_fullsized":"2021-03-15T22:35:09.379611+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:03.502854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:32:03.502854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"346","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.379334+0000","last_change":"2021-03-15T22:32:41.879032+0000","last_active":"2021-03-15T22:35:07.379334+0000","last_peered":"2021-03-15T22:35:07.379334+0000","last_clean":"2021-03-15T22:35:07.379334+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:35:07.379334+0000","last_undegraded":"2021-03-15T22:35:07.379334+0000","last_fullsized":"2021-03-15T22:35:07.379334+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:32:41.878949+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.878949+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"33","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370724+0000","last_change":"2021-03-15T22:34:43.345092+0000","last_active":"2021-03-15T22:35:05.370724+0000","last_peered":"2021-03-15T22:35:05.370724+0000","last_clean":"2021-03-15T22:35:05.370724+0000","last_became_active":"2021-03-15T22:34:43.344790+0000","last_became_peered":"2021-03-15T22:34:43.344790+0000","last_unstale":"2021-03-15T22:35:05.370724+0000","last_undegraded":"2021-03-15T22:35:05.370724+0000","last_fullsized":"2021-03-15T22:35:05.370724+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.621539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"23","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.374285+0000","last_change":"2021-03-15T22:34:55.349824+0000","last_active":"2021-03-15T22:35:07.374285+0000","last_peered":"2021-03-15T22:35:07.374285+0000","last_clean":"2021-03-15T22:35:07.374285+0000","last_became_active":"2021-03-15T22:34:55.348663+0000","last_became_peered":"2021-03-15T22:34:55.348663+0000","last_unstale":"2021-03-15T22:35:07.374285+0000","last_undegraded":"2021-03-15T22:35:07.374285+0000","last_fullsized":"2021-03-15T22:35:07.374285+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:42.894499+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.224872+0000","last_clean_scrub_stamp":"2021-03-15T22:33:42.894499+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"944","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.370059+0000","last_change":"2021-03-15T22:33:21.545910+0000","last_active":"2021-03-15T22:35:07.370059+0000","last_peered":"2021-03-15T22:35:07.370059+0000","last_clean":"2021-03-15T22:35:07.370059+0000","last_became_active":"2021-03-15T22:33:04.330195+0000","last_became_peered":"2021-03-15T22:33:04.330195+0000","last_unstale":"2021-03-15T22:35:07.370059+0000","last_undegraded":"2021-03-15T22:35:07.370059+0000","last_fullsized":"2021-03-15T22:35:07.370059+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.545826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.545826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"482","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.381933+0000","last_change":"2021-03-15T22:34:37.435070+0000","last_active":"2021-03-15T22:35:09.381933+0000","last_peered":"2021-03-15T22:35:09.381933+0000","last_clean":"2021-03-15T22:35:09.381933+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:35:09.381933+0000","last_undegraded":"2021-03-15T22:35:09.381933+0000","last_fullsized":"2021-03-15T22:35:09.381933+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:34:37.435006+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.435006+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"750","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.381160+0000","last_change":"2021-03-15T22:34:46.754611+0000","last_active":"2021-03-15T22:35:07.381160+0000","last_peered":"2021-03-15T22:35:07.381160+0000","last_clean":"2021-03-15T22:35:07.381160+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:35:07.381160+0000","last_undegraded":"2021-03-15T22:35:07.381160+0000","last_fullsized":"2021-03-15T22:35:07.381160+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:11.834104+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:11.834104+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"938","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.371137+0000","last_change":"2021-03-15T22:33:19.496857+0000","last_active":"2021-03-15T22:35:07.371137+0000","last_peered":"2021-03-15T22:35:07.371137+0000","last_clean":"2021-03-15T22:35:07.371137+0000","last_became_active":"2021-03-15T22:33:04.331013+0000","last_became_peered":"2021-03-15T22:33:04.331013+0000","last_unstale":"2021-03-15T22:35:07.371137+0000","last_undegraded":"2021-03-15T22:35:07.371137+0000","last_fullsized":"2021-03-15T22:35:07.371137+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:19.496753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:19.496753+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"134","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.384609+0000","last_change":"2021-03-15T22:34:07.203702+0000","last_active":"2021-03-15T22:35:09.384609+0000","last_peered":"2021-03-15T22:35:09.384609+0000","last_clean":"2021-03-15T22:35:09.384609+0000","last_became_active":"2021-03-15T22:33:04.336886+0000","last_became_peered":"2021-03-15T22:33:04.336886+0000","last_unstale":"2021-03-15T22:35:09.384609+0000","last_undegraded":"2021-03-15T22:35:09.384609+0000","last_fullsized":"2021-03-15T22:35:09.384609+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.203639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:39.697935+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.203639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"302","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.371600+0000","last_change":"2021-03-15T22:34:55.387006+0000","last_active":"2021-03-15T22:35:07.371600+0000","last_peered":"2021-03-15T22:35:07.371600+0000","last_clean":"2021-03-15T22:35:07.371600+0000","last_became_active":"2021-03-15T22:34:55.351977+0000","last_became_peered":"2021-03-15T22:34:55.351977+0000","last_unstale":"2021-03-15T22:35:07.371600+0000","last_undegraded":"2021-03-15T22:35:07.371600+0000","last_fullsized":"2021-03-15T22:35:07.371600+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:34:36.438541+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:34:36.438541+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"952'1324","reported_seq":"4305","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.384909+0000","last_change":"2021-03-15T22:35:07.386337+0000","last_active":"2021-03-15T22:35:07.384909+0000","last_peered":"2021-03-15T22:35:07.384909+0000","last_clean":"2021-03-15T22:35:07.384909+0000","last_became_active":"2021-03-15T22:33:08.355490+0000","last_became_peered":"2021-03-15T22:33:08.355490+0000","last_unstale":"2021-03-15T22:35:07.384909+0000","last_undegraded":"2021-03-15T22:35:07.384909+0000","last_fullsized":"2021-03-15T22:35:07.384909+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'1088","last_scrub_stamp":"2021-03-15T22:33:10.457078+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:10.457078+0000","log_size":1324,"ondisk_log_size":1324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46526635,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":457,"num_read_kb":504188,"num_write":315,"num_write_kb":111433,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":84869361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"895","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.379203+0000","last_change":"2021-03-15T22:33:42.741529+0000","last_active":"2021-03-15T22:35:07.379203+0000","last_peered":"2021-03-15T22:35:07.379203+0000","last_clean":"2021-03-15T22:35:07.379203+0000","last_became_active":"2021-03-15T22:33:04.340352+0000","last_became_peered":"2021-03-15T22:33:04.340352+0000","last_unstale":"2021-03-15T22:35:07.379203+0000","last_undegraded":"2021-03-15T22:35:07.379203+0000","last_fullsized":"2021-03-15T22:35:07.379203+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:42.741451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:42.741451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"874'4","reported_seq":"961","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.371512+0000","last_change":"2021-03-15T22:33:44.498327+0000","last_active":"2021-03-15T22:35:07.371512+0000","last_peered":"2021-03-15T22:35:07.371512+0000","last_clean":"2021-03-15T22:35:07.371512+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:35:07.371512+0000","last_undegraded":"2021-03-15T22:35:07.371512+0000","last_fullsized":"2021-03-15T22:35:07.371512+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"874'4","last_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_deep_scrub":"874'4","last_deep_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.498263+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"422","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.378814+0000","last_change":"2021-03-15T22:33:38.759477+0000","last_active":"2021-03-15T22:35:07.378814+0000","last_peered":"2021-03-15T22:35:07.378814+0000","last_clean":"2021-03-15T22:35:07.378814+0000","last_became_active":"2021-03-15T22:33:04.326070+0000","last_became_peered":"2021-03-15T22:33:04.326070+0000","last_unstale":"2021-03-15T22:35:07.378814+0000","last_undegraded":"2021-03-15T22:35:07.378814+0000","last_fullsized":"2021-03-15T22:35:07.378814+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:33:38.759395+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.759395+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"841'14","reported_seq":"1213","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.370161+0000","last_change":"2021-03-15T22:33:43.476783+0000","last_active":"2021-03-15T22:35:07.370161+0000","last_peered":"2021-03-15T22:35:07.370161+0000","last_clean":"2021-03-15T22:35:07.370161+0000","last_became_active":"2021-03-15T22:33:08.354031+0000","last_became_peered":"2021-03-15T22:33:08.354031+0000","last_unstale":"2021-03-15T22:35:07.370161+0000","last_undegraded":"2021-03-15T22:35:07.370161+0000","last_fullsized":"2021-03-15T22:35:07.370161+0000","mapping_epoch":837,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'14","last_scrub_stamp":"2021-03-15T22:33:43.476701+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.476701+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"951'1893","reported_seq":"5151","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.381397+0000","last_change":"2021-03-15T22:35:06.381626+0000","last_active":"2021-03-15T22:35:06.381397+0000","last_peered":"2021-03-15T22:35:06.381397+0000","last_clean":"2021-03-15T22:35:06.381397+0000","last_became_active":"2021-03-15T22:34:58.336162+0000","last_became_peered":"2021-03-15T22:34:58.336162+0000","last_unstale":"2021-03-15T22:35:06.381397+0000","last_undegraded":"2021-03-15T22:35:06.381397+0000","last_fullsized":"2021-03-15T22:35:06.381397+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1312","last_scrub_stamp":"2021-03-15T22:32:05.639057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.639057+0000","log_size":1893,"ondisk_log_size":1893,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":71530236,"num_objects":33,"num_object_clones":28,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":1,"num_read":650,"num_read_kb":625148,"num_write":482,"num_write_kb":193780,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":198498497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"29f","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"379","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370641+0000","last_change":"2021-03-15T22:32:42.621654+0000","last_active":"2021-03-15T22:35:05.370641+0000","last_peered":"2021-03-15T22:35:05.370641+0000","last_clean":"2021-03-15T22:35:05.370641+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:35:05.370641+0000","last_undegraded":"2021-03-15T22:35:05.370641+0000","last_fullsized":"2021-03-15T22:35:05.370641+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:42.621594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:42.621594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"928","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.371300+0000","last_change":"2021-03-15T22:33:22.542596+0000","last_active":"2021-03-15T22:35:07.371300+0000","last_peered":"2021-03-15T22:35:07.371300+0000","last_clean":"2021-03-15T22:35:07.371300+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:35:07.371300+0000","last_undegraded":"2021-03-15T22:35:07.371300+0000","last_fullsized":"2021-03-15T22:35:07.371300+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.542506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.542506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"1160","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362445+0000","last_change":"2021-03-15T22:34:11.031994+0000","last_active":"2021-03-15T22:35:05.362445+0000","last_peered":"2021-03-15T22:35:05.362445+0000","last_clean":"2021-03-15T22:35:05.362445+0000","last_became_active":"2021-03-15T22:34:10.342120+0000","last_became_peered":"2021-03-15T22:34:10.342120+0000","last_unstale":"2021-03-15T22:35:05.362445+0000","last_undegraded":"2021-03-15T22:35:05.362445+0000","last_fullsized":"2021-03-15T22:35:05.362445+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:34:07.452572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.452572+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"667","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.378111+0000","last_change":"2021-03-15T22:34:55.356651+0000","last_active":"2021-03-15T22:35:09.378111+0000","last_peered":"2021-03-15T22:35:09.378111+0000","last_clean":"2021-03-15T22:35:09.378111+0000","last_became_active":"2021-03-15T22:34:55.356328+0000","last_became_peered":"2021-03-15T22:34:55.356328+0000","last_unstale":"2021-03-15T22:35:09.378111+0000","last_undegraded":"2021-03-15T22:35:09.378111+0000","last_fullsized":"2021-03-15T22:35:09.378111+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"955'532","reported_seq":"3235","reported_epoch":"955","state":"active+clean","last_fresh":"2021-03-15T22:35:10.333644+0000","last_change":"2021-03-15T22:35:10.333821+0000","last_active":"2021-03-15T22:35:10.333644+0000","last_peered":"2021-03-15T22:35:10.333644+0000","last_clean":"2021-03-15T22:35:10.333644+0000","last_became_active":"2021-03-15T22:34:13.324667+0000","last_became_peered":"2021-03-15T22:34:13.324667+0000","last_unstale":"2021-03-15T22:35:10.333644+0000","last_undegraded":"2021-03-15T22:35:10.333644+0000","last_fullsized":"2021-03-15T22:35:10.333644+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'446","last_scrub_stamp":"2021-03-15T22:33:39.355306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.355306+0000","log_size":532,"ondisk_log_size":532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17601321,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":123,"num_read_kb":94114,"num_write":126,"num_write_kb":41271,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":23698344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"239","length":"1"},{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"},{"start":"324","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"996","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.369261+0000","last_change":"2021-03-15T22:34:55.354150+0000","last_active":"2021-03-15T22:35:07.369261+0000","last_peered":"2021-03-15T22:35:07.369261+0000","last_clean":"2021-03-15T22:35:07.369261+0000","last_became_active":"2021-03-15T22:34:55.353827+0000","last_became_peered":"2021-03-15T22:34:55.353827+0000","last_unstale":"2021-03-15T22:35:07.369261+0000","last_undegraded":"2021-03-15T22:35:07.369261+0000","last_fullsized":"2021-03-15T22:35:07.369261+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"23","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.372339+0000","last_change":"2021-03-15T22:34:55.354467+0000","last_active":"2021-03-15T22:35:07.372339+0000","last_peered":"2021-03-15T22:35:07.372339+0000","last_clean":"2021-03-15T22:35:07.372339+0000","last_became_active":"2021-03-15T22:34:55.353966+0000","last_became_peered":"2021-03-15T22:34:55.353966+0000","last_unstale":"2021-03-15T22:35:07.372339+0000","last_undegraded":"2021-03-15T22:35:07.372339+0000","last_fullsized":"2021-03-15T22:35:07.372339+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.809863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.809863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"983","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.371459+0000","last_change":"2021-03-15T22:33:04.331301+0000","last_active":"2021-03-15T22:35:07.371459+0000","last_peered":"2021-03-15T22:35:07.371459+0000","last_clean":"2021-03-15T22:35:07.371459+0000","last_became_active":"2021-03-15T22:33:04.330771+0000","last_became_peered":"2021-03-15T22:33:04.330771+0000","last_unstale":"2021-03-15T22:35:07.371459+0000","last_undegraded":"2021-03-15T22:35:07.371459+0000","last_fullsized":"2021-03-15T22:35:07.371459+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:40.430038+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:32:40.430038+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"199","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.385301+0000","last_change":"2021-03-15T22:34:10.249538+0000","last_active":"2021-03-15T22:35:09.385301+0000","last_peered":"2021-03-15T22:35:09.385301+0000","last_clean":"2021-03-15T22:35:09.385301+0000","last_became_active":"2021-03-15T22:33:45.318867+0000","last_became_peered":"2021-03-15T22:33:45.318867+0000","last_unstale":"2021-03-15T22:35:09.385301+0000","last_undegraded":"2021-03-15T22:35:09.385301+0000","last_fullsized":"2021-03-15T22:35:09.385301+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:34:10.249444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:10.249444+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":2928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"952'1172","reported_seq":"3950","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.391503+0000","last_change":"2021-03-15T22:35:07.391503+0000","last_active":"2021-03-15T22:35:07.391503+0000","last_peered":"2021-03-15T22:35:07.391503+0000","last_clean":"2021-03-15T22:35:07.391503+0000","last_became_active":"2021-03-15T22:34:59.339587+0000","last_became_peered":"2021-03-15T22:34:59.339587+0000","last_unstale":"2021-03-15T22:35:07.391503+0000","last_undegraded":"2021-03-15T22:35:07.391503+0000","last_fullsized":"2021-03-15T22:35:07.391503+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'1086","last_scrub_stamp":"2021-03-15T22:34:41.436839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.436839+0000","log_size":1172,"ondisk_log_size":1172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":61974788,"num_objects":27,"num_object_clones":24,"num_object_copies":81,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":27,"num_whiteouts":1,"num_read":399,"num_read_kb":471191,"num_write":276,"num_write_kb":99612,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":205148835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"457","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.379505+0000","last_change":"2021-03-15T22:34:55.356122+0000","last_active":"2021-03-15T22:35:09.379505+0000","last_peered":"2021-03-15T22:35:09.379505+0000","last_clean":"2021-03-15T22:35:09.379505+0000","last_became_active":"2021-03-15T22:34:55.355571+0000","last_became_peered":"2021-03-15T22:34:55.355571+0000","last_unstale":"2021-03-15T22:35:09.379505+0000","last_undegraded":"2021-03-15T22:35:09.379505+0000","last_fullsized":"2021-03-15T22:35:09.379505+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"784","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362490+0000","last_change":"2021-03-15T22:34:55.343547+0000","last_active":"2021-03-15T22:35:05.362490+0000","last_peered":"2021-03-15T22:35:05.362490+0000","last_clean":"2021-03-15T22:35:05.362490+0000","last_became_active":"2021-03-15T22:34:55.342808+0000","last_became_peered":"2021-03-15T22:34:55.342808+0000","last_unstale":"2021-03-15T22:35:05.362490+0000","last_undegraded":"2021-03-15T22:35:05.362490+0000","last_fullsized":"2021-03-15T22:35:05.362490+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:14.888119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.888119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"353","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.380252+0000","last_change":"2021-03-15T22:34:07.807466+0000","last_active":"2021-03-15T22:35:07.380252+0000","last_peered":"2021-03-15T22:35:07.380252+0000","last_clean":"2021-03-15T22:35:07.380252+0000","last_became_active":"2021-03-15T22:33:04.333291+0000","last_became_peered":"2021-03-15T22:33:04.333291+0000","last_unstale":"2021-03-15T22:35:07.380252+0000","last_undegraded":"2021-03-15T22:35:07.380252+0000","last_fullsized":"2021-03-15T22:35:07.380252+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.807411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.807411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"497","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.380001+0000","last_change":"2021-03-15T22:33:46.129541+0000","last_active":"2021-03-15T22:35:09.380001+0000","last_peered":"2021-03-15T22:35:09.380001+0000","last_clean":"2021-03-15T22:35:09.380001+0000","last_became_active":"2021-03-15T22:33:45.326747+0000","last_became_peered":"2021-03-15T22:33:45.326747+0000","last_unstale":"2021-03-15T22:35:09.380001+0000","last_undegraded":"2021-03-15T22:35:09.380001+0000","last_fullsized":"2021-03-15T22:35:09.380001+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:32:33.701675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.701675+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":2682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"955'1338","reported_seq":"4808","reported_epoch":"955","state":"active+clean","last_fresh":"2021-03-15T22:35:10.344316+0000","last_change":"2021-03-15T22:35:10.345135+0000","last_active":"2021-03-15T22:35:10.344316+0000","last_peered":"2021-03-15T22:35:10.344316+0000","last_clean":"2021-03-15T22:35:10.344316+0000","last_became_active":"2021-03-15T22:34:46.338888+0000","last_became_peered":"2021-03-15T22:34:46.338888+0000","last_unstale":"2021-03-15T22:35:10.344316+0000","last_undegraded":"2021-03-15T22:35:10.344316+0000","last_fullsized":"2021-03-15T22:35:10.344316+0000","mapping_epoch":930,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'1253","last_scrub_stamp":"2021-03-15T22:34:41.490879+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.490879+0000","log_size":1338,"ondisk_log_size":1338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42735322,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":308,"num_read_kb":250584,"num_write":327,"num_write_kb":113891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":229919087,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"239","length":"1"},{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"},{"start":"324","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"33","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.371158+0000","last_change":"2021-03-15T22:34:43.342616+0000","last_active":"2021-03-15T22:35:05.371158+0000","last_peered":"2021-03-15T22:35:05.371158+0000","last_clean":"2021-03-15T22:35:05.371158+0000","last_became_active":"2021-03-15T22:34:43.342089+0000","last_became_peered":"2021-03-15T22:34:43.342089+0000","last_unstale":"2021-03-15T22:35:05.371158+0000","last_undegraded":"2021-03-15T22:35:05.371158+0000","last_fullsized":"2021-03-15T22:35:05.371158+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"350","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.379082+0000","last_change":"2021-03-15T22:34:39.831492+0000","last_active":"2021-03-15T22:35:07.379082+0000","last_peered":"2021-03-15T22:35:07.379082+0000","last_clean":"2021-03-15T22:35:07.379082+0000","last_became_active":"2021-03-15T22:33:04.339912+0000","last_became_peered":"2021-03-15T22:33:04.339912+0000","last_unstale":"2021-03-15T22:35:07.379082+0000","last_undegraded":"2021-03-15T22:35:07.379082+0000","last_fullsized":"2021-03-15T22:35:07.379082+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.831369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.831369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"148","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362346+0000","last_change":"2021-03-15T22:33:04.336661+0000","last_active":"2021-03-15T22:35:05.362346+0000","last_peered":"2021-03-15T22:35:05.362346+0000","last_clean":"2021-03-15T22:35:05.362346+0000","last_became_active":"2021-03-15T22:33:04.335914+0000","last_became_peered":"2021-03-15T22:33:04.335914+0000","last_unstale":"2021-03-15T22:35:05.362346+0000","last_undegraded":"2021-03-15T22:35:05.362346+0000","last_fullsized":"2021-03-15T22:35:05.362346+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.051962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.051962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1320","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.370133+0000","last_change":"2021-03-15T22:34:55.382040+0000","last_active":"2021-03-15T22:35:06.370133+0000","last_peered":"2021-03-15T22:35:06.370133+0000","last_clean":"2021-03-15T22:35:06.370133+0000","last_became_active":"2021-03-15T22:34:55.350419+0000","last_became_peered":"2021-03-15T22:34:55.350419+0000","last_unstale":"2021-03-15T22:35:06.370133+0000","last_undegraded":"2021-03-15T22:35:06.370133+0000","last_fullsized":"2021-03-15T22:35:06.370133+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:33:38.133617+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.133617+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"419","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.379967+0000","last_change":"2021-03-15T22:33:15.492116+0000","last_active":"2021-03-15T22:35:09.379967+0000","last_peered":"2021-03-15T22:35:09.379967+0000","last_clean":"2021-03-15T22:35:09.379967+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:35:09.379967+0000","last_undegraded":"2021-03-15T22:35:09.379967+0000","last_fullsized":"2021-03-15T22:35:09.379967+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.492014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.492014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"955'1254","reported_seq":"3832","reported_epoch":"955","state":"active+clean","last_fresh":"2021-03-15T22:35:10.344785+0000","last_change":"2021-03-15T22:35:10.345005+0000","last_active":"2021-03-15T22:35:10.344785+0000","last_peered":"2021-03-15T22:35:10.344785+0000","last_clean":"2021-03-15T22:35:10.344785+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:35:10.344785+0000","last_undegraded":"2021-03-15T22:35:10.344785+0000","last_fullsized":"2021-03-15T22:35:10.344785+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"805'997","last_scrub_stamp":"2021-03-15T22:32:36.795301+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.795301+0000","log_size":1254,"ondisk_log_size":1254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":41192590,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":363,"num_read_kb":361421,"num_write":281,"num_write_kb":105474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"239","length":"1"},{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"},{"start":"324","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"979","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.371582+0000","last_change":"2021-03-15T22:34:34.631500+0000","last_active":"2021-03-15T22:35:07.371582+0000","last_peered":"2021-03-15T22:35:07.371582+0000","last_clean":"2021-03-15T22:35:07.371582+0000","last_became_active":"2021-03-15T22:34:34.631216+0000","last_became_peered":"2021-03-15T22:34:34.631216+0000","last_unstale":"2021-03-15T22:35:07.371582+0000","last_undegraded":"2021-03-15T22:35:07.371582+0000","last_fullsized":"2021-03-15T22:35:07.371582+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:32.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"984","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.369336+0000","last_change":"2021-03-15T22:34:55.347340+0000","last_active":"2021-03-15T22:35:06.369336+0000","last_peered":"2021-03-15T22:35:06.369336+0000","last_clean":"2021-03-15T22:35:06.369336+0000","last_became_active":"2021-03-15T22:34:55.347065+0000","last_became_peered":"2021-03-15T22:34:55.347065+0000","last_unstale":"2021-03-15T22:35:06.369336+0000","last_undegraded":"2021-03-15T22:35:06.369336+0000","last_fullsized":"2021-03-15T22:35:06.369336+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:10.222521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:10.222521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"23","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.371509+0000","last_change":"2021-03-15T22:34:55.355784+0000","last_active":"2021-03-15T22:35:07.371509+0000","last_peered":"2021-03-15T22:35:07.371509+0000","last_clean":"2021-03-15T22:35:07.371509+0000","last_became_active":"2021-03-15T22:34:55.355485+0000","last_became_peered":"2021-03-15T22:34:55.355485+0000","last_unstale":"2021-03-15T22:35:07.371509+0000","last_undegraded":"2021-03-15T22:35:07.371509+0000","last_fullsized":"2021-03-15T22:35:07.371509+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:34.095158+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.095158+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"436","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.379419+0000","last_change":"2021-03-15T22:34:10.343282+0000","last_active":"2021-03-15T22:35:07.379419+0000","last_peered":"2021-03-15T22:35:07.379419+0000","last_clean":"2021-03-15T22:35:07.379419+0000","last_became_active":"2021-03-15T22:34:10.322666+0000","last_became_peered":"2021-03-15T22:34:10.322666+0000","last_unstale":"2021-03-15T22:35:07.379419+0000","last_undegraded":"2021-03-15T22:35:07.379419+0000","last_fullsized":"2021-03-15T22:35:07.379419+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:33:40.214707+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.214707+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"23","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.374356+0000","last_change":"2021-03-15T22:34:55.353621+0000","last_active":"2021-03-15T22:35:07.374356+0000","last_peered":"2021-03-15T22:35:07.374356+0000","last_clean":"2021-03-15T22:35:07.374356+0000","last_became_active":"2021-03-15T22:34:55.353316+0000","last_became_peered":"2021-03-15T22:34:55.353316+0000","last_unstale":"2021-03-15T22:35:07.374356+0000","last_undegraded":"2021-03-15T22:35:07.374356+0000","last_fullsized":"2021-03-15T22:35:07.374356+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:38.907011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.907011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"955'3029","reported_seq":"6480","reported_epoch":"955","state":"active+clean","last_fresh":"2021-03-15T22:35:10.359160+0000","last_change":"2021-03-15T22:35:10.359360+0000","last_active":"2021-03-15T22:35:10.359160+0000","last_peered":"2021-03-15T22:35:10.359160+0000","last_clean":"2021-03-15T22:35:10.359160+0000","last_became_active":"2021-03-15T22:34:13.331560+0000","last_became_peered":"2021-03-15T22:34:13.331560+0000","last_unstale":"2021-03-15T22:35:10.359160+0000","last_undegraded":"2021-03-15T22:35:10.359160+0000","last_fullsized":"2021-03-15T22:35:10.359160+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'2007","last_scrub_stamp":"2021-03-15T22:31:02.050395+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.050395+0000","log_size":3029,"ondisk_log_size":3029,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":65439673,"num_objects":33,"num_object_clones":26,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":3,"num_read":827,"num_read_kb":848120,"num_write":643,"num_write_kb":257775,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":264088494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"239","length":"1"},{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"},{"start":"324","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"337","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.379496+0000","last_change":"2021-03-15T22:34:34.635632+0000","last_active":"2021-03-15T22:35:07.379496+0000","last_peered":"2021-03-15T22:35:07.379496+0000","last_clean":"2021-03-15T22:35:07.379496+0000","last_became_active":"2021-03-15T22:34:34.635223+0000","last_became_peered":"2021-03-15T22:34:34.635223+0000","last_unstale":"2021-03-15T22:35:07.379496+0000","last_undegraded":"2021-03-15T22:35:07.379496+0000","last_fullsized":"2021-03-15T22:35:07.379496+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.761137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"962","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.369419+0000","last_change":"2021-03-15T22:33:04.338099+0000","last_active":"2021-03-15T22:35:07.369419+0000","last_peered":"2021-03-15T22:35:07.369419+0000","last_clean":"2021-03-15T22:35:07.369419+0000","last_became_active":"2021-03-15T22:33:04.337709+0000","last_became_peered":"2021-03-15T22:33:04.337709+0000","last_unstale":"2021-03-15T22:35:07.369419+0000","last_undegraded":"2021-03-15T22:35:07.369419+0000","last_fullsized":"2021-03-15T22:35:07.369419+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:43.531349+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:16:35.535765+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.531349+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"33","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370426+0000","last_change":"2021-03-15T22:34:43.341562+0000","last_active":"2021-03-15T22:35:05.370426+0000","last_peered":"2021-03-15T22:35:05.370426+0000","last_clean":"2021-03-15T22:35:05.370426+0000","last_became_active":"2021-03-15T22:34:43.340955+0000","last_became_peered":"2021-03-15T22:34:43.340955+0000","last_unstale":"2021-03-15T22:35:05.370426+0000","last_undegraded":"2021-03-15T22:35:05.370426+0000","last_fullsized":"2021-03-15T22:35:05.370426+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.843478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.843478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"154","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.386804+0000","last_change":"2021-03-15T22:34:09.215097+0000","last_active":"2021-03-15T22:35:09.386804+0000","last_peered":"2021-03-15T22:35:09.386804+0000","last_clean":"2021-03-15T22:35:09.386804+0000","last_became_active":"2021-03-15T22:33:04.330531+0000","last_became_peered":"2021-03-15T22:33:04.330531+0000","last_unstale":"2021-03-15T22:35:09.386804+0000","last_undegraded":"2021-03-15T22:35:09.386804+0000","last_fullsized":"2021-03-15T22:35:09.386804+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:34:09.215034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:09.215034+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"983","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.370916+0000","last_change":"2021-03-15T22:34:55.352477+0000","last_active":"2021-03-15T22:35:07.370916+0000","last_peered":"2021-03-15T22:35:07.370916+0000","last_clean":"2021-03-15T22:35:07.370916+0000","last_became_active":"2021-03-15T22:34:55.352186+0000","last_became_peered":"2021-03-15T22:34:55.352186+0000","last_unstale":"2021-03-15T22:35:07.370916+0000","last_undegraded":"2021-03-15T22:35:07.370916+0000","last_fullsized":"2021-03-15T22:35:07.370916+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.451041+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.451041+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"952'928","reported_seq":"3919","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.380614+0000","last_change":"2021-03-15T22:35:07.380614+0000","last_active":"2021-03-15T22:35:07.380614+0000","last_peered":"2021-03-15T22:35:07.380614+0000","last_clean":"2021-03-15T22:35:07.380614+0000","last_became_active":"2021-03-15T22:35:03.346751+0000","last_became_peered":"2021-03-15T22:35:03.346751+0000","last_unstale":"2021-03-15T22:35:07.380614+0000","last_undegraded":"2021-03-15T22:35:07.380614+0000","last_fullsized":"2021-03-15T22:35:07.380614+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":0,"last_scrub":"891'819","last_scrub_stamp":"2021-03-15T22:34:08.508824+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.508824+0000","log_size":928,"ondisk_log_size":928,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36321465,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":0,"num_read":246,"num_read_kb":248363,"num_write":211,"num_write_kb":92564,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":109648904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"754","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.380179+0000","last_change":"2021-03-15T22:34:43.344711+0000","last_active":"2021-03-15T22:35:07.380179+0000","last_peered":"2021-03-15T22:35:07.380179+0000","last_clean":"2021-03-15T22:35:07.380179+0000","last_became_active":"2021-03-15T22:34:43.344418+0000","last_became_peered":"2021-03-15T22:34:43.344418+0000","last_unstale":"2021-03-15T22:35:07.380179+0000","last_undegraded":"2021-03-15T22:35:07.380179+0000","last_fullsized":"2021-03-15T22:35:07.380179+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"451","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.379504+0000","last_change":"2021-03-15T22:34:35.410541+0000","last_active":"2021-03-15T22:35:09.379504+0000","last_peered":"2021-03-15T22:35:09.379504+0000","last_clean":"2021-03-15T22:35:09.379504+0000","last_became_active":"2021-03-15T22:34:10.321100+0000","last_became_peered":"2021-03-15T22:34:10.321100+0000","last_unstale":"2021-03-15T22:35:09.379504+0000","last_undegraded":"2021-03-15T22:35:09.379504+0000","last_fullsized":"2021-03-15T22:35:09.379504+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:35.410485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:35.410485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"1976","reported_epoch":"953","state":"active+clean","last_fresh":"2021-03-15T22:35:08.378737+0000","last_change":"2021-03-15T22:32:30.365491+0000","last_active":"2021-03-15T22:35:08.378737+0000","last_peered":"2021-03-15T22:35:08.378737+0000","last_clean":"2021-03-15T22:35:08.378737+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:35:08.378737+0000","last_undegraded":"2021-03-15T22:35:08.378737+0000","last_fullsized":"2021-03-15T22:35:08.378737+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:32:30.365432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.365432+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1019,"num_read_kb":1014,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"549","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.382444+0000","last_change":"2021-03-15T22:34:55.414460+0000","last_active":"2021-03-15T22:35:09.382444+0000","last_peered":"2021-03-15T22:35:09.382444+0000","last_clean":"2021-03-15T22:35:09.382444+0000","last_became_active":"2021-03-15T22:34:55.359023+0000","last_became_peered":"2021-03-15T22:34:55.359023+0000","last_unstale":"2021-03-15T22:35:09.382444+0000","last_undegraded":"2021-03-15T22:35:09.382444+0000","last_fullsized":"2021-03-15T22:35:09.382444+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:34:40.472800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.472800+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"342","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.379561+0000","last_change":"2021-03-15T22:34:40.799217+0000","last_active":"2021-03-15T22:35:07.379561+0000","last_peered":"2021-03-15T22:35:07.379561+0000","last_clean":"2021-03-15T22:35:07.379561+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:35:07.379561+0000","last_undegraded":"2021-03-15T22:35:07.379561+0000","last_fullsized":"2021-03-15T22:35:07.379561+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.799153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.799153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"951'3159","reported_seq":"6943","reported_epoch":"953","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-03-15T22:35:09.192417+0000","last_change":"2021-03-15T22:35:02.369633+0000","last_active":"2021-03-15T22:35:09.192417+0000","last_peered":"2021-03-15T22:35:09.192417+0000","last_clean":"2021-03-15T22:34:53.604915+0000","last_became_active":"2021-03-15T22:34:56.439428+0000","last_became_peered":"2021-03-15T22:34:56.439428+0000","last_unstale":"2021-03-15T22:35:09.192417+0000","last_undegraded":"2021-03-15T22:35:02.360001+0000","last_fullsized":"2021-03-15T22:34:56.328883+0000","mapping_epoch":940,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":915,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'2597","last_scrub_stamp":"2021-03-15T22:32:30.909661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.909661+0000","log_size":3159,"ondisk_log_size":3159,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"stat_sum":{"num_bytes":110832439,"num_objects":49,"num_object_clones":42,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":44,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":1,"num_read":1150,"num_read_kb":1223286,"num_write":745,"num_write_kb":306276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":647,"num_bytes_recovered":260774902,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3],"avail_no_missing":["5","3","6"],"object_location_counts":[{"shards":"3,5","objects":5},{"shards":"3,5,6","objects":44}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"984","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.369129+0000","last_change":"2021-03-15T22:34:55.346293+0000","last_active":"2021-03-15T22:35:06.369129+0000","last_peered":"2021-03-15T22:35:06.369129+0000","last_clean":"2021-03-15T22:35:06.369129+0000","last_became_active":"2021-03-15T22:34:55.345948+0000","last_became_peered":"2021-03-15T22:34:55.345948+0000","last_unstale":"2021-03-15T22:35:06.369129+0000","last_undegraded":"2021-03-15T22:35:06.369129+0000","last_fullsized":"2021-03-15T22:35:06.369129+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.701955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.701955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"1002","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.369998+0000","last_change":"2021-03-15T22:34:55.346381+0000","last_active":"2021-03-15T22:35:06.369998+0000","last_peered":"2021-03-15T22:35:06.369998+0000","last_clean":"2021-03-15T22:35:06.369998+0000","last_became_active":"2021-03-15T22:34:55.346048+0000","last_became_peered":"2021-03-15T22:34:55.346048+0000","last_unstale":"2021-03-15T22:35:06.369998+0000","last_undegraded":"2021-03-15T22:35:06.369998+0000","last_fullsized":"2021-03-15T22:35:06.369998+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:48.037748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_clean_scrub_stamp":"2021-03-15T22:34:48.037748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"494","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.380071+0000","last_change":"2021-03-15T22:31:40.359761+0000","last_active":"2021-03-15T22:35:09.380071+0000","last_peered":"2021-03-15T22:35:09.380071+0000","last_clean":"2021-03-15T22:35:09.380071+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:35:09.380071+0000","last_undegraded":"2021-03-15T22:35:09.380071+0000","last_fullsized":"2021-03-15T22:35:09.380071+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:31:40.359701+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:31:40.359701+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"134","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.385224+0000","last_change":"2021-03-15T22:33:22.282663+0000","last_active":"2021-03-15T22:35:09.385224+0000","last_peered":"2021-03-15T22:35:09.385224+0000","last_clean":"2021-03-15T22:35:09.385224+0000","last_became_active":"2021-03-15T22:33:04.337244+0000","last_became_peered":"2021-03-15T22:33:04.337244+0000","last_unstale":"2021-03-15T22:35:09.385224+0000","last_undegraded":"2021-03-15T22:35:09.385224+0000","last_fullsized":"2021-03-15T22:35:09.385224+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.282586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.282586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"951'540","reported_seq":"3523","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.367537+0000","last_change":"2021-03-15T22:35:06.367782+0000","last_active":"2021-03-15T22:35:06.367537+0000","last_peered":"2021-03-15T22:35:06.367537+0000","last_clean":"2021-03-15T22:35:06.367537+0000","last_became_active":"2021-03-15T22:34:15.498612+0000","last_became_peered":"2021-03-15T22:34:15.498612+0000","last_unstale":"2021-03-15T22:35:06.367537+0000","last_undegraded":"2021-03-15T22:35:06.367537+0000","last_fullsized":"2021-03-15T22:35:06.367537+0000","mapping_epoch":899,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":900,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'463","last_scrub_stamp":"2021-03-15T22:33:40.563144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.563144+0000","log_size":540,"ondisk_log_size":540,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":19800050,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":170,"num_read_kb":209120,"num_write":106,"num_write_kb":44944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":84391207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"29f","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"475","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.378037+0000","last_change":"2021-03-15T22:34:55.356457+0000","last_active":"2021-03-15T22:35:09.378037+0000","last_peered":"2021-03-15T22:35:09.378037+0000","last_clean":"2021-03-15T22:35:09.378037+0000","last_became_active":"2021-03-15T22:34:55.356086+0000","last_became_peered":"2021-03-15T22:34:55.356086+0000","last_unstale":"2021-03-15T22:35:09.378037+0000","last_undegraded":"2021-03-15T22:35:09.378037+0000","last_fullsized":"2021-03-15T22:35:09.378037+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"925","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.379734+0000","last_change":"2021-03-15T22:34:55.356446+0000","last_active":"2021-03-15T22:35:07.379734+0000","last_peered":"2021-03-15T22:35:07.379734+0000","last_clean":"2021-03-15T22:35:07.379734+0000","last_became_active":"2021-03-15T22:34:55.355757+0000","last_became_peered":"2021-03-15T22:34:55.355757+0000","last_unstale":"2021-03-15T22:35:07.379734+0000","last_undegraded":"2021-03-15T22:35:07.379734+0000","last_fullsized":"2021-03-15T22:35:07.379734+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.883949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.883949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"906","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.378992+0000","last_change":"2021-03-15T22:34:11.330201+0000","last_active":"2021-03-15T22:35:07.378992+0000","last_peered":"2021-03-15T22:35:07.378992+0000","last_clean":"2021-03-15T22:35:07.378992+0000","last_became_active":"2021-03-15T22:34:10.321886+0000","last_became_peered":"2021-03-15T22:34:10.321886+0000","last_unstale":"2021-03-15T22:35:07.378992+0000","last_undegraded":"2021-03-15T22:35:07.378992+0000","last_fullsized":"2021-03-15T22:35:07.378992+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:32:31.747582+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.747582+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"836'11","reported_seq":"562","reported_epoch":"955","state":"active+clean","last_fresh":"2021-03-15T22:35:10.338432+0000","last_change":"2021-03-15T22:35:10.342791+0000","last_active":"2021-03-15T22:35:10.338432+0000","last_peered":"2021-03-15T22:35:10.338432+0000","last_clean":"2021-03-15T22:35:10.338432+0000","last_became_active":"2021-03-15T22:33:04.336294+0000","last_became_peered":"2021-03-15T22:33:04.336294+0000","last_unstale":"2021-03-15T22:35:10.338432+0000","last_undegraded":"2021-03-15T22:35:10.338432+0000","last_fullsized":"2021-03-15T22:35:10.338432+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'9","last_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.971303+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"951'879","reported_seq":"3892","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.376076+0000","last_change":"2021-03-15T22:35:06.376270+0000","last_active":"2021-03-15T22:35:06.376076+0000","last_peered":"2021-03-15T22:35:06.376076+0000","last_clean":"2021-03-15T22:35:06.376076+0000","last_became_active":"2021-03-15T22:34:58.350873+0000","last_became_peered":"2021-03-15T22:34:58.350873+0000","last_unstale":"2021-03-15T22:35:06.376076+0000","last_undegraded":"2021-03-15T22:35:06.376076+0000","last_fullsized":"2021-03-15T22:35:06.376076+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"744'591","last_scrub_stamp":"2021-03-15T22:31:35.474935+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:31:35.474935+0000","log_size":879,"ondisk_log_size":879,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":36578865,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":236,"num_read_kb":196881,"num_write":221,"num_write_kb":84582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":114552572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"29f","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"975","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.370489+0000","last_change":"2021-03-15T22:34:55.351577+0000","last_active":"2021-03-15T22:35:07.370489+0000","last_peered":"2021-03-15T22:35:07.370489+0000","last_clean":"2021-03-15T22:35:07.370489+0000","last_became_active":"2021-03-15T22:34:55.351249+0000","last_became_peered":"2021-03-15T22:34:55.351249+0000","last_unstale":"2021-03-15T22:35:07.370489+0000","last_undegraded":"2021-03-15T22:35:07.370489+0000","last_fullsized":"2021-03-15T22:35:07.370489+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.507204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.507204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"419","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.382522+0000","last_change":"2021-03-15T22:33:22.598240+0000","last_active":"2021-03-15T22:35:09.382522+0000","last_peered":"2021-03-15T22:35:09.382522+0000","last_clean":"2021-03-15T22:35:09.382522+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:35:09.382522+0000","last_undegraded":"2021-03-15T22:35:09.382522+0000","last_fullsized":"2021-03-15T22:35:09.382522+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.598174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.598174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"1143","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362656+0000","last_change":"2021-03-15T22:33:43.514160+0000","last_active":"2021-03-15T22:35:05.362656+0000","last_peered":"2021-03-15T22:35:05.362656+0000","last_clean":"2021-03-15T22:35:05.362656+0000","last_became_active":"2021-03-15T22:33:04.336145+0000","last_became_peered":"2021-03-15T22:33:04.336145+0000","last_unstale":"2021-03-15T22:35:05.362656+0000","last_undegraded":"2021-03-15T22:35:05.362656+0000","last_fullsized":"2021-03-15T22:35:05.362656+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:33:43.514088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.514088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"456","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370082+0000","last_change":"2021-03-15T22:33:20.685041+0000","last_active":"2021-03-15T22:35:05.370082+0000","last_peered":"2021-03-15T22:35:05.370082+0000","last_clean":"2021-03-15T22:35:05.370082+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:35:05.370082+0000","last_undegraded":"2021-03-15T22:35:05.370082+0000","last_fullsized":"2021-03-15T22:35:05.370082+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:33:20.684976+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.684976+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"952'1589","reported_seq":"4437","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.392892+0000","last_change":"2021-03-15T22:35:07.393489+0000","last_active":"2021-03-15T22:35:07.392892+0000","last_peered":"2021-03-15T22:35:07.392892+0000","last_clean":"2021-03-15T22:35:07.392892+0000","last_became_active":"2021-03-15T22:34:58.337099+0000","last_became_peered":"2021-03-15T22:34:58.337099+0000","last_unstale":"2021-03-15T22:35:07.392892+0000","last_undegraded":"2021-03-15T22:35:07.392892+0000","last_fullsized":"2021-03-15T22:35:07.392892+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"838'1298","last_scrub_stamp":"2021-03-15T22:33:09.468592+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.468592+0000","log_size":1589,"ondisk_log_size":1589,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51306599,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":505,"num_read_kb":522833,"num_write":343,"num_write_kb":120323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":226741522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"1028","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362171+0000","last_change":"2021-03-15T22:34:55.343142+0000","last_active":"2021-03-15T22:35:05.362171+0000","last_peered":"2021-03-15T22:35:05.362171+0000","last_clean":"2021-03-15T22:35:05.362171+0000","last_became_active":"2021-03-15T22:34:55.341896+0000","last_became_peered":"2021-03-15T22:34:55.341896+0000","last_unstale":"2021-03-15T22:35:05.362171+0000","last_undegraded":"2021-03-15T22:35:05.362171+0000","last_fullsized":"2021-03-15T22:35:05.362171+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"734","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370963+0000","last_change":"2021-03-15T22:34:41.870475+0000","last_active":"2021-03-15T22:35:05.370963+0000","last_peered":"2021-03-15T22:35:05.370963+0000","last_clean":"2021-03-15T22:35:05.370963+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:35:05.370963+0000","last_undegraded":"2021-03-15T22:35:05.370963+0000","last_fullsized":"2021-03-15T22:35:05.370963+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.870368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.870368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"387","reported_epoch":"953","state":"active+clean","last_fresh":"2021-03-15T22:35:08.402702+0000","last_change":"2021-03-15T22:35:08.402702+0000","last_active":"2021-03-15T22:35:08.402702+0000","last_peered":"2021-03-15T22:35:08.402702+0000","last_clean":"2021-03-15T22:35:08.402702+0000","last_became_active":"2021-03-15T22:34:10.349731+0000","last_became_peered":"2021-03-15T22:34:10.349731+0000","last_unstale":"2021-03-15T22:35:08.402702+0000","last_undegraded":"2021-03-15T22:35:08.402702+0000","last_fullsized":"2021-03-15T22:35:08.402702+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:35:08.402641+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:35:08.402641+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"996","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362115+0000","last_change":"2021-03-15T22:34:43.343527+0000","last_active":"2021-03-15T22:35:05.362115+0000","last_peered":"2021-03-15T22:35:05.362115+0000","last_clean":"2021-03-15T22:35:05.362115+0000","last_became_active":"2021-03-15T22:34:43.343255+0000","last_became_peered":"2021-03-15T22:34:43.343255+0000","last_unstale":"2021-03-15T22:35:05.362115+0000","last_undegraded":"2021-03-15T22:35:05.362115+0000","last_fullsized":"2021-03-15T22:35:05.362115+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.673338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.673338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"955'1334","reported_seq":"4458","reported_epoch":"955","state":"active+clean","last_fresh":"2021-03-15T22:35:10.350258+0000","last_change":"2021-03-15T22:35:10.350476+0000","last_active":"2021-03-15T22:35:10.350258+0000","last_peered":"2021-03-15T22:35:10.350258+0000","last_clean":"2021-03-15T22:35:10.350258+0000","last_became_active":"2021-03-15T22:34:58.344406+0000","last_became_peered":"2021-03-15T22:34:58.344406+0000","last_unstale":"2021-03-15T22:35:10.350258+0000","last_undegraded":"2021-03-15T22:35:10.350258+0000","last_fullsized":"2021-03-15T22:35:10.350258+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"872'1183","last_scrub_stamp":"2021-03-15T22:33:43.209215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.209215+0000","log_size":1334,"ondisk_log_size":1334,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35763054,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":374,"num_read_kb":340480,"num_write":294,"num_write_kb":105908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":127958264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"239","length":"1"},{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"},{"start":"324","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"897","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.379665+0000","last_change":"2021-03-15T22:34:55.356480+0000","last_active":"2021-03-15T22:35:07.379665+0000","last_peered":"2021-03-15T22:35:07.379665+0000","last_clean":"2021-03-15T22:35:07.379665+0000","last_became_active":"2021-03-15T22:34:55.356213+0000","last_became_peered":"2021-03-15T22:34:55.356213+0000","last_unstale":"2021-03-15T22:35:07.379665+0000","last_undegraded":"2021-03-15T22:35:07.379665+0000","last_fullsized":"2021-03-15T22:35:07.379665+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.729838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.729838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1236","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362822+0000","last_change":"2021-03-15T22:32:08.071278+0000","last_active":"2021-03-15T22:35:05.362822+0000","last_peered":"2021-03-15T22:35:05.362822+0000","last_clean":"2021-03-15T22:35:05.362822+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:35:05.362822+0000","last_undegraded":"2021-03-15T22:35:05.362822+0000","last_fullsized":"2021-03-15T22:35:05.362822+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:32:08.071210+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:08.071210+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"359","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.370319+0000","last_change":"2021-03-15T22:34:10.332407+0000","last_active":"2021-03-15T22:35:06.370319+0000","last_peered":"2021-03-15T22:35:06.370319+0000","last_clean":"2021-03-15T22:35:06.370319+0000","last_became_active":"2021-03-15T22:34:10.332089+0000","last_became_peered":"2021-03-15T22:34:10.332089+0000","last_unstale":"2021-03-15T22:35:06.370319+0000","last_undegraded":"2021-03-15T22:35:06.370319+0000","last_fullsized":"2021-03-15T22:35:06.370319+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.791527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.791527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"114","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.369180+0000","last_change":"2021-03-15T22:34:44.050423+0000","last_active":"2021-03-15T22:35:06.369180+0000","last_peered":"2021-03-15T22:35:06.369180+0000","last_clean":"2021-03-15T22:35:06.369180+0000","last_became_active":"2021-03-15T22:33:45.317888+0000","last_became_peered":"2021-03-15T22:33:45.317888+0000","last_unstale":"2021-03-15T22:35:06.369180+0000","last_undegraded":"2021-03-15T22:35:06.369180+0000","last_fullsized":"2021-03-15T22:35:06.369180+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:34:44.050366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:44.050366+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"955'441","reported_seq":"3146","reported_epoch":"955","state":"active+clean","last_fresh":"2021-03-15T22:35:10.357274+0000","last_change":"2021-03-15T22:35:10.342478+0000","last_active":"2021-03-15T22:35:10.357274+0000","last_peered":"2021-03-15T22:35:10.357274+0000","last_clean":"2021-03-15T22:35:10.357274+0000","last_became_active":"2021-03-15T22:33:08.354872+0000","last_became_peered":"2021-03-15T22:33:08.354872+0000","last_unstale":"2021-03-15T22:35:10.357274+0000","last_undegraded":"2021-03-15T22:35:10.357274+0000","last_fullsized":"2021-03-15T22:35:10.357274+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'373","last_scrub_stamp":"2021-03-15T22:33:44.219517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.219517+0000","log_size":441,"ondisk_log_size":441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":13706699,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":174,"num_read_kb":188293,"num_write":85,"num_write_kb":33091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":31290769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"239","length":"1"},{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"},{"start":"324","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"33","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370546+0000","last_change":"2021-03-15T22:34:43.340972+0000","last_active":"2021-03-15T22:35:05.370546+0000","last_peered":"2021-03-15T22:35:05.370546+0000","last_clean":"2021-03-15T22:35:05.370546+0000","last_became_active":"2021-03-15T22:34:43.340600+0000","last_became_peered":"2021-03-15T22:34:43.340600+0000","last_unstale":"2021-03-15T22:35:05.370546+0000","last_undegraded":"2021-03-15T22:35:05.370546+0000","last_fullsized":"2021-03-15T22:35:05.370546+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:09.612126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.612126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"990","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.379749+0000","last_change":"2021-03-15T22:34:48.811190+0000","last_active":"2021-03-15T22:35:07.379749+0000","last_peered":"2021-03-15T22:35:07.379749+0000","last_clean":"2021-03-15T22:35:07.379749+0000","last_became_active":"2021-03-15T22:33:08.352367+0000","last_became_peered":"2021-03-15T22:33:08.352367+0000","last_unstale":"2021-03-15T22:35:07.379749+0000","last_undegraded":"2021-03-15T22:35:07.379749+0000","last_fullsized":"2021-03-15T22:35:07.379749+0000","mapping_epoch":837,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'9","last_scrub_stamp":"2021-03-15T22:32:36.868932+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.868932+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"960","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362203+0000","last_change":"2021-03-15T22:34:55.347300+0000","last_active":"2021-03-15T22:35:05.362203+0000","last_peered":"2021-03-15T22:35:05.362203+0000","last_clean":"2021-03-15T22:35:05.362203+0000","last_became_active":"2021-03-15T22:34:55.346934+0000","last_became_peered":"2021-03-15T22:34:55.346934+0000","last_unstale":"2021-03-15T22:35:05.362203+0000","last_undegraded":"2021-03-15T22:35:05.362203+0000","last_fullsized":"2021-03-15T22:35:05.362203+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.461102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"1117","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362739+0000","last_change":"2021-03-15T22:34:10.369701+0000","last_active":"2021-03-15T22:35:05.362739+0000","last_peered":"2021-03-15T22:35:05.362739+0000","last_clean":"2021-03-15T22:35:05.362739+0000","last_became_active":"2021-03-15T22:34:10.330608+0000","last_became_peered":"2021-03-15T22:34:10.330608+0000","last_unstale":"2021-03-15T22:35:05.362739+0000","last_undegraded":"2021-03-15T22:35:05.362739+0000","last_fullsized":"2021-03-15T22:35:05.362739+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:32:38.318687+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.318687+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"952'464","reported_seq":"2663","reported_epoch":"952","state":"active+clean","last_fresh":"2021-03-15T22:35:07.387169+0000","last_change":"2021-03-15T22:35:07.387169+0000","last_active":"2021-03-15T22:35:07.387169+0000","last_peered":"2021-03-15T22:35:07.387169+0000","last_clean":"2021-03-15T22:35:07.387169+0000","last_became_active":"2021-03-15T22:34:58.350923+0000","last_became_peered":"2021-03-15T22:34:58.350923+0000","last_unstale":"2021-03-15T22:35:07.387169+0000","last_undegraded":"2021-03-15T22:35:07.387169+0000","last_fullsized":"2021-03-15T22:35:07.387169+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"833'385","last_scrub_stamp":"2021-03-15T22:33:13.869146+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:13.869146+0000","log_size":464,"ondisk_log_size":464,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8923593,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":155,"num_read_kb":164362,"num_write":109,"num_write_kb":33883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":52707265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"695","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370723+0000","last_change":"2021-03-15T22:34:55.345645+0000","last_active":"2021-03-15T22:35:05.370723+0000","last_peered":"2021-03-15T22:35:05.370723+0000","last_clean":"2021-03-15T22:35:05.370723+0000","last_became_active":"2021-03-15T22:34:55.345267+0000","last_became_peered":"2021-03-15T22:34:55.345267+0000","last_unstale":"2021-03-15T22:35:05.370723+0000","last_undegraded":"2021-03-15T22:35:05.370723+0000","last_fullsized":"2021-03-15T22:35:05.370723+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.958050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.958050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"984","reported_epoch":"951","state":"active+clean","last_fresh":"2021-03-15T22:35:06.370209+0000","last_change":"2021-03-15T22:33:15.406572+0000","last_active":"2021-03-15T22:35:06.370209+0000","last_peered":"2021-03-15T22:35:06.370209+0000","last_clean":"2021-03-15T22:35:06.370209+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:35:06.370209+0000","last_undegraded":"2021-03-15T22:35:06.370209+0000","last_fullsized":"2021-03-15T22:35:06.370209+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.406493+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.406493+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"143","reported_epoch":"954","state":"active+clean","last_fresh":"2021-03-15T22:35:09.385396+0000","last_change":"2021-03-15T22:34:10.332595+0000","last_active":"2021-03-15T22:35:09.385396+0000","last_peered":"2021-03-15T22:35:09.385396+0000","last_clean":"2021-03-15T22:35:09.385396+0000","last_became_active":"2021-03-15T22:34:10.332236+0000","last_became_peered":"2021-03-15T22:34:10.332236+0000","last_unstale":"2021-03-15T22:35:09.385396+0000","last_undegraded":"2021-03-15T22:35:09.385396+0000","last_fullsized":"2021-03-15T22:35:09.385396+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:08.273379+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.273379+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"1096","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362990+0000","last_change":"2021-03-15T22:34:55.371223+0000","last_active":"2021-03-15T22:35:05.362990+0000","last_peered":"2021-03-15T22:35:05.362990+0000","last_clean":"2021-03-15T22:35:05.362990+0000","last_became_active":"2021-03-15T22:34:55.341958+0000","last_became_peered":"2021-03-15T22:34:55.341958+0000","last_unstale":"2021-03-15T22:35:05.362990+0000","last_undegraded":"2021-03-15T22:35:05.362990+0000","last_fullsized":"2021-03-15T22:35:05.362990+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:32:43.307660+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.307660+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"955'1563","reported_seq":"4526","reported_epoch":"955","state":"active+clean","last_fresh":"2021-03-15T22:35:10.357697+0000","last_change":"2021-03-15T22:35:10.347048+0000","last_active":"2021-03-15T22:35:10.357697+0000","last_peered":"2021-03-15T22:35:10.357697+0000","last_clean":"2021-03-15T22:35:10.357697+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:35:10.357697+0000","last_undegraded":"2021-03-15T22:35:10.357697+0000","last_fullsized":"2021-03-15T22:35:10.357697+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"837'1315","last_scrub_stamp":"2021-03-15T22:33:08.608669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.608669+0000","log_size":1563,"ondisk_log_size":1563,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46154563,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":2,"num_read":490,"num_read_kb":501483,"num_write":353,"num_write_kb":127086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"239","length":"1"},{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"},{"start":"324","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"961","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362870+0000","last_change":"2021-03-15T22:34:55.340853+0000","last_active":"2021-03-15T22:35:05.362870+0000","last_peered":"2021-03-15T22:35:05.362870+0000","last_clean":"2021-03-15T22:35:05.362870+0000","last_became_active":"2021-03-15T22:34:55.340498+0000","last_became_peered":"2021-03-15T22:34:55.340498+0000","last_unstale":"2021-03-15T22:35:05.362870+0000","last_undegraded":"2021-03-15T22:35:05.362870+0000","last_fullsized":"2021-03-15T22:35:05.362870+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:20.500979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.500979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"941'18","reported_seq":"332","reported_epoch":"954","state":"active+remapped+backfill_toofull","last_fresh":"2021-03-15T22:35:09.379977+0000","last_change":"2021-03-15T22:34:56.377931+0000","last_active":"2021-03-15T22:35:09.379977+0000","last_peered":"2021-03-15T22:35:09.379977+0000","last_clean":"2021-03-15T22:30:29.286429+0000","last_became_active":"2021-03-15T22:34:56.340566+0000","last_became_peered":"2021-03-15T22:34:56.340566+0000","last_unstale":"2021-03-15T22:35:09.379977+0000","last_undegraded":"2021-03-15T22:35:09.379977+0000","last_fullsized":"2021-03-15T22:35:09.379977+0000","mapping_epoch":940,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'13","last_scrub_stamp":"2021-03-15T22:33:37.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.949876+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[0,7,5],"avail_no_missing":["0","2","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"379","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.370416+0000","last_change":"2021-03-15T22:33:21.690369+0000","last_active":"2021-03-15T22:35:05.370416+0000","last_peered":"2021-03-15T22:35:05.370416+0000","last_clean":"2021-03-15T22:35:05.370416+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:35:05.370416+0000","last_undegraded":"2021-03-15T22:35:05.370416+0000","last_fullsized":"2021-03-15T22:35:05.370416+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.690309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"1133","reported_epoch":"950","state":"active+clean","last_fresh":"2021-03-15T22:35:05.362955+0000","last_change":"2021-03-15T22:33:05.216013+0000","last_active":"2021-03-15T22:35:05.362955+0000","last_peered":"2021-03-15T22:35:05.362955+0000","last_clean":"2021-03-15T22:35:05.362955+0000","last_became_active":"2021-03-15T22:33:04.337113+0000","last_became_peered":"2021-03-15T22:33:04.337113+0000","last_unstale":"2021-03-15T22:35:05.362955+0000","last_undegraded":"2021-03-15T22:35:05.362955+0000","last_fullsized":"2021-03-15T22:35:05.362955+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:32:32.328526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.328526+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":706387892,"num_objects":321,"num_object_clones":271,"num_object_copies":963,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":44,"num_objects_unfound":0,"num_objects_dirty":321,"num_whiteouts":12,"num_read":6627,"num_read_kb":6749867,"num_write":4917,"num_write_kb":1871893,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4547,"num_bytes_recovered":2139777581,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1617825792,"data_stored":2254196190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21439,"ondisk_log_size":21439,"up":48,"acting":47,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1019,"num_read_kb":1014,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1482,"num_read_kb":987,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":94766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":947,"ondisk_log_size":947,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299603,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":268064,"kb_used_data":221272,"kb_used_omap":0,"kb_used_meta":46784,"kb_avail":93477088,"statfs":{"total":95995035648,"available":95720538112,"internally_reserved":0,"allocated":226582528,"data_stored":343055121,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47906816},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:23 2021","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45500000000000002,"15min":0.46999999999999997},"min":{"1min":0.32200000000000001,"5min":0.216,"15min":0.215},"max":{"1min":0.65300000000000002,"5min":0.70099999999999996,"15min":2.73},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49099999999999999,"15min":0.48999999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.222},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":3.8799999999999999},"last":0.373}]},{"osd":1,"last update":"Mon Mar 15 22:34:23 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45400000000000001,"15min":0.47499999999999998},"min":{"1min":0.28000000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.66600000000000004,"5min":0.73899999999999999,"15min":2.6859999999999999},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.34899999999999998,"5min":0.28399999999999997,"15min":0.23400000000000001},"max":{"1min":0.69699999999999995,"5min":0.81899999999999995,"15min":2.0779999999999998},"last":0.44600000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:34:38 2021","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.5,"15min":0.48199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68600000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.497,"15min":0.5},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63200000000000001,"5min":1.119,"15min":1.119},"last":0.31}]},{"osd":3,"last update":"Mon Mar 15 22:34:23 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.5,"15min":0.495},"min":{"1min":0.35399999999999998,"5min":0.33800000000000002,"15min":0.28899999999999998},"max":{"1min":0.61099999999999999,"5min":1.0249999999999999,"15min":1.702},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.52700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":0.78300000000000003,"15min":5.415},"last":0.27200000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:34:23 2021","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.28499999999999998,"5min":0.255,"15min":0.23799999999999999},"max":{"1min":0.56100000000000005,"5min":0.753,"15min":1.927},"last":0.34100000000000003},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.46600000000000003,"15min":0.46899999999999997},"min":{"1min":0.35099999999999998,"5min":0.28499999999999998,"15min":0.253},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":2.5339999999999998},"last":0.26400000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:34:38 2021","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47599999999999998,"15min":0.45400000000000001},"min":{"1min":0.318,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.28299999999999997},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.29699999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68899999999999995,"5min":1.071,"15min":1.071},"last":0.32800000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:34:23 2021","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.50700000000000001,"15min":0.51900000000000002},"min":{"1min":0.36199999999999999,"5min":0.29999999999999999,"15min":0.27100000000000002},"max":{"1min":0.64200000000000002,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53500000000000003,"15min":0.52400000000000002},"min":{"1min":0.34300000000000003,"5min":0.33800000000000002,"15min":0.28399999999999997},"max":{"1min":0.81999999999999995,"5min":1.097,"15min":1.8560000000000001},"last":0.35299999999999998}]}]},{"osd":6,"up_from":45,"seq":193273528535,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":280840,"kb_used_data":240640,"kb_used_omap":0,"kb_used_meta":40192,"kb_avail":93464312,"statfs":{"total":95995035648,"available":95707455488,"internally_reserved":0,"allocated":246415360,"data_stored":321197162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":41156608},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:05 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.47099999999999997,"15min":0.48399999999999999},"min":{"1min":0.30399999999999999,"5min":0.251,"15min":0.24299999999999999},"max":{"1min":0.70799999999999996,"5min":2.5470000000000002,"15min":3.085},"last":0.316},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46700000000000003,"15min":0.46800000000000003},"min":{"1min":0.35299999999999998,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.64100000000000001,"5min":1.137,"15min":1.323},"last":0.38800000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:34:05 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.53600000000000003,"15min":0.55500000000000005},"min":{"1min":0.33500000000000002,"5min":0.27900000000000003,"15min":0.254},"max":{"1min":0.68200000000000005,"5min":2.7839999999999998,"15min":14.637},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.45500000000000002,"5min":0.44400000000000001,"15min":0.47999999999999998},"min":{"1min":0.35799999999999998,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.73999999999999999,"5min":0.84899999999999998,"15min":5.2450000000000001},"last":0.28999999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:34:40 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47299999999999998,"15min":0.46500000000000002},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.51200000000000001},"min":{"1min":0.36199999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.63300000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.33800000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:34:05 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.49099999999999999,"15min":0.50900000000000001},"min":{"1min":0.35499999999999998,"5min":0.318,"15min":0.29899999999999999},"max":{"1min":0.58399999999999996,"5min":1.0609999999999999,"15min":2.6150000000000002},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.49199999999999999,"15min":0.50800000000000001},"min":{"1min":0.34599999999999997,"5min":0.316,"15min":0.28699999999999998},"max":{"1min":0.69199999999999995,"5min":2.4670000000000001,"15min":2.7890000000000001},"last":0.376}]},{"osd":4,"last update":"Mon Mar 15 22:34:05 2021","interfaces":[{"interface":"back","average":{"1min":0.42399999999999999,"5min":0.432,"15min":0.441},"min":{"1min":0.28899999999999998,"5min":0.224,"15min":0.224},"max":{"1min":0.68600000000000005,"5min":0.94099999999999995,"15min":4.306},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.498,"5min":0.46999999999999997,"15min":0.495},"min":{"1min":0.28799999999999998,"5min":0.252,"15min":0.23499999999999999},"max":{"1min":0.67800000000000005,"5min":1.091,"15min":4.181},"last":0.443}]},{"osd":5,"last update":"Mon Mar 15 22:34:36 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.42499999999999999},"min":{"1min":0.311,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.622,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52700000000000002,"15min":0.46899999999999997},"min":{"1min":0.33700000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63900000000000001,"5min":1.022,"15min":1.022},"last":0.49199999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:34:19 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.47399999999999998,"15min":0.53900000000000003},"min":{"1min":0.34599999999999997,"5min":0.307,"15min":0.249},"max":{"1min":0.72299999999999998,"5min":0.76300000000000001,"15min":15.340999999999999},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.55500000000000005,"15min":0.59099999999999997},"min":{"1min":0.432,"5min":0.28999999999999998,"15min":0.27200000000000002},"max":{"1min":0.65500000000000003,"5min":1.0109999999999999,"15min":15.356},"last":0.45300000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575080,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":182508,"kb_used_data":120164,"kb_used_omap":0,"kb_used_meta":62336,"kb_avail":93562644,"statfs":{"total":95995035648,"available":95808147456,"internally_reserved":0,"allocated":123047936,"data_stored":162679457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":63832064},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:52 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.379,"15min":0.51200000000000001},"min":{"1min":0.217,"5min":0.184,"15min":0.17599999999999999},"max":{"1min":4.3920000000000003,"5min":4.3920000000000003,"15min":54.427999999999997},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.42699999999999999,"15min":0.54100000000000004},"min":{"1min":0.219,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":54.356999999999999},"last":0.52500000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.46600000000000003,"15min":0.378},"min":{"1min":0.26100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.45200000000000001,"15min":0.40000000000000002},"min":{"1min":0.25700000000000001,"5min":0.254,"15min":0.254},"max":{"1min":5.0190000000000001,"5min":5.0190000000000001,"15min":5.0190000000000001},"last":0.32400000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:34:17 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.374,"15min":0.36899999999999999},"min":{"1min":0.28100000000000003,"5min":0.23100000000000001,"15min":0.215},"max":{"1min":0.501,"5min":1.647,"15min":1.647},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.42699999999999999,"15min":0.40200000000000002},"min":{"1min":0.26000000000000001,"5min":0.246,"15min":0.20999999999999999},"max":{"1min":0.61699999999999999,"5min":3.2919999999999998,"15min":3.2919999999999998},"last":0.37}]},{"osd":4,"last update":"Mon Mar 15 22:34:21 2021","interfaces":[{"interface":"back","average":{"1min":0.75600000000000001,"5min":0.60699999999999998,"15min":0.55300000000000005},"min":{"1min":0.31900000000000001,"5min":0.31900000000000001,"15min":0.22800000000000001},"max":{"1min":4.593,"5min":4.593,"15min":4.593},"last":0.53000000000000003},{"interface":"front","average":{"1min":0.57999999999999996,"5min":0.53900000000000003,"15min":0.56399999999999995},"min":{"1min":0.35499999999999998,"5min":0.317,"15min":0.253},"max":{"1min":0.874,"5min":1.284,"15min":4.6139999999999999},"last":0.38800000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:34:31 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55200000000000005,"15min":0.53300000000000003},"min":{"1min":0.34499999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78600000000000003,"5min":1.226,"15min":1.226},"last":0.435},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.58999999999999997,"15min":0.52200000000000002},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89000000000000001,"5min":3.472,"15min":3.472},"last":0.5}]},{"osd":6,"last update":"Mon Mar 15 22:34:59 2021","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61499999999999999,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.314},"max":{"1min":1.679,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.63600000000000001,"15min":0.58299999999999996},"min":{"1min":0.39800000000000002,"5min":0.375,"15min":0.33700000000000002},"max":{"1min":0.82999999999999996,"5min":4.6529999999999996,"15min":4.6529999999999996},"last":0.44900000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:34:17 2021","interfaces":[{"interface":"back","average":{"1min":0.64300000000000002,"5min":0.60899999999999999,"15min":0.59899999999999998},"min":{"1min":0.42099999999999999,"5min":0.376,"15min":0.316},"max":{"1min":1.476,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.54400000000000004},{"interface":"front","average":{"1min":0.83699999999999997,"5min":0.64500000000000002,"15min":0.59099999999999997},"min":{"1min":0.45600000000000002,"5min":0.38200000000000001,"15min":0.35799999999999998},"max":{"1min":4.625,"5min":4.625,"15min":4.625},"last":0.48799999999999999}]}]},{"osd":0,"up_from":8,"seq":34359738603,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":271600,"kb_used_data":215528,"kb_used_omap":0,"kb_used_meta":56064,"kb_avail":93473552,"statfs":{"total":95995035648,"available":95716917248,"internally_reserved":0,"allocated":220700672,"data_stored":337322863,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57409536},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:34:44 2021","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.34399999999999997,"15min":0.33800000000000002},"min":{"1min":0.27600000000000002,"5min":0.192,"15min":0.17999999999999999},"max":{"1min":0.55400000000000005,"5min":1.46,"15min":1.7070000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.436,"5min":0.38700000000000001,"15min":0.36899999999999999},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.19900000000000001},"max":{"1min":1.331,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.499}]},{"osd":2,"last update":"Mon Mar 15 22:34:40 2021","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.372,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.46100000000000002,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.49199999999999999,"15min":0.41399999999999998},"min":{"1min":0.26200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.3799999999999999,"5min":7.3799999999999999,"15min":7.3799999999999999},"last":0.47299999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:34:24 2021","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.39000000000000001,"15min":0.38400000000000001},"min":{"1min":0.30299999999999999,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":0.46999999999999997,"5min":1.544,"15min":1.544},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.60999999999999999,"5min":0.45600000000000002,"15min":0.41799999999999998},"min":{"1min":0.318,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":3.988,"5min":3.988,"15min":3.988},"last":0.56100000000000005}]},{"osd":4,"last update":"Mon Mar 15 22:34:24 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55400000000000005,"15min":0.54400000000000004},"min":{"1min":0.32400000000000001,"5min":0.27400000000000002,"15min":0.254},"max":{"1min":1.095,"5min":1.861,"15min":1.861},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58199999999999996,"15min":0.57199999999999995},"min":{"1min":0.41299999999999998,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":0.89700000000000002,"5min":1.5700000000000001,"15min":1.645},"last":0.67200000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:34:35 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.59899999999999998,"15min":0.53200000000000003},"min":{"1min":0.38100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.7879999999999998,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55400000000000005,"15min":0.5},"min":{"1min":0.35899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.163,"5min":1.909,"15min":1.909},"last":0.69499999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:34:59 2021","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.26200000000000001},"max":{"1min":2.7109999999999999,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.74199999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.437,"5min":0.32000000000000001,"15min":0.27100000000000002},"max":{"1min":0.77100000000000002,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.70599999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:34:24 2021","interfaces":[{"interface":"back","average":{"1min":0.73599999999999999,"5min":0.63400000000000001,"15min":0.58999999999999997},"min":{"1min":0.41099999999999998,"5min":0.34599999999999997,"15min":0.27900000000000003},"max":{"1min":3.8239999999999998,"5min":3.8239999999999998,"15min":3.8239999999999998},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.66500000000000004,"5min":0.59899999999999998,"15min":0.58399999999999996},"min":{"1min":0.46100000000000002,"5min":0.38900000000000001,"15min":0.317},"max":{"1min":2.665,"5min":2.665,"15min":2.665},"last":0.71999999999999997}]}]},{"osd":2,"up_from":681,"seq":2924872728633,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284924,"kb_used_data":244596,"kb_used_omap":119,"kb_used_meta":40200,"kb_avail":93460228,"statfs":{"total":95995035648,"available":95703273472,"internally_reserved":0,"allocated":250466304,"data_stored":324018803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":41165161},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.315,"15min":0.309},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.40000000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.23100000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.34599999999999997,"15min":0.32800000000000001},"min":{"1min":0.251,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.34399999999999997}]},{"osd":1,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.35399999999999998,"15min":0.33300000000000002},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.25900000000000001},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38,"15min":0.36299999999999999},"min":{"1min":0.27100000000000002,"5min":0.221,"15min":0.221},"max":{"1min":1.0269999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.312}]},{"osd":3,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.36199999999999999},"min":{"1min":0.30599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.89300000000000002,"5min":3.496,"15min":3.496},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.375,"15min":0.35899999999999999},"min":{"1min":0.252,"5min":0.248,"15min":0.248},"max":{"1min":0.624,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.27300000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53800000000000003,"15min":0.49099999999999999},"min":{"1min":0.377,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.113,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.30399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.497},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.94699999999999995,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.41599999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51500000000000001,"15min":0.502},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.65500000000000003,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55500000000000005,"15min":0.51700000000000002},"min":{"1min":0.40400000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.1870000000000001,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.38500000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.51500000000000001},"min":{"1min":0.374,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.208,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56499999999999995,"15min":0.52100000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.97799999999999998,"5min":1.387,"15min":1.387},"last":0.51100000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58799999999999997,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.222,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56599999999999995,"15min":0.52900000000000003},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0529999999999999,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.53000000000000003}]}]},{"osd":3,"up_from":26,"seq":111669149920,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300356,"kb_used_data":230460,"kb_used_omap":0,"kb_used_meta":69888,"kb_avail":93444796,"statfs":{"total":95995035648,"available":95687471104,"internally_reserved":0,"allocated":235991040,"data_stored":302697247,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":71565312},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:19 2021","interfaces":[{"interface":"back","average":{"1min":0.42599999999999999,"5min":0.34599999999999997,"15min":0.33800000000000002},"min":{"1min":0.183,"5min":0.183,"15min":0.17699999999999999},"max":{"1min":2.7509999999999999,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.32500000000000001,"15min":0.33500000000000002},"min":{"1min":0.20999999999999999,"5min":0.20000000000000001,"15min":0.189},"max":{"1min":0.76900000000000002,"5min":1.419,"15min":2.6339999999999999},"last":0.27000000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:34:19 2021","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.36599999999999999,"15min":0.39200000000000002},"min":{"1min":0.249,"5min":0.23599999999999999,"15min":0.23000000000000001},"max":{"1min":0.59999999999999998,"5min":0.72599999999999998,"15min":6.0199999999999996},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.42999999999999999,"5min":0.36299999999999999,"15min":0.40000000000000002},"min":{"1min":0.253,"5min":0.22,"15min":0.20699999999999999},"max":{"1min":2.5619999999999998,"5min":2.5619999999999998,"15min":7.6130000000000004},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35499999999999998},"min":{"1min":0.247,"5min":0.216,"15min":0.216},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.38,"5min":0.38700000000000001,"15min":0.38800000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53600000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.30599999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:34:31 2021","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.22600000000000001},"max":{"1min":0.69099999999999995,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.53400000000000003,"15min":0.52800000000000002},"min":{"1min":0.311,"5min":0.28499999999999998,"15min":0.26700000000000002},"max":{"1min":0.68000000000000005,"5min":2.1059999999999999,"15min":2.2890000000000001},"last":0.40799999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:34:31 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.59999999999999998,"15min":0.77700000000000002},"min":{"1min":0.313,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.71999999999999997,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":0.495},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.66000000000000003,"15min":0.998},"min":{"1min":0.36699999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66100000000000003,"5min":15.776,"15min":15.776},"last":0.53300000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:35:02 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55800000000000005,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.315,"15min":0.30099999999999999},"max":{"1min":0.78700000000000003,"5min":1.359,"15min":2.6499999999999999},"last":0.63600000000000001},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55400000000000005,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.35899999999999999,"15min":0.30399999999999999},"max":{"1min":0.77800000000000002,"5min":1.3400000000000001,"15min":3.472},"last":0.54500000000000004}]},{"osd":7,"last update":"Mon Mar 15 22:34:19 2021","interfaces":[{"interface":"back","average":{"1min":0.499,"5min":0.57799999999999996,"15min":0.57599999999999996},"min":{"1min":0.34799999999999998,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.69899999999999995,"5min":5.625,"15min":5.625},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.64300000000000002,"5min":0.58199999999999996,"15min":0.56599999999999995},"min":{"1min":0.39800000000000002,"5min":0.35699999999999998,"15min":0.35499999999999998},"max":{"1min":2.621,"5min":2.621,"15min":2.621},"last":0.502}]}]},{"osd":4,"up_from":31,"seq":133143986397,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":135912,"kb_used_data":96416,"kb_used_omap":0,"kb_used_meta":39488,"kb_avail":93609240,"statfs":{"total":95995035648,"available":95855861760,"internally_reserved":0,"allocated":98729984,"data_stored":158126520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40435712},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:32 2021","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.504,"15min":0.47699999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.19500000000000001},"max":{"1min":0.59999999999999998,"5min":4.7050000000000001,"15min":4.7050000000000001},"last":0.498},{"interface":"front","average":{"1min":0.438,"5min":0.46200000000000002,"15min":0.5},"min":{"1min":0.27100000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.68100000000000005,"5min":1.4179999999999999,"15min":5.0209999999999999},"last":0.40699999999999997}]},{"osd":1,"last update":"Mon Mar 15 22:34:32 2021","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.47699999999999998,"15min":0.48199999999999998},"min":{"1min":0.34499999999999997,"5min":0.25,"15min":0.23599999999999999},"max":{"1min":0.752,"5min":1.3720000000000001,"15min":4.3499999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.504},"min":{"1min":0.37,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":0.67500000000000004,"5min":1.494,"15min":4.718},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:34:38 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.49099999999999999,"15min":0.505},"min":{"1min":0.375,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.55600000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.35799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77200000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:34:32 2021","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51800000000000002,"15min":0.54900000000000004},"min":{"1min":0.40300000000000002,"5min":0.32200000000000001,"15min":0.28499999999999998},"max":{"1min":0.71999999999999997,"5min":1.486,"15min":5.0499999999999998},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71899999999999997,"5min":1.4950000000000001,"15min":4.4050000000000002},"last":0.51700000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:34:32 2021","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.42199999999999999,"15min":0.40899999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53700000000000003,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.32600000000000001},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46600000000000003,"15min":0.433},"min":{"1min":0.30299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66800000000000004,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.38100000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:34:58 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47799999999999998,"15min":0.46700000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":0.70999999999999996,"5min":0.878,"15min":4.3079999999999998},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54900000000000004,"15min":0.54000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28000000000000003},"max":{"1min":0.71899999999999997,"5min":0.83799999999999997,"15min":4.6600000000000001},"last":0.438}]},{"osd":7,"last update":"Mon Mar 15 22:34:15 2021","interfaces":[{"interface":"back","average":{"1min":0.51400000000000001,"5min":0.497,"15min":0.51000000000000001},"min":{"1min":0.35099999999999998,"5min":0.29399999999999998,"15min":0.26300000000000001},"max":{"1min":0.66500000000000004,"5min":0.80900000000000005,"15min":4.6929999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.51200000000000001,"15min":0.54700000000000004},"min":{"1min":0.35499999999999998,"5min":0.32400000000000001,"15min":0.25},"max":{"1min":0.65800000000000003,"5min":1.034,"15min":4.9900000000000002},"last":0.46999999999999997}]}]},{"osd":5,"up_from":616,"seq":2645699854406,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":273432,"kb_used_data":234576,"kb_used_omap":153,"kb_used_meta":38694,"kb_avail":93471720,"statfs":{"total":95995035648,"available":95715041280,"internally_reserved":0,"allocated":240205824,"data_stored":328241077,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":39623185},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48999999999999999,"15min":0.46400000000000002},"min":{"1min":0.28799999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0780000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":0.33400000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48199999999999998,"15min":0.45600000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.002,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.48199999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.56100000000000005,"15min":0.48399999999999999},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.046,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":0.46100000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.29399999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59799999999999998,"15min":0.54100000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.121,"5min":10.952,"15min":10.952},"last":0.312},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.51500000000000001,"15min":0.52700000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.103,"5min":1.125,"15min":1.125},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.67800000000000005,"15min":0.53800000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.1379999999999999,"5min":10.355,"15min":10.355},"last":0.432},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.505,"15min":0.45700000000000002},"min":{"1min":0.35699999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.1839999999999999,"5min":2.476,"15min":2.476},"last":0.40799999999999997}]},{"osd":4,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.504,"15min":0.41199999999999998},"min":{"1min":0.29799999999999999,"5min":0.254,"15min":0.254},"max":{"1min":1.204,"5min":7.4000000000000004,"15min":7.4000000000000004},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46100000000000002,"15min":0.40400000000000003},"min":{"1min":0.316,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.63400000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.38300000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.50800000000000001,"15min":0.46500000000000002},"min":{"1min":0.30199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.98899999999999999,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55300000000000005,"15min":0.48899999999999999},"min":{"1min":0.38100000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0860000000000001,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.52600000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.51800000000000002,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.0169999999999999,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58399999999999996,"15min":0.51600000000000001},"min":{"1min":0.35999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.0269999999999999,"5min":5.944,"15min":5.944},"last":0.39200000000000002}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":12026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":10320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":8144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":206880768,"data_stored":315589368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":121323520,"data_stored":161722260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":248913920,"data_stored":323059335,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":235335680,"data_stored":307054239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":97071104,"data_stored":157179833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":238649344,"data_stored":327281495,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":244711424,"data_stored":320216703,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":224940032,"data_stored":342092957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:35:12.651 INFO:tasks.thrashosds.thrasher:Setting 2.2 to [2, 1, 5, 7, 6, 0] 2021-03-15T22:35:12.652 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.2', '2', '1', '5', '7', '6', '0'] 2021-03-15T22:35:12.653 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pg-upmap-items 2.2 2 1 5 7 6 0 2021-03-15T22:35:12.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:12.655 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:12.655 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:12.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[62156]: cluster 2021-03-15T22:35:11 2021-03-15T22:35:12.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[62156]: .216079+0000 mgr.y (mgr.14144) 1067 : cluster [DBG] pgmap v1580: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 674 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 11 op/s; 5/1908 objects degraded (0.262%); 44/1908 objects misplaced (2.306%); 0 B/s, 14 objects/s recovering 2021-03-15T22:35:12.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[62156]: cluster 2021-03-15T22:35:11.392255+0000 mon.a ( 2021-03-15T22:35:12.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[62156]: mon.0) 2395 : cluster [DBG] osdmap e956: 8 total, 8 up, 8 in 2021-03-15T22:35:12.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[62156]: audit 2021 2021-03-15T22:35:12.679 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[62156]: -03-15T22:35:11.666262+0000 2021-03-15T22:35:12.679 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[62156]: mgr.y (mgr.14144) 1068 : audit [DBG] from='client.25561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:12.679 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: cluster 2021-03-15T22:35:12.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 2021-03-15T22:35:12.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 2021 2021-03-15T22:35:12.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: - 2021-03-15T22:35:12.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 03 2021-03-15T22:35:12.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: - 2021-03-15T22:35:12.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 15 2021-03-15T22:35:12.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: T 2021-03-15T22:35:12.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 22 2021-03-15T22:35:12.681 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: : 2021-03-15T22:35:12.682 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 35 2021-03-15T22:35:12.682 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: : 2021-03-15T22:35:12.682 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 11 2021-03-15T22:35:12.682 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: . 2021-03-15T22:35:12.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 216079 2021-03-15T22:35:12.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: +0000 2021-03-15T22:35:12.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 2021-03-15T22:35:12.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: mgr.y 2021-03-15T22:35:12.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: ( 2021-03-15T22:35:12.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: mgr 2021-03-15T22:35:12.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: . 2021-03-15T22:35:12.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 14144 2021-03-15T22:35:12.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: ) 2021-03-15T22:35:12.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 1067 2021-03-15T22:35:12.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: : 2021-03-15T22:35:12.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: cluster 2021-03-15T22:35:12.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 2021-03-15T22:35:12.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: [DBG] 2021-03-15T22:35:12.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 2021-03-15T22:35:12.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: pgmap v1580: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 674 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 11 op/s; 5/1908 objects degraded (0.262%); 44/1908 objects misplaced (2.306%); 0 B/s, 14 objects/s recovering 2021-03-15T22:35:12.686 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 2021-03-15T22:35:12.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: cluster 2021-03-15T22:35:12.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 2021-03-15T22:35:12.687 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 2021 2021-03-15T22:35:12.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: - 2021-03-15T22:35:12.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 03 2021-03-15T22:35:12.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: - 2021-03-15T22:35:12.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 15 2021-03-15T22:35:12.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: T 2021-03-15T22:35:12.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 22 2021-03-15T22:35:12.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: : 2021-03-15T22:35:12.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 35 2021-03-15T22:35:12.689 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: : 2021-03-15T22:35:12.690 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 11 2021-03-15T22:35:12.690 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: . 2021-03-15T22:35:12.690 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 392255 2021-03-15T22:35:12.690 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: +0000 2021-03-15T22:35:12.690 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: mon.a (mon.0) 2395 : cluster [DBG] 2021-03-15T22:35:12.691 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: osdmap e956: 8 total, 8 up, 8 in 2021-03-15T22:35:12.691 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: audit 2021-03- 2021-03-15T22:35:12.691 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: 15T22:35:11.666262+0000 mgr.y (mgr.14144) 1068 : audit 2021-03-15T22:35:12.691 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:12 smithi076 conmon[56514]: [DBG] from='client.25561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:12.692 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:12.757 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:12.780 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:12.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:12.782 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:12.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:12 smithi125 conmon[56739]: cluster 2021-03-15T22:35:11.216079+0000 mgr.y 2021-03-15T22:35:12.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:12 smithi125 conmon[56739]: (mgr.14144) 1067 : cluster [DBG] pgmap v1580: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 674 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 2.6 MiB/s rd, 1.9 MiB/s wr, 11 op/s; 5/1908 objects degraded (0.262%); 44/1908 objects misplaced (2.306%); 0 B/s, 14 objects/s recovering 2021-03-15T22:35:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:12 smithi125 conmon[56739]: cluster 2021-03-15T22:35:11.392255+0000 mon.a (mon.0) 2395 : cluster [DBG] osdmap e956: 8 total, 8 up, 8 in 2021-03-15T22:35:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:12 smithi125 conmon[56739]: 2021-03-15T22:35:12.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:12 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:35:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:12 smithi125 conmon[56739]: 22:35:11.666262+0000 mgr.y (mgr.14144) 1068 : audit 2021-03-15T22:35:12.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:12 smithi125 conmon[56739]: [DBG] from='client.25561 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:12.884 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:12.909 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:12.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:12.911 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:13.012 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:13.028 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:35:13.029 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:35:13.033 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:13.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:13.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:13.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:13.160 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:13.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:13.162 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:13.264 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:13.286 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:13.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:13.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:13.372 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:35:13.389 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:13.409 INFO:tasks.rados.rados.0.smithi076.stdout:3617: done (2 left) 2021-03-15T22:35:13.410 INFO:tasks.rados.rados.0.smithi076.stdout:3619: done (1 left) 2021-03-15T22:35:13.410 INFO:tasks.rados.rados.0.smithi076.stdout:3620: done (0 left) 2021-03-15T22:35:13.410 INFO:tasks.rados.rados.0.smithi076.stdout:3621: write (excl) oid 34 current snap is 426 2021-03-15T22:35:13.410 INFO:tasks.rados.rados.0.smithi076.stdout:3621: seq_num 1235 ranges {756645=581701,1935850=473428} 2021-03-15T22:35:13.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:13.412 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:13.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:13.417 INFO:tasks.rados.rados.0.smithi076.stdout:3621: writing smithi07694878-34 from 756645 to 1338346 tid 1 2021-03-15T22:35:13.423 INFO:tasks.rados.rados.0.smithi076.stdout:3621: writing smithi07694878-34 from 1935850 to 2409278 tid 2 2021-03-15T22:35:13.423 INFO:tasks.rados.rados.0.smithi076.stdout:3622: read oid 31 snap 414 2021-03-15T22:35:13.423 INFO:tasks.rados.rados.0.smithi076.stdout:3622: expect (ObjNum 1141 snap 397 seq_num 1141) 2021-03-15T22:35:13.424 INFO:tasks.rados.rados.0.smithi076.stdout:3623: write oid 46 current snap is 426 2021-03-15T22:35:13.424 INFO:tasks.rados.rados.0.smithi076.stdout:3621: finishing write tid 1 to smithi07694878-34 2021-03-15T22:35:13.424 INFO:tasks.rados.rados.0.smithi076.stdout:3623: seq_num 1236 ranges {773452=656950,1877979=617086} 2021-03-15T22:35:13.430 INFO:tasks.rados.rados.0.smithi076.stdout:3623: writing smithi07694878-46 from 773452 to 1430402 tid 1 2021-03-15T22:35:13.437 INFO:tasks.rados.rados.0.smithi076.stdout:3623: writing smithi07694878-46 from 1877979 to 2495065 tid 2 2021-03-15T22:35:13.437 INFO:tasks.rados.rados.0.smithi076.stdout:3624: write (excl) oid 48 current snap is 426 2021-03-15T22:35:13.437 INFO:tasks.rados.rados.0.smithi076.stdout:3621: finishing write tid 2 to smithi07694878-34 2021-03-15T22:35:13.438 INFO:tasks.rados.rados.0.smithi076.stdout:3624: seq_num 1237 ranges {790259=732199,1936461=477097} 2021-03-15T22:35:13.445 INFO:tasks.rados.rados.0.smithi076.stdout:3624: writing smithi07694878-48 from 790259 to 1522458 tid 1 2021-03-15T22:35:13.452 INFO:tasks.rados.rados.0.smithi076.stdout:3624: writing smithi07694878-48 from 1936461 to 2413558 tid 2 2021-03-15T22:35:13.452 INFO:tasks.rados.rados.0.smithi076.stdout:3625: snap_remove snap 415 2021-03-15T22:35:13.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:13.536 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:13.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:13.537 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:13.638 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:13.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:13.661 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:13.661 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:13.762 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:13 smithi076 conmon[56514]: cluster 2021-03-15T22:35:12.091645+0000 osd.1 (osd.1) 179 : cluster [DBG] 5.c scrub ok 2021-03-15T22:35:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:13 smithi076 conmon[56514]: cluster 2021-03-15T22:35:12.392216+0000 mon.a (mon.0) 2396 : cluster [DBG] osdmap e957: 8 total, 8 up, 8 in 2021-03-15T22:35:13.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:13 smithi076 conmon[56514]: cluster 2021-03-15T22:35:12.404891+0000 osd.7 (osd.7) 178 : cluster [DBG] 5.1a scrub ok 2021-03-15T22:35:13.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:13 smithi076 conmon[62156]: cluster 2021-03-15T22:35:12.091645+0000 osd.1 (osd. 2021-03-15T22:35:13.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:13 smithi076 conmon[62156]: 1) 179 : cluster [DBG] 5.c scrub ok 2021-03-15T22:35:13.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:13 smithi076 conmon[62156]: cluster 2021-03-15T22:35:13.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:13 smithi076 conmon[62156]: 2021-03-15T22:35:12.392216+0000 mon.a (mon.0) 2396 : cluster [DBG] 2021-03-15T22:35:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:13 smithi076 conmon[62156]: osdmap e957: 8 total, 8 up, 8 in 2021-03-15T22:35:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:13 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:35:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:13 smithi076 conmon[62156]: 15T22:35:12.404891+0000 osd.7 (osd 2021-03-15T22:35:13.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:13 smithi076 conmon[62156]: .7) 178 : cluster [DBG] 5.1a scrub ok 2021-03-15T22:35:13.785 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:13.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:13.786 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:13 smithi125 conmon[56739]: cluster 2021-03-15T22:35:12.091645+0000 2021-03-15T22:35:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:13 smithi125 conmon[56739]: osd.1 (osd.1) 179 : cluster [DBG] 5.c scrub ok 2021-03-15T22:35:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:13 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:35:13.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:13 smithi125 conmon[56739]: T22:35:12.392216+0000 mon.a (mon.0) 2396 : cluster 2021-03-15T22:35:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:13 smithi125 conmon[56739]: [DBG] osdmap e957: 8 total, 8 up, 8 in 2021-03-15T22:35:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:13 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:35:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:13 smithi125 conmon[56739]: -15T22:35:12.404891+0000 osd.7 2021-03-15T22:35:13.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:13 smithi125 conmon[56739]: (osd.7) 178 : cluster [DBG] 5.1a scrub ok 2021-03-15T22:35:13.887 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:13.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:13.910 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:13.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:14.011 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:14.033 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:14.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:14.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:14.061 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-15T22:35:14.136 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:14.158 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:14.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:14.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:14.210 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:14.260 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:14.261 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:14.282 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:14.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:14.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:14.384 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:14.409 INFO:tasks.rados.rados.0.smithi076.stdout:3625: done (4 left) 2021-03-15T22:35:14.409 INFO:tasks.rados.rados.0.smithi076.stdout:3626: rollback oid 45 current snap is 426 2021-03-15T22:35:14.410 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 45 to 357 2021-03-15T22:35:14.410 INFO:tasks.rados.rados.0.smithi076.stdout:3621: finishing write tid 3 to smithi07694878-34 2021-03-15T22:35:14.410 INFO:tasks.rados.rados.0.smithi076.stdout:3621: finishing write tid 4 to smithi07694878-34 2021-03-15T22:35:14.410 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 2440 (ObjNum 1235 snap 426 seq_num 1235) dirty exists 2021-03-15T22:35:14.411 INFO:tasks.rados.rados.0.smithi076.stdout:3621: left oid 34 (ObjNum 1235 snap 426 seq_num 1235) 2021-03-15T22:35:14.411 INFO:tasks.rados.rados.0.smithi076.stdout:3622: expect (ObjNum 1141 snap 397 seq_num 1141) 2021-03-15T22:35:14.411 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:14.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:14.413 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:14.514 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:14.537 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:14.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:14.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:14.612 INFO:tasks.rados.rados.0.smithi076.stdout:3623: finishing write tid 1 to smithi07694878-46 2021-03-15T22:35:14.707 INFO:tasks.rados.rados.0.smithi076.stdout:3623: finishing write tid 2 to smithi07694878-46 2021-03-15T22:35:14.707 INFO:tasks.rados.rados.0.smithi076.stdout:3623: finishing write tid 3 to smithi07694878-46 2021-03-15T22:35:14.708 INFO:tasks.rados.rados.0.smithi076.stdout:3623: finishing write tid 4 to smithi07694878-46 2021-03-15T22:35:14.708 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2296 (ObjNum 1236 snap 426 seq_num 1236) dirty exists 2021-03-15T22:35:14.708 INFO:tasks.rados.rados.0.smithi076.stdout:3623: left oid 46 (ObjNum 1236 snap 426 seq_num 1236) 2021-03-15T22:35:14.708 INFO:tasks.rados.rados.0.smithi076.stdout:3624: finishing write tid 1 to smithi07694878-48 2021-03-15T22:35:14.709 INFO:tasks.rados.rados.0.smithi076.stdout:3624: finishing write tid 2 to smithi07694878-48 2021-03-15T22:35:14.709 INFO:tasks.rados.rados.0.smithi076.stdout:3624: finishing write tid 3 to smithi07694878-48 2021-03-15T22:35:14.709 INFO:tasks.rados.rados.0.smithi076.stdout:3624: finishing write tid 4 to smithi07694878-48 2021-03-15T22:35:14.709 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 2430 (ObjNum 1237 snap 426 seq_num 1237) dirty exists 2021-03-15T22:35:14.709 INFO:tasks.rados.rados.0.smithi076.stdout:3624: left oid 48 (ObjNum 1237 snap 426 seq_num 1237) 2021-03-15T22:35:14.710 INFO:tasks.rados.rados.0.smithi076.stdout:3621: done (4 left) 2021-03-15T22:35:14.710 INFO:tasks.rados.rados.0.smithi076.stdout:3622: done (3 left) 2021-03-15T22:35:14.710 INFO:tasks.rados.rados.0.smithi076.stdout:3623: done (2 left) 2021-03-15T22:35:14.710 INFO:tasks.rados.rados.0.smithi076.stdout:3624: done (1 left) 2021-03-15T22:35:14.711 INFO:tasks.rados.rados.0.smithi076.stdout:3627: snap_create 2021-03-15T22:35:14.711 INFO:tasks.rados.rados.0.smithi076.stdout:3626: finishing rollback tid 0 to smithi07694878-45 2021-03-15T22:35:14.711 INFO:tasks.rados.rados.0.smithi076.stdout:3626: finishing rollback tid 1 to smithi07694878-45 2021-03-15T22:35:14.711 INFO:tasks.rados.rados.0.smithi076.stdout:3626: finishing rollback tid 2 to smithi07694878-45 2021-03-15T22:35:14.711 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 3079 (ObjNum 1034 snap 346 seq_num 1034) dirty exists 2021-03-15T22:35:14.712 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:14.733 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:14.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:14.735 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:14.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[56514]: cluster 2021-03-15T22:35:13.216556+0000 mgr.y (mgr.14144) 1069 : cluster [DBG] pgmap v1583: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 655 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 5/1884 objects degraded (0.265%); 44/1884 objects misplaced (2.335%); 0 B/s, 14 objects/s recovering 2021-03-15T22:35:14.736 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[56514]: cluster 2021-03-15T22:35:13.404109+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e958: 8 total, 8 up, 8 in 2021-03-15T22:35:14.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: cluster 2021-03-15T22:35:14.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: 2021-03-15T22:35:14.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: 2021 2021-03-15T22:35:14.737 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: - 2021-03-15T22:35:14.738 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: 03 2021-03-15T22:35:14.738 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: - 2021-03-15T22:35:14.738 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: 15 2021-03-15T22:35:14.738 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: T 2021-03-15T22:35:14.739 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: 22:35:13.216556 2021-03-15T22:35:14.739 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 1069 : cluster [DBG] pgmap v1583: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 655 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 5/1884 objects degraded (0.265%); 44/1884 objects misplaced (2.335%); 0 B/s, 14 objects/s recovering 2021-03-15T22:35:14.739 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:35:14.739 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: 15T22:35: 2021-03-15T22:35:14.740 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: 13.404109+0000 mon.a (mon.0 2021-03-15T22:35:14.740 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:14 smithi076 conmon[62156]: ) 2397 : cluster [DBG] osdmap e958: 8 total, 8 up, 8 in 2021-03-15T22:35:14.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:14 smithi125 conmon[56739]: cluster 2021-03-15T22:35:13.216556+0000 mgr.y ( 2021-03-15T22:35:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:14 smithi125 conmon[56739]: mgr.14144) 1069 : cluster [DBG] pgmap v1583: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 655 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 2.2 MiB/s wr, 15 op/s; 5/1884 objects degraded (0.265%); 44/1884 objects misplaced (2.335%); 0 B/s, 14 objects/s recovering 2021-03-15T22:35:14.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:14 smithi125 conmon[56739]: cluster 2021-03-15T22:35:13.404109+0000 mon.a (mon.0) 2397 : cluster [DBG] osdmap e958: 8 total, 8 up, 8 in 2021-03-15T22:35:14.837 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:14.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:14.859 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:14.860 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:14.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:14.983 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:14.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:14.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:15.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:15.107 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:15.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:15.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:15.209 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:15.232 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:15.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:15.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:15.335 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:15.359 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:15.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:15.360 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:15.462 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:15.479 INFO:tasks.rados.rados.0.smithi076.stdout:3626: done (1 left) 2021-03-15T22:35:15.480 INFO:tasks.rados.rados.0.smithi076.stdout:3627: done (0 left) 2021-03-15T22:35:15.480 INFO:tasks.rados.rados.0.smithi076.stdout:3628: read oid 17 snap -1 2021-03-15T22:35:15.480 INFO:tasks.rados.rados.0.smithi076.stdout:3628: expect deleted 2021-03-15T22:35:15.480 INFO:tasks.rados.rados.0.smithi076.stdout:3629: write oid 5 current snap is 427 2021-03-15T22:35:15.481 INFO:tasks.rados.rados.0.smithi076.stdout:3629: seq_num 1238 ranges {407066=407448,1594943=620755} 2021-03-15T22:35:15.484 INFO:tasks.rados.rados.0.smithi076.stdout:3629: writing smithi07694878-5 from 407066 to 814514 tid 1 2021-03-15T22:35:15.484 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:15.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:15.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:15.492 INFO:tasks.rados.rados.0.smithi076.stdout:3629: writing smithi07694878-5 from 1594943 to 2215698 tid 2 2021-03-15T22:35:15.492 INFO:tasks.rados.rados.0.smithi076.stdout:3630: rollback oid 3 current snap is 427 2021-03-15T22:35:15.492 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 3 to 360 2021-03-15T22:35:15.493 INFO:tasks.rados.rados.0.smithi076.stdout:3629: finishing write tid 1 to smithi07694878-5 2021-03-15T22:35:15.493 INFO:tasks.rados.rados.0.smithi076.stdout:3628: done (2 left) 2021-03-15T22:35:15.493 INFO:tasks.rados.rados.0.smithi076.stdout:3631: snap_create 2021-03-15T22:35:15.496 INFO:tasks.rados.rados.0.smithi076.stdout:3629: finishing write tid 2 to smithi07694878-5 2021-03-15T22:35:15.496 INFO:tasks.rados.rados.0.smithi076.stdout:3629: finishing write tid 3 to smithi07694878-5 2021-03-15T22:35:15.496 INFO:tasks.rados.rados.0.smithi076.stdout:3629: finishing write tid 4 to smithi07694878-5 2021-03-15T22:35:15.496 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 2009 (ObjNum 1238 snap 427 seq_num 1238) dirty exists 2021-03-15T22:35:15.497 INFO:tasks.rados.rados.0.smithi076.stdout:3629: left oid 5 (ObjNum 1238 snap 427 seq_num 1238) 2021-03-15T22:35:15.498 INFO:tasks.rados.rados.0.smithi076.stdout:3630: finishing rollback tid 0 to smithi07694878-3 2021-03-15T22:35:15.498 INFO:tasks.rados.rados.0.smithi076.stdout:3630: finishing rollback tid 1 to smithi07694878-3 2021-03-15T22:35:15.498 INFO:tasks.rados.rados.0.smithi076.stdout:3630: finishing rollback tid 2 to smithi07694878-3 2021-03-15T22:35:15.499 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 3 v 2686 (ObjNum 620 snap 193 seq_num 620) dirty exists 2021-03-15T22:35:15.586 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:15.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:15.611 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:15.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:15.712 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:15.734 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:15.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:15.736 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:15.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:15 smithi076 conmon[56514]: cluster 2021-03-15T22:35:14.406975+0000 mon.a (mon.0) 2398 : cluster [DBG] osdmap e959: 8 total, 8 up, 8 in 2021-03-15T22:35:15.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:15 smithi076 conmon[56514]: cluster 2021-03-15T22:35:14.433957+0000 osd.4 (osd.4) 192 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:35:15.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:15 smithi076 conmon[56514]: cluster 2021-03-15T22:35:14.592447+0000 osd.6 (osd.6) 140 : cluster [DBG] 6.23 scrub ok 2021-03-15T22:35:15.755 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:15 smithi076 conmon[56514]: cluster 2021-03-15T22:35:15.326240+0000 mon.a (mon.0) 2399 : cluster [WRN] Health check update: Degraded data redundancy: 5/1884 objects degraded (0.265%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:15.756 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:15 smithi076 conmon[62156]: cluster 2021-03-15T22:35:14.406975+0000 mon.a (mon.0) 2398 : 2021-03-15T22:35:15.756 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:15 smithi076 conmon[62156]: cluster [DBG] osdmap e959: 8 total, 8 up, 8 in 2021-03-15T22:35:15.756 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:15 smithi076 conmon[62156]: cluster 2021-03-15T22:35:14.433957+0000 osd.4 (osd.4) 192 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:35:15.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:15 smithi076 conmon[62156]: cluster 2021-03-15T22:35:14.592447+0000 osd.6 (osd. 2021-03-15T22:35:15.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:15 smithi076 conmon[62156]: 6) 140 : cluster [DBG] 6.23 scrub ok 2021-03-15T22:35:15.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:15 smithi076 conmon[62156]: cluster 2021-03-15T22:35:15.326240+0000 2021-03-15T22:35:15.757 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:15 smithi076 conmon[62156]: mon.a (mon.0) 2399 : cluster [WRN] Health check update: Degraded data redundancy: 5/1884 objects degraded (0.265%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:15.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:15 smithi125 conmon[56739]: cluster 2021-03-15T22:35:14.406975+0000 mon.a ( 2021-03-15T22:35:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:15 smithi125 conmon[56739]: mon.0) 2398 : cluster [DBG] osdmap e959: 8 total, 8 up, 8 in 2021-03-15T22:35:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:15 smithi125 conmon[56739]: cluster 2021-03-15T22:35:14.433957+0000 osd.4 (osd.4) 192 : cluster [DBG] 4.18 scrub ok 2021-03-15T22:35:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:15 smithi125 conmon[56739]: cluster 2021-03-15T22:35:14.592447+0000 2021-03-15T22:35:15.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:15 smithi125 conmon[56739]: osd.6 (osd.6) 140 : cluster [DBG] 6.23 scrub ok 2021-03-15T22:35:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:15 smithi125 conmon[56739]: 2021-03-15T22:35:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:15 smithi125 conmon[56739]: cluster 2021-03-15T22:35:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:15 smithi125 conmon[56739]: 2021-03-15T22:35:15.326240+0000 mon.a (mon 2021-03-15T22:35:15.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:15 smithi125 conmon[56739]: .0) 2399 : cluster [WRN] Health check update: Degraded data redundancy: 5/1884 objects degraded (0.265%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:15.837 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:15.859 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:15.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:15.861 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:15.962 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:15.984 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:15.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:15.986 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:16.046 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:16.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:16.109 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:16.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:16.110 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:16.212 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:16.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:16.235 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:16.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:16.337 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:16.359 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:16.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:16.361 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:16.462 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:16.478 INFO:tasks.rados.rados.0.smithi076.stdout:3629: done (2 left) 2021-03-15T22:35:16.479 INFO:tasks.rados.rados.0.smithi076.stdout:3630: done (1 left) 2021-03-15T22:35:16.479 INFO:tasks.rados.rados.0.smithi076.stdout:3631: done (0 left) 2021-03-15T22:35:16.479 INFO:tasks.rados.rados.0.smithi076.stdout:3632: read oid 34 snap -1 2021-03-15T22:35:16.479 INFO:tasks.rados.rados.0.smithi076.stdout:3632: expect (ObjNum 1235 snap 426 seq_num 1235) 2021-03-15T22:35:16.480 INFO:tasks.rados.rados.0.smithi076.stdout:3633: rollback oid 13 current snap is 428 2021-03-15T22:35:16.480 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 13 to 422 2021-03-15T22:35:16.480 INFO:tasks.rados.rados.0.smithi076.stdout:3634: read oid 16 snap 410 2021-03-15T22:35:16.480 INFO:tasks.rados.rados.0.smithi076.stdout:3634: expect (ObjNum 1141 snap 397 seq_num 1141) 2021-03-15T22:35:16.481 INFO:tasks.rados.rados.0.smithi076.stdout:3635: copy_from oid 23 from oid 2 current snap is 428 2021-03-15T22:35:16.481 INFO:tasks.rados.rados.0.smithi076.stdout:3636: read oid 24 snap -1 2021-03-15T22:35:16.481 INFO:tasks.rados.rados.0.smithi076.stdout:3636: expect deleted 2021-03-15T22:35:16.481 INFO:tasks.rados.rados.0.smithi076.stdout:3637: read oid 11 snap -1 2021-03-15T22:35:16.481 INFO:tasks.rados.rados.0.smithi076.stdout:3637: expect (ObjNum 1189 snap 416 seq_num 1189) 2021-03-15T22:35:16.482 INFO:tasks.rados.rados.0.smithi076.stdout:3638: read oid 46 snap -1 2021-03-15T22:35:16.482 INFO:tasks.rados.rados.0.smithi076.stdout:3638: expect (ObjNum 1236 snap 426 seq_num 1236) 2021-03-15T22:35:16.482 INFO:tasks.rados.rados.0.smithi076.stdout:3639: delete oid 12 current snap is 428 2021-03-15T22:35:16.487 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:16.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:16.489 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:16.496 INFO:tasks.rados.rados.0.smithi076.stdout:3638: expect (ObjNum 1236 snap 426 seq_num 1236) 2021-03-15T22:35:16.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:16 smithi076 conmon[56514]: cluster 2021-03-15T22:35:15.217004+0000 mgr.y (mgr 2021-03-15T22:35:16.579 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:16 smithi076 conmon[56514]: .14144) 1070 : cluster [DBG] pgmap v1586: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 649 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s; 6/1881 objects degraded (0.319%); 10/1881 objects misplaced (0.532%); 15 MiB/s, 14 objects/s recovering 2021-03-15T22:35:16.580 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:16 smithi076 conmon[56514]: cluster 2021-03-15T22:35:15.466353+0000 mon.a (mon.0) 2400 : cluster [DBG] osdmap e960: 8 total, 8 up, 8 in 2021-03-15T22:35:16.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:16 smithi076 conmon[62156]: cluster 2021-03-15T22:35:15.217004+0000 mgr.y ( 2021-03-15T22:35:16.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:16 smithi076 conmon[62156]: mgr.14144) 1070 : cluster [DBG] pgmap v1586: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 649 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s; 6/1881 objects degraded (0.319%); 10/1881 objects misplaced (0.532%); 15 MiB/s, 14 objects/s recovering 2021-03-15T22:35:16.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:16 smithi076 conmon[62156]: cluster 2021-03-15T22:35:15.466353+0000 mon.a (mon.0 2021-03-15T22:35:16.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:16 smithi076 conmon[62156]: ) 2400 : cluster [DBG] osdmap e960: 8 total, 8 up, 8 in 2021-03-15T22:35:16.589 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:16.613 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:16.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:16.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:16.666 INFO:tasks.rados.rados.0.smithi076.stdout:3634: expect (ObjNum 1141 snap 397 seq_num 1141) 2021-03-15T22:35:16.715 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:16.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:16.739 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:16.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:16.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:16 smithi125 conmon[56739]: cluster 2021-03-15T22:35:15.217004+0000 mgr.y (mgr 2021-03-15T22:35:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:16 smithi125 conmon[56739]: .14144) 1070 : cluster [DBG] pgmap v1586: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+remapped+backfill_toofull, 156 active+clean; 649 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.8 MiB/s rd, 1.6 MiB/s wr, 12 op/s; 6/1881 objects degraded (0.319%); 10/1881 objects misplaced (0.532%); 15 MiB/s, 14 objects/s recovering 2021-03-15T22:35:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:16 smithi125 conmon[56739]: cluster 2021-03-15T22:35:15.466353+0000 mon.a (mon.0) 2400 : 2021-03-15T22:35:16.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:16 smithi125 conmon[56739]: cluster [DBG] osdmap e960: 8 total, 8 up, 8 in 2021-03-15T22:35:16.841 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:16.860 INFO:tasks.rados.rados.0.smithi076.stdout:3633: finishing rollback tid 1 to smithi07694878-13 2021-03-15T22:35:16.860 INFO:tasks.rados.rados.0.smithi076.stdout:3633: finishing rollback tid 2 to smithi07694878-13 2021-03-15T22:35:16.860 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 2428 (ObjNum 1209 snap 418 seq_num 1209) dirty exists 2021-03-15T22:35:16.860 INFO:tasks.rados.rados.0.smithi076.stdout:3632: expect (ObjNum 1235 snap 426 seq_num 1235) 2021-03-15T22:35:16.864 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:16.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:16.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:16.951 INFO:tasks.rados.rados.0.smithi076.stdout:3637: expect (ObjNum 1189 snap 416 seq_num 1189) 2021-03-15T22:35:16.966 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:16.989 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:16.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:16.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:17.092 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:17.113 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:17.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:17.115 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:17.136 INFO:tasks.rados.rados.0.smithi076.stdout:3635: finishing copy_from to smithi07694878-23 2021-03-15T22:35:17.136 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 2424 (ObjNum 1189 snap 416 seq_num 1189) dirty exists 2021-03-15T22:35:17.137 INFO:tasks.rados.rados.0.smithi076.stdout:3635: finishing copy_from racing read to smithi07694878-23 2021-03-15T22:35:17.137 INFO:tasks.rados.rados.0.smithi076.stdout:3632: done (7 left) 2021-03-15T22:35:17.137 INFO:tasks.rados.rados.0.smithi076.stdout:3633: done (6 left) 2021-03-15T22:35:17.137 INFO:tasks.rados.rados.0.smithi076.stdout:3634: done (5 left) 2021-03-15T22:35:17.137 INFO:tasks.rados.rados.0.smithi076.stdout:3635: done (4 left) 2021-03-15T22:35:17.138 INFO:tasks.rados.rados.0.smithi076.stdout:3636: done (3 left) 2021-03-15T22:35:17.138 INFO:tasks.rados.rados.0.smithi076.stdout:3637: done (2 left) 2021-03-15T22:35:17.138 INFO:tasks.rados.rados.0.smithi076.stdout:3638: done (1 left) 2021-03-15T22:35:17.138 INFO:tasks.rados.rados.0.smithi076.stdout:3639: done (0 left) 2021-03-15T22:35:17.139 INFO:tasks.rados.rados.0.smithi076.stdout:3640: snap_remove snap 419 2021-03-15T22:35:17.216 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:17.238 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:17.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:17.240 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:17.341 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:17.365 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:17.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:17.367 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:17.468 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:17.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[56514]: cluster 2021-03-15T22:35:17.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[56514]: 2021-03-15T22:35:16.036082+0000 osd.1 (osd.1) 180 : cluster [DBG] 7.b scrub ok 2021-03-15T22:35:17.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[56514]: cluster 2021-03-15T22:35:16.256500+0000 osd.1 (osd.1) 181 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:17.477 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[56514]: cluster 2021-03-15T22:35:16.258853+0000 osd.1 (osd.1) 182 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:17.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[56514]: cluster 2021-03-15T22:35:16.472601+0000 mon.a (mon.0) 2401 : cluster [DBG] osdmap e961: 8 total, 8 up, 8 in 2021-03-15T22:35:17.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[56514]: audit 2021-03-15T22:35:16.799881+0000 mon.c (mon.1) 79 : audit [INF] from='client.? 172.21.15.76:0/2967012423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:17.478 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[56514]: audit 2021-03-15T22:35:16.800180+0000 mon.a (mon.0) 2402 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:17.486 INFO:tasks.rados.rados.0.smithi076.stdout:3640: done (0 left) 2021-03-15T22:35:17.487 INFO:tasks.rados.rados.0.smithi076.stdout:3641: write (excl) oid 13 current snap is 428 2021-03-15T22:35:17.487 INFO:tasks.rados.rados.0.smithi076.stdout:3641: seq_num 1240 ranges {440680=674299,1711907=624424} 2021-03-15T22:35:17.491 INFO:tasks.rados.rados.0.smithi076.stdout:3641: writing smithi07694878-13 from 440680 to 1114979 tid 1 2021-03-15T22:35:17.499 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:17.499 INFO:tasks.rados.rados.0.smithi076.stdout:3641: writing smithi07694878-13 from 1711907 to 2336331 tid 2 2021-03-15T22:35:17.499 INFO:tasks.rados.rados.0.smithi076.stdout:3642: write oid 10 current snap is 428 2021-03-15T22:35:17.500 INFO:tasks.rados.rados.0.smithi076.stdout:3642: seq_num 1241 ranges {457487=749548,1770389=484435,2746287=111200} 2021-03-15T22:35:17.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:17.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:17.506 INFO:tasks.rados.rados.0.smithi076.stdout:3642: writing smithi07694878-10 from 457487 to 1207035 tid 1 2021-03-15T22:35:17.508 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:35:17.512 INFO:tasks.rados.rados.0.smithi076.stdout:3642: writing smithi07694878-10 from 1770389 to 2254824 tid 2 2021-03-15T22:35:17.516 INFO:tasks.rados.rados.0.smithi076.stdout:3642: writing smithi07694878-10 from 2746287 to 2857487 tid 3 2021-03-15T22:35:17.517 INFO:tasks.rados.rados.0.smithi076.stdout:3643: rollback oid 39 current snap is 428 2021-03-15T22:35:17.517 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 39 to 375 2021-03-15T22:35:17.517 INFO:tasks.rados.rados.0.smithi076.stdout:3641: finishing write tid 1 to smithi07694878-13 2021-03-15T22:35:17.518 INFO:tasks.rados.rados.0.smithi076.stdout:3641: finishing write tid 2 to smithi07694878-13 2021-03-15T22:35:17.518 INFO:tasks.rados.rados.0.smithi076.stdout:3641: finishing write tid 3 to smithi07694878-13 2021-03-15T22:35:17.518 INFO:tasks.rados.rados.0.smithi076.stdout:3641: finishing write tid 4 to smithi07694878-13 2021-03-15T22:35:17.519 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 2431 (ObjNum 1240 snap 428 seq_num 1240) dirty exists 2021-03-15T22:35:17.519 INFO:tasks.rados.rados.0.smithi076.stdout:3641: left oid 13 (ObjNum 1240 snap 428 seq_num 1240) 2021-03-15T22:35:17.519 INFO:tasks.rados.rados.0.smithi076.stdout:3642: finishing write tid 1 to smithi07694878-10 2021-03-15T22:35:17.519 INFO:tasks.rados.rados.0.smithi076.stdout:3641: done (2 left) 2021-03-15T22:35:17.520 INFO:tasks.rados.rados.0.smithi076.stdout:3644: read oid 28 snap -1 2021-03-15T22:35:17.520 INFO:tasks.rados.rados.0.smithi076.stdout:3644: expect deleted 2021-03-15T22:35:17.520 INFO:tasks.rados.rados.0.smithi076.stdout:3645: read oid 31 snap -1 2021-03-15T22:35:17.520 INFO:tasks.rados.rados.0.smithi076.stdout:3645: expect deleted 2021-03-15T22:35:17.520 INFO:tasks.rados.rados.0.smithi076.stdout:3646: copy_from oid 42 from oid 19 current snap is 428 2021-03-15T22:35:17.521 INFO:tasks.rados.rados.0.smithi076.stdout:3647: snap_create 2021-03-15T22:35:17.522 INFO:tasks.rados.rados.0.smithi076.stdout:3642: finishing write tid 2 to smithi07694878-10 2021-03-15T22:35:17.522 INFO:tasks.rados.rados.0.smithi076.stdout:3642: finishing write tid 3 to smithi07694878-10 2021-03-15T22:35:17.522 INFO:tasks.rados.rados.0.smithi076.stdout:3642: finishing write tid 4 to smithi07694878-10 2021-03-15T22:35:17.522 INFO:tasks.rados.rados.0.smithi076.stdout:3642: finishing write tid 5 to smithi07694878-10 2021-03-15T22:35:17.523 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 2437 (ObjNum 1241 snap 428 seq_num 1241) dirty exists 2021-03-15T22:35:17.523 INFO:tasks.rados.rados.0.smithi076.stdout:3642: left oid 10 (ObjNum 1241 snap 428 seq_num 1241) 2021-03-15T22:35:17.523 INFO:tasks.rados.rados.0.smithi076.stdout:3643: finishing rollback tid 0 to smithi07694878-39 2021-03-15T22:35:17.525 INFO:tasks.rados.rados.0.smithi076.stdout:3643: finishing rollback tid 1 to smithi07694878-39 2021-03-15T22:35:17.526 INFO:tasks.rados.rados.0.smithi076.stdout:3643: finishing rollback tid 2 to smithi07694878-39 2021-03-15T22:35:17.526 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2812 (ObjNum 829 snap 275 seq_num 829) dirty exists 2021-03-15T22:35:17.542 INFO:tasks.rados.rados.0.smithi076.stdout:3646: finishing copy_from to smithi07694878-42 2021-03-15T22:35:17.542 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2800 (ObjNum 1204 snap 418 seq_num 1204) dirty exists 2021-03-15T22:35:17.543 INFO:tasks.rados.rados.0.smithi076.stdout:3646: finishing copy_from racing read to smithi07694878-42 2021-03-15T22:35:17.603 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:17.625 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:17.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:17.627 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:17.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:17.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: cluster 2021-03-15T22:35:16.036082+0000 osd.1 (osd. 2021-03-15T22:35:17.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: 1) 180 : cluster [DBG] 7.b scrub ok 2021-03-15T22:35:17.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: cluster 2021-03-15T22:35:16.256500+0000 osd.1 (osd.1) 181 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:17.733 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: cluster 2021-03-15T22:35:16.258853 2021-03-15T22:35:17.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: +0000 osd.1 (osd.1) 182 : cluster [DBG] 2021-03-15T22:35:17.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: purged_snaps scrub ok 2021-03-15T22:35:17.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: cluster 2021-03-15T22:35:16 2021-03-15T22:35:17.734 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: .472601+0000 mon.a (mon.0) 2401 : 2021-03-15T22:35:17.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: cluster [DBG] osdmap e961: 8 total, 8 up, 8 in 2021-03-15T22:35:17.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:35:17.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: :35:16.799881+0000 mon.c (mon.1 2021-03-15T22:35:17.735 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: ) 79 : audit [INF] from='client.? 172.21.15.76:0/2967012423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:17.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:35:17.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: T22:35:16.800180+0000 mon.a (mon 2021-03-15T22:35:17.736 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:17 smithi076 conmon[62156]: .0) 2402 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:17.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:17.752 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:17.752 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: cluster 2021-03-15T22:35:16.036082+0000 osd.1 (osd. 2021-03-15T22:35:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: 1) 180 : cluster [DBG] 7.b scrub ok 2021-03-15T22:35:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: cluster 2021-03-15T22:35:16.256500+0000 osd.1 (osd.1) 181 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:17.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: cluster 2021-03-15T22:35:16.258853+0000 osd.1 (osd.1) 182 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:17.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:35:17.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: 22:35:16.472601+0000 mon.a (mon.0) 2401 : cluster [DBG] 2021-03-15T22:35:17.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: osdmap e961: 8 total, 8 up, 8 in 2021-03-15T22:35:17.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: audit 2021 2021-03-15T22:35:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: -03-15T22:35:16.799881+0000 mon.c (mon.1) 79 : audit [INF] from='client.? 172.21.15.76:0/2967012423' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: 2021-03-15T22:35:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:35:17.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:17 smithi125 conmon[56739]: 22:35:16.800180+0000 mon.a (mon.0) 2402 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:17.853 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:17.876 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:17.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:17.878 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:17.979 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:18.002 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:18.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:18.004 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:18.105 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:18.128 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:18.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:18.130 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:18.231 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:18.253 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:18.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:18.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:18.356 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:18.378 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:18.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:18.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:18.481 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:18.491 INFO:tasks.rados.rados.0.smithi076.stdout:3642: done (5 left) 2021-03-15T22:35:18.492 INFO:tasks.rados.rados.0.smithi076.stdout:3643: done (4 left) 2021-03-15T22:35:18.492 INFO:tasks.rados.rados.0.smithi076.stdout:3644: done (3 left) 2021-03-15T22:35:18.492 INFO:tasks.rados.rados.0.smithi076.stdout:3645: done (2 left) 2021-03-15T22:35:18.492 INFO:tasks.rados.rados.0.smithi076.stdout:3646: done (1 left) 2021-03-15T22:35:18.493 INFO:tasks.rados.rados.0.smithi076.stdout:3647: done (0 left) 2021-03-15T22:35:18.493 INFO:tasks.rados.rados.0.smithi076.stdout:3648: delete oid 18 current snap is 429 2021-03-15T22:35:18.496 INFO:teuthology.orchestra.run.smithi076.stderr:set 2.2 pg_upmap_items mapping to [2->1,5->7,6->0] 2021-03-15T22:35:18.497 INFO:tasks.rados.rados.0.smithi076.stdout:3648: done (0 left) 2021-03-15T22:35:18.498 INFO:tasks.rados.rados.0.smithi076.stdout:3649: read oid 50 snap -1 2021-03-15T22:35:18.498 INFO:tasks.rados.rados.0.smithi076.stdout:3649: expect (ObjNum 1206 snap 418 seq_num 1206) 2021-03-15T22:35:18.498 INFO:tasks.rados.rados.0.smithi076.stdout:3650: read oid 4 snap 388 2021-03-15T22:35:18.498 INFO:tasks.rados.rados.0.smithi076.stdout:3650: expect (ObjNum 1109 snap 386 seq_num 1109) 2021-03-15T22:35:18.498 INFO:tasks.rados.rados.0.smithi076.stdout:3651: write oid 27 current snap is 429 2021-03-15T22:35:18.499 INFO:tasks.rados.rados.0.smithi076.stdout:3651: seq_num 1243 ranges {491101=500046,1487353=488104,2401133=489968} 2021-03-15T22:35:18.503 INFO:tasks.rados.rados.0.smithi076.stdout:3651: writing smithi07694878-27 from 491101 to 991147 tid 1 2021-03-15T22:35:18.508 INFO:tasks.rados.rados.0.smithi076.stdout:3651: writing smithi07694878-27 from 1487353 to 1975457 tid 2 2021-03-15T22:35:18.511 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:18.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:18.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:18.514 INFO:tasks.rados.rados.0.smithi076.stdout:3651: writing smithi07694878-27 from 2401133 to 2891101 tid 3 2021-03-15T22:35:18.514 INFO:tasks.rados.rados.0.smithi076.stdout:3652: read oid 47 snap -1 2021-03-15T22:35:18.515 INFO:tasks.rados.rados.0.smithi076.stdout:3652: expect deleted 2021-03-15T22:35:18.515 INFO:tasks.rados.rados.0.smithi076.stdout:3653: write (excl) oid 7 current snap is 429 2021-03-15T22:35:18.515 INFO:tasks.rados.rados.0.smithi076.stdout:3653: seq_num 1244 ranges {507908=575295,1829482=631762} 2021-03-15T22:35:18.521 INFO:tasks.rados.rados.0.smithi076.stdout:3653: writing smithi07694878-7 from 507908 to 1083203 tid 1 2021-03-15T22:35:18.531 INFO:tasks.rados.rados.0.smithi076.stdout:3653: writing smithi07694878-7 from 1829482 to 2461244 tid 2 2021-03-15T22:35:18.531 INFO:tasks.rados.rados.0.smithi076.stdout:3654: copy_from oid 48 from oid 32 current snap is 429 2021-03-15T22:35:18.531 INFO:tasks.rados.rados.0.smithi076.stdout:3650: expect (ObjNum 1109 snap 386 seq_num 1109) 2021-03-15T22:35:18.614 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:18.638 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:18.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:18.640 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:18.729 INFO:tasks.rados.rados.0.smithi076.stdout:3649: expect (ObjNum 1206 snap 418 seq_num 1206) 2021-03-15T22:35:18.741 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:18.767 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:18.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:18.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[56514]: cluster 2021-03-15T22:35:16.848915+0000 osd.5 (osd.5 2021-03-15T22:35:18.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[56514]: ) 53 : cluster [DBG] 5.1b scrub ok 2021-03-15T22:35:18.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[56514]: cluster 2021-03-15T22:35:16.957326+0000 osd.3 (osd.3) 169 : cluster [DBG] 7.2 scrub ok 2021-03-15T22:35:18.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[56514]: cluster 2021-03-15T22:35:17.217558+0000 mgr.y (mgr.14144) 1071 : cluster [DBG] pgmap v1589: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 658 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 6/1899 objects degraded (0.316%); 10/1899 objects misplaced (0.527%); 15 MiB/s, 14 objects/s recovering 2021-03-15T22:35:18.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[56514]: cluster 2021-03-15T22:35:17.250431+0000 osd.0 (osd.0) 154 : cluster [DBG] 6.10 scrub ok 2021-03-15T22:35:18.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[56514]: cluster 2021-03-15T22:35:17.301889+0000 osd.2 (osd.2) 44 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:35:18.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[56514]: cluster 2021-03-15T22:35:17.468955+0000 mon.a (mon.0) 2403 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:35:18.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[56514]: audit 2021-03-15T22:35:17.477916+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:18.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[56514]: cluster 2021-03-15T22:35:17.478006+0000 mon.a (mon.0) 2405 : cluster [DBG] osdmap e962: 8 total, 8 up, 8 in 2021-03-15T22:35:18.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[56514]: audit 2021-03-15T22:35:18.416336+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? 172.21.15.76:0/824830697' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.2", "id": [2, 1, 5, 7, 6, 0]}]: dispatch 2021-03-15T22:35:18.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: cluster 2021-03-15T22:35:16. 2021-03-15T22:35:18.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: 848915+0000 osd.5 (osd.5) 53 : cluster [DBG] 5.1b scrub ok 2021-03-15T22:35:18.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: cluster 2021-03-15T22:35:16. 2021-03-15T22:35:18.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: 957326+0000 osd.3 (osd.3) 169 : cluster [DBG] 7.2 scrub ok 2021-03-15T22:35:18.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:35:18.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: -03-15T22:35:17.217558+0000 2021-03-15T22:35:18.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: mgr.y (mgr.14144) 1071 : 2021-03-15T22:35:18.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: cluster [DBG] pgmap v1589: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 658 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 6/1899 objects degraded (0.316%); 10/1899 objects misplaced (0.527%); 15 MiB/s, 14 objects/s recovering 2021-03-15T22:35:18.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:35:18.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: -03-15T22:35: 2021-03-15T22:35:18.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: 17.250431+0000 osd.0 (osd 2021-03-15T22:35:18.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: .0) 154 : cluster [DBG] 2021-03-15T22:35:18.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: 6.10 scrub ok 2021-03-15T22:35:18.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: cluster 2021-03-15T22:35:18.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:35:18.774 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: 35:17.301889+0000 osd.2 2021-03-15T22:35:18.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: (osd.2) 44 : cluster 2021-03-15T22:35:18.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: [DBG] 2.1 scrub ok 2021-03-15T22:35:18.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:35:18.775 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: -03-15T22:35: 2021-03-15T22:35:18.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: 17.468955+0000 mon.a (mon 2021-03-15T22:35:18.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: .0) 2403 : cluster [INF] 2021-03-15T22:35:18.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:35:18.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: audit 2021-03-15T22:35:18.776 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:35:18.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: :35:17.477916+0000 2021-03-15T22:35:18.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: mon.a (mon.0) 2404 2021-03-15T22:35:18.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:18.777 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:35:18.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: -03-15T22:35: 2021-03-15T22:35:18.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: 17.478006+0000 mon.a (mon 2021-03-15T22:35:18.778 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: .0) 2405 : cluster [DBG] 2021-03-15T22:35:18.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: osdmap e962: 8 total, 8 up, 8 in 2021-03-15T22:35:18.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: audit 2021-03-15T22:35:18.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:35:18.779 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: 35:18.416336+0000 mon.a 2021-03-15T22:35:18.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: (mon.0) 2406 : audit 2021-03-15T22:35:18.780 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:18 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/824830697' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.2", "id": [2, 1, 5, 7, 6, 0]}]: dispatch 2021-03-15T22:35:18.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:18.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: cluster 2021-03-15T22:35:16.848915+0000 osd.5 (osd.5) 53 : cluster [DBG] 5.1b scrub ok 2021-03-15T22:35:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: cluster 2021-03-15T22:35:16. 2021-03-15T22:35:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: 957326+0000 osd.3 (osd.3) 169 : cluster [DBG] 7.2 scrub ok 2021-03-15T22:35:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: cluster 2021-03-15T22:35:17.217558+0000 mgr.y (mgr.14144) 1071 : cluster [DBG] pgmap v1589: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 658 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 6/1899 objects degraded (0.316%); 10/1899 objects misplaced (0.527%); 15 MiB/s, 14 objects/s recovering 2021-03-15T22:35:18.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: cluster 2021-03-15T22:35:17.250431+0000 osd.0 ( 2021-03-15T22:35:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: osd.0) 154 : cluster [DBG] 6.10 scrub ok 2021-03-15T22:35:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: cluster 2021-03-15T22:35:17.301889+0000 osd.2 (osd.2 2021-03-15T22:35:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: ) 44 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:35:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:35:18.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: T22:35:17.468955+0000 mon.a (mon.0) 2403 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:35:18.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: audit 2021-03-15T22:35: 2021-03-15T22:35:18.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: 17.477916+0000 mon.a (mon.0) 2404 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:18.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:35:18.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: 22:35:17.478006+0000 mon.a (mon.0) 2405 : cluster [DBG] 2021-03-15T22:35:18.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: osdmap e962: 8 total, 8 up, 8 in 2021-03-15T22:35:18.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:35:18.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: 03-15T22:35:18.416336+0000 mon.a (mon.0) 2406 : audit [INF] from='client.? 172.21.15.76:0/824830697' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.2", "id": [2, 1, 5, 7, 6, 0]}]: dispatch 2021-03-15T22:35:18.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:18 smithi125 conmon[56739]: 2021-03-15T22:35:18.832 INFO:tasks.rados.rados.0.smithi076.stdout:3651: finishing write tid 1 to smithi07694878-27 2021-03-15T22:35:18.833 INFO:tasks.rados.rados.0.smithi076.stdout:3651: finishing write tid 2 to smithi07694878-27 2021-03-15T22:35:18.833 INFO:tasks.rados.rados.0.smithi076.stdout:3651: finishing write tid 3 to smithi07694878-27 2021-03-15T22:35:18.833 INFO:tasks.rados.rados.0.smithi076.stdout:3651: finishing write tid 4 to smithi07694878-27 2021-03-15T22:35:18.833 INFO:tasks.rados.rados.0.smithi076.stdout:3649: done (5 left) 2021-03-15T22:35:18.834 INFO:tasks.rados.rados.0.smithi076.stdout:3650: done (4 left) 2021-03-15T22:35:18.834 INFO:tasks.rados.rados.0.smithi076.stdout:3652: done (3 left) 2021-03-15T22:35:18.834 INFO:tasks.rados.rados.0.smithi076.stdout:3655: copy_from oid 6 from oid 50 current snap is 429 2021-03-15T22:35:18.834 INFO:tasks.rados.rados.0.smithi076.stdout:3651: finishing write tid 5 to smithi07694878-27 2021-03-15T22:35:18.835 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 3042 (ObjNum 1243 snap 429 seq_num 1243) dirty exists 2021-03-15T22:35:18.835 INFO:tasks.rados.rados.0.smithi076.stdout:3651: left oid 27 (ObjNum 1243 snap 429 seq_num 1243) 2021-03-15T22:35:18.835 INFO:tasks.rados.rados.0.smithi076.stdout:3653: finishing write tid 1 to smithi07694878-7 2021-03-15T22:35:18.835 INFO:tasks.rados.rados.0.smithi076.stdout:3653: finishing write tid 2 to smithi07694878-7 2021-03-15T22:35:18.835 INFO:tasks.rados.rados.0.smithi076.stdout:3653: finishing write tid 3 to smithi07694878-7 2021-03-15T22:35:18.836 INFO:tasks.rados.rados.0.smithi076.stdout:3653: finishing write tid 4 to smithi07694878-7 2021-03-15T22:35:18.836 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2815 (ObjNum 1244 snap 429 seq_num 1244) dirty exists 2021-03-15T22:35:18.836 INFO:tasks.rados.rados.0.smithi076.stdout:3653: left oid 7 (ObjNum 1244 snap 429 seq_num 1244) 2021-03-15T22:35:18.837 INFO:tasks.rados.rados.0.smithi076.stdout:3654: finishing copy_from to smithi07694878-48 2021-03-15T22:35:18.837 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 48 v 3155 (ObjNum 1221 snap 423 seq_num 1221) dirty exists 2021-03-15T22:35:18.837 INFO:tasks.rados.rados.0.smithi076.stdout:3654: finishing copy_from racing read to smithi07694878-48 2021-03-15T22:35:18.837 INFO:tasks.rados.rados.0.smithi076.stdout:3651: done (3 left) 2021-03-15T22:35:18.838 INFO:tasks.rados.rados.0.smithi076.stdout:3653: done (2 left) 2021-03-15T22:35:18.838 INFO:tasks.rados.rados.0.smithi076.stdout:3654: done (1 left) 2021-03-15T22:35:18.838 INFO:tasks.rados.rados.0.smithi076.stdout:3656: write oid 8 current snap is 429 2021-03-15T22:35:18.838 INFO:tasks.rados.rados.0.smithi076.stdout:3656: seq_num 1245 ranges {524715=650544,1887964=491773} 2021-03-15T22:35:18.839 INFO:tasks.rados.rados.0.smithi076.stdout:3656: writing smithi07694878-8 from 524715 to 1175259 tid 1 2021-03-15T22:35:18.845 INFO:tasks.rados.rados.0.smithi076.stdout:3656: writing smithi07694878-8 from 1887964 to 2379737 tid 2 2021-03-15T22:35:18.845 INFO:tasks.rados.rados.0.smithi076.stdout:3657: read oid 23 snap -1 2021-03-15T22:35:18.846 INFO:tasks.rados.rados.0.smithi076.stdout:3657: expect (ObjNum 1189 snap 416 seq_num 1189) 2021-03-15T22:35:18.846 INFO:tasks.rados.rados.0.smithi076.stdout:3658: write oid 7 current snap is 429 2021-03-15T22:35:18.846 INFO:tasks.rados.rados.0.smithi076.stdout:3655: finishing copy_from to smithi07694878-6 2021-03-15T22:35:18.846 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 2680 (ObjNum 1206 snap 418 seq_num 1206) dirty exists 2021-03-15T22:35:18.847 INFO:tasks.rados.rados.0.smithi076.stdout:3655: finishing copy_from racing read to smithi07694878-6 2021-03-15T22:35:18.847 INFO:tasks.rados.rados.0.smithi076.stdout:3656: finishing write tid 1 to smithi07694878-8 2021-03-15T22:35:18.847 INFO:tasks.rados.rados.0.smithi076.stdout:3658: seq_num 1246 ranges {541522=725793,1946446=635431} 2021-03-15T22:35:18.852 INFO:tasks.rados.rados.0.smithi076.stdout:3658: writing smithi07694878-7 from 541522 to 1267315 tid 1 2021-03-15T22:35:18.859 INFO:tasks.rados.rados.0.smithi076.stdout:3658: writing smithi07694878-7 from 1946446 to 2581877 tid 2 2021-03-15T22:35:18.859 INFO:tasks.rados.rados.0.smithi076.stdout:3655: done (3 left) 2021-03-15T22:35:18.860 INFO:tasks.rados.rados.0.smithi076.stdout:3659: snap_create 2021-03-15T22:35:18.860 INFO:tasks.rados.rados.0.smithi076.stdout:3656: finishing write tid 2 to smithi07694878-8 2021-03-15T22:35:18.860 INFO:tasks.rados.rados.0.smithi076.stdout:3656: finishing write tid 3 to smithi07694878-8 2021-03-15T22:35:18.860 INFO:tasks.rados.rados.0.smithi076.stdout:3656: finishing write tid 4 to smithi07694878-8 2021-03-15T22:35:18.861 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 3158 (ObjNum 1245 snap 429 seq_num 1245) dirty exists 2021-03-15T22:35:18.861 INFO:tasks.rados.rados.0.smithi076.stdout:3656: left oid 8 (ObjNum 1245 snap 429 seq_num 1245) 2021-03-15T22:35:18.861 INFO:tasks.rados.rados.0.smithi076.stdout:3657: expect (ObjNum 1189 snap 416 seq_num 1189) 2021-03-15T22:35:18.882 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:18.908 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:18.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:18.910 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:19.011 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:19.034 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:19.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:19.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:19.044 INFO:tasks.rados.rados.0.smithi076.stdout:3658: finishing write tid 1 to smithi07694878-7 2021-03-15T22:35:19.044 INFO:tasks.rados.rados.0.smithi076.stdout:3658: finishing write tid 2 to smithi07694878-7 2021-03-15T22:35:19.044 INFO:tasks.rados.rados.0.smithi076.stdout:3658: finishing write tid 3 to smithi07694878-7 2021-03-15T22:35:19.044 INFO:tasks.rados.rados.0.smithi076.stdout:3658: finishing write tid 4 to smithi07694878-7 2021-03-15T22:35:19.045 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2818 (ObjNum 1246 snap 429 seq_num 1246) dirty exists 2021-03-15T22:35:19.045 INFO:tasks.rados.rados.0.smithi076.stdout:3658: left oid 7 (ObjNum 1246 snap 429 seq_num 1246) 2021-03-15T22:35:19.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:19.159 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:19.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:19.160 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:19.261 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:19.285 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:19.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:19.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:19.388 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:19.410 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:19.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:19.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:19.493 INFO:tasks.rados.rados.0.smithi076.stdout:3656: done (3 left) 2021-03-15T22:35:19.493 INFO:tasks.rados.rados.0.smithi076.stdout:3657: done (2 left) 2021-03-15T22:35:19.493 INFO:tasks.rados.rados.0.smithi076.stdout:3658: done (1 left) 2021-03-15T22:35:19.493 INFO:tasks.rados.rados.0.smithi076.stdout:3659: done (0 left) 2021-03-15T22:35:19.494 INFO:tasks.rados.rados.0.smithi076.stdout:3660: delete oid 40 current snap is 430 2021-03-15T22:35:19.498 INFO:tasks.rados.rados.0.smithi076.stdout:3660: done (0 left) 2021-03-15T22:35:19.499 INFO:tasks.rados.rados.0.smithi076.stdout:3661: rollback oid 15 current snap is 430 2021-03-15T22:35:19.499 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 15 to 364 2021-03-15T22:35:19.499 INFO:tasks.rados.rados.0.smithi076.stdout:3662: copy_from oid 49 from oid 21 current snap is 430 2021-03-15T22:35:19.499 INFO:tasks.rados.rados.0.smithi076.stdout:3663: read oid 26 snap 427 2021-03-15T22:35:19.499 INFO:tasks.rados.rados.0.smithi076.stdout:3663: expect deleted 2021-03-15T22:35:19.500 INFO:tasks.rados.rados.0.smithi076.stdout:3664: rollback oid 7 current snap is 430 2021-03-15T22:35:19.500 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 7 to 393 2021-03-15T22:35:19.500 INFO:tasks.rados.rados.0.smithi076.stdout:3665: copy_from oid 20 from oid 47 current snap is 430 2021-03-15T22:35:19.500 INFO:tasks.rados.rados.0.smithi076.stdout:3666: read oid 45 snap -1 2021-03-15T22:35:19.501 INFO:tasks.rados.rados.0.smithi076.stdout:3666: expect (ObjNum 1034 snap 346 seq_num 1034) 2021-03-15T22:35:19.501 INFO:tasks.rados.rados.0.smithi076.stdout:3667: delete oid 9 current snap is 430 2021-03-15T22:35:19.501 INFO:tasks.rados.rados.0.smithi076.stdout:3665: finishing copy_from to smithi07694878-20 2021-03-15T22:35:19.502 INFO:tasks.rados.rados.0.smithi076.stdout:3665: got expected ENOENT (src dne) 2021-03-15T22:35:19.502 INFO:tasks.rados.rados.0.smithi076.stdout:3665: finishing copy_from racing read to smithi07694878-20 2021-03-15T22:35:19.503 INFO:tasks.rados.rados.0.smithi076.stdout:3664: finishing rollback tid 0 to smithi07694878-7 2021-03-15T22:35:19.506 INFO:tasks.rados.rados.0.smithi076.stdout:3664: finishing rollback tid 1 to smithi07694878-7 2021-03-15T22:35:19.506 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2820 (ObjNum 262131872 snap 21896 seq_num 1902431456) dirty dne 2021-03-15T22:35:19.507 INFO:tasks.rados.rados.0.smithi076.stdout:3661: finishing rollback tid 0 to smithi07694878-15 2021-03-15T22:35:19.507 INFO:tasks.rados.rados.0.smithi076.stdout:3661: finishing rollback tid 1 to smithi07694878-15 2021-03-15T22:35:19.508 INFO:tasks.rados.rados.0.smithi076.stdout:3661: finishing rollback tid 2 to smithi07694878-15 2021-03-15T22:35:19.508 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 3073 (ObjNum 971 snap 322 seq_num 971) dirty exists 2021-03-15T22:35:19.508 INFO:tasks.rados.rados.0.smithi076.stdout:3666: expect (ObjNum 1034 snap 346 seq_num 1034) 2021-03-15T22:35:19.513 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:19.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[56514]: cluster 2021-03-15T22:35:18.111811+0000 osd.1 ( 2021-03-15T22:35:19.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[56514]: osd.1) 183 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:35:19.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[56514]: cluster 2021-03-15T22:35:18.338282+0000 osd.2 (osd.2) 45 : cluster [DBG] 7.e scrub ok 2021-03-15T22:35:19.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[56514]: audit 2021-03-15T22:35:18.485632+0000 mon.a (mon.0) 2407 : audit [INF] from='client.? 172.21.15.76:0/824830697' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.2", "id": [2, 1, 5, 7, 6, 0]}]': finished 2021-03-15T22:35:19.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[56514]: cluster 2021-03-15T22:35:18.485709+0000 mon.a (mon.0) 2408 : cluster [DBG] osdmap e963: 8 total, 8 up, 8 in 2021-03-15T22:35:19.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[56514]: cluster 2021-03-15T22:35:18.617909+0000 osd.6 (osd.6) 141 : cluster [DBG] 6.27 scrub ok 2021-03-15T22:35:19.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: cluster 2021-03-15T22:35:18.111811+0000 2021-03-15T22:35:19.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: osd.1 (osd.1) 183 : cluster [DBG] 6.1d scrub ok 2021-03-15T22:35:19.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:35:19.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: 15T22:35:18.338282+0000 osd.2 (osd.2) 45 : cluster [DBG] 7.e scrub ok 2021-03-15T22:35:19.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: 2021-03-15T22:35:19.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: audit 2021-03-15T22:35:18 2021-03-15T22:35:19.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: .485632+0000 mon.a (mon.0) 2407 : 2021-03-15T22:35:19.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: audit [INF] from='client.? 172.21.15.76:0/824830697' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.2", "id": [2, 1, 5, 7, 6, 0]}]': finished 2021-03-15T22:35:19.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: cluster 2021-03-15T22:35 2021-03-15T22:35:19.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: :18.485709+0000 mon.a (mon.0) 2408 2021-03-15T22:35:19.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: : cluster [DBG] osdmap e963: 8 total, 8 up, 8 in 2021-03-15T22:35:19.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:35:19.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: T22:35:18.617909+0000 osd.6 (osd 2021-03-15T22:35:19.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:19 smithi076 conmon[62156]: .6) 141 : cluster [DBG] 6.27 scrub ok 2021-03-15T22:35:19.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:19.539 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:19.539 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:19.618 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:35:19.634 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:35:19.641 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:19.665 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:19.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:19.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:19.667 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:35:19.699 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:35:19.717 INFO:tasks.rados.rados.0.smithi076.stdout:3662: finishing copy_from to smithi07694878-49 2021-03-15T22:35:19.718 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 49 v 3159 (ObjNum 1225 snap 424 seq_num 1225) dirty exists 2021-03-15T22:35:19.718 INFO:tasks.rados.rados.0.smithi076.stdout:3662: finishing copy_from racing read to smithi07694878-49 2021-03-15T22:35:19.719 INFO:tasks.rados.rados.0.smithi076.stdout:3661: done (6 left) 2021-03-15T22:35:19.719 INFO:tasks.rados.rados.0.smithi076.stdout:3662: done (5 left) 2021-03-15T22:35:19.719 INFO:tasks.rados.rados.0.smithi076.stdout:3663: done (4 left) 2021-03-15T22:35:19.720 INFO:tasks.rados.rados.0.smithi076.stdout:3664: done (3 left) 2021-03-15T22:35:19.720 INFO:tasks.rados.rados.0.smithi076.stdout:3665: done (2 left) 2021-03-15T22:35:19.720 INFO:tasks.rados.rados.0.smithi076.stdout:3666: done (1 left) 2021-03-15T22:35:19.721 INFO:tasks.rados.rados.0.smithi076.stdout:3667: done (0 left) 2021-03-15T22:35:19.721 INFO:tasks.rados.rados.0.smithi076.stdout:3668: read oid 5 snap -1 2021-03-15T22:35:19.721 INFO:tasks.rados.rados.0.smithi076.stdout:3668: expect (ObjNum 1238 snap 427 seq_num 1238) 2021-03-15T22:35:19.722 INFO:tasks.rados.rados.0.smithi076.stdout:3669: copy_from oid 31 from oid 9 current snap is 430 2021-03-15T22:35:19.722 INFO:tasks.rados.rados.0.smithi076.stdout:3670: read oid 36 snap -1 2021-03-15T22:35:19.722 INFO:tasks.rados.rados.0.smithi076.stdout:3670: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:35:19.723 INFO:tasks.rados.rados.0.smithi076.stdout:3671: write (excl) oid 39 current snap is 430 2021-03-15T22:35:19.723 INFO:tasks.rados.rados.0.smithi076.stdout:3671: seq_num 1249 ranges {591943=667893,1721892=782758} 2021-03-15T22:35:19.724 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:35:19.725 INFO:tasks.rados.rados.0.smithi076.stdout:3671: writing smithi07694878-39 from 591943 to 1259836 tid 1 2021-03-15T22:35:19.731 INFO:tasks.rados.rados.0.smithi076.stdout:3671: writing smithi07694878-39 from 1721892 to 2504650 tid 2 2021-03-15T22:35:19.731 INFO:tasks.rados.rados.0.smithi076.stdout:3672: snap_create 2021-03-15T22:35:19.731 INFO:tasks.rados.rados.0.smithi076.stdout:3669: finishing copy_from to smithi07694878-31 2021-03-15T22:35:19.732 INFO:tasks.rados.rados.0.smithi076.stdout:3669: got expected ENOENT (src dne) 2021-03-15T22:35:19.732 INFO:tasks.rados.rados.0.smithi076.stdout:3669: finishing copy_from racing read to smithi07694878-31 2021-03-15T22:35:19.732 INFO:tasks.rados.rados.0.smithi076.stdout:3668: expect (ObjNum 1238 snap 427 seq_num 1238) 2021-03-15T22:35:19.741 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:35:19.756 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:35:19.756 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:35:19.756 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:35:19.757 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:35:19.757 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:35:19.758 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:35:19.768 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:19.777 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:35:19.777 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:35:19.778 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:35:19.778 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:35:19.778 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:35:19.779 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:35:19.779 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:35:19.779 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:35:19.780 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:35:19.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: cluster 2021-03-15T22:35:18.111811+0000 osd.1 (osd.1) 183 : cluster 2021-03-15T22:35:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: [DBG] 6.1d scrub ok 2021-03-15T22:35:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: cluster 2021-03-15T22:35:18.338282+0000 osd.2 (osd.2) 45 : cluster [DBG] 7.e scrub ok 2021-03-15T22:35:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: audit 2021 2021-03-15T22:35:19.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: -03-15T22:35:18.485632+0000 mon.a (mon.0) 2407 : audit 2021-03-15T22:35:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/824830697' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.2", "id": [2, 1, 5, 7, 6, 0]}]': finished 2021-03-15T22:35:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:35:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: T22:35:18.485709+0000 2021-03-15T22:35:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: mon.a (mon.0) 2408 : cluster [DBG] osdmap e963: 8 total, 8 up, 8 in 2021-03-15T22:35:19.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: 2021-03-15T22:35:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:35:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: T22:35:18.617909+0000 2021-03-15T22:35:19.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:19 smithi125 conmon[56739]: osd.6 (osd.6) 141 : cluster [DBG] 6.27 scrub ok 2021-03-15T22:35:19.797 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:19.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:19.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:19.900 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:19.925 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:19.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:19.927 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:19.999 INFO:tasks.rados.rados.0.smithi076.stdout:3670: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:35:20.028 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:20.053 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:20.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:20.055 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:20.097 INFO:tasks.rados.rados.0.smithi076.stdout:3671: finishing write tid 1 to smithi07694878-39 2021-03-15T22:35:20.097 INFO:tasks.rados.rados.0.smithi076.stdout:3671: finishing write tid 2 to smithi07694878-39 2021-03-15T22:35:20.098 INFO:tasks.rados.rados.0.smithi076.stdout:3671: finishing write tid 3 to smithi07694878-39 2021-03-15T22:35:20.098 INFO:tasks.rados.rados.0.smithi076.stdout:3671: finishing write tid 4 to smithi07694878-39 2021-03-15T22:35:20.098 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2823 (ObjNum 1249 snap 430 seq_num 1249) dirty exists 2021-03-15T22:35:20.098 INFO:tasks.rados.rados.0.smithi076.stdout:3671: left oid 39 (ObjNum 1249 snap 430 seq_num 1249) 2021-03-15T22:35:20.156 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:20.178 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:20.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:20.180 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:20.281 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:20.307 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:20.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:20.308 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:20.336 INFO:tasks.rados.rados.0.smithi076.stdout:3668: done (4 left) 2021-03-15T22:35:20.337 INFO:tasks.rados.rados.0.smithi076.stdout:3669: done (3 left) 2021-03-15T22:35:20.337 INFO:tasks.rados.rados.0.smithi076.stdout:3670: done (2 left) 2021-03-15T22:35:20.337 INFO:tasks.rados.rados.0.smithi076.stdout:3671: done (1 left) 2021-03-15T22:35:20.338 INFO:tasks.rados.rados.0.smithi076.stdout:3672: done (0 left) 2021-03-15T22:35:20.338 INFO:tasks.rados.rados.0.smithi076.stdout:3673: rollback oid 22 current snap is 431 2021-03-15T22:35:20.338 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 22 to 426 2021-03-15T22:35:20.338 INFO:tasks.rados.rados.0.smithi076.stdout:3674: write (excl) oid 11 current snap is 431 2021-03-15T22:35:20.339 INFO:tasks.rados.rados.0.smithi076.stdout:3674: seq_num 1250 ranges {608750=743142,1780374=642769} 2021-03-15T22:35:20.344 INFO:tasks.rados.rados.0.smithi076.stdout:3674: writing smithi07694878-11 from 608750 to 1351892 tid 1 2021-03-15T22:35:20.350 INFO:tasks.rados.rados.0.smithi076.stdout:3674: writing smithi07694878-11 from 1780374 to 2423143 tid 2 2021-03-15T22:35:20.351 INFO:tasks.rados.rados.0.smithi076.stdout:3675: copy_from oid 20 from oid 14 current snap is 431 2021-03-15T22:35:20.351 INFO:tasks.rados.rados.0.smithi076.stdout:3673: finishing rollback tid 0 to smithi07694878-22 2021-03-15T22:35:20.351 INFO:tasks.rados.rados.0.smithi076.stdout:3673: finishing rollback tid 1 to smithi07694878-22 2021-03-15T22:35:20.351 INFO:tasks.rados.rados.0.smithi076.stdout:3673: finishing rollback tid 2 to smithi07694878-22 2021-03-15T22:35:20.351 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 3162 (ObjNum 1181 snap 414 seq_num 1181) dirty exists 2021-03-15T22:35:20.352 INFO:tasks.rados.rados.0.smithi076.stdout:3674: finishing write tid 1 to smithi07694878-11 2021-03-15T22:35:20.352 INFO:tasks.rados.rados.0.smithi076.stdout:3673: done (2 left) 2021-03-15T22:35:20.354 INFO:tasks.rados.rados.0.smithi076.stdout:3676: snap_remove snap 321 2021-03-15T22:35:20.409 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:20.431 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:20.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:20.433 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:20.442 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:35:20.534 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:20.557 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:20.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:20.559 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:20.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:20.682 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:20.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:20.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:20.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[56514]: cluster 2021-03-15T22:35:19.218047+0000 mgr.y (mgr.14144) 1072 : cluster [DBG] pgmap v1592: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 664 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 12 op/s; 6/1905 objects degraded (0.315%); 10/1905 objects misplaced (0.525%) 2021-03-15T22:35:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[56514]: cluster 2021-03-15T22:35:19.456870+0000 osd.7 (osd.7) 179 : cluster [DBG] 1.0 deep-scrub ok 2021-03-15T22:35:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[56514]: cluster 2021-03-15T22:35:19.468398+0000 osd.4 (osd.4) 193 : cluster [DBG] 5.e scrub ok 2021-03-15T22:35:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[56514]: cluster 2021-03-15T22:35:19.488893+0000 mon.a (mon.0) 2409 : cluster [DBG] osdmap e964: 8 total, 8 up, 8 in 2021-03-15T22:35:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[56514]: audit 2021-03-15T22:35:19.597650+0000 mon.a (mon.0) 2410 : audit [DBG] from='client.? 172.21.15.76:0/3883644816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[56514]: cluster 2021-03-15T22:35:20.328244+0000 mon.a (mon.0) 2411 : cluster [WRN] Health check update: Degraded data redundancy: 6/1905 objects degraded (0.315%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[56514]: cluster 2021-03-15T22:35:20.332699+0000 mon.a (mon.0) 2412 : cluster [DBG] osdmap e965: 8 total, 8 up, 8 in 2021-03-15T22:35:20.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[62156]: cluster 2021-03-15T22:35:19.218047+0000 mgr.y (mgr.14144) 1072 : cluster [DBG] pgmap v1592: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 664 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 12 op/s; 6/1905 objects degraded (0.315%); 10/1905 objects misplaced (0.525%) 2021-03-15T22:35:20.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[62156]: cluster 2021-03-15T22:35:19.456870+0000 osd.7 (osd.7) 179 : cluster [DBG] 1.0 deep-scrub ok 2021-03-15T22:35:20.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[62156]: cluster 2021-03-15T22:35:19.468398+0000 osd.4 (osd.4) 193 : cluster [DBG] 5.e scrub ok 2021-03-15T22:35:20.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[62156]: cluster 2021-03-15T22:35:19.488893+0000 mon.a (mon.0) 2409 : cluster [DBG] osdmap e964: 8 total, 8 up, 8 in 2021-03-15T22:35:20.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[62156]: audit 2021-03-15T22:35:19.597650+0000 mon.a (mon.0) 2410 : audit [DBG] from='client.? 172.21.15.76:0/3883644816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[62156]: cluster 2021-03-15T22:35:20. 2021-03-15T22:35:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[62156]: 328244+0000 mon.a (mon.0) 2411 : cluster [WRN] Health check update: Degraded data redundancy: 6/1905 objects degraded (0.315%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:20 smithi076 conmon[62156]: cluster 2021-03-15T22:35:20.332699+0000 mon.a (mon.0) 2412 : cluster [DBG] osdmap e965: 8 total, 8 up, 8 in 2021-03-15T22:35:20.785 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:20.787 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:35:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: cluster 2021-03-15T22:35:19.218047+0000 mgr.y (mgr.14144) 1072 : cluster 2021-03-15T22:35:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: [DBG] pgmap v1592: 158 pgs: 1 active+remapped+backfill_toofull, 1 active+recovering+undersized+degraded+remapped, 156 active+clean; 664 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 12 op/s; 6/1905 objects degraded (0.315%); 10/1905 objects misplaced (0.525%) 2021-03-15T22:35:20.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: cluster 2021-03-15T22:35:19.456870+0000 osd.7 (osd.7) 179 : cluster [DBG] 1.0 deep-scrub ok 2021-03-15T22:35:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: cluster 2021-03-15T22:35:19.468398 2021-03-15T22:35:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: +0000 osd.4 (osd.4) 193 : cluster [DBG] 2021-03-15T22:35:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: 5.e scrub ok 2021-03-15T22:35:20.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:35:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: -15T22:35:19.488893 2021-03-15T22:35:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2409 : cluster 2021-03-15T22:35:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: [DBG] osdmap e964: 8 total, 8 up, 8 in 2021-03-15T22:35:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:35:20.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: -15T22:35:19. 2021-03-15T22:35:20.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: 597650+0000 mon.a (mon.0) 2021-03-15T22:35:20.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: 2410 : audit [DBG] from='client.? 172.21.15.76:0/3883644816' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:20.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:35:20.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: -03-15T22:35: 2021-03-15T22:35:20.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: 20.328244+0000 mon.a (mon.0) 2021-03-15T22:35:20.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: 2411 : cluster [WRN] Health check update: Degraded data redundancy: 6/1905 objects degraded (0.315%), 1 pg degraded (PG_DEGRADED) 2021-03-15T22:35:20.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:35:20.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: :35:20.332699+0000 mon.a (mon.0) 2412 : cluster 2021-03-15T22:35:20.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:20 smithi125 conmon[56739]: [DBG] osdmap e965: 8 total, 8 up, 8 in 2021-03-15T22:35:20.807 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:20.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:20.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:20.910 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:20.934 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:20.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:20.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:21.037 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:21.060 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:21.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:21.062 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:21.084 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-15T22:35:21.106 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:21.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:21.186 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:21.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:21.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:21.288 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:21.312 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:21.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:21.356 INFO:tasks.rados.rados.0.smithi076.stdout:3676: done (2 left) 2021-03-15T22:35:21.357 INFO:tasks.rados.rados.0.smithi076.stdout:3677: rollback oid 13 current snap is 431 2021-03-15T22:35:21.357 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 13 to 367 2021-03-15T22:35:21.357 INFO:tasks.rados.rados.0.smithi076.stdout:3678: read oid 12 snap 370 2021-03-15T22:35:21.357 INFO:tasks.rados.rados.0.smithi076.stdout:3678: expect (ObjNum 999 snap 335 seq_num 999) 2021-03-15T22:35:21.357 INFO:tasks.rados.rados.0.smithi076.stdout:3679: delete oid 8 current snap is 431 2021-03-15T22:35:21.358 INFO:tasks.rados.rados.0.smithi076.stdout:3674: finishing write tid 2 to smithi07694878-11 2021-03-15T22:35:21.358 INFO:tasks.rados.rados.0.smithi076.stdout:3674: finishing write tid 3 to smithi07694878-11 2021-03-15T22:35:21.359 INFO:tasks.rados.rados.0.smithi076.stdout:3674: finishing write tid 4 to smithi07694878-11 2021-03-15T22:35:21.359 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 2434 (ObjNum 1250 snap 431 seq_num 1250) dirty exists 2021-03-15T22:35:21.359 INFO:tasks.rados.rados.0.smithi076.stdout:3674: left oid 11 (ObjNum 1250 snap 431 seq_num 1250) 2021-03-15T22:35:21.359 INFO:tasks.rados.rados.0.smithi076.stdout:3675: finishing copy_from to smithi07694878-20 2021-03-15T22:35:21.360 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 2677 (ObjNum 1140 snap 393 seq_num 1140) dirty exists 2021-03-15T22:35:21.360 INFO:tasks.rados.rados.0.smithi076.stdout:3675: finishing copy_from racing read to smithi07694878-20 2021-03-15T22:35:21.360 INFO:tasks.rados.rados.0.smithi076.stdout:3674: done (4 left) 2021-03-15T22:35:21.361 INFO:tasks.rados.rados.0.smithi076.stdout:3675: done (3 left) 2021-03-15T22:35:21.361 INFO:tasks.rados.rados.0.smithi076.stdout:3679: done (2 left) 2021-03-15T22:35:21.361 INFO:tasks.rados.rados.0.smithi076.stdout:3680: read oid 15 snap -1 2021-03-15T22:35:21.361 INFO:tasks.rados.rados.0.smithi076.stdout:3680: expect (ObjNum 971 snap 322 seq_num 971) 2021-03-15T22:35:21.362 INFO:tasks.rados.rados.0.smithi076.stdout:3681: read oid 36 snap -1 2021-03-15T22:35:21.362 INFO:tasks.rados.rados.0.smithi076.stdout:3681: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:35:21.362 INFO:tasks.rados.rados.0.smithi076.stdout:3682: read oid 3 snap 357 2021-03-15T22:35:21.362 INFO:tasks.rados.rados.0.smithi076.stdout:3682: expect (ObjNum 620 snap 193 seq_num 620) 2021-03-15T22:35:21.362 INFO:tasks.rados.rados.0.smithi076.stdout:3683: delete oid 1 current snap is 431 2021-03-15T22:35:21.367 INFO:tasks.rados.rados.0.smithi076.stdout:3683: done (5 left) 2021-03-15T22:35:21.368 INFO:tasks.rados.rados.0.smithi076.stdout:3684: copy_from oid 25 from oid 22 current snap is 431 2021-03-15T22:35:21.368 INFO:tasks.rados.rados.0.smithi076.stdout:3685: read oid 7 snap -1 2021-03-15T22:35:21.368 INFO:tasks.rados.rados.0.smithi076.stdout:3685: expect deleted 2021-03-15T22:35:21.368 INFO:tasks.rados.rados.0.smithi076.stdout:3686: copy_from oid 23 from oid 4 current snap is 431 2021-03-15T22:35:21.369 INFO:tasks.rados.rados.0.smithi076.stdout:3687: read oid 37 snap -1 2021-03-15T22:35:21.369 INFO:tasks.rados.rados.0.smithi076.stdout:3687: expect (ObjNum 1186 snap 414 seq_num 1186) 2021-03-15T22:35:21.369 INFO:tasks.rados.rados.0.smithi076.stdout:3688: delete oid 35 current snap is 431 2021-03-15T22:35:21.370 INFO:tasks.rados.rados.0.smithi076.stdout:3682: expect (ObjNum 620 snap 193 seq_num 620) 2021-03-15T22:35:21.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:21.436 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:21.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:21.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:21.505 INFO:tasks.rados.rados.0.smithi076.stdout:3682: done (9 left) 2021-03-15T22:35:21.505 INFO:tasks.rados.rados.0.smithi076.stdout:3688: done (8 left) 2021-03-15T22:35:21.505 INFO:tasks.rados.rados.0.smithi076.stdout:3689: snap_create 2021-03-15T22:35:21.506 INFO:tasks.rados.rados.0.smithi076.stdout:3678: expect (ObjNum 999 snap 335 seq_num 999) 2021-03-15T22:35:21.539 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:21.563 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:21.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:21.564 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:21.594 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:21.602 INFO:tasks.rados.rados.0.smithi076.stdout:3677: finishing rollback tid 0 to smithi07694878-13 2021-03-15T22:35:21.602 INFO:tasks.rados.rados.0.smithi076.stdout:3686: finishing copy_from to smithi07694878-23 2021-03-15T22:35:21.602 INFO:tasks.rados.rados.0.smithi076.stdout:3686: got expected ENOENT (src dne) 2021-03-15T22:35:21.603 INFO:tasks.rados.rados.0.smithi076.stdout:3686: finishing copy_from racing read to smithi07694878-23 2021-03-15T22:35:21.603 INFO:tasks.rados.rados.0.smithi076.stdout:3681: expect (ObjNum 669 snap 212 seq_num 669) 2021-03-15T22:35:21.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:21.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:21.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:21.712 INFO:tasks.rados.rados.0.smithi076.stdout:3677: finishing rollback tid 1 to smithi07694878-13 2021-03-15T22:35:21.712 INFO:tasks.rados.rados.0.smithi076.stdout:3677: finishing rollback tid 2 to smithi07694878-13 2021-03-15T22:35:21.712 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 2437 (ObjNum 1072 snap 366 seq_num 1072) dirty exists 2021-03-15T22:35:21.712 INFO:tasks.rados.rados.0.smithi076.stdout:3680: expect (ObjNum 971 snap 322 seq_num 971) 2021-03-15T22:35:21.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[56514]: cluster 2021-03-15T22:35:20.025888+0000 osd.3 (osd 2021-03-15T22:35:21.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[56514]: .3) 170 : cluster [DBG] 6.1f deep-scrub ok 2021-03-15T22:35:21.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[56514]: audit 2021-03-15T22:35:20.297425+0000 mgr.y (mgr.14144) 1073 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:21.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[56514]: cluster 2021-03-15T22:35:20.381348+0000 osd.2 (osd.2) 46 : cluster [DBG] 6.14 deep-scrub ok 2021-03-15T22:35:21.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[56514]: cluster 2021-03-15T22:35:20.678242+0000 osd.6 (osd.6) 142 : cluster [DBG] 3.a deep-scrub ok 2021-03-15T22:35:21.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[56514]: cluster 2021-03-15T22:35:21.339886+0000 mon.a (mon.0) 2413 : cluster [DBG] osdmap e966: 8 total, 8 up, 8 in 2021-03-15T22:35:21.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: cluster 2021-03-15T22:35:20 2021-03-15T22:35:21.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: .025888+0000 osd.3 (osd.3) 170 : cluster [DBG] 6.1f deep-scrub ok 2021-03-15T22:35:21.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: audit 2021-03-15T22:35 2021-03-15T22:35:21.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: :20.297425+0000 mgr.y (mgr.14144) 2021-03-15T22:35:21.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: 1073 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:21.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:35:21.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: -03-15T22:35:20. 2021-03-15T22:35:21.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: 381348+0000 osd.2 (osd.2) 46 : cluster 2021-03-15T22:35:21.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: [DBG] 6.14 deep-scrub ok 2021-03-15T22:35:21.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:35:21.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: 03-15T22:35:20.678242 2021-03-15T22:35:21.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: +0000 osd.6 (osd.6) 142 : cluster 2021-03-15T22:35:21.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: [DBG] 3.a deep-scrub ok 2021-03-15T22:35:21.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:35:21.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: 03-15T22:35:21.339886 2021-03-15T22:35:21.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2413 : cluster 2021-03-15T22:35:21.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:21 smithi076 conmon[62156]: [DBG] osdmap e966: 8 total, 8 up, 8 in 2021-03-15T22:35:21.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: cluster 2021-03-15T22:35:20.025888+0000 osd.3 (osd.3) 170 : 2021-03-15T22:35:21.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: cluster [DBG] 6.1f deep-scrub ok 2021-03-15T22:35:21.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: audit 2021-03-15T22:35:20.297425+0000 mgr.y (mgr.14144) 1073 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:21.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: cluster 2021-03-15T22:35:20.381348+0000 2021-03-15T22:35:21.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: osd.2 (osd.2) 46 : cluster [DBG] 2021-03-15T22:35:21.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: 6.14 deep-scrub ok 2021-03-15T22:35:21.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: cluster 2021-03-15T22:35 2021-03-15T22:35:21.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: :20.678242+0000 osd.6 (osd.6) 2021-03-15T22:35:21.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: 142 : cluster [DBG] 3.a deep-scrub ok 2021-03-15T22:35:21.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:35:21.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: 15T22:35:21.339886+0000 mon.a ( 2021-03-15T22:35:21.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:21 smithi125 conmon[56739]: mon.0) 2413 : cluster [DBG] osdmap e966: 8 total, 8 up, 8 in 2021-03-15T22:35:21.791 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:21.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:21.815 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:21.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:21.899 INFO:tasks.rados.rados.0.smithi076.stdout:3687: expect (ObjNum 1186 snap 414 seq_num 1186) 2021-03-15T22:35:21.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:21.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:21.941 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:21.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:22.043 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:22.066 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:22.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:22.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:22.086 INFO:tasks.rados.rados.0.smithi076.stdout:3684: finishing copy_from to smithi07694878-25 2021-03-15T22:35:22.087 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 3162 (ObjNum 1181 snap 414 seq_num 1181) dirty exists 2021-03-15T22:35:22.087 INFO:tasks.rados.rados.0.smithi076.stdout:3684: finishing copy_from racing read to smithi07694878-25 2021-03-15T22:35:22.170 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:22.192 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:22.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:22.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:22.295 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:22.320 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:22.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:22.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:22.423 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:22.446 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:22.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:22.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:22.506 INFO:tasks.rados.rados.0.smithi076.stdout:3677: done (8 left) 2021-03-15T22:35:22.507 INFO:tasks.rados.rados.0.smithi076.stdout:3678: done (7 left) 2021-03-15T22:35:22.507 INFO:tasks.rados.rados.0.smithi076.stdout:3680: done (6 left) 2021-03-15T22:35:22.507 INFO:tasks.rados.rados.0.smithi076.stdout:3681: done (5 left) 2021-03-15T22:35:22.507 INFO:tasks.rados.rados.0.smithi076.stdout:3684: done (4 left) 2021-03-15T22:35:22.508 INFO:tasks.rados.rados.0.smithi076.stdout:3685: done (3 left) 2021-03-15T22:35:22.508 INFO:tasks.rados.rados.0.smithi076.stdout:3686: done (2 left) 2021-03-15T22:35:22.508 INFO:tasks.rados.rados.0.smithi076.stdout:3687: done (1 left) 2021-03-15T22:35:22.508 INFO:tasks.rados.rados.0.smithi076.stdout:3689: done (0 left) 2021-03-15T22:35:22.509 INFO:tasks.rados.rados.0.smithi076.stdout:3690: snap_create 2021-03-15T22:35:22.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:22.575 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:22.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:22.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:22.678 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:22.702 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:22.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:22.704 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:22.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[56514]: cluster 2021-03-15T22:35:20.883984+0000 osd.5 (osd.5) 54 : cluster [DBG] 5.16 scrub ok 2021-03-15T22:35:22.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[56514]: cluster 2021-03-15T22:35:21.054185+0000 osd.1 (osd.1) 184 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:35:22.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[56514]: cluster 2021-03-15T22:35:21.218623+0000 mgr.y (mgr.14144) 1074 : cluster [DBG] pgmap v1595: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 156 active+clean; 670 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 5.0 MiB/s wr, 27 op/s; 6.4 MiB/s, 3 objects/s recovering 2021-03-15T22:35:22.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[56514]: cluster 2021-03-15T22:35:21.314759+0000 osd.0 (osd.0) 155 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:35:22.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[56514]: cluster 2021-03-15T22:35:21.384030+0000 osd.4 (osd.4) 194 : cluster [DBG] 4.e scrub ok 2021-03-15T22:35:22.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[56514]: cluster 2021-03-15T22:35:21.487286+0000 2021-03-15T22:35:22.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[56514]: mon.a (mon.0) 2414 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/1905 objects degraded (0.315%), 1 pg degraded) 2021-03-15T22:35:22.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[62156]: cluster 2021-03-15T22:35:20.883984+0000 osd.5 (osd.5) 54 : cluster [DBG] 5.16 scrub ok 2021-03-15T22:35:22.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[62156]: cluster 2021-03-15T22:35:21.054185+0000 osd.1 (osd.1) 184 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:35:22.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[62156]: cluster 2021-03-15T22:35:21.218623+0000 mgr.y (mgr.14144) 1074 : cluster [DBG] pgmap v1595: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 156 active+clean; 670 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 5.0 MiB/s wr, 27 op/s; 6.4 MiB/s, 3 objects/s recovering 2021-03-15T22:35:22.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[62156]: cluster 2021-03-15T22:35:21.314759+0000 osd.0 (osd.0) 155 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:35:22.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[62156]: cluster 2021-03-15T22:35:21.384030+0000 osd.4 (osd.4) 2021-03-15T22:35:22.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[62156]: 194 : cluster [DBG] 4.e scrub ok 2021-03-15T22:35:22.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:22 smithi076 conmon[62156]: cluster 2021-03-15T22:35:21.487286+0000 mon.a (mon.0) 2414 : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/1905 objects degraded (0.315%), 1 pg degraded) 2021-03-15T22:35:22.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: cluster 2021-03-15T22:35:20.883984+0000 osd.5 (osd.5) 54 : cluster [DBG] 5.16 scrub ok 2021-03-15T22:35:22.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: cluster 2021-03-15T22:35:21.054185+0000 osd.1 (osd.1) 184 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:35:22.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:35:22.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: :35:21.218623+0000 mgr.y (mgr.14144) 1074 : cluster [DBG] pgmap v1595: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 156 active+clean; 670 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 22 MiB/s rd, 5.0 MiB/s wr, 27 op/s; 6.4 MiB/s, 3 objects/s recovering 2021-03-15T22:35:22.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:35:22.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: 15T22:35:21.314759+0000 osd.0 ( 2021-03-15T22:35:22.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: osd.0) 155 : cluster [DBG] 4.3 scrub ok 2021-03-15T22:35:22.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:35:22.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: -03-15T22:35:21.384030+0000 2021-03-15T22:35:22.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: osd.4 (osd.4) 194 : cluster [DBG] 2021-03-15T22:35:22.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: 4.e scrub ok 2021-03-15T22:35:22.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:35:22.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: 35:21.487286+0000 mon.a (mon.0) 2414 2021-03-15T22:35:22.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:22 smithi125 conmon[56739]: : cluster [INF] Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 6/1905 objects degraded (0.315%), 1 pg degraded) 2021-03-15T22:35:22.805 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:22.826 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:22.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:22.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:22.873 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:35:22.929 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:22.955 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:22.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:22.956 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:23.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:23.082 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:23.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:23.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:23.184 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:23.209 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:23.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:23.211 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:23.312 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:23.334 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:23.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:23.336 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:23.437 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:23.447 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:23.460 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:23.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:23.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:23.516 INFO:tasks.rados.rados.0.smithi076.stdout:3690: done (0 left) 2021-03-15T22:35:23.517 INFO:tasks.rados.rados.0.smithi076.stdout:3691: snap_remove snap 418 2021-03-15T22:35:23.520 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:35:23.562 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:23.573 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[56514]: cluster 2021-03-15T22:35:21 2021-03-15T22:35:23.574 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[56514]: .917811+0000 osd.5 (osd.5) 55 : cluster [DBG] 5.f scrub ok 2021-03-15T22:35:23.574 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[56514]: cluster 2021-03-15T22:35:22.291397+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.17 deep-scrub ok 2021-03-15T22:35:23.574 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[56514]: cluster 2021-03-15T22:35:22.408958+0000 osd.7 (osd.7) 180 : cluster [DBG] 5.4 scrub ok 2021-03-15T22:35:23.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[56514]: cluster 2021-03-15T22:35:22.500444+0000 mon.a (mon.0) 2415 : cluster [DBG] osdmap e967: 8 total, 8 up, 8 in 2021-03-15T22:35:23.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[56514]: cluster 2021-03-15T22:35:22.665244+0000 osd.6 (osd.6) 143 : cluster [DBG] 5.11 scrub ok 2021-03-15T22:35:23.575 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[56514]: audit 2021-03-15T22:35:23.303056+0000 mon.a (mon.0) 2416 : audit [INF] from='client.? 172.21.15.76:0/1737589649' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:35:23.576 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: cluster 2021-03-15T22:35:21.917811+0000 osd.5 ( 2021-03-15T22:35:23.576 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: osd.5) 55 : cluster [DBG] 5.f scrub ok 2021-03-15T22:35:23.576 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: cluster 2021-03-15T22:35: 2021-03-15T22:35:23.577 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: 22.291397+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.17 deep-scrub ok 2021-03-15T22:35:23.577 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:35:23.578 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: -15T22:35:22.408958+0000 osd.7 2021-03-15T22:35:23.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: (osd.7) 180 : cluster [DBG] 5.4 scrub ok 2021-03-15T22:35:23.579 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:35:23.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: 03-15T22:35:22.500444+0000 mon.a ( 2021-03-15T22:35:23.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: mon.0) 2415 : cluster [DBG] osdmap e967: 8 total, 8 up, 8 in 2021-03-15T22:35:23.580 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: cluster 2021-03-15T22:35:23.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: 2021-03-15T22:35:22.665244 2021-03-15T22:35:23.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: +0000 osd.6 (osd.6) 143 : cluster [DBG] 2021-03-15T22:35:23.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: 5.11 scrub ok 2021-03-15T22:35:23.581 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: audit 2021-03-15T22:35 2021-03-15T22:35:23.582 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: :23.303056+0000 mon.a (mon.0) 2416 2021-03-15T22:35:23.582 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:23 smithi076 conmon[62156]: : audit [INF] from='client.? 172.21.15.76:0/1737589649' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:35:23.589 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:23.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:23.590 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:23.691 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:23.716 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:23.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:23.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:23.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: cluster 2021-03-15T22:35:21.917811+0000 osd.5 (osd.5) 55 : 2021-03-15T22:35:23.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: cluster [DBG] 5.f scrub ok 2021-03-15T22:35:23.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: cluster 2021-03-15T22:35:22.291397+0000 osd.0 (osd.0) 156 : cluster [DBG] 3.17 deep-scrub ok 2021-03-15T22:35:23.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: cluster 2021-03-15T22:35:22.408958+0000 2021-03-15T22:35:23.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: osd.7 (osd.7) 180 : cluster [DBG] 5.4 scrub ok 2021-03-15T22:35:23.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: cluster 2021-03-15T22:35:23.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: 2021-03-15T22:35: 2021-03-15T22:35:23.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: 22.500444+0000 mon.a (mon. 2021-03-15T22:35:23.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: 0) 2415 : cluster [DBG] osdmap e967: 8 total, 8 up, 8 in 2021-03-15T22:35:23.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: 2021-03-15T22:35:23.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: cluster 2021-03-15T22:35:23.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: 2021-03-15T 2021-03-15T22:35:23.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: 22:35:22.665244+0000 2021-03-15T22:35:23.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: osd.6 (osd.6) 143 : cluster 2021-03-15T22:35:23.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: [DBG] 5.11 scrub ok 2021-03-15T22:35:23.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: audit 2021 2021-03-15T22:35:23.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: -03-15T22:35:23. 2021-03-15T22:35:23.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: 303056+0000 mon.a (mon.0) 2021-03-15T22:35:23.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:23 smithi125 conmon[56739]: 2416 : audit [INF] from='client.? 172.21.15.76:0/1737589649' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:35:23.818 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:23.840 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:23.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:23.842 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:23.943 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:23.965 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:23.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:23.966 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:24.067 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:24.093 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:24.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:24.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:24.196 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:24.217 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:24.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:24.219 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:24.242 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:35:24.242 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 20, 2021-03-15T22:35:24.242 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 600, 2021-03-15T22:35:24.243 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [ 2021-03-15T22:35:24.243 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.243 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3982 7.3 7:c458c800:::smithi07694878-3:head [copy-from ver 119] snapc 13f=[13f,13e,13b,13a,138,135,134,133,131,130,12f,12e,128,126,123,122,121,118,117,114,10e,10d,10c,102,100,e7] ondisk+write+known_if_redirected e421)", 2021-03-15T22:35:24.243 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:36.115531+0000", 2021-03-15T22:35:24.243 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 588.04766866499995, 2021-03-15T22:35:24.244 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0139079740000001, 2021-03-15T22:35:24.244 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.244 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:24.244 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.245 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.245 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.245 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3982 2021-03-15T22:35:24.245 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.246 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.246 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.246 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.246 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115531+0000", 2021-03-15T22:35:24.247 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.247 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.247 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.247 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.248 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115531+0000", 2021-03-15T22:35:24.248 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4710000000000001e-06 2021-03-15T22:35:24.248 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.248 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.248 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.249 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115536+0000", 2021-03-15T22:35:24.249 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.5420000000000001e-06 2021-03-15T22:35:24.249 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.249 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.249 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.250 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115539+0000", 2021-03-15T22:35:24.250 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3487e-05 2021-03-15T22:35:24.250 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.250 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.251 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.251 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115553+0000", 2021-03-15T22:35:24.251 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4500000000000004e-06 2021-03-15T22:35:24.251 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.251 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.252 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.252 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115561+0000", 2021-03-15T22:35:24.252 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0590000000000001e-05 2021-03-15T22:35:24.252 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.253 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.253 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.254 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115622+0000", 2021-03-15T22:35:24.254 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.3449999999999995e-06 2021-03-15T22:35:24.254 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.254 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.254 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op must wait for map", 2021-03-15T22:35:24.255 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115631+0000", 2021-03-15T22:35:24.255 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00089869400000000001 2021-03-15T22:35:24.255 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.255 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.256 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.256 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.116530+0000", 2021-03-15T22:35:24.256 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8217000000000001e-05 2021-03-15T22:35:24.256 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.256 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.257 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.257 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.116598+0000", 2021-03-15T22:35:24.257 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.00515501 2021-03-15T22:35:24.258 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.258 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.258 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,4", 2021-03-15T22:35:24.258 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.121753+0000", 2021-03-15T22:35:24.258 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0073058109999999997 2021-03-15T22:35:24.259 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.259 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.259 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:24.260 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129059+0000", 2021-03-15T22:35:24.260 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1652e-05 2021-03-15T22:35:24.260 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.260 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.260 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:35:24.261 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129071+0000", 2021-03-15T22:35:24.261 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025268099999999999 2021-03-15T22:35:24.261 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.261 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.262 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:24.262 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129323+0000", 2021-03-15T22:35:24.262 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5415999999999999e-05 2021-03-15T22:35:24.262 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.263 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.263 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:24.263 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129379+0000", 2021-03-15T22:35:24.263 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0608e-05 2021-03-15T22:35:24.264 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.264 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.264 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.264 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129439+0000", 2021-03-15T22:35:24.265 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0139079740000001 2021-03-15T22:35:24.265 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.265 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.265 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.266 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.266 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.266 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:3983 7.3 7:c458c800:::smithi07694878-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e421)", 2021-03-15T22:35:24.266 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:25:36.115576+0000", 2021-03-15T22:35:24.266 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 588.04762436299995, 2021-03-15T22:35:24.267 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0140553489999999, 2021-03-15T22:35:24.267 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.267 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:35:24.267 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.268 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.268 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.268 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 3983 2021-03-15T22:35:24.268 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.268 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.269 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.269 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.269 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115576+0000", 2021-03-15T22:35:24.269 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.270 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.270 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.270 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.270 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115576+0000", 2021-03-15T22:35:24.271 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2550000000000001e-06 2021-03-15T22:35:24.271 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.271 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.271 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.271 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115577+0000", 2021-03-15T22:35:24.272 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8390000000000001e-06 2021-03-15T22:35:24.272 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.272 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.272 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.273 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115579+0000", 2021-03-15T22:35:24.273 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0740000000000004e-06 2021-03-15T22:35:24.273 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.273 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.273 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.274 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115586+0000", 2021-03-15T22:35:24.274 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.3220000000000003e-06 2021-03-15T22:35:24.274 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.274 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.274 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.275 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115590+0000", 2021-03-15T22:35:24.275 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000108726 2021-03-15T22:35:24.275 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.275 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.276 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.276 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115699+0000", 2021-03-15T22:35:24.276 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.9349999999999997e-06 2021-03-15T22:35:24.276 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.276 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.277 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:35:24.277 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.115706+0000", 2021-03-15T22:35:24.277 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0021999540000000001 2021-03-15T22:35:24.277 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.277 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.278 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.278 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.117906+0000", 2021-03-15T22:35:24.278 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6322e-05 2021-03-15T22:35:24.278 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.279 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.279 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:35:24.279 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:36.117962+0000", 2021-03-15T22:35:24.279 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.00844165 2021-03-15T22:35:24.279 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.280 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.280 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.280 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.126404+0000", 2021-03-15T22:35:24.280 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5353999999999996e-05 2021-03-15T22:35:24.281 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.281 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.281 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:24.281 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.126489+0000", 2021-03-15T22:35:24.281 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0029835399999999998 2021-03-15T22:35:24.282 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.282 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.282 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.282 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129473+0000", 2021-03-15T22:35:24.283 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5050999999999997e-05 2021-03-15T22:35:24.283 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.283 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.283 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.283 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129548+0000", 2021-03-15T22:35:24.284 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3326999999999997e-05 2021-03-15T22:35:24.284 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.284 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.284 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.284 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:25:37.129631+0000", 2021-03-15T22:35:24.285 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0140553489999999 2021-03-15T22:35:24.285 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.285 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.285 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.286 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.286 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.286 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop(client.24517.0:4380 7.6 e455/421 7:6283974b:::smithi07694878-30:head v 455'1076, mlcod=454'1072)", 2021-03-15T22:35:24.286 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:26:10.132423+0000", 2021-03-15T22:35:24.286 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 554.03077683399999, 2021-03-15T22:35:24.287 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.120130079, 2021-03-15T22:35:24.287 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.287 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:24.287 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.288 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.288 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.288 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.132423+0000", 2021-03-15T22:35:24.288 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.288 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.289 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.289 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.289 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.132423+0000", 2021-03-15T22:35:24.290 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0010757970000000001 2021-03-15T22:35:24.290 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.290 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.290 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.291 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.133499+0000", 2021-03-15T22:35:24.291 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5153e-05 2021-03-15T22:35:24.291 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.291 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.291 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.292 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.133514+0000", 2021-03-15T22:35:24.292 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.070374909999999999 2021-03-15T22:35:24.292 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.292 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.293 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.293 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.203889+0000", 2021-03-15T22:35:24.293 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.6134000000000001e-05 2021-03-15T22:35:24.293 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.294 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.294 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.294 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.203915+0000", 2021-03-15T22:35:24.294 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.8656000000000001e-05 2021-03-15T22:35:24.295 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.295 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.295 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.295 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.203984+0000", 2021-03-15T22:35:24.295 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.5831999999999999e-05 2021-03-15T22:35:24.296 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.296 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.296 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.296 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.204010+0000", 2021-03-15T22:35:24.297 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.048499043999999998 2021-03-15T22:35:24.297 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.297 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.297 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:24.297 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.252509+0000", 2021-03-15T22:35:24.298 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4552999999999998e-05 2021-03-15T22:35:24.298 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.298 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.298 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.299 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:26:10.252553+0000", 2021-03-15T22:35:24.299 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.120130079 2021-03-15T22:35:24.299 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.299 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.299 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.300 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.300 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.300 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5267 7.a 7.68cfe3da (undecoded) ondisk+write+known_if_redirected e540)", 2021-03-15T22:35:24.300 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.213130+0000", 2021-03-15T22:35:24.301 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 468.95006980800002, 2021-03-15T22:35:24.301 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99493337699999995, 2021-03-15T22:35:24.301 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.301 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:35:24.301 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.302 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.302 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.302 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5267 2021-03-15T22:35:24.302 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.303 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.303 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.303 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.303 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213130+0000", 2021-03-15T22:35:24.303 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.304 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.304 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.304 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.304 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213130+0000", 2021-03-15T22:35:24.304 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027025900000000002 2021-03-15T22:35:24.305 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.305 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.305 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.305 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213400+0000", 2021-03-15T22:35:24.306 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.7129999999999999e-06 2021-03-15T22:35:24.306 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.307 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.307 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.308 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213404+0000", 2021-03-15T22:35:24.308 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3495e-05 2021-03-15T22:35:24.309 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.309 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.310 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.310 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213418+0000", 2021-03-15T22:35:24.311 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5329999999999997e-06 2021-03-15T22:35:24.311 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.311 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.312 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.312 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213425+0000", 2021-03-15T22:35:24.313 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5578999999999998e-05 2021-03-15T22:35:24.313 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.313 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.314 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.314 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213471+0000", 2021-03-15T22:35:24.315 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0062e-05 2021-03-15T22:35:24.315 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.316 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.316 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.317 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.213481+0000", 2021-03-15T22:35:24.317 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99456081500000004 2021-03-15T22:35:24.318 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.318 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.318 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.319 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208042+0000", 2021-03-15T22:35:24.319 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1920999999999999e-05 2021-03-15T22:35:24.319 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.319 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.319 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.320 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208064+0000", 2021-03-15T22:35:24.320 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.99493337699999995 2021-03-15T22:35:24.320 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.320 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.321 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.321 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.321 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.321 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5268 7.a 7.68cfe3da (undecoded) ondisk+write+known_if_redirected e540)", 2021-03-15T22:35:24.321 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.219467+0000", 2021-03-15T22:35:24.322 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 468.943733323, 2021-03-15T22:35:24.322 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98862604499999995, 2021-03-15T22:35:24.322 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.323 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:35:24.323 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.323 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.323 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.324 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5268 2021-03-15T22:35:24.324 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.324 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.324 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.324 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.325 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219467+0000", 2021-03-15T22:35:24.325 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.325 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.325 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.326 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.326 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219467+0000", 2021-03-15T22:35:24.326 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00025054199999999997 2021-03-15T22:35:24.326 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.326 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.327 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.327 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219717+0000", 2021-03-15T22:35:24.328 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6829999999999999e-06 2021-03-15T22:35:24.328 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.329 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.329 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.329 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219722+0000", 2021-03-15T22:35:24.329 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1461e-05 2021-03-15T22:35:24.330 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.330 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.330 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.331 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219743+0000", 2021-03-15T22:35:24.331 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4905e-05 2021-03-15T22:35:24.331 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.331 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.331 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.332 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219758+0000", 2021-03-15T22:35:24.332 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.6631000000000006e-05 2021-03-15T22:35:24.333 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.333 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.333 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.333 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219825+0000", 2021-03-15T22:35:24.333 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1799e-05 2021-03-15T22:35:24.334 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.334 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.334 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.334 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219837+0000", 2021-03-15T22:35:24.335 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98824981899999997 2021-03-15T22:35:24.335 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.335 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.335 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.335 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208087+0000", 2021-03-15T22:35:24.336 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.2049999999999999e-06 2021-03-15T22:35:24.336 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.336 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.336 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.337 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208093+0000", 2021-03-15T22:35:24.337 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98862604499999995 2021-03-15T22:35:24.337 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.337 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.337 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.338 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.338 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.338 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5269 7.a 7.68cfe3da (undecoded) ondisk+write+known_if_redirected e540)", 2021-03-15T22:35:24.338 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.219771+0000", 2021-03-15T22:35:24.339 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 468.94342879499999, 2021-03-15T22:35:24.339 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98834424600000004, 2021-03-15T22:35:24.339 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.340 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:35:24.340 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.341 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.341 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.342 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5269 2021-03-15T22:35:24.342 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.342 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.343 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.343 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.343 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219771+0000", 2021-03-15T22:35:24.343 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.344 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.344 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.345 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.345 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219771+0000", 2021-03-15T22:35:24.345 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3630000000000001e-06 2021-03-15T22:35:24.346 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.346 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.346 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.347 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219774+0000", 2021-03-15T22:35:24.347 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2139999999999999e-06 2021-03-15T22:35:24.347 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.347 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.348 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.348 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219775+0000", 2021-03-15T22:35:24.348 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5919000000000001e-05 2021-03-15T22:35:24.348 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.348 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.349 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.349 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219791+0000", 2021-03-15T22:35:24.349 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0747999999999999e-05 2021-03-15T22:35:24.349 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.350 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.350 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.350 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219801+0000", 2021-03-15T22:35:24.350 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4376999999999996e-05 2021-03-15T22:35:24.350 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.351 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.351 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.351 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219876+0000", 2021-03-15T22:35:24.352 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.585e-06 2021-03-15T22:35:24.352 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.352 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.352 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.353 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219884+0000", 2021-03-15T22:35:24.353 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98822621899999996 2021-03-15T22:35:24.353 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.353 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.353 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.354 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208111+0000", 2021-03-15T22:35:24.354 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8210000000000001e-06 2021-03-15T22:35:24.354 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.354 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.355 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.355 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208115+0000", 2021-03-15T22:35:24.355 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98834424600000004 2021-03-15T22:35:24.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.356 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.356 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.356 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.356 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.357 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:5270 7.a 7.68cfe3da (undecoded) ondisk+read+rwordered+known_if_redirected e540)", 2021-03-15T22:35:24.357 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:27:35.219812+0000", 2021-03-15T22:35:24.357 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 468.94338768699998, 2021-03-15T22:35:24.358 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98832836899999998, 2021-03-15T22:35:24.358 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.358 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:35:24.358 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.359 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.359 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.359 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 5270 2021-03-15T22:35:24.359 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.359 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.360 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.360 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.360 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219812+0000", 2021-03-15T22:35:24.360 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.361 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.361 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.361 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.361 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219812+0000", 2021-03-15T22:35:24.361 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.88e-06 2021-03-15T22:35:24.362 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.362 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.362 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.362 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219814+0000", 2021-03-15T22:35:24.363 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.2960000000000001e-06 2021-03-15T22:35:24.363 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.363 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.363 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.364 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219816+0000", 2021-03-15T22:35:24.364 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0669999999999998e-06 2021-03-15T22:35:24.364 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.364 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.364 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.365 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219826+0000", 2021-03-15T22:35:24.365 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.014e-06 2021-03-15T22:35:24.365 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.365 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.366 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.366 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219830+0000", 2021-03-15T22:35:24.366 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.8069e-05 2021-03-15T22:35:24.366 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.367 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.367 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.367 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219918+0000", 2021-03-15T22:35:24.367 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.6109999999999994e-06 2021-03-15T22:35:24.367 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.368 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.368 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.368 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:35.219926+0000", 2021-03-15T22:35:24.368 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98820947100000001 2021-03-15T22:35:24.369 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.369 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.369 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.369 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208136+0000", 2021-03-15T22:35:24.369 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9609999999999996e-06 2021-03-15T22:35:24.370 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.370 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.370 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.370 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:27:36.208141+0000", 2021-03-15T22:35:24.371 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.98832836899999998 2021-03-15T22:35:24.371 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.371 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.371 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.372 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.372 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.372 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.14427.0:604 4.19 4:9ed7f5b7:::obj_delete_at_hint.0000000004:head [call lock.lock in=8b] snapc 0=[] RETRY=1 ondisk+retry+write+known_if_redirected e550)", 2021-03-15T22:35:24.372 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:28:11.512449+0000", 2021-03-15T22:35:24.372 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 432.65075115899998, 2021-03-15T22:35:24.373 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0160361099999999, 2021-03-15T22:35:24.373 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.373 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:24.373 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.373 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.14427", 2021-03-15T22:35:24.374 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2912955800", 2021-03-15T22:35:24.374 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 604 2021-03-15T22:35:24.374 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.375 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.375 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.375 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.375 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512449+0000", 2021-03-15T22:35:24.376 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.376 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.376 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.376 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.377 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512449+0000", 2021-03-15T22:35:24.377 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8390000000000001e-06 2021-03-15T22:35:24.377 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.377 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.378 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.378 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512451+0000", 2021-03-15T22:35:24.378 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.0460000000000001e-06 2021-03-15T22:35:24.378 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.378 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.379 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.379 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512454+0000", 2021-03-15T22:35:24.379 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3121e-05 2021-03-15T22:35:24.379 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.380 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.380 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.380 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512467+0000", 2021-03-15T22:35:24.380 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.8639999999999996e-06 2021-03-15T22:35:24.380 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.381 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.381 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.381 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512475+0000", 2021-03-15T22:35:24.381 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013658999999999999 2021-03-15T22:35:24.382 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.382 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.382 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.382 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512611+0000", 2021-03-15T22:35:24.383 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0465000000000001e-05 2021-03-15T22:35:24.383 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.383 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.383 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.383 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:11.512622+0000", 2021-03-15T22:35:24.384 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.002971662 2021-03-15T22:35:24.384 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.384 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.384 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.385 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.515593+0000", 2021-03-15T22:35:24.385 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000250323 2021-03-15T22:35:24.385 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.385 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.385 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for degraded object", 2021-03-15T22:35:24.386 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.515844+0000", 2021-03-15T22:35:24.386 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0096873329999999994 2021-03-15T22:35:24.386 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.386 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.387 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.387 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.525531+0000", 2021-03-15T22:35:24.387 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2487000000000001e-05 2021-03-15T22:35:24.387 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.388 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.388 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.388 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.525584+0000", 2021-03-15T22:35:24.388 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000216667 2021-03-15T22:35:24.388 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.389 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.389 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 0", 2021-03-15T22:35:24.389 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.525800+0000", 2021-03-15T22:35:24.389 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00229575 2021-03-15T22:35:24.390 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.390 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.390 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:24.390 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.528096+0000", 2021-03-15T22:35:24.390 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00027043100000000001 2021-03-15T22:35:24.391 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.391 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.391 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:35:24.391 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.528366+0000", 2021-03-15T22:35:24.392 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4045000000000003e-05 2021-03-15T22:35:24.392 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.392 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.392 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:24.392 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.528410+0000", 2021-03-15T22:35:24.393 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.4486999999999993e-05 2021-03-15T22:35:24.393 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.393 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.393 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.394 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:12.528485+0000", 2021-03-15T22:35:24.394 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0160361099999999 2021-03-15T22:35:24.394 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.394 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.395 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.395 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.395 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.395 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "MOSDPGPushReply(7.f 559/553 [PushReplyOp(7:ffef77ba:::smithi07694878-13:1b1)])", 2021-03-15T22:35:24.395 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:28:19.240832+0000", 2021-03-15T22:35:24.396 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 424.922368323, 2021-03-15T22:35:24.396 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.427929538, 2021-03-15T22:35:24.396 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.396 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "reached pg", 2021-03-15T22:35:24.397 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.397 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.397 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.397 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240832+0000", 2021-03-15T22:35:24.397 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.398 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.398 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.398 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.398 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240832+0000", 2021-03-15T22:35:24.399 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.841e-06 2021-03-15T22:35:24.399 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.400 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.400 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.401 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240835+0000", 2021-03-15T22:35:24.401 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6820000000000002e-06 2021-03-15T22:35:24.401 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.401 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.402 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.402 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240839+0000", 2021-03-15T22:35:24.402 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.1081999999999999e-05 2021-03-15T22:35:24.402 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.403 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.403 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.403 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240860+0000", 2021-03-15T22:35:24.403 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5779999999999999e-06 2021-03-15T22:35:24.403 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.404 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.404 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.404 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.240869+0000", 2021-03-15T22:35:24.404 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.426681264 2021-03-15T22:35:24.405 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.405 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.405 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.405 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.667550+0000", 2021-03-15T22:35:24.405 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0012110910000000001 2021-03-15T22:35:24.406 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.406 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.406 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.406 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:28:19.668761+0000", 2021-03-15T22:35:24.407 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.427929538 2021-03-15T22:35:24.407 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.407 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.407 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.408 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.408 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.408 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6450 7.a 7:53dee572:::smithi07694878-35:1f7 [sparse-read 0~1916845,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected e691)", 2021-03-15T22:35:24.408 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:30:40.216790+0000", 2021-03-15T22:35:24.408 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 283.94641008000002, 2021-03-15T22:35:24.409 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095085135000000001, 2021-03-15T22:35:24.409 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.409 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:35:24.409 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.409 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.410 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.410 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6450 2021-03-15T22:35:24.410 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.410 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.411 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.411 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.411 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.216790+0000", 2021-03-15T22:35:24.411 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.412 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.412 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.412 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.412 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.216790+0000", 2021-03-15T22:35:24.412 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.3280000000000002e-06 2021-03-15T22:35:24.413 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.413 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.413 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.413 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.216793+0000", 2021-03-15T22:35:24.414 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.931e-06 2021-03-15T22:35:24.414 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.414 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.414 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.415 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.216796+0000", 2021-03-15T22:35:24.415 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.080014454999999998 2021-03-15T22:35:24.415 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.415 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.416 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.416 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.296811+0000", 2021-03-15T22:35:24.417 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0380000000000001e-05 2021-03-15T22:35:24.417 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.418 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.418 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.419 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.296821+0000", 2021-03-15T22:35:24.419 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00071239899999999995 2021-03-15T22:35:24.420 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.420 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.421 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.421 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297533+0000", 2021-03-15T22:35:24.422 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000119981 2021-03-15T22:35:24.422 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.423 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.423 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.423 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.297653+0000", 2021-03-15T22:35:24.424 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.014221661 2021-03-15T22:35:24.424 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.425 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.425 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.426 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:30:40.311875+0000", 2021-03-15T22:35:24.426 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.095085135000000001 2021-03-15T22:35:24.427 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.427 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.428 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.428 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.428 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.429 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6959 7.c 7:37eb566e:::smithi07694878-48:head [write 733036~563824,stat] snapc 25b=[25b,25a,257,255,24f,24e,24c,24b,24a,248,247,245,243,23e,23d,23c,239,235,234,22d,227,225,218,211,20c,1f7,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e740)", 2021-03-15T22:35:24.429 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.842261+0000", 2021-03-15T22:35:24.430 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 233.32093905400001, 2021-03-15T22:35:24.430 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40759049800000002, 2021-03-15T22:35:24.431 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.431 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:24.432 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.432 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.433 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.433 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6959 2021-03-15T22:35:24.433 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.434 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.434 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.435 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.435 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842261+0000", 2021-03-15T22:35:24.436 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.436 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.436 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.437 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.437 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842261+0000", 2021-03-15T22:35:24.438 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00016927899999999999 2021-03-15T22:35:24.438 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.439 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.439 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.440 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842430+0000", 2021-03-15T22:35:24.440 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.097e-06 2021-03-15T22:35:24.441 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.441 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.441 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.442 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842437+0000", 2021-03-15T22:35:24.442 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2824999999999998e-05 2021-03-15T22:35:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.443 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.444 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842470+0000", 2021-03-15T22:35:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.2836999999999999e-05 2021-03-15T22:35:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.445 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.446 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842483+0000", 2021-03-15T22:35:24.447 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.8919000000000003e-05 2021-03-15T22:35:24.447 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.448 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.448 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.449 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842542+0000", 2021-03-15T22:35:24.449 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1345999999999999e-05 2021-03-15T22:35:24.449 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.450 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.450 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.451 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.842553+0000", 2021-03-15T22:35:24.451 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40354269599999998 2021-03-15T22:35:24.452 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.452 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.453 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.453 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.246096+0000", 2021-03-15T22:35:24.453 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00018614400000000001 2021-03-15T22:35:24.454 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.454 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.455 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.455 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.246282+0000", 2021-03-15T22:35:24.456 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00038109899999999998 2021-03-15T22:35:24.456 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.457 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.457 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:35:24.457 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.246663+0000", 2021-03-15T22:35:24.458 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0024403139999999999 2021-03-15T22:35:24.458 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.459 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.459 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:24.460 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249103+0000", 2021-03-15T22:35:24.460 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9490000000000002e-06 2021-03-15T22:35:24.461 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.461 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.461 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:35:24.462 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249112+0000", 2021-03-15T22:35:24.462 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000559948 2021-03-15T22:35:24.463 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.463 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:24.464 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249672+0000", 2021-03-15T22:35:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0310999999999998e-05 2021-03-15T22:35:24.465 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.466 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249743+0000", 2021-03-15T22:35:24.467 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00010873400000000001 2021-03-15T22:35:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.468 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249851+0000", 2021-03-15T22:35:24.469 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40759049800000002 2021-03-15T22:35:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.470 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.471 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.472 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6960 7.c 7:37eb566e:::smithi07694878-48:head [write 2074812~684986] snapc 25b=[25b,25a,257,255,24f,24e,24c,24b,24a,248,247,245,243,23e,23d,23c,239,235,234,22d,227,225,218,211,20c,1f7,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e740)", 2021-03-15T22:35:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.850305+0000", 2021-03-15T22:35:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 233.31289543099999, 2021-03-15T22:35:24.473 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40674566499999998, 2021-03-15T22:35:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.474 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.475 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.476 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6960 2021-03-15T22:35:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.477 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.478 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850305+0000", 2021-03-15T22:35:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.479 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850305+0000", 2021-03-15T22:35:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013730900000000001 2021-03-15T22:35:24.480 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850442+0000", 2021-03-15T22:35:24.481 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0299999999999999e-06 2021-03-15T22:35:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850448+0000", 2021-03-15T22:35:24.482 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.3971000000000001e-05 2021-03-15T22:35:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.483 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850472+0000", 2021-03-15T22:35:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.0040000000000005e-06 2021-03-15T22:35:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.484 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850481+0000", 2021-03-15T22:35:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6959000000000002e-05 2021-03-15T22:35:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.485 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850528+0000", 2021-03-15T22:35:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.0190000000000006e-06 2021-03-15T22:35:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.486 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.850536+0000", 2021-03-15T22:35:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.39859939 2021-03-15T22:35:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.487 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249135+0000", 2021-03-15T22:35:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.5754999999999994e-05 2021-03-15T22:35:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.488 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249231+0000", 2021-03-15T22:35:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00067494899999999999 2021-03-15T22:35:24.489 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249906+0000", 2021-03-15T22:35:24.490 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7687000000000003e-05 2021-03-15T22:35:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249984+0000", 2021-03-15T22:35:24.491 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00015096900000000001 2021-03-15T22:35:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:35:24.492 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.250135+0000", 2021-03-15T22:35:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0038623450000000001 2021-03-15T22:35:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.493 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:35:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.253997+0000", 2021-03-15T22:35:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.6215000000000003e-05 2021-03-15T22:35:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.494 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254053+0000", 2021-03-15T22:35:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0027848769999999998 2021-03-15T22:35:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.495 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.256838+0000", 2021-03-15T22:35:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000138264 2021-03-15T22:35:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.496 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.497 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:24.497 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.256976+0000", 2021-03-15T22:35:24.497 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.3922000000000001e-05 2021-03-15T22:35:24.497 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.498 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.498 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.498 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257050+0000", 2021-03-15T22:35:24.498 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40674566499999998 2021-03-15T22:35:24.498 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.499 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.499 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.499 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.499 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.500 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6961 7.c 7:37eb566e:::smithi07694878-48:head [write 3183703~749333] snapc 25b=[25b,25a,257,255,24f,24e,24c,24b,24a,248,247,245,243,23e,23d,23c,239,235,234,22d,227,225,218,211,20c,1f7,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e740)", 2021-03-15T22:35:24.500 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.856636+0000", 2021-03-15T22:35:24.500 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 233.30656426499999, 2021-03-15T22:35:24.500 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40065102200000002, 2021-03-15T22:35:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.501 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.502 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6961 2021-03-15T22:35:24.502 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.502 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.502 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.503 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.503 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856636+0000", 2021-03-15T22:35:24.503 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.503 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.503 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.504 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.504 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856636+0000", 2021-03-15T22:35:24.504 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00028634199999999998 2021-03-15T22:35:24.504 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.505 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.505 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.505 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856922+0000", 2021-03-15T22:35:24.505 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.4699999999999998e-06 2021-03-15T22:35:24.505 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.506 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.506 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.506 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856926+0000", 2021-03-15T22:35:24.506 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.3896999999999999e-05 2021-03-15T22:35:24.507 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.507 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.507 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.507 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856939+0000", 2021-03-15T22:35:24.507 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.3809999999999998e-06 2021-03-15T22:35:24.508 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.508 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.508 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.508 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856945+0000", 2021-03-15T22:35:24.509 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.0936999999999997e-05 2021-03-15T22:35:24.509 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.509 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.509 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.509 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856996+0000", 2021-03-15T22:35:24.510 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1702000000000001e-05 2021-03-15T22:35:24.510 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.510 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.510 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.511 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.857007+0000", 2021-03-15T22:35:24.511 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.392417246 2021-03-15T22:35:24.511 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.511 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.511 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.512 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249425+0000", 2021-03-15T22:35:24.512 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.5692999999999994e-05 2021-03-15T22:35:24.512 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.512 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.513 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:24.513 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249510+0000", 2021-03-15T22:35:24.513 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0020279209999999998 2021-03-15T22:35:24.513 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.513 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.514 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.514 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.251538+0000", 2021-03-15T22:35:24.514 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00012340800000000001 2021-03-15T22:35:24.514 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.251662+0000", 2021-03-15T22:35:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00022522300000000001 2021-03-15T22:35:24.515 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.516 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.516 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:35:24.516 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.251887+0000", 2021-03-15T22:35:24.516 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0052293959999999999 2021-03-15T22:35:24.516 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.517 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.517 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:24.517 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257116+0000", 2021-03-15T22:35:24.517 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.0851e-05 2021-03-15T22:35:24.518 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.518 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.518 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:35:24.518 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257127+0000", 2021-03-15T22:35:24.518 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8776000000000002e-05 2021-03-15T22:35:24.519 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.519 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.519 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:24.519 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257176+0000", 2021-03-15T22:35:24.520 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.2856999999999999e-05 2021-03-15T22:35:24.520 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.520 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.520 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:24.520 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257229+0000", 2021-03-15T22:35:24.521 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.7921999999999999e-05 2021-03-15T22:35:24.521 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.521 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.521 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.522 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257287+0000", 2021-03-15T22:35:24.522 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40065102200000002 2021-03-15T22:35:24.522 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.522 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.522 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.523 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.523 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.523 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6962 7.c 7:37eb566e:::smithi07694878-48:head [setxattr _header (58) in=7b,truncate 3933036] snapc 25b=[25b,25a,257,255,24f,24e,24c,24b,24a,248,247,245,243,23e,23d,23c,239,235,234,22d,227,225,218,211,20c,1f7,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e740)", 2021-03-15T22:35:24.523 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.856954+0000", 2021-03-15T22:35:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 233.30624643499999, 2021-03-15T22:35:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40052243700000001, 2021-03-15T22:35:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:24.524 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6962 2021-03-15T22:35:24.525 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856954+0000", 2021-03-15T22:35:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.527 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856954+0000", 2021-03-15T22:35:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.5260000000000001e-06 2021-03-15T22:35:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.528 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856955+0000", 2021-03-15T22:35:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7999999999999993e-07 2021-03-15T22:35:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.529 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856956+0000", 2021-03-15T22:35:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.5719999999999996e-06 2021-03-15T22:35:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.530 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856961+0000", 2021-03-15T22:35:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2320000000000001e-06 2021-03-15T22:35:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.531 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856964+0000", 2021-03-15T22:35:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.6818999999999998e-05 2021-03-15T22:35:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.532 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.533 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.533 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.857041+0000", 2021-03-15T22:35:24.533 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5550000000000002e-06 2021-03-15T22:35:24.533 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.857046+0000", 2021-03-15T22:35:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.39249579499999998 2021-03-15T22:35:24.534 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.535 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.535 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.535 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249542+0000", 2021-03-15T22:35:24.535 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.3127000000000006e-05 2021-03-15T22:35:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.249625+0000", 2021-03-15T22:35:24.536 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035889419999999999 2021-03-15T22:35:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.537 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.253214+0000", 2021-03-15T22:35:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9442000000000002e-05 2021-03-15T22:35:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.538 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.253304+0000", 2021-03-15T22:35:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00013094199999999999 2021-03-15T22:35:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.539 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for subops from 2,7", 2021-03-15T22:35:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.253434+0000", 2021-03-15T22:35:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0036997000000000002 2021-03-15T22:35:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.540 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "op_commit", 2021-03-15T22:35:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257134+0000", 2021-03-15T22:35:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00019671599999999999 2021-03-15T22:35:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.541 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257331+0000", 2021-03-15T22:35:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2697999999999998e-05 2021-03-15T22:35:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.542 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.543 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:24.543 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257364+0000", 2021-03-15T22:35:24.543 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.1310000000000002e-05 2021-03-15T22:35:24.543 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "commit_sent", 2021-03-15T22:35:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257415+0000", 2021-03-15T22:35:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.1080999999999993e-05 2021-03-15T22:35:24.544 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257476+0000", 2021-03-15T22:35:24.545 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.40052243700000001 2021-03-15T22:35:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.546 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6963 7.c 7:37eb566e:::smithi07694878-48:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e740)", 2021-03-15T22:35:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.856985+0000", 2021-03-15T22:35:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 233.306214598, 2021-03-15T22:35:24.547 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.400735811, 2021-03-15T22:35:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:35:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.548 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6963 2021-03-15T22:35:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.549 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856985+0000", 2021-03-15T22:35:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.550 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856985+0000", 2021-03-15T22:35:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.2699999999999998e-07 2021-03-15T22:35:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.551 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856986+0000", 2021-03-15T22:35:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.023e-06 2021-03-15T22:35:24.552 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856987+0000", 2021-03-15T22:35:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 6.0000000000000002e-06 2021-03-15T22:35:24.553 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856993+0000", 2021-03-15T22:35:24.554 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.694e-06 2021-03-15T22:35:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.856996+0000", 2021-03-15T22:35:24.555 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.1422999999999997e-05 2021-03-15T22:35:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.857067+0000", 2021-03-15T22:35:24.556 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4980000000000006e-06 2021-03-15T22:35:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.557 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.857076+0000", 2021-03-15T22:35:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.39701107000000002 2021-03-15T22:35:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.558 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254087+0000", 2021-03-15T22:35:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000118583 2021-03-15T22:35:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.559 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.254206+0000", 2021-03-15T22:35:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.003334371 2021-03-15T22:35:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.560 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257540+0000", 2021-03-15T22:35:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.4062999999999998e-05 2021-03-15T22:35:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.561 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257624+0000", 2021-03-15T22:35:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.7158999999999996e-05 2021-03-15T22:35:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.562 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257721+0000", 2021-03-15T22:35:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.400735811 2021-03-15T22:35:24.563 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.564 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6969 7.3 7:c7825e8f:::smithi07694878-26:head [copy-from ver 0] snapc 25b=[25b,25a,257,255,24f,24e,24c,24b,24a,248,247,245,243,23e,23d,23c,239,235,234,22d,227,225,218,211,20c,1f7,1e6,1e4,1e0,1c7,1a1,17a,176] ondisk+write+known_if_redirected e740)", 2021-03-15T22:35:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.979796+0000", 2021-03-15T22:35:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 233.18340434500001, 2021-03-15T22:35:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.28205391499999999, 2021-03-15T22:35:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.565 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:35:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.566 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6969 2021-03-15T22:35:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.567 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979796+0000", 2021-03-15T22:35:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.568 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979796+0000", 2021-03-15T22:35:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6020000000000002e-06 2021-03-15T22:35:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.569 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979800+0000", 2021-03-15T22:35:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 3.2289999999999999e-06 2021-03-15T22:35:24.570 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979803+0000", 2021-03-15T22:35:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1990999999999999e-05 2021-03-15T22:35:24.571 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979815+0000", 2021-03-15T22:35:24.572 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.0029999999999997e-06 2021-03-15T22:35:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979822+0000", 2021-03-15T22:35:24.573 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6368000000000001e-05 2021-03-15T22:35:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.574 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979869+0000", 2021-03-15T22:35:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.1099999999999992e-06 2021-03-15T22:35:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.575 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979878+0000", 2021-03-15T22:35:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27729234800000002 2021-03-15T22:35:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.576 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257170+0000", 2021-03-15T22:35:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.000218314 2021-03-15T22:35:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.577 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.257389+0000", 2021-03-15T22:35:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00446095 2021-03-15T22:35:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.578 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.261850+0000", 2021-03-15T22:35:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.28205391499999999 2021-03-15T22:35:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.579 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:6970 7.3 7:c7825e8f:::smithi07694878-26:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e740)", 2021-03-15T22:35:24.580 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:31:30.979836+0000", 2021-03-15T22:35:24.581 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 233.183363625, 2021-03-15T22:35:24.581 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.28649647299999997, 2021-03-15T22:35:24.581 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.581 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:35:24.582 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.582 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.582 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.582 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 6970 2021-03-15T22:35:24.582 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.583 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.583 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.583 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.583 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979836+0000", 2021-03-15T22:35:24.584 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.584 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.584 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.584 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.584 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979836+0000", 2021-03-15T22:35:24.585 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.246e-06 2021-03-15T22:35:24.585 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.585 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.585 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.585 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979838+0000", 2021-03-15T22:35:24.586 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.7039999999999999e-06 2021-03-15T22:35:24.586 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.586 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.586 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.587 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979839+0000", 2021-03-15T22:35:24.587 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 5.5820000000000001e-06 2021-03-15T22:35:24.587 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.587 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.587 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.588 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979845+0000", 2021-03-15T22:35:24.588 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.6890000000000003e-06 2021-03-15T22:35:24.588 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.588 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.589 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.589 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979850+0000", 2021-03-15T22:35:24.589 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9660999999999998e-05 2021-03-15T22:35:24.589 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.589 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.590 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.590 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979899+0000", 2021-03-15T22:35:24.590 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.4100000000000001e-06 2021-03-15T22:35:24.590 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.591 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.591 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for peered", 2021-03-15T22:35:24.591 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:30.979904+0000", 2021-03-15T22:35:24.591 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.27843821899999999 2021-03-15T22:35:24.591 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.592 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.592 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.592 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.258342+0000", 2021-03-15T22:35:24.592 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.5978999999999999e-05 2021-03-15T22:35:24.592 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.593 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.593 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for blocked object", 2021-03-15T22:35:24.593 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.258418+0000", 2021-03-15T22:35:24.593 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0035069020000000001 2021-03-15T22:35:24.594 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.594 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.594 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.594 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.261925+0000", 2021-03-15T22:35:24.595 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2521000000000005e-05 2021-03-15T22:35:24.595 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.595 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.595 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:24.595 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.262007+0000", 2021-03-15T22:35:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0041258969999999999 2021-03-15T22:35:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.597 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.266133+0000", 2021-03-15T22:35:24.597 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9773999999999997e-05 2021-03-15T22:35:24.597 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.597 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.597 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.598 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.266233+0000", 2021-03-15T22:35:24.598 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.9889000000000005e-05 2021-03-15T22:35:24.598 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.598 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.599 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.599 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:31:31.266333+0000", 2021-03-15T22:35:24.599 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.28649647299999997 2021-03-15T22:35:24.599 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.599 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.600 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.600 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.600 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.600 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_op(client.24517.0:7848 7.c 7:3bc1c17e:::smithi07694878-8:head [rollback 2b1] snapc 2b6=[2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,293,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e830)", 2021-03-15T22:35:24.601 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:33:00.405277+0000", 2021-03-15T22:35:24.601 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 143.757922678, 2021-03-15T22:35:24.601 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035514907999999998, 2021-03-15T22:35:24.601 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.601 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:35:24.602 INFO:teuthology.orchestra.run.smithi076.stdout: "client_info": { 2021-03-15T22:35:24.602 INFO:teuthology.orchestra.run.smithi076.stdout: "client": "client.24517", 2021-03-15T22:35:24.602 INFO:teuthology.orchestra.run.smithi076.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:24.602 INFO:teuthology.orchestra.run.smithi076.stdout: "tid": 7848 2021-03-15T22:35:24.603 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.603 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.603 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.603 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.603 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405277+0000", 2021-03-15T22:35:24.604 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.604 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.604 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.604 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.605 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405277+0000", 2021-03-15T22:35:24.605 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.8029999999999996e-06 2021-03-15T22:35:24.605 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.605 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.605 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.606 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405282+0000", 2021-03-15T22:35:24.606 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.4715e-05 2021-03-15T22:35:24.606 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.606 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.607 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.607 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405307+0000", 2021-03-15T22:35:24.607 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 2.9646e-05 2021-03-15T22:35:24.607 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.607 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.608 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.608 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405336+0000", 2021-03-15T22:35:24.608 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.8090000000000004e-06 2021-03-15T22:35:24.608 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.608 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.609 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.609 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405346+0000", 2021-03-15T22:35:24.609 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.2713999999999994e-05 2021-03-15T22:35:24.609 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.609 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.610 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.610 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405429+0000", 2021-03-15T22:35:24.610 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.6675999999999994e-05 2021-03-15T22:35:24.610 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.611 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.611 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.611 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.405526+0000", 2021-03-15T22:35:24.611 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035266545000000003 2021-03-15T22:35:24.612 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.612 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.612 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.612 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:33:00.440792+0000", 2021-03-15T22:35:24.612 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.035514907999999998 2021-03-15T22:35:24.613 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.613 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.613 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.613 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.613 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.614 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.3.0:2446 7.2 e913/897 ondisk, result = 0)", 2021-03-15T22:35:24.614 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:29.601215+0000", 2021-03-15T22:35:24.614 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.561985219999997, 2021-03-15T22:35:24.614 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073306017000000001, 2021-03-15T22:35:24.615 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.615 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:35:24.615 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.615 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.615 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.616 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.601215+0000", 2021-03-15T22:35:24.616 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.616 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.616 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.617 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.617 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.601215+0000", 2021-03-15T22:35:24.617 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.4869999999999999e-06 2021-03-15T22:35:24.617 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.618 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.618 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.618 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.601216+0000", 2021-03-15T22:35:24.618 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9199999999999998e-06 2021-03-15T22:35:24.619 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.619 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.619 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.619 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.601218+0000", 2021-03-15T22:35:24.619 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.072455513999999999 2021-03-15T22:35:24.620 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.620 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.620 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.620 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.673674+0000", 2021-03-15T22:35:24.621 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.746e-06 2021-03-15T22:35:24.621 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.621 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.621 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.621 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.673681+0000", 2021-03-15T22:35:24.622 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.00078196099999999996 2021-03-15T22:35:24.622 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.622 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.622 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.623 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.674463+0000", 2021-03-15T22:35:24.623 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 7.7489999999999998e-06 2021-03-15T22:35:24.623 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.623 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.624 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.624 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.674471+0000", 2021-03-15T22:35:24.624 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 4.9639999999999999e-05 2021-03-15T22:35:24.624 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.624 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.625 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.625 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.674521+0000", 2021-03-15T22:35:24.625 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073306017000000001 2021-03-15T22:35:24.625 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.626 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.626 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.626 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.626 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.626 INFO:teuthology.orchestra.run.smithi076.stdout: "description": "osd_repop_reply(osd.3.0:2446 7.2 e913/897 ondisk, result = 0)", 2021-03-15T22:35:24.627 INFO:teuthology.orchestra.run.smithi076.stdout: "initiated_at": "2021-03-15T22:34:29.601226+0000", 2021-03-15T22:35:24.627 INFO:teuthology.orchestra.run.smithi076.stdout: "age": 54.561973793, 2021-03-15T22:35:24.627 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073218065999999998, 2021-03-15T22:35:24.627 INFO:teuthology.orchestra.run.smithi076.stdout: "type_data": { 2021-03-15T22:35:24.628 INFO:teuthology.orchestra.run.smithi076.stdout: "flag_point": "started", 2021-03-15T22:35:24.628 INFO:teuthology.orchestra.run.smithi076.stdout: "events": [ 2021-03-15T22:35:24.628 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.628 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "initiated", 2021-03-15T22:35:24.628 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.601226+0000", 2021-03-15T22:35:24.629 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0 2021-03-15T22:35:24.629 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.629 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.629 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "throttled", 2021-03-15T22:35:24.629 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.601226+0000", 2021-03-15T22:35:24.630 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.8029999999999999e-06 2021-03-15T22:35:24.630 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.630 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.630 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "header_read", 2021-03-15T22:35:24.631 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.601228+0000", 2021-03-15T22:35:24.631 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.9360000000000002e-06 2021-03-15T22:35:24.631 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.631 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.631 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "all_read", 2021-03-15T22:35:24.632 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.601230+0000", 2021-03-15T22:35:24.632 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.070915927000000004 2021-03-15T22:35:24.632 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.632 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.632 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "dispatched", 2021-03-15T22:35:24.633 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.672146+0000", 2021-03-15T22:35:24.633 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 8.9220000000000003e-06 2021-03-15T22:35:24.633 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.633 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.634 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "queued_for_pg", 2021-03-15T22:35:24.634 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.672155+0000", 2021-03-15T22:35:24.634 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.0022687330000000002 2021-03-15T22:35:24.634 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.634 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.635 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "reached_pg", 2021-03-15T22:35:24.635 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.674424+0000", 2021-03-15T22:35:24.635 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 9.4169999999999998e-06 2021-03-15T22:35:24.635 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.636 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.636 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "started", 2021-03-15T22:35:24.636 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.674433+0000", 2021-03-15T22:35:24.636 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 1.1328000000000001e-05 2021-03-15T22:35:24.636 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:35:24.637 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:35:24.637 INFO:teuthology.orchestra.run.smithi076.stdout: "event": "done", 2021-03-15T22:35:24.637 INFO:teuthology.orchestra.run.smithi076.stdout: "time": "2021-03-15T22:34:29.674444+0000", 2021-03-15T22:35:24.637 INFO:teuthology.orchestra.run.smithi076.stdout: "duration": 0.073218065999999998 2021-03-15T22:35:24.637 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.638 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.638 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.638 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:35:24.638 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:35:24.639 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:35:24.639 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:24.640 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 7, 0, 3, 5, 6, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:35:24.640 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:35:24.640 INFO:tasks.thrashosds.thrasher:Shrinking pool default.rgw.meta 2021-03-15T22:35:24.641 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool default.rgw.meta by 10 min 8 2021-03-15T22:35:24.641 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:35:24.643 INFO:tasks.rados.rados.0.smithi076.stdout:3691: done (0 left) 2021-03-15T22:35:24.643 INFO:tasks.rados.rados.0.smithi076.stdout:3692: read oid 14 snap -1 2021-03-15T22:35:24.644 INFO:tasks.rados.rados.0.smithi076.stdout:3692: expect (ObjNum 1140 snap 393 seq_num 1140) 2021-03-15T22:35:24.644 INFO:tasks.rados.rados.0.smithi076.stdout:3693: snap_create 2021-03-15T22:35:24.644 INFO:tasks.rados.rados.0.smithi076.stdout:3692: expect (ObjNum 1140 snap 393 seq_num 1140) 2021-03-15T22:35:24.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[56514]: cluster 2021-03-15T22:35:23.219063+0000 mgr.y (mgr.14144) 1075 : cluster [DBG] pgmap v1598: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 156 active+clean; 655 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s rd, 5.6 MiB/s wr, 41 op/s; 6.4 MiB/s, 3 objects/s recovering 2021-03-15T22:35:24.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[56514]: cluster 2021-03-15T22:35:23.291503+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.11 scrub ok 2021-03-15T22:35:24.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[56514]: cluster 2021-03-15T22:35:23.428838+0000 osd.7 (osd.7) 181 : cluster [DBG] 3.d scrub ok 2021-03-15T22:35:24.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[56514]: cluster 2021-03-15T22:35:23.498065+0000 mon.a (mon.0) 2417 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:24.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[56514]: audit 2021-03-15T22:35:23.509957+0000 mon.a (mon.0) 2418 : audit [INF] from='client.? 172.21.15.76:0/1737589649' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:35:24.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[56514]: cluster 2021-03-15T22:35:23.510024+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e968: 8 total, 8 up, 8 in 2021-03-15T22:35:24.646 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[62156]: cluster 2021-03-15T22:35:23.219063+0000 mgr.y (mgr.14144) 1075 : cluster [DBG] pgmap v1598: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 156 active+clean; 655 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s rd, 5.6 MiB/s wr, 41 op/s; 6.4 MiB/s, 3 objects/s recovering 2021-03-15T22:35:24.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[62156]: cluster 2021-03-15T22:35:23.291503+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.11 scrub ok 2021-03-15T22:35:24.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[62156]: cluster 2021-03-15T22:35:23.428838+0000 osd.7 (osd.7) 181 : cluster [DBG] 3.d scrub ok 2021-03-15T22:35:24.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[62156]: cluster 2021-03-15T22:35:23.498065+0000 mon.a (mon.0) 2417 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:24.647 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[62156]: audit 2021-03-15T22:35:23.509957+0000 mon.a (mon.0) 2418 : audit [INF] from='client.? 172.21.15.76:0/1737589649' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:35:24.648 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:24 smithi076 conmon[62156]: cluster 2021-03-15T22:35:23.510024+0000 mon.a (mon.0) 2419 : cluster [DBG] osdmap e968: 8 total, 8 up, 8 in 2021-03-15T22:35:24.666 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:24.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:24.668 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:24.768 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:24.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: cluster 2021-03-15T22:35:23.219063+0000 mgr.y (mgr.14144) 1075 : cluster 2021-03-15T22:35:24.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: [DBG] pgmap v1598: 158 pgs: 1 active+remapped+backfill_toofull, 1 peering, 156 active+clean; 655 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 34 MiB/s rd, 5.6 MiB/s wr, 41 op/s; 6.4 MiB/s, 3 objects/s recovering 2021-03-15T22:35:24.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: cluster 2021-03-15T22:35:23.291503+0000 osd.0 (osd.0) 157 : cluster [DBG] 6.11 scrub ok 2021-03-15T22:35:24.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: cluster 2021-03-15T22:35:23.428838+0000 osd.7 (osd.7) 2021-03-15T22:35:24.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: 181 : cluster [DBG] 3.d scrub ok 2021-03-15T22:35:24.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:35:24.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: 15T22:35:23.498065+0000 mon.a (mon.0) 2417 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:24.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:35:24.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: 15T22:35:23.509957+0000 mon.a (mon.0) 2418 : 2021-03-15T22:35:24.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: audit [INF] from='client.? 172.21.15.76:0/1737589649' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:35:24.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:35:24.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: -15T22:35:23.510024+0000 mon.a (mon.0 2021-03-15T22:35:24.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:24 smithi125 conmon[56739]: ) 2419 : cluster [DBG] osdmap e968: 8 total, 8 up, 8 in 2021-03-15T22:35:24.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:24.797 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:24.798 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:24.899 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:24.932 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:24.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:24.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:25.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:25.059 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:25.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:25.060 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:25.161 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:25.185 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:25.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:25.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:25.287 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:25.313 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:25.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:25.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:25.338 INFO:tasks.rados.rados.0.smithi076.stdout:3692: done (1 left) 2021-03-15T22:35:25.339 INFO:tasks.rados.rados.0.smithi076.stdout:3693: done (0 left) 2021-03-15T22:35:25.339 INFO:tasks.rados.rados.0.smithi076.stdout:3694: delete oid 9 current snap is 434 2021-03-15T22:35:25.345 INFO:tasks.rados.rados.0.smithi076.stdout:3694: done (0 left) 2021-03-15T22:35:25.345 INFO:tasks.rados.rados.0.smithi076.stdout:3695: write (excl) oid 15 current snap is 434 2021-03-15T22:35:25.346 INFO:tasks.rados.rados.0.smithi076.stdout:3695: seq_num 1255 ranges {692785=435740,1672784=793765,2897503=195282} 2021-03-15T22:35:25.351 INFO:tasks.rados.rados.0.smithi076.stdout:3695: writing smithi07694878-15 from 692785 to 1128525 tid 1 2021-03-15T22:35:25.359 INFO:tasks.rados.rados.0.smithi076.stdout:3695: writing smithi07694878-15 from 1672784 to 2466549 tid 2 2021-03-15T22:35:25.363 INFO:tasks.rados.rados.0.smithi076.stdout:3695: writing smithi07694878-15 from 2897503 to 3092785 tid 3 2021-03-15T22:35:25.363 INFO:tasks.rados.rados.0.smithi076.stdout:3696: snap_create 2021-03-15T22:35:25.363 INFO:tasks.rados.rados.0.smithi076.stdout:3695: finishing write tid 1 to smithi07694878-15 2021-03-15T22:35:25.366 INFO:tasks.rados.rados.0.smithi076.stdout:3695: finishing write tid 2 to smithi07694878-15 2021-03-15T22:35:25.369 INFO:tasks.rados.rados.0.smithi076.stdout:3695: finishing write tid 3 to smithi07694878-15 2021-03-15T22:35:25.370 INFO:tasks.rados.rados.0.smithi076.stdout:3695: finishing write tid 4 to smithi07694878-15 2021-03-15T22:35:25.370 INFO:tasks.rados.rados.0.smithi076.stdout:3695: finishing write tid 5 to smithi07694878-15 2021-03-15T22:35:25.371 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 3166 (ObjNum 1255 snap 434 seq_num 1255) dirty exists 2021-03-15T22:35:25.371 INFO:tasks.rados.rados.0.smithi076.stdout:3695: left oid 15 (ObjNum 1255 snap 434 seq_num 1255) 2021-03-15T22:35:25.416 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:25.444 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:25.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:25.447 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:25.548 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:25.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:25.575 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:25.576 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:25.677 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:25.680 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_ops_in_flight 2021-03-15T22:35:25.741 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:25.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:25.743 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:25.766 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:25.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:25 smithi125 conmon[56739]: cluster 2021-03-15T22:35:24.522426+0000 mon.a (mon.0) 2420 : cluster [DBG] osdmap e969: 8 total, 8 up, 8 in 2021-03-15T22:35:25.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:25 smithi125 conmon[56739]: cluster 2021-03-15T22:35:25.336118+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e970: 8 total, 8 up, 8 in 2021-03-15T22:35:25.845 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:25.868 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:25.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:25.870 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:25.929 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:25 smithi076 conmon[56514]: cluster 2021-03-15T22:35:24.522426+0000 mon.a (mon.0) 2420 2021-03-15T22:35:25.929 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:25 smithi076 conmon[56514]: : cluster [DBG] osdmap e969: 8 total, 8 up, 8 in 2021-03-15T22:35:25.929 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:25 smithi076 conmon[56514]: cluster 2021-03-15T22:35:25.336118+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e970: 8 total, 8 up, 8 in 2021-03-15T22:35:25.930 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:25 smithi076 conmon[62156]: cluster 2021-03-15T22:35:24.522426+0000 mon.a 2021-03-15T22:35:25.930 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:25 smithi076 conmon[62156]: (mon.0) 2420 : cluster [DBG] osdmap e969: 8 total, 8 up, 8 in 2021-03-15T22:35:25.930 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:25 smithi076 conmon[62156]: cluster 2021-03-15T22:35:25. 2021-03-15T22:35:25.930 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:25 smithi076 conmon[62156]: 336118+0000 mon.a (mon.0) 2421 : cluster [DBG] osdmap e970: 8 total, 8 up, 8 in 2021-03-15T22:35:25.956 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:25.970 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:25.993 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:25.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:25.995 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:26.096 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:26.119 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:26.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:26.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:26.192 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:35:26.214 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:35:26.222 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:26.231 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:35:26.244 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:26.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:26.246 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:26.248 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:35:26.267 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:35:26.284 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:35:26.301 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:35:26.318 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:35:26.319 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:35:26.319 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:35:26.320 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:35:26.320 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:35:26.320 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:35:26.320 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:35:26.321 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:35:26.321 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:35:26.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:26.348 INFO:tasks.rados.rados.0.smithi076.stdout:3695: done (1 left) 2021-03-15T22:35:26.348 INFO:tasks.rados.rados.0.smithi076.stdout:3696: done (0 left) 2021-03-15T22:35:26.349 INFO:tasks.rados.rados.0.smithi076.stdout:3697: snap_create 2021-03-15T22:35:26.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:26.371 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:26.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:26.473 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:26.495 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:26.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:26.497 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:26.598 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:26.620 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:26.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:26.621 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:26.644 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:35:26.723 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:26.748 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:26.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:26.749 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:26.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:26 smithi076 conmon[56514]: cluster 2021-03-15T22:35:25.219419+0000 mgr.y (mgr.14144) 1076 : cluster [DBG] pgmap v1601: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 640 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2021-03-15T22:35:26.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:26 smithi076 conmon[56514]: audit 2021-03-15T22:35:26.172725+0000 mon.a (mon.0) 2422 : audit [DBG] from='client.? 172.21.15.76:0/2726926853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:26.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:26 smithi076 conmon[56514]: cluster 2021-03-15T22:35:26.344940+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e971: 8 total, 8 up, 8 in 2021-03-15T22:35:26.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:26 smithi076 conmon[62156]: cluster 2021-03-15T22:35:25.219419+0000 mgr.y (mgr.14144) 1076 : cluster [DBG] pgmap v1601: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 640 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2021-03-15T22:35:26.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:26 smithi076 conmon[62156]: audit 2021-03-15T22:35:26.172725+0000 mon.a (mon.0) 2422 : audit [DBG] from='client.? 172.21.15.76:0/2726926853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:26.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:26 smithi076 conmon[62156]: cluster 2021-03-15T22:35:26.344940+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e971: 8 total, 8 up, 8 in 2021-03-15T22:35:26.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:26 smithi125 conmon[56739]: cluster 2021-03-15T22:35:25.219419+0000 mgr.y (mgr. 2021-03-15T22:35:26.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:26 smithi125 conmon[56739]: 14144) 1076 : cluster [DBG] pgmap v1601: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 640 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 17 MiB/s rd, 1.5 MiB/s wr, 18 op/s 2021-03-15T22:35:26.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:26 smithi125 conmon[56739]: audit 2021-03-15T22:35:26.172725+0000 mon.a (mon.0) 2422 : audit [DBG] from='client.? 172.21.15.76:0/2726926853' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:26.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:26 smithi125 conmon[56739]: cluster 2021-03-15T22:35:26.344940+0000 mon.a (mon.0) 2423 : cluster [DBG] osdmap e971: 8 total, 8 up, 8 in 2021-03-15T22:35:26.851 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:26.873 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:26.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:26.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:26.976 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:26.998 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:26.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:27.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:27.101 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:27.123 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:27.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:27.125 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:27.226 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:27.248 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:27.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:27.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:27.351 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:27.353 INFO:tasks.rados.rados.0.smithi076.stdout:3697: done (0 left) 2021-03-15T22:35:27.354 INFO:tasks.rados.rados.0.smithi076.stdout:3698: snap_remove snap 400 2021-03-15T22:35:27.375 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:27.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:27.379 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:27.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:27.501 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:27.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:27.503 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:27.591 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:35:27.592 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:35:27.604 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:27.626 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:27.627 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:27.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:27.728 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:27.752 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:27.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:27.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:27.840 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:27.855 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:27.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:27.879 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:27.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:27.982 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:28.004 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:28.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:28.006 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:28.107 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:28.133 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:28.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:28.134 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:28.157 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_blocked_ops 2021-03-15T22:35:28.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:28.262 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:28.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:28.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:28.359 INFO:tasks.rados.rados.0.smithi076.stdout:3698: done (0 left) 2021-03-15T22:35:28.359 INFO:tasks.rados.rados.0.smithi076.stdout:3699: read oid 42 snap -1 2021-03-15T22:35:28.359 INFO:tasks.rados.rados.0.smithi076.stdout:3699: expect (ObjNum 1204 snap 418 seq_num 1204) 2021-03-15T22:35:28.360 INFO:tasks.rados.rados.0.smithi076.stdout:3700: read oid 39 snap -1 2021-03-15T22:35:28.360 INFO:tasks.rados.rados.0.smithi076.stdout:3700: expect (ObjNum 1249 snap 430 seq_num 1249) 2021-03-15T22:35:28.360 INFO:tasks.rados.rados.0.smithi076.stdout:3701: copy_from oid 29 from oid 14 current snap is 436 2021-03-15T22:35:28.360 INFO:tasks.rados.rados.0.smithi076.stdout:3702: read oid 33 snap 427 2021-03-15T22:35:28.360 INFO:tasks.rados.rados.0.smithi076.stdout:3702: expect (ObjNum 1174 snap 410 seq_num 1174) 2021-03-15T22:35:28.361 INFO:tasks.rados.rados.0.smithi076.stdout:3703: write (excl) oid 43 current snap is 436 2021-03-15T22:35:28.361 INFO:tasks.rados.rados.0.smithi076.stdout:3703: seq_num 1256 ranges {709592=510989,1731266=653776,3041279=68313} 2021-03-15T22:35:28.365 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:28.367 INFO:tasks.rados.rados.0.smithi076.stdout:3703: writing smithi07694878-43 from 709592 to 1220581 tid 1 2021-03-15T22:35:28.374 INFO:tasks.rados.rados.0.smithi076.stdout:3703: writing smithi07694878-43 from 1731266 to 2385042 tid 2 2021-03-15T22:35:28.380 INFO:tasks.rados.rados.0.smithi076.stdout:3703: writing smithi07694878-43 from 3041279 to 3109592 tid 3 2021-03-15T22:35:28.380 INFO:tasks.rados.rados.0.smithi076.stdout:3700: expect (ObjNum 1249 snap 430 seq_num 1249) 2021-03-15T22:35:28.397 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:28.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:28.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:28.442 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:28.500 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:28.521 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:28.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:28.523 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:28.532 INFO:tasks.rados.rados.0.smithi076.stdout:3703: finishing write tid 1 to smithi07694878-43 2021-03-15T22:35:28.532 INFO:tasks.rados.rados.0.smithi076.stdout:3703: finishing write tid 2 to smithi07694878-43 2021-03-15T22:35:28.533 INFO:tasks.rados.rados.0.smithi076.stdout:3702: expect (ObjNum 1174 snap 410 seq_num 1174) 2021-03-15T22:35:28.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:28 smithi125 conmon[56739]: cluster 2021-03-15T22:35:27.220071+0000 mgr.y (mgr.14144) 1077 : cluster [DBG] pgmap v1604: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 617 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2021-03-15T22:35:28.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:28 smithi125 conmon[56739]: cluster 2021-03-15T22:35:27.345575+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in 2021-03-15T22:35:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:28 smithi125 conmon[56739]: cluster 2021-03-15T22:35:27.417695+0000 osd.5 (osd.5) 56 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:28.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:28 smithi125 conmon[56739]: cluster 2021-03-15T22:35:27.420060+0000 osd.5 (osd.5) 57 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:28.625 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:28.648 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:28.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:28.650 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:28.722 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:35:28.726 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:35:28.727 INFO:tasks.rados.rados.0.smithi076.stdout:3703: finishing write tid 3 to smithi07694878-43 2021-03-15T22:35:28.727 INFO:tasks.rados.rados.0.smithi076.stdout:3699: expect (ObjNum 1204 snap 418 seq_num 1204) 2021-03-15T22:35:28.751 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:28.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:28 smithi076 conmon[56514]: cluster 2021-03-15T22:35:27.220071+0000 mgr.y ( 2021-03-15T22:35:28.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:28 smithi076 conmon[56514]: mgr.14144) 1077 : cluster [DBG] pgmap v1604: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 617 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2021-03-15T22:35:28.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:28 smithi076 conmon[56514]: cluster 2021-03-15T22:35:27.345575+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in 2021-03-15T22:35:28.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:28 smithi076 conmon[56514]: cluster 2021-03-15T22:35:27.417695+0000 osd.5 (osd.5) 56 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:28.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:28 smithi076 conmon[56514]: cluster 2021-03-15T22:35:27.420060+0000 osd.5 (osd.5) 57 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:28.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:28 smithi076 conmon[62156]: cluster 2021-03-15T22:35:27.220071+0000 mgr.y (mgr 2021-03-15T22:35:28.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:28 smithi076 conmon[62156]: .14144) 1077 : cluster [DBG] pgmap v1604: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 617 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.3 MiB/s wr, 13 op/s 2021-03-15T22:35:28.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:28 smithi076 conmon[62156]: cluster 2021-03-15T22:35:27.345575+0000 mon.a (mon.0) 2424 : cluster [DBG] osdmap e972: 8 total, 8 up, 8 in 2021-03-15T22:35:28.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:28 smithi076 conmon[62156]: cluster 2021-03-15T22:35:27.417695+0000 osd.5 (osd.5) 56 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:28.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:28 smithi076 conmon[62156]: cluster 2021-03-15T22:35:27.420060+0000 osd.5 (osd.5) 57 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:28.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:28.776 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:28.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:28.854 INFO:tasks.rados.rados.0.smithi076.stdout:3703: finishing write tid 4 to smithi07694878-43 2021-03-15T22:35:28.854 INFO:tasks.rados.rados.0.smithi076.stdout:3703: finishing write tid 5 to smithi07694878-43 2021-03-15T22:35:28.854 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 2438 (ObjNum 1256 snap 436 seq_num 1256) dirty exists 2021-03-15T22:35:28.855 INFO:tasks.rados.rados.0.smithi076.stdout:3703: left oid 43 (ObjNum 1256 snap 436 seq_num 1256) 2021-03-15T22:35:28.855 INFO:tasks.rados.rados.0.smithi076.stdout:3701: finishing copy_from to smithi07694878-29 2021-03-15T22:35:28.855 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 2677 (ObjNum 1140 snap 393 seq_num 1140) dirty exists 2021-03-15T22:35:28.855 INFO:tasks.rados.rados.0.smithi076.stdout:3701: finishing copy_from racing read to smithi07694878-29 2021-03-15T22:35:28.855 INFO:tasks.rados.rados.0.smithi076.stdout:3699: done (4 left) 2021-03-15T22:35:28.856 INFO:tasks.rados.rados.0.smithi076.stdout:3700: done (3 left) 2021-03-15T22:35:28.856 INFO:tasks.rados.rados.0.smithi076.stdout:3701: done (2 left) 2021-03-15T22:35:28.856 INFO:tasks.rados.rados.0.smithi076.stdout:3702: done (1 left) 2021-03-15T22:35:28.856 INFO:tasks.rados.rados.0.smithi076.stdout:3703: done (0 left) 2021-03-15T22:35:28.857 INFO:tasks.rados.rados.0.smithi076.stdout:3704: delete oid 4 current snap is 436 2021-03-15T22:35:28.857 INFO:tasks.rados.rados.0.smithi076.stdout:3704: done (0 left) 2021-03-15T22:35:28.858 INFO:tasks.rados.rados.0.smithi076.stdout:3705: write oid 15 current snap is 436 2021-03-15T22:35:28.858 INFO:tasks.rados.rados.0.smithi076.stdout:3705: seq_num 1258 ranges {743206=661487,2131877=541092} 2021-03-15T22:35:28.863 INFO:tasks.rados.rados.0.smithi076.stdout:3705: writing smithi07694878-15 from 743206 to 1404693 tid 1 2021-03-15T22:35:28.871 INFO:tasks.rados.rados.0.smithi076.stdout:3705: writing smithi07694878-15 from 2131877 to 2672969 tid 2 2021-03-15T22:35:28.871 INFO:tasks.rados.rados.0.smithi076.stdout:3706: read oid 47 snap 403 2021-03-15T22:35:28.871 INFO:tasks.rados.rados.0.smithi076.stdout:3706: expect (ObjNum 1144 snap 400 seq_num 1144) 2021-03-15T22:35:28.871 INFO:tasks.rados.rados.0.smithi076.stdout:3707: delete oid 31 current snap is 436 2021-03-15T22:35:28.872 INFO:tasks.rados.rados.0.smithi076.stdout:3705: finishing write tid 1 to smithi07694878-15 2021-03-15T22:35:28.873 INFO:tasks.rados.rados.0.smithi076.stdout:3707: done (2 left) 2021-03-15T22:35:28.873 INFO:tasks.rados.rados.0.smithi076.stdout:3708: delete oid 7 current snap is 436 2021-03-15T22:35:28.875 INFO:tasks.rados.rados.0.smithi076.stdout:3705: finishing write tid 2 to smithi07694878-15 2021-03-15T22:35:28.876 INFO:tasks.rados.rados.0.smithi076.stdout:3705: finishing write tid 3 to smithi07694878-15 2021-03-15T22:35:28.876 INFO:tasks.rados.rados.0.smithi076.stdout:3705: finishing write tid 4 to smithi07694878-15 2021-03-15T22:35:28.876 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 3169 (ObjNum 1258 snap 436 seq_num 1258) dirty exists 2021-03-15T22:35:28.877 INFO:tasks.rados.rados.0.smithi076.stdout:3705: left oid 15 (ObjNum 1258 snap 436 seq_num 1258) 2021-03-15T22:35:28.877 INFO:tasks.rados.rados.0.smithi076.stdout:3705: done (2 left) 2021-03-15T22:35:28.877 INFO:tasks.rados.rados.0.smithi076.stdout:3708: done (1 left) 2021-03-15T22:35:28.877 INFO:tasks.rados.rados.0.smithi076.stdout:3709: read oid 38 snap -1 2021-03-15T22:35:28.877 INFO:tasks.rados.rados.0.smithi076.stdout:3709: expect (ObjNum 1141 snap 397 seq_num 1141) 2021-03-15T22:35:28.878 INFO:tasks.rados.rados.0.smithi076.stdout:3710: write oid 18 current snap is 436 2021-03-15T22:35:28.878 INFO:tasks.rados.rados.0.smithi076.stdout:3710: seq_num 1261 ranges {793627=487234,1790970=404772,2829335=364292} 2021-03-15T22:35:28.878 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:28.883 INFO:tasks.rados.rados.0.smithi076.stdout:3710: writing smithi07694878-18 from 793627 to 1280861 tid 1 2021-03-15T22:35:28.888 INFO:tasks.rados.rados.0.smithi076.stdout:3710: writing smithi07694878-18 from 1790970 to 2195742 tid 2 2021-03-15T22:35:28.894 INFO:tasks.rados.rados.0.smithi076.stdout:3710: writing smithi07694878-18 from 2829335 to 3193627 tid 3 2021-03-15T22:35:28.894 INFO:tasks.rados.rados.0.smithi076.stdout:3711: delete oid 13 current snap is 436 2021-03-15T22:35:28.894 INFO:tasks.rados.rados.0.smithi076.stdout:3706: expect (ObjNum 1144 snap 400 seq_num 1144) 2021-03-15T22:35:28.902 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:28.902 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:28.903 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:29.004 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:29.026 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:29.027 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:29.028 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:29.129 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:29.139 INFO:tasks.rados.rados.0.smithi076.stdout:3709: expect (ObjNum 1141 snap 397 seq_num 1141) 2021-03-15T22:35:29.150 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:29.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:29.152 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:29.253 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:29.274 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:29.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:29.275 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:29.325 INFO:tasks.rados.rados.0.smithi076.stdout:3710: finishing write tid 1 to smithi07694878-18 2021-03-15T22:35:29.325 INFO:tasks.rados.rados.0.smithi076.stdout:3710: finishing write tid 2 to smithi07694878-18 2021-03-15T22:35:29.326 INFO:tasks.rados.rados.0.smithi076.stdout:3710: finishing write tid 3 to smithi07694878-18 2021-03-15T22:35:29.326 INFO:tasks.rados.rados.0.smithi076.stdout:3710: finishing write tid 4 to smithi07694878-18 2021-03-15T22:35:29.326 INFO:tasks.rados.rados.0.smithi076.stdout:3710: finishing write tid 5 to smithi07694878-18 2021-03-15T22:35:29.327 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 2442 (ObjNum 1261 snap 436 seq_num 1261) dirty exists 2021-03-15T22:35:29.327 INFO:tasks.rados.rados.0.smithi076.stdout:3710: left oid 18 (ObjNum 1261 snap 436 seq_num 1261) 2021-03-15T22:35:29.327 INFO:tasks.rados.rados.0.smithi076.stdout:3706: done (3 left) 2021-03-15T22:35:29.327 INFO:tasks.rados.rados.0.smithi076.stdout:3709: done (2 left) 2021-03-15T22:35:29.327 INFO:tasks.rados.rados.0.smithi076.stdout:3710: done (1 left) 2021-03-15T22:35:29.328 INFO:tasks.rados.rados.0.smithi076.stdout:3711: done (0 left) 2021-03-15T22:35:29.328 INFO:tasks.rados.rados.0.smithi076.stdout:3712: copy_from oid 29 from oid 14 current snap is 436 2021-03-15T22:35:29.328 INFO:tasks.rados.rados.0.smithi076.stdout:3713: snap_create 2021-03-15T22:35:29.345 INFO:tasks.rados.rados.0.smithi076.stdout:3712: finishing copy_from to smithi07694878-29 2021-03-15T22:35:29.345 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 2677 (ObjNum 1140 snap 393 seq_num 1140) dirty exists 2021-03-15T22:35:29.345 INFO:tasks.rados.rados.0.smithi076.stdout:3712: finishing copy_from racing read to smithi07694878-29 2021-03-15T22:35:29.359 INFO:tasks.rados.rados.0.smithi076.stdout:3712: done (1 left) 2021-03-15T22:35:29.359 INFO:tasks.rados.rados.0.smithi076.stdout:3713: done (0 left) 2021-03-15T22:35:29.359 INFO:tasks.rados.rados.0.smithi076.stdout:3714: delete oid 31 current snap is 437 2021-03-15T22:35:29.362 INFO:tasks.rados.rados.0.smithi076.stdout:3714: done (0 left) 2021-03-15T22:35:29.363 INFO:tasks.rados.rados.0.smithi076.stdout:3715: snap_remove snap 367 2021-03-15T22:35:29.376 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:29.399 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:29.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:29.400 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:29.501 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:29.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:29.524 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:29.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:29.576 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1606,"stamp":"2021-03-15T22:35:28.348152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":646712561,"num_objects":619,"num_object_clones":254,"num_object_copies":1857,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":619,"num_whiteouts":17,"num_read":9355,"num_read_kb":7012923,"num_write":5858,"num_write_kb":1917193,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5899,"num_bytes_recovered":2230123701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23083,"ondisk_log_size":23083,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":475,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1851916,"kb_used_data":1438604,"kb_used_omap":273,"kb_used_meta":412974,"kb_avail":748109300,"statfs":{"total":767960285184,"available":766063923200,"internally_reserved":0,"allocated":1473130496,"data_stored":2081170000,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":422886266},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-24316779,"num_objects":-8,"num_object_clones":-8,"num_object_copies":-24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":39,"num_read_kb":49539,"num_write":9,"num_write_kb":5152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.830418"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"470","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428272+0000","last_change":"2021-03-15T22:34:55.350872+0000","last_active":"2021-03-15T22:35:23.428272+0000","last_peered":"2021-03-15T22:35:23.428272+0000","last_clean":"2021-03-15T22:35:23.428272+0000","last_became_active":"2021-03-15T22:34:55.350524+0000","last_became_peered":"2021-03-15T22:34:55.350524+0000","last_unstale":"2021-03-15T22:35:23.428272+0000","last_undegraded":"2021-03-15T22:35:23.428272+0000","last_fullsized":"2021-03-15T22:35:23.428272+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"38","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.289156+0000","last_change":"2021-03-15T22:34:55.349985+0000","last_active":"2021-03-15T22:35:23.289156+0000","last_peered":"2021-03-15T22:35:23.289156+0000","last_clean":"2021-03-15T22:35:23.289156+0000","last_became_active":"2021-03-15T22:34:55.348740+0000","last_became_peered":"2021-03-15T22:34:55.348740+0000","last_unstale":"2021-03-15T22:35:23.289156+0000","last_undegraded":"2021-03-15T22:35:23.289156+0000","last_fullsized":"2021-03-15T22:35:23.289156+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,6],"acting":[2,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"64","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.517939+0000","last_change":"2021-03-15T22:35:18.617982+0000","last_active":"2021-03-15T22:35:24.517939+0000","last_peered":"2021-03-15T22:35:24.517939+0000","last_clean":"2021-03-15T22:35:24.517939+0000","last_became_active":"2021-03-15T22:34:36.653285+0000","last_became_peered":"2021-03-15T22:34:36.653285+0000","last_unstale":"2021-03-15T22:35:24.517939+0000","last_undegraded":"2021-03-15T22:35:24.517939+0000","last_fullsized":"2021-03-15T22:35:24.517939+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:18.617912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.617912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1],"acting":[6,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"60","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.343967+0000","last_change":"2021-03-15T22:34:36.657886+0000","last_active":"2021-03-15T22:35:26.343967+0000","last_peered":"2021-03-15T22:35:26.343967+0000","last_clean":"2021-03-15T22:35:26.343967+0000","last_became_active":"2021-03-15T22:34:36.657561+0000","last_became_peered":"2021-03-15T22:34:36.657561+0000","last_unstale":"2021-03-15T22:35:26.343967+0000","last_undegraded":"2021-03-15T22:35:26.343967+0000","last_fullsized":"2021-03-15T22:35:26.343967+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"1005","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.518169+0000","last_change":"2021-03-15T22:34:55.346316+0000","last_active":"2021-03-15T22:35:24.518169+0000","last_peered":"2021-03-15T22:35:24.518169+0000","last_clean":"2021-03-15T22:35:24.518169+0000","last_became_active":"2021-03-15T22:34:55.345943+0000","last_became_peered":"2021-03-15T22:34:55.345943+0000","last_unstale":"2021-03-15T22:35:24.518169+0000","last_undegraded":"2021-03-15T22:35:24.518169+0000","last_fullsized":"2021-03-15T22:35:24.518169+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:32.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,1],"acting":[6,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"777","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.500865+0000","last_change":"2021-03-15T22:34:55.355408+0000","last_active":"2021-03-15T22:35:22.500865+0000","last_peered":"2021-03-15T22:35:22.500865+0000","last_clean":"2021-03-15T22:35:22.500865+0000","last_became_active":"2021-03-15T22:34:55.355062+0000","last_became_peered":"2021-03-15T22:34:55.355062+0000","last_unstale":"2021-03-15T22:35:22.500865+0000","last_undegraded":"2021-03-15T22:35:22.500865+0000","last_fullsized":"2021-03-15T22:35:22.500865+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"988","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.517516+0000","last_change":"2021-03-15T22:35:14.592505+0000","last_active":"2021-03-15T22:35:24.517516+0000","last_peered":"2021-03-15T22:35:24.517516+0000","last_clean":"2021-03-15T22:35:24.517516+0000","last_became_active":"2021-03-15T22:34:36.652531+0000","last_became_peered":"2021-03-15T22:34:36.652531+0000","last_unstale":"2021-03-15T22:35:24.517516+0000","last_undegraded":"2021-03-15T22:35:24.517516+0000","last_fullsized":"2021-03-15T22:35:24.517516+0000","mapping_epoch":920,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":921,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:14.592451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:14.592451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,4],"acting":[6,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"1038","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.517403+0000","last_change":"2021-03-15T22:34:55.344965+0000","last_active":"2021-03-15T22:35:24.517403+0000","last_peered":"2021-03-15T22:35:24.517403+0000","last_clean":"2021-03-15T22:35:24.517403+0000","last_became_active":"2021-03-15T22:34:55.343726+0000","last_became_peered":"2021-03-15T22:34:55.343726+0000","last_unstale":"2021-03-15T22:35:24.517403+0000","last_undegraded":"2021-03-15T22:35:24.517403+0000","last_fullsized":"2021-03-15T22:35:24.517403+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"1000","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428344+0000","last_change":"2021-03-15T22:34:55.351171+0000","last_active":"2021-03-15T22:35:23.428344+0000","last_peered":"2021-03-15T22:35:23.428344+0000","last_clean":"2021-03-15T22:35:23.428344+0000","last_became_active":"2021-03-15T22:34:55.350597+0000","last_became_peered":"2021-03-15T22:34:55.350597+0000","last_unstale":"2021-03-15T22:35:23.428344+0000","last_undegraded":"2021-03-15T22:35:23.428344+0000","last_fullsized":"2021-03-15T22:35:23.428344+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.701955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.701955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"69","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.343770+0000","last_change":"2021-03-15T22:34:55.350719+0000","last_active":"2021-03-15T22:35:26.343770+0000","last_peered":"2021-03-15T22:35:26.343770+0000","last_clean":"2021-03-15T22:35:26.343770+0000","last_became_active":"2021-03-15T22:34:55.350384+0000","last_became_peered":"2021-03-15T22:34:55.350384+0000","last_unstale":"2021-03-15T22:35:26.343770+0000","last_undegraded":"2021-03-15T22:35:26.343770+0000","last_fullsized":"2021-03-15T22:35:26.343770+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"317'7","reported_seq":"335","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.344673+0000","last_change":"2021-03-15T22:34:39.122745+0000","last_active":"2021-03-15T22:35:26.344673+0000","last_peered":"2021-03-15T22:35:26.344673+0000","last_clean":"2021-03-15T22:35:26.344673+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:35:26.344673+0000","last_undegraded":"2021-03-15T22:35:26.344673+0000","last_fullsized":"2021-03-15T22:35:26.344673+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:34:39.122665+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.122665+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"986","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429535+0000","last_change":"2021-03-15T22:34:10.331764+0000","last_active":"2021-03-15T22:35:25.429535+0000","last_peered":"2021-03-15T22:35:25.429535+0000","last_clean":"2021-03-15T22:35:25.429535+0000","last_became_active":"2021-03-15T22:34:10.331424+0000","last_became_peered":"2021-03-15T22:34:10.331424+0000","last_unstale":"2021-03-15T22:35:25.429535+0000","last_undegraded":"2021-03-15T22:35:25.429535+0000","last_fullsized":"2021-03-15T22:35:25.429535+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.564464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.564464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"497","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.521238+0000","last_change":"2021-03-15T22:34:55.389531+0000","last_active":"2021-03-15T22:35:24.521238+0000","last_peered":"2021-03-15T22:35:24.521238+0000","last_clean":"2021-03-15T22:35:24.521238+0000","last_became_active":"2021-03-15T22:34:55.356022+0000","last_became_peered":"2021-03-15T22:34:55.356022+0000","last_unstale":"2021-03-15T22:35:24.521238+0000","last_undegraded":"2021-03-15T22:35:24.521238+0000","last_fullsized":"2021-03-15T22:35:24.521238+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:32:31.680111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.680111+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"1069","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430442+0000","last_change":"2021-03-15T22:35:14.434026+0000","last_active":"2021-03-15T22:35:25.430442+0000","last_peered":"2021-03-15T22:35:25.430442+0000","last_clean":"2021-03-15T22:35:25.430442+0000","last_became_active":"2021-03-15T22:33:04.345393+0000","last_became_peered":"2021-03-15T22:33:04.345393+0000","last_unstale":"2021-03-15T22:35:25.430442+0000","last_undegraded":"2021-03-15T22:35:25.430442+0000","last_fullsized":"2021-03-15T22:35:25.430442+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:35:14.433961+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:35:14.433961+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"273","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429318+0000","last_change":"2021-03-15T22:33:04.345933+0000","last_active":"2021-03-15T22:35:25.429318+0000","last_peered":"2021-03-15T22:35:25.429318+0000","last_clean":"2021-03-15T22:35:25.429318+0000","last_became_active":"2021-03-15T22:33:04.345518+0000","last_became_peered":"2021-03-15T22:33:04.345518+0000","last_unstale":"2021-03-15T22:35:25.429318+0000","last_undegraded":"2021-03-15T22:35:25.429318+0000","last_fullsized":"2021-03-15T22:35:25.429318+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.338100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.338100+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"52","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.524674+0000","last_change":"2021-03-15T22:34:43.345093+0000","last_active":"2021-03-15T22:35:24.524674+0000","last_peered":"2021-03-15T22:35:24.524674+0000","last_clean":"2021-03-15T22:35:24.524674+0000","last_became_active":"2021-03-15T22:34:43.344764+0000","last_became_peered":"2021-03-15T22:34:43.344764+0000","last_unstale":"2021-03-15T22:35:24.524674+0000","last_undegraded":"2021-03-15T22:35:24.524674+0000","last_fullsized":"2021-03-15T22:35:24.524674+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:08.652741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.652741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"61","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.523988+0000","last_change":"2021-03-15T22:34:55.347147+0000","last_active":"2021-03-15T22:35:24.523988+0000","last_peered":"2021-03-15T22:35:24.523988+0000","last_clean":"2021-03-15T22:35:24.523988+0000","last_became_active":"2021-03-15T22:34:55.346711+0000","last_became_peered":"2021-03-15T22:34:55.346711+0000","last_unstale":"2021-03-15T22:35:24.523988+0000","last_undegraded":"2021-03-15T22:35:24.523988+0000","last_fullsized":"2021-03-15T22:35:24.523988+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.475850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.475850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"113","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.523536+0000","last_change":"2021-03-15T22:34:55.399761+0000","last_active":"2021-03-15T22:35:24.523536+0000","last_peered":"2021-03-15T22:35:24.523536+0000","last_clean":"2021-03-15T22:35:24.523536+0000","last_became_active":"2021-03-15T22:34:55.346148+0000","last_became_peered":"2021-03-15T22:34:55.346148+0000","last_unstale":"2021-03-15T22:35:24.523536+0000","last_undegraded":"2021-03-15T22:35:24.523536+0000","last_fullsized":"2021-03-15T22:35:24.523536+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.652898+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":5340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"818","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.500661+0000","last_change":"2021-03-15T22:35:16.848985+0000","last_active":"2021-03-15T22:35:22.500661+0000","last_peered":"2021-03-15T22:35:22.500661+0000","last_clean":"2021-03-15T22:35:22.500661+0000","last_became_active":"2021-03-15T22:34:03.570306+0000","last_became_peered":"2021-03-15T22:34:03.570306+0000","last_unstale":"2021-03-15T22:35:22.500661+0000","last_undegraded":"2021-03-15T22:35:22.500661+0000","last_fullsized":"2021-03-15T22:35:22.500661+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:16.848918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.848918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"434","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.520759+0000","last_change":"2021-03-15T22:32:01.452949+0000","last_active":"2021-03-15T22:35:24.520759+0000","last_peered":"2021-03-15T22:35:24.520759+0000","last_clean":"2021-03-15T22:35:24.520759+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:35:24.520759+0000","last_undegraded":"2021-03-15T22:35:24.520759+0000","last_fullsized":"2021-03-15T22:35:24.520759+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:01.452885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:32:01.452885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"361","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501016+0000","last_change":"2021-03-15T22:33:39.756677+0000","last_active":"2021-03-15T22:35:22.501016+0000","last_peered":"2021-03-15T22:35:22.501016+0000","last_clean":"2021-03-15T22:35:22.501016+0000","last_became_active":"2021-03-15T22:33:04.325611+0000","last_became_peered":"2021-03-15T22:33:04.325611+0000","last_unstale":"2021-03-15T22:35:22.501016+0000","last_undegraded":"2021-03-15T22:35:22.501016+0000","last_fullsized":"2021-03-15T22:35:22.501016+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.756581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"1120","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429677+0000","last_change":"2021-03-15T22:33:17.502781+0000","last_active":"2021-03-15T22:35:25.429677+0000","last_peered":"2021-03-15T22:35:25.429677+0000","last_clean":"2021-03-15T22:35:25.429677+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:35:25.429677+0000","last_undegraded":"2021-03-15T22:35:25.429677+0000","last_fullsized":"2021-03-15T22:35:25.429677+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:33:17.502703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.502703+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"253","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428648+0000","last_change":"2021-03-15T22:35:12.404980+0000","last_active":"2021-03-15T22:35:23.428648+0000","last_peered":"2021-03-15T22:35:23.428648+0000","last_clean":"2021-03-15T22:35:23.428648+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:35:23.428648+0000","last_undegraded":"2021-03-15T22:35:23.428648+0000","last_fullsized":"2021-03-15T22:35:23.428648+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.404897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.404897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"606","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.500763+0000","last_change":"2021-03-15T22:34:43.344692+0000","last_active":"2021-03-15T22:35:22.500763+0000","last_peered":"2021-03-15T22:35:22.500763+0000","last_clean":"2021-03-15T22:35:22.500763+0000","last_became_active":"2021-03-15T22:34:43.344401+0000","last_became_peered":"2021-03-15T22:34:43.344401+0000","last_unstale":"2021-03-15T22:35:22.500763+0000","last_undegraded":"2021-03-15T22:35:22.500763+0000","last_fullsized":"2021-03-15T22:35:22.500763+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.774079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.774079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"343","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501272+0000","last_change":"2021-03-15T22:32:14.756556+0000","last_active":"2021-03-15T22:35:22.501272+0000","last_peered":"2021-03-15T22:35:22.501272+0000","last_clean":"2021-03-15T22:35:22.501272+0000","last_became_active":"2021-03-15T22:29:51.234868+0000","last_became_peered":"2021-03-15T22:29:51.234868+0000","last_unstale":"2021-03-15T22:35:22.501272+0000","last_undegraded":"2021-03-15T22:35:22.501272+0000","last_fullsized":"2021-03-15T22:35:22.501272+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.756460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.756460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"1111","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429858+0000","last_change":"2021-03-15T22:35:10.528771+0000","last_active":"2021-03-15T22:35:25.429858+0000","last_peered":"2021-03-15T22:35:25.429858+0000","last_clean":"2021-03-15T22:35:25.429858+0000","last_became_active":"2021-03-15T22:34:55.335550+0000","last_became_peered":"2021-03-15T22:34:55.335550+0000","last_unstale":"2021-03-15T22:35:25.429858+0000","last_undegraded":"2021-03-15T22:35:25.429858+0000","last_fullsized":"2021-03-15T22:35:25.429858+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:32:05.111031+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.111031+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"932","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.345495+0000","last_change":"2021-03-15T22:35:10.068109+0000","last_active":"2021-03-15T22:35:26.345495+0000","last_peered":"2021-03-15T22:35:26.345495+0000","last_clean":"2021-03-15T22:35:26.345495+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:35:26.345495+0000","last_undegraded":"2021-03-15T22:35:26.345495+0000","last_fullsized":"2021-03-15T22:35:26.345495+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:35:10.068030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.068030+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"993","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430594+0000","last_change":"2021-03-15T22:34:41.614750+0000","last_active":"2021-03-15T22:35:25.430594+0000","last_peered":"2021-03-15T22:35:25.430594+0000","last_clean":"2021-03-15T22:35:25.430594+0000","last_became_active":"2021-03-15T22:33:04.329078+0000","last_became_peered":"2021-03-15T22:33:04.329078+0000","last_unstale":"2021-03-15T22:35:25.430594+0000","last_undegraded":"2021-03-15T22:35:25.430594+0000","last_fullsized":"2021-03-15T22:35:25.430594+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.614684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.614684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"438","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.519108+0000","last_change":"2021-03-15T22:32:41.673856+0000","last_active":"2021-03-15T22:35:24.519108+0000","last_peered":"2021-03-15T22:35:24.519108+0000","last_clean":"2021-03-15T22:35:24.519108+0000","last_became_active":"2021-03-15T22:28:12.519619+0000","last_became_peered":"2021-03-15T22:28:12.519619+0000","last_unstale":"2021-03-15T22:35:24.519108+0000","last_undegraded":"2021-03-15T22:35:24.519108+0000","last_fullsized":"2021-03-15T22:35:24.519108+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:32:41.673792+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.673792+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"529","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.345225+0000","last_change":"2021-03-15T22:34:55.366147+0000","last_active":"2021-03-15T22:35:26.345225+0000","last_peered":"2021-03-15T22:35:26.345225+0000","last_clean":"2021-03-15T22:35:26.345225+0000","last_became_active":"2021-03-15T22:34:55.350325+0000","last_became_peered":"2021-03-15T22:34:55.350325+0000","last_unstale":"2021-03-15T22:35:26.345225+0000","last_undegraded":"2021-03-15T22:35:26.345225+0000","last_fullsized":"2021-03-15T22:35:26.345225+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:34:38.119156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:38.119156+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1019","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429984+0000","last_change":"2021-03-15T22:35:09.568158+0000","last_active":"2021-03-15T22:35:25.429984+0000","last_peered":"2021-03-15T22:35:25.429984+0000","last_clean":"2021-03-15T22:35:25.429984+0000","last_became_active":"2021-03-15T22:34:55.338001+0000","last_became_peered":"2021-03-15T22:34:55.338001+0000","last_unstale":"2021-03-15T22:35:25.429984+0000","last_undegraded":"2021-03-15T22:35:25.429984+0000","last_fullsized":"2021-03-15T22:35:25.429984+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.144132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.144132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"807","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.525097+0000","last_change":"2021-03-15T22:35:20.025959+0000","last_active":"2021-03-15T22:35:24.525097+0000","last_peered":"2021-03-15T22:35:24.525097+0000","last_clean":"2021-03-15T22:35:24.525097+0000","last_became_active":"2021-03-15T22:33:04.336804+0000","last_became_peered":"2021-03-15T22:33:04.336804+0000","last_unstale":"2021-03-15T22:35:24.525097+0000","last_undegraded":"2021-03-15T22:35:24.525097+0000","last_fullsized":"2021-03-15T22:35:24.525097+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.025892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"983","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429721+0000","last_change":"2021-03-15T22:34:55.343334+0000","last_active":"2021-03-15T22:35:25.429721+0000","last_peered":"2021-03-15T22:35:25.429721+0000","last_clean":"2021-03-15T22:35:25.429721+0000","last_became_active":"2021-03-15T22:34:55.342358+0000","last_became_peered":"2021-03-15T22:34:55.342358+0000","last_unstale":"2021-03-15T22:35:25.429721+0000","last_undegraded":"2021-03-15T22:35:25.429721+0000","last_fullsized":"2021-03-15T22:35:25.429721+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.429029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.429029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"418","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.524094+0000","last_change":"2021-03-15T22:33:15.617975+0000","last_active":"2021-03-15T22:35:24.524094+0000","last_peered":"2021-03-15T22:35:24.524094+0000","last_clean":"2021-03-15T22:35:24.524094+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:35:24.524094+0000","last_undegraded":"2021-03-15T22:35:24.524094+0000","last_fullsized":"2021-03-15T22:35:24.524094+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:33:15.617891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.617891+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"891'22","reported_seq":"531","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.518344+0000","last_change":"2021-03-15T22:35:11.551893+0000","last_active":"2021-03-15T22:35:24.518344+0000","last_peered":"2021-03-15T22:35:24.518344+0000","last_clean":"2021-03-15T22:35:24.518344+0000","last_became_active":"2021-03-15T22:34:06.311478+0000","last_became_peered":"2021-03-15T22:34:06.311478+0000","last_unstale":"2021-03-15T22:35:24.518344+0000","last_undegraded":"2021-03-15T22:35:24.518344+0000","last_fullsized":"2021-03-15T22:35:24.518344+0000","mapping_epoch":890,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"841'19","last_scrub_stamp":"2021-03-15T22:33:20.272520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.272520+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"999","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.429538+0000","last_change":"2021-03-15T22:34:55.352256+0000","last_active":"2021-03-15T22:35:23.429538+0000","last_peered":"2021-03-15T22:35:23.429538+0000","last_clean":"2021-03-15T22:35:23.429538+0000","last_became_active":"2021-03-15T22:34:55.351597+0000","last_became_peered":"2021-03-15T22:34:55.351597+0000","last_unstale":"2021-03-15T22:35:23.429538+0000","last_undegraded":"2021-03-15T22:35:23.429538+0000","last_fullsized":"2021-03-15T22:35:23.429538+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:43.393275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:43.393275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"1002","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.344823+0000","last_change":"2021-03-15T22:35:11.111684+0000","last_active":"2021-03-15T22:35:26.344823+0000","last_peered":"2021-03-15T22:35:26.344823+0000","last_clean":"2021-03-15T22:35:26.344823+0000","last_became_active":"2021-03-15T22:28:59.264559+0000","last_became_peered":"2021-03-15T22:28:59.264559+0000","last_unstale":"2021-03-15T22:35:26.344823+0000","last_undegraded":"2021-03-15T22:35:26.344823+0000","last_fullsized":"2021-03-15T22:35:26.344823+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:43.703019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.703019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"572","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.519654+0000","last_change":"2021-03-15T22:34:10.385041+0000","last_active":"2021-03-15T22:35:24.519654+0000","last_peered":"2021-03-15T22:35:24.519654+0000","last_clean":"2021-03-15T22:35:24.519654+0000","last_became_active":"2021-03-15T22:34:10.347227+0000","last_became_peered":"2021-03-15T22:34:10.347227+0000","last_unstale":"2021-03-15T22:35:24.519654+0000","last_undegraded":"2021-03-15T22:35:24.519654+0000","last_fullsized":"2021-03-15T22:35:24.519654+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:34:07.125155+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.125155+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":3736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"1048","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.518939+0000","last_change":"2021-03-15T22:34:58.356466+0000","last_active":"2021-03-15T22:35:24.518939+0000","last_peered":"2021-03-15T22:35:24.518939+0000","last_clean":"2021-03-15T22:35:24.518939+0000","last_became_active":"2021-03-15T22:34:58.353086+0000","last_became_peered":"2021-03-15T22:34:58.353086+0000","last_unstale":"2021-03-15T22:35:24.518939+0000","last_undegraded":"2021-03-15T22:35:24.518939+0000","last_fullsized":"2021-03-15T22:35:24.518939+0000","mapping_epoch":942,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:32:34.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.739637+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"981","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.345864+0000","last_change":"2021-03-15T22:35:18.111879+0000","last_active":"2021-03-15T22:35:26.345864+0000","last_peered":"2021-03-15T22:35:26.345864+0000","last_clean":"2021-03-15T22:35:26.345864+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:35:26.345864+0000","last_undegraded":"2021-03-15T22:35:26.345864+0000","last_fullsized":"2021-03-15T22:35:26.345864+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:18.111815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.111815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"595","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.525372+0000","last_change":"2021-03-15T22:34:55.345698+0000","last_active":"2021-03-15T22:35:24.525372+0000","last_peered":"2021-03-15T22:35:24.525372+0000","last_clean":"2021-03-15T22:35:24.525372+0000","last_became_active":"2021-03-15T22:34:55.345277+0000","last_became_peered":"2021-03-15T22:34:55.345277+0000","last_unstale":"2021-03-15T22:35:24.525372+0000","last_undegraded":"2021-03-15T22:35:24.525372+0000","last_fullsized":"2021-03-15T22:35:24.525372+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.844183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.844183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"178","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.518257+0000","last_change":"2021-03-15T22:34:43.447677+0000","last_active":"2021-03-15T22:35:24.518257+0000","last_peered":"2021-03-15T22:35:24.518257+0000","last_clean":"2021-03-15T22:35:24.518257+0000","last_became_active":"2021-03-15T22:33:04.336100+0000","last_became_peered":"2021-03-15T22:33:04.336100+0000","last_unstale":"2021-03-15T22:35:24.518257+0000","last_undegraded":"2021-03-15T22:35:24.518257+0000","last_fullsized":"2021-03-15T22:35:24.518257+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:34:43.447602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:43.447602+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"981","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428719+0000","last_change":"2021-03-15T22:34:55.351849+0000","last_active":"2021-03-15T22:35:23.428719+0000","last_peered":"2021-03-15T22:35:23.428719+0000","last_clean":"2021-03-15T22:35:23.428719+0000","last_became_active":"2021-03-15T22:34:55.351089+0000","last_became_peered":"2021-03-15T22:34:55.351089+0000","last_unstale":"2021-03-15T22:35:23.428719+0000","last_undegraded":"2021-03-15T22:35:23.428719+0000","last_fullsized":"2021-03-15T22:35:23.428719+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.469521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.469521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"154","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.517640+0000","last_change":"2021-03-15T22:35:22.665325+0000","last_active":"2021-03-15T22:35:24.517640+0000","last_peered":"2021-03-15T22:35:24.517640+0000","last_clean":"2021-03-15T22:35:24.517640+0000","last_became_active":"2021-03-15T22:33:04.335734+0000","last_became_peered":"2021-03-15T22:33:04.335734+0000","last_unstale":"2021-03-15T22:35:24.517640+0000","last_undegraded":"2021-03-15T22:35:24.517640+0000","last_fullsized":"2021-03-15T22:35:24.517640+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.665251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.665251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"496","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.521160+0000","last_change":"2021-03-15T22:35:22.291513+0000","last_active":"2021-03-15T22:35:24.521160+0000","last_peered":"2021-03-15T22:35:24.521160+0000","last_clean":"2021-03-15T22:35:24.521160+0000","last_became_active":"2021-03-15T22:34:43.350301+0000","last_became_peered":"2021-03-15T22:34:43.350301+0000","last_unstale":"2021-03-15T22:35:24.521160+0000","last_undegraded":"2021-03-15T22:35:24.521160+0000","last_fullsized":"2021-03-15T22:35:24.521160+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.291406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:22.291406+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.291406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"482","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.525097+0000","last_change":"2021-03-15T22:33:37.752077+0000","last_active":"2021-03-15T22:35:24.525097+0000","last_peered":"2021-03-15T22:35:24.525097+0000","last_clean":"2021-03-15T22:35:24.525097+0000","last_became_active":"2021-03-15T22:33:04.342749+0000","last_became_peered":"2021-03-15T22:33:04.342749+0000","last_unstale":"2021-03-15T22:35:24.525097+0000","last_undegraded":"2021-03-15T22:35:24.525097+0000","last_fullsized":"2021-03-15T22:35:24.525097+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:33:37.752011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.752011+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":8764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"595","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.524908+0000","last_change":"2021-03-15T22:34:34.783107+0000","last_active":"2021-03-15T22:35:24.524908+0000","last_peered":"2021-03-15T22:35:24.524908+0000","last_clean":"2021-03-15T22:35:24.524908+0000","last_became_active":"2021-03-15T22:33:04.333077+0000","last_became_peered":"2021-03-15T22:33:04.333077+0000","last_unstale":"2021-03-15T22:35:24.524908+0000","last_undegraded":"2021-03-15T22:35:24.524908+0000","last_fullsized":"2021-03-15T22:35:24.524908+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.783047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.783047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"91","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.525178+0000","last_change":"2021-03-15T22:34:03.571320+0000","last_active":"2021-03-15T22:35:24.525178+0000","last_peered":"2021-03-15T22:35:24.525178+0000","last_clean":"2021-03-15T22:35:24.525178+0000","last_became_active":"2021-03-15T22:34:03.571049+0000","last_became_peered":"2021-03-15T22:34:03.571049+0000","last_unstale":"2021-03-15T22:35:24.525178+0000","last_undegraded":"2021-03-15T22:35:24.525178+0000","last_fullsized":"2021-03-15T22:35:24.525178+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:05.522372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.522372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"343","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501665+0000","last_change":"2021-03-15T22:33:12.882245+0000","last_active":"2021-03-15T22:35:22.501665+0000","last_peered":"2021-03-15T22:35:22.501665+0000","last_clean":"2021-03-15T22:35:22.501665+0000","last_became_active":"2021-03-15T22:29:51.220261+0000","last_became_peered":"2021-03-15T22:29:51.220261+0000","last_unstale":"2021-03-15T22:35:22.501665+0000","last_undegraded":"2021-03-15T22:35:22.501665+0000","last_fullsized":"2021-03-15T22:35:22.501665+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.882164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.882164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"813","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.523823+0000","last_change":"2021-03-15T22:34:37.787302+0000","last_active":"2021-03-15T22:35:24.523823+0000","last_peered":"2021-03-15T22:35:24.523823+0000","last_clean":"2021-03-15T22:35:24.523823+0000","last_became_active":"2021-03-15T22:33:04.331607+0000","last_became_peered":"2021-03-15T22:33:04.331607+0000","last_unstale":"2021-03-15T22:35:24.523823+0000","last_undegraded":"2021-03-15T22:35:24.523823+0000","last_fullsized":"2021-03-15T22:35:24.523823+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:34:37.787235+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.787235+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"458","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.520593+0000","last_change":"2021-03-15T22:35:17.250496+0000","last_active":"2021-03-15T22:35:24.520593+0000","last_peered":"2021-03-15T22:35:24.520593+0000","last_clean":"2021-03-15T22:35:24.520593+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:35:24.520593+0000","last_undegraded":"2021-03-15T22:35:24.520593+0000","last_fullsized":"2021-03-15T22:35:24.520593+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.250434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.250434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"762","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.524818+0000","last_change":"2021-03-15T22:32:37.690585+0000","last_active":"2021-03-15T22:35:24.524818+0000","last_peered":"2021-03-15T22:35:24.524818+0000","last_clean":"2021-03-15T22:35:24.524818+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:35:24.524818+0000","last_undegraded":"2021-03-15T22:35:24.524818+0000","last_fullsized":"2021-03-15T22:35:24.524818+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:37.690525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:32:37.690525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"969","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428261+0000","last_change":"2021-03-15T22:35:07.430778+0000","last_active":"2021-03-15T22:35:23.428261+0000","last_peered":"2021-03-15T22:35:23.428261+0000","last_clean":"2021-03-15T22:35:23.428261+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:35:23.428261+0000","last_undegraded":"2021-03-15T22:35:23.428261+0000","last_fullsized":"2021-03-15T22:35:23.428261+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.438742+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.438742+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"685","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.517563+0000","last_change":"2021-03-15T22:34:56.210802+0000","last_active":"2021-03-15T22:35:24.517563+0000","last_peered":"2021-03-15T22:35:24.517563+0000","last_clean":"2021-03-15T22:35:24.517563+0000","last_became_active":"2021-03-15T22:34:55.344555+0000","last_became_peered":"2021-03-15T22:34:55.344555+0000","last_unstale":"2021-03-15T22:35:24.517563+0000","last_undegraded":"2021-03-15T22:35:24.517563+0000","last_fullsized":"2021-03-15T22:35:24.517563+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:34:42.447631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.447631+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"355","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.520333+0000","last_change":"2021-03-15T22:35:23.291568+0000","last_active":"2021-03-15T22:35:24.520333+0000","last_peered":"2021-03-15T22:35:24.520333+0000","last_clean":"2021-03-15T22:35:24.520333+0000","last_became_active":"2021-03-15T22:34:55.355119+0000","last_became_peered":"2021-03-15T22:34:55.355119+0000","last_unstale":"2021-03-15T22:35:24.520333+0000","last_undegraded":"2021-03-15T22:35:24.520333+0000","last_fullsized":"2021-03-15T22:35:24.520333+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.291507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.291507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"1029","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.344575+0000","last_change":"2021-03-15T22:34:10.353649+0000","last_active":"2021-03-15T22:35:26.344575+0000","last_peered":"2021-03-15T22:35:26.344575+0000","last_clean":"2021-03-15T22:35:26.344575+0000","last_became_active":"2021-03-15T22:34:10.353314+0000","last_became_peered":"2021-03-15T22:34:10.353314+0000","last_unstale":"2021-03-15T22:35:26.344575+0000","last_undegraded":"2021-03-15T22:35:26.344575+0000","last_fullsized":"2021-03-15T22:35:26.344575+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:43.268612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.268612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"106","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429763+0000","last_change":"2021-03-15T22:33:45.314503+0000","last_active":"2021-03-15T22:35:25.429763+0000","last_peered":"2021-03-15T22:35:25.429763+0000","last_clean":"2021-03-15T22:35:25.429763+0000","last_became_active":"2021-03-15T22:33:45.314042+0000","last_became_peered":"2021-03-15T22:33:45.314042+0000","last_unstale":"2021-03-15T22:35:25.429763+0000","last_undegraded":"2021-03-15T22:35:25.429763+0000","last_fullsized":"2021-03-15T22:35:25.429763+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:08.884878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.884878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"1057","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429784+0000","last_change":"2021-03-15T22:34:42.632034+0000","last_active":"2021-03-15T22:35:25.429784+0000","last_peered":"2021-03-15T22:35:25.429784+0000","last_clean":"2021-03-15T22:35:25.429784+0000","last_became_active":"2021-03-15T22:33:04.340460+0000","last_became_peered":"2021-03-15T22:33:04.340460+0000","last_unstale":"2021-03-15T22:35:25.429784+0000","last_undegraded":"2021-03-15T22:35:25.429784+0000","last_fullsized":"2021-03-15T22:35:25.429784+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:34:42.631962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.631962+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"461","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.519826+0000","last_change":"2021-03-15T22:34:10.327350+0000","last_active":"2021-03-15T22:35:24.519826+0000","last_peered":"2021-03-15T22:35:24.519826+0000","last_clean":"2021-03-15T22:35:24.519826+0000","last_became_active":"2021-03-15T22:34:10.326881+0000","last_became_peered":"2021-03-15T22:34:10.326881+0000","last_unstale":"2021-03-15T22:35:24.519826+0000","last_undegraded":"2021-03-15T22:35:24.519826+0000","last_fullsized":"2021-03-15T22:35:24.519826+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:19.560387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:19.560387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"808'9","reported_seq":"415","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501148+0000","last_change":"2021-03-15T22:32:38.881054+0000","last_active":"2021-03-15T22:35:22.501148+0000","last_peered":"2021-03-15T22:35:22.501148+0000","last_clean":"2021-03-15T22:35:22.501148+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:35:22.501148+0000","last_undegraded":"2021-03-15T22:35:22.501148+0000","last_fullsized":"2021-03-15T22:35:22.501148+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9","last_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_deep_scrub":"808'9","last_deep_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.880993+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"966","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428976+0000","last_change":"2021-03-15T22:34:55.351414+0000","last_active":"2021-03-15T22:35:23.428976+0000","last_peered":"2021-03-15T22:35:23.428976+0000","last_clean":"2021-03-15T22:35:23.428976+0000","last_became_active":"2021-03-15T22:34:55.351042+0000","last_became_peered":"2021-03-15T22:34:55.351042+0000","last_unstale":"2021-03-15T22:35:23.428976+0000","last_undegraded":"2021-03-15T22:35:23.428976+0000","last_fullsized":"2021-03-15T22:35:23.428976+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.516668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.516668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"844'33","reported_seq":"828","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.524637+0000","last_change":"2021-03-15T22:34:55.387712+0000","last_active":"2021-03-15T22:35:24.524637+0000","last_peered":"2021-03-15T22:35:24.524637+0000","last_clean":"2021-03-15T22:35:24.524637+0000","last_became_active":"2021-03-15T22:34:55.351721+0000","last_became_peered":"2021-03-15T22:34:55.351721+0000","last_unstale":"2021-03-15T22:35:24.524637+0000","last_undegraded":"2021-03-15T22:35:24.524637+0000","last_fullsized":"2021-03-15T22:35:24.524637+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"844'33","last_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_deep_scrub":"844'33","last_deep_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.622437+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1005","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430825+0000","last_change":"2021-03-15T22:35:08.565833+0000","last_active":"2021-03-15T22:35:25.430825+0000","last_peered":"2021-03-15T22:35:25.430825+0000","last_clean":"2021-03-15T22:35:25.430825+0000","last_became_active":"2021-03-15T22:34:55.340440+0000","last_became_peered":"2021-03-15T22:34:55.340440+0000","last_unstale":"2021-03-15T22:35:25.430825+0000","last_undegraded":"2021-03-15T22:35:25.430825+0000","last_fullsized":"2021-03-15T22:35:25.430825+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:41.407324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.407324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"941'17","reported_seq":"490","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.429361+0000","last_change":"2021-03-15T22:34:55.473459+0000","last_active":"2021-03-15T22:35:23.429361+0000","last_peered":"2021-03-15T22:35:23.429361+0000","last_clean":"2021-03-15T22:35:23.429361+0000","last_became_active":"2021-03-15T22:34:55.351682+0000","last_became_peered":"2021-03-15T22:34:55.351682+0000","last_unstale":"2021-03-15T22:35:23.429361+0000","last_undegraded":"2021-03-15T22:35:23.429361+0000","last_fullsized":"2021-03-15T22:35:23.429361+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"896'16","last_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_deep_scrub":"896'16","last_deep_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_clean_scrub_stamp":"2021-03-15T22:34:19.400191+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"461","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.518816+0000","last_change":"2021-03-15T22:34:10.347844+0000","last_active":"2021-03-15T22:35:24.518816+0000","last_peered":"2021-03-15T22:35:24.518816+0000","last_clean":"2021-03-15T22:35:24.518816+0000","last_became_active":"2021-03-15T22:34:10.347512+0000","last_became_peered":"2021-03-15T22:34:10.347512+0000","last_unstale":"2021-03-15T22:35:24.518816+0000","last_undegraded":"2021-03-15T22:35:24.518816+0000","last_fullsized":"2021-03-15T22:35:24.518816+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:03.502854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:32:03.502854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"361","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501004+0000","last_change":"2021-03-15T22:32:41.879032+0000","last_active":"2021-03-15T22:35:22.501004+0000","last_peered":"2021-03-15T22:35:22.501004+0000","last_clean":"2021-03-15T22:35:22.501004+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:35:22.501004+0000","last_undegraded":"2021-03-15T22:35:22.501004+0000","last_fullsized":"2021-03-15T22:35:22.501004+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:32:41.878949+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.878949+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"52","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.524415+0000","last_change":"2021-03-15T22:34:43.345092+0000","last_active":"2021-03-15T22:35:24.524415+0000","last_peered":"2021-03-15T22:35:24.524415+0000","last_clean":"2021-03-15T22:35:24.524415+0000","last_became_active":"2021-03-15T22:34:43.344790+0000","last_became_peered":"2021-03-15T22:34:43.344790+0000","last_unstale":"2021-03-15T22:35:24.524415+0000","last_undegraded":"2021-03-15T22:35:24.524415+0000","last_fullsized":"2021-03-15T22:35:24.524415+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.621539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"43","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.289480+0000","last_change":"2021-03-15T22:35:20.381407+0000","last_active":"2021-03-15T22:35:23.289480+0000","last_peered":"2021-03-15T22:35:23.289480+0000","last_clean":"2021-03-15T22:35:23.289480+0000","last_became_active":"2021-03-15T22:34:55.348663+0000","last_became_peered":"2021-03-15T22:34:55.348663+0000","last_unstale":"2021-03-15T22:35:23.289480+0000","last_undegraded":"2021-03-15T22:35:23.289480+0000","last_fullsized":"2021-03-15T22:35:23.289480+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.381352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"959","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428710+0000","last_change":"2021-03-15T22:33:21.545910+0000","last_active":"2021-03-15T22:35:23.428710+0000","last_peered":"2021-03-15T22:35:23.428710+0000","last_clean":"2021-03-15T22:35:23.428710+0000","last_became_active":"2021-03-15T22:33:04.330195+0000","last_became_peered":"2021-03-15T22:33:04.330195+0000","last_unstale":"2021-03-15T22:35:23.428710+0000","last_undegraded":"2021-03-15T22:35:23.428710+0000","last_fullsized":"2021-03-15T22:35:23.428710+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.545826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.545826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"497","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.519085+0000","last_change":"2021-03-15T22:34:37.435070+0000","last_active":"2021-03-15T22:35:24.519085+0000","last_peered":"2021-03-15T22:35:24.519085+0000","last_clean":"2021-03-15T22:35:24.519085+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:35:24.519085+0000","last_undegraded":"2021-03-15T22:35:24.519085+0000","last_fullsized":"2021-03-15T22:35:24.519085+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:34:37.435006+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.435006+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"770","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501669+0000","last_change":"2021-03-15T22:35:20.884067+0000","last_active":"2021-03-15T22:35:22.501669+0000","last_peered":"2021-03-15T22:35:22.501669+0000","last_clean":"2021-03-15T22:35:22.501669+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:35:22.501669+0000","last_undegraded":"2021-03-15T22:35:22.501669+0000","last_fullsized":"2021-03-15T22:35:22.501669+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.883990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.883990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"953","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.429523+0000","last_change":"2021-03-15T22:33:19.496857+0000","last_active":"2021-03-15T22:35:23.429523+0000","last_peered":"2021-03-15T22:35:23.429523+0000","last_clean":"2021-03-15T22:35:23.429523+0000","last_became_active":"2021-03-15T22:33:04.331013+0000","last_became_peered":"2021-03-15T22:33:04.331013+0000","last_unstale":"2021-03-15T22:35:23.429523+0000","last_undegraded":"2021-03-15T22:35:23.429523+0000","last_fullsized":"2021-03-15T22:35:23.429523+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:19.496753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:19.496753+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"149","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.517369+0000","last_change":"2021-03-15T22:34:07.203702+0000","last_active":"2021-03-15T22:35:24.517369+0000","last_peered":"2021-03-15T22:35:24.517369+0000","last_clean":"2021-03-15T22:35:24.517369+0000","last_became_active":"2021-03-15T22:33:04.336886+0000","last_became_peered":"2021-03-15T22:33:04.336886+0000","last_unstale":"2021-03-15T22:35:24.517369+0000","last_undegraded":"2021-03-15T22:35:24.517369+0000","last_fullsized":"2021-03-15T22:35:24.517369+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.203639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:39.697935+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.203639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"317","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.429849+0000","last_change":"2021-03-15T22:34:55.387006+0000","last_active":"2021-03-15T22:35:23.429849+0000","last_peered":"2021-03-15T22:35:23.429849+0000","last_clean":"2021-03-15T22:35:23.429849+0000","last_became_active":"2021-03-15T22:34:55.351977+0000","last_became_peered":"2021-03-15T22:34:55.351977+0000","last_unstale":"2021-03-15T22:35:23.429849+0000","last_undegraded":"2021-03-15T22:35:23.429849+0000","last_fullsized":"2021-03-15T22:35:23.429849+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:34:36.438541+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:34:36.438541+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"969'1370","reported_seq":"4404","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.346555+0000","last_change":"2021-03-15T22:35:24.545383+0000","last_active":"2021-03-15T22:35:26.346555+0000","last_peered":"2021-03-15T22:35:26.346555+0000","last_clean":"2021-03-15T22:35:26.346555+0000","last_became_active":"2021-03-15T22:33:08.355490+0000","last_became_peered":"2021-03-15T22:33:08.355490+0000","last_unstale":"2021-03-15T22:35:26.346555+0000","last_undegraded":"2021-03-15T22:35:26.346555+0000","last_fullsized":"2021-03-15T22:35:26.346555+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'1350","last_scrub_stamp":"2021-03-15T22:35:16.036089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.036089+0000","log_size":1370,"ondisk_log_size":1370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42240460,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":467,"num_read_kb":519756,"num_write":317,"num_write_kb":115324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":84869361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"910","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501428+0000","last_change":"2021-03-15T22:33:42.741529+0000","last_active":"2021-03-15T22:35:22.501428+0000","last_peered":"2021-03-15T22:35:22.501428+0000","last_clean":"2021-03-15T22:35:22.501428+0000","last_became_active":"2021-03-15T22:33:04.340352+0000","last_became_peered":"2021-03-15T22:33:04.340352+0000","last_unstale":"2021-03-15T22:35:22.501428+0000","last_undegraded":"2021-03-15T22:35:22.501428+0000","last_fullsized":"2021-03-15T22:35:22.501428+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:42.741451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:42.741451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"874'4","reported_seq":"976","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.429755+0000","last_change":"2021-03-15T22:33:44.498327+0000","last_active":"2021-03-15T22:35:23.429755+0000","last_peered":"2021-03-15T22:35:23.429755+0000","last_clean":"2021-03-15T22:35:23.429755+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:35:23.429755+0000","last_undegraded":"2021-03-15T22:35:23.429755+0000","last_fullsized":"2021-03-15T22:35:23.429755+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"874'4","last_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_deep_scrub":"874'4","last_deep_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.498263+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"437","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.500565+0000","last_change":"2021-03-15T22:33:38.759477+0000","last_active":"2021-03-15T22:35:22.500565+0000","last_peered":"2021-03-15T22:35:22.500565+0000","last_clean":"2021-03-15T22:35:22.500565+0000","last_became_active":"2021-03-15T22:33:04.326070+0000","last_became_peered":"2021-03-15T22:33:04.326070+0000","last_unstale":"2021-03-15T22:35:22.500565+0000","last_undegraded":"2021-03-15T22:35:22.500565+0000","last_fullsized":"2021-03-15T22:35:22.500565+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:33:38.759395+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.759395+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"841'14","reported_seq":"1231","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428847+0000","last_change":"2021-03-15T22:33:43.476783+0000","last_active":"2021-03-15T22:35:23.428847+0000","last_peered":"2021-03-15T22:35:23.428847+0000","last_clean":"2021-03-15T22:35:23.428847+0000","last_became_active":"2021-03-15T22:33:08.354031+0000","last_became_peered":"2021-03-15T22:33:08.354031+0000","last_unstale":"2021-03-15T22:35:23.428847+0000","last_undegraded":"2021-03-15T22:35:23.428847+0000","last_fullsized":"2021-03-15T22:35:23.428847+0000","mapping_epoch":837,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'14","last_scrub_stamp":"2021-03-15T22:33:43.476701+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.476701+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"969'1962","reported_seq":"5267","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.335611+0000","last_change":"2021-03-15T22:35:24.534585+0000","last_active":"2021-03-15T22:35:25.335611+0000","last_peered":"2021-03-15T22:35:25.335611+0000","last_clean":"2021-03-15T22:35:25.335611+0000","last_became_active":"2021-03-15T22:34:58.336162+0000","last_became_peered":"2021-03-15T22:34:58.336162+0000","last_unstale":"2021-03-15T22:35:25.335611+0000","last_undegraded":"2021-03-15T22:35:25.335611+0000","last_fullsized":"2021-03-15T22:35:25.335611+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1312","last_scrub_stamp":"2021-03-15T22:32:05.639057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.639057+0000","log_size":1962,"ondisk_log_size":1962,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":67977662,"num_objects":33,"num_object_clones":28,"num_object_copies":99,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":33,"num_whiteouts":2,"num_read":658,"num_read_kb":625153,"num_write":506,"num_write_kb":199898,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":198498497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"398","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.524794+0000","last_change":"2021-03-15T22:32:42.621654+0000","last_active":"2021-03-15T22:35:24.524794+0000","last_peered":"2021-03-15T22:35:24.524794+0000","last_clean":"2021-03-15T22:35:24.524794+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:35:24.524794+0000","last_undegraded":"2021-03-15T22:35:24.524794+0000","last_fullsized":"2021-03-15T22:35:24.524794+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:42.621594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:42.621594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"943","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.429638+0000","last_change":"2021-03-15T22:33:22.542596+0000","last_active":"2021-03-15T22:35:23.429638+0000","last_peered":"2021-03-15T22:35:23.429638+0000","last_clean":"2021-03-15T22:35:23.429638+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:35:23.429638+0000","last_undegraded":"2021-03-15T22:35:23.429638+0000","last_fullsized":"2021-03-15T22:35:23.429638+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.542506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.542506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"1180","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430408+0000","last_change":"2021-03-15T22:34:11.031994+0000","last_active":"2021-03-15T22:35:25.430408+0000","last_peered":"2021-03-15T22:35:25.430408+0000","last_clean":"2021-03-15T22:35:25.430408+0000","last_became_active":"2021-03-15T22:34:10.342120+0000","last_became_peered":"2021-03-15T22:34:10.342120+0000","last_unstale":"2021-03-15T22:35:25.430408+0000","last_undegraded":"2021-03-15T22:35:25.430408+0000","last_fullsized":"2021-03-15T22:35:25.430408+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:34:07.452572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.452572+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"682","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.520667+0000","last_change":"2021-03-15T22:34:55.356651+0000","last_active":"2021-03-15T22:35:24.520667+0000","last_peered":"2021-03-15T22:35:24.520667+0000","last_clean":"2021-03-15T22:35:24.520667+0000","last_became_active":"2021-03-15T22:34:55.356328+0000","last_became_peered":"2021-03-15T22:34:55.356328+0000","last_unstale":"2021-03-15T22:35:24.520667+0000","last_undegraded":"2021-03-15T22:35:24.520667+0000","last_fullsized":"2021-03-15T22:35:24.520667+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"969'548","reported_seq":"3294","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.339701+0000","last_change":"2021-03-15T22:35:24.536306+0000","last_active":"2021-03-15T22:35:25.339701+0000","last_peered":"2021-03-15T22:35:25.339701+0000","last_clean":"2021-03-15T22:35:25.339701+0000","last_became_active":"2021-03-15T22:34:13.324667+0000","last_became_peered":"2021-03-15T22:34:13.324667+0000","last_unstale":"2021-03-15T22:35:25.339701+0000","last_undegraded":"2021-03-15T22:35:25.339701+0000","last_fullsized":"2021-03-15T22:35:25.339701+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'446","last_scrub_stamp":"2021-03-15T22:33:39.355306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.355306+0000","log_size":548,"ondisk_log_size":548,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17801208,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":131,"num_read_kb":101310,"num_write":130,"num_write_kb":42276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":23698344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"1011","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428933+0000","last_change":"2021-03-15T22:34:55.354150+0000","last_active":"2021-03-15T22:35:23.428933+0000","last_peered":"2021-03-15T22:35:23.428933+0000","last_clean":"2021-03-15T22:35:23.428933+0000","last_became_active":"2021-03-15T22:34:55.353827+0000","last_became_peered":"2021-03-15T22:34:55.353827+0000","last_unstale":"2021-03-15T22:35:23.428933+0000","last_undegraded":"2021-03-15T22:35:23.428933+0000","last_fullsized":"2021-03-15T22:35:23.428933+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"38","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.289192+0000","last_change":"2021-03-15T22:34:55.354467+0000","last_active":"2021-03-15T22:35:23.289192+0000","last_peered":"2021-03-15T22:35:23.289192+0000","last_clean":"2021-03-15T22:35:23.289192+0000","last_became_active":"2021-03-15T22:34:55.353966+0000","last_became_peered":"2021-03-15T22:34:55.353966+0000","last_unstale":"2021-03-15T22:35:23.289192+0000","last_undegraded":"2021-03-15T22:35:23.289192+0000","last_fullsized":"2021-03-15T22:35:23.289192+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.809863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.809863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"1003","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.429784+0000","last_change":"2021-03-15T22:35:23.428929+0000","last_active":"2021-03-15T22:35:23.429784+0000","last_peered":"2021-03-15T22:35:23.429784+0000","last_clean":"2021-03-15T22:35:23.429784+0000","last_became_active":"2021-03-15T22:33:04.330771+0000","last_became_peered":"2021-03-15T22:33:04.330771+0000","last_unstale":"2021-03-15T22:35:23.429784+0000","last_undegraded":"2021-03-15T22:35:23.429784+0000","last_fullsized":"2021-03-15T22:35:23.429784+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.428842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.428842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"214","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.517695+0000","last_change":"2021-03-15T22:34:10.249538+0000","last_active":"2021-03-15T22:35:24.517695+0000","last_peered":"2021-03-15T22:35:24.517695+0000","last_clean":"2021-03-15T22:35:24.517695+0000","last_became_active":"2021-03-15T22:33:45.318867+0000","last_became_peered":"2021-03-15T22:33:45.318867+0000","last_unstale":"2021-03-15T22:35:24.517695+0000","last_undegraded":"2021-03-15T22:35:24.517695+0000","last_fullsized":"2021-03-15T22:35:24.517695+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:34:10.249444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:10.249444+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":2928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"966'1199","reported_seq":"4020","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.289221+0000","last_change":"2021-03-15T22:35:21.352073+0000","last_active":"2021-03-15T22:35:23.289221+0000","last_peered":"2021-03-15T22:35:23.289221+0000","last_clean":"2021-03-15T22:35:23.289221+0000","last_became_active":"2021-03-15T22:34:59.339587+0000","last_became_peered":"2021-03-15T22:34:59.339587+0000","last_unstale":"2021-03-15T22:35:23.289221+0000","last_undegraded":"2021-03-15T22:35:23.289221+0000","last_fullsized":"2021-03-15T22:35:23.289221+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'1086","last_scrub_stamp":"2021-03-15T22:34:41.436839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.436839+0000","log_size":1199,"ondisk_log_size":1199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":56108130,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":1,"num_read":409,"num_read_kb":484935,"num_write":281,"num_write_kb":101057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":205148835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"239","length":"1"},{"start":"266","length":"1"},{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2eb","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"},{"start":"313","length":"1"},{"start":"31d","length":"1"},{"start":"324","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"472","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.520381+0000","last_change":"2021-03-15T22:34:55.356122+0000","last_active":"2021-03-15T22:35:24.520381+0000","last_peered":"2021-03-15T22:35:24.520381+0000","last_clean":"2021-03-15T22:35:24.520381+0000","last_became_active":"2021-03-15T22:34:55.355571+0000","last_became_peered":"2021-03-15T22:34:55.355571+0000","last_unstale":"2021-03-15T22:35:24.520381+0000","last_undegraded":"2021-03-15T22:35:24.520381+0000","last_fullsized":"2021-03-15T22:35:24.520381+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"804","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429822+0000","last_change":"2021-03-15T22:34:55.343547+0000","last_active":"2021-03-15T22:35:25.429822+0000","last_peered":"2021-03-15T22:35:25.429822+0000","last_clean":"2021-03-15T22:35:25.429822+0000","last_became_active":"2021-03-15T22:34:55.342808+0000","last_became_peered":"2021-03-15T22:34:55.342808+0000","last_unstale":"2021-03-15T22:35:25.429822+0000","last_undegraded":"2021-03-15T22:35:25.429822+0000","last_fullsized":"2021-03-15T22:35:25.429822+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:14.888119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.888119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"368","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501138+0000","last_change":"2021-03-15T22:34:07.807466+0000","last_active":"2021-03-15T22:35:22.501138+0000","last_peered":"2021-03-15T22:35:22.501138+0000","last_clean":"2021-03-15T22:35:22.501138+0000","last_became_active":"2021-03-15T22:33:04.333291+0000","last_became_peered":"2021-03-15T22:33:04.333291+0000","last_unstale":"2021-03-15T22:35:22.501138+0000","last_undegraded":"2021-03-15T22:35:22.501138+0000","last_fullsized":"2021-03-15T22:35:22.501138+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.807411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.807411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"512","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.519187+0000","last_change":"2021-03-15T22:33:46.129541+0000","last_active":"2021-03-15T22:35:24.519187+0000","last_peered":"2021-03-15T22:35:24.519187+0000","last_clean":"2021-03-15T22:35:24.519187+0000","last_became_active":"2021-03-15T22:33:45.326747+0000","last_became_peered":"2021-03-15T22:33:45.326747+0000","last_unstale":"2021-03-15T22:35:24.519187+0000","last_undegraded":"2021-03-15T22:35:24.519187+0000","last_fullsized":"2021-03-15T22:35:24.519187+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:32:33.701675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.701675+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":2682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"969'1368","reported_seq":"4874","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.601259+0000","last_change":"2021-03-15T22:35:24.527307+0000","last_active":"2021-03-15T22:35:25.601259+0000","last_peered":"2021-03-15T22:35:25.601259+0000","last_clean":"2021-03-15T22:35:25.601259+0000","last_became_active":"2021-03-15T22:34:46.338888+0000","last_became_peered":"2021-03-15T22:34:46.338888+0000","last_unstale":"2021-03-15T22:35:25.601259+0000","last_undegraded":"2021-03-15T22:35:25.601259+0000","last_fullsized":"2021-03-15T22:35:25.601259+0000","mapping_epoch":930,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'1253","last_scrub_stamp":"2021-03-15T22:34:41.490879+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.490879+0000","log_size":1368,"ondisk_log_size":1368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42238984,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":310,"num_read_kb":250585,"num_write":334,"num_write_kb":118965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":229919087,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"52","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.524068+0000","last_change":"2021-03-15T22:34:43.342616+0000","last_active":"2021-03-15T22:35:24.524068+0000","last_peered":"2021-03-15T22:35:24.524068+0000","last_clean":"2021-03-15T22:35:24.524068+0000","last_became_active":"2021-03-15T22:34:43.342089+0000","last_became_peered":"2021-03-15T22:34:43.342089+0000","last_unstale":"2021-03-15T22:35:24.524068+0000","last_undegraded":"2021-03-15T22:35:24.524068+0000","last_fullsized":"2021-03-15T22:35:24.524068+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"14","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428435+0000","last_change":"2021-03-15T22:35:19.502186+0000","last_active":"2021-03-15T22:35:23.428435+0000","last_peered":"2021-03-15T22:35:23.428435+0000","last_clean":"2021-03-15T22:35:23.428435+0000","last_became_active":"2021-03-15T22:35:19.501783+0000","last_became_peered":"2021-03-15T22:35:19.501783+0000","last_unstale":"2021-03-15T22:35:23.428435+0000","last_undegraded":"2021-03-15T22:35:23.428435+0000","last_fullsized":"2021-03-15T22:35:23.428435+0000","mapping_epoch":963,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":964,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.831369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.831369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"168","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429475+0000","last_change":"2021-03-15T22:33:04.336661+0000","last_active":"2021-03-15T22:35:25.429475+0000","last_peered":"2021-03-15T22:35:25.429475+0000","last_clean":"2021-03-15T22:35:25.429475+0000","last_became_active":"2021-03-15T22:33:04.335914+0000","last_became_peered":"2021-03-15T22:33:04.335914+0000","last_unstale":"2021-03-15T22:35:25.429475+0000","last_undegraded":"2021-03-15T22:35:25.429475+0000","last_fullsized":"2021-03-15T22:35:25.429475+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.051962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.051962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1339","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.346016+0000","last_change":"2021-03-15T22:34:55.382040+0000","last_active":"2021-03-15T22:35:26.346016+0000","last_peered":"2021-03-15T22:35:26.346016+0000","last_clean":"2021-03-15T22:35:26.346016+0000","last_became_active":"2021-03-15T22:34:55.350419+0000","last_became_peered":"2021-03-15T22:34:55.350419+0000","last_unstale":"2021-03-15T22:35:26.346016+0000","last_undegraded":"2021-03-15T22:35:26.346016+0000","last_fullsized":"2021-03-15T22:35:26.346016+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:33:38.133617+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.133617+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"434","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.519707+0000","last_change":"2021-03-15T22:33:15.492116+0000","last_active":"2021-03-15T22:35:24.519707+0000","last_peered":"2021-03-15T22:35:24.519707+0000","last_clean":"2021-03-15T22:35:24.519707+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:35:24.519707+0000","last_undegraded":"2021-03-15T22:35:24.519707+0000","last_fullsized":"2021-03-15T22:35:24.519707+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.492014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.492014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"969'1279","reported_seq":"3898","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.339612+0000","last_change":"2021-03-15T22:35:24.529930+0000","last_active":"2021-03-15T22:35:25.339612+0000","last_peered":"2021-03-15T22:35:25.339612+0000","last_clean":"2021-03-15T22:35:25.339612+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:35:25.339612+0000","last_undegraded":"2021-03-15T22:35:25.339612+0000","last_fullsized":"2021-03-15T22:35:25.339612+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"805'997","last_scrub_stamp":"2021-03-15T22:32:36.795301+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.795301+0000","log_size":1279,"ondisk_log_size":1279,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":43260214,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":377,"num_read_kb":377584,"num_write":287,"num_write_kb":109592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"994","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428123+0000","last_change":"2021-03-15T22:34:34.631500+0000","last_active":"2021-03-15T22:35:23.428123+0000","last_peered":"2021-03-15T22:35:23.428123+0000","last_clean":"2021-03-15T22:35:23.428123+0000","last_became_active":"2021-03-15T22:34:34.631216+0000","last_became_peered":"2021-03-15T22:34:34.631216+0000","last_unstale":"2021-03-15T22:35:23.428123+0000","last_undegraded":"2021-03-15T22:35:23.428123+0000","last_fullsized":"2021-03-15T22:35:23.428123+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:32.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"1003","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.344806+0000","last_change":"2021-03-15T22:34:55.347340+0000","last_active":"2021-03-15T22:35:26.344806+0000","last_peered":"2021-03-15T22:35:26.344806+0000","last_clean":"2021-03-15T22:35:26.344806+0000","last_became_active":"2021-03-15T22:34:55.347065+0000","last_became_peered":"2021-03-15T22:34:55.347065+0000","last_unstale":"2021-03-15T22:35:26.344806+0000","last_undegraded":"2021-03-15T22:35:26.344806+0000","last_fullsized":"2021-03-15T22:35:26.344806+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:10.222521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:10.222521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"43","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.289244+0000","last_change":"2021-03-15T22:35:17.301942+0000","last_active":"2021-03-15T22:35:23.289244+0000","last_peered":"2021-03-15T22:35:23.289244+0000","last_clean":"2021-03-15T22:35:23.289244+0000","last_became_active":"2021-03-15T22:34:55.355485+0000","last_became_peered":"2021-03-15T22:34:55.355485+0000","last_unstale":"2021-03-15T22:35:23.289244+0000","last_undegraded":"2021-03-15T22:35:23.289244+0000","last_fullsized":"2021-03-15T22:35:23.289244+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.301892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.301892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"451","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501229+0000","last_change":"2021-03-15T22:34:10.343282+0000","last_active":"2021-03-15T22:35:22.501229+0000","last_peered":"2021-03-15T22:35:22.501229+0000","last_clean":"2021-03-15T22:35:22.501229+0000","last_became_active":"2021-03-15T22:34:10.322666+0000","last_became_peered":"2021-03-15T22:34:10.322666+0000","last_unstale":"2021-03-15T22:35:22.501229+0000","last_undegraded":"2021-03-15T22:35:22.501229+0000","last_fullsized":"2021-03-15T22:35:22.501229+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:33:40.214707+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.214707+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"38","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.289875+0000","last_change":"2021-03-15T22:34:55.353621+0000","last_active":"2021-03-15T22:35:23.289875+0000","last_peered":"2021-03-15T22:35:23.289875+0000","last_clean":"2021-03-15T22:35:23.289875+0000","last_became_active":"2021-03-15T22:34:55.353316+0000","last_became_peered":"2021-03-15T22:34:55.353316+0000","last_unstale":"2021-03-15T22:35:23.289875+0000","last_undegraded":"2021-03-15T22:35:23.289875+0000","last_fullsized":"2021-03-15T22:35:23.289875+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:38.907011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.907011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"970'3101","reported_seq":"6590","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.365995+0000","last_change":"2021-03-15T22:35:24.538222+0000","last_active":"2021-03-15T22:35:25.365995+0000","last_peered":"2021-03-15T22:35:25.365995+0000","last_clean":"2021-03-15T22:35:25.365995+0000","last_became_active":"2021-03-15T22:34:13.331560+0000","last_became_peered":"2021-03-15T22:34:13.331560+0000","last_unstale":"2021-03-15T22:35:25.365995+0000","last_undegraded":"2021-03-15T22:35:25.365995+0000","last_fullsized":"2021-03-15T22:35:25.365995+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'2007","last_scrub_stamp":"2021-03-15T22:31:02.050395+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.050395+0000","log_size":3101,"ondisk_log_size":3101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":63093949,"num_objects":34,"num_object_clones":27,"num_object_copies":102,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":34,"num_whiteouts":3,"num_read":864,"num_read_kb":897095,"num_write":654,"num_write_kb":261696,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":264088494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"352","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501342+0000","last_change":"2021-03-15T22:34:34.635632+0000","last_active":"2021-03-15T22:35:22.501342+0000","last_peered":"2021-03-15T22:35:22.501342+0000","last_clean":"2021-03-15T22:35:22.501342+0000","last_became_active":"2021-03-15T22:34:34.635223+0000","last_became_peered":"2021-03-15T22:34:34.635223+0000","last_unstale":"2021-03-15T22:35:22.501342+0000","last_undegraded":"2021-03-15T22:35:22.501342+0000","last_fullsized":"2021-03-15T22:35:22.501342+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.761137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"982","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.428284+0000","last_change":"2021-03-15T22:35:19.456953+0000","last_active":"2021-03-15T22:35:23.428284+0000","last_peered":"2021-03-15T22:35:23.428284+0000","last_clean":"2021-03-15T22:35:23.428284+0000","last_became_active":"2021-03-15T22:33:04.337709+0000","last_became_peered":"2021-03-15T22:33:04.337709+0000","last_unstale":"2021-03-15T22:35:23.428284+0000","last_undegraded":"2021-03-15T22:35:23.428284+0000","last_fullsized":"2021-03-15T22:35:23.428284+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"52","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.523427+0000","last_change":"2021-03-15T22:34:43.341562+0000","last_active":"2021-03-15T22:35:24.523427+0000","last_peered":"2021-03-15T22:35:24.523427+0000","last_clean":"2021-03-15T22:35:24.523427+0000","last_became_active":"2021-03-15T22:34:43.340955+0000","last_became_peered":"2021-03-15T22:34:43.340955+0000","last_unstale":"2021-03-15T22:35:24.523427+0000","last_undegraded":"2021-03-15T22:35:24.523427+0000","last_fullsized":"2021-03-15T22:35:24.523427+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.843478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.843478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"169","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.518292+0000","last_change":"2021-03-15T22:34:09.215097+0000","last_active":"2021-03-15T22:35:24.518292+0000","last_peered":"2021-03-15T22:35:24.518292+0000","last_clean":"2021-03-15T22:35:24.518292+0000","last_became_active":"2021-03-15T22:33:04.330531+0000","last_became_peered":"2021-03-15T22:33:04.330531+0000","last_unstale":"2021-03-15T22:35:24.518292+0000","last_undegraded":"2021-03-15T22:35:24.518292+0000","last_fullsized":"2021-03-15T22:35:24.518292+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:34:09.215034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:09.215034+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"1003","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.429657+0000","last_change":"2021-03-15T22:35:22.409047+0000","last_active":"2021-03-15T22:35:23.429657+0000","last_peered":"2021-03-15T22:35:23.429657+0000","last_clean":"2021-03-15T22:35:23.429657+0000","last_became_active":"2021-03-15T22:34:55.352186+0000","last_became_peered":"2021-03-15T22:34:55.352186+0000","last_unstale":"2021-03-15T22:35:23.429657+0000","last_undegraded":"2021-03-15T22:35:23.429657+0000","last_fullsized":"2021-03-15T22:35:23.429657+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.408965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.408965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"966'953","reported_seq":"3985","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.429381+0000","last_change":"2021-03-15T22:35:21.355296+0000","last_active":"2021-03-15T22:35:23.429381+0000","last_peered":"2021-03-15T22:35:23.429381+0000","last_clean":"2021-03-15T22:35:23.429381+0000","last_became_active":"2021-03-15T22:35:03.346751+0000","last_became_peered":"2021-03-15T22:35:03.346751+0000","last_unstale":"2021-03-15T22:35:23.429381+0000","last_undegraded":"2021-03-15T22:35:23.429381+0000","last_fullsized":"2021-03-15T22:35:23.429381+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":0,"last_scrub":"891'819","last_scrub_stamp":"2021-03-15T22:34:08.508824+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.508824+0000","log_size":953,"ondisk_log_size":953,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32591193,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":2,"num_read":256,"num_read_kb":260779,"num_write":216,"num_write_kb":96405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":109648904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"239","length":"1"},{"start":"266","length":"1"},{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2eb","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"},{"start":"313","length":"1"},{"start":"31d","length":"1"},{"start":"324","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"769","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501183+0000","last_change":"2021-03-15T22:34:43.344711+0000","last_active":"2021-03-15T22:35:22.501183+0000","last_peered":"2021-03-15T22:35:22.501183+0000","last_clean":"2021-03-15T22:35:22.501183+0000","last_became_active":"2021-03-15T22:34:43.344418+0000","last_became_peered":"2021-03-15T22:34:43.344418+0000","last_unstale":"2021-03-15T22:35:22.501183+0000","last_undegraded":"2021-03-15T22:35:22.501183+0000","last_fullsized":"2021-03-15T22:35:22.501183+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"466","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.520410+0000","last_change":"2021-03-15T22:34:35.410541+0000","last_active":"2021-03-15T22:35:24.520410+0000","last_peered":"2021-03-15T22:35:24.520410+0000","last_clean":"2021-03-15T22:35:24.520410+0000","last_became_active":"2021-03-15T22:34:10.321100+0000","last_became_peered":"2021-03-15T22:34:10.321100+0000","last_unstale":"2021-03-15T22:35:24.520410+0000","last_undegraded":"2021-03-15T22:35:24.520410+0000","last_fullsized":"2021-03-15T22:35:24.520410+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:35.410485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:35.410485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"2011","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:24.095443+0000","last_change":"2021-03-15T22:35:10.452906+0000","last_active":"2021-03-15T22:35:24.095443+0000","last_peered":"2021-03-15T22:35:24.095443+0000","last_clean":"2021-03-15T22:35:24.095443+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:35:24.095443+0000","last_undegraded":"2021-03-15T22:35:24.095443+0000","last_fullsized":"2021-03-15T22:35:24.095443+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:35:10.452809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.452809+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1034,"num_read_kb":1029,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"564","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.519716+0000","last_change":"2021-03-15T22:34:55.414460+0000","last_active":"2021-03-15T22:35:24.519716+0000","last_peered":"2021-03-15T22:35:24.519716+0000","last_clean":"2021-03-15T22:35:24.519716+0000","last_became_active":"2021-03-15T22:34:55.359023+0000","last_became_peered":"2021-03-15T22:34:55.359023+0000","last_unstale":"2021-03-15T22:35:24.519716+0000","last_undegraded":"2021-03-15T22:35:24.519716+0000","last_fullsized":"2021-03-15T22:35:24.519716+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:34:40.472800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.472800+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"357","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501347+0000","last_change":"2021-03-15T22:34:40.799217+0000","last_active":"2021-03-15T22:35:22.501347+0000","last_peered":"2021-03-15T22:35:22.501347+0000","last_clean":"2021-03-15T22:35:22.501347+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:35:22.501347+0000","last_undegraded":"2021-03-15T22:35:22.501347+0000","last_fullsized":"2021-03-15T22:35:22.501347+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.799153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.799153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"966'3227","reported_seq":"7240","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.500753+0000","last_change":"2021-03-15T22:35:21.372570+0000","last_active":"2021-03-15T22:35:22.500753+0000","last_peered":"2021-03-15T22:35:22.500753+0000","last_clean":"2021-03-15T22:35:22.500753+0000","last_became_active":"2021-03-15T22:35:17.487261+0000","last_became_peered":"2021-03-15T22:35:17.487261+0000","last_unstale":"2021-03-15T22:35:22.500753+0000","last_undegraded":"2021-03-15T22:35:22.500753+0000","last_fullsized":"2021-03-15T22:35:22.500753+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'2597","last_scrub_stamp":"2021-03-15T22:32:30.909661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.909661+0000","log_size":3227,"ondisk_log_size":3227,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":91187612,"num_objects":41,"num_object_clones":34,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":2,"num_read":1162,"num_read_kb":1241810,"num_write":751,"num_write_kb":309878,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":721,"num_bytes_recovered":351021959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"239","length":"1"},{"start":"266","length":"1"},{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2eb","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"},{"start":"313","length":"1"},{"start":"31d","length":"1"},{"start":"324","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"1003","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.344908+0000","last_change":"2021-03-15T22:34:55.346293+0000","last_active":"2021-03-15T22:35:26.344908+0000","last_peered":"2021-03-15T22:35:26.344908+0000","last_clean":"2021-03-15T22:35:26.344908+0000","last_became_active":"2021-03-15T22:34:55.345948+0000","last_became_peered":"2021-03-15T22:34:55.345948+0000","last_unstale":"2021-03-15T22:35:26.344908+0000","last_undegraded":"2021-03-15T22:35:26.344908+0000","last_fullsized":"2021-03-15T22:35:26.344908+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.701955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.701955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"1021","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.345356+0000","last_change":"2021-03-15T22:34:55.346381+0000","last_active":"2021-03-15T22:35:26.345356+0000","last_peered":"2021-03-15T22:35:26.345356+0000","last_clean":"2021-03-15T22:35:26.345356+0000","last_became_active":"2021-03-15T22:34:55.346048+0000","last_became_peered":"2021-03-15T22:34:55.346048+0000","last_unstale":"2021-03-15T22:35:26.345356+0000","last_undegraded":"2021-03-15T22:35:26.345356+0000","last_fullsized":"2021-03-15T22:35:26.345356+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:48.037748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_clean_scrub_stamp":"2021-03-15T22:34:48.037748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"514","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.519026+0000","last_change":"2021-03-15T22:35:21.314851+0000","last_active":"2021-03-15T22:35:24.519026+0000","last_peered":"2021-03-15T22:35:24.519026+0000","last_clean":"2021-03-15T22:35:24.519026+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:35:24.519026+0000","last_undegraded":"2021-03-15T22:35:24.519026+0000","last_fullsized":"2021-03-15T22:35:24.519026+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:35:21.314763+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.314763+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"149","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.517634+0000","last_change":"2021-03-15T22:33:22.282663+0000","last_active":"2021-03-15T22:35:24.517634+0000","last_peered":"2021-03-15T22:35:24.517634+0000","last_clean":"2021-03-15T22:35:24.517634+0000","last_became_active":"2021-03-15T22:33:04.337244+0000","last_became_peered":"2021-03-15T22:33:04.337244+0000","last_unstale":"2021-03-15T22:35:24.517634+0000","last_undegraded":"2021-03-15T22:35:24.517634+0000","last_fullsized":"2021-03-15T22:35:24.517634+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.282586+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.282586+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"969'558","reported_seq":"3597","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429585+0000","last_change":"2021-03-15T22:35:24.518438+0000","last_active":"2021-03-15T22:35:25.429585+0000","last_peered":"2021-03-15T22:35:25.429585+0000","last_clean":"2021-03-15T22:35:25.429585+0000","last_became_active":"2021-03-15T22:34:15.498612+0000","last_became_peered":"2021-03-15T22:34:15.498612+0000","last_unstale":"2021-03-15T22:35:25.429585+0000","last_undegraded":"2021-03-15T22:35:25.429585+0000","last_fullsized":"2021-03-15T22:35:25.429585+0000","mapping_epoch":899,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":900,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'463","last_scrub_stamp":"2021-03-15T22:33:40.563144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.563144+0000","log_size":558,"ondisk_log_size":558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14726666,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":171,"num_read_kb":209120,"num_write":107,"num_write_kb":44944,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":84391207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"490","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.520522+0000","last_change":"2021-03-15T22:34:55.356457+0000","last_active":"2021-03-15T22:35:24.520522+0000","last_peered":"2021-03-15T22:35:24.520522+0000","last_clean":"2021-03-15T22:35:24.520522+0000","last_became_active":"2021-03-15T22:34:55.356086+0000","last_became_peered":"2021-03-15T22:34:55.356086+0000","last_unstale":"2021-03-15T22:35:24.520522+0000","last_undegraded":"2021-03-15T22:35:24.520522+0000","last_fullsized":"2021-03-15T22:35:24.520522+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"940","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501325+0000","last_change":"2021-03-15T22:34:55.356446+0000","last_active":"2021-03-15T22:35:22.501325+0000","last_peered":"2021-03-15T22:35:22.501325+0000","last_clean":"2021-03-15T22:35:22.501325+0000","last_became_active":"2021-03-15T22:34:55.355757+0000","last_became_peered":"2021-03-15T22:34:55.355757+0000","last_unstale":"2021-03-15T22:35:22.501325+0000","last_undegraded":"2021-03-15T22:35:22.501325+0000","last_fullsized":"2021-03-15T22:35:22.501325+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.883949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.883949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"921","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501208+0000","last_change":"2021-03-15T22:34:11.330201+0000","last_active":"2021-03-15T22:35:22.501208+0000","last_peered":"2021-03-15T22:35:22.501208+0000","last_clean":"2021-03-15T22:35:22.501208+0000","last_became_active":"2021-03-15T22:34:10.321886+0000","last_became_peered":"2021-03-15T22:34:10.321886+0000","last_unstale":"2021-03-15T22:35:22.501208+0000","last_undegraded":"2021-03-15T22:35:22.501208+0000","last_fullsized":"2021-03-15T22:35:22.501208+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:32:31.747582+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.747582+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"836'11","reported_seq":"580","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.518889+0000","last_change":"2021-03-15T22:35:10.342791+0000","last_active":"2021-03-15T22:35:24.518889+0000","last_peered":"2021-03-15T22:35:24.518889+0000","last_clean":"2021-03-15T22:35:24.518889+0000","last_became_active":"2021-03-15T22:33:04.336294+0000","last_became_peered":"2021-03-15T22:33:04.336294+0000","last_unstale":"2021-03-15T22:35:24.518889+0000","last_undegraded":"2021-03-15T22:35:24.518889+0000","last_fullsized":"2021-03-15T22:35:24.518889+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'9","last_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.971303+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"969'908","reported_seq":"3983","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.335845+0000","last_change":"2021-03-15T22:35:24.535310+0000","last_active":"2021-03-15T22:35:25.335845+0000","last_peered":"2021-03-15T22:35:25.335845+0000","last_clean":"2021-03-15T22:35:25.335845+0000","last_became_active":"2021-03-15T22:34:58.350873+0000","last_became_peered":"2021-03-15T22:34:58.350873+0000","last_unstale":"2021-03-15T22:35:25.335845+0000","last_undegraded":"2021-03-15T22:35:25.335845+0000","last_fullsized":"2021-03-15T22:35:25.335845+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'901","last_scrub_stamp":"2021-03-15T22:35:16.957336+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.957336+0000","log_size":908,"ondisk_log_size":908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31342678,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":246,"num_read_kb":212691,"num_write":224,"num_write_kb":86799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":114552572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"990","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.429241+0000","last_change":"2021-03-15T22:34:55.351577+0000","last_active":"2021-03-15T22:35:23.429241+0000","last_peered":"2021-03-15T22:35:23.429241+0000","last_clean":"2021-03-15T22:35:23.429241+0000","last_became_active":"2021-03-15T22:34:55.351249+0000","last_became_peered":"2021-03-15T22:34:55.351249+0000","last_unstale":"2021-03-15T22:35:23.429241+0000","last_undegraded":"2021-03-15T22:35:23.429241+0000","last_fullsized":"2021-03-15T22:35:23.429241+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.507204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.507204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"434","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.519761+0000","last_change":"2021-03-15T22:33:22.598240+0000","last_active":"2021-03-15T22:35:24.519761+0000","last_peered":"2021-03-15T22:35:24.519761+0000","last_clean":"2021-03-15T22:35:24.519761+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:35:24.519761+0000","last_undegraded":"2021-03-15T22:35:24.519761+0000","last_fullsized":"2021-03-15T22:35:24.519761+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.598174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.598174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"1163","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430486+0000","last_change":"2021-03-15T22:33:43.514160+0000","last_active":"2021-03-15T22:35:25.430486+0000","last_peered":"2021-03-15T22:35:25.430486+0000","last_clean":"2021-03-15T22:35:25.430486+0000","last_became_active":"2021-03-15T22:33:04.336145+0000","last_became_peered":"2021-03-15T22:33:04.336145+0000","last_unstale":"2021-03-15T22:35:25.430486+0000","last_undegraded":"2021-03-15T22:35:25.430486+0000","last_fullsized":"2021-03-15T22:35:25.430486+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:33:43.514088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.514088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"479","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.526615+0000","last_change":"2021-03-15T22:33:20.685041+0000","last_active":"2021-03-15T22:35:24.526615+0000","last_peered":"2021-03-15T22:35:24.526615+0000","last_clean":"2021-03-15T22:35:24.526615+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:35:24.526615+0000","last_undegraded":"2021-03-15T22:35:24.526615+0000","last_fullsized":"2021-03-15T22:35:24.526615+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:33:20.684976+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.684976+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"969'1621","reported_seq":"4533","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.345113+0000","last_change":"2021-03-15T22:35:24.541807+0000","last_active":"2021-03-15T22:35:26.345113+0000","last_peered":"2021-03-15T22:35:26.345113+0000","last_clean":"2021-03-15T22:35:26.345113+0000","last_became_active":"2021-03-15T22:34:58.337099+0000","last_became_peered":"2021-03-15T22:34:58.337099+0000","last_unstale":"2021-03-15T22:35:26.345113+0000","last_undegraded":"2021-03-15T22:35:26.345113+0000","last_fullsized":"2021-03-15T22:35:26.345113+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'1611","last_scrub_stamp":"2021-03-15T22:35:21.054192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.054192+0000","log_size":1621,"ondisk_log_size":1621,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40973200,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":539,"num_read_kb":566333,"num_write":345,"num_write_kb":120323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":226741522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"1048","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.429562+0000","last_change":"2021-03-15T22:34:55.343142+0000","last_active":"2021-03-15T22:35:25.429562+0000","last_peered":"2021-03-15T22:35:25.429562+0000","last_clean":"2021-03-15T22:35:25.429562+0000","last_became_active":"2021-03-15T22:34:55.341896+0000","last_became_peered":"2021-03-15T22:34:55.341896+0000","last_unstale":"2021-03-15T22:35:25.429562+0000","last_undegraded":"2021-03-15T22:35:25.429562+0000","last_fullsized":"2021-03-15T22:35:25.429562+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"753","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.525045+0000","last_change":"2021-03-15T22:34:41.870475+0000","last_active":"2021-03-15T22:35:24.525045+0000","last_peered":"2021-03-15T22:35:24.525045+0000","last_clean":"2021-03-15T22:35:24.525045+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:35:24.525045+0000","last_undegraded":"2021-03-15T22:35:24.525045+0000","last_fullsized":"2021-03-15T22:35:24.525045+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.870368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.870368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"402","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.429487+0000","last_change":"2021-03-15T22:35:08.402702+0000","last_active":"2021-03-15T22:35:23.429487+0000","last_peered":"2021-03-15T22:35:23.429487+0000","last_clean":"2021-03-15T22:35:23.429487+0000","last_became_active":"2021-03-15T22:34:10.349731+0000","last_became_peered":"2021-03-15T22:34:10.349731+0000","last_unstale":"2021-03-15T22:35:23.429487+0000","last_undegraded":"2021-03-15T22:35:23.429487+0000","last_fullsized":"2021-03-15T22:35:23.429487+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:35:08.402641+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:35:08.402641+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1016","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430071+0000","last_change":"2021-03-15T22:34:43.343527+0000","last_active":"2021-03-15T22:35:25.430071+0000","last_peered":"2021-03-15T22:35:25.430071+0000","last_clean":"2021-03-15T22:35:25.430071+0000","last_became_active":"2021-03-15T22:34:43.343255+0000","last_became_peered":"2021-03-15T22:34:43.343255+0000","last_unstale":"2021-03-15T22:35:25.430071+0000","last_undegraded":"2021-03-15T22:35:25.430071+0000","last_fullsized":"2021-03-15T22:35:25.430071+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.673338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.673338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"969'1374","reported_seq":"4531","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.601189+0000","last_change":"2021-03-15T22:35:24.539055+0000","last_active":"2021-03-15T22:35:25.601189+0000","last_peered":"2021-03-15T22:35:25.601189+0000","last_clean":"2021-03-15T22:35:25.601189+0000","last_became_active":"2021-03-15T22:34:58.344406+0000","last_became_peered":"2021-03-15T22:34:58.344406+0000","last_unstale":"2021-03-15T22:35:25.601189+0000","last_undegraded":"2021-03-15T22:35:25.601189+0000","last_fullsized":"2021-03-15T22:35:25.601189+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"872'1183","last_scrub_stamp":"2021-03-15T22:33:43.209215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.209215+0000","log_size":1374,"ondisk_log_size":1374,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":35501249,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":379,"num_read_kb":340482,"num_write":304,"num_write_kb":110670,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":127958264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"912","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501258+0000","last_change":"2021-03-15T22:34:55.356480+0000","last_active":"2021-03-15T22:35:22.501258+0000","last_peered":"2021-03-15T22:35:22.501258+0000","last_clean":"2021-03-15T22:35:22.501258+0000","last_became_active":"2021-03-15T22:34:55.356213+0000","last_became_peered":"2021-03-15T22:34:55.356213+0000","last_unstale":"2021-03-15T22:35:22.501258+0000","last_undegraded":"2021-03-15T22:35:22.501258+0000","last_fullsized":"2021-03-15T22:35:22.501258+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.729838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.729838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1265","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430599+0000","last_change":"2021-03-15T22:35:19.468482+0000","last_active":"2021-03-15T22:35:25.430599+0000","last_peered":"2021-03-15T22:35:25.430599+0000","last_clean":"2021-03-15T22:35:25.430599+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:35:25.430599+0000","last_undegraded":"2021-03-15T22:35:25.430599+0000","last_fullsized":"2021-03-15T22:35:25.430599+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:35:19.468402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.468402+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"378","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.344940+0000","last_change":"2021-03-15T22:34:10.332407+0000","last_active":"2021-03-15T22:35:26.344940+0000","last_peered":"2021-03-15T22:35:26.344940+0000","last_clean":"2021-03-15T22:35:26.344940+0000","last_became_active":"2021-03-15T22:34:10.332089+0000","last_became_peered":"2021-03-15T22:34:10.332089+0000","last_unstale":"2021-03-15T22:35:26.344940+0000","last_undegraded":"2021-03-15T22:35:26.344940+0000","last_fullsized":"2021-03-15T22:35:26.344940+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.791527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.791527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"133","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.344225+0000","last_change":"2021-03-15T22:34:44.050423+0000","last_active":"2021-03-15T22:35:26.344225+0000","last_peered":"2021-03-15T22:35:26.344225+0000","last_clean":"2021-03-15T22:35:26.344225+0000","last_became_active":"2021-03-15T22:33:45.317888+0000","last_became_peered":"2021-03-15T22:33:45.317888+0000","last_unstale":"2021-03-15T22:35:26.344225+0000","last_undegraded":"2021-03-15T22:35:26.344225+0000","last_fullsized":"2021-03-15T22:35:26.344225+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:34:44.050366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:44.050366+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"969'458","reported_seq":"3207","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.601037+0000","last_change":"2021-03-15T22:35:24.528776+0000","last_active":"2021-03-15T22:35:25.601037+0000","last_peered":"2021-03-15T22:35:25.601037+0000","last_clean":"2021-03-15T22:35:25.601037+0000","last_became_active":"2021-03-15T22:33:08.354872+0000","last_became_peered":"2021-03-15T22:33:08.354872+0000","last_unstale":"2021-03-15T22:35:25.601037+0000","last_undegraded":"2021-03-15T22:35:25.601037+0000","last_fullsized":"2021-03-15T22:35:25.601037+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'373","last_scrub_stamp":"2021-03-15T22:33:44.219517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.219517+0000","log_size":458,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11894009,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":185,"num_read_kb":197817,"num_write":93,"num_write_kb":35609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":31290769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"52","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.524352+0000","last_change":"2021-03-15T22:34:43.340972+0000","last_active":"2021-03-15T22:35:24.524352+0000","last_peered":"2021-03-15T22:35:24.524352+0000","last_clean":"2021-03-15T22:35:24.524352+0000","last_became_active":"2021-03-15T22:34:43.340600+0000","last_became_peered":"2021-03-15T22:34:43.340600+0000","last_unstale":"2021-03-15T22:35:24.524352+0000","last_undegraded":"2021-03-15T22:35:24.524352+0000","last_fullsized":"2021-03-15T22:35:24.524352+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:09.612126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.612126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"1010","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:22.501543+0000","last_change":"2021-03-15T22:35:21.917891+0000","last_active":"2021-03-15T22:35:22.501543+0000","last_peered":"2021-03-15T22:35:22.501543+0000","last_clean":"2021-03-15T22:35:22.501543+0000","last_became_active":"2021-03-15T22:33:08.352367+0000","last_became_peered":"2021-03-15T22:33:08.352367+0000","last_unstale":"2021-03-15T22:35:22.501543+0000","last_undegraded":"2021-03-15T22:35:22.501543+0000","last_fullsized":"2021-03-15T22:35:22.501543+0000","mapping_epoch":837,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'9","last_scrub_stamp":"2021-03-15T22:35:21.917819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.917819+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"980","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430144+0000","last_change":"2021-03-15T22:34:55.347300+0000","last_active":"2021-03-15T22:35:25.430144+0000","last_peered":"2021-03-15T22:35:25.430144+0000","last_clean":"2021-03-15T22:35:25.430144+0000","last_became_active":"2021-03-15T22:34:55.346934+0000","last_became_peered":"2021-03-15T22:34:55.346934+0000","last_unstale":"2021-03-15T22:35:25.430144+0000","last_undegraded":"2021-03-15T22:35:25.430144+0000","last_fullsized":"2021-03-15T22:35:25.430144+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.461102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"1142","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430502+0000","last_change":"2021-03-15T22:35:21.384129+0000","last_active":"2021-03-15T22:35:25.430502+0000","last_peered":"2021-03-15T22:35:25.430502+0000","last_clean":"2021-03-15T22:35:25.430502+0000","last_became_active":"2021-03-15T22:34:10.330608+0000","last_became_peered":"2021-03-15T22:34:10.330608+0000","last_unstale":"2021-03-15T22:35:25.430502+0000","last_undegraded":"2021-03-15T22:35:25.430502+0000","last_fullsized":"2021-03-15T22:35:25.430502+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:35:21.384037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.384037+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"966'472","reported_seq":"2719","reported_epoch":"967","state":"active+clean","last_fresh":"2021-03-15T22:35:23.289663+0000","last_change":"2021-03-15T22:35:21.344340+0000","last_active":"2021-03-15T22:35:23.289663+0000","last_peered":"2021-03-15T22:35:23.289663+0000","last_clean":"2021-03-15T22:35:23.289663+0000","last_became_active":"2021-03-15T22:34:58.350923+0000","last_became_peered":"2021-03-15T22:34:58.350923+0000","last_unstale":"2021-03-15T22:35:23.289663+0000","last_undegraded":"2021-03-15T22:35:23.289663+0000","last_fullsized":"2021-03-15T22:35:23.289663+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"957'470","last_scrub_stamp":"2021-03-15T22:35:18.338289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.338289+0000","log_size":472,"ondisk_log_size":472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5906715,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":155,"num_read_kb":164362,"num_write":109,"num_write_kb":33883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":52707265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"239","length":"1"},{"start":"266","length":"1"},{"start":"29f","length":"1"},{"start":"2c0","length":"1"},{"start":"2d9","length":"1"},{"start":"2dc","length":"1"},{"start":"2de","length":"2"},{"start":"2eb","length":"2"},{"start":"2f5","length":"1"},{"start":"2fc","length":"1"},{"start":"30f","length":"1"},{"start":"313","length":"1"},{"start":"31d","length":"1"},{"start":"324","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"714","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.524325+0000","last_change":"2021-03-15T22:34:55.345645+0000","last_active":"2021-03-15T22:35:24.524325+0000","last_peered":"2021-03-15T22:35:24.524325+0000","last_clean":"2021-03-15T22:35:24.524325+0000","last_became_active":"2021-03-15T22:34:55.345267+0000","last_became_peered":"2021-03-15T22:34:55.345267+0000","last_unstale":"2021-03-15T22:35:24.524325+0000","last_undegraded":"2021-03-15T22:35:24.524325+0000","last_fullsized":"2021-03-15T22:35:24.524325+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.958050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.958050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"1008","reported_epoch":"971","state":"active+clean","last_fresh":"2021-03-15T22:35:26.346095+0000","last_change":"2021-03-15T22:35:12.091714+0000","last_active":"2021-03-15T22:35:26.346095+0000","last_peered":"2021-03-15T22:35:26.346095+0000","last_clean":"2021-03-15T22:35:26.346095+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:35:26.346095+0000","last_undegraded":"2021-03-15T22:35:26.346095+0000","last_fullsized":"2021-03-15T22:35:26.346095+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.091650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.091650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"163","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.517741+0000","last_change":"2021-03-15T22:35:20.678309+0000","last_active":"2021-03-15T22:35:24.517741+0000","last_peered":"2021-03-15T22:35:24.517741+0000","last_clean":"2021-03-15T22:35:24.517741+0000","last_became_active":"2021-03-15T22:34:10.332236+0000","last_became_peered":"2021-03-15T22:34:10.332236+0000","last_unstale":"2021-03-15T22:35:24.517741+0000","last_undegraded":"2021-03-15T22:35:24.517741+0000","last_fullsized":"2021-03-15T22:35:24.517741+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.678246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"1116","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430096+0000","last_change":"2021-03-15T22:34:55.371223+0000","last_active":"2021-03-15T22:35:25.430096+0000","last_peered":"2021-03-15T22:35:25.430096+0000","last_clean":"2021-03-15T22:35:25.430096+0000","last_became_active":"2021-03-15T22:34:55.341958+0000","last_became_peered":"2021-03-15T22:34:55.341958+0000","last_unstale":"2021-03-15T22:35:25.430096+0000","last_undegraded":"2021-03-15T22:35:25.430096+0000","last_fullsized":"2021-03-15T22:35:25.430096+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:32:43.307660+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.307660+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"969'1592","reported_seq":"4606","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.341480+0000","last_change":"2021-03-15T22:35:24.539385+0000","last_active":"2021-03-15T22:35:25.341480+0000","last_peered":"2021-03-15T22:35:25.341480+0000","last_clean":"2021-03-15T22:35:25.341480+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:35:25.341480+0000","last_undegraded":"2021-03-15T22:35:25.341480+0000","last_fullsized":"2021-03-15T22:35:25.341480+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"837'1315","last_scrub_stamp":"2021-03-15T22:33:08.608669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.608669+0000","log_size":1592,"ondisk_log_size":1592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":49843396,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":528,"num_read_kb":561093,"num_write":364,"num_write_kb":129709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"266","length":"1"},{"start":"31c","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"981","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430676+0000","last_change":"2021-03-15T22:34:55.340853+0000","last_active":"2021-03-15T22:35:25.430676+0000","last_peered":"2021-03-15T22:35:25.430676+0000","last_clean":"2021-03-15T22:35:25.430676+0000","last_became_active":"2021-03-15T22:34:55.340498+0000","last_became_peered":"2021-03-15T22:34:55.340498+0000","last_unstale":"2021-03-15T22:35:25.430676+0000","last_undegraded":"2021-03-15T22:35:25.430676+0000","last_fullsized":"2021-03-15T22:35:25.430676+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:20.500979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.500979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"941'18","reported_seq":"350","reported_epoch":"969","state":"active+remapped+backfill_toofull","last_fresh":"2021-03-15T22:35:24.519554+0000","last_change":"2021-03-15T22:34:56.377931+0000","last_active":"2021-03-15T22:35:24.519554+0000","last_peered":"2021-03-15T22:35:24.519554+0000","last_clean":"2021-03-15T22:30:29.286429+0000","last_became_active":"2021-03-15T22:34:56.340566+0000","last_became_peered":"2021-03-15T22:34:56.340566+0000","last_unstale":"2021-03-15T22:35:24.519554+0000","last_undegraded":"2021-03-15T22:35:24.519554+0000","last_fullsized":"2021-03-15T22:35:24.519554+0000","mapping_epoch":940,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":878,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'13","last_scrub_stamp":"2021-03-15T22:33:37.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.949876+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[0,7,5],"avail_no_missing":["0","2","5","7"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"398","reported_epoch":"969","state":"active+clean","last_fresh":"2021-03-15T22:35:24.524671+0000","last_change":"2021-03-15T22:33:21.690369+0000","last_active":"2021-03-15T22:35:24.524671+0000","last_peered":"2021-03-15T22:35:24.524671+0000","last_clean":"2021-03-15T22:35:24.524671+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:35:24.524671+0000","last_undegraded":"2021-03-15T22:35:24.524671+0000","last_fullsized":"2021-03-15T22:35:24.524671+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.690309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"1155","reported_epoch":"970","state":"active+clean","last_fresh":"2021-03-15T22:35:25.430713+0000","last_change":"2021-03-15T22:35:15.476444+0000","last_active":"2021-03-15T22:35:25.430713+0000","last_peered":"2021-03-15T22:35:25.430713+0000","last_clean":"2021-03-15T22:35:25.430713+0000","last_became_active":"2021-03-15T22:33:04.337113+0000","last_became_peered":"2021-03-15T22:33:04.337113+0000","last_unstale":"2021-03-15T22:35:25.430713+0000","last_undegraded":"2021-03-15T22:35:25.430713+0000","last_fullsized":"2021-03-15T22:35:25.430713+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:32:32.328526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.328526+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":646687325,"num_objects":304,"num_object_clones":254,"num_object_copies":912,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":304,"num_whiteouts":17,"num_read":6837,"num_read_kb":7010905,"num_write":5022,"num_write_kb":1917028,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4621,"num_bytes_recovered":2230024638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1459818496,"data_stored":2075058897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":21990,"ondisk_log_size":21990,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1034,"num_read_kb":1029,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1482,"num_read_kb":987,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":94766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":947,"ondisk_log_size":947,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":134,"ondisk_log_size":134,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299606,"num_pgs":67,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":249300,"kb_used_data":197580,"kb_used_omap":0,"kb_used_meta":51712,"kb_avail":93495852,"statfs":{"total":95995035648,"available":95739752448,"internally_reserved":0,"allocated":202321920,"data_stored":316078706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":52953088},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:23 2021","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.45500000000000002,"15min":0.46999999999999997},"min":{"1min":0.32200000000000001,"5min":0.216,"15min":0.215},"max":{"1min":0.65300000000000002,"5min":0.70099999999999996,"15min":2.73},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.44400000000000001,"5min":0.49099999999999999,"15min":0.48999999999999999},"min":{"1min":0.30299999999999999,"5min":0.30299999999999999,"15min":0.222},"max":{"1min":0.66900000000000004,"5min":0.75800000000000001,"15min":3.8799999999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:34:23 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45400000000000001,"15min":0.47499999999999998},"min":{"1min":0.28000000000000003,"5min":0.249,"15min":0.249},"max":{"1min":0.66600000000000004,"5min":0.73899999999999999,"15min":2.6859999999999999},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.496,"5min":0.501,"15min":0.49399999999999999},"min":{"1min":0.34899999999999998,"5min":0.28399999999999997,"15min":0.23400000000000001},"max":{"1min":0.69699999999999995,"5min":0.81899999999999995,"15min":2.0779999999999998},"last":0.41899999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:34:38 2021","interfaces":[{"interface":"back","average":{"1min":0.46100000000000002,"5min":0.5,"15min":0.48199999999999998},"min":{"1min":0.32300000000000001,"5min":0.32300000000000001,"15min":0.32300000000000001},"max":{"1min":0.68600000000000005,"5min":0.77400000000000002,"15min":0.77400000000000002},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.497,"15min":0.5},"min":{"1min":0.31900000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":0.63200000000000001,"5min":1.119,"15min":1.119},"last":0.378}]},{"osd":3,"last update":"Mon Mar 15 22:34:23 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.5,"15min":0.495},"min":{"1min":0.35399999999999998,"5min":0.33800000000000002,"15min":0.28899999999999998},"max":{"1min":0.61099999999999999,"5min":1.0249999999999999,"15min":1.702},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.47799999999999998,"5min":0.48899999999999999,"15min":0.52700000000000002},"min":{"1min":0.33100000000000002,"5min":0.33100000000000002,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":0.78300000000000003,"15min":5.415},"last":0.36799999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:34:23 2021","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.436,"15min":0.44700000000000001},"min":{"1min":0.28499999999999998,"5min":0.255,"15min":0.23799999999999999},"max":{"1min":0.56100000000000005,"5min":0.753,"15min":1.927},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.47299999999999998,"5min":0.46600000000000003,"15min":0.46899999999999997},"min":{"1min":0.35099999999999998,"5min":0.28499999999999998,"15min":0.253},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":2.5339999999999998},"last":0.439}]},{"osd":5,"last update":"Mon Mar 15 22:34:38 2021","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.47599999999999998,"15min":0.45400000000000001},"min":{"1min":0.318,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.61699999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.50900000000000001,"15min":0.505},"min":{"1min":0.29699999999999999,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68899999999999995,"5min":1.071,"15min":1.071},"last":0.40999999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:34:23 2021","interfaces":[{"interface":"back","average":{"1min":0.51100000000000001,"5min":0.50700000000000001,"15min":0.51900000000000002},"min":{"1min":0.36199999999999999,"5min":0.29999999999999999,"15min":0.27100000000000002},"max":{"1min":0.64200000000000002,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.53500000000000003,"15min":0.52400000000000002},"min":{"1min":0.34300000000000003,"5min":0.33800000000000002,"15min":0.28399999999999997},"max":{"1min":0.81999999999999995,"5min":1.097,"15min":1.8560000000000001},"last":0.45700000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528538,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":190904,"kb_used_data":149872,"kb_used_omap":0,"kb_used_meta":41024,"kb_avail":93554248,"statfs":{"total":95995035648,"available":95799549952,"internally_reserved":0,"allocated":153468928,"data_stored":213405101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42008576},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.49199999999999999,"15min":0.48399999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.79600000000000004,"5min":2.5470000000000002,"15min":3.085},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.47999999999999998,"15min":0.46500000000000002},"min":{"1min":0.27600000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73099999999999998,"5min":1.137,"15min":1.323},"last":0.57199999999999995}]},{"osd":1,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.254},"max":{"1min":1.1120000000000001,"5min":2.7839999999999998,"15min":14.637},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.45200000000000001,"15min":0.47499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60299999999999998,"5min":0.84899999999999998,"15min":5.2450000000000001},"last":0.48799999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:34:40 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47299999999999998,"15min":0.46500000000000002},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.503},{"interface":"front","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.51200000000000001},"min":{"1min":0.36199999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.63300000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.59899999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51100000000000001,"15min":0.50900000000000001},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.29899999999999999},"max":{"1min":0.874,"5min":1.0609999999999999,"15min":2.6150000000000002},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.502,"15min":0.502},"min":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.63900000000000001,"5min":2.4670000000000001,"15min":2.7890000000000001},"last":0.52800000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.44600000000000001,"15min":0.438},"min":{"1min":0.25,"5min":0.25,"15min":0.224},"max":{"1min":0.61799999999999999,"5min":0.94099999999999995,"15min":4.306},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.47799999999999998,"15min":0.49199999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23499999999999999},"max":{"1min":0.82799999999999996,"5min":1.091,"15min":4.181},"last":0.34699999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:34:36 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.47199999999999998,"15min":0.42499999999999999},"min":{"1min":0.311,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.622,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.38800000000000001},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.52700000000000002,"15min":0.46899999999999997},"min":{"1min":0.33700000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.63900000000000001,"5min":1.022,"15min":1.022},"last":0.439}]},{"osd":7,"last update":"Mon Mar 15 22:35:21 2021","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.47799999999999998,"15min":0.53200000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.249},"max":{"1min":0.67700000000000005,"5min":0.76300000000000001,"15min":15.340999999999999},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.502,"5min":0.56399999999999995,"15min":0.58699999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.27200000000000002},"max":{"1min":0.754,"5min":1.0109999999999999,"15min":15.356},"last":0.51100000000000001}]}]},{"osd":1,"up_from":13,"seq":55834575084,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":172172,"kb_used_data":105220,"kb_used_omap":0,"kb_used_meta":66944,"kb_avail":93572980,"statfs":{"total":95995035648,"available":95818731520,"internally_reserved":0,"allocated":107745280,"data_stored":145113682,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68550656},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:52 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.379,"15min":0.51200000000000001},"min":{"1min":0.217,"5min":0.184,"15min":0.17599999999999999},"max":{"1min":4.3920000000000003,"5min":4.3920000000000003,"15min":54.427999999999997},"last":0.26400000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.42699999999999999,"15min":0.54100000000000004},"min":{"1min":0.219,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":54.356999999999999},"last":0.38500000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.46600000000000003,"15min":0.378},"min":{"1min":0.26100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.45200000000000001,"15min":0.40000000000000002},"min":{"1min":0.25700000000000001,"5min":0.254,"15min":0.254},"max":{"1min":5.0190000000000001,"5min":5.0190000000000001,"15min":5.0190000000000001},"last":0.45100000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:35:20 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.47599999999999998,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.26400000000000001,"15min":0.215},"max":{"1min":7.718,"5min":7.718,"15min":7.718},"last":0.35299999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.53700000000000003,"15min":0.434},"min":{"1min":0.245,"5min":0.245,"15min":0.20999999999999999},"max":{"1min":8.4760000000000009,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":0.308}]},{"osd":4,"last update":"Mon Mar 15 22:35:26 2021","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.61399999999999999,"15min":0.55200000000000005},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.22800000000000001},"max":{"1min":0.78800000000000003,"5min":4.593,"15min":4.593},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.253},"max":{"1min":4.3140000000000001,"5min":4.3140000000000001,"15min":4.6139999999999999},"last":0.48499999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:34:31 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55200000000000005,"15min":0.53300000000000003},"min":{"1min":0.34499999999999997,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.78600000000000003,"5min":1.226,"15min":1.226},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.53100000000000003,"5min":0.58999999999999997,"15min":0.52200000000000002},"min":{"1min":0.34499999999999997,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.89000000000000001,"5min":3.472,"15min":3.472},"last":0.42499999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:34:59 2021","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61499999999999999,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.314},"max":{"1min":1.679,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.63600000000000001,"15min":0.58299999999999996},"min":{"1min":0.39800000000000002,"5min":0.375,"15min":0.33700000000000002},"max":{"1min":0.82999999999999996,"5min":4.6529999999999996,"15min":4.6529999999999996},"last":0.46300000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:35:20 2021","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63700000000000001,"15min":0.60299999999999998},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.316},"max":{"1min":1.716,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.66800000000000004,"15min":0.59599999999999997},"min":{"1min":0.42799999999999999,"5min":0.40300000000000002,"15min":0.35799999999999998},"max":{"1min":1.466,"5min":4.625,"15min":4.625},"last":0.498}]}]},{"osd":0,"up_from":8,"seq":34359738606,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":256708,"kb_used_data":199740,"kb_used_omap":0,"kb_used_meta":56960,"kb_avail":93488444,"statfs":{"total":95995035648,"available":95732166656,"internally_reserved":0,"allocated":204533760,"data_stored":312630699,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58327040},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:34:44 2021","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.34399999999999997,"15min":0.33800000000000002},"min":{"1min":0.27600000000000002,"5min":0.192,"15min":0.17999999999999999},"max":{"1min":0.55400000000000005,"5min":1.46,"15min":1.7070000000000001},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.38700000000000001,"15min":0.36899999999999999},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.19900000000000001},"max":{"1min":1.331,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.376}]},{"osd":2,"last update":"Mon Mar 15 22:34:40 2021","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.372,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.46100000000000002,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.49199999999999999,"15min":0.41399999999999998},"min":{"1min":0.26200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.3799999999999999,"5min":7.3799999999999999,"15min":7.3799999999999999},"last":0.33300000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39700000000000002,"15min":0.38400000000000001},"min":{"1min":0.25700000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":0.61799999999999999,"5min":1.544,"15min":1.544},"last":0.38700000000000001},{"interface":"front","average":{"1min":0.442,"5min":0.47499999999999998,"15min":0.41999999999999998},"min":{"1min":0.31,"5min":0.254,"15min":0.19400000000000001},"max":{"1min":0.67700000000000005,"5min":3.988,"15min":3.988},"last":0.41899999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.36899999999999999,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":0.76300000000000001,"5min":1.861,"15min":1.861},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.56599999999999995},"min":{"1min":0.40699999999999997,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":0.74099999999999999,"5min":1.5700000000000001,"15min":1.645},"last":0.51200000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:34:35 2021","interfaces":[{"interface":"back","average":{"1min":0.70299999999999996,"5min":0.59899999999999998,"15min":0.53200000000000003},"min":{"1min":0.38100000000000001,"5min":0.29899999999999999,"15min":0.29899999999999999},"max":{"1min":3.7879999999999998,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.55400000000000005,"15min":0.5},"min":{"1min":0.35899999999999999,"5min":0.318,"15min":0.318},"max":{"1min":1.163,"5min":1.909,"15min":1.909},"last":0.44600000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:34:59 2021","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.26200000000000001},"max":{"1min":2.7109999999999999,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.437,"5min":0.32000000000000001,"15min":0.27100000000000002},"max":{"1min":0.77100000000000002,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.60399999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.65500000000000003,"15min":0.58699999999999997},"min":{"1min":0.42899999999999999,"5min":0.38200000000000001,"15min":0.27900000000000003},"max":{"1min":0.753,"5min":3.8239999999999998,"15min":3.8239999999999998},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60699999999999998,"15min":0.57999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39100000000000001,"15min":0.317},"max":{"1min":0.752,"5min":2.665,"15min":2.665},"last":0.48899999999999999}]}]},{"osd":2,"up_from":681,"seq":2924872728636,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":328996,"kb_used_data":287068,"kb_used_omap":119,"kb_used_meta":41800,"kb_avail":93416156,"statfs":{"total":95995035648,"available":95658143744,"internally_reserved":0,"allocated":293957632,"data_stored":386394839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":42803561},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.315,"15min":0.309},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.40000000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.29499999999999998},{"interface":"front","average":{"1min":0.372,"5min":0.34599999999999997,"15min":0.32800000000000001},"min":{"1min":0.251,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.223}]},{"osd":1,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.35399999999999998,"15min":0.33300000000000002},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.28599999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38,"15min":0.36299999999999999},"min":{"1min":0.27100000000000002,"5min":0.221,"15min":0.221},"max":{"1min":1.0269999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.33600000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.36199999999999999},"min":{"1min":0.30599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.89300000000000002,"5min":3.496,"15min":3.496},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.375,"15min":0.35899999999999999},"min":{"1min":0.252,"5min":0.248,"15min":0.248},"max":{"1min":0.624,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.32500000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53800000000000003,"15min":0.49099999999999999},"min":{"1min":0.377,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.113,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.497},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.94699999999999995,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.59699999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51500000000000001,"15min":0.502},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.65500000000000003,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.64400000000000002},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55500000000000005,"15min":0.51700000000000002},"min":{"1min":0.40400000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.1870000000000001,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.63600000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.51500000000000001},"min":{"1min":0.374,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.208,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56499999999999995,"15min":0.52100000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.97799999999999998,"5min":1.387,"15min":1.387},"last":0.66000000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58799999999999997,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.222,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56599999999999995,"15min":0.52900000000000003},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0529999999999999,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.69399999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149924,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":280340,"kb_used_data":209356,"kb_used_omap":0,"kb_used_meta":70976,"kb_avail":93464812,"statfs":{"total":95995035648,"available":95707967488,"internally_reserved":0,"allocated":214380544,"data_stored":283109179,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":72679424},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34499999999999997,"15min":0.34000000000000002},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.17699999999999999},"max":{"1min":0.47799999999999998,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.33300000000000002,"15min":0.33800000000000002},"min":{"1min":0.23699999999999999,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.504,"5min":0.76900000000000002,"15min":2.6339999999999999},"last":0.34899999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.371,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23599999999999999,"15min":0.23000000000000001},"max":{"1min":0.501,"5min":0.626,"15min":6.0199999999999996},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.36799999999999999,"15min":0.39900000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.20699999999999999},"max":{"1min":0.47899999999999998,"5min":2.5619999999999998,"15min":7.6130000000000004},"last":0.36199999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35499999999999998},"min":{"1min":0.247,"5min":0.216,"15min":0.216},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.33700000000000002},{"interface":"front","average":{"1min":0.38,"5min":0.38700000000000001,"15min":0.38800000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53600000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.38400000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:34:31 2021","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.22600000000000001},"max":{"1min":0.69099999999999995,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.53400000000000003,"15min":0.52800000000000002},"min":{"1min":0.311,"5min":0.28499999999999998,"15min":0.26700000000000002},"max":{"1min":0.68000000000000005,"5min":2.1059999999999999,"15min":2.2890000000000001},"last":0.65300000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:34:31 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.59999999999999998,"15min":0.77700000000000002},"min":{"1min":0.313,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.71999999999999997,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":0.57799999999999996},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.66000000000000003,"15min":0.998},"min":{"1min":0.36699999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66100000000000003,"5min":15.776,"15min":15.776},"last":0.72599999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:35:02 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55800000000000005,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.315,"15min":0.30099999999999999},"max":{"1min":0.78700000000000003,"5min":1.359,"15min":2.6499999999999999},"last":0.81699999999999995},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55400000000000005,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.35899999999999999,"15min":0.30399999999999999},"max":{"1min":0.77800000000000002,"5min":1.3400000000000001,"15min":3.472},"last":0.78500000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.57399999999999995},"min":{"1min":0.36199999999999999,"5min":0.308,"15min":0.29599999999999999},"max":{"1min":0.77600000000000002,"5min":0.91200000000000003,"15min":5.625},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.40799999999999997,"5min":0.371,"15min":0.35499999999999998},"max":{"1min":0.74299999999999999,"5min":2.621,"15min":2.621},"last":0.67300000000000004}]}]},{"osd":4,"up_from":31,"seq":133143986401,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":131760,"kb_used_data":87656,"kb_used_omap":0,"kb_used_meta":44096,"kb_avail":93613392,"statfs":{"total":95995035648,"available":95860113408,"internally_reserved":0,"allocated":89759744,"data_stored":138091328,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45154304},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:32 2021","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.504,"15min":0.47699999999999998},"min":{"1min":0.221,"5min":0.221,"15min":0.19500000000000001},"max":{"1min":0.59999999999999998,"5min":4.7050000000000001,"15min":4.7050000000000001},"last":0.379},{"interface":"front","average":{"1min":0.438,"5min":0.46200000000000002,"15min":0.5},"min":{"1min":0.27100000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.68100000000000005,"5min":1.4179999999999999,"15min":5.0209999999999999},"last":0.33300000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:34:32 2021","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.47699999999999998,"15min":0.48199999999999998},"min":{"1min":0.34499999999999997,"5min":0.25,"15min":0.23599999999999999},"max":{"1min":0.752,"5min":1.3720000000000001,"15min":4.3499999999999996},"last":0.34899999999999998},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.496,"15min":0.504},"min":{"1min":0.37,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":0.67500000000000004,"5min":1.494,"15min":4.718},"last":0.55100000000000005}]},{"osd":2,"last update":"Mon Mar 15 22:34:38 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.49099999999999999,"15min":0.505},"min":{"1min":0.375,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.55600000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.35799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77200000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.314}]},{"osd":3,"last update":"Mon Mar 15 22:34:32 2021","interfaces":[{"interface":"back","average":{"1min":0.51200000000000001,"5min":0.51800000000000002,"15min":0.54900000000000004},"min":{"1min":0.40300000000000002,"5min":0.32200000000000001,"15min":0.28499999999999998},"max":{"1min":0.71999999999999997,"5min":1.486,"15min":5.0499999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.54700000000000004,"15min":0.54200000000000004},"min":{"1min":0.32900000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71899999999999997,"5min":1.4950000000000001,"15min":4.4050000000000002},"last":0.48699999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:34:32 2021","interfaces":[{"interface":"back","average":{"1min":0.41899999999999998,"5min":0.42199999999999999,"15min":0.40899999999999997},"min":{"1min":0.22900000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.53700000000000003,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.28499999999999998},{"interface":"front","average":{"1min":0.47399999999999998,"5min":0.46600000000000003,"15min":0.433},"min":{"1min":0.30299999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66800000000000004,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.32300000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:34:58 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47799999999999998,"15min":0.46700000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":0.70999999999999996,"5min":0.878,"15min":4.3079999999999998},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54900000000000004,"15min":0.54000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28000000000000003},"max":{"1min":0.71899999999999997,"5min":0.83799999999999997,"15min":4.6600000000000001},"last":0.36199999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:35:18 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51300000000000001,"15min":0.51000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26300000000000001},"max":{"1min":0.749,"5min":0.80900000000000005,"15min":4.6929999999999996},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.25},"max":{"1min":0.73599999999999999,"5min":1.034,"15min":4.9900000000000002},"last":0.44800000000000001}]}]},{"osd":5,"up_from":616,"seq":2645699854409,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":241736,"kb_used_data":202112,"kb_used_omap":153,"kb_used_meta":39462,"kb_avail":93503416,"statfs":{"total":95995035648,"available":95747497984,"internally_reserved":0,"allocated":206962688,"data_stored":286346466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":40409617},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48999999999999999,"15min":0.46400000000000002},"min":{"1min":0.28799999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0780000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":0.378},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48199999999999998,"15min":0.45600000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.002,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.48999999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.56100000000000005,"15min":0.48399999999999999},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.046,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.42999999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59799999999999998,"15min":0.54100000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.121,"5min":10.952,"15min":10.952},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.51500000000000001,"15min":0.52700000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.103,"5min":1.125,"15min":1.125},"last":0.623}]},{"osd":3,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.67800000000000005,"15min":0.53800000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.1379999999999999,"5min":10.355,"15min":10.355},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.505,"15min":0.45700000000000002},"min":{"1min":0.35699999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.1839999999999999,"5min":2.476,"15min":2.476},"last":0.39400000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.504,"15min":0.41199999999999998},"min":{"1min":0.29799999999999999,"5min":0.254,"15min":0.254},"max":{"1min":1.204,"5min":7.4000000000000004,"15min":7.4000000000000004},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46100000000000002,"15min":0.40400000000000003},"min":{"1min":0.316,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.63400000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.53500000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.50800000000000001,"15min":0.46500000000000002},"min":{"1min":0.30199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.98899999999999999,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55300000000000005,"15min":0.48899999999999999},"min":{"1min":0.38100000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0860000000000001,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.61099999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.51800000000000002,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.0169999999999999,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58399999999999996,"15min":0.51600000000000001},"min":{"1min":0.35999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.0269999999999999,"5min":5.944,"15min":5.944},"last":0.502}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":12026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":10320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":8144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":204095488,"data_stored":314192177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":105787392,"data_stored":143994974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292220928,"data_stored":385307880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":212500480,"data_stored":282001377,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":87867392,"data_stored":136982986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":205234176,"data_stored":285267896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":151592960,"data_stored":212305550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":200519680,"data_stored":315006057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:35:29.585 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2021-03-15T22:35:29.586 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool set default.rgw.meta pg_num 32 2021-03-15T22:35:29.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:29 smithi076 conmon[62156]: cluster 2021-03-15T22:35:28.349631+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e973: 8 total, 8 up, 8 in 2021-03-15T22:35:29.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:29 smithi076 conmon[62156]: audit 2021-03-15T22:35:28.703696+0000 mgr.y (mgr.14144) 1078 : audit [DBG] from='client.16809 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:29.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:29 smithi076 conmon[56514]: cluster 2021-03-15T22:35:29.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:29 smithi076 conmon[56514]: 2021-03-15 2021-03-15T22:35:29.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:29 smithi076 conmon[56514]: T22:35:28.349631+0000 mon.a (mon.0) 2425 : cluster [DBG] 2021-03-15T22:35:29.610 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:29 smithi076 conmon[56514]: osdmap e973: 8 total, 8 up, 8 in 2021-03-15T22:35:29.611 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:29 smithi076 conmon[56514]: audit 2021-03-15T22:35:29.611 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:29 smithi076 conmon[56514]: 2021-03-15T22:35:29.611 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:29 smithi076 conmon[56514]: 2021-03-15T22:35:28 2021-03-15T22:35:29.611 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:29 smithi076 conmon[56514]: .703696+0000 mgr.y (mgr.14144) 1078 : audit [DBG] from='client.16809 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:29.625 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:29.648 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:29.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:29.649 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:29.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:35:29.750 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:29.773 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:29.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:29.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:29.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:29 smithi125 conmon[56739]: cluster 2021-03-15T22:35:28.349631+0000 mon.a (mon.0) 2425 : cluster [DBG] osdmap e973: 8 total, 8 up, 8 in 2021-03-15T22:35:29.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:29 smithi125 conmon[56739]: audit 2021-03-15T22:35:28.703696+0000 mgr.y (mgr.14144) 1078 : audit [DBG] from='client.16809 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:29.875 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:29.898 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:29.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:29.987 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:30.001 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:30.018 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:30.024 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:30.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:30.025 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:30.030 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:35:30.031 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:35:30.127 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:30.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:30.155 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:30.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:30.256 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:30.280 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:30.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:30.281 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:30.349 INFO:tasks.rados.rados.0.smithi076.stdout:3715: done (0 left) 2021-03-15T22:35:30.350 INFO:tasks.rados.rados.0.smithi076.stdout:3716: snap_remove snap 412 2021-03-15T22:35:30.366 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:35:30.372 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:35:30.382 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:30.405 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:30.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:30.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:30.453 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[56514]: cluster 2021-03-15T22:35:29.220465+0000 mgr.y (mgr.14144) 1079 : cluster [DBG] pgmap v1607: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 617 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-03-15T22:35:30.453 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[56514]: cluster 2021-03-15T22:35:29.357160+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e974: 8 total, 8 up, 8 in 2021-03-15T22:35:30.453 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[56514]: audit 2021-03-15T22:35:30.222704+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? 172.21.15.76:0/227172315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:30.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[56514]: cluster 2021-03-15T22:35:30.333199 2021-03-15T22:35:30.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[56514]: +0000 mon.a (mon.0) 2428 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:30.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[56514]: audit 2021-03-15T22:35:30.339503+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? 172.21.15.76:0/227172315' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:30.454 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[56514]: cluster 2021-03-15T22:35:30.339554+0000 mon.a (mon.0) 2430 : cluster [DBG] osdmap e975: 8 total, 8 up, 8 in 2021-03-15T22:35:30.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[62156]: cluster 2021-03-15T22:35:29.220465+0000 2021-03-15T22:35:30.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[62156]: mgr.y (mgr.14144) 1079 : cluster [DBG] pgmap v1607: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 617 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-03-15T22:35:30.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[62156]: cluster 2021-03-15T22:35:29.357160+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e974: 8 total, 8 up, 8 in 2021-03-15T22:35:30.455 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[62156]: audit 2021-03-15T22:35:30.222704+0000 mon.a (mon.0) 2427 : audit [INF] from='client.? 172.21.15.76:0/227172315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:30.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[62156]: cluster 2021-03-15T22:35:30.333199+0000 mon.a (mon.0) 2428 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:30.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[62156]: audit 2021-03-15T22:35:30.339503+0000 mon.a (mon.0) 2429 : audit [INF] from='client.? 172.21.15.76:0/227172315' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:30.456 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:30 smithi076 conmon[62156]: cluster 2021-03-15T22:35:30.339554+0000 mon.a (mon.0) 2430 : cluster [DBG] osdmap e975: 8 total, 8 up, 8 in 2021-03-15T22:35:30.508 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:30.530 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:30.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:30.532 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:30.634 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:30.656 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:30.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:30.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:30.659 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: cluster 2021-03-15T22:35:29.220465+0000 mgr.y (mgr.14144) 1079 2021-03-15T22:35:30.659 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: : cluster [DBG] pgmap v1607: 158 pgs: 1 active+remapped+backfill_toofull, 157 active+clean; 617 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 12 MiB/s rd, 1.3 MiB/s wr, 11 op/s 2021-03-15T22:35:30.660 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: cluster 2021-03-15T22:35:29.357160+0000 mon.a (mon.0) 2426 : cluster [DBG] osdmap e974: 8 total, 8 up, 8 in 2021-03-15T22:35:30.660 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: audit 2021-03- 2021-03-15T22:35:30.660 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: 15T22:35:30.222704 2021-03-15T22:35:30.660 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2427 : 2021-03-15T22:35:30.660 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: audit [INF] from='client.? 172.21.15.76:0/227172315' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:30.661 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:35:30.661 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: -03-15T22:35:30.333199 2021-03-15T22:35:30.661 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2428 : cluster 2021-03-15T22:35:30.661 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:30.662 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:35:30.662 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: T22:35:30.339503+0000 mon.a ( 2021-03-15T22:35:30.662 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: mon.0) 2429 : audit [INF] from='client.? 172.21.15.76:0/227172315' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:30.662 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:35:30.662 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: -15T22:35:30.339554+0000 mon.a 2021-03-15T22:35:30.663 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:30 smithi125 conmon[56739]: (mon.0) 2430 : cluster [DBG] osdmap e975: 8 total, 8 up, 8 in 2021-03-15T22:35:30.681 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.4 dump_historic_ops 2021-03-15T22:35:30.758 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:30.782 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:30.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:30.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:30.884 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:30.907 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:30.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:30.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:30.959 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:31.010 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:31.032 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:31.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:31.033 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:31.134 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:31.156 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:31.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:31.158 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:31.259 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:31.281 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:31.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:31.282 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:31.369 INFO:tasks.rados.rados.0.smithi076.stdout:3716: done (0 left) 2021-03-15T22:35:31.369 INFO:tasks.rados.rados.0.smithi076.stdout:3717: delete oid 6 current snap is 437 2021-03-15T22:35:31.377 INFO:tasks.rados.rados.0.smithi076.stdout:3717: done (0 left) 2021-03-15T22:35:31.377 INFO:tasks.rados.rados.0.smithi076.stdout:3718: read oid 29 snap -1 2021-03-15T22:35:31.378 INFO:tasks.rados.rados.0.smithi076.stdout:3718: expect (ObjNum 1140 snap 393 seq_num 1140) 2021-03-15T22:35:31.378 INFO:tasks.rados.rados.0.smithi076.stdout:3719: copy_from oid 17 from oid 10 current snap is 437 2021-03-15T22:35:31.378 INFO:tasks.rados.rados.0.smithi076.stdout:3720: copy_from oid 36 from oid 27 current snap is 437 2021-03-15T22:35:31.378 INFO:tasks.rados.rados.0.smithi076.stdout:3721: write oid 18 current snap is 437 2021-03-15T22:35:31.379 INFO:tasks.rados.rados.0.smithi076.stdout:3721: seq_num 1265 ranges {460855=504583,1624898=412110,2539027=572508} 2021-03-15T22:35:31.382 INFO:tasks.rados.rados.0.smithi076.stdout:3721: writing smithi07694878-18 from 460855 to 965438 tid 1 2021-03-15T22:35:31.383 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:31.390 INFO:tasks.rados.rados.0.smithi076.stdout:3721: writing smithi07694878-18 from 1624898 to 2037008 tid 2 2021-03-15T22:35:31.396 INFO:tasks.rados.rados.0.smithi076.stdout:3721: writing smithi07694878-18 from 2539027 to 3111535 tid 3 2021-03-15T22:35:31.397 INFO:tasks.rados.rados.0.smithi076.stdout:3722: copy_from oid 48 from oid 31 current snap is 437 2021-03-15T22:35:31.397 INFO:tasks.rados.rados.0.smithi076.stdout:3718: expect (ObjNum 1140 snap 393 seq_num 1140) 2021-03-15T22:35:31.410 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:31.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:31.411 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:31.512 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:31.535 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:31.537 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:31.591 INFO:tasks.rados.rados.0.smithi076.stdout:3721: finishing write tid 1 to smithi07694878-18 2021-03-15T22:35:31.591 INFO:tasks.rados.rados.0.smithi076.stdout:3722: finishing copy_from to smithi07694878-48 2021-03-15T22:35:31.592 INFO:tasks.rados.rados.0.smithi076.stdout:3722: got expected ENOENT (src dne) 2021-03-15T22:35:31.592 INFO:tasks.rados.rados.0.smithi076.stdout:3720: finishing copy_from to smithi07694878-36 2021-03-15T22:35:31.592 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 3042 (ObjNum 1243 snap 429 seq_num 1243) dirty exists 2021-03-15T22:35:31.592 INFO:tasks.rados.rados.0.smithi076.stdout:3719: finishing copy_from to smithi07694878-17 2021-03-15T22:35:31.593 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 2437 (ObjNum 1241 snap 428 seq_num 1241) dirty exists 2021-03-15T22:35:31.593 INFO:tasks.rados.rados.0.smithi076.stdout:3720: finishing copy_from racing read to smithi07694878-36 2021-03-15T22:35:31.593 INFO:tasks.rados.rados.0.smithi076.stdout:3722: finishing copy_from racing read to smithi07694878-48 2021-03-15T22:35:31.593 INFO:tasks.rados.rados.0.smithi076.stdout:3719: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:35:31.594 INFO:tasks.rados.rados.0.smithi076.stdout:3721: finishing write tid 2 to smithi07694878-18 2021-03-15T22:35:31.594 INFO:tasks.rados.rados.0.smithi076.stdout:3718: done (4 left) 2021-03-15T22:35:31.594 INFO:tasks.rados.rados.0.smithi076.stdout:3719: done (3 left) 2021-03-15T22:35:31.594 INFO:tasks.rados.rados.0.smithi076.stdout:3720: done (2 left) 2021-03-15T22:35:31.594 INFO:tasks.rados.rados.0.smithi076.stdout:3722: done (1 left) 2021-03-15T22:35:31.595 INFO:tasks.rados.rados.0.smithi076.stdout:3723: copy_from oid 45 from oid 12 current snap is 437 2021-03-15T22:35:31.595 INFO:tasks.rados.rados.0.smithi076.stdout:3721: finishing write tid 3 to smithi07694878-18 2021-03-15T22:35:31.595 INFO:tasks.rados.rados.0.smithi076.stdout:3721: finishing write tid 4 to smithi07694878-18 2021-03-15T22:35:31.595 INFO:tasks.rados.rados.0.smithi076.stdout:3721: finishing write tid 5 to smithi07694878-18 2021-03-15T22:35:31.596 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 2446 (ObjNum 1265 snap 437 seq_num 1265) dirty exists 2021-03-15T22:35:31.596 INFO:tasks.rados.rados.0.smithi076.stdout:3721: left oid 18 (ObjNum 1265 snap 437 seq_num 1265) 2021-03-15T22:35:31.596 INFO:tasks.rados.rados.0.smithi076.stdout:3721: done (1 left) 2021-03-15T22:35:31.596 INFO:tasks.rados.rados.0.smithi076.stdout:3724: snap_remove snap 375 2021-03-15T22:35:31.638 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:31.660 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:31.662 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:31.763 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:31.785 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:31.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:31.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:31.888 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:31.910 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:31.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:31.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:32.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:32.037 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:32.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:32.038 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:32.139 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:32.162 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:32.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:32.164 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:32.261 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:32 smithi125 conmon[56739]: audit 2021-03-15T22:35:30.301411+0000 mgr.y (mgr.14144) 1080 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:32.261 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:32 smithi125 conmon[56739]: audit 2021-03-15T22:35:31.109031+0000 mon.a (mon.0) 2431 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:35:32.261 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:32 smithi125 conmon[56739]: cluster 2021-03-15T22:35:31.360416+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e976: 8 total, 8 up, 8 in 2021-03-15T22:35:32.265 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:32.288 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:32.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:32.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:32.372 INFO:tasks.rados.rados.0.smithi076.stdout:3724: done (1 left) 2021-03-15T22:35:32.373 INFO:tasks.rados.rados.0.smithi076.stdout:3725: rollback oid 6 current snap is 437 2021-03-15T22:35:32.373 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 6 to 411 2021-03-15T22:35:32.373 INFO:tasks.rados.rados.0.smithi076.stdout:3726: write oid 25 current snap is 437 2021-03-15T22:35:32.374 INFO:tasks.rados.rados.0.smithi076.stdout:3726: seq_num 1266 ranges {477662=579832,1683380=555768,2966450=311212} 2021-03-15T22:35:32.377 INFO:tasks.rados.rados.0.smithi076.stdout:3726: writing smithi07694878-25 from 477662 to 1057494 tid 1 2021-03-15T22:35:32.384 INFO:tasks.rados.rados.0.smithi076.stdout:3726: writing smithi07694878-25 from 1683380 to 2239148 tid 2 2021-03-15T22:35:32.390 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:32.391 INFO:tasks.rados.rados.0.smithi076.stdout:3726: writing smithi07694878-25 from 2966450 to 3277662 tid 3 2021-03-15T22:35:32.392 INFO:tasks.rados.rados.0.smithi076.stdout:3727: write oid 33 current snap is 437 2021-03-15T22:35:32.392 INFO:tasks.rados.rados.0.smithi076.stdout:3723: finishing copy_from to smithi07694878-45 2021-03-15T22:35:32.392 INFO:tasks.rados.rados.0.smithi076.stdout:3723: got expected ENOENT (src dne) 2021-03-15T22:35:32.392 INFO:tasks.rados.rados.0.smithi076.stdout:3727: seq_num 1267 ranges {494469=655081,1741862=415779,2710226=584243} 2021-03-15T22:35:32.397 INFO:tasks.rados.rados.0.smithi076.stdout:3727: writing smithi07694878-33 from 494469 to 1149550 tid 1 2021-03-15T22:35:32.402 INFO:tasks.rados.rados.0.smithi076.stdout:3727: writing smithi07694878-33 from 1741862 to 2157641 tid 2 2021-03-15T22:35:32.410 INFO:tasks.rados.rados.0.smithi076.stdout:3727: writing smithi07694878-33 from 2710226 to 3294469 tid 3 2021-03-15T22:35:32.410 INFO:tasks.rados.rados.0.smithi076.stdout:3728: rollback oid 24 current snap is 437 2021-03-15T22:35:32.410 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 427 2021-03-15T22:35:32.411 INFO:tasks.rados.rados.0.smithi076.stdout:3723: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:35:32.411 INFO:tasks.rados.rados.0.smithi076.stdout:3725: finishing rollback tid 1 to smithi07694878-6 2021-03-15T22:35:32.411 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 6 v 3080 (ObjNum 0 snap 0 seq_num 0) dirty dne 2021-03-15T22:35:32.411 INFO:tasks.rados.rados.0.smithi076.stdout:3726: finishing write tid 1 to smithi07694878-25 2021-03-15T22:35:32.412 INFO:tasks.rados.rados.0.smithi076.stdout:3726: finishing write tid 2 to smithi07694878-25 2021-03-15T22:35:32.412 INFO:tasks.rados.rados.0.smithi076.stdout:3726: finishing write tid 3 to smithi07694878-25 2021-03-15T22:35:32.412 INFO:tasks.rados.rados.0.smithi076.stdout:3726: finishing write tid 4 to smithi07694878-25 2021-03-15T22:35:32.412 INFO:tasks.rados.rados.0.smithi076.stdout:3726: finishing write tid 5 to smithi07694878-25 2021-03-15T22:35:32.412 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 3166 (ObjNum 1266 snap 437 seq_num 1266) dirty exists 2021-03-15T22:35:32.413 INFO:tasks.rados.rados.0.smithi076.stdout:3726: left oid 25 (ObjNum 1266 snap 437 seq_num 1266) 2021-03-15T22:35:32.413 INFO:tasks.rados.rados.0.smithi076.stdout:3727: finishing write tid 1 to smithi07694878-33 2021-03-15T22:35:32.413 INFO:tasks.rados.rados.0.smithi076.stdout:3727: finishing write tid 2 to smithi07694878-33 2021-03-15T22:35:32.414 INFO:tasks.rados.rados.0.smithi076.stdout:3723: done (4 left) 2021-03-15T22:35:32.414 INFO:tasks.rados.rados.0.smithi076.stdout:3725: done (3 left) 2021-03-15T22:35:32.414 INFO:tasks.rados.rados.0.smithi076.stdout:3726: done (2 left) 2021-03-15T22:35:32.415 INFO:tasks.rados.rados.0.smithi076.stdout:3729: rollback oid 19 current snap is 437 2021-03-15T22:35:32.415 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 19 to 370 2021-03-15T22:35:32.415 INFO:tasks.rados.rados.0.smithi076.stdout:3730: delete oid 13 current snap is 437 2021-03-15T22:35:32.416 INFO:tasks.rados.rados.0.smithi076.stdout:3728: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:35:32.416 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1740 (ObjNum 1953066355 snap 925919592 seq_num 942946614) dirty dne 2021-03-15T22:35:32.417 INFO:tasks.rados.rados.0.smithi076.stdout:3728: done (3 left) 2021-03-15T22:35:32.417 INFO:tasks.rados.rados.0.smithi076.stdout:3730: done (2 left) 2021-03-15T22:35:32.417 INFO:tasks.rados.rados.0.smithi076.stdout:3731: write oid 10 current snap is 437 2021-03-15T22:35:32.418 INFO:tasks.rados.rados.0.smithi076.stdout:3731: seq_num 1269 ranges {528083=405579,1342473=419448,2248719=733949} 2021-03-15T22:35:32.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:32.419 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:32.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:32.420 INFO:tasks.rados.rados.0.smithi076.stdout:3731: writing smithi07694878-10 from 528083 to 933662 tid 1 2021-03-15T22:35:32.422 INFO:tasks.rados.rados.0.smithi076.stdout:3731: writing smithi07694878-10 from 1342473 to 1761921 tid 2 2021-03-15T22:35:32.431 INFO:tasks.rados.rados.0.smithi076.stdout:3731: writing smithi07694878-10 from 2248719 to 2982668 tid 3 2021-03-15T22:35:32.431 INFO:tasks.rados.rados.0.smithi076.stdout:3732: rollback oid 31 current snap is 437 2021-03-15T22:35:32.431 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 31 to 427 2021-03-15T22:35:32.431 INFO:tasks.rados.rados.0.smithi076.stdout:3727: finishing write tid 3 to smithi07694878-33 2021-03-15T22:35:32.432 INFO:tasks.rados.rados.0.smithi076.stdout:3727: finishing write tid 4 to smithi07694878-33 2021-03-15T22:35:32.432 INFO:tasks.rados.rados.0.smithi076.stdout:3727: finishing write tid 5 to smithi07694878-33 2021-03-15T22:35:32.432 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 2681 (ObjNum 1267 snap 437 seq_num 1267) dirty exists 2021-03-15T22:35:32.432 INFO:tasks.rados.rados.0.smithi076.stdout:3727: left oid 33 (ObjNum 1267 snap 437 seq_num 1267) 2021-03-15T22:35:32.432 INFO:tasks.rados.rados.0.smithi076.stdout:3729: finishing rollback tid 0 to smithi07694878-19 2021-03-15T22:35:32.433 INFO:tasks.rados.rados.0.smithi076.stdout:3729: finishing rollback tid 1 to smithi07694878-19 2021-03-15T22:35:32.433 INFO:tasks.rados.rados.0.smithi076.stdout:3729: finishing rollback tid 2 to smithi07694878-19 2021-03-15T22:35:32.433 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2826 (ObjNum 1075 snap 366 seq_num 1075) dirty exists 2021-03-15T22:35:32.434 INFO:tasks.rados.rados.0.smithi076.stdout:3731: finishing write tid 1 to smithi07694878-10 2021-03-15T22:35:32.434 INFO:tasks.rados.rados.0.smithi076.stdout:3731: finishing write tid 2 to smithi07694878-10 2021-03-15T22:35:32.434 INFO:tasks.rados.rados.0.smithi076.stdout:3727: done (3 left) 2021-03-15T22:35:32.434 INFO:tasks.rados.rados.0.smithi076.stdout:3729: done (2 left) 2021-03-15T22:35:32.434 INFO:tasks.rados.rados.0.smithi076.stdout:3733: read oid 44 snap -1 2021-03-15T22:35:32.435 INFO:tasks.rados.rados.0.smithi076.stdout:3733: expect (ObjNum 1177 snap 411 seq_num 1177) 2021-03-15T22:35:32.435 INFO:tasks.rados.rados.0.smithi076.stdout:3734: read oid 24 snap -1 2021-03-15T22:35:32.435 INFO:tasks.rados.rados.0.smithi076.stdout:3734: expect deleted 2021-03-15T22:35:32.435 INFO:tasks.rados.rados.0.smithi076.stdout:3735: read oid 8 snap -1 2021-03-15T22:35:32.436 INFO:tasks.rados.rados.0.smithi076.stdout:3735: expect deleted 2021-03-15T22:35:32.436 INFO:tasks.rados.rados.0.smithi076.stdout:3736: write oid 7 current snap is 437 2021-03-15T22:35:32.436 INFO:tasks.rados.rados.0.smithi076.stdout:3736: seq_num 1270 ranges {544890=597181,1917308=563106,3076142=268748} 2021-03-15T22:35:32.436 INFO:tasks.rados.rados.0.smithi076.stdout:3736: writing smithi07694878-7 from 544890 to 1142071 tid 1 2021-03-15T22:35:32.443 INFO:tasks.rados.rados.0.smithi076.stdout:3736: writing smithi07694878-7 from 1917308 to 2480414 tid 2 2021-03-15T22:35:32.449 INFO:tasks.rados.rados.0.smithi076.stdout:3736: writing smithi07694878-7 from 3076142 to 3344890 tid 3 2021-03-15T22:35:32.449 INFO:tasks.rados.rados.0.smithi076.stdout:3737: read oid 45 snap -1 2021-03-15T22:35:32.450 INFO:tasks.rados.rados.0.smithi076.stdout:3737: expect (ObjNum 1034 snap 346 seq_num 1034) 2021-03-15T22:35:32.450 INFO:tasks.rados.rados.0.smithi076.stdout:3738: copy_from oid 17 from oid 18 current snap is 437 2021-03-15T22:35:32.450 INFO:tasks.rados.rados.0.smithi076.stdout:3732: finishing rollback tid 1 to smithi07694878-31 2021-03-15T22:35:32.450 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 2793 (ObjNum 0 snap 21896 seq_num 262222912) dirty dne 2021-03-15T22:35:32.451 INFO:tasks.rados.rados.0.smithi076.stdout:3733: expect (ObjNum 1177 snap 411 seq_num 1177) 2021-03-15T22:35:32.478 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:35:32.478 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:35:32.521 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:32.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:32 smithi076 conmon[56514]: audit 2021-03-15T22:35:30.301411+0000 2021-03-15T22:35:32.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:32 smithi076 conmon[56514]: mgr.y (mgr.14144) 1080 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:32.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:32 smithi076 conmon[56514]: audit 2021-03-15T22:35:31.109031+0000 mon.a (mon.0) 2431 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:35:32.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:32 smithi076 conmon[56514]: cluster 2021-03-15T22:35:31.360416+0000 mon.a (mon.0) 2432 : cluster [DBG] osdmap e976: 8 total, 8 up, 8 in 2021-03-15T22:35:32.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:32 smithi076 conmon[62156]: audit 2021-03-15T22:35:30.301411+0000 mgr.y (mgr 2021-03-15T22:35:32.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:32 smithi076 conmon[62156]: .14144) 1080 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:32.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:32 smithi076 conmon[62156]: audit 2021-03-15T22:35:31.109031+0000 mon.a (mon 2021-03-15T22:35:32.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:32 smithi076 conmon[62156]: .0) 2431 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:35:32.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:32 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:35:32.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:32 smithi076 conmon[62156]: -03-15T22:35:31.360416+0000 mon.a (mon.0 2021-03-15T22:35:32.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:32 smithi076 conmon[62156]: ) 2432 : cluster [DBG] osdmap e976: 8 total, 8 up, 8 in 2021-03-15T22:35:32.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:32.549 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:32.549 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:32.573 INFO:tasks.rados.rados.0.smithi076.stdout:3731: finishing write tid 3 to smithi07694878-10 2021-03-15T22:35:32.573 INFO:tasks.rados.rados.0.smithi076.stdout:3732: done (7 left) 2021-03-15T22:35:32.573 INFO:tasks.rados.rados.0.smithi076.stdout:3733: done (6 left) 2021-03-15T22:35:32.574 INFO:tasks.rados.rados.0.smithi076.stdout:3734: done (5 left) 2021-03-15T22:35:32.574 INFO:tasks.rados.rados.0.smithi076.stdout:3735: done (4 left) 2021-03-15T22:35:32.574 INFO:tasks.rados.rados.0.smithi076.stdout:3739: copy_from oid 2 from oid 43 current snap is 437 2021-03-15T22:35:32.574 INFO:tasks.rados.rados.0.smithi076.stdout:3731: finishing write tid 4 to smithi07694878-10 2021-03-15T22:35:32.575 INFO:tasks.rados.rados.0.smithi076.stdout:3731: finishing write tid 5 to smithi07694878-10 2021-03-15T22:35:32.575 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 3170 (ObjNum 1269 snap 437 seq_num 1269) dirty exists 2021-03-15T22:35:32.575 INFO:tasks.rados.rados.0.smithi076.stdout:3731: left oid 10 (ObjNum 1269 snap 437 seq_num 1269) 2021-03-15T22:35:32.575 INFO:tasks.rados.rados.0.smithi076.stdout:3736: finishing write tid 1 to smithi07694878-7 2021-03-15T22:35:32.576 INFO:tasks.rados.rados.0.smithi076.stdout:3736: finishing write tid 2 to smithi07694878-7 2021-03-15T22:35:32.576 INFO:tasks.rados.rados.0.smithi076.stdout:3736: finishing write tid 3 to smithi07694878-7 2021-03-15T22:35:32.576 INFO:tasks.rados.rados.0.smithi076.stdout:3736: finishing write tid 4 to smithi07694878-7 2021-03-15T22:35:32.576 INFO:tasks.rados.rados.0.smithi076.stdout:3736: finishing write tid 5 to smithi07694878-7 2021-03-15T22:35:32.577 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2830 (ObjNum 1270 snap 437 seq_num 1270) dirty exists 2021-03-15T22:35:32.577 INFO:tasks.rados.rados.0.smithi076.stdout:3736: left oid 7 (ObjNum 1270 snap 437 seq_num 1270) 2021-03-15T22:35:32.577 INFO:tasks.rados.rados.0.smithi076.stdout:3737: expect (ObjNum 1034 snap 346 seq_num 1034) 2021-03-15T22:35:32.651 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:32.676 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:32.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:32.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:32.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:32.788 INFO:tasks.rados.rados.0.smithi076.stdout:3738: finishing copy_from to smithi07694878-17 2021-03-15T22:35:32.788 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 2446 (ObjNum 1265 snap 437 seq_num 1265) dirty exists 2021-03-15T22:35:32.788 INFO:tasks.rados.rados.0.smithi076.stdout:3738: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:35:32.789 INFO:tasks.rados.rados.0.smithi076.stdout:3731: done (4 left) 2021-03-15T22:35:32.789 INFO:tasks.rados.rados.0.smithi076.stdout:3736: done (3 left) 2021-03-15T22:35:32.789 INFO:tasks.rados.rados.0.smithi076.stdout:3737: done (2 left) 2021-03-15T22:35:32.789 INFO:tasks.rados.rados.0.smithi076.stdout:3738: done (1 left) 2021-03-15T22:35:32.789 INFO:tasks.rados.rados.0.smithi076.stdout:3740: snap_create 2021-03-15T22:35:32.790 INFO:tasks.rados.rados.0.smithi076.stdout:3739: finishing copy_from to smithi07694878-2 2021-03-15T22:35:32.790 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 2438 (ObjNum 1256 snap 436 seq_num 1256) dirty exists 2021-03-15T22:35:32.790 INFO:tasks.rados.rados.0.smithi076.stdout:3739: finishing copy_from racing read to smithi07694878-2 2021-03-15T22:35:32.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:32.809 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:32.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:32.911 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:32.933 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:32.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:32.934 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:33.035 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:33.057 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:33.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:33.059 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:33.159 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:33.164 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_ops_in_flight 2021-03-15T22:35:33.183 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:33.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:33.184 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:33.222 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:35:33.238 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:35:33.255 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:35:33.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[56514]: cluster 2021-03-15T22:35:31.221014+0000 mgr.y (mgr.14144) 1081 : cluster [DBG] pgmap v1610: 158 pgs: 158 active+clean; 621 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.0 MiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:35:33.267 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[56514]: cluster 2021-03-15T22:35:32.107222+0000 mon.a (mon.0) 2433 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:35:33.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[56514]: cluster 2021-03-15T22:35:32.364105+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in 2021-03-15T22:35:33.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[56514]: cluster 2021-03-15T22:35:32.892016+0000 osd.2 (osd.2) 47 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:33.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[56514]: cluster 2021-03-15T22:35:32.895348+0000 osd.2 (osd.2) 48 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:33.268 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[56514]: audit 2021-03-15T22:35:32.959096+0000 mon.c (mon.1) 80 : audit [INF] from='client.? 172.21.15.76:0/3743083824' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "32"}]: dispatch 2021-03-15T22:35:33.269 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[56514]: audit 2021-03-15T22:35:32.959438+0000 mon.a (mon.0) 2435 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "32"}]: dispatch 2021-03-15T22:35:33.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[62156]: cluster 2021-03-15T22:35:31.221014+0000 mgr.y (mgr.14144) 1081 : cluster [DBG] pgmap v1610: 158 pgs: 158 active+clean; 621 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.0 MiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:35:33.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[62156]: cluster 2021-03-15T22:35:32.107222+0000 mon.a (mon.0) 2433 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:35:33.269 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[62156]: cluster 2021-03-15T22:35:32.364105+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in 2021-03-15T22:35:33.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[62156]: cluster 2021-03-15T22:35:32.892016+0000 osd.2 (osd.2) 47 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:33.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[62156]: cluster 2021-03-15T22:35:32.895348+0000 osd.2 (osd.2) 48 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:33.270 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[62156]: audit 2021-03-15T22:35:32.959096+0000 mon.c (mon.1) 80 : audit [INF] from='client.? 172.21.15.76:0/3743083824' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "32"}]: dispatch 2021-03-15T22:35:33.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[62156]: audit 2021-03-15T22:35:32.959438+0000 mon.a (mon.0) 2435 2021-03-15T22:35:33.271 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:33 smithi076 conmon[62156]: : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "32"}]: dispatch 2021-03-15T22:35:33.272 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:35:33.285 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:33.289 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:35:33.306 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:35:33.307 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:33.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:33.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:33.323 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:35:33.340 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:35:33.342 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:35:33.342 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:35:33.342 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:35:33.342 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:35:33.343 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:35:33.343 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:35:33.343 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:35:33.343 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:35:33.369 INFO:tasks.rados.rados.0.smithi076.stdout:3739: done (1 left) 2021-03-15T22:35:33.369 INFO:tasks.rados.rados.0.smithi076.stdout:3740: done (0 left) 2021-03-15T22:35:33.369 INFO:tasks.rados.rados.0.smithi076.stdout:3741: write (excl) oid 38 current snap is 438 2021-03-15T22:35:33.370 INFO:tasks.rados.rados.0.smithi076.stdout:3741: seq_num 1271 ranges {561697=672430,1975790=706764,3103565=258132} 2021-03-15T22:35:33.371 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 6 pg_num to 32 2021-03-15T22:35:33.375 INFO:tasks.rados.rados.0.smithi076.stdout:3741: writing smithi07694878-38 from 561697 to 1234127 tid 1 2021-03-15T22:35:33.384 INFO:tasks.rados.rados.0.smithi076.stdout:3741: writing smithi07694878-38 from 1975790 to 2682554 tid 2 2021-03-15T22:35:33.388 INFO:tasks.rados.rados.0.smithi076.stdout:3741: writing smithi07694878-38 from 3103565 to 3361697 tid 3 2021-03-15T22:35:33.389 INFO:tasks.rados.rados.0.smithi076.stdout:3742: snap_create 2021-03-15T22:35:33.389 INFO:tasks.rados.rados.0.smithi076.stdout:3741: finishing write tid 1 to smithi07694878-38 2021-03-15T22:35:33.389 INFO:tasks.rados.rados.0.smithi076.stdout:3741: finishing write tid 2 to smithi07694878-38 2021-03-15T22:35:33.392 INFO:tasks.rados.rados.0.smithi076.stdout:3741: finishing write tid 3 to smithi07694878-38 2021-03-15T22:35:33.392 INFO:tasks.rados.rados.0.smithi076.stdout:3741: finishing write tid 4 to smithi07694878-38 2021-03-15T22:35:33.393 INFO:tasks.rados.rados.0.smithi076.stdout:3741: finishing write tid 5 to smithi07694878-38 2021-03-15T22:35:33.393 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2834 (ObjNum 1271 snap 438 seq_num 1271) dirty exists 2021-03-15T22:35:33.394 INFO:tasks.rados.rados.0.smithi076.stdout:3741: left oid 38 (ObjNum 1271 snap 438 seq_num 1271) 2021-03-15T22:35:33.409 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:33.432 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:33.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:33.434 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:33.444 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:33.535 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:33.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: cluster 2021-03-15T22:35:31.221014+0000 mgr.y (mgr.14144) 1081 2021-03-15T22:35:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: : cluster [DBG] pgmap v1610: 158 pgs: 158 active+clean; 621 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.0 MiB/s wr, 11 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:35:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: cluster 2021-03-15T22:35:32.107222+0000 mon.a (mon.0) 2433 : cluster [INF] Health check cleared: PG_BACKFILL_FULL (was: Low space hindering backfill (add storage if this doesn't resolve itself): 1 pg backfill_toofull) 2021-03-15T22:35:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: cluster 2021-03-15T22:35:32.364105+0000 mon.a (mon.0) 2434 : cluster [DBG] osdmap e977: 8 total, 8 up, 8 in 2021-03-15T22:35:33.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: cluster 2021-03-15T22:35:32.892016+0000 osd.2 (osd.2) 47 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: cluster 2021-03-15T22:35:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: 2021-03-15T22:35:32.895348+0000 osd.2 (osd.2) 48 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: audit 2021-03-15T22:35:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: 2021-03-15T22:35:32. 2021-03-15T22:35:33.538 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: 959096+0000 mon.c (mon.1) 80 : audit 2021-03-15T22:35:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/3743083824' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "32"}]: dispatch 2021-03-15T22:35:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: audit 2021-03-15T22:35:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: 2021-03-15T22:35:32 2021-03-15T22:35:33.539 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: .959438+0000 mon.a (mon.0) 2435 : 2021-03-15T22:35:33.540 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:33 smithi125 conmon[56739]: audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "32"}]: dispatch 2021-03-15T22:35:33.561 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:33.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:33.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:33.664 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:33.687 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:33.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:33.689 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:33.735 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:35:33.790 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:33.811 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:33.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:33.920 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:33.943 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:33.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:33.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:34.046 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:34.068 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:34.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:34.069 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:34.170 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:34.192 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:34.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:34.194 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:34.286 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:34 smithi125 conmon[56739]: audit 2021-03-15T22:35:33.202964+0000 mon.a (mon.0) 2436 : audit [DBG] from='client.? 172.21.15.76:0/1658644825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:34 smithi125 conmon[56739]: cluster 2021-03-15T22:35:33.221386+0000 mgr.y (mgr.14144) 1082 : cluster [DBG] pgmap v1613: 158 pgs: 158 active+clean; 630 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:35:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:34 smithi125 conmon[56739]: audit 2021-03-15T22:35:33.363980+0000 mon.a (mon.0) 2437 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "32"}]': finished 2021-03-15T22:35:34.287 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:34 smithi125 conmon[56739]: cluster 2021-03-15T22:35:33.364042+0000 mon.a (mon.0) 2438 : cluster [DBG] osdmap e978: 8 total, 8 up, 8 in 2021-03-15T22:35:34.295 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:34.317 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:34.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:34.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:34.371 INFO:tasks.rados.rados.0.smithi076.stdout:3741: done (1 left) 2021-03-15T22:35:34.371 INFO:tasks.rados.rados.0.smithi076.stdout:3742: done (0 left) 2021-03-15T22:35:34.371 INFO:tasks.rados.rados.0.smithi076.stdout:3743: write (excl) oid 28 current snap is 439 2021-03-15T22:35:34.372 INFO:tasks.rados.rados.0.smithi076.stdout:3743: seq_num 1272 ranges {578504=747679,2034272=566775,3130988=247516} 2021-03-15T22:35:34.377 INFO:tasks.rados.rados.0.smithi076.stdout:3743: writing smithi07694878-28 from 578504 to 1326183 tid 1 2021-03-15T22:35:34.384 INFO:tasks.rados.rados.0.smithi076.stdout:3743: writing smithi07694878-28 from 2034272 to 2601047 tid 2 2021-03-15T22:35:34.385 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[56514]: audit 2021-03-15T22:35:33.202964+0000 2021-03-15T22:35:34.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[56514]: mon.a (mon.0) 2436 : audit [DBG] from='client.? 172.21.15.76:0/1658644825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:34.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[56514]: cluster 2021-03-15T22:35:33.221386+0000 mgr.y (mgr.14144) 1082 : cluster [DBG] pgmap v1613: 158 pgs: 158 active+clean; 630 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:35:34.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[56514]: audit 2021-03-15T22:35:33.363980+0000 mon.a (mon.0) 2437 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "32"}]': finished 2021-03-15T22:35:34.386 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[56514]: cluster 2021-03-15T22:35:33.364042+0000 mon.a (mon.0) 2438 : cluster [DBG] osdmap e978: 8 total, 8 up, 8 in 2021-03-15T22:35:34.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: audit 2021-03-15T22:35:33. 2021-03-15T22:35:34.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: 202964+0000 mon.a (mon.0) 2436 : audit [DBG] from='client.? 172.21.15.76:0/1658644825' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:34.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: cluster 2021-03-15T22:35:33. 2021-03-15T22:35:34.387 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: 221386+0000 mgr.y (mgr.14144) 1082 : cluster [DBG] 2021-03-15T22:35:34.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: pgmap v1613: 158 pgs: 158 active+clean; 630 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 2.7 MiB/s wr, 19 op/s; 0 B/s, 0 objects/s recovering 2021-03-15T22:35:34.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: audit 2021-03-15T 2021-03-15T22:35:34.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: 22:35:33.363980+0000 mon.a (mon.0 2021-03-15T22:35:34.388 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: ) 2437 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num", "val": "32"}]': finished 2021-03-15T22:35:34.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:35:34.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: 03-15T22:35:33.364042+0000 2021-03-15T22:35:34.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: mon.a (mon.0) 2438 : cluster 2021-03-15T22:35:34.389 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:34 smithi076 conmon[62156]: [DBG] osdmap e978: 8 total, 8 up, 8 in 2021-03-15T22:35:34.390 INFO:tasks.rados.rados.0.smithi076.stdout:3743: writing smithi07694878-28 from 3130988 to 3378504 tid 3 2021-03-15T22:35:34.391 INFO:tasks.rados.rados.0.smithi076.stdout:3744: read oid 5 snap -1 2021-03-15T22:35:34.391 INFO:tasks.rados.rados.0.smithi076.stdout:3744: expect (ObjNum 1238 snap 427 seq_num 1238) 2021-03-15T22:35:34.391 INFO:tasks.rados.rados.0.smithi076.stdout:3745: write (excl) oid 38 current snap is 439 2021-03-15T22:35:34.392 INFO:tasks.rados.rados.0.smithi076.stdout:3743: finishing write tid 1 to smithi07694878-28 2021-03-15T22:35:34.392 INFO:tasks.rados.rados.0.smithi076.stdout:3745: seq_num 1273 ranges {595311=422928,1576401=710433,3042058=353253} 2021-03-15T22:35:34.395 INFO:tasks.rados.rados.0.smithi076.stdout:3745: writing smithi07694878-38 from 595311 to 1018239 tid 1 2021-03-15T22:35:34.402 INFO:tasks.rados.rados.0.smithi076.stdout:3745: writing smithi07694878-38 from 1576401 to 2286834 tid 2 2021-03-15T22:35:34.408 INFO:tasks.rados.rados.0.smithi076.stdout:3745: writing smithi07694878-38 from 3042058 to 3395311 tid 3 2021-03-15T22:35:34.409 INFO:tasks.rados.rados.0.smithi076.stdout:3746: write (excl) oid 42 current snap is 439 2021-03-15T22:35:34.409 INFO:tasks.rados.rados.0.smithi076.stdout:3743: finishing write tid 2 to smithi07694878-28 2021-03-15T22:35:34.409 INFO:tasks.rados.rados.0.smithi076.stdout:3746: seq_num 1274 ranges {612118=498177,1634883=570444,2669481=706662} 2021-03-15T22:35:34.414 INFO:tasks.rados.rados.0.smithi076.stdout:3746: writing smithi07694878-42 from 612118 to 1110295 tid 1 2021-03-15T22:35:34.419 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:34.421 INFO:tasks.rados.rados.0.smithi076.stdout:3746: writing smithi07694878-42 from 1634883 to 2205327 tid 2 2021-03-15T22:35:34.428 INFO:tasks.rados.rados.0.smithi076.stdout:3746: writing smithi07694878-42 from 2669481 to 3376143 tid 3 2021-03-15T22:35:34.428 INFO:tasks.rados.rados.0.smithi076.stdout:3747: read oid 11 snap -1 2021-03-15T22:35:34.429 INFO:tasks.rados.rados.0.smithi076.stdout:3747: expect (ObjNum 1250 snap 431 seq_num 1250) 2021-03-15T22:35:34.429 INFO:tasks.rados.rados.0.smithi076.stdout:3748: rollback oid 24 current snap is 439 2021-03-15T22:35:34.429 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 430 2021-03-15T22:35:34.429 INFO:tasks.rados.rados.0.smithi076.stdout:3744: expect (ObjNum 1238 snap 427 seq_num 1238) 2021-03-15T22:35:34.450 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:34.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:34.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:34.552 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:34.575 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:34.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:34.576 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:34.675 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:34.677 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:34.693 INFO:tasks.rados.rados.0.smithi076.stdout:3743: finishing write tid 3 to smithi07694878-28 2021-03-15T22:35:34.694 INFO:tasks.rados.rados.0.smithi076.stdout:3743: finishing write tid 4 to smithi07694878-28 2021-03-15T22:35:34.694 INFO:tasks.rados.rados.0.smithi076.stdout:3743: finishing write tid 5 to smithi07694878-28 2021-03-15T22:35:34.694 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 3282 (ObjNum 1272 snap 439 seq_num 1272) dirty exists 2021-03-15T22:35:34.694 INFO:tasks.rados.rados.0.smithi076.stdout:3743: left oid 28 (ObjNum 1272 snap 439 seq_num 1272) 2021-03-15T22:35:34.695 INFO:tasks.rados.rados.0.smithi076.stdout:3745: finishing write tid 1 to smithi07694878-38 2021-03-15T22:35:34.695 INFO:tasks.rados.rados.0.smithi076.stdout:3745: finishing write tid 2 to smithi07694878-38 2021-03-15T22:35:34.695 INFO:tasks.rados.rados.0.smithi076.stdout:3743: done (5 left) 2021-03-15T22:35:34.695 INFO:tasks.rados.rados.0.smithi076.stdout:3744: done (4 left) 2021-03-15T22:35:34.695 INFO:tasks.rados.rados.0.smithi076.stdout:3749: rollback oid 12 current snap is 439 2021-03-15T22:35:34.696 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 12 to 411 2021-03-15T22:35:34.696 INFO:tasks.rados.rados.0.smithi076.stdout:3745: finishing write tid 3 to smithi07694878-38 2021-03-15T22:35:34.696 INFO:tasks.rados.rados.0.smithi076.stdout:3745: finishing write tid 4 to smithi07694878-38 2021-03-15T22:35:34.696 INFO:tasks.rados.rados.0.smithi076.stdout:3745: finishing write tid 5 to smithi07694878-38 2021-03-15T22:35:34.697 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2838 (ObjNum 1273 snap 439 seq_num 1273) dirty exists 2021-03-15T22:35:34.697 INFO:tasks.rados.rados.0.smithi076.stdout:3745: left oid 38 (ObjNum 1273 snap 439 seq_num 1273) 2021-03-15T22:35:34.697 INFO:tasks.rados.rados.0.smithi076.stdout:3746: finishing write tid 1 to smithi07694878-42 2021-03-15T22:35:34.697 INFO:tasks.rados.rados.0.smithi076.stdout:3746: finishing write tid 2 to smithi07694878-42 2021-03-15T22:35:34.697 INFO:tasks.rados.rados.0.smithi076.stdout:3748: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:35:34.698 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1740 (ObjNum 1611153536 snap 32564 seq_num 1610614992) dirty dne 2021-03-15T22:35:34.698 INFO:tasks.rados.rados.0.smithi076.stdout:3747: expect (ObjNum 1250 snap 431 seq_num 1250) 2021-03-15T22:35:34.699 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:34.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:34.701 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:34.802 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:34.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:34.827 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:34.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:34.854 INFO:tasks.rados.rados.0.smithi076.stdout:3746: finishing write tid 3 to smithi07694878-42 2021-03-15T22:35:34.855 INFO:tasks.rados.rados.0.smithi076.stdout:3745: done (4 left) 2021-03-15T22:35:34.855 INFO:tasks.rados.rados.0.smithi076.stdout:3747: done (3 left) 2021-03-15T22:35:34.855 INFO:tasks.rados.rados.0.smithi076.stdout:3748: done (2 left) 2021-03-15T22:35:34.856 INFO:tasks.rados.rados.0.smithi076.stdout:3750: copy_from oid 16 from oid 19 current snap is 439 2021-03-15T22:35:34.856 INFO:tasks.rados.rados.0.smithi076.stdout:3746: finishing write tid 4 to smithi07694878-42 2021-03-15T22:35:34.856 INFO:tasks.rados.rados.0.smithi076.stdout:3746: finishing write tid 5 to smithi07694878-42 2021-03-15T22:35:34.856 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 3287 (ObjNum 1274 snap 439 seq_num 1274) dirty exists 2021-03-15T22:35:34.856 INFO:tasks.rados.rados.0.smithi076.stdout:3746: left oid 42 (ObjNum 1274 snap 439 seq_num 1274) 2021-03-15T22:35:34.857 INFO:tasks.rados.rados.0.smithi076.stdout:3749: finishing rollback tid 1 to smithi07694878-12 2021-03-15T22:35:34.857 INFO:tasks.rados.rados.0.smithi076.stdout:3749: finishing rollback tid 2 to smithi07694878-12 2021-03-15T22:35:34.857 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 3171 (ObjNum 1160 snap 407 seq_num 1160) dirty exists 2021-03-15T22:35:34.857 INFO:tasks.rados.rados.0.smithi076.stdout:3746: done (2 left) 2021-03-15T22:35:34.857 INFO:tasks.rados.rados.0.smithi076.stdout:3749: done (1 left) 2021-03-15T22:35:34.858 INFO:tasks.rados.rados.0.smithi076.stdout:3751: delete oid 47 current snap is 439 2021-03-15T22:35:34.858 INFO:tasks.rados.rados.0.smithi076.stdout:3751: done (1 left) 2021-03-15T22:35:34.859 INFO:tasks.rados.rados.0.smithi076.stdout:3752: read oid 34 snap 426 2021-03-15T22:35:34.859 INFO:tasks.rados.rados.0.smithi076.stdout:3752: expect (ObjNum 1235 snap 426 seq_num 1235) 2021-03-15T22:35:34.859 INFO:tasks.rados.rados.0.smithi076.stdout:3753: snap_remove snap 370 2021-03-15T22:35:34.928 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:34.951 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:34.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:34.953 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:35.014 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:35:35.014 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:35:35.015 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:35:35.015 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:35:35.053 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:35.078 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:35.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:35.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:35.181 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:35.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:35.204 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:35.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:35.305 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:35.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:35.334 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:35.334 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:35.354 INFO:tasks.rados.rados.0.smithi076.stdout:3753: done (2 left) 2021-03-15T22:35:35.355 INFO:tasks.rados.rados.0.smithi076.stdout:3754: snap_create 2021-03-15T22:35:35.355 INFO:tasks.rados.rados.0.smithi076.stdout:3752: expect (ObjNum 1235 snap 426 seq_num 1235) 2021-03-15T22:35:35.436 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:35.452 INFO:tasks.rados.rados.0.smithi076.stdout:3750: finishing copy_from to smithi07694878-16 2021-03-15T22:35:35.453 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 16 v 2826 (ObjNum 1075 snap 366 seq_num 1075) dirty exists 2021-03-15T22:35:35.453 INFO:tasks.rados.rados.0.smithi076.stdout:3750: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:35:35.458 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:35.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:35.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:35.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[56514]: cluster 2021-03-15T22:35:34.368436+0000 mon.a (mon.0) 2439 : cluster [DBG] osdmap e979: 8 total, 8 up, 8 in 2021-03-15T22:35:35.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[56514]: audit 2021-03-15T22:35:35.222642+0000 mon.a (mon.0) 2440 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "32"}]: dispatch 2021-03-15T22:35:35.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[56514]: audit 2021-03-15T22:35:35.340052+0000 mon.a (mon.0) 2441 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "32"}]': finished 2021-03-15T22:35:35.519 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[56514]: cluster 2021-03-15T22:35:35.340129+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in 2021-03-15T22:35:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: cluster 2021-03-15T22:35:34.368436+0000 2021-03-15T22:35:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: mon.a (mon.0) 2439 : cluster [DBG] osdmap e979: 8 total, 8 up, 8 in 2021-03-15T22:35:35.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: audit 2021-03-15T22:35 2021-03-15T22:35:35.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: :35.222642+0000 mon.a (mon.0) 2440 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "32"}]: dispatch 2021-03-15T22:35:35.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: audit 2021-03-15T22:35:35.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: 2021-03- 2021-03-15T22:35:35.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: 15T22:35:35.340052+0000 mon.a 2021-03-15T22:35:35.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: (mon.0) 2441 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "32"}]': finished 2021-03-15T22:35:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: 2021-03-15T22:35:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:35:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: :35:35.340129+0000 mon.a (mon. 2021-03-15T22:35:35.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:35 smithi076 conmon[62156]: 0) 2442 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in 2021-03-15T22:35:35.562 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:35.585 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:35.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:35.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:35.688 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:35.700 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:35 smithi125 conmon[56739]: cluster 2021-03-15T22:35:34.368436+0000 mon.a (mon.0) 2021-03-15T22:35:35.700 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:35 smithi125 conmon[56739]: 2439 : cluster [DBG] osdmap e979: 8 total, 8 up, 8 in 2021-03-15T22:35:35.700 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:35 smithi125 conmon[56739]: audit 2021-03-15T22:35:35.222642+0000 mon.a (mon.0) 2440 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "32"}]: dispatch 2021-03-15T22:35:35.700 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:35 smithi125 conmon[56739]: audit 2021-03-15T22:35:35.340052+0000 mon.a (mon.0) 2441 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num_actual", "val": "32"}]': finished 2021-03-15T22:35:35.701 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:35 smithi125 conmon[56739]: cluster 2021-03-15T22:35:35.340129+0000 mon.a (mon.0) 2442 : cluster [DBG] osdmap e980: 8 total, 8 up, 8 in 2021-03-15T22:35:35.711 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:35.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:35.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:35.725 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_blocked_ops 2021-03-15T22:35:35.814 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:35.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:35.838 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:35.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:35.940 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:35.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:35.966 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:35.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:36.040 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:36.073 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:36.097 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:36.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:36.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:36.200 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:36.223 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:36.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:36.225 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:36.326 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:36.353 INFO:tasks.rados.rados.0.smithi076.stdout:3750: done (2 left) 2021-03-15T22:35:36.353 INFO:tasks.rados.rados.0.smithi076.stdout:3752: done (1 left) 2021-03-15T22:35:36.353 INFO:tasks.rados.rados.0.smithi076.stdout:3754: done (0 left) 2021-03-15T22:35:36.353 INFO:tasks.rados.rados.0.smithi076.stdout:3755: delete oid 30 current snap is 440 2021-03-15T22:35:36.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:36.355 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:36.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:36.358 INFO:tasks.rados.rados.0.smithi076.stdout:3755: done (0 left) 2021-03-15T22:35:36.358 INFO:tasks.rados.rados.0.smithi076.stdout:3756: snap_create 2021-03-15T22:35:36.422 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:35:36.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:36 smithi076 conmon[56514]: cluster 2021-03-15T22:35:35.221840+0000 mgr.y 2021-03-15T22:35:36.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:36 smithi076 conmon[56514]: (mgr.14144) 1083 : cluster [DBG] pgmap v1616: 158 pgs: 158 active+clean; 623 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-03-15T22:35:36.447 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:36 smithi076 conmon[56514]: cluster 2021-03-15T22:35:36.345543+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e981: 8 total, 8 up, 8 in 2021-03-15T22:35:36.448 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:36 smithi076 conmon[62156]: cluster 2021-03-15T22:35:35.221840+0000 2021-03-15T22:35:36.448 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:36 smithi076 conmon[62156]: mgr.y (mgr.14144) 1083 : cluster [DBG] pgmap v1616: 158 pgs: 158 active+clean; 623 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-03-15T22:35:36.448 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:36 smithi076 conmon[62156]: cluster 2021-03-15T22:35:36.345543+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e981: 8 total, 8 up, 8 in 2021-03-15T22:35:36.457 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:36.479 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:36.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:36.481 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:36.483 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:36 smithi125 conmon[56739]: cluster 2021-03-15T22:35:35.221840+0000 mgr.y (mgr.14144) 1083 : cluster [DBG] pgmap v1616: 158 pgs: 158 active+clean; 623 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 2.4 MiB/s wr, 15 op/s 2021-03-15T22:35:36.483 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:36 smithi125 conmon[56739]: cluster 2021-03-15T22:35:36.345543+0000 mon.a (mon.0) 2443 : cluster [DBG] osdmap e981: 8 total, 8 up, 8 in 2021-03-15T22:35:36.582 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:36.604 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:36.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:36.606 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:36.704 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:36.707 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:36.730 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:36.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:36.731 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:36.832 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:36.854 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:36.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:36.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:36.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:36.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:36.980 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:36.980 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:37.081 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:37.103 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:37.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:37.104 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:37.205 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:37.226 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:37.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:37.228 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:37.329 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:37.344 INFO:tasks.rados.rados.0.smithi076.stdout:3756: done (0 left) 2021-03-15T22:35:37.344 INFO:tasks.rados.rados.0.smithi076.stdout:3757: delete oid 28 current snap is 441 2021-03-15T22:35:37.351 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:35:37.351 INFO:tasks.rados.rados.0.smithi076.stdout:3757: done (0 left) 2021-03-15T22:35:37.352 INFO:tasks.rados.rados.0.smithi076.stdout:3758: copy_from oid 39 from oid 1 current snap is 441 2021-03-15T22:35:37.352 INFO:tasks.rados.rados.0.smithi076.stdout:3759: copy_from oid 20 from oid 35 current snap is 441 2021-03-15T22:35:37.352 INFO:tasks.rados.rados.0.smithi076.stdout:3760: snap_create 2021-03-15T22:35:37.353 INFO:tasks.rados.rados.0.smithi076.stdout:3759: finishing copy_from to smithi07694878-20 2021-03-15T22:35:37.353 INFO:tasks.rados.rados.0.smithi076.stdout:3759: got expected ENOENT (src dne) 2021-03-15T22:35:37.353 INFO:tasks.rados.rados.0.smithi076.stdout:3758: finishing copy_from to smithi07694878-39 2021-03-15T22:35:37.354 INFO:tasks.rados.rados.0.smithi076.stdout:3758: got expected ENOENT (src dne) 2021-03-15T22:35:37.354 INFO:tasks.rados.rados.0.smithi076.stdout:3759: finishing copy_from racing read to smithi07694878-20 2021-03-15T22:35:37.354 INFO:tasks.rados.rados.0.smithi076.stdout:3758: finishing copy_from racing read to smithi07694878-39 2021-03-15T22:35:37.358 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:37.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:37.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:37.460 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:37.484 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:37.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:37.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:37.508 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:37 smithi125 conmon[56739]: audit 2021-03-15T22:35:36.536463+0000 mon.a (mon. 2021-03-15T22:35:37.508 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:37 smithi125 conmon[56739]: 0) 2444 : audit [INF] from='client.? 172.21.15.76:0/3583121055' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:35:37.508 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:37 smithi125 conmon[56739]: cluster 2021-03-15T22:35:37.338165+0000 mon.a (mon.0) 2445 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:37.509 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:37 smithi125 conmon[56739]: audit 2021-03-15T22:35:37.342046+0000 mon.a (mon.0) 2446 : audit [INF] from='client.? 172.21.15.76:0/3583121055' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:35:37.509 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:37 smithi125 conmon[56739]: cluster 2021-03-15T22:35:37.342111+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e982: 8 total, 8 up, 8 in 2021-03-15T22:35:37.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[56514]: audit 2021-03-15T22:35:36.536463+0000 2021-03-15T22:35:37.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[56514]: mon.a (mon.0) 2444 : audit [INF] from='client.? 172.21.15.76:0/3583121055' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:35:37.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[56514]: cluster 2021-03-15T22:35:37.338165+0000 mon.a (mon.0) 2445 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:37.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[56514]: audit 2021-03-15T22:35:37.342046+0000 mon.a (mon.0) 2446 : audit [INF] from='client.? 172.21.15.76:0/3583121055' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:35:37.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[56514]: cluster 2021-03-15T22:35:37.342111+0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e982: 8 total, 8 up, 8 in 2021-03-15T22:35:37.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[62156]: audit 2021-03-15T22:35:36.536463+0000 mon.a 2021-03-15T22:35:37.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[62156]: (mon.0) 2444 : audit [INF] from='client.? 172.21.15.76:0/3583121055' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:35:37.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:35:37.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[62156]: -15T22:35:37.338165+0000 mon.a (mon.0) 2445 2021-03-15T22:35:37.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[62156]: : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:37.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:35:37.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[62156]: T22:35:37.342046+0000 mon.a (mon.0) 2446 : audit 2021-03-15T22:35:37.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/3583121055' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:35:37.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[62156]: cluster 2021-03-15T22:35:37.342111 2021-03-15T22:35:37.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:37 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2447 : cluster [DBG] osdmap e982: 8 total, 8 up, 8 in 2021-03-15T22:35:37.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:37.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:37.613 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:37.613 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:37.715 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:37.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:37.740 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:37.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:37.748 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:35:37.749 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:35:37.749 INFO:teuthology.orchestra.run.smithi125.stdout: "complaint_time": 30, 2021-03-15T22:35:37.749 INFO:teuthology.orchestra.run.smithi125.stdout: "num_blocked_ops": 0 2021-03-15T22:35:37.749 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:35:37.842 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:37.865 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:37.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:37.866 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:37.967 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:37.989 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:37.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:37.991 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:38.092 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:38.113 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:38.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:38.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:38.216 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:38.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:38.239 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:38.239 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:38.341 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:38.347 INFO:tasks.rados.rados.0.smithi076.stdout:3758: done (2 left) 2021-03-15T22:35:38.347 INFO:tasks.rados.rados.0.smithi076.stdout:3759: done (1 left) 2021-03-15T22:35:38.347 INFO:tasks.rados.rados.0.smithi076.stdout:3760: done (0 left) 2021-03-15T22:35:38.348 INFO:tasks.rados.rados.0.smithi076.stdout:3761: write oid 45 current snap is 442 2021-03-15T22:35:38.348 INFO:tasks.rados.rados.0.smithi076.stdout:3761: seq_num 1278 ranges {679346=515526,1868811=577782,3179173=300173} 2021-03-15T22:35:38.350 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.6 dump_historic_ops 2021-03-15T22:35:38.353 INFO:tasks.rados.rados.0.smithi076.stdout:3761: writing smithi07694878-45 from 679346 to 1194872 tid 1 2021-03-15T22:35:38.361 INFO:tasks.rados.rados.0.smithi076.stdout:3761: writing smithi07694878-45 from 1868811 to 2446593 tid 2 2021-03-15T22:35:38.366 INFO:tasks.rados.rados.0.smithi076.stdout:3761: writing smithi07694878-45 from 3179173 to 3479346 tid 3 2021-03-15T22:35:38.367 INFO:tasks.rados.rados.0.smithi076.stdout:3762: snap_remove snap 397 2021-03-15T22:35:38.367 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:38.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:38.369 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:38.470 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:38.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:38.493 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:38.493 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:38.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:38 smithi076 conmon[56514]: cluster 2021-03-15T22:35:37.222555+0000 mgr.y (mgr.14144) 1084 : cluster [DBG] pgmap v1619: 158 pgs: 2 peering, 156 active+clean; 645 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 5.7 MiB/s wr, 25 op/s 2021-03-15T22:35:38.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:38 smithi076 conmon[56514]: cluster 2021-03-15T22:35:37.543284+0000 osd.7 (osd.7) 182 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:35:38.518 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:38 smithi076 conmon[56514]: cluster 2021-03-15T22:35:38.343663+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e983: 8 total, 8 up, 8 in 2021-03-15T22:35:38.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:38 smithi076 conmon[62156]: cluster 2021-03-15T22:35:37.222555+0000 mgr.y (mgr.14144) 1084 : cluster [DBG] pgmap v1619: 158 pgs: 2 peering, 156 active+clean; 645 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 5.7 MiB/s wr, 25 op/s 2021-03-15T22:35:38.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:38 smithi076 conmon[62156]: cluster 2021-03-15T22:35:37.543284+0000 osd.7 (osd.7) 182 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:35:38.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:38 smithi076 conmon[62156]: cluster 2021-03-15T22:35:38.343663+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e983: 8 total, 8 up, 8 in 2021-03-15T22:35:38.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:38.618 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:38.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:38.620 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:38.625 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:38.721 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:38.745 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:35:38.746 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:38.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:38.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:38.762 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:35:38.778 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:35:38.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:38 smithi125 conmon[56739]: cluster 2021-03-15T22:35:37.222555+0000 2021-03-15T22:35:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:38 smithi125 conmon[56739]: mgr.y (mgr.14144) 1084 : cluster [DBG] pgmap v1619: 158 pgs: 2 peering, 156 active+clean; 645 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 5.7 MiB/s wr, 25 op/s 2021-03-15T22:35:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:38 smithi125 conmon[56739]: cluster 2021-03-15T22:35:37.543284+0000 osd.7 (osd.7) 182 : cluster [DBG] 3.15 scrub ok 2021-03-15T22:35:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:38 smithi125 conmon[56739]: cluster 2021-03-15T22:35:38.343663+0000 mon.a (mon.0) 2448 : cluster [DBG] osdmap e983: 8 total, 8 up, 8 in 2021-03-15T22:35:38.794 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:35:38.812 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:35:38.829 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:35:38.846 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:35:38.850 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:38.864 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:35:38.865 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:35:38.866 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:35:38.866 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:35:38.866 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:35:38.866 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:35:38.867 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:35:38.867 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:35:38.867 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:35:38.872 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:38.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:38.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:38.975 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:38.997 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:38.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:38.999 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:39.100 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:39.124 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:39.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:39.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:39.227 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:39.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:39.253 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:39.253 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:39.354 INFO:tasks.rados.rados.0.smithi076.stdout:3762: done (1 left) 2021-03-15T22:35:39.354 INFO:tasks.rados.rados.0.smithi076.stdout:3763: write oid 27 current snap is 442 2021-03-15T22:35:39.354 INFO:tasks.rados.rados.0.smithi076.stdout:3763: seq_num 1279 ranges {696153=590775,1927293=721440,3206596=289557} 2021-03-15T22:35:39.355 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:39.361 INFO:tasks.rados.rados.0.smithi076.stdout:3763: writing smithi07694878-27 from 696153 to 1286928 tid 1 2021-03-15T22:35:39.369 INFO:tasks.rados.rados.0.smithi076.stdout:3763: writing smithi07694878-27 from 1927293 to 2648733 tid 2 2021-03-15T22:35:39.375 INFO:tasks.rados.rados.0.smithi076.stdout:3763: writing smithi07694878-27 from 3206596 to 3496153 tid 3 2021-03-15T22:35:39.375 INFO:tasks.rados.rados.0.smithi076.stdout:3764: rollback oid 29 current snap is 442 2021-03-15T22:35:39.376 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 29 to 426 2021-03-15T22:35:39.376 INFO:tasks.rados.rados.0.smithi076.stdout:3761: finishing write tid 1 to smithi07694878-45 2021-03-15T22:35:39.376 INFO:tasks.rados.rados.0.smithi076.stdout:3761: finishing write tid 2 to smithi07694878-45 2021-03-15T22:35:39.376 INFO:tasks.rados.rados.0.smithi076.stdout:3761: finishing write tid 3 to smithi07694878-45 2021-03-15T22:35:39.377 INFO:tasks.rados.rados.0.smithi076.stdout:3761: finishing write tid 4 to smithi07694878-45 2021-03-15T22:35:39.377 INFO:tasks.rados.rados.0.smithi076.stdout:3761: finishing write tid 5 to smithi07694878-45 2021-03-15T22:35:39.377 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 3084 (ObjNum 1278 snap 442 seq_num 1278) dirty exists 2021-03-15T22:35:39.377 INFO:tasks.rados.rados.0.smithi076.stdout:3761: left oid 45 (ObjNum 1278 snap 442 seq_num 1278) 2021-03-15T22:35:39.377 INFO:tasks.rados.rados.0.smithi076.stdout:3763: finishing write tid 1 to smithi07694878-27 2021-03-15T22:35:39.378 INFO:tasks.rados.rados.0.smithi076.stdout:3763: finishing write tid 2 to smithi07694878-27 2021-03-15T22:35:39.378 INFO:tasks.rados.rados.0.smithi076.stdout:3761: done (2 left) 2021-03-15T22:35:39.378 INFO:tasks.rados.rados.0.smithi076.stdout:3765: read oid 25 snap -1 2021-03-15T22:35:39.378 INFO:tasks.rados.rados.0.smithi076.stdout:3765: expect (ObjNum 1266 snap 437 seq_num 1266) 2021-03-15T22:35:39.379 INFO:tasks.rados.rados.0.smithi076.stdout:3766: write oid 20 current snap is 442 2021-03-15T22:35:39.379 INFO:tasks.rados.rados.0.smithi076.stdout:3766: seq_num 1280 ranges {712960=666024,1985775=581451,3234019=278941} 2021-03-15T22:35:39.382 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:39.382 INFO:tasks.rados.rados.0.smithi076.stdout:3766: writing smithi07694878-20 from 712960 to 1378984 tid 1 2021-03-15T22:35:39.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:39.384 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:39.389 INFO:tasks.rados.rados.0.smithi076.stdout:3766: writing smithi07694878-20 from 1985775 to 2567226 tid 2 2021-03-15T22:35:39.395 INFO:tasks.rados.rados.0.smithi076.stdout:3766: writing smithi07694878-20 from 3234019 to 3512960 tid 3 2021-03-15T22:35:39.396 INFO:tasks.rados.rados.0.smithi076.stdout:3767: rollback oid 31 current snap is 442 2021-03-15T22:35:39.396 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 31 to 429 2021-03-15T22:35:39.396 INFO:tasks.rados.rados.0.smithi076.stdout:3763: finishing write tid 3 to smithi07694878-27 2021-03-15T22:35:39.396 INFO:tasks.rados.rados.0.smithi076.stdout:3763: finishing write tid 4 to smithi07694878-27 2021-03-15T22:35:39.397 INFO:tasks.rados.rados.0.smithi076.stdout:3763: finishing write tid 5 to smithi07694878-27 2021-03-15T22:35:39.397 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 3046 (ObjNum 1279 snap 442 seq_num 1279) dirty exists 2021-03-15T22:35:39.397 INFO:tasks.rados.rados.0.smithi076.stdout:3763: left oid 27 (ObjNum 1279 snap 442 seq_num 1279) 2021-03-15T22:35:39.397 INFO:tasks.rados.rados.0.smithi076.stdout:3764: finishing rollback tid 0 to smithi07694878-29 2021-03-15T22:35:39.398 INFO:tasks.rados.rados.0.smithi076.stdout:3764: finishing rollback tid 1 to smithi07694878-29 2021-03-15T22:35:39.398 INFO:tasks.rados.rados.0.smithi076.stdout:3764: finishing rollback tid 2 to smithi07694878-29 2021-03-15T22:35:39.398 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 29 v 3311 (ObjNum 1228 snap 425 seq_num 1228) dirty exists 2021-03-15T22:35:39.398 INFO:tasks.rados.rados.0.smithi076.stdout:3763: done (4 left) 2021-03-15T22:35:39.399 INFO:tasks.rados.rados.0.smithi076.stdout:3764: done (3 left) 2021-03-15T22:35:39.399 INFO:tasks.rados.rados.0.smithi076.stdout:3768: delete oid 8 current snap is 442 2021-03-15T22:35:39.399 INFO:tasks.rados.rados.0.smithi076.stdout:3766: finishing write tid 1 to smithi07694878-20 2021-03-15T22:35:39.399 INFO:tasks.rados.rados.0.smithi076.stdout:3765: expect (ObjNum 1266 snap 437 seq_num 1266) 2021-03-15T22:35:39.485 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:39.508 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:39.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:39.509 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:39.536 INFO:tasks.rados.rados.0.smithi076.stdout:3766: finishing write tid 2 to smithi07694878-20 2021-03-15T22:35:39.536 INFO:tasks.rados.rados.0.smithi076.stdout:3765: done (3 left) 2021-03-15T22:35:39.537 INFO:tasks.rados.rados.0.smithi076.stdout:3768: done (2 left) 2021-03-15T22:35:39.537 INFO:tasks.rados.rados.0.smithi076.stdout:3769: write (excl) oid 30 current snap is 442 2021-03-15T22:35:39.537 INFO:tasks.rados.rados.0.smithi076.stdout:3766: finishing write tid 3 to smithi07694878-20 2021-03-15T22:35:39.537 INFO:tasks.rados.rados.0.smithi076.stdout:3769: seq_num 1282 ranges {746574=416522,1586386=468767,2772512=629544} 2021-03-15T22:35:39.542 INFO:tasks.rados.rados.0.smithi076.stdout:3769: writing smithi07694878-30 from 746574 to 1163096 tid 1 2021-03-15T22:35:39.547 INFO:tasks.rados.rados.0.smithi076.stdout:3769: writing smithi07694878-30 from 1586386 to 2055153 tid 2 2021-03-15T22:35:39.555 INFO:tasks.rados.rados.0.smithi076.stdout:3769: writing smithi07694878-30 from 2772512 to 3402056 tid 3 2021-03-15T22:35:39.555 INFO:tasks.rados.rados.0.smithi076.stdout:3770: read oid 25 snap 411 2021-03-15T22:35:39.556 INFO:tasks.rados.rados.0.smithi076.stdout:3770: expect (ObjNum 1076 snap 366 seq_num 1076) 2021-03-15T22:35:39.556 INFO:tasks.rados.rados.0.smithi076.stdout:3771: copy_from oid 41 from oid 39 current snap is 442 2021-03-15T22:35:39.556 INFO:tasks.rados.rados.0.smithi076.stdout:3766: finishing write tid 4 to smithi07694878-20 2021-03-15T22:35:39.556 INFO:tasks.rados.rados.0.smithi076.stdout:3766: finishing write tid 5 to smithi07694878-20 2021-03-15T22:35:39.557 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 2685 (ObjNum 1280 snap 442 seq_num 1280) dirty exists 2021-03-15T22:35:39.557 INFO:tasks.rados.rados.0.smithi076.stdout:3766: left oid 20 (ObjNum 1280 snap 442 seq_num 1280) 2021-03-15T22:35:39.557 INFO:tasks.rados.rados.0.smithi076.stdout:3767: finishing rollback tid 1 to smithi07694878-31 2021-03-15T22:35:39.557 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 2793 (ObjNum 255149944 snap 21896 seq_num 1) dirty dne 2021-03-15T22:35:39.558 INFO:tasks.rados.rados.0.smithi076.stdout:3769: finishing write tid 1 to smithi07694878-30 2021-03-15T22:35:39.558 INFO:tasks.rados.rados.0.smithi076.stdout:3769: finishing write tid 2 to smithi07694878-30 2021-03-15T22:35:39.558 INFO:tasks.rados.rados.0.smithi076.stdout:3766: done (4 left) 2021-03-15T22:35:39.558 INFO:tasks.rados.rados.0.smithi076.stdout:3767: done (3 left) 2021-03-15T22:35:39.559 INFO:tasks.rados.rados.0.smithi076.stdout:3772: write oid 28 current snap is 442 2021-03-15T22:35:39.559 INFO:tasks.rados.rados.0.smithi076.stdout:3772: seq_num 1283 ranges {763381=491771,2044868=728778,3199935=363446} 2021-03-15T22:35:39.562 INFO:tasks.rados.rados.0.smithi076.stdout:3772: writing smithi07694878-28 from 763381 to 1255152 tid 1 2021-03-15T22:35:39.571 INFO:tasks.rados.rados.0.smithi076.stdout:3772: writing smithi07694878-28 from 2044868 to 2773646 tid 2 2021-03-15T22:35:39.576 INFO:tasks.rados.rados.0.smithi076.stdout:3772: writing smithi07694878-28 from 3199935 to 3563381 tid 3 2021-03-15T22:35:39.577 INFO:tasks.rados.rados.0.smithi076.stdout:3773: write (excl) oid 37 current snap is 442 2021-03-15T22:35:39.577 INFO:tasks.rados.rados.0.smithi076.stdout:3773: seq_num 1284 ranges {780188=567020,2103350=472436,3227358=352830} 2021-03-15T22:35:39.584 INFO:tasks.rados.rados.0.smithi076.stdout:3773: writing smithi07694878-37 from 780188 to 1347208 tid 1 2021-03-15T22:35:39.590 INFO:tasks.rados.rados.0.smithi076.stdout:3773: writing smithi07694878-37 from 2103350 to 2575786 tid 2 2021-03-15T22:35:39.597 INFO:tasks.rados.rados.0.smithi076.stdout:3773: writing smithi07694878-37 from 3227358 to 3580188 tid 3 2021-03-15T22:35:39.597 INFO:tasks.rados.rados.0.smithi076.stdout:3774: read oid 21 snap -1 2021-03-15T22:35:39.597 INFO:tasks.rados.rados.0.smithi076.stdout:3774: expect (ObjNum 1225 snap 424 seq_num 1225) 2021-03-15T22:35:39.597 INFO:tasks.rados.rados.0.smithi076.stdout:3775: copy_from oid 22 from oid 46 current snap is 442 2021-03-15T22:35:39.598 INFO:tasks.rados.rados.0.smithi076.stdout:3769: finishing write tid 3 to smithi07694878-30 2021-03-15T22:35:39.598 INFO:tasks.rados.rados.0.smithi076.stdout:3769: finishing write tid 4 to smithi07694878-30 2021-03-15T22:35:39.598 INFO:tasks.rados.rados.0.smithi076.stdout:3769: finishing write tid 5 to smithi07694878-30 2021-03-15T22:35:39.598 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 3184 (ObjNum 1282 snap 442 seq_num 1282) dirty exists 2021-03-15T22:35:39.599 INFO:tasks.rados.rados.0.smithi076.stdout:3769: left oid 30 (ObjNum 1282 snap 442 seq_num 1282) 2021-03-15T22:35:39.599 INFO:tasks.rados.rados.0.smithi076.stdout:3770: expect (ObjNum 1076 snap 366 seq_num 1076) 2021-03-15T22:35:39.611 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:39.632 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:39.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:39.634 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:39.643 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 7, 0, 3, 5, 6, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:35:39.644 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:35:39.644 INFO:tasks.thrashosds.thrasher:Shrinking pool .rgw.root 2021-03-15T22:35:39.644 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .rgw.root by 10 min 8 2021-03-15T22:35:39.645 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:35:39.663 INFO:tasks.rados.rados.0.smithi076.stdout:3772: finishing write tid 1 to smithi07694878-28 2021-03-15T22:35:39.663 INFO:tasks.rados.rados.0.smithi076.stdout:3769: done (6 left) 2021-03-15T22:35:39.664 INFO:tasks.rados.rados.0.smithi076.stdout:3770: done (5 left) 2021-03-15T22:35:39.664 INFO:tasks.rados.rados.0.smithi076.stdout:3776: rollback oid 36 current snap is 442 2021-03-15T22:35:39.664 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 36 to 416 2021-03-15T22:35:39.664 INFO:tasks.rados.rados.0.smithi076.stdout:3771: finishing copy_from to smithi07694878-41 2021-03-15T22:35:39.665 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 2823 (ObjNum 1249 snap 430 seq_num 1249) dirty exists 2021-03-15T22:35:39.665 INFO:tasks.rados.rados.0.smithi076.stdout:3771: finishing copy_from racing read to smithi07694878-41 2021-03-15T22:35:39.665 INFO:tasks.rados.rados.0.smithi076.stdout:3772: finishing write tid 2 to smithi07694878-28 2021-03-15T22:35:39.665 INFO:tasks.rados.rados.0.smithi076.stdout:3772: finishing write tid 3 to smithi07694878-28 2021-03-15T22:35:39.665 INFO:tasks.rados.rados.0.smithi076.stdout:3772: finishing write tid 4 to smithi07694878-28 2021-03-15T22:35:39.666 INFO:tasks.rados.rados.0.smithi076.stdout:3772: finishing write tid 5 to smithi07694878-28 2021-03-15T22:35:39.666 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 28 v 3315 (ObjNum 1283 snap 442 seq_num 1283) dirty exists 2021-03-15T22:35:39.666 INFO:tasks.rados.rados.0.smithi076.stdout:3772: left oid 28 (ObjNum 1283 snap 442 seq_num 1283) 2021-03-15T22:35:39.666 INFO:tasks.rados.rados.0.smithi076.stdout:3773: finishing write tid 1 to smithi07694878-37 2021-03-15T22:35:39.667 INFO:tasks.rados.rados.0.smithi076.stdout:3773: finishing write tid 2 to smithi07694878-37 2021-03-15T22:35:39.667 INFO:tasks.rados.rados.0.smithi076.stdout:3773: finishing write tid 3 to smithi07694878-37 2021-03-15T22:35:39.667 INFO:tasks.rados.rados.0.smithi076.stdout:3773: finishing write tid 4 to smithi07694878-37 2021-03-15T22:35:39.667 INFO:tasks.rados.rados.0.smithi076.stdout:3773: finishing write tid 5 to smithi07694878-37 2021-03-15T22:35:39.668 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 3320 (ObjNum 1284 snap 442 seq_num 1284) dirty exists 2021-03-15T22:35:39.668 INFO:tasks.rados.rados.0.smithi076.stdout:3773: left oid 37 (ObjNum 1284 snap 442 seq_num 1284) 2021-03-15T22:35:39.668 INFO:tasks.rados.rados.0.smithi076.stdout:3774: expect (ObjNum 1225 snap 424 seq_num 1225) 2021-03-15T22:35:39.669 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:39 smithi076 conmon[56514]: audit 2021-03-15T22:35:38.722787+0000 mon.a (mon.0 2021-03-15T22:35:39.669 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:39 smithi076 conmon[56514]: ) 2449 : audit [DBG] from='client.? 172.21.15.76:0/1612038195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:39.669 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:39 smithi076 conmon[56514]: cluster 2021-03-15T22:35:39.347232+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e984: 8 total, 8 up, 8 in 2021-03-15T22:35:39.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:39 smithi076 conmon[62156]: audit 2021-03-15T22:35:38. 2021-03-15T22:35:39.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:39 smithi076 conmon[62156]: 722787+0000 mon.a (mon.0) 2449 : audit [DBG] from='client.? 172.21.15.76:0/1612038195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:39.670 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:39 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:35:39.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:39 smithi076 conmon[62156]: -15T22:35:39.347232+0000 mon.a (mon.0) 2021-03-15T22:35:39.671 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:39 smithi076 conmon[62156]: 2450 : cluster [DBG] osdmap e984: 8 total, 8 up, 8 in 2021-03-15T22:35:39.682 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:39 smithi125 conmon[56739]: audit 2021-03-15T22:35:38.722787+0000 mon.a (mon.0) 2449 : audit [DBG] from='client.? 172.21.15.76:0/1612038195' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:39.683 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:39 smithi125 conmon[56739]: cluster 2021-03-15T22:35:39.347232+0000 mon.a (mon.0) 2450 : cluster [DBG] osdmap e984: 8 total, 8 up, 8 in 2021-03-15T22:35:39.735 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:39.775 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:39.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:39.777 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:39.878 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:39.900 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:39.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:39.902 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:39.920 INFO:tasks.rados.rados.0.smithi076.stdout:3771: done (5 left) 2021-03-15T22:35:39.921 INFO:tasks.rados.rados.0.smithi076.stdout:3772: done (4 left) 2021-03-15T22:35:39.921 INFO:tasks.rados.rados.0.smithi076.stdout:3773: done (3 left) 2021-03-15T22:35:39.921 INFO:tasks.rados.rados.0.smithi076.stdout:3774: done (2 left) 2021-03-15T22:35:39.921 INFO:tasks.rados.rados.0.smithi076.stdout:3777: write (excl) oid 9 current snap is 442 2021-03-15T22:35:39.922 INFO:tasks.rados.rados.0.smithi076.stdout:3775: finishing copy_from to smithi07694878-22 2021-03-15T22:35:39.922 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 2296 (ObjNum 1236 snap 426 seq_num 1236) dirty exists 2021-03-15T22:35:39.922 INFO:tasks.rados.rados.0.smithi076.stdout:3775: finishing copy_from racing read to smithi07694878-22 2021-03-15T22:35:39.922 INFO:tasks.rados.rados.0.smithi076.stdout:3777: seq_num 1285 ranges {796995=758622,2161832=732447} 2021-03-15T22:35:39.929 INFO:tasks.rados.rados.0.smithi076.stdout:3777: writing smithi07694878-9 from 796995 to 1555617 tid 1 2021-03-15T22:35:39.936 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:39.937 INFO:tasks.rados.rados.0.smithi076.stdout:3777: writing smithi07694878-9 from 2161832 to 2894279 tid 2 2021-03-15T22:35:39.937 INFO:tasks.rados.rados.0.smithi076.stdout:3775: done (2 left) 2021-03-15T22:35:39.937 INFO:tasks.rados.rados.0.smithi076.stdout:3778: write oid 34 current snap is 442 2021-03-15T22:35:39.937 INFO:tasks.rados.rados.0.smithi076.stdout:3776: finishing rollback tid 0 to smithi07694878-36 2021-03-15T22:35:39.938 INFO:tasks.rados.rados.0.smithi076.stdout:3778: seq_num 1286 ranges {413802=433871,1420314=476105,2482204=790985} 2021-03-15T22:35:39.941 INFO:tasks.rados.rados.0.smithi076.stdout:3778: writing smithi07694878-34 from 413802 to 847673 tid 1 2021-03-15T22:35:39.947 INFO:tasks.rados.rados.0.smithi076.stdout:3778: writing smithi07694878-34 from 1420314 to 1896419 tid 2 2021-03-15T22:35:39.956 INFO:tasks.rados.rados.0.smithi076.stdout:3778: writing smithi07694878-34 from 2482204 to 3273189 tid 3 2021-03-15T22:35:39.956 INFO:tasks.rados.rados.0.smithi076.stdout:3779: copy_from oid 17 from oid 44 current snap is 442 2021-03-15T22:35:39.956 INFO:tasks.rados.rados.0.smithi076.stdout:3776: finishing rollback tid 1 to smithi07694878-36 2021-03-15T22:35:39.956 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 36 v 3044 (ObjNum 262225176 snap 21896 seq_num 265145648) dirty dne 2021-03-15T22:35:39.957 INFO:tasks.rados.rados.0.smithi076.stdout:3777: finishing write tid 1 to smithi07694878-9 2021-03-15T22:35:39.957 INFO:tasks.rados.rados.0.smithi076.stdout:3777: finishing write tid 2 to smithi07694878-9 2021-03-15T22:35:39.957 INFO:tasks.rados.rados.0.smithi076.stdout:3777: finishing write tid 3 to smithi07694878-9 2021-03-15T22:35:39.957 INFO:tasks.rados.rados.0.smithi076.stdout:3777: finishing write tid 4 to smithi07694878-9 2021-03-15T22:35:39.957 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 3189 (ObjNum 1285 snap 442 seq_num 1285) dirty exists 2021-03-15T22:35:39.958 INFO:tasks.rados.rados.0.smithi076.stdout:3777: left oid 9 (ObjNum 1285 snap 442 seq_num 1285) 2021-03-15T22:35:39.958 INFO:tasks.rados.rados.0.smithi076.stdout:3776: done (3 left) 2021-03-15T22:35:39.958 INFO:tasks.rados.rados.0.smithi076.stdout:3777: done (2 left) 2021-03-15T22:35:39.958 INFO:tasks.rados.rados.0.smithi076.stdout:3780: snap_create 2021-03-15T22:35:39.959 INFO:tasks.rados.rados.0.smithi076.stdout:3778: finishing write tid 1 to smithi07694878-34 2021-03-15T22:35:39.959 INFO:tasks.rados.rados.0.smithi076.stdout:3778: finishing write tid 2 to smithi07694878-34 2021-03-15T22:35:39.960 INFO:tasks.rados.rados.0.smithi076.stdout:3778: finishing write tid 3 to smithi07694878-34 2021-03-15T22:35:39.960 INFO:tasks.rados.rados.0.smithi076.stdout:3778: finishing write tid 4 to smithi07694878-34 2021-03-15T22:35:39.960 INFO:tasks.rados.rados.0.smithi076.stdout:3778: finishing write tid 5 to smithi07694878-34 2021-03-15T22:35:39.960 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 2689 (ObjNum 1286 snap 442 seq_num 1286) dirty exists 2021-03-15T22:35:39.961 INFO:tasks.rados.rados.0.smithi076.stdout:3778: left oid 34 (ObjNum 1286 snap 442 seq_num 1286) 2021-03-15T22:35:39.971 INFO:tasks.rados.rados.0.smithi076.stdout:3779: finishing copy_from to smithi07694878-17 2021-03-15T22:35:39.971 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 3038 (ObjNum 1177 snap 411 seq_num 1177) dirty exists 2021-03-15T22:35:39.972 INFO:tasks.rados.rados.0.smithi076.stdout:3779: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:35:40.003 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:40.025 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:40.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:40.027 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:40.128 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:40.129 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:35:40.129 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:35:40.152 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:40.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:40.154 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:40.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:40.277 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:40.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:40.278 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:40.301 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:35:40.347 INFO:tasks.rados.rados.0.smithi076.stdout:3778: done (2 left) 2021-03-15T22:35:40.347 INFO:tasks.rados.rados.0.smithi076.stdout:3779: done (1 left) 2021-03-15T22:35:40.347 INFO:tasks.rados.rados.0.smithi076.stdout:3780: done (0 left) 2021-03-15T22:35:40.347 INFO:tasks.rados.rados.0.smithi076.stdout:3781: write (excl) oid 15 current snap is 443 2021-03-15T22:35:40.348 INFO:tasks.rados.rados.0.smithi076.stdout:3781: seq_num 1287 ranges {430609=509120,1478796=736116,2909627=602257} 2021-03-15T22:35:40.352 INFO:tasks.rados.rados.0.smithi076.stdout:3781: writing smithi07694878-15 from 430609 to 939729 tid 1 2021-03-15T22:35:40.359 INFO:tasks.rados.rados.0.smithi076.stdout:3781: writing smithi07694878-15 from 1478796 to 2214912 tid 2 2021-03-15T22:35:40.367 INFO:tasks.rados.rados.0.smithi076.stdout:3781: writing smithi07694878-15 from 2909627 to 3511884 tid 3 2021-03-15T22:35:40.368 INFO:tasks.rados.rados.0.smithi076.stdout:3782: rollback oid 43 current snap is 443 2021-03-15T22:35:40.368 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 43 to 441 2021-03-15T22:35:40.368 INFO:tasks.rados.rados.0.smithi076.stdout:3781: finishing write tid 1 to smithi07694878-15 2021-03-15T22:35:40.368 INFO:tasks.rados.rados.0.smithi076.stdout:3781: finishing write tid 2 to smithi07694878-15 2021-03-15T22:35:40.368 INFO:tasks.rados.rados.0.smithi076.stdout:3783: snap_remove snap 387 2021-03-15T22:35:40.379 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:40.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:40.403 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:40.404 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:40.497 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:35:40.506 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:40.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:40.530 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:40.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:40.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:40.654 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:40.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:40.656 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:40.757 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:40.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:40 smithi076 conmon[56514]: cluster 2021-03-15T22:35:38.710638+0000 2021-03-15T22:35:40.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:40 smithi076 conmon[56514]: osd.6 (osd.6) 144 : cluster [DBG] 5.1f scrub ok 2021-03-15T22:35:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:40 smithi076 conmon[56514]: cluster 2021-03-15T22:35:39.223040+0000 mgr.y (mgr.14144) 1085 : cluster [DBG] pgmap v1622: 158 pgs: 2 peering, 156 active+clean; 645 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2021-03-15T22:35:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:40 smithi076 conmon[56514]: cluster 2021-03-15T22:35:39.349233+0000 osd.6 (osd.6) 145 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:40 smithi076 conmon[56514]: cluster 2021-03-15T22:35:39.351214+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:40.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:40 smithi076 conmon[56514]: cluster 2021-03-15T22:35:40.342481+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e985: 8 total, 8 up, 8 in 2021-03-15T22:35:40.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:40 smithi076 conmon[62156]: cluster 2021-03-15T22:35:38.710638+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.1f scrub ok 2021-03-15T22:35:40.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:40 smithi076 conmon[62156]: cluster 2021-03-15T22:35:39.223040+0000 mgr.y (mgr.14144) 1085 : cluster [DBG] pgmap v1622: 158 pgs: 2 peering, 156 active+clean; 645 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2021-03-15T22:35:40.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:40 smithi076 conmon[62156]: cluster 2021-03-15T22:35:39.349233+0000 osd.6 (osd.6) 145 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:40.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:40 smithi076 conmon[62156]: cluster 2021-03-15T22:35:39.351214+0000 osd.6 (osd.6) 146 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:40.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:40 smithi076 conmon[62156]: cluster 2021-03-15T22:35:40.342481+0000 mon.a (mon.0) 2451 : cluster [DBG] osdmap e985: 8 total, 8 up, 8 in 2021-03-15T22:35:40.778 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:40.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:40.780 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:40.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[56739]: cluster 2021-03-15T22:35:38.710638+0000 osd.6 (osd.6) 144 : cluster [DBG] 5.1f scrub ok 2021-03-15T22:35:40.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[56739]: cluster 2021-03-15T22:35:39.223040+0000 mgr.y (mgr.14144) 1085 : cluster [DBG] pgmap v1622: 158 pgs: 2 peering, 156 active+clean; 645 MiB data, 1.8 GiB used, 713 GiB / 715 GiB avail; 9.8 MiB/s rd, 4.9 MiB/s wr, 20 op/s 2021-03-15T22:35:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[56739]: cluster 2021-03-15T22:35: 2021-03-15T22:35:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[56739]: 39.349233+0000 osd.6 (osd.6) 145 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:35:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[56739]: -03-15T22:35:39.351214+0000 2021-03-15T22:35:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[56739]: osd.6 (osd.6) 146 : cluster [DBG] 2021-03-15T22:35:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[56739]: purged_snaps scrub ok 2021-03-15T22:35:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:35:40.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[56739]: :35:40.342481+0000 mon.a (mon. 2021-03-15T22:35:40.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:40 smithi125 conmon[56739]: 0) 2451 : cluster [DBG] osdmap e985: 8 total, 8 up, 8 in 2021-03-15T22:35:40.821 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_ops_in_flight 2021-03-15T22:35:40.881 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:40.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:40.928 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:40.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:41.030 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:41.053 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:41.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:41.055 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:41.138 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:41.156 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:41.179 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:41.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:41.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:41.212 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:41.282 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:41.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:41.307 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:41.307 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:41.359 INFO:tasks.rados.rados.0.smithi076.stdout:3783: done (2 left) 2021-03-15T22:35:41.359 INFO:tasks.rados.rados.0.smithi076.stdout:3784: copy_from oid 12 from oid 9 current snap is 443 2021-03-15T22:35:41.359 INFO:tasks.rados.rados.0.smithi076.stdout:3785: snap_create 2021-03-15T22:35:41.360 INFO:tasks.rados.rados.0.smithi076.stdout:3781: finishing write tid 3 to smithi07694878-15 2021-03-15T22:35:41.360 INFO:tasks.rados.rados.0.smithi076.stdout:3781: finishing write tid 4 to smithi07694878-15 2021-03-15T22:35:41.360 INFO:tasks.rados.rados.0.smithi076.stdout:3781: finishing write tid 5 to smithi07694878-15 2021-03-15T22:35:41.360 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 3194 (ObjNum 1287 snap 443 seq_num 1287) dirty exists 2021-03-15T22:35:41.361 INFO:tasks.rados.rados.0.smithi076.stdout:3781: left oid 15 (ObjNum 1287 snap 443 seq_num 1287) 2021-03-15T22:35:41.361 INFO:tasks.rados.rados.0.smithi076.stdout:3782: finishing rollback tid 0 to smithi07694878-43 2021-03-15T22:35:41.361 INFO:tasks.rados.rados.0.smithi076.stdout:3782: finishing rollback tid 1 to smithi07694878-43 2021-03-15T22:35:41.361 INFO:tasks.rados.rados.0.smithi076.stdout:3782: finishing rollback tid 2 to smithi07694878-43 2021-03-15T22:35:41.361 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 2449 (ObjNum 1256 snap 436 seq_num 1256) dirty exists 2021-03-15T22:35:41.394 INFO:tasks.rados.rados.0.smithi076.stdout:3784: finishing copy_from to smithi07694878-12 2021-03-15T22:35:41.395 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 12 v 3189 (ObjNum 1285 snap 442 seq_num 1285) dirty exists 2021-03-15T22:35:41.395 INFO:tasks.rados.rados.0.smithi076.stdout:3784: finishing copy_from racing read to smithi07694878-12 2021-03-15T22:35:41.409 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:41.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:41.434 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:41.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:41.535 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:41.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:41.559 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:41.560 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:41.627 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:41 smithi125 conmon[56739]: cluster 2021-03-15T22:35:39.707896+0000 osd.6 (osd.6) 147 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:35:41.627 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:41 smithi125 conmon[56739]: cluster 2021-03-15T22:35:40.034941+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:35:41.627 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:41 smithi125 conmon[56739]: audit 2021-03-15T22:35:40.302325+0000 mgr.y (mgr.14144) 1086 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:41.628 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:41 smithi125 conmon[56739]: cluster 2021-03-15T22:35:40.412139+0000 osd.4 (osd.4) 195 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:35:41.628 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:41 smithi125 conmon[56739]: audit 2021-03-15T22:35:41.224835+0000 mon.a (mon.0) 2452 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "41"}]: dispatch 2021-03-15T22:35:41.628 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:41 smithi125 conmon[56739]: audit 2021-03-15T22:35:41.350114+0000 mon.a (mon.0) 2453 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "41"}]': finished 2021-03-15T22:35:41.628 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:41 smithi125 conmon[56739]: cluster 2021-03-15T22:35:41.350174+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e986: 8 total, 8 up, 8 in 2021-03-15T22:35:41.661 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:41.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[56514]: cluster 2021-03-15T22:35:39.707896+0000 2021-03-15T22:35:41.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[56514]: osd.6 (osd.6) 147 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:35:41.663 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[56514]: cluster 2021-03-15T22:35:40.034941+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:35:41.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[56514]: audit 2021-03-15T22:35:40.302325+0000 mgr.y (mgr.14144) 1086 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:41.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[56514]: cluster 2021-03-15T22:35:40.412139+0000 osd.4 (osd.4) 195 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:35:41.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[56514]: audit 2021-03-15T22:35:41.224835+0000 mon.a (mon.0) 2452 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "41"}]: dispatch 2021-03-15T22:35:41.664 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[56514]: audit 2021-03-15T22:35:41.350114+0000 mon.a (mon.0) 2453 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "41"}]': finished 2021-03-15T22:35:41.665 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[56514]: cluster 2021-03-15T22:35:41.350174+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e986: 8 total, 8 up, 8 in 2021-03-15T22:35:41.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[62156]: cluster 2021-03-15T22:35:39.707896+0000 2021-03-15T22:35:41.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[62156]: osd.6 (osd.6) 147 : cluster [DBG] 5.2 scrub ok 2021-03-15T22:35:41.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[62156]: cluster 2021-03-15T22:35:40.034941+0000 osd.5 (osd.5) 58 : cluster [DBG] 3.1c scrub ok 2021-03-15T22:35:41.666 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[62156]: audit 2021-03-15T22:35:40.302325+0000 mgr.y (mgr.14144) 1086 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:41.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[62156]: cluster 2021-03-15T22:35:40.412139+0000 osd.4 (osd.4) 195 : cluster [DBG] 6.17 scrub ok 2021-03-15T22:35:41.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[62156]: audit 2021-03-15T22:35:41.224835+0000 mon.a (mon.0) 2452 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "41"}]: dispatch 2021-03-15T22:35:41.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[62156]: audit 2021-03-15T22:35:41.350114+0000 mon.a (mon.0) 2453 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "41"}]': finished 2021-03-15T22:35:41.667 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:41 smithi076 conmon[62156]: cluster 2021-03-15T22:35:41.350174+0000 mon.a (mon.0) 2454 : cluster [DBG] osdmap e986: 8 total, 8 up, 8 in 2021-03-15T22:35:41.684 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:41.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:41.686 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:41.714 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:35:41.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:41.794 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:35:41.797 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:35:41.810 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:41.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:41.811 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:41.912 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:41.937 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:41.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:41.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:42.039 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:42.061 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:42.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:42.164 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:42.187 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:42.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:42.289 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:42.311 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:42.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:42.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:42.353 INFO:tasks.rados.rados.0.smithi076.stdout:3781: done (3 left) 2021-03-15T22:35:42.354 INFO:tasks.rados.rados.0.smithi076.stdout:3782: done (2 left) 2021-03-15T22:35:42.354 INFO:tasks.rados.rados.0.smithi076.stdout:3784: done (1 left) 2021-03-15T22:35:42.354 INFO:tasks.rados.rados.0.smithi076.stdout:3785: done (0 left) 2021-03-15T22:35:42.355 INFO:tasks.rados.rados.0.smithi076.stdout:3786: write (excl) oid 33 current snap is 444 2021-03-15T22:35:42.355 INFO:tasks.rados.rados.0.smithi076.stdout:3786: seq_num 1288 ranges {447416=584369,1537278=479774,2537050=529882,3524784=122632} 2021-03-15T22:35:42.359 INFO:tasks.rados.rados.0.smithi076.stdout:3786: writing smithi07694878-33 from 447416 to 1031785 tid 1 2021-03-15T22:35:42.365 INFO:tasks.rados.rados.0.smithi076.stdout:3786: writing smithi07694878-33 from 1537278 to 2017052 tid 2 2021-03-15T22:35:42.371 INFO:tasks.rados.rados.0.smithi076.stdout:3786: writing smithi07694878-33 from 2537050 to 3066932 tid 3 2021-03-15T22:35:42.374 INFO:tasks.rados.rados.0.smithi076.stdout:3786: writing smithi07694878-33 from 3524784 to 3647416 tid 4 2021-03-15T22:35:42.375 INFO:tasks.rados.rados.0.smithi076.stdout:3787: snap_create 2021-03-15T22:35:42.375 INFO:tasks.rados.rados.0.smithi076.stdout:3786: finishing write tid 1 to smithi07694878-33 2021-03-15T22:35:42.375 INFO:tasks.rados.rados.0.smithi076.stdout:3786: finishing write tid 2 to smithi07694878-33 2021-03-15T22:35:42.375 INFO:tasks.rados.rados.0.smithi076.stdout:3786: finishing write tid 3 to smithi07694878-33 2021-03-15T22:35:42.377 INFO:tasks.rados.rados.0.smithi076.stdout:3786: finishing write tid 4 to smithi07694878-33 2021-03-15T22:35:42.377 INFO:tasks.rados.rados.0.smithi076.stdout:3786: finishing write tid 5 to smithi07694878-33 2021-03-15T22:35:42.377 INFO:tasks.rados.rados.0.smithi076.stdout:3786: finishing write tid 6 to smithi07694878-33 2021-03-15T22:35:42.378 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 33 v 2694 (ObjNum 1288 snap 444 seq_num 1288) dirty exists 2021-03-15T22:35:42.378 INFO:tasks.rados.rados.0.smithi076.stdout:3786: left oid 33 (ObjNum 1288 snap 444 seq_num 1288) 2021-03-15T22:35:42.413 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:42.437 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:42.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:42.438 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:42.535 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:42 smithi125 conmon[56739]: cluster 2021-03-15T22:35:41.082297+0000 osd.1 (osd.1) 2021-03-15T22:35:42.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:42 smithi125 conmon[56739]: 185 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:35:42.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:42 smithi125 conmon[56739]: cluster 2021-03-15T22:35:41.223531+0000 mgr.y (mgr.14144) 1087 : cluster [DBG] pgmap v1625: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 667 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2021-03-15T22:35:42.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:42 smithi125 conmon[56739]: cluster 2021-03-15T22:35:41.588929+0000 osd.7 (osd.7) 183 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:35:42.536 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:42 smithi125 conmon[56739]: audit 2021-03-15T22:35:41.775740+0000 mgr.y (mgr.14144) 1088 : audit [DBG] from='client.16845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:42.537 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:42 smithi125 conmon[56739]: cluster 2021-03-15T22:35:42.350996+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e987: 8 total, 8 up, 8 in 2021-03-15T22:35:42.539 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:42.561 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:42.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:42.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:42.663 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:42.685 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:42.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:42.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:42.713 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:35:42.713 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:35:42.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[56514]: cluster 2021-03-15T22:35:41.082297+0000 osd.1 (osd.1 2021-03-15T22:35:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[56514]: ) 185 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:35:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[56514]: cluster 2021-03-15T22:35:41.223531+0000 mgr.y (mgr.14144) 1087 : cluster [DBG] pgmap v1625: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 667 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2021-03-15T22:35:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[56514]: cluster 2021-03-15T22:35:41.588929+0000 osd.7 (osd.7) 183 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:35:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[56514]: audit 2021-03-15T22:35:41.775740+0000 mgr.y (mgr.14144) 1088 : audit [DBG] from='client.16845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:42.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[56514]: cluster 2021-03-15T22:35:42.350996+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e987: 8 total, 8 up, 8 in 2021-03-15T22:35:42.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[62156]: cluster 2021-03-15T22:35:41. 2021-03-15T22:35:42.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[62156]: 082297+0000 osd.1 (osd.1) 185 : cluster [DBG] 3.19 scrub ok 2021-03-15T22:35:42.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[62156]: cluster 2021-03-15T22:35:41.223531+0000 mgr.y (mgr.14144) 1087 : cluster [DBG] pgmap v1625: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 667 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 4.1 MiB/s wr, 19 op/s 2021-03-15T22:35:42.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[62156]: cluster 2021-03-15T22:35:41.588929+0000 osd.7 (osd.7) 183 : cluster [DBG] 6.12 scrub ok 2021-03-15T22:35:42.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[62156]: audit 2021-03-15T22:35:41.775740+0000 mgr.y (mgr.14144) 1088 : audit [DBG] from='client.16845 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:42.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:42 smithi076 conmon[62156]: cluster 2021-03-15T22:35:42.350996+0000 mon.a (mon.0) 2455 : cluster [DBG] osdmap e987: 8 total, 8 up, 8 in 2021-03-15T22:35:42.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:42.811 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:42.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:42.813 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:42.914 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:42.936 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:42.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:42.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:43.038 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:43.061 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:43.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:43.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:43.133 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:43.158 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1626,"stamp":"2021-03-15T22:35:41.347535+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":699686453,"num_objects":633,"num_object_clones":268,"num_object_copies":1899,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":633,"num_whiteouts":13,"num_read":9515,"num_read_kb":7159593,"num_write":5959,"num_write_kb":1968085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5900,"num_bytes_recovered":2230123701,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23633,"ondisk_log_size":23633,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":472,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":1962208,"kb_used_data":1535392,"kb_used_omap":273,"kb_used_meta":426478,"kb_avail":747999008,"statfs":{"total":767960285184,"available":765950984192,"internally_reserved":0,"allocated":1572241408,"data_stored":2206590899,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":436714362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":22889296,"num_objects":8,"num_object_clones":8,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24169,"num_write":40,"num_write_kb":16791,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.007495"},"pg_stats":[{"pgid":"6.29","version":"0'0","reported_seq":"449","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350271+0000","last_change":"2021-03-15T22:35:36.357154+0000","last_active":"2021-03-15T22:35:39.350271+0000","last_peered":"2021-03-15T22:35:39.350271+0000","last_clean":"2021-03-15T22:35:39.350271+0000","last_became_active":"2021-03-15T22:35:36.356788+0000","last_became_peered":"2021-03-15T22:35:36.356788+0000","last_unstale":"2021-03-15T22:35:39.350271+0000","last_undegraded":"2021-03-15T22:35:39.350271+0000","last_fullsized":"2021-03-15T22:35:39.350271+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.28","version":"0'0","reported_seq":"993","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.340882+0000","last_change":"2021-03-15T22:35:36.354474+0000","last_active":"2021-03-15T22:35:37.340882+0000","last_peered":"2021-03-15T22:35:37.340882+0000","last_clean":"2021-03-15T22:35:37.340882+0000","last_became_active":"2021-03-15T22:35:36.354095+0000","last_became_peered":"2021-03-15T22:35:36.354095+0000","last_unstale":"2021-03-15T22:35:37.340882+0000","last_undegraded":"2021-03-15T22:35:37.340882+0000","last_fullsized":"2021-03-15T22:35:37.340882+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.27","version":"0'0","reported_seq":"12","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344436+0000","last_change":"2021-03-15T22:35:36.357719+0000","last_active":"2021-03-15T22:35:37.344436+0000","last_peered":"2021-03-15T22:35:37.344436+0000","last_clean":"2021-03-15T22:35:37.344436+0000","last_became_active":"2021-03-15T22:35:36.357351+0000","last_became_peered":"2021-03-15T22:35:36.357351+0000","last_unstale":"2021-03-15T22:35:37.344436+0000","last_undegraded":"2021-03-15T22:35:37.344436+0000","last_fullsized":"2021-03-15T22:35:37.344436+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:18.617912+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.617912+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.26","version":"0'0","reported_seq":"10","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.350574+0000","last_change":"2021-03-15T22:35:36.350574+0000","last_active":"2021-03-15T22:35:36.350574+0000","last_peered":"2021-03-15T22:35:36.350574+0000","last_clean":"2021-03-15T22:35:36.350574+0000","last_became_active":"2021-03-15T22:35:36.350145+0000","last_became_peered":"2021-03-15T22:35:36.350145+0000","last_unstale":"2021-03-15T22:35:36.350574+0000","last_undegraded":"2021-03-15T22:35:36.350574+0000","last_fullsized":"2021-03-15T22:35:36.350574+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"12","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341573+0000","last_change":"2021-03-15T22:35:36.755816+0000","last_active":"2021-03-15T22:35:37.341573+0000","last_peered":"2021-03-15T22:35:37.341573+0000","last_clean":"2021-03-15T22:35:37.341573+0000","last_became_active":"2021-03-15T22:35:36.755571+0000","last_became_peered":"2021-03-15T22:35:36.755571+0000","last_unstale":"2021-03-15T22:35:37.341573+0000","last_undegraded":"2021-03-15T22:35:37.341573+0000","last_fullsized":"2021-03-15T22:35:37.341573+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:32.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"801","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344600+0000","last_change":"2021-03-15T22:35:36.758818+0000","last_active":"2021-03-15T22:35:37.344600+0000","last_peered":"2021-03-15T22:35:37.344600+0000","last_clean":"2021-03-15T22:35:37.344600+0000","last_became_active":"2021-03-15T22:35:36.758418+0000","last_became_peered":"2021-03-15T22:35:36.758418+0000","last_unstale":"2021-03-15T22:35:37.344600+0000","last_undegraded":"2021-03-15T22:35:37.344600+0000","last_fullsized":"2021-03-15T22:35:37.344600+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"12","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341004+0000","last_change":"2021-03-15T22:35:36.758802+0000","last_active":"2021-03-15T22:35:37.341004+0000","last_peered":"2021-03-15T22:35:37.341004+0000","last_clean":"2021-03-15T22:35:37.341004+0000","last_became_active":"2021-03-15T22:35:36.758471+0000","last_became_peered":"2021-03-15T22:35:36.758471+0000","last_unstale":"2021-03-15T22:35:37.341004+0000","last_undegraded":"2021-03-15T22:35:37.341004+0000","last_fullsized":"2021-03-15T22:35:37.341004+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:14.592451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:14.592451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"10","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.356252+0000","last_change":"2021-03-15T22:35:36.356252+0000","last_active":"2021-03-15T22:35:36.356252+0000","last_peered":"2021-03-15T22:35:36.356252+0000","last_clean":"2021-03-15T22:35:36.356252+0000","last_became_active":"2021-03-15T22:35:36.355967+0000","last_became_peered":"2021-03-15T22:35:36.355967+0000","last_unstale":"2021-03-15T22:35:36.356252+0000","last_undegraded":"2021-03-15T22:35:36.356252+0000","last_fullsized":"2021-03-15T22:35:36.356252+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"11","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.758856+0000","last_change":"2021-03-15T22:35:36.758571+0000","last_active":"2021-03-15T22:35:36.758856+0000","last_peered":"2021-03-15T22:35:36.758856+0000","last_clean":"2021-03-15T22:35:36.758856+0000","last_became_active":"2021-03-15T22:35:36.758203+0000","last_became_peered":"2021-03-15T22:35:36.758203+0000","last_unstale":"2021-03-15T22:35:36.758856+0000","last_undegraded":"2021-03-15T22:35:36.758856+0000","last_fullsized":"2021-03-15T22:35:36.758856+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.701955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.701955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"14","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.349604+0000","last_change":"2021-03-15T22:35:36.357269+0000","last_active":"2021-03-15T22:35:39.349604+0000","last_peered":"2021-03-15T22:35:39.349604+0000","last_clean":"2021-03-15T22:35:39.349604+0000","last_became_active":"2021-03-15T22:35:36.356824+0000","last_became_peered":"2021-03-15T22:35:36.356824+0000","last_unstale":"2021-03-15T22:35:39.349604+0000","last_undegraded":"2021-03-15T22:35:39.349604+0000","last_fullsized":"2021-03-15T22:35:39.349604+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"317'7","reported_seq":"345","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.346759+0000","last_change":"2021-03-15T22:34:39.122745+0000","last_active":"2021-03-15T22:35:36.346759+0000","last_peered":"2021-03-15T22:35:36.346759+0000","last_clean":"2021-03-15T22:35:36.346759+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:35:36.346759+0000","last_undegraded":"2021-03-15T22:35:36.346759+0000","last_fullsized":"2021-03-15T22:35:36.346759+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:34:39.122665+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.122665+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"996","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.342056+0000","last_change":"2021-03-15T22:34:10.331764+0000","last_active":"2021-03-15T22:35:35.342056+0000","last_peered":"2021-03-15T22:35:35.342056+0000","last_clean":"2021-03-15T22:35:35.342056+0000","last_became_active":"2021-03-15T22:34:10.331424+0000","last_became_peered":"2021-03-15T22:34:10.331424+0000","last_unstale":"2021-03-15T22:35:35.342056+0000","last_undegraded":"2021-03-15T22:35:35.342056+0000","last_fullsized":"2021-03-15T22:35:35.342056+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.564464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.564464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"512","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.352565+0000","last_change":"2021-03-15T22:34:55.389531+0000","last_active":"2021-03-15T22:35:39.352565+0000","last_peered":"2021-03-15T22:35:39.352565+0000","last_clean":"2021-03-15T22:35:39.352565+0000","last_became_active":"2021-03-15T22:34:55.356022+0000","last_became_peered":"2021-03-15T22:34:55.356022+0000","last_unstale":"2021-03-15T22:35:39.352565+0000","last_undegraded":"2021-03-15T22:35:39.352565+0000","last_fullsized":"2021-03-15T22:35:39.352565+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:32:31.680111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.680111+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2619,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"1079","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.341987+0000","last_change":"2021-03-15T22:35:14.434026+0000","last_active":"2021-03-15T22:35:35.341987+0000","last_peered":"2021-03-15T22:35:35.341987+0000","last_clean":"2021-03-15T22:35:35.341987+0000","last_became_active":"2021-03-15T22:33:04.345393+0000","last_became_peered":"2021-03-15T22:33:04.345393+0000","last_unstale":"2021-03-15T22:35:35.341987+0000","last_undegraded":"2021-03-15T22:35:35.341987+0000","last_fullsized":"2021-03-15T22:35:35.341987+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:35:14.433961+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:35:14.433961+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"283","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.341707+0000","last_change":"2021-03-15T22:33:04.345933+0000","last_active":"2021-03-15T22:35:35.341707+0000","last_peered":"2021-03-15T22:35:35.341707+0000","last_clean":"2021-03-15T22:35:35.341707+0000","last_became_active":"2021-03-15T22:33:04.345518+0000","last_became_peered":"2021-03-15T22:33:04.345518+0000","last_unstale":"2021-03-15T22:35:35.341707+0000","last_undegraded":"2021-03-15T22:35:35.341707+0000","last_fullsized":"2021-03-15T22:35:35.341707+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.338100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.338100+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"63","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.350605+0000","last_change":"2021-03-15T22:34:43.345093+0000","last_active":"2021-03-15T22:35:35.350605+0000","last_peered":"2021-03-15T22:35:35.350605+0000","last_clean":"2021-03-15T22:35:35.350605+0000","last_became_active":"2021-03-15T22:34:43.344764+0000","last_became_peered":"2021-03-15T22:34:43.344764+0000","last_unstale":"2021-03-15T22:35:35.350605+0000","last_undegraded":"2021-03-15T22:35:35.350605+0000","last_fullsized":"2021-03-15T22:35:35.350605+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:08.652741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.652741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"72","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.351265+0000","last_change":"2021-03-15T22:34:55.347147+0000","last_active":"2021-03-15T22:35:35.351265+0000","last_peered":"2021-03-15T22:35:35.351265+0000","last_clean":"2021-03-15T22:35:35.351265+0000","last_became_active":"2021-03-15T22:34:55.346711+0000","last_became_peered":"2021-03-15T22:34:55.346711+0000","last_unstale":"2021-03-15T22:35:35.351265+0000","last_undegraded":"2021-03-15T22:35:35.351265+0000","last_fullsized":"2021-03-15T22:35:35.351265+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.475850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.475850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"124","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.350574+0000","last_change":"2021-03-15T22:34:55.399761+0000","last_active":"2021-03-15T22:35:35.350574+0000","last_peered":"2021-03-15T22:35:35.350574+0000","last_clean":"2021-03-15T22:35:35.350574+0000","last_became_active":"2021-03-15T22:34:55.346148+0000","last_became_peered":"2021-03-15T22:34:55.346148+0000","last_unstale":"2021-03-15T22:35:35.350574+0000","last_undegraded":"2021-03-15T22:35:35.350574+0000","last_fullsized":"2021-03-15T22:35:35.350574+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.652898+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":5340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"833","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344251+0000","last_change":"2021-03-15T22:35:16.848985+0000","last_active":"2021-03-15T22:35:37.344251+0000","last_peered":"2021-03-15T22:35:37.344251+0000","last_clean":"2021-03-15T22:35:37.344251+0000","last_became_active":"2021-03-15T22:34:03.570306+0000","last_became_peered":"2021-03-15T22:34:03.570306+0000","last_unstale":"2021-03-15T22:35:37.344251+0000","last_undegraded":"2021-03-15T22:35:37.344251+0000","last_fullsized":"2021-03-15T22:35:37.344251+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:16.848918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.848918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"449","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350173+0000","last_change":"2021-03-15T22:32:01.452949+0000","last_active":"2021-03-15T22:35:39.350173+0000","last_peered":"2021-03-15T22:35:39.350173+0000","last_clean":"2021-03-15T22:35:39.350173+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:35:39.350173+0000","last_undegraded":"2021-03-15T22:35:39.350173+0000","last_fullsized":"2021-03-15T22:35:39.350173+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:01.452885+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:32:01.452885+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"376","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344399+0000","last_change":"2021-03-15T22:33:39.756677+0000","last_active":"2021-03-15T22:35:37.344399+0000","last_peered":"2021-03-15T22:35:37.344399+0000","last_clean":"2021-03-15T22:35:37.344399+0000","last_became_active":"2021-03-15T22:33:04.325611+0000","last_became_peered":"2021-03-15T22:33:04.325611+0000","last_unstale":"2021-03-15T22:35:37.344399+0000","last_undegraded":"2021-03-15T22:35:37.344399+0000","last_fullsized":"2021-03-15T22:35:37.344399+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.756581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"1130","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.342330+0000","last_change":"2021-03-15T22:33:17.502781+0000","last_active":"2021-03-15T22:35:35.342330+0000","last_peered":"2021-03-15T22:35:35.342330+0000","last_clean":"2021-03-15T22:35:35.342330+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:35:35.342330+0000","last_undegraded":"2021-03-15T22:35:35.342330+0000","last_fullsized":"2021-03-15T22:35:35.342330+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:33:17.502703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.502703+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"268","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.340989+0000","last_change":"2021-03-15T22:35:12.404980+0000","last_active":"2021-03-15T22:35:37.340989+0000","last_peered":"2021-03-15T22:35:37.340989+0000","last_clean":"2021-03-15T22:35:37.340989+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:35:37.340989+0000","last_undegraded":"2021-03-15T22:35:37.340989+0000","last_fullsized":"2021-03-15T22:35:37.340989+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.404897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.404897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"621","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344205+0000","last_change":"2021-03-15T22:34:43.344692+0000","last_active":"2021-03-15T22:35:37.344205+0000","last_peered":"2021-03-15T22:35:37.344205+0000","last_clean":"2021-03-15T22:35:37.344205+0000","last_became_active":"2021-03-15T22:34:43.344401+0000","last_became_peered":"2021-03-15T22:34:43.344401+0000","last_unstale":"2021-03-15T22:35:37.344205+0000","last_undegraded":"2021-03-15T22:35:37.344205+0000","last_fullsized":"2021-03-15T22:35:37.344205+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.774079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.774079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"358","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344827+0000","last_change":"2021-03-15T22:32:14.756556+0000","last_active":"2021-03-15T22:35:37.344827+0000","last_peered":"2021-03-15T22:35:37.344827+0000","last_clean":"2021-03-15T22:35:37.344827+0000","last_became_active":"2021-03-15T22:29:51.234868+0000","last_became_peered":"2021-03-15T22:29:51.234868+0000","last_unstale":"2021-03-15T22:35:37.344827+0000","last_undegraded":"2021-03-15T22:35:37.344827+0000","last_fullsized":"2021-03-15T22:35:37.344827+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:14.756460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:14.756460+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"1121","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.344049+0000","last_change":"2021-03-15T22:35:10.528771+0000","last_active":"2021-03-15T22:35:35.344049+0000","last_peered":"2021-03-15T22:35:35.344049+0000","last_clean":"2021-03-15T22:35:35.344049+0000","last_became_active":"2021-03-15T22:34:55.335550+0000","last_became_peered":"2021-03-15T22:34:55.335550+0000","last_unstale":"2021-03-15T22:35:35.344049+0000","last_undegraded":"2021-03-15T22:35:35.344049+0000","last_fullsized":"2021-03-15T22:35:35.344049+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:32:05.111031+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.111031+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"942","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.347665+0000","last_change":"2021-03-15T22:35:10.068109+0000","last_active":"2021-03-15T22:35:36.347665+0000","last_peered":"2021-03-15T22:35:36.347665+0000","last_clean":"2021-03-15T22:35:36.347665+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:35:36.347665+0000","last_undegraded":"2021-03-15T22:35:36.347665+0000","last_fullsized":"2021-03-15T22:35:36.347665+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:35:10.068030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.068030+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1003","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.346687+0000","last_change":"2021-03-15T22:34:41.614750+0000","last_active":"2021-03-15T22:35:35.346687+0000","last_peered":"2021-03-15T22:35:35.346687+0000","last_clean":"2021-03-15T22:35:35.346687+0000","last_became_active":"2021-03-15T22:33:04.329078+0000","last_became_peered":"2021-03-15T22:33:04.329078+0000","last_unstale":"2021-03-15T22:35:35.346687+0000","last_undegraded":"2021-03-15T22:35:35.346687+0000","last_fullsized":"2021-03-15T22:35:35.346687+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.614684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.614684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"453","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350980+0000","last_change":"2021-03-15T22:32:41.673856+0000","last_active":"2021-03-15T22:35:39.350980+0000","last_peered":"2021-03-15T22:35:39.350980+0000","last_clean":"2021-03-15T22:35:39.350980+0000","last_became_active":"2021-03-15T22:28:12.519619+0000","last_became_peered":"2021-03-15T22:28:12.519619+0000","last_unstale":"2021-03-15T22:35:39.350980+0000","last_undegraded":"2021-03-15T22:35:39.350980+0000","last_fullsized":"2021-03-15T22:35:39.350980+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:32:41.673792+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.673792+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"539","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.347142+0000","last_change":"2021-03-15T22:34:55.366147+0000","last_active":"2021-03-15T22:35:36.347142+0000","last_peered":"2021-03-15T22:35:36.347142+0000","last_clean":"2021-03-15T22:35:36.347142+0000","last_became_active":"2021-03-15T22:34:55.350325+0000","last_became_peered":"2021-03-15T22:34:55.350325+0000","last_unstale":"2021-03-15T22:35:36.347142+0000","last_undegraded":"2021-03-15T22:35:36.347142+0000","last_fullsized":"2021-03-15T22:35:36.347142+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:34:38.119156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:38.119156+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1029","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.340833+0000","last_change":"2021-03-15T22:35:09.568158+0000","last_active":"2021-03-15T22:35:35.340833+0000","last_peered":"2021-03-15T22:35:35.340833+0000","last_clean":"2021-03-15T22:35:35.340833+0000","last_became_active":"2021-03-15T22:34:55.338001+0000","last_became_peered":"2021-03-15T22:34:55.338001+0000","last_unstale":"2021-03-15T22:35:35.340833+0000","last_undegraded":"2021-03-15T22:35:35.340833+0000","last_fullsized":"2021-03-15T22:35:35.340833+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.144132+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.144132+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"818","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.353995+0000","last_change":"2021-03-15T22:35:20.025959+0000","last_active":"2021-03-15T22:35:35.353995+0000","last_peered":"2021-03-15T22:35:35.353995+0000","last_clean":"2021-03-15T22:35:35.353995+0000","last_became_active":"2021-03-15T22:33:04.336804+0000","last_became_peered":"2021-03-15T22:33:04.336804+0000","last_unstale":"2021-03-15T22:35:35.353995+0000","last_undegraded":"2021-03-15T22:35:35.353995+0000","last_fullsized":"2021-03-15T22:35:35.353995+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.025892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"993","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.342513+0000","last_change":"2021-03-15T22:34:55.343334+0000","last_active":"2021-03-15T22:35:35.342513+0000","last_peered":"2021-03-15T22:35:35.342513+0000","last_clean":"2021-03-15T22:35:35.342513+0000","last_became_active":"2021-03-15T22:34:55.342358+0000","last_became_peered":"2021-03-15T22:34:55.342358+0000","last_unstale":"2021-03-15T22:35:35.342513+0000","last_undegraded":"2021-03-15T22:35:35.342513+0000","last_fullsized":"2021-03-15T22:35:35.342513+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.429029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.429029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"429","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.352609+0000","last_change":"2021-03-15T22:33:15.617975+0000","last_active":"2021-03-15T22:35:35.352609+0000","last_peered":"2021-03-15T22:35:35.352609+0000","last_clean":"2021-03-15T22:35:35.352609+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:35:35.352609+0000","last_undegraded":"2021-03-15T22:35:35.352609+0000","last_fullsized":"2021-03-15T22:35:35.352609+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:33:15.617891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.617891+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"891'22","reported_seq":"554","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.347303+0000","last_change":"2021-03-15T22:35:38.710708+0000","last_active":"2021-03-15T22:35:39.347303+0000","last_peered":"2021-03-15T22:35:39.347303+0000","last_clean":"2021-03-15T22:35:39.347303+0000","last_became_active":"2021-03-15T22:34:06.311478+0000","last_became_peered":"2021-03-15T22:34:06.311478+0000","last_unstale":"2021-03-15T22:35:39.347303+0000","last_undegraded":"2021-03-15T22:35:39.347303+0000","last_fullsized":"2021-03-15T22:35:39.347303+0000","mapping_epoch":890,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"891'22","last_scrub_stamp":"2021-03-15T22:35:38.710643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:38.710643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1014","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341371+0000","last_change":"2021-03-15T22:34:55.352256+0000","last_active":"2021-03-15T22:35:37.341371+0000","last_peered":"2021-03-15T22:35:37.341371+0000","last_clean":"2021-03-15T22:35:37.341371+0000","last_became_active":"2021-03-15T22:34:55.351597+0000","last_became_peered":"2021-03-15T22:34:55.351597+0000","last_unstale":"2021-03-15T22:35:37.341371+0000","last_undegraded":"2021-03-15T22:35:37.341371+0000","last_fullsized":"2021-03-15T22:35:37.341371+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:43.393275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:43.393275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"1012","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.347082+0000","last_change":"2021-03-15T22:35:11.111684+0000","last_active":"2021-03-15T22:35:36.347082+0000","last_peered":"2021-03-15T22:35:36.347082+0000","last_clean":"2021-03-15T22:35:36.347082+0000","last_became_active":"2021-03-15T22:28:59.264559+0000","last_became_peered":"2021-03-15T22:28:59.264559+0000","last_unstale":"2021-03-15T22:35:36.347082+0000","last_undegraded":"2021-03-15T22:35:36.347082+0000","last_fullsized":"2021-03-15T22:35:36.347082+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:43.703019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.703019+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"587","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.353672+0000","last_change":"2021-03-15T22:34:10.385041+0000","last_active":"2021-03-15T22:35:39.353672+0000","last_peered":"2021-03-15T22:35:39.353672+0000","last_clean":"2021-03-15T22:35:39.353672+0000","last_became_active":"2021-03-15T22:34:10.347227+0000","last_became_peered":"2021-03-15T22:34:10.347227+0000","last_unstale":"2021-03-15T22:35:39.353672+0000","last_undegraded":"2021-03-15T22:35:39.353672+0000","last_fullsized":"2021-03-15T22:35:39.353672+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:34:07.125155+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.125155+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":3736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"1063","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.353138+0000","last_change":"2021-03-15T22:34:58.356466+0000","last_active":"2021-03-15T22:35:39.353138+0000","last_peered":"2021-03-15T22:35:39.353138+0000","last_clean":"2021-03-15T22:35:39.353138+0000","last_became_active":"2021-03-15T22:34:58.353086+0000","last_became_peered":"2021-03-15T22:34:58.353086+0000","last_unstale":"2021-03-15T22:35:39.353138+0000","last_undegraded":"2021-03-15T22:35:39.353138+0000","last_fullsized":"2021-03-15T22:35:39.353138+0000","mapping_epoch":942,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:32:34.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.739637+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"991","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.348098+0000","last_change":"2021-03-15T22:35:18.111879+0000","last_active":"2021-03-15T22:35:36.348098+0000","last_peered":"2021-03-15T22:35:36.348098+0000","last_clean":"2021-03-15T22:35:36.348098+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:35:36.348098+0000","last_undegraded":"2021-03-15T22:35:36.348098+0000","last_fullsized":"2021-03-15T22:35:36.348098+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:18.111815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.111815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"606","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.352957+0000","last_change":"2021-03-15T22:34:55.345698+0000","last_active":"2021-03-15T22:35:35.352957+0000","last_peered":"2021-03-15T22:35:35.352957+0000","last_clean":"2021-03-15T22:35:35.352957+0000","last_became_active":"2021-03-15T22:34:55.345277+0000","last_became_peered":"2021-03-15T22:34:55.345277+0000","last_unstale":"2021-03-15T22:35:35.352957+0000","last_undegraded":"2021-03-15T22:35:35.352957+0000","last_fullsized":"2021-03-15T22:35:35.352957+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.844183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.844183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"193","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.347207+0000","last_change":"2021-03-15T22:34:43.447677+0000","last_active":"2021-03-15T22:35:39.347207+0000","last_peered":"2021-03-15T22:35:39.347207+0000","last_clean":"2021-03-15T22:35:39.347207+0000","last_became_active":"2021-03-15T22:33:04.336100+0000","last_became_peered":"2021-03-15T22:33:04.336100+0000","last_unstale":"2021-03-15T22:35:39.347207+0000","last_undegraded":"2021-03-15T22:35:39.347207+0000","last_fullsized":"2021-03-15T22:35:39.347207+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:34:43.447602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:43.447602+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"996","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341056+0000","last_change":"2021-03-15T22:34:55.351849+0000","last_active":"2021-03-15T22:35:37.341056+0000","last_peered":"2021-03-15T22:35:37.341056+0000","last_clean":"2021-03-15T22:35:37.341056+0000","last_became_active":"2021-03-15T22:34:55.351089+0000","last_became_peered":"2021-03-15T22:34:55.351089+0000","last_unstale":"2021-03-15T22:35:37.341056+0000","last_undegraded":"2021-03-15T22:35:37.341056+0000","last_fullsized":"2021-03-15T22:35:37.341056+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.469521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.469521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"169","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.346056+0000","last_change":"2021-03-15T22:35:22.665325+0000","last_active":"2021-03-15T22:35:39.346056+0000","last_peered":"2021-03-15T22:35:39.346056+0000","last_clean":"2021-03-15T22:35:39.346056+0000","last_became_active":"2021-03-15T22:33:04.335734+0000","last_became_peered":"2021-03-15T22:33:04.335734+0000","last_unstale":"2021-03-15T22:35:39.346056+0000","last_undegraded":"2021-03-15T22:35:39.346056+0000","last_fullsized":"2021-03-15T22:35:39.346056+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.665251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.665251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"511","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.352560+0000","last_change":"2021-03-15T22:35:22.291513+0000","last_active":"2021-03-15T22:35:39.352560+0000","last_peered":"2021-03-15T22:35:39.352560+0000","last_clean":"2021-03-15T22:35:39.352560+0000","last_became_active":"2021-03-15T22:34:43.350301+0000","last_became_peered":"2021-03-15T22:34:43.350301+0000","last_unstale":"2021-03-15T22:35:39.352560+0000","last_undegraded":"2021-03-15T22:35:39.352560+0000","last_fullsized":"2021-03-15T22:35:39.352560+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.291406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:22.291406+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.291406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"493","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.353257+0000","last_change":"2021-03-15T22:33:37.752077+0000","last_active":"2021-03-15T22:35:35.353257+0000","last_peered":"2021-03-15T22:35:35.353257+0000","last_clean":"2021-03-15T22:35:35.353257+0000","last_became_active":"2021-03-15T22:33:04.342749+0000","last_became_peered":"2021-03-15T22:33:04.342749+0000","last_unstale":"2021-03-15T22:35:35.353257+0000","last_undegraded":"2021-03-15T22:35:35.353257+0000","last_fullsized":"2021-03-15T22:35:35.353257+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:33:37.752011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.752011+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":8764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"606","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.352378+0000","last_change":"2021-03-15T22:34:34.783107+0000","last_active":"2021-03-15T22:35:35.352378+0000","last_peered":"2021-03-15T22:35:35.352378+0000","last_clean":"2021-03-15T22:35:35.352378+0000","last_became_active":"2021-03-15T22:33:04.333077+0000","last_became_peered":"2021-03-15T22:33:04.333077+0000","last_unstale":"2021-03-15T22:35:35.352378+0000","last_undegraded":"2021-03-15T22:35:35.352378+0000","last_fullsized":"2021-03-15T22:35:35.352378+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.783047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.783047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"102","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.353334+0000","last_change":"2021-03-15T22:34:03.571320+0000","last_active":"2021-03-15T22:35:35.353334+0000","last_peered":"2021-03-15T22:35:35.353334+0000","last_clean":"2021-03-15T22:35:35.353334+0000","last_became_active":"2021-03-15T22:34:03.571049+0000","last_became_peered":"2021-03-15T22:34:03.571049+0000","last_unstale":"2021-03-15T22:35:35.353334+0000","last_undegraded":"2021-03-15T22:35:35.353334+0000","last_fullsized":"2021-03-15T22:35:35.353334+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:05.522372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.522372+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"358","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344658+0000","last_change":"2021-03-15T22:33:12.882245+0000","last_active":"2021-03-15T22:35:37.344658+0000","last_peered":"2021-03-15T22:35:37.344658+0000","last_clean":"2021-03-15T22:35:37.344658+0000","last_became_active":"2021-03-15T22:29:51.220261+0000","last_became_peered":"2021-03-15T22:29:51.220261+0000","last_unstale":"2021-03-15T22:35:37.344658+0000","last_undegraded":"2021-03-15T22:35:37.344658+0000","last_fullsized":"2021-03-15T22:35:37.344658+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.882164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.882164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"824","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.350955+0000","last_change":"2021-03-15T22:34:37.787302+0000","last_active":"2021-03-15T22:35:35.350955+0000","last_peered":"2021-03-15T22:35:35.350955+0000","last_clean":"2021-03-15T22:35:35.350955+0000","last_became_active":"2021-03-15T22:33:04.331607+0000","last_became_peered":"2021-03-15T22:33:04.331607+0000","last_unstale":"2021-03-15T22:35:35.350955+0000","last_undegraded":"2021-03-15T22:35:35.350955+0000","last_fullsized":"2021-03-15T22:35:35.350955+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:34:37.787235+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.787235+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"473","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350019+0000","last_change":"2021-03-15T22:35:17.250496+0000","last_active":"2021-03-15T22:35:39.350019+0000","last_peered":"2021-03-15T22:35:39.350019+0000","last_clean":"2021-03-15T22:35:39.350019+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:35:39.350019+0000","last_undegraded":"2021-03-15T22:35:39.350019+0000","last_fullsized":"2021-03-15T22:35:39.350019+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.250434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.250434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"773","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.351120+0000","last_change":"2021-03-15T22:32:37.690585+0000","last_active":"2021-03-15T22:35:35.351120+0000","last_peered":"2021-03-15T22:35:35.351120+0000","last_clean":"2021-03-15T22:35:35.351120+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:35:35.351120+0000","last_undegraded":"2021-03-15T22:35:35.351120+0000","last_fullsized":"2021-03-15T22:35:35.351120+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:37.690525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:32:37.690525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"989","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.543385+0000","last_change":"2021-03-15T22:35:37.543358+0000","last_active":"2021-03-15T22:35:37.543385+0000","last_peered":"2021-03-15T22:35:37.543385+0000","last_clean":"2021-03-15T22:35:37.543385+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:35:37.543385+0000","last_undegraded":"2021-03-15T22:35:37.543385+0000","last_fullsized":"2021-03-15T22:35:37.543385+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:37.543289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:35:37.543289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"700","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.345980+0000","last_change":"2021-03-15T22:34:56.210802+0000","last_active":"2021-03-15T22:35:39.345980+0000","last_peered":"2021-03-15T22:35:39.345980+0000","last_clean":"2021-03-15T22:35:39.345980+0000","last_became_active":"2021-03-15T22:34:55.344555+0000","last_became_peered":"2021-03-15T22:34:55.344555+0000","last_unstale":"2021-03-15T22:35:39.345980+0000","last_undegraded":"2021-03-15T22:35:39.345980+0000","last_fullsized":"2021-03-15T22:35:39.345980+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:34:42.447631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.447631+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"370","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350466+0000","last_change":"2021-03-15T22:35:23.291568+0000","last_active":"2021-03-15T22:35:39.350466+0000","last_peered":"2021-03-15T22:35:39.350466+0000","last_clean":"2021-03-15T22:35:39.350466+0000","last_became_active":"2021-03-15T22:34:55.355119+0000","last_became_peered":"2021-03-15T22:34:55.355119+0000","last_unstale":"2021-03-15T22:35:39.350466+0000","last_undegraded":"2021-03-15T22:35:39.350466+0000","last_fullsized":"2021-03-15T22:35:39.350466+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.291507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.291507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"1039","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.346701+0000","last_change":"2021-03-15T22:34:10.353649+0000","last_active":"2021-03-15T22:35:36.346701+0000","last_peered":"2021-03-15T22:35:36.346701+0000","last_clean":"2021-03-15T22:35:36.346701+0000","last_became_active":"2021-03-15T22:34:10.353314+0000","last_became_peered":"2021-03-15T22:34:10.353314+0000","last_unstale":"2021-03-15T22:35:36.346701+0000","last_undegraded":"2021-03-15T22:35:36.346701+0000","last_fullsized":"2021-03-15T22:35:36.346701+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:43.268612+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.268612+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"116","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.344207+0000","last_change":"2021-03-15T22:33:45.314503+0000","last_active":"2021-03-15T22:35:35.344207+0000","last_peered":"2021-03-15T22:35:35.344207+0000","last_clean":"2021-03-15T22:35:35.344207+0000","last_became_active":"2021-03-15T22:33:45.314042+0000","last_became_peered":"2021-03-15T22:33:45.314042+0000","last_unstale":"2021-03-15T22:35:35.344207+0000","last_undegraded":"2021-03-15T22:35:35.344207+0000","last_fullsized":"2021-03-15T22:35:35.344207+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:08.884878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.884878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"1067","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.343944+0000","last_change":"2021-03-15T22:34:42.632034+0000","last_active":"2021-03-15T22:35:35.343944+0000","last_peered":"2021-03-15T22:35:35.343944+0000","last_clean":"2021-03-15T22:35:35.343944+0000","last_became_active":"2021-03-15T22:33:04.340460+0000","last_became_peered":"2021-03-15T22:33:04.340460+0000","last_unstale":"2021-03-15T22:35:35.343944+0000","last_undegraded":"2021-03-15T22:35:35.343944+0000","last_fullsized":"2021-03-15T22:35:35.343944+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:34:42.631962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.631962+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"476","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.353902+0000","last_change":"2021-03-15T22:34:10.327350+0000","last_active":"2021-03-15T22:35:39.353902+0000","last_peered":"2021-03-15T22:35:39.353902+0000","last_clean":"2021-03-15T22:35:39.353902+0000","last_became_active":"2021-03-15T22:34:10.326881+0000","last_became_peered":"2021-03-15T22:34:10.326881+0000","last_unstale":"2021-03-15T22:35:39.353902+0000","last_undegraded":"2021-03-15T22:35:39.353902+0000","last_fullsized":"2021-03-15T22:35:39.353902+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:19.560387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:19.560387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"808'9","reported_seq":"433","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344491+0000","last_change":"2021-03-15T22:32:38.881054+0000","last_active":"2021-03-15T22:35:37.344491+0000","last_peered":"2021-03-15T22:35:37.344491+0000","last_clean":"2021-03-15T22:35:37.344491+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:35:37.344491+0000","last_undegraded":"2021-03-15T22:35:37.344491+0000","last_fullsized":"2021-03-15T22:35:37.344491+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9","last_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_deep_scrub":"808'9","last_deep_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.880993+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"981","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341096+0000","last_change":"2021-03-15T22:34:55.351414+0000","last_active":"2021-03-15T22:35:37.341096+0000","last_peered":"2021-03-15T22:35:37.341096+0000","last_clean":"2021-03-15T22:35:37.341096+0000","last_became_active":"2021-03-15T22:34:55.351042+0000","last_became_peered":"2021-03-15T22:34:55.351042+0000","last_unstale":"2021-03-15T22:35:37.341096+0000","last_undegraded":"2021-03-15T22:35:37.341096+0000","last_fullsized":"2021-03-15T22:35:37.341096+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.516668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.516668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"844'33","reported_seq":"839","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.352844+0000","last_change":"2021-03-15T22:34:55.387712+0000","last_active":"2021-03-15T22:35:35.352844+0000","last_peered":"2021-03-15T22:35:35.352844+0000","last_clean":"2021-03-15T22:35:35.352844+0000","last_became_active":"2021-03-15T22:34:55.351721+0000","last_became_peered":"2021-03-15T22:34:55.351721+0000","last_unstale":"2021-03-15T22:35:35.352844+0000","last_undegraded":"2021-03-15T22:35:35.352844+0000","last_fullsized":"2021-03-15T22:35:35.352844+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"844'33","last_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_deep_scrub":"844'33","last_deep_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.622437+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1015","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.342874+0000","last_change":"2021-03-15T22:35:08.565833+0000","last_active":"2021-03-15T22:35:35.342874+0000","last_peered":"2021-03-15T22:35:35.342874+0000","last_clean":"2021-03-15T22:35:35.342874+0000","last_became_active":"2021-03-15T22:34:55.340440+0000","last_became_peered":"2021-03-15T22:34:55.340440+0000","last_unstale":"2021-03-15T22:35:35.342874+0000","last_undegraded":"2021-03-15T22:35:35.342874+0000","last_fullsized":"2021-03-15T22:35:35.342874+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:41.407324+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:41.407324+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"941'17","reported_seq":"508","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341287+0000","last_change":"2021-03-15T22:34:55.473459+0000","last_active":"2021-03-15T22:35:37.341287+0000","last_peered":"2021-03-15T22:35:37.341287+0000","last_clean":"2021-03-15T22:35:37.341287+0000","last_became_active":"2021-03-15T22:34:55.351682+0000","last_became_peered":"2021-03-15T22:34:55.351682+0000","last_unstale":"2021-03-15T22:35:37.341287+0000","last_undegraded":"2021-03-15T22:35:37.341287+0000","last_fullsized":"2021-03-15T22:35:37.341287+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"896'16","last_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_deep_scrub":"896'16","last_deep_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_clean_scrub_stamp":"2021-03-15T22:34:19.400191+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"476","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350427+0000","last_change":"2021-03-15T22:34:10.347844+0000","last_active":"2021-03-15T22:35:39.350427+0000","last_peered":"2021-03-15T22:35:39.350427+0000","last_clean":"2021-03-15T22:35:39.350427+0000","last_became_active":"2021-03-15T22:34:10.347512+0000","last_became_peered":"2021-03-15T22:34:10.347512+0000","last_unstale":"2021-03-15T22:35:39.350427+0000","last_undegraded":"2021-03-15T22:35:39.350427+0000","last_fullsized":"2021-03-15T22:35:39.350427+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:03.502854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:32:03.502854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"376","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344312+0000","last_change":"2021-03-15T22:32:41.879032+0000","last_active":"2021-03-15T22:35:37.344312+0000","last_peered":"2021-03-15T22:35:37.344312+0000","last_clean":"2021-03-15T22:35:37.344312+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:35:37.344312+0000","last_undegraded":"2021-03-15T22:35:37.344312+0000","last_fullsized":"2021-03-15T22:35:37.344312+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:32:41.878949+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.878949+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"63","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.353601+0000","last_change":"2021-03-15T22:34:43.345092+0000","last_active":"2021-03-15T22:35:35.353601+0000","last_peered":"2021-03-15T22:35:35.353601+0000","last_clean":"2021-03-15T22:35:35.353601+0000","last_became_active":"2021-03-15T22:34:43.344790+0000","last_became_peered":"2021-03-15T22:34:43.344790+0000","last_unstale":"2021-03-15T22:35:35.353601+0000","last_undegraded":"2021-03-15T22:35:35.353601+0000","last_fullsized":"2021-03-15T22:35:35.353601+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.621539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"59","reported_epoch":"983","state":"active+clean","last_fresh":"2021-03-15T22:35:38.343815+0000","last_change":"2021-03-15T22:35:20.381407+0000","last_active":"2021-03-15T22:35:38.343815+0000","last_peered":"2021-03-15T22:35:38.343815+0000","last_clean":"2021-03-15T22:35:38.343815+0000","last_became_active":"2021-03-15T22:34:55.348663+0000","last_became_peered":"2021-03-15T22:34:55.348663+0000","last_unstale":"2021-03-15T22:35:38.343815+0000","last_undegraded":"2021-03-15T22:35:38.343815+0000","last_fullsized":"2021-03-15T22:35:38.343815+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.381352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"974","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.340898+0000","last_change":"2021-03-15T22:33:21.545910+0000","last_active":"2021-03-15T22:35:37.340898+0000","last_peered":"2021-03-15T22:35:37.340898+0000","last_clean":"2021-03-15T22:35:37.340898+0000","last_became_active":"2021-03-15T22:33:04.330195+0000","last_became_peered":"2021-03-15T22:33:04.330195+0000","last_unstale":"2021-03-15T22:35:37.340898+0000","last_undegraded":"2021-03-15T22:35:37.340898+0000","last_fullsized":"2021-03-15T22:35:37.340898+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.545826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.545826+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"512","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.353336+0000","last_change":"2021-03-15T22:34:37.435070+0000","last_active":"2021-03-15T22:35:39.353336+0000","last_peered":"2021-03-15T22:35:39.353336+0000","last_clean":"2021-03-15T22:35:39.353336+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:35:39.353336+0000","last_undegraded":"2021-03-15T22:35:39.353336+0000","last_fullsized":"2021-03-15T22:35:39.353336+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:34:37.435006+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.435006+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"785","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344680+0000","last_change":"2021-03-15T22:35:20.884067+0000","last_active":"2021-03-15T22:35:37.344680+0000","last_peered":"2021-03-15T22:35:37.344680+0000","last_clean":"2021-03-15T22:35:37.344680+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:35:37.344680+0000","last_undegraded":"2021-03-15T22:35:37.344680+0000","last_fullsized":"2021-03-15T22:35:37.344680+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.883990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.883990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"968","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341525+0000","last_change":"2021-03-15T22:33:19.496857+0000","last_active":"2021-03-15T22:35:37.341525+0000","last_peered":"2021-03-15T22:35:37.341525+0000","last_clean":"2021-03-15T22:35:37.341525+0000","last_became_active":"2021-03-15T22:33:04.331013+0000","last_became_peered":"2021-03-15T22:33:04.331013+0000","last_unstale":"2021-03-15T22:35:37.341525+0000","last_undegraded":"2021-03-15T22:35:37.341525+0000","last_fullsized":"2021-03-15T22:35:37.341525+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:19.496753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:19.496753+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"164","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.346056+0000","last_change":"2021-03-15T22:34:07.203702+0000","last_active":"2021-03-15T22:35:39.346056+0000","last_peered":"2021-03-15T22:35:39.346056+0000","last_clean":"2021-03-15T22:35:39.346056+0000","last_became_active":"2021-03-15T22:33:04.336886+0000","last_became_peered":"2021-03-15T22:33:04.336886+0000","last_unstale":"2021-03-15T22:35:39.346056+0000","last_undegraded":"2021-03-15T22:35:39.346056+0000","last_fullsized":"2021-03-15T22:35:39.346056+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.203639+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:39.697935+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.203639+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"332","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341801+0000","last_change":"2021-03-15T22:34:55.387006+0000","last_active":"2021-03-15T22:35:37.341801+0000","last_peered":"2021-03-15T22:35:37.341801+0000","last_clean":"2021-03-15T22:35:37.341801+0000","last_became_active":"2021-03-15T22:34:55.351977+0000","last_became_peered":"2021-03-15T22:34:55.351977+0000","last_unstale":"2021-03-15T22:35:37.341801+0000","last_undegraded":"2021-03-15T22:35:37.341801+0000","last_fullsized":"2021-03-15T22:35:37.341801+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:34:36.438541+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:34:36.438541+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"980'1403","reported_seq":"4470","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.348148+0000","last_change":"2021-03-15T22:35:35.354417+0000","last_active":"2021-03-15T22:35:36.348148+0000","last_peered":"2021-03-15T22:35:36.348148+0000","last_clean":"2021-03-15T22:35:36.348148+0000","last_became_active":"2021-03-15T22:33:08.355490+0000","last_became_peered":"2021-03-15T22:33:08.355490+0000","last_unstale":"2021-03-15T22:35:36.348148+0000","last_undegraded":"2021-03-15T22:35:36.348148+0000","last_fullsized":"2021-03-15T22:35:36.348148+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'1350","last_scrub_stamp":"2021-03-15T22:35:16.036089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.036089+0000","log_size":1403,"ondisk_log_size":1403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50100108,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":471,"num_read_kb":519759,"num_write":332,"num_write_kb":119214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":84869361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2be","length":"1"},{"start":"2c3","length":"1"},{"start":"2d1","length":"1"},{"start":"30d","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"925","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344405+0000","last_change":"2021-03-15T22:33:42.741529+0000","last_active":"2021-03-15T22:35:37.344405+0000","last_peered":"2021-03-15T22:35:37.344405+0000","last_clean":"2021-03-15T22:35:37.344405+0000","last_became_active":"2021-03-15T22:33:04.340352+0000","last_became_peered":"2021-03-15T22:33:04.340352+0000","last_unstale":"2021-03-15T22:35:37.344405+0000","last_undegraded":"2021-03-15T22:35:37.344405+0000","last_fullsized":"2021-03-15T22:35:37.344405+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:42.741451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:42.741451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"874'4","reported_seq":"991","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341744+0000","last_change":"2021-03-15T22:33:44.498327+0000","last_active":"2021-03-15T22:35:37.341744+0000","last_peered":"2021-03-15T22:35:37.341744+0000","last_clean":"2021-03-15T22:35:37.341744+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:35:37.341744+0000","last_undegraded":"2021-03-15T22:35:37.341744+0000","last_fullsized":"2021-03-15T22:35:37.341744+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"874'4","last_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_deep_scrub":"874'4","last_deep_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.498263+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"452","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344045+0000","last_change":"2021-03-15T22:33:38.759477+0000","last_active":"2021-03-15T22:35:37.344045+0000","last_peered":"2021-03-15T22:35:37.344045+0000","last_clean":"2021-03-15T22:35:37.344045+0000","last_became_active":"2021-03-15T22:33:04.326070+0000","last_became_peered":"2021-03-15T22:33:04.326070+0000","last_unstale":"2021-03-15T22:35:37.344045+0000","last_undegraded":"2021-03-15T22:35:37.344045+0000","last_fullsized":"2021-03-15T22:35:37.344045+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:33:38.759395+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.759395+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"841'14","reported_seq":"1249","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.340987+0000","last_change":"2021-03-15T22:33:43.476783+0000","last_active":"2021-03-15T22:35:37.340987+0000","last_peered":"2021-03-15T22:35:37.340987+0000","last_clean":"2021-03-15T22:35:37.340987+0000","last_became_active":"2021-03-15T22:33:08.354031+0000","last_became_peered":"2021-03-15T22:33:08.354031+0000","last_unstale":"2021-03-15T22:35:37.340987+0000","last_undegraded":"2021-03-15T22:35:37.340987+0000","last_fullsized":"2021-03-15T22:35:37.340987+0000","mapping_epoch":837,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'14","last_scrub_stamp":"2021-03-15T22:33:43.476701+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.476701+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"980'2019","reported_seq":"5356","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.380333+0000","last_change":"2021-03-15T22:35:35.380333+0000","last_active":"2021-03-15T22:35:35.380333+0000","last_peered":"2021-03-15T22:35:35.380333+0000","last_clean":"2021-03-15T22:35:35.380333+0000","last_became_active":"2021-03-15T22:34:58.336162+0000","last_became_peered":"2021-03-15T22:34:58.336162+0000","last_unstale":"2021-03-15T22:35:35.380333+0000","last_undegraded":"2021-03-15T22:35:35.380333+0000","last_fullsized":"2021-03-15T22:35:35.380333+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"772'1312","last_scrub_stamp":"2021-03-15T22:32:05.639057+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:05.639057+0000","log_size":2019,"ondisk_log_size":2019,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":75967361,"num_objects":36,"num_object_clones":31,"num_object_copies":108,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":36,"num_whiteouts":1,"num_read":680,"num_read_kb":649871,"num_write":523,"num_write_kb":204349,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":198498497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2be","length":"1"},{"start":"2c3","length":"1"},{"start":"2d1","length":"1"},{"start":"30d","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"409","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.351078+0000","last_change":"2021-03-15T22:32:42.621654+0000","last_active":"2021-03-15T22:35:35.351078+0000","last_peered":"2021-03-15T22:35:35.351078+0000","last_clean":"2021-03-15T22:35:35.351078+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:35:35.351078+0000","last_undegraded":"2021-03-15T22:35:35.351078+0000","last_fullsized":"2021-03-15T22:35:35.351078+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:42.621594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:42.621594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"958","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341911+0000","last_change":"2021-03-15T22:33:22.542596+0000","last_active":"2021-03-15T22:35:37.341911+0000","last_peered":"2021-03-15T22:35:37.341911+0000","last_clean":"2021-03-15T22:35:37.341911+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:35:37.341911+0000","last_undegraded":"2021-03-15T22:35:37.341911+0000","last_fullsized":"2021-03-15T22:35:37.341911+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.542506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.542506+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"1190","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.343460+0000","last_change":"2021-03-15T22:34:11.031994+0000","last_active":"2021-03-15T22:35:35.343460+0000","last_peered":"2021-03-15T22:35:35.343460+0000","last_clean":"2021-03-15T22:35:35.343460+0000","last_became_active":"2021-03-15T22:34:10.342120+0000","last_became_peered":"2021-03-15T22:34:10.342120+0000","last_unstale":"2021-03-15T22:35:35.343460+0000","last_undegraded":"2021-03-15T22:35:35.343460+0000","last_fullsized":"2021-03-15T22:35:35.343460+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:34:07.452572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.452572+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"697","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350098+0000","last_change":"2021-03-15T22:34:55.356651+0000","last_active":"2021-03-15T22:35:39.350098+0000","last_peered":"2021-03-15T22:35:39.350098+0000","last_clean":"2021-03-15T22:35:39.350098+0000","last_became_active":"2021-03-15T22:34:55.356328+0000","last_became_peered":"2021-03-15T22:34:55.356328+0000","last_unstale":"2021-03-15T22:35:39.350098+0000","last_undegraded":"2021-03-15T22:35:39.350098+0000","last_fullsized":"2021-03-15T22:35:39.350098+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"984'560","reported_seq":"3351","reported_epoch":"985","state":"active+clean","last_fresh":"2021-03-15T22:35:40.344300+0000","last_change":"2021-03-15T22:35:39.359068+0000","last_active":"2021-03-15T22:35:40.344300+0000","last_peered":"2021-03-15T22:35:40.344300+0000","last_clean":"2021-03-15T22:35:40.344300+0000","last_became_active":"2021-03-15T22:34:13.324667+0000","last_became_peered":"2021-03-15T22:34:13.324667+0000","last_unstale":"2021-03-15T22:35:40.344300+0000","last_undegraded":"2021-03-15T22:35:40.344300+0000","last_fullsized":"2021-03-15T22:35:40.344300+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'446","last_scrub_stamp":"2021-03-15T22:33:39.355306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.355306+0000","log_size":560,"ondisk_log_size":560,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17801208,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":139,"num_read_kb":111592,"num_write":130,"num_write_kb":42276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":23698344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c3","length":"1"},{"start":"2f3","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"1026","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.340702+0000","last_change":"2021-03-15T22:34:55.354150+0000","last_active":"2021-03-15T22:35:37.340702+0000","last_peered":"2021-03-15T22:35:37.340702+0000","last_clean":"2021-03-15T22:35:37.340702+0000","last_became_active":"2021-03-15T22:34:55.353827+0000","last_became_peered":"2021-03-15T22:34:55.353827+0000","last_unstale":"2021-03-15T22:35:37.340702+0000","last_undegraded":"2021-03-15T22:35:37.340702+0000","last_fullsized":"2021-03-15T22:35:37.340702+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"54","reported_epoch":"983","state":"active+clean","last_fresh":"2021-03-15T22:35:38.344108+0000","last_change":"2021-03-15T22:34:55.354467+0000","last_active":"2021-03-15T22:35:38.344108+0000","last_peered":"2021-03-15T22:35:38.344108+0000","last_clean":"2021-03-15T22:35:38.344108+0000","last_became_active":"2021-03-15T22:34:55.353966+0000","last_became_peered":"2021-03-15T22:34:55.353966+0000","last_unstale":"2021-03-15T22:35:38.344108+0000","last_undegraded":"2021-03-15T22:35:38.344108+0000","last_fullsized":"2021-03-15T22:35:38.344108+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.809863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.809863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"1018","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341705+0000","last_change":"2021-03-15T22:35:23.428929+0000","last_active":"2021-03-15T22:35:37.341705+0000","last_peered":"2021-03-15T22:35:37.341705+0000","last_clean":"2021-03-15T22:35:37.341705+0000","last_became_active":"2021-03-15T22:33:04.330771+0000","last_became_peered":"2021-03-15T22:33:04.330771+0000","last_unstale":"2021-03-15T22:35:37.341705+0000","last_undegraded":"2021-03-15T22:35:37.341705+0000","last_fullsized":"2021-03-15T22:35:37.341705+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.428842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.428842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"229","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.346077+0000","last_change":"2021-03-15T22:34:10.249538+0000","last_active":"2021-03-15T22:35:39.346077+0000","last_peered":"2021-03-15T22:35:39.346077+0000","last_clean":"2021-03-15T22:35:39.346077+0000","last_became_active":"2021-03-15T22:33:45.318867+0000","last_became_peered":"2021-03-15T22:33:45.318867+0000","last_unstale":"2021-03-15T22:35:39.346077+0000","last_undegraded":"2021-03-15T22:35:39.346077+0000","last_fullsized":"2021-03-15T22:35:39.346077+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:34:10.249444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:10.249444+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":2928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"984'1238","reported_seq":"4105","reported_epoch":"984","state":"active+clean+snaptrim","last_fresh":"2021-03-15T22:35:39.357137+0000","last_change":"2021-03-15T22:35:39.346148+0000","last_active":"2021-03-15T22:35:39.357137+0000","last_peered":"2021-03-15T22:35:39.357137+0000","last_clean":"2021-03-15T22:35:39.357137+0000","last_became_active":"2021-03-15T22:34:59.339587+0000","last_became_peered":"2021-03-15T22:34:59.339587+0000","last_unstale":"2021-03-15T22:35:39.357137+0000","last_undegraded":"2021-03-15T22:35:39.357137+0000","last_fullsized":"2021-03-15T22:35:39.357137+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'1086","last_scrub_stamp":"2021-03-15T22:34:41.436839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.436839+0000","log_size":1238,"ondisk_log_size":1238,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"stat_sum":{"num_bytes":49340032,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":1,"num_read":425,"num_read_kb":511498,"num_write":281,"num_write_kb":101057,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":205148835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2be","length":"1"},{"start":"2c3","length":"1"},{"start":"2d1","length":"1"},{"start":"30d","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"487","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350543+0000","last_change":"2021-03-15T22:34:55.356122+0000","last_active":"2021-03-15T22:35:39.350543+0000","last_peered":"2021-03-15T22:35:39.350543+0000","last_clean":"2021-03-15T22:35:39.350543+0000","last_became_active":"2021-03-15T22:34:55.355571+0000","last_became_peered":"2021-03-15T22:34:55.355571+0000","last_unstale":"2021-03-15T22:35:39.350543+0000","last_undegraded":"2021-03-15T22:35:39.350543+0000","last_fullsized":"2021-03-15T22:35:39.350543+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"814","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.343713+0000","last_change":"2021-03-15T22:34:55.343547+0000","last_active":"2021-03-15T22:35:35.343713+0000","last_peered":"2021-03-15T22:35:35.343713+0000","last_clean":"2021-03-15T22:35:35.343713+0000","last_became_active":"2021-03-15T22:34:55.342808+0000","last_became_peered":"2021-03-15T22:34:55.342808+0000","last_unstale":"2021-03-15T22:35:35.343713+0000","last_undegraded":"2021-03-15T22:35:35.343713+0000","last_fullsized":"2021-03-15T22:35:35.343713+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:14.888119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.888119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"383","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344782+0000","last_change":"2021-03-15T22:34:07.807466+0000","last_active":"2021-03-15T22:35:37.344782+0000","last_peered":"2021-03-15T22:35:37.344782+0000","last_clean":"2021-03-15T22:35:37.344782+0000","last_became_active":"2021-03-15T22:33:04.333291+0000","last_became_peered":"2021-03-15T22:33:04.333291+0000","last_unstale":"2021-03-15T22:35:37.344782+0000","last_undegraded":"2021-03-15T22:35:37.344782+0000","last_fullsized":"2021-03-15T22:35:37.344782+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.807411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.807411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"527","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.351058+0000","last_change":"2021-03-15T22:33:46.129541+0000","last_active":"2021-03-15T22:35:39.351058+0000","last_peered":"2021-03-15T22:35:39.351058+0000","last_clean":"2021-03-15T22:35:39.351058+0000","last_became_active":"2021-03-15T22:33:45.326747+0000","last_became_peered":"2021-03-15T22:33:45.326747+0000","last_unstale":"2021-03-15T22:35:39.351058+0000","last_undegraded":"2021-03-15T22:35:39.351058+0000","last_fullsized":"2021-03-15T22:35:39.351058+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:32:33.701675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.701675+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":2682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"984'1410","reported_seq":"4959","reported_epoch":"985","state":"active+clean","last_fresh":"2021-03-15T22:35:40.372304+0000","last_change":"2021-03-15T22:35:39.362551+0000","last_active":"2021-03-15T22:35:40.372304+0000","last_peered":"2021-03-15T22:35:40.372304+0000","last_clean":"2021-03-15T22:35:40.372304+0000","last_became_active":"2021-03-15T22:34:46.338888+0000","last_became_peered":"2021-03-15T22:34:46.338888+0000","last_unstale":"2021-03-15T22:35:40.372304+0000","last_undegraded":"2021-03-15T22:35:40.372304+0000","last_fullsized":"2021-03-15T22:35:40.372304+0000","mapping_epoch":930,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'1253","last_scrub_stamp":"2021-03-15T22:34:41.490879+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.490879+0000","log_size":1410,"ondisk_log_size":1410,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":47539166,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":330,"num_read_kb":274744,"num_write":345,"num_write_kb":124940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":229919087,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c3","length":"1"},{"start":"2f3","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"63","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.352809+0000","last_change":"2021-03-15T22:34:43.342616+0000","last_active":"2021-03-15T22:35:35.352809+0000","last_peered":"2021-03-15T22:35:35.352809+0000","last_clean":"2021-03-15T22:35:35.352809+0000","last_became_active":"2021-03-15T22:34:43.342089+0000","last_became_peered":"2021-03-15T22:34:43.342089+0000","last_unstale":"2021-03-15T22:35:35.352809+0000","last_undegraded":"2021-03-15T22:35:35.352809+0000","last_fullsized":"2021-03-15T22:35:35.352809+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"29","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.340881+0000","last_change":"2021-03-15T22:35:19.502186+0000","last_active":"2021-03-15T22:35:37.340881+0000","last_peered":"2021-03-15T22:35:37.340881+0000","last_clean":"2021-03-15T22:35:37.340881+0000","last_became_active":"2021-03-15T22:35:19.501783+0000","last_became_peered":"2021-03-15T22:35:19.501783+0000","last_unstale":"2021-03-15T22:35:37.340881+0000","last_undegraded":"2021-03-15T22:35:37.340881+0000","last_fullsized":"2021-03-15T22:35:37.340881+0000","mapping_epoch":963,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":964,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.831369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.831369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"178","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.343752+0000","last_change":"2021-03-15T22:33:04.336661+0000","last_active":"2021-03-15T22:35:35.343752+0000","last_peered":"2021-03-15T22:35:35.343752+0000","last_clean":"2021-03-15T22:35:35.343752+0000","last_became_active":"2021-03-15T22:33:04.335914+0000","last_became_peered":"2021-03-15T22:33:04.335914+0000","last_unstale":"2021-03-15T22:35:35.343752+0000","last_undegraded":"2021-03-15T22:35:35.343752+0000","last_fullsized":"2021-03-15T22:35:35.343752+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.051962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.051962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1349","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.347442+0000","last_change":"2021-03-15T22:34:55.382040+0000","last_active":"2021-03-15T22:35:36.347442+0000","last_peered":"2021-03-15T22:35:36.347442+0000","last_clean":"2021-03-15T22:35:36.347442+0000","last_became_active":"2021-03-15T22:34:55.350419+0000","last_became_peered":"2021-03-15T22:34:55.350419+0000","last_unstale":"2021-03-15T22:35:36.347442+0000","last_undegraded":"2021-03-15T22:35:36.347442+0000","last_fullsized":"2021-03-15T22:35:36.347442+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:33:38.133617+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.133617+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"449","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350785+0000","last_change":"2021-03-15T22:33:15.492116+0000","last_active":"2021-03-15T22:35:39.350785+0000","last_peered":"2021-03-15T22:35:39.350785+0000","last_clean":"2021-03-15T22:35:39.350785+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:35:39.350785+0000","last_undegraded":"2021-03-15T22:35:39.350785+0000","last_fullsized":"2021-03-15T22:35:39.350785+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.492014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.492014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"984'1316","reported_seq":"3981","reported_epoch":"985","state":"active+clean","last_fresh":"2021-03-15T22:35:40.345351+0000","last_change":"2021-03-15T22:35:39.359715+0000","last_active":"2021-03-15T22:35:40.345351+0000","last_peered":"2021-03-15T22:35:40.345351+0000","last_clean":"2021-03-15T22:35:40.345351+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:35:40.345351+0000","last_undegraded":"2021-03-15T22:35:40.345351+0000","last_fullsized":"2021-03-15T22:35:40.345351+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"805'997","last_scrub_stamp":"2021-03-15T22:32:36.795301+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.795301+0000","log_size":1316,"ondisk_log_size":1316,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":48557346,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":0,"num_read":397,"num_read_kb":397985,"num_write":302,"num_write_kb":114364,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c3","length":"1"},{"start":"2f3","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"1009","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341777+0000","last_change":"2021-03-15T22:34:34.631500+0000","last_active":"2021-03-15T22:35:37.341777+0000","last_peered":"2021-03-15T22:35:37.341777+0000","last_clean":"2021-03-15T22:35:37.341777+0000","last_became_active":"2021-03-15T22:34:34.631216+0000","last_became_peered":"2021-03-15T22:34:34.631216+0000","last_unstale":"2021-03-15T22:35:37.341777+0000","last_undegraded":"2021-03-15T22:35:37.341777+0000","last_fullsized":"2021-03-15T22:35:37.341777+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:32.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"1013","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.346893+0000","last_change":"2021-03-15T22:34:55.347340+0000","last_active":"2021-03-15T22:35:36.346893+0000","last_peered":"2021-03-15T22:35:36.346893+0000","last_clean":"2021-03-15T22:35:36.346893+0000","last_became_active":"2021-03-15T22:34:55.347065+0000","last_became_peered":"2021-03-15T22:34:55.347065+0000","last_unstale":"2021-03-15T22:35:36.346893+0000","last_undegraded":"2021-03-15T22:35:36.346893+0000","last_fullsized":"2021-03-15T22:35:36.346893+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:10.222521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:10.222521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"59","reported_epoch":"983","state":"active+clean","last_fresh":"2021-03-15T22:35:38.342918+0000","last_change":"2021-03-15T22:35:17.301942+0000","last_active":"2021-03-15T22:35:38.342918+0000","last_peered":"2021-03-15T22:35:38.342918+0000","last_clean":"2021-03-15T22:35:38.342918+0000","last_became_active":"2021-03-15T22:34:55.355485+0000","last_became_peered":"2021-03-15T22:34:55.355485+0000","last_unstale":"2021-03-15T22:35:38.342918+0000","last_undegraded":"2021-03-15T22:35:38.342918+0000","last_fullsized":"2021-03-15T22:35:38.342918+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.301892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.301892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"466","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344579+0000","last_change":"2021-03-15T22:34:10.343282+0000","last_active":"2021-03-15T22:35:37.344579+0000","last_peered":"2021-03-15T22:35:37.344579+0000","last_clean":"2021-03-15T22:35:37.344579+0000","last_became_active":"2021-03-15T22:34:10.322666+0000","last_became_peered":"2021-03-15T22:34:10.322666+0000","last_unstale":"2021-03-15T22:35:37.344579+0000","last_undegraded":"2021-03-15T22:35:37.344579+0000","last_fullsized":"2021-03-15T22:35:37.344579+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:33:40.214707+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.214707+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"54","reported_epoch":"983","state":"active+clean","last_fresh":"2021-03-15T22:35:38.344595+0000","last_change":"2021-03-15T22:34:55.353621+0000","last_active":"2021-03-15T22:35:38.344595+0000","last_peered":"2021-03-15T22:35:38.344595+0000","last_clean":"2021-03-15T22:35:38.344595+0000","last_became_active":"2021-03-15T22:34:55.353316+0000","last_became_peered":"2021-03-15T22:34:55.353316+0000","last_unstale":"2021-03-15T22:35:38.344595+0000","last_undegraded":"2021-03-15T22:35:38.344595+0000","last_fullsized":"2021-03-15T22:35:38.344595+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:38.907011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.907011+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"985'3194","reported_seq":"6710","reported_epoch":"985","state":"active+clean","last_fresh":"2021-03-15T22:35:40.373047+0000","last_change":"2021-03-15T22:35:39.376532+0000","last_active":"2021-03-15T22:35:40.373047+0000","last_peered":"2021-03-15T22:35:40.373047+0000","last_clean":"2021-03-15T22:35:40.373047+0000","last_became_active":"2021-03-15T22:34:13.331560+0000","last_became_peered":"2021-03-15T22:34:13.331560+0000","last_unstale":"2021-03-15T22:35:40.373047+0000","last_undegraded":"2021-03-15T22:35:40.373047+0000","last_fullsized":"2021-03-15T22:35:40.373047+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"711'2007","last_scrub_stamp":"2021-03-15T22:31:02.050395+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:31:02.050395+0000","log_size":3194,"ondisk_log_size":3194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":81793833,"num_objects":38,"num_object_clones":31,"num_object_copies":114,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":38,"num_whiteouts":0,"num_read":873,"num_read_kb":897099,"num_write":676,"num_write_kb":272343,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":264088494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c3","length":"1"},{"start":"2f3","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"367","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344594+0000","last_change":"2021-03-15T22:34:34.635632+0000","last_active":"2021-03-15T22:35:37.344594+0000","last_peered":"2021-03-15T22:35:37.344594+0000","last_clean":"2021-03-15T22:35:37.344594+0000","last_became_active":"2021-03-15T22:34:34.635223+0000","last_became_peered":"2021-03-15T22:34:34.635223+0000","last_unstale":"2021-03-15T22:35:37.344594+0000","last_undegraded":"2021-03-15T22:35:37.344594+0000","last_fullsized":"2021-03-15T22:35:37.344594+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.761137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"997","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.340780+0000","last_change":"2021-03-15T22:35:19.456953+0000","last_active":"2021-03-15T22:35:37.340780+0000","last_peered":"2021-03-15T22:35:37.340780+0000","last_clean":"2021-03-15T22:35:37.340780+0000","last_became_active":"2021-03-15T22:33:04.337709+0000","last_became_peered":"2021-03-15T22:33:04.337709+0000","last_unstale":"2021-03-15T22:35:37.340780+0000","last_undegraded":"2021-03-15T22:35:37.340780+0000","last_fullsized":"2021-03-15T22:35:37.340780+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"63","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.353767+0000","last_change":"2021-03-15T22:34:43.341562+0000","last_active":"2021-03-15T22:35:35.353767+0000","last_peered":"2021-03-15T22:35:35.353767+0000","last_clean":"2021-03-15T22:35:35.353767+0000","last_became_active":"2021-03-15T22:34:43.340955+0000","last_became_peered":"2021-03-15T22:34:43.340955+0000","last_unstale":"2021-03-15T22:35:35.353767+0000","last_undegraded":"2021-03-15T22:35:35.353767+0000","last_fullsized":"2021-03-15T22:35:35.353767+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.843478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.843478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"184","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.346889+0000","last_change":"2021-03-15T22:34:09.215097+0000","last_active":"2021-03-15T22:35:39.346889+0000","last_peered":"2021-03-15T22:35:39.346889+0000","last_clean":"2021-03-15T22:35:39.346889+0000","last_became_active":"2021-03-15T22:33:04.330531+0000","last_became_peered":"2021-03-15T22:33:04.330531+0000","last_unstale":"2021-03-15T22:35:39.346889+0000","last_undegraded":"2021-03-15T22:35:39.346889+0000","last_fullsized":"2021-03-15T22:35:39.346889+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:34:09.215034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:09.215034+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"1018","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341411+0000","last_change":"2021-03-15T22:35:22.409047+0000","last_active":"2021-03-15T22:35:37.341411+0000","last_peered":"2021-03-15T22:35:37.341411+0000","last_clean":"2021-03-15T22:35:37.341411+0000","last_became_active":"2021-03-15T22:34:55.352186+0000","last_became_peered":"2021-03-15T22:34:55.352186+0000","last_unstale":"2021-03-15T22:35:37.341411+0000","last_undegraded":"2021-03-15T22:35:37.341411+0000","last_fullsized":"2021-03-15T22:35:37.341411+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.408965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.408965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"980'974","reported_seq":"4044","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341144+0000","last_change":"2021-03-15T22:35:35.358163+0000","last_active":"2021-03-15T22:35:37.341144+0000","last_peered":"2021-03-15T22:35:37.341144+0000","last_clean":"2021-03-15T22:35:37.341144+0000","last_became_active":"2021-03-15T22:35:03.346751+0000","last_became_peered":"2021-03-15T22:35:03.346751+0000","last_unstale":"2021-03-15T22:35:37.341144+0000","last_undegraded":"2021-03-15T22:35:37.341144+0000","last_fullsized":"2021-03-15T22:35:37.341144+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":0,"last_scrub":"891'819","last_scrub_stamp":"2021-03-15T22:34:08.508824+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.508824+0000","log_size":974,"ondisk_log_size":974,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31297829,"num_objects":16,"num_object_clones":14,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":256,"num_read_kb":260779,"num_write":216,"num_write_kb":96405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":109648904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2be","length":"1"},{"start":"2c3","length":"1"},{"start":"2d1","length":"1"},{"start":"30d","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"784","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344729+0000","last_change":"2021-03-15T22:34:43.344711+0000","last_active":"2021-03-15T22:35:37.344729+0000","last_peered":"2021-03-15T22:35:37.344729+0000","last_clean":"2021-03-15T22:35:37.344729+0000","last_became_active":"2021-03-15T22:34:43.344418+0000","last_became_peered":"2021-03-15T22:34:43.344418+0000","last_unstale":"2021-03-15T22:35:37.344729+0000","last_undegraded":"2021-03-15T22:35:37.344729+0000","last_fullsized":"2021-03-15T22:35:37.344729+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"481","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350619+0000","last_change":"2021-03-15T22:34:35.410541+0000","last_active":"2021-03-15T22:35:39.350619+0000","last_peered":"2021-03-15T22:35:39.350619+0000","last_clean":"2021-03-15T22:35:39.350619+0000","last_became_active":"2021-03-15T22:34:10.321100+0000","last_became_peered":"2021-03-15T22:34:10.321100+0000","last_unstale":"2021-03-15T22:35:39.350619+0000","last_undegraded":"2021-03-15T22:35:39.350619+0000","last_fullsized":"2021-03-15T22:35:39.350619+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:35.410485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:35.410485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"2041","reported_epoch":"983","state":"active+clean","last_fresh":"2021-03-15T22:35:38.345270+0000","last_change":"2021-03-15T22:35:10.452906+0000","last_active":"2021-03-15T22:35:38.345270+0000","last_peered":"2021-03-15T22:35:38.345270+0000","last_clean":"2021-03-15T22:35:38.345270+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:35:38.345270+0000","last_undegraded":"2021-03-15T22:35:38.345270+0000","last_fullsized":"2021-03-15T22:35:38.345270+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:35:10.452809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.452809+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1049,"num_read_kb":1044,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"579","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.353758+0000","last_change":"2021-03-15T22:34:55.414460+0000","last_active":"2021-03-15T22:35:39.353758+0000","last_peered":"2021-03-15T22:35:39.353758+0000","last_clean":"2021-03-15T22:35:39.353758+0000","last_became_active":"2021-03-15T22:34:55.359023+0000","last_became_peered":"2021-03-15T22:34:55.359023+0000","last_unstale":"2021-03-15T22:35:39.353758+0000","last_undegraded":"2021-03-15T22:35:39.353758+0000","last_fullsized":"2021-03-15T22:35:39.353758+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:34:40.472800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.472800+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"372","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344691+0000","last_change":"2021-03-15T22:34:40.799217+0000","last_active":"2021-03-15T22:35:37.344691+0000","last_peered":"2021-03-15T22:35:37.344691+0000","last_clean":"2021-03-15T22:35:37.344691+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:35:37.344691+0000","last_undegraded":"2021-03-15T22:35:37.344691+0000","last_fullsized":"2021-03-15T22:35:37.344691+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.799153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.799153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"982'3297","reported_seq":"7346","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.349902+0000","last_change":"2021-03-15T22:35:35.375943+0000","last_active":"2021-03-15T22:35:37.349902+0000","last_peered":"2021-03-15T22:35:37.349902+0000","last_clean":"2021-03-15T22:35:37.349902+0000","last_became_active":"2021-03-15T22:35:17.487261+0000","last_became_peered":"2021-03-15T22:35:17.487261+0000","last_unstale":"2021-03-15T22:35:37.349902+0000","last_undegraded":"2021-03-15T22:35:37.349902+0000","last_fullsized":"2021-03-15T22:35:37.349902+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'2597","last_scrub_stamp":"2021-03-15T22:32:30.909661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.909661+0000","log_size":3297,"ondisk_log_size":3297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":98794433,"num_objects":42,"num_object_clones":35,"num_object_copies":126,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":42,"num_whiteouts":2,"num_read":1184,"num_read_kb":1259814,"num_write":764,"num_write_kb":319313,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":721,"num_bytes_recovered":351021959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2be","length":"1"},{"start":"2c3","length":"1"},{"start":"2d1","length":"1"},{"start":"30d","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"1013","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.346959+0000","last_change":"2021-03-15T22:34:55.346293+0000","last_active":"2021-03-15T22:35:36.346959+0000","last_peered":"2021-03-15T22:35:36.346959+0000","last_clean":"2021-03-15T22:35:36.346959+0000","last_became_active":"2021-03-15T22:34:55.345948+0000","last_became_peered":"2021-03-15T22:34:55.345948+0000","last_unstale":"2021-03-15T22:35:36.346959+0000","last_undegraded":"2021-03-15T22:35:36.346959+0000","last_fullsized":"2021-03-15T22:35:36.346959+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.701955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.701955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"1031","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.347263+0000","last_change":"2021-03-15T22:34:55.346381+0000","last_active":"2021-03-15T22:35:36.347263+0000","last_peered":"2021-03-15T22:35:36.347263+0000","last_clean":"2021-03-15T22:35:36.347263+0000","last_became_active":"2021-03-15T22:34:55.346048+0000","last_became_peered":"2021-03-15T22:34:55.346048+0000","last_unstale":"2021-03-15T22:35:36.347263+0000","last_undegraded":"2021-03-15T22:35:36.347263+0000","last_fullsized":"2021-03-15T22:35:36.347263+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:48.037748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_clean_scrub_stamp":"2021-03-15T22:34:48.037748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"529","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350891+0000","last_change":"2021-03-15T22:35:21.314851+0000","last_active":"2021-03-15T22:35:39.350891+0000","last_peered":"2021-03-15T22:35:39.350891+0000","last_clean":"2021-03-15T22:35:39.350891+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:35:39.350891+0000","last_undegraded":"2021-03-15T22:35:39.350891+0000","last_fullsized":"2021-03-15T22:35:39.350891+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:35:21.314763+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.314763+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"169","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.708029+0000","last_change":"2021-03-15T22:35:39.707993+0000","last_active":"2021-03-15T22:35:39.708029+0000","last_peered":"2021-03-15T22:35:39.708029+0000","last_clean":"2021-03-15T22:35:39.708029+0000","last_became_active":"2021-03-15T22:33:04.337244+0000","last_became_peered":"2021-03-15T22:33:04.337244+0000","last_unstale":"2021-03-15T22:35:39.708029+0000","last_undegraded":"2021-03-15T22:35:39.708029+0000","last_fullsized":"2021-03-15T22:35:39.708029+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:39.707900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:35:39.707900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"980'570","reported_seq":"3643","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.356629+0000","last_change":"2021-03-15T22:35:35.356629+0000","last_active":"2021-03-15T22:35:35.356629+0000","last_peered":"2021-03-15T22:35:35.356629+0000","last_clean":"2021-03-15T22:35:35.356629+0000","last_became_active":"2021-03-15T22:34:15.498612+0000","last_became_peered":"2021-03-15T22:34:15.498612+0000","last_unstale":"2021-03-15T22:35:35.356629+0000","last_undegraded":"2021-03-15T22:35:35.356629+0000","last_fullsized":"2021-03-15T22:35:35.356629+0000","mapping_epoch":899,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":900,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'463","last_scrub_stamp":"2021-03-15T22:33:40.563144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.563144+0000","log_size":570,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17987521,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":173,"num_read_kb":209120,"num_write":109,"num_write_kb":50920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":84391207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2be","length":"1"},{"start":"2c3","length":"1"},{"start":"2d1","length":"1"},{"start":"30d","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"505","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350513+0000","last_change":"2021-03-15T22:34:55.356457+0000","last_active":"2021-03-15T22:35:39.350513+0000","last_peered":"2021-03-15T22:35:39.350513+0000","last_clean":"2021-03-15T22:35:39.350513+0000","last_became_active":"2021-03-15T22:34:55.356086+0000","last_became_peered":"2021-03-15T22:34:55.356086+0000","last_unstale":"2021-03-15T22:35:39.350513+0000","last_undegraded":"2021-03-15T22:35:39.350513+0000","last_fullsized":"2021-03-15T22:35:39.350513+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"955","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344711+0000","last_change":"2021-03-15T22:34:55.356446+0000","last_active":"2021-03-15T22:35:37.344711+0000","last_peered":"2021-03-15T22:35:37.344711+0000","last_clean":"2021-03-15T22:35:37.344711+0000","last_became_active":"2021-03-15T22:34:55.355757+0000","last_became_peered":"2021-03-15T22:34:55.355757+0000","last_unstale":"2021-03-15T22:35:37.344711+0000","last_undegraded":"2021-03-15T22:35:37.344711+0000","last_fullsized":"2021-03-15T22:35:37.344711+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.883949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.883949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"936","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344181+0000","last_change":"2021-03-15T22:34:11.330201+0000","last_active":"2021-03-15T22:35:37.344181+0000","last_peered":"2021-03-15T22:35:37.344181+0000","last_clean":"2021-03-15T22:35:37.344181+0000","last_became_active":"2021-03-15T22:34:10.321886+0000","last_became_peered":"2021-03-15T22:34:10.321886+0000","last_unstale":"2021-03-15T22:35:37.344181+0000","last_undegraded":"2021-03-15T22:35:37.344181+0000","last_fullsized":"2021-03-15T22:35:37.344181+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:32:31.747582+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.747582+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"836'11","reported_seq":"598","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.350737+0000","last_change":"2021-03-15T22:35:10.342791+0000","last_active":"2021-03-15T22:35:39.350737+0000","last_peered":"2021-03-15T22:35:39.350737+0000","last_clean":"2021-03-15T22:35:39.350737+0000","last_became_active":"2021-03-15T22:33:04.336294+0000","last_became_peered":"2021-03-15T22:33:04.336294+0000","last_unstale":"2021-03-15T22:35:39.350737+0000","last_undegraded":"2021-03-15T22:35:39.350737+0000","last_fullsized":"2021-03-15T22:35:39.350737+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"721'9","last_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:31:10.971303+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"980'923","reported_seq":"4032","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.365074+0000","last_change":"2021-03-15T22:35:35.365074+0000","last_active":"2021-03-15T22:35:35.365074+0000","last_peered":"2021-03-15T22:35:35.365074+0000","last_clean":"2021-03-15T22:35:35.365074+0000","last_became_active":"2021-03-15T22:34:58.350873+0000","last_became_peered":"2021-03-15T22:34:58.350873+0000","last_unstale":"2021-03-15T22:35:35.365074+0000","last_undegraded":"2021-03-15T22:35:35.365074+0000","last_fullsized":"2021-03-15T22:35:35.365074+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'901","last_scrub_stamp":"2021-03-15T22:35:16.957336+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.957336+0000","log_size":923,"ondisk_log_size":923,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":31342678,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":255,"num_read_kb":220513,"num_write":225,"num_write_kb":86799,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":114552572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2be","length":"1"},{"start":"2c3","length":"1"},{"start":"2d1","length":"1"},{"start":"30d","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"1005","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341241+0000","last_change":"2021-03-15T22:34:55.351577+0000","last_active":"2021-03-15T22:35:37.341241+0000","last_peered":"2021-03-15T22:35:37.341241+0000","last_clean":"2021-03-15T22:35:37.341241+0000","last_became_active":"2021-03-15T22:34:55.351249+0000","last_became_peered":"2021-03-15T22:34:55.351249+0000","last_unstale":"2021-03-15T22:35:37.341241+0000","last_undegraded":"2021-03-15T22:35:37.341241+0000","last_fullsized":"2021-03-15T22:35:37.341241+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.507204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.507204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"449","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.353827+0000","last_change":"2021-03-15T22:33:22.598240+0000","last_active":"2021-03-15T22:35:39.353827+0000","last_peered":"2021-03-15T22:35:39.353827+0000","last_clean":"2021-03-15T22:35:39.353827+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:35:39.353827+0000","last_undegraded":"2021-03-15T22:35:39.353827+0000","last_fullsized":"2021-03-15T22:35:39.353827+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.598174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.598174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"1173","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.343444+0000","last_change":"2021-03-15T22:33:43.514160+0000","last_active":"2021-03-15T22:35:35.343444+0000","last_peered":"2021-03-15T22:35:35.343444+0000","last_clean":"2021-03-15T22:35:35.343444+0000","last_became_active":"2021-03-15T22:33:04.336145+0000","last_became_peered":"2021-03-15T22:33:04.336145+0000","last_unstale":"2021-03-15T22:35:35.343444+0000","last_undegraded":"2021-03-15T22:35:35.343444+0000","last_fullsized":"2021-03-15T22:35:35.343444+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:33:43.514088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.514088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"492","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.353162+0000","last_change":"2021-03-15T22:33:20.685041+0000","last_active":"2021-03-15T22:35:35.353162+0000","last_peered":"2021-03-15T22:35:35.353162+0000","last_clean":"2021-03-15T22:35:35.353162+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:35:35.353162+0000","last_undegraded":"2021-03-15T22:35:35.353162+0000","last_fullsized":"2021-03-15T22:35:35.353162+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:33:20.684976+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.684976+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"980'1649","reported_seq":"4590","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.348600+0000","last_change":"2021-03-15T22:35:35.361529+0000","last_active":"2021-03-15T22:35:36.348600+0000","last_peered":"2021-03-15T22:35:36.348600+0000","last_clean":"2021-03-15T22:35:36.348600+0000","last_became_active":"2021-03-15T22:34:58.337099+0000","last_became_peered":"2021-03-15T22:34:58.337099+0000","last_unstale":"2021-03-15T22:35:36.348600+0000","last_undegraded":"2021-03-15T22:35:36.348600+0000","last_fullsized":"2021-03-15T22:35:36.348600+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'1611","last_scrub_stamp":"2021-03-15T22:35:21.054192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.054192+0000","log_size":1649,"ondisk_log_size":1649,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40111289,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":539,"num_read_kb":566333,"num_write":345,"num_write_kb":120323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":226741522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2be","length":"1"},{"start":"2c3","length":"1"},{"start":"2d1","length":"1"},{"start":"30d","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"1058","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.342175+0000","last_change":"2021-03-15T22:34:55.343142+0000","last_active":"2021-03-15T22:35:35.342175+0000","last_peered":"2021-03-15T22:35:35.342175+0000","last_clean":"2021-03-15T22:35:35.342175+0000","last_became_active":"2021-03-15T22:34:55.341896+0000","last_became_peered":"2021-03-15T22:34:55.341896+0000","last_unstale":"2021-03-15T22:35:35.342175+0000","last_undegraded":"2021-03-15T22:35:35.342175+0000","last_fullsized":"2021-03-15T22:35:35.342175+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"764","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.353813+0000","last_change":"2021-03-15T22:34:41.870475+0000","last_active":"2021-03-15T22:35:35.353813+0000","last_peered":"2021-03-15T22:35:35.353813+0000","last_clean":"2021-03-15T22:35:35.353813+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:35:35.353813+0000","last_undegraded":"2021-03-15T22:35:35.353813+0000","last_fullsized":"2021-03-15T22:35:35.353813+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.870368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.870368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"417","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.341551+0000","last_change":"2021-03-15T22:35:08.402702+0000","last_active":"2021-03-15T22:35:37.341551+0000","last_peered":"2021-03-15T22:35:37.341551+0000","last_clean":"2021-03-15T22:35:37.341551+0000","last_became_active":"2021-03-15T22:34:10.349731+0000","last_became_peered":"2021-03-15T22:34:10.349731+0000","last_unstale":"2021-03-15T22:35:37.341551+0000","last_undegraded":"2021-03-15T22:35:37.341551+0000","last_fullsized":"2021-03-15T22:35:37.341551+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:35:08.402641+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:35:08.402641+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1026","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.342444+0000","last_change":"2021-03-15T22:34:43.343527+0000","last_active":"2021-03-15T22:35:35.342444+0000","last_peered":"2021-03-15T22:35:35.342444+0000","last_clean":"2021-03-15T22:35:35.342444+0000","last_became_active":"2021-03-15T22:34:43.343255+0000","last_became_peered":"2021-03-15T22:34:43.343255+0000","last_unstale":"2021-03-15T22:35:35.342444+0000","last_undegraded":"2021-03-15T22:35:35.342444+0000","last_fullsized":"2021-03-15T22:35:35.342444+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.673338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.673338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"984'1403","reported_seq":"4597","reported_epoch":"985","state":"active+clean","last_fresh":"2021-03-15T22:35:40.372044+0000","last_change":"2021-03-15T22:35:39.361528+0000","last_active":"2021-03-15T22:35:40.372044+0000","last_peered":"2021-03-15T22:35:40.372044+0000","last_clean":"2021-03-15T22:35:40.372044+0000","last_became_active":"2021-03-15T22:34:58.344406+0000","last_became_peered":"2021-03-15T22:34:58.344406+0000","last_unstale":"2021-03-15T22:35:40.372044+0000","last_undegraded":"2021-03-15T22:35:40.372044+0000","last_fullsized":"2021-03-15T22:35:40.372044+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"872'1183","last_scrub_stamp":"2021-03-15T22:33:43.209215+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.209215+0000","log_size":1403,"ondisk_log_size":1403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":38493192,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":381,"num_read_kb":340482,"num_write":305,"num_write_kb":113592,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":127958264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c3","length":"1"},{"start":"2f3","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"927","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344651+0000","last_change":"2021-03-15T22:34:55.356480+0000","last_active":"2021-03-15T22:35:37.344651+0000","last_peered":"2021-03-15T22:35:37.344651+0000","last_clean":"2021-03-15T22:35:37.344651+0000","last_became_active":"2021-03-15T22:34:55.356213+0000","last_became_peered":"2021-03-15T22:34:55.356213+0000","last_unstale":"2021-03-15T22:35:37.344651+0000","last_undegraded":"2021-03-15T22:35:37.344651+0000","last_fullsized":"2021-03-15T22:35:37.344651+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.729838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.729838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1277","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.346715+0000","last_change":"2021-03-15T22:35:19.468482+0000","last_active":"2021-03-15T22:35:35.346715+0000","last_peered":"2021-03-15T22:35:35.346715+0000","last_clean":"2021-03-15T22:35:35.346715+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:35:35.346715+0000","last_undegraded":"2021-03-15T22:35:35.346715+0000","last_fullsized":"2021-03-15T22:35:35.346715+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:35:19.468402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.468402+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"388","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.347065+0000","last_change":"2021-03-15T22:34:10.332407+0000","last_active":"2021-03-15T22:35:36.347065+0000","last_peered":"2021-03-15T22:35:36.347065+0000","last_clean":"2021-03-15T22:35:36.347065+0000","last_became_active":"2021-03-15T22:34:10.332089+0000","last_became_peered":"2021-03-15T22:34:10.332089+0000","last_unstale":"2021-03-15T22:35:36.347065+0000","last_undegraded":"2021-03-15T22:35:36.347065+0000","last_fullsized":"2021-03-15T22:35:36.347065+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.791527+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.791527+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"143","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.347553+0000","last_change":"2021-03-15T22:34:44.050423+0000","last_active":"2021-03-15T22:35:36.347553+0000","last_peered":"2021-03-15T22:35:36.347553+0000","last_clean":"2021-03-15T22:35:36.347553+0000","last_became_active":"2021-03-15T22:33:45.317888+0000","last_became_peered":"2021-03-15T22:33:45.317888+0000","last_unstale":"2021-03-15T22:35:36.347553+0000","last_undegraded":"2021-03-15T22:35:36.347553+0000","last_fullsized":"2021-03-15T22:35:36.347553+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:34:44.050366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:44.050366+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"984'468","reported_seq":"3260","reported_epoch":"985","state":"active+clean","last_fresh":"2021-03-15T22:35:40.370453+0000","last_change":"2021-03-15T22:35:39.352231+0000","last_active":"2021-03-15T22:35:40.370453+0000","last_peered":"2021-03-15T22:35:40.370453+0000","last_clean":"2021-03-15T22:35:40.370453+0000","last_became_active":"2021-03-15T22:33:08.354872+0000","last_became_peered":"2021-03-15T22:33:08.354872+0000","last_unstale":"2021-03-15T22:35:40.370453+0000","last_undegraded":"2021-03-15T22:35:40.370453+0000","last_fullsized":"2021-03-15T22:35:40.370453+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'373","last_scrub_stamp":"2021-03-15T22:33:44.219517+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.219517+0000","log_size":468,"ondisk_log_size":468,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11894009,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":185,"num_read_kb":197817,"num_write":93,"num_write_kb":35609,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":31290769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2c3","length":"1"},{"start":"2f3","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"63","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.352563+0000","last_change":"2021-03-15T22:34:43.340972+0000","last_active":"2021-03-15T22:35:35.352563+0000","last_peered":"2021-03-15T22:35:35.352563+0000","last_clean":"2021-03-15T22:35:35.352563+0000","last_became_active":"2021-03-15T22:34:43.340600+0000","last_became_peered":"2021-03-15T22:34:43.340600+0000","last_unstale":"2021-03-15T22:35:35.352563+0000","last_undegraded":"2021-03-15T22:35:35.352563+0000","last_fullsized":"2021-03-15T22:35:35.352563+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:09.612126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.612126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"1025","reported_epoch":"982","state":"active+clean","last_fresh":"2021-03-15T22:35:37.344783+0000","last_change":"2021-03-15T22:35:21.917891+0000","last_active":"2021-03-15T22:35:37.344783+0000","last_peered":"2021-03-15T22:35:37.344783+0000","last_clean":"2021-03-15T22:35:37.344783+0000","last_became_active":"2021-03-15T22:33:08.352367+0000","last_became_peered":"2021-03-15T22:33:08.352367+0000","last_unstale":"2021-03-15T22:35:37.344783+0000","last_undegraded":"2021-03-15T22:35:37.344783+0000","last_fullsized":"2021-03-15T22:35:37.344783+0000","mapping_epoch":837,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'9","last_scrub_stamp":"2021-03-15T22:35:21.917819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.917819+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"990","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.342549+0000","last_change":"2021-03-15T22:34:55.347300+0000","last_active":"2021-03-15T22:35:35.342549+0000","last_peered":"2021-03-15T22:35:35.342549+0000","last_clean":"2021-03-15T22:35:35.342549+0000","last_became_active":"2021-03-15T22:34:55.346934+0000","last_became_peered":"2021-03-15T22:34:55.346934+0000","last_unstale":"2021-03-15T22:35:35.342549+0000","last_undegraded":"2021-03-15T22:35:35.342549+0000","last_fullsized":"2021-03-15T22:35:35.342549+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.461102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"1152","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.346632+0000","last_change":"2021-03-15T22:35:21.384129+0000","last_active":"2021-03-15T22:35:35.346632+0000","last_peered":"2021-03-15T22:35:35.346632+0000","last_clean":"2021-03-15T22:35:35.346632+0000","last_became_active":"2021-03-15T22:34:10.330608+0000","last_became_peered":"2021-03-15T22:34:10.330608+0000","last_unstale":"2021-03-15T22:35:35.346632+0000","last_undegraded":"2021-03-15T22:35:35.346632+0000","last_fullsized":"2021-03-15T22:35:35.346632+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:35:21.384037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.384037+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"984'484","reported_seq":"2778","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.349918+0000","last_change":"2021-03-15T22:35:39.350254+0000","last_active":"2021-03-15T22:35:39.349918+0000","last_peered":"2021-03-15T22:35:39.349918+0000","last_clean":"2021-03-15T22:35:39.349918+0000","last_became_active":"2021-03-15T22:34:58.350923+0000","last_became_peered":"2021-03-15T22:34:58.350923+0000","last_unstale":"2021-03-15T22:35:39.349918+0000","last_undegraded":"2021-03-15T22:35:39.349918+0000","last_fullsized":"2021-03-15T22:35:39.349918+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"957'470","last_scrub_stamp":"2021-03-15T22:35:18.338289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.338289+0000","log_size":484,"ondisk_log_size":484,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":5906715,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":155,"num_read_kb":164362,"num_write":109,"num_write_kb":33883,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":52707265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2be","length":"1"},{"start":"2c3","length":"1"},{"start":"2d1","length":"1"},{"start":"2f3","length":"1"},{"start":"30d","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"725","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.353509+0000","last_change":"2021-03-15T22:34:55.345645+0000","last_active":"2021-03-15T22:35:35.353509+0000","last_peered":"2021-03-15T22:35:35.353509+0000","last_clean":"2021-03-15T22:35:35.353509+0000","last_became_active":"2021-03-15T22:34:55.345267+0000","last_became_peered":"2021-03-15T22:34:55.345267+0000","last_unstale":"2021-03-15T22:35:35.353509+0000","last_undegraded":"2021-03-15T22:35:35.353509+0000","last_fullsized":"2021-03-15T22:35:35.353509+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.958050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.958050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"1018","reported_epoch":"981","state":"active+clean","last_fresh":"2021-03-15T22:35:36.347577+0000","last_change":"2021-03-15T22:35:12.091714+0000","last_active":"2021-03-15T22:35:36.347577+0000","last_peered":"2021-03-15T22:35:36.347577+0000","last_clean":"2021-03-15T22:35:36.347577+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:35:36.347577+0000","last_undegraded":"2021-03-15T22:35:36.347577+0000","last_fullsized":"2021-03-15T22:35:36.347577+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.091650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.091650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"178","reported_epoch":"984","state":"active+clean","last_fresh":"2021-03-15T22:35:39.346092+0000","last_change":"2021-03-15T22:35:20.678309+0000","last_active":"2021-03-15T22:35:39.346092+0000","last_peered":"2021-03-15T22:35:39.346092+0000","last_clean":"2021-03-15T22:35:39.346092+0000","last_became_active":"2021-03-15T22:34:10.332236+0000","last_became_peered":"2021-03-15T22:34:10.332236+0000","last_unstale":"2021-03-15T22:35:39.346092+0000","last_undegraded":"2021-03-15T22:35:39.346092+0000","last_fullsized":"2021-03-15T22:35:39.346092+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.678246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"1126","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.343420+0000","last_change":"2021-03-15T22:34:55.371223+0000","last_active":"2021-03-15T22:35:35.343420+0000","last_peered":"2021-03-15T22:35:35.343420+0000","last_clean":"2021-03-15T22:35:35.343420+0000","last_became_active":"2021-03-15T22:34:55.341958+0000","last_became_peered":"2021-03-15T22:34:55.341958+0000","last_unstale":"2021-03-15T22:35:35.343420+0000","last_undegraded":"2021-03-15T22:35:35.343420+0000","last_fullsized":"2021-03-15T22:35:35.343420+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:32:43.307660+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.307660+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"984'1631","reported_seq":"4681","reported_epoch":"985","state":"active+clean","last_fresh":"2021-03-15T22:35:40.345552+0000","last_change":"2021-03-15T22:35:39.364932+0000","last_active":"2021-03-15T22:35:40.345552+0000","last_peered":"2021-03-15T22:35:40.345552+0000","last_clean":"2021-03-15T22:35:40.345552+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:35:40.345552+0000","last_undegraded":"2021-03-15T22:35:40.345552+0000","last_fullsized":"2021-03-15T22:35:40.345552+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"837'1315","last_scrub_stamp":"2021-03-15T22:33:08.608669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.608669+0000","log_size":1631,"ondisk_log_size":1631,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52734497,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":539,"num_read_kb":575792,"num_write":368,"num_write_kb":132533,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2c3","length":"1"},{"start":"2f3","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"991","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.343015+0000","last_change":"2021-03-15T22:34:55.340853+0000","last_active":"2021-03-15T22:35:35.343015+0000","last_peered":"2021-03-15T22:35:35.343015+0000","last_clean":"2021-03-15T22:35:35.343015+0000","last_became_active":"2021-03-15T22:34:55.340498+0000","last_became_peered":"2021-03-15T22:34:55.340498+0000","last_unstale":"2021-03-15T22:35:35.343015+0000","last_undegraded":"2021-03-15T22:35:35.343015+0000","last_fullsized":"2021-03-15T22:35:35.343015+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:20.500979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.500979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"976'19","reported_seq":"380","reported_epoch":"983","state":"active+clean","last_fresh":"2021-03-15T22:35:38.344018+0000","last_change":"2021-03-15T22:35:28.365207+0000","last_active":"2021-03-15T22:35:38.344018+0000","last_peered":"2021-03-15T22:35:38.344018+0000","last_clean":"2021-03-15T22:35:38.344018+0000","last_became_active":"2021-03-15T22:35:28.364764+0000","last_became_peered":"2021-03-15T22:35:28.364764+0000","last_unstale":"2021-03-15T22:35:38.344018+0000","last_undegraded":"2021-03-15T22:35:38.344018+0000","last_fullsized":"2021-03-15T22:35:38.344018+0000","mapping_epoch":972,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":973,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'13","last_scrub_stamp":"2021-03-15T22:33:37.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.949876+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"409","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.350696+0000","last_change":"2021-03-15T22:33:21.690369+0000","last_active":"2021-03-15T22:35:35.350696+0000","last_peered":"2021-03-15T22:35:35.350696+0000","last_clean":"2021-03-15T22:35:35.350696+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:35:35.350696+0000","last_undegraded":"2021-03-15T22:35:35.350696+0000","last_fullsized":"2021-03-15T22:35:35.350696+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.690309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"1165","reported_epoch":"980","state":"active+clean","last_fresh":"2021-03-15T22:35:35.344539+0000","last_change":"2021-03-15T22:35:15.476444+0000","last_active":"2021-03-15T22:35:35.344539+0000","last_peered":"2021-03-15T22:35:35.344539+0000","last_clean":"2021-03-15T22:35:35.344539+0000","last_became_active":"2021-03-15T22:33:04.337113+0000","last_became_peered":"2021-03-15T22:33:04.337113+0000","last_unstale":"2021-03-15T22:35:35.344539+0000","last_undegraded":"2021-03-15T22:35:35.344539+0000","last_fullsized":"2021-03-15T22:35:35.344539+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:32:32.328526+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.328526+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":699661217,"num_objects":318,"num_object_clones":268,"num_object_copies":954,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":318,"num_whiteouts":13,"num_read":6982,"num_read_kb":7157560,"num_write":5123,"num_write_kb":1967920,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4621,"num_bytes_recovered":2230024638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1556533248,"data_stored":2197614524,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22539,"ondisk_log_size":22539,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1049,"num_read_kb":1044,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":3519,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1482,"num_read_kb":987,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":94766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":947,"ondisk_log_size":947,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":42,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":126,"acting":126,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299609,"num_pgs":68,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":255148,"kb_used_data":202724,"kb_used_omap":0,"kb_used_meta":52416,"kb_avail":93490004,"statfs":{"total":95995035648,"available":95733764096,"internally_reserved":0,"allocated":207589376,"data_stored":321534726,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53673984},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45000000000000001,"15min":0.46500000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57399999999999995,"5min":0.70099999999999996,"15min":2.73},"last":2.0390000000000001},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.222},"max":{"1min":0.61199999999999999,"5min":0.75800000000000001,"15min":3.8799999999999999},"last":1.9950000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.443,"15min":0.47199999999999998},"min":{"1min":0.254,"5min":0.249,"15min":0.249},"max":{"1min":0.57299999999999995,"5min":0.73899999999999999,"15min":2.6859999999999999},"last":0.56200000000000006},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.56000000000000005,"15min":0.51400000000000001},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.23400000000000001},"max":{"1min":5.3819999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":0.46600000000000003}]},{"osd":2,"last update":"Mon Mar 15 22:35:38 2021","interfaces":[{"interface":"back","average":{"1min":2.1760000000000002,"5min":0.83999999999999997,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":29.350999999999999,"5min":29.350999999999999,"15min":29.350999999999999},"last":0.35599999999999998},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.498,"15min":0.5},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":1.298}]},{"osd":3,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.498,"15min":0.495},"min":{"1min":0.314,"5min":0.314,"15min":0.28899999999999998},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.702},"last":1.4810000000000001},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.497,"15min":0.52500000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":5.415},"last":1.806}]},{"osd":4,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.437,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.27900000000000003,"15min":0.23799999999999999},"max":{"1min":0.58499999999999996,"5min":0.753,"15min":1.927},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.46300000000000002,"15min":0.46200000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":0.57599999999999996,"5min":0.72699999999999998,"15min":2.5339999999999998},"last":0.33700000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:35:38 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47899999999999998,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68300000000000005,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.38400000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.498,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67800000000000005,"5min":0.77700000000000002,"15min":1.071},"last":0.52300000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.29999999999999999,"15min":0.27100000000000002},"max":{"1min":0.65300000000000002,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.28399999999999997},"max":{"1min":0.73799999999999999,"5min":0.81999999999999995,"15min":1.8560000000000001},"last":0.48899999999999999}]}]},{"osd":6,"up_from":45,"seq":193273528541,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":209480,"kb_used_data":167808,"kb_used_omap":0,"kb_used_meta":41664,"kb_avail":93535672,"statfs":{"total":95995035648,"available":95780528128,"internally_reserved":0,"allocated":171835392,"data_stored":235279150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":42663936},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.49199999999999999,"15min":0.48399999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.79600000000000004,"5min":2.5470000000000002,"15min":3.085},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.47999999999999998,"15min":0.46500000000000002},"min":{"1min":0.27600000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73099999999999998,"5min":1.137,"15min":1.323},"last":0.44}]},{"osd":1,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.254},"max":{"1min":1.1120000000000001,"5min":2.7839999999999998,"15min":14.637},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.45200000000000001,"15min":0.47499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60299999999999998,"5min":0.84899999999999998,"15min":5.2450000000000001},"last":0.46200000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:34:40 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.47299999999999998,"15min":0.46500000000000002},"min":{"1min":0.33600000000000002,"5min":0.312,"15min":0.312},"max":{"1min":0.85899999999999999,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.495,"5min":0.53500000000000003,"15min":0.51200000000000001},"min":{"1min":0.36199999999999999,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.63300000000000001,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.48399999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51100000000000001,"15min":0.50900000000000001},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.29899999999999999},"max":{"1min":0.874,"5min":1.0609999999999999,"15min":2.6150000000000002},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.502,"15min":0.502},"min":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.63900000000000001,"5min":2.4670000000000001,"15min":2.7890000000000001},"last":0.42799999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.44600000000000001,"15min":0.438},"min":{"1min":0.25,"5min":0.25,"15min":0.224},"max":{"1min":0.61799999999999999,"5min":0.94099999999999995,"15min":4.306},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.47799999999999998,"15min":0.49199999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23499999999999999},"max":{"1min":0.82799999999999996,"5min":1.091,"15min":4.181},"last":0.34200000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:35:39 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.47599999999999998,"15min":0.42599999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.26700000000000002},"max":{"1min":0.63300000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.53000000000000003,"15min":0.46999999999999997},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67900000000000005,"5min":1.022,"15min":1.022},"last":0.35499999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:35:21 2021","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.47799999999999998,"15min":0.53200000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.249},"max":{"1min":0.67700000000000005,"5min":0.76300000000000001,"15min":15.340999999999999},"last":0.497},{"interface":"front","average":{"1min":0.502,"5min":0.56399999999999995,"15min":0.58699999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.27200000000000002},"max":{"1min":0.754,"5min":1.0109999999999999,"15min":15.356},"last":0.504}]}]},{"osd":1,"up_from":13,"seq":55834575086,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":177448,"kb_used_data":110112,"kb_used_omap":0,"kb_used_meta":67328,"kb_avail":93567704,"statfs":{"total":95995035648,"available":95813328896,"internally_reserved":0,"allocated":112754688,"data_stored":150432043,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68943872},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:52 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.379,"15min":0.51200000000000001},"min":{"1min":0.217,"5min":0.184,"15min":0.17599999999999999},"max":{"1min":4.3920000000000003,"5min":4.3920000000000003,"15min":54.427999999999997},"last":0.32200000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.42699999999999999,"15min":0.54100000000000004},"min":{"1min":0.219,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":54.356999999999999},"last":0.34399999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.76300000000000001,"5min":0.46600000000000003,"15min":0.378},"min":{"1min":0.26100000000000001,"5min":0.248,"15min":0.248},"max":{"1min":4.5490000000000004,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.39700000000000002},{"interface":"front","average":{"1min":0.67000000000000004,"5min":0.45200000000000001,"15min":0.40000000000000002},"min":{"1min":0.25700000000000001,"5min":0.254,"15min":0.254},"max":{"1min":5.0190000000000001,"5min":5.0190000000000001,"15min":5.0190000000000001},"last":0.41099999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:35:20 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.47599999999999998,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.26400000000000001,"15min":0.215},"max":{"1min":7.718,"5min":7.718,"15min":7.718},"last":0.36099999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.53700000000000003,"15min":0.434},"min":{"1min":0.245,"5min":0.245,"15min":0.20999999999999999},"max":{"1min":8.4760000000000009,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":0.38800000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:35:26 2021","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.61399999999999999,"15min":0.55200000000000005},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.22800000000000001},"max":{"1min":0.78800000000000003,"5min":4.593,"15min":4.593},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.253},"max":{"1min":4.3140000000000001,"5min":4.3140000000000001,"15min":4.6139999999999999},"last":0.441}]},{"osd":5,"last update":"Mon Mar 15 22:35:34 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.59499999999999997,"15min":0.54700000000000004},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.34300000000000003},"max":{"1min":4.3630000000000004,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60999999999999999,"15min":0.52900000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32100000000000001},"max":{"1min":1.794,"5min":3.472,"15min":3.472},"last":0.48999999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:34:59 2021","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61499999999999999,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.314},"max":{"1min":1.679,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.54700000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.63600000000000001,"15min":0.58299999999999996},"min":{"1min":0.39800000000000002,"5min":0.375,"15min":0.33700000000000002},"max":{"1min":0.82999999999999996,"5min":4.6529999999999996,"15min":4.6529999999999996},"last":0.55700000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:35:20 2021","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63700000000000001,"15min":0.60299999999999998},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.316},"max":{"1min":1.716,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.66800000000000004,"15min":0.59599999999999997},"min":{"1min":0.42799999999999999,"5min":0.40300000000000002,"15min":0.35799999999999998},"max":{"1min":1.466,"5min":4.625,"15min":4.625},"last":0.60399999999999998}]}]},{"osd":0,"up_from":8,"seq":34359738609,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":287484,"kb_used_data":229556,"kb_used_omap":0,"kb_used_meta":57920,"kb_avail":93457668,"statfs":{"total":95995035648,"available":95700652032,"internally_reserved":0,"allocated":235065344,"data_stored":348370404,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":59310080},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:34:44 2021","interfaces":[{"interface":"back","average":{"1min":0.38700000000000001,"5min":0.34399999999999997,"15min":0.33800000000000002},"min":{"1min":0.27600000000000002,"5min":0.192,"15min":0.17999999999999999},"max":{"1min":0.55400000000000005,"5min":1.46,"15min":1.7070000000000001},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.38700000000000001,"15min":0.36899999999999999},"min":{"1min":0.23499999999999999,"5min":0.22800000000000001,"15min":0.19900000000000001},"max":{"1min":1.331,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.41399999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:34:40 2021","interfaces":[{"interface":"back","average":{"1min":0.35499999999999998,"5min":0.372,"15min":0.36399999999999999},"min":{"1min":0.28299999999999997,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.46100000000000002,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.35799999999999998},{"interface":"front","average":{"1min":0.83099999999999996,"5min":0.49199999999999999,"15min":0.41399999999999998},"min":{"1min":0.26200000000000001,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":7.3799999999999999,"5min":7.3799999999999999,"15min":7.3799999999999999},"last":0.39400000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39700000000000002,"15min":0.38400000000000001},"min":{"1min":0.25700000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":0.61799999999999999,"5min":1.544,"15min":1.544},"last":0.42399999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.47499999999999998,"15min":0.41999999999999998},"min":{"1min":0.31,"5min":0.254,"15min":0.19400000000000001},"max":{"1min":0.67700000000000005,"5min":3.988,"15min":3.988},"last":0.36899999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.36899999999999999,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":0.76300000000000001,"5min":1.861,"15min":1.861},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.56599999999999995},"min":{"1min":0.40699999999999997,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":0.74099999999999999,"5min":1.5700000000000001,"15min":1.645},"last":0.54100000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:35:35 2021","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.60799999999999998,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.32600000000000001,"15min":0.29899999999999999},"max":{"1min":0.71599999999999997,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.504,"5min":0.56000000000000005,"15min":0.502},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.318},"max":{"1min":0.72099999999999997,"5min":1.909,"15min":1.909},"last":0.60099999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:34:59 2021","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.26200000000000001},"max":{"1min":2.7109999999999999,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.437,"5min":0.32000000000000001,"15min":0.27100000000000002},"max":{"1min":0.77100000000000002,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.57899999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.65500000000000003,"15min":0.58699999999999997},"min":{"1min":0.42899999999999999,"5min":0.38200000000000001,"15min":0.27900000000000003},"max":{"1min":0.753,"5min":3.8239999999999998,"15min":3.8239999999999998},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60699999999999998,"15min":0.57999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39100000000000001,"15min":0.317},"max":{"1min":0.752,"5min":2.665,"15min":2.665},"last":0.622}]}]},{"osd":2,"up_from":681,"seq":2924872728639,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":335556,"kb_used_data":288700,"kb_used_omap":119,"kb_used_meta":46728,"kb_avail":93409596,"statfs":{"total":95995035648,"available":95651426304,"internally_reserved":0,"allocated":295628800,"data_stored":390024829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":47849833},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.315,"15min":0.309},"min":{"1min":0.20899999999999999,"5min":0.19400000000000001,"15min":0.19400000000000001},"max":{"1min":0.40000000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.372,"5min":0.34599999999999997,"15min":0.32800000000000001},"min":{"1min":0.251,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.98699999999999999,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.246}]},{"osd":1,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.39900000000000002,"5min":0.35399999999999998,"15min":0.33300000000000002},"min":{"1min":0.28199999999999997,"5min":0.23699999999999999,"15min":0.23699999999999999},"max":{"1min":1.0169999999999999,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.39500000000000002,"5min":0.38,"15min":0.36299999999999999},"min":{"1min":0.27100000000000002,"5min":0.221,"15min":0.221},"max":{"1min":1.0269999999999999,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.308}]},{"osd":3,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.40600000000000003,"5min":0.41399999999999998,"15min":0.36199999999999999},"min":{"1min":0.30599999999999999,"5min":0.216,"15min":0.216},"max":{"1min":0.89300000000000002,"5min":3.496,"15min":3.496},"last":0.33500000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.375,"15min":0.35899999999999999},"min":{"1min":0.252,"5min":0.248,"15min":0.248},"max":{"1min":0.624,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.27700000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53800000000000003,"15min":0.49099999999999999},"min":{"1min":0.377,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":1.113,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.53100000000000003,"15min":0.497},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.94699999999999995,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.502}]},{"osd":5,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.497,"5min":0.51500000000000001,"15min":0.502},"min":{"1min":0.32400000000000001,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.65500000000000003,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56499999999999995,"5min":0.55500000000000005,"15min":0.51700000000000002},"min":{"1min":0.40400000000000003,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":1.1870000000000001,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.55200000000000005}]},{"osd":6,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.56399999999999995,"15min":0.51500000000000001},"min":{"1min":0.374,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":1.208,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.63},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.56499999999999995,"15min":0.52100000000000002},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.97799999999999998,"5min":1.387,"15min":1.387},"last":0.52500000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:34:43 2021","interfaces":[{"interface":"back","average":{"1min":0.56200000000000006,"5min":0.58799999999999997,"15min":0.54100000000000004},"min":{"1min":0.40699999999999997,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.222,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55300000000000005,"5min":0.56599999999999995,"15min":0.52900000000000003},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.0529999999999999,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.47099999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149926,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":302760,"kb_used_data":227040,"kb_used_omap":0,"kb_used_meta":75712,"kb_avail":93442392,"statfs":{"total":95995035648,"available":95685009408,"internally_reserved":0,"allocated":232488960,"data_stored":308617008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":77529088},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34499999999999997,"15min":0.34000000000000002},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.17699999999999999},"max":{"1min":0.47799999999999998,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":0.255},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.33300000000000002,"15min":0.33800000000000002},"min":{"1min":0.23699999999999999,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.504,"5min":0.76900000000000002,"15min":2.6339999999999999},"last":0.28399999999999997}]},{"osd":1,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.371,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23599999999999999,"15min":0.23000000000000001},"max":{"1min":0.501,"5min":0.626,"15min":6.0199999999999996},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.36799999999999999,"15min":0.39900000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.20699999999999999},"max":{"1min":0.47899999999999998,"5min":2.5619999999999998,"15min":7.6130000000000004},"last":0.312}]},{"osd":2,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35499999999999998},"min":{"1min":0.247,"5min":0.216,"15min":0.216},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.38,"5min":0.38700000000000001,"15min":0.38800000000000001},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.53600000000000003,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.35499999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:34:31 2021","interfaces":[{"interface":"back","average":{"1min":0.45500000000000002,"5min":0.53400000000000003,"15min":0.53400000000000003},"min":{"1min":0.33000000000000002,"5min":0.26300000000000001,"15min":0.22600000000000001},"max":{"1min":0.69099999999999995,"5min":3.3370000000000002,"15min":3.3370000000000002},"last":0.435},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.53400000000000003,"15min":0.52800000000000002},"min":{"1min":0.311,"5min":0.28499999999999998,"15min":0.26700000000000002},"max":{"1min":0.68000000000000005,"5min":2.1059999999999999,"15min":2.2890000000000001},"last":0.502}]},{"osd":5,"last update":"Mon Mar 15 22:34:31 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.59999999999999998,"15min":0.77700000000000002},"min":{"1min":0.313,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.71999999999999997,"5min":9.4870000000000001,"15min":9.4870000000000001},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.47499999999999998,"5min":0.66000000000000003,"15min":0.998},"min":{"1min":0.36699999999999999,"5min":0.32800000000000001,"15min":0.32800000000000001},"max":{"1min":0.66100000000000003,"5min":15.776,"15min":15.776},"last":0.47599999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:35:02 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55800000000000005,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.315,"15min":0.30099999999999999},"max":{"1min":0.78700000000000003,"5min":1.359,"15min":2.6499999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55400000000000005,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.35899999999999999,"15min":0.30399999999999999},"max":{"1min":0.77800000000000002,"5min":1.3400000000000001,"15min":3.472},"last":0.48699999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.57399999999999995},"min":{"1min":0.36199999999999999,"5min":0.308,"15min":0.29599999999999999},"max":{"1min":0.77600000000000002,"5min":0.91200000000000003,"15min":5.625},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.40799999999999997,"5min":0.371,"15min":0.35499999999999998},"max":{"1min":0.74299999999999999,"5min":2.621,"15min":2.621},"last":0.51800000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986403,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":133168,"kb_used_data":88744,"kb_used_omap":0,"kb_used_meta":44416,"kb_avail":93611984,"statfs":{"total":95995035648,"available":95858671616,"internally_reserved":0,"allocated":90873856,"data_stored":137250633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":45481984},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.50800000000000001,"15min":0.47599999999999998},"min":{"1min":0.30099999999999999,"5min":0.221,"15min":0.20000000000000001},"max":{"1min":0.55900000000000005,"5min":4.7050000000000001,"15min":4.7050000000000001},"last":0.32500000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.46500000000000002,"15min":0.498},"min":{"1min":0.27300000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.67100000000000004,"5min":1.4179999999999999,"15min":5.0209999999999999},"last":0.34699999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.312,"5min":0.25,"15min":0.23999999999999999},"max":{"1min":0.73399999999999999,"5min":1.3720000000000001,"15min":4.3499999999999996},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.499,"15min":0.499},"min":{"1min":0.311,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":0.627,"5min":1.494,"15min":4.718},"last":0.33500000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:34:38 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.49099999999999999,"15min":0.505},"min":{"1min":0.375,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.55600000000000005,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.48699999999999999,"15min":0.495},"min":{"1min":0.35799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.77200000000000002,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.38800000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.28499999999999998},"max":{"1min":1.0820000000000001,"5min":1.486,"15min":5.0499999999999998},"last":0.505},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.54400000000000004},"min":{"1min":0.38300000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0660000000000001,"5min":1.4950000000000001,"15min":4.4050000000000002},"last":0.45000000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.41999999999999998,"15min":0.40799999999999997},"min":{"1min":0.27600000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.623,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47499999999999998,"15min":0.436},"min":{"1min":0.29099999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66400000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.29099999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:34:58 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47799999999999998,"15min":0.46700000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":0.70999999999999996,"5min":0.878,"15min":4.3079999999999998},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54900000000000004,"15min":0.54000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28000000000000003},"max":{"1min":0.71899999999999997,"5min":0.83799999999999997,"15min":4.6600000000000001},"last":0.439}]},{"osd":7,"last update":"Mon Mar 15 22:35:18 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51300000000000001,"15min":0.51000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26300000000000001},"max":{"1min":0.749,"5min":0.80900000000000005,"15min":4.6929999999999996},"last":0.377},{"interface":"front","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.25},"max":{"1min":0.73599999999999999,"5min":1.034,"15min":4.9900000000000002},"last":0.41599999999999998}]}]},{"osd":5,"up_from":616,"seq":2645699854412,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":261164,"kb_used_data":220708,"kb_used_omap":153,"kb_used_meta":40294,"kb_avail":93483988,"statfs":{"total":95995035648,"available":95727603712,"internally_reserved":0,"allocated":226004992,"data_stored":315082106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":41261585},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.47999999999999998,"5min":0.48999999999999999,"15min":0.46400000000000002},"min":{"1min":0.28799999999999998,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":1.0780000000000001,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.48199999999999998,"15min":0.45600000000000002},"min":{"1min":0.26200000000000001,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":1.002,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.32500000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.56100000000000005,"15min":0.48399999999999999},"min":{"1min":0.29699999999999999,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":1.046,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.46899999999999997,"5min":0.49299999999999999,"15min":0.48799999999999999},"min":{"1min":0.28999999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":1.161,"5min":1.161,"15min":1.161},"last":0.31}]},{"osd":2,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.59799999999999998,"15min":0.54100000000000004},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":1.121,"5min":10.952,"15min":10.952},"last":0.377},{"interface":"front","average":{"1min":0.47999999999999998,"5min":0.51500000000000001,"15min":0.52700000000000002},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":1.103,"5min":1.125,"15min":1.125},"last":0.34300000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.495,"5min":0.67800000000000005,"15min":0.53800000000000003},"min":{"1min":0.32200000000000001,"5min":0.32200000000000001,"15min":0.32200000000000001},"max":{"1min":1.1379999999999999,"5min":10.355,"15min":10.355},"last":0.28399999999999997},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.505,"15min":0.45700000000000002},"min":{"1min":0.35699999999999998,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":1.1839999999999999,"5min":2.476,"15min":2.476},"last":0.29699999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.504,"15min":0.41199999999999998},"min":{"1min":0.29799999999999999,"5min":0.254,"15min":0.254},"max":{"1min":1.204,"5min":7.4000000000000004,"15min":7.4000000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.46100000000000002,"15min":0.40400000000000003},"min":{"1min":0.316,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.63400000000000001,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.35199999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.50800000000000001,"15min":0.46500000000000002},"min":{"1min":0.30199999999999999,"5min":0.23999999999999999,"15min":0.23999999999999999},"max":{"1min":0.98899999999999999,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.41499999999999998},{"interface":"front","average":{"1min":0.53200000000000003,"5min":0.55300000000000005,"15min":0.48899999999999999},"min":{"1min":0.38100000000000001,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":1.0860000000000001,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.42299999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:34:39 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.51800000000000002,"15min":0.46000000000000002},"min":{"1min":0.33600000000000002,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":1.0169999999999999,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.58399999999999996,"15min":0.51600000000000001},"min":{"1min":0.35999999999999999,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":1.0269999999999999,"5min":5.944,"15min":5.944},"last":0.434}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":12026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":10320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":8144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":234860544,"data_stored":350430920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":110673920,"data_stored":149228918,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":292245504,"data_stored":386248829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":230486016,"data_stored":307424836,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":88846336,"data_stored":136049563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":224079872,"data_stored":313868375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":169762816,"data_stored":234044670,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":205578240,"data_stored":320318413,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:35:43.167 INFO:tasks.cephadm.ceph_manager.ceph:decrease pool size by 10 2021-03-15T22:35:43.168 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool set .rgw.root pg_num 22 2021-03-15T22:35:43.169 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:43.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:43.196 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:43.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:43.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:43.320 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:43.322 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:43.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_blocked_ops 2021-03-15T22:35:43.362 INFO:tasks.rados.rados.0.smithi076.stdout:3786: done (1 left) 2021-03-15T22:35:43.362 INFO:tasks.rados.rados.0.smithi076.stdout:3787: done (0 left) 2021-03-15T22:35:43.363 INFO:tasks.rados.rados.0.smithi076.stdout:3788: copy_from oid 2 from oid 8 current snap is 445 2021-03-15T22:35:43.363 INFO:tasks.rados.rados.0.smithi076.stdout:3789: read oid 43 snap -1 2021-03-15T22:35:43.363 INFO:tasks.rados.rados.0.smithi076.stdout:3789: expect (ObjNum 1256 snap 436 seq_num 1256) 2021-03-15T22:35:43.363 INFO:tasks.rados.rados.0.smithi076.stdout:3790: write (excl) oid 11 current snap is 445 2021-03-15T22:35:43.363 INFO:tasks.rados.rados.0.smithi076.stdout:3790: seq_num 1289 ranges {464223=659618,1879407=739785,3248120=416103} 2021-03-15T22:35:43.369 INFO:tasks.rados.rados.0.smithi076.stdout:3790: writing smithi07694878-11 from 464223 to 1123841 tid 1 2021-03-15T22:35:43.380 INFO:tasks.rados.rados.0.smithi076.stdout:3790: writing smithi07694878-11 from 1879407 to 2619192 tid 2 2021-03-15T22:35:43.386 INFO:tasks.rados.rados.0.smithi076.stdout:3790: writing smithi07694878-11 from 3248120 to 3664223 tid 3 2021-03-15T22:35:43.386 INFO:tasks.rados.rados.0.smithi076.stdout:3791: read oid 37 snap -1 2021-03-15T22:35:43.387 INFO:tasks.rados.rados.0.smithi076.stdout:3791: expect (ObjNum 1284 snap 442 seq_num 1284) 2021-03-15T22:35:43.387 INFO:tasks.rados.rados.0.smithi076.stdout:3792: read oid 10 snap -1 2021-03-15T22:35:43.387 INFO:tasks.rados.rados.0.smithi076.stdout:3792: expect (ObjNum 1269 snap 437 seq_num 1269) 2021-03-15T22:35:43.387 INFO:tasks.rados.rados.0.smithi076.stdout:3793: read oid 31 snap 403 2021-03-15T22:35:43.388 INFO:tasks.rados.rados.0.smithi076.stdout:3793: expect (ObjNum 1141 snap 397 seq_num 1141) 2021-03-15T22:35:43.388 INFO:tasks.rados.rados.0.smithi076.stdout:3794: write (excl) oid 1 current snap is 445 2021-03-15T22:35:43.388 INFO:tasks.rados.rados.0.smithi076.stdout:3788: finishing copy_from to smithi07694878-2 2021-03-15T22:35:43.388 INFO:tasks.rados.rados.0.smithi076.stdout:3788: got expected ENOENT (src dne) 2021-03-15T22:35:43.389 INFO:tasks.rados.rados.0.smithi076.stdout:3794: seq_num 1290 ranges {481030=734867,1937889=483443,2875543=552426} 2021-03-15T22:35:43.393 INFO:tasks.rados.rados.0.smithi076.stdout:3794: writing smithi07694878-1 from 481030 to 1215897 tid 1 2021-03-15T22:35:43.400 INFO:tasks.rados.rados.0.smithi076.stdout:3794: writing smithi07694878-1 from 1937889 to 2421332 tid 2 2021-03-15T22:35:43.406 INFO:tasks.rados.rados.0.smithi076.stdout:3794: writing smithi07694878-1 from 2875543 to 3427969 tid 3 2021-03-15T22:35:43.406 INFO:tasks.rados.rados.0.smithi076.stdout:3795: delete oid 50 current snap is 445 2021-03-15T22:35:43.407 INFO:tasks.rados.rados.0.smithi076.stdout:3788: finishing copy_from racing read to smithi07694878-2 2021-03-15T22:35:43.407 INFO:tasks.rados.rados.0.smithi076.stdout:3789: expect (ObjNum 1256 snap 436 seq_num 1256) 2021-03-15T22:35:43.422 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:43.444 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:43.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:43.517 INFO:tasks.rados.rados.0.smithi076.stdout:3790: finishing write tid 1 to smithi07694878-11 2021-03-15T22:35:43.518 INFO:tasks.rados.rados.0.smithi076.stdout:3790: finishing write tid 2 to smithi07694878-11 2021-03-15T22:35:43.518 INFO:tasks.rados.rados.0.smithi076.stdout:3790: finishing write tid 3 to smithi07694878-11 2021-03-15T22:35:43.518 INFO:tasks.rados.rados.0.smithi076.stdout:3790: finishing write tid 4 to smithi07694878-11 2021-03-15T22:35:43.518 INFO:tasks.rados.rados.0.smithi076.stdout:3790: finishing write tid 5 to smithi07694878-11 2021-03-15T22:35:43.519 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 3048 (ObjNum 1289 snap 445 seq_num 1289) dirty exists 2021-03-15T22:35:43.519 INFO:tasks.rados.rados.0.smithi076.stdout:3790: left oid 11 (ObjNum 1289 snap 445 seq_num 1289) 2021-03-15T22:35:43.519 INFO:tasks.rados.rados.0.smithi076.stdout:3788: done (7 left) 2021-03-15T22:35:43.519 INFO:tasks.rados.rados.0.smithi076.stdout:3789: done (6 left) 2021-03-15T22:35:43.520 INFO:tasks.rados.rados.0.smithi076.stdout:3790: done (5 left) 2021-03-15T22:35:43.520 INFO:tasks.rados.rados.0.smithi076.stdout:3795: done (4 left) 2021-03-15T22:35:43.520 INFO:tasks.rados.rados.0.smithi076.stdout:3796: write (excl) oid 46 current snap is 445 2021-03-15T22:35:43.520 INFO:tasks.rados.rados.0.smithi076.stdout:3793: expect (ObjNum 1141 snap 397 seq_num 1141) 2021-03-15T22:35:43.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:43.569 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:43.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:43.571 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:43.587 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:43.626 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[62156]: cluster 2021-03-15T22:35:41.986314+0000 2021-03-15T22:35:43.626 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[62156]: osd.5 (osd.5) 59 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:35:43.627 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[62156]: cluster 2021-03-15T22:35:42.023323+0000 osd.3 (osd.3) 171 : cluster [DBG] 7.a scrub ok 2021-03-15T22:35:43.627 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[62156]: cluster 2021-03-15T22:35:42.271835+0000 osd.0 (osd.0) 158 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:35:43.627 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[62156]: cluster 2021-03-15T22:35:42.445907+0000 osd.4 (osd.4) 196 : cluster [DBG] 5.1c scrub ok 2021-03-15T22:35:43.627 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[62156]: cluster 2021-03-15T22:35:43.353157+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e988: 8 total, 8 up, 8 in 2021-03-15T22:35:43.628 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[56514]: cluster 2021-03-15T22:35:41.986314+0000 osd.5 (osd.5) 59 2021-03-15T22:35:43.628 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[56514]: : cluster [DBG] 5.15 scrub ok 2021-03-15T22:35:43.628 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[56514]: cluster 2021-03-15T22:35:42.023323+0000 osd.3 (osd.3) 171 : cluster [DBG] 7.a scrub ok 2021-03-15T22:35:43.629 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[56514]: cluster 2021-03-15T22:35:42.271835+0000 osd.0 (osd.0) 158 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:35:43.629 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[56514]: cluster 2021-03-15T22:35:42.445907+0000 osd.4 (osd.4) 196 : cluster [DBG] 5.1c scrub ok 2021-03-15T22:35:43.629 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:43 smithi076 conmon[56514]: cluster 2021-03-15T22:35:43.353157+0000 mon.a (mon.0) 2456 : cluster [DBG] osdmap e988: 8 total, 8 up, 8 in 2021-03-15T22:35:43.637 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:43.672 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:43.695 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:43.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:43.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:43.711 INFO:tasks.rados.rados.0.smithi076.stdout:3794: finishing write tid 1 to smithi07694878-1 2021-03-15T22:35:43.712 INFO:tasks.rados.rados.0.smithi076.stdout:3794: finishing write tid 2 to smithi07694878-1 2021-03-15T22:35:43.712 INFO:tasks.rados.rados.0.smithi076.stdout:3794: finishing write tid 3 to smithi07694878-1 2021-03-15T22:35:43.712 INFO:tasks.rados.rados.0.smithi076.stdout:3796: seq_num 1292 ranges {514644=485365,1654853=487112,2930389=574970} 2021-03-15T22:35:43.717 INFO:tasks.rados.rados.0.smithi076.stdout:3796: writing smithi07694878-46 from 514644 to 1000009 tid 1 2021-03-15T22:35:43.725 INFO:tasks.rados.rados.0.smithi076.stdout:3796: writing smithi07694878-46 from 1654853 to 2141965 tid 2 2021-03-15T22:35:43.735 INFO:tasks.rados.rados.0.smithi076.stdout:3796: writing smithi07694878-46 from 2930389 to 3505359 tid 3 2021-03-15T22:35:43.735 INFO:tasks.rados.rados.0.smithi076.stdout:3793: done (4 left) 2021-03-15T22:35:43.735 INFO:tasks.rados.rados.0.smithi076.stdout:3797: read oid 22 snap -1 2021-03-15T22:35:43.735 INFO:tasks.rados.rados.0.smithi076.stdout:3797: expect (ObjNum 1236 snap 426 seq_num 1236) 2021-03-15T22:35:43.736 INFO:tasks.rados.rados.0.smithi076.stdout:3798: read oid 11 snap -1 2021-03-15T22:35:43.736 INFO:tasks.rados.rados.0.smithi076.stdout:3798: expect (ObjNum 1289 snap 445 seq_num 1289) 2021-03-15T22:35:43.736 INFO:tasks.rados.rados.0.smithi076.stdout:3799: write oid 21 current snap is 445 2021-03-15T22:35:43.736 INFO:tasks.rados.rados.0.smithi076.stdout:3794: finishing write tid 4 to smithi07694878-1 2021-03-15T22:35:43.736 INFO:tasks.rados.rados.0.smithi076.stdout:3794: finishing write tid 5 to smithi07694878-1 2021-03-15T22:35:43.737 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 3081 (ObjNum 1290 snap 445 seq_num 1290) dirty exists 2021-03-15T22:35:43.737 INFO:tasks.rados.rados.0.smithi076.stdout:3794: left oid 1 (ObjNum 1290 snap 445 seq_num 1290) 2021-03-15T22:35:43.737 INFO:tasks.rados.rados.0.smithi076.stdout:3792: expect (ObjNum 1269 snap 437 seq_num 1269) 2021-03-15T22:35:43.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: cluster 2021-03-15T22:35 2021-03-15T22:35:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: :41.986314+0000 osd.5 (osd.5) 59 : cluster [DBG] 5.15 scrub ok 2021-03-15T22:35:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: cluster 2021-03-15T22:35:42.023323+0000 osd.3 (osd.3) 171 : cluster [DBG] 7.a scrub ok 2021-03-15T22:35:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: cluster 2021-03-15T22:35:42.271835+0000 osd.0 (osd.0) 158 : cluster [DBG] 7.6 scrub ok 2021-03-15T22:35:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:35:43.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: T22:35:42.445907+0000 osd.4 (osd.4) 196 : cluster [DBG] 2021-03-15T22:35:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: 5.1c scrub ok 2021-03-15T22:35:43.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:35:43.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: 03-15T22:35:43 2021-03-15T22:35:43.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: .353157+0000 mon.a (mon. 2021-03-15T22:35:43.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: 0) 2456 : cluster [DBG] 2021-03-15T22:35:43.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:43 smithi125 conmon[56739]: osdmap e988: 8 total, 8 up, 8 in 2021-03-15T22:35:43.798 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:43.820 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:43.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:43.821 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:43.923 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:43.933 INFO:tasks.rados.rados.0.smithi076.stdout:3791: expect (ObjNum 1284 snap 442 seq_num 1284) 2021-03-15T22:35:43.947 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:43.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:43.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:44.051 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:44.073 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:44.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:44.075 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:44.144 INFO:tasks.rados.rados.0.smithi076.stdout:3799: seq_num 1293 ranges {531451=676967,1713335=747123,2957812=502595} 2021-03-15T22:35:44.151 INFO:tasks.rados.rados.0.smithi076.stdout:3799: writing smithi07694878-21 from 531451 to 1208418 tid 1 2021-03-15T22:35:44.157 INFO:tasks.rados.rados.0.smithi076.stdout:3799: writing smithi07694878-21 from 1713335 to 2460458 tid 2 2021-03-15T22:35:44.163 INFO:tasks.rados.rados.0.smithi076.stdout:3799: writing smithi07694878-21 from 2957812 to 3460407 tid 3 2021-03-15T22:35:44.164 INFO:tasks.rados.rados.0.smithi076.stdout:3791: done (6 left) 2021-03-15T22:35:44.165 INFO:tasks.rados.rados.0.smithi076.stdout:3792: done (5 left) 2021-03-15T22:35:44.165 INFO:tasks.rados.rados.0.smithi076.stdout:3794: done (4 left) 2021-03-15T22:35:44.165 INFO:tasks.rados.rados.0.smithi076.stdout:3800: write (excl) oid 38 current snap is 445 2021-03-15T22:35:44.165 INFO:tasks.rados.rados.0.smithi076.stdout:3796: finishing write tid 1 to smithi07694878-46 2021-03-15T22:35:44.167 INFO:tasks.rados.rados.0.smithi076.stdout:3796: finishing write tid 2 to smithi07694878-46 2021-03-15T22:35:44.167 INFO:tasks.rados.rados.0.smithi076.stdout:3796: finishing write tid 3 to smithi07694878-46 2021-03-15T22:35:44.168 INFO:tasks.rados.rados.0.smithi076.stdout:3796: finishing write tid 4 to smithi07694878-46 2021-03-15T22:35:44.168 INFO:tasks.rados.rados.0.smithi076.stdout:3800: seq_num 1294 ranges {548258=752216,1771817=490781,2985235=713867} 2021-03-15T22:35:44.170 INFO:tasks.rados.rados.0.smithi076.stdout:3800: writing smithi07694878-38 from 548258 to 1300474 tid 1 2021-03-15T22:35:44.176 INFO:tasks.rados.rados.0.smithi076.stdout:3800: writing smithi07694878-38 from 1771817 to 2262598 tid 2 2021-03-15T22:35:44.176 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:44.184 INFO:tasks.rados.rados.0.smithi076.stdout:3800: writing smithi07694878-38 from 2985235 to 3699102 tid 3 2021-03-15T22:35:44.184 INFO:tasks.rados.rados.0.smithi076.stdout:3801: copy_from oid 19 from oid 40 current snap is 445 2021-03-15T22:35:44.184 INFO:tasks.rados.rados.0.smithi076.stdout:3796: finishing write tid 5 to smithi07694878-46 2021-03-15T22:35:44.185 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2300 (ObjNum 1292 snap 445 seq_num 1292) dirty exists 2021-03-15T22:35:44.185 INFO:tasks.rados.rados.0.smithi076.stdout:3796: left oid 46 (ObjNum 1292 snap 445 seq_num 1292) 2021-03-15T22:35:44.185 INFO:tasks.rados.rados.0.smithi076.stdout:3797: expect (ObjNum 1236 snap 426 seq_num 1236) 2021-03-15T22:35:44.198 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:44.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:44.199 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:44.301 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:44.322 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:44.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:44.324 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:44.348 INFO:tasks.rados.rados.0.smithi076.stdout:3798: expect (ObjNum 1289 snap 445 seq_num 1289) 2021-03-15T22:35:44.360 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:35:44.425 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:44.447 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:44.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:44.448 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:44.532 INFO:tasks.rados.rados.0.smithi076.stdout:3799: finishing write tid 1 to smithi07694878-21 2021-03-15T22:35:44.533 INFO:tasks.rados.rados.0.smithi076.stdout:3799: finishing write tid 2 to smithi07694878-21 2021-03-15T22:35:44.533 INFO:tasks.rados.rados.0.smithi076.stdout:3796: done (5 left) 2021-03-15T22:35:44.533 INFO:tasks.rados.rados.0.smithi076.stdout:3797: done (4 left) 2021-03-15T22:35:44.533 INFO:tasks.rados.rados.0.smithi076.stdout:3798: done (3 left) 2021-03-15T22:35:44.534 INFO:tasks.rados.rados.0.smithi076.stdout:3802: snap_create 2021-03-15T22:35:44.534 INFO:tasks.rados.rados.0.smithi076.stdout:3799: finishing write tid 3 to smithi07694878-21 2021-03-15T22:35:44.534 INFO:tasks.rados.rados.0.smithi076.stdout:3799: finishing write tid 4 to smithi07694878-21 2021-03-15T22:35:44.534 INFO:tasks.rados.rados.0.smithi076.stdout:3799: finishing write tid 5 to smithi07694878-21 2021-03-15T22:35:44.535 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 3333 (ObjNum 1293 snap 445 seq_num 1293) dirty exists 2021-03-15T22:35:44.535 INFO:tasks.rados.rados.0.smithi076.stdout:3799: left oid 21 (ObjNum 1293 snap 445 seq_num 1293) 2021-03-15T22:35:44.535 INFO:tasks.rados.rados.0.smithi076.stdout:3800: finishing write tid 1 to smithi07694878-38 2021-03-15T22:35:44.535 INFO:tasks.rados.rados.0.smithi076.stdout:3800: finishing write tid 2 to smithi07694878-38 2021-03-15T22:35:44.536 INFO:tasks.rados.rados.0.smithi076.stdout:3800: finishing write tid 3 to smithi07694878-38 2021-03-15T22:35:44.536 INFO:tasks.rados.rados.0.smithi076.stdout:3800: finishing write tid 4 to smithi07694878-38 2021-03-15T22:35:44.536 INFO:tasks.rados.rados.0.smithi076.stdout:3800: finishing write tid 5 to smithi07694878-38 2021-03-15T22:35:44.536 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2842 (ObjNum 1294 snap 445 seq_num 1294) dirty exists 2021-03-15T22:35:44.536 INFO:tasks.rados.rados.0.smithi076.stdout:3800: left oid 38 (ObjNum 1294 snap 445 seq_num 1294) 2021-03-15T22:35:44.537 INFO:tasks.rados.rados.0.smithi076.stdout:3801: finishing copy_from to smithi07694878-19 2021-03-15T22:35:44.537 INFO:tasks.rados.rados.0.smithi076.stdout:3801: got expected ENOENT (src dne) 2021-03-15T22:35:44.537 INFO:tasks.rados.rados.0.smithi076.stdout:3801: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:35:44.550 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:44.572 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:44.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:44.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:44.674 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:44.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:44.698 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:44.699 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:44.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[56514]: cluster 2021-03-15T22:35:42.721094+0000 osd.6 (osd.6) 148 : cluster [DBG] 7.d scrub ok 2021-03-15T22:35:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[56514]: cluster 2021-03-15T22:35:43.224077+0000 mgr.y (mgr.14144) 1089 : cluster [DBG] pgmap v1628: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 670 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 5.3 MiB/s wr, 22 op/s 2021-03-15T22:35:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[56514]: audit 2021-03-15T22:35:43.770413+0000 mon.c (mon.1) 81 : audit [INF] from='client.? 172.21.15.76:0/942385927' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[56514]: audit 2021-03-15T22:35:43.770763+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:44.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[56514]: cluster 2021-03-15T22:35:44.349862+0000 mon.a (mon.0) 2458 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:35:44.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[56514]: cluster 2021-03-15T22:35:44.349904+0000 mon.a (mon.0) 2459 : cluster [INF] Cluster is now healthy 2021-03-15T22:35:44.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[56514]: audit 2021-03-15T22:35:44.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[56514]: 2021-03-15T22:35:44.354176+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:44.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[56514]: cluster 2021-03-15T22:35:44.354247+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in 2021-03-15T22:35:44.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[62156]: cluster 2021-03-15T22:35:42.721094+0000 2021-03-15T22:35:44.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[62156]: osd.6 (osd.6) 148 : cluster [DBG] 7.d scrub ok 2021-03-15T22:35:44.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[62156]: cluster 2021-03-15T22:35:43.224077+0000 mgr.y (mgr.14144) 1089 : cluster [DBG] pgmap v1628: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 670 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 5.3 MiB/s wr, 22 op/s 2021-03-15T22:35:44.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[62156]: audit 2021-03-15T22:35:43.770413+0000 mon.c (mon.1) 81 : audit [INF] from='client.? 172.21.15.76:0/942385927' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:44.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[62156]: audit 2021-03-15T22:35:43.770763+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:44.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[62156]: cluster 2021-03-15T22:35:44.349862+0000 mon.a (mon.0) 2458 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:35:44.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[62156]: cluster 2021-03-15T22:35:44.349904+0000 mon.a (mon.0) 2459 : cluster [INF] Cluster is now healthy 2021-03-15T22:35:44.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[62156]: audit 2021-03-15T22:35:44.354176+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:44.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:44 smithi076 conmon[62156]: cluster 2021-03-15T22:35:44.354247+0000 mon.a (mon.0) 2461 : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in 2021-03-15T22:35:44.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:44 smithi125 conmon[56739]: cluster 2021-03-15T22:35:42.721094+0000 osd.6 (osd.6) 148 : cluster [DBG] 7.d scrub ok 2021-03-15T22:35:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:44 smithi125 conmon[56739]: cluster 2021-03-15T22:35:43.224077+0000 mgr.y (mgr.14144) 1089 : cluster [DBG] pgmap v1628: 158 pgs: 1 active+clean+snaptrim, 157 active+clean; 670 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 5.3 MiB/s wr, 22 op/s 2021-03-15T22:35:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:44 smithi125 conmon[56739]: audit 2021-03-15T22:35:43.770413+0000 mon.c (mon.1) 81 : audit [INF] from='client.? 172.21.15.76:0/942385927' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:44 smithi125 conmon[56739]: audit 2021-03-15T22:35:43.770763+0000 mon.a (mon.0) 2457 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:44 smithi125 conmon[56739]: cluster 2021-03-15T22:35:44.349862+0000 mon.a (mon.0) 2458 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:35:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:44 smithi125 conmon[56739]: cluster 2021-03-15T22:35:44.349904+0000 mon.a (mon.0) 2459 : cluster [INF] Cluster is now healthy 2021-03-15T22:35:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:44 smithi125 conmon[56739]: audit 2021-03-15T22:35:44.354176+0000 mon.a (mon.0) 2460 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:44 smithi125 conmon[56739]: cluster 2021-03-15T22:35:44.354247+0000 mon.a (mon.0) 2461 2021-03-15T22:35:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:44 smithi125 conmon[56739]: : cluster [DBG] osdmap e989: 8 total, 8 up, 8 in 2021-03-15T22:35:44.800 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:44.822 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:44.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:44.823 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:44.925 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:44.947 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:44.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:44.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:45.050 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:45.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:45.073 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:45.073 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:45.175 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:45.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:45.198 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:45.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:45.272 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:35:45.273 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:35:45.299 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:45.324 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:45.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:45.346 INFO:tasks.rados.rados.0.smithi076.stdout:3799: done (3 left) 2021-03-15T22:35:45.347 INFO:tasks.rados.rados.0.smithi076.stdout:3800: done (2 left) 2021-03-15T22:35:45.347 INFO:tasks.rados.rados.0.smithi076.stdout:3801: done (1 left) 2021-03-15T22:35:45.347 INFO:tasks.rados.rados.0.smithi076.stdout:3802: done (0 left) 2021-03-15T22:35:45.347 INFO:tasks.rados.rados.0.smithi076.stdout:3803: snap_create 2021-03-15T22:35:45.427 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:45.449 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:45.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:45.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:45.552 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:45.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:45 smithi125 conmon[56739]: cluster 2021-03-15T22:35:43.758910+0000 osd.6 ( 2021-03-15T22:35:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:45 smithi125 conmon[56739]: osd.6) 149 : cluster [DBG] 7.c scrub ok 2021-03-15T22:35:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:45 smithi125 conmon[56739]: cluster 2021-03-15T22:35:44.256482+0000 osd.0 (osd.0) 159 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:35:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:45 smithi125 conmon[56739]: cluster 2021-03-15T22:35:44.582510+0000 osd.7 (osd.7) 184 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:35:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:45 smithi125 conmon[56739]: audit 2021-03-15T22:35:45.225531+0000 mon.a (mon.0) 2462 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "40"}]: dispatch 2021-03-15T22:35:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:45 smithi125 conmon[56739]: audit 2021-03-15T22:35:45.344094+0000 mon.a (mon.0) 2463 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "40"}]': finished 2021-03-15T22:35:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:45 smithi125 conmon[56739]: cluster 2021-03-15T22:35:45.344167+0000 mon.a (mon.0) 2464 : cluster [DBG] osdmap e990: 8 total, 8 up, 8 in 2021-03-15T22:35:45.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:45.803 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:45.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:45.837 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.7 dump_historic_ops 2021-03-15T22:35:45.904 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:45.926 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:45.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:45.928 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:45.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[56514]: cluster 2021-03-15T22:35:43.758910+0000 osd.6 (osd.6) 2021-03-15T22:35:45.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[56514]: 149 : cluster [DBG] 7.c scrub ok 2021-03-15T22:35:45.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[56514]: cluster 2021-03-15T22:35:44.256482+0000 osd.0 (osd.0) 159 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:35:45.985 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[56514]: cluster 2021-03-15T22:35:44.582510+0000 osd.7 (osd.7) 184 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:35:45.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[56514]: audit 2021-03-15T22:35:45.225531+0000 mon.a (mon.0) 2462 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "40"}]: dispatch 2021-03-15T22:35:45.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[56514]: audit 2021-03-15T22:35:45.344094+0000 mon.a (mon.0) 2463 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "40"}]': finished 2021-03-15T22:35:45.986 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[56514]: cluster 2021-03-15T22:35:45.344167+0000 mon.a (mon.0) 2464 : cluster [DBG] osdmap e990: 8 total, 8 up, 8 in 2021-03-15T22:35:45.987 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: cluster 2021-03-15T22:35:45.987 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: 2021-03-15T22:35 2021-03-15T22:35:45.987 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: :43.758910+0000 osd.6 (osd.6) 149 : cluster [DBG] 7.c scrub ok 2021-03-15T22:35:45.988 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: cluster 2021-03-15T22:35:44.256482+0000 osd.0 (osd.0) 2021-03-15T22:35:45.988 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: 159 : cluster [DBG] 5.3 scrub ok 2021-03-15T22:35:45.988 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:35:45.988 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: -15T22:35:44.582510+0000 osd.7 (osd.7) 184 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:35:45.988 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: audit 2021-03-15T22:35: 2021-03-15T22:35:45.989 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: 45.225531+0000 mon.a (mon.0) 2462 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "40"}]: dispatch 2021-03-15T22:35:45.989 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: 2021-03-15T22:35:45.989 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: audit 2021-03-15T22:35: 2021-03-15T22:35:45.989 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: 45.344094+0000 mon.a (mon.0) 2463 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "40"}]': finished 2021-03-15T22:35:45.991 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: cluster 2021-03-15T22:35:45.344167+0000 mon.a (mon 2021-03-15T22:35:45.991 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:45 smithi076 conmon[62156]: .0) 2464 : cluster [DBG] osdmap e990: 8 total, 8 up, 8 in 2021-03-15T22:35:46.029 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:46.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:46.055 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:46.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:46.115 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:46.156 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:46.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:46.180 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:46.180 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:46.281 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:46.304 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:46.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:46.306 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:46.352 INFO:tasks.rados.rados.0.smithi076.stdout:3803: done (0 left) 2021-03-15T22:35:46.353 INFO:tasks.rados.rados.0.smithi076.stdout:3804: copy_from oid 19 from oid 16 current snap is 447 2021-03-15T22:35:46.353 INFO:tasks.rados.rados.0.smithi076.stdout:3805: write oid 24 current snap is 447 2021-03-15T22:35:46.353 INFO:tasks.rados.rados.0.smithi076.stdout:3805: seq_num 1295 ranges {565065=427465,1430299=634439,2496305=525139,3472104=292961} 2021-03-15T22:35:46.358 INFO:tasks.rados.rados.0.smithi076.stdout:3805: writing smithi07694878-24 from 565065 to 992530 tid 1 2021-03-15T22:35:46.364 INFO:tasks.rados.rados.0.smithi076.stdout:3805: writing smithi07694878-24 from 1430299 to 2064738 tid 2 2021-03-15T22:35:46.370 INFO:tasks.rados.rados.0.smithi076.stdout:3805: writing smithi07694878-24 from 2496305 to 3021444 tid 3 2021-03-15T22:35:46.375 INFO:tasks.rados.rados.0.smithi076.stdout:3805: writing smithi07694878-24 from 3472104 to 3765065 tid 4 2021-03-15T22:35:46.375 INFO:tasks.rados.rados.0.smithi076.stdout:3806: read oid 42 snap -1 2021-03-15T22:35:46.375 INFO:tasks.rados.rados.0.smithi076.stdout:3806: expect (ObjNum 1274 snap 439 seq_num 1274) 2021-03-15T22:35:46.375 INFO:tasks.rados.rados.0.smithi076.stdout:3807: rollback oid 22 current snap is 447 2021-03-15T22:35:46.376 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 22 to 436 2021-03-15T22:35:46.376 INFO:tasks.rados.rados.0.smithi076.stdout:3805: finishing write tid 1 to smithi07694878-24 2021-03-15T22:35:46.376 INFO:tasks.rados.rados.0.smithi076.stdout:3804: finishing copy_from to smithi07694878-19 2021-03-15T22:35:46.376 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2826 (ObjNum 1075 snap 366 seq_num 1075) dirty exists 2021-03-15T22:35:46.377 INFO:tasks.rados.rados.0.smithi076.stdout:3805: finishing write tid 2 to smithi07694878-24 2021-03-15T22:35:46.377 INFO:tasks.rados.rados.0.smithi076.stdout:3804: finishing copy_from racing read to smithi07694878-19 2021-03-15T22:35:46.377 INFO:tasks.rados.rados.0.smithi076.stdout:3805: finishing write tid 3 to smithi07694878-24 2021-03-15T22:35:46.377 INFO:tasks.rados.rados.0.smithi076.stdout:3804: done (3 left) 2021-03-15T22:35:46.377 INFO:tasks.rados.rados.0.smithi076.stdout:3808: copy_from oid 47 from oid 48 current snap is 447 2021-03-15T22:35:46.378 INFO:tasks.rados.rados.0.smithi076.stdout:3809: read oid 29 snap -1 2021-03-15T22:35:46.378 INFO:tasks.rados.rados.0.smithi076.stdout:3809: expect (ObjNum 1228 snap 425 seq_num 1228) 2021-03-15T22:35:46.378 INFO:tasks.rados.rados.0.smithi076.stdout:3810: snap_remove snap 360 2021-03-15T22:35:46.406 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:35:46.407 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:46.423 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:35:46.430 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:46.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:46.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:46.440 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:35:46.457 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:35:46.476 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:35:46.493 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:35:46.511 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:35:46.530 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:35:46.531 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:35:46.531 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:35:46.532 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:35:46.532 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:35:46.532 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:35:46.532 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:35:46.532 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:35:46.533 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:35:46.533 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:46.555 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:46.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:46.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:46.565 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:46 smithi125 conmon[56739]: cluster 2021-03-15T22:35:45.165830+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:35:46.565 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:46 smithi125 conmon[56739]: cluster 2021-03-15T22:35:45.224723+0000 mgr.y (mgr.14144) 1090 : cluster [DBG] pgmap v1631: 157 pgs: 157 active+clean; 682 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.2 MiB/s rd, 3.2 MiB/s wr, 18 op/s 2021-03-15T22:35:46.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:46 smithi125 conmon[56739]: cluster 2021-03-15T22:35:45.355643+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:35:46.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:46 smithi125 conmon[56739]: cluster 2021-03-15T22:35:45.410461+0000 osd.4 (osd.4) 197 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:35:46.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:46 smithi125 conmon[56739]: cluster 2021-03-15T22:35:45.547389+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.15 deep-scrub ok 2021-03-15T22:35:46.566 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:46 smithi125 conmon[56739]: cluster 2021-03-15T22:35:45.676506+0000 osd.3 (osd.3) 172 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:46.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:46 smithi125 conmon[56739]: cluster 2021-03-15T22:35:45.679551+0000 osd.3 (osd.3) 173 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:46.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:46 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:35:46.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:46 smithi125 conmon[56739]: -15T22:35:46.349437+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e991: 8 total, 8 up, 8 in 2021-03-15T22:35:46.567 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:46 smithi125 conmon[56739]: audit 2021-03-15T22:35:46.381778+0000 mon.c (mon.1) 82 : audit [DBG] from='client.? 172.21.15.76:0/2692180064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:46.657 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:46.681 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:46.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:46.683 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:46.784 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:46.807 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:46.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:46.808 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:46.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:46.932 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:46.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:46.933 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:47.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[56514]: cluster 2021-03-15T22:35:45.165830+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:35:47.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[56514]: cluster 2021-03-15T22:35:45.224723+0000 mgr.y (mgr.14144) 1090 : cluster [DBG] pgmap v1631: 157 pgs: 157 active+clean; 682 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.2 MiB/s rd, 3.2 MiB/s wr, 18 op/s 2021-03-15T22:35:47.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[56514]: cluster 2021-03-15T22:35:45.355643+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:35:47.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[56514]: cluster 2021-03-15T22:35:45.410461+0000 osd.4 (osd.4) 197 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:35:47.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[56514]: cluster 2021-03-15T22:35:45.547389+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.15 deep-scrub ok 2021-03-15T22:35:47.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[56514]: cluster 2021-03-15T22:35:45.676506+0000 osd.3 (osd.3) 172 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:47.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[56514]: cluster 2021-03-15T22:35:45.679551+0000 osd.3 (osd.3) 173 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:47.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[56514]: cluster 2021-03-15T22:35:46.349437+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e991: 8 total, 8 up, 8 in 2021-03-15T22:35:47.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[56514]: audit 2021-03-15T22:35:46.381778+0000 mon.c (mon.1) 82 : audit [DBG] from='client.? 172.21.15.76:0/2692180064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:47.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[62156]: cluster 2021-03-15T22:35:45.165830+0000 osd.1 (osd.1) 186 : cluster [DBG] 3.8 scrub ok 2021-03-15T22:35:47.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[62156]: cluster 2021-03-15T22:35:45.224723+0000 mgr.y (mgr.14144) 1090 : cluster [DBG] pgmap v1631: 157 pgs: 157 active+clean; 682 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 8.2 MiB/s rd, 3.2 MiB/s wr, 18 op/s 2021-03-15T22:35:47.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[62156]: cluster 2021-03-15T22:35:45.355643+0000 osd.2 (osd.2) 49 : cluster [DBG] 5.6 scrub ok 2021-03-15T22:35:47.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[62156]: cluster 2021-03-15T22:35:45.410461+0000 osd.4 (osd.4) 197 : cluster [DBG] 4.1b scrub ok 2021-03-15T22:35:47.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[62156]: cluster 2021-03-15T22:35:45.547389+0000 osd.7 (osd.7) 185 : cluster [DBG] 6.15 deep-scrub ok 2021-03-15T22:35:47.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[62156]: cluster 2021-03-15T22:35:45.676506+0000 osd.3 (osd.3) 172 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:47.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[62156]: cluster 2021-03-15T22:35:45.679551+0000 osd.3 (osd.3) 173 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:47.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[62156]: cluster 2021-03-15T22:35:46.349437+0000 mon.a (mon.0) 2465 : cluster [DBG] osdmap e991: 8 total, 8 up, 8 in 2021-03-15T22:35:47.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:46 smithi076 conmon[62156]: audit 2021-03-15T22:35:46.381778+0000 mon.c (mon.1) 82 : audit [DBG] from='client.? 172.21.15.76:0/2692180064' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:47.034 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:47.056 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:47.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:47.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:47.159 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:47.180 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:47.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:47.182 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:47.283 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:47.305 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:47.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:47.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:47.355 INFO:tasks.rados.rados.0.smithi076.stdout:3810: done (5 left) 2021-03-15T22:35:47.355 INFO:tasks.rados.rados.0.smithi076.stdout:3811: delete oid 5 current snap is 447 2021-03-15T22:35:47.355 INFO:tasks.rados.rados.0.smithi076.stdout:3805: finishing write tid 4 to smithi07694878-24 2021-03-15T22:35:47.355 INFO:tasks.rados.rados.0.smithi076.stdout:3805: finishing write tid 5 to smithi07694878-24 2021-03-15T22:35:47.356 INFO:tasks.rados.rados.0.smithi076.stdout:3805: finishing write tid 6 to smithi07694878-24 2021-03-15T22:35:47.356 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 1745 (ObjNum 1295 snap 447 seq_num 1295) dirty exists 2021-03-15T22:35:47.356 INFO:tasks.rados.rados.0.smithi076.stdout:3805: left oid 24 (ObjNum 1295 snap 447 seq_num 1295) 2021-03-15T22:35:47.356 INFO:tasks.rados.rados.0.smithi076.stdout:3807: finishing rollback tid 0 to smithi07694878-22 2021-03-15T22:35:47.357 INFO:tasks.rados.rados.0.smithi076.stdout:3807: finishing rollback tid 1 to smithi07694878-22 2021-03-15T22:35:47.357 INFO:tasks.rados.rados.0.smithi076.stdout:3807: finishing rollback tid 2 to smithi07694878-22 2021-03-15T22:35:47.357 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 3212 (ObjNum 1181 snap 414 seq_num 1181) dirty exists 2021-03-15T22:35:47.357 INFO:tasks.rados.rados.0.smithi076.stdout:3806: expect (ObjNum 1274 snap 439 seq_num 1274) 2021-03-15T22:35:47.370 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 3 pg_num to 22 2021-03-15T22:35:47.408 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:47.430 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:47.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:47.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:47.533 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:47.555 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:47.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:47.557 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:47.587 INFO:tasks.rados.rados.0.smithi076.stdout:3809: expect (ObjNum 1228 snap 425 seq_num 1228) 2021-03-15T22:35:47.658 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:47.679 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:47.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:47.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:47.693 INFO:tasks.rados.rados.0.smithi076.stdout:3808: finishing copy_from to smithi07694878-47 2021-03-15T22:35:47.693 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 3155 (ObjNum 1221 snap 423 seq_num 1221) dirty exists 2021-03-15T22:35:47.693 INFO:tasks.rados.rados.0.smithi076.stdout:3808: finishing copy_from racing read to smithi07694878-47 2021-03-15T22:35:47.694 INFO:tasks.rados.rados.0.smithi076.stdout:3805: done (5 left) 2021-03-15T22:35:47.694 INFO:tasks.rados.rados.0.smithi076.stdout:3806: done (4 left) 2021-03-15T22:35:47.694 INFO:tasks.rados.rados.0.smithi076.stdout:3807: done (3 left) 2021-03-15T22:35:47.694 INFO:tasks.rados.rados.0.smithi076.stdout:3808: done (2 left) 2021-03-15T22:35:47.695 INFO:tasks.rados.rados.0.smithi076.stdout:3809: done (1 left) 2021-03-15T22:35:47.695 INFO:tasks.rados.rados.0.smithi076.stdout:3811: done (0 left) 2021-03-15T22:35:47.695 INFO:tasks.rados.rados.0.smithi076.stdout:3812: snap_remove snap 435 2021-03-15T22:35:47.699 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:35:47.700 INFO:teuthology.orchestra.run.smithi125.stdout: "size": 20, 2021-03-15T22:35:47.700 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 600, 2021-03-15T22:35:47.700 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [ 2021-03-15T22:35:47.700 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.701 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4380 7.6 7:6283974b:::smithi07694878-30:head [write 484200~624934,stat] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:35:47.701 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:35:47.701 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 577.56478948200004, 2021-03-15T22:35:47.701 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.122158787, 2021-03-15T22:35:47.702 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.702 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:47.702 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.702 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.702 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.703 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4380 2021-03-15T22:35:47.703 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.703 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.703 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.703 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.704 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:35:47.704 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.704 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.704 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.705 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.705 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.130851+0000", 2021-03-15T22:35:47.705 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00065488299999999998 2021-03-15T22:35:47.705 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.705 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.706 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.706 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131506+0000", 2021-03-15T22:35:47.706 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.037e-06 2021-03-15T22:35:47.706 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.706 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.707 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.707 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131512+0000", 2021-03-15T22:35:47.707 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9637000000000001e-05 2021-03-15T22:35:47.707 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.708 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.708 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.708 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131541+0000", 2021-03-15T22:35:47.708 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9325000000000001e-05 2021-03-15T22:35:47.708 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.709 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.709 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.709 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131571+0000", 2021-03-15T22:35:47.709 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.6747000000000002e-05 2021-03-15T22:35:47.710 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.710 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.710 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.710 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131647+0000", 2021-03-15T22:35:47.710 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000176789 2021-03-15T22:35:47.711 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.711 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.711 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.711 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.131824+0000", 2021-03-15T22:35:47.711 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00042344700000000003 2021-03-15T22:35:47.712 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.713 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.713 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:35:47.713 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.132248+0000", 2021-03-15T22:35:47.713 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0026234660000000001 2021-03-15T22:35:47.714 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.714 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.714 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.714 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.134871+0000", 2021-03-15T22:35:47.715 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00031408100000000002 2021-03-15T22:35:47.715 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.715 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.715 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:47.715 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.135185+0000", 2021-03-15T22:35:47.716 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.117673658 2021-03-15T22:35:47.716 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.716 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.716 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.716 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.252859+0000", 2021-03-15T22:35:47.717 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.6511999999999997e-05 2021-03-15T22:35:47.717 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.717 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.717 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:47.718 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.252915+0000", 2021-03-15T22:35:47.718 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4204999999999997e-05 2021-03-15T22:35:47.718 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.718 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.718 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.719 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253009+0000", 2021-03-15T22:35:47.719 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.122158787 2021-03-15T22:35:47.719 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.719 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.720 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.720 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.720 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.720 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4381 7.6 7:6283974b:::smithi07694878-30:head [write 1860843~223357] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:35:47.721 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:35:47.721 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 577.55951853500005, 2021-03-15T22:35:47.721 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119715842, 2021-03-15T22:35:47.721 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.721 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:47.722 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.722 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.722 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.722 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4381 2021-03-15T22:35:47.723 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.723 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.723 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.723 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.723 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:35:47.724 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.724 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.724 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.724 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.725 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136122+0000", 2021-03-15T22:35:47.725 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019479299999999999 2021-03-15T22:35:47.725 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.725 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.725 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.726 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136316+0000", 2021-03-15T22:35:47.726 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7419999999999999e-06 2021-03-15T22:35:47.726 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.726 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.727 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.727 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136320+0000", 2021-03-15T22:35:47.727 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5999999999999999e-05 2021-03-15T22:35:47.727 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.727 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.728 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.728 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136336+0000", 2021-03-15T22:35:47.728 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4386e-05 2021-03-15T22:35:47.728 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.728 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.729 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.729 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136361+0000", 2021-03-15T22:35:47.729 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4256999999999999e-05 2021-03-15T22:35:47.729 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.729 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.730 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.730 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136415+0000", 2021-03-15T22:35:47.730 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000158257 2021-03-15T22:35:47.730 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.731 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.731 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:47.731 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136573+0000", 2021-03-15T22:35:47.731 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.116481746 2021-03-15T22:35:47.731 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.732 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.732 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.732 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253055+0000", 2021-03-15T22:35:47.732 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1245e-05 2021-03-15T22:35:47.732 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.733 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.733 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.733 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253126+0000", 2021-03-15T22:35:47.733 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000138727 2021-03-15T22:35:47.733 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.734 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.734 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:35:47.734 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.253265+0000", 2021-03-15T22:35:47.734 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0017125980000000001 2021-03-15T22:35:47.735 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.735 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.735 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:47.735 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254977+0000", 2021-03-15T22:35:47.735 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000112898 2021-03-15T22:35:47.736 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.736 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.736 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.736 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255090+0000", 2021-03-15T22:35:47.736 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00066560399999999998 2021-03-15T22:35:47.737 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.737 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.737 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.737 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255756+0000", 2021-03-15T22:35:47.737 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8887999999999998e-05 2021-03-15T22:35:47.738 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.738 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.738 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:47.738 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255795+0000", 2021-03-15T22:35:47.739 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2701000000000003e-05 2021-03-15T22:35:47.739 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.739 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.739 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.739 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255837+0000", 2021-03-15T22:35:47.740 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119715842 2021-03-15T22:35:47.740 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.740 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.740 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.740 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.741 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.741 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4382 7.6 7:6283974b:::smithi07694878-30:head [setxattr _header (58) in=7b,truncate 2084200] snapc 15f=[15f,15b,15a,153,151,148,143,141,13f,13e,133,131,12f,128,126,123,10c,102] ondisk+write+known_if_redirected e455)", 2021-03-15T22:35:47.741 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:35:47.741 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 577.55926945500005, 2021-03-15T22:35:47.742 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119633089, 2021-03-15T22:35:47.742 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.742 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:47.742 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.742 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.743 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.743 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4382 2021-03-15T22:35:47.743 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.743 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.743 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.744 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.744 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:35:47.744 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.744 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.744 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.745 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.745 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136371+0000", 2021-03-15T22:35:47.745 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4609999999999998e-06 2021-03-15T22:35:47.745 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.746 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.746 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.746 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136373+0000", 2021-03-15T22:35:47.746 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7880000000000001e-06 2021-03-15T22:35:47.746 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.747 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.747 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.747 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136375+0000", 2021-03-15T22:35:47.747 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2340000000000002e-06 2021-03-15T22:35:47.747 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.748 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.748 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.748 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136380+0000", 2021-03-15T22:35:47.748 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.3160000000000001e-06 2021-03-15T22:35:47.749 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.749 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.749 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.749 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136384+0000", 2021-03-15T22:35:47.749 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024020700000000001 2021-03-15T22:35:47.750 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.750 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.750 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.750 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136624+0000", 2021-03-15T22:35:47.750 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7928000000000003e-05 2021-03-15T22:35:47.751 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.751 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.751 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:47.751 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136702+0000", 2021-03-15T22:35:47.751 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.11744721399999999 2021-03-15T22:35:47.752 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.752 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.752 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.752 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254149+0000", 2021-03-15T22:35:47.753 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.5769000000000001e-05 2021-03-15T22:35:47.753 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.753 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.753 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.753 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254205+0000", 2021-03-15T22:35:47.754 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000127489 2021-03-15T22:35:47.754 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.754 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.754 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,5", 2021-03-15T22:35:47.755 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254332+0000", 2021-03-15T22:35:47.755 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00068339200000000001 2021-03-15T22:35:47.755 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.755 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.755 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:47.756 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255016+0000", 2021-03-15T22:35:47.756 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000233143 2021-03-15T22:35:47.756 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.756 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.757 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.757 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255249+0000", 2021-03-15T22:35:47.757 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00063040000000000004 2021-03-15T22:35:47.757 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.757 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.758 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.758 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255879+0000", 2021-03-15T22:35:47.758 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.9571000000000003e-05 2021-03-15T22:35:47.758 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.758 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.759 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:47.759 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.255939+0000", 2021-03-15T22:35:47.759 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5177000000000003e-05 2021-03-15T22:35:47.759 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.760 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.760 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.760 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256004+0000", 2021-03-15T22:35:47.760 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119633089 2021-03-15T22:35:47.760 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.761 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.761 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.761 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.761 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.762 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:4383 7.6 7:6283974b:::smithi07694878-30:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e455)", 2021-03-15T22:35:47.762 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:35:47.762 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 577.55925172399998, 2021-03-15T22:35:47.762 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119776914, 2021-03-15T22:35:47.762 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.763 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:35:47.763 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.763 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.763 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.763 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 4383 2021-03-15T22:35:47.764 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.764 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.764 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.764 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.765 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:35:47.765 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.765 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.765 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.765 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.766 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136388+0000", 2021-03-15T22:35:47.766 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9899999999999999e-07 2021-03-15T22:35:47.766 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.766 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.766 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.767 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136389+0000", 2021-03-15T22:35:47.767 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1799999999999999e-06 2021-03-15T22:35:47.767 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.767 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.768 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.768 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136390+0000", 2021-03-15T22:35:47.768 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0630000000000002e-06 2021-03-15T22:35:47.768 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.768 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.769 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.769 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136396+0000", 2021-03-15T22:35:47.769 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6900000000000001e-06 2021-03-15T22:35:47.769 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.769 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.770 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.770 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136399+0000", 2021-03-15T22:35:47.770 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00034116099999999998 2021-03-15T22:35:47.770 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.770 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.771 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.771 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136740+0000", 2021-03-15T22:35:47.771 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000123552 2021-03-15T22:35:47.771 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.772 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.772 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:47.772 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.136864+0000", 2021-03-15T22:35:47.772 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.118007994 2021-03-15T22:35:47.772 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.773 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.773 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.773 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254872+0000", 2021-03-15T22:35:47.773 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0631999999999996e-05 2021-03-15T22:35:47.773 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.774 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.774 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:47.774 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.254963+0000", 2021-03-15T22:35:47.774 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010704379999999999 2021-03-15T22:35:47.775 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.775 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.775 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.775 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256033+0000", 2021-03-15T22:35:47.775 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2941e-05 2021-03-15T22:35:47.776 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.776 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.776 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.776 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256086+0000", 2021-03-15T22:35:47.776 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9463999999999997e-05 2021-03-15T22:35:47.777 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.777 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.777 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.777 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:26:10.256165+0000", 2021-03-15T22:35:47.777 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.119776914 2021-03-15T22:35:47.778 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.778 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.778 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.778 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.779 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.779 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.34361.0:611 2.0 2:353ec8f9:::gateway.conf:head [getxattr epoch (1) in=5b] snapc 0=[] ondisk+read+known_if_redirected e551)", 2021-03-15T22:35:47.779 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:28:12.419483+0000", 2021-03-15T22:35:47.779 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 455.276157066, 2021-03-15T22:35:47.779 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.091628773999999996, 2021-03-15T22:35:47.780 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.780 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:35:47.780 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.780 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.34361", 2021-03-15T22:35:47.780 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.125:0/1442040869", 2021-03-15T22:35:47.781 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 611 2021-03-15T22:35:47.781 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.781 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.781 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.781 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.782 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419483+0000", 2021-03-15T22:35:47.782 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.782 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.782 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.783 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.783 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419483+0000", 2021-03-15T22:35:47.783 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2217000000000001e-05 2021-03-15T22:35:47.783 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.784 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.784 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.784 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419505+0000", 2021-03-15T22:35:47.784 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9129999999999996e-06 2021-03-15T22:35:47.785 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.785 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.785 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.785 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419509+0000", 2021-03-15T22:35:47.785 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5500999999999999e-05 2021-03-15T22:35:47.786 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.786 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.786 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.786 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419525+0000", 2021-03-15T22:35:47.787 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3603000000000001e-05 2021-03-15T22:35:47.787 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.787 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.787 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.787 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419548+0000", 2021-03-15T22:35:47.788 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.8461999999999996e-05 2021-03-15T22:35:47.788 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.788 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.788 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.788 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419617+0000", 2021-03-15T22:35:47.789 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.277e-05 2021-03-15T22:35:47.789 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.789 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.789 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:35:47.789 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.419630+0000", 2021-03-15T22:35:47.790 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.091151858000000002 2021-03-15T22:35:47.790 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.790 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.790 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.791 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.510781+0000", 2021-03-15T22:35:47.791 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014667799999999999 2021-03-15T22:35:47.791 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.791 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.791 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.792 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.510928+0000", 2021-03-15T22:35:47.792 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00018377199999999999 2021-03-15T22:35:47.792 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.792 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.792 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.793 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:28:12.511112+0000", 2021-03-15T22:35:47.793 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.091628773999999996 2021-03-15T22:35:47.793 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.793 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.794 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.794 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.794 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.794 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5830 7.5 7:ac84e656:::smithi07694878-31:head [copy-from ver 1320] snapc 1f0=[1f0,1ee,1e9,1e7,1e6,1e4,1e0,1df,1dc,1d5,1d4,1cc,1c7,1c5,1c3,1b6,1ae,1a6,1a4,1a1,19c,193,180,17a,176,15f,141] ondisk+write+known_if_redirected e631)", 2021-03-15T22:35:47.794 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:40.245985+0000", 2021-03-15T22:35:47.795 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 367.44965560100002, 2021-03-15T22:35:47.795 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.089874809, 2021-03-15T22:35:47.795 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.795 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:47.795 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.796 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.796 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.796 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5830 2021-03-15T22:35:47.796 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.796 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.797 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.797 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.797 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.245985+0000", 2021-03-15T22:35:47.797 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.798 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.798 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.798 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.798 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.245985+0000", 2021-03-15T22:35:47.798 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0769999999999998e-06 2021-03-15T22:35:47.799 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.799 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.799 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.799 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.245990+0000", 2021-03-15T22:35:47.799 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2939999999999999e-06 2021-03-15T22:35:47.800 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.800 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.800 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.800 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.245994+0000", 2021-03-15T22:35:47.800 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.7839999999999999e-05 2021-03-15T22:35:47.801 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.801 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.801 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.801 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246012+0000", 2021-03-15T22:35:47.802 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2398000000000001e-05 2021-03-15T22:35:47.802 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.802 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.802 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.802 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246024+0000", 2021-03-15T22:35:47.803 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.1279999999999997e-05 2021-03-15T22:35:47.803 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.803 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.803 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.803 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246075+0000", 2021-03-15T22:35:47.804 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3337000000000005e-05 2021-03-15T22:35:47.804 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.804 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.804 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.804 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246159+0000", 2021-03-15T22:35:47.805 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0051863150000000004 2021-03-15T22:35:47.805 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.805 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.805 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,1", 2021-03-15T22:35:47.806 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.251345+0000", 2021-03-15T22:35:47.806 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.054202781999999998 2021-03-15T22:35:47.806 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.806 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.806 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:47.807 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.305548+0000", 2021-03-15T22:35:47.807 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017903207000000001 2021-03-15T22:35:47.807 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.807 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.807 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.808 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.323451+0000", 2021-03-15T22:35:47.808 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.012244708 2021-03-15T22:35:47.808 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.808 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.809 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.809 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.335696+0000", 2021-03-15T22:35:47.809 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6770000000000001e-05 2021-03-15T22:35:47.809 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.809 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.810 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:47.810 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.335743+0000", 2021-03-15T22:35:47.810 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000116801 2021-03-15T22:35:47.810 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.810 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.811 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.811 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.335859+0000", 2021-03-15T22:35:47.811 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.089874809 2021-03-15T22:35:47.811 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.811 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.812 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.812 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.812 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.812 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:5831 7.5 7:ac84e656:::smithi07694878-31:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e631)", 2021-03-15T22:35:47.812 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:40.246040+0000", 2021-03-15T22:35:47.813 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 367.44960001200002, 2021-03-15T22:35:47.813 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.090216515999999997, 2021-03-15T22:35:47.813 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.813 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:35:47.814 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.814 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.814 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.814 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 5831 2021-03-15T22:35:47.814 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.815 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.815 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.815 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.815 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246040+0000", 2021-03-15T22:35:47.816 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.816 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.816 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.816 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.816 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246040+0000", 2021-03-15T22:35:47.817 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.1790000000000001e-06 2021-03-15T22:35:47.817 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.817 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.817 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.817 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246041+0000", 2021-03-15T22:35:47.818 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3319999999999999e-06 2021-03-15T22:35:47.818 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.818 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.818 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.818 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246043+0000", 2021-03-15T22:35:47.819 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6370000000000002e-06 2021-03-15T22:35:47.819 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.819 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.819 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.819 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246049+0000", 2021-03-15T22:35:47.820 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3510000000000004e-06 2021-03-15T22:35:47.820 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.820 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.820 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.820 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246054+0000", 2021-03-15T22:35:47.821 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00024099399999999999 2021-03-15T22:35:47.821 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.821 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.821 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.822 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246295+0000", 2021-03-15T22:35:47.822 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7129999999999997e-05 2021-03-15T22:35:47.822 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.822 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.822 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:35:47.823 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.246352+0000", 2021-03-15T22:35:47.823 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0096365749999999997 2021-03-15T22:35:47.823 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.823 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.824 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.824 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.255988+0000", 2021-03-15T22:35:47.824 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121428 2021-03-15T22:35:47.824 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.824 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.825 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:47.825 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.256110+0000", 2021-03-15T22:35:47.825 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.079801443 2021-03-15T22:35:47.825 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.825 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.826 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.826 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.335911+0000", 2021-03-15T22:35:47.826 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012181899999999999 2021-03-15T22:35:47.826 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.827 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.827 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.827 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.336033+0000", 2021-03-15T22:35:47.827 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00022362799999999999 2021-03-15T22:35:47.827 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.828 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.828 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.828 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:40.336257+0000", 2021-03-15T22:35:47.828 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.090216515999999997 2021-03-15T22:35:47.828 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.829 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.829 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.829 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.829 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.829 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2102 5.d 5:b4812045:::notify.4:head [watch ping cookie 94751391775488 gen 9] snapc 0=[] ondisk+write+known_if_redirected e637)", 2021-03-15T22:35:47.830 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.130297+0000", 2021-03-15T22:35:47.830 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 355.565343563, 2021-03-15T22:35:47.830 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.098088540000000002, 2021-03-15T22:35:47.830 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.831 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:35:47.831 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.831 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:35:47.831 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:35:47.831 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2102 2021-03-15T22:35:47.832 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.832 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.832 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.832 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.832 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130297+0000", 2021-03-15T22:35:47.833 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.833 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.833 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.833 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.834 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130297+0000", 2021-03-15T22:35:47.834 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.739e-06 2021-03-15T22:35:47.834 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.834 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.834 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.835 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130298+0000", 2021-03-15T22:35:47.835 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.6400000000000001e-06 2021-03-15T22:35:47.835 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.835 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.835 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.836 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130301+0000", 2021-03-15T22:35:47.836 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4290000000000007e-06 2021-03-15T22:35:47.836 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.836 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.837 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.837 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130310+0000", 2021-03-15T22:35:47.837 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0030000000000001e-05 2021-03-15T22:35:47.837 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.837 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.838 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.838 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130320+0000", 2021-03-15T22:35:47.838 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000322384 2021-03-15T22:35:47.838 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.838 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.839 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.839 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130643+0000", 2021-03-15T22:35:47.839 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5191000000000001e-05 2021-03-15T22:35:47.839 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.839 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.840 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:35:47.840 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130658+0000", 2021-03-15T22:35:47.840 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.097505270000000005 2021-03-15T22:35:47.840 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.841 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.841 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.841 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228163+0000", 2021-03-15T22:35:47.841 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00014791500000000001 2021-03-15T22:35:47.841 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.842 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.842 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.842 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228311+0000", 2021-03-15T22:35:47.842 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.3942000000000005e-05 2021-03-15T22:35:47.842 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.843 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.843 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.843 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228385+0000", 2021-03-15T22:35:47.843 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.098088540000000002 2021-03-15T22:35:47.843 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.844 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.844 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.844 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.844 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.844 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2108 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94751391775488 gen 10] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:35:47.845 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.130994+0000", 2021-03-15T22:35:47.845 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 355.56464595, 2021-03-15T22:35:47.845 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.100843209, 2021-03-15T22:35:47.845 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.846 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:35:47.846 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.846 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:35:47.846 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:35:47.846 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2108 2021-03-15T22:35:47.847 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.847 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.847 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.847 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.847 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130994+0000", 2021-03-15T22:35:47.848 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.848 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.848 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.848 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.849 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130994+0000", 2021-03-15T22:35:47.849 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.694e-06 2021-03-15T22:35:47.849 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.849 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.849 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.850 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130996+0000", 2021-03-15T22:35:47.850 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.1129999999999999e-06 2021-03-15T22:35:47.850 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.850 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.850 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.851 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.130998+0000", 2021-03-15T22:35:47.851 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.4609999999999997e-06 2021-03-15T22:35:47.851 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.851 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.851 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.852 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131007+0000", 2021-03-15T22:35:47.852 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0089999999999998e-06 2021-03-15T22:35:47.852 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.852 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.853 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.853 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131012+0000", 2021-03-15T22:35:47.853 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4428e-05 2021-03-15T22:35:47.853 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.853 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.854 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.854 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131077+0000", 2021-03-15T22:35:47.854 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2669e-05 2021-03-15T22:35:47.854 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.854 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.855 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:35:47.855 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131090+0000", 2021-03-15T22:35:47.855 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.097317492000000005 2021-03-15T22:35:47.855 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.855 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.856 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.856 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228407+0000", 2021-03-15T22:35:47.856 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6538000000000006e-05 2021-03-15T22:35:47.856 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.857 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.857 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.857 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228474+0000", 2021-03-15T22:35:47.857 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0033638050000000001 2021-03-15T22:35:47.857 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.858 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.858 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.858 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.231837+0000", 2021-03-15T22:35:47.858 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.100843209 2021-03-15T22:35:47.858 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.859 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.859 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.859 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.859 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.859 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.14397.0:2111 5.d 5:b4812045:::notify.4:head [watch reconnect cookie 94751391775488 gen 11] snapc 0=[] ondisk+write+known_if_redirected e642)", 2021-03-15T22:35:47.860 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:29:52.131034+0000", 2021-03-15T22:35:47.860 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 355.564606051, 2021-03-15T22:35:47.860 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.101203874, 2021-03-15T22:35:47.860 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.861 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:35:47.861 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.861 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.14397", 2021-03-15T22:35:47.861 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/926017807", 2021-03-15T22:35:47.861 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 2111 2021-03-15T22:35:47.862 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.862 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.862 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.862 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.862 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131034+0000", 2021-03-15T22:35:47.863 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.863 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.863 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.863 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.863 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131034+0000", 2021-03-15T22:35:47.864 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0070000000000001e-06 2021-03-15T22:35:47.864 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.864 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.864 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.865 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131035+0000", 2021-03-15T22:35:47.865 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.384e-06 2021-03-15T22:35:47.865 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.865 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.865 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.866 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131037+0000", 2021-03-15T22:35:47.866 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.1930000000000004e-06 2021-03-15T22:35:47.866 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.866 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.866 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.867 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131044+0000", 2021-03-15T22:35:47.867 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.3359999999999997e-06 2021-03-15T22:35:47.867 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.867 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.868 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.868 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131048+0000", 2021-03-15T22:35:47.868 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2633999999999996e-05 2021-03-15T22:35:47.868 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.868 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.869 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.869 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131121+0000", 2021-03-15T22:35:47.869 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.4610000000000002e-06 2021-03-15T22:35:47.869 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.869 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.870 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:35:47.870 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.131128+0000", 2021-03-15T22:35:47.870 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.097620176000000003 2021-03-15T22:35:47.870 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.871 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.871 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.871 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228748+0000", 2021-03-15T22:35:47.871 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.0249999999999999e-05 2021-03-15T22:35:47.871 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.872 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.872 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.872 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.228829+0000", 2021-03-15T22:35:47.872 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0034094329999999999 2021-03-15T22:35:47.872 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.873 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.873 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.873 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:29:52.232238+0000", 2021-03-15T22:35:47.873 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.101203874 2021-03-15T22:35:47.873 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.874 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.874 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.874 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.874 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.874 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7843 7.5 7:ac84e656:::smithi07694878-31:head [write 716262~504854] snapc 2b6=[2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,293,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e830)", 2021-03-15T22:35:47.875 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:33:00.391093+0000", 2021-03-15T22:35:47.875 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 167.30454701100001, 2021-03-15T22:35:47.875 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.050022617999999998, 2021-03-15T22:35:47.875 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.875 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:47.876 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.876 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.876 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.876 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7843 2021-03-15T22:35:47.876 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.877 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.877 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.877 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.877 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.391093+0000", 2021-03-15T22:35:47.877 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.878 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.878 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.878 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.878 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.391093+0000", 2021-03-15T22:35:47.879 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00055933000000000003 2021-03-15T22:35:47.879 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.879 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.879 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.879 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.391652+0000", 2021-03-15T22:35:47.880 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5839999999999999e-06 2021-03-15T22:35:47.880 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.880 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.880 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.880 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.391656+0000", 2021-03-15T22:35:47.881 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4188999999999999e-05 2021-03-15T22:35:47.881 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.881 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.881 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.881 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.391670+0000", 2021-03-15T22:35:47.882 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7139999999999998e-06 2021-03-15T22:35:47.882 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.882 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.882 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.883 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.391676+0000", 2021-03-15T22:35:47.883 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.9209000000000003e-05 2021-03-15T22:35:47.883 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.883 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.883 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.884 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.391725+0000", 2021-03-15T22:35:47.884 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5439000000000006e-05 2021-03-15T22:35:47.884 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.884 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.884 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.885 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.391811+0000", 2021-03-15T22:35:47.885 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00016879699999999999 2021-03-15T22:35:47.885 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.885 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.886 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:35:47.886 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.391979+0000", 2021-03-15T22:35:47.886 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001538085 2021-03-15T22:35:47.886 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.886 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.887 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:47.887 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.393518+0000", 2021-03-15T22:35:47.887 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.046364515000000002 2021-03-15T22:35:47.887 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.887 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.888 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.888 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.439882+0000", 2021-03-15T22:35:47.888 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0011277329999999999 2021-03-15T22:35:47.888 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.888 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.889 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.889 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.441010+0000", 2021-03-15T22:35:47.889 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.5376999999999994e-05 2021-03-15T22:35:47.889 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.889 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.890 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:47.890 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.441075+0000", 2021-03-15T22:35:47.890 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.0645999999999997e-05 2021-03-15T22:35:47.890 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.891 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.891 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.891 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.441116+0000", 2021-03-15T22:35:47.891 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.050022617999999998 2021-03-15T22:35:47.891 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.892 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.892 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.892 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.892 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.892 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:7844 7.5 7:ac84e656:::smithi07694878-31:head [write 1863099~719692] snapc 2b6=[2b6,2b5,2b4,2b3,2b1,2af,2ad,2ac,2ab,2a7,2a2,29f,29e,29c,296,293,291,28d,285,283,27f,27e,27d,276,26e,266,265,24c,24b,239,218] ondisk+write+known_if_redirected e830)", 2021-03-15T22:35:47.893 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:33:00.398688+0000", 2021-03-15T22:35:47.893 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 167.29695265300001, 2021-03-15T22:35:47.893 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042594346999999998, 2021-03-15T22:35:47.893 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.893 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:47.894 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.894 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.894 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.894 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 7844 2021-03-15T22:35:47.894 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.895 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.895 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.895 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.895 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.398688+0000", 2021-03-15T22:35:47.895 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.896 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.896 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.896 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.896 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.398688+0000", 2021-03-15T22:35:47.897 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00053087999999999996 2021-03-15T22:35:47.897 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.897 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.897 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.897 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.399218+0000", 2021-03-15T22:35:47.898 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.399e-06 2021-03-15T22:35:47.898 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.898 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.898 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.898 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.399222+0000", 2021-03-15T22:35:47.899 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4149e-05 2021-03-15T22:35:47.899 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.899 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.899 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.900 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.399236+0000", 2021-03-15T22:35:47.900 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.4219999999999997e-06 2021-03-15T22:35:47.900 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.900 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.900 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.901 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.399242+0000", 2021-03-15T22:35:47.901 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1807999999999997e-05 2021-03-15T22:35:47.901 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.901 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.901 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.902 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.399284+0000", 2021-03-15T22:35:47.902 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7892999999999994e-05 2021-03-15T22:35:47.902 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.902 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.902 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.903 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.399362+0000", 2021-03-15T22:35:47.903 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000132254 2021-03-15T22:35:47.903 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.903 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.903 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:35:47.904 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.399494+0000", 2021-03-15T22:35:47.904 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00127155 2021-03-15T22:35:47.904 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.904 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.904 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:47.905 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.400766+0000", 2021-03-15T22:35:47.905 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.040116950999999998 2021-03-15T22:35:47.905 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.905 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.905 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.906 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.440883+0000", 2021-03-15T22:35:47.906 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000285025 2021-03-15T22:35:47.906 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.906 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.907 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.907 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.441168+0000", 2021-03-15T22:35:47.907 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1362e-05 2021-03-15T22:35:47.907 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.907 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.908 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:47.908 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.441229+0000", 2021-03-15T22:35:47.908 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2654000000000001e-05 2021-03-15T22:35:47.908 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.908 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.909 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.909 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:00.441282+0000", 2021-03-15T22:35:47.909 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.042594346999999998 2021-03-15T22:35:47.909 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.910 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.910 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.910 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.910 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.910 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:8336 7.8 7:1ed4caeb:::smithi07694878-27:head [rollback 2e0] snapc 2e4=[2e4,2e2,2e1,2e0,2df,2de,2dd,2dc,2d9,2d5,2d4,2d3,2d1,2cb,2c7,2c3,2c2,2c0,2be,2bb,2b7,2b5,2b4,2b3,2b1,2ac,2ab,2a7,29f,296,291,28d,285,266,265,24b,239,218] ondisk+write+known_if_redirected e880)", 2021-03-15T22:35:47.911 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:33:50.591905+0000", 2021-03-15T22:35:47.911 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 117.103735324, 2021-03-15T22:35:47.911 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.421775641, 2021-03-15T22:35:47.911 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.911 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:47.912 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.912 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.912 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.912 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 8336 2021-03-15T22:35:47.913 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.913 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.913 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.913 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.913 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591905+0000", 2021-03-15T22:35:47.914 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.914 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.914 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.914 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.915 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591905+0000", 2021-03-15T22:35:47.915 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3120000000000001e-06 2021-03-15T22:35:47.915 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.915 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.915 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.916 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591906+0000", 2021-03-15T22:35:47.916 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5340000000000001e-06 2021-03-15T22:35:47.916 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.916 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.916 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.917 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591908+0000", 2021-03-15T22:35:47.917 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.9840000000000004e-06 2021-03-15T22:35:47.917 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.917 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.917 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.918 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591915+0000", 2021-03-15T22:35:47.918 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.6e-06 2021-03-15T22:35:47.918 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.918 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.919 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.919 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591919+0000", 2021-03-15T22:35:47.919 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0014086179999999999 2021-03-15T22:35:47.919 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.919 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.920 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.920 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.593328+0000", 2021-03-15T22:35:47.920 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000151018 2021-03-15T22:35:47.920 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.920 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.921 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.921 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.593479+0000", 2021-03-15T22:35:47.921 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000190538 2021-03-15T22:35:47.921 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.922 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.922 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-15T22:35:47.922 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.593669+0000", 2021-03-15T22:35:47.922 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4177680050000001 2021-03-15T22:35:47.922 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.923 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.923 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.923 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.011437+0000", 2021-03-15T22:35:47.923 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00012805 2021-03-15T22:35:47.923 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.924 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.924 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.924 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.011566+0000", 2021-03-15T22:35:47.924 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00038056599999999999 2021-03-15T22:35:47.924 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.925 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.925 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,4", 2021-03-15T22:35:47.925 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.011946+0000", 2021-03-15T22:35:47.925 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001341063 2021-03-15T22:35:47.926 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.926 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.926 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:47.926 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.013287+0000", 2021-03-15T22:35:47.926 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.0550999999999998e-05 2021-03-15T22:35:47.927 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.927 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.927 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.927 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.013338+0000", 2021-03-15T22:35:47.927 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00019508899999999999 2021-03-15T22:35:47.928 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.928 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.928 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.928 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.013533+0000", 2021-03-15T22:35:47.929 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.2493999999999997e-05 2021-03-15T22:35:47.929 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.929 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.929 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:47.929 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.013575+0000", 2021-03-15T22:35:47.930 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000105219 2021-03-15T22:35:47.930 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.930 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.930 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.930 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.013680+0000", 2021-03-15T22:35:47.931 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.421775641 2021-03-15T22:35:47.931 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.931 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.931 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.931 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.932 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.932 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:8337 7.8 7:1ed4caeb:::smithi07694878-27:head [append 0~0] snapc 2e4=[2e4,2e2,2e1,2e0,2df,2de,2dd,2dc,2d9,2d5,2d4,2d3,2d1,2cb,2c7,2c3,2c2,2c0,2be,2bb,2b7,2b5,2b4,2b3,2b1,2ac,2ab,2a7,29f,296,291,28d,285,266,265,24b,239,218] ondisk+write+known_if_redirected e880)", 2021-03-15T22:35:47.932 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:33:50.591934+0000", 2021-03-15T22:35:47.932 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 117.103706496, 2021-03-15T22:35:47.932 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4219534949999999, 2021-03-15T22:35:47.933 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.933 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:47.933 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.933 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.933 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.934 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 8337 2021-03-15T22:35:47.934 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.934 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.934 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.935 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.935 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591934+0000", 2021-03-15T22:35:47.935 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.935 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.935 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.936 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.936 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591934+0000", 2021-03-15T22:35:47.936 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.271e-06 2021-03-15T22:35:47.936 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.937 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.937 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.937 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591935+0000", 2021-03-15T22:35:47.937 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.266e-06 2021-03-15T22:35:47.937 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.938 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.938 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.938 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591936+0000", 2021-03-15T22:35:47.938 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.2140000000000001e-06 2021-03-15T22:35:47.938 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.939 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.939 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.939 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591942+0000", 2021-03-15T22:35:47.939 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9299999999999996e-06 2021-03-15T22:35:47.940 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.940 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.940 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.940 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.591946+0000", 2021-03-15T22:35:47.940 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.001785621 2021-03-15T22:35:47.941 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.941 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.941 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.941 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.593732+0000", 2021-03-15T22:35:47.941 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000121068 2021-03-15T22:35:47.942 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.942 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.942 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for degraded object", 2021-03-15T22:35:47.942 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:50.593853+0000", 2021-03-15T22:35:47.942 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.41880666 2021-03-15T22:35:47.943 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.943 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.943 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.943 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.012660+0000", 2021-03-15T22:35:47.943 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.3078000000000007e-05 2021-03-15T22:35:47.944 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.944 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.944 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.944 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.012743+0000", 2021-03-15T22:35:47.945 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00017567099999999999 2021-03-15T22:35:47.945 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.945 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.945 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 3,4", 2021-03-15T22:35:47.945 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.012918+0000", 2021-03-15T22:35:47.946 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00052855100000000004 2021-03-15T22:35:47.946 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.946 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.946 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:47.946 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.013447+0000", 2021-03-15T22:35:47.947 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00029445600000000001 2021-03-15T22:35:47.947 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.947 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.947 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.947 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.013741+0000", 2021-03-15T22:35:47.948 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.0086e-05 2021-03-15T22:35:47.948 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.948 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.948 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.949 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.013772+0000", 2021-03-15T22:35:47.949 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.7174e-05 2021-03-15T22:35:47.949 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.949 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.949 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:47.950 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.013809+0000", 2021-03-15T22:35:47.950 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.8448999999999994e-05 2021-03-15T22:35:47.950 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.950 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.950 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.951 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:33:52.013887+0000", 2021-03-15T22:35:47.951 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.4219534949999999 2021-03-15T22:35:47.951 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.951 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.951 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.952 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.952 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.952 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:8473 7.5 7:a9545d62:::smithi07694878-1:head [copy-from ver 2000] snapc 2f6=[2f6,2f5,2f3,2f2,2ef,2ee,2ed,2ec,2eb,2ea,2e8,2e4,2e2,2e1,2e0,2df,2de,2dd,2dc,2d9,2d4,2d1,2c3,2c2,2c0,2be,2bb,2b7,2b5,2b4,2b3,2b1,2ac,2a7,29f,296,28d,285,266,265,24b,239,218] ondisk+write+known_if_redirected e900)", 2021-03-15T22:35:47.952 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:34:15.480988+0000", 2021-03-15T22:35:47.952 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 92.214652173000005, 2021-03-15T22:35:47.953 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.13274593200000001, 2021-03-15T22:35:47.953 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.953 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:47.953 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.954 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.954 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.954 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 8473 2021-03-15T22:35:47.954 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.954 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.955 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.955 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.955 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.480988+0000", 2021-03-15T22:35:47.955 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.955 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.956 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.956 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.956 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.480988+0000", 2021-03-15T22:35:47.956 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.316e-06 2021-03-15T22:35:47.956 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.957 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.957 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.957 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.480989+0000", 2021-03-15T22:35:47.957 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.1500000000000003e-07 2021-03-15T22:35:47.957 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.958 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.958 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.958 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.480990+0000", 2021-03-15T22:35:47.958 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.444e-06 2021-03-15T22:35:47.958 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.959 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.959 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.959 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.480994+0000", 2021-03-15T22:35:47.959 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.2010000000000002e-06 2021-03-15T22:35:47.959 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.960 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.960 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.960 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.480996+0000", 2021-03-15T22:35:47.960 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0689000000000001e-05 2021-03-15T22:35:47.960 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.961 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.961 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.961 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.481056+0000", 2021-03-15T22:35:47.961 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0214000000000001e-05 2021-03-15T22:35:47.961 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.962 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.962 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op must wait for map", 2021-03-15T22:35:47.962 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.481067+0000", 2021-03-15T22:35:47.962 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019559342 2021-03-15T22:35:47.963 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.963 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.963 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.963 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.500626+0000", 2021-03-15T22:35:47.963 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00010205900000000001 2021-03-15T22:35:47.964 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.964 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.964 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.964 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.500728+0000", 2021-03-15T22:35:47.964 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0094230760000000007 2021-03-15T22:35:47.965 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.965 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.965 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,1", 2021-03-15T22:35:47.965 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.510151+0000", 2021-03-15T22:35:47.965 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0045897059999999998 2021-03-15T22:35:47.966 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.966 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.966 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:47.966 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.514741+0000", 2021-03-15T22:35:47.966 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.081490919999999994 2021-03-15T22:35:47.967 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.967 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.967 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.967 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.596232+0000", 2021-03-15T22:35:47.967 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.017411768000000001 2021-03-15T22:35:47.968 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.968 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.968 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:47.968 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.613644+0000", 2021-03-15T22:35:47.968 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1959000000000003e-05 2021-03-15T22:35:47.969 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.969 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.969 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:47.969 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.613686+0000", 2021-03-15T22:35:47.970 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8423e-05 2021-03-15T22:35:47.970 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.970 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.970 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.970 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.613734+0000", 2021-03-15T22:35:47.971 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.13274593200000001 2021-03-15T22:35:47.971 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.971 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.971 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.971 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.972 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.972 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:8474 7.5 7:a9545d62:::smithi07694878-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected e900)", 2021-03-15T22:35:47.972 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:34:15.481005+0000", 2021-03-15T22:35:47.972 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 92.214635337000004, 2021-03-15T22:35:47.972 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.132881206, 2021-03-15T22:35:47.973 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.973 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:35:47.973 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.973 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.973 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.974 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 8474 2021-03-15T22:35:47.974 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.974 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.974 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.975 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.975 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.481005+0000", 2021-03-15T22:35:47.975 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.975 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.975 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.975 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.976 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.481005+0000", 2021-03-15T22:35:47.976 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.0299999999999997e-07 2021-03-15T22:35:47.976 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.976 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.976 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.977 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.481006+0000", 2021-03-15T22:35:47.977 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3909999999999999e-06 2021-03-15T22:35:47.977 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.977 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.977 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.978 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.481007+0000", 2021-03-15T22:35:47.978 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.4890000000000003e-06 2021-03-15T22:35:47.978 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.978 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.978 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.979 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.481013+0000", 2021-03-15T22:35:47.979 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.8500000000000004e-06 2021-03-15T22:35:47.979 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.979 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.980 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.980 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.481016+0000", 2021-03-15T22:35:47.980 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2484000000000006e-05 2021-03-15T22:35:47.980 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.981 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.981 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.981 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.481089+0000", 2021-03-15T22:35:47.981 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.712e-06 2021-03-15T22:35:47.981 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.982 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.982 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting_for_map not empty", 2021-03-15T22:35:47.982 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.481097+0000", 2021-03-15T22:35:47.982 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.019772792000000001 2021-03-15T22:35:47.982 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.983 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.983 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.983 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.500869+0000", 2021-03-15T22:35:47.983 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2413000000000001e-05 2021-03-15T22:35:47.983 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.984 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.984 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for blocked object", 2021-03-15T22:35:47.984 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.500922+0000", 2021-03-15T22:35:47.984 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.01293161 2021-03-15T22:35:47.984 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.985 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.985 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.985 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.513853+0000", 2021-03-15T22:35:47.985 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.8745000000000004e-05 2021-03-15T22:35:47.985 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.986 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.986 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:47.986 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.513952+0000", 2021-03-15T22:35:47.986 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.099814105 2021-03-15T22:35:47.987 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.987 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.987 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:47.987 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.613766+0000", 2021-03-15T22:35:47.987 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.1896000000000005e-05 2021-03-15T22:35:47.988 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.988 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.988 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:47.988 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.613828+0000", 2021-03-15T22:35:47.988 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.7815999999999997e-05 2021-03-15T22:35:47.989 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.989 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.989 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:47.989 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:34:15.613886+0000", 2021-03-15T22:35:47.990 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.132881206 2021-03-15T22:35:47.990 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.990 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:47.990 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:47.990 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.991 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.991 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:8982 7.5 7:a9545d62:::smithi07694878-1:head [write 487733~745011,stat] snapc 325=[325,324,31f,31d,31c,318,314,313,312,30d,30b,30a,300,2fb,2fa,2f9,2f8,2f7,2f5,2f3,2ed,2ec,2eb,2e8,2e4,2e2,2df,2dc,2d1,2c3,2c2,2c0,2be,2bb,2b7,2b5,2b4,2b1,296,28d,266,239] ondisk+write+known_if_redirected e947)", 2021-03-15T22:35:47.991 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:35:02.346191+0000", 2021-03-15T22:35:47.991 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 45.349449024000002, 2021-03-15T22:35:47.992 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0086307080000001, 2021-03-15T22:35:47.992 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:47.992 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:47.992 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:47.992 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:47.993 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:47.993 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 8982 2021-03-15T22:35:47.993 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.993 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:47.993 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.994 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:47.994 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.346191+0000", 2021-03-15T22:35:47.994 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:47.994 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.994 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.995 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:47.995 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.346191+0000", 2021-03-15T22:35:47.995 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00075914799999999996 2021-03-15T22:35:47.995 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.996 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.996 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:47.996 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.346950+0000", 2021-03-15T22:35:47.996 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.9110000000000002e-06 2021-03-15T22:35:47.996 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.997 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.997 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:47.997 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.346953+0000", 2021-03-15T22:35:47.997 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.3988000000000001e-05 2021-03-15T22:35:47.997 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.998 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.998 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:47.998 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.346967+0000", 2021-03-15T22:35:47.998 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.857e-06 2021-03-15T22:35:47.999 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:47.999 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:47.999 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:47.999 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.346973+0000", 2021-03-15T22:35:47.999 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5756000000000001e-05 2021-03-15T22:35:48.000 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.000 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.000 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.000 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.347009+0000", 2021-03-15T22:35:48.000 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.0619999999999997e-06 2021-03-15T22:35:48.001 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.001 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.001 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:35:48.001 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.347015+0000", 2021-03-15T22:35:48.001 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.000930469 2021-03-15T22:35:48.002 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.002 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.002 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.002 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.347945+0000", 2021-03-15T22:35:48.003 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0001772 2021-03-15T22:35:48.003 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.003 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.003 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:48.003 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.348123+0000", 2021-03-15T22:35:48.004 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00021221400000000001 2021-03-15T22:35:48.004 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.004 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.004 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:35:48.005 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.348335+0000", 2021-03-15T22:35:48.005 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0044343680000000002 2021-03-15T22:35:48.005 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.005 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.005 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:48.006 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.352769+0000", 2021-03-15T22:35:48.006 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0010352219999999999 2021-03-15T22:35:48.006 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.006 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.006 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:48.007 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.353804+0000", 2021-03-15T22:35:48.007 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00087557100000000001 2021-03-15T22:35:48.007 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.007 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.007 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:48.008 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.354680+0000", 2021-03-15T22:35:48.008 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.5532999999999999e-05 2021-03-15T22:35:48.008 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.008 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.008 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:48.009 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.354725+0000", 2021-03-15T22:35:48.009 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.6409000000000005e-05 2021-03-15T22:35:48.009 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.009 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.009 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:48.010 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.354822+0000", 2021-03-15T22:35:48.010 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0086307080000001 2021-03-15T22:35:48.010 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:48.010 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:48.011 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:48.011 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.011 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.011 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:8983 7.5 7:a9545d62:::smithi07694878-1:head [write 1632844~560429] snapc 325=[325,324,31f,31d,31c,318,314,313,312,30d,30b,30a,300,2fb,2fa,2f9,2f8,2f7,2f5,2f3,2ed,2ec,2eb,2e8,2e4,2e2,2df,2dc,2d1,2c3,2c2,2c0,2be,2bb,2b7,2b5,2b4,2b1,296,28d,266,239] ondisk+write+known_if_redirected e947)", 2021-03-15T22:35:48.011 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:35:02.351489+0000", 2021-03-15T22:35:48.012 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 45.344151656000001, 2021-03-15T22:35:48.012 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0084808780000001, 2021-03-15T22:35:48.012 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:48.012 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:48.012 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:48.013 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:48.013 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:48.013 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 8983 2021-03-15T22:35:48.013 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.013 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:48.014 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.014 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:48.014 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351489+0000", 2021-03-15T22:35:48.014 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:48.015 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.015 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.015 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:48.015 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351489+0000", 2021-03-15T22:35:48.015 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00046598499999999998 2021-03-15T22:35:48.016 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.016 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.016 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:48.016 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351954+0000", 2021-03-15T22:35:48.016 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.3810000000000002e-06 2021-03-15T22:35:48.017 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.017 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.017 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:48.017 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351957+0000", 2021-03-15T22:35:48.017 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.2509e-05 2021-03-15T22:35:48.018 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.018 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.018 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:48.018 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351969+0000", 2021-03-15T22:35:48.018 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.8659999999999997e-06 2021-03-15T22:35:48.019 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.019 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.019 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:48.019 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351974+0000", 2021-03-15T22:35:48.020 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5938e-05 2021-03-15T22:35:48.020 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.020 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.020 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.020 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352010+0000", 2021-03-15T22:35:48.021 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.9419999999999995e-06 2021-03-15T22:35:48.021 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.021 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.021 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:35:48.022 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352018+0000", 2021-03-15T22:35:48.022 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99762336200000001 2021-03-15T22:35:48.022 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.022 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.022 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.023 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.349641+0000", 2021-03-15T22:35:48.023 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00013692899999999999 2021-03-15T22:35:48.023 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.023 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.023 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:48.024 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.349778+0000", 2021-03-15T22:35:48.024 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0051077279999999997 2021-03-15T22:35:48.024 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.024 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.025 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.025 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.354886+0000", 2021-03-15T22:35:48.025 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2477000000000004e-05 2021-03-15T22:35:48.025 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.025 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.026 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:48.026 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.354959+0000", 2021-03-15T22:35:48.026 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000246736 2021-03-15T22:35:48.026 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.026 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.027 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:35:48.027 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.355205+0000", 2021-03-15T22:35:48.027 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0023543079999999998 2021-03-15T22:35:48.027 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.028 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.028 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:48.028 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.357560+0000", 2021-03-15T22:35:48.028 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0020502760000000002 2021-03-15T22:35:48.028 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.029 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.029 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:48.029 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.359610+0000", 2021-03-15T22:35:48.029 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000285662 2021-03-15T22:35:48.029 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.030 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.030 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:48.030 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.359896+0000", 2021-03-15T22:35:48.030 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 4.1893e-05 2021-03-15T22:35:48.030 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.031 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.031 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:48.031 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.359938+0000", 2021-03-15T22:35:48.031 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.1885999999999999e-05 2021-03-15T22:35:48.031 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.032 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.032 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:48.032 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.359969+0000", 2021-03-15T22:35:48.032 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.0084808780000001 2021-03-15T22:35:48.032 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:48.033 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:48.033 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:48.033 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.033 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.034 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:8984 7.5 7:a9545d62:::smithi07694878-1:head [setxattr _header (57) in=7b,truncate 2487733] snapc 325=[325,324,31f,31d,31c,318,314,313,312,30d,30b,30a,300,2fb,2fa,2f9,2f8,2f7,2f5,2f3,2ed,2ec,2eb,2e8,2e4,2e2,2df,2dc,2d1,2c3,2c2,2c0,2be,2bb,2b7,2b5,2b4,2b1,296,28d,266,239] ondisk+write+known_if_redirected e947)", 2021-03-15T22:35:48.034 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:35:02.351983+0000", 2021-03-15T22:35:48.034 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 45.343657112999999, 2021-03-15T22:35:48.034 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.008123986, 2021-03-15T22:35:48.034 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:48.035 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "commit sent; apply or cleanup", 2021-03-15T22:35:48.035 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:48.035 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:48.035 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:48.036 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 8984 2021-03-15T22:35:48.036 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.036 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:48.036 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.036 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:48.037 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351983+0000", 2021-03-15T22:35:48.037 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:48.037 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.037 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.037 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:48.038 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351983+0000", 2021-03-15T22:35:48.038 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.5579999999999999e-06 2021-03-15T22:35:48.038 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.038 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.039 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:48.039 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351985+0000", 2021-03-15T22:35:48.039 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.7199999999999997e-07 2021-03-15T22:35:48.039 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.039 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.040 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:48.040 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351985+0000", 2021-03-15T22:35:48.040 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9020000000000001e-06 2021-03-15T22:35:48.040 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.041 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.041 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:48.041 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351989+0000", 2021-03-15T22:35:48.041 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.5059999999999998e-06 2021-03-15T22:35:48.041 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.042 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.042 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:48.042 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.351992+0000", 2021-03-15T22:35:48.042 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6172000000000003e-05 2021-03-15T22:35:48.042 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.043 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.043 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.043 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352058+0000", 2021-03-15T22:35:48.043 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.8370000000000002e-06 2021-03-15T22:35:48.044 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.044 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.044 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:35:48.044 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352064+0000", 2021-03-15T22:35:48.044 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99776494999999998 2021-03-15T22:35:48.045 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.045 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.045 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.045 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.349829+0000", 2021-03-15T22:35:48.045 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000105473 2021-03-15T22:35:48.046 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.046 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.046 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:48.046 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.349934+0000", 2021-03-15T22:35:48.047 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0066396119999999996 2021-03-15T22:35:48.047 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.047 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.047 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.047 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.356574+0000", 2021-03-15T22:35:48.048 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.5977999999999996e-05 2021-03-15T22:35:48.048 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.048 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.048 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:48.048 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.356660+0000", 2021-03-15T22:35:48.049 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000153547 2021-03-15T22:35:48.049 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.049 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.049 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for subops from 0,2", 2021-03-15T22:35:48.050 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.356813+0000", 2021-03-15T22:35:48.050 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.00099653599999999991 2021-03-15T22:35:48.050 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.050 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.050 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "op_commit", 2021-03-15T22:35:48.051 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.357810+0000", 2021-03-15T22:35:48.051 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0021907599999999999 2021-03-15T22:35:48.051 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.051 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.051 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:48.052 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.360001+0000", 2021-03-15T22:35:48.052 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5145000000000003e-05 2021-03-15T22:35:48.052 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.052 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.052 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "sub_op_commit_rec", 2021-03-15T22:35:48.053 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.360036+0000", 2021-03-15T22:35:48.053 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.6040000000000001e-05 2021-03-15T22:35:48.053 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.053 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.053 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "commit_sent", 2021-03-15T22:35:48.054 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.360072+0000", 2021-03-15T22:35:48.054 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.5098000000000002e-05 2021-03-15T22:35:48.054 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.054 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.055 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:48.055 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.360107+0000", 2021-03-15T22:35:48.055 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.008123986 2021-03-15T22:35:48.055 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:48.055 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:48.056 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:48.056 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.056 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.056 INFO:teuthology.orchestra.run.smithi125.stdout: "description": "osd_op(client.24517.0:8985 7.5 7:a9545d62:::smithi07694878-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected e947)", 2021-03-15T22:35:48.056 INFO:teuthology.orchestra.run.smithi125.stdout: "initiated_at": "2021-03-15T22:35:02.352006+0000", 2021-03-15T22:35:48.057 INFO:teuthology.orchestra.run.smithi125.stdout: "age": 45.343634367, 2021-03-15T22:35:48.057 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.008255074, 2021-03-15T22:35:48.057 INFO:teuthology.orchestra.run.smithi125.stdout: "type_data": { 2021-03-15T22:35:48.057 INFO:teuthology.orchestra.run.smithi125.stdout: "flag_point": "started", 2021-03-15T22:35:48.057 INFO:teuthology.orchestra.run.smithi125.stdout: "client_info": { 2021-03-15T22:35:48.058 INFO:teuthology.orchestra.run.smithi125.stdout: "client": "client.24517", 2021-03-15T22:35:48.058 INFO:teuthology.orchestra.run.smithi125.stdout: "client_addr": "172.21.15.76:0/2794982104", 2021-03-15T22:35:48.058 INFO:teuthology.orchestra.run.smithi125.stdout: "tid": 8985 2021-03-15T22:35:48.058 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.059 INFO:teuthology.orchestra.run.smithi125.stdout: "events": [ 2021-03-15T22:35:48.059 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.059 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "initiated", 2021-03-15T22:35:48.059 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352006+0000", 2021-03-15T22:35:48.059 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0 2021-03-15T22:35:48.059 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.060 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.060 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "throttled", 2021-03-15T22:35:48.060 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352006+0000", 2021-03-15T22:35:48.060 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.6199999999999997e-07 2021-03-15T22:35:48.060 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.061 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.061 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "header_read", 2021-03-15T22:35:48.061 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352006+0000", 2021-03-15T22:35:48.061 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 8.6700000000000002e-07 2021-03-15T22:35:48.061 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.062 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.062 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "all_read", 2021-03-15T22:35:48.062 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352007+0000", 2021-03-15T22:35:48.062 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 3.9910000000000004e-06 2021-03-15T22:35:48.062 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.063 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.063 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "dispatched", 2021-03-15T22:35:48.063 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352011+0000", 2021-03-15T22:35:48.063 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 2.4310000000000001e-06 2021-03-15T22:35:48.063 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.064 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.064 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "queued_for_pg", 2021-03-15T22:35:48.064 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352014+0000", 2021-03-15T22:35:48.064 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.2838999999999998e-05 2021-03-15T22:35:48.064 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.065 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.065 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.065 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352087+0000", 2021-03-15T22:35:48.065 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 7.7400000000000004e-06 2021-03-15T22:35:48.065 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.066 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.066 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for peered", 2021-03-15T22:35:48.066 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:02.352094+0000", 2021-03-15T22:35:48.066 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.99786613899999999 2021-03-15T22:35:48.067 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.067 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.067 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.067 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.349960+0000", 2021-03-15T22:35:48.067 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.000150791 2021-03-15T22:35:48.068 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.068 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.068 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:48.068 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.350111+0000", 2021-03-15T22:35:48.068 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0071127070000000002 2021-03-15T22:35:48.069 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.069 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.069 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.069 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.357224+0000", 2021-03-15T22:35:48.069 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 9.9820000000000003e-05 2021-03-15T22:35:48.070 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.070 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.070 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "waiting for rw locks", 2021-03-15T22:35:48.070 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.357324+0000", 2021-03-15T22:35:48.071 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 0.0028170309999999998 2021-03-15T22:35:48.071 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.071 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.071 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "reached_pg", 2021-03-15T22:35:48.071 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.360141+0000", 2021-03-15T22:35:48.072 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 5.2716999999999997e-05 2021-03-15T22:35:48.072 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.072 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.072 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "started", 2021-03-15T22:35:48.072 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.360194+0000", 2021-03-15T22:35:48.073 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 6.7339e-05 2021-03-15T22:35:48.073 INFO:teuthology.orchestra.run.smithi125.stdout: }, 2021-03-15T22:35:48.073 INFO:teuthology.orchestra.run.smithi125.stdout: { 2021-03-15T22:35:48.073 INFO:teuthology.orchestra.run.smithi125.stdout: "event": "done", 2021-03-15T22:35:48.073 INFO:teuthology.orchestra.run.smithi125.stdout: "time": "2021-03-15T22:35:03.360261+0000", 2021-03-15T22:35:48.074 INFO:teuthology.orchestra.run.smithi125.stdout: "duration": 1.008255074 2021-03-15T22:35:48.074 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:48.074 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:48.074 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:48.074 INFO:teuthology.orchestra.run.smithi125.stdout: } 2021-03-15T22:35:48.075 INFO:teuthology.orchestra.run.smithi125.stdout: ] 2021-03-15T22:35:48.075 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:35:48.075 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:35:48.076 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:48.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[56514]: cluster 2021-03-15T22:35:45.757249+0000 osd.6 (osd.6) 150 2021-03-15T22:35:48.077 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[56514]: : cluster [DBG] 3.10 deep-scrub ok 2021-03-15T22:35:48.078 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[56514]: cluster 2021-03-15T22:35:46.406367+0000 osd.4 (osd.4) 198 : cluster [DBG] 4.c scrub ok 2021-03-15T22:35:48.078 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[56514]: audit 2021-03-15T22:35:46.662288+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? 172.21.15.76:0/648468348' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]: dispatch 2021-03-15T22:35:48.078 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[56514]: audit 2021-03-15T22:35:47.345663+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? 172.21.15.76:0/648468348' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]': finished 2021-03-15T22:35:48.078 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[56514]: cluster 2021-03-15T22:35:47.345716+0000 mon.a (mon.0) 2468 : cluster [DBG] osdmap e992: 8 total, 8 up, 8 in 2021-03-15T22:35:48.079 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:47 smithi125 conmon[56739]: cluster 2021-03-15T22:35:45.757249+0000 osd.6 ( 2021-03-15T22:35:48.079 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:47 smithi125 conmon[56739]: osd.6) 150 : cluster [DBG] 3.10 deep-scrub ok 2021-03-15T22:35:48.079 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:47 smithi125 conmon[56739]: cluster 2021-03-15T22:35:46.406367+0000 osd.4 (osd.4) 198 : cluster [DBG] 4.c scrub ok 2021-03-15T22:35:48.080 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:47 smithi125 conmon[56739]: audit 2021-03-15T22:35:46.662288+0000 mon.a (mon.0) 2466 : audit [INF] from='client.? 172.21.15.76:0/648468348' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]: dispatch 2021-03-15T22:35:48.080 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:47 smithi125 conmon[56739]: audit 2021-03-15T22:35:47.345663+0000 mon.a (mon.0) 2467 : audit [INF] from='client.? 172.21.15.76:0/648468348' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]': finished 2021-03-15T22:35:48.080 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:47 smithi125 conmon[56739]: cluster 2021-03-15T22:35:47.345716+0000 mon.a (mon.0) 2468 : cluster 2021-03-15T22:35:48.080 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:47 smithi125 conmon[56739]: [DBG] osdmap e992: 8 total, 8 up, 8 in 2021-03-15T22:35:48.081 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: cluster 2021-03-15T22:35:45.757249+0000 osd.6 ( 2021-03-15T22:35:48.081 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: osd.6) 150 : cluster [DBG] 3.10 deep-scrub ok 2021-03-15T22:35:48.081 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: cluster 2021-03-15T22:35:46.406367+0000 2021-03-15T22:35:48.081 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: osd.4 (osd.4) 198 : cluster [DBG] 4.c scrub ok 2021-03-15T22:35:48.082 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:35:48.082 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: :35:46.662288+0000 mon.a (mon.0 2021-03-15T22:35:48.082 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: ) 2466 : audit [INF] from='client.? 172.21.15.76:0/648468348' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]: dispatch 2021-03-15T22:35:48.082 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:35:48.083 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: -15T22:35:47.345663+0000 mon.a 2021-03-15T22:35:48.083 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: (mon.0) 2467 : audit [INF] from='client.? 172.21.15.76:0/648468348' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num", "val": "22"}]': finished 2021-03-15T22:35:48.083 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: 2021-03-15T22:35:48.083 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: cluster 2021-03-15T22:35:48.083 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: 2021-03-15T22:35:47.345716+0000 2021-03-15T22:35:48.084 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: mon.a (mon.0) 2468 : cluster [DBG] 2021-03-15T22:35:48.084 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:47 smithi076 conmon[62156]: osdmap e992: 8 total, 8 up, 8 in 2021-03-15T22:35:48.102 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:48.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:48.103 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:48.204 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:48.226 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:48.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:48.227 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:48.296 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_ops_in_flight 2021-03-15T22:35:48.328 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:48.352 INFO:tasks.rados.rados.0.smithi076.stdout:3812: done (0 left) 2021-03-15T22:35:48.352 INFO:tasks.rados.rados.0.smithi076.stdout:3813: copy_from oid 9 from oid 13 current snap is 447 2021-03-15T22:35:48.353 INFO:tasks.rados.rados.0.smithi076.stdout:3814: snap_create 2021-03-15T22:35:48.357 INFO:tasks.rados.rados.0.smithi076.stdout:3813: finishing copy_from to smithi07694878-9 2021-03-15T22:35:48.357 INFO:tasks.rados.rados.0.smithi076.stdout:3813: got expected ENOENT (src dne) 2021-03-15T22:35:48.358 INFO:tasks.rados.rados.0.smithi076.stdout:3813: finishing copy_from racing read to smithi07694878-9 2021-03-15T22:35:48.364 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:48.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:48.367 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:48.416 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:48.467 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:48.490 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:48.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:48.491 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:48.580 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:48.592 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:48.615 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:48.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:48.617 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:48.718 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:48.740 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:48.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:48.741 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:48.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:48 smithi125 conmon[56739]: cluster 2021-03-15T22:35:47.126899+0000 osd.1 (osd.1) 187 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:35:48.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:48 smithi125 conmon[56739]: cluster 2021-03-15T22:35:47.225290+0000 mgr.y (mgr.14144) 1091 : cluster [DBG] pgmap v1634: 157 pgs: 157 active+clean; 695 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 5.0 MiB/s wr, 30 op/s 2021-03-15T22:35:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:48 smithi125 conmon[56739]: cluster 2021-03-15T22:35:47.601453+0000 osd.7 (osd.7) 186 : cluster [DBG] 3.e scrub ok 2021-03-15T22:35:48.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:48 smithi125 conmon[56739]: cluster 2021-03-15T22:35:48.349286+0000 mon.a (mon.0) 2469 : cluster [DBG] osdmap e993: 8 total, 8 up, 8 in 2021-03-15T22:35:48.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[56514]: cluster 2021-03-15T22:35:47.126899+0000 2021-03-15T22:35:48.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[56514]: osd.1 (osd.1) 187 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:35:48.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[56514]: cluster 2021-03-15T22:35:47.225290+0000 mgr.y (mgr.14144) 1091 : cluster [DBG] pgmap v1634: 157 pgs: 157 active+clean; 695 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 5.0 MiB/s wr, 30 op/s 2021-03-15T22:35:48.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[56514]: cluster 2021-03-15T22:35:47.601453+0000 osd.7 (osd.7) 186 : cluster 2021-03-15T22:35:48.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[56514]: [DBG] 3.e scrub ok 2021-03-15T22:35:48.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[56514]: cluster 2021-03-15T22:35:48.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[56514]: 2021-03-15T22:35:48.349286 2021-03-15T22:35:48.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[56514]: +0000 mon.a (mon.0) 2469 2021-03-15T22:35:48.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[56514]: : cluster [DBG] osdmap e993: 8 total, 8 up, 8 in 2021-03-15T22:35:48.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: cluster 2021-03-15T22:35 2021-03-15T22:35:48.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: :47.126899+0000 osd.1 (osd.1) 187 : cluster [DBG] 4.4 scrub ok 2021-03-15T22:35:48.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:35:48.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: 03-15T22:35:47.225290+0000 mgr.y 2021-03-15T22:35:48.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: (mgr.14144) 1091 : cluster 2021-03-15T22:35:48.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: [DBG] pgmap v1634: 157 pgs: 157 active+clean; 695 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 19 MiB/s rd, 5.0 MiB/s wr, 30 op/s 2021-03-15T22:35:48.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: cluster 2021-03-15T22:35:48.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: 2021-03-15T22 2021-03-15T22:35:48.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: :35:47.601453+0000 2021-03-15T22:35:48.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: osd.7 (osd.7) 186 : 2021-03-15T22:35:48.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: cluster [DBG] 3.e scrub ok 2021-03-15T22:35:48.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:35:48.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: 03-15T22:35:48.349286+0000 2021-03-15T22:35:48.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: mon.a (mon.0) 2469 : 2021-03-15T22:35:48.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:48 smithi076 conmon[62156]: cluster [DBG] osdmap e993: 8 total, 8 up, 8 in 2021-03-15T22:35:48.842 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:48.865 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:48.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:48.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:48.967 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:48.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:48.991 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:48.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:49.093 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:49.115 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:49.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:49.117 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:49.218 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:49.240 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:49.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:49.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:49.343 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:49.354 INFO:tasks.rados.rados.0.smithi076.stdout:3813: done (1 left) 2021-03-15T22:35:49.354 INFO:tasks.rados.rados.0.smithi076.stdout:3814: done (0 left) 2021-03-15T22:35:49.354 INFO:tasks.rados.rados.0.smithi076.stdout:3815: copy_from oid 24 from oid 17 current snap is 448 2021-03-15T22:35:49.355 INFO:tasks.rados.rados.0.smithi076.stdout:3816: write (excl) oid 8 current snap is 448 2021-03-15T22:35:49.355 INFO:tasks.rados.rados.0.smithi076.stdout:3816: seq_num 1297 ranges {598679=577963,1947263=638108,3351151=447528} 2021-03-15T22:35:49.358 INFO:tasks.rados.rados.0.smithi076.stdout:3816: writing smithi07694878-8 from 598679 to 1176642 tid 1 2021-03-15T22:35:49.367 INFO:tasks.rados.rados.0.smithi076.stdout:3816: writing smithi07694878-8 from 1947263 to 2585371 tid 2 2021-03-15T22:35:49.371 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:49.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:49.372 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:49.374 INFO:tasks.rados.rados.0.smithi076.stdout:3816: writing smithi07694878-8 from 3351151 to 3798679 tid 3 2021-03-15T22:35:49.374 INFO:tasks.rados.rados.0.smithi076.stdout:3817: delete oid 23 current snap is 448 2021-03-15T22:35:49.375 INFO:tasks.rados.rados.0.smithi076.stdout:3816: finishing write tid 1 to smithi07694878-8 2021-03-15T22:35:49.375 INFO:tasks.rados.rados.0.smithi076.stdout:3816: finishing write tid 2 to smithi07694878-8 2021-03-15T22:35:49.377 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:35:49.382 INFO:tasks.rados.rados.0.smithi076.stdout:3817: done (2 left) 2021-03-15T22:35:49.382 INFO:tasks.rados.rados.0.smithi076.stdout:3818: write oid 46 current snap is 448 2021-03-15T22:35:49.382 INFO:tasks.rados.rados.0.smithi076.stdout:3818: seq_num 1299 ranges {632293=728461,1947874=641777,3005997=686580} 2021-03-15T22:35:49.389 INFO:tasks.rados.rados.0.smithi076.stdout:3818: writing smithi07694878-46 from 632293 to 1360754 tid 1 2021-03-15T22:35:49.396 INFO:tasks.rados.rados.0.smithi076.stdout:3818: writing smithi07694878-46 from 1947874 to 2589651 tid 2 2021-03-15T22:35:49.404 INFO:tasks.rados.rados.0.smithi076.stdout:3818: writing smithi07694878-46 from 3005997 to 3692577 tid 3 2021-03-15T22:35:49.404 INFO:tasks.rados.rados.0.smithi076.stdout:3819: copy_from oid 16 from oid 6 current snap is 448 2021-03-15T22:35:49.404 INFO:tasks.rados.rados.0.smithi076.stdout:3815: finishing copy_from to smithi07694878-24 2021-03-15T22:35:49.405 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 3038 (ObjNum 1177 snap 411 seq_num 1177) dirty exists 2021-03-15T22:35:49.405 INFO:tasks.rados.rados.0.smithi076.stdout:3816: finishing write tid 3 to smithi07694878-8 2021-03-15T22:35:49.405 INFO:tasks.rados.rados.0.smithi076.stdout:3816: finishing write tid 4 to smithi07694878-8 2021-03-15T22:35:49.405 INFO:tasks.rados.rados.0.smithi076.stdout:3815: finishing copy_from racing read to smithi07694878-24 2021-03-15T22:35:49.405 INFO:tasks.rados.rados.0.smithi076.stdout:3816: finishing write tid 5 to smithi07694878-8 2021-03-15T22:35:49.406 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 3163 (ObjNum 1297 snap 448 seq_num 1297) dirty exists 2021-03-15T22:35:49.406 INFO:tasks.rados.rados.0.smithi076.stdout:3816: left oid 8 (ObjNum 1297 snap 448 seq_num 1297) 2021-03-15T22:35:49.406 INFO:tasks.rados.rados.0.smithi076.stdout:3818: finishing write tid 1 to smithi07694878-46 2021-03-15T22:35:49.406 INFO:tasks.rados.rados.0.smithi076.stdout:3818: finishing write tid 2 to smithi07694878-46 2021-03-15T22:35:49.407 INFO:tasks.rados.rados.0.smithi076.stdout:3815: done (3 left) 2021-03-15T22:35:49.407 INFO:tasks.rados.rados.0.smithi076.stdout:3816: done (2 left) 2021-03-15T22:35:49.407 INFO:tasks.rados.rados.0.smithi076.stdout:3820: rollback oid 24 current snap is 448 2021-03-15T22:35:49.407 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 24 to 417 2021-03-15T22:35:49.407 INFO:tasks.rados.rados.0.smithi076.stdout:3821: snap_remove snap 424 2021-03-15T22:35:49.473 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:49.496 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:49.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:49.498 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:49.599 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:49.621 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:49.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:49.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:49.678 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[56514]: cluster 2021-03-15T22:35:47.782089+0000 2021-03-15T22:35:49.678 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[56514]: osd.6 (osd.6) 151 : cluster [DBG] 4.a scrub ok 2021-03-15T22:35:49.679 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[56514]: cluster 2021-03-15T22:35:48.581154+0000 osd.7 (osd.7) 187 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:35:49.679 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[56514]: audit 2021-03-15T22:35:49.226380+0000 mon.a (mon.0) 2470 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "39"}]: dispatch 2021-03-15T22:35:49.679 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[56514]: audit 2021-03-15T22:35:49.226577+0000 mon.a (mon.0) 2471 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-03-15T22:35:49.679 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[56514]: audit 2021-03-15T22:35:49.348923+0000 mon.a (mon.0) 2472 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "39"}]': finished 2021-03-15T22:35:49.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[56514]: audit 2021-03-15T22:35:49.348990+0000 mon.a (mon.0) 2473 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]': finished 2021-03-15T22:35:49.680 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[56514]: cluster 2021-03-15T22:35:49.349046+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e994: 8 total, 8 up, 8 in 2021-03-15T22:35:49.680 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: cluster 2021-03-15T22:35:47.782089+0000 2021-03-15T22:35:49.680 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: osd.6 (osd.6) 151 : cluster [DBG] 4.a scrub ok 2021-03-15T22:35:49.681 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: cluster 2021-03-15T22: 2021-03-15T22:35:49.681 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: 35:48.581154+0000 osd.7 ( 2021-03-15T22:35:49.681 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: osd.7) 187 : cluster [DBG] 2021-03-15T22:35:49.681 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: 3.11 scrub ok 2021-03-15T22:35:49.682 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: audit 2021-03-15T22:35:49.682 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: 2021-03-15T22:35: 2021-03-15T22:35:49.682 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: 49.226380+0000 mon.a (mon 2021-03-15T22:35:49.682 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: .0) 2470 : audit [INF] 2021-03-15T22:35:49.682 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "39"}]: dispatch 2021-03-15T22:35:49.683 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: audit 2021-03-15T22:35:49.683 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: 2021-03-15T22:35: 2021-03-15T22:35:49.683 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: 49.226577+0000 mon.a (mon 2021-03-15T22:35:49.684 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: .0) 2471 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-03-15T22:35:49.684 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:35:49.684 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: 03-15T22:35:49.348923+0000 2021-03-15T22:35:49.684 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: mon.a (mon.0) 2472 : audit [INF] 2021-03-15T22:35:49.684 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "39"}]': finished 2021-03-15T22:35:49.685 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: audit 2021-03-15T22:35:49.685 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: 2021-03-15T22:35:49.348990+0000 mon.a (mon. 2021-03-15T22:35:49.685 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: 0) 2473 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]': finished 2021-03-15T22:35:49.685 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: cluster 2021-03-15T22:35:49 2021-03-15T22:35:49.686 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: .349046+0000 mon.a (mon.0) 2474 : 2021-03-15T22:35:49.686 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:49 smithi076 conmon[62156]: cluster [DBG] osdmap e994: 8 total, 8 up, 8 in 2021-03-15T22:35:49.723 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:49.746 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:49.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:49.747 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:49.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:49 smithi125 conmon[56739]: cluster 2021-03-15T22:35:47.782089+0000 osd.6 (osd.6 2021-03-15T22:35:49.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:49 smithi125 conmon[56739]: ) 151 : cluster [DBG] 4.a scrub ok 2021-03-15T22:35:49.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:49 smithi125 conmon[56739]: cluster 2021-03-15T22:35:48.581154+0000 osd.7 (osd.7) 187 : cluster [DBG] 3.11 scrub ok 2021-03-15T22:35:49.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:49 smithi125 conmon[56739]: audit 2021-03-15T22:35:49.226380+0000 mon.a (mon.0) 2470 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "39"}]: dispatch 2021-03-15T22:35:49.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:49 smithi125 conmon[56739]: audit 2021-03-15T22:35:49.226577+0000 mon.a (mon.0) 2471 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]: dispatch 2021-03-15T22:35:49.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:49 smithi125 conmon[56739]: audit 2021-03-15T22:35:49.348923+0000 mon.a (mon.0) 2472 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "39"}]': finished 2021-03-15T22:35:49.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:49 smithi125 conmon[56739]: audit 2021-03-15T22:35:49.348990+0000 mon.a (mon.0) 2473 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pgp_num_actual", "val": "22"}]': finished 2021-03-15T22:35:49.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:49 smithi125 conmon[56739]: cluster 2021-03-15T22:35:49.349046+0000 mon.a (mon.0) 2474 : cluster [DBG] osdmap e994: 8 total, 8 up, 8 in 2021-03-15T22:35:49.849 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:49.859 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:49.871 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:49.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:49.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:49.974 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:49.997 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:49.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:49.999 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:50.020 INFO:teuthology.orchestra.run.smithi125.stdout:{ 2021-03-15T22:35:50.020 INFO:teuthology.orchestra.run.smithi125.stdout: "ops": [], 2021-03-15T22:35:50.020 INFO:teuthology.orchestra.run.smithi125.stdout: "num_ops": 0 2021-03-15T22:35:50.020 INFO:teuthology.orchestra.run.smithi125.stdout:} 2021-03-15T22:35:50.100 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:50.122 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:50.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:50.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:50.225 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:50.246 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:50.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:50.248 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:50.349 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:50.354 INFO:tasks.rados.rados.0.smithi076.stdout:3821: done (3 left) 2021-03-15T22:35:50.354 INFO:tasks.rados.rados.0.smithi076.stdout:3822: snap_create 2021-03-15T22:35:50.354 INFO:tasks.rados.rados.0.smithi076.stdout:3819: finishing copy_from to smithi07694878-16 2021-03-15T22:35:50.355 INFO:tasks.rados.rados.0.smithi076.stdout:3819: got expected ENOENT (src dne) 2021-03-15T22:35:50.355 INFO:tasks.rados.rados.0.smithi076.stdout:3819: finishing copy_from racing read to smithi07694878-16 2021-03-15T22:35:50.355 INFO:tasks.rados.rados.0.smithi076.stdout:3820: finishing rollback tid 0 to smithi07694878-24 2021-03-15T22:35:50.355 INFO:tasks.rados.rados.0.smithi076.stdout:3820: finishing rollback tid 1 to smithi07694878-24 2021-03-15T22:35:50.355 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 3040 (ObjNum 262080960 snap 21896 seq_num 264961584) dirty dne 2021-03-15T22:35:50.356 INFO:tasks.rados.rados.0.smithi076.stdout:3818: finishing write tid 3 to smithi07694878-46 2021-03-15T22:35:50.356 INFO:tasks.rados.rados.0.smithi076.stdout:3818: finishing write tid 4 to smithi07694878-46 2021-03-15T22:35:50.356 INFO:tasks.rados.rados.0.smithi076.stdout:3818: finishing write tid 5 to smithi07694878-46 2021-03-15T22:35:50.356 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2304 (ObjNum 1299 snap 448 seq_num 1299) dirty exists 2021-03-15T22:35:50.357 INFO:tasks.rados.rados.0.smithi076.stdout:3818: left oid 46 (ObjNum 1299 snap 448 seq_num 1299) 2021-03-15T22:35:50.379 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:35:50.381 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:50.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:50.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:50.484 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:50.506 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:50.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:50.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:50.536 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:35:50.603 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_blocked_ops 2021-03-15T22:35:50.608 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:50.633 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:50.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:50.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:50.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[56514]: cluster 2021-03-15T22:35:48.884313+0000 osd.3 (osd.3) 174 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:35:50.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[56514]: cluster 2021-03-15T22:35:49.070793+0000 osd.1 (osd.1) 188 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:35:50.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[56514]: cluster 2021-03-15T22:35:49.146331+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:35:50.653 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[56514]: cluster 2021-03-15T22:35:49.225628+0000 mgr.y (mgr.14144) 1092 : cluster [DBG] pgmap v1637: 156 pgs: 156 active+clean; 695 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-03-15T22:35:50.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[56514]: cluster 2021-03-15T22:35:49.448577+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.b scrub ok 2021-03-15T22:35:50.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[56514]: audit 2021-03-15T22:35:50.106952+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? 172.21.15.76:0/2324633482' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:35:50.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[56514]: cluster 2021-03-15T22:35:50.341001+0000 mon.a (mon.0) 2476 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:50.654 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[56514]: audit 2021-03-15T22:35:50.349058+0000 mon.a (mon 2021-03-15T22:35:50.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[56514]: .0) 2477 : audit [INF] from='client.? 172.21.15.76:0/2324633482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:35:50.655 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[56514]: cluster 2021-03-15T22:35:50.349115+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e995: 8 total, 8 up, 8 in 2021-03-15T22:35:50.655 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[62156]: cluster 2021-03-15T22:35:48.884313+0000 osd.3 (osd.3) 174 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:35:50.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[62156]: cluster 2021-03-15T22:35:49.070793+0000 osd.1 (osd.1) 188 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:35:50.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[62156]: cluster 2021-03-15T22:35:49.146331+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:35:50.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[62156]: cluster 2021-03-15T22:35:49.225628+0000 mgr.y (mgr.14144) 1092 : cluster [DBG] pgmap v1637: 156 pgs: 156 active+clean; 695 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-03-15T22:35:50.656 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[62156]: cluster 2021-03-15T22:35: 2021-03-15T22:35:50.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[62156]: 49.448577+0000 osd.2 (osd.2) 50 : cluster [DBG] 5.b scrub ok 2021-03-15T22:35:50.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[62156]: audit 2021-03-15T22:35:50.106952+0000 mon.a (mon.0) 2475 : audit [INF] from='client.? 172.21.15.76:0/2324633482' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:35:50.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[62156]: cluster 2021-03-15T22:35:50.341001+0000 mon.a (mon.0) 2476 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:50.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[62156]: audit 2021-03-15T22:35:50.349058+0000 mon.a (mon.0) 2477 : audit [INF] from='client.? 172.21.15.76:0/2324633482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:35:50.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:50 smithi076 conmon[62156]: cluster 2021-03-15T22:35:50.349115+0000 mon.a (mon.0) 2478 : cluster [DBG] osdmap e995: 8 total, 8 up, 8 in 2021-03-15T22:35:50.736 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:50.758 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:50.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:50.759 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:50.861 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:50.885 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:50.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:50.886 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:50.918 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:50.987 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:51.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:51.013 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:51.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:51.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: cluster 2021-03-15T22:35:48.884313+0000 osd.3 (osd 2021-03-15T22:35:51.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: .3) 174 : cluster [DBG] 5.10 scrub ok 2021-03-15T22:35:51.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: cluster 2021-03-15T22:35:49.070793+0000 osd.1 (osd.1) 188 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:35:51.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: cluster 2021-03-15T22:35:49.146331+0000 osd.0 (osd.0) 160 : cluster [DBG] 6.18 scrub ok 2021-03-15T22:35:51.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: cluster 2021-03-15T22:35:49.225628+0000 mgr.y ( 2021-03-15T22:35:51.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: mgr.14144) 1092 : cluster [DBG] pgmap v1637: 156 pgs: 156 active+clean; 695 MiB data, 1.9 GiB used, 713 GiB / 715 GiB avail; 15 MiB/s rd, 3.0 MiB/s wr, 18 op/s 2021-03-15T22:35:51.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:35:51.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: 22:35:49.448577+0000 osd.2 (osd.2) 50 : cluster 2021-03-15T22:35:51.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: [DBG] 5.b scrub ok 2021-03-15T22:35:51.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:35:51.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: 35:50.106952+0000 mon.a (mon.0 2021-03-15T22:35:51.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: ) 2475 : audit [INF] from='client.? 172.21.15.76:0/2324633482' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:35:51.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: cluster 2021-03-15T22:35:51.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: 2021-03-15T22:35:50 2021-03-15T22:35:51.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: .341001+0000 mon.a (mon.0) 2476 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:51.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:35:51.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: 35:50.349058+0000 mon.a (mon.0) 2021-03-15T22:35:51.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: 2477 : audit [INF] from='client.? 172.21.15.76:0/2324633482' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:35:51.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:35:51.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: T22:35:50.349115+0000 mon.a (mon 2021-03-15T22:35:51.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:50 smithi125 conmon[56739]: .0) 2478 : cluster [DBG] osdmap e995: 8 total, 8 up, 8 in 2021-03-15T22:35:51.115 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:51.140 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:51.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:51.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:51.243 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:51.265 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:51.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:51.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:51.350 INFO:tasks.rados.rados.0.smithi076.stdout:3818: done (3 left) 2021-03-15T22:35:51.350 INFO:tasks.rados.rados.0.smithi076.stdout:3819: done (2 left) 2021-03-15T22:35:51.351 INFO:tasks.rados.rados.0.smithi076.stdout:3820: done (1 left) 2021-03-15T22:35:51.351 INFO:tasks.rados.rados.0.smithi076.stdout:3822: done (0 left) 2021-03-15T22:35:51.351 INFO:tasks.rados.rados.0.smithi076.stdout:3823: copy_from oid 22 from oid 49 current snap is 449 2021-03-15T22:35:51.351 INFO:tasks.rados.rados.0.smithi076.stdout:3824: write (excl) oid 25 current snap is 449 2021-03-15T22:35:51.351 INFO:tasks.rados.rados.0.smithi076.stdout:3824: seq_num 1300 ranges {649100=403710,1606356=501788,2633420=497852,3636005=213095} 2021-03-15T22:35:51.356 INFO:tasks.rados.rados.0.smithi076.stdout:3824: writing smithi07694878-25 from 649100 to 1052810 tid 1 2021-03-15T22:35:51.362 INFO:tasks.rados.rados.0.smithi076.stdout:3824: writing smithi07694878-25 from 1606356 to 2108144 tid 2 2021-03-15T22:35:51.368 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:51.369 INFO:tasks.rados.rados.0.smithi076.stdout:3824: writing smithi07694878-25 from 2633420 to 3131272 tid 3 2021-03-15T22:35:51.373 INFO:tasks.rados.rados.0.smithi076.stdout:3824: writing smithi07694878-25 from 3636005 to 3849100 tid 4 2021-03-15T22:35:51.373 INFO:tasks.rados.rados.0.smithi076.stdout:3825: snap_create 2021-03-15T22:35:51.374 INFO:tasks.rados.rados.0.smithi076.stdout:3824: finishing write tid 1 to smithi07694878-25 2021-03-15T22:35:51.374 INFO:tasks.rados.rados.0.smithi076.stdout:3823: finishing copy_from to smithi07694878-22 2021-03-15T22:35:51.374 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 3159 (ObjNum 1225 snap 424 seq_num 1225) dirty exists 2021-03-15T22:35:51.374 INFO:tasks.rados.rados.0.smithi076.stdout:3823: finishing copy_from racing read to smithi07694878-22 2021-03-15T22:35:51.374 INFO:tasks.rados.rados.0.smithi076.stdout:3824: finishing write tid 2 to smithi07694878-25 2021-03-15T22:35:51.375 INFO:tasks.rados.rados.0.smithi076.stdout:3824: finishing write tid 3 to smithi07694878-25 2021-03-15T22:35:51.375 INFO:tasks.rados.rados.0.smithi076.stdout:3824: finishing write tid 4 to smithi07694878-25 2021-03-15T22:35:51.376 INFO:tasks.rados.rados.0.smithi076.stdout:3824: finishing write tid 5 to smithi07694878-25 2021-03-15T22:35:51.376 INFO:tasks.rados.rados.0.smithi076.stdout:3824: finishing write tid 6 to smithi07694878-25 2021-03-15T22:35:51.377 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 25 v 3175 (ObjNum 1300 snap 449 seq_num 1300) dirty exists 2021-03-15T22:35:51.377 INFO:tasks.rados.rados.0.smithi076.stdout:3824: left oid 25 (ObjNum 1300 snap 449 seq_num 1300) 2021-03-15T22:35:51.391 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:51.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:51.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:51.494 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:51.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:51.518 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:51.518 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:51.620 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:51.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:51.643 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:51.644 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:51.745 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:51.766 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:51.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:51.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:51 smithi125 conmon[56739]: cluster 2021-03-15T22:35:50.069456+0000 osd.1 (osd.1) 189 2021-03-15T22:35:51.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:51 smithi125 conmon[56739]: : cluster [DBG] 5.12 scrub ok 2021-03-15T22:35:51.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:51 smithi125 conmon[56739]: audit 2021-03-15T22:35:50.306507+0000 mgr.y (mgr.14144) 1093 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:51.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:51 smithi125 conmon[56739]: cluster 2021-03-15T22:35:51.347042+0000 mon.a (mon.0) 2479 : cluster [DBG] osdmap e996: 8 total, 8 up, 8 in 2021-03-15T22:35:51.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:51 smithi076 conmon[56514]: cluster 2021-03-15T22:35:50.069456+0000 osd.1 (osd.1) 189 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:35:51.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:51 smithi076 conmon[56514]: 2021-03-15T22:35:51.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:51 smithi076 conmon[56514]: audit 2021-03-15T22:35:50.306507+0000 mgr.y (mgr.14144) 1093 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:51.804 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:51 smithi076 conmon[56514]: cluster 2021-03-15T22:35:51.347042+0000 mon.a (mon.0) 2479 : cluster [DBG] osdmap e996: 8 total, 8 up, 8 in 2021-03-15T22:35:51.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:51 smithi076 conmon[62156]: cluster 2021-03-15T22:35:50.069456+0000 osd.1 2021-03-15T22:35:51.804 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:51 smithi076 conmon[62156]: (osd.1) 189 : cluster [DBG] 5.12 scrub ok 2021-03-15T22:35:51.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:51 smithi076 conmon[62156]: audit 2021-03-15T22:35:50.306507+0000 mgr.y (mgr 2021-03-15T22:35:51.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:51 smithi076 conmon[62156]: .14144) 1093 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:35:51.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:51 smithi076 conmon[62156]: cluster 2021-03-15T22:35:51.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:51 smithi076 conmon[62156]: 2021-03-15T22:35:51.347042+0000 mon.a (mon. 2021-03-15T22:35:51.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:51 smithi076 conmon[62156]: 0) 2479 : cluster [DBG] osdmap e996: 8 total, 8 up, 8 in 2021-03-15T22:35:51.870 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:51.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:51.895 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:51.895 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:51.992 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:35:51.996 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:52.008 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:35:52.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:52.021 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:52.022 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:52.025 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:35:52.042 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:35:52.059 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:35:52.077 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:35:52.093 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:35:52.111 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:35:52.112 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:35:52.113 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:35:52.114 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:35:52.114 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:35:52.114 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:35:52.114 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:35:52.115 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:35:52.115 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:35:52.123 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:52.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:52.148 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:52.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:52.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:52.273 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:52.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:52.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:52.353 INFO:tasks.rados.rados.0.smithi076.stdout:3823: done (2 left) 2021-03-15T22:35:52.354 INFO:tasks.rados.rados.0.smithi076.stdout:3824: done (1 left) 2021-03-15T22:35:52.354 INFO:tasks.rados.rados.0.smithi076.stdout:3825: done (0 left) 2021-03-15T22:35:52.354 INFO:tasks.rados.rados.0.smithi076.stdout:3826: read oid 9 snap 449 2021-03-15T22:35:52.355 INFO:tasks.rados.rados.0.smithi076.stdout:3826: expect (ObjNum 1285 snap 442 seq_num 1285) 2021-03-15T22:35:52.355 INFO:tasks.rados.rados.0.smithi076.stdout:3827: delete oid 15 current snap is 450 2021-03-15T22:35:52.362 INFO:tasks.rados.rados.0.smithi076.stdout:3826: expect (ObjNum 1285 snap 442 seq_num 1285) 2021-03-15T22:35:52.376 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:52.399 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:52.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:52.401 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:52.480 INFO:tasks.rados.rados.0.smithi076.stdout:3826: done (1 left) 2021-03-15T22:35:52.480 INFO:tasks.rados.rados.0.smithi076.stdout:3827: done (0 left) 2021-03-15T22:35:52.481 INFO:tasks.rados.rados.0.smithi076.stdout:3828: delete oid 47 current snap is 450 2021-03-15T22:35:52.482 INFO:tasks.rados.rados.0.smithi076.stdout:3828: done (0 left) 2021-03-15T22:35:52.482 INFO:tasks.rados.rados.0.smithi076.stdout:3829: snap_create 2021-03-15T22:35:52.501 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:52.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:52.525 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:52.525 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:52.556 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:35:52.556 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:35:52.624 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:52 smithi125 conmon[56739]: cluster 2021-03-15T22:35:51.226188+0000 mgr.y (mgr.14144) 1094 : cluster [DBG] pgmap v1640: 156 pgs: 156 active+clean; 704 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2021-03-15T22:35:52.625 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:52 smithi125 conmon[56739]: audit 2021-03-15T22:35:51.973351+0000 mon.a (mon.0) 2480 : audit [DBG] from='client.? 172.21.15.76:0/2015213197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:52.625 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:52 smithi125 conmon[56739]: cluster 2021-03-15T22:35:52.350040+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e997: 8 total, 8 up, 8 in 2021-03-15T22:35:52.627 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:52.648 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:52.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:52.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:52.751 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:52.773 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:52.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:52.774 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:52.875 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:52.883 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:52 smithi076 conmon[56514]: cluster 2021-03-15T22:35:51.226188+0000 mgr.y (mgr.14144) 1094 : cluster [DBG] pgmap v1640: 156 pgs: 156 active+clean; 704 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2021-03-15T22:35:52.883 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:52 smithi076 conmon[56514]: audit 2021-03-15T22:35:51.973351+0000 mon.a (mon.0) 2480 : audit [DBG] from='client.? 172.21.15.76:0/2015213197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:52.883 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:52 smithi076 conmon[56514]: cluster 2021-03-15T22:35:52.350040+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e997: 8 total, 8 up, 8 in 2021-03-15T22:35:52.884 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:52 smithi076 conmon[62156]: cluster 2021-03-15T22:35:51.226188+0000 mgr.y (mgr.14144) 1094 : cluster [DBG] pgmap v1640: 156 pgs: 156 active+clean; 704 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.5 MiB/s rd, 1.9 MiB/s wr, 14 op/s 2021-03-15T22:35:52.884 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:52 smithi076 conmon[62156]: audit 2021-03-15T22:35:51.973351+0000 mon.a (mon.0) 2480 : audit [DBG] from='client.? 172.21.15.76:0/2015213197' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:35:52.884 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:52 smithi076 conmon[62156]: cluster 2021-03-15T22:35:52.350040+0000 mon.a (mon.0) 2481 : cluster [DBG] osdmap e997: 8 total, 8 up, 8 in 2021-03-15T22:35:52.898 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:52.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:52.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:53.000 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:53.023 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:53.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:53.024 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:53.085 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 7, 0, 3, 5, 6, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:35:53.086 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:35:53.086 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.control 2021-03-15T22:35:53.087 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:35:53.125 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:53.152 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:53.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:53.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:53.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.5 dump_historic_ops 2021-03-15T22:35:53.255 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:53.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:53.279 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:53.280 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:53.358 INFO:tasks.rados.rados.0.smithi076.stdout:3829: done (0 left) 2021-03-15T22:35:53.359 INFO:tasks.rados.rados.0.smithi076.stdout:3830: copy_from oid 42 from oid 34 current snap is 451 2021-03-15T22:35:53.359 INFO:tasks.rados.rados.0.smithi076.stdout:3831: write oid 21 current snap is 451 2021-03-15T22:35:53.360 INFO:tasks.rados.rados.0.smithi076.stdout:3831: seq_num 1303 ranges {699521=745810,2181802=649115,3515689=383832} 2021-03-15T22:35:53.366 INFO:tasks.rados.rados.0.smithi076.stdout:3831: writing smithi07694878-21 from 699521 to 1445331 tid 1 2021-03-15T22:35:53.373 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:53.374 INFO:tasks.rados.rados.0.smithi076.stdout:3831: writing smithi07694878-21 from 2181802 to 2830917 tid 2 2021-03-15T22:35:53.380 INFO:tasks.rados.rados.0.smithi076.stdout:3831: writing smithi07694878-21 from 3515689 to 3899521 tid 3 2021-03-15T22:35:53.381 INFO:tasks.rados.rados.0.smithi076.stdout:3832: snap_create 2021-03-15T22:35:53.381 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:53.387 INFO:tasks.rados.rados.0.smithi076.stdout:3830: finishing copy_from to smithi07694878-42 2021-03-15T22:35:53.388 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 2689 (ObjNum 1286 snap 442 seq_num 1286) dirty exists 2021-03-15T22:35:53.388 INFO:tasks.rados.rados.0.smithi076.stdout:3830: finishing copy_from racing read to smithi07694878-42 2021-03-15T22:35:53.388 INFO:tasks.rados.rados.0.smithi076.stdout:3831: finishing write tid 1 to smithi07694878-21 2021-03-15T22:35:53.388 INFO:tasks.rados.rados.0.smithi076.stdout:3831: finishing write tid 2 to smithi07694878-21 2021-03-15T22:35:53.389 INFO:tasks.rados.rados.0.smithi076.stdout:3831: finishing write tid 3 to smithi07694878-21 2021-03-15T22:35:53.389 INFO:tasks.rados.rados.0.smithi076.stdout:3831: finishing write tid 4 to smithi07694878-21 2021-03-15T22:35:53.389 INFO:tasks.rados.rados.0.smithi076.stdout:3831: finishing write tid 5 to smithi07694878-21 2021-03-15T22:35:53.389 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 3370 (ObjNum 1303 snap 451 seq_num 1303) dirty exists 2021-03-15T22:35:53.390 INFO:tasks.rados.rados.0.smithi076.stdout:3831: left oid 21 (ObjNum 1303 snap 451 seq_num 1303) 2021-03-15T22:35:53.407 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:53.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:53.409 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:53.487 INFO:teuthology.orchestra.run.smithi125.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.b/config 2021-03-15T22:35:53.509 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:53.534 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:53.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:53.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:53.637 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:53.659 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:53.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:53.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:53.719 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:35:53.762 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:53.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:53.788 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:53.788 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:53.833 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[56514]: audit 2021-03-15T22:35:53.227415 2021-03-15T22:35:53.833 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[56514]: +0000 mon.a (mon.0) 2482 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "38"}]: dispatch 2021-03-15T22:35:53.834 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[56514]: audit 2021-03-15T22:35:53.352229+0000 mon.a (mon.0) 2483 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "38"}]': finished 2021-03-15T22:35:53.838 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[56514]: cluster 2021-03-15T22:35:53.352664+0000 mon.a (mon.0) 2484 : cluster [DBG] osdmap e998: 8 total, 8 up, 8 in 2021-03-15T22:35:53.838 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[62156]: audit 2021-03-15T22:35:53.227415 2021-03-15T22:35:53.839 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2482 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "38"}]: dispatch 2021-03-15T22:35:53.839 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:35:53.839 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[62156]: T22:35:53.352229+0000 mon.a (mon.0) 2483 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "38"}]': finished 2021-03-15T22:35:53.839 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[62156]: 2021-03-15T22:35:53.839 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[62156]: cluster 2021-03-15T22:35:53 2021-03-15T22:35:53.840 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[62156]: .352664+0000 mon.a (mon.0) 2484 : cluster 2021-03-15T22:35:53.840 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:53 smithi076 conmon[62156]: [DBG] osdmap e998: 8 total, 8 up, 8 in 2021-03-15T22:35:53.877 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:53 smithi125 conmon[56739]: audit 2021-03-15T22:35:53.227415+0000 mon.a (mon.0) 2021-03-15T22:35:53.878 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:53 smithi125 conmon[56739]: 2482 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "38"}]: dispatch 2021-03-15T22:35:53.878 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:53 smithi125 conmon[56739]: audit 2021-03-15T22:35:53.352229+0000 mon.a (mon.0) 2483 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "38"}]': finished 2021-03-15T22:35:53.878 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:53 smithi125 conmon[56739]: cluster 2021-03-15T22:35:53.352664+0000 mon.a (mon.0) 2484 : cluster [DBG] osdmap e998: 8 total, 8 up, 8 in 2021-03-15T22:35:53.890 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:53.912 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:53.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:53.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:54.014 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:54.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:54.038 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:54.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:54.140 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:54.162 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:54.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:54.164 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:54.265 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:54.287 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:54.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:54.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:54.355 INFO:tasks.rados.rados.0.smithi076.stdout:3830: done (2 left) 2021-03-15T22:35:54.355 INFO:tasks.rados.rados.0.smithi076.stdout:3831: done (1 left) 2021-03-15T22:35:54.355 INFO:tasks.rados.rados.0.smithi076.stdout:3832: done (0 left) 2021-03-15T22:35:54.356 INFO:tasks.rados.rados.0.smithi076.stdout:3833: read oid 29 snap -1 2021-03-15T22:35:54.356 INFO:tasks.rados.rados.0.smithi076.stdout:3833: expect (ObjNum 1228 snap 425 seq_num 1228) 2021-03-15T22:35:54.356 INFO:tasks.rados.rados.0.smithi076.stdout:3834: snap_create 2021-03-15T22:35:54.363 INFO:tasks.rados.rados.0.smithi076.stdout:3833: expect (ObjNum 1228 snap 425 seq_num 1228) 2021-03-15T22:35:54.390 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:54.411 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:54.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:54.413 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:54.514 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:54.532 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:54.537 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:54.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:54.538 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:54.640 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:54.662 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:54.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:54.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:54.744 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:54 smithi076 conmon[56514]: cluster 2021-03-15T22:35:53.226636+0000 mgr.y (mgr.14144) 2021-03-15T22:35:54.744 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:54 smithi076 conmon[56514]: 1095 : cluster [DBG] pgmap v1643: 155 pgs: 155 active+clean; 704 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 89 B/s, 0 objects/s recovering 2021-03-15T22:35:54.745 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:54 smithi076 conmon[56514]: cluster 2021-03-15T22:35:54.349927+0000 mon.a (mon.0) 2485 : cluster [DBG] osdmap e999: 8 total, 8 up, 8 in 2021-03-15T22:35:54.745 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:54 smithi076 conmon[62156]: cluster 2021-03-15T22:35:53.226636+0000 2021-03-15T22:35:54.746 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:54 smithi076 conmon[62156]: mgr.y (mgr.14144) 1095 : cluster [DBG] pgmap v1643: 155 pgs: 155 active+clean; 704 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 89 B/s, 0 objects/s recovering 2021-03-15T22:35:54.746 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:54 smithi076 conmon[62156]: cluster 2021-03-15T22:35:54.349927+0000 2021-03-15T22:35:54.746 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:54 smithi076 conmon[62156]: mon.a (mon.0) 2485 : cluster [DBG] osdmap e999: 8 total, 8 up, 8 in 2021-03-15T22:35:54.764 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:54.787 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:54.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:54.788 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:54.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:54 smithi125 conmon[56739]: cluster 2021-03-15T22:35:54.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:54 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:35:54.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:54 smithi125 conmon[56739]: 35:53.226636+0000 mgr.y (mgr.14144) 1095 : cluster [DBG] pgmap v1643: 155 pgs: 155 active+clean; 704 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 1.9 MiB/s wr, 12 op/s; 89 B/s, 0 objects/s recovering 2021-03-15T22:35:54.814 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:54 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:35:54.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:54 smithi125 conmon[56739]: -15T22:35:54.349927+0000 mon.a 2021-03-15T22:35:54.815 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:54 smithi125 conmon[56739]: (mon.0) 2485 : cluster [DBG] osdmap e999: 8 total, 8 up, 8 in 2021-03-15T22:35:54.890 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:54.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:35:54.912 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:54.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:54.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:55.014 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:55.033 INFO:teuthology.orchestra.run.smithi125.stdout:ERROR: (22) Invalid argument 2021-03-15T22:35:55.033 INFO:teuthology.orchestra.run.smithi125.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:35:55.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:55.040 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:55.040 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:55.145 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:55.146 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:35:55.148 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:35:55.210 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:55.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:55.212 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:55.306 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:55.313 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:55.335 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:55.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:55.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:55.350 INFO:tasks.rados.rados.0.smithi076.stdout:3833: done (1 left) 2021-03-15T22:35:55.350 INFO:tasks.rados.rados.0.smithi076.stdout:3834: done (0 left) 2021-03-15T22:35:55.350 INFO:tasks.rados.rados.0.smithi076.stdout:3835: copy_from oid 34 from oid 18 current snap is 453 2021-03-15T22:35:55.350 INFO:tasks.rados.rados.0.smithi076.stdout:3836: snap_create 2021-03-15T22:35:55.372 INFO:tasks.rados.rados.0.smithi076.stdout:3835: finishing copy_from to smithi07694878-34 2021-03-15T22:35:55.372 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 34 v 2446 (ObjNum 1265 snap 437 seq_num 1265) dirty exists 2021-03-15T22:35:55.373 INFO:tasks.rados.rados.0.smithi076.stdout:3835: finishing copy_from racing read to smithi07694878-34 2021-03-15T22:35:55.438 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:55.460 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:55.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:55.462 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:55.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:55 smithi076 conmon[62156]: audit 2021-03-15T22:35:55.228150+0000 mon.a (mon.0) 2486 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-03-15T22:35:55.555 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:55 smithi076 conmon[62156]: audit 2021-03-15T22:35:55.347659+0000 mon.a (mon.0) 2487 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]': finished 2021-03-15T22:35:55.556 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:55 smithi076 conmon[62156]: cluster 2021-03-15T22:35:55.347731+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in 2021-03-15T22:35:55.563 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:55.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:55.587 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:55.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:55.670 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_ops_in_flight 2021-03-15T22:35:55.688 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:55.710 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:55.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:55.712 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:55.813 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:55.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:55.842 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:55.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:55.944 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:55.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:55.970 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:55.970 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:56.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:55 smithi076 conmon[56514]: audit 2021-03-15T22:35:55.228150+0000 mon.a (mon.0) 2486 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-03-15T22:35:56.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:55 smithi076 conmon[56514]: audit 2021-03-15T22:35:55.347659+0000 mon.a (mon.0) 2487 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]': finished 2021-03-15T22:35:56.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:55 smithi076 conmon[56514]: cluster 2021-03-15T22:35:55.347731+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in 2021-03-15T22:35:56.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:55 smithi125 conmon[56739]: audit 2021-03-15T22:35:55.228150+0000 mon.a (mon.0) 2486 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]: dispatch 2021-03-15T22:35:56.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:55 smithi125 conmon[56739]: audit 2021-03-15T22:35:55.347659+0000 mon.a (mon.0) 2487 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "31"}]': finished 2021-03-15T22:35:56.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:55 smithi125 conmon[56739]: cluster 2021-03-15T22:35:55.347731+0000 mon.a (mon.0) 2488 : cluster [DBG] osdmap e1000: 8 total, 8 up, 8 in 2021-03-15T22:35:56.071 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:56.095 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:56.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:56.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:56.197 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:56.220 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:56.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:56.222 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:56.323 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:56.347 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:56.349 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:56.360 INFO:tasks.rados.rados.0.smithi076.stdout:3835: done (1 left) 2021-03-15T22:35:56.360 INFO:tasks.rados.rados.0.smithi076.stdout:3836: done (0 left) 2021-03-15T22:35:56.360 INFO:tasks.rados.rados.0.smithi076.stdout:3837: copy_from oid 17 from oid 30 current snap is 454 2021-03-15T22:35:56.360 INFO:tasks.rados.rados.0.smithi076.stdout:3838: rollback oid 13 current snap is 454 2021-03-15T22:35:56.361 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 13 to 432 2021-03-15T22:35:56.361 INFO:tasks.rados.rados.0.smithi076.stdout:3839: snap_create 2021-03-15T22:35:56.362 INFO:tasks.rados.rados.0.smithi076.stdout:3838: finishing rollback tid 1 to smithi07694878-13 2021-03-15T22:35:56.362 INFO:tasks.rados.rados.0.smithi076.stdout:3838: finishing rollback tid 2 to smithi07694878-13 2021-03-15T22:35:56.362 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 13 v 3050 (ObjNum 1072 snap 366 seq_num 1072) dirty exists 2021-03-15T22:35:56.379 INFO:tasks.rados.rados.0.smithi076.stdout:3837: finishing copy_from to smithi07694878-17 2021-03-15T22:35:56.379 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 3184 (ObjNum 1282 snap 442 seq_num 1282) dirty exists 2021-03-15T22:35:56.380 INFO:tasks.rados.rados.0.smithi076.stdout:3837: finishing copy_from racing read to smithi07694878-17 2021-03-15T22:35:56.449 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:56.473 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:56.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:56.474 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:56.553 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1645,"stamp":"2021-03-15T22:35:54.347950+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":743723027,"num_objects":646,"num_object_clones":281,"num_object_copies":1938,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":646,"num_whiteouts":11,"num_read":9638,"num_read_kb":7275279,"num_write":6045,"num_write_kb":2006847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5902,"num_bytes_recovered":2230124932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24037,"ondisk_log_size":24037,"up":465,"acting":465,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":470,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2081040,"kb_used_data":1653200,"kb_used_omap":273,"kb_used_meta":427502,"kb_avail":747880176,"statfs":{"total":767960285184,"available":765829300224,"internally_reserved":0,"allocated":1692876800,"data_stored":2343089359,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":437762938},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":8945299,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":12,"num_read_kb":8,"num_write":20,"num_write_kb":8905,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1231,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.004920"},"pg_stats":[{"pgid":"6.26","version":"0'0","reported_seq":"25","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.348224+0000","last_change":"2021-03-15T22:35:36.350574+0000","last_active":"2021-03-15T22:35:50.348224+0000","last_peered":"2021-03-15T22:35:50.348224+0000","last_clean":"2021-03-15T22:35:50.348224+0000","last_became_active":"2021-03-15T22:35:36.350145+0000","last_became_peered":"2021-03-15T22:35:36.350145+0000","last_unstale":"2021-03-15T22:35:50.348224+0000","last_undegraded":"2021-03-15T22:35:50.348224+0000","last_fullsized":"2021-03-15T22:35:50.348224+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.25","version":"0'0","reported_seq":"27","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362319+0000","last_change":"2021-03-15T22:35:36.755816+0000","last_active":"2021-03-15T22:35:52.362319+0000","last_peered":"2021-03-15T22:35:52.362319+0000","last_clean":"2021-03-15T22:35:52.362319+0000","last_became_active":"2021-03-15T22:35:36.755571+0000","last_became_peered":"2021-03-15T22:35:36.755571+0000","last_unstale":"2021-03-15T22:35:52.362319+0000","last_undegraded":"2021-03-15T22:35:52.362319+0000","last_fullsized":"2021-03-15T22:35:52.362319+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:32.417221+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:32.417221+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.24","version":"0'0","reported_seq":"816","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.351688+0000","last_change":"2021-03-15T22:35:36.758818+0000","last_active":"2021-03-15T22:35:52.351688+0000","last_peered":"2021-03-15T22:35:52.351688+0000","last_clean":"2021-03-15T22:35:52.351688+0000","last_became_active":"2021-03-15T22:35:36.758418+0000","last_became_peered":"2021-03-15T22:35:36.758418+0000","last_unstale":"2021-03-15T22:35:52.351688+0000","last_undegraded":"2021-03-15T22:35:52.351688+0000","last_fullsized":"2021-03-15T22:35:52.351688+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.23","version":"0'0","reported_seq":"27","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.361837+0000","last_change":"2021-03-15T22:35:36.758802+0000","last_active":"2021-03-15T22:35:52.361837+0000","last_peered":"2021-03-15T22:35:52.361837+0000","last_clean":"2021-03-15T22:35:52.361837+0000","last_became_active":"2021-03-15T22:35:36.758471+0000","last_became_peered":"2021-03-15T22:35:36.758471+0000","last_unstale":"2021-03-15T22:35:52.361837+0000","last_undegraded":"2021-03-15T22:35:52.361837+0000","last_fullsized":"2021-03-15T22:35:52.361837+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:14.592451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:14.592451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.22","version":"0'0","reported_seq":"25","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344487+0000","last_change":"2021-03-15T22:35:36.356252+0000","last_active":"2021-03-15T22:35:50.344487+0000","last_peered":"2021-03-15T22:35:50.344487+0000","last_clean":"2021-03-15T22:35:50.344487+0000","last_became_active":"2021-03-15T22:35:36.355967+0000","last_became_peered":"2021-03-15T22:35:36.355967+0000","last_unstale":"2021-03-15T22:35:50.344487+0000","last_undegraded":"2021-03-15T22:35:50.344487+0000","last_fullsized":"2021-03-15T22:35:50.344487+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.21","version":"0'0","reported_seq":"26","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534096+0000","last_change":"2021-03-15T22:35:36.758571+0000","last_active":"2021-03-15T22:35:51.534096+0000","last_peered":"2021-03-15T22:35:51.534096+0000","last_clean":"2021-03-15T22:35:51.534096+0000","last_became_active":"2021-03-15T22:35:36.758203+0000","last_became_peered":"2021-03-15T22:35:36.758203+0000","last_unstale":"2021-03-15T22:35:51.534096+0000","last_undegraded":"2021-03-15T22:35:51.534096+0000","last_fullsized":"2021-03-15T22:35:51.534096+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.701955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.701955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"24","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.355981+0000","last_change":"2021-03-15T22:35:36.357269+0000","last_active":"2021-03-15T22:35:49.355981+0000","last_peered":"2021-03-15T22:35:49.355981+0000","last_clean":"2021-03-15T22:35:49.355981+0000","last_became_active":"2021-03-15T22:35:36.356824+0000","last_became_peered":"2021-03-15T22:35:36.356824+0000","last_unstale":"2021-03-15T22:35:49.355981+0000","last_undegraded":"2021-03-15T22:35:49.355981+0000","last_fullsized":"2021-03-15T22:35:49.355981+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"317'7","reported_seq":"360","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534129+0000","last_change":"2021-03-15T22:34:39.122745+0000","last_active":"2021-03-15T22:35:51.534129+0000","last_peered":"2021-03-15T22:35:51.534129+0000","last_clean":"2021-03-15T22:35:51.534129+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:35:51.534129+0000","last_undegraded":"2021-03-15T22:35:51.534129+0000","last_fullsized":"2021-03-15T22:35:51.534129+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:34:39.122665+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.122665+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1011","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344617+0000","last_change":"2021-03-15T22:34:10.331764+0000","last_active":"2021-03-15T22:35:50.344617+0000","last_peered":"2021-03-15T22:35:50.344617+0000","last_clean":"2021-03-15T22:35:50.344617+0000","last_became_active":"2021-03-15T22:34:10.331424+0000","last_became_peered":"2021-03-15T22:34:10.331424+0000","last_unstale":"2021-03-15T22:35:50.344617+0000","last_undegraded":"2021-03-15T22:35:50.344617+0000","last_fullsized":"2021-03-15T22:35:50.344617+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.564464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.564464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1f","version":"57'1","reported_seq":"18","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362815+0000","last_change":"2021-03-15T22:35:50.377663+0000","last_active":"2021-03-15T22:35:52.362815+0000","last_peered":"2021-03-15T22:35:52.362815+0000","last_clean":"2021-03-15T22:35:52.362815+0000","last_became_active":"2021-03-15T22:35:50.359488+0000","last_became_peered":"2021-03-15T22:35:50.359488+0000","last_unstale":"2021-03-15T22:35:52.362815+0000","last_undegraded":"2021-03-15T22:35:52.362815+0000","last_fullsized":"2021-03-15T22:35:52.362815+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"57'1","last_scrub_stamp":"2021-03-15T22:32:31.680111+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.680111+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":873,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3492,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"1094","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344488+0000","last_change":"2021-03-15T22:35:14.434026+0000","last_active":"2021-03-15T22:35:50.344488+0000","last_peered":"2021-03-15T22:35:50.344488+0000","last_clean":"2021-03-15T22:35:50.344488+0000","last_became_active":"2021-03-15T22:33:04.345393+0000","last_became_peered":"2021-03-15T22:33:04.345393+0000","last_unstale":"2021-03-15T22:35:50.344488+0000","last_undegraded":"2021-03-15T22:35:50.344488+0000","last_fullsized":"2021-03-15T22:35:50.344488+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:35:14.433961+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:35:14.433961+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"298","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344311+0000","last_change":"2021-03-15T22:33:04.345933+0000","last_active":"2021-03-15T22:35:50.344311+0000","last_peered":"2021-03-15T22:35:50.344311+0000","last_clean":"2021-03-15T22:35:50.344311+0000","last_became_active":"2021-03-15T22:33:04.345518+0000","last_became_peered":"2021-03-15T22:33:04.345518+0000","last_unstale":"2021-03-15T22:35:50.344311+0000","last_undegraded":"2021-03-15T22:35:50.344311+0000","last_fullsized":"2021-03-15T22:35:50.344311+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.338100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.338100+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"78","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347220+0000","last_change":"2021-03-15T22:34:43.345093+0000","last_active":"2021-03-15T22:35:50.347220+0000","last_peered":"2021-03-15T22:35:50.347220+0000","last_clean":"2021-03-15T22:35:50.347220+0000","last_became_active":"2021-03-15T22:34:43.344764+0000","last_became_peered":"2021-03-15T22:34:43.344764+0000","last_unstale":"2021-03-15T22:35:50.347220+0000","last_undegraded":"2021-03-15T22:35:50.347220+0000","last_fullsized":"2021-03-15T22:35:50.347220+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:08.652741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.652741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":"13","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362248+0000","last_change":"2021-03-15T22:35:50.361069+0000","last_active":"2021-03-15T22:35:52.362248+0000","last_peered":"2021-03-15T22:35:52.362248+0000","last_clean":"2021-03-15T22:35:52.362248+0000","last_became_active":"2021-03-15T22:35:50.360730+0000","last_became_peered":"2021-03-15T22:35:50.360730+0000","last_unstale":"2021-03-15T22:35:52.362248+0000","last_undegraded":"2021-03-15T22:35:52.362248+0000","last_fullsized":"2021-03-15T22:35:52.362248+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.475850+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.475850+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"139","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347246+0000","last_change":"2021-03-15T22:34:55.399761+0000","last_active":"2021-03-15T22:35:50.347246+0000","last_peered":"2021-03-15T22:35:50.347246+0000","last_clean":"2021-03-15T22:35:50.347246+0000","last_became_active":"2021-03-15T22:34:55.346148+0000","last_became_peered":"2021-03-15T22:34:55.346148+0000","last_unstale":"2021-03-15T22:35:50.347246+0000","last_undegraded":"2021-03-15T22:35:50.347246+0000","last_fullsized":"2021-03-15T22:35:50.347246+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.652898+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":5340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"848","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.351687+0000","last_change":"2021-03-15T22:35:16.848985+0000","last_active":"2021-03-15T22:35:52.351687+0000","last_peered":"2021-03-15T22:35:52.351687+0000","last_clean":"2021-03-15T22:35:52.351687+0000","last_became_active":"2021-03-15T22:34:03.570306+0000","last_became_peered":"2021-03-15T22:34:03.570306+0000","last_unstale":"2021-03-15T22:35:52.351687+0000","last_undegraded":"2021-03-15T22:35:52.351687+0000","last_fullsized":"2021-03-15T22:35:52.351687+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:16.848918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.848918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"464","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357361+0000","last_change":"2021-03-15T22:35:49.146403+0000","last_active":"2021-03-15T22:35:49.357361+0000","last_peered":"2021-03-15T22:35:49.357361+0000","last_clean":"2021-03-15T22:35:49.357361+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:35:49.357361+0000","last_undegraded":"2021-03-15T22:35:49.357361+0000","last_fullsized":"2021-03-15T22:35:49.357361+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.146337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.146337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":"13","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362135+0000","last_change":"2021-03-15T22:35:50.356948+0000","last_active":"2021-03-15T22:35:52.362135+0000","last_peered":"2021-03-15T22:35:52.362135+0000","last_clean":"2021-03-15T22:35:52.362135+0000","last_became_active":"2021-03-15T22:35:50.356083+0000","last_became_peered":"2021-03-15T22:35:50.356083+0000","last_unstale":"2021-03-15T22:35:52.362135+0000","last_undegraded":"2021-03-15T22:35:52.362135+0000","last_fullsized":"2021-03-15T22:35:52.362135+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.756581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.756581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"1145","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344711+0000","last_change":"2021-03-15T22:33:17.502781+0000","last_active":"2021-03-15T22:35:50.344711+0000","last_peered":"2021-03-15T22:35:50.344711+0000","last_clean":"2021-03-15T22:35:50.344711+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:35:50.344711+0000","last_undegraded":"2021-03-15T22:35:50.344711+0000","last_fullsized":"2021-03-15T22:35:50.344711+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:33:17.502703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.502703+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"283","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.361702+0000","last_change":"2021-03-15T22:35:12.404980+0000","last_active":"2021-03-15T22:35:52.361702+0000","last_peered":"2021-03-15T22:35:52.361702+0000","last_clean":"2021-03-15T22:35:52.361702+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:35:52.361702+0000","last_undegraded":"2021-03-15T22:35:52.361702+0000","last_fullsized":"2021-03-15T22:35:52.361702+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.404897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.404897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"636","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.352291+0000","last_change":"2021-03-15T22:34:43.344692+0000","last_active":"2021-03-15T22:35:52.352291+0000","last_peered":"2021-03-15T22:35:52.352291+0000","last_clean":"2021-03-15T22:35:52.352291+0000","last_became_active":"2021-03-15T22:34:43.344401+0000","last_became_peered":"2021-03-15T22:34:43.344401+0000","last_unstale":"2021-03-15T22:35:52.352291+0000","last_undegraded":"2021-03-15T22:35:52.352291+0000","last_fullsized":"2021-03-15T22:35:52.352291+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.774079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.774079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":"387","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.352008+0000","last_change":"2021-03-15T22:35:50.360962+0000","last_active":"2021-03-15T22:35:52.352008+0000","last_peered":"2021-03-15T22:35:52.352008+0000","last_clean":"2021-03-15T22:35:52.352008+0000","last_became_active":"2021-03-15T22:35:50.360639+0000","last_became_peered":"2021-03-15T22:35:50.360639+0000","last_unstale":"2021-03-15T22:35:52.352008+0000","last_undegraded":"2021-03-15T22:35:52.352008+0000","last_fullsized":"2021-03-15T22:35:52.352008+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:40.034949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:40.034949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"1141","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344891+0000","last_change":"2021-03-15T22:35:45.410540+0000","last_active":"2021-03-15T22:35:50.344891+0000","last_peered":"2021-03-15T22:35:50.344891+0000","last_clean":"2021-03-15T22:35:50.344891+0000","last_became_active":"2021-03-15T22:34:55.335550+0000","last_became_peered":"2021-03-15T22:34:55.335550+0000","last_unstale":"2021-03-15T22:35:50.344891+0000","last_undegraded":"2021-03-15T22:35:50.344891+0000","last_fullsized":"2021-03-15T22:35:50.344891+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:35:45.410466+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.410466+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"957","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.533962+0000","last_change":"2021-03-15T22:35:10.068109+0000","last_active":"2021-03-15T22:35:51.533962+0000","last_peered":"2021-03-15T22:35:51.533962+0000","last_clean":"2021-03-15T22:35:51.533962+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:35:51.533962+0000","last_undegraded":"2021-03-15T22:35:51.533962+0000","last_fullsized":"2021-03-15T22:35:51.533962+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:35:10.068030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.068030+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1018","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.345694+0000","last_change":"2021-03-15T22:34:41.614750+0000","last_active":"2021-03-15T22:35:50.345694+0000","last_peered":"2021-03-15T22:35:50.345694+0000","last_clean":"2021-03-15T22:35:50.345694+0000","last_became_active":"2021-03-15T22:33:04.329078+0000","last_became_peered":"2021-03-15T22:33:04.329078+0000","last_unstale":"2021-03-15T22:35:50.345694+0000","last_undegraded":"2021-03-15T22:35:50.345694+0000","last_fullsized":"2021-03-15T22:35:50.345694+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.614684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.614684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"15","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.361210+0000","last_change":"2021-03-15T22:35:50.360900+0000","last_active":"2021-03-15T22:35:50.361210+0000","last_peered":"2021-03-15T22:35:50.361210+0000","last_clean":"2021-03-15T22:35:50.361210+0000","last_became_active":"2021-03-15T22:35:50.354808+0000","last_became_peered":"2021-03-15T22:35:50.354808+0000","last_unstale":"2021-03-15T22:35:50.361210+0000","last_undegraded":"2021-03-15T22:35:50.361210+0000","last_fullsized":"2021-03-15T22:35:50.361210+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:32:41.673792+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.673792+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"554","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534241+0000","last_change":"2021-03-15T22:34:55.366147+0000","last_active":"2021-03-15T22:35:51.534241+0000","last_peered":"2021-03-15T22:35:51.534241+0000","last_clean":"2021-03-15T22:35:51.534241+0000","last_became_active":"2021-03-15T22:34:55.350325+0000","last_became_peered":"2021-03-15T22:34:55.350325+0000","last_unstale":"2021-03-15T22:35:51.534241+0000","last_undegraded":"2021-03-15T22:35:51.534241+0000","last_fullsized":"2021-03-15T22:35:51.534241+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:34:38.119156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:38.119156+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1049","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.345166+0000","last_change":"2021-03-15T22:35:42.445978+0000","last_active":"2021-03-15T22:35:50.345166+0000","last_peered":"2021-03-15T22:35:50.345166+0000","last_clean":"2021-03-15T22:35:50.345166+0000","last_became_active":"2021-03-15T22:34:55.338001+0000","last_became_peered":"2021-03-15T22:34:55.338001+0000","last_unstale":"2021-03-15T22:35:50.345166+0000","last_undegraded":"2021-03-15T22:35:50.345166+0000","last_fullsized":"2021-03-15T22:35:50.345166+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:42.445911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.445911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"833","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.348791+0000","last_change":"2021-03-15T22:35:20.025959+0000","last_active":"2021-03-15T22:35:50.348791+0000","last_peered":"2021-03-15T22:35:50.348791+0000","last_clean":"2021-03-15T22:35:50.348791+0000","last_became_active":"2021-03-15T22:33:04.336804+0000","last_became_peered":"2021-03-15T22:33:04.336804+0000","last_unstale":"2021-03-15T22:35:50.348791+0000","last_undegraded":"2021-03-15T22:35:50.348791+0000","last_fullsized":"2021-03-15T22:35:50.348791+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.025892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"10","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.355311+0000","last_change":"2021-03-15T22:35:50.355311+0000","last_active":"2021-03-15T22:35:50.355311+0000","last_peered":"2021-03-15T22:35:50.355311+0000","last_clean":"2021-03-15T22:35:50.355311+0000","last_became_active":"2021-03-15T22:35:50.354946+0000","last_became_peered":"2021-03-15T22:35:50.354946+0000","last_unstale":"2021-03-15T22:35:50.355311+0000","last_undegraded":"2021-03-15T22:35:50.355311+0000","last_fullsized":"2021-03-15T22:35:50.355311+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.429029+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.429029+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"444","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.348112+0000","last_change":"2021-03-15T22:33:15.617975+0000","last_active":"2021-03-15T22:35:50.348112+0000","last_peered":"2021-03-15T22:35:50.348112+0000","last_clean":"2021-03-15T22:35:50.348112+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:35:50.348112+0000","last_undegraded":"2021-03-15T22:35:50.348112+0000","last_fullsized":"2021-03-15T22:35:50.348112+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:33:15.617891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.617891+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"891'22","reported_seq":"566","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357150+0000","last_change":"2021-03-15T22:35:38.710708+0000","last_active":"2021-03-15T22:35:49.357150+0000","last_peered":"2021-03-15T22:35:49.357150+0000","last_clean":"2021-03-15T22:35:49.357150+0000","last_became_active":"2021-03-15T22:34:06.311478+0000","last_became_peered":"2021-03-15T22:34:06.311478+0000","last_unstale":"2021-03-15T22:35:49.357150+0000","last_undegraded":"2021-03-15T22:35:49.357150+0000","last_fullsized":"2021-03-15T22:35:49.357150+0000","mapping_epoch":890,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"891'22","last_scrub_stamp":"2021-03-15T22:35:38.710643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:38.710643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1029","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362689+0000","last_change":"2021-03-15T22:34:55.352256+0000","last_active":"2021-03-15T22:35:52.362689+0000","last_peered":"2021-03-15T22:35:52.362689+0000","last_clean":"2021-03-15T22:35:52.362689+0000","last_became_active":"2021-03-15T22:34:55.351597+0000","last_became_peered":"2021-03-15T22:34:55.351597+0000","last_unstale":"2021-03-15T22:35:52.362689+0000","last_undegraded":"2021-03-15T22:35:52.362689+0000","last_fullsized":"2021-03-15T22:35:52.362689+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:43.393275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:43.393275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"607","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.358709+0000","last_change":"2021-03-15T22:35:50.357745+0000","last_active":"2021-03-15T22:35:50.358709+0000","last_peered":"2021-03-15T22:35:50.358709+0000","last_clean":"2021-03-15T22:35:50.358709+0000","last_became_active":"2021-03-15T22:35:50.357418+0000","last_became_peered":"2021-03-15T22:35:50.357418+0000","last_unstale":"2021-03-15T22:35:50.358709+0000","last_undegraded":"2021-03-15T22:35:50.358709+0000","last_fullsized":"2021-03-15T22:35:50.358709+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:41.082301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.082301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"597","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357624+0000","last_change":"2021-03-15T22:34:10.385041+0000","last_active":"2021-03-15T22:35:49.357624+0000","last_peered":"2021-03-15T22:35:49.357624+0000","last_clean":"2021-03-15T22:35:49.357624+0000","last_became_active":"2021-03-15T22:34:10.347227+0000","last_became_peered":"2021-03-15T22:34:10.347227+0000","last_unstale":"2021-03-15T22:35:49.357624+0000","last_undegraded":"2021-03-15T22:35:49.357624+0000","last_fullsized":"2021-03-15T22:35:49.357624+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:34:07.125155+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.125155+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":3736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"1073","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357045+0000","last_change":"2021-03-15T22:34:58.356466+0000","last_active":"2021-03-15T22:35:49.357045+0000","last_peered":"2021-03-15T22:35:49.357045+0000","last_clean":"2021-03-15T22:35:49.357045+0000","last_became_active":"2021-03-15T22:34:58.353086+0000","last_became_peered":"2021-03-15T22:34:58.353086+0000","last_unstale":"2021-03-15T22:35:49.357045+0000","last_undegraded":"2021-03-15T22:35:49.357045+0000","last_fullsized":"2021-03-15T22:35:49.357045+0000","mapping_epoch":942,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:32:34.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.739637+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"1006","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534639+0000","last_change":"2021-03-15T22:35:18.111879+0000","last_active":"2021-03-15T22:35:51.534639+0000","last_peered":"2021-03-15T22:35:51.534639+0000","last_clean":"2021-03-15T22:35:51.534639+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:35:51.534639+0000","last_undegraded":"2021-03-15T22:35:51.534639+0000","last_fullsized":"2021-03-15T22:35:51.534639+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:18.111815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.111815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"12","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534087+0000","last_change":"2021-03-15T22:35:50.359970+0000","last_active":"2021-03-15T22:35:51.534087+0000","last_peered":"2021-03-15T22:35:51.534087+0000","last_clean":"2021-03-15T22:35:51.534087+0000","last_became_active":"2021-03-15T22:35:50.359484+0000","last_became_peered":"2021-03-15T22:35:50.359484+0000","last_unstale":"2021-03-15T22:35:51.534087+0000","last_undegraded":"2021-03-15T22:35:51.534087+0000","last_fullsized":"2021-03-15T22:35:51.534087+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.844183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.844183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"203","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357096+0000","last_change":"2021-03-15T22:34:43.447677+0000","last_active":"2021-03-15T22:35:49.357096+0000","last_peered":"2021-03-15T22:35:49.357096+0000","last_clean":"2021-03-15T22:35:49.357096+0000","last_became_active":"2021-03-15T22:33:04.336100+0000","last_became_peered":"2021-03-15T22:33:04.336100+0000","last_unstale":"2021-03-15T22:35:49.357096+0000","last_undegraded":"2021-03-15T22:35:49.357096+0000","last_fullsized":"2021-03-15T22:35:49.357096+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:34:43.447602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:43.447602+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"1016","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.361743+0000","last_change":"2021-03-15T22:35:41.589000+0000","last_active":"2021-03-15T22:35:52.361743+0000","last_peered":"2021-03-15T22:35:52.361743+0000","last_clean":"2021-03-15T22:35:52.361743+0000","last_became_active":"2021-03-15T22:34:55.351089+0000","last_became_peered":"2021-03-15T22:34:55.351089+0000","last_unstale":"2021-03-15T22:35:52.361743+0000","last_undegraded":"2021-03-15T22:35:52.361743+0000","last_fullsized":"2021-03-15T22:35:52.361743+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:41.588935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.588935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"179","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.356746+0000","last_change":"2021-03-15T22:35:22.665325+0000","last_active":"2021-03-15T22:35:49.356746+0000","last_peered":"2021-03-15T22:35:49.356746+0000","last_clean":"2021-03-15T22:35:49.356746+0000","last_became_active":"2021-03-15T22:33:04.335734+0000","last_became_peered":"2021-03-15T22:33:04.335734+0000","last_unstale":"2021-03-15T22:35:49.356746+0000","last_undegraded":"2021-03-15T22:35:49.356746+0000","last_fullsized":"2021-03-15T22:35:49.356746+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.665251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.665251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"453","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.360359+0000","last_change":"2021-03-15T22:35:50.357197+0000","last_active":"2021-03-15T22:35:50.360359+0000","last_peered":"2021-03-15T22:35:50.360359+0000","last_clean":"2021-03-15T22:35:50.360359+0000","last_became_active":"2021-03-15T22:35:50.356790+0000","last_became_peered":"2021-03-15T22:35:50.356790+0000","last_unstale":"2021-03-15T22:35:50.360359+0000","last_undegraded":"2021-03-15T22:35:50.360359+0000","last_fullsized":"2021-03-15T22:35:50.360359+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.291406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:22.291406+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.291406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"508","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.346871+0000","last_change":"2021-03-15T22:33:37.752077+0000","last_active":"2021-03-15T22:35:50.346871+0000","last_peered":"2021-03-15T22:35:50.346871+0000","last_clean":"2021-03-15T22:35:50.346871+0000","last_became_active":"2021-03-15T22:33:04.342749+0000","last_became_peered":"2021-03-15T22:33:04.342749+0000","last_unstale":"2021-03-15T22:35:50.346871+0000","last_undegraded":"2021-03-15T22:35:50.346871+0000","last_fullsized":"2021-03-15T22:35:50.346871+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:33:37.752011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.752011+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":8764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"621","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347600+0000","last_change":"2021-03-15T22:34:34.783107+0000","last_active":"2021-03-15T22:35:50.347600+0000","last_peered":"2021-03-15T22:35:50.347600+0000","last_clean":"2021-03-15T22:35:50.347600+0000","last_became_active":"2021-03-15T22:33:04.333077+0000","last_became_peered":"2021-03-15T22:33:04.333077+0000","last_unstale":"2021-03-15T22:35:50.347600+0000","last_undegraded":"2021-03-15T22:35:50.347600+0000","last_fullsized":"2021-03-15T22:35:50.347600+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.783047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.783047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"122","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347014+0000","last_change":"2021-03-15T22:35:48.884382+0000","last_active":"2021-03-15T22:35:50.347014+0000","last_peered":"2021-03-15T22:35:50.347014+0000","last_clean":"2021-03-15T22:35:50.347014+0000","last_became_active":"2021-03-15T22:34:03.571049+0000","last_became_peered":"2021-03-15T22:34:03.571049+0000","last_unstale":"2021-03-15T22:35:50.347014+0000","last_undegraded":"2021-03-15T22:35:50.347014+0000","last_fullsized":"2021-03-15T22:35:50.347014+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:48.884317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:48.884317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"10","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.350018+0000","last_change":"2021-03-15T22:35:50.350018+0000","last_active":"2021-03-15T22:35:50.350018+0000","last_peered":"2021-03-15T22:35:50.350018+0000","last_clean":"2021-03-15T22:35:50.350018+0000","last_became_active":"2021-03-15T22:35:50.349656+0000","last_became_peered":"2021-03-15T22:35:50.349656+0000","last_unstale":"2021-03-15T22:35:50.350018+0000","last_undegraded":"2021-03-15T22:35:50.350018+0000","last_fullsized":"2021-03-15T22:35:50.350018+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.882164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.882164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"839","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347498+0000","last_change":"2021-03-15T22:34:37.787302+0000","last_active":"2021-03-15T22:35:50.347498+0000","last_peered":"2021-03-15T22:35:50.347498+0000","last_clean":"2021-03-15T22:35:50.347498+0000","last_became_active":"2021-03-15T22:33:04.331607+0000","last_became_peered":"2021-03-15T22:33:04.331607+0000","last_unstale":"2021-03-15T22:35:50.347498+0000","last_undegraded":"2021-03-15T22:35:50.347498+0000","last_fullsized":"2021-03-15T22:35:50.347498+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:34:37.787235+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.787235+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"483","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357181+0000","last_change":"2021-03-15T22:35:17.250496+0000","last_active":"2021-03-15T22:35:49.357181+0000","last_peered":"2021-03-15T22:35:49.357181+0000","last_clean":"2021-03-15T22:35:49.357181+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:35:49.357181+0000","last_undegraded":"2021-03-15T22:35:49.357181+0000","last_fullsized":"2021-03-15T22:35:49.357181+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.250434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.250434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"788","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347514+0000","last_change":"2021-03-15T22:32:37.690585+0000","last_active":"2021-03-15T22:35:50.347514+0000","last_peered":"2021-03-15T22:35:50.347514+0000","last_clean":"2021-03-15T22:35:50.347514+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:35:50.347514+0000","last_undegraded":"2021-03-15T22:35:50.347514+0000","last_fullsized":"2021-03-15T22:35:50.347514+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:37.690525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:32:37.690525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"1004","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362280+0000","last_change":"2021-03-15T22:35:37.543358+0000","last_active":"2021-03-15T22:35:52.362280+0000","last_peered":"2021-03-15T22:35:52.362280+0000","last_clean":"2021-03-15T22:35:52.362280+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:35:52.362280+0000","last_undegraded":"2021-03-15T22:35:52.362280+0000","last_fullsized":"2021-03-15T22:35:52.362280+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:37.543289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:35:37.543289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"710","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.356441+0000","last_change":"2021-03-15T22:34:56.210802+0000","last_active":"2021-03-15T22:35:49.356441+0000","last_peered":"2021-03-15T22:35:49.356441+0000","last_clean":"2021-03-15T22:35:49.356441+0000","last_became_active":"2021-03-15T22:34:55.344555+0000","last_became_peered":"2021-03-15T22:34:55.344555+0000","last_unstale":"2021-03-15T22:35:49.356441+0000","last_undegraded":"2021-03-15T22:35:49.356441+0000","last_fullsized":"2021-03-15T22:35:49.356441+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:34:42.447631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.447631+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"380","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357077+0000","last_change":"2021-03-15T22:35:23.291568+0000","last_active":"2021-03-15T22:35:49.357077+0000","last_peered":"2021-03-15T22:35:49.357077+0000","last_clean":"2021-03-15T22:35:49.357077+0000","last_became_active":"2021-03-15T22:34:55.355119+0000","last_became_peered":"2021-03-15T22:34:55.355119+0000","last_unstale":"2021-03-15T22:35:49.357077+0000","last_undegraded":"2021-03-15T22:35:49.357077+0000","last_fullsized":"2021-03-15T22:35:49.357077+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.291507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.291507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"1059","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534359+0000","last_change":"2021-03-15T22:35:50.069546+0000","last_active":"2021-03-15T22:35:51.534359+0000","last_peered":"2021-03-15T22:35:51.534359+0000","last_clean":"2021-03-15T22:35:51.534359+0000","last_became_active":"2021-03-15T22:34:10.353314+0000","last_became_peered":"2021-03-15T22:34:10.353314+0000","last_unstale":"2021-03-15T22:35:51.534359+0000","last_undegraded":"2021-03-15T22:35:51.534359+0000","last_fullsized":"2021-03-15T22:35:51.534359+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:50.069463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:35:50.069463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"131","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344971+0000","last_change":"2021-03-15T22:33:45.314503+0000","last_active":"2021-03-15T22:35:50.344971+0000","last_peered":"2021-03-15T22:35:50.344971+0000","last_clean":"2021-03-15T22:35:50.344971+0000","last_became_active":"2021-03-15T22:33:45.314042+0000","last_became_peered":"2021-03-15T22:33:45.314042+0000","last_unstale":"2021-03-15T22:35:50.344971+0000","last_undegraded":"2021-03-15T22:35:50.344971+0000","last_fullsized":"2021-03-15T22:35:50.344971+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:08.884878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.884878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"1082","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344791+0000","last_change":"2021-03-15T22:34:42.632034+0000","last_active":"2021-03-15T22:35:50.344791+0000","last_peered":"2021-03-15T22:35:50.344791+0000","last_clean":"2021-03-15T22:35:50.344791+0000","last_became_active":"2021-03-15T22:33:04.340460+0000","last_became_peered":"2021-03-15T22:33:04.340460+0000","last_unstale":"2021-03-15T22:35:50.344791+0000","last_undegraded":"2021-03-15T22:35:50.344791+0000","last_fullsized":"2021-03-15T22:35:50.344791+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:34:42.631962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.631962+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"486","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357799+0000","last_change":"2021-03-15T22:34:10.327350+0000","last_active":"2021-03-15T22:35:49.357799+0000","last_peered":"2021-03-15T22:35:49.357799+0000","last_clean":"2021-03-15T22:35:49.357799+0000","last_became_active":"2021-03-15T22:34:10.326881+0000","last_became_peered":"2021-03-15T22:34:10.326881+0000","last_unstale":"2021-03-15T22:35:49.357799+0000","last_undegraded":"2021-03-15T22:35:49.357799+0000","last_fullsized":"2021-03-15T22:35:49.357799+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:19.560387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:19.560387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"808'9","reported_seq":"456","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.351930+0000","last_change":"2021-03-15T22:35:41.986386+0000","last_active":"2021-03-15T22:35:52.351930+0000","last_peered":"2021-03-15T22:35:52.351930+0000","last_clean":"2021-03-15T22:35:52.351930+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:35:52.351930+0000","last_undegraded":"2021-03-15T22:35:52.351930+0000","last_fullsized":"2021-03-15T22:35:52.351930+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9","last_scrub_stamp":"2021-03-15T22:35:41.986318+0000","last_deep_scrub":"808'9","last_deep_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.986318+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"996","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.361917+0000","last_change":"2021-03-15T22:34:55.351414+0000","last_active":"2021-03-15T22:35:52.361917+0000","last_peered":"2021-03-15T22:35:52.361917+0000","last_clean":"2021-03-15T22:35:52.361917+0000","last_became_active":"2021-03-15T22:34:55.351042+0000","last_became_peered":"2021-03-15T22:34:55.351042+0000","last_unstale":"2021-03-15T22:35:52.361917+0000","last_undegraded":"2021-03-15T22:35:52.361917+0000","last_fullsized":"2021-03-15T22:35:52.361917+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.516668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.516668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"844'33","reported_seq":"854","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347980+0000","last_change":"2021-03-15T22:34:55.387712+0000","last_active":"2021-03-15T22:35:50.347980+0000","last_peered":"2021-03-15T22:35:50.347980+0000","last_clean":"2021-03-15T22:35:50.347980+0000","last_became_active":"2021-03-15T22:34:55.351721+0000","last_became_peered":"2021-03-15T22:34:55.351721+0000","last_unstale":"2021-03-15T22:35:50.347980+0000","last_undegraded":"2021-03-15T22:35:50.347980+0000","last_fullsized":"2021-03-15T22:35:50.347980+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"844'33","last_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_deep_scrub":"844'33","last_deep_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.622437+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1035","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.345392+0000","last_change":"2021-03-15T22:35:40.412203+0000","last_active":"2021-03-15T22:35:50.345392+0000","last_peered":"2021-03-15T22:35:50.345392+0000","last_clean":"2021-03-15T22:35:50.345392+0000","last_became_active":"2021-03-15T22:34:55.340440+0000","last_became_peered":"2021-03-15T22:34:55.340440+0000","last_unstale":"2021-03-15T22:35:50.345392+0000","last_undegraded":"2021-03-15T22:35:50.345392+0000","last_fullsized":"2021-03-15T22:35:50.345392+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:40.412142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:40.412142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"941'17","reported_seq":"526","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362644+0000","last_change":"2021-03-15T22:34:55.473459+0000","last_active":"2021-03-15T22:35:52.362644+0000","last_peered":"2021-03-15T22:35:52.362644+0000","last_clean":"2021-03-15T22:35:52.362644+0000","last_became_active":"2021-03-15T22:34:55.351682+0000","last_became_peered":"2021-03-15T22:34:55.351682+0000","last_unstale":"2021-03-15T22:35:52.362644+0000","last_undegraded":"2021-03-15T22:35:52.362644+0000","last_fullsized":"2021-03-15T22:35:52.362644+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"896'16","last_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_deep_scrub":"896'16","last_deep_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_clean_scrub_stamp":"2021-03-15T22:34:19.400191+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"486","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.356947+0000","last_change":"2021-03-15T22:34:10.347844+0000","last_active":"2021-03-15T22:35:49.356947+0000","last_peered":"2021-03-15T22:35:49.356947+0000","last_clean":"2021-03-15T22:35:49.356947+0000","last_became_active":"2021-03-15T22:34:10.347512+0000","last_became_peered":"2021-03-15T22:34:10.347512+0000","last_unstale":"2021-03-15T22:35:49.356947+0000","last_undegraded":"2021-03-15T22:35:49.356947+0000","last_fullsized":"2021-03-15T22:35:49.356947+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:03.502854+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:32:03.502854+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"391","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.352222+0000","last_change":"2021-03-15T22:32:41.879032+0000","last_active":"2021-03-15T22:35:52.352222+0000","last_peered":"2021-03-15T22:35:52.352222+0000","last_clean":"2021-03-15T22:35:52.352222+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:35:52.352222+0000","last_undegraded":"2021-03-15T22:35:52.352222+0000","last_fullsized":"2021-03-15T22:35:52.352222+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:32:41.878949+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.878949+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"78","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347578+0000","last_change":"2021-03-15T22:34:43.345092+0000","last_active":"2021-03-15T22:35:50.347578+0000","last_peered":"2021-03-15T22:35:50.347578+0000","last_clean":"2021-03-15T22:35:50.347578+0000","last_became_active":"2021-03-15T22:34:43.344790+0000","last_became_peered":"2021-03-15T22:34:43.344790+0000","last_unstale":"2021-03-15T22:35:50.347578+0000","last_undegraded":"2021-03-15T22:35:50.347578+0000","last_fullsized":"2021-03-15T22:35:50.347578+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.621539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"69","reported_epoch":"993","state":"active+clean","last_fresh":"2021-03-15T22:35:48.348676+0000","last_change":"2021-03-15T22:35:20.381407+0000","last_active":"2021-03-15T22:35:48.348676+0000","last_peered":"2021-03-15T22:35:48.348676+0000","last_clean":"2021-03-15T22:35:48.348676+0000","last_became_active":"2021-03-15T22:34:55.348663+0000","last_became_peered":"2021-03-15T22:34:55.348663+0000","last_unstale":"2021-03-15T22:35:48.348676+0000","last_undegraded":"2021-03-15T22:35:48.348676+0000","last_fullsized":"2021-03-15T22:35:48.348676+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.381352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"994","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.361413+0000","last_change":"2021-03-15T22:35:48.581266+0000","last_active":"2021-03-15T22:35:52.361413+0000","last_peered":"2021-03-15T22:35:52.361413+0000","last_clean":"2021-03-15T22:35:52.361413+0000","last_became_active":"2021-03-15T22:33:04.330195+0000","last_became_peered":"2021-03-15T22:33:04.330195+0000","last_unstale":"2021-03-15T22:35:52.361413+0000","last_undegraded":"2021-03-15T22:35:52.361413+0000","last_fullsized":"2021-03-15T22:35:52.361413+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:48.581162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:48.581162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"522","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357317+0000","last_change":"2021-03-15T22:34:37.435070+0000","last_active":"2021-03-15T22:35:49.357317+0000","last_peered":"2021-03-15T22:35:49.357317+0000","last_clean":"2021-03-15T22:35:49.357317+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:35:49.357317+0000","last_undegraded":"2021-03-15T22:35:49.357317+0000","last_fullsized":"2021-03-15T22:35:49.357317+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:34:37.435006+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.435006+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"800","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.352551+0000","last_change":"2021-03-15T22:35:20.884067+0000","last_active":"2021-03-15T22:35:52.352551+0000","last_peered":"2021-03-15T22:35:52.352551+0000","last_clean":"2021-03-15T22:35:52.352551+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:35:52.352551+0000","last_undegraded":"2021-03-15T22:35:52.352551+0000","last_fullsized":"2021-03-15T22:35:52.352551+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.883990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.883990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"988","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362240+0000","last_change":"2021-03-15T22:35:45.547459+0000","last_active":"2021-03-15T22:35:52.362240+0000","last_peered":"2021-03-15T22:35:52.362240+0000","last_clean":"2021-03-15T22:35:52.362240+0000","last_became_active":"2021-03-15T22:33:04.331013+0000","last_became_peered":"2021-03-15T22:33:04.331013+0000","last_unstale":"2021-03-15T22:35:52.362240+0000","last_undegraded":"2021-03-15T22:35:52.362240+0000","last_fullsized":"2021-03-15T22:35:52.362240+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.547395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:45.547395+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.547395+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"179","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.356185+0000","last_change":"2021-03-15T22:35:45.757312+0000","last_active":"2021-03-15T22:35:49.356185+0000","last_peered":"2021-03-15T22:35:49.356185+0000","last_clean":"2021-03-15T22:35:49.356185+0000","last_became_active":"2021-03-15T22:33:04.336886+0000","last_became_peered":"2021-03-15T22:33:04.336886+0000","last_unstale":"2021-03-15T22:35:49.356185+0000","last_undegraded":"2021-03-15T22:35:49.356185+0000","last_fullsized":"2021-03-15T22:35:49.356185+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.757254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:45.757254+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.757254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"347","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.363082+0000","last_change":"2021-03-15T22:34:55.387006+0000","last_active":"2021-03-15T22:35:52.363082+0000","last_peered":"2021-03-15T22:35:52.363082+0000","last_clean":"2021-03-15T22:35:52.363082+0000","last_became_active":"2021-03-15T22:34:55.351977+0000","last_became_peered":"2021-03-15T22:34:55.351977+0000","last_unstale":"2021-03-15T22:35:52.363082+0000","last_undegraded":"2021-03-15T22:35:52.363082+0000","last_fullsized":"2021-03-15T22:35:52.363082+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:34:36.438541+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:34:36.438541+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"995'1425","reported_seq":"4528","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534509+0000","last_change":"2021-03-15T22:35:50.366827+0000","last_active":"2021-03-15T22:35:51.534509+0000","last_peered":"2021-03-15T22:35:51.534509+0000","last_clean":"2021-03-15T22:35:51.534509+0000","last_became_active":"2021-03-15T22:33:08.355490+0000","last_became_peered":"2021-03-15T22:33:08.355490+0000","last_unstale":"2021-03-15T22:35:51.534509+0000","last_undegraded":"2021-03-15T22:35:51.534509+0000","last_fullsized":"2021-03-15T22:35:51.534509+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'1350","last_scrub_stamp":"2021-03-15T22:35:16.036089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.036089+0000","log_size":1425,"ondisk_log_size":1425,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50100108,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":1,"num_read":481,"num_read_kb":537985,"num_write":335,"num_write_kb":119214,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":84869361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"940","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.352333+0000","last_change":"2021-03-15T22:33:42.741529+0000","last_active":"2021-03-15T22:35:52.352333+0000","last_peered":"2021-03-15T22:35:52.352333+0000","last_clean":"2021-03-15T22:35:52.352333+0000","last_became_active":"2021-03-15T22:33:04.340352+0000","last_became_peered":"2021-03-15T22:33:04.340352+0000","last_unstale":"2021-03-15T22:35:52.352333+0000","last_undegraded":"2021-03-15T22:35:52.352333+0000","last_fullsized":"2021-03-15T22:35:52.352333+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:42.741451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:42.741451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"874'4","reported_seq":"1006","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362997+0000","last_change":"2021-03-15T22:33:44.498327+0000","last_active":"2021-03-15T22:35:52.362997+0000","last_peered":"2021-03-15T22:35:52.362997+0000","last_clean":"2021-03-15T22:35:52.362997+0000","last_became_active":"2021-03-15T22:28:12.508161+0000","last_became_peered":"2021-03-15T22:28:12.508161+0000","last_unstale":"2021-03-15T22:35:52.362997+0000","last_undegraded":"2021-03-15T22:35:52.362997+0000","last_fullsized":"2021-03-15T22:35:52.362997+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"874'4","last_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_deep_scrub":"874'4","last_deep_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.498263+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"467","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.351566+0000","last_change":"2021-03-15T22:33:38.759477+0000","last_active":"2021-03-15T22:35:52.351566+0000","last_peered":"2021-03-15T22:35:52.351566+0000","last_clean":"2021-03-15T22:35:52.351566+0000","last_became_active":"2021-03-15T22:33:04.326070+0000","last_became_peered":"2021-03-15T22:33:04.326070+0000","last_unstale":"2021-03-15T22:35:52.351566+0000","last_undegraded":"2021-03-15T22:35:52.351566+0000","last_fullsized":"2021-03-15T22:35:52.351566+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:33:38.759395+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.759395+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"841'14","reported_seq":"1267","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.361502+0000","last_change":"2021-03-15T22:33:43.476783+0000","last_active":"2021-03-15T22:35:52.361502+0000","last_peered":"2021-03-15T22:35:52.361502+0000","last_clean":"2021-03-15T22:35:52.361502+0000","last_became_active":"2021-03-15T22:33:08.354031+0000","last_became_peered":"2021-03-15T22:33:08.354031+0000","last_unstale":"2021-03-15T22:35:52.361502+0000","last_undegraded":"2021-03-15T22:35:52.361502+0000","last_fullsized":"2021-03-15T22:35:52.361502+0000","mapping_epoch":837,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'14","last_scrub_stamp":"2021-03-15T22:33:43.476701+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.476701+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"995'2060","reported_seq":"5435","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.382178+0000","last_change":"2021-03-15T22:35:50.382178+0000","last_active":"2021-03-15T22:35:50.382178+0000","last_peered":"2021-03-15T22:35:50.382178+0000","last_clean":"2021-03-15T22:35:50.382178+0000","last_became_active":"2021-03-15T22:34:58.336162+0000","last_became_peered":"2021-03-15T22:34:58.336162+0000","last_unstale":"2021-03-15T22:35:50.382178+0000","last_undegraded":"2021-03-15T22:35:50.382178+0000","last_fullsized":"2021-03-15T22:35:50.382178+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'2031","last_scrub_stamp":"2021-03-15T22:35:42.023328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.023328+0000","log_size":2060,"ondisk_log_size":2060,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":77802644,"num_objects":35,"num_object_clones":30,"num_object_copies":105,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":35,"num_whiteouts":1,"num_read":685,"num_read_kb":649872,"num_write":529,"num_write_kb":208282,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":198498497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"424","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347371+0000","last_change":"2021-03-15T22:32:42.621654+0000","last_active":"2021-03-15T22:35:50.347371+0000","last_peered":"2021-03-15T22:35:50.347371+0000","last_clean":"2021-03-15T22:35:50.347371+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:35:50.347371+0000","last_undegraded":"2021-03-15T22:35:50.347371+0000","last_fullsized":"2021-03-15T22:35:50.347371+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:42.621594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:42.621594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"978","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362551+0000","last_change":"2021-03-15T22:35:47.601522+0000","last_active":"2021-03-15T22:35:52.362551+0000","last_peered":"2021-03-15T22:35:52.362551+0000","last_clean":"2021-03-15T22:35:52.362551+0000","last_became_active":"2021-03-15T22:23:52.263791+0000","last_became_peered":"2021-03-15T22:23:52.263791+0000","last_unstale":"2021-03-15T22:35:52.362551+0000","last_undegraded":"2021-03-15T22:35:52.362551+0000","last_fullsized":"2021-03-15T22:35:52.362551+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":317,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:47.601457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.601457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"1205","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.345660+0000","last_change":"2021-03-15T22:34:11.031994+0000","last_active":"2021-03-15T22:35:50.345660+0000","last_peered":"2021-03-15T22:35:50.345660+0000","last_clean":"2021-03-15T22:35:50.345660+0000","last_became_active":"2021-03-15T22:34:10.342120+0000","last_became_peered":"2021-03-15T22:34:10.342120+0000","last_unstale":"2021-03-15T22:35:50.345660+0000","last_undegraded":"2021-03-15T22:35:50.345660+0000","last_fullsized":"2021-03-15T22:35:50.345660+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:34:07.452572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.452572+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"707","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357262+0000","last_change":"2021-03-15T22:34:55.356651+0000","last_active":"2021-03-15T22:35:49.357262+0000","last_peered":"2021-03-15T22:35:49.357262+0000","last_clean":"2021-03-15T22:35:49.357262+0000","last_became_active":"2021-03-15T22:34:55.356328+0000","last_became_peered":"2021-03-15T22:34:55.356328+0000","last_unstale":"2021-03-15T22:35:49.357262+0000","last_undegraded":"2021-03-15T22:35:49.357262+0000","last_fullsized":"2021-03-15T22:35:49.357262+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"995'570","reported_seq":"3388","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.359449+0000","last_change":"2021-03-15T22:35:50.359647+0000","last_active":"2021-03-15T22:35:50.359449+0000","last_peered":"2021-03-15T22:35:50.359449+0000","last_clean":"2021-03-15T22:35:50.359449+0000","last_became_active":"2021-03-15T22:34:13.324667+0000","last_became_peered":"2021-03-15T22:34:13.324667+0000","last_unstale":"2021-03-15T22:35:50.359449+0000","last_undegraded":"2021-03-15T22:35:50.359449+0000","last_fullsized":"2021-03-15T22:35:50.359449+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'446","last_scrub_stamp":"2021-03-15T22:33:39.355306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.355306+0000","log_size":570,"ondisk_log_size":570,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17801208,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":139,"num_read_kb":111592,"num_write":131,"num_write_kb":42276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":23698344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"1059","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.361263+0000","last_change":"2021-03-15T22:35:48.365854+0000","last_active":"2021-03-15T22:35:52.361263+0000","last_peered":"2021-03-15T22:35:52.361263+0000","last_clean":"2021-03-15T22:35:52.361263+0000","last_became_active":"2021-03-15T22:35:48.365533+0000","last_became_peered":"2021-03-15T22:35:48.365533+0000","last_unstale":"2021-03-15T22:35:52.361263+0000","last_undegraded":"2021-03-15T22:35:52.361263+0000","last_fullsized":"2021-03-15T22:35:52.361263+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":993,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"64","reported_epoch":"993","state":"active+clean","last_fresh":"2021-03-15T22:35:48.349822+0000","last_change":"2021-03-15T22:34:55.354467+0000","last_active":"2021-03-15T22:35:48.349822+0000","last_peered":"2021-03-15T22:35:48.349822+0000","last_clean":"2021-03-15T22:35:48.349822+0000","last_became_active":"2021-03-15T22:34:55.353966+0000","last_became_peered":"2021-03-15T22:34:55.353966+0000","last_unstale":"2021-03-15T22:35:48.349822+0000","last_undegraded":"2021-03-15T22:35:48.349822+0000","last_fullsized":"2021-03-15T22:35:48.349822+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.809863+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.809863+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"1033","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362418+0000","last_change":"2021-03-15T22:35:23.428929+0000","last_active":"2021-03-15T22:35:52.362418+0000","last_peered":"2021-03-15T22:35:52.362418+0000","last_clean":"2021-03-15T22:35:52.362418+0000","last_became_active":"2021-03-15T22:33:04.330771+0000","last_became_peered":"2021-03-15T22:33:04.330771+0000","last_unstale":"2021-03-15T22:35:52.362418+0000","last_undegraded":"2021-03-15T22:35:52.362418+0000","last_fullsized":"2021-03-15T22:35:52.362418+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.428842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.428842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"244","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.356546+0000","last_change":"2021-03-15T22:35:47.782159+0000","last_active":"2021-03-15T22:35:49.356546+0000","last_peered":"2021-03-15T22:35:49.356546+0000","last_clean":"2021-03-15T22:35:49.356546+0000","last_became_active":"2021-03-15T22:33:45.318867+0000","last_became_peered":"2021-03-15T22:33:45.318867+0000","last_unstale":"2021-03-15T22:35:49.356546+0000","last_undegraded":"2021-03-15T22:35:49.356546+0000","last_fullsized":"2021-03-15T22:35:49.356546+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:35:47.782092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.782092+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":2928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"993'1258","reported_seq":"4150","reported_epoch":"993","state":"active+clean","last_fresh":"2021-03-15T22:35:48.352800+0000","last_change":"2021-03-15T22:35:48.352800+0000","last_active":"2021-03-15T22:35:48.352800+0000","last_peered":"2021-03-15T22:35:48.352800+0000","last_clean":"2021-03-15T22:35:48.352800+0000","last_became_active":"2021-03-15T22:34:59.339587+0000","last_became_peered":"2021-03-15T22:34:59.339587+0000","last_unstale":"2021-03-15T22:35:48.352800+0000","last_undegraded":"2021-03-15T22:35:48.352800+0000","last_fullsized":"2021-03-15T22:35:48.352800+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'1086","last_scrub_stamp":"2021-03-15T22:34:41.436839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.436839+0000","log_size":1258,"ondisk_log_size":1258,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51621216,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":427,"num_read_kb":511499,"num_write":287,"num_write_kb":105085,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":205148835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"2e2","length":"1"},{"start":"2f3","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"515","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357180+0000","last_change":"2021-03-15T22:35:44.582634+0000","last_active":"2021-03-15T22:35:49.357180+0000","last_peered":"2021-03-15T22:35:49.357180+0000","last_clean":"2021-03-15T22:35:49.357180+0000","last_became_active":"2021-03-15T22:35:44.582322+0000","last_became_peered":"2021-03-15T22:35:44.582322+0000","last_unstale":"2021-03-15T22:35:49.357180+0000","last_undegraded":"2021-03-15T22:35:49.357180+0000","last_fullsized":"2021-03-15T22:35:49.357180+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"829","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344871+0000","last_change":"2021-03-15T22:34:55.343547+0000","last_active":"2021-03-15T22:35:50.344871+0000","last_peered":"2021-03-15T22:35:50.344871+0000","last_clean":"2021-03-15T22:35:50.344871+0000","last_became_active":"2021-03-15T22:34:55.342808+0000","last_became_peered":"2021-03-15T22:34:55.342808+0000","last_unstale":"2021-03-15T22:35:50.344871+0000","last_undegraded":"2021-03-15T22:35:50.344871+0000","last_fullsized":"2021-03-15T22:35:50.344871+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:14.888119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.888119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"398","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.351923+0000","last_change":"2021-03-15T22:34:07.807466+0000","last_active":"2021-03-15T22:35:52.351923+0000","last_peered":"2021-03-15T22:35:52.351923+0000","last_clean":"2021-03-15T22:35:52.351923+0000","last_became_active":"2021-03-15T22:33:04.333291+0000","last_became_peered":"2021-03-15T22:33:04.333291+0000","last_unstale":"2021-03-15T22:35:52.351923+0000","last_undegraded":"2021-03-15T22:35:52.351923+0000","last_fullsized":"2021-03-15T22:35:52.351923+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.807411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.807411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"537","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357481+0000","last_change":"2021-03-15T22:33:46.129541+0000","last_active":"2021-03-15T22:35:49.357481+0000","last_peered":"2021-03-15T22:35:49.357481+0000","last_clean":"2021-03-15T22:35:49.357481+0000","last_became_active":"2021-03-15T22:33:45.326747+0000","last_became_peered":"2021-03-15T22:33:45.326747+0000","last_unstale":"2021-03-15T22:35:49.357481+0000","last_undegraded":"2021-03-15T22:35:49.357481+0000","last_fullsized":"2021-03-15T22:35:49.357481+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:32:33.701675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.701675+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":2682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"995'1430","reported_seq":"5008","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.380611+0000","last_change":"2021-03-15T22:35:50.380918+0000","last_active":"2021-03-15T22:35:50.380611+0000","last_peered":"2021-03-15T22:35:50.380611+0000","last_clean":"2021-03-15T22:35:50.380611+0000","last_became_active":"2021-03-15T22:34:46.338888+0000","last_became_peered":"2021-03-15T22:34:46.338888+0000","last_unstale":"2021-03-15T22:35:50.380611+0000","last_undegraded":"2021-03-15T22:35:50.380611+0000","last_fullsized":"2021-03-15T22:35:50.380611+0000","mapping_epoch":930,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'1253","last_scrub_stamp":"2021-03-15T22:34:41.490879+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.490879+0000","log_size":1430,"ondisk_log_size":1430,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":44614550,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":340,"num_read_kb":290239,"num_write":345,"num_write_kb":124940,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":229919087,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"78","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.348318+0000","last_change":"2021-03-15T22:34:43.342616+0000","last_active":"2021-03-15T22:35:50.348318+0000","last_peered":"2021-03-15T22:35:50.348318+0000","last_clean":"2021-03-15T22:35:50.348318+0000","last_became_active":"2021-03-15T22:34:43.342089+0000","last_became_peered":"2021-03-15T22:34:43.342089+0000","last_unstale":"2021-03-15T22:35:50.348318+0000","last_undegraded":"2021-03-15T22:35:50.348318+0000","last_fullsized":"2021-03-15T22:35:50.348318+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"44","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.361602+0000","last_change":"2021-03-15T22:35:19.502186+0000","last_active":"2021-03-15T22:35:52.361602+0000","last_peered":"2021-03-15T22:35:52.361602+0000","last_clean":"2021-03-15T22:35:52.361602+0000","last_became_active":"2021-03-15T22:35:19.501783+0000","last_became_peered":"2021-03-15T22:35:19.501783+0000","last_unstale":"2021-03-15T22:35:52.361602+0000","last_undegraded":"2021-03-15T22:35:52.361602+0000","last_fullsized":"2021-03-15T22:35:52.361602+0000","mapping_epoch":963,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":964,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.831369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.831369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"193","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344563+0000","last_change":"2021-03-15T22:33:04.336661+0000","last_active":"2021-03-15T22:35:50.344563+0000","last_peered":"2021-03-15T22:35:50.344563+0000","last_clean":"2021-03-15T22:35:50.344563+0000","last_became_active":"2021-03-15T22:33:04.335914+0000","last_became_peered":"2021-03-15T22:33:04.335914+0000","last_unstale":"2021-03-15T22:35:50.344563+0000","last_undegraded":"2021-03-15T22:35:50.344563+0000","last_fullsized":"2021-03-15T22:35:50.344563+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.051962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.051962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1369","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.533953+0000","last_change":"2021-03-15T22:35:47.126971+0000","last_active":"2021-03-15T22:35:51.533953+0000","last_peered":"2021-03-15T22:35:51.533953+0000","last_clean":"2021-03-15T22:35:51.533953+0000","last_became_active":"2021-03-15T22:34:55.350419+0000","last_became_peered":"2021-03-15T22:34:55.350419+0000","last_unstale":"2021-03-15T22:35:51.533953+0000","last_undegraded":"2021-03-15T22:35:51.533953+0000","last_fullsized":"2021-03-15T22:35:51.533953+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:35:47.126904+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.126904+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"459","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.356538+0000","last_change":"2021-03-15T22:33:15.492116+0000","last_active":"2021-03-15T22:35:49.356538+0000","last_peered":"2021-03-15T22:35:49.356538+0000","last_clean":"2021-03-15T22:35:49.356538+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:35:49.356538+0000","last_undegraded":"2021-03-15T22:35:49.356538+0000","last_fullsized":"2021-03-15T22:35:49.356538+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.492014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.492014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"995'1342","reported_seq":"4032","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.365441+0000","last_change":"2021-03-15T22:35:50.365880+0000","last_active":"2021-03-15T22:35:50.365441+0000","last_peered":"2021-03-15T22:35:50.365441+0000","last_clean":"2021-03-15T22:35:50.365441+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:35:50.365441+0000","last_undegraded":"2021-03-15T22:35:50.365441+0000","last_fullsized":"2021-03-15T22:35:50.365441+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"805'997","last_scrub_stamp":"2021-03-15T22:32:36.795301+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.795301+0000","log_size":1342,"ondisk_log_size":1342,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50504318,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":399,"num_read_kb":397986,"num_write":308,"num_write_kb":116042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"1029","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362490+0000","last_change":"2021-03-15T22:35:44.582587+0000","last_active":"2021-03-15T22:35:52.362490+0000","last_peered":"2021-03-15T22:35:52.362490+0000","last_clean":"2021-03-15T22:35:52.362490+0000","last_became_active":"2021-03-15T22:34:34.631216+0000","last_became_peered":"2021-03-15T22:34:34.631216+0000","last_unstale":"2021-03-15T22:35:52.362490+0000","last_undegraded":"2021-03-15T22:35:52.362490+0000","last_fullsized":"2021-03-15T22:35:52.362490+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":919,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:44.582515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:44.582515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"1028","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.533889+0000","last_change":"2021-03-15T22:34:55.347340+0000","last_active":"2021-03-15T22:35:51.533889+0000","last_peered":"2021-03-15T22:35:51.533889+0000","last_clean":"2021-03-15T22:35:51.533889+0000","last_became_active":"2021-03-15T22:34:55.347065+0000","last_became_peered":"2021-03-15T22:34:55.347065+0000","last_unstale":"2021-03-15T22:35:51.533889+0000","last_undegraded":"2021-03-15T22:35:51.533889+0000","last_fullsized":"2021-03-15T22:35:51.533889+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:10.222521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:10.222521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"69","reported_epoch":"993","state":"active+clean","last_fresh":"2021-03-15T22:35:48.348374+0000","last_change":"2021-03-15T22:35:17.301942+0000","last_active":"2021-03-15T22:35:48.348374+0000","last_peered":"2021-03-15T22:35:48.348374+0000","last_clean":"2021-03-15T22:35:48.348374+0000","last_became_active":"2021-03-15T22:34:55.355485+0000","last_became_peered":"2021-03-15T22:34:55.355485+0000","last_unstale":"2021-03-15T22:35:48.348374+0000","last_undegraded":"2021-03-15T22:35:48.348374+0000","last_fullsized":"2021-03-15T22:35:48.348374+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.301892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.301892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"481","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.352109+0000","last_change":"2021-03-15T22:34:10.343282+0000","last_active":"2021-03-15T22:35:52.352109+0000","last_peered":"2021-03-15T22:35:52.352109+0000","last_clean":"2021-03-15T22:35:52.352109+0000","last_became_active":"2021-03-15T22:34:10.322666+0000","last_became_peered":"2021-03-15T22:34:10.322666+0000","last_unstale":"2021-03-15T22:35:52.352109+0000","last_undegraded":"2021-03-15T22:35:52.352109+0000","last_fullsized":"2021-03-15T22:35:52.352109+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:33:40.214707+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.214707+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"69","reported_epoch":"993","state":"active+clean","last_fresh":"2021-03-15T22:35:48.348641+0000","last_change":"2021-03-15T22:35:45.355712+0000","last_active":"2021-03-15T22:35:48.348641+0000","last_peered":"2021-03-15T22:35:48.348641+0000","last_clean":"2021-03-15T22:35:48.348641+0000","last_became_active":"2021-03-15T22:34:55.353316+0000","last_became_peered":"2021-03-15T22:34:55.353316+0000","last_unstale":"2021-03-15T22:35:48.348641+0000","last_undegraded":"2021-03-15T22:35:48.348641+0000","last_fullsized":"2021-03-15T22:35:48.348641+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.355647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.355647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"995'3236","reported_seq":"6783","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.379080+0000","last_change":"2021-03-15T22:35:50.379294+0000","last_active":"2021-03-15T22:35:50.379080+0000","last_peered":"2021-03-15T22:35:50.379080+0000","last_clean":"2021-03-15T22:35:50.379080+0000","last_became_active":"2021-03-15T22:34:13.331560+0000","last_became_peered":"2021-03-15T22:34:13.331560+0000","last_unstale":"2021-03-15T22:35:50.379080+0000","last_undegraded":"2021-03-15T22:35:50.379080+0000","last_fullsized":"2021-03-15T22:35:50.379080+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'3208","last_scrub_stamp":"2021-03-15T22:35:42.271839+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.271839+0000","log_size":3236,"ondisk_log_size":3236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":89239895,"num_objects":40,"num_object_clones":33,"num_object_copies":120,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":40,"num_whiteouts":0,"num_read":885,"num_read_kb":910648,"num_write":679,"num_write_kb":275856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":264088494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"400","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.857677+0000","last_change":"2021-03-15T22:35:52.857478+0000","last_active":"2021-03-15T22:35:52.857677+0000","last_peered":"2021-03-15T22:35:52.857677+0000","last_clean":"2021-03-15T22:35:52.857677+0000","last_became_active":"2021-03-15T22:35:52.857192+0000","last_became_peered":"2021-03-15T22:35:52.857192+0000","last_unstale":"2021-03-15T22:35:52.857677+0000","last_undegraded":"2021-03-15T22:35:52.857677+0000","last_fullsized":"2021-03-15T22:35:52.857677+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":997,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.761137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"1012","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.361351+0000","last_change":"2021-03-15T22:35:19.456953+0000","last_active":"2021-03-15T22:35:52.361351+0000","last_peered":"2021-03-15T22:35:52.361351+0000","last_clean":"2021-03-15T22:35:52.361351+0000","last_became_active":"2021-03-15T22:33:04.337709+0000","last_became_peered":"2021-03-15T22:33:04.337709+0000","last_unstale":"2021-03-15T22:35:52.361351+0000","last_undegraded":"2021-03-15T22:35:52.361351+0000","last_fullsized":"2021-03-15T22:35:52.361351+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"78","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.349004+0000","last_change":"2021-03-15T22:34:43.341562+0000","last_active":"2021-03-15T22:35:50.349004+0000","last_peered":"2021-03-15T22:35:50.349004+0000","last_clean":"2021-03-15T22:35:50.349004+0000","last_became_active":"2021-03-15T22:34:43.340955+0000","last_became_peered":"2021-03-15T22:34:43.340955+0000","last_unstale":"2021-03-15T22:35:50.349004+0000","last_undegraded":"2021-03-15T22:35:50.349004+0000","last_fullsized":"2021-03-15T22:35:50.349004+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.843478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.843478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"194","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357075+0000","last_change":"2021-03-15T22:34:09.215097+0000","last_active":"2021-03-15T22:35:49.357075+0000","last_peered":"2021-03-15T22:35:49.357075+0000","last_clean":"2021-03-15T22:35:49.357075+0000","last_became_active":"2021-03-15T22:33:04.330531+0000","last_became_peered":"2021-03-15T22:33:04.330531+0000","last_unstale":"2021-03-15T22:35:49.357075+0000","last_undegraded":"2021-03-15T22:35:49.357075+0000","last_fullsized":"2021-03-15T22:35:49.357075+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:34:09.215034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:09.215034+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"1033","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362731+0000","last_change":"2021-03-15T22:35:22.409047+0000","last_active":"2021-03-15T22:35:52.362731+0000","last_peered":"2021-03-15T22:35:52.362731+0000","last_clean":"2021-03-15T22:35:52.362731+0000","last_became_active":"2021-03-15T22:34:55.352186+0000","last_became_peered":"2021-03-15T22:34:55.352186+0000","last_unstale":"2021-03-15T22:35:52.362731+0000","last_undegraded":"2021-03-15T22:35:52.362731+0000","last_fullsized":"2021-03-15T22:35:52.362731+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.408965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.408965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"995'995","reported_seq":"4103","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362193+0000","last_change":"2021-03-15T22:35:50.367855+0000","last_active":"2021-03-15T22:35:52.362193+0000","last_peered":"2021-03-15T22:35:52.362193+0000","last_clean":"2021-03-15T22:35:52.362193+0000","last_became_active":"2021-03-15T22:35:03.346751+0000","last_became_peered":"2021-03-15T22:35:03.346751+0000","last_unstale":"2021-03-15T22:35:52.362193+0000","last_undegraded":"2021-03-15T22:35:52.362193+0000","last_fullsized":"2021-03-15T22:35:52.362193+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":0,"last_scrub":"891'819","last_scrub_stamp":"2021-03-15T22:34:08.508824+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.508824+0000","log_size":995,"ondisk_log_size":995,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":33658111,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":266,"num_read_kb":273196,"num_write":221,"num_write_kb":98136,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":109648904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"799","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.351838+0000","last_change":"2021-03-15T22:34:43.344711+0000","last_active":"2021-03-15T22:35:52.351838+0000","last_peered":"2021-03-15T22:35:52.351838+0000","last_clean":"2021-03-15T22:35:52.351838+0000","last_became_active":"2021-03-15T22:34:43.344418+0000","last_became_peered":"2021-03-15T22:34:43.344418+0000","last_unstale":"2021-03-15T22:35:52.351838+0000","last_undegraded":"2021-03-15T22:35:52.351838+0000","last_fullsized":"2021-03-15T22:35:52.351838+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"491","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357269+0000","last_change":"2021-03-15T22:34:35.410541+0000","last_active":"2021-03-15T22:35:49.357269+0000","last_peered":"2021-03-15T22:35:49.357269+0000","last_clean":"2021-03-15T22:35:49.357269+0000","last_became_active":"2021-03-15T22:34:10.321100+0000","last_became_peered":"2021-03-15T22:34:10.321100+0000","last_unstale":"2021-03-15T22:35:49.357269+0000","last_undegraded":"2021-03-15T22:35:49.357269+0000","last_fullsized":"2021-03-15T22:35:49.357269+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:35.410485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:35.410485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"2071","reported_epoch":"998","state":"active+clean","last_fresh":"2021-03-15T22:35:53.355351+0000","last_change":"2021-03-15T22:35:10.452906+0000","last_active":"2021-03-15T22:35:53.355351+0000","last_peered":"2021-03-15T22:35:53.355351+0000","last_clean":"2021-03-15T22:35:53.355351+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:35:53.355351+0000","last_undegraded":"2021-03-15T22:35:53.355351+0000","last_fullsized":"2021-03-15T22:35:53.355351+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:35:10.452809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.452809+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1064,"num_read_kb":1059,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"589","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357710+0000","last_change":"2021-03-15T22:34:55.414460+0000","last_active":"2021-03-15T22:35:49.357710+0000","last_peered":"2021-03-15T22:35:49.357710+0000","last_clean":"2021-03-15T22:35:49.357710+0000","last_became_active":"2021-03-15T22:34:55.359023+0000","last_became_peered":"2021-03-15T22:34:55.359023+0000","last_unstale":"2021-03-15T22:35:49.357710+0000","last_undegraded":"2021-03-15T22:35:49.357710+0000","last_fullsized":"2021-03-15T22:35:49.357710+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:34:40.472800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.472800+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"387","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.352154+0000","last_change":"2021-03-15T22:34:40.799217+0000","last_active":"2021-03-15T22:35:52.352154+0000","last_peered":"2021-03-15T22:35:52.352154+0000","last_clean":"2021-03-15T22:35:52.352154+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:35:52.352154+0000","last_undegraded":"2021-03-15T22:35:52.352154+0000","last_fullsized":"2021-03-15T22:35:52.352154+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.799153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.799153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"998'3370","reported_seq":"7458","reported_epoch":"998","state":"active+clean","last_fresh":"2021-03-15T22:35:53.386653+0000","last_change":"2021-03-15T22:35:50.383126+0000","last_active":"2021-03-15T22:35:53.386653+0000","last_peered":"2021-03-15T22:35:53.386653+0000","last_clean":"2021-03-15T22:35:53.386653+0000","last_became_active":"2021-03-15T22:35:17.487261+0000","last_became_peered":"2021-03-15T22:35:17.487261+0000","last_unstale":"2021-03-15T22:35:53.386653+0000","last_undegraded":"2021-03-15T22:35:53.386653+0000","last_fullsized":"2021-03-15T22:35:53.386653+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'2597","last_scrub_stamp":"2021-03-15T22:32:30.909661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.909661+0000","log_size":3370,"ondisk_log_size":3370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":114348621,"num_objects":47,"num_object_clones":40,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":1,"num_read":1222,"num_read_kb":1301937,"num_write":787,"num_write_kb":329373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":721,"num_bytes_recovered":351021959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"1033","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.533931+0000","last_change":"2021-03-15T22:35:49.070877+0000","last_active":"2021-03-15T22:35:51.533931+0000","last_peered":"2021-03-15T22:35:51.533931+0000","last_clean":"2021-03-15T22:35:51.533931+0000","last_became_active":"2021-03-15T22:34:55.345948+0000","last_became_peered":"2021-03-15T22:34:55.345948+0000","last_unstale":"2021-03-15T22:35:51.533931+0000","last_undegraded":"2021-03-15T22:35:51.533931+0000","last_fullsized":"2021-03-15T22:35:51.533931+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.070798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.070798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"1046","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534088+0000","last_change":"2021-03-15T22:34:55.346381+0000","last_active":"2021-03-15T22:35:51.534088+0000","last_peered":"2021-03-15T22:35:51.534088+0000","last_clean":"2021-03-15T22:35:51.534088+0000","last_became_active":"2021-03-15T22:34:55.346048+0000","last_became_peered":"2021-03-15T22:34:55.346048+0000","last_unstale":"2021-03-15T22:35:51.534088+0000","last_undegraded":"2021-03-15T22:35:51.534088+0000","last_fullsized":"2021-03-15T22:35:51.534088+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:48.037748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_clean_scrub_stamp":"2021-03-15T22:34:48.037748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"539","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357358+0000","last_change":"2021-03-15T22:35:21.314851+0000","last_active":"2021-03-15T22:35:49.357358+0000","last_peered":"2021-03-15T22:35:49.357358+0000","last_clean":"2021-03-15T22:35:49.357358+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:35:49.357358+0000","last_undegraded":"2021-03-15T22:35:49.357358+0000","last_fullsized":"2021-03-15T22:35:49.357358+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:35:21.314763+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.314763+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"179","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.356492+0000","last_change":"2021-03-15T22:35:39.707993+0000","last_active":"2021-03-15T22:35:49.356492+0000","last_peered":"2021-03-15T22:35:49.356492+0000","last_clean":"2021-03-15T22:35:49.356492+0000","last_became_active":"2021-03-15T22:33:04.337244+0000","last_became_peered":"2021-03-15T22:33:04.337244+0000","last_unstale":"2021-03-15T22:35:49.356492+0000","last_undegraded":"2021-03-15T22:35:49.356492+0000","last_fullsized":"2021-03-15T22:35:49.356492+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:39.707900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:35:39.707900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"995'580","reported_seq":"3692","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.350540+0000","last_change":"2021-03-15T22:35:50.350540+0000","last_active":"2021-03-15T22:35:50.350540+0000","last_peered":"2021-03-15T22:35:50.350540+0000","last_clean":"2021-03-15T22:35:50.350540+0000","last_became_active":"2021-03-15T22:34:15.498612+0000","last_became_peered":"2021-03-15T22:34:15.498612+0000","last_unstale":"2021-03-15T22:35:50.350540+0000","last_undegraded":"2021-03-15T22:35:50.350540+0000","last_fullsized":"2021-03-15T22:35:50.350540+0000","mapping_epoch":899,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":900,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'463","last_scrub_stamp":"2021-03-15T22:33:40.563144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.563144+0000","log_size":580,"ondisk_log_size":580,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":21769360,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":174,"num_read_kb":209120,"num_write":110,"num_write_kb":54614,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":84391207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"515","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.356625+0000","last_change":"2021-03-15T22:34:55.356457+0000","last_active":"2021-03-15T22:35:49.356625+0000","last_peered":"2021-03-15T22:35:49.356625+0000","last_clean":"2021-03-15T22:35:49.356625+0000","last_became_active":"2021-03-15T22:34:55.356086+0000","last_became_peered":"2021-03-15T22:34:55.356086+0000","last_unstale":"2021-03-15T22:35:49.356625+0000","last_undegraded":"2021-03-15T22:35:49.356625+0000","last_fullsized":"2021-03-15T22:35:49.356625+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"970","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.352148+0000","last_change":"2021-03-15T22:34:55.356446+0000","last_active":"2021-03-15T22:35:52.352148+0000","last_peered":"2021-03-15T22:35:52.352148+0000","last_clean":"2021-03-15T22:35:52.352148+0000","last_became_active":"2021-03-15T22:34:55.355757+0000","last_became_peered":"2021-03-15T22:34:55.355757+0000","last_unstale":"2021-03-15T22:35:52.352148+0000","last_undegraded":"2021-03-15T22:35:52.352148+0000","last_fullsized":"2021-03-15T22:35:52.352148+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.883949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.883949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"951","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.351688+0000","last_change":"2021-03-15T22:34:11.330201+0000","last_active":"2021-03-15T22:35:52.351688+0000","last_peered":"2021-03-15T22:35:52.351688+0000","last_clean":"2021-03-15T22:35:52.351688+0000","last_became_active":"2021-03-15T22:34:10.321886+0000","last_became_peered":"2021-03-15T22:34:10.321886+0000","last_unstale":"2021-03-15T22:35:52.351688+0000","last_undegraded":"2021-03-15T22:35:52.351688+0000","last_fullsized":"2021-03-15T22:35:52.351688+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:32:31.747582+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.747582+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"836'11","reported_seq":"615","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357221+0000","last_change":"2021-03-15T22:35:44.256542+0000","last_active":"2021-03-15T22:35:49.357221+0000","last_peered":"2021-03-15T22:35:49.357221+0000","last_clean":"2021-03-15T22:35:49.357221+0000","last_became_active":"2021-03-15T22:33:04.336294+0000","last_became_peered":"2021-03-15T22:33:04.336294+0000","last_unstale":"2021-03-15T22:35:49.357221+0000","last_undegraded":"2021-03-15T22:35:49.357221+0000","last_fullsized":"2021-03-15T22:35:49.357221+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"836'11","last_scrub_stamp":"2021-03-15T22:35:44.256486+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:35:44.256486+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"995'944","reported_seq":"4088","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.377905+0000","last_change":"2021-03-15T22:35:50.377905+0000","last_active":"2021-03-15T22:35:50.377905+0000","last_peered":"2021-03-15T22:35:50.377905+0000","last_clean":"2021-03-15T22:35:50.377905+0000","last_became_active":"2021-03-15T22:34:58.350873+0000","last_became_peered":"2021-03-15T22:34:58.350873+0000","last_unstale":"2021-03-15T22:35:50.377905+0000","last_undegraded":"2021-03-15T22:35:50.377905+0000","last_fullsized":"2021-03-15T22:35:50.377905+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'901","last_scrub_stamp":"2021-03-15T22:35:16.957336+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.957336+0000","log_size":944,"ondisk_log_size":944,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":32736159,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":256,"num_read_kb":220514,"num_write":230,"num_write_kb":88162,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":114552572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"1020","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.362060+0000","last_change":"2021-03-15T22:34:55.351577+0000","last_active":"2021-03-15T22:35:52.362060+0000","last_peered":"2021-03-15T22:35:52.362060+0000","last_clean":"2021-03-15T22:35:52.362060+0000","last_became_active":"2021-03-15T22:34:55.351249+0000","last_became_peered":"2021-03-15T22:34:55.351249+0000","last_unstale":"2021-03-15T22:35:52.362060+0000","last_undegraded":"2021-03-15T22:35:52.362060+0000","last_fullsized":"2021-03-15T22:35:52.362060+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.507204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.507204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"459","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.357737+0000","last_change":"2021-03-15T22:33:22.598240+0000","last_active":"2021-03-15T22:35:49.357737+0000","last_peered":"2021-03-15T22:35:49.357737+0000","last_clean":"2021-03-15T22:35:49.357737+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:35:49.357737+0000","last_undegraded":"2021-03-15T22:35:49.357737+0000","last_fullsized":"2021-03-15T22:35:49.357737+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.598174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.598174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"1188","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.345579+0000","last_change":"2021-03-15T22:33:43.514160+0000","last_active":"2021-03-15T22:35:50.345579+0000","last_peered":"2021-03-15T22:35:50.345579+0000","last_clean":"2021-03-15T22:35:50.345579+0000","last_became_active":"2021-03-15T22:33:04.336145+0000","last_became_peered":"2021-03-15T22:33:04.336145+0000","last_unstale":"2021-03-15T22:35:50.345579+0000","last_undegraded":"2021-03-15T22:35:50.345579+0000","last_fullsized":"2021-03-15T22:35:50.345579+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:33:43.514088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.514088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"510","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.346768+0000","last_change":"2021-03-15T22:33:20.685041+0000","last_active":"2021-03-15T22:35:50.346768+0000","last_peered":"2021-03-15T22:35:50.346768+0000","last_clean":"2021-03-15T22:35:50.346768+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:35:50.346768+0000","last_undegraded":"2021-03-15T22:35:50.346768+0000","last_fullsized":"2021-03-15T22:35:50.346768+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:33:20.684976+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.684976+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"995'1673","reported_seq":"4647","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534334+0000","last_change":"2021-03-15T22:35:50.368826+0000","last_active":"2021-03-15T22:35:51.534334+0000","last_peered":"2021-03-15T22:35:51.534334+0000","last_clean":"2021-03-15T22:35:51.534334+0000","last_became_active":"2021-03-15T22:34:58.337099+0000","last_became_peered":"2021-03-15T22:34:58.337099+0000","last_unstale":"2021-03-15T22:35:51.534334+0000","last_undegraded":"2021-03-15T22:35:51.534334+0000","last_fullsized":"2021-03-15T22:35:51.534334+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'1611","last_scrub_stamp":"2021-03-15T22:35:21.054192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.054192+0000","log_size":1673,"ondisk_log_size":1673,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40111289,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":2,"num_read":540,"num_read_kb":566333,"num_write":346,"num_write_kb":120323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":226741522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"1073","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344606+0000","last_change":"2021-03-15T22:34:55.343142+0000","last_active":"2021-03-15T22:35:50.344606+0000","last_peered":"2021-03-15T22:35:50.344606+0000","last_clean":"2021-03-15T22:35:50.344606+0000","last_became_active":"2021-03-15T22:34:55.341896+0000","last_became_peered":"2021-03-15T22:34:55.341896+0000","last_unstale":"2021-03-15T22:35:50.344606+0000","last_undegraded":"2021-03-15T22:35:50.344606+0000","last_fullsized":"2021-03-15T22:35:50.344606+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"779","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347810+0000","last_change":"2021-03-15T22:34:41.870475+0000","last_active":"2021-03-15T22:35:50.347810+0000","last_peered":"2021-03-15T22:35:50.347810+0000","last_clean":"2021-03-15T22:35:50.347810+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:35:50.347810+0000","last_undegraded":"2021-03-15T22:35:50.347810+0000","last_fullsized":"2021-03-15T22:35:50.347810+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.870368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.870368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"432","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.361453+0000","last_change":"2021-03-15T22:35:08.402702+0000","last_active":"2021-03-15T22:35:52.361453+0000","last_peered":"2021-03-15T22:35:52.361453+0000","last_clean":"2021-03-15T22:35:52.361453+0000","last_became_active":"2021-03-15T22:34:10.349731+0000","last_became_peered":"2021-03-15T22:34:10.349731+0000","last_unstale":"2021-03-15T22:35:52.361453+0000","last_undegraded":"2021-03-15T22:35:52.361453+0000","last_fullsized":"2021-03-15T22:35:52.361453+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:35:08.402641+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:35:08.402641+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1041","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.345875+0000","last_change":"2021-03-15T22:34:43.343527+0000","last_active":"2021-03-15T22:35:50.345875+0000","last_peered":"2021-03-15T22:35:50.345875+0000","last_clean":"2021-03-15T22:35:50.345875+0000","last_became_active":"2021-03-15T22:34:43.343255+0000","last_became_peered":"2021-03-15T22:34:43.343255+0000","last_unstale":"2021-03-15T22:35:50.345875+0000","last_undegraded":"2021-03-15T22:35:50.345875+0000","last_fullsized":"2021-03-15T22:35:50.345875+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.673338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.673338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"995'1423","reported_seq":"4649","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.367224+0000","last_change":"2021-03-15T22:35:50.367551+0000","last_active":"2021-03-15T22:35:50.367224+0000","last_peered":"2021-03-15T22:35:50.367224+0000","last_clean":"2021-03-15T22:35:50.367224+0000","last_became_active":"2021-03-15T22:34:58.344406+0000","last_became_peered":"2021-03-15T22:34:58.344406+0000","last_unstale":"2021-03-15T22:35:50.367224+0000","last_undegraded":"2021-03-15T22:35:50.367224+0000","last_fullsized":"2021-03-15T22:35:50.367224+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'1407","last_scrub_stamp":"2021-03-15T22:35:43.758914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:43.758914+0000","log_size":1423,"ondisk_log_size":1423,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42291871,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":383,"num_read_kb":340483,"num_write":310,"num_write_kb":115219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":127958264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"942","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.352097+0000","last_change":"2021-03-15T22:34:55.356480+0000","last_active":"2021-03-15T22:35:52.352097+0000","last_peered":"2021-03-15T22:35:52.352097+0000","last_clean":"2021-03-15T22:35:52.352097+0000","last_became_active":"2021-03-15T22:34:55.356213+0000","last_became_peered":"2021-03-15T22:34:55.356213+0000","last_unstale":"2021-03-15T22:35:52.352097+0000","last_undegraded":"2021-03-15T22:35:52.352097+0000","last_fullsized":"2021-03-15T22:35:52.352097+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.729838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.729838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1295","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.345776+0000","last_change":"2021-03-15T22:35:19.468482+0000","last_active":"2021-03-15T22:35:50.345776+0000","last_peered":"2021-03-15T22:35:50.345776+0000","last_clean":"2021-03-15T22:35:50.345776+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:35:50.345776+0000","last_undegraded":"2021-03-15T22:35:50.345776+0000","last_fullsized":"2021-03-15T22:35:50.345776+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:35:19.468402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.468402+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"408","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534704+0000","last_change":"2021-03-15T22:35:45.165921+0000","last_active":"2021-03-15T22:35:51.534704+0000","last_peered":"2021-03-15T22:35:51.534704+0000","last_clean":"2021-03-15T22:35:51.534704+0000","last_became_active":"2021-03-15T22:34:10.332089+0000","last_became_peered":"2021-03-15T22:34:10.332089+0000","last_unstale":"2021-03-15T22:35:51.534704+0000","last_undegraded":"2021-03-15T22:35:51.534704+0000","last_fullsized":"2021-03-15T22:35:51.534704+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.165838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.165838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"158","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534715+0000","last_change":"2021-03-15T22:34:44.050423+0000","last_active":"2021-03-15T22:35:51.534715+0000","last_peered":"2021-03-15T22:35:51.534715+0000","last_clean":"2021-03-15T22:35:51.534715+0000","last_became_active":"2021-03-15T22:33:45.317888+0000","last_became_peered":"2021-03-15T22:33:45.317888+0000","last_unstale":"2021-03-15T22:35:51.534715+0000","last_undegraded":"2021-03-15T22:35:51.534715+0000","last_fullsized":"2021-03-15T22:35:51.534715+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:34:44.050366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:44.050366+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"995'486","reported_seq":"3311","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.358350+0000","last_change":"2021-03-15T22:35:50.358626+0000","last_active":"2021-03-15T22:35:50.358350+0000","last_peered":"2021-03-15T22:35:50.358350+0000","last_clean":"2021-03-15T22:35:50.358350+0000","last_became_active":"2021-03-15T22:33:08.354872+0000","last_became_peered":"2021-03-15T22:33:08.354872+0000","last_unstale":"2021-03-15T22:35:50.358350+0000","last_undegraded":"2021-03-15T22:35:50.358350+0000","last_fullsized":"2021-03-15T22:35:50.358350+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'470","last_scrub_stamp":"2021-03-15T22:35:42.721100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.721100+0000","log_size":486,"ondisk_log_size":486,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":15982145,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":188,"num_read_kb":197819,"num_write":103,"num_write_kb":39131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":31290769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"78","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347816+0000","last_change":"2021-03-15T22:34:43.340972+0000","last_active":"2021-03-15T22:35:50.347816+0000","last_peered":"2021-03-15T22:35:50.347816+0000","last_clean":"2021-03-15T22:35:50.347816+0000","last_became_active":"2021-03-15T22:34:43.340600+0000","last_became_peered":"2021-03-15T22:34:43.340600+0000","last_unstale":"2021-03-15T22:35:50.347816+0000","last_undegraded":"2021-03-15T22:35:50.347816+0000","last_fullsized":"2021-03-15T22:35:50.347816+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:09.612126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.612126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"1040","reported_epoch":"997","state":"active+clean","last_fresh":"2021-03-15T22:35:52.352264+0000","last_change":"2021-03-15T22:35:21.917891+0000","last_active":"2021-03-15T22:35:52.352264+0000","last_peered":"2021-03-15T22:35:52.352264+0000","last_clean":"2021-03-15T22:35:52.352264+0000","last_became_active":"2021-03-15T22:33:08.352367+0000","last_became_peered":"2021-03-15T22:33:08.352367+0000","last_unstale":"2021-03-15T22:35:52.352264+0000","last_undegraded":"2021-03-15T22:35:52.352264+0000","last_fullsized":"2021-03-15T22:35:52.352264+0000","mapping_epoch":837,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'9","last_scrub_stamp":"2021-03-15T22:35:21.917819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.917819+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1005","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.344992+0000","last_change":"2021-03-15T22:34:55.347300+0000","last_active":"2021-03-15T22:35:50.344992+0000","last_peered":"2021-03-15T22:35:50.344992+0000","last_clean":"2021-03-15T22:35:50.344992+0000","last_became_active":"2021-03-15T22:34:55.346934+0000","last_became_peered":"2021-03-15T22:34:55.346934+0000","last_unstale":"2021-03-15T22:35:50.344992+0000","last_undegraded":"2021-03-15T22:35:50.344992+0000","last_fullsized":"2021-03-15T22:35:50.344992+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.461102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"1167","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.345640+0000","last_change":"2021-03-15T22:35:21.384129+0000","last_active":"2021-03-15T22:35:50.345640+0000","last_peered":"2021-03-15T22:35:50.345640+0000","last_clean":"2021-03-15T22:35:50.345640+0000","last_became_active":"2021-03-15T22:34:10.330608+0000","last_became_peered":"2021-03-15T22:34:10.330608+0000","last_unstale":"2021-03-15T22:35:50.345640+0000","last_undegraded":"2021-03-15T22:35:50.345640+0000","last_fullsized":"2021-03-15T22:35:50.345640+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:35:21.384037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.384037+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"992'493","reported_seq":"2812","reported_epoch":"993","state":"active+clean","last_fresh":"2021-03-15T22:35:48.348953+0000","last_change":"2021-03-15T22:35:48.348953+0000","last_active":"2021-03-15T22:35:48.348953+0000","last_peered":"2021-03-15T22:35:48.348953+0000","last_clean":"2021-03-15T22:35:48.348953+0000","last_became_active":"2021-03-15T22:34:58.350923+0000","last_became_peered":"2021-03-15T22:34:58.350923+0000","last_unstale":"2021-03-15T22:35:48.348953+0000","last_undegraded":"2021-03-15T22:35:48.348953+0000","last_fullsized":"2021-03-15T22:35:48.348953+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"957'470","last_scrub_stamp":"2021-03-15T22:35:18.338289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.338289+0000","log_size":493,"ondisk_log_size":493,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9671780,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":156,"num_read_kb":164363,"num_write":115,"num_write_kb":35721,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":52707265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"2e2","length":"1"},{"start":"2f3","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"740","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347501+0000","last_change":"2021-03-15T22:34:55.345645+0000","last_active":"2021-03-15T22:35:50.347501+0000","last_peered":"2021-03-15T22:35:50.347501+0000","last_clean":"2021-03-15T22:35:50.347501+0000","last_became_active":"2021-03-15T22:34:55.345267+0000","last_became_peered":"2021-03-15T22:34:55.345267+0000","last_unstale":"2021-03-15T22:35:50.347501+0000","last_undegraded":"2021-03-15T22:35:50.347501+0000","last_fullsized":"2021-03-15T22:35:50.347501+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.958050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.958050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"1033","reported_epoch":"996","state":"active+clean","last_fresh":"2021-03-15T22:35:51.534683+0000","last_change":"2021-03-15T22:35:12.091714+0000","last_active":"2021-03-15T22:35:51.534683+0000","last_peered":"2021-03-15T22:35:51.534683+0000","last_clean":"2021-03-15T22:35:51.534683+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:35:51.534683+0000","last_undegraded":"2021-03-15T22:35:51.534683+0000","last_fullsized":"2021-03-15T22:35:51.534683+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.091650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.091650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"188","reported_epoch":"994","state":"active+clean","last_fresh":"2021-03-15T22:35:49.356595+0000","last_change":"2021-03-15T22:35:20.678309+0000","last_active":"2021-03-15T22:35:49.356595+0000","last_peered":"2021-03-15T22:35:49.356595+0000","last_clean":"2021-03-15T22:35:49.356595+0000","last_became_active":"2021-03-15T22:34:10.332236+0000","last_became_peered":"2021-03-15T22:34:10.332236+0000","last_unstale":"2021-03-15T22:35:49.356595+0000","last_undegraded":"2021-03-15T22:35:49.356595+0000","last_fullsized":"2021-03-15T22:35:49.356595+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.678246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"1141","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.345266+0000","last_change":"2021-03-15T22:34:55.371223+0000","last_active":"2021-03-15T22:35:50.345266+0000","last_peered":"2021-03-15T22:35:50.345266+0000","last_clean":"2021-03-15T22:35:50.345266+0000","last_became_active":"2021-03-15T22:34:55.341958+0000","last_became_peered":"2021-03-15T22:34:55.341958+0000","last_unstale":"2021-03-15T22:35:50.345266+0000","last_undegraded":"2021-03-15T22:35:50.345266+0000","last_fullsized":"2021-03-15T22:35:50.345266+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:32:43.307660+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.307660+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"995'1658","reported_seq":"4735","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.382979+0000","last_change":"2021-03-15T22:35:50.383225+0000","last_active":"2021-03-15T22:35:50.382979+0000","last_peered":"2021-03-15T22:35:50.382979+0000","last_clean":"2021-03-15T22:35:50.382979+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:35:50.382979+0000","last_undegraded":"2021-03-15T22:35:50.382979+0000","last_fullsized":"2021-03-15T22:35:50.382979+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"837'1315","last_scrub_stamp":"2021-03-15T22:33:08.608669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.608669+0000","log_size":1658,"ondisk_log_size":1658,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51444516,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":3,"num_read":549,"num_read_kb":589645,"num_write":373,"num_write_kb":134308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2b5","length":"1"},{"start":"32b","length":"1"},{"start":"33e","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"1006","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.345842+0000","last_change":"2021-03-15T22:34:55.340853+0000","last_active":"2021-03-15T22:35:50.345842+0000","last_peered":"2021-03-15T22:35:50.345842+0000","last_clean":"2021-03-15T22:35:50.345842+0000","last_became_active":"2021-03-15T22:34:55.340498+0000","last_became_peered":"2021-03-15T22:34:55.340498+0000","last_unstale":"2021-03-15T22:35:50.345842+0000","last_undegraded":"2021-03-15T22:35:50.345842+0000","last_fullsized":"2021-03-15T22:35:50.345842+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:20.500979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.500979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"976'19","reported_seq":"392","reported_epoch":"993","state":"active+clean","last_fresh":"2021-03-15T22:35:48.348333+0000","last_change":"2021-03-15T22:35:28.365207+0000","last_active":"2021-03-15T22:35:48.348333+0000","last_peered":"2021-03-15T22:35:48.348333+0000","last_clean":"2021-03-15T22:35:48.348333+0000","last_became_active":"2021-03-15T22:35:28.364764+0000","last_became_peered":"2021-03-15T22:35:28.364764+0000","last_unstale":"2021-03-15T22:35:48.348333+0000","last_undegraded":"2021-03-15T22:35:48.348333+0000","last_fullsized":"2021-03-15T22:35:48.348333+0000","mapping_epoch":972,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":973,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'13","last_scrub_stamp":"2021-03-15T22:33:37.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.949876+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"424","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.347290+0000","last_change":"2021-03-15T22:33:21.690369+0000","last_active":"2021-03-15T22:35:50.347290+0000","last_peered":"2021-03-15T22:35:50.347290+0000","last_clean":"2021-03-15T22:35:50.347290+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:35:50.347290+0000","last_undegraded":"2021-03-15T22:35:50.347290+0000","last_fullsized":"2021-03-15T22:35:50.347290+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.690309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"1185","reported_epoch":"995","state":"active+clean","last_fresh":"2021-03-15T22:35:50.345221+0000","last_change":"2021-03-15T22:35:46.406478+0000","last_active":"2021-03-15T22:35:50.345221+0000","last_peered":"2021-03-15T22:35:50.345221+0000","last_clean":"2021-03-15T22:35:50.345221+0000","last_became_active":"2021-03-15T22:33:04.337113+0000","last_became_peered":"2021-03-15T22:33:04.337113+0000","last_unstale":"2021-03-15T22:35:50.345221+0000","last_undegraded":"2021-03-15T22:35:50.345221+0000","last_fullsized":"2021-03-15T22:35:50.345221+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:35:46.406376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:46.406376+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":743697791,"num_objects":331,"num_object_clones":281,"num_object_copies":993,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":331,"num_whiteouts":11,"num_read":7090,"num_read_kb":7273231,"num_write":5209,"num_write_kb":2006682,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4621,"num_bytes_recovered":2230024638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1677697024,"data_stored":2331733794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22943,"ondisk_log_size":22943,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1064,"num_read_kb":1059,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":4842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1482,"num_read_kb":987,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":94766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":947,"ondisk_log_size":947,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":39,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":117,"acting":117,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299612,"num_pgs":70,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":284688,"kb_used_data":227336,"kb_used_omap":0,"kb_used_meta":57344,"kb_avail":93460464,"statfs":{"total":95995035648,"available":95703515136,"internally_reserved":0,"allocated":232792064,"data_stored":342460945,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58720256},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45000000000000001,"15min":0.46500000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57399999999999995,"5min":0.70099999999999996,"15min":2.73},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.222},"max":{"1min":0.61199999999999999,"5min":0.75800000000000001,"15min":3.8799999999999999},"last":0.45200000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.443,"15min":0.47199999999999998},"min":{"1min":0.254,"5min":0.249,"15min":0.249},"max":{"1min":0.57299999999999995,"5min":0.73899999999999999,"15min":2.6859999999999999},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.56000000000000005,"15min":0.51400000000000001},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.23400000000000001},"max":{"1min":5.3819999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":0.41699999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:35:38 2021","interfaces":[{"interface":"back","average":{"1min":2.1760000000000002,"5min":0.83999999999999997,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":29.350999999999999,"5min":29.350999999999999,"15min":29.350999999999999},"last":0.51100000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.498,"15min":0.5},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.44500000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.498,"15min":0.495},"min":{"1min":0.314,"5min":0.314,"15min":0.28899999999999998},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.702},"last":0.52400000000000002},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.497,"15min":0.52500000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":5.415},"last":0.46999999999999997}]},{"osd":4,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.437,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.27900000000000003,"15min":0.23799999999999999},"max":{"1min":0.58499999999999996,"5min":0.753,"15min":1.927},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.46300000000000002,"15min":0.46200000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":0.57599999999999996,"5min":0.72699999999999998,"15min":2.5339999999999998},"last":0.504}]},{"osd":5,"last update":"Mon Mar 15 22:35:38 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47899999999999998,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68300000000000005,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.498,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67800000000000005,"5min":0.77700000000000002,"15min":1.071},"last":0.47799999999999998}]},{"osd":6,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.29999999999999999,"15min":0.27100000000000002},"max":{"1min":0.65300000000000002,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.28399999999999997},"max":{"1min":0.73799999999999999,"5min":0.81999999999999995,"15min":1.8560000000000001},"last":0.55000000000000004}]}]},{"osd":6,"up_from":45,"seq":193273528543,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":219260,"kb_used_data":173108,"kb_used_omap":0,"kb_used_meta":46144,"kb_avail":93525892,"statfs":{"total":95995035648,"available":95770513408,"internally_reserved":0,"allocated":177262592,"data_stored":239639081,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47251456},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.49199999999999999,"15min":0.48399999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.79600000000000004,"5min":2.5470000000000002,"15min":3.085},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.47999999999999998,"15min":0.46500000000000002},"min":{"1min":0.27600000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73099999999999998,"5min":1.137,"15min":1.323},"last":0.53400000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.254},"max":{"1min":1.1120000000000001,"5min":2.7839999999999998,"15min":14.637},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.45200000000000001,"15min":0.47499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60299999999999998,"5min":0.84899999999999998,"15min":5.2450000000000001},"last":0.39100000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:35:43 2021","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64800000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70899999999999996,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.51000000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51100000000000001,"15min":0.50900000000000001},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.29899999999999999},"max":{"1min":0.874,"5min":1.0609999999999999,"15min":2.6150000000000002},"last":0.443},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.502,"15min":0.502},"min":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.63900000000000001,"5min":2.4670000000000001,"15min":2.7890000000000001},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.44600000000000001,"15min":0.438},"min":{"1min":0.25,"5min":0.25,"15min":0.224},"max":{"1min":0.61799999999999999,"5min":0.94099999999999995,"15min":4.306},"last":0.45500000000000002},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.47799999999999998,"15min":0.49199999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23499999999999999},"max":{"1min":0.82799999999999996,"5min":1.091,"15min":4.181},"last":0.40300000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:35:39 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.47599999999999998,"15min":0.42599999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.26700000000000002},"max":{"1min":0.63300000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.53000000000000003,"15min":0.46999999999999997},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67900000000000005,"5min":1.022,"15min":1.022},"last":0.48499999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:35:21 2021","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.47799999999999998,"15min":0.53200000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.249},"max":{"1min":0.67700000000000005,"5min":0.76300000000000001,"15min":15.340999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.56399999999999995,"15min":0.58699999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.27200000000000002},"max":{"1min":0.754,"5min":1.0109999999999999,"15min":15.356},"last":0.55000000000000004}]}]},{"osd":1,"up_from":13,"seq":55834575089,"num_pgs":52,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":186540,"kb_used_data":118692,"kb_used_omap":0,"kb_used_meta":67840,"kb_avail":93558612,"statfs":{"total":95995035648,"available":95804018688,"internally_reserved":0,"allocated":121540608,"data_stored":166059924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69468160},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:34:52 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.379,"15min":0.51200000000000001},"min":{"1min":0.217,"5min":0.184,"15min":0.17599999999999999},"max":{"1min":4.3920000000000003,"5min":4.3920000000000003,"15min":54.427999999999997},"last":0.313},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.42699999999999999,"15min":0.54100000000000004},"min":{"1min":0.219,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":4.6029999999999998,"5min":4.6029999999999998,"15min":54.356999999999999},"last":0.32600000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:35:45 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.46899999999999997,"15min":0.379},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59199999999999997,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45500000000000002,"15min":0.40100000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64600000000000002,"5min":5.0190000000000001,"15min":5.0190000000000001},"last":0.39500000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:35:20 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.47599999999999998,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.26400000000000001,"15min":0.215},"max":{"1min":7.718,"5min":7.718,"15min":7.718},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.53700000000000003,"15min":0.434},"min":{"1min":0.245,"5min":0.245,"15min":0.20999999999999999},"max":{"1min":8.4760000000000009,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":0.35299999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:35:26 2021","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.61399999999999999,"15min":0.55200000000000005},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.22800000000000001},"max":{"1min":0.78800000000000003,"5min":4.593,"15min":4.593},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.253},"max":{"1min":4.3140000000000001,"5min":4.3140000000000001,"15min":4.6139999999999999},"last":0.5}]},{"osd":5,"last update":"Mon Mar 15 22:35:34 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.59499999999999997,"15min":0.54700000000000004},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.34300000000000003},"max":{"1min":4.3630000000000004,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60999999999999999,"15min":0.52900000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32100000000000001},"max":{"1min":1.794,"5min":3.472,"15min":3.472},"last":0.54000000000000004}]},{"osd":6,"last update":"Mon Mar 15 22:34:59 2021","interfaces":[{"interface":"back","average":{"1min":0.58699999999999997,"5min":0.61499999999999999,"15min":0.58499999999999996},"min":{"1min":0.38500000000000001,"5min":0.35499999999999998,"15min":0.314},"max":{"1min":1.679,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.63600000000000001,"15min":0.58299999999999996},"min":{"1min":0.39800000000000002,"5min":0.375,"15min":0.33700000000000002},"max":{"1min":0.82999999999999996,"5min":4.6529999999999996,"15min":4.6529999999999996},"last":0.52700000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:35:20 2021","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63700000000000001,"15min":0.60299999999999998},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.316},"max":{"1min":1.716,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.66800000000000004,"15min":0.59599999999999997},"min":{"1min":0.42799999999999999,"5min":0.40300000000000002,"15min":0.35799999999999998},"max":{"1min":1.466,"5min":4.625,"15min":4.625},"last":0.56200000000000006}]}]},{"osd":0,"up_from":8,"seq":34359738611,"num_pgs":59,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308216,"kb_used_data":245616,"kb_used_omap":0,"kb_used_meta":62592,"kb_avail":93436936,"statfs":{"total":95995035648,"available":95679422464,"internally_reserved":0,"allocated":251510784,"data_stored":372421163,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":64094208},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:35:47 2021","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35899999999999999,"15min":0.34000000000000002},"min":{"1min":0.249,"5min":0.192,"15min":0.17999999999999999},"max":{"1min":0.68000000000000005,"5min":1.46,"15min":1.7070000000000001},"last":0.22800000000000001},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39600000000000002,"15min":0.36899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19900000000000001},"max":{"1min":0.499,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.249}]},{"osd":2,"last update":"Mon Mar 15 22:35:41 2021","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.374,"15min":0.36399999999999999},"min":{"1min":0.28000000000000003,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.70099999999999996,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.375,"5min":0.49199999999999999,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51300000000000001,"5min":7.3799999999999999,"15min":7.3799999999999999},"last":0.32700000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39700000000000002,"15min":0.38400000000000001},"min":{"1min":0.25700000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":0.61799999999999999,"5min":1.544,"15min":1.544},"last":0.316},{"interface":"front","average":{"1min":0.442,"5min":0.47499999999999998,"15min":0.41999999999999998},"min":{"1min":0.31,"5min":0.254,"15min":0.19400000000000001},"max":{"1min":0.67700000000000005,"5min":3.988,"15min":3.988},"last":0.27800000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.36899999999999999,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":0.76300000000000001,"5min":1.861,"15min":1.861},"last":0.47399999999999998},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.56599999999999995},"min":{"1min":0.40699999999999997,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":0.74099999999999999,"5min":1.5700000000000001,"15min":1.645},"last":0.46500000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:35:35 2021","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.60799999999999998,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.32600000000000001,"15min":0.29899999999999999},"max":{"1min":0.71599999999999997,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.56000000000000005,"15min":0.502},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.318},"max":{"1min":0.72099999999999997,"5min":1.909,"15min":1.909},"last":0.36399999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:34:59 2021","interfaces":[{"interface":"back","average":{"1min":0.63600000000000001,"5min":0.57199999999999995,"15min":0.55900000000000005},"min":{"1min":0.34899999999999998,"5min":0.34899999999999998,"15min":0.26200000000000001},"max":{"1min":2.7109999999999999,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.56999999999999995,"5min":0.58599999999999997,"15min":0.57199999999999995},"min":{"1min":0.437,"5min":0.32000000000000001,"15min":0.27100000000000002},"max":{"1min":0.77100000000000002,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.42699999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.65500000000000003,"15min":0.58699999999999997},"min":{"1min":0.42899999999999999,"5min":0.38200000000000001,"15min":0.27900000000000003},"max":{"1min":0.753,"5min":3.8239999999999998,"15min":3.8239999999999998},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60699999999999998,"15min":0.57999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39100000000000001,"15min":0.317},"max":{"1min":0.752,"5min":2.665,"15min":2.665},"last":0.41999999999999998}]}]},{"osd":2,"up_from":681,"seq":2924872728641,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":334028,"kb_used_data":302852,"kb_used_omap":119,"kb_used_meta":31048,"kb_avail":93411124,"statfs":{"total":95995035648,"available":95652990976,"internally_reserved":0,"allocated":310120448,"data_stored":403964629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":31793513},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.316,"15min":0.31},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.56299999999999994,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35299999999999998,"15min":0.33100000000000002},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.77800000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.314}]},{"osd":1,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.35599999999999998,"15min":0.33400000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.58099999999999996,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.28899999999999998},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.379,"15min":0.36299999999999999},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.59799999999999998,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.32400000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.42099999999999999,"15min":0.36499999999999999},"min":{"1min":0.26200000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.65500000000000003,"5min":3.496,"15min":3.496},"last":0.377},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.36199999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.85099999999999998,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.38900000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54800000000000004,"15min":0.49399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67600000000000005,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.628},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53800000000000003,"15min":0.499},"min":{"1min":0.378,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76700000000000002,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.54500000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.875,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56100000000000005,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70299999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.55600000000000005}]},{"osd":6,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57599999999999996,"15min":0.51900000000000002},"min":{"1min":0.372,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76600000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.52500000000000002},"min":{"1min":0.433,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":1.387,"15min":1.387},"last":0.50800000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59399999999999997,"15min":0.54300000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.748,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.499},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57599999999999996,"15min":0.53200000000000003},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82599999999999996,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.61099999999999999}]}]},{"osd":3,"up_from":26,"seq":111669149929,"num_pgs":62,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":313552,"kb_used_data":237064,"kb_used_omap":0,"kb_used_meta":76480,"kb_avail":93431600,"statfs":{"total":95995035648,"available":95673958400,"internally_reserved":0,"allocated":242753536,"data_stored":318800824,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":78315520},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34499999999999997,"15min":0.34000000000000002},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.17699999999999999},"max":{"1min":0.47799999999999998,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.33300000000000002,"15min":0.33800000000000002},"min":{"1min":0.23699999999999999,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.504,"5min":0.76900000000000002,"15min":2.6339999999999999},"last":0.26300000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.371,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23599999999999999,"15min":0.23000000000000001},"max":{"1min":0.501,"5min":0.626,"15min":6.0199999999999996},"last":0.32800000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.36799999999999999,"15min":0.39900000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.20699999999999999},"max":{"1min":0.47899999999999998,"5min":2.5619999999999998,"15min":7.6130000000000004},"last":0.439}]},{"osd":2,"last update":"Mon Mar 15 22:35:40 2021","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.246,"5min":0.216,"15min":0.216},"max":{"1min":0.53100000000000003,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.28000000000000003},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.30499999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.35299999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:35:36 2021","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.35599999999999998,"5min":0.31900000000000001,"15min":0.22600000000000001},"max":{"1min":0.72199999999999998,"5min":0.86899999999999999,"15min":3.3370000000000002},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53900000000000003,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.311,"15min":0.26700000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":2.2890000000000001},"last":0.48799999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:35:36 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.755},"min":{"1min":0.35099999999999998,"5min":0.311,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":0.75800000000000001,"15min":9.4870000000000001},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54100000000000004,"15min":0.95899999999999996},"min":{"1min":0.37,"5min":0.36699999999999999,"15min":0.32800000000000001},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":15.776},"last":0.45800000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:35:02 2021","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.55800000000000005,"15min":0.56499999999999995},"min":{"1min":0.38200000000000001,"5min":0.315,"15min":0.30099999999999999},"max":{"1min":0.78700000000000003,"5min":1.359,"15min":2.6499999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.55400000000000005,"15min":0.56100000000000005},"min":{"1min":0.375,"5min":0.35899999999999999,"15min":0.30399999999999999},"max":{"1min":0.77800000000000002,"5min":1.3400000000000001,"15min":3.472},"last":0.41299999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.57399999999999995},"min":{"1min":0.36199999999999999,"5min":0.308,"15min":0.29599999999999999},"max":{"1min":0.77600000000000002,"5min":0.91200000000000003,"15min":5.625},"last":0.503},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.40799999999999997,"5min":0.371,"15min":0.35499999999999998},"max":{"1min":0.74299999999999999,"5min":2.621,"15min":2.621},"last":0.40600000000000003}]}]},{"osd":4,"up_from":31,"seq":133143986406,"num_pgs":66,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":145244,"kb_used_data":100308,"kb_used_omap":0,"kb_used_meta":44928,"kb_avail":93599908,"statfs":{"total":95995035648,"available":95846305792,"internally_reserved":0,"allocated":102715392,"data_stored":148306280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46006272},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.50800000000000001,"15min":0.47599999999999998},"min":{"1min":0.30099999999999999,"5min":0.221,"15min":0.20000000000000001},"max":{"1min":0.55900000000000005,"5min":4.7050000000000001,"15min":4.7050000000000001},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.435,"5min":0.46500000000000002,"15min":0.498},"min":{"1min":0.27300000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.67100000000000004,"5min":1.4179999999999999,"15min":5.0209999999999999},"last":0.46700000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.312,"5min":0.25,"15min":0.23999999999999999},"max":{"1min":0.73399999999999999,"5min":1.3720000000000001,"15min":4.3499999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.499,"15min":0.499},"min":{"1min":0.311,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":0.627,"5min":1.494,"15min":4.718},"last":0.372}]},{"osd":2,"last update":"Mon Mar 15 22:35:41 2021","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47999999999999998,"15min":0.502},"min":{"1min":0.32200000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77700000000000002,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.497,"5min":0.48599999999999999,"15min":0.495},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.41399999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.28499999999999998},"max":{"1min":1.0820000000000001,"5min":1.486,"15min":5.0499999999999998},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.54400000000000004},"min":{"1min":0.38300000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0660000000000001,"5min":1.4950000000000001,"15min":4.4050000000000002},"last":0.44700000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.41999999999999998,"15min":0.40799999999999997},"min":{"1min":0.27600000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.623,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.29599999999999999},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47499999999999998,"15min":0.436},"min":{"1min":0.29099999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66400000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.435}]},{"osd":6,"last update":"Mon Mar 15 22:34:58 2021","interfaces":[{"interface":"back","average":{"1min":0.44400000000000001,"5min":0.47799999999999998,"15min":0.46700000000000003},"min":{"1min":0.247,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":0.70999999999999996,"5min":0.878,"15min":4.3079999999999998},"last":0.317},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.54900000000000004,"15min":0.54000000000000004},"min":{"1min":0.33000000000000002,"5min":0.33000000000000002,"15min":0.28000000000000003},"max":{"1min":0.71899999999999997,"5min":0.83799999999999997,"15min":4.6600000000000001},"last":0.48499999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:35:18 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51300000000000001,"15min":0.51000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26300000000000001},"max":{"1min":0.749,"5min":0.80900000000000005,"15min":4.6929999999999996},"last":0.40400000000000003},{"interface":"front","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.25},"max":{"1min":0.73599999999999999,"5min":1.034,"15min":4.9900000000000002},"last":0.49299999999999999}]}]},{"osd":5,"up_from":616,"seq":2645699854415,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":289512,"kb_used_data":248224,"kb_used_omap":153,"kb_used_meta":41126,"kb_avail":93455640,"statfs":{"total":95995035648,"available":95698575360,"internally_reserved":0,"allocated":254181376,"data_stored":351436513,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":42113553},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48599999999999999,"15min":0.46300000000000002},"min":{"1min":0.26400000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60799999999999998,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":0.47599999999999998},{"interface":"front","average":{"1min":0.438,"5min":0.48099999999999998,"15min":0.45500000000000002},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63700000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.42099999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.55600000000000005,"15min":0.48199999999999998},"min":{"1min":0.32000000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56599999999999995,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.29399999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.54600000000000004,"5min":1.161,"15min":1.161},"last":0.442}]},{"osd":2,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.58299999999999996,"15min":0.53600000000000003},"min":{"1min":0.312,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.60399999999999998,"5min":10.952,"15min":10.952},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.499,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.623,"5min":1.125,"15min":1.125},"last":0.41299999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.67800000000000005,"15min":0.53800000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63200000000000001,"5min":10.355,"15min":10.355},"last":0.496},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.51100000000000001,"15min":0.45900000000000002},"min":{"1min":0.29699999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65800000000000003,"5min":2.476,"15min":2.476},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.51000000000000001,"15min":0.41399999999999998},"min":{"1min":0.27600000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.66900000000000004,"5min":7.4000000000000004,"15min":7.4000000000000004},"last":0.29899999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.47299999999999998,"15min":0.40799999999999997},"min":{"1min":0.32800000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71599999999999997,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.40000000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.50900000000000001,"15min":0.46500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.23999999999999999},"max":{"1min":0.64400000000000002,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.31},{"interface":"front","average":{"1min":0.501,"5min":0.56200000000000006,"15min":0.49199999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.29699999999999999},"max":{"1min":0.70399999999999996,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.45000000000000001}]},{"osd":7,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.46300000000000002},"min":{"1min":0.36899999999999999,"5min":0.29799999999999999,"15min":0.29299999999999998},"max":{"1min":0.69199999999999995,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":0.32100000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.58799999999999997,"15min":0.51700000000000002},"min":{"1min":0.34300000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68500000000000005,"5min":5.944,"15min":5.944},"last":0.34699999999999998}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":873,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":12026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":10320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":8144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":249327616,"data_stored":366281957,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":119275520,"data_stored":164729053,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":308658176,"data_stored":403244025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":242196480,"data_stored":320971611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":100499456,"data_stored":146976591,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":252076032,"data_stored":350095563,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":175067136,"data_stored":238318969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":230596608,"data_stored":341116025,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:35:56.562 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool set default.rgw.control pgp_num 32 2021-03-15T22:35:56.576 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:56.600 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:56.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56514]: audit 2021-03-15T22:35:55.125471+0000 mgr.y (mgr.14144) 2021-03-15T22:35:56.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56514]: 1096 : audit [DBG] from='client.16881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:56.620 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56514]: cluster 2021-03-15T22:35:55.227063+0000 mgr.y (mgr.14144) 1097 : cluster [DBG] pgmap v1646: 155 pgs: 155 active+clean; 709 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 MiB/s wr, 4 op/s; 307 B/s, 0 objects/s recovering 2021-03-15T22:35:56.621 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56514]: audit 2021-03-15T22:35:56.087651+0000 mon.a (mon.0) 2489 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:35:56.621 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56514]: audit 2021-03-15T22:35:56.333026+0000 mon.a (mon.0) 2490 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:35:56.621 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56514]: cluster 2021-03-15T22:35:56.353418+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in 2021-03-15T22:35:56.622 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: debug 2021-03-15T22:35:56.613+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:35:56.623 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:35:56.623 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:56.623 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:35:56.623 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:56.624 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:35:56.624 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:56.624 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:56.624 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:35:56.625 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: debug 2021-03-15T22:35:56.613+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:35:56.625 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:35:56.625 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:56.625 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:35:56.625 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:56.626 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:35:56.626 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:56.626 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:56.626 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:35:56.627 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: debug 2021-03-15T22:35:56.614+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:35:56.627 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:35:56.627 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:56.627 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:35:56.627 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:56.628 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:35:56.628 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:56.628 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:56.628 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:35:56.629 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: debug 2021-03-15T22:35:56.614+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:35:56.629 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:35:56.629 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:56.629 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:35:56.629 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:56.630 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:35:56.630 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:56.630 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:56.630 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:35:56.631 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: debug 2021-03-15T22:35:56.615+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:35:56.631 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:35:56.631 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:56.631 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:35:56.632 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:56.632 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:35:56.632 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:56.632 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:56.632 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:35:56.633 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: debug 2021-03-15T22:35:56.616+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:35:56.633 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:35:56.633 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:56.634 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:35:56.634 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:56.634 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:35:56.635 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:56.635 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:56.635 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:35:56.636 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: debug 2021-03-15T22:35:56.616+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:35:56.636 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:35:56.636 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:56.637 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:35:56.637 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:56.637 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:35:56.637 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:56.637 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:56.638 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:35:56.638 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: audit 2021-03-15T22:35: 2021-03-15T22:35:56.638 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: 55.125471+0000 mgr.y (mgr.14144) 1096 : audit [DBG] from='client.16881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:56.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: cluster 2021-03-15T22:35:55.227063+0000 mgr.y (mgr.14144) 1097 : cluster [DBG] pgmap v1646: 155 pgs: 155 active+clean; 709 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 MiB/s wr, 4 op/s; 307 B/s, 0 objects/s recovering 2021-03-15T22:35:56.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: audit 2021-03-15T22:35: 2021-03-15T22:35:56.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: 56.087651+0000 mon.a (mon.0) 2021-03-15T22:35:56.639 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: 2489 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:35:56.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:35:56.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: 35:56.333026+0000 mon.a (mon.0) 2490 : audit [INF] 2021-03-15T22:35:56.640 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:35:56.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:35:56.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: 03-15T22:35:56.353418+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in 2021-03-15T22:35:56.641 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:56 smithi076 conmon[62156]: 2021-03-15T22:35:56.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:56.642 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:56.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:56.744 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:56.766 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:56.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:56.768 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:56.869 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:56.892 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:56.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:56.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:56.998 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:57.022 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:57.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:57.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:57.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:56 smithi125 conmon[56739]: audit 2021-03-15T22:35:55.125471+0000 mgr.y (mgr.14144 2021-03-15T22:35:57.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:56 smithi125 conmon[56739]: ) 1096 : audit [DBG] from='client.16881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:35:57.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:56 smithi125 conmon[56739]: cluster 2021-03-15T22:35:55.227063+0000 mgr.y (mgr.14144) 1097 : cluster [DBG] pgmap v1646: 155 pgs: 155 active+clean; 709 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.5 KiB/s rd, 2.2 MiB/s wr, 4 op/s; 307 B/s, 0 objects/s recovering 2021-03-15T22:35:57.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:56 smithi125 conmon[56739]: audit 2021-03-15T22:35:56.087651+0000 mon.a (mon.0) 2489 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:35:57.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:56 smithi125 conmon[56739]: audit 2021-03-15T22:35:56.333026+0000 mon.a (mon.0) 2490 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:35:57.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:56 smithi125 conmon[56739]: cluster 2021-03-15T22:35:56.353418+0000 mon.a (mon.0) 2491 : cluster [DBG] osdmap e1001: 8 total, 8 up, 8 in 2021-03-15T22:35:57.125 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:35:57.147 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:57.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:57.149 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:35:57.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:57.261 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:35:57.273 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:57.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:57.274 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:57.358 INFO:tasks.rados.rados.0.smithi076.stdout:3837: done (2 left) 2021-03-15T22:35:57.358 INFO:tasks.rados.rados.0.smithi076.stdout:3838: done (1 left) 2021-03-15T22:35:57.359 INFO:tasks.rados.rados.0.smithi076.stdout:3839: done (0 left) 2021-03-15T22:35:57.359 INFO:tasks.rados.rados.0.smithi076.stdout:3840: rollback oid 38 current snap is 455 2021-03-15T22:35:57.359 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 38 to 403 2021-03-15T22:35:57.360 INFO:tasks.rados.rados.0.smithi076.stdout:3841: write (excl) oid 20 current snap is 455 2021-03-15T22:35:57.360 INFO:tasks.rados.rados.0.smithi076.stdout:3841: seq_num 1304 ranges {716328=421059,1840284=509126,3143112=659293} 2021-03-15T22:35:57.365 INFO:tasks.rados.rados.0.smithi076.stdout:3841: writing smithi07694878-20 from 716328 to 1137387 tid 1 2021-03-15T22:35:57.371 INFO:tasks.rados.rados.0.smithi076.stdout:3841: writing smithi07694878-20 from 1840284 to 2349410 tid 2 2021-03-15T22:35:57.375 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:57.380 INFO:tasks.rados.rados.0.smithi076.stdout:3841: writing smithi07694878-20 from 3143112 to 3802405 tid 3 2021-03-15T22:35:57.380 INFO:tasks.rados.rados.0.smithi076.stdout:3842: snap_create 2021-03-15T22:35:57.380 INFO:tasks.rados.rados.0.smithi076.stdout:3840: finishing rollback tid 0 to smithi07694878-38 2021-03-15T22:35:57.381 INFO:tasks.rados.rados.0.smithi076.stdout:3840: finishing rollback tid 1 to smithi07694878-38 2021-03-15T22:35:57.381 INFO:tasks.rados.rados.0.smithi076.stdout:3840: finishing rollback tid 2 to smithi07694878-38 2021-03-15T22:35:57.381 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2845 (ObjNum 1116 snap 388 seq_num 1116) dirty exists 2021-03-15T22:35:57.381 INFO:tasks.rados.rados.0.smithi076.stdout:3841: finishing write tid 1 to smithi07694878-20 2021-03-15T22:35:57.381 INFO:tasks.rados.rados.0.smithi076.stdout:3841: finishing write tid 2 to smithi07694878-20 2021-03-15T22:35:57.383 INFO:tasks.rados.rados.0.smithi076.stdout:3841: finishing write tid 3 to smithi07694878-20 2021-03-15T22:35:57.384 INFO:tasks.rados.rados.0.smithi076.stdout:3841: finishing write tid 4 to smithi07694878-20 2021-03-15T22:35:57.384 INFO:tasks.rados.rados.0.smithi076.stdout:3841: finishing write tid 5 to smithi07694878-20 2021-03-15T22:35:57.384 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 2698 (ObjNum 1304 snap 455 seq_num 1304) dirty exists 2021-03-15T22:35:57.385 INFO:tasks.rados.rados.0.smithi076.stdout:3841: left oid 20 (ObjNum 1304 snap 455 seq_num 1304) 2021-03-15T22:35:57.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:57.403 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:57.403 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:57.504 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:57.530 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:57.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:57.532 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:57.633 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:57.657 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:57.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:57.659 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:57.760 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:57.787 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:57.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:57.788 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:57.889 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:57.915 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:57.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:57.917 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:58.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: cluster 2021-03-15T22:35:56.497095+0000 osd.7 (osd.7) 188 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:58.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: cluster 2021-03-15T22:35:56.499148+0000 osd.7 (osd.7) 189 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:58.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:35:56.607967+0000 mgr.y (mgr.14144) 1098 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:35:58.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: audit 2021-03-15T22:35:56.612722+0000 mon.a (mon.0) 2492 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:35:58.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:35:56.613819+0000 mgr.y (mgr.14144) 1099 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:35:58.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:35:58.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:35:58.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:35:58.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:35:58.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:35:56.614476+0000 mgr.y (mgr. 2021-03-15T22:35:58.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: 14144) 1100 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:35:58.021 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:35:58.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:35:58.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.022 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:35:58.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:35:58.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:35:56.615047+0000 mgr.y (mgr.14144) 1101 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:35:58.023 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:35:58.024 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cluster 2021-03-15T22:35:56.497095+0000 osd.7 (osd.7) 188 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:58.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cluster 2021-03-15T22:35:56.499148+0000 osd.7 (osd.7) 189 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:58.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:35:56.607967+0000 mgr.y (mgr.14144) 1098 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:35:58.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: audit 2021-03-15T22:35:56.612722+0000 mon.a (mon.0) 2492 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:35:58.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:35:56.613819+0000 mgr.y (mgr.14144) 1099 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:35:58.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:35:58.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:35:58.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:35:58.027 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:35:58.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:35:56.614476+0000 mgr.y (mgr.14144) 1100 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:35:58.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:35:58.028 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:35:58.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:35:58.029 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:35:58.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:35:56.615047+0000 mgr.y (mgr.14144) 1101 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:35:58.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:35:58.030 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:35:58.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.031 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:35:58.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:35:58.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:35:56.615581+0000 mgr.y (mgr.14144) 1102 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:35:58.032 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:35:58.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:35:58.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:35:58.033 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:35:58.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:35:56.616178+0000 mgr.y (mgr.14144) 1103 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:35:58.034 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:35:58.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:35:58.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:35:58.035 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:35:58.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:35:56.616761+0000 mgr.y (mgr.14144) 1104 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:35:58.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:35:58.036 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:35:58.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:35:58.037 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:35:58.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.038 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:35:58.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:35:58.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:35:56.615581+0000 mgr.y (mgr.14144) 1102 : 2021-03-15T22:35:58.039 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:35:58.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:35:58.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:35:58.040 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:35:58.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:35:58.041 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:35:56.616178+0000 mgr.y (mgr.14144) 1103 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:35:58.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:35:58.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.042 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:35:58.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:35:58.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.043 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:35:58.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:35:56.616761+0000 mgr.y (mgr.14144) 1104 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:35:58.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:35:58.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.044 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:35:58.045 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.045 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:35:58.045 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.045 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.046 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:35:58.046 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:35:56.617297+0000 mgr.y (mgr.14144) 1105 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:35:58.046 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:35:58.047 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.047 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:35:58.047 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.047 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:35:58.047 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.048 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.048 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:35:58.048 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: audit 2021-03-15T22:35:57.228489+0000 mon.a (mon.0) 2493 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "37"}]: dispatch 2021-03-15T22:35:58.048 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: audit 2021-03-15T22:35:57.354713+0000 mon.a (mon.0) 2494 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "37"}]': finished 2021-03-15T22:35:58.049 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: cluster 2021-03-15T22:35:57.354764+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e1002: 8 total, 8 up, 8 in 2021-03-15T22:35:58.049 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[56514]: audit 2021-03-15T22:35:57.362637+0000 mon.a (mon.0) 2496 : audit [INF] from='client.? 172.21.15.76:0/1139720748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:58.049 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.050 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:35:58.050 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:35:56.617297+0000 mgr.y (mgr.14144) 1105 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:35:58.050 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:35:58.050 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.051 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:35:58.051 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.051 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:35:58.051 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.052 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.052 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:35:58.052 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: audit 2021-03-15T22:35:57.228489+0000 mon.a (mon.0) 2493 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "37"}]: dispatch 2021-03-15T22:35:58.052 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: audit 2021-03-15T22:35:57.354713+0000 mon.a (mon.0) 2494 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "37"}]': finished 2021-03-15T22:35:58.052 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: cluster 2021-03-15T22:35:57.354764+0000 mon.a (mon.0) 2495 : cluster [DBG] osdmap e1002: 8 total, 8 up, 8 in 2021-03-15T22:35:58.053 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:57 smithi076 conmon[62156]: audit 2021-03-15T22:35:57.362637+0000 mon.a (mon.0) 2496 : audit [INF] from='client.? 172.21.15.76:0/1139720748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:58.053 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:58.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: cluster 2021-03-15T22:35:56.497095+0000 osd.7 (osd.7) 188 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:35:58.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: cluster 2021-03-15T22:35 2021-03-15T22:35:58.055 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: :56.499148+0000 osd.7 (osd.7) 189 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:35:58.055 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:35:56.607967+0000 mgr.y (mgr.14144) 1098 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:35:58.055 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:35:58.055 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: -15T22:35:56.612722+0000 mon.a (mon.0) 2492 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:35:58.056 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: cephadm 2021 2021-03-15T22:35:58.056 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: -03-15T22:35:56.613819+0000 2021-03-15T22:35:58.056 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: mgr.y (mgr.14144) 1099 : cephadm [ERR] 2021-03-15T22:35:58.056 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='y'), HostPlacementSpec(hostname='smithi125', network='', name='x')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.y already in use 2021-03-15T22:35:58.056 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:35:58.057 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.057 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:35:58.057 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.057 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:35:58.057 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.058 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.058 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.y already in use 2021-03-15T22:35:58.058 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:35:56.614476+0000 mgr.y (mgr.14144) 1100 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:35:58.058 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:35:58.059 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.059 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:35:58.059 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.059 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:35:58.059 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.060 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.060 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:35:58.060 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:35:58.060 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: 2021-03-15T22:35:56.615047+0000 mgr.y (mgr.14144) 1101 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:35:58.060 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:35:58.061 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.061 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:35:58.061 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.061 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:35:58.061 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.062 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.062 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:35:58.062 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:35: 2021-03-15T22:35:58.062 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: 56.615581+0000 mgr.y (mgr.14144) 1102 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:35:58.063 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:35:58.063 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.063 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:35:58.063 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.063 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:35:58.064 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.064 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.064 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:35:58.064 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:35:56.616178+0000 mgr.y (mgr.14144 2021-03-15T22:35:58.064 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: ) 1103 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:35:58.065 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:35:58.065 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.065 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:35:58.065 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.066 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:35:58.066 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.066 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.066 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:35:58.066 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:35:56.616761+0000 mgr.y (mgr.14144) 1104 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:35:58.067 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:35:58.067 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.067 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:35:58.067 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.068 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:35:58.068 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.068 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.068 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:35:58.068 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: 2021-03-15T22:35:58.069 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:35:56.617297+0000 mgr.y (mgr.14144) 1105 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:35:58.069 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:35:58.069 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:35:58.069 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:35:58.070 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:35:58.070 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:35:58.070 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:35:58.070 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:35:58.070 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:35:58.071 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: audit 2021-03-15T22:35:57.228489+0000 mon.a (mon.0) 2493 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "37"}]: dispatch 2021-03-15T22:35:58.071 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: audit 2021-03-15T22:35:57.354713+0000 mon.a (mon.0) 2494 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "37"}]': finished 2021-03-15T22:35:58.071 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:35:58.071 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: 22:35:57.354764+0000 mon.a (mon.0) 2495 : cluster 2021-03-15T22:35:58.072 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: [DBG] osdmap e1002: 8 total, 8 up, 8 in 2021-03-15T22:35:58.072 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:35:58.072 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: 35:57.362637+0000 mon.a (mon.0) 2021-03-15T22:35:58.072 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:57 smithi125 conmon[56739]: 2496 : audit [INF] from='client.? 172.21.15.76:0/1139720748' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:35:58.093 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:58.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:58.095 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:58.196 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:58.222 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:58.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:58.223 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:58.324 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:58.349 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:58.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:58.350 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:58.360 INFO:tasks.rados.rados.0.smithi076.stdout:3840: done (2 left) 2021-03-15T22:35:58.360 INFO:tasks.rados.rados.0.smithi076.stdout:3841: done (1 left) 2021-03-15T22:35:58.360 INFO:tasks.rados.rados.0.smithi076.stdout:3842: done (0 left) 2021-03-15T22:35:58.360 INFO:tasks.rados.rados.0.smithi076.stdout:3843: read oid 49 snap -1 2021-03-15T22:35:58.360 INFO:tasks.rados.rados.0.smithi076.stdout:3843: expect (ObjNum 1225 snap 424 seq_num 1225) 2021-03-15T22:35:58.362 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:35:58.363 INFO:tasks.rados.rados.0.smithi076.stdout:3844: write (excl) oid 15 current snap is 456 2021-03-15T22:35:58.363 INFO:tasks.rados.rados.0.smithi076.stdout:3844: seq_num 1305 ranges {733135=496308,1898766=652784,3170535=470565} 2021-03-15T22:35:58.370 INFO:tasks.rados.rados.0.smithi076.stdout:3844: writing smithi07694878-15 from 733135 to 1229443 tid 1 2021-03-15T22:35:58.376 INFO:tasks.rados.rados.0.smithi076.stdout:3844: writing smithi07694878-15 from 1898766 to 2551550 tid 2 2021-03-15T22:35:58.383 INFO:tasks.rados.rados.0.smithi076.stdout:3844: writing smithi07694878-15 from 3170535 to 3641100 tid 3 2021-03-15T22:35:58.383 INFO:tasks.rados.rados.0.smithi076.stdout:3845: rollback oid 20 current snap is 456 2021-03-15T22:35:58.383 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 20 to 345 2021-03-15T22:35:58.383 INFO:tasks.rados.rados.0.smithi076.stdout:3843: expect (ObjNum 1225 snap 424 seq_num 1225) 2021-03-15T22:35:58.451 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:58.475 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:58.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:58.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:58.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:58.601 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:58.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:58.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:58.623 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:58 smithi076 conmon[56514]: cluster 2021-03-15T22:35:57.227757+0000 mgr.y (mgr.14144) 1106 : cluster [DBG] pgmap v1649: 154 pgs: 154 active+clean; 720 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 10 op/s; 218 B/s, 0 objects/s recovering 2021-03-15T22:35:58.624 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:58 smithi076 conmon[56514]: cluster 2021-03-15T22:35:58.351894+0000 mon.a (mon.0) 2497 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:58.624 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:58 smithi076 conmon[56514]: audit 2021-03-15T22:35:58.355717+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? 172.21.15.76:0/1139720748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:58.624 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:58 smithi076 conmon[56514]: cluster 2021-03-15T22:35:58.355786+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e1003: 8 total, 8 up, 8 in 2021-03-15T22:35:58.624 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:58 smithi076 conmon[62156]: cluster 2021-03-15T22:35:57.227757+0000 mgr.y (mgr.14144) 1106 : cluster [DBG] pgmap v1649: 154 pgs: 154 active+clean; 720 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 10 op/s; 218 B/s, 0 objects/s recovering 2021-03-15T22:35:58.625 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:58 smithi076 conmon[62156]: cluster 2021-03-15T22:35:58.351894+0000 mon.a (mon.0) 2497 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:58.625 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:58 smithi076 conmon[62156]: audit 2021-03-15T22:35:58.355717+0000 mon.a (mon.0) 2498 : audit [INF] from='client.? 172.21.15.76:0/1139720748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:58.625 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:58 smithi076 conmon[62156]: cluster 2021-03-15T22:35:58.355786+0000 mon.a (mon.0) 2499 : cluster [DBG] osdmap e1003: 8 total, 8 up, 8 in 2021-03-15T22:35:58.647 INFO:tasks.rados.rados.0.smithi076.stdout:3844: finishing write tid 1 to smithi07694878-15 2021-03-15T22:35:58.648 INFO:tasks.rados.rados.0.smithi076.stdout:3844: finishing write tid 2 to smithi07694878-15 2021-03-15T22:35:58.648 INFO:tasks.rados.rados.0.smithi076.stdout:3845: finishing rollback tid 0 to smithi07694878-20 2021-03-15T22:35:58.648 INFO:tasks.rados.rados.0.smithi076.stdout:3845: finishing rollback tid 1 to smithi07694878-20 2021-03-15T22:35:58.648 INFO:tasks.rados.rados.0.smithi076.stdout:3844: finishing write tid 3 to smithi07694878-15 2021-03-15T22:35:58.649 INFO:tasks.rados.rados.0.smithi076.stdout:3844: finishing write tid 4 to smithi07694878-15 2021-03-15T22:35:58.649 INFO:tasks.rados.rados.0.smithi076.stdout:3844: finishing write tid 5 to smithi07694878-15 2021-03-15T22:35:58.649 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 15 v 3244 (ObjNum 1305 snap 456 seq_num 1305) dirty exists 2021-03-15T22:35:58.649 INFO:tasks.rados.rados.0.smithi076.stdout:3844: left oid 15 (ObjNum 1305 snap 456 seq_num 1305) 2021-03-15T22:35:58.650 INFO:tasks.rados.rados.0.smithi076.stdout:3845: finishing rollback tid 2 to smithi07694878-20 2021-03-15T22:35:58.650 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 20 v 2701 (ObjNum 962 snap 321 seq_num 962) dirty exists 2021-03-15T22:35:58.650 INFO:tasks.rados.rados.0.smithi076.stdout:3843: done (2 left) 2021-03-15T22:35:58.650 INFO:tasks.rados.rados.0.smithi076.stdout:3844: done (1 left) 2021-03-15T22:35:58.651 INFO:tasks.rados.rados.0.smithi076.stdout:3845: done (0 left) 2021-03-15T22:35:58.651 INFO:tasks.rados.rados.0.smithi076.stdout:3846: delete oid 9 current snap is 456 2021-03-15T22:35:58.651 INFO:tasks.rados.rados.0.smithi076.stdout:3846: done (0 left) 2021-03-15T22:35:58.652 INFO:tasks.rados.rados.0.smithi076.stdout:3847: snap_create 2021-03-15T22:35:58.704 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:58.729 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:58.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:58.730 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:58.831 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:35:58.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:58.855 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:58.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:35:58.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:35:58.983 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:58.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:58.985 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:35:59.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:58 smithi125 conmon[56739]: cluster 2021-03-15T22:35:57.227757+0000 mgr.y (mgr.14144) 1106 : cluster [DBG] 2021-03-15T22:35:59.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:58 smithi125 conmon[56739]: pgmap v1649: 154 pgs: 154 active+clean; 720 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 4.8 MiB/s wr, 10 op/s; 218 B/s, 0 objects/s recovering 2021-03-15T22:35:59.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:58 smithi125 conmon[56739]: cluster 2021-03-15T22:35:58.351894+0000 mon.a (mon.0) 2497 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:35:59.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:58 smithi125 conmon[56739]: audit 2021-03-15T22:35:58.355717 2021-03-15T22:35:59.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:58 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2498 : audit [INF] 2021-03-15T22:35:59.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:58 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1139720748' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:35:59.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:58 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:35:59.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:58 smithi125 conmon[56739]: 15T22:35:58.355786+0000 2021-03-15T22:35:59.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:58 smithi125 conmon[56739]: mon.a (mon.0) 2499 : cluster [DBG] 2021-03-15T22:35:59.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:58 smithi125 conmon[56739]: osdmap e1003: 8 total, 8 up, 8 in 2021-03-15T22:35:59.086 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:35:59.111 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:59.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:59.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:35:59.214 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:59.239 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:59.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:59.241 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:59.342 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:59.345 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:35:59.363 INFO:tasks.rados.rados.0.smithi076.stdout:3847: done (0 left) 2021-03-15T22:35:59.363 INFO:tasks.rados.rados.0.smithi076.stdout:3848: read oid 5 snap -1 2021-03-15T22:35:59.363 INFO:tasks.rados.rados.0.smithi076.stdout:3848: expect deleted 2021-03-15T22:35:59.364 INFO:tasks.rados.rados.0.smithi076.stdout:3849: copy_from oid 42 from oid 10 current snap is 457 2021-03-15T22:35:59.364 INFO:tasks.rados.rados.0.smithi076.stdout:3850: snap_create 2021-03-15T22:35:59.369 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:35:59.371 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:59.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:59.373 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:59.388 INFO:tasks.rados.rados.0.smithi076.stdout:3849: finishing copy_from to smithi07694878-42 2021-03-15T22:35:59.389 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 3170 (ObjNum 1269 snap 437 seq_num 1269) dirty exists 2021-03-15T22:35:59.389 INFO:tasks.rados.rados.0.smithi076.stdout:3849: finishing copy_from racing read to smithi07694878-42 2021-03-15T22:35:59.389 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:35:59.407 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:35:59.429 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:35:59.433 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:35:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:35:59.434 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:35:59.434 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:35:59.450 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:35:59.472 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:35:59.474 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:35:59.487 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:35:59.488 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:35:59.489 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:35:59.489 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:35:59.489 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:35:59.489 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:35:59.490 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:35:59.490 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:35:59.490 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:35:59.497 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:59.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:59.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:35:59.600 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:35:59.624 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:35:59.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:59.626 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:35:59.727 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:35:59.753 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:59.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:59.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:35:59.856 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:35:59.881 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:35:59.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:35:59.883 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:35:59.984 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:00.006 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:00.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:00.008 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:00.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:59 smithi076 conmon[56514]: audit 2021-03-15T22:35:59.228968 2021-03-15T22:36:00.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:59 smithi076 conmon[56514]: +0000 mon.a (mon.0) 2500 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-03-15T22:36:00.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:59 smithi076 conmon[56514]: audit 2021-03-15T22:35:59.325776+0000 mon.a (mon.0) 2501 : audit [DBG] from='client.? 172.21.15.76:0/212314085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:00.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:59 smithi076 conmon[56514]: audit 2021-03-15T22:35:59.359652+0000 mon.a (mon.0) 2502 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]': finished 2021-03-15T22:36:00.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:35:59 smithi076 conmon[56514]: cluster 2021-03-15T22:35:59.359728+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e1004: 8 total, 8 up, 8 in 2021-03-15T22:36:00.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:59 smithi076 conmon[62156]: audit 2021-03-15T22:35:59 2021-03-15T22:36:00.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:59 smithi076 conmon[62156]: .228968+0000 mon.a (mon.0) 2500 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-03-15T22:36:00.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:59 smithi076 conmon[62156]: audit 2021-03-15T22:35:59.325776+0000 mon.a (mon.0) 2501 : audit [DBG] from='client.? 172.21.15.76:0/212314085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:00.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:59 smithi076 conmon[62156]: audit 2021-03-15T22:35:59.359652+0000 mon.a (mon.0) 2502 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]': finished 2021-03-15T22:36:00.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:35:59 smithi076 conmon[62156]: cluster 2021-03-15T22:35:59.359728+0000 mon.a (mon.0) 2503 : cluster [DBG] osdmap e1004: 8 total, 8 up, 8 in 2021-03-15T22:36:00.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:59 smithi125 conmon[56739]: audit 2021-03-15T22:35:59.228968+0000 mon.a (mon.0) 2500 : audit [INF] 2021-03-15T22:36:00.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:59 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]: dispatch 2021-03-15T22:36:00.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:59 smithi125 conmon[56739]: audit 2021-03-15T22:35:59.325776+0000 mon.a (mon.0) 2501 : audit [DBG] from='client.? 172.21.15.76:0/212314085' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:00.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:59 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:36:00.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:59 smithi125 conmon[56739]: 03-15T22:35:59.359652+0000 mon.a (mon.0) 2502 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "30"}]': finished 2021-03-15T22:36:00.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:59 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:36:00.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:59 smithi125 conmon[56739]: -03-15T22:35: 2021-03-15T22:36:00.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:59 smithi125 conmon[56739]: 59.359728+0000 mon.a (mon.0) 2503 2021-03-15T22:36:00.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:35:59 smithi125 conmon[56739]: : cluster [DBG] osdmap e1004: 8 total, 8 up, 8 in 2021-03-15T22:36:00.109 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:00.131 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:00.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:00.133 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:00.234 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:00.261 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:00.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:00.263 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:00.353 INFO:tasks.rados.rados.0.smithi076.stdout:3848: done (2 left) 2021-03-15T22:36:00.354 INFO:tasks.rados.rados.0.smithi076.stdout:3849: done (1 left) 2021-03-15T22:36:00.354 INFO:tasks.rados.rados.0.smithi076.stdout:3850: done (0 left) 2021-03-15T22:36:00.354 INFO:tasks.rados.rados.0.smithi076.stdout:3851: rollback oid 8 current snap is 458 2021-03-15T22:36:00.354 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 8 to 422 2021-03-15T22:36:00.355 INFO:tasks.rados.rados.0.smithi076.stdout:3852: read oid 50 snap -1 2021-03-15T22:36:00.355 INFO:tasks.rados.rados.0.smithi076.stdout:3852: expect deleted 2021-03-15T22:36:00.355 INFO:tasks.rados.rados.0.smithi076.stdout:3853: read oid 12 snap -1 2021-03-15T22:36:00.356 INFO:tasks.rados.rados.0.smithi076.stdout:3853: expect (ObjNum 1285 snap 442 seq_num 1285) 2021-03-15T22:36:00.356 INFO:tasks.rados.rados.0.smithi076.stdout:3854: write (excl) oid 45 current snap is 458 2021-03-15T22:36:00.356 INFO:tasks.rados.rados.0.smithi076.stdout:3854: seq_num 1307 ranges {766749=646806,1899377=656453,3109028=609462} 2021-03-15T22:36:00.363 INFO:tasks.rados.rados.0.smithi076.stdout:3854: writing smithi07694878-45 from 766749 to 1413555 tid 1 2021-03-15T22:36:00.363 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:00.370 INFO:tasks.rados.rados.0.smithi076.stdout:3854: writing smithi07694878-45 from 1899377 to 2555830 tid 2 2021-03-15T22:36:00.377 INFO:tasks.rados.rados.0.smithi076.stdout:3854: writing smithi07694878-45 from 3109028 to 3718490 tid 3 2021-03-15T22:36:00.378 INFO:tasks.rados.rados.0.smithi076.stdout:3855: read oid 23 snap -1 2021-03-15T22:36:00.378 INFO:tasks.rados.rados.0.smithi076.stdout:3855: expect deleted 2021-03-15T22:36:00.378 INFO:tasks.rados.rados.0.smithi076.stdout:3851: finishing rollback tid 0 to smithi07694878-8 2021-03-15T22:36:00.378 INFO:tasks.rados.rados.0.smithi076.stdout:3851: finishing rollback tid 1 to smithi07694878-8 2021-03-15T22:36:00.379 INFO:tasks.rados.rados.0.smithi076.stdout:3856: read oid 29 snap 416 2021-03-15T22:36:00.379 INFO:tasks.rados.rados.0.smithi076.stdout:3856: expect deleted 2021-03-15T22:36:00.379 INFO:tasks.rados.rados.0.smithi076.stdout:3852: done (5 left) 2021-03-15T22:36:00.379 INFO:tasks.rados.rados.0.smithi076.stdout:3857: snap_create 2021-03-15T22:36:00.379 INFO:tasks.rados.rados.0.smithi076.stdout:3851: finishing rollback tid 2 to smithi07694878-8 2021-03-15T22:36:00.380 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 8 v 3166 (ObjNum 1203 snap 418 seq_num 1203) dirty exists 2021-03-15T22:36:00.380 INFO:tasks.rados.rados.0.smithi076.stdout:3853: expect (ObjNum 1285 snap 442 seq_num 1285) 2021-03-15T22:36:00.392 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:00.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:00.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:00.494 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:00.515 INFO:tasks.rados.rados.0.smithi076.stdout:3854: finishing write tid 1 to smithi07694878-45 2021-03-15T22:36:00.515 INFO:tasks.rados.rados.0.smithi076.stdout:3854: finishing write tid 2 to smithi07694878-45 2021-03-15T22:36:00.515 INFO:tasks.rados.rados.0.smithi076.stdout:3854: finishing write tid 3 to smithi07694878-45 2021-03-15T22:36:00.516 INFO:tasks.rados.rados.0.smithi076.stdout:3854: finishing write tid 4 to smithi07694878-45 2021-03-15T22:36:00.516 INFO:tasks.rados.rados.0.smithi076.stdout:3854: finishing write tid 5 to smithi07694878-45 2021-03-15T22:36:00.516 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 3088 (ObjNum 1307 snap 458 seq_num 1307) dirty exists 2021-03-15T22:36:00.516 INFO:tasks.rados.rados.0.smithi076.stdout:3854: left oid 45 (ObjNum 1307 snap 458 seq_num 1307) 2021-03-15T22:36:00.518 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:00.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:00.620 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:00.629 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:36:00 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:36:00.645 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:00.646 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:00.747 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:00.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:00.771 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:00.771 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:00.873 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:00.898 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:00.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:00.900 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:00.902 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[56514]: cluster 2021-03-15T22:35:59. 2021-03-15T22:36:00.903 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[56514]: 228205+0000 mgr.y (mgr.14144) 1107 : cluster [DBG] pgmap v1652: 153 pgs: 153 active+clean; 720 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-03-15T22:36:00.903 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[56514]: cluster 2021-03-15T22:35:59.416835+0000 osd.0 (osd.0) 161 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:00.903 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[56514]: cluster 2021-03-15T22:35:59.419038+0000 osd.0 (osd.0) 162 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:00.903 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[56514]: cluster 2021-03-15T22:36:00.349671+0000 mon.a (mon.0) 2504 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in 2021-03-15T22:36:00.904 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: cluster 2021-03-15T22:35:59 2021-03-15T22:36:00.904 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: .228205+0000 mgr.y (mgr.14144) 1107 : cluster [DBG] pgmap v1652: 153 pgs: 153 active+clean; 720 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-03-15T22:36:00.904 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: cluster 2021-03-15T22:35:59.416835 2021-03-15T22:36:00.905 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: +0000 osd.0 (osd.0) 161 2021-03-15T22:36:00.905 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:00.905 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:36:00.905 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: 03-15T22:35:59 2021-03-15T22:36:00.906 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: .419038+0000 osd.0 (osd. 2021-03-15T22:36:00.906 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: 0) 162 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:00.906 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: 2021-03-15T22:36:00.906 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: cluster 2021-03-15T22:36:00.906 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: 2021-03-15T22:36:00.349671+0000 2021-03-15T22:36:00.907 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: mon.a (mon.0) 2504 : cluster 2021-03-15T22:36:00.907 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:00 smithi076 conmon[62156]: [DBG] osdmap e1005: 8 total, 8 up, 8 in 2021-03-15T22:36:01.001 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:01.024 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:01.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:01.025 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:01.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:00 smithi125 conmon[56739]: cluster 2021-03-15T22:35:59.228205+0000 mgr.y (mgr.14144) 1107 : cluster [DBG] pgmap v1652: 153 pgs: 153 active+clean; 720 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 3.8 MiB/s rd, 2.6 MiB/s wr, 5 op/s 2021-03-15T22:36:01.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:00 smithi125 conmon[56739]: 2021-03-15T22:36:01.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:00 smithi125 conmon[56739]: cluster 2021-03-15T22:35:59.416835+0000 osd.0 (osd.0) 161 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:01.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:00 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:36:01.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:00 smithi125 conmon[56739]: -03-15T22:35:59.419038+0000 osd.0 (osd.0) 162 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:01.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:00 smithi125 conmon[56739]: cluster 2021-03-15T22:36:01.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:00 smithi125 conmon[56739]: 2021-03-15T22:36:00.349671+0000 mon.a (mon.0 2021-03-15T22:36:01.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:00 smithi125 conmon[56739]: ) 2504 : cluster [DBG] osdmap e1005: 8 total, 8 up, 8 in 2021-03-15T22:36:01.126 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:01.151 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:01.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:01.153 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:01.254 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:01.280 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:01.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:01.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:01.283 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_blocked_ops 2021-03-15T22:36:01.369 INFO:tasks.rados.rados.0.smithi076.stdout:3851: done (5 left) 2021-03-15T22:36:01.369 INFO:tasks.rados.rados.0.smithi076.stdout:3853: done (4 left) 2021-03-15T22:36:01.369 INFO:tasks.rados.rados.0.smithi076.stdout:3854: done (3 left) 2021-03-15T22:36:01.370 INFO:tasks.rados.rados.0.smithi076.stdout:3855: done (2 left) 2021-03-15T22:36:01.370 INFO:tasks.rados.rados.0.smithi076.stdout:3856: done (1 left) 2021-03-15T22:36:01.370 INFO:tasks.rados.rados.0.smithi076.stdout:3857: done (0 left) 2021-03-15T22:36:01.370 INFO:tasks.rados.rados.0.smithi076.stdout:3858: write oid 18 current snap is 459 2021-03-15T22:36:01.371 INFO:tasks.rados.rados.0.smithi076.stdout:3858: seq_num 1308 ranges {783556=438408,1674212=400111,2736451=420734,3882270=101286} 2021-03-15T22:36:01.374 INFO:tasks.rados.rados.0.smithi076.stdout:3858: writing smithi07694878-18 from 783556 to 1221964 tid 1 2021-03-15T22:36:01.379 INFO:tasks.rados.rados.0.smithi076.stdout:3858: writing smithi07694878-18 from 1674212 to 2074323 tid 2 2021-03-15T22:36:01.383 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:01.385 INFO:tasks.rados.rados.0.smithi076.stdout:3858: writing smithi07694878-18 from 2736451 to 3157185 tid 3 2021-03-15T22:36:01.390 INFO:tasks.rados.rados.0.smithi076.stdout:3858: writing smithi07694878-18 from 3882270 to 3983556 tid 4 2021-03-15T22:36:01.391 INFO:tasks.rados.rados.0.smithi076.stdout:3859: write oid 41 current snap is 459 2021-03-15T22:36:01.391 INFO:tasks.rados.rados.0.smithi076.stdout:3858: finishing write tid 1 to smithi07694878-18 2021-03-15T22:36:01.391 INFO:tasks.rados.rados.0.smithi076.stdout:3859: seq_num 1309 ranges {400363=513657,1332694=660122} 2021-03-15T22:36:01.395 INFO:tasks.rados.rados.0.smithi076.stdout:3859: writing smithi07694878-41 from 400363 to 914020 tid 1 2021-03-15T22:36:01.403 INFO:tasks.rados.rados.0.smithi076.stdout:3859: writing smithi07694878-41 from 1332694 to 1992816 tid 2 2021-03-15T22:36:01.403 INFO:tasks.rados.rados.0.smithi076.stdout:3860: read oid 7 snap -1 2021-03-15T22:36:01.403 INFO:tasks.rados.rados.0.smithi076.stdout:3860: expect (ObjNum 1270 snap 437 seq_num 1270) 2021-03-15T22:36:01.403 INFO:tasks.rados.rados.0.smithi076.stdout:3861: write oid 27 current snap is 459 2021-03-15T22:36:01.404 INFO:tasks.rados.rados.0.smithi076.stdout:3858: finishing write tid 2 to smithi07694878-18 2021-03-15T22:36:01.404 INFO:tasks.rados.rados.0.smithi076.stdout:3861: seq_num 1310 ranges {417170=588906,1674823=342347} 2021-03-15T22:36:01.408 INFO:tasks.rados.rados.0.smithi076.stdout:3861: writing smithi07694878-27 from 417170 to 1006076 tid 1 2021-03-15T22:36:01.409 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:01.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:01.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:01.414 INFO:tasks.rados.rados.0.smithi076.stdout:3861: writing smithi07694878-27 from 1674823 to 2017170 tid 2 2021-03-15T22:36:01.414 INFO:tasks.rados.rados.0.smithi076.stdout:3862: snap_remove snap 452 2021-03-15T22:36:01.512 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:01.537 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:01.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:01.539 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:01.640 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:01.663 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:01.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:01.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:01.677 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:01.766 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:01.788 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:01.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:01.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:01.849 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[62156]: audit 2021-03-15T22:36:00.317904+0000 mgr.y (mgr.14144) 1108 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:01.850 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[62156]: audit 2021-03-15T22:36:01.123456+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:36:01.850 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[62156]: audit 2021-03-15T22:36:01.124593+0000 mon.a (mon.0) 2506 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:36:01.851 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[62156]: audit 2021-03-15T22:36:01.229552+0000 mon.a (mon.0) 2507 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "36"}]: dispatch 2021-03-15T22:36:01.851 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[62156]: audit 2021-03-15T22:36:01.359759+0000 mon.a (mon.0) 2508 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "36"}]': finished 2021-03-15T22:36:01.851 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[62156]: cluster 2021-03-15T22:36:01.359817+0000 mon.a (mon.0) 2509 : cluster [DBG] osdmap e1006: 8 total, 8 up, 8 in 2021-03-15T22:36:01.852 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[56514]: audit 2021-03-15T22:36:00.317904+0000 mgr.y (mgr.14144) 1108 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:01.853 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[56514]: audit 2021-03-15T22:36:01.123456+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:36:01.853 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[56514]: audit 2021-03-15T22:36:01.124593+0000 mon.a (mon.0) 2506 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:36:01.854 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[56514]: audit 2021-03-15T22:36:01.229552+0000 mon.a (mon.0) 2507 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "36"}]: dispatch 2021-03-15T22:36:01.854 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[56514]: audit 2021-03-15T22:36:01.359759+0000 mon.a (mon.0) 2508 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "36"}]': finished 2021-03-15T22:36:01.855 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[56514]: cluster 2021-03-15T22:36 2021-03-15T22:36:01.855 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:01 smithi076 conmon[56514]: :01.359817+0000 mon.a (mon.0) 2509 : cluster [DBG] osdmap e1006: 8 total, 8 up, 8 in 2021-03-15T22:36:01.892 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:01.919 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:01.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:01.920 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:02.021 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:02.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: audit 2021-03-15T22:36:00.317904+0000 mgr.y (mgr.14144) 1108 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:02.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: 2021-03-15T22:36:02.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: audit 2021-03-15T22:36:01.123456+0000 mon.a (mon.0) 2505 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2021-03-15T22:36:02.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: audit 2021-03-15T22:36:01.124593+0000 mon.a (mon.0) 2506 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2021-03-15T22:36:02.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: audit 2021-03-15T22:36:01.229552+0000 mon.a (mon.0) 2507 : audit 2021-03-15T22:36:02.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "36"}]: dispatch 2021-03-15T22:36:02.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: audit 2021-03-15T22:36:01. 2021-03-15T22:36:02.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: 359759+0000 mon.a (mon.0) 2508 : audit [INF] 2021-03-15T22:36:02.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "36"}]': finished 2021-03-15T22:36:02.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: cluster 2021-03-15T22:36:02.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: 2021-03-15T22:36:01.359817+0000 mon.a (mon 2021-03-15T22:36:02.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:01 smithi125 conmon[56739]: .0) 2509 : cluster [DBG] osdmap e1006: 8 total, 8 up, 8 in 2021-03-15T22:36:02.047 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:02.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:02.049 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:02.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:02.173 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:02.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:02.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:02.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:02.304 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:02.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:02.306 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:02.366 INFO:tasks.rados.rados.0.smithi076.stdout:3862: done (4 left) 2021-03-15T22:36:02.366 INFO:tasks.rados.rados.0.smithi076.stdout:3863: read oid 21 snap -1 2021-03-15T22:36:02.367 INFO:tasks.rados.rados.0.smithi076.stdout:3863: expect (ObjNum 1303 snap 451 seq_num 1303) 2021-03-15T22:36:02.367 INFO:tasks.rados.rados.0.smithi076.stdout:3864: read oid 37 snap -1 2021-03-15T22:36:02.367 INFO:tasks.rados.rados.0.smithi076.stdout:3864: expect (ObjNum 1284 snap 442 seq_num 1284) 2021-03-15T22:36:02.367 INFO:tasks.rados.rados.0.smithi076.stdout:3865: read oid 8 snap -1 2021-03-15T22:36:02.367 INFO:tasks.rados.rados.0.smithi076.stdout:3865: expect (ObjNum 1203 snap 418 seq_num 1203) 2021-03-15T22:36:02.368 INFO:tasks.rados.rados.0.smithi076.stdout:3866: read oid 32 snap -1 2021-03-15T22:36:02.368 INFO:tasks.rados.rados.0.smithi076.stdout:3866: expect (ObjNum 1221 snap 423 seq_num 1221) 2021-03-15T22:36:02.368 INFO:tasks.rados.rados.0.smithi076.stdout:3867: write oid 1 current snap is 459 2021-03-15T22:36:02.369 INFO:tasks.rados.rados.0.smithi076.stdout:3858: finishing write tid 3 to smithi07694878-18 2021-03-15T22:36:02.369 INFO:tasks.rados.rados.0.smithi076.stdout:3858: finishing write tid 4 to smithi07694878-18 2021-03-15T22:36:02.369 INFO:tasks.rados.rados.0.smithi076.stdout:3858: finishing write tid 5 to smithi07694878-18 2021-03-15T22:36:02.369 INFO:tasks.rados.rados.0.smithi076.stdout:3858: finishing write tid 6 to smithi07694878-18 2021-03-15T22:36:02.369 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 18 v 2455 (ObjNum 1308 snap 459 seq_num 1308) dirty exists 2021-03-15T22:36:02.370 INFO:tasks.rados.rados.0.smithi076.stdout:3858: left oid 18 (ObjNum 1308 snap 459 seq_num 1308) 2021-03-15T22:36:02.370 INFO:tasks.rados.rados.0.smithi076.stdout:3859: finishing write tid 1 to smithi07694878-41 2021-03-15T22:36:02.370 INFO:tasks.rados.rados.0.smithi076.stdout:3860: expect (ObjNum 1270 snap 437 seq_num 1270) 2021-03-15T22:36:02.371 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 5 pgp_num to 32 2021-03-15T22:36:02.408 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:02.430 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:02.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:02.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:02.477 INFO:tasks.rados.rados.0.smithi076.stdout:3859: finishing write tid 2 to smithi07694878-41 2021-03-15T22:36:02.478 INFO:tasks.rados.rados.0.smithi076.stdout:3859: finishing write tid 3 to smithi07694878-41 2021-03-15T22:36:02.478 INFO:tasks.rados.rados.0.smithi076.stdout:3859: finishing write tid 4 to smithi07694878-41 2021-03-15T22:36:02.478 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 3169 (ObjNum 1309 snap 459 seq_num 1309) dirty exists 2021-03-15T22:36:02.479 INFO:tasks.rados.rados.0.smithi076.stdout:3859: left oid 41 (ObjNum 1309 snap 459 seq_num 1309) 2021-03-15T22:36:02.479 INFO:tasks.rados.rados.0.smithi076.stdout:3861: finishing write tid 1 to smithi07694878-27 2021-03-15T22:36:02.479 INFO:tasks.rados.rados.0.smithi076.stdout:3861: finishing write tid 2 to smithi07694878-27 2021-03-15T22:36:02.479 INFO:tasks.rados.rados.0.smithi076.stdout:3861: finishing write tid 3 to smithi07694878-27 2021-03-15T22:36:02.480 INFO:tasks.rados.rados.0.smithi076.stdout:3861: finishing write tid 4 to smithi07694878-27 2021-03-15T22:36:02.480 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 3159 (ObjNum 1310 snap 459 seq_num 1310) dirty exists 2021-03-15T22:36:02.480 INFO:tasks.rados.rados.0.smithi076.stdout:3861: left oid 27 (ObjNum 1310 snap 459 seq_num 1310) 2021-03-15T22:36:02.480 INFO:tasks.rados.rados.0.smithi076.stdout:3867: seq_num 1311 ranges {433977=664155,1733305=300672} 2021-03-15T22:36:02.483 INFO:tasks.rados.rados.0.smithi076.stdout:3867: writing smithi07694878-1 from 433977 to 1098132 tid 1 2021-03-15T22:36:02.488 INFO:tasks.rados.rados.0.smithi076.stdout:3867: writing smithi07694878-1 from 1733305 to 2033977 tid 2 2021-03-15T22:36:02.489 INFO:tasks.rados.rados.0.smithi076.stdout:3858: done (8 left) 2021-03-15T22:36:02.489 INFO:tasks.rados.rados.0.smithi076.stdout:3859: done (7 left) 2021-03-15T22:36:02.489 INFO:tasks.rados.rados.0.smithi076.stdout:3860: done (6 left) 2021-03-15T22:36:02.489 INFO:tasks.rados.rados.0.smithi076.stdout:3861: done (5 left) 2021-03-15T22:36:02.489 INFO:tasks.rados.rados.0.smithi076.stdout:3868: delete oid 46 current snap is 459 2021-03-15T22:36:02.490 INFO:tasks.rados.rados.0.smithi076.stdout:3865: expect (ObjNum 1203 snap 418 seq_num 1203) 2021-03-15T22:36:02.533 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:02.556 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:02.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:02.558 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:02.611 INFO:tasks.rados.rados.0.smithi076.stdout:3865: done (5 left) 2021-03-15T22:36:02.612 INFO:tasks.rados.rados.0.smithi076.stdout:3868: done (4 left) 2021-03-15T22:36:02.612 INFO:tasks.rados.rados.0.smithi076.stdout:3869: read oid 5 snap -1 2021-03-15T22:36:02.612 INFO:tasks.rados.rados.0.smithi076.stdout:3869: expect deleted 2021-03-15T22:36:02.613 INFO:tasks.rados.rados.0.smithi076.stdout:3870: delete oid 49 current snap is 459 2021-03-15T22:36:02.613 INFO:tasks.rados.rados.0.smithi076.stdout:3863: expect (ObjNum 1303 snap 451 seq_num 1303) 2021-03-15T22:36:02.655 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:36:02.658 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:02.720 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:02.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:02.722 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:02.823 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:02.848 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:02.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:02.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:02.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:36:02.925 INFO:tasks.rados.rados.0.smithi076.stdout:3864: expect (ObjNum 1284 snap 442 seq_num 1284) 2021-03-15T22:36:02.932 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[56514]: cluster 2021-03-15T22:36:01.228809+0000 mgr.y (mgr. 2021-03-15T22:36:02.932 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[56514]: 14144) 1109 : cluster [DBG] pgmap v1655: 152 pgs: 1 clean+premerge+peered, 151 active+clean; 732 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 784 KiB/s wr, 12 op/s 2021-03-15T22:36:02.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[56514]: audit 2021-03-15T22:36:01.885960+0000 mon.c (mon.1) 83 : audit [INF] from='client.? 172.21.15.76:0/994878946' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:36:02.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[56514]: audit 2021-03-15T22:36:01.886307+0000 mon.a (mon.0) 2510 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:36:02.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[56514]: audit 2021-03-15T22:36:02.362180+0000 mon.a (mon.0) 2511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:36:02.933 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[56514]: cluster 2021-03-15T22:36:02.362253+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e1007: 8 total, 8 up, 8 in 2021-03-15T22:36:02.934 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: cluster 2021-03-15T22:36: 2021-03-15T22:36:02.934 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: 01.228809+0000 mgr.y (mgr.14144) 1109 : cluster [DBG] pgmap v1655: 152 pgs: 1 clean+premerge+peered, 151 active+clean; 732 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 784 KiB/s wr, 12 op/s 2021-03-15T22:36:02.934 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:36:02.934 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: -15T22:36:01.885960+0000 mon.c (mon.1) 2021-03-15T22:36:02.935 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: 83 : audit [INF] from='client.? 172.21.15.76:0/994878946' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:36:02.935 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: audit 2021-03-15T22:36:02.935 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: 2021-03-15T22:36:01.886307+0000 mon.a 2021-03-15T22:36:02.935 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: (mon.0) 2510 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:36:02.935 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: audit 2021 2021-03-15T22:36:02.936 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: -03-15T22:36:02.362180+0000 2021-03-15T22:36:02.936 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: mon.a (mon.0) 2511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:36:02.936 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:36:02.936 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: -15T22:36:02. 2021-03-15T22:36:02.937 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: 362253+0000 mon.a (mon.0) 2512 : cluster [DBG] osdmap e1007: 8 total, 8 up, 8 in 2021-03-15T22:36:02.937 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:02 smithi076 conmon[62156]: 2021-03-15T22:36:02.951 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:02.973 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:02.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:02.974 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:03.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: cluster 2021-03-15T22:36:01.228809+0000 mgr.y (mgr.14144) 1109 : cluster 2021-03-15T22:36:03.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: [DBG] pgmap v1655: 152 pgs: 1 clean+premerge+peered, 151 active+clean; 732 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 9.6 MiB/s rd, 784 KiB/s wr, 12 op/s 2021-03-15T22:36:03.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: audit 2021-03-15T22:36:01.885960+0000 mon.c (mon.1) 83 : audit [INF] from='client.? 172.21.15.76:0/994878946' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:36:03.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: audit 2021-03-15T22:36:01.886307+0000 mon.a ( 2021-03-15T22:36:03.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: mon.0) 2510 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:36:03.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: audit 2021-03-15T22:36:03.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: 2021-03-15T22:36 2021-03-15T22:36:03.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: :02.362180+0000 mon.a (mon.0) 2021-03-15T22:36:03.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: 2511 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.control", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:36:03.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:36:03.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: -03-15T22:36: 2021-03-15T22:36:03.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: 02.362253+0000 mon.a (mon.0) 2021-03-15T22:36:03.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:02 smithi125 conmon[56739]: 2512 : cluster [DBG] osdmap e1007: 8 total, 8 up, 8 in 2021-03-15T22:36:03.076 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:03.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:03.100 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:03.100 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:03.134 INFO:tasks.rados.rados.0.smithi076.stdout:3866: expect (ObjNum 1221 snap 423 seq_num 1221) 2021-03-15T22:36:03.202 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:03.228 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:03.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:03.229 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:03.257 INFO:tasks.rados.rados.0.smithi076.stdout:3863: done (5 left) 2021-03-15T22:36:03.257 INFO:tasks.rados.rados.0.smithi076.stdout:3864: done (4 left) 2021-03-15T22:36:03.257 INFO:tasks.rados.rados.0.smithi076.stdout:3866: done (3 left) 2021-03-15T22:36:03.258 INFO:tasks.rados.rados.0.smithi076.stdout:3870: done (2 left) 2021-03-15T22:36:03.258 INFO:tasks.rados.rados.0.smithi076.stdout:3871: delete oid 7 current snap is 459 2021-03-15T22:36:03.258 INFO:tasks.rados.rados.0.smithi076.stdout:3867: finishing write tid 1 to smithi07694878-1 2021-03-15T22:36:03.258 INFO:tasks.rados.rados.0.smithi076.stdout:3867: finishing write tid 2 to smithi07694878-1 2021-03-15T22:36:03.259 INFO:tasks.rados.rados.0.smithi076.stdout:3867: finishing write tid 3 to smithi07694878-1 2021-03-15T22:36:03.259 INFO:tasks.rados.rados.0.smithi076.stdout:3867: finishing write tid 4 to smithi07694878-1 2021-03-15T22:36:03.259 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 3084 (ObjNum 1311 snap 459 seq_num 1311) dirty exists 2021-03-15T22:36:03.259 INFO:tasks.rados.rados.0.smithi076.stdout:3867: left oid 1 (ObjNum 1311 snap 459 seq_num 1311) 2021-03-15T22:36:03.260 INFO:tasks.rados.rados.0.smithi076.stdout:3867: done (2 left) 2021-03-15T22:36:03.261 INFO:tasks.rados.rados.0.smithi076.stdout:3869: done (1 left) 2021-03-15T22:36:03.261 INFO:tasks.rados.rados.0.smithi076.stdout:3871: done (0 left) 2021-03-15T22:36:03.261 INFO:tasks.rados.rados.0.smithi076.stdout:3872: snap_create 2021-03-15T22:36:03.330 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:03.352 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:03.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:03.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:03.367 INFO:tasks.rados.rados.0.smithi076.stdout:3872: done (0 left) 2021-03-15T22:36:03.367 INFO:tasks.rados.rados.0.smithi076.stdout:3873: snap_create 2021-03-15T22:36:03.398 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:03.425 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:03.455 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:03.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:03.482 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:03.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:03.584 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:03.608 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:03.608 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:03.710 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:03.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:03.734 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:03.734 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:03.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:03 smithi076 conmon[62156]: audit 2021-03-15T22:36:03.230054+0000 mon.a (mon.0) 2513 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-03-15T22:36:03.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:03 smithi076 conmon[62156]: audit 2021-03-15T22:36:03.365019+0000 mon.a (mon.0) 2514 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]': finished 2021-03-15T22:36:03.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:03 smithi076 conmon[62156]: cluster 2021-03-15T22:36:03.365106+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e1008: 8 total, 8 up, 8 in 2021-03-15T22:36:03.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:03 smithi076 conmon[56514]: audit 2021-03-15T22:36:03.230054+0000 mon.a (mon.0) 2513 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-03-15T22:36:03.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:03 smithi076 conmon[56514]: audit 2021-03-15T22:36:03.365019+0000 mon.a (mon.0) 2514 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]': finished 2021-03-15T22:36:03.808 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:03 smithi076 conmon[56514]: cluster 2021-03-15T22:36:03.365106+0000 mon.a (mon.0) 2515 : cluster [DBG] osdmap e1008: 8 total, 8 up, 8 in 2021-03-15T22:36:03.835 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:03.847 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:36:03.847 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:36:03.847 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:36:03.848 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:36:03.848 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:36:03.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:03.859 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:03.859 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:03.961 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:03.987 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:03.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:03.989 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:04.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:03 smithi125 conmon[56739]: audit 2021-03-15T22:36:03.230054+0000 mon.a (mon.0) 2513 : audit 2021-03-15T22:36:04.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:03 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]: dispatch 2021-03-15T22:36:04.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:03 smithi125 conmon[56739]: audit 2021-03-15T22:36:03.365019+0000 mon.a (mon.0) 2514 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "29"}]': finished 2021-03-15T22:36:04.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:03 smithi125 conmon[56739]: cluster 2021-03-15T22:36:03.365106+0000 mon.a (mon.0 2021-03-15T22:36:04.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:03 smithi125 conmon[56739]: ) 2515 : cluster [DBG] osdmap e1008: 8 total, 8 up, 8 in 2021-03-15T22:36:04.091 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:04.115 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:04.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:04.117 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:04.218 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:04.244 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:04.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:04.246 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:04.347 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:04.366 INFO:tasks.rados.rados.0.smithi076.stdout:3873: done (0 left) 2021-03-15T22:36:04.367 INFO:tasks.rados.rados.0.smithi076.stdout:3874: snap_remove snap 431 2021-03-15T22:36:04.371 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:04.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:04.373 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:04.474 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:04.500 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:04.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:04.501 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:04.603 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:04.627 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:04.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:04.629 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:04.730 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:04.752 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:04.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:04.754 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:04.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:04 smithi076 conmon[56514]: cluster 2021-03-15T22:36:03.229287+0000 mgr.y (mgr.14144) 1110 2021-03-15T22:36:04.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:04 smithi076 conmon[56514]: : cluster [DBG] pgmap v1658: 151 pgs: 1 clean+premerge+peered, 150 active+clean; 738 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 18 op/s; 872 B/s, 0 objects/s recovering 2021-03-15T22:36:04.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:04 smithi076 conmon[56514]: cluster 2021-03-15T22:36:04.363512+0000 mon.a (mon.0) 2516 : cluster [DBG] osdmap e1009: 8 total, 8 up, 8 in 2021-03-15T22:36:04.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:04 smithi076 conmon[62156]: cluster 2021-03-15T22:36:03.229287+0000 mgr.y (mgr.14144 2021-03-15T22:36:04.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:04 smithi076 conmon[62156]: ) 1110 : cluster [DBG] pgmap v1658: 151 pgs: 1 clean+premerge+peered, 150 active+clean; 738 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 18 op/s; 872 B/s, 0 objects/s recovering 2021-03-15T22:36:04.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:04 smithi076 conmon[62156]: cluster 2021-03-15T22:36:04.363512+0000 mon.a (mon.0) 2516 : cluster [DBG] 2021-03-15T22:36:04.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:04 smithi076 conmon[62156]: osdmap e1009: 8 total, 8 up, 8 in 2021-03-15T22:36:04.855 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:04.882 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:04.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:04.883 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:04.984 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:05.007 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:05.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:05.008 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:05.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:04 smithi125 conmon[56739]: cluster 2021-03-15T22:36:03.229287+0000 mgr.y (mgr.14144) 2021-03-15T22:36:05.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:04 smithi125 conmon[56739]: 1110 : cluster [DBG] pgmap v1658: 151 pgs: 1 clean+premerge+peered, 150 active+clean; 738 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 11 MiB/s rd, 1.5 MiB/s wr, 18 op/s; 872 B/s, 0 objects/s recovering 2021-03-15T22:36:05.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:04 smithi125 conmon[56739]: cluster 2021-03-15T22:36:04.363512+0000 mon.a (mon.0) 2516 : cluster [DBG] 2021-03-15T22:36:05.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:04 smithi125 conmon[56739]: osdmap e1009: 8 total, 8 up, 8 in 2021-03-15T22:36:05.109 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:05.135 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:05.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:05.137 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:05.238 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:05.263 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:05.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:05.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:05.351 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.2 dump_historic_ops 2021-03-15T22:36:05.366 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:05.374 INFO:tasks.rados.rados.0.smithi076.stdout:3874: done (0 left) 2021-03-15T22:36:05.374 INFO:tasks.rados.rados.0.smithi076.stdout:3875: write (excl) oid 19 current snap is 461 2021-03-15T22:36:05.375 INFO:tasks.rados.rados.0.smithi076.stdout:3875: seq_num 1315 ranges {501205=565151,1850880=250325} 2021-03-15T22:36:05.380 INFO:tasks.rados.rados.0.smithi076.stdout:3875: writing smithi07694878-19 from 501205 to 1066356 tid 1 2021-03-15T22:36:05.388 INFO:tasks.rados.rados.0.smithi076.stdout:3875: writing smithi07694878-19 from 1850880 to 2101205 tid 2 2021-03-15T22:36:05.388 INFO:tasks.rados.rados.0.smithi076.stdout:3876: write (excl) oid 43 current snap is 461 2021-03-15T22:36:05.389 INFO:tasks.rados.rados.0.smithi076.stdout:3876: seq_num 1316 ranges {518012=756753,2025715=92297} 2021-03-15T22:36:05.397 INFO:tasks.rados.rados.0.smithi076.stdout:3876: writing smithi07694878-43 from 518012 to 1274765 tid 1 2021-03-15T22:36:05.398 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:05.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:05.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:05.402 INFO:tasks.rados.rados.0.smithi076.stdout:3876: writing smithi07694878-43 from 2025715 to 2118012 tid 2 2021-03-15T22:36:05.402 INFO:tasks.rados.rados.0.smithi076.stdout:3877: snap_remove snap 401 2021-03-15T22:36:05.500 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:05.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:05.527 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:05.527 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:05.629 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:05.652 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:05.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:05.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:05.656 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:05 smithi076 conmon[56514]: audit 2021-03-15T22:36:05.230558+0000 mon.a (mon.0) 2517 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "35"}]: dispatch 2021-03-15T22:36:05.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:05 smithi076 conmon[56514]: audit 2021-03-15T22:36:05.370363+0000 mon.a (mon.0) 2518 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "35"}]': finished 2021-03-15T22:36:05.657 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:05 smithi076 conmon[56514]: cluster 2021-03-15T22:36:05.370408+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e1010: 8 total, 8 up, 8 in 2021-03-15T22:36:05.657 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:05 smithi076 conmon[62156]: audit 2021-03-15T22:36:05.230558 2021-03-15T22:36:05.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:05 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2517 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "35"}]: dispatch 2021-03-15T22:36:05.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:05 smithi076 conmon[62156]: audit 2021-03-15T22:36:05.370363+0000 mon.a (mon.0) 2518 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "35"}]': finished 2021-03-15T22:36:05.658 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:05 smithi076 conmon[62156]: cluster 2021-03-15T22:36:05.370408+0000 mon.a (mon.0) 2519 : cluster [DBG] osdmap e1010: 8 total, 8 up, 8 in 2021-03-15T22:36:05.755 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:05.779 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:05.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:05.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:05.882 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:05.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:05.906 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:05.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:06.008 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:06.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:06.034 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:06.035 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:06.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:05 smithi125 conmon[56739]: audit 2021-03-15T22:36:05.230558+0000 mon.a (mon.0) 2517 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "35"}]: dispatch 2021-03-15T22:36:06.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:05 smithi125 conmon[56739]: 2021-03-15T22:36:06.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:05 smithi125 conmon[56739]: audit 2021-03-15T22:36:05.370363+0000 mon.a (mon.0) 2518 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "35"}]': finished 2021-03-15T22:36:06.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:05 smithi125 conmon[56739]: cluster 2021-03-15T22:36:05.370408+0000 mon.a (mon 2021-03-15T22:36:06.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:05 smithi125 conmon[56739]: .0) 2519 : cluster [DBG] osdmap e1010: 8 total, 8 up, 8 in 2021-03-15T22:36:06.105 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:06.136 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:06.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:06.161 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:06.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:06.262 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:06.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:06.297 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:06.297 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:06.374 INFO:tasks.rados.rados.0.smithi076.stdout:3877: done (2 left) 2021-03-15T22:36:06.374 INFO:tasks.rados.rados.0.smithi076.stdout:3878: write oid 7 current snap is 461 2021-03-15T22:36:06.375 INFO:tasks.rados.rados.0.smithi076.stdout:3876: finishing write tid 1 to smithi07694878-43 2021-03-15T22:36:06.375 INFO:tasks.rados.rados.0.smithi076.stdout:3878: seq_num 1317 ranges {534819=432002,1684197=450622} 2021-03-15T22:36:06.380 INFO:tasks.rados.rados.0.smithi076.stdout:3878: writing smithi07694878-7 from 534819 to 966821 tid 1 2021-03-15T22:36:06.389 INFO:tasks.rados.rados.0.smithi076.stdout:3878: writing smithi07694878-7 from 1684197 to 2134819 tid 2 2021-03-15T22:36:06.390 INFO:tasks.rados.rados.0.smithi076.stdout:3879: read oid 36 snap -1 2021-03-15T22:36:06.390 INFO:tasks.rados.rados.0.smithi076.stdout:3879: expect deleted 2021-03-15T22:36:06.390 INFO:tasks.rados.rados.0.smithi076.stdout:3875: finishing write tid 1 to smithi07694878-19 2021-03-15T22:36:06.390 INFO:tasks.rados.rados.0.smithi076.stdout:3875: finishing write tid 2 to smithi07694878-19 2021-03-15T22:36:06.391 INFO:tasks.rados.rados.0.smithi076.stdout:3875: finishing write tid 3 to smithi07694878-19 2021-03-15T22:36:06.391 INFO:tasks.rados.rados.0.smithi076.stdout:3875: finishing write tid 4 to smithi07694878-19 2021-03-15T22:36:06.391 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2849 (ObjNum 1315 snap 461 seq_num 1315) dirty exists 2021-03-15T22:36:06.392 INFO:tasks.rados.rados.0.smithi076.stdout:3875: left oid 19 (ObjNum 1315 snap 461 seq_num 1315) 2021-03-15T22:36:06.392 INFO:tasks.rados.rados.0.smithi076.stdout:3876: finishing write tid 2 to smithi07694878-43 2021-03-15T22:36:06.392 INFO:tasks.rados.rados.0.smithi076.stdout:3876: finishing write tid 3 to smithi07694878-43 2021-03-15T22:36:06.393 INFO:tasks.rados.rados.0.smithi076.stdout:3876: finishing write tid 4 to smithi07694878-43 2021-03-15T22:36:06.393 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 2458 (ObjNum 1316 snap 461 seq_num 1316) dirty exists 2021-03-15T22:36:06.393 INFO:tasks.rados.rados.0.smithi076.stdout:3876: left oid 43 (ObjNum 1316 snap 461 seq_num 1316) 2021-03-15T22:36:06.394 INFO:tasks.rados.rados.0.smithi076.stdout:3875: done (3 left) 2021-03-15T22:36:06.394 INFO:tasks.rados.rados.0.smithi076.stdout:3876: done (2 left) 2021-03-15T22:36:06.394 INFO:tasks.rados.rados.0.smithi076.stdout:3880: rollback oid 10 current snap is 461 2021-03-15T22:36:06.395 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 10 to 416 2021-03-15T22:36:06.395 INFO:tasks.rados.rados.0.smithi076.stdout:3881: read oid 14 snap -1 2021-03-15T22:36:06.395 INFO:tasks.rados.rados.0.smithi076.stdout:3881: expect (ObjNum 1140 snap 393 seq_num 1140) 2021-03-15T22:36:06.396 INFO:tasks.rados.rados.0.smithi076.stdout:3882: read oid 18 snap -1 2021-03-15T22:36:06.396 INFO:tasks.rados.rados.0.smithi076.stdout:3882: expect (ObjNum 1308 snap 459 seq_num 1308) 2021-03-15T22:36:06.396 INFO:tasks.rados.rados.0.smithi076.stdout:3883: write oid 42 current snap is 461 2021-03-15T22:36:06.397 INFO:tasks.rados.rados.0.smithi076.stdout:3883: seq_num 1318 ranges {551626=507251,1626326=418456} 2021-03-15T22:36:06.397 INFO:tasks.rados.rados.0.smithi076.stdout:3883: writing smithi07694878-42 from 551626 to 1058877 tid 1 2021-03-15T22:36:06.398 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:06.401 INFO:tasks.rados.rados.0.smithi076.stdout:3883: writing smithi07694878-42 from 1626326 to 2044782 tid 2 2021-03-15T22:36:06.405 INFO:tasks.rados.rados.0.smithi076.stdout:3878: finishing write tid 1 to smithi07694878-7 2021-03-15T22:36:06.405 INFO:tasks.rados.rados.0.smithi076.stdout:3878: finishing write tid 2 to smithi07694878-7 2021-03-15T22:36:06.406 INFO:tasks.rados.rados.0.smithi076.stdout:3882: expect (ObjNum 1308 snap 459 seq_num 1308) 2021-03-15T22:36:06.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:06.430 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:06.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:06.525 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:36:06.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:06.542 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:36:06.557 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:06.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:06.559 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:06.560 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:36:06.578 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:36:06.596 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:36:06.615 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:36:06.635 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:36:06.638 INFO:tasks.rados.rados.0.smithi076.stdout:3878: finishing write tid 3 to smithi07694878-7 2021-03-15T22:36:06.638 INFO:tasks.rados.rados.0.smithi076.stdout:3878: finishing write tid 4 to smithi07694878-7 2021-03-15T22:36:06.638 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 7 v 2852 (ObjNum 1317 snap 461 seq_num 1317) dirty exists 2021-03-15T22:36:06.639 INFO:tasks.rados.rados.0.smithi076.stdout:3878: left oid 7 (ObjNum 1317 snap 461 seq_num 1317) 2021-03-15T22:36:06.639 INFO:tasks.rados.rados.0.smithi076.stdout:3878: done (5 left) 2021-03-15T22:36:06.639 INFO:tasks.rados.rados.0.smithi076.stdout:3882: done (4 left) 2021-03-15T22:36:06.639 INFO:tasks.rados.rados.0.smithi076.stdout:3884: read oid 22 snap -1 2021-03-15T22:36:06.640 INFO:tasks.rados.rados.0.smithi076.stdout:3884: expect (ObjNum 1225 snap 424 seq_num 1225) 2021-03-15T22:36:06.640 INFO:tasks.rados.rados.0.smithi076.stdout:3885: read oid 30 snap -1 2021-03-15T22:36:06.640 INFO:tasks.rados.rados.0.smithi076.stdout:3885: expect (ObjNum 1282 snap 442 seq_num 1282) 2021-03-15T22:36:06.640 INFO:tasks.rados.rados.0.smithi076.stdout:3886: read oid 8 snap -1 2021-03-15T22:36:06.640 INFO:tasks.rados.rados.0.smithi076.stdout:3886: expect (ObjNum 1203 snap 418 seq_num 1203) 2021-03-15T22:36:06.641 INFO:tasks.rados.rados.0.smithi076.stdout:3887: read oid 33 snap 455 2021-03-15T22:36:06.641 INFO:tasks.rados.rados.0.smithi076.stdout:3887: expect (ObjNum 1288 snap 444 seq_num 1288) 2021-03-15T22:36:06.641 INFO:tasks.rados.rados.0.smithi076.stdout:3888: rollback oid 1 current snap is 461 2021-03-15T22:36:06.641 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 1 to 440 2021-03-15T22:36:06.642 INFO:tasks.rados.rados.0.smithi076.stdout:3880: finishing rollback tid 0 to smithi07694878-10 2021-03-15T22:36:06.642 INFO:tasks.rados.rados.0.smithi076.stdout:3880: finishing rollback tid 1 to smithi07694878-10 2021-03-15T22:36:06.642 INFO:tasks.rados.rados.0.smithi076.stdout:3880: finishing rollback tid 2 to smithi07694878-10 2021-03-15T22:36:06.643 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 3178 (ObjNum 1183 snap 414 seq_num 1183) dirty exists 2021-03-15T22:36:06.643 INFO:tasks.rados.rados.0.smithi076.stdout:3881: expect (ObjNum 1140 snap 393 seq_num 1140) 2021-03-15T22:36:06.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[62156]: cluster 2021-03-15T22:36:05 2021-03-15T22:36:06.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[62156]: .114003+0000 osd.4 (osd.4) 199 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:06.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[62156]: cluster 2021-03-15T22:36:05.116065+0000 osd.4 (osd.4) 200 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:06.659 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[62156]: cluster 2021-03-15T22:36:05.229871+0000 mgr.y (mgr.14144) 1111 : cluster [DBG] pgmap v1661: 150 pgs: 3 peering, 147 active+clean; 743 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2021-03-15T22:36:06.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[62156]: cluster 2021-03-15T22:36:06.368973+0000 mon.a (mon.0) 2520 : cluster [DBG] osdmap e1011: 8 total, 8 up, 8 in 2021-03-15T22:36:06.660 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[62156]: audit 2021-03-15T22:36:06.505318+0000 mon.a (mon.0) 2521 : audit [DBG] from='client.? 172.21.15.76:0/2985021091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:06.661 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:06.662 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:36:06.666 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:36:06.666 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:36:06.666 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:36:06.667 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:36:06.667 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:36:06.667 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:36:06.667 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:36:06.667 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:36:06.684 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:06.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:06.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:06.787 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:06.812 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:06.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:06.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:06.842 INFO:tasks.rados.rados.0.smithi076.stdout:3883: finishing write tid 1 to smithi07694878-42 2021-03-15T22:36:06.842 INFO:tasks.rados.rados.0.smithi076.stdout:3883: finishing write tid 2 to smithi07694878-42 2021-03-15T22:36:06.843 INFO:tasks.rados.rados.0.smithi076.stdout:3883: finishing write tid 3 to smithi07694878-42 2021-03-15T22:36:06.843 INFO:tasks.rados.rados.0.smithi076.stdout:3883: finishing write tid 4 to smithi07694878-42 2021-03-15T22:36:06.843 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 42 v 3396 (ObjNum 1318 snap 461 seq_num 1318) dirty exists 2021-03-15T22:36:06.843 INFO:tasks.rados.rados.0.smithi076.stdout:3883: left oid 42 (ObjNum 1318 snap 461 seq_num 1318) 2021-03-15T22:36:06.844 INFO:tasks.rados.rados.0.smithi076.stdout:3879: done (8 left) 2021-03-15T22:36:06.844 INFO:tasks.rados.rados.0.smithi076.stdout:3880: done (7 left) 2021-03-15T22:36:06.844 INFO:tasks.rados.rados.0.smithi076.stdout:3881: done (6 left) 2021-03-15T22:36:06.844 INFO:tasks.rados.rados.0.smithi076.stdout:3883: done (5 left) 2021-03-15T22:36:06.844 INFO:tasks.rados.rados.0.smithi076.stdout:3889: delete oid 27 current snap is 461 2021-03-15T22:36:06.845 INFO:tasks.rados.rados.0.smithi076.stdout:3888: finishing rollback tid 0 to smithi07694878-1 2021-03-15T22:36:06.845 INFO:tasks.rados.rados.0.smithi076.stdout:3884: expect (ObjNum 1225 snap 424 seq_num 1225) 2021-03-15T22:36:06.915 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:06.939 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:06.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:06.940 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:07.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[56514]: cluster 2021-03-15T22:36:05 2021-03-15T22:36:07.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[56514]: .114003+0000 osd.4 (osd.4) 199 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:07.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[56514]: cluster 2021-03-15T22:36:05.116065+0000 osd.4 (osd.4) 200 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:07.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[56514]: cluster 2021-03-15T22:36:05.229871+0000 mgr.y (mgr.14144) 1111 : cluster [DBG] pgmap v1661: 150 pgs: 3 peering, 147 active+clean; 743 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2021-03-15T22:36:07.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[56514]: cluster 2021-03-15T22:36:06.368973+0000 mon.a (mon.0) 2520 : cluster [DBG] osdmap e1011: 8 total, 8 up, 8 in 2021-03-15T22:36:07.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:06 smithi076 conmon[56514]: audit 2021-03-15T22:36:06.505318+0000 mon.a (mon.0) 2521 : audit [DBG] from='client.? 172.21.15.76:0/2985021091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:07.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:06 smithi125 conmon[56739]: cluster 2021-03-15T22:36:05.114003+0000 osd.4 (osd.4) 2021-03-15T22:36:07.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:06 smithi125 conmon[56739]: 199 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:07.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:06 smithi125 conmon[56739]: cluster 2021-03-15T22:36:05.116065+0000 osd.4 (osd.4) 200 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:07.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:06 smithi125 conmon[56739]: cluster 2021-03-15T22:36:07.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:06 smithi125 conmon[56739]: 2021-03-15T22:36:05.229871+0000 mgr.y (mgr.14144) 1111 : cluster [DBG] pgmap v1661: 150 pgs: 3 peering, 147 active+clean; 743 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 7.6 MiB/s rd, 2.0 MiB/s wr, 15 op/s 2021-03-15T22:36:07.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:06 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:36:07.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:06 smithi125 conmon[56739]: T22:36:06.368973+0000 mon.a (mon.0) 2520 : cluster 2021-03-15T22:36:07.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:06 smithi125 conmon[56739]: [DBG] osdmap e1011: 8 total, 8 up, 8 in 2021-03-15T22:36:07.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:06 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:36:07.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:06 smithi125 conmon[56739]: 03-15T22:36:06.505318+0000 mon.a 2021-03-15T22:36:07.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:06 smithi125 conmon[56739]: (mon.0) 2521 : audit [DBG] from='client.? 172.21.15.76:0/2985021091' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:07.042 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:07.064 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:07.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:07.065 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:07.089 INFO:tasks.rados.rados.0.smithi076.stdout:3886: expect (ObjNum 1203 snap 418 seq_num 1203) 2021-03-15T22:36:07.167 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:07.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:07.189 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:07.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:07.208 INFO:tasks.rados.rados.0.smithi076.stdout:3887: expect (ObjNum 1288 snap 444 seq_num 1288) 2021-03-15T22:36:07.290 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:07.317 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:07.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:07.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:07.419 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:07.445 INFO:tasks.rados.rados.0.smithi076.stdout:3888: finishing rollback tid 1 to smithi07694878-1 2021-03-15T22:36:07.445 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 1 v 3086 (ObjNum 262269168 snap 21896 seq_num 262300688) dirty dne 2021-03-15T22:36:07.445 INFO:tasks.rados.rados.0.smithi076.stdout:3885: expect (ObjNum 1282 snap 442 seq_num 1282) 2021-03-15T22:36:07.446 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:07.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:07.447 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:07.548 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:07.566 INFO:tasks.rados.rados.0.smithi076.stdout:3884: done (5 left) 2021-03-15T22:36:07.566 INFO:tasks.rados.rados.0.smithi076.stdout:3885: done (4 left) 2021-03-15T22:36:07.566 INFO:tasks.rados.rados.0.smithi076.stdout:3886: done (3 left) 2021-03-15T22:36:07.566 INFO:tasks.rados.rados.0.smithi076.stdout:3887: done (2 left) 2021-03-15T22:36:07.567 INFO:tasks.rados.rados.0.smithi076.stdout:3888: done (1 left) 2021-03-15T22:36:07.567 INFO:tasks.rados.rados.0.smithi076.stdout:3889: done (0 left) 2021-03-15T22:36:07.567 INFO:tasks.rados.rados.0.smithi076.stdout:3890: snap_remove snap 407 2021-03-15T22:36:07.573 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:07.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:07.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:07.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:07.699 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:07.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:07.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:07.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:07 smithi076 conmon[62156]: audit 2021-03-15T22:36:07.231651+0000 mon.a (mon.0) 2522 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-03-15T22:36:07.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:07 smithi076 conmon[62156]: audit 2021-03-15T22:36:07.372320+0000 mon.a (mon.0) 2523 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]': finished 2021-03-15T22:36:07.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:07 smithi076 conmon[62156]: cluster 2021-03-15T22:36:07.372381+0000 mon.a (mon.0) 2524 : cluster [DBG] osdmap e1012: 8 total, 8 up, 8 in 2021-03-15T22:36:07.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:07 smithi076 conmon[62156]: audit 2021-03-15T22:36:07.506930+0000 mon.c (mon.1) 84 : audit [INF] from='client.? 172.21.15.76:0/3193757794' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:07.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:07 smithi076 conmon[62156]: audit 2021-03-15T22:36:07.507226+0000 mon.a (mon.0) 2525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:07.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:07 smithi076 conmon[56514]: audit 2021-03-15T22:36:07.231651+0000 mon.a (mon.0) 2522 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-03-15T22:36:07.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:07 smithi076 conmon[56514]: audit 2021-03-15T22:36:07.372320+0000 mon.a (mon.0) 2523 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]': finished 2021-03-15T22:36:07.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:07 smithi076 conmon[56514]: cluster 2021-03-15T22:36:07.372381+0000 mon.a (mon.0) 2524 : cluster [DBG] osdmap e1012: 8 total, 8 up, 8 in 2021-03-15T22:36:07.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:07 smithi076 conmon[56514]: audit 2021-03-15T22:36:07.506930+0000 mon.c (mon.1) 84 : audit [INF] from='client.? 172.21.15.76:0/3193757794' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:07.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:07 smithi076 conmon[56514]: audit 2021-03-15T22:36:07.507226+0000 mon.a (mon.0) 2525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:07.802 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:07.823 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:07.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:07.926 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:07.951 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:07.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:07.953 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:08.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:07 smithi125 conmon[56739]: audit 2021-03-15T22:36:07.231651+0000 mon.a (mon.0) 2522 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]: dispatch 2021-03-15T22:36:08.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:07 smithi125 conmon[56739]: audit 2021-03-15T22:36:07.372320+0000 mon.a (mon 2021-03-15T22:36:08.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:07 smithi125 conmon[56739]: .0) 2523 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "28"}]': finished 2021-03-15T22:36:08.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:07 smithi125 conmon[56739]: cluster 2021-03-15T22:36:07.372381+0000 mon.a (mon.0) 2524 : cluster [DBG] osdmap e1012: 8 total, 8 up, 8 in 2021-03-15T22:36:08.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:07 smithi125 conmon[56739]: audit 2021-03-15T22:36:07. 2021-03-15T22:36:08.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:07 smithi125 conmon[56739]: 506930+0000 mon.c (mon.1) 84 : audit [INF] from='client.? 172.21.15.76:0/3193757794' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:08.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:07 smithi125 conmon[56739]: audit 2021-03-15T22:36 2021-03-15T22:36:08.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:07 smithi125 conmon[56739]: :07.507226+0000 mon.a (mon.0) 2021-03-15T22:36:08.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:07 smithi125 conmon[56739]: 2525 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:08.054 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:08.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:08.077 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:08.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:08.178 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:08.204 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:08.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:08.206 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:08.307 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:08.333 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:08.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:08.334 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:08.379 INFO:tasks.rados.rados.0.smithi076.stdout:3890: done (0 left) 2021-03-15T22:36:08.379 INFO:tasks.rados.rados.0.smithi076.stdout:3891: read oid 17 snap 439 2021-03-15T22:36:08.379 INFO:tasks.rados.rados.0.smithi076.stdout:3891: expect (ObjNum 1265 snap 437 seq_num 1265) 2021-03-15T22:36:08.379 INFO:tasks.rados.rados.0.smithi076.stdout:3892: snap_create 2021-03-15T22:36:08.396 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:36:08.396 INFO:tasks.rados.rados.0.smithi076.stdout:3891: expect (ObjNum 1265 snap 437 seq_num 1265) 2021-03-15T22:36:08.420 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 7, 0, 3, 5, 6, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:36:08.420 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:36:08.421 INFO:tasks.thrashosds.thrasher:Growing pool device_health_metrics 2021-03-15T22:36:08.421 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:36:08.435 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:08.491 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:08.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:08.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:08.594 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:08.616 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:08.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:08.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:08.719 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:08.741 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:08.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:08.742 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:08.843 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:08.870 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:08.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:08.872 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:08.974 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:08.999 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:09.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:09.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:09.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:08 smithi076 conmon[56514]: cluster 2021-03-15T22:36:07.230412+0000 mgr.y ( 2021-03-15T22:36:09.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:08 smithi076 conmon[56514]: mgr.14144) 1112 : cluster [DBG] pgmap v1664: 149 pgs: 2 peering, 147 active+clean; 747 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2021-03-15T22:36:09.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:08 smithi076 conmon[56514]: cluster 2021-03-15T22:36:08.367464+0000 mon.a (mon.0) 2526 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:09.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:08 smithi076 conmon[56514]: audit 2021-03-15T22:36:08.374289+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:36:09.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:08 smithi076 conmon[56514]: cluster 2021-03-15T22:36:08.374427+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e1013: 8 total, 8 up, 8 in 2021-03-15T22:36:09.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:08 smithi076 conmon[62156]: cluster 2021-03-15T22:36:07.230412+0000 2021-03-15T22:36:09.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:08 smithi076 conmon[62156]: mgr.y (mgr.14144) 1112 : cluster [DBG] pgmap v1664: 149 pgs: 2 peering, 147 active+clean; 747 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2021-03-15T22:36:09.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:08 smithi076 conmon[62156]: cluster 2021-03-15T22:36:08.367464+0000 mon.a (mon.0) 2526 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:09.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:08 smithi076 conmon[62156]: audit 2021-03-15T22:36:08.374289+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:36:09.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:08 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:36:09.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:08 smithi076 conmon[62156]: 22:36:08.374427+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e1013: 8 total, 8 up, 8 in 2021-03-15T22:36:09.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:08 smithi125 conmon[56739]: cluster 2021-03-15T22:36:07.230412+0000 mgr.y (mgr.14144) 1112 : cluster [DBG] pgmap v1664: 149 pgs: 2 peering, 147 active+clean; 747 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 8.7 MiB/s rd, 1.9 MiB/s wr, 17 op/s 2021-03-15T22:36:09.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:08 smithi125 conmon[56739]: cluster 2021-03-15T22:36:08.367464+0000 mon.a (mon.0) 2526 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:09.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:08 smithi125 conmon[56739]: audit 2021-03-15T22:36:08.374289+0000 mon.a (mon.0) 2527 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:36:09.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:08 smithi125 conmon[56739]: 2021-03-15T22:36:09.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:08 smithi125 conmon[56739]: cluster 2021-03-15T22:36 2021-03-15T22:36:09.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:08 smithi125 conmon[56739]: :08.374427+0000 mon.a (mon.0) 2528 : cluster [DBG] osdmap e1013: 8 total, 8 up, 8 in 2021-03-15T22:36:09.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:08 smithi125 conmon[56739]: 2021-03-15T22:36:09.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:09.127 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:09.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:09.129 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:09.207 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:09.230 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:09.253 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:09.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:09.255 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:09.351 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:36:09.351 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:36:09.356 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:09.380 INFO:tasks.rados.rados.0.smithi076.stdout:3891: done (1 left) 2021-03-15T22:36:09.381 INFO:tasks.rados.rados.0.smithi076.stdout:3892: done (0 left) 2021-03-15T22:36:09.381 INFO:tasks.rados.rados.0.smithi076.stdout:3893: delete oid 47 current snap is 462 2021-03-15T22:36:09.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:09.383 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:09.383 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:09.384 INFO:tasks.rados.rados.0.smithi076.stdout:3893: done (0 left) 2021-03-15T22:36:09.386 INFO:tasks.rados.rados.0.smithi076.stdout:3894: copy_from oid 22 from oid 24 current snap is 462 2021-03-15T22:36:09.386 INFO:tasks.rados.rados.0.smithi076.stdout:3895: read oid 4 snap -1 2021-03-15T22:36:09.387 INFO:tasks.rados.rados.0.smithi076.stdout:3895: expect deleted 2021-03-15T22:36:09.387 INFO:tasks.rados.rados.0.smithi076.stdout:3896: read oid 43 snap -1 2021-03-15T22:36:09.387 INFO:tasks.rados.rados.0.smithi076.stdout:3896: expect (ObjNum 1316 snap 461 seq_num 1316) 2021-03-15T22:36:09.387 INFO:tasks.rados.rados.0.smithi076.stdout:3897: delete oid 2 current snap is 462 2021-03-15T22:36:09.388 INFO:tasks.rados.rados.0.smithi076.stdout:3894: finishing copy_from to smithi07694878-22 2021-03-15T22:36:09.389 INFO:tasks.rados.rados.0.smithi076.stdout:3894: got expected ENOENT (src dne) 2021-03-15T22:36:09.389 INFO:tasks.rados.rados.0.smithi076.stdout:3894: finishing copy_from racing read to smithi07694878-22 2021-03-15T22:36:09.389 INFO:tasks.rados.rados.0.smithi076.stdout:3894: done (3 left) 2021-03-15T22:36:09.389 INFO:tasks.rados.rados.0.smithi076.stdout:3895: done (2 left) 2021-03-15T22:36:09.390 INFO:tasks.rados.rados.0.smithi076.stdout:3897: done (1 left) 2021-03-15T22:36:09.390 INFO:tasks.rados.rados.0.smithi076.stdout:3898: write oid 2 current snap is 462 2021-03-15T22:36:09.390 INFO:tasks.rados.rados.0.smithi076.stdout:3898: seq_num 1322 ranges {618854=408247,1743901=425794} 2021-03-15T22:36:09.393 INFO:tasks.rados.rados.0.smithi076.stdout:3898: writing smithi07694878-2 from 618854 to 1027101 tid 1 2021-03-15T22:36:09.400 INFO:tasks.rados.rados.0.smithi076.stdout:3898: writing smithi07694878-2 from 1743901 to 2169695 tid 2 2021-03-15T22:36:09.400 INFO:tasks.rados.rados.0.smithi076.stdout:3899: read oid 44 snap -1 2021-03-15T22:36:09.401 INFO:tasks.rados.rados.0.smithi076.stdout:3899: expect (ObjNum 1177 snap 411 seq_num 1177) 2021-03-15T22:36:09.401 INFO:tasks.rados.rados.0.smithi076.stdout:3900: read oid 48 snap -1 2021-03-15T22:36:09.401 INFO:tasks.rados.rados.0.smithi076.stdout:3900: expect (ObjNum 1221 snap 423 seq_num 1221) 2021-03-15T22:36:09.401 INFO:tasks.rados.rados.0.smithi076.stdout:3901: snap_remove snap 449 2021-03-15T22:36:09.484 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:09.506 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:09.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:09.508 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:09.539 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:36:09.608 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:09.635 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:09.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:09.637 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:09.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[62156]: cluster 2021-03-15T22:36:08.386201+0000 osd.4 (osd.4) 201 : cluster [DBG] 5.18 scrub ok 2021-03-15T22:36:09.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[62156]: cluster 2021-03-15T22:36:08.685694+0000 osd.6 (osd.6) 152 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:36:09.675 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[62156]: cluster 2021-03-15T22:36:08.831554+0000 osd.3 (osd.3) 175 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:36:09.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[62156]: audit 2021-03-15T22:36:09.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[62156]: 2021-03-15T22:36:09.231798+0000 mon.a (mon.0) 2529 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "34"}]: dispatch 2021-03-15T22:36:09.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[62156]: audit 2021-03-15T22:36:09.375472+0000 mon.a (mon.0) 2530 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "34"}]': finished 2021-03-15T22:36:09.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[62156]: cluster 2021-03-15T22:36:09.375820+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e1014: 8 total, 8 up, 8 in 2021-03-15T22:36:09.738 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:09.760 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:09.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:09.762 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:09.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:09.889 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:09.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:09.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:09.991 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:10.016 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:10.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[56514]: cluster 2021-03-15T22:36:08.386201+0000 osd.4 (osd.4) 201 : cluster [DBG] 5.18 scrub ok 2021-03-15T22:36:10.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[56514]: cluster 2021-03-15T22:36:08.685694+0000 osd.6 (osd.6) 152 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:36:10.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[56514]: cluster 2021-03-15T22:36:08.831554+0000 osd.3 (osd.3) 175 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:36:10.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[56514]: audit 2021-03-15T22:36:09.231798+0000 mon.a (mon.0) 2529 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "34"}]: dispatch 2021-03-15T22:36:10.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[56514]: audit 2021-03-15T22:36:09.375472+0000 mon.a (mon.0) 2530 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "34"}]': finished 2021-03-15T22:36:10.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:09 smithi076 conmon[56514]: cluster 2021-03-15T22:36:09.375820+0000 mon.a (mon.0) 2531 : cluster [DBG] osdmap e1014: 8 total, 8 up, 8 in 2021-03-15T22:36:10.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:10.020 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:10.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: cluster 2021-03-15T22:36:08.386201+0000 osd.4 (osd.4) 201 : cluster [DBG] 5.18 scrub ok 2021-03-15T22:36:10.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: cluster 2021-03-15T22:36:08.685694+0000 2021-03-15T22:36:10.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: osd.6 (osd.6) 152 : cluster [DBG] 3.1a scrub ok 2021-03-15T22:36:10.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: cluster 2021-03-15T22:36:08.831554+0000 osd.3 (osd.3) 175 : cluster [DBG] 4.19 scrub ok 2021-03-15T22:36:10.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:36:10.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: :36:09.231798+0000 mon.a (mon.0) 2529 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "34"}]: dispatch 2021-03-15T22:36:10.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:36:10.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: T22:36:09.375472+0000 mon.a (mon 2021-03-15T22:36:10.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: .0) 2530 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "34"}]': finished 2021-03-15T22:36:10.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:36:10.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: -03-15T22:36:09.375820+0000 2021-03-15T22:36:10.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: mon.a (mon.0) 2531 : cluster [DBG] 2021-03-15T22:36:10.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:09 smithi125 conmon[56739]: osdmap e1014: 8 total, 8 up, 8 in 2021-03-15T22:36:10.121 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:10.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:10.147 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:10.148 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:10.249 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:10.266 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_ops_in_flight 2021-03-15T22:36:10.275 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:10.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:10.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:10.353 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:10.376 INFO:tasks.rados.rados.0.smithi076.stdout:3901: done (4 left) 2021-03-15T22:36:10.376 INFO:tasks.rados.rados.0.smithi076.stdout:3902: write (excl) oid 19 current snap is 462 2021-03-15T22:36:10.376 INFO:tasks.rados.rados.0.smithi076.stdout:3902: seq_num 1323 ranges {635661=599849,1918736=316925} 2021-03-15T22:36:10.377 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:10.389 INFO:tasks.rados.rados.0.smithi076.stdout:3902: writing smithi07694878-19 from 635661 to 1235510 tid 1 2021-03-15T22:36:10.396 INFO:tasks.rados.rados.0.smithi076.stdout:3902: writing smithi07694878-19 from 1918736 to 2235661 tid 2 2021-03-15T22:36:10.396 INFO:tasks.rados.rados.0.smithi076.stdout:3896: expect (ObjNum 1316 snap 461 seq_num 1316) 2021-03-15T22:36:10.403 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:10.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:10.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:10.504 INFO:tasks.rados.rados.0.smithi076.stdout:3896: done (4 left) 2021-03-15T22:36:10.504 INFO:tasks.rados.rados.0.smithi076.stdout:3903: write (excl) oid 39 current snap is 462 2021-03-15T22:36:10.505 INFO:tasks.rados.rados.0.smithi076.stdout:3898: finishing write tid 1 to smithi07694878-2 2021-03-15T22:36:10.505 INFO:tasks.rados.rados.0.smithi076.stdout:3903: seq_num 1324 ranges {652468=675098,1977218=275250} 2021-03-15T22:36:10.506 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:10.511 INFO:tasks.rados.rados.0.smithi076.stdout:3903: writing smithi07694878-39 from 652468 to 1327566 tid 1 2021-03-15T22:36:10.516 INFO:tasks.rados.rados.0.smithi076.stdout:3903: writing smithi07694878-39 from 1977218 to 2252468 tid 2 2021-03-15T22:36:10.516 INFO:tasks.rados.rados.0.smithi076.stdout:3904: snap_remove snap 453 2021-03-15T22:36:10.530 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:10.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:10.531 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:10.632 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:10.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:10.658 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:10.658 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:10.675 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:36:10 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:36:10.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:10.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:10.783 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:10.783 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:10.884 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:10.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:10.911 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:10.911 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:11.012 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:11.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:10 smithi076 conmon[56514]: cluster 2021-03-15T22 2021-03-15T22:36:11.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:10 smithi076 conmon[56514]: :36:09.230872+0000 mgr.y (mgr.14144) 1113 : cluster [DBG] pgmap v1667: 148 pgs: 2 peering, 146 active+clean; 748 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.4 MiB/s rd, 909 KiB/s wr, 12 op/s 2021-03-15T22:36:11.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:10 smithi076 conmon[56514]: cluster 2021-03-15T22:36:09.696241+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.d scrub ok 2021-03-15T22:36:11.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:10 smithi076 conmon[56514]: cluster 2021-03-15T22:36:10.354294+0000 mon.a (mon.0) 2532 : cluster [DBG] osdmap e1015: 8 total, 8 up, 8 in 2021-03-15T22:36:11.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:10 smithi076 conmon[62156]: cluster 2021-03-15T22:36:09 2021-03-15T22:36:11.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:10 smithi076 conmon[62156]: .230872+0000 mgr.y (mgr.14144) 1113 : cluster [DBG] pgmap v1667: 148 pgs: 2 peering, 146 active+clean; 748 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.4 MiB/s rd, 909 KiB/s wr, 12 op/s 2021-03-15T22:36:11.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:10 smithi076 conmon[62156]: cluster 2021-03-15T22:36:09.696241+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.d scrub ok 2021-03-15T22:36:11.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:10 smithi076 conmon[62156]: cluster 2021-03-15T22:36:10.354294+0000 mon.a (mon.0) 2532 : cluster [DBG] osdmap e1015: 8 total, 8 up, 8 in 2021-03-15T22:36:11.035 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:11.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:11.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:11.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:10 smithi125 conmon[56739]: cluster 2021-03-15T22:36:09.230872+0000 mgr.y (mgr.14144) 1113 : cluster [DBG] pgmap v1667: 148 pgs: 2 peering, 146 active+clean; 748 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 7.4 MiB/s rd, 909 KiB/s wr, 12 op/s 2021-03-15T22:36:11.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:10 smithi125 conmon[56739]: cluster 2021-03-15T22:36:09.696241+0000 osd.2 (osd.2) 51 : cluster [DBG] 5.d scrub ok 2021-03-15T22:36:11.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:10 smithi125 conmon[56739]: cluster 2021-03-15T22:36:10.354294+0000 mon.a (mon.0) 2532 : cluster 2021-03-15T22:36:11.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:10 smithi125 conmon[56739]: [DBG] osdmap e1015: 8 total, 8 up, 8 in 2021-03-15T22:36:11.138 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:11.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:11.167 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:11.168 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:11.189 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:11.225 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:36:11.270 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:11.292 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:11.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:11.294 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:11.374 INFO:tasks.rados.rados.0.smithi076.stdout:3904: done (5 left) 2021-03-15T22:36:11.375 INFO:tasks.rados.rados.0.smithi076.stdout:3905: rollback oid 46 current snap is 462 2021-03-15T22:36:11.375 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 46 to 364 2021-03-15T22:36:11.376 INFO:tasks.rados.rados.0.smithi076.stdout:3899: expect (ObjNum 1177 snap 411 seq_num 1177) 2021-03-15T22:36:11.395 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:11.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:11.423 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:11.424 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:11.515 INFO:tasks.rados.rados.0.smithi076.stdout:3899: done (5 left) 2021-03-15T22:36:11.515 INFO:tasks.rados.rados.0.smithi076.stdout:3906: copy_from oid 4 from oid 49 current snap is 462 2021-03-15T22:36:11.515 INFO:tasks.rados.rados.0.smithi076.stdout:3900: expect (ObjNum 1221 snap 423 seq_num 1221) 2021-03-15T22:36:11.524 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:11.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:11.555 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:11.555 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:11.637 INFO:tasks.rados.rados.0.smithi076.stdout:3898: finishing write tid 2 to smithi07694878-2 2021-03-15T22:36:11.637 INFO:tasks.rados.rados.0.smithi076.stdout:3898: finishing write tid 3 to smithi07694878-2 2021-03-15T22:36:11.637 INFO:tasks.rados.rados.0.smithi076.stdout:3898: finishing write tid 4 to smithi07694878-2 2021-03-15T22:36:11.637 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 3182 (ObjNum 1322 snap 462 seq_num 1322) dirty exists 2021-03-15T22:36:11.638 INFO:tasks.rados.rados.0.smithi076.stdout:3898: left oid 2 (ObjNum 1322 snap 462 seq_num 1322) 2021-03-15T22:36:11.638 INFO:tasks.rados.rados.0.smithi076.stdout:3902: finishing write tid 1 to smithi07694878-19 2021-03-15T22:36:11.638 INFO:tasks.rados.rados.0.smithi076.stdout:3902: finishing write tid 2 to smithi07694878-19 2021-03-15T22:36:11.638 INFO:tasks.rados.rados.0.smithi076.stdout:3898: done (5 left) 2021-03-15T22:36:11.639 INFO:tasks.rados.rados.0.smithi076.stdout:3900: done (4 left) 2021-03-15T22:36:11.639 INFO:tasks.rados.rados.0.smithi076.stdout:3907: delete oid 12 current snap is 462 2021-03-15T22:36:11.639 INFO:tasks.rados.rados.0.smithi076.stdout:3902: finishing write tid 3 to smithi07694878-19 2021-03-15T22:36:11.639 INFO:tasks.rados.rados.0.smithi076.stdout:3902: finishing write tid 4 to smithi07694878-19 2021-03-15T22:36:11.640 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 19 v 2855 (ObjNum 1323 snap 462 seq_num 1323) dirty exists 2021-03-15T22:36:11.640 INFO:tasks.rados.rados.0.smithi076.stdout:3902: left oid 19 (ObjNum 1323 snap 462 seq_num 1323) 2021-03-15T22:36:11.640 INFO:tasks.rados.rados.0.smithi076.stdout:3903: finishing write tid 1 to smithi07694878-39 2021-03-15T22:36:11.640 INFO:tasks.rados.rados.0.smithi076.stdout:3903: finishing write tid 2 to smithi07694878-39 2021-03-15T22:36:11.641 INFO:tasks.rados.rados.0.smithi076.stdout:3903: finishing write tid 3 to smithi07694878-39 2021-03-15T22:36:11.641 INFO:tasks.rados.rados.0.smithi076.stdout:3903: finishing write tid 4 to smithi07694878-39 2021-03-15T22:36:11.641 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2858 (ObjNum 1324 snap 462 seq_num 1324) dirty exists 2021-03-15T22:36:11.641 INFO:tasks.rados.rados.0.smithi076.stdout:3903: left oid 39 (ObjNum 1324 snap 462 seq_num 1324) 2021-03-15T22:36:11.642 INFO:tasks.rados.rados.0.smithi076.stdout:3905: finishing rollback tid 1 to smithi07694878-46 2021-03-15T22:36:11.642 INFO:tasks.rados.rados.0.smithi076.stdout:3905: finishing rollback tid 2 to smithi07694878-46 2021-03-15T22:36:11.642 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2307 (ObjNum 1044 snap 350 seq_num 1044) dirty exists 2021-03-15T22:36:11.642 INFO:tasks.rados.rados.0.smithi076.stdout:3906: finishing copy_from to smithi07694878-4 2021-03-15T22:36:11.643 INFO:tasks.rados.rados.0.smithi076.stdout:3906: got expected ENOENT (src dne) 2021-03-15T22:36:11.643 INFO:tasks.rados.rados.0.smithi076.stdout:3906: finishing copy_from racing read to smithi07694878-4 2021-03-15T22:36:11.644 INFO:tasks.rados.rados.0.smithi076.stdout:3902: done (4 left) 2021-03-15T22:36:11.644 INFO:tasks.rados.rados.0.smithi076.stdout:3903: done (3 left) 2021-03-15T22:36:11.644 INFO:tasks.rados.rados.0.smithi076.stdout:3905: done (2 left) 2021-03-15T22:36:11.645 INFO:tasks.rados.rados.0.smithi076.stdout:3906: done (1 left) 2021-03-15T22:36:11.645 INFO:tasks.rados.rados.0.smithi076.stdout:3907: done (0 left) 2021-03-15T22:36:11.645 INFO:tasks.rados.rados.0.smithi076.stdout:3908: rollback oid 11 current snap is 462 2021-03-15T22:36:11.646 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 11 to 362 2021-03-15T22:36:11.646 INFO:tasks.rados.rados.0.smithi076.stdout:3909: rollback oid 27 current snap is 462 2021-03-15T22:36:11.646 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 27 to 450 2021-03-15T22:36:11.647 INFO:tasks.rados.rados.0.smithi076.stdout:3910: write (excl) oid 44 current snap is 462 2021-03-15T22:36:11.647 INFO:tasks.rados.rados.0.smithi076.stdout:3910: seq_num 1326 ranges {686082=425596,1577829=433132} 2021-03-15T22:36:11.648 INFO:tasks.rados.rados.0.smithi076.stdout:3910: writing smithi07694878-44 from 686082 to 1111678 tid 1 2021-03-15T22:36:11.651 INFO:tasks.rados.rados.0.smithi076.stdout:3910: writing smithi07694878-44 from 1577829 to 2010961 tid 2 2021-03-15T22:36:11.651 INFO:tasks.rados.rados.0.smithi076.stdout:3911: snap_remove snap 459 2021-03-15T22:36:11.656 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:11.679 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:11.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:11.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:11.783 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:11.804 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:11.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:11.806 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:11.907 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:11.929 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:11.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:11.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:11.947 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[56514]: cluster 2021-03-15T22:36:09.833547+0000 osd.7 (osd.7) 190 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:36:11.947 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[56514]: cluster 2021-03-15T22:36:10.121531+0000 osd.0 (osd.0) 163 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:36:11.947 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[56514]: audit 2021-03-15T22:36:10.328830+0000 mgr.y (mgr.14144) 1114 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:11.948 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[56514]: cluster 2021-03-15T22:36:11.021560+0000 osd.5 (osd.5) 60 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:36:11.948 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[56514]: audit 2021-03-15T22:36:11.232778+0000 mon.a (mon.0) 2533 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-03-15T22:36:11.948 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[56514]: audit 2021-03-15T22:36:11.372707+0000 mon.a (mon.0) 2534 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]': finished 2021-03-15T22:36:11.948 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[56514]: cluster 2021-03-15T22:36:11.372768+0000 mon.a (mon.0) 2535 2021-03-15T22:36:11.949 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[56514]: : cluster [DBG] osdmap e1016: 8 total, 8 up, 8 in 2021-03-15T22:36:11.949 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[62156]: cluster 2021-03-15T22:36:09.833547+0000 osd.7 (osd.7) 190 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:36:11.949 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[62156]: cluster 2021-03-15T22:36:10.121531+0000 osd.0 (osd.0) 163 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:36:11.949 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[62156]: audit 2021-03-15T22:36:10.328830+0000 mgr.y (mgr.14144) 1114 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:11.950 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[62156]: cluster 2021-03-15T22:36:11.021560+0000 osd.5 (osd.5) 60 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:36:11.950 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[62156]: audit 2021-03-15T22:36:11.232778+0000 mon.a (mon.0) 2533 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-03-15T22:36:11.950 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[62156]: audit 2021-03-15T22:36:11.372707+0000 mon.a (mon.0) 2534 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]': finished 2021-03-15T22:36:11.950 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:11 smithi076 conmon[62156]: cluster 2021-03-15T22:36:11.372768+0000 mon.a (mon.0) 2535 : cluster [DBG] osdmap e1016: 8 total, 8 up, 8 in 2021-03-15T22:36:12.031 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:12.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: cluster 2021-03-15T22:36:09.833547+0000 osd.7 (osd.7) 190 : cluster [DBG] 3.13 scrub ok 2021-03-15T22:36:12.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: cluster 2021-03-15T22:36:10.121531+0000 2021-03-15T22:36:12.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: osd.0 (osd.0) 163 : cluster [DBG] 3.12 scrub ok 2021-03-15T22:36:12.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: audit 2021-03-15T22:36:10.328830+0000 mgr.y (mgr.14144) 1114 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:12.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: cluster 2021-03-15T22:36:11.021560+0000 osd.5 (osd.5) 60 : cluster [DBG] 4.15 scrub ok 2021-03-15T22:36:12.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: 2021-03-15T22:36:12.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: audit 2021-03-15T22:36:11.232778+0000 2021-03-15T22:36:12.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: mon.a (mon.0) 2533 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]: dispatch 2021-03-15T22:36:12.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:36:12.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: 03-15T22:36:11.372707+0000 2021-03-15T22:36:12.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: mon.a (mon.0) 2534 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "27"}]': finished 2021-03-15T22:36:12.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: 2021-03-15T22:36:12.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: cluster 2021-03-15T22:36:12.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: 2021- 2021-03-15T22:36:12.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: 03-15T22:36:11.372768+0000 2021-03-15T22:36:12.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: mon.a (mon.0) 2535 : cluster [DBG] 2021-03-15T22:36:12.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:11 smithi125 conmon[56739]: osdmap e1016: 8 total, 8 up, 8 in 2021-03-15T22:36:12.057 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:12.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:12.058 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:12.159 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:12.185 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:12.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:12.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:12.233 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:12.288 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:12.313 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:12.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:12.315 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:12.387 INFO:tasks.rados.rados.0.smithi076.stdout:3911: done (3 left) 2021-03-15T22:36:12.387 INFO:tasks.rados.rados.0.smithi076.stdout:3912: copy_from oid 41 from oid 38 current snap is 462 2021-03-15T22:36:12.387 INFO:tasks.rados.rados.0.smithi076.stdout:3913: read oid 29 snap -1 2021-03-15T22:36:12.387 INFO:tasks.rados.rados.0.smithi076.stdout:3913: expect (ObjNum 1228 snap 425 seq_num 1228) 2021-03-15T22:36:12.388 INFO:tasks.rados.rados.0.smithi076.stdout:3914: write oid 50 current snap is 462 2021-03-15T22:36:12.388 INFO:tasks.rados.rados.0.smithi076.stdout:3914: seq_num 1327 ranges {702889=500845,1636311=576790} 2021-03-15T22:36:12.393 INFO:tasks.rados.rados.0.smithi076.stdout:3914: writing smithi07694878-50 from 702889 to 1203734 tid 1 2021-03-15T22:36:12.400 INFO:tasks.rados.rados.0.smithi076.stdout:3914: writing smithi07694878-50 from 1636311 to 2213101 tid 2 2021-03-15T22:36:12.400 INFO:tasks.rados.rados.0.smithi076.stdout:3915: write (excl) oid 4 current snap is 462 2021-03-15T22:36:12.400 INFO:tasks.rados.rados.0.smithi076.stdout:3909: finishing rollback tid 1 to smithi07694878-27 2021-03-15T22:36:12.401 INFO:tasks.rados.rados.0.smithi076.stdout:3915: seq_num 1328 ranges {719696=576094,2094793=224903} 2021-03-15T22:36:12.407 INFO:tasks.rados.rados.0.smithi076.stdout:3915: writing smithi07694878-4 from 719696 to 1295790 tid 1 2021-03-15T22:36:12.414 INFO:tasks.rados.rados.0.smithi076.stdout:3915: writing smithi07694878-4 from 2094793 to 2319696 tid 2 2021-03-15T22:36:12.414 INFO:tasks.rados.rados.0.smithi076.stdout:3916: read oid 19 snap -1 2021-03-15T22:36:12.414 INFO:tasks.rados.rados.0.smithi076.stdout:3916: expect (ObjNum 1323 snap 462 seq_num 1323) 2021-03-15T22:36:12.414 INFO:tasks.rados.rados.0.smithi076.stdout:3917: read oid 37 snap -1 2021-03-15T22:36:12.415 INFO:tasks.rados.rados.0.smithi076.stdout:3917: expect (ObjNum 1284 snap 442 seq_num 1284) 2021-03-15T22:36:12.415 INFO:tasks.rados.rados.0.smithi076.stdout:3918: write oid 10 current snap is 462 2021-03-15T22:36:12.415 INFO:tasks.rados.rados.0.smithi076.stdout:3909: finishing rollback tid 2 to smithi07694878-27 2021-03-15T22:36:12.415 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 3162 (ObjNum 1279 snap 442 seq_num 1279) dirty exists 2021-03-15T22:36:12.416 INFO:tasks.rados.rados.0.smithi076.stdout:3908: finishing rollback tid 0 to smithi07694878-11 2021-03-15T22:36:12.416 INFO:tasks.rados.rados.0.smithi076.stdout:3908: finishing rollback tid 1 to smithi07694878-11 2021-03-15T22:36:12.416 INFO:tasks.rados.rados.0.smithi076.stdout:3908: finishing rollback tid 2 to smithi07694878-11 2021-03-15T22:36:12.416 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 11 v 3053 (ObjNum 1041 snap 349 seq_num 1041) dirty exists 2021-03-15T22:36:12.416 INFO:tasks.rados.rados.0.smithi076.stdout:3910: finishing write tid 1 to smithi07694878-44 2021-03-15T22:36:12.417 INFO:tasks.rados.rados.0.smithi076.stdout:3910: finishing write tid 2 to smithi07694878-44 2021-03-15T22:36:12.417 INFO:tasks.rados.rados.0.smithi076.stdout:3910: finishing write tid 3 to smithi07694878-44 2021-03-15T22:36:12.417 INFO:tasks.rados.rados.0.smithi076.stdout:3910: finishing write tid 4 to smithi07694878-44 2021-03-15T22:36:12.417 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 44 v 3165 (ObjNum 1326 snap 462 seq_num 1326) dirty exists 2021-03-15T22:36:12.417 INFO:tasks.rados.rados.0.smithi076.stdout:3910: left oid 44 (ObjNum 1326 snap 462 seq_num 1326) 2021-03-15T22:36:12.418 INFO:tasks.rados.rados.0.smithi076.stdout:3913: expect (ObjNum 1228 snap 425 seq_num 1228) 2021-03-15T22:36:12.418 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:12.444 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:12.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:12.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:12.521 INFO:tasks.rados.rados.0.smithi076.stdout:3914: finishing write tid 1 to smithi07694878-50 2021-03-15T22:36:12.522 INFO:tasks.rados.rados.0.smithi076.stdout:3914: finishing write tid 2 to smithi07694878-50 2021-03-15T22:36:12.522 INFO:tasks.rados.rados.0.smithi076.stdout:3914: finishing write tid 3 to smithi07694878-50 2021-03-15T22:36:12.522 INFO:tasks.rados.rados.0.smithi076.stdout:3914: finishing write tid 4 to smithi07694878-50 2021-03-15T22:36:12.522 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 2690 (ObjNum 1327 snap 462 seq_num 1327) dirty exists 2021-03-15T22:36:12.522 INFO:tasks.rados.rados.0.smithi076.stdout:3914: left oid 50 (ObjNum 1327 snap 462 seq_num 1327) 2021-03-15T22:36:12.523 INFO:tasks.rados.rados.0.smithi076.stdout:3912: finishing copy_from to smithi07694878-41 2021-03-15T22:36:12.523 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 41 v 2845 (ObjNum 1116 snap 388 seq_num 1116) dirty exists 2021-03-15T22:36:12.523 INFO:tasks.rados.rados.0.smithi076.stdout:3912: finishing copy_from racing read to smithi07694878-41 2021-03-15T22:36:12.523 INFO:tasks.rados.rados.0.smithi076.stdout:3915: finishing write tid 1 to smithi07694878-4 2021-03-15T22:36:12.524 INFO:tasks.rados.rados.0.smithi076.stdout:3918: seq_num 1329 ranges {736503=651343,2153275=183228} 2021-03-15T22:36:12.528 INFO:tasks.rados.rados.0.smithi076.stdout:3918: writing smithi07694878-10 from 736503 to 1387846 tid 1 2021-03-15T22:36:12.534 INFO:tasks.rados.rados.0.smithi076.stdout:3918: writing smithi07694878-10 from 2153275 to 2336503 tid 2 2021-03-15T22:36:12.534 INFO:tasks.rados.rados.0.smithi076.stdout:3908: done (9 left) 2021-03-15T22:36:12.535 INFO:tasks.rados.rados.0.smithi076.stdout:3909: done (8 left) 2021-03-15T22:36:12.535 INFO:tasks.rados.rados.0.smithi076.stdout:3910: done (7 left) 2021-03-15T22:36:12.535 INFO:tasks.rados.rados.0.smithi076.stdout:3912: done (6 left) 2021-03-15T22:36:12.535 INFO:tasks.rados.rados.0.smithi076.stdout:3913: done (5 left) 2021-03-15T22:36:12.536 INFO:tasks.rados.rados.0.smithi076.stdout:3914: done (4 left) 2021-03-15T22:36:12.536 INFO:tasks.rados.rados.0.smithi076.stdout:3919: copy_from oid 23 from oid 45 current snap is 462 2021-03-15T22:36:12.536 INFO:tasks.rados.rados.0.smithi076.stdout:3915: finishing write tid 2 to smithi07694878-4 2021-03-15T22:36:12.536 INFO:tasks.rados.rados.0.smithi076.stdout:3915: finishing write tid 3 to smithi07694878-4 2021-03-15T22:36:12.536 INFO:tasks.rados.rados.0.smithi076.stdout:3915: finishing write tid 4 to smithi07694878-4 2021-03-15T22:36:12.537 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 4 v 3056 (ObjNum 1328 snap 462 seq_num 1328) dirty exists 2021-03-15T22:36:12.537 INFO:tasks.rados.rados.0.smithi076.stdout:3915: left oid 4 (ObjNum 1328 snap 462 seq_num 1328) 2021-03-15T22:36:12.537 INFO:tasks.rados.rados.0.smithi076.stdout:3916: expect (ObjNum 1323 snap 462 seq_num 1323) 2021-03-15T22:36:12.547 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:12.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:12.570 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:12.571 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:12.672 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:12.696 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:12.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:12.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:12.746 INFO:tasks.rados.rados.0.smithi076.stdout:3917: expect (ObjNum 1284 snap 442 seq_num 1284) 2021-03-15T22:36:12.799 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:12.821 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:12.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:12.823 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:12.881 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:12 smithi076 conmon[56514]: cluster 2021-03-15T22:36:11. 2021-03-15T22:36:12.881 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:12 smithi076 conmon[56514]: 231512+0000 mgr.y (mgr.14144) 1115 : cluster [DBG] pgmap v1670: 147 pgs: 1 clean+premerge+peered, 146 active+clean; 753 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 638 KiB/s wr, 22 op/s 2021-03-15T22:36:12.882 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:12 smithi076 conmon[56514]: cluster 2021-03-15T22:36:11.805408+0000 osd.3 (osd.3) 176 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:36:12.882 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:12 smithi076 conmon[56514]: cluster 2021-03-15T22:36:12.378879+0000 mon.a (mon.0) 2536 : cluster [DBG] osdmap e1017: 8 total, 8 up, 8 in 2021-03-15T22:36:12.883 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:12 smithi076 conmon[62156]: cluster 2021-03-15T22:36:11.231512+0000 mgr.y ( 2021-03-15T22:36:12.883 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:12 smithi076 conmon[62156]: mgr.14144) 1115 : cluster [DBG] pgmap v1670: 147 pgs: 1 clean+premerge+peered, 146 active+clean; 753 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 638 KiB/s wr, 22 op/s 2021-03-15T22:36:12.884 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:12 smithi076 conmon[62156]: cluster 2021-03-15T22:36:11.805408+0000 osd.3 (osd.3) 176 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:36:12.884 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:12 smithi076 conmon[62156]: cluster 2021-03-15T22:36:12.378879+0000 mon.a (mon.0) 2536 : cluster [DBG] osdmap e1017: 8 total, 8 up, 8 in 2021-03-15T22:36:12.903 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:36:12.906 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:36:12.924 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:12.947 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:12.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:12.949 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:12.955 INFO:tasks.rados.rados.0.smithi076.stdout:3915: done (4 left) 2021-03-15T22:36:12.956 INFO:tasks.rados.rados.0.smithi076.stdout:3916: done (3 left) 2021-03-15T22:36:12.956 INFO:tasks.rados.rados.0.smithi076.stdout:3917: done (2 left) 2021-03-15T22:36:12.956 INFO:tasks.rados.rados.0.smithi076.stdout:3920: snap_remove snap 417 2021-03-15T22:36:13.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:12 smithi125 conmon[56739]: cluster 2021-03-15T22:36:11.231512+0000 mgr.y (mgr.14144) 1115 2021-03-15T22:36:13.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:12 smithi125 conmon[56739]: : cluster [DBG] pgmap v1670: 147 pgs: 1 clean+premerge+peered, 146 active+clean; 753 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 638 KiB/s wr, 22 op/s 2021-03-15T22:36:13.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:12 smithi125 conmon[56739]: cluster 2021-03-15T22:36:11.805408+0000 osd.3 (osd.3) 176 : cluster [DBG] 5.13 scrub ok 2021-03-15T22:36:13.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:12 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:36:13.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:12 smithi125 conmon[56739]: :36:12.378879+0000 mon.a (mon.0) 2021-03-15T22:36:13.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:12 smithi125 conmon[56739]: 2536 : cluster [DBG] osdmap e1017: 8 total, 8 up, 8 in 2021-03-15T22:36:13.050 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:13.072 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:13.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:13.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:13.175 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:13.204 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:13.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:13.205 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:13.307 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:13.333 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:13.334 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:13.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:13.378 INFO:tasks.rados.rados.0.smithi076.stdout:3920: done (2 left) 2021-03-15T22:36:13.379 INFO:tasks.rados.rados.0.smithi076.stdout:3921: snap_remove snap 450 2021-03-15T22:36:13.435 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:13.458 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:13.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:13.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:13.561 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:13.583 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:13.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:13.585 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:13.686 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:13.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:13.713 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:13.713 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:13.814 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:13.840 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:13.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:13.842 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:13.943 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:13.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:13.969 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:13.969 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:14.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[56514]: cluster 2021-03-15T22:36:12.698325+0000 osd.6 (osd.6) 153 : cluster 2021-03-15T22:36:14.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[56514]: [DBG] 4.5 scrub ok 2021-03-15T22:36:14.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[56514]: audit 2021-03-15T22:36:12.883114+0000 mgr.y (mgr.14144) 1116 : audit [DBG] from='client.25633 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:14.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[56514]: cluster 2021-03-15T22:36:13.027636+0000 osd.5 (osd.5) 61 : cluster [DBG] 4.2 scrub ok 2021-03-15T22:36:14.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[56514]: audit 2021-03-15T22:36:13.232850+0000 mon.a (mon.0) 2537 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "33"}]: dispatch 2021-03-15T22:36:14.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[56514]: audit 2021-03-15T22:36:13.375301+0000 mon.a (mon.0) 2538 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "33"}]': finished 2021-03-15T22:36:14.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[56514]: cluster 2021-03-15T22:36:13.375349+0000 mon.a (mon.0) 2539 : cluster [DBG] osdmap e1018: 8 total, 8 up, 8 in 2021-03-15T22:36:14.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: cluster 2021-03-15T22:36:12.698325 2021-03-15T22:36:14.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: +0000 osd.6 (osd.6) 153 : cluster [DBG] 4.5 scrub ok 2021-03-15T22:36:14.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:36:14.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: 15T22:36:12.883114+0000 mgr.y (mgr.14144) 1116 : audit [DBG] from='client.25633 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:14.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: 2021-03-15T22:36:14.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:36:14.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: 03-15T22:36:13.027636+0000 osd.5 (osd.5) 61 : cluster [DBG] 2021-03-15T22:36:14.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: 4.2 scrub ok 2021-03-15T22:36:14.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: audit 2021-03-15T22:36:14.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: 2021-03-15T22:36:13.232850+0000 mon.a (mon.0) 2537 : audit 2021-03-15T22:36:14.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "33"}]: dispatch 2021-03-15T22:36:14.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: audit 2021-03-15T22:36:13.375301+0000 mon.a (mon. 2021-03-15T22:36:14.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: 0) 2538 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "33"}]': finished 2021-03-15T22:36:14.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: cluster 2021-03-15T22:36:13.375349+0000 mon.a 2021-03-15T22:36:14.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:13 smithi076 conmon[62156]: (mon.0) 2539 : cluster [DBG] osdmap e1018: 8 total, 8 up, 8 in 2021-03-15T22:36:14.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: cluster 2021-03-15T22:36:12.698325+0000 osd.6 (osd.6) 153 : cluster 2021-03-15T22:36:14.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: [DBG] 4.5 scrub ok 2021-03-15T22:36:14.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: audit 2021-03-15T22:36:12.883114+0000 mgr.y (mgr.14144) 1116 : audit [DBG] from='client.25633 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:14.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: cluster 2021-03-15T22:36:13. 2021-03-15T22:36:14.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: 027636+0000 osd.5 (osd.5) 61 2021-03-15T22:36:14.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: : cluster [DBG] 4.2 scrub ok 2021-03-15T22:36:14.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:36:14.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: 03-15T22:36:13.232850+0000 2021-03-15T22:36:14.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: mon.a (mon.0) 2537 : audit 2021-03-15T22:36:14.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "33"}]: dispatch 2021-03-15T22:36:14.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:36:14.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: 03-15T22:36:13.375301+0000 2021-03-15T22:36:14.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: mon.a (mon.0) 2538 : 2021-03-15T22:36:14.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "33"}]': finished 2021-03-15T22:36:14.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:36:14.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: -03-15T22:36:13 2021-03-15T22:36:14.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: .375349+0000 mon.a (mon.0 2021-03-15T22:36:14.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:13 smithi125 conmon[56739]: ) 2539 : cluster [DBG] osdmap e1018: 8 total, 8 up, 8 in 2021-03-15T22:36:14.071 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:14.097 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:14.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:14.099 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:14.200 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:14.225 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:14.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:14.227 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:14.328 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:14.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:14.353 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:14.353 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:14.381 INFO:tasks.rados.rados.0.smithi076.stdout:3921: done (2 left) 2021-03-15T22:36:14.381 INFO:tasks.rados.rados.0.smithi076.stdout:3922: delete oid 32 current snap is 462 2021-03-15T22:36:14.382 INFO:tasks.rados.rados.0.smithi076.stdout:3918: finishing write tid 1 to smithi07694878-10 2021-03-15T22:36:14.382 INFO:tasks.rados.rados.0.smithi076.stdout:3918: finishing write tid 2 to smithi07694878-10 2021-03-15T22:36:14.382 INFO:tasks.rados.rados.0.smithi076.stdout:3918: finishing write tid 3 to smithi07694878-10 2021-03-15T22:36:14.382 INFO:tasks.rados.rados.0.smithi076.stdout:3918: finishing write tid 4 to smithi07694878-10 2021-03-15T22:36:14.382 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 3185 (ObjNum 1329 snap 462 seq_num 1329) dirty exists 2021-03-15T22:36:14.383 INFO:tasks.rados.rados.0.smithi076.stdout:3918: left oid 10 (ObjNum 1329 snap 462 seq_num 1329) 2021-03-15T22:36:14.383 INFO:tasks.rados.rados.0.smithi076.stdout:3919: finishing copy_from to smithi07694878-23 2021-03-15T22:36:14.383 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 23 v 3088 (ObjNum 1307 snap 458 seq_num 1307) dirty exists 2021-03-15T22:36:14.383 INFO:tasks.rados.rados.0.smithi076.stdout:3919: finishing copy_from racing read to smithi07694878-23 2021-03-15T22:36:14.393 INFO:tasks.rados.rados.0.smithi076.stdout:3918: done (2 left) 2021-03-15T22:36:14.394 INFO:tasks.rados.rados.0.smithi076.stdout:3919: done (1 left) 2021-03-15T22:36:14.394 INFO:tasks.rados.rados.0.smithi076.stdout:3922: done (0 left) 2021-03-15T22:36:14.394 INFO:tasks.rados.rados.0.smithi076.stdout:3923: read oid 3 snap -1 2021-03-15T22:36:14.394 INFO:tasks.rados.rados.0.smithi076.stdout:3923: expect (ObjNum 620 snap 193 seq_num 620) 2021-03-15T22:36:14.395 INFO:tasks.rados.rados.0.smithi076.stdout:3924: snap_remove snap 430 2021-03-15T22:36:14.454 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:14.480 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:14.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:14.481 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:14.582 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:14.607 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:14.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:14.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:14.710 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:14.736 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:14.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:14.738 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:14.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:14 smithi076 conmon[56514]: cluster 2021-03-15T22:36:12.855071+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.f scrub ok 2021-03-15T22:36:14.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:14 smithi076 conmon[56514]: cluster 2021-03-15T22:36:13.232035+0000 mgr.y (mgr.14144) 1117 : cluster [DBG] pgmap v1673: 146 pgs: 3 peering, 1 clean+premerge+peered, 142 active+clean; 757 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 2.1 MiB/s wr, 25 op/s 2021-03-15T22:36:14.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:14 smithi076 conmon[56514]: cluster 2021-03-15T22:36:13.313273+0000 osd.1 (osd.1) 190 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:36:14.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:14 smithi076 conmon[56514]: cluster 2021-03-15T22:36:14.037753+0000 osd.5 (osd.5) 62 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:36:14.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:14 smithi076 conmon[56514]: cluster 2021-03-15T22:36:14.376697+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e1019: 8 total, 8 up, 8 in 2021-03-15T22:36:14.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:14 smithi076 conmon[62156]: cluster 2021-03-15T22:36:12.855071+0000 osd.3 (osd.3) 177 2021-03-15T22:36:14.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:14 smithi076 conmon[62156]: : cluster [DBG] 6.f scrub ok 2021-03-15T22:36:14.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:14 smithi076 conmon[62156]: cluster 2021-03-15T22:36:13.232035+0000 mgr.y (mgr.14144) 1117 : cluster [DBG] pgmap v1673: 146 pgs: 3 peering, 1 clean+premerge+peered, 142 active+clean; 757 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 2.1 MiB/s wr, 25 op/s 2021-03-15T22:36:14.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:14 smithi076 conmon[62156]: cluster 2021-03-15T22:36:13.313273+0000 osd.1 (osd.1) 190 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:36:14.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:14 smithi076 conmon[62156]: cluster 2021-03-15T22:36:14.037753+0000 osd.5 (osd.5) 62 : cluster [DBG] 3.5 scrub ok 2021-03-15T22:36:14.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:14 smithi076 conmon[62156]: cluster 2021-03-15T22:36:14.376697+0000 mon.a (mon.0) 2540 : cluster [DBG] osdmap e1019: 8 total, 8 up, 8 in 2021-03-15T22:36:14.796 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:36:14.797 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:36:14.839 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:14.852 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1672,"stamp":"2021-03-15T22:36:12.376374+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":789631843,"num_objects":674,"num_object_clones":309,"num_object_copies":2022,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":674,"num_whiteouts":17,"num_read":9827,"num_read_kb":7469821,"num_write":6138,"num_write_kb":2039925,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5902,"num_bytes_recovered":2230124932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24409,"ondisk_log_size":24409,"up":438,"acting":438,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":443,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2157392,"kb_used_data":1735776,"kb_used_omap":273,"kb_used_meta":420654,"kb_avail":747803824,"statfs":{"total":767960285184,"available":765751115776,"internally_reserved":0,"allocated":1777434624,"data_stored":2478506570,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":430750586},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5400396,"num_objects":5,"num_object_clones":5,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":65,"num_read_kb":78403,"num_write":22,"num_write_kb":3545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.005149"},"pg_stats":[{"pgid":"6.21","version":"0'0","reported_seq":"40","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.370264+0000","last_change":"2021-03-15T22:35:36.758571+0000","last_active":"2021-03-15T22:36:05.370264+0000","last_peered":"2021-03-15T22:36:05.370264+0000","last_clean":"2021-03-15T22:36:05.370264+0000","last_became_active":"2021-03-15T22:35:36.758203+0000","last_became_peered":"2021-03-15T22:35:36.758203+0000","last_unstale":"2021-03-15T22:36:05.370264+0000","last_undegraded":"2021-03-15T22:36:05.370264+0000","last_fullsized":"2021-03-15T22:36:05.370264+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.701955+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.701955+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.20","version":"0'0","reported_seq":"44","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384243+0000","last_change":"2021-03-15T22:35:36.357269+0000","last_active":"2021-03-15T22:36:09.384243+0000","last_peered":"2021-03-15T22:36:09.384243+0000","last_clean":"2021-03-15T22:36:09.384243+0000","last_became_active":"2021-03-15T22:35:36.356824+0000","last_became_peered":"2021-03-15T22:35:36.356824+0000","last_unstale":"2021-03-15T22:36:09.384243+0000","last_undegraded":"2021-03-15T22:36:09.384243+0000","last_fullsized":"2021-03-15T22:36:09.384243+0000","mapping_epoch":980,"log_start":"0'0","ondisk_log_start":"0'0","created":918,"last_epoch_clean":981,"parent":"0.0","parent_split_bits":6,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.19","version":"317'7","reported_seq":"374","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.370353+0000","last_change":"2021-03-15T22:34:39.122745+0000","last_active":"2021-03-15T22:36:05.370353+0000","last_peered":"2021-03-15T22:36:05.370353+0000","last_clean":"2021-03-15T22:36:05.370353+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:36:05.370353+0000","last_undegraded":"2021-03-15T22:36:05.370353+0000","last_fullsized":"2021-03-15T22:36:05.370353+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:34:39.122665+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.122665+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1031","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.358405+0000","last_change":"2021-03-15T22:34:10.331764+0000","last_active":"2021-03-15T22:36:10.358405+0000","last_peered":"2021-03-15T22:36:10.358405+0000","last_clean":"2021-03-15T22:36:10.358405+0000","last_became_active":"2021-03-15T22:34:10.331424+0000","last_became_peered":"2021-03-15T22:34:10.331424+0000","last_unstale":"2021-03-15T22:36:10.358405+0000","last_undegraded":"2021-03-15T22:36:10.358405+0000","last_fullsized":"2021-03-15T22:36:10.358405+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.564464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.564464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"1114","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.357854+0000","last_change":"2021-03-15T22:35:14.434026+0000","last_active":"2021-03-15T22:36:10.357854+0000","last_peered":"2021-03-15T22:36:10.357854+0000","last_clean":"2021-03-15T22:36:10.357854+0000","last_became_active":"2021-03-15T22:33:04.345393+0000","last_became_peered":"2021-03-15T22:33:04.345393+0000","last_unstale":"2021-03-15T22:36:10.357854+0000","last_undegraded":"2021-03-15T22:36:10.357854+0000","last_fullsized":"2021-03-15T22:36:10.357854+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:35:14.433961+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:35:14.433961+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"323","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.357599+0000","last_change":"2021-03-15T22:36:08.386295+0000","last_active":"2021-03-15T22:36:10.357599+0000","last_peered":"2021-03-15T22:36:10.357599+0000","last_clean":"2021-03-15T22:36:10.357599+0000","last_became_active":"2021-03-15T22:33:04.345518+0000","last_became_peered":"2021-03-15T22:33:04.345518+0000","last_unstale":"2021-03-15T22:36:10.357599+0000","last_undegraded":"2021-03-15T22:36:10.357599+0000","last_fullsized":"2021-03-15T22:36:10.357599+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:08.386208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:08.386208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"98","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.354928+0000","last_change":"2021-03-15T22:34:43.345093+0000","last_active":"2021-03-15T22:36:10.354928+0000","last_peered":"2021-03-15T22:36:10.354928+0000","last_clean":"2021-03-15T22:36:10.354928+0000","last_became_active":"2021-03-15T22:34:43.344764+0000","last_became_peered":"2021-03-15T22:34:43.344764+0000","last_unstale":"2021-03-15T22:36:10.354928+0000","last_undegraded":"2021-03-15T22:36:10.354928+0000","last_fullsized":"2021-03-15T22:36:10.354928+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:08.652741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.652741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"164","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.354854+0000","last_change":"2021-03-15T22:36:08.831621+0000","last_active":"2021-03-15T22:36:10.354854+0000","last_peered":"2021-03-15T22:36:10.354854+0000","last_clean":"2021-03-15T22:36:10.354854+0000","last_became_active":"2021-03-15T22:34:55.346148+0000","last_became_peered":"2021-03-15T22:34:55.346148+0000","last_unstale":"2021-03-15T22:36:10.354854+0000","last_undegraded":"2021-03-15T22:36:10.354854+0000","last_fullsized":"2021-03-15T22:36:10.354854+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:36:08.831557+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:36:08.831557+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":5340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"863","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.376882+0000","last_change":"2021-03-15T22:35:16.848985+0000","last_active":"2021-03-15T22:36:07.376882+0000","last_peered":"2021-03-15T22:36:07.376882+0000","last_clean":"2021-03-15T22:36:07.376882+0000","last_became_active":"2021-03-15T22:34:03.570306+0000","last_became_peered":"2021-03-15T22:34:03.570306+0000","last_unstale":"2021-03-15T22:36:07.376882+0000","last_undegraded":"2021-03-15T22:36:07.376882+0000","last_fullsized":"2021-03-15T22:36:07.376882+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:16.848918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.848918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"484","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384549+0000","last_change":"2021-03-15T22:35:49.146403+0000","last_active":"2021-03-15T22:36:09.384549+0000","last_peered":"2021-03-15T22:36:09.384549+0000","last_clean":"2021-03-15T22:36:09.384549+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:36:09.384549+0000","last_undegraded":"2021-03-15T22:36:09.384549+0000","last_fullsized":"2021-03-15T22:36:09.384549+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.146337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.146337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"1165","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.358582+0000","last_change":"2021-03-15T22:33:17.502781+0000","last_active":"2021-03-15T22:36:10.358582+0000","last_peered":"2021-03-15T22:36:10.358582+0000","last_clean":"2021-03-15T22:36:10.358582+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:36:10.358582+0000","last_undegraded":"2021-03-15T22:36:10.358582+0000","last_fullsized":"2021-03-15T22:36:10.358582+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:33:17.502703+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.502703+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"298","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.376210+0000","last_change":"2021-03-15T22:35:12.404980+0000","last_active":"2021-03-15T22:36:07.376210+0000","last_peered":"2021-03-15T22:36:07.376210+0000","last_clean":"2021-03-15T22:36:07.376210+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:36:07.376210+0000","last_undegraded":"2021-03-15T22:36:07.376210+0000","last_fullsized":"2021-03-15T22:36:07.376210+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.404897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.404897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"651","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377140+0000","last_change":"2021-03-15T22:34:43.344692+0000","last_active":"2021-03-15T22:36:07.377140+0000","last_peered":"2021-03-15T22:36:07.377140+0000","last_clean":"2021-03-15T22:36:07.377140+0000","last_became_active":"2021-03-15T22:34:43.344401+0000","last_became_peered":"2021-03-15T22:34:43.344401+0000","last_unstale":"2021-03-15T22:36:07.377140+0000","last_undegraded":"2021-03-15T22:36:07.377140+0000","last_fullsized":"2021-03-15T22:36:07.377140+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.774079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.774079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"1161","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.358853+0000","last_change":"2021-03-15T22:35:45.410540+0000","last_active":"2021-03-15T22:36:10.358853+0000","last_peered":"2021-03-15T22:36:10.358853+0000","last_clean":"2021-03-15T22:36:10.358853+0000","last_became_active":"2021-03-15T22:34:55.335550+0000","last_became_peered":"2021-03-15T22:34:55.335550+0000","last_unstale":"2021-03-15T22:36:10.358853+0000","last_undegraded":"2021-03-15T22:36:10.358853+0000","last_fullsized":"2021-03-15T22:36:10.358853+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:35:45.410466+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.410466+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"971","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.371783+0000","last_change":"2021-03-15T22:35:10.068109+0000","last_active":"2021-03-15T22:36:05.371783+0000","last_peered":"2021-03-15T22:36:05.371783+0000","last_clean":"2021-03-15T22:36:05.371783+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:36:05.371783+0000","last_undegraded":"2021-03-15T22:36:05.371783+0000","last_fullsized":"2021-03-15T22:36:05.371783+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:35:10.068030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.068030+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1038","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359497+0000","last_change":"2021-03-15T22:34:41.614750+0000","last_active":"2021-03-15T22:36:10.359497+0000","last_peered":"2021-03-15T22:36:10.359497+0000","last_clean":"2021-03-15T22:36:10.359497+0000","last_became_active":"2021-03-15T22:33:04.329078+0000","last_became_peered":"2021-03-15T22:33:04.329078+0000","last_unstale":"2021-03-15T22:36:10.359497+0000","last_undegraded":"2021-03-15T22:36:10.359497+0000","last_fullsized":"2021-03-15T22:36:10.359497+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.614684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.614684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"84'2","reported_seq":"36","reported_epoch":"1016","state":"peering","last_fresh":"2021-03-15T22:36:11.392403+0000","last_change":"2021-03-15T22:36:11.392704+0000","last_active":"2021-03-15T22:36:11.392403+0000","last_peered":"2021-03-15T22:36:10.354841+0000","last_clean":"2021-03-15T22:36:10.354841+0000","last_became_active":"2021-03-15T22:35:50.354808+0000","last_became_peered":"2021-03-15T22:35:50.354808+0000","last_unstale":"2021-03-15T22:36:11.392403+0000","last_undegraded":"2021-03-15T22:36:11.392403+0000","last_fullsized":"2021-03-15T22:36:11.392403+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"84'2","last_scrub_stamp":"2021-03-15T22:32:41.673792+0000","last_deep_scrub":"84'2","last_deep_scrub_stamp":"2021-03-15T22:19:14.859443+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.673792+0000","log_size":2,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"568","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.372442+0000","last_change":"2021-03-15T22:34:55.366147+0000","last_active":"2021-03-15T22:36:05.372442+0000","last_peered":"2021-03-15T22:36:05.372442+0000","last_clean":"2021-03-15T22:36:05.372442+0000","last_became_active":"2021-03-15T22:34:55.350325+0000","last_became_peered":"2021-03-15T22:34:55.350325+0000","last_unstale":"2021-03-15T22:36:05.372442+0000","last_undegraded":"2021-03-15T22:36:05.372442+0000","last_fullsized":"2021-03-15T22:36:05.372442+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:34:38.119156+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:38.119156+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1069","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.358797+0000","last_change":"2021-03-15T22:35:42.445978+0000","last_active":"2021-03-15T22:36:10.358797+0000","last_peered":"2021-03-15T22:36:10.358797+0000","last_clean":"2021-03-15T22:36:10.358797+0000","last_became_active":"2021-03-15T22:34:55.338001+0000","last_became_peered":"2021-03-15T22:34:55.338001+0000","last_unstale":"2021-03-15T22:36:10.358797+0000","last_undegraded":"2021-03-15T22:36:10.358797+0000","last_fullsized":"2021-03-15T22:36:10.358797+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:42.445911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.445911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"853","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359842+0000","last_change":"2021-03-15T22:35:20.025959+0000","last_active":"2021-03-15T22:36:10.359842+0000","last_peered":"2021-03-15T22:36:10.359842+0000","last_clean":"2021-03-15T22:36:10.359842+0000","last_became_active":"2021-03-15T22:33:04.336804+0000","last_became_peered":"2021-03-15T22:33:04.336804+0000","last_unstale":"2021-03-15T22:36:10.359842+0000","last_undegraded":"2021-03-15T22:36:10.359842+0000","last_fullsized":"2021-03-15T22:36:10.359842+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.025892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":"35","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.372443+0000","last_change":"2021-03-15T22:36:08.685772+0000","last_active":"2021-03-15T22:36:09.372443+0000","last_peered":"2021-03-15T22:36:09.372443+0000","last_clean":"2021-03-15T22:36:09.372443+0000","last_became_active":"2021-03-15T22:35:50.354946+0000","last_became_peered":"2021-03-15T22:35:50.354946+0000","last_unstale":"2021-03-15T22:36:09.372443+0000","last_undegraded":"2021-03-15T22:36:09.372443+0000","last_fullsized":"2021-03-15T22:36:09.372443+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:08.685698+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.443613+0000","last_clean_scrub_stamp":"2021-03-15T22:36:08.685698+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"464","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.354532+0000","last_change":"2021-03-15T22:33:15.617975+0000","last_active":"2021-03-15T22:36:10.354532+0000","last_peered":"2021-03-15T22:36:10.354532+0000","last_clean":"2021-03-15T22:36:10.354532+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:36:10.354532+0000","last_undegraded":"2021-03-15T22:36:10.354532+0000","last_fullsized":"2021-03-15T22:36:10.354532+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:33:15.617891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.617891+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"891'22","reported_seq":"590","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.373129+0000","last_change":"2021-03-15T22:35:38.710708+0000","last_active":"2021-03-15T22:36:09.373129+0000","last_peered":"2021-03-15T22:36:09.373129+0000","last_clean":"2021-03-15T22:36:09.373129+0000","last_became_active":"2021-03-15T22:34:06.311478+0000","last_became_peered":"2021-03-15T22:34:06.311478+0000","last_unstale":"2021-03-15T22:36:09.373129+0000","last_undegraded":"2021-03-15T22:36:09.373129+0000","last_fullsized":"2021-03-15T22:36:09.373129+0000","mapping_epoch":890,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"891'22","last_scrub_stamp":"2021-03-15T22:35:38.710643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:38.710643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1044","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.376577+0000","last_change":"2021-03-15T22:34:55.352256+0000","last_active":"2021-03-15T22:36:07.376577+0000","last_peered":"2021-03-15T22:36:07.376577+0000","last_clean":"2021-03-15T22:36:07.376577+0000","last_became_active":"2021-03-15T22:34:55.351597+0000","last_became_peered":"2021-03-15T22:34:55.351597+0000","last_unstale":"2021-03-15T22:36:07.376577+0000","last_undegraded":"2021-03-15T22:36:07.376577+0000","last_fullsized":"2021-03-15T22:36:07.376577+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:43.393275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:43.393275+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":"627","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359324+0000","last_change":"2021-03-15T22:35:50.357745+0000","last_active":"2021-03-15T22:36:10.359324+0000","last_peered":"2021-03-15T22:36:10.359324+0000","last_clean":"2021-03-15T22:36:10.359324+0000","last_became_active":"2021-03-15T22:35:50.357418+0000","last_became_peered":"2021-03-15T22:35:50.357418+0000","last_unstale":"2021-03-15T22:36:10.359324+0000","last_undegraded":"2021-03-15T22:36:10.359324+0000","last_fullsized":"2021-03-15T22:36:10.359324+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:41.082301+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.082301+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"617","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.385220+0000","last_change":"2021-03-15T22:34:10.385041+0000","last_active":"2021-03-15T22:36:09.385220+0000","last_peered":"2021-03-15T22:36:09.385220+0000","last_clean":"2021-03-15T22:36:09.385220+0000","last_became_active":"2021-03-15T22:34:10.347227+0000","last_became_peered":"2021-03-15T22:34:10.347227+0000","last_unstale":"2021-03-15T22:36:09.385220+0000","last_undegraded":"2021-03-15T22:36:09.385220+0000","last_fullsized":"2021-03-15T22:36:09.385220+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:34:07.125155+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.125155+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":3736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"1093","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384987+0000","last_change":"2021-03-15T22:34:58.356466+0000","last_active":"2021-03-15T22:36:09.384987+0000","last_peered":"2021-03-15T22:36:09.384987+0000","last_clean":"2021-03-15T22:36:09.384987+0000","last_became_active":"2021-03-15T22:34:58.353086+0000","last_became_peered":"2021-03-15T22:34:58.353086+0000","last_unstale":"2021-03-15T22:36:09.384987+0000","last_undegraded":"2021-03-15T22:36:09.384987+0000","last_fullsized":"2021-03-15T22:36:09.384987+0000","mapping_epoch":942,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:32:34.739637+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:32:34.739637+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"1020","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.372117+0000","last_change":"2021-03-15T22:35:18.111879+0000","last_active":"2021-03-15T22:36:05.372117+0000","last_peered":"2021-03-15T22:36:05.372117+0000","last_clean":"2021-03-15T22:36:05.372117+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:36:05.372117+0000","last_undegraded":"2021-03-15T22:36:05.372117+0000","last_fullsized":"2021-03-15T22:36:05.372117+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:18.111815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.111815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":"26","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.371692+0000","last_change":"2021-03-15T22:35:50.359970+0000","last_active":"2021-03-15T22:36:05.371692+0000","last_peered":"2021-03-15T22:36:05.371692+0000","last_clean":"2021-03-15T22:36:05.371692+0000","last_became_active":"2021-03-15T22:35:50.359484+0000","last_became_peered":"2021-03-15T22:35:50.359484+0000","last_unstale":"2021-03-15T22:36:05.371692+0000","last_undegraded":"2021-03-15T22:36:05.371692+0000","last_fullsized":"2021-03-15T22:36:05.371692+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.844183+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.844183+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"223","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.373276+0000","last_change":"2021-03-15T22:34:43.447677+0000","last_active":"2021-03-15T22:36:09.373276+0000","last_peered":"2021-03-15T22:36:09.373276+0000","last_clean":"2021-03-15T22:36:09.373276+0000","last_became_active":"2021-03-15T22:33:04.336100+0000","last_became_peered":"2021-03-15T22:33:04.336100+0000","last_unstale":"2021-03-15T22:36:09.373276+0000","last_undegraded":"2021-03-15T22:36:09.373276+0000","last_fullsized":"2021-03-15T22:36:09.373276+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:34:43.447602+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:43.447602+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"1031","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.376275+0000","last_change":"2021-03-15T22:35:41.589000+0000","last_active":"2021-03-15T22:36:07.376275+0000","last_peered":"2021-03-15T22:36:07.376275+0000","last_clean":"2021-03-15T22:36:07.376275+0000","last_became_active":"2021-03-15T22:34:55.351089+0000","last_became_peered":"2021-03-15T22:34:55.351089+0000","last_unstale":"2021-03-15T22:36:07.376275+0000","last_undegraded":"2021-03-15T22:36:07.376275+0000","last_fullsized":"2021-03-15T22:36:07.376275+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:41.588935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.588935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"199","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.372739+0000","last_change":"2021-03-15T22:35:22.665325+0000","last_active":"2021-03-15T22:36:09.372739+0000","last_peered":"2021-03-15T22:36:09.372739+0000","last_clean":"2021-03-15T22:36:09.372739+0000","last_became_active":"2021-03-15T22:33:04.335734+0000","last_became_peered":"2021-03-15T22:33:04.335734+0000","last_unstale":"2021-03-15T22:36:09.372739+0000","last_undegraded":"2021-03-15T22:36:09.372739+0000","last_fullsized":"2021-03-15T22:36:09.372739+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.665251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.665251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":"473","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.355082+0000","last_change":"2021-03-15T22:35:50.357197+0000","last_active":"2021-03-15T22:36:10.355082+0000","last_peered":"2021-03-15T22:36:10.355082+0000","last_clean":"2021-03-15T22:36:10.355082+0000","last_became_active":"2021-03-15T22:35:50.356790+0000","last_became_peered":"2021-03-15T22:35:50.356790+0000","last_unstale":"2021-03-15T22:36:10.355082+0000","last_undegraded":"2021-03-15T22:36:10.355082+0000","last_fullsized":"2021-03-15T22:36:10.355082+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.291406+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:22.291406+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.291406+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"528","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.362189+0000","last_change":"2021-03-15T22:33:37.752077+0000","last_active":"2021-03-15T22:36:10.362189+0000","last_peered":"2021-03-15T22:36:10.362189+0000","last_clean":"2021-03-15T22:36:10.362189+0000","last_became_active":"2021-03-15T22:33:04.342749+0000","last_became_peered":"2021-03-15T22:33:04.342749+0000","last_unstale":"2021-03-15T22:36:10.362189+0000","last_undegraded":"2021-03-15T22:36:10.362189+0000","last_fullsized":"2021-03-15T22:36:10.362189+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:33:37.752011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.752011+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":8764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"641","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.355392+0000","last_change":"2021-03-15T22:34:34.783107+0000","last_active":"2021-03-15T22:36:10.355392+0000","last_peered":"2021-03-15T22:36:10.355392+0000","last_clean":"2021-03-15T22:36:10.355392+0000","last_became_active":"2021-03-15T22:33:04.333077+0000","last_became_peered":"2021-03-15T22:33:04.333077+0000","last_unstale":"2021-03-15T22:36:10.355392+0000","last_undegraded":"2021-03-15T22:36:10.355392+0000","last_fullsized":"2021-03-15T22:36:10.355392+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.783047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.783047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"142","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.362330+0000","last_change":"2021-03-15T22:35:48.884382+0000","last_active":"2021-03-15T22:36:10.362330+0000","last_peered":"2021-03-15T22:36:10.362330+0000","last_clean":"2021-03-15T22:36:10.362330+0000","last_became_active":"2021-03-15T22:34:03.571049+0000","last_became_peered":"2021-03-15T22:34:03.571049+0000","last_unstale":"2021-03-15T22:36:10.362330+0000","last_undegraded":"2021-03-15T22:36:10.362330+0000","last_fullsized":"2021-03-15T22:36:10.362330+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:48.884317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:48.884317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":"30","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384843+0000","last_change":"2021-03-15T22:35:50.350018+0000","last_active":"2021-03-15T22:36:09.384843+0000","last_peered":"2021-03-15T22:36:09.384843+0000","last_clean":"2021-03-15T22:36:09.384843+0000","last_became_active":"2021-03-15T22:35:50.349656+0000","last_became_peered":"2021-03-15T22:35:50.349656+0000","last_unstale":"2021-03-15T22:36:09.384843+0000","last_undegraded":"2021-03-15T22:36:09.384843+0000","last_fullsized":"2021-03-15T22:36:09.384843+0000","mapping_epoch":994,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":995,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:12.882164+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:42.233098+0000","last_clean_scrub_stamp":"2021-03-15T22:33:12.882164+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"859","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.354993+0000","last_change":"2021-03-15T22:34:37.787302+0000","last_active":"2021-03-15T22:36:10.354993+0000","last_peered":"2021-03-15T22:36:10.354993+0000","last_clean":"2021-03-15T22:36:10.354993+0000","last_became_active":"2021-03-15T22:33:04.331607+0000","last_became_peered":"2021-03-15T22:33:04.331607+0000","last_unstale":"2021-03-15T22:36:10.354993+0000","last_undegraded":"2021-03-15T22:36:10.354993+0000","last_fullsized":"2021-03-15T22:36:10.354993+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:34:37.787235+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.787235+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"503","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384451+0000","last_change":"2021-03-15T22:35:17.250496+0000","last_active":"2021-03-15T22:36:09.384451+0000","last_peered":"2021-03-15T22:36:09.384451+0000","last_clean":"2021-03-15T22:36:09.384451+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:36:09.384451+0000","last_undegraded":"2021-03-15T22:36:09.384451+0000","last_fullsized":"2021-03-15T22:36:09.384451+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.250434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.250434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"808","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.355310+0000","last_change":"2021-03-15T22:32:37.690585+0000","last_active":"2021-03-15T22:36:10.355310+0000","last_peered":"2021-03-15T22:36:10.355310+0000","last_clean":"2021-03-15T22:36:10.355310+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:36:10.355310+0000","last_undegraded":"2021-03-15T22:36:10.355310+0000","last_fullsized":"2021-03-15T22:36:10.355310+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:37.690525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:32:37.690525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"1019","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377333+0000","last_change":"2021-03-15T22:35:37.543358+0000","last_active":"2021-03-15T22:36:07.377333+0000","last_peered":"2021-03-15T22:36:07.377333+0000","last_clean":"2021-03-15T22:36:07.377333+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:36:07.377333+0000","last_undegraded":"2021-03-15T22:36:07.377333+0000","last_fullsized":"2021-03-15T22:36:07.377333+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:37.543289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:35:37.543289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"730","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.372561+0000","last_change":"2021-03-15T22:34:56.210802+0000","last_active":"2021-03-15T22:36:09.372561+0000","last_peered":"2021-03-15T22:36:09.372561+0000","last_clean":"2021-03-15T22:36:09.372561+0000","last_became_active":"2021-03-15T22:34:55.344555+0000","last_became_peered":"2021-03-15T22:34:55.344555+0000","last_unstale":"2021-03-15T22:36:09.372561+0000","last_undegraded":"2021-03-15T22:36:09.372561+0000","last_fullsized":"2021-03-15T22:36:09.372561+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:34:42.447631+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.447631+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"400","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384427+0000","last_change":"2021-03-15T22:35:23.291568+0000","last_active":"2021-03-15T22:36:09.384427+0000","last_peered":"2021-03-15T22:36:09.384427+0000","last_clean":"2021-03-15T22:36:09.384427+0000","last_became_active":"2021-03-15T22:34:55.355119+0000","last_became_peered":"2021-03-15T22:34:55.355119+0000","last_unstale":"2021-03-15T22:36:09.384427+0000","last_undegraded":"2021-03-15T22:36:09.384427+0000","last_fullsized":"2021-03-15T22:36:09.384427+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.291507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.291507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"1073","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.370923+0000","last_change":"2021-03-15T22:35:50.069546+0000","last_active":"2021-03-15T22:36:05.370923+0000","last_peered":"2021-03-15T22:36:05.370923+0000","last_clean":"2021-03-15T22:36:05.370923+0000","last_became_active":"2021-03-15T22:34:10.353314+0000","last_became_peered":"2021-03-15T22:34:10.353314+0000","last_unstale":"2021-03-15T22:36:05.370923+0000","last_undegraded":"2021-03-15T22:36:05.370923+0000","last_fullsized":"2021-03-15T22:36:05.370923+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:50.069463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:35:50.069463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"151","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.358668+0000","last_change":"2021-03-15T22:33:45.314503+0000","last_active":"2021-03-15T22:36:10.358668+0000","last_peered":"2021-03-15T22:36:10.358668+0000","last_clean":"2021-03-15T22:36:10.358668+0000","last_became_active":"2021-03-15T22:33:45.314042+0000","last_became_peered":"2021-03-15T22:33:45.314042+0000","last_unstale":"2021-03-15T22:36:10.358668+0000","last_undegraded":"2021-03-15T22:36:10.358668+0000","last_fullsized":"2021-03-15T22:36:10.358668+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:08.884878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.884878+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"1102","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.358758+0000","last_change":"2021-03-15T22:34:42.632034+0000","last_active":"2021-03-15T22:36:10.358758+0000","last_peered":"2021-03-15T22:36:10.358758+0000","last_clean":"2021-03-15T22:36:10.358758+0000","last_became_active":"2021-03-15T22:33:04.340460+0000","last_became_peered":"2021-03-15T22:33:04.340460+0000","last_unstale":"2021-03-15T22:36:10.358758+0000","last_undegraded":"2021-03-15T22:36:10.358758+0000","last_fullsized":"2021-03-15T22:36:10.358758+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:34:42.631962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.631962+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"506","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.385421+0000","last_change":"2021-03-15T22:34:10.327350+0000","last_active":"2021-03-15T22:36:09.385421+0000","last_peered":"2021-03-15T22:36:09.385421+0000","last_clean":"2021-03-15T22:36:09.385421+0000","last_became_active":"2021-03-15T22:34:10.326881+0000","last_became_peered":"2021-03-15T22:34:10.326881+0000","last_unstale":"2021-03-15T22:36:09.385421+0000","last_undegraded":"2021-03-15T22:36:09.385421+0000","last_fullsized":"2021-03-15T22:36:09.385421+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:19.560387+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:19.560387+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"808'9","reported_seq":"474","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377603+0000","last_change":"2021-03-15T22:35:41.986386+0000","last_active":"2021-03-15T22:36:07.377603+0000","last_peered":"2021-03-15T22:36:07.377603+0000","last_clean":"2021-03-15T22:36:07.377603+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:36:07.377603+0000","last_undegraded":"2021-03-15T22:36:07.377603+0000","last_fullsized":"2021-03-15T22:36:07.377603+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9","last_scrub_stamp":"2021-03-15T22:35:41.986318+0000","last_deep_scrub":"808'9","last_deep_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.986318+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"1011","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.375146+0000","last_change":"2021-03-15T22:34:55.351414+0000","last_active":"2021-03-15T22:36:07.375146+0000","last_peered":"2021-03-15T22:36:07.375146+0000","last_clean":"2021-03-15T22:36:07.375146+0000","last_became_active":"2021-03-15T22:34:55.351042+0000","last_became_peered":"2021-03-15T22:34:55.351042+0000","last_unstale":"2021-03-15T22:36:07.375146+0000","last_undegraded":"2021-03-15T22:36:07.375146+0000","last_fullsized":"2021-03-15T22:36:07.375146+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:37.516668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.516668+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"844'33","reported_seq":"874","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.354907+0000","last_change":"2021-03-15T22:34:55.387712+0000","last_active":"2021-03-15T22:36:10.354907+0000","last_peered":"2021-03-15T22:36:10.354907+0000","last_clean":"2021-03-15T22:36:10.354907+0000","last_became_active":"2021-03-15T22:34:55.351721+0000","last_became_peered":"2021-03-15T22:34:55.351721+0000","last_unstale":"2021-03-15T22:36:10.354907+0000","last_undegraded":"2021-03-15T22:36:10.354907+0000","last_fullsized":"2021-03-15T22:36:10.354907+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"844'33","last_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_deep_scrub":"844'33","last_deep_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.622437+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1055","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359454+0000","last_change":"2021-03-15T22:35:40.412203+0000","last_active":"2021-03-15T22:36:10.359454+0000","last_peered":"2021-03-15T22:36:10.359454+0000","last_clean":"2021-03-15T22:36:10.359454+0000","last_became_active":"2021-03-15T22:34:55.340440+0000","last_became_peered":"2021-03-15T22:34:55.340440+0000","last_unstale":"2021-03-15T22:36:10.359454+0000","last_undegraded":"2021-03-15T22:36:10.359454+0000","last_fullsized":"2021-03-15T22:36:10.359454+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:40.412142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:40.412142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"941'17","reported_seq":"544","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.376537+0000","last_change":"2021-03-15T22:34:55.473459+0000","last_active":"2021-03-15T22:36:07.376537+0000","last_peered":"2021-03-15T22:36:07.376537+0000","last_clean":"2021-03-15T22:36:07.376537+0000","last_became_active":"2021-03-15T22:34:55.351682+0000","last_became_peered":"2021-03-15T22:34:55.351682+0000","last_unstale":"2021-03-15T22:36:07.376537+0000","last_undegraded":"2021-03-15T22:36:07.376537+0000","last_fullsized":"2021-03-15T22:36:07.376537+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"896'16","last_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_deep_scrub":"896'16","last_deep_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_clean_scrub_stamp":"2021-03-15T22:34:19.400191+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"511","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:10.121627+0000","last_change":"2021-03-15T22:36:10.121601+0000","last_active":"2021-03-15T22:36:10.121627+0000","last_peered":"2021-03-15T22:36:10.121627+0000","last_clean":"2021-03-15T22:36:10.121627+0000","last_became_active":"2021-03-15T22:34:10.347512+0000","last_became_peered":"2021-03-15T22:34:10.347512+0000","last_unstale":"2021-03-15T22:36:10.121627+0000","last_undegraded":"2021-03-15T22:36:10.121627+0000","last_fullsized":"2021-03-15T22:36:10.121627+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:10.121536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:36:10.121536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"406","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377497+0000","last_change":"2021-03-15T22:32:41.879032+0000","last_active":"2021-03-15T22:36:07.377497+0000","last_peered":"2021-03-15T22:36:07.377497+0000","last_clean":"2021-03-15T22:36:07.377497+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:36:07.377497+0000","last_undegraded":"2021-03-15T22:36:07.377497+0000","last_fullsized":"2021-03-15T22:36:07.377497+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:32:41.878949+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:32:41.878949+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"98","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.355393+0000","last_change":"2021-03-15T22:34:43.345092+0000","last_active":"2021-03-15T22:36:10.355393+0000","last_peered":"2021-03-15T22:36:10.355393+0000","last_clean":"2021-03-15T22:36:10.355393+0000","last_became_active":"2021-03-15T22:34:43.344790+0000","last_became_peered":"2021-03-15T22:34:43.344790+0000","last_unstale":"2021-03-15T22:36:10.355393+0000","last_undegraded":"2021-03-15T22:36:10.355393+0000","last_fullsized":"2021-03-15T22:36:10.355393+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.621539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"88","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.378051+0000","last_change":"2021-03-15T22:35:20.381407+0000","last_active":"2021-03-15T22:36:07.378051+0000","last_peered":"2021-03-15T22:36:07.378051+0000","last_clean":"2021-03-15T22:36:07.378051+0000","last_became_active":"2021-03-15T22:34:55.348663+0000","last_became_peered":"2021-03-15T22:34:55.348663+0000","last_unstale":"2021-03-15T22:36:07.378051+0000","last_undegraded":"2021-03-15T22:36:07.378051+0000","last_fullsized":"2021-03-15T22:36:07.378051+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.381352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"1009","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.374815+0000","last_change":"2021-03-15T22:35:48.581266+0000","last_active":"2021-03-15T22:36:07.374815+0000","last_peered":"2021-03-15T22:36:07.374815+0000","last_clean":"2021-03-15T22:36:07.374815+0000","last_became_active":"2021-03-15T22:33:04.330195+0000","last_became_peered":"2021-03-15T22:33:04.330195+0000","last_unstale":"2021-03-15T22:36:07.374815+0000","last_undegraded":"2021-03-15T22:36:07.374815+0000","last_fullsized":"2021-03-15T22:36:07.374815+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:48.581162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:48.581162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"542","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.385060+0000","last_change":"2021-03-15T22:34:37.435070+0000","last_active":"2021-03-15T22:36:09.385060+0000","last_peered":"2021-03-15T22:36:09.385060+0000","last_clean":"2021-03-15T22:36:09.385060+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:36:09.385060+0000","last_undegraded":"2021-03-15T22:36:09.385060+0000","last_fullsized":"2021-03-15T22:36:09.385060+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:34:37.435006+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.435006+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"815","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.378590+0000","last_change":"2021-03-15T22:35:20.884067+0000","last_active":"2021-03-15T22:36:07.378590+0000","last_peered":"2021-03-15T22:36:07.378590+0000","last_clean":"2021-03-15T22:36:07.378590+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:36:07.378590+0000","last_undegraded":"2021-03-15T22:36:07.378590+0000","last_fullsized":"2021-03-15T22:36:07.378590+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.883990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.883990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"1003","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377237+0000","last_change":"2021-03-15T22:35:45.547459+0000","last_active":"2021-03-15T22:36:07.377237+0000","last_peered":"2021-03-15T22:36:07.377237+0000","last_clean":"2021-03-15T22:36:07.377237+0000","last_became_active":"2021-03-15T22:33:04.331013+0000","last_became_peered":"2021-03-15T22:33:04.331013+0000","last_unstale":"2021-03-15T22:36:07.377237+0000","last_undegraded":"2021-03-15T22:36:07.377237+0000","last_fullsized":"2021-03-15T22:36:07.377237+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.547395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:45.547395+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.547395+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"199","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.372339+0000","last_change":"2021-03-15T22:35:45.757312+0000","last_active":"2021-03-15T22:36:09.372339+0000","last_peered":"2021-03-15T22:36:09.372339+0000","last_clean":"2021-03-15T22:36:09.372339+0000","last_became_active":"2021-03-15T22:33:04.336886+0000","last_became_peered":"2021-03-15T22:33:04.336886+0000","last_unstale":"2021-03-15T22:36:09.372339+0000","last_undegraded":"2021-03-15T22:36:09.372339+0000","last_fullsized":"2021-03-15T22:36:09.372339+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.757254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:45.757254+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.757254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"362","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377017+0000","last_change":"2021-03-15T22:34:55.387006+0000","last_active":"2021-03-15T22:36:07.377017+0000","last_peered":"2021-03-15T22:36:07.377017+0000","last_clean":"2021-03-15T22:36:07.377017+0000","last_became_active":"2021-03-15T22:34:55.351977+0000","last_became_peered":"2021-03-15T22:34:55.351977+0000","last_unstale":"2021-03-15T22:36:07.377017+0000","last_undegraded":"2021-03-15T22:36:07.377017+0000","last_fullsized":"2021-03-15T22:36:07.377017+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:34:36.438541+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:34:36.438541+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"1011'1445","reported_seq":"4579","reported_epoch":"1011","state":"active+clean","last_fresh":"2021-03-15T22:36:06.393490+0000","last_change":"2021-03-15T22:36:06.394374+0000","last_active":"2021-03-15T22:36:06.393490+0000","last_peered":"2021-03-15T22:36:06.393490+0000","last_clean":"2021-03-15T22:36:06.393490+0000","last_became_active":"2021-03-15T22:33:08.355490+0000","last_became_peered":"2021-03-15T22:33:08.355490+0000","last_unstale":"2021-03-15T22:36:06.393490+0000","last_undegraded":"2021-03-15T22:36:06.393490+0000","last_fullsized":"2021-03-15T22:36:06.393490+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'1350","last_scrub_stamp":"2021-03-15T22:35:16.036089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.036089+0000","log_size":1445,"ondisk_log_size":1445,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52931112,"num_objects":24,"num_object_clones":21,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":491,"num_read_kb":544282,"num_write":345,"num_write_kb":121375,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":84869361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"339","length":"1"},{"start":"359","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"955","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.378583+0000","last_change":"2021-03-15T22:33:42.741529+0000","last_active":"2021-03-15T22:36:07.378583+0000","last_peered":"2021-03-15T22:36:07.378583+0000","last_clean":"2021-03-15T22:36:07.378583+0000","last_became_active":"2021-03-15T22:33:04.340352+0000","last_became_peered":"2021-03-15T22:33:04.340352+0000","last_unstale":"2021-03-15T22:36:07.378583+0000","last_undegraded":"2021-03-15T22:36:07.378583+0000","last_fullsized":"2021-03-15T22:36:07.378583+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:42.741451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:42.741451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"874'4","reported_seq":"1039","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377015+0000","last_change":"2021-03-15T22:35:58.361251+0000","last_active":"2021-03-15T22:36:07.377015+0000","last_peered":"2021-03-15T22:36:07.377015+0000","last_clean":"2021-03-15T22:36:07.377015+0000","last_became_active":"2021-03-15T22:35:58.360849+0000","last_became_peered":"2021-03-15T22:35:58.360849+0000","last_unstale":"2021-03-15T22:36:07.377015+0000","last_undegraded":"2021-03-15T22:36:07.377015+0000","last_fullsized":"2021-03-15T22:36:07.377015+0000","mapping_epoch":552,"log_start":"874'4","ondisk_log_start":"874'4","created":56,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"874'4","last_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_deep_scrub":"874'4","last_deep_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.498263+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":965,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"482","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.379613+0000","last_change":"2021-03-15T22:33:38.759477+0000","last_active":"2021-03-15T22:36:07.379613+0000","last_peered":"2021-03-15T22:36:07.379613+0000","last_clean":"2021-03-15T22:36:07.379613+0000","last_became_active":"2021-03-15T22:33:04.326070+0000","last_became_peered":"2021-03-15T22:33:04.326070+0000","last_unstale":"2021-03-15T22:36:07.379613+0000","last_undegraded":"2021-03-15T22:36:07.379613+0000","last_fullsized":"2021-03-15T22:36:07.379613+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:33:38.759395+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.759395+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"841'14","reported_seq":"1285","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.374911+0000","last_change":"2021-03-15T22:33:43.476783+0000","last_active":"2021-03-15T22:36:07.374911+0000","last_peered":"2021-03-15T22:36:07.374911+0000","last_clean":"2021-03-15T22:36:07.374911+0000","last_became_active":"2021-03-15T22:33:08.354031+0000","last_became_peered":"2021-03-15T22:33:08.354031+0000","last_unstale":"2021-03-15T22:36:07.374911+0000","last_undegraded":"2021-03-15T22:36:07.374911+0000","last_fullsized":"2021-03-15T22:36:07.374911+0000","mapping_epoch":837,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'14","last_scrub_stamp":"2021-03-15T22:33:43.476701+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.476701+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"1016'2115","reported_seq":"5534","reported_epoch":"1016","state":"active+clean","last_fresh":"2021-03-15T22:36:11.407798+0000","last_change":"2021-03-15T22:36:11.407965+0000","last_active":"2021-03-15T22:36:11.407798+0000","last_peered":"2021-03-15T22:36:11.407798+0000","last_clean":"2021-03-15T22:36:11.407798+0000","last_became_active":"2021-03-15T22:34:58.336162+0000","last_became_peered":"2021-03-15T22:34:58.336162+0000","last_unstale":"2021-03-15T22:36:11.407798+0000","last_undegraded":"2021-03-15T22:36:11.407798+0000","last_fullsized":"2021-03-15T22:36:11.407798+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'2031","last_scrub_stamp":"2021-03-15T22:35:42.023328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.023328+0000","log_size":2115,"ondisk_log_size":2115,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":83943366,"num_objects":39,"num_object_clones":34,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":1,"num_read":700,"num_read_kb":654068,"num_write":548,"num_write_kb":211767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":198498497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"356","length":"1"},{"start":"35a","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"444","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.355172+0000","last_change":"2021-03-15T22:32:42.621654+0000","last_active":"2021-03-15T22:36:10.355172+0000","last_peered":"2021-03-15T22:36:10.355172+0000","last_clean":"2021-03-15T22:36:10.355172+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:36:10.355172+0000","last_undegraded":"2021-03-15T22:36:10.355172+0000","last_fullsized":"2021-03-15T22:36:10.355172+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:42.621594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:42.621594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"1011","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.378030+0000","last_change":"2021-03-15T22:36:02.374943+0000","last_active":"2021-03-15T22:36:07.378030+0000","last_peered":"2021-03-15T22:36:07.378030+0000","last_clean":"2021-03-15T22:36:07.378030+0000","last_became_active":"2021-03-15T22:36:02.374583+0000","last_became_peered":"2021-03-15T22:36:02.374583+0000","last_unstale":"2021-03-15T22:36:07.378030+0000","last_undegraded":"2021-03-15T22:36:07.378030+0000","last_fullsized":"2021-03-15T22:36:07.378030+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1007,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:47.601457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.601457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"1225","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359425+0000","last_change":"2021-03-15T22:34:11.031994+0000","last_active":"2021-03-15T22:36:10.359425+0000","last_peered":"2021-03-15T22:36:10.359425+0000","last_clean":"2021-03-15T22:36:10.359425+0000","last_became_active":"2021-03-15T22:34:10.342120+0000","last_became_peered":"2021-03-15T22:34:10.342120+0000","last_unstale":"2021-03-15T22:36:10.359425+0000","last_undegraded":"2021-03-15T22:36:10.359425+0000","last_fullsized":"2021-03-15T22:36:10.359425+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:34:07.452572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.452572+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"727","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384511+0000","last_change":"2021-03-15T22:34:55.356651+0000","last_active":"2021-03-15T22:36:09.384511+0000","last_peered":"2021-03-15T22:36:09.384511+0000","last_clean":"2021-03-15T22:36:09.384511+0000","last_became_active":"2021-03-15T22:34:55.356328+0000","last_became_peered":"2021-03-15T22:34:55.356328+0000","last_unstale":"2021-03-15T22:36:09.384511+0000","last_undegraded":"2021-03-15T22:36:09.384511+0000","last_fullsized":"2021-03-15T22:36:09.384511+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"1013'576","reported_seq":"3437","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.354458+0000","last_change":"2021-03-15T22:36:10.354752+0000","last_active":"2021-03-15T22:36:10.354458+0000","last_peered":"2021-03-15T22:36:10.354458+0000","last_clean":"2021-03-15T22:36:10.354458+0000","last_became_active":"2021-03-15T22:34:13.324667+0000","last_became_peered":"2021-03-15T22:34:13.324667+0000","last_unstale":"2021-03-15T22:36:10.354458+0000","last_undegraded":"2021-03-15T22:36:10.354458+0000","last_fullsized":"2021-03-15T22:36:10.354458+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'446","last_scrub_stamp":"2021-03-15T22:33:39.355306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.355306+0000","log_size":576,"ondisk_log_size":576,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":17801208,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":139,"num_read_kb":111592,"num_write":131,"num_write_kb":42276,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":23698344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"356","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"1074","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.374558+0000","last_change":"2021-03-15T22:35:48.365854+0000","last_active":"2021-03-15T22:36:07.374558+0000","last_peered":"2021-03-15T22:36:07.374558+0000","last_clean":"2021-03-15T22:36:07.374558+0000","last_became_active":"2021-03-15T22:35:48.365533+0000","last_became_peered":"2021-03-15T22:35:48.365533+0000","last_unstale":"2021-03-15T22:36:07.374558+0000","last_undegraded":"2021-03-15T22:36:07.374558+0000","last_fullsized":"2021-03-15T22:36:07.374558+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":993,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:31.390348+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.390348+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"88","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377534+0000","last_change":"2021-03-15T22:35:49.448634+0000","last_active":"2021-03-15T22:36:07.377534+0000","last_peered":"2021-03-15T22:36:07.377534+0000","last_clean":"2021-03-15T22:36:07.377534+0000","last_became_active":"2021-03-15T22:34:55.353966+0000","last_became_peered":"2021-03-15T22:34:55.353966+0000","last_unstale":"2021-03-15T22:36:07.377534+0000","last_undegraded":"2021-03-15T22:36:07.377534+0000","last_fullsized":"2021-03-15T22:36:07.377534+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.448581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.448581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"1066","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377548+0000","last_change":"2021-03-15T22:36:06.382789+0000","last_active":"2021-03-15T22:36:07.377548+0000","last_peered":"2021-03-15T22:36:07.377548+0000","last_clean":"2021-03-15T22:36:07.377548+0000","last_became_active":"2021-03-15T22:36:06.382477+0000","last_became_peered":"2021-03-15T22:36:06.382477+0000","last_unstale":"2021-03-15T22:36:07.377548+0000","last_undegraded":"2021-03-15T22:36:07.377548+0000","last_fullsized":"2021-03-15T22:36:07.377548+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1011,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.428842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.428842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"264","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.372843+0000","last_change":"2021-03-15T22:35:47.782159+0000","last_active":"2021-03-15T22:36:09.372843+0000","last_peered":"2021-03-15T22:36:09.372843+0000","last_clean":"2021-03-15T22:36:09.372843+0000","last_became_active":"2021-03-15T22:33:45.318867+0000","last_became_peered":"2021-03-15T22:33:45.318867+0000","last_unstale":"2021-03-15T22:36:09.372843+0000","last_undegraded":"2021-03-15T22:36:09.372843+0000","last_fullsized":"2021-03-15T22:36:09.372843+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:35:47.782092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.782092+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":2928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"1013'1284","reported_seq":"4214","reported_epoch":"1013","state":"active+clean","last_fresh":"2021-03-15T22:36:08.389591+0000","last_change":"2021-03-15T22:36:08.389841+0000","last_active":"2021-03-15T22:36:08.389591+0000","last_peered":"2021-03-15T22:36:08.389591+0000","last_clean":"2021-03-15T22:36:08.389591+0000","last_became_active":"2021-03-15T22:34:59.339587+0000","last_became_peered":"2021-03-15T22:34:59.339587+0000","last_unstale":"2021-03-15T22:36:08.389591+0000","last_undegraded":"2021-03-15T22:36:08.389591+0000","last_fullsized":"2021-03-15T22:36:08.389591+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'1086","last_scrub_stamp":"2021-03-15T22:34:41.436839+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.436839+0000","log_size":1284,"ondisk_log_size":1284,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52552469,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":428,"num_read_kb":511500,"num_write":293,"num_write_kb":105996,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":205148835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"359","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"535","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384453+0000","last_change":"2021-03-15T22:35:44.582634+0000","last_active":"2021-03-15T22:36:09.384453+0000","last_peered":"2021-03-15T22:36:09.384453+0000","last_clean":"2021-03-15T22:36:09.384453+0000","last_became_active":"2021-03-15T22:35:44.582322+0000","last_became_peered":"2021-03-15T22:35:44.582322+0000","last_unstale":"2021-03-15T22:36:09.384453+0000","last_undegraded":"2021-03-15T22:36:09.384453+0000","last_fullsized":"2021-03-15T22:36:09.384453+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"849","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359118+0000","last_change":"2021-03-15T22:34:55.343547+0000","last_active":"2021-03-15T22:36:10.359118+0000","last_peered":"2021-03-15T22:36:10.359118+0000","last_clean":"2021-03-15T22:36:10.359118+0000","last_became_active":"2021-03-15T22:34:55.342808+0000","last_became_peered":"2021-03-15T22:34:55.342808+0000","last_unstale":"2021-03-15T22:36:10.359118+0000","last_undegraded":"2021-03-15T22:36:10.359118+0000","last_fullsized":"2021-03-15T22:36:10.359118+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:14.888119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.888119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"422","reported_epoch":"1013","state":"clean+premerge+peered","last_fresh":"2021-03-15T22:36:08.390376+0000","last_change":"2021-03-15T22:36:08.390376+0000","last_active":"2021-03-15T22:36:07.379966+0000","last_peered":"2021-03-15T22:36:08.390376+0000","last_clean":"2021-03-15T22:36:08.390376+0000","last_became_active":"2021-03-15T22:33:04.333291+0000","last_became_peered":"2021-03-15T22:36:08.389968+0000","last_unstale":"2021-03-15T22:36:08.390376+0000","last_undegraded":"2021-03-15T22:36:08.390376+0000","last_fullsized":"2021-03-15T22:36:08.390376+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.807411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.807411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"557","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384870+0000","last_change":"2021-03-15T22:33:46.129541+0000","last_active":"2021-03-15T22:36:09.384870+0000","last_peered":"2021-03-15T22:36:09.384870+0000","last_clean":"2021-03-15T22:36:09.384870+0000","last_became_active":"2021-03-15T22:33:45.326747+0000","last_became_peered":"2021-03-15T22:33:45.326747+0000","last_unstale":"2021-03-15T22:36:09.384870+0000","last_undegraded":"2021-03-15T22:36:09.384870+0000","last_fullsized":"2021-03-15T22:36:09.384870+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:32:33.701675+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.701675+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":2682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.7","version":"1015'1463","reported_seq":"5080","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.380560+0000","last_change":"2021-03-15T22:36:10.380930+0000","last_active":"2021-03-15T22:36:10.380560+0000","last_peered":"2021-03-15T22:36:10.380560+0000","last_clean":"2021-03-15T22:36:10.380560+0000","last_became_active":"2021-03-15T22:34:46.338888+0000","last_became_peered":"2021-03-15T22:34:46.338888+0000","last_unstale":"2021-03-15T22:36:10.380560+0000","last_undegraded":"2021-03-15T22:36:10.380560+0000","last_fullsized":"2021-03-15T22:36:10.380560+0000","mapping_epoch":930,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"926'1253","last_scrub_stamp":"2021-03-15T22:34:41.490879+0000","last_deep_scrub":"212'88","last_deep_scrub_stamp":"2021-03-15T22:22:07.718413+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.490879+0000","log_size":1463,"ondisk_log_size":1463,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52089480,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":344,"num_read_kb":290241,"num_write":358,"num_write_kb":127337,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":229919087,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"356","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"116","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.354643+0000","last_change":"2021-03-15T22:35:56.358841+0000","last_active":"2021-03-15T22:36:10.354643+0000","last_peered":"2021-03-15T22:36:10.354643+0000","last_clean":"2021-03-15T22:36:10.354643+0000","last_became_active":"2021-03-15T22:35:56.358504+0000","last_became_peered":"2021-03-15T22:35:56.358504+0000","last_unstale":"2021-03-15T22:36:10.354643+0000","last_undegraded":"2021-03-15T22:36:10.354643+0000","last_fullsized":"2021-03-15T22:36:10.354643+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1001,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"59","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.375978+0000","last_change":"2021-03-15T22:35:19.502186+0000","last_active":"2021-03-15T22:36:07.375978+0000","last_peered":"2021-03-15T22:36:07.375978+0000","last_clean":"2021-03-15T22:36:07.375978+0000","last_became_active":"2021-03-15T22:35:19.501783+0000","last_became_peered":"2021-03-15T22:35:19.501783+0000","last_unstale":"2021-03-15T22:36:07.375978+0000","last_undegraded":"2021-03-15T22:36:07.375978+0000","last_fullsized":"2021-03-15T22:36:07.375978+0000","mapping_epoch":963,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":964,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.831369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.831369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"213","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.358626+0000","last_change":"2021-03-15T22:33:04.336661+0000","last_active":"2021-03-15T22:36:10.358626+0000","last_peered":"2021-03-15T22:36:10.358626+0000","last_clean":"2021-03-15T22:36:10.358626+0000","last_became_active":"2021-03-15T22:33:04.335914+0000","last_became_peered":"2021-03-15T22:33:04.335914+0000","last_unstale":"2021-03-15T22:36:10.358626+0000","last_undegraded":"2021-03-15T22:36:10.358626+0000","last_fullsized":"2021-03-15T22:36:10.358626+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.051962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.051962+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1383","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.373900+0000","last_change":"2021-03-15T22:35:47.126971+0000","last_active":"2021-03-15T22:36:05.373900+0000","last_peered":"2021-03-15T22:36:05.373900+0000","last_clean":"2021-03-15T22:36:05.373900+0000","last_became_active":"2021-03-15T22:34:55.350419+0000","last_became_peered":"2021-03-15T22:34:55.350419+0000","last_unstale":"2021-03-15T22:36:05.373900+0000","last_undegraded":"2021-03-15T22:36:05.373900+0000","last_fullsized":"2021-03-15T22:36:05.373900+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:35:47.126904+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.126904+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"479","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384835+0000","last_change":"2021-03-15T22:33:15.492116+0000","last_active":"2021-03-15T22:36:09.384835+0000","last_peered":"2021-03-15T22:36:09.384835+0000","last_clean":"2021-03-15T22:36:09.384835+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:36:09.384835+0000","last_undegraded":"2021-03-15T22:36:09.384835+0000","last_fullsized":"2021-03-15T22:36:09.384835+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.492014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.492014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.4","version":"1015'1375","reported_seq":"4106","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.373137+0000","last_change":"2021-03-15T22:36:10.376508+0000","last_active":"2021-03-15T22:36:10.373137+0000","last_peered":"2021-03-15T22:36:10.373137+0000","last_clean":"2021-03-15T22:36:10.373137+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:36:10.373137+0000","last_undegraded":"2021-03-15T22:36:10.373137+0000","last_fullsized":"2021-03-15T22:36:10.373137+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"805'997","last_scrub_stamp":"2021-03-15T22:32:36.795301+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:32:36.795301+0000","log_size":1375,"ondisk_log_size":1375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57636908,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":411,"num_read_kb":414806,"num_write":316,"num_write_kb":120781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"356","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"1062","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377713+0000","last_change":"2021-03-15T22:36:00.364664+0000","last_active":"2021-03-15T22:36:07.377713+0000","last_peered":"2021-03-15T22:36:07.377713+0000","last_clean":"2021-03-15T22:36:07.377713+0000","last_became_active":"2021-03-15T22:36:00.364246+0000","last_became_peered":"2021-03-15T22:36:00.364246+0000","last_unstale":"2021-03-15T22:36:07.377713+0000","last_undegraded":"2021-03-15T22:36:07.377713+0000","last_fullsized":"2021-03-15T22:36:07.377713+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1005,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:44.582515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:44.582515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"1042","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.372183+0000","last_change":"2021-03-15T22:34:55.347340+0000","last_active":"2021-03-15T22:36:05.372183+0000","last_peered":"2021-03-15T22:36:05.372183+0000","last_clean":"2021-03-15T22:36:05.372183+0000","last_became_active":"2021-03-15T22:34:55.347065+0000","last_became_peered":"2021-03-15T22:34:55.347065+0000","last_unstale":"2021-03-15T22:36:05.372183+0000","last_undegraded":"2021-03-15T22:36:05.372183+0000","last_fullsized":"2021-03-15T22:36:05.372183+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:10.222521+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:10.222521+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"88","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377725+0000","last_change":"2021-03-15T22:35:17.301942+0000","last_active":"2021-03-15T22:36:07.377725+0000","last_peered":"2021-03-15T22:36:07.377725+0000","last_clean":"2021-03-15T22:36:07.377725+0000","last_became_active":"2021-03-15T22:34:55.355485+0000","last_became_peered":"2021-03-15T22:34:55.355485+0000","last_unstale":"2021-03-15T22:36:07.377725+0000","last_undegraded":"2021-03-15T22:36:07.377725+0000","last_fullsized":"2021-03-15T22:36:07.377725+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.301892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.301892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"496","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377585+0000","last_change":"2021-03-15T22:34:10.343282+0000","last_active":"2021-03-15T22:36:07.377585+0000","last_peered":"2021-03-15T22:36:07.377585+0000","last_clean":"2021-03-15T22:36:07.377585+0000","last_became_active":"2021-03-15T22:34:10.322666+0000","last_became_peered":"2021-03-15T22:34:10.322666+0000","last_unstale":"2021-03-15T22:36:07.377585+0000","last_undegraded":"2021-03-15T22:36:07.377585+0000","last_fullsized":"2021-03-15T22:36:07.377585+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:33:40.214707+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.214707+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"88","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.378139+0000","last_change":"2021-03-15T22:35:45.355712+0000","last_active":"2021-03-15T22:36:07.378139+0000","last_peered":"2021-03-15T22:36:07.378139+0000","last_clean":"2021-03-15T22:36:07.378139+0000","last_became_active":"2021-03-15T22:34:55.353316+0000","last_became_peered":"2021-03-15T22:34:55.353316+0000","last_unstale":"2021-03-15T22:36:07.378139+0000","last_undegraded":"2021-03-15T22:36:07.378139+0000","last_fullsized":"2021-03-15T22:36:07.378139+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.355647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.355647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"1015'3286","reported_seq":"6884","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.387967+0000","last_change":"2021-03-15T22:36:10.388226+0000","last_active":"2021-03-15T22:36:10.387967+0000","last_peered":"2021-03-15T22:36:10.387967+0000","last_clean":"2021-03-15T22:36:10.387967+0000","last_became_active":"2021-03-15T22:34:13.331560+0000","last_became_peered":"2021-03-15T22:34:13.331560+0000","last_unstale":"2021-03-15T22:36:10.387967+0000","last_undegraded":"2021-03-15T22:36:10.387967+0000","last_fullsized":"2021-03-15T22:36:10.387967+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'3208","last_scrub_stamp":"2021-03-15T22:35:42.271839+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.271839+0000","log_size":3286,"ondisk_log_size":3286,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":95761605,"num_objects":44,"num_object_clones":37,"num_object_copies":132,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":44,"num_whiteouts":2,"num_read":936,"num_read_kb":984359,"num_write":688,"num_write_kb":279967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":264088494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"356","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"415","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377701+0000","last_change":"2021-03-15T22:35:52.857478+0000","last_active":"2021-03-15T22:36:07.377701+0000","last_peered":"2021-03-15T22:36:07.377701+0000","last_clean":"2021-03-15T22:36:07.377701+0000","last_became_active":"2021-03-15T22:35:52.857192+0000","last_became_peered":"2021-03-15T22:35:52.857192+0000","last_unstale":"2021-03-15T22:36:07.377701+0000","last_undegraded":"2021-03-15T22:36:07.377701+0000","last_fullsized":"2021-03-15T22:36:07.377701+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":997,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.761137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"1027","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.374641+0000","last_change":"2021-03-15T22:35:19.456953+0000","last_active":"2021-03-15T22:36:07.374641+0000","last_peered":"2021-03-15T22:36:07.374641+0000","last_clean":"2021-03-15T22:36:07.374641+0000","last_became_active":"2021-03-15T22:33:04.337709+0000","last_became_peered":"2021-03-15T22:33:04.337709+0000","last_unstale":"2021-03-15T22:36:07.374641+0000","last_undegraded":"2021-03-15T22:36:07.374641+0000","last_fullsized":"2021-03-15T22:36:07.374641+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"98","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359603+0000","last_change":"2021-03-15T22:34:43.341562+0000","last_active":"2021-03-15T22:36:10.359603+0000","last_peered":"2021-03-15T22:36:10.359603+0000","last_clean":"2021-03-15T22:36:10.359603+0000","last_became_active":"2021-03-15T22:34:43.340955+0000","last_became_peered":"2021-03-15T22:34:43.340955+0000","last_unstale":"2021-03-15T22:36:10.359603+0000","last_undegraded":"2021-03-15T22:36:10.359603+0000","last_fullsized":"2021-03-15T22:36:10.359603+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.843478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.843478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"214","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.373001+0000","last_change":"2021-03-15T22:34:09.215097+0000","last_active":"2021-03-15T22:36:09.373001+0000","last_peered":"2021-03-15T22:36:09.373001+0000","last_clean":"2021-03-15T22:36:09.373001+0000","last_became_active":"2021-03-15T22:33:04.330531+0000","last_became_peered":"2021-03-15T22:33:04.330531+0000","last_unstale":"2021-03-15T22:36:09.373001+0000","last_undegraded":"2021-03-15T22:36:09.373001+0000","last_fullsized":"2021-03-15T22:36:09.373001+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:34:09.215034+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:09.215034+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"1048","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.376628+0000","last_change":"2021-03-15T22:35:22.409047+0000","last_active":"2021-03-15T22:36:07.376628+0000","last_peered":"2021-03-15T22:36:07.376628+0000","last_clean":"2021-03-15T22:36:07.376628+0000","last_became_active":"2021-03-15T22:34:55.352186+0000","last_became_peered":"2021-03-15T22:34:55.352186+0000","last_unstale":"2021-03-15T22:36:07.376628+0000","last_undegraded":"2021-03-15T22:36:07.376628+0000","last_fullsized":"2021-03-15T22:36:07.376628+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.408965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.408965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"1013'1010","reported_seq":"4148","reported_epoch":"1013","state":"active+clean","last_fresh":"2021-03-15T22:36:08.389672+0000","last_change":"2021-03-15T22:36:08.389958+0000","last_active":"2021-03-15T22:36:08.389672+0000","last_peered":"2021-03-15T22:36:08.389672+0000","last_clean":"2021-03-15T22:36:08.389672+0000","last_became_active":"2021-03-15T22:35:03.346751+0000","last_became_peered":"2021-03-15T22:35:03.346751+0000","last_unstale":"2021-03-15T22:36:08.389672+0000","last_undegraded":"2021-03-15T22:36:08.389672+0000","last_fullsized":"2021-03-15T22:36:08.389672+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":0,"last_scrub":"891'819","last_scrub_stamp":"2021-03-15T22:34:08.508824+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.508824+0000","log_size":1010,"ondisk_log_size":1010,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34622938,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":2,"num_read":267,"num_read_kb":273197,"num_write":227,"num_write_kb":99079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":109648904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"359","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"832","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.380113+0000","last_change":"2021-03-15T22:36:04.904466+0000","last_active":"2021-03-15T22:36:07.380113+0000","last_peered":"2021-03-15T22:36:07.380113+0000","last_clean":"2021-03-15T22:36:07.380113+0000","last_became_active":"2021-03-15T22:36:04.904112+0000","last_became_peered":"2021-03-15T22:36:04.904112+0000","last_unstale":"2021-03-15T22:36:07.380113+0000","last_undegraded":"2021-03-15T22:36:07.380113+0000","last_fullsized":"2021-03-15T22:36:07.380113+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"511","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384553+0000","last_change":"2021-03-15T22:34:35.410541+0000","last_active":"2021-03-15T22:36:09.384553+0000","last_peered":"2021-03-15T22:36:09.384553+0000","last_clean":"2021-03-15T22:36:09.384553+0000","last_became_active":"2021-03-15T22:34:10.321100+0000","last_became_peered":"2021-03-15T22:34:10.321100+0000","last_unstale":"2021-03-15T22:36:09.384553+0000","last_undegraded":"2021-03-15T22:36:09.384553+0000","last_fullsized":"2021-03-15T22:36:09.384553+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:35.410485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:35.410485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"2101","reported_epoch":"1013","state":"active+clean","last_fresh":"2021-03-15T22:36:08.378643+0000","last_change":"2021-03-15T22:35:10.452906+0000","last_active":"2021-03-15T22:36:08.378643+0000","last_peered":"2021-03-15T22:36:08.378643+0000","last_clean":"2021-03-15T22:36:08.378643+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:36:08.378643+0000","last_undegraded":"2021-03-15T22:36:08.378643+0000","last_fullsized":"2021-03-15T22:36:08.378643+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:35:10.452809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.452809+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1079,"num_read_kb":1074,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"609","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.385291+0000","last_change":"2021-03-15T22:34:55.414460+0000","last_active":"2021-03-15T22:36:09.385291+0000","last_peered":"2021-03-15T22:36:09.385291+0000","last_clean":"2021-03-15T22:36:09.385291+0000","last_became_active":"2021-03-15T22:34:55.359023+0000","last_became_peered":"2021-03-15T22:34:55.359023+0000","last_unstale":"2021-03-15T22:36:09.385291+0000","last_undegraded":"2021-03-15T22:36:09.385291+0000","last_fullsized":"2021-03-15T22:36:09.385291+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:34:40.472800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.472800+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"402","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.378251+0000","last_change":"2021-03-15T22:34:40.799217+0000","last_active":"2021-03-15T22:36:07.378251+0000","last_peered":"2021-03-15T22:36:07.378251+0000","last_clean":"2021-03-15T22:36:07.378251+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:36:07.378251+0000","last_undegraded":"2021-03-15T22:36:07.378251+0000","last_fullsized":"2021-03-15T22:36:07.378251+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.799153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.799153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"1013'3406","reported_seq":"7533","reported_epoch":"1013","state":"active+clean","last_fresh":"2021-03-15T22:36:08.402897+0000","last_change":"2021-03-15T22:36:08.403119+0000","last_active":"2021-03-15T22:36:08.402897+0000","last_peered":"2021-03-15T22:36:08.402897+0000","last_clean":"2021-03-15T22:36:08.402897+0000","last_became_active":"2021-03-15T22:35:17.487261+0000","last_became_peered":"2021-03-15T22:35:17.487261+0000","last_unstale":"2021-03-15T22:36:08.402897+0000","last_undegraded":"2021-03-15T22:36:08.402897+0000","last_fullsized":"2021-03-15T22:36:08.402897+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":0,"last_scrub":"800'2597","last_scrub_stamp":"2021-03-15T22:32:30.909661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:32:30.909661+0000","log_size":3406,"ondisk_log_size":3406,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":118602411,"num_objects":49,"num_object_clones":42,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":1,"num_read":1254,"num_read_kb":1338011,"num_write":792,"num_write_kb":333529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":721,"num_bytes_recovered":351021959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"359","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"1047","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.370495+0000","last_change":"2021-03-15T22:35:49.070877+0000","last_active":"2021-03-15T22:36:05.370495+0000","last_peered":"2021-03-15T22:36:05.370495+0000","last_clean":"2021-03-15T22:36:05.370495+0000","last_became_active":"2021-03-15T22:34:55.345948+0000","last_became_peered":"2021-03-15T22:34:55.345948+0000","last_unstale":"2021-03-15T22:36:05.370495+0000","last_undegraded":"2021-03-15T22:36:05.370495+0000","last_fullsized":"2021-03-15T22:36:05.370495+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.070798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.070798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"1060","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.373835+0000","last_change":"2021-03-15T22:34:55.346381+0000","last_active":"2021-03-15T22:36:05.373835+0000","last_peered":"2021-03-15T22:36:05.373835+0000","last_clean":"2021-03-15T22:36:05.373835+0000","last_became_active":"2021-03-15T22:34:55.346048+0000","last_became_peered":"2021-03-15T22:34:55.346048+0000","last_unstale":"2021-03-15T22:36:05.373835+0000","last_undegraded":"2021-03-15T22:36:05.373835+0000","last_fullsized":"2021-03-15T22:36:05.373835+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:48.037748+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_clean_scrub_stamp":"2021-03-15T22:34:48.037748+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"559","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384785+0000","last_change":"2021-03-15T22:35:21.314851+0000","last_active":"2021-03-15T22:36:09.384785+0000","last_peered":"2021-03-15T22:36:09.384785+0000","last_clean":"2021-03-15T22:36:09.384785+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:36:09.384785+0000","last_undegraded":"2021-03-15T22:36:09.384785+0000","last_fullsized":"2021-03-15T22:36:09.384785+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:35:21.314763+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.314763+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"199","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.372659+0000","last_change":"2021-03-15T22:35:39.707993+0000","last_active":"2021-03-15T22:36:09.372659+0000","last_peered":"2021-03-15T22:36:09.372659+0000","last_clean":"2021-03-15T22:36:09.372659+0000","last_became_active":"2021-03-15T22:33:04.337244+0000","last_became_peered":"2021-03-15T22:33:04.337244+0000","last_unstale":"2021-03-15T22:36:09.372659+0000","last_undegraded":"2021-03-15T22:36:09.372659+0000","last_fullsized":"2021-03-15T22:36:09.372659+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:39.707900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:35:39.707900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"1016'592","reported_seq":"3754","reported_epoch":"1016","state":"active+clean","last_fresh":"2021-03-15T22:36:11.376415+0000","last_change":"2021-03-15T22:36:11.376692+0000","last_active":"2021-03-15T22:36:11.376415+0000","last_peered":"2021-03-15T22:36:11.376415+0000","last_clean":"2021-03-15T22:36:11.376415+0000","last_became_active":"2021-03-15T22:34:15.498612+0000","last_became_peered":"2021-03-15T22:34:15.498612+0000","last_unstale":"2021-03-15T22:36:11.376415+0000","last_undegraded":"2021-03-15T22:36:11.376415+0000","last_fullsized":"2021-03-15T22:36:11.376415+0000","mapping_epoch":899,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":900,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'463","last_scrub_stamp":"2021-03-15T22:33:40.563144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.563144+0000","log_size":592,"ondisk_log_size":592,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":25315934,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":183,"num_read_kb":221864,"num_write":111,"num_write_kb":58078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":84391207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"356","length":"1"},{"start":"35a","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"535","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384431+0000","last_change":"2021-03-15T22:34:55.356457+0000","last_active":"2021-03-15T22:36:09.384431+0000","last_peered":"2021-03-15T22:36:09.384431+0000","last_clean":"2021-03-15T22:36:09.384431+0000","last_became_active":"2021-03-15T22:34:55.356086+0000","last_became_peered":"2021-03-15T22:34:55.356086+0000","last_unstale":"2021-03-15T22:36:09.384431+0000","last_undegraded":"2021-03-15T22:36:09.384431+0000","last_fullsized":"2021-03-15T22:36:09.384431+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"985","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377876+0000","last_change":"2021-03-15T22:34:55.356446+0000","last_active":"2021-03-15T22:36:07.377876+0000","last_peered":"2021-03-15T22:36:07.377876+0000","last_clean":"2021-03-15T22:36:07.377876+0000","last_became_active":"2021-03-15T22:34:55.355757+0000","last_became_peered":"2021-03-15T22:34:55.355757+0000","last_unstale":"2021-03-15T22:36:07.377876+0000","last_undegraded":"2021-03-15T22:36:07.377876+0000","last_fullsized":"2021-03-15T22:36:07.377876+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:33.883949+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:32:33.883949+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"966","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377130+0000","last_change":"2021-03-15T22:34:11.330201+0000","last_active":"2021-03-15T22:36:07.377130+0000","last_peered":"2021-03-15T22:36:07.377130+0000","last_clean":"2021-03-15T22:36:07.377130+0000","last_became_active":"2021-03-15T22:34:10.321886+0000","last_became_peered":"2021-03-15T22:34:10.321886+0000","last_unstale":"2021-03-15T22:36:07.377130+0000","last_undegraded":"2021-03-15T22:36:07.377130+0000","last_fullsized":"2021-03-15T22:36:07.377130+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:32:31.747582+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:32:31.747582+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"836'11","reported_seq":"639","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.384738+0000","last_change":"2021-03-15T22:35:44.256542+0000","last_active":"2021-03-15T22:36:09.384738+0000","last_peered":"2021-03-15T22:36:09.384738+0000","last_clean":"2021-03-15T22:36:09.384738+0000","last_became_active":"2021-03-15T22:33:04.336294+0000","last_became_peered":"2021-03-15T22:33:04.336294+0000","last_unstale":"2021-03-15T22:36:09.384738+0000","last_undegraded":"2021-03-15T22:36:09.384738+0000","last_fullsized":"2021-03-15T22:36:09.384738+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"836'11","last_scrub_stamp":"2021-03-15T22:35:44.256486+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:35:44.256486+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.2","version":"1016'967","reported_seq":"4155","reported_epoch":"1016","state":"active+clean","last_fresh":"2021-03-15T22:36:11.402070+0000","last_change":"2021-03-15T22:36:11.402331+0000","last_active":"2021-03-15T22:36:11.402070+0000","last_peered":"2021-03-15T22:36:11.402070+0000","last_clean":"2021-03-15T22:36:11.402070+0000","last_became_active":"2021-03-15T22:34:58.350873+0000","last_became_peered":"2021-03-15T22:34:58.350873+0000","last_unstale":"2021-03-15T22:36:11.402070+0000","last_undegraded":"2021-03-15T22:36:11.402070+0000","last_fullsized":"2021-03-15T22:36:11.402070+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'901","last_scrub_stamp":"2021-03-15T22:35:16.957336+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.957336+0000","log_size":967,"ondisk_log_size":967,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34897139,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":258,"num_read_kb":220515,"num_write":235,"num_write_kb":90032,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":114552572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"356","length":"1"},{"start":"35a","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"1053","reported_epoch":"1013","state":"active+clean","last_fresh":"2021-03-15T22:36:08.382982+0000","last_change":"2021-03-15T22:36:08.382982+0000","last_active":"2021-03-15T22:36:08.382982+0000","last_peered":"2021-03-15T22:36:08.382982+0000","last_clean":"2021-03-15T22:36:08.382982+0000","last_became_active":"2021-03-15T22:36:08.382713+0000","last_became_peered":"2021-03-15T22:36:08.382713+0000","last_unstale":"2021-03-15T22:36:08.382982+0000","last_undegraded":"2021-03-15T22:36:08.382982+0000","last_fullsized":"2021-03-15T22:36:08.382982+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.507204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.507204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"479","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.385306+0000","last_change":"2021-03-15T22:33:22.598240+0000","last_active":"2021-03-15T22:36:09.385306+0000","last_peered":"2021-03-15T22:36:09.385306+0000","last_clean":"2021-03-15T22:36:09.385306+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:36:09.385306+0000","last_undegraded":"2021-03-15T22:36:09.385306+0000","last_fullsized":"2021-03-15T22:36:09.385306+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.598174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.598174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"1208","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359390+0000","last_change":"2021-03-15T22:33:43.514160+0000","last_active":"2021-03-15T22:36:10.359390+0000","last_peered":"2021-03-15T22:36:10.359390+0000","last_clean":"2021-03-15T22:36:10.359390+0000","last_became_active":"2021-03-15T22:33:04.336145+0000","last_became_peered":"2021-03-15T22:33:04.336145+0000","last_unstale":"2021-03-15T22:36:10.359390+0000","last_undegraded":"2021-03-15T22:36:10.359390+0000","last_fullsized":"2021-03-15T22:36:10.359390+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:33:43.514088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.514088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"534","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.362096+0000","last_change":"2021-03-15T22:33:20.685041+0000","last_active":"2021-03-15T22:36:10.362096+0000","last_peered":"2021-03-15T22:36:10.362096+0000","last_clean":"2021-03-15T22:36:10.362096+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:36:10.362096+0000","last_undegraded":"2021-03-15T22:36:10.362096+0000","last_fullsized":"2021-03-15T22:36:10.362096+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:33:20.684976+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.684976+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.3","version":"1011'1681","reported_seq":"4685","reported_epoch":"1011","state":"active+clean","last_fresh":"2021-03-15T22:36:06.400631+0000","last_change":"2021-03-15T22:36:06.401096+0000","last_active":"2021-03-15T22:36:06.400631+0000","last_peered":"2021-03-15T22:36:06.400631+0000","last_clean":"2021-03-15T22:36:06.400631+0000","last_became_active":"2021-03-15T22:34:58.337099+0000","last_became_peered":"2021-03-15T22:34:58.337099+0000","last_unstale":"2021-03-15T22:36:06.400631+0000","last_undegraded":"2021-03-15T22:36:06.400631+0000","last_fullsized":"2021-03-15T22:36:06.400631+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'1611","last_scrub_stamp":"2021-03-15T22:35:21.054192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.054192+0000","log_size":1681,"ondisk_log_size":1681,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":40111289,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":2,"num_read":549,"num_read_kb":581637,"num_write":346,"num_write_kb":120323,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":226741522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"339","length":"1"},{"start":"359","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"1103","reported_epoch":"1016","state":"peering","last_fresh":"2021-03-15T22:36:11.373528+0000","last_change":"2021-03-15T22:36:11.373776+0000","last_active":"2021-03-15T22:36:09.377282+0000","last_peered":"2021-03-15T22:36:10.380283+0000","last_clean":"2021-03-15T22:36:10.380283+0000","last_became_active":"2021-03-15T22:34:55.341896+0000","last_became_peered":"2021-03-15T22:36:10.379534+0000","last_unstale":"2021-03-15T22:36:11.373528+0000","last_undegraded":"2021-03-15T22:36:11.373528+0000","last_fullsized":"2021-03-15T22:36:11.373528+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1015,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,3],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"799","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359909+0000","last_change":"2021-03-15T22:34:41.870475+0000","last_active":"2021-03-15T22:36:10.359909+0000","last_peered":"2021-03-15T22:36:10.359909+0000","last_clean":"2021-03-15T22:36:10.359909+0000","last_became_active":"2021-03-15T22:29:00.234381+0000","last_became_peered":"2021-03-15T22:29:00.234381+0000","last_unstale":"2021-03-15T22:36:10.359909+0000","last_undegraded":"2021-03-15T22:36:10.359909+0000","last_fullsized":"2021-03-15T22:36:10.359909+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.870368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.870368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"447","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.374884+0000","last_change":"2021-03-15T22:35:08.402702+0000","last_active":"2021-03-15T22:36:07.374884+0000","last_peered":"2021-03-15T22:36:07.374884+0000","last_clean":"2021-03-15T22:36:07.374884+0000","last_became_active":"2021-03-15T22:34:10.349731+0000","last_became_peered":"2021-03-15T22:34:10.349731+0000","last_unstale":"2021-03-15T22:36:07.374884+0000","last_undegraded":"2021-03-15T22:36:07.374884+0000","last_fullsized":"2021-03-15T22:36:07.374884+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:35:08.402641+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:35:08.402641+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1061","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359200+0000","last_change":"2021-03-15T22:34:43.343527+0000","last_active":"2021-03-15T22:36:10.359200+0000","last_peered":"2021-03-15T22:36:10.359200+0000","last_clean":"2021-03-15T22:36:10.359200+0000","last_became_active":"2021-03-15T22:34:43.343255+0000","last_became_peered":"2021-03-15T22:34:43.343255+0000","last_unstale":"2021-03-15T22:36:10.359200+0000","last_undegraded":"2021-03-15T22:36:10.359200+0000","last_fullsized":"2021-03-15T22:36:10.359200+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.673338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.673338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.c","version":"1015'1451","reported_seq":"4724","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.381263+0000","last_change":"2021-03-15T22:36:10.381564+0000","last_active":"2021-03-15T22:36:10.381263+0000","last_peered":"2021-03-15T22:36:10.381263+0000","last_clean":"2021-03-15T22:36:10.381263+0000","last_became_active":"2021-03-15T22:34:58.344406+0000","last_became_peered":"2021-03-15T22:34:58.344406+0000","last_unstale":"2021-03-15T22:36:10.381263+0000","last_undegraded":"2021-03-15T22:36:10.381263+0000","last_fullsized":"2021-03-15T22:36:10.381263+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'1407","last_scrub_stamp":"2021-03-15T22:35:43.758914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:43.758914+0000","log_size":1451,"ondisk_log_size":1451,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":45684471,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":409,"num_read_kb":369859,"num_write":316,"num_write_kb":116366,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":127958264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"356","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"957","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.377777+0000","last_change":"2021-03-15T22:34:55.356480+0000","last_active":"2021-03-15T22:36:07.377777+0000","last_peered":"2021-03-15T22:36:07.377777+0000","last_clean":"2021-03-15T22:36:07.377777+0000","last_became_active":"2021-03-15T22:34:55.356213+0000","last_became_peered":"2021-03-15T22:34:55.356213+0000","last_unstale":"2021-03-15T22:36:07.377777+0000","last_undegraded":"2021-03-15T22:36:07.377777+0000","last_fullsized":"2021-03-15T22:36:07.377777+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.729838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.729838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1319","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359548+0000","last_change":"2021-03-15T22:35:19.468482+0000","last_active":"2021-03-15T22:36:10.359548+0000","last_peered":"2021-03-15T22:36:10.359548+0000","last_clean":"2021-03-15T22:36:10.359548+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:36:10.359548+0000","last_undegraded":"2021-03-15T22:36:10.359548+0000","last_fullsized":"2021-03-15T22:36:10.359548+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:35:19.468402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.468402+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"422","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.372267+0000","last_change":"2021-03-15T22:35:45.165921+0000","last_active":"2021-03-15T22:36:05.372267+0000","last_peered":"2021-03-15T22:36:05.372267+0000","last_clean":"2021-03-15T22:36:05.372267+0000","last_became_active":"2021-03-15T22:34:10.332089+0000","last_became_peered":"2021-03-15T22:34:10.332089+0000","last_unstale":"2021-03-15T22:36:05.372267+0000","last_undegraded":"2021-03-15T22:36:05.372267+0000","last_fullsized":"2021-03-15T22:36:05.372267+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.165838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.165838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"172","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.371330+0000","last_change":"2021-03-15T22:34:44.050423+0000","last_active":"2021-03-15T22:36:05.371330+0000","last_peered":"2021-03-15T22:36:05.371330+0000","last_clean":"2021-03-15T22:36:05.371330+0000","last_became_active":"2021-03-15T22:33:45.317888+0000","last_became_peered":"2021-03-15T22:33:45.317888+0000","last_unstale":"2021-03-15T22:36:05.371330+0000","last_undegraded":"2021-03-15T22:36:05.371330+0000","last_fullsized":"2021-03-15T22:36:05.371330+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:34:44.050366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:44.050366+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"1015'496","reported_seq":"3363","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.366847+0000","last_change":"2021-03-15T22:36:10.367108+0000","last_active":"2021-03-15T22:36:10.366847+0000","last_peered":"2021-03-15T22:36:10.366847+0000","last_clean":"2021-03-15T22:36:10.366847+0000","last_became_active":"2021-03-15T22:33:08.354872+0000","last_became_peered":"2021-03-15T22:33:08.354872+0000","last_unstale":"2021-03-15T22:36:10.366847+0000","last_undegraded":"2021-03-15T22:36:10.366847+0000","last_fullsized":"2021-03-15T22:36:10.366847+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'470","last_scrub_stamp":"2021-03-15T22:35:42.721100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.721100+0000","log_size":496,"ondisk_log_size":496,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":14522877,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":189,"num_read_kb":197819,"num_write":104,"num_write_kb":39131,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":31290769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"356","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"98","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.354694+0000","last_change":"2021-03-15T22:34:43.340972+0000","last_active":"2021-03-15T22:36:10.354694+0000","last_peered":"2021-03-15T22:36:10.354694+0000","last_clean":"2021-03-15T22:36:10.354694+0000","last_became_active":"2021-03-15T22:34:43.340600+0000","last_became_peered":"2021-03-15T22:34:43.340600+0000","last_unstale":"2021-03-15T22:36:10.354694+0000","last_undegraded":"2021-03-15T22:36:10.354694+0000","last_fullsized":"2021-03-15T22:36:10.354694+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:09.612126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.612126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"1055","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.378182+0000","last_change":"2021-03-15T22:35:21.917891+0000","last_active":"2021-03-15T22:36:07.378182+0000","last_peered":"2021-03-15T22:36:07.378182+0000","last_clean":"2021-03-15T22:36:07.378182+0000","last_became_active":"2021-03-15T22:33:08.352367+0000","last_became_peered":"2021-03-15T22:33:08.352367+0000","last_unstale":"2021-03-15T22:36:07.378182+0000","last_undegraded":"2021-03-15T22:36:07.378182+0000","last_fullsized":"2021-03-15T22:36:07.378182+0000","mapping_epoch":837,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'9","last_scrub_stamp":"2021-03-15T22:35:21.917819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.917819+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1025","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359296+0000","last_change":"2021-03-15T22:34:55.347300+0000","last_active":"2021-03-15T22:36:10.359296+0000","last_peered":"2021-03-15T22:36:10.359296+0000","last_clean":"2021-03-15T22:36:10.359296+0000","last_became_active":"2021-03-15T22:34:55.346934+0000","last_became_peered":"2021-03-15T22:34:55.346934+0000","last_unstale":"2021-03-15T22:36:10.359296+0000","last_undegraded":"2021-03-15T22:36:10.359296+0000","last_fullsized":"2021-03-15T22:36:10.359296+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.461102+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"1187","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359443+0000","last_change":"2021-03-15T22:35:21.384129+0000","last_active":"2021-03-15T22:36:10.359443+0000","last_peered":"2021-03-15T22:36:10.359443+0000","last_clean":"2021-03-15T22:36:10.359443+0000","last_became_active":"2021-03-15T22:34:10.330608+0000","last_became_peered":"2021-03-15T22:34:10.330608+0000","last_unstale":"2021-03-15T22:36:10.359443+0000","last_undegraded":"2021-03-15T22:36:10.359443+0000","last_fullsized":"2021-03-15T22:36:10.359443+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:35:21.384037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.384037+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.e","version":"1011'499","reported_seq":"2860","reported_epoch":"1013","state":"active+clean","last_fresh":"2021-03-15T22:36:08.373293+0000","last_change":"2021-03-15T22:36:08.373558+0000","last_active":"2021-03-15T22:36:08.373293+0000","last_peered":"2021-03-15T22:36:08.373293+0000","last_clean":"2021-03-15T22:36:08.373293+0000","last_became_active":"2021-03-15T22:34:58.350923+0000","last_became_peered":"2021-03-15T22:34:58.350923+0000","last_unstale":"2021-03-15T22:36:08.373293+0000","last_undegraded":"2021-03-15T22:36:08.373293+0000","last_fullsized":"2021-03-15T22:36:08.373293+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"957'470","last_scrub_stamp":"2021-03-15T22:35:18.338289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.338289+0000","log_size":499,"ondisk_log_size":499,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":9671780,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":157,"num_read_kb":164363,"num_write":118,"num_write_kb":39415,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":52707265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"359","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"760","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.355315+0000","last_change":"2021-03-15T22:34:55.345645+0000","last_active":"2021-03-15T22:36:10.355315+0000","last_peered":"2021-03-15T22:36:10.355315+0000","last_clean":"2021-03-15T22:36:10.355315+0000","last_became_active":"2021-03-15T22:34:55.345267+0000","last_became_peered":"2021-03-15T22:34:55.345267+0000","last_unstale":"2021-03-15T22:36:10.355315+0000","last_undegraded":"2021-03-15T22:36:10.355315+0000","last_fullsized":"2021-03-15T22:36:10.355315+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:38.958050+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:32:38.958050+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"1047","reported_epoch":"1010","state":"active+clean","last_fresh":"2021-03-15T22:36:05.373909+0000","last_change":"2021-03-15T22:35:12.091714+0000","last_active":"2021-03-15T22:36:05.373909+0000","last_peered":"2021-03-15T22:36:05.373909+0000","last_clean":"2021-03-15T22:36:05.373909+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:36:05.373909+0000","last_undegraded":"2021-03-15T22:36:05.373909+0000","last_fullsized":"2021-03-15T22:36:05.373909+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.091650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.091650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"208","reported_epoch":"1014","state":"active+clean","last_fresh":"2021-03-15T22:36:09.372778+0000","last_change":"2021-03-15T22:35:20.678309+0000","last_active":"2021-03-15T22:36:09.372778+0000","last_peered":"2021-03-15T22:36:09.372778+0000","last_clean":"2021-03-15T22:36:09.372778+0000","last_became_active":"2021-03-15T22:34:10.332236+0000","last_became_peered":"2021-03-15T22:34:10.332236+0000","last_unstale":"2021-03-15T22:36:09.372778+0000","last_undegraded":"2021-03-15T22:36:09.372778+0000","last_fullsized":"2021-03-15T22:36:09.372778+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.678246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"1161","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359051+0000","last_change":"2021-03-15T22:34:55.371223+0000","last_active":"2021-03-15T22:36:10.359051+0000","last_peered":"2021-03-15T22:36:10.359051+0000","last_clean":"2021-03-15T22:36:10.359051+0000","last_became_active":"2021-03-15T22:34:55.341958+0000","last_became_peered":"2021-03-15T22:34:55.341958+0000","last_unstale":"2021-03-15T22:36:10.359051+0000","last_undegraded":"2021-03-15T22:36:10.359051+0000","last_fullsized":"2021-03-15T22:36:10.359051+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:32:43.307660+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:32:43.307660+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.f","version":"1013'1674","reported_seq":"4791","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.376621+0000","last_change":"2021-03-15T22:36:10.377423+0000","last_active":"2021-03-15T22:36:10.376621+0000","last_peered":"2021-03-15T22:36:10.376621+0000","last_clean":"2021-03-15T22:36:10.376621+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:36:10.376621+0000","last_undegraded":"2021-03-15T22:36:10.376621+0000","last_fullsized":"2021-03-15T22:36:10.376621+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"837'1315","last_scrub_stamp":"2021-03-15T22:33:08.608669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:08.608669+0000","log_size":1674,"ondisk_log_size":1674,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":53461620,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":549,"num_read_kb":589645,"num_write":374,"num_write_kb":134308,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2f8","length":"1"},{"start":"300","length":"1"},{"start":"339","length":"1"},{"start":"356","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"1026","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.359606+0000","last_change":"2021-03-15T22:34:55.340853+0000","last_active":"2021-03-15T22:36:10.359606+0000","last_peered":"2021-03-15T22:36:10.359606+0000","last_clean":"2021-03-15T22:36:10.359606+0000","last_became_active":"2021-03-15T22:34:55.340498+0000","last_became_peered":"2021-03-15T22:34:55.340498+0000","last_unstale":"2021-03-15T22:36:10.359606+0000","last_undegraded":"2021-03-15T22:36:10.359606+0000","last_fullsized":"2021-03-15T22:36:10.359606+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:20.500979+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:36.417221+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.500979+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"976'19","reported_seq":"415","reported_epoch":"1012","state":"active+clean","last_fresh":"2021-03-15T22:36:07.378380+0000","last_change":"2021-03-15T22:35:28.365207+0000","last_active":"2021-03-15T22:36:07.378380+0000","last_peered":"2021-03-15T22:36:07.378380+0000","last_clean":"2021-03-15T22:36:07.378380+0000","last_became_active":"2021-03-15T22:35:28.364764+0000","last_became_peered":"2021-03-15T22:35:28.364764+0000","last_unstale":"2021-03-15T22:36:07.378380+0000","last_undegraded":"2021-03-15T22:36:07.378380+0000","last_fullsized":"2021-03-15T22:36:07.378380+0000","mapping_epoch":972,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":973,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'13","last_scrub_stamp":"2021-03-15T22:33:37.949876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.949876+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":"445","reported_epoch":"1016","state":"peering","last_fresh":"2021-03-15T22:36:11.374743+0000","last_change":"2021-03-15T22:36:11.374951+0000","last_active":"2021-03-15T22:36:11.374743+0000","last_peered":"2021-03-15T22:36:10.355003+0000","last_clean":"2021-03-15T22:36:10.355003+0000","last_became_active":"2021-03-15T22:28:59.275051+0000","last_became_peered":"2021-03-15T22:28:59.275051+0000","last_unstale":"2021-03-15T22:36:11.374743+0000","last_undegraded":"2021-03-15T22:36:11.374743+0000","last_fullsized":"2021-03-15T22:36:11.374743+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.690309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[0,4],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"1205","reported_epoch":"1015","state":"active+clean","last_fresh":"2021-03-15T22:36:10.358833+0000","last_change":"2021-03-15T22:35:46.406478+0000","last_active":"2021-03-15T22:36:10.358833+0000","last_peered":"2021-03-15T22:36:10.358833+0000","last_clean":"2021-03-15T22:36:10.358833+0000","last_became_active":"2021-03-15T22:33:04.337113+0000","last_became_peered":"2021-03-15T22:33:04.337113+0000","last_unstale":"2021-03-15T22:36:10.358833+0000","last_undegraded":"2021-03-15T22:36:10.358833+0000","last_fullsized":"2021-03-15T22:36:10.358833+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:35:46.406376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:46.406376+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":789606607,"num_objects":359,"num_object_clones":309,"num_object_copies":1077,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":359,"num_whiteouts":17,"num_read":7264,"num_read_kb":7467758,"num_write":5302,"num_write_kb":2039760,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4621,"num_bytes_recovered":2230024638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1763155968,"data_stored":2475749326,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":23320,"ondisk_log_size":23320,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1079,"num_read_kb":1074,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":3,"acting":3,"num_store_stats":7},{"poolid":3,"num_pg":28,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":84,"acting":84,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1482,"num_read_kb":987,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":94766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":947,"ondisk_log_size":947,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":34,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":102,"acting":102,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299615,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":291512,"kb_used_data":233520,"kb_used_omap":0,"kb_used_meta":57984,"kb_avail":93453640,"statfs":{"total":95995035648,"available":95696527360,"internally_reserved":0,"allocated":239124480,"data_stored":354446692,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":59375616},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.41099999999999998,"5min":0.45000000000000001,"15min":0.46500000000000002},"min":{"1min":0.20499999999999999,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":0.57399999999999995,"5min":0.70099999999999996,"15min":2.73},"last":0.316},{"interface":"front","average":{"1min":0.45300000000000001,"5min":0.48799999999999999,"15min":0.48899999999999999},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.222},"max":{"1min":0.61199999999999999,"5min":0.75800000000000001,"15min":3.8799999999999999},"last":0.28000000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.40400000000000003,"5min":0.443,"15min":0.47199999999999998},"min":{"1min":0.254,"5min":0.249,"15min":0.249},"max":{"1min":0.57299999999999995,"5min":0.73899999999999999,"15min":2.6859999999999999},"last":0.375},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.56000000000000005,"15min":0.51400000000000001},"min":{"1min":0.29099999999999998,"5min":0.28399999999999997,"15min":0.23400000000000001},"max":{"1min":5.3819999999999997,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":0.41299999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:35:38 2021","interfaces":[{"interface":"back","average":{"1min":2.1760000000000002,"5min":0.83999999999999997,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":29.350999999999999,"5min":29.350999999999999,"15min":29.350999999999999},"last":0.45900000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.498,"15min":0.5},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.55600000000000005}]},{"osd":3,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.48999999999999999,"5min":0.498,"15min":0.495},"min":{"1min":0.314,"5min":0.314,"15min":0.28899999999999998},"max":{"1min":1.1819999999999999,"5min":1.1819999999999999,"15min":1.702},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.49099999999999999,"5min":0.497,"15min":0.52500000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.1930000000000001,"5min":1.1930000000000001,"15min":5.415},"last":0.43099999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.41399999999999998,"5min":0.437,"15min":0.44400000000000001},"min":{"1min":0.28999999999999998,"5min":0.27900000000000003,"15min":0.23799999999999999},"max":{"1min":0.58499999999999996,"5min":0.753,"15min":1.927},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.41499999999999998,"5min":0.46300000000000002,"15min":0.46200000000000002},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":0.57599999999999996,"5min":0.72699999999999998,"15min":2.5339999999999998},"last":0.57899999999999996}]},{"osd":5,"last update":"Mon Mar 15 22:35:38 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47899999999999998,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68300000000000005,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.503},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.498,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67800000000000005,"5min":0.77700000000000002,"15min":1.071},"last":0.61399999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51800000000000002,"15min":0.51600000000000001},"min":{"1min":0.311,"5min":0.29999999999999999,"15min":0.27100000000000002},"max":{"1min":0.65300000000000002,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.53500000000000003,"15min":0.52100000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.28399999999999997},"max":{"1min":0.73799999999999999,"5min":0.81999999999999995,"15min":1.8560000000000001},"last":0.747}]}]},{"osd":6,"up_from":45,"seq":193273528547,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":225852,"kb_used_data":178996,"kb_used_omap":0,"kb_used_meta":46848,"kb_avail":93519300,"statfs":{"total":95995035648,"available":95763763200,"internally_reserved":0,"allocated":183291904,"data_stored":251412912,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":47972352},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.49199999999999999,"15min":0.48399999999999999},"min":{"1min":0.23100000000000001,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.79600000000000004,"5min":2.5470000000000002,"15min":3.085},"last":0.57099999999999995},{"interface":"front","average":{"1min":0.45200000000000001,"5min":0.47999999999999998,"15min":0.46500000000000002},"min":{"1min":0.27600000000000002,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.73099999999999998,"5min":1.137,"15min":1.323},"last":0.39100000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.505,"5min":0.55200000000000005,"15min":0.55500000000000005},"min":{"1min":0.26400000000000001,"5min":0.26400000000000001,"15min":0.254},"max":{"1min":1.1120000000000001,"5min":2.7839999999999998,"15min":14.637},"last":0.40500000000000003},{"interface":"front","average":{"1min":0.42699999999999999,"5min":0.45200000000000001,"15min":0.47499999999999998},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60299999999999998,"5min":0.84899999999999998,"15min":5.2450000000000001},"last":0.501}]},{"osd":2,"last update":"Mon Mar 15 22:35:43 2021","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64800000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.53200000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70899999999999996,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.55800000000000005}]},{"osd":3,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.51100000000000001,"15min":0.50900000000000001},"min":{"1min":0.35099999999999998,"5min":0.33000000000000002,"15min":0.29899999999999999},"max":{"1min":0.874,"5min":1.0609999999999999,"15min":2.6150000000000002},"last":2.1560000000000001},{"interface":"front","average":{"1min":0.45900000000000002,"5min":0.502,"15min":0.502},"min":{"1min":0.32900000000000001,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.63900000000000001,"5min":2.4670000000000001,"15min":2.7890000000000001},"last":0.379}]},{"osd":4,"last update":"Mon Mar 15 22:35:11 2021","interfaces":[{"interface":"back","average":{"1min":0.39200000000000002,"5min":0.44600000000000001,"15min":0.438},"min":{"1min":0.25,"5min":0.25,"15min":0.224},"max":{"1min":0.61799999999999999,"5min":0.94099999999999995,"15min":4.306},"last":0.433},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.47799999999999998,"15min":0.49199999999999999},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.23499999999999999},"max":{"1min":0.82799999999999996,"5min":1.091,"15min":4.181},"last":0.46800000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:35:39 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.47599999999999998,"15min":0.42599999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.26700000000000002},"max":{"1min":0.63300000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.53000000000000003,"15min":0.46999999999999997},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67900000000000005,"5min":1.022,"15min":1.022},"last":0.51800000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:35:21 2021","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.47799999999999998,"15min":0.53200000000000003},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.249},"max":{"1min":0.67700000000000005,"5min":0.76300000000000001,"15min":15.340999999999999},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.502,"5min":0.56399999999999995,"15min":0.58699999999999997},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.27200000000000002},"max":{"1min":0.754,"5min":1.0109999999999999,"15min":15.356},"last":0.59399999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575092,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":194040,"kb_used_data":125316,"kb_used_omap":0,"kb_used_meta":68224,"kb_avail":93551112,"statfs":{"total":95995035648,"available":95796338688,"internally_reserved":0,"allocated":128323584,"data_stored":178368421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":69861376},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:56 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.372,"15min":0.34000000000000002},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":0.50800000000000001,"5min":4.3920000000000003,"15min":4.3920000000000003},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.42499999999999999,"15min":0.37},"min":{"1min":0.215,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.52500000000000002,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:35:45 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.46899999999999997,"15min":0.379},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59199999999999997,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45500000000000002,"15min":0.40100000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64600000000000002,"5min":5.0190000000000001,"15min":5.0190000000000001},"last":0.29999999999999999}]},{"osd":3,"last update":"Mon Mar 15 22:35:20 2021","interfaces":[{"interface":"back","average":{"1min":0.82299999999999995,"5min":0.47599999999999998,"15min":0.40000000000000002},"min":{"1min":0.28599999999999998,"5min":0.26400000000000001,"15min":0.215},"max":{"1min":7.718,"5min":7.718,"15min":7.718},"last":0.38},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.53700000000000003,"15min":0.434},"min":{"1min":0.245,"5min":0.245,"15min":0.20999999999999999},"max":{"1min":8.4760000000000009,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":0.442}]},{"osd":4,"last update":"Mon Mar 15 22:35:26 2021","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.61399999999999999,"15min":0.55200000000000005},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.22800000000000001},"max":{"1min":0.78800000000000003,"5min":4.593,"15min":4.593},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.58299999999999996,"15min":0.57799999999999996},"min":{"1min":0.36799999999999999,"5min":0.34499999999999997,"15min":0.253},"max":{"1min":4.3140000000000001,"5min":4.3140000000000001,"15min":4.6139999999999999},"last":0.627}]},{"osd":5,"last update":"Mon Mar 15 22:35:34 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.59499999999999997,"15min":0.54700000000000004},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.34300000000000003},"max":{"1min":4.3630000000000004,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60999999999999999,"15min":0.52900000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32100000000000001},"max":{"1min":1.794,"5min":3.472,"15min":3.472},"last":0.67000000000000004}]},{"osd":6,"last update":"Mon Mar 15 22:35:59 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.60799999999999998,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.38500000000000001,"15min":0.314},"max":{"1min":0.755,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.61599999999999999},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.63100000000000001,"15min":0.57899999999999996},"min":{"1min":0.41199999999999998,"5min":0.375,"15min":0.33700000000000002},"max":{"1min":0.68300000000000005,"5min":4.6529999999999996,"15min":4.6529999999999996},"last":0.59499999999999997}]},{"osd":7,"last update":"Mon Mar 15 22:35:20 2021","interfaces":[{"interface":"back","average":{"1min":0.626,"5min":0.63700000000000001,"15min":0.60299999999999998},"min":{"1min":0.443,"5min":0.39800000000000002,"15min":0.316},"max":{"1min":1.716,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.64900000000000002},{"interface":"front","average":{"1min":0.623,"5min":0.66800000000000004,"15min":0.59599999999999997},"min":{"1min":0.42799999999999999,"5min":0.40300000000000002,"15min":0.35799999999999998},"max":{"1min":1.466,"5min":4.625,"15min":4.625},"last":0.54700000000000004}]}]},{"osd":0,"up_from":8,"seq":34359738615,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":306756,"kb_used_data":259644,"kb_used_omap":0,"kb_used_meta":47104,"kb_avail":93438396,"statfs":{"total":95995035648,"available":95680917504,"internally_reserved":0,"allocated":265875456,"data_stored":390035431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48234496},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:35:47 2021","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35899999999999999,"15min":0.34000000000000002},"min":{"1min":0.249,"5min":0.192,"15min":0.17999999999999999},"max":{"1min":0.68000000000000005,"5min":1.46,"15min":1.7070000000000001},"last":0.27300000000000002},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39600000000000002,"15min":0.36899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19900000000000001},"max":{"1min":0.499,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.26000000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:35:41 2021","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.374,"15min":0.36399999999999999},"min":{"1min":0.28000000000000003,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.70099999999999996,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.375,"5min":0.49199999999999999,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51300000000000001,"5min":7.3799999999999999,"15min":7.3799999999999999},"last":0.31}]},{"osd":3,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.40699999999999997,"5min":0.39700000000000002,"15min":0.38400000000000001},"min":{"1min":0.25700000000000001,"5min":0.23100000000000001,"15min":0.22700000000000001},"max":{"1min":0.61799999999999999,"5min":1.544,"15min":1.544},"last":0.36799999999999999},{"interface":"front","average":{"1min":0.442,"5min":0.47499999999999998,"15min":0.41999999999999998},"min":{"1min":0.31,"5min":0.254,"15min":0.19400000000000001},"max":{"1min":0.67700000000000005,"5min":3.988,"15min":3.988},"last":0.33100000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.55700000000000005,"15min":0.53800000000000003},"min":{"1min":0.36899999999999999,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":0.76300000000000001,"5min":1.861,"15min":1.861},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.59399999999999997,"15min":0.56599999999999995},"min":{"1min":0.40699999999999997,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":0.74099999999999999,"5min":1.5700000000000001,"15min":1.645},"last":0.57599999999999996}]},{"osd":5,"last update":"Mon Mar 15 22:35:35 2021","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.60799999999999998,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.32600000000000001,"15min":0.29899999999999999},"max":{"1min":0.71599999999999997,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.504,"5min":0.56000000000000005,"15min":0.502},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.318},"max":{"1min":0.72099999999999997,"5min":1.909,"15min":1.909},"last":0.55800000000000005}]},{"osd":6,"last update":"Mon Mar 15 22:36:01 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.57399999999999995,"15min":0.55600000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.26200000000000001},"max":{"1min":0.78800000000000003,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.53800000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.59299999999999997,"15min":0.56899999999999995},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.27100000000000002},"max":{"1min":0.77300000000000002,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.54600000000000004}]},{"osd":7,"last update":"Mon Mar 15 22:35:24 2021","interfaces":[{"interface":"back","average":{"1min":0.59199999999999997,"5min":0.65500000000000003,"15min":0.58699999999999997},"min":{"1min":0.42899999999999999,"5min":0.38200000000000001,"15min":0.27900000000000003},"max":{"1min":0.753,"5min":3.8239999999999998,"15min":3.8239999999999998},"last":0.52900000000000003},{"interface":"front","average":{"1min":0.57099999999999995,"5min":0.60699999999999998,"15min":0.57999999999999996},"min":{"1min":0.44500000000000001,"5min":0.39100000000000001,"15min":0.317},"max":{"1min":0.752,"5min":2.665,"15min":2.665},"last":0.48899999999999999}]}]},{"osd":2,"up_from":681,"seq":2924872728645,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":351376,"kb_used_data":319172,"kb_used_omap":119,"kb_used_meta":31944,"kb_avail":93393776,"statfs":{"total":95995035648,"available":95635226624,"internally_reserved":0,"allocated":326832128,"data_stored":430970732,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":32711017},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.316,"15min":0.31},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.56299999999999994,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.254},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35299999999999998,"15min":0.33100000000000002},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.77800000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.314}]},{"osd":1,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.35599999999999998,"15min":0.33400000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.58099999999999996,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.27800000000000002},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.379,"15min":0.36299999999999999},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.59799999999999998,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.35199999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.42099999999999999,"15min":0.36499999999999999},"min":{"1min":0.26200000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.65500000000000003,"5min":3.496,"15min":3.496},"last":0.39000000000000001},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.36199999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.85099999999999998,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.44600000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54800000000000004,"15min":0.49399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67600000000000005,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53800000000000003,"15min":0.499},"min":{"1min":0.378,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76700000000000002,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.56200000000000006}]},{"osd":5,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.875,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.501},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56100000000000005,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70299999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.53900000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57599999999999996,"15min":0.51900000000000002},"min":{"1min":0.372,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76600000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.51000000000000001},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.52500000000000002},"min":{"1min":0.433,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":1.387,"15min":1.387},"last":0.61499999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59399999999999997,"15min":0.54300000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.748,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.48799999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57599999999999996,"15min":0.53200000000000003},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82599999999999996,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.59199999999999997}]}]},{"osd":3,"up_from":26,"seq":111669149933,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":331384,"kb_used_data":254064,"kb_used_omap":0,"kb_used_meta":77312,"kb_avail":93413768,"statfs":{"total":95995035648,"available":95655698432,"internally_reserved":0,"allocated":260161536,"data_stored":349042327,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":79167488},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.32500000000000001,"5min":0.34499999999999997,"15min":0.34000000000000002},"min":{"1min":0.23000000000000001,"5min":0.183,"15min":0.17699999999999999},"max":{"1min":0.47799999999999998,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":0.25600000000000001},{"interface":"front","average":{"1min":0.36599999999999999,"5min":0.33300000000000002,"15min":0.33800000000000002},"min":{"1min":0.23699999999999999,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.504,"5min":0.76900000000000002,"15min":2.6339999999999999},"last":0.23599999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.371,"15min":0.39200000000000002},"min":{"1min":0.27300000000000002,"5min":0.23599999999999999,"15min":0.23000000000000001},"max":{"1min":0.501,"5min":0.626,"15min":6.0199999999999996},"last":0.27500000000000002},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.36799999999999999,"15min":0.39900000000000002},"min":{"1min":0.219,"5min":0.219,"15min":0.20699999999999999},"max":{"1min":0.47899999999999998,"5min":2.5619999999999998,"15min":7.6130000000000004},"last":0.33800000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:35:40 2021","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.246,"5min":0.216,"15min":0.216},"max":{"1min":0.53100000000000003,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.311},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.30499999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.32000000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:35:36 2021","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.35599999999999998,"5min":0.31900000000000001,"15min":0.22600000000000001},"max":{"1min":0.72199999999999998,"5min":0.86899999999999999,"15min":3.3370000000000002},"last":0.58799999999999997},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53900000000000003,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.311,"15min":0.26700000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":2.2890000000000001},"last":0.52300000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:35:36 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.755},"min":{"1min":0.35099999999999998,"5min":0.311,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":0.75800000000000001,"15min":9.4870000000000001},"last":0.65800000000000003},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54100000000000004,"15min":0.95899999999999996},"min":{"1min":0.37,"5min":0.36699999999999999,"15min":0.32800000000000001},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":15.776},"last":0.498}]},{"osd":6,"last update":"Mon Mar 15 22:36:06 2021","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38900000000000001,"5min":0.38200000000000001,"15min":0.30099999999999999},"max":{"1min":0.81699999999999995,"5min":1.359,"15min":2.6499999999999999},"last":0.68200000000000005},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.38900000000000001,"5min":0.375,"15min":0.30399999999999999},"max":{"1min":0.78500000000000003,"5min":1.3400000000000001,"15min":3.472},"last":0.64200000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:35:23 2021","interfaces":[{"interface":"back","average":{"1min":0.53800000000000003,"5min":0.54400000000000004,"15min":0.57399999999999995},"min":{"1min":0.36199999999999999,"5min":0.308,"15min":0.29599999999999999},"max":{"1min":0.77600000000000002,"5min":0.91200000000000003,"15min":5.625},"last":0.748},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.57899999999999996,"15min":0.56499999999999995},"min":{"1min":0.40799999999999997,"5min":0.371,"15min":0.35499999999999998},"max":{"1min":0.74299999999999999,"5min":2.621,"15min":2.621},"last":0.59999999999999998}]}]},{"osd":4,"up_from":31,"seq":133143986410,"num_pgs":63,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":154628,"kb_used_data":109180,"kb_used_omap":0,"kb_used_meta":45440,"kb_avail":93590524,"statfs":{"total":95995035648,"available":95836696576,"internally_reserved":0,"allocated":111800320,"data_stored":162313748,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46530560},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.50800000000000001,"15min":0.47599999999999998},"min":{"1min":0.30099999999999999,"5min":0.221,"15min":0.20000000000000001},"max":{"1min":0.55900000000000005,"5min":4.7050000000000001,"15min":4.7050000000000001},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.435,"5min":0.46500000000000002,"15min":0.498},"min":{"1min":0.27300000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.67100000000000004,"5min":1.4179999999999999,"15min":5.0209999999999999},"last":0.68799999999999994}]},{"osd":1,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.312,"5min":0.25,"15min":0.23999999999999999},"max":{"1min":0.73399999999999999,"5min":1.3720000000000001,"15min":4.3499999999999996},"last":0.64300000000000002},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.499,"15min":0.499},"min":{"1min":0.311,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":0.627,"5min":1.494,"15min":4.718},"last":0.72899999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:35:41 2021","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47999999999999998,"15min":0.502},"min":{"1min":0.32200000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77700000000000002,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.57299999999999995},{"interface":"front","average":{"1min":0.497,"5min":0.48599999999999999,"15min":0.495},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.67400000000000004}]},{"osd":3,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.28499999999999998},"max":{"1min":1.0820000000000001,"5min":1.486,"15min":5.0499999999999998},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.54400000000000004},"min":{"1min":0.38300000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0660000000000001,"5min":1.4950000000000001,"15min":4.4050000000000002},"last":0.71099999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.41999999999999998,"15min":0.40799999999999997},"min":{"1min":0.27600000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.623,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47499999999999998,"15min":0.436},"min":{"1min":0.29099999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66400000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.61099999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:35:59 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.27500000000000002,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":4.3079999999999998},"last":0.52100000000000002},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.36199999999999999,"5min":0.33000000000000002,"15min":0.28000000000000003},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":4.6600000000000001},"last":0.70199999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:35:18 2021","interfaces":[{"interface":"back","average":{"1min":0.49199999999999999,"5min":0.51300000000000001,"15min":0.51000000000000001},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.26300000000000001},"max":{"1min":0.749,"5min":0.80900000000000005,"15min":4.6929999999999996},"last":0.65300000000000002},{"interface":"front","average":{"1min":0.505,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.29999999999999999,"5min":0.29999999999999999,"15min":0.25},"max":{"1min":0.73599999999999999,"5min":1.034,"15min":4.9900000000000002},"last":0.63}]}]},{"osd":5,"up_from":616,"seq":2645699854418,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301844,"kb_used_data":255884,"kb_used_omap":153,"kb_used_meta":45798,"kb_avail":93443308,"statfs":{"total":95995035648,"available":95685947392,"internally_reserved":0,"allocated":262025216,"data_stored":361916307,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":46897681},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48599999999999999,"15min":0.46300000000000002},"min":{"1min":0.26400000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60799999999999998,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":0.68100000000000005},{"interface":"front","average":{"1min":0.438,"5min":0.48099999999999998,"15min":0.45500000000000002},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63700000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.56399999999999995}]},{"osd":1,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.55600000000000005,"15min":0.48199999999999998},"min":{"1min":0.32000000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56599999999999995,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":0.61299999999999999},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.29399999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.54600000000000004,"5min":1.161,"15min":1.161},"last":0.63800000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.58299999999999996,"15min":0.53600000000000003},"min":{"1min":0.312,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.60399999999999998,"5min":10.952,"15min":10.952},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.499,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.623,"5min":1.125,"15min":1.125},"last":0.53600000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.67800000000000005,"15min":0.53800000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63200000000000001,"5min":10.355,"15min":10.355},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.51100000000000001,"15min":0.45900000000000002},"min":{"1min":0.29699999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65800000000000003,"5min":2.476,"15min":2.476},"last":0.55100000000000005}]},{"osd":4,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.51000000000000001,"15min":0.41399999999999998},"min":{"1min":0.27600000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.66900000000000004,"5min":7.4000000000000004,"15min":7.4000000000000004},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.47299999999999998,"15min":0.40799999999999997},"min":{"1min":0.32800000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71599999999999997,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.49299999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.50900000000000001,"15min":0.46500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.23999999999999999},"max":{"1min":0.64400000000000002,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.501,"5min":0.56200000000000006,"15min":0.49199999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.29699999999999999},"max":{"1min":0.70399999999999996,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.65100000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.46300000000000002},"min":{"1min":0.36899999999999999,"5min":0.29799999999999999,"15min":0.29299999999999998},"max":{"1min":0.69199999999999995,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.58799999999999997,"15min":0.51700000000000002},"min":{"1min":0.34300000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68500000000000005,"5min":5.944,"15min":5.944},"last":0.59099999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":12026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":10320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":8144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":263938048,"data_stored":388884677,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126377984,"data_stored":177250882,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":325050368,"data_stored":429843150,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":258228224,"data_stored":349457374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":110718976,"data_stored":165946646,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":260239360,"data_stored":360788611,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":181354496,"data_stored":250262960,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":237248512,"data_stored":353315026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:36:14.862 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2021-03-15T22:36:14.862 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool set device_health_metrics pg_num 11 2021-03-15T22:36:14.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:14.864 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:14.865 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:14.966 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:14.988 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:14.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:14.990 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:15.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: cluster 2021-03-15T22:36:12.855071+0000 osd.3 (osd.3) 177 : cluster [DBG] 6.f scrub ok 2021-03-15T22:36:15.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: cluster 2021-03-15T22:36:13.232035+0000 mgr.y (mgr.14144) 1117 : cluster [DBG] pgmap v1673: 146 pgs: 3 peering, 1 clean+premerge+peered, 142 active+clean; 757 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 21 MiB/s rd, 2.1 MiB/s wr, 25 op/s 2021-03-15T22:36:15.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: cluster 2021-03-15T22:36:13.313273+0000 osd.1 (osd.1 2021-03-15T22:36:15.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: ) 190 : cluster [DBG] 3.0 scrub ok 2021-03-15T22:36:15.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:36:15.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: -15T22:36:14. 2021-03-15T22:36:15.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: 037753+0000 osd.5 (osd.5) 62 2021-03-15T22:36:15.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: : cluster [DBG] 3.5 scrub ok 2021-03-15T22:36:15.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:36:15.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: -15T22:36:14.376697+0000 mon.a 2021-03-15T22:36:15.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: (mon.0) 2540 : cluster [DBG] 2021-03-15T22:36:15.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:14 smithi125 conmon[56739]: osdmap e1019: 8 total, 8 up, 8 in 2021-03-15T22:36:15.091 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:15.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:15.119 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:15.120 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:15.221 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:15.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:15.247 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:15.247 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:15.319 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:36:15.342 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:36:15.349 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:15.361 INFO:tasks.rados.rados.0.smithi076.stdout:3924: done (1 left) 2021-03-15T22:36:15.361 INFO:tasks.rados.rados.0.smithi076.stdout:3925: write (excl) oid 37 current snap is 462 2021-03-15T22:36:15.362 INFO:tasks.rados.rados.0.smithi076.stdout:3925: seq_num 1331 ranges {770117=518194,1870239=499878} 2021-03-15T22:36:15.364 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:36:15.369 INFO:tasks.rados.rados.0.smithi076.stdout:3925: writing smithi07694878-37 from 770117 to 1288311 tid 1 2021-03-15T22:36:15.376 INFO:tasks.rados.rados.0.smithi076.stdout:3925: writing smithi07694878-37 from 1870239 to 2370117 tid 2 2021-03-15T22:36:15.376 INFO:tasks.rados.rados.0.smithi076.stdout:3926: copy_from oid 5 from oid 16 current snap is 462 2021-03-15T22:36:15.376 INFO:tasks.rados.rados.0.smithi076.stdout:3927: read oid 1 snap -1 2021-03-15T22:36:15.376 INFO:tasks.rados.rados.0.smithi076.stdout:3927: expect deleted 2021-03-15T22:36:15.377 INFO:tasks.rados.rados.0.smithi076.stdout:3928: snap_create 2021-03-15T22:36:15.377 INFO:tasks.rados.rados.0.smithi076.stdout:3923: expect (ObjNum 620 snap 193 seq_num 620) 2021-03-15T22:36:15.385 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:15.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:15.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:15.397 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:36:15.417 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:36:15.434 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:36:15.452 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:36:15.469 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:36:15.470 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:36:15.471 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:36:15.471 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:36:15.471 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:36:15.471 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:36:15.472 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:36:15.472 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:36:15.472 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:36:15.488 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:15.512 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:15.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:15.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:15.515 INFO:tasks.rados.rados.0.smithi076.stdout:3925: finishing write tid 1 to smithi07694878-37 2021-03-15T22:36:15.515 INFO:tasks.rados.rados.0.smithi076.stdout:3926: finishing copy_from to smithi07694878-5 2021-03-15T22:36:15.516 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 2826 (ObjNum 1075 snap 366 seq_num 1075) dirty exists 2021-03-15T22:36:15.516 INFO:tasks.rados.rados.0.smithi076.stdout:3926: finishing copy_from racing read to smithi07694878-5 2021-03-15T22:36:15.516 INFO:tasks.rados.rados.0.smithi076.stdout:3925: finishing write tid 2 to smithi07694878-37 2021-03-15T22:36:15.516 INFO:tasks.rados.rados.0.smithi076.stdout:3925: finishing write tid 3 to smithi07694878-37 2021-03-15T22:36:15.516 INFO:tasks.rados.rados.0.smithi076.stdout:3925: finishing write tid 4 to smithi07694878-37 2021-03-15T22:36:15.517 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 37 v 3440 (ObjNum 1331 snap 462 seq_num 1331) dirty exists 2021-03-15T22:36:15.517 INFO:tasks.rados.rados.0.smithi076.stdout:3925: left oid 37 (ObjNum 1331 snap 462 seq_num 1331) 2021-03-15T22:36:15.615 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:15.638 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:15.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:15.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:15.741 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:15.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:15.765 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:15.765 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:15.866 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:15.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:15.891 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:15.891 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:15.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:16.015 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:16.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:16.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:16.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[56514]: cluster 2021-03-15T22:36:13.797836+0000 osd.7 (osd.7) 191 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:36:16.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[56514]: cluster 2021-03-15T22:36:14.097427+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.b scrub ok 2021-03-15T22:36:16.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[56514]: cluster 2021-03-15T22:36:14.641349+0000 osd.2 (osd.2) 52 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:36:16.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[56514]: audit 2021-03-15T22:36:15.233337+0000 mon.a (mon.0) 2541 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-15T22:36:16.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[56514]: audit 2021-03-15T22:36:15.300333+0000 mon.a (mon.0) 2542 : audit [DBG] from='client.? 172.21.15.76:0/1369159702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:16.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[56514]: audit 2021-03-15T22:36:15.354958+0000 mon.a (mon.0) 2543 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-15T22:36:16.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[56514]: cluster 2021-03-15T22:36:15.355008+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e1020: 8 total, 8 up, 8 in 2021-03-15T22:36:16.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[62156]: cluster 2021-03-15T22:36:13.797836+0000 osd.7 (osd.7) 191 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:36:16.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[62156]: cluster 2021-03-15T22:36:14.097427+0000 osd.0 (osd.0) 164 : cluster [DBG] 4.b scrub ok 2021-03-15T22:36:16.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[62156]: cluster 2021-03-15T22:36:14.641349+0000 osd.2 (osd.2) 52 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:36:16.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[62156]: audit 2021-03-15T22:36:15.233337+0000 mon.a (mon.0) 2541 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-15T22:36:16.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[62156]: audit 2021-03-15T22:36:15.300333+0000 mon.a (mon.0) 2542 : audit [DBG] from='client.? 172.21.15.76:0/1369159702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:16.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[62156]: audit 2021-03-15T22:36:15.354958+0000 mon.a (mon.0) 2543 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-15T22:36:16.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:15 smithi076 conmon[62156]: cluster 2021-03-15T22:36:15.355008+0000 mon.a (mon.0) 2544 : cluster [DBG] osdmap e1020: 8 total, 8 up, 8 in 2021-03-15T22:36:16.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: cluster 2021-03-15T22:36:13.797836+0000 osd.7 (osd.7) 191 : cluster [DBG] 5.9 scrub ok 2021-03-15T22:36:16.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: cluster 2021-03-15T22:36:14.097427 2021-03-15T22:36:16.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: +0000 osd.0 (osd.0) 164 : cluster [DBG] 4.b scrub ok 2021-03-15T22:36:16.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: cluster 2021-03-15T22:36:14.641349+0000 osd.2 (osd.2) 52 : cluster [DBG] 7.8 scrub ok 2021-03-15T22:36:16.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:36:16.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: T22:36:15.233337+0000 mon.a (mon.0) 2541 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]: dispatch 2021-03-15T22:36:16.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:36:16.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: -15T22:36:15.300333+0000 mon.a 2021-03-15T22:36:16.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: (mon.0) 2542 : audit [DBG] from='client.? 172.21.15.76:0/1369159702' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:16.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: 2021-03-15T22:36:16.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: audit 2021-03-15T22:36:16.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: 2021-03-15T22:36:15.354958 2021-03-15T22:36:16.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2543 : audit [INF] 2021-03-15T22:36:16.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "26"}]': finished 2021-03-15T22:36:16.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: cluster 2021-03-15T22:36 2021-03-15T22:36:16.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: :15.355008+0000 mon.a (mon.0) 2544 2021-03-15T22:36:16.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:15 smithi125 conmon[56739]: : cluster [DBG] osdmap e1020: 8 total, 8 up, 8 in 2021-03-15T22:36:16.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:16.144 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:16.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:16.146 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:16.247 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:16.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:16.272 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:16.272 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:16.372 INFO:tasks.rados.rados.0.smithi076.stdout:3923: done (4 left) 2021-03-15T22:36:16.372 INFO:tasks.rados.rados.0.smithi076.stdout:3925: done (3 left) 2021-03-15T22:36:16.373 INFO:tasks.rados.rados.0.smithi076.stdout:3926: done (2 left) 2021-03-15T22:36:16.373 INFO:tasks.rados.rados.0.smithi076.stdout:3927: done (1 left) 2021-03-15T22:36:16.373 INFO:tasks.rados.rados.0.smithi076.stdout:3928: done (0 left) 2021-03-15T22:36:16.373 INFO:tasks.rados.rados.0.smithi076.stdout:3929: write oid 47 current snap is 463 2021-03-15T22:36:16.374 INFO:tasks.rados.rados.0.smithi076.stdout:3929: seq_num 1332 ranges {786924=593443,1928721=458203} 2021-03-15T22:36:16.375 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:16.379 INFO:tasks.rados.rados.0.smithi076.stdout:3929: writing smithi07694878-47 from 786924 to 1380367 tid 1 2021-03-15T22:36:16.385 INFO:tasks.rados.rados.0.smithi076.stdout:3929: writing smithi07694878-47 from 1928721 to 2386924 tid 2 2021-03-15T22:36:16.385 INFO:tasks.rados.rados.0.smithi076.stdout:3929: finishing write tid 1 to smithi07694878-47 2021-03-15T22:36:16.386 INFO:tasks.rados.rados.0.smithi076.stdout:3930: rollback oid 50 current snap is 463 2021-03-15T22:36:16.386 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 50 to 416 2021-03-15T22:36:16.386 INFO:tasks.rados.rados.0.smithi076.stdout:3931: read oid 37 snap -1 2021-03-15T22:36:16.386 INFO:tasks.rados.rados.0.smithi076.stdout:3931: expect (ObjNum 1331 snap 462 seq_num 1331) 2021-03-15T22:36:16.387 INFO:tasks.rados.rados.0.smithi076.stdout:3932: snap_remove snap 426 2021-03-15T22:36:16.403 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:16.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:16.405 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:16.419 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:16.443 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_blocked_ops 2021-03-15T22:36:16.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:16.675 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:16.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:16.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:16.778 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:16.804 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:16.805 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:16.806 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:16.907 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:16.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:16.933 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:16.933 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:17.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[56514]: cluster 2021-03-15T22: 2021-03-15T22:36:17.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[56514]: 36:14.798784+0000 osd.7 (osd.7) 192 : cluster [DBG] 3.f scrub ok 2021-03-15T22:36:17.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[56514]: cluster 2021-03-15T22:36:15.114635+0000 osd.0 (osd.0) 165 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:36:17.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[56514]: cluster 2021-03-15T22:36:15.232441+0000 mgr.y (mgr.14144) 1118 : cluster [DBG] pgmap v1676: 145 pgs: 2 peering, 143 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2021-03-15T22:36:17.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[56514]: cluster 2021-03-15T22:36:15.261668+0000 osd.4 (osd.4) 202 : cluster [DBG] 4.d scrub ok 2021-03-15T22:36:17.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[56514]: cluster 2021-03-15T22:36:16.067776+0000 osd.5 (osd.5) 63 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:36:17.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:36:17.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[56514]: -15T22:36:16.369507+0000 mon.a (mon.0) 2545 : cluster [DBG] osdmap e1021: 8 total, 8 up, 8 in 2021-03-15T22:36:17.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: cluster 2021-03-15T22:36:14.798784+0000 osd.7 (osd 2021-03-15T22:36:17.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: .7) 192 : cluster [DBG] 3.f scrub ok 2021-03-15T22:36:17.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: cluster 2021-03-15T22:36:15.114635+0000 osd.0 (osd 2021-03-15T22:36:17.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: .0) 165 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:36:17.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:36:17.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: :36:15.232441+0000 mgr.y (mgr.14144 2021-03-15T22:36:17.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: ) 1118 : cluster [DBG] pgmap v1676: 145 pgs: 2 peering, 143 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2021-03-15T22:36:17.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: cluster 2021-03-15T22:36:17.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: 2021-03-15T 2021-03-15T22:36:17.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: 22:36:15.261668+0000 osd.4 (osd 2021-03-15T22:36:17.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: .4) 202 : cluster [DBG] 4.d scrub ok 2021-03-15T22:36:17.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: cluster 2021-03 2021-03-15T22:36:17.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: -15T22:36:16.067776+0000 osd.5 (osd 2021-03-15T22:36:17.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: .5) 63 : cluster [DBG] 7.0 scrub ok 2021-03-15T22:36:17.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:36:17.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: 03-15T22:36:16.369507+0000 mon.a 2021-03-15T22:36:17.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: (mon.0) 2545 : cluster [DBG] osdmap e1021: 8 total, 8 up, 8 in 2021-03-15T22:36:17.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:16 smithi076 conmon[62156]: 2021-03-15T22:36:17.035 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:17.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: cluster 2021-03-15T22:36:14.798784+0000 osd.7 (osd.7) 192 : cluster 2021-03-15T22:36:17.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: [DBG] 3.f scrub ok 2021-03-15T22:36:17.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: cluster 2021-03-15T22:36:15.114635+0000 osd.0 (osd.0) 165 : cluster [DBG] 7.4 scrub ok 2021-03-15T22:36:17.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: cluster 2021-03-15T22:36: 2021-03-15T22:36:17.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: 15.232441+0000 mgr.y (mgr.14144) 1118 : cluster [DBG] pgmap v1676: 145 pgs: 2 peering, 143 active+clean; 761 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.9 MiB/s wr, 16 op/s 2021-03-15T22:36:17.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: cluster 2021-03-15T22:36:15 2021-03-15T22:36:17.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: .261668+0000 osd.4 (osd.4) 202 : cluster 2021-03-15T22:36:17.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: [DBG] 4.d scrub ok 2021-03-15T22:36:17.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:36:17.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: 36:16.067776+0000 osd.5 (osd.5) 63 2021-03-15T22:36:17.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: : cluster [DBG] 7.0 scrub ok 2021-03-15T22:36:17.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:36:17.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: T22:36:16.369507+0000 mon.a (mon 2021-03-15T22:36:17.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:16 smithi125 conmon[56739]: .0) 2545 : cluster [DBG] osdmap e1021: 8 total, 8 up, 8 in 2021-03-15T22:36:17.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:17.066 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:17.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:17.135 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:17.170 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:17.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:17.194 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:17.194 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:17.296 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:17.318 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:17.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:17.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:17.382 INFO:tasks.rados.rados.0.smithi076.stdout:3932: done (3 left) 2021-03-15T22:36:17.382 INFO:tasks.rados.rados.0.smithi076.stdout:3933: write (excl) oid 17 current snap is 463 2021-03-15T22:36:17.382 INFO:tasks.rados.rados.0.smithi076.stdout:3933: seq_num 1333 ranges {403731=668692,1587203=587797} 2021-03-15T22:36:17.388 INFO:tasks.rados.rados.0.smithi076.stdout:3933: writing smithi07694878-17 from 403731 to 1072423 tid 1 2021-03-15T22:36:17.395 INFO:tasks.rados.rados.0.smithi076.stdout:3933: writing smithi07694878-17 from 1587203 to 2175000 tid 2 2021-03-15T22:36:17.395 INFO:tasks.rados.rados.0.smithi076.stdout:3934: snap_remove snap 445 2021-03-15T22:36:17.421 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:17.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:17.447 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:17.448 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:17.455 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:36:17.549 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:17.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:17.576 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:17.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:17.678 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:17.703 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:17.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:17.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:17.806 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:17.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:17.830 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:17.831 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:17.933 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:17.955 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:17.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:17.956 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:18.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: cluster 2021-03-15T22:36:16.071883+0000 osd.0 (osd. 2021-03-15T22:36:18.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: 0) 166 : cluster [DBG] 5.1e scrub ok 2021-03-15T22:36:18.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: cluster 2021-03-15T22:36:16.252574+0000 osd.4 (osd.4) 203 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:36:18.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: audit 2021-03-15T22:36:16.960225+0000 mon.a (mon.0) 2546 : audit [INF] from='client.? 172.21.15.76:0/4241541601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:18.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: cluster 2021-03-15T22:36:17.148375+0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:18.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: cluster 2021-03-15T22:36:17.150533+0000 osd.1 (osd.1) 192 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:18.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: audit 2021-03-15T22:36:17.233697+0000 mon.a (mon.0) 2547 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "32"}]: dispatch 2021-03-15T22:36:18.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: cluster 2021-03-15T22:36:17.367020+0000 mon.a (mon.0) 2548 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:36:18.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: cluster 2021-03-15T22:36:17.367049+0000 mon.a (mon.0) 2549 : cluster [INF] Cluster is now healthy 2021-03-15T22:36:18.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: audit 2021-03-15T22:36:17.373657+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? 172.21.15.76:0/4241541601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:36:18.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: audit 2021-03-15T22:36:17.373730+0000 mon.a (mon.0) 2551 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "32"}]': finished 2021-03-15T22:36:18.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[56514]: cluster 2021-03-15T22:36:17.373781+0000 mon.a (mon.0) 2552 : cluster [DBG] osdmap e1022: 8 total, 8 up, 8 in 2021-03-15T22:36:18.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: cluster 2021-03-15T22:36:16.071883+0000 osd.0 ( 2021-03-15T22:36:18.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: osd.0) 166 : cluster [DBG] 5.1e scrub ok 2021-03-15T22:36:18.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: cluster 2021-03-15T22:36:16.252574+0000 osd.4 (osd.4) 203 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:36:18.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: audit 2021-03-15T22:36:16.960225+0000 mon.a (mon.0) 2546 : audit [INF] from='client.? 172.21.15.76:0/4241541601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:18.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: cluster 2021-03-15T22:36:17.148375+0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:18.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: cluster 2021-03-15T22:36:17.150533+0000 osd.1 (osd.1) 192 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:18.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: audit 2021-03-15T22:36:17.233697+0000 mon.a (mon.0) 2547 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "32"}]: dispatch 2021-03-15T22:36:18.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: cluster 2021-03-15T22:36:17.367020+0000 mon.a (mon.0) 2548 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:36:18.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: cluster 2021-03-15T22:36:17.367049+0000 mon.a (mon.0) 2549 : cluster 2021-03-15T22:36:18.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: [INF] Cluster is now healthy 2021-03-15T22:36:18.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: audit 2021-03-15T22:36:17.373657+0000 mon.a (mon.0) 2550 : audit [INF] from='client.? 172.21.15.76:0/4241541601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:36:18.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: audit 2021-03-15T22:36:18.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: 2021-03-15T22:36:17.373730+0000 mon.a (mon.0) 2021-03-15T22:36:18.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: 2551 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "32"}]': finished 2021-03-15T22:36:18.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:36:18.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: 22:36:17.373781+0000 mon.a (mon. 2021-03-15T22:36:18.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:17 smithi076 conmon[62156]: 0) 2552 : cluster [DBG] osdmap e1022: 8 total, 8 up, 8 in 2021-03-15T22:36:18.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: cluster 2021-03-15T22:36:16.071883+0000 osd.0 (osd.0) 166 : cluster [DBG] 5.1e scrub ok 2021-03-15T22:36:18.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: cluster 2021-03-15T22:36:16.252574+0000 osd.4 (osd.4) 203 : cluster [DBG] 3.3 scrub ok 2021-03-15T22:36:18.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: audit 2021-03-15T22:36:16.960225+0000 mon.a ( 2021-03-15T22:36:18.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: mon.0) 2546 : audit [INF] from='client.? 172.21.15.76:0/4241541601' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:18.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: cluster 2021-03-15T22:36:17.148375 2021-03-15T22:36:18.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: +0000 osd.1 (osd.1) 191 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:18.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:36:18.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: 15T22:36:17.150533+0000 osd.1 (osd. 2021-03-15T22:36:18.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: 1) 192 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:18.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: audit 2021-03-15T22:36:18.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: 2021-03-15T22:36:17.233697+0000 mon.a 2021-03-15T22:36:18.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: (mon.0) 2547 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "32"}]: dispatch 2021-03-15T22:36:18.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: cluster 2021-03-15T22:36:17. 2021-03-15T22:36:18.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: 367020+0000 mon.a (mon.0) 2548 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:36:18.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:36:18.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: -15T22:36:17.367049+0000 mon.a (mon. 2021-03-15T22:36:18.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: 0) 2549 : cluster [INF] Cluster is now healthy 2021-03-15T22:36:18.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:36:18.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: 03-15T22:36:17.373657+0000 mon.a (mon. 2021-03-15T22:36:18.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: 0) 2550 : audit [INF] from='client.? 172.21.15.76:0/4241541601' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:36:18.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: audit 2021-03-15T22:36:18.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: 2021-03-15T22:36:17.373730+0000 2021-03-15T22:36:18.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: mon.a (mon.0) 2551 : audit [INF] 2021-03-15T22:36:18.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_num_actual", "val": "32"}]': finished 2021-03-15T22:36:18.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: cluster 2021-03-15T22: 2021-03-15T22:36:18.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: 36:17.373781+0000 mon.a (mon.0) 2021-03-15T22:36:18.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:17 smithi125 conmon[56739]: 2552 : cluster [DBG] osdmap e1022: 8 total, 8 up, 8 in 2021-03-15T22:36:18.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:18.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:18.081 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:18.081 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:18.183 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:18.205 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:18.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:18.206 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:18.308 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:18.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:18.331 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:18.332 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:18.381 INFO:tasks.rados.rados.0.smithi076.stdout:3934: done (4 left) 2021-03-15T22:36:18.381 INFO:tasks.rados.rados.0.smithi076.stdout:3935: snap_create 2021-03-15T22:36:18.381 INFO:tasks.rados.rados.0.smithi076.stdout:3930: finishing rollback tid 0 to smithi07694878-50 2021-03-15T22:36:18.382 INFO:tasks.rados.rados.0.smithi076.stdout:3930: finishing rollback tid 1 to smithi07694878-50 2021-03-15T22:36:18.382 INFO:tasks.rados.rados.0.smithi076.stdout:3930: finishing rollback tid 2 to smithi07694878-50 2021-03-15T22:36:18.382 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 50 v 2693 (ObjNum 1076 snap 366 seq_num 1076) dirty exists 2021-03-15T22:36:18.382 INFO:tasks.rados.rados.0.smithi076.stdout:3929: finishing write tid 2 to smithi07694878-47 2021-03-15T22:36:18.383 INFO:tasks.rados.rados.0.smithi076.stdout:3929: finishing write tid 3 to smithi07694878-47 2021-03-15T22:36:18.383 INFO:tasks.rados.rados.0.smithi076.stdout:3929: finishing write tid 4 to smithi07694878-47 2021-03-15T22:36:18.383 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 47 v 3168 (ObjNum 1332 snap 463 seq_num 1332) dirty exists 2021-03-15T22:36:18.383 INFO:tasks.rados.rados.0.smithi076.stdout:3929: left oid 47 (ObjNum 1332 snap 463 seq_num 1332) 2021-03-15T22:36:18.383 INFO:tasks.rados.rados.0.smithi076.stdout:3931: expect (ObjNum 1331 snap 462 seq_num 1331) 2021-03-15T22:36:18.433 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:18.456 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:18.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:18.514 INFO:tasks.rados.rados.0.smithi076.stdout:3933: finishing write tid 1 to smithi07694878-17 2021-03-15T22:36:18.514 INFO:tasks.rados.rados.0.smithi076.stdout:3933: finishing write tid 2 to smithi07694878-17 2021-03-15T22:36:18.514 INFO:tasks.rados.rados.0.smithi076.stdout:3933: finishing write tid 3 to smithi07694878-17 2021-03-15T22:36:18.515 INFO:tasks.rados.rados.0.smithi076.stdout:3933: finishing write tid 4 to smithi07694878-17 2021-03-15T22:36:18.515 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 17 v 3187 (ObjNum 1333 snap 463 seq_num 1333) dirty exists 2021-03-15T22:36:18.515 INFO:tasks.rados.rados.0.smithi076.stdout:3933: left oid 17 (ObjNum 1333 snap 463 seq_num 1333) 2021-03-15T22:36:18.559 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:18.585 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:18.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:18.587 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:18.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:18.712 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:18.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:18.713 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:18.814 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:18.837 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:18.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:18.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:18.839 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:36:18.940 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:18.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:18.967 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:18.968 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:19.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[56514]: cluster 2021-03-15T22:36:16.818671+0000 osd.7 (osd.7) 193 : cluster [DBG] 7.5 scrub ok 2021-03-15T22:36:19.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[56514]: cluster 2021-03-15T22:36:17.232995+0000 mgr.y (mgr.14144) 1119 : cluster [DBG] pgmap v1679: 144 pgs: 2 peering, 142 active+clean; 773 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-03-15T22:36:19.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[56514]: cluster 2021-03-15T22:36:17.753369+0000 osd.6 (osd.6) 154 : cluster [DBG] 4.1f scrub ok 2021-03-15T22:36:19.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[56514]: cluster 2021-03-15T22:36:18.276094+0000 osd.1 (osd.1) 193 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:36:19.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[56514]: cluster 2021-03-15T22:36:18.371443+0000 mon.a (mon.0) 2553 : cluster [DBG] osdmap e1023: 8 total, 8 up, 8 in 2021-03-15T22:36:19.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: cluster 2021-03-15T22:36:16.818671 2021-03-15T22:36:19.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: +0000 osd.7 (osd.7) 193 : cluster [DBG] 7.5 scrub ok 2021-03-15T22:36:19.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: cluster 2021-03-15T22:36:17.232995 2021-03-15T22:36:19.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 1119 : cluster [DBG] 2021-03-15T22:36:19.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: pgmap v1679: 144 pgs: 2 peering, 142 active+clean; 773 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-03-15T22:36:19.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:36:19.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: :36:17.753369+0000 osd.6 (osd.6) 154 : 2021-03-15T22:36:19.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: cluster [DBG] 4.1f scrub ok 2021-03-15T22:36:19.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: cluster 2021-03-15T22:36 2021-03-15T22:36:19.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: :18.276094+0000 osd.1 (osd.1) 193 : 2021-03-15T22:36:19.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: cluster [DBG] 5.19 scrub ok 2021-03-15T22:36:19.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: cluster 2021-03-15T22:36:18 2021-03-15T22:36:19.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: .371443+0000 mon.a (mon.0) 2553 : cluster [DBG] 2021-03-15T22:36:19.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:18 smithi076 conmon[62156]: osdmap e1023: 8 total, 8 up, 8 in 2021-03-15T22:36:19.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: cluster 2021-03-15T22:36:16.818671+0000 osd.7 (osd 2021-03-15T22:36:19.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: .7) 193 : cluster [DBG] 7.5 scrub ok 2021-03-15T22:36:19.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: cluster 2021-03-15T22:36:17.232995+0000 mgr.y (mgr.14144) 1119 : cluster 2021-03-15T22:36:19.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: [DBG] pgmap v1679: 144 pgs: 2 peering, 142 active+clean; 773 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 10 MiB/s rd, 1.7 MiB/s wr, 15 op/s 2021-03-15T22:36:19.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: cluster 2021-03-15T22:36:17. 2021-03-15T22:36:19.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: 753369+0000 osd.6 (osd.6) 154 : cluster [DBG] 2021-03-15T22:36:19.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: 4.1f scrub ok 2021-03-15T22:36:19.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: cluster 2021-03-15T22:36 2021-03-15T22:36:19.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: :18.276094+0000 osd.1 (osd.1) 193 : cluster [DBG] 5.19 scrub ok 2021-03-15T22:36:19.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: cluster 2021-03-15T22:36:19.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: 2021-03-15T22:36:18.371443+0000 2021-03-15T22:36:19.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: mon.a (mon.0) 2553 : cluster [DBG] osdmap e1023: 8 total, 8 up, 8 in 2021-03-15T22:36:19.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:18 smithi125 conmon[56739]: 2021-03-15T22:36:19.069 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:19.094 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:19.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:19.096 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:19.198 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:19.223 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:19.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:19.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:19.326 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:19.349 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:19.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:19.351 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:19.378 INFO:tasks.rados.rados.0.smithi076.stdout:3929: done (4 left) 2021-03-15T22:36:19.379 INFO:tasks.rados.rados.0.smithi076.stdout:3930: done (3 left) 2021-03-15T22:36:19.379 INFO:tasks.rados.rados.0.smithi076.stdout:3931: done (2 left) 2021-03-15T22:36:19.379 INFO:tasks.rados.rados.0.smithi076.stdout:3933: done (1 left) 2021-03-15T22:36:19.380 INFO:tasks.rados.rados.0.smithi076.stdout:3935: done (0 left) 2021-03-15T22:36:19.380 INFO:tasks.rados.rados.0.smithi076.stdout:3936: rollback oid 31 current snap is 464 2021-03-15T22:36:19.380 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 31 to 423 2021-03-15T22:36:19.380 INFO:tasks.rados.rados.0.smithi076.stdout:3937: snap_remove snap 440 2021-03-15T22:36:19.452 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:19.477 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:19.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:19.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:19.512 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:19.580 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:19.606 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:19.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:19.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:19.709 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:19.735 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:19.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:19.737 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:19.838 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:19.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:19.862 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:19.863 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:19.896 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[56514]: cluster 2021-03-15T22:36:17.843199 2021-03-15T22:36:19.896 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[56514]: +0000 osd.7 (osd.7) 194 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:36:19.897 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[56514]: cluster 2021-03-15T22:36:18.800815+0000 osd.6 (osd.6) 155 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:36:19.897 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[56514]: audit 2021-03-15T22:36:19.234446+0000 mon.a (mon.0) 2554 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-03-15T22:36:19.897 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[56514]: audit 2021-03-15T22:36:19.375061+0000 mon.a (mon.0) 2555 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]': finished 2021-03-15T22:36:19.897 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[56514]: cluster 2021-03-15T22:36:19.375125+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e1024: 8 total, 8 up, 8 in 2021-03-15T22:36:19.898 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[62156]: cluster 2021-03-15T22:36:17.843199+0000 2021-03-15T22:36:19.898 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[62156]: osd.7 (osd.7) 194 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:36:19.898 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[62156]: cluster 2021-03-15T22:36:18.800815+0000 osd.6 (osd.6) 155 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:36:19.898 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[62156]: audit 2021-03-15T22:36:19.234446+0000 mon.a (mon.0) 2554 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-03-15T22:36:19.899 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[62156]: audit 2021-03-15T22:36:19.375061+0000 mon.a (mon.0) 2555 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]': finished 2021-03-15T22:36:19.899 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:19 smithi076 conmon[62156]: cluster 2021-03-15T22:36:19.375125+0000 mon.a (mon.0) 2556 : cluster [DBG] osdmap e1024: 8 total, 8 up, 8 in 2021-03-15T22:36:19.964 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:19.990 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:19.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:19.992 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:20.018 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:36:20.019 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:36:20.019 INFO:teuthology.orchestra.run.smithi076.stdout: "complaint_time": 30, 2021-03-15T22:36:20.019 INFO:teuthology.orchestra.run.smithi076.stdout: "num_blocked_ops": 0 2021-03-15T22:36:20.020 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:36:20.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:19 smithi125 conmon[56739]: cluster 2021-03-15T22:36:17.843199+0000 osd.7 (osd.7) 194 : cluster [DBG] 5.14 scrub ok 2021-03-15T22:36:20.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:19 smithi125 conmon[56739]: cluster 2021-03-15T22:36:18.800815+0000 osd.6 (osd.6) 155 : cluster [DBG] 4.12 scrub ok 2021-03-15T22:36:20.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:19 smithi125 conmon[56739]: audit 2021-03-15T22:36:19.234446+0000 mon.a (mon. 2021-03-15T22:36:20.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:19 smithi125 conmon[56739]: 0) 2554 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]: dispatch 2021-03-15T22:36:20.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:19 smithi125 conmon[56739]: audit 2021-03-15T22:36:19.375061+0000 mon.a (mon 2021-03-15T22:36:20.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:19 smithi125 conmon[56739]: .0) 2555 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "25"}]': finished 2021-03-15T22:36:20.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:19 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:36:20.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:19 smithi125 conmon[56739]: 03-15T22:36:19.375125+0000 mon.a (mon. 2021-03-15T22:36:20.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:19 smithi125 conmon[56739]: 0) 2556 : cluster [DBG] osdmap e1024: 8 total, 8 up, 8 in 2021-03-15T22:36:20.093 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:20.119 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:20.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:20.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:20.221 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:20.243 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:20.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:20.245 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:20.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:20.365 INFO:tasks.rados.rados.0.smithi076.stdout:3937: done (1 left) 2021-03-15T22:36:20.366 INFO:tasks.rados.rados.0.smithi076.stdout:3938: snap_remove snap 447 2021-03-15T22:36:20.376 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:20.377 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 1 pg_num to 11 2021-03-15T22:36:20.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:20.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:20.479 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:20.480 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:36:20.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:20.505 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:20.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:20.607 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:20.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:20.635 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:20.635 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:20.719 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:36:20.736 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:20.759 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:20.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:20.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[56514]: cluster 2021-03-15T22:36:18.835121+0000 osd.7 (osd.7) 195 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:36:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[56514]: cluster 2021-03-15T22:36:19.035903+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:36:20.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[56514]: cluster 2021-03-15T22:36:19.233649+0000 mgr.y (mgr.14144) 1120 : cluster [DBG] pgmap v1682: 143 pgs: 1 peering, 142 active+clean; 775 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 9 op/s; 268 B/s, 0 objects/s recovering 2021-03-15T22:36:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[56514]: audit 2021-03-15T22:36:19.860132+0000 mon.a (mon.0) 2557 : audit [INF] from='client.? 172.21.15.76:0/789532442' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]: dispatch 2021-03-15T22:36:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[56514]: cluster 2021-03-15T22:36:20.050445+0000 osd.5 (osd.5) 64 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:36:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[56514]: cluster 2021-03-15T22:36:20.132177+0000 osd.4 (osd. 2021-03-15T22:36:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[56514]: 4) 204 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:36:20.769 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[56514]: cluster 2021-03-15T22:36:20.306354+0000 osd.1 (osd.1) 194 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:36:20.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[56514]: audit 2021-03-15T22:36:20.358052+0000 mon.a (mon.0) 2558 : audit [INF] from='client.? 172.21.15.76:0/789532442' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]': finished 2021-03-15T22:36:20.770 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[56514]: cluster 2021-03-15T22:36:20.358110+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in 2021-03-15T22:36:20.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[62156]: cluster 2021-03-15T22:36:18.835121+0000 osd.7 (osd.7) 195 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:36:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[62156]: cluster 2021-03-15T22:36:19.035903+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:36:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[62156]: cluster 2021-03-15T22:36:19.233649+0000 mgr.y (mgr.14144) 1120 : cluster [DBG] pgmap v1682: 143 pgs: 1 peering, 142 active+clean; 775 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 9 op/s; 268 B/s, 0 objects/s recovering 2021-03-15T22:36:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[62156]: audit 2021-03-15T22:36:19.860132+0000 mon.a (mon.0) 2557 : audit [INF] from='client.? 172.21.15.76:0/789532442' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]: dispatch 2021-03-15T22:36:20.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[62156]: cluster 2021-03-15T22:36:20.050445+0000 osd.5 (osd.5) 64 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:36:20.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[62156]: cluster 2021-03-15T22:36:20.132177+0000 osd.4 (osd.4) 204 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:36:20.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[62156]: cluster 2021-03-15T22:36:20.306354+0000 osd.1 (osd.1) 194 : cluster [DBG] 3.18 scrub ok 2021-03-15T22:36:20.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[62156]: audit 2021-03-15T22:36:20.358052+0000 mon.a (mon.0) 2558 : audit [INF] from='client.? 172.21.15.76:0/789532442' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]': finished 2021-03-15T22:36:20.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:20 smithi076 conmon[62156]: cluster 2021-03-15T22:36:20.358110+0000 mon.a (mon.0) 2559 : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in 2021-03-15T22:36:20.862 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:20.890 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:20.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:20.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:20.994 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:21.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:21.020 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:21.021 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:21.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: cluster 2021-03-15T22:36:18.835121+0000 osd.7 (osd.7) 195 : cluster [DBG] 6.8 scrub ok 2021-03-15T22:36:21.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: 2021-03-15T22:36:21.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: cluster 2021-03-15T22:36:19.035903+0000 osd.0 (osd.0) 167 : cluster [DBG] 3.16 scrub ok 2021-03-15T22:36:21.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:36:21.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: 03-15T22:36:19.233649+0000 mgr.y (mgr.14144) 1120 : cluster [DBG] pgmap v1682: 143 pgs: 1 peering, 142 active+clean; 775 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 5.0 MiB/s rd, 1.5 MiB/s wr, 9 op/s; 268 B/s, 0 objects/s recovering 2021-03-15T22:36:21.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:36:21.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: T22:36:19.860132+0000 mon.a (mon.0 2021-03-15T22:36:21.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: ) 2557 : audit [INF] from='client.? 172.21.15.76:0/789532442' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]: dispatch 2021-03-15T22:36:21.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: cluster 2021-03-15T22:36:20.050445+0000 osd.5 (osd.5) 64 : cluster [DBG] 4.7 scrub ok 2021-03-15T22:36:21.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: cluster 2021-03-15T22:36:20.132177+0000 osd.4 (osd.4) 204 : cluster [DBG] 3.14 scrub ok 2021-03-15T22:36:21.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: cluster 2021-03-15T22:36:20.306354+0000 osd.1 (osd.1) 194 : cluster 2021-03-15T22:36:21.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: [DBG] 3.18 scrub ok 2021-03-15T22:36:21.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:36:21.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: T22:36:20.358052+0000 mon.a (mon. 2021-03-15T22:36:21.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: 0) 2558 : audit [INF] from='client.? 172.21.15.76:0/789532442' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num", "val": "11"}]': finished 2021-03-15T22:36:21.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: cluster 2021-03-15T22:36:21.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:36:21.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: 36:20.358110+0000 mon.a (mon.0) 2021-03-15T22:36:21.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:20 smithi125 conmon[56739]: 2559 : cluster [DBG] osdmap e1025: 8 total, 8 up, 8 in 2021-03-15T22:36:21.122 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:21.144 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:21.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:21.146 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:21.247 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:21.260 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.0 dump_historic_ops 2021-03-15T22:36:21.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:21.270 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:21.271 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:21.373 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:21.375 INFO:tasks.rados.rados.0.smithi076.stdout:3938: done (1 left) 2021-03-15T22:36:21.375 INFO:tasks.rados.rados.0.smithi076.stdout:3939: read oid 44 snap -1 2021-03-15T22:36:21.375 INFO:tasks.rados.rados.0.smithi076.stdout:3939: expect (ObjNum 1326 snap 462 seq_num 1326) 2021-03-15T22:36:21.375 INFO:tasks.rados.rados.0.smithi076.stdout:3940: rollback oid 5 current snap is 464 2021-03-15T22:36:21.376 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 5 to 438 2021-03-15T22:36:21.376 INFO:tasks.rados.rados.0.smithi076.stdout:3941: delete oid 21 current snap is 464 2021-03-15T22:36:21.376 INFO:tasks.rados.rados.0.smithi076.stdout:3936: finishing rollback tid 1 to smithi07694878-31 2021-03-15T22:36:21.376 INFO:tasks.rados.rados.0.smithi076.stdout:3936: finishing rollback tid 2 to smithi07694878-31 2021-03-15T22:36:21.377 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 31 v 3088 (ObjNum 1216 snap 421 seq_num 1216) dirty exists 2021-03-15T22:36:21.395 INFO:tasks.rados.rados.0.smithi076.stdout:3939: expect (ObjNum 1326 snap 462 seq_num 1326) 2021-03-15T22:36:21.411 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:21.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:21.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:21.506 INFO:tasks.rados.rados.0.smithi076.stdout:3940: finishing rollback tid 0 to smithi07694878-5 2021-03-15T22:36:21.507 INFO:tasks.rados.rados.0.smithi076.stdout:3940: finishing rollback tid 1 to smithi07694878-5 2021-03-15T22:36:21.507 INFO:tasks.rados.rados.0.smithi076.stdout:3940: finishing rollback tid 2 to smithi07694878-5 2021-03-15T22:36:21.507 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 5 v 2829 (ObjNum 1238 snap 427 seq_num 1238) dirty exists 2021-03-15T22:36:21.507 INFO:tasks.rados.rados.0.smithi076.stdout:3936: done (3 left) 2021-03-15T22:36:21.508 INFO:tasks.rados.rados.0.smithi076.stdout:3939: done (2 left) 2021-03-15T22:36:21.508 INFO:tasks.rados.rados.0.smithi076.stdout:3940: done (1 left) 2021-03-15T22:36:21.508 INFO:tasks.rados.rados.0.smithi076.stdout:3941: done (0 left) 2021-03-15T22:36:21.508 INFO:tasks.rados.rados.0.smithi076.stdout:3942: read oid 18 snap -1 2021-03-15T22:36:21.508 INFO:tasks.rados.rados.0.smithi076.stdout:3942: expect (ObjNum 1308 snap 459 seq_num 1308) 2021-03-15T22:36:21.509 INFO:tasks.rados.rados.0.smithi076.stdout:3943: snap_remove snap 369 2021-03-15T22:36:21.514 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:21.539 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:21.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:21.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:21.642 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:21.668 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:21.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:21.670 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:21.771 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:21.793 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:21.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:21.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:21.896 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:21.918 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:21.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:21.919 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:22.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[56514]: cluster 2021-03-15T22:36:20.059740+0000 2021-03-15T22:36:22.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[56514]: osd.0 (osd.0) 168 : cluster [DBG] 7.f scrub ok 2021-03-15T22:36:22.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[56514]: audit 2021-03-15T22:36:20.339705+0000 mgr.y (mgr.14144) 1121 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:22.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[56514]: cluster 2021-03-15T22:36:20.962676+0000 osd.6 (osd.6) 156 : cluster [DBG] 7.7 deep-scrub ok 2021-03-15T22:36:22.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[56514]: cluster 2021-03-15T22:36:21.176396+0000 osd.4 (osd.4) 205 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:36:22.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[56514]: audit 2021-03-15T22:36:21.234878+0000 mon.a (mon.0) 2560 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]: dispatch 2021-03-15T22:36:22.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[56514]: cluster 2021-03-15T22:36:21.293214+0000 osd.1 (osd.1) 195 : cluster [DBG] 4.f scrub ok 2021-03-15T22:36:22.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[56514]: audit 2021-03-15T22:36:21.369914+0000 mon.a (mon.0) 2561 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]': finished 2021-03-15T22:36:22.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[56514]: cluster 2021-03-15T22:36:21.369976+0000 mon.a (mon.0) 2562 : cluster [DBG] osdmap e1026: 8 total, 8 up, 8 in 2021-03-15T22:36:22.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: cluster 2021-03-15T22:36:20.059740+0000 osd.0 2021-03-15T22:36:22.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: (osd.0) 168 : cluster [DBG] 7.f scrub ok 2021-03-15T22:36:22.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: audit 2021-03-15T22:36:20.339705+0000 mgr.y (mgr 2021-03-15T22:36:22.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: .14144) 1121 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:22.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:36:22.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: 03-15T22:36:20. 2021-03-15T22:36:22.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: 962676+0000 osd.6 (osd.6) 2021-03-15T22:36:22.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: 156 : cluster [DBG] 7.7 deep-scrub ok 2021-03-15T22:36:22.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:36:22.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: -03-15T22:36:21.176396 2021-03-15T22:36:22.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: +0000 osd.4 (osd.4) 205 : cluster 2021-03-15T22:36:22.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: [DBG] 4.1a scrub ok 2021-03-15T22:36:22.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:36:22.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: T22:36:21.234878+0000 mon.a ( 2021-03-15T22:36:22.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: mon.0) 2560 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]: dispatch 2021-03-15T22:36:22.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: cluster 2021-03-15T22:36:22.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: 2021-03-15T22:36:21. 2021-03-15T22:36:22.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: 293214+0000 osd.1 (osd.1) 195 : cluster 2021-03-15T22:36:22.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: [DBG] 4.f scrub ok 2021-03-15T22:36:22.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: audit 2021-03-15 2021-03-15T22:36:22.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: T22:36:21.369914+0000 mon.a 2021-03-15T22:36:22.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: (mon.0) 2561 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]': finished 2021-03-15T22:36:22.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: 2021-03-15T22:36:22.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: cluster 2021-03-15T22:36:22.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: 2021-03-15T22:36:21.369976+0000 2021-03-15T22:36:22.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: mon.a (mon.0) 2562 : cluster [DBG] 2021-03-15T22:36:22.026 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:21 smithi076 conmon[62156]: osdmap e1026: 8 total, 8 up, 8 in 2021-03-15T22:36:22.026 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:22.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: cluster 2021-03-15T22:36:20.059740+0000 osd.0 ( 2021-03-15T22:36:22.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: osd.0) 168 : cluster [DBG] 7.f scrub ok 2021-03-15T22:36:22.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: audit 2021-03-15T22:36:20.339705+0000 mgr.y (mgr.14144) 1121 : audit 2021-03-15T22:36:22.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:22.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:36:22.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: T22:36:20.962676+0000 osd.6 (osd.6) 156 : cluster [DBG] 2021-03-15T22:36:22.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: 7.7 deep-scrub ok 2021-03-15T22:36:22.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:36:22.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: :36:21.176396+0000 osd.4 (osd.4) 2021-03-15T22:36:22.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: 205 : cluster [DBG] 4.1a scrub ok 2021-03-15T22:36:22.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: audit 2021 2021-03-15T22:36:22.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: -03-15T22:36:21.234878+0000 mon.a 2021-03-15T22:36:22.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: (mon.0) 2560 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]: dispatch 2021-03-15T22:36:22.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: 2021-03-15T22:36:22.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: cluster 2021-03-15T22:36:21 2021-03-15T22:36:22.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: .293214+0000 osd.1 (osd.1) 195 : cluster 2021-03-15T22:36:22.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: [DBG] 4.f scrub ok 2021-03-15T22:36:22.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:36:22.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: T22:36:21.369914+0000 mon.a (mon 2021-03-15T22:36:22.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: .0) 2561 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pg_num_actual", "val": "11"}]': finished 2021-03-15T22:36:22.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: cluster 2021-03-15T22:36:22.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: 2021-03-15T22:36:21.369976+0000 2021-03-15T22:36:22.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: mon.a (mon.0) 2562 : cluster [DBG] 2021-03-15T22:36:22.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:21 smithi125 conmon[56739]: osdmap e1026: 8 total, 8 up, 8 in 2021-03-15T22:36:22.049 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:22.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:22.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:22.117 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:22.154 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:22.179 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:22.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:22.181 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:22.282 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:22.309 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:22.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:22.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:22.374 INFO:tasks.rados.rados.0.smithi076.stdout:3943: done (1 left) 2021-03-15T22:36:22.374 INFO:tasks.rados.rados.0.smithi076.stdout:3944: copy_from oid 2 from oid 22 current snap is 464 2021-03-15T22:36:22.375 INFO:tasks.rados.rados.0.smithi076.stdout:3942: expect (ObjNum 1308 snap 459 seq_num 1308) 2021-03-15T22:36:22.412 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:22.436 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:22.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:22.438 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:22.539 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:22.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:22.562 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:22.563 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:22.622 INFO:tasks.rados.rados.0.smithi076.stdout:3942: done (1 left) 2021-03-15T22:36:22.623 INFO:tasks.rados.rados.0.smithi076.stdout:3945: snap_create 2021-03-15T22:36:22.623 INFO:tasks.rados.rados.0.smithi076.stdout:3944: finishing copy_from to smithi07694878-2 2021-03-15T22:36:22.623 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 3159 (ObjNum 1225 snap 424 seq_num 1225) dirty exists 2021-03-15T22:36:22.623 INFO:tasks.rados.rados.0.smithi076.stdout:3944: finishing copy_from racing read to smithi07694878-2 2021-03-15T22:36:22.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:22.688 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:22.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:22.689 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:22.769 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:36:22.788 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:36:22.790 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:22.806 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:36:22.812 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:22.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:22.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:22.849 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:36:22.849 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:36:22.860 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:36:22.880 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:36:22.897 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:36:22.898 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:36:22.899 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:36:22.899 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:36:22.899 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:36:22.900 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:36:22.900 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:36:22.900 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:36:22.901 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:36:22.915 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:22.937 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:22.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:22.939 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:22.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:22 smithi076 conmon[56514]: cluster 2021-03-15T22:36:21.234149+0000 mgr.y (mgr.14144) 1122 : cluster [DBG] pgmap v1685: 142 pgs: 142 active+clean; 778 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 506 KiB/s wr, 9 op/s 2021-03-15T22:36:22.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:22 smithi076 conmon[56514]: cluster 2021-03-15T22:36:22.138301+0000 osd.5 (osd.5) 65 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:36:22.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:22 smithi076 conmon[56514]: cluster 2021-03-15T22:36:22.338196+0000 osd.1 (osd.1) 196 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:36:22.962 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:22 smithi076 conmon[56514]: cluster 2021-03-15T22:36:22.370820+0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e1027: 8 total, 8 up, 8 in 2021-03-15T22:36:22.962 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:22 smithi076 conmon[62156]: cluster 2021-03-15T22:36:21.234149+0000 mgr.y (mgr.14144 2021-03-15T22:36:22.963 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:22 smithi076 conmon[62156]: ) 1122 : cluster [DBG] pgmap v1685: 142 pgs: 142 active+clean; 778 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 506 KiB/s wr, 9 op/s 2021-03-15T22:36:22.963 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:22 smithi076 conmon[62156]: cluster 2021-03-15T22:36:22.138301+0000 osd.5 (osd.5) 65 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:36:22.963 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:22 smithi076 conmon[62156]: cluster 2021-03-15T22:36:22.338196+0000 osd.1 (osd.1) 196 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:36:22.963 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:22 smithi076 conmon[62156]: cluster 2021-03-15T22:36:22.370820+0000 mon.a (mon 2021-03-15T22:36:22.964 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:22 smithi076 conmon[62156]: .0) 2563 : cluster [DBG] osdmap e1027: 8 total, 8 up, 8 in 2021-03-15T22:36:23.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:22 smithi125 conmon[56739]: cluster 2021-03-15T22:36:21.234149+0000 mgr.y (mgr.14144) 1122 : cluster 2021-03-15T22:36:23.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:22 smithi125 conmon[56739]: [DBG] pgmap v1685: 142 pgs: 142 active+clean; 778 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 506 KiB/s wr, 9 op/s 2021-03-15T22:36:23.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:22 smithi125 conmon[56739]: cluster 2021-03-15T22:36:22.138301+0000 osd.5 (osd.5) 65 : cluster [DBG] 6.19 scrub ok 2021-03-15T22:36:23.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:22 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:36:23.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:22 smithi125 conmon[56739]: 22:36:22.338196+0000 osd.1 (osd.1) 196 : cluster [DBG] 4.1c scrub ok 2021-03-15T22:36:23.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:22 smithi125 conmon[56739]: cluster 2021-03-15T22:36:22.370820 2021-03-15T22:36:23.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:22 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2563 : cluster [DBG] osdmap e1027: 8 total, 8 up, 8 in 2021-03-15T22:36:23.040 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:23.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:23.063 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:23.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:23.166 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:23.188 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:23.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:23.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:23.291 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:23.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:23.319 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:23.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:23.376 INFO:tasks.rados.rados.0.smithi076.stdout:3944: done (1 left) 2021-03-15T22:36:23.377 INFO:tasks.rados.rados.0.smithi076.stdout:3945: done (0 left) 2021-03-15T22:36:23.377 INFO:tasks.rados.rados.0.smithi076.stdout:3946: read oid 32 snap -1 2021-03-15T22:36:23.377 INFO:tasks.rados.rados.0.smithi076.stdout:3946: expect deleted 2021-03-15T22:36:23.378 INFO:tasks.rados.rados.0.smithi076.stdout:3947: snap_remove snap 411 2021-03-15T22:36:23.421 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:23.444 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:23.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:23.445 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:23.546 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:23.572 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:23.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:23.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:23.675 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:23.700 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:23.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:23.702 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:23.803 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:23.825 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:23.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:23.826 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:23.927 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:23.955 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:23.956 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:24.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[56514]: cluster 2021-03-15T22:36:22.004536+0000 osd.0 (osd.0) 169 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:36:24.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[56514]: 2021-03-15T22:36:24.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[56514]: cluster 2021-03-15T22:36:22.729841+0000 osd.7 (osd.7) 196 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:36:24.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[56514]: audit 2021-03-15T22:36:22.749416+0000 mon.a (mon.0) 2564 : audit [DBG] from='client.? 172.21.15.76:0/1311769517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:24.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[56514]: audit 2021-03-15T22:36:23.235413+0000 mon.a (mon.0) 2565 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-03-15T22:36:24.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[56514]: audit 2021-03-15T22:36:23.235602+0000 mon.a (mon.0) 2566 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]: dispatch 2021-03-15T22:36:24.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[56514]: audit 2021-03-15T22:36:23.373955+0000 mon.a (mon.0) 2567 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]': finished 2021-03-15T22:36:24.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[56514]: audit 2021-03-15T22:36:23.374031+0000 mon.a (mon.0) 2568 : audit [INF] 2021-03-15T22:36:24.019 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[56514]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]': finished 2021-03-15T22:36:24.020 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[56514]: cluster 2021-03-15T22:36:23.374089+0000 mon.a (mon.0) 2569 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in 2021-03-15T22:36:24.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: cluster 2021-03-15T22:36:22.004536+0000 2021-03-15T22:36:24.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: osd.0 (osd.0) 169 : cluster [DBG] 6.16 scrub ok 2021-03-15T22:36:24.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: cluster 2021-03-15T22:36: 2021-03-15T22:36:24.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: 22.729841+0000 osd.7 (osd.7) 196 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:36:24.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: audit 2021-03 2021-03-15T22:36:24.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: -15T22:36:22.749416+0000 mon.a (mon.0) 2564 : audit [DBG] 2021-03-15T22:36:24.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/1311769517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:24.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: audit 2021-03-15T22: 2021-03-15T22:36:24.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: 36:23.235413+0000 mon.a (mon.0) 2565 : audit [INF] 2021-03-15T22:36:24.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-03-15T22:36:24.022 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:36:24.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: 03-15T22:36:23.235602+0000 mon.a (mon.0) 2566 : audit [INF] 2021-03-15T22:36:24.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]: dispatch 2021-03-15T22:36:24.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:36:24.023 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: :36:23.373955+0000 mon.a (mon.0) 2567 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]': finished 2021-03-15T22:36:24.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: audit 2021-03-15T22:36:23.374031+0000 mon.a (mon 2021-03-15T22:36:24.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: .0) 2568 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]': finished 2021-03-15T22:36:24.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: cluster 2021-03-15T22:36:24.024 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: 2021-03-15T22:36:23.374089+0000 mon.a (mon.0) 2569 : cluster [DBG] 2021-03-15T22:36:24.025 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:23 smithi076 conmon[62156]: osdmap e1028: 8 total, 8 up, 8 in 2021-03-15T22:36:24.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: cluster 2021-03-15T22:36:22.004536+0000 osd.0 (osd.0) 169 : cluster [DBG] 2021-03-15T22:36:24.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: 6.16 scrub ok 2021-03-15T22:36:24.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: cluster 2021-03-15T22:36:22.729841+0000 osd.7 (osd.7) 196 : cluster [DBG] 6.1c scrub ok 2021-03-15T22:36:24.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: audit 2021-03-15T22:36:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: 2021-03-15T22:36:22.749416+0000 mon.a (mon.0) 2564 : audit [DBG] 2021-03-15T22:36:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: from='client.? 172.21.15.76:0/1311769517' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: audit 2021-03-15T22:36:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: 2021-03-15T22:36:23.235413+0000 mon.a (mon. 2021-03-15T22:36:24.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: 0) 2565 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]: dispatch 2021-03-15T22:36:24.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:36:24.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: :36:23.235602+0000 mon.a (mon.0 2021-03-15T22:36:24.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: ) 2566 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]: dispatch 2021-03-15T22:36:24.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:36:24.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: :36:23.373955+0000 mon.a (mon.0 2021-03-15T22:36:24.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: ) 2567 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "24"}]': finished 2021-03-15T22:36:24.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:36:24.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: 03-15T22:36:23.374031+0000 2021-03-15T22:36:24.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: mon.a (mon.0) 2568 : audit [INF] 2021-03-15T22:36:24.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "device_health_metrics", "var": "pgp_num_actual", "val": "11"}]': finished 2021-03-15T22:36:24.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: cluster 2021-03-15T22:36:23.374089+0000 2021-03-15T22:36:24.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:23 smithi125 conmon[56739]: mon.a (mon.0) 2569 : cluster [DBG] osdmap e1028: 8 total, 8 up, 8 in 2021-03-15T22:36:24.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:24.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:24.081 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:24.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:24.183 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:24.205 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:24.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:24.207 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:24.308 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:24.331 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:24.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:24.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:24.379 INFO:tasks.rados.rados.0.smithi076.stdout:3947: done (1 left) 2021-03-15T22:36:24.380 INFO:tasks.rados.rados.0.smithi076.stdout:3948: copy_from oid 2 from oid 48 current snap is 465 2021-03-15T22:36:24.380 INFO:tasks.rados.rados.0.smithi076.stdout:3949: delete oid 30 current snap is 465 2021-03-15T22:36:24.396 INFO:tasks.rados.rados.0.smithi076.stdout:3946: done (2 left) 2021-03-15T22:36:24.396 INFO:tasks.rados.rados.0.smithi076.stdout:3949: done (1 left) 2021-03-15T22:36:24.397 INFO:tasks.rados.rados.0.smithi076.stdout:3950: copy_from oid 45 from oid 29 current snap is 465 2021-03-15T22:36:24.397 INFO:tasks.rados.rados.0.smithi076.stdout:3951: delete oid 13 current snap is 465 2021-03-15T22:36:24.401 INFO:tasks.rados.rados.0.smithi076.stdout:3951: done (2 left) 2021-03-15T22:36:24.401 INFO:tasks.rados.rados.0.smithi076.stdout:3952: rollback oid 30 current snap is 465 2021-03-15T22:36:24.401 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 30 to 423 2021-03-15T22:36:24.402 INFO:tasks.rados.rados.0.smithi076.stdout:3953: read oid 37 snap 416 2021-03-15T22:36:24.402 INFO:tasks.rados.rados.0.smithi076.stdout:3953: expect (ObjNum 1186 snap 414 seq_num 1186) 2021-03-15T22:36:24.402 INFO:tasks.rados.rados.0.smithi076.stdout:3954: snap_remove snap 446 2021-03-15T22:36:24.435 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:24.457 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:24.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:24.458 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:24.559 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:24.586 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:24.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:24.588 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:24.689 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:24.714 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:24.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:24.715 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:24.816 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:24.842 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:24.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:24.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:24.945 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:24.970 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:24.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:25.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[56514]: cluster 2021-03-15T22:36:23.234602+0000 mgr.y ( 2021-03-15T22:36:25.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[56514]: mgr.14144) 1123 : cluster [DBG] pgmap v1688: 151 pgs: 1 peering, 10 unknown, 140 active+clean; 779 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 813 KiB/s wr, 10 op/s 2021-03-15T22:36:25.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[56514]: cluster 2021-03-15T22:36:24.285987+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.9 scrub ok 2021-03-15T22:36:25.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[56514]: cluster 2021-03-15T22:36:24.373998+0000 mon.a (mon.0) 2570 : cluster [DBG] osdmap e1029: 8 total, 8 up, 8 in 2021-03-15T22:36:25.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[56514]: cluster 2021-03-15T22:36:24.397813+0000 osd.1 (osd.1) 197 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:36:25.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: cluster 2021-03-15T22:36:23.234602+0000 2021-03-15T22:36:25.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: mgr.y (mgr.14144) 1123 : cluster [DBG] pgmap v1688: 151 pgs: 1 peering, 10 unknown, 140 active+clean; 779 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 813 KiB/s wr, 10 op/s 2021-03-15T22:36:25.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:36:25.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: 03-15T22:36:24.285987+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.9 scrub ok 2021-03-15T22:36:25.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: 2021-03-15T22:36:25.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: cluster 2021-03-15T 2021-03-15T22:36:25.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: 22:36:24.373998+0000 mon.a (mon 2021-03-15T22:36:25.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: .0) 2570 : cluster [DBG] osdmap e1029: 8 total, 8 up, 8 in 2021-03-15T22:36:25.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: cluster 2021-03-15T22:36:25.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: 2021-03-15T22:36:24.397813+0000 2021-03-15T22:36:25.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: osd.1 (osd.1) 197 : cluster [DBG] 2021-03-15T22:36:25.021 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:24 smithi076 conmon[62156]: 3.4 scrub ok 2021-03-15T22:36:25.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:24 smithi125 conmon[56739]: cluster 2021-03-15T22:36:23.234602+0000 mgr.y (mgr.14144) 1123 : cluster [DBG] pgmap v1688: 151 pgs: 1 peering, 10 unknown, 140 active+clean; 779 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 6.3 MiB/s rd, 813 KiB/s wr, 10 op/s 2021-03-15T22:36:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:24 smithi125 conmon[56739]: cluster 2021-03-15T22:36:24.285987+0000 osd.4 (osd.4) 206 : cluster [DBG] 3.9 scrub ok 2021-03-15T22:36:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:24 smithi125 conmon[56739]: cluster 2021-03-15T22:36:24.373998 2021-03-15T22:36:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:24 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2570 : cluster [DBG] osdmap e1029: 8 total, 8 up, 8 in 2021-03-15T22:36:25.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:24 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:36:25.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:24 smithi125 conmon[56739]: 15T22:36:24.397813+0000 osd.1 ( 2021-03-15T22:36:25.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:24 smithi125 conmon[56739]: osd.1) 197 : cluster [DBG] 3.4 scrub ok 2021-03-15T22:36:25.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:25.097 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:25.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:25.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:25.200 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:25.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:25.226 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:25.226 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:25.327 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:25.349 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:25.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:25.351 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:25.368 INFO:tasks.rados.rados.0.smithi076.stdout:3954: done (4 left) 2021-03-15T22:36:25.368 INFO:tasks.rados.rados.0.smithi076.stdout:3955: copy_from oid 22 from oid 5 current snap is 465 2021-03-15T22:36:25.368 INFO:tasks.rados.rados.0.smithi076.stdout:3952: finishing rollback tid 1 to smithi07694878-30 2021-03-15T22:36:25.368 INFO:tasks.rados.rados.0.smithi076.stdout:3952: finishing rollback tid 2 to smithi07694878-30 2021-03-15T22:36:25.369 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 3378 (ObjNum 774 snap 256 seq_num 774) dirty exists 2021-03-15T22:36:25.369 INFO:tasks.rados.rados.0.smithi076.stdout:3950: finishing copy_from to smithi07694878-45 2021-03-15T22:36:25.369 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 45 v 3311 (ObjNum 1228 snap 425 seq_num 1228) dirty exists 2021-03-15T22:36:25.369 INFO:tasks.rados.rados.0.smithi076.stdout:3950: finishing copy_from racing read to smithi07694878-45 2021-03-15T22:36:25.370 INFO:tasks.rados.rados.0.smithi076.stdout:3948: finishing copy_from to smithi07694878-2 2021-03-15T22:36:25.370 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 2 v 3155 (ObjNum 1221 snap 423 seq_num 1221) dirty exists 2021-03-15T22:36:25.370 INFO:tasks.rados.rados.0.smithi076.stdout:3948: finishing copy_from racing read to smithi07694878-2 2021-03-15T22:36:25.370 INFO:tasks.rados.rados.0.smithi076.stdout:3953: expect (ObjNum 1186 snap 414 seq_num 1186) 2021-03-15T22:36:25.377 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is set 2021-03-15T22:36:25.423 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:36:25.423 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:36:25.452 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:25.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:25.477 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:25.477 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:25.570 INFO:tasks.rados.rados.0.smithi076.stdout:3948: done (4 left) 2021-03-15T22:36:25.571 INFO:tasks.rados.rados.0.smithi076.stdout:3950: done (3 left) 2021-03-15T22:36:25.571 INFO:tasks.rados.rados.0.smithi076.stdout:3952: done (2 left) 2021-03-15T22:36:25.572 INFO:tasks.rados.rados.0.smithi076.stdout:3953: done (1 left) 2021-03-15T22:36:25.572 INFO:tasks.rados.rados.0.smithi076.stdout:3956: snap_create 2021-03-15T22:36:25.572 INFO:tasks.rados.rados.0.smithi076.stdout:3955: finishing copy_from to smithi07694878-22 2021-03-15T22:36:25.572 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 2829 (ObjNum 1238 snap 427 seq_num 1238) dirty exists 2021-03-15T22:36:25.573 INFO:tasks.rados.rados.0.smithi076.stdout:3955: finishing copy_from racing read to smithi07694878-22 2021-03-15T22:36:25.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:25.601 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:25.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:25.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:25.704 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:25.731 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:25.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:25.732 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:25.833 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:25.854 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:25.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:25.856 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:25.957 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:25.983 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:25.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:25.985 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:26.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[56514]: cluster 2021-03-15T22:36:24.720619+0000 osd.2 (osd.2) 53 : cluster [DBG] 7.e scrub ok 2021-03-15T22:36:26.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[56514]: audit 2021-03-15T22:36:25.194154+0000 mon.a (mon.0) 2571 : audit [INF] from='client.? 172.21.15.76:0/3675563901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:36:26.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[56514]: cluster 2021-03-15T22:36:25.253183+0000 osd.4 (osd.4) 207 : cluster [DBG] 6.e deep-scrub ok 2021-03-15T22:36:26.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[56514]: cluster 2021-03-15T22:36:25.355309+0000 mon.a (mon.0) 2572 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:26.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[56514]: audit 2021-03-15T22:36:25.359452+0000 mon.a (mon.0) 2573 : audit [INF] from='client.? 172.21.15.76:0/3675563901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:36:26.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[56514]: cluster 2021-03-15T22:36:25.359513+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e1030: 8 total, 8 up, 8 in 2021-03-15T22:36:26.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[62156]: cluster 2021-03-15T22:36:24.720619+0000 osd.2 (osd.2) 53 : cluster [DBG] 7.e scrub ok 2021-03-15T22:36:26.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[62156]: audit 2021-03-15T22:36:25.194154+0000 mon.a (mon.0) 2571 : audit [INF] from='client.? 172.21.15.76:0/3675563901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:36:26.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[62156]: cluster 2021-03-15T22:36:25.253183+0000 osd.4 (osd.4) 207 : cluster [DBG] 6.e deep-scrub ok 2021-03-15T22:36:26.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[62156]: cluster 2021-03-15T22:36:25.355309+0000 mon.a (mon.0) 2572 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:26.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[62156]: audit 2021-03-15T22:36:25.359452+0000 mon.a (mon.0) 2573 : audit [INF] from='client.? 172.21.15.76:0/3675563901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:36:26.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:25 smithi076 conmon[62156]: cluster 2021-03-15T22:36:25.359513+0000 mon.a (mon.0) 2574 : cluster [DBG] osdmap e1030: 8 total, 8 up, 8 in 2021-03-15T22:36:26.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: cluster 2021-03-15T22:36:24.720619+0000 osd.2 (osd 2021-03-15T22:36:26.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: .2) 53 : cluster [DBG] 7.e scrub ok 2021-03-15T22:36:26.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: audit 2021-03-15T22:36:25.194154+0000 mon.a (mon.0) 2571 : audit [INF] from='client.? 172.21.15.76:0/3675563901' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2021-03-15T22:36:26.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: cluster 2021-03-15T22:36:25. 2021-03-15T22:36:26.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: 253183+0000 osd.4 (osd.4) 207 : cluster [DBG] 2021-03-15T22:36:26.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: 6.e deep-scrub ok 2021-03-15T22:36:26.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: cluster 2021-03-15T22 2021-03-15T22:36:26.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: :36:25.355309+0000 mon.a (mon.0 2021-03-15T22:36:26.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: ) 2572 : cluster [WRN] Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:26.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:36:26.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: -15T22:36:25.359452+0000 mon.a 2021-03-15T22:36:26.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: (mon.0) 2573 : audit [INF] from='client.? 172.21.15.76:0/3675563901' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2021-03-15T22:36:26.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: 2021-03-15T22:36:26.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: cluster 2021-03-15T22:36:26.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: 2021-03-15T22:36:25.359513+0000 mon.a (mon 2021-03-15T22:36:26.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:25 smithi125 conmon[56739]: .0) 2574 : cluster [DBG] osdmap e1030: 8 total, 8 up, 8 in 2021-03-15T22:36:26.085 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:26.110 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:26.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:26.112 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:26.146 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:36:26.213 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:26.238 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:26.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:26.240 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:26.341 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:26.364 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:26.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:26.366 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:26.377 INFO:tasks.rados.rados.0.smithi076.stdout:3955: done (1 left) 2021-03-15T22:36:26.378 INFO:tasks.rados.rados.0.smithi076.stdout:3956: done (0 left) 2021-03-15T22:36:26.378 INFO:tasks.rados.rados.0.smithi076.stdout:3957: read oid 45 snap -1 2021-03-15T22:36:26.378 INFO:tasks.rados.rados.0.smithi076.stdout:3957: expect (ObjNum 1228 snap 425 seq_num 1228) 2021-03-15T22:36:26.378 INFO:tasks.rados.rados.0.smithi076.stdout:3958: rollback oid 38 current snap is 466 2021-03-15T22:36:26.379 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 38 to 420 2021-03-15T22:36:26.379 INFO:tasks.rados.rados.0.smithi076.stdout:3959: snap_remove snap 458 2021-03-15T22:36:26.467 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:26.488 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:26.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:26.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:26.549 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 7, 0, 3, 5, 6, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:36:26.549 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:36:26.550 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump -f json-pretty 2021-03-15T22:36:26.591 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:26.615 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:26.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:26.616 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:26.676 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_ops_in_flight 2021-03-15T22:36:26.714 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:26.717 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:26.745 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:26.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:26.747 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:26.848 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:26.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:26.879 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:26.880 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:26.895 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:26.929 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:26 smithi076 conmon[56514]: cluster 2021-03-15T22:36:25. 2021-03-15T22:36:26.930 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:26 smithi076 conmon[56514]: 235121+0000 mgr.y (mgr.14144) 1124 : cluster [DBG] pgmap v1691: 151 pgs: 6 peering, 5 unknown, 140 active+clean; 769 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 307 KiB/s wr, 7 op/s 2021-03-15T22:36:26.930 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:26 smithi076 conmon[56514]: cluster 2021-03-15T22:36:26.372673+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e1031: 8 total, 8 up, 8 in 2021-03-15T22:36:26.930 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:26 smithi076 conmon[62156]: cluster 2021-03-15T22:36:25 2021-03-15T22:36:26.930 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:26 smithi076 conmon[62156]: .235121+0000 mgr.y (mgr.14144) 1124 : cluster [DBG] pgmap v1691: 151 pgs: 6 peering, 5 unknown, 140 active+clean; 769 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 307 KiB/s wr, 7 op/s 2021-03-15T22:36:26.931 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:26 smithi076 conmon[62156]: cluster 2021-03-15T22:36:26.931 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:26 smithi076 conmon[62156]: 2021-03-15T22:36:26.372673+0000 2021-03-15T22:36:26.931 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:26 smithi076 conmon[62156]: mon.a (mon.0) 2575 : cluster [DBG] osdmap e1031: 8 total, 8 up, 8 in 2021-03-15T22:36:26.981 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:27.015 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:27.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:27.017 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:27.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:26 smithi125 conmon[56739]: cluster 2021-03-15T22:36:25.235121+0000 mgr.y (mgr.14144) 1124 : cluster [DBG] pgmap v1691: 151 pgs: 6 peering, 5 unknown, 140 active+clean; 769 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.1 MiB/s rd, 307 KiB/s wr, 7 op/s 2021-03-15T22:36:27.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:26 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:36:27.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:26 smithi125 conmon[56739]: 15T22:36:26.372673+0000 mon.a (mon.0) 2575 : cluster [DBG] osdmap e1031: 8 total, 8 up, 8 in 2021-03-15T22:36:27.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:27.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:27.151 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:27.152 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:27.254 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:27.283 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:27.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:27.285 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:27.323 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:27.381 INFO:tasks.rados.rados.0.smithi076.stdout:3959: done (2 left) 2021-03-15T22:36:27.381 INFO:tasks.rados.rados.0.smithi076.stdout:3960: snap_remove snap 404 2021-03-15T22:36:27.386 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:27.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:27.417 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:27.417 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:27.519 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:27.541 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:27.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:27.542 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:27.643 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:27.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:27.667 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:27.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:27.768 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:27.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:27.791 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:27.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:27.893 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:27.915 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:27.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:27.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:28.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:27 smithi076 conmon[56514]: audit 2021-03-15T22:36:27.236520+0000 mon.a (mon.0) 2576 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-03-15T22:36:28.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:27 smithi076 conmon[56514]: audit 2021-03-15T22:36:27.377312+0000 mon.a (mon.0) 2577 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]': finished 2021-03-15T22:36:28.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:27 smithi076 conmon[56514]: cluster 2021-03-15T22:36:27.377372+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e1032: 8 total, 8 up, 8 in 2021-03-15T22:36:28.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:27 smithi076 conmon[62156]: audit 2021-03-15T22:36:27.236520+0000 mon.a (mon.0) 2576 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-03-15T22:36:28.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:27 smithi076 conmon[62156]: audit 2021-03-15T22:36:27.377312+0000 mon.a (mon.0) 2577 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]': finished 2021-03-15T22:36:28.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:27 smithi076 conmon[62156]: cluster 2021-03-15T22:36:27.377372+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e1032: 8 total, 8 up, 8 in 2021-03-15T22:36:28.019 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:28.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:27 smithi125 conmon[56739]: audit 2021-03-15T22:36:27.236520+0000 mon.a (mon.0) 2576 : audit [INF] 2021-03-15T22:36:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:27 smithi125 conmon[56739]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]: dispatch 2021-03-15T22:36:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:27 smithi125 conmon[56739]: audit 2021-03-15T22:36:27.377312+0000 mon.a (mon.0) 2577 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "23"}]': finished 2021-03-15T22:36:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:27 smithi125 conmon[56739]: cluster 2021-03-15T22:36:28.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:27 smithi125 conmon[56739]: 2021-03-15T22:36:27.377372+0000 mon.a (mon.0) 2578 : cluster [DBG] osdmap e1032: 8 total, 8 up, 8 in 2021-03-15T22:36:28.045 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:28.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:28.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:28.148 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:28.170 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:28.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:28.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:28.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:28.294 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:28.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:28.296 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:28.383 INFO:tasks.rados.rados.0.smithi076.stdout:3960: done (2 left) 2021-03-15T22:36:28.383 INFO:tasks.rados.rados.0.smithi076.stdout:3961: read oid 48 snap -1 2021-03-15T22:36:28.383 INFO:tasks.rados.rados.0.smithi076.stdout:3961: expect (ObjNum 1221 snap 423 seq_num 1221) 2021-03-15T22:36:28.384 INFO:tasks.rados.rados.0.smithi076.stdout:3962: write oid 24 current snap is 466 2021-03-15T22:36:28.384 INFO:tasks.rados.rados.0.smithi076.stdout:3962: seq_num 1337 ranges {470959=569688,1704778=595135} 2021-03-15T22:36:28.385 INFO:tasks.rados.rados.0.smithi076.stdout:3962: writing smithi07694878-24 from 470959 to 1040647 tid 1 2021-03-15T22:36:28.393 INFO:tasks.rados.rados.0.smithi076.stdout:3962: writing smithi07694878-24 from 1704778 to 2299913 tid 2 2021-03-15T22:36:28.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:28.403 INFO:tasks.rados.rados.0.smithi076.stdout:3957: expect (ObjNum 1228 snap 425 seq_num 1228) 2021-03-15T22:36:28.422 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:28.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:28.424 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:28.500 INFO:tasks.rados.rados.0.smithi076.stdout:3958: finishing rollback tid 0 to smithi07694878-38 2021-03-15T22:36:28.501 INFO:tasks.rados.rados.0.smithi076.stdout:3958: finishing rollback tid 1 to smithi07694878-38 2021-03-15T22:36:28.501 INFO:tasks.rados.rados.0.smithi076.stdout:3958: finishing rollback tid 2 to smithi07694878-38 2021-03-15T22:36:28.501 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2861 (ObjNum 1141 snap 397 seq_num 1141) dirty exists 2021-03-15T22:36:28.501 INFO:tasks.rados.rados.0.smithi076.stdout:3957: done (3 left) 2021-03-15T22:36:28.501 INFO:tasks.rados.rados.0.smithi076.stdout:3958: done (2 left) 2021-03-15T22:36:28.502 INFO:tasks.rados.rados.0.smithi076.stdout:3963: copy_from oid 46 from oid 34 current snap is 466 2021-03-15T22:36:28.502 INFO:tasks.rados.rados.0.smithi076.stdout:3961: expect (ObjNum 1221 snap 423 seq_num 1221) 2021-03-15T22:36:28.526 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:28.547 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:28.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:28.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:28.618 INFO:tasks.rados.rados.0.smithi076.stdout:3962: finishing write tid 1 to smithi07694878-24 2021-03-15T22:36:28.618 INFO:tasks.rados.rados.0.smithi076.stdout:3962: finishing write tid 2 to smithi07694878-24 2021-03-15T22:36:28.618 INFO:tasks.rados.rados.0.smithi076.stdout:3962: finishing write tid 3 to smithi07694878-24 2021-03-15T22:36:28.619 INFO:tasks.rados.rados.0.smithi076.stdout:3962: finishing write tid 4 to smithi07694878-24 2021-03-15T22:36:28.619 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 24 v 3043 (ObjNum 1337 snap 466 seq_num 1337) dirty exists 2021-03-15T22:36:28.619 INFO:tasks.rados.rados.0.smithi076.stdout:3962: left oid 24 (ObjNum 1337 snap 466 seq_num 1337) 2021-03-15T22:36:28.619 INFO:tasks.rados.rados.0.smithi076.stdout:3961: done (2 left) 2021-03-15T22:36:28.620 INFO:tasks.rados.rados.0.smithi076.stdout:3962: done (1 left) 2021-03-15T22:36:28.620 INFO:tasks.rados.rados.0.smithi076.stdout:3964: read oid 5 snap -1 2021-03-15T22:36:28.620 INFO:tasks.rados.rados.0.smithi076.stdout:3964: expect (ObjNum 1238 snap 427 seq_num 1238) 2021-03-15T22:36:28.620 INFO:tasks.rados.rados.0.smithi076.stdout:3963: finishing copy_from to smithi07694878-46 2021-03-15T22:36:28.621 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 46 v 2446 (ObjNum 1265 snap 437 seq_num 1265) dirty exists 2021-03-15T22:36:28.621 INFO:tasks.rados.rados.0.smithi076.stdout:3963: finishing copy_from racing read to smithi07694878-46 2021-03-15T22:36:28.621 INFO:tasks.rados.rados.0.smithi076.stdout:3965: read oid 20 snap -1 2021-03-15T22:36:28.621 INFO:tasks.rados.rados.0.smithi076.stdout:3965: expect (ObjNum 962 snap 321 seq_num 962) 2021-03-15T22:36:28.622 INFO:tasks.rados.rados.0.smithi076.stdout:3963: done (2 left) 2021-03-15T22:36:28.622 INFO:tasks.rados.rados.0.smithi076.stdout:3966: write oid 39 current snap is 466 2021-03-15T22:36:28.622 INFO:tasks.rados.rados.0.smithi076.stdout:3966: seq_num 1338 ranges {487766=644937,1646907=738793} 2021-03-15T22:36:28.623 INFO:tasks.rados.rados.0.smithi076.stdout:3966: writing smithi07694878-39 from 487766 to 1132703 tid 1 2021-03-15T22:36:28.630 INFO:tasks.rados.rados.0.smithi076.stdout:3966: writing smithi07694878-39 from 1646907 to 2385700 tid 2 2021-03-15T22:36:28.631 INFO:tasks.rados.rados.0.smithi076.stdout:3967: snap_create 2021-03-15T22:36:28.631 INFO:tasks.rados.rados.0.smithi076.stdout:3964: expect (ObjNum 1238 snap 427 seq_num 1238) 2021-03-15T22:36:28.649 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:28.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set nodeep-scrub 2021-03-15T22:36:28.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:28.673 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:28.674 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:28.776 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:28.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:28.799 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:28.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:28.901 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:28.902 INFO:tasks.rados.rados.0.smithi076.stdout:3965: expect (ObjNum 962 snap 321 seq_num 962) 2021-03-15T22:36:28.931 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:28.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:28.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:28.952 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[56514]: cluster 2021-03-15T22:36:27.235655 2021-03-15T22:36:28.953 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[56514]: +0000 mgr.y (mgr.14144) 1125 : cluster [DBG] pgmap v1694: 150 pgs: 5 peering, 1 unknown, 144 active+clean; 778 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-03-15T22:36:28.953 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[56514]: cluster 2021-03-15T22:36:28.330143+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:28.953 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[56514]: cluster 2021-03-15T22:36:28.333465+0000 osd.5 (osd.5) 67 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:28.953 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[56514]: cluster 2021-03-15T22:36:28.372926+0000 mon.a (mon.0) 2579 : cluster [DBG] osdmap e1033: 8 total, 8 up, 8 in 2021-03-15T22:36:28.954 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[62156]: cluster 2021-03-15T22:36:27.235655+0000 mgr.y (mgr.14144) 1125 : cluster 2021-03-15T22:36:28.954 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[62156]: [DBG] pgmap v1694: 150 pgs: 5 peering, 1 unknown, 144 active+clean; 778 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-03-15T22:36:28.955 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[62156]: cluster 2021-03-15T22:36:28.330143+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:28.955 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[62156]: cluster 2021-03-15T22:36:28.333465 2021-03-15T22:36:28.955 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[62156]: +0000 osd.5 (osd.5) 67 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:28.955 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[62156]: cluster 2021-03-15T22:36:28.372926+0000 2021-03-15T22:36:28.955 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:28 smithi076 conmon[62156]: mon.a (mon.0) 2579 : cluster [DBG] osdmap e1033: 8 total, 8 up, 8 in 2021-03-15T22:36:29.034 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:29.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:28 smithi125 conmon[56739]: cluster 2021-03-15T22:36:27.235655+0000 mgr.y (mgr.14144) 1125 : cluster [DBG] 2021-03-15T22:36:29.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:28 smithi125 conmon[56739]: pgmap v1694: 150 pgs: 5 peering, 1 unknown, 144 active+clean; 778 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 4.7 MiB/s rd, 2.5 MiB/s wr, 10 op/s 2021-03-15T22:36:29.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:28 smithi125 conmon[56739]: cluster 2021-03-15T22:36:28.330143+0000 osd.5 (osd.5) 66 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:29.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:28 smithi125 conmon[56739]: cluster 2021-03-15T22:36: 2021-03-15T22:36:29.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:28 smithi125 conmon[56739]: 28.333465+0000 osd.5 (osd.5) 67 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:29.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:28 smithi125 conmon[56739]: cluster 2021-03-15T22:36:29.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:28 smithi125 conmon[56739]: 2021-03-15T22:36:28.372926+0000 mon.a (mon.0) 2579 : 2021-03-15T22:36:29.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:28 smithi125 conmon[56739]: cluster [DBG] osdmap e1033: 8 total, 8 up, 8 in 2021-03-15T22:36:29.056 INFO:tasks.rados.rados.0.smithi076.stdout:3966: finishing write tid 1 to smithi07694878-39 2021-03-15T22:36:29.057 INFO:tasks.rados.rados.0.smithi076.stdout:3966: finishing write tid 2 to smithi07694878-39 2021-03-15T22:36:29.057 INFO:tasks.rados.rados.0.smithi076.stdout:3966: finishing write tid 3 to smithi07694878-39 2021-03-15T22:36:29.057 INFO:tasks.rados.rados.0.smithi076.stdout:3966: finishing write tid 4 to smithi07694878-39 2021-03-15T22:36:29.058 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 39 v 2864 (ObjNum 1338 snap 466 seq_num 1338) dirty exists 2021-03-15T22:36:29.058 INFO:tasks.rados.rados.0.smithi076.stdout:3966: left oid 39 (ObjNum 1338 snap 466 seq_num 1338) 2021-03-15T22:36:29.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:29.059 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:29.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:29.155 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:36:29.160 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:29.173 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:36:29.183 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:29.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:29.184 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:29.190 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:36:29.208 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:36:29.226 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:36:29.243 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:36:29.264 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:36:29.281 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:36:29.282 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:36:29.283 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:36:29.283 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:36:29.283 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:36:29.283 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:36:29.284 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:36:29.284 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:36:29.284 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:36:29.286 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:29.298 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:29.311 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:29.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:29.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:29.382 INFO:tasks.rados.rados.0.smithi076.stdout:3964: done (3 left) 2021-03-15T22:36:29.382 INFO:tasks.rados.rados.0.smithi076.stdout:3965: done (2 left) 2021-03-15T22:36:29.383 INFO:tasks.rados.rados.0.smithi076.stdout:3966: done (1 left) 2021-03-15T22:36:29.383 INFO:tasks.rados.rados.0.smithi076.stdout:3967: done (0 left) 2021-03-15T22:36:29.383 INFO:tasks.rados.rados.0.smithi076.stdout:3968: snap_remove snap 422 2021-03-15T22:36:29.414 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:29.439 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:29.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:29.440 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:29.542 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:29.565 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:29.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:29.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:29.668 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:29.693 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:29.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:29.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:29.796 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:29.821 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:29.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:29.924 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:29.949 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:29.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:29.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:30.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:29 smithi076 conmon[56514]: audit 2021-03-15T22:36:29.136136+0000 mon.a (mon.0) 2580 : audit [DBG] from='client.? 172.21.15.76:0/142100710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:30.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:29 smithi076 conmon[56514]: cluster 2021-03-15T22:36:29.377421+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e1034: 8 total, 8 up, 8 in 2021-03-15T22:36:30.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:29 smithi076 conmon[62156]: audit 2021-03-15T22:36:29 2021-03-15T22:36:30.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:29 smithi076 conmon[62156]: .136136+0000 mon.a (mon.0) 2580 : audit [DBG] from='client.? 172.21.15.76:0/142100710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:30.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:29 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:36:30.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:29 smithi076 conmon[62156]: 03-15T22:36:29.377421+0000 mon.a (mon.0) 2581 : cluster [DBG] 2021-03-15T22:36:30.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:29 smithi076 conmon[62156]: osdmap e1034: 8 total, 8 up, 8 in 2021-03-15T22:36:30.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:29 smithi125 conmon[56739]: audit 2021-03-15T22:36:29.136136+0000 mon.a (mon.0) 2580 : audit 2021-03-15T22:36:30.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:29 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/142100710' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:30.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:29 smithi125 conmon[56739]: cluster 2021-03-15T22:36:29.377421+0000 mon.a (mon.0) 2581 : cluster [DBG] osdmap e1034: 8 total, 8 up, 8 in 2021-03-15T22:36:30.052 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:30.075 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:30.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:30.076 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:30.177 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:30.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:30.200 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:30.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:30.302 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:30.325 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:30.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:30.326 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:30.381 INFO:tasks.rados.rados.0.smithi076.stdout:3968: done (0 left) 2021-03-15T22:36:30.381 INFO:tasks.rados.rados.0.smithi076.stdout:3969: write oid 43 current snap is 467 2021-03-15T22:36:30.381 INFO:tasks.rados.rados.0.smithi076.stdout:3969: seq_num 1339 ranges {504573=436539,1421742=598804,2437505=67068} 2021-03-15T22:36:30.382 INFO:tasks.rados.rados.0.smithi076.stdout:3969: writing smithi07694878-43 from 504573 to 941112 tid 1 2021-03-15T22:36:30.385 INFO:tasks.rados.rados.0.smithi076.stdout:3969: writing smithi07694878-43 from 1421742 to 2020546 tid 2 2021-03-15T22:36:30.388 INFO:tasks.rados.rados.0.smithi076.stdout:3969: writing smithi07694878-43 from 2437505 to 2504573 tid 3 2021-03-15T22:36:30.389 INFO:tasks.rados.rados.0.smithi076.stdout:3970: snap_remove snap 389 2021-03-15T22:36:30.427 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:30.450 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:30.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:30.552 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:30.575 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:30.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:30.576 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:30.674 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:36:30.675 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:36:30.675 INFO:teuthology.orchestra.run.smithi076.stdout: "epoch": 1035, 2021-03-15T22:36:30.675 INFO:teuthology.orchestra.run.smithi076.stdout: "fsid": "8a572802-85db-11eb-9091-001a4aab830c", 2021-03-15T22:36:30.675 INFO:teuthology.orchestra.run.smithi076.stdout: "created": "2021-03-15T22:13:20.370795+0000", 2021-03-15T22:36:30.676 INFO:teuthology.orchestra.run.smithi076.stdout: "modified": "2021-03-15T22:36:30.356703+0000", 2021-03-15T22:36:30.676 INFO:teuthology.orchestra.run.smithi076.stdout: "last_up_change": "2021-03-15T22:30:30.205134+0000", 2021-03-15T22:36:30.676 INFO:teuthology.orchestra.run.smithi076.stdout: "last_in_change": "2021-03-15T22:34:54.369110+0000", 2021-03-15T22:36:30.676 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2021-03-15T22:36:30.677 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_num": 5801984, 2021-03-15T22:36:30.677 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_set": [ 2021-03-15T22:36:30.677 INFO:teuthology.orchestra.run.smithi076.stdout: "noscrub", 2021-03-15T22:36:30.677 INFO:teuthology.orchestra.run.smithi076.stdout: "pglog_hardlimit", 2021-03-15T22:36:30.678 INFO:teuthology.orchestra.run.smithi076.stdout: "purged_snapdirs", 2021-03-15T22:36:30.678 INFO:teuthology.orchestra.run.smithi076.stdout: "recovery_deletes", 2021-03-15T22:36:30.678 INFO:teuthology.orchestra.run.smithi076.stdout: "sortbitwise" 2021-03-15T22:36:30.678 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:36:30.679 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_version": 20, 2021-03-15T22:36:30.679 INFO:teuthology.orchestra.run.smithi076.stdout: "full_ratio": 0.94999998807907104, 2021-03-15T22:36:30.679 INFO:teuthology.orchestra.run.smithi076.stdout: "backfillfull_ratio": 0.89999997615814209, 2021-03-15T22:36:30.679 INFO:teuthology.orchestra.run.smithi076.stdout: "nearfull_ratio": 0.85000002384185791, 2021-03-15T22:36:30.679 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_snapshot": "", 2021-03-15T22:36:30.680 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_max": 7, 2021-03-15T22:36:30.680 INFO:teuthology.orchestra.run.smithi076.stdout: "max_osd": 8, 2021-03-15T22:36:30.680 INFO:teuthology.orchestra.run.smithi076.stdout: "require_min_compat_client": "luminous", 2021-03-15T22:36:30.680 INFO:teuthology.orchestra.run.smithi076.stdout: "min_compat_client": "luminous", 2021-03-15T22:36:30.681 INFO:teuthology.orchestra.run.smithi076.stdout: "require_osd_release": "quincy", 2021-03-15T22:36:30.681 INFO:teuthology.orchestra.run.smithi076.stdout: "pools": [ 2021-03-15T22:36:30.682 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.682 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 1, 2021-03-15T22:36:30.682 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "device_health_metrics", 2021-03-15T22:36:30.683 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:16:34.817191+0000", 2021-03-15T22:36:30.683 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:36:30.683 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:36:30.683 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:36:30.684 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:36:30.684 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:36:30.684 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:36:30.684 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:36:30.685 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:36:30.685 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:36:30.685 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:36:30.685 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:36:30.685 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:36:30.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 11, 2021-03-15T22:36:30.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 11, 2021-03-15T22:36:30.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 11, 2021-03-15T22:36:30.686 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 11, 2021-03-15T22:36:30.687 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 11, 2021-03-15T22:36:30.687 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:36:30.687 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:36:30.687 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:36:30.688 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:36:30.688 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:36:30.688 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:36:30.689 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:36:30.689 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.689 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "1028", 2021-03-15T22:36:30.689 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:36:30.690 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:36:30.690 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "1026", 2021-03-15T22:36:30.690 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:36:30.690 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:36:30.690 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:36:30.691 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:36:30.691 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:36:30.691 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:36:30.691 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:36:30.692 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:36:30.692 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:36:30.692 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:36:30.692 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:36:30.693 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:36:30.693 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:36:30.693 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:36:30.693 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:36:30.693 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:36:30.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:36:30.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:36:30.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:36:30.694 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:36:30.695 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:36:30.695 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:36:30.695 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:36:30.695 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.695 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:36:30.696 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:36:30.696 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:36:30.696 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:36:30.696 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:36:30.696 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:36:30.697 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:36:30.697 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:36:30.697 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:36:30.697 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:36:30.698 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:36:30.698 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:36:30.698 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 1 2021-03-15T22:36:30.698 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.699 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:36:30.699 INFO:teuthology.orchestra.run.smithi076.stdout: "mgr_devicehealth": {} 2021-03-15T22:36:30.699 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.699 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.700 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.700 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 2, 2021-03-15T22:36:30.700 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "iscsi", 2021-03-15T22:36:30.700 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:49.906400+0000", 2021-03-15T22:36:30.700 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:36:30.701 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:36:30.701 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:36:30.701 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:36:30.701 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:36:30.701 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:36:30.702 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:36:30.702 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:36:30.702 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:36:30.702 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:36:30.703 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:36:30.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:36:30.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 3, 2021-03-15T22:36:30.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 3, 2021-03-15T22:36:30.703 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 3, 2021-03-15T22:36:30.704 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 3, 2021-03-15T22:36:30.704 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 3, 2021-03-15T22:36:30.705 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:36:30.705 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:36:30.705 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:36:30.705 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:36:30.705 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:36:30.706 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:36:30.706 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:36:30.706 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.706 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "56", 2021-03-15T22:36:30.706 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:36:30.707 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:36:30.707 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:36:30.707 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:36:30.707 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:36:30.708 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:36:30.708 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:36:30.708 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:36:30.708 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:36:30.708 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:36:30.709 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:36:30.709 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:36:30.709 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:36:30.709 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:36:30.709 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:36:30.710 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:36:30.710 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:36:30.710 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:36:30.710 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:36:30.710 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:36:30.711 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:36:30.711 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:36:30.711 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:36:30.711 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:36:30.711 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:36:30.712 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:36:30.712 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.712 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:36:30.712 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:36:30.713 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:36:30.713 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:36:30.713 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:36:30.713 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:36:30.713 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:36:30.714 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:36:30.714 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:36:30.714 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:36:30.714 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:36:30.715 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:36:30.715 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:36:30.715 INFO:teuthology.orchestra.run.smithi076.stdout: "rbd": {} 2021-03-15T22:36:30.715 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.715 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.716 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.716 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 3, 2021-03-15T22:36:30.716 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": ".rgw.root", 2021-03-15T22:36:30.716 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:52.257970+0000", 2021-03-15T22:36:30.716 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:36:30.717 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:36:30.717 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:36:30.717 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:36:30.717 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:36:30.717 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:36:30.718 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:36:30.718 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:36:30.718 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:36:30.718 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:36:30.719 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:36:30.719 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:36:30.719 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 23, 2021-03-15T22:36:30.719 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 22, 2021-03-15T22:36:30.720 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 22, 2021-03-15T22:36:30.720 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 22, 2021-03-15T22:36:30.720 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 23, 2021-03-15T22:36:30.720 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:36:30.720 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "3.17", 2021-03-15T22:36:30.721 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 1034, 2021-03-15T22:36:30.721 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 1033, 2021-03-15T22:36:30.721 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 1033, 2021-03-15T22:36:30.721 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:36:30.722 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:36:30.722 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.722 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "1034", 2021-03-15T22:36:30.722 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:36:30.722 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "1034", 2021-03-15T22:36:30.723 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "1032", 2021-03-15T22:36:30.723 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:36:30.723 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:36:30.723 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:36:30.723 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:36:30.724 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:36:30.724 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:36:30.724 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:36:30.724 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:36:30.724 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:36:30.725 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:36:30.725 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:36:30.725 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:36:30.726 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:36:30.726 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:36:30.726 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:36:30.726 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:36:30.727 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:36:30.727 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:36:30.727 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:36:30.727 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:36:30.727 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:36:30.728 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:36:30.728 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:36:30.728 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.728 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:36:30.728 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:36:30.729 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:36:30.729 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:36:30.729 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:36:30.729 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:36:30.730 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:36:30.730 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:36:30.730 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:36:30.730 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:36:30.730 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:36:30.731 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:36:30.731 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:36:30.731 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:36:30.731 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.731 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.732 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.732 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 4, 2021-03-15T22:36:30.732 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.log", 2021-03-15T22:36:30.732 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:54.562498+0000", 2021-03-15T22:36:30.733 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:36:30.733 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:36:30.733 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:36:30.733 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:36:30.733 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:36:30.734 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:36:30.734 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:36:30.734 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:36:30.734 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:36:30.735 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:36:30.735 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:36:30.735 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:36:30.735 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:36:30.736 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:36:30.736 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:36:30.736 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:36:30.736 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:36:30.736 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:36:30.737 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:36:30.737 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:36:30.737 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:36:30.737 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:36:30.738 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:36:30.738 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:36:30.738 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.738 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "60", 2021-03-15T22:36:30.738 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:36:30.739 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:36:30.739 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:36:30.739 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:36:30.739 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:36:30.740 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:36:30.740 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:36:30.740 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:36:30.740 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:36:30.740 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:36:30.741 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:36:30.741 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:36:30.741 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:36:30.741 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:36:30.742 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:36:30.742 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:36:30.742 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:36:30.742 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:36:30.742 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:36:30.743 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:36:30.743 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:36:30.743 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:36:30.743 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:36:30.744 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:36:30.744 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:36:30.744 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:36:30.744 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.744 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:36:30.745 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:36:30.745 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:36:30.745 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:36:30.745 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:36:30.746 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:36:30.746 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:36:30.746 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:36:30.746 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:36:30.746 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:36:30.747 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:36:30.747 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:36:30.747 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:36:30.747 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:36:30.748 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.748 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.748 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.748 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 5, 2021-03-15T22:36:30.749 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.control", 2021-03-15T22:36:30.749 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:56.243806+0000", 2021-03-15T22:36:30.749 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:36:30.749 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:36:30.750 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:36:30.750 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:36:30.750 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:36:30.750 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:36:30.750 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:36:30.751 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:36:30.751 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:36:30.751 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:36:30.751 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:36:30.751 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:36:30.752 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:36:30.752 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:36:30.752 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:36:30.752 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:36:30.752 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:36:30.753 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:36:30.753 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "5.16", 2021-03-15T22:36:30.753 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 182, 2021-03-15T22:36:30.753 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 181, 2021-03-15T22:36:30.753 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 181, 2021-03-15T22:36:30.754 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:36:30.754 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:36:30.754 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.754 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "1007", 2021-03-15T22:36:30.755 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:36:30.755 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "182", 2021-03-15T22:36:30.755 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "728", 2021-03-15T22:36:30.755 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:36:30.756 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:36:30.756 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:36:30.756 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:36:30.756 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:36:30.757 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:36:30.757 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:36:30.757 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:36:30.757 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:36:30.757 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:36:30.758 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:36:30.758 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:36:30.758 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:36:30.758 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:36:30.758 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:36:30.759 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:36:30.759 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:36:30.759 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:36:30.759 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:36:30.759 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:36:30.760 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:36:30.760 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:36:30.760 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:36:30.760 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.760 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:36:30.761 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:36:30.761 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:36:30.761 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:36:30.761 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:36:30.762 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:36:30.762 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:36:30.763 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:36:30.764 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:36:30.764 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:36:30.764 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:36:30.764 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:36:30.764 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:36:30.765 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:36:30.765 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.765 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.765 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.766 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 6, 2021-03-15T22:36:30.766 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "default.rgw.meta", 2021-03-15T22:36:30.766 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:17:58.730534+0000", 2021-03-15T22:36:30.766 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 1, 2021-03-15T22:36:30.766 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool", 2021-03-15T22:36:30.767 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:36:30.767 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:36:30.767 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:36:30.767 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:36:30.768 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:36:30.768 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:36:30.768 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:36:30.768 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:36:30.768 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:36:30.769 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:36:30.769 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 32, 2021-03-15T22:36:30.769 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 32, 2021-03-15T22:36:30.769 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 32, 2021-03-15T22:36:30.770 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 32, 2021-03-15T22:36:30.770 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 32, 2021-03-15T22:36:30.770 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:36:30.770 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "6.20", 2021-03-15T22:36:30.771 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 1024, 2021-03-15T22:36:30.771 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 1023, 2021-03-15T22:36:30.771 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 1023, 2021-03-15T22:36:30.771 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:36:30.771 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:36:30.772 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.772 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "1024", 2021-03-15T22:36:30.772 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:36:30.772 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "1024", 2021-03-15T22:36:30.772 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "1022", 2021-03-15T22:36:30.773 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:36:30.773 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:36:30.773 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 0, 2021-03-15T22:36:30.773 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 0, 2021-03-15T22:36:30.774 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:36:30.774 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:36:30.774 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:36:30.774 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:36:30.774 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:36:30.775 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:36:30.775 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:36:30.775 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:36:30.775 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:36:30.775 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:36:30.776 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:36:30.776 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:36:30.776 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:36:30.776 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:36:30.777 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:36:30.777 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:36:30.777 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:36:30.777 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:36:30.777 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:36:30.778 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.778 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:36:30.778 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:36:30.778 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:36:30.779 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:36:30.779 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:36:30.779 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:36:30.779 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:36:30.779 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:36:30.780 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:36:30.780 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:36:30.780 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:36:30.780 INFO:teuthology.orchestra.run.smithi076.stdout: "options": { 2021-03-15T22:36:30.780 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_bias": 4, 2021-03-15T22:36:30.781 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_min": 8 2021-03-15T22:36:30.781 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.781 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:36:30.781 INFO:teuthology.orchestra.run.smithi076.stdout: "rgw": {} 2021-03-15T22:36:30.782 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.782 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.782 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.782 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:36:30.782 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_name": "unique_pool_0", 2021-03-15T22:36:30.783 INFO:teuthology.orchestra.run.smithi076.stdout: "create_time": "2021-03-15T22:20:03.377294+0000", 2021-03-15T22:36:30.783 INFO:teuthology.orchestra.run.smithi076.stdout: "flags": 8193, 2021-03-15T22:36:30.783 INFO:teuthology.orchestra.run.smithi076.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2021-03-15T22:36:30.783 INFO:teuthology.orchestra.run.smithi076.stdout: "type": 1, 2021-03-15T22:36:30.784 INFO:teuthology.orchestra.run.smithi076.stdout: "size": 3, 2021-03-15T22:36:30.784 INFO:teuthology.orchestra.run.smithi076.stdout: "min_size": 2, 2021-03-15T22:36:30.784 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_rule": 0, 2021-03-15T22:36:30.784 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_count": 0, 2021-03-15T22:36:30.784 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_target": 0, 2021-03-15T22:36:30.785 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_barrier": 0, 2021-03-15T22:36:30.785 INFO:teuthology.orchestra.run.smithi076.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2021-03-15T22:36:30.785 INFO:teuthology.orchestra.run.smithi076.stdout: "object_hash": 2, 2021-03-15T22:36:30.785 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_autoscale_mode": "off", 2021-03-15T22:36:30.786 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num": 16, 2021-03-15T22:36:30.786 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num": 16, 2021-03-15T22:36:30.786 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_placement_num_target": 16, 2021-03-15T22:36:30.786 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_target": 16, 2021-03-15T22:36:30.786 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_num_pending": 16, 2021-03-15T22:36:30.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_pg_merge_meta": { 2021-03-15T22:36:30.787 INFO:teuthology.orchestra.run.smithi076.stdout: "source_pgid": "0.0", 2021-03-15T22:36:30.787 INFO:teuthology.orchestra.run.smithi076.stdout: "ready_epoch": 0, 2021-03-15T22:36:30.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_started": 0, 2021-03-15T22:36:30.787 INFO:teuthology.orchestra.run.smithi076.stdout: "last_epoch_clean": 0, 2021-03-15T22:36:30.788 INFO:teuthology.orchestra.run.smithi076.stdout: "source_version": "0'0", 2021-03-15T22:36:30.788 INFO:teuthology.orchestra.run.smithi076.stdout: "target_version": "0'0" 2021-03-15T22:36:30.788 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.788 INFO:teuthology.orchestra.run.smithi076.stdout: "last_change": "1035", 2021-03-15T22:36:30.789 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend": "0", 2021-03-15T22:36:30.789 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_prenautilus": "0", 2021-03-15T22:36:30.789 INFO:teuthology.orchestra.run.smithi076.stdout: "last_force_op_resend_preluminous": "0", 2021-03-15T22:36:30.789 INFO:teuthology.orchestra.run.smithi076.stdout: "auid": 0, 2021-03-15T22:36:30.789 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_mode": "selfmanaged", 2021-03-15T22:36:30.790 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_seq": 891, 2021-03-15T22:36:30.790 INFO:teuthology.orchestra.run.smithi076.stdout: "snap_epoch": 1035, 2021-03-15T22:36:30.790 INFO:teuthology.orchestra.run.smithi076.stdout: "pool_snaps": [], 2021-03-15T22:36:30.790 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps": "[]", 2021-03-15T22:36:30.790 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_bytes": 0, 2021-03-15T22:36:30.791 INFO:teuthology.orchestra.run.smithi076.stdout: "quota_max_objects": 0, 2021-03-15T22:36:30.791 INFO:teuthology.orchestra.run.smithi076.stdout: "tiers": [], 2021-03-15T22:36:30.791 INFO:teuthology.orchestra.run.smithi076.stdout: "tier_of": -1, 2021-03-15T22:36:30.791 INFO:teuthology.orchestra.run.smithi076.stdout: "read_tier": -1, 2021-03-15T22:36:30.792 INFO:teuthology.orchestra.run.smithi076.stdout: "write_tier": -1, 2021-03-15T22:36:30.792 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_mode": "none", 2021-03-15T22:36:30.792 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_bytes": 0, 2021-03-15T22:36:30.792 INFO:teuthology.orchestra.run.smithi076.stdout: "target_max_objects": 0, 2021-03-15T22:36:30.792 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_ratio_micro": 400000, 2021-03-15T22:36:30.793 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2021-03-15T22:36:30.793 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_target_full_ratio_micro": 800000, 2021-03-15T22:36:30.793 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_flush_age": 0, 2021-03-15T22:36:30.793 INFO:teuthology.orchestra.run.smithi076.stdout: "cache_min_evict_age": 0, 2021-03-15T22:36:30.793 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profile": "", 2021-03-15T22:36:30.794 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_params": { 2021-03-15T22:36:30.794 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "none" 2021-03-15T22:36:30.794 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.794 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_period": 0, 2021-03-15T22:36:30.795 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_count": 0, 2021-03-15T22:36:30.795 INFO:teuthology.orchestra.run.smithi076.stdout: "use_gmt_hitset": true, 2021-03-15T22:36:30.795 INFO:teuthology.orchestra.run.smithi076.stdout: "min_read_recency_for_promote": 0, 2021-03-15T22:36:30.795 INFO:teuthology.orchestra.run.smithi076.stdout: "min_write_recency_for_promote": 0, 2021-03-15T22:36:30.795 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_grade_decay_rate": 0, 2021-03-15T22:36:30.796 INFO:teuthology.orchestra.run.smithi076.stdout: "hit_set_search_last_n": 0, 2021-03-15T22:36:30.796 INFO:teuthology.orchestra.run.smithi076.stdout: "grade_table": [], 2021-03-15T22:36:30.796 INFO:teuthology.orchestra.run.smithi076.stdout: "stripe_width": 0, 2021-03-15T22:36:30.796 INFO:teuthology.orchestra.run.smithi076.stdout: "expected_num_objects": 0, 2021-03-15T22:36:30.796 INFO:teuthology.orchestra.run.smithi076.stdout: "fast_read": false, 2021-03-15T22:36:30.797 INFO:teuthology.orchestra.run.smithi076.stdout: "options": {}, 2021-03-15T22:36:30.797 INFO:teuthology.orchestra.run.smithi076.stdout: "application_metadata": { 2021-03-15T22:36:30.797 INFO:teuthology.orchestra.run.smithi076.stdout: "rados": {} 2021-03-15T22:36:30.797 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.798 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.798 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:36:30.798 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:36:30.798 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.798 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:36:30.799 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd", 2021-03-15T22:36:30.799 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:36:30.799 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:36:30.799 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:36:30.799 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:36:30.800 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:36:30.800 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:36:30.800 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 8, 2021-03-15T22:36:30.800 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 1024, 2021-03-15T22:36:30.801 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:36:30.801 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:36:30.801 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:36:30.801 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.801 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.802 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.802 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6802", 2021-03-15T22:36:30.802 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:36:30.802 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.802 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.803 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.803 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6803", 2021-03-15T22:36:30.803 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:36:30.803 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.804 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.804 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.804 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:36:30.804 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.804 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.805 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.805 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6804", 2021-03-15T22:36:30.805 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:36:30.805 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.805 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.806 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.806 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6805", 2021-03-15T22:36:30.806 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:36:30.806 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.807 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.807 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.807 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:36:30.807 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.807 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.808 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.808 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6808", 2021-03-15T22:36:30.808 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:36:30.808 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.808 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.809 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.809 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6809", 2021-03-15T22:36:30.809 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:36:30.809 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.810 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.810 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.810 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:36:30.810 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.810 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.811 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.811 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6806", 2021-03-15T22:36:30.811 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:36:30.811 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.812 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.812 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.812 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6807", 2021-03-15T22:36:30.812 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 188228176 2021-03-15T22:36:30.812 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.813 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.813 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.813 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6803/188228176", 2021-03-15T22:36:30.813 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6805/188228176", 2021-03-15T22:36:30.813 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6809/188228176", 2021-03-15T22:36:30.814 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6807/188228176", 2021-03-15T22:36:30.814 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:36:30.814 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:36:30.814 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:36:30.815 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.815 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.815 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.815 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:36:30.815 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "97bd3ffc-15c8-4a6f-b37a-217813def526", 2021-03-15T22:36:30.816 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:36:30.816 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:36:30.816 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0.8248748779296875, 2021-03-15T22:36:30.816 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.999237060546875, 2021-03-15T22:36:30.817 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:36:30.817 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:36:30.817 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 13, 2021-03-15T22:36:30.817 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 1030, 2021-03-15T22:36:30.817 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:36:30.818 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:36:30.818 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:36:30.818 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.818 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.818 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.819 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6810", 2021-03-15T22:36:30.819 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:36:30.819 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.819 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.820 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.820 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6811", 2021-03-15T22:36:30.820 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:36:30.820 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.820 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.821 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.821 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:36:30.821 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.821 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.822 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.822 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6812", 2021-03-15T22:36:30.822 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:36:30.822 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.822 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.823 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.823 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6813", 2021-03-15T22:36:30.823 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:36:30.823 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.823 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.824 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.824 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:36:30.824 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.824 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.825 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.825 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6816", 2021-03-15T22:36:30.825 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:36:30.825 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.825 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.826 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.826 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6817", 2021-03-15T22:36:30.826 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:36:30.826 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.826 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.827 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.827 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:36:30.827 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.827 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.828 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.828 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6814", 2021-03-15T22:36:30.828 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:36:30.828 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.828 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.829 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.829 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6815", 2021-03-15T22:36:30.829 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3924955423 2021-03-15T22:36:30.829 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.830 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.830 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.830 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6811/3924955423", 2021-03-15T22:36:30.830 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6813/3924955423", 2021-03-15T22:36:30.830 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6817/3924955423", 2021-03-15T22:36:30.831 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6815/3924955423", 2021-03-15T22:36:30.831 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:36:30.831 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:36:30.831 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:36:30.831 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.832 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.832 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.832 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:36:30.832 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a7f5f115-e69e-47d9-b6c3-ff386ddfaa05", 2021-03-15T22:36:30.833 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:36:30.833 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:36:30.833 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:36:30.833 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 0.53363037109375, 2021-03-15T22:36:30.833 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 18, 2021-03-15T22:36:30.834 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 489, 2021-03-15T22:36:30.834 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 681, 2021-03-15T22:36:30.834 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 972, 2021-03-15T22:36:30.834 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 490, 2021-03-15T22:36:30.835 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:36:30.835 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:36:30.835 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.835 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.835 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.836 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6818", 2021-03-15T22:36:30.836 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:36:30.836 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.836 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.836 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.837 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6819", 2021-03-15T22:36:30.837 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:36:30.837 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.837 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.838 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.838 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:36:30.838 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.838 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.838 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.839 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6820", 2021-03-15T22:36:30.839 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:36:30.839 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.839 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.839 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.840 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6821", 2021-03-15T22:36:30.840 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:36:30.840 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.840 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.841 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.841 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:36:30.841 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.841 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.841 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.842 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6824", 2021-03-15T22:36:30.842 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:36:30.842 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.842 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.842 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.843 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6825", 2021-03-15T22:36:30.843 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:36:30.843 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.843 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.844 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.844 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:36:30.844 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.844 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.844 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.845 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6822", 2021-03-15T22:36:30.845 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:36:30.845 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.845 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.845 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.846 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6823", 2021-03-15T22:36:30.846 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 1676192618 2021-03-15T22:36:30.846 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.846 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.847 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.847 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6819/1676192618", 2021-03-15T22:36:30.847 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6821/1676192618", 2021-03-15T22:36:30.847 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6825/1676192618", 2021-03-15T22:36:30.847 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6823/1676192618", 2021-03-15T22:36:30.848 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:36:30.848 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:36:30.848 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:36:30.848 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.849 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.849 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.849 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:36:30.849 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "a0bb7f88-0114-47cb-a82c-d8fb002aac56", 2021-03-15T22:36:30.849 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:36:30.850 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:36:30.850 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 0.7639923095703125, 2021-03-15T22:36:30.850 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:36:30.850 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:36:30.850 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:36:30.851 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 26, 2021-03-15T22:36:30.851 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 1034, 2021-03-15T22:36:30.851 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:36:30.851 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:36:30.852 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:36:30.852 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.852 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.852 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.852 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6826", 2021-03-15T22:36:30.853 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:36:30.853 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.853 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.853 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.853 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6827", 2021-03-15T22:36:30.854 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:36:30.854 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.854 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.854 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.855 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:36:30.855 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.855 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.855 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.855 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6828", 2021-03-15T22:36:30.856 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:36:30.856 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.856 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.856 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.856 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6829", 2021-03-15T22:36:30.857 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:36:30.857 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.857 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.857 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.858 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:36:30.858 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.858 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.858 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.858 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6832", 2021-03-15T22:36:30.859 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:36:30.859 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.859 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.859 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.860 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6833", 2021-03-15T22:36:30.860 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:36:30.860 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.860 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.860 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.861 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:36:30.861 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.861 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.861 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.861 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6830", 2021-03-15T22:36:30.862 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:36:30.862 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.862 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.862 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.863 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.76:6831", 2021-03-15T22:36:30.863 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 3960968636 2021-03-15T22:36:30.863 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.863 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.863 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.864 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.76:6827/3960968636", 2021-03-15T22:36:30.864 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.76:6829/3960968636", 2021-03-15T22:36:30.864 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.76:6833/3960968636", 2021-03-15T22:36:30.864 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.76:6831/3960968636", 2021-03-15T22:36:30.865 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:36:30.865 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:36:30.865 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:36:30.865 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.865 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.866 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.866 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:36:30.866 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "b650611c-6101-47e8-9ca9-94e02cf8b237", 2021-03-15T22:36:30.866 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:36:30.867 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:36:30.867 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:36:30.867 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:36:30.867 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:36:30.867 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:36:30.868 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 31, 2021-03-15T22:36:30.868 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 1028, 2021-03-15T22:36:30.868 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:36:30.868 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:36:30.869 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:36:30.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.869 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.869 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.869 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6800", 2021-03-15T22:36:30.870 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:36:30.870 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.870 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.870 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.871 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6801", 2021-03-15T22:36:30.871 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:36:30.871 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.871 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.871 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.872 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:36:30.872 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.872 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.872 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.873 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6802", 2021-03-15T22:36:30.873 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:36:30.873 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.873 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.873 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.874 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6803", 2021-03-15T22:36:30.874 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:36:30.874 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.874 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.874 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.875 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:36:30.875 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.875 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.875 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.876 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6806", 2021-03-15T22:36:30.876 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:36:30.876 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.876 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.876 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.877 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6807", 2021-03-15T22:36:30.877 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:36:30.877 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.877 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.877 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.878 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:36:30.878 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.878 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.878 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.878 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6804", 2021-03-15T22:36:30.879 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:36:30.879 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.879 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.879 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.880 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6805", 2021-03-15T22:36:30.880 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 37386122 2021-03-15T22:36:30.880 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.880 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.880 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.881 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6801/37386122", 2021-03-15T22:36:30.881 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6803/37386122", 2021-03-15T22:36:30.881 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6807/37386122", 2021-03-15T22:36:30.881 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6805/37386122", 2021-03-15T22:36:30.881 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:36:30.882 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:36:30.882 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:36:30.882 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.882 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.883 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.883 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:36:30.883 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "7d6ceff2-7093-4ab9-8b96-ade448093bf2", 2021-03-15T22:36:30.883 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:36:30.883 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:36:30.884 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:36:30.884 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:36:30.884 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 38, 2021-03-15T22:36:30.884 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 539, 2021-03-15T22:36:30.884 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 616, 2021-03-15T22:36:30.885 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 1028, 2021-03-15T22:36:30.885 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 540, 2021-03-15T22:36:30.885 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:36:30.885 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:36:30.886 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.886 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.886 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.886 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6808", 2021-03-15T22:36:30.886 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:36:30.887 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.887 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.887 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.887 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6809", 2021-03-15T22:36:30.887 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:36:30.888 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.888 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.888 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.888 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:36:30.889 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.889 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.889 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.889 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6810", 2021-03-15T22:36:30.889 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:36:30.890 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.890 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.890 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.890 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6811", 2021-03-15T22:36:30.891 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:36:30.891 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.891 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.891 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.891 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:36:30.892 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.892 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.892 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.892 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6814", 2021-03-15T22:36:30.892 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:36:30.893 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.893 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.893 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.893 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6815", 2021-03-15T22:36:30.894 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:36:30.894 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.894 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.894 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.894 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:36:30.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.895 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.895 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.895 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6812", 2021-03-15T22:36:30.896 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:36:30.896 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.896 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.896 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.896 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6813", 2021-03-15T22:36:30.897 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 978017133 2021-03-15T22:36:30.897 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.897 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.897 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.898 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6809/978017133", 2021-03-15T22:36:30.898 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6811/978017133", 2021-03-15T22:36:30.898 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6815/978017133", 2021-03-15T22:36:30.898 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6813/978017133", 2021-03-15T22:36:30.898 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:36:30.899 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:36:30.899 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:36:30.899 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.899 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.899 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.900 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:36:30.900 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "98b72022-a9a7-4ee2-89a1-cac037de7afe", 2021-03-15T22:36:30.900 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:36:30.900 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:36:30.900 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:36:30.901 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:36:30.901 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:36:30.901 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:36:30.901 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 45, 2021-03-15T22:36:30.902 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 1028, 2021-03-15T22:36:30.902 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:36:30.902 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:36:30.902 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:36:30.902 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.903 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.903 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.903 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6816", 2021-03-15T22:36:30.903 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:36:30.903 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.904 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.904 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.904 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6817", 2021-03-15T22:36:30.904 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:36:30.905 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.905 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.905 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.905 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:36:30.905 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.906 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.906 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.906 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6818", 2021-03-15T22:36:30.906 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:36:30.907 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.907 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.907 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.907 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6819", 2021-03-15T22:36:30.907 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:36:30.908 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.908 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.908 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.908 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:36:30.908 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.909 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.909 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.909 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6822", 2021-03-15T22:36:30.909 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:36:30.910 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.910 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.910 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.910 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6823", 2021-03-15T22:36:30.910 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:36:30.911 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.911 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.911 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.911 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:36:30.911 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.912 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.912 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.912 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6820", 2021-03-15T22:36:30.912 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:36:30.913 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.913 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.913 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.913 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6821", 2021-03-15T22:36:30.913 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 171378275 2021-03-15T22:36:30.914 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.914 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.914 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.914 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6817/171378275", 2021-03-15T22:36:30.914 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6819/171378275", 2021-03-15T22:36:30.915 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6823/171378275", 2021-03-15T22:36:30.915 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6821/171378275", 2021-03-15T22:36:30.915 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:36:30.915 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:36:30.916 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:36:30.916 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.916 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.916 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.916 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:36:30.917 INFO:teuthology.orchestra.run.smithi076.stdout: "uuid": "f99ea650-b2dc-4d87-a44c-56a897edd614", 2021-03-15T22:36:30.917 INFO:teuthology.orchestra.run.smithi076.stdout: "up": 1, 2021-03-15T22:36:30.917 INFO:teuthology.orchestra.run.smithi076.stdout: "in": 1, 2021-03-15T22:36:30.917 INFO:teuthology.orchestra.run.smithi076.stdout: "weight": 1, 2021-03-15T22:36:30.917 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_affinity": 1, 2021-03-15T22:36:30.918 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_begin": 0, 2021-03-15T22:36:30.918 INFO:teuthology.orchestra.run.smithi076.stdout: "last_clean_end": 0, 2021-03-15T22:36:30.918 INFO:teuthology.orchestra.run.smithi076.stdout: "up_from": 52, 2021-03-15T22:36:30.919 INFO:teuthology.orchestra.run.smithi076.stdout: "up_thru": 1028, 2021-03-15T22:36:30.919 INFO:teuthology.orchestra.run.smithi076.stdout: "down_at": 0, 2021-03-15T22:36:30.919 INFO:teuthology.orchestra.run.smithi076.stdout: "lost_at": 0, 2021-03-15T22:36:30.919 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addrs": { 2021-03-15T22:36:30.920 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.920 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.920 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.920 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6824", 2021-03-15T22:36:30.921 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:36:30.921 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.921 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.921 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.921 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6825", 2021-03-15T22:36:30.922 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:36:30.922 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.922 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.922 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.922 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addrs": { 2021-03-15T22:36:30.923 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.923 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.923 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.923 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6826", 2021-03-15T22:36:30.924 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:36:30.924 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.924 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.924 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.924 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6827", 2021-03-15T22:36:30.925 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:36:30.925 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.925 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.925 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.925 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addrs": { 2021-03-15T22:36:30.926 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.926 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.926 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.926 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6830", 2021-03-15T22:36:30.927 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:36:30.927 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.927 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.927 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.927 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6831", 2021-03-15T22:36:30.928 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:36:30.928 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.928 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.928 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.928 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addrs": { 2021-03-15T22:36:30.929 INFO:teuthology.orchestra.run.smithi076.stdout: "addrvec": [ 2021-03-15T22:36:30.929 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.929 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v2", 2021-03-15T22:36:30.929 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6828", 2021-03-15T22:36:30.930 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:36:30.930 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.930 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.930 INFO:teuthology.orchestra.run.smithi076.stdout: "type": "v1", 2021-03-15T22:36:30.930 INFO:teuthology.orchestra.run.smithi076.stdout: "addr": "172.21.15.125:6829", 2021-03-15T22:36:30.931 INFO:teuthology.orchestra.run.smithi076.stdout: "nonce": 652003948 2021-03-15T22:36:30.931 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.931 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.931 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.931 INFO:teuthology.orchestra.run.smithi076.stdout: "public_addr": "172.21.15.125:6825/652003948", 2021-03-15T22:36:30.932 INFO:teuthology.orchestra.run.smithi076.stdout: "cluster_addr": "172.21.15.125:6827/652003948", 2021-03-15T22:36:30.932 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_back_addr": "172.21.15.125:6831/652003948", 2021-03-15T22:36:30.932 INFO:teuthology.orchestra.run.smithi076.stdout: "heartbeat_front_addr": "172.21.15.125:6829/652003948", 2021-03-15T22:36:30.932 INFO:teuthology.orchestra.run.smithi076.stdout: "state": [ 2021-03-15T22:36:30.933 INFO:teuthology.orchestra.run.smithi076.stdout: "exists", 2021-03-15T22:36:30.933 INFO:teuthology.orchestra.run.smithi076.stdout: "up" 2021-03-15T22:36:30.933 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.933 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.933 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:36:30.934 INFO:teuthology.orchestra.run.smithi076.stdout: "osd_xinfo": [ 2021-03-15T22:36:30.934 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.934 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 0, 2021-03-15T22:36:30.934 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:36:30.935 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:36:30.935 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:36:30.935 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:36:30.935 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:36:30.935 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:35:59.419042+0000", 2021-03-15T22:36:30.936 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:36:30.936 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.936 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.936 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 1, 2021-03-15T22:36:30.936 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:36:30.937 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:36:30.937 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:36:30.937 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:36:30.937 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:36:30.937 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:36:17.150537+0000", 2021-03-15T22:36:30.938 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:36:30.938 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.938 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.938 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 2, 2021-03-15T22:36:30.939 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:36:30.939 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:36:30.939 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:36:30.939 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:36:30.939 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:36:30.940 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:35:32.895351+0000", 2021-03-15T22:36:30.940 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 490 2021-03-15T22:36:30.940 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.940 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.940 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 3, 2021-03-15T22:36:30.941 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:36:30.941 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:36:30.941 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:36:30.941 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:36:30.942 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:36:30.942 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:35:45.679555+0000", 2021-03-15T22:36:30.942 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:36:30.942 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.942 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.943 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 4, 2021-03-15T22:36:30.943 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:36:30.943 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:36:30.943 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:36:30.944 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:36:30.944 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:36:30.944 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:36:05.116068+0000", 2021-03-15T22:36:30.944 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:36:30.944 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.945 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.945 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 5, 2021-03-15T22:36:30.945 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:36:30.945 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:36:30.945 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:36:30.946 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:36:30.946 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:36:30.946 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:36:28.333468+0000", 2021-03-15T22:36:30.946 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 540 2021-03-15T22:36:30.947 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.947 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.947 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 6, 2021-03-15T22:36:30.947 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:36:30.947 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:36:30.948 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:36:30.948 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:36:30.948 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:36:30.948 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:35:39.351216+0000", 2021-03-15T22:36:30.949 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:36:30.949 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.949 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.949 INFO:teuthology.orchestra.run.smithi076.stdout: "osd": 7, 2021-03-15T22:36:30.949 INFO:teuthology.orchestra.run.smithi076.stdout: "down_stamp": "0.000000", 2021-03-15T22:36:30.950 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_probability": 0, 2021-03-15T22:36:30.950 INFO:teuthology.orchestra.run.smithi076.stdout: "laggy_interval": 0, 2021-03-15T22:36:30.950 INFO:teuthology.orchestra.run.smithi076.stdout: "features": 4540138303579357183, 2021-03-15T22:36:30.950 INFO:teuthology.orchestra.run.smithi076.stdout: "old_weight": 0, 2021-03-15T22:36:30.950 INFO:teuthology.orchestra.run.smithi076.stdout: "last_purged_snaps_scrub": "2021-03-15T22:35:56.499151+0000", 2021-03-15T22:36:30.951 INFO:teuthology.orchestra.run.smithi076.stdout: "dead_epoch": 0 2021-03-15T22:36:30.951 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.951 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:36:30.951 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap": [ 2021-03-15T22:36:30.952 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.952 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "7.3", 2021-03-15T22:36:30.952 INFO:teuthology.orchestra.run.smithi076.stdout: "osds": [ 2021-03-15T22:36:30.952 INFO:teuthology.orchestra.run.smithi076.stdout: 1, 2021-03-15T22:36:30.952 INFO:teuthology.orchestra.run.smithi076.stdout: 6, 2021-03-15T22:36:30.953 INFO:teuthology.orchestra.run.smithi076.stdout: 2 2021-03-15T22:36:30.953 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.953 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.953 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:36:30.953 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_upmap_items": [ 2021-03-15T22:36:30.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.954 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "2.2", 2021-03-15T22:36:30.954 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:36:30.954 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.955 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 5, 2021-03-15T22:36:30.955 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 7 2021-03-15T22:36:30.955 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.955 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.955 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 6, 2021-03-15T22:36:30.956 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 0 2021-03-15T22:36:30.956 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.956 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.956 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.956 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.957 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.10", 2021-03-15T22:36:30.957 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:36:30.957 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.957 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 7, 2021-03-15T22:36:30.958 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 3 2021-03-15T22:36:30.958 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.958 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.958 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.958 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.959 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.17", 2021-03-15T22:36:30.959 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:36:30.959 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.959 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 1, 2021-03-15T22:36:30.960 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 6 2021-03-15T22:36:30.960 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.960 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.960 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.960 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.961 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.1b", 2021-03-15T22:36:30.961 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:36:30.961 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.961 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 0, 2021-03-15T22:36:30.961 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 3 2021-03-15T22:36:30.962 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.962 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.962 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.962 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.963 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "5.1f", 2021-03-15T22:36:30.963 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:36:30.963 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.963 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:36:30.963 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 3 2021-03-15T22:36:30.964 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.964 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.964 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.964 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.964 INFO:teuthology.orchestra.run.smithi076.stdout: "pgid": "6.f", 2021-03-15T22:36:30.965 INFO:teuthology.orchestra.run.smithi076.stdout: "mappings": [ 2021-03-15T22:36:30.965 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.965 INFO:teuthology.orchestra.run.smithi076.stdout: "from": 4, 2021-03-15T22:36:30.965 INFO:teuthology.orchestra.run.smithi076.stdout: "to": 0 2021-03-15T22:36:30.966 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.966 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.966 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.966 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:36:30.966 INFO:teuthology.orchestra.run.smithi076.stdout: "pg_temp": [], 2021-03-15T22:36:30.967 INFO:teuthology.orchestra.run.smithi076.stdout: "primary_temp": [], 2021-03-15T22:36:30.967 INFO:teuthology.orchestra.run.smithi076.stdout: "blocklist": { 2021-03-15T22:36:30.967 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:36:30.967 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4087324200": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:36:30.967 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/4141143304": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:36:30.968 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2604045245": "2021-03-16T22:14:00.771702+0000", 2021-03-15T22:36:30.968 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2779879991": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:36:30.968 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6801/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:36:30.968 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:6800/2764179881": "2021-03-16T22:13:41.603652+0000", 2021-03-15T22:36:30.969 INFO:teuthology.orchestra.run.smithi076.stdout: "172.21.15.76:0/2483102204": "2021-03-16T22:13:41.603652+0000" 2021-03-15T22:36:30.969 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.969 INFO:teuthology.orchestra.run.smithi076.stdout: "erasure_code_profiles": { 2021-03-15T22:36:30.969 INFO:teuthology.orchestra.run.smithi076.stdout: "default": { 2021-03-15T22:36:30.969 INFO:teuthology.orchestra.run.smithi076.stdout: "crush-failure-domain": "osd", 2021-03-15T22:36:30.970 INFO:teuthology.orchestra.run.smithi076.stdout: "k": "2", 2021-03-15T22:36:30.970 INFO:teuthology.orchestra.run.smithi076.stdout: "m": "1", 2021-03-15T22:36:30.970 INFO:teuthology.orchestra.run.smithi076.stdout: "plugin": "jerasure", 2021-03-15T22:36:30.970 INFO:teuthology.orchestra.run.smithi076.stdout: "ruleset-failure-domain": "osd", 2021-03-15T22:36:30.970 INFO:teuthology.orchestra.run.smithi076.stdout: "technique": "reed_sol_van" 2021-03-15T22:36:30.971 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.971 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.971 INFO:teuthology.orchestra.run.smithi076.stdout: "removed_snaps_queue": [ 2021-03-15T22:36:30.971 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.972 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:36:30.972 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:36:30.972 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.972 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 763, 2021-03-15T22:36:30.972 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:36:30.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.973 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.973 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 779, 2021-03-15T22:36:30.973 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:36:30.973 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.974 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.974 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 805, 2021-03-15T22:36:30.974 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:36:30.974 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.975 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 848, 2021-03-15T22:36:30.975 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:36:30.975 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.975 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.976 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 863, 2021-03-15T22:36:30.976 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:36:30.976 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.976 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.976 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.977 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:36:30.977 INFO:teuthology.orchestra.run.smithi076.stdout: "new_removed_snaps": [ 2021-03-15T22:36:30.977 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.977 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:36:30.977 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:36:30.978 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.978 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 805, 2021-03-15T22:36:30.978 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:36:30.978 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.979 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.979 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.979 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:36:30.979 INFO:teuthology.orchestra.run.smithi076.stdout: "new_purged_snaps": [ 2021-03-15T22:36:30.979 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.980 INFO:teuthology.orchestra.run.smithi076.stdout: "pool": 7, 2021-03-15T22:36:30.980 INFO:teuthology.orchestra.run.smithi076.stdout: "snaps": [ 2021-03-15T22:36:30.980 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.980 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 706, 2021-03-15T22:36:30.980 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:36:30.981 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.981 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.981 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 840, 2021-03-15T22:36:30.981 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:36:30.982 INFO:teuthology.orchestra.run.smithi076.stdout: }, 2021-03-15T22:36:30.982 INFO:teuthology.orchestra.run.smithi076.stdout: { 2021-03-15T22:36:30.982 INFO:teuthology.orchestra.run.smithi076.stdout: "begin": 851, 2021-03-15T22:36:30.982 INFO:teuthology.orchestra.run.smithi076.stdout: "length": 1 2021-03-15T22:36:30.982 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.983 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2021-03-15T22:36:30.983 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.983 INFO:teuthology.orchestra.run.smithi076.stdout: ], 2021-03-15T22:36:30.983 INFO:teuthology.orchestra.run.smithi076.stdout: "crush_node_flags": {}, 2021-03-15T22:36:30.983 INFO:teuthology.orchestra.run.smithi076.stdout: "device_class_flags": {}, 2021-03-15T22:36:30.984 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode": { 2021-03-15T22:36:30.984 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_enabled": false, 2021-03-15T22:36:30.984 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_bucket_count": 0, 2021-03-15T22:36:30.984 INFO:teuthology.orchestra.run.smithi076.stdout: "degraded_stretch_mode": 0, 2021-03-15T22:36:30.985 INFO:teuthology.orchestra.run.smithi076.stdout: "recovering_stretch_mode": 0, 2021-03-15T22:36:30.985 INFO:teuthology.orchestra.run.smithi076.stdout: "stretch_mode_bucket": 0 2021-03-15T22:36:30.985 INFO:teuthology.orchestra.run.smithi076.stdout: } 2021-03-15T22:36:30.985 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:36:30.986 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:30.987 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:36:30 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:36:31.009 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:31.010 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:31.011 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:31.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:30 smithi125 conmon[56739]: cluster 2021-03-15T22:36:29.236067+0000 2021-03-15T22:36:31.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:30 smithi125 conmon[56739]: mgr.y (mgr.14144) 1126 : cluster [DBG] pgmap v1697: 150 pgs: 5 peering, 145 active+clean; 778 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-03-15T22:36:31.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:30 smithi125 conmon[56739]: cluster 2021-03-15T22:36:30.361502+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e1035: 8 total, 8 up, 8 in 2021-03-15T22:36:31.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:30 smithi125 conmon[56739]: audit 2021-03-15T22:36:30.670799+0000 mon.a (mon.0) 2583 : audit 2021-03-15T22:36:31.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:30 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/2940542297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:36:31.112 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:31.138 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:31.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:31.140 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:31.241 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:31.259 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:30 smithi076 conmon[56514]: cluster 2021-03-15T22:36:29.236067+0000 mgr.y (mgr.14144) 1126 : cluster [DBG] pgmap v1697: 150 pgs: 5 peering, 145 active+clean; 778 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-03-15T22:36:31.259 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:30 smithi076 conmon[56514]: 2021-03-15T22:36:31.259 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:30 smithi076 conmon[56514]: cluster 2021-03-15T22:36:30.361502+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e1035: 8 total, 8 up, 8 in 2021-03-15T22:36:31.259 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:30 smithi076 conmon[56514]: audit 2021-03-15T22:36:30.670799+0000 mon.a (mon.0) 2583 : audit [DBG] from='client.? 172.21.15.76:0/2940542297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:36:31.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:30 smithi076 conmon[62156]: cluster 2021-03-15T22:36:29.236067+0000 mgr.y (mgr.14144 2021-03-15T22:36:31.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:30 smithi076 conmon[62156]: ) 1126 : cluster [DBG] pgmap v1697: 150 pgs: 5 peering, 145 active+clean; 778 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 2.5 MiB/s rd, 2.5 MiB/s wr, 6 op/s 2021-03-15T22:36:31.260 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:30 smithi076 conmon[62156]: cluster 2021-03-15T22:36:30.361502+0000 mon.a (mon.0) 2582 : cluster [DBG] osdmap e1035: 8 total, 8 up, 8 in 2021-03-15T22:36:31.261 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:30 smithi076 conmon[62156]: audit 2021-03-15T22:36:30.670799+0000 mon.a (mon.0) 2583 : audit [DBG] from='client.? 172.21.15.76:0/2940542297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2021-03-15T22:36:31.263 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:31.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:31.265 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:31.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:31.376 INFO:tasks.rados.rados.0.smithi076.stdout:3970: done (1 left) 2021-03-15T22:36:31.377 INFO:tasks.rados.rados.0.smithi076.stdout:3971: snap_remove snap 410 2021-03-15T22:36:31.401 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:31.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:31.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:31.503 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:31.529 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:31.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:31.531 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:31.632 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:31.659 INFO:tasks.thrashosds.thrasher:j is {'epoch': 1035, 'fsid': '8a572802-85db-11eb-9091-001a4aab830c', 'created': '2021-03-15T22:13:20.370795+0000', 'modified': '2021-03-15T22:36:30.356703+0000', 'last_up_change': '2021-03-15T22:30:30.205134+0000', 'last_in_change': '2021-03-15T22:34:54.369110+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 20, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'quincy', 'pools': [{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-15T22:16:34.817191+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1028', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '1026', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'iscsi', 'create_time': '2021-03-15T22:17:49.906400+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '56', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2021-03-15T22:17:52.257970+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 23, 'pg_placement_num': 22, 'pg_placement_num_target': 22, 'pg_num_target': 22, 'pg_num_pending': 23, 'last_pg_merge_meta': {'source_pgid': '3.17', 'ready_epoch': 1034, 'last_epoch_started': 1033, 'last_epoch_clean': 1033, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1034', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '1034', 'last_force_op_resend_preluminous': '1032', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2021-03-15T22:17:54.562498+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2021-03-15T22:17:56.243806+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '5.16', 'ready_epoch': 182, 'last_epoch_started': 181, 'last_epoch_clean': 181, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1007', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '182', 'last_force_op_resend_preluminous': '728', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2021-03-15T22:17:58.730534+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '6.20', 'ready_epoch': 1024, 'last_epoch_started': 1023, 'last_epoch_clean': 1023, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1024', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '1024', 'last_force_op_resend_preluminous': '1022', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 8}, 'application_metadata': {'rgw': {}}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2021-03-15T22:20:03.377294+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '1035', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 891, 'snap_epoch': 1035, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}}], 'osds': [{'osd': 0, 'uuid': 'aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 1024, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6802', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6803', 'nonce': 188228176}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6804', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6805', 'nonce': 188228176}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6808', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6809', 'nonce': 188228176}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6806', 'nonce': 188228176}, {'type': 'v1', 'addr': '172.21.15.76:6807', 'nonce': 188228176}]}, 'public_addr': '172.21.15.76:6803/188228176', 'cluster_addr': '172.21.15.76:6805/188228176', 'heartbeat_back_addr': '172.21.15.76:6809/188228176', 'heartbeat_front_addr': '172.21.15.76:6807/188228176', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '97bd3ffc-15c8-4a6f-b37a-217813def526', 'up': 1, 'in': 1, 'weight': 0.8248748779296875, 'primary_affinity': 0.999237060546875, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 1030, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6810', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6811', 'nonce': 3924955423}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6812', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6813', 'nonce': 3924955423}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6816', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6817', 'nonce': 3924955423}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6814', 'nonce': 3924955423}, {'type': 'v1', 'addr': '172.21.15.76:6815', 'nonce': 3924955423}]}, 'public_addr': '172.21.15.76:6811/3924955423', 'cluster_addr': '172.21.15.76:6813/3924955423', 'heartbeat_back_addr': '172.21.15.76:6817/3924955423', 'heartbeat_front_addr': '172.21.15.76:6815/3924955423', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': 'a7f5f115-e69e-47d9-b6c3-ff386ddfaa05', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.53363037109375, 'last_clean_begin': 18, 'last_clean_end': 489, 'up_from': 681, 'up_thru': 972, 'down_at': 490, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6818', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6819', 'nonce': 1676192618}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6820', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6821', 'nonce': 1676192618}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6824', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6825', 'nonce': 1676192618}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6822', 'nonce': 1676192618}, {'type': 'v1', 'addr': '172.21.15.76:6823', 'nonce': 1676192618}]}, 'public_addr': '172.21.15.76:6819/1676192618', 'cluster_addr': '172.21.15.76:6821/1676192618', 'heartbeat_back_addr': '172.21.15.76:6825/1676192618', 'heartbeat_front_addr': '172.21.15.76:6823/1676192618', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': 'a0bb7f88-0114-47cb-a82c-d8fb002aac56', 'up': 1, 'in': 1, 'weight': 0.7639923095703125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 26, 'up_thru': 1034, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6826', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6827', 'nonce': 3960968636}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6828', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6829', 'nonce': 3960968636}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6832', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6833', 'nonce': 3960968636}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.76:6830', 'nonce': 3960968636}, {'type': 'v1', 'addr': '172.21.15.76:6831', 'nonce': 3960968636}]}, 'public_addr': '172.21.15.76:6827/3960968636', 'cluster_addr': '172.21.15.76:6829/3960968636', 'heartbeat_back_addr': '172.21.15.76:6833/3960968636', 'heartbeat_front_addr': '172.21.15.76:6831/3960968636', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'b650611c-6101-47e8-9ca9-94e02cf8b237', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 1028, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6800', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6801', 'nonce': 37386122}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6802', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6803', 'nonce': 37386122}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6806', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6807', 'nonce': 37386122}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6804', 'nonce': 37386122}, {'type': 'v1', 'addr': '172.21.15.125:6805', 'nonce': 37386122}]}, 'public_addr': '172.21.15.125:6801/37386122', 'cluster_addr': '172.21.15.125:6803/37386122', 'heartbeat_back_addr': '172.21.15.125:6807/37386122', 'heartbeat_front_addr': '172.21.15.125:6805/37386122', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': '7d6ceff2-7093-4ab9-8b96-ade448093bf2', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 38, 'last_clean_end': 539, 'up_from': 616, 'up_thru': 1028, 'down_at': 540, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6808', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6809', 'nonce': 978017133}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6810', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6811', 'nonce': 978017133}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6814', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6815', 'nonce': 978017133}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6812', 'nonce': 978017133}, {'type': 'v1', 'addr': '172.21.15.125:6813', 'nonce': 978017133}]}, 'public_addr': '172.21.15.125:6809/978017133', 'cluster_addr': '172.21.15.125:6811/978017133', 'heartbeat_back_addr': '172.21.15.125:6815/978017133', 'heartbeat_front_addr': '172.21.15.125:6813/978017133', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': '98b72022-a9a7-4ee2-89a1-cac037de7afe', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 45, 'up_thru': 1028, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6816', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6817', 'nonce': 171378275}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6818', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6819', 'nonce': 171378275}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6822', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6823', 'nonce': 171378275}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6820', 'nonce': 171378275}, {'type': 'v1', 'addr': '172.21.15.125:6821', 'nonce': 171378275}]}, 'public_addr': '172.21.15.125:6817/171378275', 'cluster_addr': '172.21.15.125:6819/171378275', 'heartbeat_back_addr': '172.21.15.125:6823/171378275', 'heartbeat_front_addr': '172.21.15.125:6821/171378275', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'f99ea650-b2dc-4d87-a44c-56a897edd614', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 52, 'up_thru': 1028, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6824', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6825', 'nonce': 652003948}]}, 'cluster_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6826', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6827', 'nonce': 652003948}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6830', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6831', 'nonce': 652003948}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.125:6828', 'nonce': 652003948}, {'type': 'v1', 'addr': '172.21.15.125:6829', 'nonce': 652003948}]}, 'public_addr': '172.21.15.125:6825/652003948', 'cluster_addr': '172.21.15.125:6827/652003948', 'heartbeat_back_addr': '172.21.15.125:6831/652003948', 'heartbeat_front_addr': '172.21.15.125:6829/652003948', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:35:59.419042+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:36:17.150537+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:35:32.895351+0000', 'dead_epoch': 490}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:35:45.679555+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:36:05.116068+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:36:28.333468+0000', 'dead_epoch': 540}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:35:39.351216+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540138303579357183, 'old_weight': 0, 'last_purged_snaps_scrub': '2021-03-15T22:35:56.499151+0000', 'dead_epoch': 0}], 'pg_upmap': [{'pgid': '7.3', 'osds': [1, 6, 2]}], 'pg_upmap_items': [{'pgid': '2.2', 'mappings': [{'from': 5, 'to': 7}, {'from': 6, 'to': 0}]}, {'pgid': '5.10', 'mappings': [{'from': 7, 'to': 3}]}, {'pgid': '5.17', 'mappings': [{'from': 1, 'to': 6}]}, {'pgid': '5.1b', 'mappings': [{'from': 0, 'to': 3}]}, {'pgid': '5.1f', 'mappings': [{'from': 4, 'to': 3}]}, {'pgid': '6.f', 'mappings': [{'from': 4, 'to': 0}]}], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.76:6800/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4087324200': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/4141143304': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:6801/2604045245': '2021-03-16T22:14:00.771702+0000', '172.21.15.76:0/2779879991': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6801/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:6800/2764179881': '2021-03-16T22:13:41.603652+0000', '172.21.15.76:0/2483102204': '2021-03-16T22:13:41.603652+0000'}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'ruleset-failure-domain': 'osd', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 763, 'length': 1}, {'begin': 779, 'length': 1}, {'begin': 805, 'length': 1}, {'begin': 848, 'length': 1}, {'begin': 863, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 805, 'length': 1}]}], 'new_purged_snaps': [{'pool': 7, 'snaps': [{'begin': 706, 'length': 1}, {'begin': 840, 'length': 1}, {'begin': 851, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2021-03-15T22:36:31.660 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:36:31.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:31.662 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:31.663 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:31.763 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:31.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:31.791 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:31.792 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:31.893 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:31.917 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:31.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:31.919 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:32.020 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:32.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:31 smithi125 conmon[56739]: audit 2021-03-15T22:36:30.349599+0000 mgr.y (mgr.14144) 1127 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:32.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:31 smithi125 conmon[56739]: audit 2021-03-15T22:36:31 2021-03-15T22:36:32.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:31 smithi125 conmon[56739]: .237432+0000 mon.a (mon.0) 2584 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-03-15T22:36:32.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:31 smithi125 conmon[56739]: audit 2021-03-15T22:36:31.373686+0000 mon.a (mon.0) 2585 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]': finished 2021-03-15T22:36:32.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:31 smithi125 conmon[56739]: cluster 2021-03-15T22:36 2021-03-15T22:36:32.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:31 smithi125 conmon[56739]: :31.373751+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e1036: 8 total, 8 up, 8 in 2021-03-15T22:36:32.045 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:32.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:32.046 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:32.072 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:31 smithi076 conmon[56514]: audit 2021-03-15T22:36:30.349599+0000 mgr.y (mgr.14144) 1127 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:32.073 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:31 smithi076 conmon[56514]: audit 2021-03-15T22:36:31.237432+0000 mon.a (mon.0) 2584 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-03-15T22:36:32.073 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:31 smithi076 conmon[56514]: audit 2021-03-15T22:36:31.373686+0000 mon.a (mon.0) 2585 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]': finished 2021-03-15T22:36:32.073 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:31 smithi076 conmon[56514]: cluster 2021-03-15T22:36:31.373751+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e1036: 8 total, 8 up, 8 in 2021-03-15T22:36:32.073 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:31 smithi076 conmon[62156]: audit 2021-03-15T22:36:30.349599+0000 mgr.y (mgr.14144) 1127 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:32.074 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:31 smithi076 conmon[62156]: audit 2021-03-15T22:36:31.237432+0000 mon.a (mon.0) 2584 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]: dispatch 2021-03-15T22:36:32.074 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:31 smithi076 conmon[62156]: audit 2021-03-15T22:36:31.373686+0000 mon.a (mon.0) 2585 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".rgw.root", "var": "pg_num_actual", "val": "22"}]': finished 2021-03-15T22:36:32.074 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:31 smithi076 conmon[62156]: cluster 2021-03-15T22:36:31.373751+0000 mon.a (mon.0) 2586 : cluster [DBG] osdmap e1036: 8 total, 8 up, 8 in 2021-03-15T22:36:32.147 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:32.175 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:32.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:32.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:32.278 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:32.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:32.304 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:32.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:32.328 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:36:32.328 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:36:32.328 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:36:32.328 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:36:32.382 INFO:tasks.rados.rados.0.smithi076.stdout:3971: done (1 left) 2021-03-15T22:36:32.382 INFO:tasks.rados.rados.0.smithi076.stdout:3972: rollback oid 38 current snap is 467 2021-03-15T22:36:32.383 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 38 to 442 2021-03-15T22:36:32.383 INFO:tasks.rados.rados.0.smithi076.stdout:3969: finishing write tid 1 to smithi07694878-43 2021-03-15T22:36:32.383 INFO:tasks.rados.rados.0.smithi076.stdout:3969: finishing write tid 2 to smithi07694878-43 2021-03-15T22:36:32.383 INFO:tasks.rados.rados.0.smithi076.stdout:3969: finishing write tid 3 to smithi07694878-43 2021-03-15T22:36:32.384 INFO:tasks.rados.rados.0.smithi076.stdout:3969: finishing write tid 4 to smithi07694878-43 2021-03-15T22:36:32.384 INFO:tasks.rados.rados.0.smithi076.stdout:3969: finishing write tid 5 to smithi07694878-43 2021-03-15T22:36:32.384 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 43 v 3092 (ObjNum 1339 snap 467 seq_num 1339) dirty exists 2021-03-15T22:36:32.384 INFO:tasks.rados.rados.0.smithi076.stdout:3969: left oid 43 (ObjNum 1339 snap 467 seq_num 1339) 2021-03-15T22:36:32.384 INFO:tasks.rados.rados.0.smithi076.stdout:3969: done (1 left) 2021-03-15T22:36:32.385 INFO:tasks.rados.rados.0.smithi076.stdout:3973: read oid 45 snap 402 2021-03-15T22:36:32.385 INFO:tasks.rados.rados.0.smithi076.stdout:3973: expect (ObjNum 1023 snap 341 seq_num 1023) 2021-03-15T22:36:32.385 INFO:tasks.rados.rados.0.smithi076.stdout:3974: read oid 29 snap 393 2021-03-15T22:36:32.385 INFO:tasks.rados.rados.0.smithi076.stdout:3974: expect deleted 2021-03-15T22:36:32.386 INFO:tasks.rados.rados.0.smithi076.stdout:3975: snap_create 2021-03-15T22:36:32.395 INFO:tasks.rados.rados.0.smithi076.stdout:3973: expect (ObjNum 1023 snap 341 seq_num 1023) 2021-03-15T22:36:32.406 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:32.428 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:32.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:32.429 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:32.456 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:36:32.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:32.553 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:32.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:32.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:32.582 INFO:tasks.rados.rados.0.smithi076.stdout:3972: finishing rollback tid 0 to smithi07694878-38 2021-03-15T22:36:32.583 INFO:tasks.rados.rados.0.smithi076.stdout:3972: finishing rollback tid 1 to smithi07694878-38 2021-03-15T22:36:32.583 INFO:tasks.rados.rados.0.smithi076.stdout:3972: finishing rollback tid 2 to smithi07694878-38 2021-03-15T22:36:32.583 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 2867 (ObjNum 1273 snap 439 seq_num 1273) dirty exists 2021-03-15T22:36:32.657 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:32.681 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:32.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:32.683 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:32.755 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:32.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:32.811 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:32.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:32.813 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:32.823 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_blocked_ops 2021-03-15T22:36:32.829 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:32.847 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:32 smithi076 conmon[56514]: cluster 2021-03-15T22:36:31.236723+0000 mgr.y (mgr.14144) 1128 : cluster [DBG] pgmap v1700: 149 pgs: 149 active+clean; 780 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2021-03-15T22:36:32.847 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:32 smithi076 conmon[56514]: cluster 2021-03-15T22:36:32.375186+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e1037: 8 total, 8 up, 8 in 2021-03-15T22:36:32.847 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:32 smithi076 conmon[62156]: cluster 2021-03-15T22:36:31. 2021-03-15T22:36:32.848 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:32 smithi076 conmon[62156]: 236723+0000 mgr.y (mgr.14144) 1128 : cluster [DBG] pgmap v1700: 149 pgs: 149 active+clean; 780 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2021-03-15T22:36:32.848 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:32 smithi076 conmon[62156]: cluster 2021-03-15T22:36:32.375186+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e1037: 8 total, 8 up, 8 in 2021-03-15T22:36:32.914 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:32.939 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:32.940 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:33.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:32 smithi125 conmon[56739]: cluster 2021-03-15T22:36:31.236723+0000 mgr.y (mgr.14144) 1128 : cluster [DBG] pgmap v1700: 149 pgs: 149 active+clean; 780 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.1 MiB/s wr, 11 op/s 2021-03-15T22:36:33.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:32 smithi125 conmon[56739]: cluster 2021-03-15T22:36:32.375186+0000 mon.a (mon.0) 2587 : cluster [DBG] osdmap e1037: 8 total, 8 up, 8 in 2021-03-15T22:36:33.042 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:33.069 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:33.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:33.071 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:33.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:33.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:33.196 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:33.196 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:33.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:33.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:33.323 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:33.323 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:33.383 INFO:tasks.rados.rados.0.smithi076.stdout:3972: done (3 left) 2021-03-15T22:36:33.384 INFO:tasks.rados.rados.0.smithi076.stdout:3973: done (2 left) 2021-03-15T22:36:33.384 INFO:tasks.rados.rados.0.smithi076.stdout:3974: done (1 left) 2021-03-15T22:36:33.384 INFO:tasks.rados.rados.0.smithi076.stdout:3975: done (0 left) 2021-03-15T22:36:33.385 INFO:tasks.rados.rados.0.smithi076.stdout:3976: delete oid 16 current snap is 468 2021-03-15T22:36:33.389 INFO:tasks.rados.rados.0.smithi076.stdout:3976: done (0 left) 2021-03-15T22:36:33.389 INFO:tasks.rados.rados.0.smithi076.stdout:3977: write oid 9 current snap is 468 2021-03-15T22:36:33.389 INFO:tasks.rados.rados.0.smithi076.stdout:3977: seq_num 1341 ranges {538187=587037,1538706=602473} 2021-03-15T22:36:33.395 INFO:tasks.rados.rados.0.smithi076.stdout:3977: writing smithi07694878-9 from 538187 to 1125224 tid 1 2021-03-15T22:36:33.401 INFO:tasks.rados.rados.0.smithi076.stdout:3977: writing smithi07694878-9 from 1538706 to 2141179 tid 2 2021-03-15T22:36:33.401 INFO:tasks.rados.rados.0.smithi076.stdout:3978: snap_create 2021-03-15T22:36:33.401 INFO:tasks.rados.rados.0.smithi076.stdout:3977: finishing write tid 1 to smithi07694878-9 2021-03-15T22:36:33.406 INFO:tasks.rados.rados.0.smithi076.stdout:3977: finishing write tid 2 to smithi07694878-9 2021-03-15T22:36:33.406 INFO:tasks.rados.rados.0.smithi076.stdout:3977: finishing write tid 3 to smithi07694878-9 2021-03-15T22:36:33.406 INFO:tasks.rados.rados.0.smithi076.stdout:3977: finishing write tid 4 to smithi07694878-9 2021-03-15T22:36:33.407 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 3457 (ObjNum 1341 snap 468 seq_num 1341) dirty exists 2021-03-15T22:36:33.407 INFO:tasks.rados.rados.0.smithi076.stdout:3977: left oid 9 (ObjNum 1341 snap 468 seq_num 1341) 2021-03-15T22:36:33.424 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:33.447 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:33.448 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:33.551 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:33.575 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:33.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:33.593 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:33.677 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:33.703 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:33.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:33.705 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:33.806 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:33.831 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:33.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:33.833 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:33.934 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:33.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:33.959 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:33.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:34.061 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:34.088 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:34.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:34.090 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:34.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:34.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:34.215 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:34.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:34.317 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:34.344 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:34.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:34.345 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:34.384 INFO:tasks.rados.rados.0.smithi076.stdout:3977: done (1 left) 2021-03-15T22:36:34.384 INFO:tasks.rados.rados.0.smithi076.stdout:3978: done (0 left) 2021-03-15T22:36:34.384 INFO:tasks.rados.rados.0.smithi076.stdout:3979: snap_remove snap 442 2021-03-15T22:36:34.388 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is set 2021-03-15T22:36:34.446 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:34.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:34.470 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:34.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:34.517 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: cluster 2021-03-15T22:36:33.237172+0000 mgr.y 2021-03-15T22:36:34.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: (mgr.14144) 1129 : cluster [DBG] pgmap v1703: 149 pgs: 149 active+clean; 772 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2021-03-15T22:36:34.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:36:34.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: :36:33.319236+0000 osd.2 (osd.2) 54 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:34.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: cluster 2021-03-15T22:36:34.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: 2021-03-15T22:36 2021-03-15T22:36:34.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: :33.323708+0000 osd.2 (osd.2) 2021-03-15T22:36:34.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: 55 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:34.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: cluster 2021-03-15T22:36:34.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:36:34.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: 36:33.374574+0000 mon.a (mon. 2021-03-15T22:36:34.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: 0) 2588 : cluster [DBG] osdmap e1038: 8 total, 8 up, 8 in 2021-03-15T22:36:34.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: audit 2021-03-15T22:36:34.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: 2021-03-15T22: 2021-03-15T22:36:34.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: 36:33.497790+0000 mon.c (mon.1 2021-03-15T22:36:34.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: ) 85 : audit [INF] from='client.? 172.21.15.76:0/3496975364' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:34.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: audit 2021-03-15T22:36:34.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: 2021-03-15T22:36 2021-03-15T22:36:34.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: :33.498077+0000 mon.a (mon.0 2021-03-15T22:36:34.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[62156]: ) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:34.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: cluster 2021-03-15T22:36:34.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: 2021-03-15 2021-03-15T22:36:34.523 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: T22:36:33.237172+0000 mgr.y (mgr.14144) 1129 : cluster [DBG] pgmap v1703: 149 pgs: 149 active+clean; 772 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2021-03-15T22:36:34.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:36:34.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: T22:36:33.319236+0000 osd.2 (osd.2) 54 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:34.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: cluster 2021-03- 2021-03-15T22:36:34.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: 15T22:36:33.323708+0000 osd.2 (osd.2) 55 : cluster 2021-03-15T22:36:34.524 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: [DBG] purged_snaps scrub ok 2021-03-15T22:36:34.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: cluster 2021-03-15T22:36:33.374574+0000 mon.a ( 2021-03-15T22:36:34.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: mon.0) 2588 : cluster [DBG] osdmap e1038: 8 total, 8 up, 8 in 2021-03-15T22:36:34.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: audit 2021-03-15T22 2021-03-15T22:36:34.525 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: :36:33.497790+0000 mon.c (mon.1) 85 : audit [INF] 2021-03-15T22:36:34.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/3496975364' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:34.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: audit 2021-03-15T22:36:33.498077+0000 mon.a (mon 2021-03-15T22:36:34.526 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:34 smithi076 conmon[56514]: .0) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:34.571 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:34.597 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:34.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:34.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:34.700 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:34.726 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:34.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:34.728 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: cluster 2021-03-15T22:36:33.237172+0000 mgr.y (mgr.14144) 1129 : cluster 2021-03-15T22:36:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: [DBG] pgmap v1703: 149 pgs: 149 active+clean; 772 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 13 MiB/s rd, 1.6 MiB/s wr, 14 op/s 2021-03-15T22:36:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: cluster 2021-03-15T22:36:33.319236+0000 osd.2 (osd.2) 54 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:34.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: cluster 2021-03-15T22:36:33.323708+0000 2021-03-15T22:36:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: osd.2 (osd.2) 55 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: cluster 2021-03-15T22:36: 2021-03-15T22:36:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: 33.374574+0000 mon.a (mon.0) 2588 : cluster [DBG] osdmap e1038: 8 total, 8 up, 8 in 2021-03-15T22:36:34.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:36:34.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: 03-15T22:36:33.497790+0000 mon.c 2021-03-15T22:36:34.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: (mon.1) 85 : audit [INF] from='client.? 172.21.15.76:0/3496975364' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:34.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:36:34.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: T22:36:33.498077+0000 2021-03-15T22:36:34.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:34 smithi125 conmon[56739]: mon.a (mon.0) 2589 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:34.829 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:34.852 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:34.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:34.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:34.955 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:34.976 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:34.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:34.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:35.079 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:35.106 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:35.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:35.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:35.210 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:35.236 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:35.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:35.238 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:35.339 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:35.361 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:35.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:35.363 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:35.368 INFO:tasks.rados.rados.0.smithi076.stdout:3979: done (0 left) 2021-03-15T22:36:35.368 INFO:tasks.rados.rados.0.smithi076.stdout:3980: read oid 8 snap -1 2021-03-15T22:36:35.368 INFO:tasks.rados.rados.0.smithi076.stdout:3980: expect (ObjNum 1203 snap 418 seq_num 1203) 2021-03-15T22:36:35.368 INFO:tasks.rados.rados.0.smithi076.stdout:3981: copy_from oid 21 from oid 50 current snap is 469 2021-03-15T22:36:35.369 INFO:tasks.rados.rados.0.smithi076.stdout:3982: write (excl) oid 9 current snap is 469 2021-03-15T22:36:35.369 INFO:tasks.rados.rados.0.smithi076.stdout:3982: seq_num 1342 ranges {554994=662286,1997188=557806} 2021-03-15T22:36:35.374 INFO:tasks.rados.rados.0.smithi076.stdout:3982: writing smithi07694878-9 from 554994 to 1217280 tid 1 2021-03-15T22:36:35.381 INFO:tasks.rados.rados.0.smithi076.stdout:3982: writing smithi07694878-9 from 1997188 to 2554994 tid 2 2021-03-15T22:36:35.382 INFO:tasks.rados.rados.0.smithi076.stdout:3983: snap_create 2021-03-15T22:36:35.382 INFO:tasks.rados.rados.0.smithi076.stdout:3980: expect (ObjNum 1203 snap 418 seq_num 1203) 2021-03-15T22:36:35.464 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:35.489 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:35.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:35.490 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:35.507 INFO:tasks.rados.rados.0.smithi076.stdout:3982: finishing write tid 1 to smithi07694878-9 2021-03-15T22:36:35.508 INFO:tasks.rados.rados.0.smithi076.stdout:3982: finishing write tid 2 to smithi07694878-9 2021-03-15T22:36:35.508 INFO:tasks.rados.rados.0.smithi076.stdout:3982: finishing write tid 3 to smithi07694878-9 2021-03-15T22:36:35.508 INFO:tasks.rados.rados.0.smithi076.stdout:3982: finishing write tid 4 to smithi07694878-9 2021-03-15T22:36:35.508 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 3461 (ObjNum 1342 snap 469 seq_num 1342) dirty exists 2021-03-15T22:36:35.508 INFO:tasks.rados.rados.0.smithi076.stdout:3982: left oid 9 (ObjNum 1342 snap 469 seq_num 1342) 2021-03-15T22:36:35.509 INFO:tasks.rados.rados.0.smithi076.stdout:3981: finishing copy_from to smithi07694878-21 2021-03-15T22:36:35.509 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 2693 (ObjNum 1076 snap 366 seq_num 1076) dirty exists 2021-03-15T22:36:35.509 INFO:tasks.rados.rados.0.smithi076.stdout:3981: finishing copy_from racing read to smithi07694878-21 2021-03-15T22:36:35.591 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:35.613 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:35.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:35.615 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:35.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: cluster 2021-03-15T22:36:35.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: 2021-03-15T22:36:35.673 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: 2021 2021-03-15T22:36:35.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: -03-15T22 2021-03-15T22:36:35.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: :36:34.371233+0000 mon.a (mon.0) 2590 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:35.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: audit 2021-03 2021-03-15T22:36:35.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: -15T22:36:34.378256+0000 mon.a (mon.0) 2021-03-15T22:36:35.674 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: 2591 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:36:35.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:36:35.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: -15T22:36:34.378413+0000 mon.a 2021-03-15T22:36:35.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: (mon.0) 2592 : cluster [DBG] osdmap e1039: 8 total, 8 up, 8 in 2021-03-15T22:36:35.675 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:36:35.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: -03-15T22:36:35.363322+0000 2021-03-15T22:36:35.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: mon.a (mon.0) 2593 : cluster [DBG] 2021-03-15T22:36:35.676 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[56514]: osdmap e1040: 8 total, 8 up, 8 in 2021-03-15T22:36:35.676 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[62156]: cluster 2021-03-15T22:36:34.371233+0000 mon.a 2021-03-15T22:36:35.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[62156]: (mon.0) 2590 : cluster [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:35.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[62156]: audit 2021-03-15T22:36:34.378256+0000 mon.a (mon.0) 2591 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:36:35.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[62156]: cluster 2021-03-15T22:36:35.677 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[62156]: 2021-03-15T22:36:34.378413+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e1039: 8 total, 8 up, 8 in 2021-03-15T22:36:35.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[62156]: cluster 2021-03-15T22:36:35 2021-03-15T22:36:35.678 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[62156]: .363322+0000 mon.a (mon.0) 2021-03-15T22:36:35.679 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:35 smithi076 conmon[62156]: 2593 : cluster [DBG] osdmap e1040: 8 total, 8 up, 8 in 2021-03-15T22:36:35.717 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:35.742 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:35.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:35.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:35 smithi125 conmon[56739]: cluster 2021-03-15T22:36:34.371233+0000 mon.a (mon.0) 2590 : cluster 2021-03-15T22:36:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:35 smithi125 conmon[56739]: [WRN] Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:35 smithi125 conmon[56739]: audit 2021-03-15T22:36:34.378256+0000 mon.a (mon.0) 2591 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2021-03-15T22:36:35.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:35 smithi125 conmon[56739]: cluster 2021-03-15T22:36:34.378413+0000 mon.a (mon.0) 2592 : cluster [DBG] osdmap e1039: 8 total, 8 up, 8 in 2021-03-15T22:36:35.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:35 smithi125 conmon[56739]: cluster 2021-03-15T22:36:35.363322+0000 mon.a (mon.0) 2593 : cluster [DBG] osdmap e1040: 8 total, 8 up, 8 in 2021-03-15T22:36:35.844 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:35.867 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:35.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:35.868 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:35.969 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:35.993 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:35.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:35.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:36.096 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:36.120 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:36.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:36.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:36.187 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:36:36.190 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:36:36.222 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:36.245 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:36.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:36.246 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:36.348 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:36.374 INFO:tasks.rados.rados.0.smithi076.stdout:3980: done (3 left) 2021-03-15T22:36:36.374 INFO:tasks.rados.rados.0.smithi076.stdout:3981: done (2 left) 2021-03-15T22:36:36.374 INFO:tasks.rados.rados.0.smithi076.stdout:3982: done (1 left) 2021-03-15T22:36:36.375 INFO:tasks.rados.rados.0.smithi076.stdout:3983: done (0 left) 2021-03-15T22:36:36.375 INFO:tasks.rados.rados.0.smithi076.stdout:3984: write (excl) oid 9 current snap is 470 2021-03-15T22:36:36.375 INFO:tasks.rados.rados.0.smithi076.stdout:3984: seq_num 1343 ranges {571801=737535,1939317=489789} 2021-03-15T22:36:36.376 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:36.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:36.378 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:36.381 INFO:tasks.rados.rados.0.smithi076.stdout:3984: writing smithi07694878-9 from 571801 to 1309336 tid 1 2021-03-15T22:36:36.387 INFO:tasks.rados.rados.0.smithi076.stdout:3984: writing smithi07694878-9 from 1939317 to 2429106 tid 2 2021-03-15T22:36:36.388 INFO:tasks.rados.rados.0.smithi076.stdout:3985: write (excl) oid 30 current snap is 470 2021-03-15T22:36:36.388 INFO:tasks.rados.rados.0.smithi076.stdout:3985: seq_num 1344 ranges {588608=412784,1597799=749800} 2021-03-15T22:36:36.393 INFO:tasks.rados.rados.0.smithi076.stdout:3985: writing smithi07694878-30 from 588608 to 1001392 tid 1 2021-03-15T22:36:36.394 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:36:36.394 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:36:36.395 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:36:36.396 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:36:36.402 INFO:tasks.rados.rados.0.smithi076.stdout:3985: writing smithi07694878-30 from 1597799 to 2347599 tid 2 2021-03-15T22:36:36.403 INFO:tasks.rados.rados.0.smithi076.stdout:3986: rollback oid 21 current snap is 470 2021-03-15T22:36:36.403 INFO:tasks.rados.rados.0.smithi076.stdout:rollback oid 21 to 403 2021-03-15T22:36:36.403 INFO:tasks.rados.rados.0.smithi076.stdout:3984: finishing write tid 1 to smithi07694878-9 2021-03-15T22:36:36.404 INFO:tasks.rados.rados.0.smithi076.stdout:3984: finishing write tid 2 to smithi07694878-9 2021-03-15T22:36:36.404 INFO:tasks.rados.rados.0.smithi076.stdout:3984: finishing write tid 3 to smithi07694878-9 2021-03-15T22:36:36.404 INFO:tasks.rados.rados.0.smithi076.stdout:3984: finishing write tid 4 to smithi07694878-9 2021-03-15T22:36:36.404 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 9 v 3479 (ObjNum 1343 snap 470 seq_num 1343) dirty exists 2021-03-15T22:36:36.404 INFO:tasks.rados.rados.0.smithi076.stdout:3984: left oid 9 (ObjNum 1343 snap 470 seq_num 1343) 2021-03-15T22:36:36.405 INFO:tasks.rados.rados.0.smithi076.stdout:3985: finishing write tid 1 to smithi07694878-30 2021-03-15T22:36:36.405 INFO:tasks.rados.rados.0.smithi076.stdout:3984: done (2 left) 2021-03-15T22:36:36.405 INFO:tasks.rados.rados.0.smithi076.stdout:3987: copy_from oid 14 from oid 34 current snap is 470 2021-03-15T22:36:36.405 INFO:tasks.rados.rados.0.smithi076.stdout:3988: delete oid 36 current snap is 470 2021-03-15T22:36:36.410 INFO:tasks.rados.rados.0.smithi076.stdout:3985: finishing write tid 2 to smithi07694878-30 2021-03-15T22:36:36.411 INFO:tasks.rados.rados.0.smithi076.stdout:3986: finishing rollback tid 0 to smithi07694878-21 2021-03-15T22:36:36.411 INFO:tasks.rados.rados.0.smithi076.stdout:3986: finishing rollback tid 1 to smithi07694878-21 2021-03-15T22:36:36.412 INFO:tasks.rados.rados.0.smithi076.stdout:3988: done (3 left) 2021-03-15T22:36:36.412 INFO:tasks.rados.rados.0.smithi076.stdout:3989: delete oid 3 current snap is 470 2021-03-15T22:36:36.413 INFO:tasks.rados.rados.0.smithi076.stdout:3986: finishing rollback tid 2 to smithi07694878-21 2021-03-15T22:36:36.413 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 21 v 3571 (ObjNum 1044 snap 350 seq_num 1044) dirty exists 2021-03-15T22:36:36.415 INFO:tasks.rados.rados.0.smithi076.stdout:3985: finishing write tid 3 to smithi07694878-30 2021-03-15T22:36:36.417 INFO:tasks.rados.rados.0.smithi076.stdout:3985: finishing write tid 4 to smithi07694878-30 2021-03-15T22:36:36.417 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 3483 (ObjNum 1344 snap 470 seq_num 1344) dirty exists 2021-03-15T22:36:36.417 INFO:tasks.rados.rados.0.smithi076.stdout:3985: left oid 30 (ObjNum 1344 snap 470 seq_num 1344) 2021-03-15T22:36:36.422 INFO:tasks.rados.rados.0.smithi076.stdout:3987: finishing copy_from to smithi07694878-14 2021-03-15T22:36:36.422 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 14 v 2446 (ObjNum 1265 snap 437 seq_num 1265) dirty exists 2021-03-15T22:36:36.423 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:36:36.424 INFO:tasks.rados.rados.0.smithi076.stdout:3987: finishing copy_from racing read to smithi07694878-14 2021-03-15T22:36:36.424 INFO:tasks.rados.rados.0.smithi076.stdout:3985: done (3 left) 2021-03-15T22:36:36.424 INFO:tasks.rados.rados.0.smithi076.stdout:3986: done (2 left) 2021-03-15T22:36:36.424 INFO:tasks.rados.rados.0.smithi076.stdout:3987: done (1 left) 2021-03-15T22:36:36.424 INFO:tasks.rados.rados.0.smithi076.stdout:3989: done (0 left) 2021-03-15T22:36:36.425 INFO:tasks.rados.rados.0.smithi076.stdout:3990: read oid 37 snap -1 2021-03-15T22:36:36.425 INFO:tasks.rados.rados.0.smithi076.stdout:3990: expect (ObjNum 1331 snap 462 seq_num 1331) 2021-03-15T22:36:36.425 INFO:tasks.rados.rados.0.smithi076.stdout:3991: write (excl) oid 30 current snap is 470 2021-03-15T22:36:36.425 INFO:tasks.rados.rados.0.smithi076.stdout:3991: seq_num 1347 ranges {639029=754884,2173245=465784} 2021-03-15T22:36:36.431 INFO:tasks.rados.rados.0.smithi076.stdout:3991: writing smithi07694878-30 from 639029 to 1393913 tid 1 2021-03-15T22:36:36.438 INFO:tasks.rados.rados.0.smithi076.stdout:3991: writing smithi07694878-30 from 2173245 to 2639029 tid 2 2021-03-15T22:36:36.438 INFO:tasks.rados.rados.0.smithi076.stdout:3992: delete oid 6 current snap is 470 2021-03-15T22:36:36.438 INFO:tasks.rados.rados.0.smithi076.stdout:3990: expect (ObjNum 1331 snap 462 seq_num 1331) 2021-03-15T22:36:36.459 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:36:36.479 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:36.500 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:36.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:36.502 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:36.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[56514]: cluster 2021-03-15T22:36:35.237757+0000 mgr.y (mgr.14144) 1130 : cluster [DBG] pgmap v1706: 148 pgs: 148 active+clean; 772 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 607 KiB/s wr, 3 op/s 2021-03-15T22:36:36.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[56514]: audit 2021-03-15T22:36:36.319203+0000 mon.a (mon.0) 2594 : audit [DBG] from='client.? 172.21.15.76:0/3426255880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:36.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[56514]: cluster 2021-03-15T22:36:36.371122+0000 mon.a (mon.0) 2595 : cluster [DBG] osdmap e1041: 8 total, 8 up, 8 in 2021-03-15T22:36:36.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: cluster 2021-03-15T22:36:36.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021 2021-03-15T22:36:36.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: - 2021-03-15T22:36:36.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 03 2021-03-15T22:36:36.519 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: - 2021-03-15T22:36:36.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 15 2021-03-15T22:36:36.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: T 2021-03-15T22:36:36.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 22 2021-03-15T22:36:36.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: : 2021-03-15T22:36:36.520 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 36 2021-03-15T22:36:36.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: : 2021-03-15T22:36:36.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 35 2021-03-15T22:36:36.521 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: . 2021-03-15T22:36:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 237757 2021-03-15T22:36:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: +0000 2021-03-15T22:36:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: mgr.y 2021-03-15T22:36:36.522 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: ( 2021-03-15T22:36:36.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: mgr 2021-03-15T22:36:36.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: . 2021-03-15T22:36:36.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 14144 2021-03-15T22:36:36.523 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: ) 2021-03-15T22:36:36.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 1130 2021-03-15T22:36:36.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: : 2021-03-15T22:36:36.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: cluster 2021-03-15T22:36:36.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.524 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: [DBG] 2021-03-15T22:36:36.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: pgmap v1706: 148 pgs: 148 active+clean; 772 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 607 KiB/s wr, 3 op/s 2021-03-15T22:36:36.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: audit 2021-03-15T22:36:36.525 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021 2021-03-15T22:36:36.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: - 2021-03-15T22:36:36.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 03 2021-03-15T22:36:36.526 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: - 2021-03-15T22:36:36.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 15 2021-03-15T22:36:36.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: T 2021-03-15T22:36:36.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 22 2021-03-15T22:36:36.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: : 2021-03-15T22:36:36.527 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 36 2021-03-15T22:36:36.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: : 2021-03-15T22:36:36.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 36 2021-03-15T22:36:36.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: . 2021-03-15T22:36:36.528 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 319203 2021-03-15T22:36:36.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: +0000 2021-03-15T22:36:36.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: mon.a 2021-03-15T22:36:36.529 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: ( 2021-03-15T22:36:36.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: mon 2021-03-15T22:36:36.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: . 2021-03-15T22:36:36.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 0 2021-03-15T22:36:36.530 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: ) 2021-03-15T22:36:36.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2594 2021-03-15T22:36:36.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: : 2021-03-15T22:36:36.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: audit 2021-03-15T22:36:36.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.531 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: [DBG] 2021-03-15T22:36:36.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3426255880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:36.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.532 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: cluster 2021-03-15T22:36:36.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021 2021-03-15T22:36:36.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: - 2021-03-15T22:36:36.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 03 2021-03-15T22:36:36.533 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: - 2021-03-15T22:36:36.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 15 2021-03-15T22:36:36.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: T 2021-03-15T22:36:36.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 22 2021-03-15T22:36:36.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: : 2021-03-15T22:36:36.534 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 36 2021-03-15T22:36:36.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: : 2021-03-15T22:36:36.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 36 2021-03-15T22:36:36.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: . 2021-03-15T22:36:36.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 371122 2021-03-15T22:36:36.535 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: +0000 2021-03-15T22:36:36.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: mon.a 2021-03-15T22:36:36.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: ( 2021-03-15T22:36:36.536 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: mon 2021-03-15T22:36:36.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: . 2021-03-15T22:36:36.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 0 2021-03-15T22:36:36.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: ) 2021-03-15T22:36:36.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2595 2021-03-15T22:36:36.537 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: : 2021-03-15T22:36:36.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: cluster 2021-03-15T22:36:36.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: [DBG] 2021-03-15T22:36:36.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.538 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: osdmap e1041: 8 total, 8 up, 8 in 2021-03-15T22:36:36.539 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:36 smithi076 conmon[62156]: 2021-03-15T22:36:36.568 INFO:tasks.rados.rados.0.smithi076.stdout:3991: finishing write tid 1 to smithi07694878-30 2021-03-15T22:36:36.568 INFO:tasks.rados.rados.0.smithi076.stdout:3991: finishing write tid 2 to smithi07694878-30 2021-03-15T22:36:36.568 INFO:tasks.rados.rados.0.smithi076.stdout:3991: finishing write tid 3 to smithi07694878-30 2021-03-15T22:36:36.568 INFO:tasks.rados.rados.0.smithi076.stdout:3991: finishing write tid 4 to smithi07694878-30 2021-03-15T22:36:36.569 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 30 v 3486 (ObjNum 1347 snap 470 seq_num 1347) dirty exists 2021-03-15T22:36:36.569 INFO:tasks.rados.rados.0.smithi076.stdout:3991: left oid 30 (ObjNum 1347 snap 470 seq_num 1347) 2021-03-15T22:36:36.569 INFO:tasks.rados.rados.0.smithi076.stdout:3990: done (2 left) 2021-03-15T22:36:36.569 INFO:tasks.rados.rados.0.smithi076.stdout:3991: done (1 left) 2021-03-15T22:36:36.570 INFO:tasks.rados.rados.0.smithi076.stdout:3992: done (0 left) 2021-03-15T22:36:36.570 INFO:tasks.rados.rados.0.smithi076.stdout:3993: snap_create 2021-03-15T22:36:36.573 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:36:36.574 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:36:36.574 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:36:36.574 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:36:36.575 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:36:36.575 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:36:36.575 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:36:36.575 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:36:36.576 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:36:36.576 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:36:36.603 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:36.625 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:36.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:36.627 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:36.728 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:36.750 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:36.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:36.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:36.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:36 smithi125 conmon[56739]: cluster 2021-03-15T22:36:35.237757+0000 mgr.y (mgr.14144) 2021-03-15T22:36:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:36 smithi125 conmon[56739]: 1130 : cluster [DBG] pgmap v1706: 148 pgs: 148 active+clean; 772 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 1.7 KiB/s rd, 607 KiB/s wr, 3 op/s 2021-03-15T22:36:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:36 smithi125 conmon[56739]: audit 2021-03-15T22:36:36.319203+0000 mon.a (mon.0) 2594 : audit [DBG] from='client.? 172.21.15.76:0/3426255880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:36.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:36 smithi125 conmon[56739]: cluster 2021-03-15T22:36:36.371122+0000 mon.a (mon.0) 2021-03-15T22:36:36.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:36 smithi125 conmon[56739]: 2595 : cluster [DBG] osdmap e1041: 8 total, 8 up, 8 in 2021-03-15T22:36:36.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:36.874 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:36.875 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:36.875 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:36.976 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:37.004 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:37.006 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:37.107 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:37.133 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:37.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:37.134 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:37.235 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:37.263 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:37.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:37.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:37.365 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:37.391 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:37.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:37.393 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:37.413 INFO:tasks.rados.rados.0.smithi076.stdout:3993: done (0 left) 2021-03-15T22:36:37.414 INFO:tasks.rados.rados.0.smithi076.stdout:3994: copy_from oid 22 from oid 29 current snap is 471 2021-03-15T22:36:37.414 INFO:tasks.rados.rados.0.smithi076.stdout:3995: write (excl) oid 10 current snap is 471 2021-03-15T22:36:37.414 INFO:tasks.rados.rados.0.smithi076.stdout:3995: seq_num 1349 ranges {672643=505382,1890209=500796} 2021-03-15T22:36:37.419 INFO:tasks.rados.rados.0.smithi076.stdout:3995: writing smithi07694878-10 from 672643 to 1178025 tid 1 2021-03-15T22:36:37.426 INFO:tasks.rados.rados.0.smithi076.stdout:3995: writing smithi07694878-10 from 1890209 to 2391005 tid 2 2021-03-15T22:36:37.427 INFO:tasks.rados.rados.0.smithi076.stdout:3996: snap_create 2021-03-15T22:36:37.427 INFO:tasks.rados.rados.0.smithi076.stdout:3995: finishing write tid 1 to smithi07694878-10 2021-03-15T22:36:37.433 INFO:tasks.rados.rados.0.smithi076.stdout:3995: finishing write tid 2 to smithi07694878-10 2021-03-15T22:36:37.436 INFO:tasks.rados.rados.0.smithi076.stdout:3995: finishing write tid 3 to smithi07694878-10 2021-03-15T22:36:37.436 INFO:tasks.rados.rados.0.smithi076.stdout:3995: finishing write tid 4 to smithi07694878-10 2021-03-15T22:36:37.436 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 3188 (ObjNum 1349 snap 471 seq_num 1349) dirty exists 2021-03-15T22:36:37.436 INFO:tasks.rados.rados.0.smithi076.stdout:3995: left oid 10 (ObjNum 1349 snap 471 seq_num 1349) 2021-03-15T22:36:37.437 INFO:tasks.rados.rados.0.smithi076.stdout:3994: finishing copy_from to smithi07694878-22 2021-03-15T22:36:37.437 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 22 v 3311 (ObjNum 1228 snap 425 seq_num 1228) dirty exists 2021-03-15T22:36:37.437 INFO:tasks.rados.rados.0.smithi076.stdout:3994: finishing copy_from racing read to smithi07694878-22 2021-03-15T22:36:37.495 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:37.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:37 smithi076 conmon[56514]: audit 2021-03-15T22:36:36.169485+0000 mgr.y (mgr.14144 2021-03-15T22:36:37.517 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:37 smithi076 conmon[56514]: ) 1131 : audit [DBG] from='client.16971 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:37.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:37 smithi076 conmon[62156]: audit 2021-03-15T22:36:36.169485+0000 2021-03-15T22:36:37.518 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:37 smithi076 conmon[62156]: mgr.y (mgr.14144) 1131 : audit [DBG] from='client.16971 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:37.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:37.520 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:37.520 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:37.615 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1707,"stamp":"2021-03-15T22:36:35.360889+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":809590063,"num_objects":673,"num_object_clones":307,"num_object_copies":2019,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":673,"num_whiteouts":12,"num_read":9995,"num_read_kb":7629735,"num_write":6204,"num_write_kb":2072322,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5900,"num_bytes_recovered":2230124932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25503,"ondisk_log_size":25503,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":447,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2144664,"kb_used_data":1754264,"kb_used_omap":273,"kb_used_meta":390062,"kb_avail":747816552,"statfs":{"total":767960285184,"available":765764149248,"internally_reserved":0,"allocated":1796366336,"data_stored":2486835063,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":399424378},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-8076324,"num_objects":-3,"num_object_clones":-3,"num_object_copies":-9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-3,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":9,"num_write_kb":2430,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.989714"},"pg_stats":[{"pgid":"5.19","version":"317'7","reported_seq":"405","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.381524+0000","last_change":"2021-03-15T22:36:18.276167+0000","last_active":"2021-03-15T22:36:31.381524+0000","last_peered":"2021-03-15T22:36:31.381524+0000","last_clean":"2021-03-15T22:36:31.381524+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:36:31.381524+0000","last_undegraded":"2021-03-15T22:36:31.381524+0000","last_fullsized":"2021-03-15T22:36:31.381524+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:36:18.276099+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:36:18.276099+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1051","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365158+0000","last_change":"2021-03-15T22:34:10.331764+0000","last_active":"2021-03-15T22:36:30.365158+0000","last_peered":"2021-03-15T22:36:30.365158+0000","last_clean":"2021-03-15T22:36:30.365158+0000","last_became_active":"2021-03-15T22:34:10.331424+0000","last_became_peered":"2021-03-15T22:34:10.331424+0000","last_unstale":"2021-03-15T22:36:30.365158+0000","last_undegraded":"2021-03-15T22:36:30.365158+0000","last_fullsized":"2021-03-15T22:36:30.365158+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.564464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.564464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"1134","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.364873+0000","last_change":"2021-03-15T22:35:14.434026+0000","last_active":"2021-03-15T22:36:30.364873+0000","last_peered":"2021-03-15T22:36:30.364873+0000","last_clean":"2021-03-15T22:36:30.364873+0000","last_became_active":"2021-03-15T22:33:04.345393+0000","last_became_peered":"2021-03-15T22:33:04.345393+0000","last_unstale":"2021-03-15T22:36:30.364873+0000","last_undegraded":"2021-03-15T22:36:30.364873+0000","last_fullsized":"2021-03-15T22:36:30.364873+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:35:14.433961+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:35:14.433961+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"343","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.364576+0000","last_change":"2021-03-15T22:36:08.386295+0000","last_active":"2021-03-15T22:36:30.364576+0000","last_peered":"2021-03-15T22:36:30.364576+0000","last_clean":"2021-03-15T22:36:30.364576+0000","last_became_active":"2021-03-15T22:33:04.345518+0000","last_became_peered":"2021-03-15T22:33:04.345518+0000","last_unstale":"2021-03-15T22:36:30.364576+0000","last_undegraded":"2021-03-15T22:36:30.364576+0000","last_fullsized":"2021-03-15T22:36:30.364576+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:08.386208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:08.386208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"118","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.364814+0000","last_change":"2021-03-15T22:34:43.345093+0000","last_active":"2021-03-15T22:36:30.364814+0000","last_peered":"2021-03-15T22:36:30.364814+0000","last_clean":"2021-03-15T22:36:30.364814+0000","last_became_active":"2021-03-15T22:34:43.344764+0000","last_became_peered":"2021-03-15T22:34:43.344764+0000","last_unstale":"2021-03-15T22:36:30.364814+0000","last_undegraded":"2021-03-15T22:36:30.364814+0000","last_fullsized":"2021-03-15T22:36:30.364814+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:08.652741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.652741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"184","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.366012+0000","last_change":"2021-03-15T22:36:08.831621+0000","last_active":"2021-03-15T22:36:30.366012+0000","last_peered":"2021-03-15T22:36:30.366012+0000","last_clean":"2021-03-15T22:36:30.366012+0000","last_became_active":"2021-03-15T22:34:55.346148+0000","last_became_peered":"2021-03-15T22:34:55.346148+0000","last_unstale":"2021-03-15T22:36:30.366012+0000","last_undegraded":"2021-03-15T22:36:30.366012+0000","last_fullsized":"2021-03-15T22:36:30.366012+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:36:08.831557+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:36:08.831557+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":5340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"888","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.382480+0000","last_change":"2021-03-15T22:35:16.848985+0000","last_active":"2021-03-15T22:36:32.382480+0000","last_peered":"2021-03-15T22:36:32.382480+0000","last_clean":"2021-03-15T22:36:32.382480+0000","last_became_active":"2021-03-15T22:34:03.570306+0000","last_became_peered":"2021-03-15T22:34:03.570306+0000","last_unstale":"2021-03-15T22:36:32.382480+0000","last_undegraded":"2021-03-15T22:36:32.382480+0000","last_fullsized":"2021-03-15T22:36:32.382480+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:16.848918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.848918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"504","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381337+0000","last_change":"2021-03-15T22:35:49.146403+0000","last_active":"2021-03-15T22:36:29.381337+0000","last_peered":"2021-03-15T22:36:29.381337+0000","last_clean":"2021-03-15T22:36:29.381337+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:36:29.381337+0000","last_undegraded":"2021-03-15T22:36:29.381337+0000","last_fullsized":"2021-03-15T22:36:29.381337+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.146337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.146337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"1190","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365385+0000","last_change":"2021-03-15T22:36:21.176476+0000","last_active":"2021-03-15T22:36:30.365385+0000","last_peered":"2021-03-15T22:36:30.365385+0000","last_clean":"2021-03-15T22:36:30.365385+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:36:30.365385+0000","last_undegraded":"2021-03-15T22:36:30.365385+0000","last_fullsized":"2021-03-15T22:36:30.365385+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:36:21.176400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:21.176400+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"323","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.376987+0000","last_change":"2021-03-15T22:35:12.404980+0000","last_active":"2021-03-15T22:36:32.376987+0000","last_peered":"2021-03-15T22:36:32.376987+0000","last_clean":"2021-03-15T22:36:32.376987+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:36:32.376987+0000","last_undegraded":"2021-03-15T22:36:32.376987+0000","last_fullsized":"2021-03-15T22:36:32.376987+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.404897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.404897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"681","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.382690+0000","last_change":"2021-03-15T22:36:22.138406+0000","last_active":"2021-03-15T22:36:32.382690+0000","last_peered":"2021-03-15T22:36:32.382690+0000","last_clean":"2021-03-15T22:36:32.382690+0000","last_became_active":"2021-03-15T22:34:43.344401+0000","last_became_peered":"2021-03-15T22:34:43.344401+0000","last_unstale":"2021-03-15T22:36:32.382690+0000","last_undegraded":"2021-03-15T22:36:32.382690+0000","last_fullsized":"2021-03-15T22:36:32.382690+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:22.138306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.138306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"1181","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365871+0000","last_change":"2021-03-15T22:35:45.410540+0000","last_active":"2021-03-15T22:36:30.365871+0000","last_peered":"2021-03-15T22:36:30.365871+0000","last_clean":"2021-03-15T22:36:30.365871+0000","last_became_active":"2021-03-15T22:34:55.335550+0000","last_became_peered":"2021-03-15T22:34:55.335550+0000","last_unstale":"2021-03-15T22:36:30.365871+0000","last_undegraded":"2021-03-15T22:36:30.365871+0000","last_fullsized":"2021-03-15T22:36:30.365871+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:35:45.410466+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.410466+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"997","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.382415+0000","last_change":"2021-03-15T22:35:10.068109+0000","last_active":"2021-03-15T22:36:31.382415+0000","last_peered":"2021-03-15T22:36:31.382415+0000","last_clean":"2021-03-15T22:36:31.382415+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:36:31.382415+0000","last_undegraded":"2021-03-15T22:36:31.382415+0000","last_fullsized":"2021-03-15T22:36:31.382415+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:35:10.068030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.068030+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1058","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.366722+0000","last_change":"2021-03-15T22:34:41.614750+0000","last_active":"2021-03-15T22:36:30.366722+0000","last_peered":"2021-03-15T22:36:30.366722+0000","last_clean":"2021-03-15T22:36:30.366722+0000","last_became_active":"2021-03-15T22:33:04.329078+0000","last_became_peered":"2021-03-15T22:33:04.329078+0000","last_unstale":"2021-03-15T22:36:30.366722+0000","last_undegraded":"2021-03-15T22:36:30.366722+0000","last_fullsized":"2021-03-15T22:36:30.366722+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.614684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.614684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"599","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.383015+0000","last_change":"2021-03-15T22:36:22.338276+0000","last_active":"2021-03-15T22:36:31.383015+0000","last_peered":"2021-03-15T22:36:31.383015+0000","last_clean":"2021-03-15T22:36:31.383015+0000","last_became_active":"2021-03-15T22:34:55.350325+0000","last_became_peered":"2021-03-15T22:34:55.350325+0000","last_unstale":"2021-03-15T22:36:31.383015+0000","last_undegraded":"2021-03-15T22:36:31.383015+0000","last_fullsized":"2021-03-15T22:36:31.383015+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:36:22.338202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.338202+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1089","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.367305+0000","last_change":"2021-03-15T22:35:42.445978+0000","last_active":"2021-03-15T22:36:30.367305+0000","last_peered":"2021-03-15T22:36:30.367305+0000","last_clean":"2021-03-15T22:36:30.367305+0000","last_became_active":"2021-03-15T22:34:55.338001+0000","last_became_peered":"2021-03-15T22:34:55.338001+0000","last_unstale":"2021-03-15T22:36:30.367305+0000","last_undegraded":"2021-03-15T22:36:30.367305+0000","last_fullsized":"2021-03-15T22:36:30.367305+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:42.445911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.445911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"873","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.370469+0000","last_change":"2021-03-15T22:35:20.025959+0000","last_active":"2021-03-15T22:36:30.370469+0000","last_peered":"2021-03-15T22:36:30.370469+0000","last_clean":"2021-03-15T22:36:30.370469+0000","last_became_active":"2021-03-15T22:33:04.336804+0000","last_became_peered":"2021-03-15T22:33:04.336804+0000","last_unstale":"2021-03-15T22:36:30.370469+0000","last_undegraded":"2021-03-15T22:36:30.370469+0000","last_fullsized":"2021-03-15T22:36:30.370469+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.025892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"484","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.367510+0000","last_change":"2021-03-15T22:33:15.617975+0000","last_active":"2021-03-15T22:36:30.367510+0000","last_peered":"2021-03-15T22:36:30.367510+0000","last_clean":"2021-03-15T22:36:30.367510+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:36:30.367510+0000","last_undegraded":"2021-03-15T22:36:30.367510+0000","last_fullsized":"2021-03-15T22:36:30.367510+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:33:15.617891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.617891+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"891'22","reported_seq":"614","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.763599+0000","last_change":"2021-03-15T22:35:38.710708+0000","last_active":"2021-03-15T22:36:29.763599+0000","last_peered":"2021-03-15T22:36:29.763599+0000","last_clean":"2021-03-15T22:36:29.763599+0000","last_became_active":"2021-03-15T22:34:06.311478+0000","last_became_peered":"2021-03-15T22:34:06.311478+0000","last_unstale":"2021-03-15T22:36:29.763599+0000","last_undegraded":"2021-03-15T22:36:29.763599+0000","last_fullsized":"2021-03-15T22:36:29.763599+0000","mapping_epoch":890,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"891'22","last_scrub_stamp":"2021-03-15T22:35:38.710643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:38.710643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1074","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.377926+0000","last_change":"2021-03-15T22:36:22.729938+0000","last_active":"2021-03-15T22:36:32.377926+0000","last_peered":"2021-03-15T22:36:32.377926+0000","last_clean":"2021-03-15T22:36:32.377926+0000","last_became_active":"2021-03-15T22:34:55.351597+0000","last_became_peered":"2021-03-15T22:34:55.351597+0000","last_unstale":"2021-03-15T22:36:32.377926+0000","last_undegraded":"2021-03-15T22:36:32.377926+0000","last_fullsized":"2021-03-15T22:36:32.377926+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:22.729845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.729845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"637","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381958+0000","last_change":"2021-03-15T22:34:10.385041+0000","last_active":"2021-03-15T22:36:29.381958+0000","last_peered":"2021-03-15T22:36:29.381958+0000","last_clean":"2021-03-15T22:36:29.381958+0000","last_became_active":"2021-03-15T22:34:10.347227+0000","last_became_peered":"2021-03-15T22:34:10.347227+0000","last_unstale":"2021-03-15T22:36:29.381958+0000","last_undegraded":"2021-03-15T22:36:29.381958+0000","last_fullsized":"2021-03-15T22:36:29.381958+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:34:07.125155+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.125155+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":3736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"1118","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381731+0000","last_change":"2021-03-15T22:36:16.071953+0000","last_active":"2021-03-15T22:36:29.381731+0000","last_peered":"2021-03-15T22:36:29.381731+0000","last_clean":"2021-03-15T22:36:29.381731+0000","last_became_active":"2021-03-15T22:34:58.353086+0000","last_became_peered":"2021-03-15T22:34:58.353086+0000","last_unstale":"2021-03-15T22:36:29.381731+0000","last_undegraded":"2021-03-15T22:36:29.381731+0000","last_fullsized":"2021-03-15T22:36:29.381731+0000","mapping_epoch":942,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:36:16.071889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.071889+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"1046","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.382827+0000","last_change":"2021-03-15T22:35:18.111879+0000","last_active":"2021-03-15T22:36:31.382827+0000","last_peered":"2021-03-15T22:36:31.382827+0000","last_clean":"2021-03-15T22:36:31.382827+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:36:31.382827+0000","last_undegraded":"2021-03-15T22:36:31.382827+0000","last_fullsized":"2021-03-15T22:36:31.382827+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:18.111815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.111815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"248","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.763509+0000","last_change":"2021-03-15T22:36:17.753463+0000","last_active":"2021-03-15T22:36:29.763509+0000","last_peered":"2021-03-15T22:36:29.763509+0000","last_clean":"2021-03-15T22:36:29.763509+0000","last_became_active":"2021-03-15T22:33:04.336100+0000","last_became_peered":"2021-03-15T22:33:04.336100+0000","last_unstale":"2021-03-15T22:36:29.763509+0000","last_undegraded":"2021-03-15T22:36:29.763509+0000","last_fullsized":"2021-03-15T22:36:29.763509+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:36:17.753373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:17.753373+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"1056","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.377904+0000","last_change":"2021-03-15T22:35:41.589000+0000","last_active":"2021-03-15T22:36:32.377904+0000","last_peered":"2021-03-15T22:36:32.377904+0000","last_clean":"2021-03-15T22:36:32.377904+0000","last_became_active":"2021-03-15T22:34:55.351089+0000","last_became_peered":"2021-03-15T22:34:55.351089+0000","last_unstale":"2021-03-15T22:36:32.377904+0000","last_undegraded":"2021-03-15T22:36:32.377904+0000","last_fullsized":"2021-03-15T22:36:32.377904+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:41.588935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.588935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"219","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.763044+0000","last_change":"2021-03-15T22:35:22.665325+0000","last_active":"2021-03-15T22:36:29.763044+0000","last_peered":"2021-03-15T22:36:29.763044+0000","last_clean":"2021-03-15T22:36:29.763044+0000","last_became_active":"2021-03-15T22:33:04.335734+0000","last_became_peered":"2021-03-15T22:33:04.335734+0000","last_unstale":"2021-03-15T22:36:29.763044+0000","last_undegraded":"2021-03-15T22:36:29.763044+0000","last_fullsized":"2021-03-15T22:36:29.763044+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.665251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.665251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"548","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365353+0000","last_change":"2021-03-15T22:33:37.752077+0000","last_active":"2021-03-15T22:36:30.365353+0000","last_peered":"2021-03-15T22:36:30.365353+0000","last_clean":"2021-03-15T22:36:30.365353+0000","last_became_active":"2021-03-15T22:33:04.342749+0000","last_became_peered":"2021-03-15T22:33:04.342749+0000","last_unstale":"2021-03-15T22:36:30.365353+0000","last_undegraded":"2021-03-15T22:36:30.365353+0000","last_fullsized":"2021-03-15T22:36:30.365353+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:33:37.752011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.752011+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":8764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"661","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365222+0000","last_change":"2021-03-15T22:34:34.783107+0000","last_active":"2021-03-15T22:36:30.365222+0000","last_peered":"2021-03-15T22:36:30.365222+0000","last_clean":"2021-03-15T22:36:30.365222+0000","last_became_active":"2021-03-15T22:33:04.333077+0000","last_became_peered":"2021-03-15T22:33:04.333077+0000","last_unstale":"2021-03-15T22:36:30.365222+0000","last_undegraded":"2021-03-15T22:36:30.365222+0000","last_fullsized":"2021-03-15T22:36:30.365222+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.783047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.783047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"162","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365437+0000","last_change":"2021-03-15T22:35:48.884382+0000","last_active":"2021-03-15T22:36:30.365437+0000","last_peered":"2021-03-15T22:36:30.365437+0000","last_clean":"2021-03-15T22:36:30.365437+0000","last_became_active":"2021-03-15T22:34:03.571049+0000","last_became_peered":"2021-03-15T22:34:03.571049+0000","last_unstale":"2021-03-15T22:36:30.365437+0000","last_undegraded":"2021-03-15T22:36:30.365437+0000","last_fullsized":"2021-03-15T22:36:30.365437+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:48.884317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:48.884317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"879","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.366092+0000","last_change":"2021-03-15T22:34:37.787302+0000","last_active":"2021-03-15T22:36:30.366092+0000","last_peered":"2021-03-15T22:36:30.366092+0000","last_clean":"2021-03-15T22:36:30.366092+0000","last_became_active":"2021-03-15T22:33:04.331607+0000","last_became_peered":"2021-03-15T22:33:04.331607+0000","last_unstale":"2021-03-15T22:36:30.366092+0000","last_undegraded":"2021-03-15T22:36:30.366092+0000","last_fullsized":"2021-03-15T22:36:30.366092+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:34:37.787235+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.787235+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"523","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381243+0000","last_change":"2021-03-15T22:35:17.250496+0000","last_active":"2021-03-15T22:36:29.381243+0000","last_peered":"2021-03-15T22:36:29.381243+0000","last_clean":"2021-03-15T22:36:29.381243+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:36:29.381243+0000","last_undegraded":"2021-03-15T22:36:29.381243+0000","last_fullsized":"2021-03-15T22:36:29.381243+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.250434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.250434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"833","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365152+0000","last_change":"2021-03-15T22:36:11.805477+0000","last_active":"2021-03-15T22:36:30.365152+0000","last_peered":"2021-03-15T22:36:30.365152+0000","last_clean":"2021-03-15T22:36:30.365152+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:36:30.365152+0000","last_undegraded":"2021-03-15T22:36:30.365152+0000","last_fullsized":"2021-03-15T22:36:30.365152+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:11.805413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:36:11.805413+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"1044","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.377598+0000","last_change":"2021-03-15T22:35:37.543358+0000","last_active":"2021-03-15T22:36:32.377598+0000","last_peered":"2021-03-15T22:36:32.377598+0000","last_clean":"2021-03-15T22:36:32.377598+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:36:32.377598+0000","last_undegraded":"2021-03-15T22:36:32.377598+0000","last_fullsized":"2021-03-15T22:36:32.377598+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:37.543289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:35:37.543289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"755","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.762742+0000","last_change":"2021-03-15T22:36:18.800914+0000","last_active":"2021-03-15T22:36:29.762742+0000","last_peered":"2021-03-15T22:36:29.762742+0000","last_clean":"2021-03-15T22:36:29.762742+0000","last_became_active":"2021-03-15T22:34:55.344555+0000","last_became_peered":"2021-03-15T22:34:55.344555+0000","last_unstale":"2021-03-15T22:36:29.762742+0000","last_undegraded":"2021-03-15T22:36:29.762742+0000","last_fullsized":"2021-03-15T22:36:29.762742+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:36:18.800824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:18.800824+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"420","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381378+0000","last_change":"2021-03-15T22:35:23.291568+0000","last_active":"2021-03-15T22:36:29.381378+0000","last_peered":"2021-03-15T22:36:29.381378+0000","last_clean":"2021-03-15T22:36:29.381378+0000","last_became_active":"2021-03-15T22:34:55.355119+0000","last_became_peered":"2021-03-15T22:34:55.355119+0000","last_unstale":"2021-03-15T22:36:29.381378+0000","last_undegraded":"2021-03-15T22:36:29.381378+0000","last_fullsized":"2021-03-15T22:36:29.381378+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.291507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.291507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"1099","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.382147+0000","last_change":"2021-03-15T22:35:50.069546+0000","last_active":"2021-03-15T22:36:31.382147+0000","last_peered":"2021-03-15T22:36:31.382147+0000","last_clean":"2021-03-15T22:36:31.382147+0000","last_became_active":"2021-03-15T22:34:10.353314+0000","last_became_peered":"2021-03-15T22:34:10.353314+0000","last_unstale":"2021-03-15T22:36:31.382147+0000","last_undegraded":"2021-03-15T22:36:31.382147+0000","last_fullsized":"2021-03-15T22:36:31.382147+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:50.069463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:35:50.069463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"176","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.367070+0000","last_change":"2021-03-15T22:36:20.132263+0000","last_active":"2021-03-15T22:36:30.367070+0000","last_peered":"2021-03-15T22:36:30.367070+0000","last_clean":"2021-03-15T22:36:30.367070+0000","last_became_active":"2021-03-15T22:33:45.314042+0000","last_became_peered":"2021-03-15T22:33:45.314042+0000","last_unstale":"2021-03-15T22:36:30.367070+0000","last_undegraded":"2021-03-15T22:36:30.367070+0000","last_fullsized":"2021-03-15T22:36:30.367070+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:20.132182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.132182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"1122","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365797+0000","last_change":"2021-03-15T22:34:42.632034+0000","last_active":"2021-03-15T22:36:30.365797+0000","last_peered":"2021-03-15T22:36:30.365797+0000","last_clean":"2021-03-15T22:36:30.365797+0000","last_became_active":"2021-03-15T22:33:04.340460+0000","last_became_peered":"2021-03-15T22:33:04.340460+0000","last_unstale":"2021-03-15T22:36:30.365797+0000","last_undegraded":"2021-03-15T22:36:30.365797+0000","last_fullsized":"2021-03-15T22:36:30.365797+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:34:42.631962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.631962+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"531","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.382068+0000","last_change":"2021-03-15T22:36:22.004600+0000","last_active":"2021-03-15T22:36:29.382068+0000","last_peered":"2021-03-15T22:36:29.382068+0000","last_clean":"2021-03-15T22:36:29.382068+0000","last_became_active":"2021-03-15T22:34:10.326881+0000","last_became_peered":"2021-03-15T22:34:10.326881+0000","last_unstale":"2021-03-15T22:36:29.382068+0000","last_undegraded":"2021-03-15T22:36:29.382068+0000","last_fullsized":"2021-03-15T22:36:29.382068+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:22.004539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.004539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"808'9","reported_seq":"504","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.384675+0000","last_change":"2021-03-15T22:35:41.986386+0000","last_active":"2021-03-15T22:36:32.384675+0000","last_peered":"2021-03-15T22:36:32.384675+0000","last_clean":"2021-03-15T22:36:32.384675+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:36:32.384675+0000","last_undegraded":"2021-03-15T22:36:32.384675+0000","last_fullsized":"2021-03-15T22:36:32.384675+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9","last_scrub_stamp":"2021-03-15T22:35:41.986318+0000","last_deep_scrub":"808'9","last_deep_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.986318+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"1041","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.377442+0000","last_change":"2021-03-15T22:36:09.833634+0000","last_active":"2021-03-15T22:36:32.377442+0000","last_peered":"2021-03-15T22:36:32.377442+0000","last_clean":"2021-03-15T22:36:32.377442+0000","last_became_active":"2021-03-15T22:34:55.351042+0000","last_became_peered":"2021-03-15T22:34:55.351042+0000","last_unstale":"2021-03-15T22:36:32.377442+0000","last_undegraded":"2021-03-15T22:36:32.377442+0000","last_fullsized":"2021-03-15T22:36:32.377442+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:09.833552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:36:09.833552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"844'33","reported_seq":"894","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.366940+0000","last_change":"2021-03-15T22:34:55.387712+0000","last_active":"2021-03-15T22:36:30.366940+0000","last_peered":"2021-03-15T22:36:30.366940+0000","last_clean":"2021-03-15T22:36:30.366940+0000","last_became_active":"2021-03-15T22:34:55.351721+0000","last_became_peered":"2021-03-15T22:34:55.351721+0000","last_unstale":"2021-03-15T22:36:30.366940+0000","last_undegraded":"2021-03-15T22:36:30.366940+0000","last_fullsized":"2021-03-15T22:36:30.366940+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"844'33","last_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_deep_scrub":"844'33","last_deep_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.622437+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1075","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.371077+0000","last_change":"2021-03-15T22:35:40.412203+0000","last_active":"2021-03-15T22:36:30.371077+0000","last_peered":"2021-03-15T22:36:30.371077+0000","last_clean":"2021-03-15T22:36:30.371077+0000","last_became_active":"2021-03-15T22:34:55.340440+0000","last_became_peered":"2021-03-15T22:34:55.340440+0000","last_unstale":"2021-03-15T22:36:30.371077+0000","last_undegraded":"2021-03-15T22:36:30.371077+0000","last_fullsized":"2021-03-15T22:36:30.371077+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:40.412142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:40.412142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"941'17","reported_seq":"579","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.377849+0000","last_change":"2021-03-15T22:36:17.843284+0000","last_active":"2021-03-15T22:36:32.377849+0000","last_peered":"2021-03-15T22:36:32.377849+0000","last_clean":"2021-03-15T22:36:32.377849+0000","last_became_active":"2021-03-15T22:34:55.351682+0000","last_became_peered":"2021-03-15T22:34:55.351682+0000","last_unstale":"2021-03-15T22:36:32.377849+0000","last_undegraded":"2021-03-15T22:36:32.377849+0000","last_fullsized":"2021-03-15T22:36:32.377849+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'17","last_scrub_stamp":"2021-03-15T22:36:17.843205+0000","last_deep_scrub":"896'16","last_deep_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_clean_scrub_stamp":"2021-03-15T22:36:17.843205+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"531","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381603+0000","last_change":"2021-03-15T22:36:10.121601+0000","last_active":"2021-03-15T22:36:29.381603+0000","last_peered":"2021-03-15T22:36:29.381603+0000","last_clean":"2021-03-15T22:36:29.381603+0000","last_became_active":"2021-03-15T22:34:10.347512+0000","last_became_peered":"2021-03-15T22:34:10.347512+0000","last_unstale":"2021-03-15T22:36:29.381603+0000","last_undegraded":"2021-03-15T22:36:29.381603+0000","last_fullsized":"2021-03-15T22:36:29.381603+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:10.121536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:36:10.121536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"436","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.383407+0000","last_change":"2021-03-15T22:36:11.021642+0000","last_active":"2021-03-15T22:36:32.383407+0000","last_peered":"2021-03-15T22:36:32.383407+0000","last_clean":"2021-03-15T22:36:32.383407+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:36:32.383407+0000","last_undegraded":"2021-03-15T22:36:32.383407+0000","last_fullsized":"2021-03-15T22:36:32.383407+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:36:11.021564+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:36:11.021564+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"118","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.368190+0000","last_change":"2021-03-15T22:34:43.345092+0000","last_active":"2021-03-15T22:36:30.368190+0000","last_peered":"2021-03-15T22:36:30.368190+0000","last_clean":"2021-03-15T22:36:30.368190+0000","last_became_active":"2021-03-15T22:34:43.344790+0000","last_became_peered":"2021-03-15T22:34:43.344790+0000","last_unstale":"2021-03-15T22:36:30.368190+0000","last_undegraded":"2021-03-15T22:36:30.368190+0000","last_fullsized":"2021-03-15T22:36:30.368190+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.621539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"113","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.374899+0000","last_change":"2021-03-15T22:35:20.381407+0000","last_active":"2021-03-15T22:36:32.374899+0000","last_peered":"2021-03-15T22:36:32.374899+0000","last_clean":"2021-03-15T22:36:32.374899+0000","last_became_active":"2021-03-15T22:34:55.348663+0000","last_became_peered":"2021-03-15T22:34:55.348663+0000","last_unstale":"2021-03-15T22:36:32.374899+0000","last_undegraded":"2021-03-15T22:36:32.374899+0000","last_fullsized":"2021-03-15T22:36:32.374899+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.381352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"1034","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.376932+0000","last_change":"2021-03-15T22:35:48.581266+0000","last_active":"2021-03-15T22:36:32.376932+0000","last_peered":"2021-03-15T22:36:32.376932+0000","last_clean":"2021-03-15T22:36:32.376932+0000","last_became_active":"2021-03-15T22:33:04.330195+0000","last_became_peered":"2021-03-15T22:33:04.330195+0000","last_unstale":"2021-03-15T22:36:32.376932+0000","last_undegraded":"2021-03-15T22:36:32.376932+0000","last_fullsized":"2021-03-15T22:36:32.376932+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:48.581162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:48.581162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"562","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381806+0000","last_change":"2021-03-15T22:34:37.435070+0000","last_active":"2021-03-15T22:36:29.381806+0000","last_peered":"2021-03-15T22:36:29.381806+0000","last_clean":"2021-03-15T22:36:29.381806+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:36:29.381806+0000","last_undegraded":"2021-03-15T22:36:29.381806+0000","last_fullsized":"2021-03-15T22:36:29.381806+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:34:37.435006+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.435006+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"840","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.384919+0000","last_change":"2021-03-15T22:35:20.884067+0000","last_active":"2021-03-15T22:36:32.384919+0000","last_peered":"2021-03-15T22:36:32.384919+0000","last_clean":"2021-03-15T22:36:32.384919+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:36:32.384919+0000","last_undegraded":"2021-03-15T22:36:32.384919+0000","last_fullsized":"2021-03-15T22:36:32.384919+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.883990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.883990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"1028","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.377505+0000","last_change":"2021-03-15T22:35:45.547459+0000","last_active":"2021-03-15T22:36:32.377505+0000","last_peered":"2021-03-15T22:36:32.377505+0000","last_clean":"2021-03-15T22:36:32.377505+0000","last_became_active":"2021-03-15T22:33:04.331013+0000","last_became_peered":"2021-03-15T22:33:04.331013+0000","last_unstale":"2021-03-15T22:36:32.377505+0000","last_undegraded":"2021-03-15T22:36:32.377505+0000","last_fullsized":"2021-03-15T22:36:32.377505+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.547395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:45.547395+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.547395+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"219","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.762238+0000","last_change":"2021-03-15T22:35:45.757312+0000","last_active":"2021-03-15T22:36:29.762238+0000","last_peered":"2021-03-15T22:36:29.762238+0000","last_clean":"2021-03-15T22:36:29.762238+0000","last_became_active":"2021-03-15T22:33:04.336886+0000","last_became_peered":"2021-03-15T22:33:04.336886+0000","last_unstale":"2021-03-15T22:36:29.762238+0000","last_undegraded":"2021-03-15T22:36:29.762238+0000","last_fullsized":"2021-03-15T22:36:29.762238+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.757254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:45.757254+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.757254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"387","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.379235+0000","last_change":"2021-03-15T22:34:55.387006+0000","last_active":"2021-03-15T22:36:32.379235+0000","last_peered":"2021-03-15T22:36:32.379235+0000","last_clean":"2021-03-15T22:36:32.379235+0000","last_became_active":"2021-03-15T22:34:55.351977+0000","last_became_peered":"2021-03-15T22:34:55.351977+0000","last_unstale":"2021-03-15T22:36:32.379235+0000","last_undegraded":"2021-03-15T22:36:32.379235+0000","last_fullsized":"2021-03-15T22:36:32.379235+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:34:36.438541+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:34:36.438541+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"1037'1541","reported_seq":"4773","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.399450+0000","last_change":"2021-03-15T22:36:32.399890+0000","last_active":"2021-03-15T22:36:32.399450+0000","last_peered":"2021-03-15T22:36:32.399450+0000","last_clean":"2021-03-15T22:36:32.399450+0000","last_became_active":"2021-03-15T22:33:08.355490+0000","last_became_peered":"2021-03-15T22:33:08.355490+0000","last_unstale":"2021-03-15T22:36:32.399450+0000","last_undegraded":"2021-03-15T22:36:32.399450+0000","last_fullsized":"2021-03-15T22:36:32.399450+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'1350","last_scrub_stamp":"2021-03-15T22:35:16.036089+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.036089+0000","log_size":1541,"ondisk_log_size":1541,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":53562661,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":509,"num_read_kb":562639,"num_write":351,"num_write_kb":126327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":84869361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2e8","length":"1"},{"start":"2fb","length":"1"},{"start":"30a","length":"2"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.a","version":"0'0","reported_seq":"980","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.385330+0000","last_change":"2021-03-15T22:33:42.741529+0000","last_active":"2021-03-15T22:36:32.385330+0000","last_peered":"2021-03-15T22:36:32.385330+0000","last_clean":"2021-03-15T22:36:32.385330+0000","last_became_active":"2021-03-15T22:33:04.340352+0000","last_became_peered":"2021-03-15T22:33:04.340352+0000","last_unstale":"2021-03-15T22:36:32.385330+0000","last_undegraded":"2021-03-15T22:36:32.385330+0000","last_fullsized":"2021-03-15T22:36:32.385330+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:42.741451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:42.741451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"874'4","reported_seq":"1069","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.378529+0000","last_change":"2021-03-15T22:36:14.798866+0000","last_active":"2021-03-15T22:36:32.378529+0000","last_peered":"2021-03-15T22:36:32.378529+0000","last_clean":"2021-03-15T22:36:32.378529+0000","last_became_active":"2021-03-15T22:35:58.360849+0000","last_became_peered":"2021-03-15T22:35:58.360849+0000","last_unstale":"2021-03-15T22:36:32.378529+0000","last_undegraded":"2021-03-15T22:36:32.378529+0000","last_fullsized":"2021-03-15T22:36:32.378529+0000","mapping_epoch":552,"log_start":"874'4","ondisk_log_start":"874'4","created":56,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"874'4","last_scrub_stamp":"2021-03-15T22:36:14.798788+0000","last_deep_scrub":"874'4","last_deep_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.798788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":965,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"507","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.385502+0000","last_change":"2021-03-15T22:33:38.759477+0000","last_active":"2021-03-15T22:36:32.385502+0000","last_peered":"2021-03-15T22:36:32.385502+0000","last_clean":"2021-03-15T22:36:32.385502+0000","last_became_active":"2021-03-15T22:33:04.326070+0000","last_became_peered":"2021-03-15T22:33:04.326070+0000","last_unstale":"2021-03-15T22:36:32.385502+0000","last_undegraded":"2021-03-15T22:36:32.385502+0000","last_fullsized":"2021-03-15T22:36:32.385502+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:33:38.759395+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.759395+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"841'14","reported_seq":"1320","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.377218+0000","last_change":"2021-03-15T22:36:13.797925+0000","last_active":"2021-03-15T22:36:32.377218+0000","last_peered":"2021-03-15T22:36:32.377218+0000","last_clean":"2021-03-15T22:36:32.377218+0000","last_became_active":"2021-03-15T22:33:08.354031+0000","last_became_peered":"2021-03-15T22:33:08.354031+0000","last_unstale":"2021-03-15T22:36:32.377218+0000","last_undegraded":"2021-03-15T22:36:32.377218+0000","last_fullsized":"2021-03-15T22:36:32.377218+0000","mapping_epoch":837,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'14","last_scrub_stamp":"2021-03-15T22:36:13.797841+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:36:13.797841+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"1036'2235","reported_seq":"5715","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.395489+0000","last_change":"2021-03-15T22:36:31.395764+0000","last_active":"2021-03-15T22:36:31.395489+0000","last_peered":"2021-03-15T22:36:31.395489+0000","last_clean":"2021-03-15T22:36:31.395489+0000","last_became_active":"2021-03-15T22:34:58.336162+0000","last_became_peered":"2021-03-15T22:34:58.336162+0000","last_unstale":"2021-03-15T22:36:31.395489+0000","last_undegraded":"2021-03-15T22:36:31.395489+0000","last_fullsized":"2021-03-15T22:36:31.395489+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'2031","last_scrub_stamp":"2021-03-15T22:35:42.023328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.023328+0000","log_size":2235,"ondisk_log_size":2235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":84134651,"num_objects":39,"num_object_clones":34,"num_object_copies":117,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":39,"num_whiteouts":1,"num_read":709,"num_read_kb":662809,"num_write":554,"num_write_kb":213119,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":198498497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2e8","length":"1"},{"start":"2fb","length":"1"},{"start":"30b","length":"1"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.b","version":"0'0","reported_seq":"464","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365029+0000","last_change":"2021-03-15T22:32:42.621654+0000","last_active":"2021-03-15T22:36:30.365029+0000","last_peered":"2021-03-15T22:36:30.365029+0000","last_clean":"2021-03-15T22:36:30.365029+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:36:30.365029+0000","last_undegraded":"2021-03-15T22:36:30.365029+0000","last_fullsized":"2021-03-15T22:36:30.365029+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:42.621594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:42.621594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"1036","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.378354+0000","last_change":"2021-03-15T22:36:02.374943+0000","last_active":"2021-03-15T22:36:32.378354+0000","last_peered":"2021-03-15T22:36:32.378354+0000","last_clean":"2021-03-15T22:36:32.378354+0000","last_became_active":"2021-03-15T22:36:02.374583+0000","last_became_peered":"2021-03-15T22:36:02.374583+0000","last_unstale":"2021-03-15T22:36:32.378354+0000","last_undegraded":"2021-03-15T22:36:32.378354+0000","last_fullsized":"2021-03-15T22:36:32.378354+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1007,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:47.601457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.601457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"1245","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.369162+0000","last_change":"2021-03-15T22:34:11.031994+0000","last_active":"2021-03-15T22:36:30.369162+0000","last_peered":"2021-03-15T22:36:30.369162+0000","last_clean":"2021-03-15T22:36:30.369162+0000","last_became_active":"2021-03-15T22:34:10.342120+0000","last_became_peered":"2021-03-15T22:34:10.342120+0000","last_unstale":"2021-03-15T22:36:30.369162+0000","last_undegraded":"2021-03-15T22:36:30.369162+0000","last_fullsized":"2021-03-15T22:36:30.369162+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:34:07.452572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.452572+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"747","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381295+0000","last_change":"2021-03-15T22:34:55.356651+0000","last_active":"2021-03-15T22:36:29.381295+0000","last_peered":"2021-03-15T22:36:29.381295+0000","last_clean":"2021-03-15T22:36:29.381295+0000","last_became_active":"2021-03-15T22:34:55.356328+0000","last_became_peered":"2021-03-15T22:34:55.356328+0000","last_unstale":"2021-03-15T22:36:29.381295+0000","last_undegraded":"2021-03-15T22:36:29.381295+0000","last_fullsized":"2021-03-15T22:36:29.381295+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"1035'601","reported_seq":"3550","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.386195+0000","last_change":"2021-03-15T22:36:30.386422+0000","last_active":"2021-03-15T22:36:30.386195+0000","last_peered":"2021-03-15T22:36:30.386195+0000","last_clean":"2021-03-15T22:36:30.386195+0000","last_became_active":"2021-03-15T22:34:13.324667+0000","last_became_peered":"2021-03-15T22:34:13.324667+0000","last_unstale":"2021-03-15T22:36:30.386195+0000","last_undegraded":"2021-03-15T22:36:30.386195+0000","last_fullsized":"2021-03-15T22:36:30.386195+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'446","last_scrub_stamp":"2021-03-15T22:33:39.355306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.355306+0000","log_size":601,"ondisk_log_size":601,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20558245,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":149,"num_read_kb":124961,"num_write":134,"num_write_kb":44296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":23698344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2fb","length":"1"},{"start":"30b","length":"1"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.8","version":"0'0","reported_seq":"1104","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.376494+0000","last_change":"2021-03-15T22:36:18.835190+0000","last_active":"2021-03-15T22:36:32.376494+0000","last_peered":"2021-03-15T22:36:32.376494+0000","last_clean":"2021-03-15T22:36:32.376494+0000","last_became_active":"2021-03-15T22:35:48.365533+0000","last_became_peered":"2021-03-15T22:35:48.365533+0000","last_unstale":"2021-03-15T22:36:32.376494+0000","last_undegraded":"2021-03-15T22:36:32.376494+0000","last_fullsized":"2021-03-15T22:36:32.376494+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":993,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:18.835125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:36:18.835125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"113","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.375281+0000","last_change":"2021-03-15T22:35:49.448634+0000","last_active":"2021-03-15T22:36:32.375281+0000","last_peered":"2021-03-15T22:36:32.375281+0000","last_clean":"2021-03-15T22:36:32.375281+0000","last_became_active":"2021-03-15T22:34:55.353966+0000","last_became_peered":"2021-03-15T22:34:55.353966+0000","last_unstale":"2021-03-15T22:36:32.375281+0000","last_undegraded":"2021-03-15T22:36:32.375281+0000","last_fullsized":"2021-03-15T22:36:32.375281+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.448581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.448581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"1091","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.378251+0000","last_change":"2021-03-15T22:36:06.382789+0000","last_active":"2021-03-15T22:36:32.378251+0000","last_peered":"2021-03-15T22:36:32.378251+0000","last_clean":"2021-03-15T22:36:32.378251+0000","last_became_active":"2021-03-15T22:36:06.382477+0000","last_became_peered":"2021-03-15T22:36:06.382477+0000","last_unstale":"2021-03-15T22:36:32.378251+0000","last_undegraded":"2021-03-15T22:36:32.378251+0000","last_fullsized":"2021-03-15T22:36:32.378251+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1011,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.428842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.428842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"284","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.762869+0000","last_change":"2021-03-15T22:35:47.782159+0000","last_active":"2021-03-15T22:36:29.762869+0000","last_peered":"2021-03-15T22:36:29.762869+0000","last_clean":"2021-03-15T22:36:29.762869+0000","last_became_active":"2021-03-15T22:33:45.318867+0000","last_became_peered":"2021-03-15T22:33:45.318867+0000","last_unstale":"2021-03-15T22:36:29.762869+0000","last_undegraded":"2021-03-15T22:36:29.762869+0000","last_fullsized":"2021-03-15T22:36:29.762869+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:35:47.782092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.782092+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":2928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"1037'1376","reported_seq":"4403","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.394216+0000","last_change":"2021-03-15T22:36:32.394216+0000","last_active":"2021-03-15T22:36:32.394216+0000","last_peered":"2021-03-15T22:36:32.394216+0000","last_clean":"2021-03-15T22:36:32.394216+0000","last_became_active":"2021-03-15T22:34:59.339587+0000","last_became_peered":"2021-03-15T22:34:59.339587+0000","last_unstale":"2021-03-15T22:36:32.394216+0000","last_undegraded":"2021-03-15T22:36:32.394216+0000","last_fullsized":"2021-03-15T22:36:32.394216+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'1309","last_scrub_stamp":"2021-03-15T22:36:14.641356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.641356+0000","log_size":1376,"ondisk_log_size":1376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51061425,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":448,"num_read_kb":538912,"num_write":302,"num_write_kb":107863,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":205148835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2e8","length":"1"},{"start":"2fb","length":"1"},{"start":"30a","length":"2"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"555","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381421+0000","last_change":"2021-03-15T22:35:44.582634+0000","last_active":"2021-03-15T22:36:29.381421+0000","last_peered":"2021-03-15T22:36:29.381421+0000","last_clean":"2021-03-15T22:36:29.381421+0000","last_became_active":"2021-03-15T22:35:44.582322+0000","last_became_peered":"2021-03-15T22:35:44.582322+0000","last_unstale":"2021-03-15T22:36:29.381421+0000","last_undegraded":"2021-03-15T22:36:29.381421+0000","last_fullsized":"2021-03-15T22:36:29.381421+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"869","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365536+0000","last_change":"2021-03-15T22:34:55.343547+0000","last_active":"2021-03-15T22:36:30.365536+0000","last_peered":"2021-03-15T22:36:30.365536+0000","last_clean":"2021-03-15T22:36:30.365536+0000","last_became_active":"2021-03-15T22:34:55.342808+0000","last_became_peered":"2021-03-15T22:34:55.342808+0000","last_unstale":"2021-03-15T22:36:30.365536+0000","last_undegraded":"2021-03-15T22:36:30.365536+0000","last_fullsized":"2021-03-15T22:36:30.365536+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:14.888119+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.888119+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"456","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.383042+0000","last_change":"2021-03-15T22:36:10.362976+0000","last_active":"2021-03-15T22:36:32.383042+0000","last_peered":"2021-03-15T22:36:32.383042+0000","last_clean":"2021-03-15T22:36:32.383042+0000","last_became_active":"2021-03-15T22:36:10.362612+0000","last_became_peered":"2021-03-15T22:36:10.362612+0000","last_unstale":"2021-03-15T22:36:32.383042+0000","last_undegraded":"2021-03-15T22:36:32.383042+0000","last_fullsized":"2021-03-15T22:36:32.383042+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1015,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.807411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.807411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"582","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381957+0000","last_change":"2021-03-15T22:36:14.097504+0000","last_active":"2021-03-15T22:36:29.381957+0000","last_peered":"2021-03-15T22:36:29.381957+0000","last_clean":"2021-03-15T22:36:29.381957+0000","last_became_active":"2021-03-15T22:33:45.326747+0000","last_became_peered":"2021-03-15T22:33:45.326747+0000","last_unstale":"2021-03-15T22:36:29.381957+0000","last_undegraded":"2021-03-15T22:36:29.381957+0000","last_fullsized":"2021-03-15T22:36:29.381957+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:36:14.097432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.097432+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":2682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"19","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.382323+0000","last_change":"2021-03-15T22:36:24.384474+0000","last_active":"2021-03-15T22:36:32.382323+0000","last_peered":"2021-03-15T22:36:32.382323+0000","last_clean":"2021-03-15T22:36:32.382323+0000","last_became_active":"2021-03-15T22:36:24.384123+0000","last_became_peered":"2021-03-15T22:36:24.384123+0000","last_unstale":"2021-03-15T22:36:32.382323+0000","last_undegraded":"2021-03-15T22:36:32.382323+0000","last_fullsized":"2021-03-15T22:36:32.382323+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"1035'1565","reported_seq":"5256","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.402251+0000","last_change":"2021-03-15T22:36:30.402637+0000","last_active":"2021-03-15T22:36:30.402251+0000","last_peered":"2021-03-15T22:36:30.402251+0000","last_clean":"2021-03-15T22:36:30.402251+0000","last_became_active":"2021-03-15T22:34:46.338888+0000","last_became_peered":"2021-03-15T22:34:46.338888+0000","last_unstale":"2021-03-15T22:36:30.402251+0000","last_undegraded":"2021-03-15T22:36:30.402251+0000","last_fullsized":"2021-03-15T22:36:30.402251+0000","mapping_epoch":930,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"1025'1523","last_scrub_stamp":"2021-03-15T22:36:20.962682+0000","last_deep_scrub":"1025'1523","last_deep_scrub_stamp":"2021-03-15T22:36:20.962682+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.962682+0000","log_size":1565,"ondisk_log_size":1565,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":58033973,"num_objects":25,"num_object_clones":22,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":347,"num_read_kb":290242,"num_write":364,"num_write_kb":133144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":229919087,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2fb","length":"1"},{"start":"30b","length":"1"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.6","version":"0'0","reported_seq":"136","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.367237+0000","last_change":"2021-03-15T22:35:56.358841+0000","last_active":"2021-03-15T22:36:30.367237+0000","last_peered":"2021-03-15T22:36:30.367237+0000","last_clean":"2021-03-15T22:36:30.367237+0000","last_became_active":"2021-03-15T22:35:56.358504+0000","last_became_peered":"2021-03-15T22:35:56.358504+0000","last_unstale":"2021-03-15T22:36:30.367237+0000","last_undegraded":"2021-03-15T22:36:30.367237+0000","last_fullsized":"2021-03-15T22:36:30.367237+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1001,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"84","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.376758+0000","last_change":"2021-03-15T22:35:19.502186+0000","last_active":"2021-03-15T22:36:32.376758+0000","last_peered":"2021-03-15T22:36:32.376758+0000","last_clean":"2021-03-15T22:36:32.376758+0000","last_became_active":"2021-03-15T22:35:19.501783+0000","last_became_peered":"2021-03-15T22:35:19.501783+0000","last_unstale":"2021-03-15T22:36:32.376758+0000","last_undegraded":"2021-03-15T22:36:32.376758+0000","last_fullsized":"2021-03-15T22:36:32.376758+0000","mapping_epoch":963,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":964,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.831369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.831369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"238","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.364787+0000","last_change":"2021-03-15T22:36:16.252660+0000","last_active":"2021-03-15T22:36:30.364787+0000","last_peered":"2021-03-15T22:36:30.364787+0000","last_clean":"2021-03-15T22:36:30.364787+0000","last_became_active":"2021-03-15T22:33:04.335914+0000","last_became_peered":"2021-03-15T22:33:04.335914+0000","last_unstale":"2021-03-15T22:36:30.364787+0000","last_undegraded":"2021-03-15T22:36:30.364787+0000","last_fullsized":"2021-03-15T22:36:30.364787+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:16.252579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.252579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1409","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.383245+0000","last_change":"2021-03-15T22:35:47.126971+0000","last_active":"2021-03-15T22:36:31.383245+0000","last_peered":"2021-03-15T22:36:31.383245+0000","last_clean":"2021-03-15T22:36:31.383245+0000","last_became_active":"2021-03-15T22:34:55.350419+0000","last_became_peered":"2021-03-15T22:34:55.350419+0000","last_unstale":"2021-03-15T22:36:31.383245+0000","last_undegraded":"2021-03-15T22:36:31.383245+0000","last_fullsized":"2021-03-15T22:36:31.383245+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:35:47.126904+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.126904+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"499","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381553+0000","last_change":"2021-03-15T22:33:15.492116+0000","last_active":"2021-03-15T22:36:29.381553+0000","last_peered":"2021-03-15T22:36:29.381553+0000","last_clean":"2021-03-15T22:36:29.381553+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:36:29.381553+0000","last_undegraded":"2021-03-15T22:36:29.381553+0000","last_fullsized":"2021-03-15T22:36:29.381553+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.492014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.492014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1069","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.377086+0000","last_change":"2021-03-15T22:36:24.388928+0000","last_active":"2021-03-15T22:36:32.377086+0000","last_peered":"2021-03-15T22:36:32.377086+0000","last_clean":"2021-03-15T22:36:32.377086+0000","last_became_active":"2021-03-15T22:36:24.388564+0000","last_became_peered":"2021-03-15T22:36:24.388564+0000","last_unstale":"2021-03-15T22:36:32.377086+0000","last_undegraded":"2021-03-15T22:36:32.377086+0000","last_fullsized":"2021-03-15T22:36:32.377086+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1035'1435","reported_seq":"4247","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.389518+0000","last_change":"2021-03-15T22:36:30.389719+0000","last_active":"2021-03-15T22:36:30.389518+0000","last_peered":"2021-03-15T22:36:30.389518+0000","last_clean":"2021-03-15T22:36:30.389518+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:36:30.389518+0000","last_undegraded":"2021-03-15T22:36:30.389518+0000","last_fullsized":"2021-03-15T22:36:30.389518+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'1385","last_scrub_stamp":"2021-03-15T22:36:15.114641+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:36:15.114641+0000","log_size":1435,"ondisk_log_size":1435,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":51972454,"num_objects":21,"num_object_clones":18,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":420,"num_read_kb":425400,"num_write":316,"num_write_kb":120781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2fb","length":"1"},{"start":"30b","length":"1"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.5","version":"0'0","reported_seq":"1087","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.378406+0000","last_change":"2021-03-15T22:36:00.364664+0000","last_active":"2021-03-15T22:36:32.378406+0000","last_peered":"2021-03-15T22:36:32.378406+0000","last_clean":"2021-03-15T22:36:32.378406+0000","last_became_active":"2021-03-15T22:36:00.364246+0000","last_became_peered":"2021-03-15T22:36:00.364246+0000","last_unstale":"2021-03-15T22:36:32.378406+0000","last_undegraded":"2021-03-15T22:36:32.378406+0000","last_fullsized":"2021-03-15T22:36:32.378406+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1005,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:44.582515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:44.582515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"1073","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.383390+0000","last_change":"2021-03-15T22:36:13.313358+0000","last_active":"2021-03-15T22:36:31.383390+0000","last_peered":"2021-03-15T22:36:31.383390+0000","last_clean":"2021-03-15T22:36:31.383390+0000","last_became_active":"2021-03-15T22:34:55.347065+0000","last_became_peered":"2021-03-15T22:34:55.347065+0000","last_unstale":"2021-03-15T22:36:31.383390+0000","last_undegraded":"2021-03-15T22:36:31.383390+0000","last_fullsized":"2021-03-15T22:36:31.383390+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:13.313278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:36:13.313278+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"113","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.374125+0000","last_change":"2021-03-15T22:35:17.301942+0000","last_active":"2021-03-15T22:36:32.374125+0000","last_peered":"2021-03-15T22:36:32.374125+0000","last_clean":"2021-03-15T22:36:32.374125+0000","last_became_active":"2021-03-15T22:34:55.355485+0000","last_became_peered":"2021-03-15T22:34:55.355485+0000","last_unstale":"2021-03-15T22:36:32.374125+0000","last_undegraded":"2021-03-15T22:36:32.374125+0000","last_fullsized":"2021-03-15T22:36:32.374125+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.301892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.301892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"526","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.384756+0000","last_change":"2021-03-15T22:36:20.050533+0000","last_active":"2021-03-15T22:36:32.384756+0000","last_peered":"2021-03-15T22:36:32.384756+0000","last_clean":"2021-03-15T22:36:32.384756+0000","last_became_active":"2021-03-15T22:34:10.322666+0000","last_became_peered":"2021-03-15T22:34:10.322666+0000","last_unstale":"2021-03-15T22:36:32.384756+0000","last_undegraded":"2021-03-15T22:36:32.384756+0000","last_fullsized":"2021-03-15T22:36:32.384756+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:36:20.050450+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.050450+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"113","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.376204+0000","last_change":"2021-03-15T22:35:45.355712+0000","last_active":"2021-03-15T22:36:32.376204+0000","last_peered":"2021-03-15T22:36:32.376204+0000","last_clean":"2021-03-15T22:36:32.376204+0000","last_became_active":"2021-03-15T22:34:55.353316+0000","last_became_peered":"2021-03-15T22:34:55.353316+0000","last_unstale":"2021-03-15T22:36:32.376204+0000","last_undegraded":"2021-03-15T22:36:32.376204+0000","last_fullsized":"2021-03-15T22:36:32.376204+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.355647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.355647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"1035'3428","reported_seq":"7080","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.404536+0000","last_change":"2021-03-15T22:36:30.404814+0000","last_active":"2021-03-15T22:36:30.404536+0000","last_peered":"2021-03-15T22:36:30.404536+0000","last_clean":"2021-03-15T22:36:30.404536+0000","last_became_active":"2021-03-15T22:34:13.331560+0000","last_became_peered":"2021-03-15T22:34:13.331560+0000","last_unstale":"2021-03-15T22:36:30.404536+0000","last_undegraded":"2021-03-15T22:36:30.404536+0000","last_fullsized":"2021-03-15T22:36:30.404536+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'3208","last_scrub_stamp":"2021-03-15T22:35:42.271839+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.271839+0000","log_size":3428,"ondisk_log_size":3428,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":100464208,"num_objects":45,"num_object_clones":38,"num_object_copies":135,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":3,"num_read":938,"num_read_kb":984359,"num_write":692,"num_write_kb":282709,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":264088494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2fb","length":"1"},{"start":"30b","length":"1"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.7","version":"0'0","reported_seq":"440","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.384856+0000","last_change":"2021-03-15T22:35:52.857478+0000","last_active":"2021-03-15T22:36:32.384856+0000","last_peered":"2021-03-15T22:36:32.384856+0000","last_clean":"2021-03-15T22:36:32.384856+0000","last_became_active":"2021-03-15T22:35:52.857192+0000","last_became_peered":"2021-03-15T22:35:52.857192+0000","last_unstale":"2021-03-15T22:36:32.384856+0000","last_undegraded":"2021-03-15T22:36:32.384856+0000","last_fullsized":"2021-03-15T22:36:32.384856+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":997,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.761137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"1061","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.376641+0000","last_change":"2021-03-15T22:36:22.385705+0000","last_active":"2021-03-15T22:36:32.376641+0000","last_peered":"2021-03-15T22:36:32.376641+0000","last_clean":"2021-03-15T22:36:32.376641+0000","last_became_active":"2021-03-15T22:36:22.385402+0000","last_became_peered":"2021-03-15T22:36:22.385402+0000","last_unstale":"2021-03-15T22:36:32.376641+0000","last_undegraded":"2021-03-15T22:36:32.376641+0000","last_fullsized":"2021-03-15T22:36:32.376641+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":1027,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"118","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.364837+0000","last_change":"2021-03-15T22:34:43.341562+0000","last_active":"2021-03-15T22:36:30.364837+0000","last_peered":"2021-03-15T22:36:30.364837+0000","last_clean":"2021-03-15T22:36:30.364837+0000","last_became_active":"2021-03-15T22:34:43.340955+0000","last_became_peered":"2021-03-15T22:34:43.340955+0000","last_unstale":"2021-03-15T22:36:30.364837+0000","last_undegraded":"2021-03-15T22:36:30.364837+0000","last_fullsized":"2021-03-15T22:36:30.364837+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.843478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.843478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"239","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.763204+0000","last_change":"2021-03-15T22:36:12.698401+0000","last_active":"2021-03-15T22:36:29.763204+0000","last_peered":"2021-03-15T22:36:29.763204+0000","last_clean":"2021-03-15T22:36:29.763204+0000","last_became_active":"2021-03-15T22:33:04.330531+0000","last_became_peered":"2021-03-15T22:33:04.330531+0000","last_unstale":"2021-03-15T22:36:29.763204+0000","last_undegraded":"2021-03-15T22:36:29.763204+0000","last_fullsized":"2021-03-15T22:36:29.763204+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:36:12.698329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:12.698329+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"1073","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.377937+0000","last_change":"2021-03-15T22:35:22.409047+0000","last_active":"2021-03-15T22:36:32.377937+0000","last_peered":"2021-03-15T22:36:32.377937+0000","last_clean":"2021-03-15T22:36:32.377937+0000","last_became_active":"2021-03-15T22:34:55.352186+0000","last_became_peered":"2021-03-15T22:34:55.352186+0000","last_unstale":"2021-03-15T22:36:32.377937+0000","last_undegraded":"2021-03-15T22:36:32.377937+0000","last_fullsized":"2021-03-15T22:36:32.377937+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.408965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.408965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"18","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.384474+0000","last_change":"2021-03-15T22:36:24.391414+0000","last_active":"2021-03-15T22:36:31.384474+0000","last_peered":"2021-03-15T22:36:31.384474+0000","last_clean":"2021-03-15T22:36:31.384474+0000","last_became_active":"2021-03-15T22:36:24.390984+0000","last_became_peered":"2021-03-15T22:36:24.390984+0000","last_unstale":"2021-03-15T22:36:31.384474+0000","last_undegraded":"2021-03-15T22:36:31.384474+0000","last_fullsized":"2021-03-15T22:36:31.384474+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1037'1052","reported_seq":"4293","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.384392+0000","last_change":"2021-03-15T22:36:32.384392+0000","last_active":"2021-03-15T22:36:32.384392+0000","last_peered":"2021-03-15T22:36:32.384392+0000","last_clean":"2021-03-15T22:36:32.384392+0000","last_became_active":"2021-03-15T22:35:03.346751+0000","last_became_peered":"2021-03-15T22:35:03.346751+0000","last_unstale":"2021-03-15T22:36:32.384392+0000","last_undegraded":"2021-03-15T22:36:32.384392+0000","last_fullsized":"2021-03-15T22:36:32.384392+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":0,"last_scrub":"1020'1020","last_scrub_stamp":"2021-03-15T22:36:16.818677+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.818677+0000","log_size":1052,"ondisk_log_size":1052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37060250,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":267,"num_read_kb":273197,"num_write":228,"num_write_kb":99079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":109648904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"2e8","length":"1"},{"start":"2fb","length":"1"},{"start":"30a","length":"2"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"857","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.382947+0000","last_change":"2021-03-15T22:36:04.904466+0000","last_active":"2021-03-15T22:36:32.382947+0000","last_peered":"2021-03-15T22:36:32.382947+0000","last_clean":"2021-03-15T22:36:32.382947+0000","last_became_active":"2021-03-15T22:36:04.904112+0000","last_became_peered":"2021-03-15T22:36:04.904112+0000","last_unstale":"2021-03-15T22:36:32.382947+0000","last_undegraded":"2021-03-15T22:36:32.382947+0000","last_fullsized":"2021-03-15T22:36:32.382947+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"531","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381475+0000","last_change":"2021-03-15T22:34:35.410541+0000","last_active":"2021-03-15T22:36:29.381475+0000","last_peered":"2021-03-15T22:36:29.381475+0000","last_clean":"2021-03-15T22:36:29.381475+0000","last_became_active":"2021-03-15T22:34:10.321100+0000","last_became_peered":"2021-03-15T22:34:10.321100+0000","last_unstale":"2021-03-15T22:36:29.381475+0000","last_undegraded":"2021-03-15T22:36:29.381475+0000","last_fullsized":"2021-03-15T22:36:29.381475+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:35.410485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:35.410485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"2151","reported_epoch":"1038","state":"active+clean","last_fresh":"2021-03-15T22:36:33.386639+0000","last_change":"2021-03-15T22:35:10.452906+0000","last_active":"2021-03-15T22:36:33.386639+0000","last_peered":"2021-03-15T22:36:33.386639+0000","last_clean":"2021-03-15T22:36:33.386639+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:36:33.386639+0000","last_undegraded":"2021-03-15T22:36:33.386639+0000","last_fullsized":"2021-03-15T22:36:33.386639+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:35:10.452809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.452809+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1104,"num_read_kb":1099,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"629","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.382002+0000","last_change":"2021-03-15T22:34:55.414460+0000","last_active":"2021-03-15T22:36:29.382002+0000","last_peered":"2021-03-15T22:36:29.382002+0000","last_clean":"2021-03-15T22:36:29.382002+0000","last_became_active":"2021-03-15T22:34:55.359023+0000","last_became_peered":"2021-03-15T22:34:55.359023+0000","last_unstale":"2021-03-15T22:36:29.382002+0000","last_undegraded":"2021-03-15T22:36:29.382002+0000","last_fullsized":"2021-03-15T22:36:29.382002+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:34:40.472800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.472800+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"427","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.384926+0000","last_change":"2021-03-15T22:34:40.799217+0000","last_active":"2021-03-15T22:36:32.384926+0000","last_peered":"2021-03-15T22:36:32.384926+0000","last_clean":"2021-03-15T22:36:32.384926+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:36:32.384926+0000","last_undegraded":"2021-03-15T22:36:32.384926+0000","last_fullsized":"2021-03-15T22:36:32.384926+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.799153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.799153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1058","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.762863+0000","last_change":"2021-03-15T22:36:24.387855+0000","last_active":"2021-03-15T22:36:29.762863+0000","last_peered":"2021-03-15T22:36:29.762863+0000","last_clean":"2021-03-15T22:36:29.762863+0000","last_became_active":"2021-03-15T22:36:24.387249+0000","last_became_peered":"2021-03-15T22:36:24.387249+0000","last_unstale":"2021-03-15T22:36:29.762863+0000","last_undegraded":"2021-03-15T22:36:29.762863+0000","last_fullsized":"2021-03-15T22:36:29.762863+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"1037'3558","reported_seq":"7772","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.406223+0000","last_change":"2021-03-15T22:36:32.406223+0000","last_active":"2021-03-15T22:36:32.406223+0000","last_peered":"2021-03-15T22:36:32.406223+0000","last_clean":"2021-03-15T22:36:32.406223+0000","last_became_active":"2021-03-15T22:35:17.487261+0000","last_became_peered":"2021-03-15T22:35:17.487261+0000","last_unstale":"2021-03-15T22:36:32.406223+0000","last_undegraded":"2021-03-15T22:36:32.406223+0000","last_fullsized":"2021-03-15T22:36:32.406223+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":0,"last_scrub":"1020'3444","last_scrub_stamp":"2021-03-15T22:36:16.067782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.067782+0000","log_size":3558,"ondisk_log_size":3558,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":115989940,"num_objects":51,"num_object_clones":44,"num_object_copies":153,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":51,"num_whiteouts":3,"num_read":1290,"num_read_kb":1384963,"num_write":798,"num_write_kb":334525,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":721,"num_bytes_recovered":351021959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"2e8","length":"1"},{"start":"2fb","length":"1"},{"start":"30a","length":"2"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"1091","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.381710+0000","last_change":"2021-03-15T22:36:16.372506+0000","last_active":"2021-03-15T22:36:31.381710+0000","last_peered":"2021-03-15T22:36:31.381710+0000","last_clean":"2021-03-15T22:36:31.381710+0000","last_became_active":"2021-03-15T22:36:16.372177+0000","last_became_peered":"2021-03-15T22:36:16.372177+0000","last_unstale":"2021-03-15T22:36:31.381710+0000","last_undegraded":"2021-03-15T22:36:31.381710+0000","last_fullsized":"2021-03-15T22:36:31.381710+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1021,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.070798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.070798+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"1091","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.383103+0000","last_change":"2021-03-15T22:36:24.397895+0000","last_active":"2021-03-15T22:36:31.383103+0000","last_peered":"2021-03-15T22:36:31.383103+0000","last_clean":"2021-03-15T22:36:31.383103+0000","last_became_active":"2021-03-15T22:34:55.346048+0000","last_became_peered":"2021-03-15T22:34:55.346048+0000","last_unstale":"2021-03-15T22:36:31.383103+0000","last_undegraded":"2021-03-15T22:36:31.383103+0000","last_fullsized":"2021-03-15T22:36:31.383103+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:24.397819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_clean_scrub_stamp":"2021-03-15T22:36:24.397819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"579","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381904+0000","last_change":"2021-03-15T22:35:21.314851+0000","last_active":"2021-03-15T22:36:29.381904+0000","last_peered":"2021-03-15T22:36:29.381904+0000","last_clean":"2021-03-15T22:36:29.381904+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:36:29.381904+0000","last_undegraded":"2021-03-15T22:36:29.381904+0000","last_fullsized":"2021-03-15T22:36:29.381904+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:35:21.314763+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.314763+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"219","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.762766+0000","last_change":"2021-03-15T22:35:39.707993+0000","last_active":"2021-03-15T22:36:29.762766+0000","last_peered":"2021-03-15T22:36:29.762766+0000","last_clean":"2021-03-15T22:36:29.762766+0000","last_became_active":"2021-03-15T22:33:04.337244+0000","last_became_peered":"2021-03-15T22:33:04.337244+0000","last_unstale":"2021-03-15T22:36:29.762766+0000","last_undegraded":"2021-03-15T22:36:29.762766+0000","last_fullsized":"2021-03-15T22:36:29.762766+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:39.707900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:35:39.707900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1058","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.763090+0000","last_change":"2021-03-15T22:36:24.393232+0000","last_active":"2021-03-15T22:36:29.763090+0000","last_peered":"2021-03-15T22:36:29.763090+0000","last_clean":"2021-03-15T22:36:29.763090+0000","last_became_active":"2021-03-15T22:36:24.392909+0000","last_became_peered":"2021-03-15T22:36:24.392909+0000","last_unstale":"2021-03-15T22:36:29.763090+0000","last_undegraded":"2021-03-15T22:36:29.763090+0000","last_fullsized":"2021-03-15T22:36:29.763090+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"1036'620","reported_seq":"3866","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.386169+0000","last_change":"2021-03-15T22:36:31.386471+0000","last_active":"2021-03-15T22:36:31.386169+0000","last_peered":"2021-03-15T22:36:31.386169+0000","last_clean":"2021-03-15T22:36:31.386169+0000","last_became_active":"2021-03-15T22:34:15.498612+0000","last_became_peered":"2021-03-15T22:34:15.498612+0000","last_unstale":"2021-03-15T22:36:31.386169+0000","last_undegraded":"2021-03-15T22:36:31.386169+0000","last_fullsized":"2021-03-15T22:36:31.386169+0000","mapping_epoch":899,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":900,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'463","last_scrub_stamp":"2021-03-15T22:33:40.563144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.563144+0000","log_size":620,"ondisk_log_size":620,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23933460,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":185,"num_read_kb":221865,"num_write":115,"num_write_kb":59307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":84391207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"2e8","length":"1"},{"start":"2fb","length":"1"},{"start":"30b","length":"1"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.0","version":"0'0","reported_seq":"573","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381257+0000","last_change":"2021-03-15T22:36:20.369600+0000","last_active":"2021-03-15T22:36:29.381257+0000","last_peered":"2021-03-15T22:36:29.381257+0000","last_clean":"2021-03-15T22:36:29.381257+0000","last_became_active":"2021-03-15T22:36:20.369276+0000","last_became_peered":"2021-03-15T22:36:20.369276+0000","last_unstale":"2021-03-15T22:36:29.381257+0000","last_undegraded":"2021-03-15T22:36:29.381257+0000","last_fullsized":"2021-03-15T22:36:29.381257+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1025,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"1015","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.384431+0000","last_change":"2021-03-15T22:36:14.037839+0000","last_active":"2021-03-15T22:36:32.384431+0000","last_peered":"2021-03-15T22:36:32.384431+0000","last_clean":"2021-03-15T22:36:32.384431+0000","last_became_active":"2021-03-15T22:34:55.355757+0000","last_became_peered":"2021-03-15T22:34:55.355757+0000","last_unstale":"2021-03-15T22:36:32.384431+0000","last_undegraded":"2021-03-15T22:36:32.384431+0000","last_fullsized":"2021-03-15T22:36:32.384431+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:14.037757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.037757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"996","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.383205+0000","last_change":"2021-03-15T22:36:13.027720+0000","last_active":"2021-03-15T22:36:32.383205+0000","last_peered":"2021-03-15T22:36:32.383205+0000","last_clean":"2021-03-15T22:36:32.383205+0000","last_became_active":"2021-03-15T22:34:10.321886+0000","last_became_peered":"2021-03-15T22:34:10.321886+0000","last_unstale":"2021-03-15T22:36:32.383205+0000","last_undegraded":"2021-03-15T22:36:32.383205+0000","last_fullsized":"2021-03-15T22:36:32.383205+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:36:13.027640+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:36:13.027640+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"836'11","reported_seq":"663","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.381778+0000","last_change":"2021-03-15T22:35:44.256542+0000","last_active":"2021-03-15T22:36:29.381778+0000","last_peered":"2021-03-15T22:36:29.381778+0000","last_clean":"2021-03-15T22:36:29.381778+0000","last_became_active":"2021-03-15T22:33:04.336294+0000","last_became_peered":"2021-03-15T22:33:04.336294+0000","last_unstale":"2021-03-15T22:36:29.381778+0000","last_undegraded":"2021-03-15T22:36:29.381778+0000","last_fullsized":"2021-03-15T22:36:29.381778+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"836'11","last_scrub_stamp":"2021-03-15T22:35:44.256486+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:35:44.256486+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"17","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.364812+0000","last_change":"2021-03-15T22:36:24.387641+0000","last_active":"2021-03-15T22:36:30.364812+0000","last_peered":"2021-03-15T22:36:30.364812+0000","last_clean":"2021-03-15T22:36:30.364812+0000","last_became_active":"2021-03-15T22:36:24.387299+0000","last_became_peered":"2021-03-15T22:36:24.387299+0000","last_unstale":"2021-03-15T22:36:30.364812+0000","last_undegraded":"2021-03-15T22:36:30.364812+0000","last_fullsized":"2021-03-15T22:36:30.364812+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"1036'999","reported_seq":"4273","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.392016+0000","last_change":"2021-03-15T22:36:31.392483+0000","last_active":"2021-03-15T22:36:31.392016+0000","last_peered":"2021-03-15T22:36:31.392016+0000","last_clean":"2021-03-15T22:36:31.392016+0000","last_became_active":"2021-03-15T22:34:58.350873+0000","last_became_peered":"2021-03-15T22:34:58.350873+0000","last_unstale":"2021-03-15T22:36:31.392016+0000","last_undegraded":"2021-03-15T22:36:31.392016+0000","last_fullsized":"2021-03-15T22:36:31.392016+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'901","last_scrub_stamp":"2021-03-15T22:35:16.957336+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.957336+0000","log_size":999,"ondisk_log_size":999,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37536135,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":267,"num_read_kb":230831,"num_write":236,"num_write_kb":92610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":114552572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"2e8","length":"1"},{"start":"2fb","length":"1"},{"start":"30b","length":"1"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.3","version":"0'0","reported_seq":"1078","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.377731+0000","last_change":"2021-03-15T22:36:08.382982+0000","last_active":"2021-03-15T22:36:32.377731+0000","last_peered":"2021-03-15T22:36:32.377731+0000","last_clean":"2021-03-15T22:36:32.377731+0000","last_became_active":"2021-03-15T22:36:08.382713+0000","last_became_peered":"2021-03-15T22:36:08.382713+0000","last_unstale":"2021-03-15T22:36:32.377731+0000","last_undegraded":"2021-03-15T22:36:32.377731+0000","last_fullsized":"2021-03-15T22:36:32.377731+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.507204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.507204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"499","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.382009+0000","last_change":"2021-03-15T22:33:22.598240+0000","last_active":"2021-03-15T22:36:29.382009+0000","last_peered":"2021-03-15T22:36:29.382009+0000","last_clean":"2021-03-15T22:36:29.382009+0000","last_became_active":"2021-03-15T22:28:12.526143+0000","last_became_peered":"2021-03-15T22:28:12.526143+0000","last_unstale":"2021-03-15T22:36:29.382009+0000","last_undegraded":"2021-03-15T22:36:29.382009+0000","last_fullsized":"2021-03-15T22:36:29.382009+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.598174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.598174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"1228","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.369073+0000","last_change":"2021-03-15T22:33:43.514160+0000","last_active":"2021-03-15T22:36:30.369073+0000","last_peered":"2021-03-15T22:36:30.369073+0000","last_clean":"2021-03-15T22:36:30.369073+0000","last_became_active":"2021-03-15T22:33:04.336145+0000","last_became_peered":"2021-03-15T22:33:04.336145+0000","last_unstale":"2021-03-15T22:36:30.369073+0000","last_undegraded":"2021-03-15T22:36:30.369073+0000","last_fullsized":"2021-03-15T22:36:30.369073+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:33:43.514088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.514088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"558","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365251+0000","last_change":"2021-03-15T22:33:20.685041+0000","last_active":"2021-03-15T22:36:30.365251+0000","last_peered":"2021-03-15T22:36:30.365251+0000","last_clean":"2021-03-15T22:36:30.365251+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:36:30.365251+0000","last_undegraded":"2021-03-15T22:36:30.365251+0000","last_fullsized":"2021-03-15T22:36:30.365251+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:33:20.684976+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.684976+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"17","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365933+0000","last_change":"2021-03-15T22:36:24.391783+0000","last_active":"2021-03-15T22:36:30.365933+0000","last_peered":"2021-03-15T22:36:30.365933+0000","last_clean":"2021-03-15T22:36:30.365933+0000","last_became_active":"2021-03-15T22:36:24.391417+0000","last_became_peered":"2021-03-15T22:36:24.391417+0000","last_unstale":"2021-03-15T22:36:30.365933+0000","last_undegraded":"2021-03-15T22:36:30.365933+0000","last_fullsized":"2021-03-15T22:36:30.365933+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1037'1734","reported_seq":"4839","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.398721+0000","last_change":"2021-03-15T22:36:32.399153+0000","last_active":"2021-03-15T22:36:32.398721+0000","last_peered":"2021-03-15T22:36:32.398721+0000","last_clean":"2021-03-15T22:36:32.398721+0000","last_became_active":"2021-03-15T22:34:58.337099+0000","last_became_peered":"2021-03-15T22:34:58.337099+0000","last_unstale":"2021-03-15T22:36:32.398721+0000","last_undegraded":"2021-03-15T22:36:32.398721+0000","last_fullsized":"2021-03-15T22:36:32.398721+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'1611","last_scrub_stamp":"2021-03-15T22:35:21.054192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.054192+0000","log_size":1734,"ondisk_log_size":1734,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":42621465,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":559,"num_read_kb":593462,"num_write":352,"num_write_kb":121377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":226741522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2e8","length":"1"},{"start":"2fb","length":"1"},{"start":"30a","length":"2"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.2","version":"0'0","reported_seq":"1131","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.365238+0000","last_change":"2021-03-15T22:36:12.386049+0000","last_active":"2021-03-15T22:36:30.365238+0000","last_peered":"2021-03-15T22:36:30.365238+0000","last_clean":"2021-03-15T22:36:30.365238+0000","last_became_active":"2021-03-15T22:36:12.385681+0000","last_became_peered":"2021-03-15T22:36:12.385681+0000","last_unstale":"2021-03-15T22:36:30.365238+0000","last_undegraded":"2021-03-15T22:36:30.365238+0000","last_fullsized":"2021-03-15T22:36:30.365238+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1017,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"837","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.398303+0000","last_change":"2021-03-15T22:36:30.397474+0000","last_active":"2021-03-15T22:36:30.398303+0000","last_peered":"2021-03-15T22:36:30.398303+0000","last_clean":"2021-03-15T22:36:30.398303+0000","last_became_active":"2021-03-15T22:36:30.390223+0000","last_became_peered":"2021-03-15T22:36:30.390223+0000","last_unstale":"2021-03-15T22:36:30.398303+0000","last_undegraded":"2021-03-15T22:36:30.398303+0000","last_fullsized":"2021-03-15T22:36:30.398303+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1035,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.870368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.870368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"472","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.376880+0000","last_change":"2021-03-15T22:35:08.402702+0000","last_active":"2021-03-15T22:36:32.376880+0000","last_peered":"2021-03-15T22:36:32.376880+0000","last_clean":"2021-03-15T22:36:32.376880+0000","last_became_active":"2021-03-15T22:34:10.349731+0000","last_became_peered":"2021-03-15T22:34:10.349731+0000","last_unstale":"2021-03-15T22:36:32.376880+0000","last_undegraded":"2021-03-15T22:36:32.376880+0000","last_fullsized":"2021-03-15T22:36:32.376880+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:35:08.402641+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:35:08.402641+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1081","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.368532+0000","last_change":"2021-03-15T22:34:43.343527+0000","last_active":"2021-03-15T22:36:30.368532+0000","last_peered":"2021-03-15T22:36:30.368532+0000","last_clean":"2021-03-15T22:36:30.368532+0000","last_became_active":"2021-03-15T22:34:43.343255+0000","last_became_peered":"2021-03-15T22:34:43.343255+0000","last_unstale":"2021-03-15T22:36:30.368532+0000","last_undegraded":"2021-03-15T22:36:30.368532+0000","last_fullsized":"2021-03-15T22:36:30.368532+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.673338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.673338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"19","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.383318+0000","last_change":"2021-03-15T22:36:24.389841+0000","last_active":"2021-03-15T22:36:32.383318+0000","last_peered":"2021-03-15T22:36:32.383318+0000","last_clean":"2021-03-15T22:36:32.383318+0000","last_became_active":"2021-03-15T22:36:24.389490+0000","last_became_peered":"2021-03-15T22:36:24.389490+0000","last_unstale":"2021-03-15T22:36:32.383318+0000","last_undegraded":"2021-03-15T22:36:32.383318+0000","last_fullsized":"2021-03-15T22:36:32.383318+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1035'1507","reported_seq":"4859","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.395643+0000","last_change":"2021-03-15T22:36:30.395950+0000","last_active":"2021-03-15T22:36:30.395643+0000","last_peered":"2021-03-15T22:36:30.395643+0000","last_clean":"2021-03-15T22:36:30.395643+0000","last_became_active":"2021-03-15T22:34:58.344406+0000","last_became_peered":"2021-03-15T22:34:58.344406+0000","last_unstale":"2021-03-15T22:36:30.395643+0000","last_undegraded":"2021-03-15T22:36:30.395643+0000","last_fullsized":"2021-03-15T22:36:30.395643+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'1407","last_scrub_stamp":"2021-03-15T22:35:43.758914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:43.758914+0000","log_size":1507,"ondisk_log_size":1507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":46916134,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":419,"num_read_kb":382178,"num_write":317,"num_write_kb":119059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":127958264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2fb","length":"1"},{"start":"30b","length":"1"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.d","version":"0'0","reported_seq":"982","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.384346+0000","last_change":"2021-03-15T22:34:55.356480+0000","last_active":"2021-03-15T22:36:32.384346+0000","last_peered":"2021-03-15T22:36:32.384346+0000","last_clean":"2021-03-15T22:36:32.384346+0000","last_became_active":"2021-03-15T22:34:55.356213+0000","last_became_peered":"2021-03-15T22:34:55.356213+0000","last_unstale":"2021-03-15T22:36:32.384346+0000","last_undegraded":"2021-03-15T22:36:32.384346+0000","last_fullsized":"2021-03-15T22:36:32.384346+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.729838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.729838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1343","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.366902+0000","last_change":"2021-03-15T22:35:19.468482+0000","last_active":"2021-03-15T22:36:30.366902+0000","last_peered":"2021-03-15T22:36:30.366902+0000","last_clean":"2021-03-15T22:36:30.366902+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:36:30.366902+0000","last_undegraded":"2021-03-15T22:36:30.366902+0000","last_fullsized":"2021-03-15T22:36:30.366902+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:35:19.468402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.468402+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"466","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.383528+0000","last_change":"2021-03-15T22:36:26.941424+0000","last_active":"2021-03-15T22:36:31.383528+0000","last_peered":"2021-03-15T22:36:31.383528+0000","last_clean":"2021-03-15T22:36:31.383528+0000","last_became_active":"2021-03-15T22:36:26.941149+0000","last_became_peered":"2021-03-15T22:36:26.941149+0000","last_unstale":"2021-03-15T22:36:31.383528+0000","last_undegraded":"2021-03-15T22:36:31.383528+0000","last_fullsized":"2021-03-15T22:36:31.383528+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1031,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.165838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.165838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"203","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.381675+0000","last_change":"2021-03-15T22:36:21.293286+0000","last_active":"2021-03-15T22:36:31.381675+0000","last_peered":"2021-03-15T22:36:31.381675+0000","last_clean":"2021-03-15T22:36:31.381675+0000","last_became_active":"2021-03-15T22:33:45.317888+0000","last_became_peered":"2021-03-15T22:33:45.317888+0000","last_unstale":"2021-03-15T22:36:31.381675+0000","last_undegraded":"2021-03-15T22:36:31.381675+0000","last_fullsized":"2021-03-15T22:36:31.381675+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:36:21.293219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:21.293219+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"1035'526","reported_seq":"3477","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.382850+0000","last_change":"2021-03-15T22:36:30.383160+0000","last_active":"2021-03-15T22:36:30.382850+0000","last_peered":"2021-03-15T22:36:30.382850+0000","last_clean":"2021-03-15T22:36:30.382850+0000","last_became_active":"2021-03-15T22:33:08.354872+0000","last_became_peered":"2021-03-15T22:33:08.354872+0000","last_unstale":"2021-03-15T22:36:30.382850+0000","last_undegraded":"2021-03-15T22:36:30.382850+0000","last_fullsized":"2021-03-15T22:36:30.382850+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'470","last_scrub_stamp":"2021-03-15T22:35:42.721100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.721100+0000","log_size":526,"ondisk_log_size":526,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20081853,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":190,"num_read_kb":197819,"num_write":106,"num_write_kb":42316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":31290769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[{"start":"2fb","length":"1"},{"start":"30b","length":"1"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.c","version":"0'0","reported_seq":"118","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.366719+0000","last_change":"2021-03-15T22:34:43.340972+0000","last_active":"2021-03-15T22:36:30.366719+0000","last_peered":"2021-03-15T22:36:30.366719+0000","last_clean":"2021-03-15T22:36:30.366719+0000","last_became_active":"2021-03-15T22:34:43.340600+0000","last_became_peered":"2021-03-15T22:34:43.340600+0000","last_unstale":"2021-03-15T22:36:30.366719+0000","last_undegraded":"2021-03-15T22:36:30.366719+0000","last_fullsized":"2021-03-15T22:36:30.366719+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:09.612126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.612126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"1080","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.385110+0000","last_change":"2021-03-15T22:35:21.917891+0000","last_active":"2021-03-15T22:36:32.385110+0000","last_peered":"2021-03-15T22:36:32.385110+0000","last_clean":"2021-03-15T22:36:32.385110+0000","last_became_active":"2021-03-15T22:33:08.352367+0000","last_became_peered":"2021-03-15T22:33:08.352367+0000","last_unstale":"2021-03-15T22:36:32.385110+0000","last_undegraded":"2021-03-15T22:36:32.385110+0000","last_fullsized":"2021-03-15T22:36:32.385110+0000","mapping_epoch":837,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'9","last_scrub_stamp":"2021-03-15T22:35:21.917819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.917819+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"1068","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.368916+0000","last_change":"2021-03-15T22:36:24.286085+0000","last_active":"2021-03-15T22:36:30.368916+0000","last_peered":"2021-03-15T22:36:30.368916+0000","last_clean":"2021-03-15T22:36:30.368916+0000","last_became_active":"2021-03-15T22:36:22.389688+0000","last_became_peered":"2021-03-15T22:36:22.389688+0000","last_unstale":"2021-03-15T22:36:30.368916+0000","last_undegraded":"2021-03-15T22:36:30.368916+0000","last_fullsized":"2021-03-15T22:36:30.368916+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1027,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:24.285991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_clean_scrub_stamp":"2021-03-15T22:36:24.285991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"1207","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.366648+0000","last_change":"2021-03-15T22:35:21.384129+0000","last_active":"2021-03-15T22:36:30.366648+0000","last_peered":"2021-03-15T22:36:30.366648+0000","last_clean":"2021-03-15T22:36:30.366648+0000","last_became_active":"2021-03-15T22:34:10.330608+0000","last_became_peered":"2021-03-15T22:34:10.330608+0000","last_unstale":"2021-03-15T22:36:30.366648+0000","last_undegraded":"2021-03-15T22:36:30.366648+0000","last_fullsized":"2021-03-15T22:36:30.366648+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:35:21.384037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.384037+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"19","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.384609+0000","last_change":"2021-03-15T22:36:24.389116+0000","last_active":"2021-03-15T22:36:32.384609+0000","last_peered":"2021-03-15T22:36:32.384609+0000","last_clean":"2021-03-15T22:36:32.384609+0000","last_became_active":"2021-03-15T22:36:24.388690+0000","last_became_peered":"2021-03-15T22:36:24.388690+0000","last_unstale":"2021-03-15T22:36:32.384609+0000","last_undegraded":"2021-03-15T22:36:32.384609+0000","last_fullsized":"2021-03-15T22:36:32.384609+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"1036'508","reported_seq":"2977","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.377013+0000","last_change":"2021-03-15T22:36:32.377013+0000","last_active":"2021-03-15T22:36:32.377013+0000","last_peered":"2021-03-15T22:36:32.377013+0000","last_clean":"2021-03-15T22:36:32.377013+0000","last_became_active":"2021-03-15T22:34:58.350923+0000","last_became_peered":"2021-03-15T22:34:58.350923+0000","last_unstale":"2021-03-15T22:36:32.377013+0000","last_undegraded":"2021-03-15T22:36:32.377013+0000","last_fullsized":"2021-03-15T22:36:32.377013+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"1027'503","last_scrub_stamp":"2021-03-15T22:36:24.720629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:24.720629+0000","log_size":508,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8377674,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":158,"num_read_kb":164364,"num_write":122,"num_write_kb":40554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":52707265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2e8","length":"1"},{"start":"2fb","length":"1"},{"start":"30a","length":"2"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"785","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.368121+0000","last_change":"2021-03-15T22:36:12.855165+0000","last_active":"2021-03-15T22:36:30.368121+0000","last_peered":"2021-03-15T22:36:30.368121+0000","last_clean":"2021-03-15T22:36:30.368121+0000","last_became_active":"2021-03-15T22:34:55.345267+0000","last_became_peered":"2021-03-15T22:34:55.345267+0000","last_unstale":"2021-03-15T22:36:30.368121+0000","last_undegraded":"2021-03-15T22:36:30.368121+0000","last_fullsized":"2021-03-15T22:36:30.368121+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:12.855079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:36:12.855079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"1073","reported_epoch":"1036","state":"active+clean","last_fresh":"2021-03-15T22:36:31.383327+0000","last_change":"2021-03-15T22:35:12.091714+0000","last_active":"2021-03-15T22:36:31.383327+0000","last_peered":"2021-03-15T22:36:31.383327+0000","last_clean":"2021-03-15T22:36:31.383327+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:36:31.383327+0000","last_undegraded":"2021-03-15T22:36:31.383327+0000","last_fullsized":"2021-03-15T22:36:31.383327+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.091650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.091650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"246","reported_epoch":"1034","state":"active+clean","last_fresh":"2021-03-15T22:36:29.762886+0000","last_change":"2021-03-15T22:36:18.384469+0000","last_active":"2021-03-15T22:36:29.762886+0000","last_peered":"2021-03-15T22:36:29.762886+0000","last_clean":"2021-03-15T22:36:29.762886+0000","last_became_active":"2021-03-15T22:36:18.384088+0000","last_became_peered":"2021-03-15T22:36:18.384088+0000","last_unstale":"2021-03-15T22:36:29.762886+0000","last_undegraded":"2021-03-15T22:36:29.762886+0000","last_fullsized":"2021-03-15T22:36:29.762886+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.678246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"1186","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.366252+0000","last_change":"2021-03-15T22:36:15.261768+0000","last_active":"2021-03-15T22:36:30.366252+0000","last_peered":"2021-03-15T22:36:30.366252+0000","last_clean":"2021-03-15T22:36:30.366252+0000","last_became_active":"2021-03-15T22:34:55.341958+0000","last_became_peered":"2021-03-15T22:34:55.341958+0000","last_unstale":"2021-03-15T22:36:30.366252+0000","last_undegraded":"2021-03-15T22:36:30.366252+0000","last_fullsized":"2021-03-15T22:36:30.366252+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:36:15.261676+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:36:15.261676+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1069","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.376499+0000","last_change":"2021-03-15T22:36:24.385650+0000","last_active":"2021-03-15T22:36:32.376499+0000","last_peered":"2021-03-15T22:36:32.376499+0000","last_clean":"2021-03-15T22:36:32.376499+0000","last_became_active":"2021-03-15T22:36:24.383694+0000","last_became_peered":"2021-03-15T22:36:24.383694+0000","last_unstale":"2021-03-15T22:36:32.376499+0000","last_undegraded":"2021-03-15T22:36:32.376499+0000","last_fullsized":"2021-03-15T22:36:32.376499+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"1035'1731","reported_seq":"4928","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.400249+0000","last_change":"2021-03-15T22:36:30.401557+0000","last_active":"2021-03-15T22:36:30.400249+0000","last_peered":"2021-03-15T22:36:30.400249+0000","last_clean":"2021-03-15T22:36:30.400249+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:36:30.400249+0000","last_undegraded":"2021-03-15T22:36:30.400249+0000","last_fullsized":"2021-03-15T22:36:30.400249+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"1023'1699","last_scrub_stamp":"2021-03-15T22:36:20.059745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.059745+0000","log_size":1731,"ondisk_log_size":1731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":57260299,"num_objects":25,"num_object_clones":21,"num_object_copies":75,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":2,"num_read":552,"num_read_kb":589646,"num_write":381,"num_write_kb":135091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2fb","length":"1"},{"start":"30b","length":"1"},{"start":"325","length":"1"},{"start":"350","length":"1"},{"start":"35f","length":"1"}]},{"pgid":"6.e","version":"0'0","reported_seq":"1051","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.366956+0000","last_change":"2021-03-15T22:36:25.253269+0000","last_active":"2021-03-15T22:36:30.366956+0000","last_peered":"2021-03-15T22:36:30.366956+0000","last_clean":"2021-03-15T22:36:30.366956+0000","last_became_active":"2021-03-15T22:34:55.340498+0000","last_became_peered":"2021-03-15T22:34:55.340498+0000","last_unstale":"2021-03-15T22:36:30.366956+0000","last_undegraded":"2021-03-15T22:36:30.366956+0000","last_fullsized":"2021-03-15T22:36:30.366956+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:25.253188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:36:25.253188+0000","last_clean_scrub_stamp":"2021-03-15T22:36:25.253188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"976'19","reported_seq":"450","reported_epoch":"1037","state":"active+clean","last_fresh":"2021-03-15T22:36:32.375591+0000","last_change":"2021-03-15T22:36:09.696306+0000","last_active":"2021-03-15T22:36:32.375591+0000","last_peered":"2021-03-15T22:36:32.375591+0000","last_clean":"2021-03-15T22:36:32.375591+0000","last_became_active":"2021-03-15T22:35:28.364764+0000","last_became_peered":"2021-03-15T22:35:28.364764+0000","last_unstale":"2021-03-15T22:36:32.375591+0000","last_undegraded":"2021-03-15T22:36:32.375591+0000","last_fullsized":"2021-03-15T22:36:32.375591+0000","mapping_epoch":972,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":973,"parent":"0.0","parent_split_bits":0,"last_scrub":"976'19","last_scrub_stamp":"2021-03-15T22:36:09.696246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:09.696246+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"84'2","reported_seq":"482","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.364903+0000","last_change":"2021-03-15T22:36:14.388343+0000","last_active":"2021-03-15T22:36:30.364903+0000","last_peered":"2021-03-15T22:36:30.364903+0000","last_clean":"2021-03-15T22:36:30.364903+0000","last_became_active":"2021-03-15T22:36:14.387999+0000","last_became_peered":"2021-03-15T22:36:14.387999+0000","last_unstale":"2021-03-15T22:36:30.364903+0000","last_undegraded":"2021-03-15T22:36:30.364903+0000","last_fullsized":"2021-03-15T22:36:30.364903+0000","mapping_epoch":589,"log_start":"84'2","ondisk_log_start":"84'2","created":56,"last_epoch_clean":1019,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.690309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"1225","reported_epoch":"1035","state":"active+clean","last_fresh":"2021-03-15T22:36:30.367391+0000","last_change":"2021-03-15T22:35:46.406478+0000","last_active":"2021-03-15T22:36:30.367391+0000","last_peered":"2021-03-15T22:36:30.367391+0000","last_clean":"2021-03-15T22:36:30.367391+0000","last_became_active":"2021-03-15T22:33:04.337113+0000","last_became_peered":"2021-03-15T22:33:04.337113+0000","last_unstale":"2021-03-15T22:36:30.367391+0000","last_undegraded":"2021-03-15T22:36:30.367391+0000","last_fullsized":"2021-03-15T22:36:30.367391+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:35:46.406376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:46.406376+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":809564827,"num_objects":357,"num_object_clones":307,"num_object_copies":1071,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":357,"num_whiteouts":12,"num_read":7407,"num_read_kb":7627647,"num_write":5368,"num_write_kb":2072157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4621,"num_bytes_recovered":2230024638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1778999296,"data_stored":2473290519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24416,"ondisk_log_size":24416,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1104,"num_read_kb":1099,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1482,"num_read_kb":987,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":94766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":947,"ondisk_log_size":947,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299620,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":280880,"kb_used_data":233640,"kb_used_omap":0,"kb_used_meta":47232,"kb_avail":93464272,"statfs":{"total":95995035648,"available":95707414528,"internally_reserved":0,"allocated":239247360,"data_stored":348109421,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48365568},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.47199999999999998,"15min":0.46899999999999997},"min":{"1min":0.25,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.73},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.222},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":3.8799999999999999},"last":0.46600000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.47099999999999997},"min":{"1min":0.34399999999999997,"5min":0.249,"15min":0.249},"max":{"1min":0.58799999999999997,"5min":0.73899999999999999,"15min":2.6859999999999999},"last":0.36399999999999999},{"interface":"front","average":{"1min":0.502,"5min":0.56299999999999994,"15min":0.51500000000000001},"min":{"1min":0.311,"5min":0.28399999999999997,"15min":0.23400000000000001},"max":{"1min":0.67100000000000004,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":0.60799999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:35:38 2021","interfaces":[{"interface":"back","average":{"1min":2.1760000000000002,"5min":0.83999999999999997,"15min":0.59599999999999997},"min":{"1min":0.28599999999999998,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":29.350999999999999,"5min":29.350999999999999,"15min":29.350999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.498,"15min":0.5},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":1.298,"5min":1.298,"15min":1.298},"last":0.34200000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.33500000000000002,"5min":0.314,"15min":0.28899999999999998},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.702},"last":0.47999999999999998},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.30099999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.806,"5min":1.806,"15min":5.415},"last":0.432}]},{"osd":4,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.436,"15min":0.439},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.23799999999999999},"max":{"1min":0.63500000000000001,"5min":0.753,"15min":1.927},"last":0.38600000000000001},{"interface":"front","average":{"1min":0.502,"5min":0.47399999999999998,"15min":0.46200000000000002},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":0.68600000000000005,"5min":0.72699999999999998,"15min":2.5339999999999998},"last":0.41299999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:35:38 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.47899999999999998,"15min":0.45500000000000002},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.68300000000000005,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.50900000000000001},{"interface":"front","average":{"1min":0.45000000000000001,"5min":0.498,"15min":0.502},"min":{"1min":0.32800000000000001,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.67800000000000005,"5min":0.77700000000000002,"15min":1.071},"last":0.55700000000000005}]},{"osd":6,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.29999999999999999,"15min":0.27100000000000002},"max":{"1min":0.77800000000000002,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54200000000000004,"15min":0.51900000000000002},"min":{"1min":0.38200000000000001,"5min":0.33200000000000002,"15min":0.28399999999999997},"max":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":1.8560000000000001},"last":0.52500000000000002}]}]},{"osd":6,"up_from":45,"seq":193273528551,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":241776,"kb_used_data":189672,"kb_used_omap":0,"kb_used_meta":52096,"kb_avail":93503376,"statfs":{"total":95995035648,"available":95747457024,"internally_reserved":0,"allocated":194224128,"data_stored":260327544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53346304},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.49399999999999999,"15min":0.47999999999999998},"min":{"1min":0.33900000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57099999999999995,"5min":2.5470000000000002,"15min":3.085},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.47299999999999998,"15min":0.46000000000000002},"min":{"1min":0.315,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69699999999999995,"5min":1.137,"15min":1.323},"last":0.503}]},{"osd":1,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.54200000000000004,"15min":0.54800000000000004},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.254},"max":{"1min":0.61099999999999999,"5min":2.7839999999999998,"15min":14.637},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.44700000000000001,"15min":0.46899999999999997},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60999999999999999,"5min":0.84499999999999997,"15min":5.2450000000000001},"last":0.41899999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:35:43 2021","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64800000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.57199999999999995},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70899999999999996,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.379}]},{"osd":3,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.52600000000000002,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.29899999999999999},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.6150000000000002},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.498,"15min":0.496},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.65100000000000002,"5min":2.4670000000000001,"15min":2.7890000000000001},"last":0.48299999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.44900000000000001,"15min":0.437},"min":{"1min":0.26900000000000002,"5min":0.25,"15min":0.224},"max":{"1min":0.70499999999999996,"5min":0.94099999999999995,"15min":4.306},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.47099999999999997,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.26600000000000001,"15min":0.23499999999999999},"max":{"1min":0.65300000000000002,"5min":1.091,"15min":4.181},"last":0.40600000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:35:39 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.47599999999999998,"15min":0.42599999999999999},"min":{"1min":0.30599999999999999,"5min":0.30599999999999999,"15min":0.26700000000000002},"max":{"1min":0.63300000000000001,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.53000000000000003,"15min":0.46999999999999997},"min":{"1min":0.33100000000000002,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.67900000000000005,"5min":1.022,"15min":1.022},"last":0.53600000000000003}]},{"osd":7,"last update":"Mon Mar 15 22:36:25 2021","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47599999999999998,"15min":0.52600000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.249},"max":{"1min":0.63600000000000001,"5min":0.76300000000000001,"15min":15.340999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56999999999999995,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.35399999999999998,"15min":0.27200000000000002},"max":{"1min":0.78200000000000003,"5min":1.0109999999999999,"15min":15.356},"last":0.58399999999999996}]}]},{"osd":1,"up_from":13,"seq":55834575097,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":182064,"kb_used_data":125224,"kb_used_omap":0,"kb_used_meta":56832,"kb_avail":93563088,"statfs":{"total":95995035648,"available":95808602112,"internally_reserved":0,"allocated":128229376,"data_stored":170892403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58195968},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:56 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.372,"15min":0.34000000000000002},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":0.50800000000000001,"5min":4.3920000000000003,"15min":4.3920000000000003},"last":0.27900000000000003},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.42499999999999999,"15min":0.37},"min":{"1min":0.215,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.52500000000000002,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.29199999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:35:45 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.46899999999999997,"15min":0.379},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59199999999999997,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45500000000000002,"15min":0.40100000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64600000000000002,"5min":5.0190000000000001,"15min":5.0190000000000001},"last":0.317}]},{"osd":3,"last update":"Mon Mar 15 22:36:21 2021","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.47999999999999998,"15min":0.40100000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.215},"max":{"1min":0.76500000000000001,"5min":7.718,"15min":7.718},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.51800000000000002,"15min":0.432},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.21199999999999999},"max":{"1min":0.54400000000000004,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":0.33200000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.61199999999999999,"15min":0.55100000000000005},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.22800000000000001},"max":{"1min":0.78800000000000003,"5min":4.593,"15min":4.593},"last":0.41299999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58599999999999997,"15min":0.57699999999999996},"min":{"1min":0.38,"5min":0.35499999999999998,"15min":0.253},"max":{"1min":0.67100000000000004,"5min":4.3140000000000001,"15min":4.6139999999999999},"last":0.45300000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:35:34 2021","interfaces":[{"interface":"back","average":{"1min":0.74199999999999999,"5min":0.59499999999999997,"15min":0.54700000000000004},"min":{"1min":0.41599999999999998,"5min":0.34499999999999997,"15min":0.34300000000000003},"max":{"1min":4.3630000000000004,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.497},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.60999999999999999,"15min":0.52900000000000003},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.32100000000000001},"max":{"1min":1.794,"5min":3.472,"15min":3.472},"last":0.40699999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:35:59 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.60799999999999998,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.38500000000000001,"15min":0.314},"max":{"1min":0.755,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.442},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.63100000000000001,"15min":0.57899999999999996},"min":{"1min":0.41199999999999998,"5min":0.375,"15min":0.33700000000000002},"max":{"1min":0.68300000000000005,"5min":4.6529999999999996,"15min":4.6529999999999996},"last":0.48099999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:36:21 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.63600000000000001,"15min":0.59999999999999998},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.32100000000000001},"max":{"1min":0.71899999999999997,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.66500000000000004,"15min":0.59399999999999997},"min":{"1min":0.45400000000000001,"5min":0.40300000000000002,"15min":0.35799999999999998},"max":{"1min":0.73099999999999998,"5min":4.625,"15min":4.625},"last":0.50600000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738619,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":316680,"kb_used_data":268096,"kb_used_omap":0,"kb_used_meta":48576,"kb_avail":93428472,"statfs":{"total":95995035648,"available":95670755328,"internally_reserved":0,"allocated":274530304,"data_stored":397066578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":49741824},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:35:47 2021","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35899999999999999,"15min":0.34000000000000002},"min":{"1min":0.249,"5min":0.192,"15min":0.17999999999999999},"max":{"1min":0.68000000000000005,"5min":1.46,"15min":1.7070000000000001},"last":0.372},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39600000000000002,"15min":0.36899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19900000000000001},"max":{"1min":0.499,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.26900000000000002}]},{"osd":2,"last update":"Mon Mar 15 22:35:41 2021","interfaces":[{"interface":"back","average":{"1min":0.36599999999999999,"5min":0.374,"15min":0.36399999999999999},"min":{"1min":0.28000000000000003,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.70099999999999996,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.432},{"interface":"front","average":{"1min":0.375,"5min":0.49199999999999999,"15min":0.41399999999999998},"min":{"1min":0.24399999999999999,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.51300000000000001,"5min":7.3799999999999999,"15min":7.3799999999999999},"last":0.312}]},{"osd":3,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":0.71399999999999997,"5min":1.544,"15min":1.544},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.46700000000000003,"15min":0.41999999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.60399999999999998,"5min":3.988,"15min":3.988},"last":0.46300000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.55200000000000005,"15min":0.53700000000000003},"min":{"1min":0.34799999999999998,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":0.76100000000000001,"5min":1.861,"15min":1.861},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.36799999999999999,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":0.78300000000000003,"5min":1.5700000000000001,"15min":1.645},"last":0.48299999999999998}]},{"osd":5,"last update":"Mon Mar 15 22:35:35 2021","interfaces":[{"interface":"back","average":{"1min":0.54400000000000004,"5min":0.60799999999999998,"15min":0.53500000000000003},"min":{"1min":0.371,"5min":0.32600000000000001,"15min":0.29899999999999999},"max":{"1min":0.71599999999999997,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.504,"5min":0.56000000000000005,"15min":0.502},"min":{"1min":0.377,"5min":0.35899999999999999,"15min":0.318},"max":{"1min":0.72099999999999997,"5min":1.909,"15min":1.909},"last":0.40899999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:36:01 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.57399999999999995,"15min":0.55600000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.26200000000000001},"max":{"1min":0.78800000000000003,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.59299999999999997,"15min":0.56899999999999995},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.27100000000000002},"max":{"1min":0.77300000000000002,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.46999999999999997}]},{"osd":7,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.65000000000000002,"15min":0.58499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.27900000000000003},"max":{"1min":0.82499999999999996,"5min":3.8239999999999998,"15min":3.8239999999999998},"last":0.442},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.60899999999999999,"15min":0.57699999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.317},"max":{"1min":0.72499999999999998,"5min":2.665,"15min":2.665},"last":0.53300000000000003}]}]},{"osd":2,"up_from":681,"seq":2924872728650,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":352964,"kb_used_data":315068,"kb_used_omap":119,"kb_used_meta":37768,"kb_avail":93392188,"statfs":{"total":95995035648,"available":95633600512,"internally_reserved":0,"allocated":322629632,"data_stored":430664286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":38674793},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.316,"15min":0.31},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.56299999999999994,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.35099999999999998},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35299999999999998,"15min":0.33100000000000002},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.77800000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":1.4099999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.35599999999999998,"15min":0.33400000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.58099999999999996,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.379,"15min":0.36299999999999999},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.59799999999999998,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.92900000000000005}]},{"osd":3,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.42099999999999999,"15min":0.36499999999999999},"min":{"1min":0.26200000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.65500000000000003,"5min":3.496,"15min":3.496},"last":0.96599999999999997},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.36199999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.85099999999999998,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.38200000000000001}]},{"osd":4,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54800000000000004,"15min":0.49399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67600000000000005,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53800000000000003,"15min":0.499},"min":{"1min":0.378,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76700000000000002,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":1.0069999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.875,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.59799999999999998},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56100000000000005,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70299999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.42899999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57599999999999996,"15min":0.51900000000000002},"min":{"1min":0.372,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76600000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.52500000000000002},"min":{"1min":0.433,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":1.387,"15min":1.387},"last":0.98999999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59399999999999997,"15min":0.54300000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.748,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.58399999999999996},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57599999999999996,"15min":0.53200000000000003},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82599999999999996,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":1.022}]}]},{"osd":3,"up_from":26,"seq":111669149937,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":326960,"kb_used_data":260648,"kb_used_omap":0,"kb_used_meta":66304,"kb_avail":93418192,"statfs":{"total":95995035648,"available":95660228608,"internally_reserved":0,"allocated":266903552,"data_stored":356313036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67895296},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:24 2021","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.17699999999999999},"max":{"1min":0.39500000000000002,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.32600000000000001,"15min":0.33600000000000002},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.41999999999999998,"5min":0.76900000000000002,"15min":2.6339999999999999},"last":0.25900000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:36:24 2021","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36899999999999999,"15min":0.39200000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55900000000000005,"5min":0.626,"15min":6.0199999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.37,"15min":0.39900000000000002},"min":{"1min":0.25900000000000001,"5min":0.219,"15min":0.20699999999999999},"max":{"1min":0.48599999999999999,"5min":2.5619999999999998,"15min":7.6130000000000004},"last":0.30299999999999999}]},{"osd":2,"last update":"Mon Mar 15 22:35:40 2021","interfaces":[{"interface":"back","average":{"1min":0.36699999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"min":{"1min":0.246,"5min":0.216,"15min":0.216},"max":{"1min":0.53100000000000003,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.24099999999999999},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.39000000000000001,"15min":0.38900000000000001},"min":{"1min":0.30499999999999999,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54000000000000004,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.28999999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:35:36 2021","interfaces":[{"interface":"back","average":{"1min":0.52100000000000002,"5min":0.51900000000000002,"15min":0.53200000000000003},"min":{"1min":0.35599999999999998,"5min":0.31900000000000001,"15min":0.22600000000000001},"max":{"1min":0.72199999999999998,"5min":0.86899999999999999,"15min":3.3370000000000002},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.55700000000000005,"5min":0.53900000000000003,"15min":0.52600000000000002},"min":{"1min":0.314,"5min":0.311,"15min":0.26700000000000002},"max":{"1min":0.84499999999999997,"5min":0.84499999999999997,"15min":2.2890000000000001},"last":0.442}]},{"osd":5,"last update":"Mon Mar 15 22:35:36 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.755},"min":{"1min":0.35099999999999998,"5min":0.311,"15min":0.28299999999999997},"max":{"1min":0.73099999999999998,"5min":0.75800000000000001,"15min":9.4870000000000001},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.57299999999999995,"5min":0.54100000000000004,"15min":0.95899999999999996},"min":{"1min":0.37,"5min":0.36699999999999999,"15min":0.32800000000000001},"max":{"1min":0.79400000000000004,"5min":0.84799999999999998,"15min":15.776},"last":0.64000000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:36:06 2021","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38900000000000001,"5min":0.38200000000000001,"15min":0.30099999999999999},"max":{"1min":0.81699999999999995,"5min":1.359,"15min":2.6499999999999999},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.38900000000000001,"5min":0.375,"15min":0.30399999999999999},"max":{"1min":0.78500000000000003,"5min":1.3400000000000001,"15min":3.472},"last":0.54800000000000004}]},{"osd":7,"last update":"Mon Mar 15 22:36:24 2021","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55100000000000005,"15min":0.57499999999999996},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.29599999999999999},"max":{"1min":0.752,"5min":0.91200000000000003,"15min":5.625},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.57599999999999996,"15min":0.56200000000000006},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73199999999999998,"5min":2.621,"15min":2.621},"last":0.39800000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986414,"num_pgs":61,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":154064,"kb_used_data":103752,"kb_used_omap":0,"kb_used_meta":50304,"kb_avail":93591088,"statfs":{"total":95995035648,"available":95837274112,"internally_reserved":0,"allocated":106242048,"data_stored":154615056,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51511296},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.45200000000000001,"5min":0.50800000000000001,"15min":0.47599999999999998},"min":{"1min":0.30099999999999999,"5min":0.221,"15min":0.20000000000000001},"max":{"1min":0.55900000000000005,"5min":4.7050000000000001,"15min":4.7050000000000001},"last":0.748},{"interface":"front","average":{"1min":0.435,"5min":0.46500000000000002,"15min":0.498},"min":{"1min":0.27300000000000002,"5min":0.246,"15min":0.246},"max":{"1min":0.67100000000000004,"5min":1.4179999999999999,"15min":5.0209999999999999},"last":0.65900000000000003}]},{"osd":1,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.47599999999999998,"15min":0.48099999999999998},"min":{"1min":0.312,"5min":0.25,"15min":0.23999999999999999},"max":{"1min":0.73399999999999999,"5min":1.3720000000000001,"15min":4.3499999999999996},"last":0.77900000000000003},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.499,"15min":0.499},"min":{"1min":0.311,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":0.627,"5min":1.494,"15min":4.718},"last":0.441}]},{"osd":2,"last update":"Mon Mar 15 22:35:41 2021","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47999999999999998,"15min":0.502},"min":{"1min":0.32200000000000001,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.77700000000000002,"5min":0.78000000000000003,"15min":0.78000000000000003},"last":0.48899999999999999},{"interface":"front","average":{"1min":0.497,"5min":0.48599999999999999,"15min":0.495},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.94899999999999995,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.69399999999999995}]},{"osd":3,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53200000000000003,"15min":0.54800000000000004},"min":{"1min":0.35699999999999998,"5min":0.32200000000000001,"15min":0.28499999999999998},"max":{"1min":1.0820000000000001,"5min":1.486,"15min":5.0499999999999998},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.56399999999999995,"5min":0.56299999999999994,"15min":0.54400000000000004},"min":{"1min":0.38300000000000001,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":1.0660000000000001,"5min":1.4950000000000001,"15min":4.4050000000000002},"last":0.60999999999999999}]},{"osd":5,"last update":"Mon Mar 15 22:35:32 2021","interfaces":[{"interface":"back","average":{"1min":0.39100000000000001,"5min":0.41999999999999998,"15min":0.40799999999999997},"min":{"1min":0.27600000000000002,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.623,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.45800000000000002,"5min":0.47499999999999998,"15min":0.436},"min":{"1min":0.29099999999999998,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.66400000000000003,"5min":0.77000000000000002,"15min":0.77000000000000002},"last":0.71599999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:35:59 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.27500000000000002,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":4.3079999999999998},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.36199999999999999,"5min":0.33000000000000002,"15min":0.28000000000000003},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":4.6600000000000001},"last":0.76800000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:36:21 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.503,"15min":0.505},"min":{"1min":0.34799999999999998,"5min":0.28299999999999997,"15min":0.26300000000000001},"max":{"1min":0.71699999999999997,"5min":0.79900000000000004,"15min":4.6929999999999996},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52100000000000002,"15min":0.54200000000000004},"min":{"1min":0.40899999999999997,"5min":0.29999999999999999,"15min":0.25},"max":{"1min":0.66200000000000003,"5min":1.034,"15min":4.9900000000000002},"last":0.79300000000000004}]}]},{"osd":5,"up_from":616,"seq":2645699854423,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":289276,"kb_used_data":258164,"kb_used_omap":153,"kb_used_meta":30950,"kb_avail":93455876,"statfs":{"total":95995035648,"available":95698817024,"internally_reserved":0,"allocated":264359936,"data_stored":368846739,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":31693329},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42899999999999999,"5min":0.48599999999999999,"15min":0.46300000000000002},"min":{"1min":0.26400000000000001,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.60799999999999998,"5min":2.2130000000000001,"15min":2.2130000000000001},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.438,"5min":0.48099999999999998,"15min":0.45500000000000002},"min":{"1min":0.28699999999999998,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":0.63700000000000001,"5min":2.5030000000000001,"15min":2.5030000000000001},"last":0.52000000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.42099999999999999,"5min":0.55600000000000005,"15min":0.48199999999999998},"min":{"1min":0.32000000000000001,"5min":0.27500000000000002,"15min":0.27500000000000002},"max":{"1min":0.56599999999999995,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":0.47299999999999998},{"interface":"front","average":{"1min":0.44900000000000001,"5min":0.48499999999999999,"15min":0.48499999999999999},"min":{"1min":0.29399999999999998,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.54600000000000004,"5min":1.161,"15min":1.161},"last":0.48099999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.58299999999999996,"15min":0.53600000000000003},"min":{"1min":0.312,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.60399999999999998,"5min":10.952,"15min":10.952},"last":0.50800000000000001},{"interface":"front","average":{"1min":0.499,"5min":0.50800000000000001,"15min":0.52400000000000002},"min":{"1min":0.34300000000000003,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.623,"5min":1.125,"15min":1.125},"last":0.54600000000000004}]},{"osd":3,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.67800000000000005,"15min":0.53800000000000003},"min":{"1min":0.28399999999999997,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.63200000000000001,"5min":10.355,"15min":10.355},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.51100000000000001,"15min":0.45900000000000002},"min":{"1min":0.29699999999999999,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.65800000000000003,"5min":2.476,"15min":2.476},"last":0.55800000000000005}]},{"osd":4,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.39800000000000002,"5min":0.51000000000000001,"15min":0.41399999999999998},"min":{"1min":0.27600000000000002,"5min":0.254,"15min":0.254},"max":{"1min":0.66900000000000004,"5min":7.4000000000000004,"15min":7.4000000000000004},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.436,"5min":0.47299999999999998,"15min":0.40799999999999997},"min":{"1min":0.32800000000000001,"5min":0.25700000000000001,"15min":0.25700000000000001},"max":{"1min":0.71599999999999997,"5min":2.4100000000000001,"15min":2.4100000000000001},"last":0.45500000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.50900000000000001,"15min":0.46500000000000002},"min":{"1min":0.30099999999999999,"5min":0.30099999999999999,"15min":0.23999999999999999},"max":{"1min":0.64400000000000002,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.501,"5min":0.56200000000000006,"15min":0.49199999999999999},"min":{"1min":0.371,"5min":0.371,"15min":0.29699999999999999},"max":{"1min":0.70399999999999996,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.56699999999999995}]},{"osd":7,"last update":"Mon Mar 15 22:35:42 2021","interfaces":[{"interface":"back","average":{"1min":0.48499999999999999,"5min":0.52900000000000003,"15min":0.46300000000000002},"min":{"1min":0.36899999999999999,"5min":0.29799999999999999,"15min":0.29299999999999998},"max":{"1min":0.69199999999999995,"5min":2.5699999999999998,"15min":2.5699999999999998},"last":0.53500000000000003},{"interface":"front","average":{"1min":0.504,"5min":0.58799999999999997,"15min":0.51700000000000002},"min":{"1min":0.34300000000000003,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.68500000000000005,"5min":5.944,"15min":5.944},"last":0.58799999999999997}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":12026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":10320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":8144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":272347136,"data_stored":395740881,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":125964288,"data_stored":169548271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":320540672,"data_stored":429318417,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":264724480,"data_stored":354988283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":104050688,"data_stored":150431794,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":262266880,"data_stored":367500756,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":192040960,"data_stored":259002649,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":237064192,"data_stored":346759468,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:36:37.624 INFO:tasks.thrashosds.thrasher:Setting 3.9 to [5, 4, 6] 2021-03-15T22:36:37.625 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '3.9', '5', '4', '6'] 2021-03-15T22:36:37.625 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pg-upmap 3.9 5 4 6 2021-03-15T22:36:37.626 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:37.653 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:37.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:37.654 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:37.755 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:37.781 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:37.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:37.783 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:37.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:37 smithi125 conmon[56739]: audit 2021-03-15T22:36:36.169485+0000 mgr.y ( 2021-03-15T22:36:37.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:37 smithi125 conmon[56739]: mgr.14144) 1131 : audit [DBG] from='client.16971 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:37.802 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset noscrub 2021-03-15T22:36:37.884 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:37.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:37.912 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:37.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:37.946 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:38.013 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:38.038 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:38.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:38.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:38.143 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:38.169 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:38.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:38.171 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:38.232 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:36:38.233 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:36:38.271 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:38.297 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:38.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:38.299 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:38.400 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:38.416 INFO:tasks.rados.rados.0.smithi076.stdout:3994: done (2 left) 2021-03-15T22:36:38.416 INFO:tasks.rados.rados.0.smithi076.stdout:3995: done (1 left) 2021-03-15T22:36:38.416 INFO:tasks.rados.rados.0.smithi076.stdout:3996: done (0 left) 2021-03-15T22:36:38.417 INFO:tasks.rados.rados.0.smithi076.stdout:3997: copy_from oid 10 from oid 44 current snap is 472 2021-03-15T22:36:38.417 INFO:tasks.rados.rados.0.smithi076.stdout:3998: copy_from oid 38 from oid 2 current snap is 472 2021-03-15T22:36:38.417 INFO:tasks.rados.rados.0.smithi076.stdout:3999: write (excl) oid 27 current snap is 472 2021-03-15T22:36:38.417 INFO:tasks.rados.rados.0.smithi076.stdout:3999: seq_num 1350 ranges {689450=580631,1948691=740759} 2021-03-15T22:36:38.423 INFO:tasks.rados.rados.0.smithi076.stdout:3999: writing smithi07694878-27 from 689450 to 1270081 tid 1 2021-03-15T22:36:38.431 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:38.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:38.432 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:38.433 INFO:tasks.rados.rados.0.smithi076.stdout:3999: writing smithi07694878-27 from 1948691 to 2689450 tid 2 2021-03-15T22:36:38.434 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 3 2021-03-15T22:36:38.434 INFO:tasks.rados.rados.0.smithi076.stdout:3999: finishing write tid 1 to smithi07694878-27 2021-03-15T22:36:38.440 INFO:tasks.rados.rados.0.smithi076.stdout:3997: finishing copy_from to smithi07694878-10 2021-03-15T22:36:38.440 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 10 v 3165 (ObjNum 1326 snap 462 seq_num 1326) dirty exists 2021-03-15T22:36:38.440 INFO:tasks.rados.rados.0.smithi076.stdout:3997: finishing copy_from racing read to smithi07694878-10 2021-03-15T22:36:38.441 INFO:tasks.rados.rados.0.smithi076.stdout:3997: done (2 left) 2021-03-15T22:36:38.441 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 2 2021-03-15T22:36:38.441 INFO:tasks.rados.rados.0.smithi076.stdout:3998: finishing copy_from to smithi07694878-38 2021-03-15T22:36:38.441 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 38 v 3155 (ObjNum 1221 snap 423 seq_num 1221) dirty exists 2021-03-15T22:36:38.443 INFO:tasks.rados.rados.0.smithi076.stdout:3999: finishing write tid 2 to smithi07694878-27 2021-03-15T22:36:38.443 INFO:tasks.rados.rados.0.smithi076.stdout:3998: finishing copy_from racing read to smithi07694878-38 2021-03-15T22:36:38.443 INFO:tasks.rados.rados.0.smithi076.stdout:3998: done (1 left) 2021-03-15T22:36:38.443 INFO:tasks.rados.rados.0.smithi076.stdout: waiting on 1 2021-03-15T22:36:38.444 INFO:tasks.rados.rados.0.smithi076.stdout:3999: finishing write tid 3 to smithi07694878-27 2021-03-15T22:36:38.444 INFO:tasks.rados.rados.0.smithi076.stdout:3999: finishing write tid 4 to smithi07694878-27 2021-03-15T22:36:38.444 INFO:tasks.rados.rados.0.smithi076.stdout:update_object_version oid 27 v 3171 (ObjNum 1350 snap 472 seq_num 1350) dirty exists 2021-03-15T22:36:38.445 INFO:tasks.rados.rados.0.smithi076.stdout:3999: left oid 27 (ObjNum 1350 snap 472 seq_num 1350) 2021-03-15T22:36:38.445 INFO:tasks.rados.rados.0.smithi076.stdout:3999: done (0 left) 2021-03-15T22:36:38.446 INFO:tasks.rados.rados.0.smithi076.stderr:0 errors. 2021-03-15T22:36:38.447 INFO:tasks.rados.rados.0.smithi076.stderr: 2021-03-15T22:36:38.447 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd dump --format=json 2021-03-15T22:36:38.534 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:38.555 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:38.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:38.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:38.658 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:38.682 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:38.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:38.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:38.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:38 smithi076 conmon[56514]: cluster 2021-03-15T22:36:37.238245+0000 mgr.y (mgr.14144) 1132 : cluster [DBG] pgmap v1709: 148 pgs: 148 active+clean; 772 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 589 KiB/s wr, 9 op/s 2021-03-15T22:36:38.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:38 smithi076 conmon[56514]: cluster 2021-03-15T22:36:37.406719+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e1042: 8 total, 8 up, 8 in 2021-03-15T22:36:38.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:38 smithi076 conmon[62156]: cluster 2021-03-15T22:36:37.238245+0000 mgr.y (mgr.14144) 1132 : cluster [DBG] pgmap v1709: 148 pgs: 148 active+clean; 772 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 589 KiB/s wr, 9 op/s 2021-03-15T22:36:38.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:38 smithi076 conmon[62156]: cluster 2021-03-15T22:36:37.406719+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e1042: 8 total, 8 up, 8 in 2021-03-15T22:36:38.785 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:38 smithi125 conmon[56739]: cluster 2021-03-15T22:36:37.238245+0000 mgr.y (mgr.14144) 1132 : cluster 2021-03-15T22:36:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:38 smithi125 conmon[56739]: [DBG] pgmap v1709: 148 pgs: 148 active+clean; 772 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 589 KiB/s wr, 9 op/s 2021-03-15T22:36:38.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:38 smithi125 conmon[56739]: cluster 2021-03-15T22:36:37.406719+0000 mon.a (mon.0) 2596 : cluster [DBG] osdmap e1042: 8 total, 8 up, 8 in 2021-03-15T22:36:38.809 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:38.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:38.811 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:38.912 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:38.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:38.937 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:38.937 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:39.039 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:39.065 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:39.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:39.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:39.148 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.1 dump_historic_ops 2021-03-15T22:36:39.168 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:39.214 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:39.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:39.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:39.316 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:39.342 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:39.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:39.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:39.374 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:39.375 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:39.422 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:39 smithi076 conmon[56514]: cluster 2021-03-15T22:36:38.413055+0000 mon.a (mon.0) 2597 : cluster [DBG] 2021-03-15T22:36:39.422 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:39 smithi076 conmon[56514]: osdmap e1043: 8 total, 8 up, 8 in 2021-03-15T22:36:39.444 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:39.469 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:39.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:39.471 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:39.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:39.594 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:39.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:39.595 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:39.685 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:39.697 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:39.723 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:39.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:39.725 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:39.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:39 smithi076 conmon[62156]: cluster 2021-03-15T22:36:38. 2021-03-15T22:36:39.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:39 smithi076 conmon[62156]: 413055+0000 mon.a (mon.0) 2597 : cluster [DBG] osdmap e1043: 8 total, 8 up, 8 in 2021-03-15T22:36:39.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:39 smithi125 conmon[56739]: cluster 2021-03-15T22:36:38.413055+0000 mon.a (mon.0) 2597 : cluster [DBG] osdmap e1043: 8 total, 8 up, 8 in 2021-03-15T22:36:39.826 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:39.852 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:39.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:39.853 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:39.878 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:36:39.955 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:39.979 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:39.979 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:39.980 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:40.081 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:40.106 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:40.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:40.107 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:40.209 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:40.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:40.234 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:40.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:40.335 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:40.358 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:40.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:40.360 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:40.461 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:40.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:40.485 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:40.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:40.587 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:40.612 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:40.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:40.613 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:40.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[62156]: cluster 2021-03-15T22:36:40.643 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[62156]: 2021-03-15T22:36:39 2021-03-15T22:36:40.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[62156]: .238633+0000 mgr.y (mgr.14144) 1133 : cluster [DBG] pgmap v1712: 148 pgs: 148 active+clean; 775 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-03-15T22:36:40.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[62156]: cluster 2021-03- 2021-03-15T22:36:40.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[62156]: 15T22:36:40.365136+0000 mon.a (mon 2021-03-15T22:36:40.644 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[62156]: .0) 2598 : cluster [DBG] osdmap e1044: 8 total, 8 up, 8 in 2021-03-15T22:36:40.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: cluster 2021-03-15T22:36:40.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 2021-03-15T22:36:40.645 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 2021-03-15T 2021-03-15T22:36:40.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 22:36:39.238633+0000 mgr.y (mgr.14144) 1133 : cluster [DBG] pgmap v1712: 148 pgs: 148 active+clean; 775 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-03-15T22:36:40.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: cluster 2021-03-15T22:36:40.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 2021-03-15T22:36:40.646 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 2021 2021-03-15T22:36:40.647 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: - 2021-03-15T22:36:40.647 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 03 2021-03-15T22:36:40.647 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: - 2021-03-15T22:36:40.647 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 15 2021-03-15T22:36:40.647 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: T 2021-03-15T22:36:40.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 22 2021-03-15T22:36:40.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: : 2021-03-15T22:36:40.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 36 2021-03-15T22:36:40.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: : 2021-03-15T22:36:40.648 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 40 2021-03-15T22:36:40.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: . 2021-03-15T22:36:40.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 365136 2021-03-15T22:36:40.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: +0000 2021-03-15T22:36:40.649 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 2021-03-15T22:36:40.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: mon.a 2021-03-15T22:36:40.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: ( 2021-03-15T22:36:40.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: mon 2021-03-15T22:36:40.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: . 2021-03-15T22:36:40.650 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 0 2021-03-15T22:36:40.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: ) 2021-03-15T22:36:40.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 2598 2021-03-15T22:36:40.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: : 2021-03-15T22:36:40.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: cluster 2021-03-15T22:36:40.651 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 2021-03-15T22:36:40.652 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: [DBG] 2021-03-15T22:36:40.652 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: 2021-03-15T22:36:40.652 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:40 smithi076 conmon[56514]: osdmap e1044: 8 total, 8 up, 8 in 2021-03-15T22:36:40.714 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:40.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:40.739 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:40.740 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:40.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:36:40 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:36:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:40 smithi125 conmon[56739]: cluster 2021-03-15T22:36:39.238633+0000 mgr.y (mgr.14144) 2021-03-15T22:36:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:40 smithi125 conmon[56739]: 1133 : cluster [DBG] pgmap v1712: 148 pgs: 148 active+clean; 775 MiB data, 2.0 GiB used, 713 GiB / 715 GiB avail; 5.9 MiB/s rd, 1.4 MiB/s wr, 8 op/s 2021-03-15T22:36:40.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:40 smithi125 conmon[56739]: cluster 2021-03-15T22:36:40.365136+0000 mon.a (mon.0) 2598 : cluster [DBG] osdmap e1044: 8 total, 8 up, 8 in 2021-03-15T22:36:40.841 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:40.866 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:40.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:40.867 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:40.968 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:40.994 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:40.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:41.097 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:41.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:41.120 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:41.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:41.222 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:41.244 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:41.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:41.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:41.347 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:41.380 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:41.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:41.381 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:41.483 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:41.509 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:41.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:41.511 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:41.516 INFO:teuthology.orchestra.run.smithi076.stderr:set 3.9 pg_upmap mapping to [5,4,6] 2021-03-15T22:36:41.612 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:41.635 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:41.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:41.637 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:41.738 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:41.760 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:41.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:41.762 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:41.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[56514]: cluster 2021-03-15T22:36:40.329291+0000 osd.6 (osd 2021-03-15T22:36:41.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[56514]: .6) 157 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:41.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[56514]: cluster 2021-03-15T22:36:40.331606+0000 osd.6 (osd.6) 158 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:41.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[56514]: audit 2021-03-15T22:36:40.358443+0000 mgr.y (mgr.14144) 1134 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:41.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[56514]: audit 2021-03-15T22:36:40.689536+0000 mon.a (mon.0) 2599 : audit [INF] from='client.? 172.21.15.76:0/3192816236' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.9", "id": [5, 4, 6]}]: dispatch 2021-03-15T22:36:41.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[62156]: cluster 2021-03-15T22:36:40.329291+0000 osd.6 2021-03-15T22:36:41.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[62156]: (osd.6) 157 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:41.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[62156]: cluster 2021-03-15T22:36:40.331606+0000 2021-03-15T22:36:41.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[62156]: osd.6 (osd.6) 158 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:41.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[62156]: audit 2021-03-15T22 2021-03-15T22:36:41.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[62156]: :36:40.358443+0000 mgr.y (mgr.14144 2021-03-15T22:36:41.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[62156]: ) 1134 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:41.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:36:41.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[62156]: 03-15T22:36:40.689536+0000 2021-03-15T22:36:41.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[62156]: mon.a (mon.0) 2599 : audit [INF] 2021-03-15T22:36:41.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:41 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/3192816236' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.9", "id": [5, 4, 6]}]: dispatch 2021-03-15T22:36:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:41 smithi125 conmon[56739]: cluster 2021-03-15T22:36:40.329291+0000 osd.6 (osd.6) 157 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:41 smithi125 conmon[56739]: 2021-03-15T22:36:41.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:41 smithi125 conmon[56739]: cluster 2021-03-15T22:36:40.331606+0000 osd.6 (osd.6) 158 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:41.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:41 smithi125 conmon[56739]: audit 2021-03-15T22:36:40. 2021-03-15T22:36:41.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:41 smithi125 conmon[56739]: 358443+0000 mgr.y (mgr.14144) 1134 2021-03-15T22:36:41.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:41 smithi125 conmon[56739]: : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:41.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:41 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:36:41.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:41 smithi125 conmon[56739]: -15T22:36:40.689536+0000 2021-03-15T22:36:41.790 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:41 smithi125 conmon[56739]: mon.a (mon.0) 2599 : audit 2021-03-15T22:36:41.791 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:41 smithi125 conmon[56739]: [INF] from='client.? 172.21.15.76:0/3192816236' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "3.9", "id": [5, 4, 6]}]: dispatch 2021-03-15T22:36:41.863 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:41.889 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:41.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:41.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:41.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:42.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:42.016 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:42.017 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:42.118 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:42.146 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:42.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:42.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:42.249 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:42.272 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:42.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:42.274 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:42.375 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:42.396 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:42.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:42.398 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:42.499 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:42.523 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:42.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:42.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:42.626 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:42.649 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:42.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:42.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:42.752 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:42.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:42 smithi076 conmon[56514]: cluster 2021-03-15T22:36:41. 2021-03-15T22:36:42.767 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:42 smithi076 conmon[56514]: 239051+0000 mgr.y (mgr.14144) 1135 : cluster [DBG] pgmap v1714: 148 pgs: 148 active+clean; 791 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2021-03-15T22:36:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:42 smithi076 conmon[56514]: audit 2021-03-15T22:36:41.511514+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? 172.21.15.76:0/3192816236' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.9", "id": [5, 4, 6]}]': finished 2021-03-15T22:36:42.768 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:42 smithi076 conmon[56514]: cluster 2021-03-15T22:36:41.511558+0000 mon.a (mon.0) 2601 : cluster [DBG] osdmap e1045: 8 total, 8 up, 8 in 2021-03-15T22:36:42.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:42 smithi076 conmon[62156]: cluster 2021-03-15T22:36:41.239051 2021-03-15T22:36:42.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:42 smithi076 conmon[62156]: +0000 mgr.y (mgr.14144) 1135 : cluster [DBG] pgmap v1714: 148 pgs: 148 active+clean; 791 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2021-03-15T22:36:42.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:42 smithi076 conmon[62156]: audit 2021-03-15T22:36:41.511514+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? 172.21.15.76:0/3192816236' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.9", "id": [5, 4, 6]}]': finished 2021-03-15T22:36:42.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:42 smithi076 conmon[62156]: cluster 2021-03-15T22:36:41.511558+0000 mon.a (mon.0) 2021-03-15T22:36:42.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:42 smithi076 conmon[62156]: 2601 : cluster [DBG] osdmap e1045: 8 total, 8 up, 8 in 2021-03-15T22:36:42.778 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:42.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:42.780 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:42.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:42 smithi125 conmon[56739]: cluster 2021-03-15T22:36:41.239051+0000 mgr.y (mgr.14144) 1135 : cluster [DBG] 2021-03-15T22:36:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:42 smithi125 conmon[56739]: pgmap v1714: 148 pgs: 148 active+clean; 791 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 4.1 MiB/s rd, 3.2 MiB/s wr, 13 op/s 2021-03-15T22:36:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:42 smithi125 conmon[56739]: audit 2021-03-15T22:36:41.511514+0000 mon.a (mon.0) 2600 : audit [INF] from='client.? 172.21.15.76:0/3192816236' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "3.9", "id": [5, 4, 6]}]': finished 2021-03-15T22:36:42.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:42 smithi125 conmon[56739]: cluster 2021-03-15T22:36:41.511558+0000 mon.a (mon.0) 2021-03-15T22:36:42.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:42 smithi125 conmon[56739]: 2601 : cluster [DBG] osdmap e1045: 8 total, 8 up, 8 in 2021-03-15T22:36:42.852 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:42.881 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:42.907 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:42.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:42.908 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:43.010 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:43.031 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:43.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:43.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:43.134 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:43.160 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:43.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:43.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:43.263 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:43.288 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:43.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:43.290 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:43.369 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:36:43.369 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:36:43.391 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:43.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:43.415 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:43.415 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:43.517 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:43.542 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:43.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:43.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:43.645 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:43.670 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:43.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:43.672 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:43.688 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:43 smithi076 conmon[56514]: cluster 2021-03-15T22:36:42.513169+0000 mon.a (mon.0) 2602 : cluster [DBG] osdmap e1046: 8 total, 8 up, 8 in 2021-03-15T22:36:43.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:43 smithi076 conmon[62156]: cluster 2021-03-15T22:36:42.513169+0000 mon.a 2021-03-15T22:36:43.689 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:43 smithi076 conmon[62156]: (mon.0) 2602 : cluster [DBG] osdmap e1046: 8 total, 8 up, 8 in 2021-03-15T22:36:43.773 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:43.786 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:43 smithi125 conmon[56739]: cluster 2021-03-15T22:36:42.513169+0000 mon.a (mon.0) 2602 : 2021-03-15T22:36:43.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:43 smithi125 conmon[56739]: cluster [DBG] osdmap e1046: 8 total, 8 up, 8 in 2021-03-15T22:36:43.796 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:43.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:43.797 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:43.898 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:43.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:43.922 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:43.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:44.024 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:44.047 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:36:44.048 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1046,"fsid":"8a572802-85db-11eb-9091-001a4aab830c","created":"2021-03-15T22:13:20.370795+0000","modified":"2021-03-15T22:36:42.509058+0000","last_up_change":"2021-03-15T22:30:30.205134+0000","last_in_change":"2021-03-15T22:34:54.369110+0000","flags":"noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5806080,"flags_set":["nodeep-scrub","noscrub","pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":20,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":7,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-15T22:16:34.817191+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":11,"pg_placement_num":11,"pg_placement_num_target":11,"pg_num_target":11,"pg_num_pending":11,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1028","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"1026","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"iscsi","create_time":"2021-03-15T22:17:49.906400+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"56","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":".rgw.root","create_time":"2021-03-15T22:17:52.257970+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":22,"pg_placement_num":22,"pg_placement_num_target":22,"pg_num_target":22,"pg_num_pending":22,"last_pg_merge_meta":{"source_pgid":"3.16","ready_epoch":1038,"last_epoch_started":1037,"last_epoch_clean":1037,"source_version":"0'0","target_version":"0'0"},"last_change":"1038","last_force_op_resend":"0","last_force_op_resend_prenautilus":"1038","last_force_op_resend_preluminous":"1036","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2021-03-15T22:17:54.562498+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2021-03-15T22:17:56.243806+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"5.16","ready_epoch":182,"last_epoch_started":181,"last_epoch_clean":181,"source_version":"0'0","target_version":"0'0"},"last_change":"1007","last_force_op_resend":"0","last_force_op_resend_prenautilus":"182","last_force_op_resend_preluminous":"728","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2021-03-15T22:17:58.730534+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"6.20","ready_epoch":1024,"last_epoch_started":1023,"last_epoch_clean":1023,"source_version":"0'0","target_version":"0'0"},"last_change":"1024","last_force_op_resend":"0","last_force_op_resend_prenautilus":"1024","last_force_op_resend_preluminous":"1022","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"unique_pool_0","create_time":"2021-03-15T22:20:03.377294+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":16,"pg_placement_num":16,"pg_placement_num_target":16,"pg_num_target":16,"pg_num_pending":16,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"1043","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":899,"snap_epoch":1043,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rados":{}}}],"osds":[{"osd":0,"uuid":"aa2209da-9bd7-4d7e-ae89-58ee8d1a16bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":1038,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6802","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6803","nonce":188228176}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6804","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6805","nonce":188228176}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6808","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6809","nonce":188228176}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6806","nonce":188228176},{"type":"v1","addr":"172.21.15.76:6807","nonce":188228176}]},"public_addr":"172.21.15.76:6803/188228176","cluster_addr":"172.21.15.76:6805/188228176","heartbeat_back_addr":"172.21.15.76:6809/188228176","heartbeat_front_addr":"172.21.15.76:6807/188228176","state":["exists","up"]},{"osd":1,"uuid":"97bd3ffc-15c8-4a6f-b37a-217813def526","up":1,"in":1,"weight":0.8248748779296875,"primary_affinity":0.999237060546875,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":1030,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6810","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6811","nonce":3924955423}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6812","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6813","nonce":3924955423}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6816","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6817","nonce":3924955423}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6814","nonce":3924955423},{"type":"v1","addr":"172.21.15.76:6815","nonce":3924955423}]},"public_addr":"172.21.15.76:6811/3924955423","cluster_addr":"172.21.15.76:6813/3924955423","heartbeat_back_addr":"172.21.15.76:6817/3924955423","heartbeat_front_addr":"172.21.15.76:6815/3924955423","state":["exists","up"]},{"osd":2,"uuid":"a7f5f115-e69e-47d9-b6c3-ff386ddfaa05","up":1,"in":1,"weight":1,"primary_affinity":0.53363037109375,"last_clean_begin":18,"last_clean_end":489,"up_from":681,"up_thru":972,"down_at":490,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6818","nonce":1676192618},{"type":"v1","addr":"172.21.15.76:6819","nonce":1676192618}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6820","nonce":1676192618},{"type":"v1","addr":"172.21.15.76:6821","nonce":1676192618}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6824","nonce":1676192618},{"type":"v1","addr":"172.21.15.76:6825","nonce":1676192618}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6822","nonce":1676192618},{"type":"v1","addr":"172.21.15.76:6823","nonce":1676192618}]},"public_addr":"172.21.15.76:6819/1676192618","cluster_addr":"172.21.15.76:6821/1676192618","heartbeat_back_addr":"172.21.15.76:6825/1676192618","heartbeat_front_addr":"172.21.15.76:6823/1676192618","state":["exists","up"]},{"osd":3,"uuid":"a0bb7f88-0114-47cb-a82c-d8fb002aac56","up":1,"in":1,"weight":0.7639923095703125,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":1034,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6826","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6827","nonce":3960968636}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6828","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6829","nonce":3960968636}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6832","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6833","nonce":3960968636}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.76:6830","nonce":3960968636},{"type":"v1","addr":"172.21.15.76:6831","nonce":3960968636}]},"public_addr":"172.21.15.76:6827/3960968636","cluster_addr":"172.21.15.76:6829/3960968636","heartbeat_back_addr":"172.21.15.76:6833/3960968636","heartbeat_front_addr":"172.21.15.76:6831/3960968636","state":["exists","up"]},{"osd":4,"uuid":"b650611c-6101-47e8-9ca9-94e02cf8b237","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":1028,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6800","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6801","nonce":37386122}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6802","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6803","nonce":37386122}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6806","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6807","nonce":37386122}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6804","nonce":37386122},{"type":"v1","addr":"172.21.15.125:6805","nonce":37386122}]},"public_addr":"172.21.15.125:6801/37386122","cluster_addr":"172.21.15.125:6803/37386122","heartbeat_back_addr":"172.21.15.125:6807/37386122","heartbeat_front_addr":"172.21.15.125:6805/37386122","state":["exists","up"]},{"osd":5,"uuid":"7d6ceff2-7093-4ab9-8b96-ade448093bf2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":38,"last_clean_end":539,"up_from":616,"up_thru":1045,"down_at":540,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6808","nonce":978017133},{"type":"v1","addr":"172.21.15.125:6809","nonce":978017133}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6810","nonce":978017133},{"type":"v1","addr":"172.21.15.125:6811","nonce":978017133}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6814","nonce":978017133},{"type":"v1","addr":"172.21.15.125:6815","nonce":978017133}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6812","nonce":978017133},{"type":"v1","addr":"172.21.15.125:6813","nonce":978017133}]},"public_addr":"172.21.15.125:6809/978017133","cluster_addr":"172.21.15.125:6811/978017133","heartbeat_back_addr":"172.21.15.125:6815/978017133","heartbeat_front_addr":"172.21.15.125:6813/978017133","state":["exists","up"]},{"osd":6,"uuid":"98b72022-a9a7-4ee2-89a1-cac037de7afe","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":45,"up_thru":1028,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6816","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6817","nonce":171378275}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6818","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6819","nonce":171378275}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6822","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6823","nonce":171378275}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6820","nonce":171378275},{"type":"v1","addr":"172.21.15.125:6821","nonce":171378275}]},"public_addr":"172.21.15.125:6817/171378275","cluster_addr":"172.21.15.125:6819/171378275","heartbeat_back_addr":"172.21.15.125:6823/171378275","heartbeat_front_addr":"172.21.15.125:6821/171378275","state":["exists","up"]},{"osd":7,"uuid":"f99ea650-b2dc-4d87-a44c-56a897edd614","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":1028,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6824","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6825","nonce":652003948}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6826","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6827","nonce":652003948}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6830","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6831","nonce":652003948}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.125:6828","nonce":652003948},{"type":"v1","addr":"172.21.15.125:6829","nonce":652003948}]},"public_addr":"172.21.15.125:6825/652003948","cluster_addr":"172.21.15.125:6827/652003948","heartbeat_back_addr":"172.21.15.125:6831/652003948","heartbeat_front_addr":"172.21.15.125:6829/652003948","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:35:59.419042+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:36:17.150537+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:36:33.323713+0000","dead_epoch":490},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:35:45.679555+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:36:05.116068+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:36:28.333468+0000","dead_epoch":540},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:36:40.331609+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-03-15T22:35:56.499151+0000","dead_epoch":0}],"pg_upmap":[{"pgid":"3.9","osds":[5,4,6]},{"pgid":"7.3","osds":[1,6,2]}],"pg_upmap_items":[{"pgid":"2.2","mappings":[{"from":5,"to":7},{"from":6,"to":0}]},{"pgid":"5.10","mappings":[{"from":7,"to":3}]},{"pgid":"5.17","mappings":[{"from":1,"to":6}]},{"pgid":"5.1b","mappings":[{"from":0,"to":3}]},{"pgid":"5.1f","mappings":[{"from":4,"to":3}]},{"pgid":"6.f","mappings":[{"from":4,"to":0}]}],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.76:6800/2604045245":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/4087324200":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/4141143304":"2021-03-16T22:14:00.771702+0000","172.21.15.76:6801/2604045245":"2021-03-16T22:14:00.771702+0000","172.21.15.76:0/2779879991":"2021-03-16T22:13:41.603652+0000","172.21.15.76:6801/2764179881":"2021-03-16T22:13:41.603652+0000","172.21.15.76:6800/2764179881":"2021-03-16T22:13:41.603652+0000","172.21.15.76:0/2483102204":"2021-03-16T22:13:41.603652+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":7,"snaps":[{"begin":843,"length":1}]}],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-03-15T22:36:44.049 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:44.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:44.051 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:44.152 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:44.174 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:44.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:44.176 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:44.277 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:44.303 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:44.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:44.304 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:44.405 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:44.427 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:44.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:44.429 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:44.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:44.542 INFO:teuthology.orchestra.run.smithi076.stderr:noscrub is unset 2021-03-15T22:36:44.558 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:44.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:44.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:44.660 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:44.670 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[56514]: cluster 2021-03-15T22:36:43.239420+0000 mgr.y (mgr.14144) 1136 : cluster [DBG] pgmap v1717: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 10 op/s 2021-03-15T22:36:44.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[56514]: audit 2021-03-15T22:36:44.044122+0000 mon.a (mon.0) 2603 : audit [DBG] from='client.? 172.21.15.76:0/1293723927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:36:44.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[56514]: audit 2021-03-15T22:36:44.131667+0000 mon.c (mon.1) 86 : audit [INF] from='client.? 172.21.15.76:0/4112103213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:44.671 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[56514]: audit 2021-03-15T22:36:44.132009+0000 mon.a (mon.0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:44.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[62156]: cluster 2021-03-15T22:36: 2021-03-15T22:36:44.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[62156]: 43.239420+0000 mgr.y (mgr.14144) 1136 : cluster [DBG] pgmap v1717: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 10 op/s 2021-03-15T22:36:44.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[62156]: audit 2021-03-15T22:36:44.044122+0000 2021-03-15T22:36:44.672 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[62156]: mon.a (mon.0) 2603 : audit [DBG] from='client.? 172.21.15.76:0/1293723927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:36:44.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[62156]: audit 2021-03-15T22:36:44.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[62156]: 2021-03-15T22:36:44.131667+0000 mon.c 2021-03-15T22:36:44.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[62156]: (mon.1) 86 : audit [INF] from='client.? 172.21.15.76:0/4112103213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:44.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[62156]: audit 2021-03-15T22:36:44.132009+0000 mon.a 2021-03-15T22:36:44.673 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:44 smithi076 conmon[62156]: (mon.0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:44.686 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:44.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:44.687 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:44 smithi125 conmon[56739]: cluster 2021-03-15T22:36:43.239420+0000 mgr.y (mgr.14144) 1136 : cluster [DBG] pgmap v1717: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 10 op/s 2021-03-15T22:36:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:44 smithi125 conmon[56739]: audit 2021-03-15T22:36:44.044122+0000 mon.a (mon.0) 2603 : audit [DBG] from='client.? 172.21.15.76:0/1293723927' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2021-03-15T22:36:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:44 smithi125 conmon[56739]: audit 2021-03-15T22: 2021-03-15T22:36:44.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:44 smithi125 conmon[56739]: 36:44.131667+0000 mon.c (mon.1) 86 : audit [INF] from='client.? 172.21.15.76:0/4112103213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:44 smithi125 conmon[56739]: audit 2021-03-15T 2021-03-15T22:36:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:44 smithi125 conmon[56739]: 22:36:44.132009+0000 mon.a (mon. 2021-03-15T22:36:44.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:44 smithi125 conmon[56739]: 0) 2604 : audit [INF] from='client.? ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2021-03-15T22:36:44.788 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:44.814 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:44.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:44.816 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:44.856 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_ops_in_flight 2021-03-15T22:36:44.918 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:44.940 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:44.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:44.941 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:45.043 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:45.066 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:45.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:45.068 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:45.169 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:45.194 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:45.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:45.196 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:45.297 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:45.321 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:45.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:45.323 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:45.424 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:45.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:45.448 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:45.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:45.550 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:45.572 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:45.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:45.574 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:45.675 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:45.699 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:45.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:45.701 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:45 smithi125 conmon[56739]: cluster 2021-03-15T22:36:44.523993+0000 mon.a (mon.0) 2605 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:45 smithi125 conmon[56739]: audit 2021-03-15T22:36:45.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:45 smithi125 conmon[56739]: 2021-03-15T22:36:44.537641+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:36:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:45 smithi125 conmon[56739]: cluster 2021-03-15T22:36:44.537727+0000 mon.a (mon.0) 2607 : cluster [DBG] osdmap e1047: 8 total, 8 up, 8 in 2021-03-15T22:36:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:45 smithi125 conmon[56739]: cluster 2021-03-15T22:36:44.551566+0000 osd.2 (osd.2) 56 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:36:45.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:45 smithi125 conmon[56739]: cluster 2021-03-15T22:36:45.368320+0000 mon.a (mon.0) 2608 : cluster [DBG] osdmap e1048: 8 total, 8 up, 8 in 2021-03-15T22:36:45.802 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:45.820 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[56514]: cluster 2021-03-15T22:36: 2021-03-15T22:36:45.820 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[56514]: 44.523993+0000 mon.a (mon.0) 2605 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:45.821 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[56514]: audit 2021-03-15T22:36:44.537641+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:36:45.821 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[56514]: cluster 2021-03-15T22:36:44.537727+0000 mon.a (mon.0) 2607 : cluster [DBG] osdmap e1047: 8 total, 8 up, 8 in 2021-03-15T22:36:45.821 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[56514]: cluster 2021-03-15T22:36:44.551566+0000 osd.2 (osd.2) 56 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:36:45.821 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[56514]: cluster 2021-03-15T22:36:45.368320+0000 mon.a (mon.0) 2608 : cluster [DBG] osdmap e1048: 8 total, 8 up, 8 in 2021-03-15T22:36:45.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[62156]: cluster 2021-03-15T22:36:44.523993+0000 mon.a (mon.0) 2605 : cluster [WRN] Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2021-03-15T22:36:45.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[62156]: 2021-03-15T22:36:45.822 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[62156]: audit 2021-03-15T22:36:44.537641+0000 mon.a (mon.0) 2606 : audit [INF] from='client.? ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2021-03-15T22:36:45.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[62156]: cluster 2021-03-15T22:36:44.537727+0000 mon.a (mon.0) 2607 : cluster [DBG] osdmap e1047: 8 total, 8 up, 8 in 2021-03-15T22:36:45.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[62156]: cluster 2021-03-15T22:36:44.551566+0000 osd.2 (osd.2) 56 : cluster [DBG] 2.1 scrub ok 2021-03-15T22:36:45.823 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:45 smithi076 conmon[62156]: cluster 2021-03-15T22:36:45.368320+0000 mon.a (mon.0) 2608 : cluster [DBG] osdmap e1048: 8 total, 8 up, 8 in 2021-03-15T22:36:45.826 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:45.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:45.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:45.928 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:45.951 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:45.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:45.953 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:46.032 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:46.054 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:46.061 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:36:46.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:46.080 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:46.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:46.183 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:46.210 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:46.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:46.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:46.313 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:46.337 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:46.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:46.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:46.440 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:46.466 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:46.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:46.468 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:46.510 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:36:46.540 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:36:46.558 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:36:46.569 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:46.575 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:36:46.592 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:46.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:46.593 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:46.594 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:36:46.612 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:36:46.630 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:46 smithi076 conmon[56514]: cluster 2021-03-15T22:36:45.239846+0000 mgr.y (mgr.14144) 1137 : cluster [DBG] pgmap v1719: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.4 MiB/s wr, 7 op/s 2021-03-15T22:36:46.630 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:46 smithi076 conmon[56514]: cluster 2021-03-15T22:36:45.344258+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.a scrub ok 2021-03-15T22:36:46.631 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:46 smithi076 conmon[56514]: cluster 2021-03-15T22:36:45.550858+0000 osd.1 (osd.1) 198 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:36:46.631 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:46 smithi076 conmon[56514]: audit 2021-03-15T22:36:46.489071+0000 mon.a (mon.0) 2609 : audit [DBG] from='client.? 172.21.15.76:0/3307392203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:46.631 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:46 smithi076 conmon[62156]: cluster 2021-03-15T22:36:45.239846+0000 mgr.y (mgr.14144) 1137 : cluster [DBG] pgmap v1719: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.4 MiB/s wr, 7 op/s 2021-03-15T22:36:46.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:46 smithi076 conmon[62156]: cluster 2021-03-15T22:36:45.344258+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.a scrub ok 2021-03-15T22:36:46.632 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:46 smithi076 conmon[62156]: cluster 2021-03-15T22:36:45.550858+0000 osd.1 (osd.1) 198 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:36:46.633 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:46 smithi076 conmon[62156]: audit 2021-03-15T22:36:46.489071+0000 mon.a (mon.0) 2609 : audit [DBG] from='client.? 172.21.15.76:0/3307392203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:46.633 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:36:46.649 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:36:46.650 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'true' 2021-03-15T22:36:46.650 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'true' 2021-03-15T22:36:46.651 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'true' 2021-03-15T22:36:46.651 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'true' 2021-03-15T22:36:46.651 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'true' 2021-03-15T22:36:46.651 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'true' 2021-03-15T22:36:46.652 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'true' 2021-03-15T22:36:46.652 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'true' 2021-03-15T22:36:46.695 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:46.721 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:46.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:46.722 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:46.746 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:46.823 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:46.846 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:46.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:46.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:46.948 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:46.971 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:46.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:46.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:47.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:46 smithi125 conmon[56739]: cluster 2021-03-15T22:36:45.239846+0000 mgr.y (mgr.14144) 1137 : cluster [DBG] pgmap v1719: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.8 KiB/s rd, 2.4 MiB/s wr, 7 op/s 2021-03-15T22:36:47.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:46 smithi125 conmon[56739]: cluster 2021-03-15T22:36:45.344258+0000 osd.4 (osd.4) 208 : cluster [DBG] 5.a scrub ok 2021-03-15T22:36:47.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:46 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:36:47.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:46 smithi125 conmon[56739]: 03-15T22:36:45.550858+0000 osd.1 (osd.1 2021-03-15T22:36:47.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:46 smithi125 conmon[56739]: ) 198 : cluster [DBG] 6.1 scrub ok 2021-03-15T22:36:47.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:46 smithi125 conmon[56739]: audit 2021-03 2021-03-15T22:36:47.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:46 smithi125 conmon[56739]: -15T22:36:46.489071+0000 2021-03-15T22:36:47.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:46 smithi125 conmon[56739]: mon.a (mon.0) 2609 : audit 2021-03-15T22:36:47.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:46 smithi125 conmon[56739]: [DBG] from='client.? 172.21.15.76:0/3307392203' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:47.074 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:47.096 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:47.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:47.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:47.199 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:47.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:47.222 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:47.222 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:47.324 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:47.345 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:47.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:47.347 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:47.448 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:47.470 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:47.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:47.471 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:47.566 INFO:tasks.thrashosds.thrasher:in_osds: [1, 4, 7, 0, 3, 5, 6, 2] out_osds: [] dead_osds: [] live_osds: [0, 1, 3, 4, 6, 7, 5, 2] 2021-03-15T22:36:47.567 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 2021-03-15T22:36:47.567 INFO:tasks.thrashosds.thrasher:fixing pg num pool default.rgw.meta 2021-03-15T22:36:47.568 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph pg dump --format=json 2021-03-15T22:36:47.573 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:47.634 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:47.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:47.736 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:47.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: cluster 2021-03-15T22:36:45.831914+0000 osd.7 2021-03-15T22:36:47.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: (osd.7) 197 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:36:47.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:36:47.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: 03-15T22:36:46.065357+0000 osd.3 (osd.3) 178 : 2021-03-15T22:36:47.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:47.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: cluster 2021-03-15T22:36:47.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: 2021-03-15T22:36:46.068224 2021-03-15T22:36:47.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: +0000 osd.3 (osd.3) 179 : cluster [DBG] 2021-03-15T22:36:47.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: purged_snaps scrub ok 2021-03-15T22:36:47.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: cluster 2021-03-15T22:36:46.541268 2021-03-15T22:36:47.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: +0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e1049: 8 total, 8 up, 8 in 2021-03-15T22:36:47.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: 2021-03-15T22:36:47.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: cluster 2021-03-15T22:36:47.834 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: 2021-03-15T22:36:46.565269+0000 osd.1 (osd.1) 199 : 2021-03-15T22:36:47.834 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[62156]: cluster [DBG] 7.b scrub ok 2021-03-15T22:36:47.834 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[56514]: cluster 2021-03-15T22:36:45.831914+0000 osd.7 ( 2021-03-15T22:36:47.834 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[56514]: osd.7) 197 : cluster [DBG] 4.17 scrub ok 2021-03-15T22:36:47.835 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[56514]: cluster 2021-03-15T22:36:46.065357+0000 osd.3 (osd.3) 178 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:47.835 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[56514]: cluster 2021-03-15T22:36:46.068224+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:47.835 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[56514]: cluster 2021-03-15T22:36:46.541268+0000 mon.a (mon.0) 2610 : cluster [DBG] osdmap e1049: 8 total, 8 up, 8 in 2021-03-15T22:36:47.835 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:47 smithi076 conmon[56514]: cluster 2021-03-15T22:36:46.565269+0000 osd.1 (osd.1) 199 : cluster [DBG] 7.b scrub ok 2021-03-15T22:36:47.847 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd unset nodeep-scrub 2021-03-15T22:36:47.860 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:47.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:47.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:47.963 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:47.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:47.986 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:47.987 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:48.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:47 smithi125 conmon[56739]: cluster 2021-03-15T22:36:45.831914+0000 osd.7 (osd.7) 197 : cluster [DBG] 2021-03-15T22:36:48.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:47 smithi125 conmon[56739]: 4.17 scrub ok 2021-03-15T22:36:48.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:47 smithi125 conmon[56739]: cluster 2021-03-15T22:36:46.065357+0000 osd.3 (osd.3) 178 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:48.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:47 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:36:48.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:47 smithi125 conmon[56739]: 03-15T22:36:46.068224+0000 osd.3 (osd.3) 179 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:48.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:47 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:36:48.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:47 smithi125 conmon[56739]: 22:36:46.541268+0000 mon.a (mon.0 2021-03-15T22:36:48.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:47 smithi125 conmon[56739]: ) 2610 : cluster [DBG] osdmap e1049: 8 total, 8 up, 8 in 2021-03-15T22:36:48.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:47 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:36:48.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:47 smithi125 conmon[56739]: 03-15T22:36:46.565269+0000 osd.1 (osd 2021-03-15T22:36:48.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:47 smithi125 conmon[56739]: .1) 199 : cluster [DBG] 7.b scrub ok 2021-03-15T22:36:48.088 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:48.114 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:48.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:48.116 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:48.217 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:48.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:48.241 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:48.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:48.245 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:48.343 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:48.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:48.368 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:48.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:48.470 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:48.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:48.493 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:48.493 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:48.595 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:48.617 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:48.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:48.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:48.719 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:48.745 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:48.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:48.747 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:48.848 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:48.870 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:48.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:48.872 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:48.973 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:48.986 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2021-03-15T22:36:48.986 INFO:teuthology.orchestra.run.smithi076.stdout: "ops": [], 2021-03-15T22:36:48.987 INFO:teuthology.orchestra.run.smithi076.stdout: "num_ops": 0 2021-03-15T22:36:48.987 INFO:teuthology.orchestra.run.smithi076.stdout:} 2021-03-15T22:36:48.988 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:48 smithi076 conmon[56514]: cluster 2021-03-15T22:36:47.240331+0000 mgr.y (mgr.14144) 1138 : cluster [DBG] pgmap v1722: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-15T22:36:48.989 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:48 smithi076 conmon[56514]: cluster 2021-03-15T22:36:47.636239+0000 osd.2 (osd.2) 57 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:36:48.989 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:48 smithi076 conmon[62156]: cluster 2021-03-15T22:36:47.240331+0000 mgr.y (mgr.14144) 1138 : cluster [DBG] pgmap v1722: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-15T22:36:48.990 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:48 smithi076 conmon[62156]: cluster 2021-03-15T22:36:47.636239+0000 osd.2 (osd.2) 57 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:36:48.999 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:49.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:49.001 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:49.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:48 smithi125 conmon[56739]: cluster 2021-03-15T22:36:47.240331+0000 mgr.y (mgr.14144) 1138 : cluster 2021-03-15T22:36:49.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:48 smithi125 conmon[56739]: [DBG] pgmap v1722: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.1 KiB/s rd, 1 op/s 2021-03-15T22:36:49.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:48 smithi125 conmon[56739]: cluster 2021-03-15T22:36:47.636239+0000 osd.2 (osd.2) 57 : cluster [DBG] 6.14 scrub ok 2021-03-15T22:36:49.102 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:49.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:49.131 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:49.131 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:49.233 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:49.255 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:49.256 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:49.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:49.379 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:49.381 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:49.482 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:49.505 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:49.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:49.506 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:49.608 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:49.634 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:49.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:49.635 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:49.683 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[56514]: cluster 2021-03-15T22:36:47.829890+0000 2021-03-15T22:36:49.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[56514]: osd.7 (osd.7) 198 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:36:49.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[56514]: cluster 2021-03-15T22:36:48.012434+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:36:49.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[56514]: cluster 2021-03-15T22:36:48.136801+0000 osd.3 (osd.3) 180 : cluster [DBG] 6.b scrub ok 2021-03-15T22:36:49.684 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[56514]: cluster 2021-03-15T22:36:48.228758+0000 osd.5 (osd.5) 68 : cluster [DBG] 6.a scrub ok 2021-03-15T22:36:49.685 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[56514]: cluster 2021-03-15T22:36:48.578763+0000 osd.1 (osd.1) 200 : cluster [DBG] 1.3 scrub ok 2021-03-15T22:36:49.685 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[62156]: cluster 2021-03-15T22:36:47.829890+0000 osd.7 (osd. 2021-03-15T22:36:49.685 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[62156]: 7) 198 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:36:49.686 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[62156]: cluster 2021-03-15T22:36:48.012434+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:36:49.686 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[62156]: cluster 2021-03-15T22:36:48.136801+0000 osd.3 (osd.3) 180 : cluster [DBG] 6.b scrub ok 2021-03-15T22:36:49.686 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[62156]: cluster 2021-03-15T22:36:48.228758+0000 osd.5 (osd.5) 68 : cluster [DBG] 6.a scrub ok 2021-03-15T22:36:49.686 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:49 smithi076 conmon[62156]: cluster 2021-03-15T22:36:48.578763+0000 osd.1 (osd.1) 200 : cluster [DBG] 1.3 scrub ok 2021-03-15T22:36:49.736 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:49.762 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:49.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:49.864 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:49.870 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2021-03-15T22:36:49.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:49.891 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:49.891 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:49.992 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:50.018 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:50.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:50.021 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:50.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: cluster 2021-03-15T22:36:47.829890+0000 osd.7 (osd.7) 198 : cluster [DBG] 6.5 scrub ok 2021-03-15T22:36:50.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: cluster 2021-03-15T22:36:48.012434+0000 osd.0 (osd.0) 170 : cluster [DBG] 5.5 scrub ok 2021-03-15T22:36:50.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: cluster 2021-03-15T22:36:48.136801+0000 osd.3 (osd.3) 180 2021-03-15T22:36:50.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: : cluster [DBG] 6.b scrub ok 2021-03-15T22:36:50.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: cluster 2021-03-15T22:36:50.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: 2021-03-15T22:36:48.228758+0000 2021-03-15T22:36:50.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: osd.5 (osd.5) 68 : cluster [DBG] 2021-03-15T22:36:50.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: 6.a scrub ok 2021-03-15T22:36:50.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:36:50.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: 15T22:36:48.578763+0000 2021-03-15T22:36:50.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: osd.1 (osd.1) 200 : cluster [DBG] 2021-03-15T22:36:50.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:49 smithi125 conmon[56739]: 1.3 scrub ok 2021-03-15T22:36:50.122 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:50.148 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:50.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:50.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:50.251 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:50.273 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:50.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:50.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:50.289 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:50.313 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_blocked_ops 2021-03-15T22:36:50.377 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:50.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:50.405 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:50.405 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:50.506 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:50.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:50.529 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:50.529 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:50.631 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:50.657 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:50.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:50.659 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:50.723 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:50.760 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:50.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:50.785 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:50.785 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:50.786 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:36:50 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:36:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:50 smithi125 conmon[56739]: cluster 2021-03-15T22:36:49.240789+0000 mgr.y (mgr.14144) 1139 : 2021-03-15T22:36:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:50 smithi125 conmon[56739]: cluster [DBG] pgmap v1723: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:36:50.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:50 smithi125 conmon[56739]: cluster 2021-03-15T22:36:49.529919+0000 osd.1 (osd.1) 201 : cluster [DBG] 5.c scrub ok 2021-03-15T22:36:50.798 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:50.875 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:50 smithi076 conmon[56514]: cluster 2021-03-15T22:36:49.240789+0000 mgr.y (mgr.14144) 1139 : cluster [DBG] pgmap v1723: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:36:50.876 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:50 smithi076 conmon[56514]: cluster 2021-03-15T22:36:49.529919+0000 osd.1 (osd.1) 201 : cluster [DBG] 5.c scrub ok 2021-03-15T22:36:50.876 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:50 smithi076 conmon[62156]: cluster 2021-03-15T22:36:49.240789+0000 mgr.y 2021-03-15T22:36:50.876 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:50 smithi076 conmon[62156]: (mgr.14144) 1139 : cluster [DBG] pgmap v1723: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:36:50.877 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:50 smithi076 conmon[62156]: cluster 2021-03-15T22:36:49.529919+0000 osd.1 (osd.1) 201 : cluster [DBG] 5.c scrub ok 2021-03-15T22:36:50.884 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:36:50.887 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:50.888 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:36:50.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:50.952 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:50.953 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:51.054 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:51.080 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:51.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:51.082 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:51.183 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:51.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:51.208 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:51.209 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:51.310 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:51.336 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:51.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:51.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:51.439 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:51.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:51.466 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:51.466 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:51.568 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:51.590 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:51.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:51.591 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:51.692 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:51.714 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:51.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:51.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:51.796 INFO:teuthology.orchestra.run.smithi076.stdout: 2021-03-15T22:36:51.799 INFO:teuthology.orchestra.run.smithi076.stderr:dumped all 2021-03-15T22:36:51.818 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:51.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:51.842 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:51.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:51.943 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[62156]: cluster 2021-03-15T22:36:50.117032+0000 osd.3 (osd.3) 181 2021-03-15T22:36:51.943 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[62156]: : cluster [DBG] 4.1d scrub ok 2021-03-15T22:36:51.943 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[62156]: cluster 2021-03-15T22:36:50.238937+0000 osd.5 (osd.5) 69 : cluster [DBG] 6.d scrub ok 2021-03-15T22:36:51.943 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[62156]: audit 2021-03-15T22:36:50.359324+0000 mgr.y (mgr.14144) 1140 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:51.944 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[62156]: cluster 2021-03-15T22:36:50.391156+0000 osd.4 (osd.4) 209 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:36:51.944 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[62156]: audit 2021-03-15T22:36:50.866423+0000 mgr.y (mgr.14144) 1141 : audit [DBG] from='client.25678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:51.944 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[56514]: cluster 2021-03-15T22:36:50.117032+0000 osd.3 2021-03-15T22:36:51.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[56514]: (osd.3) 181 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:36:51.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[56514]: cluster 2021-03-15T22:36:50.238937+0000 osd.5 (osd.5) 69 : cluster [DBG] 6.d scrub ok 2021-03-15T22:36:51.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[56514]: audit 2021-03-15T22:36:50.359324+0000 mgr.y (mgr.14144) 1140 : audit [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:51.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[56514]: cluster 2021-03-15T22:36:50.391156+0000 osd.4 (osd.4) 209 : cluster [DBG] 6.1a scrub ok 2021-03-15T22:36:51.946 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:51 smithi076 conmon[56514]: audit 2021-03-15T22:36:50.866423+0000 mgr.y (mgr.14144) 1141 : audit [DBG] from='client.25678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:51.946 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:51.969 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:51.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:51.970 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:52.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:51 smithi125 conmon[56739]: cluster 2021-03-15T22:36:50.117032+0000 osd.3 (osd.3) 181 : cluster [DBG] 4.1d scrub ok 2021-03-15T22:36:52.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:51 smithi125 conmon[56739]: cluster 2021-03-15T22:36:50.238937+0000 osd.5 (osd.5) 69 : cluster [DBG] 6.d scrub ok 2021-03-15T22:36:52.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:51 smithi125 conmon[56739]: audit 2021-03-15T22:36:50.359324+0000 mgr.y (mgr.14144) 1140 : audit 2021-03-15T22:36:52.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:51 smithi125 conmon[56739]: [DBG] from='client.34364 -' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2021-03-15T22:36:52.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:51 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:36:52.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:51 smithi125 conmon[56739]: 03-15T22:36:50.391156+0000 osd.4 (osd.4) 209 2021-03-15T22:36:52.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:51 smithi125 conmon[56739]: : cluster [DBG] 6.1a scrub ok 2021-03-15T22:36:52.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:51 smithi125 conmon[56739]: audit 2021-03-15 2021-03-15T22:36:52.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:51 smithi125 conmon[56739]: T22:36:50.866423+0000 mgr.y ( 2021-03-15T22:36:52.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:51 smithi125 conmon[56739]: mgr.14144) 1141 : audit [DBG] from='client.25678 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:52.072 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:52.098 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:52.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:52.099 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:52.201 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:52.227 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:52.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:52.228 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:52.329 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:52.352 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:52.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:52.354 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:52.449 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1723,"stamp":"2021-03-15T22:36:49.240530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":831847580,"num_objects":680,"num_object_clones":314,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":680,"num_whiteouts":12,"num_read":10050,"num_read_kb":7663020,"num_write":6243,"num_write_kb":2092978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5900,"num_bytes_recovered":2230124932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25690,"ondisk_log_size":25690,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":445,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2197972,"kb_used_data":1804948,"kb_used_omap":273,"kb_used_meta":392686,"kb_avail":747763244,"statfs":{"total":767960285184,"available":765709561856,"internally_reserved":0,"allocated":1848266752,"data_stored":2546537862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":402111354},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001340"},"pg_stats":[{"pgid":"5.19","version":"317'7","reported_seq":"417","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.548919+0000","last_change":"2021-03-15T22:36:18.276167+0000","last_active":"2021-03-15T22:36:45.548919+0000","last_peered":"2021-03-15T22:36:45.548919+0000","last_clean":"2021-03-15T22:36:45.548919+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:36:45.548919+0000","last_undegraded":"2021-03-15T22:36:45.548919+0000","last_fullsized":"2021-03-15T22:36:45.548919+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:36:18.276099+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:36:18.276099+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1064","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367139+0000","last_change":"2021-03-15T22:34:10.331764+0000","last_active":"2021-03-15T22:36:45.367139+0000","last_peered":"2021-03-15T22:36:45.367139+0000","last_clean":"2021-03-15T22:36:45.367139+0000","last_became_active":"2021-03-15T22:34:10.331424+0000","last_became_peered":"2021-03-15T22:34:10.331424+0000","last_unstale":"2021-03-15T22:36:45.367139+0000","last_undegraded":"2021-03-15T22:36:45.367139+0000","last_fullsized":"2021-03-15T22:36:45.367139+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.564464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.564464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"1147","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367004+0000","last_change":"2021-03-15T22:35:14.434026+0000","last_active":"2021-03-15T22:36:45.367004+0000","last_peered":"2021-03-15T22:36:45.367004+0000","last_clean":"2021-03-15T22:36:45.367004+0000","last_became_active":"2021-03-15T22:33:04.345393+0000","last_became_peered":"2021-03-15T22:33:04.345393+0000","last_unstale":"2021-03-15T22:36:45.367004+0000","last_undegraded":"2021-03-15T22:36:45.367004+0000","last_fullsized":"2021-03-15T22:36:45.367004+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:35:14.433961+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:35:14.433961+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"356","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.366872+0000","last_change":"2021-03-15T22:36:08.386295+0000","last_active":"2021-03-15T22:36:45.366872+0000","last_peered":"2021-03-15T22:36:45.366872+0000","last_clean":"2021-03-15T22:36:45.366872+0000","last_became_active":"2021-03-15T22:33:04.345518+0000","last_became_peered":"2021-03-15T22:33:04.345518+0000","last_unstale":"2021-03-15T22:36:45.366872+0000","last_undegraded":"2021-03-15T22:36:45.366872+0000","last_fullsized":"2021-03-15T22:36:45.366872+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:08.386208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:08.386208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"131","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367259+0000","last_change":"2021-03-15T22:34:43.345093+0000","last_active":"2021-03-15T22:36:45.367259+0000","last_peered":"2021-03-15T22:36:45.367259+0000","last_clean":"2021-03-15T22:36:45.367259+0000","last_became_active":"2021-03-15T22:34:43.344764+0000","last_became_peered":"2021-03-15T22:34:43.344764+0000","last_unstale":"2021-03-15T22:36:45.367259+0000","last_undegraded":"2021-03-15T22:36:45.367259+0000","last_fullsized":"2021-03-15T22:36:45.367259+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:08.652741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.652741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"197","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367423+0000","last_change":"2021-03-15T22:36:08.831621+0000","last_active":"2021-03-15T22:36:45.367423+0000","last_peered":"2021-03-15T22:36:45.367423+0000","last_clean":"2021-03-15T22:36:45.367423+0000","last_became_active":"2021-03-15T22:34:55.346148+0000","last_became_peered":"2021-03-15T22:34:55.346148+0000","last_unstale":"2021-03-15T22:36:45.367423+0000","last_undegraded":"2021-03-15T22:36:45.367423+0000","last_fullsized":"2021-03-15T22:36:45.367423+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:36:08.831557+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:36:08.831557+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":5340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"896","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.514866+0000","last_change":"2021-03-15T22:35:16.848985+0000","last_active":"2021-03-15T22:36:41.514866+0000","last_peered":"2021-03-15T22:36:41.514866+0000","last_clean":"2021-03-15T22:36:41.514866+0000","last_became_active":"2021-03-15T22:34:03.570306+0000","last_became_peered":"2021-03-15T22:34:03.570306+0000","last_unstale":"2021-03-15T22:36:41.514866+0000","last_undegraded":"2021-03-15T22:36:41.514866+0000","last_fullsized":"2021-03-15T22:36:41.514866+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:16.848918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.848918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"517","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.548337+0000","last_change":"2021-03-15T22:35:49.146403+0000","last_active":"2021-03-15T22:36:44.548337+0000","last_peered":"2021-03-15T22:36:44.548337+0000","last_clean":"2021-03-15T22:36:44.548337+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:36:44.548337+0000","last_undegraded":"2021-03-15T22:36:44.548337+0000","last_fullsized":"2021-03-15T22:36:44.548337+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.146337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.146337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"1203","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367324+0000","last_change":"2021-03-15T22:36:21.176476+0000","last_active":"2021-03-15T22:36:45.367324+0000","last_peered":"2021-03-15T22:36:45.367324+0000","last_clean":"2021-03-15T22:36:45.367324+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:36:45.367324+0000","last_undegraded":"2021-03-15T22:36:45.367324+0000","last_fullsized":"2021-03-15T22:36:45.367324+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:36:21.176400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:21.176400+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"331","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.515756+0000","last_change":"2021-03-15T22:35:12.404980+0000","last_active":"2021-03-15T22:36:41.515756+0000","last_peered":"2021-03-15T22:36:41.515756+0000","last_clean":"2021-03-15T22:36:41.515756+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:36:41.515756+0000","last_undegraded":"2021-03-15T22:36:41.515756+0000","last_fullsized":"2021-03-15T22:36:41.515756+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.404897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.404897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"689","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.514353+0000","last_change":"2021-03-15T22:36:22.138406+0000","last_active":"2021-03-15T22:36:41.514353+0000","last_peered":"2021-03-15T22:36:41.514353+0000","last_clean":"2021-03-15T22:36:41.514353+0000","last_became_active":"2021-03-15T22:34:43.344401+0000","last_became_peered":"2021-03-15T22:34:43.344401+0000","last_unstale":"2021-03-15T22:36:41.514353+0000","last_undegraded":"2021-03-15T22:36:41.514353+0000","last_fullsized":"2021-03-15T22:36:41.514353+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:22.138306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.138306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"1194","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367750+0000","last_change":"2021-03-15T22:35:45.410540+0000","last_active":"2021-03-15T22:36:45.367750+0000","last_peered":"2021-03-15T22:36:45.367750+0000","last_clean":"2021-03-15T22:36:45.367750+0000","last_became_active":"2021-03-15T22:34:55.335550+0000","last_became_peered":"2021-03-15T22:34:55.335550+0000","last_unstale":"2021-03-15T22:36:45.367750+0000","last_undegraded":"2021-03-15T22:36:45.367750+0000","last_fullsized":"2021-03-15T22:36:45.367750+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:35:45.410466+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.410466+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"1009","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549225+0000","last_change":"2021-03-15T22:35:10.068109+0000","last_active":"2021-03-15T22:36:45.549225+0000","last_peered":"2021-03-15T22:36:45.549225+0000","last_clean":"2021-03-15T22:36:45.549225+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:36:45.549225+0000","last_undegraded":"2021-03-15T22:36:45.549225+0000","last_fullsized":"2021-03-15T22:36:45.549225+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:35:10.068030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.068030+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1071","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368017+0000","last_change":"2021-03-15T22:34:41.614750+0000","last_active":"2021-03-15T22:36:45.368017+0000","last_peered":"2021-03-15T22:36:45.368017+0000","last_clean":"2021-03-15T22:36:45.368017+0000","last_became_active":"2021-03-15T22:33:04.329078+0000","last_became_peered":"2021-03-15T22:33:04.329078+0000","last_unstale":"2021-03-15T22:36:45.368017+0000","last_undegraded":"2021-03-15T22:36:45.368017+0000","last_fullsized":"2021-03-15T22:36:45.368017+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.614684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.614684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"611","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549243+0000","last_change":"2021-03-15T22:36:22.338276+0000","last_active":"2021-03-15T22:36:45.549243+0000","last_peered":"2021-03-15T22:36:45.549243+0000","last_clean":"2021-03-15T22:36:45.549243+0000","last_became_active":"2021-03-15T22:34:55.350325+0000","last_became_peered":"2021-03-15T22:34:55.350325+0000","last_unstale":"2021-03-15T22:36:45.549243+0000","last_undegraded":"2021-03-15T22:36:45.549243+0000","last_fullsized":"2021-03-15T22:36:45.549243+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:36:22.338202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.338202+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1102","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367620+0000","last_change":"2021-03-15T22:35:42.445978+0000","last_active":"2021-03-15T22:36:45.367620+0000","last_peered":"2021-03-15T22:36:45.367620+0000","last_clean":"2021-03-15T22:36:45.367620+0000","last_became_active":"2021-03-15T22:34:55.338001+0000","last_became_peered":"2021-03-15T22:34:55.338001+0000","last_unstale":"2021-03-15T22:36:45.367620+0000","last_undegraded":"2021-03-15T22:36:45.367620+0000","last_fullsized":"2021-03-15T22:36:45.367620+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:42.445911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.445911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"886","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.369605+0000","last_change":"2021-03-15T22:35:20.025959+0000","last_active":"2021-03-15T22:36:45.369605+0000","last_peered":"2021-03-15T22:36:45.369605+0000","last_clean":"2021-03-15T22:36:45.369605+0000","last_became_active":"2021-03-15T22:33:04.336804+0000","last_became_peered":"2021-03-15T22:33:04.336804+0000","last_unstale":"2021-03-15T22:36:45.369605+0000","last_undegraded":"2021-03-15T22:36:45.369605+0000","last_fullsized":"2021-03-15T22:36:45.369605+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.025892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"497","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368962+0000","last_change":"2021-03-15T22:33:15.617975+0000","last_active":"2021-03-15T22:36:45.368962+0000","last_peered":"2021-03-15T22:36:45.368962+0000","last_clean":"2021-03-15T22:36:45.368962+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:36:45.368962+0000","last_undegraded":"2021-03-15T22:36:45.368962+0000","last_fullsized":"2021-03-15T22:36:45.368962+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:33:15.617891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.617891+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"891'22","reported_seq":"630","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.755398+0000","last_change":"2021-03-15T22:35:38.710708+0000","last_active":"2021-03-15T22:36:44.755398+0000","last_peered":"2021-03-15T22:36:44.755398+0000","last_clean":"2021-03-15T22:36:44.755398+0000","last_became_active":"2021-03-15T22:34:06.311478+0000","last_became_peered":"2021-03-15T22:34:06.311478+0000","last_unstale":"2021-03-15T22:36:44.755398+0000","last_undegraded":"2021-03-15T22:36:44.755398+0000","last_fullsized":"2021-03-15T22:36:44.755398+0000","mapping_epoch":890,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"891'22","last_scrub_stamp":"2021-03-15T22:35:38.710643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:38.710643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1082","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.516414+0000","last_change":"2021-03-15T22:36:22.729938+0000","last_active":"2021-03-15T22:36:41.516414+0000","last_peered":"2021-03-15T22:36:41.516414+0000","last_clean":"2021-03-15T22:36:41.516414+0000","last_became_active":"2021-03-15T22:34:55.351597+0000","last_became_peered":"2021-03-15T22:34:55.351597+0000","last_unstale":"2021-03-15T22:36:41.516414+0000","last_undegraded":"2021-03-15T22:36:41.516414+0000","last_fullsized":"2021-03-15T22:36:41.516414+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:22.729845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.729845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"650","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.549621+0000","last_change":"2021-03-15T22:34:10.385041+0000","last_active":"2021-03-15T22:36:44.549621+0000","last_peered":"2021-03-15T22:36:44.549621+0000","last_clean":"2021-03-15T22:36:44.549621+0000","last_became_active":"2021-03-15T22:34:10.347227+0000","last_became_peered":"2021-03-15T22:34:10.347227+0000","last_unstale":"2021-03-15T22:36:44.549621+0000","last_undegraded":"2021-03-15T22:36:44.549621+0000","last_fullsized":"2021-03-15T22:36:44.549621+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:34:07.125155+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.125155+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":3736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"1131","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.549240+0000","last_change":"2021-03-15T22:36:16.071953+0000","last_active":"2021-03-15T22:36:44.549240+0000","last_peered":"2021-03-15T22:36:44.549240+0000","last_clean":"2021-03-15T22:36:44.549240+0000","last_became_active":"2021-03-15T22:34:58.353086+0000","last_became_peered":"2021-03-15T22:34:58.353086+0000","last_unstale":"2021-03-15T22:36:44.549240+0000","last_undegraded":"2021-03-15T22:36:44.549240+0000","last_fullsized":"2021-03-15T22:36:44.549240+0000","mapping_epoch":942,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:36:16.071889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.071889+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"1058","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549813+0000","last_change":"2021-03-15T22:35:18.111879+0000","last_active":"2021-03-15T22:36:45.549813+0000","last_peered":"2021-03-15T22:36:45.549813+0000","last_clean":"2021-03-15T22:36:45.549813+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:36:45.549813+0000","last_undegraded":"2021-03-15T22:36:45.549813+0000","last_fullsized":"2021-03-15T22:36:45.549813+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:18.111815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.111815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"261","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.755377+0000","last_change":"2021-03-15T22:36:17.753463+0000","last_active":"2021-03-15T22:36:44.755377+0000","last_peered":"2021-03-15T22:36:44.755377+0000","last_clean":"2021-03-15T22:36:44.755377+0000","last_became_active":"2021-03-15T22:33:04.336100+0000","last_became_peered":"2021-03-15T22:33:04.336100+0000","last_unstale":"2021-03-15T22:36:44.755377+0000","last_undegraded":"2021-03-15T22:36:44.755377+0000","last_fullsized":"2021-03-15T22:36:44.755377+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:36:17.753373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:17.753373+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"1064","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.515851+0000","last_change":"2021-03-15T22:35:41.589000+0000","last_active":"2021-03-15T22:36:41.515851+0000","last_peered":"2021-03-15T22:36:41.515851+0000","last_clean":"2021-03-15T22:36:41.515851+0000","last_became_active":"2021-03-15T22:34:55.351089+0000","last_became_peered":"2021-03-15T22:34:55.351089+0000","last_unstale":"2021-03-15T22:36:41.515851+0000","last_undegraded":"2021-03-15T22:36:41.515851+0000","last_fullsized":"2021-03-15T22:36:41.515851+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:41.588935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.588935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"232","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.755329+0000","last_change":"2021-03-15T22:35:22.665325+0000","last_active":"2021-03-15T22:36:44.755329+0000","last_peered":"2021-03-15T22:36:44.755329+0000","last_clean":"2021-03-15T22:36:44.755329+0000","last_became_active":"2021-03-15T22:33:04.335734+0000","last_became_peered":"2021-03-15T22:33:04.335734+0000","last_unstale":"2021-03-15T22:36:44.755329+0000","last_undegraded":"2021-03-15T22:36:44.755329+0000","last_fullsized":"2021-03-15T22:36:44.755329+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.665251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.665251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"561","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.369036+0000","last_change":"2021-03-15T22:33:37.752077+0000","last_active":"2021-03-15T22:36:45.369036+0000","last_peered":"2021-03-15T22:36:45.369036+0000","last_clean":"2021-03-15T22:36:45.369036+0000","last_became_active":"2021-03-15T22:33:04.342749+0000","last_became_peered":"2021-03-15T22:33:04.342749+0000","last_unstale":"2021-03-15T22:36:45.369036+0000","last_undegraded":"2021-03-15T22:36:45.369036+0000","last_fullsized":"2021-03-15T22:36:45.369036+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:33:37.752011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.752011+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":8764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"674","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368098+0000","last_change":"2021-03-15T22:34:34.783107+0000","last_active":"2021-03-15T22:36:45.368098+0000","last_peered":"2021-03-15T22:36:45.368098+0000","last_clean":"2021-03-15T22:36:45.368098+0000","last_became_active":"2021-03-15T22:33:04.333077+0000","last_became_peered":"2021-03-15T22:33:04.333077+0000","last_unstale":"2021-03-15T22:36:45.368098+0000","last_undegraded":"2021-03-15T22:36:45.368098+0000","last_fullsized":"2021-03-15T22:36:45.368098+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.783047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.783047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"175","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.369109+0000","last_change":"2021-03-15T22:35:48.884382+0000","last_active":"2021-03-15T22:36:45.369109+0000","last_peered":"2021-03-15T22:36:45.369109+0000","last_clean":"2021-03-15T22:36:45.369109+0000","last_became_active":"2021-03-15T22:34:03.571049+0000","last_became_peered":"2021-03-15T22:34:03.571049+0000","last_unstale":"2021-03-15T22:36:45.369109+0000","last_undegraded":"2021-03-15T22:36:45.369109+0000","last_fullsized":"2021-03-15T22:36:45.369109+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:48.884317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:48.884317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"892","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367551+0000","last_change":"2021-03-15T22:34:37.787302+0000","last_active":"2021-03-15T22:36:45.367551+0000","last_peered":"2021-03-15T22:36:45.367551+0000","last_clean":"2021-03-15T22:36:45.367551+0000","last_became_active":"2021-03-15T22:33:04.331607+0000","last_became_peered":"2021-03-15T22:33:04.331607+0000","last_unstale":"2021-03-15T22:36:45.367551+0000","last_undegraded":"2021-03-15T22:36:45.367551+0000","last_fullsized":"2021-03-15T22:36:45.367551+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:34:37.787235+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.787235+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"536","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.548247+0000","last_change":"2021-03-15T22:35:17.250496+0000","last_active":"2021-03-15T22:36:44.548247+0000","last_peered":"2021-03-15T22:36:44.548247+0000","last_clean":"2021-03-15T22:36:44.548247+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:36:44.548247+0000","last_undegraded":"2021-03-15T22:36:44.548247+0000","last_fullsized":"2021-03-15T22:36:44.548247+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.250434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.250434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"846","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368017+0000","last_change":"2021-03-15T22:36:11.805477+0000","last_active":"2021-03-15T22:36:45.368017+0000","last_peered":"2021-03-15T22:36:45.368017+0000","last_clean":"2021-03-15T22:36:45.368017+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:36:45.368017+0000","last_undegraded":"2021-03-15T22:36:45.368017+0000","last_fullsized":"2021-03-15T22:36:45.368017+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:11.805413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:36:11.805413+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"1052","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.516787+0000","last_change":"2021-03-15T22:35:37.543358+0000","last_active":"2021-03-15T22:36:41.516787+0000","last_peered":"2021-03-15T22:36:41.516787+0000","last_clean":"2021-03-15T22:36:41.516787+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:36:41.516787+0000","last_undegraded":"2021-03-15T22:36:41.516787+0000","last_fullsized":"2021-03-15T22:36:41.516787+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:37.543289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:35:37.543289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"768","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.755158+0000","last_change":"2021-03-15T22:36:18.800914+0000","last_active":"2021-03-15T22:36:44.755158+0000","last_peered":"2021-03-15T22:36:44.755158+0000","last_clean":"2021-03-15T22:36:44.755158+0000","last_became_active":"2021-03-15T22:34:55.344555+0000","last_became_peered":"2021-03-15T22:34:55.344555+0000","last_unstale":"2021-03-15T22:36:44.755158+0000","last_undegraded":"2021-03-15T22:36:44.755158+0000","last_fullsized":"2021-03-15T22:36:44.755158+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:36:18.800824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:18.800824+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"433","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.548607+0000","last_change":"2021-03-15T22:35:23.291568+0000","last_active":"2021-03-15T22:36:44.548607+0000","last_peered":"2021-03-15T22:36:44.548607+0000","last_clean":"2021-03-15T22:36:44.548607+0000","last_became_active":"2021-03-15T22:34:55.355119+0000","last_became_peered":"2021-03-15T22:34:55.355119+0000","last_unstale":"2021-03-15T22:36:44.548607+0000","last_undegraded":"2021-03-15T22:36:44.548607+0000","last_fullsized":"2021-03-15T22:36:44.548607+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.291507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.291507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"1111","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549124+0000","last_change":"2021-03-15T22:35:50.069546+0000","last_active":"2021-03-15T22:36:45.549124+0000","last_peered":"2021-03-15T22:36:45.549124+0000","last_clean":"2021-03-15T22:36:45.549124+0000","last_became_active":"2021-03-15T22:34:10.353314+0000","last_became_peered":"2021-03-15T22:34:10.353314+0000","last_unstale":"2021-03-15T22:36:45.549124+0000","last_undegraded":"2021-03-15T22:36:45.549124+0000","last_fullsized":"2021-03-15T22:36:45.549124+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:50.069463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:35:50.069463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"189","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367439+0000","last_change":"2021-03-15T22:36:20.132263+0000","last_active":"2021-03-15T22:36:45.367439+0000","last_peered":"2021-03-15T22:36:45.367439+0000","last_clean":"2021-03-15T22:36:45.367439+0000","last_became_active":"2021-03-15T22:33:45.314042+0000","last_became_peered":"2021-03-15T22:33:45.314042+0000","last_unstale":"2021-03-15T22:36:45.367439+0000","last_undegraded":"2021-03-15T22:36:45.367439+0000","last_fullsized":"2021-03-15T22:36:45.367439+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:20.132182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.132182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"1135","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367689+0000","last_change":"2021-03-15T22:34:42.632034+0000","last_active":"2021-03-15T22:36:45.367689+0000","last_peered":"2021-03-15T22:36:45.367689+0000","last_clean":"2021-03-15T22:36:45.367689+0000","last_became_active":"2021-03-15T22:33:04.340460+0000","last_became_peered":"2021-03-15T22:33:04.340460+0000","last_unstale":"2021-03-15T22:36:45.367689+0000","last_undegraded":"2021-03-15T22:36:45.367689+0000","last_fullsized":"2021-03-15T22:36:45.367689+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:34:42.631962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.631962+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"544","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.549867+0000","last_change":"2021-03-15T22:36:22.004600+0000","last_active":"2021-03-15T22:36:44.549867+0000","last_peered":"2021-03-15T22:36:44.549867+0000","last_clean":"2021-03-15T22:36:44.549867+0000","last_became_active":"2021-03-15T22:34:10.326881+0000","last_became_peered":"2021-03-15T22:34:10.326881+0000","last_unstale":"2021-03-15T22:36:44.549867+0000","last_undegraded":"2021-03-15T22:36:44.549867+0000","last_fullsized":"2021-03-15T22:36:44.549867+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:22.004539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.004539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"808'9","reported_seq":"514","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:42.199829+0000","last_change":"2021-03-15T22:35:41.986386+0000","last_active":"2021-03-15T22:36:42.199829+0000","last_peered":"2021-03-15T22:36:42.199829+0000","last_clean":"2021-03-15T22:36:42.199829+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:36:42.199829+0000","last_undegraded":"2021-03-15T22:36:42.199829+0000","last_fullsized":"2021-03-15T22:36:42.199829+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9","last_scrub_stamp":"2021-03-15T22:35:41.986318+0000","last_deep_scrub":"808'9","last_deep_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.986318+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"1049","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.515502+0000","last_change":"2021-03-15T22:36:09.833634+0000","last_active":"2021-03-15T22:36:41.515502+0000","last_peered":"2021-03-15T22:36:41.515502+0000","last_clean":"2021-03-15T22:36:41.515502+0000","last_became_active":"2021-03-15T22:34:55.351042+0000","last_became_peered":"2021-03-15T22:34:55.351042+0000","last_unstale":"2021-03-15T22:36:41.515502+0000","last_undegraded":"2021-03-15T22:36:41.515502+0000","last_fullsized":"2021-03-15T22:36:41.515502+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:09.833552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:36:09.833552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"844'33","reported_seq":"907","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368755+0000","last_change":"2021-03-15T22:34:55.387712+0000","last_active":"2021-03-15T22:36:45.368755+0000","last_peered":"2021-03-15T22:36:45.368755+0000","last_clean":"2021-03-15T22:36:45.368755+0000","last_became_active":"2021-03-15T22:34:55.351721+0000","last_became_peered":"2021-03-15T22:34:55.351721+0000","last_unstale":"2021-03-15T22:36:45.368755+0000","last_undegraded":"2021-03-15T22:36:45.368755+0000","last_fullsized":"2021-03-15T22:36:45.368755+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"844'33","last_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_deep_scrub":"844'33","last_deep_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.622437+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1088","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368258+0000","last_change":"2021-03-15T22:35:40.412203+0000","last_active":"2021-03-15T22:36:45.368258+0000","last_peered":"2021-03-15T22:36:45.368258+0000","last_clean":"2021-03-15T22:36:45.368258+0000","last_became_active":"2021-03-15T22:34:55.340440+0000","last_became_peered":"2021-03-15T22:34:55.340440+0000","last_unstale":"2021-03-15T22:36:45.368258+0000","last_undegraded":"2021-03-15T22:36:45.368258+0000","last_fullsized":"2021-03-15T22:36:45.368258+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:40.412142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:40.412142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"941'17","reported_seq":"589","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:42.199677+0000","last_change":"2021-03-15T22:36:17.843284+0000","last_active":"2021-03-15T22:36:42.199677+0000","last_peered":"2021-03-15T22:36:42.199677+0000","last_clean":"2021-03-15T22:36:42.199677+0000","last_became_active":"2021-03-15T22:34:55.351682+0000","last_became_peered":"2021-03-15T22:34:55.351682+0000","last_unstale":"2021-03-15T22:36:42.199677+0000","last_undegraded":"2021-03-15T22:36:42.199677+0000","last_fullsized":"2021-03-15T22:36:42.199677+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'17","last_scrub_stamp":"2021-03-15T22:36:17.843205+0000","last_deep_scrub":"896'16","last_deep_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_clean_scrub_stamp":"2021-03-15T22:36:17.843205+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"544","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.548750+0000","last_change":"2021-03-15T22:36:10.121601+0000","last_active":"2021-03-15T22:36:44.548750+0000","last_peered":"2021-03-15T22:36:44.548750+0000","last_clean":"2021-03-15T22:36:44.548750+0000","last_became_active":"2021-03-15T22:34:10.347512+0000","last_became_peered":"2021-03-15T22:34:10.347512+0000","last_unstale":"2021-03-15T22:36:44.548750+0000","last_undegraded":"2021-03-15T22:36:44.548750+0000","last_fullsized":"2021-03-15T22:36:44.548750+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:10.121536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:36:10.121536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"444","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.517799+0000","last_change":"2021-03-15T22:36:11.021642+0000","last_active":"2021-03-15T22:36:41.517799+0000","last_peered":"2021-03-15T22:36:41.517799+0000","last_clean":"2021-03-15T22:36:41.517799+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:36:41.517799+0000","last_undegraded":"2021-03-15T22:36:41.517799+0000","last_fullsized":"2021-03-15T22:36:41.517799+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:36:11.021564+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:36:11.021564+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"131","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368564+0000","last_change":"2021-03-15T22:34:43.345092+0000","last_active":"2021-03-15T22:36:45.368564+0000","last_peered":"2021-03-15T22:36:45.368564+0000","last_clean":"2021-03-15T22:36:45.368564+0000","last_became_active":"2021-03-15T22:34:43.344790+0000","last_became_peered":"2021-03-15T22:34:43.344790+0000","last_unstale":"2021-03-15T22:36:45.368564+0000","last_undegraded":"2021-03-15T22:36:45.368564+0000","last_fullsized":"2021-03-15T22:36:45.368564+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.621539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"121","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.511276+0000","last_change":"2021-03-15T22:35:20.381407+0000","last_active":"2021-03-15T22:36:41.511276+0000","last_peered":"2021-03-15T22:36:41.511276+0000","last_clean":"2021-03-15T22:36:41.511276+0000","last_became_active":"2021-03-15T22:34:55.348663+0000","last_became_peered":"2021-03-15T22:34:55.348663+0000","last_unstale":"2021-03-15T22:36:41.511276+0000","last_undegraded":"2021-03-15T22:36:41.511276+0000","last_fullsized":"2021-03-15T22:36:41.511276+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.381352+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"1042","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.515763+0000","last_change":"2021-03-15T22:35:48.581266+0000","last_active":"2021-03-15T22:36:41.515763+0000","last_peered":"2021-03-15T22:36:41.515763+0000","last_clean":"2021-03-15T22:36:41.515763+0000","last_became_active":"2021-03-15T22:33:04.330195+0000","last_became_peered":"2021-03-15T22:33:04.330195+0000","last_unstale":"2021-03-15T22:36:41.515763+0000","last_undegraded":"2021-03-15T22:36:41.515763+0000","last_fullsized":"2021-03-15T22:36:41.515763+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:48.581162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:48.581162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"575","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.549384+0000","last_change":"2021-03-15T22:34:37.435070+0000","last_active":"2021-03-15T22:36:44.549384+0000","last_peered":"2021-03-15T22:36:44.549384+0000","last_clean":"2021-03-15T22:36:44.549384+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:36:44.549384+0000","last_undegraded":"2021-03-15T22:36:44.549384+0000","last_fullsized":"2021-03-15T22:36:44.549384+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:34:37.435006+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.435006+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"848","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.518252+0000","last_change":"2021-03-15T22:35:20.884067+0000","last_active":"2021-03-15T22:36:41.518252+0000","last_peered":"2021-03-15T22:36:41.518252+0000","last_clean":"2021-03-15T22:36:41.518252+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:36:41.518252+0000","last_undegraded":"2021-03-15T22:36:41.518252+0000","last_fullsized":"2021-03-15T22:36:41.518252+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.883990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.883990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"1036","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.516743+0000","last_change":"2021-03-15T22:35:45.547459+0000","last_active":"2021-03-15T22:36:41.516743+0000","last_peered":"2021-03-15T22:36:41.516743+0000","last_clean":"2021-03-15T22:36:41.516743+0000","last_became_active":"2021-03-15T22:33:04.331013+0000","last_became_peered":"2021-03-15T22:33:04.331013+0000","last_unstale":"2021-03-15T22:36:41.516743+0000","last_undegraded":"2021-03-15T22:36:41.516743+0000","last_fullsized":"2021-03-15T22:36:41.516743+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.547395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:45.547395+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.547395+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"232","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.754928+0000","last_change":"2021-03-15T22:35:45.757312+0000","last_active":"2021-03-15T22:36:44.754928+0000","last_peered":"2021-03-15T22:36:44.754928+0000","last_clean":"2021-03-15T22:36:44.754928+0000","last_became_active":"2021-03-15T22:33:04.336886+0000","last_became_peered":"2021-03-15T22:33:04.336886+0000","last_unstale":"2021-03-15T22:36:44.754928+0000","last_undegraded":"2021-03-15T22:36:44.754928+0000","last_fullsized":"2021-03-15T22:36:44.754928+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.757254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:45.757254+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.757254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"395","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.517332+0000","last_change":"2021-03-15T22:34:55.387006+0000","last_active":"2021-03-15T22:36:41.517332+0000","last_peered":"2021-03-15T22:36:41.517332+0000","last_clean":"2021-03-15T22:36:41.517332+0000","last_became_active":"2021-03-15T22:34:55.351977+0000","last_became_peered":"2021-03-15T22:34:55.351977+0000","last_unstale":"2021-03-15T22:36:41.517332+0000","last_undegraded":"2021-03-15T22:36:41.517332+0000","last_fullsized":"2021-03-15T22:36:41.517332+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:34:36.438541+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:34:36.438541+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"1040'1547","reported_seq":"4802","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:46.565341+0000","last_change":"2021-03-15T22:36:46.565341+0000","last_active":"2021-03-15T22:36:46.565341+0000","last_peered":"2021-03-15T22:36:46.565341+0000","last_clean":"2021-03-15T22:36:46.565341+0000","last_became_active":"2021-03-15T22:33:08.355490+0000","last_became_peered":"2021-03-15T22:33:08.355490+0000","last_unstale":"2021-03-15T22:36:46.565341+0000","last_undegraded":"2021-03-15T22:36:46.565341+0000","last_fullsized":"2021-03-15T22:36:46.565341+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"1040'1547","last_scrub_stamp":"2021-03-15T22:36:46.565275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:46.565275+0000","log_size":1547,"ondisk_log_size":1547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":53562661,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":509,"num_read_kb":562639,"num_write":351,"num_write_kb":126327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":84869361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"988","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.514902+0000","last_change":"2021-03-15T22:33:42.741529+0000","last_active":"2021-03-15T22:36:41.514902+0000","last_peered":"2021-03-15T22:36:41.514902+0000","last_clean":"2021-03-15T22:36:41.514902+0000","last_became_active":"2021-03-15T22:33:04.340352+0000","last_became_peered":"2021-03-15T22:33:04.340352+0000","last_unstale":"2021-03-15T22:36:41.514902+0000","last_undegraded":"2021-03-15T22:36:41.514902+0000","last_fullsized":"2021-03-15T22:36:41.514902+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:42.741451+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:42.741451+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"874'4","reported_seq":"1077","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.517314+0000","last_change":"2021-03-15T22:36:14.798866+0000","last_active":"2021-03-15T22:36:41.517314+0000","last_peered":"2021-03-15T22:36:41.517314+0000","last_clean":"2021-03-15T22:36:41.517314+0000","last_became_active":"2021-03-15T22:35:58.360849+0000","last_became_peered":"2021-03-15T22:35:58.360849+0000","last_unstale":"2021-03-15T22:36:41.517314+0000","last_undegraded":"2021-03-15T22:36:41.517314+0000","last_fullsized":"2021-03-15T22:36:41.517314+0000","mapping_epoch":552,"log_start":"874'4","ondisk_log_start":"874'4","created":56,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"874'4","last_scrub_stamp":"2021-03-15T22:36:14.798788+0000","last_deep_scrub":"874'4","last_deep_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.798788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":965,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"515","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.514437+0000","last_change":"2021-03-15T22:33:38.759477+0000","last_active":"2021-03-15T22:36:41.514437+0000","last_peered":"2021-03-15T22:36:41.514437+0000","last_clean":"2021-03-15T22:36:41.514437+0000","last_became_active":"2021-03-15T22:33:04.326070+0000","last_became_peered":"2021-03-15T22:33:04.326070+0000","last_unstale":"2021-03-15T22:36:41.514437+0000","last_undegraded":"2021-03-15T22:36:41.514437+0000","last_fullsized":"2021-03-15T22:36:41.514437+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:33:38.759395+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.759395+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"841'14","reported_seq":"1330","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:42.199677+0000","last_change":"2021-03-15T22:36:13.797925+0000","last_active":"2021-03-15T22:36:42.199677+0000","last_peered":"2021-03-15T22:36:42.199677+0000","last_clean":"2021-03-15T22:36:42.199677+0000","last_became_active":"2021-03-15T22:33:08.354031+0000","last_became_peered":"2021-03-15T22:33:08.354031+0000","last_unstale":"2021-03-15T22:36:42.199677+0000","last_undegraded":"2021-03-15T22:36:42.199677+0000","last_fullsized":"2021-03-15T22:36:42.199677+0000","mapping_epoch":837,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'14","last_scrub_stamp":"2021-03-15T22:36:13.797841+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:36:13.797841+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"1043'2255","reported_seq":"5755","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367671+0000","last_change":"2021-03-15T22:36:35.395939+0000","last_active":"2021-03-15T22:36:45.367671+0000","last_peered":"2021-03-15T22:36:45.367671+0000","last_clean":"2021-03-15T22:36:45.367671+0000","last_became_active":"2021-03-15T22:34:58.336162+0000","last_became_peered":"2021-03-15T22:34:58.336162+0000","last_unstale":"2021-03-15T22:36:45.367671+0000","last_undegraded":"2021-03-15T22:36:45.367671+0000","last_fullsized":"2021-03-15T22:36:45.367671+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'2031","last_scrub_stamp":"2021-03-15T22:35:42.023328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.023328+0000","log_size":2255,"ondisk_log_size":2255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90051309,"num_objects":41,"num_object_clones":36,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":1,"num_read":710,"num_read_kb":662809,"num_write":557,"num_write_kb":215582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":198498497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"477","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367855+0000","last_change":"2021-03-15T22:32:42.621654+0000","last_active":"2021-03-15T22:36:45.367855+0000","last_peered":"2021-03-15T22:36:45.367855+0000","last_clean":"2021-03-15T22:36:45.367855+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:36:45.367855+0000","last_undegraded":"2021-03-15T22:36:45.367855+0000","last_fullsized":"2021-03-15T22:36:45.367855+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:42.621594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:42.621594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"1044","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.517196+0000","last_change":"2021-03-15T22:36:02.374943+0000","last_active":"2021-03-15T22:36:41.517196+0000","last_peered":"2021-03-15T22:36:41.517196+0000","last_clean":"2021-03-15T22:36:41.517196+0000","last_became_active":"2021-03-15T22:36:02.374583+0000","last_became_peered":"2021-03-15T22:36:02.374583+0000","last_unstale":"2021-03-15T22:36:41.517196+0000","last_undegraded":"2021-03-15T22:36:41.517196+0000","last_fullsized":"2021-03-15T22:36:41.517196+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1007,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:47.601457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.601457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"1258","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367363+0000","last_change":"2021-03-15T22:34:11.031994+0000","last_active":"2021-03-15T22:36:45.367363+0000","last_peered":"2021-03-15T22:36:45.367363+0000","last_clean":"2021-03-15T22:36:45.367363+0000","last_became_active":"2021-03-15T22:34:10.342120+0000","last_became_peered":"2021-03-15T22:34:10.342120+0000","last_unstale":"2021-03-15T22:36:45.367363+0000","last_undegraded":"2021-03-15T22:36:45.367363+0000","last_fullsized":"2021-03-15T22:36:45.367363+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:34:07.452572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.452572+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"760","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.548285+0000","last_change":"2021-03-15T22:34:55.356651+0000","last_active":"2021-03-15T22:36:44.548285+0000","last_peered":"2021-03-15T22:36:44.548285+0000","last_clean":"2021-03-15T22:36:44.548285+0000","last_became_active":"2021-03-15T22:34:55.356328+0000","last_became_peered":"2021-03-15T22:34:55.356328+0000","last_unstale":"2021-03-15T22:36:44.548285+0000","last_undegraded":"2021-03-15T22:36:44.548285+0000","last_fullsized":"2021-03-15T22:36:44.548285+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"1040'607","reported_seq":"3584","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367526+0000","last_change":"2021-03-15T22:36:35.381989+0000","last_active":"2021-03-15T22:36:45.367526+0000","last_peered":"2021-03-15T22:36:45.367526+0000","last_clean":"2021-03-15T22:36:45.367526+0000","last_became_active":"2021-03-15T22:34:13.324667+0000","last_became_peered":"2021-03-15T22:34:13.324667+0000","last_unstale":"2021-03-15T22:36:45.367526+0000","last_undegraded":"2021-03-15T22:36:45.367526+0000","last_fullsized":"2021-03-15T22:36:45.367526+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'446","last_scrub_stamp":"2021-03-15T22:33:39.355306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.355306+0000","log_size":607,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20558245,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":149,"num_read_kb":124961,"num_write":134,"num_write_kb":44296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":23698344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"1112","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.515261+0000","last_change":"2021-03-15T22:36:18.835190+0000","last_active":"2021-03-15T22:36:41.515261+0000","last_peered":"2021-03-15T22:36:41.515261+0000","last_clean":"2021-03-15T22:36:41.515261+0000","last_became_active":"2021-03-15T22:35:48.365533+0000","last_became_peered":"2021-03-15T22:35:48.365533+0000","last_unstale":"2021-03-15T22:36:41.515261+0000","last_undegraded":"2021-03-15T22:36:41.515261+0000","last_fullsized":"2021-03-15T22:36:41.515261+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":993,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:18.835125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:36:18.835125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"121","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.510998+0000","last_change":"2021-03-15T22:35:49.448634+0000","last_active":"2021-03-15T22:36:41.510998+0000","last_peered":"2021-03-15T22:36:41.510998+0000","last_clean":"2021-03-15T22:36:41.510998+0000","last_became_active":"2021-03-15T22:34:55.353966+0000","last_became_peered":"2021-03-15T22:34:55.353966+0000","last_unstale":"2021-03-15T22:36:41.510998+0000","last_undegraded":"2021-03-15T22:36:41.510998+0000","last_fullsized":"2021-03-15T22:36:41.510998+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.448581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.448581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"1099","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.516894+0000","last_change":"2021-03-15T22:36:06.382789+0000","last_active":"2021-03-15T22:36:41.516894+0000","last_peered":"2021-03-15T22:36:41.516894+0000","last_clean":"2021-03-15T22:36:41.516894+0000","last_became_active":"2021-03-15T22:36:06.382477+0000","last_became_peered":"2021-03-15T22:36:06.382477+0000","last_unstale":"2021-03-15T22:36:41.516894+0000","last_undegraded":"2021-03-15T22:36:41.516894+0000","last_fullsized":"2021-03-15T22:36:41.516894+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1011,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.428842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.428842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"297","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.755482+0000","last_change":"2021-03-15T22:35:47.782159+0000","last_active":"2021-03-15T22:36:44.755482+0000","last_peered":"2021-03-15T22:36:44.755482+0000","last_clean":"2021-03-15T22:36:44.755482+0000","last_became_active":"2021-03-15T22:33:45.318867+0000","last_became_peered":"2021-03-15T22:33:45.318867+0000","last_unstale":"2021-03-15T22:36:44.755482+0000","last_undegraded":"2021-03-15T22:36:44.755482+0000","last_fullsized":"2021-03-15T22:36:44.755482+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:35:47.782092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.782092+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":2928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"1043'1384","reported_seq":"4425","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.510580+0000","last_change":"2021-03-15T22:36:35.375257+0000","last_active":"2021-03-15T22:36:41.510580+0000","last_peered":"2021-03-15T22:36:41.510580+0000","last_clean":"2021-03-15T22:36:41.510580+0000","last_became_active":"2021-03-15T22:34:59.339587+0000","last_became_peered":"2021-03-15T22:34:59.339587+0000","last_unstale":"2021-03-15T22:36:41.510580+0000","last_undegraded":"2021-03-15T22:36:41.510580+0000","last_fullsized":"2021-03-15T22:36:41.510580+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'1309","last_scrub_stamp":"2021-03-15T22:36:14.641356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.641356+0000","log_size":1384,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52382815,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":450,"num_read_kb":538913,"num_write":306,"num_write_kb":109155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":205148835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"34b","length":"1"}]},{"pgid":"6.9","version":"0'0","reported_seq":"568","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.548709+0000","last_change":"2021-03-15T22:35:44.582634+0000","last_active":"2021-03-15T22:36:44.548709+0000","last_peered":"2021-03-15T22:36:44.548709+0000","last_clean":"2021-03-15T22:36:44.548709+0000","last_became_active":"2021-03-15T22:35:44.582322+0000","last_became_peered":"2021-03-15T22:35:44.582322+0000","last_unstale":"2021-03-15T22:36:44.548709+0000","last_undegraded":"2021-03-15T22:36:44.548709+0000","last_fullsized":"2021-03-15T22:36:44.548709+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"887","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367427+0000","last_change":"2021-03-15T22:36:45.344330+0000","last_active":"2021-03-15T22:36:45.367427+0000","last_peered":"2021-03-15T22:36:45.367427+0000","last_clean":"2021-03-15T22:36:45.367427+0000","last_became_active":"2021-03-15T22:34:55.342808+0000","last_became_peered":"2021-03-15T22:34:55.342808+0000","last_unstale":"2021-03-15T22:36:45.367427+0000","last_undegraded":"2021-03-15T22:36:45.367427+0000","last_fullsized":"2021-03-15T22:36:45.367427+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:45.344264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:45.344264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"464","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.517748+0000","last_change":"2021-03-15T22:36:10.362976+0000","last_active":"2021-03-15T22:36:41.517748+0000","last_peered":"2021-03-15T22:36:41.517748+0000","last_clean":"2021-03-15T22:36:41.517748+0000","last_became_active":"2021-03-15T22:36:10.362612+0000","last_became_peered":"2021-03-15T22:36:10.362612+0000","last_unstale":"2021-03-15T22:36:41.517748+0000","last_undegraded":"2021-03-15T22:36:41.517748+0000","last_fullsized":"2021-03-15T22:36:41.517748+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1015,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.807411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.807411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"595","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.549288+0000","last_change":"2021-03-15T22:36:14.097504+0000","last_active":"2021-03-15T22:36:44.549288+0000","last_peered":"2021-03-15T22:36:44.549288+0000","last_clean":"2021-03-15T22:36:44.549288+0000","last_became_active":"2021-03-15T22:33:45.326747+0000","last_became_peered":"2021-03-15T22:33:45.326747+0000","last_unstale":"2021-03-15T22:36:44.549288+0000","last_undegraded":"2021-03-15T22:36:44.549288+0000","last_fullsized":"2021-03-15T22:36:44.549288+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:36:14.097432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.097432+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":2682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"27","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.514643+0000","last_change":"2021-03-15T22:36:24.384474+0000","last_active":"2021-03-15T22:36:41.514643+0000","last_peered":"2021-03-15T22:36:41.514643+0000","last_clean":"2021-03-15T22:36:41.514643+0000","last_became_active":"2021-03-15T22:36:24.384123+0000","last_became_peered":"2021-03-15T22:36:24.384123+0000","last_unstale":"2021-03-15T22:36:41.514643+0000","last_undegraded":"2021-03-15T22:36:41.514643+0000","last_fullsized":"2021-03-15T22:36:41.514643+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"1043'1585","reported_seq":"5302","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533459+0000","last_change":"2021-03-15T22:36:35.395167+0000","last_active":"2021-03-15T22:36:45.533459+0000","last_peered":"2021-03-15T22:36:45.533459+0000","last_clean":"2021-03-15T22:36:45.533459+0000","last_became_active":"2021-03-15T22:34:46.338888+0000","last_became_peered":"2021-03-15T22:34:46.338888+0000","last_unstale":"2021-03-15T22:36:45.533459+0000","last_undegraded":"2021-03-15T22:36:45.533459+0000","last_fullsized":"2021-03-15T22:36:45.533459+0000","mapping_epoch":930,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"1025'1523","last_scrub_stamp":"2021-03-15T22:36:20.962682+0000","last_deep_scrub":"1025'1523","last_deep_scrub_stamp":"2021-03-15T22:36:20.962682+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.962682+0000","log_size":1585,"ondisk_log_size":1585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60313905,"num_objects":26,"num_object_clones":23,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":350,"num_read_kb":290243,"num_write":369,"num_write_kb":136361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":229919087,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"149","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.369140+0000","last_change":"2021-03-15T22:35:56.358841+0000","last_active":"2021-03-15T22:36:45.369140+0000","last_peered":"2021-03-15T22:36:45.369140+0000","last_clean":"2021-03-15T22:36:45.369140+0000","last_became_active":"2021-03-15T22:35:56.358504+0000","last_became_peered":"2021-03-15T22:35:56.358504+0000","last_unstale":"2021-03-15T22:36:45.369140+0000","last_undegraded":"2021-03-15T22:36:45.369140+0000","last_fullsized":"2021-03-15T22:36:45.369140+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1001,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"92","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.515572+0000","last_change":"2021-03-15T22:35:19.502186+0000","last_active":"2021-03-15T22:36:41.515572+0000","last_peered":"2021-03-15T22:36:41.515572+0000","last_clean":"2021-03-15T22:36:41.515572+0000","last_became_active":"2021-03-15T22:35:19.501783+0000","last_became_peered":"2021-03-15T22:35:19.501783+0000","last_unstale":"2021-03-15T22:36:41.515572+0000","last_undegraded":"2021-03-15T22:36:41.515572+0000","last_fullsized":"2021-03-15T22:36:41.515572+0000","mapping_epoch":963,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":964,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.831369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.831369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"251","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367497+0000","last_change":"2021-03-15T22:36:16.252660+0000","last_active":"2021-03-15T22:36:45.367497+0000","last_peered":"2021-03-15T22:36:45.367497+0000","last_clean":"2021-03-15T22:36:45.367497+0000","last_became_active":"2021-03-15T22:33:04.335914+0000","last_became_peered":"2021-03-15T22:33:04.335914+0000","last_unstale":"2021-03-15T22:36:45.367497+0000","last_undegraded":"2021-03-15T22:36:45.367497+0000","last_fullsized":"2021-03-15T22:36:45.367497+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:16.252579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.252579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1421","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549483+0000","last_change":"2021-03-15T22:35:47.126971+0000","last_active":"2021-03-15T22:36:45.549483+0000","last_peered":"2021-03-15T22:36:45.549483+0000","last_clean":"2021-03-15T22:36:45.549483+0000","last_became_active":"2021-03-15T22:34:55.350419+0000","last_became_peered":"2021-03-15T22:34:55.350419+0000","last_unstale":"2021-03-15T22:36:45.549483+0000","last_undegraded":"2021-03-15T22:36:45.549483+0000","last_fullsized":"2021-03-15T22:36:45.549483+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:35:47.126904+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.126904+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"512","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.548768+0000","last_change":"2021-03-15T22:33:15.492116+0000","last_active":"2021-03-15T22:36:44.548768+0000","last_peered":"2021-03-15T22:36:44.548768+0000","last_clean":"2021-03-15T22:36:44.548768+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:36:44.548768+0000","last_undegraded":"2021-03-15T22:36:44.548768+0000","last_fullsized":"2021-03-15T22:36:44.548768+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:15.492014+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.492014+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1077","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.515784+0000","last_change":"2021-03-15T22:36:24.388928+0000","last_active":"2021-03-15T22:36:41.515784+0000","last_peered":"2021-03-15T22:36:41.515784+0000","last_clean":"2021-03-15T22:36:41.515784+0000","last_became_active":"2021-03-15T22:36:24.388564+0000","last_became_peered":"2021-03-15T22:36:24.388564+0000","last_unstale":"2021-03-15T22:36:41.515784+0000","last_undegraded":"2021-03-15T22:36:41.515784+0000","last_fullsized":"2021-03-15T22:36:41.515784+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1040'1449","reported_seq":"4287","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368392+0000","last_change":"2021-03-15T22:36:35.394677+0000","last_active":"2021-03-15T22:36:45.368392+0000","last_peered":"2021-03-15T22:36:45.368392+0000","last_clean":"2021-03-15T22:36:45.368392+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:36:45.368392+0000","last_undegraded":"2021-03-15T22:36:45.368392+0000","last_fullsized":"2021-03-15T22:36:45.368392+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'1385","last_scrub_stamp":"2021-03-15T22:36:15.114641+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:36:15.114641+0000","log_size":1449,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50649467,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":420,"num_read_kb":425400,"num_write":316,"num_write_kb":120781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1095","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.516973+0000","last_change":"2021-03-15T22:36:00.364664+0000","last_active":"2021-03-15T22:36:41.516973+0000","last_peered":"2021-03-15T22:36:41.516973+0000","last_clean":"2021-03-15T22:36:41.516973+0000","last_became_active":"2021-03-15T22:36:00.364246+0000","last_became_peered":"2021-03-15T22:36:00.364246+0000","last_unstale":"2021-03-15T22:36:41.516973+0000","last_undegraded":"2021-03-15T22:36:41.516973+0000","last_fullsized":"2021-03-15T22:36:41.516973+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1005,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:44.582515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:44.582515+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"1085","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549074+0000","last_change":"2021-03-15T22:36:13.313358+0000","last_active":"2021-03-15T22:36:45.549074+0000","last_peered":"2021-03-15T22:36:45.549074+0000","last_clean":"2021-03-15T22:36:45.549074+0000","last_became_active":"2021-03-15T22:34:55.347065+0000","last_became_peered":"2021-03-15T22:34:55.347065+0000","last_unstale":"2021-03-15T22:36:45.549074+0000","last_undegraded":"2021-03-15T22:36:45.549074+0000","last_fullsized":"2021-03-15T22:36:45.549074+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:13.313278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:36:13.313278+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"121","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.510704+0000","last_change":"2021-03-15T22:35:17.301942+0000","last_active":"2021-03-15T22:36:41.510704+0000","last_peered":"2021-03-15T22:36:41.510704+0000","last_clean":"2021-03-15T22:36:41.510704+0000","last_became_active":"2021-03-15T22:34:55.355485+0000","last_became_peered":"2021-03-15T22:34:55.355485+0000","last_unstale":"2021-03-15T22:36:41.510704+0000","last_undegraded":"2021-03-15T22:36:41.510704+0000","last_fullsized":"2021-03-15T22:36:41.510704+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.301892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.301892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"534","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.518355+0000","last_change":"2021-03-15T22:36:20.050533+0000","last_active":"2021-03-15T22:36:41.518355+0000","last_peered":"2021-03-15T22:36:41.518355+0000","last_clean":"2021-03-15T22:36:41.518355+0000","last_became_active":"2021-03-15T22:34:10.322666+0000","last_became_peered":"2021-03-15T22:34:10.322666+0000","last_unstale":"2021-03-15T22:36:41.518355+0000","last_undegraded":"2021-03-15T22:36:41.518355+0000","last_fullsized":"2021-03-15T22:36:41.518355+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:36:20.050450+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.050450+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"121","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.511357+0000","last_change":"2021-03-15T22:35:45.355712+0000","last_active":"2021-03-15T22:36:41.511357+0000","last_peered":"2021-03-15T22:36:41.511357+0000","last_clean":"2021-03-15T22:36:41.511357+0000","last_became_active":"2021-03-15T22:34:55.353316+0000","last_became_peered":"2021-03-15T22:34:55.353316+0000","last_unstale":"2021-03-15T22:36:41.511357+0000","last_undegraded":"2021-03-15T22:36:41.511357+0000","last_fullsized":"2021-03-15T22:36:41.511357+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.355647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.355647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"1042'3488","reported_seq":"7160","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.370347+0000","last_change":"2021-03-15T22:36:35.405911+0000","last_active":"2021-03-15T22:36:45.370347+0000","last_peered":"2021-03-15T22:36:45.370347+0000","last_clean":"2021-03-15T22:36:45.370347+0000","last_became_active":"2021-03-15T22:34:13.331560+0000","last_became_peered":"2021-03-15T22:34:13.331560+0000","last_unstale":"2021-03-15T22:36:45.370347+0000","last_undegraded":"2021-03-15T22:36:45.370347+0000","last_fullsized":"2021-03-15T22:36:45.370347+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'3208","last_scrub_stamp":"2021-03-15T22:35:42.271839+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.271839+0000","log_size":3488,"ondisk_log_size":3488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":108494076,"num_objects":49,"num_object_clones":42,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":3,"num_read":948,"num_read_kb":984364,"num_write":714,"num_write_kb":291172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":264088494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"448","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.518476+0000","last_change":"2021-03-15T22:35:52.857478+0000","last_active":"2021-03-15T22:36:41.518476+0000","last_peered":"2021-03-15T22:36:41.518476+0000","last_clean":"2021-03-15T22:36:41.518476+0000","last_became_active":"2021-03-15T22:35:52.857192+0000","last_became_peered":"2021-03-15T22:35:52.857192+0000","last_unstale":"2021-03-15T22:36:41.518476+0000","last_undegraded":"2021-03-15T22:36:41.518476+0000","last_fullsized":"2021-03-15T22:36:41.518476+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":997,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.761137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"1069","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.515446+0000","last_change":"2021-03-15T22:36:22.385705+0000","last_active":"2021-03-15T22:36:41.515446+0000","last_peered":"2021-03-15T22:36:41.515446+0000","last_clean":"2021-03-15T22:36:41.515446+0000","last_became_active":"2021-03-15T22:36:22.385402+0000","last_became_peered":"2021-03-15T22:36:22.385402+0000","last_unstale":"2021-03-15T22:36:41.515446+0000","last_undegraded":"2021-03-15T22:36:41.515446+0000","last_fullsized":"2021-03-15T22:36:41.515446+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":1027,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"131","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367374+0000","last_change":"2021-03-15T22:34:43.341562+0000","last_active":"2021-03-15T22:36:45.367374+0000","last_peered":"2021-03-15T22:36:45.367374+0000","last_clean":"2021-03-15T22:36:45.367374+0000","last_became_active":"2021-03-15T22:34:43.340955+0000","last_became_peered":"2021-03-15T22:34:43.340955+0000","last_unstale":"2021-03-15T22:36:45.367374+0000","last_undegraded":"2021-03-15T22:36:45.367374+0000","last_fullsized":"2021-03-15T22:36:45.367374+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.843478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.843478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"252","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.755451+0000","last_change":"2021-03-15T22:36:12.698401+0000","last_active":"2021-03-15T22:36:44.755451+0000","last_peered":"2021-03-15T22:36:44.755451+0000","last_clean":"2021-03-15T22:36:44.755451+0000","last_became_active":"2021-03-15T22:33:04.330531+0000","last_became_peered":"2021-03-15T22:33:04.330531+0000","last_unstale":"2021-03-15T22:36:44.755451+0000","last_undegraded":"2021-03-15T22:36:44.755451+0000","last_fullsized":"2021-03-15T22:36:44.755451+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:36:12.698329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:12.698329+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"1081","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.516455+0000","last_change":"2021-03-15T22:35:22.409047+0000","last_active":"2021-03-15T22:36:41.516455+0000","last_peered":"2021-03-15T22:36:41.516455+0000","last_clean":"2021-03-15T22:36:41.516455+0000","last_became_active":"2021-03-15T22:34:55.352186+0000","last_became_peered":"2021-03-15T22:34:55.352186+0000","last_unstale":"2021-03-15T22:36:41.516455+0000","last_undegraded":"2021-03-15T22:36:41.516455+0000","last_fullsized":"2021-03-15T22:36:41.516455+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.408965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.408965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"30","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549096+0000","last_change":"2021-03-15T22:36:24.391414+0000","last_active":"2021-03-15T22:36:45.549096+0000","last_peered":"2021-03-15T22:36:45.549096+0000","last_clean":"2021-03-15T22:36:45.549096+0000","last_became_active":"2021-03-15T22:36:24.390984+0000","last_became_peered":"2021-03-15T22:36:24.390984+0000","last_unstale":"2021-03-15T22:36:45.549096+0000","last_undegraded":"2021-03-15T22:36:45.549096+0000","last_fullsized":"2021-03-15T22:36:45.549096+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1037'1052","reported_seq":"4307","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.516697+0000","last_change":"2021-03-15T22:36:35.374467+0000","last_active":"2021-03-15T22:36:41.516697+0000","last_peered":"2021-03-15T22:36:41.516697+0000","last_clean":"2021-03-15T22:36:41.516697+0000","last_became_active":"2021-03-15T22:35:03.346751+0000","last_became_peered":"2021-03-15T22:35:03.346751+0000","last_unstale":"2021-03-15T22:36:41.516697+0000","last_undegraded":"2021-03-15T22:36:41.516697+0000","last_fullsized":"2021-03-15T22:36:41.516697+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":0,"last_scrub":"1020'1020","last_scrub_stamp":"2021-03-15T22:36:16.818677+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.818677+0000","log_size":1052,"ondisk_log_size":1052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37060250,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":267,"num_read_kb":273197,"num_write":228,"num_write_kb":99079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":109648904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"34b","length":"1"}]},{"pgid":"6.4","version":"0'0","reported_seq":"865","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.517751+0000","last_change":"2021-03-15T22:36:04.904466+0000","last_active":"2021-03-15T22:36:41.517751+0000","last_peered":"2021-03-15T22:36:41.517751+0000","last_clean":"2021-03-15T22:36:41.517751+0000","last_became_active":"2021-03-15T22:36:04.904112+0000","last_became_peered":"2021-03-15T22:36:04.904112+0000","last_unstale":"2021-03-15T22:36:41.517751+0000","last_undegraded":"2021-03-15T22:36:41.517751+0000","last_fullsized":"2021-03-15T22:36:41.517751+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"544","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.548792+0000","last_change":"2021-03-15T22:34:35.410541+0000","last_active":"2021-03-15T22:36:44.548792+0000","last_peered":"2021-03-15T22:36:44.548792+0000","last_clean":"2021-03-15T22:36:44.548792+0000","last_became_active":"2021-03-15T22:34:10.321100+0000","last_became_peered":"2021-03-15T22:34:10.321100+0000","last_unstale":"2021-03-15T22:36:44.548792+0000","last_undegraded":"2021-03-15T22:36:44.548792+0000","last_fullsized":"2021-03-15T22:36:44.548792+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:35.410485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:35.410485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"2169","reported_epoch":"1046","state":"active+clean","last_fresh":"2021-03-15T22:36:43.214209+0000","last_change":"2021-03-15T22:35:10.452906+0000","last_active":"2021-03-15T22:36:43.214209+0000","last_peered":"2021-03-15T22:36:43.214209+0000","last_clean":"2021-03-15T22:36:43.214209+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:36:43.214209+0000","last_undegraded":"2021-03-15T22:36:43.214209+0000","last_fullsized":"2021-03-15T22:36:43.214209+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:35:10.452809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.452809+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1114,"num_read_kb":1109,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"642","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.549711+0000","last_change":"2021-03-15T22:34:55.414460+0000","last_active":"2021-03-15T22:36:44.549711+0000","last_peered":"2021-03-15T22:36:44.549711+0000","last_clean":"2021-03-15T22:36:44.549711+0000","last_became_active":"2021-03-15T22:34:55.359023+0000","last_became_peered":"2021-03-15T22:34:55.359023+0000","last_unstale":"2021-03-15T22:36:44.549711+0000","last_undegraded":"2021-03-15T22:36:44.549711+0000","last_fullsized":"2021-03-15T22:36:44.549711+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:34:40.472800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.472800+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"435","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.518561+0000","last_change":"2021-03-15T22:34:40.799217+0000","last_active":"2021-03-15T22:36:41.518561+0000","last_peered":"2021-03-15T22:36:41.518561+0000","last_clean":"2021-03-15T22:36:41.518561+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:36:41.518561+0000","last_undegraded":"2021-03-15T22:36:41.518561+0000","last_fullsized":"2021-03-15T22:36:41.518561+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.799153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.799153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1071","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.754683+0000","last_change":"2021-03-15T22:36:24.387855+0000","last_active":"2021-03-15T22:36:44.754683+0000","last_peered":"2021-03-15T22:36:44.754683+0000","last_clean":"2021-03-15T22:36:44.754683+0000","last_became_active":"2021-03-15T22:36:24.387249+0000","last_became_peered":"2021-03-15T22:36:24.387249+0000","last_unstale":"2021-03-15T22:36:44.754683+0000","last_undegraded":"2021-03-15T22:36:44.754683+0000","last_fullsized":"2021-03-15T22:36:44.754683+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"1041'3571","reported_seq":"7801","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.514248+0000","last_change":"2021-03-15T22:36:35.389045+0000","last_active":"2021-03-15T22:36:41.514248+0000","last_peered":"2021-03-15T22:36:41.514248+0000","last_clean":"2021-03-15T22:36:41.514248+0000","last_became_active":"2021-03-15T22:35:17.487261+0000","last_became_peered":"2021-03-15T22:35:17.487261+0000","last_unstale":"2021-03-15T22:36:41.514248+0000","last_undegraded":"2021-03-15T22:36:41.514248+0000","last_fullsized":"2021-03-15T22:36:41.514248+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":0,"last_scrub":"1020'3444","last_scrub_stamp":"2021-03-15T22:36:16.067782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.067782+0000","log_size":3571,"ondisk_log_size":3571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":121620780,"num_objects":52,"num_object_clones":45,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":2,"num_read":1299,"num_read_kb":1394227,"num_write":801,"num_write_kb":336561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":721,"num_bytes_recovered":351021959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"34b","length":"1"}]},{"pgid":"6.1","version":"0'0","reported_seq":"1108","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.550953+0000","last_change":"2021-03-15T22:36:45.550914+0000","last_active":"2021-03-15T22:36:45.550953+0000","last_peered":"2021-03-15T22:36:45.550953+0000","last_clean":"2021-03-15T22:36:45.550953+0000","last_became_active":"2021-03-15T22:36:16.372177+0000","last_became_peered":"2021-03-15T22:36:16.372177+0000","last_unstale":"2021-03-15T22:36:45.550953+0000","last_undegraded":"2021-03-15T22:36:45.550953+0000","last_fullsized":"2021-03-15T22:36:45.550953+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1021,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:45.550861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:36:45.550861+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"1103","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549329+0000","last_change":"2021-03-15T22:36:24.397895+0000","last_active":"2021-03-15T22:36:45.549329+0000","last_peered":"2021-03-15T22:36:45.549329+0000","last_clean":"2021-03-15T22:36:45.549329+0000","last_became_active":"2021-03-15T22:34:55.346048+0000","last_became_peered":"2021-03-15T22:34:55.346048+0000","last_unstale":"2021-03-15T22:36:45.549329+0000","last_undegraded":"2021-03-15T22:36:45.549329+0000","last_fullsized":"2021-03-15T22:36:45.549329+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:24.397819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_clean_scrub_stamp":"2021-03-15T22:36:24.397819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"592","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.549148+0000","last_change":"2021-03-15T22:35:21.314851+0000","last_active":"2021-03-15T22:36:44.549148+0000","last_peered":"2021-03-15T22:36:44.549148+0000","last_clean":"2021-03-15T22:36:44.549148+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:36:44.549148+0000","last_undegraded":"2021-03-15T22:36:44.549148+0000","last_fullsized":"2021-03-15T22:36:44.549148+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:35:21.314763+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.314763+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"232","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.755383+0000","last_change":"2021-03-15T22:35:39.707993+0000","last_active":"2021-03-15T22:36:44.755383+0000","last_peered":"2021-03-15T22:36:44.755383+0000","last_clean":"2021-03-15T22:36:44.755383+0000","last_became_active":"2021-03-15T22:33:04.337244+0000","last_became_peered":"2021-03-15T22:33:04.337244+0000","last_unstale":"2021-03-15T22:36:44.755383+0000","last_undegraded":"2021-03-15T22:36:44.755383+0000","last_fullsized":"2021-03-15T22:36:44.755383+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:39.707900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:35:39.707900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1071","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.755045+0000","last_change":"2021-03-15T22:36:24.393232+0000","last_active":"2021-03-15T22:36:44.755045+0000","last_peered":"2021-03-15T22:36:44.755045+0000","last_clean":"2021-03-15T22:36:44.755045+0000","last_became_active":"2021-03-15T22:36:24.392909+0000","last_became_peered":"2021-03-15T22:36:24.392909+0000","last_unstale":"2021-03-15T22:36:44.755045+0000","last_undegraded":"2021-03-15T22:36:44.755045+0000","last_fullsized":"2021-03-15T22:36:44.755045+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"1040'624","reported_seq":"3894","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367069+0000","last_change":"2021-03-15T22:36:35.373209+0000","last_active":"2021-03-15T22:36:45.367069+0000","last_peered":"2021-03-15T22:36:45.367069+0000","last_clean":"2021-03-15T22:36:45.367069+0000","last_became_active":"2021-03-15T22:34:15.498612+0000","last_became_peered":"2021-03-15T22:34:15.498612+0000","last_unstale":"2021-03-15T22:36:45.367069+0000","last_undegraded":"2021-03-15T22:36:45.367069+0000","last_fullsized":"2021-03-15T22:36:45.367069+0000","mapping_epoch":899,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":900,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'463","last_scrub_stamp":"2021-03-15T22:33:40.563144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.563144+0000","log_size":624,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23933460,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":185,"num_read_kb":221865,"num_write":115,"num_write_kb":59307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":84391207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"586","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.548202+0000","last_change":"2021-03-15T22:36:20.369600+0000","last_active":"2021-03-15T22:36:44.548202+0000","last_peered":"2021-03-15T22:36:44.548202+0000","last_clean":"2021-03-15T22:36:44.548202+0000","last_became_active":"2021-03-15T22:36:20.369276+0000","last_became_peered":"2021-03-15T22:36:20.369276+0000","last_unstale":"2021-03-15T22:36:44.548202+0000","last_undegraded":"2021-03-15T22:36:44.548202+0000","last_fullsized":"2021-03-15T22:36:44.548202+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1025,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"1023","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.517990+0000","last_change":"2021-03-15T22:36:14.037839+0000","last_active":"2021-03-15T22:36:41.517990+0000","last_peered":"2021-03-15T22:36:41.517990+0000","last_clean":"2021-03-15T22:36:41.517990+0000","last_became_active":"2021-03-15T22:34:55.355757+0000","last_became_peered":"2021-03-15T22:34:55.355757+0000","last_unstale":"2021-03-15T22:36:41.517990+0000","last_undegraded":"2021-03-15T22:36:41.517990+0000","last_fullsized":"2021-03-15T22:36:41.517990+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:14.037757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.037757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"1004","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.514401+0000","last_change":"2021-03-15T22:36:13.027720+0000","last_active":"2021-03-15T22:36:41.514401+0000","last_peered":"2021-03-15T22:36:41.514401+0000","last_clean":"2021-03-15T22:36:41.514401+0000","last_became_active":"2021-03-15T22:34:10.321886+0000","last_became_peered":"2021-03-15T22:34:10.321886+0000","last_unstale":"2021-03-15T22:36:41.514401+0000","last_undegraded":"2021-03-15T22:36:41.514401+0000","last_fullsized":"2021-03-15T22:36:41.514401+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:36:13.027640+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:36:13.027640+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"836'11","reported_seq":"679","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.549007+0000","last_change":"2021-03-15T22:35:44.256542+0000","last_active":"2021-03-15T22:36:44.549007+0000","last_peered":"2021-03-15T22:36:44.549007+0000","last_clean":"2021-03-15T22:36:44.549007+0000","last_became_active":"2021-03-15T22:33:04.336294+0000","last_became_peered":"2021-03-15T22:33:04.336294+0000","last_unstale":"2021-03-15T22:36:44.549007+0000","last_undegraded":"2021-03-15T22:36:44.549007+0000","last_fullsized":"2021-03-15T22:36:44.549007+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"836'11","last_scrub_stamp":"2021-03-15T22:35:44.256486+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:35:44.256486+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"30","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367709+0000","last_change":"2021-03-15T22:36:24.387641+0000","last_active":"2021-03-15T22:36:45.367709+0000","last_peered":"2021-03-15T22:36:45.367709+0000","last_clean":"2021-03-15T22:36:45.367709+0000","last_became_active":"2021-03-15T22:36:24.387299+0000","last_became_peered":"2021-03-15T22:36:24.387299+0000","last_unstale":"2021-03-15T22:36:45.367709+0000","last_undegraded":"2021-03-15T22:36:45.367709+0000","last_fullsized":"2021-03-15T22:36:45.367709+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"1041'1002","reported_seq":"4303","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367860+0000","last_change":"2021-03-15T22:36:35.384139+0000","last_active":"2021-03-15T22:36:45.367860+0000","last_peered":"2021-03-15T22:36:45.367860+0000","last_clean":"2021-03-15T22:36:45.367860+0000","last_became_active":"2021-03-15T22:34:58.350873+0000","last_became_peered":"2021-03-15T22:34:58.350873+0000","last_unstale":"2021-03-15T22:36:45.367860+0000","last_undegraded":"2021-03-15T22:36:45.367860+0000","last_fullsized":"2021-03-15T22:36:45.367860+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'901","last_scrub_stamp":"2021-03-15T22:35:16.957336+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.957336+0000","log_size":1002,"ondisk_log_size":1002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37536135,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":275,"num_read_kb":241829,"num_write":236,"num_write_kb":92610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":114552572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1086","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.516638+0000","last_change":"2021-03-15T22:36:08.382982+0000","last_active":"2021-03-15T22:36:41.516638+0000","last_peered":"2021-03-15T22:36:41.516638+0000","last_clean":"2021-03-15T22:36:41.516638+0000","last_became_active":"2021-03-15T22:36:08.382713+0000","last_became_peered":"2021-03-15T22:36:08.382713+0000","last_unstale":"2021-03-15T22:36:41.516638+0000","last_undegraded":"2021-03-15T22:36:41.516638+0000","last_fullsized":"2021-03-15T22:36:41.516638+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.507204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.507204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"530","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.549788+0000","last_change":"2021-03-15T22:36:34.383378+0000","last_active":"2021-03-15T22:36:44.549788+0000","last_peered":"2021-03-15T22:36:44.549788+0000","last_clean":"2021-03-15T22:36:44.549788+0000","last_became_active":"2021-03-15T22:36:34.382987+0000","last_became_peered":"2021-03-15T22:36:34.382987+0000","last_unstale":"2021-03-15T22:36:44.549788+0000","last_undegraded":"2021-03-15T22:36:44.549788+0000","last_fullsized":"2021-03-15T22:36:44.549788+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1039,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.598174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.598174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"1241","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367386+0000","last_change":"2021-03-15T22:33:43.514160+0000","last_active":"2021-03-15T22:36:45.367386+0000","last_peered":"2021-03-15T22:36:45.367386+0000","last_clean":"2021-03-15T22:36:45.367386+0000","last_became_active":"2021-03-15T22:33:04.336145+0000","last_became_peered":"2021-03-15T22:33:04.336145+0000","last_unstale":"2021-03-15T22:36:45.367386+0000","last_undegraded":"2021-03-15T22:36:45.367386+0000","last_fullsized":"2021-03-15T22:36:45.367386+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:33:43.514088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.514088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"574","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368963+0000","last_change":"2021-03-15T22:33:20.685041+0000","last_active":"2021-03-15T22:36:45.368963+0000","last_peered":"2021-03-15T22:36:45.368963+0000","last_clean":"2021-03-15T22:36:45.368963+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:36:45.368963+0000","last_undegraded":"2021-03-15T22:36:45.368963+0000","last_fullsized":"2021-03-15T22:36:45.368963+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:33:20.684976+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.684976+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"30","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367640+0000","last_change":"2021-03-15T22:36:24.391783+0000","last_active":"2021-03-15T22:36:45.367640+0000","last_peered":"2021-03-15T22:36:45.367640+0000","last_clean":"2021-03-15T22:36:45.367640+0000","last_became_active":"2021-03-15T22:36:24.391417+0000","last_became_peered":"2021-03-15T22:36:24.391417+0000","last_unstale":"2021-03-15T22:36:45.367640+0000","last_undegraded":"2021-03-15T22:36:45.367640+0000","last_fullsized":"2021-03-15T22:36:45.367640+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1041'1740","reported_seq":"4864","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549250+0000","last_change":"2021-03-15T22:36:35.375063+0000","last_active":"2021-03-15T22:36:45.549250+0000","last_peered":"2021-03-15T22:36:45.549250+0000","last_clean":"2021-03-15T22:36:45.549250+0000","last_became_active":"2021-03-15T22:34:58.337099+0000","last_became_peered":"2021-03-15T22:34:58.337099+0000","last_unstale":"2021-03-15T22:36:45.549250+0000","last_undegraded":"2021-03-15T22:36:45.549250+0000","last_fullsized":"2021-03-15T22:36:45.549250+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'1611","last_scrub_stamp":"2021-03-15T22:35:21.054192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.054192+0000","log_size":1740,"ondisk_log_size":1740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":45882320,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":561,"num_read_kb":593462,"num_write":354,"num_write_kb":124562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":226741522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1144","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367170+0000","last_change":"2021-03-15T22:36:12.386049+0000","last_active":"2021-03-15T22:36:45.367170+0000","last_peered":"2021-03-15T22:36:45.367170+0000","last_clean":"2021-03-15T22:36:45.367170+0000","last_became_active":"2021-03-15T22:36:12.385681+0000","last_became_peered":"2021-03-15T22:36:12.385681+0000","last_unstale":"2021-03-15T22:36:45.367170+0000","last_undegraded":"2021-03-15T22:36:45.367170+0000","last_fullsized":"2021-03-15T22:36:45.367170+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1017,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"850","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.369524+0000","last_change":"2021-03-15T22:36:30.397474+0000","last_active":"2021-03-15T22:36:45.369524+0000","last_peered":"2021-03-15T22:36:45.369524+0000","last_clean":"2021-03-15T22:36:45.369524+0000","last_became_active":"2021-03-15T22:36:30.390223+0000","last_became_peered":"2021-03-15T22:36:30.390223+0000","last_unstale":"2021-03-15T22:36:45.369524+0000","last_undegraded":"2021-03-15T22:36:45.369524+0000","last_fullsized":"2021-03-15T22:36:45.369524+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1035,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.870368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.870368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"480","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.515506+0000","last_change":"2021-03-15T22:35:08.402702+0000","last_active":"2021-03-15T22:36:41.515506+0000","last_peered":"2021-03-15T22:36:41.515506+0000","last_clean":"2021-03-15T22:36:41.515506+0000","last_became_active":"2021-03-15T22:34:10.349731+0000","last_became_peered":"2021-03-15T22:34:10.349731+0000","last_unstale":"2021-03-15T22:36:41.515506+0000","last_undegraded":"2021-03-15T22:36:41.515506+0000","last_fullsized":"2021-03-15T22:36:41.515506+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:35:08.402641+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:35:08.402641+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1094","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367211+0000","last_change":"2021-03-15T22:34:43.343527+0000","last_active":"2021-03-15T22:36:45.367211+0000","last_peered":"2021-03-15T22:36:45.367211+0000","last_clean":"2021-03-15T22:36:45.367211+0000","last_became_active":"2021-03-15T22:34:43.343255+0000","last_became_peered":"2021-03-15T22:34:43.343255+0000","last_unstale":"2021-03-15T22:36:45.367211+0000","last_undegraded":"2021-03-15T22:36:45.367211+0000","last_fullsized":"2021-03-15T22:36:45.367211+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.673338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.673338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"27","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.514749+0000","last_change":"2021-03-15T22:36:24.389841+0000","last_active":"2021-03-15T22:36:41.514749+0000","last_peered":"2021-03-15T22:36:41.514749+0000","last_clean":"2021-03-15T22:36:41.514749+0000","last_became_active":"2021-03-15T22:36:24.389490+0000","last_became_peered":"2021-03-15T22:36:24.389490+0000","last_unstale":"2021-03-15T22:36:41.514749+0000","last_undegraded":"2021-03-15T22:36:41.514749+0000","last_fullsized":"2021-03-15T22:36:41.514749+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1040'1519","reported_seq":"4900","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533347+0000","last_change":"2021-03-15T22:36:35.379125+0000","last_active":"2021-03-15T22:36:45.533347+0000","last_peered":"2021-03-15T22:36:45.533347+0000","last_clean":"2021-03-15T22:36:45.533347+0000","last_became_active":"2021-03-15T22:34:58.344406+0000","last_became_peered":"2021-03-15T22:34:58.344406+0000","last_unstale":"2021-03-15T22:36:45.533347+0000","last_undegraded":"2021-03-15T22:36:45.533347+0000","last_fullsized":"2021-03-15T22:36:45.533347+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'1407","last_scrub_stamp":"2021-03-15T22:35:43.758914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:43.758914+0000","log_size":1519,"ondisk_log_size":1519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":45609496,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":429,"num_read_kb":395184,"num_write":317,"num_write_kb":119059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":127958264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"990","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.518009+0000","last_change":"2021-03-15T22:34:55.356480+0000","last_active":"2021-03-15T22:36:41.518009+0000","last_peered":"2021-03-15T22:36:41.518009+0000","last_clean":"2021-03-15T22:36:41.518009+0000","last_became_active":"2021-03-15T22:34:55.356213+0000","last_became_peered":"2021-03-15T22:34:55.356213+0000","last_unstale":"2021-03-15T22:36:41.518009+0000","last_undegraded":"2021-03-15T22:36:41.518009+0000","last_fullsized":"2021-03-15T22:36:41.518009+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.729838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.729838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1359","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368106+0000","last_change":"2021-03-15T22:35:19.468482+0000","last_active":"2021-03-15T22:36:45.368106+0000","last_peered":"2021-03-15T22:36:45.368106+0000","last_clean":"2021-03-15T22:36:45.368106+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:36:45.368106+0000","last_undegraded":"2021-03-15T22:36:45.368106+0000","last_fullsized":"2021-03-15T22:36:45.368106+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:35:19.468402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.468402+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"478","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549160+0000","last_change":"2021-03-15T22:36:26.941424+0000","last_active":"2021-03-15T22:36:45.549160+0000","last_peered":"2021-03-15T22:36:45.549160+0000","last_clean":"2021-03-15T22:36:45.549160+0000","last_became_active":"2021-03-15T22:36:26.941149+0000","last_became_peered":"2021-03-15T22:36:26.941149+0000","last_unstale":"2021-03-15T22:36:45.549160+0000","last_undegraded":"2021-03-15T22:36:45.549160+0000","last_fullsized":"2021-03-15T22:36:45.549160+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1031,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.165838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.165838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"215","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549786+0000","last_change":"2021-03-15T22:36:21.293286+0000","last_active":"2021-03-15T22:36:45.549786+0000","last_peered":"2021-03-15T22:36:45.549786+0000","last_clean":"2021-03-15T22:36:45.549786+0000","last_became_active":"2021-03-15T22:33:45.317888+0000","last_became_peered":"2021-03-15T22:33:45.317888+0000","last_unstale":"2021-03-15T22:36:45.549786+0000","last_undegraded":"2021-03-15T22:36:45.549786+0000","last_fullsized":"2021-03-15T22:36:45.549786+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:36:21.293219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:21.293219+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"1040'530","reported_seq":"3509","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533384+0000","last_change":"2021-03-15T22:36:35.372629+0000","last_active":"2021-03-15T22:36:45.533384+0000","last_peered":"2021-03-15T22:36:45.533384+0000","last_clean":"2021-03-15T22:36:45.533384+0000","last_became_active":"2021-03-15T22:33:08.354872+0000","last_became_peered":"2021-03-15T22:33:08.354872+0000","last_unstale":"2021-03-15T22:36:45.533384+0000","last_undegraded":"2021-03-15T22:36:45.533384+0000","last_fullsized":"2021-03-15T22:36:45.533384+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'470","last_scrub_stamp":"2021-03-15T22:35:42.721100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.721100+0000","log_size":530,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20081853,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":190,"num_read_kb":197819,"num_write":106,"num_write_kb":42316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":31290769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"131","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368503+0000","last_change":"2021-03-15T22:34:43.340972+0000","last_active":"2021-03-15T22:36:45.368503+0000","last_peered":"2021-03-15T22:36:45.368503+0000","last_clean":"2021-03-15T22:36:45.368503+0000","last_became_active":"2021-03-15T22:34:43.340600+0000","last_became_peered":"2021-03-15T22:34:43.340600+0000","last_unstale":"2021-03-15T22:36:45.368503+0000","last_undegraded":"2021-03-15T22:36:45.368503+0000","last_fullsized":"2021-03-15T22:36:45.368503+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:09.612126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.612126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"1088","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.518776+0000","last_change":"2021-03-15T22:35:21.917891+0000","last_active":"2021-03-15T22:36:41.518776+0000","last_peered":"2021-03-15T22:36:41.518776+0000","last_clean":"2021-03-15T22:36:41.518776+0000","last_became_active":"2021-03-15T22:33:08.352367+0000","last_became_peered":"2021-03-15T22:33:08.352367+0000","last_unstale":"2021-03-15T22:36:41.518776+0000","last_undegraded":"2021-03-15T22:36:41.518776+0000","last_fullsized":"2021-03-15T22:36:41.518776+0000","mapping_epoch":837,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'9","last_scrub_stamp":"2021-03-15T22:35:21.917819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.917819+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"10","reported_epoch":"1046","state":"active+clean","last_fresh":"2021-03-15T22:36:42.665723+0000","last_change":"2021-03-15T22:36:42.665723+0000","last_active":"2021-03-15T22:36:42.665723+0000","last_peered":"2021-03-15T22:36:42.665723+0000","last_clean":"2021-03-15T22:36:42.665723+0000","last_became_active":"2021-03-15T22:36:42.665274+0000","last_became_peered":"2021-03-15T22:36:42.665274+0000","last_unstale":"2021-03-15T22:36:42.665723+0000","last_undegraded":"2021-03-15T22:36:42.665723+0000","last_fullsized":"2021-03-15T22:36:42.665723+0000","mapping_epoch":1045,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1046,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:24.285991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_clean_scrub_stamp":"2021-03-15T22:36:24.285991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"1220","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367971+0000","last_change":"2021-03-15T22:35:21.384129+0000","last_active":"2021-03-15T22:36:45.367971+0000","last_peered":"2021-03-15T22:36:45.367971+0000","last_clean":"2021-03-15T22:36:45.367971+0000","last_became_active":"2021-03-15T22:34:10.330608+0000","last_became_peered":"2021-03-15T22:34:10.330608+0000","last_unstale":"2021-03-15T22:36:45.367971+0000","last_undegraded":"2021-03-15T22:36:45.367971+0000","last_fullsized":"2021-03-15T22:36:45.367971+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:35:21.384037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.384037+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"27","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.514453+0000","last_change":"2021-03-15T22:36:24.389116+0000","last_active":"2021-03-15T22:36:41.514453+0000","last_peered":"2021-03-15T22:36:41.514453+0000","last_clean":"2021-03-15T22:36:41.514453+0000","last_became_active":"2021-03-15T22:36:24.388690+0000","last_became_peered":"2021-03-15T22:36:24.388690+0000","last_unstale":"2021-03-15T22:36:41.514453+0000","last_undegraded":"2021-03-15T22:36:41.514453+0000","last_fullsized":"2021-03-15T22:36:41.514453+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"1036'508","reported_seq":"2991","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.511307+0000","last_change":"2021-03-15T22:36:35.367176+0000","last_active":"2021-03-15T22:36:41.511307+0000","last_peered":"2021-03-15T22:36:41.511307+0000","last_clean":"2021-03-15T22:36:41.511307+0000","last_became_active":"2021-03-15T22:34:58.350923+0000","last_became_peered":"2021-03-15T22:34:58.350923+0000","last_unstale":"2021-03-15T22:36:41.511307+0000","last_undegraded":"2021-03-15T22:36:41.511307+0000","last_fullsized":"2021-03-15T22:36:41.511307+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"1027'503","last_scrub_stamp":"2021-03-15T22:36:24.720629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:24.720629+0000","log_size":508,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8377674,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":158,"num_read_kb":164364,"num_write":122,"num_write_kb":40554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":52707265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"34b","length":"1"}]},{"pgid":"6.f","version":"0'0","reported_seq":"798","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368480+0000","last_change":"2021-03-15T22:36:12.855165+0000","last_active":"2021-03-15T22:36:45.368480+0000","last_peered":"2021-03-15T22:36:45.368480+0000","last_clean":"2021-03-15T22:36:45.368480+0000","last_became_active":"2021-03-15T22:34:55.345267+0000","last_became_peered":"2021-03-15T22:34:55.345267+0000","last_unstale":"2021-03-15T22:36:45.368480+0000","last_undegraded":"2021-03-15T22:36:45.368480+0000","last_fullsized":"2021-03-15T22:36:45.368480+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:12.855079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:36:12.855079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"1085","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549567+0000","last_change":"2021-03-15T22:35:12.091714+0000","last_active":"2021-03-15T22:36:45.549567+0000","last_peered":"2021-03-15T22:36:45.549567+0000","last_clean":"2021-03-15T22:36:45.549567+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:36:45.549567+0000","last_undegraded":"2021-03-15T22:36:45.549567+0000","last_fullsized":"2021-03-15T22:36:45.549567+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.091650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.091650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"259","reported_epoch":"1047","state":"active+clean","last_fresh":"2021-03-15T22:36:44.755534+0000","last_change":"2021-03-15T22:36:18.384469+0000","last_active":"2021-03-15T22:36:44.755534+0000","last_peered":"2021-03-15T22:36:44.755534+0000","last_clean":"2021-03-15T22:36:44.755534+0000","last_became_active":"2021-03-15T22:36:18.384088+0000","last_became_peered":"2021-03-15T22:36:18.384088+0000","last_unstale":"2021-03-15T22:36:44.755534+0000","last_undegraded":"2021-03-15T22:36:44.755534+0000","last_fullsized":"2021-03-15T22:36:44.755534+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.678246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"1199","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367776+0000","last_change":"2021-03-15T22:36:15.261768+0000","last_active":"2021-03-15T22:36:45.367776+0000","last_peered":"2021-03-15T22:36:45.367776+0000","last_clean":"2021-03-15T22:36:45.367776+0000","last_became_active":"2021-03-15T22:34:55.341958+0000","last_became_peered":"2021-03-15T22:34:55.341958+0000","last_unstale":"2021-03-15T22:36:45.367776+0000","last_undegraded":"2021-03-15T22:36:45.367776+0000","last_fullsized":"2021-03-15T22:36:45.367776+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:36:15.261676+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:36:15.261676+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1077","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:41.515268+0000","last_change":"2021-03-15T22:36:24.385650+0000","last_active":"2021-03-15T22:36:41.515268+0000","last_peered":"2021-03-15T22:36:41.515268+0000","last_clean":"2021-03-15T22:36:41.515268+0000","last_became_active":"2021-03-15T22:36:24.383694+0000","last_became_peered":"2021-03-15T22:36:24.383694+0000","last_unstale":"2021-03-15T22:36:41.515268+0000","last_undegraded":"2021-03-15T22:36:41.515268+0000","last_fullsized":"2021-03-15T22:36:41.515268+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"1041'1742","reported_seq":"4965","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.370026+0000","last_change":"2021-03-15T22:36:35.389598+0000","last_active":"2021-03-15T22:36:45.370026+0000","last_peered":"2021-03-15T22:36:45.370026+0000","last_clean":"2021-03-15T22:36:45.370026+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:36:45.370026+0000","last_undegraded":"2021-03-15T22:36:45.370026+0000","last_fullsized":"2021-03-15T22:36:45.370026+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"1023'1699","last_scrub_stamp":"2021-03-15T22:36:20.059745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.059745+0000","log_size":1742,"ondisk_log_size":1742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55707898,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":552,"num_read_kb":589646,"num_write":381,"num_write_kb":135091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1064","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368120+0000","last_change":"2021-03-15T22:36:25.253269+0000","last_active":"2021-03-15T22:36:45.368120+0000","last_peered":"2021-03-15T22:36:45.368120+0000","last_clean":"2021-03-15T22:36:45.368120+0000","last_became_active":"2021-03-15T22:34:55.340498+0000","last_became_peered":"2021-03-15T22:34:55.340498+0000","last_unstale":"2021-03-15T22:36:45.368120+0000","last_undegraded":"2021-03-15T22:36:45.368120+0000","last_fullsized":"2021-03-15T22:36:45.368120+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:25.253188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:36:25.253188+0000","last_clean_scrub_stamp":"2021-03-15T22:36:25.253188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"976'19","reported_seq":"460","reported_epoch":"1045","state":"active+clean","last_fresh":"2021-03-15T22:36:42.199295+0000","last_change":"2021-03-15T22:36:09.696306+0000","last_active":"2021-03-15T22:36:42.199295+0000","last_peered":"2021-03-15T22:36:42.199295+0000","last_clean":"2021-03-15T22:36:42.199295+0000","last_became_active":"2021-03-15T22:35:28.364764+0000","last_became_peered":"2021-03-15T22:35:28.364764+0000","last_unstale":"2021-03-15T22:36:42.199295+0000","last_undegraded":"2021-03-15T22:36:42.199295+0000","last_fullsized":"2021-03-15T22:36:42.199295+0000","mapping_epoch":972,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":973,"parent":"0.0","parent_split_bits":0,"last_scrub":"976'19","last_scrub_stamp":"2021-03-15T22:36:09.696246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:09.696246+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"84'2","reported_seq":"495","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367718+0000","last_change":"2021-03-15T22:36:14.388343+0000","last_active":"2021-03-15T22:36:45.367718+0000","last_peered":"2021-03-15T22:36:45.367718+0000","last_clean":"2021-03-15T22:36:45.367718+0000","last_became_active":"2021-03-15T22:36:14.387999+0000","last_became_peered":"2021-03-15T22:36:14.387999+0000","last_unstale":"2021-03-15T22:36:45.367718+0000","last_undegraded":"2021-03-15T22:36:45.367718+0000","last_fullsized":"2021-03-15T22:36:45.367718+0000","mapping_epoch":589,"log_start":"84'2","ondisk_log_start":"84'2","created":56,"last_epoch_clean":1019,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.690309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"1238","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367728+0000","last_change":"2021-03-15T22:35:46.406478+0000","last_active":"2021-03-15T22:36:45.367728+0000","last_peered":"2021-03-15T22:36:45.367728+0000","last_clean":"2021-03-15T22:36:45.367728+0000","last_became_active":"2021-03-15T22:33:04.337113+0000","last_became_peered":"2021-03-15T22:33:04.337113+0000","last_unstale":"2021-03-15T22:36:45.367728+0000","last_undegraded":"2021-03-15T22:36:45.367728+0000","last_fullsized":"2021-03-15T22:36:45.367728+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:35:46.406376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:46.406376+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":831822344,"num_objects":364,"num_object_clones":314,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":364,"num_whiteouts":12,"num_read":7452,"num_read_kb":7660922,"num_write":5407,"num_write_kb":2092813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4621,"num_bytes_recovered":2230024638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1829830656,"data_stored":2535132102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24603,"ondisk_log_size":24603,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1114,"num_read_kb":1109,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1482,"num_read_kb":987,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":94766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":947,"ondisk_log_size":947,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299622,"num_pgs":65,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":289396,"kb_used_data":241836,"kb_used_omap":0,"kb_used_meta":47552,"kb_avail":93455756,"statfs":{"total":95995035648,"available":95698694144,"internally_reserved":0,"allocated":247640064,"data_stored":358942578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48693248},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.47199999999999998,"15min":0.46899999999999997},"min":{"1min":0.25,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.73},"last":0.622},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.222},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":3.8799999999999999},"last":0.54900000000000004}]},{"osd":1,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.47099999999999997},"min":{"1min":0.34399999999999997,"5min":0.249,"15min":0.249},"max":{"1min":0.58799999999999997,"5min":0.73899999999999999,"15min":2.6859999999999999},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.56299999999999994,"15min":0.51500000000000001},"min":{"1min":0.311,"5min":0.28399999999999997,"15min":0.23400000000000001},"max":{"1min":0.67100000000000004,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":0.50700000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:36:38 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.83799999999999997,"15min":0.59499999999999997},"min":{"1min":0.33000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.65900000000000003,"5min":29.350999999999999,"15min":29.350999999999999},"last":0.64100000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.33500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65900000000000003,"5min":1.298,"15min":1.298},"last":0.67100000000000004}]},{"osd":3,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.33500000000000002,"5min":0.314,"15min":0.28899999999999998},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.702},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.30099999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.806,"5min":1.806,"15min":5.415},"last":0.65600000000000003}]},{"osd":4,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.436,"15min":0.439},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.23799999999999999},"max":{"1min":0.63500000000000001,"5min":0.753,"15min":1.927},"last":0.40600000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.47399999999999998,"15min":0.46200000000000002},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":0.68600000000000005,"5min":0.72699999999999998,"15min":2.5339999999999998},"last":0.45500000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:36:38 2021","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.48999999999999999,"15min":0.46000000000000002},"min":{"1min":0.33400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.74299999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.503},"min":{"1min":0.35999999999999999,"5min":0.29699999999999999,"15min":0.27900000000000003},"max":{"1min":0.68100000000000005,"5min":0.77700000000000002,"15min":1.071},"last":0.502}]},{"osd":6,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.29999999999999999,"15min":0.27100000000000002},"max":{"1min":0.77800000000000002,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54200000000000004,"15min":0.51900000000000002},"min":{"1min":0.38200000000000001,"5min":0.33200000000000002,"15min":0.28399999999999997},"max":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":1.8560000000000001},"last":0.73099999999999998}]}]},{"osd":6,"up_from":45,"seq":193273528554,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":245712,"kb_used_data":193288,"kb_used_omap":0,"kb_used_meta":52416,"kb_avail":93499440,"statfs":{"total":95995035648,"available":95743426560,"internally_reserved":0,"allocated":197926912,"data_stored":260369653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53673984},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.49399999999999999,"15min":0.47999999999999998},"min":{"1min":0.33900000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57099999999999995,"5min":2.5470000000000002,"15min":3.085},"last":0.41699999999999998},{"interface":"front","average":{"1min":0.436,"5min":0.47299999999999998,"15min":0.46000000000000002},"min":{"1min":0.315,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69699999999999995,"5min":1.137,"15min":1.323},"last":0.45800000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.54200000000000004,"15min":0.54800000000000004},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.254},"max":{"1min":0.61099999999999999,"5min":2.7839999999999998,"15min":14.637},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.44700000000000001,"15min":0.46899999999999997},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60999999999999999,"5min":0.84499999999999997,"15min":5.2450000000000001},"last":0.38}]},{"osd":2,"last update":"Mon Mar 15 22:35:43 2021","interfaces":[{"interface":"back","average":{"1min":0.441,"5min":0.46899999999999997,"15min":0.46300000000000002},"min":{"1min":0.25800000000000001,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.64800000000000002,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.502},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.52600000000000002,"15min":0.50900000000000001},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.70899999999999996,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.53300000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.52600000000000002,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.29899999999999999},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.6150000000000002},"last":0.625},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.498,"15min":0.496},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.65100000000000002,"5min":2.4670000000000001,"15min":2.7890000000000001},"last":0.48899999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.44900000000000001,"15min":0.437},"min":{"1min":0.26900000000000002,"5min":0.25,"15min":0.224},"max":{"1min":0.70499999999999996,"5min":0.94099999999999995,"15min":4.306},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.47099999999999997,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.26600000000000001,"15min":0.23499999999999999},"max":{"1min":0.65300000000000002,"5min":1.091,"15min":4.181},"last":0.65800000000000003}]},{"osd":5,"last update":"Mon Mar 15 22:36:41 2021","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.49099999999999999,"15min":0.435},"min":{"1min":0.39600000000000002,"5min":0.30599999999999999,"15min":0.26700000000000002},"max":{"1min":0.73899999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.64800000000000002},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54500000000000004,"15min":0.47999999999999998},"min":{"1min":0.42199999999999999,"5min":0.317,"15min":0.30199999999999999},"max":{"1min":0.86599999999999999,"5min":1.022,"15min":1.022},"last":0.68000000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:36:25 2021","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47599999999999998,"15min":0.52600000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.249},"max":{"1min":0.63600000000000001,"5min":0.76300000000000001,"15min":15.340999999999999},"last":0.59399999999999997},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56999999999999995,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.35399999999999998,"15min":0.27200000000000002},"max":{"1min":0.78200000000000003,"5min":1.0109999999999999,"15min":15.356},"last":0.71299999999999997}]}]},{"osd":1,"up_from":13,"seq":55834575100,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":185652,"kb_used_data":128556,"kb_used_omap":0,"kb_used_meta":57088,"kb_avail":93559500,"statfs":{"total":95995035648,"available":95804928000,"internally_reserved":0,"allocated":131641344,"data_stored":174251752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58458112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:56 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.372,"15min":0.34000000000000002},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":0.50800000000000001,"5min":4.3920000000000003,"15min":4.3920000000000003},"last":0.312},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.42499999999999999,"15min":0.37},"min":{"1min":0.215,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.52500000000000002,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.249}]},{"osd":2,"last update":"Mon Mar 15 22:35:45 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.46899999999999997,"15min":0.379},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59199999999999997,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45500000000000002,"15min":0.40100000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64600000000000002,"5min":5.0190000000000001,"15min":5.0190000000000001},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:36:21 2021","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.47999999999999998,"15min":0.40100000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.215},"max":{"1min":0.76500000000000001,"5min":7.718,"15min":7.718},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.51800000000000002,"15min":0.432},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.21199999999999999},"max":{"1min":0.54400000000000004,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":0.27700000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.61199999999999999,"15min":0.55100000000000005},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.22800000000000001},"max":{"1min":0.78800000000000003,"5min":4.593,"15min":4.593},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58599999999999997,"15min":0.57699999999999996},"min":{"1min":0.38,"5min":0.35499999999999998,"15min":0.253},"max":{"1min":0.67100000000000004,"5min":4.3140000000000001,"15min":4.6139999999999999},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:36:37 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58699999999999997,"15min":0.54800000000000004},"min":{"1min":0.38200000000000001,"5min":0.34499999999999997,"15min":0.34300000000000003},"max":{"1min":0.79800000000000004,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.60399999999999998,"15min":0.53200000000000003},"min":{"1min":0.36499999999999999,"5min":0.33600000000000002,"15min":0.32100000000000001},"max":{"1min":0.81000000000000005,"5min":3.472,"15min":3.472},"last":0.53300000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:35:59 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.60799999999999998,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.38500000000000001,"15min":0.314},"max":{"1min":0.755,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.621},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.63100000000000001,"15min":0.57899999999999996},"min":{"1min":0.41199999999999998,"5min":0.375,"15min":0.33700000000000002},"max":{"1min":0.68300000000000005,"5min":4.6529999999999996,"15min":4.6529999999999996},"last":0.64800000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:36:21 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.63600000000000001,"15min":0.59999999999999998},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.32100000000000001},"max":{"1min":0.71899999999999997,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.66500000000000004,"15min":0.59399999999999997},"min":{"1min":0.45400000000000001,"5min":0.40300000000000002,"15min":0.35799999999999998},"max":{"1min":0.73099999999999998,"5min":4.625,"15min":4.625},"last":0.57799999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738622,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":324980,"kb_used_data":275884,"kb_used_omap":0,"kb_used_meta":49088,"kb_avail":93420172,"statfs":{"total":95995035648,"available":95662256128,"internally_reserved":0,"allocated":282505216,"data_stored":400948051,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50266112},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:35:47 2021","interfaces":[{"interface":"back","average":{"1min":0.35399999999999998,"5min":0.35899999999999999,"15min":0.34000000000000002},"min":{"1min":0.249,"5min":0.192,"15min":0.17999999999999999},"max":{"1min":0.68000000000000005,"5min":1.46,"15min":1.7070000000000001},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.36699999999999999,"5min":0.39600000000000002,"15min":0.36899999999999999},"min":{"1min":0.20000000000000001,"5min":0.20000000000000001,"15min":0.19900000000000001},"max":{"1min":0.499,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.44400000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:36:41 2021","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.371,"15min":0.36299999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":0.53600000000000003,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.34200000000000003},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.48499999999999999,"15min":0.41199999999999998},"min":{"1min":0.247,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.55100000000000005,"5min":7.3799999999999999,"15min":7.3799999999999999},"last":0.38800000000000001}]},{"osd":3,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":0.71399999999999997,"5min":1.544,"15min":1.544},"last":0.38200000000000001},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.46700000000000003,"15min":0.41999999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.60399999999999998,"5min":3.988,"15min":3.988},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.55200000000000005,"15min":0.53700000000000003},"min":{"1min":0.34799999999999998,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":0.76100000000000001,"5min":1.861,"15min":1.861},"last":0.65200000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.36799999999999999,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":0.78300000000000003,"5min":1.5700000000000001,"15min":1.645},"last":0.68100000000000005}]},{"osd":5,"last update":"Mon Mar 15 22:36:36 2021","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61699999999999999,"15min":0.54100000000000004},"min":{"1min":0.38400000000000001,"5min":0.33900000000000002,"15min":0.29899999999999999},"max":{"1min":1.6459999999999999,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.75},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56200000000000006,"15min":0.50600000000000001},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.318},"max":{"1min":0.81000000000000005,"5min":1.909,"15min":1.909},"last":0.58899999999999997}]},{"osd":6,"last update":"Mon Mar 15 22:36:01 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.57399999999999995,"15min":0.55600000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.26200000000000001},"max":{"1min":0.78800000000000003,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.59299999999999997,"15min":0.56899999999999995},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.27100000000000002},"max":{"1min":0.77300000000000002,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.73199999999999998}]},{"osd":7,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.65000000000000002,"15min":0.58499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.27900000000000003},"max":{"1min":0.82499999999999996,"5min":3.8239999999999998,"15min":3.8239999999999998},"last":0.78500000000000003},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.60899999999999999,"15min":0.57699999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.317},"max":{"1min":0.72499999999999998,"5min":2.665,"15min":2.665},"last":0.77700000000000002}]}]},{"osd":2,"up_from":681,"seq":2924872728652,"num_pgs":51,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":362304,"kb_used_data":324152,"kb_used_omap":119,"kb_used_meta":38024,"kb_avail":93382848,"statfs":{"total":95995035648,"available":95624036352,"internally_reserved":0,"allocated":331931648,"data_stored":444149642,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":38936937},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.316,"15min":0.31},"min":{"1min":0.183,"5min":0.183,"15min":0.183},"max":{"1min":0.56299999999999994,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.35299999999999998,"15min":0.33100000000000002},"min":{"1min":0.223,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":0.77800000000000002,"5min":0.98699999999999999,"15min":0.98699999999999999},"last":0.35099999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.33200000000000002,"5min":0.35599999999999998,"15min":0.33400000000000002},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.58099999999999996,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.29799999999999999},{"interface":"front","average":{"1min":0.34799999999999998,"5min":0.379,"15min":0.36299999999999999},"min":{"1min":0.22900000000000001,"5min":0.221,"15min":0.221},"max":{"1min":0.59799999999999998,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.33200000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.42099999999999999,"15min":0.36499999999999999},"min":{"1min":0.26200000000000001,"5min":0.216,"15min":0.216},"max":{"1min":0.65500000000000003,"5min":3.496,"15min":3.496},"last":0.376},{"interface":"front","average":{"1min":0.38800000000000001,"5min":0.38300000000000001,"15min":0.36199999999999999},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":0.85099999999999998,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.312}]},{"osd":4,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.51700000000000002,"5min":0.54800000000000004,"15min":0.49399999999999999},"min":{"1min":0.30399999999999999,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.67600000000000005,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.53800000000000003,"15min":0.499},"min":{"1min":0.378,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.76700000000000002,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.5}]},{"osd":5,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.52200000000000002,"15min":0.504},"min":{"1min":0.35499999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.875,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.52700000000000002,"5min":0.56100000000000005,"15min":0.51900000000000002},"min":{"1min":0.38500000000000001,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.70299999999999996,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.57399999999999995}]},{"osd":6,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.55200000000000005,"5min":0.57599999999999996,"15min":0.51900000000000002},"min":{"1min":0.372,"5min":0.33500000000000002,"15min":0.33500000000000002},"max":{"1min":0.76600000000000001,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.56299999999999994,"5min":0.57699999999999996,"15min":0.52500000000000002},"min":{"1min":0.433,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.77900000000000003,"5min":1.387,"15min":1.387},"last":0.55100000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:35:44 2021","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.59399999999999997,"15min":0.54300000000000004},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.748,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.54200000000000004},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.57599999999999996,"15min":0.53200000000000003},"min":{"1min":0.40100000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":0.82599999999999996,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.58499999999999996}]}]},{"osd":3,"up_from":26,"seq":111669149940,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333160,"kb_used_data":266464,"kb_used_omap":0,"kb_used_meta":66688,"kb_avail":93411992,"statfs":{"total":95995035648,"available":95653879808,"internally_reserved":0,"allocated":272859136,"data_stored":368178842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68288512},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:24 2021","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.17699999999999999},"max":{"1min":0.39500000000000002,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.32600000000000001,"15min":0.33600000000000002},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.41999999999999998,"5min":0.76900000000000002,"15min":2.6339999999999999},"last":0.67900000000000005}]},{"osd":1,"last update":"Mon Mar 15 22:36:24 2021","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36899999999999999,"15min":0.39200000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55900000000000005,"5min":0.626,"15min":6.0199999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.37,"15min":0.39900000000000002},"min":{"1min":0.25900000000000001,"5min":0.219,"15min":0.20699999999999999},"max":{"1min":0.48599999999999999,"5min":2.5619999999999998,"15min":7.6130000000000004},"last":0.63200000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:36:42 2021","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35399999999999998,"15min":0.35499999999999998},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.216},"max":{"1min":0.58099999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.372,"5min":0.38700000000000001,"15min":0.38800000000000001},"min":{"1min":0.28999999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70899999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.70899999999999996}]},{"osd":4,"last update":"Mon Mar 15 22:36:39 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.53100000000000003},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.22600000000000001},"max":{"1min":0.66900000000000004,"5min":0.86899999999999999,"15min":3.3370000000000002},"last":0.999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54000000000000004,"15min":0.52400000000000002},"min":{"1min":0.36599999999999999,"5min":0.311,"15min":0.26700000000000002},"max":{"1min":0.80900000000000005,"5min":0.84499999999999997,"15min":2.2890000000000001},"last":0.79300000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:36:39 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53500000000000003,"15min":0.73299999999999998},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.28299999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":9.4870000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54100000000000004,"15min":0.91500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.32800000000000001},"max":{"1min":0.64000000000000001,"5min":0.84799999999999998,"15min":15.776},"last":1.0269999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:36:06 2021","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38900000000000001,"5min":0.38200000000000001,"15min":0.30099999999999999},"max":{"1min":0.81699999999999995,"5min":1.359,"15min":2.6499999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.38900000000000001,"5min":0.375,"15min":0.30399999999999999},"max":{"1min":0.78500000000000003,"5min":1.3400000000000001,"15min":3.472},"last":0.92800000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:36:24 2021","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55100000000000005,"15min":0.57499999999999996},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.29599999999999999},"max":{"1min":0.752,"5min":0.91200000000000003,"15min":5.625},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.57599999999999996,"15min":0.56200000000000006},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73199999999999998,"5min":2.621,"15min":2.621},"last":0.89600000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986417,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":155840,"kb_used_data":105208,"kb_used_omap":0,"kb_used_meta":50624,"kb_avail":93589312,"statfs":{"total":95995035648,"available":95835455488,"internally_reserved":0,"allocated":107732992,"data_stored":153871734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51838976},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:32 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.46500000000000002,"15min":0.48199999999999998},"min":{"1min":0.32600000000000001,"5min":0.221,"15min":0.20000000000000001},"max":{"1min":0.748,"5min":0.748,"15min":4.7050000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.47099999999999997,"15min":0.499},"min":{"1min":0.28499999999999998,"5min":0.27100000000000002,"15min":0.246},"max":{"1min":0.68799999999999994,"5min":1.4179999999999999,"15min":5.0209999999999999},"last":0.60199999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:36:32 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.34699999999999998,"5min":0.25,"15min":0.23999999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":4.3499999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.48499999999999999,"15min":0.495},"min":{"1min":0.34599999999999997,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":0.72899999999999998,"5min":0.94599999999999995,"15min":4.718},"last":0.51500000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.48399999999999999,"15min":0.503},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.49099999999999999,"15min":0.497},"min":{"1min":0.34100000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.748,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.48099999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:36:32 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.28499999999999998},"max":{"1min":0.69399999999999995,"5min":1.486,"15min":5.0499999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55500000000000005,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71099999999999997,"5min":1.4950000000000001,"15min":4.4050000000000002},"last":0.71499999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:36:32 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.67000000000000004,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.628},{"interface":"front","average":{"1min":0.503,"5min":0.47299999999999998,"15min":0.442},"min":{"1min":0.308,"5min":0.29099999999999998,"15min":0.24099999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.77000000000000002},"last":0.55300000000000005}]},{"osd":6,"last update":"Mon Mar 15 22:35:59 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.27500000000000002,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":4.3079999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.36199999999999999,"5min":0.33000000000000002,"15min":0.28000000000000003},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":4.6600000000000001},"last":0.68600000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:36:21 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.503,"15min":0.505},"min":{"1min":0.34799999999999998,"5min":0.28299999999999997,"15min":0.26300000000000001},"max":{"1min":0.71699999999999997,"5min":0.79900000000000004,"15min":4.6929999999999996},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52100000000000002,"15min":0.54200000000000004},"min":{"1min":0.40899999999999997,"5min":0.29999999999999999,"15min":0.25},"max":{"1min":0.66200000000000003,"5min":1.034,"15min":4.9900000000000002},"last":0.64700000000000002}]}]},{"osd":5,"up_from":616,"seq":2645699854425,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":300928,"kb_used_data":269560,"kb_used_omap":153,"kb_used_meta":31206,"kb_avail":93444224,"statfs":{"total":95995035648,"available":95686885376,"internally_reserved":0,"allocated":276029440,"data_stored":385825610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":31955473},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.51100000000000001,"15min":0.47499999999999998},"min":{"1min":0.307,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.62,"5min":0.503,"15min":0.46700000000000003},"min":{"1min":0.377,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.5939999999999999,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":0.45500000000000002}]},{"osd":1,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58099999999999996,"15min":0.49399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.27500000000000002},"max":{"1min":2.6699999999999999,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":0.371},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.48599999999999999,"15min":0.48699999999999999},"min":{"1min":0.35899999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":1.161,"15min":1.161},"last":0.47299999999999998}]},{"osd":2,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.58699999999999997,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71899999999999997,"5min":10.952,"15min":10.952},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.39800000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71399999999999997,"5min":1.125,"15min":1.125},"last":0.44}]},{"osd":3,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.68600000000000005,"15min":0.54400000000000004},"min":{"1min":0.308,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76400000000000001,"5min":10.355,"15min":10.355},"last":0.54800000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51200000000000001,"15min":0.46400000000000002},"min":{"1min":0.32800000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77100000000000002,"5min":2.476,"15min":2.476},"last":0.42399999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.505,"15min":0.41899999999999998},"min":{"1min":0.29099999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.68500000000000005,"5min":7.4000000000000004,"15min":7.4000000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.49199999999999999,"15min":0.41999999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.5739999999999998,"5min":2.5739999999999998,"15min":2.5739999999999998},"last":0.52700000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52400000000000002,"15min":0.47299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23999999999999999},"max":{"1min":2.613,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.53700000000000003},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.58299999999999996,"15min":0.504},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.29699999999999999},"max":{"1min":2.698,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.58599999999999997}]},{"osd":7,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.54600000000000004,"15min":0.47499999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.58499999999999996,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75600000000000001,"5min":5.944,"15min":5.944},"last":0.505}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":12026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":10320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":8144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280162304,"data_stored":399514695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129228800,"data_stored":172809126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329744384,"data_stored":442737932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270520320,"data_stored":366746430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":105381888,"data_stored":152438267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":273838080,"data_stored":384413786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":195596288,"data_stored":258945082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":245358592,"data_stored":357526784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:36:52.458 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:52.484 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:52.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:52.486 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:52.587 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:52.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:52.611 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:52.611 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:52.712 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:52.739 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:52.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:52.740 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:52.841 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:52.867 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:52.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:52.868 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:52.958 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:52 smithi076 conmon[62156]: cluster 2021-03-15T22:36:51.083501+0000 osd.3 (osd.3) 182 : cluster [DBG] 5.17 scrub ok 2021-03-15T22:36:52.959 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:52 smithi076 conmon[62156]: cluster 2021-03-15T22:36:51.241343+0000 mgr.y (mgr.14144) 1142 : cluster [DBG] pgmap v1724: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 763 B/s rd, 0 op/s 2021-03-15T22:36:52.959 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:52 smithi076 conmon[62156]: cluster 2021-03-15T22:36:51.269108+0000 osd.5 (osd.5) 70 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:36:52.959 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:52 smithi076 conmon[62156]: cluster 2021-03-15T22:36:51.568520+0000 osd.6 (osd.6) 159 : cluster [DBG] 1.6 scrub ok 2021-03-15T22:36:52.959 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:52 smithi076 conmon[62156]: audit 2021-03-15T22:36:51.679429+0000 mgr.y (mgr.14144) 1143 : audit [DBG] from='client.17013 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:52.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:52 smithi076 conmon[56514]: cluster 2021-03-15T22:36:51.083501+0000 osd.3 (osd.3) 182 : cluster [DBG] 5.17 scrub ok 2021-03-15T22:36:52.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:52 smithi076 conmon[56514]: cluster 2021-03-15T22:36:51.241343+0000 mgr.y (mgr.14144) 1142 : cluster [DBG] pgmap v1724: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 763 B/s rd, 0 op/s 2021-03-15T22:36:52.960 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:52 smithi076 conmon[56514]: cluster 2021-03-15T22:36:51.269108+0000 osd.5 (osd.5) 70 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:36:52.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:52 smithi076 conmon[56514]: cluster 2021-03-15T22:36:51.568520+0000 osd.6 (osd.6) 159 : cluster [DBG] 1.6 scrub ok 2021-03-15T22:36:52.961 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:52 smithi076 conmon[56514]: audit 2021-03-15T22:36:51.679429+0000 mgr.y (mgr.14144) 1143 : audit [DBG] from='client.17013 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:52.969 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:52.995 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:52.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:52.996 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:53.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: cluster 2021-03-15T22:36:51.083501+0000 osd.3 (osd.3) 182 2021-03-15T22:36:53.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: : cluster [DBG] 5.17 scrub ok 2021-03-15T22:36:53.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: cluster 2021-03-15T22:36:51.241343+0000 mgr.y (mgr.14144) 1142 : cluster [DBG] pgmap v1724: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 763 B/s rd, 0 op/s 2021-03-15T22:36:53.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: cluster 2021-03-15T22:36:51.269108+0000 osd.5 (osd.5) 2021-03-15T22:36:53.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: 70 : cluster [DBG] 4.8 scrub ok 2021-03-15T22:36:53.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:36:53.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: -15T22:36:51.568520+0000 osd.6 2021-03-15T22:36:53.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: (osd.6) 159 : cluster [DBG] 1.6 scrub ok 2021-03-15T22:36:53.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: 2021-03-15T22:36:53.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: audit 2021-03-15T22:36:51 2021-03-15T22:36:53.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: .679429+0000 mgr.y (mgr.14144) 1143 : audit 2021-03-15T22:36:53.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:52 smithi125 conmon[56739]: [DBG] from='client.17013 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2021-03-15T22:36:53.047 INFO:teuthology.orchestra.run.smithi076.stdout:{"pg_ready":true,"pg_map":{"version":1724,"stamp":"2021-03-15T22:36:51.240946+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":831847580,"num_objects":680,"num_object_clones":314,"num_object_copies":2040,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":680,"num_whiteouts":12,"num_read":10055,"num_read_kb":7663025,"num_write":6243,"num_write_kb":2092978,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5900,"num_bytes_recovered":2230124932,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":25690,"ondisk_log_size":25690,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":2198308,"kb_used_data":1805092,"kb_used_omap":273,"kb_used_meta":392878,"kb_avail":747762908,"statfs":{"total":767960285184,"available":765709217792,"internally_reserved":0,"allocated":1848414208,"data_stored":2546633544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":279686,"internal_metadata":402307962},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.706053"},"pg_stats":[{"pgid":"5.19","version":"317'7","reported_seq":"417","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.548919+0000","last_change":"2021-03-15T22:36:18.276167+0000","last_active":"2021-03-15T22:36:45.548919+0000","last_peered":"2021-03-15T22:36:45.548919+0000","last_clean":"2021-03-15T22:36:45.548919+0000","last_became_active":"2021-03-15T22:31:23.256594+0000","last_became_peered":"2021-03-15T22:31:23.256594+0000","last_unstale":"2021-03-15T22:36:45.548919+0000","last_undegraded":"2021-03-15T22:36:45.548919+0000","last_fullsized":"2021-03-15T22:36:45.548919+0000","mapping_epoch":733,"log_start":"89'3","ondisk_log_start":"89'3","created":728,"last_epoch_clean":734,"parent":"0.0","parent_split_bits":5,"last_scrub":"317'7","last_scrub_stamp":"2021-03-15T22:36:18.276099+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:36:18.276099+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":"1064","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367139+0000","last_change":"2021-03-15T22:34:10.331764+0000","last_active":"2021-03-15T22:36:45.367139+0000","last_peered":"2021-03-15T22:36:45.367139+0000","last_clean":"2021-03-15T22:36:45.367139+0000","last_became_active":"2021-03-15T22:34:10.331424+0000","last_became_peered":"2021-03-15T22:34:10.331424+0000","last_unstale":"2021-03-15T22:36:45.367139+0000","last_undegraded":"2021-03-15T22:36:45.367139+0000","last_fullsized":"2021-03-15T22:36:45.367139+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:39.564464+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:41.431052+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.564464+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.18","version":"781'31","reported_seq":"1147","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367004+0000","last_change":"2021-03-15T22:35:14.434026+0000","last_active":"2021-03-15T22:36:45.367004+0000","last_peered":"2021-03-15T22:36:45.367004+0000","last_clean":"2021-03-15T22:36:45.367004+0000","last_became_active":"2021-03-15T22:33:04.345393+0000","last_became_peered":"2021-03-15T22:33:04.345393+0000","last_unstale":"2021-03-15T22:36:45.367004+0000","last_undegraded":"2021-03-15T22:36:45.367004+0000","last_fullsized":"2021-03-15T22:36:45.367004+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"781'31","last_scrub_stamp":"2021-03-15T22:35:14.433961+0000","last_deep_scrub":"781'31","last_deep_scrub_stamp":"2021-03-15T22:32:12.177019+0000","last_clean_scrub_stamp":"2021-03-15T22:35:14.433961+0000","log_size":31,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1038,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":44,"num_read_kb":26,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2076,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":"356","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.366872+0000","last_change":"2021-03-15T22:36:08.386295+0000","last_active":"2021-03-15T22:36:45.366872+0000","last_peered":"2021-03-15T22:36:45.366872+0000","last_clean":"2021-03-15T22:36:45.366872+0000","last_became_active":"2021-03-15T22:33:04.345518+0000","last_became_peered":"2021-03-15T22:33:04.345518+0000","last_unstale":"2021-03-15T22:36:45.366872+0000","last_undegraded":"2021-03-15T22:36:45.366872+0000","last_fullsized":"2021-03-15T22:36:45.366872+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:08.386208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:08.386208+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":"131","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367259+0000","last_change":"2021-03-15T22:34:43.345093+0000","last_active":"2021-03-15T22:36:45.367259+0000","last_peered":"2021-03-15T22:36:45.367259+0000","last_clean":"2021-03-15T22:36:45.367259+0000","last_became_active":"2021-03-15T22:34:43.344764+0000","last_became_peered":"2021-03-15T22:34:43.344764+0000","last_unstale":"2021-03-15T22:36:45.367259+0000","last_undegraded":"2021-03-15T22:36:45.367259+0000","last_fullsized":"2021-03-15T22:36:45.367259+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:08.652741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:34:08.652741+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"780'44","reported_seq":"197","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367423+0000","last_change":"2021-03-15T22:36:08.831621+0000","last_active":"2021-03-15T22:36:45.367423+0000","last_peered":"2021-03-15T22:36:45.367423+0000","last_clean":"2021-03-15T22:36:45.367423+0000","last_became_active":"2021-03-15T22:34:55.346148+0000","last_became_peered":"2021-03-15T22:34:55.346148+0000","last_unstale":"2021-03-15T22:36:45.367423+0000","last_undegraded":"2021-03-15T22:36:45.367423+0000","last_fullsized":"2021-03-15T22:36:45.367423+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"780'44","last_scrub_stamp":"2021-03-15T22:36:08.831557+0000","last_deep_scrub":"780'44","last_deep_scrub_stamp":"2021-03-15T22:32:10.652898+0000","last_clean_scrub_stamp":"2021-03-15T22:36:08.831557+0000","log_size":44,"ondisk_log_size":44,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":890,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":58,"num_read_kb":39,"num_write":33,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":84,"num_bytes_recovered":5340,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":"899","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103105+0000","last_change":"2021-03-15T22:35:16.848985+0000","last_active":"2021-03-15T22:36:46.103105+0000","last_peered":"2021-03-15T22:36:46.103105+0000","last_clean":"2021-03-15T22:36:46.103105+0000","last_became_active":"2021-03-15T22:34:03.570306+0000","last_became_peered":"2021-03-15T22:34:03.570306+0000","last_unstale":"2021-03-15T22:36:46.103105+0000","last_undegraded":"2021-03-15T22:36:46.103105+0000","last_fullsized":"2021-03-15T22:36:46.103105+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:16.848918+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.848918+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":"518","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368162+0000","last_change":"2021-03-15T22:35:49.146403+0000","last_active":"2021-03-15T22:36:45.368162+0000","last_peered":"2021-03-15T22:36:45.368162+0000","last_clean":"2021-03-15T22:36:45.368162+0000","last_became_active":"2021-03-15T22:28:12.507402+0000","last_became_peered":"2021-03-15T22:28:12.507402+0000","last_unstale":"2021-03-15T22:36:45.368162+0000","last_undegraded":"2021-03-15T22:36:45.368162+0000","last_fullsized":"2021-03-15T22:36:45.368162+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.146337+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:12.608082+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.146337+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1a","version":"555'19","reported_seq":"1203","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367324+0000","last_change":"2021-03-15T22:36:21.176476+0000","last_active":"2021-03-15T22:36:45.367324+0000","last_peered":"2021-03-15T22:36:45.367324+0000","last_clean":"2021-03-15T22:36:45.367324+0000","last_became_active":"2021-03-15T22:29:00.214167+0000","last_became_peered":"2021-03-15T22:29:00.214167+0000","last_unstale":"2021-03-15T22:36:45.367324+0000","last_undegraded":"2021-03-15T22:36:45.367324+0000","last_fullsized":"2021-03-15T22:36:45.367324+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'19","last_scrub_stamp":"2021-03-15T22:36:21.176400+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:21.176400+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":2936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":"334","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831245+0000","last_change":"2021-03-15T22:35:12.404980+0000","last_active":"2021-03-15T22:36:45.831245+0000","last_peered":"2021-03-15T22:36:45.831245+0000","last_clean":"2021-03-15T22:36:45.831245+0000","last_became_active":"2021-03-15T22:31:20.249649+0000","last_became_peered":"2021-03-15T22:31:20.249649+0000","last_unstale":"2021-03-15T22:36:45.831245+0000","last_undegraded":"2021-03-15T22:36:45.831245+0000","last_fullsized":"2021-03-15T22:36:45.831245+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.404897+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.404897+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"0'0","reported_seq":"692","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.102862+0000","last_change":"2021-03-15T22:36:22.138406+0000","last_active":"2021-03-15T22:36:46.102862+0000","last_peered":"2021-03-15T22:36:46.102862+0000","last_clean":"2021-03-15T22:36:46.102862+0000","last_became_active":"2021-03-15T22:34:43.344401+0000","last_became_peered":"2021-03-15T22:34:43.344401+0000","last_unstale":"2021-03-15T22:36:46.102862+0000","last_undegraded":"2021-03-15T22:36:46.102862+0000","last_fullsized":"2021-03-15T22:36:46.102862+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:22.138306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:24:36.674895+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.138306+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"555'11","reported_seq":"1194","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367750+0000","last_change":"2021-03-15T22:35:45.410540+0000","last_active":"2021-03-15T22:36:45.367750+0000","last_peered":"2021-03-15T22:36:45.367750+0000","last_clean":"2021-03-15T22:36:45.367750+0000","last_became_active":"2021-03-15T22:34:55.335550+0000","last_became_peered":"2021-03-15T22:34:55.335550+0000","last_unstale":"2021-03-15T22:36:45.367750+0000","last_undegraded":"2021-03-15T22:36:45.367750+0000","last_fullsized":"2021-03-15T22:36:45.367750+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'11","last_scrub_stamp":"2021-03-15T22:35:45.410466+0000","last_deep_scrub":"121'9","last_deep_scrub_stamp":"2021-03-15T22:20:36.258799+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.410466+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":264,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":9,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":1848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1d","version":"683'12","reported_seq":"1009","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549225+0000","last_change":"2021-03-15T22:35:10.068109+0000","last_active":"2021-03-15T22:36:45.549225+0000","last_peered":"2021-03-15T22:36:45.549225+0000","last_clean":"2021-03-15T22:36:45.549225+0000","last_became_active":"2021-03-15T22:32:23.476572+0000","last_became_peered":"2021-03-15T22:32:23.476572+0000","last_unstale":"2021-03-15T22:36:45.549225+0000","last_undegraded":"2021-03-15T22:36:45.549225+0000","last_fullsized":"2021-03-15T22:36:45.549225+0000","mapping_epoch":792,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":793,"parent":"0.0","parent_split_bits":5,"last_scrub":"683'12","last_scrub_stamp":"2021-03-15T22:35:10.068030+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.068030+0000","log_size":10,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":"1071","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368017+0000","last_change":"2021-03-15T22:34:41.614750+0000","last_active":"2021-03-15T22:36:45.368017+0000","last_peered":"2021-03-15T22:36:45.368017+0000","last_clean":"2021-03-15T22:36:45.368017+0000","last_became_active":"2021-03-15T22:33:04.329078+0000","last_became_peered":"2021-03-15T22:33:04.329078+0000","last_unstale":"2021-03-15T22:36:45.368017+0000","last_undegraded":"2021-03-15T22:36:45.368017+0000","last_fullsized":"2021-03-15T22:36:45.368017+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.614684+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:37.466443+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.614684+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"555'27","reported_seq":"611","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549243+0000","last_change":"2021-03-15T22:36:22.338276+0000","last_active":"2021-03-15T22:36:45.549243+0000","last_peered":"2021-03-15T22:36:45.549243+0000","last_clean":"2021-03-15T22:36:45.549243+0000","last_became_active":"2021-03-15T22:34:55.350325+0000","last_became_peered":"2021-03-15T22:34:55.350325+0000","last_unstale":"2021-03-15T22:36:45.549243+0000","last_undegraded":"2021-03-15T22:36:45.549243+0000","last_fullsized":"2021-03-15T22:36:45.549243+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'27","last_scrub_stamp":"2021-03-15T22:36:22.338202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.338202+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":43,"num_read_kb":34,"num_write":28,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":"1102","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367620+0000","last_change":"2021-03-15T22:35:42.445978+0000","last_active":"2021-03-15T22:36:45.367620+0000","last_peered":"2021-03-15T22:36:45.367620+0000","last_clean":"2021-03-15T22:36:45.367620+0000","last_became_active":"2021-03-15T22:34:55.338001+0000","last_became_peered":"2021-03-15T22:34:55.338001+0000","last_unstale":"2021-03-15T22:36:45.367620+0000","last_undegraded":"2021-03-15T22:36:45.367620+0000","last_fullsized":"2021-03-15T22:36:45.367620+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:42.445911+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.445911+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":"886","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.369605+0000","last_change":"2021-03-15T22:35:20.025959+0000","last_active":"2021-03-15T22:36:45.369605+0000","last_peered":"2021-03-15T22:36:45.369605+0000","last_clean":"2021-03-15T22:36:45.369605+0000","last_became_active":"2021-03-15T22:33:04.336804+0000","last_became_peered":"2021-03-15T22:33:04.336804+0000","last_unstale":"2021-03-15T22:36:45.369605+0000","last_undegraded":"2021-03-15T22:36:45.369605+0000","last_fullsized":"2021-03-15T22:36:45.369605+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.025892+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.025892+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1d","version":"556'22","reported_seq":"497","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368962+0000","last_change":"2021-03-15T22:33:15.617975+0000","last_active":"2021-03-15T22:36:45.368962+0000","last_peered":"2021-03-15T22:36:45.368962+0000","last_clean":"2021-03-15T22:36:45.368962+0000","last_became_active":"2021-03-15T22:28:59.274551+0000","last_became_peered":"2021-03-15T22:28:59.274551+0000","last_unstale":"2021-03-15T22:36:45.368962+0000","last_undegraded":"2021-03-15T22:36:45.368962+0000","last_fullsized":"2021-03-15T22:36:45.368962+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'22","last_scrub_stamp":"2021-03-15T22:33:15.617891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:15.617891+0000","log_size":22,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":844,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":27,"num_write":24,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":3376,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1f","version":"891'22","reported_seq":"632","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:46.561129+0000","last_change":"2021-03-15T22:35:38.710708+0000","last_active":"2021-03-15T22:36:46.561129+0000","last_peered":"2021-03-15T22:36:46.561129+0000","last_clean":"2021-03-15T22:36:46.561129+0000","last_became_active":"2021-03-15T22:34:06.311478+0000","last_became_peered":"2021-03-15T22:34:06.311478+0000","last_unstale":"2021-03-15T22:36:46.561129+0000","last_undegraded":"2021-03-15T22:36:46.561129+0000","last_fullsized":"2021-03-15T22:36:46.561129+0000","mapping_epoch":890,"log_start":"142'3","ondisk_log_start":"142'3","created":728,"last_epoch_clean":891,"parent":"0.0","parent_split_bits":5,"last_scrub":"891'22","last_scrub_stamp":"2021-03-15T22:35:38.710643+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:38.710643+0000","log_size":19,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,7],"acting":[6,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1c","version":"0'0","reported_seq":"1085","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831631+0000","last_change":"2021-03-15T22:36:22.729938+0000","last_active":"2021-03-15T22:36:45.831631+0000","last_peered":"2021-03-15T22:36:45.831631+0000","last_clean":"2021-03-15T22:36:45.831631+0000","last_became_active":"2021-03-15T22:34:55.351597+0000","last_became_peered":"2021-03-15T22:34:55.351597+0000","last_unstale":"2021-03-15T22:36:45.831631+0000","last_undegraded":"2021-03-15T22:36:45.831631+0000","last_fullsized":"2021-03-15T22:36:45.831631+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:22.729845+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.729845+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"556'37","reported_seq":"651","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.370429+0000","last_change":"2021-03-15T22:34:10.385041+0000","last_active":"2021-03-15T22:36:45.370429+0000","last_peered":"2021-03-15T22:36:45.370429+0000","last_clean":"2021-03-15T22:36:45.370429+0000","last_became_active":"2021-03-15T22:34:10.347227+0000","last_became_peered":"2021-03-15T22:34:10.347227+0000","last_unstale":"2021-03-15T22:36:45.370429+0000","last_undegraded":"2021-03-15T22:36:45.370429+0000","last_fullsized":"2021-03-15T22:36:45.370429+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'37","last_scrub_stamp":"2021-03-15T22:34:07.125155+0000","last_deep_scrub":"87'27","last_deep_scrub_stamp":"2021-03-15T22:19:40.666518+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.125155+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":934,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":3736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1e","version":"643'13","reported_seq":"1132","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.370232+0000","last_change":"2021-03-15T22:36:16.071953+0000","last_active":"2021-03-15T22:36:45.370232+0000","last_peered":"2021-03-15T22:36:45.370232+0000","last_clean":"2021-03-15T22:36:45.370232+0000","last_became_active":"2021-03-15T22:34:58.353086+0000","last_became_peered":"2021-03-15T22:34:58.353086+0000","last_unstale":"2021-03-15T22:36:45.370232+0000","last_undegraded":"2021-03-15T22:36:45.370232+0000","last_fullsized":"2021-03-15T22:36:45.370232+0000","mapping_epoch":942,"log_start":"61'2","ondisk_log_start":"61'2","created":728,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":5,"last_scrub":"643'13","last_scrub_stamp":"2021-03-15T22:36:16.071889+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.071889+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":"1058","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549813+0000","last_change":"2021-03-15T22:35:18.111879+0000","last_active":"2021-03-15T22:36:45.549813+0000","last_peered":"2021-03-15T22:36:45.549813+0000","last_clean":"2021-03-15T22:36:45.549813+0000","last_became_active":"2021-03-15T22:29:52.229977+0000","last_became_peered":"2021-03-15T22:29:52.229977+0000","last_unstale":"2021-03-15T22:36:45.549813+0000","last_undegraded":"2021-03-15T22:36:45.549813+0000","last_fullsized":"2021-03-15T22:36:45.549813+0000","mapping_epoch":642,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:18.111815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:18.111815+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"556'25","reported_seq":"262","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533590+0000","last_change":"2021-03-15T22:36:17.753463+0000","last_active":"2021-03-15T22:36:45.533590+0000","last_peered":"2021-03-15T22:36:45.533590+0000","last_clean":"2021-03-15T22:36:45.533590+0000","last_became_active":"2021-03-15T22:33:04.336100+0000","last_became_peered":"2021-03-15T22:33:04.336100+0000","last_unstale":"2021-03-15T22:36:45.533590+0000","last_undegraded":"2021-03-15T22:36:45.533590+0000","last_fullsized":"2021-03-15T22:36:45.533590+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'25","last_scrub_stamp":"2021-03-15T22:36:17.753373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:17.753373+0000","log_size":25,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":50,"num_read_kb":31,"num_write":26,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":2100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":"1067","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831345+0000","last_change":"2021-03-15T22:35:41.589000+0000","last_active":"2021-03-15T22:36:45.831345+0000","last_peered":"2021-03-15T22:36:45.831345+0000","last_clean":"2021-03-15T22:36:45.831345+0000","last_became_active":"2021-03-15T22:34:55.351089+0000","last_became_peered":"2021-03-15T22:34:55.351089+0000","last_unstale":"2021-03-15T22:36:45.831345+0000","last_undegraded":"2021-03-15T22:36:45.831345+0000","last_fullsized":"2021-03-15T22:36:45.831345+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:41.588935+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.759034+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.588935+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":"233","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533006+0000","last_change":"2021-03-15T22:35:22.665325+0000","last_active":"2021-03-15T22:36:45.533006+0000","last_peered":"2021-03-15T22:36:45.533006+0000","last_clean":"2021-03-15T22:36:45.533006+0000","last_became_active":"2021-03-15T22:33:04.335734+0000","last_became_peered":"2021-03-15T22:33:04.335734+0000","last_unstale":"2021-03-15T22:36:45.533006+0000","last_undegraded":"2021-03-15T22:36:45.533006+0000","last_fullsized":"2021-03-15T22:36:45.533006+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.665251+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.665251+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.10","version":"554'16","reported_seq":"561","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.369036+0000","last_change":"2021-03-15T22:33:37.752077+0000","last_active":"2021-03-15T22:36:45.369036+0000","last_peered":"2021-03-15T22:36:45.369036+0000","last_clean":"2021-03-15T22:36:45.369036+0000","last_became_active":"2021-03-15T22:33:04.342749+0000","last_became_peered":"2021-03-15T22:33:04.342749+0000","last_unstale":"2021-03-15T22:36:45.369036+0000","last_undegraded":"2021-03-15T22:36:45.369036+0000","last_fullsized":"2021-03-15T22:36:45.369036+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'16","last_scrub_stamp":"2021-03-15T22:33:37.752011+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:37.752011+0000","log_size":16,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1252,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":34,"num_read_kb":16,"num_write":16,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":70,"num_bytes_recovered":8764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":"674","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368098+0000","last_change":"2021-03-15T22:34:34.783107+0000","last_active":"2021-03-15T22:36:45.368098+0000","last_peered":"2021-03-15T22:36:45.368098+0000","last_clean":"2021-03-15T22:36:45.368098+0000","last_became_active":"2021-03-15T22:33:04.333077+0000","last_became_peered":"2021-03-15T22:33:04.333077+0000","last_unstale":"2021-03-15T22:36:45.368098+0000","last_undegraded":"2021-03-15T22:36:45.368098+0000","last_fullsized":"2021-03-15T22:36:45.368098+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.783047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:08.317817+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.783047+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":"175","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.369109+0000","last_change":"2021-03-15T22:35:48.884382+0000","last_active":"2021-03-15T22:36:45.369109+0000","last_peered":"2021-03-15T22:36:45.369109+0000","last_clean":"2021-03-15T22:36:45.369109+0000","last_became_active":"2021-03-15T22:34:03.571049+0000","last_became_peered":"2021-03-15T22:34:03.571049+0000","last_unstale":"2021-03-15T22:36:45.369109+0000","last_undegraded":"2021-03-15T22:36:45.369109+0000","last_fullsized":"2021-03-15T22:36:45.369109+0000","mapping_epoch":887,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":888,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:48.884317+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:48.884317+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,6],"acting":[3,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.11","version":"556'30","reported_seq":"892","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367551+0000","last_change":"2021-03-15T22:34:37.787302+0000","last_active":"2021-03-15T22:36:45.367551+0000","last_peered":"2021-03-15T22:36:45.367551+0000","last_clean":"2021-03-15T22:36:45.367551+0000","last_became_active":"2021-03-15T22:33:04.331607+0000","last_became_peered":"2021-03-15T22:33:04.331607+0000","last_unstale":"2021-03-15T22:36:45.367551+0000","last_undegraded":"2021-03-15T22:36:45.367551+0000","last_fullsized":"2021-03-15T22:36:45.367551+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'30","last_scrub_stamp":"2021-03-15T22:34:37.787235+0000","last_deep_scrub":"90'22","last_deep_scrub_stamp":"2021-03-15T22:19:58.799544+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.787235+0000","log_size":30,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":580,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":23,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":1740,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":"537","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368003+0000","last_change":"2021-03-15T22:35:17.250496+0000","last_active":"2021-03-15T22:36:45.368003+0000","last_peered":"2021-03-15T22:36:45.368003+0000","last_clean":"2021-03-15T22:36:45.368003+0000","last_became_active":"2021-03-15T22:29:51.212402+0000","last_became_peered":"2021-03-15T22:29:51.212402+0000","last_unstale":"2021-03-15T22:36:45.368003+0000","last_undegraded":"2021-03-15T22:36:45.368003+0000","last_fullsized":"2021-03-15T22:36:45.368003+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:17.250434+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:46.756523+0000","last_clean_scrub_stamp":"2021-03-15T22:35:17.250434+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":"846","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368017+0000","last_change":"2021-03-15T22:36:11.805477+0000","last_active":"2021-03-15T22:36:45.368017+0000","last_peered":"2021-03-15T22:36:45.368017+0000","last_clean":"2021-03-15T22:36:45.368017+0000","last_became_active":"2021-03-15T22:29:00.216021+0000","last_became_peered":"2021-03-15T22:29:00.216021+0000","last_unstale":"2021-03-15T22:36:45.368017+0000","last_undegraded":"2021-03-15T22:36:45.368017+0000","last_fullsized":"2021-03-15T22:36:45.368017+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:11.805413+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:52.558905+0000","last_clean_scrub_stamp":"2021-03-15T22:36:11.805413+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":"1055","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831911+0000","last_change":"2021-03-15T22:35:37.543358+0000","last_active":"2021-03-15T22:36:45.831911+0000","last_peered":"2021-03-15T22:36:45.831911+0000","last_clean":"2021-03-15T22:36:45.831911+0000","last_became_active":"2021-03-15T22:28:59.279829+0000","last_became_peered":"2021-03-15T22:28:59.279829+0000","last_unstale":"2021-03-15T22:36:45.831911+0000","last_undegraded":"2021-03-15T22:36:45.831911+0000","last_fullsized":"2021-03-15T22:36:45.831911+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:37.543289+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:41.449472+0000","last_clean_scrub_stamp":"2021-03-15T22:35:37.543289+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.12","version":"555'21","reported_seq":"769","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.532927+0000","last_change":"2021-03-15T22:36:18.800914+0000","last_active":"2021-03-15T22:36:45.532927+0000","last_peered":"2021-03-15T22:36:45.532927+0000","last_clean":"2021-03-15T22:36:45.532927+0000","last_became_active":"2021-03-15T22:34:55.344555+0000","last_became_peered":"2021-03-15T22:34:55.344555+0000","last_unstale":"2021-03-15T22:36:45.532927+0000","last_undegraded":"2021-03-15T22:36:45.532927+0000","last_fullsized":"2021-03-15T22:36:45.532927+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'21","last_scrub_stamp":"2021-03-15T22:36:18.800824+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:18.800824+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1050,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":43,"num_read_kb":25,"num_write":22,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":3150,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,0],"acting":[6,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":"434","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367296+0000","last_change":"2021-03-15T22:35:23.291568+0000","last_active":"2021-03-15T22:36:45.367296+0000","last_peered":"2021-03-15T22:36:45.367296+0000","last_clean":"2021-03-15T22:36:45.367296+0000","last_became_active":"2021-03-15T22:34:55.355119+0000","last_became_peered":"2021-03-15T22:34:55.355119+0000","last_unstale":"2021-03-15T22:36:45.367296+0000","last_undegraded":"2021-03-15T22:36:45.367296+0000","last_fullsized":"2021-03-15T22:36:45.367296+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.291507+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:26:10.680621+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.291507+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":"1111","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549124+0000","last_change":"2021-03-15T22:35:50.069546+0000","last_active":"2021-03-15T22:36:45.549124+0000","last_peered":"2021-03-15T22:36:45.549124+0000","last_clean":"2021-03-15T22:36:45.549124+0000","last_became_active":"2021-03-15T22:34:10.353314+0000","last_became_peered":"2021-03-15T22:34:10.353314+0000","last_unstale":"2021-03-15T22:36:45.549124+0000","last_undegraded":"2021-03-15T22:36:45.549124+0000","last_fullsized":"2021-03-15T22:36:45.549124+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:50.069463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:44.116934+0000","last_clean_scrub_stamp":"2021-03-15T22:35:50.069463+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":"189","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367439+0000","last_change":"2021-03-15T22:36:20.132263+0000","last_active":"2021-03-15T22:36:45.367439+0000","last_peered":"2021-03-15T22:36:45.367439+0000","last_clean":"2021-03-15T22:36:45.367439+0000","last_became_active":"2021-03-15T22:33:45.314042+0000","last_became_peered":"2021-03-15T22:33:45.314042+0000","last_unstale":"2021-03-15T22:36:45.367439+0000","last_undegraded":"2021-03-15T22:36:45.367439+0000","last_fullsized":"2021-03-15T22:36:45.367439+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:20.132182+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.132182+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.13","version":"555'23","reported_seq":"1135","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367689+0000","last_change":"2021-03-15T22:34:42.632034+0000","last_active":"2021-03-15T22:36:45.367689+0000","last_peered":"2021-03-15T22:36:45.367689+0000","last_clean":"2021-03-15T22:36:45.367689+0000","last_became_active":"2021-03-15T22:33:04.340460+0000","last_became_peered":"2021-03-15T22:33:04.340460+0000","last_unstale":"2021-03-15T22:36:45.367689+0000","last_undegraded":"2021-03-15T22:36:45.367689+0000","last_fullsized":"2021-03-15T22:36:45.367689+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:34:42.631962+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:42.631962+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":736,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":"545","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.370569+0000","last_change":"2021-03-15T22:36:22.004600+0000","last_active":"2021-03-15T22:36:45.370569+0000","last_peered":"2021-03-15T22:36:45.370569+0000","last_clean":"2021-03-15T22:36:45.370569+0000","last_became_active":"2021-03-15T22:34:10.326881+0000","last_became_peered":"2021-03-15T22:34:10.326881+0000","last_unstale":"2021-03-15T22:36:45.370569+0000","last_undegraded":"2021-03-15T22:36:45.370569+0000","last_fullsized":"2021-03-15T22:36:45.370569+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:22.004539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:36:22.004539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,4],"acting":[0,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.15","version":"808'9","reported_seq":"518","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:46.545675+0000","last_change":"2021-03-15T22:35:41.986386+0000","last_active":"2021-03-15T22:36:46.545675+0000","last_peered":"2021-03-15T22:36:46.545675+0000","last_clean":"2021-03-15T22:36:46.545675+0000","last_became_active":"2021-03-15T22:29:51.234854+0000","last_became_peered":"2021-03-15T22:29:51.234854+0000","last_unstale":"2021-03-15T22:36:46.545675+0000","last_undegraded":"2021-03-15T22:36:46.545675+0000","last_fullsized":"2021-03-15T22:36:46.545675+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":642,"parent":"0.0","parent_split_bits":0,"last_scrub":"808'9","last_scrub_stamp":"2021-03-15T22:35:41.986318+0000","last_deep_scrub":"808'9","last_deep_scrub_stamp":"2021-03-15T22:32:38.880993+0000","last_clean_scrub_stamp":"2021-03-15T22:35:41.986318+0000","log_size":9,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":"1052","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831380+0000","last_change":"2021-03-15T22:36:09.833634+0000","last_active":"2021-03-15T22:36:45.831380+0000","last_peered":"2021-03-15T22:36:45.831380+0000","last_clean":"2021-03-15T22:36:45.831380+0000","last_became_active":"2021-03-15T22:34:55.351042+0000","last_became_peered":"2021-03-15T22:34:55.351042+0000","last_unstale":"2021-03-15T22:36:45.831380+0000","last_undegraded":"2021-03-15T22:36:45.831380+0000","last_fullsized":"2021-03-15T22:36:45.831380+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:09.833552+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:36:09.833552+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.14","version":"844'33","reported_seq":"907","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368755+0000","last_change":"2021-03-15T22:34:55.387712+0000","last_active":"2021-03-15T22:36:45.368755+0000","last_peered":"2021-03-15T22:36:45.368755+0000","last_clean":"2021-03-15T22:36:45.368755+0000","last_became_active":"2021-03-15T22:34:55.351721+0000","last_became_peered":"2021-03-15T22:34:55.351721+0000","last_unstale":"2021-03-15T22:36:45.368755+0000","last_undegraded":"2021-03-15T22:36:45.368755+0000","last_fullsized":"2021-03-15T22:36:45.368755+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"844'33","last_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_deep_scrub":"844'33","last_deep_scrub_stamp":"2021-03-15T22:33:14.622437+0000","last_clean_scrub_stamp":"2021-03-15T22:33:14.622437+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":624,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":28,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":2496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":"1088","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368258+0000","last_change":"2021-03-15T22:35:40.412203+0000","last_active":"2021-03-15T22:36:45.368258+0000","last_peered":"2021-03-15T22:36:45.368258+0000","last_clean":"2021-03-15T22:36:45.368258+0000","last_became_active":"2021-03-15T22:34:55.340440+0000","last_became_peered":"2021-03-15T22:34:55.340440+0000","last_unstale":"2021-03-15T22:36:45.368258+0000","last_undegraded":"2021-03-15T22:36:45.368258+0000","last_fullsized":"2021-03-15T22:36:45.368258+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:40.412142+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:35:40.412142+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.14","version":"941'17","reported_seq":"593","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:46.624055+0000","last_change":"2021-03-15T22:36:17.843284+0000","last_active":"2021-03-15T22:36:46.624055+0000","last_peered":"2021-03-15T22:36:46.624055+0000","last_clean":"2021-03-15T22:36:46.624055+0000","last_became_active":"2021-03-15T22:34:55.351682+0000","last_became_peered":"2021-03-15T22:34:55.351682+0000","last_unstale":"2021-03-15T22:36:46.624055+0000","last_undegraded":"2021-03-15T22:36:46.624055+0000","last_fullsized":"2021-03-15T22:36:46.624055+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"941'17","last_scrub_stamp":"2021-03-15T22:36:17.843205+0000","last_deep_scrub":"896'16","last_deep_scrub_stamp":"2021-03-15T22:34:19.400191+0000","last_clean_scrub_stamp":"2021-03-15T22:36:17.843205+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":"545","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.369926+0000","last_change":"2021-03-15T22:36:10.121601+0000","last_active":"2021-03-15T22:36:45.369926+0000","last_peered":"2021-03-15T22:36:45.369926+0000","last_clean":"2021-03-15T22:36:45.369926+0000","last_became_active":"2021-03-15T22:34:10.347512+0000","last_became_peered":"2021-03-15T22:34:10.347512+0000","last_unstale":"2021-03-15T22:36:45.369926+0000","last_undegraded":"2021-03-15T22:36:45.369926+0000","last_fullsized":"2021-03-15T22:36:45.369926+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:10.121536+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:39.695671+0000","last_clean_scrub_stamp":"2021-03-15T22:36:10.121536+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.15","version":"556'27","reported_seq":"447","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103434+0000","last_change":"2021-03-15T22:36:11.021642+0000","last_active":"2021-03-15T22:36:46.103434+0000","last_peered":"2021-03-15T22:36:46.103434+0000","last_clean":"2021-03-15T22:36:46.103434+0000","last_became_active":"2021-03-15T22:29:51.236724+0000","last_became_peered":"2021-03-15T22:29:51.236724+0000","last_unstale":"2021-03-15T22:36:46.103434+0000","last_undegraded":"2021-03-15T22:36:46.103434+0000","last_fullsized":"2021-03-15T22:36:46.103434+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:36:11.021564+0000","last_deep_scrub":"497'21","last_deep_scrub_stamp":"2021-03-15T22:26:52.609858+0000","last_clean_scrub_stamp":"2021-03-15T22:36:11.021564+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":37,"num_read_kb":23,"num_write":20,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":2202,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":"131","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368564+0000","last_change":"2021-03-15T22:34:43.345092+0000","last_active":"2021-03-15T22:36:45.368564+0000","last_peered":"2021-03-15T22:36:45.368564+0000","last_clean":"2021-03-15T22:36:45.368564+0000","last_became_active":"2021-03-15T22:34:43.344790+0000","last_became_peered":"2021-03-15T22:34:43.344790+0000","last_unstale":"2021-03-15T22:36:45.368564+0000","last_undegraded":"2021-03-15T22:36:45.368564+0000","last_fullsized":"2021-03-15T22:36:45.368564+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.621539+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.621539+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,7],"acting":[3,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":"129","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:47.636303+0000","last_change":"2021-03-15T22:36:47.636303+0000","last_active":"2021-03-15T22:36:47.636303+0000","last_peered":"2021-03-15T22:36:47.636303+0000","last_clean":"2021-03-15T22:36:47.636303+0000","last_became_active":"2021-03-15T22:34:55.348663+0000","last_became_peered":"2021-03-15T22:34:55.348663+0000","last_unstale":"2021-03-15T22:36:47.636303+0000","last_undegraded":"2021-03-15T22:36:47.636303+0000","last_fullsized":"2021-03-15T22:36:47.636303+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:47.636243+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.381352+0000","last_clean_scrub_stamp":"2021-03-15T22:36:47.636243+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":"1045","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831108+0000","last_change":"2021-03-15T22:35:48.581266+0000","last_active":"2021-03-15T22:36:45.831108+0000","last_peered":"2021-03-15T22:36:45.831108+0000","last_clean":"2021-03-15T22:36:45.831108+0000","last_became_active":"2021-03-15T22:33:04.330195+0000","last_became_peered":"2021-03-15T22:33:04.330195+0000","last_unstale":"2021-03-15T22:36:45.831108+0000","last_undegraded":"2021-03-15T22:36:45.831108+0000","last_fullsized":"2021-03-15T22:36:45.831108+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:48.581162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:48.581162+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"575'33","reported_seq":"576","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.370263+0000","last_change":"2021-03-15T22:34:37.435070+0000","last_active":"2021-03-15T22:36:45.370263+0000","last_peered":"2021-03-15T22:36:45.370263+0000","last_clean":"2021-03-15T22:36:45.370263+0000","last_became_active":"2021-03-15T22:29:00.213158+0000","last_became_peered":"2021-03-15T22:29:00.213158+0000","last_unstale":"2021-03-15T22:36:45.370263+0000","last_undegraded":"2021-03-15T22:36:45.370263+0000","last_fullsized":"2021-03-15T22:36:45.370263+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":591,"parent":"0.0","parent_split_bits":0,"last_scrub":"575'33","last_scrub_stamp":"2021-03-15T22:34:37.435006+0000","last_deep_scrub":"575'33","last_deep_scrub_stamp":"2021-03-15T22:28:44.765846+0000","last_clean_scrub_stamp":"2021-03-15T22:34:37.435006+0000","log_size":33,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1206,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":47,"num_read_kb":27,"num_write":23,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":4824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":"851","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103692+0000","last_change":"2021-03-15T22:35:20.884067+0000","last_active":"2021-03-15T22:36:46.103692+0000","last_peered":"2021-03-15T22:36:46.103692+0000","last_clean":"2021-03-15T22:36:46.103692+0000","last_became_active":"2021-03-15T22:31:20.260875+0000","last_became_peered":"2021-03-15T22:31:20.260875+0000","last_unstale":"2021-03-15T22:36:46.103692+0000","last_undegraded":"2021-03-15T22:36:46.103692+0000","last_fullsized":"2021-03-15T22:36:46.103692+0000","mapping_epoch":730,"log_start":"0'0","ondisk_log_start":"0'0","created":728,"last_epoch_clean":731,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.883990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.883990+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":"1039","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831813+0000","last_change":"2021-03-15T22:35:45.547459+0000","last_active":"2021-03-15T22:36:45.831813+0000","last_peered":"2021-03-15T22:36:45.831813+0000","last_clean":"2021-03-15T22:36:45.831813+0000","last_became_active":"2021-03-15T22:33:04.331013+0000","last_became_peered":"2021-03-15T22:33:04.331013+0000","last_unstale":"2021-03-15T22:36:45.831813+0000","last_undegraded":"2021-03-15T22:36:45.831813+0000","last_fullsized":"2021-03-15T22:36:45.831813+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.547395+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:45.547395+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.547395+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":"233","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.532756+0000","last_change":"2021-03-15T22:35:45.757312+0000","last_active":"2021-03-15T22:36:45.532756+0000","last_peered":"2021-03-15T22:36:45.532756+0000","last_clean":"2021-03-15T22:36:45.532756+0000","last_became_active":"2021-03-15T22:33:04.336886+0000","last_became_peered":"2021-03-15T22:33:04.336886+0000","last_unstale":"2021-03-15T22:36:45.532756+0000","last_undegraded":"2021-03-15T22:36:45.532756+0000","last_fullsized":"2021-03-15T22:36:45.532756+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.757254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:45.757254+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.757254+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"555'17","reported_seq":"403","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.832332+0000","last_change":"2021-03-15T22:36:45.831996+0000","last_active":"2021-03-15T22:36:45.832332+0000","last_peered":"2021-03-15T22:36:45.832332+0000","last_clean":"2021-03-15T22:36:45.832332+0000","last_became_active":"2021-03-15T22:34:55.351977+0000","last_became_peered":"2021-03-15T22:34:55.351977+0000","last_unstale":"2021-03-15T22:36:45.832332+0000","last_undegraded":"2021-03-15T22:36:45.832332+0000","last_fullsized":"2021-03-15T22:36:45.832332+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'17","last_scrub_stamp":"2021-03-15T22:36:45.831919+0000","last_deep_scrub":"398'11","last_deep_scrub_stamp":"2021-03-15T22:25:13.181225+0000","last_clean_scrub_stamp":"2021-03-15T22:36:45.831919+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":158,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":19,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":790,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"1040'1547","reported_seq":"4802","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:46.565341+0000","last_change":"2021-03-15T22:36:46.565341+0000","last_active":"2021-03-15T22:36:46.565341+0000","last_peered":"2021-03-15T22:36:46.565341+0000","last_clean":"2021-03-15T22:36:46.565341+0000","last_became_active":"2021-03-15T22:33:08.355490+0000","last_became_peered":"2021-03-15T22:33:08.355490+0000","last_unstale":"2021-03-15T22:36:46.565341+0000","last_undegraded":"2021-03-15T22:36:46.565341+0000","last_fullsized":"2021-03-15T22:36:46.565341+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"1040'1547","last_scrub_stamp":"2021-03-15T22:36:46.565275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:46.565275+0000","log_size":1547,"ondisk_log_size":1547,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":53562661,"num_objects":22,"num_object_clones":19,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":509,"num_read_kb":562639,"num_write":351,"num_write_kb":126327,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":170,"num_bytes_recovered":84869361,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6],"acting":[1,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":"996","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:48.228840+0000","last_change":"2021-03-15T22:36:48.228840+0000","last_active":"2021-03-15T22:36:48.228840+0000","last_peered":"2021-03-15T22:36:48.228840+0000","last_clean":"2021-03-15T22:36:48.228840+0000","last_became_active":"2021-03-15T22:33:04.340352+0000","last_became_peered":"2021-03-15T22:33:04.340352+0000","last_unstale":"2021-03-15T22:36:48.228840+0000","last_undegraded":"2021-03-15T22:36:48.228840+0000","last_fullsized":"2021-03-15T22:36:48.228840+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:48.228763+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:36:48.228763+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"874'4","reported_seq":"1080","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.832247+0000","last_change":"2021-03-15T22:36:14.798866+0000","last_active":"2021-03-15T22:36:45.832247+0000","last_peered":"2021-03-15T22:36:45.832247+0000","last_clean":"2021-03-15T22:36:45.832247+0000","last_became_active":"2021-03-15T22:35:58.360849+0000","last_became_peered":"2021-03-15T22:35:58.360849+0000","last_unstale":"2021-03-15T22:36:45.832247+0000","last_undegraded":"2021-03-15T22:36:45.832247+0000","last_fullsized":"2021-03-15T22:36:45.832247+0000","mapping_epoch":552,"log_start":"874'4","ondisk_log_start":"874'4","created":56,"last_epoch_clean":1003,"parent":"0.0","parent_split_bits":0,"last_scrub":"874'4","last_scrub_stamp":"2021-03-15T22:36:14.798788+0000","last_deep_scrub":"874'4","last_deep_scrub_stamp":"2021-03-15T22:33:44.498263+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.798788+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":965,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":3676,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"691'40","reported_seq":"518","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.102995+0000","last_change":"2021-03-15T22:33:38.759477+0000","last_active":"2021-03-15T22:36:46.102995+0000","last_peered":"2021-03-15T22:36:46.102995+0000","last_clean":"2021-03-15T22:36:46.102995+0000","last_became_active":"2021-03-15T22:33:04.326070+0000","last_became_peered":"2021-03-15T22:33:04.326070+0000","last_unstale":"2021-03-15T22:36:46.102995+0000","last_undegraded":"2021-03-15T22:36:46.102995+0000","last_fullsized":"2021-03-15T22:36:46.102995+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"691'40","last_scrub_stamp":"2021-03-15T22:33:38.759395+0000","last_deep_scrub":"691'40","last_deep_scrub_stamp":"2021-03-15T22:30:41.227473+0000","last_clean_scrub_stamp":"2021-03-15T22:33:38.759395+0000","log_size":40,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":578,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":52,"num_read_kb":37,"num_write":31,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":2312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.9","version":"841'14","reported_seq":"1334","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:46.622928+0000","last_change":"2021-03-15T22:36:13.797925+0000","last_active":"2021-03-15T22:36:46.622928+0000","last_peered":"2021-03-15T22:36:46.622928+0000","last_clean":"2021-03-15T22:36:46.622928+0000","last_became_active":"2021-03-15T22:33:08.354031+0000","last_became_peered":"2021-03-15T22:33:08.354031+0000","last_unstale":"2021-03-15T22:36:46.622928+0000","last_undegraded":"2021-03-15T22:36:46.622928+0000","last_fullsized":"2021-03-15T22:36:46.622928+0000","mapping_epoch":837,"log_start":"89'3","ondisk_log_start":"89'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"841'14","last_scrub_stamp":"2021-03-15T22:36:13.797841+0000","last_deep_scrub":"89'3","last_deep_scrub_stamp":"2021-03-15T22:19:51.938126+0000","last_clean_scrub_stamp":"2021-03-15T22:36:13.797841+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"1043'2255","reported_seq":"5755","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367671+0000","last_change":"2021-03-15T22:36:35.395939+0000","last_active":"2021-03-15T22:36:45.367671+0000","last_peered":"2021-03-15T22:36:45.367671+0000","last_clean":"2021-03-15T22:36:45.367671+0000","last_became_active":"2021-03-15T22:34:58.336162+0000","last_became_peered":"2021-03-15T22:34:58.336162+0000","last_unstale":"2021-03-15T22:36:45.367671+0000","last_undegraded":"2021-03-15T22:36:45.367671+0000","last_fullsized":"2021-03-15T22:36:45.367671+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'2031","last_scrub_stamp":"2021-03-15T22:35:42.023328+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.023328+0000","log_size":2255,"ondisk_log_size":2255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":90051309,"num_objects":41,"num_object_clones":36,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":1,"num_read":710,"num_read_kb":662809,"num_write":557,"num_write_kb":215582,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":374,"num_bytes_recovered":198498497,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":"477","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367855+0000","last_change":"2021-03-15T22:32:42.621654+0000","last_active":"2021-03-15T22:36:45.367855+0000","last_peered":"2021-03-15T22:36:45.367855+0000","last_clean":"2021-03-15T22:36:45.367855+0000","last_became_active":"2021-03-15T22:28:59.274629+0000","last_became_peered":"2021-03-15T22:28:59.274629+0000","last_unstale":"2021-03-15T22:36:45.367855+0000","last_undegraded":"2021-03-15T22:36:45.367855+0000","last_fullsized":"2021-03-15T22:36:45.367855+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:42.621594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:42.621594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":"1047","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831920+0000","last_change":"2021-03-15T22:36:02.374943+0000","last_active":"2021-03-15T22:36:45.831920+0000","last_peered":"2021-03-15T22:36:45.831920+0000","last_clean":"2021-03-15T22:36:45.831920+0000","last_became_active":"2021-03-15T22:36:02.374583+0000","last_became_peered":"2021-03-15T22:36:02.374583+0000","last_unstale":"2021-03-15T22:36:45.831920+0000","last_undegraded":"2021-03-15T22:36:45.831920+0000","last_fullsized":"2021-03-15T22:36:45.831920+0000","mapping_epoch":316,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1007,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:47.601457+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.601457+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"554'21","reported_seq":"1258","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367363+0000","last_change":"2021-03-15T22:34:11.031994+0000","last_active":"2021-03-15T22:36:45.367363+0000","last_peered":"2021-03-15T22:36:45.367363+0000","last_clean":"2021-03-15T22:36:45.367363+0000","last_became_active":"2021-03-15T22:34:10.342120+0000","last_became_peered":"2021-03-15T22:34:10.342120+0000","last_unstale":"2021-03-15T22:36:45.367363+0000","last_undegraded":"2021-03-15T22:36:45.367363+0000","last_fullsized":"2021-03-15T22:36:45.367363+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"554'21","last_scrub_stamp":"2021-03-15T22:34:07.452572+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.452572+0000","log_size":21,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":692,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":43,"num_read_kb":28,"num_write":23,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":56,"num_bytes_recovered":4844,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,6],"acting":[4,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":"761","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368076+0000","last_change":"2021-03-15T22:34:55.356651+0000","last_active":"2021-03-15T22:36:45.368076+0000","last_peered":"2021-03-15T22:36:45.368076+0000","last_clean":"2021-03-15T22:36:45.368076+0000","last_became_active":"2021-03-15T22:34:55.356328+0000","last_became_peered":"2021-03-15T22:34:55.356328+0000","last_unstale":"2021-03-15T22:36:45.368076+0000","last_undegraded":"2021-03-15T22:36:45.368076+0000","last_fullsized":"2021-03-15T22:36:45.368076+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:42.055640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:31:42.055640+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"1040'607","reported_seq":"3585","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367564+0000","last_change":"2021-03-15T22:36:35.381989+0000","last_active":"2021-03-15T22:36:45.367564+0000","last_peered":"2021-03-15T22:36:45.367564+0000","last_clean":"2021-03-15T22:36:45.367564+0000","last_became_active":"2021-03-15T22:34:13.324667+0000","last_became_peered":"2021-03-15T22:34:13.324667+0000","last_unstale":"2021-03-15T22:36:45.367564+0000","last_undegraded":"2021-03-15T22:36:45.367564+0000","last_fullsized":"2021-03-15T22:36:45.367564+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"867'446","last_scrub_stamp":"2021-03-15T22:33:39.355306+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:39.355306+0000","log_size":607,"ondisk_log_size":607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20558245,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":149,"num_read_kb":124961,"num_write":134,"num_write_kb":44296,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":23698344,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":"1115","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.830914+0000","last_change":"2021-03-15T22:36:18.835190+0000","last_active":"2021-03-15T22:36:45.830914+0000","last_peered":"2021-03-15T22:36:45.830914+0000","last_clean":"2021-03-15T22:36:45.830914+0000","last_became_active":"2021-03-15T22:35:48.365533+0000","last_became_peered":"2021-03-15T22:35:48.365533+0000","last_unstale":"2021-03-15T22:36:45.830914+0000","last_undegraded":"2021-03-15T22:36:45.830914+0000","last_fullsized":"2021-03-15T22:36:45.830914+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":993,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:18.835125+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:36:18.835125+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":"124","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.551684+0000","last_change":"2021-03-15T22:35:49.448634+0000","last_active":"2021-03-15T22:36:45.551684+0000","last_peered":"2021-03-15T22:36:45.551684+0000","last_clean":"2021-03-15T22:36:45.551684+0000","last_became_active":"2021-03-15T22:34:55.353966+0000","last_became_peered":"2021-03-15T22:34:55.353966+0000","last_unstale":"2021-03-15T22:36:45.551684+0000","last_undegraded":"2021-03-15T22:36:45.551684+0000","last_fullsized":"2021-03-15T22:36:45.551684+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:49.448581+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:49.448581+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":"1102","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.832108+0000","last_change":"2021-03-15T22:36:06.382789+0000","last_active":"2021-03-15T22:36:45.832108+0000","last_peered":"2021-03-15T22:36:45.832108+0000","last_clean":"2021-03-15T22:36:45.832108+0000","last_became_active":"2021-03-15T22:36:06.382477+0000","last_became_peered":"2021-03-15T22:36:06.382477+0000","last_unstale":"2021-03-15T22:36:45.832108+0000","last_undegraded":"2021-03-15T22:36:45.832108+0000","last_fullsized":"2021-03-15T22:36:45.832108+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1011,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:23.428842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:06.006086+0000","last_clean_scrub_stamp":"2021-03-15T22:35:23.428842+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"556'39","reported_seq":"298","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533059+0000","last_change":"2021-03-15T22:35:47.782159+0000","last_active":"2021-03-15T22:36:45.533059+0000","last_peered":"2021-03-15T22:36:45.533059+0000","last_clean":"2021-03-15T22:36:45.533059+0000","last_became_active":"2021-03-15T22:33:45.318867+0000","last_became_peered":"2021-03-15T22:33:45.318867+0000","last_unstale":"2021-03-15T22:36:45.533059+0000","last_undegraded":"2021-03-15T22:36:45.533059+0000","last_fullsized":"2021-03-15T22:36:45.533059+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":876,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'39","last_scrub_stamp":"2021-03-15T22:35:47.782092+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.782092+0000","log_size":39,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":732,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":49,"num_write":40,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":2928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.8","version":"1043'1384","reported_seq":"4429","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.551889+0000","last_change":"2021-03-15T22:36:35.375257+0000","last_active":"2021-03-15T22:36:45.551889+0000","last_peered":"2021-03-15T22:36:45.551889+0000","last_clean":"2021-03-15T22:36:45.551889+0000","last_became_active":"2021-03-15T22:34:59.339587+0000","last_became_peered":"2021-03-15T22:34:59.339587+0000","last_unstale":"2021-03-15T22:36:45.551889+0000","last_undegraded":"2021-03-15T22:36:45.551889+0000","last_fullsized":"2021-03-15T22:36:45.551889+0000","mapping_epoch":943,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":944,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'1309","last_scrub_stamp":"2021-03-15T22:36:14.641356+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.641356+0000","log_size":1384,"ondisk_log_size":1384,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":52382815,"num_objects":23,"num_object_clones":20,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":450,"num_read_kb":538913,"num_write":306,"num_write_kb":109155,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":305,"num_bytes_recovered":205148835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,4],"acting":[2,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":"569","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367364+0000","last_change":"2021-03-15T22:35:44.582634+0000","last_active":"2021-03-15T22:36:45.367364+0000","last_peered":"2021-03-15T22:36:45.367364+0000","last_clean":"2021-03-15T22:36:45.367364+0000","last_became_active":"2021-03-15T22:35:44.582322+0000","last_became_peered":"2021-03-15T22:35:44.582322+0000","last_unstale":"2021-03-15T22:36:45.367364+0000","last_undegraded":"2021-03-15T22:36:45.367364+0000","last_fullsized":"2021-03-15T22:36:45.367364+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":989,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:31:08.968594+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:31:08.968594+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":"887","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367427+0000","last_change":"2021-03-15T22:36:45.344330+0000","last_active":"2021-03-15T22:36:45.367427+0000","last_peered":"2021-03-15T22:36:45.367427+0000","last_clean":"2021-03-15T22:36:45.367427+0000","last_became_active":"2021-03-15T22:34:55.342808+0000","last_became_peered":"2021-03-15T22:34:55.342808+0000","last_unstale":"2021-03-15T22:36:45.367427+0000","last_undegraded":"2021-03-15T22:36:45.367427+0000","last_fullsized":"2021-03-15T22:36:45.367427+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:45.344264+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:45.344264+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,3],"acting":[4,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":"467","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103399+0000","last_change":"2021-03-15T22:36:10.362976+0000","last_active":"2021-03-15T22:36:46.103399+0000","last_peered":"2021-03-15T22:36:46.103399+0000","last_clean":"2021-03-15T22:36:46.103399+0000","last_became_active":"2021-03-15T22:36:10.362612+0000","last_became_peered":"2021-03-15T22:36:10.362612+0000","last_unstale":"2021-03-15T22:36:46.103399+0000","last_undegraded":"2021-03-15T22:36:46.103399+0000","last_fullsized":"2021-03-15T22:36:46.103399+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1015,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:07.807411+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:07.807411+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.b","version":"555'20","reported_seq":"596","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368834+0000","last_change":"2021-03-15T22:36:14.097504+0000","last_active":"2021-03-15T22:36:45.368834+0000","last_peered":"2021-03-15T22:36:45.368834+0000","last_clean":"2021-03-15T22:36:45.368834+0000","last_became_active":"2021-03-15T22:33:45.326747+0000","last_became_peered":"2021-03-15T22:33:45.326747+0000","last_unstale":"2021-03-15T22:36:45.368834+0000","last_undegraded":"2021-03-15T22:36:45.368834+0000","last_fullsized":"2021-03-15T22:36:45.368834+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'20","last_scrub_stamp":"2021-03-15T22:36:14.097432+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.097432+0000","log_size":20,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":894,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":41,"num_read_kb":25,"num_write":21,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":2682,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"30","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103112+0000","last_change":"2021-03-15T22:36:24.384474+0000","last_active":"2021-03-15T22:36:46.103112+0000","last_peered":"2021-03-15T22:36:46.103112+0000","last_clean":"2021-03-15T22:36:46.103112+0000","last_became_active":"2021-03-15T22:36:24.384123+0000","last_became_peered":"2021-03-15T22:36:24.384123+0000","last_unstale":"2021-03-15T22:36:46.103112+0000","last_undegraded":"2021-03-15T22:36:46.103112+0000","last_fullsized":"2021-03-15T22:36:46.103112+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"1043'1585","reported_seq":"5303","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533504+0000","last_change":"2021-03-15T22:36:35.395167+0000","last_active":"2021-03-15T22:36:45.533504+0000","last_peered":"2021-03-15T22:36:45.533504+0000","last_clean":"2021-03-15T22:36:45.533504+0000","last_became_active":"2021-03-15T22:34:46.338888+0000","last_became_peered":"2021-03-15T22:34:46.338888+0000","last_unstale":"2021-03-15T22:36:45.533504+0000","last_undegraded":"2021-03-15T22:36:45.533504+0000","last_fullsized":"2021-03-15T22:36:45.533504+0000","mapping_epoch":930,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":931,"parent":"0.0","parent_split_bits":0,"last_scrub":"1025'1523","last_scrub_stamp":"2021-03-15T22:36:20.962682+0000","last_deep_scrub":"1025'1523","last_deep_scrub_stamp":"2021-03-15T22:36:20.962682+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.962682+0000","log_size":1585,"ondisk_log_size":1585,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":60313905,"num_objects":26,"num_object_clones":23,"num_object_copies":78,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":350,"num_read_kb":290243,"num_write":369,"num_write_kb":136361,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":229919087,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":"149","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.369140+0000","last_change":"2021-03-15T22:35:56.358841+0000","last_active":"2021-03-15T22:36:45.369140+0000","last_peered":"2021-03-15T22:36:45.369140+0000","last_clean":"2021-03-15T22:36:45.369140+0000","last_became_active":"2021-03-15T22:35:56.358504+0000","last_became_peered":"2021-03-15T22:35:56.358504+0000","last_unstale":"2021-03-15T22:36:45.369140+0000","last_undegraded":"2021-03-15T22:36:45.369140+0000","last_fullsized":"2021-03-15T22:36:45.369140+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1001,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:44.778568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:44.778568+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"95","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831165+0000","last_change":"2021-03-15T22:35:19.502186+0000","last_active":"2021-03-15T22:36:45.831165+0000","last_peered":"2021-03-15T22:36:45.831165+0000","last_clean":"2021-03-15T22:36:45.831165+0000","last_became_active":"2021-03-15T22:35:19.501783+0000","last_became_peered":"2021-03-15T22:35:19.501783+0000","last_unstale":"2021-03-15T22:36:45.831165+0000","last_undegraded":"2021-03-15T22:36:45.831165+0000","last_fullsized":"2021-03-15T22:36:45.831165+0000","mapping_epoch":963,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":964,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.831369+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:08.268707+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.831369+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":"251","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367497+0000","last_change":"2021-03-15T22:36:16.252660+0000","last_active":"2021-03-15T22:36:45.367497+0000","last_peered":"2021-03-15T22:36:45.367497+0000","last_clean":"2021-03-15T22:36:45.367497+0000","last_became_active":"2021-03-15T22:33:04.335914+0000","last_became_peered":"2021-03-15T22:33:04.335914+0000","last_unstale":"2021-03-15T22:36:45.367497+0000","last_undegraded":"2021-03-15T22:36:45.367497+0000","last_fullsized":"2021-03-15T22:36:45.367497+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:16.252579+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:39.012676+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.252579+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.4","version":"556'50","reported_seq":"1421","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549483+0000","last_change":"2021-03-15T22:35:47.126971+0000","last_active":"2021-03-15T22:36:45.549483+0000","last_peered":"2021-03-15T22:36:45.549483+0000","last_clean":"2021-03-15T22:36:45.549483+0000","last_became_active":"2021-03-15T22:34:55.350419+0000","last_became_peered":"2021-03-15T22:34:55.350419+0000","last_unstale":"2021-03-15T22:36:45.549483+0000","last_undegraded":"2021-03-15T22:36:45.549483+0000","last_fullsized":"2021-03-15T22:36:45.549483+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'50","last_scrub_stamp":"2021-03-15T22:35:47.126904+0000","last_deep_scrub":"84'38","last_deep_scrub_stamp":"2021-03-15T22:19:17.649963+0000","last_clean_scrub_stamp":"2021-03-15T22:35:47.126904+0000","log_size":50,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":954,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":72,"num_read_kb":49,"num_write":40,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":97,"num_bytes_recovered":6678,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":"518","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:48.012498+0000","last_change":"2021-03-15T22:36:48.012498+0000","last_active":"2021-03-15T22:36:48.012498+0000","last_peered":"2021-03-15T22:36:48.012498+0000","last_clean":"2021-03-15T22:36:48.012498+0000","last_became_active":"2021-03-15T22:28:12.519433+0000","last_became_peered":"2021-03-15T22:28:12.519433+0000","last_unstale":"2021-03-15T22:36:48.012498+0000","last_undegraded":"2021-03-15T22:36:48.012498+0000","last_fullsized":"2021-03-15T22:36:48.012498+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:48.012438+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:48.012438+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"1080","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831328+0000","last_change":"2021-03-15T22:36:24.388928+0000","last_active":"2021-03-15T22:36:45.831328+0000","last_peered":"2021-03-15T22:36:45.831328+0000","last_clean":"2021-03-15T22:36:45.831328+0000","last_became_active":"2021-03-15T22:36:24.388564+0000","last_became_peered":"2021-03-15T22:36:24.388564+0000","last_unstale":"2021-03-15T22:36:45.831328+0000","last_undegraded":"2021-03-15T22:36:45.831328+0000","last_fullsized":"2021-03-15T22:36:45.831328+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,4],"acting":[7,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"1040'1449","reported_seq":"4288","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368444+0000","last_change":"2021-03-15T22:36:35.394677+0000","last_active":"2021-03-15T22:36:45.368444+0000","last_peered":"2021-03-15T22:36:45.368444+0000","last_clean":"2021-03-15T22:36:45.368444+0000","last_became_active":"2021-03-15T22:28:17.178984+0000","last_became_peered":"2021-03-15T22:28:17.178984+0000","last_unstale":"2021-03-15T22:36:45.368444+0000","last_undegraded":"2021-03-15T22:36:45.368444+0000","last_fullsized":"2021-03-15T22:36:45.368444+0000","mapping_epoch":557,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":558,"parent":"0.0","parent_split_bits":0,"last_scrub":"1019'1385","last_scrub_stamp":"2021-03-15T22:36:15.114641+0000","last_deep_scrub":"690'772","last_deep_scrub_stamp":"2021-03-15T22:30:39.836165+0000","last_clean_scrub_stamp":"2021-03-15T22:36:15.114641+0000","log_size":1449,"ondisk_log_size":1449,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":50649467,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":420,"num_read_kb":425400,"num_write":316,"num_write_kb":120781,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":72,"num_bytes_recovered":34756985,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,1],"acting":[0,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":"1103","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:47.830031+0000","last_change":"2021-03-15T22:36:47.830031+0000","last_active":"2021-03-15T22:36:47.830031+0000","last_peered":"2021-03-15T22:36:47.830031+0000","last_clean":"2021-03-15T22:36:47.830031+0000","last_became_active":"2021-03-15T22:36:00.364246+0000","last_became_peered":"2021-03-15T22:36:00.364246+0000","last_unstale":"2021-03-15T22:36:47.830031+0000","last_undegraded":"2021-03-15T22:36:47.830031+0000","last_fullsized":"2021-03-15T22:36:47.830031+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1005,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:47.829896+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:36:47.829896+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":"1085","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549074+0000","last_change":"2021-03-15T22:36:13.313358+0000","last_active":"2021-03-15T22:36:45.549074+0000","last_peered":"2021-03-15T22:36:45.549074+0000","last_clean":"2021-03-15T22:36:45.549074+0000","last_became_active":"2021-03-15T22:34:55.347065+0000","last_became_peered":"2021-03-15T22:34:55.347065+0000","last_unstale":"2021-03-15T22:36:45.549074+0000","last_undegraded":"2021-03-15T22:36:45.549074+0000","last_fullsized":"2021-03-15T22:36:45.549074+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:13.313278+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:36:13.313278+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"129","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.551314+0000","last_change":"2021-03-15T22:36:44.551628+0000","last_active":"2021-03-15T22:36:45.551314+0000","last_peered":"2021-03-15T22:36:45.551314+0000","last_clean":"2021-03-15T22:36:45.551314+0000","last_became_active":"2021-03-15T22:34:55.355485+0000","last_became_peered":"2021-03-15T22:34:55.355485+0000","last_unstale":"2021-03-15T22:36:45.551314+0000","last_undegraded":"2021-03-15T22:36:45.551314+0000","last_fullsized":"2021-03-15T22:36:45.551314+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:44.551570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:36:44.551570+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.7","version":"555'26","reported_seq":"537","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103699+0000","last_change":"2021-03-15T22:36:20.050533+0000","last_active":"2021-03-15T22:36:46.103699+0000","last_peered":"2021-03-15T22:36:46.103699+0000","last_clean":"2021-03-15T22:36:46.103699+0000","last_became_active":"2021-03-15T22:34:10.322666+0000","last_became_peered":"2021-03-15T22:34:10.322666+0000","last_unstale":"2021-03-15T22:36:46.103699+0000","last_undegraded":"2021-03-15T22:36:46.103699+0000","last_fullsized":"2021-03-15T22:36:46.103699+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'26","last_scrub_stamp":"2021-03-15T22:36:20.050450+0000","last_deep_scrub":"149'22","last_deep_scrub_stamp":"2021-03-15T22:21:04.762292+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.050450+0000","log_size":26,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":644,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":42,"num_read_kb":27,"num_write":22,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":2576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":"124","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.552291+0000","last_change":"2021-03-15T22:35:45.355712+0000","last_active":"2021-03-15T22:36:45.552291+0000","last_peered":"2021-03-15T22:36:45.552291+0000","last_clean":"2021-03-15T22:36:45.552291+0000","last_became_active":"2021-03-15T22:34:55.353316+0000","last_became_peered":"2021-03-15T22:34:55.353316+0000","last_unstale":"2021-03-15T22:36:45.552291+0000","last_undegraded":"2021-03-15T22:36:45.552291+0000","last_fullsized":"2021-03-15T22:36:45.552291+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.355647+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.355647+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.6","version":"1042'3488","reported_seq":"7161","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.370395+0000","last_change":"2021-03-15T22:36:35.405911+0000","last_active":"2021-03-15T22:36:45.370395+0000","last_peered":"2021-03-15T22:36:45.370395+0000","last_clean":"2021-03-15T22:36:45.370395+0000","last_became_active":"2021-03-15T22:34:13.331560+0000","last_became_peered":"2021-03-15T22:34:13.331560+0000","last_unstale":"2021-03-15T22:36:45.370395+0000","last_undegraded":"2021-03-15T22:36:45.370395+0000","last_fullsized":"2021-03-15T22:36:45.370395+0000","mapping_epoch":897,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":898,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'3208","last_scrub_stamp":"2021-03-15T22:35:42.271839+0000","last_deep_scrub":"245'350","last_deep_scrub_stamp":"2021-03-15T22:22:40.995789+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.271839+0000","log_size":3488,"ondisk_log_size":3488,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":108494076,"num_objects":49,"num_object_clones":42,"num_object_copies":147,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":49,"num_whiteouts":3,"num_read":948,"num_read_kb":984364,"num_write":714,"num_write_kb":291172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":640,"num_bytes_recovered":264088494,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":"451","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103707+0000","last_change":"2021-03-15T22:35:52.857478+0000","last_active":"2021-03-15T22:36:46.103707+0000","last_peered":"2021-03-15T22:36:46.103707+0000","last_clean":"2021-03-15T22:36:46.103707+0000","last_became_active":"2021-03-15T22:35:52.857192+0000","last_became_peered":"2021-03-15T22:35:52.857192+0000","last_unstale":"2021-03-15T22:36:46.103707+0000","last_undegraded":"2021-03-15T22:36:46.103707+0000","last_fullsized":"2021-03-15T22:36:46.103707+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":997,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:17.761137+0000","last_clean_scrub_stamp":"2021-03-15T22:33:17.761137+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"1072","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831035+0000","last_change":"2021-03-15T22:36:22.385705+0000","last_active":"2021-03-15T22:36:45.831035+0000","last_peered":"2021-03-15T22:36:45.831035+0000","last_clean":"2021-03-15T22:36:45.831035+0000","last_became_active":"2021-03-15T22:36:22.385402+0000","last_became_peered":"2021-03-15T22:36:22.385402+0000","last_unstale":"2021-03-15T22:36:45.831035+0000","last_undegraded":"2021-03-15T22:36:45.831035+0000","last_fullsized":"2021-03-15T22:36:45.831035+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":1027,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":"131","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367374+0000","last_change":"2021-03-15T22:34:43.341562+0000","last_active":"2021-03-15T22:36:45.367374+0000","last_peered":"2021-03-15T22:36:45.367374+0000","last_clean":"2021-03-15T22:36:45.367374+0000","last_became_active":"2021-03-15T22:34:43.340955+0000","last_became_peered":"2021-03-15T22:34:43.340955+0000","last_unstale":"2021-03-15T22:36:45.367374+0000","last_undegraded":"2021-03-15T22:36:45.367374+0000","last_fullsized":"2021-03-15T22:36:45.367374+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:34.843478+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:31:39.555487+0000","last_clean_scrub_stamp":"2021-03-15T22:34:34.843478+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.5","version":"556'27","reported_seq":"253","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533363+0000","last_change":"2021-03-15T22:36:12.698401+0000","last_active":"2021-03-15T22:36:45.533363+0000","last_peered":"2021-03-15T22:36:45.533363+0000","last_clean":"2021-03-15T22:36:45.533363+0000","last_became_active":"2021-03-15T22:33:04.330531+0000","last_became_peered":"2021-03-15T22:33:04.330531+0000","last_unstale":"2021-03-15T22:36:45.533363+0000","last_undegraded":"2021-03-15T22:36:45.533363+0000","last_fullsized":"2021-03-15T22:36:45.533363+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'27","last_scrub_stamp":"2021-03-15T22:36:12.698329+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:12.698329+0000","log_size":27,"ondisk_log_size":27,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":466,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":32,"num_write":27,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":"1084","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831652+0000","last_change":"2021-03-15T22:35:22.409047+0000","last_active":"2021-03-15T22:36:45.831652+0000","last_peered":"2021-03-15T22:36:45.831652+0000","last_clean":"2021-03-15T22:36:45.831652+0000","last_became_active":"2021-03-15T22:34:55.352186+0000","last_became_peered":"2021-03-15T22:34:55.352186+0000","last_unstale":"2021-03-15T22:36:45.831652+0000","last_undegraded":"2021-03-15T22:36:45.831652+0000","last_fullsized":"2021-03-15T22:36:45.831652+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:22.408965+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:03.257476+0000","last_clean_scrub_stamp":"2021-03-15T22:35:22.408965+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"30","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549096+0000","last_change":"2021-03-15T22:36:24.391414+0000","last_active":"2021-03-15T22:36:45.549096+0000","last_peered":"2021-03-15T22:36:45.549096+0000","last_clean":"2021-03-15T22:36:45.549096+0000","last_became_active":"2021-03-15T22:36:24.390984+0000","last_became_peered":"2021-03-15T22:36:24.390984+0000","last_unstale":"2021-03-15T22:36:45.549096+0000","last_undegraded":"2021-03-15T22:36:45.549096+0000","last_fullsized":"2021-03-15T22:36:45.549096+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,4],"acting":[1,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"1037'1052","reported_seq":"4311","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831645+0000","last_change":"2021-03-15T22:36:35.374467+0000","last_active":"2021-03-15T22:36:45.831645+0000","last_peered":"2021-03-15T22:36:45.831645+0000","last_clean":"2021-03-15T22:36:45.831645+0000","last_became_active":"2021-03-15T22:35:03.346751+0000","last_became_peered":"2021-03-15T22:35:03.346751+0000","last_unstale":"2021-03-15T22:36:45.831645+0000","last_undegraded":"2021-03-15T22:36:45.831645+0000","last_fullsized":"2021-03-15T22:36:45.831645+0000","mapping_epoch":947,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":948,"parent":"0.0","parent_split_bits":0,"last_scrub":"1020'1020","last_scrub_stamp":"2021-03-15T22:36:16.818677+0000","last_deep_scrub":"781'659","last_deep_scrub_stamp":"2021-03-15T22:32:11.467520+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.818677+0000","log_size":1052,"ondisk_log_size":1052,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37060250,"num_objects":17,"num_object_clones":15,"num_object_copies":51,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":17,"num_whiteouts":1,"num_read":267,"num_read_kb":273197,"num_write":228,"num_write_kb":99079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":182,"num_bytes_recovered":109648904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":"868","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103431+0000","last_change":"2021-03-15T22:36:04.904466+0000","last_active":"2021-03-15T22:36:46.103431+0000","last_peered":"2021-03-15T22:36:46.103431+0000","last_clean":"2021-03-15T22:36:46.103431+0000","last_became_active":"2021-03-15T22:36:04.904112+0000","last_became_peered":"2021-03-15T22:36:04.904112+0000","last_unstale":"2021-03-15T22:36:46.103431+0000","last_undegraded":"2021-03-15T22:36:46.103431+0000","last_fullsized":"2021-03-15T22:36:46.103431+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1009,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:12.830484+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:12.830484+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":"545","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367426+0000","last_change":"2021-03-15T22:34:35.410541+0000","last_active":"2021-03-15T22:36:45.367426+0000","last_peered":"2021-03-15T22:36:45.367426+0000","last_clean":"2021-03-15T22:36:45.367426+0000","last_became_active":"2021-03-15T22:34:10.321100+0000","last_became_peered":"2021-03-15T22:34:10.321100+0000","last_unstale":"2021-03-15T22:36:45.367426+0000","last_undegraded":"2021-03-15T22:36:45.367426+0000","last_fullsized":"2021-03-15T22:36:45.367426+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:35.410485+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:52.953223+0000","last_clean_scrub_stamp":"2021-03-15T22:34:35.410485+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"62'5","reported_seq":"2177","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:48.222484+0000","last_change":"2021-03-15T22:35:10.452906+0000","last_active":"2021-03-15T22:36:48.222484+0000","last_peered":"2021-03-15T22:36:48.222484+0000","last_clean":"2021-03-15T22:36:48.222484+0000","last_became_active":"2021-03-15T22:28:12.510452+0000","last_became_peered":"2021-03-15T22:28:12.510452+0000","last_unstale":"2021-03-15T22:36:48.222484+0000","last_undegraded":"2021-03-15T22:36:48.222484+0000","last_fullsized":"2021-03-15T22:36:48.222484+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":53,"last_epoch_clean":553,"parent":"0.0","parent_split_bits":0,"last_scrub":"62'5","last_scrub_stamp":"2021-03-15T22:35:10.452809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:49.940895+0000","last_clean_scrub_stamp":"2021-03-15T22:35:10.452809+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1119,"num_read_kb":1114,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"555'28","reported_seq":"643","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.370507+0000","last_change":"2021-03-15T22:34:55.414460+0000","last_active":"2021-03-15T22:36:45.370507+0000","last_peered":"2021-03-15T22:36:45.370507+0000","last_clean":"2021-03-15T22:36:45.370507+0000","last_became_active":"2021-03-15T22:34:55.359023+0000","last_became_peered":"2021-03-15T22:34:55.359023+0000","last_unstale":"2021-03-15T22:36:45.370507+0000","last_undegraded":"2021-03-15T22:36:45.370507+0000","last_fullsized":"2021-03-15T22:36:45.370507+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'28","last_scrub_stamp":"2021-03-15T22:34:40.472800+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.472800+0000","log_size":28,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":630,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":33,"num_write":28,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":2520,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":"438","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103777+0000","last_change":"2021-03-15T22:34:40.799217+0000","last_active":"2021-03-15T22:36:46.103777+0000","last_peered":"2021-03-15T22:36:46.103777+0000","last_clean":"2021-03-15T22:36:46.103777+0000","last_became_active":"2021-03-15T22:31:18.270276+0000","last_became_peered":"2021-03-15T22:31:18.270276+0000","last_unstale":"2021-03-15T22:36:46.103777+0000","last_undegraded":"2021-03-15T22:36:46.103777+0000","last_fullsized":"2021-03-15T22:36:46.103777+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:40.799153+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:36.215040+0000","last_clean_scrub_stamp":"2021-03-15T22:34:40.799153+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"1072","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533227+0000","last_change":"2021-03-15T22:36:24.387855+0000","last_active":"2021-03-15T22:36:45.533227+0000","last_peered":"2021-03-15T22:36:45.533227+0000","last_clean":"2021-03-15T22:36:45.533227+0000","last_became_active":"2021-03-15T22:36:24.387249+0000","last_became_peered":"2021-03-15T22:36:24.387249+0000","last_unstale":"2021-03-15T22:36:45.533227+0000","last_undegraded":"2021-03-15T22:36:45.533227+0000","last_fullsized":"2021-03-15T22:36:45.533227+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,3],"acting":[6,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.0","version":"1041'3571","reported_seq":"7805","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.102699+0000","last_change":"2021-03-15T22:36:35.389045+0000","last_active":"2021-03-15T22:36:46.102699+0000","last_peered":"2021-03-15T22:36:46.102699+0000","last_clean":"2021-03-15T22:36:46.102699+0000","last_became_active":"2021-03-15T22:35:17.487261+0000","last_became_peered":"2021-03-15T22:35:17.487261+0000","last_unstale":"2021-03-15T22:36:46.102699+0000","last_undegraded":"2021-03-15T22:36:46.102699+0000","last_fullsized":"2021-03-15T22:36:46.102699+0000","mapping_epoch":961,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":962,"parent":"0.0","parent_split_bits":0,"last_scrub":"1020'3444","last_scrub_stamp":"2021-03-15T22:36:16.067782+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:16.067782+0000","log_size":3571,"ondisk_log_size":3571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":121620780,"num_objects":52,"num_object_clones":45,"num_object_copies":156,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":52,"num_whiteouts":2,"num_read":1299,"num_read_kb":1394227,"num_write":801,"num_write_kb":336561,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":721,"num_bytes_recovered":351021959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":"1108","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.550953+0000","last_change":"2021-03-15T22:36:45.550914+0000","last_active":"2021-03-15T22:36:45.550953+0000","last_peered":"2021-03-15T22:36:45.550953+0000","last_clean":"2021-03-15T22:36:45.550953+0000","last_became_active":"2021-03-15T22:36:16.372177+0000","last_became_peered":"2021-03-15T22:36:16.372177+0000","last_unstale":"2021-03-15T22:36:45.550953+0000","last_undegraded":"2021-03-15T22:36:45.550953+0000","last_fullsized":"2021-03-15T22:36:45.550953+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1021,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:45.550861+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:15.593307+0000","last_clean_scrub_stamp":"2021-03-15T22:36:45.550861+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":"1103","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549329+0000","last_change":"2021-03-15T22:36:24.397895+0000","last_active":"2021-03-15T22:36:45.549329+0000","last_peered":"2021-03-15T22:36:45.549329+0000","last_clean":"2021-03-15T22:36:45.549329+0000","last_became_active":"2021-03-15T22:34:55.346048+0000","last_became_peered":"2021-03-15T22:34:55.346048+0000","last_unstale":"2021-03-15T22:36:45.549329+0000","last_undegraded":"2021-03-15T22:36:45.549329+0000","last_fullsized":"2021-03-15T22:36:45.549329+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:24.397819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:32:44.675891+0000","last_clean_scrub_stamp":"2021-03-15T22:36:24.397819+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"556'41","reported_seq":"593","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368788+0000","last_change":"2021-03-15T22:35:21.314851+0000","last_active":"2021-03-15T22:36:45.368788+0000","last_peered":"2021-03-15T22:36:45.368788+0000","last_clean":"2021-03-15T22:36:45.368788+0000","last_became_active":"2021-03-15T22:29:51.212525+0000","last_became_peered":"2021-03-15T22:29:51.212525+0000","last_unstale":"2021-03-15T22:36:45.368788+0000","last_undegraded":"2021-03-15T22:36:45.368788+0000","last_fullsized":"2021-03-15T22:36:45.368788+0000","mapping_epoch":641,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":643,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'41","last_scrub_stamp":"2021-03-15T22:35:21.314763+0000","last_deep_scrub":"78'31","last_deep_scrub_stamp":"2021-03-15T22:18:56.832638+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.314763+0000","log_size":41,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":642,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":59,"num_read_kb":41,"num_write":34,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":1284,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":"233","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.532930+0000","last_change":"2021-03-15T22:35:39.707993+0000","last_active":"2021-03-15T22:36:45.532930+0000","last_peered":"2021-03-15T22:36:45.532930+0000","last_clean":"2021-03-15T22:36:45.532930+0000","last_became_active":"2021-03-15T22:33:04.337244+0000","last_became_peered":"2021-03-15T22:33:04.337244+0000","last_unstale":"2021-03-15T22:36:45.532930+0000","last_undegraded":"2021-03-15T22:36:45.532930+0000","last_fullsized":"2021-03-15T22:36:45.532930+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:39.707900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:16.581514+0000","last_clean_scrub_stamp":"2021-03-15T22:35:39.707900+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.7","version":"0'0","reported_seq":"1072","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533327+0000","last_change":"2021-03-15T22:36:24.393232+0000","last_active":"2021-03-15T22:36:45.533327+0000","last_peered":"2021-03-15T22:36:45.533327+0000","last_clean":"2021-03-15T22:36:45.533327+0000","last_became_active":"2021-03-15T22:36:24.392909+0000","last_became_peered":"2021-03-15T22:36:24.392909+0000","last_unstale":"2021-03-15T22:36:45.533327+0000","last_undegraded":"2021-03-15T22:36:45.533327+0000","last_fullsized":"2021-03-15T22:36:45.533327+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,5],"acting":[6,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"7.1","version":"1040'624","reported_seq":"3894","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367069+0000","last_change":"2021-03-15T22:36:35.373209+0000","last_active":"2021-03-15T22:36:45.367069+0000","last_peered":"2021-03-15T22:36:45.367069+0000","last_clean":"2021-03-15T22:36:45.367069+0000","last_became_active":"2021-03-15T22:34:15.498612+0000","last_became_peered":"2021-03-15T22:34:15.498612+0000","last_unstale":"2021-03-15T22:36:45.367069+0000","last_undegraded":"2021-03-15T22:36:45.367069+0000","last_fullsized":"2021-03-15T22:36:45.367069+0000","mapping_epoch":899,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":900,"parent":"0.0","parent_split_bits":0,"last_scrub":"864'463","last_scrub_stamp":"2021-03-15T22:33:40.563144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.563144+0000","log_size":624,"ondisk_log_size":624,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":23933460,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":185,"num_read_kb":221865,"num_write":115,"num_write_kb":59307,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":132,"num_bytes_recovered":84391207,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,0],"acting":[4,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":"587","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367927+0000","last_change":"2021-03-15T22:36:20.369600+0000","last_active":"2021-03-15T22:36:45.367927+0000","last_peered":"2021-03-15T22:36:45.367927+0000","last_clean":"2021-03-15T22:36:45.367927+0000","last_became_active":"2021-03-15T22:36:20.369276+0000","last_became_peered":"2021-03-15T22:36:20.369276+0000","last_unstale":"2021-03-15T22:36:45.367927+0000","last_undegraded":"2021-03-15T22:36:45.367927+0000","last_fullsized":"2021-03-15T22:36:45.367927+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1025,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:10.674805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:10.674805+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":"1026","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103837+0000","last_change":"2021-03-15T22:36:14.037839+0000","last_active":"2021-03-15T22:36:46.103837+0000","last_peered":"2021-03-15T22:36:46.103837+0000","last_clean":"2021-03-15T22:36:46.103837+0000","last_became_active":"2021-03-15T22:34:55.355757+0000","last_became_peered":"2021-03-15T22:34:55.355757+0000","last_unstale":"2021-03-15T22:36:46.103837+0000","last_undegraded":"2021-03-15T22:36:46.103837+0000","last_fullsized":"2021-03-15T22:36:46.103837+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:14.037757+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:32.017806+0000","last_clean_scrub_stamp":"2021-03-15T22:36:14.037757+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.2","version":"610'37","reported_seq":"1007","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103049+0000","last_change":"2021-03-15T22:36:13.027720+0000","last_active":"2021-03-15T22:36:46.103049+0000","last_peered":"2021-03-15T22:36:46.103049+0000","last_clean":"2021-03-15T22:36:46.103049+0000","last_became_active":"2021-03-15T22:34:10.321886+0000","last_became_peered":"2021-03-15T22:34:10.321886+0000","last_unstale":"2021-03-15T22:36:46.103049+0000","last_undegraded":"2021-03-15T22:36:46.103049+0000","last_fullsized":"2021-03-15T22:36:46.103049+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"610'37","last_scrub_stamp":"2021-03-15T22:36:13.027640+0000","last_deep_scrub":"610'37","last_deep_scrub_stamp":"2021-03-15T22:29:20.139319+0000","last_clean_scrub_stamp":"2021-03-15T22:36:13.027640+0000","log_size":37,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":11,"num_object_clones":0,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":48,"num_read_kb":31,"num_write":26,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":55,"num_bytes_recovered":4700,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,0],"acting":[5,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"836'11","reported_seq":"681","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:46.541474+0000","last_change":"2021-03-15T22:35:44.256542+0000","last_active":"2021-03-15T22:36:46.541474+0000","last_peered":"2021-03-15T22:36:46.541474+0000","last_clean":"2021-03-15T22:36:46.541474+0000","last_became_active":"2021-03-15T22:33:04.336294+0000","last_became_peered":"2021-03-15T22:33:04.336294+0000","last_unstale":"2021-03-15T22:36:46.541474+0000","last_undegraded":"2021-03-15T22:36:46.541474+0000","last_fullsized":"2021-03-15T22:36:46.541474+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"836'11","last_scrub_stamp":"2021-03-15T22:35:44.256486+0000","last_deep_scrub":"721'9","last_deep_scrub_stamp":"2021-03-15T22:31:10.971303+0000","last_clean_scrub_stamp":"2021-03-15T22:35:44.256486+0000","log_size":11,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"30","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367709+0000","last_change":"2021-03-15T22:36:24.387641+0000","last_active":"2021-03-15T22:36:45.367709+0000","last_peered":"2021-03-15T22:36:45.367709+0000","last_clean":"2021-03-15T22:36:45.367709+0000","last_became_active":"2021-03-15T22:36:24.387299+0000","last_became_peered":"2021-03-15T22:36:24.387299+0000","last_unstale":"2021-03-15T22:36:45.367709+0000","last_undegraded":"2021-03-15T22:36:45.367709+0000","last_fullsized":"2021-03-15T22:36:45.367709+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"1041'1002","reported_seq":"4303","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367860+0000","last_change":"2021-03-15T22:36:35.384139+0000","last_active":"2021-03-15T22:36:45.367860+0000","last_peered":"2021-03-15T22:36:45.367860+0000","last_clean":"2021-03-15T22:36:45.367860+0000","last_became_active":"2021-03-15T22:34:58.350873+0000","last_became_peered":"2021-03-15T22:34:58.350873+0000","last_unstale":"2021-03-15T22:36:45.367860+0000","last_undegraded":"2021-03-15T22:36:45.367860+0000","last_fullsized":"2021-03-15T22:36:45.367860+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"959'901","last_scrub_stamp":"2021-03-15T22:35:16.957336+0000","last_deep_scrub":"366'202","last_deep_scrub_stamp":"2021-03-15T22:24:42.006150+0000","last_clean_scrub_stamp":"2021-03-15T22:35:16.957336+0000","log_size":1002,"ondisk_log_size":1002,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":37536135,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":275,"num_read_kb":241829,"num_write":236,"num_write_kb":92610,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":114552572,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":"1089","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831485+0000","last_change":"2021-03-15T22:36:08.382982+0000","last_active":"2021-03-15T22:36:45.831485+0000","last_peered":"2021-03-15T22:36:45.831485+0000","last_clean":"2021-03-15T22:36:45.831485+0000","last_became_active":"2021-03-15T22:36:08.382713+0000","last_became_peered":"2021-03-15T22:36:08.382713+0000","last_unstale":"2021-03-15T22:36:45.831485+0000","last_undegraded":"2021-03-15T22:36:45.831485+0000","last_fullsized":"2021-03-15T22:36:45.831485+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1013,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:18.507204+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:18.507204+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":"531","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.370575+0000","last_change":"2021-03-15T22:36:34.383378+0000","last_active":"2021-03-15T22:36:45.370575+0000","last_peered":"2021-03-15T22:36:45.370575+0000","last_clean":"2021-03-15T22:36:45.370575+0000","last_became_active":"2021-03-15T22:36:34.382987+0000","last_became_peered":"2021-03-15T22:36:34.382987+0000","last_unstale":"2021-03-15T22:36:45.370575+0000","last_undegraded":"2021-03-15T22:36:45.370575+0000","last_fullsized":"2021-03-15T22:36:45.370575+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1039,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:22.598174+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:30:37.798388+0000","last_clean_scrub_stamp":"2021-03-15T22:33:22.598174+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1","version":"556'34","reported_seq":"1241","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367386+0000","last_change":"2021-03-15T22:33:43.514160+0000","last_active":"2021-03-15T22:36:45.367386+0000","last_peered":"2021-03-15T22:36:45.367386+0000","last_clean":"2021-03-15T22:36:45.367386+0000","last_became_active":"2021-03-15T22:33:04.336145+0000","last_became_peered":"2021-03-15T22:33:04.336145+0000","last_unstale":"2021-03-15T22:36:45.367386+0000","last_undegraded":"2021-03-15T22:36:45.367386+0000","last_fullsized":"2021-03-15T22:36:45.367386+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":835,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'34","last_scrub_stamp":"2021-03-15T22:33:43.514088+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:33:43.514088+0000","log_size":34,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":940,"num_objects":13,"num_object_clones":0,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":60,"num_read_kb":41,"num_write":34,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":52,"num_bytes_recovered":3760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.0","version":"592'8","reported_seq":"574","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368963+0000","last_change":"2021-03-15T22:33:20.685041+0000","last_active":"2021-03-15T22:36:45.368963+0000","last_peered":"2021-03-15T22:36:45.368963+0000","last_clean":"2021-03-15T22:36:45.368963+0000","last_became_active":"2021-03-15T22:28:59.276230+0000","last_became_peered":"2021-03-15T22:28:59.276230+0000","last_unstale":"2021-03-15T22:36:45.368963+0000","last_undegraded":"2021-03-15T22:36:45.368963+0000","last_fullsized":"2021-03-15T22:36:45.368963+0000","mapping_epoch":589,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":590,"parent":"0.0","parent_split_bits":0,"last_scrub":"592'8","last_scrub_stamp":"2021-03-15T22:33:20.684976+0000","last_deep_scrub":"421'6","last_deep_scrub_stamp":"2021-03-15T22:27:47.109636+0000","last_clean_scrub_stamp":"2021-03-15T22:33:20.684976+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"30","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367640+0000","last_change":"2021-03-15T22:36:24.391783+0000","last_active":"2021-03-15T22:36:45.367640+0000","last_peered":"2021-03-15T22:36:45.367640+0000","last_clean":"2021-03-15T22:36:45.367640+0000","last_became_active":"2021-03-15T22:36:24.391417+0000","last_became_peered":"2021-03-15T22:36:24.391417+0000","last_unstale":"2021-03-15T22:36:45.367640+0000","last_undegraded":"2021-03-15T22:36:45.367640+0000","last_fullsized":"2021-03-15T22:36:45.367640+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":3,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,6],"acting":[4,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"1041'1740","reported_seq":"4864","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549250+0000","last_change":"2021-03-15T22:36:35.375063+0000","last_active":"2021-03-15T22:36:45.549250+0000","last_peered":"2021-03-15T22:36:45.549250+0000","last_clean":"2021-03-15T22:36:45.549250+0000","last_became_active":"2021-03-15T22:34:58.337099+0000","last_became_peered":"2021-03-15T22:34:58.337099+0000","last_unstale":"2021-03-15T22:36:45.549250+0000","last_undegraded":"2021-03-15T22:36:45.549250+0000","last_fullsized":"2021-03-15T22:36:45.549250+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"962'1611","last_scrub_stamp":"2021-03-15T22:35:21.054192+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.054192+0000","log_size":1740,"ondisk_log_size":1740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":45882320,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":561,"num_read_kb":593462,"num_write":354,"num_write_kb":124562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":466,"num_bytes_recovered":226741522,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":"1144","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367170+0000","last_change":"2021-03-15T22:36:12.386049+0000","last_active":"2021-03-15T22:36:45.367170+0000","last_peered":"2021-03-15T22:36:45.367170+0000","last_clean":"2021-03-15T22:36:45.367170+0000","last_became_active":"2021-03-15T22:36:12.385681+0000","last_became_peered":"2021-03-15T22:36:12.385681+0000","last_unstale":"2021-03-15T22:36:45.367170+0000","last_undegraded":"2021-03-15T22:36:45.367170+0000","last_fullsized":"2021-03-15T22:36:45.367170+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":1017,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:32:39.271262+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:32:39.271262+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":"850","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.369524+0000","last_change":"2021-03-15T22:36:30.397474+0000","last_active":"2021-03-15T22:36:45.369524+0000","last_peered":"2021-03-15T22:36:45.369524+0000","last_clean":"2021-03-15T22:36:45.369524+0000","last_became_active":"2021-03-15T22:36:30.390223+0000","last_became_peered":"2021-03-15T22:36:30.390223+0000","last_unstale":"2021-03-15T22:36:45.369524+0000","last_undegraded":"2021-03-15T22:36:45.369524+0000","last_fullsized":"2021-03-15T22:36:45.369524+0000","mapping_epoch":590,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1035,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:41.870368+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:38.487735+0000","last_clean_scrub_stamp":"2021-03-15T22:34:41.870368+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.0","version":"555'46","reported_seq":"483","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.831178+0000","last_change":"2021-03-15T22:35:08.402702+0000","last_active":"2021-03-15T22:36:45.831178+0000","last_peered":"2021-03-15T22:36:45.831178+0000","last_clean":"2021-03-15T22:36:45.831178+0000","last_became_active":"2021-03-15T22:34:10.349731+0000","last_became_peered":"2021-03-15T22:34:10.349731+0000","last_unstale":"2021-03-15T22:36:45.831178+0000","last_undegraded":"2021-03-15T22:36:45.831178+0000","last_fullsized":"2021-03-15T22:36:45.831178+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":896,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'46","last_scrub_stamp":"2021-03-15T22:35:08.402641+0000","last_deep_scrub":"82'34","last_deep_scrub_stamp":"2021-03-15T22:26:46.453319+0000","last_clean_scrub_stamp":"2021-03-15T22:35:08.402641+0000","log_size":46,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":846,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":63,"num_read_kb":43,"num_write":36,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":3384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":"1094","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367211+0000","last_change":"2021-03-15T22:34:43.343527+0000","last_active":"2021-03-15T22:36:45.367211+0000","last_peered":"2021-03-15T22:36:45.367211+0000","last_clean":"2021-03-15T22:36:45.367211+0000","last_became_active":"2021-03-15T22:34:43.343255+0000","last_became_peered":"2021-03-15T22:34:43.343255+0000","last_unstale":"2021-03-15T22:36:45.367211+0000","last_undegraded":"2021-03-15T22:36:45.367211+0000","last_fullsized":"2021-03-15T22:36:45.367211+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:34:39.673338+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:21:59.228452+0000","last_clean_scrub_stamp":"2021-03-15T22:34:39.673338+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"0'0","reported_seq":"30","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103097+0000","last_change":"2021-03-15T22:36:24.389841+0000","last_active":"2021-03-15T22:36:46.103097+0000","last_peered":"2021-03-15T22:36:46.103097+0000","last_clean":"2021-03-15T22:36:46.103097+0000","last_became_active":"2021-03-15T22:36:24.389490+0000","last_became_peered":"2021-03-15T22:36:24.389490+0000","last_unstale":"2021-03-15T22:36:46.103097+0000","last_undegraded":"2021-03-15T22:36:46.103097+0000","last_fullsized":"2021-03-15T22:36:46.103097+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,6],"acting":[5,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"1040'1519","reported_seq":"4901","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533398+0000","last_change":"2021-03-15T22:36:35.379125+0000","last_active":"2021-03-15T22:36:45.533398+0000","last_peered":"2021-03-15T22:36:45.533398+0000","last_clean":"2021-03-15T22:36:45.533398+0000","last_became_active":"2021-03-15T22:34:58.344406+0000","last_became_peered":"2021-03-15T22:34:58.344406+0000","last_unstale":"2021-03-15T22:36:45.533398+0000","last_undegraded":"2021-03-15T22:36:45.533398+0000","last_fullsized":"2021-03-15T22:36:45.533398+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'1407","last_scrub_stamp":"2021-03-15T22:35:43.758914+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:43.758914+0000","log_size":1519,"ondisk_log_size":1519,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":45609496,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":429,"num_read_kb":395184,"num_write":317,"num_write_kb":119059,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":332,"num_bytes_recovered":127958264,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,2],"acting":[6,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":"993","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103824+0000","last_change":"2021-03-15T22:34:55.356480+0000","last_active":"2021-03-15T22:36:46.103824+0000","last_peered":"2021-03-15T22:36:46.103824+0000","last_clean":"2021-03-15T22:36:46.103824+0000","last_became_active":"2021-03-15T22:34:55.356213+0000","last_became_peered":"2021-03-15T22:34:55.356213+0000","last_unstale":"2021-03-15T22:36:46.103824+0000","last_undegraded":"2021-03-15T22:36:46.103824+0000","last_fullsized":"2021-03-15T22:36:46.103824+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:40.729838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:58.999017+0000","last_clean_scrub_stamp":"2021-03-15T22:33:40.729838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.e","version":"732'14","reported_seq":"1359","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368106+0000","last_change":"2021-03-15T22:35:19.468482+0000","last_active":"2021-03-15T22:36:45.368106+0000","last_peered":"2021-03-15T22:36:45.368106+0000","last_clean":"2021-03-15T22:36:45.368106+0000","last_became_active":"2021-03-15T22:31:18.257261+0000","last_became_peered":"2021-03-15T22:31:18.257261+0000","last_unstale":"2021-03-15T22:36:45.368106+0000","last_undegraded":"2021-03-15T22:36:45.368106+0000","last_fullsized":"2021-03-15T22:36:45.368106+0000","mapping_epoch":642,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"732'14","last_scrub_stamp":"2021-03-15T22:35:19.468402+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.468402+0000","log_size":12,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":"478","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549160+0000","last_change":"2021-03-15T22:36:26.941424+0000","last_active":"2021-03-15T22:36:45.549160+0000","last_peered":"2021-03-15T22:36:45.549160+0000","last_clean":"2021-03-15T22:36:45.549160+0000","last_became_active":"2021-03-15T22:36:26.941149+0000","last_became_peered":"2021-03-15T22:36:26.941149+0000","last_unstale":"2021-03-15T22:36:45.549160+0000","last_undegraded":"2021-03-15T22:36:45.549160+0000","last_fullsized":"2021-03-15T22:36:45.549160+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1031,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:45.165838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:27:43.184429+0000","last_clean_scrub_stamp":"2021-03-15T22:35:45.165838+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.f","version":"556'29","reported_seq":"215","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549786+0000","last_change":"2021-03-15T22:36:21.293286+0000","last_active":"2021-03-15T22:36:45.549786+0000","last_peered":"2021-03-15T22:36:45.549786+0000","last_clean":"2021-03-15T22:36:45.549786+0000","last_became_active":"2021-03-15T22:33:45.317888+0000","last_became_peered":"2021-03-15T22:33:45.317888+0000","last_unstale":"2021-03-15T22:36:45.549786+0000","last_undegraded":"2021-03-15T22:36:45.549786+0000","last_fullsized":"2021-03-15T22:36:45.549786+0000","mapping_epoch":874,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":875,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'29","last_scrub_stamp":"2021-03-15T22:36:21.293219+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:36:21.293219+0000","log_size":29,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":422,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":50,"num_read_kb":37,"num_write":30,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33,"num_bytes_recovered":1266,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.d","version":"1040'530","reported_seq":"3510","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533420+0000","last_change":"2021-03-15T22:36:35.372629+0000","last_active":"2021-03-15T22:36:45.533420+0000","last_peered":"2021-03-15T22:36:45.533420+0000","last_clean":"2021-03-15T22:36:45.533420+0000","last_became_active":"2021-03-15T22:33:08.354872+0000","last_became_peered":"2021-03-15T22:33:08.354872+0000","last_unstale":"2021-03-15T22:36:45.533420+0000","last_undegraded":"2021-03-15T22:36:45.533420+0000","last_fullsized":"2021-03-15T22:36:45.533420+0000","mapping_epoch":837,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"986'470","last_scrub_stamp":"2021-03-15T22:35:42.721100+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:35:42.721100+0000","log_size":530,"ondisk_log_size":530,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":20081853,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":190,"num_read_kb":197819,"num_write":106,"num_write_kb":42316,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":31290769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,0],"acting":[6,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":"131","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368503+0000","last_change":"2021-03-15T22:34:43.340972+0000","last_active":"2021-03-15T22:36:45.368503+0000","last_peered":"2021-03-15T22:36:45.368503+0000","last_clean":"2021-03-15T22:36:45.368503+0000","last_became_active":"2021-03-15T22:34:43.340600+0000","last_became_peered":"2021-03-15T22:34:43.340600+0000","last_unstale":"2021-03-15T22:36:45.368503+0000","last_undegraded":"2021-03-15T22:36:45.368503+0000","last_fullsized":"2021-03-15T22:36:45.368503+0000","mapping_epoch":927,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":928,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:09.612126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:19:25.878008+0000","last_clean_scrub_stamp":"2021-03-15T22:33:09.612126+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.f","version":"643'9","reported_seq":"1091","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.103864+0000","last_change":"2021-03-15T22:35:21.917891+0000","last_active":"2021-03-15T22:36:46.103864+0000","last_peered":"2021-03-15T22:36:46.103864+0000","last_clean":"2021-03-15T22:36:46.103864+0000","last_became_active":"2021-03-15T22:33:08.352367+0000","last_became_peered":"2021-03-15T22:33:08.352367+0000","last_unstale":"2021-03-15T22:36:46.103864+0000","last_undegraded":"2021-03-15T22:36:46.103864+0000","last_fullsized":"2021-03-15T22:36:46.103864+0000","mapping_epoch":837,"log_start":"142'3","ondisk_log_start":"142'3","created":60,"last_epoch_clean":838,"parent":"0.0","parent_split_bits":0,"last_scrub":"643'9","last_scrub_stamp":"2021-03-15T22:35:21.917819+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.917819+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":"13","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.102794+0000","last_change":"2021-03-15T22:36:42.665723+0000","last_active":"2021-03-15T22:36:46.102794+0000","last_peered":"2021-03-15T22:36:46.102794+0000","last_clean":"2021-03-15T22:36:46.102794+0000","last_became_active":"2021-03-15T22:36:42.665274+0000","last_became_peered":"2021-03-15T22:36:42.665274+0000","last_unstale":"2021-03-15T22:36:46.102794+0000","last_undegraded":"2021-03-15T22:36:46.102794+0000","last_fullsized":"2021-03-15T22:36:46.102794+0000","mapping_epoch":1045,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1046,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:24.285991+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:18.461102+0000","last_clean_scrub_stamp":"2021-03-15T22:36:24.285991+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"556'23","reported_seq":"1220","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367971+0000","last_change":"2021-03-15T22:35:21.384129+0000","last_active":"2021-03-15T22:36:45.367971+0000","last_peered":"2021-03-15T22:36:45.367971+0000","last_clean":"2021-03-15T22:36:45.367971+0000","last_became_active":"2021-03-15T22:34:10.330608+0000","last_became_peered":"2021-03-15T22:34:10.330608+0000","last_unstale":"2021-03-15T22:36:45.367971+0000","last_undegraded":"2021-03-15T22:36:45.367971+0000","last_fullsized":"2021-03-15T22:36:45.367971+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":895,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'23","last_scrub_stamp":"2021-03-15T22:35:21.384037+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:21.384037+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":738,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":44,"num_read_kb":29,"num_write":24,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":45,"num_bytes_recovered":3690,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,3],"acting":[4,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.8","version":"0'0","reported_seq":"30","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:46.102729+0000","last_change":"2021-03-15T22:36:24.389116+0000","last_active":"2021-03-15T22:36:46.102729+0000","last_peered":"2021-03-15T22:36:46.102729+0000","last_clean":"2021-03-15T22:36:46.102729+0000","last_became_active":"2021-03-15T22:36:24.388690+0000","last_became_peered":"2021-03-15T22:36:24.388690+0000","last_unstale":"2021-03-15T22:36:46.102729+0000","last_undegraded":"2021-03-15T22:36:46.102729+0000","last_fullsized":"2021-03-15T22:36:46.102729+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"1036'508","reported_seq":"2995","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.552291+0000","last_change":"2021-03-15T22:36:35.367176+0000","last_active":"2021-03-15T22:36:45.552291+0000","last_peered":"2021-03-15T22:36:45.552291+0000","last_clean":"2021-03-15T22:36:45.552291+0000","last_became_active":"2021-03-15T22:34:58.350923+0000","last_became_peered":"2021-03-15T22:34:58.350923+0000","last_unstale":"2021-03-15T22:36:45.552291+0000","last_undegraded":"2021-03-15T22:36:45.552291+0000","last_fullsized":"2021-03-15T22:36:45.552291+0000","mapping_epoch":942,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":943,"parent":"0.0","parent_split_bits":0,"last_scrub":"1027'503","last_scrub_stamp":"2021-03-15T22:36:24.720629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:24.720629+0000","log_size":508,"ondisk_log_size":508,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":8377674,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":158,"num_read_kb":164364,"num_write":122,"num_write_kb":40554,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":103,"num_bytes_recovered":52707265,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":"798","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368480+0000","last_change":"2021-03-15T22:36:12.855165+0000","last_active":"2021-03-15T22:36:45.368480+0000","last_peered":"2021-03-15T22:36:45.368480+0000","last_clean":"2021-03-15T22:36:45.368480+0000","last_became_active":"2021-03-15T22:34:55.345267+0000","last_became_peered":"2021-03-15T22:34:55.345267+0000","last_unstale":"2021-03-15T22:36:45.368480+0000","last_undegraded":"2021-03-15T22:36:45.368480+0000","last_fullsized":"2021-03-15T22:36:45.368480+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:12.855079+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:23:02.474236+0000","last_clean_scrub_stamp":"2021-03-15T22:36:12.855079+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":"1085","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.549567+0000","last_change":"2021-03-15T22:35:12.091714+0000","last_active":"2021-03-15T22:36:45.549567+0000","last_peered":"2021-03-15T22:36:45.549567+0000","last_clean":"2021-03-15T22:36:45.549567+0000","last_became_active":"2021-03-15T22:31:18.250332+0000","last_became_peered":"2021-03-15T22:31:18.250332+0000","last_unstale":"2021-03-15T22:36:45.549567+0000","last_undegraded":"2021-03-15T22:36:45.549567+0000","last_fullsized":"2021-03-15T22:36:45.549567+0000","mapping_epoch":552,"log_start":"0'0","ondisk_log_start":"0'0","created":60,"last_epoch_clean":729,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:12.091650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:22:03.851047+0000","last_clean_scrub_stamp":"2021-03-15T22:35:12.091650+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":"260","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.533053+0000","last_change":"2021-03-15T22:36:18.384469+0000","last_active":"2021-03-15T22:36:45.533053+0000","last_peered":"2021-03-15T22:36:45.533053+0000","last_clean":"2021-03-15T22:36:45.533053+0000","last_became_active":"2021-03-15T22:36:18.384088+0000","last_became_peered":"2021-03-15T22:36:18.384088+0000","last_unstale":"2021-03-15T22:36:45.533053+0000","last_undegraded":"2021-03-15T22:36:45.533053+0000","last_fullsized":"2021-03-15T22:36:45.533053+0000","mapping_epoch":894,"log_start":"0'0","ondisk_log_start":"0'0","created":56,"last_epoch_clean":1023,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:20.678246+0000","last_clean_scrub_stamp":"2021-03-15T22:35:20.678246+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,3],"acting":[6,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.d","version":"556'47","reported_seq":"1199","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367776+0000","last_change":"2021-03-15T22:36:15.261768+0000","last_active":"2021-03-15T22:36:45.367776+0000","last_peered":"2021-03-15T22:36:45.367776+0000","last_clean":"2021-03-15T22:36:45.367776+0000","last_became_active":"2021-03-15T22:34:55.341958+0000","last_became_peered":"2021-03-15T22:34:55.341958+0000","last_unstale":"2021-03-15T22:36:45.367776+0000","last_undegraded":"2021-03-15T22:36:45.367776+0000","last_fullsized":"2021-03-15T22:36:45.367776+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"556'47","last_scrub_stamp":"2021-03-15T22:36:15.261676+0000","last_deep_scrub":"115'33","last_deep_scrub_stamp":"2021-03-15T22:20:30.227196+0000","last_clean_scrub_stamp":"2021-03-15T22:36:15.261676+0000","log_size":47,"ondisk_log_size":47,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":734,"num_objects":12,"num_object_clones":0,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":62,"num_read_kb":44,"num_write":36,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":1468,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"0'0","reported_seq":"1080","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.830984+0000","last_change":"2021-03-15T22:36:24.385650+0000","last_active":"2021-03-15T22:36:45.830984+0000","last_peered":"2021-03-15T22:36:45.830984+0000","last_clean":"2021-03-15T22:36:45.830984+0000","last_became_active":"2021-03-15T22:36:24.383694+0000","last_became_peered":"2021-03-15T22:36:24.383694+0000","last_unstale":"2021-03-15T22:36:45.830984+0000","last_undegraded":"2021-03-15T22:36:45.830984+0000","last_fullsized":"2021-03-15T22:36:45.830984+0000","mapping_epoch":1028,"log_start":"0'0","ondisk_log_start":"0'0","created":1026,"last_epoch_clean":1029,"parent":"0.0","parent_split_bits":4,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:35:19.456875+0000","last_clean_scrub_stamp":"2021-03-15T22:35:19.456875+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.f","version":"1041'1742","reported_seq":"4966","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.370070+0000","last_change":"2021-03-15T22:36:35.389598+0000","last_active":"2021-03-15T22:36:45.370070+0000","last_peered":"2021-03-15T22:36:45.370070+0000","last_clean":"2021-03-15T22:36:45.370070+0000","last_became_active":"2021-03-15T22:29:02.355714+0000","last_became_peered":"2021-03-15T22:29:02.355714+0000","last_unstale":"2021-03-15T22:36:45.370070+0000","last_undegraded":"2021-03-15T22:36:45.370070+0000","last_fullsized":"2021-03-15T22:36:45.370070+0000","mapping_epoch":592,"log_start":"0'0","ondisk_log_start":"0'0","created":91,"last_epoch_clean":593,"parent":"0.0","parent_split_bits":0,"last_scrub":"1023'1699","last_scrub_stamp":"2021-03-15T22:36:20.059745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:20:04.009088+0000","last_clean_scrub_stamp":"2021-03-15T22:36:20.059745+0000","log_size":1742,"ondisk_log_size":1742,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":55707898,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":552,"num_read_kb":589646,"num_write":381,"num_write_kb":135091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":296,"num_bytes_recovered":90732573,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":"1064","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.368120+0000","last_change":"2021-03-15T22:36:25.253269+0000","last_active":"2021-03-15T22:36:45.368120+0000","last_peered":"2021-03-15T22:36:45.368120+0000","last_clean":"2021-03-15T22:36:45.368120+0000","last_became_active":"2021-03-15T22:34:55.340498+0000","last_became_peered":"2021-03-15T22:34:55.340498+0000","last_unstale":"2021-03-15T22:36:45.368120+0000","last_undegraded":"2021-03-15T22:36:45.368120+0000","last_fullsized":"2021-03-15T22:36:45.368120+0000","mapping_epoch":939,"log_start":"0'0","ondisk_log_start":"0'0","created":62,"last_epoch_clean":940,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:36:25.253188+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:36:25.253188+0000","last_clean_scrub_stamp":"2021-03-15T22:36:25.253188+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"976'19","reported_seq":"464","reported_epoch":"1049","state":"active+clean","last_fresh":"2021-03-15T22:36:46.623139+0000","last_change":"2021-03-15T22:36:09.696306+0000","last_active":"2021-03-15T22:36:46.623139+0000","last_peered":"2021-03-15T22:36:46.623139+0000","last_clean":"2021-03-15T22:36:46.623139+0000","last_became_active":"2021-03-15T22:35:28.364764+0000","last_became_peered":"2021-03-15T22:35:28.364764+0000","last_unstale":"2021-03-15T22:36:46.623139+0000","last_undegraded":"2021-03-15T22:36:46.623139+0000","last_fullsized":"2021-03-15T22:36:46.623139+0000","mapping_epoch":972,"log_start":"61'2","ondisk_log_start":"61'2","created":60,"last_epoch_clean":973,"parent":"0.0","parent_split_bits":0,"last_scrub":"976'19","last_scrub_stamp":"2021-03-15T22:36:09.696246+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:56.987106+0000","last_clean_scrub_stamp":"2021-03-15T22:36:09.696246+0000","log_size":17,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"84'2","reported_seq":"495","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367718+0000","last_change":"2021-03-15T22:36:14.388343+0000","last_active":"2021-03-15T22:36:45.367718+0000","last_peered":"2021-03-15T22:36:45.367718+0000","last_clean":"2021-03-15T22:36:45.367718+0000","last_became_active":"2021-03-15T22:36:14.387999+0000","last_became_peered":"2021-03-15T22:36:14.387999+0000","last_unstale":"2021-03-15T22:36:45.367718+0000","last_undegraded":"2021-03-15T22:36:45.367718+0000","last_fullsized":"2021-03-15T22:36:45.367718+0000","mapping_epoch":589,"log_start":"84'2","ondisk_log_start":"84'2","created":56,"last_epoch_clean":1019,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:33:21.690309+0000","last_clean_scrub_stamp":"2021-03-15T22:33:21.690309+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":358,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.c","version":"555'23","reported_seq":"1238","reported_epoch":"1048","state":"active+clean","last_fresh":"2021-03-15T22:36:45.367728+0000","last_change":"2021-03-15T22:35:46.406478+0000","last_active":"2021-03-15T22:36:45.367728+0000","last_peered":"2021-03-15T22:36:45.367728+0000","last_clean":"2021-03-15T22:36:45.367728+0000","last_became_active":"2021-03-15T22:33:04.337113+0000","last_became_peered":"2021-03-15T22:33:04.337113+0000","last_unstale":"2021-03-15T22:36:45.367728+0000","last_undegraded":"2021-03-15T22:36:45.367728+0000","last_fullsized":"2021-03-15T22:36:45.367728+0000","mapping_epoch":833,"log_start":"0'0","ondisk_log_start":"0'0","created":58,"last_epoch_clean":834,"parent":"0.0","parent_split_bits":0,"last_scrub":"555'23","last_scrub_stamp":"2021-03-15T22:35:46.406376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-15T22:17:54.969195+0000","last_clean_scrub_stamp":"2021-03-15T22:35:46.406376+0000","log_size":23,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":468,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":28,"num_write":23,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":48,"num_bytes_recovered":2808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":831822344,"num_objects":364,"num_object_clones":314,"num_object_copies":1092,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":364,"num_whiteouts":12,"num_read":7452,"num_read_kb":7660922,"num_write":5407,"num_write_kb":2092813,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4621,"num_bytes_recovered":2230024638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1829830656,"data_stored":2535132102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":24603,"ondisk_log_size":24603,"up":48,"acting":48,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1119,"num_read_kb":1114,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1167,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":33,"acting":33,"num_store_stats":8},{"poolid":3,"num_pg":22,"stat_sum":{"num_bytes":1323,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4750,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":66,"acting":66,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":23524,"num_objects":303,"num_object_clones":0,"num_object_copies":909,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":303,"num_whiteouts":0,"num_read":1482,"num_read_kb":987,"num_write":828,"num_write_kb":160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1253,"num_bytes_recovered":94766,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1966080,"data_stored":70572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},"log_size":947,"ondisk_log_size":947,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":135,"ondisk_log_size":135,"up":96,"acting":96,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":96,"acting":96,"num_store_stats":8}],"osd_stats":[{"osd":7,"up_from":52,"seq":223338299623,"num_pgs":64,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":289432,"kb_used_data":241872,"kb_used_omap":0,"kb_used_meta":47552,"kb_avail":93455720,"statfs":{"total":95995035648,"available":95698657280,"internally_reserved":0,"allocated":247676928,"data_stored":358966493,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48693248},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.53400000000000003,"5min":0.47199999999999998,"15min":0.46899999999999997},"min":{"1min":0.25,"5min":0.20499999999999999,"15min":0.20499999999999999},"max":{"1min":2.0390000000000001,"5min":2.0390000000000001,"15min":2.73},"last":0.61099999999999999},{"interface":"front","average":{"1min":0.56100000000000005,"5min":0.498,"15min":0.49399999999999999},"min":{"1min":0.28000000000000003,"5min":0.28000000000000003,"15min":0.222},"max":{"1min":1.9950000000000001,"5min":1.9950000000000001,"15min":3.8799999999999999},"last":0.72799999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.44800000000000001,"5min":0.44800000000000001,"15min":0.47099999999999997},"min":{"1min":0.34399999999999997,"5min":0.249,"15min":0.249},"max":{"1min":0.58799999999999997,"5min":0.73899999999999999,"15min":2.6859999999999999},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.502,"5min":0.56299999999999994,"15min":0.51500000000000001},"min":{"1min":0.311,"5min":0.28399999999999997,"15min":0.23400000000000001},"max":{"1min":0.67100000000000004,"5min":5.3819999999999997,"15min":5.3819999999999997},"last":0.79200000000000004}]},{"osd":2,"last update":"Mon Mar 15 22:36:38 2021","interfaces":[{"interface":"back","average":{"1min":0.46400000000000002,"5min":0.83799999999999997,"15min":0.59499999999999997},"min":{"1min":0.33000000000000002,"5min":0.28599999999999998,"15min":0.28599999999999998},"max":{"1min":0.65900000000000003,"5min":29.350999999999999,"15min":29.350999999999999},"last":0.76200000000000001},{"interface":"front","average":{"1min":0.47199999999999998,"5min":0.49199999999999999,"15min":0.498},"min":{"1min":0.33500000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.65900000000000003,"5min":1.298,"15min":1.298},"last":0.54900000000000004}]},{"osd":3,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.54800000000000004,"5min":0.51200000000000001,"15min":0.498},"min":{"1min":0.33500000000000002,"5min":0.314,"15min":0.28899999999999998},"max":{"1min":1.4810000000000001,"5min":1.4810000000000001,"15min":1.702},"last":0.58999999999999997},{"interface":"front","average":{"1min":0.53800000000000003,"5min":0.51200000000000001,"15min":0.52700000000000002},"min":{"1min":0.30099999999999999,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":1.806,"5min":1.806,"15min":5.415},"last":0.69499999999999995}]},{"osd":4,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.43099999999999999,"5min":0.436,"15min":0.439},"min":{"1min":0.27400000000000002,"5min":0.27400000000000002,"15min":0.23799999999999999},"max":{"1min":0.63500000000000001,"5min":0.753,"15min":1.927},"last":0.443},{"interface":"front","average":{"1min":0.502,"5min":0.47399999999999998,"15min":0.46200000000000002},"min":{"1min":0.32400000000000001,"5min":0.26400000000000001,"15min":0.253},"max":{"1min":0.68600000000000005,"5min":0.72699999999999998,"15min":2.5339999999999998},"last":0.46000000000000002}]},{"osd":5,"last update":"Mon Mar 15 22:36:38 2021","interfaces":[{"interface":"back","average":{"1min":0.52000000000000002,"5min":0.48999999999999999,"15min":0.46000000000000002},"min":{"1min":0.33400000000000002,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.74299999999999999,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.52300000000000002,"5min":0.51000000000000001,"15min":0.503},"min":{"1min":0.35999999999999999,"5min":0.29699999999999999,"15min":0.27900000000000003},"max":{"1min":0.68100000000000005,"5min":0.77700000000000002,"15min":1.071},"last":0.65100000000000002}]},{"osd":6,"last update":"Mon Mar 15 22:36:27 2021","interfaces":[{"interface":"back","average":{"1min":0.53600000000000003,"5min":0.52000000000000002,"15min":0.51400000000000001},"min":{"1min":0.35299999999999998,"5min":0.29999999999999999,"15min":0.27100000000000002},"max":{"1min":0.77800000000000002,"5min":0.78400000000000003,"15min":2.1179999999999999},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54200000000000004,"15min":0.51900000000000002},"min":{"1min":0.38200000000000001,"5min":0.33200000000000002,"15min":0.28399999999999997},"max":{"1min":0.76800000000000002,"5min":0.81999999999999995,"15min":1.8560000000000001},"last":0.51000000000000001}]}]},{"osd":6,"up_from":45,"seq":193273528555,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":245736,"kb_used_data":193312,"kb_used_omap":0,"kb_used_meta":52416,"kb_avail":93499416,"statfs":{"total":95995035648,"available":95743401984,"internally_reserved":0,"allocated":197951488,"data_stored":260385613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":53673984},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.49399999999999999,"15min":0.47999999999999998},"min":{"1min":0.33900000000000002,"5min":0.23100000000000001,"15min":0.23100000000000001},"max":{"1min":0.57099999999999995,"5min":2.5470000000000002,"15min":3.085},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.436,"5min":0.47299999999999998,"15min":0.46000000000000002},"min":{"1min":0.315,"5min":0.26400000000000001,"15min":0.26400000000000001},"max":{"1min":0.69699999999999995,"5min":1.137,"15min":1.323},"last":0.43099999999999999}]},{"osd":1,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.54200000000000004,"15min":0.54800000000000004},"min":{"1min":0.29399999999999998,"5min":0.26400000000000001,"15min":0.254},"max":{"1min":0.61099999999999999,"5min":2.7839999999999998,"15min":14.637},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.42099999999999999,"5min":0.44700000000000001,"15min":0.46899999999999997},"min":{"1min":0.28999999999999998,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.60999999999999999,"5min":0.84499999999999997,"15min":5.2450000000000001},"last":0.46999999999999997}]},{"osd":2,"last update":"Mon Mar 15 22:36:46 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.47899999999999998,"15min":0.46700000000000003},"min":{"1min":0.34899999999999998,"5min":0.25800000000000001,"15min":0.25800000000000001},"max":{"1min":0.72199999999999998,"5min":0.85899999999999999,"15min":0.85899999999999999},"last":0.60499999999999998},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.52400000000000002,"15min":0.50800000000000001},"min":{"1min":0.36699999999999999,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.66000000000000003,"5min":2.2210000000000001,"15min":2.2210000000000001},"last":0.54700000000000004}]},{"osd":3,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.55600000000000005,"5min":0.52600000000000002,"15min":0.51000000000000001},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.29899999999999999},"max":{"1min":2.1560000000000001,"5min":2.1560000000000001,"15min":2.6150000000000002},"last":0.67200000000000004},{"interface":"front","average":{"1min":0.45400000000000001,"5min":0.498,"15min":0.496},"min":{"1min":0.32100000000000001,"5min":0.32100000000000001,"15min":0.28699999999999998},"max":{"1min":0.65100000000000002,"5min":2.4670000000000001,"15min":2.7890000000000001},"last":0.53200000000000003}]},{"osd":4,"last update":"Mon Mar 15 22:36:15 2021","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.44900000000000001,"15min":0.437},"min":{"1min":0.26900000000000002,"5min":0.25,"15min":0.224},"max":{"1min":0.70499999999999996,"5min":0.94099999999999995,"15min":4.306},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.41599999999999998,"5min":0.47099999999999997,"15min":0.48499999999999999},"min":{"1min":0.28999999999999998,"5min":0.26600000000000001,"15min":0.23499999999999999},"max":{"1min":0.65300000000000002,"5min":1.091,"15min":4.181},"last":0.69399999999999995}]},{"osd":5,"last update":"Mon Mar 15 22:36:41 2021","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.49099999999999999,"15min":0.435},"min":{"1min":0.39600000000000002,"5min":0.30599999999999999,"15min":0.26700000000000002},"max":{"1min":0.73899999999999999,"5min":0.89300000000000002,"15min":0.89300000000000002},"last":0.56699999999999995},{"interface":"front","average":{"1min":0.58299999999999996,"5min":0.54500000000000004,"15min":0.47999999999999998},"min":{"1min":0.42199999999999999,"5min":0.317,"15min":0.30199999999999999},"max":{"1min":0.86599999999999999,"5min":1.022,"15min":1.022},"last":0.58299999999999996}]},{"osd":7,"last update":"Mon Mar 15 22:36:25 2021","interfaces":[{"interface":"back","average":{"1min":0.45900000000000002,"5min":0.47599999999999998,"15min":0.52600000000000002},"min":{"1min":0.28199999999999997,"5min":0.28199999999999997,"15min":0.249},"max":{"1min":0.63600000000000001,"5min":0.76300000000000001,"15min":15.340999999999999},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.56999999999999995,"15min":0.58599999999999997},"min":{"1min":0.38600000000000001,"5min":0.35399999999999998,"15min":0.27200000000000002},"max":{"1min":0.78200000000000003,"5min":1.0109999999999999,"15min":15.356},"last":0.64500000000000002}]}]},{"osd":1,"up_from":13,"seq":55834575100,"num_pgs":49,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":185652,"kb_used_data":128556,"kb_used_omap":0,"kb_used_meta":57088,"kb_avail":93559500,"statfs":{"total":95995035648,"available":95804928000,"internally_reserved":0,"allocated":131641344,"data_stored":174251752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":58458112},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:35:56 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.372,"15min":0.34000000000000002},"min":{"1min":0.22800000000000001,"5min":0.20200000000000001,"15min":0.17599999999999999},"max":{"1min":0.50800000000000001,"5min":4.3920000000000003,"15min":4.3920000000000003},"last":0.312},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.42499999999999999,"15min":0.37},"min":{"1min":0.215,"5min":0.19500000000000001,"15min":0.19500000000000001},"max":{"1min":0.52500000000000002,"5min":4.6029999999999998,"15min":4.6029999999999998},"last":0.249}]},{"osd":2,"last update":"Mon Mar 15 22:35:45 2021","interfaces":[{"interface":"back","average":{"1min":0.35099999999999998,"5min":0.46899999999999997,"15min":0.379},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.59199999999999997,"5min":4.5490000000000004,"15min":4.5490000000000004},"last":0.28999999999999998},{"interface":"front","average":{"1min":0.38900000000000001,"5min":0.45500000000000002,"15min":0.40100000000000002},"min":{"1min":0.25,"5min":0.25,"15min":0.25},"max":{"1min":0.64600000000000002,"5min":5.0190000000000001,"15min":5.0190000000000001},"last":0.34100000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:36:21 2021","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.47999999999999998,"15min":0.40100000000000002},"min":{"1min":0.27100000000000002,"5min":0.27100000000000002,"15min":0.215},"max":{"1min":0.76500000000000001,"5min":7.718,"15min":7.718},"last":0.26600000000000001},{"interface":"front","average":{"1min":0.35699999999999998,"5min":0.51800000000000002,"15min":0.432},"min":{"1min":0.26600000000000001,"5min":0.245,"15min":0.21199999999999999},"max":{"1min":0.54400000000000004,"5min":8.4760000000000009,"15min":8.4760000000000009},"last":0.27700000000000002}]},{"osd":4,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.61199999999999999,"15min":0.55100000000000005},"min":{"1min":0.35499999999999998,"5min":0.31900000000000001,"15min":0.22800000000000001},"max":{"1min":0.78800000000000003,"5min":4.593,"15min":4.593},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.58599999999999997,"15min":0.57699999999999996},"min":{"1min":0.38,"5min":0.35499999999999998,"15min":0.253},"max":{"1min":0.67100000000000004,"5min":4.3140000000000001,"15min":4.6139999999999999},"last":0.50600000000000001}]},{"osd":5,"last update":"Mon Mar 15 22:36:37 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.58699999999999997,"15min":0.54800000000000004},"min":{"1min":0.38200000000000001,"5min":0.34499999999999997,"15min":0.34300000000000003},"max":{"1min":0.79800000000000004,"5min":4.3630000000000004,"15min":4.3630000000000004},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.53700000000000003,"5min":0.60399999999999998,"15min":0.53200000000000003},"min":{"1min":0.36499999999999999,"5min":0.33600000000000002,"15min":0.32100000000000001},"max":{"1min":0.81000000000000005,"5min":3.472,"15min":3.472},"last":0.53300000000000003}]},{"osd":6,"last update":"Mon Mar 15 22:35:59 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.60799999999999998,"15min":0.58199999999999996},"min":{"1min":0.42799999999999999,"5min":0.38500000000000001,"15min":0.314},"max":{"1min":0.755,"5min":3.4209999999999998,"15min":3.4209999999999998},"last":0.621},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.63100000000000001,"15min":0.57899999999999996},"min":{"1min":0.41199999999999998,"5min":0.375,"15min":0.33700000000000002},"max":{"1min":0.68300000000000005,"5min":4.6529999999999996,"15min":4.6529999999999996},"last":0.64800000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:36:21 2021","interfaces":[{"interface":"back","average":{"1min":0.56499999999999995,"5min":0.63600000000000001,"15min":0.59999999999999998},"min":{"1min":0.42399999999999999,"5min":0.39800000000000002,"15min":0.32100000000000001},"max":{"1min":0.71899999999999997,"5min":3.5070000000000001,"15min":3.5070000000000001},"last":0.46400000000000002},{"interface":"front","average":{"1min":0.55900000000000005,"5min":0.66500000000000004,"15min":0.59399999999999997},"min":{"1min":0.45400000000000001,"5min":0.40300000000000002,"15min":0.35799999999999998},"max":{"1min":0.73099999999999998,"5min":4.625,"15min":4.625},"last":0.57799999999999996}]}]},{"osd":0,"up_from":8,"seq":34359738623,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":325056,"kb_used_data":275896,"kb_used_omap":0,"kb_used_meta":49152,"kb_avail":93420096,"statfs":{"total":95995035648,"available":95662178304,"internally_reserved":0,"allocated":282517504,"data_stored":400956028,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50331648},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Mon Mar 15 22:36:47 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.35899999999999999,"15min":0.34300000000000003},"min":{"1min":0.22800000000000001,"5min":0.192,"15min":0.17999999999999999},"max":{"1min":0.50800000000000001,"5min":1.46,"15min":1.7070000000000001},"last":0.30099999999999999},{"interface":"front","average":{"1min":0.32700000000000001,"5min":0.39400000000000002,"15min":0.37},"min":{"1min":0.22500000000000001,"5min":0.20000000000000001,"15min":0.19900000000000001},"max":{"1min":0.58499999999999996,"5min":2.4830000000000001,"15min":2.4830000000000001},"last":0.31900000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:36:41 2021","interfaces":[{"interface":"back","average":{"1min":0.34899999999999998,"5min":0.371,"15min":0.36299999999999999},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23000000000000001},"max":{"1min":0.53600000000000003,"5min":1.4990000000000001,"15min":1.4990000000000001},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.34200000000000003,"5min":0.48499999999999999,"15min":0.41199999999999998},"min":{"1min":0.247,"5min":0.20999999999999999,"15min":0.20999999999999999},"max":{"1min":0.55100000000000005,"5min":7.3799999999999999,"15min":7.3799999999999999},"last":0.45900000000000002}]},{"osd":3,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.38500000000000001,"5min":0.40000000000000002,"15min":0.38800000000000001},"min":{"1min":0.25600000000000001,"5min":0.25600000000000001,"15min":0.22700000000000001},"max":{"1min":0.71399999999999997,"5min":1.544,"15min":1.544},"last":0.33200000000000002},{"interface":"front","average":{"1min":0.36299999999999999,"5min":0.46700000000000003,"15min":0.41999999999999998},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.19400000000000001},"max":{"1min":0.60399999999999998,"5min":3.988,"15min":3.988},"last":0.35999999999999999}]},{"osd":4,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.55200000000000005,"15min":0.53700000000000003},"min":{"1min":0.34799999999999998,"5min":0.27500000000000002,"15min":0.254},"max":{"1min":0.76100000000000001,"5min":1.861,"15min":1.861},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.59199999999999997,"15min":0.56399999999999995},"min":{"1min":0.36799999999999999,"5min":0.33000000000000002,"15min":0.28799999999999998},"max":{"1min":0.78300000000000003,"5min":1.5700000000000001,"15min":1.645},"last":0.498}]},{"osd":5,"last update":"Mon Mar 15 22:36:36 2021","interfaces":[{"interface":"back","average":{"1min":0.58799999999999997,"5min":0.61699999999999999,"15min":0.54100000000000004},"min":{"1min":0.38400000000000001,"5min":0.33900000000000002,"15min":0.29899999999999999},"max":{"1min":1.6459999999999999,"5min":3.7879999999999998,"15min":3.7879999999999998},"last":0.433},{"interface":"front","average":{"1min":0.54200000000000004,"5min":0.56200000000000006,"15min":0.50600000000000001},"min":{"1min":0.36399999999999999,"5min":0.35899999999999999,"15min":0.318},"max":{"1min":0.81000000000000005,"5min":1.909,"15min":1.909},"last":0.51100000000000001}]},{"osd":6,"last update":"Mon Mar 15 22:36:01 2021","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.57399999999999995,"15min":0.55600000000000005},"min":{"1min":0.34699999999999998,"5min":0.34699999999999998,"15min":0.26200000000000001},"max":{"1min":0.78800000000000003,"5min":2.7109999999999999,"15min":2.7109999999999999},"last":0.53400000000000003},{"interface":"front","average":{"1min":0.56000000000000005,"5min":0.59299999999999997,"15min":0.56899999999999995},"min":{"1min":0.42699999999999999,"5min":0.40500000000000003,"15min":0.27100000000000002},"max":{"1min":0.77300000000000002,"5min":1.5880000000000001,"15min":1.5880000000000001},"last":0.52600000000000002}]},{"osd":7,"last update":"Mon Mar 15 22:36:28 2021","interfaces":[{"interface":"back","average":{"1min":0.55500000000000005,"5min":0.65000000000000002,"15min":0.58499999999999996},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.27900000000000003},"max":{"1min":0.82499999999999996,"5min":3.8239999999999998,"15min":3.8239999999999998},"last":0.56299999999999994},{"interface":"front","average":{"1min":0.54600000000000004,"5min":0.60899999999999999,"15min":0.57699999999999996},"min":{"1min":0.41999999999999998,"5min":0.41999999999999998,"15min":0.317},"max":{"1min":0.72499999999999998,"5min":2.665,"15min":2.665},"last":0.55900000000000005}]}]},{"osd":2,"up_from":681,"seq":2924872728653,"num_pgs":50,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":362404,"kb_used_data":324188,"kb_used_omap":119,"kb_used_meta":38088,"kb_avail":93382748,"statfs":{"total":95995035648,"available":95623933952,"internally_reserved":0,"allocated":331968512,"data_stored":444173557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":122519,"internal_metadata":39002473},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:46 2021","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.32100000000000001,"15min":0.311},"min":{"1min":0.24199999999999999,"5min":0.183,"15min":0.183},"max":{"1min":0.45700000000000002,"5min":0.71799999999999997,"15min":0.71799999999999997},"last":0.32700000000000001},{"interface":"front","average":{"1min":0.42499999999999999,"5min":0.374,"15min":0.33800000000000002},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.20000000000000001},"max":{"1min":1.4099999999999999,"5min":1.4099999999999999,"15min":1.4099999999999999},"last":0.25800000000000001}]},{"osd":1,"last update":"Mon Mar 15 22:36:46 2021","interfaces":[{"interface":"back","average":{"1min":0.33900000000000002,"5min":0.35899999999999999,"15min":0.33500000000000002},"min":{"1min":0.26800000000000002,"5min":0.214,"15min":0.214},"max":{"1min":0.46700000000000003,"5min":1.0169999999999999,"15min":1.0169999999999999},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.433,"5min":0.39400000000000002,"15min":0.36799999999999999},"min":{"1min":0.24199999999999999,"5min":0.221,"15min":0.221},"max":{"1min":0.92900000000000005,"5min":1.3069999999999999,"15min":1.3069999999999999},"last":0.34399999999999997}]},{"osd":3,"last update":"Mon Mar 15 22:36:46 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.44,"15min":0.371},"min":{"1min":0.28999999999999998,"5min":0.224,"15min":0.216},"max":{"1min":0.96599999999999997,"5min":3.496,"15min":3.496},"last":0.308},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.38400000000000001,"15min":0.36199999999999999},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.46800000000000003,"5min":1.3440000000000001,"15min":1.3440000000000001},"last":0.41699999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:36:46 2021","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.55800000000000005,"15min":0.498},"min":{"1min":0.33200000000000002,"5min":0.30399999999999999,"15min":0.30399999999999999},"max":{"1min":0.70299999999999996,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.58799999999999997,"5min":0.56000000000000005,"15min":0.50600000000000001},"min":{"1min":0.35499999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":1.0069999999999999,"5min":1.4239999999999999,"15min":1.4239999999999999},"last":0.67200000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:36:46 2021","interfaces":[{"interface":"back","average":{"1min":0.56599999999999995,"5min":0.53600000000000003,"15min":0.50900000000000001},"min":{"1min":0.41999999999999998,"5min":0.32400000000000001,"15min":0.32400000000000001},"max":{"1min":0.71899999999999997,"5min":1.0009999999999999,"15min":1.0009999999999999},"last":0.64700000000000002},{"interface":"front","average":{"1min":0.55000000000000004,"5min":0.57099999999999995,"15min":0.52200000000000002},"min":{"1min":0.42899999999999999,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.73399999999999999,"5min":1.2849999999999999,"15min":1.2849999999999999},"last":0.57699999999999996}]},{"osd":6,"last update":"Mon Mar 15 22:36:46 2021","interfaces":[{"interface":"back","average":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.52600000000000002},"min":{"1min":0.47299999999999998,"5min":0.372,"15min":0.33500000000000002},"max":{"1min":0.80200000000000005,"5min":1.3320000000000001,"15min":1.3320000000000001},"last":0.748},{"interface":"front","average":{"1min":0.629,"5min":0.60299999999999998,"15min":0.53300000000000003},"min":{"1min":0.47799999999999998,"5min":0.34499999999999997,"15min":0.34499999999999997},"max":{"1min":0.98999999999999999,"5min":1.387,"15min":1.387},"last":0.73399999999999999}]},{"osd":7,"last update":"Mon Mar 15 22:36:46 2021","interfaces":[{"interface":"back","average":{"1min":0.59099999999999997,"5min":0.60799999999999998,"15min":0.54800000000000004},"min":{"1min":0.41999999999999998,"5min":0.374,"15min":0.374},"max":{"1min":0.77400000000000002,"5min":1.9239999999999999,"15min":1.9239999999999999},"last":0.69299999999999995},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.60099999999999998,"15min":0.54100000000000004},"min":{"1min":0.44900000000000001,"5min":0.39300000000000002,"15min":0.39300000000000002},"max":{"1min":1.022,"5min":1.4590000000000001,"15min":1.4590000000000001},"last":0.56499999999999995}]}]},{"osd":3,"up_from":26,"seq":111669149940,"num_pgs":55,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333160,"kb_used_data":266464,"kb_used_omap":0,"kb_used_meta":66688,"kb_avail":93411992,"statfs":{"total":95995035648,"available":95653879808,"internally_reserved":0,"allocated":272859136,"data_stored":368178842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":68288512},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:24 2021","interfaces":[{"interface":"back","average":{"1min":0.312,"5min":0.34499999999999997,"15min":0.33900000000000002},"min":{"1min":0.22900000000000001,"5min":0.183,"15min":0.17699999999999999},"max":{"1min":0.39500000000000002,"5min":2.7509999999999999,"15min":2.7509999999999999},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.28899999999999998,"5min":0.32600000000000001,"15min":0.33600000000000002},"min":{"1min":0.22600000000000001,"5min":0.20000000000000001,"15min":0.19800000000000001},"max":{"1min":0.41999999999999998,"5min":0.76900000000000002,"15min":2.6339999999999999},"last":0.67900000000000005}]},{"osd":1,"last update":"Mon Mar 15 22:36:24 2021","interfaces":[{"interface":"back","average":{"1min":0.35899999999999999,"5min":0.36899999999999999,"15min":0.39200000000000002},"min":{"1min":0.21199999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.55900000000000005,"5min":0.626,"15min":6.0199999999999996},"last":0.59599999999999997},{"interface":"front","average":{"1min":0.35799999999999998,"5min":0.37,"15min":0.39900000000000002},"min":{"1min":0.25900000000000001,"5min":0.219,"15min":0.20699999999999999},"max":{"1min":0.48599999999999999,"5min":2.5619999999999998,"15min":7.6130000000000004},"last":0.63200000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:36:42 2021","interfaces":[{"interface":"back","average":{"1min":0.34699999999999998,"5min":0.35399999999999998,"15min":0.35499999999999998},"min":{"1min":0.24099999999999999,"5min":0.23699999999999999,"15min":0.216},"max":{"1min":0.58099999999999996,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.372,"5min":0.38700000000000001,"15min":0.38800000000000001},"min":{"1min":0.28999999999999998,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.70899999999999996,"5min":0.79000000000000004,"15min":0.79000000000000004},"last":0.70899999999999996}]},{"osd":4,"last update":"Mon Mar 15 22:36:39 2021","interfaces":[{"interface":"back","average":{"1min":0.51000000000000001,"5min":0.51900000000000002,"15min":0.53100000000000003},"min":{"1min":0.33000000000000002,"5min":0.31900000000000001,"15min":0.22600000000000001},"max":{"1min":0.66900000000000004,"5min":0.86899999999999999,"15min":3.3370000000000002},"last":0.999},{"interface":"front","average":{"1min":0.51600000000000001,"5min":0.54000000000000004,"15min":0.52400000000000002},"min":{"1min":0.36599999999999999,"5min":0.311,"15min":0.26700000000000002},"max":{"1min":0.80900000000000005,"5min":0.84499999999999997,"15min":2.2890000000000001},"last":0.79300000000000004}]},{"osd":5,"last update":"Mon Mar 15 22:36:39 2021","interfaces":[{"interface":"back","average":{"1min":0.54000000000000004,"5min":0.53500000000000003,"15min":0.73299999999999998},"min":{"1min":0.39100000000000001,"5min":0.311,"15min":0.28299999999999997},"max":{"1min":0.80200000000000005,"5min":0.80200000000000005,"15min":9.4870000000000001},"last":0.88500000000000001},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.54100000000000004,"15min":0.91500000000000004},"min":{"1min":0.33800000000000002,"5min":0.33800000000000002,"15min":0.32800000000000001},"max":{"1min":0.64000000000000001,"5min":0.84799999999999998,"15min":15.776},"last":1.0269999999999999}]},{"osd":6,"last update":"Mon Mar 15 22:36:06 2021","interfaces":[{"interface":"back","average":{"1min":0.53000000000000003,"5min":0.55900000000000005,"15min":0.55900000000000005},"min":{"1min":0.38900000000000001,"5min":0.38200000000000001,"15min":0.30099999999999999},"max":{"1min":0.81699999999999995,"5min":1.359,"15min":2.6499999999999999},"last":0.95799999999999996},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.55300000000000005,"15min":0.55900000000000005},"min":{"1min":0.38900000000000001,"5min":0.375,"15min":0.30399999999999999},"max":{"1min":0.78500000000000003,"5min":1.3400000000000001,"15min":3.472},"last":0.92800000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:36:24 2021","interfaces":[{"interface":"back","average":{"1min":0.56399999999999995,"5min":0.55100000000000005,"15min":0.57499999999999996},"min":{"1min":0.41699999999999998,"5min":0.34799999999999998,"15min":0.29599999999999999},"max":{"1min":0.752,"5min":0.91200000000000003,"15min":5.625},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.53300000000000003,"5min":0.57599999999999996,"15min":0.56200000000000006},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.73199999999999998,"5min":2.621,"15min":2.621},"last":0.89600000000000002}]}]},{"osd":4,"up_from":31,"seq":133143986417,"num_pgs":60,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":155840,"kb_used_data":105208,"kb_used_omap":0,"kb_used_meta":50624,"kb_avail":93589312,"statfs":{"total":95995035648,"available":95835455488,"internally_reserved":0,"allocated":107732992,"data_stored":153871734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":51838976},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:32 2021","interfaces":[{"interface":"back","average":{"1min":0.50900000000000001,"5min":0.46500000000000002,"15min":0.48199999999999998},"min":{"1min":0.32600000000000001,"5min":0.221,"15min":0.20000000000000001},"max":{"1min":0.748,"5min":0.748,"15min":4.7050000000000001},"last":0.65900000000000003},{"interface":"front","average":{"1min":0.502,"5min":0.47099999999999997,"15min":0.499},"min":{"1min":0.28499999999999998,"5min":0.27100000000000002,"15min":0.246},"max":{"1min":0.68799999999999994,"5min":1.4179999999999999,"15min":5.0209999999999999},"last":0.60199999999999998}]},{"osd":1,"last update":"Mon Mar 15 22:36:32 2021","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.46899999999999997,"15min":0.47899999999999998},"min":{"1min":0.34699999999999998,"5min":0.25,"15min":0.23999999999999999},"max":{"1min":0.77900000000000003,"5min":0.77900000000000003,"15min":4.3499999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.496,"5min":0.48499999999999999,"15min":0.495},"min":{"1min":0.34599999999999997,"5min":0.28399999999999997,"15min":0.25700000000000001},"max":{"1min":0.72899999999999998,"5min":0.94599999999999995,"15min":4.718},"last":0.51500000000000001}]},{"osd":2,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.53100000000000003,"5min":0.48399999999999999,"15min":0.503},"min":{"1min":0.36699999999999999,"5min":0.27000000000000002,"15min":0.27000000000000002},"max":{"1min":0.78700000000000003,"5min":0.78700000000000003,"15min":0.78700000000000003},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.52400000000000002,"5min":0.49099999999999999,"15min":0.497},"min":{"1min":0.34100000000000003,"5min":0.309,"15min":0.309},"max":{"1min":0.748,"5min":0.94899999999999995,"15min":0.94899999999999995},"last":0.48099999999999998}]},{"osd":3,"last update":"Mon Mar 15 22:36:32 2021","interfaces":[{"interface":"back","average":{"1min":0.51500000000000001,"5min":0.52800000000000002,"15min":0.54600000000000004},"min":{"1min":0.32000000000000001,"5min":0.32000000000000001,"15min":0.28499999999999998},"max":{"1min":0.69399999999999995,"5min":1.486,"15min":5.0499999999999998},"last":0.67500000000000004},{"interface":"front","average":{"1min":0.55200000000000005,"5min":0.55500000000000005,"15min":0.54600000000000004},"min":{"1min":0.40400000000000003,"5min":0.32900000000000001,"15min":0.32900000000000001},"max":{"1min":0.71099999999999997,"5min":1.4950000000000001,"15min":4.4050000000000002},"last":0.71499999999999997}]},{"osd":5,"last update":"Mon Mar 15 22:36:32 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.41199999999999998,"15min":0.40999999999999998},"min":{"1min":0.26500000000000001,"5min":0.22900000000000001,"15min":0.22900000000000001},"max":{"1min":0.67000000000000004,"5min":0.75900000000000001,"15min":0.75900000000000001},"last":0.628},{"interface":"front","average":{"1min":0.503,"5min":0.47299999999999998,"15min":0.442},"min":{"1min":0.308,"5min":0.29099999999999998,"15min":0.24099999999999999},"max":{"1min":0.73599999999999999,"5min":0.73599999999999999,"15min":0.77000000000000002},"last":0.55300000000000005}]},{"osd":6,"last update":"Mon Mar 15 22:35:59 2021","interfaces":[{"interface":"back","average":{"1min":0.435,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.27500000000000002,"5min":0.247,"15min":0.22600000000000001},"max":{"1min":0.80300000000000005,"5min":0.80300000000000005,"15min":4.3079999999999998},"last":0.56599999999999995},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.53800000000000003,"15min":0.53500000000000003},"min":{"1min":0.36199999999999999,"5min":0.33000000000000002,"15min":0.28000000000000003},"max":{"1min":0.78600000000000003,"5min":0.78600000000000003,"15min":4.6600000000000001},"last":0.68600000000000005}]},{"osd":7,"last update":"Mon Mar 15 22:36:21 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.503,"15min":0.505},"min":{"1min":0.34799999999999998,"5min":0.28299999999999997,"15min":0.26300000000000001},"max":{"1min":0.71699999999999997,"5min":0.79900000000000004,"15min":4.6929999999999996},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.52100000000000002,"15min":0.54200000000000004},"min":{"1min":0.40899999999999997,"5min":0.29999999999999999,"15min":0.25},"max":{"1min":0.66200000000000003,"5min":1.034,"15min":4.9900000000000002},"last":0.64700000000000002}]}]},{"osd":5,"up_from":616,"seq":2645699854426,"num_pgs":54,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":301028,"kb_used_data":269596,"kb_used_omap":153,"kb_used_meta":31270,"kb_avail":93444124,"statfs":{"total":95995035648,"available":95686782976,"internally_reserved":0,"allocated":276066304,"data_stored":385849525,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":157167,"internal_metadata":32021009},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.63400000000000001,"5min":0.51100000000000001,"15min":0.47499999999999998},"min":{"1min":0.307,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":2.5600000000000001,"5min":2.5600000000000001,"15min":2.5600000000000001},"last":0.437},{"interface":"front","average":{"1min":0.62,"5min":0.503,"15min":0.46700000000000003},"min":{"1min":0.377,"5min":0.26200000000000001,"15min":0.26200000000000001},"max":{"1min":2.5939999999999999,"5min":2.5939999999999999,"15min":2.5939999999999999},"last":0.55800000000000005}]},{"osd":1,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.61899999999999999,"5min":0.58099999999999996,"15min":0.49399999999999999},"min":{"1min":0.28899999999999998,"5min":0.28899999999999998,"15min":0.27500000000000002},"max":{"1min":2.6699999999999999,"5min":6.4359999999999999,"15min":6.4359999999999999},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.51500000000000001,"5min":0.48599999999999999,"15min":0.48699999999999999},"min":{"1min":0.35899999999999999,"5min":0.28999999999999998,"15min":0.28999999999999998},"max":{"1min":0.68500000000000005,"5min":1.161,"15min":1.161},"last":0.496}]},{"osd":2,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.53300000000000003,"5min":0.58699999999999997,"15min":0.53700000000000003},"min":{"1min":0.33600000000000002,"5min":0.28199999999999997,"15min":0.28199999999999997},"max":{"1min":0.71899999999999997,"5min":10.952,"15min":10.952},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.52900000000000003,"5min":0.50700000000000001,"15min":0.52400000000000002},"min":{"1min":0.39800000000000002,"5min":0.32100000000000001,"15min":0.32100000000000001},"max":{"1min":0.71399999999999997,"5min":1.125,"15min":1.125},"last":0.53000000000000003}]},{"osd":3,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.54700000000000004,"5min":0.68600000000000005,"15min":0.54400000000000004},"min":{"1min":0.308,"5min":0.28399999999999997,"15min":0.28399999999999997},"max":{"1min":0.76400000000000001,"5min":10.355,"15min":10.355},"last":0.51400000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.51200000000000001,"15min":0.46400000000000002},"min":{"1min":0.32800000000000001,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.77100000000000002,"5min":2.476,"15min":2.476},"last":0.41799999999999998}]},{"osd":4,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.505,"15min":0.41899999999999998},"min":{"1min":0.29099999999999998,"5min":0.254,"15min":0.254},"max":{"1min":0.68500000000000005,"5min":7.4000000000000004,"15min":7.4000000000000004},"last":0.59699999999999998},{"interface":"front","average":{"1min":0.56200000000000006,"5min":0.49199999999999999,"15min":0.41999999999999998},"min":{"1min":0.247,"5min":0.247,"15min":0.247},"max":{"1min":2.5739999999999998,"5min":2.5739999999999998,"15min":2.5739999999999998},"last":0.54100000000000004}]},{"osd":6,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.56799999999999995,"5min":0.52400000000000002,"15min":0.47299999999999998},"min":{"1min":0.25700000000000001,"5min":0.25700000000000001,"15min":0.23999999999999999},"max":{"1min":2.613,"5min":3.9060000000000001,"15min":3.9060000000000001},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.63900000000000001,"5min":0.58299999999999996,"15min":0.504},"min":{"1min":0.34499999999999997,"5min":0.34499999999999997,"15min":0.29699999999999999},"max":{"1min":2.698,"5min":4.0190000000000001,"15min":4.0190000000000001},"last":0.628}]},{"osd":7,"last update":"Mon Mar 15 22:36:43 2021","interfaces":[{"interface":"back","average":{"1min":0.60099999999999998,"5min":0.54600000000000004,"15min":0.47499999999999998},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29299999999999998},"max":{"1min":2.6469999999999998,"5min":2.6469999999999998,"15min":2.6469999999999998},"last":0.58099999999999996},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.58499999999999996,"15min":0.51900000000000002},"min":{"1min":0.32200000000000001,"5min":0.29999999999999999,"15min":0.29999999999999999},"max":{"1min":0.75600000000000001,"5min":5.944,"15min":5.944},"last":0.57399999999999995}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":984,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":9410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":335872,"data_stored":12026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":151552,"data_stored":5414,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":274432,"data_stored":9910,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":155648,"data_stored":5528,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3137,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":10320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":229376,"data_stored":8144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":280162304,"data_stored":399514695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":129228800,"data_stored":172809126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":329744384,"data_stored":442737932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":270520320,"data_stored":366746430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":105381888,"data_stored":152438267,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":273838080,"data_stored":384413786,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":195596288,"data_stored":258945082,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":245358592,"data_stored":357526784,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2021-03-15T22:36:53.057 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool set default.rgw.meta pgp_num 32 2021-03-15T22:36:53.098 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:53.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:53.126 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:53.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:53.228 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:53.254 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:53.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:53.257 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:53.359 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:53.384 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:53.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:53.386 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:53.487 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:53.512 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:53.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:53.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:53.615 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:53.640 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:53.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:53.642 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:53.743 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:53.768 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:53.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:53.770 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:53.871 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:53.896 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:53.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:53.897 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:53.944 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:53 smithi076 conmon[56514]: cluster 2021-03-15T22:36:52.122434+0000 osd.3 (osd. 2021-03-15T22:36:53.944 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:53 smithi076 conmon[56514]: 3) 183 : cluster [DBG] 6.c scrub ok 2021-03-15T22:36:53.945 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:53 smithi076 conmon[56514]: cluster 2021-03-15T22:36:52.300208+0000 osd.5 (osd.5) 71 : cluster [DBG] 3.c scrub ok 2021-03-15T22:36:53.945 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:53 smithi076 conmon[62156]: cluster 2021-03-15T22:36:52.122434+0000 osd.3 2021-03-15T22:36:53.945 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:53 smithi076 conmon[62156]: (osd.3) 183 : cluster [DBG] 6.c scrub ok 2021-03-15T22:36:53.945 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:53 smithi076 conmon[62156]: cluster 2021-03-15T22:36:52.300208+0000 2021-03-15T22:36:53.946 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:53 smithi076 conmon[62156]: osd.5 (osd.5) 71 : cluster [DBG] 3.c scrub ok 2021-03-15T22:36:53.998 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:54.023 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:54.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:54.024 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:54.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:53 smithi125 conmon[56739]: cluster 2021-03-15T22:36:52.122434+0000 osd.3 (osd.3) 2021-03-15T22:36:54.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:53 smithi125 conmon[56739]: 183 : cluster [DBG] 6.c scrub ok 2021-03-15T22:36:54.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:53 smithi125 conmon[56739]: cluster 2021-03-15T22:36:52.300208+0000 osd.5 (osd.5) 71 : cluster [DBG] 3.c scrub ok 2021-03-15T22:36:54.125 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:54.148 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:54.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:54.149 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:54.251 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:54.275 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:54.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:54.276 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:54.377 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:54.401 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:54.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:54.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:54.503 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:54.526 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:54.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:54.527 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:54.628 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:54.655 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:54.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:54.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:54.758 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:54.784 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:54.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:54.785 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:54.796 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:54.886 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:54.911 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:54.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:54.913 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:54.941 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:54 smithi076 conmon[62156]: cluster 2021-03-15T22:36:52.991319+0000 osd.0 (osd. 2021-03-15T22:36:54.941 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:54 smithi076 conmon[62156]: 0) 171 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:36:54.942 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:54 smithi076 conmon[62156]: cluster 2021-03-15T22:36:53.241865+0000 mgr.y (mgr.14144) 1144 : cluster [DBG] pgmap v1725: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:36:54.942 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:54 smithi076 conmon[56514]: cluster 2021-03-15T22:36:52.991319+0000 osd.0 (osd.0) 171 : cluster [DBG] 2021-03-15T22:36:54.942 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:54 smithi076 conmon[56514]: 3.6 scrub ok 2021-03-15T22:36:54.942 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:54 smithi076 conmon[56514]: cluster 2021-03-15T22:36:53.241865+0000 mgr.y (mgr.14144) 1144 : cluster [DBG] pgmap v1725: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:36:55.014 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:55.036 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:54 smithi125 conmon[56739]: cluster 2021-03-15T22:36:52.991319+0000 osd.0 (osd.0) 171 : cluster [DBG] 3.6 scrub ok 2021-03-15T22:36:55.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:54 smithi125 conmon[56739]: cluster 2021-03-15T22:36:53.241865 2021-03-15T22:36:55.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:54 smithi125 conmon[56739]: +0000 mgr.y (mgr.14144) 1144 : cluster [DBG] pgmap v1725: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 639 B/s rd, 0 op/s 2021-03-15T22:36:55.048 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:55.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:55.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:55.150 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:55.176 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:55.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:55.177 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:55.226 INFO:teuthology.orchestra.run.smithi076.stdout:osd.0: {} 2021-03-15T22:36:55.243 INFO:teuthology.orchestra.run.smithi076.stdout:osd.1: {} 2021-03-15T22:36:55.260 INFO:teuthology.orchestra.run.smithi076.stdout:osd.2: {} 2021-03-15T22:36:55.278 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:55.279 INFO:teuthology.orchestra.run.smithi076.stdout:osd.3: {} 2021-03-15T22:36:55.298 INFO:teuthology.orchestra.run.smithi076.stdout:osd.4: {} 2021-03-15T22:36:55.303 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:55.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:55.304 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:55.318 INFO:teuthology.orchestra.run.smithi076.stdout:osd.5: {} 2021-03-15T22:36:55.336 INFO:teuthology.orchestra.run.smithi076.stdout:osd.6: {} 2021-03-15T22:36:55.356 INFO:teuthology.orchestra.run.smithi076.stdout:osd.7: {} 2021-03-15T22:36:55.358 INFO:teuthology.orchestra.run.smithi076.stderr:osd.0: osd_enable_op_tracker = 'false' 2021-03-15T22:36:55.358 INFO:teuthology.orchestra.run.smithi076.stderr:osd.1: osd_enable_op_tracker = 'false' 2021-03-15T22:36:55.358 INFO:teuthology.orchestra.run.smithi076.stderr:osd.2: osd_enable_op_tracker = 'false' 2021-03-15T22:36:55.359 INFO:teuthology.orchestra.run.smithi076.stderr:osd.3: osd_enable_op_tracker = 'false' 2021-03-15T22:36:55.359 INFO:teuthology.orchestra.run.smithi076.stderr:osd.4: osd_enable_op_tracker = 'false' 2021-03-15T22:36:55.359 INFO:teuthology.orchestra.run.smithi076.stderr:osd.5: osd_enable_op_tracker = 'false' 2021-03-15T22:36:55.359 INFO:teuthology.orchestra.run.smithi076.stderr:osd.6: osd_enable_op_tracker = 'false' 2021-03-15T22:36:55.359 INFO:teuthology.orchestra.run.smithi076.stderr:osd.7: osd_enable_op_tracker = 'false' 2021-03-15T22:36:55.405 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:55.429 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:55.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:55.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:55.532 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:55.555 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:55.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:55.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:55.584 INFO:teuthology.orchestra.run.smithi076.stderr:nodeep-scrub is unset 2021-03-15T22:36:55.657 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:55.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:55.682 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:55.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:55.784 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:55.807 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:55.808 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:55.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:55.933 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:55.934 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:55.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:56.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[56514]: cluster 2021-03-15T22:36 2021-03-15T22:36:56.017 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[56514]: :54.310383+0000 osd.5 (osd.5) 72 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:36:56.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[56514]: audit 2021-03-15T22:36:55.037109+0000 mon.a (mon.0) 2611 : audit [INF] from='client.? 172.21.15.76:0/3441222555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:56.018 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[56514]: audit 2021-03-15T22:36:55.206675+0000 mon.c (mon.1) 87 : audit [DBG] from='client.? 172.21.15.76:0/2481200234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:56.018 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[62156]: cluster 2021-03-15T22:36: 2021-03-15T22:36:56.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[62156]: 54.310383+0000 osd.5 (osd.5) 72 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:36:56.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[62156]: audit 2021-03- 2021-03-15T22:36:56.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[62156]: 15T22:36:55.037109+0000 mon.a (mon.0) 2611 : audit [INF] from='client.? 172.21.15.76:0/3441222555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:56.019 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[62156]: 2021-03-15T22:36:56.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[62156]: audit 2021-03-15T22:36:56.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[62156]: 2021-03-15T22:36: 2021-03-15T22:36:56.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[62156]: 55.206675+0000 mon.c (mon.1) 87 2021-03-15T22:36:56.020 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:55 smithi076 conmon[62156]: : audit [DBG] from='client.? 172.21.15.76:0/2481200234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:56.036 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7 2021-03-15T22:36:56.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:55 smithi125 conmon[56739]: cluster 2021-03-15T22:36:54.310383+0000 osd.5 (osd.5) 72 : cluster [DBG] 6.7 scrub ok 2021-03-15T22:36:56.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:55 smithi125 conmon[56739]: audit 2021-03-15T22:36:55.037109+0000 mon.a (mon.0) 2611 : audit [INF] from='client.? 172.21.15.76:0/3441222555' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2021-03-15T22:36:56.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:55 smithi125 conmon[56739]: audit 2021-03-15T22:36:55.206675+0000 mon.c (mon.1) 87 : 2021-03-15T22:36:56.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:55 smithi125 conmon[56739]: audit [DBG] from='client.? 172.21.15.76:0/2481200234' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2021-03-15T22:36:56.060 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:56.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:56.061 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.7' 2021-03-15T22:36:56.162 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:56.185 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:56.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:56.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:56.288 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:56.311 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:56.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:56.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:56.411 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:36:56.412 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:36:56.413 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:56.435 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:56.437 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:56.538 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:56.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:56.562 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:56.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:56.664 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:36:56.687 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:56.688 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:56.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:36:56.767 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: cluster 2021-03-15T22:36:55 2021-03-15T22:36:56.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: .242358+0000 mgr.y (mgr.14144) 1145 : cluster [DBG] pgmap v1726: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:36:56.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: cluster 2021-03-15T22:36:55. 2021-03-15T22:36:56.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: 451089+0000 osd.4 (osd.4) 210 : 2021-03-15T22:36:56.768 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: cluster [DBG] 4.1 scrub ok 2021-03-15T22:36:56.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: cluster 2021-03-15T22:36:56.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: 2021-03-15T22:36 2021-03-15T22:36:56.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: :55.472352+0000 osd.1 (osd. 2021-03-15T22:36:56.769 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: 1) 202 : cluster [DBG] 2021-03-15T22:36:56.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: 5.1d scrub ok 2021-03-15T22:36:56.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: cluster 2021-03-15T22:36:56.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: 2021-03-15T22:36:55.507093+0000 2021-03-15T22:36:56.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: osd.6 (osd.6) 160 : cluster 2021-03-15T22:36:56.770 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: [DBG] 1.7 scrub ok 2021-03-15T22:36:56.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:36:56.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: 03-15T22:36:55.567860+0000 mon.a ( 2021-03-15T22:36:56.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: mon.0) 2612 : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:36:56.771 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: cluster 2021-03-15T22:36 2021-03-15T22:36:56.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: :55.567894+0000 mon.a (mon.0) 2613 : cluster [INF] 2021-03-15T22:36:56.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: Cluster is now healthy 2021-03-15T22:36:56.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: audit 2021-03-15T22:36:56.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: 2021-03-15T22:36:55. 2021-03-15T22:36:56.772 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: 578292+0000 mon.a (mon.0) 2614 : audit 2021-03-15T22:36:56.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: [INF] from='client.? 172.21.15.76:0/3441222555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:36:56.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:36:56.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: 03-15T22:36:55.578361+0000 2021-03-15T22:36:56.773 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[62156]: mon.a (mon.0) 2615 : cluster [DBG] osdmap e1050: 8 total, 8 up, 8 in 2021-03-15T22:36:56.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: cluster 2021-03-15T22:36:55. 2021-03-15T22:36:56.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: 242358+0000 mgr.y (mgr.14144) 1145 : cluster [DBG] pgmap v1726: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:36:56.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: cluster 2021-03-15T22:36:55.451089+0000 osd.4 (osd.4) 210 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:36:56.774 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: cluster 2021-03-15T22:36:55.472352+0000 osd.1 (osd.1) 202 : cluster [DBG] 5.1d scrub ok 2021-03-15T22:36:56.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: cluster 2021-03-15T22:36:55.507093+0000 osd.6 (osd.6) 160 : cluster [DBG] 1.7 scrub ok 2021-03-15T22:36:56.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: cluster 2021-03-15 2021-03-15T22:36:56.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: T22:36:55.567860+0000 2021-03-15T22:36:56.775 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: mon.a (mon.0) 2612 2021-03-15T22:36:56.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: : cluster [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:36:56.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: cluster 2021 2021-03-15T22:36:56.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: -03-15T22:36:55. 2021-03-15T22:36:56.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: 567894+0000 mon.a (mon.0 2021-03-15T22:36:56.776 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: ) 2613 : cluster [INF] Cluster is now healthy 2021-03-15T22:36:56.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: 2021-03-15T22:36:56.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: audit 2021-03-15T22:36:56.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: 2021-03-15T22:36 2021-03-15T22:36:56.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: :55.578292+0000 mon.a ( 2021-03-15T22:36:56.777 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: mon.0) 2614 : audit [INF] 2021-03-15T22:36:56.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: from='client.? 172.21.15.76:0/3441222555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:36:56.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: cluster 2021-03 2021-03-15T22:36:56.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: -15T22:36:55.578361 2021-03-15T22:36:56.778 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: +0000 mon.a (mon.0) 2021-03-15T22:36:56.779 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56514]: 2615 : cluster [DBG] osdmap e1050: 8 total, 8 up, 8 in 2021-03-15T22:36:56.789 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:56.812 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:56.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:56.813 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:56.915 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:56.936 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:56.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:56.938 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:57.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: cluster 2021-03-15T22:36:55.242358+0000 mgr.y (mgr.14144) 1145 : cluster [DBG] pgmap v1726: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1.0 KiB/s rd, 1 op/s 2021-03-15T22:36:57.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:36:57.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: 15T22:36:55.451089+0000 osd.4 (osd.4) 210 : cluster [DBG] 4.1 scrub ok 2021-03-15T22:36:57.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: cluster 2021-03-15T22:36:55.472352+0000 osd.1 (osd.1) 202 : cluster 2021-03-15T22:36:57.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: [DBG] 5.1d scrub ok 2021-03-15T22:36:57.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: cluster 2021-03-15T22:36:57.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: 2021-03-15T22:36 2021-03-15T22:36:57.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: :55.507093+0000 osd.6 (osd.6) 2021-03-15T22:36:57.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: 160 : cluster [DBG] 1.7 scrub ok 2021-03-15T22:36:57.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: cluster 2021 2021-03-15T22:36:57.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: -03-15T22:36:55.567860 2021-03-15T22:36:57.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: +0000 mon.a (mon.0) 2612 : cluster 2021-03-15T22:36:57.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: [INF] Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2021-03-15T22:36:57.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:36:57.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: 22:36:55.567894+0000 mon.a (mon 2021-03-15T22:36:57.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: .0) 2613 : cluster [INF] Cluster is now healthy 2021-03-15T22:36:57.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: audit 2021-03-15T22:36:57.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: 2021-03-15T22:36: 2021-03-15T22:36:57.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: 55.578292+0000 mon.a (mon.0) 2021-03-15T22:36:57.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: 2614 : audit [INF] from='client.? 172.21.15.76:0/3441222555' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2021-03-15T22:36:57.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: cluster 2021- 2021-03-15T22:36:57.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: 03-15T22:36:55.578361+0000 2021-03-15T22:36:57.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: mon.a (mon.0) 2615 : cluster [DBG] 2021-03-15T22:36:57.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:56 smithi125 conmon[56739]: osdmap e1050: 8 total, 8 up, 8 in 2021-03-15T22:36:57.043 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:57.065 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:57.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:57.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:57.168 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:57.191 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:57.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:57.192 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:57.242 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: debug 2021-03-15T22:36:56.900+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:36:57.242 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:36:57.242 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.243 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:36:57.243 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.243 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:36:57.243 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.244 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.244 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:36:57.244 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: debug 2021-03-15T22:36:56.901+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:36:57.244 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:36:57.244 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.245 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:36:57.245 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.245 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:36:57.245 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.246 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.246 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:36:57.246 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: debug 2021-03-15T22:36:56.901+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:36:57.246 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:36:57.246 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.247 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:36:57.247 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.247 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:36:57.247 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.247 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.248 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:36:57.248 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: debug 2021-03-15T22:36:56.902+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:36:57.248 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:36:57.248 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.248 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:36:57.249 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.249 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:36:57.249 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.249 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.250 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:36:57.250 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: debug 2021-03-15T22:36:56.902+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:36:57.250 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:36:57.250 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.250 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:36:57.251 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.251 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:36:57.251 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.251 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.251 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:36:57.252 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: debug 2021-03-15T22:36:56.903+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:36:57.252 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:36:57.252 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.252 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:36:57.253 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.253 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:36:57.253 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.253 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.253 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:36:57.254 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: debug 2021-03-15T22:36:56.903+0000 7fc66c9ed700 -1 log_channel(cephadm) log [ERR] : Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:36:57.254 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: Traceback (most recent call last): 2021-03-15T22:36:57.254 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.254 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: if self._apply_service(spec): 2021-03-15T22:36:57.254 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.255 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: forcename=name) 2021-03-15T22:36:57.255 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.255 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.255 INFO:journalctl@ceph.mgr.y.smithi076.stdout:Mar 15 22:36:56 smithi076 conmon[56844]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:36:57.293 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:57.315 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:57.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:57.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:57.418 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:57.441 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:57.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:57.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:57.543 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:57.566 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:57.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:57.668 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:57.692 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:57.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:57.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:57.794 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6 2021-03-15T22:36:57.796 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cluster 2021-03-15T22:36:56.029249+0000 osd.0 (osd 2021-03-15T22:36:57.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: .0) 172 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:36:57.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cluster 2021-03-15T22:36:56.462312+0000 osd.6 (osd.6) 161 : cluster [DBG] 3.a scrub ok 2021-03-15T22:36:57.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: audit 2021-03-15T22:36:56.618661+0000 mon.a (mon.0) 2616 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:36:57.797 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cluster 2021-03-15T22:36:56.694437+0000 osd.7 (osd.7) 199 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:57.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cluster 2021-03-15T22:36:56.696658+0000 osd.7 (osd.7) 200 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:57.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:36:56.897287+0000 mgr.y (mgr.14144) 1146 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:36:57.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: audit 2021-03-15T22:36:56.900680+0000 mon.a (mon.0) 2617 : audit [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:36:57.798 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:36:56.901591+0000 mgr.y (mgr.14144) 1147 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:36:57.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:36:57.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:36:57.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.799 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:36:57.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:36:57.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:36:56.902081+0000 mgr.y (mgr.14144) 1148 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:36:57.800 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:36:57.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:36:57.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.801 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:36:57.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.802 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.803 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:36:57.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cluster 2021-03-15T22:36:56.029249+0000 2021-03-15T22:36:57.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: osd.0 (osd.0) 172 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:36:57.805 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cluster 2021-03-15T22:36:56.462312+0000 osd.6 (osd.6) 161 : cluster [DBG] 3.a scrub ok 2021-03-15T22:36:57.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: audit 2021-03-15T22:36:56.618661+0000 mon.a (mon.0) 2616 : audit [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:36:57.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cluster 2021-03-15T22:36:56.694437+0000 osd.7 (osd.7) 199 : cluster [DBG] purged_snaps scrub starts 2021-03-15T22:36:57.806 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cluster 2021-03-15T22:36:56.696658+0000 osd.7 (osd.7) 200 : cluster [DBG] purged_snaps scrub ok 2021-03-15T22:36:57.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:36:56.897287+0000 mgr.y (mgr.14144) 1146 : cephadm [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:36:57.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: audit 2021-03-15T22:36:56.900680+0000 mon.a (mon.0) 2617 : audit [INF] 2021-03-15T22:36:57.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:36:57.807 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:36:56.901591+0000 mgr.y (mgr 2021-03-15T22:36:57.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: .14144) 1147 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:36:57.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:36:57.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.808 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:36:57.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:36:57.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.809 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:36:57.810 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:36:56.902081+0000 mgr.y (mgr.14144) 1148 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:36:57.810 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:36:57.810 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.810 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:36:57.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:36:57.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.811 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:36:57.812 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:36:56.902496+0000 mgr.y (mgr.14144) 1149 : cephadm [ERR] 2021-03-15T22:36:57.812 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:36:56.902496+0000 mgr.y (mgr.14144) 1149 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:36:57.814 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:36:57.814 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:36:57.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:36:57.815 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:36:57.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:36:56.902910+0000 mgr.y (mgr.14144) 1150 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:36:57.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:36:57.816 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:36:57.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:36:57.817 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:36:57.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:36:56.903316+0000 mgr.y (mgr.14144) 1151 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:36:57.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:36:57.818 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:36:57.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:36:57.819 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.820 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.820 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:36:57.820 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:36:56.903719+0000 mgr.y (mgr.14144) 1152 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:36:57.820 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:36:57.822 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph daemon osd.3 dump_historic_ops 2021-03-15T22:36:57.822 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.823 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:36:57.823 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.823 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:36:57.823 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.824 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.824 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:36:57.824 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: cephadm 2021-03-15T22:36:56.904114+0000 mgr.y (mgr.14144) 1153 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:36:57.824 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: Traceback (most recent call last): 2021-03-15T22:36:57.824 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.825 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: if self._apply_service(spec): 2021-03-15T22:36:57.825 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.825 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: forcename=name) 2021-03-15T22:36:57.825 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.826 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.826 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[56514]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:36:57.826 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:36:57.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:36:57.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:36:57.827 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:36:57.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.828 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:36:57.829 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:36:56.902910+0000 mgr.y (mgr.14144) 1150 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:36:57.829 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:36:57.829 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.829 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:36:57.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:36:57.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.830 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:36:57.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:36: 2021-03-15T22:36:57.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: 56.903316+0000 mgr.y (mgr.14144) 1151 : cephadm [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:36:57.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:36:57.831 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:36:57.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:36:57.832 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:36:57.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:36:56.903719+0000 mgr.y (mgr.14144) 1152 : cephadm 2021-03-15T22:36:57.833 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:36:57.834 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:36:57.834 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.834 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:36:57.834 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.834 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:36:57.835 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.835 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.835 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:36:57.835 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: cephadm 2021-03-15T22:36:56.904114+0000 mgr.y (mgr.14144) 1153 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:36:57.836 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: Traceback (most recent call last): 2021-03-15T22:36:57.836 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:57.836 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: if self._apply_service(spec): 2021-03-15T22:36:57.836 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:57.836 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: forcename=name) 2021-03-15T22:36:57.837 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:57.837 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:57.837 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:57 smithi076 conmon[62156]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:36:57.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:57.839 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:57.839 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.6' 2021-03-15T22:36:57.941 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:57.963 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:57.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:57.964 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:58.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cluster 2021-03-15T22:36:56.029249+0000 osd.0 ( 2021-03-15T22:36:58.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: osd.0) 172 : cluster [DBG] 7.9 scrub ok 2021-03-15T22:36:58.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cluster 2021-03-15T22:36:56.462312+0000 2021-03-15T22:36:58.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: osd.6 (osd.6) 161 : cluster [DBG] 3.a scrub ok 2021-03-15T22:36:58.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: audit 2021- 2021-03-15T22:36:58.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: 03-15T22:36:56. 2021-03-15T22:36:58.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: 618661+0000 mon.a (mon.0) 2616 : audit 2021-03-15T22:36:58.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: [DBG] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2021-03-15T22:36:58.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:36:58.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: -15T22:36:56.694437 2021-03-15T22:36:58.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: +0000 osd.7 (osd.7) 199 : cluster [DBG] 2021-03-15T22:36:58.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: purged_snaps scrub starts 2021-03-15T22:36:58.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cluster 2021-03- 2021-03-15T22:36:58.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: 15T22:36:56.696658+0000 2021-03-15T22:36:58.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: osd.7 (osd.7) 200 : cluster [DBG] 2021-03-15T22:36:58.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: purged_snaps scrub ok 2021-03-15T22:36:58.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cephadm 2021-03-15T 2021-03-15T22:36:58.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: 22:36:56.897287+0000 mgr.y (mgr.14144) 1146 : cephadm 2021-03-15T22:36:58.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: [INF] ALL cephadm checks are disabled, use 'ceph config set mgr mgr/cephadm/config_checks_enabled true' to enable 2021-03-15T22:36:58.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: audit 2021-03-15T22:36:58.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: 2021-03-15T22:36:56. 2021-03-15T22:36:58.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: 900680+0000 mon.a (mon.0) 2617 : audit 2021-03-15T22:36:58.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: [INF] from='mgr.14144 172.21.15.76:0/3931659084' entity='mgr.y' 2021-03-15T22:36:58.042 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cephadm 2021- 2021-03-15T22:36:58.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: 03-15T22:36:56 2021-03-15T22:36:58.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: .901591+0000 mgr.y (mgr. 2021-03-15T22:36:58.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: 14144) 1147 : cephadm [ERR] Failed to apply mgr spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='x'), HostPlacementSpec(hostname='smithi076', network='', name='y')]), 'service_type': 'mgr', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name mgr.x already in use 2021-03-15T22:36:58.043 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:36:58.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:58.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:36:58.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:58.044 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:36:58.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:58.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:58.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name mgr.x already in use 2021-03-15T22:36:58.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:36:56.902081+0000 mgr.y (mgr.14144) 1148 : cephadm [ERR] Failed to apply rgw.foo.a spec RGWSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='foo.a')]), 'service_type': 'rgw', 'service_id': 'foo.a', 'unmanaged': False, 'preview_only': False, 'config': None, 'rgw_realm': None, 'rgw_zone': None, 'rgw_frontend_port': None, 'rgw_frontend_ssl_certificate': None, 'rgw_frontend_ssl_key': None, 'ssl': False}): name rgw.foo.a already in use 2021-03-15T22:36:58.045 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:36:58.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:58.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:36:58.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:58.046 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:36:58.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:58.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:58.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name rgw.foo.a already in use 2021-03-15T22:36:58.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:36:56.902496+0000 mgr.y (mgr.14144) 1149 : cephadm [ERR] Failed to apply iscsi.iscsi spec IscsiServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='iscsi.a')]), 'service_type': 'iscsi', 'service_id': 'iscsi', 'unmanaged': False, 'preview_only': False, 'config': None, 'pool': 'iscsi', 'trusted_ip_list': None, 'api_port': None, 'api_user': 'user', 'api_password': 'password', 'api_secure': None, 'ssl_cert': None, 'ssl_key': None}): name iscsi.iscsi.a already in use 2021-03-15T22:36:58.047 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:36:58.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:58.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:36:58.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:58.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:36:58.048 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:58.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:58.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name iscsi.iscsi.a already in use 2021-03-15T22:36:58.049 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:36:56.902910+0000 mgr.y (mgr.14144) 1150 : cephadm [ERR] Failed to apply prometheus spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'prometheus', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name prometheus.a already in use 2021-03-15T22:36:58.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:36:58.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:58.050 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:36:58.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:58.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:36:58.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:58.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:58.051 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name prometheus.a already in use 2021-03-15T22:36:58.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: 2021-03-15T22:36:58.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:36:56.903316+0000 mgr.y (mgr.14144) 1151 : cephadm 2021-03-15T22:36:58.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: [ERR] Failed to apply node-exporter spec ServiceSpec({'placement': PlacementSpec(count=2, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a'), HostPlacementSpec(hostname='smithi125', network='', name='b')]), 'service_type': 'node-exporter', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name node-exporter.a already in use 2021-03-15T22:36:58.052 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:36:58.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:58.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:36:58.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:58.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:36:58.053 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:58.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:58.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name node-exporter.a already in use 2021-03-15T22:36:58.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:36:56.903719+0000 mgr.y (mgr.14144) 1152 : cephadm [ERR] Failed to apply alertmanager spec AlertManagerSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi076', network='', name='a')]), 'service_type': 'alertmanager', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None, 'user_data': {}}): name alertmanager.a already in use 2021-03-15T22:36:58.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:36:58.054 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:58.055 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:36:58.055 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:58.055 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:36:58.055 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:58.056 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:58.056 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name alertmanager.a already in use 2021-03-15T22:36:58.056 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: cephadm 2021-03-15T22:36:56.904114+0000 mgr.y (mgr.14144) 1153 : cephadm [ERR] Failed to apply grafana spec ServiceSpec({'placement': PlacementSpec(count=1, hosts=[HostPlacementSpec(hostname='smithi125', network='', name='a')]), 'service_type': 'grafana', 'service_id': None, 'unmanaged': False, 'preview_only': False, 'config': None}): name grafana.a already in use 2021-03-15T22:36:58.056 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: Traceback (most recent call last): 2021-03-15T22:36:58.056 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 460, in _apply_all_services 2021-03-15T22:36:58.057 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: if self._apply_service(spec): 2021-03-15T22:36:58.057 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 591, in _apply_service 2021-03-15T22:36:58.057 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: forcename=name) 2021-03-15T22:36:58.057 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: File "/usr/share/ceph/mgr/cephadm/module.py", line 548, in get_unique_name 2021-03-15T22:36:58.057 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: f'name {daemon_type}.{forcename} already in use') 2021-03-15T22:36:58.058 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:57 smithi125 conmon[56739]: orchestrator._interface.OrchestratorValidationError: name grafana.a already in use 2021-03-15T22:36:58.065 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:58.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:58.091 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:58.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:58.134 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:58.192 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:58.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:58.216 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:58.217 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:58.318 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:58.340 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:58.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:58.342 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:58.443 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:58.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:58.467 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:58.467 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:58.569 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:58.591 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:58.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:58.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:58.607 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[56514]: cluster 2021-03-15T22:36:56.817486+0000 osd.7 (osd.7) 201 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:36:58.608 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[56514]: cluster 2021-03-15T22:36:57.242886+0000 mgr.y (mgr.14144) 1154 : cluster [DBG] pgmap v1728: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:36:58.608 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[56514]: cluster 2021-03-15T22:36:57.314470+0000 osd.5 (osd.5) 73 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:36:58.608 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[56514]: cluster 2021-03-15T22:36:57.441891+0000 osd.4 (osd.4) 211 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:36:58.608 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[56514]: cluster 2021-03-15T22:36:57.585196+0000 mon.a (mon.0) 2618 : cluster [DBG] osdmap e1051: 8 total, 8 up, 8 in 2021-03-15T22:36:58.608 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[56514]: audit 2021-03-15T22:36:58.352536+0000 mon.a (mon.0) 2619 : audit [INF] from='client.? 172.21.15.76:0/547209345' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:36:58.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: cluster 2021-03-15T22:36: 2021-03-15T22:36:58.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: 56.817486+0000 osd.7 (osd.7) 201 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:36:58.609 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: cluster 2021-03-15T22:36:58.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: 2021-03-15T22:36:57.242886+0000 2021-03-15T22:36:58.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: mgr.y (mgr.14144) 1154 : cluster [DBG] pgmap v1728: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:36:58.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: 2021-03-15T22:36:58.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:36:58.610 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: 03-15T22:36:57.314470+0000 osd.5 (osd 2021-03-15T22:36:58.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: .5) 73 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:36:58.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: cluster 2021-03-15 2021-03-15T22:36:58.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: T22:36:57.441891+0000 osd.4 (osd.4) 2021-03-15T22:36:58.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: 211 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:36:58.611 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: cluster 2021-03-15T22:36:57 2021-03-15T22:36:58.612 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: .585196+0000 mon.a (mon.0) 2618 : cluster [DBG] 2021-03-15T22:36:58.612 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: osdmap e1051: 8 total, 8 up, 8 in 2021-03-15T22:36:58.612 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: audit 2021 2021-03-15T22:36:58.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: -03-15T22:36:58.352536+0000 mon.a (mon.0) 2619 : audit [INF] 2021-03-15T22:36:58.613 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:58 smithi076 conmon[62156]: from='client.? 172.21.15.76:0/547209345' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:36:58.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:58.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:58.717 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:58.718 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:58.819 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:58.843 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:58.843 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:58.909 INFO:teuthology.orchestra.run.smithi076.stderr:set pool 6 pgp_num to 32 2021-03-15T22:36:58.945 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:58.966 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:58.968 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:59.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: cluster 2021-03-15T22:36:56.817486+0000 osd.7 (osd 2021-03-15T22:36:59.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: .7) 201 : cluster [DBG] 6.3 scrub ok 2021-03-15T22:36:59.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: cluster 2021-03-15T22:36:57.242886+0000 mgr.y 2021-03-15T22:36:59.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: (mgr.14144) 1154 : cluster [DBG] pgmap v1728: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2021-03-15T22:36:59.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: cluster 2021-03-15 2021-03-15T22:36:59.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: T22:36:57.314470+0000 osd.5 2021-03-15T22:36:59.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: (osd.5) 73 : cluster [DBG] 5.7 scrub ok 2021-03-15T22:36:59.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: 2021-03-15T22:36:59.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: cluster 2021-03-15T 2021-03-15T22:36:59.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: 22:36:57.441891+0000 osd.4 ( 2021-03-15T22:36:59.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: osd.4) 211 : cluster [DBG] 7.1 scrub ok 2021-03-15T22:36:59.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: cluster 2021-03-15T22:36:59.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: 2021-03-15T22: 2021-03-15T22:36:59.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: 36:57.585196+0000 mon.a (mon.0 2021-03-15T22:36:59.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: ) 2618 : cluster [DBG] osdmap e1051: 8 total, 8 up, 8 in 2021-03-15T22:36:59.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: audit 2021-03-15T22:36:59.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: 2021-03-15T22:36 2021-03-15T22:36:59.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: :58.352536+0000 mon.a (mon.0) 2619 2021-03-15T22:36:59.041 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:58 smithi125 conmon[56739]: : audit [INF] from='client.? 172.21.15.76:0/547209345' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]: dispatch 2021-03-15T22:36:59.069 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:59.090 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:59.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:59.092 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:59.193 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:59.215 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:59.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:59.216 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:59.262 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2021-03-15T22:36:59.317 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:36:59.339 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:59.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:59.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:36:59.442 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:36:59.468 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:59.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:59.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:36:59.555 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd set noscrub 2021-03-15T22:36:59.570 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:36:59.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:59.597 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:36:59.597 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:36:59.698 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:59.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:59.723 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:59.723 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:59.825 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:36:59.847 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:59.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:59.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:36:59.949 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:36:59.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:36:59.972 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:36:59.973 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:36:59.982 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[56514]: cluster 2021-03-15T22:36:57.986762+0000 osd.0 (osd.0 2021-03-15T22:36:59.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[56514]: ) 173 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:36:59.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[56514]: cluster 2021-03-15T22:36:58.296435+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:36:59.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[56514]: cluster 2021-03-15T22:36:58.499124+0000 osd.6 (osd.6) 162 : cluster [DBG] 5.11 deep-scrub ok 2021-03-15T22:36:59.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[56514]: audit 2021-03-15T22:36:58.904002+0000 mon.a (mon.0) 2620 : audit [INF] from='client.? 172.21.15.76:0/547209345' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:36:59.984 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[56514]: cluster 2021-03-15T22:36:58.904068+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e1052: 8 total, 8 up, 8 in 2021-03-15T22:36:59.985 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: cluster 2021-03-15T22:36:57 2021-03-15T22:36:59.985 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: .986762+0000 osd.0 (osd.0) 173 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:36:59.985 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: cluster 2021 2021-03-15T22:36:59.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: -03-15T22:36:58.296435+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:36:59.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: 2021-03-15T22:36:59.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: cluster 2021-03-15T22 2021-03-15T22:36:59.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: :36:58.499124+0000 osd.6 (osd.6) 2021-03-15T22:36:59.986 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: 162 : cluster [DBG] 5.11 deep-scrub ok 2021-03-15T22:36:59.987 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: audit 2021- 2021-03-15T22:36:59.987 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: 03-15T22:36:58.904002+0000 mon.a (mon. 2021-03-15T22:36:59.987 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: 0) 2620 : audit [INF] from='client.? 172.21.15.76:0/547209345' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:36:59.987 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: cluster 2021-03-15T22:36:59.988 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: 2021-03-15T22:36:58. 2021-03-15T22:36:59.988 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: 904068+0000 mon.a (mon.0) 2621 : cluster [DBG] osdmap e1052: 8 total, 8 up, 8 in 2021-03-15T22:36:59.988 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:36:59 smithi076 conmon[62156]: 2021-03-15T22:36:59.990 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:36:59.991 INFO:teuthology.orchestra.run.smithi076.stderr:Inferring config /var/lib/ceph/8a572802-85db-11eb-9091-001a4aab830c/mon.a/config 2021-03-15T22:37:00.016 INFO:tasks.cephadm.ceph_manager.ceph:removing pool_name unique_pool_0 2021-03-15T22:37:00.016 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:b658018acf46e32d2ec283d302076352e17476f5 shell --fsid 8a572802-85db-11eb-9091-001a4aab830c -- ceph osd pool rm unique_pool_0 unique_pool_0 --yes-i-really-really-mean-it 2021-03-15T22:37:00.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: cluster 2021-03-15T22:36:57.986762+0000 osd.0 (osd.0) 173 : cluster [DBG] 5.8 scrub ok 2021-03-15T22:37:00.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: 2021-03-15T22:37:00.037 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: cluster 2021-03-15T22:36:58.296435+0000 osd.5 (osd.5) 74 : cluster [DBG] 6.4 scrub ok 2021-03-15T22:37:00.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: cluster 2021-03-15T22:37:00.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: 2021-03-15T22:36:58.499124+0000 osd.6 (osd.6) 162 : cluster 2021-03-15T22:37:00.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: [DBG] 5.11 deep-scrub ok 2021-03-15T22:37:00.038 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: audit 2021-03-15T22 2021-03-15T22:37:00.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: :36:58.904002+0000 mon.a (mon. 2021-03-15T22:37:00.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: 0) 2620 : audit [INF] from='client.? 172.21.15.76:0/547209345' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pgp_num", "val": "32"}]': finished 2021-03-15T22:37:00.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: cluster 2021-03 2021-03-15T22:37:00.039 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: -15T22:36:58.904068+0000 2021-03-15T22:37:00.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: mon.a (mon.0) 2621 : cluster [DBG] osdmap e1052: 8 total, 8 up, 8 in 2021-03-15T22:37:00.040 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:36:59 smithi125 conmon[56739]: 2021-03-15T22:37:00.074 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4 2021-03-15T22:37:00.101 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:37:00.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:37:00.103 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.4' 2021-03-15T22:37:00.204 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1 2021-03-15T22:37:00.227 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:37:00.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:37:00.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.1' 2021-03-15T22:37:00.331 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2 2021-03-15T22:37:00.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:37:00.354 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:37:00.354 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.2' 2021-03-15T22:37:00.446 INFO:teuthology.orchestra.run.smithi076.stdout:ERROR: (22) Invalid argument 2021-03-15T22:37:00.446 INFO:teuthology.orchestra.run.smithi076.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2021-03-15T22:37:00.455 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:37:00.479 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:37:00.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:37:00.481 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:37:00.582 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:37:00.592 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:37:00 smithi076 conmon[56514]: cluster 2021-03-15T22:36:59.243229+0000 mgr.y (mgr.14144) 1155 : cluster [DBG] pgmap v1731: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:37:00.593 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:37:00 smithi076 conmon[56514]: cluster 2021-03-15T22:36:59.531642+0000 osd.1 (osd.1) 203 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:37:00.593 INFO:journalctl@ceph.mon.a.smithi076.stdout:Mar 15 22:37:00 smithi076 conmon[56514]: cluster 2021-03-15T22:37:00.373235+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e1053: 8 total, 8 up, 8 in 2021-03-15T22:37:00.593 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:37:00 smithi076 conmon[62156]: cluster 2021-03-15T22:36:59.243229+0000 mgr.y (mgr.14144) 1155 : cluster [DBG] pgmap v1731: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:37:00.594 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:37:00 smithi076 conmon[62156]: cluster 2021-03-15T22:36:59.531642+0000 osd.1 (osd.1) 203 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:37:00.594 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:37:00 smithi076 conmon[62156]: cluster 2021- 2021-03-15T22:37:00.594 INFO:journalctl@ceph.mon.c.smithi076.stdout:Mar 15 22:37:00 smithi076 conmon[62156]: 03-15T22:37:00.373235+0000 mon.a (mon.0) 2622 : cluster [DBG] osdmap e1053: 8 total, 8 up, 8 in 2021-03-15T22:37:00.605 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:37:00.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:37:00.607 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:37:00.708 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0 2021-03-15T22:37:00.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:37:00.732 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:37:00.732 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.0' 2021-03-15T22:37:00.787 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:37:00 smithi125 conmon[56739]: cluster 2021-03-15T22:36:59.243229+0000 mgr.y ( 2021-03-15T22:37:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:37:00 smithi125 conmon[56739]: mgr.14144) 1155 : cluster [DBG] pgmap v1731: 148 pgs: 148 active+clean; 793 MiB data, 2.1 GiB used, 713 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2021-03-15T22:37:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:37:00 smithi125 conmon[56739]: cluster 2021-03-15T22:36:59.531642+0000 osd.1 (osd.1) 203 : cluster [DBG] 7.3 scrub ok 2021-03-15T22:37:00.788 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:37:00 smithi125 conmon[56739]: cluster 2021-03-15T22:37:00.373235+0000 mon.a (mon. 2021-03-15T22:37:00.789 INFO:journalctl@ceph.mon.b.smithi125.stdout:Mar 15 22:37:00 smithi125 conmon[56739]: 0) 2622 : cluster [DBG] osdmap e1053: 8 total, 8 up, 8 in 2021-03-15T22:37:00.789 INFO:journalctl@ceph.iscsi.iscsi.a.smithi125.stdout:Mar 15 22:37:00 smithi125 conmon[73528]: debug there is no tcmu-runner data avaliable 2021-03-15T22:37:00.833 DEBUG:teuthology.orchestra.run.smithi125:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5 2021-03-15T22:37:00.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:37:00.862 INFO:teuthology.orchestra.run.smithi125.stderr:sudo: docker: command not found 2021-03-15T22:37:00.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi125 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.5' 2021-03-15T22:37:00.963 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:37:00.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:37:00.988 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:37:00.988 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:37:01.089 DEBUG:teuthology.orchestra.run.smithi076:> sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3 2021-03-15T22:37:01.112 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: docker: command not found 2021-03-15T22:37:01.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-15T22:37:01.114 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi076 with status 1: 'sudo docker kill -s 1 ceph-8a572802-85db-11eb-9091-001a4aab830c-osd.3' 2021-03-15T22:37:01.212 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-03-15T22:37:01.215 DEBUG:teuthology.task.console_log:Killing console logger for smithi076 2021-03-15T22:37:01.216 DEBUG:teuthology.task.console_log:Killing console logger for smithi125 2021-03-15T22:37:01.217 DEBUG:teuthology.exit:Finished running handlers